2023-05-22T15:12:43.092 INFO:root:teuthology version: 0.0.1.dev100+g01abd1e 2023-05-22T15:12:43.094 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_yuriw@teuthology --verbose --name yuriw-2023-05-22_14:44:12-fs-wip-yuri3-testing-2023-05-21-0740-reef-distro-default-smithi --archive /home/teuthworker/archive/yuriw-2023-05-22_14:44:12-fs-wip-yuri3-testing-2023-05-21-0740-reef-distro-default-smithi/7282496 --description fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snap-schedule} -- /home/teuthworker/archive/yuriw-2023-05-22_14:44:12-fs-wip-yuri3-testing-2023-05-21-0740-reef-distro-default-smithi/7282496/orig.config.yaml 2023-05-22T15:12:43.136 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-05-22T15:12:43.229 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2023-05-22_14:44:12-fs-wip-yuri3-testing-2023-05-21-0740-reef-distro-default-smithi/7282496 branch: wip-yuri3-testing-2023-05-21-0740-reef description: fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snap-schedule} email: yweinste@redhat.com first_in_suite: false job_id: '7282496' kernel: kdb: true sha1: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: yuriw-2023-05-22_14:44:12-fs-wip-yuri3-testing-2023-05-21-0740-reef-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: ubuntu os_version: '22.04' overrides: admin_socket: branch: wip-yuri3-testing-2023-05-21-0740-reef ceph: cephfs: ec_profile: - m=2 - k=2 - crush-failure-domain=osd conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 pid file: '' rados mon op timeout: 900 rados osd op timeout: 900 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug client: 20 debug finisher: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding log-whitelist: - OSD full dropping all updates - OSD near full - pausewr flag - failsafe engaged, dropping updates - failsafe disengaged, no longer dropping - is full \(reached quota - POOL_FULL - POOL_BACKFILLFULL sha1: 9c9ba4df22024d9d8d6cf9f98a4b96751fc78d0c ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 9c9ba4df22024d9d8d6cf9f98a4b96751fc78d0c thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-yuri3-testing-2023-05-21-0740-reef sha1: 9c9ba4df22024d9d8d6cf9f98a4b96751fc78d0c owner: scheduled_yuriw@teuthology priority: 75 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mgr.y - mds.a - mds.c - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - client.2 - - mon.b - mon.c - mgr.x - mds.b - mds.d - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - client.3 seed: 1398 sha1: 9c9ba4df22024d9d8d6cf9f98a4b96751fc78d0c sleep_before_teardown: 0 subset: 111/12000 suite: fs suite_branch: wip-yuri3-testing-2023-05-21-0740-reef suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_9c9ba4df22024d9d8d6cf9f98a4b96751fc78d0c/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 9c9ba4df22024d9d8d6cf9f98a4b96751fc78d0c syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi016.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLlVQ8mnKu7mabeHXD6q1SCUofl2/7TQGJtZoDkWRaEN8SjRlNYMe5o9bDtiZPZ2Z7+Mb61B2A6l71rddBvoLjg= smithi062.front.sepia.ceph.com: ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIA1xuKgcCm2/BCZTETfSU7EAds5jWkUYadR3nrazn66o tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - ceph: null - ceph-fuse: null - cephfs_test_runner: modules: - tasks.cephfs.test_snap_schedules teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 01abd1ed90624b09fd81d9cb3a79e2834351f0ca timestamp: 2023-05-22_14:44:12 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.23459 2023-05-22T15:12:43.230 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_9c9ba4df22024d9d8d6cf9f98a4b96751fc78d0c/qa; will attempt to use it 2023-05-22T15:12:43.230 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_9c9ba4df22024d9d8d6cf9f98a4b96751fc78d0c/qa/tasks 2023-05-22T15:12:43.231 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-05-22T15:12:43.237 INFO:teuthology.task.internal:Checking packages... 2023-05-22T15:12:43.265 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash '9c9ba4df22024d9d8d6cf9f98a4b96751fc78d0c' 2023-05-22T15:12:43.266 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-05-22T15:12:43.266 INFO:teuthology.packaging:ref: None 2023-05-22T15:12:43.267 INFO:teuthology.packaging:tag: None 2023-05-22T15:12:43.267 INFO:teuthology.packaging:branch: wip-yuri3-testing-2023-05-21-0740-reef 2023-05-22T15:12:43.267 INFO:teuthology.packaging:sha1: 9c9ba4df22024d9d8d6cf9f98a4b96751fc78d0c 2023-05-22T15:12:43.267 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=wip-yuri3-testing-2023-05-21-0740-reef 2023-05-22T15:12:43.406 INFO:teuthology.task.internal:Found packages for ceph version 18.0.0-3192-g9c9ba4df-1jammy 2023-05-22T15:12:43.406 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-05-22T15:12:43.413 INFO:teuthology.task.internal:no buildpackages task found 2023-05-22T15:12:43.413 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-05-22T15:12:43.419 INFO:teuthology.task.internal:Saving configuration 2023-05-22T15:12:43.434 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-05-22T15:12:43.440 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-05-22T15:12:43.468 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi016.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-05-22_14:44:12-fs-wip-yuri3-testing-2023-05-21-0740-reef-distro-default-smithi/7282496', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-05-22 15:07:05.114694', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIJSBVHse2WRw1rRC6Pgf8MKl8GYAQq2k8s3Jla0hldc3'} 2023-05-22T15:12:43.496 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi062.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-05-22_14:44:12-fs-wip-yuri3-testing-2023-05-21-0740-reef-distro-default-smithi/7282496', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-05-22 15:07:05.113244', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIA1xuKgcCm2/BCZTETfSU7EAds5jWkUYadR3nrazn66o'} 2023-05-22T15:12:43.496 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-05-22T15:12:43.502 INFO:teuthology.task.internal:roles: ubuntu@smithi016.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'] 2023-05-22T15:12:43.502 INFO:teuthology.task.internal:roles: ubuntu@smithi062.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3'] 2023-05-22T15:12:43.503 INFO:teuthology.run_tasks:Running task console_log... 2023-05-22T15:12:43.579 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fc0d87c0f70>, signals=[15]) 2023-05-22T15:12:43.581 INFO:teuthology.run_tasks:Running task internal.connect... 2023-05-22T15:12:43.588 INFO:teuthology.task.internal:Opening connections... 2023-05-22T15:12:43.589 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi016.front.sepia.ceph.com 2023-05-22T15:12:43.591 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi016.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-05-22T15:12:43.665 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi062.front.sepia.ceph.com 2023-05-22T15:12:43.666 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi062.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-05-22T15:12:43.744 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-05-22T15:12:43.751 DEBUG:teuthology.orchestra.run.smithi016:> uname -m 2023-05-22T15:12:43.756 INFO:teuthology.orchestra.run.smithi016.stdout:x86_64 2023-05-22T15:12:43.756 DEBUG:teuthology.orchestra.run.smithi016:> cat /etc/os-release 2023-05-22T15:12:43.803 INFO:teuthology.orchestra.run.smithi016.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2023-05-22T15:12:43.803 INFO:teuthology.orchestra.run.smithi016.stdout:NAME="Ubuntu" 2023-05-22T15:12:43.804 INFO:teuthology.orchestra.run.smithi016.stdout:VERSION_ID="22.04" 2023-05-22T15:12:43.804 INFO:teuthology.orchestra.run.smithi016.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2023-05-22T15:12:43.804 INFO:teuthology.orchestra.run.smithi016.stdout:VERSION_CODENAME=jammy 2023-05-22T15:12:43.804 INFO:teuthology.orchestra.run.smithi016.stdout:ID=ubuntu 2023-05-22T15:12:43.804 INFO:teuthology.orchestra.run.smithi016.stdout:ID_LIKE=debian 2023-05-22T15:12:43.804 INFO:teuthology.orchestra.run.smithi016.stdout:HOME_URL="https://www.ubuntu.com/" 2023-05-22T15:12:43.804 INFO:teuthology.orchestra.run.smithi016.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2023-05-22T15:12:43.805 INFO:teuthology.orchestra.run.smithi016.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2023-05-22T15:12:43.805 INFO:teuthology.orchestra.run.smithi016.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2023-05-22T15:12:43.805 INFO:teuthology.orchestra.run.smithi016.stdout:UBUNTU_CODENAME=jammy 2023-05-22T15:12:43.806 INFO:teuthology.lock.ops:Updating smithi016.front.sepia.ceph.com on lock server 2023-05-22T15:12:43.862 DEBUG:teuthology.orchestra.run.smithi062:> uname -m 2023-05-22T15:12:43.868 INFO:teuthology.orchestra.run.smithi062.stdout:x86_64 2023-05-22T15:12:43.869 DEBUG:teuthology.orchestra.run.smithi062:> cat /etc/os-release 2023-05-22T15:12:43.914 INFO:teuthology.orchestra.run.smithi062.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2023-05-22T15:12:43.914 INFO:teuthology.orchestra.run.smithi062.stdout:NAME="Ubuntu" 2023-05-22T15:12:43.914 INFO:teuthology.orchestra.run.smithi062.stdout:VERSION_ID="22.04" 2023-05-22T15:12:43.914 INFO:teuthology.orchestra.run.smithi062.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2023-05-22T15:12:43.914 INFO:teuthology.orchestra.run.smithi062.stdout:VERSION_CODENAME=jammy 2023-05-22T15:12:43.915 INFO:teuthology.orchestra.run.smithi062.stdout:ID=ubuntu 2023-05-22T15:12:43.915 INFO:teuthology.orchestra.run.smithi062.stdout:ID_LIKE=debian 2023-05-22T15:12:43.915 INFO:teuthology.orchestra.run.smithi062.stdout:HOME_URL="https://www.ubuntu.com/" 2023-05-22T15:12:43.915 INFO:teuthology.orchestra.run.smithi062.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2023-05-22T15:12:43.915 INFO:teuthology.orchestra.run.smithi062.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2023-05-22T15:12:43.915 INFO:teuthology.orchestra.run.smithi062.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2023-05-22T15:12:43.915 INFO:teuthology.orchestra.run.smithi062.stdout:UBUNTU_CODENAME=jammy 2023-05-22T15:12:43.916 INFO:teuthology.lock.ops:Updating smithi062.front.sepia.ceph.com on lock server 2023-05-22T15:12:43.936 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-05-22T15:12:43.947 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-05-22T15:12:43.952 INFO:teuthology.task.internal:Checking for old test directory... 2023-05-22T15:12:43.953 DEBUG:teuthology.orchestra.run.smithi016:> test '!' -e /home/ubuntu/cephtest 2023-05-22T15:12:43.956 DEBUG:teuthology.orchestra.run.smithi062:> test '!' -e /home/ubuntu/cephtest 2023-05-22T15:12:43.961 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-05-22T15:12:43.967 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-05-22T15:12:43.967 DEBUG:teuthology.orchestra.run.smithi016:> test -z $(ls -A /var/lib/ceph) 2023-05-22T15:12:44.001 DEBUG:teuthology.orchestra.run.smithi062:> test -z $(ls -A /var/lib/ceph) 2023-05-22T15:12:44.028 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-05-22T15:12:44.090 INFO:teuthology.run_tasks:Running task kernel... 2023-05-22T15:12:44.108 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-05-22T15:12:44.109 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-05-22T15:12:44.110 DEBUG:teuthology.orchestra.run.smithi016:> test -f /run/.containerenv -o -f /.dockerenv 2023-05-22T15:12:44.110 DEBUG:teuthology.orchestra.run.smithi062:> test -f /run/.containerenv -o -f /.dockerenv 2023-05-22T15:12:44.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-05-22T15:12:44.115 DEBUG:teuthology.orchestra.run.smithi062:> uname -r 2023-05-22T15:12:44.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-05-22T15:12:44.116 DEBUG:teuthology.orchestra.run.smithi016:> uname -r 2023-05-22T15:12:44.162 INFO:teuthology.orchestra.run.smithi062.stdout:5.15.0-56-generic 2023-05-22T15:12:44.163 INFO:teuthology.task.kernel:Running kernel on smithi062: 5.15.0-56-generic 2023-05-22T15:12:44.163 DEBUG:teuthology.orchestra.run.smithi062:> sudo apt-get clean 2023-05-22T15:12:44.164 INFO:teuthology.orchestra.run.smithi016.stdout:5.15.0-56-generic 2023-05-22T15:12:44.164 INFO:teuthology.task.kernel:Running kernel on smithi016: 5.15.0-56-generic 2023-05-22T15:12:44.164 DEBUG:teuthology.orchestra.run.smithi016:> sudo apt-get clean 2023-05-22T15:12:44.277 DEBUG:teuthology.orchestra.run.smithi016:> sudo apt-get update 2023-05-22T15:12:44.303 DEBUG:teuthology.orchestra.run.smithi062:> sudo apt-get update 2023-05-22T15:12:44.434 INFO:teuthology.orchestra.run.smithi016.stdout:Get:1 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB] 2023-05-22T15:12:44.473 INFO:teuthology.orchestra.run.smithi062.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2023-05-22T15:12:44.474 INFO:teuthology.orchestra.run.smithi062.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB] 2023-05-22T15:12:44.474 INFO:teuthology.orchestra.run.smithi062.stdout:Get:3 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB] 2023-05-22T15:12:44.525 INFO:teuthology.orchestra.run.smithi062.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [108 kB] 2023-05-22T15:12:44.543 INFO:teuthology.orchestra.run.smithi016.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2023-05-22T15:12:44.624 INFO:teuthology.orchestra.run.smithi016.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB] 2023-05-22T15:12:44.640 INFO:teuthology.orchestra.run.smithi016.stdout:Get:4 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [211 kB] 2023-05-22T15:12:44.714 INFO:teuthology.orchestra.run.smithi016.stdout:Get:5 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [385 kB] 2023-05-22T15:12:44.714 INFO:teuthology.orchestra.run.smithi062.stdout:Get:5 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [385 kB] 2023-05-22T15:12:44.734 INFO:teuthology.orchestra.run.smithi016.stdout:Get:6 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [111 kB] 2023-05-22T15:12:44.738 INFO:teuthology.orchestra.run.smithi016.stdout:Get:7 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [9,812 B] 2023-05-22T15:12:44.739 INFO:teuthology.orchestra.run.smithi016.stdout:Get:8 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [28.0 kB] 2023-05-22T15:12:44.740 INFO:teuthology.orchestra.run.smithi016.stdout:Get:9 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [250 kB] 2023-05-22T15:12:44.749 INFO:teuthology.orchestra.run.smithi016.stdout:Get:10 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [36.6 kB] 2023-05-22T15:12:44.750 INFO:teuthology.orchestra.run.smithi016.stdout:Get:11 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [604 B] 2023-05-22T15:12:44.750 INFO:teuthology.orchestra.run.smithi016.stdout:Get:12 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [529 kB] 2023-05-22T15:12:44.760 INFO:teuthology.orchestra.run.smithi016.stdout:Get:13 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [726 kB] 2023-05-22T15:12:44.779 INFO:teuthology.orchestra.run.smithi062.stdout:Get:6 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [211 kB] 2023-05-22T15:12:44.785 INFO:teuthology.orchestra.run.smithi062.stdout:Get:7 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [111 kB] 2023-05-22T15:12:44.789 INFO:teuthology.orchestra.run.smithi062.stdout:Get:8 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [9,812 B] 2023-05-22T15:12:44.789 INFO:teuthology.orchestra.run.smithi062.stdout:Get:9 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [28.0 kB] 2023-05-22T15:12:44.790 INFO:teuthology.orchestra.run.smithi062.stdout:Get:10 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [250 kB] 2023-05-22T15:12:44.794 INFO:teuthology.orchestra.run.smithi062.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [393 kB] 2023-05-22T15:12:44.800 INFO:teuthology.orchestra.run.smithi062.stdout:Get:12 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [36.6 kB] 2023-05-22T15:12:44.801 INFO:teuthology.orchestra.run.smithi062.stdout:Get:13 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [604 B] 2023-05-22T15:12:44.811 INFO:teuthology.orchestra.run.smithi016.stdout:Get:14 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [126 kB] 2023-05-22T15:12:44.811 INFO:teuthology.orchestra.run.smithi062.stdout:Get:14 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [726 kB] 2023-05-22T15:12:44.820 INFO:teuthology.orchestra.run.smithi062.stdout:Get:15 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [529 kB] 2023-05-22T15:12:44.844 INFO:teuthology.orchestra.run.smithi062.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [605 kB] 2023-05-22T15:12:44.860 INFO:teuthology.orchestra.run.smithi062.stdout:Get:17 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [126 kB] 2023-05-22T15:12:44.862 INFO:teuthology.orchestra.run.smithi062.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [171 kB] 2023-05-22T15:12:44.868 INFO:teuthology.orchestra.run.smithi062.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [14.5 kB] 2023-05-22T15:12:44.868 INFO:teuthology.orchestra.run.smithi062.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [28.3 kB] 2023-05-22T15:12:44.869 INFO:teuthology.orchestra.run.smithi062.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [260 kB] 2023-05-22T15:12:44.875 INFO:teuthology.orchestra.run.smithi062.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [38.7 kB] 2023-05-22T15:12:44.876 INFO:teuthology.orchestra.run.smithi062.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [604 B] 2023-05-22T15:12:44.881 INFO:teuthology.orchestra.run.smithi016.stdout:Get:15 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [14.6 kB] 2023-05-22T15:12:44.882 INFO:teuthology.orchestra.run.smithi016.stdout:Get:16 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [30.2 kB] 2023-05-22T15:12:44.885 INFO:teuthology.orchestra.run.smithi016.stdout:Get:17 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [5,828 B] 2023-05-22T15:12:44.886 INFO:teuthology.orchestra.run.smithi016.stdout:Get:18 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [252 B] 2023-05-22T15:12:44.911 INFO:teuthology.orchestra.run.smithi062.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [905 kB] 2023-05-22T15:12:44.935 INFO:teuthology.orchestra.run.smithi062.stdout:Get:25 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [14.6 kB] 2023-05-22T15:12:44.936 INFO:teuthology.orchestra.run.smithi062.stdout:Get:26 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [30.2 kB] 2023-05-22T15:12:44.939 INFO:teuthology.orchestra.run.smithi062.stdout:Get:27 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [5,828 B] 2023-05-22T15:12:44.939 INFO:teuthology.orchestra.run.smithi062.stdout:Get:28 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [252 B] 2023-05-22T15:12:44.982 INFO:teuthology.orchestra.run.smithi016.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [108 kB] 2023-05-22T15:12:45.038 INFO:teuthology.orchestra.run.smithi062.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [611 kB] 2023-05-22T15:12:45.050 INFO:teuthology.orchestra.run.smithi062.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [186 kB] 2023-05-22T15:12:45.054 INFO:teuthology.orchestra.run.smithi062.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [18.9 kB] 2023-05-22T15:12:45.055 INFO:teuthology.orchestra.run.smithi062.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [35.3 kB] 2023-05-22T15:12:45.056 INFO:teuthology.orchestra.run.smithi062.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [3,884 B] 2023-05-22T15:12:45.057 INFO:teuthology.orchestra.run.smithi062.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [8,452 B] 2023-05-22T15:12:45.058 INFO:teuthology.orchestra.run.smithi062.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [468 B] 2023-05-22T15:12:45.058 INFO:teuthology.orchestra.run.smithi062.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [33.7 kB] 2023-05-22T15:12:45.059 INFO:teuthology.orchestra.run.smithi062.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [40.9 kB] 2023-05-22T15:12:45.095 INFO:teuthology.orchestra.run.smithi062.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [10.2 kB] 2023-05-22T15:12:45.096 INFO:teuthology.orchestra.run.smithi062.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2023-05-22T15:12:45.096 INFO:teuthology.orchestra.run.smithi062.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [12.7 kB] 2023-05-22T15:12:45.116 INFO:teuthology.orchestra.run.smithi062.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [22.2 kB] 2023-05-22T15:12:45.118 INFO:teuthology.orchestra.run.smithi016.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [605 kB] 2023-05-22T15:12:45.130 INFO:teuthology.orchestra.run.smithi062.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [15.0 kB] 2023-05-22T15:12:45.136 INFO:teuthology.orchestra.run.smithi062.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [548 B] 2023-05-22T15:12:45.315 INFO:teuthology.orchestra.run.smithi016.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [393 kB] 2023-05-22T15:12:45.378 INFO:teuthology.orchestra.run.smithi016.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [171 kB] 2023-05-22T15:12:45.395 INFO:teuthology.orchestra.run.smithi016.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [14.5 kB] 2023-05-22T15:12:45.395 INFO:teuthology.orchestra.run.smithi016.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [28.3 kB] 2023-05-22T15:12:45.399 INFO:teuthology.orchestra.run.smithi016.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [260 kB] 2023-05-22T15:12:45.418 INFO:teuthology.orchestra.run.smithi016.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [38.7 kB] 2023-05-22T15:12:45.421 INFO:teuthology.orchestra.run.smithi016.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [604 B] 2023-05-22T15:12:45.585 INFO:teuthology.orchestra.run.smithi016.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [611 kB] 2023-05-22T15:12:46.147 INFO:teuthology.orchestra.run.smithi016.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [905 kB] 2023-05-22T15:12:46.255 INFO:teuthology.orchestra.run.smithi016.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [186 kB] 2023-05-22T15:12:46.273 INFO:teuthology.orchestra.run.smithi016.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [18.9 kB] 2023-05-22T15:12:46.274 INFO:teuthology.orchestra.run.smithi016.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [3,884 B] 2023-05-22T15:12:46.275 INFO:teuthology.orchestra.run.smithi016.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [35.3 kB] 2023-05-22T15:12:46.278 INFO:teuthology.orchestra.run.smithi016.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [8,452 B] 2023-05-22T15:12:46.279 INFO:teuthology.orchestra.run.smithi016.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [468 B] 2023-05-22T15:12:46.279 INFO:teuthology.orchestra.run.smithi016.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [40.9 kB] 2023-05-22T15:12:46.282 INFO:teuthology.orchestra.run.smithi016.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [33.7 kB] 2023-05-22T15:12:46.285 INFO:teuthology.orchestra.run.smithi016.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [10.2 kB] 2023-05-22T15:12:46.450 INFO:teuthology.orchestra.run.smithi016.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2023-05-22T15:12:46.533 INFO:teuthology.orchestra.run.smithi016.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [22.2 kB] 2023-05-22T15:12:46.622 INFO:teuthology.orchestra.run.smithi016.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [12.7 kB] 2023-05-22T15:12:46.651 INFO:teuthology.orchestra.run.smithi016.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [15.0 kB] 2023-05-22T15:12:46.703 INFO:teuthology.orchestra.run.smithi016.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [548 B] 2023-05-22T15:12:49.835 INFO:teuthology.orchestra.run.smithi062.stdout:Fetched 6,218 kB in 1s (5,822 kB/s) 2023-05-22T15:12:50.968 INFO:teuthology.orchestra.run.smithi016.stdout:Fetched 6,218 kB in 2s (2,647 kB/s) 2023-05-22T15:12:50.995 INFO:teuthology.orchestra.run.smithi062.stdout:Reading package lists... 2023-05-22T15:12:51.011 DEBUG:teuthology.orchestra.run.smithi062:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2023-05-22T15:12:51.072 INFO:teuthology.orchestra.run.smithi062.stdout:Reading package lists... 2023-05-22T15:12:51.247 INFO:teuthology.orchestra.run.smithi062.stdout:Building dependency tree... 2023-05-22T15:12:51.247 INFO:teuthology.orchestra.run.smithi062.stdout:Reading state information... 2023-05-22T15:12:51.397 INFO:teuthology.orchestra.run.smithi062.stdout:The following packages were automatically installed and are no longer required: 2023-05-22T15:12:51.398 INFO:teuthology.orchestra.run.smithi062.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2023-05-22T15:12:51.398 INFO:teuthology.orchestra.run.smithi062.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2023-05-22T15:12:51.398 INFO:teuthology.orchestra.run.smithi062.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2023-05-22T15:12:51.398 INFO:teuthology.orchestra.run.smithi062.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2023-05-22T15:12:51.398 INFO:teuthology.orchestra.run.smithi062.stdout: python2.7-minimal 2023-05-22T15:12:51.398 INFO:teuthology.orchestra.run.smithi062.stdout:Use 'sudo apt autoremove' to remove them. 2023-05-22T15:12:51.399 INFO:teuthology.orchestra.run.smithi062.stdout:The following additional packages will be installed: 2023-05-22T15:12:51.399 INFO:teuthology.orchestra.run.smithi062.stdout: linux-generic linux-headers-5.15.0-72 linux-headers-5.15.0-72-generic 2023-05-22T15:12:51.399 INFO:teuthology.orchestra.run.smithi062.stdout: linux-headers-generic linux-image-5.15.0-72-generic 2023-05-22T15:12:51.400 INFO:teuthology.orchestra.run.smithi062.stdout: linux-modules-5.15.0-72-generic linux-modules-extra-5.15.0-72-generic 2023-05-22T15:12:51.400 INFO:teuthology.orchestra.run.smithi062.stdout:Suggested packages: 2023-05-22T15:12:51.400 INFO:teuthology.orchestra.run.smithi062.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2023-05-22T15:12:51.400 INFO:teuthology.orchestra.run.smithi062.stdout:Recommended packages: 2023-05-22T15:12:51.401 INFO:teuthology.orchestra.run.smithi062.stdout: thermald 2023-05-22T15:12:51.455 INFO:teuthology.orchestra.run.smithi062.stdout:The following NEW packages will be installed: 2023-05-22T15:12:51.455 INFO:teuthology.orchestra.run.smithi062.stdout: linux-headers-5.15.0-72 linux-headers-5.15.0-72-generic 2023-05-22T15:12:51.455 INFO:teuthology.orchestra.run.smithi062.stdout: linux-image-5.15.0-72-generic linux-modules-5.15.0-72-generic 2023-05-22T15:12:51.456 INFO:teuthology.orchestra.run.smithi062.stdout: linux-modules-extra-5.15.0-72-generic 2023-05-22T15:12:51.456 INFO:teuthology.orchestra.run.smithi062.stdout:The following packages will be upgraded: 2023-05-22T15:12:51.457 INFO:teuthology.orchestra.run.smithi062.stdout: linux-generic linux-headers-generic linux-image-generic 2023-05-22T15:12:51.507 INFO:teuthology.orchestra.run.smithi062.stdout:3 upgraded, 5 newly installed, 0 to remove and 170 not upgraded. 2023-05-22T15:12:51.507 INFO:teuthology.orchestra.run.smithi062.stdout:Need to get 114 MB of archives. 2023-05-22T15:12:51.507 INFO:teuthology.orchestra.run.smithi062.stdout:After this operation, 596 MB of additional disk space will be used. 2023-05-22T15:12:51.507 INFO:teuthology.orchestra.run.smithi062.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-72-generic amd64 5.15.0-72.79 [23.2 MB] 2023-05-22T15:12:51.956 INFO:teuthology.orchestra.run.smithi062.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-72-generic amd64 5.15.0-72.79 [11.4 MB] 2023-05-22T15:12:52.065 INFO:teuthology.orchestra.run.smithi062.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-72-generic amd64 5.15.0-72.79 [64.2 MB] 2023-05-22T15:12:52.192 INFO:teuthology.orchestra.run.smithi016.stdout:Reading package lists... 2023-05-22T15:12:52.209 DEBUG:teuthology.orchestra.run.smithi016:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2023-05-22T15:12:52.262 INFO:teuthology.orchestra.run.smithi016.stdout:Reading package lists... 2023-05-22T15:12:52.424 INFO:teuthology.orchestra.run.smithi016.stdout:Building dependency tree... 2023-05-22T15:12:52.424 INFO:teuthology.orchestra.run.smithi016.stdout:Reading state information... 2023-05-22T15:12:52.557 INFO:teuthology.orchestra.run.smithi016.stdout:The following packages were automatically installed and are no longer required: 2023-05-22T15:12:52.559 INFO:teuthology.orchestra.run.smithi016.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2023-05-22T15:12:52.559 INFO:teuthology.orchestra.run.smithi016.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2023-05-22T15:12:52.560 INFO:teuthology.orchestra.run.smithi016.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2023-05-22T15:12:52.560 INFO:teuthology.orchestra.run.smithi016.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2023-05-22T15:12:52.560 INFO:teuthology.orchestra.run.smithi016.stdout: python2.7-minimal 2023-05-22T15:12:52.561 INFO:teuthology.orchestra.run.smithi016.stdout:Use 'sudo apt autoremove' to remove them. 2023-05-22T15:12:52.561 INFO:teuthology.orchestra.run.smithi016.stdout:The following additional packages will be installed: 2023-05-22T15:12:52.561 INFO:teuthology.orchestra.run.smithi016.stdout: linux-generic linux-headers-5.15.0-72 linux-headers-5.15.0-72-generic 2023-05-22T15:12:52.561 INFO:teuthology.orchestra.run.smithi016.stdout: linux-headers-generic linux-image-5.15.0-72-generic 2023-05-22T15:12:52.562 INFO:teuthology.orchestra.run.smithi016.stdout: linux-modules-5.15.0-72-generic linux-modules-extra-5.15.0-72-generic 2023-05-22T15:12:52.562 INFO:teuthology.orchestra.run.smithi016.stdout:Suggested packages: 2023-05-22T15:12:52.562 INFO:teuthology.orchestra.run.smithi016.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2023-05-22T15:12:52.562 INFO:teuthology.orchestra.run.smithi016.stdout:Recommended packages: 2023-05-22T15:12:52.562 INFO:teuthology.orchestra.run.smithi016.stdout: thermald 2023-05-22T15:12:52.597 INFO:teuthology.orchestra.run.smithi016.stdout:The following NEW packages will be installed: 2023-05-22T15:12:52.597 INFO:teuthology.orchestra.run.smithi016.stdout: linux-headers-5.15.0-72 linux-headers-5.15.0-72-generic 2023-05-22T15:12:52.597 INFO:teuthology.orchestra.run.smithi016.stdout: linux-image-5.15.0-72-generic linux-modules-5.15.0-72-generic 2023-05-22T15:12:52.598 INFO:teuthology.orchestra.run.smithi016.stdout: linux-modules-extra-5.15.0-72-generic 2023-05-22T15:12:52.598 INFO:teuthology.orchestra.run.smithi016.stdout:The following packages will be upgraded: 2023-05-22T15:12:52.599 INFO:teuthology.orchestra.run.smithi016.stdout: linux-generic linux-headers-generic linux-image-generic 2023-05-22T15:12:52.647 INFO:teuthology.orchestra.run.smithi016.stdout:3 upgraded, 5 newly installed, 0 to remove and 170 not upgraded. 2023-05-22T15:12:52.647 INFO:teuthology.orchestra.run.smithi016.stdout:Need to get 114 MB of archives. 2023-05-22T15:12:52.647 INFO:teuthology.orchestra.run.smithi016.stdout:After this operation, 596 MB of additional disk space will be used. 2023-05-22T15:12:52.647 INFO:teuthology.orchestra.run.smithi016.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-72-generic amd64 5.15.0-72.79 [23.2 MB] 2023-05-22T15:12:52.867 INFO:teuthology.orchestra.run.smithi062.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.72.70 [1,700 B] 2023-05-22T15:12:52.868 INFO:teuthology.orchestra.run.smithi062.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.72.70 [2,496 B] 2023-05-22T15:12:52.868 INFO:teuthology.orchestra.run.smithi062.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-72 all 5.15.0-72.79 [12.3 MB] 2023-05-22T15:12:52.998 INFO:teuthology.orchestra.run.smithi062.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-72-generic amd64 5.15.0-72.79 [2,895 kB] 2023-05-22T15:12:53.021 INFO:teuthology.orchestra.run.smithi062.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.72.70 [2,346 B] 2023-05-22T15:12:53.085 INFO:teuthology.orchestra.run.smithi016.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-72-generic amd64 5.15.0-72.79 [11.4 MB] 2023-05-22T15:12:53.181 INFO:teuthology.orchestra.run.smithi016.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-72-generic amd64 5.15.0-72.79 [64.2 MB] 2023-05-22T15:12:53.434 INFO:teuthology.orchestra.run.smithi062.stdout:Fetched 114 MB in 2s (73.2 MB/s) 2023-05-22T15:12:53.610 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package linux-modules-5.15.0-72-generic. 2023-05-22T15:12:54.030 INFO:teuthology.orchestra.run.smithi016.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.72.70 [1,700 B] 2023-05-22T15:12:54.030 INFO:teuthology.orchestra.run.smithi016.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.72.70 [2,496 B] 2023-05-22T15:12:54.031 INFO:teuthology.orchestra.run.smithi016.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-72 all 5.15.0-72.79 [12.3 MB] 2023-05-22T15:12:54.160 INFO:teuthology.orchestra.run.smithi016.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-72-generic amd64 5.15.0-72.79 [2,895 kB] 2023-05-22T15:12:54.182 INFO:teuthology.orchestra.run.smithi016.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.72.70 [2,346 B] 2023-05-22T15:12:54.586 INFO:teuthology.orchestra.run.smithi016.stdout:Fetched 114 MB in 2s (72.3 MB/s) 2023-05-22T15:12:54.750 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package linux-modules-5.15.0-72-generic. 2023-05-22T15:12:55.564 INFO:teuthology.orchestra.run.smithi062.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2023-05-22T15:12:55.571 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../0-linux-modules-5.15.0-72-generic_5.15.0-72.79_amd64.deb ... 2023-05-22T15:12:55.649 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking linux-modules-5.15.0-72-generic (5.15.0-72.79) ... 2023-05-22T15:12:56.756 INFO:teuthology.orchestra.run.smithi016.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2023-05-22T15:12:56.761 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../0-linux-modules-5.15.0-72-generic_5.15.0-72.79_amd64.deb ... 2023-05-22T15:12:56.980 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking linux-modules-5.15.0-72-generic (5.15.0-72.79) ... 2023-05-22T15:12:57.734 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package linux-image-5.15.0-72-generic. 2023-05-22T15:12:57.756 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../1-linux-image-5.15.0-72-generic_5.15.0-72.79_amd64.deb ... 2023-05-22T15:12:57.870 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking linux-image-5.15.0-72-generic (5.15.0-72.79) ... 2023-05-22T15:12:58.287 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-72-generic. 2023-05-22T15:12:58.308 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-72-generic_5.15.0-72.79_amd64.deb ... 2023-05-22T15:12:58.345 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking linux-modules-extra-5.15.0-72-generic (5.15.0-72.79) ... 2023-05-22T15:12:59.323 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package linux-image-5.15.0-72-generic. 2023-05-22T15:12:59.345 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../1-linux-image-5.15.0-72-generic_5.15.0-72.79_amd64.deb ... 2023-05-22T15:12:59.437 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking linux-image-5.15.0-72-generic (5.15.0-72.79) ... 2023-05-22T15:12:59.901 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-72-generic. 2023-05-22T15:12:59.911 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-72-generic_5.15.0-72.79_amd64.deb ... 2023-05-22T15:12:59.943 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking linux-modules-extra-5.15.0-72-generic (5.15.0-72.79) ... 2023-05-22T15:13:04.456 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../3-linux-generic_5.15.0.72.70_amd64.deb ... 2023-05-22T15:13:04.583 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking linux-generic (5.15.0.72.70) over (5.15.0.56.54) ... 2023-05-22T15:13:04.900 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.72.70_amd64.deb ... 2023-05-22T15:13:05.028 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking linux-image-generic (5.15.0.72.70) over (5.15.0.56.54) ... 2023-05-22T15:13:05.331 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package linux-headers-5.15.0-72. 2023-05-22T15:13:05.353 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../5-linux-headers-5.15.0-72_5.15.0-72.79_all.deb ... 2023-05-22T15:13:05.397 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking linux-headers-5.15.0-72 (5.15.0-72.79) ... 2023-05-22T15:13:06.813 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../3-linux-generic_5.15.0.72.70_amd64.deb ... 2023-05-22T15:13:06.913 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking linux-generic (5.15.0.72.70) over (5.15.0.56.54) ... 2023-05-22T15:13:07.254 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.72.70_amd64.deb ... 2023-05-22T15:13:07.408 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking linux-image-generic (5.15.0.72.70) over (5.15.0.56.54) ... 2023-05-22T15:13:07.744 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package linux-headers-5.15.0-72. 2023-05-22T15:13:07.766 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../5-linux-headers-5.15.0-72_5.15.0-72.79_all.deb ... 2023-05-22T15:13:07.810 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking linux-headers-5.15.0-72 (5.15.0-72.79) ... 2023-05-22T15:13:11.338 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package linux-headers-5.15.0-72-generic. 2023-05-22T15:13:11.376 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../6-linux-headers-5.15.0-72-generic_5.15.0-72.79_amd64.deb ... 2023-05-22T15:13:11.413 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking linux-headers-5.15.0-72-generic (5.15.0-72.79) ... 2023-05-22T15:13:13.855 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package linux-headers-5.15.0-72-generic. 2023-05-22T15:13:13.892 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../6-linux-headers-5.15.0-72-generic_5.15.0-72.79_amd64.deb ... 2023-05-22T15:13:13.930 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking linux-headers-5.15.0-72-generic (5.15.0-72.79) ... 2023-05-22T15:13:14.103 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.72.70_amd64.deb ... 2023-05-22T15:13:14.226 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking linux-headers-generic (5.15.0.72.70) over (5.15.0.56.54) ... 2023-05-22T15:13:14.551 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up linux-headers-5.15.0-72 (5.15.0-72.79) ... 2023-05-22T15:13:14.672 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up linux-headers-5.15.0-72-generic (5.15.0-72.79) ... 2023-05-22T15:13:14.798 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up linux-headers-generic (5.15.0.72.70) ... 2023-05-22T15:13:14.952 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up linux-image-5.15.0-72-generic (5.15.0-72.79) ... 2023-05-22T15:13:16.300 INFO:teuthology.orchestra.run.smithi062.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2023-05-22T15:13:16.300 INFO:teuthology.orchestra.run.smithi062.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2023-05-22T15:13:16.300 INFO:teuthology.orchestra.run.smithi062.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-72-generic 2023-05-22T15:13:16.300 INFO:teuthology.orchestra.run.smithi062.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-72-generic 2023-05-22T15:13:16.471 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up linux-modules-5.15.0-72-generic (5.15.0-72.79) ... 2023-05-22T15:13:16.794 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.72.70_amd64.deb ... 2023-05-22T15:13:16.916 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking linux-headers-generic (5.15.0.72.70) over (5.15.0.56.54) ... 2023-05-22T15:13:17.249 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up linux-headers-5.15.0-72 (5.15.0-72.79) ... 2023-05-22T15:13:17.371 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up linux-headers-5.15.0-72-generic (5.15.0-72.79) ... 2023-05-22T15:13:17.497 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up linux-headers-generic (5.15.0.72.70) ... 2023-05-22T15:13:17.643 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up linux-image-5.15.0-72-generic (5.15.0-72.79) ... 2023-05-22T15:13:17.914 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up linux-modules-extra-5.15.0-72-generic (5.15.0-72.79) ... 2023-05-22T15:13:18.973 INFO:teuthology.orchestra.run.smithi016.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2023-05-22T15:13:18.974 INFO:teuthology.orchestra.run.smithi016.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2023-05-22T15:13:18.974 INFO:teuthology.orchestra.run.smithi016.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-72-generic 2023-05-22T15:13:18.974 INFO:teuthology.orchestra.run.smithi016.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-72-generic 2023-05-22T15:13:19.111 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up linux-modules-5.15.0-72-generic (5.15.0-72.79) ... 2023-05-22T15:13:19.323 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up linux-image-generic (5.15.0.72.70) ... 2023-05-22T15:13:19.566 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up linux-generic (5.15.0.72.70) ... 2023-05-22T15:13:19.701 INFO:teuthology.orchestra.run.smithi062.stdout:Processing triggers for linux-image-5.15.0-72-generic (5.15.0-72.79) ... 2023-05-22T15:13:19.778 INFO:teuthology.orchestra.run.smithi062.stdout:/etc/kernel/postinst.d/initramfs-tools: 2023-05-22T15:13:19.779 INFO:teuthology.orchestra.run.smithi062.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-72-generic 2023-05-22T15:13:20.664 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up linux-modules-extra-5.15.0-72-generic (5.15.0-72.79) ... 2023-05-22T15:13:22.097 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up linux-image-generic (5.15.0.72.70) ... 2023-05-22T15:13:22.223 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up linux-generic (5.15.0.72.70) ... 2023-05-22T15:13:22.349 INFO:teuthology.orchestra.run.smithi016.stdout:Processing triggers for linux-image-5.15.0-72-generic (5.15.0-72.79) ... 2023-05-22T15:13:22.434 INFO:teuthology.orchestra.run.smithi016.stdout:/etc/kernel/postinst.d/initramfs-tools: 2023-05-22T15:13:22.434 INFO:teuthology.orchestra.run.smithi016.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-72-generic 2023-05-22T15:13:34.024 INFO:teuthology.orchestra.run.smithi062.stdout:/etc/kernel/postinst.d/zz-update-grub: 2023-05-22T15:13:34.024 INFO:teuthology.orchestra.run.smithi062.stdout:Sourcing file `/etc/default/grub' 2023-05-22T15:13:34.043 INFO:teuthology.orchestra.run.smithi062.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2023-05-22T15:13:34.061 INFO:teuthology.orchestra.run.smithi062.stdout:Generating grub configuration file ... 2023-05-22T15:13:34.326 INFO:teuthology.orchestra.run.smithi062.stdout:Found linux image: /boot/vmlinuz-5.15.0-72-generic 2023-05-22T15:13:34.355 INFO:teuthology.orchestra.run.smithi062.stdout:Found initrd image: /boot/initrd.img-5.15.0-72-generic 2023-05-22T15:13:34.639 INFO:teuthology.orchestra.run.smithi062.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2023-05-22T15:13:34.645 INFO:teuthology.orchestra.run.smithi062.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2023-05-22T15:13:34.711 INFO:teuthology.orchestra.run.smithi062.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2023-05-22T15:13:34.716 INFO:teuthology.orchestra.run.smithi062.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2023-05-22T15:13:34.878 INFO:teuthology.orchestra.run.smithi062.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2023-05-22T15:13:34.878 INFO:teuthology.orchestra.run.smithi062.stdout:Systems on them will not be added to the GRUB boot configuration. 2023-05-22T15:13:34.878 INFO:teuthology.orchestra.run.smithi062.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2023-05-22T15:13:34.917 INFO:teuthology.orchestra.run.smithi062.stdout:done 2023-05-22T15:13:35.402 DEBUG:teuthology.orchestra.run.smithi062:> dpkg -s linux-image-generic 2023-05-22T15:13:35.424 INFO:teuthology.orchestra.run.smithi062.stdout:Package: linux-image-generic 2023-05-22T15:13:35.424 INFO:teuthology.orchestra.run.smithi062.stdout:Status: install ok installed 2023-05-22T15:13:35.424 INFO:teuthology.orchestra.run.smithi062.stdout:Priority: optional 2023-05-22T15:13:35.425 INFO:teuthology.orchestra.run.smithi062.stdout:Section: kernel 2023-05-22T15:13:35.425 INFO:teuthology.orchestra.run.smithi062.stdout:Installed-Size: 20 2023-05-22T15:13:35.425 INFO:teuthology.orchestra.run.smithi062.stdout:Maintainer: Ubuntu Kernel Team 2023-05-22T15:13:35.425 INFO:teuthology.orchestra.run.smithi062.stdout:Architecture: amd64 2023-05-22T15:13:35.425 INFO:teuthology.orchestra.run.smithi062.stdout:Source: linux-meta 2023-05-22T15:13:35.425 INFO:teuthology.orchestra.run.smithi062.stdout:Version: 5.15.0.72.70 2023-05-22T15:13:35.425 INFO:teuthology.orchestra.run.smithi062.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.1), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-72), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.1) 2023-05-22T15:13:35.426 INFO:teuthology.orchestra.run.smithi062.stdout:Depends: linux-image-5.15.0-72-generic, linux-modules-extra-5.15.0-72-generic, linux-firmware, intel-microcode, amd64-microcode 2023-05-22T15:13:35.426 INFO:teuthology.orchestra.run.smithi062.stdout:Recommends: thermald 2023-05-22T15:13:35.426 INFO:teuthology.orchestra.run.smithi062.stdout:Description: Generic Linux kernel image 2023-05-22T15:13:35.426 INFO:teuthology.orchestra.run.smithi062.stdout: This package will always depend on the latest generic kernel image 2023-05-22T15:13:35.426 INFO:teuthology.orchestra.run.smithi062.stdout: available. 2023-05-22T15:13:35.426 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-72-generic 2023-05-22T15:13:35.427 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-05-22T15:13:35.427 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-05-22T15:13:35.427 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2023-05-22T15:13:35.427 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi062.front.sepia.ceph.com, path=None, version=distro) 2023-05-22T15:13:35.427 DEBUG:teuthology.orchestra.run.smithi062:> sudo apt-get clean 2023-05-22T15:13:35.590 DEBUG:teuthology.orchestra.run.smithi062:> sudo apt-get update 2023-05-22T15:13:35.718 INFO:teuthology.orchestra.run.smithi062.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2023-05-22T15:13:35.718 INFO:teuthology.orchestra.run.smithi062.stdout:Hit:2 http://security.ubuntu.com/ubuntu jammy-security InRelease 2023-05-22T15:13:35.719 INFO:teuthology.orchestra.run.smithi062.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2023-05-22T15:13:35.737 INFO:teuthology.orchestra.run.smithi062.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2023-05-22T15:13:36.980 INFO:teuthology.orchestra.run.smithi062.stdout:Reading package lists... 2023-05-22T15:13:36.995 DEBUG:teuthology.orchestra.run.smithi062:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2023-05-22T15:13:37.055 INFO:teuthology.orchestra.run.smithi062.stdout:Reading package lists... 2023-05-22T15:13:37.237 INFO:teuthology.orchestra.run.smithi062.stdout:Building dependency tree... 2023-05-22T15:13:37.238 INFO:teuthology.orchestra.run.smithi062.stdout:Reading state information... 2023-05-22T15:13:37.267 INFO:teuthology.orchestra.run.smithi016.stdout:/etc/kernel/postinst.d/zz-update-grub: 2023-05-22T15:13:37.267 INFO:teuthology.orchestra.run.smithi016.stdout:Sourcing file `/etc/default/grub' 2023-05-22T15:13:37.290 INFO:teuthology.orchestra.run.smithi016.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2023-05-22T15:13:37.308 INFO:teuthology.orchestra.run.smithi016.stdout:Generating grub configuration file ... 2023-05-22T15:13:37.396 INFO:teuthology.orchestra.run.smithi062.stdout:linux-image-generic is already the newest version (5.15.0.72.70). 2023-05-22T15:13:37.396 INFO:teuthology.orchestra.run.smithi062.stdout:The following packages were automatically installed and are no longer required: 2023-05-22T15:13:37.396 INFO:teuthology.orchestra.run.smithi062.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2023-05-22T15:13:37.396 INFO:teuthology.orchestra.run.smithi062.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2023-05-22T15:13:37.397 INFO:teuthology.orchestra.run.smithi062.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2023-05-22T15:13:37.397 INFO:teuthology.orchestra.run.smithi062.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2023-05-22T15:13:37.397 INFO:teuthology.orchestra.run.smithi062.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2023-05-22T15:13:37.397 INFO:teuthology.orchestra.run.smithi062.stdout:Use 'sudo apt autoremove' to remove them. 2023-05-22T15:13:37.435 INFO:teuthology.orchestra.run.smithi062.stdout:0 upgraded, 0 newly installed, 0 to remove and 170 not upgraded. 2023-05-22T15:13:37.437 DEBUG:teuthology.orchestra.run.smithi062:> dpkg -s linux-image-generic 2023-05-22T15:13:37.453 INFO:teuthology.orchestra.run.smithi062.stdout:Package: linux-image-generic 2023-05-22T15:13:37.453 INFO:teuthology.orchestra.run.smithi062.stdout:Status: install ok installed 2023-05-22T15:13:37.453 INFO:teuthology.orchestra.run.smithi062.stdout:Priority: optional 2023-05-22T15:13:37.454 INFO:teuthology.orchestra.run.smithi062.stdout:Section: kernel 2023-05-22T15:13:37.454 INFO:teuthology.orchestra.run.smithi062.stdout:Installed-Size: 20 2023-05-22T15:13:37.454 INFO:teuthology.orchestra.run.smithi062.stdout:Maintainer: Ubuntu Kernel Team 2023-05-22T15:13:37.454 INFO:teuthology.orchestra.run.smithi062.stdout:Architecture: amd64 2023-05-22T15:13:37.454 INFO:teuthology.orchestra.run.smithi062.stdout:Source: linux-meta 2023-05-22T15:13:37.454 INFO:teuthology.orchestra.run.smithi062.stdout:Version: 5.15.0.72.70 2023-05-22T15:13:37.454 INFO:teuthology.orchestra.run.smithi062.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.1), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-72), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.1) 2023-05-22T15:13:37.454 INFO:teuthology.orchestra.run.smithi062.stdout:Depends: linux-image-5.15.0-72-generic, linux-modules-extra-5.15.0-72-generic, linux-firmware, intel-microcode, amd64-microcode 2023-05-22T15:13:37.455 INFO:teuthology.orchestra.run.smithi062.stdout:Recommends: thermald 2023-05-22T15:13:37.455 INFO:teuthology.orchestra.run.smithi062.stdout:Description: Generic Linux kernel image 2023-05-22T15:13:37.455 INFO:teuthology.orchestra.run.smithi062.stdout: This package will always depend on the latest generic kernel image 2023-05-22T15:13:37.455 INFO:teuthology.orchestra.run.smithi062.stdout: available. 2023-05-22T15:13:37.455 DEBUG:teuthology.orchestra.run.smithi062:> mktemp 2023-05-22T15:13:37.505 INFO:teuthology.orchestra.run.smithi062.stdout:/tmp/tmp.PvpFP8Apt2 2023-05-22T15:13:37.505 DEBUG:teuthology.orchestra.run.smithi062:> sudo cp /boot/grub/grub.cfg /tmp/tmp.PvpFP8Apt2 2023-05-22T15:13:37.544 INFO:teuthology.orchestra.run.smithi016.stdout:Found linux image: /boot/vmlinuz-5.15.0-72-generic 2023-05-22T15:13:37.556 INFO:teuthology.orchestra.run.smithi016.stdout:Found initrd image: /boot/initrd.img-5.15.0-72-generic 2023-05-22T15:13:37.561 DEBUG:teuthology.orchestra.run.smithi062:> sudo chmod 0666 /tmp/tmp.PvpFP8Apt2 2023-05-22T15:13:37.685 DEBUG:teuthology.orchestra.remote:smithi062:/tmp/tmp.PvpFP8Apt2 is 10KB 2023-05-22T15:13:37.697 DEBUG:teuthology.orchestra.run.smithi062:> rm -fr /tmp/tmp.PvpFP8Apt2 2023-05-22T15:13:37.703 DEBUG:teuthology.orchestra.run.smithi062:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2023-05-22T15:13:37.757 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2023-05-22T15:13:37.758 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd of=/etc/grub.d/01_ceph_kernel 2023-05-22T15:13:37.758 DEBUG:teuthology.orchestra.run.smithi062:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2023-05-22T15:13:37.824 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-72-generic 2023-05-22T15:13:37.824 DEBUG:teuthology.orchestra.run.smithi062:> sudo update-grub 2023-05-22T15:13:37.855 INFO:teuthology.orchestra.run.smithi016.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2023-05-22T15:13:37.861 INFO:teuthology.orchestra.run.smithi016.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2023-05-22T15:13:37.936 INFO:teuthology.orchestra.run.smithi016.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2023-05-22T15:13:37.942 INFO:teuthology.orchestra.run.smithi016.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2023-05-22T15:13:38.114 INFO:teuthology.orchestra.run.smithi016.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2023-05-22T15:13:38.114 INFO:teuthology.orchestra.run.smithi016.stdout:Systems on them will not be added to the GRUB boot configuration. 2023-05-22T15:13:38.114 INFO:teuthology.orchestra.run.smithi016.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2023-05-22T15:13:38.154 INFO:teuthology.orchestra.run.smithi016.stdout:done 2023-05-22T15:13:38.612 DEBUG:teuthology.orchestra.run.smithi016:> dpkg -s linux-image-generic 2023-05-22T15:13:38.627 INFO:teuthology.orchestra.run.smithi016.stdout:Package: linux-image-generic 2023-05-22T15:13:38.627 INFO:teuthology.orchestra.run.smithi016.stdout:Status: install ok installed 2023-05-22T15:13:38.627 INFO:teuthology.orchestra.run.smithi016.stdout:Priority: optional 2023-05-22T15:13:38.628 INFO:teuthology.orchestra.run.smithi016.stdout:Section: kernel 2023-05-22T15:13:38.628 INFO:teuthology.orchestra.run.smithi016.stdout:Installed-Size: 20 2023-05-22T15:13:38.628 INFO:teuthology.orchestra.run.smithi016.stdout:Maintainer: Ubuntu Kernel Team 2023-05-22T15:13:38.628 INFO:teuthology.orchestra.run.smithi016.stdout:Architecture: amd64 2023-05-22T15:13:38.628 INFO:teuthology.orchestra.run.smithi016.stdout:Source: linux-meta 2023-05-22T15:13:38.628 INFO:teuthology.orchestra.run.smithi016.stdout:Version: 5.15.0.72.70 2023-05-22T15:13:38.629 INFO:teuthology.orchestra.run.smithi016.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.1), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-72), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.1) 2023-05-22T15:13:38.629 INFO:teuthology.orchestra.run.smithi016.stdout:Depends: linux-image-5.15.0-72-generic, linux-modules-extra-5.15.0-72-generic, linux-firmware, intel-microcode, amd64-microcode 2023-05-22T15:13:38.629 INFO:teuthology.orchestra.run.smithi016.stdout:Recommends: thermald 2023-05-22T15:13:38.629 INFO:teuthology.orchestra.run.smithi016.stdout:Description: Generic Linux kernel image 2023-05-22T15:13:38.629 INFO:teuthology.orchestra.run.smithi016.stdout: This package will always depend on the latest generic kernel image 2023-05-22T15:13:38.629 INFO:teuthology.orchestra.run.smithi016.stdout: available. 2023-05-22T15:13:38.630 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-72-generic 2023-05-22T15:13:38.630 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-05-22T15:13:38.630 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-05-22T15:13:38.630 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2023-05-22T15:13:38.630 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi016.front.sepia.ceph.com, path=None, version=distro) 2023-05-22T15:13:38.630 DEBUG:teuthology.orchestra.run.smithi016:> sudo apt-get clean 2023-05-22T15:13:38.797 DEBUG:teuthology.orchestra.run.smithi016:> sudo apt-get update 2023-05-22T15:13:38.850 INFO:teuthology.orchestra.run.smithi062.stderr:Sourcing file `/etc/default/grub' 2023-05-22T15:13:38.851 INFO:teuthology.orchestra.run.smithi062.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2023-05-22T15:13:38.853 INFO:teuthology.orchestra.run.smithi062.stderr:Generating grub configuration file ... 2023-05-22T15:13:38.909 INFO:teuthology.orchestra.run.smithi016.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2023-05-22T15:13:38.910 INFO:teuthology.orchestra.run.smithi016.stdout:Hit:2 http://security.ubuntu.com/ubuntu jammy-security InRelease 2023-05-22T15:13:38.917 INFO:teuthology.orchestra.run.smithi016.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2023-05-22T15:13:38.935 INFO:teuthology.orchestra.run.smithi016.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2023-05-22T15:13:39.055 INFO:teuthology.orchestra.run.smithi062.stderr:Found linux image: /boot/vmlinuz-5.15.0-72-generic 2023-05-22T15:13:39.061 INFO:teuthology.orchestra.run.smithi062.stderr:Found initrd image: /boot/initrd.img-5.15.0-72-generic 2023-05-22T15:13:39.322 INFO:teuthology.orchestra.run.smithi062.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2023-05-22T15:13:39.327 INFO:teuthology.orchestra.run.smithi062.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2023-05-22T15:13:39.368 INFO:teuthology.orchestra.run.smithi062.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2023-05-22T15:13:39.373 INFO:teuthology.orchestra.run.smithi062.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2023-05-22T15:13:39.487 INFO:teuthology.orchestra.run.smithi062.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2023-05-22T15:13:39.487 INFO:teuthology.orchestra.run.smithi062.stderr:Systems on them will not be added to the GRUB boot configuration. 2023-05-22T15:13:39.487 INFO:teuthology.orchestra.run.smithi062.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2023-05-22T15:13:39.506 INFO:teuthology.orchestra.run.smithi062.stderr:done 2023-05-22T15:13:39.524 DEBUG:teuthology.orchestra.run.smithi062:> sudo shutdown -r now 2023-05-22T15:13:40.136 INFO:teuthology.orchestra.run.smithi016.stdout:Reading package lists... 2023-05-22T15:13:40.153 DEBUG:teuthology.orchestra.run.smithi016:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2023-05-22T15:13:40.214 INFO:teuthology.orchestra.run.smithi016.stdout:Reading package lists... 2023-05-22T15:13:40.394 INFO:teuthology.orchestra.run.smithi016.stdout:Building dependency tree... 2023-05-22T15:13:40.394 INFO:teuthology.orchestra.run.smithi016.stdout:Reading state information... 2023-05-22T15:13:40.555 INFO:teuthology.orchestra.run.smithi016.stdout:linux-image-generic is already the newest version (5.15.0.72.70). 2023-05-22T15:13:40.555 INFO:teuthology.orchestra.run.smithi016.stdout:The following packages were automatically installed and are no longer required: 2023-05-22T15:13:40.555 INFO:teuthology.orchestra.run.smithi016.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2023-05-22T15:13:40.556 INFO:teuthology.orchestra.run.smithi016.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2023-05-22T15:13:40.556 INFO:teuthology.orchestra.run.smithi016.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2023-05-22T15:13:40.556 INFO:teuthology.orchestra.run.smithi016.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2023-05-22T15:13:40.556 INFO:teuthology.orchestra.run.smithi016.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2023-05-22T15:13:40.557 INFO:teuthology.orchestra.run.smithi016.stdout:Use 'sudo apt autoremove' to remove them. 2023-05-22T15:13:40.595 INFO:teuthology.orchestra.run.smithi016.stdout:0 upgraded, 0 newly installed, 0 to remove and 170 not upgraded. 2023-05-22T15:13:40.597 DEBUG:teuthology.orchestra.run.smithi016:> dpkg -s linux-image-generic 2023-05-22T15:13:40.613 INFO:teuthology.orchestra.run.smithi016.stdout:Package: linux-image-generic 2023-05-22T15:13:40.614 INFO:teuthology.orchestra.run.smithi016.stdout:Status: install ok installed 2023-05-22T15:13:40.614 INFO:teuthology.orchestra.run.smithi016.stdout:Priority: optional 2023-05-22T15:13:40.614 INFO:teuthology.orchestra.run.smithi016.stdout:Section: kernel 2023-05-22T15:13:40.614 INFO:teuthology.orchestra.run.smithi016.stdout:Installed-Size: 20 2023-05-22T15:13:40.614 INFO:teuthology.orchestra.run.smithi016.stdout:Maintainer: Ubuntu Kernel Team 2023-05-22T15:13:40.614 INFO:teuthology.orchestra.run.smithi016.stdout:Architecture: amd64 2023-05-22T15:13:40.615 INFO:teuthology.orchestra.run.smithi016.stdout:Source: linux-meta 2023-05-22T15:13:40.615 INFO:teuthology.orchestra.run.smithi016.stdout:Version: 5.15.0.72.70 2023-05-22T15:13:40.615 INFO:teuthology.orchestra.run.smithi016.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.1), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-72), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.1) 2023-05-22T15:13:40.615 INFO:teuthology.orchestra.run.smithi016.stdout:Depends: linux-image-5.15.0-72-generic, linux-modules-extra-5.15.0-72-generic, linux-firmware, intel-microcode, amd64-microcode 2023-05-22T15:13:40.615 INFO:teuthology.orchestra.run.smithi016.stdout:Recommends: thermald 2023-05-22T15:13:40.615 INFO:teuthology.orchestra.run.smithi016.stdout:Description: Generic Linux kernel image 2023-05-22T15:13:40.615 INFO:teuthology.orchestra.run.smithi016.stdout: This package will always depend on the latest generic kernel image 2023-05-22T15:13:40.616 INFO:teuthology.orchestra.run.smithi016.stdout: available. 2023-05-22T15:13:40.616 DEBUG:teuthology.orchestra.run.smithi016:> mktemp 2023-05-22T15:13:40.659 INFO:teuthology.orchestra.run.smithi016.stdout:/tmp/tmp.eLXPnXsfPS 2023-05-22T15:13:40.659 DEBUG:teuthology.orchestra.run.smithi016:> sudo cp /boot/grub/grub.cfg /tmp/tmp.eLXPnXsfPS 2023-05-22T15:13:40.716 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod 0666 /tmp/tmp.eLXPnXsfPS 2023-05-22T15:13:40.829 DEBUG:teuthology.orchestra.remote:smithi016:/tmp/tmp.eLXPnXsfPS is 10KB 2023-05-22T15:13:40.841 DEBUG:teuthology.orchestra.run.smithi016:> rm -fr /tmp/tmp.eLXPnXsfPS 2023-05-22T15:13:40.847 DEBUG:teuthology.orchestra.run.smithi016:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2023-05-22T15:13:40.904 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-05-22T15:13:40.904 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd of=/etc/grub.d/01_ceph_kernel 2023-05-22T15:13:40.904 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2023-05-22T15:13:40.980 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-72-generic 2023-05-22T15:13:40.980 DEBUG:teuthology.orchestra.run.smithi016:> sudo update-grub 2023-05-22T15:13:42.160 INFO:teuthology.orchestra.run.smithi016.stderr:Sourcing file `/etc/default/grub' 2023-05-22T15:13:42.161 INFO:teuthology.orchestra.run.smithi016.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2023-05-22T15:13:42.162 INFO:teuthology.orchestra.run.smithi016.stderr:Generating grub configuration file ... 2023-05-22T15:13:42.345 INFO:teuthology.orchestra.run.smithi016.stderr:Found linux image: /boot/vmlinuz-5.15.0-72-generic 2023-05-22T15:13:42.351 INFO:teuthology.orchestra.run.smithi016.stderr:Found initrd image: /boot/initrd.img-5.15.0-72-generic 2023-05-22T15:13:42.592 INFO:teuthology.orchestra.run.smithi016.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2023-05-22T15:13:42.598 INFO:teuthology.orchestra.run.smithi016.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2023-05-22T15:13:42.640 INFO:teuthology.orchestra.run.smithi016.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2023-05-22T15:13:42.646 INFO:teuthology.orchestra.run.smithi016.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2023-05-22T15:13:42.787 INFO:teuthology.orchestra.run.smithi016.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2023-05-22T15:13:42.787 INFO:teuthology.orchestra.run.smithi016.stderr:Systems on them will not be added to the GRUB boot configuration. 2023-05-22T15:13:42.787 INFO:teuthology.orchestra.run.smithi016.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2023-05-22T15:13:42.798 INFO:teuthology.orchestra.run.smithi016.stderr:done 2023-05-22T15:13:42.816 DEBUG:teuthology.orchestra.run.smithi016:> sudo shutdown -r now 2023-05-22T15:14:09.552 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2023-05-22T15:14:09.553 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi062.front.sepia.ceph.com' 2023-05-22T15:14:09.553 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi062.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-05-22T15:14:12.821 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2023-05-22T15:14:12.822 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi016.front.sepia.ceph.com' 2023-05-22T15:14:12.823 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi016.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-05-22T15:14:28.080 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.62 2023-05-22T15:14:31.156 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.16 2023-05-22T15:14:58.108 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi062.front.sepia.ceph.com' 2023-05-22T15:14:58.109 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi062.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-05-22T15:15:01.160 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi016.front.sepia.ceph.com' 2023-05-22T15:15:01.161 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi016.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-05-22T15:15:01.168 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.62 2023-05-22T15:15:04.240 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.16 2023-05-22T15:15:31.190 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi062.front.sepia.ceph.com' 2023-05-22T15:15:31.191 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi062.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-05-22T15:15:31.423 DEBUG:teuthology.orchestra.run.smithi062:> true 2023-05-22T15:15:31.978 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi062.front.sepia.ceph.com' 2023-05-22T15:15:31.979 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.15.0-72-generic"... 2023-05-22T15:15:31.979 DEBUG:teuthology.orchestra.run.smithi062:> uname -r 2023-05-22T15:15:32.026 INFO:teuthology.orchestra.run.smithi062.stdout:5.15.0-72-generic 2023-05-22T15:15:32.027 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-72-generic vs 5.15.0-72-generic 2023-05-22T15:15:32.027 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-05-22T15:15:32.028 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2023-05-22T15:15:33.029 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2023-05-22T15:15:33.030 DEBUG:teuthology.orchestra.run.smithi062:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-05-22T15:15:33.129 INFO:teuthology.orchestra.run.smithi062.stdout:ttyS1 2023-05-22T15:15:33.156 DEBUG:teuthology.parallel:result is None 2023-05-22T15:15:34.241 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi016.front.sepia.ceph.com' 2023-05-22T15:15:34.242 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi016.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-05-22T15:15:34.445 DEBUG:teuthology.orchestra.run.smithi016:> true 2023-05-22T15:15:35.160 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi016.front.sepia.ceph.com' 2023-05-22T15:15:35.161 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.15.0-72-generic"... 2023-05-22T15:15:35.161 DEBUG:teuthology.orchestra.run.smithi016:> uname -r 2023-05-22T15:15:35.206 INFO:teuthology.orchestra.run.smithi016.stdout:5.15.0-72-generic 2023-05-22T15:15:35.206 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-72-generic vs 5.15.0-72-generic 2023-05-22T15:15:35.207 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-05-22T15:15:35.207 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2023-05-22T15:15:36.208 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2023-05-22T15:15:36.208 DEBUG:teuthology.orchestra.run.smithi016:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-05-22T15:15:36.271 INFO:teuthology.orchestra.run.smithi016.stdout:ttyS1 2023-05-22T15:15:36.303 DEBUG:teuthology.parallel:result is None 2023-05-22T15:15:36.303 INFO:teuthology.run_tasks:Running task internal.base... 2023-05-22T15:15:36.310 INFO:teuthology.task.internal:Creating test directory... 2023-05-22T15:15:36.310 DEBUG:teuthology.orchestra.run.smithi016:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-05-22T15:15:36.313 DEBUG:teuthology.orchestra.run.smithi062:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-05-22T15:15:36.318 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-05-22T15:15:36.325 INFO:teuthology.run_tasks:Running task internal.archive... 2023-05-22T15:15:36.335 INFO:teuthology.task.internal:Creating archive directory... 2023-05-22T15:15:36.335 DEBUG:teuthology.orchestra.run.smithi016:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-05-22T15:15:36.359 DEBUG:teuthology.orchestra.run.smithi062:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-05-22T15:15:36.387 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-05-22T15:15:36.394 INFO:teuthology.task.internal:Enabling coredump saving... 2023-05-22T15:15:36.395 DEBUG:teuthology.orchestra.run.smithi016:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-05-22T15:15:36.411 DEBUG:teuthology.orchestra.run.smithi062:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-05-22T15:15:36.440 INFO:teuthology.orchestra.run.smithi062.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-05-22T15:15:36.448 INFO:teuthology.orchestra.run.smithi062.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-05-22T15:15:36.450 INFO:teuthology.orchestra.run.smithi016.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-05-22T15:15:36.460 INFO:teuthology.orchestra.run.smithi016.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-05-22T15:15:36.462 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-05-22T15:15:36.470 INFO:teuthology.task.internal:Configuring sudo... 2023-05-22T15:15:36.470 DEBUG:teuthology.orchestra.run.smithi016:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-05-22T15:15:36.506 DEBUG:teuthology.orchestra.run.smithi062:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-05-22T15:15:36.520 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-05-22T15:15:36.531 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-05-22T15:15:36.531 DEBUG:teuthology.orchestra.run.smithi016:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-05-22T15:15:36.563 DEBUG:teuthology.orchestra.run.smithi062:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-05-22T15:15:36.569 DEBUG:teuthology.orchestra.run.smithi016:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-05-22T15:15:36.614 DEBUG:teuthology.orchestra.run.smithi016:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-05-22T15:15:36.662 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-05-22T15:15:36.662 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-05-22T15:15:36.722 DEBUG:teuthology.orchestra.run.smithi062:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-05-22T15:15:36.728 DEBUG:teuthology.orchestra.run.smithi062:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-05-22T15:15:36.774 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2023-05-22T15:15:36.775 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-05-22T15:15:36.838 DEBUG:teuthology.orchestra.run.smithi016:> sudo service rsyslog restart 2023-05-22T15:15:36.840 DEBUG:teuthology.orchestra.run.smithi062:> sudo service rsyslog restart 2023-05-22T15:15:36.962 INFO:teuthology.run_tasks:Running task internal.timer... 2023-05-22T15:15:36.969 INFO:teuthology.task.internal:Starting timer... 2023-05-22T15:15:36.970 INFO:teuthology.run_tasks:Running task pcp... 2023-05-22T15:15:36.989 INFO:teuthology.run_tasks:Running task selinux... 2023-05-22T15:15:36.998 DEBUG:teuthology.task.selinux:Excluding smithi016: OS 'ubuntu' does not support SELinux 2023-05-22T15:15:36.998 DEBUG:teuthology.task.selinux:Excluding smithi062: OS 'ubuntu' does not support SELinux 2023-05-22T15:15:36.999 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-05-22T15:15:36.999 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2023-05-22T15:15:36.999 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-05-22T15:15:36.999 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-05-22T15:15:37.007 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2023-05-22T15:15:37.012 INFO:teuthology.repo_utils:Fetching main from origin 2023-05-22T15:15:37.128 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-05-22T15:15:37.151 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-05-22T15:15:37.153 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi016.front.sepia.ceph.com,smithi062.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-05-22T15:23:12.629 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi016.front.sepia.ceph.com'), Remote(name='ubuntu@smithi062.front.sepia.ceph.com')] 2023-05-22T15:23:12.631 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi016.front.sepia.ceph.com' 2023-05-22T15:23:12.633 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi016.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-05-22T15:23:12.711 DEBUG:teuthology.orchestra.run.smithi016:> true 2023-05-22T15:23:12.836 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi062.front.sepia.ceph.com' 2023-05-22T15:23:12.837 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi062.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-05-22T15:23:12.915 DEBUG:teuthology.orchestra.run.smithi062:> true 2023-05-22T15:23:13.012 INFO:teuthology.run_tasks:Running task clock... 2023-05-22T15:23:13.025 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-05-22T15:23:13.026 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-05-22T15:23:13.026 DEBUG:teuthology.orchestra.run.smithi016:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-05-22T15:23:13.029 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-05-22T15:23:13.029 DEBUG:teuthology.orchestra.run.smithi062:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-05-22T15:23:13.056 INFO:teuthology.orchestra.run.smithi016.stdout:22 May 15:23:13 ntpd[26923]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2023-05-22T15:23:13.057 INFO:teuthology.orchestra.run.smithi016.stdout:22 May 15:23:13 ntpd[26923]: Command line: ntpd -gq 2023-05-22T15:23:13.057 INFO:teuthology.orchestra.run.smithi016.stdout:22 May 15:23:13 ntpd[26923]: ---------------------------------------------------- 2023-05-22T15:23:13.057 INFO:teuthology.orchestra.run.smithi016.stdout:22 May 15:23:13 ntpd[26923]: ntp-4 is maintained by Network Time Foundation, 2023-05-22T15:23:13.057 INFO:teuthology.orchestra.run.smithi016.stdout:22 May 15:23:13 ntpd[26923]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2023-05-22T15:23:13.057 INFO:teuthology.orchestra.run.smithi016.stdout:22 May 15:23:13 ntpd[26923]: corporation. Support and training for ntp-4 are 2023-05-22T15:23:13.057 INFO:teuthology.orchestra.run.smithi016.stdout:22 May 15:23:13 ntpd[26923]: available at https://www.nwtime.org/support 2023-05-22T15:23:13.057 INFO:teuthology.orchestra.run.smithi016.stdout:22 May 15:23:13 ntpd[26923]: ---------------------------------------------------- 2023-05-22T15:23:13.059 INFO:teuthology.orchestra.run.smithi016.stdout:22 May 15:23:13 ntpd[26923]: proto: precision = 0.046 usec (-24) 2023-05-22T15:23:13.059 INFO:teuthology.orchestra.run.smithi016.stdout:22 May 15:23:13 ntpd[26923]: basedate set to 2022-02-04 2023-05-22T15:23:13.059 INFO:teuthology.orchestra.run.smithi016.stdout:22 May 15:23:13 ntpd[26923]: gps base set to 2022-02-06 (week 2196) 2023-05-22T15:23:13.059 INFO:teuthology.orchestra.run.smithi016.stdout:22 May 15:23:13 ntpd[26923]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-05-22T15:23:13.059 INFO:teuthology.orchestra.run.smithi016.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-05-22T15:23:13.060 INFO:teuthology.orchestra.run.smithi016.stderr:restrict ::: KOD does nothing without LIMITED. 2023-05-22T15:23:13.060 INFO:teuthology.orchestra.run.smithi016.stdout:22 May 15:23:13 ntpd[26923]: restrict ::: KOD does nothing without LIMITED. 2023-05-22T15:23:13.061 INFO:teuthology.orchestra.run.smithi016.stdout:22 May 15:23:13 ntpd[26923]: Listen and drop on 0 v6wildcard [::]:123 2023-05-22T15:23:13.061 INFO:teuthology.orchestra.run.smithi016.stdout:22 May 15:23:13 ntpd[26923]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2023-05-22T15:23:13.061 INFO:teuthology.orchestra.run.smithi016.stdout:22 May 15:23:13 ntpd[26923]: Listen normally on 2 lo 127.0.0.1:123 2023-05-22T15:23:13.061 INFO:teuthology.orchestra.run.smithi016.stdout:22 May 15:23:13 ntpd[26923]: Listen normally on 3 ens1f0 172.21.15.16:123 2023-05-22T15:23:13.061 INFO:teuthology.orchestra.run.smithi016.stdout:22 May 15:23:13 ntpd[26923]: Listen normally on 4 lo [::1]:123 2023-05-22T15:23:13.061 INFO:teuthology.orchestra.run.smithi016.stdout:22 May 15:23:13 ntpd[26923]: Listen normally on 5 ens1f0 [fe80::ec4:7aff:febd:1564%4]:123 2023-05-22T15:23:13.061 INFO:teuthology.orchestra.run.smithi016.stdout:22 May 15:23:13 ntpd[26923]: Listening on routing socket on fd #22 for interface updates 2023-05-22T15:23:13.082 INFO:teuthology.orchestra.run.smithi062.stdout:22 May 15:23:13 ntpd[27015]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2023-05-22T15:23:13.083 INFO:teuthology.orchestra.run.smithi062.stdout:22 May 15:23:13 ntpd[27015]: Command line: ntpd -gq 2023-05-22T15:23:13.083 INFO:teuthology.orchestra.run.smithi062.stdout:22 May 15:23:13 ntpd[27015]: ---------------------------------------------------- 2023-05-22T15:23:13.083 INFO:teuthology.orchestra.run.smithi062.stdout:22 May 15:23:13 ntpd[27015]: ntp-4 is maintained by Network Time Foundation, 2023-05-22T15:23:13.083 INFO:teuthology.orchestra.run.smithi062.stdout:22 May 15:23:13 ntpd[27015]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2023-05-22T15:23:13.083 INFO:teuthology.orchestra.run.smithi062.stdout:22 May 15:23:13 ntpd[27015]: corporation. Support and training for ntp-4 are 2023-05-22T15:23:13.083 INFO:teuthology.orchestra.run.smithi062.stdout:22 May 15:23:13 ntpd[27015]: available at https://www.nwtime.org/support 2023-05-22T15:23:13.083 INFO:teuthology.orchestra.run.smithi062.stdout:22 May 15:23:13 ntpd[27015]: ---------------------------------------------------- 2023-05-22T15:23:13.084 INFO:teuthology.orchestra.run.smithi062.stdout:22 May 15:23:13 ntpd[27015]: proto: precision = 0.072 usec (-24) 2023-05-22T15:23:13.084 INFO:teuthology.orchestra.run.smithi062.stdout:22 May 15:23:13 ntpd[27015]: basedate set to 2022-02-04 2023-05-22T15:23:13.084 INFO:teuthology.orchestra.run.smithi062.stdout:22 May 15:23:13 ntpd[27015]: gps base set to 2022-02-06 (week 2196) 2023-05-22T15:23:13.085 INFO:teuthology.orchestra.run.smithi062.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-05-22T15:23:13.085 INFO:teuthology.orchestra.run.smithi062.stdout:22 May 15:23:13 ntpd[27015]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-05-22T15:23:13.085 INFO:teuthology.orchestra.run.smithi062.stdout:22 May 15:23:13 ntpd[27015]: restrict ::: KOD does nothing without LIMITED. 2023-05-22T15:23:13.086 INFO:teuthology.orchestra.run.smithi062.stderr:restrict ::: KOD does nothing without LIMITED. 2023-05-22T15:23:13.086 INFO:teuthology.orchestra.run.smithi062.stdout:22 May 15:23:13 ntpd[27015]: Listen and drop on 0 v6wildcard [::]:123 2023-05-22T15:23:13.087 INFO:teuthology.orchestra.run.smithi062.stdout:22 May 15:23:13 ntpd[27015]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2023-05-22T15:23:13.087 INFO:teuthology.orchestra.run.smithi062.stdout:22 May 15:23:13 ntpd[27015]: Listen normally on 2 lo 127.0.0.1:123 2023-05-22T15:23:13.087 INFO:teuthology.orchestra.run.smithi062.stdout:22 May 15:23:13 ntpd[27015]: Listen normally on 3 enp3s0f1 172.21.15.62:123 2023-05-22T15:23:13.087 INFO:teuthology.orchestra.run.smithi062.stdout:22 May 15:23:13 ntpd[27015]: Listen normally on 4 lo [::1]:123 2023-05-22T15:23:13.087 INFO:teuthology.orchestra.run.smithi062.stdout:22 May 15:23:13 ntpd[27015]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:febb:d1c3%5]:123 2023-05-22T15:23:13.087 INFO:teuthology.orchestra.run.smithi062.stdout:22 May 15:23:13 ntpd[27015]: Listening on routing socket on fd #22 for interface updates 2023-05-22T15:23:14.057 INFO:teuthology.orchestra.run.smithi016.stderr:22 May 15:23:14 ntpd[26923]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-05-22T15:23:14.057 INFO:teuthology.orchestra.run.smithi016.stderr:22 May 15:23:14 ntpd[26923]: can't open /var/log/ntpstats/rawstats.20230522: Permission denied 2023-05-22T15:23:14.057 INFO:teuthology.orchestra.run.smithi016.stderr:22 May 15:23:14 ntpd[26923]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-05-22T15:23:14.058 INFO:teuthology.orchestra.run.smithi016.stderr:22 May 15:23:14 ntpd[26923]: can't open /var/log/ntpstats/rawstats.20230522: Permission denied 2023-05-22T15:23:14.058 INFO:teuthology.orchestra.run.smithi016.stderr:22 May 15:23:14 ntpd[26923]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-05-22T15:23:14.058 INFO:teuthology.orchestra.run.smithi016.stderr:22 May 15:23:14 ntpd[26923]: can't open /var/log/ntpstats/peerstats.20230522: Permission denied 2023-05-22T15:23:14.083 INFO:teuthology.orchestra.run.smithi062.stderr:22 May 15:23:14 ntpd[27015]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-05-22T15:23:14.083 INFO:teuthology.orchestra.run.smithi062.stderr:22 May 15:23:14 ntpd[27015]: can't open /var/log/ntpstats/rawstats.20230522: Permission denied 2023-05-22T15:23:14.083 INFO:teuthology.orchestra.run.smithi062.stderr:22 May 15:23:14 ntpd[27015]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-05-22T15:23:14.083 INFO:teuthology.orchestra.run.smithi062.stderr:22 May 15:23:14 ntpd[27015]: can't open /var/log/ntpstats/rawstats.20230522: Permission denied 2023-05-22T15:23:14.084 INFO:teuthology.orchestra.run.smithi062.stderr:22 May 15:23:14 ntpd[27015]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-05-22T15:23:14.084 INFO:teuthology.orchestra.run.smithi062.stderr:22 May 15:23:14 ntpd[27015]: can't open /var/log/ntpstats/rawstats.20230522: Permission denied 2023-05-22T15:23:14.085 INFO:teuthology.orchestra.run.smithi062.stderr:22 May 15:23:14 ntpd[27015]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-05-22T15:23:14.085 INFO:teuthology.orchestra.run.smithi062.stderr:22 May 15:23:14 ntpd[27015]: can't open /var/log/ntpstats/peerstats.20230522: Permission denied 2023-05-22T15:23:14.085 INFO:teuthology.orchestra.run.smithi062.stderr:22 May 15:23:14 ntpd[27015]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-05-22T15:23:14.085 INFO:teuthology.orchestra.run.smithi062.stderr:22 May 15:23:14 ntpd[27015]: can't open /var/log/ntpstats/rawstats.20230522: Permission denied 2023-05-22T15:23:14.085 INFO:teuthology.orchestra.run.smithi062.stderr:22 May 15:23:14 ntpd[27015]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-05-22T15:23:14.085 INFO:teuthology.orchestra.run.smithi062.stderr:22 May 15:23:14 ntpd[27015]: can't open /var/log/ntpstats/peerstats.20230522: Permission denied 2023-05-22T15:23:15.057 INFO:teuthology.orchestra.run.smithi016.stderr:22 May 15:23:15 ntpd[26923]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-05-22T15:23:15.057 INFO:teuthology.orchestra.run.smithi016.stderr:22 May 15:23:15 ntpd[26923]: can't open /var/log/ntpstats/rawstats.20230522: Permission denied 2023-05-22T15:23:16.057 INFO:teuthology.orchestra.run.smithi016.stderr:22 May 15:23:16 ntpd[26923]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-05-22T15:23:16.057 INFO:teuthology.orchestra.run.smithi016.stderr:22 May 15:23:16 ntpd[26923]: can't open /var/log/ntpstats/rawstats.20230522: Permission denied 2023-05-22T15:23:16.058 INFO:teuthology.orchestra.run.smithi016.stderr:22 May 15:23:16 ntpd[26923]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-05-22T15:23:16.058 INFO:teuthology.orchestra.run.smithi016.stderr:22 May 15:23:16 ntpd[26923]: can't open /var/log/ntpstats/rawstats.20230522: Permission denied 2023-05-22T15:23:16.058 INFO:teuthology.orchestra.run.smithi016.stderr:22 May 15:23:16 ntpd[26923]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-05-22T15:23:16.058 INFO:teuthology.orchestra.run.smithi016.stderr:22 May 15:23:16 ntpd[26923]: can't open /var/log/ntpstats/peerstats.20230522: Permission denied 2023-05-22T15:23:16.084 INFO:teuthology.orchestra.run.smithi062.stderr:22 May 15:23:16 ntpd[27015]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-05-22T15:23:16.084 INFO:teuthology.orchestra.run.smithi062.stderr:22 May 15:23:16 ntpd[27015]: can't open /var/log/ntpstats/rawstats.20230522: Permission denied 2023-05-22T15:23:16.084 INFO:teuthology.orchestra.run.smithi062.stderr:22 May 15:23:16 ntpd[27015]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-05-22T15:23:16.084 INFO:teuthology.orchestra.run.smithi062.stderr:22 May 15:23:16 ntpd[27015]: can't open /var/log/ntpstats/rawstats.20230522: Permission denied 2023-05-22T15:23:16.085 INFO:teuthology.orchestra.run.smithi062.stderr:22 May 15:23:16 ntpd[27015]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-05-22T15:23:16.085 INFO:teuthology.orchestra.run.smithi062.stderr:22 May 15:23:16 ntpd[27015]: can't open /var/log/ntpstats/rawstats.20230522: Permission denied 2023-05-22T15:23:16.085 INFO:teuthology.orchestra.run.smithi062.stderr:22 May 15:23:16 ntpd[27015]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-05-22T15:23:16.085 INFO:teuthology.orchestra.run.smithi062.stderr:22 May 15:23:16 ntpd[27015]: can't open /var/log/ntpstats/rawstats.20230522: Permission denied 2023-05-22T15:23:16.085 INFO:teuthology.orchestra.run.smithi062.stderr:22 May 15:23:16 ntpd[27015]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-05-22T15:23:16.085 INFO:teuthology.orchestra.run.smithi062.stderr:22 May 15:23:16 ntpd[27015]: can't open /var/log/ntpstats/peerstats.20230522: Permission denied 2023-05-22T15:23:16.085 INFO:teuthology.orchestra.run.smithi062.stderr:22 May 15:23:16 ntpd[27015]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-05-22T15:23:16.085 INFO:teuthology.orchestra.run.smithi062.stderr:22 May 15:23:16 ntpd[27015]: can't open /var/log/ntpstats/rawstats.20230522: Permission denied 2023-05-22T15:23:16.086 INFO:teuthology.orchestra.run.smithi062.stderr:22 May 15:23:16 ntpd[27015]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-05-22T15:23:16.086 INFO:teuthology.orchestra.run.smithi062.stderr:22 May 15:23:16 ntpd[27015]: can't open /var/log/ntpstats/peerstats.20230522: Permission denied 2023-05-22T15:23:17.057 INFO:teuthology.orchestra.run.smithi016.stderr:22 May 15:23:17 ntpd[26923]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-05-22T15:23:17.058 INFO:teuthology.orchestra.run.smithi016.stderr:22 May 15:23:17 ntpd[26923]: can't open /var/log/ntpstats/rawstats.20230522: Permission denied 2023-05-22T15:23:17.058 INFO:teuthology.orchestra.run.smithi016.stderr:22 May 15:23:17 ntpd[26923]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-05-22T15:23:17.058 INFO:teuthology.orchestra.run.smithi016.stderr:22 May 15:23:17 ntpd[26923]: can't open /var/log/ntpstats/rawstats.20230522: Permission denied 2023-05-22T15:23:17.058 INFO:teuthology.orchestra.run.smithi016.stderr:22 May 15:23:17 ntpd[26923]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-05-22T15:23:17.058 INFO:teuthology.orchestra.run.smithi016.stderr:22 May 15:23:17 ntpd[26923]: can't open /var/log/ntpstats/rawstats.20230522: Permission denied 2023-05-22T15:23:17.058 INFO:teuthology.orchestra.run.smithi016.stderr:22 May 15:23:17 ntpd[26923]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-05-22T15:23:17.059 INFO:teuthology.orchestra.run.smithi016.stderr:22 May 15:23:17 ntpd[26923]: can't open /var/log/ntpstats/peerstats.20230522: Permission denied 2023-05-22T15:23:17.059 INFO:teuthology.orchestra.run.smithi016.stderr:22 May 15:23:17 ntpd[26923]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-05-22T15:23:17.059 INFO:teuthology.orchestra.run.smithi016.stderr:22 May 15:23:17 ntpd[26923]: can't open /var/log/ntpstats/rawstats.20230522: Permission denied 2023-05-22T15:23:17.059 INFO:teuthology.orchestra.run.smithi016.stderr:22 May 15:23:17 ntpd[26923]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-05-22T15:23:17.059 INFO:teuthology.orchestra.run.smithi016.stderr:22 May 15:23:17 ntpd[26923]: can't open /var/log/ntpstats/peerstats.20230522: Permission denied 2023-05-22T15:23:17.083 INFO:teuthology.orchestra.run.smithi062.stderr:22 May 15:23:17 ntpd[27015]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-05-22T15:23:17.083 INFO:teuthology.orchestra.run.smithi062.stderr:22 May 15:23:17 ntpd[27015]: can't open /var/log/ntpstats/rawstats.20230522: Permission denied 2023-05-22T15:23:17.083 INFO:teuthology.orchestra.run.smithi062.stderr:22 May 15:23:17 ntpd[27015]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-05-22T15:23:17.083 INFO:teuthology.orchestra.run.smithi062.stderr:22 May 15:23:17 ntpd[27015]: can't open /var/log/ntpstats/rawstats.20230522: Permission denied 2023-05-22T15:23:17.084 INFO:teuthology.orchestra.run.smithi062.stderr:22 May 15:23:17 ntpd[27015]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-05-22T15:23:17.084 INFO:teuthology.orchestra.run.smithi062.stderr:22 May 15:23:17 ntpd[27015]: can't open /var/log/ntpstats/peerstats.20230522: Permission denied 2023-05-22T15:23:18.057 INFO:teuthology.orchestra.run.smithi016.stderr:22 May 15:23:18 ntpd[26923]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-05-22T15:23:18.057 INFO:teuthology.orchestra.run.smithi016.stderr:22 May 15:23:18 ntpd[26923]: can't open /var/log/ntpstats/rawstats.20230522: Permission denied 2023-05-22T15:23:18.057 INFO:teuthology.orchestra.run.smithi016.stderr:22 May 15:23:18 ntpd[26923]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-05-22T15:23:18.058 INFO:teuthology.orchestra.run.smithi016.stderr:22 May 15:23:18 ntpd[26923]: can't open /var/log/ntpstats/rawstats.20230522: Permission denied 2023-05-22T15:23:18.058 INFO:teuthology.orchestra.run.smithi016.stderr:22 May 15:23:18 ntpd[26923]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-05-22T15:23:18.058 INFO:teuthology.orchestra.run.smithi016.stderr:22 May 15:23:18 ntpd[26923]: can't open /var/log/ntpstats/peerstats.20230522: Permission denied 2023-05-22T15:23:18.083 INFO:teuthology.orchestra.run.smithi062.stderr:22 May 15:23:18 ntpd[27015]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-05-22T15:23:18.084 INFO:teuthology.orchestra.run.smithi062.stderr:22 May 15:23:18 ntpd[27015]: can't open /var/log/ntpstats/rawstats.20230522: Permission denied 2023-05-22T15:23:18.084 INFO:teuthology.orchestra.run.smithi062.stderr:22 May 15:23:18 ntpd[27015]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-05-22T15:23:18.084 INFO:teuthology.orchestra.run.smithi062.stderr:22 May 15:23:18 ntpd[27015]: can't open /var/log/ntpstats/rawstats.20230522: Permission denied 2023-05-22T15:23:18.084 INFO:teuthology.orchestra.run.smithi062.stderr:22 May 15:23:18 ntpd[27015]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-05-22T15:23:18.084 INFO:teuthology.orchestra.run.smithi062.stderr:22 May 15:23:18 ntpd[27015]: can't open /var/log/ntpstats/rawstats.20230522: Permission denied 2023-05-22T15:23:18.085 INFO:teuthology.orchestra.run.smithi062.stderr:22 May 15:23:18 ntpd[27015]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-05-22T15:23:18.085 INFO:teuthology.orchestra.run.smithi062.stderr:22 May 15:23:18 ntpd[27015]: can't open /var/log/ntpstats/peerstats.20230522: Permission denied 2023-05-22T15:23:18.085 INFO:teuthology.orchestra.run.smithi062.stderr:22 May 15:23:18 ntpd[27015]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-05-22T15:23:18.085 INFO:teuthology.orchestra.run.smithi062.stderr:22 May 15:23:18 ntpd[27015]: can't open /var/log/ntpstats/rawstats.20230522: Permission denied 2023-05-22T15:23:18.085 INFO:teuthology.orchestra.run.smithi062.stderr:22 May 15:23:18 ntpd[27015]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-05-22T15:23:18.086 INFO:teuthology.orchestra.run.smithi062.stderr:22 May 15:23:18 ntpd[27015]: can't open /var/log/ntpstats/peerstats.20230522: Permission denied 2023-05-22T15:23:19.058 INFO:teuthology.orchestra.run.smithi016.stderr:22 May 15:23:19 ntpd[26923]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-05-22T15:23:19.058 INFO:teuthology.orchestra.run.smithi016.stderr:22 May 15:23:19 ntpd[26923]: can't open /var/log/ntpstats/rawstats.20230522: Permission denied 2023-05-22T15:23:19.059 INFO:teuthology.orchestra.run.smithi016.stderr:22 May 15:23:19 ntpd[26923]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-05-22T15:23:19.059 INFO:teuthology.orchestra.run.smithi016.stderr:22 May 15:23:19 ntpd[26923]: can't open /var/log/ntpstats/rawstats.20230522: Permission denied 2023-05-22T15:23:19.059 INFO:teuthology.orchestra.run.smithi016.stderr:22 May 15:23:19 ntpd[26923]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-05-22T15:23:19.059 INFO:teuthology.orchestra.run.smithi016.stderr:22 May 15:23:19 ntpd[26923]: can't open /var/log/ntpstats/rawstats.20230522: Permission denied 2023-05-22T15:23:19.060 INFO:teuthology.orchestra.run.smithi016.stderr:22 May 15:23:19 ntpd[26923]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-05-22T15:23:19.060 INFO:teuthology.orchestra.run.smithi016.stderr:22 May 15:23:19 ntpd[26923]: can't open /var/log/ntpstats/peerstats.20230522: Permission denied 2023-05-22T15:23:19.060 INFO:teuthology.orchestra.run.smithi016.stderr:22 May 15:23:19 ntpd[26923]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-05-22T15:23:19.060 INFO:teuthology.orchestra.run.smithi016.stderr:22 May 15:23:19 ntpd[26923]: can't open /var/log/ntpstats/rawstats.20230522: Permission denied 2023-05-22T15:23:19.061 INFO:teuthology.orchestra.run.smithi016.stderr:22 May 15:23:19 ntpd[26923]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-05-22T15:23:19.061 INFO:teuthology.orchestra.run.smithi016.stderr:22 May 15:23:19 ntpd[26923]: can't open /var/log/ntpstats/peerstats.20230522: Permission denied 2023-05-22T15:23:19.084 INFO:teuthology.orchestra.run.smithi062.stderr:22 May 15:23:19 ntpd[27015]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-05-22T15:23:19.084 INFO:teuthology.orchestra.run.smithi062.stderr:22 May 15:23:19 ntpd[27015]: can't open /var/log/ntpstats/rawstats.20230522: Permission denied 2023-05-22T15:23:19.084 INFO:teuthology.orchestra.run.smithi062.stderr:22 May 15:23:19 ntpd[27015]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-05-22T15:23:19.085 INFO:teuthology.orchestra.run.smithi062.stderr:22 May 15:23:19 ntpd[27015]: can't open /var/log/ntpstats/rawstats.20230522: Permission denied 2023-05-22T15:23:19.085 INFO:teuthology.orchestra.run.smithi062.stderr:22 May 15:23:19 ntpd[27015]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-05-22T15:23:19.085 INFO:teuthology.orchestra.run.smithi062.stderr:22 May 15:23:19 ntpd[27015]: can't open /var/log/ntpstats/peerstats.20230522: Permission denied 2023-05-22T15:23:20.057 INFO:teuthology.orchestra.run.smithi016.stderr:22 May 15:23:20 ntpd[26923]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-05-22T15:23:20.057 INFO:teuthology.orchestra.run.smithi016.stderr:22 May 15:23:20 ntpd[26923]: can't open /var/log/ntpstats/rawstats.20230522: Permission denied 2023-05-22T15:23:20.058 INFO:teuthology.orchestra.run.smithi016.stderr:22 May 15:23:20 ntpd[26923]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-05-22T15:23:20.058 INFO:teuthology.orchestra.run.smithi016.stderr:22 May 15:23:20 ntpd[26923]: can't open /var/log/ntpstats/rawstats.20230522: Permission denied 2023-05-22T15:23:20.058 INFO:teuthology.orchestra.run.smithi016.stderr:22 May 15:23:20 ntpd[26923]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-05-22T15:23:20.058 INFO:teuthology.orchestra.run.smithi016.stdout:22 May 15:23:20 ntpd[26923]: ntpd: time slew -0.004003 s 2023-05-22T15:23:20.058 INFO:teuthology.orchestra.run.smithi016.stdout:ntpd: time slew -0.004003s 2023-05-22T15:23:20.059 INFO:teuthology.orchestra.run.smithi016.stderr:22 May 15:23:20 ntpd[26923]: can't open /var/log/ntpstats/peerstats.20230522: Permission denied 2023-05-22T15:23:20.059 INFO:teuthology.orchestra.run.smithi016.stderr:22 May 15:23:20 ntpd[26923]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2023-05-22T15:23:20.059 INFO:teuthology.orchestra.run.smithi016.stderr:22 May 15:23:20 ntpd[26923]: can't open /var/log/ntpstats/loopstats.20230522: Permission denied 2023-05-22T15:23:20.083 INFO:teuthology.orchestra.run.smithi062.stderr:22 May 15:23:20 ntpd[27015]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-05-22T15:23:20.083 INFO:teuthology.orchestra.run.smithi062.stderr:22 May 15:23:20 ntpd[27015]: can't open /var/log/ntpstats/rawstats.20230522: Permission denied 2023-05-22T15:23:20.084 INFO:teuthology.orchestra.run.smithi062.stderr:22 May 15:23:20 ntpd[27015]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-05-22T15:23:20.084 INFO:teuthology.orchestra.run.smithi062.stderr:22 May 15:23:20 ntpd[27015]: can't open /var/log/ntpstats/rawstats.20230522: Permission denied 2023-05-22T15:23:20.084 INFO:teuthology.orchestra.run.smithi062.stderr:22 May 15:23:20 ntpd[27015]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-05-22T15:23:20.084 INFO:teuthology.orchestra.run.smithi062.stderr:22 May 15:23:20 ntpd[27015]: can't open /var/log/ntpstats/rawstats.20230522: Permission denied 2023-05-22T15:23:20.084 INFO:teuthology.orchestra.run.smithi062.stderr:22 May 15:23:20 ntpd[27015]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-05-22T15:23:20.084 INFO:teuthology.orchestra.run.smithi062.stderr:22 May 15:23:20 ntpd[27015]: can't open /var/log/ntpstats/peerstats.20230522: Permission denied 2023-05-22T15:23:20.085 INFO:teuthology.orchestra.run.smithi062.stdout:22 May 15:23:20 ntpd[27015]: ntpd: time slew -0.004138 s 2023-05-22T15:23:20.085 INFO:teuthology.orchestra.run.smithi062.stdout:ntpd: time slew -0.004138s 2023-05-22T15:23:20.085 INFO:teuthology.orchestra.run.smithi062.stderr:22 May 15:23:20 ntpd[27015]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2023-05-22T15:23:20.085 INFO:teuthology.orchestra.run.smithi062.stderr:22 May 15:23:20 ntpd[27015]: can't open /var/log/ntpstats/loopstats.20230522: Permission denied 2023-05-22T15:23:20.151 INFO:teuthology.orchestra.run.smithi016.stdout: remote refid st t when poll reach delay offset jitter 2023-05-22T15:23:20.151 INFO:teuthology.orchestra.run.smithi016.stdout:============================================================================== 2023-05-22T15:23:20.152 INFO:teuthology.orchestra.run.smithi016.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-05-22T15:23:20.152 INFO:teuthology.orchestra.run.smithi016.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-05-22T15:23:20.152 INFO:teuthology.orchestra.run.smithi016.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-05-22T15:23:20.152 INFO:teuthology.orchestra.run.smithi016.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-05-22T15:23:20.164 INFO:teuthology.orchestra.run.smithi062.stdout: remote refid st t when poll reach delay offset jitter 2023-05-22T15:23:20.164 INFO:teuthology.orchestra.run.smithi062.stdout:============================================================================== 2023-05-22T15:23:20.164 INFO:teuthology.orchestra.run.smithi062.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-05-22T15:23:20.164 INFO:teuthology.orchestra.run.smithi062.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-05-22T15:23:20.164 INFO:teuthology.orchestra.run.smithi062.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-05-22T15:23:20.164 INFO:teuthology.orchestra.run.smithi062.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-05-22T15:23:20.165 INFO:teuthology.run_tasks:Running task install... 2023-05-22T15:23:20.181 DEBUG:teuthology.task.install:project ceph 2023-05-22T15:23:20.182 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '9c9ba4df22024d9d8d6cf9f98a4b96751fc78d0c'}} 2023-05-22T15:23:20.183 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '9c9ba4df22024d9d8d6cf9f98a4b96751fc78d0c'} 2023-05-22T15:23:20.183 INFO:teuthology.task.install:Using flavor: default 2023-05-22T15:23:20.189 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-05-22T15:23:20.190 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']} 2023-05-22T15:23:20.190 DEBUG:teuthology.orchestra.run.smithi016:> sudo apt-key list | grep Ceph 2023-05-22T15:23:20.191 DEBUG:teuthology.orchestra.run.smithi062:> sudo apt-key list | grep Ceph 2023-05-22T15:23:20.251 INFO:teuthology.orchestra.run.smithi016.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2023-05-22T15:23:20.261 INFO:teuthology.orchestra.run.smithi062.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2023-05-22T15:23:20.279 INFO:teuthology.orchestra.run.smithi016.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2023-05-22T15:23:20.279 INFO:teuthology.orchestra.run.smithi016.stdout:uid [ unknown] Ceph.com (release key) 2023-05-22T15:23:20.280 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror on remote deb x86_64 2023-05-22T15:23:20.280 INFO:teuthology.task.install.deb:Installing system (non-project) packages: bison, flex, libelf-dev, libssl-dev, network-manager, iproute2, util-linux, dump, indent, libaio-dev, libtool-bin, uuid-dev, xfslibs-dev, postgresql, postgresql-client, postgresql-common, postgresql-contrib on remote deb x86_64 2023-05-22T15:23:20.281 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=9c9ba4df22024d9d8d6cf9f98a4b96751fc78d0c 2023-05-22T15:23:20.292 INFO:teuthology.orchestra.run.smithi062.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2023-05-22T15:23:20.293 INFO:teuthology.orchestra.run.smithi062.stdout:uid [ unknown] Ceph.com (release key) 2023-05-22T15:23:20.293 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror on remote deb x86_64 2023-05-22T15:23:20.294 INFO:teuthology.task.install.deb:Installing system (non-project) packages: bison, flex, libelf-dev, libssl-dev, network-manager, iproute2, util-linux, dump, indent, libaio-dev, libtool-bin, uuid-dev, xfslibs-dev, postgresql, postgresql-client, postgresql-common, postgresql-contrib on remote deb x86_64 2023-05-22T15:23:20.294 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=9c9ba4df22024d9d8d6cf9f98a4b96751fc78d0c 2023-05-22T15:23:20.428 INFO:teuthology.task.install.deb:Pulling from https://4.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-05-21-0740-reef/9c9ba4df22024d9d8d6cf9f98a4b96751fc78d0c/ubuntu/jammy/flavors/default/ 2023-05-22T15:23:20.428 INFO:teuthology.task.install.deb:Package version is 18.0.0-3192-g9c9ba4df-1jammy 2023-05-22T15:23:20.442 INFO:teuthology.task.install.deb:Pulling from https://4.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-05-21-0740-reef/9c9ba4df22024d9d8d6cf9f98a4b96751fc78d0c/ubuntu/jammy/flavors/default/ 2023-05-22T15:23:20.443 INFO:teuthology.task.install.deb:Package version is 18.0.0-3192-g9c9ba4df-1jammy 2023-05-22T15:23:20.543 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-05-22T15:23:20.543 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2023-05-22T15:23:20.550 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2023-05-22T15:23:20.550 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2023-05-22T15:23:20.557 DEBUG:teuthology.orchestra.run.smithi016:> sudo apt-get update 2023-05-22T15:23:20.564 DEBUG:teuthology.orchestra.run.smithi062:> sudo apt-get update 2023-05-22T15:23:20.723 INFO:teuthology.orchestra.run.smithi016.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2023-05-22T15:23:20.732 INFO:teuthology.orchestra.run.smithi062.stdout:Hit:1 http://security.ubuntu.com/ubuntu jammy-security InRelease 2023-05-22T15:23:20.733 INFO:teuthology.orchestra.run.smithi016.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2023-05-22T15:23:20.750 INFO:teuthology.orchestra.run.smithi016.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2023-05-22T15:23:20.778 INFO:teuthology.orchestra.run.smithi016.stdout:Ign:4 https://4.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-05-21-0740-reef/9c9ba4df22024d9d8d6cf9f98a4b96751fc78d0c/ubuntu/jammy/flavors/default jammy InRelease 2023-05-22T15:23:20.789 INFO:teuthology.orchestra.run.smithi062.stdout:Ign:2 https://4.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-05-21-0740-reef/9c9ba4df22024d9d8d6cf9f98a4b96751fc78d0c/ubuntu/jammy/flavors/default jammy InRelease 2023-05-22T15:23:20.803 INFO:teuthology.orchestra.run.smithi016.stdout:Get:5 https://4.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-05-21-0740-reef/9c9ba4df22024d9d8d6cf9f98a4b96751fc78d0c/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2023-05-22T15:23:20.810 INFO:teuthology.orchestra.run.smithi062.stdout:Get:3 https://4.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-05-21-0740-reef/9c9ba4df22024d9d8d6cf9f98a4b96751fc78d0c/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2023-05-22T15:23:20.825 INFO:teuthology.orchestra.run.smithi016.stdout:Ign:6 https://4.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-05-21-0740-reef/9c9ba4df22024d9d8d6cf9f98a4b96751fc78d0c/ubuntu/jammy/flavors/default jammy Release.gpg 2023-05-22T15:23:20.831 INFO:teuthology.orchestra.run.smithi062.stdout:Ign:4 https://4.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-05-21-0740-reef/9c9ba4df22024d9d8d6cf9f98a4b96751fc78d0c/ubuntu/jammy/flavors/default jammy Release.gpg 2023-05-22T15:23:20.846 INFO:teuthology.orchestra.run.smithi016.stdout:Get:7 https://4.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-05-21-0740-reef/9c9ba4df22024d9d8d6cf9f98a4b96751fc78d0c/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,407 B] 2023-05-22T15:23:20.852 INFO:teuthology.orchestra.run.smithi016.stdout:Hit:8 http://security.ubuntu.com/ubuntu jammy-security InRelease 2023-05-22T15:23:20.852 INFO:teuthology.orchestra.run.smithi062.stdout:Get:5 https://4.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-05-21-0740-reef/9c9ba4df22024d9d8d6cf9f98a4b96751fc78d0c/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,407 B] 2023-05-22T15:23:20.858 INFO:teuthology.orchestra.run.smithi062.stdout:Hit:6 http://archive.ubuntu.com/ubuntu jammy InRelease 2023-05-22T15:23:20.872 INFO:teuthology.orchestra.run.smithi016.stdout:Get:9 https://4.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-05-21-0740-reef/9c9ba4df22024d9d8d6cf9f98a4b96751fc78d0c/ubuntu/jammy/flavors/default jammy/main amd64 Packages [17.4 kB] 2023-05-22T15:23:20.880 INFO:teuthology.orchestra.run.smithi062.stdout:Get:7 https://4.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-05-21-0740-reef/9c9ba4df22024d9d8d6cf9f98a4b96751fc78d0c/ubuntu/jammy/flavors/default jammy/main amd64 Packages [17.4 kB] 2023-05-22T15:23:20.936 INFO:teuthology.orchestra.run.smithi062.stdout:Hit:8 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2023-05-22T15:23:21.018 INFO:teuthology.orchestra.run.smithi062.stdout:Hit:9 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2023-05-22T15:23:21.084 INFO:teuthology.orchestra.run.smithi016.stdout:Fetched 28.5 kB in 0s (83.2 kB/s) 2023-05-22T15:23:21.161 INFO:teuthology.orchestra.run.smithi062.stdout:Fetched 28.5 kB in 0s (68.5 kB/s) 2023-05-22T15:23:21.885 INFO:teuthology.orchestra.run.smithi016.stdout:Reading package lists... 2023-05-22T15:23:21.902 DEBUG:teuthology.orchestra.run.smithi016:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=18.0.0-3192-g9c9ba4df-1jammy cephadm=18.0.0-3192-g9c9ba4df-1jammy ceph-mds=18.0.0-3192-g9c9ba4df-1jammy ceph-mgr=18.0.0-3192-g9c9ba4df-1jammy ceph-common=18.0.0-3192-g9c9ba4df-1jammy ceph-fuse=18.0.0-3192-g9c9ba4df-1jammy ceph-test=18.0.0-3192-g9c9ba4df-1jammy ceph-volume=18.0.0-3192-g9c9ba4df-1jammy radosgw=18.0.0-3192-g9c9ba4df-1jammy python3-rados=18.0.0-3192-g9c9ba4df-1jammy python3-rgw=18.0.0-3192-g9c9ba4df-1jammy python3-cephfs=18.0.0-3192-g9c9ba4df-1jammy python3-rbd=18.0.0-3192-g9c9ba4df-1jammy libcephfs2=18.0.0-3192-g9c9ba4df-1jammy libcephfs-dev=18.0.0-3192-g9c9ba4df-1jammy librados2=18.0.0-3192-g9c9ba4df-1jammy librbd1=18.0.0-3192-g9c9ba4df-1jammy rbd-fuse=18.0.0-3192-g9c9ba4df-1jammy python3-cephfs=18.0.0-3192-g9c9ba4df-1jammy cephfs-shell=18.0.0-3192-g9c9ba4df-1jammy cephfs-top=18.0.0-3192-g9c9ba4df-1jammy cephfs-mirror=18.0.0-3192-g9c9ba4df-1jammy 2023-05-22T15:23:21.933 INFO:teuthology.orchestra.run.smithi062.stdout:Reading package lists... 2023-05-22T15:23:21.949 DEBUG:teuthology.orchestra.run.smithi062:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=18.0.0-3192-g9c9ba4df-1jammy cephadm=18.0.0-3192-g9c9ba4df-1jammy ceph-mds=18.0.0-3192-g9c9ba4df-1jammy ceph-mgr=18.0.0-3192-g9c9ba4df-1jammy ceph-common=18.0.0-3192-g9c9ba4df-1jammy ceph-fuse=18.0.0-3192-g9c9ba4df-1jammy ceph-test=18.0.0-3192-g9c9ba4df-1jammy ceph-volume=18.0.0-3192-g9c9ba4df-1jammy radosgw=18.0.0-3192-g9c9ba4df-1jammy python3-rados=18.0.0-3192-g9c9ba4df-1jammy python3-rgw=18.0.0-3192-g9c9ba4df-1jammy python3-cephfs=18.0.0-3192-g9c9ba4df-1jammy python3-rbd=18.0.0-3192-g9c9ba4df-1jammy libcephfs2=18.0.0-3192-g9c9ba4df-1jammy libcephfs-dev=18.0.0-3192-g9c9ba4df-1jammy librados2=18.0.0-3192-g9c9ba4df-1jammy librbd1=18.0.0-3192-g9c9ba4df-1jammy rbd-fuse=18.0.0-3192-g9c9ba4df-1jammy python3-cephfs=18.0.0-3192-g9c9ba4df-1jammy cephfs-shell=18.0.0-3192-g9c9ba4df-1jammy cephfs-top=18.0.0-3192-g9c9ba4df-1jammy cephfs-mirror=18.0.0-3192-g9c9ba4df-1jammy 2023-05-22T15:23:21.962 INFO:teuthology.orchestra.run.smithi016.stdout:Reading package lists... 2023-05-22T15:23:22.001 INFO:teuthology.orchestra.run.smithi062.stdout:Reading package lists... 2023-05-22T15:23:22.135 INFO:teuthology.orchestra.run.smithi016.stdout:Building dependency tree... 2023-05-22T15:23:22.136 INFO:teuthology.orchestra.run.smithi016.stdout:Reading state information... 2023-05-22T15:23:22.158 INFO:teuthology.orchestra.run.smithi062.stdout:Building dependency tree... 2023-05-22T15:23:22.159 INFO:teuthology.orchestra.run.smithi062.stdout:Reading state information... 2023-05-22T15:23:22.300 INFO:teuthology.orchestra.run.smithi016.stdout:The following packages were automatically installed and are no longer required: 2023-05-22T15:23:22.301 INFO:teuthology.orchestra.run.smithi016.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2023-05-22T15:23:22.301 INFO:teuthology.orchestra.run.smithi016.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2023-05-22T15:23:22.302 INFO:teuthology.orchestra.run.smithi016.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2023-05-22T15:23:22.302 INFO:teuthology.orchestra.run.smithi016.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2023-05-22T15:23:22.302 INFO:teuthology.orchestra.run.smithi016.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2023-05-22T15:23:22.302 INFO:teuthology.orchestra.run.smithi016.stdout:Use 'sudo apt autoremove' to remove them. 2023-05-22T15:23:22.305 INFO:teuthology.orchestra.run.smithi016.stdout:The following additional packages will be installed: 2023-05-22T15:23:22.305 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2023-05-22T15:23:22.306 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2023-05-22T15:23:22.306 INFO:teuthology.orchestra.run.smithi016.stdout: libdouble-conversion3 libfmt8 libjq1 liblttng-ust1 liboath0 libonig5 2023-05-22T15:23:22.306 INFO:teuthology.orchestra.run.smithi016.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2023-05-22T15:23:22.306 INFO:teuthology.orchestra.run.smithi016.stdout: librdkafka1 librgw2 libsqlite3-mod-ceph libthrift-0.16.0 nvme-cli 2023-05-22T15:23:22.306 INFO:teuthology.orchestra.run.smithi016.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2023-05-22T15:23:22.306 INFO:teuthology.orchestra.run.smithi016.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2023-05-22T15:23:22.306 INFO:teuthology.orchestra.run.smithi016.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2023-05-22T15:23:22.307 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cherrypy3 python3-cmd2 python3-colorama python3-google-auth 2023-05-22T15:23:22.307 INFO:teuthology.orchestra.run.smithi016.stdout: python3-idna python3-iniconfig python3-jaraco.classes 2023-05-22T15:23:22.307 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jaraco.collections python3-jaraco.functools python3-jaraco.text 2023-05-22T15:23:22.307 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jinja2 python3-joblib python3-kubernetes python3-logutils 2023-05-22T15:23:22.307 INFO:teuthology.orchestra.run.smithi016.stdout: python3-mako python3-markupsafe python3-natsort python3-openssl 2023-05-22T15:23:22.307 INFO:teuthology.orchestra.run.smithi016.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2023-05-22T15:23:22.307 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pluggy python3-portend python3-prettytable python3-psutil python3-py 2023-05-22T15:23:22.308 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pyasn1 python3-pyasn1-modules python3-pygments python3-pyinotify 2023-05-22T15:23:22.308 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pyperclip python3-pytest python3-repoze.lru python3-requests 2023-05-22T15:23:22.308 INFO:teuthology.orchestra.run.smithi016.stdout: python3-requests-oauthlib python3-routes python3-rsa python3-simplegeneric 2023-05-22T15:23:22.308 INFO:teuthology.orchestra.run.smithi016.stdout: python3-simplejson python3-singledispatch python3-sklearn 2023-05-22T15:23:22.308 INFO:teuthology.orchestra.run.smithi016.stdout: python3-sklearn-lib python3-tempita python3-tempora python3-threadpoolctl 2023-05-22T15:23:22.308 INFO:teuthology.orchestra.run.smithi016.stdout: python3-toml python3-urllib3 python3-waitress python3-wcwidth python3-webob 2023-05-22T15:23:22.308 INFO:teuthology.orchestra.run.smithi016.stdout: python3-websocket python3-webtest python3-werkzeug python3-zc.lockfile 2023-05-22T15:23:22.308 INFO:teuthology.orchestra.run.smithi016.stdout: qttranslations5-l10n socat xmlstarlet 2023-05-22T15:23:22.309 INFO:teuthology.orchestra.run.smithi016.stdout:Suggested packages: 2023-05-22T15:23:22.309 INFO:teuthology.orchestra.run.smithi016.stdout: python3-influxdb python-jinja2-doc python3-beaker python-mako-doc 2023-05-22T15:23:22.309 INFO:teuthology.orchestra.run.smithi016.stdout: python-natsort-doc python-openssl-doc python3-openssl-dbg httpd-wsgi 2023-05-22T15:23:22.309 INFO:teuthology.orchestra.run.smithi016.stdout: libapache2-mod-python libapache2-mod-scgi libjs-mochikit python-pecan-doc 2023-05-22T15:23:22.309 INFO:teuthology.orchestra.run.smithi016.stdout: python-psutil-doc subversion python-pygments-doc python-pyinotify-doc 2023-05-22T15:23:22.309 INFO:teuthology.orchestra.run.smithi016.stdout: python3-socks python-requests-doc python3-dap python-sklearn-doc ipython3 2023-05-22T15:23:22.309 INFO:teuthology.orchestra.run.smithi016.stdout: python-waitress-doc python-webob-doc python-webtest-doc python-werkzeug-doc 2023-05-22T15:23:22.309 INFO:teuthology.orchestra.run.smithi016.stdout: python3-watchdog 2023-05-22T15:23:22.310 INFO:teuthology.orchestra.run.smithi016.stdout:Recommended packages: 2023-05-22T15:23:22.310 INFO:teuthology.orchestra.run.smithi016.stdout: btrfs-tools 2023-05-22T15:23:22.311 INFO:teuthology.orchestra.run.smithi062.stdout:The following packages were automatically installed and are no longer required: 2023-05-22T15:23:22.311 INFO:teuthology.orchestra.run.smithi062.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2023-05-22T15:23:22.311 INFO:teuthology.orchestra.run.smithi062.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2023-05-22T15:23:22.311 INFO:teuthology.orchestra.run.smithi062.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2023-05-22T15:23:22.311 INFO:teuthology.orchestra.run.smithi062.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2023-05-22T15:23:22.311 INFO:teuthology.orchestra.run.smithi062.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2023-05-22T15:23:22.312 INFO:teuthology.orchestra.run.smithi062.stdout:Use 'sudo apt autoremove' to remove them. 2023-05-22T15:23:22.312 INFO:teuthology.orchestra.run.smithi062.stdout:The following additional packages will be installed: 2023-05-22T15:23:22.312 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2023-05-22T15:23:22.312 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2023-05-22T15:23:22.312 INFO:teuthology.orchestra.run.smithi062.stdout: libdouble-conversion3 libfmt8 libjq1 liblttng-ust1 liboath0 libonig5 2023-05-22T15:23:22.312 INFO:teuthology.orchestra.run.smithi062.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2023-05-22T15:23:22.312 INFO:teuthology.orchestra.run.smithi062.stdout: librdkafka1 librgw2 libsqlite3-mod-ceph libthrift-0.16.0 nvme-cli 2023-05-22T15:23:22.312 INFO:teuthology.orchestra.run.smithi062.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2023-05-22T15:23:22.313 INFO:teuthology.orchestra.run.smithi062.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2023-05-22T15:23:22.313 INFO:teuthology.orchestra.run.smithi062.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2023-05-22T15:23:22.313 INFO:teuthology.orchestra.run.smithi062.stdout: python3-cherrypy3 python3-cmd2 python3-colorama python3-google-auth 2023-05-22T15:23:22.313 INFO:teuthology.orchestra.run.smithi062.stdout: python3-idna python3-iniconfig python3-jaraco.classes 2023-05-22T15:23:22.313 INFO:teuthology.orchestra.run.smithi062.stdout: python3-jaraco.collections python3-jaraco.functools python3-jaraco.text 2023-05-22T15:23:22.313 INFO:teuthology.orchestra.run.smithi062.stdout: python3-jinja2 python3-joblib python3-kubernetes python3-logutils 2023-05-22T15:23:22.313 INFO:teuthology.orchestra.run.smithi062.stdout: python3-mako python3-markupsafe python3-natsort python3-openssl 2023-05-22T15:23:22.314 INFO:teuthology.orchestra.run.smithi062.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2023-05-22T15:23:22.314 INFO:teuthology.orchestra.run.smithi062.stdout: python3-pluggy python3-portend python3-prettytable python3-psutil python3-py 2023-05-22T15:23:22.314 INFO:teuthology.orchestra.run.smithi062.stdout: python3-pyasn1 python3-pyasn1-modules python3-pygments python3-pyinotify 2023-05-22T15:23:22.314 INFO:teuthology.orchestra.run.smithi062.stdout: python3-pyperclip python3-pytest python3-repoze.lru python3-requests 2023-05-22T15:23:22.314 INFO:teuthology.orchestra.run.smithi062.stdout: python3-requests-oauthlib python3-routes python3-rsa python3-simplegeneric 2023-05-22T15:23:22.314 INFO:teuthology.orchestra.run.smithi062.stdout: python3-simplejson python3-singledispatch python3-sklearn 2023-05-22T15:23:22.314 INFO:teuthology.orchestra.run.smithi062.stdout: python3-sklearn-lib python3-tempita python3-tempora python3-threadpoolctl 2023-05-22T15:23:22.314 INFO:teuthology.orchestra.run.smithi062.stdout: python3-toml python3-urllib3 python3-waitress python3-wcwidth python3-webob 2023-05-22T15:23:22.315 INFO:teuthology.orchestra.run.smithi062.stdout: python3-websocket python3-webtest python3-werkzeug python3-zc.lockfile 2023-05-22T15:23:22.315 INFO:teuthology.orchestra.run.smithi062.stdout: qttranslations5-l10n socat xmlstarlet 2023-05-22T15:23:22.315 INFO:teuthology.orchestra.run.smithi062.stdout:Suggested packages: 2023-05-22T15:23:22.315 INFO:teuthology.orchestra.run.smithi062.stdout: python3-influxdb python-jinja2-doc python3-beaker python-mako-doc 2023-05-22T15:23:22.315 INFO:teuthology.orchestra.run.smithi062.stdout: python-natsort-doc python-openssl-doc python3-openssl-dbg httpd-wsgi 2023-05-22T15:23:22.315 INFO:teuthology.orchestra.run.smithi062.stdout: libapache2-mod-python libapache2-mod-scgi libjs-mochikit python-pecan-doc 2023-05-22T15:23:22.315 INFO:teuthology.orchestra.run.smithi062.stdout: python-psutil-doc subversion python-pygments-doc python-pyinotify-doc 2023-05-22T15:23:22.315 INFO:teuthology.orchestra.run.smithi062.stdout: python3-socks python-requests-doc python3-dap python-sklearn-doc ipython3 2023-05-22T15:23:22.316 INFO:teuthology.orchestra.run.smithi062.stdout: python-waitress-doc python-webob-doc python-webtest-doc python-werkzeug-doc 2023-05-22T15:23:22.316 INFO:teuthology.orchestra.run.smithi062.stdout: python3-watchdog 2023-05-22T15:23:22.316 INFO:teuthology.orchestra.run.smithi062.stdout:Recommended packages: 2023-05-22T15:23:22.316 INFO:teuthology.orchestra.run.smithi062.stdout: btrfs-tools 2023-05-22T15:23:22.406 INFO:teuthology.orchestra.run.smithi062.stdout:The following NEW packages will be installed: 2023-05-22T15:23:22.407 INFO:teuthology.orchestra.run.smithi062.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2023-05-22T15:23:22.407 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2023-05-22T15:23:22.407 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm 2023-05-22T15:23:22.407 INFO:teuthology.orchestra.run.smithi062.stdout: cephfs-mirror cephfs-shell cephfs-top jq libcephfs-dev libcephfs2 2023-05-22T15:23:22.407 INFO:teuthology.orchestra.run.smithi062.stdout: libdouble-conversion3 libfmt8 libjq1 liblttng-ust1 liboath0 libonig5 2023-05-22T15:23:22.408 INFO:teuthology.orchestra.run.smithi062.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 librados2 2023-05-22T15:23:22.408 INFO:teuthology.orchestra.run.smithi062.stdout: libradosstriper1 librbd1 librdkafka1 librgw2 libsqlite3-mod-ceph 2023-05-22T15:23:22.408 INFO:teuthology.orchestra.run.smithi062.stdout: libthrift-0.16.0 nvme-cli python-asyncssh-doc python-babel-localedata 2023-05-22T15:23:22.408 INFO:teuthology.orchestra.run.smithi062.stdout: python-pastedeploy-tpl python3-asyncssh python3-babel python3-bcrypt 2023-05-22T15:23:22.409 INFO:teuthology.orchestra.run.smithi062.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-cephfs 2023-05-22T15:23:22.409 INFO:teuthology.orchestra.run.smithi062.stdout: python3-certifi python3-cheroot python3-cherrypy3 python3-cmd2 2023-05-22T15:23:22.409 INFO:teuthology.orchestra.run.smithi062.stdout: python3-colorama python3-google-auth python3-idna python3-iniconfig 2023-05-22T15:23:22.410 INFO:teuthology.orchestra.run.smithi062.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2023-05-22T15:23:22.410 INFO:teuthology.orchestra.run.smithi062.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2023-05-22T15:23:22.410 INFO:teuthology.orchestra.run.smithi062.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2023-05-22T15:23:22.410 INFO:teuthology.orchestra.run.smithi062.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2023-05-22T15:23:22.410 INFO:teuthology.orchestra.run.smithi062.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2023-05-22T15:23:22.411 INFO:teuthology.orchestra.run.smithi062.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2023-05-22T15:23:22.411 INFO:teuthology.orchestra.run.smithi062.stdout: python3-pygments python3-pyinotify python3-pyperclip python3-pytest 2023-05-22T15:23:22.411 INFO:teuthology.orchestra.run.smithi062.stdout: python3-rados python3-rbd python3-repoze.lru python3-requests 2023-05-22T15:23:22.411 INFO:teuthology.orchestra.run.smithi062.stdout: python3-requests-oauthlib python3-rgw python3-routes python3-rsa 2023-05-22T15:23:22.411 INFO:teuthology.orchestra.run.smithi062.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2023-05-22T15:23:22.411 INFO:teuthology.orchestra.run.smithi062.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2023-05-22T15:23:22.411 INFO:teuthology.orchestra.run.smithi062.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2023-05-22T15:23:22.412 INFO:teuthology.orchestra.run.smithi062.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2023-05-22T15:23:22.412 INFO:teuthology.orchestra.run.smithi062.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n radosgw rbd-fuse 2023-05-22T15:23:22.412 INFO:teuthology.orchestra.run.smithi062.stdout: socat xmlstarlet 2023-05-22T15:23:22.413 INFO:teuthology.orchestra.run.smithi016.stdout:The following NEW packages will be installed: 2023-05-22T15:23:22.414 INFO:teuthology.orchestra.run.smithi016.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2023-05-22T15:23:22.414 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2023-05-22T15:23:22.414 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm 2023-05-22T15:23:22.414 INFO:teuthology.orchestra.run.smithi016.stdout: cephfs-mirror cephfs-shell cephfs-top jq libcephfs-dev libcephfs2 2023-05-22T15:23:22.414 INFO:teuthology.orchestra.run.smithi016.stdout: libdouble-conversion3 libfmt8 libjq1 liblttng-ust1 liboath0 libonig5 2023-05-22T15:23:22.414 INFO:teuthology.orchestra.run.smithi016.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 librados2 2023-05-22T15:23:22.415 INFO:teuthology.orchestra.run.smithi016.stdout: libradosstriper1 librbd1 librdkafka1 librgw2 libsqlite3-mod-ceph 2023-05-22T15:23:22.415 INFO:teuthology.orchestra.run.smithi016.stdout: libthrift-0.16.0 nvme-cli python-asyncssh-doc python-babel-localedata 2023-05-22T15:23:22.415 INFO:teuthology.orchestra.run.smithi016.stdout: python-pastedeploy-tpl python3-asyncssh python3-babel python3-bcrypt 2023-05-22T15:23:22.415 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-cephfs 2023-05-22T15:23:22.415 INFO:teuthology.orchestra.run.smithi016.stdout: python3-certifi python3-cheroot python3-cherrypy3 python3-cmd2 2023-05-22T15:23:22.415 INFO:teuthology.orchestra.run.smithi016.stdout: python3-colorama python3-google-auth python3-idna python3-iniconfig 2023-05-22T15:23:22.415 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2023-05-22T15:23:22.416 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2023-05-22T15:23:22.416 INFO:teuthology.orchestra.run.smithi016.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2023-05-22T15:23:22.416 INFO:teuthology.orchestra.run.smithi016.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2023-05-22T15:23:22.417 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2023-05-22T15:23:22.417 INFO:teuthology.orchestra.run.smithi016.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2023-05-22T15:23:22.417 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pygments python3-pyinotify python3-pyperclip python3-pytest 2023-05-22T15:23:22.417 INFO:teuthology.orchestra.run.smithi016.stdout: python3-rados python3-rbd python3-repoze.lru python3-requests 2023-05-22T15:23:22.417 INFO:teuthology.orchestra.run.smithi016.stdout: python3-requests-oauthlib python3-rgw python3-routes python3-rsa 2023-05-22T15:23:22.417 INFO:teuthology.orchestra.run.smithi016.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2023-05-22T15:23:22.418 INFO:teuthology.orchestra.run.smithi016.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2023-05-22T15:23:22.418 INFO:teuthology.orchestra.run.smithi016.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2023-05-22T15:23:22.418 INFO:teuthology.orchestra.run.smithi016.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2023-05-22T15:23:22.418 INFO:teuthology.orchestra.run.smithi016.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n radosgw rbd-fuse 2023-05-22T15:23:22.418 INFO:teuthology.orchestra.run.smithi016.stdout: socat xmlstarlet 2023-05-22T15:23:22.470 INFO:teuthology.orchestra.run.smithi016.stdout:0 upgraded, 115 newly installed, 0 to remove and 158 not upgraded. 2023-05-22T15:23:22.470 INFO:teuthology.orchestra.run.smithi016.stdout:Need to get 180 MB of archives. 2023-05-22T15:23:22.470 INFO:teuthology.orchestra.run.smithi016.stdout:After this operation, 808 MB of additional disk space will be used. 2023-05-22T15:23:22.471 INFO:teuthology.orchestra.run.smithi016.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyperclip all 1.8.2-2 [11.5 kB] 2023-05-22T15:23:22.487 INFO:teuthology.orchestra.run.smithi016.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2023-05-22T15:23:22.497 INFO:teuthology.orchestra.run.smithi016.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cmd2 all 2.3.3-1 [124 kB] 2023-05-22T15:23:22.537 INFO:teuthology.orchestra.run.smithi016.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libfmt8 amd64 8.1.1+ds1-2 [60.2 kB] 2023-05-22T15:23:22.546 INFO:teuthology.orchestra.run.smithi016.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2023-05-22T15:23:22.551 INFO:teuthology.orchestra.run.smithi062.stdout:0 upgraded, 115 newly installed, 0 to remove and 158 not upgraded. 2023-05-22T15:23:22.551 INFO:teuthology.orchestra.run.smithi062.stdout:Need to get 180 MB of archives. 2023-05-22T15:23:22.551 INFO:teuthology.orchestra.run.smithi062.stdout:After this operation, 808 MB of additional disk space will be used. 2023-05-22T15:23:22.551 INFO:teuthology.orchestra.run.smithi062.stdout:Get:1 https://4.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-05-21-0740-reef/9c9ba4df22024d9d8d6cf9f98a4b96751fc78d0c/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 18.0.0-3192-g9c9ba4df-1jammy [3,400 kB] 2023-05-22T15:23:22.554 INFO:teuthology.orchestra.run.smithi016.stdout:Get:6 https://4.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-05-21-0740-reef/9c9ba4df22024d9d8d6cf9f98a4b96751fc78d0c/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 18.0.0-3192-g9c9ba4df-1jammy [3,400 kB] 2023-05-22T15:23:22.562 INFO:teuthology.orchestra.run.smithi016.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2023-05-22T15:23:22.565 INFO:teuthology.orchestra.run.smithi016.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2023-05-22T15:23:22.574 INFO:teuthology.orchestra.run.smithi016.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2023-05-22T15:23:22.593 INFO:teuthology.orchestra.run.smithi062.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyperclip all 1.8.2-2 [11.5 kB] 2023-05-22T15:23:22.621 INFO:teuthology.orchestra.run.smithi016.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2023-05-22T15:23:22.625 INFO:teuthology.orchestra.run.smithi016.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2023-05-22T15:23:22.637 INFO:teuthology.orchestra.run.smithi016.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2023-05-22T15:23:22.675 INFO:teuthology.orchestra.run.smithi016.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2023-05-22T15:23:22.676 INFO:teuthology.orchestra.run.smithi062.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2023-05-22T15:23:22.716 INFO:teuthology.orchestra.run.smithi016.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2023-05-22T15:23:22.721 INFO:teuthology.orchestra.run.smithi062.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cmd2 all 2.3.3-1 [124 kB] 2023-05-22T15:23:22.791 INFO:teuthology.orchestra.run.smithi016.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2023-05-22T15:23:22.792 INFO:teuthology.orchestra.run.smithi016.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2023-05-22T15:23:22.793 INFO:teuthology.orchestra.run.smithi016.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2023-05-22T15:23:22.796 INFO:teuthology.orchestra.run.smithi016.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2023-05-22T15:23:22.796 INFO:teuthology.orchestra.run.smithi016.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2023-05-22T15:23:22.797 INFO:teuthology.orchestra.run.smithi016.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2023-05-22T15:23:22.833 INFO:teuthology.orchestra.run.smithi016.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2023-05-22T15:23:22.838 INFO:teuthology.orchestra.run.smithi016.stdout:Get:22 https://4.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-05-21-0740-reef/9c9ba4df22024d9d8d6cf9f98a4b96751fc78d0c/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 18.0.0-3192-g9c9ba4df-1jammy [3,095 kB] 2023-05-22T15:23:22.850 INFO:teuthology.orchestra.run.smithi016.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2023-05-22T15:23:22.853 INFO:teuthology.orchestra.run.smithi016.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2023-05-22T15:23:22.857 INFO:teuthology.orchestra.run.smithi016.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2023-05-22T15:23:22.860 INFO:teuthology.orchestra.run.smithi062.stdout:Get:5 https://4.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-05-21-0740-reef/9c9ba4df22024d9d8d6cf9f98a4b96751fc78d0c/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 18.0.0-3192-g9c9ba4df-1jammy [3,095 kB] 2023-05-22T15:23:22.909 INFO:teuthology.orchestra.run.smithi016.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2023-05-22T15:23:22.912 INFO:teuthology.orchestra.run.smithi016.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2023-05-22T15:23:22.913 INFO:teuthology.orchestra.run.smithi016.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2023-05-22T15:23:22.914 INFO:teuthology.orchestra.run.smithi016.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2023-05-22T15:23:22.916 INFO:teuthology.orchestra.run.smithi062.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libfmt8 amd64 8.1.1+ds1-2 [60.2 kB] 2023-05-22T15:23:22.919 INFO:teuthology.orchestra.run.smithi016.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2023-05-22T15:23:22.920 INFO:teuthology.orchestra.run.smithi016.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2023-05-22T15:23:22.921 INFO:teuthology.orchestra.run.smithi016.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB] 2023-05-22T15:23:22.960 INFO:teuthology.orchestra.run.smithi062.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2023-05-22T15:23:22.960 INFO:teuthology.orchestra.run.smithi016.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2023-05-22T15:23:23.013 INFO:teuthology.orchestra.run.smithi016.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2023-05-22T15:23:23.015 INFO:teuthology.orchestra.run.smithi016.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2023-05-22T15:23:23.026 INFO:teuthology.orchestra.run.smithi016.stdout:Get:36 https://4.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-05-21-0740-reef/9c9ba4df22024d9d8d6cf9f98a4b96751fc78d0c/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 18.0.0-3192-g9c9ba4df-1jammy [652 kB] 2023-05-22T15:23:23.034 INFO:teuthology.orchestra.run.smithi062.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2023-05-22T15:23:23.046 INFO:teuthology.orchestra.run.smithi062.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2023-05-22T15:23:23.046 INFO:teuthology.orchestra.run.smithi016.stdout:Get:37 https://4.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-05-21-0740-reef/9c9ba4df22024d9d8d6cf9f98a4b96751fc78d0c/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 18.0.0-3192-g9c9ba4df-1jammy [343 kB] 2023-05-22T15:23:23.066 INFO:teuthology.orchestra.run.smithi062.stdout:Get:10 https://4.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-05-21-0740-reef/9c9ba4df22024d9d8d6cf9f98a4b96751fc78d0c/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 18.0.0-3192-g9c9ba4df-1jammy [652 kB] 2023-05-22T15:23:23.072 INFO:teuthology.orchestra.run.smithi016.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2023-05-22T15:23:23.072 INFO:teuthology.orchestra.run.smithi016.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2023-05-22T15:23:23.074 INFO:teuthology.orchestra.run.smithi016.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2023-05-22T15:23:23.075 INFO:teuthology.orchestra.run.smithi016.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2023-05-22T15:23:23.078 INFO:teuthology.orchestra.run.smithi016.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2023-05-22T15:23:23.083 INFO:teuthology.orchestra.run.smithi016.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-idna all 3.3-1 [49.3 kB] 2023-05-22T15:23:23.085 INFO:teuthology.orchestra.run.smithi016.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-urllib3 all 1.26.5-1~exp1 [96.4 kB] 2023-05-22T15:23:23.086 INFO:teuthology.orchestra.run.smithi062.stdout:Get:11 https://4.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-05-21-0740-reef/9c9ba4df22024d9d8d6cf9f98a4b96751fc78d0c/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 18.0.0-3192-g9c9ba4df-1jammy [343 kB] 2023-05-22T15:23:23.088 INFO:teuthology.orchestra.run.smithi016.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-requests all 2.25.1+dfsg-2 [47.9 kB] 2023-05-22T15:23:23.091 INFO:teuthology.orchestra.run.smithi062.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2023-05-22T15:23:23.093 INFO:teuthology.orchestra.run.smithi016.stdout:Get:46 https://4.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-05-21-0740-reef/9c9ba4df22024d9d8d6cf9f98a4b96751fc78d0c/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 18.0.0-3192-g9c9ba4df-1jammy [22.9 kB] 2023-05-22T15:23:23.093 INFO:teuthology.orchestra.run.smithi016.stdout:Get:47 https://4.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-05-21-0740-reef/9c9ba4df22024d9d8d6cf9f98a4b96751fc78d0c/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 18.0.0-3192-g9c9ba4df-1jammy [164 kB] 2023-05-22T15:23:23.098 INFO:teuthology.orchestra.run.smithi016.stdout:Get:48 https://4.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-05-21-0740-reef/9c9ba4df22024d9d8d6cf9f98a4b96751fc78d0c/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 18.0.0-3192-g9c9ba4df-1jammy [48.5 kB] 2023-05-22T15:23:23.100 INFO:teuthology.orchestra.run.smithi016.stdout:Get:49 https://4.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-05-21-0740-reef/9c9ba4df22024d9d8d6cf9f98a4b96751fc78d0c/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 18.0.0-3192-g9c9ba4df-1jammy [319 kB] 2023-05-22T15:23:23.111 INFO:teuthology.orchestra.run.smithi016.stdout:Get:50 https://4.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-05-21-0740-reef/9c9ba4df22024d9d8d6cf9f98a4b96751fc78d0c/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 18.0.0-3192-g9c9ba4df-1jammy [5,839 kB] 2023-05-22T15:23:23.124 INFO:teuthology.orchestra.run.smithi016.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.1 [181 kB] 2023-05-22T15:23:23.137 INFO:teuthology.orchestra.run.smithi062.stdout:Get:13 https://4.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-05-21-0740-reef/9c9ba4df22024d9d8d6cf9f98a4b96751fc78d0c/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 18.0.0-3192-g9c9ba4df-1jammy [22.9 kB] 2023-05-22T15:23:23.137 INFO:teuthology.orchestra.run.smithi062.stdout:Get:14 https://4.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-05-21-0740-reef/9c9ba4df22024d9d8d6cf9f98a4b96751fc78d0c/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 18.0.0-3192-g9c9ba4df-1jammy [164 kB] 2023-05-22T15:23:23.142 INFO:teuthology.orchestra.run.smithi062.stdout:Get:15 https://4.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-05-21-0740-reef/9c9ba4df22024d9d8d6cf9f98a4b96751fc78d0c/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 18.0.0-3192-g9c9ba4df-1jammy [48.5 kB] 2023-05-22T15:23:23.144 INFO:teuthology.orchestra.run.smithi062.stdout:Get:16 https://4.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-05-21-0740-reef/9c9ba4df22024d9d8d6cf9f98a4b96751fc78d0c/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 18.0.0-3192-g9c9ba4df-1jammy [319 kB] 2023-05-22T15:23:23.154 INFO:teuthology.orchestra.run.smithi062.stdout:Get:17 https://4.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-05-21-0740-reef/9c9ba4df22024d9d8d6cf9f98a4b96751fc78d0c/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 18.0.0-3192-g9c9ba4df-1jammy [5,839 kB] 2023-05-22T15:23:23.210 INFO:teuthology.orchestra.run.smithi016.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2023-05-22T15:23:23.213 INFO:teuthology.orchestra.run.smithi016.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2023-05-22T15:23:23.229 INFO:teuthology.orchestra.run.smithi016.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2023-05-22T15:23:23.257 INFO:teuthology.orchestra.run.smithi062.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2023-05-22T15:23:23.266 INFO:teuthology.orchestra.run.smithi062.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2023-05-22T15:23:23.292 INFO:teuthology.orchestra.run.smithi062.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2023-05-22T15:23:23.332 INFO:teuthology.orchestra.run.smithi016.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2023-05-22T15:23:23.333 INFO:teuthology.orchestra.run.smithi016.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jinja2 all 3.0.3-1 [108 kB] 2023-05-22T15:23:23.335 INFO:teuthology.orchestra.run.smithi016.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2023-05-22T15:23:23.335 INFO:teuthology.orchestra.run.smithi016.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2023-05-22T15:23:23.335 INFO:teuthology.orchestra.run.smithi016.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2023-05-22T15:23:23.337 INFO:teuthology.orchestra.run.smithi016.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2023-05-22T15:23:23.372 INFO:teuthology.orchestra.run.smithi016.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2023-05-22T15:23:23.390 INFO:teuthology.orchestra.run.smithi016.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2023-05-22T15:23:23.446 INFO:teuthology.orchestra.run.smithi016.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2023-05-22T15:23:23.448 INFO:teuthology.orchestra.run.smithi016.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2023-05-22T15:23:23.455 INFO:teuthology.orchestra.run.smithi016.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2023-05-22T15:23:23.464 INFO:teuthology.orchestra.run.smithi062.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2023-05-22T15:23:23.483 INFO:teuthology.orchestra.run.smithi016.stdout:Get:66 https://4.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-05-21-0740-reef/9c9ba4df22024d9d8d6cf9f98a4b96751fc78d0c/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 18.0.0-3192-g9c9ba4df-1jammy [102 kB] 2023-05-22T15:23:23.485 INFO:teuthology.orchestra.run.smithi016.stdout:Get:67 https://4.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-05-21-0740-reef/9c9ba4df22024d9d8d6cf9f98a4b96751fc78d0c/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 18.0.0-3192-g9c9ba4df-1jammy [431 kB] 2023-05-22T15:23:23.515 INFO:teuthology.orchestra.run.smithi016.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2023-05-22T15:23:23.519 INFO:teuthology.orchestra.run.smithi016.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2023-05-22T15:23:23.520 INFO:teuthology.orchestra.run.smithi016.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2023-05-22T15:23:23.528 INFO:teuthology.orchestra.run.smithi016.stdout:Get:71 https://4.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-05-21-0740-reef/9c9ba4df22024d9d8d6cf9f98a4b96751fc78d0c/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 18.0.0-3192-g9c9ba4df-1jammy [21.8 MB] 2023-05-22T15:23:23.547 INFO:teuthology.orchestra.run.smithi016.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2023-05-22T15:23:23.547 INFO:teuthology.orchestra.run.smithi016.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2023-05-22T15:23:23.548 INFO:teuthology.orchestra.run.smithi016.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2023-05-22T15:23:23.557 INFO:teuthology.orchestra.run.smithi062.stdout:Get:22 https://4.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-05-21-0740-reef/9c9ba4df22024d9d8d6cf9f98a4b96751fc78d0c/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 18.0.0-3192-g9c9ba4df-1jammy [102 kB] 2023-05-22T15:23:23.559 INFO:teuthology.orchestra.run.smithi062.stdout:Get:23 https://4.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-05-21-0740-reef/9c9ba4df22024d9d8d6cf9f98a4b96751fc78d0c/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 18.0.0-3192-g9c9ba4df-1jammy [431 kB] 2023-05-22T15:23:23.573 INFO:teuthology.orchestra.run.smithi062.stdout:Get:24 https://4.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-05-21-0740-reef/9c9ba4df22024d9d8d6cf9f98a4b96751fc78d0c/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 18.0.0-3192-g9c9ba4df-1jammy [21.8 MB] 2023-05-22T15:23:23.583 INFO:teuthology.orchestra.run.smithi016.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2023-05-22T15:23:23.623 INFO:teuthology.orchestra.run.smithi016.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2023-05-22T15:23:23.635 INFO:teuthology.orchestra.run.smithi016.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2023-05-22T15:23:23.643 INFO:teuthology.orchestra.run.smithi016.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2023-05-22T15:23:23.655 INFO:teuthology.orchestra.run.smithi016.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2023-05-22T15:23:23.658 INFO:teuthology.orchestra.run.smithi062.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2023-05-22T15:23:23.690 INFO:teuthology.orchestra.run.smithi016.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2023-05-22T15:23:23.698 INFO:teuthology.orchestra.run.smithi016.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2023-05-22T15:23:23.703 INFO:teuthology.orchestra.run.smithi016.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2023-05-22T15:23:23.705 INFO:teuthology.orchestra.run.smithi016.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2023-05-22T15:23:23.714 INFO:teuthology.orchestra.run.smithi016.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2023-05-22T15:23:23.721 INFO:teuthology.orchestra.run.smithi016.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-colorama all 0.4.4-1 [24.5 kB] 2023-05-22T15:23:23.756 INFO:teuthology.orchestra.run.smithi016.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.1 [474 kB] 2023-05-22T15:23:23.867 INFO:teuthology.orchestra.run.smithi016.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2023-05-22T15:23:23.875 INFO:teuthology.orchestra.run.smithi016.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2023-05-22T15:23:23.877 INFO:teuthology.orchestra.run.smithi016.stdout:Get:89 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2023-05-22T15:23:23.885 INFO:teuthology.orchestra.run.smithi016.stdout:Get:90 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2023-05-22T15:23:23.904 INFO:teuthology.orchestra.run.smithi016.stdout:Get:91 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2023-05-22T15:23:23.905 INFO:teuthology.orchestra.run.smithi016.stdout:Get:92 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2023-05-22T15:23:23.906 INFO:teuthology.orchestra.run.smithi016.stdout:Get:93 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2023-05-22T15:23:24.025 INFO:teuthology.orchestra.run.smithi062.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2023-05-22T15:23:24.032 INFO:teuthology.orchestra.run.smithi062.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2023-05-22T15:23:24.033 INFO:teuthology.orchestra.run.smithi062.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2023-05-22T15:23:24.045 INFO:teuthology.orchestra.run.smithi062.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2023-05-22T15:23:24.046 INFO:teuthology.orchestra.run.smithi062.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2023-05-22T15:23:24.047 INFO:teuthology.orchestra.run.smithi062.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2023-05-22T15:23:24.049 INFO:teuthology.orchestra.run.smithi062.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2023-05-22T15:23:24.052 INFO:teuthology.orchestra.run.smithi062.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2023-05-22T15:23:24.053 INFO:teuthology.orchestra.run.smithi062.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2023-05-22T15:23:24.218 INFO:teuthology.orchestra.run.smithi062.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2023-05-22T15:23:24.649 INFO:teuthology.orchestra.run.smithi062.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2023-05-22T15:23:24.659 INFO:teuthology.orchestra.run.smithi062.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2023-05-22T15:23:24.665 INFO:teuthology.orchestra.run.smithi062.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2023-05-22T15:23:24.669 INFO:teuthology.orchestra.run.smithi062.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2023-05-22T15:23:24.686 INFO:teuthology.orchestra.run.smithi062.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2023-05-22T15:23:24.689 INFO:teuthology.orchestra.run.smithi062.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2023-05-22T15:23:24.692 INFO:teuthology.orchestra.run.smithi062.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB] 2023-05-22T15:23:24.734 INFO:teuthology.orchestra.run.smithi062.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2023-05-22T15:23:24.748 INFO:teuthology.orchestra.run.smithi062.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2023-05-22T15:23:24.752 INFO:teuthology.orchestra.run.smithi062.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2023-05-22T15:23:24.976 INFO:teuthology.orchestra.run.smithi016.stdout:Get:94 https://4.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-05-21-0740-reef/9c9ba4df22024d9d8d6cf9f98a4b96751fc78d0c/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 18.0.0-3192-g9c9ba4df-1jammy [6,821 kB] 2023-05-22T15:23:25.005 INFO:teuthology.orchestra.run.smithi062.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2023-05-22T15:23:25.089 INFO:teuthology.orchestra.run.smithi062.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2023-05-22T15:23:25.119 INFO:teuthology.orchestra.run.smithi062.stdout:Get:48 https://4.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-05-21-0740-reef/9c9ba4df22024d9d8d6cf9f98a4b96751fc78d0c/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 18.0.0-3192-g9c9ba4df-1jammy [6,821 kB] 2023-05-22T15:23:25.179 INFO:teuthology.orchestra.run.smithi062.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2023-05-22T15:23:25.236 INFO:teuthology.orchestra.run.smithi062.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2023-05-22T15:23:25.350 INFO:teuthology.orchestra.run.smithi016.stdout:Get:95 https://4.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-05-21-0740-reef/9c9ba4df22024d9d8d6cf9f98a4b96751fc78d0c/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 18.0.0-3192-g9c9ba4df-1jammy [225 kB] 2023-05-22T15:23:25.353 INFO:teuthology.orchestra.run.smithi062.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2023-05-22T15:23:25.379 INFO:teuthology.orchestra.run.smithi016.stdout:Get:96 https://4.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-05-21-0740-reef/9c9ba4df22024d9d8d6cf9f98a4b96751fc78d0c/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 18.0.0-3192-g9c9ba4df-1jammy [114 kB] 2023-05-22T15:23:25.383 INFO:teuthology.orchestra.run.smithi016.stdout:Get:97 https://4.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-05-21-0740-reef/9c9ba4df22024d9d8d6cf9f98a4b96751fc78d0c/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 18.0.0-3192-g9c9ba4df-1jammy [1,465 kB] 2023-05-22T15:23:25.435 INFO:teuthology.orchestra.run.smithi062.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-idna all 3.3-1 [49.3 kB] 2023-05-22T15:23:25.454 INFO:teuthology.orchestra.run.smithi062.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-urllib3 all 1.26.5-1~exp1 [96.4 kB] 2023-05-22T15:23:25.467 INFO:teuthology.orchestra.run.smithi016.stdout:Get:98 https://4.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-05-21-0740-reef/9c9ba4df22024d9d8d6cf9f98a4b96751fc78d0c/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 18.0.0-3192-g9c9ba4df-1jammy [9,864 kB] 2023-05-22T15:23:25.483 INFO:teuthology.orchestra.run.smithi062.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-requests all 2.25.1+dfsg-2 [47.9 kB] 2023-05-22T15:23:25.495 INFO:teuthology.orchestra.run.smithi062.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.1 [181 kB] 2023-05-22T15:23:25.508 INFO:teuthology.orchestra.run.smithi062.stdout:Get:56 https://4.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-05-21-0740-reef/9c9ba4df22024d9d8d6cf9f98a4b96751fc78d0c/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 18.0.0-3192-g9c9ba4df-1jammy [225 kB] 2023-05-22T15:23:25.514 INFO:teuthology.orchestra.run.smithi062.stdout:Get:57 https://4.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-05-21-0740-reef/9c9ba4df22024d9d8d6cf9f98a4b96751fc78d0c/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 18.0.0-3192-g9c9ba4df-1jammy [114 kB] 2023-05-22T15:23:25.532 INFO:teuthology.orchestra.run.smithi062.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2023-05-22T15:23:25.541 INFO:teuthology.orchestra.run.smithi062.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2023-05-22T15:23:25.548 INFO:teuthology.orchestra.run.smithi062.stdout:Get:60 https://4.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-05-21-0740-reef/9c9ba4df22024d9d8d6cf9f98a4b96751fc78d0c/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 18.0.0-3192-g9c9ba4df-1jammy [1,465 kB] 2023-05-22T15:23:25.640 INFO:teuthology.orchestra.run.smithi062.stdout:Get:61 https://4.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-05-21-0740-reef/9c9ba4df22024d9d8d6cf9f98a4b96751fc78d0c/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 18.0.0-3192-g9c9ba4df-1jammy [9,864 kB] 2023-05-22T15:23:25.731 INFO:teuthology.orchestra.run.smithi062.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2023-05-22T15:23:26.170 INFO:teuthology.orchestra.run.smithi016.stdout:Get:99 https://4.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-05-21-0740-reef/9c9ba4df22024d9d8d6cf9f98a4b96751fc78d0c/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 18.0.0-3192-g9c9ba4df-1jammy [31.5 MB] 2023-05-22T15:23:26.322 INFO:teuthology.orchestra.run.smithi062.stdout:Get:63 https://4.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-05-21-0740-reef/9c9ba4df22024d9d8d6cf9f98a4b96751fc78d0c/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 18.0.0-3192-g9c9ba4df-1jammy [31.5 MB] 2023-05-22T15:23:26.541 INFO:teuthology.orchestra.run.smithi062.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2023-05-22T15:23:26.544 INFO:teuthology.orchestra.run.smithi062.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jinja2 all 3.0.3-1 [108 kB] 2023-05-22T15:23:26.547 INFO:teuthology.orchestra.run.smithi062.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2023-05-22T15:23:26.548 INFO:teuthology.orchestra.run.smithi062.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2023-05-22T15:23:26.549 INFO:teuthology.orchestra.run.smithi062.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2023-05-22T15:23:26.551 INFO:teuthology.orchestra.run.smithi062.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2023-05-22T15:23:26.552 INFO:teuthology.orchestra.run.smithi062.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2023-05-22T15:23:26.553 INFO:teuthology.orchestra.run.smithi062.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2023-05-22T15:23:26.558 INFO:teuthology.orchestra.run.smithi062.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2023-05-22T15:23:26.623 INFO:teuthology.orchestra.run.smithi062.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2023-05-22T15:23:26.789 INFO:teuthology.orchestra.run.smithi062.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2023-05-22T15:23:27.493 INFO:teuthology.orchestra.run.smithi062.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2023-05-22T15:23:27.498 INFO:teuthology.orchestra.run.smithi062.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2023-05-22T15:23:27.499 INFO:teuthology.orchestra.run.smithi062.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2023-05-22T15:23:27.551 INFO:teuthology.orchestra.run.smithi062.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2023-05-22T15:23:27.575 INFO:teuthology.orchestra.run.smithi062.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2023-05-22T15:23:27.576 INFO:teuthology.orchestra.run.smithi062.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2023-05-22T15:23:27.577 INFO:teuthology.orchestra.run.smithi062.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2023-05-22T15:23:27.579 INFO:teuthology.orchestra.run.smithi062.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2023-05-22T15:23:27.580 INFO:teuthology.orchestra.run.smithi062.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2023-05-22T15:23:27.747 INFO:teuthology.orchestra.run.smithi062.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2023-05-22T15:23:27.868 INFO:teuthology.orchestra.run.smithi062.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2023-05-22T15:23:28.171 INFO:teuthology.orchestra.run.smithi062.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2023-05-22T15:23:28.216 INFO:teuthology.orchestra.run.smithi062.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2023-05-22T15:23:28.242 INFO:teuthology.orchestra.run.smithi062.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2023-05-22T15:23:28.307 INFO:teuthology.orchestra.run.smithi062.stdout:Get:89 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2023-05-22T15:23:28.382 INFO:teuthology.orchestra.run.smithi062.stdout:Get:90 https://4.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-05-21-0740-reef/9c9ba4df22024d9d8d6cf9f98a4b96751fc78d0c/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 18.0.0-3192-g9c9ba4df-1jammy [3,594 B] 2023-05-22T15:23:28.382 INFO:teuthology.orchestra.run.smithi062.stdout:Get:91 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2023-05-22T15:23:28.382 INFO:teuthology.orchestra.run.smithi062.stdout:Get:92 https://4.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-05-21-0740-reef/9c9ba4df22024d9d8d6cf9f98a4b96751fc78d0c/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 18.0.0-3192-g9c9ba4df-1jammy [806 kB] 2023-05-22T15:23:28.388 INFO:teuthology.orchestra.run.smithi062.stdout:Get:93 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-colorama all 0.4.4-1 [24.5 kB] 2023-05-22T15:23:28.389 INFO:teuthology.orchestra.run.smithi062.stdout:Get:94 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.1 [474 kB] 2023-05-22T15:23:28.390 INFO:teuthology.orchestra.run.smithi016.stdout:Get:100 https://4.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-05-21-0740-reef/9c9ba4df22024d9d8d6cf9f98a4b96751fc78d0c/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 18.0.0-3192-g9c9ba4df-1jammy [3,594 B] 2023-05-22T15:23:28.390 INFO:teuthology.orchestra.run.smithi016.stdout:Get:101 https://4.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-05-21-0740-reef/9c9ba4df22024d9d8d6cf9f98a4b96751fc78d0c/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 18.0.0-3192-g9c9ba4df-1jammy [806 kB] 2023-05-22T15:23:28.398 INFO:teuthology.orchestra.run.smithi062.stdout:Get:95 https://4.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-05-21-0740-reef/9c9ba4df22024d9d8d6cf9f98a4b96751fc78d0c/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 18.0.0-3192-g9c9ba4df-1jammy [2,144 kB] 2023-05-22T15:23:28.400 INFO:teuthology.orchestra.run.smithi062.stdout:Get:96 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2023-05-22T15:23:28.407 INFO:teuthology.orchestra.run.smithi016.stdout:Get:102 https://4.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-05-21-0740-reef/9c9ba4df22024d9d8d6cf9f98a4b96751fc78d0c/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 18.0.0-3192-g9c9ba4df-1jammy [2,144 kB] 2023-05-22T15:23:28.407 INFO:teuthology.orchestra.run.smithi062.stdout:Get:97 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2023-05-22T15:23:28.438 INFO:teuthology.orchestra.run.smithi062.stdout:Get:98 https://4.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-05-21-0740-reef/9c9ba4df22024d9d8d6cf9f98a4b96751fc78d0c/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 18.0.0-3192-g9c9ba4df-1jammy [96.3 kB] 2023-05-22T15:23:28.439 INFO:teuthology.orchestra.run.smithi062.stdout:Get:99 https://4.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-05-21-0740-reef/9c9ba4df22024d9d8d6cf9f98a4b96751fc78d0c/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 18.0.0-3192-g9c9ba4df-1jammy [120 kB] 2023-05-22T15:23:28.442 INFO:teuthology.orchestra.run.smithi062.stdout:Get:100 https://4.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-05-21-0740-reef/9c9ba4df22024d9d8d6cf9f98a4b96751fc78d0c/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 18.0.0-3192-g9c9ba4df-1jammy [1,823 kB] 2023-05-22T15:23:28.522 INFO:teuthology.orchestra.run.smithi016.stdout:Get:103 https://4.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-05-21-0740-reef/9c9ba4df22024d9d8d6cf9f98a4b96751fc78d0c/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 18.0.0-3192-g9c9ba4df-1jammy [96.3 kB] 2023-05-22T15:23:28.525 INFO:teuthology.orchestra.run.smithi016.stdout:Get:104 https://4.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-05-21-0740-reef/9c9ba4df22024d9d8d6cf9f98a4b96751fc78d0c/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 18.0.0-3192-g9c9ba4df-1jammy [120 kB] 2023-05-22T15:23:28.529 INFO:teuthology.orchestra.run.smithi016.stdout:Get:105 https://4.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-05-21-0740-reef/9c9ba4df22024d9d8d6cf9f98a4b96751fc78d0c/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 18.0.0-3192-g9c9ba4df-1jammy [1,823 kB] 2023-05-22T15:23:28.553 INFO:teuthology.orchestra.run.smithi062.stdout:Get:101 https://4.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-05-21-0740-reef/9c9ba4df22024d9d8d6cf9f98a4b96751fc78d0c/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 18.0.0-3192-g9c9ba4df-1jammy [8,615 kB] 2023-05-22T15:23:28.573 INFO:teuthology.orchestra.run.smithi062.stdout:Get:102 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2023-05-22T15:23:28.670 INFO:teuthology.orchestra.run.smithi016.stdout:Get:106 https://4.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-05-21-0740-reef/9c9ba4df22024d9d8d6cf9f98a4b96751fc78d0c/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 18.0.0-3192-g9c9ba4df-1jammy [8,615 kB] 2023-05-22T15:23:28.969 INFO:teuthology.orchestra.run.smithi062.stdout:Get:103 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2023-05-22T15:23:29.059 INFO:teuthology.orchestra.run.smithi062.stdout:Get:104 https://4.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-05-21-0740-reef/9c9ba4df22024d9d8d6cf9f98a4b96751fc78d0c/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 18.0.0-3192-g9c9ba4df-1jammy [3,722 B] 2023-05-22T15:23:29.060 INFO:teuthology.orchestra.run.smithi062.stdout:Get:105 https://4.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-05-21-0740-reef/9c9ba4df22024d9d8d6cf9f98a4b96751fc78d0c/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 18.0.0-3192-g9c9ba4df-1jammy [46.2 MB] 2023-05-22T15:23:29.190 INFO:teuthology.orchestra.run.smithi062.stdout:Get:106 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2023-05-22T15:23:29.193 INFO:teuthology.orchestra.run.smithi062.stdout:Get:107 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2023-05-22T15:23:29.202 INFO:teuthology.orchestra.run.smithi062.stdout:Get:108 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2023-05-22T15:23:29.320 INFO:teuthology.orchestra.run.smithi016.stdout:Get:107 https://4.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-05-21-0740-reef/9c9ba4df22024d9d8d6cf9f98a4b96751fc78d0c/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 18.0.0-3192-g9c9ba4df-1jammy [3,722 B] 2023-05-22T15:23:29.320 INFO:teuthology.orchestra.run.smithi016.stdout:Get:108 https://4.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-05-21-0740-reef/9c9ba4df22024d9d8d6cf9f98a4b96751fc78d0c/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 18.0.0-3192-g9c9ba4df-1jammy [46.2 MB] 2023-05-22T15:23:31.965 INFO:teuthology.orchestra.run.smithi062.stdout:Get:109 https://4.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-05-21-0740-reef/9c9ba4df22024d9d8d6cf9f98a4b96751fc78d0c/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 18.0.0-3192-g9c9ba4df-1jammy [237 kB] 2023-05-22T15:23:31.969 INFO:teuthology.orchestra.run.smithi062.stdout:Get:110 https://4.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-05-21-0740-reef/9c9ba4df22024d9d8d6cf9f98a4b96751fc78d0c/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-mirror amd64 18.0.0-3192-g9c9ba4df-1jammy [213 kB] 2023-05-22T15:23:31.973 INFO:teuthology.orchestra.run.smithi062.stdout:Get:111 https://4.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-05-21-0740-reef/9c9ba4df22024d9d8d6cf9f98a4b96751fc78d0c/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-shell all 18.0.0-3192-g9c9ba4df-1jammy [23.8 kB] 2023-05-22T15:23:32.032 INFO:teuthology.orchestra.run.smithi062.stdout:Get:112 https://4.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-05-21-0740-reef/9c9ba4df22024d9d8d6cf9f98a4b96751fc78d0c/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-top all 18.0.0-3192-g9c9ba4df-1jammy [13.2 kB] 2023-05-22T15:23:32.033 INFO:teuthology.orchestra.run.smithi062.stdout:Get:113 https://4.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-05-21-0740-reef/9c9ba4df22024d9d8d6cf9f98a4b96751fc78d0c/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 18.0.0-3192-g9c9ba4df-1jammy [27.7 kB] 2023-05-22T15:23:32.034 INFO:teuthology.orchestra.run.smithi062.stdout:Get:114 https://4.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-05-21-0740-reef/9c9ba4df22024d9d8d6cf9f98a4b96751fc78d0c/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 18.0.0-3192-g9c9ba4df-1jammy [11.5 MB] 2023-05-22T15:23:32.553 INFO:teuthology.orchestra.run.smithi062.stdout:Get:115 https://4.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-05-21-0740-reef/9c9ba4df22024d9d8d6cf9f98a4b96751fc78d0c/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 18.0.0-3192-g9c9ba4df-1jammy [81.9 kB] 2023-05-22T15:23:32.771 INFO:teuthology.orchestra.run.smithi016.stdout:Get:109 https://4.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-05-21-0740-reef/9c9ba4df22024d9d8d6cf9f98a4b96751fc78d0c/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 18.0.0-3192-g9c9ba4df-1jammy [237 kB] 2023-05-22T15:23:32.775 INFO:teuthology.orchestra.run.smithi016.stdout:Get:110 https://4.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-05-21-0740-reef/9c9ba4df22024d9d8d6cf9f98a4b96751fc78d0c/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-mirror amd64 18.0.0-3192-g9c9ba4df-1jammy [213 kB] 2023-05-22T15:23:32.780 INFO:teuthology.orchestra.run.smithi016.stdout:Get:111 https://4.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-05-21-0740-reef/9c9ba4df22024d9d8d6cf9f98a4b96751fc78d0c/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-shell all 18.0.0-3192-g9c9ba4df-1jammy [23.8 kB] 2023-05-22T15:23:32.841 INFO:teuthology.orchestra.run.smithi016.stdout:Get:112 https://4.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-05-21-0740-reef/9c9ba4df22024d9d8d6cf9f98a4b96751fc78d0c/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-top all 18.0.0-3192-g9c9ba4df-1jammy [13.2 kB] 2023-05-22T15:23:32.841 INFO:teuthology.orchestra.run.smithi016.stdout:Get:113 https://4.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-05-21-0740-reef/9c9ba4df22024d9d8d6cf9f98a4b96751fc78d0c/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 18.0.0-3192-g9c9ba4df-1jammy [27.7 kB] 2023-05-22T15:23:32.842 INFO:teuthology.orchestra.run.smithi016.stdout:Get:114 https://4.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-05-21-0740-reef/9c9ba4df22024d9d8d6cf9f98a4b96751fc78d0c/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 18.0.0-3192-g9c9ba4df-1jammy [11.5 MB] 2023-05-22T15:23:33.106 INFO:teuthology.orchestra.run.smithi016.stdout:Get:115 https://4.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-05-21-0740-reef/9c9ba4df22024d9d8d6cf9f98a4b96751fc78d0c/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 18.0.0-3192-g9c9ba4df-1jammy [81.9 kB] 2023-05-22T15:23:33.116 INFO:teuthology.orchestra.run.smithi062.stdout:Fetched 180 MB in 10s (17.7 MB/s) 2023-05-22T15:23:33.230 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-pyperclip. 2023-05-22T15:23:33.324 INFO:teuthology.orchestra.run.smithi062.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165999 files and directories currently installed.) 2023-05-22T15:23:33.330 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../000-python3-pyperclip_1.8.2-2_all.deb ... 2023-05-22T15:23:33.370 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-pyperclip (1.8.2-2) ... 2023-05-22T15:23:33.632 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-wcwidth. 2023-05-22T15:23:33.658 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../001-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2023-05-22T15:23:33.670 INFO:teuthology.orchestra.run.smithi016.stdout:Fetched 180 MB in 11s (16.8 MB/s) 2023-05-22T15:23:33.698 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2023-05-22T15:23:33.817 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-pyperclip. 2023-05-22T15:23:33.911 INFO:teuthology.orchestra.run.smithi016.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165999 files and directories currently installed.) 2023-05-22T15:23:33.917 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../000-python3-pyperclip_1.8.2-2_all.deb ... 2023-05-22T15:23:33.957 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-pyperclip (1.8.2-2) ... 2023-05-22T15:23:34.010 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-cmd2. 2023-05-22T15:23:34.036 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../002-python3-cmd2_2.3.3-1_all.deb ... 2023-05-22T15:23:34.076 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-cmd2 (2.3.3-1) ... 2023-05-22T15:23:34.260 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-wcwidth. 2023-05-22T15:23:34.287 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../001-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2023-05-22T15:23:34.326 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2023-05-22T15:23:34.421 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package libfmt8:amd64. 2023-05-22T15:23:34.447 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../003-libfmt8_8.1.1+ds1-2_amd64.deb ... 2023-05-22T15:23:34.529 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking libfmt8:amd64 (8.1.1+ds1-2) ... 2023-05-22T15:23:34.824 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2023-05-22T15:23:34.830 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-cmd2. 2023-05-22T15:23:34.851 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../004-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2023-05-22T15:23:34.856 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../002-python3-cmd2_2.3.3-1_all.deb ... 2023-05-22T15:23:34.882 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2023-05-22T15:23:34.896 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-cmd2 (2.3.3-1) ... 2023-05-22T15:23:35.169 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2023-05-22T15:23:35.185 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../005-libdouble-conversion3_3.1.7-4_amd64.deb ... 2023-05-22T15:23:35.199 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package libfmt8:amd64. 2023-05-22T15:23:35.215 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../003-libfmt8_8.1.1+ds1-2_amd64.deb ... 2023-05-22T15:23:35.219 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2023-05-22T15:23:35.291 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking libfmt8:amd64 (8.1.1+ds1-2) ... 2023-05-22T15:23:35.547 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2023-05-22T15:23:35.574 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../006-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2023-05-22T15:23:35.604 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2023-05-22T15:23:35.614 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2023-05-22T15:23:35.631 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../004-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2023-05-22T15:23:35.671 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2023-05-22T15:23:35.999 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2023-05-22T15:23:36.026 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../005-libdouble-conversion3_3.1.7-4_amd64.deb ... 2023-05-22T15:23:36.066 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2023-05-22T15:23:36.226 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package libqt5core5a:amd64. 2023-05-22T15:23:36.253 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../007-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2023-05-22T15:23:36.301 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-05-22T15:23:36.344 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2023-05-22T15:23:36.371 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../006-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2023-05-22T15:23:36.402 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2023-05-22T15:23:36.645 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2023-05-22T15:23:36.661 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../008-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2023-05-22T15:23:36.704 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-05-22T15:23:36.839 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package libqt5core5a:amd64. 2023-05-22T15:23:36.866 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../007-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2023-05-22T15:23:36.914 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-05-22T15:23:37.057 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package libqt5network5:amd64. 2023-05-22T15:23:37.084 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../009-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2023-05-22T15:23:37.124 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-05-22T15:23:37.343 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2023-05-22T15:23:37.369 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../008-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2023-05-22T15:23:37.409 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-05-22T15:23:37.485 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2023-05-22T15:23:37.512 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../010-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2023-05-22T15:23:37.552 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2023-05-22T15:23:37.696 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package libqt5network5:amd64. 2023-05-22T15:23:37.723 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../009-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2023-05-22T15:23:37.754 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-05-22T15:23:37.796 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package librados2. 2023-05-22T15:23:37.812 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../011-librados2_18.0.0-3192-g9c9ba4df-1jammy_amd64.deb ... 2023-05-22T15:23:37.846 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking librados2 (18.0.0-3192-g9c9ba4df-1jammy) ... 2023-05-22T15:23:38.124 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2023-05-22T15:23:38.151 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../010-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2023-05-22T15:23:38.190 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2023-05-22T15:23:38.299 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package librbd1. 2023-05-22T15:23:38.326 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../012-librbd1_18.0.0-3192-g9c9ba4df-1jammy_amd64.deb ... 2023-05-22T15:23:38.366 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking librbd1 (18.0.0-3192-g9c9ba4df-1jammy) ... 2023-05-22T15:23:38.477 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package librados2. 2023-05-22T15:23:38.503 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../011-librados2_18.0.0-3192-g9c9ba4df-1jammy_amd64.deb ... 2023-05-22T15:23:38.535 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking librados2 (18.0.0-3192-g9c9ba4df-1jammy) ... 2023-05-22T15:23:38.793 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package libcephfs2. 2023-05-22T15:23:38.821 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../013-libcephfs2_18.0.0-3192-g9c9ba4df-1jammy_amd64.deb ... 2023-05-22T15:23:38.860 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking libcephfs2 (18.0.0-3192-g9c9ba4df-1jammy) ... 2023-05-22T15:23:38.996 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package librbd1. 2023-05-22T15:23:39.023 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../012-librbd1_18.0.0-3192-g9c9ba4df-1jammy_amd64.deb ... 2023-05-22T15:23:39.063 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking librbd1 (18.0.0-3192-g9c9ba4df-1jammy) ... 2023-05-22T15:23:39.112 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-rados. 2023-05-22T15:23:39.128 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../014-python3-rados_18.0.0-3192-g9c9ba4df-1jammy_amd64.deb ... 2023-05-22T15:23:39.162 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-rados (18.0.0-3192-g9c9ba4df-1jammy) ... 2023-05-22T15:23:39.466 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-ceph-argparse. 2023-05-22T15:23:39.474 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package libcephfs2. 2023-05-22T15:23:39.493 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../015-python3-ceph-argparse_18.0.0-3192-g9c9ba4df-1jammy_all.deb ... 2023-05-22T15:23:39.501 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../013-libcephfs2_18.0.0-3192-g9c9ba4df-1jammy_amd64.deb ... 2023-05-22T15:23:39.532 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-ceph-argparse (18.0.0-3192-g9c9ba4df-1jammy) ... 2023-05-22T15:23:39.541 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking libcephfs2 (18.0.0-3192-g9c9ba4df-1jammy) ... 2023-05-22T15:23:39.785 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-cephfs. 2023-05-22T15:23:39.811 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../016-python3-cephfs_18.0.0-3192-g9c9ba4df-1jammy_amd64.deb ... 2023-05-22T15:23:39.851 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-cephfs (18.0.0-3192-g9c9ba4df-1jammy) ... 2023-05-22T15:23:39.869 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-rados. 2023-05-22T15:23:39.895 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../014-python3-rados_18.0.0-3192-g9c9ba4df-1jammy_amd64.deb ... 2023-05-22T15:23:39.935 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-rados (18.0.0-3192-g9c9ba4df-1jammy) ... 2023-05-22T15:23:40.179 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-ceph-common. 2023-05-22T15:23:40.206 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../017-python3-ceph-common_18.0.0-3192-g9c9ba4df-1jammy_all.deb ... 2023-05-22T15:23:40.230 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-ceph-argparse. 2023-05-22T15:23:40.245 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-ceph-common (18.0.0-3192-g9c9ba4df-1jammy) ... 2023-05-22T15:23:40.256 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../015-python3-ceph-argparse_18.0.0-3192-g9c9ba4df-1jammy_all.deb ... 2023-05-22T15:23:40.296 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-ceph-argparse (18.0.0-3192-g9c9ba4df-1jammy) ... 2023-05-22T15:23:40.532 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-prettytable. 2023-05-22T15:23:40.532 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-cephfs. 2023-05-22T15:23:40.548 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../018-python3-prettytable_2.5.0-2_all.deb ... 2023-05-22T15:23:40.548 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../016-python3-cephfs_18.0.0-3192-g9c9ba4df-1jammy_amd64.deb ... 2023-05-22T15:23:40.582 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2023-05-22T15:23:40.590 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-cephfs (18.0.0-3192-g9c9ba4df-1jammy) ... 2023-05-22T15:23:40.843 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-rbd. 2023-05-22T15:23:40.869 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../019-python3-rbd_18.0.0-3192-g9c9ba4df-1jammy_amd64.deb ... 2023-05-22T15:23:40.909 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-rbd (18.0.0-3192-g9c9ba4df-1jammy) ... 2023-05-22T15:23:40.926 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-ceph-common. 2023-05-22T15:23:40.953 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../017-python3-ceph-common_18.0.0-3192-g9c9ba4df-1jammy_all.deb ... 2023-05-22T15:23:40.993 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-ceph-common (18.0.0-3192-g9c9ba4df-1jammy) ... 2023-05-22T15:23:41.296 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-prettytable. 2023-05-22T15:23:41.323 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../018-python3-prettytable_2.5.0-2_all.deb ... 2023-05-22T15:23:41.355 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2023-05-22T15:23:41.360 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package librdkafka1:amd64. 2023-05-22T15:23:41.387 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../020-librdkafka1_1.8.0-1build1_amd64.deb ... 2023-05-22T15:23:41.571 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2023-05-22T15:23:41.599 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-rbd. 2023-05-22T15:23:41.614 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../019-python3-rbd_18.0.0-3192-g9c9ba4df-1jammy_amd64.deb ... 2023-05-22T15:23:41.657 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-rbd (18.0.0-3192-g9c9ba4df-1jammy) ... 2023-05-22T15:23:41.882 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package librgw2. 2023-05-22T15:23:41.898 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../021-librgw2_18.0.0-3192-g9c9ba4df-1jammy_amd64.deb ... 2023-05-22T15:23:41.932 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking librgw2 (18.0.0-3192-g9c9ba4df-1jammy) ... 2023-05-22T15:23:42.002 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package librdkafka1:amd64. 2023-05-22T15:23:42.029 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../020-librdkafka1_1.8.0-1build1_amd64.deb ... 2023-05-22T15:23:42.068 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2023-05-22T15:23:42.401 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-rgw. 2023-05-22T15:23:42.405 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package librgw2. 2023-05-22T15:23:42.417 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../022-python3-rgw_18.0.0-3192-g9c9ba4df-1jammy_amd64.deb ... 2023-05-22T15:23:42.432 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../021-librgw2_18.0.0-3192-g9c9ba4df-1jammy_amd64.deb ... 2023-05-22T15:23:42.451 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-rgw (18.0.0-3192-g9c9ba4df-1jammy) ... 2023-05-22T15:23:42.471 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking librgw2 (18.0.0-3192-g9c9ba4df-1jammy) ... 2023-05-22T15:23:42.763 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package liboath0:amd64. 2023-05-22T15:23:42.789 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../023-liboath0_2.6.7-3build1_amd64.deb ... 2023-05-22T15:23:42.829 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2023-05-22T15:23:42.957 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-rgw. 2023-05-22T15:23:42.973 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../022-python3-rgw_18.0.0-3192-g9c9ba4df-1jammy_amd64.deb ... 2023-05-22T15:23:43.007 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-rgw (18.0.0-3192-g9c9ba4df-1jammy) ... 2023-05-22T15:23:43.141 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package libradosstriper1. 2023-05-22T15:23:43.168 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../024-libradosstriper1_18.0.0-3192-g9c9ba4df-1jammy_amd64.deb ... 2023-05-22T15:23:43.207 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking libradosstriper1 (18.0.0-3192-g9c9ba4df-1jammy) ... 2023-05-22T15:23:43.335 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package liboath0:amd64. 2023-05-22T15:23:43.362 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../023-liboath0_2.6.7-3build1_amd64.deb ... 2023-05-22T15:23:43.402 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2023-05-22T15:23:43.586 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package ceph-common. 2023-05-22T15:23:43.612 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../025-ceph-common_18.0.0-3192-g9c9ba4df-1jammy_amd64.deb ... 2023-05-22T15:23:43.652 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking ceph-common (18.0.0-3192-g9c9ba4df-1jammy) ... 2023-05-22T15:23:43.721 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package libradosstriper1. 2023-05-22T15:23:43.748 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../024-libradosstriper1_18.0.0-3192-g9c9ba4df-1jammy_amd64.deb ... 2023-05-22T15:23:43.788 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking libradosstriper1 (18.0.0-3192-g9c9ba4df-1jammy) ... 2023-05-22T15:23:44.150 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package ceph-common. 2023-05-22T15:23:44.176 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../025-ceph-common_18.0.0-3192-g9c9ba4df-1jammy_amd64.deb ... 2023-05-22T15:23:44.208 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking ceph-common (18.0.0-3192-g9c9ba4df-1jammy) ... 2023-05-22T15:23:44.966 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package ceph-base. 2023-05-22T15:23:44.993 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../026-ceph-base_18.0.0-3192-g9c9ba4df-1jammy_amd64.deb ... 2023-05-22T15:23:45.042 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking ceph-base (18.0.0-3192-g9c9ba4df-1jammy) ... 2023-05-22T15:23:45.647 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package ceph-base. 2023-05-22T15:23:45.674 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../026-ceph-base_18.0.0-3192-g9c9ba4df-1jammy_amd64.deb ... 2023-05-22T15:23:45.724 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking ceph-base (18.0.0-3192-g9c9ba4df-1jammy) ... 2023-05-22T15:23:45.754 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-jaraco.functools. 2023-05-22T15:23:45.781 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../027-python3-jaraco.functools_3.4.0-2_all.deb ... 2023-05-22T15:23:45.820 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2023-05-22T15:23:46.115 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-cheroot. 2023-05-22T15:23:46.142 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../028-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2023-05-22T15:23:46.182 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2023-05-22T15:23:46.460 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-jaraco.functools. 2023-05-22T15:23:46.487 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../027-python3-jaraco.functools_3.4.0-2_all.deb ... 2023-05-22T15:23:46.526 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2023-05-22T15:23:46.668 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-jaraco.classes. 2023-05-22T15:23:46.695 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../029-python3-jaraco.classes_3.2.1-3_all.deb ... 2023-05-22T15:23:46.735 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2023-05-22T15:23:46.830 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-cheroot. 2023-05-22T15:23:46.856 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../028-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2023-05-22T15:23:46.896 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2023-05-22T15:23:47.012 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-jaraco.text. 2023-05-22T15:23:47.028 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../030-python3-jaraco.text_3.6.0-2_all.deb ... 2023-05-22T15:23:47.062 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2023-05-22T15:23:47.216 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-jaraco.classes. 2023-05-22T15:23:47.232 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../029-python3-jaraco.classes_3.2.1-3_all.deb ... 2023-05-22T15:23:47.266 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2023-05-22T15:23:47.340 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-jaraco.collections. 2023-05-22T15:23:47.367 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../031-python3-jaraco.collections_3.4.0-2_all.deb ... 2023-05-22T15:23:47.407 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2023-05-22T15:23:47.519 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-jaraco.text. 2023-05-22T15:23:47.546 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../030-python3-jaraco.text_3.6.0-2_all.deb ... 2023-05-22T15:23:47.585 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2023-05-22T15:23:47.693 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-tempora. 2023-05-22T15:23:47.720 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../032-python3-tempora_4.1.2-1_all.deb ... 2023-05-22T15:23:47.760 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-tempora (4.1.2-1) ... 2023-05-22T15:23:47.889 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-jaraco.collections. 2023-05-22T15:23:47.916 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../031-python3-jaraco.collections_3.4.0-2_all.deb ... 2023-05-22T15:23:47.955 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2023-05-22T15:23:48.063 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-portend. 2023-05-22T15:23:48.089 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../033-python3-portend_3.0.0-1_all.deb ... 2023-05-22T15:23:48.129 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-portend (3.0.0-1) ... 2023-05-22T15:23:48.258 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-tempora. 2023-05-22T15:23:48.285 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../032-python3-tempora_4.1.2-1_all.deb ... 2023-05-22T15:23:48.324 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-tempora (4.1.2-1) ... 2023-05-22T15:23:48.399 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-zc.lockfile. 2023-05-22T15:23:48.415 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../034-python3-zc.lockfile_2.0-1_all.deb ... 2023-05-22T15:23:48.449 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2023-05-22T15:23:48.619 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-portend. 2023-05-22T15:23:48.646 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../033-python3-portend_3.0.0-1_all.deb ... 2023-05-22T15:23:48.677 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-portend (3.0.0-1) ... 2023-05-22T15:23:48.768 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-cherrypy3. 2023-05-22T15:23:48.795 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../035-python3-cherrypy3_18.6.1-4_all.deb ... 2023-05-22T15:23:48.835 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2023-05-22T15:23:49.005 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-zc.lockfile. 2023-05-22T15:23:49.032 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../034-python3-zc.lockfile_2.0-1_all.deb ... 2023-05-22T15:23:49.072 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2023-05-22T15:23:49.155 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-natsort. 2023-05-22T15:23:49.171 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../036-python3-natsort_8.0.2-1_all.deb ... 2023-05-22T15:23:49.213 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-natsort (8.0.2-1) ... 2023-05-22T15:23:49.359 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-cherrypy3. 2023-05-22T15:23:49.386 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../035-python3-cherrypy3_18.6.1-4_all.deb ... 2023-05-22T15:23:49.425 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2023-05-22T15:23:49.525 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-logutils. 2023-05-22T15:23:49.552 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../037-python3-logutils_0.3.3-8_all.deb ... 2023-05-22T15:23:49.592 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-logutils (0.3.3-8) ... 2023-05-22T15:23:49.754 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-natsort. 2023-05-22T15:23:49.781 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../036-python3-natsort_8.0.2-1_all.deb ... 2023-05-22T15:23:49.820 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-natsort (8.0.2-1) ... 2023-05-22T15:23:49.911 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-markupsafe. 2023-05-22T15:23:49.938 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../038-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2023-05-22T15:23:49.978 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2023-05-22T15:23:50.115 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-logutils. 2023-05-22T15:23:50.142 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../037-python3-logutils_0.3.3-8_all.deb ... 2023-05-22T15:23:50.181 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-logutils (0.3.3-8) ... 2023-05-22T15:23:50.239 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-mako. 2023-05-22T15:23:50.266 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../039-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2023-05-22T15:23:50.297 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2023-05-22T15:23:50.484 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-markupsafe. 2023-05-22T15:23:50.511 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../038-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2023-05-22T15:23:50.551 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2023-05-22T15:23:50.609 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-simplegeneric. 2023-05-22T15:23:50.636 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../040-python3-simplegeneric_0.8.1-3_all.deb ... 2023-05-22T15:23:50.675 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2023-05-22T15:23:50.812 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-mako. 2023-05-22T15:23:50.839 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../039-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2023-05-22T15:23:50.879 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2023-05-22T15:23:50.995 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-singledispatch. 2023-05-22T15:23:51.019 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../041-python3-singledispatch_3.4.0.3-3_all.deb ... 2023-05-22T15:23:51.062 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2023-05-22T15:23:51.166 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-simplegeneric. 2023-05-22T15:23:51.193 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../040-python3-simplegeneric_0.8.1-3_all.deb ... 2023-05-22T15:23:51.232 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2023-05-22T15:23:51.314 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-webob. 2023-05-22T15:23:51.330 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../042-python3-webob_1%3a1.8.6-1.1_all.deb ... 2023-05-22T15:23:51.365 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-webob (1:1.8.6-1.1) ... 2023-05-22T15:23:51.535 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-singledispatch. 2023-05-22T15:23:51.562 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../041-python3-singledispatch_3.4.0.3-3_all.deb ... 2023-05-22T15:23:51.602 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2023-05-22T15:23:51.676 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-waitress. 2023-05-22T15:23:51.703 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../043-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2023-05-22T15:23:51.854 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2023-05-22T15:23:51.913 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-webob. 2023-05-22T15:23:51.940 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../042-python3-webob_1%3a1.8.6-1.1_all.deb ... 2023-05-22T15:23:51.980 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-webob (1:1.8.6-1.1) ... 2023-05-22T15:23:52.162 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-tempita. 2023-05-22T15:23:52.189 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../044-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2023-05-22T15:23:52.254 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2023-05-22T15:23:52.308 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-waitress. 2023-05-22T15:23:52.335 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../043-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2023-05-22T15:23:52.436 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2023-05-22T15:23:52.565 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-paste. 2023-05-22T15:23:52.592 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../045-python3-paste_3.5.0+dfsg1-1_all.deb ... 2023-05-22T15:23:52.632 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2023-05-22T15:23:52.744 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-tempita. 2023-05-22T15:23:52.771 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../044-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2023-05-22T15:23:52.811 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2023-05-22T15:23:52.943 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2023-05-22T15:23:52.959 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../046-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2023-05-22T15:23:52.993 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2023-05-22T15:23:53.089 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-paste. 2023-05-22T15:23:53.112 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../045-python3-paste_3.5.0+dfsg1-1_all.deb ... 2023-05-22T15:23:53.147 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2023-05-22T15:23:53.279 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-pastedeploy. 2023-05-22T15:23:53.306 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../047-python3-pastedeploy_2.1.1-1_all.deb ... 2023-05-22T15:23:53.346 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2023-05-22T15:23:53.475 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2023-05-22T15:23:53.491 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../046-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2023-05-22T15:23:53.534 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2023-05-22T15:23:53.649 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-webtest. 2023-05-22T15:23:53.676 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../048-python3-webtest_2.0.35-1_all.deb ... 2023-05-22T15:23:53.716 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-webtest (2.0.35-1) ... 2023-05-22T15:23:53.803 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-pastedeploy. 2023-05-22T15:23:53.830 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../047-python3-pastedeploy_2.1.1-1_all.deb ... 2023-05-22T15:23:53.870 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2023-05-22T15:23:54.027 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-pecan. 2023-05-22T15:23:54.054 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../049-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2023-05-22T15:23:54.094 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2023-05-22T15:23:54.140 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-webtest. 2023-05-22T15:23:54.167 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../048-python3-webtest_2.0.35-1_all.deb ... 2023-05-22T15:23:54.206 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-webtest (2.0.35-1) ... 2023-05-22T15:23:54.457 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-certifi. 2023-05-22T15:23:54.484 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../050-python3-certifi_2020.6.20-1_all.deb ... 2023-05-22T15:23:54.518 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-pecan. 2023-05-22T15:23:54.523 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2023-05-22T15:23:54.544 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../049-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2023-05-22T15:23:54.584 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2023-05-22T15:23:54.785 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-idna. 2023-05-22T15:23:54.797 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../051-python3-idna_3.3-1_all.deb ... 2023-05-22T15:23:54.834 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-idna (3.3-1) ... 2023-05-22T15:23:54.964 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-certifi. 2023-05-22T15:23:54.991 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../050-python3-certifi_2020.6.20-1_all.deb ... 2023-05-22T15:23:55.031 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2023-05-22T15:23:55.146 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-urllib3. 2023-05-22T15:23:55.173 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../052-python3-urllib3_1.26.5-1~exp1_all.deb ... 2023-05-22T15:23:55.212 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1) ... 2023-05-22T15:23:55.534 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-idna. 2023-05-22T15:23:55.541 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-requests. 2023-05-22T15:23:55.561 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../051-python3-idna_3.3-1_all.deb ... 2023-05-22T15:23:55.568 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../053-python3-requests_2.25.1+dfsg-2_all.deb ... 2023-05-22T15:23:55.600 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-idna (3.3-1) ... 2023-05-22T15:23:55.607 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-requests (2.25.1+dfsg-2) ... 2023-05-22T15:23:55.870 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-urllib3. 2023-05-22T15:23:55.886 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../052-python3-urllib3_1.26.5-1~exp1_all.deb ... 2023-05-22T15:23:55.920 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1) ... 2023-05-22T15:23:55.935 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-werkzeug. 2023-05-22T15:23:55.962 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../054-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.1_all.deb ... 2023-05-22T15:23:56.002 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2023-05-22T15:23:56.982 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-requests. 2023-05-22T15:23:56.999 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../053-python3-requests_2.25.1+dfsg-2_all.deb ... 2023-05-22T15:23:57.032 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-requests (2.25.1+dfsg-2) ... 2023-05-22T15:23:57.063 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2023-05-22T15:23:57.079 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../055-ceph-mgr-modules-core_18.0.0-3192-g9c9ba4df-1jammy_all.deb ... 2023-05-22T15:23:57.121 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking ceph-mgr-modules-core (18.0.0-3192-g9c9ba4df-1jammy) ... 2023-05-22T15:23:57.310 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-werkzeug. 2023-05-22T15:23:57.337 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../054-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.1_all.deb ... 2023-05-22T15:23:57.368 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2023-05-22T15:23:57.493 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2023-05-22T15:23:57.521 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../056-libsqlite3-mod-ceph_18.0.0-3192-g9c9ba4df-1jammy_amd64.deb ... 2023-05-22T15:23:57.559 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking libsqlite3-mod-ceph (18.0.0-3192-g9c9ba4df-1jammy) ... 2023-05-22T15:23:57.671 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2023-05-22T15:23:57.687 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../055-ceph-mgr-modules-core_18.0.0-3192-g9c9ba4df-1jammy_all.deb ... 2023-05-22T15:23:57.729 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking ceph-mgr-modules-core (18.0.0-3192-g9c9ba4df-1jammy) ... 2023-05-22T15:23:57.871 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-openssl. 2023-05-22T15:23:57.898 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../057-python3-openssl_21.0.0-1_all.deb ... 2023-05-22T15:23:57.937 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-openssl (21.0.0-1) ... 2023-05-22T15:23:58.143 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2023-05-22T15:23:58.171 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../056-libsqlite3-mod-ceph_18.0.0-3192-g9c9ba4df-1jammy_amd64.deb ... 2023-05-22T15:23:58.210 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking libsqlite3-mod-ceph (18.0.0-3192-g9c9ba4df-1jammy) ... 2023-05-22T15:23:58.240 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package ceph-mgr. 2023-05-22T15:23:58.268 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../058-ceph-mgr_18.0.0-3192-g9c9ba4df-1jammy_amd64.deb ... 2023-05-22T15:23:58.307 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking ceph-mgr (18.0.0-3192-g9c9ba4df-1jammy) ... 2023-05-22T15:23:58.521 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-openssl. 2023-05-22T15:23:58.548 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../057-python3-openssl_21.0.0-1_all.deb ... 2023-05-22T15:23:58.587 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-openssl (21.0.0-1) ... 2023-05-22T15:23:58.677 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package ceph-mon. 2023-05-22T15:23:58.704 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../059-ceph-mon_18.0.0-3192-g9c9ba4df-1jammy_amd64.deb ... 2023-05-22T15:23:58.743 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking ceph-mon (18.0.0-3192-g9c9ba4df-1jammy) ... 2023-05-22T15:23:59.007 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package ceph-mgr. 2023-05-22T15:23:59.035 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../058-ceph-mgr_18.0.0-3192-g9c9ba4df-1jammy_amd64.deb ... 2023-05-22T15:23:59.065 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking ceph-mgr (18.0.0-3192-g9c9ba4df-1jammy) ... 2023-05-22T15:23:59.427 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package ceph-mon. 2023-05-22T15:23:59.454 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../059-ceph-mon_18.0.0-3192-g9c9ba4df-1jammy_amd64.deb ... 2023-05-22T15:23:59.494 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking ceph-mon (18.0.0-3192-g9c9ba4df-1jammy) ... 2023-05-22T15:23:59.689 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package ceph-osd. 2023-05-22T15:23:59.716 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../060-ceph-osd_18.0.0-3192-g9c9ba4df-1jammy_amd64.deb ... 2023-05-22T15:23:59.755 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking ceph-osd (18.0.0-3192-g9c9ba4df-1jammy) ... 2023-05-22T15:24:00.456 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package ceph-osd. 2023-05-22T15:24:00.484 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../060-ceph-osd_18.0.0-3192-g9c9ba4df-1jammy_amd64.deb ... 2023-05-22T15:24:00.522 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking ceph-osd (18.0.0-3192-g9c9ba4df-1jammy) ... 2023-05-22T15:24:01.635 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package ceph. 2023-05-22T15:24:01.652 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../061-ceph_18.0.0-3192-g9c9ba4df-1jammy_amd64.deb ... 2023-05-22T15:24:01.685 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking ceph (18.0.0-3192-g9c9ba4df-1jammy) ... 2023-05-22T15:24:01.921 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package ceph-fuse. 2023-05-22T15:24:01.948 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../062-ceph-fuse_18.0.0-3192-g9c9ba4df-1jammy_amd64.deb ... 2023-05-22T15:24:01.988 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking ceph-fuse (18.0.0-3192-g9c9ba4df-1jammy) ... 2023-05-22T15:24:02.366 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package ceph-mds. 2023-05-22T15:24:02.393 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../063-ceph-mds_18.0.0-3192-g9c9ba4df-1jammy_amd64.deb ... 2023-05-22T15:24:02.432 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking ceph-mds (18.0.0-3192-g9c9ba4df-1jammy) ... 2023-05-22T15:24:02.451 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package ceph. 2023-05-22T15:24:02.467 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../061-ceph_18.0.0-3192-g9c9ba4df-1jammy_amd64.deb ... 2023-05-22T15:24:02.501 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking ceph (18.0.0-3192-g9c9ba4df-1jammy) ... 2023-05-22T15:24:02.812 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package ceph-fuse. 2023-05-22T15:24:02.819 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package cephadm. 2023-05-22T15:24:02.836 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../062-ceph-fuse_18.0.0-3192-g9c9ba4df-1jammy_amd64.deb ... 2023-05-22T15:24:02.846 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../064-cephadm_18.0.0-3192-g9c9ba4df-1jammy_amd64.deb ... 2023-05-22T15:24:02.878 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking ceph-fuse (18.0.0-3192-g9c9ba4df-1jammy) ... 2023-05-22T15:24:02.885 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking cephadm (18.0.0-3192-g9c9ba4df-1jammy) ... 2023-05-22T15:24:03.163 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-asyncssh. 2023-05-22T15:24:03.179 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../065-python3-asyncssh_2.5.0-1_all.deb ... 2023-05-22T15:24:03.213 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2023-05-22T15:24:03.266 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package ceph-mds. 2023-05-22T15:24:03.293 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../063-ceph-mds_18.0.0-3192-g9c9ba4df-1jammy_amd64.deb ... 2023-05-22T15:24:03.332 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking ceph-mds (18.0.0-3192-g9c9ba4df-1jammy) ... 2023-05-22T15:24:03.507 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python-babel-localedata. 2023-05-22T15:24:03.523 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../066-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2023-05-22T15:24:03.557 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2023-05-22T15:24:03.677 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package cephadm. 2023-05-22T15:24:03.704 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../064-cephadm_18.0.0-3192-g9c9ba4df-1jammy_amd64.deb ... 2023-05-22T15:24:03.735 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking cephadm (18.0.0-3192-g9c9ba4df-1jammy) ... 2023-05-22T15:24:04.038 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-asyncssh. 2023-05-22T15:24:04.054 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../065-python3-asyncssh_2.5.0-1_all.deb ... 2023-05-22T15:24:04.088 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2023-05-22T15:24:04.416 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python-babel-localedata. 2023-05-22T15:24:04.432 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../066-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2023-05-22T15:24:04.474 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2023-05-22T15:24:04.562 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-babel. 2023-05-22T15:24:04.590 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../067-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2023-05-22T15:24:04.629 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2023-05-22T15:24:04.965 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-jinja2. 2023-05-22T15:24:04.981 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../068-python3-jinja2_3.0.3-1_all.deb ... 2023-05-22T15:24:05.023 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-jinja2 (3.0.3-1) ... 2023-05-22T15:24:05.377 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2023-05-22T15:24:05.404 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../069-ceph-mgr-cephadm_18.0.0-3192-g9c9ba4df-1jammy_all.deb ... 2023-05-22T15:24:05.443 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking ceph-mgr-cephadm (18.0.0-3192-g9c9ba4df-1jammy) ... 2023-05-22T15:24:05.471 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-babel. 2023-05-22T15:24:05.499 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../067-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2023-05-22T15:24:05.529 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2023-05-22T15:24:05.780 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-bcrypt. 2023-05-22T15:24:05.799 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-jinja2. 2023-05-22T15:24:05.807 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../070-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2023-05-22T15:24:05.827 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../068-python3-jinja2_3.0.3-1_all.deb ... 2023-05-22T15:24:05.846 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2023-05-22T15:24:05.857 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-jinja2 (3.0.3-1) ... 2023-05-22T15:24:06.149 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-pluggy. 2023-05-22T15:24:06.169 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2023-05-22T15:24:06.176 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../071-python3-pluggy_0.13.0-7.1_all.deb ... 2023-05-22T15:24:06.196 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../069-ceph-mgr-cephadm_18.0.0-3192-g9c9ba4df-1jammy_all.deb ... 2023-05-22T15:24:06.215 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2023-05-22T15:24:06.236 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking ceph-mgr-cephadm (18.0.0-3192-g9c9ba4df-1jammy) ... 2023-05-22T15:24:06.510 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-py. 2023-05-22T15:24:06.537 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../072-python3-py_1.10.0-1_all.deb ... 2023-05-22T15:24:06.565 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-bcrypt. 2023-05-22T15:24:06.592 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../070-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2023-05-22T15:24:06.601 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-py (1.10.0-1) ... 2023-05-22T15:24:06.631 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2023-05-22T15:24:06.888 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-iniconfig. 2023-05-22T15:24:06.915 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../073-python3-iniconfig_1.1.1-2_all.deb ... 2023-05-22T15:24:06.934 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-pluggy. 2023-05-22T15:24:06.946 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2023-05-22T15:24:06.962 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../071-python3-pluggy_0.13.0-7.1_all.deb ... 2023-05-22T15:24:07.009 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2023-05-22T15:24:07.321 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-py. 2023-05-22T15:24:07.348 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../072-python3-py_1.10.0-1_all.deb ... 2023-05-22T15:24:07.387 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-py (1.10.0-1) ... 2023-05-22T15:24:07.458 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-toml. 2023-05-22T15:24:07.485 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../074-python3-toml_0.10.2-1_all.deb ... 2023-05-22T15:24:07.524 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-toml (0.10.2-1) ... 2023-05-22T15:24:07.716 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-iniconfig. 2023-05-22T15:24:07.743 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../073-python3-iniconfig_1.1.1-2_all.deb ... 2023-05-22T15:24:07.782 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2023-05-22T15:24:07.794 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-pytest. 2023-05-22T15:24:07.821 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../075-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2023-05-22T15:24:07.852 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2023-05-22T15:24:08.027 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-toml. 2023-05-22T15:24:08.054 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../074-python3-toml_0.10.2-1_all.deb ... 2023-05-22T15:24:08.094 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-toml (0.10.2-1) ... 2023-05-22T15:24:08.189 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-repoze.lru. 2023-05-22T15:24:08.216 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../076-python3-repoze.lru_0.7-2_all.deb ... 2023-05-22T15:24:08.255 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2023-05-22T15:24:08.397 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-pytest. 2023-05-22T15:24:08.424 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../075-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2023-05-22T15:24:08.463 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2023-05-22T15:24:08.525 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-routes. 2023-05-22T15:24:08.552 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../077-python3-routes_2.5.1-1ubuntu1_all.deb ... 2023-05-22T15:24:08.583 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2023-05-22T15:24:08.750 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-repoze.lru. 2023-05-22T15:24:08.777 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../076-python3-repoze.lru_0.7-2_all.deb ... 2023-05-22T15:24:08.817 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2023-05-22T15:24:08.911 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2023-05-22T15:24:08.939 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../078-ceph-mgr-dashboard_18.0.0-3192-g9c9ba4df-1jammy_all.deb ... 2023-05-22T15:24:08.977 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking ceph-mgr-dashboard (18.0.0-3192-g9c9ba4df-1jammy) ... 2023-05-22T15:24:09.103 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-routes. 2023-05-22T15:24:09.130 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../077-python3-routes_2.5.1-1ubuntu1_all.deb ... 2023-05-22T15:24:09.170 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2023-05-22T15:24:09.356 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2023-05-22T15:24:09.372 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../079-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2023-05-22T15:24:09.406 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2023-05-22T15:24:09.499 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2023-05-22T15:24:09.526 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../078-ceph-mgr-dashboard_18.0.0-3192-g9c9ba4df-1jammy_all.deb ... 2023-05-22T15:24:09.565 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking ceph-mgr-dashboard (18.0.0-3192-g9c9ba4df-1jammy) ... 2023-05-22T15:24:09.867 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-joblib. 2023-05-22T15:24:09.895 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../080-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2023-05-22T15:24:09.927 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2023-05-22T15:24:09.934 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2023-05-22T15:24:09.943 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../079-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2023-05-22T15:24:09.976 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2023-05-22T15:24:10.337 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-threadpoolctl. 2023-05-22T15:24:10.365 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../081-python3-threadpoolctl_3.1.0-1_all.deb ... 2023-05-22T15:24:10.404 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2023-05-22T15:24:10.439 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-joblib. 2023-05-22T15:24:10.466 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../080-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2023-05-22T15:24:10.505 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2023-05-22T15:24:10.657 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-sklearn. 2023-05-22T15:24:10.684 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../082-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2023-05-22T15:24:10.723 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2023-05-22T15:24:11.059 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-threadpoolctl. 2023-05-22T15:24:11.086 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../081-python3-threadpoolctl_3.1.0-1_all.deb ... 2023-05-22T15:24:11.125 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2023-05-22T15:24:11.420 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-sklearn. 2023-05-22T15:24:11.447 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../082-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2023-05-22T15:24:11.478 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2023-05-22T15:24:11.500 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2023-05-22T15:24:11.528 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../083-ceph-mgr-diskprediction-local_18.0.0-3192-g9c9ba4df-1jammy_all.deb ... 2023-05-22T15:24:11.558 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking ceph-mgr-diskprediction-local (18.0.0-3192-g9c9ba4df-1jammy) ... 2023-05-22T15:24:12.306 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2023-05-22T15:24:12.334 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../083-ceph-mgr-diskprediction-local_18.0.0-3192-g9c9ba4df-1jammy_all.deb ... 2023-05-22T15:24:12.372 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking ceph-mgr-diskprediction-local (18.0.0-3192-g9c9ba4df-1jammy) ... 2023-05-22T15:24:12.478 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-cachetools. 2023-05-22T15:24:12.505 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../084-python3-cachetools_5.0.0-1_all.deb ... 2023-05-22T15:24:12.544 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2023-05-22T15:24:12.847 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-pyasn1. 2023-05-22T15:24:12.875 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../085-python3-pyasn1_0.4.8-1_all.deb ... 2023-05-22T15:24:12.913 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2023-05-22T15:24:13.200 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-pyasn1-modules. 2023-05-22T15:24:13.228 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../086-python3-pyasn1-modules_0.2.1-1_all.deb ... 2023-05-22T15:24:13.258 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2023-05-22T15:24:13.326 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-cachetools. 2023-05-22T15:24:13.353 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../084-python3-cachetools_5.0.0-1_all.deb ... 2023-05-22T15:24:13.392 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2023-05-22T15:24:13.545 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-rsa. 2023-05-22T15:24:13.572 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../087-python3-rsa_4.8-1_all.deb ... 2023-05-22T15:24:13.611 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-rsa (4.8-1) ... 2023-05-22T15:24:13.670 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-pyasn1. 2023-05-22T15:24:13.698 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../085-python3-pyasn1_0.4.8-1_all.deb ... 2023-05-22T15:24:13.728 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2023-05-22T15:24:13.956 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-google-auth. 2023-05-22T15:24:13.984 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../088-python3-google-auth_1.5.1-3_all.deb ... 2023-05-22T15:24:14.023 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2023-05-22T15:24:14.059 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-pyasn1-modules. 2023-05-22T15:24:14.087 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../086-python3-pyasn1-modules_0.2.1-1_all.deb ... 2023-05-22T15:24:14.125 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2023-05-22T15:24:14.343 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-requests-oauthlib. 2023-05-22T15:24:14.370 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../089-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2023-05-22T15:24:14.409 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2023-05-22T15:24:14.437 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-rsa. 2023-05-22T15:24:14.464 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../087-python3-rsa_4.8-1_all.deb ... 2023-05-22T15:24:14.503 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-rsa (4.8-1) ... 2023-05-22T15:24:14.729 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-websocket. 2023-05-22T15:24:14.756 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../090-python3-websocket_1.2.3-1_all.deb ... 2023-05-22T15:24:14.795 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-websocket (1.2.3-1) ... 2023-05-22T15:24:14.824 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-google-auth. 2023-05-22T15:24:14.851 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../088-python3-google-auth_1.5.1-3_all.deb ... 2023-05-22T15:24:14.890 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2023-05-22T15:24:15.073 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-kubernetes. 2023-05-22T15:24:15.101 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../091-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2023-05-22T15:24:15.131 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2023-05-22T15:24:15.218 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-requests-oauthlib. 2023-05-22T15:24:15.246 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../089-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2023-05-22T15:24:15.285 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2023-05-22T15:24:15.580 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-websocket. 2023-05-22T15:24:15.611 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../090-python3-websocket_1.2.3-1_all.deb ... 2023-05-22T15:24:15.638 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-websocket (1.2.3-1) ... 2023-05-22T15:24:15.795 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2023-05-22T15:24:15.812 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../092-ceph-mgr-k8sevents_18.0.0-3192-g9c9ba4df-1jammy_all.deb ... 2023-05-22T15:24:15.853 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking ceph-mgr-k8sevents (18.0.0-3192-g9c9ba4df-1jammy) ... 2023-05-22T15:24:15.949 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-kubernetes. 2023-05-22T15:24:15.977 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../091-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2023-05-22T15:24:16.016 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2023-05-22T15:24:16.173 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package libonig5:amd64. 2023-05-22T15:24:16.201 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../093-libonig5_6.9.7.1-2build1_amd64.deb ... 2023-05-22T15:24:16.239 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2023-05-22T15:24:16.534 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package libjq1:amd64. 2023-05-22T15:24:16.561 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../094-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2023-05-22T15:24:16.600 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2023-05-22T15:24:16.813 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2023-05-22T15:24:16.828 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package jq. 2023-05-22T15:24:16.830 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../092-ceph-mgr-k8sevents_18.0.0-3192-g9c9ba4df-1jammy_all.deb ... 2023-05-22T15:24:16.853 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../095-jq_1.6-2.1ubuntu3_amd64.deb ... 2023-05-22T15:24:16.863 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking ceph-mgr-k8sevents (18.0.0-3192-g9c9ba4df-1jammy) ... 2023-05-22T15:24:16.894 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2023-05-22T15:24:17.147 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package socat. 2023-05-22T15:24:17.163 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../096-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2023-05-22T15:24:17.182 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package libonig5:amd64. 2023-05-22T15:24:17.205 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2023-05-22T15:24:17.210 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../093-libonig5_6.9.7.1-2build1_amd64.deb ... 2023-05-22T15:24:17.299 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2023-05-22T15:24:17.652 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package libjq1:amd64. 2023-05-22T15:24:17.680 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../094-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2023-05-22T15:24:17.719 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2023-05-22T15:24:17.892 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package xmlstarlet. 2023-05-22T15:24:17.908 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../097-xmlstarlet_1.6.1-2.1_amd64.deb ... 2023-05-22T15:24:17.941 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2023-05-22T15:24:17.963 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package jq. 2023-05-22T15:24:17.979 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../095-jq_1.6-2.1ubuntu3_amd64.deb ... 2023-05-22T15:24:18.013 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2023-05-22T15:24:18.232 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package socat. 2023-05-22T15:24:18.249 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../096-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2023-05-22T15:24:18.279 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package ceph-test. 2023-05-22T15:24:18.290 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2023-05-22T15:24:18.308 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../098-ceph-test_18.0.0-3192-g9c9ba4df-1jammy_amd64.deb ... 2023-05-22T15:24:18.345 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking ceph-test (18.0.0-3192-g9c9ba4df-1jammy) ... 2023-05-22T15:24:18.627 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package xmlstarlet. 2023-05-22T15:24:18.655 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../097-xmlstarlet_1.6.1-2.1_amd64.deb ... 2023-05-22T15:24:18.694 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2023-05-22T15:24:19.015 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package ceph-test. 2023-05-22T15:24:19.031 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../098-ceph-test_18.0.0-3192-g9c9ba4df-1jammy_amd64.deb ... 2023-05-22T15:24:19.073 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking ceph-test (18.0.0-3192-g9c9ba4df-1jammy) ... 2023-05-22T15:24:20.524 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package ceph-volume. 2023-05-22T15:24:20.548 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../099-ceph-volume_18.0.0-3192-g9c9ba4df-1jammy_all.deb ... 2023-05-22T15:24:20.590 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking ceph-volume (18.0.0-3192-g9c9ba4df-1jammy) ... 2023-05-22T15:24:20.937 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package cephfs-mirror. 2023-05-22T15:24:20.965 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../100-cephfs-mirror_18.0.0-3192-g9c9ba4df-1jammy_amd64.deb ... 2023-05-22T15:24:21.003 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking cephfs-mirror (18.0.0-3192-g9c9ba4df-1jammy) ... 2023-05-22T15:24:21.306 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-colorama. 2023-05-22T15:24:21.334 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../101-python3-colorama_0.4.4-1_all.deb ... 2023-05-22T15:24:21.372 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-colorama (0.4.4-1) ... 2023-05-22T15:24:21.493 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package ceph-volume. 2023-05-22T15:24:21.510 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../099-ceph-volume_18.0.0-3192-g9c9ba4df-1jammy_all.deb ... 2023-05-22T15:24:21.543 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking ceph-volume (18.0.0-3192-g9c9ba4df-1jammy) ... 2023-05-22T15:24:21.633 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package cephfs-shell. 2023-05-22T15:24:21.650 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../102-cephfs-shell_18.0.0-3192-g9c9ba4df-1jammy_all.deb ... 2023-05-22T15:24:21.692 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking cephfs-shell (18.0.0-3192-g9c9ba4df-1jammy) ... 2023-05-22T15:24:21.856 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package cephfs-mirror. 2023-05-22T15:24:21.884 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../100-cephfs-mirror_18.0.0-3192-g9c9ba4df-1jammy_amd64.deb ... 2023-05-22T15:24:21.914 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking cephfs-mirror (18.0.0-3192-g9c9ba4df-1jammy) ... 2023-05-22T15:24:21.936 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package cephfs-top. 2023-05-22T15:24:21.953 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../103-cephfs-top_18.0.0-3192-g9c9ba4df-1jammy_all.deb ... 2023-05-22T15:24:21.995 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking cephfs-top (18.0.0-3192-g9c9ba4df-1jammy) ... 2023-05-22T15:24:22.209 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-colorama. 2023-05-22T15:24:22.222 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package libcephfs-dev. 2023-05-22T15:24:22.237 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../101-python3-colorama_0.4.4-1_all.deb ... 2023-05-22T15:24:22.239 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../104-libcephfs-dev_18.0.0-3192-g9c9ba4df-1jammy_amd64.deb ... 2023-05-22T15:24:22.272 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking libcephfs-dev (18.0.0-3192-g9c9ba4df-1jammy) ... 2023-05-22T15:24:22.276 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-colorama (0.4.4-1) ... 2023-05-22T15:24:22.537 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package cephfs-shell. 2023-05-22T15:24:22.554 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../102-cephfs-shell_18.0.0-3192-g9c9ba4df-1jammy_all.deb ... 2023-05-22T15:24:22.595 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking cephfs-shell (18.0.0-3192-g9c9ba4df-1jammy) ... 2023-05-22T15:24:22.642 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package nvme-cli. 2023-05-22T15:24:22.669 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../105-nvme-cli_1.16-3ubuntu0.1_amd64.deb ... 2023-05-22T15:24:22.708 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking nvme-cli (1.16-3ubuntu0.1) ... 2023-05-22T15:24:22.839 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package cephfs-top. 2023-05-22T15:24:22.856 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../103-cephfs-top_18.0.0-3192-g9c9ba4df-1jammy_all.deb ... 2023-05-22T15:24:22.898 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking cephfs-top (18.0.0-3192-g9c9ba4df-1jammy) ... 2023-05-22T15:24:23.150 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package libcephfs-dev. 2023-05-22T15:24:23.178 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../104-libcephfs-dev_18.0.0-3192-g9c9ba4df-1jammy_amd64.deb ... 2023-05-22T15:24:23.188 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python-asyncssh-doc. 2023-05-22T15:24:23.213 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../106-python-asyncssh-doc_2.5.0-1_all.deb ... 2023-05-22T15:24:23.217 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking libcephfs-dev (18.0.0-3192-g9c9ba4df-1jammy) ... 2023-05-22T15:24:23.254 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2023-05-22T15:24:23.579 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package nvme-cli. 2023-05-22T15:24:23.606 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../105-nvme-cli_1.16-3ubuntu0.1_amd64.deb ... 2023-05-22T15:24:23.645 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking nvme-cli (1.16-3ubuntu0.1) ... 2023-05-22T15:24:23.682 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-pastescript. 2023-05-22T15:24:23.710 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../107-python3-pastescript_2.0.2-4_all.deb ... 2023-05-22T15:24:23.749 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2023-05-22T15:24:23.992 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python-asyncssh-doc. 2023-05-22T15:24:24.020 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../106-python-asyncssh-doc_2.5.0-1_all.deb ... 2023-05-22T15:24:24.058 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2023-05-22T15:24:24.077 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-psutil. 2023-05-22T15:24:24.105 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../108-python3-psutil_5.9.0-1build1_amd64.deb ... 2023-05-22T15:24:24.143 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2023-05-22T15:24:24.470 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-pastescript. 2023-05-22T15:24:24.496 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-pygments. 2023-05-22T15:24:24.497 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../107-python3-pastescript_2.0.2-4_all.deb ... 2023-05-22T15:24:24.524 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../109-python3-pygments_2.11.2+dfsg-2_all.deb ... 2023-05-22T15:24:24.536 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2023-05-22T15:24:24.563 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2023-05-22T15:24:24.864 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-psutil. 2023-05-22T15:24:24.892 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../108-python3-psutil_5.9.0-1build1_amd64.deb ... 2023-05-22T15:24:24.931 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2023-05-22T15:24:25.058 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-pyinotify. 2023-05-22T15:24:25.087 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../110-python3-pyinotify_0.9.6-1.3_all.deb ... 2023-05-22T15:24:25.125 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2023-05-22T15:24:25.267 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-pygments. 2023-05-22T15:24:25.295 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../109-python3-pygments_2.11.2+dfsg-2_all.deb ... 2023-05-22T15:24:25.333 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2023-05-22T15:24:25.403 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-simplejson. 2023-05-22T15:24:25.431 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../111-python3-simplejson_3.17.6-1build1_amd64.deb ... 2023-05-22T15:24:25.461 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2023-05-22T15:24:25.688 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package qttranslations5-l10n. 2023-05-22T15:24:25.705 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../112-qttranslations5-l10n_5.15.3-1_all.deb ... 2023-05-22T15:24:25.738 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2023-05-22T15:24:25.804 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-pyinotify. 2023-05-22T15:24:25.833 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../110-python3-pyinotify_0.9.6-1.3_all.deb ... 2023-05-22T15:24:25.871 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2023-05-22T15:24:26.132 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-simplejson. 2023-05-22T15:24:26.160 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../111-python3-simplejson_3.17.6-1build1_amd64.deb ... 2023-05-22T15:24:26.198 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2023-05-22T15:24:26.450 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package radosgw. 2023-05-22T15:24:26.451 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package qttranslations5-l10n. 2023-05-22T15:24:26.468 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../112-qttranslations5-l10n_5.15.3-1_all.deb ... 2023-05-22T15:24:26.479 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../113-radosgw_18.0.0-3192-g9c9ba4df-1jammy_amd64.deb ... 2023-05-22T15:24:26.509 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2023-05-22T15:24:26.517 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking radosgw (18.0.0-3192-g9c9ba4df-1jammy) ... 2023-05-22T15:24:27.262 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package rbd-fuse. 2023-05-22T15:24:27.289 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../114-rbd-fuse_18.0.0-3192-g9c9ba4df-1jammy_amd64.deb ... 2023-05-22T15:24:27.305 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package radosgw. 2023-05-22T15:24:27.320 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking rbd-fuse (18.0.0-3192-g9c9ba4df-1jammy) ... 2023-05-22T15:24:27.334 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../113-radosgw_18.0.0-3192-g9c9ba4df-1jammy_amd64.deb ... 2023-05-22T15:24:27.371 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking radosgw (18.0.0-3192-g9c9ba4df-1jammy) ... 2023-05-22T15:24:27.607 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2023-05-22T15:24:27.866 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2023-05-22T15:24:27.975 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up nvme-cli (1.16-3ubuntu0.1) ... 2023-05-22T15:24:28.091 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package rbd-fuse. 2023-05-22T15:24:28.108 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../114-rbd-fuse_18.0.0-3192-g9c9ba4df-1jammy_amd64.deb ... 2023-05-22T15:24:28.149 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking rbd-fuse (18.0.0-3192-g9c9ba4df-1jammy) ... 2023-05-22T15:24:28.259 INFO:teuthology.orchestra.run.smithi062.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2023-05-22T15:24:28.462 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2023-05-22T15:24:28.638 INFO:teuthology.orchestra.run.smithi062.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2023-05-22T15:24:28.779 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2023-05-22T15:24:28.905 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up nvme-cli (1.16-3ubuntu0.1) ... 2023-05-22T15:24:29.120 INFO:teuthology.orchestra.run.smithi062.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2023-05-22T15:24:29.177 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2023-05-22T15:24:29.218 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up cephadm (18.0.0-3192-g9c9ba4df-1jammy) ... 2023-05-22T15:24:29.538 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2023-05-22T15:24:29.745 INFO:teuthology.orchestra.run.smithi062.stdout:Adding system user cephadm....done 2023-05-22T15:24:29.899 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2023-05-22T15:24:29.993 INFO:teuthology.orchestra.run.smithi016.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2023-05-22T15:24:30.090 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up cephadm (18.0.0-3192-g9c9ba4df-1jammy) ... 2023-05-22T15:24:30.217 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2023-05-22T15:24:30.460 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2023-05-22T15:24:30.595 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2023-05-22T15:24:30.650 INFO:teuthology.orchestra.run.smithi016.stdout:Adding system user cephadm....done 2023-05-22T15:24:30.821 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2023-05-22T15:24:30.862 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-repoze.lru (0.7-2) ... 2023-05-22T15:24:31.089 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2023-05-22T15:24:31.122 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2023-05-22T15:24:31.248 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-py (1.10.0-1) ... 2023-05-22T15:24:31.332 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2023-05-22T15:24:31.458 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2023-05-22T15:24:31.566 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2023-05-22T15:24:31.726 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-repoze.lru (0.7-2) ... 2023-05-22T15:24:31.926 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-cachetools (5.0.0-1) ... 2023-05-22T15:24:32.119 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2023-05-22T15:24:32.169 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-colorama (0.4.4-1) ... 2023-05-22T15:24:32.253 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-py (1.10.0-1) ... 2023-05-22T15:24:32.420 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-openssl (21.0.0-1) ... 2023-05-22T15:24:32.563 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2023-05-22T15:24:32.704 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2023-05-22T15:24:32.923 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-cachetools (5.0.0-1) ... 2023-05-22T15:24:32.964 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2023-05-22T15:24:33.183 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-colorama (0.4.4-1) ... 2023-05-22T15:24:33.240 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2023-05-22T15:24:33.443 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-openssl (21.0.0-1) ... 2023-05-22T15:24:33.719 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2023-05-22T15:24:33.758 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-ceph-argparse (18.0.0-3192-g9c9ba4df-1jammy) ... 2023-05-22T15:24:33.970 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2023-05-22T15:24:34.025 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2023-05-22T15:24:34.126 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-pyperclip (1.8.2-2) ... 2023-05-22T15:24:34.221 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2023-05-22T15:24:34.386 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2023-05-22T15:24:34.472 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-ceph-argparse (18.0.0-3192-g9c9ba4df-1jammy) ... 2023-05-22T15:24:34.645 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2023-05-22T15:24:34.731 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2023-05-22T15:24:34.771 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2023-05-22T15:24:34.841 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-pyperclip (1.8.2-2) ... 2023-05-22T15:24:35.100 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2023-05-22T15:24:35.139 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2023-05-22T15:24:35.290 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-natsort (8.0.2-1) ... 2023-05-22T15:24:35.351 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2023-05-22T15:24:35.478 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2023-05-22T15:24:35.574 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2023-05-22T15:24:35.842 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2023-05-22T15:24:35.854 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2023-05-22T15:24:35.980 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-natsort (8.0.2-1) ... 2023-05-22T15:24:36.135 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2023-05-22T15:24:36.239 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2023-05-22T15:24:36.490 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2023-05-22T15:24:36.791 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2023-05-22T15:24:36.845 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2023-05-22T15:24:37.104 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2023-05-22T15:24:37.230 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2023-05-22T15:24:37.356 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2023-05-22T15:24:37.460 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2023-05-22T15:24:37.657 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2023-05-22T15:24:37.720 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2023-05-22T15:24:37.846 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2023-05-22T15:24:37.972 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2023-05-22T15:24:38.042 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-certifi (2020.6.20-1) ... 2023-05-22T15:24:38.264 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2023-05-22T15:24:38.276 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2023-05-22T15:24:38.641 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-certifi (2020.6.20-1) ... 2023-05-22T15:24:38.653 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2023-05-22T15:24:38.891 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2023-05-22T15:24:38.946 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-cmd2 (2.3.3-1) ... 2023-05-22T15:24:39.255 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2023-05-22T15:24:39.277 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2023-05-22T15:24:39.561 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-cmd2 (2.3.3-1) ... 2023-05-22T15:24:39.606 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2023-05-22T15:24:39.866 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2023-05-22T15:24:39.904 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2023-05-22T15:24:39.975 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-ceph-common (18.0.0-3192-g9c9ba4df-1jammy) ... 2023-05-22T15:24:40.251 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-idna (3.3-1) ... 2023-05-22T15:24:40.280 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2023-05-22T15:24:40.523 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2023-05-22T15:24:40.561 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2023-05-22T15:24:40.632 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-ceph-common (18.0.0-3192-g9c9ba4df-1jammy) ... 2023-05-22T15:24:40.933 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-idna (3.3-1) ... 2023-05-22T15:24:41.293 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2023-05-22T15:24:41.783 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-05-22T15:24:41.909 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-toml (0.10.2-1) ... 2023-05-22T15:24:42.143 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2023-05-22T15:24:42.269 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2023-05-22T15:24:42.416 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-urllib3 (1.26.5-1~exp1) ... 2023-05-22T15:24:42.499 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-05-22T15:24:42.633 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-toml (0.10.2-1) ... 2023-05-22T15:24:42.738 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2023-05-22T15:24:42.901 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2023-05-22T15:24:42.997 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2023-05-22T15:24:43.002 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2023-05-22T15:24:43.103 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-urllib3 (1.26.5-1~exp1) ... 2023-05-22T15:24:43.257 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-05-22T15:24:43.383 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up libfmt8:amd64 (8.1.1+ds1-2) ... 2023-05-22T15:24:43.405 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2023-05-22T15:24:43.492 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2023-05-22T15:24:43.656 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2023-05-22T15:24:43.785 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2023-05-22T15:24:43.898 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-05-22T15:24:44.016 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up libfmt8:amd64 (8.1.1+ds1-2) ... 2023-05-22T15:24:44.053 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-logutils (0.3.3-8) ... 2023-05-22T15:24:44.142 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2023-05-22T15:24:44.321 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-tempora (4.1.2-1) ... 2023-05-22T15:24:44.477 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2023-05-22T15:24:44.580 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2023-05-22T15:24:44.728 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-logutils (0.3.3-8) ... 2023-05-22T15:24:44.839 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-prettytable (2.5.0-2) ... 2023-05-22T15:24:45.004 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-tempora (4.1.2-1) ... 2023-05-22T15:24:45.115 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2023-05-22T15:24:45.241 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-websocket (1.2.3-1) ... 2023-05-22T15:24:45.255 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2023-05-22T15:24:45.490 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-prettytable (2.5.0-2) ... 2023-05-22T15:24:45.517 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2023-05-22T15:24:45.618 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2023-05-22T15:24:45.749 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2023-05-22T15:24:45.875 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-websocket (1.2.3-1) ... 2023-05-22T15:24:45.921 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-webob (1:1.8.6-1.1) ... 2023-05-22T15:24:46.193 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2023-05-22T15:24:46.256 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2023-05-22T15:24:46.319 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2023-05-22T15:24:46.515 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2023-05-22T15:24:46.620 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-webob (1:1.8.6-1.1) ... 2023-05-22T15:24:46.930 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2023-05-22T15:24:46.947 INFO:teuthology.orchestra.run.smithi062.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2023-05-22T15:24:47.181 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2023-05-22T15:24:47.201 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2023-05-22T15:24:47.311 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2023-05-22T15:24:47.564 INFO:teuthology.orchestra.run.smithi016.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2023-05-22T15:24:47.720 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2023-05-22T15:24:47.989 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-portend (3.0.0-1) ... 2023-05-22T15:24:48.000 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2023-05-22T15:24:48.135 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2023-05-22T15:24:48.272 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-05-22T15:24:48.398 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-jinja2 (3.0.3-1) ... 2023-05-22T15:24:48.553 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2023-05-22T15:24:48.732 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2023-05-22T15:24:48.804 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-portend (3.0.0-1) ... 2023-05-22T15:24:49.042 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-requests (2.25.1+dfsg-2) ... 2023-05-22T15:24:49.063 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-05-22T15:24:49.189 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-jinja2 (3.0.3-1) ... 2023-05-22T15:24:49.532 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2023-05-22T15:24:49.534 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2023-05-22T15:24:49.669 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-rsa (4.8-1) ... 2023-05-22T15:24:49.850 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-requests (2.25.1+dfsg-2) ... 2023-05-22T15:24:49.945 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-webtest (2.0.35-1) ... 2023-05-22T15:24:50.134 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2023-05-22T15:24:50.230 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2023-05-22T15:24:50.260 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-rsa (4.8-1) ... 2023-05-22T15:24:50.505 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2023-05-22T15:24:50.528 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-webtest (2.0.35-1) ... 2023-05-22T15:24:50.804 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2023-05-22T15:24:50.932 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-pastescript (2.0.2-4) ... 2023-05-22T15:24:51.089 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2023-05-22T15:24:51.225 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2023-05-22T15:24:51.473 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-pastescript (2.0.2-4) ... 2023-05-22T15:24:51.543 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2023-05-22T15:24:51.661 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up librados2 (18.0.0-3192-g9c9ba4df-1jammy) ... 2023-05-22T15:24:51.766 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2023-05-22T15:24:51.787 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up librgw2 (18.0.0-3192-g9c9ba4df-1jammy) ... 2023-05-22T15:24:51.998 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up libsqlite3-mod-ceph (18.0.0-3192-g9c9ba4df-1jammy) ... 2023-05-22T15:24:52.125 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up libcephfs2 (18.0.0-3192-g9c9ba4df-1jammy) ... 2023-05-22T15:24:52.143 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2023-05-22T15:24:52.234 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up libradosstriper1 (18.0.0-3192-g9c9ba4df-1jammy) ... 2023-05-22T15:24:52.269 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up librados2 (18.0.0-3192-g9c9ba4df-1jammy) ... 2023-05-22T15:24:52.343 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-google-auth (1.5.1-3) ... 2023-05-22T15:24:52.396 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up librgw2 (18.0.0-3192-g9c9ba4df-1jammy) ... 2023-05-22T15:24:52.620 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up librbd1 (18.0.0-3192-g9c9ba4df-1jammy) ... 2023-05-22T15:24:52.632 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up libsqlite3-mod-ceph (18.0.0-3192-g9c9ba4df-1jammy) ... 2023-05-22T15:24:52.746 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up ceph-mgr-modules-core (18.0.0-3192-g9c9ba4df-1jammy) ... 2023-05-22T15:24:52.750 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up libcephfs2 (18.0.0-3192-g9c9ba4df-1jammy) ... 2023-05-22T15:24:52.872 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up ceph-fuse (18.0.0-3192-g9c9ba4df-1jammy) ... 2023-05-22T15:24:52.876 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up libradosstriper1 (18.0.0-3192-g9c9ba4df-1jammy) ... 2023-05-22T15:24:53.010 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-google-auth (1.5.1-3) ... 2023-05-22T15:24:53.090 INFO:teuthology.orchestra.run.smithi062.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2023-05-22T15:24:53.090 INFO:teuthology.orchestra.run.smithi062.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2023-05-22T15:24:53.287 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up librbd1 (18.0.0-3192-g9c9ba4df-1jammy) ... 2023-05-22T15:24:53.413 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up ceph-mgr-modules-core (18.0.0-3192-g9c9ba4df-1jammy) ... 2023-05-22T15:24:53.539 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up ceph-fuse (18.0.0-3192-g9c9ba4df-1jammy) ... 2023-05-22T15:24:53.606 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up libcephfs-dev (18.0.0-3192-g9c9ba4df-1jammy) ... 2023-05-22T15:24:53.732 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-rados (18.0.0-3192-g9c9ba4df-1jammy) ... 2023-05-22T15:24:53.757 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2023-05-22T15:24:53.757 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2023-05-22T15:24:53.858 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up cephfs-top (18.0.0-3192-g9c9ba4df-1jammy) ... 2023-05-22T15:24:53.960 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2023-05-22T15:24:54.290 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up libcephfs-dev (18.0.0-3192-g9c9ba4df-1jammy) ... 2023-05-22T15:24:54.416 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-rados (18.0.0-3192-g9c9ba4df-1jammy) ... 2023-05-22T15:24:54.542 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up cephfs-top (18.0.0-3192-g9c9ba4df-1jammy) ... 2023-05-22T15:24:54.660 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2023-05-22T15:24:55.247 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-rbd (18.0.0-3192-g9c9ba4df-1jammy) ... 2023-05-22T15:24:55.373 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up rbd-fuse (18.0.0-3192-g9c9ba4df-1jammy) ... 2023-05-22T15:24:55.499 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-rgw (18.0.0-3192-g9c9ba4df-1jammy) ... 2023-05-22T15:24:55.608 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-cephfs (18.0.0-3192-g9c9ba4df-1jammy) ... 2023-05-22T15:24:55.709 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up cephfs-shell (18.0.0-3192-g9c9ba4df-1jammy) ... 2023-05-22T15:24:55.835 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up ceph-common (18.0.0-3192-g9c9ba4df-1jammy) ... 2023-05-22T15:24:55.890 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-rbd (18.0.0-3192-g9c9ba4df-1jammy) ... 2023-05-22T15:24:55.999 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up rbd-fuse (18.0.0-3192-g9c9ba4df-1jammy) ... 2023-05-22T15:24:56.125 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-rgw (18.0.0-3192-g9c9ba4df-1jammy) ... 2023-05-22T15:24:56.251 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-cephfs (18.0.0-3192-g9c9ba4df-1jammy) ... 2023-05-22T15:24:56.379 INFO:teuthology.orchestra.run.smithi062.stdout:Adding group ceph....done 2023-05-22T15:24:56.380 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up cephfs-shell (18.0.0-3192-g9c9ba4df-1jammy) ... 2023-05-22T15:24:56.512 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up ceph-common (18.0.0-3192-g9c9ba4df-1jammy) ... 2023-05-22T15:24:56.705 INFO:teuthology.orchestra.run.smithi062.stdout:Adding system user ceph....done 2023-05-22T15:24:56.898 INFO:teuthology.orchestra.run.smithi062.stdout:Setting system user ceph properties....done 2023-05-22T15:24:56.910 INFO:teuthology.orchestra.run.smithi062.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2023-05-22T15:24:57.021 INFO:teuthology.orchestra.run.smithi016.stdout:Adding group ceph....done 2023-05-22T15:24:57.054 INFO:teuthology.orchestra.run.smithi062.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2023-05-22T15:24:57.398 INFO:teuthology.orchestra.run.smithi016.stdout:Adding system user ceph....done 2023-05-22T15:24:57.425 INFO:teuthology.orchestra.run.smithi062.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2023-05-22T15:24:57.591 INFO:teuthology.orchestra.run.smithi016.stdout:Setting system user ceph properties....done 2023-05-22T15:24:57.604 INFO:teuthology.orchestra.run.smithi016.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2023-05-22T15:24:57.740 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2023-05-22T15:24:57.920 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up cephfs-mirror (18.0.0-3192-g9c9ba4df-1jammy) ... 2023-05-22T15:24:58.054 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up ceph-test (18.0.0-3192-g9c9ba4df-1jammy) ... 2023-05-22T15:24:58.092 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2023-05-22T15:24:58.180 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up radosgw (18.0.0-3192-g9c9ba4df-1jammy) ... 2023-05-22T15:24:58.565 INFO:teuthology.orchestra.run.smithi062.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2023-05-22T15:24:58.566 INFO:teuthology.orchestra.run.smithi062.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2023-05-22T15:24:58.613 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up cephfs-mirror (18.0.0-3192-g9c9ba4df-1jammy) ... 2023-05-22T15:24:58.739 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up ceph-test (18.0.0-3192-g9c9ba4df-1jammy) ... 2023-05-22T15:24:58.865 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up radosgw (18.0.0-3192-g9c9ba4df-1jammy) ... 2023-05-22T15:24:59.239 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2023-05-22T15:24:59.240 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2023-05-22T15:24:59.635 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up ceph-base (18.0.0-3192-g9c9ba4df-1jammy) ... 2023-05-22T15:24:59.975 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up ceph-base (18.0.0-3192-g9c9ba4df-1jammy) ... 2023-05-22T15:25:00.128 INFO:teuthology.orchestra.run.smithi062.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2023-05-22T15:25:00.332 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2023-05-22T15:25:00.677 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up ceph-mds (18.0.0-3192-g9c9ba4df-1jammy) ... 2023-05-22T15:25:00.869 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up ceph-mds (18.0.0-3192-g9c9ba4df-1jammy) ... 2023-05-22T15:25:00.904 INFO:teuthology.orchestra.run.smithi062.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2023-05-22T15:25:00.905 INFO:teuthology.orchestra.run.smithi062.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2023-05-22T15:25:01.100 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2023-05-22T15:25:01.101 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2023-05-22T15:25:01.412 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up ceph-mgr (18.0.0-3192-g9c9ba4df-1jammy) ... 2023-05-22T15:25:01.629 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up ceph-mgr (18.0.0-3192-g9c9ba4df-1jammy) ... 2023-05-22T15:25:01.636 INFO:teuthology.orchestra.run.smithi062.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2023-05-22T15:25:01.636 INFO:teuthology.orchestra.run.smithi062.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2023-05-22T15:25:01.858 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2023-05-22T15:25:01.858 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2023-05-22T15:25:02.155 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up ceph-osd (18.0.0-3192-g9c9ba4df-1jammy) ... 2023-05-22T15:25:02.389 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up ceph-osd (18.0.0-3192-g9c9ba4df-1jammy) ... 2023-05-22T15:25:02.490 INFO:teuthology.orchestra.run.smithi062.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2023-05-22T15:25:02.491 INFO:teuthology.orchestra.run.smithi062.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2023-05-22T15:25:02.720 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2023-05-22T15:25:02.720 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2023-05-22T15:25:03.032 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up ceph-mgr-k8sevents (18.0.0-3192-g9c9ba4df-1jammy) ... 2023-05-22T15:25:03.157 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up ceph-mgr-diskprediction-local (18.0.0-3192-g9c9ba4df-1jammy) ... 2023-05-22T15:25:03.224 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up ceph-mgr-k8sevents (18.0.0-3192-g9c9ba4df-1jammy) ... 2023-05-22T15:25:03.292 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up ceph-mon (18.0.0-3192-g9c9ba4df-1jammy) ... 2023-05-22T15:25:03.350 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up ceph-mgr-diskprediction-local (18.0.0-3192-g9c9ba4df-1jammy) ... 2023-05-22T15:25:03.492 INFO:teuthology.orchestra.run.smithi062.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2023-05-22T15:25:03.493 INFO:teuthology.orchestra.run.smithi062.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2023-05-22T15:25:03.501 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up ceph-mon (18.0.0-3192-g9c9ba4df-1jammy) ... 2023-05-22T15:25:03.723 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2023-05-22T15:25:03.724 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2023-05-22T15:25:04.001 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up ceph-mgr-cephadm (18.0.0-3192-g9c9ba4df-1jammy) ... 2023-05-22T15:25:04.136 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up ceph (18.0.0-3192-g9c9ba4df-1jammy) ... 2023-05-22T15:25:04.244 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up ceph-mgr-cephadm (18.0.0-3192-g9c9ba4df-1jammy) ... 2023-05-22T15:25:04.261 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up ceph-mgr-dashboard (18.0.0-3192-g9c9ba4df-1jammy) ... 2023-05-22T15:25:04.413 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up ceph-volume (18.0.0-3192-g9c9ba4df-1jammy) ... 2023-05-22T15:25:04.478 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up ceph (18.0.0-3192-g9c9ba4df-1jammy) ... 2023-05-22T15:25:04.540 INFO:teuthology.orchestra.run.smithi062.stdout:Processing triggers for man-db (2.10.2-1) ... 2023-05-22T15:25:04.604 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up ceph-mgr-dashboard (18.0.0-3192-g9c9ba4df-1jammy) ... 2023-05-22T15:25:04.755 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up ceph-volume (18.0.0-3192-g9c9ba4df-1jammy) ... 2023-05-22T15:25:04.882 INFO:teuthology.orchestra.run.smithi016.stdout:Processing triggers for man-db (2.10.2-1) ... 2023-05-22T15:25:05.978 INFO:teuthology.orchestra.run.smithi062.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2023-05-22T15:25:06.092 INFO:teuthology.orchestra.run.smithi016.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2023-05-22T15:25:06.449 INFO:teuthology.orchestra.run.smithi062.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2023-05-22T15:25:06.451 DEBUG:teuthology.orchestra.run.smithi062:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install bison flex libelf-dev libssl-dev network-manager iproute2 util-linux dump indent libaio-dev libtool-bin uuid-dev xfslibs-dev postgresql postgresql-client postgresql-common postgresql-contrib 2023-05-22T15:25:06.547 INFO:teuthology.orchestra.run.smithi016.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2023-05-22T15:25:06.550 DEBUG:teuthology.orchestra.run.smithi016:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install bison flex libelf-dev libssl-dev network-manager iproute2 util-linux dump indent libaio-dev libtool-bin uuid-dev xfslibs-dev postgresql postgresql-client postgresql-common postgresql-contrib 2023-05-22T15:25:06.551 INFO:teuthology.orchestra.run.smithi062.stdout:Reading package lists... 2023-05-22T15:25:06.605 INFO:teuthology.orchestra.run.smithi016.stdout:Reading package lists... 2023-05-22T15:25:06.728 INFO:teuthology.orchestra.run.smithi062.stdout:Building dependency tree... 2023-05-22T15:25:06.729 INFO:teuthology.orchestra.run.smithi062.stdout:Reading state information... 2023-05-22T15:25:06.781 INFO:teuthology.orchestra.run.smithi016.stdout:Building dependency tree... 2023-05-22T15:25:06.782 INFO:teuthology.orchestra.run.smithi016.stdout:Reading state information... 2023-05-22T15:25:06.890 INFO:teuthology.orchestra.run.smithi062.stdout:iproute2 is already the newest version (5.15.0-1ubuntu2). 2023-05-22T15:25:06.891 INFO:teuthology.orchestra.run.smithi062.stdout:libtool-bin is already the newest version (2.4.6-15build2). 2023-05-22T15:25:06.891 INFO:teuthology.orchestra.run.smithi062.stdout:util-linux is already the newest version (2.37.2-4ubuntu3). 2023-05-22T15:25:06.891 INFO:teuthology.orchestra.run.smithi062.stdout:uuid-dev is already the newest version (2.37.2-4ubuntu3). 2023-05-22T15:25:06.892 INFO:teuthology.orchestra.run.smithi062.stdout:xfslibs-dev is already the newest version (5.13.0-1ubuntu2). 2023-05-22T15:25:06.892 INFO:teuthology.orchestra.run.smithi062.stdout:The following packages were automatically installed and are no longer required: 2023-05-22T15:25:06.892 INFO:teuthology.orchestra.run.smithi062.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2023-05-22T15:25:06.892 INFO:teuthology.orchestra.run.smithi062.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2023-05-22T15:25:06.893 INFO:teuthology.orchestra.run.smithi062.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2023-05-22T15:25:06.893 INFO:teuthology.orchestra.run.smithi062.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2023-05-22T15:25:06.894 INFO:teuthology.orchestra.run.smithi062.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2023-05-22T15:25:06.894 INFO:teuthology.orchestra.run.smithi062.stdout:Use 'sudo apt autoremove' to remove them. 2023-05-22T15:25:06.894 INFO:teuthology.orchestra.run.smithi062.stdout:The following additional packages will be installed: 2023-05-22T15:25:06.894 INFO:teuthology.orchestra.run.smithi062.stdout: libbluetooth3 libfl-dev libfl2 libjson-perl libllvm14 liblzo2-2 2023-05-22T15:25:06.894 INFO:teuthology.orchestra.run.smithi062.stdout: libmbim-glib4 libmbim-proxy libmm-glib0 libndp0 libnm0 libpq5 libqmi-glib5 2023-05-22T15:25:06.894 INFO:teuthology.orchestra.run.smithi062.stdout: libqmi-proxy libssl3 libteamdctl0 modemmanager network-manager-pptp 2023-05-22T15:25:06.894 INFO:teuthology.orchestra.run.smithi062.stdout: postgresql-14 postgresql-client-14 postgresql-client-common ppp pptp-linux 2023-05-22T15:25:06.894 INFO:teuthology.orchestra.run.smithi062.stdout: tcl tcl8.6 usb-modeswitch usb-modeswitch-data wpasupplicant 2023-05-22T15:25:06.895 INFO:teuthology.orchestra.run.smithi062.stdout:Suggested packages: 2023-05-22T15:25:06.895 INFO:teuthology.orchestra.run.smithi062.stdout: bison-doc flex-doc indent-doc libssl-doc avahi-autoipd libteam-utils 2023-05-22T15:25:06.895 INFO:teuthology.orchestra.run.smithi062.stdout: postgresql-doc postgresql-doc-14 tcl-tclreadline comgt wvdial wpagui 2023-05-22T15:25:06.895 INFO:teuthology.orchestra.run.smithi062.stdout: libengine-pkcs11-openssl 2023-05-22T15:25:06.929 INFO:teuthology.orchestra.run.smithi062.stdout:The following NEW packages will be installed: 2023-05-22T15:25:06.930 INFO:teuthology.orchestra.run.smithi062.stdout: bison dump flex indent libaio-dev libbluetooth3 libelf-dev libfl-dev libfl2 2023-05-22T15:25:06.930 INFO:teuthology.orchestra.run.smithi062.stdout: libjson-perl libllvm14 liblzo2-2 libmbim-glib4 libmbim-proxy libmm-glib0 2023-05-22T15:25:06.931 INFO:teuthology.orchestra.run.smithi062.stdout: libndp0 libnm0 libpq5 libqmi-glib5 libqmi-proxy libssl-dev libteamdctl0 2023-05-22T15:25:06.931 INFO:teuthology.orchestra.run.smithi062.stdout: modemmanager network-manager network-manager-pptp postgresql postgresql-14 2023-05-22T15:25:06.931 INFO:teuthology.orchestra.run.smithi062.stdout: postgresql-client postgresql-client-14 postgresql-client-common 2023-05-22T15:25:06.931 INFO:teuthology.orchestra.run.smithi062.stdout: postgresql-common postgresql-contrib ppp pptp-linux tcl tcl8.6 2023-05-22T15:25:06.932 INFO:teuthology.orchestra.run.smithi062.stdout: usb-modeswitch usb-modeswitch-data wpasupplicant 2023-05-22T15:25:06.932 INFO:teuthology.orchestra.run.smithi062.stdout:The following packages will be upgraded: 2023-05-22T15:25:06.932 INFO:teuthology.orchestra.run.smithi062.stdout: libssl3 2023-05-22T15:25:06.943 INFO:teuthology.orchestra.run.smithi016.stdout:iproute2 is already the newest version (5.15.0-1ubuntu2). 2023-05-22T15:25:06.943 INFO:teuthology.orchestra.run.smithi016.stdout:libtool-bin is already the newest version (2.4.6-15build2). 2023-05-22T15:25:06.943 INFO:teuthology.orchestra.run.smithi016.stdout:util-linux is already the newest version (2.37.2-4ubuntu3). 2023-05-22T15:25:06.943 INFO:teuthology.orchestra.run.smithi016.stdout:uuid-dev is already the newest version (2.37.2-4ubuntu3). 2023-05-22T15:25:06.943 INFO:teuthology.orchestra.run.smithi016.stdout:xfslibs-dev is already the newest version (5.13.0-1ubuntu2). 2023-05-22T15:25:06.944 INFO:teuthology.orchestra.run.smithi016.stdout:The following packages were automatically installed and are no longer required: 2023-05-22T15:25:06.944 INFO:teuthology.orchestra.run.smithi016.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2023-05-22T15:25:06.944 INFO:teuthology.orchestra.run.smithi016.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2023-05-22T15:25:06.945 INFO:teuthology.orchestra.run.smithi016.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2023-05-22T15:25:06.945 INFO:teuthology.orchestra.run.smithi016.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2023-05-22T15:25:06.945 INFO:teuthology.orchestra.run.smithi016.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2023-05-22T15:25:06.945 INFO:teuthology.orchestra.run.smithi016.stdout:Use 'sudo apt autoremove' to remove them. 2023-05-22T15:25:06.945 INFO:teuthology.orchestra.run.smithi016.stdout:The following additional packages will be installed: 2023-05-22T15:25:06.945 INFO:teuthology.orchestra.run.smithi016.stdout: libbluetooth3 libfl-dev libfl2 libjson-perl libllvm14 liblzo2-2 2023-05-22T15:25:06.946 INFO:teuthology.orchestra.run.smithi016.stdout: libmbim-glib4 libmbim-proxy libmm-glib0 libndp0 libnm0 libpq5 libqmi-glib5 2023-05-22T15:25:06.946 INFO:teuthology.orchestra.run.smithi016.stdout: libqmi-proxy libssl3 libteamdctl0 modemmanager network-manager-pptp 2023-05-22T15:25:06.946 INFO:teuthology.orchestra.run.smithi016.stdout: postgresql-14 postgresql-client-14 postgresql-client-common ppp pptp-linux 2023-05-22T15:25:06.947 INFO:teuthology.orchestra.run.smithi016.stdout: tcl tcl8.6 usb-modeswitch usb-modeswitch-data wpasupplicant 2023-05-22T15:25:06.947 INFO:teuthology.orchestra.run.smithi016.stdout:Suggested packages: 2023-05-22T15:25:06.947 INFO:teuthology.orchestra.run.smithi016.stdout: bison-doc flex-doc indent-doc libssl-doc avahi-autoipd libteam-utils 2023-05-22T15:25:06.947 INFO:teuthology.orchestra.run.smithi016.stdout: postgresql-doc postgresql-doc-14 tcl-tclreadline comgt wvdial wpagui 2023-05-22T15:25:06.947 INFO:teuthology.orchestra.run.smithi016.stdout: libengine-pkcs11-openssl 2023-05-22T15:25:06.983 INFO:teuthology.orchestra.run.smithi016.stdout:The following NEW packages will be installed: 2023-05-22T15:25:06.984 INFO:teuthology.orchestra.run.smithi062.stdout:1 upgraded, 39 newly installed, 0 to remove and 157 not upgraded. 2023-05-22T15:25:06.984 INFO:teuthology.orchestra.run.smithi062.stdout:Need to get 54.6 MB of archives. 2023-05-22T15:25:06.984 INFO:teuthology.orchestra.run.smithi062.stdout:After this operation, 202 MB of additional disk space will be used. 2023-05-22T15:25:06.984 INFO:teuthology.orchestra.run.smithi062.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 flex amd64 2.6.4-8build2 [307 kB] 2023-05-22T15:25:06.985 INFO:teuthology.orchestra.run.smithi016.stdout: bison dump flex indent libaio-dev libbluetooth3 libelf-dev libfl-dev libfl2 2023-05-22T15:25:06.985 INFO:teuthology.orchestra.run.smithi016.stdout: libjson-perl libllvm14 liblzo2-2 libmbim-glib4 libmbim-proxy libmm-glib0 2023-05-22T15:25:06.985 INFO:teuthology.orchestra.run.smithi016.stdout: libndp0 libnm0 libpq5 libqmi-glib5 libqmi-proxy libssl-dev libteamdctl0 2023-05-22T15:25:06.985 INFO:teuthology.orchestra.run.smithi016.stdout: modemmanager network-manager network-manager-pptp postgresql postgresql-14 2023-05-22T15:25:06.985 INFO:teuthology.orchestra.run.smithi016.stdout: postgresql-client postgresql-client-14 postgresql-client-common 2023-05-22T15:25:06.986 INFO:teuthology.orchestra.run.smithi016.stdout: postgresql-common postgresql-contrib ppp pptp-linux tcl tcl8.6 2023-05-22T15:25:06.986 INFO:teuthology.orchestra.run.smithi016.stdout: usb-modeswitch usb-modeswitch-data wpasupplicant 2023-05-22T15:25:06.986 INFO:teuthology.orchestra.run.smithi016.stdout:The following packages will be upgraded: 2023-05-22T15:25:06.987 INFO:teuthology.orchestra.run.smithi016.stdout: libssl3 2023-05-22T15:25:07.036 INFO:teuthology.orchestra.run.smithi016.stdout:1 upgraded, 39 newly installed, 0 to remove and 157 not upgraded. 2023-05-22T15:25:07.037 INFO:teuthology.orchestra.run.smithi016.stdout:Need to get 54.6 MB of archives. 2023-05-22T15:25:07.037 INFO:teuthology.orchestra.run.smithi016.stdout:After this operation, 202 MB of additional disk space will be used. 2023-05-22T15:25:07.037 INFO:teuthology.orchestra.run.smithi016.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 flex amd64 2.6.4-8build2 [307 kB] 2023-05-22T15:25:07.082 INFO:teuthology.orchestra.run.smithi062.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libssl3 amd64 3.0.2-0ubuntu1.9 [1,902 kB] 2023-05-22T15:25:07.136 INFO:teuthology.orchestra.run.smithi016.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libssl3 amd64 3.0.2-0ubuntu1.9 [1,902 kB] 2023-05-22T15:25:07.141 INFO:teuthology.orchestra.run.smithi062.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/main amd64 bison amd64 2:3.8.2+dfsg-1build1 [748 kB] 2023-05-22T15:25:07.156 INFO:teuthology.orchestra.run.smithi062.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblzo2-2 amd64 2.10-2build3 [53.7 kB] 2023-05-22T15:25:07.157 INFO:teuthology.orchestra.run.smithi062.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy/universe amd64 dump amd64 0.4b47-3 [133 kB] 2023-05-22T15:25:07.160 INFO:teuthology.orchestra.run.smithi062.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy/main amd64 indent amd64 2.2.12-1build2 [90.8 kB] 2023-05-22T15:25:07.161 INFO:teuthology.orchestra.run.smithi062.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy/main amd64 libaio-dev amd64 0.3.112-13build1 [21.2 kB] 2023-05-22T15:25:07.162 INFO:teuthology.orchestra.run.smithi062.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy/main amd64 libbluetooth3 amd64 5.64-0ubuntu1 [86.9 kB] 2023-05-22T15:25:07.164 INFO:teuthology.orchestra.run.smithi062.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy/main amd64 libelf-dev amd64 0.186-1build1 [64.4 kB] 2023-05-22T15:25:07.165 INFO:teuthology.orchestra.run.smithi062.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 libfl2 amd64 2.6.4-8build2 [10.7 kB] 2023-05-22T15:25:07.196 INFO:teuthology.orchestra.run.smithi016.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/main amd64 bison amd64 2:3.8.2+dfsg-1build1 [748 kB] 2023-05-22T15:25:07.201 INFO:teuthology.orchestra.run.smithi062.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/main amd64 libfl-dev amd64 2.6.4-8build2 [6,236 B] 2023-05-22T15:25:07.201 INFO:teuthology.orchestra.run.smithi062.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjson-perl all 4.04000-1 [81.8 kB] 2023-05-22T15:25:07.210 INFO:teuthology.orchestra.run.smithi016.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblzo2-2 amd64 2.10-2build3 [53.7 kB] 2023-05-22T15:25:07.211 INFO:teuthology.orchestra.run.smithi016.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy/universe amd64 dump amd64 0.4b47-3 [133 kB] 2023-05-22T15:25:07.214 INFO:teuthology.orchestra.run.smithi016.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy/main amd64 indent amd64 2.2.12-1build2 [90.8 kB] 2023-05-22T15:25:07.216 INFO:teuthology.orchestra.run.smithi016.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy/main amd64 libaio-dev amd64 0.3.112-13build1 [21.2 kB] 2023-05-22T15:25:07.217 INFO:teuthology.orchestra.run.smithi016.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy/main amd64 libbluetooth3 amd64 5.64-0ubuntu1 [86.9 kB] 2023-05-22T15:25:07.218 INFO:teuthology.orchestra.run.smithi016.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy/main amd64 libelf-dev amd64 0.186-1build1 [64.4 kB] 2023-05-22T15:25:07.220 INFO:teuthology.orchestra.run.smithi016.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 libfl2 amd64 2.6.4-8build2 [10.7 kB] 2023-05-22T15:25:07.251 INFO:teuthology.orchestra.run.smithi062.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/main amd64 libllvm14 amd64 1:14.0.0-1ubuntu1 [24.0 MB] 2023-05-22T15:25:07.255 INFO:teuthology.orchestra.run.smithi016.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/main amd64 libfl-dev amd64 2.6.4-8build2 [6,236 B] 2023-05-22T15:25:07.256 INFO:teuthology.orchestra.run.smithi016.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjson-perl all 4.04000-1 [81.8 kB] 2023-05-22T15:25:07.306 INFO:teuthology.orchestra.run.smithi016.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/main amd64 libllvm14 amd64 1:14.0.0-1ubuntu1 [24.0 MB] 2023-05-22T15:25:07.656 INFO:teuthology.orchestra.run.smithi062.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmbim-glib4 amd64 1.28.0-1~ubuntu20.04.1 [191 kB] 2023-05-22T15:25:07.658 INFO:teuthology.orchestra.run.smithi062.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmbim-proxy amd64 1.28.0-1~ubuntu20.04.1 [6,130 B] 2023-05-22T15:25:07.658 INFO:teuthology.orchestra.run.smithi062.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmm-glib0 amd64 1.20.0-1~ubuntu22.04.1 [263 kB] 2023-05-22T15:25:07.658 INFO:teuthology.orchestra.run.smithi016.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmbim-glib4 amd64 1.28.0-1~ubuntu20.04.1 [191 kB] 2023-05-22T15:25:07.660 INFO:teuthology.orchestra.run.smithi016.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmbim-proxy amd64 1.28.0-1~ubuntu20.04.1 [6,130 B] 2023-05-22T15:25:07.661 INFO:teuthology.orchestra.run.smithi062.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy/main amd64 libndp0 amd64 1.8-0ubuntu3 [12.3 kB] 2023-05-22T15:25:07.661 INFO:teuthology.orchestra.run.smithi062.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libnm0 amd64 1.36.6-0ubuntu2 [456 kB] 2023-05-22T15:25:07.664 INFO:teuthology.orchestra.run.smithi062.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpq5 amd64 14.7-0ubuntu0.22.04.1 [141 kB] 2023-05-22T15:25:07.666 INFO:teuthology.orchestra.run.smithi062.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libqmi-glib5 amd64 1.32.0-1ubuntu0.22.04.1 [772 kB] 2023-05-22T15:25:07.672 INFO:teuthology.orchestra.run.smithi062.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libqmi-proxy amd64 1.32.0-1ubuntu0.22.04.1 [6,072 B] 2023-05-22T15:25:07.707 INFO:teuthology.orchestra.run.smithi062.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libssl-dev amd64 3.0.2-0ubuntu1.9 [2,373 kB] 2023-05-22T15:25:07.716 INFO:teuthology.orchestra.run.smithi016.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmm-glib0 amd64 1.20.0-1~ubuntu22.04.1 [263 kB] 2023-05-22T15:25:07.719 INFO:teuthology.orchestra.run.smithi016.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy/main amd64 libndp0 amd64 1.8-0ubuntu3 [12.3 kB] 2023-05-22T15:25:07.720 INFO:teuthology.orchestra.run.smithi016.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libnm0 amd64 1.36.6-0ubuntu2 [456 kB] 2023-05-22T15:25:07.724 INFO:teuthology.orchestra.run.smithi016.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpq5 amd64 14.7-0ubuntu0.22.04.1 [141 kB] 2023-05-22T15:25:07.726 INFO:teuthology.orchestra.run.smithi016.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libqmi-glib5 amd64 1.32.0-1ubuntu0.22.04.1 [772 kB] 2023-05-22T15:25:07.733 INFO:teuthology.orchestra.run.smithi016.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libqmi-proxy amd64 1.32.0-1ubuntu0.22.04.1 [6,072 B] 2023-05-22T15:25:07.768 INFO:teuthology.orchestra.run.smithi016.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libssl-dev amd64 3.0.2-0ubuntu1.9 [2,373 kB] 2023-05-22T15:25:07.868 INFO:teuthology.orchestra.run.smithi062.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 libteamdctl0 amd64 1.31-1build2 [12.6 kB] 2023-05-22T15:25:07.868 INFO:teuthology.orchestra.run.smithi062.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 modemmanager amd64 1.20.0-1~ubuntu22.04.1 [1,094 kB] 2023-05-22T15:25:07.891 INFO:teuthology.orchestra.run.smithi062.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 network-manager amd64 1.36.6-0ubuntu2 [2,203 kB] 2023-05-22T15:25:07.926 INFO:teuthology.orchestra.run.smithi062.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 ppp amd64 2.4.9-1+1ubuntu3 [363 kB] 2023-05-22T15:25:07.929 INFO:teuthology.orchestra.run.smithi016.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 libteamdctl0 amd64 1.31-1build2 [12.6 kB] 2023-05-22T15:25:07.929 INFO:teuthology.orchestra.run.smithi016.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 modemmanager amd64 1.20.0-1~ubuntu22.04.1 [1,094 kB] 2023-05-22T15:25:07.930 INFO:teuthology.orchestra.run.smithi062.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 pptp-linux amd64 1.10.0-1build3 [42.5 kB] 2023-05-22T15:25:07.931 INFO:teuthology.orchestra.run.smithi062.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 network-manager-pptp amd64 1.2.10-1 [30.9 kB] 2023-05-22T15:25:07.932 INFO:teuthology.orchestra.run.smithi062.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-client-common all 238 [29.6 kB] 2023-05-22T15:25:07.932 INFO:teuthology.orchestra.run.smithi062.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 postgresql-client-14 amd64 14.7-0ubuntu0.22.04.1 [1,220 kB] 2023-05-22T15:25:07.946 INFO:teuthology.orchestra.run.smithi062.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-common all 238 [169 kB] 2023-05-22T15:25:07.948 INFO:teuthology.orchestra.run.smithi062.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 postgresql-14 amd64 14.7-0ubuntu0.22.04.1 [16.1 MB] 2023-05-22T15:25:07.950 INFO:teuthology.orchestra.run.smithi016.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 network-manager amd64 1.36.6-0ubuntu2 [2,203 kB] 2023-05-22T15:25:07.984 INFO:teuthology.orchestra.run.smithi016.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 ppp amd64 2.4.9-1+1ubuntu3 [363 kB] 2023-05-22T15:25:07.989 INFO:teuthology.orchestra.run.smithi016.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 pptp-linux amd64 1.10.0-1build3 [42.5 kB] 2023-05-22T15:25:07.989 INFO:teuthology.orchestra.run.smithi016.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 network-manager-pptp amd64 1.2.10-1 [30.9 kB] 2023-05-22T15:25:07.990 INFO:teuthology.orchestra.run.smithi016.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-client-common all 238 [29.6 kB] 2023-05-22T15:25:07.991 INFO:teuthology.orchestra.run.smithi016.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 postgresql-client-14 amd64 14.7-0ubuntu0.22.04.1 [1,220 kB] 2023-05-22T15:25:08.004 INFO:teuthology.orchestra.run.smithi016.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-common all 238 [169 kB] 2023-05-22T15:25:08.006 INFO:teuthology.orchestra.run.smithi016.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 postgresql-14 amd64 14.7-0ubuntu0.22.04.1 [16.1 MB] 2023-05-22T15:25:08.187 INFO:teuthology.orchestra.run.smithi062.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql all 14+238 [3,288 B] 2023-05-22T15:25:08.204 INFO:teuthology.orchestra.run.smithi062.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-client all 14+238 [3,292 B] 2023-05-22T15:25:08.205 INFO:teuthology.orchestra.run.smithi062.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-contrib all 14+238 [3,292 B] 2023-05-22T15:25:08.205 INFO:teuthology.orchestra.run.smithi062.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 tcl8.6 amd64 8.6.12+dfsg-1build1 [15.0 kB] 2023-05-22T15:25:08.223 INFO:teuthology.orchestra.run.smithi062.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 tcl amd64 8.6.11+1build2 [4,678 B] 2023-05-22T15:25:08.224 INFO:teuthology.orchestra.run.smithi062.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 wpasupplicant amd64 2:2.10-6ubuntu2 [1,481 kB] 2023-05-22T15:25:08.251 INFO:teuthology.orchestra.run.smithi016.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql all 14+238 [3,288 B] 2023-05-22T15:25:08.269 INFO:teuthology.orchestra.run.smithi016.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-client all 14+238 [3,292 B] 2023-05-22T15:25:08.269 INFO:teuthology.orchestra.run.smithi016.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-contrib all 14+238 [3,292 B] 2023-05-22T15:25:08.270 INFO:teuthology.orchestra.run.smithi016.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 tcl8.6 amd64 8.6.12+dfsg-1build1 [15.0 kB] 2023-05-22T15:25:08.287 INFO:teuthology.orchestra.run.smithi016.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 tcl amd64 8.6.11+1build2 [4,678 B] 2023-05-22T15:25:08.288 INFO:teuthology.orchestra.run.smithi016.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 wpasupplicant amd64 2:2.10-6ubuntu2 [1,481 kB] 2023-05-22T15:25:08.322 INFO:teuthology.orchestra.run.smithi062.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 usb-modeswitch-data all 20191128-4 [33.2 kB] 2023-05-22T15:25:08.323 INFO:teuthology.orchestra.run.smithi062.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 usb-modeswitch amd64 2.6.1-3ubuntu2 [46.0 kB] 2023-05-22T15:25:08.386 INFO:teuthology.orchestra.run.smithi016.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 usb-modeswitch-data all 20191128-4 [33.2 kB] 2023-05-22T15:25:08.387 INFO:teuthology.orchestra.run.smithi016.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 usb-modeswitch amd64 2.6.1-3ubuntu2 [46.0 kB] 2023-05-22T15:25:08.740 INFO:teuthology.orchestra.run.smithi062.stdout:Preconfiguring packages ... 2023-05-22T15:25:08.802 INFO:teuthology.orchestra.run.smithi016.stdout:Preconfiguring packages ... 2023-05-22T15:25:08.910 INFO:teuthology.orchestra.run.smithi062.stdout:Fetched 54.6 MB in 1s (39.4 MB/s) 2023-05-22T15:25:08.973 INFO:teuthology.orchestra.run.smithi016.stdout:Fetched 54.6 MB in 1s (39.1 MB/s) 2023-05-22T15:25:09.051 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package flex. 2023-05-22T15:25:09.118 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package flex. 2023-05-22T15:25:09.148 INFO:teuthology.orchestra.run.smithi062.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 172965 files and directories currently installed.) 2023-05-22T15:25:09.154 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../flex_2.6.4-8build2_amd64.deb ... 2023-05-22T15:25:09.194 INFO:teuthology.orchestra.run.smithi016.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 172965 files and directories currently installed.) 2023-05-22T15:25:09.199 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking flex (2.6.4-8build2) ... 2023-05-22T15:25:09.200 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../flex_2.6.4-8build2_amd64.deb ... 2023-05-22T15:25:09.241 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking flex (2.6.4-8build2) ... 2023-05-22T15:25:09.732 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../libssl3_3.0.2-0ubuntu1.9_amd64.deb ... 2023-05-22T15:25:09.775 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../libssl3_3.0.2-0ubuntu1.9_amd64.deb ... 2023-05-22T15:25:09.888 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking libssl3:amd64 (3.0.2-0ubuntu1.9) over (3.0.2-0ubuntu1.7) ... 2023-05-22T15:25:09.914 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking libssl3:amd64 (3.0.2-0ubuntu1.9) over (3.0.2-0ubuntu1.7) ... 2023-05-22T15:25:10.290 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up libssl3:amd64 (3.0.2-0ubuntu1.9) ... 2023-05-22T15:25:10.316 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up libssl3:amd64 (3.0.2-0ubuntu1.9) ... 2023-05-22T15:25:10.824 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package bison. 2023-05-22T15:25:10.855 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package bison. 2023-05-22T15:25:10.925 INFO:teuthology.orchestra.run.smithi016.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 173038 files and directories currently installed.) 2023-05-22T15:25:10.931 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../00-bison_2%3a3.8.2+dfsg-1build1_amd64.deb ... 2023-05-22T15:25:10.962 INFO:teuthology.orchestra.run.smithi062.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 173038 files and directories currently installed.) 2023-05-22T15:25:10.968 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../00-bison_2%3a3.8.2+dfsg-1build1_amd64.deb ... 2023-05-22T15:25:10.976 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking bison (2:3.8.2+dfsg-1build1) ... 2023-05-22T15:25:11.007 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking bison (2:3.8.2+dfsg-1build1) ... 2023-05-22T15:25:11.351 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package liblzo2-2:amd64. 2023-05-22T15:25:11.368 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../01-liblzo2-2_2.10-2build3_amd64.deb ... 2023-05-22T15:25:11.401 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking liblzo2-2:amd64 (2.10-2build3) ... 2023-05-22T15:25:11.433 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package liblzo2-2:amd64. 2023-05-22T15:25:11.461 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../01-liblzo2-2_2.10-2build3_amd64.deb ... 2023-05-22T15:25:11.500 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking liblzo2-2:amd64 (2.10-2build3) ... 2023-05-22T15:25:11.730 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package dump. 2023-05-22T15:25:11.758 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../02-dump_0.4b47-3_amd64.deb ... 2023-05-22T15:25:11.796 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking dump (0.4b47-3) ... 2023-05-22T15:25:11.827 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package dump. 2023-05-22T15:25:11.855 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../02-dump_0.4b47-3_amd64.deb ... 2023-05-22T15:25:11.894 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking dump (0.4b47-3) ... 2023-05-22T15:25:12.075 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package indent. 2023-05-22T15:25:12.091 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../03-indent_2.2.12-1build2_amd64.deb ... 2023-05-22T15:25:12.133 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking indent (2.2.12-1build2) ... 2023-05-22T15:25:12.173 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package indent. 2023-05-22T15:25:12.189 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../03-indent_2.2.12-1build2_amd64.deb ... 2023-05-22T15:25:12.231 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking indent (2.2.12-1build2) ... 2023-05-22T15:25:12.410 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package libaio-dev:amd64. 2023-05-22T15:25:12.438 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../04-libaio-dev_0.3.112-13build1_amd64.deb ... 2023-05-22T15:25:12.458 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package libaio-dev:amd64. 2023-05-22T15:25:12.475 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../04-libaio-dev_0.3.112-13build1_amd64.deb ... 2023-05-22T15:25:12.477 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking libaio-dev:amd64 (0.3.112-13build1) ... 2023-05-22T15:25:12.508 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking libaio-dev:amd64 (0.3.112-13build1) ... 2023-05-22T15:25:12.781 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package libbluetooth3:amd64. 2023-05-22T15:25:12.809 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../05-libbluetooth3_5.64-0ubuntu1_amd64.deb ... 2023-05-22T15:25:12.837 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package libbluetooth3:amd64. 2023-05-22T15:25:12.847 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking libbluetooth3:amd64 (5.64-0ubuntu1) ... 2023-05-22T15:25:12.865 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../05-libbluetooth3_5.64-0ubuntu1_amd64.deb ... 2023-05-22T15:25:12.903 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking libbluetooth3:amd64 (5.64-0ubuntu1) ... 2023-05-22T15:25:13.165 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package libelf-dev:amd64. 2023-05-22T15:25:13.192 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../06-libelf-dev_0.186-1build1_amd64.deb ... 2023-05-22T15:25:13.231 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking libelf-dev:amd64 (0.186-1build1) ... 2023-05-22T15:25:13.260 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package libelf-dev:amd64. 2023-05-22T15:25:13.277 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../06-libelf-dev_0.186-1build1_amd64.deb ... 2023-05-22T15:25:13.318 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking libelf-dev:amd64 (0.186-1build1) ... 2023-05-22T15:25:13.534 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package libfl2:amd64. 2023-05-22T15:25:13.562 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../07-libfl2_2.6.4-8build2_amd64.deb ... 2023-05-22T15:25:13.600 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking libfl2:amd64 (2.6.4-8build2) ... 2023-05-22T15:25:13.621 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package libfl2:amd64. 2023-05-22T15:25:13.650 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../07-libfl2_2.6.4-8build2_amd64.deb ... 2023-05-22T15:25:13.680 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking libfl2:amd64 (2.6.4-8build2) ... 2023-05-22T15:25:13.819 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package libfl-dev:amd64. 2023-05-22T15:25:13.836 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../08-libfl-dev_2.6.4-8build2_amd64.deb ... 2023-05-22T15:25:13.878 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking libfl-dev:amd64 (2.6.4-8build2) ... 2023-05-22T15:25:13.899 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package libfl-dev:amd64. 2023-05-22T15:25:13.916 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../08-libfl-dev_2.6.4-8build2_amd64.deb ... 2023-05-22T15:25:13.948 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking libfl-dev:amd64 (2.6.4-8build2) ... 2023-05-22T15:25:14.139 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package libjson-perl. 2023-05-22T15:25:14.167 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../09-libjson-perl_4.04000-1_all.deb ... 2023-05-22T15:25:14.205 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking libjson-perl (4.04000-1) ... 2023-05-22T15:25:14.209 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package libjson-perl. 2023-05-22T15:25:14.228 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../09-libjson-perl_4.04000-1_all.deb ... 2023-05-22T15:25:14.268 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking libjson-perl (4.04000-1) ... 2023-05-22T15:25:14.517 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package libllvm14:amd64. 2023-05-22T15:25:14.544 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../10-libllvm14_1%3a14.0.0-1ubuntu1_amd64.deb ... 2023-05-22T15:25:14.579 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package libllvm14:amd64. 2023-05-22T15:25:14.583 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking libllvm14:amd64 (1:14.0.0-1ubuntu1) ... 2023-05-22T15:25:14.608 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../10-libllvm14_1%3a14.0.0-1ubuntu1_amd64.deb ... 2023-05-22T15:25:14.646 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking libllvm14:amd64 (1:14.0.0-1ubuntu1) ... 2023-05-22T15:25:16.303 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package libmbim-glib4:amd64. 2023-05-22T15:25:16.330 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../11-libmbim-glib4_1.28.0-1~ubuntu20.04.1_amd64.deb ... 2023-05-22T15:25:16.365 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package libmbim-glib4:amd64. 2023-05-22T15:25:16.369 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking libmbim-glib4:amd64 (1.28.0-1~ubuntu20.04.1) ... 2023-05-22T15:25:16.393 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../11-libmbim-glib4_1.28.0-1~ubuntu20.04.1_amd64.deb ... 2023-05-22T15:25:16.432 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking libmbim-glib4:amd64 (1.28.0-1~ubuntu20.04.1) ... 2023-05-22T15:25:16.613 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package libmbim-proxy. 2023-05-22T15:25:16.630 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../12-libmbim-proxy_1.28.0-1~ubuntu20.04.1_amd64.deb ... 2023-05-22T15:25:16.671 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking libmbim-proxy (1.28.0-1~ubuntu20.04.1) ... 2023-05-22T15:25:16.676 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package libmbim-proxy. 2023-05-22T15:25:16.693 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../12-libmbim-proxy_1.28.0-1~ubuntu20.04.1_amd64.deb ... 2023-05-22T15:25:16.726 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking libmbim-proxy (1.28.0-1~ubuntu20.04.1) ... 2023-05-22T15:25:16.982 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package libmm-glib0:amd64. 2023-05-22T15:25:17.011 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../13-libmm-glib0_1.20.0-1~ubuntu22.04.1_amd64.deb ... 2023-05-22T15:25:17.049 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking libmm-glib0:amd64 (1.20.0-1~ubuntu22.04.1) ... 2023-05-22T15:25:17.053 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package libmm-glib0:amd64. 2023-05-22T15:25:17.082 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../13-libmm-glib0_1.20.0-1~ubuntu22.04.1_amd64.deb ... 2023-05-22T15:25:17.120 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking libmm-glib0:amd64 (1.20.0-1~ubuntu22.04.1) ... 2023-05-22T15:25:17.393 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package libndp0:amd64. 2023-05-22T15:25:17.421 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../14-libndp0_1.8-0ubuntu3_amd64.deb ... 2023-05-22T15:25:17.459 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking libndp0:amd64 (1.8-0ubuntu3) ... 2023-05-22T15:25:17.465 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package libndp0:amd64. 2023-05-22T15:25:17.493 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../14-libndp0_1.8-0ubuntu3_amd64.deb ... 2023-05-22T15:25:17.531 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking libndp0:amd64 (1.8-0ubuntu3) ... 2023-05-22T15:25:17.762 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package libnm0:amd64. 2023-05-22T15:25:17.790 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../15-libnm0_1.36.6-0ubuntu2_amd64.deb ... 2023-05-22T15:25:17.829 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking libnm0:amd64 (1.36.6-0ubuntu2) ... 2023-05-22T15:25:17.834 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package libnm0:amd64. 2023-05-22T15:25:17.851 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../15-libnm0_1.36.6-0ubuntu2_amd64.deb ... 2023-05-22T15:25:17.892 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking libnm0:amd64 (1.36.6-0ubuntu2) ... 2023-05-22T15:25:18.165 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package libpq5:amd64. 2023-05-22T15:25:18.193 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../16-libpq5_14.7-0ubuntu0.22.04.1_amd64.deb ... 2023-05-22T15:25:18.250 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking libpq5:amd64 (14.7-0ubuntu0.22.04.1) ... 2023-05-22T15:25:18.565 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package libpq5:amd64. 2023-05-22T15:25:18.584 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package libqmi-glib5:amd64. 2023-05-22T15:25:18.594 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../16-libpq5_14.7-0ubuntu0.22.04.1_amd64.deb ... 2023-05-22T15:25:18.612 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../17-libqmi-glib5_1.32.0-1ubuntu0.22.04.1_amd64.deb ... 2023-05-22T15:25:18.651 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking libqmi-glib5:amd64 (1.32.0-1ubuntu0.22.04.1) ... 2023-05-22T15:25:18.729 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking libpq5:amd64 (14.7-0ubuntu0.22.04.1) ... 2023-05-22T15:25:18.961 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package libqmi-proxy. 2023-05-22T15:25:18.989 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../18-libqmi-proxy_1.32.0-1ubuntu0.22.04.1_amd64.deb ... 2023-05-22T15:25:19.028 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking libqmi-proxy (1.32.0-1ubuntu0.22.04.1) ... 2023-05-22T15:25:19.082 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package libqmi-glib5:amd64. 2023-05-22T15:25:19.110 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../17-libqmi-glib5_1.32.0-1ubuntu0.22.04.1_amd64.deb ... 2023-05-22T15:25:19.167 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking libqmi-glib5:amd64 (1.32.0-1ubuntu0.22.04.1) ... 2023-05-22T15:25:19.255 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package libssl-dev:amd64. 2023-05-22T15:25:19.272 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../19-libssl-dev_3.0.2-0ubuntu1.9_amd64.deb ... 2023-05-22T15:25:19.314 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking libssl-dev:amd64 (3.0.2-0ubuntu1.9) ... 2023-05-22T15:25:19.484 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package libqmi-proxy. 2023-05-22T15:25:19.501 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../18-libqmi-proxy_1.32.0-1ubuntu0.22.04.1_amd64.deb ... 2023-05-22T15:25:19.542 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking libqmi-proxy (1.32.0-1ubuntu0.22.04.1) ... 2023-05-22T15:25:19.753 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package libssl-dev:amd64. 2023-05-22T15:25:19.771 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../19-libssl-dev_3.0.2-0ubuntu1.9_amd64.deb ... 2023-05-22T15:25:19.803 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking libssl-dev:amd64 (3.0.2-0ubuntu1.9) ... 2023-05-22T15:25:19.825 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package libteamdctl0:amd64. 2023-05-22T15:25:19.853 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../20-libteamdctl0_1.31-1build2_amd64.deb ... 2023-05-22T15:25:19.892 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking libteamdctl0:amd64 (1.31-1build2) ... 2023-05-22T15:25:20.220 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package modemmanager. 2023-05-22T15:25:20.248 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../21-modemmanager_1.20.0-1~ubuntu22.04.1_amd64.deb ... 2023-05-22T15:25:20.278 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking modemmanager (1.20.0-1~ubuntu22.04.1) ... 2023-05-22T15:25:20.357 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package libteamdctl0:amd64. 2023-05-22T15:25:20.385 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../20-libteamdctl0_1.31-1build2_amd64.deb ... 2023-05-22T15:25:20.423 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking libteamdctl0:amd64 (1.31-1build2) ... 2023-05-22T15:25:20.743 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package modemmanager. 2023-05-22T15:25:20.771 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../21-modemmanager_1.20.0-1~ubuntu22.04.1_amd64.deb ... 2023-05-22T15:25:20.801 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking modemmanager (1.20.0-1~ubuntu22.04.1) ... 2023-05-22T15:25:20.892 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package network-manager. 2023-05-22T15:25:20.920 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../22-network-manager_1.36.6-0ubuntu2_amd64.deb ... 2023-05-22T15:25:20.967 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking network-manager (1.36.6-0ubuntu2) ... 2023-05-22T15:25:21.440 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package network-manager. 2023-05-22T15:25:21.469 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../22-network-manager_1.36.6-0ubuntu2_amd64.deb ... 2023-05-22T15:25:21.516 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking network-manager (1.36.6-0ubuntu2) ... 2023-05-22T15:25:21.574 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package ppp. 2023-05-22T15:25:21.602 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../23-ppp_2.4.9-1+1ubuntu3_amd64.deb ... 2023-05-22T15:25:21.892 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking ppp (2.4.9-1+1ubuntu3) ... 2023-05-22T15:25:22.072 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package ppp. 2023-05-22T15:25:22.101 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../23-ppp_2.4.9-1+1ubuntu3_amd64.deb ... 2023-05-22T15:25:22.195 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package pptp-linux. 2023-05-22T15:25:22.212 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../24-pptp-linux_1.10.0-1build3_amd64.deb ... 2023-05-22T15:25:22.253 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking pptp-linux (1.10.0-1build3) ... 2023-05-22T15:25:22.373 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking ppp (2.4.9-1+1ubuntu3) ... 2023-05-22T15:25:22.514 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package network-manager-pptp. 2023-05-22T15:25:22.532 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../25-network-manager-pptp_1.2.10-1_amd64.deb ... 2023-05-22T15:25:22.572 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking network-manager-pptp (1.2.10-1) ... 2023-05-22T15:25:22.660 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package pptp-linux. 2023-05-22T15:25:22.673 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../24-pptp-linux_1.10.0-1build3_amd64.deb ... 2023-05-22T15:25:22.701 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking pptp-linux (1.10.0-1build3) ... 2023-05-22T15:25:22.875 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package postgresql-client-common. 2023-05-22T15:25:22.904 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../26-postgresql-client-common_238_all.deb ... 2023-05-22T15:25:22.912 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package network-manager-pptp. 2023-05-22T15:25:22.937 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../25-network-manager-pptp_1.2.10-1_amd64.deb ... 2023-05-22T15:25:22.942 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking postgresql-client-common (238) ... 2023-05-22T15:25:22.979 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking network-manager-pptp (1.2.10-1) ... 2023-05-22T15:25:23.196 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package postgresql-client-14. 2023-05-22T15:25:23.224 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../27-postgresql-client-14_14.7-0ubuntu0.22.04.1_amd64.deb ... 2023-05-22T15:25:23.253 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking postgresql-client-14 (14.7-0ubuntu0.22.04.1) ... 2023-05-22T15:25:23.282 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package postgresql-client-common. 2023-05-22T15:25:23.311 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../26-postgresql-client-common_238_all.deb ... 2023-05-22T15:25:23.348 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking postgresql-client-common (238) ... 2023-05-22T15:25:23.652 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package postgresql-client-14. 2023-05-22T15:25:23.681 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../27-postgresql-client-14_14.7-0ubuntu0.22.04.1_amd64.deb ... 2023-05-22T15:25:23.719 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking postgresql-client-14 (14.7-0ubuntu0.22.04.1) ... 2023-05-22T15:25:23.842 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package postgresql-common. 2023-05-22T15:25:23.870 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../28-postgresql-common_238_all.deb ... 2023-05-22T15:25:23.936 INFO:teuthology.orchestra.run.smithi062.stdout:Adding 'diversion of /usr/bin/pg_config to /usr/bin/pg_config.libpq-dev by postgresql-common' 2023-05-22T15:25:23.965 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking postgresql-common (238) ... 2023-05-22T15:25:24.286 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package postgresql-common. 2023-05-22T15:25:24.314 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../28-postgresql-common_238_all.deb ... 2023-05-22T15:25:24.368 INFO:teuthology.orchestra.run.smithi016.stdout:Adding 'diversion of /usr/bin/pg_config to /usr/bin/pg_config.libpq-dev by postgresql-common' 2023-05-22T15:25:24.397 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking postgresql-common (238) ... 2023-05-22T15:25:24.407 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package postgresql-14. 2023-05-22T15:25:24.436 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../29-postgresql-14_14.7-0ubuntu0.22.04.1_amd64.deb ... 2023-05-22T15:25:24.476 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking postgresql-14 (14.7-0ubuntu0.22.04.1) ... 2023-05-22T15:25:24.890 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package postgresql-14. 2023-05-22T15:25:24.919 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../29-postgresql-14_14.7-0ubuntu0.22.04.1_amd64.deb ... 2023-05-22T15:25:24.960 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking postgresql-14 (14.7-0ubuntu0.22.04.1) ... 2023-05-22T15:25:25.760 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package postgresql. 2023-05-22T15:25:25.777 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../30-postgresql_14+238_all.deb ... 2023-05-22T15:25:25.818 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking postgresql (14+238) ... 2023-05-22T15:25:26.062 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package postgresql-client. 2023-05-22T15:25:26.090 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../31-postgresql-client_14+238_all.deb ... 2023-05-22T15:25:26.128 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking postgresql-client (14+238) ... 2023-05-22T15:25:26.268 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package postgresql. 2023-05-22T15:25:26.286 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../30-postgresql_14+238_all.deb ... 2023-05-22T15:25:26.318 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking postgresql (14+238) ... 2023-05-22T15:25:26.372 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package postgresql-contrib. 2023-05-22T15:25:26.389 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../32-postgresql-contrib_14+238_all.deb ... 2023-05-22T15:25:26.430 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking postgresql-contrib (14+238) ... 2023-05-22T15:25:26.562 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package postgresql-client. 2023-05-22T15:25:26.592 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../31-postgresql-client_14+238_all.deb ... 2023-05-22T15:25:26.628 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking postgresql-client (14+238) ... 2023-05-22T15:25:26.758 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package tcl8.6. 2023-05-22T15:25:26.786 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../33-tcl8.6_8.6.12+dfsg-1build1_amd64.deb ... 2023-05-22T15:25:26.816 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking tcl8.6 (8.6.12+dfsg-1build1) ... 2023-05-22T15:25:26.881 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package postgresql-contrib. 2023-05-22T15:25:26.898 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../32-postgresql-contrib_14+238_all.deb ... 2023-05-22T15:25:26.939 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking postgresql-contrib (14+238) ... 2023-05-22T15:25:27.085 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package tcl. 2023-05-22T15:25:27.114 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../34-tcl_8.6.11+1build2_amd64.deb ... 2023-05-22T15:25:27.157 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking tcl (8.6.11+1build2) ... 2023-05-22T15:25:27.242 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package tcl8.6. 2023-05-22T15:25:27.271 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../33-tcl8.6_8.6.12+dfsg-1build1_amd64.deb ... 2023-05-22T15:25:27.308 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking tcl8.6 (8.6.12+dfsg-1build1) ... 2023-05-22T15:25:27.505 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package wpasupplicant. 2023-05-22T15:25:27.533 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../35-wpasupplicant_2%3a2.10-6ubuntu2_amd64.deb ... 2023-05-22T15:25:27.552 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package tcl. 2023-05-22T15:25:27.570 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../34-tcl_8.6.11+1build2_amd64.deb ... 2023-05-22T15:25:27.571 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking wpasupplicant (2:2.10-6ubuntu2) ... 2023-05-22T15:25:27.607 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking tcl (8.6.11+1build2) ... 2023-05-22T15:25:27.934 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package usb-modeswitch-data. 2023-05-22T15:25:27.963 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../36-usb-modeswitch-data_20191128-4_all.deb ... 2023-05-22T15:25:27.964 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package wpasupplicant. 2023-05-22T15:25:27.993 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../35-wpasupplicant_2%3a2.10-6ubuntu2_amd64.deb ... 2023-05-22T15:25:28.000 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking usb-modeswitch-data (20191128-4) ... 2023-05-22T15:25:28.030 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking wpasupplicant (2:2.10-6ubuntu2) ... 2023-05-22T15:25:28.320 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package usb-modeswitch. 2023-05-22T15:25:28.348 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../37-usb-modeswitch_2.6.1-3ubuntu2_amd64.deb ... 2023-05-22T15:25:28.395 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking usb-modeswitch (2.6.1-3ubuntu2) ... 2023-05-22T15:25:28.418 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package usb-modeswitch-data. 2023-05-22T15:25:28.447 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../36-usb-modeswitch-data_20191128-4_all.deb ... 2023-05-22T15:25:28.484 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking usb-modeswitch-data (20191128-4) ... 2023-05-22T15:25:28.690 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up postgresql-client-common (238) ... 2023-05-22T15:25:28.796 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package usb-modeswitch. 2023-05-22T15:25:28.813 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../37-usb-modeswitch_2.6.1-3ubuntu2_amd64.deb ... 2023-05-22T15:25:28.854 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking usb-modeswitch (2.6.1-3ubuntu2) ... 2023-05-22T15:25:28.900 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up flex (2.6.4-8build2) ... 2023-05-22T15:25:29.076 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up tcl8.6 (8.6.12+dfsg-1build1) ... 2023-05-22T15:25:29.202 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up libpq5:amd64 (14.7-0ubuntu0.22.04.1) ... 2023-05-22T15:25:29.302 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up postgresql-client-common (238) ... 2023-05-22T15:25:29.478 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up liblzo2-2:amd64 (2.10-2build3) ... 2023-05-22T15:25:29.537 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up flex (2.6.4-8build2) ... 2023-05-22T15:25:29.604 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up libaio-dev:amd64 (0.3.112-13build1) ... 2023-05-22T15:25:29.679 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up tcl8.6 (8.6.12+dfsg-1build1) ... 2023-05-22T15:25:29.730 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up postgresql-client-14 (14.7-0ubuntu0.22.04.1) ... 2023-05-22T15:25:29.814 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up libpq5:amd64 (14.7-0ubuntu0.22.04.1) ... 2023-05-22T15:25:29.940 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up liblzo2-2:amd64 (2.10-2build3) ... 2023-05-22T15:25:30.049 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up libaio-dev:amd64 (0.3.112-13build1) ... 2023-05-22T15:25:30.175 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up postgresql-client-14 (14.7-0ubuntu0.22.04.1) ... 2023-05-22T15:25:30.257 INFO:teuthology.orchestra.run.smithi062.stdout:update-alternatives: using /usr/share/postgresql/14/man/man1/psql.1.gz to provide /usr/share/man/man1/psql.1.gz (psql.1.gz) in auto mode 2023-05-22T15:25:30.335 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up wpasupplicant (2:2.10-6ubuntu2) ... 2023-05-22T15:25:30.688 INFO:teuthology.orchestra.run.smithi016.stdout:update-alternatives: using /usr/share/postgresql/14/man/man1/psql.1.gz to provide /usr/share/man/man1/psql.1.gz (psql.1.gz) in auto mode 2023-05-22T15:25:30.773 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up wpasupplicant (2:2.10-6ubuntu2) ... 2023-05-22T15:25:30.823 INFO:teuthology.orchestra.run.smithi062.stdout:Created symlink /etc/systemd/system/dbus-fi.w1.wpa_supplicant1.service -> /lib/systemd/system/wpa_supplicant.service. 2023-05-22T15:25:30.823 INFO:teuthology.orchestra.run.smithi062.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/wpa_supplicant.service -> /lib/systemd/system/wpa_supplicant.service. 2023-05-22T15:25:31.163 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/dbus-fi.w1.wpa_supplicant1.service -> /lib/systemd/system/wpa_supplicant.service. 2023-05-22T15:25:31.163 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/wpa_supplicant.service -> /lib/systemd/system/wpa_supplicant.service. 2023-05-22T15:25:31.380 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up libteamdctl0:amd64 (1.31-1build2) ... 2023-05-22T15:25:31.515 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up libssl-dev:amd64 (3.0.2-0ubuntu1.9) ... 2023-05-22T15:25:31.785 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up libteamdctl0:amd64 (1.31-1build2) ... 2023-05-22T15:25:31.911 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up libssl-dev:amd64 (3.0.2-0ubuntu1.9) ... 2023-05-22T15:25:32.037 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up usb-modeswitch-data (20191128-4) ... 2023-05-22T15:25:32.057 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up usb-modeswitch-data (20191128-4) ... 2023-05-22T15:25:32.163 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up libelf-dev:amd64 (0.186-1build1) ... 2023-05-22T15:25:32.194 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up libelf-dev:amd64 (0.186-1build1) ... 2023-05-22T15:25:32.314 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up libfl2:amd64 (2.6.4-8build2) ... 2023-05-22T15:25:32.326 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up libfl2:amd64 (2.6.4-8build2) ... 2023-05-22T15:25:32.440 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up libnm0:amd64 (1.36.6-0ubuntu2) ... 2023-05-22T15:25:32.528 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up libnm0:amd64 (1.36.6-0ubuntu2) ... 2023-05-22T15:25:32.541 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up ppp (2.4.9-1+1ubuntu3) ... 2023-05-22T15:25:32.797 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up ppp (2.4.9-1+1ubuntu3) ... 2023-05-22T15:25:33.146 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up libllvm14:amd64 (1:14.0.0-1ubuntu1) ... 2023-05-22T15:25:33.281 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up dump (0.4b47-3) ... 2023-05-22T15:25:33.374 INFO:teuthology.orchestra.run.smithi016.stdout:update-alternatives: using /usr/sbin/rmt-dump to provide /usr/sbin/rmt (rmt) in auto mode 2023-05-22T15:25:33.440 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up libmm-glib0:amd64 (1.20.0-1~ubuntu22.04.1) ... 2023-05-22T15:25:33.541 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up libbluetooth3:amd64 (5.64-0ubuntu1) ... 2023-05-22T15:25:33.642 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up tcl (8.6.11+1build2) ... 2023-05-22T15:25:33.760 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up libjson-perl (4.04000-1) ... 2023-05-22T15:25:33.886 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up bison (2:3.8.2+dfsg-1build1) ... 2023-05-22T15:25:33.903 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up libllvm14:amd64 (1:14.0.0-1ubuntu1) ... 2023-05-22T15:25:33.977 INFO:teuthology.orchestra.run.smithi016.stdout:update-alternatives: using /usr/bin/bison.yacc to provide /usr/bin/yacc (yacc) in auto mode 2023-05-22T15:25:34.029 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up dump (0.4b47-3) ... 2023-05-22T15:25:34.046 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up indent (2.2.12-1build2) ... 2023-05-22T15:25:34.124 INFO:teuthology.orchestra.run.smithi062.stdout:update-alternatives: using /usr/sbin/rmt-dump to provide /usr/sbin/rmt (rmt) in auto mode 2023-05-22T15:25:34.172 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up usb-modeswitch (2.6.1-3ubuntu2) ... 2023-05-22T15:25:34.197 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up libmm-glib0:amd64 (1.20.0-1~ubuntu22.04.1) ... 2023-05-22T15:25:34.298 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up libbluetooth3:amd64 (5.64-0ubuntu1) ... 2023-05-22T15:25:34.349 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up libndp0:amd64 (1.8-0ubuntu3) ... 2023-05-22T15:25:34.416 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up tcl (8.6.11+1build2) ... 2023-05-22T15:25:34.458 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up libmbim-glib4:amd64 (1.28.0-1~ubuntu20.04.1) ... 2023-05-22T15:25:34.550 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up libjson-perl (4.04000-1) ... 2023-05-22T15:25:34.576 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up pptp-linux (1.10.0-1build3) ... 2023-05-22T15:25:34.676 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up bison (2:3.8.2+dfsg-1build1) ... 2023-05-22T15:25:34.719 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up postgresql-client (14+238) ... 2023-05-22T15:25:34.766 INFO:teuthology.orchestra.run.smithi062.stdout:update-alternatives: using /usr/bin/bison.yacc to provide /usr/bin/yacc (yacc) in auto mode 2023-05-22T15:25:34.828 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up indent (2.2.12-1build2) ... 2023-05-22T15:25:34.844 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up network-manager (1.36.6-0ubuntu2) ... 2023-05-22T15:25:34.954 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up usb-modeswitch (2.6.1-3ubuntu2) ... 2023-05-22T15:25:35.122 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up libndp0:amd64 (1.8-0ubuntu3) ... 2023-05-22T15:25:35.231 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up libmbim-glib4:amd64 (1.28.0-1~ubuntu20.04.1) ... 2023-05-22T15:25:35.327 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:25:35.327 INFO:teuthology.orchestra.run.smithi016.stdout:The following network interfaces were found in /etc/network/interfaces 2023-05-22T15:25:35.328 INFO:teuthology.orchestra.run.smithi016.stdout:which means they are currently configured by ifupdown: 2023-05-22T15:25:35.328 INFO:teuthology.orchestra.run.smithi016.stdout:- ens1f0 2023-05-22T15:25:35.328 INFO:teuthology.orchestra.run.smithi016.stdout:If you want to manage those interfaces with NetworkManager instead 2023-05-22T15:25:35.328 INFO:teuthology.orchestra.run.smithi016.stdout:remove their configuration from /etc/network/interfaces. 2023-05-22T15:25:35.328 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:25:35.341 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up pptp-linux (1.10.0-1build3) ... 2023-05-22T15:25:35.442 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.nm-dispatcher.service -> /lib/systemd/system/NetworkManager-dispatcher.service. 2023-05-22T15:25:35.509 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up postgresql-client (14+238) ... 2023-05-22T15:25:35.635 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up network-manager (1.36.6-0ubuntu2) ... 2023-05-22T15:25:35.766 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/network-online.target.wants/NetworkManager-wait-online.service -> /lib/systemd/system/NetworkManager-wait-online.service. 2023-05-22T15:25:36.111 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/NetworkManager.service -> /lib/systemd/system/NetworkManager.service. 2023-05-22T15:25:36.135 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-05-22T15:25:36.135 INFO:teuthology.orchestra.run.smithi062.stdout:The following network interfaces were found in /etc/network/interfaces 2023-05-22T15:25:36.135 INFO:teuthology.orchestra.run.smithi062.stdout:which means they are currently configured by ifupdown: 2023-05-22T15:25:36.135 INFO:teuthology.orchestra.run.smithi062.stdout:- enp3s0f1 2023-05-22T15:25:36.135 INFO:teuthology.orchestra.run.smithi062.stdout:If you want to manage those interfaces with NetworkManager instead 2023-05-22T15:25:36.135 INFO:teuthology.orchestra.run.smithi062.stdout:remove their configuration from /etc/network/interfaces. 2023-05-22T15:25:36.136 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-05-22T15:25:36.259 INFO:teuthology.orchestra.run.smithi062.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.nm-dispatcher.service -> /lib/systemd/system/NetworkManager-dispatcher.service. 2023-05-22T15:25:36.613 INFO:teuthology.orchestra.run.smithi062.stdout:Created symlink /etc/systemd/system/network-online.target.wants/NetworkManager-wait-online.service -> /lib/systemd/system/NetworkManager-wait-online.service. 2023-05-22T15:25:36.639 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up libmbim-proxy (1.28.0-1~ubuntu20.04.1) ... 2023-05-22T15:25:36.782 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up libfl-dev:amd64 (2.6.4-8build2) ... 2023-05-22T15:25:36.892 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up network-manager-pptp (1.2.10-1) ... 2023-05-22T15:25:36.962 INFO:teuthology.orchestra.run.smithi062.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/NetworkManager.service -> /lib/systemd/system/NetworkManager.service. 2023-05-22T15:25:36.993 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up postgresql-common (238) ... 2023-05-22T15:25:37.563 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up libmbim-proxy (1.28.0-1~ubuntu20.04.1) ... 2023-05-22T15:25:37.706 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up libfl-dev:amd64 (2.6.4-8build2) ... 2023-05-22T15:25:37.826 INFO:teuthology.orchestra.run.smithi016.stdout:Adding user postgres to group ssl-cert 2023-05-22T15:25:37.832 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up network-manager-pptp (1.2.10-1) ... 2023-05-22T15:25:37.958 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up postgresql-common (238) ... 2023-05-22T15:25:38.026 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:25:38.026 INFO:teuthology.orchestra.run.smithi016.stdout:Creating config file /etc/postgresql-common/createcluster.conf with new version 2023-05-22T15:25:38.210 INFO:teuthology.orchestra.run.smithi016.stdout:Building PostgreSQL dictionaries from installed myspell/hunspell packages... 2023-05-22T15:25:38.211 INFO:teuthology.orchestra.run.smithi016.stdout:Removing obsolete dictionary files: 2023-05-22T15:25:38.789 INFO:teuthology.orchestra.run.smithi062.stdout:Adding user postgres to group ssl-cert 2023-05-22T15:25:38.814 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/postgresql.service -> /lib/systemd/system/postgresql.service. 2023-05-22T15:25:39.052 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-05-22T15:25:39.052 INFO:teuthology.orchestra.run.smithi062.stdout:Creating config file /etc/postgresql-common/createcluster.conf with new version 2023-05-22T15:25:39.227 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up postgresql-14 (14.7-0ubuntu0.22.04.1) ... 2023-05-22T15:25:39.279 INFO:teuthology.orchestra.run.smithi062.stdout:Building PostgreSQL dictionaries from installed myspell/hunspell packages... 2023-05-22T15:25:39.280 INFO:teuthology.orchestra.run.smithi062.stdout:Removing obsolete dictionary files: 2023-05-22T15:25:39.576 INFO:teuthology.orchestra.run.smithi016.stdout:Creating new PostgreSQL cluster 14/main ... 2023-05-22T15:25:39.590 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/lib/postgresql/14/bin/initdb -D /var/lib/postgresql/14/main --auth-local peer --auth-host scram-sha-256 --no-instructions 2023-05-22T15:25:39.632 INFO:teuthology.orchestra.run.smithi016.stdout:The files belonging to this database system will be owned by user "postgres". 2023-05-22T15:25:39.632 INFO:teuthology.orchestra.run.smithi016.stdout:This user must also own the server process. 2023-05-22T15:25:39.632 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:25:39.632 INFO:teuthology.orchestra.run.smithi016.stdout:The database cluster will be initialized with locale "en_US". 2023-05-22T15:25:39.632 INFO:teuthology.orchestra.run.smithi016.stdout:The default database encoding has accordingly been set to "LATIN1". 2023-05-22T15:25:39.633 INFO:teuthology.orchestra.run.smithi016.stdout:The default text search configuration will be set to "english". 2023-05-22T15:25:39.633 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:25:39.633 INFO:teuthology.orchestra.run.smithi016.stdout:Data page checksums are disabled. 2023-05-22T15:25:39.633 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:25:39.633 INFO:teuthology.orchestra.run.smithi016.stdout:fixing permissions on existing directory /var/lib/postgresql/14/main ... ok 2023-05-22T15:25:39.634 INFO:teuthology.orchestra.run.smithi016.stdout:creating subdirectories ... ok 2023-05-22T15:25:39.634 INFO:teuthology.orchestra.run.smithi016.stdout:selecting dynamic shared memory implementation ... posix 2023-05-22T15:25:39.645 INFO:teuthology.orchestra.run.smithi016.stdout:selecting default max_connections ... 100 2023-05-22T15:25:39.663 INFO:teuthology.orchestra.run.smithi016.stdout:selecting default shared_buffers ... 128MB 2023-05-22T15:25:39.672 INFO:teuthology.orchestra.run.smithi016.stdout:selecting default time zone ... Etc/UTC 2023-05-22T15:25:39.673 INFO:teuthology.orchestra.run.smithi016.stdout:creating configuration files ... ok 2023-05-22T15:25:39.786 INFO:teuthology.orchestra.run.smithi016.stdout:running bootstrap script ... ok 2023-05-22T15:25:39.884 INFO:teuthology.orchestra.run.smithi062.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/postgresql.service -> /lib/systemd/system/postgresql.service. 2023-05-22T15:25:40.286 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up postgresql-14 (14.7-0ubuntu0.22.04.1) ... 2023-05-22T15:25:40.438 INFO:teuthology.orchestra.run.smithi016.stdout:performing post-bootstrap initialization ... ok 2023-05-22T15:25:40.620 INFO:teuthology.orchestra.run.smithi062.stdout:Creating new PostgreSQL cluster 14/main ... 2023-05-22T15:25:40.637 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/lib/postgresql/14/bin/initdb -D /var/lib/postgresql/14/main --auth-local peer --auth-host scram-sha-256 --no-instructions 2023-05-22T15:25:40.703 INFO:teuthology.orchestra.run.smithi062.stdout:The files belonging to this database system will be owned by user "postgres". 2023-05-22T15:25:40.704 INFO:teuthology.orchestra.run.smithi062.stdout:This user must also own the server process. 2023-05-22T15:25:40.704 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-05-22T15:25:40.704 INFO:teuthology.orchestra.run.smithi062.stdout:The database cluster will be initialized with locale "en_US". 2023-05-22T15:25:40.704 INFO:teuthology.orchestra.run.smithi062.stdout:The default database encoding has accordingly been set to "LATIN1". 2023-05-22T15:25:40.704 INFO:teuthology.orchestra.run.smithi062.stdout:The default text search configuration will be set to "english". 2023-05-22T15:25:40.704 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-05-22T15:25:40.705 INFO:teuthology.orchestra.run.smithi062.stdout:Data page checksums are disabled. 2023-05-22T15:25:40.705 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-05-22T15:25:40.705 INFO:teuthology.orchestra.run.smithi062.stdout:fixing permissions on existing directory /var/lib/postgresql/14/main ... ok 2023-05-22T15:25:40.705 INFO:teuthology.orchestra.run.smithi062.stdout:creating subdirectories ... ok 2023-05-22T15:25:40.705 INFO:teuthology.orchestra.run.smithi062.stdout:selecting dynamic shared memory implementation ... posix 2023-05-22T15:25:40.717 INFO:teuthology.orchestra.run.smithi062.stdout:selecting default max_connections ... 100 2023-05-22T15:25:40.739 INFO:teuthology.orchestra.run.smithi062.stdout:selecting default shared_buffers ... 128MB 2023-05-22T15:25:40.749 INFO:teuthology.orchestra.run.smithi062.stdout:selecting default time zone ... Etc/UTC 2023-05-22T15:25:40.750 INFO:teuthology.orchestra.run.smithi062.stdout:creating configuration files ... ok 2023-05-22T15:25:40.867 INFO:teuthology.orchestra.run.smithi062.stdout:running bootstrap script ... ok 2023-05-22T15:25:41.104 INFO:teuthology.orchestra.run.smithi016.stdout:syncing data to disk ... ok 2023-05-22T15:25:41.520 INFO:teuthology.orchestra.run.smithi016.stdout:update-alternatives: using /usr/share/postgresql/14/man/man1/postmaster.1.gz to provide /usr/share/man/man1/postmaster.1.gz (postmaster.1.gz) in auto mode 2023-05-22T15:25:41.542 INFO:teuthology.orchestra.run.smithi062.stdout:performing post-bootstrap initialization ... ok 2023-05-22T15:25:42.179 INFO:teuthology.orchestra.run.smithi062.stdout:syncing data to disk ... ok 2023-05-22T15:25:42.593 INFO:teuthology.orchestra.run.smithi062.stdout:update-alternatives: using /usr/share/postgresql/14/man/man1/postmaster.1.gz to provide /usr/share/man/man1/postmaster.1.gz (postmaster.1.gz) in auto mode 2023-05-22T15:25:44.359 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up libqmi-glib5:amd64 (1.32.0-1ubuntu0.22.04.1) ... 2023-05-22T15:25:44.477 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up postgresql-contrib (14+238) ... 2023-05-22T15:25:44.603 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up postgresql (14+238) ... 2023-05-22T15:25:44.729 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up libqmi-proxy (1.32.0-1ubuntu0.22.04.1) ... 2023-05-22T15:25:44.854 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up modemmanager (1.20.0-1~ubuntu22.04.1) ... 2023-05-22T15:25:45.116 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.ModemManager1.service -> /lib/systemd/system/ModemManager.service. 2023-05-22T15:25:45.117 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ModemManager.service -> /lib/systemd/system/ModemManager.service. 2023-05-22T15:25:45.409 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up libqmi-glib5:amd64 (1.32.0-1ubuntu0.22.04.1) ... 2023-05-22T15:25:45.535 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up postgresql-contrib (14+238) ... 2023-05-22T15:25:45.674 INFO:teuthology.orchestra.run.smithi016.stdout:Processing triggers for hicolor-icon-theme (0.17-2) ... 2023-05-22T15:25:45.677 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up postgresql (14+238) ... 2023-05-22T15:25:45.804 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up libqmi-proxy (1.32.0-1ubuntu0.22.04.1) ... 2023-05-22T15:25:45.921 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up modemmanager (1.20.0-1~ubuntu22.04.1) ... 2023-05-22T15:25:46.018 INFO:teuthology.orchestra.run.smithi016.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2023-05-22T15:25:46.161 INFO:teuthology.orchestra.run.smithi016.stdout:Processing triggers for man-db (2.10.2-1) ... 2023-05-22T15:25:46.164 INFO:teuthology.orchestra.run.smithi062.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.ModemManager1.service -> /lib/systemd/system/ModemManager.service. 2023-05-22T15:25:46.164 INFO:teuthology.orchestra.run.smithi062.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ModemManager.service -> /lib/systemd/system/ModemManager.service. 2023-05-22T15:25:46.749 INFO:teuthology.orchestra.run.smithi062.stdout:Processing triggers for hicolor-icon-theme (0.17-2) ... 2023-05-22T15:25:47.060 INFO:teuthology.orchestra.run.smithi062.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2023-05-22T15:25:47.219 INFO:teuthology.orchestra.run.smithi062.stdout:Processing triggers for man-db (2.10.2-1) ... 2023-05-22T15:25:47.473 INFO:teuthology.orchestra.run.smithi016.stdout:Processing triggers for dbus (1.12.20-2ubuntu4.1) ... 2023-05-22T15:25:47.598 INFO:teuthology.orchestra.run.smithi016.stdout:Processing triggers for install-info (6.8-4build1) ... 2023-05-22T15:25:48.262 INFO:teuthology.orchestra.run.smithi016.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2023-05-22T15:25:48.264 DEBUG:teuthology.parallel:result is None 2023-05-22T15:25:48.304 INFO:teuthology.orchestra.run.smithi062.stdout:Processing triggers for dbus (1.12.20-2ubuntu4.1) ... 2023-05-22T15:25:48.404 INFO:teuthology.orchestra.run.smithi062.stdout:Processing triggers for install-info (6.8-4build1) ... 2023-05-22T15:25:49.068 INFO:teuthology.orchestra.run.smithi062.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2023-05-22T15:25:49.070 DEBUG:teuthology.parallel:result is None 2023-05-22T15:25:49.071 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=9c9ba4df22024d9d8d6cf9f98a4b96751fc78d0c 2023-05-22T15:25:49.210 DEBUG:teuthology.orchestra.run.smithi016:> dpkg-query -W -f '${Version}' ceph 2023-05-22T15:25:49.230 INFO:teuthology.orchestra.run.smithi016.stdout:18.0.0-3192-g9c9ba4df-1jammy 2023-05-22T15:25:49.231 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-3192-g9c9ba4df-1jammy 2023-05-22T15:25:49.231 INFO:teuthology.task.install:The correct ceph version 18.0.0-3192-g9c9ba4df-1jammy is installed. 2023-05-22T15:25:49.232 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=9c9ba4df22024d9d8d6cf9f98a4b96751fc78d0c 2023-05-22T15:25:49.371 DEBUG:teuthology.orchestra.run.smithi062:> dpkg-query -W -f '${Version}' ceph 2023-05-22T15:25:49.391 INFO:teuthology.orchestra.run.smithi062.stdout:18.0.0-3192-g9c9ba4df-1jammy 2023-05-22T15:25:49.392 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-3192-g9c9ba4df-1jammy 2023-05-22T15:25:49.393 INFO:teuthology.task.install:The correct ceph version 18.0.0-3192-g9c9ba4df-1jammy is installed. 2023-05-22T15:25:49.393 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-05-22T15:25:49.394 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-05-22T15:25:49.394 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-05-22T15:25:49.409 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2023-05-22T15:25:49.410 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-05-22T15:25:49.447 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-05-22T15:25:49.448 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-05-22T15:25:49.448 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd of=/usr/bin/daemon-helper 2023-05-22T15:25:49.464 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-05-22T15:25:49.520 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2023-05-22T15:25:49.521 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd of=/usr/bin/daemon-helper 2023-05-22T15:25:49.535 DEBUG:teuthology.orchestra.run.smithi062:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-05-22T15:25:49.590 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-05-22T15:25:49.590 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-05-22T15:25:49.591 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd of=/usr/bin/adjust-ulimits 2023-05-22T15:25:49.604 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-05-22T15:25:49.663 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2023-05-22T15:25:49.663 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd of=/usr/bin/adjust-ulimits 2023-05-22T15:25:49.676 DEBUG:teuthology.orchestra.run.smithi062:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-05-22T15:25:49.731 INFO:teuthology.run_tasks:Running task ceph... 2023-05-22T15:25:49.810 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2023-05-22T15:25:49.811 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod 777 /var/log/ceph 2023-05-22T15:25:49.814 DEBUG:teuthology.orchestra.run.smithi062:> sudo chmod 777 /var/log/ceph 2023-05-22T15:25:49.827 INFO:tasks.ceph:Disabling ceph logrotate... 2023-05-22T15:25:49.828 DEBUG:teuthology.orchestra.run.smithi016:> sudo rm -f -- /etc/logrotate.d/ceph 2023-05-22T15:25:49.869 DEBUG:teuthology.orchestra.run.smithi062:> sudo rm -f -- /etc/logrotate.d/ceph 2023-05-22T15:25:49.882 INFO:tasks.ceph:Creating extra log directories... 2023-05-22T15:25:49.882 DEBUG:teuthology.orchestra.run.smithi016:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-05-22T15:25:49.925 DEBUG:teuthology.orchestra.run.smithi062:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-05-22T15:25:49.945 INFO:tasks.ceph:Setting up log rotation with {'ceph-mds': '10G', 'ceph-osd': '10G'} 2023-05-22T15:25:49.945 INFO:tasks.ceph:writing logrotate stanza for ceph-mds 2023-05-22T15:25:49.946 INFO:tasks.ceph:writing logrotate stanza for ceph-osd 2023-05-22T15:25:49.946 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-05-22T15:25:49.946 DEBUG:teuthology.orchestra.run.smithi016:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2023-05-22T15:25:49.987 DEBUG:teuthology.orchestra.run.smithi016:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2023-05-22T15:25:50.066 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2023-05-22T15:25:50.066 DEBUG:teuthology.orchestra.run.smithi062:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2023-05-22T15:25:50.072 DEBUG:teuthology.orchestra.run.smithi062:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2023-05-22T15:25:50.155 INFO:tasks.ceph:Creating ceph cluster ceph... 2023-05-22T15:25:50.155 INFO:tasks.ceph:config {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug finisher': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-05-22T15:25:50.156 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yuriw-2023-05-22_14:44:12-fs-wip-yuri3-testing-2023-05-21-0740-reef-distro-default-smithi/7282496', 'branch': 'wip-yuri3-testing-2023-05-21-0740-reef', 'description': 'fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snap-schedule}', 'email': 'yweinste@redhat.com', 'first_in_suite': False, 'job_id': '7282496', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'log-rotate': {'ceph-mds': '10G', 'ceph-osd': '10G'}, 'machine_type': 'smithi', 'name': 'yuriw-2023-05-22_14:44:12-fs-wip-yuri3-testing-2023-05-21-0740-reef-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 20}}, {'machine': {'disk': 200}}], 'os_type': 'ubuntu', 'os_version': '22.04', 'overrides': {'admin_socket': {'branch': 'wip-yuri3-testing-2023-05-21-0740-reef'}, 'ceph': {'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd']}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug finisher': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'log-whitelist': ['OSD full dropping all updates', 'OSD near full', 'pausewr flag', 'failsafe engaged, dropping updates', 'failsafe disengaged, no longer dropping', 'is full \\(reached quota', 'POOL_FULL', 'POOL_BACKFILLFULL'], 'sha1': '9c9ba4df22024d9d8d6cf9f98a4b96751fc78d0c'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': '9c9ba4df22024d9d8d6cf9f98a4b96751fc78d0c'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-yuri3-testing-2023-05-21-0740-reef', 'sha1': '9c9ba4df22024d9d8d6cf9f98a4b96751fc78d0c'}}, 'owner': 'scheduled_yuriw@teuthology', 'priority': 75, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'], ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3']], 'seed': 1398, 'sha1': '9c9ba4df22024d9d8d6cf9f98a4b96751fc78d0c', 'sleep_before_teardown': 0, 'subset': '111/12000', 'suite': 'fs', 'suite_branch': 'wip-yuri3-testing-2023-05-21-0740-reef', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_9c9ba4df22024d9d8d6cf9f98a4b96751fc78d0c/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': '9c9ba4df22024d9d8d6cf9f98a4b96751fc78d0c', 'syslog': {'ignorelist': ['WARNING*.*check_session_state', 'WARNING*.*__ceph_remove_cap']}, 'targets': {'smithi016.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLlVQ8mnKu7mabeHXD6q1SCUofl2/7TQGJtZoDkWRaEN8SjRlNYMe5o9bDtiZPZ2Z7+Mb61B2A6l71rddBvoLjg=', 'smithi062.front.sepia.ceph.com': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIA1xuKgcCm2/BCZTETfSU7EAds5jWkUYadR3nrazn66o'}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '9c9ba4df22024d9d8d6cf9f98a4b96751fc78d0c'}}, {'ceph': None}, {'ceph-fuse': None}, {'cephfs_test_runner': {'modules': ['tasks.cephfs.test_snap_schedules']}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '01abd1ed90624b09fd81d9cb3a79e2834351f0ca', 'timestamp': '2023-05-22_14:44:12', 'tube': 'smithi', 'user': 'yuriw', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.23459'} 2023-05-22T15:25:50.157 DEBUG:teuthology.orchestra.run.smithi016:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-05-22T15:25:50.159 DEBUG:teuthology.orchestra.run.smithi062:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-05-22T15:25:50.195 DEBUG:teuthology.orchestra.run.smithi016:> sudo install -d -m0777 -- /var/run/ceph 2023-05-22T15:25:50.204 DEBUG:teuthology.orchestra.run.smithi062:> sudo install -d -m0777 -- /var/run/ceph 2023-05-22T15:25:50.251 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-05-22T15:25:50.251 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/scratch_devs of=/dev/stdout 2023-05-22T15:25:50.263 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-05-22T15:25:50.263 DEBUG:teuthology.orchestra.run.smithi016:> stat /dev/vg_nvme/lv_1 2023-05-22T15:25:50.311 INFO:teuthology.orchestra.run.smithi016.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-05-22T15:25:50.311 INFO:teuthology.orchestra.run.smithi016.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-05-22T15:25:50.312 INFO:teuthology.orchestra.run.smithi016.stdout:Device: 5h/5d Inode: 879 Links: 1 2023-05-22T15:25:50.312 INFO:teuthology.orchestra.run.smithi016.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-05-22T15:25:50.312 INFO:teuthology.orchestra.run.smithi016.stdout:Access: 2023-05-22 15:19:46.503894817 +0000 2023-05-22T15:25:50.312 INFO:teuthology.orchestra.run.smithi016.stdout:Modify: 2023-05-22 15:19:45.967899607 +0000 2023-05-22T15:25:50.312 INFO:teuthology.orchestra.run.smithi016.stdout:Change: 2023-05-22 15:19:45.967899607 +0000 2023-05-22T15:25:50.312 INFO:teuthology.orchestra.run.smithi016.stdout: Birth: - 2023-05-22T15:25:50.313 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-05-22T15:25:50.366 INFO:teuthology.orchestra.run.smithi016.stderr:1+0 records in 2023-05-22T15:25:50.366 INFO:teuthology.orchestra.run.smithi016.stderr:1+0 records out 2023-05-22T15:25:50.366 INFO:teuthology.orchestra.run.smithi016.stderr:512 bytes copied, 0.000341681 s, 1.5 MB/s 2023-05-22T15:25:50.367 DEBUG:teuthology.orchestra.run.smithi016:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-05-22T15:25:50.417 DEBUG:teuthology.orchestra.run.smithi016:> stat /dev/vg_nvme/lv_2 2023-05-22T15:25:50.464 INFO:teuthology.orchestra.run.smithi016.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-05-22T15:25:50.464 INFO:teuthology.orchestra.run.smithi016.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-05-22T15:25:50.464 INFO:teuthology.orchestra.run.smithi016.stdout:Device: 5h/5d Inode: 895 Links: 1 2023-05-22T15:25:50.465 INFO:teuthology.orchestra.run.smithi016.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-05-22T15:25:50.465 INFO:teuthology.orchestra.run.smithi016.stdout:Access: 2023-05-22 15:19:47.111889387 +0000 2023-05-22T15:25:50.465 INFO:teuthology.orchestra.run.smithi016.stdout:Modify: 2023-05-22 15:19:46.723892852 +0000 2023-05-22T15:25:50.465 INFO:teuthology.orchestra.run.smithi016.stdout:Change: 2023-05-22 15:19:46.723892852 +0000 2023-05-22T15:25:50.465 INFO:teuthology.orchestra.run.smithi016.stdout: Birth: - 2023-05-22T15:25:50.465 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-05-22T15:25:50.522 INFO:teuthology.orchestra.run.smithi016.stderr:1+0 records in 2023-05-22T15:25:50.522 INFO:teuthology.orchestra.run.smithi016.stderr:1+0 records out 2023-05-22T15:25:50.522 INFO:teuthology.orchestra.run.smithi016.stderr:512 bytes copied, 0.000304951 s, 1.7 MB/s 2023-05-22T15:25:50.522 DEBUG:teuthology.orchestra.run.smithi016:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-05-22T15:25:50.569 DEBUG:teuthology.orchestra.run.smithi016:> stat /dev/vg_nvme/lv_3 2023-05-22T15:25:50.616 INFO:teuthology.orchestra.run.smithi016.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-05-22T15:25:50.616 INFO:teuthology.orchestra.run.smithi016.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-05-22T15:25:50.616 INFO:teuthology.orchestra.run.smithi016.stdout:Device: 5h/5d Inode: 907 Links: 1 2023-05-22T15:25:50.616 INFO:teuthology.orchestra.run.smithi016.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-05-22T15:25:50.617 INFO:teuthology.orchestra.run.smithi016.stdout:Access: 2023-05-22 15:19:47.363887137 +0000 2023-05-22T15:25:50.617 INFO:teuthology.orchestra.run.smithi016.stdout:Modify: 2023-05-22 15:19:47.359887172 +0000 2023-05-22T15:25:50.617 INFO:teuthology.orchestra.run.smithi016.stdout:Change: 2023-05-22 15:19:47.359887172 +0000 2023-05-22T15:25:50.617 INFO:teuthology.orchestra.run.smithi016.stdout: Birth: - 2023-05-22T15:25:50.617 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-05-22T15:25:50.670 INFO:teuthology.orchestra.run.smithi016.stderr:1+0 records in 2023-05-22T15:25:50.670 INFO:teuthology.orchestra.run.smithi016.stderr:1+0 records out 2023-05-22T15:25:50.670 INFO:teuthology.orchestra.run.smithi016.stderr:512 bytes copied, 0.000221402 s, 2.3 MB/s 2023-05-22T15:25:50.671 DEBUG:teuthology.orchestra.run.smithi016:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-05-22T15:25:50.721 DEBUG:teuthology.orchestra.run.smithi016:> stat /dev/vg_nvme/lv_4 2023-05-22T15:25:50.768 INFO:teuthology.orchestra.run.smithi016.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-05-22T15:25:50.768 INFO:teuthology.orchestra.run.smithi016.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-05-22T15:25:50.768 INFO:teuthology.orchestra.run.smithi016.stdout:Device: 5h/5d Inode: 925 Links: 1 2023-05-22T15:25:50.769 INFO:teuthology.orchestra.run.smithi016.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-05-22T15:25:50.769 INFO:teuthology.orchestra.run.smithi016.stdout:Access: 2023-05-22 15:19:48.347878347 +0000 2023-05-22T15:25:50.769 INFO:teuthology.orchestra.run.smithi016.stdout:Modify: 2023-05-22 15:19:47.927882099 +0000 2023-05-22T15:25:50.769 INFO:teuthology.orchestra.run.smithi016.stdout:Change: 2023-05-22 15:19:47.927882099 +0000 2023-05-22T15:25:50.770 INFO:teuthology.orchestra.run.smithi016.stdout: Birth: - 2023-05-22T15:25:50.770 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-05-22T15:25:50.822 INFO:teuthology.orchestra.run.smithi016.stderr:1+0 records in 2023-05-22T15:25:50.822 INFO:teuthology.orchestra.run.smithi016.stderr:1+0 records out 2023-05-22T15:25:50.823 INFO:teuthology.orchestra.run.smithi016.stderr:512 bytes copied, 0.00032352 s, 1.6 MB/s 2023-05-22T15:25:50.824 DEBUG:teuthology.orchestra.run.smithi016:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-05-22T15:25:50.873 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-05-22T15:25:50.874 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2023-05-22T15:25:50.874 DEBUG:teuthology.orchestra.run.smithi062:> dd if=/scratch_devs of=/dev/stdout 2023-05-22T15:25:50.880 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-05-22T15:25:50.880 DEBUG:teuthology.orchestra.run.smithi062:> stat /dev/vg_nvme/lv_1 2023-05-22T15:25:50.927 INFO:teuthology.orchestra.run.smithi062.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-05-22T15:25:50.927 INFO:teuthology.orchestra.run.smithi062.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-05-22T15:25:50.928 INFO:teuthology.orchestra.run.smithi062.stdout:Device: 5h/5d Inode: 878 Links: 1 2023-05-22T15:25:50.928 INFO:teuthology.orchestra.run.smithi062.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-05-22T15:25:50.928 INFO:teuthology.orchestra.run.smithi062.stdout:Access: 2023-05-22 15:19:48.785568076 +0000 2023-05-22T15:25:50.928 INFO:teuthology.orchestra.run.smithi062.stdout:Modify: 2023-05-22 15:19:48.401571504 +0000 2023-05-22T15:25:50.928 INFO:teuthology.orchestra.run.smithi062.stdout:Change: 2023-05-22 15:19:48.401571504 +0000 2023-05-22T15:25:50.929 INFO:teuthology.orchestra.run.smithi062.stdout: Birth: - 2023-05-22T15:25:50.929 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-05-22T15:25:50.982 INFO:teuthology.orchestra.run.smithi062.stderr:1+0 records in 2023-05-22T15:25:50.982 INFO:teuthology.orchestra.run.smithi062.stderr:1+0 records out 2023-05-22T15:25:50.982 INFO:teuthology.orchestra.run.smithi062.stderr:512 bytes copied, 0.00044253 s, 1.2 MB/s 2023-05-22T15:25:50.983 DEBUG:teuthology.orchestra.run.smithi062:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-05-22T15:25:51.033 DEBUG:teuthology.orchestra.run.smithi062:> stat /dev/vg_nvme/lv_2 2023-05-22T15:25:51.079 INFO:teuthology.orchestra.run.smithi062.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-05-22T15:25:51.079 INFO:teuthology.orchestra.run.smithi062.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-05-22T15:25:51.080 INFO:teuthology.orchestra.run.smithi062.stdout:Device: 5h/5d Inode: 896 Links: 1 2023-05-22T15:25:51.080 INFO:teuthology.orchestra.run.smithi062.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-05-22T15:25:51.080 INFO:teuthology.orchestra.run.smithi062.stdout:Access: 2023-05-22 15:19:49.429562331 +0000 2023-05-22T15:25:51.080 INFO:teuthology.orchestra.run.smithi062.stdout:Modify: 2023-05-22 15:19:49.049565721 +0000 2023-05-22T15:25:51.080 INFO:teuthology.orchestra.run.smithi062.stdout:Change: 2023-05-22 15:19:49.049565721 +0000 2023-05-22T15:25:51.080 INFO:teuthology.orchestra.run.smithi062.stdout: Birth: - 2023-05-22T15:25:51.081 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-05-22T15:25:51.133 INFO:teuthology.orchestra.run.smithi062.stderr:1+0 records in 2023-05-22T15:25:51.134 INFO:teuthology.orchestra.run.smithi062.stderr:1+0 records out 2023-05-22T15:25:51.134 INFO:teuthology.orchestra.run.smithi062.stderr:512 bytes copied, 0.000311124 s, 1.6 MB/s 2023-05-22T15:25:51.135 DEBUG:teuthology.orchestra.run.smithi062:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-05-22T15:25:51.184 DEBUG:teuthology.orchestra.run.smithi062:> stat /dev/vg_nvme/lv_3 2023-05-22T15:25:51.231 INFO:teuthology.orchestra.run.smithi062.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-05-22T15:25:51.232 INFO:teuthology.orchestra.run.smithi062.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-05-22T15:25:51.232 INFO:teuthology.orchestra.run.smithi062.stdout:Device: 5h/5d Inode: 918 Links: 1 2023-05-22T15:25:51.232 INFO:teuthology.orchestra.run.smithi062.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-05-22T15:25:51.233 INFO:teuthology.orchestra.run.smithi062.stdout:Access: 2023-05-22 15:19:50.085556478 +0000 2023-05-22T15:25:51.233 INFO:teuthology.orchestra.run.smithi062.stdout:Modify: 2023-05-22 15:19:49.693559976 +0000 2023-05-22T15:25:51.233 INFO:teuthology.orchestra.run.smithi062.stdout:Change: 2023-05-22 15:19:49.693559976 +0000 2023-05-22T15:25:51.233 INFO:teuthology.orchestra.run.smithi062.stdout: Birth: - 2023-05-22T15:25:51.234 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-05-22T15:25:51.286 INFO:teuthology.orchestra.run.smithi062.stderr:1+0 records in 2023-05-22T15:25:51.286 INFO:teuthology.orchestra.run.smithi062.stderr:1+0 records out 2023-05-22T15:25:51.286 INFO:teuthology.orchestra.run.smithi062.stderr:512 bytes copied, 0.000407927 s, 1.3 MB/s 2023-05-22T15:25:51.287 DEBUG:teuthology.orchestra.run.smithi062:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-05-22T15:25:51.337 DEBUG:teuthology.orchestra.run.smithi062:> stat /dev/vg_nvme/lv_4 2023-05-22T15:25:51.383 INFO:teuthology.orchestra.run.smithi062.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-05-22T15:25:51.384 INFO:teuthology.orchestra.run.smithi062.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-05-22T15:25:51.384 INFO:teuthology.orchestra.run.smithi062.stdout:Device: 5h/5d Inode: 933 Links: 1 2023-05-22T15:25:51.384 INFO:teuthology.orchestra.run.smithi062.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-05-22T15:25:51.385 INFO:teuthology.orchestra.run.smithi062.stdout:Access: 2023-05-22 15:19:50.709550911 +0000 2023-05-22T15:25:51.385 INFO:teuthology.orchestra.run.smithi062.stdout:Modify: 2023-05-22 15:19:50.321554373 +0000 2023-05-22T15:25:51.385 INFO:teuthology.orchestra.run.smithi062.stdout:Change: 2023-05-22 15:19:50.321554373 +0000 2023-05-22T15:25:51.385 INFO:teuthology.orchestra.run.smithi062.stdout: Birth: - 2023-05-22T15:25:51.386 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-05-22T15:25:51.438 INFO:teuthology.orchestra.run.smithi062.stderr:1+0 records in 2023-05-22T15:25:51.438 INFO:teuthology.orchestra.run.smithi062.stderr:1+0 records out 2023-05-22T15:25:51.438 INFO:teuthology.orchestra.run.smithi062.stderr:512 bytes copied, 0.000381414 s, 1.3 MB/s 2023-05-22T15:25:51.440 DEBUG:teuthology.orchestra.run.smithi062:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-05-22T15:25:51.488 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-05-22T15:25:51.488 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi016.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi062.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2023-05-22T15:25:51.489 INFO:tasks.ceph:Generating config... 2023-05-22T15:25:51.490 INFO:tasks.ceph:[client] client mount timeout = 600 2023-05-22T15:25:51.490 INFO:tasks.ceph:[client] debug client = 20 2023-05-22T15:25:51.491 INFO:tasks.ceph:[client] debug ms = 1 2023-05-22T15:25:51.491 INFO:tasks.ceph:[client] pid file = 2023-05-22T15:25:51.491 INFO:tasks.ceph:[client] rados mon op timeout = 900 2023-05-22T15:25:51.491 INFO:tasks.ceph:[client] rados osd op timeout = 900 2023-05-22T15:25:51.491 INFO:tasks.ceph:[mds] debug mds = 20 2023-05-22T15:25:51.492 INFO:tasks.ceph:[mds] debug mds balancer = 20 2023-05-22T15:25:51.492 INFO:tasks.ceph:[mds] debug ms = 1 2023-05-22T15:25:51.492 INFO:tasks.ceph:[mds] mds debug frag = True 2023-05-22T15:25:51.492 INFO:tasks.ceph:[mds] mds debug scatterstat = True 2023-05-22T15:25:51.492 INFO:tasks.ceph:[mds] mds op complaint time = 180 2023-05-22T15:25:51.493 INFO:tasks.ceph:[mds] mds verify scatter = True 2023-05-22T15:25:51.493 INFO:tasks.ceph:[mds] osd op complaint time = 180 2023-05-22T15:25:51.493 INFO:tasks.ceph:[mds] rados mon op timeout = 900 2023-05-22T15:25:51.493 INFO:tasks.ceph:[mds] rados osd op timeout = 900 2023-05-22T15:25:51.493 INFO:tasks.ceph:[mgr] debug client = 20 2023-05-22T15:25:51.494 INFO:tasks.ceph:[mgr] debug finisher = 20 2023-05-22T15:25:51.494 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-05-22T15:25:51.494 INFO:tasks.ceph:[mgr] debug ms = 1 2023-05-22T15:25:51.494 INFO:tasks.ceph:[mon] debug mon = 20 2023-05-22T15:25:51.494 INFO:tasks.ceph:[mon] debug ms = 1 2023-05-22T15:25:51.495 INFO:tasks.ceph:[mon] debug paxos = 20 2023-05-22T15:25:51.495 INFO:tasks.ceph:[mon] mon op complaint time = 120 2023-05-22T15:25:51.495 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2023-05-22T15:25:51.495 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2023-05-22T15:25:51.495 INFO:tasks.ceph:[osd] debug bluefs = 20 2023-05-22T15:25:51.496 INFO:tasks.ceph:[osd] debug bluestore = 20 2023-05-22T15:25:51.496 INFO:tasks.ceph:[osd] debug ms = 1 2023-05-22T15:25:51.496 INFO:tasks.ceph:[osd] debug osd = 20 2023-05-22T15:25:51.497 INFO:tasks.ceph:[osd] debug rocksdb = 10 2023-05-22T15:25:51.497 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2023-05-22T15:25:51.497 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2023-05-22T15:25:51.497 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2023-05-22T15:25:51.497 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2023-05-22T15:25:51.498 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2023-05-22T15:25:51.498 INFO:tasks.ceph:[osd] osd op complaint time = 180 2023-05-22T15:25:51.498 INFO:tasks.ceph:Setting up mon.a... 2023-05-22T15:25:51.498 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2023-05-22T15:25:51.534 INFO:teuthology.orchestra.run.smithi016.stdout:creating /etc/ceph/ceph.keyring 2023-05-22T15:25:51.541 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2023-05-22T15:25:51.618 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-05-22T15:25:51.672 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.16'), ('mon.b', '172.21.15.62'), ('mon.c', '[v2:172.21.15.62:3301,v1:172.21.15.62:6790]')] 2023-05-22T15:25:51.673 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.16,172.21.15.62,[v2:172.21.15.62:3301,v1:172.21.15.62:6790]'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false', 'debug client': 20, 'debug finisher': 20}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon op complaint time': 120}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}, 'mds.a': {}, 'mds.c': {}, 'mds.b': {}, 'mds.d': {}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}} 2023-05-22T15:25:51.674 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-05-22T15:25:51.674 DEBUG:teuthology.orchestra.run.smithi016:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-05-22T15:25:51.719 DEBUG:teuthology.orchestra.run.smithi016:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.16 --add b 172.21.15.62 --addv c '[v2:172.21.15.62:3301,v1:172.21.15.62:6790]' --print /home/ubuntu/cephtest/ceph.monmap 2023-05-22T15:25:51.806 INFO:teuthology.orchestra.run.smithi016.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2023-05-22T15:25:51.806 INFO:teuthology.orchestra.run.smithi016.stdout:monmaptool: generated fsid b2719834-7847-4d98-90a6-f65d57f85c49 2023-05-22T15:25:51.806 INFO:teuthology.orchestra.run.smithi016.stdout:setting min_mon_release = pacific 2023-05-22T15:25:51.807 INFO:teuthology.orchestra.run.smithi016.stdout:epoch 0 2023-05-22T15:25:51.807 INFO:teuthology.orchestra.run.smithi016.stdout:fsid b2719834-7847-4d98-90a6-f65d57f85c49 2023-05-22T15:25:51.807 INFO:teuthology.orchestra.run.smithi016.stdout:last_changed 2023-05-22T15:25:51.801223+0000 2023-05-22T15:25:51.807 INFO:teuthology.orchestra.run.smithi016.stdout:created 2023-05-22T15:25:51.801223+0000 2023-05-22T15:25:51.807 INFO:teuthology.orchestra.run.smithi016.stdout:min_mon_release 16 (pacific) 2023-05-22T15:25:51.807 INFO:teuthology.orchestra.run.smithi016.stdout:election_strategy: 1 2023-05-22T15:25:51.807 INFO:teuthology.orchestra.run.smithi016.stdout:0: [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] mon.a 2023-05-22T15:25:51.808 INFO:teuthology.orchestra.run.smithi016.stdout:1: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.b 2023-05-22T15:25:51.808 INFO:teuthology.orchestra.run.smithi016.stdout:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2023-05-22T15:25:51.808 INFO:teuthology.orchestra.run.smithi016.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2023-05-22T15:25:51.809 DEBUG:teuthology.orchestra.run.smithi016:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-05-22T15:25:51.855 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID b2719834-7847-4d98-90a6-f65d57f85c49... 2023-05-22T15:25:51.857 DEBUG:teuthology.orchestra.run.smithi016:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-05-22T15:25:51.901 DEBUG:teuthology.orchestra.run.smithi062:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-05-22T15:25:51.931 INFO:teuthology.orchestra.run.smithi016.stdout:[global] 2023-05-22T15:25:51.932 INFO:teuthology.orchestra.run.smithi016.stdout: chdir = "" 2023-05-22T15:25:51.932 INFO:teuthology.orchestra.run.smithi016.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-05-22T15:25:51.932 INFO:teuthology.orchestra.run.smithi016.stdout: auth supported = cephx 2023-05-22T15:25:51.932 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:25:51.933 INFO:teuthology.orchestra.run.smithi016.stdout: filestore xattr use omap = true 2023-05-22T15:25:51.933 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:25:51.933 INFO:teuthology.orchestra.run.smithi016.stdout: mon clock drift allowed = 1.000 2023-05-22T15:25:51.933 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:25:51.933 INFO:teuthology.orchestra.run.smithi016.stdout: osd crush chooseleaf type = 0 2023-05-22T15:25:51.933 INFO:teuthology.orchestra.run.smithi016.stdout: auth debug = true 2023-05-22T15:25:51.933 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:25:51.934 INFO:teuthology.orchestra.run.smithi016.stdout: ms die on old message = true 2023-05-22T15:25:51.934 INFO:teuthology.orchestra.run.smithi016.stdout: ms die on bug = true 2023-05-22T15:25:51.934 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:25:51.934 INFO:teuthology.orchestra.run.smithi016.stdout: mon max pg per osd = 10000 # >= luminous 2023-05-22T15:25:51.934 INFO:teuthology.orchestra.run.smithi016.stdout: mon pg warn max object skew = 0 2023-05-22T15:25:51.934 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:25:51.934 INFO:teuthology.orchestra.run.smithi016.stdout: # disable pg_autoscaler by default for new pools 2023-05-22T15:25:51.934 INFO:teuthology.orchestra.run.smithi016.stdout: osd_pool_default_pg_autoscale_mode = off 2023-05-22T15:25:51.935 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:25:51.935 INFO:teuthology.orchestra.run.smithi016.stdout: osd pool default size = 2 2023-05-22T15:25:51.935 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:25:51.935 INFO:teuthology.orchestra.run.smithi016.stdout: mon osd allow primary affinity = true 2023-05-22T15:25:51.935 INFO:teuthology.orchestra.run.smithi016.stdout: mon osd allow pg remap = true 2023-05-22T15:25:51.935 INFO:teuthology.orchestra.run.smithi016.stdout: mon warn on legacy crush tunables = false 2023-05-22T15:25:51.935 INFO:teuthology.orchestra.run.smithi016.stdout: mon warn on crush straw calc version zero = false 2023-05-22T15:25:51.935 INFO:teuthology.orchestra.run.smithi016.stdout: mon warn on no sortbitwise = false 2023-05-22T15:25:51.936 INFO:teuthology.orchestra.run.smithi016.stdout: mon warn on osd down out interval zero = false 2023-05-22T15:25:51.936 INFO:teuthology.orchestra.run.smithi016.stdout: mon warn on too few osds = false 2023-05-22T15:25:51.936 INFO:teuthology.orchestra.run.smithi016.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-05-22T15:25:51.936 INFO:teuthology.orchestra.run.smithi016.stdout: mon_warn_on_pool_no_redundancy = false 2023-05-22T15:25:51.936 INFO:teuthology.orchestra.run.smithi016.stdout: mon_allow_pool_size_one = true 2023-05-22T15:25:51.936 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:25:51.936 INFO:teuthology.orchestra.run.smithi016.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-05-22T15:25:51.937 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:25:51.937 INFO:teuthology.orchestra.run.smithi016.stdout: osd default data pool replay window = 5 2023-05-22T15:25:51.937 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:25:51.937 INFO:teuthology.orchestra.run.smithi016.stdout: mon allow pool delete = true 2023-05-22T15:25:51.937 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:25:51.938 INFO:teuthology.orchestra.run.smithi016.stdout: mon cluster log file level = debug 2023-05-22T15:25:51.938 INFO:teuthology.orchestra.run.smithi016.stdout: debug asserts on shutdown = true 2023-05-22T15:25:51.938 INFO:teuthology.orchestra.run.smithi016.stdout: mon health detail to clog = false 2023-05-22T15:25:51.938 INFO:teuthology.orchestra.run.smithi016.stdout: mon host = "172.21.15.16,172.21.15.62,[v2:172.21.15.62:3301,v1:172.21.15.62:6790]" 2023-05-22T15:25:51.938 INFO:teuthology.orchestra.run.smithi016.stdout: fsid = b2719834-7847-4d98-90a6-f65d57f85c49 2023-05-22T15:25:51.938 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:25:51.939 INFO:teuthology.orchestra.run.smithi016.stdout:[osd] 2023-05-22T15:25:51.939 INFO:teuthology.orchestra.run.smithi016.stdout: osd journal size = 100 2023-05-22T15:25:51.939 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:25:51.939 INFO:teuthology.orchestra.run.smithi016.stdout: osd scrub load threshold = 5.0 2023-05-22T15:25:51.939 INFO:teuthology.orchestra.run.smithi016.stdout: osd scrub max interval = 600 2023-05-22T15:25:51.939 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:25:51.939 INFO:teuthology.orchestra.run.smithi016.stdout: osd recover clone overlap = true 2023-05-22T15:25:51.940 INFO:teuthology.orchestra.run.smithi016.stdout: osd recovery max chunk = 1048576 2023-05-22T15:25:51.940 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:25:51.940 INFO:teuthology.orchestra.run.smithi016.stdout: osd debug shutdown = true 2023-05-22T15:25:51.940 INFO:teuthology.orchestra.run.smithi016.stdout: osd debug op order = true 2023-05-22T15:25:51.940 INFO:teuthology.orchestra.run.smithi016.stdout: osd debug verify stray on activate = true 2023-05-22T15:25:51.940 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:25:51.941 INFO:teuthology.orchestra.run.smithi016.stdout: osd open classes on start = true 2023-05-22T15:25:51.941 INFO:teuthology.orchestra.run.smithi016.stdout: osd debug pg log writeout = true 2023-05-22T15:25:51.941 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:25:51.941 INFO:teuthology.orchestra.run.smithi016.stdout: osd deep scrub update digest min age = 30 2023-05-22T15:25:51.941 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:25:51.941 INFO:teuthology.orchestra.run.smithi016.stdout: osd map max advance = 10 2023-05-22T15:25:51.941 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:25:51.942 INFO:teuthology.orchestra.run.smithi016.stdout: journal zero on create = true 2023-05-22T15:25:51.942 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:25:51.942 INFO:teuthology.orchestra.run.smithi016.stdout: filestore ondisk finisher threads = 3 2023-05-22T15:25:51.942 INFO:teuthology.orchestra.run.smithi016.stdout: filestore apply finisher threads = 3 2023-05-22T15:25:51.942 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:25:51.942 INFO:teuthology.orchestra.run.smithi016.stdout: bdev debug aio = true 2023-05-22T15:25:51.942 INFO:teuthology.orchestra.run.smithi016.stdout: osd debug misdirected ops = true 2023-05-22T15:25:51.943 INFO:teuthology.orchestra.run.smithi016.stdout: bluestore block size = 96636764160 2023-05-22T15:25:51.943 INFO:teuthology.orchestra.run.smithi016.stdout: bluestore fsck on mount = True 2023-05-22T15:25:51.943 INFO:teuthology.orchestra.run.smithi016.stdout: debug bluefs = 20 2023-05-22T15:25:51.943 INFO:teuthology.orchestra.run.smithi016.stdout: debug bluestore = 20 2023-05-22T15:25:51.943 INFO:teuthology.orchestra.run.smithi016.stdout: debug ms = 1 2023-05-22T15:25:51.943 INFO:teuthology.orchestra.run.smithi016.stdout: debug osd = 20 2023-05-22T15:25:51.944 INFO:teuthology.orchestra.run.smithi016.stdout: debug rocksdb = 10 2023-05-22T15:25:51.944 INFO:teuthology.orchestra.run.smithi016.stdout: mon osd backfillfull_ratio = 0.85 2023-05-22T15:25:51.944 INFO:teuthology.orchestra.run.smithi016.stdout: mon osd full ratio = 0.9 2023-05-22T15:25:51.944 INFO:teuthology.orchestra.run.smithi016.stdout: mon osd nearfull ratio = 0.8 2023-05-22T15:25:51.944 INFO:teuthology.orchestra.run.smithi016.stdout: osd failsafe full ratio = 0.95 2023-05-22T15:25:51.944 INFO:teuthology.orchestra.run.smithi016.stdout: osd objectstore = bluestore 2023-05-22T15:25:51.944 INFO:teuthology.orchestra.run.smithi016.stdout: osd op complaint time = 180 2023-05-22T15:25:51.944 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:25:51.945 INFO:teuthology.orchestra.run.smithi016.stdout:[mgr] 2023-05-22T15:25:51.945 INFO:teuthology.orchestra.run.smithi016.stdout: debug ms = 1 2023-05-22T15:25:51.945 INFO:teuthology.orchestra.run.smithi016.stdout: debug mgr = 20 2023-05-22T15:25:51.945 INFO:teuthology.orchestra.run.smithi016.stdout: debug mon = 20 2023-05-22T15:25:51.945 INFO:teuthology.orchestra.run.smithi016.stdout: debug auth = 20 2023-05-22T15:25:51.945 INFO:teuthology.orchestra.run.smithi016.stdout: mon reweight min pgs per osd = 4 2023-05-22T15:25:51.946 INFO:teuthology.orchestra.run.smithi016.stdout: mon reweight min bytes per osd = 10 2023-05-22T15:25:51.946 INFO:teuthology.orchestra.run.smithi016.stdout: mgr/telemetry/nag = false 2023-05-22T15:25:51.946 INFO:teuthology.orchestra.run.smithi016.stdout: debug client = 20 2023-05-22T15:25:51.946 INFO:teuthology.orchestra.run.smithi016.stdout: debug finisher = 20 2023-05-22T15:25:51.946 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:25:51.946 INFO:teuthology.orchestra.run.smithi016.stdout:[mon] 2023-05-22T15:25:51.946 INFO:teuthology.orchestra.run.smithi016.stdout: debug ms = 1 2023-05-22T15:25:51.947 INFO:teuthology.orchestra.run.smithi016.stdout: debug mon = 20 2023-05-22T15:25:51.947 INFO:teuthology.orchestra.run.smithi016.stdout: debug paxos = 20 2023-05-22T15:25:51.947 INFO:teuthology.orchestra.run.smithi016.stdout: debug auth = 20 2023-05-22T15:25:51.947 INFO:teuthology.orchestra.run.smithi016.stdout: mon data avail warn = 5 2023-05-22T15:25:51.947 INFO:teuthology.orchestra.run.smithi016.stdout: mon mgr mkfs grace = 240 2023-05-22T15:25:51.947 INFO:teuthology.orchestra.run.smithi016.stdout: mon reweight min pgs per osd = 4 2023-05-22T15:25:51.947 INFO:teuthology.orchestra.run.smithi016.stdout: mon osd reporter subtree level = osd 2023-05-22T15:25:51.947 INFO:teuthology.orchestra.run.smithi016.stdout: mon osd prime pg temp = true 2023-05-22T15:25:51.948 INFO:teuthology.orchestra.run.smithi016.stdout: mon reweight min bytes per osd = 10 2023-05-22T15:25:51.948 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:25:51.948 INFO:teuthology.orchestra.run.smithi016.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-05-22T15:25:51.948 INFO:teuthology.orchestra.run.smithi016.stdout: auth mon ticket ttl = 660 # 11m 2023-05-22T15:25:51.948 INFO:teuthology.orchestra.run.smithi016.stdout: auth service ticket ttl = 240 # 4m 2023-05-22T15:25:51.948 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:25:51.948 INFO:teuthology.orchestra.run.smithi016.stdout: # don't complain about insecure global_id in the test suite 2023-05-22T15:25:51.948 INFO:teuthology.orchestra.run.smithi016.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-05-22T15:25:51.949 INFO:teuthology.orchestra.run.smithi016.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-05-22T15:25:51.949 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:25:51.949 INFO:teuthology.orchestra.run.smithi016.stdout: # 1m isn't quite enough 2023-05-22T15:25:51.949 INFO:teuthology.orchestra.run.smithi016.stdout: mon_down_mkfs_grace = 2m 2023-05-22T15:25:51.949 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:25:51.949 INFO:teuthology.orchestra.run.smithi016.stdout: mon_warn_on_filestore_osds = false 2023-05-22T15:25:51.949 INFO:teuthology.orchestra.run.smithi016.stdout: mon op complaint time = 120 2023-05-22T15:25:51.950 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:25:51.950 INFO:teuthology.orchestra.run.smithi016.stdout:[client] 2023-05-22T15:25:51.950 INFO:teuthology.orchestra.run.smithi016.stdout: rgw cache enabled = true 2023-05-22T15:25:51.950 INFO:teuthology.orchestra.run.smithi016.stdout: rgw enable ops log = true 2023-05-22T15:25:51.950 INFO:teuthology.orchestra.run.smithi016.stdout: rgw enable usage log = true 2023-05-22T15:25:51.950 INFO:teuthology.orchestra.run.smithi016.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-05-22T15:25:51.950 INFO:teuthology.orchestra.run.smithi016.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-05-22T15:25:51.950 INFO:teuthology.orchestra.run.smithi016.stdout: client mount timeout = 600 2023-05-22T15:25:51.951 INFO:teuthology.orchestra.run.smithi016.stdout: debug client = 20 2023-05-22T15:25:51.951 INFO:teuthology.orchestra.run.smithi016.stdout: debug ms = 1 2023-05-22T15:25:51.951 INFO:teuthology.orchestra.run.smithi016.stdout: pid file = "" 2023-05-22T15:25:51.951 INFO:teuthology.orchestra.run.smithi016.stdout: rados mon op timeout = 900 2023-05-22T15:25:51.951 INFO:teuthology.orchestra.run.smithi016.stdout: rados osd op timeout = 900 2023-05-22T15:25:51.951 INFO:teuthology.orchestra.run.smithi016.stdout:[mon.a] 2023-05-22T15:25:51.951 INFO:teuthology.orchestra.run.smithi016.stdout:[mon.b] 2023-05-22T15:25:51.951 INFO:teuthology.orchestra.run.smithi016.stdout:[mon.c] 2023-05-22T15:25:51.952 INFO:teuthology.orchestra.run.smithi016.stdout:[mds.a] 2023-05-22T15:25:51.952 INFO:teuthology.orchestra.run.smithi016.stdout:[mds.c] 2023-05-22T15:25:51.952 INFO:teuthology.orchestra.run.smithi016.stdout:[mds.b] 2023-05-22T15:25:51.952 INFO:teuthology.orchestra.run.smithi016.stdout:[mds.d] 2023-05-22T15:25:51.952 INFO:teuthology.orchestra.run.smithi016.stdout:[mds] 2023-05-22T15:25:51.952 INFO:teuthology.orchestra.run.smithi016.stdout: debug mds = 20 2023-05-22T15:25:51.952 INFO:teuthology.orchestra.run.smithi016.stdout: debug mds balancer = 20 2023-05-22T15:25:51.953 INFO:teuthology.orchestra.run.smithi016.stdout: debug ms = 1 2023-05-22T15:25:51.953 INFO:teuthology.orchestra.run.smithi016.stdout: mds debug frag = True 2023-05-22T15:25:51.953 INFO:teuthology.orchestra.run.smithi016.stdout: mds debug scatterstat = True 2023-05-22T15:25:51.953 INFO:teuthology.orchestra.run.smithi016.stdout: mds op complaint time = 180 2023-05-22T15:25:51.953 INFO:teuthology.orchestra.run.smithi016.stdout: mds verify scatter = True 2023-05-22T15:25:51.954 INFO:teuthology.orchestra.run.smithi016.stdout: osd op complaint time = 180 2023-05-22T15:25:51.954 INFO:teuthology.orchestra.run.smithi016.stdout: rados mon op timeout = 900 2023-05-22T15:25:51.954 INFO:teuthology.orchestra.run.smithi016.stdout: rados osd op timeout = 900 2023-05-22T15:25:51.955 INFO:teuthology.orchestra.run.smithi062.stdout:[global] 2023-05-22T15:25:51.955 INFO:teuthology.orchestra.run.smithi062.stdout: chdir = "" 2023-05-22T15:25:51.956 INFO:teuthology.orchestra.run.smithi062.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-05-22T15:25:51.956 INFO:teuthology.orchestra.run.smithi062.stdout: auth supported = cephx 2023-05-22T15:25:51.956 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-05-22T15:25:51.956 INFO:teuthology.orchestra.run.smithi062.stdout: filestore xattr use omap = true 2023-05-22T15:25:51.956 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-05-22T15:25:51.956 INFO:teuthology.orchestra.run.smithi062.stdout: mon clock drift allowed = 1.000 2023-05-22T15:25:51.956 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-05-22T15:25:51.957 INFO:teuthology.orchestra.run.smithi062.stdout: osd crush chooseleaf type = 0 2023-05-22T15:25:51.957 INFO:teuthology.orchestra.run.smithi062.stdout: auth debug = true 2023-05-22T15:25:51.957 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-05-22T15:25:51.957 INFO:teuthology.orchestra.run.smithi062.stdout: ms die on old message = true 2023-05-22T15:25:51.957 INFO:teuthology.orchestra.run.smithi062.stdout: ms die on bug = true 2023-05-22T15:25:51.957 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-05-22T15:25:51.957 INFO:teuthology.orchestra.run.smithi062.stdout: mon max pg per osd = 10000 # >= luminous 2023-05-22T15:25:51.958 INFO:teuthology.orchestra.run.smithi062.stdout: mon pg warn max object skew = 0 2023-05-22T15:25:51.958 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-05-22T15:25:51.958 INFO:teuthology.orchestra.run.smithi062.stdout: # disable pg_autoscaler by default for new pools 2023-05-22T15:25:51.958 INFO:teuthology.orchestra.run.smithi062.stdout: osd_pool_default_pg_autoscale_mode = off 2023-05-22T15:25:51.958 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-05-22T15:25:51.958 INFO:teuthology.orchestra.run.smithi062.stdout: osd pool default size = 2 2023-05-22T15:25:51.959 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-05-22T15:25:51.959 INFO:teuthology.orchestra.run.smithi062.stdout: mon osd allow primary affinity = true 2023-05-22T15:25:51.959 INFO:teuthology.orchestra.run.smithi062.stdout: mon osd allow pg remap = true 2023-05-22T15:25:51.959 INFO:teuthology.orchestra.run.smithi062.stdout: mon warn on legacy crush tunables = false 2023-05-22T15:25:51.959 INFO:teuthology.orchestra.run.smithi062.stdout: mon warn on crush straw calc version zero = false 2023-05-22T15:25:51.959 INFO:teuthology.orchestra.run.smithi062.stdout: mon warn on no sortbitwise = false 2023-05-22T15:25:51.959 INFO:teuthology.orchestra.run.smithi062.stdout: mon warn on osd down out interval zero = false 2023-05-22T15:25:51.960 INFO:teuthology.orchestra.run.smithi062.stdout: mon warn on too few osds = false 2023-05-22T15:25:51.960 INFO:teuthology.orchestra.run.smithi062.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-05-22T15:25:51.960 INFO:teuthology.orchestra.run.smithi062.stdout: mon_warn_on_pool_no_redundancy = false 2023-05-22T15:25:51.960 INFO:teuthology.orchestra.run.smithi062.stdout: mon_allow_pool_size_one = true 2023-05-22T15:25:51.960 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-05-22T15:25:51.960 INFO:teuthology.orchestra.run.smithi062.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-05-22T15:25:51.961 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-05-22T15:25:51.961 INFO:teuthology.orchestra.run.smithi062.stdout: osd default data pool replay window = 5 2023-05-22T15:25:51.961 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-05-22T15:25:51.961 INFO:teuthology.orchestra.run.smithi062.stdout: mon allow pool delete = true 2023-05-22T15:25:51.962 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-05-22T15:25:51.962 INFO:teuthology.orchestra.run.smithi062.stdout: mon cluster log file level = debug 2023-05-22T15:25:51.962 INFO:teuthology.orchestra.run.smithi062.stdout: debug asserts on shutdown = true 2023-05-22T15:25:51.962 INFO:teuthology.orchestra.run.smithi062.stdout: mon health detail to clog = false 2023-05-22T15:25:51.963 INFO:teuthology.orchestra.run.smithi062.stdout: mon host = "172.21.15.16,172.21.15.62,[v2:172.21.15.62:3301,v1:172.21.15.62:6790]" 2023-05-22T15:25:51.963 INFO:teuthology.orchestra.run.smithi062.stdout: fsid = b2719834-7847-4d98-90a6-f65d57f85c49 2023-05-22T15:25:51.963 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-05-22T15:25:51.963 INFO:teuthology.orchestra.run.smithi062.stdout:[osd] 2023-05-22T15:25:51.964 INFO:teuthology.orchestra.run.smithi062.stdout: osd journal size = 100 2023-05-22T15:25:51.964 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-05-22T15:25:51.964 INFO:teuthology.orchestra.run.smithi062.stdout: osd scrub load threshold = 5.0 2023-05-22T15:25:51.965 INFO:teuthology.orchestra.run.smithi062.stdout: osd scrub max interval = 600 2023-05-22T15:25:51.965 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-05-22T15:25:51.965 INFO:teuthology.orchestra.run.smithi062.stdout: osd recover clone overlap = true 2023-05-22T15:25:51.965 INFO:teuthology.orchestra.run.smithi062.stdout: osd recovery max chunk = 1048576 2023-05-22T15:25:51.965 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-05-22T15:25:51.966 INFO:teuthology.orchestra.run.smithi062.stdout: osd debug shutdown = true 2023-05-22T15:25:51.966 INFO:teuthology.orchestra.run.smithi062.stdout: osd debug op order = true 2023-05-22T15:25:51.966 INFO:teuthology.orchestra.run.smithi062.stdout: osd debug verify stray on activate = true 2023-05-22T15:25:51.966 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-05-22T15:25:51.966 INFO:teuthology.orchestra.run.smithi062.stdout: osd open classes on start = true 2023-05-22T15:25:51.966 INFO:teuthology.orchestra.run.smithi062.stdout: osd debug pg log writeout = true 2023-05-22T15:25:51.966 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-05-22T15:25:51.967 INFO:teuthology.orchestra.run.smithi062.stdout: osd deep scrub update digest min age = 30 2023-05-22T15:25:51.967 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-05-22T15:25:51.967 INFO:teuthology.orchestra.run.smithi062.stdout: osd map max advance = 10 2023-05-22T15:25:51.967 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-05-22T15:25:51.967 INFO:teuthology.orchestra.run.smithi062.stdout: journal zero on create = true 2023-05-22T15:25:51.967 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-05-22T15:25:51.967 INFO:teuthology.orchestra.run.smithi062.stdout: filestore ondisk finisher threads = 3 2023-05-22T15:25:51.967 INFO:teuthology.orchestra.run.smithi062.stdout: filestore apply finisher threads = 3 2023-05-22T15:25:51.968 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-05-22T15:25:51.968 INFO:teuthology.orchestra.run.smithi062.stdout: bdev debug aio = true 2023-05-22T15:25:51.968 INFO:teuthology.orchestra.run.smithi062.stdout: osd debug misdirected ops = true 2023-05-22T15:25:51.968 INFO:teuthology.orchestra.run.smithi062.stdout: bluestore block size = 96636764160 2023-05-22T15:25:51.968 INFO:teuthology.orchestra.run.smithi062.stdout: bluestore fsck on mount = True 2023-05-22T15:25:51.968 INFO:teuthology.orchestra.run.smithi062.stdout: debug bluefs = 20 2023-05-22T15:25:51.968 INFO:teuthology.orchestra.run.smithi062.stdout: debug bluestore = 20 2023-05-22T15:25:51.969 INFO:teuthology.orchestra.run.smithi062.stdout: debug ms = 1 2023-05-22T15:25:51.969 INFO:teuthology.orchestra.run.smithi062.stdout: debug osd = 20 2023-05-22T15:25:51.969 INFO:teuthology.orchestra.run.smithi062.stdout: debug rocksdb = 10 2023-05-22T15:25:51.969 INFO:teuthology.orchestra.run.smithi062.stdout: mon osd backfillfull_ratio = 0.85 2023-05-22T15:25:51.969 INFO:teuthology.orchestra.run.smithi062.stdout: mon osd full ratio = 0.9 2023-05-22T15:25:51.969 INFO:teuthology.orchestra.run.smithi062.stdout: mon osd nearfull ratio = 0.8 2023-05-22T15:25:51.970 INFO:teuthology.orchestra.run.smithi062.stdout: osd failsafe full ratio = 0.95 2023-05-22T15:25:51.970 INFO:teuthology.orchestra.run.smithi062.stdout: osd objectstore = bluestore 2023-05-22T15:25:51.970 INFO:teuthology.orchestra.run.smithi062.stdout: osd op complaint time = 180 2023-05-22T15:25:51.970 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-05-22T15:25:51.970 INFO:teuthology.orchestra.run.smithi062.stdout:[mgr] 2023-05-22T15:25:51.970 INFO:teuthology.orchestra.run.smithi062.stdout: debug ms = 1 2023-05-22T15:25:51.970 INFO:teuthology.orchestra.run.smithi062.stdout: debug mgr = 20 2023-05-22T15:25:51.970 INFO:teuthology.orchestra.run.smithi062.stdout: debug mon = 20 2023-05-22T15:25:51.971 INFO:teuthology.orchestra.run.smithi062.stdout: debug auth = 20 2023-05-22T15:25:51.971 INFO:teuthology.orchestra.run.smithi062.stdout: mon reweight min pgs per osd = 4 2023-05-22T15:25:51.971 INFO:teuthology.orchestra.run.smithi062.stdout: mon reweight min bytes per osd = 10 2023-05-22T15:25:51.971 INFO:teuthology.orchestra.run.smithi062.stdout: mgr/telemetry/nag = false 2023-05-22T15:25:51.971 INFO:teuthology.orchestra.run.smithi062.stdout: debug client = 20 2023-05-22T15:25:51.971 INFO:teuthology.orchestra.run.smithi062.stdout: debug finisher = 20 2023-05-22T15:25:51.972 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-05-22T15:25:51.972 INFO:teuthology.orchestra.run.smithi062.stdout:[mon] 2023-05-22T15:25:51.972 INFO:teuthology.orchestra.run.smithi062.stdout: debug ms = 1 2023-05-22T15:25:51.972 INFO:teuthology.orchestra.run.smithi062.stdout: debug mon = 20 2023-05-22T15:25:51.972 INFO:teuthology.orchestra.run.smithi062.stdout: debug paxos = 20 2023-05-22T15:25:51.972 INFO:teuthology.orchestra.run.smithi062.stdout: debug auth = 20 2023-05-22T15:25:51.973 INFO:teuthology.orchestra.run.smithi062.stdout: mon data avail warn = 5 2023-05-22T15:25:51.973 INFO:teuthology.orchestra.run.smithi062.stdout: mon mgr mkfs grace = 240 2023-05-22T15:25:51.973 INFO:teuthology.orchestra.run.smithi062.stdout: mon reweight min pgs per osd = 4 2023-05-22T15:25:51.973 INFO:teuthology.orchestra.run.smithi062.stdout: mon osd reporter subtree level = osd 2023-05-22T15:25:51.973 INFO:teuthology.orchestra.run.smithi062.stdout: mon osd prime pg temp = true 2023-05-22T15:25:51.974 INFO:teuthology.orchestra.run.smithi062.stdout: mon reweight min bytes per osd = 10 2023-05-22T15:25:51.974 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-05-22T15:25:51.974 INFO:teuthology.orchestra.run.smithi062.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-05-22T15:25:51.974 INFO:teuthology.orchestra.run.smithi062.stdout: auth mon ticket ttl = 660 # 11m 2023-05-22T15:25:51.974 INFO:teuthology.orchestra.run.smithi062.stdout: auth service ticket ttl = 240 # 4m 2023-05-22T15:25:51.975 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-05-22T15:25:51.975 INFO:teuthology.orchestra.run.smithi062.stdout: # don't complain about insecure global_id in the test suite 2023-05-22T15:25:51.975 INFO:teuthology.orchestra.run.smithi062.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-05-22T15:25:51.975 INFO:teuthology.orchestra.run.smithi062.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-05-22T15:25:51.975 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-05-22T15:25:51.975 INFO:teuthology.orchestra.run.smithi062.stdout: # 1m isn't quite enough 2023-05-22T15:25:51.976 INFO:teuthology.orchestra.run.smithi062.stdout: mon_down_mkfs_grace = 2m 2023-05-22T15:25:51.976 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-05-22T15:25:51.976 INFO:teuthology.orchestra.run.smithi062.stdout: mon_warn_on_filestore_osds = false 2023-05-22T15:25:51.976 INFO:teuthology.orchestra.run.smithi062.stdout: mon op complaint time = 120 2023-05-22T15:25:51.976 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-05-22T15:25:51.976 INFO:teuthology.orchestra.run.smithi062.stdout:[client] 2023-05-22T15:25:51.977 INFO:teuthology.orchestra.run.smithi062.stdout: rgw cache enabled = true 2023-05-22T15:25:51.977 INFO:teuthology.orchestra.run.smithi062.stdout: rgw enable ops log = true 2023-05-22T15:25:51.977 INFO:teuthology.orchestra.run.smithi062.stdout: rgw enable usage log = true 2023-05-22T15:25:51.977 INFO:teuthology.orchestra.run.smithi062.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-05-22T15:25:51.978 INFO:teuthology.orchestra.run.smithi062.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-05-22T15:25:51.978 INFO:teuthology.orchestra.run.smithi062.stdout: client mount timeout = 600 2023-05-22T15:25:51.978 INFO:teuthology.orchestra.run.smithi062.stdout: debug client = 20 2023-05-22T15:25:51.978 INFO:teuthology.orchestra.run.smithi062.stdout: debug ms = 1 2023-05-22T15:25:51.978 INFO:teuthology.orchestra.run.smithi062.stdout: pid file = "" 2023-05-22T15:25:51.978 INFO:teuthology.orchestra.run.smithi062.stdout: rados mon op timeout = 900 2023-05-22T15:25:51.978 INFO:teuthology.orchestra.run.smithi062.stdout: rados osd op timeout = 900 2023-05-22T15:25:51.979 INFO:teuthology.orchestra.run.smithi062.stdout:[mon.a] 2023-05-22T15:25:51.979 INFO:teuthology.orchestra.run.smithi062.stdout:[mon.b] 2023-05-22T15:25:51.979 INFO:teuthology.orchestra.run.smithi062.stdout:[mon.c] 2023-05-22T15:25:51.979 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.a] 2023-05-22T15:25:51.979 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.c] 2023-05-22T15:25:51.979 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.b] 2023-05-22T15:25:51.979 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.d] 2023-05-22T15:25:51.980 INFO:teuthology.orchestra.run.smithi062.stdout:[mds] 2023-05-22T15:25:51.980 INFO:teuthology.orchestra.run.smithi062.stdout: debug mds = 20 2023-05-22T15:25:51.980 INFO:teuthology.orchestra.run.smithi062.stdout: debug mds balancer = 20 2023-05-22T15:25:51.981 INFO:teuthology.orchestra.run.smithi062.stdout: debug ms = 1 2023-05-22T15:25:51.981 INFO:teuthology.orchestra.run.smithi062.stdout: mds debug frag = True 2023-05-22T15:25:51.981 INFO:teuthology.orchestra.run.smithi062.stdout: mds debug scatterstat = True 2023-05-22T15:25:51.981 INFO:teuthology.orchestra.run.smithi062.stdout: mds op complaint time = 180 2023-05-22T15:25:51.981 INFO:teuthology.orchestra.run.smithi062.stdout: mds verify scatter = True 2023-05-22T15:25:51.982 INFO:teuthology.orchestra.run.smithi062.stdout: osd op complaint time = 180 2023-05-22T15:25:51.982 INFO:teuthology.orchestra.run.smithi062.stdout: rados mon op timeout = 900 2023-05-22T15:25:51.982 INFO:teuthology.orchestra.run.smithi062.stdout: rados osd op timeout = 900 2023-05-22T15:25:51.983 INFO:tasks.ceph:Creating admin key on mon.a... 2023-05-22T15:25:51.983 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2023-05-22T15:25:52.047 INFO:tasks.ceph:Copying monmap to all nodes... 2023-05-22T15:25:52.048 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-05-22T15:25:52.048 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2023-05-22T15:25:52.053 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-05-22T15:25:52.054 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2023-05-22T15:25:52.100 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi016.front.sepia.ceph.com 2023-05-22T15:25:52.100 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-05-22T15:25:52.100 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd of=/etc/ceph/ceph.keyring 2023-05-22T15:25:52.100 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-05-22T15:25:52.164 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-05-22T15:25:52.164 DEBUG:teuthology.orchestra.run.smithi016:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-05-22T15:25:52.212 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi062.front.sepia.ceph.com 2023-05-22T15:25:52.212 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2023-05-22T15:25:52.212 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd of=/etc/ceph/ceph.keyring 2023-05-22T15:25:52.213 DEBUG:teuthology.orchestra.run.smithi062:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-05-22T15:25:52.236 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2023-05-22T15:25:52.236 DEBUG:teuthology.orchestra.run.smithi062:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-05-22T15:25:52.283 INFO:tasks.ceph:Setting up mon nodes... 2023-05-22T15:25:52.283 INFO:tasks.ceph:Setting up mgr nodes... 2023-05-22T15:25:52.284 DEBUG:teuthology.orchestra.run.smithi016:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2023-05-22T15:25:52.329 INFO:teuthology.orchestra.run.smithi016.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2023-05-22T15:25:52.332 DEBUG:teuthology.orchestra.run.smithi062:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2023-05-22T15:25:52.377 INFO:teuthology.orchestra.run.smithi062.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2023-05-22T15:25:52.392 INFO:tasks.ceph:Setting up mds nodes... 2023-05-22T15:25:52.392 DEBUG:teuthology.orchestra.run.smithi016:> sudo mkdir -p /var/lib/ceph/mds/ceph-a && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.a /var/lib/ceph/mds/ceph-a/keyring 2023-05-22T15:25:52.436 INFO:teuthology.orchestra.run.smithi016.stdout:creating /var/lib/ceph/mds/ceph-a/keyring 2023-05-22T15:25:52.439 DEBUG:teuthology.orchestra.run.smithi016:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-a 2023-05-22T15:25:52.451 DEBUG:teuthology.orchestra.run.smithi016:> sudo mkdir -p /var/lib/ceph/mds/ceph-c && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.c /var/lib/ceph/mds/ceph-c/keyring 2023-05-22T15:25:52.534 INFO:teuthology.orchestra.run.smithi016.stdout:creating /var/lib/ceph/mds/ceph-c/keyring 2023-05-22T15:25:52.536 DEBUG:teuthology.orchestra.run.smithi016:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-c 2023-05-22T15:25:52.593 DEBUG:teuthology.orchestra.run.smithi062:> sudo mkdir -p /var/lib/ceph/mds/ceph-b && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.b /var/lib/ceph/mds/ceph-b/keyring 2023-05-22T15:25:52.640 INFO:teuthology.orchestra.run.smithi062.stdout:creating /var/lib/ceph/mds/ceph-b/keyring 2023-05-22T15:25:52.643 DEBUG:teuthology.orchestra.run.smithi062:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-b 2023-05-22T15:25:52.655 DEBUG:teuthology.orchestra.run.smithi062:> sudo mkdir -p /var/lib/ceph/mds/ceph-d && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.d /var/lib/ceph/mds/ceph-d/keyring 2023-05-22T15:25:52.740 INFO:teuthology.orchestra.run.smithi062.stdout:creating /var/lib/ceph/mds/ceph-d/keyring 2023-05-22T15:25:52.743 DEBUG:teuthology.orchestra.run.smithi062:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-d 2023-05-22T15:25:52.799 INFO:tasks.ceph_client:Setting up client nodes... 2023-05-22T15:25:52.800 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2023-05-22T15:25:52.836 INFO:teuthology.orchestra.run.smithi016.stdout:creating /etc/ceph/ceph.client.0.keyring 2023-05-22T15:25:52.846 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.2 /etc/ceph/ceph.client.2.keyring && sudo chmod 0644 /etc/ceph/ceph.client.2.keyring 2023-05-22T15:25:52.924 INFO:teuthology.orchestra.run.smithi016.stdout:creating /etc/ceph/ceph.client.2.keyring 2023-05-22T15:25:52.936 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2023-05-22T15:25:52.972 INFO:teuthology.orchestra.run.smithi062.stdout:creating /etc/ceph/ceph.client.1.keyring 2023-05-22T15:25:52.985 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.3 /etc/ceph/ceph.client.3.keyring && sudo chmod 0644 /etc/ceph/ceph.client.3.keyring 2023-05-22T15:25:53.061 INFO:teuthology.orchestra.run.smithi062.stdout:creating /etc/ceph/ceph.client.3.keyring 2023-05-22T15:25:53.072 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-05-22T15:25:53.072 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi016.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi062.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2023-05-22T15:25:53.073 DEBUG:teuthology.orchestra.run.smithi016:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2023-05-22T15:25:53.087 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-05-22T15:25:53.087 INFO:tasks.ceph:role: osd.0 2023-05-22T15:25:53.087 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi016.front.sepia.ceph.com 2023-05-22T15:25:53.087 DEBUG:teuthology.orchestra.run.smithi016:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-05-22T15:25:53.145 INFO:teuthology.orchestra.run.smithi016.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-05-22T15:25:53.145 INFO:teuthology.orchestra.run.smithi016.stdout: = sectsz=512 attr=2, projid32bit=1 2023-05-22T15:25:53.145 INFO:teuthology.orchestra.run.smithi016.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-05-22T15:25:53.145 INFO:teuthology.orchestra.run.smithi016.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-05-22T15:25:53.145 INFO:teuthology.orchestra.run.smithi016.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-05-22T15:25:53.145 INFO:teuthology.orchestra.run.smithi016.stdout: = sunit=0 swidth=0 blks 2023-05-22T15:25:53.146 INFO:teuthology.orchestra.run.smithi016.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-05-22T15:25:53.146 INFO:teuthology.orchestra.run.smithi016.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-05-22T15:25:53.146 INFO:teuthology.orchestra.run.smithi016.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-05-22T15:25:53.146 INFO:teuthology.orchestra.run.smithi016.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-05-22T15:25:53.176 INFO:teuthology.orchestra.run.smithi016.stdout:Discarding blocks...Done. 2023-05-22T15:25:53.178 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi016.front.sepia.ceph.com -o noatime 2023-05-22T15:25:53.178 DEBUG:teuthology.orchestra.run.smithi016:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2023-05-22T15:25:53.234 DEBUG:teuthology.orchestra.run.smithi016:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2023-05-22T15:25:53.284 INFO:teuthology.orchestra.run.smithi016.stderr:sudo: /sbin/restorecon: command not found 2023-05-22T15:25:53.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-05-22T15:25:53.286 DEBUG:teuthology.orchestra.run.smithi016:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2023-05-22T15:25:53.338 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-05-22T15:25:53.339 INFO:tasks.ceph:role: osd.1 2023-05-22T15:25:53.339 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi016.front.sepia.ceph.com 2023-05-22T15:25:53.339 DEBUG:teuthology.orchestra.run.smithi016:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-05-22T15:25:53.398 INFO:teuthology.orchestra.run.smithi016.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-05-22T15:25:53.398 INFO:teuthology.orchestra.run.smithi016.stdout: = sectsz=512 attr=2, projid32bit=1 2023-05-22T15:25:53.398 INFO:teuthology.orchestra.run.smithi016.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-05-22T15:25:53.399 INFO:teuthology.orchestra.run.smithi016.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-05-22T15:25:53.399 INFO:teuthology.orchestra.run.smithi016.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-05-22T15:25:53.399 INFO:teuthology.orchestra.run.smithi016.stdout: = sunit=0 swidth=0 blks 2023-05-22T15:25:53.399 INFO:teuthology.orchestra.run.smithi016.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-05-22T15:25:53.400 INFO:teuthology.orchestra.run.smithi016.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-05-22T15:25:53.400 INFO:teuthology.orchestra.run.smithi016.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-05-22T15:25:53.400 INFO:teuthology.orchestra.run.smithi016.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-05-22T15:25:53.429 INFO:teuthology.orchestra.run.smithi016.stdout:Discarding blocks...Done. 2023-05-22T15:25:53.430 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi016.front.sepia.ceph.com -o noatime 2023-05-22T15:25:53.431 DEBUG:teuthology.orchestra.run.smithi016:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2023-05-22T15:25:53.489 DEBUG:teuthology.orchestra.run.smithi016:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2023-05-22T15:25:53.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-05-22T15:25:53.541 INFO:teuthology.orchestra.run.smithi016.stderr:sudo: /sbin/restorecon: command not found 2023-05-22T15:25:53.542 DEBUG:teuthology.orchestra.run.smithi016:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2023-05-22T15:25:53.595 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-05-22T15:25:53.595 INFO:tasks.ceph:role: osd.2 2023-05-22T15:25:53.595 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi016.front.sepia.ceph.com 2023-05-22T15:25:53.596 DEBUG:teuthology.orchestra.run.smithi016:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-05-22T15:25:53.655 INFO:teuthology.orchestra.run.smithi016.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-05-22T15:25:53.655 INFO:teuthology.orchestra.run.smithi016.stdout: = sectsz=512 attr=2, projid32bit=1 2023-05-22T15:25:53.655 INFO:teuthology.orchestra.run.smithi016.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-05-22T15:25:53.656 INFO:teuthology.orchestra.run.smithi016.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-05-22T15:25:53.656 INFO:teuthology.orchestra.run.smithi016.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-05-22T15:25:53.656 INFO:teuthology.orchestra.run.smithi016.stdout: = sunit=0 swidth=0 blks 2023-05-22T15:25:53.656 INFO:teuthology.orchestra.run.smithi016.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-05-22T15:25:53.656 INFO:teuthology.orchestra.run.smithi016.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-05-22T15:25:53.657 INFO:teuthology.orchestra.run.smithi016.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-05-22T15:25:53.657 INFO:teuthology.orchestra.run.smithi016.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-05-22T15:25:53.690 INFO:teuthology.orchestra.run.smithi016.stdout:Discarding blocks...Done. 2023-05-22T15:25:53.691 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi016.front.sepia.ceph.com -o noatime 2023-05-22T15:25:53.692 DEBUG:teuthology.orchestra.run.smithi016:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2023-05-22T15:25:53.752 DEBUG:teuthology.orchestra.run.smithi016:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2023-05-22T15:25:53.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-05-22T15:25:53.804 INFO:teuthology.orchestra.run.smithi016.stderr:sudo: /sbin/restorecon: command not found 2023-05-22T15:25:53.805 DEBUG:teuthology.orchestra.run.smithi016:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2023-05-22T15:25:53.857 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-05-22T15:25:53.858 INFO:tasks.ceph:role: osd.3 2023-05-22T15:25:53.858 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi016.front.sepia.ceph.com 2023-05-22T15:25:53.858 DEBUG:teuthology.orchestra.run.smithi016:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2023-05-22T15:25:53.912 INFO:teuthology.orchestra.run.smithi016.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2023-05-22T15:25:53.912 INFO:teuthology.orchestra.run.smithi016.stdout: = sectsz=512 attr=2, projid32bit=1 2023-05-22T15:25:53.912 INFO:teuthology.orchestra.run.smithi016.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-05-22T15:25:53.913 INFO:teuthology.orchestra.run.smithi016.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-05-22T15:25:53.913 INFO:teuthology.orchestra.run.smithi016.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-05-22T15:25:53.913 INFO:teuthology.orchestra.run.smithi016.stdout: = sunit=0 swidth=0 blks 2023-05-22T15:25:53.913 INFO:teuthology.orchestra.run.smithi016.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-05-22T15:25:53.913 INFO:teuthology.orchestra.run.smithi016.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-05-22T15:25:53.913 INFO:teuthology.orchestra.run.smithi016.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-05-22T15:25:53.913 INFO:teuthology.orchestra.run.smithi016.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-05-22T15:25:53.943 INFO:teuthology.orchestra.run.smithi016.stdout:Discarding blocks...Done. 2023-05-22T15:25:53.945 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi016.front.sepia.ceph.com -o noatime 2023-05-22T15:25:53.945 DEBUG:teuthology.orchestra.run.smithi016:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2023-05-22T15:25:54.005 DEBUG:teuthology.orchestra.run.smithi016:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2023-05-22T15:25:54.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-05-22T15:25:54.057 INFO:teuthology.orchestra.run.smithi016.stderr:sudo: /sbin/restorecon: command not found 2023-05-22T15:25:54.057 DEBUG:teuthology.orchestra.run.smithi016:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-05-22T15:25:54.171 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:25:54.158+0000 7fc484ac9640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2023-05-22T15:25:54.172 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:25:54.162+0000 7fc484ac9640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2023-05-22T15:25:54.172 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:25:54.162+0000 7fc484ac9640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-05-22T15:25:54.172 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:25:54.162+0000 7fc484ac9640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-05-22T15:25:54.173 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:25:54.162+0000 7fc484ac9640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-05-22T15:25:54.174 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:25:54.162+0000 7fc484ac9640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2023-05-22T15:25:57.198 DEBUG:teuthology.orchestra.run.smithi016:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2023-05-22T15:25:57.212 DEBUG:teuthology.orchestra.run.smithi016:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-05-22T15:25:57.297 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:25:57.286+0000 7f17b1127640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2023-05-22T15:25:57.298 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:25:57.286+0000 7f17b1127640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2023-05-22T15:25:57.298 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:25:57.286+0000 7f17b1127640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-05-22T15:25:57.298 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:25:57.286+0000 7f17b1127640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-05-22T15:25:57.298 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:25:57.286+0000 7f17b1127640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-05-22T15:25:57.299 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:25:57.286+0000 7f17b1127640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2023-05-22T15:26:00.355 DEBUG:teuthology.orchestra.run.smithi016:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2023-05-22T15:26:00.413 DEBUG:teuthology.orchestra.run.smithi016:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-05-22T15:26:00.501 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:00.490+0000 7fdb0e1e0640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2023-05-22T15:26:00.501 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:00.490+0000 7fdb0e1e0640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2023-05-22T15:26:00.502 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:00.490+0000 7fdb0e1e0640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-05-22T15:26:00.502 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:00.490+0000 7fdb0e1e0640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-05-22T15:26:00.502 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:00.490+0000 7fdb0e1e0640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-05-22T15:26:00.502 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:00.490+0000 7fdb0e1e0640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2023-05-22T15:26:03.511 DEBUG:teuthology.orchestra.run.smithi016:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2023-05-22T15:26:03.569 DEBUG:teuthology.orchestra.run.smithi016:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-05-22T15:26:03.653 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:03.642+0000 7f3270fd5640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2023-05-22T15:26:03.653 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:03.642+0000 7f3270fd5640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2023-05-22T15:26:03.653 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:03.642+0000 7f3270fd5640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-05-22T15:26:03.653 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:03.642+0000 7f3270fd5640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-05-22T15:26:03.653 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:03.642+0000 7f3270fd5640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-05-22T15:26:03.654 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:03.642+0000 7f3270fd5640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2023-05-22T15:26:06.663 DEBUG:teuthology.orchestra.run.smithi016:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2023-05-22T15:26:06.720 DEBUG:teuthology.orchestra.run.smithi062:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2023-05-22T15:26:06.735 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-05-22T15:26:06.736 INFO:tasks.ceph:role: osd.4 2023-05-22T15:26:06.736 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi062.front.sepia.ceph.com 2023-05-22T15:26:06.736 DEBUG:teuthology.orchestra.run.smithi062:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-05-22T15:26:06.798 INFO:teuthology.orchestra.run.smithi062.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-05-22T15:26:06.798 INFO:teuthology.orchestra.run.smithi062.stdout: = sectsz=512 attr=2, projid32bit=1 2023-05-22T15:26:06.798 INFO:teuthology.orchestra.run.smithi062.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-05-22T15:26:06.799 INFO:teuthology.orchestra.run.smithi062.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-05-22T15:26:06.799 INFO:teuthology.orchestra.run.smithi062.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-05-22T15:26:06.799 INFO:teuthology.orchestra.run.smithi062.stdout: = sunit=0 swidth=0 blks 2023-05-22T15:26:06.799 INFO:teuthology.orchestra.run.smithi062.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-05-22T15:26:06.799 INFO:teuthology.orchestra.run.smithi062.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-05-22T15:26:06.799 INFO:teuthology.orchestra.run.smithi062.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-05-22T15:26:06.799 INFO:teuthology.orchestra.run.smithi062.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-05-22T15:26:07.258 INFO:teuthology.orchestra.run.smithi062.stdout:Discarding blocks...Done. 2023-05-22T15:26:07.260 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi062.front.sepia.ceph.com -o noatime 2023-05-22T15:26:07.261 DEBUG:teuthology.orchestra.run.smithi062:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2023-05-22T15:26:07.362 DEBUG:teuthology.orchestra.run.smithi062:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2023-05-22T15:26:07.372 INFO:teuthology.orchestra.run.smithi062.stderr:sudo: /sbin/restorecon: command not found 2023-05-22T15:26:07.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-05-22T15:26:07.373 DEBUG:teuthology.orchestra.run.smithi062:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2023-05-22T15:26:07.425 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-05-22T15:26:07.426 INFO:tasks.ceph:role: osd.5 2023-05-22T15:26:07.426 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi062.front.sepia.ceph.com 2023-05-22T15:26:07.426 DEBUG:teuthology.orchestra.run.smithi062:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-05-22T15:26:07.486 INFO:teuthology.orchestra.run.smithi062.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-05-22T15:26:07.486 INFO:teuthology.orchestra.run.smithi062.stdout: = sectsz=512 attr=2, projid32bit=1 2023-05-22T15:26:07.486 INFO:teuthology.orchestra.run.smithi062.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-05-22T15:26:07.486 INFO:teuthology.orchestra.run.smithi062.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-05-22T15:26:07.487 INFO:teuthology.orchestra.run.smithi062.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-05-22T15:26:07.487 INFO:teuthology.orchestra.run.smithi062.stdout: = sunit=0 swidth=0 blks 2023-05-22T15:26:07.487 INFO:teuthology.orchestra.run.smithi062.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-05-22T15:26:07.487 INFO:teuthology.orchestra.run.smithi062.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-05-22T15:26:07.487 INFO:teuthology.orchestra.run.smithi062.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-05-22T15:26:07.487 INFO:teuthology.orchestra.run.smithi062.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-05-22T15:26:07.949 INFO:teuthology.orchestra.run.smithi062.stdout:Discarding blocks...Done. 2023-05-22T15:26:07.952 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi062.front.sepia.ceph.com -o noatime 2023-05-22T15:26:07.952 DEBUG:teuthology.orchestra.run.smithi062:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2023-05-22T15:26:08.085 DEBUG:teuthology.orchestra.run.smithi062:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2023-05-22T15:26:08.097 INFO:teuthology.orchestra.run.smithi062.stderr:sudo: /sbin/restorecon: command not found 2023-05-22T15:26:08.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-05-22T15:26:08.098 DEBUG:teuthology.orchestra.run.smithi062:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2023-05-22T15:26:08.155 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-05-22T15:26:08.155 INFO:tasks.ceph:role: osd.6 2023-05-22T15:26:08.155 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi062.front.sepia.ceph.com 2023-05-22T15:26:08.156 DEBUG:teuthology.orchestra.run.smithi062:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-05-22T15:26:08.219 INFO:teuthology.orchestra.run.smithi062.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-05-22T15:26:08.219 INFO:teuthology.orchestra.run.smithi062.stdout: = sectsz=512 attr=2, projid32bit=1 2023-05-22T15:26:08.219 INFO:teuthology.orchestra.run.smithi062.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-05-22T15:26:08.220 INFO:teuthology.orchestra.run.smithi062.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-05-22T15:26:08.220 INFO:teuthology.orchestra.run.smithi062.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-05-22T15:26:08.220 INFO:teuthology.orchestra.run.smithi062.stdout: = sunit=0 swidth=0 blks 2023-05-22T15:26:08.220 INFO:teuthology.orchestra.run.smithi062.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-05-22T15:26:08.220 INFO:teuthology.orchestra.run.smithi062.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-05-22T15:26:08.220 INFO:teuthology.orchestra.run.smithi062.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-05-22T15:26:08.220 INFO:teuthology.orchestra.run.smithi062.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-05-22T15:26:08.686 INFO:teuthology.orchestra.run.smithi062.stdout:Discarding blocks...Done. 2023-05-22T15:26:08.689 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi062.front.sepia.ceph.com -o noatime 2023-05-22T15:26:08.689 DEBUG:teuthology.orchestra.run.smithi062:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2023-05-22T15:26:08.820 DEBUG:teuthology.orchestra.run.smithi062:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2023-05-22T15:26:08.830 INFO:teuthology.orchestra.run.smithi062.stderr:sudo: /sbin/restorecon: command not found 2023-05-22T15:26:08.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-05-22T15:26:08.831 DEBUG:teuthology.orchestra.run.smithi062:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2023-05-22T15:26:08.886 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-05-22T15:26:08.887 INFO:tasks.ceph:role: osd.7 2023-05-22T15:26:08.887 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi062.front.sepia.ceph.com 2023-05-22T15:26:08.887 DEBUG:teuthology.orchestra.run.smithi062:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2023-05-22T15:26:08.951 INFO:teuthology.orchestra.run.smithi062.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2023-05-22T15:26:08.951 INFO:teuthology.orchestra.run.smithi062.stdout: = sectsz=512 attr=2, projid32bit=1 2023-05-22T15:26:08.951 INFO:teuthology.orchestra.run.smithi062.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-05-22T15:26:08.951 INFO:teuthology.orchestra.run.smithi062.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-05-22T15:26:08.951 INFO:teuthology.orchestra.run.smithi062.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-05-22T15:26:08.952 INFO:teuthology.orchestra.run.smithi062.stdout: = sunit=0 swidth=0 blks 2023-05-22T15:26:08.952 INFO:teuthology.orchestra.run.smithi062.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-05-22T15:26:08.952 INFO:teuthology.orchestra.run.smithi062.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-05-22T15:26:08.952 INFO:teuthology.orchestra.run.smithi062.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-05-22T15:26:08.952 INFO:teuthology.orchestra.run.smithi062.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-05-22T15:26:09.365 INFO:teuthology.orchestra.run.smithi062.stdout:Discarding blocks...Done. 2023-05-22T15:26:09.367 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi062.front.sepia.ceph.com -o noatime 2023-05-22T15:26:09.368 DEBUG:teuthology.orchestra.run.smithi062:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2023-05-22T15:26:09.453 DEBUG:teuthology.orchestra.run.smithi062:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2023-05-22T15:26:09.505 INFO:teuthology.orchestra.run.smithi062.stderr:sudo: /sbin/restorecon: command not found 2023-05-22T15:26:09.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-05-22T15:26:09.506 DEBUG:teuthology.orchestra.run.smithi062:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-05-22T15:26:09.628 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:09.617+0000 7f842c0c2640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2023-05-22T15:26:09.632 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:09.621+0000 7f842c0c2640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2023-05-22T15:26:09.632 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:09.621+0000 7f842c0c2640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-05-22T15:26:09.632 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:09.621+0000 7f842c0c2640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-05-22T15:26:09.632 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:09.621+0000 7f842c0c2640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-05-22T15:26:09.635 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:09.625+0000 7f842c0c2640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2023-05-22T15:26:12.665 DEBUG:teuthology.orchestra.run.smithi062:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2023-05-22T15:26:12.679 DEBUG:teuthology.orchestra.run.smithi062:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-05-22T15:26:12.767 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:12.757+0000 7f08e0294640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2023-05-22T15:26:12.768 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:12.757+0000 7f08e0294640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2023-05-22T15:26:12.768 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:12.757+0000 7f08e0294640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-05-22T15:26:12.768 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:12.757+0000 7f08e0294640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-05-22T15:26:12.769 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:12.757+0000 7f08e0294640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-05-22T15:26:12.769 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:12.757+0000 7f08e0294640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2023-05-22T15:26:15.783 DEBUG:teuthology.orchestra.run.smithi062:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2023-05-22T15:26:15.840 DEBUG:teuthology.orchestra.run.smithi062:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-05-22T15:26:15.923 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:15.913+0000 7f15a57c1640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2023-05-22T15:26:15.924 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:15.913+0000 7f15a57c1640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2023-05-22T15:26:15.924 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:15.913+0000 7f15a57c1640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-05-22T15:26:15.924 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:15.913+0000 7f15a57c1640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-05-22T15:26:15.925 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:15.913+0000 7f15a57c1640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-05-22T15:26:15.925 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:15.913+0000 7f15a57c1640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2023-05-22T15:26:18.940 DEBUG:teuthology.orchestra.run.smithi062:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2023-05-22T15:26:19.000 DEBUG:teuthology.orchestra.run.smithi062:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-05-22T15:26:19.087 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:19.077+0000 7fc4c1f7e640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2023-05-22T15:26:19.088 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:19.077+0000 7fc4c1f7e640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2023-05-22T15:26:19.088 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:19.077+0000 7fc4c1f7e640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-05-22T15:26:19.088 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:19.077+0000 7fc4c1f7e640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-05-22T15:26:19.088 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:19.077+0000 7fc4c1f7e640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-05-22T15:26:19.089 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:19.077+0000 7fc4c1f7e640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2023-05-22T15:26:20.158 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-05-22T15:26:20.161 DEBUG:teuthology.orchestra.run.smithi062:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-05-22T15:26:22.117 DEBUG:teuthology.orchestra.run.smithi062:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2023-05-22T15:26:22.131 INFO:tasks.ceph:Reading keys from all nodes... 2023-05-22T15:26:22.132 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-05-22T15:26:22.132 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2023-05-22T15:26:22.147 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-05-22T15:26:22.147 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd if=/var/lib/ceph/mds/ceph-a/keyring of=/dev/stdout 2023-05-22T15:26:22.204 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-05-22T15:26:22.205 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd if=/var/lib/ceph/mds/ceph-c/keyring of=/dev/stdout 2023-05-22T15:26:22.260 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-05-22T15:26:22.261 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2023-05-22T15:26:22.315 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-05-22T15:26:22.316 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2023-05-22T15:26:22.371 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-05-22T15:26:22.372 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2023-05-22T15:26:22.427 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-05-22T15:26:22.427 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2023-05-22T15:26:22.483 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2023-05-22T15:26:22.484 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2023-05-22T15:26:22.498 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2023-05-22T15:26:22.499 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd if=/var/lib/ceph/mds/ceph-b/keyring of=/dev/stdout 2023-05-22T15:26:22.555 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2023-05-22T15:26:22.555 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd if=/var/lib/ceph/mds/ceph-d/keyring of=/dev/stdout 2023-05-22T15:26:22.611 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2023-05-22T15:26:22.611 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2023-05-22T15:26:22.666 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2023-05-22T15:26:22.666 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2023-05-22T15:26:22.718 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2023-05-22T15:26:22.718 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2023-05-22T15:26:22.775 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2023-05-22T15:26:22.776 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2023-05-22T15:26:22.830 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-05-22T15:26:22.830 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2023-05-22T15:26:22.837 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-05-22T15:26:22.837 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/etc/ceph/ceph.client.2.keyring of=/dev/stdout 2023-05-22T15:26:22.884 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2023-05-22T15:26:22.885 DEBUG:teuthology.orchestra.run.smithi062:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2023-05-22T15:26:22.891 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2023-05-22T15:26:22.892 DEBUG:teuthology.orchestra.run.smithi062:> dd if=/etc/ceph/ceph.client.3.keyring of=/dev/stdout 2023-05-22T15:26:22.939 INFO:tasks.ceph:Adding keys to all mons... 2023-05-22T15:26:22.939 DEBUG:teuthology.orchestra.run.smithi016:> sudo tee -a /etc/ceph/ceph.keyring 2023-05-22T15:26:22.942 DEBUG:teuthology.orchestra.run.smithi062:> sudo tee -a /etc/ceph/ceph.keyring 2023-05-22T15:26:22.984 INFO:teuthology.orchestra.run.smithi016.stdout:[mgr.y] 2023-05-22T15:26:22.985 INFO:teuthology.orchestra.run.smithi016.stdout: key = AQCAiWtkhgVWExAAjDoh7cexFVfsigsYfGkmVg== 2023-05-22T15:26:22.985 INFO:teuthology.orchestra.run.smithi016.stdout:[mds.a] 2023-05-22T15:26:22.985 INFO:teuthology.orchestra.run.smithi016.stdout: key = AQCAiWtkw/K0GRAA9hvTMKMmcbgKV68GeoC4ag== 2023-05-22T15:26:22.985 INFO:teuthology.orchestra.run.smithi016.stdout:[mds.c] 2023-05-22T15:26:22.985 INFO:teuthology.orchestra.run.smithi016.stdout: key = AQCAiWtkvgOKHxAAMuASPtMqwyoqpZ5I+MRS7A== 2023-05-22T15:26:22.985 INFO:teuthology.orchestra.run.smithi016.stdout:[osd.0] 2023-05-22T15:26:22.985 INFO:teuthology.orchestra.run.smithi016.stdout: key = AQCCiWtk6ZLoCRAAKdCQitZuPnWcIpkiXhJeww== 2023-05-22T15:26:22.986 INFO:teuthology.orchestra.run.smithi016.stdout:[osd.1] 2023-05-22T15:26:22.986 INFO:teuthology.orchestra.run.smithi016.stdout: key = AQCFiWtkXtVeERAAkgPeDKuFmWXrDiCyqWOYwQ== 2023-05-22T15:26:22.986 INFO:teuthology.orchestra.run.smithi016.stdout:[osd.2] 2023-05-22T15:26:22.986 INFO:teuthology.orchestra.run.smithi016.stdout: key = AQCIiWtkdGaRHRAAx3POIp3mnz7p7XcCVMXm5Q== 2023-05-22T15:26:22.986 INFO:teuthology.orchestra.run.smithi016.stdout:[osd.3] 2023-05-22T15:26:22.986 INFO:teuthology.orchestra.run.smithi016.stdout: key = AQCLiWtkSBydJhAAsgc19Xaf/GOyy9nm9MLdJQ== 2023-05-22T15:26:22.986 INFO:teuthology.orchestra.run.smithi016.stdout:[mgr.x] 2023-05-22T15:26:22.987 INFO:teuthology.orchestra.run.smithi016.stdout: key = AQCAiWtkkaIqFhAAb/4BI6FYWOy3qOSaylXLSw== 2023-05-22T15:26:22.987 INFO:teuthology.orchestra.run.smithi016.stdout:[mds.b] 2023-05-22T15:26:22.987 INFO:teuthology.orchestra.run.smithi016.stdout: key = AQCAiWtkoFzfJRAAxCiB0MGM5apMVA81WDq+dA== 2023-05-22T15:26:22.987 INFO:teuthology.orchestra.run.smithi016.stdout:[mds.d] 2023-05-22T15:26:22.987 INFO:teuthology.orchestra.run.smithi016.stdout: key = AQCAiWtk6hPQKxAAlCtfC36vja5ftz6uOqUhHg== 2023-05-22T15:26:22.987 INFO:teuthology.orchestra.run.smithi016.stdout:[osd.4] 2023-05-22T15:26:22.987 INFO:teuthology.orchestra.run.smithi016.stdout: key = AQCRiWtkueYvJRAARXbHlKHrTbgTqBArRJfvBA== 2023-05-22T15:26:22.988 INFO:teuthology.orchestra.run.smithi016.stdout:[osd.5] 2023-05-22T15:26:22.988 INFO:teuthology.orchestra.run.smithi016.stdout: key = AQCUiWtkH3hwLRAAUekP6y+wkJVPh+OPDiZbzQ== 2023-05-22T15:26:22.988 INFO:teuthology.orchestra.run.smithi016.stdout:[osd.6] 2023-05-22T15:26:22.988 INFO:teuthology.orchestra.run.smithi016.stdout: key = AQCXiWtkLsnBNhAAQ8ilX8ZU4ncJ2JJU0I+U4w== 2023-05-22T15:26:22.988 INFO:teuthology.orchestra.run.smithi016.stdout:[osd.7] 2023-05-22T15:26:22.988 INFO:teuthology.orchestra.run.smithi016.stdout: key = AQCbiWtkUvXjBBAApMWw7wdfxWq0WEQDL2qprA== 2023-05-22T15:26:22.988 INFO:teuthology.orchestra.run.smithi016.stdout:[client.0] 2023-05-22T15:26:22.989 INFO:teuthology.orchestra.run.smithi016.stdout: key = AQCAiWtkZwuUMRAAnYqSFiaT/QRsajdjNgTetA== 2023-05-22T15:26:22.989 INFO:teuthology.orchestra.run.smithi016.stdout:[client.2] 2023-05-22T15:26:22.990 INFO:teuthology.orchestra.run.smithi016.stdout: key = AQCAiWtkqkPMNhAAqugrnD4Mh/4At8QAFmzqcg== 2023-05-22T15:26:22.990 INFO:teuthology.orchestra.run.smithi016.stdout:[client.1] 2023-05-22T15:26:22.990 INFO:teuthology.orchestra.run.smithi016.stdout: key = AQCAiWtkPimmORAABzNL2rANz5TgCl3p++GVng== 2023-05-22T15:26:22.990 INFO:teuthology.orchestra.run.smithi016.stdout:[client.3] 2023-05-22T15:26:22.990 INFO:teuthology.orchestra.run.smithi016.stdout: key = AQCBiWtkpvlWAxAAPuedivXBGZGHIO6hSnz8PQ== 2023-05-22T15:26:22.994 INFO:teuthology.orchestra.run.smithi062.stdout:[mgr.y] 2023-05-22T15:26:22.994 INFO:teuthology.orchestra.run.smithi062.stdout: key = AQCAiWtkhgVWExAAjDoh7cexFVfsigsYfGkmVg== 2023-05-22T15:26:22.994 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.a] 2023-05-22T15:26:22.994 INFO:teuthology.orchestra.run.smithi062.stdout: key = AQCAiWtkw/K0GRAA9hvTMKMmcbgKV68GeoC4ag== 2023-05-22T15:26:22.994 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.c] 2023-05-22T15:26:22.994 INFO:teuthology.orchestra.run.smithi062.stdout: key = AQCAiWtkvgOKHxAAMuASPtMqwyoqpZ5I+MRS7A== 2023-05-22T15:26:22.995 INFO:teuthology.orchestra.run.smithi062.stdout:[osd.0] 2023-05-22T15:26:22.995 INFO:teuthology.orchestra.run.smithi062.stdout: key = AQCCiWtk6ZLoCRAAKdCQitZuPnWcIpkiXhJeww== 2023-05-22T15:26:22.995 INFO:teuthology.orchestra.run.smithi062.stdout:[osd.1] 2023-05-22T15:26:22.995 INFO:teuthology.orchestra.run.smithi062.stdout: key = AQCFiWtkXtVeERAAkgPeDKuFmWXrDiCyqWOYwQ== 2023-05-22T15:26:22.995 INFO:teuthology.orchestra.run.smithi062.stdout:[osd.2] 2023-05-22T15:26:22.995 INFO:teuthology.orchestra.run.smithi062.stdout: key = AQCIiWtkdGaRHRAAx3POIp3mnz7p7XcCVMXm5Q== 2023-05-22T15:26:22.995 INFO:teuthology.orchestra.run.smithi062.stdout:[osd.3] 2023-05-22T15:26:22.995 INFO:teuthology.orchestra.run.smithi062.stdout: key = AQCLiWtkSBydJhAAsgc19Xaf/GOyy9nm9MLdJQ== 2023-05-22T15:26:22.996 INFO:teuthology.orchestra.run.smithi062.stdout:[mgr.x] 2023-05-22T15:26:22.996 INFO:teuthology.orchestra.run.smithi062.stdout: key = AQCAiWtkkaIqFhAAb/4BI6FYWOy3qOSaylXLSw== 2023-05-22T15:26:22.996 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.b] 2023-05-22T15:26:22.996 INFO:teuthology.orchestra.run.smithi062.stdout: key = AQCAiWtkoFzfJRAAxCiB0MGM5apMVA81WDq+dA== 2023-05-22T15:26:22.996 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.d] 2023-05-22T15:26:22.996 INFO:teuthology.orchestra.run.smithi062.stdout: key = AQCAiWtk6hPQKxAAlCtfC36vja5ftz6uOqUhHg== 2023-05-22T15:26:22.996 INFO:teuthology.orchestra.run.smithi062.stdout:[osd.4] 2023-05-22T15:26:22.996 INFO:teuthology.orchestra.run.smithi062.stdout: key = AQCRiWtkueYvJRAARXbHlKHrTbgTqBArRJfvBA== 2023-05-22T15:26:22.997 INFO:teuthology.orchestra.run.smithi062.stdout:[osd.5] 2023-05-22T15:26:22.997 INFO:teuthology.orchestra.run.smithi062.stdout: key = AQCUiWtkH3hwLRAAUekP6y+wkJVPh+OPDiZbzQ== 2023-05-22T15:26:22.997 INFO:teuthology.orchestra.run.smithi062.stdout:[osd.6] 2023-05-22T15:26:22.997 INFO:teuthology.orchestra.run.smithi062.stdout: key = AQCXiWtkLsnBNhAAQ8ilX8ZU4ncJ2JJU0I+U4w== 2023-05-22T15:26:22.997 INFO:teuthology.orchestra.run.smithi062.stdout:[osd.7] 2023-05-22T15:26:22.997 INFO:teuthology.orchestra.run.smithi062.stdout: key = AQCbiWtkUvXjBBAApMWw7wdfxWq0WEQDL2qprA== 2023-05-22T15:26:22.997 INFO:teuthology.orchestra.run.smithi062.stdout:[client.0] 2023-05-22T15:26:22.998 INFO:teuthology.orchestra.run.smithi062.stdout: key = AQCAiWtkZwuUMRAAnYqSFiaT/QRsajdjNgTetA== 2023-05-22T15:26:22.998 INFO:teuthology.orchestra.run.smithi062.stdout:[client.2] 2023-05-22T15:26:22.998 INFO:teuthology.orchestra.run.smithi062.stdout: key = AQCAiWtkqkPMNhAAqugrnD4Mh/4At8QAFmzqcg== 2023-05-22T15:26:22.998 INFO:teuthology.orchestra.run.smithi062.stdout:[client.1] 2023-05-22T15:26:22.998 INFO:teuthology.orchestra.run.smithi062.stdout: key = AQCAiWtkPimmORAABzNL2rANz5TgCl3p++GVng== 2023-05-22T15:26:22.998 INFO:teuthology.orchestra.run.smithi062.stdout:[client.3] 2023-05-22T15:26:22.998 INFO:teuthology.orchestra.run.smithi062.stdout: key = AQCBiWtkpvlWAxAAPuedivXBGZGHIO6hSnz8PQ== 2023-05-22T15:26:23.000 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-05-22T15:26:23.036 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-05-22T15:26:23.089 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-05-22T15:26:23.091 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-05-22T15:26:23.136 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-05-22T15:26:23.177 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-05-22T15:26:23.225 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-05-22T15:26:23.228 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-05-22T15:26:23.308 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-05-22T15:26:23.313 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-05-22T15:26:23.393 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-05-22T15:26:23.396 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-05-22T15:26:23.474 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-05-22T15:26:23.480 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-05-22T15:26:23.558 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-05-22T15:26:23.568 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-05-22T15:26:23.641 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-05-22T15:26:23.657 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-05-22T15:26:23.729 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-05-22T15:26:23.732 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-05-22T15:26:23.812 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-05-22T15:26:23.816 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-05-22T15:26:23.893 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-05-22T15:26:23.896 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-05-22T15:26:23.980 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-05-22T15:26:23.983 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-05-22T15:26:24.025 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-05-22T15:26:24.068 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-05-22T15:26:24.111 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-05-22T15:26:24.153 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-05-22T15:26:24.200 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-05-22T15:26:24.203 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-05-22T15:26:24.247 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-05-22T15:26:24.289 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-05-22T15:26:24.336 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-05-22T15:26:24.338 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-05-22T15:26:24.427 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-05-22T15:26:24.428 DEBUG:teuthology.orchestra.run.smithi016:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2023-05-22T15:26:24.440 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-05-22T15:26:24.536 DEBUG:teuthology.orchestra.run.smithi016:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2023-05-22T15:26:24.547 DEBUG:teuthology.orchestra.run.smithi062:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2023-05-22T15:26:24.562 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-05-22T15:26:24.664 DEBUG:teuthology.orchestra.run.smithi062:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2023-05-22T15:26:24.675 DEBUG:teuthology.orchestra.run.smithi062:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2023-05-22T15:26:24.730 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-05-22T15:26:24.829 DEBUG:teuthology.orchestra.run.smithi062:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2023-05-22T15:26:24.840 DEBUG:teuthology.orchestra.run.smithi016:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-05-22T15:26:24.843 DEBUG:teuthology.orchestra.run.smithi062:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-05-22T15:26:24.885 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2023-05-22T15:26:24.886 INFO:tasks.ceph.mon.a:Restarting daemon 2023-05-22T15:26:24.886 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2023-05-22T15:26:24.889 INFO:tasks.ceph.mon.a:Started 2023-05-22T15:26:24.889 INFO:tasks.ceph.mon.b:Restarting daemon 2023-05-22T15:26:24.889 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2023-05-22T15:26:24.928 INFO:tasks.ceph.mon.b:Started 2023-05-22T15:26:24.929 INFO:tasks.ceph.mon.c:Restarting daemon 2023-05-22T15:26:24.930 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2023-05-22T15:26:24.933 INFO:tasks.ceph.mon.c:Started 2023-05-22T15:26:24.933 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2023-05-22T15:26:24.934 INFO:tasks.ceph.mgr.y:Restarting daemon 2023-05-22T15:26:24.934 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2023-05-22T15:26:24.936 INFO:tasks.ceph.mgr.y:Started 2023-05-22T15:26:24.937 INFO:tasks.ceph.mgr.x:Restarting daemon 2023-05-22T15:26:24.937 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2023-05-22T15:26:24.939 INFO:tasks.ceph.mgr.x:Started 2023-05-22T15:26:24.940 INFO:tasks.ceph:Setting crush tunables to default 2023-05-22T15:26:24.940 DEBUG:teuthology.orchestra.run.smithi016:> sudo ceph --cluster ceph osd crush tunables default 2023-05-22T15:26:25.061 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:25.050+0000 7f9290ee7640 1 Processor -- start 2023-05-22T15:26:25.061 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:25.050+0000 7f9290ee7640 1 -- start start 2023-05-22T15:26:25.062 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:25.050+0000 7f9290ee7640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f928c159a90 0x7f928c0cbb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:25.062 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:25.050+0000 7f9290ee7640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f928c0cd770 0x7f928c15dbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:25.062 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:25.050+0000 7f9290ee7640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f928c159e60 con 0x7f928c15e130 2023-05-22T15:26:25.062 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:25.050+0000 7f9290ee7640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f928c15d4f0 con 0x7f928c159a90 2023-05-22T15:26:25.062 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:25.050+0000 7f9290ee7640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f928c0cc1d0 con 0x7f928c0cd770 2023-05-22T15:26:25.063 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:25.050+0000 7f9289d9b640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f928c0cd770 0x7f928c15dbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:25.063 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:25.050+0000 7f928ad9d640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f928c15e130 0x7f928c0cdd90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.16:42682/0 (socket says 172.21.15.16:42682) 2023-05-22T15:26:25.063 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:25.050+0000 7f928ad9d640 1 -- 172.21.15.16:0/1608529784 learned_addr learned my addr 172.21.15.16:0/1608529784 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:26:25.063 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:25.050+0000 7f928a59c640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f928c159a90 0x7f928c0cbb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:25.063 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:25.050+0000 7f9289d9b640 1 -- 172.21.15.16:0/1608529784 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f928c0cd770 msgr2=0x7f928c15dbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-05-22T15:26:25.063 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:25.050+0000 7f9289d9b640 1 -- 172.21.15.16:0/1608529784 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f928c0cd770 msgr2=0x7f928c15dbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-05-22T15:26:25.063 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:25.050+0000 7f928a59c640 1 -- 172.21.15.16:0/1608529784 >> v2:172.21.15.62:3300/0 conn(0x7f928c159a90 msgr2=0x7f928c0cbb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-05-22T15:26:25.064 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:25.050+0000 7f928a59c640 1 -- 172.21.15.16:0/1608529784 >> v2:172.21.15.62:3300/0 conn(0x7f928c159a90 msgr2=0x7f928c0cbb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-05-22T15:26:25.064 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:25.050+0000 7f9289d9b640 1 --2- 172.21.15.16:0/1608529784 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f928c0cd770 0x7f928c15dbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-05-22T15:26:25.064 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:25.050+0000 7f928a59c640 1 --2- 172.21.15.16:0/1608529784 >> v2:172.21.15.62:3300/0 conn(0x7f928c159a90 0x7f928c0cbb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-05-22T15:26:25.064 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:25.050+0000 7f9289d9b640 1 --2- 172.21.15.16:0/1608529784 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f928c0cd770 0x7f928c15dbf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-05-22T15:26:25.064 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:25.050+0000 7f928a59c640 1 --2- 172.21.15.16:0/1608529784 >> v2:172.21.15.62:3300/0 conn(0x7f928c159a90 0x7f928c0cbb30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-05-22T15:26:25.264 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:25.254+0000 7f928a59c640 1 --2- 172.21.15.16:0/1608529784 >> v2:172.21.15.62:3300/0 conn(0x7f928c159a90 0x7f928c0cbb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:25.264 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:25.254+0000 7f9289d9b640 1 --2- 172.21.15.16:0/1608529784 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f928c0cd770 0x7f928c15dbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:25.264 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:25.254+0000 7f928a59c640 1 -- 172.21.15.16:0/1608529784 >> v2:172.21.15.62:3300/0 conn(0x7f928c159a90 msgr2=0x7f928c0cbb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-05-22T15:26:25.264 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:25.254+0000 7f928a59c640 1 -- 172.21.15.16:0/1608529784 >> v2:172.21.15.62:3300/0 conn(0x7f928c159a90 msgr2=0x7f928c0cbb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-05-22T15:26:25.265 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:25.254+0000 7f928a59c640 1 --2- 172.21.15.16:0/1608529784 >> v2:172.21.15.62:3300/0 conn(0x7f928c159a90 0x7f928c0cbb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-05-22T15:26:25.265 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:25.254+0000 7f928a59c640 1 --2- 172.21.15.16:0/1608529784 >> v2:172.21.15.62:3300/0 conn(0x7f928c159a90 0x7f928c0cbb30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-05-22T15:26:25.265 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:25.254+0000 7f9289d9b640 1 -- 172.21.15.16:0/1608529784 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f928c0cd770 msgr2=0x7f928c15dbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-05-22T15:26:25.265 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:25.254+0000 7f9289d9b640 1 -- 172.21.15.16:0/1608529784 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f928c0cd770 msgr2=0x7f928c15dbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-05-22T15:26:25.265 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:25.254+0000 7f9289d9b640 1 --2- 172.21.15.16:0/1608529784 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f928c0cd770 0x7f928c15dbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-05-22T15:26:25.265 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:25.254+0000 7f9289d9b640 1 --2- 172.21.15.16:0/1608529784 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f928c0cd770 0x7f928c15dbf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-05-22T15:26:25.335 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:25.322+0000 7f928959a640 1 -- 172.21.15.16:0/1608529784 <== mon.1 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2245029347 0 0) 0x7f928c159e60 con 0x7f928c15e130 2023-05-22T15:26:25.335 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:25.322+0000 7f928959a640 1 -- 172.21.15.16:0/1608529784 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9274003530 con 0x7f928c15e130 2023-05-22T15:26:25.336 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:25.326+0000 7f928959a640 1 -- 172.21.15.16:0/1608529784 <== mon.1 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2348246691 0 0) 0x7f9274003530 con 0x7f928c15e130 2023-05-22T15:26:25.336 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:25.326+0000 7f928959a640 1 -- 172.21.15.16:0/1608529784 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f928c0cd770 msgr2=0x7f928c15dbf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-05-22T15:26:25.336 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:25.326+0000 7f928959a640 1 --2- 172.21.15.16:0/1608529784 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f928c0cd770 0x7f928c15dbf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:25.336 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:25.326+0000 7f928959a640 1 -- 172.21.15.16:0/1608529784 >> v2:172.21.15.62:3300/0 conn(0x7f928c159a90 msgr2=0x7f928c0cbb30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-05-22T15:26:25.337 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:25.326+0000 7f928959a640 1 --2- 172.21.15.16:0/1608529784 >> v2:172.21.15.62:3300/0 conn(0x7f928c159a90 0x7f928c0cbb30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:25.337 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:25.326+0000 7f928959a640 1 -- 172.21.15.16:0/1608529784 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f928c0cc4b0 con 0x7f928c15e130 2023-05-22T15:26:25.337 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:25.326+0000 7f928959a640 1 -- 172.21.15.16:0/1608529784 <== mon.1 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7f9284001a00 con 0x7f928c15e130 2023-05-22T15:26:25.338 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:25.326+0000 7f928959a640 1 -- 172.21.15.16:0/1608529784 >> v1:172.21.15.62:6789/0 conn(0x7f928c15e130 legacy=0x7f928c0cdd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:25.338 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:25.326+0000 7f928959a640 1 --2- 172.21.15.16:0/1608529784 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9274003980 0x7f9274003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:25.338 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:25.326+0000 7f928959a640 1 --2- 172.21.15.16:0/1608529784 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f928c0cd770 0x7f92740047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:25.338 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:25.326+0000 7f928959a640 1 --2- 172.21.15.16:0/1608529784 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9274004d10 0x7f9274005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:25.338 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:25.326+0000 7f928959a640 1 -- 172.21.15.16:0/1608529784 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f928c15d4f0 con 0x7f9274004d10 2023-05-22T15:26:25.338 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:25.326+0000 7f928959a640 1 -- 172.21.15.16:0/1608529784 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f928c0cc1d0 con 0x7f9274003980 2023-05-22T15:26:25.339 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:25.326+0000 7f928959a640 1 -- 172.21.15.16:0/1608529784 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f927400baf0 con 0x7f928c0cd770 2023-05-22T15:26:25.339 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:25.326+0000 7f928ad9d640 1 --2- 172.21.15.16:0/1608529784 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9274004d10 0x7f9274005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:25.339 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:25.326+0000 7f9289d9b640 1 --2- 172.21.15.16:0/1608529784 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f928c0cd770 0x7f92740047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:25.339 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:25.326+0000 7f928a59c640 1 --2- 172.21.15.16:0/1608529784 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9274003980 0x7f9274003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:25.339 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:25.326+0000 7f9289d9b640 1 -- 172.21.15.16:0/1608529784 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9274003980 msgr2=0x7f9274003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:26:25.339 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:25.326+0000 7f9289d9b640 1 --2- 172.21.15.16:0/1608529784 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9274003980 0x7f9274003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:25.339 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:25.326+0000 7f9289d9b640 1 -- 172.21.15.16:0/1608529784 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9274004d10 msgr2=0x7f9274005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:26:25.340 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:25.326+0000 7f9289d9b640 1 --2- 172.21.15.16:0/1608529784 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9274004d10 0x7f9274005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:25.340 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:25.326+0000 7f9289d9b640 1 -- 172.21.15.16:0/1608529784 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f928c0cc4b0 con 0x7f928c0cd770 2023-05-22T15:26:25.342 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:25.330+0000 7f9289d9b640 1 --2- 172.21.15.16:0/1608529784 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f928c0cd770 0x7f92740047d0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f928000e8d0 tx=0x7f928001c890 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2fe6f05785bef43a server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:26:25.343 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:25.330+0000 7f928959a640 1 -- 172.21.15.16:0/1608529784 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92800a5790 con 0x7f928c0cd770 2023-05-22T15:26:25.343 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:25.330+0000 7f928959a640 1 -- 172.21.15.16:0/1608529784 <== mon.2 v2:172.21.15.62:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92800a9070 con 0x7f928c0cd770 2023-05-22T15:26:25.343 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:25.330+0000 7f928959a640 1 -- 172.21.15.16:0/1608529784 <== mon.2 v2:172.21.15.62:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92800a5b30 con 0x7f928c0cd770 2023-05-22T15:26:25.344 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:25.334+0000 7f9290ee7640 1 -- 172.21.15.16:0/1608529784 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f928c0cd770 msgr2=0x7f92740047d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:25.344 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:25.334+0000 7f9290ee7640 1 --2- 172.21.15.16:0/1608529784 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f928c0cd770 0x7f92740047d0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f928000e8d0 tx=0x7f928001c890 comp rx=0 tx=0).stop 2023-05-22T15:26:25.344 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:25.334+0000 7f928ad9d640 1 -- 172.21.15.16:0/1608529784 reap_dead start 2023-05-22T15:26:25.345 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:25.334+0000 7f9290ee7640 1 -- 172.21.15.16:0/1608529784 shutdown_connections 2023-05-22T15:26:25.345 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:25.334+0000 7f9290ee7640 1 -- 172.21.15.16:0/1608529784 >> 172.21.15.16:0/1608529784 conn(0x7f928c06f4f0 msgr2=0x7f928c06f8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:26:25.345 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:25.334+0000 7f9290ee7640 1 -- 172.21.15.16:0/1608529784 shutdown_connections 2023-05-22T15:26:25.345 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:25.334+0000 7f9290ee7640 1 -- 172.21.15.16:0/1608529784 wait complete. 2023-05-22T15:26:25.347 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:25.338+0000 7f9290ee7640 1 Processor -- start 2023-05-22T15:26:25.347 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:25.338+0000 7f9290ee7640 1 -- start start 2023-05-22T15:26:25.350 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:25.338+0000 7f9290ee7640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f928c0cd770 0x7f928c1f1f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:25.350 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:25.338+0000 7f9290ee7640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f928c0ce4e0 0x7f928c0c6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:25.350 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:25.338+0000 7f9290ee7640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f928c0ce950 0x7f928c153db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:25.350 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:25.338+0000 7f9290ee7640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f928c06a830 con 0x7f928c0ce950 2023-05-22T15:26:25.350 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:25.338+0000 7f9290ee7640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f928c06ab10 con 0x7f928c0cd770 2023-05-22T15:26:25.350 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:25.338+0000 7f9290ee7640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f928c06adf0 con 0x7f928c0ce4e0 2023-05-22T15:26:25.351 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:25.338+0000 7f928ad9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f928c0ce950 0x7f928c153db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:25.351 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:25.338+0000 7f928ad9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f928c0ce950 0x7f928c153db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47300/0 (socket says 172.21.15.16:47300) 2023-05-22T15:26:25.351 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:25.338+0000 7f928ad9d640 1 -- 172.21.15.16:0/3028789350 learned_addr learned my addr 172.21.15.16:0/3028789350 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:26:25.351 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:25.338+0000 7f928a59c640 1 --2- 172.21.15.16:0/3028789350 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f928c0cd770 0x7f928c1f1f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:25.351 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:25.338+0000 7f9289d9b640 1 --2- 172.21.15.16:0/3028789350 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f928c0ce4e0 0x7f928c0c6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:25.351 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:25.338+0000 7f928ad9d640 1 -- 172.21.15.16:0/3028789350 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f928c0ce4e0 msgr2=0x7f928c0c6300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:25.351 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:25.338+0000 7f928ad9d640 1 --2- 172.21.15.16:0/3028789350 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f928c0ce4e0 0x7f928c0c6300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:25.352 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:25.338+0000 7f928ad9d640 1 -- 172.21.15.16:0/3028789350 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f928c0cd770 msgr2=0x7f928c1f1f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:25.352 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:25.338+0000 7f928ad9d640 1 --2- 172.21.15.16:0/3028789350 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f928c0cd770 0x7f928c1f1f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:25.352 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:25.338+0000 7f928ad9d640 1 -- 172.21.15.16:0/3028789350 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f928c15d6d0 con 0x7f928c0ce950 2023-05-22T15:26:25.352 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:25.338+0000 7f928a59c640 1 --2- 172.21.15.16:0/3028789350 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f928c0cd770 0x7f928c1f1f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T15:26:25.352 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:25.338+0000 7f9289d9b640 1 --2- 172.21.15.16:0/3028789350 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f928c0ce4e0 0x7f928c0c6300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T15:26:25.352 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:25.338+0000 7f928ad9d640 1 --2- 172.21.15.16:0/3028789350 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f928c0ce950 0x7f928c153db0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f928c071640 tx=0x7f9284007a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:26:25.353 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:25.338+0000 7f927affd640 1 -- 172.21.15.16:0/3028789350 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9284014630 con 0x7f928c0ce950 2023-05-22T15:26:25.353 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:25.338+0000 7f927affd640 1 -- 172.21.15.16:0/3028789350 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9284014790 con 0x7f928c0ce950 2023-05-22T15:26:25.354 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:25.338+0000 7f927affd640 1 -- 172.21.15.16:0/3028789350 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9284014b00 con 0x7f928c0ce950 2023-05-22T15:26:25.354 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:25.338+0000 7f9290ee7640 1 -- 172.21.15.16:0/3028789350 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f928c0cc790 con 0x7f928c0ce950 2023-05-22T15:26:25.354 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:25.338+0000 7f9290ee7640 1 -- 172.21.15.16:0/3028789350 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f928c0cd2e0 con 0x7f928c0ce950 2023-05-22T15:26:25.354 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:25.342+0000 7f927affd640 1 -- 172.21.15.16:0/3028789350 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f9284014c60 con 0x7f928c0ce950 2023-05-22T15:26:25.354 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:25.342+0000 7f927affd640 1 -- 172.21.15.16:0/3028789350 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f928401fd00 con 0x7f928c0ce950 2023-05-22T15:26:25.354 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:25.342+0000 7f9290ee7640 1 -- 172.21.15.16:0/3028789350 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f928c04dbe0 con 0x7f928c0ce950 2023-05-22T15:26:25.355 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:25.342+0000 7f927affd640 1 -- 172.21.15.16:0/3028789350 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72422 (secure 0 0 0) 0x7f9284010040 con 0x7f928c0ce950 2023-05-22T15:26:25.423 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:25.414+0000 7f9290ee7640 1 -- 172.21.15.16:0/3028789350 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f928c0c7650 con 0x7f928c0ce950 2023-05-22T15:26:25.429 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:25.418+0000 7f927affd640 1 -- 172.21.15.16:0/3028789350 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (secure 0 0 0) 0x7f928401f460 con 0x7f928c0ce950 2023-05-22T15:26:25.430 INFO:teuthology.orchestra.run.smithi016.stderr:adjusted tunables profile to default 2023-05-22T15:26:25.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:25.418+0000 7f9290ee7640 1 -- 172.21.15.16:0/3028789350 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f928c0ce950 msgr2=0x7f928c153db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:25.432 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:25.418+0000 7f9290ee7640 1 --2- 172.21.15.16:0/3028789350 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f928c0ce950 0x7f928c153db0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f928c071640 tx=0x7f9284007a10 comp rx=0 tx=0).stop 2023-05-22T15:26:25.432 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:25.418+0000 7f9290ee7640 1 -- 172.21.15.16:0/3028789350 shutdown_connections 2023-05-22T15:26:25.432 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:25.418+0000 7f9290ee7640 1 --2- 172.21.15.16:0/3028789350 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f928c0ce950 0x7f928c153db0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:25.433 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:25.418+0000 7f9290ee7640 1 --2- 172.21.15.16:0/3028789350 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f928c0ce4e0 0x7f928c0c6300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:25.433 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:25.418+0000 7f9290ee7640 1 --2- 172.21.15.16:0/3028789350 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f928c0cd770 0x7f928c1f1f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:25.433 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:25.418+0000 7f9290ee7640 1 -- 172.21.15.16:0/3028789350 >> 172.21.15.16:0/3028789350 conn(0x7f928c06f4f0 msgr2=0x7f928c0c8160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:26:25.434 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:25.418+0000 7f9290ee7640 1 -- 172.21.15.16:0/3028789350 shutdown_connections 2023-05-22T15:26:25.434 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:25.418+0000 7f9290ee7640 1 -- 172.21.15.16:0/3028789350 wait complete. 2023-05-22T15:26:25.449 INFO:tasks.ceph:check_enable_crimson: False 2023-05-22T15:26:25.449 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2023-05-22T15:26:25.450 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-05-22T15:26:25.450 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2023-05-22T15:26:25.460 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-05-22T15:26:25.460 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2023-05-22T15:26:25.516 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-05-22T15:26:25.517 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2023-05-22T15:26:25.526 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:26:25.514+0000 7f723d3e1180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-05-22T15:26:25.536 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-05-22T15:26:25.537 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2023-05-22T15:26:25.546 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:26:25.537+0000 7f90244ca180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-05-22T15:26:25.593 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2023-05-22T15:26:25.593 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2023-05-22T15:26:25.607 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2023-05-22T15:26:25.607 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2023-05-22T15:26:25.662 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2023-05-22T15:26:25.663 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2023-05-22T15:26:25.718 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2023-05-22T15:26:25.718 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2023-05-22T15:26:25.774 DEBUG:teuthology.orchestra.run.smithi062:> sudo ceph --cluster ceph osd new 9e7ff854-b25f-4cbb-a5c2-1799d9c53d2d 0 2023-05-22T15:26:25.965 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:25.953+0000 7feb76a88640 1 Processor -- start 2023-05-22T15:26:25.966 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:25.953+0000 7feb76a88640 1 -- start start 2023-05-22T15:26:25.967 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:25.957+0000 7feb76a88640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7feb700705d0 0x7feb700cda20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:25.967 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:25.957+0000 7feb76a88640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb700ce6c0 con 0x7feb7006ac00 2023-05-22T15:26:25.967 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:25.957+0000 7feb76a88640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb700ce100 con 0x7feb7015bdf0 2023-05-22T15:26:25.967 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:25.957+0000 7feb76a88640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb7015c1c0 con 0x7feb700705d0 2023-05-22T15:26:25.968 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:25.957+0000 7feb75025640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7feb700705d0 0x7feb700cda20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:25.968 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:25.957+0000 7feb75025640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7feb700705d0 0x7feb700cda20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3301/0 says I am v2:172.21.15.62:33866/0 (socket says 172.21.15.62:33866) 2023-05-22T15:26:25.968 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:25.957+0000 7feb75025640 1 -- 172.21.15.62:0/3921359147 learned_addr learned my addr 172.21.15.62:0/3921359147 (peer_addr_for_me v2:172.21.15.62:0/0) 2023-05-22T15:26:25.968 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:25.957+0000 7feb75025640 1 -- 172.21.15.62:0/3921359147 >> v1:172.21.15.62:6789/0 conn(0x7feb7015bdf0 legacy=0x7feb7006c080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:25.968 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:25.957+0000 7feb75025640 1 -- 172.21.15.62:0/3921359147 >> v1:172.21.15.16:6789/0 conn(0x7feb7006ac00 legacy=0x7feb7006fec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:26:25.968 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:25.957+0000 7feb75025640 1 -- 172.21.15.62:0/3921359147 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb700ce3e0 con 0x7feb700705d0 2023-05-22T15:26:25.970 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:25.957+0000 7feb75025640 1 --2- 172.21.15.62:0/3921359147 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7feb700705d0 0x7feb700cda20 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7feb68009e80 tx=0x7feb6801c8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=34ff77e10c3183b3 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:26:25.970 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:25.957+0000 7feb677fe640 1 -- 172.21.15.62:0/3921359147 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb680a5760 con 0x7feb700705d0 2023-05-22T15:26:25.970 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:25.961+0000 7feb677fe640 1 -- 172.21.15.62:0/3921359147 <== mon.2 v2:172.21.15.62:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb680a58c0 con 0x7feb700705d0 2023-05-22T15:26:25.970 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:25.961+0000 7feb677fe640 1 -- 172.21.15.62:0/3921359147 <== mon.2 v2:172.21.15.62:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb680a5c50 con 0x7feb700705d0 2023-05-22T15:26:25.971 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:25.961+0000 7feb76a88640 1 -- 172.21.15.62:0/3921359147 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7feb700705d0 msgr2=0x7feb700cda20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:25.971 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:25.961+0000 7feb76a88640 1 --2- 172.21.15.62:0/3921359147 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7feb700705d0 0x7feb700cda20 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7feb68009e80 tx=0x7feb6801c8a0 comp rx=0 tx=0).stop 2023-05-22T15:26:25.971 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:25.961+0000 7feb76a88640 1 -- 172.21.15.62:0/3921359147 shutdown_connections 2023-05-22T15:26:25.972 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:25.961+0000 7feb76a88640 1 --2- 172.21.15.62:0/3921359147 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7feb700705d0 0x7feb700cda20 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:25.972 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:25.961+0000 7feb76a88640 1 -- 172.21.15.62:0/3921359147 >> 172.21.15.62:0/3921359147 conn(0x7feb700ce890 msgr2=0x7feb700cec90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:26:25.972 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:25.961+0000 7feb76a88640 1 -- 172.21.15.62:0/3921359147 shutdown_connections 2023-05-22T15:26:25.972 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:25.961+0000 7feb76a88640 1 -- 172.21.15.62:0/3921359147 wait complete. 2023-05-22T15:26:25.972 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:25.961+0000 7feb76a88640 1 Processor -- start 2023-05-22T15:26:25.973 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:25.961+0000 7feb76a88640 1 -- start start 2023-05-22T15:26:25.974 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:25.961+0000 7feb76a88640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feb7006ac00 0x7feb70152e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:25.975 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:25.961+0000 7feb76a88640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7feb700705d0 0x7feb700c4d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:25.975 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:25.961+0000 7feb76a88640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7feb7015bdf0 0x7feb700b2dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:25.975 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:25.961+0000 7feb76a88640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb700c48a0 con 0x7feb7006ac00 2023-05-22T15:26:25.975 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:25.961+0000 7feb76a88640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb700c3b80 con 0x7feb7015bdf0 2023-05-22T15:26:25.975 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:25.961+0000 7feb76a88640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb700c3f70 con 0x7feb700705d0 2023-05-22T15:26:25.975 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:25.965+0000 7feb75025640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7feb7015bdf0 0x7feb700b2dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:25.975 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:25.965+0000 7feb67fff640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7feb700705d0 0x7feb700c4d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:25.975 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:25.965+0000 7feb75025640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7feb7015bdf0 0x7feb700b2dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:38024/0 (socket says 172.21.15.62:38024) 2023-05-22T15:26:25.976 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:25.965+0000 7feb67fff640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7feb700705d0 0x7feb700c4d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3301/0 says I am v2:172.21.15.62:33878/0 (socket says 172.21.15.62:33878) 2023-05-22T15:26:25.976 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:25.965+0000 7feb75025640 1 -- 172.21.15.62:0/2957720284 learned_addr learned my addr 172.21.15.62:0/2957720284 (peer_addr_for_me v2:172.21.15.62:0/0) 2023-05-22T15:26:25.976 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:25.965+0000 7feb74824640 1 --2- 172.21.15.62:0/2957720284 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feb7006ac00 0x7feb70152e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:25.976 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:25.965+0000 7feb75025640 1 -- 172.21.15.62:0/2957720284 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7feb700705d0 msgr2=0x7feb700c4d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:25.976 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:25.965+0000 7feb75025640 1 --2- 172.21.15.62:0/2957720284 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7feb700705d0 0x7feb700c4d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:25.976 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:25.965+0000 7feb75025640 1 -- 172.21.15.62:0/2957720284 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feb7006ac00 msgr2=0x7feb70152e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:25.976 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:25.965+0000 7feb75025640 1 --2- 172.21.15.62:0/2957720284 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feb7006ac00 0x7feb70152e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:25.977 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:25.965+0000 7feb75025640 1 -- 172.21.15.62:0/2957720284 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb68009b30 con 0x7feb7015bdf0 2023-05-22T15:26:25.977 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:25.965+0000 7feb75025640 1 --2- 172.21.15.62:0/2957720284 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7feb7015bdf0 0x7feb700b2dd0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7feb680af040 tx=0x7feb68019640 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:26:25.977 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:25.965+0000 7feb657fa640 1 -- 172.21.15.62:0/2957720284 <== mon.1 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb680b8670 con 0x7feb7015bdf0 2023-05-22T15:26:25.978 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:25.965+0000 7feb657fa640 1 -- 172.21.15.62:0/2957720284 <== mon.1 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb680b87d0 con 0x7feb7015bdf0 2023-05-22T15:26:25.978 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:25.965+0000 7feb657fa640 1 -- 172.21.15.62:0/2957720284 <== mon.1 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb680b8b80 con 0x7feb7015bdf0 2023-05-22T15:26:25.978 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:25.965+0000 7feb76a88640 1 -- 172.21.15.62:0/2957720284 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb70069f30 con 0x7feb7015bdf0 2023-05-22T15:26:25.978 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:25.965+0000 7feb76a88640 1 -- 172.21.15.62:0/2957720284 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb700c8be0 con 0x7feb7015bdf0 2023-05-22T15:26:25.978 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:25.965+0000 7feb657fa640 1 -- 172.21.15.62:0/2957720284 <== mon.1 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7feb680c1400 con 0x7feb7015bdf0 2023-05-22T15:26:25.978 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:25.965+0000 7feb657fa640 1 -- 172.21.15.62:0/2957720284 <== mon.1 v2:172.21.15.62:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (secure 0 0 0) 0x7feb680b6480 con 0x7feb7015bdf0 2023-05-22T15:26:25.979 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:25.965+0000 7feb76a88640 1 -- 172.21.15.62:0/2957720284 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb7006a0c0 con 0x7feb7015bdf0 2023-05-22T15:26:25.989 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:25.977+0000 7feb657fa640 1 -- 172.21.15.62:0/2957720284 <== mon.1 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72422 (secure 0 0 0) 0x7feb680dfb00 con 0x7feb7015bdf0 2023-05-22T15:26:26.064 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:26:26.054+0000 7f723d3e1180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-05-22T15:26:26.079 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.069+0000 7feb76a88640 1 -- 172.21.15.62:0/2957720284 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd new", "uuid": "9e7ff854-b25f-4cbb-a5c2-1799d9c53d2d", "id": 0} v 0) v1 -- 0x7feb700cd0f0 con 0x7feb7015bdf0 2023-05-22T15:26:26.087 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.077+0000 7feb657fa640 1 -- 172.21.15.62:0/2957720284 <== mon.1 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "9e7ff854-b25f-4cbb-a5c2-1799d9c53d2d", "id": 0}]=0 v3) v1 ==== 112+0+2 (secure 0 0 0) 0x7feb680ce030 con 0x7feb7015bdf0 2023-05-22T15:26:26.088 INFO:teuthology.orchestra.run.smithi062.stdout:0 2023-05-22T15:26:26.089 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.077+0000 7feb76a88640 1 -- 172.21.15.62:0/2957720284 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7feb7015bdf0 msgr2=0x7feb700b2dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:26.089 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.077+0000 7feb76a88640 1 --2- 172.21.15.62:0/2957720284 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7feb7015bdf0 0x7feb700b2dd0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7feb680af040 tx=0x7feb68019640 comp rx=0 tx=0).stop 2023-05-22T15:26:26.089 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.077+0000 7feb76a88640 1 -- 172.21.15.62:0/2957720284 shutdown_connections 2023-05-22T15:26:26.089 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.077+0000 7feb76a88640 1 --2- 172.21.15.62:0/2957720284 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7feb7015bdf0 0x7feb700b2dd0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:26.089 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.077+0000 7feb76a88640 1 --2- 172.21.15.62:0/2957720284 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7feb700705d0 0x7feb700c4d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:26.089 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.077+0000 7feb76a88640 1 --2- 172.21.15.62:0/2957720284 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feb7006ac00 0x7feb70152e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:26.089 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.077+0000 7feb76a88640 1 -- 172.21.15.62:0/2957720284 >> 172.21.15.62:0/2957720284 conn(0x7feb700ce890 msgr2=0x7feb700cec60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:26:26.090 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.077+0000 7feb76a88640 1 -- 172.21.15.62:0/2957720284 shutdown_connections 2023-05-22T15:26:26.090 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.077+0000 7feb76a88640 1 -- 172.21.15.62:0/2957720284 wait complete. 2023-05-22T15:26:26.104 DEBUG:teuthology.orchestra.run.smithi062:> sudo ceph --cluster ceph osd new 4702fbc9-a927-4565-aca5-41dc69cc8264 1 2023-05-22T15:26:26.129 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:26:26.117+0000 7f90244ca180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-05-22T15:26:26.214 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.205+0000 7f2e2329b640 1 Processor -- start 2023-05-22T15:26:26.215 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.205+0000 7f2e2329b640 1 -- start start 2023-05-22T15:26:26.216 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.205+0000 7f2e2329b640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2e1c15cfa0 0x7f2e1c06db00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:26.216 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.205+0000 7f2e2329b640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f2e1c06e740 0x7f2e1c06ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:26.216 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.205+0000 7f2e2329b640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e1c06e040 con 0x7f2e1c0cf870 2023-05-22T15:26:26.217 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.205+0000 7f2e2329b640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e1c0cb9f0 con 0x7f2e1c06e740 2023-05-22T15:26:26.217 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.205+0000 7f2e2329b640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e1c06f100 con 0x7f2e1c15cfa0 2023-05-22T15:26:26.217 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.205+0000 7f2e20836640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f2e1c0cf870 0x7f2e1c0b1fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:42222/0 (socket says 172.21.15.62:42222) 2023-05-22T15:26:26.217 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.205+0000 7f2e20836640 1 -- 172.21.15.62:0/3612971944 learned_addr learned my addr 172.21.15.62:0/3612971944 (peer_addr_for_me v1:172.21.15.62:0/0) 2023-05-22T15:26:26.217 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.205+0000 7f2e21037640 1 --2- 172.21.15.62:0/3612971944 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2e1c15cfa0 0x7f2e1c06db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:26.217 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.205+0000 7f2e21838640 1 --2- 172.21.15.62:0/3612971944 >> v2:172.21.15.62:3300/0 conn(0x7f2e1c06e740 0x7f2e1c06ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:26.217 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.205+0000 7f2e13fff640 1 -- 172.21.15.62:0/3612971944 <== mon.1 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 110822949 0 0) 0x7f2e0c001120 con 0x7f2e1c0cf870 2023-05-22T15:26:26.217 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.205+0000 7f2e21037640 1 -- 172.21.15.62:0/3612971944 >> v2:172.21.15.62:3300/0 conn(0x7f2e1c06e740 msgr2=0x7f2e1c06ebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:26:26.218 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.205+0000 7f2e21037640 1 --2- 172.21.15.62:0/3612971944 >> v2:172.21.15.62:3300/0 conn(0x7f2e1c06e740 0x7f2e1c06ebc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:26.218 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.205+0000 7f2e21037640 1 -- 172.21.15.62:0/3612971944 >> v1:172.21.15.62:6789/0 conn(0x7f2e1c0cf870 legacy=0x7f2e1c0b1fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:26.218 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.205+0000 7f2e21037640 1 -- 172.21.15.62:0/3612971944 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e1c06e490 con 0x7f2e1c15cfa0 2023-05-22T15:26:26.219 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.205+0000 7f2e21037640 1 --2- 172.21.15.62:0/3612971944 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2e1c15cfa0 0x7f2e1c06db00 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f2e180097c0 tx=0x7f2e1801c3c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=85ff7200f32fcc48 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:26:26.219 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.205+0000 7f2e13fff640 1 -- 172.21.15.62:0/3612971944 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e180ac620 con 0x7f2e1c15cfa0 2023-05-22T15:26:26.219 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.205+0000 7f2e13fff640 1 -- 172.21.15.62:0/3612971944 <== mon.2 v2:172.21.15.62:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e180ac780 con 0x7f2e1c15cfa0 2023-05-22T15:26:26.219 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.205+0000 7f2e13fff640 1 -- 172.21.15.62:0/3612971944 <== mon.2 v2:172.21.15.62:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e180acb10 con 0x7f2e1c15cfa0 2023-05-22T15:26:26.220 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.209+0000 7f2e2329b640 1 -- 172.21.15.62:0/3612971944 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2e1c15cfa0 msgr2=0x7f2e1c06db00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:26.220 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.209+0000 7f2e2329b640 1 --2- 172.21.15.62:0/3612971944 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2e1c15cfa0 0x7f2e1c06db00 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f2e180097c0 tx=0x7f2e1801c3c0 comp rx=0 tx=0).stop 2023-05-22T15:26:26.220 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.209+0000 7f2e2329b640 1 -- 172.21.15.62:0/3612971944 shutdown_connections 2023-05-22T15:26:26.220 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.209+0000 7f2e2329b640 1 --2- 172.21.15.62:0/3612971944 >> v2:172.21.15.62:3300/0 conn(0x7f2e1c06e740 0x7f2e1c06ebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:26.220 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.209+0000 7f2e2329b640 1 --2- 172.21.15.62:0/3612971944 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2e1c15cfa0 0x7f2e1c06db00 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:26.221 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.209+0000 7f2e2329b640 1 -- 172.21.15.62:0/3612971944 >> 172.21.15.62:0/3612971944 conn(0x7f2e1c0cc870 msgr2=0x7f2e1c0cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:26:26.221 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.209+0000 7f2e2329b640 1 -- 172.21.15.62:0/3612971944 shutdown_connections 2023-05-22T15:26:26.221 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.209+0000 7f2e2329b640 1 -- 172.21.15.62:0/3612971944 wait complete. 2023-05-22T15:26:26.223 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.209+0000 7f2e2329b640 1 Processor -- start 2023-05-22T15:26:26.223 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.209+0000 7f2e2329b640 1 -- start start 2023-05-22T15:26:26.223 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.209+0000 7f2e2329b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e1c06e740 0x7f2e1c0ce780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:26.223 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.209+0000 7f2e2329b640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2e1c0cf870 0x7f2e1c1624d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:26.223 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.209+0000 7f2e2329b640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2e1c15cfa0 0x7f2e1c162a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:26.223 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.209+0000 7f2e2329b640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e1c1633c0 con 0x7f2e1c06e740 2023-05-22T15:26:26.223 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.209+0000 7f2e2329b640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e1c0b3eb0 con 0x7f2e1c0cf870 2023-05-22T15:26:26.224 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.209+0000 7f2e2329b640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e1c161c60 con 0x7f2e1c15cfa0 2023-05-22T15:26:26.224 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.209+0000 7f2e21838640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2e1c15cfa0 0x7f2e1c162a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:26.224 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.209+0000 7f2e20836640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2e1c0cf870 0x7f2e1c1624d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:26.224 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.209+0000 7f2e21838640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2e1c15cfa0 0x7f2e1c162a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3301/0 says I am v2:172.21.15.62:33890/0 (socket says 172.21.15.62:33890) 2023-05-22T15:26:26.224 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.209+0000 7f2e21838640 1 -- 172.21.15.62:0/3352049278 learned_addr learned my addr 172.21.15.62:0/3352049278 (peer_addr_for_me v2:172.21.15.62:0/0) 2023-05-22T15:26:26.224 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.209+0000 7f2e21037640 1 --2- 172.21.15.62:0/3352049278 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e1c06e740 0x7f2e1c0ce780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:26.224 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.209+0000 7f2e20836640 1 -- 172.21.15.62:0/3352049278 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2e1c15cfa0 msgr2=0x7f2e1c162a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:26.225 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.209+0000 7f2e20836640 1 --2- 172.21.15.62:0/3352049278 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2e1c15cfa0 0x7f2e1c162a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:26.225 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.209+0000 7f2e20836640 1 -- 172.21.15.62:0/3352049278 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e1c06e740 msgr2=0x7f2e1c0ce780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:26.225 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.209+0000 7f2e20836640 1 --2- 172.21.15.62:0/3352049278 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e1c06e740 0x7f2e1c0ce780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:26.225 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.209+0000 7f2e20836640 1 -- 172.21.15.62:0/3352049278 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e18009420 con 0x7f2e1c0cf870 2023-05-22T15:26:26.225 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.209+0000 7f2e21838640 1 --2- 172.21.15.62:0/3352049278 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2e1c15cfa0 0x7f2e1c162a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-05-22T15:26:26.225 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.209+0000 7f2e21037640 1 --2- 172.21.15.62:0/3352049278 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e1c06e740 0x7f2e1c0ce780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:26:26.225 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.209+0000 7f2e20836640 1 --2- 172.21.15.62:0/3352049278 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2e1c0cf870 0x7f2e1c1624d0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f2e0c00c780 tx=0x7f2e0c00cc40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:26:26.225 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.209+0000 7f2e11ffb640 1 -- 172.21.15.62:0/3352049278 <== mon.1 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e0c0072d0 con 0x7f2e1c0cf870 2023-05-22T15:26:26.226 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.209+0000 7f2e11ffb640 1 -- 172.21.15.62:0/3352049278 <== mon.1 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e0c007450 con 0x7f2e1c0cf870 2023-05-22T15:26:26.226 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.209+0000 7f2e2329b640 1 -- 172.21.15.62:0/3352049278 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e1c0cb610 con 0x7f2e1c0cf870 2023-05-22T15:26:26.226 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.209+0000 7f2e11ffb640 1 -- 172.21.15.62:0/3352049278 <== mon.1 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e0c001d20 con 0x7f2e1c0cf870 2023-05-22T15:26:26.226 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.209+0000 7f2e2329b640 1 -- 172.21.15.62:0/3352049278 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e1c0cb330 con 0x7f2e1c0cf870 2023-05-22T15:26:26.226 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.209+0000 7f2e11ffb640 1 -- 172.21.15.62:0/3352049278 <== mon.1 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f2e0c004af0 con 0x7f2e1c0cf870 2023-05-22T15:26:26.226 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.213+0000 7f2e11ffb640 1 -- 172.21.15.62:0/3352049278 <== mon.1 v2:172.21.15.62:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (secure 0 0 0) 0x7f2e0c01d7f0 con 0x7f2e1c0cf870 2023-05-22T15:26:26.226 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.213+0000 7f2e2329b640 1 -- 172.21.15.62:0/3352049278 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e1c15d370 con 0x7f2e1c0cf870 2023-05-22T15:26:26.227 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.213+0000 7f2e11ffb640 1 -- 172.21.15.62:0/3352049278 <== mon.1 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72422 (secure 0 0 0) 0x7f2e0c018070 con 0x7f2e1c0cf870 2023-05-22T15:26:26.252 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:26:26.242+0000 7f723d3e1180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-05-22T15:26:26.292 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.281+0000 7f2e2329b640 1 -- 172.21.15.62:0/3352049278 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd new", "uuid": "4702fbc9-a927-4565-aca5-41dc69cc8264", "id": 1} v 0) v1 -- 0x7f2e1c161ac0 con 0x7f2e1c0cf870 2023-05-22T15:26:26.303 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.293+0000 7f2e11ffb640 1 -- 172.21.15.62:0/3352049278 <== mon.1 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "4702fbc9-a927-4565-aca5-41dc69cc8264", "id": 1}]=0 v4) v1 ==== 112+0+2 (secure 0 0 0) 0x7f2e1c161ac0 con 0x7f2e1c0cf870 2023-05-22T15:26:26.303 INFO:teuthology.orchestra.run.smithi062.stdout:1 2023-05-22T15:26:26.304 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.293+0000 7f2e2329b640 1 -- 172.21.15.62:0/3352049278 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2e1c0cf870 msgr2=0x7f2e1c1624d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:26.305 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.293+0000 7f2e2329b640 1 --2- 172.21.15.62:0/3352049278 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2e1c0cf870 0x7f2e1c1624d0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f2e0c00c780 tx=0x7f2e0c00cc40 comp rx=0 tx=0).stop 2023-05-22T15:26:26.305 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.293+0000 7f2e2329b640 1 -- 172.21.15.62:0/3352049278 shutdown_connections 2023-05-22T15:26:26.305 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.293+0000 7f2e2329b640 1 --2- 172.21.15.62:0/3352049278 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2e1c15cfa0 0x7f2e1c162a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:26.305 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.293+0000 7f2e2329b640 1 --2- 172.21.15.62:0/3352049278 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2e1c0cf870 0x7f2e1c1624d0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:26.305 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.293+0000 7f2e2329b640 1 --2- 172.21.15.62:0/3352049278 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e1c06e740 0x7f2e1c0ce780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:26.305 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.293+0000 7f2e2329b640 1 -- 172.21.15.62:0/3352049278 >> 172.21.15.62:0/3352049278 conn(0x7f2e1c0cc870 msgr2=0x7f2e1c15a340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:26:26.305 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.293+0000 7f2e2329b640 1 -- 172.21.15.62:0/3352049278 shutdown_connections 2023-05-22T15:26:26.306 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.293+0000 7f2e2329b640 1 -- 172.21.15.62:0/3352049278 wait complete. 2023-05-22T15:26:26.312 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:26:26.301+0000 7f90244ca180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-05-22T15:26:26.320 DEBUG:teuthology.orchestra.run.smithi062:> sudo ceph --cluster ceph osd new 1e7bba46-4eba-47ef-bb81-5e2d371638ad 2 2023-05-22T15:26:26.473 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.461+0000 7f2293976640 1 Processor -- start 2023-05-22T15:26:26.473 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.461+0000 7f2293976640 1 -- start start 2023-05-22T15:26:26.475 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.461+0000 7f2293976640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f228c0cf870 0x7f228c06db00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:26.475 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.461+0000 7f2293976640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f228c06e740 0x7f228c06ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:26.475 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.461+0000 7f2293976640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f228c06e040 con 0x7f228c15cfa0 2023-05-22T15:26:26.476 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.461+0000 7f2293976640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f228c0cb9f0 con 0x7f228c06e740 2023-05-22T15:26:26.476 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.461+0000 7f2293976640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f228c06f100 con 0x7f228c0cf870 2023-05-22T15:26:26.476 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.465+0000 7f2291712640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f228c15cfa0 0x7f228c0b1fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:42228/0 (socket says 172.21.15.62:42228) 2023-05-22T15:26:26.476 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.465+0000 7f2291712640 1 -- 172.21.15.62:0/514847353 learned_addr learned my addr 172.21.15.62:0/514847353 (peer_addr_for_me v1:172.21.15.62:0/0) 2023-05-22T15:26:26.476 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.465+0000 7f2290f11640 1 --2- 172.21.15.62:0/514847353 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f228c0cf870 0x7f228c06db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:26.477 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.465+0000 7f2283fff640 1 -- 172.21.15.62:0/514847353 <== mon.1 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 572857356 0 0) 0x7f2288001120 con 0x7f228c15cfa0 2023-05-22T15:26:26.477 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.465+0000 7f2291f13640 1 --2- 172.21.15.62:0/514847353 >> v2:172.21.15.16:3300/0 conn(0x7f228c06e740 0x7f228c06ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:26.477 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.465+0000 7f2283fff640 1 -- 172.21.15.62:0/514847353 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2278003530 con 0x7f228c15cfa0 2023-05-22T15:26:26.477 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.465+0000 7f2290f11640 1 -- 172.21.15.62:0/514847353 >> v2:172.21.15.16:3300/0 conn(0x7f228c06e740 msgr2=0x7f228c06ebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:26:26.477 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.465+0000 7f2290f11640 1 --2- 172.21.15.62:0/514847353 >> v2:172.21.15.16:3300/0 conn(0x7f228c06e740 0x7f228c06ebc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:26.477 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.465+0000 7f2290f11640 1 -- 172.21.15.62:0/514847353 >> v1:172.21.15.62:6789/0 conn(0x7f228c15cfa0 legacy=0x7f228c0b1fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:26.478 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.465+0000 7f2290f11640 1 -- 172.21.15.62:0/514847353 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f228c06e490 con 0x7f228c0cf870 2023-05-22T15:26:26.479 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.465+0000 7f2290f11640 1 --2- 172.21.15.62:0/514847353 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f228c0cf870 0x7f228c06db00 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f227c0097c0 tx=0x7f227c01c3c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5baff04188ce5c14 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:26:26.479 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.465+0000 7f2283fff640 1 -- 172.21.15.62:0/514847353 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f227c0ac620 con 0x7f228c0cf870 2023-05-22T15:26:26.479 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.465+0000 7f2283fff640 1 -- 172.21.15.62:0/514847353 <== mon.2 v2:172.21.15.62:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f227c0ac780 con 0x7f228c0cf870 2023-05-22T15:26:26.479 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.465+0000 7f2283fff640 1 -- 172.21.15.62:0/514847353 <== mon.2 v2:172.21.15.62:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f227c0acb10 con 0x7f228c0cf870 2023-05-22T15:26:26.479 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.465+0000 7f2293976640 1 -- 172.21.15.62:0/514847353 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f228c0cf870 msgr2=0x7f228c06db00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:26.479 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.465+0000 7f2293976640 1 --2- 172.21.15.62:0/514847353 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f228c0cf870 0x7f228c06db00 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f227c0097c0 tx=0x7f227c01c3c0 comp rx=0 tx=0).stop 2023-05-22T15:26:26.480 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.465+0000 7f2293976640 1 -- 172.21.15.62:0/514847353 shutdown_connections 2023-05-22T15:26:26.480 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.465+0000 7f2293976640 1 --2- 172.21.15.62:0/514847353 >> v2:172.21.15.16:3300/0 conn(0x7f228c06e740 0x7f228c06ebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:26.480 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.465+0000 7f2293976640 1 --2- 172.21.15.62:0/514847353 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f228c0cf870 0x7f228c06db00 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:26.480 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.465+0000 7f2293976640 1 -- 172.21.15.62:0/514847353 >> 172.21.15.62:0/514847353 conn(0x7f228c0cc870 msgr2=0x7f228c0cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:26:26.480 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.465+0000 7f2293976640 1 -- 172.21.15.62:0/514847353 shutdown_connections 2023-05-22T15:26:26.480 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.465+0000 7f2293976640 1 -- 172.21.15.62:0/514847353 wait complete. 2023-05-22T15:26:26.480 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.469+0000 7f2293976640 1 Processor -- start 2023-05-22T15:26:26.481 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.469+0000 7f2293976640 1 -- start start 2023-05-22T15:26:26.482 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.469+0000 7f2293976640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f228c06e740 0x7f228c0c6c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:26.483 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.469+0000 7f2293976640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f228c0cf870 0x7f228c159260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:26.483 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.469+0000 7f2293976640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f228c15cfa0 0x7f228c1597a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:26.483 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.469+0000 7f2293976640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f228c0c8310 con 0x7f228c15cfa0 2023-05-22T15:26:26.483 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.469+0000 7f2293976640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f228c15d850 con 0x7f228c0cf870 2023-05-22T15:26:26.483 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.469+0000 7f2293976640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f228c0c7690 con 0x7f228c06e740 2023-05-22T15:26:26.483 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.469+0000 7f2290f11640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f228c0cf870 0x7f228c159260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:26.484 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.469+0000 7f2290f11640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f228c0cf870 0x7f228c159260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:38044/0 (socket says 172.21.15.62:38044) 2023-05-22T15:26:26.484 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.469+0000 7f2290f11640 1 -- 172.21.15.62:0/2614362128 learned_addr learned my addr 172.21.15.62:0/2614362128 (peer_addr_for_me v2:172.21.15.62:0/0) 2023-05-22T15:26:26.484 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.469+0000 7f2291712640 1 --2- 172.21.15.62:0/2614362128 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f228c06e740 0x7f228c0c6c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:26.484 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.469+0000 7f2291f13640 1 --2- 172.21.15.62:0/2614362128 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f228c15cfa0 0x7f228c1597a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:26.484 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.469+0000 7f2290f11640 1 -- 172.21.15.62:0/2614362128 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f228c06e740 msgr2=0x7f228c0c6c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:26.484 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.469+0000 7f2290f11640 1 --2- 172.21.15.62:0/2614362128 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f228c06e740 0x7f228c0c6c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:26.484 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.469+0000 7f2290f11640 1 -- 172.21.15.62:0/2614362128 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f228c15cfa0 msgr2=0x7f228c1597a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:26.485 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.469+0000 7f2290f11640 1 --2- 172.21.15.62:0/2614362128 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f228c15cfa0 0x7f228c1597a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:26.485 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.469+0000 7f2290f11640 1 -- 172.21.15.62:0/2614362128 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f227c009420 con 0x7f228c0cf870 2023-05-22T15:26:26.485 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.469+0000 7f2290f11640 1 --2- 172.21.15.62:0/2614362128 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f228c0cf870 0x7f228c159260 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f227c019ff0 tx=0x7f227c019160 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:26:26.485 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.469+0000 7f2281ffb640 1 -- 172.21.15.62:0/2614362128 <== mon.1 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f227c0b5410 con 0x7f228c0cf870 2023-05-22T15:26:26.485 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.469+0000 7f2281ffb640 1 -- 172.21.15.62:0/2614362128 <== mon.1 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f227c0b5590 con 0x7f228c0cf870 2023-05-22T15:26:26.485 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.469+0000 7f2293976640 1 -- 172.21.15.62:0/2614362128 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f228c0cb610 con 0x7f228c0cf870 2023-05-22T15:26:26.485 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.469+0000 7f2281ffb640 1 -- 172.21.15.62:0/2614362128 <== mon.1 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f227c0b5940 con 0x7f228c0cf870 2023-05-22T15:26:26.485 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.469+0000 7f2293976640 1 -- 172.21.15.62:0/2614362128 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f228c0cb330 con 0x7f228c0cf870 2023-05-22T15:26:26.486 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.469+0000 7f2281ffb640 1 -- 172.21.15.62:0/2614362128 <== mon.1 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f227c0b3040 con 0x7f228c0cf870 2023-05-22T15:26:26.486 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.469+0000 7f2281ffb640 1 -- 172.21.15.62:0/2614362128 <== mon.1 v2:172.21.15.62:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (secure 0 0 0) 0x7f227c0bdd70 con 0x7f228c0cf870 2023-05-22T15:26:26.486 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.469+0000 7f2293976640 1 -- 172.21.15.62:0/2614362128 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f228c15d370 con 0x7f228c0cf870 2023-05-22T15:26:26.486 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.473+0000 7f2281ffb640 1 -- 172.21.15.62:0/2614362128 <== mon.1 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72422 (secure 0 0 0) 0x7f227c0b1070 con 0x7f228c0cf870 2023-05-22T15:26:26.550 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.537+0000 7f2293976640 1 -- 172.21.15.62:0/2614362128 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd new", "uuid": "1e7bba46-4eba-47ef-bb81-5e2d371638ad", "id": 2} v 0) v1 -- 0x7f228c04dbe0 con 0x7f228c0cf870 2023-05-22T15:26:26.557 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.545+0000 7f2281ffb640 1 -- 172.21.15.62:0/2614362128 <== mon.1 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "1e7bba46-4eba-47ef-bb81-5e2d371638ad", "id": 2}]=0 v5) v1 ==== 112+0+2 (secure 0 0 0) 0x7f227c0bd400 con 0x7f228c0cf870 2023-05-22T15:26:26.557 INFO:teuthology.orchestra.run.smithi062.stdout:2 2023-05-22T15:26:26.558 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.549+0000 7f2293976640 1 -- 172.21.15.62:0/2614362128 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f228c0cf870 msgr2=0x7f228c159260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:26.558 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.549+0000 7f2293976640 1 --2- 172.21.15.62:0/2614362128 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f228c0cf870 0x7f228c159260 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f227c019ff0 tx=0x7f227c019160 comp rx=0 tx=0).stop 2023-05-22T15:26:26.559 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.549+0000 7f2293976640 1 -- 172.21.15.62:0/2614362128 shutdown_connections 2023-05-22T15:26:26.559 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.549+0000 7f2293976640 1 --2- 172.21.15.62:0/2614362128 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f228c15cfa0 0x7f228c1597a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:26.559 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.549+0000 7f2293976640 1 --2- 172.21.15.62:0/2614362128 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f228c0cf870 0x7f228c159260 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:26.559 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.549+0000 7f2293976640 1 --2- 172.21.15.62:0/2614362128 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f228c06e740 0x7f228c0c6c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:26.559 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.549+0000 7f2293976640 1 -- 172.21.15.62:0/2614362128 >> 172.21.15.62:0/2614362128 conn(0x7f228c0cc870 msgr2=0x7f228c1578c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:26:26.559 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.549+0000 7f2293976640 1 -- 172.21.15.62:0/2614362128 shutdown_connections 2023-05-22T15:26:26.559 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.549+0000 7f2293976640 1 -- 172.21.15.62:0/2614362128 wait complete. 2023-05-22T15:26:26.579 DEBUG:teuthology.orchestra.run.smithi062:> sudo ceph --cluster ceph osd new 383cacc0-5a88-4904-990e-498ca59ef48a 3 2023-05-22T15:26:26.701 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.689+0000 7fb2f5707640 1 Processor -- start 2023-05-22T15:26:26.701 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.689+0000 7fb2f5707640 1 -- start start 2023-05-22T15:26:26.703 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.689+0000 7fb2f5707640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb2f01563c0 0x7fb2f00c7b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:26.703 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.689+0000 7fb2f5707640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2f015c210 con 0x7fb2f00cc6c0 2023-05-22T15:26:26.703 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.689+0000 7fb2f5707640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2f015c480 con 0x7fb2f00c8080 2023-05-22T15:26:26.704 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.689+0000 7fb2f5707640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2f00c8520 con 0x7fb2f01563c0 2023-05-22T15:26:26.705 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.693+0000 7fb2eeffd640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb2f01563c0 0x7fb2f00c7b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:26.705 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.693+0000 7fb2ef7fe640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7fb2f00c8080 0x7fb2f015f6f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:42240/0 (socket says 172.21.15.62:42240) 2023-05-22T15:26:26.705 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.693+0000 7fb2ef7fe640 1 -- 172.21.15.62:0/877228747 learned_addr learned my addr 172.21.15.62:0/877228747 (peer_addr_for_me v1:172.21.15.62:0/0) 2023-05-22T15:26:26.705 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.693+0000 7fb2eeffd640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb2f01563c0 0x7fb2f00c7b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3301/0 says I am v2:172.21.15.62:33906/0 (socket says 172.21.15.62:33906) 2023-05-22T15:26:26.705 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.693+0000 7fb2eeffd640 1 -- 172.21.15.62:0/877228747 >> v1:172.21.15.62:6789/0 conn(0x7fb2f00c8080 legacy=0x7fb2f015f6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:26.705 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.693+0000 7fb2eeffd640 1 -- 172.21.15.62:0/877228747 >> v1:172.21.15.16:6789/0 conn(0x7fb2f00cc6c0 legacy=0x7fb2f006c220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:26:26.706 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.693+0000 7fb2edffb640 1 -- 172.21.15.62:0/877228747 <== mon.1 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1684517400 0 0) 0x7fb2f015c480 con 0x7fb2f00c8080 2023-05-22T15:26:26.706 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.693+0000 7fb2eeffd640 1 -- 172.21.15.62:0/877228747 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2f015c760 con 0x7fb2f01563c0 2023-05-22T15:26:26.707 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.693+0000 7fb2eeffd640 1 --2- 172.21.15.62:0/877228747 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb2f01563c0 0x7fb2f00c7b40 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fb2e00097f0 tx=0x7fb2e001c450 comp rx=0 tx=0).ready entity=mon.2 client_cookie=83b8d1bd26c2aa4a server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:26:26.707 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.693+0000 7fb2edffb640 1 -- 172.21.15.62:0/877228747 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2e00ab650 con 0x7fb2f01563c0 2023-05-22T15:26:26.707 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.693+0000 7fb2edffb640 1 -- 172.21.15.62:0/877228747 <== mon.2 v2:172.21.15.62:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2e00ab7b0 con 0x7fb2f01563c0 2023-05-22T15:26:26.707 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.693+0000 7fb2edffb640 1 -- 172.21.15.62:0/877228747 <== mon.2 v2:172.21.15.62:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2e00abb40 con 0x7fb2f01563c0 2023-05-22T15:26:26.707 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.693+0000 7fb2f5707640 1 -- 172.21.15.62:0/877228747 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb2f01563c0 msgr2=0x7fb2f00c7b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:26.708 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.693+0000 7fb2f5707640 1 --2- 172.21.15.62:0/877228747 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb2f01563c0 0x7fb2f00c7b40 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fb2e00097f0 tx=0x7fb2e001c450 comp rx=0 tx=0).stop 2023-05-22T15:26:26.708 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.693+0000 7fb2f5707640 1 -- 172.21.15.62:0/877228747 shutdown_connections 2023-05-22T15:26:26.708 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.693+0000 7fb2f5707640 1 --2- 172.21.15.62:0/877228747 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb2f01563c0 0x7fb2f00c7b40 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:26.708 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.693+0000 7fb2f5707640 1 -- 172.21.15.62:0/877228747 >> 172.21.15.62:0/877228747 conn(0x7fb2f00b1110 msgr2=0x7fb2f00ce830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:26:26.708 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.693+0000 7fb2f5707640 1 -- 172.21.15.62:0/877228747 shutdown_connections 2023-05-22T15:26:26.708 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.693+0000 7fb2f5707640 1 -- 172.21.15.62:0/877228747 wait complete. 2023-05-22T15:26:26.708 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.697+0000 7fb2f5707640 1 Processor -- start 2023-05-22T15:26:26.709 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.697+0000 7fb2f5707640 1 -- start start 2023-05-22T15:26:26.710 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.697+0000 7fb2f5707640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb2f00c8080 0x7fb2f00b2ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:26.710 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.697+0000 7fb2f5707640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb2f00cc6c0 0x7fb2f014ac90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:26.711 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.697+0000 7fb2f5707640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb2f00b40a0 0x7fb2f00af5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:26.711 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.697+0000 7fb2f5707640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2f014b1d0 con 0x7fb2f00cc6c0 2023-05-22T15:26:26.711 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.697+0000 7fb2f5707640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2f014dbd0 con 0x7fb2f00b40a0 2023-05-22T15:26:26.711 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.697+0000 7fb2f5707640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2f00b3ed0 con 0x7fb2f00c8080 2023-05-22T15:26:26.711 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.697+0000 7fb2eeffd640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb2f00c8080 0x7fb2f00b2ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:26.711 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.697+0000 7fb2ef7fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb2f00b40a0 0x7fb2f00af5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:26.711 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.697+0000 7fb2eeffd640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb2f00c8080 0x7fb2f00b2ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3301/0 says I am v2:172.21.15.62:33912/0 (socket says 172.21.15.62:33912) 2023-05-22T15:26:26.712 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.697+0000 7fb2eeffd640 1 -- 172.21.15.62:0/1560024547 learned_addr learned my addr 172.21.15.62:0/1560024547 (peer_addr_for_me v2:172.21.15.62:0/0) 2023-05-22T15:26:26.712 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.697+0000 7fb2ee7fc640 1 --2- 172.21.15.62:0/1560024547 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb2f00cc6c0 0x7fb2f014ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:26.713 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.697+0000 7fb2eeffd640 1 -- 172.21.15.62:0/1560024547 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb2f00b40a0 msgr2=0x7fb2f00af5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:26.713 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.697+0000 7fb2eeffd640 1 --2- 172.21.15.62:0/1560024547 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb2f00b40a0 0x7fb2f00af5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:26.713 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.697+0000 7fb2eeffd640 1 -- 172.21.15.62:0/1560024547 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb2f00cc6c0 msgr2=0x7fb2f014ac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:26.713 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.697+0000 7fb2eeffd640 1 --2- 172.21.15.62:0/1560024547 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb2f00cc6c0 0x7fb2f014ac90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:26.713 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.697+0000 7fb2eeffd640 1 -- 172.21.15.62:0/1560024547 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2e0009420 con 0x7fb2f00c8080 2023-05-22T15:26:26.713 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.697+0000 7fb2ef7fe640 1 --2- 172.21.15.62:0/1560024547 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb2f00b40a0 0x7fb2f00af5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-05-22T15:26:26.713 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.697+0000 7fb2eeffd640 1 --2- 172.21.15.62:0/1560024547 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb2f00c8080 0x7fb2f00b2ab0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fb2e00ad040 tx=0x7fb2e001ac90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:26:26.714 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.697+0000 7fb2d77fe640 1 -- 172.21.15.62:0/1560024547 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2e00b5810 con 0x7fb2f00c8080 2023-05-22T15:26:26.714 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.697+0000 7fb2d77fe640 1 -- 172.21.15.62:0/1560024547 <== mon.2 v2:172.21.15.62:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2e00b5990 con 0x7fb2f00c8080 2023-05-22T15:26:26.714 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.697+0000 7fb2f5707640 1 -- 172.21.15.62:0/1560024547 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2f0169aa0 con 0x7fb2f00c8080 2023-05-22T15:26:26.714 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.697+0000 7fb2d77fe640 1 -- 172.21.15.62:0/1560024547 <== mon.2 v2:172.21.15.62:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2e00b5d40 con 0x7fb2f00c8080 2023-05-22T15:26:26.714 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.697+0000 7fb2f5707640 1 -- 172.21.15.62:0/1560024547 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2f006cd40 con 0x7fb2f00c8080 2023-05-22T15:26:26.714 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.697+0000 7fb2d77fe640 1 -- 172.21.15.62:0/1560024547 <== mon.2 v2:172.21.15.62:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fb2e00bf710 con 0x7fb2f00c8080 2023-05-22T15:26:26.714 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.697+0000 7fb2d77fe640 1 -- 172.21.15.62:0/1560024547 <== mon.2 v2:172.21.15.62:3301/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (secure 0 0 0) 0x7fb2e00b4390 con 0x7fb2f00c8080 2023-05-22T15:26:26.715 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.697+0000 7fb2f5707640 1 -- 172.21.15.62:0/1560024547 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2f0169c30 con 0x7fb2f00c8080 2023-05-22T15:26:26.715 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.701+0000 7fb2d77fe640 1 -- 172.21.15.62:0/1560024547 <== mon.2 v2:172.21.15.62:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72422 (secure 0 0 0) 0x7fb2e00b4550 con 0x7fb2f00c8080 2023-05-22T15:26:26.779 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.769+0000 7fb2f5707640 1 -- 172.21.15.62:0/1560024547 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_command({"prefix": "osd new", "uuid": "383cacc0-5a88-4904-990e-498ca59ef48a", "id": 3} v 0) v1 -- 0x7fb2f00b17c0 con 0x7fb2f00c8080 2023-05-22T15:26:26.787 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.777+0000 7fb2d77fe640 1 -- 172.21.15.62:0/1560024547 <== mon.2 v2:172.21.15.62:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "383cacc0-5a88-4904-990e-498ca59ef48a", "id": 3}]=0 v6) v1 ==== 112+0+2 (secure 0 0 0) 0x7fb2e00bf9c0 con 0x7fb2f00c8080 2023-05-22T15:26:26.787 INFO:teuthology.orchestra.run.smithi062.stdout:3 2023-05-22T15:26:26.788 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.777+0000 7fb2f5707640 1 -- 172.21.15.62:0/1560024547 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb2f00c8080 msgr2=0x7fb2f00b2ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:26.788 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.777+0000 7fb2f5707640 1 --2- 172.21.15.62:0/1560024547 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb2f00c8080 0x7fb2f00b2ab0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fb2e00ad040 tx=0x7fb2e001ac90 comp rx=0 tx=0).stop 2023-05-22T15:26:26.788 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.777+0000 7fb2f5707640 1 -- 172.21.15.62:0/1560024547 shutdown_connections 2023-05-22T15:26:26.788 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.777+0000 7fb2f5707640 1 --2- 172.21.15.62:0/1560024547 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb2f00b40a0 0x7fb2f00af5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:26.788 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.777+0000 7fb2f5707640 1 --2- 172.21.15.62:0/1560024547 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb2f00cc6c0 0x7fb2f014ac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:26.789 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.777+0000 7fb2f5707640 1 --2- 172.21.15.62:0/1560024547 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb2f00c8080 0x7fb2f00b2ab0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:26.789 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.777+0000 7fb2f5707640 1 -- 172.21.15.62:0/1560024547 >> 172.21.15.62:0/1560024547 conn(0x7fb2f00b1110 msgr2=0x7fb2f006fdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:26:26.789 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.777+0000 7fb2f5707640 1 -- 172.21.15.62:0/1560024547 shutdown_connections 2023-05-22T15:26:26.789 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.777+0000 7fb2f5707640 1 -- 172.21.15.62:0/1560024547 wait complete. 2023-05-22T15:26:26.803 DEBUG:teuthology.orchestra.run.smithi062:> sudo ceph --cluster ceph osd new 970500ef-95b7-42c1-b080-eed4fae7013d 4 2023-05-22T15:26:26.818 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:26:26.806+0000 7f723d3e1180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-05-22T15:26:26.875 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:26:26.862+0000 7f723d3e1180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-05-22T15:26:26.918 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.905+0000 7f2f72db4640 1 Processor -- start 2023-05-22T15:26:26.918 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.905+0000 7f2f72db4640 1 -- start start 2023-05-22T15:26:26.919 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.909+0000 7f2f72db4640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f2f6c15cfa0 0x7f2f6c06db00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:26.919 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.909+0000 7f2f72db4640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2f6c06e740 0x7f2f6c06ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:26.920 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.909+0000 7f2f72db4640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f6c06e040 con 0x7f2f6c0cf870 2023-05-22T15:26:26.920 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.909+0000 7f2f72db4640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f6c0cb9f0 con 0x7f2f6c15cfa0 2023-05-22T15:26:26.920 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.909+0000 7f2f72db4640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f6c06f100 con 0x7f2f6c06e740 2023-05-22T15:26:26.920 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.909+0000 7f2f71351640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2f6c06e740 0x7f2f6c06ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:26.920 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.909+0000 7f2f71351640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2f6c06e740 0x7f2f6c06ebc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3301/0 says I am v2:172.21.15.62:33928/0 (socket says 172.21.15.62:33928) 2023-05-22T15:26:26.920 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.909+0000 7f2f71351640 1 -- 172.21.15.62:0/1112929859 learned_addr learned my addr 172.21.15.62:0/1112929859 (peer_addr_for_me v2:172.21.15.62:0/0) 2023-05-22T15:26:26.921 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.909+0000 7f2f70b50640 1 --2- 172.21.15.62:0/1112929859 >> v2:172.21.15.62:3300/0 conn(0x7f2f6c15cfa0 0x7f2f6c06db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:26.921 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.909+0000 7f2f71351640 1 -- 172.21.15.62:0/1112929859 >> v2:172.21.15.62:3300/0 conn(0x7f2f6c15cfa0 msgr2=0x7f2f6c06db00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:26:26.921 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.909+0000 7f2f71351640 1 --2- 172.21.15.62:0/1112929859 >> v2:172.21.15.62:3300/0 conn(0x7f2f6c15cfa0 0x7f2f6c06db00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:26.921 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.909+0000 7f2f71351640 1 -- 172.21.15.62:0/1112929859 >> v1:172.21.15.16:6789/0 conn(0x7f2f6c0cf870 legacy=0x7f2f6c0b1fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:26.921 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.909+0000 7f2f71351640 1 -- 172.21.15.62:0/1112929859 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f6c06e490 con 0x7f2f6c06e740 2023-05-22T15:26:26.922 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.909+0000 7f2f71351640 1 --2- 172.21.15.62:0/1112929859 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2f6c06e740 0x7f2f6c06ebc0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f2f64009e50 tx=0x7f2f6401c9e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b5be9ad49256468a server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:26:26.922 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.909+0000 7f2f6b7fe640 1 -- 172.21.15.62:0/1112929859 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f640a58a0 con 0x7f2f6c06e740 2023-05-22T15:26:26.923 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:26:26.909+0000 7f90244ca180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-05-22T15:26:26.923 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.909+0000 7f2f6b7fe640 1 -- 172.21.15.62:0/1112929859 <== mon.2 v2:172.21.15.62:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f640a5a00 con 0x7f2f6c06e740 2023-05-22T15:26:26.923 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.909+0000 7f2f6b7fe640 1 -- 172.21.15.62:0/1112929859 <== mon.2 v2:172.21.15.62:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f640a5d90 con 0x7f2f6c06e740 2023-05-22T15:26:26.924 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.913+0000 7f2f72db4640 1 -- 172.21.15.62:0/1112929859 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2f6c06e740 msgr2=0x7f2f6c06ebc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:26.924 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.913+0000 7f2f72db4640 1 --2- 172.21.15.62:0/1112929859 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2f6c06e740 0x7f2f6c06ebc0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f2f64009e50 tx=0x7f2f6401c9e0 comp rx=0 tx=0).stop 2023-05-22T15:26:26.924 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.913+0000 7f2f72db4640 1 -- 172.21.15.62:0/1112929859 shutdown_connections 2023-05-22T15:26:26.924 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.913+0000 7f2f72db4640 1 --2- 172.21.15.62:0/1112929859 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2f6c06e740 0x7f2f6c06ebc0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:26.924 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.913+0000 7f2f72db4640 1 --2- 172.21.15.62:0/1112929859 >> v2:172.21.15.62:3300/0 conn(0x7f2f6c15cfa0 0x7f2f6c06db00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:26.924 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.913+0000 7f2f72db4640 1 -- 172.21.15.62:0/1112929859 >> 172.21.15.62:0/1112929859 conn(0x7f2f6c0cc870 msgr2=0x7f2f6c0cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:26:26.924 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.913+0000 7f2f72db4640 1 -- 172.21.15.62:0/1112929859 shutdown_connections 2023-05-22T15:26:26.925 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.913+0000 7f2f72db4640 1 -- 172.21.15.62:0/1112929859 wait complete. 2023-05-22T15:26:26.927 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.913+0000 7f2f72db4640 1 Processor -- start 2023-05-22T15:26:26.927 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.913+0000 7f2f72db4640 1 -- start start 2023-05-22T15:26:26.927 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.913+0000 7f2f72db4640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2f6c06e740 0x7f2f6c1578b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:26.928 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.913+0000 7f2f72db4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f6c0cf870 0x7f2f6c1437a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:26.928 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.913+0000 7f2f72db4640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2f6c15cfa0 0x7f2f6c146240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:26.928 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.913+0000 7f2f72db4640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f6c146780 con 0x7f2f6c0cf870 2023-05-22T15:26:26.928 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.913+0000 7f2f72db4640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f6c0c9030 con 0x7f2f6c06e740 2023-05-22T15:26:26.928 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.913+0000 7f2f72db4640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f6c0c8a70 con 0x7f2f6c15cfa0 2023-05-22T15:26:26.928 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.913+0000 7f2f70b50640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2f6c06e740 0x7f2f6c1578b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:26.929 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.913+0000 7f2f71351640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2f6c15cfa0 0x7f2f6c146240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:26.929 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.913+0000 7f2f71351640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2f6c15cfa0 0x7f2f6c146240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3301/0 says I am v2:172.21.15.62:33944/0 (socket says 172.21.15.62:33944) 2023-05-22T15:26:26.929 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.913+0000 7f2f70b50640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2f6c06e740 0x7f2f6c1578b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:38062/0 (socket says 172.21.15.62:38062) 2023-05-22T15:26:26.929 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.913+0000 7f2f71351640 1 -- 172.21.15.62:0/2387343381 learned_addr learned my addr 172.21.15.62:0/2387343381 (peer_addr_for_me v2:172.21.15.62:0/0) 2023-05-22T15:26:26.929 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.913+0000 7f2f6bfff640 1 --2- 172.21.15.62:0/2387343381 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f6c0cf870 0x7f2f6c1437a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:26.929 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.913+0000 7f2f70b50640 1 -- 172.21.15.62:0/2387343381 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2f6c15cfa0 msgr2=0x7f2f6c146240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:26.929 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.913+0000 7f2f70b50640 1 --2- 172.21.15.62:0/2387343381 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2f6c15cfa0 0x7f2f6c146240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:26.929 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.913+0000 7f2f70b50640 1 -- 172.21.15.62:0/2387343381 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f6c0cf870 msgr2=0x7f2f6c1437a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:26.930 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.913+0000 7f2f70b50640 1 --2- 172.21.15.62:0/2387343381 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f6c0cf870 0x7f2f6c1437a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:26.930 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.913+0000 7f2f70b50640 1 -- 172.21.15.62:0/2387343381 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f64009b30 con 0x7f2f6c06e740 2023-05-22T15:26:26.930 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.913+0000 7f2f71351640 1 --2- 172.21.15.62:0/2387343381 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2f6c15cfa0 0x7f2f6c146240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-05-22T15:26:26.930 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.913+0000 7f2f70b50640 1 --2- 172.21.15.62:0/2387343381 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2f6c06e740 0x7f2f6c1578b0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f2f6000e870 tx=0x7f2f6000ed30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:26:26.930 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.913+0000 7f2f697fa640 1 -- 172.21.15.62:0/2387343381 <== mon.1 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f6000ee60 con 0x7f2f6c06e740 2023-05-22T15:26:26.930 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.913+0000 7f2f697fa640 1 -- 172.21.15.62:0/2387343381 <== mon.1 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f6000c940 con 0x7f2f6c06e740 2023-05-22T15:26:26.930 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.913+0000 7f2f72db4640 1 -- 172.21.15.62:0/2387343381 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f6c0cb610 con 0x7f2f6c06e740 2023-05-22T15:26:26.931 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.913+0000 7f2f697fa640 1 -- 172.21.15.62:0/2387343381 <== mon.1 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f6000ccd0 con 0x7f2f6c06e740 2023-05-22T15:26:26.931 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.913+0000 7f2f72db4640 1 -- 172.21.15.62:0/2387343381 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f6c0cb330 con 0x7f2f6c06e740 2023-05-22T15:26:26.931 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.917+0000 7f2f697fa640 1 -- 172.21.15.62:0/2387343381 <== mon.1 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f2f60013770 con 0x7f2f6c06e740 2023-05-22T15:26:26.931 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.917+0000 7f2f697fa640 1 -- 172.21.15.62:0/2387343381 <== mon.1 v2:172.21.15.62:3300/0 5 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (secure 0 0 0) 0x7f2f6001cbe0 con 0x7f2f6c06e740 2023-05-22T15:26:26.931 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.917+0000 7f2f72db4640 1 -- 172.21.15.62:0/2387343381 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f6c15d370 con 0x7f2f6c06e740 2023-05-22T15:26:26.932 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:26:26.918+0000 7f723d3e1180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-05-22T15:26:26.932 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.921+0000 7f2f697fa640 1 -- 172.21.15.62:0/2387343381 <== mon.1 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72422 (secure 0 0 0) 0x7f2f60021060 con 0x7f2f6c06e740 2023-05-22T15:26:26.981 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:26:26.969+0000 7f90244ca180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-05-22T15:26:26.996 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.985+0000 7f2f72db4640 1 -- 172.21.15.62:0/2387343381 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd new", "uuid": "970500ef-95b7-42c1-b080-eed4fae7013d", "id": 4} v 0) v1 -- 0x7f2f6c04dbe0 con 0x7f2f6c06e740 2023-05-22T15:26:27.004 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.993+0000 7f2f697fa640 1 -- 172.21.15.62:0/2387343381 <== mon.1 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "970500ef-95b7-42c1-b080-eed4fae7013d", "id": 4}]=0 v7) v1 ==== 112+0+2 (secure 0 0 0) 0x7f2f60017070 con 0x7f2f6c06e740 2023-05-22T15:26:27.004 INFO:teuthology.orchestra.run.smithi062.stdout:4 2023-05-22T15:26:27.005 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.993+0000 7f2f72db4640 1 -- 172.21.15.62:0/2387343381 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2f6c06e740 msgr2=0x7f2f6c1578b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:27.006 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.993+0000 7f2f72db4640 1 --2- 172.21.15.62:0/2387343381 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2f6c06e740 0x7f2f6c1578b0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f2f6000e870 tx=0x7f2f6000ed30 comp rx=0 tx=0).stop 2023-05-22T15:26:27.006 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.993+0000 7f2f72db4640 1 -- 172.21.15.62:0/2387343381 shutdown_connections 2023-05-22T15:26:27.006 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.993+0000 7f2f72db4640 1 --2- 172.21.15.62:0/2387343381 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2f6c15cfa0 0x7f2f6c146240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:27.006 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.993+0000 7f2f72db4640 1 --2- 172.21.15.62:0/2387343381 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f6c0cf870 0x7f2f6c1437a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:27.006 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.993+0000 7f2f72db4640 1 --2- 172.21.15.62:0/2387343381 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2f6c06e740 0x7f2f6c1578b0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:27.006 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.993+0000 7f2f72db4640 1 -- 172.21.15.62:0/2387343381 >> 172.21.15.62:0/2387343381 conn(0x7f2f6c0cc870 msgr2=0x7f2f6c0b4590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:26:27.007 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.993+0000 7f2f72db4640 1 -- 172.21.15.62:0/2387343381 shutdown_connections 2023-05-22T15:26:27.007 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:26.993+0000 7f2f72db4640 1 -- 172.21.15.62:0/2387343381 wait complete. 2023-05-22T15:26:27.010 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:26:26.998+0000 7f723d3e1180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-05-22T15:26:27.020 DEBUG:teuthology.orchestra.run.smithi062:> sudo ceph --cluster ceph osd new 022d67db-d4d8-445b-b1b9-8b446a00f5c7 5 2023-05-22T15:26:27.040 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:26:27.029+0000 7f90244ca180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-05-22T15:26:27.064 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:26:27.054+0000 7f723d3e1180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-05-22T15:26:27.123 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:26:27.113+0000 7f90244ca180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-05-22T15:26:27.141 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.129+0000 7f072b4a9640 1 Processor -- start 2023-05-22T15:26:27.141 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.129+0000 7f072b4a9640 1 -- start start 2023-05-22T15:26:27.143 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.133+0000 7f072b4a9640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f07241563c0 0x7f07240c7b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:27.143 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.133+0000 7f072b4a9640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f07240cc6c0 0x7f07240c8080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:27.143 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.133+0000 7f072b4a9640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f072406c220 0x7f072406c6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:27.143 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.133+0000 7f072b4a9640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f072415c210 con 0x7f07241563c0 2023-05-22T15:26:27.143 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.133+0000 7f072b4a9640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f072415c480 con 0x7f072406c220 2023-05-22T15:26:27.144 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.133+0000 7f072b4a9640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f072415c760 con 0x7f07240cc6c0 2023-05-22T15:26:27.144 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.133+0000 7f0728a44640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f07240cc6c0 0x7f07240c8080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:27.144 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.133+0000 7f0729a46640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f072406c220 0x7f072406c6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:27.144 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.133+0000 7f0728a44640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f07240cc6c0 0x7f07240c8080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3301/0 says I am v2:172.21.15.62:33954/0 (socket says 172.21.15.62:33954) 2023-05-22T15:26:27.144 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.133+0000 7f0728a44640 1 -- 172.21.15.62:0/968580582 learned_addr learned my addr 172.21.15.62:0/968580582 (peer_addr_for_me v2:172.21.15.62:0/0) 2023-05-22T15:26:27.144 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.133+0000 7f0729245640 1 --2- 172.21.15.62:0/968580582 >> v2:172.21.15.16:3300/0 conn(0x7f07241563c0 0x7f07240c7b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:27.144 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.133+0000 7f0728a44640 1 -- 172.21.15.62:0/968580582 >> v2:172.21.15.62:3300/0 conn(0x7f072406c220 msgr2=0x7f072406c6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:26:27.145 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.133+0000 7f0728a44640 1 --2- 172.21.15.62:0/968580582 >> v2:172.21.15.62:3300/0 conn(0x7f072406c220 0x7f072406c6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:27.145 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.133+0000 7f0728a44640 1 -- 172.21.15.62:0/968580582 >> v2:172.21.15.16:3300/0 conn(0x7f07241563c0 msgr2=0x7f07240c7b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:26:27.145 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.133+0000 7f0728a44640 1 --2- 172.21.15.62:0/968580582 >> v2:172.21.15.16:3300/0 conn(0x7f07241563c0 0x7f07240c7b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:27.145 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.133+0000 7f0728a44640 1 -- 172.21.15.62:0/968580582 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f072415ca40 con 0x7f07240cc6c0 2023-05-22T15:26:27.145 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.133+0000 7f0729245640 1 --2- 172.21.15.62:0/968580582 >> v2:172.21.15.16:3300/0 conn(0x7f07241563c0 0x7f07240c7b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:26:27.146 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:26:27.134+0000 7f723d3e1180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-05-22T15:26:27.147 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.133+0000 7f0728a44640 1 --2- 172.21.15.62:0/968580582 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f07240cc6c0 0x7f07240c8080 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f0724159cd0 tx=0x7f071401c4d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=388d2cc216b6f362 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:26:27.148 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.133+0000 7f071bfff640 1 -- 172.21.15.62:0/968580582 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07140aa610 con 0x7f07240cc6c0 2023-05-22T15:26:27.148 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.133+0000 7f071bfff640 1 -- 172.21.15.62:0/968580582 <== mon.2 v2:172.21.15.62:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07140a6040 con 0x7f07240cc6c0 2023-05-22T15:26:27.148 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.133+0000 7f071bfff640 1 -- 172.21.15.62:0/968580582 <== mon.2 v2:172.21.15.62:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07140aa990 con 0x7f07240cc6c0 2023-05-22T15:26:27.148 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.133+0000 7f072b4a9640 1 -- 172.21.15.62:0/968580582 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f07240cc6c0 msgr2=0x7f07240c8080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:27.148 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.133+0000 7f072b4a9640 1 --2- 172.21.15.62:0/968580582 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f07240cc6c0 0x7f07240c8080 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f0724159cd0 tx=0x7f071401c4d0 comp rx=0 tx=0).stop 2023-05-22T15:26:27.148 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.133+0000 7f072b4a9640 1 -- 172.21.15.62:0/968580582 shutdown_connections 2023-05-22T15:26:27.149 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.133+0000 7f072b4a9640 1 --2- 172.21.15.62:0/968580582 >> v2:172.21.15.62:3300/0 conn(0x7f072406c220 0x7f072406c6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:27.149 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.133+0000 7f072b4a9640 1 --2- 172.21.15.62:0/968580582 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f07240cc6c0 0x7f07240c8080 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:27.149 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.133+0000 7f072b4a9640 1 --2- 172.21.15.62:0/968580582 >> v2:172.21.15.16:3300/0 conn(0x7f07241563c0 0x7f07240c7b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:27.149 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.133+0000 7f072b4a9640 1 -- 172.21.15.62:0/968580582 >> 172.21.15.62:0/968580582 conn(0x7f07240b1110 msgr2=0x7f07240ce830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:26:27.149 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.137+0000 7f072b4a9640 1 -- 172.21.15.62:0/968580582 shutdown_connections 2023-05-22T15:26:27.149 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.137+0000 7f072b4a9640 1 -- 172.21.15.62:0/968580582 wait complete. 2023-05-22T15:26:27.149 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.137+0000 7f072b4a9640 1 Processor -- start 2023-05-22T15:26:27.149 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.137+0000 7f072b4a9640 1 -- start start 2023-05-22T15:26:27.152 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.137+0000 7f072b4a9640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f072406c220 0x7f072414fda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:27.152 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.137+0000 7f072b4a9640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f07240cc6c0 0x7f07240c90c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:27.152 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.137+0000 7f072b4a9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f072414de10 0x7f07241f7a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:27.153 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.137+0000 7f072b4a9640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0724151150 con 0x7f072414de10 2023-05-22T15:26:27.153 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.137+0000 7f072b4a9640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07241505d0 con 0x7f07240cc6c0 2023-05-22T15:26:27.153 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.137+0000 7f072b4a9640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07241508b0 con 0x7f072406c220 2023-05-22T15:26:27.153 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.137+0000 7f0729245640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f072406c220 0x7f072414fda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:27.153 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.137+0000 7f0728a44640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f07240cc6c0 0x7f07240c90c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:27.153 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.137+0000 7f0729245640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f072406c220 0x7f072414fda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3301/0 says I am v2:172.21.15.62:33970/0 (socket says 172.21.15.62:33970) 2023-05-22T15:26:27.153 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.137+0000 7f0729245640 1 -- 172.21.15.62:0/2055650705 learned_addr learned my addr 172.21.15.62:0/2055650705 (peer_addr_for_me v2:172.21.15.62:0/0) 2023-05-22T15:26:27.154 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.137+0000 7f0729a46640 1 --2- 172.21.15.62:0/2055650705 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f072414de10 0x7f07241f7a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:27.154 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.137+0000 7f0729245640 1 -- 172.21.15.62:0/2055650705 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f07240cc6c0 msgr2=0x7f07240c90c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:27.154 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.137+0000 7f0729245640 1 --2- 172.21.15.62:0/2055650705 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f07240cc6c0 0x7f07240c90c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:27.154 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.137+0000 7f0729245640 1 -- 172.21.15.62:0/2055650705 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f072414de10 msgr2=0x7f07241f7a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:27.154 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.137+0000 7f0729245640 1 --2- 172.21.15.62:0/2055650705 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f072414de10 0x7f07241f7a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:27.154 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.137+0000 7f0729245640 1 -- 172.21.15.62:0/2055650705 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07140094d0 con 0x7f072406c220 2023-05-22T15:26:27.154 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.137+0000 7f0728a44640 1 --2- 172.21.15.62:0/2055650705 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f07240cc6c0 0x7f07240c90c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-05-22T15:26:27.155 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.137+0000 7f0729245640 1 --2- 172.21.15.62:0/2055650705 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f072406c220 0x7f072414fda0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f072000c7f0 tx=0x7f072000ccb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:26:27.155 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.137+0000 7f0719ffb640 1 -- 172.21.15.62:0/2055650705 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0720007820 con 0x7f072406c220 2023-05-22T15:26:27.155 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.137+0000 7f0719ffb640 1 -- 172.21.15.62:0/2055650705 <== mon.2 v2:172.21.15.62:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0720007980 con 0x7f072406c220 2023-05-22T15:26:27.155 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.137+0000 7f072b4a9640 1 -- 172.21.15.62:0/2055650705 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0724070890 con 0x7f072406c220 2023-05-22T15:26:27.155 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.137+0000 7f0719ffb640 1 -- 172.21.15.62:0/2055650705 <== mon.2 v2:172.21.15.62:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0720007d30 con 0x7f072406c220 2023-05-22T15:26:27.155 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.137+0000 7f072b4a9640 1 -- 172.21.15.62:0/2055650705 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0724070700 con 0x7f072406c220 2023-05-22T15:26:27.155 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.137+0000 7f0719ffb640 1 -- 172.21.15.62:0/2055650705 <== mon.2 v2:172.21.15.62:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f072001a660 con 0x7f072406c220 2023-05-22T15:26:27.155 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.137+0000 7f0719ffb640 1 -- 172.21.15.62:0/2055650705 <== mon.2 v2:172.21.15.62:3301/0 5 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (secure 0 0 0) 0x7f0720011b50 con 0x7f072406c220 2023-05-22T15:26:27.156 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.137+0000 7f072b4a9640 1 -- 172.21.15.62:0/2055650705 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0724070a20 con 0x7f072406c220 2023-05-22T15:26:27.156 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.141+0000 7f0719ffb640 1 -- 172.21.15.62:0/2055650705 <== mon.2 v2:172.21.15.62:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72422 (secure 0 0 0) 0x7f0720039b00 con 0x7f072406c220 2023-05-22T15:26:27.179 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:26:27.169+0000 7f90244ca180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-05-22T15:26:27.198 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:26:27.186+0000 7f723d3e1180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-05-22T15:26:27.218 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.205+0000 7f072b4a9640 1 -- 172.21.15.62:0/2055650705 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_command({"prefix": "osd new", "uuid": "022d67db-d4d8-445b-b1b9-8b446a00f5c7", "id": 5} v 0) v1 -- 0x7f072404dbe0 con 0x7f072406c220 2023-05-22T15:26:27.226 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.213+0000 7f0719ffb640 1 -- 172.21.15.62:0/2055650705 <== mon.2 v2:172.21.15.62:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "022d67db-d4d8-445b-b1b9-8b446a00f5c7", "id": 5}]=0 v8) v1 ==== 112+0+2 (secure 0 0 0) 0x7f0720028030 con 0x7f072406c220 2023-05-22T15:26:27.226 INFO:teuthology.orchestra.run.smithi062.stdout:5 2023-05-22T15:26:27.227 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.217+0000 7f072b4a9640 1 -- 172.21.15.62:0/2055650705 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f072406c220 msgr2=0x7f072414fda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:27.227 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.217+0000 7f072b4a9640 1 --2- 172.21.15.62:0/2055650705 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f072406c220 0x7f072414fda0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f072000c7f0 tx=0x7f072000ccb0 comp rx=0 tx=0).stop 2023-05-22T15:26:27.227 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.217+0000 7f072b4a9640 1 -- 172.21.15.62:0/2055650705 shutdown_connections 2023-05-22T15:26:27.227 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.217+0000 7f072b4a9640 1 --2- 172.21.15.62:0/2055650705 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f072414de10 0x7f07241f7a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:27.227 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.217+0000 7f072b4a9640 1 --2- 172.21.15.62:0/2055650705 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f07240cc6c0 0x7f07240c90c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:27.228 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.217+0000 7f072b4a9640 1 --2- 172.21.15.62:0/2055650705 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f072406c220 0x7f072414fda0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:27.228 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.217+0000 7f072b4a9640 1 -- 172.21.15.62:0/2055650705 >> 172.21.15.62:0/2055650705 conn(0x7f07240b1110 msgr2=0x7f072406aaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:26:27.228 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.217+0000 7f072b4a9640 1 -- 172.21.15.62:0/2055650705 shutdown_connections 2023-05-22T15:26:27.228 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.217+0000 7f072b4a9640 1 -- 172.21.15.62:0/2055650705 wait complete. 2023-05-22T15:26:27.242 DEBUG:teuthology.orchestra.run.smithi062:> sudo ceph --cluster ceph osd new 42bd372b-6688-4a1b-8407-e16e8c266e33 6 2023-05-22T15:26:27.263 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:26:27.253+0000 7f90244ca180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-05-22T15:26:27.319 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:26:27.309+0000 7f90244ca180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-05-22T15:26:27.342 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:26:27.330+0000 7f723d3e1180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-05-22T15:26:27.381 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.369+0000 7f2f698ee640 1 Processor -- start 2023-05-22T15:26:27.381 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.369+0000 7f2f698ee640 1 -- start start 2023-05-22T15:26:27.382 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.369+0000 7f2f698ee640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2f6406dab0 0x7f2f640653f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:27.382 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.369+0000 7f2f698ee640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2f6406fef0 0x7f2f64070340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:27.382 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.369+0000 7f2f698ee640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f64070880 con 0x7f2f6415bbe0 2023-05-22T15:26:27.382 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.369+0000 7f2f698ee640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f6406de80 con 0x7f2f6406fef0 2023-05-22T15:26:27.383 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.369+0000 7f2f698ee640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f6415c770 con 0x7f2f6406dab0 2023-05-22T15:26:27.383 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.369+0000 7f2f62ffd640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2f6406dab0 0x7f2f640653f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:27.383 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.369+0000 7f2f62ffd640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2f6406dab0 0x7f2f640653f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3301/0 says I am v2:172.21.15.62:33974/0 (socket says 172.21.15.62:33974) 2023-05-22T15:26:27.383 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.369+0000 7f2f62ffd640 1 -- 172.21.15.62:0/2725057991 learned_addr learned my addr 172.21.15.62:0/2725057991 (peer_addr_for_me v2:172.21.15.62:0/0) 2023-05-22T15:26:27.384 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.373+0000 7f2f627fc640 1 --2- 172.21.15.62:0/2725057991 >> v2:172.21.15.16:3300/0 conn(0x7f2f6406fef0 0x7f2f64070340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:27.384 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.373+0000 7f2f62ffd640 1 -- 172.21.15.62:0/2725057991 >> v2:172.21.15.16:3300/0 conn(0x7f2f6406fef0 msgr2=0x7f2f64070340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:26:27.384 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.373+0000 7f2f62ffd640 1 --2- 172.21.15.62:0/2725057991 >> v2:172.21.15.16:3300/0 conn(0x7f2f6406fef0 0x7f2f64070340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:27.384 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.373+0000 7f2f62ffd640 1 -- 172.21.15.62:0/2725057991 >> v1:172.21.15.62:6789/0 conn(0x7f2f6415bbe0 legacy=0x7f2f6415c060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:27.384 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.373+0000 7f2f62ffd640 1 -- 172.21.15.62:0/2725057991 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f64158eb0 con 0x7f2f6406dab0 2023-05-22T15:26:27.384 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.373+0000 7f2f627fc640 1 --2- 172.21.15.62:0/2725057991 >> v2:172.21.15.16:3300/0 conn(0x7f2f6406fef0 0x7f2f64070340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:26:27.385 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.373+0000 7f2f62ffd640 1 --2- 172.21.15.62:0/2725057991 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2f6406dab0 0x7f2f640653f0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f2f5c009e50 tx=0x7f2f5c01c9e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=71d6783227821177 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:26:27.386 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.373+0000 7f2f61ffb640 1 -- 172.21.15.62:0/2725057991 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f5c0a58a0 con 0x7f2f6406dab0 2023-05-22T15:26:27.386 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.373+0000 7f2f61ffb640 1 -- 172.21.15.62:0/2725057991 <== mon.2 v2:172.21.15.62:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f5c0a5a00 con 0x7f2f6406dab0 2023-05-22T15:26:27.386 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.373+0000 7f2f61ffb640 1 -- 172.21.15.62:0/2725057991 <== mon.2 v2:172.21.15.62:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f5c0a5d90 con 0x7f2f6406dab0 2023-05-22T15:26:27.386 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.373+0000 7f2f698ee640 1 -- 172.21.15.62:0/2725057991 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2f6406dab0 msgr2=0x7f2f640653f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:27.386 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.373+0000 7f2f698ee640 1 --2- 172.21.15.62:0/2725057991 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2f6406dab0 0x7f2f640653f0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f2f5c009e50 tx=0x7f2f5c01c9e0 comp rx=0 tx=0).stop 2023-05-22T15:26:27.386 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.373+0000 7f2f698ee640 1 -- 172.21.15.62:0/2725057991 shutdown_connections 2023-05-22T15:26:27.386 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.373+0000 7f2f698ee640 1 --2- 172.21.15.62:0/2725057991 >> v2:172.21.15.16:3300/0 conn(0x7f2f6406fef0 0x7f2f64070340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:27.387 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.373+0000 7f2f698ee640 1 --2- 172.21.15.62:0/2725057991 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2f6406dab0 0x7f2f640653f0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:27.387 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.373+0000 7f2f698ee640 1 -- 172.21.15.62:0/2725057991 >> 172.21.15.62:0/2725057991 conn(0x7f2f640b1110 msgr2=0x7f2f640ce830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:26:27.387 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.373+0000 7f2f698ee640 1 -- 172.21.15.62:0/2725057991 shutdown_connections 2023-05-22T15:26:27.387 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.373+0000 7f2f698ee640 1 -- 172.21.15.62:0/2725057991 wait complete. 2023-05-22T15:26:27.393 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.377+0000 7f2f698ee640 1 Processor -- start 2023-05-22T15:26:27.393 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.377+0000 7f2f698ee640 1 -- start start 2023-05-22T15:26:27.393 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.377+0000 7f2f698ee640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2f6406dab0 0x7f2f64145900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:27.393 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.377+0000 7f2f698ee640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f6406fef0 0x7f2f64162300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:27.393 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.377+0000 7f2f698ee640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2f6415bbe0 0x7f2f64157ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:27.394 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.377+0000 7f2f698ee640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f64162d50 con 0x7f2f6406fef0 2023-05-22T15:26:27.394 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.377+0000 7f2f698ee640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f64162840 con 0x7f2f6406dab0 2023-05-22T15:26:27.394 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.377+0000 7f2f698ee640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f64142550 con 0x7f2f6415bbe0 2023-05-22T15:26:27.394 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.377+0000 7f2f637fe640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2f6415bbe0 0x7f2f64157ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:27.394 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.377+0000 7f2f62ffd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2f6406dab0 0x7f2f64145900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:27.394 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.377+0000 7f2f637fe640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2f6415bbe0 0x7f2f64157ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3301/0 says I am v2:172.21.15.62:33990/0 (socket says 172.21.15.62:33990) 2023-05-22T15:26:27.394 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.377+0000 7f2f637fe640 1 -- 172.21.15.62:0/980159794 learned_addr learned my addr 172.21.15.62:0/980159794 (peer_addr_for_me v2:172.21.15.62:0/0) 2023-05-22T15:26:27.395 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.377+0000 7f2f627fc640 1 --2- 172.21.15.62:0/980159794 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f6406fef0 0x7f2f64162300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:27.395 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.377+0000 7f2f637fe640 1 -- 172.21.15.62:0/980159794 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2f6406dab0 msgr2=0x7f2f64145900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:27.395 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.377+0000 7f2f637fe640 1 --2- 172.21.15.62:0/980159794 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2f6406dab0 0x7f2f64145900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:27.395 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.377+0000 7f2f637fe640 1 -- 172.21.15.62:0/980159794 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f6406fef0 msgr2=0x7f2f64162300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:27.395 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.377+0000 7f2f637fe640 1 --2- 172.21.15.62:0/980159794 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f6406fef0 0x7f2f64162300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:27.396 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.377+0000 7f2f637fe640 1 -- 172.21.15.62:0/980159794 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f5c009b30 con 0x7f2f6415bbe0 2023-05-22T15:26:27.396 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.377+0000 7f2f637fe640 1 --2- 172.21.15.62:0/980159794 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2f6415bbe0 0x7f2f64157ab0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f2f5400c430 tx=0x7f2f5400c8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:26:27.396 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.377+0000 7f2f5bfff640 1 -- 172.21.15.62:0/980159794 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f54005220 con 0x7f2f6415bbe0 2023-05-22T15:26:27.397 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.377+0000 7f2f5bfff640 1 -- 172.21.15.62:0/980159794 <== mon.2 v2:172.21.15.62:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f54005380 con 0x7f2f6415bbe0 2023-05-22T15:26:27.397 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.377+0000 7f2f5bfff640 1 -- 172.21.15.62:0/980159794 <== mon.2 v2:172.21.15.62:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f54007270 con 0x7f2f6415bbe0 2023-05-22T15:26:27.397 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.377+0000 7f2f698ee640 1 -- 172.21.15.62:0/980159794 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f64158510 con 0x7f2f6415bbe0 2023-05-22T15:26:27.397 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.377+0000 7f2f698ee640 1 -- 172.21.15.62:0/980159794 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f64158b40 con 0x7f2f6415bbe0 2023-05-22T15:26:27.397 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.377+0000 7f2f5bfff640 1 -- 172.21.15.62:0/980159794 <== mon.2 v2:172.21.15.62:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f2f540067e0 con 0x7f2f6415bbe0 2023-05-22T15:26:27.397 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.377+0000 7f2f5bfff640 1 -- 172.21.15.62:0/980159794 <== mon.2 v2:172.21.15.62:3301/0 5 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (secure 0 0 0) 0x7f2f54011590 con 0x7f2f6415bbe0 2023-05-22T15:26:27.397 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.377+0000 7f2f698ee640 1 -- 172.21.15.62:0/980159794 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f64065930 con 0x7f2f6415bbe0 2023-05-22T15:26:27.397 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.381+0000 7f2f5bfff640 1 -- 172.21.15.62:0/980159794 <== mon.2 v2:172.21.15.62:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72422 (secure 0 0 0) 0x7f2f64070880 con 0x7f2f6415bbe0 2023-05-22T15:26:27.405 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:26:27.394+0000 7f723d3e1180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-05-22T15:26:27.456 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.445+0000 7f2f698ee640 1 -- 172.21.15.62:0/980159794 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_command({"prefix": "osd new", "uuid": "42bd372b-6688-4a1b-8407-e16e8c266e33", "id": 6} v 0) v1 -- 0x7f2f6406b0d0 con 0x7f2f6415bbe0 2023-05-22T15:26:27.464 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.453+0000 7f2f5bfff640 1 -- 172.21.15.62:0/980159794 <== mon.2 v2:172.21.15.62:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "42bd372b-6688-4a1b-8407-e16e8c266e33", "id": 6}]=0 v9) v1 ==== 112+0+2 (secure 0 0 0) 0x7f2f54006cc0 con 0x7f2f6415bbe0 2023-05-22T15:26:27.465 INFO:teuthology.orchestra.run.smithi062.stdout:6 2023-05-22T15:26:27.466 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.453+0000 7f2f698ee640 1 -- 172.21.15.62:0/980159794 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2f6415bbe0 msgr2=0x7f2f64157ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:27.467 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.453+0000 7f2f698ee640 1 --2- 172.21.15.62:0/980159794 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2f6415bbe0 0x7f2f64157ab0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f2f5400c430 tx=0x7f2f5400c8f0 comp rx=0 tx=0).stop 2023-05-22T15:26:27.467 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.453+0000 7f2f698ee640 1 -- 172.21.15.62:0/980159794 shutdown_connections 2023-05-22T15:26:27.467 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.453+0000 7f2f698ee640 1 --2- 172.21.15.62:0/980159794 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2f6415bbe0 0x7f2f64157ab0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:27.468 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.453+0000 7f2f698ee640 1 --2- 172.21.15.62:0/980159794 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f6406fef0 0x7f2f64162300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:27.468 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.453+0000 7f2f698ee640 1 --2- 172.21.15.62:0/980159794 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2f6406dab0 0x7f2f64145900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:27.468 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.453+0000 7f2f698ee640 1 -- 172.21.15.62:0/980159794 >> 172.21.15.62:0/980159794 conn(0x7f2f640b1110 msgr2=0x7f2f6415c060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:26:27.468 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.453+0000 7f2f698ee640 1 -- 172.21.15.62:0/980159794 shutdown_connections 2023-05-22T15:26:27.468 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.453+0000 7f2f698ee640 1 -- 172.21.15.62:0/980159794 wait complete. 2023-05-22T15:26:27.472 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:26:27.461+0000 7f90244ca180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-05-22T15:26:27.486 DEBUG:teuthology.orchestra.run.smithi062:> sudo ceph --cluster ceph osd new ac70b2a3-f41b-4bf2-b227-7f2cbb55426d 7 2023-05-22T15:26:27.541 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:26:27.529+0000 7f90244ca180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-05-22T15:26:27.601 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.589+0000 7efd1b1c1640 1 Processor -- start 2023-05-22T15:26:27.601 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.589+0000 7efd1b1c1640 1 -- start start 2023-05-22T15:26:27.602 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.589+0000 7efd1b1c1640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7efd140c67d0 0x7efd140cf730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:27.602 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.589+0000 7efd1b1c1640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7efd140cc6f0 0x7efd140c3db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:27.602 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.589+0000 7efd1b1c1640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd140cfc70 con 0x7efd140b2880 2023-05-22T15:26:27.602 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.589+0000 7efd1b1c1640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd1415a300 con 0x7efd140cc6f0 2023-05-22T15:26:27.603 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.589+0000 7efd1b1c1640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd140c6ba0 con 0x7efd140c67d0 2023-05-22T15:26:27.603 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.593+0000 7efd18f5d640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7efd140c67d0 0x7efd140cf730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:27.603 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.593+0000 7efd13fff640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7efd140b2880 0x7efd140cbfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:42252/0 (socket says 172.21.15.62:42252) 2023-05-22T15:26:27.603 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.593+0000 7efd13fff640 1 -- 172.21.15.62:0/2704143297 learned_addr learned my addr 172.21.15.62:0/2704143297 (peer_addr_for_me v1:172.21.15.62:0/0) 2023-05-22T15:26:27.604 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.593+0000 7efd1975e640 1 --2- 172.21.15.62:0/2704143297 >> v2:172.21.15.16:3300/0 conn(0x7efd140cc6f0 0x7efd140c3db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:27.604 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.593+0000 7efd18f5d640 1 -- 172.21.15.62:0/2704143297 >> v2:172.21.15.16:3300/0 conn(0x7efd140cc6f0 msgr2=0x7efd140c3db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:26:27.604 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.593+0000 7efd18f5d640 1 --2- 172.21.15.62:0/2704143297 >> v2:172.21.15.16:3300/0 conn(0x7efd140cc6f0 0x7efd140c3db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:27.604 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.593+0000 7efd18f5d640 1 -- 172.21.15.62:0/2704143297 >> v1:172.21.15.62:6789/0 conn(0x7efd140b2880 legacy=0x7efd140cbfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:27.604 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.593+0000 7efd18f5d640 1 -- 172.21.15.62:0/2704143297 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd140cfef0 con 0x7efd140c67d0 2023-05-22T15:26:27.604 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.593+0000 7efd137fe640 1 -- 172.21.15.62:0/2704143297 <== mon.1 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3439545638 0 0) 0x7efd140cfc70 con 0x7efd140b2880 2023-05-22T15:26:27.605 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.593+0000 7efd1975e640 1 --2- 172.21.15.62:0/2704143297 >> v2:172.21.15.16:3300/0 conn(0x7efd140cc6f0 0x7efd140c3db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:26:27.606 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.593+0000 7efd18f5d640 1 --2- 172.21.15.62:0/2704143297 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7efd140c67d0 0x7efd140cf730 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7efcfc0097c0 tx=0x7efcfc01c3c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=401cfbd55d4012ee server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:26:27.606 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.593+0000 7efd137fe640 1 -- 172.21.15.62:0/2704143297 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcfc0ac640 con 0x7efd140c67d0 2023-05-22T15:26:27.606 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.593+0000 7efd137fe640 1 -- 172.21.15.62:0/2704143297 <== mon.2 v2:172.21.15.62:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcfc0ac7a0 con 0x7efd140c67d0 2023-05-22T15:26:27.606 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.593+0000 7efd137fe640 1 -- 172.21.15.62:0/2704143297 <== mon.2 v2:172.21.15.62:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcfc0acb30 con 0x7efd140c67d0 2023-05-22T15:26:27.606 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.593+0000 7efd1b1c1640 1 -- 172.21.15.62:0/2704143297 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7efd140c67d0 msgr2=0x7efd140cf730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:27.606 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.593+0000 7efd1b1c1640 1 --2- 172.21.15.62:0/2704143297 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7efd140c67d0 0x7efd140cf730 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7efcfc0097c0 tx=0x7efcfc01c3c0 comp rx=0 tx=0).stop 2023-05-22T15:26:27.606 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.593+0000 7efd1b1c1640 1 -- 172.21.15.62:0/2704143297 shutdown_connections 2023-05-22T15:26:27.607 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.593+0000 7efd1b1c1640 1 --2- 172.21.15.62:0/2704143297 >> v2:172.21.15.16:3300/0 conn(0x7efd140cc6f0 0x7efd140c3db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:27.607 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.593+0000 7efd1b1c1640 1 --2- 172.21.15.62:0/2704143297 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7efd140c67d0 0x7efd140cf730 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:27.607 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.593+0000 7efd1b1c1640 1 -- 172.21.15.62:0/2704143297 >> 172.21.15.62:0/2704143297 conn(0x7efd1406f5a0 msgr2=0x7efd1406f9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:26:27.607 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.593+0000 7efd1b1c1640 1 -- 172.21.15.62:0/2704143297 shutdown_connections 2023-05-22T15:26:27.607 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.593+0000 7efd1b1c1640 1 -- 172.21.15.62:0/2704143297 wait complete. 2023-05-22T15:26:27.609 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.597+0000 7efd1b1c1640 1 Processor -- start 2023-05-22T15:26:27.609 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.597+0000 7efd1b1c1640 1 -- start start 2023-05-22T15:26:27.609 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.597+0000 7efd1b1c1640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efd140b2880 0x7efd14153260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:27.609 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.597+0000 7efd1b1c1640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efd140c67d0 0x7efd141410e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:27.609 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.597+0000 7efd1b1c1640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7efd140cc6f0 0x7efd14152a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:27.610 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.597+0000 7efd1b1c1640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd14157e90 con 0x7efd140b2880 2023-05-22T15:26:27.610 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.597+0000 7efd1b1c1640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd14153840 con 0x7efd140c67d0 2023-05-22T15:26:27.610 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.597+0000 7efd1b1c1640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd141574f0 con 0x7efd140cc6f0 2023-05-22T15:26:27.610 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.597+0000 7efd13fff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efd140c67d0 0x7efd141410e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:27.610 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.597+0000 7efd1975e640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7efd140cc6f0 0x7efd14152a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:27.610 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.597+0000 7efd13fff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efd140c67d0 0x7efd141410e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:38110/0 (socket says 172.21.15.62:38110) 2023-05-22T15:26:27.610 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.597+0000 7efd13fff640 1 -- 172.21.15.62:0/362501983 learned_addr learned my addr 172.21.15.62:0/362501983 (peer_addr_for_me v2:172.21.15.62:0/0) 2023-05-22T15:26:27.611 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.597+0000 7efd18f5d640 1 --2- 172.21.15.62:0/362501983 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efd140b2880 0x7efd14153260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:27.611 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.597+0000 7efd13fff640 1 -- 172.21.15.62:0/362501983 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7efd140cc6f0 msgr2=0x7efd14152a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:27.611 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.597+0000 7efd13fff640 1 --2- 172.21.15.62:0/362501983 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7efd140cc6f0 0x7efd14152a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:27.611 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.597+0000 7efd13fff640 1 -- 172.21.15.62:0/362501983 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efd140b2880 msgr2=0x7efd14153260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:27.611 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.597+0000 7efd13fff640 1 --2- 172.21.15.62:0/362501983 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efd140b2880 0x7efd14153260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:27.611 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.597+0000 7efd13fff640 1 -- 172.21.15.62:0/362501983 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcfc009420 con 0x7efd140c67d0 2023-05-22T15:26:27.611 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.597+0000 7efd13fff640 1 --2- 172.21.15.62:0/362501983 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efd140c67d0 0x7efd141410e0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7efd0400c480 tx=0x7efd0400c940 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:26:27.611 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.597+0000 7efd117fa640 1 -- 172.21.15.62:0/362501983 <== mon.1 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd040074e0 con 0x7efd140c67d0 2023-05-22T15:26:27.612 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.597+0000 7efd117fa640 1 -- 172.21.15.62:0/362501983 <== mon.1 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd04007660 con 0x7efd140c67d0 2023-05-22T15:26:27.612 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.597+0000 7efd1b1c1640 1 -- 172.21.15.62:0/362501983 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd140d0a70 con 0x7efd140c67d0 2023-05-22T15:26:27.612 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.597+0000 7efd117fa640 1 -- 172.21.15.62:0/362501983 <== mon.1 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd04004a30 con 0x7efd140c67d0 2023-05-22T15:26:27.612 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.597+0000 7efd1b1c1640 1 -- 172.21.15.62:0/362501983 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd140d0790 con 0x7efd140c67d0 2023-05-22T15:26:27.612 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.597+0000 7efd117fa640 1 -- 172.21.15.62:0/362501983 <== mon.1 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7efd0401d440 con 0x7efd140c67d0 2023-05-22T15:26:27.612 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.597+0000 7efd117fa640 1 -- 172.21.15.62:0/362501983 <== mon.1 v2:172.21.15.62:3300/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (secure 0 0 0) 0x7efd04012a60 con 0x7efd140c67d0 2023-05-22T15:26:27.613 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.597+0000 7efd1b1c1640 1 -- 172.21.15.62:0/362501983 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd140d0f90 con 0x7efd140c67d0 2023-05-22T15:26:27.613 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.601+0000 7efd117fa640 1 -- 172.21.15.62:0/362501983 <== mon.1 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72422 (secure 0 0 0) 0x7efd04018070 con 0x7efd140c67d0 2023-05-22T15:26:27.679 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.669+0000 7efd1b1c1640 1 -- 172.21.15.62:0/362501983 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd new", "uuid": "ac70b2a3-f41b-4bf2-b227-7f2cbb55426d", "id": 7} v 0) v1 -- 0x7efd14157fd0 con 0x7efd140c67d0 2023-05-22T15:26:27.687 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.677+0000 7efd117fa640 1 -- 172.21.15.62:0/362501983 <== mon.1 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "ac70b2a3-f41b-4bf2-b227-7f2cbb55426d", "id": 7}]=0 v10) v1 ==== 112+0+2 (secure 0 0 0) 0x7efd14157fd0 con 0x7efd140c67d0 2023-05-22T15:26:27.688 INFO:teuthology.orchestra.run.smithi062.stdout:7 2023-05-22T15:26:27.689 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.677+0000 7efd1b1c1640 1 -- 172.21.15.62:0/362501983 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efd140c67d0 msgr2=0x7efd141410e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:27.689 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.677+0000 7efd1b1c1640 1 --2- 172.21.15.62:0/362501983 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efd140c67d0 0x7efd141410e0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7efd0400c480 tx=0x7efd0400c940 comp rx=0 tx=0).stop 2023-05-22T15:26:27.689 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.677+0000 7efd1b1c1640 1 -- 172.21.15.62:0/362501983 shutdown_connections 2023-05-22T15:26:27.689 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.677+0000 7efd1b1c1640 1 --2- 172.21.15.62:0/362501983 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7efd140cc6f0 0x7efd14152a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:27.689 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.677+0000 7efd1b1c1640 1 --2- 172.21.15.62:0/362501983 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efd140c67d0 0x7efd141410e0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:27.690 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.677+0000 7efd1b1c1640 1 --2- 172.21.15.62:0/362501983 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efd140b2880 0x7efd14153260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:27.690 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.677+0000 7efd1b1c1640 1 -- 172.21.15.62:0/362501983 >> 172.21.15.62:0/362501983 conn(0x7efd1406f5a0 msgr2=0x7efd140b1af0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:26:27.690 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.677+0000 7efd1b1c1640 1 -- 172.21.15.62:0/362501983 shutdown_connections 2023-05-22T15:26:27.690 INFO:teuthology.orchestra.run.smithi062.stderr:2023-05-22T15:26:27.677+0000 7efd1b1c1640 1 -- 172.21.15.62:0/362501983 wait complete. 2023-05-22T15:26:27.705 INFO:tasks.ceph.osd.0:Restarting daemon 2023-05-22T15:26:27.706 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2023-05-22T15:26:27.709 INFO:tasks.ceph.osd.0:Started 2023-05-22T15:26:27.709 INFO:tasks.ceph.osd.1:Restarting daemon 2023-05-22T15:26:27.710 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2023-05-22T15:26:27.712 INFO:tasks.ceph.osd.1:Started 2023-05-22T15:26:27.713 INFO:tasks.ceph.osd.2:Restarting daemon 2023-05-22T15:26:27.713 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2023-05-22T15:26:27.715 INFO:tasks.ceph.osd.2:Started 2023-05-22T15:26:27.716 INFO:tasks.ceph.osd.3:Restarting daemon 2023-05-22T15:26:27.716 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2023-05-22T15:26:27.719 INFO:tasks.ceph.osd.3:Started 2023-05-22T15:26:27.719 INFO:tasks.ceph.osd.4:Restarting daemon 2023-05-22T15:26:27.719 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2023-05-22T15:26:27.722 INFO:tasks.ceph.osd.4:Started 2023-05-22T15:26:27.722 INFO:tasks.ceph.osd.5:Restarting daemon 2023-05-22T15:26:27.722 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2023-05-22T15:26:27.724 INFO:tasks.ceph.osd.5:Started 2023-05-22T15:26:27.724 INFO:tasks.ceph.osd.6:Restarting daemon 2023-05-22T15:26:27.725 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2023-05-22T15:26:27.726 INFO:tasks.ceph.osd.6:Started 2023-05-22T15:26:27.727 INFO:tasks.ceph.osd.7:Restarting daemon 2023-05-22T15:26:27.727 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2023-05-22T15:26:27.729 INFO:tasks.ceph.osd.7:Started 2023-05-22T15:26:27.730 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-05-22T15:26:27.894 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:27.882+0000 7fd912a52640 1 Processor -- start 2023-05-22T15:26:27.894 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:27.882+0000 7fd912a52640 1 -- start start 2023-05-22T15:26:27.896 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:27.882+0000 7fd912a52640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fd90c0ca080 0x7fd90c0c8670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:27.896 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:27.882+0000 7fd912a52640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd90c15e470 0x7fd90c0c8bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:27.896 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:27.882+0000 7fd912a52640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd90c15a070 con 0x7fd90c070f10 2023-05-22T15:26:27.896 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:27.882+0000 7fd912a52640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd90c0ca580 con 0x7fd90c15e470 2023-05-22T15:26:27.896 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:27.882+0000 7fd912a52640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd90c15d990 con 0x7fd90c0ca080 2023-05-22T15:26:27.897 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:27.882+0000 7fd90bfff640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fd90c070f10 0x7fd90c0712e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:43006/0 (socket says 172.21.15.16:43006) 2023-05-22T15:26:27.897 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:27.882+0000 7fd90bfff640 1 -- 172.21.15.16:0/1489224542 learned_addr learned my addr 172.21.15.16:0/1489224542 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:26:27.897 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:27.886+0000 7fd910fef640 1 --2- 172.21.15.16:0/1489224542 >> v2:172.21.15.16:3300/0 conn(0x7fd90c15e470 0x7fd90c0c8bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:27.897 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:27.886+0000 7fd90b7fe640 1 --2- 172.21.15.16:0/1489224542 >> v2:172.21.15.62:3300/0 conn(0x7fd90c0ca080 0x7fd90c0c8670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:27.897 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:27.886+0000 7fd90affd640 1 -- 172.21.15.16:0/1489224542 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 980607137 0 0) 0x7fd90c15a070 con 0x7fd90c070f10 2023-05-22T15:26:27.898 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:27.886+0000 7fd90affd640 1 -- 172.21.15.16:0/1489224542 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd8f8003530 con 0x7fd90c070f10 2023-05-22T15:26:27.898 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:27.886+0000 7fd910fef640 1 -- 172.21.15.16:0/1489224542 >> v2:172.21.15.62:3300/0 conn(0x7fd90c0ca080 msgr2=0x7fd90c0c8670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:26:27.898 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:27.886+0000 7fd910fef640 1 --2- 172.21.15.16:0/1489224542 >> v2:172.21.15.62:3300/0 conn(0x7fd90c0ca080 0x7fd90c0c8670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:27.898 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:27.886+0000 7fd910fef640 1 -- 172.21.15.16:0/1489224542 >> v1:172.21.15.16:6789/0 conn(0x7fd90c070f10 legacy=0x7fd90c0712e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:27.898 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:27.886+0000 7fd910fef640 1 -- 172.21.15.16:0/1489224542 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd90c15dc70 con 0x7fd90c15e470 2023-05-22T15:26:27.902 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:27.886+0000 7fd910fef640 1 --2- 172.21.15.16:0/1489224542 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd90c15e470 0x7fd90c0c8bb0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fd90c0c1a80 tx=0x7fd90401c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24981789e802d314 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:26:27.902 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:27.886+0000 7fd90affd640 1 -- 172.21.15.16:0/1489224542 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9040ac620 con 0x7fd90c15e470 2023-05-22T15:26:27.902 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:27.886+0000 7fd90affd640 1 -- 172.21.15.16:0/1489224542 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9040ac780 con 0x7fd90c15e470 2023-05-22T15:26:27.903 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:27.886+0000 7fd90affd640 1 -- 172.21.15.16:0/1489224542 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9040acaf0 con 0x7fd90c15e470 2023-05-22T15:26:27.903 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:27.886+0000 7fd912a52640 1 -- 172.21.15.16:0/1489224542 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd90c15e470 msgr2=0x7fd90c0c8bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:27.903 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:27.886+0000 7fd912a52640 1 --2- 172.21.15.16:0/1489224542 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd90c15e470 0x7fd90c0c8bb0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fd90c0c1a80 tx=0x7fd90401c3a0 comp rx=0 tx=0).stop 2023-05-22T15:26:27.903 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:27.886+0000 7fd912a52640 1 -- 172.21.15.16:0/1489224542 shutdown_connections 2023-05-22T15:26:27.903 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:27.886+0000 7fd912a52640 1 --2- 172.21.15.16:0/1489224542 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd90c15e470 0x7fd90c0c8bb0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:27.903 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:27.886+0000 7fd912a52640 1 --2- 172.21.15.16:0/1489224542 >> v2:172.21.15.62:3300/0 conn(0x7fd90c0ca080 0x7fd90c0c8670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:27.904 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:27.886+0000 7fd912a52640 1 -- 172.21.15.16:0/1489224542 >> 172.21.15.16:0/1489224542 conn(0x7fd90c069720 msgr2=0x7fd90c069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:26:27.904 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:27.886+0000 7fd912a52640 1 -- 172.21.15.16:0/1489224542 shutdown_connections 2023-05-22T15:26:27.904 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:27.886+0000 7fd912a52640 1 -- 172.21.15.16:0/1489224542 wait complete. 2023-05-22T15:26:27.904 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:27.890+0000 7fd912a52640 1 Processor -- start 2023-05-22T15:26:27.905 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:27.890+0000 7fd912a52640 1 -- start start 2023-05-22T15:26:27.905 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:27.890+0000 7fd912a52640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd90c070f10 0x7fd90c0b12b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:27.905 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:27.890+0000 7fd912a52640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd90c0ca080 0x7fd90c0cc350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:27.905 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:27.890+0000 7fd912a52640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd90c15e470 0x7fd90c0c7360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:27.906 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:27.890+0000 7fd912a52640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd90c162ed0 con 0x7fd90c15e470 2023-05-22T15:26:27.906 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:27.890+0000 7fd912a52640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd90c0ccc50 con 0x7fd90c070f10 2023-05-22T15:26:27.906 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:27.890+0000 7fd912a52640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd90c163170 con 0x7fd90c0ca080 2023-05-22T15:26:27.906 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:27.890+0000 7fd90b7fe640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd90c0ca080 0x7fd90c0cc350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:27.907 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:27.890+0000 7fd910fef640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd90c15e470 0x7fd90c0c7360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:27.907 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:27.890+0000 7fd90bfff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd90c070f10 0x7fd90c0b12b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:27.907 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:27.890+0000 7fd90b7fe640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd90c0ca080 0x7fd90c0cc350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3301/0 says I am v2:172.21.15.16:47024/0 (socket says 172.21.15.16:47024) 2023-05-22T15:26:27.907 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:27.890+0000 7fd90b7fe640 1 -- 172.21.15.16:0/644645133 learned_addr learned my addr 172.21.15.16:0/644645133 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:26:27.908 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:27.890+0000 7fd910fef640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd90c15e470 0x7fd90c0c7360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47328/0 (socket says 172.21.15.16:47328) 2023-05-22T15:26:27.908 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:27.890+0000 7fd910fef640 1 -- 172.21.15.16:0/644645133 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd90c0ca080 msgr2=0x7fd90c0cc350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:27.908 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:27.890+0000 7fd910fef640 1 --2- 172.21.15.16:0/644645133 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd90c0ca080 0x7fd90c0cc350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:27.908 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:27.890+0000 7fd910fef640 1 -- 172.21.15.16:0/644645133 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd90c070f10 msgr2=0x7fd90c0b12b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:27.908 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:27.890+0000 7fd910fef640 1 --2- 172.21.15.16:0/644645133 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd90c070f10 0x7fd90c0b12b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:27.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:27.890+0000 7fd910fef640 1 -- 172.21.15.16:0/644645133 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd904009420 con 0x7fd90c15e470 2023-05-22T15:26:27.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:27.890+0000 7fd90bfff640 1 --2- 172.21.15.16:0/644645133 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd90c070f10 0x7fd90c0b12b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-05-22T15:26:27.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:27.890+0000 7fd910fef640 1 --2- 172.21.15.16:0/644645133 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd90c15e470 0x7fd90c0c7360 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fd9040ac000 tx=0x7fd904019140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:26:27.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:27.890+0000 7fd908ff9640 1 -- 172.21.15.16:0/644645133 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9040ac8f0 con 0x7fd90c15e470 2023-05-22T15:26:27.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:27.890+0000 7fd908ff9640 1 -- 172.21.15.16:0/644645133 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9040aca50 con 0x7fd90c15e470 2023-05-22T15:26:27.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:27.890+0000 7fd912a52640 1 -- 172.21.15.16:0/644645133 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd90c1673a0 con 0x7fd90c15e470 2023-05-22T15:26:27.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:27.890+0000 7fd908ff9640 1 -- 172.21.15.16:0/644645133 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9040acdc0 con 0x7fd90c15e470 2023-05-22T15:26:27.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:27.890+0000 7fd912a52640 1 -- 172.21.15.16:0/644645133 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd90c0c9500 con 0x7fd90c15e470 2023-05-22T15:26:27.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:27.890+0000 7fd908ff9640 1 -- 172.21.15.16:0/644645133 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fd9040b3040 con 0x7fd90c15e470 2023-05-22T15:26:27.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:27.890+0000 7fd908ff9640 1 -- 172.21.15.16:0/644645133 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7fd9040bead0 con 0x7fd90c15e470 2023-05-22T15:26:27.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:27.890+0000 7fd912a52640 1 -- 172.21.15.16:0/644645133 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd90c167530 con 0x7fd90c15e470 2023-05-22T15:26:27.914 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:27.902+0000 7fd908ff9640 1 -- 172.21.15.16:0/644645133 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72422 (secure 0 0 0) 0x7fd9040b9020 con 0x7fd90c15e470 2023-05-22T15:26:27.971 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:27.962+0000 7fd912a52640 1 -- 172.21.15.16:0/644645133 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd90c1632b0 con 0x7fd90c15e470 2023-05-22T15:26:27.972 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:27.962+0000 7fd908ff9640 1 -- 172.21.15.16:0/644645133 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7057 (secure 0 0 0) 0x7fd90c1632b0 con 0x7fd90c15e470 2023-05-22T15:26:27.973 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:26:27.973 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":10,"fsid":"b2719834-7847-4d98-90a6-f65d57f85c49","created":"2023-05-22T15:26:25.259314+0000","modified":"2023-05-22T15:26:27.675668+0000","last_up_change":"0.000000","last_in_change":"2023-05-22T15:26:27.675668+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9e7ff854-b25f-4cbb-a5c2-1799d9c53d2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4702fbc9-a927-4565-aca5-41dc69cc8264","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1e7bba46-4eba-47ef-bb81-5e2d371638ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"383cacc0-5a88-4904-990e-498ca59ef48a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"970500ef-95b7-42c1-b080-eed4fae7013d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"022d67db-d4d8-445b-b1b9-8b446a00f5c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"42bd372b-6688-4a1b-8407-e16e8c266e33","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ac70b2a3-f41b-4bf2-b227-7f2cbb55426d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-05-22T15:26:27.974 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:27.962+0000 7fd912a52640 1 -- 172.21.15.16:0/644645133 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd90c15e470 msgr2=0x7fd90c0c7360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:27.974 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:27.962+0000 7fd912a52640 1 --2- 172.21.15.16:0/644645133 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd90c15e470 0x7fd90c0c7360 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fd9040ac000 tx=0x7fd904019140 comp rx=0 tx=0).stop 2023-05-22T15:26:27.974 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:27.962+0000 7fd912a52640 1 -- 172.21.15.16:0/644645133 shutdown_connections 2023-05-22T15:26:27.975 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:27.962+0000 7fd912a52640 1 --2- 172.21.15.16:0/644645133 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd90c15e470 0x7fd90c0c7360 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:27.975 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:27.962+0000 7fd912a52640 1 --2- 172.21.15.16:0/644645133 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd90c0ca080 0x7fd90c0cc350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:27.975 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:27.962+0000 7fd912a52640 1 --2- 172.21.15.16:0/644645133 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd90c070f10 0x7fd90c0b12b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:27.975 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:27.962+0000 7fd912a52640 1 -- 172.21.15.16:0/644645133 >> 172.21.15.16:0/644645133 conn(0x7fd90c069720 msgr2=0x7fd90c1564a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:26:27.975 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:27.962+0000 7fd912a52640 1 -- 172.21.15.16:0/644645133 shutdown_connections 2023-05-22T15:26:27.975 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:27.962+0000 7fd912a52640 1 -- 172.21.15.16:0/644645133 wait complete. 2023-05-22T15:26:27.990 INFO:tasks.ceph.ceph_manager.ceph:[] 2023-05-22T15:26:27.990 INFO:tasks.ceph:Waiting for OSDs to come up 2023-05-22T15:26:28.077 INFO:tasks.ceph.mgr.y.smithi016.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2023-05-22T15:26:28.077 INFO:tasks.ceph.mgr.y.smithi016.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2023-05-22T15:26:28.078 INFO:tasks.ceph.mgr.y.smithi016.stderr: from numpy import show_config as show_numpy_config 2023-05-22T15:26:28.124 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:26:28.114+0000 7f723d3e1180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-05-22T15:26:28.210 INFO:tasks.ceph.mgr.x.smithi062.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2023-05-22T15:26:28.211 INFO:tasks.ceph.mgr.x.smithi062.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2023-05-22T15:26:28.211 INFO:tasks.ceph.mgr.x.smithi062.stderr: from numpy import show_config as show_numpy_config 2023-05-22T15:26:28.259 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:26:28.246+0000 7f723d3e1180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-05-22T15:26:28.294 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:26:28.285+0000 7f90244ca180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-05-22T15:26:28.323 INFO:tasks.ceph.osd.5.smithi062.stderr:2023-05-22T15:26:28.313+0000 7f627d961640 -1 Falling back to public interface 2023-05-22T15:26:28.327 INFO:tasks.ceph.osd.7.smithi062.stderr:2023-05-22T15:26:28.317+0000 7f8a35d78640 -1 Falling back to public interface 2023-05-22T15:26:28.340 INFO:tasks.ceph.osd.4.smithi062.stderr:2023-05-22T15:26:28.329+0000 7efd9fea8640 -1 Falling back to public interface 2023-05-22T15:26:28.341 INFO:tasks.ceph.osd.6.smithi062.stderr:2023-05-22T15:26:28.329+0000 7efd7e2ad640 -1 Falling back to public interface 2023-05-22T15:26:28.364 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-05-22T15:26:28.354+0000 7f1c87e23640 -1 Falling back to public interface 2023-05-22T15:26:28.376 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-05-22T15:26:28.366+0000 7f8bb4db1640 -1 Falling back to public interface 2023-05-22T15:26:28.384 INFO:tasks.ceph.osd.3.smithi016.stderr:2023-05-22T15:26:28.374+0000 7ff74b8d8640 -1 Falling back to public interface 2023-05-22T15:26:28.424 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-05-22T15:26:28.414+0000 7f8572e33640 -1 Falling back to public interface 2023-05-22T15:26:28.442 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:26:28.430+0000 7f723d3e1180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-05-22T15:26:28.472 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:26:28.461+0000 7f90244ca180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-05-22T15:26:28.511 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:26:28.498+0000 7f723d3e1180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-05-22T15:26:28.638 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:26:28.626+0000 7f723d3e1180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-05-22T15:26:28.639 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:26:28.629+0000 7f90244ca180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-05-22T15:26:28.706 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:26:28.693+0000 7f90244ca180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-05-22T15:26:28.794 DEBUG:teuthology.orchestra.run.smithi016:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-05-22T15:26:28.834 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:26:28.821+0000 7f90244ca180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-05-22T15:26:28.910 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:28.898+0000 7fc132639640 1 Processor -- start 2023-05-22T15:26:28.911 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:28.898+0000 7fc132639640 1 -- start start 2023-05-22T15:26:28.912 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:28.902+0000 7fc132639640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc12c0ceef0 0x7fc12c06d660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:28.912 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:28.902+0000 7fc132639640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fc12c069710 0x7fc12c069b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:28.913 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:28.902+0000 7fc132639640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc12c06e210 con 0x7fc12c070be0 2023-05-22T15:26:28.913 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:28.902+0000 7fc132639640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc12c06a050 con 0x7fc12c0ceef0 2023-05-22T15:26:28.913 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:28.902+0000 7fc132639640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc12c071b70 con 0x7fc12c069710 2023-05-22T15:26:28.914 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:28.902+0000 7fc12bfff640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc12c0ceef0 0x7fc12c06d660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:28.914 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:28.902+0000 7fc12b7fe640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fc12c070be0 0x7fc12c15f880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:43010/0 (socket says 172.21.15.16:43010) 2023-05-22T15:26:28.914 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:28.902+0000 7fc12b7fe640 1 -- 172.21.15.16:0/1812740515 learned_addr learned my addr 172.21.15.16:0/1812740515 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:26:28.914 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:28.902+0000 7fc130bd6640 1 --2- 172.21.15.16:0/1812740515 >> v2:172.21.15.62:3300/0 conn(0x7fc12c069710 0x7fc12c069b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:28.915 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:28.902+0000 7fc12bfff640 1 -- 172.21.15.16:0/1812740515 >> v2:172.21.15.62:3300/0 conn(0x7fc12c069710 msgr2=0x7fc12c069b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:26:28.916 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:28.902+0000 7fc12bfff640 1 --2- 172.21.15.16:0/1812740515 >> v2:172.21.15.62:3300/0 conn(0x7fc12c069710 0x7fc12c069b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:28.916 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:28.902+0000 7fc12bfff640 1 -- 172.21.15.16:0/1812740515 >> v1:172.21.15.16:6789/0 conn(0x7fc12c070be0 legacy=0x7fc12c15f880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:28.916 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:28.902+0000 7fc12bfff640 1 -- 172.21.15.16:0/1812740515 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc12c15c0d0 con 0x7fc12c0ceef0 2023-05-22T15:26:28.916 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:28.902+0000 7fc12affd640 1 -- 172.21.15.16:0/1812740515 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1714189276 0 0) 0x7fc12c06e210 con 0x7fc12c070be0 2023-05-22T15:26:28.921 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:28.902+0000 7fc12bfff640 1 --2- 172.21.15.16:0/1812740515 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc12c0ceef0 0x7fc12c06d660 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fc12c0c98a0 tx=0x7fc12001c430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7734957c5db3ae02 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:26:28.922 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:28.906+0000 7fc12affd640 1 -- 172.21.15.16:0/1812740515 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1200a7070 con 0x7fc12c0ceef0 2023-05-22T15:26:28.922 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:28.906+0000 7fc12affd640 1 -- 172.21.15.16:0/1812740515 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1200ab430 con 0x7fc12c0ceef0 2023-05-22T15:26:28.922 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:28.906+0000 7fc12affd640 1 -- 172.21.15.16:0/1812740515 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1200ab7c0 con 0x7fc12c0ceef0 2023-05-22T15:26:28.923 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:28.906+0000 7fc132639640 1 -- 172.21.15.16:0/1812740515 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc12c0ceef0 msgr2=0x7fc12c06d660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:28.923 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:28.906+0000 7fc132639640 1 --2- 172.21.15.16:0/1812740515 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc12c0ceef0 0x7fc12c06d660 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fc12c0c98a0 tx=0x7fc12001c430 comp rx=0 tx=0).stop 2023-05-22T15:26:28.923 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:28.906+0000 7fc132639640 1 -- 172.21.15.16:0/1812740515 shutdown_connections 2023-05-22T15:26:28.923 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:28.906+0000 7fc132639640 1 --2- 172.21.15.16:0/1812740515 >> v2:172.21.15.62:3300/0 conn(0x7fc12c069710 0x7fc12c069b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:28.924 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:28.906+0000 7fc132639640 1 --2- 172.21.15.16:0/1812740515 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc12c0ceef0 0x7fc12c06d660 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:28.924 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:28.906+0000 7fc132639640 1 -- 172.21.15.16:0/1812740515 >> 172.21.15.16:0/1812740515 conn(0x7fc12c0b0dd0 msgr2=0x7fc12c0b11d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:26:28.924 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:28.906+0000 7fc132639640 1 -- 172.21.15.16:0/1812740515 shutdown_connections 2023-05-22T15:26:28.925 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:28.906+0000 7fc132639640 1 -- 172.21.15.16:0/1812740515 wait complete. 2023-05-22T15:26:28.925 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:28.906+0000 7fc132639640 1 Processor -- start 2023-05-22T15:26:28.925 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:28.906+0000 7fc132639640 1 -- start start 2023-05-22T15:26:28.925 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:28.906+0000 7fc132639640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc12c069710 0x7fc12c15ced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:28.926 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:28.906+0000 7fc132639640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc12c070be0 0x7fc12c15e870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:28.926 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:28.906+0000 7fc132639640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc12c0ceef0 0x7fc12c1500e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:28.926 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:28.906+0000 7fc132639640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc12c1513d0 con 0x7fc12c070be0 2023-05-22T15:26:28.926 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:28.906+0000 7fc132639640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc12c15f160 con 0x7fc12c0ceef0 2023-05-22T15:26:28.927 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:28.906+0000 7fc132639640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc12c15d410 con 0x7fc12c069710 2023-05-22T15:26:28.927 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:28.906+0000 7fc12b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc12c070be0 0x7fc12c15e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:28.927 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:28.906+0000 7fc12b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc12c070be0 0x7fc12c15e870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47348/0 (socket says 172.21.15.16:47348) 2023-05-22T15:26:28.928 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:28.906+0000 7fc12b7fe640 1 -- 172.21.15.16:0/1199338770 learned_addr learned my addr 172.21.15.16:0/1199338770 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:26:28.928 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:28.906+0000 7fc12bfff640 1 --2- 172.21.15.16:0/1199338770 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc12c069710 0x7fc12c15ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:28.928 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:28.906+0000 7fc130bd6640 1 --2- 172.21.15.16:0/1199338770 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc12c0ceef0 0x7fc12c1500e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:28.928 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:28.906+0000 7fc12b7fe640 1 -- 172.21.15.16:0/1199338770 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc12c069710 msgr2=0x7fc12c15ced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:28.929 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:28.906+0000 7fc12b7fe640 1 --2- 172.21.15.16:0/1199338770 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc12c069710 0x7fc12c15ced0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:28.929 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:28.906+0000 7fc12b7fe640 1 -- 172.21.15.16:0/1199338770 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc12c0ceef0 msgr2=0x7fc12c1500e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:28.929 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:28.906+0000 7fc12b7fe640 1 --2- 172.21.15.16:0/1199338770 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc12c0ceef0 0x7fc12c1500e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:28.929 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:28.906+0000 7fc12b7fe640 1 -- 172.21.15.16:0/1199338770 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc120009420 con 0x7fc12c070be0 2023-05-22T15:26:28.929 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:28.906+0000 7fc12b7fe640 1 --2- 172.21.15.16:0/1199338770 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc12c070be0 0x7fc12c15e870 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fc11c00c550 tx=0x7fc11c00ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:26:28.929 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:28.906+0000 7fc128ff9640 1 -- 172.21.15.16:0/1199338770 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc11c0074e0 con 0x7fc12c070be0 2023-05-22T15:26:28.930 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:28.906+0000 7fc128ff9640 1 -- 172.21.15.16:0/1199338770 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc11c007640 con 0x7fc12c070be0 2023-05-22T15:26:28.930 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:28.906+0000 7fc132639640 1 -- 172.21.15.16:0/1199338770 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc12c15ff90 con 0x7fc12c070be0 2023-05-22T15:26:28.930 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:28.906+0000 7fc128ff9640 1 -- 172.21.15.16:0/1199338770 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc11c0049f0 con 0x7fc12c070be0 2023-05-22T15:26:28.931 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:28.906+0000 7fc132639640 1 -- 172.21.15.16:0/1199338770 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc12c071890 con 0x7fc12c070be0 2023-05-22T15:26:28.931 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:28.906+0000 7fc128ff9640 1 -- 172.21.15.16:0/1199338770 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fc11c01d440 con 0x7fc12c070be0 2023-05-22T15:26:28.931 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:28.906+0000 7fc128ff9640 1 -- 172.21.15.16:0/1199338770 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7fc11c012ad0 con 0x7fc12c070be0 2023-05-22T15:26:28.931 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:28.910+0000 7fc132639640 1 -- 172.21.15.16:0/1199338770 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc12c168c20 con 0x7fc12c070be0 2023-05-22T15:26:28.932 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:28.910+0000 7fc128ff9640 1 -- 172.21.15.16:0/1199338770 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72422 (secure 0 0 0) 0x7fc11c018070 con 0x7fc12c070be0 2023-05-22T15:26:28.986 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:28.974+0000 7fc132639640 1 -- 172.21.15.16:0/1199338770 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc12c151580 con 0x7fc12c070be0 2023-05-22T15:26:28.986 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:28.974+0000 7fc128ff9640 1 -- 172.21.15.16:0/1199338770 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7057 (secure 0 0 0) 0x7fc12c151580 con 0x7fc12c070be0 2023-05-22T15:26:28.987 INFO:teuthology.misc.health.smithi016.stdout: 2023-05-22T15:26:28.987 INFO:teuthology.misc.health.smithi016.stdout:{"epoch":10,"fsid":"b2719834-7847-4d98-90a6-f65d57f85c49","created":"2023-05-22T15:26:25.259314+0000","modified":"2023-05-22T15:26:27.675668+0000","last_up_change":"0.000000","last_in_change":"2023-05-22T15:26:27.675668+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9e7ff854-b25f-4cbb-a5c2-1799d9c53d2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4702fbc9-a927-4565-aca5-41dc69cc8264","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1e7bba46-4eba-47ef-bb81-5e2d371638ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"383cacc0-5a88-4904-990e-498ca59ef48a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"970500ef-95b7-42c1-b080-eed4fae7013d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"022d67db-d4d8-445b-b1b9-8b446a00f5c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"42bd372b-6688-4a1b-8407-e16e8c266e33","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ac70b2a3-f41b-4bf2-b227-7f2cbb55426d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-05-22T15:26:28.988 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:28.978+0000 7fc132639640 1 -- 172.21.15.16:0/1199338770 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc12c070be0 msgr2=0x7fc12c15e870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:28.989 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:28.978+0000 7fc132639640 1 --2- 172.21.15.16:0/1199338770 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc12c070be0 0x7fc12c15e870 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fc11c00c550 tx=0x7fc11c00ca10 comp rx=0 tx=0).stop 2023-05-22T15:26:28.989 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:28.978+0000 7fc132639640 1 -- 172.21.15.16:0/1199338770 shutdown_connections 2023-05-22T15:26:28.989 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:28.978+0000 7fc132639640 1 --2- 172.21.15.16:0/1199338770 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc12c0ceef0 0x7fc12c1500e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:28.990 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:28.978+0000 7fc132639640 1 --2- 172.21.15.16:0/1199338770 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc12c070be0 0x7fc12c15e870 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:28.990 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:28.978+0000 7fc132639640 1 --2- 172.21.15.16:0/1199338770 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc12c069710 0x7fc12c15ced0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:28.990 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:28.978+0000 7fc132639640 1 -- 172.21.15.16:0/1199338770 >> 172.21.15.16:0/1199338770 conn(0x7fc12c0b0dd0 msgr2=0x7fc12c0ca340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:26:28.991 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:28.978+0000 7fc132639640 1 -- 172.21.15.16:0/1199338770 shutdown_connections 2023-05-22T15:26:28.991 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:28.978+0000 7fc132639640 1 -- 172.21.15.16:0/1199338770 wait complete. 2023-05-22T15:26:29.002 DEBUG:teuthology.misc:0 of 8 OSDs are up 2023-05-22T15:26:29.265 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:26:29.254+0000 7f723d3e1180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-05-22T15:26:29.345 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:26:29.334+0000 7f723d3e1180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-05-22T15:26:29.437 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:26:29.425+0000 7f90244ca180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-05-22T15:26:29.517 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:26:29.505+0000 7f90244ca180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-05-22T15:26:29.521 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:26:29.510+0000 7f723d3e1180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-05-22T15:26:29.598 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:26:29.586+0000 7f723d3e1180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-05-22T15:26:29.695 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:26:29.685+0000 7f90244ca180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-05-22T15:26:29.772 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:26:29.761+0000 7f90244ca180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-05-22T15:26:29.855 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:26:29.842+0000 7f723d3e1180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-05-22T15:26:29.916 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:26:29.906+0000 7f723d3e1180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-05-22T15:26:30.044 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:26:30.033+0000 7f90244ca180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-05-22T15:26:30.106 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:26:30.097+0000 7f90244ca180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-05-22T15:26:30.962 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:26:30.950+0000 7f71e4ea9640 -1 mgr.server handle_report got status from non-daemon mon.c 2023-05-22T15:26:30.965 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:26:30.954+0000 7f71e4ea9640 -1 mgr.server handle_report got status from non-daemon mon.b 2023-05-22T15:26:32.146 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-05-22T15:26:32.138+0000 7f1c87e23640 -1 osd.1 0 log_to_monitors true 2023-05-22T15:26:32.163 INFO:tasks.ceph.osd.4.smithi062.stderr:2023-05-22T15:26:32.153+0000 7efd9fea8640 -1 osd.4 0 log_to_monitors true 2023-05-22T15:26:32.190 INFO:tasks.ceph.osd.7.smithi062.stderr:2023-05-22T15:26:32.177+0000 7f8a35d78640 -1 osd.7 0 log_to_monitors true 2023-05-22T15:26:32.267 INFO:tasks.ceph.osd.5.smithi062.stderr:2023-05-22T15:26:32.257+0000 7f627d961640 -1 osd.5 0 log_to_monitors true 2023-05-22T15:26:32.280 INFO:tasks.ceph.osd.6.smithi062.stderr:2023-05-22T15:26:32.269+0000 7efd7e2ad640 -1 osd.6 0 log_to_monitors true 2023-05-22T15:26:32.291 INFO:tasks.ceph.osd.3.smithi016.stderr:2023-05-22T15:26:32.282+0000 7ff74b8d8640 -1 osd.3 0 log_to_monitors true 2023-05-22T15:26:32.318 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-05-22T15:26:32.310+0000 7f8bb4db1640 -1 osd.2 0 log_to_monitors true 2023-05-22T15:26:32.367 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-05-22T15:26:32.362+0000 7f8572e33640 -1 osd.0 0 log_to_monitors true 2023-05-22T15:26:35.451 INFO:tasks.ceph.osd.3.smithi016.stderr:2023-05-22T15:26:35.442+0000 7ff74787d640 -1 osd.3 0 waiting for initial osdmap 2023-05-22T15:26:35.475 INFO:tasks.ceph.osd.3.smithi016.stderr:2023-05-22T15:26:35.470+0000 7ff73f6b0640 -1 osd.3 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-05-22T15:26:35.546 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-05-22T15:26:35.538+0000 7f1c83dc8640 -1 osd.1 0 waiting for initial osdmap 2023-05-22T15:26:35.575 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-05-22T15:26:35.570+0000 7f1c7bbfb640 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-05-22T15:26:35.634 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-05-22T15:26:35.626+0000 7f8bb0d56640 -1 osd.2 0 waiting for initial osdmap 2023-05-22T15:26:35.655 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-05-22T15:26:35.650+0000 7f8ba8b89640 -1 osd.2 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-05-22T15:26:35.702 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-05-22T15:26:35.694+0000 7f856edd8640 -1 osd.0 0 waiting for initial osdmap 2023-05-22T15:26:35.727 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-05-22T15:26:35.722+0000 7f8566c0b640 -1 osd.0 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-05-22T15:26:35.808 DEBUG:teuthology.orchestra.run.smithi016:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-05-22T15:26:35.926 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:35.918+0000 7fd21e42a640 1 Processor -- start 2023-05-22T15:26:35.927 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:35.918+0000 7fd21e42a640 1 -- start start 2023-05-22T15:26:35.929 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:35.922+0000 7fd21e42a640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fd21815c6c0 0x7fd2180d01f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:35.930 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:35.922+0000 7fd21e42a640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd2180d0730 0x7fd21815e1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:35.930 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:35.922+0000 7fd21e42a640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd21815cb20 con 0x7fd2180b19d0 2023-05-22T15:26:35.930 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:35.922+0000 7fd21e42a640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2180d0bb0 con 0x7fd21815c6c0 2023-05-22T15:26:35.930 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:35.922+0000 7fd21e42a640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2180b4910 con 0x7fd2180d0730 2023-05-22T15:26:35.930 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:35.922+0000 7fd217fff640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fd2180b19d0 0x7fd2180b4130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:36300/0 (socket says 172.21.15.16:36300) 2023-05-22T15:26:35.931 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:35.922+0000 7fd217fff640 1 -- 172.21.15.16:0/4005700310 learned_addr learned my addr 172.21.15.16:0/4005700310 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:26:35.931 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:35.922+0000 7fd21c9c7640 1 --2- 172.21.15.16:0/4005700310 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd2180d0730 0x7fd21815e1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:35.931 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:35.922+0000 7fd2177fe640 1 --2- 172.21.15.16:0/4005700310 >> v2:172.21.15.62:3300/0 conn(0x7fd21815c6c0 0x7fd2180d01f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:35.931 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:35.922+0000 7fd216ffd640 1 -- 172.21.15.16:0/4005700310 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4247871405 0 0) 0x7fd21815cb20 con 0x7fd2180b19d0 2023-05-22T15:26:35.931 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:35.922+0000 7fd216ffd640 1 -- 172.21.15.16:0/4005700310 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd200003530 con 0x7fd2180b19d0 2023-05-22T15:26:35.931 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:35.922+0000 7fd216ffd640 1 -- 172.21.15.16:0/4005700310 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4093970528 0 0) 0x7fd200003530 con 0x7fd2180b19d0 2023-05-22T15:26:35.932 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:35.922+0000 7fd216ffd640 1 -- 172.21.15.16:0/4005700310 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd2180d0730 msgr2=0x7fd21815e1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:26:35.932 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:35.922+0000 7fd216ffd640 1 --2- 172.21.15.16:0/4005700310 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd2180d0730 0x7fd21815e1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:35.932 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:35.922+0000 7fd216ffd640 1 -- 172.21.15.16:0/4005700310 >> v2:172.21.15.62:3300/0 conn(0x7fd21815c6c0 msgr2=0x7fd2180d01f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:26:35.932 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:35.922+0000 7fd216ffd640 1 --2- 172.21.15.16:0/4005700310 >> v2:172.21.15.62:3300/0 conn(0x7fd21815c6c0 0x7fd2180d01f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:35.932 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:35.922+0000 7fd216ffd640 1 -- 172.21.15.16:0/4005700310 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2180b4bf0 con 0x7fd2180b19d0 2023-05-22T15:26:35.932 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:35.922+0000 7fd216ffd640 1 -- 172.21.15.16:0/4005700310 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7fd208001680 con 0x7fd2180b19d0 2023-05-22T15:26:35.932 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:35.922+0000 7fd216ffd640 1 -- 172.21.15.16:0/4005700310 >> v1:172.21.15.16:6789/0 conn(0x7fd2180b19d0 legacy=0x7fd2180b4130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:35.933 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:35.922+0000 7fd216ffd640 1 --2- 172.21.15.16:0/4005700310 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd2180d0730 0x7fd200003980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:35.933 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:35.922+0000 7fd216ffd640 1 --2- 172.21.15.16:0/4005700310 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd200003f00 0x7fd2000047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:35.933 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:35.922+0000 7fd216ffd640 1 --2- 172.21.15.16:0/4005700310 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd200004d10 0x7fd200005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:35.933 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:35.922+0000 7fd216ffd640 1 -- 172.21.15.16:0/4005700310 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2180d0bb0 con 0x7fd200003f00 2023-05-22T15:26:35.933 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:35.922+0000 7fd216ffd640 1 -- 172.21.15.16:0/4005700310 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2180b4910 con 0x7fd200004d10 2023-05-22T15:26:35.933 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:35.922+0000 7fd216ffd640 1 -- 172.21.15.16:0/4005700310 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd20000b9c0 con 0x7fd2180d0730 2023-05-22T15:26:35.933 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:35.922+0000 7fd2177fe640 1 --2- 172.21.15.16:0/4005700310 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd200003f00 0x7fd2000047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:35.934 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:35.922+0000 7fd217fff640 1 --2- 172.21.15.16:0/4005700310 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd2180d0730 0x7fd200003980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:35.934 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:35.922+0000 7fd21c9c7640 1 --2- 172.21.15.16:0/4005700310 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd200004d10 0x7fd200005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:35.934 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:35.922+0000 7fd2177fe640 1 -- 172.21.15.16:0/4005700310 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd2180d0730 msgr2=0x7fd200003980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:26:35.934 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:35.922+0000 7fd2177fe640 1 --2- 172.21.15.16:0/4005700310 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd2180d0730 0x7fd200003980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:35.934 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:35.922+0000 7fd2177fe640 1 -- 172.21.15.16:0/4005700310 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd200004d10 msgr2=0x7fd200005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:26:35.934 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:35.922+0000 7fd2177fe640 1 --2- 172.21.15.16:0/4005700310 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd200004d10 0x7fd200005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:35.934 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:35.922+0000 7fd2177fe640 1 -- 172.21.15.16:0/4005700310 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd200003530 con 0x7fd200003f00 2023-05-22T15:26:35.935 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:35.922+0000 7fd217fff640 1 --2- 172.21.15.16:0/4005700310 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd2180d0730 0x7fd200003980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T15:26:35.935 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:35.922+0000 7fd21c9c7640 1 --2- 172.21.15.16:0/4005700310 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd200004d10 0x7fd200005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T15:26:35.937 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:35.922+0000 7fd2177fe640 1 --2- 172.21.15.16:0/4005700310 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd200003f00 0x7fd2000047d0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fd20c009ae0 tx=0x7fd20c01ca90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f3650b1f10023c6 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:26:35.938 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:35.922+0000 7fd216ffd640 1 -- 172.21.15.16:0/4005700310 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd20c0a5680 con 0x7fd200003f00 2023-05-22T15:26:35.938 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:35.922+0000 7fd216ffd640 1 -- 172.21.15.16:0/4005700310 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd20c0a57e0 con 0x7fd200003f00 2023-05-22T15:26:35.938 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:35.922+0000 7fd216ffd640 1 -- 172.21.15.16:0/4005700310 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd20c0a5b90 con 0x7fd200003f00 2023-05-22T15:26:35.938 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:35.926+0000 7fd21e42a640 1 -- 172.21.15.16:0/4005700310 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd200003f00 msgr2=0x7fd2000047d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:35.938 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:35.926+0000 7fd21e42a640 1 --2- 172.21.15.16:0/4005700310 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd200003f00 0x7fd2000047d0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fd20c009ae0 tx=0x7fd20c01ca90 comp rx=0 tx=0).stop 2023-05-22T15:26:35.938 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:35.926+0000 7fd21c9c7640 1 -- 172.21.15.16:0/4005700310 reap_dead start 2023-05-22T15:26:35.939 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:35.926+0000 7fd21e42a640 1 -- 172.21.15.16:0/4005700310 shutdown_connections 2023-05-22T15:26:35.939 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:35.926+0000 7fd21e42a640 1 -- 172.21.15.16:0/4005700310 >> 172.21.15.16:0/4005700310 conn(0x7fd21809b8a0 msgr2=0x7fd2180d26b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:26:35.939 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:35.926+0000 7fd21e42a640 1 -- 172.21.15.16:0/4005700310 shutdown_connections 2023-05-22T15:26:35.939 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:35.926+0000 7fd21e42a640 1 -- 172.21.15.16:0/4005700310 wait complete. 2023-05-22T15:26:35.939 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:35.926+0000 7fd21e42a640 1 Processor -- start 2023-05-22T15:26:35.939 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:35.926+0000 7fd21e42a640 1 -- start start 2023-05-22T15:26:35.940 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:35.926+0000 7fd21e42a640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd200003f00 0x7fd218153e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:35.940 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:35.926+0000 7fd21e42a640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd2180b0250 0x7fd2180b06c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:35.940 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:35.926+0000 7fd21e42a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd2180b3b60 0x7fd2180b3fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:35.940 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:35.926+0000 7fd21e42a640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2180b0c00 con 0x7fd2180b3b60 2023-05-22T15:26:35.941 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:35.926+0000 7fd21e42a640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2181543b0 con 0x7fd2180b0250 2023-05-22T15:26:35.941 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:35.926+0000 7fd21e42a640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2180b21c0 con 0x7fd200003f00 2023-05-22T15:26:35.941 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:35.926+0000 7fd21c9c7640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd2180b3b60 0x7fd2180b3fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:35.942 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:35.926+0000 7fd21c9c7640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd2180b3b60 0x7fd2180b3fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59302/0 (socket says 172.21.15.16:59302) 2023-05-22T15:26:35.942 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:35.926+0000 7fd21c9c7640 1 -- 172.21.15.16:0/568244210 learned_addr learned my addr 172.21.15.16:0/568244210 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:26:35.942 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:35.926+0000 7fd217fff640 1 --2- 172.21.15.16:0/568244210 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd200003f00 0x7fd218153e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:35.942 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:35.926+0000 7fd2177fe640 1 --2- 172.21.15.16:0/568244210 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd2180b0250 0x7fd2180b06c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:35.943 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:35.926+0000 7fd21c9c7640 1 -- 172.21.15.16:0/568244210 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd200003f00 msgr2=0x7fd218153e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:35.943 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:35.926+0000 7fd21c9c7640 1 --2- 172.21.15.16:0/568244210 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd200003f00 0x7fd218153e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:35.943 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:35.926+0000 7fd21c9c7640 1 -- 172.21.15.16:0/568244210 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd2180b0250 msgr2=0x7fd2180b06c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:35.943 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:35.926+0000 7fd21c9c7640 1 --2- 172.21.15.16:0/568244210 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd2180b0250 0x7fd2180b06c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:35.944 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:35.926+0000 7fd21c9c7640 1 -- 172.21.15.16:0/568244210 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd21815dc90 con 0x7fd2180b3b60 2023-05-22T15:26:35.944 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:35.926+0000 7fd217fff640 1 --2- 172.21.15.16:0/568244210 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd200003f00 0x7fd218153e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-05-22T15:26:35.944 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:35.926+0000 7fd21c9c7640 1 --2- 172.21.15.16:0/568244210 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd2180b3b60 0x7fd2180b3fd0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fd21815ca90 tx=0x7fd210007d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:26:35.944 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:35.926+0000 7fd214ff9640 1 -- 172.21.15.16:0/568244210 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd21000e070 con 0x7fd2180b3b60 2023-05-22T15:26:35.945 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:35.926+0000 7fd214ff9640 1 -- 172.21.15.16:0/568244210 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd21000f040 con 0x7fd2180b3b60 2023-05-22T15:26:35.945 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:35.926+0000 7fd21e42a640 1 -- 172.21.15.16:0/568244210 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2180b4ed0 con 0x7fd2180b3b60 2023-05-22T15:26:35.945 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:35.926+0000 7fd214ff9640 1 -- 172.21.15.16:0/568244210 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd210003bc0 con 0x7fd2180b3b60 2023-05-22T15:26:35.946 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:35.926+0000 7fd21e42a640 1 -- 172.21.15.16:0/568244210 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2180b1da0 con 0x7fd2180b3b60 2023-05-22T15:26:35.946 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:35.930+0000 7fd21e42a640 1 -- 172.21.15.16:0/568244210 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd21804dbe0 con 0x7fd2180b3b60 2023-05-22T15:26:35.946 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:35.930+0000 7fd214ff9640 1 -- 172.21.15.16:0/568244210 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 4) v1 ==== 84174+0+0 (secure 0 0 0) 0x7fd210015650 con 0x7fd2180b3b60 2023-05-22T15:26:35.946 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:35.930+0000 7fd214ff9640 1 --2- 172.21.15.16:0/568244210 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd2040652f0 0x7fd2040677a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:35.946 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:35.930+0000 7fd214ff9640 1 -- 172.21.15.16:0/568244210 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(12..12 src has 1..12) v4 ==== 2231+0+0 (secure 0 0 0) 0x7fd210081170 con 0x7fd2180b3b60 2023-05-22T15:26:35.946 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:35.930+0000 7fd217fff640 1 --2- 172.21.15.16:0/568244210 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd2040652f0 0x7fd2040677a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:35.947 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:35.930+0000 7fd217fff640 1 --2- 172.21.15.16:0/568244210 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd2040652f0 0x7fd2040677a0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fd208000f70 tx=0x7fd208007040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:26:35.947 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:35.934+0000 7fd214ff9640 1 -- 172.21.15.16:0/568244210 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fd21004fdf0 con 0x7fd2180b3b60 2023-05-22T15:26:36.091 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:36.086+0000 7fd21e42a640 1 -- 172.21.15.16:0/568244210 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd21809bc70 con 0x7fd2180b3b60 2023-05-22T15:26:36.093 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:36.086+0000 7fd214ff9640 1 -- 172.21.15.16:0/568244210 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v12) v1 ==== 74+0+7057 (secure 0 0 0) 0x7fd210053410 con 0x7fd2180b3b60 2023-05-22T15:26:36.093 INFO:teuthology.misc.health.smithi016.stdout: 2023-05-22T15:26:36.093 INFO:teuthology.misc.health.smithi016.stdout:{"epoch":12,"fsid":"b2719834-7847-4d98-90a6-f65d57f85c49","created":"2023-05-22T15:26:25.259314+0000","modified":"2023-05-22T15:26:33.958512+0000","last_up_change":"0.000000","last_in_change":"2023-05-22T15:26:27.675668+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9e7ff854-b25f-4cbb-a5c2-1799d9c53d2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4702fbc9-a927-4565-aca5-41dc69cc8264","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1e7bba46-4eba-47ef-bb81-5e2d371638ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"383cacc0-5a88-4904-990e-498ca59ef48a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"970500ef-95b7-42c1-b080-eed4fae7013d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"022d67db-d4d8-445b-b1b9-8b446a00f5c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"42bd372b-6688-4a1b-8407-e16e8c266e33","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ac70b2a3-f41b-4bf2-b227-7f2cbb55426d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-05-22T15:26:36.094 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:36.086+0000 7fd21e42a640 1 -- 172.21.15.16:0/568244210 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd2040652f0 msgr2=0x7fd2040677a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:36.095 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:36.086+0000 7fd21e42a640 1 --2- 172.21.15.16:0/568244210 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd2040652f0 0x7fd2040677a0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fd208000f70 tx=0x7fd208007040 comp rx=0 tx=0).stop 2023-05-22T15:26:36.095 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:36.086+0000 7fd21e42a640 1 -- 172.21.15.16:0/568244210 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd2180b3b60 msgr2=0x7fd2180b3fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:36.095 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:36.086+0000 7fd21e42a640 1 --2- 172.21.15.16:0/568244210 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd2180b3b60 0x7fd2180b3fd0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fd21815ca90 tx=0x7fd210007d80 comp rx=0 tx=0).stop 2023-05-22T15:26:36.095 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:36.086+0000 7fd21e42a640 1 -- 172.21.15.16:0/568244210 shutdown_connections 2023-05-22T15:26:36.095 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:36.086+0000 7fd21e42a640 1 --2- 172.21.15.16:0/568244210 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd2180b3b60 0x7fd2180b3fd0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:36.095 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:36.086+0000 7fd21e42a640 1 --2- 172.21.15.16:0/568244210 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd2180b0250 0x7fd2180b06c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:36.095 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:36.086+0000 7fd21e42a640 1 --2- 172.21.15.16:0/568244210 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd2040652f0 0x7fd2040677a0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:36.096 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:36.086+0000 7fd21e42a640 1 --2- 172.21.15.16:0/568244210 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd200003f00 0x7fd218153e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:36.096 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:36.086+0000 7fd21e42a640 1 -- 172.21.15.16:0/568244210 >> 172.21.15.16:0/568244210 conn(0x7fd21809b8a0 msgr2=0x7fd2180cf110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:26:36.096 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:36.086+0000 7fd21e42a640 1 -- 172.21.15.16:0/568244210 shutdown_connections 2023-05-22T15:26:36.096 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:36.086+0000 7fd21e42a640 1 -- 172.21.15.16:0/568244210 wait complete. 2023-05-22T15:26:36.108 DEBUG:teuthology.misc:0 of 8 OSDs are up 2023-05-22T15:26:36.216 INFO:tasks.ceph.osd.4.smithi062.stderr:2023-05-22T15:26:36.209+0000 7efd9be4d640 -1 osd.4 0 waiting for initial osdmap 2023-05-22T15:26:36.246 INFO:tasks.ceph.osd.4.smithi062.stderr:2023-05-22T15:26:36.241+0000 7efd93c80640 -1 osd.4 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-05-22T15:26:36.285 INFO:tasks.ceph.osd.6.smithi062.stderr:2023-05-22T15:26:36.277+0000 7efd7a252640 -1 osd.6 0 waiting for initial osdmap 2023-05-22T15:26:36.296 INFO:tasks.ceph.osd.5.smithi062.stderr:2023-05-22T15:26:36.289+0000 7f6279906640 -1 osd.5 0 waiting for initial osdmap 2023-05-22T15:26:36.314 INFO:tasks.ceph.osd.6.smithi062.stderr:2023-05-22T15:26:36.309+0000 7efd72085640 -1 osd.6 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-05-22T15:26:36.318 INFO:tasks.ceph.osd.5.smithi062.stderr:2023-05-22T15:26:36.313+0000 7f6271739640 -1 osd.5 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-05-22T15:26:36.339 INFO:tasks.ceph.osd.7.smithi062.stderr:2023-05-22T15:26:36.333+0000 7f8a31d1d640 -1 osd.7 0 waiting for initial osdmap 2023-05-22T15:26:36.362 INFO:tasks.ceph.osd.7.smithi062.stderr:2023-05-22T15:26:36.357+0000 7f8a29b50640 -1 osd.7 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-05-22T15:26:42.915 DEBUG:teuthology.orchestra.run.smithi016:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-05-22T15:26:43.045 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:43.038+0000 7f270a961640 1 Processor -- start 2023-05-22T15:26:43.045 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:43.038+0000 7f270a961640 1 -- start start 2023-05-22T15:26:43.046 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:43.038+0000 7f270a961640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f270406b600 0x7f27040d0250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:43.046 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:43.038+0000 7f270a961640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f270406a730 con 0x7f27040b1ac0 2023-05-22T15:26:43.046 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:43.038+0000 7f270a961640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f270406ba80 con 0x7f270406b180 2023-05-22T15:26:43.047 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:43.038+0000 7f270a961640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f270415d1d0 con 0x7f270406b600 2023-05-22T15:26:43.047 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:43.038+0000 7f2703fff640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f27040b1ac0 0x7f270415da30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:36854/0 (socket says 172.21.15.16:36854) 2023-05-22T15:26:43.047 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:43.038+0000 7f2703fff640 1 -- 172.21.15.16:0/1653918322 learned_addr learned my addr 172.21.15.16:0/1653918322 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:26:43.047 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:43.038+0000 7f2708efe640 1 --2- 172.21.15.16:0/1653918322 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f270406b600 0x7f27040d0250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:43.049 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:43.038+0000 7f2702ffd640 1 -- 172.21.15.16:0/1653918322 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2265304337 0 0) 0x7f270406a730 con 0x7f27040b1ac0 2023-05-22T15:26:43.049 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:43.038+0000 7f2702ffd640 1 -- 172.21.15.16:0/1653918322 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f26ec003530 con 0x7f27040b1ac0 2023-05-22T15:26:43.049 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:43.042+0000 7f2702ffd640 1 -- 172.21.15.16:0/1653918322 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 555635497 0 0) 0x7f26ec003530 con 0x7f27040b1ac0 2023-05-22T15:26:43.050 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:43.042+0000 7f2702ffd640 1 -- 172.21.15.16:0/1653918322 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f270406b600 msgr2=0x7f27040d0250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:26:43.050 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:43.042+0000 7f2702ffd640 1 --2- 172.21.15.16:0/1653918322 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f270406b600 0x7f27040d0250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:43.050 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:43.042+0000 7f2702ffd640 1 -- 172.21.15.16:0/1653918322 >> v1:172.21.15.62:6789/0 conn(0x7f270406b180 legacy=0x7f270415c670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:43.050 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:43.042+0000 7f2702ffd640 1 -- 172.21.15.16:0/1653918322 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f270415d4b0 con 0x7f27040b1ac0 2023-05-22T15:26:43.050 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:43.042+0000 7f2702ffd640 1 -- 172.21.15.16:0/1653918322 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7f26f4001a00 con 0x7f27040b1ac0 2023-05-22T15:26:43.050 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:43.042+0000 7f2702ffd640 1 -- 172.21.15.16:0/1653918322 >> v1:172.21.15.16:6789/0 conn(0x7f27040b1ac0 legacy=0x7f270415da30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:43.051 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:43.042+0000 7f2702ffd640 1 --2- 172.21.15.16:0/1653918322 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f26ec0039b0 0x7f26ec003d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:43.051 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:43.042+0000 7f2702ffd640 1 --2- 172.21.15.16:0/1653918322 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f270406b600 0x7f26ec0047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:43.051 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:43.042+0000 7f2702ffd640 1 --2- 172.21.15.16:0/1653918322 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f26ec004d10 0x7f26ec005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:43.051 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:43.042+0000 7f2702ffd640 1 -- 172.21.15.16:0/1653918322 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f270415d1d0 con 0x7f26ec0039b0 2023-05-22T15:26:43.051 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:43.042+0000 7f2702ffd640 1 -- 172.21.15.16:0/1653918322 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26ec00ba40 con 0x7f26ec004d10 2023-05-22T15:26:43.051 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:43.042+0000 7f2702ffd640 1 -- 172.21.15.16:0/1653918322 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26ec00bbd0 con 0x7f270406b600 2023-05-22T15:26:43.051 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:43.042+0000 7f2703fff640 1 --2- 172.21.15.16:0/1653918322 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f26ec0039b0 0x7f26ec003d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:43.052 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:43.042+0000 7f2708efe640 1 --2- 172.21.15.16:0/1653918322 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f26ec004d10 0x7f26ec005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:43.052 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:43.042+0000 7f27037fe640 1 --2- 172.21.15.16:0/1653918322 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f270406b600 0x7f26ec0047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:43.052 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:43.042+0000 7f2703fff640 1 -- 172.21.15.16:0/1653918322 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f270406b600 msgr2=0x7f26ec0047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:26:43.052 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:43.042+0000 7f2703fff640 1 --2- 172.21.15.16:0/1653918322 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f270406b600 0x7f26ec0047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:43.052 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:43.042+0000 7f2703fff640 1 -- 172.21.15.16:0/1653918322 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f26ec004d10 msgr2=0x7f26ec005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:26:43.053 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:43.042+0000 7f2703fff640 1 --2- 172.21.15.16:0/1653918322 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f26ec004d10 0x7f26ec005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:43.053 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:43.042+0000 7f2703fff640 1 -- 172.21.15.16:0/1653918322 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26ec003530 con 0x7f26ec0039b0 2023-05-22T15:26:43.059 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:43.042+0000 7f2703fff640 1 --2- 172.21.15.16:0/1653918322 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f26ec0039b0 0x7f26ec003d80 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f26f400a7d0 tx=0x7f26f401d2e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8321e1053df2ec7f server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:26:43.059 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:43.042+0000 7f2702ffd640 1 -- 172.21.15.16:0/1653918322 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26f40ad630 con 0x7f26ec0039b0 2023-05-22T15:26:43.059 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:43.042+0000 7f2702ffd640 1 -- 172.21.15.16:0/1653918322 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26f40a8070 con 0x7f26ec0039b0 2023-05-22T15:26:43.059 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:43.042+0000 7f2702ffd640 1 -- 172.21.15.16:0/1653918322 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26f40ad9f0 con 0x7f26ec0039b0 2023-05-22T15:26:43.060 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:43.042+0000 7f270a961640 1 -- 172.21.15.16:0/1653918322 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f26ec0039b0 msgr2=0x7f26ec003d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:43.060 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:43.042+0000 7f270a961640 1 --2- 172.21.15.16:0/1653918322 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f26ec0039b0 0x7f26ec003d80 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f26f400a7d0 tx=0x7f26f401d2e0 comp rx=0 tx=0).stop 2023-05-22T15:26:43.060 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:43.042+0000 7f2708efe640 1 -- 172.21.15.16:0/1653918322 reap_dead start 2023-05-22T15:26:43.060 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:43.042+0000 7f270a961640 1 -- 172.21.15.16:0/1653918322 shutdown_connections 2023-05-22T15:26:43.060 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:43.042+0000 7f270a961640 1 -- 172.21.15.16:0/1653918322 >> 172.21.15.16:0/1653918322 conn(0x7f2704071760 msgr2=0x7f270409b8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:26:43.061 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:43.042+0000 7f270a961640 1 -- 172.21.15.16:0/1653918322 shutdown_connections 2023-05-22T15:26:43.061 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:43.042+0000 7f270a961640 1 -- 172.21.15.16:0/1653918322 wait complete. 2023-05-22T15:26:43.061 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:43.046+0000 7f270a961640 1 Processor -- start 2023-05-22T15:26:43.061 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:43.046+0000 7f270a961640 1 -- start start 2023-05-22T15:26:43.061 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:43.046+0000 7f270a961640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f26ec0039b0 0x7f2704150790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:43.062 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:43.046+0000 7f270a961640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2704155420 0x7f2704150db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:43.062 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:43.046+0000 7f270a961640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2704152e50 0x7f27041532c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:43.062 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:43.046+0000 7f270a961640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2704152180 con 0x7f2704152e50 2023-05-22T15:26:43.062 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:43.046+0000 7f270a961640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27040c6930 con 0x7f2704155420 2023-05-22T15:26:43.063 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:43.046+0000 7f270a961640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2704153800 con 0x7f26ec0039b0 2023-05-22T15:26:43.063 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:43.046+0000 7f2708efe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2704152e50 0x7f27041532c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:43.063 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:43.046+0000 7f2708efe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2704152e50 0x7f27041532c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49644/0 (socket says 172.21.15.16:49644) 2023-05-22T15:26:43.063 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:43.046+0000 7f2708efe640 1 -- 172.21.15.16:0/145745579 learned_addr learned my addr 172.21.15.16:0/145745579 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:26:43.063 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:43.046+0000 7f27037fe640 1 --2- 172.21.15.16:0/145745579 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2704155420 0x7f2704150db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:43.063 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:43.046+0000 7f2703fff640 1 --2- 172.21.15.16:0/145745579 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f26ec0039b0 0x7f2704150790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:43.063 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:43.046+0000 7f2708efe640 1 -- 172.21.15.16:0/145745579 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f26ec0039b0 msgr2=0x7f2704150790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:43.064 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:43.046+0000 7f2708efe640 1 --2- 172.21.15.16:0/145745579 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f26ec0039b0 0x7f2704150790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:43.064 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:43.046+0000 7f2708efe640 1 -- 172.21.15.16:0/145745579 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2704155420 msgr2=0x7f2704150db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:43.064 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:43.046+0000 7f2708efe640 1 --2- 172.21.15.16:0/145745579 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2704155420 0x7f2704150db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:43.064 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:43.046+0000 7f2708efe640 1 -- 172.21.15.16:0/145745579 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f270415e140 con 0x7f2704152e50 2023-05-22T15:26:43.064 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:43.046+0000 7f2708efe640 1 --2- 172.21.15.16:0/145745579 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2704152e50 0x7f27041532c0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f27040d32c0 tx=0x7f26fc011550 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:26:43.064 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:43.046+0000 7f2700ff9640 1 -- 172.21.15.16:0/145745579 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26fc004500 con 0x7f2704152e50 2023-05-22T15:26:43.064 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:43.046+0000 7f2700ff9640 1 -- 172.21.15.16:0/145745579 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26fc011870 con 0x7f2704152e50 2023-05-22T15:26:43.065 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:43.046+0000 7f2700ff9640 1 -- 172.21.15.16:0/145745579 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26fc011c20 con 0x7f2704152e50 2023-05-22T15:26:43.065 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:43.046+0000 7f270a961640 1 -- 172.21.15.16:0/145745579 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27040d2570 con 0x7f2704152e50 2023-05-22T15:26:43.065 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:43.046+0000 7f270a961640 1 -- 172.21.15.16:0/145745579 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f270415cd80 con 0x7f2704152e50 2023-05-22T15:26:43.065 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:43.046+0000 7f270a961640 1 -- 172.21.15.16:0/145745579 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f270404dbe0 con 0x7f2704152e50 2023-05-22T15:26:43.065 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:43.050+0000 7f2700ff9640 1 -- 172.21.15.16:0/145745579 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7f26fc011d80 con 0x7f2704152e50 2023-05-22T15:26:43.065 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:43.050+0000 7f2700ff9640 1 --2- 172.21.15.16:0/145745579 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f26f0065380 0x7f26f0067830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:43.065 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:43.050+0000 7f2700ff9640 1 -- 172.21.15.16:0/145745579 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f26fc082060 con 0x7f2704152e50 2023-05-22T15:26:43.066 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:43.050+0000 7f2703fff640 1 --2- 172.21.15.16:0/145745579 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f26f0065380 0x7f26f0067830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:43.066 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:43.050+0000 7f2703fff640 1 --2- 172.21.15.16:0/145745579 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f26f0065380 0x7f26f0067830 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f26f40ad000 tx=0x7f26f40a7000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:26:43.066 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:43.054+0000 7f2700ff9640 1 -- 172.21.15.16:0/145745579 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f26fc0500f0 con 0x7f2704152e50 2023-05-22T15:26:43.214 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:43.206+0000 7f270a961640 1 -- 172.21.15.16:0/145745579 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f270408bd50 con 0x7f2704152e50 2023-05-22T15:26:43.217 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:43.210+0000 7f2700ff9640 1 -- 172.21.15.16:0/145745579 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) v1 ==== 74+0+12334 (secure 0 0 0) 0x7f26fc053710 con 0x7f2704152e50 2023-05-22T15:26:43.217 INFO:teuthology.misc.health.smithi016.stdout: 2023-05-22T15:26:43.218 INFO:teuthology.misc.health.smithi016.stdout:{"epoch":16,"fsid":"b2719834-7847-4d98-90a6-f65d57f85c49","created":"2023-05-22T15:26:25.259314+0000","modified":"2023-05-22T15:26:40.231951+0000","last_up_change":"2023-05-22T15:26:36.448524+0000","last_in_change":"2023-05-22T15:26:27.675668+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-05-22T15:26:38.019781+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9e7ff854-b25f-4cbb-a5c2-1799d9c53d2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":37021},{"type":"v1","addr":"172.21.15.16:6825","nonce":37021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":37021},{"type":"v1","addr":"172.21.15.16:6827","nonce":37021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":37021},{"type":"v1","addr":"172.21.15.16:6831","nonce":37021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":37021},{"type":"v1","addr":"172.21.15.16:6829","nonce":37021}]},"public_addr":"172.21.15.16:6825/37021","cluster_addr":"172.21.15.16:6827/37021","heartbeat_back_addr":"172.21.15.16:6831/37021","heartbeat_front_addr":"172.21.15.16:6829/37021","state":["exists","up"]},{"osd":1,"uuid":"4702fbc9-a927-4565-aca5-41dc69cc8264","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":37008},{"type":"v1","addr":"172.21.15.16:6801","nonce":37008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":37008},{"type":"v1","addr":"172.21.15.16:6803","nonce":37008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":37008},{"type":"v1","addr":"172.21.15.16:6807","nonce":37008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":37008},{"type":"v1","addr":"172.21.15.16:6805","nonce":37008}]},"public_addr":"172.21.15.16:6801/37008","cluster_addr":"172.21.15.16:6803/37008","heartbeat_back_addr":"172.21.15.16:6807/37008","heartbeat_front_addr":"172.21.15.16:6805/37008","state":["exists","up"]},{"osd":2,"uuid":"1e7bba46-4eba-47ef-bb81-5e2d371638ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":37009},{"type":"v1","addr":"172.21.15.16:6809","nonce":37009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":37009},{"type":"v1","addr":"172.21.15.16:6811","nonce":37009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":37009},{"type":"v1","addr":"172.21.15.16:6815","nonce":37009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":37009},{"type":"v1","addr":"172.21.15.16:6813","nonce":37009}]},"public_addr":"172.21.15.16:6809/37009","cluster_addr":"172.21.15.16:6811/37009","heartbeat_back_addr":"172.21.15.16:6815/37009","heartbeat_front_addr":"172.21.15.16:6813/37009","state":["exists","up"]},{"osd":3,"uuid":"383cacc0-5a88-4904-990e-498ca59ef48a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":37020},{"type":"v1","addr":"172.21.15.16:6817","nonce":37020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":37020},{"type":"v1","addr":"172.21.15.16:6819","nonce":37020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":37020},{"type":"v1","addr":"172.21.15.16:6823","nonce":37020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":37020},{"type":"v1","addr":"172.21.15.16:6821","nonce":37020}]},"public_addr":"172.21.15.16:6817/37020","cluster_addr":"172.21.15.16:6819/37020","heartbeat_back_addr":"172.21.15.16:6823/37020","heartbeat_front_addr":"172.21.15.16:6821/37020","state":["exists","up"]},{"osd":4,"uuid":"970500ef-95b7-42c1-b080-eed4fae7013d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":37353},{"type":"v1","addr":"172.21.15.62:6817","nonce":37353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6819","nonce":37353},{"type":"v1","addr":"172.21.15.62:6822","nonce":37353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37353},{"type":"v1","addr":"172.21.15.62:6831","nonce":37353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37353},{"type":"v1","addr":"172.21.15.62:6826","nonce":37353}]},"public_addr":"172.21.15.62:6817/37353","cluster_addr":"172.21.15.62:6822/37353","heartbeat_back_addr":"172.21.15.62:6831/37353","heartbeat_front_addr":"172.21.15.62:6826/37353","state":["exists","up"]},{"osd":5,"uuid":"022d67db-d4d8-445b-b1b9-8b446a00f5c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":37352},{"type":"v1","addr":"172.21.15.62:6801","nonce":37352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":37352},{"type":"v1","addr":"172.21.15.62:6803","nonce":37352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":37352},{"type":"v1","addr":"172.21.15.62:6807","nonce":37352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":37352},{"type":"v1","addr":"172.21.15.62:6805","nonce":37352}]},"public_addr":"172.21.15.62:6801/37352","cluster_addr":"172.21.15.62:6803/37352","heartbeat_back_addr":"172.21.15.62:6807/37352","heartbeat_front_addr":"172.21.15.62:6805/37352","state":["exists","up"]},{"osd":6,"uuid":"42bd372b-6688-4a1b-8407-e16e8c266e33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":37354},{"type":"v1","addr":"172.21.15.62:6820","nonce":37354}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":37354},{"type":"v1","addr":"172.21.15.62:6823","nonce":37354}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6829","nonce":37354},{"type":"v1","addr":"172.21.15.62:6830","nonce":37354}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6825","nonce":37354},{"type":"v1","addr":"172.21.15.62:6827","nonce":37354}]},"public_addr":"172.21.15.62:6820/37354","cluster_addr":"172.21.15.62:6823/37354","heartbeat_back_addr":"172.21.15.62:6830/37354","heartbeat_front_addr":"172.21.15.62:6827/37354","state":["exists","up"]},{"osd":7,"uuid":"ac70b2a3-f41b-4bf2-b227-7f2cbb55426d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37355},{"type":"v1","addr":"172.21.15.62:6809","nonce":37355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37355},{"type":"v1","addr":"172.21.15.62:6811","nonce":37355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":37355},{"type":"v1","addr":"172.21.15.62:6815","nonce":37355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":37355},{"type":"v1","addr":"172.21.15.62:6813","nonce":37355}]},"public_addr":"172.21.15.62:6809/37355","cluster_addr":"172.21.15.62:6811/37355","heartbeat_back_addr":"172.21.15.62:6815/37355","heartbeat_front_addr":"172.21.15.62:6813/37355","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.368896+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.107273+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.276888+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.287502+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.124791+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.251937+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.255679+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.232797+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-05-22T15:26:43.220 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:43.210+0000 7f270a961640 1 -- 172.21.15.16:0/145745579 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f26f0065380 msgr2=0x7f26f0067830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:43.220 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:43.210+0000 7f270a961640 1 --2- 172.21.15.16:0/145745579 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f26f0065380 0x7f26f0067830 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f26f40ad000 tx=0x7f26f40a7000 comp rx=0 tx=0).stop 2023-05-22T15:26:43.221 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:43.210+0000 7f270a961640 1 -- 172.21.15.16:0/145745579 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2704152e50 msgr2=0x7f27041532c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:43.221 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:43.210+0000 7f270a961640 1 --2- 172.21.15.16:0/145745579 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2704152e50 0x7f27041532c0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f27040d32c0 tx=0x7f26fc011550 comp rx=0 tx=0).stop 2023-05-22T15:26:43.222 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:43.210+0000 7f270a961640 1 -- 172.21.15.16:0/145745579 shutdown_connections 2023-05-22T15:26:43.222 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:43.210+0000 7f270a961640 1 --2- 172.21.15.16:0/145745579 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2704152e50 0x7f27041532c0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:43.222 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:43.210+0000 7f270a961640 1 --2- 172.21.15.16:0/145745579 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2704155420 0x7f2704150db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:43.223 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:43.210+0000 7f270a961640 1 --2- 172.21.15.16:0/145745579 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f26f0065380 0x7f26f0067830 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:43.223 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:43.210+0000 7f270a961640 1 --2- 172.21.15.16:0/145745579 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f26ec0039b0 0x7f2704150790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:43.224 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:43.210+0000 7f270a961640 1 -- 172.21.15.16:0/145745579 >> 172.21.15.16:0/145745579 conn(0x7f2704071760 msgr2=0x7f270406ba60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:26:43.224 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:43.210+0000 7f270a961640 1 -- 172.21.15.16:0/145745579 shutdown_connections 2023-05-22T15:26:43.225 INFO:teuthology.misc.health.smithi016.stderr:2023-05-22T15:26:43.210+0000 7f270a961640 1 -- 172.21.15.16:0/145745579 wait complete. 2023-05-22T15:26:43.234 DEBUG:teuthology.misc:8 of 8 OSDs are up 2023-05-22T15:26:43.235 INFO:tasks.ceph:Creating RBD pool 2023-05-22T15:26:43.235 DEBUG:teuthology.orchestra.run.smithi016:> sudo ceph --cluster ceph osd pool create rbd 8 2023-05-22T15:26:43.365 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:43.358+0000 7f7e62d28640 1 Processor -- start 2023-05-22T15:26:43.365 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:43.358+0000 7f7e62d28640 1 -- start start 2023-05-22T15:26:43.365 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:43.358+0000 7f7e62d28640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f7e5c0cae20 0x7f7e5c06fef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:43.365 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:43.358+0000 7f7e62d28640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7e5c070430 0x7f7e5c0ad410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:43.366 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:43.358+0000 7f7e62d28640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e5c069720 con 0x7f7e5c153fe0 2023-05-22T15:26:43.366 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:43.358+0000 7f7e62d28640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e5c0708d0 con 0x7f7e5c0cae20 2023-05-22T15:26:43.366 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:43.358+0000 7f7e62d28640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e5c06db00 con 0x7f7e5c070430 2023-05-22T15:26:43.366 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:43.358+0000 7f7e5bfff640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f7e5c0cae20 0x7f7e5c06fef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:43.366 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:43.358+0000 7f7e60ac4640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f7e5c153fe0 0x7f7e5c06d3f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:36856/0 (socket says 172.21.15.16:36856) 2023-05-22T15:26:43.366 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:43.358+0000 7f7e5bfff640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f7e5c0cae20 0x7f7e5c06fef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49650/0 (socket says 172.21.15.16:49650) 2023-05-22T15:26:43.367 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:43.358+0000 7f7e60ac4640 1 -- 172.21.15.16:0/1513564300 learned_addr learned my addr 172.21.15.16:0/1513564300 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:26:43.367 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:43.358+0000 7f7e612c5640 1 --2- 172.21.15.16:0/1513564300 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7e5c070430 0x7f7e5c0ad410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:43.368 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:43.358+0000 7f7e5bfff640 1 -- 172.21.15.16:0/1513564300 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7e5c070430 msgr2=0x7f7e5c0ad410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:26:43.368 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:43.358+0000 7f7e5bfff640 1 --2- 172.21.15.16:0/1513564300 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7e5c070430 0x7f7e5c0ad410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:43.368 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:43.358+0000 7f7e5bfff640 1 -- 172.21.15.16:0/1513564300 >> v1:172.21.15.16:6789/0 conn(0x7f7e5c153fe0 legacy=0x7f7e5c06d3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:43.368 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:43.358+0000 7f7e5b7fe640 1 -- 172.21.15.16:0/1513564300 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 189625145 0 0) 0x7f7e5c069720 con 0x7f7e5c153fe0 2023-05-22T15:26:43.368 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:43.358+0000 7f7e5bfff640 1 -- 172.21.15.16:0/1513564300 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e5c1543b0 con 0x7f7e5c0cae20 2023-05-22T15:26:43.369 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:43.362+0000 7f7e5bfff640 1 --2- 172.21.15.16:0/1513564300 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e5c0cae20 0x7f7e5c06fef0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f7e540097f0 tx=0x7f7e5401c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d939df3167fa2e1 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:26:43.369 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:43.362+0000 7f7e5b7fe640 1 -- 172.21.15.16:0/1513564300 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e540ab650 con 0x7f7e5c0cae20 2023-05-22T15:26:43.369 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:43.362+0000 7f7e5b7fe640 1 -- 172.21.15.16:0/1513564300 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e540ab7b0 con 0x7f7e5c0cae20 2023-05-22T15:26:43.369 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:43.362+0000 7f7e5b7fe640 1 -- 172.21.15.16:0/1513564300 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e540abb40 con 0x7f7e5c0cae20 2023-05-22T15:26:43.371 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:43.362+0000 7f7e62d28640 1 -- 172.21.15.16:0/1513564300 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e5c0cae20 msgr2=0x7f7e5c06fef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:43.371 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:43.362+0000 7f7e62d28640 1 --2- 172.21.15.16:0/1513564300 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e5c0cae20 0x7f7e5c06fef0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f7e540097f0 tx=0x7f7e5401c450 comp rx=0 tx=0).stop 2023-05-22T15:26:43.371 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:43.362+0000 7f7e62d28640 1 -- 172.21.15.16:0/1513564300 shutdown_connections 2023-05-22T15:26:43.371 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:43.362+0000 7f7e62d28640 1 --2- 172.21.15.16:0/1513564300 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7e5c070430 0x7f7e5c0ad410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:43.372 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:43.362+0000 7f7e62d28640 1 --2- 172.21.15.16:0/1513564300 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e5c0cae20 0x7f7e5c06fef0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:43.372 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:43.362+0000 7f7e62d28640 1 -- 172.21.15.16:0/1513564300 >> 172.21.15.16:0/1513564300 conn(0x7f7e5c0b1110 msgr2=0x7f7e5c0ce830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:26:43.372 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:43.362+0000 7f7e62d28640 1 -- 172.21.15.16:0/1513564300 shutdown_connections 2023-05-22T15:26:43.372 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:43.362+0000 7f7e62d28640 1 -- 172.21.15.16:0/1513564300 wait complete. 2023-05-22T15:26:43.372 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:43.362+0000 7f7e62d28640 1 Processor -- start 2023-05-22T15:26:43.372 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:43.362+0000 7f7e62d28640 1 -- start start 2023-05-22T15:26:43.372 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:43.362+0000 7f7e62d28640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7e5c070430 0x7f7e5c158900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:43.373 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:43.362+0000 7f7e62d28640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e5c0cae20 0x7f7e5c152c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:43.373 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:43.362+0000 7f7e62d28640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7e5c153fe0 0x7f7e5c0b1a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:43.373 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:43.362+0000 7f7e62d28640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e5c151bd0 con 0x7f7e5c0cae20 2023-05-22T15:26:43.373 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:43.362+0000 7f7e62d28640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e5c151eb0 con 0x7f7e5c153fe0 2023-05-22T15:26:43.373 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:43.362+0000 7f7e62d28640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e5c152190 con 0x7f7e5c070430 2023-05-22T15:26:43.373 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:43.362+0000 7f7e5bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e5c0cae20 0x7f7e5c152c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:43.374 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:43.366+0000 7f7e5bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e5c0cae20 0x7f7e5c152c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49658/0 (socket says 172.21.15.16:49658) 2023-05-22T15:26:43.374 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:43.366+0000 7f7e5bfff640 1 -- 172.21.15.16:0/4264742373 learned_addr learned my addr 172.21.15.16:0/4264742373 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:26:43.375 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:43.366+0000 7f7e60ac4640 1 --2- 172.21.15.16:0/4264742373 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7e5c070430 0x7f7e5c158900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:43.375 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:43.366+0000 7f7e612c5640 1 --2- 172.21.15.16:0/4264742373 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7e5c153fe0 0x7f7e5c0b1a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:43.375 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:43.366+0000 7f7e5bfff640 1 -- 172.21.15.16:0/4264742373 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7e5c070430 msgr2=0x7f7e5c158900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:43.375 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:43.366+0000 7f7e5bfff640 1 --2- 172.21.15.16:0/4264742373 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7e5c070430 0x7f7e5c158900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:43.375 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:43.366+0000 7f7e5bfff640 1 -- 172.21.15.16:0/4264742373 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7e5c153fe0 msgr2=0x7f7e5c0b1a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:43.376 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:43.366+0000 7f7e5bfff640 1 --2- 172.21.15.16:0/4264742373 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7e5c153fe0 0x7f7e5c0b1a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:43.376 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:43.366+0000 7f7e5bfff640 1 -- 172.21.15.16:0/4264742373 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e54009420 con 0x7f7e5c0cae20 2023-05-22T15:26:43.377 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:43.366+0000 7f7e5bfff640 1 --2- 172.21.15.16:0/4264742373 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e5c0cae20 0x7f7e5c152c70 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f7e540ad040 tx=0x7f7e5401ac90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:26:43.377 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:43.366+0000 7f7e597fa640 1 -- 172.21.15.16:0/4264742373 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e54004220 con 0x7f7e5c0cae20 2023-05-22T15:26:43.377 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:43.366+0000 7f7e597fa640 1 -- 172.21.15.16:0/4264742373 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e540b9070 con 0x7f7e5c0cae20 2023-05-22T15:26:43.378 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:43.366+0000 7f7e62d28640 1 -- 172.21.15.16:0/4264742373 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e5c15db60 con 0x7f7e5c0cae20 2023-05-22T15:26:43.378 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:43.366+0000 7f7e597fa640 1 -- 172.21.15.16:0/4264742373 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e54003620 con 0x7f7e5c0cae20 2023-05-22T15:26:43.378 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:43.366+0000 7f7e62d28640 1 -- 172.21.15.16:0/4264742373 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e5c15d880 con 0x7f7e5c0cae20 2023-05-22T15:26:43.378 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:43.366+0000 7f7e62d28640 1 -- 172.21.15.16:0/4264742373 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e5c15de40 con 0x7f7e5c0cae20 2023-05-22T15:26:43.378 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:43.366+0000 7f7e597fa640 1 -- 172.21.15.16:0/4264742373 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7f7e54003780 con 0x7f7e5c0cae20 2023-05-22T15:26:43.378 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:43.366+0000 7f7e597fa640 1 --2- 172.21.15.16:0/4264742373 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f7e3c065380 0x7f7e3c067830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:43.378 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:43.366+0000 7f7e597fa640 1 -- 172.21.15.16:0/4264742373 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f7e540bb070 con 0x7f7e5c0cae20 2023-05-22T15:26:43.379 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:43.366+0000 7f7e60ac4640 1 --2- 172.21.15.16:0/4264742373 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f7e3c065380 0x7f7e3c067830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:43.379 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:43.366+0000 7f7e60ac4640 1 --2- 172.21.15.16:0/4264742373 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f7e3c065380 0x7f7e3c067830 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f7e440012f0 tx=0x7f7e44005ec0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:26:43.379 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:43.370+0000 7f7e597fa640 1 -- 172.21.15.16:0/4264742373 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f7e540eb530 con 0x7f7e5c0cae20 2023-05-22T15:26:43.534 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:43.526+0000 7f7e62d28640 1 -- 172.21.15.16:0/4264742373 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f7e5c04dbe0 con 0x7f7e5c0cae20 2023-05-22T15:26:43.966 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:43.958+0000 7f7e597fa640 1 -- 172.21.15.16:0/4264742373 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]=0 pool 'rbd' created v17) v1 ==== 109+0+0 (secure 0 0 0) 0x7f7e540c4020 con 0x7f7e5c0cae20 2023-05-22T15:26:43.967 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'rbd' created 2023-05-22T15:26:43.970 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:43.962+0000 7f7e62d28640 1 -- 172.21.15.16:0/4264742373 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f7e3c065380 msgr2=0x7f7e3c067830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:43.970 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:43.962+0000 7f7e62d28640 1 --2- 172.21.15.16:0/4264742373 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f7e3c065380 0x7f7e3c067830 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f7e440012f0 tx=0x7f7e44005ec0 comp rx=0 tx=0).stop 2023-05-22T15:26:43.970 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:43.962+0000 7f7e62d28640 1 -- 172.21.15.16:0/4264742373 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e5c0cae20 msgr2=0x7f7e5c152c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:43.970 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:43.962+0000 7f7e62d28640 1 --2- 172.21.15.16:0/4264742373 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e5c0cae20 0x7f7e5c152c70 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f7e540ad040 tx=0x7f7e5401ac90 comp rx=0 tx=0).stop 2023-05-22T15:26:43.971 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:43.962+0000 7f7e62d28640 1 -- 172.21.15.16:0/4264742373 shutdown_connections 2023-05-22T15:26:43.971 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:43.962+0000 7f7e62d28640 1 --2- 172.21.15.16:0/4264742373 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7e5c153fe0 0x7f7e5c0b1a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:43.971 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:43.962+0000 7f7e62d28640 1 --2- 172.21.15.16:0/4264742373 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e5c0cae20 0x7f7e5c152c70 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:43.971 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:43.962+0000 7f7e62d28640 1 --2- 172.21.15.16:0/4264742373 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f7e3c065380 0x7f7e3c067830 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:43.971 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:43.962+0000 7f7e62d28640 1 --2- 172.21.15.16:0/4264742373 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7e5c070430 0x7f7e5c158900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:43.972 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:43.962+0000 7f7e62d28640 1 -- 172.21.15.16:0/4264742373 >> 172.21.15.16:0/4264742373 conn(0x7f7e5c0b1110 msgr2=0x7f7e5c06a610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:26:43.972 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:43.966+0000 7f7e62d28640 1 -- 172.21.15.16:0/4264742373 shutdown_connections 2023-05-22T15:26:43.972 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:43.966+0000 7f7e62d28640 1 -- 172.21.15.16:0/4264742373 wait complete. 2023-05-22T15:26:43.991 DEBUG:teuthology.orchestra.run.smithi016:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2023-05-22T15:26:44.118 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:44.110+0000 7f93a51fe640 1 Processor -- start 2023-05-22T15:26:44.118 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:44.110+0000 7f93a51fe640 1 -- start start 2023-05-22T15:26:44.120 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:44.114+0000 7f93a51fe640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f93a00ce900 0x7f93a015c540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:44.120 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:44.114+0000 7f93a51fe640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f93a00cbe80 0x7f93a00cc680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:44.120 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:44.114+0000 7f93a51fe640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93a00c3930 con 0x7f93a00ce530 2023-05-22T15:26:44.120 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:44.114+0000 7f93a51fe640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93a0158f00 con 0x7f93a00ce900 2023-05-22T15:26:44.120 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:44.114+0000 7f93a51fe640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93a0159d90 con 0x7f93a00cbe80 2023-05-22T15:26:44.120 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:44.114+0000 7f939ed9d640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f93a00ce530 0x7f93a0159680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:36860/0 (socket says 172.21.15.16:36860) 2023-05-22T15:26:44.121 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:44.114+0000 7f939ed9d640 1 -- 172.21.15.16:0/1995557758 learned_addr learned my addr 172.21.15.16:0/1995557758 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:26:44.121 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:44.114+0000 7f939f59e640 1 --2- 172.21.15.16:0/1995557758 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f93a00cbe80 0x7f93a00cc680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:44.121 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:44.114+0000 7f939e59c640 1 --2- 172.21.15.16:0/1995557758 >> v2:172.21.15.62:3300/0 conn(0x7f93a00ce900 0x7f93a015c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:44.121 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:44.114+0000 7f939dd9b640 1 -- 172.21.15.16:0/1995557758 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 97839261 0 0) 0x7f9390001120 con 0x7f93a00ce530 2023-05-22T15:26:44.121 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:44.114+0000 7f939dd9b640 1 -- 172.21.15.16:0/1995557758 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f938c003530 con 0x7f93a00ce530 2023-05-22T15:26:44.122 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:44.114+0000 7f939dd9b640 1 -- 172.21.15.16:0/1995557758 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1955377442 0 0) 0x7f938c003530 con 0x7f93a00ce530 2023-05-22T15:26:44.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:44.114+0000 7f939dd9b640 1 -- 172.21.15.16:0/1995557758 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f93a00cbe80 msgr2=0x7f93a00cc680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:26:44.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:44.114+0000 7f939dd9b640 1 --2- 172.21.15.16:0/1995557758 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f93a00cbe80 0x7f93a00cc680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:44.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:44.114+0000 7f939dd9b640 1 -- 172.21.15.16:0/1995557758 >> v2:172.21.15.62:3300/0 conn(0x7f93a00ce900 msgr2=0x7f93a015c540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:26:44.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:44.114+0000 7f939dd9b640 1 --2- 172.21.15.16:0/1995557758 >> v2:172.21.15.62:3300/0 conn(0x7f93a00ce900 0x7f93a015c540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:44.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:44.114+0000 7f939dd9b640 1 -- 172.21.15.16:0/1995557758 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93a015cc40 con 0x7f93a00ce530 2023-05-22T15:26:44.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:44.114+0000 7f939dd9b640 1 -- 172.21.15.16:0/1995557758 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7f9390001810 con 0x7f93a00ce530 2023-05-22T15:26:44.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:44.114+0000 7f939dd9b640 1 -- 172.21.15.16:0/1995557758 >> v1:172.21.15.16:6789/0 conn(0x7f93a00ce530 legacy=0x7f93a0159680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:44.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:44.114+0000 7f939dd9b640 1 --2- 172.21.15.16:0/1995557758 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f93a00cbe80 0x7f938c003980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:44.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:44.114+0000 7f939dd9b640 1 --2- 172.21.15.16:0/1995557758 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f938c003f00 0x7f938c0047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:44.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:44.114+0000 7f939dd9b640 1 --2- 172.21.15.16:0/1995557758 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f938c004d10 0x7f938c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:44.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:44.114+0000 7f939dd9b640 1 -- 172.21.15.16:0/1995557758 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93a0158f00 con 0x7f938c003f00 2023-05-22T15:26:44.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:44.114+0000 7f939dd9b640 1 -- 172.21.15.16:0/1995557758 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93a0159d90 con 0x7f938c004d10 2023-05-22T15:26:44.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:44.114+0000 7f939dd9b640 1 -- 172.21.15.16:0/1995557758 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f938c00b990 con 0x7f93a00cbe80 2023-05-22T15:26:44.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:44.114+0000 7f939e59c640 1 --2- 172.21.15.16:0/1995557758 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f938c003f00 0x7f938c0047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:44.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:44.114+0000 7f939ed9d640 1 --2- 172.21.15.16:0/1995557758 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f93a00cbe80 0x7f938c003980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:44.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:44.114+0000 7f939f59e640 1 --2- 172.21.15.16:0/1995557758 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f938c004d10 0x7f938c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:44.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:44.114+0000 7f939e59c640 1 -- 172.21.15.16:0/1995557758 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f93a00cbe80 msgr2=0x7f938c003980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:26:44.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:44.114+0000 7f939e59c640 1 --2- 172.21.15.16:0/1995557758 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f93a00cbe80 0x7f938c003980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:44.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:44.114+0000 7f939e59c640 1 -- 172.21.15.16:0/1995557758 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f938c004d10 msgr2=0x7f938c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:26:44.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:44.114+0000 7f939e59c640 1 --2- 172.21.15.16:0/1995557758 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f938c004d10 0x7f938c005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:44.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:44.114+0000 7f939e59c640 1 -- 172.21.15.16:0/1995557758 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f938c003530 con 0x7f938c003f00 2023-05-22T15:26:44.127 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:44.118+0000 7f939e59c640 1 --2- 172.21.15.16:0/1995557758 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f938c003f00 0x7f938c0047d0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f9388009ae0 tx=0x7f938801ca90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d765835f35b7e42 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:26:44.127 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:44.118+0000 7f939dd9b640 1 -- 172.21.15.16:0/1995557758 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93880a5680 con 0x7f938c003f00 2023-05-22T15:26:44.127 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:44.118+0000 7f939dd9b640 1 -- 172.21.15.16:0/1995557758 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93880a57e0 con 0x7f938c003f00 2023-05-22T15:26:44.127 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:44.118+0000 7f939dd9b640 1 -- 172.21.15.16:0/1995557758 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93880a5b90 con 0x7f938c003f00 2023-05-22T15:26:44.128 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:44.118+0000 7f93a51fe640 1 -- 172.21.15.16:0/1995557758 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f938c003f00 msgr2=0x7f938c0047d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:44.128 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:44.118+0000 7f93a51fe640 1 --2- 172.21.15.16:0/1995557758 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f938c003f00 0x7f938c0047d0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f9388009ae0 tx=0x7f938801ca90 comp rx=0 tx=0).stop 2023-05-22T15:26:44.128 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:44.118+0000 7f939f59e640 1 -- 172.21.15.16:0/1995557758 reap_dead start 2023-05-22T15:26:44.128 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:44.118+0000 7f93a51fe640 1 -- 172.21.15.16:0/1995557758 shutdown_connections 2023-05-22T15:26:44.128 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:44.118+0000 7f93a51fe640 1 -- 172.21.15.16:0/1995557758 >> 172.21.15.16:0/1995557758 conn(0x7f93a0069720 msgr2=0x7f93a0069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:26:44.128 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:44.118+0000 7f93a51fe640 1 -- 172.21.15.16:0/1995557758 shutdown_connections 2023-05-22T15:26:44.128 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:44.118+0000 7f93a51fe640 1 -- 172.21.15.16:0/1995557758 wait complete. 2023-05-22T15:26:44.129 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:44.118+0000 7f93a51fe640 1 Processor -- start 2023-05-22T15:26:44.129 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:44.118+0000 7f93a51fe640 1 -- start start 2023-05-22T15:26:44.129 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:44.122+0000 7f93a51fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f938c003f00 0x7f93a0148f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:44.129 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:44.122+0000 7f93a51fe640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f93a00c5b30 0x7f93a00c5fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:44.129 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:44.122+0000 7f93a51fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f93a00ae070 0x7f93a00ae4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:44.129 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:44.122+0000 7f93a51fe640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93a01494c0 con 0x7f938c003f00 2023-05-22T15:26:44.129 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:44.122+0000 7f93a51fe640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93a00c6600 con 0x7f93a00ae070 2023-05-22T15:26:44.129 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:44.122+0000 7f93a51fe640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93a00c68e0 con 0x7f93a00c5b30 2023-05-22T15:26:44.130 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:44.122+0000 7f939ed9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f938c003f00 0x7f93a0148f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:44.130 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:44.122+0000 7f939ed9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f938c003f00 0x7f93a0148f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49676/0 (socket says 172.21.15.16:49676) 2023-05-22T15:26:44.130 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:44.122+0000 7f939ed9d640 1 -- 172.21.15.16:0/3013243597 learned_addr learned my addr 172.21.15.16:0/3013243597 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:26:44.131 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:44.122+0000 7f939e59c640 1 --2- 172.21.15.16:0/3013243597 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f93a00c5b30 0x7f93a00c5fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:44.131 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:44.122+0000 7f939f59e640 1 --2- 172.21.15.16:0/3013243597 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f93a00ae070 0x7f93a00ae4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:44.132 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:44.122+0000 7f939ed9d640 1 -- 172.21.15.16:0/3013243597 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f93a00c5b30 msgr2=0x7f93a00c5fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:44.132 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:44.122+0000 7f939ed9d640 1 --2- 172.21.15.16:0/3013243597 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f93a00c5b30 0x7f93a00c5fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:44.132 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:44.122+0000 7f939ed9d640 1 -- 172.21.15.16:0/3013243597 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f93a00ae070 msgr2=0x7f93a00ae4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:44.132 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:44.122+0000 7f939ed9d640 1 --2- 172.21.15.16:0/3013243597 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f93a00ae070 0x7f93a00ae4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:44.132 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:44.122+0000 7f939ed9d640 1 -- 172.21.15.16:0/3013243597 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93a015d200 con 0x7f938c003f00 2023-05-22T15:26:44.132 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:44.122+0000 7f939f59e640 1 --2- 172.21.15.16:0/3013243597 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f93a00ae070 0x7f93a00ae4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:26:44.132 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:44.122+0000 7f939ed9d640 1 --2- 172.21.15.16:0/3013243597 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f938c003f00 0x7f93a0148f80 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f9390011810 tx=0x7f9390011cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:26:44.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:44.122+0000 7f93977fe640 1 -- 172.21.15.16:0/3013243597 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f939000f960 con 0x7f938c003f00 2023-05-22T15:26:44.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:44.122+0000 7f93977fe640 1 -- 172.21.15.16:0/3013243597 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f939000fac0 con 0x7f938c003f00 2023-05-22T15:26:44.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:44.122+0000 7f93a51fe640 1 -- 172.21.15.16:0/3013243597 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93a015cf20 con 0x7f938c003f00 2023-05-22T15:26:44.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:44.122+0000 7f93977fe640 1 -- 172.21.15.16:0/3013243597 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f939000fe50 con 0x7f938c003f00 2023-05-22T15:26:44.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:44.122+0000 7f93a51fe640 1 -- 172.21.15.16:0/3013243597 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93a0159160 con 0x7f938c003f00 2023-05-22T15:26:44.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:44.122+0000 7f93a51fe640 1 -- 172.21.15.16:0/3013243597 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93a00ca0c0 con 0x7f938c003f00 2023-05-22T15:26:44.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:44.122+0000 7f93977fe640 1 -- 172.21.15.16:0/3013243597 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7f939001f460 con 0x7f938c003f00 2023-05-22T15:26:44.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:44.122+0000 7f93977fe640 1 --2- 172.21.15.16:0/3013243597 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f9380065380 0x7f9380067830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:44.134 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:44.122+0000 7f93977fe640 1 -- 172.21.15.16:0/3013243597 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 5199+0+0 (secure 0 0 0) 0x7f9390085630 con 0x7f938c003f00 2023-05-22T15:26:44.134 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:44.122+0000 7f939e59c640 1 --2- 172.21.15.16:0/3013243597 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f9380065380 0x7f9380067830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:44.134 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:44.126+0000 7f939e59c640 1 --2- 172.21.15.16:0/3013243597 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f9380065380 0x7f9380067830 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f93880af000 tx=0x7f93880046d0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:26:44.134 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:44.130+0000 7f93977fe640 1 -- 172.21.15.16:0/3013243597 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f93900535b0 con 0x7f938c003f00 2023-05-22T15:26:44.290 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:44.278+0000 7f93a51fe640 1 -- 172.21.15.16:0/3013243597 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f93a00c3930 con 0x7f938c003f00 2023-05-22T15:26:44.972 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:44.962+0000 7f93977fe640 1 -- 172.21.15.16:0/3013243597 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}]=0 enabled application 'rbd' on pool 'rbd' v18) v1 ==== 173+0+0 (secure 0 0 0) 0x7f93a00c3930 con 0x7f938c003f00 2023-05-22T15:26:44.972 INFO:teuthology.orchestra.run.smithi016.stderr:enabled application 'rbd' on pool 'rbd' 2023-05-22T15:26:44.977 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:44.966+0000 7f93a51fe640 1 -- 172.21.15.16:0/3013243597 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f9380065380 msgr2=0x7f9380067830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:44.977 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:44.966+0000 7f93a51fe640 1 --2- 172.21.15.16:0/3013243597 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f9380065380 0x7f9380067830 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f93880af000 tx=0x7f93880046d0 comp rx=0 tx=0).stop 2023-05-22T15:26:44.977 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:44.966+0000 7f93a51fe640 1 -- 172.21.15.16:0/3013243597 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f938c003f00 msgr2=0x7f93a0148f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:44.977 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:44.966+0000 7f93a51fe640 1 --2- 172.21.15.16:0/3013243597 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f938c003f00 0x7f93a0148f80 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f9390011810 tx=0x7f9390011cd0 comp rx=0 tx=0).stop 2023-05-22T15:26:44.978 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:44.966+0000 7f93a51fe640 1 -- 172.21.15.16:0/3013243597 shutdown_connections 2023-05-22T15:26:44.978 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:44.966+0000 7f93a51fe640 1 --2- 172.21.15.16:0/3013243597 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f93a00ae070 0x7f93a00ae4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:44.978 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:44.966+0000 7f93a51fe640 1 --2- 172.21.15.16:0/3013243597 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f9380065380 0x7f9380067830 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:44.978 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:44.966+0000 7f93a51fe640 1 --2- 172.21.15.16:0/3013243597 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f93a00c5b30 0x7f93a00c5fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:44.978 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:44.966+0000 7f93a51fe640 1 --2- 172.21.15.16:0/3013243597 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f938c003f00 0x7f93a0148f80 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:44.979 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:44.966+0000 7f93a51fe640 1 -- 172.21.15.16:0/3013243597 >> 172.21.15.16:0/3013243597 conn(0x7f93a0069720 msgr2=0x7f93a00cb4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:26:44.979 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:44.966+0000 7f93a51fe640 1 -- 172.21.15.16:0/3013243597 shutdown_connections 2023-05-22T15:26:44.979 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:44.966+0000 7f93a51fe640 1 -- 172.21.15.16:0/3013243597 wait complete. 2023-05-22T15:26:44.999 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2023-05-22T15:26:45.000 INFO:tasks.ceph.mds.a:Restarting daemon 2023-05-22T15:26:45.000 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-05-22T15:26:45.005 INFO:tasks.ceph.mds.a:Started 2023-05-22T15:26:45.005 INFO:tasks.ceph.mds.c:Restarting daemon 2023-05-22T15:26:45.010 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-05-22T15:26:45.013 INFO:tasks.ceph.mds.c:Started 2023-05-22T15:26:45.013 INFO:tasks.ceph.mds.b:Restarting daemon 2023-05-22T15:26:45.013 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-05-22T15:26:45.016 INFO:tasks.ceph.mds.b:Started 2023-05-22T15:26:45.016 INFO:tasks.ceph.mds.d:Restarting daemon 2023-05-22T15:26:45.017 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-05-22T15:26:45.019 INFO:tasks.ceph.mds.d:Started 2023-05-22T15:26:45.019 INFO:tasks.ceph:Setting up CephFS filesystem(s)... 2023-05-22T15:26:45.020 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-05-22T15:26:45.097 INFO:tasks.ceph.mds.d.smithi062.stdout:starting mds.d at 2023-05-22T15:26:45.098 INFO:tasks.ceph.mds.b.smithi062.stdout:starting mds.b at 2023-05-22T15:26:45.141 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.130+0000 7f86ba418640 1 Processor -- start 2023-05-22T15:26:45.141 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.130+0000 7f86ba418640 1 -- start start 2023-05-22T15:26:45.147 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.130+0000 7f86ba418640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f86b40c9b40 0x7f86b415e460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:45.147 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.130+0000 7f86ba418640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f86b4070df0 0x7f86b4071210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:45.148 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.130+0000 7f86ba418640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86b40b0650 con 0x7f86b40ca4e0 2023-05-22T15:26:45.148 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.130+0000 7f86ba418640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86b40b0370 con 0x7f86b40c9b40 2023-05-22T15:26:45.148 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.130+0000 7f86ba418640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86b415d980 con 0x7f86b4070df0 2023-05-22T15:26:45.148 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.130+0000 7f86b3fff640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f86b40ca4e0 0x7f86b40afb40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:36872/0 (socket says 172.21.15.16:36872) 2023-05-22T15:26:45.149 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.130+0000 7f86b3fff640 1 -- 172.21.15.16:0/630059913 learned_addr learned my addr 172.21.15.16:0/630059913 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:26:45.149 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.130+0000 7f86b2ffd640 1 -- 172.21.15.16:0/630059913 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 932426891 0 0) 0x7f86b40b0650 con 0x7f86b40ca4e0 2023-05-22T15:26:45.149 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.130+0000 7f86b2ffd640 1 -- 172.21.15.16:0/630059913 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f869c003530 con 0x7f86b40ca4e0 2023-05-22T15:26:45.149 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.130+0000 7f86b89b5640 1 --2- 172.21.15.16:0/630059913 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f86b4070df0 0x7f86b4071210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:45.150 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.130+0000 7f86b37fe640 1 --2- 172.21.15.16:0/630059913 >> v2:172.21.15.62:3300/0 conn(0x7f86b40c9b40 0x7f86b415e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:45.150 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.130+0000 7f86b2ffd640 1 -- 172.21.15.16:0/630059913 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3298939054 0 0) 0x7f869c003530 con 0x7f86b40ca4e0 2023-05-22T15:26:45.150 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.130+0000 7f86b2ffd640 1 -- 172.21.15.16:0/630059913 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f86b4070df0 msgr2=0x7f86b4071210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:26:45.151 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.130+0000 7f86b2ffd640 1 --2- 172.21.15.16:0/630059913 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f86b4070df0 0x7f86b4071210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:45.151 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.130+0000 7f86b2ffd640 1 -- 172.21.15.16:0/630059913 >> v2:172.21.15.62:3300/0 conn(0x7f86b40c9b40 msgr2=0x7f86b415e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:26:45.151 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.130+0000 7f86b2ffd640 1 --2- 172.21.15.16:0/630059913 >> v2:172.21.15.62:3300/0 conn(0x7f86b40c9b40 0x7f86b415e460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:45.151 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.130+0000 7f86b2ffd640 1 -- 172.21.15.16:0/630059913 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86b415dc60 con 0x7f86b40ca4e0 2023-05-22T15:26:45.151 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.130+0000 7f86b2ffd640 1 -- 172.21.15.16:0/630059913 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7f86ac001a20 con 0x7f86b40ca4e0 2023-05-22T15:26:45.152 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.130+0000 7f86b2ffd640 1 -- 172.21.15.16:0/630059913 >> v1:172.21.15.16:6789/0 conn(0x7f86b40ca4e0 legacy=0x7f86b40afb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:45.152 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.130+0000 7f86b2ffd640 1 --2- 172.21.15.16:0/630059913 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f86b4070df0 0x7f869c003980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:45.152 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.130+0000 7f86b2ffd640 1 --2- 172.21.15.16:0/630059913 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f869c003f00 0x7f869c0047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:45.153 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.130+0000 7f86b2ffd640 1 --2- 172.21.15.16:0/630059913 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f869c004d10 0x7f869c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:45.153 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.130+0000 7f86b2ffd640 1 -- 172.21.15.16:0/630059913 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86b40b0370 con 0x7f869c003f00 2023-05-22T15:26:45.153 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.130+0000 7f86b2ffd640 1 -- 172.21.15.16:0/630059913 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86b415d980 con 0x7f869c004d10 2023-05-22T15:26:45.153 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.130+0000 7f86b2ffd640 1 -- 172.21.15.16:0/630059913 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f869c00b990 con 0x7f86b4070df0 2023-05-22T15:26:45.154 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.130+0000 7f86b37fe640 1 --2- 172.21.15.16:0/630059913 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f869c003f00 0x7f869c0047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:45.154 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.130+0000 7f86b3fff640 1 --2- 172.21.15.16:0/630059913 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f86b4070df0 0x7f869c003980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:45.154 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.130+0000 7f86b89b5640 1 --2- 172.21.15.16:0/630059913 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f869c004d10 0x7f869c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:45.154 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.134+0000 7f86b37fe640 1 -- 172.21.15.16:0/630059913 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f86b4070df0 msgr2=0x7f869c003980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:26:45.155 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.134+0000 7f86b37fe640 1 --2- 172.21.15.16:0/630059913 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f86b4070df0 0x7f869c003980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:45.155 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.134+0000 7f86b37fe640 1 -- 172.21.15.16:0/630059913 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f869c004d10 msgr2=0x7f869c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:26:45.155 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.134+0000 7f86b37fe640 1 --2- 172.21.15.16:0/630059913 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f869c004d10 0x7f869c005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:45.155 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.134+0000 7f86b37fe640 1 -- 172.21.15.16:0/630059913 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f869c003530 con 0x7f869c003f00 2023-05-22T15:26:45.156 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.134+0000 7f86b37fe640 1 --2- 172.21.15.16:0/630059913 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f869c003f00 0x7f869c0047d0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f86a8009ae0 tx=0x7f86a801ca90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27bd7272ee549a4e server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:26:45.156 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.134+0000 7f86b2ffd640 1 -- 172.21.15.16:0/630059913 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86a80a5570 con 0x7f869c003f00 2023-05-22T15:26:45.156 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.134+0000 7f86b2ffd640 1 -- 172.21.15.16:0/630059913 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86a80a56d0 con 0x7f869c003f00 2023-05-22T15:26:45.157 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.134+0000 7f86b2ffd640 1 -- 172.21.15.16:0/630059913 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86a80a5a80 con 0x7f869c003f00 2023-05-22T15:26:45.157 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.134+0000 7f86ba418640 1 -- 172.21.15.16:0/630059913 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f869c003f00 msgr2=0x7f869c0047d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:45.157 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.134+0000 7f86ba418640 1 --2- 172.21.15.16:0/630059913 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f869c003f00 0x7f869c0047d0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f86a8009ae0 tx=0x7f86a801ca90 comp rx=0 tx=0).stop 2023-05-22T15:26:45.157 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.134+0000 7f86b89b5640 1 -- 172.21.15.16:0/630059913 reap_dead start 2023-05-22T15:26:45.157 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.134+0000 7f86ba418640 1 -- 172.21.15.16:0/630059913 shutdown_connections 2023-05-22T15:26:45.157 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.134+0000 7f86ba418640 1 -- 172.21.15.16:0/630059913 >> 172.21.15.16:0/630059913 conn(0x7f86b4069720 msgr2=0x7f86b4069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:26:45.158 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.134+0000 7f86ba418640 1 -- 172.21.15.16:0/630059913 shutdown_connections 2023-05-22T15:26:45.158 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.134+0000 7f86ba418640 1 -- 172.21.15.16:0/630059913 wait complete. 2023-05-22T15:26:45.158 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.138+0000 7f86ba418640 1 Processor -- start 2023-05-22T15:26:45.159 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.138+0000 7f86ba418640 1 -- start start 2023-05-22T15:26:45.162 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.138+0000 7f86ba418640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f869c003f00 0x7f86b41f16e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:45.162 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.138+0000 7f86ba418640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f86b4151860 0x7f86b415c9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:45.163 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.138+0000 7f86ba418640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f86b415cee0 0x7f86b4152f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:45.163 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.138+0000 7f86ba418640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86b41f1c20 con 0x7f86b415cee0 2023-05-22T15:26:45.164 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.138+0000 7f86ba418640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86b4153ea0 con 0x7f869c003f00 2023-05-22T15:26:45.164 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.138+0000 7f86ba418640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86b4154180 con 0x7f86b4151860 2023-05-22T15:26:45.164 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.138+0000 7f86b89b5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f86b415cee0 0x7f86b4152f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:45.165 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.138+0000 7f86b89b5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f86b415cee0 0x7f86b4152f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49680/0 (socket says 172.21.15.16:49680) 2023-05-22T15:26:45.165 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.138+0000 7f86b89b5640 1 -- 172.21.15.16:0/3788358102 learned_addr learned my addr 172.21.15.16:0/3788358102 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:26:45.165 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.138+0000 7f86b37fe640 1 --2- 172.21.15.16:0/3788358102 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f86b4151860 0x7f86b415c9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:45.165 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.138+0000 7f86b3fff640 1 --2- 172.21.15.16:0/3788358102 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f869c003f00 0x7f86b41f16e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:45.166 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.138+0000 7f86b89b5640 1 -- 172.21.15.16:0/3788358102 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f86b4151860 msgr2=0x7f86b415c9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:45.166 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.138+0000 7f86b89b5640 1 --2- 172.21.15.16:0/3788358102 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f86b4151860 0x7f86b415c9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:45.166 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.138+0000 7f86b89b5640 1 -- 172.21.15.16:0/3788358102 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f869c003f00 msgr2=0x7f86b41f16e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:45.166 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.138+0000 7f86b89b5640 1 --2- 172.21.15.16:0/3788358102 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f869c003f00 0x7f86b41f16e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:45.167 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.138+0000 7f86b89b5640 1 -- 172.21.15.16:0/3788358102 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86b40ca8b0 con 0x7f86b415cee0 2023-05-22T15:26:45.167 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.138+0000 7f86b3fff640 1 --2- 172.21.15.16:0/3788358102 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f869c003f00 0x7f86b41f16e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T15:26:45.167 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.138+0000 7f86b89b5640 1 --2- 172.21.15.16:0/3788358102 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f86b415cee0 0x7f86b4152f90 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f86b40cdef0 tx=0x7f86a4011920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:26:45.167 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.138+0000 7f86b0ff9640 1 -- 172.21.15.16:0/3788358102 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86a4011dc0 con 0x7f86b415cee0 2023-05-22T15:26:45.167 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.138+0000 7f86b0ff9640 1 -- 172.21.15.16:0/3788358102 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86a4004030 con 0x7f86b415cee0 2023-05-22T15:26:45.168 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.138+0000 7f86ba418640 1 -- 172.21.15.16:0/3788358102 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86b415df40 con 0x7f86b415cee0 2023-05-22T15:26:45.168 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.138+0000 7f86b0ff9640 1 -- 172.21.15.16:0/3788358102 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86a4004420 con 0x7f86b415cee0 2023-05-22T15:26:45.168 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.138+0000 7f86ba418640 1 -- 172.21.15.16:0/3788358102 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86b4071210 con 0x7f86b415cee0 2023-05-22T15:26:45.168 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.138+0000 7f86ba418640 1 -- 172.21.15.16:0/3788358102 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86b404dbe0 con 0x7f86b415cee0 2023-05-22T15:26:45.169 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.138+0000 7f86b0ff9640 1 -- 172.21.15.16:0/3788358102 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7f86a4003620 con 0x7f86b415cee0 2023-05-22T15:26:45.169 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.138+0000 7f86b0ff9640 1 --2- 172.21.15.16:0/3788358102 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f86a00653b0 0x7f86a0067860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:45.169 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.138+0000 7f86b0ff9640 1 -- 172.21.15.16:0/3788358102 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f86a4081f20 con 0x7f86b415cee0 2023-05-22T15:26:45.170 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.138+0000 7f86b3fff640 1 --2- 172.21.15.16:0/3788358102 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f86a00653b0 0x7f86a0067860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:45.170 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.142+0000 7f86b3fff640 1 --2- 172.21.15.16:0/3788358102 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f86a00653b0 0x7f86a0067860 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f86ac002070 tx=0x7f86ac00a060 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:26:45.170 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.146+0000 7f86b0ff9640 1 -- 172.21.15.16:0/3788358102 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f86a404fe60 con 0x7f86b415cee0 2023-05-22T15:26:45.199 INFO:tasks.ceph.mds.a.smithi016.stdout:starting mds.a at 2023-05-22T15:26:45.199 INFO:tasks.ceph.mds.c.smithi016.stdout:starting mds.c at 2023-05-22T15:26:45.317 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.306+0000 7f86ba418640 1 -- 172.21.15.16:0/3788358102 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f86b40b0030 con 0x7f86b415cee0 2023-05-22T15:26:45.319 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.310+0000 7f86b0ff9640 1 -- 172.21.15.16:0/3788358102 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v18) v1 ==== 74+0+14133 (secure 0 0 0) 0x7f86a4053480 con 0x7f86b415cee0 2023-05-22T15:26:45.320 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:26:45.320 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":18,"fsid":"b2719834-7847-4d98-90a6-f65d57f85c49","created":"2023-05-22T15:26:25.259314+0000","modified":"2023-05-22T15:26:44.955896+0000","last_up_change":"2023-05-22T15:26:36.448524+0000","last_in_change":"2023-05-22T15:26:27.675668+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-05-22T15:26:38.019781+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-05-22T15:26:43.570514+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9e7ff854-b25f-4cbb-a5c2-1799d9c53d2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":37021},{"type":"v1","addr":"172.21.15.16:6825","nonce":37021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":37021},{"type":"v1","addr":"172.21.15.16:6827","nonce":37021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":37021},{"type":"v1","addr":"172.21.15.16:6831","nonce":37021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":37021},{"type":"v1","addr":"172.21.15.16:6829","nonce":37021}]},"public_addr":"172.21.15.16:6825/37021","cluster_addr":"172.21.15.16:6827/37021","heartbeat_back_addr":"172.21.15.16:6831/37021","heartbeat_front_addr":"172.21.15.16:6829/37021","state":["exists","up"]},{"osd":1,"uuid":"4702fbc9-a927-4565-aca5-41dc69cc8264","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":37008},{"type":"v1","addr":"172.21.15.16:6801","nonce":37008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":37008},{"type":"v1","addr":"172.21.15.16:6803","nonce":37008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":37008},{"type":"v1","addr":"172.21.15.16:6807","nonce":37008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":37008},{"type":"v1","addr":"172.21.15.16:6805","nonce":37008}]},"public_addr":"172.21.15.16:6801/37008","cluster_addr":"172.21.15.16:6803/37008","heartbeat_back_addr":"172.21.15.16:6807/37008","heartbeat_front_addr":"172.21.15.16:6805/37008","state":["exists","up"]},{"osd":2,"uuid":"1e7bba46-4eba-47ef-bb81-5e2d371638ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":37009},{"type":"v1","addr":"172.21.15.16:6809","nonce":37009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":37009},{"type":"v1","addr":"172.21.15.16:6811","nonce":37009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":37009},{"type":"v1","addr":"172.21.15.16:6815","nonce":37009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":37009},{"type":"v1","addr":"172.21.15.16:6813","nonce":37009}]},"public_addr":"172.21.15.16:6809/37009","cluster_addr":"172.21.15.16:6811/37009","heartbeat_back_addr":"172.21.15.16:6815/37009","heartbeat_front_addr":"172.21.15.16:6813/37009","state":["exists","up"]},{"osd":3,"uuid":"383cacc0-5a88-4904-990e-498ca59ef48a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":37020},{"type":"v1","addr":"172.21.15.16:6817","nonce":37020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":37020},{"type":"v1","addr":"172.21.15.16:6819","nonce":37020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":37020},{"type":"v1","addr":"172.21.15.16:6823","nonce":37020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":37020},{"type":"v1","addr":"172.21.15.16:6821","nonce":37020}]},"public_addr":"172.21.15.16:6817/37020","cluster_addr":"172.21.15.16:6819/37020","heartbeat_back_addr":"172.21.15.16:6823/37020","heartbeat_front_addr":"172.21.15.16:6821/37020","state":["exists","up"]},{"osd":4,"uuid":"970500ef-95b7-42c1-b080-eed4fae7013d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":37353},{"type":"v1","addr":"172.21.15.62:6817","nonce":37353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6819","nonce":37353},{"type":"v1","addr":"172.21.15.62:6822","nonce":37353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37353},{"type":"v1","addr":"172.21.15.62:6831","nonce":37353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37353},{"type":"v1","addr":"172.21.15.62:6826","nonce":37353}]},"public_addr":"172.21.15.62:6817/37353","cluster_addr":"172.21.15.62:6822/37353","heartbeat_back_addr":"172.21.15.62:6831/37353","heartbeat_front_addr":"172.21.15.62:6826/37353","state":["exists","up"]},{"osd":5,"uuid":"022d67db-d4d8-445b-b1b9-8b446a00f5c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":37352},{"type":"v1","addr":"172.21.15.62:6801","nonce":37352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":37352},{"type":"v1","addr":"172.21.15.62:6803","nonce":37352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":37352},{"type":"v1","addr":"172.21.15.62:6807","nonce":37352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":37352},{"type":"v1","addr":"172.21.15.62:6805","nonce":37352}]},"public_addr":"172.21.15.62:6801/37352","cluster_addr":"172.21.15.62:6803/37352","heartbeat_back_addr":"172.21.15.62:6807/37352","heartbeat_front_addr":"172.21.15.62:6805/37352","state":["exists","up"]},{"osd":6,"uuid":"42bd372b-6688-4a1b-8407-e16e8c266e33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":37354},{"type":"v1","addr":"172.21.15.62:6820","nonce":37354}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":37354},{"type":"v1","addr":"172.21.15.62:6823","nonce":37354}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6829","nonce":37354},{"type":"v1","addr":"172.21.15.62:6830","nonce":37354}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6825","nonce":37354},{"type":"v1","addr":"172.21.15.62:6827","nonce":37354}]},"public_addr":"172.21.15.62:6820/37354","cluster_addr":"172.21.15.62:6823/37354","heartbeat_back_addr":"172.21.15.62:6830/37354","heartbeat_front_addr":"172.21.15.62:6827/37354","state":["exists","up"]},{"osd":7,"uuid":"ac70b2a3-f41b-4bf2-b227-7f2cbb55426d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37355},{"type":"v1","addr":"172.21.15.62:6809","nonce":37355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37355},{"type":"v1","addr":"172.21.15.62:6811","nonce":37355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":37355},{"type":"v1","addr":"172.21.15.62:6815","nonce":37355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":37355},{"type":"v1","addr":"172.21.15.62:6813","nonce":37355}]},"public_addr":"172.21.15.62:6809/37355","cluster_addr":"172.21.15.62:6811/37355","heartbeat_back_addr":"172.21.15.62:6815/37355","heartbeat_front_addr":"172.21.15.62:6813/37355","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.368896+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.107273+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.276888+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.287502+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.124791+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.251937+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.255679+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.232797+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-05-22T15:26:45.322 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.310+0000 7f86ba418640 1 -- 172.21.15.16:0/3788358102 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f86a00653b0 msgr2=0x7f86a0067860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:45.322 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.310+0000 7f86ba418640 1 --2- 172.21.15.16:0/3788358102 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f86a00653b0 0x7f86a0067860 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f86ac002070 tx=0x7f86ac00a060 comp rx=0 tx=0).stop 2023-05-22T15:26:45.322 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.310+0000 7f86ba418640 1 -- 172.21.15.16:0/3788358102 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f86b415cee0 msgr2=0x7f86b4152f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:45.322 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.310+0000 7f86ba418640 1 --2- 172.21.15.16:0/3788358102 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f86b415cee0 0x7f86b4152f90 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f86b40cdef0 tx=0x7f86a4011920 comp rx=0 tx=0).stop 2023-05-22T15:26:45.322 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.310+0000 7f86ba418640 1 -- 172.21.15.16:0/3788358102 shutdown_connections 2023-05-22T15:26:45.323 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.310+0000 7f86ba418640 1 --2- 172.21.15.16:0/3788358102 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f86b415cee0 0x7f86b4152f90 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:45.323 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.310+0000 7f86ba418640 1 --2- 172.21.15.16:0/3788358102 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f86a00653b0 0x7f86a0067860 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:45.323 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.310+0000 7f86ba418640 1 --2- 172.21.15.16:0/3788358102 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f86b4151860 0x7f86b415c9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:45.323 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.310+0000 7f86ba418640 1 --2- 172.21.15.16:0/3788358102 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f869c003f00 0x7f86b41f16e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:45.323 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.310+0000 7f86ba418640 1 -- 172.21.15.16:0/3788358102 >> 172.21.15.16:0/3788358102 conn(0x7f86b4069720 msgr2=0x7f86b41589d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:26:45.323 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.310+0000 7f86ba418640 1 -- 172.21.15.16:0/3788358102 shutdown_connections 2023-05-22T15:26:45.323 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.310+0000 7f86ba418640 1 -- 172.21.15.16:0/3788358102 wait complete. 2023-05-22T15:26:45.338 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-05-22T15:26:38.019781+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-05-22T15:26:43.570514+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-05-22T15:26:45.339 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-05-22T15:26:45.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.450+0000 7fd98258c640 1 Processor -- start 2023-05-22T15:26:45.460 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.450+0000 7fd98258c640 1 -- start start 2023-05-22T15:26:45.462 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.450+0000 7fd98258c640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fd97c15bca0 0x7fd97c160160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:45.462 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.450+0000 7fd98258c640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd97c0b4a70 0x7fd97c0d0890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:45.462 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.450+0000 7fd98258c640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd97c0d0400 con 0x7fd97c0b3ee0 2023-05-22T15:26:45.462 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.450+0000 7fd98258c640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd97c15c070 con 0x7fd97c15bca0 2023-05-22T15:26:45.462 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.450+0000 7fd98258c640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd97c1606a0 con 0x7fd97c0b4a70 2023-05-22T15:26:45.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.450+0000 7fd97b7fe640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fd97c0b3ee0 0x7fd97c0b4360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:36912/0 (socket says 172.21.15.16:36912) 2023-05-22T15:26:45.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.450+0000 7fd97b7fe640 1 -- 172.21.15.16:0/3202838883 learned_addr learned my addr 172.21.15.16:0/3202838883 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:26:45.464 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.450+0000 7fd97bfff640 1 --2- 172.21.15.16:0/3202838883 >> v2:172.21.15.62:3300/0 conn(0x7fd97c15bca0 0x7fd97c160160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:45.464 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.450+0000 7fd980b29640 1 --2- 172.21.15.16:0/3202838883 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd97c0b4a70 0x7fd97c0d0890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:45.464 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.450+0000 7fd97affd640 1 -- 172.21.15.16:0/3202838883 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1474193310 0 0) 0x7fd97c0d0400 con 0x7fd97c0b3ee0 2023-05-22T15:26:45.465 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.450+0000 7fd97affd640 1 -- 172.21.15.16:0/3202838883 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd964003530 con 0x7fd97c0b3ee0 2023-05-22T15:26:45.465 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.450+0000 7fd97affd640 1 -- 172.21.15.16:0/3202838883 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2093995357 0 0) 0x7fd964003530 con 0x7fd97c0b3ee0 2023-05-22T15:26:45.465 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.450+0000 7fd97affd640 1 -- 172.21.15.16:0/3202838883 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd97c0b4a70 msgr2=0x7fd97c0d0890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:26:45.465 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.450+0000 7fd97affd640 1 --2- 172.21.15.16:0/3202838883 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd97c0b4a70 0x7fd97c0d0890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:45.465 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.450+0000 7fd97affd640 1 -- 172.21.15.16:0/3202838883 >> v2:172.21.15.62:3300/0 conn(0x7fd97c15bca0 msgr2=0x7fd97c160160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:26:45.465 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.450+0000 7fd97affd640 1 --2- 172.21.15.16:0/3202838883 >> v2:172.21.15.62:3300/0 conn(0x7fd97c15bca0 0x7fd97c160160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:45.465 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.450+0000 7fd97affd640 1 -- 172.21.15.16:0/3202838883 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd97c0d00d0 con 0x7fd97c0b3ee0 2023-05-22T15:26:45.466 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.450+0000 7fd97affd640 1 -- 172.21.15.16:0/3202838883 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7fd970001680 con 0x7fd97c0b3ee0 2023-05-22T15:26:45.466 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.450+0000 7fd97affd640 1 -- 172.21.15.16:0/3202838883 >> v1:172.21.15.16:6789/0 conn(0x7fd97c0b3ee0 legacy=0x7fd97c0b4360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:45.466 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.450+0000 7fd97affd640 1 --2- 172.21.15.16:0/3202838883 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd964003980 0x7fd964003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:45.466 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.450+0000 7fd97affd640 1 --2- 172.21.15.16:0/3202838883 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd9640047d0 0x7fd964004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:45.466 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.450+0000 7fd97affd640 1 --2- 172.21.15.16:0/3202838883 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd97c0b4a70 0x7fd964005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:45.466 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.450+0000 7fd97affd640 1 -- 172.21.15.16:0/3202838883 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd97c15c070 con 0x7fd9640047d0 2023-05-22T15:26:45.466 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.450+0000 7fd97affd640 1 -- 172.21.15.16:0/3202838883 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd97c1606a0 con 0x7fd964003980 2023-05-22T15:26:45.466 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.450+0000 7fd97affd640 1 -- 172.21.15.16:0/3202838883 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd96400ba10 con 0x7fd97c0b4a70 2023-05-22T15:26:45.467 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.450+0000 7fd97b7fe640 1 --2- 172.21.15.16:0/3202838883 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd9640047d0 0x7fd964004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:45.467 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.450+0000 7fd97bfff640 1 --2- 172.21.15.16:0/3202838883 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd964003980 0x7fd964003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:45.467 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.450+0000 7fd980b29640 1 --2- 172.21.15.16:0/3202838883 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd97c0b4a70 0x7fd964005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:45.467 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.450+0000 7fd97b7fe640 1 -- 172.21.15.16:0/3202838883 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd97c0b4a70 msgr2=0x7fd964005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:26:45.467 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.450+0000 7fd97b7fe640 1 --2- 172.21.15.16:0/3202838883 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd97c0b4a70 0x7fd964005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:45.467 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.450+0000 7fd97b7fe640 1 -- 172.21.15.16:0/3202838883 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd964003980 msgr2=0x7fd964003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:26:45.467 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.450+0000 7fd97b7fe640 1 --2- 172.21.15.16:0/3202838883 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd964003980 0x7fd964003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:45.468 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.450+0000 7fd97b7fe640 1 -- 172.21.15.16:0/3202838883 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd964003530 con 0x7fd9640047d0 2023-05-22T15:26:45.468 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.450+0000 7fd97bfff640 1 --2- 172.21.15.16:0/3202838883 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd964003980 0x7fd964003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:26:45.476 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.454+0000 7fd97b7fe640 1 --2- 172.21.15.16:0/3202838883 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd9640047d0 0x7fd964004c40 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fd97000c290 tx=0x7fd97001ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e477a7394548b4f server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:26:45.476 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.454+0000 7fd97affd640 1 -- 172.21.15.16:0/3202838883 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9700a5900 con 0x7fd9640047d0 2023-05-22T15:26:45.476 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.454+0000 7fd97affd640 1 -- 172.21.15.16:0/3202838883 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9700a5a60 con 0x7fd9640047d0 2023-05-22T15:26:45.476 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.454+0000 7fd97affd640 1 -- 172.21.15.16:0/3202838883 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9700a5e10 con 0x7fd9640047d0 2023-05-22T15:26:45.477 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.454+0000 7fd98258c640 1 -- 172.21.15.16:0/3202838883 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd9640047d0 msgr2=0x7fd964004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:45.477 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.454+0000 7fd98258c640 1 --2- 172.21.15.16:0/3202838883 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd9640047d0 0x7fd964004c40 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fd97000c290 tx=0x7fd97001ccb0 comp rx=0 tx=0).stop 2023-05-22T15:26:45.477 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.454+0000 7fd980b29640 1 -- 172.21.15.16:0/3202838883 reap_dead start 2023-05-22T15:26:45.477 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.454+0000 7fd98258c640 1 -- 172.21.15.16:0/3202838883 shutdown_connections 2023-05-22T15:26:45.478 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.454+0000 7fd98258c640 1 -- 172.21.15.16:0/3202838883 >> 172.21.15.16:0/3202838883 conn(0x7fd97c0657f0 msgr2=0x7fd97c070e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:26:45.478 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.454+0000 7fd98258c640 1 -- 172.21.15.16:0/3202838883 shutdown_connections 2023-05-22T15:26:45.478 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.454+0000 7fd98258c640 1 -- 172.21.15.16:0/3202838883 wait complete. 2023-05-22T15:26:45.478 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.454+0000 7fd98258c640 1 Processor -- start 2023-05-22T15:26:45.478 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.454+0000 7fd98258c640 1 -- start start 2023-05-22T15:26:45.478 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.458+0000 7fd98258c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd9640047d0 0x7fd97c0b3280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:45.479 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.458+0000 7fd98258c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd97c154cc0 0x7fd97c15eb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:45.479 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.458+0000 7fd98258c640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd97c142a90 0x7fd97c142f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:45.479 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.458+0000 7fd98258c640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd97c0b2e60 con 0x7fd97c154cc0 2023-05-22T15:26:45.479 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.458+0000 7fd98258c640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd97c143470 con 0x7fd9640047d0 2023-05-22T15:26:45.479 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.458+0000 7fd98258c640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd97c0b27e0 con 0x7fd97c142a90 2023-05-22T15:26:45.479 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.458+0000 7fd97b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd97c154cc0 0x7fd97c15eb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:45.479 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.458+0000 7fd97b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd97c154cc0 0x7fd97c15eb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49724/0 (socket says 172.21.15.16:49724) 2023-05-22T15:26:45.480 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.458+0000 7fd97b7fe640 1 -- 172.21.15.16:0/178569177 learned_addr learned my addr 172.21.15.16:0/178569177 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:26:45.480 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.458+0000 7fd97bfff640 1 --2- 172.21.15.16:0/178569177 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd9640047d0 0x7fd97c0b3280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:45.480 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.458+0000 7fd980b29640 1 --2- 172.21.15.16:0/178569177 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd97c142a90 0x7fd97c142f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:45.480 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.458+0000 7fd97b7fe640 1 -- 172.21.15.16:0/178569177 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd97c142a90 msgr2=0x7fd97c142f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:45.480 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.458+0000 7fd97b7fe640 1 --2- 172.21.15.16:0/178569177 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd97c142a90 0x7fd97c142f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:45.480 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.458+0000 7fd97b7fe640 1 -- 172.21.15.16:0/178569177 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd9640047d0 msgr2=0x7fd97c0b3280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:45.480 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.458+0000 7fd97b7fe640 1 --2- 172.21.15.16:0/178569177 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd9640047d0 0x7fd97c0b3280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:45.481 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.458+0000 7fd97b7fe640 1 -- 172.21.15.16:0/178569177 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd97c0d1080 con 0x7fd97c154cc0 2023-05-22T15:26:45.481 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.458+0000 7fd97b7fe640 1 --2- 172.21.15.16:0/178569177 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd97c154cc0 0x7fd97c15eb10 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fd97000c290 tx=0x7fd964005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:26:45.481 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.458+0000 7fd978ff9640 1 -- 172.21.15.16:0/178569177 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9700047f0 con 0x7fd97c154cc0 2023-05-22T15:26:45.481 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.458+0000 7fd978ff9640 1 -- 172.21.15.16:0/178569177 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd970004970 con 0x7fd97c154cc0 2023-05-22T15:26:45.481 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.458+0000 7fd98258c640 1 -- 172.21.15.16:0/178569177 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd97c0cd1a0 con 0x7fd97c154cc0 2023-05-22T15:26:45.481 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.458+0000 7fd978ff9640 1 -- 172.21.15.16:0/178569177 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd970004d20 con 0x7fd97c154cc0 2023-05-22T15:26:45.481 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.458+0000 7fd98258c640 1 -- 172.21.15.16:0/178569177 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd97c0d1210 con 0x7fd97c154cc0 2023-05-22T15:26:45.481 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.458+0000 7fd98258c640 1 -- 172.21.15.16:0/178569177 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd97c0b2fa0 con 0x7fd97c154cc0 2023-05-22T15:26:45.482 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.458+0000 7fd978ff9640 1 -- 172.21.15.16:0/178569177 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7fd9700b5070 con 0x7fd97c154cc0 2023-05-22T15:26:45.482 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.458+0000 7fd978ff9640 1 --2- 172.21.15.16:0/178569177 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd968065380 0x7fd968067830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:45.482 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.458+0000 7fd978ff9640 1 -- 172.21.15.16:0/178569177 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fd97011bc70 con 0x7fd97c154cc0 2023-05-22T15:26:45.482 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.458+0000 7fd97bfff640 1 --2- 172.21.15.16:0/178569177 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd968065380 0x7fd968067830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:45.482 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.458+0000 7fd97bfff640 1 --2- 172.21.15.16:0/178569177 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd968065380 0x7fd968067830 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fd96c009b40 tx=0x7fd96c008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:26:45.482 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.462+0000 7fd978ff9640 1 -- 172.21.15.16:0/178569177 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fd97c0b2fa0 con 0x7fd97c154cc0 2023-05-22T15:26:45.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.618+0000 7fd98258c640 1 -- 172.21.15.16:0/178569177 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fd97c0b2fa0 con 0x7fd97c154cc0 2023-05-22T15:26:45.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.622+0000 7fd978ff9640 1 -- 172.21.15.16:0/178569177 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v18) v1 ==== 93+0+10 (secure 0 0 0) 0x7fd97c0b2fa0 con 0x7fd97c154cc0 2023-05-22T15:26:45.631 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2023-05-22T15:26:45.633 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.622+0000 7fd98258c640 1 -- 172.21.15.16:0/178569177 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd968065380 msgr2=0x7fd968067830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:45.633 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.622+0000 7fd98258c640 1 --2- 172.21.15.16:0/178569177 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd968065380 0x7fd968067830 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fd96c009b40 tx=0x7fd96c008040 comp rx=0 tx=0).stop 2023-05-22T15:26:45.633 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.622+0000 7fd98258c640 1 -- 172.21.15.16:0/178569177 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd97c154cc0 msgr2=0x7fd97c15eb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:45.633 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.622+0000 7fd98258c640 1 --2- 172.21.15.16:0/178569177 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd97c154cc0 0x7fd97c15eb10 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fd97000c290 tx=0x7fd964005800 comp rx=0 tx=0).stop 2023-05-22T15:26:45.633 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.622+0000 7fd98258c640 1 -- 172.21.15.16:0/178569177 shutdown_connections 2023-05-22T15:26:45.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.622+0000 7fd98258c640 1 --2- 172.21.15.16:0/178569177 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd968065380 0x7fd968067830 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:45.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.622+0000 7fd98258c640 1 --2- 172.21.15.16:0/178569177 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd97c142a90 0x7fd97c142f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:45.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.622+0000 7fd98258c640 1 --2- 172.21.15.16:0/178569177 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd97c154cc0 0x7fd97c15eb10 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:45.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.622+0000 7fd98258c640 1 --2- 172.21.15.16:0/178569177 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd9640047d0 0x7fd97c0b3280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:45.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.622+0000 7fd98258c640 1 -- 172.21.15.16:0/178569177 >> 172.21.15.16:0/178569177 conn(0x7fd97c0657f0 msgr2=0x7fd97c155620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:26:45.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.622+0000 7fd98258c640 1 -- 172.21.15.16:0/178569177 shutdown_connections 2023-05-22T15:26:45.635 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.622+0000 7fd98258c640 1 -- 172.21.15.16:0/178569177 wait complete. 2023-05-22T15:26:45.651 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-05-22T15:26:45.776 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.766+0000 7f7a784a0640 1 Processor -- start 2023-05-22T15:26:45.777 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.766+0000 7f7a784a0640 1 -- start start 2023-05-22T15:26:45.777 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.766+0000 7f7a784a0640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f7a700b5a50 0x7f7a700d2900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:45.777 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.766+0000 7f7a784a0640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a700cccc0 con 0x7f7a7015bca0 2023-05-22T15:26:45.777 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.766+0000 7f7a784a0640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a700ccfa0 con 0x7f7a700d2480 2023-05-22T15:26:45.777 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.766+0000 7f7a784a0640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a700cd660 con 0x7f7a700b5a50 2023-05-22T15:26:45.777 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.766+0000 7f7a7623c640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f7a7015bca0 0x7f7a700cda80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:36920/0 (socket says 172.21.15.16:36920) 2023-05-22T15:26:45.778 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.766+0000 7f7a7623c640 1 -- 172.21.15.16:0/3251778130 learned_addr learned my addr 172.21.15.16:0/3251778130 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:26:45.778 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.766+0000 7f7a76a3d640 1 --2- 172.21.15.16:0/3251778130 >> v2:172.21.15.62:3300/0 conn(0x7f7a700b5a50 0x7f7a700d2900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:45.778 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.766+0000 7f7a7523a640 1 -- 172.21.15.16:0/3251778130 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3304728913 0 0) 0x7f7a700cccc0 con 0x7f7a7015bca0 2023-05-22T15:26:45.778 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.766+0000 7f7a7523a640 1 -- 172.21.15.16:0/3251778130 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a5c003530 con 0x7f7a7015bca0 2023-05-22T15:26:45.779 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.766+0000 7f7a7523a640 1 -- 172.21.15.16:0/3251778130 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1698467644 0 0) 0x7f7a5c003530 con 0x7f7a7015bca0 2023-05-22T15:26:45.780 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.766+0000 7f7a7523a640 1 -- 172.21.15.16:0/3251778130 >> v2:172.21.15.62:3300/0 conn(0x7f7a700b5a50 msgr2=0x7f7a700d2900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:26:45.780 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.766+0000 7f7a7523a640 1 --2- 172.21.15.16:0/3251778130 >> v2:172.21.15.62:3300/0 conn(0x7f7a700b5a50 0x7f7a700d2900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:45.780 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.766+0000 7f7a7523a640 1 -- 172.21.15.16:0/3251778130 >> v1:172.21.15.62:6789/0 conn(0x7f7a700d2480 legacy=0x7f7a700b5340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:45.780 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.766+0000 7f7a7523a640 1 -- 172.21.15.16:0/3251778130 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a700cd280 con 0x7f7a7015bca0 2023-05-22T15:26:45.780 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.766+0000 7f7a7523a640 1 -- 172.21.15.16:0/3251778130 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7f7a6c001680 con 0x7f7a7015bca0 2023-05-22T15:26:45.780 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.766+0000 7f7a7523a640 1 -- 172.21.15.16:0/3251778130 >> v1:172.21.15.16:6789/0 conn(0x7f7a7015bca0 legacy=0x7f7a700cda80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:45.781 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.766+0000 7f7a7523a640 1 --2- 172.21.15.16:0/3251778130 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7a5c003980 0x7f7a5c003d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:45.781 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.766+0000 7f7a7523a640 1 --2- 172.21.15.16:0/3251778130 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7a5c004370 0x7f7a5c004830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:45.781 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.766+0000 7f7a7523a640 1 --2- 172.21.15.16:0/3251778130 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7a5c005030 0x7f7a5c00b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:45.781 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.766+0000 7f7a7523a640 1 -- 172.21.15.16:0/3251778130 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a700cd660 con 0x7f7a5c003980 2023-05-22T15:26:45.781 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.766+0000 7f7a7523a640 1 -- 172.21.15.16:0/3251778130 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a5c00bf80 con 0x7f7a5c004370 2023-05-22T15:26:45.781 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.766+0000 7f7a7523a640 1 -- 172.21.15.16:0/3251778130 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a5c00c140 con 0x7f7a5c005030 2023-05-22T15:26:45.781 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.766+0000 7f7a7623c640 1 --2- 172.21.15.16:0/3251778130 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7a5c003980 0x7f7a5c003d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:45.781 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.766+0000 7f7a75a3b640 1 --2- 172.21.15.16:0/3251778130 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7a5c004370 0x7f7a5c004830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:45.782 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.766+0000 7f7a76a3d640 1 --2- 172.21.15.16:0/3251778130 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7a5c005030 0x7f7a5c00b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:45.782 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.766+0000 7f7a7623c640 1 -- 172.21.15.16:0/3251778130 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7a5c005030 msgr2=0x7f7a5c00b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:26:45.782 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.766+0000 7f7a7623c640 1 --2- 172.21.15.16:0/3251778130 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7a5c005030 0x7f7a5c00b990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:45.782 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.766+0000 7f7a7623c640 1 -- 172.21.15.16:0/3251778130 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7a5c004370 msgr2=0x7f7a5c004830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:26:45.782 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.766+0000 7f7a7623c640 1 --2- 172.21.15.16:0/3251778130 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7a5c004370 0x7f7a5c004830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:45.782 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.766+0000 7f7a7623c640 1 -- 172.21.15.16:0/3251778130 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a5c003530 con 0x7f7a5c003980 2023-05-22T15:26:45.782 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.766+0000 7f7a76a3d640 1 -- 172.21.15.16:0/3251778130 reap_dead start 2023-05-22T15:26:45.783 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.766+0000 7f7a75a3b640 1 --2- 172.21.15.16:0/3251778130 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7a5c004370 0x7f7a5c004830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:26:45.785 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.770+0000 7f7a7623c640 1 --2- 172.21.15.16:0/3251778130 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7a5c003980 0x7f7a5c003d50 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f7a6c00c210 tx=0x7f7a6c01cba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25c5e32d2e745c0d server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:26:45.786 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.770+0000 7f7a7523a640 1 -- 172.21.15.16:0/3251778130 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a6c005ae0 con 0x7f7a5c003980 2023-05-22T15:26:45.786 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.770+0000 7f7a7523a640 1 -- 172.21.15.16:0/3251778130 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a6c005c60 con 0x7f7a5c003980 2023-05-22T15:26:45.786 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.770+0000 7f7a7523a640 1 -- 172.21.15.16:0/3251778130 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a6c0072b0 con 0x7f7a5c003980 2023-05-22T15:26:45.786 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.770+0000 7f7a784a0640 1 -- 172.21.15.16:0/3251778130 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7a5c003980 msgr2=0x7f7a5c003d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:45.786 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.770+0000 7f7a784a0640 1 --2- 172.21.15.16:0/3251778130 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7a5c003980 0x7f7a5c003d50 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f7a6c00c210 tx=0x7f7a6c01cba0 comp rx=0 tx=0).stop 2023-05-22T15:26:45.786 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.770+0000 7f7a784a0640 1 -- 172.21.15.16:0/3251778130 shutdown_connections 2023-05-22T15:26:45.786 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.770+0000 7f7a784a0640 1 --2- 172.21.15.16:0/3251778130 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7a5c003980 0x7f7a5c003d50 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:45.787 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.770+0000 7f7a784a0640 1 -- 172.21.15.16:0/3251778130 >> 172.21.15.16:0/3251778130 conn(0x7f7a700655c0 msgr2=0x7f7a70070e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:26:45.787 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.770+0000 7f7a784a0640 1 -- 172.21.15.16:0/3251778130 shutdown_connections 2023-05-22T15:26:45.787 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.770+0000 7f7a784a0640 1 -- 172.21.15.16:0/3251778130 wait complete. 2023-05-22T15:26:45.787 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.774+0000 7f7a784a0640 1 Processor -- start 2023-05-22T15:26:45.787 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.774+0000 7f7a784a0640 1 -- start start 2023-05-22T15:26:45.787 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.774+0000 7f7a784a0640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7a5c003980 0x7f7a7015c280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:45.787 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.774+0000 7f7a784a0640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7a701f5840 0x7f7a7015ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:45.787 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.774+0000 7f7a784a0640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7a701f85f0 0x7f7a700c84a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:45.788 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.774+0000 7f7a784a0640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a700ce190 con 0x7f7a701f85f0 2023-05-22T15:26:45.788 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.774+0000 7f7a784a0640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a7015e970 con 0x7f7a5c003980 2023-05-22T15:26:45.788 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.774+0000 7f7a784a0640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a7015f340 con 0x7f7a701f5840 2023-05-22T15:26:45.788 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.774+0000 7f7a76a3d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7a701f85f0 0x7f7a700c84a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:45.788 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.774+0000 7f7a76a3d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7a701f85f0 0x7f7a700c84a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49742/0 (socket says 172.21.15.16:49742) 2023-05-22T15:26:45.788 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.774+0000 7f7a76a3d640 1 -- 172.21.15.16:0/4007936182 learned_addr learned my addr 172.21.15.16:0/4007936182 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:26:45.788 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.774+0000 7f7a7623c640 1 --2- 172.21.15.16:0/4007936182 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7a5c003980 0x7f7a7015c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:45.789 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.774+0000 7f7a75a3b640 1 --2- 172.21.15.16:0/4007936182 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7a701f5840 0x7f7a7015ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:45.789 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.774+0000 7f7a76a3d640 1 -- 172.21.15.16:0/4007936182 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7a701f5840 msgr2=0x7f7a7015ee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:45.789 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.774+0000 7f7a76a3d640 1 --2- 172.21.15.16:0/4007936182 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7a701f5840 0x7f7a7015ee00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:45.790 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.774+0000 7f7a76a3d640 1 -- 172.21.15.16:0/4007936182 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7a5c003980 msgr2=0x7f7a7015c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:45.790 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.774+0000 7f7a76a3d640 1 --2- 172.21.15.16:0/4007936182 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7a5c003980 0x7f7a7015c280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:45.790 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.774+0000 7f7a76a3d640 1 -- 172.21.15.16:0/4007936182 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a70160360 con 0x7f7a701f85f0 2023-05-22T15:26:45.790 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.774+0000 7f7a7623c640 1 --2- 172.21.15.16:0/4007936182 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7a5c003980 0x7f7a7015c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:26:45.790 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.774+0000 7f7a76a3d640 1 --2- 172.21.15.16:0/4007936182 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7a701f85f0 0x7f7a700c84a0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f7a7008b260 tx=0x7f7a6800f150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:26:45.790 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.774+0000 7f7a66ffd640 1 -- 172.21.15.16:0/4007936182 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a6800f910 con 0x7f7a701f85f0 2023-05-22T15:26:45.790 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.774+0000 7f7a66ffd640 1 -- 172.21.15.16:0/4007936182 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a6800fa90 con 0x7f7a701f85f0 2023-05-22T15:26:45.791 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.774+0000 7f7a784a0640 1 -- 172.21.15.16:0/4007936182 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a700b5750 con 0x7f7a701f85f0 2023-05-22T15:26:45.791 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.774+0000 7f7a66ffd640 1 -- 172.21.15.16:0/4007936182 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a6800fe40 con 0x7f7a701f85f0 2023-05-22T15:26:45.791 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.774+0000 7f7a784a0640 1 -- 172.21.15.16:0/4007936182 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a701604f0 con 0x7f7a701f85f0 2023-05-22T15:26:45.792 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.774+0000 7f7a784a0640 1 -- 172.21.15.16:0/4007936182 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a700b4860 con 0x7f7a701f85f0 2023-05-22T15:26:45.792 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.774+0000 7f7a66ffd640 1 -- 172.21.15.16:0/4007936182 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7f7a68020460 con 0x7f7a701f85f0 2023-05-22T15:26:45.792 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.778+0000 7f7a66ffd640 1 --2- 172.21.15.16:0/4007936182 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f7a540653b0 0x7f7a54067860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:45.792 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.778+0000 7f7a66ffd640 1 -- 172.21.15.16:0/4007936182 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f7a68082aa0 con 0x7f7a701f85f0 2023-05-22T15:26:45.792 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.778+0000 7f7a7623c640 1 --2- 172.21.15.16:0/4007936182 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f7a540653b0 0x7f7a54067860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:45.793 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.778+0000 7f7a7623c640 1 --2- 172.21.15.16:0/4007936182 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f7a540653b0 0x7f7a54067860 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f7a6c0ab000 tx=0x7f7a6c0a7070 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:26:45.793 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.782+0000 7f7a66ffd640 1 -- 172.21.15.16:0/4007936182 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f7a68050a10 con 0x7f7a701f85f0 2023-05-22T15:26:45.945 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.934+0000 7f7a784a0640 1 -- 172.21.15.16:0/4007936182 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f7a7004dbe0 con 0x7f7a701f85f0 2023-05-22T15:26:45.946 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.934+0000 7f7a66ffd640 1 -- 172.21.15.16:0/4007936182 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v18) v1 ==== 92+0+10 (secure 0 0 0) 0x7f7a68054030 con 0x7f7a701f85f0 2023-05-22T15:26:45.946 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 8 2023-05-22T15:26:45.948 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.938+0000 7f7a784a0640 1 -- 172.21.15.16:0/4007936182 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f7a540653b0 msgr2=0x7f7a54067860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:45.948 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.938+0000 7f7a784a0640 1 --2- 172.21.15.16:0/4007936182 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f7a540653b0 0x7f7a54067860 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f7a6c0ab000 tx=0x7f7a6c0a7070 comp rx=0 tx=0).stop 2023-05-22T15:26:45.948 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.938+0000 7f7a784a0640 1 -- 172.21.15.16:0/4007936182 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7a701f85f0 msgr2=0x7f7a700c84a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:45.948 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.938+0000 7f7a784a0640 1 --2- 172.21.15.16:0/4007936182 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7a701f85f0 0x7f7a700c84a0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f7a7008b260 tx=0x7f7a6800f150 comp rx=0 tx=0).stop 2023-05-22T15:26:45.949 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.938+0000 7f7a784a0640 1 -- 172.21.15.16:0/4007936182 shutdown_connections 2023-05-22T15:26:45.949 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.938+0000 7f7a784a0640 1 --2- 172.21.15.16:0/4007936182 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7a701f85f0 0x7f7a700c84a0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:45.949 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.938+0000 7f7a784a0640 1 --2- 172.21.15.16:0/4007936182 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f7a540653b0 0x7f7a54067860 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:45.949 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.938+0000 7f7a784a0640 1 --2- 172.21.15.16:0/4007936182 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7a701f5840 0x7f7a7015ee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:45.949 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.938+0000 7f7a784a0640 1 --2- 172.21.15.16:0/4007936182 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7a5c003980 0x7f7a7015c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:45.949 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.938+0000 7f7a784a0640 1 -- 172.21.15.16:0/4007936182 >> 172.21.15.16:0/4007936182 conn(0x7f7a700655c0 msgr2=0x7f7a700b6510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:26:45.949 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.938+0000 7f7a784a0640 1 -- 172.21.15.16:0/4007936182 shutdown_connections 2023-05-22T15:26:45.950 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:45.938+0000 7f7a784a0640 1 -- 172.21.15.16:0/4007936182 wait complete. 2023-05-22T15:26:45.966 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-05-22T15:26:46.087 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.074+0000 7f7231eac640 1 Processor -- start 2023-05-22T15:26:46.088 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.074+0000 7f7231eac640 1 -- start start 2023-05-22T15:26:46.091 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.078+0000 7f7231eac640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f722c0ce480 0x7f722c071640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:46.091 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.078+0000 7f7231eac640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f722c15c270 0x7f722c15c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:46.091 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.078+0000 7f7231eac640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f722c069ae0 con 0x7f722c0ce850 2023-05-22T15:26:46.092 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.078+0000 7f7231eac640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f722c15f990 con 0x7f722c0ce480 2023-05-22T15:26:46.092 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.078+0000 7f7231eac640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f722c069dc0 con 0x7f722c15c270 2023-05-22T15:26:46.092 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.078+0000 7f722affd640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f722c0ce850 0x7f722c160090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:36934/0 (socket says 172.21.15.16:36934) 2023-05-22T15:26:46.092 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.078+0000 7f722affd640 1 -- 172.21.15.16:0/433913390 learned_addr learned my addr 172.21.15.16:0/433913390 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:26:46.092 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.078+0000 7f722bfff640 1 --2- 172.21.15.16:0/433913390 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f722c15c270 0x7f722c15c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:46.092 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.078+0000 7f722b7fe640 1 --2- 172.21.15.16:0/433913390 >> v2:172.21.15.62:3300/0 conn(0x7f722c0ce480 0x7f722c071640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:46.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.078+0000 7f722a7fc640 1 -- 172.21.15.16:0/433913390 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3475026620 0 0) 0x7f722c069ae0 con 0x7f722c0ce850 2023-05-22T15:26:46.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.078+0000 7f722a7fc640 1 -- 172.21.15.16:0/433913390 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7214003530 con 0x7f722c0ce850 2023-05-22T15:26:46.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.078+0000 7f722a7fc640 1 -- 172.21.15.16:0/433913390 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1611589113 0 0) 0x7f7214003530 con 0x7f722c0ce850 2023-05-22T15:26:46.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.078+0000 7f722a7fc640 1 -- 172.21.15.16:0/433913390 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f722c15c270 msgr2=0x7f722c15c6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:26:46.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.078+0000 7f722a7fc640 1 --2- 172.21.15.16:0/433913390 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f722c15c270 0x7f722c15c6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:46.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.078+0000 7f722a7fc640 1 -- 172.21.15.16:0/433913390 >> v2:172.21.15.62:3300/0 conn(0x7f722c0ce480 msgr2=0x7f722c071640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:26:46.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.078+0000 7f722a7fc640 1 --2- 172.21.15.16:0/433913390 >> v2:172.21.15.62:3300/0 conn(0x7f722c0ce480 0x7f722c071640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:46.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.078+0000 7f722a7fc640 1 -- 172.21.15.16:0/433913390 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f722c06a0a0 con 0x7f722c0ce850 2023-05-22T15:26:46.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.078+0000 7f722a7fc640 1 -- 172.21.15.16:0/433913390 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7f721c001680 con 0x7f722c0ce850 2023-05-22T15:26:46.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.078+0000 7f722a7fc640 1 -- 172.21.15.16:0/433913390 >> v1:172.21.15.16:6789/0 conn(0x7f722c0ce850 legacy=0x7f722c160090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:46.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.078+0000 7f722a7fc640 1 --2- 172.21.15.16:0/433913390 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7214003980 0x7f7214003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:46.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.078+0000 7f722a7fc640 1 --2- 172.21.15.16:0/433913390 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f722c15c270 0x7f72140047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:46.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.078+0000 7f722a7fc640 1 --2- 172.21.15.16:0/433913390 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7214004d10 0x7f7214005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:46.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.078+0000 7f722a7fc640 1 -- 172.21.15.16:0/433913390 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f722c15f990 con 0x7f7214003980 2023-05-22T15:26:46.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.078+0000 7f722a7fc640 1 -- 172.21.15.16:0/433913390 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f722c069dc0 con 0x7f7214004d10 2023-05-22T15:26:46.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.078+0000 7f722a7fc640 1 -- 172.21.15.16:0/433913390 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f721400b990 con 0x7f722c15c270 2023-05-22T15:26:46.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.078+0000 7f722b7fe640 1 --2- 172.21.15.16:0/433913390 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7214003980 0x7f7214003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:46.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.078+0000 7f722affd640 1 --2- 172.21.15.16:0/433913390 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f722c15c270 0x7f72140047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:46.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.078+0000 7f722bfff640 1 --2- 172.21.15.16:0/433913390 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7214004d10 0x7f7214005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:46.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.078+0000 7f722b7fe640 1 -- 172.21.15.16:0/433913390 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f722c15c270 msgr2=0x7f72140047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:26:46.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.078+0000 7f722b7fe640 1 --2- 172.21.15.16:0/433913390 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f722c15c270 0x7f72140047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:46.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.078+0000 7f722b7fe640 1 -- 172.21.15.16:0/433913390 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7214004d10 msgr2=0x7f7214005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:26:46.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.078+0000 7f722b7fe640 1 --2- 172.21.15.16:0/433913390 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7214004d10 0x7f7214005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:46.096 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.078+0000 7f722b7fe640 1 -- 172.21.15.16:0/433913390 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7214003530 con 0x7f7214003980 2023-05-22T15:26:46.096 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.078+0000 7f722affd640 1 --2- 172.21.15.16:0/433913390 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f722c15c270 0x7f72140047d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:26:46.096 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.078+0000 7f722bfff640 1 --2- 172.21.15.16:0/433913390 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7214004d10 0x7f7214005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:26:46.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.082+0000 7f722b7fe640 1 --2- 172.21.15.16:0/433913390 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7214003980 0x7f7214003d90 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f7220009ae0 tx=0x7f722001ca00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8670a6fa8204c95b server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:26:46.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.082+0000 7f722a7fc640 1 -- 172.21.15.16:0/433913390 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72200a5570 con 0x7f7214003980 2023-05-22T15:26:46.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.082+0000 7f722a7fc640 1 -- 172.21.15.16:0/433913390 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72200a56d0 con 0x7f7214003980 2023-05-22T15:26:46.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.082+0000 7f722a7fc640 1 -- 172.21.15.16:0/433913390 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72200a5a80 con 0x7f7214003980 2023-05-22T15:26:46.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.082+0000 7f7231eac640 1 -- 172.21.15.16:0/433913390 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7214003980 msgr2=0x7f7214003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:46.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.082+0000 7f7231eac640 1 --2- 172.21.15.16:0/433913390 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7214003980 0x7f7214003d90 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f7220009ae0 tx=0x7f722001ca00 comp rx=0 tx=0).stop 2023-05-22T15:26:46.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.082+0000 7f722bfff640 1 -- 172.21.15.16:0/433913390 reap_dead start 2023-05-22T15:26:46.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.082+0000 7f7231eac640 1 -- 172.21.15.16:0/433913390 shutdown_connections 2023-05-22T15:26:46.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.082+0000 7f7231eac640 1 -- 172.21.15.16:0/433913390 >> 172.21.15.16:0/433913390 conn(0x7f722c0cc870 msgr2=0x7f722c0cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:26:46.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.082+0000 7f7231eac640 1 -- 172.21.15.16:0/433913390 shutdown_connections 2023-05-22T15:26:46.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.082+0000 7f7231eac640 1 -- 172.21.15.16:0/433913390 wait complete. 2023-05-22T15:26:46.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.082+0000 7f7231eac640 1 Processor -- start 2023-05-22T15:26:46.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.082+0000 7f7231eac640 1 -- start start 2023-05-22T15:26:46.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.082+0000 7f7231eac640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7214003980 0x7f722c0af370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:46.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.082+0000 7f7231eac640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f722c15dda0 0x7f722c15e210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:46.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.082+0000 7f7231eac640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f722c0ab920 0x7f722c0abd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:46.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.082+0000 7f7231eac640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f722c15e750 con 0x7f722c15dda0 2023-05-22T15:26:46.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.082+0000 7f7231eac640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f722c0ac650 con 0x7f7214003980 2023-05-22T15:26:46.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.082+0000 7f7231eac640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f722c155110 con 0x7f722c0ab920 2023-05-22T15:26:46.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.082+0000 7f722affd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f722c15dda0 0x7f722c15e210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:46.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.082+0000 7f722affd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f722c15dda0 0x7f722c15e210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49748/0 (socket says 172.21.15.16:49748) 2023-05-22T15:26:46.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.082+0000 7f722affd640 1 -- 172.21.15.16:0/118629607 learned_addr learned my addr 172.21.15.16:0/118629607 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:26:46.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.086+0000 7f722b7fe640 1 --2- 172.21.15.16:0/118629607 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7214003980 0x7f722c0af370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:46.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.086+0000 7f722bfff640 1 --2- 172.21.15.16:0/118629607 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f722c0ab920 0x7f722c0abd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:46.103 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.086+0000 7f722affd640 1 -- 172.21.15.16:0/118629607 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f722c0ab920 msgr2=0x7f722c0abd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:46.103 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.086+0000 7f722affd640 1 --2- 172.21.15.16:0/118629607 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f722c0ab920 0x7f722c0abd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:46.103 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.086+0000 7f722affd640 1 -- 172.21.15.16:0/118629607 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7214003980 msgr2=0x7f722c0af370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:46.103 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.086+0000 7f722affd640 1 --2- 172.21.15.16:0/118629607 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7214003980 0x7f722c0af370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:46.103 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.086+0000 7f722affd640 1 -- 172.21.15.16:0/118629607 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f722c0698b0 con 0x7f722c15dda0 2023-05-22T15:26:46.103 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.086+0000 7f722b7fe640 1 --2- 172.21.15.16:0/118629607 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7214003980 0x7f722c0af370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:26:46.103 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.086+0000 7f722affd640 1 --2- 172.21.15.16:0/118629607 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f722c15dda0 0x7f722c15e210 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f721c00d7a0 tx=0x7f721c00dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:26:46.103 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.086+0000 7f7213fff640 1 -- 172.21.15.16:0/118629607 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f721c0074e0 con 0x7f722c15dda0 2023-05-22T15:26:46.104 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.086+0000 7f7213fff640 1 -- 172.21.15.16:0/118629607 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f721c007640 con 0x7f722c15dda0 2023-05-22T15:26:46.104 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.086+0000 7f7231eac640 1 -- 172.21.15.16:0/118629607 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f722c069720 con 0x7f722c15dda0 2023-05-22T15:26:46.104 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.086+0000 7f7213fff640 1 -- 172.21.15.16:0/118629607 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f721c004a30 con 0x7f722c15dda0 2023-05-22T15:26:46.104 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.086+0000 7f7231eac640 1 -- 172.21.15.16:0/118629607 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f722c0cec70 con 0x7f722c15dda0 2023-05-22T15:26:46.104 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.086+0000 7f7231eac640 1 -- 172.21.15.16:0/118629607 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f722c04dbe0 con 0x7f722c15dda0 2023-05-22T15:26:46.104 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.086+0000 7f7213fff640 1 -- 172.21.15.16:0/118629607 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7f721c004b90 con 0x7f722c15dda0 2023-05-22T15:26:46.104 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.086+0000 7f7213fff640 1 --2- 172.21.15.16:0/118629607 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f7218065380 0x7f7218067830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:46.104 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.086+0000 7f7213fff640 1 -- 172.21.15.16:0/118629607 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f721c081ce0 con 0x7f722c15dda0 2023-05-22T15:26:46.105 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.086+0000 7f722b7fe640 1 --2- 172.21.15.16:0/118629607 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f7218065380 0x7f7218067830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:46.105 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.086+0000 7f722b7fe640 1 --2- 172.21.15.16:0/118629607 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f7218065380 0x7f7218067830 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f72200af000 tx=0x7f72200046d0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:26:46.105 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.090+0000 7f7213fff640 1 -- 172.21.15.16:0/118629607 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f721c04b620 con 0x7f722c15dda0 2023-05-22T15:26:46.296 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.286+0000 7f7231eac640 1 -- 172.21.15.16:0/118629607 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f722c14e230 con 0x7f722c15dda0 2023-05-22T15:26:46.297 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.286+0000 7f7213fff640 1 -- 172.21.15.16:0/118629607 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) v1 ==== 93+0+3521 (secure 0 0 0) 0x7f721c019090 con 0x7f722c15dda0 2023-05-22T15:26:46.297 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:26:46.297 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4261,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6834/3661808159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":3661808159},{"type":"v1","addr":"172.21.15.62:6834","nonce":3661808159}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4264,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/2227467222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6833","nonce":2227467222},{"type":"v1","addr":"172.21.15.62:6835","nonce":2227467222}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4284,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6836/2808106087","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2808106087},{"type":"v1","addr":"172.21.15.16:6836","nonce":2808106087}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4287,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/283322289","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6835","nonce":283322289},{"type":"v1","addr":"172.21.15.16:6837","nonce":283322289}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[]} 2023-05-22T15:26:46.298 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 3 2023-05-22T15:26:46.299 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.286+0000 7f7231eac640 1 -- 172.21.15.16:0/118629607 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f7218065380 msgr2=0x7f7218067830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:46.299 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.286+0000 7f7231eac640 1 --2- 172.21.15.16:0/118629607 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f7218065380 0x7f7218067830 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f72200af000 tx=0x7f72200046d0 comp rx=0 tx=0).stop 2023-05-22T15:26:46.299 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.290+0000 7f7231eac640 1 -- 172.21.15.16:0/118629607 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f722c15dda0 msgr2=0x7f722c15e210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:46.299 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.290+0000 7f7231eac640 1 --2- 172.21.15.16:0/118629607 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f722c15dda0 0x7f722c15e210 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f721c00d7a0 tx=0x7f721c00dc60 comp rx=0 tx=0).stop 2023-05-22T15:26:46.299 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.290+0000 7f7231eac640 1 -- 172.21.15.16:0/118629607 shutdown_connections 2023-05-22T15:26:46.300 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.290+0000 7f7231eac640 1 --2- 172.21.15.16:0/118629607 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f7218065380 0x7f7218067830 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:46.300 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.290+0000 7f7231eac640 1 --2- 172.21.15.16:0/118629607 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f722c0ab920 0x7f722c0abd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:46.300 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.290+0000 7f7231eac640 1 --2- 172.21.15.16:0/118629607 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f722c15dda0 0x7f722c15e210 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:46.300 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.290+0000 7f7231eac640 1 --2- 172.21.15.16:0/118629607 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7214003980 0x7f722c0af370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:46.300 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.290+0000 7f7231eac640 1 -- 172.21.15.16:0/118629607 >> 172.21.15.16:0/118629607 conn(0x7f722c0cc870 msgr2=0x7f722c157c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:26:46.300 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.290+0000 7f7231eac640 1 -- 172.21.15.16:0/118629607 shutdown_connections 2023-05-22T15:26:46.300 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.290+0000 7f7231eac640 1 -- 172.21.15.16:0/118629607 wait complete. 2023-05-22T15:26:46.316 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-05-22T15:26:46.438 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.426+0000 7f4990fae640 1 Processor -- start 2023-05-22T15:26:46.439 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.426+0000 7f4990fae640 1 -- start start 2023-05-22T15:26:46.440 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.430+0000 7f4990fae640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f498c071500 0x7f498c15e460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:46.440 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.430+0000 7f4990fae640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f498c0b0650 con 0x7f498c0ca4e0 2023-05-22T15:26:46.440 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.430+0000 7f4990fae640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f498c0b0370 con 0x7f498c0c9b40 2023-05-22T15:26:46.440 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.430+0000 7f4990fae640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f498c15d980 con 0x7f498c071500 2023-05-22T15:26:46.440 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.430+0000 7f498a59c640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f498c0ca4e0 0x7f498c0afb40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:36938/0 (socket says 172.21.15.16:36938) 2023-05-22T15:26:46.440 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.430+0000 7f498a59c640 1 -- 172.21.15.16:0/3172878450 learned_addr learned my addr 172.21.15.16:0/3172878450 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:26:46.441 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.430+0000 7f498ad9d640 1 --2- 172.21.15.16:0/3172878450 >> v2:172.21.15.16:3300/0 conn(0x7f498c071500 0x7f498c15e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:46.441 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.430+0000 7f498ad9d640 1 -- 172.21.15.16:0/3172878450 >> v1:172.21.15.62:6789/0 conn(0x7f498c0c9b40 legacy=0x7f498c070df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:46.445 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.430+0000 7f498ad9d640 1 -- 172.21.15.16:0/3172878450 >> v1:172.21.15.16:6789/0 conn(0x7f498c0ca4e0 legacy=0x7f498c0afb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:46.445 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.430+0000 7f498ad9d640 1 -- 172.21.15.16:0/3172878450 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f498c15dc60 con 0x7f498c071500 2023-05-22T15:26:46.445 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.434+0000 7f498ad9d640 1 --2- 172.21.15.16:0/3172878450 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f498c071500 0x7f498c15e460 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f4984009d90 tx=0x7f498401c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df48a8acb954be88 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:26:46.445 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.434+0000 7f498959a640 1 -- 172.21.15.16:0/3172878450 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49840a5900 con 0x7f498c071500 2023-05-22T15:26:46.446 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.434+0000 7f498959a640 1 -- 172.21.15.16:0/3172878450 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49840a5a60 con 0x7f498c071500 2023-05-22T15:26:46.446 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.434+0000 7f498959a640 1 -- 172.21.15.16:0/3172878450 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49840a5df0 con 0x7f498c071500 2023-05-22T15:26:46.446 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.434+0000 7f4990fae640 1 -- 172.21.15.16:0/3172878450 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f498c071500 msgr2=0x7f498c15e460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:46.446 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.434+0000 7f4990fae640 1 --2- 172.21.15.16:0/3172878450 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f498c071500 0x7f498c15e460 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f4984009d90 tx=0x7f498401c920 comp rx=0 tx=0).stop 2023-05-22T15:26:46.447 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.434+0000 7f4990fae640 1 -- 172.21.15.16:0/3172878450 shutdown_connections 2023-05-22T15:26:46.447 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.434+0000 7f4990fae640 1 --2- 172.21.15.16:0/3172878450 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f498c071500 0x7f498c15e460 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:46.447 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.434+0000 7f4990fae640 1 -- 172.21.15.16:0/3172878450 >> 172.21.15.16:0/3172878450 conn(0x7f498c069720 msgr2=0x7f498c069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:26:46.447 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.434+0000 7f4990fae640 1 -- 172.21.15.16:0/3172878450 shutdown_connections 2023-05-22T15:26:46.447 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.434+0000 7f4990fae640 1 -- 172.21.15.16:0/3172878450 wait complete. 2023-05-22T15:26:46.447 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.434+0000 7f4990fae640 1 Processor -- start 2023-05-22T15:26:46.447 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.434+0000 7f4990fae640 1 -- start start 2023-05-22T15:26:46.449 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.434+0000 7f4990fae640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f498c071500 0x7f498c0c7a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:46.450 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.434+0000 7f4990fae640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f498c0c9b40 0x7f498c1519c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:46.450 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.434+0000 7f4990fae640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f498c0ca4e0 0x7f498c15c6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:46.450 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.434+0000 7f4990fae640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f498c0c1830 con 0x7f498c0ca4e0 2023-05-22T15:26:46.450 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.434+0000 7f4990fae640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f498c157190 con 0x7f498c0c9b40 2023-05-22T15:26:46.450 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.434+0000 7f4990fae640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f498c156580 con 0x7f498c071500 2023-05-22T15:26:46.450 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.434+0000 7f498ad9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f498c0ca4e0 0x7f498c15c6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:46.450 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.434+0000 7f498ad9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f498c0ca4e0 0x7f498c15c6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49764/0 (socket says 172.21.15.16:49764) 2023-05-22T15:26:46.451 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.434+0000 7f498ad9d640 1 -- 172.21.15.16:0/496308513 learned_addr learned my addr 172.21.15.16:0/496308513 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:26:46.451 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.434+0000 7f498a59c640 1 --2- 172.21.15.16:0/496308513 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f498c071500 0x7f498c0c7a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:46.451 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.434+0000 7f4989d9b640 1 --2- 172.21.15.16:0/496308513 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f498c0c9b40 0x7f498c1519c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:46.451 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.434+0000 7f498ad9d640 1 -- 172.21.15.16:0/496308513 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f498c071500 msgr2=0x7f498c0c7a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:46.451 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.434+0000 7f498ad9d640 1 --2- 172.21.15.16:0/496308513 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f498c071500 0x7f498c0c7a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:46.451 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.434+0000 7f498ad9d640 1 -- 172.21.15.16:0/496308513 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f498c0c9b40 msgr2=0x7f498c1519c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:46.451 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.434+0000 7f498ad9d640 1 --2- 172.21.15.16:0/496308513 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f498c0c9b40 0x7f498c1519c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:46.451 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.434+0000 7f498ad9d640 1 -- 172.21.15.16:0/496308513 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4984009a70 con 0x7f498c0ca4e0 2023-05-22T15:26:46.452 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.438+0000 7f498ad9d640 1 --2- 172.21.15.16:0/496308513 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f498c0ca4e0 0x7f498c15c6e0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f49840b0040 tx=0x7f49840196c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:26:46.452 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.438+0000 7f4982ffd640 1 -- 172.21.15.16:0/496308513 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49840ad8d0 con 0x7f498c0ca4e0 2023-05-22T15:26:46.452 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.438+0000 7f4982ffd640 1 -- 172.21.15.16:0/496308513 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49840ada30 con 0x7f498c0ca4e0 2023-05-22T15:26:46.452 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.438+0000 7f4990fae640 1 -- 172.21.15.16:0/496308513 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f498c0c9010 con 0x7f498c0ca4e0 2023-05-22T15:26:46.452 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.438+0000 7f4982ffd640 1 -- 172.21.15.16:0/496308513 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49840adde0 con 0x7f498c0ca4e0 2023-05-22T15:26:46.452 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.438+0000 7f4990fae640 1 -- 172.21.15.16:0/496308513 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f498c0c8e80 con 0x7f498c0ca4e0 2023-05-22T15:26:46.452 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.438+0000 7f4990fae640 1 -- 172.21.15.16:0/496308513 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f498c0c91a0 con 0x7f498c0ca4e0 2023-05-22T15:26:46.453 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.438+0000 7f4982ffd640 1 -- 172.21.15.16:0/496308513 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7f49840c4460 con 0x7f498c0ca4e0 2023-05-22T15:26:46.453 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.438+0000 7f4982ffd640 1 --2- 172.21.15.16:0/496308513 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f496c065380 0x7f496c067830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:46.453 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.438+0000 7f4982ffd640 1 -- 172.21.15.16:0/496308513 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f498411dae0 con 0x7f498c0ca4e0 2023-05-22T15:26:46.453 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.438+0000 7f498a59c640 1 --2- 172.21.15.16:0/496308513 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f496c065380 0x7f496c067830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:46.453 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.438+0000 7f498a59c640 1 --2- 172.21.15.16:0/496308513 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f496c065380 0x7f496c067830 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f497c005890 tx=0x7f497c009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:26:46.454 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.442+0000 7f4982ffd640 1 -- 172.21.15.16:0/496308513 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f49840eb9f0 con 0x7f498c0ca4e0 2023-05-22T15:26:46.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.598+0000 7f4990fae640 1 -- 172.21.15.16:0/496308513 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f498c157340 con 0x7f498c0ca4e0 2023-05-22T15:26:46.609 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.598+0000 7f4982ffd640 1 -- 172.21.15.16:0/496308513 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v19) v1 ==== 74+0+14120 (secure 0 0 0) 0x7f498c157340 con 0x7f498c0ca4e0 2023-05-22T15:26:46.609 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:26:46.610 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":19,"fsid":"b2719834-7847-4d98-90a6-f65d57f85c49","created":"2023-05-22T15:26:25.259314+0000","modified":"2023-05-22T15:26:45.959070+0000","last_up_change":"2023-05-22T15:26:36.448524+0000","last_in_change":"2023-05-22T15:26:27.675668+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-05-22T15:26:38.019781+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-05-22T15:26:43.570514+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9e7ff854-b25f-4cbb-a5c2-1799d9c53d2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":37021},{"type":"v1","addr":"172.21.15.16:6825","nonce":37021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":37021},{"type":"v1","addr":"172.21.15.16:6827","nonce":37021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":37021},{"type":"v1","addr":"172.21.15.16:6831","nonce":37021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":37021},{"type":"v1","addr":"172.21.15.16:6829","nonce":37021}]},"public_addr":"172.21.15.16:6825/37021","cluster_addr":"172.21.15.16:6827/37021","heartbeat_back_addr":"172.21.15.16:6831/37021","heartbeat_front_addr":"172.21.15.16:6829/37021","state":["exists","up"]},{"osd":1,"uuid":"4702fbc9-a927-4565-aca5-41dc69cc8264","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":37008},{"type":"v1","addr":"172.21.15.16:6801","nonce":37008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":37008},{"type":"v1","addr":"172.21.15.16:6803","nonce":37008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":37008},{"type":"v1","addr":"172.21.15.16:6807","nonce":37008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":37008},{"type":"v1","addr":"172.21.15.16:6805","nonce":37008}]},"public_addr":"172.21.15.16:6801/37008","cluster_addr":"172.21.15.16:6803/37008","heartbeat_back_addr":"172.21.15.16:6807/37008","heartbeat_front_addr":"172.21.15.16:6805/37008","state":["exists","up"]},{"osd":2,"uuid":"1e7bba46-4eba-47ef-bb81-5e2d371638ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":37009},{"type":"v1","addr":"172.21.15.16:6809","nonce":37009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":37009},{"type":"v1","addr":"172.21.15.16:6811","nonce":37009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":37009},{"type":"v1","addr":"172.21.15.16:6815","nonce":37009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":37009},{"type":"v1","addr":"172.21.15.16:6813","nonce":37009}]},"public_addr":"172.21.15.16:6809/37009","cluster_addr":"172.21.15.16:6811/37009","heartbeat_back_addr":"172.21.15.16:6815/37009","heartbeat_front_addr":"172.21.15.16:6813/37009","state":["exists","up"]},{"osd":3,"uuid":"383cacc0-5a88-4904-990e-498ca59ef48a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":37020},{"type":"v1","addr":"172.21.15.16:6817","nonce":37020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":37020},{"type":"v1","addr":"172.21.15.16:6819","nonce":37020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":37020},{"type":"v1","addr":"172.21.15.16:6823","nonce":37020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":37020},{"type":"v1","addr":"172.21.15.16:6821","nonce":37020}]},"public_addr":"172.21.15.16:6817/37020","cluster_addr":"172.21.15.16:6819/37020","heartbeat_back_addr":"172.21.15.16:6823/37020","heartbeat_front_addr":"172.21.15.16:6821/37020","state":["exists","up"]},{"osd":4,"uuid":"970500ef-95b7-42c1-b080-eed4fae7013d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":37353},{"type":"v1","addr":"172.21.15.62:6817","nonce":37353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6819","nonce":37353},{"type":"v1","addr":"172.21.15.62:6822","nonce":37353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37353},{"type":"v1","addr":"172.21.15.62:6831","nonce":37353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37353},{"type":"v1","addr":"172.21.15.62:6826","nonce":37353}]},"public_addr":"172.21.15.62:6817/37353","cluster_addr":"172.21.15.62:6822/37353","heartbeat_back_addr":"172.21.15.62:6831/37353","heartbeat_front_addr":"172.21.15.62:6826/37353","state":["exists","up"]},{"osd":5,"uuid":"022d67db-d4d8-445b-b1b9-8b446a00f5c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":37352},{"type":"v1","addr":"172.21.15.62:6801","nonce":37352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":37352},{"type":"v1","addr":"172.21.15.62:6803","nonce":37352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":37352},{"type":"v1","addr":"172.21.15.62:6807","nonce":37352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":37352},{"type":"v1","addr":"172.21.15.62:6805","nonce":37352}]},"public_addr":"172.21.15.62:6801/37352","cluster_addr":"172.21.15.62:6803/37352","heartbeat_back_addr":"172.21.15.62:6807/37352","heartbeat_front_addr":"172.21.15.62:6805/37352","state":["exists","up"]},{"osd":6,"uuid":"42bd372b-6688-4a1b-8407-e16e8c266e33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":37354},{"type":"v1","addr":"172.21.15.62:6820","nonce":37354}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":37354},{"type":"v1","addr":"172.21.15.62:6823","nonce":37354}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6829","nonce":37354},{"type":"v1","addr":"172.21.15.62:6830","nonce":37354}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6825","nonce":37354},{"type":"v1","addr":"172.21.15.62:6827","nonce":37354}]},"public_addr":"172.21.15.62:6820/37354","cluster_addr":"172.21.15.62:6823/37354","heartbeat_back_addr":"172.21.15.62:6830/37354","heartbeat_front_addr":"172.21.15.62:6827/37354","state":["exists","up"]},{"osd":7,"uuid":"ac70b2a3-f41b-4bf2-b227-7f2cbb55426d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37355},{"type":"v1","addr":"172.21.15.62:6809","nonce":37355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37355},{"type":"v1","addr":"172.21.15.62:6811","nonce":37355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":37355},{"type":"v1","addr":"172.21.15.62:6815","nonce":37355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":37355},{"type":"v1","addr":"172.21.15.62:6813","nonce":37355}]},"public_addr":"172.21.15.62:6809/37355","cluster_addr":"172.21.15.62:6811/37355","heartbeat_back_addr":"172.21.15.62:6815/37355","heartbeat_front_addr":"172.21.15.62:6813/37355","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.368896+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.107273+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.276888+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.287502+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.124791+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.251937+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.255679+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.232797+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-05-22T15:26:46.611 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.602+0000 7f4990fae640 1 -- 172.21.15.16:0/496308513 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f496c065380 msgr2=0x7f496c067830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:46.611 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.602+0000 7f4990fae640 1 --2- 172.21.15.16:0/496308513 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f496c065380 0x7f496c067830 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f497c005890 tx=0x7f497c009040 comp rx=0 tx=0).stop 2023-05-22T15:26:46.611 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.602+0000 7f4990fae640 1 -- 172.21.15.16:0/496308513 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f498c0ca4e0 msgr2=0x7f498c15c6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:46.612 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.602+0000 7f4990fae640 1 --2- 172.21.15.16:0/496308513 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f498c0ca4e0 0x7f498c15c6e0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f49840b0040 tx=0x7f49840196c0 comp rx=0 tx=0).stop 2023-05-22T15:26:46.612 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.602+0000 7f4990fae640 1 -- 172.21.15.16:0/496308513 shutdown_connections 2023-05-22T15:26:46.612 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.602+0000 7f4990fae640 1 --2- 172.21.15.16:0/496308513 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f498c0ca4e0 0x7f498c15c6e0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:46.612 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.602+0000 7f4990fae640 1 --2- 172.21.15.16:0/496308513 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f498c0c9b40 0x7f498c1519c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:46.612 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.602+0000 7f4990fae640 1 --2- 172.21.15.16:0/496308513 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f496c065380 0x7f496c067830 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:46.612 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.602+0000 7f4990fae640 1 --2- 172.21.15.16:0/496308513 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f498c071500 0x7f498c0c7a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:46.612 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.602+0000 7f4990fae640 1 -- 172.21.15.16:0/496308513 >> 172.21.15.16:0/496308513 conn(0x7f498c069720 msgr2=0x7f498c0adb00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:26:46.613 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.602+0000 7f4990fae640 1 -- 172.21.15.16:0/496308513 shutdown_connections 2023-05-22T15:26:46.613 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.602+0000 7f4990fae640 1 -- 172.21.15.16:0/496308513 wait complete. 2023-05-22T15:26:46.628 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-05-22T15:26:38.019781+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-05-22T15:26:43.570514+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-05-22T15:26:46.629 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-05-22T15:26:46.749 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.738+0000 7fa79b34a640 1 Processor -- start 2023-05-22T15:26:46.749 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.738+0000 7fa79b34a640 1 -- start start 2023-05-22T15:26:46.751 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.738+0000 7fa79b34a640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa7940d2480 0x7fa7940d28d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:46.751 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.738+0000 7fa79b34a640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7940d2e10 con 0x7fa7940b5340 2023-05-22T15:26:46.751 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.738+0000 7fa79b34a640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7940cccc0 con 0x7fa79415bca0 2023-05-22T15:26:46.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.738+0000 7fa79b34a640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7940cd660 con 0x7fa7940d2480 2023-05-22T15:26:46.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.738+0000 7fa7998e7640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fa7940b5340 0x7fa7940b57c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:36954/0 (socket says 172.21.15.16:36954) 2023-05-22T15:26:46.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.738+0000 7fa7998e7640 1 -- 172.21.15.16:0/2164985993 learned_addr learned my addr 172.21.15.16:0/2164985993 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:26:46.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.738+0000 7fa7988e5640 1 --2- 172.21.15.16:0/2164985993 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa7940d2480 0x7fa7940d28d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:46.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.738+0000 7fa787fff640 1 -- 172.21.15.16:0/2164985993 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1539964358 0 0) 0x7fa78c001120 con 0x7fa7940b5340 2023-05-22T15:26:46.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.738+0000 7fa787fff640 1 -- 172.21.15.16:0/2164985993 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa780003530 con 0x7fa7940b5340 2023-05-22T15:26:46.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.738+0000 7fa787fff640 1 -- 172.21.15.16:0/2164985993 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 683642788 0 0) 0x7fa780003530 con 0x7fa7940b5340 2023-05-22T15:26:46.753 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.738+0000 7fa787fff640 1 -- 172.21.15.16:0/2164985993 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa7940d2480 msgr2=0x7fa7940d28d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:26:46.753 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.738+0000 7fa787fff640 1 --2- 172.21.15.16:0/2164985993 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa7940d2480 0x7fa7940d28d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:46.753 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.738+0000 7fa787fff640 1 -- 172.21.15.16:0/2164985993 >> v1:172.21.15.62:6789/0 conn(0x7fa79415bca0 legacy=0x7fa7940cda80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:46.753 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.738+0000 7fa787fff640 1 -- 172.21.15.16:0/2164985993 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7940ccfa0 con 0x7fa7940b5340 2023-05-22T15:26:46.753 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.738+0000 7fa787fff640 1 -- 172.21.15.16:0/2164985993 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7fa78c001810 con 0x7fa7940b5340 2023-05-22T15:26:46.754 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.738+0000 7fa787fff640 1 -- 172.21.15.16:0/2164985993 >> v1:172.21.15.16:6789/0 conn(0x7fa7940b5340 legacy=0x7fa7940b57c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:46.754 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.738+0000 7fa787fff640 1 --2- 172.21.15.16:0/2164985993 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa7800039b0 0x7fa780003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:46.754 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.742+0000 7fa787fff640 1 --2- 172.21.15.16:0/2164985993 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa7800047d0 0x7fa780004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:46.754 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.742+0000 7fa787fff640 1 --2- 172.21.15.16:0/2164985993 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa7940d2480 0x7fa780005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:46.754 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.742+0000 7fa787fff640 1 -- 172.21.15.16:0/2164985993 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7940cd660 con 0x7fa7800047d0 2023-05-22T15:26:46.754 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.742+0000 7fa787fff640 1 -- 172.21.15.16:0/2164985993 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa78000ba40 con 0x7fa7800039b0 2023-05-22T15:26:46.755 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.742+0000 7fa787fff640 1 -- 172.21.15.16:0/2164985993 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa78000bc00 con 0x7fa7940d2480 2023-05-22T15:26:46.755 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.742+0000 7fa7988e5640 1 --2- 172.21.15.16:0/2164985993 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa7800047d0 0x7fa780004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:46.755 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.742+0000 7fa7988e5640 1 -- 172.21.15.16:0/2164985993 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa7940d2480 msgr2=0x7fa780005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:26:46.755 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.742+0000 7fa7988e5640 1 --2- 172.21.15.16:0/2164985993 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa7940d2480 0x7fa780005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:46.755 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.742+0000 7fa7988e5640 1 -- 172.21.15.16:0/2164985993 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa7800039b0 msgr2=0x7fa780003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:26:46.756 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.742+0000 7fa7988e5640 1 --2- 172.21.15.16:0/2164985993 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa7800039b0 0x7fa780003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:46.756 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.742+0000 7fa7988e5640 1 -- 172.21.15.16:0/2164985993 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa780003530 con 0x7fa7800047d0 2023-05-22T15:26:46.760 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.742+0000 7fa7988e5640 1 --2- 172.21.15.16:0/2164985993 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa7800047d0 0x7fa780004c40 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fa790009fc0 tx=0x7fa79001cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e70ecfdd5eb3ec5 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:26:46.760 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.742+0000 7fa787fff640 1 -- 172.21.15.16:0/2164985993 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7900a5cb0 con 0x7fa7800047d0 2023-05-22T15:26:46.760 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.742+0000 7fa787fff640 1 -- 172.21.15.16:0/2164985993 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa79000f040 con 0x7fa7800047d0 2023-05-22T15:26:46.761 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.742+0000 7fa787fff640 1 -- 172.21.15.16:0/2164985993 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7900ad680 con 0x7fa7800047d0 2023-05-22T15:26:46.761 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.742+0000 7fa79b34a640 1 -- 172.21.15.16:0/2164985993 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa7800047d0 msgr2=0x7fa780004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:46.761 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.742+0000 7fa79b34a640 1 --2- 172.21.15.16:0/2164985993 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa7800047d0 0x7fa780004c40 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fa790009fc0 tx=0x7fa79001cca0 comp rx=0 tx=0).stop 2023-05-22T15:26:46.761 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.742+0000 7fa7998e7640 1 -- 172.21.15.16:0/2164985993 reap_dead start 2023-05-22T15:26:46.761 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.742+0000 7fa79b34a640 1 -- 172.21.15.16:0/2164985993 shutdown_connections 2023-05-22T15:26:46.762 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.742+0000 7fa79b34a640 1 -- 172.21.15.16:0/2164985993 >> 172.21.15.16:0/2164985993 conn(0x7fa7940655c0 msgr2=0x7fa794070e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:26:46.762 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.742+0000 7fa79b34a640 1 -- 172.21.15.16:0/2164985993 shutdown_connections 2023-05-22T15:26:46.762 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.742+0000 7fa79b34a640 1 -- 172.21.15.16:0/2164985993 wait complete. 2023-05-22T15:26:46.762 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.746+0000 7fa79b34a640 1 Processor -- start 2023-05-22T15:26:46.762 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.746+0000 7fa79b34a640 1 -- start start 2023-05-22T15:26:46.762 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.746+0000 7fa79b34a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa7800047d0 0x7fa7941f0fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:46.763 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.746+0000 7fa79b34a640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa7940c7dc0 0x7fa79415cc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:46.763 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.746+0000 7fa79b34a640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa7940b2fe0 0x7fa7940b3480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:46.763 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.746+0000 7fa79b34a640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7940ce190 con 0x7fa7800047d0 2023-05-22T15:26:46.764 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.746+0000 7fa79b34a640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7940b39c0 con 0x7fa7940b2fe0 2023-05-22T15:26:46.764 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.746+0000 7fa79b34a640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7941f14f0 con 0x7fa7940c7dc0 2023-05-22T15:26:46.764 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.746+0000 7fa7990e6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa7800047d0 0x7fa7941f0fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:46.764 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.746+0000 7fa7990e6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa7800047d0 0x7fa7941f0fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49770/0 (socket says 172.21.15.16:49770) 2023-05-22T15:26:46.765 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.746+0000 7fa7990e6640 1 -- 172.21.15.16:0/451024060 learned_addr learned my addr 172.21.15.16:0/451024060 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:26:46.765 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.746+0000 7fa7988e5640 1 --2- 172.21.15.16:0/451024060 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa7940c7dc0 0x7fa79415cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:46.765 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.746+0000 7fa7998e7640 1 --2- 172.21.15.16:0/451024060 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa7940b2fe0 0x7fa7940b3480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:46.766 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.746+0000 7fa7990e6640 1 -- 172.21.15.16:0/451024060 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa7940c7dc0 msgr2=0x7fa79415cc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:46.766 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.746+0000 7fa7990e6640 1 --2- 172.21.15.16:0/451024060 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa7940c7dc0 0x7fa79415cc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:46.766 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.746+0000 7fa7990e6640 1 -- 172.21.15.16:0/451024060 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa7940b2fe0 msgr2=0x7fa7940b3480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:46.766 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.746+0000 7fa7990e6640 1 --2- 172.21.15.16:0/451024060 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa7940b2fe0 0x7fa7940b3480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:46.767 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.746+0000 7fa7990e6640 1 -- 172.21.15.16:0/451024060 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa79415c070 con 0x7fa7800047d0 2023-05-22T15:26:46.767 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.746+0000 7fa7990e6640 1 --2- 172.21.15.16:0/451024060 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa7800047d0 0x7fa7941f0fb0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fa78800c500 tx=0x7fa78800c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:26:46.767 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.746+0000 7fa785ffb640 1 -- 172.21.15.16:0/451024060 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7880052d0 con 0x7fa7800047d0 2023-05-22T15:26:46.767 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.746+0000 7fa785ffb640 1 -- 172.21.15.16:0/451024060 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa788005450 con 0x7fa7800047d0 2023-05-22T15:26:46.768 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.746+0000 7fa79b34a640 1 -- 172.21.15.16:0/451024060 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7940cd280 con 0x7fa7800047d0 2023-05-22T15:26:46.768 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.746+0000 7fa785ffb640 1 -- 172.21.15.16:0/451024060 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa788012870 con 0x7fa7800047d0 2023-05-22T15:26:46.768 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.746+0000 7fa79b34a640 1 -- 172.21.15.16:0/451024060 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7940d28d0 con 0x7fa7800047d0 2023-05-22T15:26:46.768 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.746+0000 7fa79b34a640 1 -- 172.21.15.16:0/451024060 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa79404dbe0 con 0x7fa7800047d0 2023-05-22T15:26:46.768 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.746+0000 7fa785ffb640 1 -- 172.21.15.16:0/451024060 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7fa7880129d0 con 0x7fa7800047d0 2023-05-22T15:26:46.768 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.746+0000 7fa785ffb640 1 --2- 172.21.15.16:0/451024060 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fa778065380 0x7fa778067830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:46.769 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.746+0000 7fa785ffb640 1 -- 172.21.15.16:0/451024060 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fa7880820f0 con 0x7fa7800047d0 2023-05-22T15:26:46.769 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.746+0000 7fa7988e5640 1 --2- 172.21.15.16:0/451024060 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fa778065380 0x7fa778067830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:46.769 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.750+0000 7fa7988e5640 1 --2- 172.21.15.16:0/451024060 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fa778065380 0x7fa778067830 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fa7900af000 tx=0x7fa7900a7040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:26:46.770 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.750+0000 7fa785ffb640 1 -- 172.21.15.16:0/451024060 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fa7880500b0 con 0x7fa7800047d0 2023-05-22T15:26:46.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.898+0000 7fa79b34a640 1 -- 172.21.15.16:0/451024060 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fa79415d3f0 con 0x7fa7800047d0 2023-05-22T15:26:46.911 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.898+0000 7fa785ffb640 1 -- 172.21.15.16:0/451024060 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v19) v1 ==== 93+0+10 (secure 0 0 0) 0x7fa7880536d0 con 0x7fa7800047d0 2023-05-22T15:26:46.911 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2023-05-22T15:26:46.912 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.902+0000 7fa79b34a640 1 -- 172.21.15.16:0/451024060 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fa778065380 msgr2=0x7fa778067830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:46.913 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.902+0000 7fa79b34a640 1 --2- 172.21.15.16:0/451024060 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fa778065380 0x7fa778067830 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fa7900af000 tx=0x7fa7900a7040 comp rx=0 tx=0).stop 2023-05-22T15:26:46.913 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.902+0000 7fa79b34a640 1 -- 172.21.15.16:0/451024060 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa7800047d0 msgr2=0x7fa7941f0fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:46.914 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.902+0000 7fa79b34a640 1 --2- 172.21.15.16:0/451024060 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa7800047d0 0x7fa7941f0fb0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fa78800c500 tx=0x7fa78800c9c0 comp rx=0 tx=0).stop 2023-05-22T15:26:46.914 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.902+0000 7fa79b34a640 1 -- 172.21.15.16:0/451024060 shutdown_connections 2023-05-22T15:26:46.914 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.902+0000 7fa79b34a640 1 --2- 172.21.15.16:0/451024060 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa7940b2fe0 0x7fa7940b3480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:46.915 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.902+0000 7fa79b34a640 1 --2- 172.21.15.16:0/451024060 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fa778065380 0x7fa778067830 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:46.915 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.902+0000 7fa79b34a640 1 --2- 172.21.15.16:0/451024060 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa7940c7dc0 0x7fa79415cc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:46.915 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.902+0000 7fa79b34a640 1 --2- 172.21.15.16:0/451024060 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa7800047d0 0x7fa7941f0fb0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:46.915 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.902+0000 7fa79b34a640 1 -- 172.21.15.16:0/451024060 >> 172.21.15.16:0/451024060 conn(0x7fa7940655c0 msgr2=0x7fa7940b57c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:26:46.916 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.902+0000 7fa79b34a640 1 -- 172.21.15.16:0/451024060 shutdown_connections 2023-05-22T15:26:46.916 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:46.902+0000 7fa79b34a640 1 -- 172.21.15.16:0/451024060 wait complete. 2023-05-22T15:26:46.928 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-05-22T15:26:47.047 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.038+0000 7f194d2ed640 1 Processor -- start 2023-05-22T15:26:47.048 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.038+0000 7f194d2ed640 1 -- start start 2023-05-22T15:26:47.049 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.038+0000 7f194d2ed640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f194815c8b0 0x7f19480cee30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:47.049 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.038+0000 7f194d2ed640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f194806bf60 0x7f194815cc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:47.049 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.038+0000 7f194d2ed640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f194815d1c0 0x7f194806db80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:47.049 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.038+0000 7f194d2ed640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f194815a0c0 con 0x7f194806bf60 2023-05-22T15:26:47.050 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.038+0000 7f194d2ed640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19480ce160 con 0x7f194815c8b0 2023-05-22T15:26:47.050 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.038+0000 7f194d2ed640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f194806c3e0 con 0x7f194815d1c0 2023-05-22T15:26:47.050 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.038+0000 7f19467fc640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f194806bf60 0x7f194815cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:47.050 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.038+0000 7f19467fc640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f194806bf60 0x7f194815cc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49774/0 (socket says 172.21.15.16:49774) 2023-05-22T15:26:47.050 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.038+0000 7f19467fc640 1 -- 172.21.15.16:0/4192426446 learned_addr learned my addr 172.21.15.16:0/4192426446 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:26:47.050 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.038+0000 7f19477fe640 1 --2- 172.21.15.16:0/4192426446 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f194815d1c0 0x7f194806db80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:47.050 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.038+0000 7f1946ffd640 1 --2- 172.21.15.16:0/4192426446 >> v2:172.21.15.62:3300/0 conn(0x7f194815c8b0 0x7f19480cee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:47.051 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.038+0000 7f19467fc640 1 -- 172.21.15.16:0/4192426446 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f194815d1c0 msgr2=0x7f194806db80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:26:47.052 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.038+0000 7f19467fc640 1 --2- 172.21.15.16:0/4192426446 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f194815d1c0 0x7f194806db80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:47.052 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.038+0000 7f19467fc640 1 -- 172.21.15.16:0/4192426446 >> v2:172.21.15.62:3300/0 conn(0x7f194815c8b0 msgr2=0x7f19480cee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:26:47.052 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.038+0000 7f19467fc640 1 --2- 172.21.15.16:0/4192426446 >> v2:172.21.15.62:3300/0 conn(0x7f194815c8b0 0x7f19480cee30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:47.052 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.038+0000 7f19467fc640 1 -- 172.21.15.16:0/4192426446 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f194806c640 con 0x7f194806bf60 2023-05-22T15:26:47.052 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.038+0000 7f19477fe640 1 --2- 172.21.15.16:0/4192426446 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f194815d1c0 0x7f194806db80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:26:47.052 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.038+0000 7f1946ffd640 1 --2- 172.21.15.16:0/4192426446 >> v2:172.21.15.62:3300/0 conn(0x7f194815c8b0 0x7f19480cee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:26:47.053 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.042+0000 7f19467fc640 1 --2- 172.21.15.16:0/4192426446 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f194806bf60 0x7f194815cc80 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f19480b2860 tx=0x7f193c01c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=871c6ad218e2acd6 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:26:47.054 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.042+0000 7f1945ffb640 1 -- 172.21.15.16:0/4192426446 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f193c0ac600 con 0x7f194806bf60 2023-05-22T15:26:47.054 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.042+0000 7f1945ffb640 1 -- 172.21.15.16:0/4192426446 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f193c0a8050 con 0x7f194806bf60 2023-05-22T15:26:47.054 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.042+0000 7f1945ffb640 1 -- 172.21.15.16:0/4192426446 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f193c0ac980 con 0x7f194806bf60 2023-05-22T15:26:47.054 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.042+0000 7f194d2ed640 1 -- 172.21.15.16:0/4192426446 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f194806bf60 msgr2=0x7f194815cc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:47.054 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.042+0000 7f194d2ed640 1 --2- 172.21.15.16:0/4192426446 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f194806bf60 0x7f194815cc80 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f19480b2860 tx=0x7f193c01c4d0 comp rx=0 tx=0).stop 2023-05-22T15:26:47.054 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.042+0000 7f194d2ed640 1 -- 172.21.15.16:0/4192426446 shutdown_connections 2023-05-22T15:26:47.054 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.042+0000 7f194d2ed640 1 --2- 172.21.15.16:0/4192426446 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f194806bf60 0x7f194815cc80 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:47.055 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.042+0000 7f194d2ed640 1 --2- 172.21.15.16:0/4192426446 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f194815d1c0 0x7f194806db80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:47.055 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.042+0000 7f194d2ed640 1 --2- 172.21.15.16:0/4192426446 >> v2:172.21.15.62:3300/0 conn(0x7f194815c8b0 0x7f19480cee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:47.055 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.042+0000 7f194d2ed640 1 -- 172.21.15.16:0/4192426446 >> 172.21.15.16:0/4192426446 conn(0x7f194806f5a0 msgr2=0x7f194806f9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:26:47.055 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.042+0000 7f194d2ed640 1 -- 172.21.15.16:0/4192426446 shutdown_connections 2023-05-22T15:26:47.055 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.042+0000 7f194d2ed640 1 -- 172.21.15.16:0/4192426446 wait complete. 2023-05-22T15:26:47.055 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.042+0000 7f194d2ed640 1 Processor -- start 2023-05-22T15:26:47.055 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.042+0000 7f194d2ed640 1 -- start start 2023-05-22T15:26:47.057 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.042+0000 7f194d2ed640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f194806bf60 0x7f1948151670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:47.057 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.042+0000 7f194d2ed640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f194815c8b0 0x7f19481537f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:47.058 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.042+0000 7f194d2ed640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f194815d1c0 0x7f19480cb9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:47.058 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.042+0000 7f194d2ed640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19481542b0 con 0x7f194815d1c0 2023-05-22T15:26:47.058 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.042+0000 7f194d2ed640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19480c6240 con 0x7f194806bf60 2023-05-22T15:26:47.058 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.042+0000 7f194d2ed640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19480c6520 con 0x7f194815c8b0 2023-05-22T15:26:47.058 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.042+0000 7f19477fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f194815d1c0 0x7f19480cb9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:47.058 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.042+0000 7f19477fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f194815d1c0 0x7f19480cb9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49776/0 (socket says 172.21.15.16:49776) 2023-05-22T15:26:47.058 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.042+0000 7f19477fe640 1 -- 172.21.15.16:0/4145359767 learned_addr learned my addr 172.21.15.16:0/4145359767 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:26:47.058 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.046+0000 7f19467fc640 1 --2- 172.21.15.16:0/4145359767 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f194815c8b0 0x7f19481537f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:47.059 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.046+0000 7f1946ffd640 1 --2- 172.21.15.16:0/4145359767 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f194806bf60 0x7f1948151670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:47.059 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.046+0000 7f19477fe640 1 -- 172.21.15.16:0/4145359767 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f194815c8b0 msgr2=0x7f19481537f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:47.059 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.046+0000 7f19477fe640 1 --2- 172.21.15.16:0/4145359767 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f194815c8b0 0x7f19481537f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:47.059 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.046+0000 7f19477fe640 1 -- 172.21.15.16:0/4145359767 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f194806bf60 msgr2=0x7f1948151670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:47.059 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.046+0000 7f19477fe640 1 --2- 172.21.15.16:0/4145359767 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f194806bf60 0x7f1948151670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:47.059 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.046+0000 7f19477fe640 1 -- 172.21.15.16:0/4145359767 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f193c0094d0 con 0x7f194815d1c0 2023-05-22T15:26:47.059 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.046+0000 7f1946ffd640 1 --2- 172.21.15.16:0/4145359767 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f194806bf60 0x7f1948151670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:26:47.060 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.046+0000 7f19467fc640 1 --2- 172.21.15.16:0/4145359767 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f194815c8b0 0x7f19481537f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:26:47.060 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.046+0000 7f19477fe640 1 --2- 172.21.15.16:0/4145359767 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f194815d1c0 0x7f19480cb9c0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f194000cd50 tx=0x7f194000a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:26:47.060 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.046+0000 7f192f7fe640 1 -- 172.21.15.16:0/4145359767 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f194000f680 con 0x7f194815d1c0 2023-05-22T15:26:47.060 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.046+0000 7f192f7fe640 1 -- 172.21.15.16:0/4145359767 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f194000f800 con 0x7f194815d1c0 2023-05-22T15:26:47.060 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.046+0000 7f194d2ed640 1 -- 172.21.15.16:0/4145359767 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19480ce930 con 0x7f194815d1c0 2023-05-22T15:26:47.060 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.046+0000 7f192f7fe640 1 -- 172.21.15.16:0/4145359767 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f194000fbb0 con 0x7f194815d1c0 2023-05-22T15:26:47.060 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.046+0000 7f194d2ed640 1 -- 172.21.15.16:0/4145359767 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f194806e4d0 con 0x7f194815d1c0 2023-05-22T15:26:47.060 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.046+0000 7f194d2ed640 1 -- 172.21.15.16:0/4145359767 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19480d0100 con 0x7f194815d1c0 2023-05-22T15:26:47.061 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.046+0000 7f192f7fe640 1 -- 172.21.15.16:0/4145359767 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7f194000b040 con 0x7f194815d1c0 2023-05-22T15:26:47.061 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.046+0000 7f192f7fe640 1 --2- 172.21.15.16:0/4145359767 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f1934065380 0x7f1934067830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:47.061 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.046+0000 7f192f7fe640 1 -- 172.21.15.16:0/4145359767 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f1940082f00 con 0x7f194815d1c0 2023-05-22T15:26:47.061 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.046+0000 7f1946ffd640 1 --2- 172.21.15.16:0/4145359767 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f1934065380 0x7f1934067830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:47.061 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.046+0000 7f1946ffd640 1 --2- 172.21.15.16:0/4145359767 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f1934065380 0x7f1934067830 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f19481f7f00 tx=0x7f1938008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:26:47.062 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.050+0000 7f192f7fe640 1 -- 172.21.15.16:0/4145359767 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f1940050e70 con 0x7f194815d1c0 2023-05-22T15:26:47.209 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.198+0000 7f194d2ed640 1 -- 172.21.15.16:0/4145359767 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f19480c6a40 con 0x7f194815d1c0 2023-05-22T15:26:47.210 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.198+0000 7f192f7fe640 1 -- 172.21.15.16:0/4145359767 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v19) v1 ==== 92+0+10 (secure 0 0 0) 0x7f19480c6a40 con 0x7f194815d1c0 2023-05-22T15:26:47.210 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 8 2023-05-22T15:26:47.213 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.202+0000 7f194d2ed640 1 -- 172.21.15.16:0/4145359767 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f1934065380 msgr2=0x7f1934067830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:47.213 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.202+0000 7f194d2ed640 1 --2- 172.21.15.16:0/4145359767 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f1934065380 0x7f1934067830 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f19481f7f00 tx=0x7f1938008040 comp rx=0 tx=0).stop 2023-05-22T15:26:47.214 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.202+0000 7f194d2ed640 1 -- 172.21.15.16:0/4145359767 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f194815d1c0 msgr2=0x7f19480cb9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:47.214 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.202+0000 7f194d2ed640 1 --2- 172.21.15.16:0/4145359767 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f194815d1c0 0x7f19480cb9c0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f194000cd50 tx=0x7f194000a650 comp rx=0 tx=0).stop 2023-05-22T15:26:47.214 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.202+0000 7f194d2ed640 1 -- 172.21.15.16:0/4145359767 shutdown_connections 2023-05-22T15:26:47.215 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.202+0000 7f194d2ed640 1 --2- 172.21.15.16:0/4145359767 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f194815d1c0 0x7f19480cb9c0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:47.215 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.202+0000 7f194d2ed640 1 --2- 172.21.15.16:0/4145359767 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f1934065380 0x7f1934067830 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:47.215 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.202+0000 7f194d2ed640 1 --2- 172.21.15.16:0/4145359767 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f194815c8b0 0x7f19481537f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:47.215 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.202+0000 7f194d2ed640 1 --2- 172.21.15.16:0/4145359767 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f194806bf60 0x7f1948151670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:47.216 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.202+0000 7f194d2ed640 1 -- 172.21.15.16:0/4145359767 >> 172.21.15.16:0/4145359767 conn(0x7f194806f5a0 msgr2=0x7f194806f970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:26:47.216 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.202+0000 7f194d2ed640 1 -- 172.21.15.16:0/4145359767 shutdown_connections 2023-05-22T15:26:47.217 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.202+0000 7f194d2ed640 1 -- 172.21.15.16:0/4145359767 wait complete. 2023-05-22T15:26:47.228 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-05-22T15:26:47.350 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.338+0000 7fe7da498640 1 Processor -- start 2023-05-22T15:26:47.351 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.342+0000 7fe7da498640 1 -- start start 2023-05-22T15:26:47.352 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.342+0000 7fe7da498640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe7d40b2a40 0x7fe7d40cf510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:47.352 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.342+0000 7fe7da498640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7d40ce160 con 0x7fe7d415c130 2023-05-22T15:26:47.352 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.342+0000 7fe7da498640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7d40cece0 con 0x7fe7d40cfa50 2023-05-22T15:26:47.353 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.342+0000 7fe7da498640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7d406c4a0 con 0x7fe7d40b2a40 2023-05-22T15:26:47.353 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.342+0000 7fe7d3fff640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fe7d415c130 0x7fe7d40b2330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:36968/0 (socket says 172.21.15.16:36968) 2023-05-22T15:26:47.353 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.342+0000 7fe7d3fff640 1 -- 172.21.15.16:0/280811093 learned_addr learned my addr 172.21.15.16:0/280811093 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:26:47.353 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.342+0000 7fe7d37fe640 1 --2- 172.21.15.16:0/280811093 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe7d40b2a40 0x7fe7d40cf510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:47.355 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.342+0000 7fe7d2ffd640 1 -- 172.21.15.16:0/280811093 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2756051819 0 0) 0x7fe7d40ce160 con 0x7fe7d415c130 2023-05-22T15:26:47.356 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.342+0000 7fe7d2ffd640 1 -- 172.21.15.16:0/280811093 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe7bc003530 con 0x7fe7d415c130 2023-05-22T15:26:47.356 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.342+0000 7fe7d2ffd640 1 -- 172.21.15.16:0/280811093 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1602191542 0 0) 0x7fe7bc003530 con 0x7fe7d415c130 2023-05-22T15:26:47.356 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.342+0000 7fe7d2ffd640 1 -- 172.21.15.16:0/280811093 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe7d40b2a40 msgr2=0x7fe7d40cf510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:26:47.356 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.342+0000 7fe7d2ffd640 1 --2- 172.21.15.16:0/280811093 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe7d40b2a40 0x7fe7d40cf510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:47.356 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.342+0000 7fe7d2ffd640 1 -- 172.21.15.16:0/280811093 >> v1:172.21.15.62:6789/0 conn(0x7fe7d40cfa50 legacy=0x7fe7d40ca8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:47.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.342+0000 7fe7d2ffd640 1 -- 172.21.15.16:0/280811093 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7d40cfed0 con 0x7fe7d415c130 2023-05-22T15:26:47.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.342+0000 7fe7d2ffd640 1 -- 172.21.15.16:0/280811093 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7fe7c4001680 con 0x7fe7d415c130 2023-05-22T15:26:47.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.342+0000 7fe7d2ffd640 1 -- 172.21.15.16:0/280811093 >> v1:172.21.15.16:6789/0 conn(0x7fe7d415c130 legacy=0x7fe7d40b2330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:47.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.342+0000 7fe7d2ffd640 1 --2- 172.21.15.16:0/280811093 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe7bc0039b0 0x7fe7bc003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:47.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.342+0000 7fe7d2ffd640 1 --2- 172.21.15.16:0/280811093 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe7d40b2a40 0x7fe7bc0047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:47.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.342+0000 7fe7d2ffd640 1 --2- 172.21.15.16:0/280811093 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe7bc004d10 0x7fe7bc005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:47.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.342+0000 7fe7d2ffd640 1 -- 172.21.15.16:0/280811093 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7d406c4a0 con 0x7fe7bc004d10 2023-05-22T15:26:47.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.342+0000 7fe7d2ffd640 1 -- 172.21.15.16:0/280811093 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7bc00ba40 con 0x7fe7bc0039b0 2023-05-22T15:26:47.358 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.342+0000 7fe7d2ffd640 1 -- 172.21.15.16:0/280811093 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7bc00bbd0 con 0x7fe7d40b2a40 2023-05-22T15:26:47.358 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.342+0000 7fe7d8a35640 1 --2- 172.21.15.16:0/280811093 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe7bc004d10 0x7fe7bc005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:47.358 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.342+0000 7fe7d3fff640 1 --2- 172.21.15.16:0/280811093 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe7bc0039b0 0x7fe7bc003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:47.358 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.342+0000 7fe7d37fe640 1 --2- 172.21.15.16:0/280811093 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe7d40b2a40 0x7fe7bc0047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:47.358 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.342+0000 7fe7d8a35640 1 -- 172.21.15.16:0/280811093 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe7d40b2a40 msgr2=0x7fe7bc0047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:26:47.358 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.342+0000 7fe7d8a35640 1 --2- 172.21.15.16:0/280811093 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe7d40b2a40 0x7fe7bc0047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:47.358 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.342+0000 7fe7d8a35640 1 -- 172.21.15.16:0/280811093 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe7bc0039b0 msgr2=0x7fe7bc003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:26:47.359 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.342+0000 7fe7d8a35640 1 --2- 172.21.15.16:0/280811093 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe7bc0039b0 0x7fe7bc003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:47.359 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.342+0000 7fe7d8a35640 1 -- 172.21.15.16:0/280811093 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7bc003530 con 0x7fe7bc004d10 2023-05-22T15:26:47.359 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.342+0000 7fe7d37fe640 1 --2- 172.21.15.16:0/280811093 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe7d40b2a40 0x7fe7bc0047d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:26:47.359 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.346+0000 7fe7d8a35640 1 --2- 172.21.15.16:0/280811093 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe7bc004d10 0x7fe7bc005180 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fe7cc001e70 tx=0x7fe7cc01c760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44edad698c74b93 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:26:47.359 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.346+0000 7fe7d2ffd640 1 -- 172.21.15.16:0/280811093 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7cc003b80 con 0x7fe7bc004d10 2023-05-22T15:26:47.359 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.346+0000 7fe7d2ffd640 1 -- 172.21.15.16:0/280811093 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7cc003ce0 con 0x7fe7bc004d10 2023-05-22T15:26:47.360 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.346+0000 7fe7d2ffd640 1 -- 172.21.15.16:0/280811093 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7cc0052c0 con 0x7fe7bc004d10 2023-05-22T15:26:47.362 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.346+0000 7fe7da498640 1 -- 172.21.15.16:0/280811093 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe7bc004d10 msgr2=0x7fe7bc005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:47.362 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.346+0000 7fe7da498640 1 --2- 172.21.15.16:0/280811093 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe7bc004d10 0x7fe7bc005180 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fe7cc001e70 tx=0x7fe7cc01c760 comp rx=0 tx=0).stop 2023-05-22T15:26:47.362 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.346+0000 7fe7d8a35640 1 -- 172.21.15.16:0/280811093 reap_dead start 2023-05-22T15:26:47.363 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.346+0000 7fe7da498640 1 -- 172.21.15.16:0/280811093 shutdown_connections 2023-05-22T15:26:47.363 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.346+0000 7fe7da498640 1 -- 172.21.15.16:0/280811093 >> 172.21.15.16:0/280811093 conn(0x7fe7d406f5c0 msgr2=0x7fe7d406f9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:26:47.363 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.346+0000 7fe7da498640 1 -- 172.21.15.16:0/280811093 shutdown_connections 2023-05-22T15:26:47.363 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.346+0000 7fe7da498640 1 -- 172.21.15.16:0/280811093 wait complete. 2023-05-22T15:26:47.364 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.346+0000 7fe7da498640 1 Processor -- start 2023-05-22T15:26:47.364 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.346+0000 7fe7da498640 1 -- start start 2023-05-22T15:26:47.364 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.346+0000 7fe7da498640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe7d41580f0 0x7fe7d415d080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:47.364 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.346+0000 7fe7da498640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe7d41584c0 0x7fe7d41f4770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:47.365 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.346+0000 7fe7da498640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe7d41f4cb0 0x7fe7d40b4d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:47.365 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.346+0000 7fe7da498640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7d415cc60 con 0x7fe7d41f4cb0 2023-05-22T15:26:47.365 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.346+0000 7fe7da498640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7d41f5100 con 0x7fe7d41584c0 2023-05-22T15:26:47.365 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.346+0000 7fe7da498640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7d40af660 con 0x7fe7d41580f0 2023-05-22T15:26:47.366 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.350+0000 7fe7d8a35640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe7d41f4cb0 0x7fe7d40b4d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:47.366 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.350+0000 7fe7d8a35640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe7d41f4cb0 0x7fe7d40b4d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49788/0 (socket says 172.21.15.16:49788) 2023-05-22T15:26:47.366 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.350+0000 7fe7d8a35640 1 -- 172.21.15.16:0/4060878256 learned_addr learned my addr 172.21.15.16:0/4060878256 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:26:47.367 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.350+0000 7fe7d3fff640 1 --2- 172.21.15.16:0/4060878256 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe7d41580f0 0x7fe7d415d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:47.367 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.350+0000 7fe7d37fe640 1 --2- 172.21.15.16:0/4060878256 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe7d41584c0 0x7fe7d41f4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:47.367 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.350+0000 7fe7d8a35640 1 -- 172.21.15.16:0/4060878256 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe7d41580f0 msgr2=0x7fe7d415d080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:47.368 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.350+0000 7fe7d8a35640 1 --2- 172.21.15.16:0/4060878256 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe7d41580f0 0x7fe7d415d080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:47.368 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.350+0000 7fe7d8a35640 1 -- 172.21.15.16:0/4060878256 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe7d41584c0 msgr2=0x7fe7d41f4770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:47.368 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.350+0000 7fe7d8a35640 1 --2- 172.21.15.16:0/4060878256 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe7d41584c0 0x7fe7d41f4770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:47.368 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.350+0000 7fe7d8a35640 1 -- 172.21.15.16:0/4060878256 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7d40ce690 con 0x7fe7d41f4cb0 2023-05-22T15:26:47.369 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.350+0000 7fe7d3fff640 1 --2- 172.21.15.16:0/4060878256 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe7d41580f0 0x7fe7d415d080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:26:47.369 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.350+0000 7fe7d37fe640 1 --2- 172.21.15.16:0/4060878256 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe7d41584c0 0x7fe7d41f4770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:26:47.369 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.350+0000 7fe7d8a35640 1 --2- 172.21.15.16:0/4060878256 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe7d41f4cb0 0x7fe7d40b4d70 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fe7cc01cc40 tx=0x7fe7bc005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:26:47.370 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.350+0000 7fe7d0ff9640 1 -- 172.21.15.16:0/4060878256 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7cc006d60 con 0x7fe7d41f4cb0 2023-05-22T15:26:47.370 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.350+0000 7fe7d0ff9640 1 -- 172.21.15.16:0/4060878256 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7cc007dc0 con 0x7fe7d41f4cb0 2023-05-22T15:26:47.370 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.350+0000 7fe7da498640 1 -- 172.21.15.16:0/4060878256 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7d40ce3b0 con 0x7fe7d41f4cb0 2023-05-22T15:26:47.370 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.350+0000 7fe7d0ff9640 1 -- 172.21.15.16:0/4060878256 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7cc007230 con 0x7fe7d41f4cb0 2023-05-22T15:26:47.372 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.350+0000 7fe7da498640 1 -- 172.21.15.16:0/4060878256 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7d40ce970 con 0x7fe7d41f4cb0 2023-05-22T15:26:47.372 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.350+0000 7fe7da498640 1 -- 172.21.15.16:0/4060878256 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7d404dbe0 con 0x7fe7d41f4cb0 2023-05-22T15:26:47.372 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.350+0000 7fe7d0ff9640 1 -- 172.21.15.16:0/4060878256 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7fe7cc007390 con 0x7fe7d41f4cb0 2023-05-22T15:26:47.372 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.350+0000 7fe7d0ff9640 1 --2- 172.21.15.16:0/4060878256 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fe7c00653b0 0x7fe7c0067860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:47.373 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.350+0000 7fe7d0ff9640 1 -- 172.21.15.16:0/4060878256 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fe7cc11b690 con 0x7fe7d41f4cb0 2023-05-22T15:26:47.373 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.350+0000 7fe7d3fff640 1 --2- 172.21.15.16:0/4060878256 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fe7c00653b0 0x7fe7c0067860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:47.373 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.350+0000 7fe7d3fff640 1 --2- 172.21.15.16:0/4060878256 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fe7c00653b0 0x7fe7c0067860 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fe7c4000f70 tx=0x7fe7c4000ee0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:26:47.373 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.354+0000 7fe7d0ff9640 1 -- 172.21.15.16:0/4060878256 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fe7cc0e96f0 con 0x7fe7d41f4cb0 2023-05-22T15:26:47.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.510+0000 7fe7da498640 1 -- 172.21.15.16:0/4060878256 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fe7d406e6a0 con 0x7fe7d41f4cb0 2023-05-22T15:26:47.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.514+0000 7fe7d0ff9640 1 -- 172.21.15.16:0/4060878256 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v19) v1 ==== 84+0+126 (secure 0 0 0) 0x7fe7cc0b8020 con 0x7fe7d41f4cb0 2023-05-22T15:26:47.524 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:26:47.524 INFO:teuthology.orchestra.run.smithi016.stdout:[ 2023-05-22T15:26:47.524 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T15:26:47.524 INFO:teuthology.orchestra.run.smithi016.stdout: "poolnum": 1, 2023-05-22T15:26:47.524 INFO:teuthology.orchestra.run.smithi016.stdout: "poolname": ".mgr" 2023-05-22T15:26:47.525 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:26:47.525 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T15:26:47.525 INFO:teuthology.orchestra.run.smithi016.stdout: "poolnum": 2, 2023-05-22T15:26:47.525 INFO:teuthology.orchestra.run.smithi016.stdout: "poolname": "rbd" 2023-05-22T15:26:47.525 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T15:26:47.525 INFO:teuthology.orchestra.run.smithi016.stdout:] 2023-05-22T15:26:47.526 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.514+0000 7fe7da498640 1 -- 172.21.15.16:0/4060878256 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fe7c00653b0 msgr2=0x7fe7c0067860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:47.526 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.514+0000 7fe7da498640 1 --2- 172.21.15.16:0/4060878256 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fe7c00653b0 0x7fe7c0067860 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fe7c4000f70 tx=0x7fe7c4000ee0 comp rx=0 tx=0).stop 2023-05-22T15:26:47.527 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.514+0000 7fe7da498640 1 -- 172.21.15.16:0/4060878256 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe7d41f4cb0 msgr2=0x7fe7d40b4d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:47.527 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.514+0000 7fe7da498640 1 --2- 172.21.15.16:0/4060878256 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe7d41f4cb0 0x7fe7d40b4d70 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fe7cc01cc40 tx=0x7fe7bc005800 comp rx=0 tx=0).stop 2023-05-22T15:26:47.527 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.514+0000 7fe7da498640 1 -- 172.21.15.16:0/4060878256 shutdown_connections 2023-05-22T15:26:47.527 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.514+0000 7fe7da498640 1 --2- 172.21.15.16:0/4060878256 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe7d41f4cb0 0x7fe7d40b4d70 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:47.527 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.514+0000 7fe7da498640 1 --2- 172.21.15.16:0/4060878256 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe7d41584c0 0x7fe7d41f4770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:47.527 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.514+0000 7fe7da498640 1 --2- 172.21.15.16:0/4060878256 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fe7c00653b0 0x7fe7c0067860 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:47.527 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.514+0000 7fe7da498640 1 --2- 172.21.15.16:0/4060878256 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe7d41580f0 0x7fe7d415d080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:47.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.514+0000 7fe7da498640 1 -- 172.21.15.16:0/4060878256 >> 172.21.15.16:0/4060878256 conn(0x7fe7d406f5c0 msgr2=0x7fe7d406f9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:26:47.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.514+0000 7fe7da498640 1 -- 172.21.15.16:0/4060878256 shutdown_connections 2023-05-22T15:26:47.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.514+0000 7fe7da498640 1 -- 172.21.15.16:0/4060878256 wait complete. 2023-05-22T15:26:47.543 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-05-22T15:26:47.543 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-05-22T15:26:47.675 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.666+0000 7f51ddf10640 1 Processor -- start 2023-05-22T15:26:47.676 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.666+0000 7f51ddf10640 1 -- start start 2023-05-22T15:26:47.677 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.666+0000 7f51ddf10640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f51d8070c10 0x7f51d80cc4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:47.677 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.666+0000 7f51ddf10640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f51d815d650 0x7f51d815da70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:47.677 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.666+0000 7f51ddf10640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51d815dfb0 con 0x7f51d80c9e40 2023-05-22T15:26:47.677 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.666+0000 7f51ddf10640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51d80ccbb0 con 0x7f51d8070c10 2023-05-22T15:26:47.677 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.666+0000 7f51ddf10640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51d80cd7c0 con 0x7f51d815d650 2023-05-22T15:26:47.678 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.666+0000 7f51d77fe640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f51d8070c10 0x7f51d80cc4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:47.678 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.666+0000 7f51d77fe640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f51d8070c10 0x7f51d80cc4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49792/0 (socket says 172.21.15.16:49792) 2023-05-22T15:26:47.678 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.666+0000 7f51d77fe640 1 -- 172.21.15.16:0/1070786601 learned_addr learned my addr 172.21.15.16:0/1070786601 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:26:47.678 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.666+0000 7f51d77fe640 1 -- 172.21.15.16:0/1070786601 >> v2:172.21.15.62:3300/0 conn(0x7f51d815d650 msgr2=0x7f51d815da70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:26:47.678 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.666+0000 7f51d77fe640 1 --2- 172.21.15.16:0/1070786601 >> v2:172.21.15.62:3300/0 conn(0x7f51d815d650 0x7f51d815da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:47.679 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.666+0000 7f51d77fe640 1 -- 172.21.15.16:0/1070786601 >> v1:172.21.15.16:6789/0 conn(0x7f51d80c9e40 legacy=0x7f51d806bfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:47.679 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.666+0000 7f51d77fe640 1 -- 172.21.15.16:0/1070786601 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51d80cdaa0 con 0x7f51d8070c10 2023-05-22T15:26:47.681 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.670+0000 7f51d77fe640 1 --2- 172.21.15.16:0/1070786601 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f51d8070c10 0x7f51d80cc4e0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f51d80c9c10 tx=0x7f51c801c9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=191525de927e2932 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:26:47.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.670+0000 7f51d67fc640 1 -- 172.21.15.16:0/1070786601 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51c80a5880 con 0x7f51d8070c10 2023-05-22T15:26:47.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.670+0000 7f51d67fc640 1 -- 172.21.15.16:0/1070786601 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51c80a59e0 con 0x7f51d8070c10 2023-05-22T15:26:47.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.670+0000 7f51d67fc640 1 -- 172.21.15.16:0/1070786601 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51c80a5d70 con 0x7f51d8070c10 2023-05-22T15:26:47.683 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.670+0000 7f51ddf10640 1 -- 172.21.15.16:0/1070786601 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f51d8070c10 msgr2=0x7f51d80cc4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:47.683 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.670+0000 7f51ddf10640 1 --2- 172.21.15.16:0/1070786601 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f51d8070c10 0x7f51d80cc4e0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f51d80c9c10 tx=0x7f51c801c9a0 comp rx=0 tx=0).stop 2023-05-22T15:26:47.683 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.670+0000 7f51ddf10640 1 -- 172.21.15.16:0/1070786601 shutdown_connections 2023-05-22T15:26:47.683 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.670+0000 7f51ddf10640 1 --2- 172.21.15.16:0/1070786601 >> v2:172.21.15.62:3300/0 conn(0x7f51d815d650 0x7f51d815da70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:47.683 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.670+0000 7f51ddf10640 1 --2- 172.21.15.16:0/1070786601 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f51d8070c10 0x7f51d80cc4e0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:47.683 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.670+0000 7f51ddf10640 1 -- 172.21.15.16:0/1070786601 >> 172.21.15.16:0/1070786601 conn(0x7f51d8069720 msgr2=0x7f51d8069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:26:47.683 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.670+0000 7f51ddf10640 1 -- 172.21.15.16:0/1070786601 shutdown_connections 2023-05-22T15:26:47.684 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.670+0000 7f51ddf10640 1 -- 172.21.15.16:0/1070786601 wait complete. 2023-05-22T15:26:47.684 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.670+0000 7f51ddf10640 1 Processor -- start 2023-05-22T15:26:47.684 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.670+0000 7f51ddf10640 1 -- start start 2023-05-22T15:26:47.686 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.670+0000 7f51ddf10640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f51d8070c10 0x7f51d814c750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:47.686 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.670+0000 7f51ddf10640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f51d80c9e40 0x7f51d80caf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:47.686 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.670+0000 7f51ddf10640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f51d815d650 0x7f51d80cb4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:47.686 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.670+0000 7f51ddf10640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51d814cc90 con 0x7f51d815d650 2023-05-22T15:26:47.686 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.670+0000 7f51ddf10640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51d81f8280 con 0x7f51d8070c10 2023-05-22T15:26:47.686 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.670+0000 7f51ddf10640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51d81f7670 con 0x7f51d80c9e40 2023-05-22T15:26:47.687 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.670+0000 7f51d7fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f51d815d650 0x7f51d80cb4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:47.687 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.670+0000 7f51d7fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f51d815d650 0x7f51d80cb4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49796/0 (socket says 172.21.15.16:49796) 2023-05-22T15:26:47.687 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.670+0000 7f51d7fff640 1 -- 172.21.15.16:0/3595821795 learned_addr learned my addr 172.21.15.16:0/3595821795 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:26:47.687 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.670+0000 7f51d6ffd640 1 --2- 172.21.15.16:0/3595821795 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f51d80c9e40 0x7f51d80caf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:47.687 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.670+0000 7f51d77fe640 1 --2- 172.21.15.16:0/3595821795 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f51d8070c10 0x7f51d814c750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:47.687 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.670+0000 7f51d7fff640 1 -- 172.21.15.16:0/3595821795 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f51d80c9e40 msgr2=0x7f51d80caf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:47.687 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.670+0000 7f51d7fff640 1 --2- 172.21.15.16:0/3595821795 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f51d80c9e40 0x7f51d80caf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:47.688 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.670+0000 7f51d7fff640 1 -- 172.21.15.16:0/3595821795 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f51d8070c10 msgr2=0x7f51d814c750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:47.688 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.670+0000 7f51d7fff640 1 --2- 172.21.15.16:0/3595821795 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f51d8070c10 0x7f51d814c750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:47.688 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.670+0000 7f51d7fff640 1 -- 172.21.15.16:0/3595821795 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51c8009b30 con 0x7f51d815d650 2023-05-22T15:26:47.688 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.674+0000 7f51d7fff640 1 --2- 172.21.15.16:0/3595821795 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f51d815d650 0x7f51d80cb4d0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f51d000b430 tx=0x7f51d000b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:26:47.688 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.674+0000 7f51bffff640 1 -- 172.21.15.16:0/3595821795 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51d0011830 con 0x7f51d815d650 2023-05-22T15:26:47.688 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.674+0000 7f51bffff640 1 -- 172.21.15.16:0/3595821795 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51d0011990 con 0x7f51d815d650 2023-05-22T15:26:47.689 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.674+0000 7f51ddf10640 1 -- 172.21.15.16:0/3595821795 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51d80ca240 con 0x7f51d815d650 2023-05-22T15:26:47.690 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.674+0000 7f51bffff640 1 -- 172.21.15.16:0/3595821795 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51d0011d40 con 0x7f51d815d650 2023-05-22T15:26:47.690 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.674+0000 7f51ddf10640 1 -- 172.21.15.16:0/3595821795 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51d80cce90 con 0x7f51d815d650 2023-05-22T15:26:47.690 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.674+0000 7f51ddf10640 1 -- 172.21.15.16:0/3595821795 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51d80ca3d0 con 0x7f51d815d650 2023-05-22T15:26:47.690 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.674+0000 7f51bffff640 1 -- 172.21.15.16:0/3595821795 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7f51d001a400 con 0x7f51d815d650 2023-05-22T15:26:47.690 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.674+0000 7f51bffff640 1 --2- 172.21.15.16:0/3595821795 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f51c4065380 0x7f51c4067830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:47.691 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.674+0000 7f51bffff640 1 -- 172.21.15.16:0/3595821795 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f51d00826e0 con 0x7f51d815d650 2023-05-22T15:26:47.691 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.674+0000 7f51d77fe640 1 --2- 172.21.15.16:0/3595821795 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f51c4065380 0x7f51c4067830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:47.691 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.674+0000 7f51d77fe640 1 --2- 172.21.15.16:0/3595821795 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f51c4065380 0x7f51c4067830 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f51c80ae000 tx=0x7f51c80a7070 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:26:47.691 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.678+0000 7f51bffff640 1 -- 172.21.15.16:0/3595821795 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f51d0050640 con 0x7f51d815d650 2023-05-22T15:26:47.844 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:47.834+0000 7f51ddf10640 1 -- 172.21.15.16:0/3595821795 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f51d81f83f0 con 0x7f51d815d650 2023-05-22T15:26:48.157 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:48.146+0000 7f51bffff640 1 -- 172.21.15.16:0/3595821795 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v20) v1 ==== 138+0+0 (secure 0 0 0) 0x7f51d00150b0 con 0x7f51d815d650 2023-05-22T15:26:48.159 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'cephfs_metadata' created 2023-05-22T15:26:48.162 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:48.150+0000 7f51ddf10640 1 -- 172.21.15.16:0/3595821795 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f51c4065380 msgr2=0x7f51c4067830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:48.162 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:48.150+0000 7f51ddf10640 1 --2- 172.21.15.16:0/3595821795 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f51c4065380 0x7f51c4067830 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f51c80ae000 tx=0x7f51c80a7070 comp rx=0 tx=0).stop 2023-05-22T15:26:48.162 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:48.150+0000 7f51ddf10640 1 -- 172.21.15.16:0/3595821795 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f51d815d650 msgr2=0x7f51d80cb4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:48.162 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:48.150+0000 7f51ddf10640 1 --2- 172.21.15.16:0/3595821795 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f51d815d650 0x7f51d80cb4d0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f51d000b430 tx=0x7f51d000b8f0 comp rx=0 tx=0).stop 2023-05-22T15:26:48.163 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:48.150+0000 7f51ddf10640 1 -- 172.21.15.16:0/3595821795 shutdown_connections 2023-05-22T15:26:48.163 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:48.150+0000 7f51ddf10640 1 --2- 172.21.15.16:0/3595821795 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f51d815d650 0x7f51d80cb4d0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:48.163 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:48.150+0000 7f51ddf10640 1 --2- 172.21.15.16:0/3595821795 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f51c4065380 0x7f51c4067830 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:48.164 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:48.150+0000 7f51ddf10640 1 --2- 172.21.15.16:0/3595821795 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f51d80c9e40 0x7f51d80caf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:48.164 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:48.150+0000 7f51ddf10640 1 --2- 172.21.15.16:0/3595821795 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f51d8070c10 0x7f51d814c750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:48.164 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:48.150+0000 7f51ddf10640 1 -- 172.21.15.16:0/3595821795 >> 172.21.15.16:0/3595821795 conn(0x7f51d8069720 msgr2=0x7f51d806bfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:26:48.164 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:48.150+0000 7f51ddf10640 1 -- 172.21.15.16:0/3595821795 shutdown_connections 2023-05-22T15:26:48.164 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:48.154+0000 7f51ddf10640 1 -- 172.21.15.16:0/3595821795 wait complete. 2023-05-22T15:26:48.183 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.05 2023-05-22T15:26:48.316 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:48.306+0000 7f4f4b918640 1 Processor -- start 2023-05-22T15:26:48.316 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:48.306+0000 7f4f4b918640 1 -- start start 2023-05-22T15:26:48.317 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:48.306+0000 7f4f4b918640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4f4406b6c0 0x7f4f440cb4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:48.318 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:48.306+0000 7f4f4b918640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f4f4406ba90 0x7f4f44158fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:48.318 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:48.306+0000 7f4f4b918640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f44159f70 con 0x7f4f44159510 2023-05-22T15:26:48.318 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:48.306+0000 7f4f4b918640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f44071760 con 0x7f4f4406ba90 2023-05-22T15:26:48.318 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:48.306+0000 7f4f4b918640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f440ca7d0 con 0x7f4f4406b6c0 2023-05-22T15:26:48.318 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:48.306+0000 7f4f48eb3640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f4f4406ba90 0x7f4f44158fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:48.318 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:48.306+0000 7f4f48eb3640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f4f4406ba90 0x7f4f44158fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49800/0 (socket says 172.21.15.16:49800) 2023-05-22T15:26:48.318 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:48.306+0000 7f4f48eb3640 1 -- 172.21.15.16:0/3776831465 learned_addr learned my addr 172.21.15.16:0/3776831465 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:26:48.319 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:48.306+0000 7f4f496b4640 1 --2- 172.21.15.16:0/3776831465 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4f4406b6c0 0x7f4f440cb4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:48.319 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:48.306+0000 7f4f48eb3640 1 -- 172.21.15.16:0/3776831465 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4f4406b6c0 msgr2=0x7f4f440cb4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:26:48.319 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:48.306+0000 7f4f48eb3640 1 --2- 172.21.15.16:0/3776831465 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4f4406b6c0 0x7f4f440cb4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:48.319 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:48.306+0000 7f4f48eb3640 1 -- 172.21.15.16:0/3776831465 >> v1:172.21.15.16:6789/0 conn(0x7f4f44159510 legacy=0x7f4f440cc740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:48.319 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:48.306+0000 7f4f48eb3640 1 -- 172.21.15.16:0/3776831465 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f440cb030 con 0x7f4f4406ba90 2023-05-22T15:26:48.319 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:48.306+0000 7f4f496b4640 1 --2- 172.21.15.16:0/3776831465 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4f4406b6c0 0x7f4f440cb4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:26:48.320 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:48.310+0000 7f4f48eb3640 1 --2- 172.21.15.16:0/3776831465 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4f4406ba90 0x7f4f44158fd0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f4f34009870 tx=0x7f4f3401c510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9be22787c99c70c1 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:26:48.320 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:48.310+0000 7f4f3bfff640 1 -- 172.21.15.16:0/3776831465 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f340a7070 con 0x7f4f4406ba90 2023-05-22T15:26:48.320 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:48.310+0000 7f4f3bfff640 1 -- 172.21.15.16:0/3776831465 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f340ab430 con 0x7f4f4406ba90 2023-05-22T15:26:48.320 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:48.310+0000 7f4f3bfff640 1 -- 172.21.15.16:0/3776831465 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f340ab7c0 con 0x7f4f4406ba90 2023-05-22T15:26:48.321 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:48.310+0000 7f4f4b918640 1 -- 172.21.15.16:0/3776831465 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4f4406ba90 msgr2=0x7f4f44158fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:48.321 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:48.310+0000 7f4f4b918640 1 --2- 172.21.15.16:0/3776831465 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4f4406ba90 0x7f4f44158fd0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f4f34009870 tx=0x7f4f3401c510 comp rx=0 tx=0).stop 2023-05-22T15:26:48.321 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:48.310+0000 7f4f4b918640 1 -- 172.21.15.16:0/3776831465 shutdown_connections 2023-05-22T15:26:48.322 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:48.310+0000 7f4f4b918640 1 --2- 172.21.15.16:0/3776831465 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4f4406ba90 0x7f4f44158fd0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:48.322 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:48.310+0000 7f4f4b918640 1 --2- 172.21.15.16:0/3776831465 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4f4406b6c0 0x7f4f440cb4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:48.322 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:48.310+0000 7f4f4b918640 1 -- 172.21.15.16:0/3776831465 >> 172.21.15.16:0/3776831465 conn(0x7f4f44069720 msgr2=0x7f4f44069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:26:48.322 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:48.310+0000 7f4f4b918640 1 -- 172.21.15.16:0/3776831465 shutdown_connections 2023-05-22T15:26:48.322 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:48.310+0000 7f4f4b918640 1 -- 172.21.15.16:0/3776831465 wait complete. 2023-05-22T15:26:48.323 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:48.310+0000 7f4f4b918640 1 Processor -- start 2023-05-22T15:26:48.323 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:48.310+0000 7f4f4b918640 1 -- start start 2023-05-22T15:26:48.324 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:48.314+0000 7f4f4b918640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4f4406b6c0 0x7f4f440b27c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:48.324 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:48.314+0000 7f4f4b918640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4f4406ba90 0x7f4f440c0910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:48.324 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:48.314+0000 7f4f4b918640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4f44159510 0x7f4f440c23f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:48.324 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:48.314+0000 7f4f4b918640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f440b3a90 con 0x7f4f44159510 2023-05-22T15:26:48.324 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:48.314+0000 7f4f4b918640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f440c0e50 con 0x7f4f4406ba90 2023-05-22T15:26:48.324 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:48.314+0000 7f4f4b918640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f440c2930 con 0x7f4f4406b6c0 2023-05-22T15:26:48.324 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:48.314+0000 7f4f49eb5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4f44159510 0x7f4f440c23f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:48.325 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:48.314+0000 7f4f49eb5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4f44159510 0x7f4f440c23f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49816/0 (socket says 172.21.15.16:49816) 2023-05-22T15:26:48.325 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:48.314+0000 7f4f49eb5640 1 -- 172.21.15.16:0/4124028662 learned_addr learned my addr 172.21.15.16:0/4124028662 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:26:48.325 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:48.314+0000 7f4f496b4640 1 --2- 172.21.15.16:0/4124028662 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4f4406b6c0 0x7f4f440b27c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:48.325 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:48.314+0000 7f4f48eb3640 1 --2- 172.21.15.16:0/4124028662 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4f4406ba90 0x7f4f440c0910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:48.325 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:48.314+0000 7f4f49eb5640 1 -- 172.21.15.16:0/4124028662 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4f4406b6c0 msgr2=0x7f4f440b27c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:48.325 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:48.314+0000 7f4f49eb5640 1 --2- 172.21.15.16:0/4124028662 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4f4406b6c0 0x7f4f440b27c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:48.326 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:48.314+0000 7f4f49eb5640 1 -- 172.21.15.16:0/4124028662 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4f4406ba90 msgr2=0x7f4f440c0910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:48.326 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:48.314+0000 7f4f49eb5640 1 --2- 172.21.15.16:0/4124028662 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4f4406ba90 0x7f4f440c0910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:48.326 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:48.314+0000 7f4f49eb5640 1 -- 172.21.15.16:0/4124028662 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f340094d0 con 0x7f4f44159510 2023-05-22T15:26:48.326 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:48.314+0000 7f4f49eb5640 1 --2- 172.21.15.16:0/4124028662 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4f44159510 0x7f4f440c23f0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f4f3c00c730 tx=0x7f4f3c00cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:26:48.328 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:48.314+0000 7f4f39ffb640 1 -- 172.21.15.16:0/4124028662 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f3c003b80 con 0x7f4f44159510 2023-05-22T15:26:48.328 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:48.314+0000 7f4f39ffb640 1 -- 172.21.15.16:0/4124028662 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f3c003ce0 con 0x7f4f44159510 2023-05-22T15:26:48.328 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:48.314+0000 7f4f4b918640 1 -- 172.21.15.16:0/4124028662 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f44159910 con 0x7f4f44159510 2023-05-22T15:26:48.328 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:48.314+0000 7f4f39ffb640 1 -- 172.21.15.16:0/4124028662 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f3c00aa30 con 0x7f4f44159510 2023-05-22T15:26:48.329 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:48.314+0000 7f4f4b918640 1 -- 172.21.15.16:0/4124028662 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f440cacc0 con 0x7f4f44159510 2023-05-22T15:26:48.329 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:48.314+0000 7f4f4b918640 1 -- 172.21.15.16:0/4124028662 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f4406bee0 con 0x7f4f44159510 2023-05-22T15:26:48.329 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:48.314+0000 7f4f39ffb640 1 -- 172.21.15.16:0/4124028662 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7f4f3c00ab90 con 0x7f4f44159510 2023-05-22T15:26:48.329 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:48.314+0000 7f4f39ffb640 1 --2- 172.21.15.16:0/4124028662 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f4f28065380 0x7f4f28067830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:48.330 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:48.314+0000 7f4f39ffb640 1 -- 172.21.15.16:0/4124028662 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5580+0+0 (secure 0 0 0) 0x7f4f3c082430 con 0x7f4f44159510 2023-05-22T15:26:48.330 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:48.314+0000 7f4f496b4640 1 --2- 172.21.15.16:0/4124028662 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f4f28065380 0x7f4f28067830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:48.330 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:48.318+0000 7f4f496b4640 1 --2- 172.21.15.16:0/4124028662 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f4f28065380 0x7f4f28067830 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f4f40006fd0 tx=0x7f4f40008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:26:48.331 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:48.322+0000 7f4f39ffb640 1 -- 172.21.15.16:0/4124028662 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f4f3c050290 con 0x7f4f44159510 2023-05-22T15:26:48.486 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:48.474+0000 7f4f4b918640 1 -- 172.21.15.16:0/4124028662 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05} v 0) v1 -- 0x7f4f4404dbe0 con 0x7f4f44159510 2023-05-22T15:26:49.162 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:49.150+0000 7f4f39ffb640 1 -- 172.21.15.16:0/4124028662 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]=0 pool 'cephfs_data' created v21) v1 ==== 171+0+0 (secure 0 0 0) 0x7f4f3c0538b0 con 0x7f4f44159510 2023-05-22T15:26:49.163 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'cephfs_data' created 2023-05-22T15:26:49.169 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:49.158+0000 7f4f4b918640 1 -- 172.21.15.16:0/4124028662 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f4f28065380 msgr2=0x7f4f28067830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:49.169 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:49.158+0000 7f4f4b918640 1 --2- 172.21.15.16:0/4124028662 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f4f28065380 0x7f4f28067830 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f4f40006fd0 tx=0x7f4f40008040 comp rx=0 tx=0).stop 2023-05-22T15:26:49.170 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:49.158+0000 7f4f4b918640 1 -- 172.21.15.16:0/4124028662 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4f44159510 msgr2=0x7f4f440c23f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:49.170 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:49.158+0000 7f4f4b918640 1 --2- 172.21.15.16:0/4124028662 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4f44159510 0x7f4f440c23f0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f4f3c00c730 tx=0x7f4f3c00cbf0 comp rx=0 tx=0).stop 2023-05-22T15:26:49.188 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:49.178+0000 7f4f4b918640 1 -- 172.21.15.16:0/4124028662 shutdown_connections 2023-05-22T15:26:49.188 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:49.178+0000 7f4f4b918640 1 --2- 172.21.15.16:0/4124028662 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4f44159510 0x7f4f440c23f0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:49.188 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:49.178+0000 7f4f4b918640 1 --2- 172.21.15.16:0/4124028662 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4f4406ba90 0x7f4f440c0910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:49.188 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:49.178+0000 7f4f4b918640 1 --2- 172.21.15.16:0/4124028662 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f4f28065380 0x7f4f28067830 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:49.189 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:49.178+0000 7f4f4b918640 1 --2- 172.21.15.16:0/4124028662 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4f4406b6c0 0x7f4f440b27c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:49.189 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:49.178+0000 7f4f4b918640 1 -- 172.21.15.16:0/4124028662 >> 172.21.15.16:0/4124028662 conn(0x7f4f44069720 msgr2=0x7f4f440c94f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:26:49.190 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:49.182+0000 7f4f4b918640 1 -- 172.21.15.16:0/4124028662 shutdown_connections 2023-05-22T15:26:49.192 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:49.182+0000 7f4f4b918640 1 -- 172.21.15.16:0/4124028662 wait complete. 2023-05-22T15:26:49.217 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-05-22T15:26:49.346 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:49.334+0000 7f4b3ac8a640 1 Processor -- start 2023-05-22T15:26:49.347 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:49.334+0000 7f4b3ac8a640 1 -- start start 2023-05-22T15:26:49.350 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:49.338+0000 7f4b3ac8a640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4b340cb4c0 0x7f4b34158fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:49.350 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:49.338+0000 7f4b3ac8a640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f4b34159510 0x7f4b3406fba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:49.350 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:49.338+0000 7f4b3ac8a640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b34159f70 con 0x7f4b3406b6c0 2023-05-22T15:26:49.350 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:49.338+0000 7f4b3ac8a640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b34071760 con 0x7f4b34159510 2023-05-22T15:26:49.351 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:49.338+0000 7f4b3ac8a640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b340cb910 con 0x7f4b340cb4c0 2023-05-22T15:26:49.351 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:49.338+0000 7f4b38a26640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f4b3406b6c0 0x7f4b3406ba90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:43434/0 (socket says 172.21.15.16:43434) 2023-05-22T15:26:49.351 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:49.338+0000 7f4b38a26640 1 -- 172.21.15.16:0/2431793214 learned_addr learned my addr 172.21.15.16:0/2431793214 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:26:49.351 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:49.338+0000 7f4b337fe640 1 -- 172.21.15.16:0/2431793214 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1803973061 0 0) 0x7f4b34159f70 con 0x7f4b3406b6c0 2023-05-22T15:26:49.351 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:49.338+0000 7f4b337fe640 1 -- 172.21.15.16:0/2431793214 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b1c003530 con 0x7f4b3406b6c0 2023-05-22T15:26:49.351 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:49.338+0000 7f4b39227640 1 --2- 172.21.15.16:0/2431793214 >> v2:172.21.15.62:3300/0 conn(0x7f4b34159510 0x7f4b3406fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:49.352 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:49.338+0000 7f4b33fff640 1 --2- 172.21.15.16:0/2431793214 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4b340cb4c0 0x7f4b34158fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:49.352 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:49.338+0000 7f4b337fe640 1 -- 172.21.15.16:0/2431793214 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3443763965 0 0) 0x7f4b1c003530 con 0x7f4b3406b6c0 2023-05-22T15:26:49.352 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:49.338+0000 7f4b337fe640 1 -- 172.21.15.16:0/2431793214 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4b340cb4c0 msgr2=0x7f4b34158fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:26:49.352 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:49.338+0000 7f4b337fe640 1 --2- 172.21.15.16:0/2431793214 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4b340cb4c0 0x7f4b34158fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:49.352 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:49.338+0000 7f4b337fe640 1 -- 172.21.15.16:0/2431793214 >> v2:172.21.15.62:3300/0 conn(0x7f4b34159510 msgr2=0x7f4b3406fba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:26:49.353 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:49.338+0000 7f4b337fe640 1 --2- 172.21.15.16:0/2431793214 >> v2:172.21.15.62:3300/0 conn(0x7f4b34159510 0x7f4b3406fba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:49.353 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:49.338+0000 7f4b337fe640 1 -- 172.21.15.16:0/2431793214 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b340cb030 con 0x7f4b3406b6c0 2023-05-22T15:26:49.353 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:49.338+0000 7f4b337fe640 1 -- 172.21.15.16:0/2431793214 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7f4b24001680 con 0x7f4b3406b6c0 2023-05-22T15:26:49.353 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:49.338+0000 7f4b337fe640 1 -- 172.21.15.16:0/2431793214 >> v1:172.21.15.16:6789/0 conn(0x7f4b3406b6c0 legacy=0x7f4b3406ba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:49.353 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:49.338+0000 7f4b337fe640 1 --2- 172.21.15.16:0/2431793214 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b1c003980 0x7f4b1c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:49.353 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:49.338+0000 7f4b337fe640 1 --2- 172.21.15.16:0/2431793214 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4b340cb4c0 0x7f4b1c0047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:49.353 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:49.338+0000 7f4b337fe640 1 --2- 172.21.15.16:0/2431793214 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4b1c004d10 0x7f4b1c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:49.354 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:49.338+0000 7f4b337fe640 1 -- 172.21.15.16:0/2431793214 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b34071760 con 0x7f4b1c003980 2023-05-22T15:26:49.354 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:49.338+0000 7f4b337fe640 1 -- 172.21.15.16:0/2431793214 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b340cb910 con 0x7f4b1c004d10 2023-05-22T15:26:49.354 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:49.338+0000 7f4b337fe640 1 -- 172.21.15.16:0/2431793214 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b1c00ba40 con 0x7f4b340cb4c0 2023-05-22T15:26:49.354 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:49.338+0000 7f4b38a26640 1 --2- 172.21.15.16:0/2431793214 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b1c003980 0x7f4b1c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:49.354 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:49.338+0000 7f4b39227640 1 --2- 172.21.15.16:0/2431793214 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4b1c004d10 0x7f4b1c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:49.354 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:49.338+0000 7f4b33fff640 1 --2- 172.21.15.16:0/2431793214 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4b340cb4c0 0x7f4b1c0047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:49.355 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:49.338+0000 7f4b38a26640 1 -- 172.21.15.16:0/2431793214 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4b340cb4c0 msgr2=0x7f4b1c0047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:26:49.355 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:49.338+0000 7f4b38a26640 1 --2- 172.21.15.16:0/2431793214 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4b340cb4c0 0x7f4b1c0047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:49.355 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:49.338+0000 7f4b38a26640 1 -- 172.21.15.16:0/2431793214 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4b1c004d10 msgr2=0x7f4b1c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:26:49.355 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:49.338+0000 7f4b38a26640 1 --2- 172.21.15.16:0/2431793214 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4b1c004d10 0x7f4b1c005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:49.355 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:49.338+0000 7f4b38a26640 1 -- 172.21.15.16:0/2431793214 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b1c003530 con 0x7f4b1c003980 2023-05-22T15:26:49.356 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:49.338+0000 7f4b39227640 1 --2- 172.21.15.16:0/2431793214 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4b1c004d10 0x7f4b1c005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:26:49.356 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:49.338+0000 7f4b33fff640 1 --2- 172.21.15.16:0/2431793214 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4b340cb4c0 0x7f4b1c0047d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:26:49.359 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:49.342+0000 7f4b38a26640 1 --2- 172.21.15.16:0/2431793214 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b1c003980 0x7f4b1c003d90 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f4b2400c210 tx=0x7f4b2401cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3eac7e78e2953d90 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:26:49.359 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:49.342+0000 7f4b337fe640 1 -- 172.21.15.16:0/2431793214 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b240058e0 con 0x7f4b1c003980 2023-05-22T15:26:49.360 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:49.342+0000 7f4b337fe640 1 -- 172.21.15.16:0/2431793214 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b24005a40 con 0x7f4b1c003980 2023-05-22T15:26:49.360 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:49.342+0000 7f4b337fe640 1 -- 172.21.15.16:0/2431793214 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b24005e10 con 0x7f4b1c003980 2023-05-22T15:26:49.360 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:49.342+0000 7f4b3ac8a640 1 -- 172.21.15.16:0/2431793214 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b1c003980 msgr2=0x7f4b1c003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:49.360 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:49.342+0000 7f4b3ac8a640 1 --2- 172.21.15.16:0/2431793214 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b1c003980 0x7f4b1c003d90 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f4b2400c210 tx=0x7f4b2401cc60 comp rx=0 tx=0).stop 2023-05-22T15:26:49.360 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:49.342+0000 7f4b39227640 1 -- 172.21.15.16:0/2431793214 reap_dead start 2023-05-22T15:26:49.361 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:49.342+0000 7f4b3ac8a640 1 -- 172.21.15.16:0/2431793214 shutdown_connections 2023-05-22T15:26:49.361 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:49.342+0000 7f4b3ac8a640 1 -- 172.21.15.16:0/2431793214 >> 172.21.15.16:0/2431793214 conn(0x7f4b34069720 msgr2=0x7f4b34069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:26:49.361 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:49.342+0000 7f4b3ac8a640 1 -- 172.21.15.16:0/2431793214 shutdown_connections 2023-05-22T15:26:49.361 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:49.342+0000 7f4b3ac8a640 1 -- 172.21.15.16:0/2431793214 wait complete. 2023-05-22T15:26:49.362 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:49.342+0000 7f4b3ac8a640 1 Processor -- start 2023-05-22T15:26:49.362 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:49.342+0000 7f4b3ac8a640 1 -- start start 2023-05-22T15:26:49.362 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:49.342+0000 7f4b3ac8a640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4b1c003980 0x7f4b341f43f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:49.362 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:49.342+0000 7f4b3ac8a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b34155f30 0x7f4b340c43d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:49.362 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:49.342+0000 7f4b3ac8a640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4b340b3480 0x7f4b3415b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:49.362 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:49.342+0000 7f4b3ac8a640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b340c1df0 con 0x7f4b34155f30 2023-05-22T15:26:49.363 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:49.342+0000 7f4b3ac8a640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b341f4930 con 0x7f4b1c003980 2023-05-22T15:26:49.363 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:49.342+0000 7f4b3ac8a640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b340b2ff0 con 0x7f4b340b3480 2023-05-22T15:26:49.363 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:49.342+0000 7f4b33fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b34155f30 0x7f4b340c43d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:49.363 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:49.342+0000 7f4b33fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b34155f30 0x7f4b340c43d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49382/0 (socket says 172.21.15.16:49382) 2023-05-22T15:26:49.363 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:49.342+0000 7f4b33fff640 1 -- 172.21.15.16:0/3670799929 learned_addr learned my addr 172.21.15.16:0/3670799929 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:26:49.363 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:49.346+0000 7f4b38a26640 1 --2- 172.21.15.16:0/3670799929 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4b1c003980 0x7f4b341f43f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:49.363 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:49.346+0000 7f4b39227640 1 --2- 172.21.15.16:0/3670799929 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4b340b3480 0x7f4b3415b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:49.364 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:49.346+0000 7f4b33fff640 1 -- 172.21.15.16:0/3670799929 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4b340b3480 msgr2=0x7f4b3415b1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:49.364 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:49.346+0000 7f4b33fff640 1 --2- 172.21.15.16:0/3670799929 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4b340b3480 0x7f4b3415b1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:49.364 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:49.346+0000 7f4b33fff640 1 -- 172.21.15.16:0/3670799929 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4b1c003980 msgr2=0x7f4b341f43f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:49.364 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:49.346+0000 7f4b33fff640 1 --2- 172.21.15.16:0/3670799929 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4b1c003980 0x7f4b341f43f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:49.364 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:49.346+0000 7f4b33fff640 1 -- 172.21.15.16:0/3670799929 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b34159910 con 0x7f4b34155f30 2023-05-22T15:26:49.365 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:49.346+0000 7f4b39227640 1 --2- 172.21.15.16:0/3670799929 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4b340b3480 0x7f4b3415b1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:26:49.365 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:49.346+0000 7f4b33fff640 1 --2- 172.21.15.16:0/3670799929 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b34155f30 0x7f4b340c43d0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f4b2800e840 tx=0x7f4b2800ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:26:49.365 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:49.346+0000 7f4b317fa640 1 -- 172.21.15.16:0/3670799929 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b2800c6d0 con 0x7f4b34155f30 2023-05-22T15:26:49.365 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:49.346+0000 7f4b317fa640 1 -- 172.21.15.16:0/3670799929 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b2800c830 con 0x7f4b34155f30 2023-05-22T15:26:49.365 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:49.346+0000 7f4b3ac8a640 1 -- 172.21.15.16:0/3670799929 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b340cacc0 con 0x7f4b34155f30 2023-05-22T15:26:49.365 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:49.346+0000 7f4b317fa640 1 -- 172.21.15.16:0/3670799929 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b2800cba0 con 0x7f4b34155f30 2023-05-22T15:26:49.365 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:49.346+0000 7f4b3ac8a640 1 -- 172.21.15.16:0/3670799929 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b34156840 con 0x7f4b34155f30 2023-05-22T15:26:49.366 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:49.346+0000 7f4b3ac8a640 1 -- 172.21.15.16:0/3670799929 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b3404dbe0 con 0x7f4b34155f30 2023-05-22T15:26:49.366 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:49.346+0000 7f4b317fa640 1 -- 172.21.15.16:0/3670799929 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7f4b2800cd00 con 0x7f4b34155f30 2023-05-22T15:26:49.366 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:49.346+0000 7f4b317fa640 1 --2- 172.21.15.16:0/3670799929 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f4b200653b0 0x7f4b20067860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:49.366 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:49.346+0000 7f4b317fa640 1 -- 172.21.15.16:0/3670799929 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5962+0+0 (secure 0 0 0) 0x7f4b28082650 con 0x7f4b34155f30 2023-05-22T15:26:49.366 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:49.346+0000 7f4b38a26640 1 --2- 172.21.15.16:0/3670799929 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f4b200653b0 0x7f4b20067860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:49.366 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:49.346+0000 7f4b38a26640 1 --2- 172.21.15.16:0/3670799929 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f4b200653b0 0x7f4b20067860 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f4b2400c210 tx=0x7f4b24001a10 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:26:49.367 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:49.350+0000 7f4b317fa640 1 -- 172.21.15.16:0/3670799929 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f4b280503c0 con 0x7f4b34155f30 2023-05-22T15:26:49.549 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:49.538+0000 7f4b3ac8a640 1 -- 172.21.15.16:0/3670799929 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f4b3414f980 con 0x7f4b34155f30 2023-05-22T15:26:49.569 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:49.558+0000 7f4b317fa640 1 -- 172.21.15.16:0/3670799929 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 3 and data pool 4 v4) v1 ==== 172+0+0 (secure 0 0 0) 0x7f4b280539e0 con 0x7f4b34155f30 2023-05-22T15:26:49.573 INFO:teuthology.orchestra.run.smithi016.stderr:new fs with metadata pool 3 and data pool 4 2023-05-22T15:26:49.574 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:49.562+0000 7f4b3ac8a640 1 -- 172.21.15.16:0/3670799929 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f4b200653b0 msgr2=0x7f4b20067860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:49.575 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:49.562+0000 7f4b3ac8a640 1 --2- 172.21.15.16:0/3670799929 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f4b200653b0 0x7f4b20067860 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f4b2400c210 tx=0x7f4b24001a10 comp rx=0 tx=0).stop 2023-05-22T15:26:49.575 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:49.562+0000 7f4b3ac8a640 1 -- 172.21.15.16:0/3670799929 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b34155f30 msgr2=0x7f4b340c43d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:49.576 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:49.562+0000 7f4b3ac8a640 1 --2- 172.21.15.16:0/3670799929 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b34155f30 0x7f4b340c43d0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f4b2800e840 tx=0x7f4b2800ed00 comp rx=0 tx=0).stop 2023-05-22T15:26:49.579 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:49.566+0000 7f4b3ac8a640 1 -- 172.21.15.16:0/3670799929 shutdown_connections 2023-05-22T15:26:49.579 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:49.566+0000 7f4b3ac8a640 1 --2- 172.21.15.16:0/3670799929 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f4b200653b0 0x7f4b20067860 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:49.579 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:49.566+0000 7f4b3ac8a640 1 --2- 172.21.15.16:0/3670799929 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4b340b3480 0x7f4b3415b1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:49.580 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:49.566+0000 7f4b3ac8a640 1 --2- 172.21.15.16:0/3670799929 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b34155f30 0x7f4b340c43d0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:49.580 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:49.566+0000 7f4b3ac8a640 1 --2- 172.21.15.16:0/3670799929 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4b1c003980 0x7f4b341f43f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:49.580 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:49.566+0000 7f4b3ac8a640 1 -- 172.21.15.16:0/3670799929 >> 172.21.15.16:0/3670799929 conn(0x7f4b34069720 msgr2=0x7f4b34156b50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:26:49.583 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:49.574+0000 7f4b3ac8a640 1 -- 172.21.15.16:0/3670799929 shutdown_connections 2023-05-22T15:26:49.584 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:49.574+0000 7f4b3ac8a640 1 -- 172.21.15.16:0/3670799929 wait complete. 2023-05-22T15:26:49.611 DEBUG:tasks.cephfs.filesystem:EC profile is ['m=2', 'k=2', 'crush-failure-domain=osd'] 2023-05-22T15:26:49.612 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile set cephfs_data_ec m=2 k=2 crush-failure-domain=osd 2023-05-22T15:26:49.737 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:49.726+0000 7ff675b9b640 1 Processor -- start 2023-05-22T15:26:49.737 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:49.726+0000 7ff675b9b640 1 -- start start 2023-05-22T15:26:49.738 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:49.726+0000 7ff675b9b640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7ff670070c10 0x7ff6700cc4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:49.739 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:49.726+0000 7ff675b9b640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff67015d650 0x7ff67015da70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:49.739 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:49.726+0000 7ff675b9b640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff67015dfb0 con 0x7ff6700c9e40 2023-05-22T15:26:49.739 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:49.726+0000 7ff675b9b640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6700ccbb0 con 0x7ff67015d650 2023-05-22T15:26:49.739 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:49.726+0000 7ff675b9b640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6700cd7c0 con 0x7ff670070c10 2023-05-22T15:26:49.739 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:49.730+0000 7ff66ffff640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff67015d650 0x7ff67015da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:49.739 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:49.730+0000 7ff66ffff640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff67015d650 0x7ff67015da70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49396/0 (socket says 172.21.15.16:49396) 2023-05-22T15:26:49.740 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:49.730+0000 7ff66ffff640 1 -- 172.21.15.16:0/4011918084 learned_addr learned my addr 172.21.15.16:0/4011918084 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:26:49.740 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:49.730+0000 7ff66f7fe640 1 --2- 172.21.15.16:0/4011918084 >> v2:172.21.15.62:3300/0 conn(0x7ff670070c10 0x7ff6700cc4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:49.740 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:49.730+0000 7ff66ffff640 1 -- 172.21.15.16:0/4011918084 >> v2:172.21.15.62:3300/0 conn(0x7ff670070c10 msgr2=0x7ff6700cc4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:26:49.741 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:49.730+0000 7ff66ffff640 1 --2- 172.21.15.16:0/4011918084 >> v2:172.21.15.62:3300/0 conn(0x7ff670070c10 0x7ff6700cc4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:49.741 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:49.730+0000 7ff66ffff640 1 -- 172.21.15.16:0/4011918084 >> v1:172.21.15.62:6789/0 conn(0x7ff6700c9e40 legacy=0x7ff67006bfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:49.741 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:49.730+0000 7ff66ffff640 1 -- 172.21.15.16:0/4011918084 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6700cdaa0 con 0x7ff67015d650 2023-05-22T15:26:49.742 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:49.730+0000 7ff66ffff640 1 --2- 172.21.15.16:0/4011918084 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff67015d650 0x7ff67015da70 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7ff6700cf110 tx=0x7ff66801c9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a37e305ad53afe8d server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:26:49.742 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:49.730+0000 7ff66e7fc640 1 -- 172.21.15.16:0/4011918084 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6680a5880 con 0x7ff67015d650 2023-05-22T15:26:49.742 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:49.730+0000 7ff66e7fc640 1 -- 172.21.15.16:0/4011918084 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6680a59e0 con 0x7ff67015d650 2023-05-22T15:26:49.742 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:49.730+0000 7ff66e7fc640 1 -- 172.21.15.16:0/4011918084 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6680a5d70 con 0x7ff67015d650 2023-05-22T15:26:49.742 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:49.730+0000 7ff675b9b640 1 -- 172.21.15.16:0/4011918084 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff67015d650 msgr2=0x7ff67015da70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:49.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:49.730+0000 7ff675b9b640 1 --2- 172.21.15.16:0/4011918084 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff67015d650 0x7ff67015da70 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7ff6700cf110 tx=0x7ff66801c9a0 comp rx=0 tx=0).stop 2023-05-22T15:26:49.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:49.730+0000 7ff675b9b640 1 -- 172.21.15.16:0/4011918084 shutdown_connections 2023-05-22T15:26:49.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:49.730+0000 7ff675b9b640 1 --2- 172.21.15.16:0/4011918084 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff67015d650 0x7ff67015da70 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:49.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:49.730+0000 7ff675b9b640 1 --2- 172.21.15.16:0/4011918084 >> v2:172.21.15.62:3300/0 conn(0x7ff670070c10 0x7ff6700cc4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:49.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:49.730+0000 7ff675b9b640 1 -- 172.21.15.16:0/4011918084 >> 172.21.15.16:0/4011918084 conn(0x7ff670069720 msgr2=0x7ff670069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:26:49.744 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:49.730+0000 7ff675b9b640 1 -- 172.21.15.16:0/4011918084 shutdown_connections 2023-05-22T15:26:49.744 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:49.730+0000 7ff675b9b640 1 -- 172.21.15.16:0/4011918084 wait complete. 2023-05-22T15:26:49.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:49.734+0000 7ff675b9b640 1 Processor -- start 2023-05-22T15:26:49.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:49.734+0000 7ff675b9b640 1 -- start start 2023-05-22T15:26:49.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:49.734+0000 7ff675b9b640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff670070c10 0x7ff6701508a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:49.746 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:49.734+0000 7ff675b9b640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff6700c9e40 0x7ff670145c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:49.746 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:49.734+0000 7ff675b9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff67015d650 0x7ff67014af20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:49.746 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:49.734+0000 7ff675b9b640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff67015d1d0 con 0x7ff67015d650 2023-05-22T15:26:49.746 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:49.734+0000 7ff675b9b640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff67015b2a0 con 0x7ff6700c9e40 2023-05-22T15:26:49.746 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:49.734+0000 7ff675b9b640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6700c5d80 con 0x7ff670070c10 2023-05-22T15:26:49.746 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:49.734+0000 7ff66ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff67015d650 0x7ff67014af20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:49.746 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:49.734+0000 7ff66ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff67015d650 0x7ff67014af20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49412/0 (socket says 172.21.15.16:49412) 2023-05-22T15:26:49.747 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:49.734+0000 7ff66ffff640 1 -- 172.21.15.16:0/1022553221 learned_addr learned my addr 172.21.15.16:0/1022553221 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:26:49.747 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:49.734+0000 7ff66f7fe640 1 --2- 172.21.15.16:0/1022553221 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff670070c10 0x7ff6701508a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:49.747 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:49.734+0000 7ff66effd640 1 --2- 172.21.15.16:0/1022553221 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff6700c9e40 0x7ff670145c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:49.747 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:49.734+0000 7ff66ffff640 1 -- 172.21.15.16:0/1022553221 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff670070c10 msgr2=0x7ff6701508a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:49.747 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:49.734+0000 7ff66ffff640 1 --2- 172.21.15.16:0/1022553221 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff670070c10 0x7ff6701508a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:49.747 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:49.734+0000 7ff66ffff640 1 -- 172.21.15.16:0/1022553221 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff6700c9e40 msgr2=0x7ff670145c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:49.747 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:49.734+0000 7ff66ffff640 1 --2- 172.21.15.16:0/1022553221 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff6700c9e40 0x7ff670145c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:49.748 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:49.734+0000 7ff66ffff640 1 -- 172.21.15.16:0/1022553221 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff668009b30 con 0x7ff67015d650 2023-05-22T15:26:49.748 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:49.734+0000 7ff66ffff640 1 --2- 172.21.15.16:0/1022553221 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff67015d650 0x7ff67014af20 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7ff6680195b0 tx=0x7ff668019740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:26:49.748 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:49.734+0000 7ff657fff640 1 -- 172.21.15.16:0/1022553221 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6680b8670 con 0x7ff67015d650 2023-05-22T15:26:49.748 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:49.734+0000 7ff657fff640 1 -- 172.21.15.16:0/1022553221 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6680b87d0 con 0x7ff67015d650 2023-05-22T15:26:49.748 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:49.734+0000 7ff675b9b640 1 -- 172.21.15.16:0/1022553221 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6700ca240 con 0x7ff67015d650 2023-05-22T15:26:49.748 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:49.734+0000 7ff657fff640 1 -- 172.21.15.16:0/1022553221 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6680b8b80 con 0x7ff67015d650 2023-05-22T15:26:49.748 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:49.734+0000 7ff675b9b640 1 -- 172.21.15.16:0/1022553221 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6700cce90 con 0x7ff67015d650 2023-05-22T15:26:49.749 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:49.734+0000 7ff675b9b640 1 -- 172.21.15.16:0/1022553221 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6700ca3d0 con 0x7ff67015d650 2023-05-22T15:26:49.750 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:49.734+0000 7ff657fff640 1 -- 172.21.15.16:0/1022553221 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7ff6680b8ce0 con 0x7ff67015d650 2023-05-22T15:26:49.750 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:49.738+0000 7ff657fff640 1 --2- 172.21.15.16:0/1022553221 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ff65c065380 0x7ff65c067830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:49.750 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:49.738+0000 7ff657fff640 1 -- 172.21.15.16:0/1022553221 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6062+0+0 (secure 0 0 0) 0x7ff66811de50 con 0x7ff67015d650 2023-05-22T15:26:49.750 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:49.738+0000 7ff66f7fe640 1 --2- 172.21.15.16:0/1022553221 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ff65c065380 0x7ff65c067830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:49.750 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:49.738+0000 7ff66f7fe640 1 --2- 172.21.15.16:0/1022553221 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ff65c065380 0x7ff65c067830 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7ff66000b930 tx=0x7ff660008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:26:49.751 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:49.742+0000 7ff657fff640 1 -- 172.21.15.16:0/1022553221 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7ff6680ebad0 con 0x7ff67015d650 2023-05-22T15:26:49.914 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:49.902+0000 7ff675b9b640 1 -- 172.21.15.16:0/1022553221 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]} v 0) v1 -- 0x7ff67004dbe0 con 0x7ff67015d650 2023-05-22T15:26:50.218 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-05-22T15:26:50.221 DEBUG:teuthology.orchestra.run.smithi062:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-05-22T15:26:50.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:50.554+0000 7ff657fff640 1 -- 172.21.15.16:0/1022553221 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]=0 v23) v1 ==== 157+0+0 (secure 0 0 0) 0x7ff6680ef0f0 con 0x7ff67015d650 2023-05-22T15:26:50.568 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:50.558+0000 7ff675b9b640 1 -- 172.21.15.16:0/1022553221 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ff65c065380 msgr2=0x7ff65c067830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:50.568 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:50.558+0000 7ff675b9b640 1 --2- 172.21.15.16:0/1022553221 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ff65c065380 0x7ff65c067830 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7ff66000b930 tx=0x7ff660008040 comp rx=0 tx=0).stop 2023-05-22T15:26:50.568 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:50.558+0000 7ff675b9b640 1 -- 172.21.15.16:0/1022553221 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff67015d650 msgr2=0x7ff67014af20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:50.568 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:50.558+0000 7ff675b9b640 1 --2- 172.21.15.16:0/1022553221 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff67015d650 0x7ff67014af20 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7ff6680195b0 tx=0x7ff668019740 comp rx=0 tx=0).stop 2023-05-22T15:26:50.568 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:50.558+0000 7ff675b9b640 1 -- 172.21.15.16:0/1022553221 shutdown_connections 2023-05-22T15:26:50.569 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:50.558+0000 7ff675b9b640 1 --2- 172.21.15.16:0/1022553221 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff67015d650 0x7ff67014af20 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:50.569 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:50.558+0000 7ff675b9b640 1 --2- 172.21.15.16:0/1022553221 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff6700c9e40 0x7ff670145c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:50.569 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:50.558+0000 7ff675b9b640 1 --2- 172.21.15.16:0/1022553221 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ff65c065380 0x7ff65c067830 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:50.569 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:50.558+0000 7ff675b9b640 1 --2- 172.21.15.16:0/1022553221 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff670070c10 0x7ff6701508a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:50.569 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:50.558+0000 7ff675b9b640 1 -- 172.21.15.16:0/1022553221 >> 172.21.15.16:0/1022553221 conn(0x7ff670069720 msgr2=0x7ff670069af0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:26:50.569 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:50.558+0000 7ff675b9b640 1 -- 172.21.15.16:0/1022553221 shutdown_connections 2023-05-22T15:26:50.569 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:50.558+0000 7ff675b9b640 1 -- 172.21.15.16:0/1022553221 wait complete. 2023-05-22T15:26:50.589 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data_ec erasure cephfs_data_ec --pg_num_min 64 --target_size_ratio 0.9 2023-05-22T15:26:50.721 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:50.710+0000 7f2506f8f640 1 Processor -- start 2023-05-22T15:26:50.722 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:50.710+0000 7f2506f8f640 1 -- start start 2023-05-22T15:26:50.723 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:50.710+0000 7f2506f8f640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f250006e320 0x7f250015f9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:50.724 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:50.710+0000 7f2506f8f640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f25000ccd40 0x7f25000ce490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:50.725 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:50.710+0000 7f2506f8f640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f250006f260 con 0x7f250006e6f0 2023-05-22T15:26:50.725 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:50.710+0000 7f2506f8f640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25000697c0 con 0x7f25000ccd40 2023-05-22T15:26:50.726 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:50.710+0000 7f2506f8f640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2500069a60 con 0x7f250006e320 2023-05-22T15:26:50.726 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:50.710+0000 7f250552c640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f25000ccd40 0x7f25000ce490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:50.726 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:50.710+0000 7f250552c640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f25000ccd40 0x7f25000ce490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49414/0 (socket says 172.21.15.16:49414) 2023-05-22T15:26:50.726 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:50.710+0000 7f250552c640 1 -- 172.21.15.16:0/2536727823 learned_addr learned my addr 172.21.15.16:0/2536727823 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:26:50.726 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:50.714+0000 7f2504d2b640 1 --2- 172.21.15.16:0/2536727823 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f250006e320 0x7f250015f9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:50.726 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:50.714+0000 7f250552c640 1 -- 172.21.15.16:0/2536727823 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f250006e320 msgr2=0x7f250015f9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:26:50.726 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:50.714+0000 7f250552c640 1 --2- 172.21.15.16:0/2536727823 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f250006e320 0x7f250015f9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:50.727 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:50.714+0000 7f250552c640 1 -- 172.21.15.16:0/2536727823 >> v1:172.21.15.62:6789/0 conn(0x7f250006e6f0 legacy=0x7f250006eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:26:50.727 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:50.714+0000 7f250552c640 1 -- 172.21.15.16:0/2536727823 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2500069d40 con 0x7f25000ccd40 2023-05-22T15:26:50.727 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:50.714+0000 7f2504d2b640 1 --2- 172.21.15.16:0/2536727823 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f250006e320 0x7f250015f9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:26:50.729 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:50.714+0000 7f250552c640 1 --2- 172.21.15.16:0/2536727823 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f25000ccd40 0x7f25000ce490 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f24f8009e50 tx=0x7f24f801c9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=956d11a6019170cd server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:26:50.729 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:50.714+0000 7f24ff7fe640 1 -- 172.21.15.16:0/2536727823 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24f80a5760 con 0x7f25000ccd40 2023-05-22T15:26:50.730 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:50.714+0000 7f24ff7fe640 1 -- 172.21.15.16:0/2536727823 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24f80a58c0 con 0x7f25000ccd40 2023-05-22T15:26:50.730 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:50.714+0000 7f24ff7fe640 1 -- 172.21.15.16:0/2536727823 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24f80a5c50 con 0x7f25000ccd40 2023-05-22T15:26:50.730 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:50.714+0000 7f2506f8f640 1 -- 172.21.15.16:0/2536727823 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f25000ccd40 msgr2=0x7f25000ce490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:50.730 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:50.714+0000 7f2506f8f640 1 --2- 172.21.15.16:0/2536727823 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f25000ccd40 0x7f25000ce490 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f24f8009e50 tx=0x7f24f801c9d0 comp rx=0 tx=0).stop 2023-05-22T15:26:50.730 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:50.714+0000 7f2506f8f640 1 -- 172.21.15.16:0/2536727823 shutdown_connections 2023-05-22T15:26:50.730 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:50.714+0000 7f2506f8f640 1 --2- 172.21.15.16:0/2536727823 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f25000ccd40 0x7f25000ce490 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:50.731 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:50.714+0000 7f2506f8f640 1 --2- 172.21.15.16:0/2536727823 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f250006e320 0x7f250015f9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:50.731 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:50.714+0000 7f2506f8f640 1 -- 172.21.15.16:0/2536727823 >> 172.21.15.16:0/2536727823 conn(0x7f25000b1950 msgr2=0x7f25000cf170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:26:50.731 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:50.714+0000 7f2506f8f640 1 -- 172.21.15.16:0/2536727823 shutdown_connections 2023-05-22T15:26:50.731 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:50.714+0000 7f2506f8f640 1 -- 172.21.15.16:0/2536727823 wait complete. 2023-05-22T15:26:50.731 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:50.718+0000 7f2506f8f640 1 Processor -- start 2023-05-22T15:26:50.731 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:50.718+0000 7f2506f8f640 1 -- start start 2023-05-22T15:26:50.731 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:50.718+0000 7f2506f8f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f250006e320 0x7f25001528e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:50.732 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:50.718+0000 7f2506f8f640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f25000ccd40 0x7f25001585c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:50.732 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:50.718+0000 7f2506f8f640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2500158b00 0x7f25000b3c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:50.732 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:50.718+0000 7f2506f8f640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2500158f80 con 0x7f250006e320 2023-05-22T15:26:50.732 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:50.718+0000 7f2506f8f640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2500152e20 con 0x7f25000ccd40 2023-05-22T15:26:50.732 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:50.718+0000 7f2506f8f640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25000ad6e0 con 0x7f2500158b00 2023-05-22T15:26:50.733 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:50.718+0000 7f2504d2b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f250006e320 0x7f25001528e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:50.733 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:50.718+0000 7f2504d2b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f250006e320 0x7f25001528e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49420/0 (socket says 172.21.15.16:49420) 2023-05-22T15:26:50.733 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:50.718+0000 7f2504d2b640 1 -- 172.21.15.16:0/1563356226 learned_addr learned my addr 172.21.15.16:0/1563356226 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:26:50.733 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:50.718+0000 7f250552c640 1 --2- 172.21.15.16:0/1563356226 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2500158b00 0x7f25000b3c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:50.733 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:50.718+0000 7f24fffff640 1 --2- 172.21.15.16:0/1563356226 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f25000ccd40 0x7f25001585c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:50.733 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:50.718+0000 7f2504d2b640 1 -- 172.21.15.16:0/1563356226 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2500158b00 msgr2=0x7f25000b3c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:50.733 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:50.718+0000 7f2504d2b640 1 --2- 172.21.15.16:0/1563356226 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2500158b00 0x7f25000b3c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:50.734 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:50.718+0000 7f2504d2b640 1 -- 172.21.15.16:0/1563356226 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f25000ccd40 msgr2=0x7f25001585c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:50.734 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:50.718+0000 7f2504d2b640 1 --2- 172.21.15.16:0/1563356226 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f25000ccd40 0x7f25001585c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:50.734 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:50.718+0000 7f2504d2b640 1 -- 172.21.15.16:0/1563356226 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24f8009b30 con 0x7f250006e320 2023-05-22T15:26:50.734 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:50.718+0000 7f250552c640 1 --2- 172.21.15.16:0/1563356226 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2500158b00 0x7f25000b3c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:26:50.734 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:50.718+0000 7f24fffff640 1 --2- 172.21.15.16:0/1563356226 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f25000ccd40 0x7f25001585c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:26:50.734 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:50.718+0000 7f2504d2b640 1 --2- 172.21.15.16:0/1563356226 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f250006e320 0x7f25001528e0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f24f000c7f0 tx=0x7f24f000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:26:50.734 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:50.718+0000 7f24fd7fa640 1 -- 172.21.15.16:0/1563356226 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24f000cea0 con 0x7f250006e320 2023-05-22T15:26:50.735 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:50.718+0000 7f24fd7fa640 1 -- 172.21.15.16:0/1563356226 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24f0007a30 con 0x7f250006e320 2023-05-22T15:26:50.735 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:50.718+0000 7f2506f8f640 1 -- 172.21.15.16:0/1563356226 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25000717f0 con 0x7f250006e320 2023-05-22T15:26:50.735 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:50.718+0000 7f24fd7fa640 1 -- 172.21.15.16:0/1563356226 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24f0007de0 con 0x7f250006e320 2023-05-22T15:26:50.735 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:50.718+0000 7f2506f8f640 1 -- 172.21.15.16:0/1563356226 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25000cede0 con 0x7f250006e320 2023-05-22T15:26:50.736 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:50.718+0000 7f2506f8f640 1 -- 172.21.15.16:0/1563356226 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2500071980 con 0x7f250006e320 2023-05-22T15:26:50.736 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:50.722+0000 7f24fd7fa640 1 -- 172.21.15.16:0/1563356226 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7f24f00113f0 con 0x7f250006e320 2023-05-22T15:26:50.737 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:50.722+0000 7f24fd7fa640 1 --2- 172.21.15.16:0/1563356226 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f24ec0653b0 0x7f24ec067860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:50.737 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:50.722+0000 7f24fd7fa640 1 -- 172.21.15.16:0/1563356226 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6288+0+0 (secure 0 0 0) 0x7f24f0082a90 con 0x7f250006e320 2023-05-22T15:26:50.737 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:50.722+0000 7f24fffff640 1 --2- 172.21.15.16:0/1563356226 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f24ec0653b0 0x7f24ec067860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:50.737 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:50.722+0000 7f24fffff640 1 --2- 172.21.15.16:0/1563356226 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f24ec0653b0 0x7f24ec067860 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f2500159150 tx=0x7f24f4009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:26:50.737 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:50.726+0000 7f24fd7fa640 1 -- 172.21.15.16:0/1563356226 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f24f0050550 con 0x7f250006e320 2023-05-22T15:26:50.886 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:50.874+0000 7f2506f8f640 1 -- 172.21.15.16:0/1563356226 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f2500073960 con 0x7f250006e320 2023-05-22T15:26:51.135 INFO:tasks.ceph.mds.c.smithi016.stderr:2023-05-22T15:26:51.130+0000 7fcda2cbc640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-05-22T15:26:52.592 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:52.582+0000 7f24fd7fa640 1 -- 172.21.15.16:0/1563356226 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data_ec' created v25) v1 ==== 228+0+0 (secure 0 0 0) 0x7f24f001e100 con 0x7f250006e320 2023-05-22T15:26:52.594 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'cephfs_data_ec' created 2023-05-22T15:26:52.602 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:52.590+0000 7f2506f8f640 1 -- 172.21.15.16:0/1563356226 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f24ec0653b0 msgr2=0x7f24ec067860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:52.603 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:52.590+0000 7f2506f8f640 1 --2- 172.21.15.16:0/1563356226 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f24ec0653b0 0x7f24ec067860 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f2500159150 tx=0x7f24f4009040 comp rx=0 tx=0).stop 2023-05-22T15:26:52.603 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:52.590+0000 7f2506f8f640 1 -- 172.21.15.16:0/1563356226 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f250006e320 msgr2=0x7f25001528e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:52.603 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:52.590+0000 7f2506f8f640 1 --2- 172.21.15.16:0/1563356226 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f250006e320 0x7f25001528e0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f24f000c7f0 tx=0x7f24f000ccb0 comp rx=0 tx=0).stop 2023-05-22T15:26:52.604 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:52.590+0000 7f2506f8f640 1 -- 172.21.15.16:0/1563356226 shutdown_connections 2023-05-22T15:26:52.604 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:52.590+0000 7f2506f8f640 1 --2- 172.21.15.16:0/1563356226 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f24ec0653b0 0x7f24ec067860 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:52.604 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:52.590+0000 7f2506f8f640 1 --2- 172.21.15.16:0/1563356226 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2500158b00 0x7f25000b3c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:52.604 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:52.590+0000 7f2506f8f640 1 --2- 172.21.15.16:0/1563356226 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f25000ccd40 0x7f25001585c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:52.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:52.590+0000 7f2506f8f640 1 --2- 172.21.15.16:0/1563356226 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f250006e320 0x7f25001528e0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:52.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:52.590+0000 7f2506f8f640 1 -- 172.21.15.16:0/1563356226 >> 172.21.15.16:0/1563356226 conn(0x7f25000b1950 msgr2=0x7f25000b1fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:26:52.606 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:52.594+0000 7f2506f8f640 1 -- 172.21.15.16:0/1563356226 shutdown_connections 2023-05-22T15:26:52.606 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:52.594+0000 7f2506f8f640 1 -- 172.21.15.16:0/1563356226 wait complete. 2023-05-22T15:26:52.628 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set cephfs_data_ec allow_ec_overwrites true 2023-05-22T15:26:52.755 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:52.746+0000 7f5b1f698640 1 Processor -- start 2023-05-22T15:26:52.756 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:52.746+0000 7f5b1f698640 1 -- start start 2023-05-22T15:26:52.756 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:52.746+0000 7f5b1f698640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5b1815d610 0x7f5b180cbe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:52.756 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:52.746+0000 7f5b1f698640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b180cde70 con 0x7f5b18159a30 2023-05-22T15:26:52.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:52.746+0000 7f5b1f698640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b1815db20 con 0x7f5b180cf570 2023-05-22T15:26:52.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:52.746+0000 7f5b1f698640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b1815dde0 con 0x7f5b1815d610 2023-05-22T15:26:52.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:52.746+0000 7f5b1d434640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f5b18159a30 0x7f5b1806e380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:43442/0 (socket says 172.21.15.16:43442) 2023-05-22T15:26:52.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:52.746+0000 7f5b1d434640 1 -- 172.21.15.16:0/2436598142 learned_addr learned my addr 172.21.15.16:0/2436598142 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:26:52.758 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:52.746+0000 7f5b1cc33640 1 --2- 172.21.15.16:0/2436598142 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5b1815d610 0x7f5b180cbe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:52.758 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:52.746+0000 7f5b0ffff640 1 -- 172.21.15.16:0/2436598142 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 125300862 0 0) 0x7f5b180cde70 con 0x7f5b18159a30 2023-05-22T15:26:52.758 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:52.746+0000 7f5b0ffff640 1 -- 172.21.15.16:0/2436598142 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b04003530 con 0x7f5b18159a30 2023-05-22T15:26:52.759 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:52.746+0000 7f5b0ffff640 1 -- 172.21.15.16:0/2436598142 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 91365993 0 0) 0x7f5b04003530 con 0x7f5b18159a30 2023-05-22T15:26:52.759 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:52.746+0000 7f5b0ffff640 1 -- 172.21.15.16:0/2436598142 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5b1815d610 msgr2=0x7f5b180cbe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:26:52.759 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:52.746+0000 7f5b0ffff640 1 --2- 172.21.15.16:0/2436598142 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5b1815d610 0x7f5b180cbe10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:52.759 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:52.746+0000 7f5b0ffff640 1 -- 172.21.15.16:0/2436598142 >> v1:172.21.15.62:6789/0 conn(0x7f5b180cf570 legacy=0x7f5b180ce300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:52.759 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:52.746+0000 7f5b0ffff640 1 -- 172.21.15.16:0/2436598142 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b1815e0c0 con 0x7f5b18159a30 2023-05-22T15:26:52.759 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:52.746+0000 7f5b0ffff640 1 -- 172.21.15.16:0/2436598142 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7f5b14001680 con 0x7f5b18159a30 2023-05-22T15:26:52.759 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:52.746+0000 7f5b0ffff640 1 -- 172.21.15.16:0/2436598142 >> v1:172.21.15.16:6789/0 conn(0x7f5b18159a30 legacy=0x7f5b1806e380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:52.759 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:52.746+0000 7f5b0ffff640 1 --2- 172.21.15.16:0/2436598142 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5b040039b0 0x7f5b04003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:52.760 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:52.746+0000 7f5b0ffff640 1 --2- 172.21.15.16:0/2436598142 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b040047d0 0x7f5b04004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:52.760 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:52.746+0000 7f5b0ffff640 1 --2- 172.21.15.16:0/2436598142 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5b1815d610 0x7f5b04005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:52.760 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:52.746+0000 7f5b0ffff640 1 -- 172.21.15.16:0/2436598142 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b1815dde0 con 0x7f5b040047d0 2023-05-22T15:26:52.760 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:52.746+0000 7f5b0ffff640 1 -- 172.21.15.16:0/2436598142 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b0400ba40 con 0x7f5b040039b0 2023-05-22T15:26:52.760 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:52.746+0000 7f5b0ffff640 1 -- 172.21.15.16:0/2436598142 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b0400bc00 con 0x7f5b1815d610 2023-05-22T15:26:52.760 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:52.746+0000 7f5b1cc33640 1 --2- 172.21.15.16:0/2436598142 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b040047d0 0x7f5b04004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:52.760 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:52.746+0000 7f5b1d434640 1 --2- 172.21.15.16:0/2436598142 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5b040039b0 0x7f5b04003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:52.761 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:52.746+0000 7f5b1cc33640 1 -- 172.21.15.16:0/2436598142 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5b1815d610 msgr2=0x7f5b04005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:26:52.761 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:52.746+0000 7f5b1dc35640 1 --2- 172.21.15.16:0/2436598142 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5b1815d610 0x7f5b04005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:52.761 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:52.746+0000 7f5b1cc33640 1 --2- 172.21.15.16:0/2436598142 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5b1815d610 0x7f5b04005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:52.761 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:52.746+0000 7f5b1cc33640 1 -- 172.21.15.16:0/2436598142 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5b040039b0 msgr2=0x7f5b04003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:26:52.761 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:52.746+0000 7f5b1cc33640 1 --2- 172.21.15.16:0/2436598142 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5b040039b0 0x7f5b04003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:52.761 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:52.746+0000 7f5b1cc33640 1 -- 172.21.15.16:0/2436598142 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b04003530 con 0x7f5b040047d0 2023-05-22T15:26:52.761 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:52.746+0000 7f5b1d434640 1 --2- 172.21.15.16:0/2436598142 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5b040039b0 0x7f5b04003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:26:52.761 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:52.746+0000 7f5b1dc35640 1 --2- 172.21.15.16:0/2436598142 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5b1815d610 0x7f5b04005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:26:52.763 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:52.750+0000 7f5b1cc33640 1 --2- 172.21.15.16:0/2436598142 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b040047d0 0x7f5b04004c40 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f5b08009ab0 tx=0x7f5b0801ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=281fb6bbd3ad7127 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:26:52.764 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:52.750+0000 7f5b0ffff640 1 -- 172.21.15.16:0/2436598142 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b080a5620 con 0x7f5b040047d0 2023-05-22T15:26:52.764 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:52.750+0000 7f5b0ffff640 1 -- 172.21.15.16:0/2436598142 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b14001d80 con 0x7f5b040047d0 2023-05-22T15:26:52.764 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:52.750+0000 7f5b0ffff640 1 -- 172.21.15.16:0/2436598142 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b080a59c0 con 0x7f5b040047d0 2023-05-22T15:26:52.764 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:52.750+0000 7f5b1f698640 1 -- 172.21.15.16:0/2436598142 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b040047d0 msgr2=0x7f5b04004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:52.764 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:52.750+0000 7f5b1f698640 1 --2- 172.21.15.16:0/2436598142 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b040047d0 0x7f5b04004c40 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f5b08009ab0 tx=0x7f5b0801ca30 comp rx=0 tx=0).stop 2023-05-22T15:26:52.764 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:52.750+0000 7f5b1dc35640 1 -- 172.21.15.16:0/2436598142 reap_dead start 2023-05-22T15:26:52.765 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:52.750+0000 7f5b1f698640 1 -- 172.21.15.16:0/2436598142 shutdown_connections 2023-05-22T15:26:52.765 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:52.750+0000 7f5b1f698640 1 -- 172.21.15.16:0/2436598142 >> 172.21.15.16:0/2436598142 conn(0x7f5b180700b0 msgr2=0x7f5b1806d9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:26:52.765 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:52.750+0000 7f5b1f698640 1 -- 172.21.15.16:0/2436598142 shutdown_connections 2023-05-22T15:26:52.765 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:52.750+0000 7f5b1f698640 1 -- 172.21.15.16:0/2436598142 wait complete. 2023-05-22T15:26:52.765 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:52.750+0000 7f5b1f698640 1 Processor -- start 2023-05-22T15:26:52.765 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:52.750+0000 7f5b1f698640 1 -- start start 2023-05-22T15:26:52.765 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:52.754+0000 7f5b1f698640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5b040047d0 0x7f5b181516b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:52.765 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:52.754+0000 7f5b1f698640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5b1815aa40 0x7f5b181efe30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:52.766 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:52.754+0000 7f5b1f698640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b181f0370 0x7f5b1815c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:52.766 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:52.754+0000 7f5b1f698640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b18159f90 con 0x7f5b181f0370 2023-05-22T15:26:52.766 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:52.754+0000 7f5b1f698640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b1815a270 con 0x7f5b040047d0 2023-05-22T15:26:52.766 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:52.754+0000 7f5b1f698640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b1815a550 con 0x7f5b1815aa40 2023-05-22T15:26:52.766 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:52.754+0000 7f5b1dc35640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b181f0370 0x7f5b1815c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:52.766 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:52.754+0000 7f5b1dc35640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b181f0370 0x7f5b1815c370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49432/0 (socket says 172.21.15.16:49432) 2023-05-22T15:26:52.766 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:52.754+0000 7f5b1dc35640 1 -- 172.21.15.16:0/1429999617 learned_addr learned my addr 172.21.15.16:0/1429999617 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:26:52.767 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:52.754+0000 7f5b1d434640 1 --2- 172.21.15.16:0/1429999617 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5b040047d0 0x7f5b181516b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:52.767 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:52.754+0000 7f5b1cc33640 1 --2- 172.21.15.16:0/1429999617 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5b1815aa40 0x7f5b181efe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:52.767 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:52.754+0000 7f5b1dc35640 1 -- 172.21.15.16:0/1429999617 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5b1815aa40 msgr2=0x7f5b181efe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:52.767 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:52.754+0000 7f5b1dc35640 1 --2- 172.21.15.16:0/1429999617 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5b1815aa40 0x7f5b181efe30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:52.767 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:52.754+0000 7f5b1dc35640 1 -- 172.21.15.16:0/1429999617 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5b040047d0 msgr2=0x7f5b181516b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:52.767 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:52.754+0000 7f5b1dc35640 1 --2- 172.21.15.16:0/1429999617 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5b040047d0 0x7f5b181516b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:52.767 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:52.754+0000 7f5b1dc35640 1 -- 172.21.15.16:0/1429999617 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b18159e00 con 0x7f5b181f0370 2023-05-22T15:26:52.767 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:52.754+0000 7f5b1d434640 1 --2- 172.21.15.16:0/1429999617 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5b040047d0 0x7f5b181516b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T15:26:52.768 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:52.754+0000 7f5b1dc35640 1 --2- 172.21.15.16:0/1429999617 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b181f0370 0x7f5b1815c370 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f5b1815d9e0 tx=0x7f5b1000d830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:26:52.768 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:52.754+0000 7f5b0dffb640 1 -- 172.21.15.16:0/1429999617 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b1000b070 con 0x7f5b181f0370 2023-05-22T15:26:52.768 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:52.754+0000 7f5b0dffb640 1 -- 172.21.15.16:0/1429999617 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b1000db40 con 0x7f5b181f0370 2023-05-22T15:26:52.768 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:52.754+0000 7f5b0dffb640 1 -- 172.21.15.16:0/1429999617 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b10003980 con 0x7f5b181f0370 2023-05-22T15:26:52.768 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:52.754+0000 7f5b1f698640 1 -- 172.21.15.16:0/1429999617 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b1815e3a0 con 0x7f5b181f0370 2023-05-22T15:26:52.768 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:52.754+0000 7f5b1f698640 1 -- 172.21.15.16:0/1429999617 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b180cea10 con 0x7f5b181f0370 2023-05-22T15:26:52.769 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:52.754+0000 7f5b1f698640 1 -- 172.21.15.16:0/1429999617 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b18066910 con 0x7f5b181f0370 2023-05-22T15:26:52.769 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:52.754+0000 7f5b0dffb640 1 -- 172.21.15.16:0/1429999617 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7f5b10003ae0 con 0x7f5b181f0370 2023-05-22T15:26:52.769 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:52.754+0000 7f5b0dffb640 1 --2- 172.21.15.16:0/1429999617 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f5afc065380 0x7f5afc067830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:52.770 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:52.754+0000 7f5b0dffb640 1 -- 172.21.15.16:0/1429999617 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6781+0+0 (secure 0 0 0) 0x7f5b100820f0 con 0x7f5b181f0370 2023-05-22T15:26:52.770 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:52.754+0000 7f5b1d434640 1 --2- 172.21.15.16:0/1429999617 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f5afc065380 0x7f5afc067830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:52.770 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:52.758+0000 7f5b1d434640 1 --2- 172.21.15.16:0/1429999617 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f5afc065380 0x7f5afc067830 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f5b14000c10 tx=0x7f5b14001420 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:26:52.770 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:52.758+0000 7f5b0dffb640 1 -- 172.21.15.16:0/1429999617 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f5b1004fa40 con 0x7f5b181f0370 2023-05-22T15:26:52.923 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:52.914+0000 7f5b1f698640 1 -- 172.21.15.16:0/1429999617 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"} v 0) v1 -- 0x7f5b1804dbe0 con 0x7f5b181f0370 2023-05-22T15:26:53.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:53.594+0000 7f5b0dffb640 1 -- 172.21.15.16:0/1429999617 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]=0 set pool 5 allow_ec_overwrites to true v26) v1 ==== 169+0+0 (secure 0 0 0) 0x7f5b10053060 con 0x7f5b181f0370 2023-05-22T15:26:53.607 INFO:teuthology.orchestra.run.smithi016.stderr:set pool 5 allow_ec_overwrites to true 2023-05-22T15:26:53.613 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:53.602+0000 7f5b1f698640 1 -- 172.21.15.16:0/1429999617 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f5afc065380 msgr2=0x7f5afc067830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:53.614 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:53.602+0000 7f5b1f698640 1 --2- 172.21.15.16:0/1429999617 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f5afc065380 0x7f5afc067830 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f5b14000c10 tx=0x7f5b14001420 comp rx=0 tx=0).stop 2023-05-22T15:26:53.614 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:53.602+0000 7f5b1f698640 1 -- 172.21.15.16:0/1429999617 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b181f0370 msgr2=0x7f5b1815c370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:53.614 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:53.602+0000 7f5b1f698640 1 --2- 172.21.15.16:0/1429999617 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b181f0370 0x7f5b1815c370 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f5b1815d9e0 tx=0x7f5b1000d830 comp rx=0 tx=0).stop 2023-05-22T15:26:53.619 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:53.610+0000 7f5b1f698640 1 -- 172.21.15.16:0/1429999617 shutdown_connections 2023-05-22T15:26:53.619 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:53.610+0000 7f5b1f698640 1 --2- 172.21.15.16:0/1429999617 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b181f0370 0x7f5b1815c370 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:53.619 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:53.610+0000 7f5b1f698640 1 --2- 172.21.15.16:0/1429999617 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f5afc065380 0x7f5afc067830 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:53.619 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:53.610+0000 7f5b1f698640 1 --2- 172.21.15.16:0/1429999617 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5b1815aa40 0x7f5b181efe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:53.620 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:53.610+0000 7f5b1f698640 1 --2- 172.21.15.16:0/1429999617 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5b040047d0 0x7f5b181516b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:53.620 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:53.610+0000 7f5b1f698640 1 -- 172.21.15.16:0/1429999617 >> 172.21.15.16:0/1429999617 conn(0x7f5b180700b0 msgr2=0x7f5b180cbdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:26:53.620 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:53.610+0000 7f5b1f698640 1 -- 172.21.15.16:0/1429999617 shutdown_connections 2023-05-22T15:26:53.620 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:53.610+0000 7f5b1f698640 1 -- 172.21.15.16:0/1429999617 wait complete. 2023-05-22T15:26:53.649 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs add_data_pool cephfs cephfs_data_ec 2023-05-22T15:26:53.775 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:53.766+0000 7fbbd635b640 1 Processor -- start 2023-05-22T15:26:53.776 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:53.766+0000 7fbbd635b640 1 -- start start 2023-05-22T15:26:53.779 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:53.766+0000 7fbbd635b640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbbd00b79b0 0x7fbbd015cc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:53.779 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:53.766+0000 7fbbd635b640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbbd00cca40 con 0x7fbbd00b7d80 2023-05-22T15:26:53.780 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:53.766+0000 7fbbd635b640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbbd00ccc90 con 0x7fbbd00cee70 2023-05-22T15:26:53.780 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:53.766+0000 7fbbd635b640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbbd00cd250 con 0x7fbbd00b79b0 2023-05-22T15:26:53.780 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:53.766+0000 7fbbcf7fe640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fbbd00b7d80 0x7fbbd00cffc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:43446/0 (socket says 172.21.15.16:43446) 2023-05-22T15:26:53.780 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:53.766+0000 7fbbcf7fe640 1 -- 172.21.15.16:0/2437805693 learned_addr learned my addr 172.21.15.16:0/2437805693 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:26:53.781 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:53.766+0000 7fbbceffd640 1 -- 172.21.15.16:0/2437805693 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3566599058 0 0) 0x7fbbd00cca40 con 0x7fbbd00b7d80 2023-05-22T15:26:53.781 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:53.766+0000 7fbbcffff640 1 --2- 172.21.15.16:0/2437805693 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbbd00b79b0 0x7fbbd015cc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:53.781 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:53.766+0000 7fbbceffd640 1 -- 172.21.15.16:0/2437805693 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbbbc003530 con 0x7fbbd00b7d80 2023-05-22T15:26:53.781 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:53.766+0000 7fbbceffd640 1 -- 172.21.15.16:0/2437805693 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3610262848 0 0) 0x7fbbbc003530 con 0x7fbbd00b7d80 2023-05-22T15:26:53.782 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:53.766+0000 7fbbceffd640 1 -- 172.21.15.16:0/2437805693 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbbd00b79b0 msgr2=0x7fbbd015cc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:26:53.782 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:53.766+0000 7fbbceffd640 1 --2- 172.21.15.16:0/2437805693 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbbd00b79b0 0x7fbbd015cc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:53.782 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:53.766+0000 7fbbceffd640 1 -- 172.21.15.16:0/2437805693 >> v1:172.21.15.62:6789/0 conn(0x7fbbd00cee70 legacy=0x7fbbd015f7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:53.783 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:53.766+0000 7fbbceffd640 1 -- 172.21.15.16:0/2437805693 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbd00ccf70 con 0x7fbbd00b7d80 2023-05-22T15:26:53.783 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:53.766+0000 7fbbceffd640 1 -- 172.21.15.16:0/2437805693 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7fbbb8001680 con 0x7fbbd00b7d80 2023-05-22T15:26:53.783 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:53.766+0000 7fbbceffd640 1 -- 172.21.15.16:0/2437805693 >> v1:172.21.15.16:6789/0 conn(0x7fbbd00b7d80 legacy=0x7fbbd00cffc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:53.783 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:53.766+0000 7fbbceffd640 1 --2- 172.21.15.16:0/2437805693 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbbbc0039b0 0x7fbbbc003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:53.784 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:53.766+0000 7fbbceffd640 1 --2- 172.21.15.16:0/2437805693 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbbbc0047d0 0x7fbbbc004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:53.784 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:53.766+0000 7fbbceffd640 1 --2- 172.21.15.16:0/2437805693 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbbd00b79b0 0x7fbbbc0051a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:53.784 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:53.766+0000 7fbbceffd640 1 -- 172.21.15.16:0/2437805693 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbd00cd250 con 0x7fbbbc0039b0 2023-05-22T15:26:53.784 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:53.766+0000 7fbbceffd640 1 -- 172.21.15.16:0/2437805693 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbbc00ba90 con 0x7fbbbc0047d0 2023-05-22T15:26:53.785 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:53.766+0000 7fbbceffd640 1 -- 172.21.15.16:0/2437805693 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbbbc00bc50 con 0x7fbbd00b79b0 2023-05-22T15:26:53.785 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:53.766+0000 7fbbcffff640 1 --2- 172.21.15.16:0/2437805693 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbbbc0039b0 0x7fbbbc003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:53.785 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:53.766+0000 7fbbcf7fe640 1 --2- 172.21.15.16:0/2437805693 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbbbc0047d0 0x7fbbbc004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:53.785 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:53.766+0000 7fbbd48f8640 1 --2- 172.21.15.16:0/2437805693 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbbd00b79b0 0x7fbbbc0051a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:53.786 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:53.766+0000 7fbbcffff640 1 -- 172.21.15.16:0/2437805693 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbbd00b79b0 msgr2=0x7fbbbc0051a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:26:53.786 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:53.766+0000 7fbbcffff640 1 --2- 172.21.15.16:0/2437805693 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbbd00b79b0 0x7fbbbc0051a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:53.786 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:53.766+0000 7fbbcffff640 1 -- 172.21.15.16:0/2437805693 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbbbc0047d0 msgr2=0x7fbbbc004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:26:53.787 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:53.766+0000 7fbbcffff640 1 --2- 172.21.15.16:0/2437805693 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbbbc0047d0 0x7fbbbc004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:53.787 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:53.766+0000 7fbbcffff640 1 -- 172.21.15.16:0/2437805693 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbbc003530 con 0x7fbbbc0039b0 2023-05-22T15:26:53.787 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:53.766+0000 7fbbd48f8640 1 --2- 172.21.15.16:0/2437805693 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbbd00b79b0 0x7fbbbc0051a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:26:53.792 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:53.770+0000 7fbbcffff640 1 --2- 172.21.15.16:0/2437805693 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbbbc0039b0 0x7fbbbc003dc0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fbbc0009fc0 tx=0x7fbbc001cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6bcd1bc71eade9f server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:26:53.792 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:53.770+0000 7fbbceffd640 1 -- 172.21.15.16:0/2437805693 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbc00a5b30 con 0x7fbbbc0039b0 2023-05-22T15:26:53.792 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:53.770+0000 7fbbceffd640 1 -- 172.21.15.16:0/2437805693 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbbc000f040 con 0x7fbbbc0039b0 2023-05-22T15:26:53.793 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:53.770+0000 7fbbceffd640 1 -- 172.21.15.16:0/2437805693 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbc00ad440 con 0x7fbbbc0039b0 2023-05-22T15:26:53.793 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:53.770+0000 7fbbd635b640 1 -- 172.21.15.16:0/2437805693 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbbbc0039b0 msgr2=0x7fbbbc003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:53.793 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:53.770+0000 7fbbd635b640 1 --2- 172.21.15.16:0/2437805693 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbbbc0039b0 0x7fbbbc003dc0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fbbc0009fc0 tx=0x7fbbc001cc30 comp rx=0 tx=0).stop 2023-05-22T15:26:53.793 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:53.770+0000 7fbbd48f8640 1 -- 172.21.15.16:0/2437805693 reap_dead start 2023-05-22T15:26:53.793 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:53.770+0000 7fbbd635b640 1 -- 172.21.15.16:0/2437805693 shutdown_connections 2023-05-22T15:26:53.794 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:53.770+0000 7fbbd635b640 1 -- 172.21.15.16:0/2437805693 >> 172.21.15.16:0/2437805693 conn(0x7fbbd0070e50 msgr2=0x7fbbd0071250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:26:53.794 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:53.770+0000 7fbbd635b640 1 -- 172.21.15.16:0/2437805693 shutdown_connections 2023-05-22T15:26:53.794 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:53.770+0000 7fbbd635b640 1 -- 172.21.15.16:0/2437805693 wait complete. 2023-05-22T15:26:53.794 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:53.770+0000 7fbbd635b640 1 Processor -- start 2023-05-22T15:26:53.794 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:53.770+0000 7fbbd635b640 1 -- start start 2023-05-22T15:26:53.795 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:53.774+0000 7fbbd635b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbbbc0039b0 0x7fbbd015a000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:53.795 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:53.774+0000 7fbbd635b640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbbd00cab60 0x7fbbd0156a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:53.795 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:53.774+0000 7fbbd635b640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbbd01f4380 0x7fbbd00b1d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:53.795 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:53.774+0000 7fbbd635b640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbd0157d80 con 0x7fbbbc0039b0 2023-05-22T15:26:53.795 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:53.774+0000 7fbbd635b640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbd00b0990 con 0x7fbbd01f4380 2023-05-22T15:26:53.796 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:53.774+0000 7fbbd635b640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbbd01f4820 con 0x7fbbd00cab60 2023-05-22T15:26:53.796 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:53.774+0000 7fbbcffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbbbc0039b0 0x7fbbd015a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:53.796 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:53.774+0000 7fbbcffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbbbc0039b0 0x7fbbd015a000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49444/0 (socket says 172.21.15.16:49444) 2023-05-22T15:26:53.796 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:53.774+0000 7fbbcffff640 1 -- 172.21.15.16:0/3284955811 learned_addr learned my addr 172.21.15.16:0/3284955811 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:26:53.797 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:53.774+0000 7fbbd48f8640 1 --2- 172.21.15.16:0/3284955811 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbbd01f4380 0x7fbbd00b1d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:53.797 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:53.774+0000 7fbbcf7fe640 1 --2- 172.21.15.16:0/3284955811 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbbd00cab60 0x7fbbd0156a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:53.797 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:53.774+0000 7fbbcffff640 1 -- 172.21.15.16:0/3284955811 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbbd00cab60 msgr2=0x7fbbd0156a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:53.798 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:53.774+0000 7fbbcffff640 1 --2- 172.21.15.16:0/3284955811 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbbd00cab60 0x7fbbd0156a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:53.798 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:53.774+0000 7fbbcffff640 1 -- 172.21.15.16:0/3284955811 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbbd01f4380 msgr2=0x7fbbd00b1d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:53.798 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:53.774+0000 7fbbcffff640 1 --2- 172.21.15.16:0/3284955811 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbbd01f4380 0x7fbbd00b1d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:53.798 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:53.774+0000 7fbbcffff640 1 -- 172.21.15.16:0/3284955811 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbd00ce640 con 0x7fbbbc0039b0 2023-05-22T15:26:53.799 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:53.774+0000 7fbbcffff640 1 --2- 172.21.15.16:0/3284955811 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbbbc0039b0 0x7fbbd015a000 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fbbc001c6a0 tx=0x7fbbc0009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:26:53.799 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:53.774+0000 7fbbccff9640 1 -- 172.21.15.16:0/3284955811 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbc0003ba0 con 0x7fbbbc0039b0 2023-05-22T15:26:53.799 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:53.774+0000 7fbbccff9640 1 -- 172.21.15.16:0/3284955811 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbbc0003d00 con 0x7fbbbc0039b0 2023-05-22T15:26:53.800 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:53.774+0000 7fbbd635b640 1 -- 172.21.15.16:0/3284955811 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbbd00d06d0 con 0x7fbbbc0039b0 2023-05-22T15:26:53.800 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:53.774+0000 7fbbccff9640 1 -- 172.21.15.16:0/3284955811 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbc0004260 con 0x7fbbbc0039b0 2023-05-22T15:26:53.800 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:53.774+0000 7fbbd635b640 1 -- 172.21.15.16:0/3284955811 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbbd00cf680 con 0x7fbbbc0039b0 2023-05-22T15:26:53.800 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:53.774+0000 7fbbd635b640 1 -- 172.21.15.16:0/3284955811 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbbd00738f0 con 0x7fbbbc0039b0 2023-05-22T15:26:53.800 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:53.774+0000 7fbbccff9640 1 -- 172.21.15.16:0/3284955811 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7fbbc00b4040 con 0x7fbbbc0039b0 2023-05-22T15:26:53.801 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:53.774+0000 7fbbccff9640 1 --2- 172.21.15.16:0/3284955811 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fbbb0065380 0x7fbbb0067830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:53.801 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:53.774+0000 7fbbccff9640 1 -- 172.21.15.16:0/3284955811 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6781+0+0 (secure 0 0 0) 0x7fbbc011d170 con 0x7fbbbc0039b0 2023-05-22T15:26:53.801 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:53.774+0000 7fbbcf7fe640 1 --2- 172.21.15.16:0/3284955811 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fbbb0065380 0x7fbbb0067830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:53.801 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:53.778+0000 7fbbcf7fe640 1 --2- 172.21.15.16:0/3284955811 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fbbb0065380 0x7fbbb0067830 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fbbb8001060 tx=0x7fbbb8009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:26:53.801 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:53.782+0000 7fbbccff9640 1 -- 172.21.15.16:0/3284955811 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fbbc00eabd0 con 0x7fbbbc0039b0 2023-05-22T15:26:53.975 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:53.966+0000 7fbbd635b640 1 -- 172.21.15.16:0/3284955811 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"} v 0) v1 -- 0x7fbbd00c94a0 con 0x7fbbbc0039b0 2023-05-22T15:26:53.981 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:53.970+0000 7fbbccff9640 1 -- 172.21.15.16:0/3284955811 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]=0 added data pool 5 to fsmap v6) v1 ==== 137+0+0 (secure 0 0 0) 0x7fbbc00ee1f0 con 0x7fbbbc0039b0 2023-05-22T15:26:53.982 INFO:teuthology.orchestra.run.smithi016.stderr:added data pool 5 to fsmap 2023-05-22T15:26:53.984 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:53.974+0000 7fbbd635b640 1 -- 172.21.15.16:0/3284955811 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fbbb0065380 msgr2=0x7fbbb0067830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:53.985 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:53.974+0000 7fbbd635b640 1 --2- 172.21.15.16:0/3284955811 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fbbb0065380 0x7fbbb0067830 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fbbb8001060 tx=0x7fbbb8009040 comp rx=0 tx=0).stop 2023-05-22T15:26:53.985 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:53.974+0000 7fbbd635b640 1 -- 172.21.15.16:0/3284955811 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbbbc0039b0 msgr2=0x7fbbd015a000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:53.985 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:53.974+0000 7fbbd635b640 1 --2- 172.21.15.16:0/3284955811 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbbbc0039b0 0x7fbbd015a000 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fbbc001c6a0 tx=0x7fbbc0009750 comp rx=0 tx=0).stop 2023-05-22T15:26:53.987 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:53.974+0000 7fbbd635b640 1 -- 172.21.15.16:0/3284955811 shutdown_connections 2023-05-22T15:26:53.987 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:53.974+0000 7fbbd635b640 1 --2- 172.21.15.16:0/3284955811 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbbd01f4380 0x7fbbd00b1d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:53.987 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:53.974+0000 7fbbd635b640 1 --2- 172.21.15.16:0/3284955811 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fbbb0065380 0x7fbbb0067830 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:53.987 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:53.974+0000 7fbbd635b640 1 --2- 172.21.15.16:0/3284955811 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbbd00cab60 0x7fbbd0156a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:53.988 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:53.974+0000 7fbbd635b640 1 --2- 172.21.15.16:0/3284955811 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbbbc0039b0 0x7fbbd015a000 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:53.988 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:53.974+0000 7fbbd635b640 1 -- 172.21.15.16:0/3284955811 >> 172.21.15.16:0/3284955811 conn(0x7fbbd0070e50 msgr2=0x7fbbd015d370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:26:53.988 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:53.974+0000 7fbbd635b640 1 -- 172.21.15.16:0/3284955811 shutdown_connections 2023-05-22T15:26:53.988 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:53.974+0000 7fbbd635b640 1 -- 172.21.15.16:0/3284955811 wait complete. 2023-05-22T15:26:54.000 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-05-22T15:26:54.110 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.098+0000 7f7b2385b640 1 Processor -- start 2023-05-22T15:26:54.110 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.098+0000 7f7b2385b640 1 -- start start 2023-05-22T15:26:54.111 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.102+0000 7f7b2385b640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7b1c071500 0x7f7b1c15e460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:54.111 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.102+0000 7f7b2385b640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b1c0b0650 con 0x7f7b1c0ca4e0 2023-05-22T15:26:54.111 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.102+0000 7f7b2385b640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b1c0b0370 con 0x7f7b1c0c9b40 2023-05-22T15:26:54.111 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.102+0000 7f7b2385b640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b1c15d980 con 0x7f7b1c071500 2023-05-22T15:26:54.111 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.102+0000 7f7b215f7640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f7b1c0ca4e0 0x7f7b1c0afb40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:43458/0 (socket says 172.21.15.16:43458) 2023-05-22T15:26:54.111 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.102+0000 7f7b215f7640 1 -- 172.21.15.16:0/2190778012 learned_addr learned my addr 172.21.15.16:0/2190778012 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:26:54.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.102+0000 7f7b21df8640 1 --2- 172.21.15.16:0/2190778012 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7b1c071500 0x7f7b1c15e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:54.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.102+0000 7f7b1bfff640 1 -- 172.21.15.16:0/2190778012 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3139189607 0 0) 0x7f7b1c0b0650 con 0x7f7b1c0ca4e0 2023-05-22T15:26:54.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.102+0000 7f7b1bfff640 1 -- 172.21.15.16:0/2190778012 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b04003530 con 0x7f7b1c0ca4e0 2023-05-22T15:26:54.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.102+0000 7f7b1bfff640 1 -- 172.21.15.16:0/2190778012 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1877290957 0 0) 0x7f7b04003530 con 0x7f7b1c0ca4e0 2023-05-22T15:26:54.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.102+0000 7f7b1bfff640 1 -- 172.21.15.16:0/2190778012 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7b1c071500 msgr2=0x7f7b1c15e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:26:54.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.102+0000 7f7b1bfff640 1 --2- 172.21.15.16:0/2190778012 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7b1c071500 0x7f7b1c15e460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:54.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.102+0000 7f7b1bfff640 1 -- 172.21.15.16:0/2190778012 >> v1:172.21.15.62:6789/0 conn(0x7f7b1c0c9b40 legacy=0x7f7b1c070df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:54.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.102+0000 7f7b1bfff640 1 -- 172.21.15.16:0/2190778012 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b1c15dc60 con 0x7f7b1c0ca4e0 2023-05-22T15:26:54.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.102+0000 7f7b1bfff640 1 -- 172.21.15.16:0/2190778012 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7f7b14001a20 con 0x7f7b1c0ca4e0 2023-05-22T15:26:54.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.102+0000 7f7b21df8640 1 --2- 172.21.15.16:0/2190778012 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7b1c071500 0x7f7b1c15e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T15:26:54.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.102+0000 7f7b1bfff640 1 -- 172.21.15.16:0/2190778012 >> v1:172.21.15.16:6789/0 conn(0x7f7b1c0ca4e0 legacy=0x7f7b1c0afb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:54.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.102+0000 7f7b1bfff640 1 --2- 172.21.15.16:0/2190778012 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7b040039b0 0x7f7b04003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:54.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.102+0000 7f7b1bfff640 1 --2- 172.21.15.16:0/2190778012 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7b040047d0 0x7f7b04004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:54.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.102+0000 7f7b1bfff640 1 --2- 172.21.15.16:0/2190778012 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7b1c071500 0x7f7b04005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:54.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.102+0000 7f7b1bfff640 1 -- 172.21.15.16:0/2190778012 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b1c15d980 con 0x7f7b040047d0 2023-05-22T15:26:54.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.102+0000 7f7b1bfff640 1 -- 172.21.15.16:0/2190778012 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b0400baa0 con 0x7f7b040039b0 2023-05-22T15:26:54.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.102+0000 7f7b1bfff640 1 -- 172.21.15.16:0/2190778012 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b0400bc60 con 0x7f7b1c071500 2023-05-22T15:26:54.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.102+0000 7f7b20df6640 1 --2- 172.21.15.16:0/2190778012 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7b040047d0 0x7f7b04004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:54.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.102+0000 7f7b215f7640 1 --2- 172.21.15.16:0/2190778012 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7b040039b0 0x7f7b04003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:54.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.102+0000 7f7b21df8640 1 --2- 172.21.15.16:0/2190778012 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7b1c071500 0x7f7b04005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:54.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.102+0000 7f7b20df6640 1 -- 172.21.15.16:0/2190778012 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7b1c071500 msgr2=0x7f7b04005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:26:54.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.102+0000 7f7b20df6640 1 --2- 172.21.15.16:0/2190778012 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7b1c071500 0x7f7b04005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:54.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.102+0000 7f7b20df6640 1 -- 172.21.15.16:0/2190778012 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7b040039b0 msgr2=0x7f7b04003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:26:54.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.102+0000 7f7b20df6640 1 --2- 172.21.15.16:0/2190778012 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7b040039b0 0x7f7b04003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:54.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.102+0000 7f7b20df6640 1 -- 172.21.15.16:0/2190778012 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b04003530 con 0x7f7b040047d0 2023-05-22T15:26:54.120 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.102+0000 7f7b20df6640 1 --2- 172.21.15.16:0/2190778012 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7b040047d0 0x7f7b04004c40 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f7b10001780 tx=0x7f7b1001c400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6472d1244a378faf server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:26:54.120 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.106+0000 7f7b1bfff640 1 -- 172.21.15.16:0/2190778012 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b100052d0 con 0x7f7b040047d0 2023-05-22T15:26:54.120 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.106+0000 7f7b1bfff640 1 -- 172.21.15.16:0/2190778012 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b100a7050 con 0x7f7b040047d0 2023-05-22T15:26:54.120 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.106+0000 7f7b1bfff640 1 -- 172.21.15.16:0/2190778012 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b10003600 con 0x7f7b040047d0 2023-05-22T15:26:54.120 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.106+0000 7f7b2385b640 1 -- 172.21.15.16:0/2190778012 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7b040047d0 msgr2=0x7f7b04004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:54.121 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.106+0000 7f7b2385b640 1 --2- 172.21.15.16:0/2190778012 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7b040047d0 0x7f7b04004c40 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f7b10001780 tx=0x7f7b1001c400 comp rx=0 tx=0).stop 2023-05-22T15:26:54.121 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.106+0000 7f7b21df8640 1 -- 172.21.15.16:0/2190778012 reap_dead start 2023-05-22T15:26:54.121 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.106+0000 7f7b2385b640 1 -- 172.21.15.16:0/2190778012 shutdown_connections 2023-05-22T15:26:54.121 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.106+0000 7f7b2385b640 1 -- 172.21.15.16:0/2190778012 >> 172.21.15.16:0/2190778012 conn(0x7f7b1c069720 msgr2=0x7f7b1c069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:26:54.121 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.106+0000 7f7b2385b640 1 -- 172.21.15.16:0/2190778012 shutdown_connections 2023-05-22T15:26:54.122 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.106+0000 7f7b2385b640 1 -- 172.21.15.16:0/2190778012 wait complete. 2023-05-22T15:26:54.122 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.106+0000 7f7b2385b640 1 Processor -- start 2023-05-22T15:26:54.122 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.106+0000 7f7b2385b640 1 -- start start 2023-05-22T15:26:54.122 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.106+0000 7f7b2385b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7b040047d0 0x7f7b1c1575b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:54.122 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.106+0000 7f7b2385b640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7b1c0c3cb0 0x7f7b1c0ccc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:54.122 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.106+0000 7f7b2385b640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7b1c1f1cd0 0x7f7b1c1f2150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:54.122 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.106+0000 7f7b2385b640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b1c0cdbc0 con 0x7f7b040047d0 2023-05-22T15:26:54.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.106+0000 7f7b2385b640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b1c0cd600 con 0x7f7b1c1f1cd0 2023-05-22T15:26:54.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.106+0000 7f7b2385b640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b1c0cd8e0 con 0x7f7b1c0c3cb0 2023-05-22T15:26:54.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.106+0000 7f7b215f7640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7b040047d0 0x7f7b1c1575b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:54.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.106+0000 7f7b215f7640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7b040047d0 0x7f7b1c1575b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49464/0 (socket says 172.21.15.16:49464) 2023-05-22T15:26:54.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.106+0000 7f7b215f7640 1 -- 172.21.15.16:0/1670361716 learned_addr learned my addr 172.21.15.16:0/1670361716 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:26:54.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.106+0000 7f7b21df8640 1 --2- 172.21.15.16:0/1670361716 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7b1c1f1cd0 0x7f7b1c1f2150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:54.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.106+0000 7f7b20df6640 1 --2- 172.21.15.16:0/1670361716 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7b1c0c3cb0 0x7f7b1c0ccc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:54.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.106+0000 7f7b215f7640 1 -- 172.21.15.16:0/1670361716 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7b1c0c3cb0 msgr2=0x7f7b1c0ccc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:54.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.106+0000 7f7b215f7640 1 --2- 172.21.15.16:0/1670361716 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7b1c0c3cb0 0x7f7b1c0ccc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:54.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.106+0000 7f7b215f7640 1 -- 172.21.15.16:0/1670361716 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7b1c1f1cd0 msgr2=0x7f7b1c1f2150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:54.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.106+0000 7f7b215f7640 1 --2- 172.21.15.16:0/1670361716 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7b1c1f1cd0 0x7f7b1c1f2150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:54.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.106+0000 7f7b215f7640 1 -- 172.21.15.16:0/1670361716 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b1c0ca8b0 con 0x7f7b040047d0 2023-05-22T15:26:54.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.106+0000 7f7b215f7640 1 --2- 172.21.15.16:0/1670361716 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7b040047d0 0x7f7b1c1575b0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f7b14011820 tx=0x7f7b14011ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:26:54.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.106+0000 7f7b19ffb640 1 -- 172.21.15.16:0/1670361716 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b1400fb60 con 0x7f7b040047d0 2023-05-22T15:26:54.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.106+0000 7f7b19ffb640 1 -- 172.21.15.16:0/1670361716 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b1400fd00 con 0x7f7b040047d0 2023-05-22T15:26:54.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.106+0000 7f7b2385b640 1 -- 172.21.15.16:0/1670361716 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b1c15df40 con 0x7f7b040047d0 2023-05-22T15:26:54.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.106+0000 7f7b19ffb640 1 -- 172.21.15.16:0/1670361716 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b14015650 con 0x7f7b040047d0 2023-05-22T15:26:54.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.106+0000 7f7b2385b640 1 -- 172.21.15.16:0/1670361716 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b1c15e460 con 0x7f7b040047d0 2023-05-22T15:26:54.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.110+0000 7f7b2385b640 1 -- 172.21.15.16:0/1670361716 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b1c0b1470 con 0x7f7b040047d0 2023-05-22T15:26:54.126 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.110+0000 7f7b19ffb640 1 -- 172.21.15.16:0/1670361716 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7f7b140157b0 con 0x7f7b040047d0 2023-05-22T15:26:54.126 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.110+0000 7f7b19ffb640 1 --2- 172.21.15.16:0/1670361716 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f7b08065380 0x7f7b08067830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:54.126 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.110+0000 7f7b19ffb640 1 -- 172.21.15.16:0/1670361716 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f7b14085ff0 con 0x7f7b040047d0 2023-05-22T15:26:54.126 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.110+0000 7f7b20df6640 1 --2- 172.21.15.16:0/1670361716 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f7b08065380 0x7f7b08067830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:54.126 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.110+0000 7f7b20df6640 1 --2- 172.21.15.16:0/1670361716 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f7b08065380 0x7f7b08067830 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f7b10001750 tx=0x7f7b100011a0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:26:54.127 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.114+0000 7f7b19ffb640 1 -- 172.21.15.16:0/1670361716 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f7b14053980 con 0x7f7b040047d0 2023-05-22T15:26:54.311 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.298+0000 7f7b2385b640 1 -- 172.21.15.16:0/1670361716 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7b1c04dbe0 con 0x7f7b040047d0 2023-05-22T15:26:54.311 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.298+0000 7f7b19ffb640 1 -- 172.21.15.16:0/1670361716 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) v1 ==== 93+0+4641 (secure 0 0 0) 0x7f7b14001910 con 0x7f7b040047d0 2023-05-22T15:26:54.312 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:26:54.312 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":6,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4261,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6834/3661808159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":3661808159},{"type":"v1","addr":"172.21.15.62:6834","nonce":3661808159}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4264,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/2227467222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6833","nonce":2227467222},{"type":"v1","addr":"172.21.15.62:6835","nonce":2227467222}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4284,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6836/2808106087","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2808106087},{"type":"v1","addr":"172.21.15.16:6836","nonce":2808106087}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":6,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-05-22T15:26:49.546008+0000","modified":"2023-05-22T15:26:53.971878+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4287},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4287":{"gid":4287,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.16:6837/283322289","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6835","nonce":283322289},{"type":"v1","addr":"172.21.15.16:6837","nonce":283322289}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2023-05-22T15:26:54.313 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 6 2023-05-22T15:26:54.313 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.302+0000 7f7b2385b640 1 -- 172.21.15.16:0/1670361716 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f7b08065380 msgr2=0x7f7b08067830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:54.313 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.302+0000 7f7b2385b640 1 --2- 172.21.15.16:0/1670361716 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f7b08065380 0x7f7b08067830 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f7b10001750 tx=0x7f7b100011a0 comp rx=0 tx=0).stop 2023-05-22T15:26:54.314 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.302+0000 7f7b2385b640 1 -- 172.21.15.16:0/1670361716 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7b040047d0 msgr2=0x7f7b1c1575b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:54.314 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.302+0000 7f7b2385b640 1 --2- 172.21.15.16:0/1670361716 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7b040047d0 0x7f7b1c1575b0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f7b14011820 tx=0x7f7b14011ce0 comp rx=0 tx=0).stop 2023-05-22T15:26:54.314 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.302+0000 7f7b2385b640 1 -- 172.21.15.16:0/1670361716 shutdown_connections 2023-05-22T15:26:54.314 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.302+0000 7f7b2385b640 1 --2- 172.21.15.16:0/1670361716 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7b1c1f1cd0 0x7f7b1c1f2150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:54.314 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.302+0000 7f7b2385b640 1 --2- 172.21.15.16:0/1670361716 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f7b08065380 0x7f7b08067830 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:54.314 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.302+0000 7f7b2385b640 1 --2- 172.21.15.16:0/1670361716 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7b1c0c3cb0 0x7f7b1c0ccc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:54.315 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.302+0000 7f7b2385b640 1 --2- 172.21.15.16:0/1670361716 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7b040047d0 0x7f7b1c1575b0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:54.315 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.302+0000 7f7b2385b640 1 -- 172.21.15.16:0/1670361716 >> 172.21.15.16:0/1670361716 conn(0x7f7b1c069720 msgr2=0x7f7b1c0676c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:26:54.315 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.302+0000 7f7b2385b640 1 -- 172.21.15.16:0/1670361716 shutdown_connections 2023-05-22T15:26:54.315 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.302+0000 7f7b2385b640 1 -- 172.21.15.16:0/1670361716 wait complete. 2023-05-22T15:26:54.329 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-05-22T15:26:54.455 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.446+0000 7fe48ad16640 1 Processor -- start 2023-05-22T15:26:54.456 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.446+0000 7fe48ad16640 1 -- start start 2023-05-22T15:26:54.458 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.446+0000 7fe48ad16640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe484159cd0 0x7fe48415a0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:54.458 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.446+0000 7fe48ad16640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fe4840b2880 0x7fe4840cbfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:54.458 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.446+0000 7fe48ad16640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4840cf9a0 con 0x7fe48415a5e0 2023-05-22T15:26:54.458 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.446+0000 7fe48ad16640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe48415aba0 con 0x7fe4840b2880 2023-05-22T15:26:54.458 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.446+0000 7fe48ad16640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe48415ae80 con 0x7fe484159cd0 2023-05-22T15:26:54.458 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.446+0000 7fe4892b3640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fe48415a5e0 0x7fe4840ce160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:43466/0 (socket says 172.21.15.16:43466) 2023-05-22T15:26:54.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.446+0000 7fe4892b3640 1 -- 172.21.15.16:0/3432031456 learned_addr learned my addr 172.21.15.16:0/3432031456 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:26:54.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.446+0000 7fe483fff640 1 --2- 172.21.15.16:0/3432031456 >> v2:172.21.15.62:3300/0 conn(0x7fe4840b2880 0x7fe4840cbfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:54.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.446+0000 7fe488ab2640 1 --2- 172.21.15.16:0/3432031456 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe484159cd0 0x7fe48415a0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:54.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.446+0000 7fe4837fe640 1 -- 172.21.15.16:0/3432031456 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1008868890 0 0) 0x7fe4840cf9a0 con 0x7fe48415a5e0 2023-05-22T15:26:54.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.446+0000 7fe4837fe640 1 -- 172.21.15.16:0/3432031456 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe470003530 con 0x7fe48415a5e0 2023-05-22T15:26:54.460 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.446+0000 7fe4837fe640 1 -- 172.21.15.16:0/3432031456 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 591961028 0 0) 0x7fe470003530 con 0x7fe48415a5e0 2023-05-22T15:26:54.460 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.446+0000 7fe4837fe640 1 -- 172.21.15.16:0/3432031456 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe484159cd0 msgr2=0x7fe48415a0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:26:54.461 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.446+0000 7fe4837fe640 1 --2- 172.21.15.16:0/3432031456 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe484159cd0 0x7fe48415a0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:54.461 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.446+0000 7fe4837fe640 1 -- 172.21.15.16:0/3432031456 >> v2:172.21.15.62:3300/0 conn(0x7fe4840b2880 msgr2=0x7fe4840cbfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:26:54.461 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.446+0000 7fe4837fe640 1 --2- 172.21.15.16:0/3432031456 >> v2:172.21.15.62:3300/0 conn(0x7fe4840b2880 0x7fe4840cbfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:54.461 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.446+0000 7fe4837fe640 1 -- 172.21.15.16:0/3432031456 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe48415b160 con 0x7fe48415a5e0 2023-05-22T15:26:54.461 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.446+0000 7fe4837fe640 1 -- 172.21.15.16:0/3432031456 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7fe474001680 con 0x7fe48415a5e0 2023-05-22T15:26:54.461 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.446+0000 7fe488ab2640 1 --2- 172.21.15.16:0/3432031456 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe484159cd0 0x7fe48415a0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T15:26:54.461 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.446+0000 7fe4837fe640 1 -- 172.21.15.16:0/3432031456 >> v1:172.21.15.16:6789/0 conn(0x7fe48415a5e0 legacy=0x7fe4840ce160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:54.462 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.446+0000 7fe4837fe640 1 --2- 172.21.15.16:0/3432031456 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe470003980 0x7fe470003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:54.462 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.446+0000 7fe4837fe640 1 --2- 172.21.15.16:0/3432031456 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe470004360 0x7fe470004830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:54.462 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.446+0000 7fe4837fe640 1 --2- 172.21.15.16:0/3432031456 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe470004f50 0x7fe47000b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:54.462 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.446+0000 7fe4837fe640 1 -- 172.21.15.16:0/3432031456 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe48415aba0 con 0x7fe470004f50 2023-05-22T15:26:54.462 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.446+0000 7fe4837fe640 1 -- 172.21.15.16:0/3432031456 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe48415ae80 con 0x7fe470004360 2023-05-22T15:26:54.462 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.446+0000 7fe4837fe640 1 -- 172.21.15.16:0/3432031456 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe47000bed0 con 0x7fe470003980 2023-05-22T15:26:54.462 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.446+0000 7fe4892b3640 1 --2- 172.21.15.16:0/3432031456 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe470004f50 0x7fe47000b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:54.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.446+0000 7fe488ab2640 1 --2- 172.21.15.16:0/3432031456 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe470003980 0x7fe470003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:54.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.446+0000 7fe483fff640 1 --2- 172.21.15.16:0/3432031456 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe470004360 0x7fe470004830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:54.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.446+0000 7fe4892b3640 1 -- 172.21.15.16:0/3432031456 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe470003980 msgr2=0x7fe470003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:26:54.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.446+0000 7fe4892b3640 1 --2- 172.21.15.16:0/3432031456 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe470003980 0x7fe470003d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:54.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.446+0000 7fe4892b3640 1 -- 172.21.15.16:0/3432031456 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe470004360 msgr2=0x7fe470004830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:26:54.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.446+0000 7fe4892b3640 1 --2- 172.21.15.16:0/3432031456 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe470004360 0x7fe470004830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:54.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.446+0000 7fe4892b3640 1 -- 172.21.15.16:0/3432031456 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe470003530 con 0x7fe470004f50 2023-05-22T15:26:54.466 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.450+0000 7fe4892b3640 1 --2- 172.21.15.16:0/3432031456 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe470004f50 0x7fe47000b990 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fe47400c240 tx=0x7fe47401cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7923af044729fb server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:26:54.466 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.450+0000 7fe4837fe640 1 -- 172.21.15.16:0/3432031456 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4740a5960 con 0x7fe470004f50 2023-05-22T15:26:54.466 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.450+0000 7fe4837fe640 1 -- 172.21.15.16:0/3432031456 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4740a5ac0 con 0x7fe470004f50 2023-05-22T15:26:54.466 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.450+0000 7fe4837fe640 1 -- 172.21.15.16:0/3432031456 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4740058e0 con 0x7fe470004f50 2023-05-22T15:26:54.466 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.450+0000 7fe48ad16640 1 -- 172.21.15.16:0/3432031456 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe470004f50 msgr2=0x7fe47000b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:54.467 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.450+0000 7fe48ad16640 1 --2- 172.21.15.16:0/3432031456 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe470004f50 0x7fe47000b990 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fe47400c240 tx=0x7fe47401cc60 comp rx=0 tx=0).stop 2023-05-22T15:26:54.467 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.450+0000 7fe4892b3640 1 -- 172.21.15.16:0/3432031456 reap_dead start 2023-05-22T15:26:54.467 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.450+0000 7fe48ad16640 1 -- 172.21.15.16:0/3432031456 shutdown_connections 2023-05-22T15:26:54.467 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.450+0000 7fe48ad16640 1 -- 172.21.15.16:0/3432031456 >> 172.21.15.16:0/3432031456 conn(0x7fe48406f5c0 msgr2=0x7fe48406f9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:26:54.467 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.450+0000 7fe48ad16640 1 -- 172.21.15.16:0/3432031456 shutdown_connections 2023-05-22T15:26:54.467 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.450+0000 7fe48ad16640 1 -- 172.21.15.16:0/3432031456 wait complete. 2023-05-22T15:26:54.467 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.450+0000 7fe48ad16640 1 Processor -- start 2023-05-22T15:26:54.467 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.450+0000 7fe48ad16640 1 -- start start 2023-05-22T15:26:54.468 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.454+0000 7fe48ad16640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe470004f50 0x7fe4840b5a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:54.468 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.454+0000 7fe48ad16640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe4840a73c0 0x7fe4840c95a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:54.468 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.454+0000 7fe48ad16640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe4841f18c0 0x7fe4841f1d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:54.468 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.454+0000 7fe48ad16640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4841f2270 con 0x7fe470004f50 2023-05-22T15:26:54.468 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.454+0000 7fe48ad16640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4840b33e0 con 0x7fe4841f18c0 2023-05-22T15:26:54.468 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.454+0000 7fe48ad16640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4840b36c0 con 0x7fe4840a73c0 2023-05-22T15:26:54.468 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.454+0000 7fe488ab2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe470004f50 0x7fe4840b5a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:54.469 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.454+0000 7fe488ab2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe470004f50 0x7fe4840b5a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49476/0 (socket says 172.21.15.16:49476) 2023-05-22T15:26:54.469 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.454+0000 7fe488ab2640 1 -- 172.21.15.16:0/3828341565 learned_addr learned my addr 172.21.15.16:0/3828341565 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:26:54.469 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.454+0000 7fe4892b3640 1 --2- 172.21.15.16:0/3828341565 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe4841f18c0 0x7fe4841f1d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:54.469 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.454+0000 7fe483fff640 1 --2- 172.21.15.16:0/3828341565 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe4840a73c0 0x7fe4840c95a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:54.469 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.454+0000 7fe488ab2640 1 -- 172.21.15.16:0/3828341565 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe4840a73c0 msgr2=0x7fe4840c95a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:54.469 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.454+0000 7fe488ab2640 1 --2- 172.21.15.16:0/3828341565 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe4840a73c0 0x7fe4840c95a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:54.469 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.454+0000 7fe488ab2640 1 -- 172.21.15.16:0/3828341565 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe4841f18c0 msgr2=0x7fe4841f1d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:54.470 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.454+0000 7fe488ab2640 1 --2- 172.21.15.16:0/3828341565 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe4841f18c0 0x7fe4841f1d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:54.470 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.454+0000 7fe488ab2640 1 -- 172.21.15.16:0/3828341565 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4840d0e40 con 0x7fe470004f50 2023-05-22T15:26:54.470 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.454+0000 7fe488ab2640 1 --2- 172.21.15.16:0/3828341565 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe470004f50 0x7fe4840b5a70 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fe47c007cd0 tx=0x7fe47c007dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:26:54.470 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.454+0000 7fe4817fa640 1 -- 172.21.15.16:0/3828341565 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe47c00ebf0 con 0x7fe470004f50 2023-05-22T15:26:54.470 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.454+0000 7fe4817fa640 1 -- 172.21.15.16:0/3828341565 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe47c00f040 con 0x7fe470004f50 2023-05-22T15:26:54.470 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.454+0000 7fe48ad16640 1 -- 172.21.15.16:0/3828341565 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4840cf630 con 0x7fe470004f50 2023-05-22T15:26:54.470 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.454+0000 7fe4817fa640 1 -- 172.21.15.16:0/3828341565 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe47c015410 con 0x7fe470004f50 2023-05-22T15:26:54.470 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.454+0000 7fe48ad16640 1 -- 172.21.15.16:0/3828341565 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4840d0fd0 con 0x7fe470004f50 2023-05-22T15:26:54.471 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.454+0000 7fe48ad16640 1 -- 172.21.15.16:0/3828341565 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe48404dbe0 con 0x7fe470004f50 2023-05-22T15:26:54.471 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.454+0000 7fe4817fa640 1 -- 172.21.15.16:0/3828341565 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7fe47c015570 con 0x7fe470004f50 2023-05-22T15:26:54.471 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.454+0000 7fe4817fa640 1 --2- 172.21.15.16:0/3828341565 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fe4640653b0 0x7fe464067860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:54.471 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.454+0000 7fe4817fa640 1 -- 172.21.15.16:0/3828341565 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6813+0+0 (secure 0 0 0) 0x7fe47c0833a0 con 0x7fe470004f50 2023-05-22T15:26:54.471 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.454+0000 7fe483fff640 1 --2- 172.21.15.16:0/3828341565 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fe4640653b0 0x7fe464067860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:54.471 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.454+0000 7fe483fff640 1 --2- 172.21.15.16:0/3828341565 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fe4640653b0 0x7fe464067860 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fe4840c9f50 tx=0x7fe46c008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:26:54.472 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.458+0000 7fe4817fa640 1 -- 172.21.15.16:0/3828341565 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fe47c050d30 con 0x7fe470004f50 2023-05-22T15:26:54.625 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.614+0000 7fe48ad16640 1 -- 172.21.15.16:0/3828341565 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe4841595b0 con 0x7fe470004f50 2023-05-22T15:26:54.626 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.618+0000 7fe4817fa640 1 -- 172.21.15.16:0/3828341565 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) v1 ==== 74+0+19604 (secure 0 0 0) 0x7fe47c054350 con 0x7fe470004f50 2023-05-22T15:26:54.627 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:26:54.627 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":27,"fsid":"b2719834-7847-4d98-90a6-f65d57f85c49","created":"2023-05-22T15:26:25.259314+0000","modified":"2023-05-22T15:26:53.971378+0000","last_up_change":"2023-05-22T15:26:36.448524+0000","last_in_change":"2023-05-22T15:26:27.675668+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-05-22T15:26:38.019781+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-05-22T15:26:43.570514+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-05-22T15:26:47.879159+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-05-22T15:26:48.522130+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data_ec","create_time":"2023-05-22T15:26:51.616142+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"9e7ff854-b25f-4cbb-a5c2-1799d9c53d2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":37021},{"type":"v1","addr":"172.21.15.16:6825","nonce":37021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":37021},{"type":"v1","addr":"172.21.15.16:6827","nonce":37021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":37021},{"type":"v1","addr":"172.21.15.16:6831","nonce":37021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":37021},{"type":"v1","addr":"172.21.15.16:6829","nonce":37021}]},"public_addr":"172.21.15.16:6825/37021","cluster_addr":"172.21.15.16:6827/37021","heartbeat_back_addr":"172.21.15.16:6831/37021","heartbeat_front_addr":"172.21.15.16:6829/37021","state":["exists","up"]},{"osd":1,"uuid":"4702fbc9-a927-4565-aca5-41dc69cc8264","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":37008},{"type":"v1","addr":"172.21.15.16:6801","nonce":37008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":37008},{"type":"v1","addr":"172.21.15.16:6803","nonce":37008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":37008},{"type":"v1","addr":"172.21.15.16:6807","nonce":37008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":37008},{"type":"v1","addr":"172.21.15.16:6805","nonce":37008}]},"public_addr":"172.21.15.16:6801/37008","cluster_addr":"172.21.15.16:6803/37008","heartbeat_back_addr":"172.21.15.16:6807/37008","heartbeat_front_addr":"172.21.15.16:6805/37008","state":["exists","up"]},{"osd":2,"uuid":"1e7bba46-4eba-47ef-bb81-5e2d371638ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":37009},{"type":"v1","addr":"172.21.15.16:6809","nonce":37009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":37009},{"type":"v1","addr":"172.21.15.16:6811","nonce":37009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":37009},{"type":"v1","addr":"172.21.15.16:6815","nonce":37009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":37009},{"type":"v1","addr":"172.21.15.16:6813","nonce":37009}]},"public_addr":"172.21.15.16:6809/37009","cluster_addr":"172.21.15.16:6811/37009","heartbeat_back_addr":"172.21.15.16:6815/37009","heartbeat_front_addr":"172.21.15.16:6813/37009","state":["exists","up"]},{"osd":3,"uuid":"383cacc0-5a88-4904-990e-498ca59ef48a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":37020},{"type":"v1","addr":"172.21.15.16:6817","nonce":37020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":37020},{"type":"v1","addr":"172.21.15.16:6819","nonce":37020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":37020},{"type":"v1","addr":"172.21.15.16:6823","nonce":37020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":37020},{"type":"v1","addr":"172.21.15.16:6821","nonce":37020}]},"public_addr":"172.21.15.16:6817/37020","cluster_addr":"172.21.15.16:6819/37020","heartbeat_back_addr":"172.21.15.16:6823/37020","heartbeat_front_addr":"172.21.15.16:6821/37020","state":["exists","up"]},{"osd":4,"uuid":"970500ef-95b7-42c1-b080-eed4fae7013d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":37353},{"type":"v1","addr":"172.21.15.62:6817","nonce":37353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6819","nonce":37353},{"type":"v1","addr":"172.21.15.62:6822","nonce":37353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37353},{"type":"v1","addr":"172.21.15.62:6831","nonce":37353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37353},{"type":"v1","addr":"172.21.15.62:6826","nonce":37353}]},"public_addr":"172.21.15.62:6817/37353","cluster_addr":"172.21.15.62:6822/37353","heartbeat_back_addr":"172.21.15.62:6831/37353","heartbeat_front_addr":"172.21.15.62:6826/37353","state":["exists","up"]},{"osd":5,"uuid":"022d67db-d4d8-445b-b1b9-8b446a00f5c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":37352},{"type":"v1","addr":"172.21.15.62:6801","nonce":37352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":37352},{"type":"v1","addr":"172.21.15.62:6803","nonce":37352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":37352},{"type":"v1","addr":"172.21.15.62:6807","nonce":37352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":37352},{"type":"v1","addr":"172.21.15.62:6805","nonce":37352}]},"public_addr":"172.21.15.62:6801/37352","cluster_addr":"172.21.15.62:6803/37352","heartbeat_back_addr":"172.21.15.62:6807/37352","heartbeat_front_addr":"172.21.15.62:6805/37352","state":["exists","up"]},{"osd":6,"uuid":"42bd372b-6688-4a1b-8407-e16e8c266e33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":37354},{"type":"v1","addr":"172.21.15.62:6820","nonce":37354}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":37354},{"type":"v1","addr":"172.21.15.62:6823","nonce":37354}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6829","nonce":37354},{"type":"v1","addr":"172.21.15.62:6830","nonce":37354}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6825","nonce":37354},{"type":"v1","addr":"172.21.15.62:6827","nonce":37354}]},"public_addr":"172.21.15.62:6820/37354","cluster_addr":"172.21.15.62:6823/37354","heartbeat_back_addr":"172.21.15.62:6830/37354","heartbeat_front_addr":"172.21.15.62:6827/37354","state":["exists","up"]},{"osd":7,"uuid":"ac70b2a3-f41b-4bf2-b227-7f2cbb55426d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37355},{"type":"v1","addr":"172.21.15.62:6809","nonce":37355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37355},{"type":"v1","addr":"172.21.15.62:6811","nonce":37355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":37355},{"type":"v1","addr":"172.21.15.62:6815","nonce":37355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":37355},{"type":"v1","addr":"172.21.15.62:6813","nonce":37355}]},"public_addr":"172.21.15.62:6809/37355","cluster_addr":"172.21.15.62:6811/37355","heartbeat_back_addr":"172.21.15.62:6815/37355","heartbeat_front_addr":"172.21.15.62:6813/37355","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.368896+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.107273+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.276888+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.287502+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.124791+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.251937+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.255679+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.232797+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-05-22T15:26:54.629 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.618+0000 7fe48ad16640 1 -- 172.21.15.16:0/3828341565 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fe4640653b0 msgr2=0x7fe464067860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:54.629 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.618+0000 7fe48ad16640 1 --2- 172.21.15.16:0/3828341565 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fe4640653b0 0x7fe464067860 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fe4840c9f50 tx=0x7fe46c008040 comp rx=0 tx=0).stop 2023-05-22T15:26:54.629 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.618+0000 7fe48ad16640 1 -- 172.21.15.16:0/3828341565 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe470004f50 msgr2=0x7fe4840b5a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:54.629 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.618+0000 7fe48ad16640 1 --2- 172.21.15.16:0/3828341565 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe470004f50 0x7fe4840b5a70 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fe47c007cd0 tx=0x7fe47c007dc0 comp rx=0 tx=0).stop 2023-05-22T15:26:54.629 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.618+0000 7fe48ad16640 1 -- 172.21.15.16:0/3828341565 shutdown_connections 2023-05-22T15:26:54.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.618+0000 7fe48ad16640 1 --2- 172.21.15.16:0/3828341565 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe4841f18c0 0x7fe4841f1d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:54.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.618+0000 7fe48ad16640 1 --2- 172.21.15.16:0/3828341565 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fe4640653b0 0x7fe464067860 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:54.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.618+0000 7fe48ad16640 1 --2- 172.21.15.16:0/3828341565 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe4840a73c0 0x7fe4840c95a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:54.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.618+0000 7fe48ad16640 1 --2- 172.21.15.16:0/3828341565 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe470004f50 0x7fe4840b5a70 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:54.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.618+0000 7fe48ad16640 1 -- 172.21.15.16:0/3828341565 >> 172.21.15.16:0/3828341565 conn(0x7fe48406f5c0 msgr2=0x7fe484158b00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:26:54.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.618+0000 7fe48ad16640 1 -- 172.21.15.16:0/3828341565 shutdown_connections 2023-05-22T15:26:54.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.618+0000 7fe48ad16640 1 -- 172.21.15.16:0/3828341565 wait complete. 2023-05-22T15:26:54.646 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-05-22T15:26:54.767 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.758+0000 7f5280d45640 1 Processor -- start 2023-05-22T15:26:54.767 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.758+0000 7f5280d45640 1 -- start start 2023-05-22T15:26:54.768 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.758+0000 7f5280d45640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f527c153fe0 0x7f527c06d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:54.769 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.758+0000 7f5280d45640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f527c0cae20 0x7f527c06fef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:54.769 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.758+0000 7f5280d45640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f527c069720 con 0x7f527c070430 2023-05-22T15:26:54.769 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.758+0000 7f5280d45640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f527c0708b0 con 0x7f527c153fe0 2023-05-22T15:26:54.769 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.758+0000 7f5280d45640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f527c1543b0 con 0x7f527c0cae20 2023-05-22T15:26:54.769 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.758+0000 7f527a59c640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f527c153fe0 0x7f527c06d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:54.769 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.758+0000 7f527ad9d640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f527c070430 0x7f527c0ad410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:43468/0 (socket says 172.21.15.16:43468) 2023-05-22T15:26:54.770 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.758+0000 7f527ad9d640 1 -- 172.21.15.16:0/905666658 learned_addr learned my addr 172.21.15.16:0/905666658 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:26:54.772 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.758+0000 7f5279d9b640 1 --2- 172.21.15.16:0/905666658 >> v2:172.21.15.62:3300/0 conn(0x7f527c0cae20 0x7f527c06fef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:54.772 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.758+0000 7f527a59c640 1 -- 172.21.15.16:0/905666658 >> v2:172.21.15.62:3300/0 conn(0x7f527c0cae20 msgr2=0x7f527c06fef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:26:54.773 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.758+0000 7f527a59c640 1 --2- 172.21.15.16:0/905666658 >> v2:172.21.15.62:3300/0 conn(0x7f527c0cae20 0x7f527c06fef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:54.773 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.758+0000 7f527a59c640 1 -- 172.21.15.16:0/905666658 >> v1:172.21.15.16:6789/0 conn(0x7f527c070430 legacy=0x7f527c0ad410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:54.773 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.758+0000 7f527a59c640 1 -- 172.21.15.16:0/905666658 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f527c15e360 con 0x7f527c153fe0 2023-05-22T15:26:54.775 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.762+0000 7f527a59c640 1 --2- 172.21.15.16:0/905666658 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f527c153fe0 0x7f527c06d3f0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f527c0b46f0 tx=0x7f526c01c2f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c46d86dc53aee991 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:26:54.776 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.762+0000 7f527959a640 1 -- 172.21.15.16:0/905666658 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f526c0ab600 con 0x7f527c153fe0 2023-05-22T15:26:54.776 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.762+0000 7f527959a640 1 -- 172.21.15.16:0/905666658 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f526c0a7050 con 0x7f527c153fe0 2023-05-22T15:26:54.776 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.762+0000 7f527959a640 1 -- 172.21.15.16:0/905666658 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f526c0ab980 con 0x7f527c153fe0 2023-05-22T15:26:54.776 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.762+0000 7f5280d45640 1 -- 172.21.15.16:0/905666658 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f527c153fe0 msgr2=0x7f527c06d3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:54.776 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.762+0000 7f5280d45640 1 --2- 172.21.15.16:0/905666658 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f527c153fe0 0x7f527c06d3f0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f527c0b46f0 tx=0x7f526c01c2f0 comp rx=0 tx=0).stop 2023-05-22T15:26:54.776 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.762+0000 7f5280d45640 1 -- 172.21.15.16:0/905666658 shutdown_connections 2023-05-22T15:26:54.777 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.762+0000 7f5280d45640 1 --2- 172.21.15.16:0/905666658 >> v2:172.21.15.62:3300/0 conn(0x7f527c0cae20 0x7f527c06fef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:54.777 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.762+0000 7f5280d45640 1 --2- 172.21.15.16:0/905666658 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f527c153fe0 0x7f527c06d3f0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:54.777 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.762+0000 7f5280d45640 1 -- 172.21.15.16:0/905666658 >> 172.21.15.16:0/905666658 conn(0x7f527c0b1110 msgr2=0x7f527c0ce830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:26:54.777 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.762+0000 7f5280d45640 1 -- 172.21.15.16:0/905666658 shutdown_connections 2023-05-22T15:26:54.777 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.762+0000 7f5280d45640 1 -- 172.21.15.16:0/905666658 wait complete. 2023-05-22T15:26:54.777 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.762+0000 7f5280d45640 1 Processor -- start 2023-05-22T15:26:54.777 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.762+0000 7f5280d45640 1 -- start start 2023-05-22T15:26:54.778 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.762+0000 7f5280d45640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f527c070430 0x7f527c1588f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:54.778 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.762+0000 7f5280d45640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f527c0cae20 0x7f527c156920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:54.778 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.762+0000 7f5280d45640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f527c153fe0 0x7f527c144990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:54.778 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.762+0000 7f5280d45640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f527c15c670 con 0x7f527c070430 2023-05-22T15:26:54.778 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.762+0000 7f5280d45640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f527c145700 con 0x7f527c0cae20 2023-05-22T15:26:54.778 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.762+0000 7f5280d45640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f527c157b30 con 0x7f527c153fe0 2023-05-22T15:26:54.778 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.762+0000 7f527a59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f527c070430 0x7f527c1588f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:54.779 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.762+0000 7f527a59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f527c070430 0x7f527c1588f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49488/0 (socket says 172.21.15.16:49488) 2023-05-22T15:26:54.779 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.762+0000 7f527a59c640 1 -- 172.21.15.16:0/3452574803 learned_addr learned my addr 172.21.15.16:0/3452574803 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:26:54.779 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.766+0000 7f5279d9b640 1 --2- 172.21.15.16:0/3452574803 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f527c0cae20 0x7f527c156920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:54.779 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.766+0000 7f527a59c640 1 -- 172.21.15.16:0/3452574803 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f527c153fe0 msgr2=0x7f527c144990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:54.779 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.766+0000 7f527ad9d640 1 --2- 172.21.15.16:0/3452574803 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f527c153fe0 0x7f527c144990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:54.779 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.766+0000 7f527a59c640 1 --2- 172.21.15.16:0/3452574803 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f527c153fe0 0x7f527c144990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:54.779 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.766+0000 7f527a59c640 1 -- 172.21.15.16:0/3452574803 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f527c0cae20 msgr2=0x7f527c156920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:54.779 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.766+0000 7f527a59c640 1 --2- 172.21.15.16:0/3452574803 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f527c0cae20 0x7f527c156920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:54.780 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.766+0000 7f527a59c640 1 -- 172.21.15.16:0/3452574803 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f526c009420 con 0x7f527c070430 2023-05-22T15:26:54.780 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.766+0000 7f527a59c640 1 --2- 172.21.15.16:0/3452574803 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f527c070430 0x7f527c1588f0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f526c0ae040 tx=0x7f526c019090 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:26:54.780 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.766+0000 7f526affd640 1 -- 172.21.15.16:0/3452574803 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f526c0b48e0 con 0x7f527c070430 2023-05-22T15:26:54.780 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.766+0000 7f526affd640 1 -- 172.21.15.16:0/3452574803 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f526c0b4a60 con 0x7f527c070430 2023-05-22T15:26:54.780 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.766+0000 7f526affd640 1 -- 172.21.15.16:0/3452574803 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f526c0b4e10 con 0x7f527c070430 2023-05-22T15:26:54.780 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.766+0000 7f5280d45640 1 -- 172.21.15.16:0/3452574803 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f527c15d880 con 0x7f527c070430 2023-05-22T15:26:54.780 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.766+0000 7f5280d45640 1 -- 172.21.15.16:0/3452574803 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f527c15d5a0 con 0x7f527c070430 2023-05-22T15:26:54.781 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.766+0000 7f5280d45640 1 -- 172.21.15.16:0/3452574803 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f527c15db60 con 0x7f527c070430 2023-05-22T15:26:54.782 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.766+0000 7f526affd640 1 -- 172.21.15.16:0/3452574803 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7f526c0b2040 con 0x7f527c070430 2023-05-22T15:26:54.782 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.766+0000 7f526affd640 1 --2- 172.21.15.16:0/3452574803 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f525c065380 0x7f525c067830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:54.782 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.766+0000 7f526affd640 1 -- 172.21.15.16:0/3452574803 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f526c11d360 con 0x7f527c070430 2023-05-22T15:26:54.782 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.766+0000 7f5279d9b640 1 --2- 172.21.15.16:0/3452574803 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f525c065380 0x7f525c067830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:54.782 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.766+0000 7f5279d9b640 1 --2- 172.21.15.16:0/3452574803 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f525c065380 0x7f525c067830 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f527c1f25a0 tx=0x7f5270008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:26:54.783 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.770+0000 7f526affd640 1 -- 172.21.15.16:0/3452574803 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f526c0eae60 con 0x7f527c070430 2023-05-22T15:26:54.928 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.918+0000 7f5280d45640 1 -- 172.21.15.16:0/3452574803 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f527c066aa0 con 0x7f527c070430 2023-05-22T15:26:54.930 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.922+0000 7f526affd640 1 -- 172.21.15.16:0/3452574803 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) v1 ==== 74+0+19604 (secure 0 0 0) 0x7f526c0ee480 con 0x7f527c070430 2023-05-22T15:26:54.931 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:26:54.931 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":27,"fsid":"b2719834-7847-4d98-90a6-f65d57f85c49","created":"2023-05-22T15:26:25.259314+0000","modified":"2023-05-22T15:26:53.971378+0000","last_up_change":"2023-05-22T15:26:36.448524+0000","last_in_change":"2023-05-22T15:26:27.675668+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-05-22T15:26:38.019781+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-05-22T15:26:43.570514+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-05-22T15:26:47.879159+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-05-22T15:26:48.522130+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data_ec","create_time":"2023-05-22T15:26:51.616142+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"9e7ff854-b25f-4cbb-a5c2-1799d9c53d2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":37021},{"type":"v1","addr":"172.21.15.16:6825","nonce":37021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":37021},{"type":"v1","addr":"172.21.15.16:6827","nonce":37021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":37021},{"type":"v1","addr":"172.21.15.16:6831","nonce":37021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":37021},{"type":"v1","addr":"172.21.15.16:6829","nonce":37021}]},"public_addr":"172.21.15.16:6825/37021","cluster_addr":"172.21.15.16:6827/37021","heartbeat_back_addr":"172.21.15.16:6831/37021","heartbeat_front_addr":"172.21.15.16:6829/37021","state":["exists","up"]},{"osd":1,"uuid":"4702fbc9-a927-4565-aca5-41dc69cc8264","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":37008},{"type":"v1","addr":"172.21.15.16:6801","nonce":37008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":37008},{"type":"v1","addr":"172.21.15.16:6803","nonce":37008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":37008},{"type":"v1","addr":"172.21.15.16:6807","nonce":37008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":37008},{"type":"v1","addr":"172.21.15.16:6805","nonce":37008}]},"public_addr":"172.21.15.16:6801/37008","cluster_addr":"172.21.15.16:6803/37008","heartbeat_back_addr":"172.21.15.16:6807/37008","heartbeat_front_addr":"172.21.15.16:6805/37008","state":["exists","up"]},{"osd":2,"uuid":"1e7bba46-4eba-47ef-bb81-5e2d371638ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":37009},{"type":"v1","addr":"172.21.15.16:6809","nonce":37009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":37009},{"type":"v1","addr":"172.21.15.16:6811","nonce":37009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":37009},{"type":"v1","addr":"172.21.15.16:6815","nonce":37009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":37009},{"type":"v1","addr":"172.21.15.16:6813","nonce":37009}]},"public_addr":"172.21.15.16:6809/37009","cluster_addr":"172.21.15.16:6811/37009","heartbeat_back_addr":"172.21.15.16:6815/37009","heartbeat_front_addr":"172.21.15.16:6813/37009","state":["exists","up"]},{"osd":3,"uuid":"383cacc0-5a88-4904-990e-498ca59ef48a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":37020},{"type":"v1","addr":"172.21.15.16:6817","nonce":37020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":37020},{"type":"v1","addr":"172.21.15.16:6819","nonce":37020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":37020},{"type":"v1","addr":"172.21.15.16:6823","nonce":37020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":37020},{"type":"v1","addr":"172.21.15.16:6821","nonce":37020}]},"public_addr":"172.21.15.16:6817/37020","cluster_addr":"172.21.15.16:6819/37020","heartbeat_back_addr":"172.21.15.16:6823/37020","heartbeat_front_addr":"172.21.15.16:6821/37020","state":["exists","up"]},{"osd":4,"uuid":"970500ef-95b7-42c1-b080-eed4fae7013d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":37353},{"type":"v1","addr":"172.21.15.62:6817","nonce":37353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6819","nonce":37353},{"type":"v1","addr":"172.21.15.62:6822","nonce":37353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37353},{"type":"v1","addr":"172.21.15.62:6831","nonce":37353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37353},{"type":"v1","addr":"172.21.15.62:6826","nonce":37353}]},"public_addr":"172.21.15.62:6817/37353","cluster_addr":"172.21.15.62:6822/37353","heartbeat_back_addr":"172.21.15.62:6831/37353","heartbeat_front_addr":"172.21.15.62:6826/37353","state":["exists","up"]},{"osd":5,"uuid":"022d67db-d4d8-445b-b1b9-8b446a00f5c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":37352},{"type":"v1","addr":"172.21.15.62:6801","nonce":37352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":37352},{"type":"v1","addr":"172.21.15.62:6803","nonce":37352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":37352},{"type":"v1","addr":"172.21.15.62:6807","nonce":37352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":37352},{"type":"v1","addr":"172.21.15.62:6805","nonce":37352}]},"public_addr":"172.21.15.62:6801/37352","cluster_addr":"172.21.15.62:6803/37352","heartbeat_back_addr":"172.21.15.62:6807/37352","heartbeat_front_addr":"172.21.15.62:6805/37352","state":["exists","up"]},{"osd":6,"uuid":"42bd372b-6688-4a1b-8407-e16e8c266e33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":37354},{"type":"v1","addr":"172.21.15.62:6820","nonce":37354}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":37354},{"type":"v1","addr":"172.21.15.62:6823","nonce":37354}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6829","nonce":37354},{"type":"v1","addr":"172.21.15.62:6830","nonce":37354}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6825","nonce":37354},{"type":"v1","addr":"172.21.15.62:6827","nonce":37354}]},"public_addr":"172.21.15.62:6820/37354","cluster_addr":"172.21.15.62:6823/37354","heartbeat_back_addr":"172.21.15.62:6830/37354","heartbeat_front_addr":"172.21.15.62:6827/37354","state":["exists","up"]},{"osd":7,"uuid":"ac70b2a3-f41b-4bf2-b227-7f2cbb55426d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37355},{"type":"v1","addr":"172.21.15.62:6809","nonce":37355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37355},{"type":"v1","addr":"172.21.15.62:6811","nonce":37355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":37355},{"type":"v1","addr":"172.21.15.62:6815","nonce":37355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":37355},{"type":"v1","addr":"172.21.15.62:6813","nonce":37355}]},"public_addr":"172.21.15.62:6809/37355","cluster_addr":"172.21.15.62:6811/37355","heartbeat_back_addr":"172.21.15.62:6815/37355","heartbeat_front_addr":"172.21.15.62:6813/37355","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.368896+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.107273+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.276888+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.287502+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.124791+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.251937+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.255679+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.232797+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-05-22T15:26:54.933 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.922+0000 7f5280d45640 1 -- 172.21.15.16:0/3452574803 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f525c065380 msgr2=0x7f525c067830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:54.933 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.922+0000 7f5280d45640 1 --2- 172.21.15.16:0/3452574803 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f525c065380 0x7f525c067830 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f527c1f25a0 tx=0x7f5270008040 comp rx=0 tx=0).stop 2023-05-22T15:26:54.933 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.922+0000 7f5280d45640 1 -- 172.21.15.16:0/3452574803 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f527c070430 msgr2=0x7f527c1588f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:54.933 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.922+0000 7f5280d45640 1 --2- 172.21.15.16:0/3452574803 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f527c070430 0x7f527c1588f0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f526c0ae040 tx=0x7f526c019090 comp rx=0 tx=0).stop 2023-05-22T15:26:54.933 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.922+0000 7f5280d45640 1 -- 172.21.15.16:0/3452574803 shutdown_connections 2023-05-22T15:26:54.933 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.922+0000 7f5280d45640 1 --2- 172.21.15.16:0/3452574803 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f525c065380 0x7f525c067830 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:54.934 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.922+0000 7f5280d45640 1 --2- 172.21.15.16:0/3452574803 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f527c153fe0 0x7f527c144990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:54.934 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.922+0000 7f5280d45640 1 --2- 172.21.15.16:0/3452574803 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f527c0cae20 0x7f527c156920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:54.934 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.922+0000 7f5280d45640 1 --2- 172.21.15.16:0/3452574803 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f527c070430 0x7f527c1588f0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:54.934 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.922+0000 7f5280d45640 1 -- 172.21.15.16:0/3452574803 >> 172.21.15.16:0/3452574803 conn(0x7f527c0b1110 msgr2=0x7f527c0aeb40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:26:54.934 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.922+0000 7f5280d45640 1 -- 172.21.15.16:0/3452574803 shutdown_connections 2023-05-22T15:26:54.934 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:54.922+0000 7f5280d45640 1 -- 172.21.15.16:0/3452574803 wait complete. 2023-05-22T15:26:54.965 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-05-22T15:26:55.089 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.078+0000 7fcf0f419640 1 Processor -- start 2023-05-22T15:26:55.090 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.078+0000 7fcf0f419640 1 -- start start 2023-05-22T15:26:55.091 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.082+0000 7fcf0f419640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fcf08158660 0x7fcf080cdfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:55.091 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.082+0000 7fcf0f419640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fcf080cce20 0x7fcf080ce4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:55.091 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.082+0000 7fcf0f419640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf080c92b0 con 0x7fcf08158290 2023-05-22T15:26:55.091 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.082+0000 7fcf0f419640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf080cc060 con 0x7fcf080cce20 2023-05-22T15:26:55.091 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.082+0000 7fcf0f419640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf080cc340 con 0x7fcf08158660 2023-05-22T15:26:55.091 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.082+0000 7fcf0d9b6640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fcf080cce20 0x7fcf080ce4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:55.092 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.082+0000 7fcf0d9b6640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fcf080cce20 0x7fcf080ce4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49490/0 (socket says 172.21.15.16:49490) 2023-05-22T15:26:55.092 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.082+0000 7fcf0d9b6640 1 -- 172.21.15.16:0/1617062150 learned_addr learned my addr 172.21.15.16:0/1617062150 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:26:55.092 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.082+0000 7fcf0c9b4640 1 --2- 172.21.15.16:0/1617062150 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fcf08158660 0x7fcf080cdfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:55.092 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.082+0000 7fcf0d9b6640 1 -- 172.21.15.16:0/1617062150 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fcf08158660 msgr2=0x7fcf080cdfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:26:55.092 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.082+0000 7fcf0d9b6640 1 --2- 172.21.15.16:0/1617062150 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fcf08158660 0x7fcf080cdfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:55.092 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.082+0000 7fcf0d9b6640 1 -- 172.21.15.16:0/1617062150 >> v1:172.21.15.16:6789/0 conn(0x7fcf08158290 legacy=0x7fcf080c8880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:55.092 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.082+0000 7fcf0d9b6640 1 -- 172.21.15.16:0/1617062150 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf080cc620 con 0x7fcf080cce20 2023-05-22T15:26:55.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.082+0000 7fcf0c9b4640 1 --2- 172.21.15.16:0/1617062150 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fcf08158660 0x7fcf080cdfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:26:55.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.082+0000 7fcf0d9b6640 1 --2- 172.21.15.16:0/1617062150 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcf080cce20 0x7fcf080ce4f0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fcf00009e50 tx=0x7fcf0001d440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21f2b9628507a418 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:26:55.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.082+0000 7fcefffff640 1 -- 172.21.15.16:0/1617062150 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf000a5850 con 0x7fcf080cce20 2023-05-22T15:26:55.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.082+0000 7fcefffff640 1 -- 172.21.15.16:0/1617062150 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf000a59b0 con 0x7fcf080cce20 2023-05-22T15:26:55.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.082+0000 7fcefffff640 1 -- 172.21.15.16:0/1617062150 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf000a5d40 con 0x7fcf080cce20 2023-05-22T15:26:55.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.082+0000 7fcf0f419640 1 -- 172.21.15.16:0/1617062150 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcf080cce20 msgr2=0x7fcf080ce4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:55.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.082+0000 7fcf0f419640 1 --2- 172.21.15.16:0/1617062150 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcf080cce20 0x7fcf080ce4f0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fcf00009e50 tx=0x7fcf0001d440 comp rx=0 tx=0).stop 2023-05-22T15:26:55.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.082+0000 7fcf0f419640 1 -- 172.21.15.16:0/1617062150 shutdown_connections 2023-05-22T15:26:55.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.082+0000 7fcf0f419640 1 --2- 172.21.15.16:0/1617062150 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcf080cce20 0x7fcf080ce4f0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:55.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.082+0000 7fcf0f419640 1 --2- 172.21.15.16:0/1617062150 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fcf08158660 0x7fcf080cdfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:55.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.082+0000 7fcf0f419640 1 -- 172.21.15.16:0/1617062150 >> 172.21.15.16:0/1617062150 conn(0x7fcf08069720 msgr2=0x7fcf08069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:26:55.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.082+0000 7fcf0f419640 1 -- 172.21.15.16:0/1617062150 shutdown_connections 2023-05-22T15:26:55.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.082+0000 7fcf0f419640 1 -- 172.21.15.16:0/1617062150 wait complete. 2023-05-22T15:26:55.096 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.086+0000 7fcf0f419640 1 Processor -- start 2023-05-22T15:26:55.097 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.086+0000 7fcf0f419640 1 -- start start 2023-05-22T15:26:55.097 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.086+0000 7fcf0f419640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcf080cce20 0x7fcf080c5560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:55.097 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.086+0000 7fcf0f419640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcf08158290 0x7fcf08149da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:55.097 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.086+0000 7fcf0f419640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fcf08158660 0x7fcf0814a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:55.097 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.086+0000 7fcf0f419640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf080b3640 con 0x7fcf08158290 2023-05-22T15:26:55.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.086+0000 7fcf0f419640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf0814a820 con 0x7fcf080cce20 2023-05-22T15:26:55.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.086+0000 7fcf0f419640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf080b29c0 con 0x7fcf08158660 2023-05-22T15:26:55.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.086+0000 7fcf0c9b4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcf08158290 0x7fcf08149da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:55.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.086+0000 7fcf0c9b4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcf08158290 0x7fcf08149da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49506/0 (socket says 172.21.15.16:49506) 2023-05-22T15:26:55.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.086+0000 7fcf0c9b4640 1 -- 172.21.15.16:0/701177511 learned_addr learned my addr 172.21.15.16:0/701177511 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:26:55.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.086+0000 7fcf0d9b6640 1 --2- 172.21.15.16:0/701177511 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fcf08158660 0x7fcf0814a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:55.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.086+0000 7fcf0d1b5640 1 --2- 172.21.15.16:0/701177511 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcf080cce20 0x7fcf080c5560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:55.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.086+0000 7fcf0c9b4640 1 -- 172.21.15.16:0/701177511 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fcf08158660 msgr2=0x7fcf0814a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:55.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.086+0000 7fcf0c9b4640 1 --2- 172.21.15.16:0/701177511 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fcf08158660 0x7fcf0814a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:55.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.086+0000 7fcf0c9b4640 1 -- 172.21.15.16:0/701177511 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcf080cce20 msgr2=0x7fcf080c5560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:55.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.086+0000 7fcf0c9b4640 1 --2- 172.21.15.16:0/701177511 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcf080cce20 0x7fcf080c5560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:55.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.086+0000 7fcf0c9b4640 1 -- 172.21.15.16:0/701177511 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf00009b30 con 0x7fcf08158290 2023-05-22T15:26:55.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.086+0000 7fcf0d9b6640 1 --2- 172.21.15.16:0/701177511 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fcf08158660 0x7fcf0814a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T15:26:55.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.086+0000 7fcf0c9b4640 1 --2- 172.21.15.16:0/701177511 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcf08158290 0x7fcf08149da0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fcef800c7f0 tx=0x7fcef800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:26:55.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.086+0000 7fcefdffb640 1 -- 172.21.15.16:0/701177511 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcef8007820 con 0x7fcf08158290 2023-05-22T15:26:55.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.086+0000 7fcefdffb640 1 -- 172.21.15.16:0/701177511 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcef8007980 con 0x7fcf08158290 2023-05-22T15:26:55.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.086+0000 7fcefdffb640 1 -- 172.21.15.16:0/701177511 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcef8007d30 con 0x7fcf08158290 2023-05-22T15:26:55.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.086+0000 7fcf0f419640 1 -- 172.21.15.16:0/701177511 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf0815cea0 con 0x7fcf08158290 2023-05-22T15:26:55.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.086+0000 7fcf0f419640 1 -- 172.21.15.16:0/701177511 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf0815cd10 con 0x7fcf08158290 2023-05-22T15:26:55.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.086+0000 7fcf0f419640 1 -- 172.21.15.16:0/701177511 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf0806f630 con 0x7fcf08158290 2023-05-22T15:26:55.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.090+0000 7fcefdffb640 1 -- 172.21.15.16:0/701177511 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7fcef80113f0 con 0x7fcf08158290 2023-05-22T15:26:55.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.090+0000 7fcefdffb640 1 --2- 172.21.15.16:0/701177511 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fceec065380 0x7fceec067830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:55.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.090+0000 7fcefdffb640 1 -- 172.21.15.16:0/701177511 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6813+0+0 (secure 0 0 0) 0x7fcef8082d50 con 0x7fcf08158290 2023-05-22T15:26:55.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.090+0000 7fcf0d1b5640 1 --2- 172.21.15.16:0/701177511 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fceec065380 0x7fceec067830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:55.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.090+0000 7fcf0d1b5640 1 --2- 172.21.15.16:0/701177511 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fceec065380 0x7fceec067830 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fcf04000f70 tx=0x7fcf04009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:26:55.103 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.094+0000 7fcefdffb640 1 -- 172.21.15.16:0/701177511 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fcef80505e0 con 0x7fcf08158290 2023-05-22T15:26:55.293 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.282+0000 7fcf0f419640 1 -- 172.21.15.16:0/701177511 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcf0804dbe0 con 0x7fcf08158290 2023-05-22T15:26:55.294 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.282+0000 7fcefdffb640 1 -- 172.21.15.16:0/701177511 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) v1 ==== 93+0+4641 (secure 0 0 0) 0x7fcef801f090 con 0x7fcf08158290 2023-05-22T15:26:55.294 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:26:55.295 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":6,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4261,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6834/3661808159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":3661808159},{"type":"v1","addr":"172.21.15.62:6834","nonce":3661808159}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4264,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/2227467222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6833","nonce":2227467222},{"type":"v1","addr":"172.21.15.62:6835","nonce":2227467222}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4284,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6836/2808106087","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2808106087},{"type":"v1","addr":"172.21.15.16:6836","nonce":2808106087}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":6,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-05-22T15:26:49.546008+0000","modified":"2023-05-22T15:26:53.971878+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4287},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4287":{"gid":4287,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.16:6837/283322289","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6835","nonce":283322289},{"type":"v1","addr":"172.21.15.16:6837","nonce":283322289}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2023-05-22T15:26:55.295 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 6 2023-05-22T15:26:55.297 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.286+0000 7fcf0f419640 1 -- 172.21.15.16:0/701177511 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fceec065380 msgr2=0x7fceec067830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:55.297 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.286+0000 7fcf0f419640 1 --2- 172.21.15.16:0/701177511 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fceec065380 0x7fceec067830 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fcf04000f70 tx=0x7fcf04009040 comp rx=0 tx=0).stop 2023-05-22T15:26:55.297 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.286+0000 7fcf0f419640 1 -- 172.21.15.16:0/701177511 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcf08158290 msgr2=0x7fcf08149da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:55.297 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.286+0000 7fcf0f419640 1 --2- 172.21.15.16:0/701177511 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcf08158290 0x7fcf08149da0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fcef800c7f0 tx=0x7fcef800ccb0 comp rx=0 tx=0).stop 2023-05-22T15:26:55.297 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.286+0000 7fcf0f419640 1 -- 172.21.15.16:0/701177511 shutdown_connections 2023-05-22T15:26:55.297 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.286+0000 7fcf0f419640 1 --2- 172.21.15.16:0/701177511 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fceec065380 0x7fceec067830 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:55.298 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.286+0000 7fcf0f419640 1 --2- 172.21.15.16:0/701177511 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fcf08158660 0x7fcf0814a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:55.298 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.286+0000 7fcf0f419640 1 --2- 172.21.15.16:0/701177511 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcf08158290 0x7fcf08149da0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:55.298 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.286+0000 7fcf0f419640 1 --2- 172.21.15.16:0/701177511 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcf080cce20 0x7fcf080c5560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:55.298 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.286+0000 7fcf0f419640 1 -- 172.21.15.16:0/701177511 >> 172.21.15.16:0/701177511 conn(0x7fcf08069720 msgr2=0x7fcf08069af0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:26:55.298 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.286+0000 7fcf0f419640 1 -- 172.21.15.16:0/701177511 shutdown_connections 2023-05-22T15:26:55.298 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.286+0000 7fcf0f419640 1 -- 172.21.15.16:0/701177511 wait complete. 2023-05-22T15:26:55.714 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 6, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-05-22T15:26:49.546008+0000', 'modified': '2023-05-22T15:26:53.971878+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4287}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4287': {'gid': 4287, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.16:6837/283322289', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6835', 'nonce': 283322289}, {'type': 'v1', 'addr': '172.21.15.16:6837', 'nonce': 283322289}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [4, 5], 'metadata_pool': 3, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 1} 2023-05-22T15:26:55.715 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-05-22T15:26:55.715 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2023-05-22T15:26:55.851 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.842+0000 7ff20de6d640 1 Processor -- start 2023-05-22T15:26:55.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.842+0000 7ff20de6d640 1 -- start start 2023-05-22T15:26:55.853 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.842+0000 7ff20de6d640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff208071640 0x7ff208160090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:55.853 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.842+0000 7ff20de6d640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff208069ae0 con 0x7ff2080ce480 2023-05-22T15:26:55.853 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.842+0000 7ff20de6d640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff208069dc0 con 0x7ff2081605d0 2023-05-22T15:26:55.853 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.842+0000 7ff20de6d640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff20815f990 con 0x7ff208071640 2023-05-22T15:26:55.854 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.842+0000 7ff206ffd640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff208071640 0x7ff208160090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:55.854 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.842+0000 7ff206ffd640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff208071640 0x7ff208160090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49518/0 (socket says 172.21.15.16:49518) 2023-05-22T15:26:55.854 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.842+0000 7ff206ffd640 1 -- 172.21.15.16:0/1862031478 learned_addr learned my addr 172.21.15.16:0/1862031478 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:26:55.854 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.842+0000 7ff2077fe640 1 --1- 172.21.15.16:0/1862031478 >> v1:172.21.15.16:6789/0 conn(0x7ff2080ce480 0x7ff2080ce850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:43488/0 (socket says 172.21.15.16:43488) 2023-05-22T15:26:55.854 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.842+0000 7ff206ffd640 1 -- 172.21.15.16:0/1862031478 >> v1:172.21.15.62:6789/0 conn(0x7ff2081605d0 legacy=0x7ff20815c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:26:55.854 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.842+0000 7ff206ffd640 1 -- 172.21.15.16:0/1862031478 >> v1:172.21.15.16:6789/0 conn(0x7ff2080ce480 legacy=0x7ff2080ce850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:55.854 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.842+0000 7ff206ffd640 1 -- 172.21.15.16:0/1862031478 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff20806a0a0 con 0x7ff208071640 2023-05-22T15:26:55.855 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.846+0000 7ff206ffd640 1 --2- 172.21.15.16:0/1862031478 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff208071640 0x7ff208160090 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7ff1f8009870 tx=0x7ff1f801c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dbfb721942163f0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:26:55.855 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.846+0000 7ff2067fc640 1 -- 172.21.15.16:0/1862031478 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1f80ac640 con 0x7ff208071640 2023-05-22T15:26:55.855 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.846+0000 7ff2067fc640 1 -- 172.21.15.16:0/1862031478 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1f80ac7a0 con 0x7ff208071640 2023-05-22T15:26:55.855 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.846+0000 7ff2067fc640 1 -- 172.21.15.16:0/1862031478 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1f80acb30 con 0x7ff208071640 2023-05-22T15:26:55.856 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.846+0000 7ff20de6d640 1 -- 172.21.15.16:0/1862031478 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff208071640 msgr2=0x7ff208160090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:55.856 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.846+0000 7ff20de6d640 1 --2- 172.21.15.16:0/1862031478 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff208071640 0x7ff208160090 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7ff1f8009870 tx=0x7ff1f801c4d0 comp rx=0 tx=0).stop 2023-05-22T15:26:55.856 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.846+0000 7ff20de6d640 1 -- 172.21.15.16:0/1862031478 shutdown_connections 2023-05-22T15:26:55.857 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.846+0000 7ff20de6d640 1 --2- 172.21.15.16:0/1862031478 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff208071640 0x7ff208160090 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:55.857 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.846+0000 7ff20de6d640 1 -- 172.21.15.16:0/1862031478 >> 172.21.15.16:0/1862031478 conn(0x7ff2080cc870 msgr2=0x7ff2080cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:26:55.857 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.846+0000 7ff20de6d640 1 -- 172.21.15.16:0/1862031478 shutdown_connections 2023-05-22T15:26:55.857 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.846+0000 7ff20de6d640 1 -- 172.21.15.16:0/1862031478 wait complete. 2023-05-22T15:26:55.857 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.846+0000 7ff20de6d640 1 Processor -- start 2023-05-22T15:26:55.858 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.846+0000 7ff20de6d640 1 -- start start 2023-05-22T15:26:55.859 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.846+0000 7ff20de6d640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff208071640 0x7ff2081535a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:55.859 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.846+0000 7ff20de6d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff2080ce480 0x7ff2080ad2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:55.859 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.846+0000 7ff20de6d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff2081605d0 0x7ff2080b3af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:55.860 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.846+0000 7ff20de6d640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff208071ac0 con 0x7ff2081605d0 2023-05-22T15:26:55.860 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.846+0000 7ff20de6d640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2080b4ee0 con 0x7ff2080ce480 2023-05-22T15:26:55.860 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.846+0000 7ff20de6d640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2080b4360 con 0x7ff208071640 2023-05-22T15:26:55.860 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.850+0000 7ff207fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff2081605d0 0x7ff2080b3af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:55.860 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.850+0000 7ff207fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff2081605d0 0x7ff2080b3af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49528/0 (socket says 172.21.15.16:49528) 2023-05-22T15:26:55.860 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.850+0000 7ff207fff640 1 -- 172.21.15.16:0/2873015283 learned_addr learned my addr 172.21.15.16:0/2873015283 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:26:55.860 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.850+0000 7ff206ffd640 1 --2- 172.21.15.16:0/2873015283 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff2080ce480 0x7ff2080ad2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:55.861 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.850+0000 7ff2077fe640 1 --2- 172.21.15.16:0/2873015283 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff208071640 0x7ff2081535a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:55.861 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.850+0000 7ff207fff640 1 -- 172.21.15.16:0/2873015283 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff208071640 msgr2=0x7ff2081535a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:55.861 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.850+0000 7ff207fff640 1 --2- 172.21.15.16:0/2873015283 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff208071640 0x7ff2081535a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:55.861 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.850+0000 7ff207fff640 1 -- 172.21.15.16:0/2873015283 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff2080ce480 msgr2=0x7ff2080ad2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:55.861 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.850+0000 7ff207fff640 1 --2- 172.21.15.16:0/2873015283 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff2080ce480 0x7ff2080ad2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:55.861 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.850+0000 7ff207fff640 1 -- 172.21.15.16:0/2873015283 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1f80094d0 con 0x7ff2081605d0 2023-05-22T15:26:55.861 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.850+0000 7ff2077fe640 1 --2- 172.21.15.16:0/2873015283 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff208071640 0x7ff2081535a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:26:55.861 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.850+0000 7ff207fff640 1 --2- 172.21.15.16:0/2873015283 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff2081605d0 0x7ff2080b3af0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7ff20000b970 tx=0x7ff20000be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:26:55.862 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.850+0000 7ff1fffff640 1 -- 172.21.15.16:0/2873015283 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff20000cb40 con 0x7ff2081605d0 2023-05-22T15:26:55.862 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.850+0000 7ff1fffff640 1 -- 172.21.15.16:0/2873015283 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff20000cca0 con 0x7ff2081605d0 2023-05-22T15:26:55.862 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.850+0000 7ff20de6d640 1 -- 172.21.15.16:0/2873015283 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2080c6000 con 0x7ff2081605d0 2023-05-22T15:26:55.862 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.850+0000 7ff1fffff640 1 -- 172.21.15.16:0/2873015283 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff200014680 con 0x7ff2081605d0 2023-05-22T15:26:55.862 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.850+0000 7ff20de6d640 1 -- 172.21.15.16:0/2873015283 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2080c5e70 con 0x7ff2081605d0 2023-05-22T15:26:55.864 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.850+0000 7ff1fffff640 1 -- 172.21.15.16:0/2873015283 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7ff20001d3b0 con 0x7ff2081605d0 2023-05-22T15:26:55.865 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.850+0000 7ff1fffff640 1 --2- 172.21.15.16:0/2873015283 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ff1e8065380 0x7ff1e8067830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:55.865 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.850+0000 7ff1fffff640 1 -- 172.21.15.16:0/2873015283 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6813+0+0 (secure 0 0 0) 0x7ff200082920 con 0x7ff2081605d0 2023-05-22T15:26:55.865 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.850+0000 7ff2077fe640 1 --2- 172.21.15.16:0/2873015283 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ff1e8065380 0x7ff1e8067830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:55.865 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.850+0000 7ff2077fe640 1 --2- 172.21.15.16:0/2873015283 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ff1e8065380 0x7ff1e8067830 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7ff1f0000f70 tx=0x7ff1f0009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:26:55.865 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.850+0000 7ff210f8f000 1 Processor -- start 2023-05-22T15:26:55.865 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.850+0000 7ff210f8f000 1 -- start start 2023-05-22T15:26:55.865 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.854+0000 7ff210f8f000 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559c58a42fd0 0x559c58cb5c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:55.866 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.854+0000 7ff210f8f000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559c58cb6170 0x559c58cb6540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:55.866 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.854+0000 7ff210f8f000 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x559c58c69a10 0x559c58c69de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:55.866 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.854+0000 7ff210f8f000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x559c58b33bd0 con 0x559c58cb6170 2023-05-22T15:26:55.866 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.854+0000 7ff210f8f000 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x559c58a50dd0 con 0x559c58a42fd0 2023-05-22T15:26:55.866 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.854+0000 7ff210f8f000 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x559c58ac1640 con 0x559c58c69a10 2023-05-22T15:26:55.866 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.854+0000 7ff206ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559c58cb6170 0x559c58cb6540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:55.866 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.854+0000 7ff206ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559c58cb6170 0x559c58cb6540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49534/0 (socket says 172.21.15.16:49534) 2023-05-22T15:26:55.867 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.854+0000 7ff206ffd640 1 -- 172.21.15.16:0/515576570 learned_addr learned my addr 172.21.15.16:0/515576570 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:26:55.867 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.854+0000 7ff207fff640 1 --2- 172.21.15.16:0/515576570 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x559c58c69a10 0x559c58c69de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:55.867 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.854+0000 7ff2077fe640 1 --2- 172.21.15.16:0/515576570 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559c58a42fd0 0x559c58cb5c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:55.867 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.854+0000 7ff206ffd640 1 -- 172.21.15.16:0/515576570 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x559c58c69a10 msgr2=0x559c58c69de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:26:55.867 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.854+0000 7ff206ffd640 1 --2- 172.21.15.16:0/515576570 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x559c58c69a10 0x559c58c69de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:55.867 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.854+0000 7ff206ffd640 1 -- 172.21.15.16:0/515576570 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559c58a42fd0 msgr2=0x559c58cb5c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:26:55.867 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.854+0000 7ff206ffd640 1 --2- 172.21.15.16:0/515576570 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559c58a42fd0 0x559c58cb5c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:55.867 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.854+0000 7ff206ffd640 1 -- 172.21.15.16:0/515576570 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559c58a800b0 con 0x559c58cb6170 2023-05-22T15:26:55.868 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.854+0000 7ff207fff640 1 --2- 172.21.15.16:0/515576570 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x559c58c69a10 0x559c58c69de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:26:55.868 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.854+0000 7ff206ffd640 1 --2- 172.21.15.16:0/515576570 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559c58cb6170 0x559c58cb6540 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7ff1f80af040 tx=0x7ff1f80a7000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8588b3178d2b1fcf server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:26:55.868 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.854+0000 7ff1fd7fa640 1 -- 172.21.15.16:0/515576570 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1f80acb10 con 0x559c58cb6170 2023-05-22T15:26:55.868 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.854+0000 7ff1fd7fa640 1 -- 172.21.15.16:0/515576570 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1f80acc70 con 0x559c58cb6170 2023-05-22T15:26:55.868 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.854+0000 7ff1fd7fa640 1 -- 172.21.15.16:0/515576570 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1f80b7420 con 0x559c58cb6170 2023-05-22T15:26:55.868 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.854+0000 7ff210f8f000 1 -- 172.21.15.16:0/515576570 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559c58cb6170 msgr2=0x559c58cb6540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:55.868 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.854+0000 7ff210f8f000 1 --2- 172.21.15.16:0/515576570 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559c58cb6170 0x559c58cb6540 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7ff1f80af040 tx=0x7ff1f80a7000 comp rx=0 tx=0).stop 2023-05-22T15:26:55.868 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.854+0000 7ff210f8f000 1 -- 172.21.15.16:0/515576570 shutdown_connections 2023-05-22T15:26:55.869 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.854+0000 7ff210f8f000 1 --2- 172.21.15.16:0/515576570 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x559c58c69a10 0x559c58c69de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:55.869 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.854+0000 7ff210f8f000 1 --2- 172.21.15.16:0/515576570 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559c58cb6170 0x559c58cb6540 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:55.869 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.854+0000 7ff210f8f000 1 --2- 172.21.15.16:0/515576570 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559c58a42fd0 0x559c58cb5c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:55.869 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.854+0000 7ff210f8f000 1 -- 172.21.15.16:0/515576570 >> 172.21.15.16:0/515576570 conn(0x559c58b12270 msgr2=0x559c58c99220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:26:55.869 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.854+0000 7ff210f8f000 1 -- 172.21.15.16:0/515576570 shutdown_connections 2023-05-22T15:26:55.869 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.854+0000 7ff210f8f000 1 -- 172.21.15.16:0/515576570 wait complete. 2023-05-22T15:26:55.872 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.854+0000 7ff210f8f000 1 -- start start 2023-05-22T15:26:55.872 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.854+0000 7ff210f8f000 1 Processor -- start 2023-05-22T15:26:55.872 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.854+0000 7ff210f8f000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559c58a42fd0 0x559c58cbcba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:55.872 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.854+0000 7ff210f8f000 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559c58c69a10 0x559c58cbd0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:55.872 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.854+0000 7ff210f8f000 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x559c58cb6170 0x559c58c9a450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:55.873 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.854+0000 7ff210f8f000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x559c58aba780 con 0x559c58a42fd0 2023-05-22T15:26:55.873 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.854+0000 7ff210f8f000 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x559c58a78250 con 0x559c58c69a10 2023-05-22T15:26:55.873 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.854+0000 7ff210f8f000 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x559c58a77ef0 con 0x559c58cb6170 2023-05-22T15:26:55.873 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.854+0000 7ff210f8f000 10 client.0 _ll_register_callbacks cb 0x559c58c57b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-05-22T15:26:55.873 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.854+0000 7ff2077fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559c58a42fd0 0x559c58cbcba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:55.873 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.854+0000 7ff2077fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559c58a42fd0 0x559c58cbcba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49546/0 (socket says 172.21.15.16:49546) 2023-05-22T15:26:55.873 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.854+0000 7ff2077fe640 1 -- 172.21.15.16:0/3152915982 learned_addr learned my addr 172.21.15.16:0/3152915982 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:26:55.873 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.858+0000 7ff207fff640 1 --2- 172.21.15.16:0/3152915982 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x559c58cb6170 0x559c58c9a450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:55.874 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.858+0000 7ff206ffd640 1 --2- 172.21.15.16:0/3152915982 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559c58c69a10 0x559c58cbd0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:55.874 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.858+0000 7ff2077fe640 1 -- 172.21.15.16:0/3152915982 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x559c58cb6170 msgr2=0x559c58c9a450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:26:55.874 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.858+0000 7ff2077fe640 1 --2- 172.21.15.16:0/3152915982 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x559c58cb6170 0x559c58c9a450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:55.874 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.858+0000 7ff2077fe640 1 -- 172.21.15.16:0/3152915982 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559c58c69a10 msgr2=0x559c58cbd0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:26:55.874 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.858+0000 7ff2077fe640 1 --2- 172.21.15.16:0/3152915982 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559c58c69a10 0x559c58cbd0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:55.874 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.858+0000 7ff2077fe640 1 -- 172.21.15.16:0/3152915982 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff20806a0a0 con 0x559c58a42fd0 2023-05-22T15:26:55.874 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.858+0000 7ff2077fe640 1 --2- 172.21.15.16:0/3152915982 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559c58a42fd0 0x559c58cbcba0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7ff1f0005ad0 tx=0x7ff1f0000ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b5b190fe62d180 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:26:55.875 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.858+0000 7ff1e67fc640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2023-05-22T15:26:55.875 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.858+0000 7ff1e67fc640 1 -- 172.21.15.16:0/3152915982 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1f0021630 con 0x559c58a42fd0 2023-05-22T15:26:55.875 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.858+0000 7ff1e67fc640 1 -- 172.21.15.16:0/3152915982 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(27..27 src has 1..27) v4 ==== 6813+0+0 (secure 0 0 0) 0x7ff1f0021790 con 0x559c58a42fd0 2023-05-22T15:26:55.875 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.858+0000 7ff210f8f000 1 -- 172.21.15.16:0/3152915982 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x559c58a7ec60 con 0x559c58a42fd0 2023-05-22T15:26:55.875 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.858+0000 7ff1e67fc640 1 -- 172.21.15.16:0/3152915982 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=6) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff1f0021950 con 0x559c58a42fd0 2023-05-22T15:26:55.875 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.858+0000 7ff210f8f000 10 client.4416 fetch_fsmap learned FSMap version 6 2023-05-22T15:26:55.875 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.858+0000 7ff210f8f000 1 -- 172.21.15.16:0/3152915982 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=6,monmap=2+}) v3 -- 0x559c587b7890 con 0x559c58a42fd0 2023-05-22T15:26:55.875 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.858+0000 7ff1e67fc640 1 -- 172.21.15.16:0/3152915982 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x559c58a7ec60 con 0x559c58a42fd0 2023-05-22T15:26:55.876 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.858+0000 7ff1e67fc640 1 -- 172.21.15.16:0/3152915982 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 6) v1 ==== 3114+0+0 (secure 0 0 0) 0x7ff1f0021ab0 con 0x559c58a42fd0 2023-05-22T15:26:55.876 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.858+0000 7ff210f8f000 10 client.4416 fetch_fsmap finished waiting for FSMap version 6 2023-05-22T15:26:55.876 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.858+0000 7ff210f8f000 10 client.4416 resolve_mds: resolved name 'c' to daemon mds.c 2023-05-22T15:26:55.876 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.858+0000 7ff210f8f000 20 client.4416 populate_metadata read hostname 'smithi016' 2023-05-22T15:26:55.876 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.858+0000 7ff210f8f000 1 --2- 172.21.15.16:0/3152915982 >> [v2:172.21.15.16:6835/283322289,v1:172.21.15.16:6837/283322289] conn(0x559c58c9b190 0x559c58c9b560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:55.876 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.858+0000 7ff210f8f000 4 client.4416 mds_command: new command op to 4287 tid=0[{"prefix": "get_command_descriptions"}] 2023-05-22T15:26:55.876 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.858+0000 7ff210f8f000 1 -- 172.21.15.16:0/3152915982 --> [v2:172.21.15.16:6835/283322289,v1:172.21.15.16:6837/283322289] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x559c58ab90d0 con 0x559c58c9b190 2023-05-22T15:26:55.876 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.858+0000 7ff206ffd640 1 --2- 172.21.15.16:0/3152915982 >> [v2:172.21.15.16:6835/283322289,v1:172.21.15.16:6837/283322289] conn(0x559c58c9b190 0x559c58c9b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:55.877 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.858+0000 7ff1e67fc640 10 client.4416 ms_handle_connect on v2:172.21.15.16:6835/283322289 2023-05-22T15:26:55.877 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.858+0000 7ff206ffd640 1 --2- 172.21.15.16:0/3152915982 >> [v2:172.21.15.16:6835/283322289,v1:172.21.15.16:6837/283322289] conn(0x559c58c9b190 0x559c58c9b560 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d66919f342ac874f server_cookie=dddc7b04f4b29282 in_seq=0 out_seq=0 2023-05-22T15:26:55.877 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.858+0000 7ff1e67fc640 1 -- 172.21.15.16:0/3152915982 <== mds.0 v2:172.21.15.16:6835/283322289 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x559c58ab90d0 con 0x559c58c9b190 2023-05-22T15:26:55.877 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.858+0000 7ff1e67fc640 10 client.4416 handle_command_reply: tid=0 2023-05-22T15:26:55.877 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.858+0000 7ff210f8f000 1 client.4416 shutdown 2023-05-22T15:26:55.877 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.858+0000 7ff210f8f000 1 -- 172.21.15.16:0/3152915982 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559c58a42fd0 msgr2=0x559c58cbcba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:55.877 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.858+0000 7ff210f8f000 1 --2- 172.21.15.16:0/3152915982 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559c58a42fd0 0x559c58cbcba0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7ff1f0005ad0 tx=0x7ff1f0000ee0 comp rx=0 tx=0).stop 2023-05-22T15:26:55.878 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.858+0000 7ff210f8f000 1 -- 172.21.15.16:0/3152915982 shutdown_connections 2023-05-22T15:26:55.878 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.858+0000 7ff210f8f000 1 --2- 172.21.15.16:0/3152915982 >> [v2:172.21.15.16:6835/283322289,v1:172.21.15.16:6837/283322289] conn(0x559c58c9b190 0x559c58c9b560 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:55.878 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.858+0000 7ff210f8f000 1 --2- 172.21.15.16:0/3152915982 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x559c58cb6170 0x559c58c9a450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:55.878 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.858+0000 7ff210f8f000 1 --2- 172.21.15.16:0/3152915982 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559c58c69a10 0x559c58cbd0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:55.878 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.858+0000 7ff210f8f000 1 --2- 172.21.15.16:0/3152915982 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559c58a42fd0 0x559c58cbcba0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:55.878 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.858+0000 7ff1e67fc640 0 client.4416 ms_handle_reset on v2:172.21.15.16:6835/283322289 2023-05-22T15:26:55.878 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.858+0000 7ff210f8f000 1 -- 172.21.15.16:0/3152915982 >> 172.21.15.16:0/3152915982 conn(0x559c58b12270 msgr2=0x559c58c69100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:26:55.878 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.858+0000 7ff210f8f000 1 -- 172.21.15.16:0/3152915982 shutdown_connections 2023-05-22T15:26:55.879 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.858+0000 7ff210f8f000 1 -- 172.21.15.16:0/3152915982 wait complete. 2023-05-22T15:26:55.879 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.858+0000 7ff210f8f000 20 client.4416 trim_cache size 0 max 16384 2023-05-22T15:26:55.883 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.874+0000 7ff210f8f000 1 Processor -- start 2023-05-22T15:26:55.883 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.874+0000 7ff210f8f000 1 -- start start 2023-05-22T15:26:55.884 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.874+0000 7ff210f8f000 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x559c58a42fd0 0x559c58c71c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:55.885 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.874+0000 7ff210f8f000 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559c58c69a10 0x559c58c72140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:55.885 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.874+0000 7ff210f8f000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559c58cb6170 0x559c58c6bdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:55.885 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.874+0000 7ff210f8f000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x559c58778940 con 0x559c58cb6170 2023-05-22T15:26:55.885 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.874+0000 7ff210f8f000 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x559c58cb5860 con 0x559c58c69a10 2023-05-22T15:26:55.885 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.874+0000 7ff210f8f000 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x559c58b44250 con 0x559c58a42fd0 2023-05-22T15:26:55.885 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.874+0000 7ff207fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559c58cb6170 0x559c58c6bdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:55.885 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.874+0000 7ff207fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559c58cb6170 0x559c58c6bdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49548/0 (socket says 172.21.15.16:49548) 2023-05-22T15:26:55.886 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.874+0000 7ff207fff640 1 -- 172.21.15.16:0/881667232 learned_addr learned my addr 172.21.15.16:0/881667232 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:26:55.886 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.874+0000 7ff207fff640 1 -- 172.21.15.16:0/881667232 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x559c58a42fd0 msgr2=0x559c58c71c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:26:55.886 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.874+0000 7ff207fff640 1 --2- 172.21.15.16:0/881667232 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x559c58a42fd0 0x559c58c71c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:55.886 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.874+0000 7ff207fff640 1 -- 172.21.15.16:0/881667232 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559c58c69a10 msgr2=0x559c58c72140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:26:55.886 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.874+0000 7ff207fff640 1 --2- 172.21.15.16:0/881667232 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559c58c69a10 0x559c58c72140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:55.886 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.874+0000 7ff207fff640 1 -- 172.21.15.16:0/881667232 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559c58a80360 con 0x559c58cb6170 2023-05-22T15:26:55.886 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.874+0000 7ff207fff640 1 --2- 172.21.15.16:0/881667232 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559c58cb6170 0x559c58c6bdf0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7ff20005bb60 tx=0x7ff20005c150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5876cc125cc5d630 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:26:55.886 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.874+0000 7ff1fdffb640 1 -- 172.21.15.16:0/881667232 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2000521a0 con 0x559c58cb6170 2023-05-22T15:26:55.887 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.874+0000 7ff1fdffb640 1 -- 172.21.15.16:0/881667232 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff20005bc00 con 0x559c58cb6170 2023-05-22T15:26:55.887 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.874+0000 7ff1fdffb640 1 -- 172.21.15.16:0/881667232 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff20006b210 con 0x559c58cb6170 2023-05-22T15:26:55.890 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.874+0000 7ff210f8f000 1 -- 172.21.15.16:0/881667232 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559c58cb6170 msgr2=0x559c58c6bdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:55.890 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.874+0000 7ff210f8f000 1 --2- 172.21.15.16:0/881667232 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559c58cb6170 0x559c58c6bdf0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7ff20005bb60 tx=0x7ff20005c150 comp rx=0 tx=0).stop 2023-05-22T15:26:55.890 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.874+0000 7ff210f8f000 1 -- 172.21.15.16:0/881667232 shutdown_connections 2023-05-22T15:26:55.890 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.874+0000 7ff210f8f000 1 --2- 172.21.15.16:0/881667232 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559c58cb6170 0x559c58c6bdf0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:55.891 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.874+0000 7ff210f8f000 1 --2- 172.21.15.16:0/881667232 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559c58c69a10 0x559c58c72140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:55.891 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.874+0000 7ff210f8f000 1 --2- 172.21.15.16:0/881667232 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x559c58a42fd0 0x559c58c71c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:55.891 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.874+0000 7ff210f8f000 1 -- 172.21.15.16:0/881667232 >> 172.21.15.16:0/881667232 conn(0x559c58b12270 msgr2=0x559c58c69080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:26:55.891 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.874+0000 7ff210f8f000 1 -- 172.21.15.16:0/881667232 shutdown_connections 2023-05-22T15:26:55.891 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.874+0000 7ff210f8f000 1 -- 172.21.15.16:0/881667232 wait complete. 2023-05-22T15:26:55.891 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.874+0000 7ff210f8f000 1 -- start start 2023-05-22T15:26:55.891 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.874+0000 7ff210f8f000 1 Processor -- start 2023-05-22T15:26:55.892 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.878+0000 7ff210f8f000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559c58a42fd0 0x559c58d12320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:55.892 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.878+0000 7ff210f8f000 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x559c58c69a10 0x559c58d12860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:55.892 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.878+0000 7ff210f8f000 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559c58cb6170 0x559c58c95f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:55.892 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.878+0000 7ff210f8f000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x559c58a59720 con 0x559c58a42fd0 2023-05-22T15:26:55.892 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.878+0000 7ff210f8f000 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x559c58ab9610 con 0x559c58cb6170 2023-05-22T15:26:55.892 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.878+0000 7ff210f8f000 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x559c58c63e80 con 0x559c58c69a10 2023-05-22T15:26:55.892 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.878+0000 7ff210f8f000 10 client.0 _ll_register_callbacks cb 0x559c58b0aeb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-05-22T15:26:55.892 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.878+0000 7ff2077fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559c58a42fd0 0x559c58d12320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:55.893 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.878+0000 7ff2077fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559c58a42fd0 0x559c58d12320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49554/0 (socket says 172.21.15.16:49554) 2023-05-22T15:26:55.893 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.878+0000 7ff2077fe640 1 -- 172.21.15.16:0/2370210371 learned_addr learned my addr 172.21.15.16:0/2370210371 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:26:55.893 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.878+0000 7ff2077fe640 1 -- 172.21.15.16:0/2370210371 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x559c58c69a10 msgr2=0x559c58d12860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:26:55.893 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.878+0000 7ff2077fe640 1 --2- 172.21.15.16:0/2370210371 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x559c58c69a10 0x559c58d12860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:55.893 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.878+0000 7ff2077fe640 1 -- 172.21.15.16:0/2370210371 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559c58cb6170 msgr2=0x559c58c95f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:26:55.893 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.878+0000 7ff2077fe640 1 --2- 172.21.15.16:0/2370210371 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559c58cb6170 0x559c58c95f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:55.893 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.878+0000 7ff2077fe640 1 -- 172.21.15.16:0/2370210371 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2080c5e70 con 0x559c58a42fd0 2023-05-22T15:26:55.894 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.878+0000 7ff2077fe640 1 --2- 172.21.15.16:0/2370210371 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559c58a42fd0 0x559c58d12320 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7ff1f000d520 tx=0x7ff1f0001130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59574e81806ca71 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:26:55.894 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.878+0000 7ff1e67fc640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2023-05-22T15:26:55.894 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.878+0000 7ff1e67fc640 1 -- 172.21.15.16:0/2370210371 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1f000dba0 con 0x559c58a42fd0 2023-05-22T15:26:55.894 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.878+0000 7ff1e67fc640 1 -- 172.21.15.16:0/2370210371 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(27..27 src has 1..27) v4 ==== 6813+0+0 (secure 0 0 0) 0x7ff1f002b640 con 0x559c58a42fd0 2023-05-22T15:26:55.894 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.878+0000 7ff210f8f000 1 -- 172.21.15.16:0/2370210371 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x559c58c701c0 con 0x559c58a42fd0 2023-05-22T15:26:55.894 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.878+0000 7ff1e67fc640 1 -- 172.21.15.16:0/2370210371 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=6) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff1f0007c60 con 0x559c58a42fd0 2023-05-22T15:26:55.894 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.878+0000 7ff210f8f000 10 client.4422 fetch_fsmap learned FSMap version 6 2023-05-22T15:26:55.894 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.878+0000 7ff210f8f000 1 -- 172.21.15.16:0/2370210371 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=6,monmap=2+}) v3 -- 0x559c58a65be0 con 0x559c58a42fd0 2023-05-22T15:26:55.895 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.878+0000 7ff1e67fc640 1 -- 172.21.15.16:0/2370210371 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x559c58c701c0 con 0x559c58a42fd0 2023-05-22T15:26:55.895 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.878+0000 7ff1e67fc640 1 -- 172.21.15.16:0/2370210371 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 6) v1 ==== 3114+0+0 (secure 0 0 0) 0x7ff1f002b840 con 0x559c58a42fd0 2023-05-22T15:26:55.895 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.878+0000 7ff210f8f000 10 client.4422 fetch_fsmap finished waiting for FSMap version 6 2023-05-22T15:26:55.895 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.878+0000 7ff210f8f000 10 client.4422 resolve_mds: resolved name 'c' to daemon mds.c 2023-05-22T15:26:55.895 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.878+0000 7ff210f8f000 20 client.4422 populate_metadata read hostname 'smithi016' 2023-05-22T15:26:55.895 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.878+0000 7ff210f8f000 1 --2- 172.21.15.16:0/2370210371 >> [v2:172.21.15.16:6835/283322289,v1:172.21.15.16:6837/283322289] conn(0x559c58c96480 0x559c58c96850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:55.895 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.878+0000 7ff210f8f000 4 client.4422 mds_command: new command op to 4287 tid=0[{"prefix": "status"}] 2023-05-22T15:26:55.895 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.878+0000 7ff210f8f000 1 -- 172.21.15.16:0/2370210371 --> [v2:172.21.15.16:6835/283322289,v1:172.21.15.16:6837/283322289] -- command(tid 0: {"prefix": "status"}) v1 -- 0x559c58a614d0 con 0x559c58c96480 2023-05-22T15:26:55.896 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.878+0000 7ff206ffd640 1 --2- 172.21.15.16:0/2370210371 >> [v2:172.21.15.16:6835/283322289,v1:172.21.15.16:6837/283322289] conn(0x559c58c96480 0x559c58c96850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:55.896 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.878+0000 7ff1e67fc640 10 client.4422 ms_handle_connect on v2:172.21.15.16:6835/283322289 2023-05-22T15:26:55.896 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.878+0000 7ff206ffd640 1 --2- 172.21.15.16:0/2370210371 >> [v2:172.21.15.16:6835/283322289,v1:172.21.15.16:6837/283322289] conn(0x559c58c96480 0x559c58c96850 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3f2ff8ea503eb766 server_cookie=ce58bc8c8e25ca43 in_seq=0 out_seq=0 2023-05-22T15:26:55.896 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.878+0000 7ff1e67fc640 1 -- 172.21.15.16:0/2370210371 <== mds.0 v2:172.21.15.16:6835/283322289 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+324 (crc 0 0 0) 0x559c58a614d0 con 0x559c58c96480 2023-05-22T15:26:55.896 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.878+0000 7ff1e67fc640 10 client.4422 handle_command_reply: tid=0 2023-05-22T15:26:55.896 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.878+0000 7ff210f8f000 1 client.4422 shutdown 2023-05-22T15:26:55.896 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.878+0000 7ff210f8f000 1 -- 172.21.15.16:0/2370210371 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559c58a42fd0 msgr2=0x559c58d12320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:55.897 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.878+0000 7ff210f8f000 1 --2- 172.21.15.16:0/2370210371 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559c58a42fd0 0x559c58d12320 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7ff1f000d520 tx=0x7ff1f0001130 comp rx=0 tx=0).stop 2023-05-22T15:26:55.897 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.878+0000 7ff210f8f000 1 -- 172.21.15.16:0/2370210371 shutdown_connections 2023-05-22T15:26:55.897 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.878+0000 7ff210f8f000 1 --2- 172.21.15.16:0/2370210371 >> [v2:172.21.15.16:6835/283322289,v1:172.21.15.16:6837/283322289] conn(0x559c58c96480 0x559c58c96850 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:55.897 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.878+0000 7ff210f8f000 1 --2- 172.21.15.16:0/2370210371 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559c58cb6170 0x559c58c95f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:55.897 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.878+0000 7ff210f8f000 1 --2- 172.21.15.16:0/2370210371 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x559c58c69a10 0x559c58d12860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:55.897 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.878+0000 7ff1e67fc640 0 client.4422 ms_handle_reset on v2:172.21.15.16:6835/283322289 2023-05-22T15:26:55.897 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.878+0000 7ff210f8f000 1 --2- 172.21.15.16:0/2370210371 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559c58a42fd0 0x559c58d12320 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:55.897 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.878+0000 7ff210f8f000 1 -- 172.21.15.16:0/2370210371 >> 172.21.15.16:0/2370210371 conn(0x559c58b12270 msgr2=0x559c58c71480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:26:55.898 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.878+0000 7ff210f8f000 1 -- 172.21.15.16:0/2370210371 shutdown_connections 2023-05-22T15:26:55.898 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.878+0000 7ff210f8f000 1 -- 172.21.15.16:0/2370210371 wait complete. 2023-05-22T15:26:55.898 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.878+0000 7ff210f8f000 20 client.4422 trim_cache size 0 max 16384 2023-05-22T15:26:55.898 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-05-22T15:26:55.898 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_fsid": "b2719834-7847-4d98-90a6-f65d57f85c49", 2023-05-22T15:26:55.899 INFO:teuthology.orchestra.run.smithi016.stdout: "whoami": 0, 2023-05-22T15:26:55.899 INFO:teuthology.orchestra.run.smithi016.stdout: "id": 4287, 2023-05-22T15:26:55.899 INFO:teuthology.orchestra.run.smithi016.stdout: "want_state": "up:active", 2023-05-22T15:26:55.899 INFO:teuthology.orchestra.run.smithi016.stdout: "state": "up:active", 2023-05-22T15:26:55.899 INFO:teuthology.orchestra.run.smithi016.stdout: "fs_name": "cephfs", 2023-05-22T15:26:55.899 INFO:teuthology.orchestra.run.smithi016.stdout: "rank_uptime": 6.3341643230000004, 2023-05-22T15:26:55.899 INFO:teuthology.orchestra.run.smithi016.stdout: "mdsmap_epoch": 6, 2023-05-22T15:26:55.899 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch": 27, 2023-05-22T15:26:55.900 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch_barrier": 26, 2023-05-22T15:26:55.900 INFO:teuthology.orchestra.run.smithi016.stdout: "uptime": 10.688196376 2023-05-22T15:26:55.900 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-05-22T15:26:55.901 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.878+0000 7ff20de6d640 1 -- 172.21.15.16:0/2873015283 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ff1e8065380 msgr2=0x7ff1e8067830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:55.901 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.878+0000 7ff20de6d640 1 --2- 172.21.15.16:0/2873015283 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ff1e8065380 0x7ff1e8067830 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7ff1f0000f70 tx=0x7ff1f0009040 comp rx=0 tx=0).stop 2023-05-22T15:26:55.901 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.878+0000 7ff20de6d640 1 -- 172.21.15.16:0/2873015283 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff2081605d0 msgr2=0x7ff2080b3af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:55.901 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.878+0000 7ff20de6d640 1 --2- 172.21.15.16:0/2873015283 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff2081605d0 0x7ff2080b3af0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7ff20000b970 tx=0x7ff20000be30 comp rx=0 tx=0).stop 2023-05-22T15:26:55.901 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.878+0000 7ff20de6d640 1 -- 172.21.15.16:0/2873015283 shutdown_connections 2023-05-22T15:26:55.901 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.878+0000 7ff20de6d640 1 --2- 172.21.15.16:0/2873015283 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff2081605d0 0x7ff2080b3af0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:55.902 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.878+0000 7ff20de6d640 1 --2- 172.21.15.16:0/2873015283 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff2080ce480 0x7ff2080ad2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:55.902 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.878+0000 7ff20de6d640 1 --2- 172.21.15.16:0/2873015283 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ff1e8065380 0x7ff1e8067830 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:55.902 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.878+0000 7ff20de6d640 1 --2- 172.21.15.16:0/2873015283 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff208071640 0x7ff2081535a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:55.902 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.878+0000 7ff20de6d640 1 -- 172.21.15.16:0/2873015283 >> 172.21.15.16:0/2873015283 conn(0x7ff2080cc870 msgr2=0x7ff2080cefb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:26:55.902 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.878+0000 7ff20de6d640 1 -- 172.21.15.16:0/2873015283 shutdown_connections 2023-05-22T15:26:55.902 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:55.878+0000 7ff20de6d640 1 -- 172.21.15.16:0/2873015283 wait complete. 2023-05-22T15:26:55.905 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2023-05-22T15:26:55.906 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2023-05-22T15:26:55.907 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-05-22T15:26:56.019 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:56.010+0000 7ff800c72640 1 Processor -- start 2023-05-22T15:26:56.019 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:56.010+0000 7ff800c72640 1 -- start start 2023-05-22T15:26:56.020 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:56.010+0000 7ff800c72640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff7fc0d1420 0x7ff7fc166880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:56.020 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:56.010+0000 7ff800c72640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7fc0ccef0 con 0x7ff7fc0b7dd0 2023-05-22T15:26:56.021 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:56.010+0000 7ff800c72640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7fc160520 con 0x7ff7fc0ccab0 2023-05-22T15:26:56.021 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:56.010+0000 7ff800c72640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7fc0d3360 con 0x7ff7fc0d1420 2023-05-22T15:26:56.021 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:56.010+0000 7ff7fad9d640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff7fc0d1420 0x7ff7fc166880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:56.021 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:56.010+0000 7ff7fad9d640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff7fc0d1420 0x7ff7fc166880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49568/0 (socket says 172.21.15.16:49568) 2023-05-22T15:26:56.021 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:56.010+0000 7ff7fad9d640 1 -- 172.21.15.16:0/1467497949 learned_addr learned my addr 172.21.15.16:0/1467497949 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:26:56.022 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:56.010+0000 7ff7fad9d640 1 -- 172.21.15.16:0/1467497949 >> v1:172.21.15.62:6789/0 conn(0x7ff7fc0ccab0 legacy=0x7ff7fc15fe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:26:56.022 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:56.010+0000 7ff7fad9d640 1 -- 172.21.15.16:0/1467497949 >> v1:172.21.15.16:6789/0 conn(0x7ff7fc0b7dd0 legacy=0x7ff7fc15f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:56.022 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:56.010+0000 7ff7fad9d640 1 -- 172.21.15.16:0/1467497949 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7fc0cd170 con 0x7ff7fc0d1420 2023-05-22T15:26:56.023 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:56.014+0000 7ff7fad9d640 1 --2- 172.21.15.16:0/1467497949 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff7fc0d1420 0x7ff7fc166880 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7ff7f4009e50 tx=0x7ff7f401c9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f7466e4084dc201 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:26:56.023 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:56.014+0000 7ff7f959a640 1 -- 172.21.15.16:0/1467497949 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7f40a5860 con 0x7ff7fc0d1420 2023-05-22T15:26:56.023 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:56.014+0000 7ff7f959a640 1 -- 172.21.15.16:0/1467497949 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7f40a59c0 con 0x7ff7fc0d1420 2023-05-22T15:26:56.024 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:56.014+0000 7ff7f959a640 1 -- 172.21.15.16:0/1467497949 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7f40a5d50 con 0x7ff7fc0d1420 2023-05-22T15:26:56.024 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:56.014+0000 7ff800c72640 1 -- 172.21.15.16:0/1467497949 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff7fc0d1420 msgr2=0x7ff7fc166880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:56.025 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:56.014+0000 7ff800c72640 1 --2- 172.21.15.16:0/1467497949 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff7fc0d1420 0x7ff7fc166880 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7ff7f4009e50 tx=0x7ff7f401c9a0 comp rx=0 tx=0).stop 2023-05-22T15:26:56.025 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:56.014+0000 7ff800c72640 1 -- 172.21.15.16:0/1467497949 shutdown_connections 2023-05-22T15:26:56.025 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:56.014+0000 7ff800c72640 1 --2- 172.21.15.16:0/1467497949 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff7fc0d1420 0x7ff7fc166880 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:56.025 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:56.014+0000 7ff800c72640 1 -- 172.21.15.16:0/1467497949 >> 172.21.15.16:0/1467497949 conn(0x7ff7fc0655c0 msgr2=0x7ff7fc070e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:26:56.025 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:56.014+0000 7ff800c72640 1 -- 172.21.15.16:0/1467497949 shutdown_connections 2023-05-22T15:26:56.025 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:56.014+0000 7ff800c72640 1 -- 172.21.15.16:0/1467497949 wait complete. 2023-05-22T15:26:56.026 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:56.014+0000 7ff800c72640 1 Processor -- start 2023-05-22T15:26:56.026 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:56.014+0000 7ff800c72640 1 -- start start 2023-05-22T15:26:56.027 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:56.018+0000 7ff800c72640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff7fc0b7dd0 0x7ff7fc157f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:56.027 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:56.018+0000 7ff800c72640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff7fc0ccab0 0x7ff7fc153a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:56.027 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:56.018+0000 7ff800c72640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff7fc0d1420 0x7ff7fc159240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:56.027 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:56.018+0000 7ff800c72640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7fc156980 con 0x7ff7fc0ccab0 2023-05-22T15:26:56.027 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:56.018+0000 7ff800c72640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7fc156c20 con 0x7ff7fc0d1420 2023-05-22T15:26:56.027 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:56.018+0000 7ff800c72640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7fc156f00 con 0x7ff7fc0b7dd0 2023-05-22T15:26:56.028 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:56.018+0000 7ff7f9d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff7fc0ccab0 0x7ff7fc153a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:56.028 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:56.018+0000 7ff7f9d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff7fc0ccab0 0x7ff7fc153a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49570/0 (socket says 172.21.15.16:49570) 2023-05-22T15:26:56.028 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:56.018+0000 7ff7f9d9b640 1 -- 172.21.15.16:0/1758982633 learned_addr learned my addr 172.21.15.16:0/1758982633 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:26:56.029 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:56.018+0000 7ff7fa59c640 1 --2- 172.21.15.16:0/1758982633 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff7fc0b7dd0 0x7ff7fc157f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:56.030 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:56.018+0000 7ff7fad9d640 1 --2- 172.21.15.16:0/1758982633 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff7fc0d1420 0x7ff7fc159240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:56.030 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:56.018+0000 7ff7f9d9b640 1 -- 172.21.15.16:0/1758982633 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff7fc0b7dd0 msgr2=0x7ff7fc157f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:56.030 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:56.018+0000 7ff7f9d9b640 1 --2- 172.21.15.16:0/1758982633 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff7fc0b7dd0 0x7ff7fc157f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:56.030 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:56.018+0000 7ff7f9d9b640 1 -- 172.21.15.16:0/1758982633 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff7fc0d1420 msgr2=0x7ff7fc159240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:56.030 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:56.018+0000 7ff7f9d9b640 1 --2- 172.21.15.16:0/1758982633 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff7fc0d1420 0x7ff7fc159240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:56.030 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:56.018+0000 7ff7f9d9b640 1 -- 172.21.15.16:0/1758982633 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7f4009b30 con 0x7ff7fc0ccab0 2023-05-22T15:26:56.030 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:56.018+0000 7ff7fad9d640 1 --2- 172.21.15.16:0/1758982633 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff7fc0d1420 0x7ff7fc159240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:26:56.030 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:56.018+0000 7ff7f9d9b640 1 --2- 172.21.15.16:0/1758982633 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff7fc0ccab0 0x7ff7fc153a90 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7ff7ec00c450 tx=0x7ff7ec00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:26:56.031 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:56.018+0000 7ff7eaffd640 1 -- 172.21.15.16:0/1758982633 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7ec012830 con 0x7ff7fc0ccab0 2023-05-22T15:26:56.031 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:56.018+0000 7ff7eaffd640 1 -- 172.21.15.16:0/1758982633 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7ec0129b0 con 0x7ff7fc0ccab0 2023-05-22T15:26:56.031 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:56.018+0000 7ff800c72640 1 -- 172.21.15.16:0/1758982633 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7fc0cf730 con 0x7ff7fc0ccab0 2023-05-22T15:26:56.031 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:56.018+0000 7ff7eaffd640 1 -- 172.21.15.16:0/1758982633 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7ec012d60 con 0x7ff7fc0ccab0 2023-05-22T15:26:56.031 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:56.018+0000 7ff800c72640 1 -- 172.21.15.16:0/1758982633 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7fc0d3080 con 0x7ff7fc0ccab0 2023-05-22T15:26:56.031 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:56.018+0000 7ff800c72640 1 -- 172.21.15.16:0/1758982633 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7fc1695e0 con 0x7ff7fc0ccab0 2023-05-22T15:26:56.031 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:56.018+0000 7ff7eaffd640 1 -- 172.21.15.16:0/1758982633 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7ff7ec01c460 con 0x7ff7fc0ccab0 2023-05-22T15:26:56.032 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:56.018+0000 7ff7eaffd640 1 --2- 172.21.15.16:0/1758982633 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ff7dc065380 0x7ff7dc067830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:56.032 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:56.018+0000 7ff7eaffd640 1 -- 172.21.15.16:0/1758982633 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6813+0+0 (secure 0 0 0) 0x7ff7ec082990 con 0x7ff7fc0ccab0 2023-05-22T15:26:56.032 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:56.018+0000 7ff7fa59c640 1 --2- 172.21.15.16:0/1758982633 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ff7dc065380 0x7ff7dc067830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:56.032 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:56.022+0000 7ff7fa59c640 1 --2- 172.21.15.16:0/1758982633 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ff7dc065380 0x7ff7dc067830 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7ff7f0000f70 tx=0x7ff7f0009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:26:56.033 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:56.022+0000 7ff7eaffd640 1 -- 172.21.15.16:0/1758982633 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7ff7ec050350 con 0x7ff7fc0ccab0 2023-05-22T15:26:56.218 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:56.206+0000 7ff800c72640 1 -- 172.21.15.16:0/1758982633 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7ff7fc04dbe0 con 0x7ff7fc0ccab0 2023-05-22T15:26:56.219 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:56.210+0000 7ff7eaffd640 1 -- 172.21.15.16:0/1758982633 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v6) v1 ==== 53+0+88 (secure 0 0 0) 0x7ff7ec053970 con 0x7ff7fc0ccab0 2023-05-22T15:26:56.219 INFO:teuthology.orchestra.run.smithi016.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2023-05-22T15:26:56.221 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:56.210+0000 7ff800c72640 1 -- 172.21.15.16:0/1758982633 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ff7dc065380 msgr2=0x7ff7dc067830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:56.221 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:56.210+0000 7ff800c72640 1 --2- 172.21.15.16:0/1758982633 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ff7dc065380 0x7ff7dc067830 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7ff7f0000f70 tx=0x7ff7f0009040 comp rx=0 tx=0).stop 2023-05-22T15:26:56.221 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:56.210+0000 7ff800c72640 1 -- 172.21.15.16:0/1758982633 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff7fc0ccab0 msgr2=0x7ff7fc153a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:56.221 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:56.210+0000 7ff800c72640 1 --2- 172.21.15.16:0/1758982633 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff7fc0ccab0 0x7ff7fc153a90 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7ff7ec00c450 tx=0x7ff7ec00c910 comp rx=0 tx=0).stop 2023-05-22T15:26:56.221 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:56.210+0000 7ff800c72640 1 -- 172.21.15.16:0/1758982633 shutdown_connections 2023-05-22T15:26:56.221 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:56.210+0000 7ff800c72640 1 --2- 172.21.15.16:0/1758982633 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff7fc0d1420 0x7ff7fc159240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:56.221 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:56.210+0000 7ff800c72640 1 --2- 172.21.15.16:0/1758982633 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff7fc0ccab0 0x7ff7fc153a90 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:56.222 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:56.210+0000 7ff800c72640 1 --2- 172.21.15.16:0/1758982633 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ff7dc065380 0x7ff7dc067830 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:56.222 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:56.210+0000 7ff800c72640 1 --2- 172.21.15.16:0/1758982633 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff7fc0b7dd0 0x7ff7fc157f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:56.222 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:56.210+0000 7ff800c72640 1 -- 172.21.15.16:0/1758982633 >> 172.21.15.16:0/1758982633 conn(0x7ff7fc0655c0 msgr2=0x7ff7fc0b6ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:26:56.222 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:56.210+0000 7ff800c72640 1 -- 172.21.15.16:0/1758982633 shutdown_connections 2023-05-22T15:26:56.223 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:56.210+0000 7ff800c72640 1 -- 172.21.15.16:0/1758982633 wait complete. 2023-05-22T15:26:56.235 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2023-05-22T15:26:56.236 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-05-22T15:26:56.236 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi016.front.sepia.ceph.com 2023-05-22T15:26:56.236 INFO:tasks.cephfs.mount:self.client.name = client.admin 2023-05-22T15:26:56.236 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2023-05-22T15:26:56.237 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2023-05-22T15:26:56.237 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-05-22T15:26:56.237 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-05-22T15:26:56.237 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.admin' 2023-05-22T15:26:56.238 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:26:56.238 DEBUG:teuthology.orchestra.run.smithi016:> ip addr 2023-05-22T15:26:56.244 INFO:teuthology.orchestra.run.smithi016.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-05-22T15:26:56.244 INFO:teuthology.orchestra.run.smithi016.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-05-22T15:26:56.244 INFO:teuthology.orchestra.run.smithi016.stdout: inet 127.0.0.1/8 scope host lo 2023-05-22T15:26:56.244 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-05-22T15:26:56.245 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 ::1/128 scope host 2023-05-22T15:26:56.245 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-05-22T15:26:56.245 INFO:teuthology.orchestra.run.smithi016.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-05-22T15:26:56.245 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:6b:fd:e0 brd ff:ff:ff:ff:ff:ff 2023-05-22T15:26:56.245 INFO:teuthology.orchestra.run.smithi016.stdout: altname enp7s0f0 2023-05-22T15:26:56.245 INFO:teuthology.orchestra.run.smithi016.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-05-22T15:26:56.245 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:6b:fd:e1 brd ff:ff:ff:ff:ff:ff 2023-05-22T15:26:56.246 INFO:teuthology.orchestra.run.smithi016.stdout: altname enp7s0f1 2023-05-22T15:26:56.246 INFO:teuthology.orchestra.run.smithi016.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2023-05-22T15:26:56.246 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:bd:15:64 brd ff:ff:ff:ff:ff:ff 2023-05-22T15:26:56.246 INFO:teuthology.orchestra.run.smithi016.stdout: altname enp1s0f0 2023-05-22T15:26:56.246 INFO:teuthology.orchestra.run.smithi016.stdout: inet 172.21.15.16/20 brd 172.21.15.255 scope global dynamic ens1f0 2023-05-22T15:26:56.246 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft 42500sec preferred_lft 42500sec 2023-05-22T15:26:56.246 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::ec4:7aff:febd:1564/64 scope link 2023-05-22T15:26:56.246 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-05-22T15:26:56.247 INFO:teuthology.orchestra.run.smithi016.stdout:5: ens1f1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-05-22T15:26:56.247 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:bd:15:65 brd ff:ff:ff:ff:ff:ff 2023-05-22T15:26:56.247 INFO:teuthology.orchestra.run.smithi016.stdout: altname enp1s0f1 2023-05-22T15:26:56.247 INFO:teuthology.orchestra.run.smithi016.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2023-05-22T15:26:56.247 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 02:42:1f:b8:98:0a brd ff:ff:ff:ff:ff:ff 2023-05-22T15:26:56.247 INFO:teuthology.orchestra.run.smithi016.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2023-05-22T15:26:56.247 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-05-22T15:26:56.248 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2023-05-22T15:26:56.248 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:26:56.249 DEBUG:teuthology.orchestra.run.smithi016:> (cd / && exec bash -c ' 2023-05-22T15:26:56.249 DEBUG:teuthology.orchestra.run.smithi016:> set -e 2023-05-22T15:26:56.249 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip link add name ceph-brx type bridge 2023-05-22T15:26:56.250 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip addr flush dev ceph-brx 2023-05-22T15:26:56.250 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip link set ceph-brx up 2023-05-22T15:26:56.250 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2023-05-22T15:26:56.250 DEBUG:teuthology.orchestra.run.smithi016:> ') 2023-05-22T15:26:56.318 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:26:56.318 DEBUG:teuthology.orchestra.run.smithi016:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-05-22T15:26:56.370 INFO:teuthology.orchestra.run.smithi016.stdout:1 2023-05-22T15:26:56.371 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:26:56.371 DEBUG:teuthology.orchestra.run.smithi016:> route 2023-05-22T15:26:56.438 INFO:teuthology.orchestra.run.smithi016.stdout:Kernel IP routing table 2023-05-22T15:26:56.439 INFO:teuthology.orchestra.run.smithi016.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-05-22T15:26:56.439 INFO:teuthology.orchestra.run.smithi016.stdout:default _gateway 0.0.0.0 UG 0 0 0 ens1f0 2023-05-22T15:26:56.439 INFO:teuthology.orchestra.run.smithi016.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2023-05-22T15:26:56.439 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 0 0 0 ens1f0 2023-05-22T15:26:56.439 INFO:teuthology.orchestra.run.smithi016.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-05-22T15:26:56.440 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:26:56.440 DEBUG:teuthology.orchestra.run.smithi016:> (cd / && exec bash -c ' 2023-05-22T15:26:56.440 DEBUG:teuthology.orchestra.run.smithi016:> set -e 2023-05-22T15:26:56.441 DEBUG:teuthology.orchestra.run.smithi016:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2023-05-22T15:26:56.441 DEBUG:teuthology.orchestra.run.smithi016:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2023-05-22T15:26:56.441 DEBUG:teuthology.orchestra.run.smithi016:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2023-05-22T15:26:56.441 DEBUG:teuthology.orchestra.run.smithi016:> ') 2023-05-22T15:26:56.514 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:26:56.515 DEBUG:teuthology.orchestra.run.smithi016:> ip netns list 2023-05-22T15:26:56.563 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:26:56.563 DEBUG:teuthology.orchestra.run.smithi016:> ip netns list-id 2023-05-22T15:26:56.611 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:26:56.611 DEBUG:teuthology.orchestra.run.smithi016:> (cd / && exec bash -c ' 2023-05-22T15:26:56.612 DEBUG:teuthology.orchestra.run.smithi016:> set -e 2023-05-22T15:26:56.612 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.admin 2023-05-22T15:26:56.612 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.admin 0 2023-05-22T15:26:56.612 DEBUG:teuthology.orchestra.run.smithi016:> ') 2023-05-22T15:26:56.683 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.admin' with 192.168.0.1/16 2023-05-22T15:26:56.684 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:26:56.685 DEBUG:teuthology.orchestra.run.smithi016:> (cd / && exec bash -c ' 2023-05-22T15:26:56.685 DEBUG:teuthology.orchestra.run.smithi016:> set -e 2023-05-22T15:26:56.685 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.admin type veth peer name brx.0 2023-05-22T15:26:56.685 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2023-05-22T15:26:56.686 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set veth0 up 2023-05-22T15:26:56.686 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set lo up 2023-05-22T15:26:56.686 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip route add default via 192.168.255.254 2023-05-22T15:26:56.686 DEBUG:teuthology.orchestra.run.smithi016:> ') 2023-05-22T15:26:56.793 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:26:56.793 DEBUG:teuthology.orchestra.run.smithi016:> (cd / && exec bash -c ' 2023-05-22T15:26:56.794 DEBUG:teuthology.orchestra.run.smithi016:> set -e 2023-05-22T15:26:56.794 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip link set brx.0 up 2023-05-22T15:26:56.794 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip link set dev brx.0 master ceph-brx 2023-05-22T15:26:56.794 DEBUG:teuthology.orchestra.run.smithi016:> ') 2023-05-22T15:26:56.814 INFO:tasks.cephfs.fuse_mount:Client client.admin config is {} 2023-05-22T15:26:56.814 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-05-22T15:26:56.815 DEBUG:teuthology.orchestra.run.smithi016:> mkdir -p -v /home/ubuntu/cephtest/mnt.admin 2023-05-22T15:26:56.862 INFO:teuthology.orchestra.run.smithi016.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.admin' 2023-05-22T15:26:56.862 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-05-22T15:26:56.863 DEBUG:teuthology.orchestra.run.smithi016:> chmod 0000 /home/ubuntu/cephtest/mnt.admin 2023-05-22T15:26:56.909 DEBUG:teuthology.orchestra.run.smithi016:> sudo modprobe fuse 2023-05-22T15:26:56.962 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-05-22T15:26:56.962 DEBUG:teuthology.orchestra.run.smithi016:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-05-22T15:26:57.019 INFO:teuthology.orchestra.run.smithi016.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-05-22T15:26:57.020 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-05-22T15:26:57.021 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:26:57.021 DEBUG:teuthology.orchestra.run.smithi016:> ls /sys/fs/fuse/connections 2023-05-22T15:26:57.067 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2023-05-22T15:26:57.068 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs) 2023-05-22T15:26:57.112 DEBUG:teuthology.orchestra.run.smithi016:> sudo modprobe fuse 2023-05-22T15:26:57.127 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-05-22T15:26:57.128 DEBUG:teuthology.orchestra.run.smithi016:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-05-22T15:26:57.181 INFO:teuthology.orchestra.run.smithi016.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-05-22T15:26:57.182 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-05-22T15:26:57.183 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:26:57.183 DEBUG:teuthology.orchestra.run.smithi016:> ls /sys/fs/fuse/connections 2023-05-22T15:26:57.202 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi016.stderr:2023-05-22T15:26:57.190+0000 7f33d6cfc580 -1 init, newargv = 0x5644c594c660 newargc=15 2023-05-22T15:26:57.202 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi016.stderr:ceph-fuse[43384]: starting ceph client 2023-05-22T15:26:57.210 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi016.stderr:ceph-fuse[43384]: starting fuse 2023-05-22T15:26:58.307 DEBUG:teuthology.orchestra.run.smithi016:> sudo modprobe fuse 2023-05-22T15:26:58.324 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-05-22T15:26:58.325 DEBUG:teuthology.orchestra.run.smithi016:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-05-22T15:26:58.378 INFO:teuthology.orchestra.run.smithi016.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-05-22T15:26:58.379 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-05-22T15:26:58.380 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:26:58.380 DEBUG:teuthology.orchestra.run.smithi016:> ls /sys/fs/fuse/connections 2023-05-22T15:26:58.431 INFO:teuthology.orchestra.run.smithi016.stdout:45 2023-05-22T15:26:58.431 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [45] 2023-05-22T15:26:58.432 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2023-05-22T15:26:58.432 DEBUG:teuthology.orchestra.run.smithi016:> import glob 2023-05-22T15:26:58.432 DEBUG:teuthology.orchestra.run.smithi016:> import re 2023-05-22T15:26:58.433 DEBUG:teuthology.orchestra.run.smithi016:> import os 2023-05-22T15:26:58.433 DEBUG:teuthology.orchestra.run.smithi016:> import subprocess 2023-05-22T15:26:58.433 DEBUG:teuthology.orchestra.run.smithi016:> 2023-05-22T15:26:58.433 DEBUG:teuthology.orchestra.run.smithi016:> def _find_admin_socket(client_name): 2023-05-22T15:26:58.434 DEBUG:teuthology.orchestra.run.smithi016:> asok_path = "/var/run/ceph/ceph-client.admin.*.asok" 2023-05-22T15:26:58.434 DEBUG:teuthology.orchestra.run.smithi016:> files = glob.glob(asok_path) 2023-05-22T15:26:58.434 DEBUG:teuthology.orchestra.run.smithi016:> mountpoint = "/home/ubuntu/cephtest/mnt.admin" 2023-05-22T15:26:58.434 DEBUG:teuthology.orchestra.run.smithi016:> 2023-05-22T15:26:58.435 DEBUG:teuthology.orchestra.run.smithi016:> # Given a non-glob path, it better be there 2023-05-22T15:26:58.435 DEBUG:teuthology.orchestra.run.smithi016:> if "*" not in asok_path: 2023-05-22T15:26:58.435 DEBUG:teuthology.orchestra.run.smithi016:> assert(len(files) == 1) 2023-05-22T15:26:58.436 DEBUG:teuthology.orchestra.run.smithi016:> return files[0] 2023-05-22T15:26:58.436 DEBUG:teuthology.orchestra.run.smithi016:> 2023-05-22T15:26:58.436 DEBUG:teuthology.orchestra.run.smithi016:> for f in files: 2023-05-22T15:26:58.436 DEBUG:teuthology.orchestra.run.smithi016:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-05-22T15:26:58.437 DEBUG:teuthology.orchestra.run.smithi016:> if os.path.exists("/proc/{0}".format(pid)): 2023-05-22T15:26:58.437 DEBUG:teuthology.orchestra.run.smithi016:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-05-22T15:26:58.437 DEBUG:teuthology.orchestra.run.smithi016:> contents = proc_f.read() 2023-05-22T15:26:58.438 DEBUG:teuthology.orchestra.run.smithi016:> if mountpoint in contents: 2023-05-22T15:26:58.438 DEBUG:teuthology.orchestra.run.smithi016:> return f 2023-05-22T15:26:58.438 DEBUG:teuthology.orchestra.run.smithi016:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-05-22T15:26:58.438 DEBUG:teuthology.orchestra.run.smithi016:> 2023-05-22T15:26:58.439 DEBUG:teuthology.orchestra.run.smithi016:> print(_find_admin_socket("client.admin")) 2023-05-22T15:26:58.439 DEBUG:teuthology.orchestra.run.smithi016:> ' 2023-05-22T15:26:58.555 INFO:teuthology.orchestra.run.smithi016.stdout:/var/run/ceph/ceph-client.admin.43384.asok 2023-05-22T15:26:58.736 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.admin.43384.asok 2023-05-22T15:26:58.737 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:26:58.737 DEBUG:teuthology.orchestra.run.smithi016:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.admin.43384.asok status 2023-05-22T15:26:58.897 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-05-22T15:26:58.898 INFO:teuthology.orchestra.run.smithi016.stdout: "metadata": { 2023-05-22T15:26:58.898 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph_sha1": "9c9ba4df22024d9d8d6cf9f98a4b96751fc78d0c", 2023-05-22T15:26:58.898 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph_version": "ceph version 18.0.0-3192-g9c9ba4df (9c9ba4df22024d9d8d6cf9f98a4b96751fc78d0c) reef (dev)", 2023-05-22T15:26:58.898 INFO:teuthology.orchestra.run.smithi016.stdout: "entity_id": "admin", 2023-05-22T15:26:58.898 INFO:teuthology.orchestra.run.smithi016.stdout: "hostname": "smithi016", 2023-05-22T15:26:58.898 INFO:teuthology.orchestra.run.smithi016.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.admin", 2023-05-22T15:26:58.899 INFO:teuthology.orchestra.run.smithi016.stdout: "pid": "43384", 2023-05-22T15:26:58.899 INFO:teuthology.orchestra.run.smithi016.stdout: "root": "/" 2023-05-22T15:26:58.899 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:26:58.899 INFO:teuthology.orchestra.run.smithi016.stdout: "dentry_count": 0, 2023-05-22T15:26:58.899 INFO:teuthology.orchestra.run.smithi016.stdout: "dentry_pinned_count": 0, 2023-05-22T15:26:58.899 INFO:teuthology.orchestra.run.smithi016.stdout: "id": 4437, 2023-05-22T15:26:58.899 INFO:teuthology.orchestra.run.smithi016.stdout: "inst": { 2023-05-22T15:26:58.900 INFO:teuthology.orchestra.run.smithi016.stdout: "name": { 2023-05-22T15:26:58.900 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "client", 2023-05-22T15:26:58.900 INFO:teuthology.orchestra.run.smithi016.stdout: "num": 4437 2023-05-22T15:26:58.901 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:26:58.901 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": { 2023-05-22T15:26:58.901 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-05-22T15:26:58.901 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "192.168.0.1:0", 2023-05-22T15:26:58.901 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 3976912500 2023-05-22T15:26:58.901 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T15:26:58.901 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:26:58.902 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": { 2023-05-22T15:26:58.902 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-05-22T15:26:58.902 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "192.168.0.1:0", 2023-05-22T15:26:58.902 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 3976912500 2023-05-22T15:26:58.902 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:26:58.902 INFO:teuthology.orchestra.run.smithi016.stdout: "inst_str": "client.4437 192.168.0.1:0/3976912500", 2023-05-22T15:26:58.902 INFO:teuthology.orchestra.run.smithi016.stdout: "addr_str": "192.168.0.1:0/3976912500", 2023-05-22T15:26:58.902 INFO:teuthology.orchestra.run.smithi016.stdout: "inode_count": 1, 2023-05-22T15:26:58.903 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_epoch": 6, 2023-05-22T15:26:58.903 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_epoch": 27, 2023-05-22T15:26:58.903 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_epoch_barrier": 0, 2023-05-22T15:26:58.903 INFO:teuthology.orchestra.run.smithi016.stdout: "blocklisted": false, 2023-05-22T15:26:58.903 INFO:teuthology.orchestra.run.smithi016.stdout: "fs_name": "cephfs" 2023-05-22T15:26:58.903 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-05-22T15:26:58.911 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:26:58.911 DEBUG:teuthology.orchestra.run.smithi016:> stat --file-system '--printf=%T 2023-05-22T15:26:58.912 DEBUG:teuthology.orchestra.run.smithi016:> ' -- /home/ubuntu/cephtest/mnt.admin 2023-05-22T15:26:58.919 INFO:teuthology.orchestra.run.smithi016.stdout:fuseblk 2023-05-22T15:26:58.919 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.admin 2023-05-22T15:26:58.920 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:26:58.921 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.admin 2023-05-22T15:26:58.972 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:26:58.973 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.admin && exec bash -c 'setfattr -n ceph.dir.layout.pool -v cephfs_data_ec . && getfattr -n ceph.dir.layout .') 2023-05-22T15:26:59.049 INFO:teuthology.orchestra.run.smithi016.stdout:# file: . 2023-05-22T15:26:59.049 INFO:teuthology.orchestra.run.smithi016.stdout:ceph.dir.layout="stripe_unit=4194304 stripe_count=1 object_size=4194304 pool=cephfs_data_ec" 2023-05-22T15:26:59.049 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:26:59.050 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-05-22T15:26:59.050 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2023-05-22T15:26:59.099 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-05-22T15:26:59.274 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.266+0000 7efede70b640 1 Processor -- start 2023-05-22T15:26:59.274 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.266+0000 7efede70b640 1 -- start start 2023-05-22T15:26:59.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.266+0000 7efede70b640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7efed815fc50 0x7efed8160020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:59.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.266+0000 7efede70b640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7efed815c920 0x7efed80d0030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:59.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.266+0000 7efede70b640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efed815c490 con 0x7efed815cd70 2023-05-22T15:26:59.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.266+0000 7efede70b640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efed80cf7a0 con 0x7efed815fc50 2023-05-22T15:26:59.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.266+0000 7efede70b640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efed815c140 con 0x7efed815c920 2023-05-22T15:26:59.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.266+0000 7efed7fff640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7efed815fc50 0x7efed8160020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:59.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.266+0000 7efed7fff640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7efed815fc50 0x7efed8160020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57300/0 (socket says 172.21.15.16:57300) 2023-05-22T15:26:59.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.266+0000 7efed7fff640 1 -- 172.21.15.16:0/3180854379 learned_addr learned my addr 172.21.15.16:0/3180854379 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:26:59.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.266+0000 7efed77fe640 1 --2- 172.21.15.16:0/3180854379 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7efed815c920 0x7efed80d0030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:59.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.266+0000 7efed7fff640 1 -- 172.21.15.16:0/3180854379 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7efed815c920 msgr2=0x7efed80d0030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:26:59.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.266+0000 7efed7fff640 1 --2- 172.21.15.16:0/3180854379 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7efed815c920 0x7efed80d0030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:59.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.266+0000 7efed7fff640 1 -- 172.21.15.16:0/3180854379 >> v1:172.21.15.62:6789/0 conn(0x7efed815cd70 legacy=0x7efed80b3ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:26:59.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.266+0000 7efed7fff640 1 -- 172.21.15.16:0/3180854379 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efed80ccb50 con 0x7efed815fc50 2023-05-22T15:26:59.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.266+0000 7efed7fff640 1 --2- 172.21.15.16:0/3180854379 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efed815fc50 0x7efed8160020 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7efec8009870 tx=0x7efec801c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a00d84bcaed7855a server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:26:59.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.270+0000 7efed6ffd640 1 -- 172.21.15.16:0/3180854379 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efec80aa5f0 con 0x7efed815fc50 2023-05-22T15:26:59.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.270+0000 7efed6ffd640 1 -- 172.21.15.16:0/3180854379 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efec80aa750 con 0x7efed815fc50 2023-05-22T15:26:59.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.270+0000 7efed6ffd640 1 -- 172.21.15.16:0/3180854379 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efec80aaae0 con 0x7efed815fc50 2023-05-22T15:26:59.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.270+0000 7efede70b640 1 -- 172.21.15.16:0/3180854379 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efed815fc50 msgr2=0x7efed8160020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:59.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.270+0000 7efede70b640 1 --2- 172.21.15.16:0/3180854379 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efed815fc50 0x7efed8160020 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7efec8009870 tx=0x7efec801c4d0 comp rx=0 tx=0).stop 2023-05-22T15:26:59.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.270+0000 7efede70b640 1 -- 172.21.15.16:0/3180854379 shutdown_connections 2023-05-22T15:26:59.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.270+0000 7efede70b640 1 --2- 172.21.15.16:0/3180854379 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7efed815c920 0x7efed80d0030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:59.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.270+0000 7efede70b640 1 --2- 172.21.15.16:0/3180854379 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efed815fc50 0x7efed8160020 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:59.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.270+0000 7efede70b640 1 -- 172.21.15.16:0/3180854379 >> 172.21.15.16:0/3180854379 conn(0x7efed80657f0 msgr2=0x7efed8070e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:26:59.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.270+0000 7efede70b640 1 -- 172.21.15.16:0/3180854379 shutdown_connections 2023-05-22T15:26:59.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.270+0000 7efede70b640 1 -- 172.21.15.16:0/3180854379 wait complete. 2023-05-22T15:26:59.282 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.270+0000 7efede70b640 1 Processor -- start 2023-05-22T15:26:59.282 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.270+0000 7efede70b640 1 -- start start 2023-05-22T15:26:59.282 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.270+0000 7efede70b640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7efed815c920 0x7efed80b3200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:59.282 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.270+0000 7efede70b640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efed815cd70 0x7efed814aca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:59.282 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.270+0000 7efede70b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efed815fc50 0x7efed814c5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:59.282 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.270+0000 7efede70b640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efed814b540 con 0x7efed815fc50 2023-05-22T15:26:59.282 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.270+0000 7efede70b640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efed814b820 con 0x7efed815cd70 2023-05-22T15:26:59.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.270+0000 7efede70b640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efed814bb00 con 0x7efed815c920 2023-05-22T15:26:59.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.270+0000 7efedcca8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efed815fc50 0x7efed814c5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:59.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.270+0000 7efedcca8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efed815fc50 0x7efed814c5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57314/0 (socket says 172.21.15.16:57314) 2023-05-22T15:26:59.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.270+0000 7efedcca8640 1 -- 172.21.15.16:0/1214826133 learned_addr learned my addr 172.21.15.16:0/1214826133 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:26:59.285 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.274+0000 7efed7fff640 1 --2- 172.21.15.16:0/1214826133 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7efed815c920 0x7efed80b3200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:59.285 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.274+0000 7efed77fe640 1 --2- 172.21.15.16:0/1214826133 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efed815cd70 0x7efed814aca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:59.285 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.274+0000 7efedcca8640 1 -- 172.21.15.16:0/1214826133 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7efed815c920 msgr2=0x7efed80b3200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:59.285 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.274+0000 7efedcca8640 1 --2- 172.21.15.16:0/1214826133 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7efed815c920 0x7efed80b3200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:59.285 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.274+0000 7efedcca8640 1 -- 172.21.15.16:0/1214826133 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efed815cd70 msgr2=0x7efed814aca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:59.285 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.274+0000 7efedcca8640 1 --2- 172.21.15.16:0/1214826133 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efed815cd70 0x7efed814aca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:59.286 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.274+0000 7efedcca8640 1 -- 172.21.15.16:0/1214826133 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efec80094d0 con 0x7efed815fc50 2023-05-22T15:26:59.286 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.274+0000 7efedcca8640 1 --2- 172.21.15.16:0/1214826133 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efed815fc50 0x7efed814c5e0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7efed000c780 tx=0x7efed000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:26:59.286 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.274+0000 7efed4ff9640 1 -- 172.21.15.16:0/1214826133 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efed0004b70 con 0x7efed815fc50 2023-05-22T15:26:59.286 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.274+0000 7efed4ff9640 1 -- 172.21.15.16:0/1214826133 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efed000ce90 con 0x7efed815fc50 2023-05-22T15:26:59.286 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.274+0000 7efed4ff9640 1 -- 172.21.15.16:0/1214826133 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efed000a850 con 0x7efed815fc50 2023-05-22T15:26:59.286 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.274+0000 7efede70b640 1 -- 172.21.15.16:0/1214826133 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efed8160560 con 0x7efed815fc50 2023-05-22T15:26:59.286 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.274+0000 7efede70b640 1 -- 172.21.15.16:0/1214826133 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efed80d0980 con 0x7efed815fc50 2023-05-22T15:26:59.287 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.274+0000 7efede70b640 1 -- 172.21.15.16:0/1214826133 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efed8168ad0 con 0x7efed815fc50 2023-05-22T15:26:59.287 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.274+0000 7efed4ff9640 1 -- 172.21.15.16:0/1214826133 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7efed000a9b0 con 0x7efed815fc50 2023-05-22T15:26:59.287 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.274+0000 7efed4ff9640 1 --2- 172.21.15.16:0/1214826133 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7efec4065380 0x7efec4067830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:59.287 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.274+0000 7efed4ff9640 1 -- 172.21.15.16:0/1214826133 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6813+0+0 (secure 0 0 0) 0x7efed0082cb0 con 0x7efed815fc50 2023-05-22T15:26:59.287 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.274+0000 7efed7fff640 1 --2- 172.21.15.16:0/1214826133 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7efec4065380 0x7efec4067830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:59.288 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.274+0000 7efed7fff640 1 --2- 172.21.15.16:0/1214826133 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7efec4065380 0x7efec4067830 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7efec80ae000 tx=0x7efec80a7000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:26:59.289 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.278+0000 7efed4ff9640 1 -- 172.21.15.16:0/1214826133 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7efed0054100 con 0x7efed815fc50 2023-05-22T15:26:59.446 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.438+0000 7efede70b640 1 -- 172.21.15.16:0/1214826133 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efed804dbe0 con 0x7efed815fc50 2023-05-22T15:26:59.447 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.438+0000 7efed4ff9640 1 -- 172.21.15.16:0/1214826133 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) v1 ==== 74+0+19604 (secure 0 0 0) 0x7efed0053cf0 con 0x7efed815fc50 2023-05-22T15:26:59.448 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:26:59.448 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":27,"fsid":"b2719834-7847-4d98-90a6-f65d57f85c49","created":"2023-05-22T15:26:25.259314+0000","modified":"2023-05-22T15:26:53.971378+0000","last_up_change":"2023-05-22T15:26:36.448524+0000","last_in_change":"2023-05-22T15:26:27.675668+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-05-22T15:26:38.019781+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-05-22T15:26:43.570514+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-05-22T15:26:47.879159+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-05-22T15:26:48.522130+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data_ec","create_time":"2023-05-22T15:26:51.616142+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"9e7ff854-b25f-4cbb-a5c2-1799d9c53d2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":37021},{"type":"v1","addr":"172.21.15.16:6825","nonce":37021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":37021},{"type":"v1","addr":"172.21.15.16:6827","nonce":37021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":37021},{"type":"v1","addr":"172.21.15.16:6831","nonce":37021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":37021},{"type":"v1","addr":"172.21.15.16:6829","nonce":37021}]},"public_addr":"172.21.15.16:6825/37021","cluster_addr":"172.21.15.16:6827/37021","heartbeat_back_addr":"172.21.15.16:6831/37021","heartbeat_front_addr":"172.21.15.16:6829/37021","state":["exists","up"]},{"osd":1,"uuid":"4702fbc9-a927-4565-aca5-41dc69cc8264","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":37008},{"type":"v1","addr":"172.21.15.16:6801","nonce":37008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":37008},{"type":"v1","addr":"172.21.15.16:6803","nonce":37008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":37008},{"type":"v1","addr":"172.21.15.16:6807","nonce":37008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":37008},{"type":"v1","addr":"172.21.15.16:6805","nonce":37008}]},"public_addr":"172.21.15.16:6801/37008","cluster_addr":"172.21.15.16:6803/37008","heartbeat_back_addr":"172.21.15.16:6807/37008","heartbeat_front_addr":"172.21.15.16:6805/37008","state":["exists","up"]},{"osd":2,"uuid":"1e7bba46-4eba-47ef-bb81-5e2d371638ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":37009},{"type":"v1","addr":"172.21.15.16:6809","nonce":37009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":37009},{"type":"v1","addr":"172.21.15.16:6811","nonce":37009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":37009},{"type":"v1","addr":"172.21.15.16:6815","nonce":37009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":37009},{"type":"v1","addr":"172.21.15.16:6813","nonce":37009}]},"public_addr":"172.21.15.16:6809/37009","cluster_addr":"172.21.15.16:6811/37009","heartbeat_back_addr":"172.21.15.16:6815/37009","heartbeat_front_addr":"172.21.15.16:6813/37009","state":["exists","up"]},{"osd":3,"uuid":"383cacc0-5a88-4904-990e-498ca59ef48a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":37020},{"type":"v1","addr":"172.21.15.16:6817","nonce":37020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":37020},{"type":"v1","addr":"172.21.15.16:6819","nonce":37020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":37020},{"type":"v1","addr":"172.21.15.16:6823","nonce":37020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":37020},{"type":"v1","addr":"172.21.15.16:6821","nonce":37020}]},"public_addr":"172.21.15.16:6817/37020","cluster_addr":"172.21.15.16:6819/37020","heartbeat_back_addr":"172.21.15.16:6823/37020","heartbeat_front_addr":"172.21.15.16:6821/37020","state":["exists","up"]},{"osd":4,"uuid":"970500ef-95b7-42c1-b080-eed4fae7013d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":37353},{"type":"v1","addr":"172.21.15.62:6817","nonce":37353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6819","nonce":37353},{"type":"v1","addr":"172.21.15.62:6822","nonce":37353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37353},{"type":"v1","addr":"172.21.15.62:6831","nonce":37353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37353},{"type":"v1","addr":"172.21.15.62:6826","nonce":37353}]},"public_addr":"172.21.15.62:6817/37353","cluster_addr":"172.21.15.62:6822/37353","heartbeat_back_addr":"172.21.15.62:6831/37353","heartbeat_front_addr":"172.21.15.62:6826/37353","state":["exists","up"]},{"osd":5,"uuid":"022d67db-d4d8-445b-b1b9-8b446a00f5c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":37352},{"type":"v1","addr":"172.21.15.62:6801","nonce":37352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":37352},{"type":"v1","addr":"172.21.15.62:6803","nonce":37352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":37352},{"type":"v1","addr":"172.21.15.62:6807","nonce":37352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":37352},{"type":"v1","addr":"172.21.15.62:6805","nonce":37352}]},"public_addr":"172.21.15.62:6801/37352","cluster_addr":"172.21.15.62:6803/37352","heartbeat_back_addr":"172.21.15.62:6807/37352","heartbeat_front_addr":"172.21.15.62:6805/37352","state":["exists","up"]},{"osd":6,"uuid":"42bd372b-6688-4a1b-8407-e16e8c266e33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":37354},{"type":"v1","addr":"172.21.15.62:6820","nonce":37354}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":37354},{"type":"v1","addr":"172.21.15.62:6823","nonce":37354}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6829","nonce":37354},{"type":"v1","addr":"172.21.15.62:6830","nonce":37354}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6825","nonce":37354},{"type":"v1","addr":"172.21.15.62:6827","nonce":37354}]},"public_addr":"172.21.15.62:6820/37354","cluster_addr":"172.21.15.62:6823/37354","heartbeat_back_addr":"172.21.15.62:6830/37354","heartbeat_front_addr":"172.21.15.62:6827/37354","state":["exists","up"]},{"osd":7,"uuid":"ac70b2a3-f41b-4bf2-b227-7f2cbb55426d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37355},{"type":"v1","addr":"172.21.15.62:6809","nonce":37355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37355},{"type":"v1","addr":"172.21.15.62:6811","nonce":37355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":37355},{"type":"v1","addr":"172.21.15.62:6815","nonce":37355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":37355},{"type":"v1","addr":"172.21.15.62:6813","nonce":37355}]},"public_addr":"172.21.15.62:6809/37355","cluster_addr":"172.21.15.62:6811/37355","heartbeat_back_addr":"172.21.15.62:6815/37355","heartbeat_front_addr":"172.21.15.62:6813/37355","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.368896+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.107273+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.276888+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.287502+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.124791+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.251937+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.255679+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.232797+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-05-22T15:26:59.450 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.438+0000 7efede70b640 1 -- 172.21.15.16:0/1214826133 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7efec4065380 msgr2=0x7efec4067830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:59.450 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.438+0000 7efede70b640 1 --2- 172.21.15.16:0/1214826133 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7efec4065380 0x7efec4067830 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7efec80ae000 tx=0x7efec80a7000 comp rx=0 tx=0).stop 2023-05-22T15:26:59.450 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.438+0000 7efede70b640 1 -- 172.21.15.16:0/1214826133 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efed815fc50 msgr2=0x7efed814c5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:59.450 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.438+0000 7efede70b640 1 --2- 172.21.15.16:0/1214826133 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efed815fc50 0x7efed814c5e0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7efed000c780 tx=0x7efed000cc40 comp rx=0 tx=0).stop 2023-05-22T15:26:59.451 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.442+0000 7efede70b640 1 -- 172.21.15.16:0/1214826133 shutdown_connections 2023-05-22T15:26:59.451 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.442+0000 7efede70b640 1 --2- 172.21.15.16:0/1214826133 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efed815fc50 0x7efed814c5e0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:59.451 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.442+0000 7efede70b640 1 --2- 172.21.15.16:0/1214826133 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efed815cd70 0x7efed814aca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:59.451 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.442+0000 7efede70b640 1 --2- 172.21.15.16:0/1214826133 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7efec4065380 0x7efec4067830 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:59.451 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.442+0000 7efede70b640 1 --2- 172.21.15.16:0/1214826133 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7efed815c920 0x7efed80b3200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:59.452 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.442+0000 7efede70b640 1 -- 172.21.15.16:0/1214826133 >> 172.21.15.16:0/1214826133 conn(0x7efed80657f0 msgr2=0x7efed80b3be0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:26:59.452 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.442+0000 7efede70b640 1 -- 172.21.15.16:0/1214826133 shutdown_connections 2023-05-22T15:26:59.452 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.442+0000 7efede70b640 1 -- 172.21.15.16:0/1214826133 wait complete. 2023-05-22T15:26:59.467 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-05-22T15:26:38.019781+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-05-22T15:26:43.570514+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2023-05-22T15:26:47.879159+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2023-05-22T15:26:48.522130+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-05-22T15:26:51.616142+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '27', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-05-22T15:26:59.468 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-05-22T15:26:59.595 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.586+0000 7f9ba8ea4640 1 Processor -- start 2023-05-22T15:26:59.595 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.586+0000 7f9ba8ea4640 1 -- start start 2023-05-22T15:26:59.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.586+0000 7f9ba8ea4640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f9ba4159d40 0x7f9ba40cb260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:59.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.586+0000 7f9ba8ea4640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f9ba406b6c0 0x7f9ba406bb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:59.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.586+0000 7f9ba8ea4640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ba4070c30 con 0x7f9ba4159970 2023-05-22T15:26:59.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.586+0000 7f9ba8ea4640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ba415dd90 con 0x7f9ba4159d40 2023-05-22T15:26:59.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.586+0000 7f9ba8ea4640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ba406c040 con 0x7f9ba406b6c0 2023-05-22T15:26:59.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.586+0000 7f9ba1d9b640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f9ba4159d40 0x7f9ba40cb260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:59.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.586+0000 7f9ba1d9b640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f9ba4159d40 0x7f9ba40cb260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57324/0 (socket says 172.21.15.16:57324) 2023-05-22T15:26:59.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.586+0000 7f9ba1d9b640 1 -- 172.21.15.16:0/574887492 learned_addr learned my addr 172.21.15.16:0/574887492 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:26:59.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.586+0000 7f9ba259c640 1 --1- 172.21.15.16:0/574887492 >> v1:172.21.15.16:6789/0 conn(0x7f9ba4159970 0x7f9ba415d680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:50942/0 (socket says 172.21.15.16:50942) 2023-05-22T15:26:59.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.586+0000 7f9ba2d9d640 1 --2- 172.21.15.16:0/574887492 >> v2:172.21.15.62:3300/0 conn(0x7f9ba406b6c0 0x7f9ba406bb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:59.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.586+0000 7f9ba159a640 1 -- 172.21.15.16:0/574887492 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1441639122 0 0) 0x7f9ba4070c30 con 0x7f9ba4159970 2023-05-22T15:26:59.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.586+0000 7f9ba159a640 1 -- 172.21.15.16:0/574887492 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b90003530 con 0x7f9ba4159970 2023-05-22T15:26:59.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.586+0000 7f9ba1d9b640 1 -- 172.21.15.16:0/574887492 >> v2:172.21.15.62:3300/0 conn(0x7f9ba406b6c0 msgr2=0x7f9ba406bb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:26:59.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.586+0000 7f9ba1d9b640 1 --2- 172.21.15.16:0/574887492 >> v2:172.21.15.62:3300/0 conn(0x7f9ba406b6c0 0x7f9ba406bb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:59.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.586+0000 7f9ba1d9b640 1 -- 172.21.15.16:0/574887492 >> v1:172.21.15.16:6789/0 conn(0x7f9ba4159970 legacy=0x7f9ba415d680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:59.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.586+0000 7f9ba1d9b640 1 -- 172.21.15.16:0/574887492 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ba4070e60 con 0x7f9ba4159d40 2023-05-22T15:26:59.600 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.590+0000 7f9ba1d9b640 1 --2- 172.21.15.16:0/574887492 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9ba4159d40 0x7f9ba40cb260 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f9ba40707a0 tx=0x7f9b9401c470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42bcbcb6df213b4e server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:26:59.601 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.590+0000 7f9ba159a640 1 -- 172.21.15.16:0/574887492 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b940ad660 con 0x7f9ba4159d40 2023-05-22T15:26:59.602 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.590+0000 7f9ba159a640 1 -- 172.21.15.16:0/574887492 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b940ad7c0 con 0x7f9ba4159d40 2023-05-22T15:26:59.602 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.590+0000 7f9ba159a640 1 -- 172.21.15.16:0/574887492 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b940adb30 con 0x7f9ba4159d40 2023-05-22T15:26:59.602 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.590+0000 7f9ba8ea4640 1 -- 172.21.15.16:0/574887492 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9ba4159d40 msgr2=0x7f9ba40cb260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:59.602 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.590+0000 7f9ba8ea4640 1 --2- 172.21.15.16:0/574887492 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9ba4159d40 0x7f9ba40cb260 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f9ba40707a0 tx=0x7f9b9401c470 comp rx=0 tx=0).stop 2023-05-22T15:26:59.602 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.590+0000 7f9ba8ea4640 1 -- 172.21.15.16:0/574887492 shutdown_connections 2023-05-22T15:26:59.602 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.590+0000 7f9ba8ea4640 1 --2- 172.21.15.16:0/574887492 >> v2:172.21.15.62:3300/0 conn(0x7f9ba406b6c0 0x7f9ba406bb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:59.602 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.590+0000 7f9ba8ea4640 1 --2- 172.21.15.16:0/574887492 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9ba4159d40 0x7f9ba40cb260 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:59.603 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.590+0000 7f9ba8ea4640 1 -- 172.21.15.16:0/574887492 >> 172.21.15.16:0/574887492 conn(0x7f9ba4069720 msgr2=0x7f9ba4069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:26:59.603 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.590+0000 7f9ba8ea4640 1 -- 172.21.15.16:0/574887492 shutdown_connections 2023-05-22T15:26:59.603 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.590+0000 7f9ba8ea4640 1 -- 172.21.15.16:0/574887492 wait complete. 2023-05-22T15:26:59.604 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.590+0000 7f9ba8ea4640 1 Processor -- start 2023-05-22T15:26:59.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.590+0000 7f9ba8ea4640 1 -- start start 2023-05-22T15:26:59.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.594+0000 7f9ba8ea4640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9ba406b6c0 0x7f9ba4150710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:59.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.594+0000 7f9ba8ea4640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9ba4159970 0x7f9ba4152950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:59.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.594+0000 7f9ba8ea4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9ba4163430 0x7f9ba41f7d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:59.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.594+0000 7f9ba8ea4640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ba41518b0 con 0x7f9ba4163430 2023-05-22T15:26:59.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.594+0000 7f9ba8ea4640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ba4151010 con 0x7f9ba406b6c0 2023-05-22T15:26:59.606 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.594+0000 7f9ba8ea4640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ba41512f0 con 0x7f9ba4159970 2023-05-22T15:26:59.606 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.594+0000 7f9ba2d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9ba4163430 0x7f9ba41f7d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:59.606 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.594+0000 7f9ba2d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9ba4163430 0x7f9ba41f7d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57330/0 (socket says 172.21.15.16:57330) 2023-05-22T15:26:59.606 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.594+0000 7f9ba2d9d640 1 -- 172.21.15.16:0/3904354382 learned_addr learned my addr 172.21.15.16:0/3904354382 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:26:59.606 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.594+0000 7f9ba259c640 1 --2- 172.21.15.16:0/3904354382 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9ba406b6c0 0x7f9ba4150710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:59.606 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.594+0000 7f9ba1d9b640 1 --2- 172.21.15.16:0/3904354382 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9ba4159970 0x7f9ba4152950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:59.606 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.594+0000 7f9ba2d9d640 1 -- 172.21.15.16:0/3904354382 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9ba4159970 msgr2=0x7f9ba4152950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:59.606 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.594+0000 7f9ba2d9d640 1 --2- 172.21.15.16:0/3904354382 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9ba4159970 0x7f9ba4152950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:59.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.594+0000 7f9ba2d9d640 1 -- 172.21.15.16:0/3904354382 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9ba406b6c0 msgr2=0x7f9ba4150710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:59.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.594+0000 7f9ba2d9d640 1 --2- 172.21.15.16:0/3904354382 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9ba406b6c0 0x7f9ba4150710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:59.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.594+0000 7f9ba2d9d640 1 -- 172.21.15.16:0/3904354382 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b940094d0 con 0x7f9ba4163430 2023-05-22T15:26:59.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.594+0000 7f9ba259c640 1 --2- 172.21.15.16:0/3904354382 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9ba406b6c0 0x7f9ba4150710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:26:59.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.594+0000 7f9ba2d9d640 1 --2- 172.21.15.16:0/3904354382 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9ba4163430 0x7f9ba41f7d00 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f9b9c007e40 tx=0x7f9b9c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:26:59.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.594+0000 7f9b9affd640 1 -- 172.21.15.16:0/3904354382 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b9c014620 con 0x7f9ba4163430 2023-05-22T15:26:59.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.594+0000 7f9b9affd640 1 -- 172.21.15.16:0/3904354382 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b9c00ee90 con 0x7f9ba4163430 2023-05-22T15:26:59.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.594+0000 7f9b9affd640 1 -- 172.21.15.16:0/3904354382 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b9c0149e0 con 0x7f9ba4163430 2023-05-22T15:26:59.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.594+0000 7f9ba8ea4640 1 -- 172.21.15.16:0/3904354382 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ba40cb7a0 con 0x7f9ba4163430 2023-05-22T15:26:59.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.594+0000 7f9ba8ea4640 1 -- 172.21.15.16:0/3904354382 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ba4071700 con 0x7f9ba4163430 2023-05-22T15:26:59.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.594+0000 7f9ba8ea4640 1 -- 172.21.15.16:0/3904354382 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ba40cb930 con 0x7f9ba4163430 2023-05-22T15:26:59.609 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.594+0000 7f9b9affd640 1 -- 172.21.15.16:0/3904354382 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7f9b9c014b40 con 0x7f9ba4163430 2023-05-22T15:26:59.609 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.594+0000 7f9b9affd640 1 --2- 172.21.15.16:0/3904354382 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f9b84065380 0x7f9b84067830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:59.609 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.594+0000 7f9b9affd640 1 -- 172.21.15.16:0/3904354382 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f9b9c0834c0 con 0x7f9ba4163430 2023-05-22T15:26:59.609 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.594+0000 7f9ba259c640 1 --2- 172.21.15.16:0/3904354382 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f9b84065380 0x7f9b84067830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:59.609 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.598+0000 7f9ba259c640 1 --2- 172.21.15.16:0/3904354382 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f9b84065380 0x7f9b84067830 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f9b8c000f70 tx=0x7f9b8c009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:26:59.610 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.598+0000 7f9b9affd640 1 -- 172.21.15.16:0/3904354382 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f9b9c050ee0 con 0x7f9ba4163430 2023-05-22T15:26:59.763 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.754+0000 7f9ba8ea4640 1 -- 172.21.15.16:0/3904354382 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f9ba404dbe0 con 0x7f9ba4163430 2023-05-22T15:26:59.764 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.754+0000 7f9b9affd640 1 -- 172.21.15.16:0/3904354382 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) v1 ==== 93+0+10 (secure 0 0 0) 0x7f9b9c054500 con 0x7f9ba4163430 2023-05-22T15:26:59.764 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2023-05-22T15:26:59.766 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.758+0000 7f9ba8ea4640 1 -- 172.21.15.16:0/3904354382 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f9b84065380 msgr2=0x7f9b84067830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:59.767 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.758+0000 7f9ba8ea4640 1 --2- 172.21.15.16:0/3904354382 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f9b84065380 0x7f9b84067830 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f9b8c000f70 tx=0x7f9b8c009040 comp rx=0 tx=0).stop 2023-05-22T15:26:59.767 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.758+0000 7f9ba8ea4640 1 -- 172.21.15.16:0/3904354382 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9ba4163430 msgr2=0x7f9ba41f7d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:59.767 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.758+0000 7f9ba8ea4640 1 --2- 172.21.15.16:0/3904354382 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9ba4163430 0x7f9ba41f7d00 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f9b9c007e40 tx=0x7f9b9c00e4b0 comp rx=0 tx=0).stop 2023-05-22T15:26:59.767 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.758+0000 7f9ba8ea4640 1 -- 172.21.15.16:0/3904354382 shutdown_connections 2023-05-22T15:26:59.767 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.758+0000 7f9ba8ea4640 1 --2- 172.21.15.16:0/3904354382 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9ba4163430 0x7f9ba41f7d00 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:59.767 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.758+0000 7f9ba8ea4640 1 --2- 172.21.15.16:0/3904354382 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f9b84065380 0x7f9b84067830 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:59.767 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.758+0000 7f9ba8ea4640 1 --2- 172.21.15.16:0/3904354382 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9ba4159970 0x7f9ba4152950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:59.768 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.758+0000 7f9ba8ea4640 1 --2- 172.21.15.16:0/3904354382 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9ba406b6c0 0x7f9ba4150710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:59.768 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.758+0000 7f9ba8ea4640 1 -- 172.21.15.16:0/3904354382 >> 172.21.15.16:0/3904354382 conn(0x7f9ba4069720 msgr2=0x7f9ba40c9890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:26:59.768 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.758+0000 7f9ba8ea4640 1 -- 172.21.15.16:0/3904354382 shutdown_connections 2023-05-22T15:26:59.768 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.758+0000 7f9ba8ea4640 1 -- 172.21.15.16:0/3904354382 wait complete. 2023-05-22T15:26:59.783 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-05-22T15:26:59.906 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.898+0000 7ff600668640 1 Processor -- start 2023-05-22T15:26:59.907 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.898+0000 7ff600668640 1 -- start start 2023-05-22T15:26:59.908 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.898+0000 7ff600668640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff5f80d18a0 0x7ff5f8166ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:59.908 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.898+0000 7ff600668640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5f80d0400 con 0x7ff5f815bca0 2023-05-22T15:26:59.908 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.898+0000 7ff600668640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5f80d00d0 con 0x7ff5f80ce900 2023-05-22T15:26:59.908 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.898+0000 7ff600668640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5f815c070 con 0x7ff5f80d18a0 2023-05-22T15:26:59.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.898+0000 7ff5fe404640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7ff5f815bca0 0x7ff5f8160160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:50956/0 (socket says 172.21.15.16:50956) 2023-05-22T15:26:59.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.898+0000 7ff5fe404640 1 -- 172.21.15.16:0/3584282064 learned_addr learned my addr 172.21.15.16:0/3584282064 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:26:59.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.898+0000 7ff5fec05640 1 --2- 172.21.15.16:0/3584282064 >> v2:172.21.15.16:3300/0 conn(0x7ff5f80d18a0 0x7ff5f8166ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:59.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.898+0000 7ff5fd402640 1 -- 172.21.15.16:0/3584282064 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 199224633 0 0) 0x7ff5f80d0400 con 0x7ff5f815bca0 2023-05-22T15:26:59.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.898+0000 7ff5fd402640 1 -- 172.21.15.16:0/3584282064 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff5e4003530 con 0x7ff5f815bca0 2023-05-22T15:26:59.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.898+0000 7ff5fec05640 1 -- 172.21.15.16:0/3584282064 >> v1:172.21.15.62:6789/0 conn(0x7ff5f80ce900 legacy=0x7ff5f80ced80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:59.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.898+0000 7ff5fec05640 1 -- 172.21.15.16:0/3584282064 >> v1:172.21.15.16:6789/0 conn(0x7ff5f815bca0 legacy=0x7ff5f8160160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:59.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.898+0000 7ff5fec05640 1 -- 172.21.15.16:0/3584282064 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5f80ccbe0 con 0x7ff5f80d18a0 2023-05-22T15:26:59.911 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.902+0000 7ff5fec05640 1 --2- 172.21.15.16:0/3584282064 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff5f80d18a0 0x7ff5f8166ae0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7ff5f0009d90 tx=0x7ff5f001c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66b332196270f7d0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:26:59.911 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.902+0000 7ff5fd402640 1 -- 172.21.15.16:0/3584282064 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5f00a5880 con 0x7ff5f80d18a0 2023-05-22T15:26:59.911 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.902+0000 7ff5fd402640 1 -- 172.21.15.16:0/3584282064 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5f00a59e0 con 0x7ff5f80d18a0 2023-05-22T15:26:59.911 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.902+0000 7ff5fd402640 1 -- 172.21.15.16:0/3584282064 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5f00a5d70 con 0x7ff5f80d18a0 2023-05-22T15:26:59.911 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.902+0000 7ff600668640 1 -- 172.21.15.16:0/3584282064 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff5f80d18a0 msgr2=0x7ff5f8166ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:59.911 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.902+0000 7ff600668640 1 --2- 172.21.15.16:0/3584282064 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff5f80d18a0 0x7ff5f8166ae0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7ff5f0009d90 tx=0x7ff5f001c890 comp rx=0 tx=0).stop 2023-05-22T15:26:59.912 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.902+0000 7ff600668640 1 -- 172.21.15.16:0/3584282064 shutdown_connections 2023-05-22T15:26:59.912 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.902+0000 7ff600668640 1 --2- 172.21.15.16:0/3584282064 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff5f80d18a0 0x7ff5f8166ae0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:59.912 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.902+0000 7ff600668640 1 -- 172.21.15.16:0/3584282064 >> 172.21.15.16:0/3584282064 conn(0x7ff5f80657f0 msgr2=0x7ff5f8070e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:26:59.912 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.902+0000 7ff600668640 1 -- 172.21.15.16:0/3584282064 shutdown_connections 2023-05-22T15:26:59.912 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.902+0000 7ff600668640 1 -- 172.21.15.16:0/3584282064 wait complete. 2023-05-22T15:26:59.914 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.902+0000 7ff600668640 1 Processor -- start 2023-05-22T15:26:59.914 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.902+0000 7ff600668640 1 -- start start 2023-05-22T15:26:59.914 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.902+0000 7ff600668640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff5f80ce900 0x7ff5f80ca5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:59.914 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.902+0000 7ff600668640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff5f80d18a0 0x7ff5f80c85f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:59.915 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.902+0000 7ff600668640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff5f815bca0 0x7ff5f8152400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:59.915 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.902+0000 7ff600668640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5f80c8160 con 0x7ff5f80d18a0 2023-05-22T15:26:59.915 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.902+0000 7ff600668640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5f80c7550 con 0x7ff5f80ce900 2023-05-22T15:26:59.915 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.902+0000 7ff600668640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5f80c7830 con 0x7ff5f815bca0 2023-05-22T15:26:59.915 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.902+0000 7ff5fdc03640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff5f80d18a0 0x7ff5f80c85f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:59.915 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.902+0000 7ff5fdc03640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff5f80d18a0 0x7ff5f80c85f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57346/0 (socket says 172.21.15.16:57346) 2023-05-22T15:26:59.915 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.902+0000 7ff5fdc03640 1 -- 172.21.15.16:0/2733675343 learned_addr learned my addr 172.21.15.16:0/2733675343 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:26:59.915 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.902+0000 7ff5fe404640 1 --2- 172.21.15.16:0/2733675343 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff5f80ce900 0x7ff5f80ca5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:59.916 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.902+0000 7ff5fec05640 1 --2- 172.21.15.16:0/2733675343 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff5f815bca0 0x7ff5f8152400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:59.916 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.902+0000 7ff5fdc03640 1 -- 172.21.15.16:0/2733675343 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff5f815bca0 msgr2=0x7ff5f8152400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:59.916 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.902+0000 7ff5fdc03640 1 --2- 172.21.15.16:0/2733675343 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff5f815bca0 0x7ff5f8152400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:59.916 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.902+0000 7ff5fdc03640 1 -- 172.21.15.16:0/2733675343 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff5f80ce900 msgr2=0x7ff5f80ca5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:26:59.916 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.902+0000 7ff5fdc03640 1 --2- 172.21.15.16:0/2733675343 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff5f80ce900 0x7ff5f80ca5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:26:59.916 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.902+0000 7ff5fdc03640 1 -- 172.21.15.16:0/2733675343 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5f0009a70 con 0x7ff5f80d18a0 2023-05-22T15:26:59.916 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.902+0000 7ff5fdc03640 1 --2- 172.21.15.16:0/2733675343 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff5f80d18a0 0x7ff5f80c85f0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7ff5f400c500 tx=0x7ff5f400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:26:59.917 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.902+0000 7ff5eeffd640 1 -- 172.21.15.16:0/2733675343 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5f4005220 con 0x7ff5f80d18a0 2023-05-22T15:26:59.917 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.902+0000 7ff5eeffd640 1 -- 172.21.15.16:0/2733675343 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5f4005380 con 0x7ff5f80d18a0 2023-05-22T15:26:59.917 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.902+0000 7ff600668640 1 -- 172.21.15.16:0/2733675343 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5f81699d0 con 0x7ff5f80d18a0 2023-05-22T15:26:59.917 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.902+0000 7ff5eeffd640 1 -- 172.21.15.16:0/2733675343 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5f4007270 con 0x7ff5f80d18a0 2023-05-22T15:26:59.917 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.902+0000 7ff600668640 1 -- 172.21.15.16:0/2733675343 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5f8169840 con 0x7ff5f80d18a0 2023-05-22T15:26:59.918 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.906+0000 7ff600668640 1 -- 172.21.15.16:0/2733675343 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5f8169b60 con 0x7ff5f80d18a0 2023-05-22T15:26:59.918 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.906+0000 7ff5eeffd640 1 -- 172.21.15.16:0/2733675343 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7ff5f40073d0 con 0x7ff5f80d18a0 2023-05-22T15:26:59.918 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.906+0000 7ff5eeffd640 1 --2- 172.21.15.16:0/2733675343 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ff5dc065380 0x7ff5dc067830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:26:59.918 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.906+0000 7ff5fe404640 1 --2- 172.21.15.16:0/2733675343 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ff5dc065380 0x7ff5dc067830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:26:59.918 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.906+0000 7ff5eeffd640 1 -- 172.21.15.16:0/2733675343 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6813+0+0 (secure 0 0 0) 0x7ff5f4082540 con 0x7ff5f80d18a0 2023-05-22T15:26:59.919 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.906+0000 7ff5fe404640 1 --2- 172.21.15.16:0/2733675343 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ff5dc065380 0x7ff5dc067830 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7ff5e80011f0 tx=0x7ff5e8009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:26:59.920 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:26:59.910+0000 7ff5eeffd640 1 -- 172.21.15.16:0/2733675343 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7ff5f404fe40 con 0x7ff5f80d18a0 2023-05-22T15:27:00.074 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.062+0000 7ff600668640 1 -- 172.21.15.16:0/2733675343 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7ff5f804dbe0 con 0x7ff5f80d18a0 2023-05-22T15:27:00.074 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.066+0000 7ff5eeffd640 1 -- 172.21.15.16:0/2733675343 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v27) v1 ==== 92+0+10 (secure 0 0 0) 0x7ff5f4053460 con 0x7ff5f80d18a0 2023-05-22T15:27:00.074 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 8 2023-05-22T15:27:00.077 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.066+0000 7ff600668640 1 -- 172.21.15.16:0/2733675343 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ff5dc065380 msgr2=0x7ff5dc067830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:00.077 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.066+0000 7ff600668640 1 --2- 172.21.15.16:0/2733675343 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ff5dc065380 0x7ff5dc067830 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7ff5e80011f0 tx=0x7ff5e8009040 comp rx=0 tx=0).stop 2023-05-22T15:27:00.078 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.066+0000 7ff600668640 1 -- 172.21.15.16:0/2733675343 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff5f80d18a0 msgr2=0x7ff5f80c85f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:00.078 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.066+0000 7ff600668640 1 --2- 172.21.15.16:0/2733675343 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff5f80d18a0 0x7ff5f80c85f0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7ff5f400c500 tx=0x7ff5f400c9c0 comp rx=0 tx=0).stop 2023-05-22T15:27:00.078 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.066+0000 7ff600668640 1 -- 172.21.15.16:0/2733675343 shutdown_connections 2023-05-22T15:27:00.078 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.066+0000 7ff600668640 1 --2- 172.21.15.16:0/2733675343 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ff5dc065380 0x7ff5dc067830 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:00.078 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.066+0000 7ff600668640 1 --2- 172.21.15.16:0/2733675343 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff5f815bca0 0x7ff5f8152400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:00.079 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.066+0000 7ff600668640 1 --2- 172.21.15.16:0/2733675343 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff5f80d18a0 0x7ff5f80c85f0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:00.079 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.066+0000 7ff600668640 1 --2- 172.21.15.16:0/2733675343 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff5f80ce900 0x7ff5f80ca5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:00.079 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.066+0000 7ff600668640 1 -- 172.21.15.16:0/2733675343 >> 172.21.15.16:0/2733675343 conn(0x7ff5f80657f0 msgr2=0x7ff5f80b23d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:00.079 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.066+0000 7ff600668640 1 -- 172.21.15.16:0/2733675343 shutdown_connections 2023-05-22T15:27:00.079 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.066+0000 7ff600668640 1 -- 172.21.15.16:0/2733675343 wait complete. 2023-05-22T15:27:00.094 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-05-22T15:27:00.214 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.206+0000 7fe0cf93b640 1 Processor -- start 2023-05-22T15:27:00.214 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.206+0000 7fe0cf93b640 1 -- start start 2023-05-22T15:27:00.215 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.206+0000 7fe0cf93b640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe0c806db10 0x7fe0c806dee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:00.215 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.206+0000 7fe0cf93b640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe0c806a830 0x7fe0c815fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:00.215 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.206+0000 7fe0cf93b640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0c815bfc0 con 0x7fe0c806c5f0 2023-05-22T15:27:00.216 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.206+0000 7fe0cf93b640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0c815f790 con 0x7fe0c806a830 2023-05-22T15:27:00.216 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.206+0000 7fe0cf93b640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0c806e420 con 0x7fe0c806db10 2023-05-22T15:27:00.216 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.206+0000 7fe0cded8640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe0c806a830 0x7fe0c815fbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:00.217 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.206+0000 7fe0cded8640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe0c806a830 0x7fe0c815fbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57362/0 (socket says 172.21.15.16:57362) 2023-05-22T15:27:00.217 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.206+0000 7fe0cded8640 1 -- 172.21.15.16:0/1209103338 learned_addr learned my addr 172.21.15.16:0/1209103338 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:00.218 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.206+0000 7fe0cd6d7640 1 --2- 172.21.15.16:0/1209103338 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe0c806db10 0x7fe0c806dee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:00.218 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.206+0000 7fe0cded8640 1 -- 172.21.15.16:0/1209103338 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe0c806db10 msgr2=0x7fe0c806dee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:00.218 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.206+0000 7fe0cded8640 1 --2- 172.21.15.16:0/1209103338 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe0c806db10 0x7fe0c806dee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:00.218 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.206+0000 7fe0cded8640 1 -- 172.21.15.16:0/1209103338 >> v1:172.21.15.62:6789/0 conn(0x7fe0c806c5f0 legacy=0x7fe0c80cc7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:00.218 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.206+0000 7fe0cded8640 1 -- 172.21.15.16:0/1209103338 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0c80cf620 con 0x7fe0c806a830 2023-05-22T15:27:00.219 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.206+0000 7fe0cded8640 1 --2- 172.21.15.16:0/1209103338 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe0c806a830 0x7fe0c815fbe0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fe0c0009870 tx=0x7fe0c001c510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=751551afc0f86d38 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:00.219 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.210+0000 7fe0bffff640 1 -- 172.21.15.16:0/1209103338 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0c00aa670 con 0x7fe0c806a830 2023-05-22T15:27:00.219 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.210+0000 7fe0bffff640 1 -- 172.21.15.16:0/1209103338 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0c00aa7d0 con 0x7fe0c806a830 2023-05-22T15:27:00.219 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.210+0000 7fe0bffff640 1 -- 172.21.15.16:0/1209103338 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0c00aab60 con 0x7fe0c806a830 2023-05-22T15:27:00.220 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.210+0000 7fe0cf93b640 1 -- 172.21.15.16:0/1209103338 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe0c806a830 msgr2=0x7fe0c815fbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:00.220 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.210+0000 7fe0cf93b640 1 --2- 172.21.15.16:0/1209103338 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe0c806a830 0x7fe0c815fbe0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fe0c0009870 tx=0x7fe0c001c510 comp rx=0 tx=0).stop 2023-05-22T15:27:00.220 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.210+0000 7fe0cf93b640 1 -- 172.21.15.16:0/1209103338 shutdown_connections 2023-05-22T15:27:00.220 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.210+0000 7fe0cf93b640 1 --2- 172.21.15.16:0/1209103338 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe0c806a830 0x7fe0c815fbe0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:00.221 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.210+0000 7fe0cf93b640 1 --2- 172.21.15.16:0/1209103338 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe0c806db10 0x7fe0c806dee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:00.221 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.210+0000 7fe0cf93b640 1 -- 172.21.15.16:0/1209103338 >> 172.21.15.16:0/1209103338 conn(0x7fe0c806f4f0 msgr2=0x7fe0c806f8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:00.221 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.210+0000 7fe0cf93b640 1 -- 172.21.15.16:0/1209103338 shutdown_connections 2023-05-22T15:27:00.221 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.210+0000 7fe0cf93b640 1 -- 172.21.15.16:0/1209103338 wait complete. 2023-05-22T15:27:00.221 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.210+0000 7fe0cf93b640 1 Processor -- start 2023-05-22T15:27:00.221 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.210+0000 7fe0cf93b640 1 -- start start 2023-05-22T15:27:00.223 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.210+0000 7fe0cf93b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe0c806a830 0x7fe0c81557c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:00.223 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.210+0000 7fe0cf93b640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe0c806c5f0 0x7fe0c80c9a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:00.223 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.210+0000 7fe0cf93b640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe0c806db10 0x7fe0c815a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:00.223 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.210+0000 7fe0cf93b640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0c80cb080 con 0x7fe0c806a830 2023-05-22T15:27:00.223 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.210+0000 7fe0cf93b640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0c80cb4a0 con 0x7fe0c806db10 2023-05-22T15:27:00.224 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.210+0000 7fe0cf93b640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0c80ca400 con 0x7fe0c806c5f0 2023-05-22T15:27:00.224 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.210+0000 7fe0cd6d7640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe0c806a830 0x7fe0c81557c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:00.224 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.210+0000 7fe0cd6d7640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe0c806a830 0x7fe0c81557c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57366/0 (socket says 172.21.15.16:57366) 2023-05-22T15:27:00.224 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.210+0000 7fe0cd6d7640 1 -- 172.21.15.16:0/3791102177 learned_addr learned my addr 172.21.15.16:0/3791102177 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:00.224 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.210+0000 7fe0cded8640 1 --2- 172.21.15.16:0/3791102177 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe0c806db10 0x7fe0c815a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:00.224 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.210+0000 7fe0cced6640 1 --2- 172.21.15.16:0/3791102177 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe0c806c5f0 0x7fe0c80c9a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:00.224 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.210+0000 7fe0cd6d7640 1 -- 172.21.15.16:0/3791102177 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe0c806c5f0 msgr2=0x7fe0c80c9a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:00.225 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.210+0000 7fe0cd6d7640 1 --2- 172.21.15.16:0/3791102177 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe0c806c5f0 0x7fe0c80c9a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:00.225 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.210+0000 7fe0cd6d7640 1 -- 172.21.15.16:0/3791102177 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe0c806db10 msgr2=0x7fe0c815a890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:00.225 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.210+0000 7fe0cd6d7640 1 --2- 172.21.15.16:0/3791102177 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe0c806db10 0x7fe0c815a890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:00.225 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.210+0000 7fe0cd6d7640 1 -- 172.21.15.16:0/3791102177 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0c00094d0 con 0x7fe0c806a830 2023-05-22T15:27:00.225 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.210+0000 7fe0cded8640 1 --2- 172.21.15.16:0/3791102177 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe0c806db10 0x7fe0c815a890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T15:27:00.225 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.210+0000 7fe0cd6d7640 1 --2- 172.21.15.16:0/3791102177 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe0c806a830 0x7fe0c81557c0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fe0c4007e40 tx=0x7fe0c400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:00.225 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.210+0000 7fe0bdffb640 1 -- 172.21.15.16:0/3791102177 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0c4014620 con 0x7fe0c806a830 2023-05-22T15:27:00.226 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.210+0000 7fe0bdffb640 1 -- 172.21.15.16:0/3791102177 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0c400ee90 con 0x7fe0c806a830 2023-05-22T15:27:00.226 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.210+0000 7fe0cf93b640 1 -- 172.21.15.16:0/3791102177 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0c80cfec0 con 0x7fe0c806a830 2023-05-22T15:27:00.226 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.210+0000 7fe0bdffb640 1 -- 172.21.15.16:0/3791102177 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0c40149c0 con 0x7fe0c806a830 2023-05-22T15:27:00.226 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.210+0000 7fe0cf93b640 1 -- 172.21.15.16:0/3791102177 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0c8160530 con 0x7fe0c806a830 2023-05-22T15:27:00.226 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.214+0000 7fe0cf93b640 1 -- 172.21.15.16:0/3791102177 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0c80cf260 con 0x7fe0c806a830 2023-05-22T15:27:00.226 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.214+0000 7fe0bdffb640 1 -- 172.21.15.16:0/3791102177 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7fe0c4014b20 con 0x7fe0c806a830 2023-05-22T15:27:00.227 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.214+0000 7fe0bdffb640 1 --2- 172.21.15.16:0/3791102177 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fe0ac0653b0 0x7fe0ac067860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:00.227 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.214+0000 7fe0bdffb640 1 -- 172.21.15.16:0/3791102177 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6813+0+0 (secure 0 0 0) 0x7fe0c4083460 con 0x7fe0c806a830 2023-05-22T15:27:00.227 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.214+0000 7fe0cced6640 1 --2- 172.21.15.16:0/3791102177 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fe0ac0653b0 0x7fe0ac067860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:00.227 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.214+0000 7fe0cced6640 1 --2- 172.21.15.16:0/3791102177 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fe0ac0653b0 0x7fe0ac067860 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fe0c80af970 tx=0x7fe0b8009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:00.228 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.218+0000 7fe0bdffb640 1 -- 172.21.15.16:0/3791102177 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fe0c4054890 con 0x7fe0c806a830 2023-05-22T15:27:00.381 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.370+0000 7fe0cf93b640 1 -- 172.21.15.16:0/3791102177 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fe0c80cb2f0 con 0x7fe0c806a830 2023-05-22T15:27:00.382 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.370+0000 7fe0bdffb640 1 -- 172.21.15.16:0/3791102177 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) v1 ==== 104+0+11 (secure 0 0 0) 0x7fe0c4054480 con 0x7fe0c806a830 2023-05-22T15:27:00.382 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2023-05-22T15:27:00.384 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.374+0000 7fe0cf93b640 1 -- 172.21.15.16:0/3791102177 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fe0ac0653b0 msgr2=0x7fe0ac067860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:00.384 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.374+0000 7fe0cf93b640 1 --2- 172.21.15.16:0/3791102177 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fe0ac0653b0 0x7fe0ac067860 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fe0c80af970 tx=0x7fe0b8009040 comp rx=0 tx=0).stop 2023-05-22T15:27:00.385 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.374+0000 7fe0cf93b640 1 -- 172.21.15.16:0/3791102177 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe0c806a830 msgr2=0x7fe0c81557c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:00.385 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.374+0000 7fe0cf93b640 1 --2- 172.21.15.16:0/3791102177 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe0c806a830 0x7fe0c81557c0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fe0c4007e40 tx=0x7fe0c400e4b0 comp rx=0 tx=0).stop 2023-05-22T15:27:00.385 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.374+0000 7fe0cf93b640 1 -- 172.21.15.16:0/3791102177 shutdown_connections 2023-05-22T15:27:00.385 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.374+0000 7fe0cf93b640 1 --2- 172.21.15.16:0/3791102177 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe0c806db10 0x7fe0c815a890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:00.386 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.374+0000 7fe0cf93b640 1 --2- 172.21.15.16:0/3791102177 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fe0ac0653b0 0x7fe0ac067860 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:00.386 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.374+0000 7fe0cf93b640 1 --2- 172.21.15.16:0/3791102177 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe0c806c5f0 0x7fe0c80c9a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:00.386 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.374+0000 7fe0cf93b640 1 --2- 172.21.15.16:0/3791102177 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe0c806a830 0x7fe0c81557c0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:00.386 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.374+0000 7fe0cf93b640 1 -- 172.21.15.16:0/3791102177 >> 172.21.15.16:0/3791102177 conn(0x7fe0c806f4f0 msgr2=0x7fe0c80b2ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:00.386 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.374+0000 7fe0cf93b640 1 -- 172.21.15.16:0/3791102177 shutdown_connections 2023-05-22T15:27:00.387 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.374+0000 7fe0cf93b640 1 -- 172.21.15.16:0/3791102177 wait complete. 2023-05-22T15:27:00.400 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-05-22T15:27:00.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.522+0000 7fed71dcd640 1 Processor -- start 2023-05-22T15:27:00.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.522+0000 7fed71dcd640 1 -- start start 2023-05-22T15:27:00.534 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.526+0000 7fed71dcd640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fed6c15cfa0 0x7fed6c06db00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:00.535 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.526+0000 7fed71dcd640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fed6c0cf870 0x7fed6c0b1fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:00.535 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.526+0000 7fed71dcd640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed6c06e040 con 0x7fed6c06e740 2023-05-22T15:27:00.535 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.526+0000 7fed71dcd640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed6c0cb9f0 con 0x7fed6c15cfa0 2023-05-22T15:27:00.535 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.526+0000 7fed71dcd640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed6c0b24f0 con 0x7fed6c0cf870 2023-05-22T15:27:00.536 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.526+0000 7fed6b7fe640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fed6c15cfa0 0x7fed6c06db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:00.536 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.526+0000 7fed6b7fe640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fed6c15cfa0 0x7fed6c06db00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57378/0 (socket says 172.21.15.16:57378) 2023-05-22T15:27:00.536 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.526+0000 7fed6b7fe640 1 -- 172.21.15.16:0/1870835423 learned_addr learned my addr 172.21.15.16:0/1870835423 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:00.536 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.526+0000 7fed6affd640 1 --2- 172.21.15.16:0/1870835423 >> v2:172.21.15.62:3300/0 conn(0x7fed6c0cf870 0x7fed6c0b1fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:00.536 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.526+0000 7fed6b7fe640 1 -- 172.21.15.16:0/1870835423 >> v2:172.21.15.62:3300/0 conn(0x7fed6c0cf870 msgr2=0x7fed6c0b1fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:00.537 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.526+0000 7fed6b7fe640 1 --2- 172.21.15.16:0/1870835423 >> v2:172.21.15.62:3300/0 conn(0x7fed6c0cf870 0x7fed6c0b1fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:00.537 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.526+0000 7fed6b7fe640 1 -- 172.21.15.16:0/1870835423 >> v1:172.21.15.62:6789/0 conn(0x7fed6c06e740 legacy=0x7fed6c070de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:00.537 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.526+0000 7fed6b7fe640 1 -- 172.21.15.16:0/1870835423 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed6c06e490 con 0x7fed6c15cfa0 2023-05-22T15:27:00.541 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.526+0000 7fed6b7fe640 1 --2- 172.21.15.16:0/1870835423 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed6c15cfa0 0x7fed6c06db00 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fed6c0cef60 tx=0x7fed5c01c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c219069de9a1d99c server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:00.541 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.526+0000 7fed6a7fc640 1 -- 172.21.15.16:0/1870835423 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed5c0aa430 con 0x7fed6c15cfa0 2023-05-22T15:27:00.541 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.526+0000 7fed6a7fc640 1 -- 172.21.15.16:0/1870835423 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed5c0aa590 con 0x7fed6c15cfa0 2023-05-22T15:27:00.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.526+0000 7fed6a7fc640 1 -- 172.21.15.16:0/1870835423 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed5c0aa920 con 0x7fed6c15cfa0 2023-05-22T15:27:00.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.526+0000 7fed71dcd640 1 -- 172.21.15.16:0/1870835423 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed6c15cfa0 msgr2=0x7fed6c06db00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:00.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.526+0000 7fed71dcd640 1 --2- 172.21.15.16:0/1870835423 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed6c15cfa0 0x7fed6c06db00 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fed6c0cef60 tx=0x7fed5c01c4d0 comp rx=0 tx=0).stop 2023-05-22T15:27:00.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.530+0000 7fed71dcd640 1 -- 172.21.15.16:0/1870835423 shutdown_connections 2023-05-22T15:27:00.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.530+0000 7fed71dcd640 1 --2- 172.21.15.16:0/1870835423 >> v2:172.21.15.62:3300/0 conn(0x7fed6c0cf870 0x7fed6c0b1fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:00.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.530+0000 7fed71dcd640 1 --2- 172.21.15.16:0/1870835423 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed6c15cfa0 0x7fed6c06db00 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:00.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.530+0000 7fed71dcd640 1 -- 172.21.15.16:0/1870835423 >> 172.21.15.16:0/1870835423 conn(0x7fed6c0cc870 msgr2=0x7fed6c0cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:00.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.530+0000 7fed71dcd640 1 -- 172.21.15.16:0/1870835423 shutdown_connections 2023-05-22T15:27:00.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.530+0000 7fed71dcd640 1 -- 172.21.15.16:0/1870835423 wait complete. 2023-05-22T15:27:00.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.530+0000 7fed71dcd640 1 Processor -- start 2023-05-22T15:27:00.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.530+0000 7fed71dcd640 1 -- start start 2023-05-22T15:27:00.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.530+0000 7fed71dcd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fed6c06e740 0x7fed6c0c5be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:00.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.530+0000 7fed71dcd640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fed6c0cf870 0x7fed6c149b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:00.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.530+0000 7fed71dcd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed6c15cfa0 0x7fed6c0b0c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:00.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.530+0000 7fed71dcd640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed6c14ab20 con 0x7fed6c15cfa0 2023-05-22T15:27:00.544 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.530+0000 7fed71dcd640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed6c14a070 con 0x7fed6c06e740 2023-05-22T15:27:00.544 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.530+0000 7fed71dcd640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed6c14a280 con 0x7fed6c0cf870 2023-05-22T15:27:00.544 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.530+0000 7fed6bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed6c15cfa0 0x7fed6c0b0c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:00.544 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.530+0000 7fed6bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed6c15cfa0 0x7fed6c0b0c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57386/0 (socket says 172.21.15.16:57386) 2023-05-22T15:27:00.544 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.530+0000 7fed6bfff640 1 -- 172.21.15.16:0/2176996634 learned_addr learned my addr 172.21.15.16:0/2176996634 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:00.544 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.530+0000 7fed6b7fe640 1 --2- 172.21.15.16:0/2176996634 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fed6c06e740 0x7fed6c0c5be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:00.544 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.530+0000 7fed6affd640 1 --2- 172.21.15.16:0/2176996634 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fed6c0cf870 0x7fed6c149b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:00.544 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.530+0000 7fed6bfff640 1 -- 172.21.15.16:0/2176996634 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fed6c0cf870 msgr2=0x7fed6c149b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:00.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.530+0000 7fed6bfff640 1 --2- 172.21.15.16:0/2176996634 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fed6c0cf870 0x7fed6c149b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:00.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.530+0000 7fed6bfff640 1 -- 172.21.15.16:0/2176996634 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fed6c06e740 msgr2=0x7fed6c0c5be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:00.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.530+0000 7fed6bfff640 1 --2- 172.21.15.16:0/2176996634 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fed6c06e740 0x7fed6c0c5be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:00.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.530+0000 7fed6bfff640 1 -- 172.21.15.16:0/2176996634 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed5c0094d0 con 0x7fed6c15cfa0 2023-05-22T15:27:00.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.530+0000 7fed6bfff640 1 --2- 172.21.15.16:0/2176996634 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed6c15cfa0 0x7fed6c0b0c30 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fed6400c770 tx=0x7fed6400cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:00.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.530+0000 7fed53fff640 1 -- 172.21.15.16:0/2176996634 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed64003b80 con 0x7fed6c15cfa0 2023-05-22T15:27:00.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.530+0000 7fed53fff640 1 -- 172.21.15.16:0/2176996634 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed64003ce0 con 0x7fed6c15cfa0 2023-05-22T15:27:00.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.530+0000 7fed71dcd640 1 -- 172.21.15.16:0/2176996634 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed6c0cb610 con 0x7fed6c15cfa0 2023-05-22T15:27:00.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.530+0000 7fed53fff640 1 -- 172.21.15.16:0/2176996634 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed6400aa90 con 0x7fed6c15cfa0 2023-05-22T15:27:00.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.530+0000 7fed71dcd640 1 -- 172.21.15.16:0/2176996634 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed6c0cb330 con 0x7fed6c15cfa0 2023-05-22T15:27:00.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.534+0000 7fed71dcd640 1 -- 172.21.15.16:0/2176996634 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed6c071900 con 0x7fed6c15cfa0 2023-05-22T15:27:00.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.534+0000 7fed53fff640 1 -- 172.21.15.16:0/2176996634 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7fed6400abf0 con 0x7fed6c15cfa0 2023-05-22T15:27:00.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.534+0000 7fed53fff640 1 --2- 172.21.15.16:0/2176996634 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fed58065380 0x7fed58067830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:00.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.534+0000 7fed53fff640 1 -- 172.21.15.16:0/2176996634 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6813+0+0 (secure 0 0 0) 0x7fed64082760 con 0x7fed6c15cfa0 2023-05-22T15:27:00.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.534+0000 7fed6b7fe640 1 --2- 172.21.15.16:0/2176996634 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fed58065380 0x7fed58067830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:00.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.534+0000 7fed6b7fe640 1 --2- 172.21.15.16:0/2176996634 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fed58065380 0x7fed58067830 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fed5c0ae000 tx=0x7fed5c0b5040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:00.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.538+0000 7fed53fff640 1 -- 172.21.15.16:0/2176996634 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fed640500f0 con 0x7fed6c15cfa0 2023-05-22T15:27:00.703 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.694+0000 7fed71dcd640 1 -- 172.21.15.16:0/2176996634 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fed6c06e040 con 0x7fed6c15cfa0 2023-05-22T15:27:00.704 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.694+0000 7fed53fff640 1 -- 172.21.15.16:0/2176996634 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) v1 ==== 100+0+11 (secure 0 0 0) 0x7fed6c06e040 con 0x7fed6c15cfa0 2023-05-22T15:27:00.704 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 64 2023-05-22T15:27:00.707 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.698+0000 7fed71dcd640 1 -- 172.21.15.16:0/2176996634 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fed58065380 msgr2=0x7fed58067830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:00.708 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.698+0000 7fed71dcd640 1 --2- 172.21.15.16:0/2176996634 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fed58065380 0x7fed58067830 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fed5c0ae000 tx=0x7fed5c0b5040 comp rx=0 tx=0).stop 2023-05-22T15:27:00.708 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.698+0000 7fed71dcd640 1 -- 172.21.15.16:0/2176996634 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed6c15cfa0 msgr2=0x7fed6c0b0c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:00.708 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.698+0000 7fed71dcd640 1 --2- 172.21.15.16:0/2176996634 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed6c15cfa0 0x7fed6c0b0c30 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fed6400c770 tx=0x7fed6400cc30 comp rx=0 tx=0).stop 2023-05-22T15:27:00.709 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.698+0000 7fed71dcd640 1 -- 172.21.15.16:0/2176996634 shutdown_connections 2023-05-22T15:27:00.709 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.698+0000 7fed71dcd640 1 --2- 172.21.15.16:0/2176996634 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed6c15cfa0 0x7fed6c0b0c30 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:00.709 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.698+0000 7fed71dcd640 1 --2- 172.21.15.16:0/2176996634 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fed58065380 0x7fed58067830 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:00.710 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.698+0000 7fed71dcd640 1 --2- 172.21.15.16:0/2176996634 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fed6c0cf870 0x7fed6c149b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:00.710 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.698+0000 7fed71dcd640 1 --2- 172.21.15.16:0/2176996634 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fed6c06e740 0x7fed6c0c5be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:00.710 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.698+0000 7fed71dcd640 1 -- 172.21.15.16:0/2176996634 >> 172.21.15.16:0/2176996634 conn(0x7fed6c0cc870 msgr2=0x7fed6c154300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:00.710 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.698+0000 7fed71dcd640 1 -- 172.21.15.16:0/2176996634 shutdown_connections 2023-05-22T15:27:00.711 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.698+0000 7fed71dcd640 1 -- 172.21.15.16:0/2176996634 wait complete. 2023-05-22T15:27:00.725 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-05-22T15:27:00.843 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.834+0000 7f669efba640 1 Processor -- start 2023-05-22T15:27:00.844 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.834+0000 7f669efba640 1 -- start start 2023-05-22T15:27:00.845 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.834+0000 7f669efba640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f66980d2480 0x7f66980d2900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:00.845 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.834+0000 7f669efba640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66980cccc0 con 0x7f66980b5340 2023-05-22T15:27:00.846 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.834+0000 7f669efba640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66980ccfa0 con 0x7f669815bca0 2023-05-22T15:27:00.846 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.834+0000 7f669efba640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66980cd660 con 0x7f66980d2480 2023-05-22T15:27:00.846 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.834+0000 7f6697fff640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f66980d2480 0x7f66980d2900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:00.847 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.834+0000 7f669d557640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f66980b5340 0x7f66980b5790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:50968/0 (socket says 172.21.15.16:50968) 2023-05-22T15:27:00.847 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.834+0000 7f6697fff640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f66980d2480 0x7f66980d2900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57400/0 (socket says 172.21.15.16:57400) 2023-05-22T15:27:00.847 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.834+0000 7f669d557640 1 -- 172.21.15.16:0/3883784082 learned_addr learned my addr 172.21.15.16:0/3883784082 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:27:00.847 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.834+0000 7f6697fff640 1 -- 172.21.15.16:0/3883784082 >> v1:172.21.15.62:6789/0 conn(0x7f669815bca0 legacy=0x7f66980cda80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:00.848 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.834+0000 7f6697fff640 1 -- 172.21.15.16:0/3883784082 >> v1:172.21.15.16:6789/0 conn(0x7f66980b5340 legacy=0x7f66980b5790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:00.848 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.834+0000 7f6697fff640 1 -- 172.21.15.16:0/3883784082 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66980cd280 con 0x7f66980d2480 2023-05-22T15:27:00.849 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.838+0000 7f6697fff640 1 --2- 172.21.15.16:0/3883784082 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66980d2480 0x7f66980d2900 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f668c0097c0 tx=0x7f668c01c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8573a885c2ad5b85 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:00.850 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.838+0000 7f66977fe640 1 -- 172.21.15.16:0/3883784082 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f668c0a8050 con 0x7f66980d2480 2023-05-22T15:27:00.850 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.838+0000 7f66977fe640 1 -- 172.21.15.16:0/3883784082 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f668c0ac5f0 con 0x7f66980d2480 2023-05-22T15:27:00.850 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.838+0000 7f66977fe640 1 -- 172.21.15.16:0/3883784082 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f668c0ac980 con 0x7f66980d2480 2023-05-22T15:27:00.850 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.838+0000 7f669efba640 1 -- 172.21.15.16:0/3883784082 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66980d2480 msgr2=0x7f66980d2900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:00.850 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.838+0000 7f669efba640 1 --2- 172.21.15.16:0/3883784082 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66980d2480 0x7f66980d2900 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f668c0097c0 tx=0x7f668c01c480 comp rx=0 tx=0).stop 2023-05-22T15:27:00.851 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.838+0000 7f669efba640 1 -- 172.21.15.16:0/3883784082 shutdown_connections 2023-05-22T15:27:00.851 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.838+0000 7f669efba640 1 --2- 172.21.15.16:0/3883784082 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66980d2480 0x7f66980d2900 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:00.851 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.838+0000 7f669efba640 1 -- 172.21.15.16:0/3883784082 >> 172.21.15.16:0/3883784082 conn(0x7f66980655c0 msgr2=0x7f6698070e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:00.851 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.838+0000 7f669efba640 1 -- 172.21.15.16:0/3883784082 shutdown_connections 2023-05-22T15:27:00.851 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.838+0000 7f669efba640 1 -- 172.21.15.16:0/3883784082 wait complete. 2023-05-22T15:27:00.851 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.838+0000 7f669efba640 1 Processor -- start 2023-05-22T15:27:00.851 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.838+0000 7f669efba640 1 -- start start 2023-05-22T15:27:00.854 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.838+0000 7f669efba640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f66980b5340 0x7f6698156130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:00.854 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.838+0000 7f669efba640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f66980d2480 0x7f669815e700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:00.854 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.838+0000 7f669efba640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f669815bca0 0x7f6698158350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:00.854 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.838+0000 7f669efba640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66980ce190 con 0x7f669815bca0 2023-05-22T15:27:00.854 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.838+0000 7f669efba640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f669815b520 con 0x7f66980b5340 2023-05-22T15:27:00.854 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.838+0000 7f669efba640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f669815c5d0 con 0x7f66980d2480 2023-05-22T15:27:00.855 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.842+0000 7f669d557640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f669815bca0 0x7f6698158350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:00.855 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.842+0000 7f669d557640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f669815bca0 0x7f6698158350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57412/0 (socket says 172.21.15.16:57412) 2023-05-22T15:27:00.855 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.842+0000 7f669d557640 1 -- 172.21.15.16:0/700754186 learned_addr learned my addr 172.21.15.16:0/700754186 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:00.855 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.842+0000 7f669cd56640 1 --2- 172.21.15.16:0/700754186 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f66980b5340 0x7f6698156130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:00.855 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.842+0000 7f6697fff640 1 --2- 172.21.15.16:0/700754186 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f66980d2480 0x7f669815e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:00.855 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.842+0000 7f669d557640 1 -- 172.21.15.16:0/700754186 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f66980d2480 msgr2=0x7f669815e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:00.856 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.842+0000 7f669d557640 1 --2- 172.21.15.16:0/700754186 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f66980d2480 0x7f669815e700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:00.856 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.842+0000 7f669d557640 1 -- 172.21.15.16:0/700754186 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f66980b5340 msgr2=0x7f6698156130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:00.856 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.842+0000 7f669d557640 1 --2- 172.21.15.16:0/700754186 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f66980b5340 0x7f6698156130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:00.856 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.842+0000 7f669d557640 1 -- 172.21.15.16:0/700754186 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f668c009420 con 0x7f669815bca0 2023-05-22T15:27:00.856 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.842+0000 7f669cd56640 1 --2- 172.21.15.16:0/700754186 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f66980b5340 0x7f6698156130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T15:27:00.856 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.842+0000 7f669d557640 1 --2- 172.21.15.16:0/700754186 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f669815bca0 0x7f6698158350 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f669000cbf0 tx=0x7f669000a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:00.856 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.842+0000 7f66957fa640 1 -- 172.21.15.16:0/700754186 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f669000ad40 con 0x7f669815bca0 2023-05-22T15:27:00.857 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.842+0000 7f66957fa640 1 -- 172.21.15.16:0/700754186 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6690012400 con 0x7f669815bca0 2023-05-22T15:27:00.857 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.842+0000 7f669efba640 1 -- 172.21.15.16:0/700754186 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66981604f0 con 0x7f669815bca0 2023-05-22T15:27:00.857 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.842+0000 7f66957fa640 1 -- 172.21.15.16:0/700754186 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f669000aea0 con 0x7f669815bca0 2023-05-22T15:27:00.857 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.842+0000 7f669efba640 1 -- 172.21.15.16:0/700754186 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6698160360 con 0x7f669815bca0 2023-05-22T15:27:00.857 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.842+0000 7f669efba640 1 -- 172.21.15.16:0/700754186 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6698169c60 con 0x7f669815bca0 2023-05-22T15:27:00.857 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.842+0000 7f66957fa640 1 -- 172.21.15.16:0/700754186 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7f6690007070 con 0x7f669815bca0 2023-05-22T15:27:00.857 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.842+0000 7f66957fa640 1 --2- 172.21.15.16:0/700754186 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f6684065380 0x7f6684067830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:00.858 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.842+0000 7f66957fa640 1 -- 172.21.15.16:0/700754186 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f6690082a10 con 0x7f669815bca0 2023-05-22T15:27:00.858 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.842+0000 7f669cd56640 1 --2- 172.21.15.16:0/700754186 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f6684065380 0x7f6684067830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:00.858 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.842+0000 7f669cd56640 1 --2- 172.21.15.16:0/700754186 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f6684065380 0x7f6684067830 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f6688001670 tx=0x7f6688009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:00.859 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.846+0000 7f66957fa640 1 -- 172.21.15.16:0/700754186 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f669004a620 con 0x7f669815bca0 2023-05-22T15:27:01.005 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.994+0000 7f669efba640 1 -- 172.21.15.16:0/700754186 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f669804dbe0 con 0x7f669815bca0 2023-05-22T15:27:01.006 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.998+0000 7f66957fa640 1 -- 172.21.15.16:0/700754186 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) v1 ==== 103+0+11 (secure 0 0 0) 0x7f6690017020 con 0x7f669815bca0 2023-05-22T15:27:01.006 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2023-05-22T15:27:01.009 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.998+0000 7f669efba640 1 -- 172.21.15.16:0/700754186 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f6684065380 msgr2=0x7f6684067830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:01.009 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.998+0000 7f669efba640 1 --2- 172.21.15.16:0/700754186 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f6684065380 0x7f6684067830 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f6688001670 tx=0x7f6688009040 comp rx=0 tx=0).stop 2023-05-22T15:27:01.010 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.998+0000 7f669efba640 1 -- 172.21.15.16:0/700754186 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f669815bca0 msgr2=0x7f6698158350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:01.010 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.998+0000 7f669efba640 1 --2- 172.21.15.16:0/700754186 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f669815bca0 0x7f6698158350 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f669000cbf0 tx=0x7f669000a430 comp rx=0 tx=0).stop 2023-05-22T15:27:01.010 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.998+0000 7f669efba640 1 -- 172.21.15.16:0/700754186 shutdown_connections 2023-05-22T15:27:01.010 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.998+0000 7f669efba640 1 --2- 172.21.15.16:0/700754186 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f669815bca0 0x7f6698158350 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:01.011 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.998+0000 7f669efba640 1 --2- 172.21.15.16:0/700754186 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f6684065380 0x7f6684067830 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:01.011 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.998+0000 7f669efba640 1 --2- 172.21.15.16:0/700754186 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f66980d2480 0x7f669815e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:01.011 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.998+0000 7f669efba640 1 --2- 172.21.15.16:0/700754186 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f66980b5340 0x7f6698156130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:01.011 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.998+0000 7f669efba640 1 -- 172.21.15.16:0/700754186 >> 172.21.15.16:0/700754186 conn(0x7f66980655c0 msgr2=0x7f66980b34a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:01.011 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.998+0000 7f669efba640 1 -- 172.21.15.16:0/700754186 shutdown_connections 2023-05-22T15:27:01.011 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:00.998+0000 7f669efba640 1 -- 172.21.15.16:0/700754186 wait complete. 2023-05-22T15:27:01.025 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-05-22T15:27:01.136 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.126+0000 7f549b079640 1 Processor -- start 2023-05-22T15:27:01.136 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.126+0000 7f549b079640 1 -- start start 2023-05-22T15:27:01.137 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.126+0000 7f549b079640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f54941567c0 0x7f54940afb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:01.137 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.126+0000 7f549b079640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5494156b90 con 0x7f54940cb6a0 2023-05-22T15:27:01.137 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.126+0000 7f549b079640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f549415a0b0 con 0x7f5494159a50 2023-05-22T15:27:01.138 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.126+0000 7f549b079640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5494159ec0 con 0x7f54941567c0 2023-05-22T15:27:01.138 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.126+0000 7f5493fff640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f54940cb6a0 0x7f5494159340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:50970/0 (socket says 172.21.15.16:50970) 2023-05-22T15:27:01.138 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.126+0000 7f5493fff640 1 -- 172.21.15.16:0/2473731304 learned_addr learned my addr 172.21.15.16:0/2473731304 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:27:01.139 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.126+0000 7f54937fe640 1 -- 172.21.15.16:0/2473731304 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1925198763 0 0) 0x7f547c001120 con 0x7f54940cb6a0 2023-05-22T15:27:01.140 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.126+0000 7f54937fe640 1 -- 172.21.15.16:0/2473731304 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5480003530 con 0x7f54940cb6a0 2023-05-22T15:27:01.140 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.126+0000 7f5498e15640 1 --2- 172.21.15.16:0/2473731304 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f54941567c0 0x7f54940afb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:01.140 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.130+0000 7f54937fe640 1 -- 172.21.15.16:0/2473731304 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 937387704 0 0) 0x7f5480003530 con 0x7f54940cb6a0 2023-05-22T15:27:01.140 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.130+0000 7f54937fe640 1 -- 172.21.15.16:0/2473731304 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f54941567c0 msgr2=0x7f54940afb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:01.140 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.130+0000 7f54937fe640 1 --2- 172.21.15.16:0/2473731304 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f54941567c0 0x7f54940afb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:01.140 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.130+0000 7f54937fe640 1 -- 172.21.15.16:0/2473731304 >> v1:172.21.15.62:6789/0 conn(0x7f5494159a50 legacy=0x7f54940c8670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:01.140 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.130+0000 7f54937fe640 1 -- 172.21.15.16:0/2473731304 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54940ca8e0 con 0x7f54940cb6a0 2023-05-22T15:27:01.141 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.130+0000 7f54937fe640 1 -- 172.21.15.16:0/2473731304 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7f547c001810 con 0x7f54940cb6a0 2023-05-22T15:27:01.141 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.130+0000 7f54937fe640 1 -- 172.21.15.16:0/2473731304 >> v1:172.21.15.16:6789/0 conn(0x7f54940cb6a0 legacy=0x7f5494159340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:01.141 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.130+0000 7f54937fe640 1 --2- 172.21.15.16:0/2473731304 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f54941567c0 0x7f54800039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:01.141 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.130+0000 7f54937fe640 1 --2- 172.21.15.16:0/2473731304 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5480003f30 0x7f54800047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:01.141 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.130+0000 7f54937fe640 1 --2- 172.21.15.16:0/2473731304 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5480004d10 0x7f5480005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:01.141 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.130+0000 7f54937fe640 1 -- 172.21.15.16:0/2473731304 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f548000ba40 con 0x7f5480003f30 2023-05-22T15:27:01.141 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.130+0000 7f54937fe640 1 -- 172.21.15.16:0/2473731304 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f548000bbb0 con 0x7f5480004d10 2023-05-22T15:27:01.142 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.130+0000 7f54937fe640 1 -- 172.21.15.16:0/2473731304 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f548000bd70 con 0x7f54941567c0 2023-05-22T15:27:01.142 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.130+0000 7f5493fff640 1 --2- 172.21.15.16:0/2473731304 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5480003f30 0x7f54800047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:01.142 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.130+0000 7f5499616640 1 --2- 172.21.15.16:0/2473731304 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5480004d10 0x7f5480005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:01.142 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.130+0000 7f5498e15640 1 --2- 172.21.15.16:0/2473731304 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f54941567c0 0x7f54800039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:01.142 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.130+0000 7f5493fff640 1 -- 172.21.15.16:0/2473731304 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f54941567c0 msgr2=0x7f54800039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:01.142 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.130+0000 7f5493fff640 1 --2- 172.21.15.16:0/2473731304 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f54941567c0 0x7f54800039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:01.142 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.130+0000 7f5493fff640 1 -- 172.21.15.16:0/2473731304 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5480004d10 msgr2=0x7f5480005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:01.142 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.130+0000 7f5493fff640 1 --2- 172.21.15.16:0/2473731304 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5480004d10 0x7f5480005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:01.143 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.130+0000 7f5493fff640 1 -- 172.21.15.16:0/2473731304 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5480003530 con 0x7f5480003f30 2023-05-22T15:27:01.143 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.130+0000 7f5499616640 1 --2- 172.21.15.16:0/2473731304 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5480004d10 0x7f5480005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:27:01.143 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.130+0000 7f5498e15640 1 --2- 172.21.15.16:0/2473731304 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f54941567c0 0x7f54800039b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:27:01.145 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.130+0000 7f5493fff640 1 --2- 172.21.15.16:0/2473731304 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5480003f30 0x7f54800047d0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f547c00c210 tx=0x7f547c01f370 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17b8dadbf36e3194 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:01.145 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.130+0000 7f54937fe640 1 -- 172.21.15.16:0/2473731304 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f547c005ae0 con 0x7f5480003f30 2023-05-22T15:27:01.145 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.130+0000 7f54937fe640 1 -- 172.21.15.16:0/2473731304 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f547c0ab070 con 0x7f5480003f30 2023-05-22T15:27:01.145 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.130+0000 7f54937fe640 1 -- 172.21.15.16:0/2473731304 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f547c007070 con 0x7f5480003f30 2023-05-22T15:27:01.146 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.130+0000 7f549b079640 1 -- 172.21.15.16:0/2473731304 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5480003f30 msgr2=0x7f54800047d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:01.146 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.130+0000 7f549b079640 1 --2- 172.21.15.16:0/2473731304 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5480003f30 0x7f54800047d0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f547c00c210 tx=0x7f547c01f370 comp rx=0 tx=0).stop 2023-05-22T15:27:01.146 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.130+0000 7f5499616640 1 -- 172.21.15.16:0/2473731304 reap_dead start 2023-05-22T15:27:01.146 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.130+0000 7f549b079640 1 -- 172.21.15.16:0/2473731304 shutdown_connections 2023-05-22T15:27:01.146 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.130+0000 7f549b079640 1 -- 172.21.15.16:0/2473731304 >> 172.21.15.16:0/2473731304 conn(0x7f5494069720 msgr2=0x7f5494069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:01.146 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.130+0000 7f549b079640 1 -- 172.21.15.16:0/2473731304 shutdown_connections 2023-05-22T15:27:01.146 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.130+0000 7f549b079640 1 -- 172.21.15.16:0/2473731304 wait complete. 2023-05-22T15:27:01.147 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.134+0000 7f549b079640 1 Processor -- start 2023-05-22T15:27:01.147 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.134+0000 7f549b079640 1 -- start start 2023-05-22T15:27:01.147 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.134+0000 7f549b079640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5480003f30 0x7f549414ba00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:01.147 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.134+0000 7f549b079640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5494147600 0x7f5494146ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:01.147 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.134+0000 7f549b079640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f549414c410 0x7f549414c880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:01.147 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.134+0000 7f549b079640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f549414b890 con 0x7f549414c410 2023-05-22T15:27:01.147 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.134+0000 7f549b079640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f549414cdc0 con 0x7f5494147600 2023-05-22T15:27:01.147 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.134+0000 7f549b079640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54940b3d40 con 0x7f5480003f30 2023-05-22T15:27:01.148 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.134+0000 7f5499616640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f549414c410 0x7f549414c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:01.148 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.134+0000 7f5499616640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f549414c410 0x7f549414c880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57442/0 (socket says 172.21.15.16:57442) 2023-05-22T15:27:01.148 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.134+0000 7f5499616640 1 -- 172.21.15.16:0/3540727778 learned_addr learned my addr 172.21.15.16:0/3540727778 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:01.148 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.134+0000 7f5493fff640 1 --2- 172.21.15.16:0/3540727778 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5494147600 0x7f5494146ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:01.148 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.134+0000 7f5499616640 1 -- 172.21.15.16:0/3540727778 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5480003f30 msgr2=0x7f549414ba00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:01.148 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.134+0000 7f5499616640 1 --2- 172.21.15.16:0/3540727778 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5480003f30 0x7f549414ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:01.148 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.134+0000 7f5499616640 1 -- 172.21.15.16:0/3540727778 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5494147600 msgr2=0x7f5494146ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:01.149 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.134+0000 7f5499616640 1 --2- 172.21.15.16:0/3540727778 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5494147600 0x7f5494146ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:01.149 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.134+0000 7f5499616640 1 -- 172.21.15.16:0/3540727778 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54940caea0 con 0x7f549414c410 2023-05-22T15:27:01.149 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.134+0000 7f5499616640 1 --2- 172.21.15.16:0/3540727778 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f549414c410 0x7f549414c880 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f54940b0720 tx=0x7f548c00d4c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:01.149 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.134+0000 7f54917fa640 1 -- 172.21.15.16:0/3540727778 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f548c00b070 con 0x7f549414c410 2023-05-22T15:27:01.149 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.134+0000 7f54917fa640 1 -- 172.21.15.16:0/3540727778 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f548c0054a0 con 0x7f549414c410 2023-05-22T15:27:01.149 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.134+0000 7f549b079640 1 -- 172.21.15.16:0/3540727778 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54940cabc0 con 0x7f549414c410 2023-05-22T15:27:01.149 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.134+0000 7f54917fa640 1 -- 172.21.15.16:0/3540727778 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f548c007270 con 0x7f549414c410 2023-05-22T15:27:01.149 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.134+0000 7f549b079640 1 -- 172.21.15.16:0/3540727778 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54940cb180 con 0x7f549414c410 2023-05-22T15:27:01.150 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.134+0000 7f549b079640 1 -- 172.21.15.16:0/3540727778 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f549404dbe0 con 0x7f549414c410 2023-05-22T15:27:01.150 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.134+0000 7f54917fa640 1 -- 172.21.15.16:0/3540727778 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7f548c0073d0 con 0x7f549414c410 2023-05-22T15:27:01.150 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.134+0000 7f54917fa640 1 --2- 172.21.15.16:0/3540727778 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f5474065380 0x7f5474067830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:01.151 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.134+0000 7f54917fa640 1 -- 172.21.15.16:0/3540727778 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f548c081e20 con 0x7f549414c410 2023-05-22T15:27:01.151 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.134+0000 7f5498e15640 1 --2- 172.21.15.16:0/3540727778 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f5474065380 0x7f5474067830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:01.151 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.138+0000 7f5498e15640 1 --2- 172.21.15.16:0/3540727778 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f5474065380 0x7f5474067830 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f5484007460 tx=0x7f5484009750 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:01.151 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.142+0000 7f54917fa640 1 -- 172.21.15.16:0/3540727778 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f548c053210 con 0x7f549414c410 2023-05-22T15:27:01.316 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.306+0000 7f549b079640 1 -- 172.21.15.16:0/3540727778 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f549414b5b0 con 0x7f549414c410 2023-05-22T15:27:01.317 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.306+0000 7f54917fa640 1 -- 172.21.15.16:0/3540727778 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v27) v1 ==== 80+0+0 (secure 0 0 0) 0x7f548c052e00 con 0x7f549414c410 2023-05-22T15:27:01.317 INFO:teuthology.orchestra.run.smithi016.stderr:listed 0 entries 2023-05-22T15:27:01.319 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.310+0000 7f549b079640 1 -- 172.21.15.16:0/3540727778 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f5474065380 msgr2=0x7f5474067830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:01.320 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.310+0000 7f549b079640 1 --2- 172.21.15.16:0/3540727778 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f5474065380 0x7f5474067830 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f5484007460 tx=0x7f5484009750 comp rx=0 tx=0).stop 2023-05-22T15:27:01.320 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.310+0000 7f549b079640 1 -- 172.21.15.16:0/3540727778 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f549414c410 msgr2=0x7f549414c880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:01.321 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.310+0000 7f549b079640 1 --2- 172.21.15.16:0/3540727778 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f549414c410 0x7f549414c880 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f54940b0720 tx=0x7f548c00d4c0 comp rx=0 tx=0).stop 2023-05-22T15:27:01.321 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.310+0000 7f549b079640 1 -- 172.21.15.16:0/3540727778 shutdown_connections 2023-05-22T15:27:01.321 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.310+0000 7f549b079640 1 --2- 172.21.15.16:0/3540727778 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f549414c410 0x7f549414c880 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:01.321 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.310+0000 7f549b079640 1 --2- 172.21.15.16:0/3540727778 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5494147600 0x7f5494146ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:01.321 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.310+0000 7f549b079640 1 --2- 172.21.15.16:0/3540727778 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f5474065380 0x7f5474067830 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:01.321 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.310+0000 7f549b079640 1 --2- 172.21.15.16:0/3540727778 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5480003f30 0x7f549414ba00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:01.321 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.310+0000 7f549b079640 1 -- 172.21.15.16:0/3540727778 >> 172.21.15.16:0/3540727778 conn(0x7f5494069720 msgr2=0x7f549415c810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:01.322 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.310+0000 7f549b079640 1 -- 172.21.15.16:0/3540727778 shutdown_connections 2023-05-22T15:27:01.322 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.310+0000 7f549b079640 1 -- 172.21.15.16:0/3540727778 wait complete. 2023-05-22T15:27:01.336 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-05-22T15:27:01.336 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2023-05-22T15:27:01.342 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-05-22T15:27:01.510 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.502+0000 7fbe8aea7640 1 Processor -- start 2023-05-22T15:27:01.510 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.502+0000 7fbe8aea7640 1 -- start start 2023-05-22T15:27:01.511 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.502+0000 7fbe8aea7640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbe840ca4e0 0x7fbe8415e460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:01.511 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.502+0000 7fbe8aea7640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fbe84070df0 0x7fbe84071210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:01.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.502+0000 7fbe8aea7640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe840b0650 con 0x7fbe840c9b40 2023-05-22T15:27:01.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.502+0000 7fbe8aea7640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe840b0370 con 0x7fbe84070df0 2023-05-22T15:27:01.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.502+0000 7fbe8aea7640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe8415d980 con 0x7fbe840ca4e0 2023-05-22T15:27:01.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.502+0000 7fbe89444640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fbe84070df0 0x7fbe84071210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:01.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.502+0000 7fbe88c43640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbe840ca4e0 0x7fbe8415e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:01.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.502+0000 7fbe83fff640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7fbe840c9b40 0x7fbe840afb40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.16:35850/0 (socket says 172.21.15.16:35850) 2023-05-22T15:27:01.513 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.502+0000 7fbe83fff640 1 -- 172.21.15.16:0/2708049 learned_addr learned my addr 172.21.15.16:0/2708049 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:27:01.513 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.502+0000 7fbe837fe640 1 -- 172.21.15.16:0/2708049 <== mon.1 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3040742377 0 0) 0x7fbe840b0650 con 0x7fbe840c9b40 2023-05-22T15:27:01.513 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.502+0000 7fbe837fe640 1 -- 172.21.15.16:0/2708049 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe6c003530 con 0x7fbe840c9b40 2023-05-22T15:27:01.514 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.502+0000 7fbe88c43640 1 -- 172.21.15.16:0/2708049 >> v2:172.21.15.62:3300/0 conn(0x7fbe84070df0 msgr2=0x7fbe84071210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:01.514 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.502+0000 7fbe88c43640 1 --2- 172.21.15.16:0/2708049 >> v2:172.21.15.62:3300/0 conn(0x7fbe84070df0 0x7fbe84071210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:01.514 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.502+0000 7fbe88c43640 1 -- 172.21.15.16:0/2708049 >> v1:172.21.15.62:6789/0 conn(0x7fbe840c9b40 legacy=0x7fbe840afb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:01.514 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.502+0000 7fbe88c43640 1 -- 172.21.15.16:0/2708049 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe8415dc60 con 0x7fbe840ca4e0 2023-05-22T15:27:01.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.506+0000 7fbe88c43640 1 --2- 172.21.15.16:0/2708049 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbe840ca4e0 0x7fbe8415e460 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fbe740097c0 tx=0x7fbe7401c3c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c9ff76f71dbc2816 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:01.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.506+0000 7fbe837fe640 1 -- 172.21.15.16:0/2708049 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe740ab600 con 0x7fbe840ca4e0 2023-05-22T15:27:01.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.506+0000 7fbe837fe640 1 -- 172.21.15.16:0/2708049 <== mon.2 v2:172.21.15.62:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe740a7050 con 0x7fbe840ca4e0 2023-05-22T15:27:01.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.506+0000 7fbe837fe640 1 -- 172.21.15.16:0/2708049 <== mon.2 v2:172.21.15.62:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe740ab960 con 0x7fbe840ca4e0 2023-05-22T15:27:01.516 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.506+0000 7fbe8aea7640 1 -- 172.21.15.16:0/2708049 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbe840ca4e0 msgr2=0x7fbe8415e460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:01.516 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.506+0000 7fbe8aea7640 1 --2- 172.21.15.16:0/2708049 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbe840ca4e0 0x7fbe8415e460 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fbe740097c0 tx=0x7fbe7401c3c0 comp rx=0 tx=0).stop 2023-05-22T15:27:01.516 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.506+0000 7fbe8aea7640 1 -- 172.21.15.16:0/2708049 shutdown_connections 2023-05-22T15:27:01.516 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.506+0000 7fbe8aea7640 1 --2- 172.21.15.16:0/2708049 >> v2:172.21.15.62:3300/0 conn(0x7fbe84070df0 0x7fbe84071210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:01.516 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.506+0000 7fbe8aea7640 1 --2- 172.21.15.16:0/2708049 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbe840ca4e0 0x7fbe8415e460 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:01.516 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.506+0000 7fbe8aea7640 1 -- 172.21.15.16:0/2708049 >> 172.21.15.16:0/2708049 conn(0x7fbe84069720 msgr2=0x7fbe84069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:01.517 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.506+0000 7fbe8aea7640 1 -- 172.21.15.16:0/2708049 shutdown_connections 2023-05-22T15:27:01.517 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.506+0000 7fbe8aea7640 1 -- 172.21.15.16:0/2708049 wait complete. 2023-05-22T15:27:01.517 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.506+0000 7fbe8aea7640 1 Processor -- start 2023-05-22T15:27:01.518 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.506+0000 7fbe8aea7640 1 -- start start 2023-05-22T15:27:01.519 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.510+0000 7fbe8aea7640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbe84070df0 0x7fbe8414bf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:01.519 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.510+0000 7fbe8aea7640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbe840c9b40 0x7fbe8415a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:01.519 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.510+0000 7fbe8aea7640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbe840ca4e0 0x7fbe841592b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:01.519 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.510+0000 7fbe8aea7640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe8414c490 con 0x7fbe84070df0 2023-05-22T15:27:01.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.510+0000 7fbe8aea7640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe84156d80 con 0x7fbe840ca4e0 2023-05-22T15:27:01.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.510+0000 7fbe8aea7640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe8415a7f0 con 0x7fbe840c9b40 2023-05-22T15:27:01.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.510+0000 7fbe88c43640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbe84070df0 0x7fbe8414bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:01.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.510+0000 7fbe88c43640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbe84070df0 0x7fbe8414bf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57454/0 (socket says 172.21.15.16:57454) 2023-05-22T15:27:01.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.510+0000 7fbe88c43640 1 -- 172.21.15.16:0/2688090225 learned_addr learned my addr 172.21.15.16:0/2688090225 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:01.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.510+0000 7fbe83fff640 1 --2- 172.21.15.16:0/2688090225 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbe840c9b40 0x7fbe8415a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:01.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.510+0000 7fbe89444640 1 --2- 172.21.15.16:0/2688090225 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbe840ca4e0 0x7fbe841592b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:01.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.510+0000 7fbe88c43640 1 -- 172.21.15.16:0/2688090225 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbe840c9b40 msgr2=0x7fbe8415a2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:01.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.510+0000 7fbe88c43640 1 --2- 172.21.15.16:0/2688090225 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbe840c9b40 0x7fbe8415a2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:01.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.510+0000 7fbe88c43640 1 -- 172.21.15.16:0/2688090225 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbe840ca4e0 msgr2=0x7fbe841592b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:01.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.510+0000 7fbe88c43640 1 --2- 172.21.15.16:0/2688090225 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbe840ca4e0 0x7fbe841592b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:01.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.510+0000 7fbe88c43640 1 -- 172.21.15.16:0/2688090225 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe74009420 con 0x7fbe84070df0 2023-05-22T15:27:01.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.510+0000 7fbe88c43640 1 --2- 172.21.15.16:0/2688090225 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbe84070df0 0x7fbe8414bf50 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fbe74000c10 tx=0x7fbe74019160 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:01.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.510+0000 7fbe817fa640 1 -- 172.21.15.16:0/2688090225 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe74003a30 con 0x7fbe84070df0 2023-05-22T15:27:01.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.510+0000 7fbe8aea7640 1 -- 172.21.15.16:0/2688090225 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe840c8e80 con 0x7fbe84070df0 2023-05-22T15:27:01.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.510+0000 7fbe817fa640 1 -- 172.21.15.16:0/2688090225 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe74003b90 con 0x7fbe84070df0 2023-05-22T15:27:01.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.510+0000 7fbe817fa640 1 -- 172.21.15.16:0/2688090225 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe740ab400 con 0x7fbe84070df0 2023-05-22T15:27:01.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.510+0000 7fbe8aea7640 1 -- 172.21.15.16:0/2688090225 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe840ca8b0 con 0x7fbe84070df0 2023-05-22T15:27:01.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.510+0000 7fbe8aea7640 1 -- 172.21.15.16:0/2688090225 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe840c9010 con 0x7fbe84070df0 2023-05-22T15:27:01.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.510+0000 7fbe817fa640 1 -- 172.21.15.16:0/2688090225 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7fbe740b2040 con 0x7fbe84070df0 2023-05-22T15:27:01.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.510+0000 7fbe817fa640 1 --2- 172.21.15.16:0/2688090225 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fbe70065380 0x7fbe70067830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:01.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.510+0000 7fbe817fa640 1 -- 172.21.15.16:0/2688090225 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6813+0+0 (secure 0 0 0) 0x7fbe7411ca10 con 0x7fbe84070df0 2023-05-22T15:27:01.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.510+0000 7fbe83fff640 1 --2- 172.21.15.16:0/2688090225 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fbe70065380 0x7fbe70067830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:01.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.510+0000 7fbe83fff640 1 --2- 172.21.15.16:0/2688090225 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fbe70065380 0x7fbe70067830 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fbe840713e0 tx=0x7fbe78009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:01.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.514+0000 7fbe817fa640 1 -- 172.21.15.16:0/2688090225 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fbe740ea320 con 0x7fbe84070df0 2023-05-22T15:27:01.681 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.670+0000 7fbe8aea7640 1 -- 172.21.15.16:0/2688090225 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbe84072160 con 0x7fbe84070df0 2023-05-22T15:27:01.684 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.674+0000 7fbe817fa640 1 -- 172.21.15.16:0/2688090225 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) v1 ==== 74+0+19604 (secure 0 0 0) 0x7fbe740ed940 con 0x7fbe84070df0 2023-05-22T15:27:01.684 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:27:01.685 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":27,"fsid":"b2719834-7847-4d98-90a6-f65d57f85c49","created":"2023-05-22T15:26:25.259314+0000","modified":"2023-05-22T15:26:53.971378+0000","last_up_change":"2023-05-22T15:26:36.448524+0000","last_in_change":"2023-05-22T15:26:27.675668+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-05-22T15:26:38.019781+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-05-22T15:26:43.570514+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-05-22T15:26:47.879159+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-05-22T15:26:48.522130+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data_ec","create_time":"2023-05-22T15:26:51.616142+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"9e7ff854-b25f-4cbb-a5c2-1799d9c53d2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":37021},{"type":"v1","addr":"172.21.15.16:6825","nonce":37021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":37021},{"type":"v1","addr":"172.21.15.16:6827","nonce":37021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":37021},{"type":"v1","addr":"172.21.15.16:6831","nonce":37021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":37021},{"type":"v1","addr":"172.21.15.16:6829","nonce":37021}]},"public_addr":"172.21.15.16:6825/37021","cluster_addr":"172.21.15.16:6827/37021","heartbeat_back_addr":"172.21.15.16:6831/37021","heartbeat_front_addr":"172.21.15.16:6829/37021","state":["exists","up"]},{"osd":1,"uuid":"4702fbc9-a927-4565-aca5-41dc69cc8264","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":37008},{"type":"v1","addr":"172.21.15.16:6801","nonce":37008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":37008},{"type":"v1","addr":"172.21.15.16:6803","nonce":37008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":37008},{"type":"v1","addr":"172.21.15.16:6807","nonce":37008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":37008},{"type":"v1","addr":"172.21.15.16:6805","nonce":37008}]},"public_addr":"172.21.15.16:6801/37008","cluster_addr":"172.21.15.16:6803/37008","heartbeat_back_addr":"172.21.15.16:6807/37008","heartbeat_front_addr":"172.21.15.16:6805/37008","state":["exists","up"]},{"osd":2,"uuid":"1e7bba46-4eba-47ef-bb81-5e2d371638ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":37009},{"type":"v1","addr":"172.21.15.16:6809","nonce":37009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":37009},{"type":"v1","addr":"172.21.15.16:6811","nonce":37009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":37009},{"type":"v1","addr":"172.21.15.16:6815","nonce":37009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":37009},{"type":"v1","addr":"172.21.15.16:6813","nonce":37009}]},"public_addr":"172.21.15.16:6809/37009","cluster_addr":"172.21.15.16:6811/37009","heartbeat_back_addr":"172.21.15.16:6815/37009","heartbeat_front_addr":"172.21.15.16:6813/37009","state":["exists","up"]},{"osd":3,"uuid":"383cacc0-5a88-4904-990e-498ca59ef48a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":37020},{"type":"v1","addr":"172.21.15.16:6817","nonce":37020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":37020},{"type":"v1","addr":"172.21.15.16:6819","nonce":37020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":37020},{"type":"v1","addr":"172.21.15.16:6823","nonce":37020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":37020},{"type":"v1","addr":"172.21.15.16:6821","nonce":37020}]},"public_addr":"172.21.15.16:6817/37020","cluster_addr":"172.21.15.16:6819/37020","heartbeat_back_addr":"172.21.15.16:6823/37020","heartbeat_front_addr":"172.21.15.16:6821/37020","state":["exists","up"]},{"osd":4,"uuid":"970500ef-95b7-42c1-b080-eed4fae7013d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":37353},{"type":"v1","addr":"172.21.15.62:6817","nonce":37353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6819","nonce":37353},{"type":"v1","addr":"172.21.15.62:6822","nonce":37353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37353},{"type":"v1","addr":"172.21.15.62:6831","nonce":37353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37353},{"type":"v1","addr":"172.21.15.62:6826","nonce":37353}]},"public_addr":"172.21.15.62:6817/37353","cluster_addr":"172.21.15.62:6822/37353","heartbeat_back_addr":"172.21.15.62:6831/37353","heartbeat_front_addr":"172.21.15.62:6826/37353","state":["exists","up"]},{"osd":5,"uuid":"022d67db-d4d8-445b-b1b9-8b446a00f5c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":37352},{"type":"v1","addr":"172.21.15.62:6801","nonce":37352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":37352},{"type":"v1","addr":"172.21.15.62:6803","nonce":37352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":37352},{"type":"v1","addr":"172.21.15.62:6807","nonce":37352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":37352},{"type":"v1","addr":"172.21.15.62:6805","nonce":37352}]},"public_addr":"172.21.15.62:6801/37352","cluster_addr":"172.21.15.62:6803/37352","heartbeat_back_addr":"172.21.15.62:6807/37352","heartbeat_front_addr":"172.21.15.62:6805/37352","state":["exists","up"]},{"osd":6,"uuid":"42bd372b-6688-4a1b-8407-e16e8c266e33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":37354},{"type":"v1","addr":"172.21.15.62:6820","nonce":37354}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":37354},{"type":"v1","addr":"172.21.15.62:6823","nonce":37354}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6829","nonce":37354},{"type":"v1","addr":"172.21.15.62:6830","nonce":37354}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6825","nonce":37354},{"type":"v1","addr":"172.21.15.62:6827","nonce":37354}]},"public_addr":"172.21.15.62:6820/37354","cluster_addr":"172.21.15.62:6823/37354","heartbeat_back_addr":"172.21.15.62:6830/37354","heartbeat_front_addr":"172.21.15.62:6827/37354","state":["exists","up"]},{"osd":7,"uuid":"ac70b2a3-f41b-4bf2-b227-7f2cbb55426d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37355},{"type":"v1","addr":"172.21.15.62:6809","nonce":37355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37355},{"type":"v1","addr":"172.21.15.62:6811","nonce":37355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":37355},{"type":"v1","addr":"172.21.15.62:6815","nonce":37355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":37355},{"type":"v1","addr":"172.21.15.62:6813","nonce":37355}]},"public_addr":"172.21.15.62:6809/37355","cluster_addr":"172.21.15.62:6811/37355","heartbeat_back_addr":"172.21.15.62:6815/37355","heartbeat_front_addr":"172.21.15.62:6813/37355","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.368896+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.107273+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.276888+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.287502+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.124791+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.251937+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.255679+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.232797+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-05-22T15:27:01.686 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.674+0000 7fbe8aea7640 1 -- 172.21.15.16:0/2688090225 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fbe70065380 msgr2=0x7fbe70067830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:01.687 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.674+0000 7fbe8aea7640 1 --2- 172.21.15.16:0/2688090225 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fbe70065380 0x7fbe70067830 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fbe840713e0 tx=0x7fbe78009040 comp rx=0 tx=0).stop 2023-05-22T15:27:01.687 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.674+0000 7fbe8aea7640 1 -- 172.21.15.16:0/2688090225 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbe84070df0 msgr2=0x7fbe8414bf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:01.687 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.674+0000 7fbe8aea7640 1 --2- 172.21.15.16:0/2688090225 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbe84070df0 0x7fbe8414bf50 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fbe74000c10 tx=0x7fbe74019160 comp rx=0 tx=0).stop 2023-05-22T15:27:01.688 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.674+0000 7fbe8aea7640 1 -- 172.21.15.16:0/2688090225 shutdown_connections 2023-05-22T15:27:01.688 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.674+0000 7fbe8aea7640 1 --2- 172.21.15.16:0/2688090225 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbe840ca4e0 0x7fbe841592b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:01.688 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.674+0000 7fbe8aea7640 1 --2- 172.21.15.16:0/2688090225 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fbe70065380 0x7fbe70067830 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:01.688 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.674+0000 7fbe8aea7640 1 --2- 172.21.15.16:0/2688090225 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbe840c9b40 0x7fbe8415a2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:01.688 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.674+0000 7fbe8aea7640 1 --2- 172.21.15.16:0/2688090225 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbe84070df0 0x7fbe8414bf50 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:01.689 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.674+0000 7fbe8aea7640 1 -- 172.21.15.16:0/2688090225 >> 172.21.15.16:0/2688090225 conn(0x7fbe84069720 msgr2=0x7fbe841560f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:01.689 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.674+0000 7fbe8aea7640 1 -- 172.21.15.16:0/2688090225 shutdown_connections 2023-05-22T15:27:01.689 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.674+0000 7fbe8aea7640 1 -- 172.21.15.16:0/2688090225 wait complete. 2023-05-22T15:27:01.701 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-05-22T15:26:38.019781+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-05-22T15:26:43.570514+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2023-05-22T15:26:47.879159+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2023-05-22T15:26:48.522130+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-05-22T15:26:51.616142+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '27', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-05-22T15:27:01.704 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-05-22T15:27:01.818 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.810+0000 7f47e220d640 1 Processor -- start 2023-05-22T15:27:01.819 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.810+0000 7f47e220d640 1 -- start start 2023-05-22T15:27:01.820 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.810+0000 7f47e220d640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f47dc15bca0 0x7f47dc0cda80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:01.820 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.810+0000 7f47e220d640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f47dc0d2480 0x7f47dc0d28d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:01.820 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.810+0000 7f47e220d640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47dc0d2e10 con 0x7f47dc0b5340 2023-05-22T15:27:01.820 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.810+0000 7f47e220d640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47dc0cd660 con 0x7f47dc0d2480 2023-05-22T15:27:01.821 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.810+0000 7f47e220d640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47dc0cccc0 con 0x7f47dc15bca0 2023-05-22T15:27:01.821 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.810+0000 7f47d2ffd640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f47dc0d2480 0x7f47dc0d28d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:01.821 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.810+0000 7f47d2ffd640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f47dc0d2480 0x7f47dc0d28d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57464/0 (socket says 172.21.15.16:57464) 2023-05-22T15:27:01.821 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.810+0000 7f47d2ffd640 1 -- 172.21.15.16:0/2982307800 learned_addr learned my addr 172.21.15.16:0/2982307800 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:01.821 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.810+0000 7f47db7fe640 1 --2- 172.21.15.16:0/2982307800 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f47dc15bca0 0x7f47dc0cda80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:01.821 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.810+0000 7f47d2ffd640 1 -- 172.21.15.16:0/2982307800 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f47dc15bca0 msgr2=0x7f47dc0cda80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:01.821 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.810+0000 7f47d2ffd640 1 --2- 172.21.15.16:0/2982307800 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f47dc15bca0 0x7f47dc0cda80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:01.822 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.810+0000 7f47d2ffd640 1 -- 172.21.15.16:0/2982307800 >> v1:172.21.15.62:6789/0 conn(0x7f47dc0b5340 legacy=0x7f47dc0b57c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:01.822 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.810+0000 7f47d2ffd640 1 -- 172.21.15.16:0/2982307800 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47dc0ccfa0 con 0x7f47dc0d2480 2023-05-22T15:27:01.822 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.810+0000 7f47db7fe640 1 --2- 172.21.15.16:0/2982307800 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f47dc15bca0 0x7f47dc0cda80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:27:01.823 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.810+0000 7f47d2ffd640 1 --2- 172.21.15.16:0/2982307800 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f47dc0d2480 0x7f47dc0d28d0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f47c4009870 tx=0x7f47c401c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd7fcfb4ae3e78e4 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:01.823 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.814+0000 7f47daffd640 1 -- 172.21.15.16:0/2982307800 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47c40aa610 con 0x7f47dc0d2480 2023-05-22T15:27:01.823 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.814+0000 7f47daffd640 1 -- 172.21.15.16:0/2982307800 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47c40a6040 con 0x7f47dc0d2480 2023-05-22T15:27:01.824 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.814+0000 7f47daffd640 1 -- 172.21.15.16:0/2982307800 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47c40aa990 con 0x7f47dc0d2480 2023-05-22T15:27:01.824 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.814+0000 7f47e220d640 1 -- 172.21.15.16:0/2982307800 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f47dc0d2480 msgr2=0x7f47dc0d28d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:01.824 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.814+0000 7f47e220d640 1 --2- 172.21.15.16:0/2982307800 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f47dc0d2480 0x7f47dc0d28d0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f47c4009870 tx=0x7f47c401c4d0 comp rx=0 tx=0).stop 2023-05-22T15:27:01.824 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.814+0000 7f47e220d640 1 -- 172.21.15.16:0/2982307800 shutdown_connections 2023-05-22T15:27:01.824 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.814+0000 7f47e220d640 1 --2- 172.21.15.16:0/2982307800 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f47dc0d2480 0x7f47dc0d28d0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:01.824 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.814+0000 7f47e220d640 1 --2- 172.21.15.16:0/2982307800 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f47dc15bca0 0x7f47dc0cda80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:01.825 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.814+0000 7f47e220d640 1 -- 172.21.15.16:0/2982307800 >> 172.21.15.16:0/2982307800 conn(0x7f47dc0655c0 msgr2=0x7f47dc070e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:01.825 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.814+0000 7f47e220d640 1 -- 172.21.15.16:0/2982307800 shutdown_connections 2023-05-22T15:27:01.825 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.814+0000 7f47e220d640 1 -- 172.21.15.16:0/2982307800 wait complete. 2023-05-22T15:27:01.827 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.814+0000 7f47e220d640 1 Processor -- start 2023-05-22T15:27:01.827 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.814+0000 7f47e220d640 1 -- start start 2023-05-22T15:27:01.828 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.814+0000 7f47e220d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f47dc0b5340 0x7f47dc149d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:01.828 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.814+0000 7f47e220d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f47dc0d2480 0x7f47dc0c5350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:01.828 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.814+0000 7f47e220d640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f47dc15bca0 0x7f47dc1486b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:01.828 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.814+0000 7f47e220d640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47dc1498c0 con 0x7f47dc0d2480 2023-05-22T15:27:01.828 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.814+0000 7f47e220d640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47dc149000 con 0x7f47dc0b5340 2023-05-22T15:27:01.828 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.814+0000 7f47e220d640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47dc14a290 con 0x7f47dc15bca0 2023-05-22T15:27:01.829 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.814+0000 7f47d2ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f47dc0d2480 0x7f47dc0c5350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:01.829 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.814+0000 7f47d2ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f47dc0d2480 0x7f47dc0c5350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57478/0 (socket says 172.21.15.16:57478) 2023-05-22T15:27:01.829 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.814+0000 7f47d2ffd640 1 -- 172.21.15.16:0/174675088 learned_addr learned my addr 172.21.15.16:0/174675088 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:01.829 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.814+0000 7f47db7fe640 1 --2- 172.21.15.16:0/174675088 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f47dc0b5340 0x7f47dc149d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:01.829 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.814+0000 7f47dbfff640 1 --2- 172.21.15.16:0/174675088 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f47dc15bca0 0x7f47dc1486b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:01.829 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.814+0000 7f47d2ffd640 1 -- 172.21.15.16:0/174675088 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f47dc15bca0 msgr2=0x7f47dc1486b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:01.830 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.814+0000 7f47d2ffd640 1 --2- 172.21.15.16:0/174675088 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f47dc15bca0 0x7f47dc1486b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:01.830 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.814+0000 7f47d2ffd640 1 -- 172.21.15.16:0/174675088 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f47dc0b5340 msgr2=0x7f47dc149d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:01.830 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.814+0000 7f47d2ffd640 1 --2- 172.21.15.16:0/174675088 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f47dc0b5340 0x7f47dc149d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:01.830 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.814+0000 7f47d2ffd640 1 -- 172.21.15.16:0/174675088 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47c40094d0 con 0x7f47dc0d2480 2023-05-22T15:27:01.830 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.814+0000 7f47dbfff640 1 --2- 172.21.15.16:0/174675088 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f47dc15bca0 0x7f47dc1486b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:27:01.831 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.814+0000 7f47db7fe640 1 --2- 172.21.15.16:0/174675088 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f47dc0b5340 0x7f47dc149d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:27:01.831 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.814+0000 7f47d2ffd640 1 --2- 172.21.15.16:0/174675088 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f47dc0d2480 0x7f47dc0c5350 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f47c40ac040 tx=0x7f47c401ad10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:01.831 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.814+0000 7f47d8ff9640 1 -- 172.21.15.16:0/174675088 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47c40bd5f0 con 0x7f47dc0d2480 2023-05-22T15:27:01.831 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.814+0000 7f47d8ff9640 1 -- 172.21.15.16:0/174675088 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47c40bd770 con 0x7f47dc0d2480 2023-05-22T15:27:01.831 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.814+0000 7f47e220d640 1 -- 172.21.15.16:0/174675088 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47dc160360 con 0x7f47dc0d2480 2023-05-22T15:27:01.831 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.814+0000 7f47d8ff9640 1 -- 172.21.15.16:0/174675088 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47c40bdb20 con 0x7f47dc0d2480 2023-05-22T15:27:01.831 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.814+0000 7f47e220d640 1 -- 172.21.15.16:0/174675088 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47dc0cdfc0 con 0x7f47dc0d2480 2023-05-22T15:27:01.832 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.818+0000 7f47e220d640 1 -- 172.21.15.16:0/174675088 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47dc1604f0 con 0x7f47dc0d2480 2023-05-22T15:27:01.832 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.818+0000 7f47d8ff9640 1 -- 172.21.15.16:0/174675088 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7f47c40b33f0 con 0x7f47dc0d2480 2023-05-22T15:27:01.832 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.818+0000 7f47d8ff9640 1 --2- 172.21.15.16:0/174675088 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f47bc065380 0x7f47bc067830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:01.832 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.818+0000 7f47db7fe640 1 --2- 172.21.15.16:0/174675088 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f47bc065380 0x7f47bc067830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:01.832 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.818+0000 7f47d8ff9640 1 -- 172.21.15.16:0/174675088 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f47c411d810 con 0x7f47dc0d2480 2023-05-22T15:27:01.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.818+0000 7f47db7fe640 1 --2- 172.21.15.16:0/174675088 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f47bc065380 0x7f47bc067830 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f47dc154070 tx=0x7f47cc008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:01.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.822+0000 7f47d8ff9640 1 -- 172.21.15.16:0/174675088 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f47c40eb1e0 con 0x7f47dc0d2480 2023-05-22T15:27:01.979 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.970+0000 7f47e220d640 1 -- 172.21.15.16:0/174675088 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f47dc149a70 con 0x7f47dc0d2480 2023-05-22T15:27:01.980 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.970+0000 7f47d8ff9640 1 -- 172.21.15.16:0/174675088 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) v1 ==== 93+0+10 (secure 0 0 0) 0x7f47dc149a70 con 0x7f47dc0d2480 2023-05-22T15:27:01.980 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2023-05-22T15:27:01.982 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.970+0000 7f47e220d640 1 -- 172.21.15.16:0/174675088 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f47bc065380 msgr2=0x7f47bc067830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:01.982 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.970+0000 7f47e220d640 1 --2- 172.21.15.16:0/174675088 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f47bc065380 0x7f47bc067830 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f47dc154070 tx=0x7f47cc008040 comp rx=0 tx=0).stop 2023-05-22T15:27:01.982 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.970+0000 7f47e220d640 1 -- 172.21.15.16:0/174675088 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f47dc0d2480 msgr2=0x7f47dc0c5350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:01.982 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.970+0000 7f47e220d640 1 --2- 172.21.15.16:0/174675088 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f47dc0d2480 0x7f47dc0c5350 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f47c40ac040 tx=0x7f47c401ad10 comp rx=0 tx=0).stop 2023-05-22T15:27:01.982 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.970+0000 7f47e220d640 1 -- 172.21.15.16:0/174675088 shutdown_connections 2023-05-22T15:27:01.983 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.970+0000 7f47e220d640 1 --2- 172.21.15.16:0/174675088 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f47bc065380 0x7f47bc067830 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:01.983 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.970+0000 7f47e220d640 1 --2- 172.21.15.16:0/174675088 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f47dc15bca0 0x7f47dc1486b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:01.983 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.970+0000 7f47e220d640 1 --2- 172.21.15.16:0/174675088 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f47dc0d2480 0x7f47dc0c5350 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:01.983 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.970+0000 7f47e220d640 1 --2- 172.21.15.16:0/174675088 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f47dc0b5340 0x7f47dc149d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:01.983 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.970+0000 7f47e220d640 1 -- 172.21.15.16:0/174675088 >> 172.21.15.16:0/174675088 conn(0x7f47dc0655c0 msgr2=0x7f47dc154880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:01.983 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.970+0000 7f47e220d640 1 -- 172.21.15.16:0/174675088 shutdown_connections 2023-05-22T15:27:01.983 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:01.970+0000 7f47e220d640 1 -- 172.21.15.16:0/174675088 wait complete. 2023-05-22T15:27:01.998 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-05-22T15:27:02.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.106+0000 7fbbed169640 1 Processor -- start 2023-05-22T15:27:02.118 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.106+0000 7fbbed169640 1 -- start start 2023-05-22T15:27:02.119 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.110+0000 7fbbed169640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbbe815bcd0 0x7fbbe815c120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:02.119 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.110+0000 7fbbed169640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fbbe80c8660 0x7fbbe80c8ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:02.119 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.110+0000 7fbbed169640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbbe8158cd0 con 0x7fbbe8158210 2023-05-22T15:27:02.119 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.110+0000 7fbbed169640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbbe80c9020 con 0x7fbbe80c8660 2023-05-22T15:27:02.119 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.110+0000 7fbbed169640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbbe815c790 con 0x7fbbe815bcd0 2023-05-22T15:27:02.119 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.110+0000 7fbbe6d9d640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fbbe8158210 0x7fbbe8159160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:50976/0 (socket says 172.21.15.16:50976) 2023-05-22T15:27:02.120 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.110+0000 7fbbe6d9d640 1 -- 172.21.15.16:0/2494074687 learned_addr learned my addr 172.21.15.16:0/2494074687 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:27:02.121 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.110+0000 7fbbe5d9b640 1 -- 172.21.15.16:0/2494074687 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1201566021 0 0) 0x7fbbe8158cd0 con 0x7fbbe8158210 2023-05-22T15:27:02.121 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.110+0000 7fbbe659c640 1 --2- 172.21.15.16:0/2494074687 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbbe815bcd0 0x7fbbe815c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:02.121 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.110+0000 7fbbe5d9b640 1 -- 172.21.15.16:0/2494074687 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbbd0003530 con 0x7fbbe8158210 2023-05-22T15:27:02.121 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.110+0000 7fbbe759e640 1 --2- 172.21.15.16:0/2494074687 >> v2:172.21.15.62:3300/0 conn(0x7fbbe80c8660 0x7fbbe80c8ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:02.122 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.110+0000 7fbbe5d9b640 1 -- 172.21.15.16:0/2494074687 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 881898858 0 0) 0x7fbbd0003530 con 0x7fbbe8158210 2023-05-22T15:27:02.122 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.110+0000 7fbbe5d9b640 1 -- 172.21.15.16:0/2494074687 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbbe815bcd0 msgr2=0x7fbbe815c120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:02.122 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.110+0000 7fbbe5d9b640 1 --2- 172.21.15.16:0/2494074687 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbbe815bcd0 0x7fbbe815c120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:02.122 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.110+0000 7fbbe5d9b640 1 -- 172.21.15.16:0/2494074687 >> v2:172.21.15.62:3300/0 conn(0x7fbbe80c8660 msgr2=0x7fbbe80c8ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:02.122 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.110+0000 7fbbe5d9b640 1 --2- 172.21.15.16:0/2494074687 >> v2:172.21.15.62:3300/0 conn(0x7fbbe80c8660 0x7fbbe80c8ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:02.122 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.110+0000 7fbbe5d9b640 1 -- 172.21.15.16:0/2494074687 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbe815ca70 con 0x7fbbe8158210 2023-05-22T15:27:02.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.110+0000 7fbbe5d9b640 1 -- 172.21.15.16:0/2494074687 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7fbbd8001a00 con 0x7fbbe8158210 2023-05-22T15:27:02.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.110+0000 7fbbe5d9b640 1 -- 172.21.15.16:0/2494074687 >> v1:172.21.15.16:6789/0 conn(0x7fbbe8158210 legacy=0x7fbbe8159160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:02.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.110+0000 7fbbe5d9b640 1 --2- 172.21.15.16:0/2494074687 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbbd0003980 0x7fbbd0003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:02.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.110+0000 7fbbe5d9b640 1 --2- 172.21.15.16:0/2494074687 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbbe815bcd0 0x7fbbd00047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:02.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.110+0000 7fbbe5d9b640 1 --2- 172.21.15.16:0/2494074687 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbbd0004d10 0x7fbbd0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:02.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.110+0000 7fbbe5d9b640 1 -- 172.21.15.16:0/2494074687 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbe80c9020 con 0x7fbbd0003980 2023-05-22T15:27:02.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.110+0000 7fbbe5d9b640 1 -- 172.21.15.16:0/2494074687 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbe815c790 con 0x7fbbd0004d10 2023-05-22T15:27:02.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.110+0000 7fbbe5d9b640 1 -- 172.21.15.16:0/2494074687 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbbd000bac0 con 0x7fbbe815bcd0 2023-05-22T15:27:02.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.110+0000 7fbbe6d9d640 1 --2- 172.21.15.16:0/2494074687 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbbd0003980 0x7fbbd0003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:02.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.110+0000 7fbbe659c640 1 --2- 172.21.15.16:0/2494074687 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbbe815bcd0 0x7fbbd00047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:02.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.110+0000 7fbbe759e640 1 --2- 172.21.15.16:0/2494074687 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbbd0004d10 0x7fbbd0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:02.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.110+0000 7fbbe6d9d640 1 -- 172.21.15.16:0/2494074687 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbbe815bcd0 msgr2=0x7fbbd00047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:02.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.110+0000 7fbbe6d9d640 1 --2- 172.21.15.16:0/2494074687 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbbe815bcd0 0x7fbbd00047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:02.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.110+0000 7fbbe6d9d640 1 -- 172.21.15.16:0/2494074687 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbbd0004d10 msgr2=0x7fbbd0005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:02.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.110+0000 7fbbe6d9d640 1 --2- 172.21.15.16:0/2494074687 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbbd0004d10 0x7fbbd0005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:02.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.110+0000 7fbbe6d9d640 1 -- 172.21.15.16:0/2494074687 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbd0003530 con 0x7fbbd0003980 2023-05-22T15:27:02.127 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.114+0000 7fbbe6d9d640 1 --2- 172.21.15.16:0/2494074687 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbbd0003980 0x7fbbd0003d90 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fbbd800a860 tx=0x7fbbd801d330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49d5568202f0d9e9 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:02.127 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.114+0000 7fbbe5d9b640 1 -- 172.21.15.16:0/2494074687 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbd80ad630 con 0x7fbbd0003980 2023-05-22T15:27:02.127 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.114+0000 7fbbe5d9b640 1 -- 172.21.15.16:0/2494074687 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbbd80a8070 con 0x7fbbd0003980 2023-05-22T15:27:02.127 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.114+0000 7fbbe5d9b640 1 -- 172.21.15.16:0/2494074687 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbd80ad9f0 con 0x7fbbd0003980 2023-05-22T15:27:02.127 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.114+0000 7fbbed169640 1 -- 172.21.15.16:0/2494074687 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbbd0003980 msgr2=0x7fbbd0003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:02.128 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.114+0000 7fbbed169640 1 --2- 172.21.15.16:0/2494074687 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbbd0003980 0x7fbbd0003d90 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fbbd800a860 tx=0x7fbbd801d330 comp rx=0 tx=0).stop 2023-05-22T15:27:02.128 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.114+0000 7fbbe759e640 1 -- 172.21.15.16:0/2494074687 reap_dead start 2023-05-22T15:27:02.128 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.114+0000 7fbbed169640 1 -- 172.21.15.16:0/2494074687 shutdown_connections 2023-05-22T15:27:02.128 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.114+0000 7fbbed169640 1 -- 172.21.15.16:0/2494074687 >> 172.21.15.16:0/2494074687 conn(0x7fbbe8069720 msgr2=0x7fbbe8069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:02.128 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.114+0000 7fbbed169640 1 -- 172.21.15.16:0/2494074687 shutdown_connections 2023-05-22T15:27:02.128 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.114+0000 7fbbed169640 1 -- 172.21.15.16:0/2494074687 wait complete. 2023-05-22T15:27:02.128 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.114+0000 7fbbed169640 1 Processor -- start 2023-05-22T15:27:02.129 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.114+0000 7fbbed169640 1 -- start start 2023-05-22T15:27:02.129 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.114+0000 7fbbed169640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbbd0003980 0x7fbbe80c50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:02.129 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.114+0000 7fbbed169640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbbe815d760 0x7fbbe815dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:02.129 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.114+0000 7fbbed169640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbbe80c75c0 0x7fbbe80c7a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:02.129 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.114+0000 7fbbed169640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbe80c59e0 con 0x7fbbe815d760 2023-05-22T15:27:02.129 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.114+0000 7fbbed169640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbe815e110 con 0x7fbbe80c75c0 2023-05-22T15:27:02.129 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.114+0000 7fbbed169640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbbe8159110 con 0x7fbbd0003980 2023-05-22T15:27:02.130 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.114+0000 7fbbe659c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbbe815d760 0x7fbbe815dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:02.130 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.114+0000 7fbbe659c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbbe815d760 0x7fbbe815dbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57506/0 (socket says 172.21.15.16:57506) 2023-05-22T15:27:02.130 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.114+0000 7fbbe659c640 1 -- 172.21.15.16:0/295178297 learned_addr learned my addr 172.21.15.16:0/295178297 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:02.130 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.114+0000 7fbbe759e640 1 --2- 172.21.15.16:0/295178297 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbbe80c75c0 0x7fbbe80c7a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:02.130 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.114+0000 7fbbe6d9d640 1 --2- 172.21.15.16:0/295178297 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbbd0003980 0x7fbbe80c50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:02.130 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.114+0000 7fbbe659c640 1 -- 172.21.15.16:0/295178297 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbbd0003980 msgr2=0x7fbbe80c50e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:02.130 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.114+0000 7fbbe659c640 1 --2- 172.21.15.16:0/295178297 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbbd0003980 0x7fbbe80c50e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:02.131 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.114+0000 7fbbe659c640 1 -- 172.21.15.16:0/295178297 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbbe80c75c0 msgr2=0x7fbbe80c7a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:02.131 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.118+0000 7fbbe659c640 1 --2- 172.21.15.16:0/295178297 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbbe80c75c0 0x7fbbe80c7a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:02.131 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.118+0000 7fbbe659c640 1 -- 172.21.15.16:0/295178297 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbe80ca690 con 0x7fbbe815d760 2023-05-22T15:27:02.131 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.118+0000 7fbbe659c640 1 --2- 172.21.15.16:0/295178297 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbbe815d760 0x7fbbe815dbd0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fbbdc00f580 tx=0x7fbbdc00fa40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:02.131 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.118+0000 7fbbcf7fe640 1 -- 172.21.15.16:0/295178297 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbdc014870 con 0x7fbbe815d760 2023-05-22T15:27:02.131 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.118+0000 7fbbcf7fe640 1 -- 172.21.15.16:0/295178297 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbbdc0149d0 con 0x7fbbe815d760 2023-05-22T15:27:02.131 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.118+0000 7fbbed169640 1 -- 172.21.15.16:0/295178297 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbbe81585e0 con 0x7fbbe815d760 2023-05-22T15:27:02.132 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.118+0000 7fbbcf7fe640 1 -- 172.21.15.16:0/295178297 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbdc014d60 con 0x7fbbe815d760 2023-05-22T15:27:02.132 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.118+0000 7fbbed169640 1 -- 172.21.15.16:0/295178297 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbbe80ca820 con 0x7fbbe815d760 2023-05-22T15:27:02.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.118+0000 7fbbed169640 1 -- 172.21.15.16:0/295178297 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbbe804dbe0 con 0x7fbbe815d760 2023-05-22T15:27:02.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.118+0000 7fbbcf7fe640 1 -- 172.21.15.16:0/295178297 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7fbbdc015410 con 0x7fbbe815d760 2023-05-22T15:27:02.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.118+0000 7fbbcf7fe640 1 --2- 172.21.15.16:0/295178297 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fbbd4065380 0x7fbbd4067830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:02.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.118+0000 7fbbcf7fe640 1 -- 172.21.15.16:0/295178297 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6813+0+0 (secure 0 0 0) 0x7fbbdc0835e0 con 0x7fbbe815d760 2023-05-22T15:27:02.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.118+0000 7fbbe6d9d640 1 --2- 172.21.15.16:0/295178297 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fbbd4065380 0x7fbbd4067830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:02.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.118+0000 7fbbe6d9d640 1 --2- 172.21.15.16:0/295178297 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fbbd4065380 0x7fbbd4067830 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fbbd80ad000 tx=0x7fbbd8001e00 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:02.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.122+0000 7fbbcf7fe640 1 -- 172.21.15.16:0/295178297 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fbbdc050f00 con 0x7fbbe815d760 2023-05-22T15:27:02.286 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.274+0000 7fbbed169640 1 -- 172.21.15.16:0/295178297 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fbbe80ad850 con 0x7fbbe815d760 2023-05-22T15:27:02.286 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.278+0000 7fbbcf7fe640 1 -- 172.21.15.16:0/295178297 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v27) v1 ==== 92+0+10 (secure 0 0 0) 0x7fbbdc054520 con 0x7fbbe815d760 2023-05-22T15:27:02.287 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 8 2023-05-22T15:27:02.289 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.278+0000 7fbbed169640 1 -- 172.21.15.16:0/295178297 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fbbd4065380 msgr2=0x7fbbd4067830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:02.289 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.278+0000 7fbbed169640 1 --2- 172.21.15.16:0/295178297 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fbbd4065380 0x7fbbd4067830 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fbbd80ad000 tx=0x7fbbd8001e00 comp rx=0 tx=0).stop 2023-05-22T15:27:02.289 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.278+0000 7fbbed169640 1 -- 172.21.15.16:0/295178297 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbbe815d760 msgr2=0x7fbbe815dbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:02.289 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.278+0000 7fbbed169640 1 --2- 172.21.15.16:0/295178297 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbbe815d760 0x7fbbe815dbd0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fbbdc00f580 tx=0x7fbbdc00fa40 comp rx=0 tx=0).stop 2023-05-22T15:27:02.289 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.278+0000 7fbbed169640 1 -- 172.21.15.16:0/295178297 shutdown_connections 2023-05-22T15:27:02.289 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.278+0000 7fbbed169640 1 --2- 172.21.15.16:0/295178297 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbbe80c75c0 0x7fbbe80c7a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:02.290 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.278+0000 7fbbed169640 1 --2- 172.21.15.16:0/295178297 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbbe815d760 0x7fbbe815dbd0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:02.290 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.278+0000 7fbbed169640 1 --2- 172.21.15.16:0/295178297 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fbbd4065380 0x7fbbd4067830 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:02.290 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.278+0000 7fbbed169640 1 --2- 172.21.15.16:0/295178297 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbbd0003980 0x7fbbe80c50e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:02.290 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.278+0000 7fbbed169640 1 -- 172.21.15.16:0/295178297 >> 172.21.15.16:0/295178297 conn(0x7fbbe8069720 msgr2=0x7fbbe80ae170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:02.290 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.278+0000 7fbbed169640 1 -- 172.21.15.16:0/295178297 shutdown_connections 2023-05-22T15:27:02.290 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.278+0000 7fbbed169640 1 -- 172.21.15.16:0/295178297 wait complete. 2023-05-22T15:27:02.306 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-05-22T15:27:02.434 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.426+0000 7f2ede657640 1 Processor -- start 2023-05-22T15:27:02.435 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.426+0000 7f2ede657640 1 -- start start 2023-05-22T15:27:02.435 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.426+0000 7f2ede657640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2ed8159970 0x7f2ed8159d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:02.435 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.426+0000 7f2ede657640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f2ed815d680 0x7f2ed80cb260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:02.435 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.426+0000 7f2ede657640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ed8070c30 con 0x7f2ed815dad0 2023-05-22T15:27:02.436 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.426+0000 7f2ede657640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ed8070e60 con 0x7f2ed815d680 2023-05-22T15:27:02.436 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.426+0000 7f2ede657640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ed8071140 con 0x7f2ed8159970 2023-05-22T15:27:02.436 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.426+0000 7f2ed7fff640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2ed8159970 0x7f2ed8159d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:02.437 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.426+0000 7f2ed7fff640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2ed8159970 0x7f2ed8159d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3301/0 says I am v2:172.21.15.16:52910/0 (socket says 172.21.15.16:52910) 2023-05-22T15:27:02.437 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.426+0000 7f2ed7fff640 1 -- 172.21.15.16:0/4130129150 learned_addr learned my addr 172.21.15.16:0/4130129150 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:02.437 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.426+0000 7f2ecffff640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f2ed815d680 0x7f2ed80cb260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:02.437 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.426+0000 7f2ecffff640 1 -- 172.21.15.16:0/4130129150 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2ed8159970 msgr2=0x7f2ed8159d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:02.437 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.426+0000 7f2ecffff640 1 --2- 172.21.15.16:0/4130129150 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2ed8159970 0x7f2ed8159d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:02.437 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.426+0000 7f2ecffff640 1 -- 172.21.15.16:0/4130129150 >> v1:172.21.15.62:6789/0 conn(0x7f2ed815dad0 legacy=0x7f2ed806b6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:02.437 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.426+0000 7f2ecffff640 1 -- 172.21.15.16:0/4130129150 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ed8071420 con 0x7f2ed815d680 2023-05-22T15:27:02.438 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.430+0000 7f2ecffff640 1 --2- 172.21.15.16:0/4130129150 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2ed815d680 0x7f2ed80cb260 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f2ec00097c0 tx=0x7f2ec001c3c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bcce2fd096149446 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:02.439 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.430+0000 7f2ed77fe640 1 -- 172.21.15.16:0/4130129150 <== mon.1 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ec00ac640 con 0x7f2ed815d680 2023-05-22T15:27:02.439 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.430+0000 7f2ed77fe640 1 -- 172.21.15.16:0/4130129150 <== mon.1 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ec00ac7a0 con 0x7f2ed815d680 2023-05-22T15:27:02.439 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.430+0000 7f2ed77fe640 1 -- 172.21.15.16:0/4130129150 <== mon.1 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ec00acb30 con 0x7f2ed815d680 2023-05-22T15:27:02.440 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.430+0000 7f2ede657640 1 -- 172.21.15.16:0/4130129150 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2ed815d680 msgr2=0x7f2ed80cb260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:02.440 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.430+0000 7f2ede657640 1 --2- 172.21.15.16:0/4130129150 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2ed815d680 0x7f2ed80cb260 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f2ec00097c0 tx=0x7f2ec001c3c0 comp rx=0 tx=0).stop 2023-05-22T15:27:02.440 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.430+0000 7f2ede657640 1 -- 172.21.15.16:0/4130129150 shutdown_connections 2023-05-22T15:27:02.441 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.430+0000 7f2ede657640 1 --2- 172.21.15.16:0/4130129150 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2ed815d680 0x7f2ed80cb260 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:02.441 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.430+0000 7f2ede657640 1 --2- 172.21.15.16:0/4130129150 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2ed8159970 0x7f2ed8159d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:02.441 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.430+0000 7f2ede657640 1 -- 172.21.15.16:0/4130129150 >> 172.21.15.16:0/4130129150 conn(0x7f2ed8069720 msgr2=0x7f2ed8069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:02.441 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.430+0000 7f2ede657640 1 -- 172.21.15.16:0/4130129150 shutdown_connections 2023-05-22T15:27:02.441 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.430+0000 7f2ede657640 1 -- 172.21.15.16:0/4130129150 wait complete. 2023-05-22T15:27:02.442 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.430+0000 7f2ede657640 1 Processor -- start 2023-05-22T15:27:02.443 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.430+0000 7f2ede657640 1 -- start start 2023-05-22T15:27:02.443 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.430+0000 7f2ede657640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2ed8159970 0x7f2ed80cd460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:02.443 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.430+0000 7f2ede657640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2ed815d680 0x7f2ed81f8fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:02.443 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.430+0000 7f2ede657640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2ed815dad0 0x7f2ed80afbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:02.443 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.430+0000 7f2ede657640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ed8150b90 con 0x7f2ed815d680 2023-05-22T15:27:02.443 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.430+0000 7f2ede657640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ed8150fb0 con 0x7f2ed815dad0 2023-05-22T15:27:02.443 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.430+0000 7f2ede657640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ed814fc70 con 0x7f2ed8159970 2023-05-22T15:27:02.444 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.434+0000 7f2ecffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2ed815d680 0x7f2ed81f8fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:02.444 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.434+0000 7f2ecffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2ed815d680 0x7f2ed81f8fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57514/0 (socket says 172.21.15.16:57514) 2023-05-22T15:27:02.444 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.434+0000 7f2ecffff640 1 -- 172.21.15.16:0/3825076779 learned_addr learned my addr 172.21.15.16:0/3825076779 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:02.444 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.434+0000 7f2edcbf4640 1 --2- 172.21.15.16:0/3825076779 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2ed815dad0 0x7f2ed80afbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:02.445 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.434+0000 7f2ed7fff640 1 --2- 172.21.15.16:0/3825076779 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2ed8159970 0x7f2ed80cd460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:02.445 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.434+0000 7f2ecffff640 1 -- 172.21.15.16:0/3825076779 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2ed8159970 msgr2=0x7f2ed80cd460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:02.445 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.434+0000 7f2ecffff640 1 --2- 172.21.15.16:0/3825076779 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2ed8159970 0x7f2ed80cd460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:02.445 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.434+0000 7f2ecffff640 1 -- 172.21.15.16:0/3825076779 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2ed815dad0 msgr2=0x7f2ed80afbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:02.445 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.434+0000 7f2ecffff640 1 --2- 172.21.15.16:0/3825076779 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2ed815dad0 0x7f2ed80afbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:02.445 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.434+0000 7f2ecffff640 1 -- 172.21.15.16:0/3825076779 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ec0009420 con 0x7f2ed815d680 2023-05-22T15:27:02.445 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.434+0000 7f2edcbf4640 1 --2- 172.21.15.16:0/3825076779 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2ed815dad0 0x7f2ed80afbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:27:02.446 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.434+0000 7f2ecffff640 1 --2- 172.21.15.16:0/3825076779 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2ed815d680 0x7f2ed81f8fa0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f2ec001aa70 tx=0x7f2ec0019160 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:02.446 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.434+0000 7f2ed57fa640 1 -- 172.21.15.16:0/3825076779 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ec00bb070 con 0x7f2ed815d680 2023-05-22T15:27:02.446 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.434+0000 7f2ed57fa640 1 -- 172.21.15.16:0/3825076779 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ec00b5400 con 0x7f2ed815d680 2023-05-22T15:27:02.446 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.434+0000 7f2ed57fa640 1 -- 172.21.15.16:0/3825076779 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ec00b57d0 con 0x7f2ed815d680 2023-05-22T15:27:02.446 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.434+0000 7f2ede657640 1 -- 172.21.15.16:0/3825076779 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ed80cae20 con 0x7f2ed815d680 2023-05-22T15:27:02.446 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.434+0000 7f2ede657640 1 -- 172.21.15.16:0/3825076779 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ed80cac90 con 0x7f2ed815d680 2023-05-22T15:27:02.447 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.434+0000 7f2ede657640 1 -- 172.21.15.16:0/3825076779 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ed806bdd0 con 0x7f2ed815d680 2023-05-22T15:27:02.447 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.434+0000 7f2ed57fa640 1 -- 172.21.15.16:0/3825076779 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7f2ec00b3040 con 0x7f2ed815d680 2023-05-22T15:27:02.447 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.434+0000 7f2ed57fa640 1 --2- 172.21.15.16:0/3825076779 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f2eb8065380 0x7f2eb8067830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:02.447 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.434+0000 7f2ed57fa640 1 -- 172.21.15.16:0/3825076779 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f2ec011c610 con 0x7f2ed815d680 2023-05-22T15:27:02.447 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.434+0000 7f2ed7fff640 1 --2- 172.21.15.16:0/3825076779 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f2eb8065380 0x7f2eb8067830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:02.448 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.434+0000 7f2ed7fff640 1 --2- 172.21.15.16:0/3825076779 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f2eb8065380 0x7f2eb8067830 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f2ec8009ae0 tx=0x7f2ec8008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:02.449 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.438+0000 7f2ed57fa640 1 -- 172.21.15.16:0/3825076779 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f2ec00e9f90 con 0x7f2ed815d680 2023-05-22T15:27:02.595 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.586+0000 7f2ede657640 1 -- 172.21.15.16:0/3825076779 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f2ed804dbe0 con 0x7f2ed815d680 2023-05-22T15:27:02.596 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.586+0000 7f2ed57fa640 1 -- 172.21.15.16:0/3825076779 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) v1 ==== 104+0+11 (secure 0 0 0) 0x7f2ec00ed5b0 con 0x7f2ed815d680 2023-05-22T15:27:02.596 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2023-05-22T15:27:02.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.586+0000 7f2ede657640 1 -- 172.21.15.16:0/3825076779 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f2eb8065380 msgr2=0x7f2eb8067830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:02.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.586+0000 7f2ede657640 1 --2- 172.21.15.16:0/3825076779 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f2eb8065380 0x7f2eb8067830 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f2ec8009ae0 tx=0x7f2ec8008040 comp rx=0 tx=0).stop 2023-05-22T15:27:02.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.586+0000 7f2ede657640 1 -- 172.21.15.16:0/3825076779 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2ed815d680 msgr2=0x7f2ed81f8fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:02.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.586+0000 7f2ede657640 1 --2- 172.21.15.16:0/3825076779 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2ed815d680 0x7f2ed81f8fa0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f2ec001aa70 tx=0x7f2ec0019160 comp rx=0 tx=0).stop 2023-05-22T15:27:02.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.586+0000 7f2ede657640 1 -- 172.21.15.16:0/3825076779 shutdown_connections 2023-05-22T15:27:02.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.586+0000 7f2ede657640 1 --2- 172.21.15.16:0/3825076779 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2ed815dad0 0x7f2ed80afbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:02.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.586+0000 7f2ede657640 1 --2- 172.21.15.16:0/3825076779 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2ed815d680 0x7f2ed81f8fa0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:02.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.586+0000 7f2ede657640 1 --2- 172.21.15.16:0/3825076779 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f2eb8065380 0x7f2eb8067830 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:02.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.586+0000 7f2ede657640 1 --2- 172.21.15.16:0/3825076779 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2ed8159970 0x7f2ed80cd460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:02.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.586+0000 7f2ede657640 1 -- 172.21.15.16:0/3825076779 >> 172.21.15.16:0/3825076779 conn(0x7f2ed8069720 msgr2=0x7f2ed8156f70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:02.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.586+0000 7f2ede657640 1 -- 172.21.15.16:0/3825076779 shutdown_connections 2023-05-22T15:27:02.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.586+0000 7f2ede657640 1 -- 172.21.15.16:0/3825076779 wait complete. 2023-05-22T15:27:02.614 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-05-22T15:27:02.740 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.730+0000 7f0fa5274640 1 Processor -- start 2023-05-22T15:27:02.741 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.730+0000 7f0fa5274640 1 -- start start 2023-05-22T15:27:02.742 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.730+0000 7f0fa5274640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f0fa0159d40 0x7f0fa00cb260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:02.742 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.730+0000 7f0fa5274640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0fa006b6c0 0x7f0fa006bb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:02.742 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.730+0000 7f0fa5274640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0fa0070c30 con 0x7f0fa0159970 2023-05-22T15:27:02.742 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.730+0000 7f0fa5274640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0fa015dd90 con 0x7f0fa0159d40 2023-05-22T15:27:02.742 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.730+0000 7f0fa5274640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0fa006c040 con 0x7f0fa006b6c0 2023-05-22T15:27:02.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.730+0000 7f0f9e7fc640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f0fa0159d40 0x7f0fa00cb260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:02.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.730+0000 7f0f9e7fc640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f0fa0159d40 0x7f0fa00cb260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57522/0 (socket says 172.21.15.16:57522) 2023-05-22T15:27:02.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.730+0000 7f0f9e7fc640 1 -- 172.21.15.16:0/3964263040 learned_addr learned my addr 172.21.15.16:0/3964263040 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:02.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.734+0000 7f0f9f7fe640 1 --2- 172.21.15.16:0/3964263040 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0fa006b6c0 0x7f0fa006bb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:02.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.734+0000 7f0f9e7fc640 1 -- 172.21.15.16:0/3964263040 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0fa006b6c0 msgr2=0x7f0fa006bb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:02.744 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.734+0000 7f0f9e7fc640 1 --2- 172.21.15.16:0/3964263040 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0fa006b6c0 0x7f0fa006bb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:02.744 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.734+0000 7f0f9e7fc640 1 -- 172.21.15.16:0/3964263040 >> v1:172.21.15.62:6789/0 conn(0x7f0fa0159970 legacy=0x7f0fa015d680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:02.744 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.734+0000 7f0f9e7fc640 1 -- 172.21.15.16:0/3964263040 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fa0070e60 con 0x7f0fa0159d40 2023-05-22T15:27:02.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.734+0000 7f0f9e7fc640 1 --2- 172.21.15.16:0/3964263040 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0fa0159d40 0x7f0fa00cb260 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f0f88009e50 tx=0x7f0f8801c980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3da0ea0ea84a17da server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:02.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.734+0000 7f0f9dffb640 1 -- 172.21.15.16:0/3964263040 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f880a5730 con 0x7f0fa0159d40 2023-05-22T15:27:02.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.734+0000 7f0f9dffb640 1 -- 172.21.15.16:0/3964263040 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f880a5890 con 0x7f0fa0159d40 2023-05-22T15:27:02.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.734+0000 7f0f9dffb640 1 -- 172.21.15.16:0/3964263040 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f880a5c20 con 0x7f0fa0159d40 2023-05-22T15:27:02.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.734+0000 7f0fa5274640 1 -- 172.21.15.16:0/3964263040 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0fa0159d40 msgr2=0x7f0fa00cb260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:02.746 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.734+0000 7f0fa5274640 1 --2- 172.21.15.16:0/3964263040 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0fa0159d40 0x7f0fa00cb260 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f0f88009e50 tx=0x7f0f8801c980 comp rx=0 tx=0).stop 2023-05-22T15:27:02.746 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.734+0000 7f0fa5274640 1 -- 172.21.15.16:0/3964263040 shutdown_connections 2023-05-22T15:27:02.746 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.734+0000 7f0fa5274640 1 --2- 172.21.15.16:0/3964263040 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0fa006b6c0 0x7f0fa006bb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:02.746 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.734+0000 7f0fa5274640 1 --2- 172.21.15.16:0/3964263040 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0fa0159d40 0x7f0fa00cb260 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:02.746 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.734+0000 7f0fa5274640 1 -- 172.21.15.16:0/3964263040 >> 172.21.15.16:0/3964263040 conn(0x7f0fa0069720 msgr2=0x7f0fa0069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:02.746 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.734+0000 7f0fa5274640 1 -- 172.21.15.16:0/3964263040 shutdown_connections 2023-05-22T15:27:02.746 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.734+0000 7f0fa5274640 1 -- 172.21.15.16:0/3964263040 wait complete. 2023-05-22T15:27:02.748 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.738+0000 7f0fa5274640 1 Processor -- start 2023-05-22T15:27:02.748 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.738+0000 7f0fa5274640 1 -- start start 2023-05-22T15:27:02.748 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.738+0000 7f0fa5274640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0fa006b6c0 0x7f0fa0158560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:02.748 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.738+0000 7f0fa5274640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0fa0159970 0x7f0fa0158aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:02.748 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.738+0000 7f0fa5274640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0fa0159d40 0x7f0fa00c1d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:02.748 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.738+0000 7f0fa5274640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fa00c3680 con 0x7f0fa0159d40 2023-05-22T15:27:02.749 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.738+0000 7f0fa5274640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fa00c2a70 con 0x7f0fa0159970 2023-05-22T15:27:02.749 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.738+0000 7f0fa5274640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0fa00c2d50 con 0x7f0fa006b6c0 2023-05-22T15:27:02.749 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.738+0000 7f0f9f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0fa0159d40 0x7f0fa00c1d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:02.749 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.738+0000 7f0f9f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0fa0159d40 0x7f0fa00c1d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57532/0 (socket says 172.21.15.16:57532) 2023-05-22T15:27:02.749 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.738+0000 7f0f9f7fe640 1 -- 172.21.15.16:0/2873861259 learned_addr learned my addr 172.21.15.16:0/2873861259 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:02.749 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.738+0000 7f0f9f7fe640 1 -- 172.21.15.16:0/2873861259 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0fa006b6c0 msgr2=0x7f0fa0158560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:02.749 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.738+0000 7f0f9f7fe640 1 --2- 172.21.15.16:0/2873861259 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0fa006b6c0 0x7f0fa0158560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:02.750 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.738+0000 7f0f9f7fe640 1 -- 172.21.15.16:0/2873861259 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0fa0159970 msgr2=0x7f0fa0158aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:02.750 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.738+0000 7f0f9f7fe640 1 --2- 172.21.15.16:0/2873861259 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0fa0159970 0x7f0fa0158aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:02.750 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.738+0000 7f0f9f7fe640 1 -- 172.21.15.16:0/2873861259 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f88009b30 con 0x7f0fa0159d40 2023-05-22T15:27:02.750 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.738+0000 7f0f9f7fe640 1 --2- 172.21.15.16:0/2873861259 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0fa0159d40 0x7f0fa00c1d00 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f0f9000d800 tx=0x7f0f9000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:02.750 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.738+0000 7f0f977fe640 1 -- 172.21.15.16:0/2873861259 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f9000b6c0 con 0x7f0fa0159d40 2023-05-22T15:27:02.750 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.738+0000 7f0f977fe640 1 -- 172.21.15.16:0/2873861259 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f9000b820 con 0x7f0fa0159d40 2023-05-22T15:27:02.750 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.738+0000 7f0fa5274640 1 -- 172.21.15.16:0/2873861259 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0fa00cb7a0 con 0x7f0fa0159d40 2023-05-22T15:27:02.751 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.738+0000 7f0f977fe640 1 -- 172.21.15.16:0/2873861259 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f9000bbd0 con 0x7f0fa0159d40 2023-05-22T15:27:02.751 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.738+0000 7f0fa5274640 1 -- 172.21.15.16:0/2873861259 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0fa0071700 con 0x7f0fa0159d40 2023-05-22T15:27:02.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.738+0000 7f0fa5274640 1 -- 172.21.15.16:0/2873861259 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0fa00cb930 con 0x7f0fa0159d40 2023-05-22T15:27:02.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.738+0000 7f0f977fe640 1 -- 172.21.15.16:0/2873861259 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7f0f9000bd30 con 0x7f0fa0159d40 2023-05-22T15:27:02.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.738+0000 7f0f977fe640 1 --2- 172.21.15.16:0/2873861259 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f0f800653b0 0x7f0f80067860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:02.753 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.738+0000 7f0f977fe640 1 -- 172.21.15.16:0/2873861259 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f0f90083090 con 0x7f0fa0159d40 2023-05-22T15:27:02.753 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.738+0000 7f0f9effd640 1 --2- 172.21.15.16:0/2873861259 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f0f800653b0 0x7f0f80067860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:02.753 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.742+0000 7f0f9effd640 1 --2- 172.21.15.16:0/2873861259 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f0f800653b0 0x7f0f80067860 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f0f98000f70 tx=0x7f0f980079c0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:02.753 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.742+0000 7f0f977fe640 1 -- 172.21.15.16:0/2873861259 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f0f90050a20 con 0x7f0fa0159d40 2023-05-22T15:27:02.901 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.890+0000 7f0fa5274640 1 -- 172.21.15.16:0/2873861259 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f0fa004dbe0 con 0x7f0fa0159d40 2023-05-22T15:27:02.902 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.890+0000 7f0f977fe640 1 -- 172.21.15.16:0/2873861259 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) v1 ==== 100+0+11 (secure 0 0 0) 0x7f0f90054040 con 0x7f0fa0159d40 2023-05-22T15:27:02.902 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 64 2023-05-22T15:27:02.904 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.894+0000 7f0fa5274640 1 -- 172.21.15.16:0/2873861259 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f0f800653b0 msgr2=0x7f0f80067860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:02.904 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.894+0000 7f0fa5274640 1 --2- 172.21.15.16:0/2873861259 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f0f800653b0 0x7f0f80067860 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f0f98000f70 tx=0x7f0f980079c0 comp rx=0 tx=0).stop 2023-05-22T15:27:02.904 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.894+0000 7f0fa5274640 1 -- 172.21.15.16:0/2873861259 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0fa0159d40 msgr2=0x7f0fa00c1d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:02.904 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.894+0000 7f0fa5274640 1 --2- 172.21.15.16:0/2873861259 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0fa0159d40 0x7f0fa00c1d00 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f0f9000d800 tx=0x7f0f9000dcc0 comp rx=0 tx=0).stop 2023-05-22T15:27:02.904 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.894+0000 7f0fa5274640 1 -- 172.21.15.16:0/2873861259 shutdown_connections 2023-05-22T15:27:02.905 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.894+0000 7f0fa5274640 1 --2- 172.21.15.16:0/2873861259 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0fa0159d40 0x7f0fa00c1d00 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:02.905 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.894+0000 7f0fa5274640 1 --2- 172.21.15.16:0/2873861259 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0fa0159970 0x7f0fa0158aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:02.905 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.894+0000 7f0fa5274640 1 --2- 172.21.15.16:0/2873861259 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f0f800653b0 0x7f0f80067860 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:02.905 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.894+0000 7f0fa5274640 1 --2- 172.21.15.16:0/2873861259 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0fa006b6c0 0x7f0fa0158560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:02.905 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.894+0000 7f0fa5274640 1 -- 172.21.15.16:0/2873861259 >> 172.21.15.16:0/2873861259 conn(0x7f0fa0069720 msgr2=0x7f0fa015ade0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:02.905 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.894+0000 7f0fa5274640 1 -- 172.21.15.16:0/2873861259 shutdown_connections 2023-05-22T15:27:02.905 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:02.894+0000 7f0fa5274640 1 -- 172.21.15.16:0/2873861259 wait complete. 2023-05-22T15:27:02.919 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-05-22T15:27:03.035 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.026+0000 7fc030949640 1 Processor -- start 2023-05-22T15:27:03.036 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.026+0000 7fc030949640 1 -- start start 2023-05-22T15:27:03.037 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.026+0000 7fc030949640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc02c06bdd0 0x7fc02c0c8af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:03.037 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.026+0000 7fc030949640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc02c0cb730 0x7fc02c0c9030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:03.037 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.026+0000 7fc030949640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc02c0c8660 con 0x7fc02c15bee0 2023-05-22T15:27:03.037 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.026+0000 7fc030949640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc02c0ce4a0 con 0x7fc02c06bdd0 2023-05-22T15:27:03.038 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.026+0000 7fc030949640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc02c0ce780 con 0x7fc02c0cb730 2023-05-22T15:27:03.038 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.026+0000 7fc029d9b640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc02c06bdd0 0x7fc02c0c8af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:03.038 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.026+0000 7fc02a59c640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fc02c15bee0 0x7fc02c06b6c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:50978/0 (socket says 172.21.15.16:50978) 2023-05-22T15:27:03.038 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.026+0000 7fc02a59c640 1 -- 172.21.15.16:0/654729703 learned_addr learned my addr 172.21.15.16:0/654729703 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:27:03.038 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.026+0000 7fc02ad9d640 1 --2- 172.21.15.16:0/654729703 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc02c0cb730 0x7fc02c0c9030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:03.038 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.026+0000 7fc029d9b640 1 -- 172.21.15.16:0/654729703 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc02c0cb730 msgr2=0x7fc02c0c9030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:03.038 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.026+0000 7fc029d9b640 1 --2- 172.21.15.16:0/654729703 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc02c0cb730 0x7fc02c0c9030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:03.039 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.026+0000 7fc029d9b640 1 -- 172.21.15.16:0/654729703 >> v1:172.21.15.16:6789/0 conn(0x7fc02c15bee0 legacy=0x7fc02c06b6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:03.039 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.026+0000 7fc029d9b640 1 -- 172.21.15.16:0/654729703 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc02c0cea60 con 0x7fc02c06bdd0 2023-05-22T15:27:03.039 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.026+0000 7fc02ad9d640 1 --2- 172.21.15.16:0/654729703 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc02c0cb730 0x7fc02c0c9030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:27:03.040 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.030+0000 7fc029d9b640 1 --2- 172.21.15.16:0/654729703 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc02c06bdd0 0x7fc02c0c8af0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fc0200097c0 tx=0x7fc02001c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c0a45667c7c339a server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:03.040 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.030+0000 7fc02959a640 1 -- 172.21.15.16:0/654729703 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0200ab670 con 0x7fc02c06bdd0 2023-05-22T15:27:03.041 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.030+0000 7fc02959a640 1 -- 172.21.15.16:0/654729703 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0200ab7d0 con 0x7fc02c06bdd0 2023-05-22T15:27:03.041 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.030+0000 7fc02959a640 1 -- 172.21.15.16:0/654729703 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0200abb60 con 0x7fc02c06bdd0 2023-05-22T15:27:03.041 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.030+0000 7fc030949640 1 -- 172.21.15.16:0/654729703 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc02c06bdd0 msgr2=0x7fc02c0c8af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:03.041 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.030+0000 7fc030949640 1 --2- 172.21.15.16:0/654729703 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc02c06bdd0 0x7fc02c0c8af0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fc0200097c0 tx=0x7fc02001c450 comp rx=0 tx=0).stop 2023-05-22T15:27:03.041 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.030+0000 7fc030949640 1 -- 172.21.15.16:0/654729703 shutdown_connections 2023-05-22T15:27:03.041 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.030+0000 7fc030949640 1 --2- 172.21.15.16:0/654729703 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc02c0cb730 0x7fc02c0c9030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:03.042 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.030+0000 7fc030949640 1 --2- 172.21.15.16:0/654729703 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc02c06bdd0 0x7fc02c0c8af0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:03.042 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.030+0000 7fc030949640 1 -- 172.21.15.16:0/654729703 >> 172.21.15.16:0/654729703 conn(0x7fc02c069720 msgr2=0x7fc02c069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:03.042 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.030+0000 7fc030949640 1 -- 172.21.15.16:0/654729703 shutdown_connections 2023-05-22T15:27:03.042 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.030+0000 7fc030949640 1 -- 172.21.15.16:0/654729703 wait complete. 2023-05-22T15:27:03.045 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.030+0000 7fc030949640 1 Processor -- start 2023-05-22T15:27:03.045 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.030+0000 7fc030949640 1 -- start start 2023-05-22T15:27:03.045 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.034+0000 7fc030949640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc02c06bdd0 0x7fc02c150790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:03.048 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.034+0000 7fc030949640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc02c0cb730 0x7fc02c153780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:03.048 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.034+0000 7fc030949640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc02c15bee0 0x7fc02c153cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:03.049 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.034+0000 7fc030949640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc02c14fef0 con 0x7fc02c06bdd0 2023-05-22T15:27:03.050 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.034+0000 7fc030949640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc02c14f2e0 con 0x7fc02c0cb730 2023-05-22T15:27:03.050 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.034+0000 7fc030949640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc02c14f5c0 con 0x7fc02c15bee0 2023-05-22T15:27:03.050 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.034+0000 7fc02a59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc02c06bdd0 0x7fc02c150790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:03.051 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.034+0000 7fc02a59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc02c06bdd0 0x7fc02c150790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57554/0 (socket says 172.21.15.16:57554) 2023-05-22T15:27:03.051 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.034+0000 7fc02a59c640 1 -- 172.21.15.16:0/1352743324 learned_addr learned my addr 172.21.15.16:0/1352743324 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:03.051 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.034+0000 7fc02ad9d640 1 --2- 172.21.15.16:0/1352743324 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc02c15bee0 0x7fc02c153cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:03.051 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.034+0000 7fc029d9b640 1 --2- 172.21.15.16:0/1352743324 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc02c0cb730 0x7fc02c153780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:03.051 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.034+0000 7fc02a59c640 1 -- 172.21.15.16:0/1352743324 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc02c15bee0 msgr2=0x7fc02c153cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:03.051 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.034+0000 7fc02a59c640 1 --2- 172.21.15.16:0/1352743324 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc02c15bee0 0x7fc02c153cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:03.051 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.034+0000 7fc02a59c640 1 -- 172.21.15.16:0/1352743324 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc02c0cb730 msgr2=0x7fc02c153780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:03.052 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.034+0000 7fc02a59c640 1 --2- 172.21.15.16:0/1352743324 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc02c0cb730 0x7fc02c153780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:03.052 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.034+0000 7fc02a59c640 1 -- 172.21.15.16:0/1352743324 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc020009420 con 0x7fc02c06bdd0 2023-05-22T15:27:03.052 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.034+0000 7fc02a59c640 1 --2- 172.21.15.16:0/1352743324 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc02c06bdd0 0x7fc02c150790 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fc01c00c370 tx=0x7fc01c00c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:03.052 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.034+0000 7fc01affd640 1 -- 172.21.15.16:0/1352743324 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc01c005cf0 con 0x7fc02c06bdd0 2023-05-22T15:27:03.052 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.034+0000 7fc01affd640 1 -- 172.21.15.16:0/1352743324 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc01c005e50 con 0x7fc02c06bdd0 2023-05-22T15:27:03.052 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.034+0000 7fc030949640 1 -- 172.21.15.16:0/1352743324 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc02c168d40 con 0x7fc02c06bdd0 2023-05-22T15:27:03.052 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.034+0000 7fc01affd640 1 -- 172.21.15.16:0/1352743324 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc01c0072f0 con 0x7fc02c06bdd0 2023-05-22T15:27:03.053 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.034+0000 7fc030949640 1 -- 172.21.15.16:0/1352743324 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc02c168bb0 con 0x7fc02c06bdd0 2023-05-22T15:27:03.053 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.034+0000 7fc030949640 1 -- 172.21.15.16:0/1352743324 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc02c168ed0 con 0x7fc02c06bdd0 2023-05-22T15:27:03.053 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.034+0000 7fc01affd640 1 -- 172.21.15.16:0/1352743324 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7fc01c007450 con 0x7fc02c06bdd0 2023-05-22T15:27:03.053 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.034+0000 7fc01affd640 1 --2- 172.21.15.16:0/1352743324 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fc00c0653b0 0x7fc00c067860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:03.053 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.034+0000 7fc01affd640 1 -- 172.21.15.16:0/1352743324 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6813+0+0 (secure 0 0 0) 0x7fc01c0820b0 con 0x7fc02c06bdd0 2023-05-22T15:27:03.053 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.034+0000 7fc029d9b640 1 --2- 172.21.15.16:0/1352743324 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fc00c0653b0 0x7fc00c067860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:03.054 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.034+0000 7fc029d9b640 1 --2- 172.21.15.16:0/1352743324 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fc00c0653b0 0x7fc00c067860 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fc0200af000 tx=0x7fc020008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:03.054 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.038+0000 7fc01affd640 1 -- 172.21.15.16:0/1352743324 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fc01c04fa40 con 0x7fc02c06bdd0 2023-05-22T15:27:03.194 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.182+0000 7fc030949640 1 -- 172.21.15.16:0/1352743324 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7fc02c04dbe0 con 0x7fc02c06bdd0 2023-05-22T15:27:03.195 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.186+0000 7fc01affd640 1 -- 172.21.15.16:0/1352743324 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) v1 ==== 103+0+11 (secure 0 0 0) 0x7fc01c017020 con 0x7fc02c06bdd0 2023-05-22T15:27:03.195 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2023-05-22T15:27:03.197 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.186+0000 7fc030949640 1 -- 172.21.15.16:0/1352743324 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fc00c0653b0 msgr2=0x7fc00c067860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:03.197 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.186+0000 7fc030949640 1 --2- 172.21.15.16:0/1352743324 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fc00c0653b0 0x7fc00c067860 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fc0200af000 tx=0x7fc020008040 comp rx=0 tx=0).stop 2023-05-22T15:27:03.197 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.186+0000 7fc030949640 1 -- 172.21.15.16:0/1352743324 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc02c06bdd0 msgr2=0x7fc02c150790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:03.197 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.186+0000 7fc030949640 1 --2- 172.21.15.16:0/1352743324 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc02c06bdd0 0x7fc02c150790 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fc01c00c370 tx=0x7fc01c00c830 comp rx=0 tx=0).stop 2023-05-22T15:27:03.197 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.186+0000 7fc030949640 1 -- 172.21.15.16:0/1352743324 shutdown_connections 2023-05-22T15:27:03.197 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.186+0000 7fc030949640 1 --2- 172.21.15.16:0/1352743324 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fc00c0653b0 0x7fc00c067860 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:03.198 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.186+0000 7fc030949640 1 --2- 172.21.15.16:0/1352743324 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc02c15bee0 0x7fc02c153cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:03.198 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.186+0000 7fc030949640 1 --2- 172.21.15.16:0/1352743324 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc02c0cb730 0x7fc02c153780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:03.198 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.186+0000 7fc030949640 1 --2- 172.21.15.16:0/1352743324 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc02c06bdd0 0x7fc02c150790 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:03.198 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.186+0000 7fc030949640 1 -- 172.21.15.16:0/1352743324 >> 172.21.15.16:0/1352743324 conn(0x7fc02c069720 msgr2=0x7fc02c0aeeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:03.198 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.186+0000 7fc030949640 1 -- 172.21.15.16:0/1352743324 shutdown_connections 2023-05-22T15:27:03.198 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.186+0000 7fc030949640 1 -- 172.21.15.16:0/1352743324 wait complete. 2023-05-22T15:27:03.212 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-05-22T15:27:03.338 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.326+0000 7f402d5bc640 1 Processor -- start 2023-05-22T15:27:03.338 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.326+0000 7f402d5bc640 1 -- start start 2023-05-22T15:27:03.339 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.330+0000 7f402d5bc640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f40280c9b40 0x7f402815e460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:03.339 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.330+0000 7f402d5bc640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f4028070df0 0x7f4028071210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:03.339 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.330+0000 7f402d5bc640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40280b0650 con 0x7f40280ca4e0 2023-05-22T15:27:03.339 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.330+0000 7f402d5bc640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40280b0370 con 0x7f4028070df0 2023-05-22T15:27:03.339 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.330+0000 7f402d5bc640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f402815d980 con 0x7f40280c9b40 2023-05-22T15:27:03.341 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.330+0000 7f40277fe640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f4028070df0 0x7f4028071210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:03.341 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.330+0000 7f40267fc640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f40280c9b40 0x7f402815e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:03.341 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.330+0000 7f4026ffd640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f40280ca4e0 0x7f40280afb40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.16:35894/0 (socket says 172.21.15.16:35894) 2023-05-22T15:27:03.341 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.330+0000 7f4026ffd640 1 -- 172.21.15.16:0/1247144675 learned_addr learned my addr 172.21.15.16:0/1247144675 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:27:03.341 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.330+0000 7f40277fe640 1 -- 172.21.15.16:0/1247144675 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f40280c9b40 msgr2=0x7f402815e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:03.341 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.330+0000 7f40277fe640 1 --2- 172.21.15.16:0/1247144675 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f40280c9b40 0x7f402815e460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:03.341 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.330+0000 7f40277fe640 1 -- 172.21.15.16:0/1247144675 >> v1:172.21.15.62:6789/0 conn(0x7f40280ca4e0 legacy=0x7f40280afb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:03.342 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.330+0000 7f40277fe640 1 -- 172.21.15.16:0/1247144675 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f402815dc60 con 0x7f4028070df0 2023-05-22T15:27:03.342 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.334+0000 7f40277fe640 1 --2- 172.21.15.16:0/1247144675 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4028070df0 0x7f4028071210 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f40180097c0 tx=0x7f401801c3c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1eb3e8716ff3a674 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:03.343 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.334+0000 7f4025ffb640 1 -- 172.21.15.16:0/1247144675 <== mon.1 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40180ab600 con 0x7f4028070df0 2023-05-22T15:27:03.343 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.334+0000 7f4025ffb640 1 -- 172.21.15.16:0/1247144675 <== mon.1 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40180a7050 con 0x7f4028070df0 2023-05-22T15:27:03.343 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.334+0000 7f4025ffb640 1 -- 172.21.15.16:0/1247144675 <== mon.1 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40180ab980 con 0x7f4028070df0 2023-05-22T15:27:03.344 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.334+0000 7f402d5bc640 1 -- 172.21.15.16:0/1247144675 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4028070df0 msgr2=0x7f4028071210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:03.344 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.334+0000 7f402d5bc640 1 --2- 172.21.15.16:0/1247144675 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4028070df0 0x7f4028071210 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f40180097c0 tx=0x7f401801c3c0 comp rx=0 tx=0).stop 2023-05-22T15:27:03.344 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.334+0000 7f402d5bc640 1 -- 172.21.15.16:0/1247144675 shutdown_connections 2023-05-22T15:27:03.344 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.334+0000 7f402d5bc640 1 --2- 172.21.15.16:0/1247144675 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4028070df0 0x7f4028071210 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:03.345 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.334+0000 7f402d5bc640 1 --2- 172.21.15.16:0/1247144675 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f40280c9b40 0x7f402815e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:03.345 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.334+0000 7f402d5bc640 1 -- 172.21.15.16:0/1247144675 >> 172.21.15.16:0/1247144675 conn(0x7f4028069720 msgr2=0x7f4028069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:03.345 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.334+0000 7f402d5bc640 1 -- 172.21.15.16:0/1247144675 shutdown_connections 2023-05-22T15:27:03.345 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.334+0000 7f402d5bc640 1 -- 172.21.15.16:0/1247144675 wait complete. 2023-05-22T15:27:03.347 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.334+0000 7f402d5bc640 1 Processor -- start 2023-05-22T15:27:03.348 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.334+0000 7f402d5bc640 1 -- start start 2023-05-22T15:27:03.348 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.334+0000 7f402d5bc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4028070df0 0x7f40280c7a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:03.348 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.334+0000 7f402d5bc640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f40280c9b40 0x7f4028154ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:03.348 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.334+0000 7f402d5bc640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f40280ca4e0 0x7f40281f9270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:03.348 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.334+0000 7f402d5bc640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40280c7f90 con 0x7f4028070df0 2023-05-22T15:27:03.348 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.334+0000 7f402d5bc640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40281553e0 con 0x7f40280c9b40 2023-05-22T15:27:03.349 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.334+0000 7f402d5bc640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40281f97b0 con 0x7f40280ca4e0 2023-05-22T15:27:03.349 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.334+0000 7f4026ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4028070df0 0x7f40280c7a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:03.349 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.334+0000 7f4026ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4028070df0 0x7f40280c7a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57566/0 (socket says 172.21.15.16:57566) 2023-05-22T15:27:03.349 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.334+0000 7f4026ffd640 1 -- 172.21.15.16:0/2575194517 learned_addr learned my addr 172.21.15.16:0/2575194517 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:03.349 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.334+0000 7f40277fe640 1 --2- 172.21.15.16:0/2575194517 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f40280ca4e0 0x7f40281f9270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:03.349 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.334+0000 7f40267fc640 1 --2- 172.21.15.16:0/2575194517 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f40280c9b40 0x7f4028154ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:03.349 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.338+0000 7f4026ffd640 1 -- 172.21.15.16:0/2575194517 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f40280ca4e0 msgr2=0x7f40281f9270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:03.350 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.338+0000 7f4026ffd640 1 --2- 172.21.15.16:0/2575194517 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f40280ca4e0 0x7f40281f9270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:03.350 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.338+0000 7f4026ffd640 1 -- 172.21.15.16:0/2575194517 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f40280c9b40 msgr2=0x7f4028154ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:03.351 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.338+0000 7f4026ffd640 1 --2- 172.21.15.16:0/2575194517 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f40280c9b40 0x7f4028154ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:03.351 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.338+0000 7f4026ffd640 1 -- 172.21.15.16:0/2575194517 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4018009420 con 0x7f4028070df0 2023-05-22T15:27:03.351 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.338+0000 7f40267fc640 1 --2- 172.21.15.16:0/2575194517 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f40280c9b40 0x7f4028154ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:27:03.351 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.338+0000 7f4026ffd640 1 --2- 172.21.15.16:0/2575194517 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4028070df0 0x7f40280c7a50 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f402000cbc0 tx=0x7f402000a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:03.351 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.338+0000 7f401f7fe640 1 -- 172.21.15.16:0/2575194517 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f402000ac40 con 0x7f4028070df0 2023-05-22T15:27:03.351 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.338+0000 7f401f7fe640 1 -- 172.21.15.16:0/2575194517 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f402000ada0 con 0x7f4028070df0 2023-05-22T15:27:03.352 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.338+0000 7f402d5bc640 1 -- 172.21.15.16:0/2575194517 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40280c8e80 con 0x7f4028070df0 2023-05-22T15:27:03.352 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.338+0000 7f401f7fe640 1 -- 172.21.15.16:0/2575194517 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4020012600 con 0x7f4028070df0 2023-05-22T15:27:03.352 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.338+0000 7f402d5bc640 1 -- 172.21.15.16:0/2575194517 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40280ca8b0 con 0x7f4028070df0 2023-05-22T15:27:03.352 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.338+0000 7f402d5bc640 1 -- 172.21.15.16:0/2575194517 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40280c9010 con 0x7f4028070df0 2023-05-22T15:27:03.352 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.338+0000 7f401f7fe640 1 -- 172.21.15.16:0/2575194517 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7f4020007070 con 0x7f4028070df0 2023-05-22T15:27:03.353 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.338+0000 7f401f7fe640 1 --2- 172.21.15.16:0/2575194517 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f40080653b0 0x7f4008067860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:03.353 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.338+0000 7f401f7fe640 1 -- 172.21.15.16:0/2575194517 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f4020082970 con 0x7f4028070df0 2023-05-22T15:27:03.354 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.338+0000 7f40267fc640 1 --2- 172.21.15.16:0/2575194517 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f40080653b0 0x7f4008067860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:03.354 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.338+0000 7f40267fc640 1 --2- 172.21.15.16:0/2575194517 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f40080653b0 0x7f4008067860 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f40280b2ab0 tx=0x7f4010008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:03.354 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.342+0000 7f401f7fe640 1 -- 172.21.15.16:0/2575194517 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f402004a620 con 0x7f4028070df0 2023-05-22T15:27:03.518 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.506+0000 7f402d5bc640 1 -- 172.21.15.16:0/2575194517 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f402804dbe0 con 0x7f4028070df0 2023-05-22T15:27:03.518 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.510+0000 7f401f7fe640 1 -- 172.21.15.16:0/2575194517 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v27) v1 ==== 80+0+0 (secure 0 0 0) 0x7f4020050350 con 0x7f4028070df0 2023-05-22T15:27:03.519 INFO:teuthology.orchestra.run.smithi016.stderr:listed 0 entries 2023-05-22T15:27:03.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.510+0000 7f402d5bc640 1 -- 172.21.15.16:0/2575194517 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f40080653b0 msgr2=0x7f4008067860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:03.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.510+0000 7f402d5bc640 1 --2- 172.21.15.16:0/2575194517 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f40080653b0 0x7f4008067860 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f40280b2ab0 tx=0x7f4010008040 comp rx=0 tx=0).stop 2023-05-22T15:27:03.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.510+0000 7f402d5bc640 1 -- 172.21.15.16:0/2575194517 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4028070df0 msgr2=0x7f40280c7a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:03.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.510+0000 7f402d5bc640 1 --2- 172.21.15.16:0/2575194517 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4028070df0 0x7f40280c7a50 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f402000cbc0 tx=0x7f402000a060 comp rx=0 tx=0).stop 2023-05-22T15:27:03.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.510+0000 7f402d5bc640 1 -- 172.21.15.16:0/2575194517 shutdown_connections 2023-05-22T15:27:03.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.510+0000 7f402d5bc640 1 --2- 172.21.15.16:0/2575194517 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f40080653b0 0x7f4008067860 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:03.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.510+0000 7f402d5bc640 1 --2- 172.21.15.16:0/2575194517 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f40280ca4e0 0x7f40281f9270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:03.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.510+0000 7f402d5bc640 1 --2- 172.21.15.16:0/2575194517 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f40280c9b40 0x7f4028154ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:03.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.510+0000 7f402d5bc640 1 --2- 172.21.15.16:0/2575194517 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4028070df0 0x7f40280c7a50 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:03.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.510+0000 7f402d5bc640 1 -- 172.21.15.16:0/2575194517 >> 172.21.15.16:0/2575194517 conn(0x7f4028069720 msgr2=0x7f40280aec00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:03.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.510+0000 7f402d5bc640 1 -- 172.21.15.16:0/2575194517 shutdown_connections 2023-05-22T15:27:03.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.510+0000 7f402d5bc640 1 -- 172.21.15.16:0/2575194517 wait complete. 2023-05-22T15:27:03.538 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi016.front.sepia.ceph.com... 2023-05-22T15:27:03.538 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:27:03.539 DEBUG:teuthology.orchestra.run.smithi016:> sudo fusermount -u /home/ubuntu/cephtest/mnt.admin 2023-05-22T15:27:03.564 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi016.stderr:ceph-fuse[43384]: fuse finished with error 0 and tester_r 0 2023-05-22T15:27:03.568 INFO:teuthology.orchestra.run:waiting for 300 2023-05-22T15:27:03.594 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi016.front.sepia.ceph.com 2023-05-22T15:27:03.595 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:27:03.595 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.admin) 2023-05-22T15:27:03.652 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-05-22T15:27:03.822 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.814+0000 7f145e5e2640 1 Processor -- start 2023-05-22T15:27:03.822 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.814+0000 7f145e5e2640 1 -- start start 2023-05-22T15:27:03.823 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.814+0000 7f145e5e2640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f14580ccab0 0x7f145815fe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:03.823 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.814+0000 7f145e5e2640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14580ccef0 con 0x7f14580d0410 2023-05-22T15:27:03.823 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.814+0000 7f145e5e2640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14580cd170 con 0x7f14580b7dd0 2023-05-22T15:27:03.824 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.814+0000 7f145e5e2640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14580d3360 con 0x7f14580ccab0 2023-05-22T15:27:03.824 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.814+0000 7f145cb7f640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f14580d0410 0x7f14580d0890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:50986/0 (socket says 172.21.15.16:50986) 2023-05-22T15:27:03.824 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.814+0000 7f145cb7f640 1 -- 172.21.15.16:0/1915858940 learned_addr learned my addr 172.21.15.16:0/1915858940 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:27:03.826 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.814+0000 7f144f7fe640 1 --2- 172.21.15.16:0/1915858940 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f14580ccab0 0x7f145815fe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:03.826 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.814+0000 7f14577fe640 1 -- 172.21.15.16:0/1915858940 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3273162394 0 0) 0x7f14580ccef0 con 0x7f14580d0410 2023-05-22T15:27:03.826 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.814+0000 7f14577fe640 1 -- 172.21.15.16:0/1915858940 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1444003530 con 0x7f14580d0410 2023-05-22T15:27:03.826 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.814+0000 7f14577fe640 1 -- 172.21.15.16:0/1915858940 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3612542994 0 0) 0x7f1444003530 con 0x7f14580d0410 2023-05-22T15:27:03.826 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.814+0000 7f14577fe640 1 -- 172.21.15.16:0/1915858940 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f14580ccab0 msgr2=0x7f145815fe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:03.827 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.814+0000 7f14577fe640 1 --2- 172.21.15.16:0/1915858940 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f14580ccab0 0x7f145815fe10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:03.827 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.814+0000 7f14577fe640 1 -- 172.21.15.16:0/1915858940 >> v1:172.21.15.62:6789/0 conn(0x7f14580b7dd0 legacy=0x7f145815f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:03.827 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.814+0000 7f14577fe640 1 -- 172.21.15.16:0/1915858940 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14580d2ac0 con 0x7f14580d0410 2023-05-22T15:27:03.827 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.814+0000 7f14577fe640 1 -- 172.21.15.16:0/1915858940 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7f1450001a00 con 0x7f14580d0410 2023-05-22T15:27:03.827 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.814+0000 7f14577fe640 1 -- 172.21.15.16:0/1915858940 >> v1:172.21.15.16:6789/0 conn(0x7f14580d0410 legacy=0x7f14580d0890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:03.827 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.814+0000 7f14577fe640 1 --2- 172.21.15.16:0/1915858940 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f14580ccab0 0x7f14440039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:03.827 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.814+0000 7f14577fe640 1 --2- 172.21.15.16:0/1915858940 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1444003f30 0x7f14440047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:03.828 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.814+0000 7f14577fe640 1 --2- 172.21.15.16:0/1915858940 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1444004d10 0x7f1444005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:03.828 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.814+0000 7f14577fe640 1 -- 172.21.15.16:0/1915858940 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14580d3360 con 0x7f1444004d10 2023-05-22T15:27:03.829 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.814+0000 7f14577fe640 1 -- 172.21.15.16:0/1915858940 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f144400ba70 con 0x7f1444003f30 2023-05-22T15:27:03.830 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.814+0000 7f14577fe640 1 -- 172.21.15.16:0/1915858940 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f144400bc30 con 0x7f14580ccab0 2023-05-22T15:27:03.830 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.814+0000 7f145cb7f640 1 --2- 172.21.15.16:0/1915858940 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1444004d10 0x7f1444005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:03.830 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.814+0000 7f144f7fe640 1 --2- 172.21.15.16:0/1915858940 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1444003f30 0x7f14440047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:03.830 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.814+0000 7f1457fff640 1 --2- 172.21.15.16:0/1915858940 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f14580ccab0 0x7f14440039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:03.831 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.814+0000 7f145cb7f640 1 -- 172.21.15.16:0/1915858940 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f14580ccab0 msgr2=0x7f14440039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:03.831 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.814+0000 7f145cb7f640 1 --2- 172.21.15.16:0/1915858940 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f14580ccab0 0x7f14440039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:03.831 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.814+0000 7f145cb7f640 1 -- 172.21.15.16:0/1915858940 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1444003f30 msgr2=0x7f14440047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:03.831 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.814+0000 7f145cb7f640 1 --2- 172.21.15.16:0/1915858940 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1444003f30 0x7f14440047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:03.831 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.814+0000 7f145cb7f640 1 -- 172.21.15.16:0/1915858940 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1444003530 con 0x7f1444004d10 2023-05-22T15:27:03.831 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.814+0000 7f144f7fe640 1 --2- 172.21.15.16:0/1915858940 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1444003f30 0x7f14440047d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:27:03.834 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.818+0000 7f145cb7f640 1 --2- 172.21.15.16:0/1915858940 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1444004d10 0x7f1444005180 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f145000a800 tx=0x7f145001d300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a6922336ee6a76 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:03.834 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.818+0000 7f14577fe640 1 -- 172.21.15.16:0/1915858940 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14500072d0 con 0x7f1444004d10 2023-05-22T15:27:03.834 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.818+0000 7f14577fe640 1 -- 172.21.15.16:0/1915858940 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14500a8070 con 0x7f1444004d10 2023-05-22T15:27:03.835 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.818+0000 7f14577fe640 1 -- 172.21.15.16:0/1915858940 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1450007690 con 0x7f1444004d10 2023-05-22T15:27:03.835 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.818+0000 7f145e5e2640 1 -- 172.21.15.16:0/1915858940 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1444004d10 msgr2=0x7f1444005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:03.835 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.818+0000 7f145e5e2640 1 --2- 172.21.15.16:0/1915858940 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1444004d10 0x7f1444005180 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f145000a800 tx=0x7f145001d300 comp rx=0 tx=0).stop 2023-05-22T15:27:03.835 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.818+0000 7f145cb7f640 1 -- 172.21.15.16:0/1915858940 reap_dead start 2023-05-22T15:27:03.835 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.818+0000 7f145e5e2640 1 -- 172.21.15.16:0/1915858940 shutdown_connections 2023-05-22T15:27:03.835 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.818+0000 7f145e5e2640 1 -- 172.21.15.16:0/1915858940 >> 172.21.15.16:0/1915858940 conn(0x7f14580655c0 msgr2=0x7f1458070e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:03.835 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.818+0000 7f145e5e2640 1 -- 172.21.15.16:0/1915858940 shutdown_connections 2023-05-22T15:27:03.835 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.818+0000 7f145e5e2640 1 -- 172.21.15.16:0/1915858940 wait complete. 2023-05-22T15:27:03.836 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.818+0000 7f145e5e2640 1 Processor -- start 2023-05-22T15:27:03.836 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.818+0000 7f145e5e2640 1 -- start start 2023-05-22T15:27:03.836 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.818+0000 7f145e5e2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1444004d10 0x7f14580c9530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:03.836 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.818+0000 7f145e5e2640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f145815ca90 0x7f145814fe10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:03.836 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.818+0000 7f145e5e2640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f14580b23c0 0x7f14580b2830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:03.836 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.818+0000 7f145e5e2640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14580b2d70 con 0x7f1444004d10 2023-05-22T15:27:03.837 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.818+0000 7f145e5e2640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14580c9d80 con 0x7f14580b23c0 2023-05-22T15:27:03.837 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.818+0000 7f145e5e2640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14580ca060 con 0x7f145815ca90 2023-05-22T15:27:03.837 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.818+0000 7f1457fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1444004d10 0x7f14580c9530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:03.837 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.818+0000 7f1457fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1444004d10 0x7f14580c9530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57584/0 (socket says 172.21.15.16:57584) 2023-05-22T15:27:03.837 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.818+0000 7f1457fff640 1 -- 172.21.15.16:0/2325470551 learned_addr learned my addr 172.21.15.16:0/2325470551 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:03.837 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.818+0000 7f145cb7f640 1 --2- 172.21.15.16:0/2325470551 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f14580b23c0 0x7f14580b2830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:03.837 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.818+0000 7f144f7fe640 1 --2- 172.21.15.16:0/2325470551 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f145815ca90 0x7f145814fe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:03.837 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.818+0000 7f1457fff640 1 -- 172.21.15.16:0/2325470551 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f145815ca90 msgr2=0x7f145814fe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:03.838 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.818+0000 7f1457fff640 1 --2- 172.21.15.16:0/2325470551 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f145815ca90 0x7f145814fe10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:03.838 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.822+0000 7f1457fff640 1 -- 172.21.15.16:0/2325470551 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f14580b23c0 msgr2=0x7f14580b2830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:03.838 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.822+0000 7f1457fff640 1 --2- 172.21.15.16:0/2325470551 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f14580b23c0 0x7f14580b2830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:03.838 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.822+0000 7f1457fff640 1 -- 172.21.15.16:0/2325470551 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14580d3080 con 0x7f1444004d10 2023-05-22T15:27:03.838 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.822+0000 7f1457fff640 1 --2- 172.21.15.16:0/2325470551 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1444004d10 0x7f14580c9530 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f144800e4b0 tx=0x7f144800e970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:03.838 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.822+0000 7f14557fa640 1 -- 172.21.15.16:0/2325470551 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14480052e0 con 0x7f1444004d10 2023-05-22T15:27:03.838 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.822+0000 7f14557fa640 1 -- 172.21.15.16:0/2325470551 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1448005440 con 0x7f1444004d10 2023-05-22T15:27:03.838 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.822+0000 7f14557fa640 1 -- 172.21.15.16:0/2325470551 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1448003820 con 0x7f1444004d10 2023-05-22T15:27:03.839 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.822+0000 7f145e5e2640 1 -- 172.21.15.16:0/2325470551 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14580d2da0 con 0x7f1444004d10 2023-05-22T15:27:03.839 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.822+0000 7f145e5e2640 1 -- 172.21.15.16:0/2325470551 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14580cf730 con 0x7f1444004d10 2023-05-22T15:27:03.839 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.822+0000 7f145e5e2640 1 -- 172.21.15.16:0/2325470551 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14580d1e80 con 0x7f1444004d10 2023-05-22T15:27:03.840 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.822+0000 7f14557fa640 1 -- 172.21.15.16:0/2325470551 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7f1448003980 con 0x7f1444004d10 2023-05-22T15:27:03.840 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.822+0000 7f14557fa640 1 --2- 172.21.15.16:0/2325470551 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f14380653b0 0x7f1438067860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:03.840 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.822+0000 7f14557fa640 1 -- 172.21.15.16:0/2325470551 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f14480825e0 con 0x7f1444004d10 2023-05-22T15:27:03.840 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.822+0000 7f144f7fe640 1 --2- 172.21.15.16:0/2325470551 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f14380653b0 0x7f1438067860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:03.840 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.822+0000 7f144f7fe640 1 --2- 172.21.15.16:0/2325470551 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f14380653b0 0x7f1438067860 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f14580caa90 tx=0x7f1440008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:03.841 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.826+0000 7f14557fa640 1 -- 172.21.15.16:0/2325470551 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f144804fed0 con 0x7f1444004d10 2023-05-22T15:27:03.988 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.978+0000 7f145e5e2640 1 -- 172.21.15.16:0/2325470551 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f145804dbe0 con 0x7f1444004d10 2023-05-22T15:27:03.990 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.978+0000 7f14557fa640 1 -- 172.21.15.16:0/2325470551 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) v1 ==== 74+0+19604 (secure 0 0 0) 0x7f14480534f0 con 0x7f1444004d10 2023-05-22T15:27:03.990 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:27:03.991 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":27,"fsid":"b2719834-7847-4d98-90a6-f65d57f85c49","created":"2023-05-22T15:26:25.259314+0000","modified":"2023-05-22T15:26:53.971378+0000","last_up_change":"2023-05-22T15:26:36.448524+0000","last_in_change":"2023-05-22T15:26:27.675668+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-05-22T15:26:38.019781+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-05-22T15:26:43.570514+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-05-22T15:26:47.879159+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-05-22T15:26:48.522130+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data_ec","create_time":"2023-05-22T15:26:51.616142+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"9e7ff854-b25f-4cbb-a5c2-1799d9c53d2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":37021},{"type":"v1","addr":"172.21.15.16:6825","nonce":37021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":37021},{"type":"v1","addr":"172.21.15.16:6827","nonce":37021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":37021},{"type":"v1","addr":"172.21.15.16:6831","nonce":37021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":37021},{"type":"v1","addr":"172.21.15.16:6829","nonce":37021}]},"public_addr":"172.21.15.16:6825/37021","cluster_addr":"172.21.15.16:6827/37021","heartbeat_back_addr":"172.21.15.16:6831/37021","heartbeat_front_addr":"172.21.15.16:6829/37021","state":["exists","up"]},{"osd":1,"uuid":"4702fbc9-a927-4565-aca5-41dc69cc8264","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":37008},{"type":"v1","addr":"172.21.15.16:6801","nonce":37008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":37008},{"type":"v1","addr":"172.21.15.16:6803","nonce":37008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":37008},{"type":"v1","addr":"172.21.15.16:6807","nonce":37008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":37008},{"type":"v1","addr":"172.21.15.16:6805","nonce":37008}]},"public_addr":"172.21.15.16:6801/37008","cluster_addr":"172.21.15.16:6803/37008","heartbeat_back_addr":"172.21.15.16:6807/37008","heartbeat_front_addr":"172.21.15.16:6805/37008","state":["exists","up"]},{"osd":2,"uuid":"1e7bba46-4eba-47ef-bb81-5e2d371638ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":37009},{"type":"v1","addr":"172.21.15.16:6809","nonce":37009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":37009},{"type":"v1","addr":"172.21.15.16:6811","nonce":37009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":37009},{"type":"v1","addr":"172.21.15.16:6815","nonce":37009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":37009},{"type":"v1","addr":"172.21.15.16:6813","nonce":37009}]},"public_addr":"172.21.15.16:6809/37009","cluster_addr":"172.21.15.16:6811/37009","heartbeat_back_addr":"172.21.15.16:6815/37009","heartbeat_front_addr":"172.21.15.16:6813/37009","state":["exists","up"]},{"osd":3,"uuid":"383cacc0-5a88-4904-990e-498ca59ef48a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":37020},{"type":"v1","addr":"172.21.15.16:6817","nonce":37020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":37020},{"type":"v1","addr":"172.21.15.16:6819","nonce":37020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":37020},{"type":"v1","addr":"172.21.15.16:6823","nonce":37020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":37020},{"type":"v1","addr":"172.21.15.16:6821","nonce":37020}]},"public_addr":"172.21.15.16:6817/37020","cluster_addr":"172.21.15.16:6819/37020","heartbeat_back_addr":"172.21.15.16:6823/37020","heartbeat_front_addr":"172.21.15.16:6821/37020","state":["exists","up"]},{"osd":4,"uuid":"970500ef-95b7-42c1-b080-eed4fae7013d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":37353},{"type":"v1","addr":"172.21.15.62:6817","nonce":37353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6819","nonce":37353},{"type":"v1","addr":"172.21.15.62:6822","nonce":37353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37353},{"type":"v1","addr":"172.21.15.62:6831","nonce":37353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37353},{"type":"v1","addr":"172.21.15.62:6826","nonce":37353}]},"public_addr":"172.21.15.62:6817/37353","cluster_addr":"172.21.15.62:6822/37353","heartbeat_back_addr":"172.21.15.62:6831/37353","heartbeat_front_addr":"172.21.15.62:6826/37353","state":["exists","up"]},{"osd":5,"uuid":"022d67db-d4d8-445b-b1b9-8b446a00f5c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":37352},{"type":"v1","addr":"172.21.15.62:6801","nonce":37352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":37352},{"type":"v1","addr":"172.21.15.62:6803","nonce":37352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":37352},{"type":"v1","addr":"172.21.15.62:6807","nonce":37352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":37352},{"type":"v1","addr":"172.21.15.62:6805","nonce":37352}]},"public_addr":"172.21.15.62:6801/37352","cluster_addr":"172.21.15.62:6803/37352","heartbeat_back_addr":"172.21.15.62:6807/37352","heartbeat_front_addr":"172.21.15.62:6805/37352","state":["exists","up"]},{"osd":6,"uuid":"42bd372b-6688-4a1b-8407-e16e8c266e33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":37354},{"type":"v1","addr":"172.21.15.62:6820","nonce":37354}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":37354},{"type":"v1","addr":"172.21.15.62:6823","nonce":37354}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6829","nonce":37354},{"type":"v1","addr":"172.21.15.62:6830","nonce":37354}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6825","nonce":37354},{"type":"v1","addr":"172.21.15.62:6827","nonce":37354}]},"public_addr":"172.21.15.62:6820/37354","cluster_addr":"172.21.15.62:6823/37354","heartbeat_back_addr":"172.21.15.62:6830/37354","heartbeat_front_addr":"172.21.15.62:6827/37354","state":["exists","up"]},{"osd":7,"uuid":"ac70b2a3-f41b-4bf2-b227-7f2cbb55426d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37355},{"type":"v1","addr":"172.21.15.62:6809","nonce":37355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37355},{"type":"v1","addr":"172.21.15.62:6811","nonce":37355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":37355},{"type":"v1","addr":"172.21.15.62:6815","nonce":37355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":37355},{"type":"v1","addr":"172.21.15.62:6813","nonce":37355}]},"public_addr":"172.21.15.62:6809/37355","cluster_addr":"172.21.15.62:6811/37355","heartbeat_back_addr":"172.21.15.62:6815/37355","heartbeat_front_addr":"172.21.15.62:6813/37355","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.368896+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.107273+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.276888+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.287502+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.124791+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.251937+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.255679+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.232797+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-05-22T15:27:03.992 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.982+0000 7f145e5e2640 1 -- 172.21.15.16:0/2325470551 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f14380653b0 msgr2=0x7f1438067860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:03.993 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.982+0000 7f145e5e2640 1 --2- 172.21.15.16:0/2325470551 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f14380653b0 0x7f1438067860 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f14580caa90 tx=0x7f1440008040 comp rx=0 tx=0).stop 2023-05-22T15:27:03.993 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.982+0000 7f145e5e2640 1 -- 172.21.15.16:0/2325470551 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1444004d10 msgr2=0x7f14580c9530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:03.993 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.982+0000 7f145e5e2640 1 --2- 172.21.15.16:0/2325470551 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1444004d10 0x7f14580c9530 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f144800e4b0 tx=0x7f144800e970 comp rx=0 tx=0).stop 2023-05-22T15:27:03.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.982+0000 7f145e5e2640 1 -- 172.21.15.16:0/2325470551 shutdown_connections 2023-05-22T15:27:03.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.982+0000 7f145e5e2640 1 --2- 172.21.15.16:0/2325470551 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f14580b23c0 0x7f14580b2830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:03.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.982+0000 7f145e5e2640 1 --2- 172.21.15.16:0/2325470551 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f14380653b0 0x7f1438067860 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:03.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.982+0000 7f145e5e2640 1 --2- 172.21.15.16:0/2325470551 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f145815ca90 0x7f145814fe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:03.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.982+0000 7f145e5e2640 1 --2- 172.21.15.16:0/2325470551 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1444004d10 0x7f14580c9530 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:03.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.982+0000 7f145e5e2640 1 -- 172.21.15.16:0/2325470551 >> 172.21.15.16:0/2325470551 conn(0x7f14580655c0 msgr2=0x7f14580b3970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:03.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.982+0000 7f145e5e2640 1 -- 172.21.15.16:0/2325470551 shutdown_connections 2023-05-22T15:27:03.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:03.982+0000 7f145e5e2640 1 -- 172.21.15.16:0/2325470551 wait complete. 2023-05-22T15:27:04.008 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-05-22T15:27:04.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:04.122+0000 7feb8d703640 1 Processor -- start 2023-05-22T15:27:04.134 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:04.122+0000 7feb8d703640 1 -- start start 2023-05-22T15:27:04.135 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:04.126+0000 7feb8d703640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7feb8815bdf0 0x7feb880afac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:04.135 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:04.126+0000 7feb8d703640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7feb8806d220 0x7feb880c8720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:04.136 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:04.126+0000 7feb8d703640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb880b0450 con 0x7feb880b0000 2023-05-22T15:27:04.136 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:04.126+0000 7feb8d703640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb8815fb80 con 0x7feb8806d220 2023-05-22T15:27:04.136 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:04.126+0000 7feb8d703640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb8815fe60 con 0x7feb8815bdf0 2023-05-22T15:27:04.137 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:04.126+0000 7feb877fe640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7feb8806d220 0x7feb880c8720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:04.137 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:04.126+0000 7feb877fe640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7feb8806d220 0x7feb880c8720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57598/0 (socket says 172.21.15.16:57598) 2023-05-22T15:27:04.137 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:04.126+0000 7feb877fe640 1 -- 172.21.15.16:0/2992299177 learned_addr learned my addr 172.21.15.16:0/2992299177 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:04.137 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:04.126+0000 7feb86ffd640 1 --2- 172.21.15.16:0/2992299177 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7feb8815bdf0 0x7feb880afac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:04.137 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:04.126+0000 7feb877fe640 1 -- 172.21.15.16:0/2992299177 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7feb8815bdf0 msgr2=0x7feb880afac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:04.137 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:04.126+0000 7feb877fe640 1 --2- 172.21.15.16:0/2992299177 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7feb8815bdf0 0x7feb880afac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:04.138 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:04.126+0000 7feb877fe640 1 -- 172.21.15.16:0/2992299177 >> v1:172.21.15.16:6789/0 conn(0x7feb880b0000 legacy=0x7feb8806cb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:04.138 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:04.126+0000 7feb877fe640 1 -- 172.21.15.16:0/2992299177 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb88160140 con 0x7feb8806d220 2023-05-22T15:27:04.139 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:04.126+0000 7feb877fe640 1 --2- 172.21.15.16:0/2992299177 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feb8806d220 0x7feb880c8720 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7feb80009e50 tx=0x7feb8001c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aafec2714b26c440 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:04.140 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:04.126+0000 7feb867fc640 1 -- 172.21.15.16:0/2992299177 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb800a57a0 con 0x7feb8806d220 2023-05-22T15:27:04.140 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:04.126+0000 7feb867fc640 1 -- 172.21.15.16:0/2992299177 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb800a5900 con 0x7feb8806d220 2023-05-22T15:27:04.140 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:04.126+0000 7feb867fc640 1 -- 172.21.15.16:0/2992299177 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb800a5c90 con 0x7feb8806d220 2023-05-22T15:27:04.140 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:04.126+0000 7feb8d703640 1 -- 172.21.15.16:0/2992299177 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feb8806d220 msgr2=0x7feb880c8720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:04.140 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:04.126+0000 7feb8d703640 1 --2- 172.21.15.16:0/2992299177 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feb8806d220 0x7feb880c8720 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7feb80009e50 tx=0x7feb8001c8c0 comp rx=0 tx=0).stop 2023-05-22T15:27:04.141 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:04.130+0000 7feb8d703640 1 -- 172.21.15.16:0/2992299177 shutdown_connections 2023-05-22T15:27:04.141 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:04.130+0000 7feb8d703640 1 --2- 172.21.15.16:0/2992299177 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feb8806d220 0x7feb880c8720 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:04.141 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:04.130+0000 7feb8d703640 1 --2- 172.21.15.16:0/2992299177 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7feb8815bdf0 0x7feb880afac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:04.141 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:04.130+0000 7feb8d703640 1 -- 172.21.15.16:0/2992299177 >> 172.21.15.16:0/2992299177 conn(0x7feb880ce040 msgr2=0x7feb880ce440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:04.141 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:04.130+0000 7feb8d703640 1 -- 172.21.15.16:0/2992299177 shutdown_connections 2023-05-22T15:27:04.142 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:04.130+0000 7feb8d703640 1 -- 172.21.15.16:0/2992299177 wait complete. 2023-05-22T15:27:04.142 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:04.130+0000 7feb8d703640 1 Processor -- start 2023-05-22T15:27:04.142 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:04.130+0000 7feb8d703640 1 -- start start 2023-05-22T15:27:04.144 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:04.130+0000 7feb8d703640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7feb8806d220 0x7feb8815ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:04.145 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:04.130+0000 7feb8d703640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feb880b0000 0x7feb881539c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:04.145 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:04.130+0000 7feb8d703640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7feb8815bdf0 0x7feb880b3630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:04.145 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:04.130+0000 7feb8d703640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb8815f230 con 0x7feb880b0000 2023-05-22T15:27:04.145 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:04.130+0000 7feb8d703640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb880c7fd0 con 0x7feb8806d220 2023-05-22T15:27:04.145 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:04.130+0000 7feb8d703640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb8815e860 con 0x7feb8815bdf0 2023-05-22T15:27:04.146 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:04.130+0000 7feb7e7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feb880b0000 0x7feb881539c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:04.146 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:04.130+0000 7feb7e7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feb880b0000 0x7feb881539c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57606/0 (socket says 172.21.15.16:57606) 2023-05-22T15:27:04.146 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:04.130+0000 7feb7e7fc640 1 -- 172.21.15.16:0/3546209774 learned_addr learned my addr 172.21.15.16:0/3546209774 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:04.146 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:04.130+0000 7feb877fe640 1 --2- 172.21.15.16:0/3546209774 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7feb8815bdf0 0x7feb880b3630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:04.146 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:04.130+0000 7feb86ffd640 1 --2- 172.21.15.16:0/3546209774 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7feb8806d220 0x7feb8815ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:04.146 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:04.130+0000 7feb7e7fc640 1 -- 172.21.15.16:0/3546209774 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7feb8815bdf0 msgr2=0x7feb880b3630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:04.147 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:04.130+0000 7feb7e7fc640 1 --2- 172.21.15.16:0/3546209774 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7feb8815bdf0 0x7feb880b3630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:04.147 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:04.130+0000 7feb7e7fc640 1 -- 172.21.15.16:0/3546209774 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7feb8806d220 msgr2=0x7feb8815ecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:04.147 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:04.130+0000 7feb7e7fc640 1 --2- 172.21.15.16:0/3546209774 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7feb8806d220 0x7feb8815ecf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:04.147 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:04.130+0000 7feb7e7fc640 1 -- 172.21.15.16:0/3546209774 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb80009b30 con 0x7feb880b0000 2023-05-22T15:27:04.147 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:04.130+0000 7feb877fe640 1 --2- 172.21.15.16:0/3546209774 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7feb8815bdf0 0x7feb880b3630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T15:27:04.147 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:04.130+0000 7feb86ffd640 1 --2- 172.21.15.16:0/3546209774 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7feb8806d220 0x7feb8815ecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T15:27:04.147 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:04.130+0000 7feb7e7fc640 1 --2- 172.21.15.16:0/3546209774 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feb880b0000 0x7feb881539c0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7feb7000c500 tx=0x7feb7000c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:04.148 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:04.130+0000 7feb7ffff640 1 -- 172.21.15.16:0/3546209774 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb70005220 con 0x7feb880b0000 2023-05-22T15:27:04.148 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:04.130+0000 7feb7ffff640 1 -- 172.21.15.16:0/3546209774 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb70005380 con 0x7feb880b0000 2023-05-22T15:27:04.148 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:04.130+0000 7feb8d703640 1 -- 172.21.15.16:0/3546209774 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb880c9600 con 0x7feb880b0000 2023-05-22T15:27:04.148 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:04.130+0000 7feb7ffff640 1 -- 172.21.15.16:0/3546209774 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb70007270 con 0x7feb880b0000 2023-05-22T15:27:04.148 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:04.130+0000 7feb8d703640 1 -- 172.21.15.16:0/3546209774 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb880c9470 con 0x7feb880b0000 2023-05-22T15:27:04.148 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:04.134+0000 7feb8d703640 1 -- 172.21.15.16:0/3546209774 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb880c9790 con 0x7feb880b0000 2023-05-22T15:27:04.148 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:04.134+0000 7feb7ffff640 1 -- 172.21.15.16:0/3546209774 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7feb700073d0 con 0x7feb880b0000 2023-05-22T15:27:04.149 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:04.134+0000 7feb7ffff640 1 --2- 172.21.15.16:0/3546209774 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7feb68065380 0x7feb68067830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:04.149 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:04.134+0000 7feb7ffff640 1 -- 172.21.15.16:0/3546209774 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6813+0+0 (secure 0 0 0) 0x7feb70082280 con 0x7feb880b0000 2023-05-22T15:27:04.149 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:04.134+0000 7feb86ffd640 1 --2- 172.21.15.16:0/3546209774 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7feb68065380 0x7feb68067830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:04.149 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:04.134+0000 7feb86ffd640 1 --2- 172.21.15.16:0/3546209774 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7feb68065380 0x7feb68067830 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7feb8815aed0 tx=0x7feb78008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:04.150 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:04.138+0000 7feb7ffff640 1 -- 172.21.15.16:0/3546209774 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7feb7004fc20 con 0x7feb880b0000 2023-05-22T15:27:04.302 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:04.290+0000 7feb8d703640 1 -- 172.21.15.16:0/3546209774 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7feb880b0450 con 0x7feb880b0000 2023-05-22T15:27:04.303 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:04.294+0000 7feb7ffff640 1 -- 172.21.15.16:0/3546209774 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) v1 ==== 74+0+19604 (secure 0 0 0) 0x7feb880b0450 con 0x7feb880b0000 2023-05-22T15:27:04.304 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:27:04.304 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":27,"fsid":"b2719834-7847-4d98-90a6-f65d57f85c49","created":"2023-05-22T15:26:25.259314+0000","modified":"2023-05-22T15:26:53.971378+0000","last_up_change":"2023-05-22T15:26:36.448524+0000","last_in_change":"2023-05-22T15:26:27.675668+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-05-22T15:26:38.019781+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-05-22T15:26:43.570514+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-05-22T15:26:47.879159+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-05-22T15:26:48.522130+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data_ec","create_time":"2023-05-22T15:26:51.616142+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"9e7ff854-b25f-4cbb-a5c2-1799d9c53d2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":37021},{"type":"v1","addr":"172.21.15.16:6825","nonce":37021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":37021},{"type":"v1","addr":"172.21.15.16:6827","nonce":37021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":37021},{"type":"v1","addr":"172.21.15.16:6831","nonce":37021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":37021},{"type":"v1","addr":"172.21.15.16:6829","nonce":37021}]},"public_addr":"172.21.15.16:6825/37021","cluster_addr":"172.21.15.16:6827/37021","heartbeat_back_addr":"172.21.15.16:6831/37021","heartbeat_front_addr":"172.21.15.16:6829/37021","state":["exists","up"]},{"osd":1,"uuid":"4702fbc9-a927-4565-aca5-41dc69cc8264","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":37008},{"type":"v1","addr":"172.21.15.16:6801","nonce":37008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":37008},{"type":"v1","addr":"172.21.15.16:6803","nonce":37008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":37008},{"type":"v1","addr":"172.21.15.16:6807","nonce":37008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":37008},{"type":"v1","addr":"172.21.15.16:6805","nonce":37008}]},"public_addr":"172.21.15.16:6801/37008","cluster_addr":"172.21.15.16:6803/37008","heartbeat_back_addr":"172.21.15.16:6807/37008","heartbeat_front_addr":"172.21.15.16:6805/37008","state":["exists","up"]},{"osd":2,"uuid":"1e7bba46-4eba-47ef-bb81-5e2d371638ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":37009},{"type":"v1","addr":"172.21.15.16:6809","nonce":37009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":37009},{"type":"v1","addr":"172.21.15.16:6811","nonce":37009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":37009},{"type":"v1","addr":"172.21.15.16:6815","nonce":37009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":37009},{"type":"v1","addr":"172.21.15.16:6813","nonce":37009}]},"public_addr":"172.21.15.16:6809/37009","cluster_addr":"172.21.15.16:6811/37009","heartbeat_back_addr":"172.21.15.16:6815/37009","heartbeat_front_addr":"172.21.15.16:6813/37009","state":["exists","up"]},{"osd":3,"uuid":"383cacc0-5a88-4904-990e-498ca59ef48a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":37020},{"type":"v1","addr":"172.21.15.16:6817","nonce":37020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":37020},{"type":"v1","addr":"172.21.15.16:6819","nonce":37020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":37020},{"type":"v1","addr":"172.21.15.16:6823","nonce":37020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":37020},{"type":"v1","addr":"172.21.15.16:6821","nonce":37020}]},"public_addr":"172.21.15.16:6817/37020","cluster_addr":"172.21.15.16:6819/37020","heartbeat_back_addr":"172.21.15.16:6823/37020","heartbeat_front_addr":"172.21.15.16:6821/37020","state":["exists","up"]},{"osd":4,"uuid":"970500ef-95b7-42c1-b080-eed4fae7013d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":37353},{"type":"v1","addr":"172.21.15.62:6817","nonce":37353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6819","nonce":37353},{"type":"v1","addr":"172.21.15.62:6822","nonce":37353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37353},{"type":"v1","addr":"172.21.15.62:6831","nonce":37353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37353},{"type":"v1","addr":"172.21.15.62:6826","nonce":37353}]},"public_addr":"172.21.15.62:6817/37353","cluster_addr":"172.21.15.62:6822/37353","heartbeat_back_addr":"172.21.15.62:6831/37353","heartbeat_front_addr":"172.21.15.62:6826/37353","state":["exists","up"]},{"osd":5,"uuid":"022d67db-d4d8-445b-b1b9-8b446a00f5c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":37352},{"type":"v1","addr":"172.21.15.62:6801","nonce":37352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":37352},{"type":"v1","addr":"172.21.15.62:6803","nonce":37352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":37352},{"type":"v1","addr":"172.21.15.62:6807","nonce":37352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":37352},{"type":"v1","addr":"172.21.15.62:6805","nonce":37352}]},"public_addr":"172.21.15.62:6801/37352","cluster_addr":"172.21.15.62:6803/37352","heartbeat_back_addr":"172.21.15.62:6807/37352","heartbeat_front_addr":"172.21.15.62:6805/37352","state":["exists","up"]},{"osd":6,"uuid":"42bd372b-6688-4a1b-8407-e16e8c266e33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":37354},{"type":"v1","addr":"172.21.15.62:6820","nonce":37354}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":37354},{"type":"v1","addr":"172.21.15.62:6823","nonce":37354}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6829","nonce":37354},{"type":"v1","addr":"172.21.15.62:6830","nonce":37354}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6825","nonce":37354},{"type":"v1","addr":"172.21.15.62:6827","nonce":37354}]},"public_addr":"172.21.15.62:6820/37354","cluster_addr":"172.21.15.62:6823/37354","heartbeat_back_addr":"172.21.15.62:6830/37354","heartbeat_front_addr":"172.21.15.62:6827/37354","state":["exists","up"]},{"osd":7,"uuid":"ac70b2a3-f41b-4bf2-b227-7f2cbb55426d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37355},{"type":"v1","addr":"172.21.15.62:6809","nonce":37355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37355},{"type":"v1","addr":"172.21.15.62:6811","nonce":37355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":37355},{"type":"v1","addr":"172.21.15.62:6815","nonce":37355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":37355},{"type":"v1","addr":"172.21.15.62:6813","nonce":37355}]},"public_addr":"172.21.15.62:6809/37355","cluster_addr":"172.21.15.62:6811/37355","heartbeat_back_addr":"172.21.15.62:6815/37355","heartbeat_front_addr":"172.21.15.62:6813/37355","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.368896+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.107273+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.276888+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.287502+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.124791+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.251937+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.255679+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.232797+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-05-22T15:27:04.306 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:04.294+0000 7feb8d703640 1 -- 172.21.15.16:0/3546209774 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7feb68065380 msgr2=0x7feb68067830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:04.306 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:04.294+0000 7feb8d703640 1 --2- 172.21.15.16:0/3546209774 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7feb68065380 0x7feb68067830 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7feb8815aed0 tx=0x7feb78008040 comp rx=0 tx=0).stop 2023-05-22T15:27:04.307 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:04.294+0000 7feb8d703640 1 -- 172.21.15.16:0/3546209774 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feb880b0000 msgr2=0x7feb881539c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:04.307 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:04.294+0000 7feb8d703640 1 --2- 172.21.15.16:0/3546209774 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feb880b0000 0x7feb881539c0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7feb7000c500 tx=0x7feb7000c9c0 comp rx=0 tx=0).stop 2023-05-22T15:27:04.307 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:04.294+0000 7feb8d703640 1 -- 172.21.15.16:0/3546209774 shutdown_connections 2023-05-22T15:27:04.307 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:04.294+0000 7feb8d703640 1 --2- 172.21.15.16:0/3546209774 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7feb68065380 0x7feb68067830 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:04.308 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:04.294+0000 7feb8d703640 1 --2- 172.21.15.16:0/3546209774 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7feb8815bdf0 0x7feb880b3630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:04.308 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:04.294+0000 7feb8d703640 1 --2- 172.21.15.16:0/3546209774 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feb880b0000 0x7feb881539c0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:04.308 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:04.294+0000 7feb8d703640 1 --2- 172.21.15.16:0/3546209774 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7feb8806d220 0x7feb8815ecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:04.308 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:04.294+0000 7feb8d703640 1 -- 172.21.15.16:0/3546209774 >> 172.21.15.16:0/3546209774 conn(0x7feb880ce040 msgr2=0x7feb880aca00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:04.309 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:04.298+0000 7feb8d703640 1 -- 172.21.15.16:0/3546209774 shutdown_connections 2023-05-22T15:27:04.309 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:04.298+0000 7feb8d703640 1 -- 172.21.15.16:0/3546209774 wait complete. 2023-05-22T15:27:04.324 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-05-22T15:27:04.445 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:04.434+0000 7f67ff5d6640 1 Processor -- start 2023-05-22T15:27:04.446 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:04.434+0000 7f67ff5d6640 1 -- start start 2023-05-22T15:27:04.447 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:04.438+0000 7f67ff5d6640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f67f8159970 0x7f67f8159d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:04.447 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:04.438+0000 7f67ff5d6640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f67f815d680 0x7f67f80cb260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:04.447 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:04.438+0000 7f67ff5d6640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67f8070c30 con 0x7f67f815db00 2023-05-22T15:27:04.447 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:04.438+0000 7f67ff5d6640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67f8070e60 con 0x7f67f815d680 2023-05-22T15:27:04.447 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:04.438+0000 7f67ff5d6640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67f8071140 con 0x7f67f8159970 2023-05-22T15:27:04.448 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:04.438+0000 7f67fcb71640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f67f815d680 0x7f67f80cb260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:04.448 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:04.438+0000 7f67fcb71640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f67f815d680 0x7f67f80cb260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57618/0 (socket says 172.21.15.16:57618) 2023-05-22T15:27:04.448 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:04.438+0000 7f67fcb71640 1 -- 172.21.15.16:0/1746360910 learned_addr learned my addr 172.21.15.16:0/1746360910 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:04.448 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:04.438+0000 7f67fd372640 1 --2- 172.21.15.16:0/1746360910 >> v2:172.21.15.62:3300/0 conn(0x7f67f8159970 0x7f67f8159d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:04.448 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:04.438+0000 7f67fcb71640 1 -- 172.21.15.16:0/1746360910 >> v2:172.21.15.62:3300/0 conn(0x7f67f8159970 msgr2=0x7f67f8159d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:04.448 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:04.438+0000 7f67fcb71640 1 --2- 172.21.15.16:0/1746360910 >> v2:172.21.15.62:3300/0 conn(0x7f67f8159970 0x7f67f8159d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:04.449 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:04.438+0000 7f67fcb71640 1 -- 172.21.15.16:0/1746360910 >> v1:172.21.15.62:6789/0 conn(0x7f67f815db00 legacy=0x7f67f806b6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:04.449 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:04.438+0000 7f67fcb71640 1 -- 172.21.15.16:0/1746360910 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67f8071420 con 0x7f67f815d680 2023-05-22T15:27:04.450 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:04.438+0000 7f67fcb71640 1 --2- 172.21.15.16:0/1746360910 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f67f815d680 0x7f67f80cb260 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f67f806daf0 tx=0x7f67e801c400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3970462e8e5ed328 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:04.450 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:04.438+0000 7f67effff640 1 -- 172.21.15.16:0/1746360910 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67e80ab620 con 0x7f67f815d680 2023-05-22T15:27:04.450 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:04.438+0000 7f67effff640 1 -- 172.21.15.16:0/1746360910 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67e80ab780 con 0x7f67f815d680 2023-05-22T15:27:04.450 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:04.438+0000 7f67effff640 1 -- 172.21.15.16:0/1746360910 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67e80abb10 con 0x7f67f815d680 2023-05-22T15:27:04.450 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:04.438+0000 7f67ff5d6640 1 -- 172.21.15.16:0/1746360910 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f67f815d680 msgr2=0x7f67f80cb260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:04.450 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:04.438+0000 7f67ff5d6640 1 --2- 172.21.15.16:0/1746360910 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f67f815d680 0x7f67f80cb260 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f67f806daf0 tx=0x7f67e801c400 comp rx=0 tx=0).stop 2023-05-22T15:27:04.451 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:04.438+0000 7f67ff5d6640 1 -- 172.21.15.16:0/1746360910 shutdown_connections 2023-05-22T15:27:04.451 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:04.438+0000 7f67ff5d6640 1 --2- 172.21.15.16:0/1746360910 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f67f815d680 0x7f67f80cb260 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:04.451 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:04.438+0000 7f67ff5d6640 1 --2- 172.21.15.16:0/1746360910 >> v2:172.21.15.62:3300/0 conn(0x7f67f8159970 0x7f67f8159d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:04.451 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:04.438+0000 7f67ff5d6640 1 -- 172.21.15.16:0/1746360910 >> 172.21.15.16:0/1746360910 conn(0x7f67f8069720 msgr2=0x7f67f8069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:04.451 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:04.438+0000 7f67ff5d6640 1 -- 172.21.15.16:0/1746360910 shutdown_connections 2023-05-22T15:27:04.451 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:04.438+0000 7f67ff5d6640 1 -- 172.21.15.16:0/1746360910 wait complete. 2023-05-22T15:27:04.452 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:04.442+0000 7f67ff5d6640 1 Processor -- start 2023-05-22T15:27:04.452 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:04.442+0000 7f67ff5d6640 1 -- start start 2023-05-22T15:27:04.454 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:04.442+0000 7f67ff5d6640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f67f8159970 0x7f67f80cddf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:04.455 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:04.442+0000 7f67ff5d6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f67f815d680 0x7f67f80ce330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:04.455 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:04.442+0000 7f67ff5d6640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f67f815db00 0x7f67f81508c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:04.455 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:04.442+0000 7f67ff5d6640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67f81641f0 con 0x7f67f815d680 2023-05-22T15:27:04.455 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:04.442+0000 7f67ff5d6640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67f80b32d0 con 0x7f67f815db00 2023-05-22T15:27:04.456 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:04.442+0000 7f67ff5d6640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67f80c1be0 con 0x7f67f8159970 2023-05-22T15:27:04.456 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:04.442+0000 7f67fcb71640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f67f815d680 0x7f67f80ce330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:04.456 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:04.442+0000 7f67fcb71640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f67f815d680 0x7f67f80ce330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57630/0 (socket says 172.21.15.16:57630) 2023-05-22T15:27:04.457 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:04.442+0000 7f67fcb71640 1 -- 172.21.15.16:0/3870936901 learned_addr learned my addr 172.21.15.16:0/3870936901 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:04.457 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:04.442+0000 7f67fd372640 1 --2- 172.21.15.16:0/3870936901 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f67f8159970 0x7f67f80cddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:04.457 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:04.442+0000 7f67fdb73640 1 --2- 172.21.15.16:0/3870936901 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f67f815db00 0x7f67f81508c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:04.458 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:04.442+0000 7f67fcb71640 1 -- 172.21.15.16:0/3870936901 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f67f8159970 msgr2=0x7f67f80cddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:04.458 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:04.442+0000 7f67fcb71640 1 --2- 172.21.15.16:0/3870936901 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f67f8159970 0x7f67f80cddf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:04.458 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:04.442+0000 7f67fcb71640 1 -- 172.21.15.16:0/3870936901 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f67f815db00 msgr2=0x7f67f81508c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:04.458 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:04.442+0000 7f67fcb71640 1 --2- 172.21.15.16:0/3870936901 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f67f815db00 0x7f67f81508c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:04.458 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:04.442+0000 7f67fcb71640 1 -- 172.21.15.16:0/3870936901 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67e80094d0 con 0x7f67f815d680 2023-05-22T15:27:04.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:04.442+0000 7f67fd372640 1 --2- 172.21.15.16:0/3870936901 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f67f8159970 0x7f67f80cddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T15:27:04.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:04.442+0000 7f67fcb71640 1 --2- 172.21.15.16:0/3870936901 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f67f815d680 0x7f67f80ce330 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f67e80ad040 tx=0x7f67e801ac40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:04.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:04.442+0000 7f67edffb640 1 -- 172.21.15.16:0/3870936901 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67e80abe80 con 0x7f67f815d680 2023-05-22T15:27:04.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:04.442+0000 7f67edffb640 1 -- 172.21.15.16:0/3870936901 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67e80be3f0 con 0x7f67f815d680 2023-05-22T15:27:04.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:04.442+0000 7f67edffb640 1 -- 172.21.15.16:0/3870936901 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67e80be780 con 0x7f67f815d680 2023-05-22T15:27:04.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:04.442+0000 7f67ff5d6640 1 -- 172.21.15.16:0/3870936901 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67f80cae20 con 0x7f67f815d680 2023-05-22T15:27:04.460 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:04.442+0000 7f67ff5d6640 1 -- 172.21.15.16:0/3870936901 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67f80cac90 con 0x7f67f815d680 2023-05-22T15:27:04.460 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:04.446+0000 7f67ff5d6640 1 -- 172.21.15.16:0/3870936901 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67f806bdd0 con 0x7f67f815d680 2023-05-22T15:27:04.461 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:04.446+0000 7f67edffb640 1 -- 172.21.15.16:0/3870936901 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7f67e80c7440 con 0x7f67f815d680 2023-05-22T15:27:04.461 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:04.446+0000 7f67edffb640 1 --2- 172.21.15.16:0/3870936901 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f67dc065380 0x7f67dc067830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:04.461 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:04.446+0000 7f67edffb640 1 -- 172.21.15.16:0/3870936901 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f67e811d4d0 con 0x7f67f815d680 2023-05-22T15:27:04.461 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:04.446+0000 7f67fd372640 1 --2- 172.21.15.16:0/3870936901 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f67dc065380 0x7f67dc067830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:04.461 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:04.446+0000 7f67fd372640 1 --2- 172.21.15.16:0/3870936901 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f67dc065380 0x7f67dc067830 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f67f815c600 tx=0x7f67f4008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:04.462 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:04.450+0000 7f67edffb640 1 -- 172.21.15.16:0/3870936901 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f67e80eae90 con 0x7f67f815d680 2023-05-22T15:27:04.643 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:04.634+0000 7f67ff5d6640 1 -- 172.21.15.16:0/3870936901 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f67f80702f0 con 0x7f67f815d680 2023-05-22T15:27:05.054 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:05.046+0000 7f67edffb640 1 -- 172.21.15.16:0/3870936901 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v7) v1 ==== 118+0+0 (secure 0 0 0) 0x7f67e80ee4b0 con 0x7f67f815d680 2023-05-22T15:27:05.059 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:05.050+0000 7f67ff5d6640 1 -- 172.21.15.16:0/3870936901 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f67dc065380 msgr2=0x7f67dc067830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:05.059 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:05.050+0000 7f67ff5d6640 1 --2- 172.21.15.16:0/3870936901 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f67dc065380 0x7f67dc067830 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f67f815c600 tx=0x7f67f4008040 comp rx=0 tx=0).stop 2023-05-22T15:27:05.059 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:05.050+0000 7f67ff5d6640 1 -- 172.21.15.16:0/3870936901 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f67f815d680 msgr2=0x7f67f80ce330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:05.059 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:05.050+0000 7f67ff5d6640 1 --2- 172.21.15.16:0/3870936901 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f67f815d680 0x7f67f80ce330 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f67e80ad040 tx=0x7f67e801ac40 comp rx=0 tx=0).stop 2023-05-22T15:27:05.059 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:05.050+0000 7f67ff5d6640 1 -- 172.21.15.16:0/3870936901 shutdown_connections 2023-05-22T15:27:05.060 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:05.050+0000 7f67ff5d6640 1 --2- 172.21.15.16:0/3870936901 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f67f815db00 0x7f67f81508c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:05.060 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:05.050+0000 7f67ff5d6640 1 --2- 172.21.15.16:0/3870936901 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f67f815d680 0x7f67f80ce330 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:05.060 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:05.050+0000 7f67ff5d6640 1 --2- 172.21.15.16:0/3870936901 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f67dc065380 0x7f67dc067830 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:05.060 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:05.050+0000 7f67ff5d6640 1 --2- 172.21.15.16:0/3870936901 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f67f8159970 0x7f67f80cddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:05.060 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:05.050+0000 7f67ff5d6640 1 -- 172.21.15.16:0/3870936901 >> 172.21.15.16:0/3870936901 conn(0x7f67f8069720 msgr2=0x7f67f8156f70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:05.060 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:05.050+0000 7f67ff5d6640 1 -- 172.21.15.16:0/3870936901 shutdown_connections 2023-05-22T15:27:05.061 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:05.050+0000 7f67ff5d6640 1 -- 172.21.15.16:0/3870936901 wait complete. 2023-05-22T15:27:05.080 DEBUG:tasks.cephfs.filesystem:fs_config: {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd']} 2023-05-22T15:27:05.081 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-05-22T15:27:05.214 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:05.206+0000 7fc7a125b640 1 Processor -- start 2023-05-22T15:27:05.215 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:05.206+0000 7fc7a125b640 1 -- start start 2023-05-22T15:27:05.217 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:05.206+0000 7fc7a125b640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fc79c0ce480 0x7fc79c071640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:05.217 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:05.206+0000 7fc7a125b640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc79c15cde0 con 0x7fc79c15c270 2023-05-22T15:27:05.217 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:05.206+0000 7fc7a125b640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc79c069ae0 con 0x7fc79c0ce850 2023-05-22T15:27:05.217 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:05.206+0000 7fc7a125b640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc79c15f990 con 0x7fc79c0ce480 2023-05-22T15:27:05.217 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:05.206+0000 7fc79b59e640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fc79c15c270 0x7fc79c15c6d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:51000/0 (socket says 172.21.15.16:51000) 2023-05-22T15:27:05.218 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:05.206+0000 7fc79b59e640 1 -- 172.21.15.16:0/1194980379 learned_addr learned my addr 172.21.15.16:0/1194980379 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:27:05.218 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:05.206+0000 7fc79ad9d640 1 --2- 172.21.15.16:0/1194980379 >> v2:172.21.15.62:3300/0 conn(0x7fc79c0ce480 0x7fc79c071640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:05.218 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:05.206+0000 7fc799d9b640 1 -- 172.21.15.16:0/1194980379 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1336915817 0 0) 0x7fc79c15cde0 con 0x7fc79c15c270 2023-05-22T15:27:05.218 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:05.206+0000 7fc799d9b640 1 -- 172.21.15.16:0/1194980379 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc784003530 con 0x7fc79c15c270 2023-05-22T15:27:05.218 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:05.206+0000 7fc799d9b640 1 -- 172.21.15.16:0/1194980379 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3778335226 0 0) 0x7fc784003530 con 0x7fc79c15c270 2023-05-22T15:27:05.219 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:05.206+0000 7fc799d9b640 1 -- 172.21.15.16:0/1194980379 >> v2:172.21.15.62:3300/0 conn(0x7fc79c0ce480 msgr2=0x7fc79c071640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:05.219 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:05.206+0000 7fc799d9b640 1 --2- 172.21.15.16:0/1194980379 >> v2:172.21.15.62:3300/0 conn(0x7fc79c0ce480 0x7fc79c071640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:05.219 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:05.206+0000 7fc799d9b640 1 -- 172.21.15.16:0/1194980379 >> v1:172.21.15.62:6789/0 conn(0x7fc79c0ce850 legacy=0x7fc79c160090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:05.219 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:05.206+0000 7fc799d9b640 1 -- 172.21.15.16:0/1194980379 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc79c069dc0 con 0x7fc79c15c270 2023-05-22T15:27:05.219 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:05.206+0000 7fc799d9b640 1 -- 172.21.15.16:0/1194980379 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7fc794001a00 con 0x7fc79c15c270 2023-05-22T15:27:05.219 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:05.206+0000 7fc79ad9d640 1 --2- 172.21.15.16:0/1194980379 >> v2:172.21.15.62:3300/0 conn(0x7fc79c0ce480 0x7fc79c071640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T15:27:05.219 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:05.206+0000 7fc799d9b640 1 -- 172.21.15.16:0/1194980379 >> v1:172.21.15.16:6789/0 conn(0x7fc79c15c270 legacy=0x7fc79c15c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:05.219 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:05.206+0000 7fc799d9b640 1 --2- 172.21.15.16:0/1194980379 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc784003980 0x7fc784003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:05.220 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:05.206+0000 7fc799d9b640 1 --2- 172.21.15.16:0/1194980379 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc7840047d0 0x7fc784004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:05.220 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:05.206+0000 7fc799d9b640 1 --2- 172.21.15.16:0/1194980379 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc784005180 0x7fc78400b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:05.220 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:05.206+0000 7fc799d9b640 1 -- 172.21.15.16:0/1194980379 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc79c15f990 con 0x7fc7840047d0 2023-05-22T15:27:05.220 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:05.206+0000 7fc799d9b640 1 -- 172.21.15.16:0/1194980379 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc78400c110 con 0x7fc784005180 2023-05-22T15:27:05.220 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:05.206+0000 7fc799d9b640 1 -- 172.21.15.16:0/1194980379 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc78400c2d0 con 0x7fc784003980 2023-05-22T15:27:05.220 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:05.206+0000 7fc79a59c640 1 --2- 172.21.15.16:0/1194980379 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc7840047d0 0x7fc784004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:05.220 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:05.206+0000 7fc79ad9d640 1 --2- 172.21.15.16:0/1194980379 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc784003980 0x7fc784003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:05.221 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:05.206+0000 7fc79b59e640 1 --2- 172.21.15.16:0/1194980379 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc784005180 0x7fc78400b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:05.221 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:05.206+0000 7fc79a59c640 1 -- 172.21.15.16:0/1194980379 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc784003980 msgr2=0x7fc784003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:05.221 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:05.206+0000 7fc79a59c640 1 --2- 172.21.15.16:0/1194980379 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc784003980 0x7fc784003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:05.221 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:05.206+0000 7fc79a59c640 1 -- 172.21.15.16:0/1194980379 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc784005180 msgr2=0x7fc78400b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:05.221 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:05.206+0000 7fc79a59c640 1 --2- 172.21.15.16:0/1194980379 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc784005180 0x7fc78400b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:05.221 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:05.206+0000 7fc79a59c640 1 -- 172.21.15.16:0/1194980379 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc784003530 con 0x7fc7840047d0 2023-05-22T15:27:05.221 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:05.206+0000 7fc79b59e640 1 -- 172.21.15.16:0/1194980379 reap_dead start 2023-05-22T15:27:05.224 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:05.210+0000 7fc79a59c640 1 --2- 172.21.15.16:0/1194980379 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc7840047d0 0x7fc784004c40 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fc78c001750 tx=0x7fc78c01c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e1ce4f19d837b80 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:05.224 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:05.210+0000 7fc799d9b640 1 -- 172.21.15.16:0/1194980379 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc78c0052d0 con 0x7fc7840047d0 2023-05-22T15:27:05.225 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:05.210+0000 7fc799d9b640 1 -- 172.21.15.16:0/1194980379 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc78c0a7050 con 0x7fc7840047d0 2023-05-22T15:27:05.225 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:05.210+0000 7fc799d9b640 1 -- 172.21.15.16:0/1194980379 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc78c003600 con 0x7fc7840047d0 2023-05-22T15:27:05.225 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:05.210+0000 7fc7a125b640 1 -- 172.21.15.16:0/1194980379 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc7840047d0 msgr2=0x7fc784004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:05.225 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:05.210+0000 7fc7a125b640 1 --2- 172.21.15.16:0/1194980379 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc7840047d0 0x7fc784004c40 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fc78c001750 tx=0x7fc78c01c3a0 comp rx=0 tx=0).stop 2023-05-22T15:27:05.225 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:05.210+0000 7fc7a125b640 1 -- 172.21.15.16:0/1194980379 shutdown_connections 2023-05-22T15:27:05.225 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:05.210+0000 7fc7a125b640 1 --2- 172.21.15.16:0/1194980379 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc7840047d0 0x7fc784004c40 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:05.225 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:05.210+0000 7fc7a125b640 1 -- 172.21.15.16:0/1194980379 >> 172.21.15.16:0/1194980379 conn(0x7fc79c0cc870 msgr2=0x7fc79c0cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:05.226 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:05.210+0000 7fc7a125b640 1 -- 172.21.15.16:0/1194980379 shutdown_connections 2023-05-22T15:27:05.226 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:05.210+0000 7fc7a125b640 1 -- 172.21.15.16:0/1194980379 wait complete. 2023-05-22T15:27:05.226 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:05.210+0000 7fc7a125b640 1 Processor -- start 2023-05-22T15:27:05.226 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:05.210+0000 7fc7a125b640 1 -- start start 2023-05-22T15:27:05.226 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:05.210+0000 7fc7a125b640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc7840047d0 0x7fc79c149cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:05.226 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:05.210+0000 7fc7a125b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc79c0b0620 0x7fc79c15f070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:05.226 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:05.210+0000 7fc7a125b640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc79c1f4f60 0x7fc79c1f53d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:05.227 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:05.210+0000 7fc7a125b640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc79c0c7880 con 0x7fc79c0b0620 2023-05-22T15:27:05.227 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:05.210+0000 7fc7a125b640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc79c0c79f0 con 0x7fc79c1f4f60 2023-05-22T15:27:05.227 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:05.210+0000 7fc7a125b640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc79c1f5910 con 0x7fc7840047d0 2023-05-22T15:27:05.227 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:05.210+0000 7fc79a59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc79c0b0620 0x7fc79c15f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:05.227 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:05.210+0000 7fc79a59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc79c0b0620 0x7fc79c15f070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57652/0 (socket says 172.21.15.16:57652) 2023-05-22T15:27:05.227 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:05.210+0000 7fc79a59c640 1 -- 172.21.15.16:0/4250684847 learned_addr learned my addr 172.21.15.16:0/4250684847 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:05.227 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:05.210+0000 7fc79ad9d640 1 --2- 172.21.15.16:0/4250684847 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc7840047d0 0x7fc79c149cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:05.227 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:05.210+0000 7fc79b59e640 1 --2- 172.21.15.16:0/4250684847 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc79c1f4f60 0x7fc79c1f53d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:05.228 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:05.210+0000 7fc79a59c640 1 -- 172.21.15.16:0/4250684847 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc7840047d0 msgr2=0x7fc79c149cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:05.228 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:05.210+0000 7fc79a59c640 1 --2- 172.21.15.16:0/4250684847 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc7840047d0 0x7fc79c149cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:05.228 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:05.210+0000 7fc79a59c640 1 -- 172.21.15.16:0/4250684847 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc79c1f4f60 msgr2=0x7fc79c1f53d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:05.228 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:05.214+0000 7fc79a59c640 1 --2- 172.21.15.16:0/4250684847 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc79c1f4f60 0x7fc79c1f53d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:05.229 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:05.214+0000 7fc79a59c640 1 -- 172.21.15.16:0/4250684847 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc79c15fb70 con 0x7fc79c0b0620 2023-05-22T15:27:05.229 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:05.214+0000 7fc79b59e640 1 --2- 172.21.15.16:0/4250684847 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc79c1f4f60 0x7fc79c1f53d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:27:05.229 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:05.214+0000 7fc79a59c640 1 --2- 172.21.15.16:0/4250684847 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc79c0b0620 0x7fc79c15f070 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fc78c01c1d0 tx=0x7fc784005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:05.230 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:05.214+0000 7fc7837fe640 1 -- 172.21.15.16:0/4250684847 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc78c003e40 con 0x7fc79c0b0620 2023-05-22T15:27:05.230 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:05.214+0000 7fc7837fe640 1 -- 172.21.15.16:0/4250684847 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc78c007030 con 0x7fc79c0b0620 2023-05-22T15:27:05.230 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:05.214+0000 7fc7a125b640 1 -- 172.21.15.16:0/4250684847 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc79c06a0a0 con 0x7fc79c0b0620 2023-05-22T15:27:05.230 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:05.214+0000 7fc7837fe640 1 -- 172.21.15.16:0/4250684847 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc78c0073c0 con 0x7fc79c0b0620 2023-05-22T15:27:05.231 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:05.214+0000 7fc7a125b640 1 -- 172.21.15.16:0/4250684847 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc79c0cecd0 con 0x7fc79c0b0620 2023-05-22T15:27:05.231 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:05.214+0000 7fc7a125b640 1 -- 172.21.15.16:0/4250684847 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc79c04dbe0 con 0x7fc79c0b0620 2023-05-22T15:27:05.231 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:05.214+0000 7fc7837fe640 1 -- 172.21.15.16:0/4250684847 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7fc78c007520 con 0x7fc79c0b0620 2023-05-22T15:27:05.231 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:05.214+0000 7fc7837fe640 1 --2- 172.21.15.16:0/4250684847 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fc788065380 0x7fc788067830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:05.232 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:05.214+0000 7fc7837fe640 1 -- 172.21.15.16:0/4250684847 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6813+0+0 (secure 0 0 0) 0x7fc78c11cf50 con 0x7fc79c0b0620 2023-05-22T15:27:05.232 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:05.214+0000 7fc79ad9d640 1 --2- 172.21.15.16:0/4250684847 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fc788065380 0x7fc788067830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:05.232 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:05.214+0000 7fc79ad9d640 1 --2- 172.21.15.16:0/4250684847 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fc788065380 0x7fc788067830 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fc79c15aa70 tx=0x7fc790008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:05.233 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:05.218+0000 7fc7837fe640 1 -- 172.21.15.16:0/4250684847 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fc78c0ea840 con 0x7fc79c0b0620 2023-05-22T15:27:05.417 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:05.406+0000 7fc7a125b640 1 -- 172.21.15.16:0/4250684847 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fc79c070a10 con 0x7fc79c0b0620 2023-05-22T15:27:06.066 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.058+0000 7fc7837fe640 1 -- 172.21.15.16:0/4250684847 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v8) v1 ==== 122+0+0 (secure 0 0 0) 0x7fc78c0a4e60 con 0x7fc79c0b0620 2023-05-22T15:27:06.070 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.058+0000 7fc7a125b640 1 -- 172.21.15.16:0/4250684847 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fc788065380 msgr2=0x7fc788067830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:06.070 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.058+0000 7fc7a125b640 1 --2- 172.21.15.16:0/4250684847 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fc788065380 0x7fc788067830 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fc79c15aa70 tx=0x7fc790008040 comp rx=0 tx=0).stop 2023-05-22T15:27:06.070 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.058+0000 7fc7a125b640 1 -- 172.21.15.16:0/4250684847 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc79c0b0620 msgr2=0x7fc79c15f070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:06.071 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.058+0000 7fc7a125b640 1 --2- 172.21.15.16:0/4250684847 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc79c0b0620 0x7fc79c15f070 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fc78c01c1d0 tx=0x7fc784005800 comp rx=0 tx=0).stop 2023-05-22T15:27:06.071 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.062+0000 7fc7a125b640 1 -- 172.21.15.16:0/4250684847 shutdown_connections 2023-05-22T15:27:06.071 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.062+0000 7fc7a125b640 1 --2- 172.21.15.16:0/4250684847 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc79c1f4f60 0x7fc79c1f53d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:06.071 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.062+0000 7fc7a125b640 1 --2- 172.21.15.16:0/4250684847 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc79c0b0620 0x7fc79c15f070 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:06.071 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.062+0000 7fc7a125b640 1 --2- 172.21.15.16:0/4250684847 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fc788065380 0x7fc788067830 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:06.071 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.062+0000 7fc7a125b640 1 --2- 172.21.15.16:0/4250684847 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc7840047d0 0x7fc79c149cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:06.071 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.062+0000 7fc7a125b640 1 -- 172.21.15.16:0/4250684847 >> 172.21.15.16:0/4250684847 conn(0x7fc79c0cc870 msgr2=0x7fc79c0b3fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:06.072 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.062+0000 7fc7a125b640 1 -- 172.21.15.16:0/4250684847 shutdown_connections 2023-05-22T15:27:06.072 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.062+0000 7fc7a125b640 1 -- 172.21.15.16:0/4250684847 wait complete. 2023-05-22T15:27:06.097 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2023-05-22T15:27:06.097 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-05-22T15:27:06.226 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.214+0000 7f6566f31640 1 Processor -- start 2023-05-22T15:27:06.226 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.214+0000 7f6566f31640 1 -- start start 2023-05-22T15:27:06.227 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.218+0000 7f6566f31640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f65600d1850 0x7f65600d1c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:06.227 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.218+0000 7f6566f31640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f65600d0f90 0x7f65600cdf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:06.227 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.218+0000 7f6566f31640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65600d2160 con 0x7f65600ce4c0 2023-05-22T15:27:06.227 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.218+0000 7f6566f31640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65600cecd0 con 0x7f65600d1850 2023-05-22T15:27:06.227 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.218+0000 7f6566f31640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f656006abb0 con 0x7f65600d0f90 2023-05-22T15:27:06.227 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.218+0000 7f65654ce640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f65600ce4c0 0x7f65600b3ee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:51002/0 (socket says 172.21.15.16:51002) 2023-05-22T15:27:06.227 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.218+0000 7f65654ce640 1 -- 172.21.15.16:0/3382930369 learned_addr learned my addr 172.21.15.16:0/3382930369 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:27:06.229 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.218+0000 7f655ffff640 1 --2- 172.21.15.16:0/3382930369 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f65600d0f90 0x7f65600cdf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:06.229 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.218+0000 7f6564ccd640 1 --2- 172.21.15.16:0/3382930369 >> v2:172.21.15.62:3300/0 conn(0x7f65600d1850 0x7f65600d1c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:06.229 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.218+0000 7f655f7fe640 1 -- 172.21.15.16:0/3382930369 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3004799562 0 0) 0x7f65600d2160 con 0x7f65600ce4c0 2023-05-22T15:27:06.229 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.218+0000 7f655f7fe640 1 -- 172.21.15.16:0/3382930369 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6548003530 con 0x7f65600ce4c0 2023-05-22T15:27:06.230 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.218+0000 7f655f7fe640 1 -- 172.21.15.16:0/3382930369 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3353790273 0 0) 0x7f6548003530 con 0x7f65600ce4c0 2023-05-22T15:27:06.230 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.218+0000 7f655f7fe640 1 -- 172.21.15.16:0/3382930369 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f65600d0f90 msgr2=0x7f65600cdf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:06.230 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.218+0000 7f655f7fe640 1 --2- 172.21.15.16:0/3382930369 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f65600d0f90 0x7f65600cdf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:06.230 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.218+0000 7f655f7fe640 1 -- 172.21.15.16:0/3382930369 >> v2:172.21.15.62:3300/0 conn(0x7f65600d1850 msgr2=0x7f65600d1c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:06.230 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.218+0000 7f655f7fe640 1 --2- 172.21.15.16:0/3382930369 >> v2:172.21.15.62:3300/0 conn(0x7f65600d1850 0x7f65600d1c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:06.230 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.218+0000 7f655f7fe640 1 -- 172.21.15.16:0/3382930369 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65600d01d0 con 0x7f65600ce4c0 2023-05-22T15:27:06.230 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.218+0000 7f655f7fe640 1 -- 172.21.15.16:0/3382930369 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7f6550001680 con 0x7f65600ce4c0 2023-05-22T15:27:06.231 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.218+0000 7f655f7fe640 1 -- 172.21.15.16:0/3382930369 >> v1:172.21.15.16:6789/0 conn(0x7f65600ce4c0 legacy=0x7f65600b3ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:06.231 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.218+0000 7f655f7fe640 1 --2- 172.21.15.16:0/3382930369 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6548003980 0x7f6548003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:06.231 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.218+0000 7f655f7fe640 1 --2- 172.21.15.16:0/3382930369 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f65600d0f90 0x7f65480047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:06.231 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.218+0000 7f655f7fe640 1 --2- 172.21.15.16:0/3382930369 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6548004d10 0x7f6548005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:06.231 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.218+0000 7f655f7fe640 1 -- 172.21.15.16:0/3382930369 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65600cecd0 con 0x7f6548003980 2023-05-22T15:27:06.231 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.218+0000 7f655f7fe640 1 -- 172.21.15.16:0/3382930369 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f654800b990 con 0x7f6548004d10 2023-05-22T15:27:06.231 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.218+0000 7f655f7fe640 1 -- 172.21.15.16:0/3382930369 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f654800bad0 con 0x7f65600d0f90 2023-05-22T15:27:06.231 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.218+0000 7f6564ccd640 1 --2- 172.21.15.16:0/3382930369 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6548003980 0x7f6548003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:06.232 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.218+0000 7f655ffff640 1 --2- 172.21.15.16:0/3382930369 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f65600d0f90 0x7f65480047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:06.232 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.218+0000 7f65654ce640 1 --2- 172.21.15.16:0/3382930369 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6548004d10 0x7f6548005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:06.232 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.218+0000 7f6564ccd640 1 -- 172.21.15.16:0/3382930369 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f65600d0f90 msgr2=0x7f65480047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:06.232 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.218+0000 7f6564ccd640 1 --2- 172.21.15.16:0/3382930369 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f65600d0f90 0x7f65480047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:06.232 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.218+0000 7f6564ccd640 1 -- 172.21.15.16:0/3382930369 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6548004d10 msgr2=0x7f6548005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:06.232 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.218+0000 7f6564ccd640 1 --2- 172.21.15.16:0/3382930369 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6548004d10 0x7f6548005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:06.233 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.218+0000 7f6564ccd640 1 -- 172.21.15.16:0/3382930369 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6548003530 con 0x7f6548003980 2023-05-22T15:27:06.233 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.218+0000 7f65654ce640 1 --2- 172.21.15.16:0/3382930369 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6548004d10 0x7f6548005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:27:06.235 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.222+0000 7f6564ccd640 1 --2- 172.21.15.16:0/3382930369 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6548003980 0x7f6548003d90 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f6558009fc0 tx=0x7f655801cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc95e437771fd14c server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:06.235 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.222+0000 7f655f7fe640 1 -- 172.21.15.16:0/3382930369 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65580a5a90 con 0x7f6548003980 2023-05-22T15:27:06.235 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.222+0000 7f655f7fe640 1 -- 172.21.15.16:0/3382930369 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f655800f040 con 0x7f6548003980 2023-05-22T15:27:06.236 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.222+0000 7f655f7fe640 1 -- 172.21.15.16:0/3382930369 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65580a5e30 con 0x7f6548003980 2023-05-22T15:27:06.236 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.222+0000 7f6566f31640 1 -- 172.21.15.16:0/3382930369 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6548003980 msgr2=0x7f6548003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:06.236 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.222+0000 7f6566f31640 1 --2- 172.21.15.16:0/3382930369 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6548003980 0x7f6548003d90 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f6558009fc0 tx=0x7f655801cc40 comp rx=0 tx=0).stop 2023-05-22T15:27:06.236 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.222+0000 7f65654ce640 1 -- 172.21.15.16:0/3382930369 reap_dead start 2023-05-22T15:27:06.236 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.222+0000 7f6566f31640 1 -- 172.21.15.16:0/3382930369 shutdown_connections 2023-05-22T15:27:06.236 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.222+0000 7f6566f31640 1 -- 172.21.15.16:0/3382930369 >> 172.21.15.16:0/3382930369 conn(0x7f65600657f0 msgr2=0x7f6560070e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:06.236 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.222+0000 7f6566f31640 1 -- 172.21.15.16:0/3382930369 shutdown_connections 2023-05-22T15:27:06.237 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.222+0000 7f6566f31640 1 -- 172.21.15.16:0/3382930369 wait complete. 2023-05-22T15:27:06.237 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.222+0000 7f6566f31640 1 Processor -- start 2023-05-22T15:27:06.237 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.222+0000 7f6566f31640 1 -- start start 2023-05-22T15:27:06.237 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.222+0000 7f6566f31640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6548003980 0x7f6560156d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:06.237 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.222+0000 7f6566f31640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f65601fc300 0x7f65600b2da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:06.237 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.222+0000 7f6566f31640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f65601fc770 0x7f65600b32e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:06.237 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.222+0000 7f6566f31640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65601fbbd0 con 0x7f65601fc300 2023-05-22T15:27:06.238 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.222+0000 7f6566f31640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65601fbe70 con 0x7f6548003980 2023-05-22T15:27:06.238 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.222+0000 7f6566f31640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65601f8da0 con 0x7f65601fc770 2023-05-22T15:27:06.238 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.222+0000 7f655ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f65601fc300 0x7f65600b2da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:06.238 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.222+0000 7f655ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f65601fc300 0x7f65600b2da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57658/0 (socket says 172.21.15.16:57658) 2023-05-22T15:27:06.238 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.222+0000 7f655ffff640 1 -- 172.21.15.16:0/1928388172 learned_addr learned my addr 172.21.15.16:0/1928388172 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:06.238 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.222+0000 7f6564ccd640 1 --2- 172.21.15.16:0/1928388172 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6548003980 0x7f6560156d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:06.238 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.222+0000 7f65654ce640 1 --2- 172.21.15.16:0/1928388172 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f65601fc770 0x7f65600b32e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:06.238 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.222+0000 7f655ffff640 1 -- 172.21.15.16:0/1928388172 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f65601fc770 msgr2=0x7f65600b32e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:06.239 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.222+0000 7f655ffff640 1 --2- 172.21.15.16:0/1928388172 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f65601fc770 0x7f65600b32e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:06.239 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.222+0000 7f655ffff640 1 -- 172.21.15.16:0/1928388172 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6548003980 msgr2=0x7f6560156d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:06.239 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.222+0000 7f655ffff640 1 --2- 172.21.15.16:0/1928388172 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6548003980 0x7f6560156d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:06.239 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.222+0000 7f655ffff640 1 -- 172.21.15.16:0/1928388172 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65600ce8e0 con 0x7f65601fc300 2023-05-22T15:27:06.239 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.222+0000 7f6564ccd640 1 --2- 172.21.15.16:0/1928388172 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6548003980 0x7f6560156d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T15:27:06.239 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.222+0000 7f655ffff640 1 --2- 172.21.15.16:0/1928388172 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f65601fc300 0x7f65600b2da0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f6554011590 tx=0x7f6554011a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:06.240 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.226+0000 7f655d7fa640 1 -- 172.21.15.16:0/1928388172 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6554004220 con 0x7f65601fc300 2023-05-22T15:27:06.240 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.226+0000 7f655d7fa640 1 -- 172.21.15.16:0/1928388172 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6554004380 con 0x7f65601fc300 2023-05-22T15:27:06.240 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.226+0000 7f655d7fa640 1 -- 172.21.15.16:0/1928388172 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6554015870 con 0x7f65601fc300 2023-05-22T15:27:06.240 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.226+0000 7f6566f31640 1 -- 172.21.15.16:0/1928388172 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65600d0a70 con 0x7f65601fc300 2023-05-22T15:27:06.240 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.226+0000 7f6566f31640 1 -- 172.21.15.16:0/1928388172 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65600cea70 con 0x7f65601fc300 2023-05-22T15:27:06.240 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.226+0000 7f6566f31640 1 -- 172.21.15.16:0/1928388172 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f656004dbe0 con 0x7f65601fc300 2023-05-22T15:27:06.240 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.226+0000 7f655d7fa640 1 -- 172.21.15.16:0/1928388172 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7f65540159d0 con 0x7f65601fc300 2023-05-22T15:27:06.241 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.226+0000 7f655d7fa640 1 --2- 172.21.15.16:0/1928388172 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f654c065380 0x7f654c067830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:06.241 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.226+0000 7f655d7fa640 1 -- 172.21.15.16:0/1928388172 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f6554082ca0 con 0x7f65601fc300 2023-05-22T15:27:06.242 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.226+0000 7f6564ccd640 1 --2- 172.21.15.16:0/1928388172 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f654c065380 0x7f654c067830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:06.242 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.226+0000 7f6564ccd640 1 --2- 172.21.15.16:0/1928388172 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f654c065380 0x7f654c067830 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f65580af000 tx=0x7f65580a7040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:06.242 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.230+0000 7f655d7fa640 1 -- 172.21.15.16:0/1928388172 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f65540506a0 con 0x7f65601fc300 2023-05-22T15:27:06.427 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.418+0000 7f6566f31640 1 -- 172.21.15.16:0/1928388172 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f65601560c0 con 0x7f65601fc300 2023-05-22T15:27:06.428 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.418+0000 7f655d7fa640 1 -- 172.21.15.16:0/1928388172 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4641 (secure 0 0 0) 0x7f6554022090 con 0x7f65601fc300 2023-05-22T15:27:06.428 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:27:06.429 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4261,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6834/3661808159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":3661808159},{"type":"v1","addr":"172.21.15.62:6834","nonce":3661808159}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4264,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/2227467222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6833","nonce":2227467222},{"type":"v1","addr":"172.21.15.62:6835","nonce":2227467222}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4284,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6836/2808106087","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2808106087},{"type":"v1","addr":"172.21.15.16:6836","nonce":2808106087}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-05-22T15:26:49.546008+0000","modified":"2023-05-22T15:27:06.047204+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4287},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4287":{"gid":4287,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.16:6837/283322289","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6835","nonce":283322289},{"type":"v1","addr":"172.21.15.16:6837","nonce":283322289}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-05-22T15:27:06.429 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 8 2023-05-22T15:27:06.430 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.422+0000 7f6566f31640 1 -- 172.21.15.16:0/1928388172 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f654c065380 msgr2=0x7f654c067830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:06.430 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.422+0000 7f6566f31640 1 --2- 172.21.15.16:0/1928388172 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f654c065380 0x7f654c067830 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f65580af000 tx=0x7f65580a7040 comp rx=0 tx=0).stop 2023-05-22T15:27:06.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.422+0000 7f6566f31640 1 -- 172.21.15.16:0/1928388172 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f65601fc300 msgr2=0x7f65600b2da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:06.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.422+0000 7f6566f31640 1 --2- 172.21.15.16:0/1928388172 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f65601fc300 0x7f65600b2da0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f6554011590 tx=0x7f6554011a50 comp rx=0 tx=0).stop 2023-05-22T15:27:06.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.422+0000 7f6566f31640 1 -- 172.21.15.16:0/1928388172 shutdown_connections 2023-05-22T15:27:06.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.422+0000 7f6566f31640 1 --2- 172.21.15.16:0/1928388172 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f654c065380 0x7f654c067830 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:06.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.422+0000 7f6566f31640 1 --2- 172.21.15.16:0/1928388172 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f65601fc770 0x7f65600b32e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:06.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.422+0000 7f6566f31640 1 --2- 172.21.15.16:0/1928388172 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f65601fc300 0x7f65600b2da0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:06.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.422+0000 7f6566f31640 1 --2- 172.21.15.16:0/1928388172 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6548003980 0x7f6560156d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:06.432 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.422+0000 7f6566f31640 1 -- 172.21.15.16:0/1928388172 >> 172.21.15.16:0/1928388172 conn(0x7f65600657f0 msgr2=0x7f65601590f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:06.432 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.422+0000 7f6566f31640 1 -- 172.21.15.16:0/1928388172 shutdown_connections 2023-05-22T15:27:06.432 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.422+0000 7f6566f31640 1 -- 172.21.15.16:0/1928388172 wait complete. 2023-05-22T15:27:06.448 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-05-22T15:27:06.574 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.566+0000 7f301b734640 1 Processor -- start 2023-05-22T15:27:06.574 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.566+0000 7f301b734640 1 -- start start 2023-05-22T15:27:06.576 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.566+0000 7f301b734640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f30140b2a40 0x7f30140cf510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:06.576 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.566+0000 7f301b734640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f30140cfa50 0x7f30140ca8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:06.576 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.566+0000 7f301b734640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30140ce160 con 0x7f301415c130 2023-05-22T15:27:06.577 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.566+0000 7f301b734640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f301406c4a0 con 0x7f30140b2a40 2023-05-22T15:27:06.577 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.566+0000 7f301b734640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30140cece0 con 0x7f30140cfa50 2023-05-22T15:27:06.577 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.566+0000 7f3018ccf640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f30140b2a40 0x7f30140cf510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:06.577 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.566+0000 7f3018ccf640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f30140b2a40 0x7f30140cf510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57668/0 (socket says 172.21.15.16:57668) 2023-05-22T15:27:06.578 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.566+0000 7f3018ccf640 1 -- 172.21.15.16:0/838511076 learned_addr learned my addr 172.21.15.16:0/838511076 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:06.578 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.566+0000 7f3019cd1640 1 --2- 172.21.15.16:0/838511076 >> v2:172.21.15.62:3300/0 conn(0x7f30140cfa50 0x7f30140ca8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:06.579 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.566+0000 7f3018ccf640 1 -- 172.21.15.16:0/838511076 >> v2:172.21.15.62:3300/0 conn(0x7f30140cfa50 msgr2=0x7f30140ca8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:06.579 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.566+0000 7f3018ccf640 1 --2- 172.21.15.16:0/838511076 >> v2:172.21.15.62:3300/0 conn(0x7f30140cfa50 0x7f30140ca8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:06.580 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.566+0000 7f3018ccf640 1 -- 172.21.15.16:0/838511076 >> v1:172.21.15.16:6789/0 conn(0x7f301415c130 legacy=0x7f30140b2330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:06.580 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.566+0000 7f3018ccf640 1 -- 172.21.15.16:0/838511076 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30140cfe70 con 0x7f30140b2a40 2023-05-22T15:27:06.582 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.570+0000 7f3018ccf640 1 --2- 172.21.15.16:0/838511076 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f30140b2a40 0x7f30140cf510 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f30140b1f80 tx=0x7f300401c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad63207358453aee server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:06.583 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.570+0000 7f300bfff640 1 -- 172.21.15.16:0/838511076 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30040ab5f0 con 0x7f30140b2a40 2023-05-22T15:27:06.583 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.570+0000 7f300bfff640 1 -- 172.21.15.16:0/838511076 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30040ab750 con 0x7f30140b2a40 2023-05-22T15:27:06.583 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.570+0000 7f300bfff640 1 -- 172.21.15.16:0/838511076 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30040abae0 con 0x7f30140b2a40 2023-05-22T15:27:06.584 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.570+0000 7f301b734640 1 -- 172.21.15.16:0/838511076 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f30140b2a40 msgr2=0x7f30140cf510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:06.584 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.570+0000 7f301b734640 1 --2- 172.21.15.16:0/838511076 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f30140b2a40 0x7f30140cf510 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f30140b1f80 tx=0x7f300401c500 comp rx=0 tx=0).stop 2023-05-22T15:27:06.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.570+0000 7f301b734640 1 -- 172.21.15.16:0/838511076 shutdown_connections 2023-05-22T15:27:06.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.570+0000 7f301b734640 1 --2- 172.21.15.16:0/838511076 >> v2:172.21.15.62:3300/0 conn(0x7f30140cfa50 0x7f30140ca8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:06.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.570+0000 7f301b734640 1 --2- 172.21.15.16:0/838511076 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f30140b2a40 0x7f30140cf510 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:06.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.570+0000 7f301b734640 1 -- 172.21.15.16:0/838511076 >> 172.21.15.16:0/838511076 conn(0x7f301406f5c0 msgr2=0x7f301406f9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:06.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.570+0000 7f301b734640 1 -- 172.21.15.16:0/838511076 shutdown_connections 2023-05-22T15:27:06.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.570+0000 7f301b734640 1 -- 172.21.15.16:0/838511076 wait complete. 2023-05-22T15:27:06.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.570+0000 7f301b734640 1 Processor -- start 2023-05-22T15:27:06.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.570+0000 7f301b734640 1 -- start start 2023-05-22T15:27:06.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.570+0000 7f301b734640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f30140b2a40 0x7f30140c6170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:06.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.570+0000 7f301b734640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f30140cfa50 0x7f3014157310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:06.587 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.570+0000 7f301b734640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f301415c130 0x7f3014157850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:06.587 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.570+0000 7f301b734640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f301413fe20 con 0x7f30140b2a40 2023-05-22T15:27:06.587 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.570+0000 7f301b734640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3014158140 con 0x7f30140cfa50 2023-05-22T15:27:06.587 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.570+0000 7f301b734640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3014140670 con 0x7f301415c130 2023-05-22T15:27:06.588 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.570+0000 7f30194d0640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f30140b2a40 0x7f30140c6170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:06.588 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.570+0000 7f30194d0640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f30140b2a40 0x7f30140c6170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57684/0 (socket says 172.21.15.16:57684) 2023-05-22T15:27:06.588 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.570+0000 7f30194d0640 1 -- 172.21.15.16:0/405940504 learned_addr learned my addr 172.21.15.16:0/405940504 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:06.589 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.570+0000 7f3019cd1640 1 --2- 172.21.15.16:0/405940504 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f301415c130 0x7f3014157850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:06.589 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.570+0000 7f3018ccf640 1 --2- 172.21.15.16:0/405940504 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f30140cfa50 0x7f3014157310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:06.589 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.574+0000 7f30194d0640 1 -- 172.21.15.16:0/405940504 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f301415c130 msgr2=0x7f3014157850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:06.589 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.574+0000 7f30194d0640 1 --2- 172.21.15.16:0/405940504 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f301415c130 0x7f3014157850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:06.590 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.574+0000 7f30194d0640 1 -- 172.21.15.16:0/405940504 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f30140cfa50 msgr2=0x7f3014157310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:06.590 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.574+0000 7f30194d0640 1 --2- 172.21.15.16:0/405940504 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f30140cfa50 0x7f3014157310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:06.590 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.574+0000 7f30194d0640 1 -- 172.21.15.16:0/405940504 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30040094d0 con 0x7f30140b2a40 2023-05-22T15:27:06.591 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.574+0000 7f3018ccf640 1 --2- 172.21.15.16:0/405940504 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f30140cfa50 0x7f3014157310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:27:06.591 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.574+0000 7f30194d0640 1 --2- 172.21.15.16:0/405940504 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f30140b2a40 0x7f30140c6170 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f301000c3a0 tx=0x7f301000c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:06.591 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.574+0000 7f3009ffb640 1 -- 172.21.15.16:0/405940504 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30100052d0 con 0x7f30140b2a40 2023-05-22T15:27:06.591 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.574+0000 7f3009ffb640 1 -- 172.21.15.16:0/405940504 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3010005430 con 0x7f30140b2a40 2023-05-22T15:27:06.592 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.574+0000 7f301b734640 1 -- 172.21.15.16:0/405940504 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30140cb600 con 0x7f30140b2a40 2023-05-22T15:27:06.592 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.574+0000 7f3009ffb640 1 -- 172.21.15.16:0/405940504 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3010007260 con 0x7f30140b2a40 2023-05-22T15:27:06.592 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.574+0000 7f301b734640 1 -- 172.21.15.16:0/405940504 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30140ce970 con 0x7f30140b2a40 2023-05-22T15:27:06.594 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.574+0000 7f301b734640 1 -- 172.21.15.16:0/405940504 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30140cb790 con 0x7f30140b2a40 2023-05-22T15:27:06.594 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.574+0000 7f3009ffb640 1 -- 172.21.15.16:0/405940504 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7f30100073c0 con 0x7f30140b2a40 2023-05-22T15:27:06.594 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.574+0000 7f3009ffb640 1 --2- 172.21.15.16:0/405940504 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f2ff8065380 0x7f2ff8067830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:06.594 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.574+0000 7f3009ffb640 1 -- 172.21.15.16:0/405940504 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f3010081e70 con 0x7f30140b2a40 2023-05-22T15:27:06.594 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.574+0000 7f3018ccf640 1 --2- 172.21.15.16:0/405940504 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f2ff8065380 0x7f2ff8067830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:06.595 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.574+0000 7f3018ccf640 1 --2- 172.21.15.16:0/405940504 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f2ff8065380 0x7f2ff8067830 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f30040af000 tx=0x7f3004008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:06.595 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.578+0000 7f3009ffb640 1 -- 172.21.15.16:0/405940504 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f3010049620 con 0x7f30140b2a40 2023-05-22T15:27:06.737 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.726+0000 7f301b734640 1 -- 172.21.15.16:0/405940504 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f301404dbe0 con 0x7f30140b2a40 2023-05-22T15:27:06.739 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.730+0000 7f3009ffb640 1 -- 172.21.15.16:0/405940504 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) v1 ==== 74+0+19604 (secure 0 0 0) 0x7f301004f7a0 con 0x7f30140b2a40 2023-05-22T15:27:06.741 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:27:06.741 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":27,"fsid":"b2719834-7847-4d98-90a6-f65d57f85c49","created":"2023-05-22T15:26:25.259314+0000","modified":"2023-05-22T15:26:53.971378+0000","last_up_change":"2023-05-22T15:26:36.448524+0000","last_in_change":"2023-05-22T15:26:27.675668+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-05-22T15:26:38.019781+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-05-22T15:26:43.570514+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-05-22T15:26:47.879159+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-05-22T15:26:48.522130+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data_ec","create_time":"2023-05-22T15:26:51.616142+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"9e7ff854-b25f-4cbb-a5c2-1799d9c53d2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":37021},{"type":"v1","addr":"172.21.15.16:6825","nonce":37021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":37021},{"type":"v1","addr":"172.21.15.16:6827","nonce":37021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":37021},{"type":"v1","addr":"172.21.15.16:6831","nonce":37021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":37021},{"type":"v1","addr":"172.21.15.16:6829","nonce":37021}]},"public_addr":"172.21.15.16:6825/37021","cluster_addr":"172.21.15.16:6827/37021","heartbeat_back_addr":"172.21.15.16:6831/37021","heartbeat_front_addr":"172.21.15.16:6829/37021","state":["exists","up"]},{"osd":1,"uuid":"4702fbc9-a927-4565-aca5-41dc69cc8264","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":37008},{"type":"v1","addr":"172.21.15.16:6801","nonce":37008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":37008},{"type":"v1","addr":"172.21.15.16:6803","nonce":37008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":37008},{"type":"v1","addr":"172.21.15.16:6807","nonce":37008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":37008},{"type":"v1","addr":"172.21.15.16:6805","nonce":37008}]},"public_addr":"172.21.15.16:6801/37008","cluster_addr":"172.21.15.16:6803/37008","heartbeat_back_addr":"172.21.15.16:6807/37008","heartbeat_front_addr":"172.21.15.16:6805/37008","state":["exists","up"]},{"osd":2,"uuid":"1e7bba46-4eba-47ef-bb81-5e2d371638ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":37009},{"type":"v1","addr":"172.21.15.16:6809","nonce":37009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":37009},{"type":"v1","addr":"172.21.15.16:6811","nonce":37009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":37009},{"type":"v1","addr":"172.21.15.16:6815","nonce":37009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":37009},{"type":"v1","addr":"172.21.15.16:6813","nonce":37009}]},"public_addr":"172.21.15.16:6809/37009","cluster_addr":"172.21.15.16:6811/37009","heartbeat_back_addr":"172.21.15.16:6815/37009","heartbeat_front_addr":"172.21.15.16:6813/37009","state":["exists","up"]},{"osd":3,"uuid":"383cacc0-5a88-4904-990e-498ca59ef48a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":37020},{"type":"v1","addr":"172.21.15.16:6817","nonce":37020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":37020},{"type":"v1","addr":"172.21.15.16:6819","nonce":37020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":37020},{"type":"v1","addr":"172.21.15.16:6823","nonce":37020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":37020},{"type":"v1","addr":"172.21.15.16:6821","nonce":37020}]},"public_addr":"172.21.15.16:6817/37020","cluster_addr":"172.21.15.16:6819/37020","heartbeat_back_addr":"172.21.15.16:6823/37020","heartbeat_front_addr":"172.21.15.16:6821/37020","state":["exists","up"]},{"osd":4,"uuid":"970500ef-95b7-42c1-b080-eed4fae7013d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":37353},{"type":"v1","addr":"172.21.15.62:6817","nonce":37353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6819","nonce":37353},{"type":"v1","addr":"172.21.15.62:6822","nonce":37353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37353},{"type":"v1","addr":"172.21.15.62:6831","nonce":37353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37353},{"type":"v1","addr":"172.21.15.62:6826","nonce":37353}]},"public_addr":"172.21.15.62:6817/37353","cluster_addr":"172.21.15.62:6822/37353","heartbeat_back_addr":"172.21.15.62:6831/37353","heartbeat_front_addr":"172.21.15.62:6826/37353","state":["exists","up"]},{"osd":5,"uuid":"022d67db-d4d8-445b-b1b9-8b446a00f5c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":37352},{"type":"v1","addr":"172.21.15.62:6801","nonce":37352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":37352},{"type":"v1","addr":"172.21.15.62:6803","nonce":37352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":37352},{"type":"v1","addr":"172.21.15.62:6807","nonce":37352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":37352},{"type":"v1","addr":"172.21.15.62:6805","nonce":37352}]},"public_addr":"172.21.15.62:6801/37352","cluster_addr":"172.21.15.62:6803/37352","heartbeat_back_addr":"172.21.15.62:6807/37352","heartbeat_front_addr":"172.21.15.62:6805/37352","state":["exists","up"]},{"osd":6,"uuid":"42bd372b-6688-4a1b-8407-e16e8c266e33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":37354},{"type":"v1","addr":"172.21.15.62:6820","nonce":37354}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":37354},{"type":"v1","addr":"172.21.15.62:6823","nonce":37354}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6829","nonce":37354},{"type":"v1","addr":"172.21.15.62:6830","nonce":37354}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6825","nonce":37354},{"type":"v1","addr":"172.21.15.62:6827","nonce":37354}]},"public_addr":"172.21.15.62:6820/37354","cluster_addr":"172.21.15.62:6823/37354","heartbeat_back_addr":"172.21.15.62:6830/37354","heartbeat_front_addr":"172.21.15.62:6827/37354","state":["exists","up"]},{"osd":7,"uuid":"ac70b2a3-f41b-4bf2-b227-7f2cbb55426d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37355},{"type":"v1","addr":"172.21.15.62:6809","nonce":37355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37355},{"type":"v1","addr":"172.21.15.62:6811","nonce":37355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":37355},{"type":"v1","addr":"172.21.15.62:6815","nonce":37355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":37355},{"type":"v1","addr":"172.21.15.62:6813","nonce":37355}]},"public_addr":"172.21.15.62:6809/37355","cluster_addr":"172.21.15.62:6811/37355","heartbeat_back_addr":"172.21.15.62:6815/37355","heartbeat_front_addr":"172.21.15.62:6813/37355","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.368896+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.107273+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.276888+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.287502+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.124791+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.251937+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.255679+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.232797+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-05-22T15:27:06.742 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.730+0000 7f301b734640 1 -- 172.21.15.16:0/405940504 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f2ff8065380 msgr2=0x7f2ff8067830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:06.742 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.730+0000 7f301b734640 1 --2- 172.21.15.16:0/405940504 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f2ff8065380 0x7f2ff8067830 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f30040af000 tx=0x7f3004008040 comp rx=0 tx=0).stop 2023-05-22T15:27:06.742 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.730+0000 7f301b734640 1 -- 172.21.15.16:0/405940504 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f30140b2a40 msgr2=0x7f30140c6170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:06.742 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.730+0000 7f301b734640 1 --2- 172.21.15.16:0/405940504 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f30140b2a40 0x7f30140c6170 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f301000c3a0 tx=0x7f301000c860 comp rx=0 tx=0).stop 2023-05-22T15:27:06.742 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.730+0000 7f301b734640 1 -- 172.21.15.16:0/405940504 shutdown_connections 2023-05-22T15:27:06.742 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.730+0000 7f301b734640 1 --2- 172.21.15.16:0/405940504 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f2ff8065380 0x7f2ff8067830 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:06.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.730+0000 7f301b734640 1 --2- 172.21.15.16:0/405940504 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f301415c130 0x7f3014157850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:06.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.730+0000 7f301b734640 1 --2- 172.21.15.16:0/405940504 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f30140cfa50 0x7f3014157310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:06.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.730+0000 7f301b734640 1 --2- 172.21.15.16:0/405940504 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f30140b2a40 0x7f30140c6170 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:06.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.730+0000 7f301b734640 1 -- 172.21.15.16:0/405940504 >> 172.21.15.16:0/405940504 conn(0x7f301406f5c0 msgr2=0x7f3014158df0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:06.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.730+0000 7f301b734640 1 -- 172.21.15.16:0/405940504 shutdown_connections 2023-05-22T15:27:06.744 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.730+0000 7f301b734640 1 -- 172.21.15.16:0/405940504 wait complete. 2023-05-22T15:27:06.758 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-05-22T15:27:06.759 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-05-22T15:27:06.868 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.858+0000 7f80ea479640 1 Processor -- start 2023-05-22T15:27:06.868 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.858+0000 7f80ea479640 1 -- start start 2023-05-22T15:27:06.869 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.858+0000 7f80ea479640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f80e40ccab0 0x7f80e415fe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:06.869 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.858+0000 7f80ea479640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f80e40d0410 0x7f80e40d0890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:06.869 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.858+0000 7f80ea479640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80e40ccef0 con 0x7f80e40b7dd0 2023-05-22T15:27:06.869 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.858+0000 7f80ea479640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80e40d3360 con 0x7f80e40d0410 2023-05-22T15:27:06.869 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.858+0000 7f80ea479640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80e40cd170 con 0x7f80e40ccab0 2023-05-22T15:27:06.870 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.858+0000 7f80e8a16640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f80e40d0410 0x7f80e40d0890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:06.870 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.858+0000 7f80e37fe640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f80e40ccab0 0x7f80e415fe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:06.870 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.858+0000 7f80e3fff640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f80e40b7dd0 0x7f80e415f700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.16:35950/0 (socket says 172.21.15.16:35950) 2023-05-22T15:27:06.870 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.858+0000 7f80e8a16640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f80e40d0410 0x7f80e40d0890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.16:37674/0 (socket says 172.21.15.16:37674) 2023-05-22T15:27:06.870 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.858+0000 7f80e3fff640 1 -- 172.21.15.16:0/1412569388 learned_addr learned my addr 172.21.15.16:0/1412569388 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:27:06.871 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.858+0000 7f80e37fe640 1 -- 172.21.15.16:0/1412569388 >> v2:172.21.15.62:3300/0 conn(0x7f80e40d0410 msgr2=0x7f80e40d0890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:06.871 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.858+0000 7f80e37fe640 1 --2- 172.21.15.16:0/1412569388 >> v2:172.21.15.62:3300/0 conn(0x7f80e40d0410 0x7f80e40d0890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:06.871 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.858+0000 7f80e37fe640 1 -- 172.21.15.16:0/1412569388 >> v1:172.21.15.62:6789/0 conn(0x7f80e40b7dd0 legacy=0x7f80e415f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:06.871 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.858+0000 7f80e37fe640 1 -- 172.21.15.16:0/1412569388 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80e40d2ac0 con 0x7f80e40ccab0 2023-05-22T15:27:06.871 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.862+0000 7f80e37fe640 1 --2- 172.21.15.16:0/1412569388 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f80e40ccab0 0x7f80e415fe10 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f80d80097c0 tx=0x7f80d801c450 comp rx=0 tx=0).ready entity=mon.2 client_cookie=34347a75f1d680f5 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:06.872 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.862+0000 7f80e2ffd640 1 -- 172.21.15.16:0/1412569388 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80d80ab650 con 0x7f80e40ccab0 2023-05-22T15:27:06.872 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.862+0000 7f80e2ffd640 1 -- 172.21.15.16:0/1412569388 <== mon.2 v2:172.21.15.62:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80d80ab7b0 con 0x7f80e40ccab0 2023-05-22T15:27:06.872 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.862+0000 7f80e2ffd640 1 -- 172.21.15.16:0/1412569388 <== mon.2 v2:172.21.15.62:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80d80abb40 con 0x7f80e40ccab0 2023-05-22T15:27:06.873 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.862+0000 7f80ea479640 1 -- 172.21.15.16:0/1412569388 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f80e40ccab0 msgr2=0x7f80e415fe10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:06.873 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.862+0000 7f80ea479640 1 --2- 172.21.15.16:0/1412569388 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f80e40ccab0 0x7f80e415fe10 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f80d80097c0 tx=0x7f80d801c450 comp rx=0 tx=0).stop 2023-05-22T15:27:06.873 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.862+0000 7f80ea479640 1 -- 172.21.15.16:0/1412569388 shutdown_connections 2023-05-22T15:27:06.874 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.862+0000 7f80ea479640 1 --2- 172.21.15.16:0/1412569388 >> v2:172.21.15.62:3300/0 conn(0x7f80e40d0410 0x7f80e40d0890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:06.874 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.862+0000 7f80ea479640 1 --2- 172.21.15.16:0/1412569388 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f80e40ccab0 0x7f80e415fe10 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:06.874 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.862+0000 7f80ea479640 1 -- 172.21.15.16:0/1412569388 >> 172.21.15.16:0/1412569388 conn(0x7f80e40655c0 msgr2=0x7f80e4070e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:06.874 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.862+0000 7f80ea479640 1 -- 172.21.15.16:0/1412569388 shutdown_connections 2023-05-22T15:27:06.874 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.862+0000 7f80ea479640 1 -- 172.21.15.16:0/1412569388 wait complete. 2023-05-22T15:27:06.876 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.866+0000 7f80ea479640 1 Processor -- start 2023-05-22T15:27:06.876 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.866+0000 7f80ea479640 1 -- start start 2023-05-22T15:27:06.876 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.866+0000 7f80ea479640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f80e40b7dd0 0x7f80e4154280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:06.876 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.866+0000 7f80ea479640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f80e40d0410 0x7f80e40c9f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:06.876 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.866+0000 7f80ea479640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f80e40ca4b0 0x7f80e4156500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:06.876 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.866+0000 7f80ea479640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80e4154f60 con 0x7f80e40d0410 2023-05-22T15:27:06.876 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.866+0000 7f80ea479640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80e41562d0 con 0x7f80e40b7dd0 2023-05-22T15:27:06.877 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.866+0000 7f80ea479640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80e41570e0 con 0x7f80e40ca4b0 2023-05-22T15:27:06.877 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.866+0000 7f80e37fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f80e40d0410 0x7f80e40c9f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:06.877 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.866+0000 7f80e37fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f80e40d0410 0x7f80e40c9f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57686/0 (socket says 172.21.15.16:57686) 2023-05-22T15:27:06.877 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.866+0000 7f80e37fe640 1 -- 172.21.15.16:0/4217886360 learned_addr learned my addr 172.21.15.16:0/4217886360 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:06.877 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.866+0000 7f80e8a16640 1 --2- 172.21.15.16:0/4217886360 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f80e40ca4b0 0x7f80e4156500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:06.877 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.866+0000 7f80e3fff640 1 --2- 172.21.15.16:0/4217886360 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f80e40b7dd0 0x7f80e4154280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:06.877 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.866+0000 7f80e37fe640 1 -- 172.21.15.16:0/4217886360 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f80e40ca4b0 msgr2=0x7f80e4156500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:06.878 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.866+0000 7f80e37fe640 1 --2- 172.21.15.16:0/4217886360 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f80e40ca4b0 0x7f80e4156500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:06.878 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.866+0000 7f80e37fe640 1 -- 172.21.15.16:0/4217886360 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f80e40b7dd0 msgr2=0x7f80e4154280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:06.878 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.866+0000 7f80e37fe640 1 --2- 172.21.15.16:0/4217886360 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f80e40b7dd0 0x7f80e4154280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:06.878 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.866+0000 7f80e37fe640 1 -- 172.21.15.16:0/4217886360 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80d8009420 con 0x7f80e40d0410 2023-05-22T15:27:06.878 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.866+0000 7f80e3fff640 1 --2- 172.21.15.16:0/4217886360 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f80e40b7dd0 0x7f80e4154280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T15:27:06.878 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.866+0000 7f80e37fe640 1 --2- 172.21.15.16:0/4217886360 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f80e40d0410 0x7f80e40c9f70 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f80d801ab00 tx=0x7f80d801ac90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:06.878 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.866+0000 7f80e0ff9640 1 -- 172.21.15.16:0/4217886360 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80d8004220 con 0x7f80e40d0410 2023-05-22T15:27:06.879 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.866+0000 7f80e0ff9640 1 -- 172.21.15.16:0/4217886360 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80d8004380 con 0x7f80e40d0410 2023-05-22T15:27:06.879 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.866+0000 7f80ea479640 1 -- 172.21.15.16:0/4217886360 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80e4160350 con 0x7f80e40d0410 2023-05-22T15:27:06.879 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.866+0000 7f80e0ff9640 1 -- 172.21.15.16:0/4217886360 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80d8003820 con 0x7f80e40d0410 2023-05-22T15:27:06.879 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.866+0000 7f80ea479640 1 -- 172.21.15.16:0/4217886360 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80e40cf730 con 0x7f80e40d0410 2023-05-22T15:27:06.880 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.866+0000 7f80ea479640 1 -- 172.21.15.16:0/4217886360 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80e41604e0 con 0x7f80e40d0410 2023-05-22T15:27:06.880 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.866+0000 7f80e0ff9640 1 -- 172.21.15.16:0/4217886360 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7f80d8003980 con 0x7f80e40d0410 2023-05-22T15:27:06.880 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.866+0000 7f80e0ff9640 1 --2- 172.21.15.16:0/4217886360 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f80d0065380 0x7f80d0067830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:06.880 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.866+0000 7f80e0ff9640 1 -- 172.21.15.16:0/4217886360 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f80d811cd70 con 0x7f80e40d0410 2023-05-22T15:27:06.880 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.866+0000 7f80e3fff640 1 --2- 172.21.15.16:0/4217886360 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f80d0065380 0x7f80d0067830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:06.880 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.870+0000 7f80e3fff640 1 --2- 172.21.15.16:0/4217886360 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f80d0065380 0x7f80d0067830 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f80d4005890 tx=0x7f80d4009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:06.881 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:06.870+0000 7f80e0ff9640 1 -- 172.21.15.16:0/4217886360 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f80d80ea840 con 0x7f80e40d0410 2023-05-22T15:27:07.033 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.022+0000 7f80ea479640 1 -- 172.21.15.16:0/4217886360 --> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f80e404dbe0 con 0x7f80d0065380 2023-05-22T15:27:07.048 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.038+0000 7f80e0ff9640 1 -- 172.21.15.16:0/4217886360 <== mgr.4101 v2:172.21.15.16:6832/36921 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+367319 (secure 0 0 0) 0x7f80e404dbe0 con 0x7f80d0065380 2023-05-22T15:27:07.051 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:27:07.054 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2023-05-22T15:27:07.055 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.042+0000 7f80ea479640 1 -- 172.21.15.16:0/4217886360 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f80d0065380 msgr2=0x7f80d0067830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:07.056 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.042+0000 7f80ea479640 1 --2- 172.21.15.16:0/4217886360 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f80d0065380 0x7f80d0067830 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f80d4005890 tx=0x7f80d4009040 comp rx=0 tx=0).stop 2023-05-22T15:27:07.056 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.042+0000 7f80ea479640 1 -- 172.21.15.16:0/4217886360 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f80e40d0410 msgr2=0x7f80e40c9f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:07.056 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.042+0000 7f80ea479640 1 --2- 172.21.15.16:0/4217886360 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f80e40d0410 0x7f80e40c9f70 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f80d801ab00 tx=0x7f80d801ac90 comp rx=0 tx=0).stop 2023-05-22T15:27:07.056 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.042+0000 7f80ea479640 1 -- 172.21.15.16:0/4217886360 shutdown_connections 2023-05-22T15:27:07.056 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.042+0000 7f80ea479640 1 --2- 172.21.15.16:0/4217886360 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f80d0065380 0x7f80d0067830 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:07.056 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.042+0000 7f80ea479640 1 --2- 172.21.15.16:0/4217886360 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f80e40ca4b0 0x7f80e4156500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:07.057 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.042+0000 7f80ea479640 1 --2- 172.21.15.16:0/4217886360 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f80e40d0410 0x7f80e40c9f70 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:07.057 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.042+0000 7f80ea479640 1 --2- 172.21.15.16:0/4217886360 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f80e40b7dd0 0x7f80e4154280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:07.057 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.042+0000 7f80ea479640 1 -- 172.21.15.16:0/4217886360 >> 172.21.15.16:0/4217886360 conn(0x7f80e40655c0 msgr2=0x7f80e4070e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:07.057 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.046+0000 7f80ea479640 1 -- 172.21.15.16:0/4217886360 shutdown_connections 2023-05-22T15:27:07.057 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.046+0000 7f80ea479640 1 -- 172.21.15.16:0/4217886360 wait complete. 2023-05-22T15:27:07.072 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":35,"stamp":"2023-05-22T15:27:05.949158+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":594754,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":206,"num_write_kb":2433,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":158,"ondisk_log_size":158,"up":338,"acting":338,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":338,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":75688,"kb_used_data":3496,"kb_used_omap":0,"kb_used_meta":72128,"kb_avail":754899032,"statfs":{"total":773094113280,"available":773016608768,"internally_reserved":0,"allocated":3579904,"data_stored":2148716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":73859072},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2056,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.973981"},"pg_stats":[{"pgid":"5.19","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.979395+0000","last_change":"2023-05-22T15:26:53.651924+0000","last_active":"2023-05-22T15:26:53.979395+0000","last_peered":"2023-05-22T15:26:53.979395+0000","last_clean":"2023-05-22T15:26:53.979395+0000","last_became_active":"2023-05-22T15:26:53.651533+0000","last_became_peered":"2023-05-22T15:26:53.651533+0000","last_unstale":"2023-05-22T15:26:53.979395+0000","last_undegraded":"2023-05-22T15:26:53.979395+0000","last_fullsized":"2023-05-22T15:26:53.979395+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_clean_scrub_stamp":"2023-05-22T15:26:52.568670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:27:41.065402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7,2],"acting":[1,5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"22'4","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.612099+0000","last_change":"2023-05-22T15:26:49.186238+0000","last_active":"2023-05-22T15:26:53.612099+0000","last_peered":"2023-05-22T15:26:53.612099+0000","last_clean":"2023-05-22T15:26:53.612099+0000","last_became_active":"2023-05-22T15:26:49.185455+0000","last_became_peered":"2023-05-22T15:26:49.185455+0000","last_unstale":"2023-05-22T15:26:53.612099+0000","last_undegraded":"2023-05-22T15:26:53.612099+0000","last_fullsized":"2023-05-22T15:26:53.612099+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_clean_scrub_stamp":"2023-05-22T15:26:48.127413+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:21:54.376718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":566,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.612858+0000","last_change":"2023-05-22T15:26:49.622435+0000","last_active":"2023-05-22T15:26:53.612858+0000","last_peered":"2023-05-22T15:26:53.612858+0000","last_clean":"2023-05-22T15:26:53.612858+0000","last_became_active":"2023-05-22T15:26:49.621643+0000","last_became_peered":"2023-05-22T15:26:49.621643+0000","last_unstale":"2023-05-22T15:26:53.612858+0000","last_undegraded":"2023-05-22T15:26:53.612858+0000","last_fullsized":"2023-05-22T15:26:53.612858+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:08:07.637726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-05-22T15:26:54.054128+0000","last_change":"2023-05-22T15:26:53.654741+0000","last_active":"2023-05-22T15:26:54.054128+0000","last_peered":"2023-05-22T15:26:54.054128+0000","last_clean":"2023-05-22T15:26:54.054128+0000","last_became_active":"2023-05-22T15:26:53.653669+0000","last_became_peered":"2023-05-22T15:26:53.653669+0000","last_unstale":"2023-05-22T15:26:54.054128+0000","last_undegraded":"2023-05-22T15:26:54.054128+0000","last_fullsized":"2023-05-22T15:26:54.054128+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_clean_scrub_stamp":"2023-05-22T15:26:52.568670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:01:12.890843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,2],"acting":[4,6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"22'1","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.602985+0000","last_change":"2023-05-22T15:26:49.187902+0000","last_active":"2023-05-22T15:26:53.602985+0000","last_peered":"2023-05-22T15:26:53.602985+0000","last_clean":"2023-05-22T15:26:53.602985+0000","last_became_active":"2023-05-22T15:26:49.187361+0000","last_became_peered":"2023-05-22T15:26:49.187361+0000","last_unstale":"2023-05-22T15:26:53.602985+0000","last_undegraded":"2023-05-22T15:26:53.602985+0000","last_fullsized":"2023-05-22T15:26:53.602985+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_clean_scrub_stamp":"2023-05-22T15:26:48.127413+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T18:13:58.291903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.603896+0000","last_change":"2023-05-22T15:26:49.560606+0000","last_active":"2023-05-22T15:26:53.603896+0000","last_peered":"2023-05-22T15:26:53.603896+0000","last_clean":"2023-05-22T15:26:53.603896+0000","last_became_active":"2023-05-22T15:26:49.560318+0000","last_became_peered":"2023-05-22T15:26:49.560318+0000","last_unstale":"2023-05-22T15:26:53.603896+0000","last_undegraded":"2023-05-22T15:26:53.603896+0000","last_fullsized":"2023-05-22T15:26:53.603896+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:43:00.518715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-05-22T15:26:54.055130+0000","last_change":"2023-05-22T15:26:53.648550+0000","last_active":"2023-05-22T15:26:54.055130+0000","last_peered":"2023-05-22T15:26:54.055130+0000","last_clean":"2023-05-22T15:26:54.055130+0000","last_became_active":"2023-05-22T15:26:53.646940+0000","last_became_peered":"2023-05-22T15:26:53.646940+0000","last_unstale":"2023-05-22T15:26:54.055130+0000","last_undegraded":"2023-05-22T15:26:54.055130+0000","last_fullsized":"2023-05-22T15:26:54.055130+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_clean_scrub_stamp":"2023-05-22T15:26:52.568670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T15:41:55.740195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7,4],"acting":[5,0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"22'2","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.634944+0000","last_change":"2023-05-22T15:26:49.210260+0000","last_active":"2023-05-22T15:26:53.634944+0000","last_peered":"2023-05-22T15:26:53.634944+0000","last_clean":"2023-05-22T15:26:53.634944+0000","last_became_active":"2023-05-22T15:26:49.209487+0000","last_became_peered":"2023-05-22T15:26:49.209487+0000","last_unstale":"2023-05-22T15:26:53.634944+0000","last_undegraded":"2023-05-22T15:26:53.634944+0000","last_fullsized":"2023-05-22T15:26:53.634944+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_clean_scrub_stamp":"2023-05-22T15:26:48.127413+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:07:49.933860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.619548+0000","last_change":"2023-05-22T15:26:49.595133+0000","last_active":"2023-05-22T15:26:53.619548+0000","last_peered":"2023-05-22T15:26:53.619548+0000","last_clean":"2023-05-22T15:26:53.619548+0000","last_became_active":"2023-05-22T15:26:49.594382+0000","last_became_peered":"2023-05-22T15:26:49.594382+0000","last_unstale":"2023-05-22T15:26:53.619548+0000","last_undegraded":"2023-05-22T15:26:53.619548+0000","last_fullsized":"2023-05-22T15:26:53.619548+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:11:42.159966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.982645+0000","last_change":"2023-05-22T15:26:53.657958+0000","last_active":"2023-05-22T15:26:53.982645+0000","last_peered":"2023-05-22T15:26:53.982645+0000","last_clean":"2023-05-22T15:26:53.982645+0000","last_became_active":"2023-05-22T15:26:53.656910+0000","last_became_peered":"2023-05-22T15:26:53.656910+0000","last_unstale":"2023-05-22T15:26:53.982645+0000","last_undegraded":"2023-05-22T15:26:53.982645+0000","last_fullsized":"2023-05-22T15:26:53.982645+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_clean_scrub_stamp":"2023-05-22T15:26:52.568670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:56:11.423640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1,6],"acting":[7,4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"22'2","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.634146+0000","last_change":"2023-05-22T15:26:49.210149+0000","last_active":"2023-05-22T15:26:53.634146+0000","last_peered":"2023-05-22T15:26:53.634146+0000","last_clean":"2023-05-22T15:26:53.634146+0000","last_became_active":"2023-05-22T15:26:49.209497+0000","last_became_peered":"2023-05-22T15:26:49.209497+0000","last_unstale":"2023-05-22T15:26:53.634146+0000","last_undegraded":"2023-05-22T15:26:53.634146+0000","last_fullsized":"2023-05-22T15:26:53.634146+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_clean_scrub_stamp":"2023-05-22T15:26:48.127413+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:47:17.588354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.620540+0000","last_change":"2023-05-22T15:26:49.598141+0000","last_active":"2023-05-22T15:26:53.620540+0000","last_peered":"2023-05-22T15:26:53.620540+0000","last_clean":"2023-05-22T15:26:53.620540+0000","last_became_active":"2023-05-22T15:26:49.597262+0000","last_became_peered":"2023-05-22T15:26:49.597262+0000","last_unstale":"2023-05-22T15:26:53.620540+0000","last_undegraded":"2023-05-22T15:26:53.620540+0000","last_fullsized":"2023-05-22T15:26:53.620540+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:33:23.357846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.980103+0000","last_change":"2023-05-22T15:26:53.656672+0000","last_active":"2023-05-22T15:26:53.980103+0000","last_peered":"2023-05-22T15:26:53.980103+0000","last_clean":"2023-05-22T15:26:53.980103+0000","last_became_active":"2023-05-22T15:26:53.656058+0000","last_became_peered":"2023-05-22T15:26:53.656058+0000","last_unstale":"2023-05-22T15:26:53.980103+0000","last_undegraded":"2023-05-22T15:26:53.980103+0000","last_fullsized":"2023-05-22T15:26:53.980103+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_clean_scrub_stamp":"2023-05-22T15:26:52.568670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:38:53.647917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,0],"acting":[1,3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"22'1","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.612141+0000","last_change":"2023-05-22T15:26:49.206147+0000","last_active":"2023-05-22T15:26:53.612141+0000","last_peered":"2023-05-22T15:26:53.612141+0000","last_clean":"2023-05-22T15:26:53.612141+0000","last_became_active":"2023-05-22T15:26:49.205706+0000","last_became_peered":"2023-05-22T15:26:49.205706+0000","last_unstale":"2023-05-22T15:26:53.612141+0000","last_undegraded":"2023-05-22T15:26:53.612141+0000","last_fullsized":"2023-05-22T15:26:53.612141+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_clean_scrub_stamp":"2023-05-22T15:26:48.127413+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:09:00.816941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.601812+0000","last_change":"2023-05-22T15:26:49.578140+0000","last_active":"2023-05-22T15:26:53.601812+0000","last_peered":"2023-05-22T15:26:53.601812+0000","last_clean":"2023-05-22T15:26:53.601812+0000","last_became_active":"2023-05-22T15:26:49.576922+0000","last_became_peered":"2023-05-22T15:26:49.576922+0000","last_unstale":"2023-05-22T15:26:53.601812+0000","last_undegraded":"2023-05-22T15:26:53.601812+0000","last_fullsized":"2023-05-22T15:26:53.601812+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T16:16:40.288204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-05-22T15:26:54.054907+0000","last_change":"2023-05-22T15:26:53.643098+0000","last_active":"2023-05-22T15:26:54.054907+0000","last_peered":"2023-05-22T15:26:54.054907+0000","last_clean":"2023-05-22T15:26:54.054907+0000","last_became_active":"2023-05-22T15:26:53.642038+0000","last_became_peered":"2023-05-22T15:26:53.642038+0000","last_unstale":"2023-05-22T15:26:54.054907+0000","last_undegraded":"2023-05-22T15:26:54.054907+0000","last_fullsized":"2023-05-22T15:26:54.054907+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_clean_scrub_stamp":"2023-05-22T15:26:52.568670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T18:43:09.309787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2,0],"acting":[4,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"22'1","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.620161+0000","last_change":"2023-05-22T15:26:49.199743+0000","last_active":"2023-05-22T15:26:53.620161+0000","last_peered":"2023-05-22T15:26:53.620161+0000","last_clean":"2023-05-22T15:26:53.620161+0000","last_became_active":"2023-05-22T15:26:49.197695+0000","last_became_peered":"2023-05-22T15:26:49.197695+0000","last_unstale":"2023-05-22T15:26:53.620161+0000","last_undegraded":"2023-05-22T15:26:53.620161+0000","last_fullsized":"2023-05-22T15:26:53.620161+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_clean_scrub_stamp":"2023-05-22T15:26:48.127413+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:21:54.376718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.602293+0000","last_change":"2023-05-22T15:26:49.580932+0000","last_active":"2023-05-22T15:26:53.602293+0000","last_peered":"2023-05-22T15:26:53.602293+0000","last_clean":"2023-05-22T15:26:53.602293+0000","last_became_active":"2023-05-22T15:26:49.580612+0000","last_became_peered":"2023-05-22T15:26:49.580612+0000","last_unstale":"2023-05-22T15:26:53.602293+0000","last_undegraded":"2023-05-22T15:26:53.602293+0000","last_fullsized":"2023-05-22T15:26:53.602293+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:19:30.391062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-05-22T15:26:54.051798+0000","last_change":"2023-05-22T15:26:53.662170+0000","last_active":"2023-05-22T15:26:54.051798+0000","last_peered":"2023-05-22T15:26:54.051798+0000","last_clean":"2023-05-22T15:26:54.051798+0000","last_became_active":"2023-05-22T15:26:53.661164+0000","last_became_peered":"2023-05-22T15:26:53.661164+0000","last_unstale":"2023-05-22T15:26:54.051798+0000","last_undegraded":"2023-05-22T15:26:54.051798+0000","last_fullsized":"2023-05-22T15:26:54.051798+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_clean_scrub_stamp":"2023-05-22T15:26:52.568670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:46:53.737748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3,7],"acting":[6,4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.613865+0000","last_change":"2023-05-22T15:26:49.187546+0000","last_active":"2023-05-22T15:26:53.613865+0000","last_peered":"2023-05-22T15:26:53.613865+0000","last_clean":"2023-05-22T15:26:53.613865+0000","last_became_active":"2023-05-22T15:26:49.187022+0000","last_became_peered":"2023-05-22T15:26:49.187022+0000","last_unstale":"2023-05-22T15:26:53.613865+0000","last_undegraded":"2023-05-22T15:26:53.613865+0000","last_fullsized":"2023-05-22T15:26:53.613865+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_clean_scrub_stamp":"2023-05-22T15:26:48.127413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:53:34.802286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.611910+0000","last_change":"2023-05-22T15:26:49.623535+0000","last_active":"2023-05-22T15:26:53.611910+0000","last_peered":"2023-05-22T15:26:53.611910+0000","last_clean":"2023-05-22T15:26:53.611910+0000","last_became_active":"2023-05-22T15:26:49.622454+0000","last_became_peered":"2023-05-22T15:26:49.622454+0000","last_unstale":"2023-05-22T15:26:53.611910+0000","last_undegraded":"2023-05-22T15:26:53.611910+0000","last_fullsized":"2023-05-22T15:26:53.611910+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:43:00.518715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-05-22T15:26:54.048982+0000","last_change":"2023-05-22T15:26:53.645181+0000","last_active":"2023-05-22T15:26:54.048982+0000","last_peered":"2023-05-22T15:26:54.048982+0000","last_clean":"2023-05-22T15:26:54.048982+0000","last_became_active":"2023-05-22T15:26:53.644764+0000","last_became_peered":"2023-05-22T15:26:53.644764+0000","last_unstale":"2023-05-22T15:26:54.048982+0000","last_undegraded":"2023-05-22T15:26:54.048982+0000","last_fullsized":"2023-05-22T15:26:54.048982+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_clean_scrub_stamp":"2023-05-22T15:26:52.568670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:12:09.126657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1,2],"acting":[0,7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.603942+0000","last_change":"2023-05-22T15:26:49.188920+0000","last_active":"2023-05-22T15:26:53.603942+0000","last_peered":"2023-05-22T15:26:53.603942+0000","last_clean":"2023-05-22T15:26:53.603942+0000","last_became_active":"2023-05-22T15:26:49.188538+0000","last_became_peered":"2023-05-22T15:26:49.188538+0000","last_unstale":"2023-05-22T15:26:53.603942+0000","last_undegraded":"2023-05-22T15:26:53.603942+0000","last_fullsized":"2023-05-22T15:26:53.603942+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_clean_scrub_stamp":"2023-05-22T15:26:48.127413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:09:00.816941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.603003+0000","last_change":"2023-05-22T15:26:49.606092+0000","last_active":"2023-05-22T15:26:53.603003+0000","last_peered":"2023-05-22T15:26:53.603003+0000","last_clean":"2023-05-22T15:26:53.603003+0000","last_became_active":"2023-05-22T15:26:49.605324+0000","last_became_peered":"2023-05-22T15:26:49.605324+0000","last_unstale":"2023-05-22T15:26:53.603003+0000","last_undegraded":"2023-05-22T15:26:53.603003+0000","last_fullsized":"2023-05-22T15:26:53.603003+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:10:51.435508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.603748+0000","last_change":"2023-05-22T15:26:49.578105+0000","last_active":"2023-05-22T15:26:53.603748+0000","last_peered":"2023-05-22T15:26:53.603748+0000","last_clean":"2023-05-22T15:26:53.603748+0000","last_became_active":"2023-05-22T15:26:49.577534+0000","last_became_peered":"2023-05-22T15:26:49.577534+0000","last_unstale":"2023-05-22T15:26:53.603748+0000","last_undegraded":"2023-05-22T15:26:53.603748+0000","last_fullsized":"2023-05-22T15:26:53.603748+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:44:52.369716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.612292+0000","last_change":"2023-05-22T15:26:49.186504+0000","last_active":"2023-05-22T15:26:53.612292+0000","last_peered":"2023-05-22T15:26:53.612292+0000","last_clean":"2023-05-22T15:26:53.612292+0000","last_became_active":"2023-05-22T15:26:49.185880+0000","last_became_peered":"2023-05-22T15:26:49.185880+0000","last_unstale":"2023-05-22T15:26:53.612292+0000","last_undegraded":"2023-05-22T15:26:53.612292+0000","last_fullsized":"2023-05-22T15:26:53.612292+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_clean_scrub_stamp":"2023-05-22T15:26:48.127413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T18:13:58.291903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-05-22T15:26:54.051126+0000","last_change":"2023-05-22T15:26:53.647905+0000","last_active":"2023-05-22T15:26:54.051126+0000","last_peered":"2023-05-22T15:26:54.051126+0000","last_clean":"2023-05-22T15:26:54.051126+0000","last_became_active":"2023-05-22T15:26:53.646790+0000","last_became_peered":"2023-05-22T15:26:53.646790+0000","last_unstale":"2023-05-22T15:26:54.051126+0000","last_undegraded":"2023-05-22T15:26:54.051126+0000","last_fullsized":"2023-05-22T15:26:54.051126+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_clean_scrub_stamp":"2023-05-22T15:26:52.568670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T03:18:03.204915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7,1],"acting":[6,4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.603724+0000","last_change":"2023-05-22T15:26:49.623613+0000","last_active":"2023-05-22T15:26:53.603724+0000","last_peered":"2023-05-22T15:26:53.603724+0000","last_clean":"2023-05-22T15:26:53.603724+0000","last_became_active":"2023-05-22T15:26:49.623385+0000","last_became_peered":"2023-05-22T15:26:49.623385+0000","last_unstale":"2023-05-22T15:26:53.603724+0000","last_undegraded":"2023-05-22T15:26:53.603724+0000","last_fullsized":"2023-05-22T15:26:53.603724+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:22:10.117410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.635389+0000","last_change":"2023-05-22T15:26:49.178387+0000","last_active":"2023-05-22T15:26:53.635389+0000","last_peered":"2023-05-22T15:26:53.635389+0000","last_clean":"2023-05-22T15:26:53.635389+0000","last_became_active":"2023-05-22T15:26:49.177428+0000","last_became_peered":"2023-05-22T15:26:49.177428+0000","last_unstale":"2023-05-22T15:26:53.635389+0000","last_undegraded":"2023-05-22T15:26:53.635389+0000","last_fullsized":"2023-05-22T15:26:53.635389+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_clean_scrub_stamp":"2023-05-22T15:26:48.127413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:07:41.791660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.981431+0000","last_change":"2023-05-22T15:26:53.660574+0000","last_active":"2023-05-22T15:26:53.981431+0000","last_peered":"2023-05-22T15:26:53.981431+0000","last_clean":"2023-05-22T15:26:53.981431+0000","last_became_active":"2023-05-22T15:26:53.659567+0000","last_became_peered":"2023-05-22T15:26:53.659567+0000","last_unstale":"2023-05-22T15:26:53.981431+0000","last_undegraded":"2023-05-22T15:26:53.981431+0000","last_fullsized":"2023-05-22T15:26:53.981431+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_clean_scrub_stamp":"2023-05-22T15:26:52.568670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:07:24.608031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6,0],"acting":[7,4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.15","version":"22'3","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.610146+0000","last_change":"2023-05-22T15:26:49.188354+0000","last_active":"2023-05-22T15:26:53.610146+0000","last_peered":"2023-05-22T15:26:53.610146+0000","last_clean":"2023-05-22T15:26:53.610146+0000","last_became_active":"2023-05-22T15:26:49.187902+0000","last_became_peered":"2023-05-22T15:26:49.187902+0000","last_unstale":"2023-05-22T15:26:53.610146+0000","last_undegraded":"2023-05-22T15:26:53.610146+0000","last_fullsized":"2023-05-22T15:26:53.610146+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_clean_scrub_stamp":"2023-05-22T15:26:48.127413+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:07:49.933860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.612914+0000","last_change":"2023-05-22T15:26:49.619577+0000","last_active":"2023-05-22T15:26:53.612914+0000","last_peered":"2023-05-22T15:26:53.612914+0000","last_clean":"2023-05-22T15:26:53.612914+0000","last_became_active":"2023-05-22T15:26:49.619210+0000","last_became_peered":"2023-05-22T15:26:49.619210+0000","last_unstale":"2023-05-22T15:26:53.612914+0000","last_undegraded":"2023-05-22T15:26:53.612914+0000","last_fullsized":"2023-05-22T15:26:53.612914+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:48:33.611833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.992745+0000","last_change":"2023-05-22T15:26:53.650745+0000","last_active":"2023-05-22T15:26:53.992745+0000","last_peered":"2023-05-22T15:26:53.992745+0000","last_clean":"2023-05-22T15:26:53.992745+0000","last_became_active":"2023-05-22T15:26:53.650209+0000","last_became_peered":"2023-05-22T15:26:53.650209+0000","last_unstale":"2023-05-22T15:26:53.992745+0000","last_undegraded":"2023-05-22T15:26:53.992745+0000","last_fullsized":"2023-05-22T15:26:53.992745+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_clean_scrub_stamp":"2023-05-22T15:26:52.568670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:15:56.755592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,5],"acting":[3,0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.14","version":"27'8","reported_seq":22,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-05-22T15:26:57.205645+0000","last_change":"2023-05-22T15:26:49.201484+0000","last_active":"2023-05-22T15:26:57.205645+0000","last_peered":"2023-05-22T15:26:57.205645+0000","last_clean":"2023-05-22T15:26:57.205645+0000","last_became_active":"2023-05-22T15:26:49.200972+0000","last_became_peered":"2023-05-22T15:26:49.200972+0000","last_unstale":"2023-05-22T15:26:57.205645+0000","last_undegraded":"2023-05-22T15:26:57.205645+0000","last_fullsized":"2023-05-22T15:26:57.205645+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_clean_scrub_stamp":"2023-05-22T15:26:48.127413+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:59:54.009838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3604,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.620650+0000","last_change":"2023-05-22T15:26:49.619268+0000","last_active":"2023-05-22T15:26:53.620650+0000","last_peered":"2023-05-22T15:26:53.620650+0000","last_clean":"2023-05-22T15:26:53.620650+0000","last_became_active":"2023-05-22T15:26:49.618724+0000","last_became_peered":"2023-05-22T15:26:49.618724+0000","last_unstale":"2023-05-22T15:26:53.620650+0000","last_undegraded":"2023-05-22T15:26:53.620650+0000","last_fullsized":"2023-05-22T15:26:53.620650+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:28:29.607131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.979878+0000","last_change":"2023-05-22T15:26:53.653181+0000","last_active":"2023-05-22T15:26:53.979878+0000","last_peered":"2023-05-22T15:26:53.979878+0000","last_clean":"2023-05-22T15:26:53.979878+0000","last_became_active":"2023-05-22T15:26:53.652676+0000","last_became_peered":"2023-05-22T15:26:53.652676+0000","last_unstale":"2023-05-22T15:26:53.979878+0000","last_undegraded":"2023-05-22T15:26:53.979878+0000","last_fullsized":"2023-05-22T15:26:53.979878+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_clean_scrub_stamp":"2023-05-22T15:26:52.568670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T15:46:20.544673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3,7],"acting":[1,5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.616683+0000","last_change":"2023-05-22T15:26:49.597250+0000","last_active":"2023-05-22T15:26:53.616683+0000","last_peered":"2023-05-22T15:26:53.616683+0000","last_clean":"2023-05-22T15:26:53.616683+0000","last_became_active":"2023-05-22T15:26:49.596487+0000","last_became_peered":"2023-05-22T15:26:49.596487+0000","last_unstale":"2023-05-22T15:26:53.616683+0000","last_undegraded":"2023-05-22T15:26:53.616683+0000","last_fullsized":"2023-05-22T15:26:53.616683+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:18:20.756810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.603054+0000","last_change":"2023-05-22T15:26:44.982421+0000","last_active":"2023-05-22T15:26:53.603054+0000","last_peered":"2023-05-22T15:26:53.603054+0000","last_clean":"2023-05-22T15:26:53.603054+0000","last_became_active":"2023-05-22T15:26:44.981586+0000","last_became_peered":"2023-05-22T15:26:44.981586+0000","last_unstale":"2023-05-22T15:26:53.603054+0000","last_undegraded":"2023-05-22T15:26:53.603054+0000","last_fullsized":"2023-05-22T15:26:53.603054+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:14:11.156544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"22'1","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.612191+0000","last_change":"2023-05-22T15:26:49.185412+0000","last_active":"2023-05-22T15:26:53.612191+0000","last_peered":"2023-05-22T15:26:53.612191+0000","last_clean":"2023-05-22T15:26:53.612191+0000","last_became_active":"2023-05-22T15:26:49.184896+0000","last_became_peered":"2023-05-22T15:26:49.184896+0000","last_unstale":"2023-05-22T15:26:53.612191+0000","last_undegraded":"2023-05-22T15:26:53.612191+0000","last_fullsized":"2023-05-22T15:26:53.612191+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_clean_scrub_stamp":"2023-05-22T15:26:48.127413+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:13:03.975127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.992364+0000","last_change":"2023-05-22T15:26:53.669441+0000","last_active":"2023-05-22T15:26:53.992364+0000","last_peered":"2023-05-22T15:26:53.992364+0000","last_clean":"2023-05-22T15:26:53.992364+0000","last_became_active":"2023-05-22T15:26:53.669000+0000","last_became_peered":"2023-05-22T15:26:53.669000+0000","last_unstale":"2023-05-22T15:26:53.992364+0000","last_undegraded":"2023-05-22T15:26:53.992364+0000","last_fullsized":"2023-05-22T15:26:53.992364+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_clean_scrub_stamp":"2023-05-22T15:26:52.568670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T18:55:06.398032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,0],"acting":[3,1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.603843+0000","last_change":"2023-05-22T15:26:49.623319+0000","last_active":"2023-05-22T15:26:53.603843+0000","last_peered":"2023-05-22T15:26:53.603843+0000","last_clean":"2023-05-22T15:26:53.603843+0000","last_became_active":"2023-05-22T15:26:49.622892+0000","last_became_peered":"2023-05-22T15:26:49.622892+0000","last_unstale":"2023-05-22T15:26:53.603843+0000","last_undegraded":"2023-05-22T15:26:53.603843+0000","last_fullsized":"2023-05-22T15:26:53.603843+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:48:33.611833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.612572+0000","last_change":"2023-05-22T15:26:44.978858+0000","last_active":"2023-05-22T15:26:53.612572+0000","last_peered":"2023-05-22T15:26:53.612572+0000","last_clean":"2023-05-22T15:26:53.612572+0000","last_became_active":"2023-05-22T15:26:44.978443+0000","last_became_peered":"2023-05-22T15:26:44.978443+0000","last_unstale":"2023-05-22T15:26:53.612572+0000","last_undegraded":"2023-05-22T15:26:53.612572+0000","last_fullsized":"2023-05-22T15:26:53.612572+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:46:29.290243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"22'2","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.606372+0000","last_change":"2023-05-22T15:26:49.188060+0000","last_active":"2023-05-22T15:26:53.606372+0000","last_peered":"2023-05-22T15:26:53.606372+0000","last_clean":"2023-05-22T15:26:53.606372+0000","last_became_active":"2023-05-22T15:26:49.187586+0000","last_became_peered":"2023-05-22T15:26:49.187586+0000","last_unstale":"2023-05-22T15:26:53.606372+0000","last_undegraded":"2023-05-22T15:26:53.606372+0000","last_fullsized":"2023-05-22T15:26:53.606372+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_clean_scrub_stamp":"2023-05-22T15:26:48.127413+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:59:54.009838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-05-22T15:26:54.053713+0000","last_change":"2023-05-22T15:26:53.659559+0000","last_active":"2023-05-22T15:26:54.053713+0000","last_peered":"2023-05-22T15:26:54.053713+0000","last_clean":"2023-05-22T15:26:54.053713+0000","last_became_active":"2023-05-22T15:26:53.655056+0000","last_became_peered":"2023-05-22T15:26:53.655056+0000","last_unstale":"2023-05-22T15:26:54.053713+0000","last_undegraded":"2023-05-22T15:26:54.053713+0000","last_fullsized":"2023-05-22T15:26:54.053713+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_clean_scrub_stamp":"2023-05-22T15:26:52.568670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:47:56.845713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7,0],"acting":[4,3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.611858+0000","last_change":"2023-05-22T15:26:49.589781+0000","last_active":"2023-05-22T15:26:53.611858+0000","last_peered":"2023-05-22T15:26:53.611858+0000","last_clean":"2023-05-22T15:26:53.611858+0000","last_became_active":"2023-05-22T15:26:49.588830+0000","last_became_peered":"2023-05-22T15:26:49.588830+0000","last_unstale":"2023-05-22T15:26:53.611858+0000","last_undegraded":"2023-05-22T15:26:53.611858+0000","last_fullsized":"2023-05-22T15:26:53.611858+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:52:48.123148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.615526+0000","last_change":"2023-05-22T15:26:45.092367+0000","last_active":"2023-05-22T15:26:53.615526+0000","last_peered":"2023-05-22T15:26:53.615526+0000","last_clean":"2023-05-22T15:26:53.615526+0000","last_became_active":"2023-05-22T15:26:45.091951+0000","last_became_peered":"2023-05-22T15:26:45.091951+0000","last_unstale":"2023-05-22T15:26:53.615526+0000","last_undegraded":"2023-05-22T15:26:53.615526+0000","last_fullsized":"2023-05-22T15:26:53.615526+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:42:57.973030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.619125+0000","last_change":"2023-05-22T15:26:49.180237+0000","last_active":"2023-05-22T15:26:53.619125+0000","last_peered":"2023-05-22T15:26:53.619125+0000","last_clean":"2023-05-22T15:26:53.619125+0000","last_became_active":"2023-05-22T15:26:49.179622+0000","last_became_peered":"2023-05-22T15:26:49.179622+0000","last_unstale":"2023-05-22T15:26:53.619125+0000","last_undegraded":"2023-05-22T15:26:53.619125+0000","last_fullsized":"2023-05-22T15:26:53.619125+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_clean_scrub_stamp":"2023-05-22T15:26:48.127413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T16:16:39.277093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-05-22T15:26:54.051370+0000","last_change":"2023-05-22T15:26:53.623285+0000","last_active":"2023-05-22T15:26:54.051370+0000","last_peered":"2023-05-22T15:26:54.051370+0000","last_clean":"2023-05-22T15:26:54.051370+0000","last_became_active":"2023-05-22T15:26:53.622116+0000","last_became_peered":"2023-05-22T15:26:53.622116+0000","last_unstale":"2023-05-22T15:26:54.051370+0000","last_undegraded":"2023-05-22T15:26:54.051370+0000","last_fullsized":"2023-05-22T15:26:54.051370+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_clean_scrub_stamp":"2023-05-22T15:26:52.568670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T16:31:40.104597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5,2],"acting":[6,0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.613088+0000","last_change":"2023-05-22T15:26:49.576833+0000","last_active":"2023-05-22T15:26:53.613088+0000","last_peered":"2023-05-22T15:26:53.613088+0000","last_clean":"2023-05-22T15:26:53.613088+0000","last_became_active":"2023-05-22T15:26:49.576559+0000","last_became_peered":"2023-05-22T15:26:49.576559+0000","last_unstale":"2023-05-22T15:26:53.613088+0000","last_undegraded":"2023-05-22T15:26:53.613088+0000","last_fullsized":"2023-05-22T15:26:53.613088+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:32:19.582542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.619335+0000","last_change":"2023-05-22T15:26:45.091475+0000","last_active":"2023-05-22T15:26:53.619335+0000","last_peered":"2023-05-22T15:26:53.619335+0000","last_clean":"2023-05-22T15:26:53.619335+0000","last_became_active":"2023-05-22T15:26:45.091143+0000","last_became_peered":"2023-05-22T15:26:45.091143+0000","last_unstale":"2023-05-22T15:26:53.619335+0000","last_undegraded":"2023-05-22T15:26:53.619335+0000","last_fullsized":"2023-05-22T15:26:53.619335+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:12:41.464054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.635151+0000","last_change":"2023-05-22T15:26:49.187061+0000","last_active":"2023-05-22T15:26:53.635151+0000","last_peered":"2023-05-22T15:26:53.635151+0000","last_clean":"2023-05-22T15:26:53.635151+0000","last_became_active":"2023-05-22T15:26:49.185965+0000","last_became_peered":"2023-05-22T15:26:49.185965+0000","last_unstale":"2023-05-22T15:26:53.635151+0000","last_undegraded":"2023-05-22T15:26:53.635151+0000","last_fullsized":"2023-05-22T15:26:53.635151+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_clean_scrub_stamp":"2023-05-22T15:26:48.127413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T16:16:39.277093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-05-22T15:26:54.048601+0000","last_change":"2023-05-22T15:26:53.650446+0000","last_active":"2023-05-22T15:26:54.048601+0000","last_peered":"2023-05-22T15:26:54.048601+0000","last_clean":"2023-05-22T15:26:54.048601+0000","last_became_active":"2023-05-22T15:26:53.649492+0000","last_became_peered":"2023-05-22T15:26:53.649492+0000","last_unstale":"2023-05-22T15:26:54.048601+0000","last_undegraded":"2023-05-22T15:26:54.048601+0000","last_fullsized":"2023-05-22T15:26:54.048601+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_clean_scrub_stamp":"2023-05-22T15:26:52.568670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:56:11.423640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6,5],"acting":[0,1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.619014+0000","last_change":"2023-05-22T15:26:49.581808+0000","last_active":"2023-05-22T15:26:53.619014+0000","last_peered":"2023-05-22T15:26:53.619014+0000","last_clean":"2023-05-22T15:26:53.619014+0000","last_became_active":"2023-05-22T15:26:49.581501+0000","last_became_peered":"2023-05-22T15:26:49.581501+0000","last_unstale":"2023-05-22T15:26:53.619014+0000","last_undegraded":"2023-05-22T15:26:53.619014+0000","last_fullsized":"2023-05-22T15:26:53.619014+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:52:48.123148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.620868+0000","last_change":"2023-05-22T15:26:49.204122+0000","last_active":"2023-05-22T15:26:53.620868+0000","last_peered":"2023-05-22T15:26:53.620868+0000","last_clean":"2023-05-22T15:26:53.620868+0000","last_became_active":"2023-05-22T15:26:49.199596+0000","last_became_peered":"2023-05-22T15:26:49.199596+0000","last_unstale":"2023-05-22T15:26:53.620868+0000","last_undegraded":"2023-05-22T15:26:53.620868+0000","last_fullsized":"2023-05-22T15:26:53.620868+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_clean_scrub_stamp":"2023-05-22T15:26:48.127413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:09:00.816941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.635108+0000","last_change":"2023-05-22T15:26:44.976632+0000","last_active":"2023-05-22T15:26:53.635108+0000","last_peered":"2023-05-22T15:26:53.635108+0000","last_clean":"2023-05-22T15:26:53.635108+0000","last_became_active":"2023-05-22T15:26:44.975546+0000","last_became_peered":"2023-05-22T15:26:44.975546+0000","last_unstale":"2023-05-22T15:26:53.635108+0000","last_undegraded":"2023-05-22T15:26:53.635108+0000","last_fullsized":"2023-05-22T15:26:53.635108+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:42:57.973030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-05-22T15:26:54.048781+0000","last_change":"2023-05-22T15:26:53.669482+0000","last_active":"2023-05-22T15:26:54.048781+0000","last_peered":"2023-05-22T15:26:54.048781+0000","last_clean":"2023-05-22T15:26:54.048781+0000","last_became_active":"2023-05-22T15:26:53.669119+0000","last_became_peered":"2023-05-22T15:26:53.669119+0000","last_unstale":"2023-05-22T15:26:54.048781+0000","last_undegraded":"2023-05-22T15:26:54.048781+0000","last_fullsized":"2023-05-22T15:26:54.048781+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_clean_scrub_stamp":"2023-05-22T15:26:52.568670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:38:53.647917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,6],"acting":[0,1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.613087+0000","last_change":"2023-05-22T15:26:49.589555+0000","last_active":"2023-05-22T15:26:53.613087+0000","last_peered":"2023-05-22T15:26:53.613087+0000","last_clean":"2023-05-22T15:26:53.613087+0000","last_became_active":"2023-05-22T15:26:49.589271+0000","last_became_peered":"2023-05-22T15:26:49.589271+0000","last_unstale":"2023-05-22T15:26:53.613087+0000","last_undegraded":"2023-05-22T15:26:53.613087+0000","last_fullsized":"2023-05-22T15:26:53.613087+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:43:00.518715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.614048+0000","last_change":"2023-05-22T15:26:49.179359+0000","last_active":"2023-05-22T15:26:53.614048+0000","last_peered":"2023-05-22T15:26:53.614048+0000","last_clean":"2023-05-22T15:26:53.614048+0000","last_became_active":"2023-05-22T15:26:49.178930+0000","last_became_peered":"2023-05-22T15:26:49.178930+0000","last_unstale":"2023-05-22T15:26:53.614048+0000","last_undegraded":"2023-05-22T15:26:53.614048+0000","last_fullsized":"2023-05-22T15:26:53.614048+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_clean_scrub_stamp":"2023-05-22T15:26:48.127413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:59:54.009838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.597765+0000","last_change":"2023-05-22T15:26:44.969469+0000","last_active":"2023-05-22T15:26:53.597765+0000","last_peered":"2023-05-22T15:26:53.597765+0000","last_clean":"2023-05-22T15:26:53.597765+0000","last_became_active":"2023-05-22T15:26:44.969083+0000","last_became_peered":"2023-05-22T15:26:44.969083+0000","last_unstale":"2023-05-22T15:26:53.597765+0000","last_undegraded":"2023-05-22T15:26:53.597765+0000","last_fullsized":"2023-05-22T15:26:53.597765+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:29:23.670067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.978603+0000","last_change":"2023-05-22T15:26:53.636902+0000","last_active":"2023-05-22T15:26:53.978603+0000","last_peered":"2023-05-22T15:26:53.978603+0000","last_clean":"2023-05-22T15:26:53.978603+0000","last_became_active":"2023-05-22T15:26:53.636337+0000","last_became_peered":"2023-05-22T15:26:53.636337+0000","last_unstale":"2023-05-22T15:26:53.978603+0000","last_undegraded":"2023-05-22T15:26:53.978603+0000","last_fullsized":"2023-05-22T15:26:53.978603+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_clean_scrub_stamp":"2023-05-22T15:26:52.568670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:58:01.437826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1,7],"acting":[2,5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.612001+0000","last_change":"2023-05-22T15:26:49.581103+0000","last_active":"2023-05-22T15:26:53.612001+0000","last_peered":"2023-05-22T15:26:53.612001+0000","last_clean":"2023-05-22T15:26:53.612001+0000","last_became_active":"2023-05-22T15:26:49.580823+0000","last_became_peered":"2023-05-22T15:26:49.580823+0000","last_unstale":"2023-05-22T15:26:53.612001+0000","last_undegraded":"2023-05-22T15:26:53.612001+0000","last_fullsized":"2023-05-22T15:26:53.612001+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:32:19.582542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.611473+0000","last_change":"2023-05-22T15:26:49.205877+0000","last_active":"2023-05-22T15:26:53.611473+0000","last_peered":"2023-05-22T15:26:53.611473+0000","last_clean":"2023-05-22T15:26:53.611473+0000","last_became_active":"2023-05-22T15:26:49.205432+0000","last_became_peered":"2023-05-22T15:26:49.205432+0000","last_unstale":"2023-05-22T15:26:53.611473+0000","last_undegraded":"2023-05-22T15:26:53.611473+0000","last_fullsized":"2023-05-22T15:26:53.611473+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_clean_scrub_stamp":"2023-05-22T15:26:48.127413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:59:54.009838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.609876+0000","last_change":"2023-05-22T15:26:44.977198+0000","last_active":"2023-05-22T15:26:53.609876+0000","last_peered":"2023-05-22T15:26:53.609876+0000","last_clean":"2023-05-22T15:26:53.609876+0000","last_became_active":"2023-05-22T15:26:44.976445+0000","last_became_peered":"2023-05-22T15:26:44.976445+0000","last_unstale":"2023-05-22T15:26:53.609876+0000","last_undegraded":"2023-05-22T15:26:53.609876+0000","last_fullsized":"2023-05-22T15:26:53.609876+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:12:41.464054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-05-22T15:26:54.056650+0000","last_change":"2023-05-22T15:26:53.667459+0000","last_active":"2023-05-22T15:26:54.056650+0000","last_peered":"2023-05-22T15:26:54.056650+0000","last_clean":"2023-05-22T15:26:54.056650+0000","last_became_active":"2023-05-22T15:26:53.666553+0000","last_became_peered":"2023-05-22T15:26:53.666553+0000","last_unstale":"2023-05-22T15:26:54.056650+0000","last_undegraded":"2023-05-22T15:26:54.056650+0000","last_fullsized":"2023-05-22T15:26:54.056650+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_clean_scrub_stamp":"2023-05-22T15:26:52.568670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:08:31.638326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0,4],"acting":[5,1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.602418+0000","last_change":"2023-05-22T15:26:49.606241+0000","last_active":"2023-05-22T15:26:53.602418+0000","last_peered":"2023-05-22T15:26:53.602418+0000","last_clean":"2023-05-22T15:26:53.602418+0000","last_became_active":"2023-05-22T15:26:49.605628+0000","last_became_peered":"2023-05-22T15:26:49.605628+0000","last_unstale":"2023-05-22T15:26:53.602418+0000","last_undegraded":"2023-05-22T15:26:53.602418+0000","last_fullsized":"2023-05-22T15:26:53.602418+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:59:23.805894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.602217+0000","last_change":"2023-05-22T15:26:49.185421+0000","last_active":"2023-05-22T15:26:53.602217+0000","last_peered":"2023-05-22T15:26:53.602217+0000","last_clean":"2023-05-22T15:26:53.602217+0000","last_became_active":"2023-05-22T15:26:49.184948+0000","last_became_peered":"2023-05-22T15:26:49.184948+0000","last_unstale":"2023-05-22T15:26:53.602217+0000","last_undegraded":"2023-05-22T15:26:53.602217+0000","last_fullsized":"2023-05-22T15:26:53.602217+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_clean_scrub_stamp":"2023-05-22T15:26:48.127413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:21:54.376718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.634293+0000","last_change":"2023-05-22T15:26:44.971951+0000","last_active":"2023-05-22T15:26:53.634293+0000","last_peered":"2023-05-22T15:26:53.634293+0000","last_clean":"2023-05-22T15:26:53.634293+0000","last_became_active":"2023-05-22T15:26:44.970872+0000","last_became_peered":"2023-05-22T15:26:44.970872+0000","last_unstale":"2023-05-22T15:26:53.634293+0000","last_undegraded":"2023-05-22T15:26:53.634293+0000","last_fullsized":"2023-05-22T15:26:53.634293+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:55:34.752949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":169,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.610124+0000","last_change":"2023-05-22T15:26:39.318204+0000","last_active":"2023-05-22T15:26:53.610124+0000","last_peered":"2023-05-22T15:26:53.610124+0000","last_clean":"2023-05-22T15:26:53.610124+0000","last_became_active":"2023-05-22T15:26:39.315370+0000","last_became_peered":"2023-05-22T15:26:39.315370+0000","last_unstale":"2023-05-22T15:26:53.610124+0000","last_undegraded":"2023-05-22T15:26:53.610124+0000","last_fullsized":"2023-05-22T15:26:53.610124+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:38.020076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:38.020076+0000","last_clean_scrub_stamp":"2023-05-22T15:26:38.020076+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:06:39.207543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.982716+0000","last_change":"2023-05-22T15:26:53.649648+0000","last_active":"2023-05-22T15:26:53.982716+0000","last_peered":"2023-05-22T15:26:53.982716+0000","last_clean":"2023-05-22T15:26:53.982716+0000","last_became_active":"2023-05-22T15:26:53.648664+0000","last_became_peered":"2023-05-22T15:26:53.648664+0000","last_unstale":"2023-05-22T15:26:53.982716+0000","last_undegraded":"2023-05-22T15:26:53.982716+0000","last_fullsized":"2023-05-22T15:26:53.982716+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_clean_scrub_stamp":"2023-05-22T15:26:52.568670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T03:14:12.733785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5,3],"acting":[7,2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.612919+0000","last_change":"2023-05-22T15:26:49.578502+0000","last_active":"2023-05-22T15:26:53.612919+0000","last_peered":"2023-05-22T15:26:53.612919+0000","last_clean":"2023-05-22T15:26:53.612919+0000","last_became_active":"2023-05-22T15:26:49.577743+0000","last_became_peered":"2023-05-22T15:26:49.577743+0000","last_unstale":"2023-05-22T15:26:53.612919+0000","last_undegraded":"2023-05-22T15:26:53.612919+0000","last_fullsized":"2023-05-22T15:26:53.612919+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:22:10.117410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.604031+0000","last_change":"2023-05-22T15:26:49.191706+0000","last_active":"2023-05-22T15:26:53.604031+0000","last_peered":"2023-05-22T15:26:53.604031+0000","last_clean":"2023-05-22T15:26:53.604031+0000","last_became_active":"2023-05-22T15:26:49.190396+0000","last_became_peered":"2023-05-22T15:26:49.190396+0000","last_unstale":"2023-05-22T15:26:53.604031+0000","last_undegraded":"2023-05-22T15:26:53.604031+0000","last_fullsized":"2023-05-22T15:26:53.604031+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_clean_scrub_stamp":"2023-05-22T15:26:48.127413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:13:03.975127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-05-22T15:26:54.055266+0000","last_change":"2023-05-22T15:26:53.656007+0000","last_active":"2023-05-22T15:26:54.055266+0000","last_peered":"2023-05-22T15:26:54.055266+0000","last_clean":"2023-05-22T15:26:54.055266+0000","last_became_active":"2023-05-22T15:26:53.652989+0000","last_became_peered":"2023-05-22T15:26:53.652989+0000","last_unstale":"2023-05-22T15:26:54.055266+0000","last_undegraded":"2023-05-22T15:26:54.055266+0000","last_fullsized":"2023-05-22T15:26:54.055266+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_clean_scrub_stamp":"2023-05-22T15:26:52.568670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:36:00.298083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0,2],"acting":[4,5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.621541+0000","last_change":"2023-05-22T15:26:49.622122+0000","last_active":"2023-05-22T15:26:53.621541+0000","last_peered":"2023-05-22T15:26:53.621541+0000","last_clean":"2023-05-22T15:26:53.621541+0000","last_became_active":"2023-05-22T15:26:49.621591+0000","last_became_peered":"2023-05-22T15:26:49.621591+0000","last_unstale":"2023-05-22T15:26:53.621541+0000","last_undegraded":"2023-05-22T15:26:53.621541+0000","last_fullsized":"2023-05-22T15:26:53.621541+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T18:05:09.939239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.617051+0000","last_change":"2023-05-22T15:26:49.200650+0000","last_active":"2023-05-22T15:26:53.617051+0000","last_peered":"2023-05-22T15:26:53.617051+0000","last_clean":"2023-05-22T15:26:53.617051+0000","last_became_active":"2023-05-22T15:26:49.199682+0000","last_became_peered":"2023-05-22T15:26:49.199682+0000","last_unstale":"2023-05-22T15:26:53.617051+0000","last_undegraded":"2023-05-22T15:26:53.617051+0000","last_fullsized":"2023-05-22T15:26:53.617051+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_clean_scrub_stamp":"2023-05-22T15:26:48.127413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:13:03.975127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-05-22T15:26:54.056477+0000","last_change":"2023-05-22T15:26:53.667847+0000","last_active":"2023-05-22T15:26:54.056477+0000","last_peered":"2023-05-22T15:26:54.056477+0000","last_clean":"2023-05-22T15:26:54.056477+0000","last_became_active":"2023-05-22T15:26:53.666866+0000","last_became_peered":"2023-05-22T15:26:53.666866+0000","last_unstale":"2023-05-22T15:26:54.056477+0000","last_undegraded":"2023-05-22T15:26:54.056477+0000","last_fullsized":"2023-05-22T15:26:54.056477+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_clean_scrub_stamp":"2023-05-22T15:26:52.568670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:30:40.980509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6,2],"acting":[5,4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.631889+0000","last_change":"2023-05-22T15:26:49.598177+0000","last_active":"2023-05-22T15:26:53.631889+0000","last_peered":"2023-05-22T15:26:53.631889+0000","last_clean":"2023-05-22T15:26:53.631889+0000","last_became_active":"2023-05-22T15:26:49.594848+0000","last_became_peered":"2023-05-22T15:26:49.594848+0000","last_unstale":"2023-05-22T15:26:53.631889+0000","last_undegraded":"2023-05-22T15:26:53.631889+0000","last_fullsized":"2023-05-22T15:26:53.631889+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T03:14:09.303640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.603009+0000","last_change":"2023-05-22T15:26:49.206461+0000","last_active":"2023-05-22T15:26:53.603009+0000","last_peered":"2023-05-22T15:26:53.603009+0000","last_clean":"2023-05-22T15:26:53.603009+0000","last_became_active":"2023-05-22T15:26:49.205870+0000","last_became_peered":"2023-05-22T15:26:49.205870+0000","last_unstale":"2023-05-22T15:26:53.603009+0000","last_undegraded":"2023-05-22T15:26:53.603009+0000","last_fullsized":"2023-05-22T15:26:53.603009+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_clean_scrub_stamp":"2023-05-22T15:26:48.127413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T18:13:58.291903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.980043+0000","last_change":"2023-05-22T15:26:53.642805+0000","last_active":"2023-05-22T15:26:53.980043+0000","last_peered":"2023-05-22T15:26:53.980043+0000","last_clean":"2023-05-22T15:26:53.980043+0000","last_became_active":"2023-05-22T15:26:53.642142+0000","last_became_peered":"2023-05-22T15:26:53.642142+0000","last_unstale":"2023-05-22T15:26:53.980043+0000","last_undegraded":"2023-05-22T15:26:53.980043+0000","last_fullsized":"2023-05-22T15:26:53.980043+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_clean_scrub_stamp":"2023-05-22T15:26:52.568670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:12:09.126657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0,6],"acting":[1,4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.623794+0000","last_change":"2023-05-22T15:26:49.600035+0000","last_active":"2023-05-22T15:26:53.623794+0000","last_peered":"2023-05-22T15:26:53.623794+0000","last_clean":"2023-05-22T15:26:53.623794+0000","last_became_active":"2023-05-22T15:26:49.597436+0000","last_became_peered":"2023-05-22T15:26:49.597436+0000","last_unstale":"2023-05-22T15:26:53.623794+0000","last_undegraded":"2023-05-22T15:26:53.623794+0000","last_fullsized":"2023-05-22T15:26:53.623794+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:46:50.307602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"22'2","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.606082+0000","last_change":"2023-05-22T15:26:49.191345+0000","last_active":"2023-05-22T15:26:53.606082+0000","last_peered":"2023-05-22T15:26:53.606082+0000","last_clean":"2023-05-22T15:26:53.606082+0000","last_became_active":"2023-05-22T15:26:49.190291+0000","last_became_peered":"2023-05-22T15:26:49.190291+0000","last_unstale":"2023-05-22T15:26:53.606082+0000","last_undegraded":"2023-05-22T15:26:53.606082+0000","last_fullsized":"2023-05-22T15:26:53.606082+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_clean_scrub_stamp":"2023-05-22T15:26:48.127413+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:13:21.177657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.979168+0000","last_change":"2023-05-22T15:26:53.649119+0000","last_active":"2023-05-22T15:26:53.979168+0000","last_peered":"2023-05-22T15:26:53.979168+0000","last_clean":"2023-05-22T15:26:53.979168+0000","last_became_active":"2023-05-22T15:26:53.648706+0000","last_became_peered":"2023-05-22T15:26:53.648706+0000","last_unstale":"2023-05-22T15:26:53.979168+0000","last_undegraded":"2023-05-22T15:26:53.979168+0000","last_fullsized":"2023-05-22T15:26:53.979168+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_clean_scrub_stamp":"2023-05-22T15:26:52.568670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T18:05:13.369385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5,6],"acting":[2,7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.611714+0000","last_change":"2023-05-22T15:26:49.581996+0000","last_active":"2023-05-22T15:26:53.611714+0000","last_peered":"2023-05-22T15:26:53.611714+0000","last_clean":"2023-05-22T15:26:53.611714+0000","last_became_active":"2023-05-22T15:26:49.581611+0000","last_became_peered":"2023-05-22T15:26:49.581611+0000","last_unstale":"2023-05-22T15:26:53.611714+0000","last_undegraded":"2023-05-22T15:26:53.611714+0000","last_fullsized":"2023-05-22T15:26:53.611714+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:44:52.369716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.634393+0000","last_change":"2023-05-22T15:26:49.192121+0000","last_active":"2023-05-22T15:26:53.634393+0000","last_peered":"2023-05-22T15:26:53.634393+0000","last_clean":"2023-05-22T15:26:53.634393+0000","last_became_active":"2023-05-22T15:26:49.191470+0000","last_became_peered":"2023-05-22T15:26:49.191470+0000","last_unstale":"2023-05-22T15:26:53.634393+0000","last_undegraded":"2023-05-22T15:26:53.634393+0000","last_fullsized":"2023-05-22T15:26:53.634393+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_clean_scrub_stamp":"2023-05-22T15:26:48.127413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:53:34.802286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.979064+0000","last_change":"2023-05-22T15:26:53.656554+0000","last_active":"2023-05-22T15:26:53.979064+0000","last_peered":"2023-05-22T15:26:53.979064+0000","last_clean":"2023-05-22T15:26:53.979064+0000","last_became_active":"2023-05-22T15:26:53.655950+0000","last_became_peered":"2023-05-22T15:26:53.655950+0000","last_unstale":"2023-05-22T15:26:53.979064+0000","last_undegraded":"2023-05-22T15:26:53.979064+0000","last_fullsized":"2023-05-22T15:26:53.979064+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_clean_scrub_stamp":"2023-05-22T15:26:52.568670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:11:45.590112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3,6],"acting":[2,4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.602652+0000","last_change":"2023-05-22T15:26:49.618498+0000","last_active":"2023-05-22T15:26:53.602652+0000","last_peered":"2023-05-22T15:26:53.602652+0000","last_clean":"2023-05-22T15:26:53.602652+0000","last_became_active":"2023-05-22T15:26:49.618068+0000","last_became_peered":"2023-05-22T15:26:49.618068+0000","last_unstale":"2023-05-22T15:26:53.602652+0000","last_undegraded":"2023-05-22T15:26:53.602652+0000","last_fullsized":"2023-05-22T15:26:53.602652+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:55:20.098659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"22'1","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.610411+0000","last_change":"2023-05-22T15:26:49.186875+0000","last_active":"2023-05-22T15:26:53.610411+0000","last_peered":"2023-05-22T15:26:53.610411+0000","last_clean":"2023-05-22T15:26:53.610411+0000","last_became_active":"2023-05-22T15:26:49.186431+0000","last_became_peered":"2023-05-22T15:26:49.186431+0000","last_unstale":"2023-05-22T15:26:53.610411+0000","last_undegraded":"2023-05-22T15:26:53.610411+0000","last_fullsized":"2023-05-22T15:26:53.610411+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_clean_scrub_stamp":"2023-05-22T15:26:48.127413+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:59:22.794782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.978460+0000","last_change":"2023-05-22T15:26:53.649077+0000","last_active":"2023-05-22T15:26:53.978460+0000","last_peered":"2023-05-22T15:26:53.978460+0000","last_clean":"2023-05-22T15:26:53.978460+0000","last_became_active":"2023-05-22T15:26:53.648415+0000","last_became_peered":"2023-05-22T15:26:53.648415+0000","last_unstale":"2023-05-22T15:26:53.978460+0000","last_undegraded":"2023-05-22T15:26:53.978460+0000","last_fullsized":"2023-05-22T15:26:53.978460+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_clean_scrub_stamp":"2023-05-22T15:26:52.568670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:33:26.787992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5,6],"acting":[2,0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.616451+0000","last_change":"2023-05-22T15:26:49.597065+0000","last_active":"2023-05-22T15:26:53.616451+0000","last_peered":"2023-05-22T15:26:53.616451+0000","last_clean":"2023-05-22T15:26:53.616451+0000","last_became_active":"2023-05-22T15:26:49.596334+0000","last_became_peered":"2023-05-22T15:26:49.596334+0000","last_unstale":"2023-05-22T15:26:53.616451+0000","last_undegraded":"2023-05-22T15:26:53.616451+0000","last_fullsized":"2023-05-22T15:26:53.616451+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:19:30.391062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"22'2","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.610463+0000","last_change":"2023-05-22T15:26:49.201296+0000","last_active":"2023-05-22T15:26:53.610463+0000","last_peered":"2023-05-22T15:26:53.610463+0000","last_clean":"2023-05-22T15:26:53.610463+0000","last_became_active":"2023-05-22T15:26:49.200707+0000","last_became_peered":"2023-05-22T15:26:49.200707+0000","last_unstale":"2023-05-22T15:26:53.610463+0000","last_undegraded":"2023-05-22T15:26:53.610463+0000","last_fullsized":"2023-05-22T15:26:53.610463+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_clean_scrub_stamp":"2023-05-22T15:26:48.127413+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:07:41.791660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.977970+0000","last_change":"2023-05-22T15:26:53.622351+0000","last_active":"2023-05-22T15:26:53.977970+0000","last_peered":"2023-05-22T15:26:53.977970+0000","last_clean":"2023-05-22T15:26:53.977970+0000","last_became_active":"2023-05-22T15:26:53.621754+0000","last_became_peered":"2023-05-22T15:26:53.621754+0000","last_unstale":"2023-05-22T15:26:53.977970+0000","last_undegraded":"2023-05-22T15:26:53.977970+0000","last_fullsized":"2023-05-22T15:26:53.977970+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_clean_scrub_stamp":"2023-05-22T15:26:52.568670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T16:31:40.104597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1,5],"acting":[2,0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.634315+0000","last_change":"2023-05-22T15:26:49.626518+0000","last_active":"2023-05-22T15:26:53.634315+0000","last_peered":"2023-05-22T15:26:53.634315+0000","last_clean":"2023-05-22T15:26:53.634315+0000","last_became_active":"2023-05-22T15:26:49.626026+0000","last_became_peered":"2023-05-22T15:26:49.626026+0000","last_unstale":"2023-05-22T15:26:53.634315+0000","last_undegraded":"2023-05-22T15:26:53.634315+0000","last_fullsized":"2023-05-22T15:26:53.634315+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:32:19.582542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"22'1","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.622234+0000","last_change":"2023-05-22T15:26:49.201252+0000","last_active":"2023-05-22T15:26:53.622234+0000","last_peered":"2023-05-22T15:26:53.622234+0000","last_clean":"2023-05-22T15:26:53.622234+0000","last_became_active":"2023-05-22T15:26:49.200748+0000","last_became_peered":"2023-05-22T15:26:49.200748+0000","last_unstale":"2023-05-22T15:26:53.622234+0000","last_undegraded":"2023-05-22T15:26:53.622234+0000","last_fullsized":"2023-05-22T15:26:53.622234+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_clean_scrub_stamp":"2023-05-22T15:26:48.127413+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:10:50.424396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.982126+0000","last_change":"2023-05-22T15:26:53.663379+0000","last_active":"2023-05-22T15:26:53.982126+0000","last_peered":"2023-05-22T15:26:53.982126+0000","last_clean":"2023-05-22T15:26:53.982126+0000","last_became_active":"2023-05-22T15:26:53.662170+0000","last_became_peered":"2023-05-22T15:26:53.662170+0000","last_unstale":"2023-05-22T15:26:53.982126+0000","last_undegraded":"2023-05-22T15:26:53.982126+0000","last_fullsized":"2023-05-22T15:26:53.982126+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_clean_scrub_stamp":"2023-05-22T15:26:52.568670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:38:53.647917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3,4],"acting":[7,6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-05-22T15:26:54.056358+0000","last_change":"2023-05-22T15:26:53.647770+0000","last_active":"2023-05-22T15:26:54.056358+0000","last_peered":"2023-05-22T15:26:54.056358+0000","last_clean":"2023-05-22T15:26:54.056358+0000","last_became_active":"2023-05-22T15:26:53.646585+0000","last_became_peered":"2023-05-22T15:26:53.646585+0000","last_unstale":"2023-05-22T15:26:54.056358+0000","last_undegraded":"2023-05-22T15:26:54.056358+0000","last_fullsized":"2023-05-22T15:26:54.056358+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_clean_scrub_stamp":"2023-05-22T15:26:52.568670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:48:44.211681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1,6],"acting":[5,3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"22'8","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.602514+0000","last_change":"2023-05-22T15:26:49.189894+0000","last_active":"2023-05-22T15:26:53.602514+0000","last_peered":"2023-05-22T15:26:53.602514+0000","last_clean":"2023-05-22T15:26:53.602514+0000","last_became_active":"2023-05-22T15:26:49.189156+0000","last_became_peered":"2023-05-22T15:26:49.189156+0000","last_unstale":"2023-05-22T15:26:53.602514+0000","last_undegraded":"2023-05-22T15:26:53.602514+0000","last_fullsized":"2023-05-22T15:26:53.602514+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_clean_scrub_stamp":"2023-05-22T15:26:48.127413+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:13:21.177657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.605587+0000","last_change":"2023-05-22T15:26:49.622540+0000","last_active":"2023-05-22T15:26:53.605587+0000","last_peered":"2023-05-22T15:26:53.605587+0000","last_clean":"2023-05-22T15:26:53.605587+0000","last_became_active":"2023-05-22T15:26:49.622187+0000","last_became_peered":"2023-05-22T15:26:49.622187+0000","last_unstale":"2023-05-22T15:26:53.605587+0000","last_undegraded":"2023-05-22T15:26:53.605587+0000","last_fullsized":"2023-05-22T15:26:53.605587+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T16:31:36.674451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.993571+0000","last_change":"2023-05-22T15:26:53.645178+0000","last_active":"2023-05-22T15:26:53.993571+0000","last_peered":"2023-05-22T15:26:53.993571+0000","last_clean":"2023-05-22T15:26:53.993571+0000","last_became_active":"2023-05-22T15:26:53.644770+0000","last_became_peered":"2023-05-22T15:26:53.644770+0000","last_unstale":"2023-05-22T15:26:53.993571+0000","last_undegraded":"2023-05-22T15:26:53.993571+0000","last_fullsized":"2023-05-22T15:26:53.993571+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_clean_scrub_stamp":"2023-05-22T15:26:52.568670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T15:46:20.544673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,0],"acting":[3,1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.628444+0000","last_change":"2023-05-22T15:26:49.191146+0000","last_active":"2023-05-22T15:26:53.628444+0000","last_peered":"2023-05-22T15:26:53.628444+0000","last_clean":"2023-05-22T15:26:53.628444+0000","last_became_active":"2023-05-22T15:26:49.190714+0000","last_became_peered":"2023-05-22T15:26:49.190714+0000","last_unstale":"2023-05-22T15:26:53.628444+0000","last_undegraded":"2023-05-22T15:26:53.628444+0000","last_fullsized":"2023-05-22T15:26:53.628444+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_clean_scrub_stamp":"2023-05-22T15:26:48.127413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:55:19.087548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.612090+0000","last_change":"2023-05-22T15:26:49.585916+0000","last_active":"2023-05-22T15:26:53.612090+0000","last_peered":"2023-05-22T15:26:53.612090+0000","last_clean":"2023-05-22T15:26:53.612090+0000","last_became_active":"2023-05-22T15:26:49.585261+0000","last_became_peered":"2023-05-22T15:26:49.585261+0000","last_unstale":"2023-05-22T15:26:53.612090+0000","last_undegraded":"2023-05-22T15:26:53.612090+0000","last_fullsized":"2023-05-22T15:26:53.612090+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:49:00.232177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.611499+0000","last_change":"2023-05-22T15:26:49.179340+0000","last_active":"2023-05-22T15:26:53.611499+0000","last_peered":"2023-05-22T15:26:53.611499+0000","last_clean":"2023-05-22T15:26:53.611499+0000","last_became_active":"2023-05-22T15:26:49.178875+0000","last_became_peered":"2023-05-22T15:26:49.178875+0000","last_unstale":"2023-05-22T15:26:53.611499+0000","last_undegraded":"2023-05-22T15:26:53.611499+0000","last_fullsized":"2023-05-22T15:26:53.611499+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_clean_scrub_stamp":"2023-05-22T15:26:48.127413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T16:16:39.277093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.634708+0000","last_change":"2023-05-22T15:26:49.623344+0000","last_active":"2023-05-22T15:26:53.634708+0000","last_peered":"2023-05-22T15:26:53.634708+0000","last_clean":"2023-05-22T15:26:53.634708+0000","last_became_active":"2023-05-22T15:26:49.622700+0000","last_became_peered":"2023-05-22T15:26:49.622700+0000","last_unstale":"2023-05-22T15:26:53.634708+0000","last_undegraded":"2023-05-22T15:26:53.634708+0000","last_fullsized":"2023-05-22T15:26:53.634708+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:22:10.117410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.993134+0000","last_change":"2023-05-22T15:26:53.646185+0000","last_active":"2023-05-22T15:26:53.993134+0000","last_peered":"2023-05-22T15:26:53.993134+0000","last_clean":"2023-05-22T15:26:53.993134+0000","last_became_active":"2023-05-22T15:26:53.645789+0000","last_became_peered":"2023-05-22T15:26:53.645789+0000","last_unstale":"2023-05-22T15:26:53.993134+0000","last_undegraded":"2023-05-22T15:26:53.993134+0000","last_fullsized":"2023-05-22T15:26:53.993134+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_clean_scrub_stamp":"2023-05-22T15:26:52.568670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:30:40.980509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2,1],"acting":[3,7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"22'2","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.608324+0000","last_change":"2023-05-22T15:26:49.201586+0000","last_active":"2023-05-22T15:26:53.608324+0000","last_peered":"2023-05-22T15:26:53.608324+0000","last_clean":"2023-05-22T15:26:53.608324+0000","last_became_active":"2023-05-22T15:26:49.201013+0000","last_became_peered":"2023-05-22T15:26:49.201013+0000","last_unstale":"2023-05-22T15:26:53.608324+0000","last_undegraded":"2023-05-22T15:26:53.608324+0000","last_fullsized":"2023-05-22T15:26:53.608324+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_clean_scrub_stamp":"2023-05-22T15:26:48.127413+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:47:17.588354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.607130+0000","last_change":"2023-05-22T15:26:49.577112+0000","last_active":"2023-05-22T15:26:53.607130+0000","last_peered":"2023-05-22T15:26:53.607130+0000","last_clean":"2023-05-22T15:26:53.607130+0000","last_became_active":"2023-05-22T15:26:49.576802+0000","last_became_peered":"2023-05-22T15:26:49.576802+0000","last_unstale":"2023-05-22T15:26:53.607130+0000","last_undegraded":"2023-05-22T15:26:53.607130+0000","last_fullsized":"2023-05-22T15:26:53.607130+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:52:48.123148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-05-22T15:26:54.056338+0000","last_change":"2023-05-22T15:26:53.655225+0000","last_active":"2023-05-22T15:26:54.056338+0000","last_peered":"2023-05-22T15:26:54.056338+0000","last_clean":"2023-05-22T15:26:54.056338+0000","last_became_active":"2023-05-22T15:26:53.648610+0000","last_became_peered":"2023-05-22T15:26:53.648610+0000","last_unstale":"2023-05-22T15:26:54.056338+0000","last_undegraded":"2023-05-22T15:26:54.056338+0000","last_fullsized":"2023-05-22T15:26:54.056338+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_clean_scrub_stamp":"2023-05-22T15:26:52.568670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T15:46:20.544673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3,0],"acting":[5,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.20","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.602276+0000","last_change":"2023-05-22T15:26:49.614417+0000","last_active":"2023-05-22T15:26:53.602276+0000","last_peered":"2023-05-22T15:26:53.602276+0000","last_clean":"2023-05-22T15:26:53.602276+0000","last_became_active":"2023-05-22T15:26:49.610439+0000","last_became_peered":"2023-05-22T15:26:49.610439+0000","last_unstale":"2023-05-22T15:26:53.602276+0000","last_undegraded":"2023-05-22T15:26:53.602276+0000","last_fullsized":"2023-05-22T15:26:53.602276+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:07:42.802772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.21","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.603531+0000","last_change":"2023-05-22T15:26:49.577132+0000","last_active":"2023-05-22T15:26:53.603531+0000","last_peered":"2023-05-22T15:26:53.603531+0000","last_clean":"2023-05-22T15:26:53.603531+0000","last_became_active":"2023-05-22T15:26:49.576859+0000","last_became_peered":"2023-05-22T15:26:49.576859+0000","last_unstale":"2023-05-22T15:26:53.603531+0000","last_undegraded":"2023-05-22T15:26:53.603531+0000","last_fullsized":"2023-05-22T15:26:53.603531+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:08:07.637726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.22","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.619920+0000","last_change":"2023-05-22T15:26:49.622334+0000","last_active":"2023-05-22T15:26:53.619920+0000","last_peered":"2023-05-22T15:26:53.619920+0000","last_clean":"2023-05-22T15:26:53.619920+0000","last_became_active":"2023-05-22T15:26:49.621648+0000","last_became_peered":"2023-05-22T15:26:49.621648+0000","last_unstale":"2023-05-22T15:26:53.619920+0000","last_undegraded":"2023-05-22T15:26:53.619920+0000","last_fullsized":"2023-05-22T15:26:53.619920+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:57:58.007680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.23","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.605756+0000","last_change":"2023-05-22T15:26:49.612753+0000","last_active":"2023-05-22T15:26:53.605756+0000","last_peered":"2023-05-22T15:26:53.605756+0000","last_clean":"2023-05-22T15:26:53.605756+0000","last_became_active":"2023-05-22T15:26:49.612495+0000","last_became_peered":"2023-05-22T15:26:49.612495+0000","last_unstale":"2023-05-22T15:26:53.605756+0000","last_undegraded":"2023-05-22T15:26:53.605756+0000","last_fullsized":"2023-05-22T15:26:53.605756+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:11:51.327926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.24","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.619231+0000","last_change":"2023-05-22T15:26:49.627768+0000","last_active":"2023-05-22T15:26:53.619231+0000","last_peered":"2023-05-22T15:26:53.619231+0000","last_clean":"2023-05-22T15:26:53.619231+0000","last_became_active":"2023-05-22T15:26:49.627340+0000","last_became_peered":"2023-05-22T15:26:49.627340+0000","last_unstale":"2023-05-22T15:26:53.619231+0000","last_undegraded":"2023-05-22T15:26:53.619231+0000","last_fullsized":"2023-05-22T15:26:53.619231+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:44:52.369716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.25","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.634508+0000","last_change":"2023-05-22T15:26:49.613627+0000","last_active":"2023-05-22T15:26:53.634508+0000","last_peered":"2023-05-22T15:26:53.634508+0000","last_clean":"2023-05-22T15:26:53.634508+0000","last_became_active":"2023-05-22T15:26:49.612997+0000","last_became_peered":"2023-05-22T15:26:49.612997+0000","last_unstale":"2023-05-22T15:26:53.634508+0000","last_undegraded":"2023-05-22T15:26:53.634508+0000","last_fullsized":"2023-05-22T15:26:53.634508+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:11:51.327926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.26","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.612444+0000","last_change":"2023-05-22T15:26:49.581772+0000","last_active":"2023-05-22T15:26:53.612444+0000","last_peered":"2023-05-22T15:26:53.612444+0000","last_clean":"2023-05-22T15:26:53.612444+0000","last_became_active":"2023-05-22T15:26:49.581493+0000","last_became_peered":"2023-05-22T15:26:49.581493+0000","last_unstale":"2023-05-22T15:26:53.612444+0000","last_undegraded":"2023-05-22T15:26:53.612444+0000","last_fullsized":"2023-05-22T15:26:53.612444+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:49:00.232177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.27","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.612724+0000","last_change":"2023-05-22T15:26:49.619433+0000","last_active":"2023-05-22T15:26:53.612724+0000","last_peered":"2023-05-22T15:26:53.612724+0000","last_clean":"2023-05-22T15:26:53.612724+0000","last_became_active":"2023-05-22T15:26:49.619107+0000","last_became_peered":"2023-05-22T15:26:49.619107+0000","last_unstale":"2023-05-22T15:26:53.612724+0000","last_undegraded":"2023-05-22T15:26:53.612724+0000","last_fullsized":"2023-05-22T15:26:53.612724+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:11:51.327926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.28","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.609649+0000","last_change":"2023-05-22T15:26:49.624224+0000","last_active":"2023-05-22T15:26:53.609649+0000","last_peered":"2023-05-22T15:26:53.609649+0000","last_clean":"2023-05-22T15:26:53.609649+0000","last_became_active":"2023-05-22T15:26:49.623023+0000","last_became_peered":"2023-05-22T15:26:49.623023+0000","last_unstale":"2023-05-22T15:26:53.609649+0000","last_undegraded":"2023-05-22T15:26:53.609649+0000","last_fullsized":"2023-05-22T15:26:53.609649+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:32:19.582542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.29","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.603353+0000","last_change":"2023-05-22T15:26:49.573154+0000","last_active":"2023-05-22T15:26:53.603353+0000","last_peered":"2023-05-22T15:26:53.603353+0000","last_clean":"2023-05-22T15:26:53.603353+0000","last_became_active":"2023-05-22T15:26:49.572881+0000","last_became_peered":"2023-05-22T15:26:49.572881+0000","last_unstale":"2023-05-22T15:26:53.603353+0000","last_undegraded":"2023-05-22T15:26:53.603353+0000","last_fullsized":"2023-05-22T15:26:53.603353+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:32:19.582542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2a","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.611407+0000","last_change":"2023-05-22T15:26:49.581918+0000","last_active":"2023-05-22T15:26:53.611407+0000","last_peered":"2023-05-22T15:26:53.611407+0000","last_clean":"2023-05-22T15:26:53.611407+0000","last_became_active":"2023-05-22T15:26:49.581462+0000","last_became_peered":"2023-05-22T15:26:49.581462+0000","last_unstale":"2023-05-22T15:26:53.611407+0000","last_undegraded":"2023-05-22T15:26:53.611407+0000","last_fullsized":"2023-05-22T15:26:53.611407+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:19:30.391062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2b","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.608076+0000","last_change":"2023-05-22T15:26:49.624711+0000","last_active":"2023-05-22T15:26:53.608076+0000","last_peered":"2023-05-22T15:26:53.608076+0000","last_clean":"2023-05-22T15:26:53.608076+0000","last_became_active":"2023-05-22T15:26:49.624180+0000","last_became_peered":"2023-05-22T15:26:49.624180+0000","last_unstale":"2023-05-22T15:26:53.608076+0000","last_undegraded":"2023-05-22T15:26:53.608076+0000","last_fullsized":"2023-05-22T15:26:53.608076+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:43:00.518715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2c","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.623768+0000","last_change":"2023-05-22T15:26:49.620689+0000","last_active":"2023-05-22T15:26:53.623768+0000","last_peered":"2023-05-22T15:26:53.623768+0000","last_clean":"2023-05-22T15:26:53.623768+0000","last_became_active":"2023-05-22T15:26:49.620183+0000","last_became_peered":"2023-05-22T15:26:49.620183+0000","last_unstale":"2023-05-22T15:26:53.623768+0000","last_undegraded":"2023-05-22T15:26:53.623768+0000","last_fullsized":"2023-05-22T15:26:53.623768+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:47:22.628767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2d","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.601734+0000","last_change":"2023-05-22T15:26:49.618556+0000","last_active":"2023-05-22T15:26:53.601734+0000","last_peered":"2023-05-22T15:26:53.601734+0000","last_clean":"2023-05-22T15:26:53.601734+0000","last_became_active":"2023-05-22T15:26:49.618215+0000","last_became_peered":"2023-05-22T15:26:49.618215+0000","last_unstale":"2023-05-22T15:26:53.601734+0000","last_undegraded":"2023-05-22T15:26:53.601734+0000","last_fullsized":"2023-05-22T15:26:53.601734+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:07:50.944971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2e","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.621273+0000","last_change":"2023-05-22T15:26:49.597573+0000","last_active":"2023-05-22T15:26:53.621273+0000","last_peered":"2023-05-22T15:26:53.621273+0000","last_clean":"2023-05-22T15:26:53.621273+0000","last_became_active":"2023-05-22T15:26:49.596607+0000","last_became_peered":"2023-05-22T15:26:49.596607+0000","last_unstale":"2023-05-22T15:26:53.621273+0000","last_undegraded":"2023-05-22T15:26:53.621273+0000","last_fullsized":"2023-05-22T15:26:53.621273+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:31:03.890085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2f","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.611892+0000","last_change":"2023-05-22T15:26:49.578237+0000","last_active":"2023-05-22T15:26:53.611892+0000","last_peered":"2023-05-22T15:26:53.611892+0000","last_clean":"2023-05-22T15:26:53.611892+0000","last_became_active":"2023-05-22T15:26:49.577927+0000","last_became_peered":"2023-05-22T15:26:49.577927+0000","last_unstale":"2023-05-22T15:26:53.611892+0000","last_undegraded":"2023-05-22T15:26:53.611892+0000","last_fullsized":"2023-05-22T15:26:53.611892+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:11:51.327926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.30","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.612647+0000","last_change":"2023-05-22T15:26:49.598449+0000","last_active":"2023-05-22T15:26:53.612647+0000","last_peered":"2023-05-22T15:26:53.612647+0000","last_clean":"2023-05-22T15:26:53.612647+0000","last_became_active":"2023-05-22T15:26:49.597652+0000","last_became_peered":"2023-05-22T15:26:49.597652+0000","last_unstale":"2023-05-22T15:26:53.612647+0000","last_undegraded":"2023-05-22T15:26:53.612647+0000","last_fullsized":"2023-05-22T15:26:53.612647+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:48:33.611833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.31","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.628230+0000","last_change":"2023-05-22T15:26:49.625943+0000","last_active":"2023-05-22T15:26:53.628230+0000","last_peered":"2023-05-22T15:26:53.628230+0000","last_clean":"2023-05-22T15:26:53.628230+0000","last_became_active":"2023-05-22T15:26:49.624306+0000","last_became_peered":"2023-05-22T15:26:49.624306+0000","last_unstale":"2023-05-22T15:26:53.628230+0000","last_undegraded":"2023-05-22T15:26:53.628230+0000","last_fullsized":"2023-05-22T15:26:53.628230+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:44:52.369716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.32","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.634913+0000","last_change":"2023-05-22T15:26:49.563010+0000","last_active":"2023-05-22T15:26:53.634913+0000","last_peered":"2023-05-22T15:26:53.634913+0000","last_clean":"2023-05-22T15:26:53.634913+0000","last_became_active":"2023-05-22T15:26:49.562210+0000","last_became_peered":"2023-05-22T15:26:49.562210+0000","last_unstale":"2023-05-22T15:26:53.634913+0000","last_undegraded":"2023-05-22T15:26:53.634913+0000","last_fullsized":"2023-05-22T15:26:53.634913+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:19:30.391062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.33","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.600262+0000","last_change":"2023-05-22T15:26:49.562260+0000","last_active":"2023-05-22T15:26:53.600262+0000","last_peered":"2023-05-22T15:26:53.600262+0000","last_clean":"2023-05-22T15:26:53.600262+0000","last_became_active":"2023-05-22T15:26:49.561965+0000","last_became_peered":"2023-05-22T15:26:49.561965+0000","last_unstale":"2023-05-22T15:26:53.600262+0000","last_undegraded":"2023-05-22T15:26:53.600262+0000","last_fullsized":"2023-05-22T15:26:53.600262+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:59:55.020950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.34","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.598383+0000","last_change":"2023-05-22T15:26:49.601428+0000","last_active":"2023-05-22T15:26:53.598383+0000","last_peered":"2023-05-22T15:26:53.598383+0000","last_clean":"2023-05-22T15:26:53.598383+0000","last_became_active":"2023-05-22T15:26:49.601157+0000","last_became_peered":"2023-05-22T15:26:49.601157+0000","last_unstale":"2023-05-22T15:26:53.598383+0000","last_undegraded":"2023-05-22T15:26:53.598383+0000","last_fullsized":"2023-05-22T15:26:53.598383+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:47:18.599465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.35","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.631671+0000","last_change":"2023-05-22T15:26:49.598414+0000","last_active":"2023-05-22T15:26:53.631671+0000","last_peered":"2023-05-22T15:26:53.631671+0000","last_clean":"2023-05-22T15:26:53.631671+0000","last_became_active":"2023-05-22T15:26:49.595631+0000","last_became_peered":"2023-05-22T15:26:49.595631+0000","last_unstale":"2023-05-22T15:26:53.631671+0000","last_undegraded":"2023-05-22T15:26:53.631671+0000","last_fullsized":"2023-05-22T15:26:53.631671+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:11:51.327926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.36","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.635151+0000","last_change":"2023-05-22T15:26:49.580957+0000","last_active":"2023-05-22T15:26:53.635151+0000","last_peered":"2023-05-22T15:26:53.635151+0000","last_clean":"2023-05-22T15:26:53.635151+0000","last_became_active":"2023-05-22T15:26:49.580172+0000","last_became_peered":"2023-05-22T15:26:49.580172+0000","last_unstale":"2023-05-22T15:26:53.635151+0000","last_undegraded":"2023-05-22T15:26:53.635151+0000","last_fullsized":"2023-05-22T15:26:53.635151+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:43:00.518715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.37","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.615519+0000","last_change":"2023-05-22T15:26:49.587437+0000","last_active":"2023-05-22T15:26:53.615519+0000","last_peered":"2023-05-22T15:26:53.615519+0000","last_clean":"2023-05-22T15:26:53.615519+0000","last_became_active":"2023-05-22T15:26:49.587019+0000","last_became_peered":"2023-05-22T15:26:49.587019+0000","last_unstale":"2023-05-22T15:26:53.615519+0000","last_undegraded":"2023-05-22T15:26:53.615519+0000","last_fullsized":"2023-05-22T15:26:53.615519+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:48:33.611833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.38","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.634076+0000","last_change":"2023-05-22T15:26:49.599720+0000","last_active":"2023-05-22T15:26:53.634076+0000","last_peered":"2023-05-22T15:26:53.634076+0000","last_clean":"2023-05-22T15:26:53.634076+0000","last_became_active":"2023-05-22T15:26:49.599079+0000","last_became_peered":"2023-05-22T15:26:49.599079+0000","last_unstale":"2023-05-22T15:26:53.634076+0000","last_undegraded":"2023-05-22T15:26:53.634076+0000","last_fullsized":"2023-05-22T15:26:53.634076+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:48:33.611833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.39","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.628016+0000","last_change":"2023-05-22T15:26:49.624934+0000","last_active":"2023-05-22T15:26:53.628016+0000","last_peered":"2023-05-22T15:26:53.628016+0000","last_clean":"2023-05-22T15:26:53.628016+0000","last_became_active":"2023-05-22T15:26:49.623557+0000","last_became_peered":"2023-05-22T15:26:49.623557+0000","last_unstale":"2023-05-22T15:26:53.628016+0000","last_undegraded":"2023-05-22T15:26:53.628016+0000","last_fullsized":"2023-05-22T15:26:53.628016+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:19:30.391062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3a","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.619318+0000","last_change":"2023-05-22T15:26:49.596704+0000","last_active":"2023-05-22T15:26:53.619318+0000","last_peered":"2023-05-22T15:26:53.619318+0000","last_clean":"2023-05-22T15:26:53.619318+0000","last_became_active":"2023-05-22T15:26:49.595941+0000","last_became_peered":"2023-05-22T15:26:49.595941+0000","last_unstale":"2023-05-22T15:26:53.619318+0000","last_undegraded":"2023-05-22T15:26:53.619318+0000","last_fullsized":"2023-05-22T15:26:53.619318+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T03:17:59.774769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3b","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.611996+0000","last_change":"2023-05-22T15:26:49.585396+0000","last_active":"2023-05-22T15:26:53.611996+0000","last_peered":"2023-05-22T15:26:53.611996+0000","last_clean":"2023-05-22T15:26:53.611996+0000","last_became_active":"2023-05-22T15:26:49.585110+0000","last_became_peered":"2023-05-22T15:26:49.585110+0000","last_unstale":"2023-05-22T15:26:53.611996+0000","last_undegraded":"2023-05-22T15:26:53.611996+0000","last_fullsized":"2023-05-22T15:26:53.611996+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:22:10.117410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3c","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.602523+0000","last_change":"2023-05-22T15:26:49.607125+0000","last_active":"2023-05-22T15:26:53.602523+0000","last_peered":"2023-05-22T15:26:53.602523+0000","last_clean":"2023-05-22T15:26:53.602523+0000","last_became_active":"2023-05-22T15:26:49.606510+0000","last_became_peered":"2023-05-22T15:26:49.606510+0000","last_unstale":"2023-05-22T15:26:53.602523+0000","last_undegraded":"2023-05-22T15:26:53.602523+0000","last_fullsized":"2023-05-22T15:26:53.602523+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:47:18.599465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3d","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.631452+0000","last_change":"2023-05-22T15:26:49.600011+0000","last_active":"2023-05-22T15:26:53.631452+0000","last_peered":"2023-05-22T15:26:53.631452+0000","last_clean":"2023-05-22T15:26:53.631452+0000","last_became_active":"2023-05-22T15:26:49.596414+0000","last_became_peered":"2023-05-22T15:26:49.596414+0000","last_unstale":"2023-05-22T15:26:53.631452+0000","last_undegraded":"2023-05-22T15:26:53.631452+0000","last_fullsized":"2023-05-22T15:26:53.631452+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T16:31:36.674451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3e","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.634607+0000","last_change":"2023-05-22T15:26:49.580509+0000","last_active":"2023-05-22T15:26:53.634607+0000","last_peered":"2023-05-22T15:26:53.634607+0000","last_clean":"2023-05-22T15:26:53.634607+0000","last_became_active":"2023-05-22T15:26:49.579662+0000","last_became_peered":"2023-05-22T15:26:49.579662+0000","last_unstale":"2023-05-22T15:26:53.634607+0000","last_undegraded":"2023-05-22T15:26:53.634607+0000","last_fullsized":"2023-05-22T15:26:53.634607+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:44:52.369716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3f","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.599109+0000","last_change":"2023-05-22T15:26:49.561088+0000","last_active":"2023-05-22T15:26:53.599109+0000","last_peered":"2023-05-22T15:26:53.599109+0000","last_clean":"2023-05-22T15:26:53.599109+0000","last_became_active":"2023-05-22T15:26:49.560859+0000","last_became_peered":"2023-05-22T15:26:49.560859+0000","last_unstale":"2023-05-22T15:26:53.599109+0000","last_undegraded":"2023-05-22T15:26:53.599109+0000","last_fullsized":"2023-05-22T15:26:53.599109+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:53:35.813397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":4386,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":47,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":8772,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":41,"ondisk_log_size":41,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574855,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9284,"kb_used_data":892,"kb_used_omap":0,"kb_used_meta":8384,"kb_avail":94362556,"statfs":{"total":96636764160,"available":96627257344,"internally_reserved":0,"allocated":913408,"data_stored":714089,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8585216},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97999999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71499999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93100000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1779999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2310000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.149}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3999999999999999}]}]},{"osd":6,"up_from":13,"seq":55834574856,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8100,"kb_used_data":284,"kb_used_omap":0,"kb_used_meta":7808,"kb_avail":94363740,"statfs":{"total":96636764160,"available":96628469760,"internally_reserved":0,"allocated":290816,"data_stored":119901,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7995392},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69899999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91200000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64500000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66500000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84499999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68600000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79600000000000004}]}]},{"osd":5,"up_from":13,"seq":55834574856,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8172,"kb_used_data":292,"kb_used_omap":0,"kb_used_meta":7872,"kb_avail":94363668,"statfs":{"total":96636764160,"available":96628396032,"internally_reserved":0,"allocated":299008,"data_stored":120537,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8060928},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.159}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90900000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77500000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71099999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1279999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94199999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2370000000000001}]}]},{"osd":4,"up_from":13,"seq":55834574854,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8248,"kb_used_data":304,"kb_used_omap":0,"kb_used_meta":7936,"kb_avail":94363592,"statfs":{"total":96636764160,"available":96628318208,"internally_reserved":0,"allocated":311296,"data_stored":123561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8126464},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72299999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78000000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84999999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1040000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1879999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1359999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1699999999999999}]}]},{"osd":3,"up_from":13,"seq":55834574854,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8168,"kb_used_data":288,"kb_used_omap":0,"kb_used_meta":7872,"kb_avail":94363672,"statfs":{"total":96636764160,"available":96628400128,"internally_reserved":0,"allocated":294912,"data_stored":119991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8060928},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34100000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38400000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46600000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85499999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92100000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71299999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79100000000000004}]}]},{"osd":2,"up_from":13,"seq":55834574855,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":12196,"kb_used_data":284,"kb_used_omap":0,"kb_used_meta":11904,"kb_avail":94359644,"statfs":{"total":96636764160,"available":96624275456,"internally_reserved":0,"allocated":290816,"data_stored":119901,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":12189696},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.503}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40400000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61199999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79900000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78900000000000003}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72799999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86699999999999999}]}]},{"osd":1,"up_from":13,"seq":55834574855,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8164,"kb_used_data":284,"kb_used_omap":0,"kb_used_meta":7872,"kb_avail":94363676,"statfs":{"total":96636764160,"available":96628404224,"internally_reserved":0,"allocated":290816,"data_stored":119901,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8060928},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66600000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57099999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77700000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79200000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.034}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0129999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0760000000000001}]}]},{"osd":0,"up_from":13,"seq":55834574856,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":13356,"kb_used_data":868,"kb_used_omap":0,"kb_used_meta":12480,"kb_avail":94358484,"statfs":{"total":96636764160,"available":96623087616,"internally_reserved":0,"allocated":888832,"data_stored":710835,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":12779520},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1839999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.23}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2410000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2969999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.365}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.264}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.28}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":3660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":3820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-05-22T15:27:07.082 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-05-22T15:27:07.237 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.226+0000 7f0c7bb2a640 1 Processor -- start 2023-05-22T15:27:07.237 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.226+0000 7f0c7bb2a640 1 -- start start 2023-05-22T15:27:07.238 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.230+0000 7f0c7bb2a640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f0c740b7dd0 0x7f0c740ccab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:07.238 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.230+0000 7f0c7bb2a640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f0c74160250 0x7f0c740d0410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:07.238 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.230+0000 7f0c7bb2a640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c740cd170 con 0x7f0c7415f700 2023-05-22T15:27:07.239 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.230+0000 7f0c7bb2a640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c740d3360 con 0x7f0c74160250 2023-05-22T15:27:07.239 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.230+0000 7f0c7bb2a640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c740d2ac0 con 0x7f0c740b7dd0 2023-05-22T15:27:07.239 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.230+0000 7f0c7a0c7640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f0c74160250 0x7f0c740d0410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:07.240 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.230+0000 7f0c790c5640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f0c7415f700 0x7f0c7415fb40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:51020/0 (socket says 172.21.15.16:51020) 2023-05-22T15:27:07.241 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.230+0000 7f0c790c5640 1 -- 172.21.15.16:0/2665486535 learned_addr learned my addr 172.21.15.16:0/2665486535 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:27:07.241 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.230+0000 7f0c798c6640 1 --2- 172.21.15.16:0/2665486535 >> v2:172.21.15.62:3300/0 conn(0x7f0c740b7dd0 0x7f0c740ccab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:07.241 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.230+0000 7f0c7a0c7640 1 -- 172.21.15.16:0/2665486535 >> v2:172.21.15.62:3300/0 conn(0x7f0c740b7dd0 msgr2=0x7f0c740ccab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:07.242 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.230+0000 7f0c7a0c7640 1 --2- 172.21.15.16:0/2665486535 >> v2:172.21.15.62:3300/0 conn(0x7f0c740b7dd0 0x7f0c740ccab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:07.242 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.230+0000 7f0c7a0c7640 1 -- 172.21.15.16:0/2665486535 >> v1:172.21.15.16:6789/0 conn(0x7f0c7415f700 legacy=0x7f0c7415fb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:07.242 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.230+0000 7f0c7a0c7640 1 -- 172.21.15.16:0/2665486535 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c740d2da0 con 0x7f0c74160250 2023-05-22T15:27:07.247 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.230+0000 7f0c7a0c7640 1 --2- 172.21.15.16:0/2665486535 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0c74160250 0x7f0c740d0410 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f0c74071e30 tx=0x7f0c6c01c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8ee2175cff8df96 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:07.248 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.230+0000 7f0c788c4640 1 -- 172.21.15.16:0/2665486535 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c6c0a5960 con 0x7f0c74160250 2023-05-22T15:27:07.248 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.230+0000 7f0c788c4640 1 -- 172.21.15.16:0/2665486535 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c6c0a5ac0 con 0x7f0c74160250 2023-05-22T15:27:07.249 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.230+0000 7f0c788c4640 1 -- 172.21.15.16:0/2665486535 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c6c0a5e50 con 0x7f0c74160250 2023-05-22T15:27:07.249 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.230+0000 7f0c7bb2a640 1 -- 172.21.15.16:0/2665486535 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0c74160250 msgr2=0x7f0c740d0410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:07.249 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.230+0000 7f0c7bb2a640 1 --2- 172.21.15.16:0/2665486535 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0c74160250 0x7f0c740d0410 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f0c74071e30 tx=0x7f0c6c01c8e0 comp rx=0 tx=0).stop 2023-05-22T15:27:07.250 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.230+0000 7f0c7bb2a640 1 -- 172.21.15.16:0/2665486535 shutdown_connections 2023-05-22T15:27:07.250 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.234+0000 7f0c7bb2a640 1 --2- 172.21.15.16:0/2665486535 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0c74160250 0x7f0c740d0410 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:07.250 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.234+0000 7f0c7bb2a640 1 --2- 172.21.15.16:0/2665486535 >> v2:172.21.15.62:3300/0 conn(0x7f0c740b7dd0 0x7f0c740ccab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:07.250 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.234+0000 7f0c7bb2a640 1 -- 172.21.15.16:0/2665486535 >> 172.21.15.16:0/2665486535 conn(0x7f0c740655c0 msgr2=0x7f0c74070e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:07.250 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.234+0000 7f0c7bb2a640 1 -- 172.21.15.16:0/2665486535 shutdown_connections 2023-05-22T15:27:07.251 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.234+0000 7f0c7bb2a640 1 -- 172.21.15.16:0/2665486535 wait complete. 2023-05-22T15:27:07.251 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.234+0000 7f0c7bb2a640 1 Processor -- start 2023-05-22T15:27:07.251 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.234+0000 7f0c7bb2a640 1 -- start start 2023-05-22T15:27:07.251 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.234+0000 7f0c7bb2a640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0c740b7dd0 0x7f0c74156d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:07.252 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.234+0000 7f0c7bb2a640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0c7415f700 0x7f0c7415e460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:07.252 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.234+0000 7f0c7bb2a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0c74160250 0x7f0c74156040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:07.252 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.234+0000 7f0c7bb2a640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c7415c8b0 con 0x7f0c74160250 2023-05-22T15:27:07.252 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.234+0000 7f0c7bb2a640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c74155120 con 0x7f0c740b7dd0 2023-05-22T15:27:07.253 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.234+0000 7f0c7bb2a640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c7415c2a0 con 0x7f0c7415f700 2023-05-22T15:27:07.253 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.234+0000 7f0c7a0c7640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0c74160250 0x7f0c74156040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:07.253 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.234+0000 7f0c7a0c7640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0c74160250 0x7f0c74156040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57712/0 (socket says 172.21.15.16:57712) 2023-05-22T15:27:07.254 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.234+0000 7f0c7a0c7640 1 -- 172.21.15.16:0/3826145296 learned_addr learned my addr 172.21.15.16:0/3826145296 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:07.254 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.234+0000 7f0c798c6640 1 --2- 172.21.15.16:0/3826145296 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0c740b7dd0 0x7f0c74156d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:07.254 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.234+0000 7f0c7a0c7640 1 -- 172.21.15.16:0/3826145296 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0c7415f700 msgr2=0x7f0c7415e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:07.254 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.234+0000 7f0c790c5640 1 --2- 172.21.15.16:0/3826145296 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0c7415f700 0x7f0c7415e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:07.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.234+0000 7f0c7a0c7640 1 --2- 172.21.15.16:0/3826145296 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0c7415f700 0x7f0c7415e460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:07.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.234+0000 7f0c7a0c7640 1 -- 172.21.15.16:0/3826145296 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0c740b7dd0 msgr2=0x7f0c74156d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:07.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.234+0000 7f0c7a0c7640 1 --2- 172.21.15.16:0/3826145296 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0c740b7dd0 0x7f0c74156d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:07.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.234+0000 7f0c7a0c7640 1 -- 172.21.15.16:0/3826145296 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c6c009a70 con 0x7f0c74160250 2023-05-22T15:27:07.256 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.234+0000 7f0c798c6640 1 --2- 172.21.15.16:0/3826145296 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0c740b7dd0 0x7f0c74156d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:27:07.256 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.234+0000 7f0c7a0c7640 1 --2- 172.21.15.16:0/3826145296 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0c74160250 0x7f0c74156040 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f0c6c0194f0 tx=0x7f0c6c019680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:07.256 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.234+0000 7f0c6a7fc640 1 -- 172.21.15.16:0/3826145296 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c6c0aed60 con 0x7f0c74160250 2023-05-22T15:27:07.256 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.234+0000 7f0c6a7fc640 1 -- 172.21.15.16:0/3826145296 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c6c009450 con 0x7f0c74160250 2023-05-22T15:27:07.257 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.234+0000 7f0c6a7fc640 1 -- 172.21.15.16:0/3826145296 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c6c0b7650 con 0x7f0c74160250 2023-05-22T15:27:07.257 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.234+0000 7f0c7bb2a640 1 -- 172.21.15.16:0/3826145296 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c740d12f0 con 0x7f0c74160250 2023-05-22T15:27:07.257 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.234+0000 7f0c7bb2a640 1 -- 172.21.15.16:0/3826145296 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c740d1160 con 0x7f0c74160250 2023-05-22T15:27:07.257 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.234+0000 7f0c7bb2a640 1 -- 172.21.15.16:0/3826145296 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c740d1480 con 0x7f0c74160250 2023-05-22T15:27:07.257 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.238+0000 7f0c6a7fc640 1 -- 172.21.15.16:0/3826145296 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7f0c6c0b4050 con 0x7f0c74160250 2023-05-22T15:27:07.258 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.238+0000 7f0c6a7fc640 1 --2- 172.21.15.16:0/3826145296 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f0c58065380 0x7f0c58067830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:07.258 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.238+0000 7f0c6a7fc640 1 -- 172.21.15.16:0/3826145296 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f0c6c11dc60 con 0x7f0c74160250 2023-05-22T15:27:07.258 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.238+0000 7f0c798c6640 1 --2- 172.21.15.16:0/3826145296 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f0c58065380 0x7f0c58067830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:07.258 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.238+0000 7f0c798c6640 1 --2- 172.21.15.16:0/3826145296 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f0c58065380 0x7f0c58067830 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f0c64009ae0 tx=0x7f0c64008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:07.259 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.242+0000 7f0c6a7fc640 1 -- 172.21.15.16:0/3826145296 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f0c6c0eb5f0 con 0x7f0c74160250 2023-05-22T15:27:07.395 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.386+0000 7f0c7bb2a640 1 -- 172.21.15.16:0/3826145296 --> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f0c7415ca60 con 0x7f0c58065380 2023-05-22T15:27:07.404 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.394+0000 7f0c6a7fc640 1 -- 172.21.15.16:0/3826145296 <== mgr.4101 v2:172.21.15.16:6832/36921 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+367319 (secure 0 0 0) 0x7f0c7415ca60 con 0x7f0c58065380 2023-05-22T15:27:07.408 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:27:07.410 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2023-05-22T15:27:07.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.398+0000 7f0c7bb2a640 1 -- 172.21.15.16:0/3826145296 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f0c58065380 msgr2=0x7f0c58067830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:07.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.398+0000 7f0c7bb2a640 1 --2- 172.21.15.16:0/3826145296 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f0c58065380 0x7f0c58067830 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f0c64009ae0 tx=0x7f0c64008040 comp rx=0 tx=0).stop 2023-05-22T15:27:07.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.398+0000 7f0c7bb2a640 1 -- 172.21.15.16:0/3826145296 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0c74160250 msgr2=0x7f0c74156040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:07.412 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.398+0000 7f0c7bb2a640 1 --2- 172.21.15.16:0/3826145296 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0c74160250 0x7f0c74156040 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f0c6c0194f0 tx=0x7f0c6c019680 comp rx=0 tx=0).stop 2023-05-22T15:27:07.412 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.398+0000 7f0c7bb2a640 1 -- 172.21.15.16:0/3826145296 shutdown_connections 2023-05-22T15:27:07.412 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.398+0000 7f0c7bb2a640 1 --2- 172.21.15.16:0/3826145296 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0c74160250 0x7f0c74156040 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:07.412 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.398+0000 7f0c7bb2a640 1 --2- 172.21.15.16:0/3826145296 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f0c58065380 0x7f0c58067830 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:07.412 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.398+0000 7f0c7bb2a640 1 --2- 172.21.15.16:0/3826145296 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0c7415f700 0x7f0c7415e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:07.412 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.398+0000 7f0c7bb2a640 1 --2- 172.21.15.16:0/3826145296 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0c740b7dd0 0x7f0c74156d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:07.412 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.398+0000 7f0c7bb2a640 1 -- 172.21.15.16:0/3826145296 >> 172.21.15.16:0/3826145296 conn(0x7f0c740655c0 msgr2=0x7f0c74070e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:07.413 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.398+0000 7f0c7bb2a640 1 -- 172.21.15.16:0/3826145296 shutdown_connections 2023-05-22T15:27:07.413 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.398+0000 7f0c7bb2a640 1 -- 172.21.15.16:0/3826145296 wait complete. 2023-05-22T15:27:07.424 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":35,"stamp":"2023-05-22T15:27:05.949158+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":594754,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":206,"num_write_kb":2433,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":158,"ondisk_log_size":158,"up":338,"acting":338,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":338,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":75688,"kb_used_data":3496,"kb_used_omap":0,"kb_used_meta":72128,"kb_avail":754899032,"statfs":{"total":773094113280,"available":773016608768,"internally_reserved":0,"allocated":3579904,"data_stored":2148716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":73859072},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2056,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.973981"},"pg_stats":[{"pgid":"5.19","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.979395+0000","last_change":"2023-05-22T15:26:53.651924+0000","last_active":"2023-05-22T15:26:53.979395+0000","last_peered":"2023-05-22T15:26:53.979395+0000","last_clean":"2023-05-22T15:26:53.979395+0000","last_became_active":"2023-05-22T15:26:53.651533+0000","last_became_peered":"2023-05-22T15:26:53.651533+0000","last_unstale":"2023-05-22T15:26:53.979395+0000","last_undegraded":"2023-05-22T15:26:53.979395+0000","last_fullsized":"2023-05-22T15:26:53.979395+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_clean_scrub_stamp":"2023-05-22T15:26:52.568670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:27:41.065402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7,2],"acting":[1,5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"22'4","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.612099+0000","last_change":"2023-05-22T15:26:49.186238+0000","last_active":"2023-05-22T15:26:53.612099+0000","last_peered":"2023-05-22T15:26:53.612099+0000","last_clean":"2023-05-22T15:26:53.612099+0000","last_became_active":"2023-05-22T15:26:49.185455+0000","last_became_peered":"2023-05-22T15:26:49.185455+0000","last_unstale":"2023-05-22T15:26:53.612099+0000","last_undegraded":"2023-05-22T15:26:53.612099+0000","last_fullsized":"2023-05-22T15:26:53.612099+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_clean_scrub_stamp":"2023-05-22T15:26:48.127413+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:21:54.376718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":566,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.612858+0000","last_change":"2023-05-22T15:26:49.622435+0000","last_active":"2023-05-22T15:26:53.612858+0000","last_peered":"2023-05-22T15:26:53.612858+0000","last_clean":"2023-05-22T15:26:53.612858+0000","last_became_active":"2023-05-22T15:26:49.621643+0000","last_became_peered":"2023-05-22T15:26:49.621643+0000","last_unstale":"2023-05-22T15:26:53.612858+0000","last_undegraded":"2023-05-22T15:26:53.612858+0000","last_fullsized":"2023-05-22T15:26:53.612858+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:08:07.637726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-05-22T15:26:54.054128+0000","last_change":"2023-05-22T15:26:53.654741+0000","last_active":"2023-05-22T15:26:54.054128+0000","last_peered":"2023-05-22T15:26:54.054128+0000","last_clean":"2023-05-22T15:26:54.054128+0000","last_became_active":"2023-05-22T15:26:53.653669+0000","last_became_peered":"2023-05-22T15:26:53.653669+0000","last_unstale":"2023-05-22T15:26:54.054128+0000","last_undegraded":"2023-05-22T15:26:54.054128+0000","last_fullsized":"2023-05-22T15:26:54.054128+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_clean_scrub_stamp":"2023-05-22T15:26:52.568670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:01:12.890843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,2],"acting":[4,6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"22'1","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.602985+0000","last_change":"2023-05-22T15:26:49.187902+0000","last_active":"2023-05-22T15:26:53.602985+0000","last_peered":"2023-05-22T15:26:53.602985+0000","last_clean":"2023-05-22T15:26:53.602985+0000","last_became_active":"2023-05-22T15:26:49.187361+0000","last_became_peered":"2023-05-22T15:26:49.187361+0000","last_unstale":"2023-05-22T15:26:53.602985+0000","last_undegraded":"2023-05-22T15:26:53.602985+0000","last_fullsized":"2023-05-22T15:26:53.602985+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_clean_scrub_stamp":"2023-05-22T15:26:48.127413+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T18:13:58.291903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.603896+0000","last_change":"2023-05-22T15:26:49.560606+0000","last_active":"2023-05-22T15:26:53.603896+0000","last_peered":"2023-05-22T15:26:53.603896+0000","last_clean":"2023-05-22T15:26:53.603896+0000","last_became_active":"2023-05-22T15:26:49.560318+0000","last_became_peered":"2023-05-22T15:26:49.560318+0000","last_unstale":"2023-05-22T15:26:53.603896+0000","last_undegraded":"2023-05-22T15:26:53.603896+0000","last_fullsized":"2023-05-22T15:26:53.603896+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:43:00.518715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-05-22T15:26:54.055130+0000","last_change":"2023-05-22T15:26:53.648550+0000","last_active":"2023-05-22T15:26:54.055130+0000","last_peered":"2023-05-22T15:26:54.055130+0000","last_clean":"2023-05-22T15:26:54.055130+0000","last_became_active":"2023-05-22T15:26:53.646940+0000","last_became_peered":"2023-05-22T15:26:53.646940+0000","last_unstale":"2023-05-22T15:26:54.055130+0000","last_undegraded":"2023-05-22T15:26:54.055130+0000","last_fullsized":"2023-05-22T15:26:54.055130+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_clean_scrub_stamp":"2023-05-22T15:26:52.568670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T15:41:55.740195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7,4],"acting":[5,0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"22'2","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.634944+0000","last_change":"2023-05-22T15:26:49.210260+0000","last_active":"2023-05-22T15:26:53.634944+0000","last_peered":"2023-05-22T15:26:53.634944+0000","last_clean":"2023-05-22T15:26:53.634944+0000","last_became_active":"2023-05-22T15:26:49.209487+0000","last_became_peered":"2023-05-22T15:26:49.209487+0000","last_unstale":"2023-05-22T15:26:53.634944+0000","last_undegraded":"2023-05-22T15:26:53.634944+0000","last_fullsized":"2023-05-22T15:26:53.634944+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_clean_scrub_stamp":"2023-05-22T15:26:48.127413+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:07:49.933860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.619548+0000","last_change":"2023-05-22T15:26:49.595133+0000","last_active":"2023-05-22T15:26:53.619548+0000","last_peered":"2023-05-22T15:26:53.619548+0000","last_clean":"2023-05-22T15:26:53.619548+0000","last_became_active":"2023-05-22T15:26:49.594382+0000","last_became_peered":"2023-05-22T15:26:49.594382+0000","last_unstale":"2023-05-22T15:26:53.619548+0000","last_undegraded":"2023-05-22T15:26:53.619548+0000","last_fullsized":"2023-05-22T15:26:53.619548+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:11:42.159966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.982645+0000","last_change":"2023-05-22T15:26:53.657958+0000","last_active":"2023-05-22T15:26:53.982645+0000","last_peered":"2023-05-22T15:26:53.982645+0000","last_clean":"2023-05-22T15:26:53.982645+0000","last_became_active":"2023-05-22T15:26:53.656910+0000","last_became_peered":"2023-05-22T15:26:53.656910+0000","last_unstale":"2023-05-22T15:26:53.982645+0000","last_undegraded":"2023-05-22T15:26:53.982645+0000","last_fullsized":"2023-05-22T15:26:53.982645+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_clean_scrub_stamp":"2023-05-22T15:26:52.568670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:56:11.423640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1,6],"acting":[7,4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"22'2","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.634146+0000","last_change":"2023-05-22T15:26:49.210149+0000","last_active":"2023-05-22T15:26:53.634146+0000","last_peered":"2023-05-22T15:26:53.634146+0000","last_clean":"2023-05-22T15:26:53.634146+0000","last_became_active":"2023-05-22T15:26:49.209497+0000","last_became_peered":"2023-05-22T15:26:49.209497+0000","last_unstale":"2023-05-22T15:26:53.634146+0000","last_undegraded":"2023-05-22T15:26:53.634146+0000","last_fullsized":"2023-05-22T15:26:53.634146+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_clean_scrub_stamp":"2023-05-22T15:26:48.127413+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:47:17.588354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.620540+0000","last_change":"2023-05-22T15:26:49.598141+0000","last_active":"2023-05-22T15:26:53.620540+0000","last_peered":"2023-05-22T15:26:53.620540+0000","last_clean":"2023-05-22T15:26:53.620540+0000","last_became_active":"2023-05-22T15:26:49.597262+0000","last_became_peered":"2023-05-22T15:26:49.597262+0000","last_unstale":"2023-05-22T15:26:53.620540+0000","last_undegraded":"2023-05-22T15:26:53.620540+0000","last_fullsized":"2023-05-22T15:26:53.620540+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:33:23.357846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.980103+0000","last_change":"2023-05-22T15:26:53.656672+0000","last_active":"2023-05-22T15:26:53.980103+0000","last_peered":"2023-05-22T15:26:53.980103+0000","last_clean":"2023-05-22T15:26:53.980103+0000","last_became_active":"2023-05-22T15:26:53.656058+0000","last_became_peered":"2023-05-22T15:26:53.656058+0000","last_unstale":"2023-05-22T15:26:53.980103+0000","last_undegraded":"2023-05-22T15:26:53.980103+0000","last_fullsized":"2023-05-22T15:26:53.980103+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_clean_scrub_stamp":"2023-05-22T15:26:52.568670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:38:53.647917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,0],"acting":[1,3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"22'1","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.612141+0000","last_change":"2023-05-22T15:26:49.206147+0000","last_active":"2023-05-22T15:26:53.612141+0000","last_peered":"2023-05-22T15:26:53.612141+0000","last_clean":"2023-05-22T15:26:53.612141+0000","last_became_active":"2023-05-22T15:26:49.205706+0000","last_became_peered":"2023-05-22T15:26:49.205706+0000","last_unstale":"2023-05-22T15:26:53.612141+0000","last_undegraded":"2023-05-22T15:26:53.612141+0000","last_fullsized":"2023-05-22T15:26:53.612141+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_clean_scrub_stamp":"2023-05-22T15:26:48.127413+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:09:00.816941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.601812+0000","last_change":"2023-05-22T15:26:49.578140+0000","last_active":"2023-05-22T15:26:53.601812+0000","last_peered":"2023-05-22T15:26:53.601812+0000","last_clean":"2023-05-22T15:26:53.601812+0000","last_became_active":"2023-05-22T15:26:49.576922+0000","last_became_peered":"2023-05-22T15:26:49.576922+0000","last_unstale":"2023-05-22T15:26:53.601812+0000","last_undegraded":"2023-05-22T15:26:53.601812+0000","last_fullsized":"2023-05-22T15:26:53.601812+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T16:16:40.288204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-05-22T15:26:54.054907+0000","last_change":"2023-05-22T15:26:53.643098+0000","last_active":"2023-05-22T15:26:54.054907+0000","last_peered":"2023-05-22T15:26:54.054907+0000","last_clean":"2023-05-22T15:26:54.054907+0000","last_became_active":"2023-05-22T15:26:53.642038+0000","last_became_peered":"2023-05-22T15:26:53.642038+0000","last_unstale":"2023-05-22T15:26:54.054907+0000","last_undegraded":"2023-05-22T15:26:54.054907+0000","last_fullsized":"2023-05-22T15:26:54.054907+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_clean_scrub_stamp":"2023-05-22T15:26:52.568670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T18:43:09.309787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2,0],"acting":[4,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"22'1","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.620161+0000","last_change":"2023-05-22T15:26:49.199743+0000","last_active":"2023-05-22T15:26:53.620161+0000","last_peered":"2023-05-22T15:26:53.620161+0000","last_clean":"2023-05-22T15:26:53.620161+0000","last_became_active":"2023-05-22T15:26:49.197695+0000","last_became_peered":"2023-05-22T15:26:49.197695+0000","last_unstale":"2023-05-22T15:26:53.620161+0000","last_undegraded":"2023-05-22T15:26:53.620161+0000","last_fullsized":"2023-05-22T15:26:53.620161+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_clean_scrub_stamp":"2023-05-22T15:26:48.127413+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:21:54.376718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.602293+0000","last_change":"2023-05-22T15:26:49.580932+0000","last_active":"2023-05-22T15:26:53.602293+0000","last_peered":"2023-05-22T15:26:53.602293+0000","last_clean":"2023-05-22T15:26:53.602293+0000","last_became_active":"2023-05-22T15:26:49.580612+0000","last_became_peered":"2023-05-22T15:26:49.580612+0000","last_unstale":"2023-05-22T15:26:53.602293+0000","last_undegraded":"2023-05-22T15:26:53.602293+0000","last_fullsized":"2023-05-22T15:26:53.602293+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:19:30.391062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-05-22T15:26:54.051798+0000","last_change":"2023-05-22T15:26:53.662170+0000","last_active":"2023-05-22T15:26:54.051798+0000","last_peered":"2023-05-22T15:26:54.051798+0000","last_clean":"2023-05-22T15:26:54.051798+0000","last_became_active":"2023-05-22T15:26:53.661164+0000","last_became_peered":"2023-05-22T15:26:53.661164+0000","last_unstale":"2023-05-22T15:26:54.051798+0000","last_undegraded":"2023-05-22T15:26:54.051798+0000","last_fullsized":"2023-05-22T15:26:54.051798+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_clean_scrub_stamp":"2023-05-22T15:26:52.568670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:46:53.737748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3,7],"acting":[6,4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.613865+0000","last_change":"2023-05-22T15:26:49.187546+0000","last_active":"2023-05-22T15:26:53.613865+0000","last_peered":"2023-05-22T15:26:53.613865+0000","last_clean":"2023-05-22T15:26:53.613865+0000","last_became_active":"2023-05-22T15:26:49.187022+0000","last_became_peered":"2023-05-22T15:26:49.187022+0000","last_unstale":"2023-05-22T15:26:53.613865+0000","last_undegraded":"2023-05-22T15:26:53.613865+0000","last_fullsized":"2023-05-22T15:26:53.613865+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_clean_scrub_stamp":"2023-05-22T15:26:48.127413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:53:34.802286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.611910+0000","last_change":"2023-05-22T15:26:49.623535+0000","last_active":"2023-05-22T15:26:53.611910+0000","last_peered":"2023-05-22T15:26:53.611910+0000","last_clean":"2023-05-22T15:26:53.611910+0000","last_became_active":"2023-05-22T15:26:49.622454+0000","last_became_peered":"2023-05-22T15:26:49.622454+0000","last_unstale":"2023-05-22T15:26:53.611910+0000","last_undegraded":"2023-05-22T15:26:53.611910+0000","last_fullsized":"2023-05-22T15:26:53.611910+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:43:00.518715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-05-22T15:26:54.048982+0000","last_change":"2023-05-22T15:26:53.645181+0000","last_active":"2023-05-22T15:26:54.048982+0000","last_peered":"2023-05-22T15:26:54.048982+0000","last_clean":"2023-05-22T15:26:54.048982+0000","last_became_active":"2023-05-22T15:26:53.644764+0000","last_became_peered":"2023-05-22T15:26:53.644764+0000","last_unstale":"2023-05-22T15:26:54.048982+0000","last_undegraded":"2023-05-22T15:26:54.048982+0000","last_fullsized":"2023-05-22T15:26:54.048982+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_clean_scrub_stamp":"2023-05-22T15:26:52.568670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:12:09.126657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1,2],"acting":[0,7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.603942+0000","last_change":"2023-05-22T15:26:49.188920+0000","last_active":"2023-05-22T15:26:53.603942+0000","last_peered":"2023-05-22T15:26:53.603942+0000","last_clean":"2023-05-22T15:26:53.603942+0000","last_became_active":"2023-05-22T15:26:49.188538+0000","last_became_peered":"2023-05-22T15:26:49.188538+0000","last_unstale":"2023-05-22T15:26:53.603942+0000","last_undegraded":"2023-05-22T15:26:53.603942+0000","last_fullsized":"2023-05-22T15:26:53.603942+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_clean_scrub_stamp":"2023-05-22T15:26:48.127413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:09:00.816941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.603003+0000","last_change":"2023-05-22T15:26:49.606092+0000","last_active":"2023-05-22T15:26:53.603003+0000","last_peered":"2023-05-22T15:26:53.603003+0000","last_clean":"2023-05-22T15:26:53.603003+0000","last_became_active":"2023-05-22T15:26:49.605324+0000","last_became_peered":"2023-05-22T15:26:49.605324+0000","last_unstale":"2023-05-22T15:26:53.603003+0000","last_undegraded":"2023-05-22T15:26:53.603003+0000","last_fullsized":"2023-05-22T15:26:53.603003+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:10:51.435508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.603748+0000","last_change":"2023-05-22T15:26:49.578105+0000","last_active":"2023-05-22T15:26:53.603748+0000","last_peered":"2023-05-22T15:26:53.603748+0000","last_clean":"2023-05-22T15:26:53.603748+0000","last_became_active":"2023-05-22T15:26:49.577534+0000","last_became_peered":"2023-05-22T15:26:49.577534+0000","last_unstale":"2023-05-22T15:26:53.603748+0000","last_undegraded":"2023-05-22T15:26:53.603748+0000","last_fullsized":"2023-05-22T15:26:53.603748+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:44:52.369716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.612292+0000","last_change":"2023-05-22T15:26:49.186504+0000","last_active":"2023-05-22T15:26:53.612292+0000","last_peered":"2023-05-22T15:26:53.612292+0000","last_clean":"2023-05-22T15:26:53.612292+0000","last_became_active":"2023-05-22T15:26:49.185880+0000","last_became_peered":"2023-05-22T15:26:49.185880+0000","last_unstale":"2023-05-22T15:26:53.612292+0000","last_undegraded":"2023-05-22T15:26:53.612292+0000","last_fullsized":"2023-05-22T15:26:53.612292+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_clean_scrub_stamp":"2023-05-22T15:26:48.127413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T18:13:58.291903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-05-22T15:26:54.051126+0000","last_change":"2023-05-22T15:26:53.647905+0000","last_active":"2023-05-22T15:26:54.051126+0000","last_peered":"2023-05-22T15:26:54.051126+0000","last_clean":"2023-05-22T15:26:54.051126+0000","last_became_active":"2023-05-22T15:26:53.646790+0000","last_became_peered":"2023-05-22T15:26:53.646790+0000","last_unstale":"2023-05-22T15:26:54.051126+0000","last_undegraded":"2023-05-22T15:26:54.051126+0000","last_fullsized":"2023-05-22T15:26:54.051126+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_clean_scrub_stamp":"2023-05-22T15:26:52.568670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T03:18:03.204915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7,1],"acting":[6,4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.603724+0000","last_change":"2023-05-22T15:26:49.623613+0000","last_active":"2023-05-22T15:26:53.603724+0000","last_peered":"2023-05-22T15:26:53.603724+0000","last_clean":"2023-05-22T15:26:53.603724+0000","last_became_active":"2023-05-22T15:26:49.623385+0000","last_became_peered":"2023-05-22T15:26:49.623385+0000","last_unstale":"2023-05-22T15:26:53.603724+0000","last_undegraded":"2023-05-22T15:26:53.603724+0000","last_fullsized":"2023-05-22T15:26:53.603724+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:22:10.117410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.635389+0000","last_change":"2023-05-22T15:26:49.178387+0000","last_active":"2023-05-22T15:26:53.635389+0000","last_peered":"2023-05-22T15:26:53.635389+0000","last_clean":"2023-05-22T15:26:53.635389+0000","last_became_active":"2023-05-22T15:26:49.177428+0000","last_became_peered":"2023-05-22T15:26:49.177428+0000","last_unstale":"2023-05-22T15:26:53.635389+0000","last_undegraded":"2023-05-22T15:26:53.635389+0000","last_fullsized":"2023-05-22T15:26:53.635389+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_clean_scrub_stamp":"2023-05-22T15:26:48.127413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:07:41.791660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.981431+0000","last_change":"2023-05-22T15:26:53.660574+0000","last_active":"2023-05-22T15:26:53.981431+0000","last_peered":"2023-05-22T15:26:53.981431+0000","last_clean":"2023-05-22T15:26:53.981431+0000","last_became_active":"2023-05-22T15:26:53.659567+0000","last_became_peered":"2023-05-22T15:26:53.659567+0000","last_unstale":"2023-05-22T15:26:53.981431+0000","last_undegraded":"2023-05-22T15:26:53.981431+0000","last_fullsized":"2023-05-22T15:26:53.981431+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_clean_scrub_stamp":"2023-05-22T15:26:52.568670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:07:24.608031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6,0],"acting":[7,4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.15","version":"22'3","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.610146+0000","last_change":"2023-05-22T15:26:49.188354+0000","last_active":"2023-05-22T15:26:53.610146+0000","last_peered":"2023-05-22T15:26:53.610146+0000","last_clean":"2023-05-22T15:26:53.610146+0000","last_became_active":"2023-05-22T15:26:49.187902+0000","last_became_peered":"2023-05-22T15:26:49.187902+0000","last_unstale":"2023-05-22T15:26:53.610146+0000","last_undegraded":"2023-05-22T15:26:53.610146+0000","last_fullsized":"2023-05-22T15:26:53.610146+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_clean_scrub_stamp":"2023-05-22T15:26:48.127413+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:07:49.933860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.612914+0000","last_change":"2023-05-22T15:26:49.619577+0000","last_active":"2023-05-22T15:26:53.612914+0000","last_peered":"2023-05-22T15:26:53.612914+0000","last_clean":"2023-05-22T15:26:53.612914+0000","last_became_active":"2023-05-22T15:26:49.619210+0000","last_became_peered":"2023-05-22T15:26:49.619210+0000","last_unstale":"2023-05-22T15:26:53.612914+0000","last_undegraded":"2023-05-22T15:26:53.612914+0000","last_fullsized":"2023-05-22T15:26:53.612914+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:48:33.611833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.992745+0000","last_change":"2023-05-22T15:26:53.650745+0000","last_active":"2023-05-22T15:26:53.992745+0000","last_peered":"2023-05-22T15:26:53.992745+0000","last_clean":"2023-05-22T15:26:53.992745+0000","last_became_active":"2023-05-22T15:26:53.650209+0000","last_became_peered":"2023-05-22T15:26:53.650209+0000","last_unstale":"2023-05-22T15:26:53.992745+0000","last_undegraded":"2023-05-22T15:26:53.992745+0000","last_fullsized":"2023-05-22T15:26:53.992745+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_clean_scrub_stamp":"2023-05-22T15:26:52.568670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:15:56.755592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,5],"acting":[3,0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.14","version":"27'8","reported_seq":22,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-05-22T15:26:57.205645+0000","last_change":"2023-05-22T15:26:49.201484+0000","last_active":"2023-05-22T15:26:57.205645+0000","last_peered":"2023-05-22T15:26:57.205645+0000","last_clean":"2023-05-22T15:26:57.205645+0000","last_became_active":"2023-05-22T15:26:49.200972+0000","last_became_peered":"2023-05-22T15:26:49.200972+0000","last_unstale":"2023-05-22T15:26:57.205645+0000","last_undegraded":"2023-05-22T15:26:57.205645+0000","last_fullsized":"2023-05-22T15:26:57.205645+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_clean_scrub_stamp":"2023-05-22T15:26:48.127413+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:59:54.009838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3604,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.620650+0000","last_change":"2023-05-22T15:26:49.619268+0000","last_active":"2023-05-22T15:26:53.620650+0000","last_peered":"2023-05-22T15:26:53.620650+0000","last_clean":"2023-05-22T15:26:53.620650+0000","last_became_active":"2023-05-22T15:26:49.618724+0000","last_became_peered":"2023-05-22T15:26:49.618724+0000","last_unstale":"2023-05-22T15:26:53.620650+0000","last_undegraded":"2023-05-22T15:26:53.620650+0000","last_fullsized":"2023-05-22T15:26:53.620650+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:28:29.607131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.979878+0000","last_change":"2023-05-22T15:26:53.653181+0000","last_active":"2023-05-22T15:26:53.979878+0000","last_peered":"2023-05-22T15:26:53.979878+0000","last_clean":"2023-05-22T15:26:53.979878+0000","last_became_active":"2023-05-22T15:26:53.652676+0000","last_became_peered":"2023-05-22T15:26:53.652676+0000","last_unstale":"2023-05-22T15:26:53.979878+0000","last_undegraded":"2023-05-22T15:26:53.979878+0000","last_fullsized":"2023-05-22T15:26:53.979878+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_clean_scrub_stamp":"2023-05-22T15:26:52.568670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T15:46:20.544673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3,7],"acting":[1,5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.616683+0000","last_change":"2023-05-22T15:26:49.597250+0000","last_active":"2023-05-22T15:26:53.616683+0000","last_peered":"2023-05-22T15:26:53.616683+0000","last_clean":"2023-05-22T15:26:53.616683+0000","last_became_active":"2023-05-22T15:26:49.596487+0000","last_became_peered":"2023-05-22T15:26:49.596487+0000","last_unstale":"2023-05-22T15:26:53.616683+0000","last_undegraded":"2023-05-22T15:26:53.616683+0000","last_fullsized":"2023-05-22T15:26:53.616683+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:18:20.756810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.603054+0000","last_change":"2023-05-22T15:26:44.982421+0000","last_active":"2023-05-22T15:26:53.603054+0000","last_peered":"2023-05-22T15:26:53.603054+0000","last_clean":"2023-05-22T15:26:53.603054+0000","last_became_active":"2023-05-22T15:26:44.981586+0000","last_became_peered":"2023-05-22T15:26:44.981586+0000","last_unstale":"2023-05-22T15:26:53.603054+0000","last_undegraded":"2023-05-22T15:26:53.603054+0000","last_fullsized":"2023-05-22T15:26:53.603054+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:14:11.156544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"22'1","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.612191+0000","last_change":"2023-05-22T15:26:49.185412+0000","last_active":"2023-05-22T15:26:53.612191+0000","last_peered":"2023-05-22T15:26:53.612191+0000","last_clean":"2023-05-22T15:26:53.612191+0000","last_became_active":"2023-05-22T15:26:49.184896+0000","last_became_peered":"2023-05-22T15:26:49.184896+0000","last_unstale":"2023-05-22T15:26:53.612191+0000","last_undegraded":"2023-05-22T15:26:53.612191+0000","last_fullsized":"2023-05-22T15:26:53.612191+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_clean_scrub_stamp":"2023-05-22T15:26:48.127413+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:13:03.975127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.992364+0000","last_change":"2023-05-22T15:26:53.669441+0000","last_active":"2023-05-22T15:26:53.992364+0000","last_peered":"2023-05-22T15:26:53.992364+0000","last_clean":"2023-05-22T15:26:53.992364+0000","last_became_active":"2023-05-22T15:26:53.669000+0000","last_became_peered":"2023-05-22T15:26:53.669000+0000","last_unstale":"2023-05-22T15:26:53.992364+0000","last_undegraded":"2023-05-22T15:26:53.992364+0000","last_fullsized":"2023-05-22T15:26:53.992364+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_clean_scrub_stamp":"2023-05-22T15:26:52.568670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T18:55:06.398032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,0],"acting":[3,1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.603843+0000","last_change":"2023-05-22T15:26:49.623319+0000","last_active":"2023-05-22T15:26:53.603843+0000","last_peered":"2023-05-22T15:26:53.603843+0000","last_clean":"2023-05-22T15:26:53.603843+0000","last_became_active":"2023-05-22T15:26:49.622892+0000","last_became_peered":"2023-05-22T15:26:49.622892+0000","last_unstale":"2023-05-22T15:26:53.603843+0000","last_undegraded":"2023-05-22T15:26:53.603843+0000","last_fullsized":"2023-05-22T15:26:53.603843+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:48:33.611833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.612572+0000","last_change":"2023-05-22T15:26:44.978858+0000","last_active":"2023-05-22T15:26:53.612572+0000","last_peered":"2023-05-22T15:26:53.612572+0000","last_clean":"2023-05-22T15:26:53.612572+0000","last_became_active":"2023-05-22T15:26:44.978443+0000","last_became_peered":"2023-05-22T15:26:44.978443+0000","last_unstale":"2023-05-22T15:26:53.612572+0000","last_undegraded":"2023-05-22T15:26:53.612572+0000","last_fullsized":"2023-05-22T15:26:53.612572+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:46:29.290243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"22'2","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.606372+0000","last_change":"2023-05-22T15:26:49.188060+0000","last_active":"2023-05-22T15:26:53.606372+0000","last_peered":"2023-05-22T15:26:53.606372+0000","last_clean":"2023-05-22T15:26:53.606372+0000","last_became_active":"2023-05-22T15:26:49.187586+0000","last_became_peered":"2023-05-22T15:26:49.187586+0000","last_unstale":"2023-05-22T15:26:53.606372+0000","last_undegraded":"2023-05-22T15:26:53.606372+0000","last_fullsized":"2023-05-22T15:26:53.606372+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_clean_scrub_stamp":"2023-05-22T15:26:48.127413+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:59:54.009838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-05-22T15:26:54.053713+0000","last_change":"2023-05-22T15:26:53.659559+0000","last_active":"2023-05-22T15:26:54.053713+0000","last_peered":"2023-05-22T15:26:54.053713+0000","last_clean":"2023-05-22T15:26:54.053713+0000","last_became_active":"2023-05-22T15:26:53.655056+0000","last_became_peered":"2023-05-22T15:26:53.655056+0000","last_unstale":"2023-05-22T15:26:54.053713+0000","last_undegraded":"2023-05-22T15:26:54.053713+0000","last_fullsized":"2023-05-22T15:26:54.053713+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_clean_scrub_stamp":"2023-05-22T15:26:52.568670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:47:56.845713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7,0],"acting":[4,3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.611858+0000","last_change":"2023-05-22T15:26:49.589781+0000","last_active":"2023-05-22T15:26:53.611858+0000","last_peered":"2023-05-22T15:26:53.611858+0000","last_clean":"2023-05-22T15:26:53.611858+0000","last_became_active":"2023-05-22T15:26:49.588830+0000","last_became_peered":"2023-05-22T15:26:49.588830+0000","last_unstale":"2023-05-22T15:26:53.611858+0000","last_undegraded":"2023-05-22T15:26:53.611858+0000","last_fullsized":"2023-05-22T15:26:53.611858+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:52:48.123148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.615526+0000","last_change":"2023-05-22T15:26:45.092367+0000","last_active":"2023-05-22T15:26:53.615526+0000","last_peered":"2023-05-22T15:26:53.615526+0000","last_clean":"2023-05-22T15:26:53.615526+0000","last_became_active":"2023-05-22T15:26:45.091951+0000","last_became_peered":"2023-05-22T15:26:45.091951+0000","last_unstale":"2023-05-22T15:26:53.615526+0000","last_undegraded":"2023-05-22T15:26:53.615526+0000","last_fullsized":"2023-05-22T15:26:53.615526+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:42:57.973030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.619125+0000","last_change":"2023-05-22T15:26:49.180237+0000","last_active":"2023-05-22T15:26:53.619125+0000","last_peered":"2023-05-22T15:26:53.619125+0000","last_clean":"2023-05-22T15:26:53.619125+0000","last_became_active":"2023-05-22T15:26:49.179622+0000","last_became_peered":"2023-05-22T15:26:49.179622+0000","last_unstale":"2023-05-22T15:26:53.619125+0000","last_undegraded":"2023-05-22T15:26:53.619125+0000","last_fullsized":"2023-05-22T15:26:53.619125+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_clean_scrub_stamp":"2023-05-22T15:26:48.127413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T16:16:39.277093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-05-22T15:26:54.051370+0000","last_change":"2023-05-22T15:26:53.623285+0000","last_active":"2023-05-22T15:26:54.051370+0000","last_peered":"2023-05-22T15:26:54.051370+0000","last_clean":"2023-05-22T15:26:54.051370+0000","last_became_active":"2023-05-22T15:26:53.622116+0000","last_became_peered":"2023-05-22T15:26:53.622116+0000","last_unstale":"2023-05-22T15:26:54.051370+0000","last_undegraded":"2023-05-22T15:26:54.051370+0000","last_fullsized":"2023-05-22T15:26:54.051370+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_clean_scrub_stamp":"2023-05-22T15:26:52.568670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T16:31:40.104597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5,2],"acting":[6,0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.613088+0000","last_change":"2023-05-22T15:26:49.576833+0000","last_active":"2023-05-22T15:26:53.613088+0000","last_peered":"2023-05-22T15:26:53.613088+0000","last_clean":"2023-05-22T15:26:53.613088+0000","last_became_active":"2023-05-22T15:26:49.576559+0000","last_became_peered":"2023-05-22T15:26:49.576559+0000","last_unstale":"2023-05-22T15:26:53.613088+0000","last_undegraded":"2023-05-22T15:26:53.613088+0000","last_fullsized":"2023-05-22T15:26:53.613088+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:32:19.582542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.619335+0000","last_change":"2023-05-22T15:26:45.091475+0000","last_active":"2023-05-22T15:26:53.619335+0000","last_peered":"2023-05-22T15:26:53.619335+0000","last_clean":"2023-05-22T15:26:53.619335+0000","last_became_active":"2023-05-22T15:26:45.091143+0000","last_became_peered":"2023-05-22T15:26:45.091143+0000","last_unstale":"2023-05-22T15:26:53.619335+0000","last_undegraded":"2023-05-22T15:26:53.619335+0000","last_fullsized":"2023-05-22T15:26:53.619335+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:12:41.464054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.635151+0000","last_change":"2023-05-22T15:26:49.187061+0000","last_active":"2023-05-22T15:26:53.635151+0000","last_peered":"2023-05-22T15:26:53.635151+0000","last_clean":"2023-05-22T15:26:53.635151+0000","last_became_active":"2023-05-22T15:26:49.185965+0000","last_became_peered":"2023-05-22T15:26:49.185965+0000","last_unstale":"2023-05-22T15:26:53.635151+0000","last_undegraded":"2023-05-22T15:26:53.635151+0000","last_fullsized":"2023-05-22T15:26:53.635151+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_clean_scrub_stamp":"2023-05-22T15:26:48.127413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T16:16:39.277093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-05-22T15:26:54.048601+0000","last_change":"2023-05-22T15:26:53.650446+0000","last_active":"2023-05-22T15:26:54.048601+0000","last_peered":"2023-05-22T15:26:54.048601+0000","last_clean":"2023-05-22T15:26:54.048601+0000","last_became_active":"2023-05-22T15:26:53.649492+0000","last_became_peered":"2023-05-22T15:26:53.649492+0000","last_unstale":"2023-05-22T15:26:54.048601+0000","last_undegraded":"2023-05-22T15:26:54.048601+0000","last_fullsized":"2023-05-22T15:26:54.048601+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_clean_scrub_stamp":"2023-05-22T15:26:52.568670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:56:11.423640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6,5],"acting":[0,1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.619014+0000","last_change":"2023-05-22T15:26:49.581808+0000","last_active":"2023-05-22T15:26:53.619014+0000","last_peered":"2023-05-22T15:26:53.619014+0000","last_clean":"2023-05-22T15:26:53.619014+0000","last_became_active":"2023-05-22T15:26:49.581501+0000","last_became_peered":"2023-05-22T15:26:49.581501+0000","last_unstale":"2023-05-22T15:26:53.619014+0000","last_undegraded":"2023-05-22T15:26:53.619014+0000","last_fullsized":"2023-05-22T15:26:53.619014+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:52:48.123148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.620868+0000","last_change":"2023-05-22T15:26:49.204122+0000","last_active":"2023-05-22T15:26:53.620868+0000","last_peered":"2023-05-22T15:26:53.620868+0000","last_clean":"2023-05-22T15:26:53.620868+0000","last_became_active":"2023-05-22T15:26:49.199596+0000","last_became_peered":"2023-05-22T15:26:49.199596+0000","last_unstale":"2023-05-22T15:26:53.620868+0000","last_undegraded":"2023-05-22T15:26:53.620868+0000","last_fullsized":"2023-05-22T15:26:53.620868+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_clean_scrub_stamp":"2023-05-22T15:26:48.127413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:09:00.816941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.635108+0000","last_change":"2023-05-22T15:26:44.976632+0000","last_active":"2023-05-22T15:26:53.635108+0000","last_peered":"2023-05-22T15:26:53.635108+0000","last_clean":"2023-05-22T15:26:53.635108+0000","last_became_active":"2023-05-22T15:26:44.975546+0000","last_became_peered":"2023-05-22T15:26:44.975546+0000","last_unstale":"2023-05-22T15:26:53.635108+0000","last_undegraded":"2023-05-22T15:26:53.635108+0000","last_fullsized":"2023-05-22T15:26:53.635108+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:42:57.973030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-05-22T15:26:54.048781+0000","last_change":"2023-05-22T15:26:53.669482+0000","last_active":"2023-05-22T15:26:54.048781+0000","last_peered":"2023-05-22T15:26:54.048781+0000","last_clean":"2023-05-22T15:26:54.048781+0000","last_became_active":"2023-05-22T15:26:53.669119+0000","last_became_peered":"2023-05-22T15:26:53.669119+0000","last_unstale":"2023-05-22T15:26:54.048781+0000","last_undegraded":"2023-05-22T15:26:54.048781+0000","last_fullsized":"2023-05-22T15:26:54.048781+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_clean_scrub_stamp":"2023-05-22T15:26:52.568670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:38:53.647917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,6],"acting":[0,1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.613087+0000","last_change":"2023-05-22T15:26:49.589555+0000","last_active":"2023-05-22T15:26:53.613087+0000","last_peered":"2023-05-22T15:26:53.613087+0000","last_clean":"2023-05-22T15:26:53.613087+0000","last_became_active":"2023-05-22T15:26:49.589271+0000","last_became_peered":"2023-05-22T15:26:49.589271+0000","last_unstale":"2023-05-22T15:26:53.613087+0000","last_undegraded":"2023-05-22T15:26:53.613087+0000","last_fullsized":"2023-05-22T15:26:53.613087+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:43:00.518715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.614048+0000","last_change":"2023-05-22T15:26:49.179359+0000","last_active":"2023-05-22T15:26:53.614048+0000","last_peered":"2023-05-22T15:26:53.614048+0000","last_clean":"2023-05-22T15:26:53.614048+0000","last_became_active":"2023-05-22T15:26:49.178930+0000","last_became_peered":"2023-05-22T15:26:49.178930+0000","last_unstale":"2023-05-22T15:26:53.614048+0000","last_undegraded":"2023-05-22T15:26:53.614048+0000","last_fullsized":"2023-05-22T15:26:53.614048+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_clean_scrub_stamp":"2023-05-22T15:26:48.127413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:59:54.009838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.597765+0000","last_change":"2023-05-22T15:26:44.969469+0000","last_active":"2023-05-22T15:26:53.597765+0000","last_peered":"2023-05-22T15:26:53.597765+0000","last_clean":"2023-05-22T15:26:53.597765+0000","last_became_active":"2023-05-22T15:26:44.969083+0000","last_became_peered":"2023-05-22T15:26:44.969083+0000","last_unstale":"2023-05-22T15:26:53.597765+0000","last_undegraded":"2023-05-22T15:26:53.597765+0000","last_fullsized":"2023-05-22T15:26:53.597765+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:29:23.670067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.978603+0000","last_change":"2023-05-22T15:26:53.636902+0000","last_active":"2023-05-22T15:26:53.978603+0000","last_peered":"2023-05-22T15:26:53.978603+0000","last_clean":"2023-05-22T15:26:53.978603+0000","last_became_active":"2023-05-22T15:26:53.636337+0000","last_became_peered":"2023-05-22T15:26:53.636337+0000","last_unstale":"2023-05-22T15:26:53.978603+0000","last_undegraded":"2023-05-22T15:26:53.978603+0000","last_fullsized":"2023-05-22T15:26:53.978603+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_clean_scrub_stamp":"2023-05-22T15:26:52.568670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:58:01.437826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1,7],"acting":[2,5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.612001+0000","last_change":"2023-05-22T15:26:49.581103+0000","last_active":"2023-05-22T15:26:53.612001+0000","last_peered":"2023-05-22T15:26:53.612001+0000","last_clean":"2023-05-22T15:26:53.612001+0000","last_became_active":"2023-05-22T15:26:49.580823+0000","last_became_peered":"2023-05-22T15:26:49.580823+0000","last_unstale":"2023-05-22T15:26:53.612001+0000","last_undegraded":"2023-05-22T15:26:53.612001+0000","last_fullsized":"2023-05-22T15:26:53.612001+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:32:19.582542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.611473+0000","last_change":"2023-05-22T15:26:49.205877+0000","last_active":"2023-05-22T15:26:53.611473+0000","last_peered":"2023-05-22T15:26:53.611473+0000","last_clean":"2023-05-22T15:26:53.611473+0000","last_became_active":"2023-05-22T15:26:49.205432+0000","last_became_peered":"2023-05-22T15:26:49.205432+0000","last_unstale":"2023-05-22T15:26:53.611473+0000","last_undegraded":"2023-05-22T15:26:53.611473+0000","last_fullsized":"2023-05-22T15:26:53.611473+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_clean_scrub_stamp":"2023-05-22T15:26:48.127413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:59:54.009838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.609876+0000","last_change":"2023-05-22T15:26:44.977198+0000","last_active":"2023-05-22T15:26:53.609876+0000","last_peered":"2023-05-22T15:26:53.609876+0000","last_clean":"2023-05-22T15:26:53.609876+0000","last_became_active":"2023-05-22T15:26:44.976445+0000","last_became_peered":"2023-05-22T15:26:44.976445+0000","last_unstale":"2023-05-22T15:26:53.609876+0000","last_undegraded":"2023-05-22T15:26:53.609876+0000","last_fullsized":"2023-05-22T15:26:53.609876+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:12:41.464054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-05-22T15:26:54.056650+0000","last_change":"2023-05-22T15:26:53.667459+0000","last_active":"2023-05-22T15:26:54.056650+0000","last_peered":"2023-05-22T15:26:54.056650+0000","last_clean":"2023-05-22T15:26:54.056650+0000","last_became_active":"2023-05-22T15:26:53.666553+0000","last_became_peered":"2023-05-22T15:26:53.666553+0000","last_unstale":"2023-05-22T15:26:54.056650+0000","last_undegraded":"2023-05-22T15:26:54.056650+0000","last_fullsized":"2023-05-22T15:26:54.056650+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_clean_scrub_stamp":"2023-05-22T15:26:52.568670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:08:31.638326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0,4],"acting":[5,1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.602418+0000","last_change":"2023-05-22T15:26:49.606241+0000","last_active":"2023-05-22T15:26:53.602418+0000","last_peered":"2023-05-22T15:26:53.602418+0000","last_clean":"2023-05-22T15:26:53.602418+0000","last_became_active":"2023-05-22T15:26:49.605628+0000","last_became_peered":"2023-05-22T15:26:49.605628+0000","last_unstale":"2023-05-22T15:26:53.602418+0000","last_undegraded":"2023-05-22T15:26:53.602418+0000","last_fullsized":"2023-05-22T15:26:53.602418+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:59:23.805894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.602217+0000","last_change":"2023-05-22T15:26:49.185421+0000","last_active":"2023-05-22T15:26:53.602217+0000","last_peered":"2023-05-22T15:26:53.602217+0000","last_clean":"2023-05-22T15:26:53.602217+0000","last_became_active":"2023-05-22T15:26:49.184948+0000","last_became_peered":"2023-05-22T15:26:49.184948+0000","last_unstale":"2023-05-22T15:26:53.602217+0000","last_undegraded":"2023-05-22T15:26:53.602217+0000","last_fullsized":"2023-05-22T15:26:53.602217+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_clean_scrub_stamp":"2023-05-22T15:26:48.127413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:21:54.376718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.634293+0000","last_change":"2023-05-22T15:26:44.971951+0000","last_active":"2023-05-22T15:26:53.634293+0000","last_peered":"2023-05-22T15:26:53.634293+0000","last_clean":"2023-05-22T15:26:53.634293+0000","last_became_active":"2023-05-22T15:26:44.970872+0000","last_became_peered":"2023-05-22T15:26:44.970872+0000","last_unstale":"2023-05-22T15:26:53.634293+0000","last_undegraded":"2023-05-22T15:26:53.634293+0000","last_fullsized":"2023-05-22T15:26:53.634293+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:55:34.752949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":169,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.610124+0000","last_change":"2023-05-22T15:26:39.318204+0000","last_active":"2023-05-22T15:26:53.610124+0000","last_peered":"2023-05-22T15:26:53.610124+0000","last_clean":"2023-05-22T15:26:53.610124+0000","last_became_active":"2023-05-22T15:26:39.315370+0000","last_became_peered":"2023-05-22T15:26:39.315370+0000","last_unstale":"2023-05-22T15:26:53.610124+0000","last_undegraded":"2023-05-22T15:26:53.610124+0000","last_fullsized":"2023-05-22T15:26:53.610124+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:38.020076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:38.020076+0000","last_clean_scrub_stamp":"2023-05-22T15:26:38.020076+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:06:39.207543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.982716+0000","last_change":"2023-05-22T15:26:53.649648+0000","last_active":"2023-05-22T15:26:53.982716+0000","last_peered":"2023-05-22T15:26:53.982716+0000","last_clean":"2023-05-22T15:26:53.982716+0000","last_became_active":"2023-05-22T15:26:53.648664+0000","last_became_peered":"2023-05-22T15:26:53.648664+0000","last_unstale":"2023-05-22T15:26:53.982716+0000","last_undegraded":"2023-05-22T15:26:53.982716+0000","last_fullsized":"2023-05-22T15:26:53.982716+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_clean_scrub_stamp":"2023-05-22T15:26:52.568670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T03:14:12.733785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5,3],"acting":[7,2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.612919+0000","last_change":"2023-05-22T15:26:49.578502+0000","last_active":"2023-05-22T15:26:53.612919+0000","last_peered":"2023-05-22T15:26:53.612919+0000","last_clean":"2023-05-22T15:26:53.612919+0000","last_became_active":"2023-05-22T15:26:49.577743+0000","last_became_peered":"2023-05-22T15:26:49.577743+0000","last_unstale":"2023-05-22T15:26:53.612919+0000","last_undegraded":"2023-05-22T15:26:53.612919+0000","last_fullsized":"2023-05-22T15:26:53.612919+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:22:10.117410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.604031+0000","last_change":"2023-05-22T15:26:49.191706+0000","last_active":"2023-05-22T15:26:53.604031+0000","last_peered":"2023-05-22T15:26:53.604031+0000","last_clean":"2023-05-22T15:26:53.604031+0000","last_became_active":"2023-05-22T15:26:49.190396+0000","last_became_peered":"2023-05-22T15:26:49.190396+0000","last_unstale":"2023-05-22T15:26:53.604031+0000","last_undegraded":"2023-05-22T15:26:53.604031+0000","last_fullsized":"2023-05-22T15:26:53.604031+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_clean_scrub_stamp":"2023-05-22T15:26:48.127413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:13:03.975127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-05-22T15:26:54.055266+0000","last_change":"2023-05-22T15:26:53.656007+0000","last_active":"2023-05-22T15:26:54.055266+0000","last_peered":"2023-05-22T15:26:54.055266+0000","last_clean":"2023-05-22T15:26:54.055266+0000","last_became_active":"2023-05-22T15:26:53.652989+0000","last_became_peered":"2023-05-22T15:26:53.652989+0000","last_unstale":"2023-05-22T15:26:54.055266+0000","last_undegraded":"2023-05-22T15:26:54.055266+0000","last_fullsized":"2023-05-22T15:26:54.055266+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_clean_scrub_stamp":"2023-05-22T15:26:52.568670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:36:00.298083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0,2],"acting":[4,5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.621541+0000","last_change":"2023-05-22T15:26:49.622122+0000","last_active":"2023-05-22T15:26:53.621541+0000","last_peered":"2023-05-22T15:26:53.621541+0000","last_clean":"2023-05-22T15:26:53.621541+0000","last_became_active":"2023-05-22T15:26:49.621591+0000","last_became_peered":"2023-05-22T15:26:49.621591+0000","last_unstale":"2023-05-22T15:26:53.621541+0000","last_undegraded":"2023-05-22T15:26:53.621541+0000","last_fullsized":"2023-05-22T15:26:53.621541+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T18:05:09.939239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.617051+0000","last_change":"2023-05-22T15:26:49.200650+0000","last_active":"2023-05-22T15:26:53.617051+0000","last_peered":"2023-05-22T15:26:53.617051+0000","last_clean":"2023-05-22T15:26:53.617051+0000","last_became_active":"2023-05-22T15:26:49.199682+0000","last_became_peered":"2023-05-22T15:26:49.199682+0000","last_unstale":"2023-05-22T15:26:53.617051+0000","last_undegraded":"2023-05-22T15:26:53.617051+0000","last_fullsized":"2023-05-22T15:26:53.617051+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_clean_scrub_stamp":"2023-05-22T15:26:48.127413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:13:03.975127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-05-22T15:26:54.056477+0000","last_change":"2023-05-22T15:26:53.667847+0000","last_active":"2023-05-22T15:26:54.056477+0000","last_peered":"2023-05-22T15:26:54.056477+0000","last_clean":"2023-05-22T15:26:54.056477+0000","last_became_active":"2023-05-22T15:26:53.666866+0000","last_became_peered":"2023-05-22T15:26:53.666866+0000","last_unstale":"2023-05-22T15:26:54.056477+0000","last_undegraded":"2023-05-22T15:26:54.056477+0000","last_fullsized":"2023-05-22T15:26:54.056477+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_clean_scrub_stamp":"2023-05-22T15:26:52.568670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:30:40.980509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6,2],"acting":[5,4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.631889+0000","last_change":"2023-05-22T15:26:49.598177+0000","last_active":"2023-05-22T15:26:53.631889+0000","last_peered":"2023-05-22T15:26:53.631889+0000","last_clean":"2023-05-22T15:26:53.631889+0000","last_became_active":"2023-05-22T15:26:49.594848+0000","last_became_peered":"2023-05-22T15:26:49.594848+0000","last_unstale":"2023-05-22T15:26:53.631889+0000","last_undegraded":"2023-05-22T15:26:53.631889+0000","last_fullsized":"2023-05-22T15:26:53.631889+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T03:14:09.303640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.603009+0000","last_change":"2023-05-22T15:26:49.206461+0000","last_active":"2023-05-22T15:26:53.603009+0000","last_peered":"2023-05-22T15:26:53.603009+0000","last_clean":"2023-05-22T15:26:53.603009+0000","last_became_active":"2023-05-22T15:26:49.205870+0000","last_became_peered":"2023-05-22T15:26:49.205870+0000","last_unstale":"2023-05-22T15:26:53.603009+0000","last_undegraded":"2023-05-22T15:26:53.603009+0000","last_fullsized":"2023-05-22T15:26:53.603009+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_clean_scrub_stamp":"2023-05-22T15:26:48.127413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T18:13:58.291903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.980043+0000","last_change":"2023-05-22T15:26:53.642805+0000","last_active":"2023-05-22T15:26:53.980043+0000","last_peered":"2023-05-22T15:26:53.980043+0000","last_clean":"2023-05-22T15:26:53.980043+0000","last_became_active":"2023-05-22T15:26:53.642142+0000","last_became_peered":"2023-05-22T15:26:53.642142+0000","last_unstale":"2023-05-22T15:26:53.980043+0000","last_undegraded":"2023-05-22T15:26:53.980043+0000","last_fullsized":"2023-05-22T15:26:53.980043+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_clean_scrub_stamp":"2023-05-22T15:26:52.568670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:12:09.126657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0,6],"acting":[1,4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.623794+0000","last_change":"2023-05-22T15:26:49.600035+0000","last_active":"2023-05-22T15:26:53.623794+0000","last_peered":"2023-05-22T15:26:53.623794+0000","last_clean":"2023-05-22T15:26:53.623794+0000","last_became_active":"2023-05-22T15:26:49.597436+0000","last_became_peered":"2023-05-22T15:26:49.597436+0000","last_unstale":"2023-05-22T15:26:53.623794+0000","last_undegraded":"2023-05-22T15:26:53.623794+0000","last_fullsized":"2023-05-22T15:26:53.623794+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:46:50.307602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"22'2","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.606082+0000","last_change":"2023-05-22T15:26:49.191345+0000","last_active":"2023-05-22T15:26:53.606082+0000","last_peered":"2023-05-22T15:26:53.606082+0000","last_clean":"2023-05-22T15:26:53.606082+0000","last_became_active":"2023-05-22T15:26:49.190291+0000","last_became_peered":"2023-05-22T15:26:49.190291+0000","last_unstale":"2023-05-22T15:26:53.606082+0000","last_undegraded":"2023-05-22T15:26:53.606082+0000","last_fullsized":"2023-05-22T15:26:53.606082+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_clean_scrub_stamp":"2023-05-22T15:26:48.127413+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:13:21.177657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.979168+0000","last_change":"2023-05-22T15:26:53.649119+0000","last_active":"2023-05-22T15:26:53.979168+0000","last_peered":"2023-05-22T15:26:53.979168+0000","last_clean":"2023-05-22T15:26:53.979168+0000","last_became_active":"2023-05-22T15:26:53.648706+0000","last_became_peered":"2023-05-22T15:26:53.648706+0000","last_unstale":"2023-05-22T15:26:53.979168+0000","last_undegraded":"2023-05-22T15:26:53.979168+0000","last_fullsized":"2023-05-22T15:26:53.979168+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_clean_scrub_stamp":"2023-05-22T15:26:52.568670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T18:05:13.369385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5,6],"acting":[2,7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.611714+0000","last_change":"2023-05-22T15:26:49.581996+0000","last_active":"2023-05-22T15:26:53.611714+0000","last_peered":"2023-05-22T15:26:53.611714+0000","last_clean":"2023-05-22T15:26:53.611714+0000","last_became_active":"2023-05-22T15:26:49.581611+0000","last_became_peered":"2023-05-22T15:26:49.581611+0000","last_unstale":"2023-05-22T15:26:53.611714+0000","last_undegraded":"2023-05-22T15:26:53.611714+0000","last_fullsized":"2023-05-22T15:26:53.611714+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:44:52.369716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.634393+0000","last_change":"2023-05-22T15:26:49.192121+0000","last_active":"2023-05-22T15:26:53.634393+0000","last_peered":"2023-05-22T15:26:53.634393+0000","last_clean":"2023-05-22T15:26:53.634393+0000","last_became_active":"2023-05-22T15:26:49.191470+0000","last_became_peered":"2023-05-22T15:26:49.191470+0000","last_unstale":"2023-05-22T15:26:53.634393+0000","last_undegraded":"2023-05-22T15:26:53.634393+0000","last_fullsized":"2023-05-22T15:26:53.634393+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_clean_scrub_stamp":"2023-05-22T15:26:48.127413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:53:34.802286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.979064+0000","last_change":"2023-05-22T15:26:53.656554+0000","last_active":"2023-05-22T15:26:53.979064+0000","last_peered":"2023-05-22T15:26:53.979064+0000","last_clean":"2023-05-22T15:26:53.979064+0000","last_became_active":"2023-05-22T15:26:53.655950+0000","last_became_peered":"2023-05-22T15:26:53.655950+0000","last_unstale":"2023-05-22T15:26:53.979064+0000","last_undegraded":"2023-05-22T15:26:53.979064+0000","last_fullsized":"2023-05-22T15:26:53.979064+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_clean_scrub_stamp":"2023-05-22T15:26:52.568670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:11:45.590112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3,6],"acting":[2,4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.602652+0000","last_change":"2023-05-22T15:26:49.618498+0000","last_active":"2023-05-22T15:26:53.602652+0000","last_peered":"2023-05-22T15:26:53.602652+0000","last_clean":"2023-05-22T15:26:53.602652+0000","last_became_active":"2023-05-22T15:26:49.618068+0000","last_became_peered":"2023-05-22T15:26:49.618068+0000","last_unstale":"2023-05-22T15:26:53.602652+0000","last_undegraded":"2023-05-22T15:26:53.602652+0000","last_fullsized":"2023-05-22T15:26:53.602652+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:55:20.098659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"22'1","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.610411+0000","last_change":"2023-05-22T15:26:49.186875+0000","last_active":"2023-05-22T15:26:53.610411+0000","last_peered":"2023-05-22T15:26:53.610411+0000","last_clean":"2023-05-22T15:26:53.610411+0000","last_became_active":"2023-05-22T15:26:49.186431+0000","last_became_peered":"2023-05-22T15:26:49.186431+0000","last_unstale":"2023-05-22T15:26:53.610411+0000","last_undegraded":"2023-05-22T15:26:53.610411+0000","last_fullsized":"2023-05-22T15:26:53.610411+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_clean_scrub_stamp":"2023-05-22T15:26:48.127413+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:59:22.794782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.978460+0000","last_change":"2023-05-22T15:26:53.649077+0000","last_active":"2023-05-22T15:26:53.978460+0000","last_peered":"2023-05-22T15:26:53.978460+0000","last_clean":"2023-05-22T15:26:53.978460+0000","last_became_active":"2023-05-22T15:26:53.648415+0000","last_became_peered":"2023-05-22T15:26:53.648415+0000","last_unstale":"2023-05-22T15:26:53.978460+0000","last_undegraded":"2023-05-22T15:26:53.978460+0000","last_fullsized":"2023-05-22T15:26:53.978460+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_clean_scrub_stamp":"2023-05-22T15:26:52.568670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:33:26.787992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5,6],"acting":[2,0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.616451+0000","last_change":"2023-05-22T15:26:49.597065+0000","last_active":"2023-05-22T15:26:53.616451+0000","last_peered":"2023-05-22T15:26:53.616451+0000","last_clean":"2023-05-22T15:26:53.616451+0000","last_became_active":"2023-05-22T15:26:49.596334+0000","last_became_peered":"2023-05-22T15:26:49.596334+0000","last_unstale":"2023-05-22T15:26:53.616451+0000","last_undegraded":"2023-05-22T15:26:53.616451+0000","last_fullsized":"2023-05-22T15:26:53.616451+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:19:30.391062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"22'2","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.610463+0000","last_change":"2023-05-22T15:26:49.201296+0000","last_active":"2023-05-22T15:26:53.610463+0000","last_peered":"2023-05-22T15:26:53.610463+0000","last_clean":"2023-05-22T15:26:53.610463+0000","last_became_active":"2023-05-22T15:26:49.200707+0000","last_became_peered":"2023-05-22T15:26:49.200707+0000","last_unstale":"2023-05-22T15:26:53.610463+0000","last_undegraded":"2023-05-22T15:26:53.610463+0000","last_fullsized":"2023-05-22T15:26:53.610463+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_clean_scrub_stamp":"2023-05-22T15:26:48.127413+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:07:41.791660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.977970+0000","last_change":"2023-05-22T15:26:53.622351+0000","last_active":"2023-05-22T15:26:53.977970+0000","last_peered":"2023-05-22T15:26:53.977970+0000","last_clean":"2023-05-22T15:26:53.977970+0000","last_became_active":"2023-05-22T15:26:53.621754+0000","last_became_peered":"2023-05-22T15:26:53.621754+0000","last_unstale":"2023-05-22T15:26:53.977970+0000","last_undegraded":"2023-05-22T15:26:53.977970+0000","last_fullsized":"2023-05-22T15:26:53.977970+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_clean_scrub_stamp":"2023-05-22T15:26:52.568670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T16:31:40.104597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1,5],"acting":[2,0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.634315+0000","last_change":"2023-05-22T15:26:49.626518+0000","last_active":"2023-05-22T15:26:53.634315+0000","last_peered":"2023-05-22T15:26:53.634315+0000","last_clean":"2023-05-22T15:26:53.634315+0000","last_became_active":"2023-05-22T15:26:49.626026+0000","last_became_peered":"2023-05-22T15:26:49.626026+0000","last_unstale":"2023-05-22T15:26:53.634315+0000","last_undegraded":"2023-05-22T15:26:53.634315+0000","last_fullsized":"2023-05-22T15:26:53.634315+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:32:19.582542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"22'1","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.622234+0000","last_change":"2023-05-22T15:26:49.201252+0000","last_active":"2023-05-22T15:26:53.622234+0000","last_peered":"2023-05-22T15:26:53.622234+0000","last_clean":"2023-05-22T15:26:53.622234+0000","last_became_active":"2023-05-22T15:26:49.200748+0000","last_became_peered":"2023-05-22T15:26:49.200748+0000","last_unstale":"2023-05-22T15:26:53.622234+0000","last_undegraded":"2023-05-22T15:26:53.622234+0000","last_fullsized":"2023-05-22T15:26:53.622234+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_clean_scrub_stamp":"2023-05-22T15:26:48.127413+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:10:50.424396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.982126+0000","last_change":"2023-05-22T15:26:53.663379+0000","last_active":"2023-05-22T15:26:53.982126+0000","last_peered":"2023-05-22T15:26:53.982126+0000","last_clean":"2023-05-22T15:26:53.982126+0000","last_became_active":"2023-05-22T15:26:53.662170+0000","last_became_peered":"2023-05-22T15:26:53.662170+0000","last_unstale":"2023-05-22T15:26:53.982126+0000","last_undegraded":"2023-05-22T15:26:53.982126+0000","last_fullsized":"2023-05-22T15:26:53.982126+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_clean_scrub_stamp":"2023-05-22T15:26:52.568670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:38:53.647917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3,4],"acting":[7,6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-05-22T15:26:54.056358+0000","last_change":"2023-05-22T15:26:53.647770+0000","last_active":"2023-05-22T15:26:54.056358+0000","last_peered":"2023-05-22T15:26:54.056358+0000","last_clean":"2023-05-22T15:26:54.056358+0000","last_became_active":"2023-05-22T15:26:53.646585+0000","last_became_peered":"2023-05-22T15:26:53.646585+0000","last_unstale":"2023-05-22T15:26:54.056358+0000","last_undegraded":"2023-05-22T15:26:54.056358+0000","last_fullsized":"2023-05-22T15:26:54.056358+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_clean_scrub_stamp":"2023-05-22T15:26:52.568670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:48:44.211681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1,6],"acting":[5,3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"22'8","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.602514+0000","last_change":"2023-05-22T15:26:49.189894+0000","last_active":"2023-05-22T15:26:53.602514+0000","last_peered":"2023-05-22T15:26:53.602514+0000","last_clean":"2023-05-22T15:26:53.602514+0000","last_became_active":"2023-05-22T15:26:49.189156+0000","last_became_peered":"2023-05-22T15:26:49.189156+0000","last_unstale":"2023-05-22T15:26:53.602514+0000","last_undegraded":"2023-05-22T15:26:53.602514+0000","last_fullsized":"2023-05-22T15:26:53.602514+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_clean_scrub_stamp":"2023-05-22T15:26:48.127413+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:13:21.177657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.605587+0000","last_change":"2023-05-22T15:26:49.622540+0000","last_active":"2023-05-22T15:26:53.605587+0000","last_peered":"2023-05-22T15:26:53.605587+0000","last_clean":"2023-05-22T15:26:53.605587+0000","last_became_active":"2023-05-22T15:26:49.622187+0000","last_became_peered":"2023-05-22T15:26:49.622187+0000","last_unstale":"2023-05-22T15:26:53.605587+0000","last_undegraded":"2023-05-22T15:26:53.605587+0000","last_fullsized":"2023-05-22T15:26:53.605587+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T16:31:36.674451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.993571+0000","last_change":"2023-05-22T15:26:53.645178+0000","last_active":"2023-05-22T15:26:53.993571+0000","last_peered":"2023-05-22T15:26:53.993571+0000","last_clean":"2023-05-22T15:26:53.993571+0000","last_became_active":"2023-05-22T15:26:53.644770+0000","last_became_peered":"2023-05-22T15:26:53.644770+0000","last_unstale":"2023-05-22T15:26:53.993571+0000","last_undegraded":"2023-05-22T15:26:53.993571+0000","last_fullsized":"2023-05-22T15:26:53.993571+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_clean_scrub_stamp":"2023-05-22T15:26:52.568670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T15:46:20.544673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,0],"acting":[3,1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.628444+0000","last_change":"2023-05-22T15:26:49.191146+0000","last_active":"2023-05-22T15:26:53.628444+0000","last_peered":"2023-05-22T15:26:53.628444+0000","last_clean":"2023-05-22T15:26:53.628444+0000","last_became_active":"2023-05-22T15:26:49.190714+0000","last_became_peered":"2023-05-22T15:26:49.190714+0000","last_unstale":"2023-05-22T15:26:53.628444+0000","last_undegraded":"2023-05-22T15:26:53.628444+0000","last_fullsized":"2023-05-22T15:26:53.628444+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_clean_scrub_stamp":"2023-05-22T15:26:48.127413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:55:19.087548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.612090+0000","last_change":"2023-05-22T15:26:49.585916+0000","last_active":"2023-05-22T15:26:53.612090+0000","last_peered":"2023-05-22T15:26:53.612090+0000","last_clean":"2023-05-22T15:26:53.612090+0000","last_became_active":"2023-05-22T15:26:49.585261+0000","last_became_peered":"2023-05-22T15:26:49.585261+0000","last_unstale":"2023-05-22T15:26:53.612090+0000","last_undegraded":"2023-05-22T15:26:53.612090+0000","last_fullsized":"2023-05-22T15:26:53.612090+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:49:00.232177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.611499+0000","last_change":"2023-05-22T15:26:49.179340+0000","last_active":"2023-05-22T15:26:53.611499+0000","last_peered":"2023-05-22T15:26:53.611499+0000","last_clean":"2023-05-22T15:26:53.611499+0000","last_became_active":"2023-05-22T15:26:49.178875+0000","last_became_peered":"2023-05-22T15:26:49.178875+0000","last_unstale":"2023-05-22T15:26:53.611499+0000","last_undegraded":"2023-05-22T15:26:53.611499+0000","last_fullsized":"2023-05-22T15:26:53.611499+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_clean_scrub_stamp":"2023-05-22T15:26:48.127413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T16:16:39.277093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.634708+0000","last_change":"2023-05-22T15:26:49.623344+0000","last_active":"2023-05-22T15:26:53.634708+0000","last_peered":"2023-05-22T15:26:53.634708+0000","last_clean":"2023-05-22T15:26:53.634708+0000","last_became_active":"2023-05-22T15:26:49.622700+0000","last_became_peered":"2023-05-22T15:26:49.622700+0000","last_unstale":"2023-05-22T15:26:53.634708+0000","last_undegraded":"2023-05-22T15:26:53.634708+0000","last_fullsized":"2023-05-22T15:26:53.634708+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:22:10.117410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.993134+0000","last_change":"2023-05-22T15:26:53.646185+0000","last_active":"2023-05-22T15:26:53.993134+0000","last_peered":"2023-05-22T15:26:53.993134+0000","last_clean":"2023-05-22T15:26:53.993134+0000","last_became_active":"2023-05-22T15:26:53.645789+0000","last_became_peered":"2023-05-22T15:26:53.645789+0000","last_unstale":"2023-05-22T15:26:53.993134+0000","last_undegraded":"2023-05-22T15:26:53.993134+0000","last_fullsized":"2023-05-22T15:26:53.993134+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_clean_scrub_stamp":"2023-05-22T15:26:52.568670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:30:40.980509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2,1],"acting":[3,7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"22'2","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.608324+0000","last_change":"2023-05-22T15:26:49.201586+0000","last_active":"2023-05-22T15:26:53.608324+0000","last_peered":"2023-05-22T15:26:53.608324+0000","last_clean":"2023-05-22T15:26:53.608324+0000","last_became_active":"2023-05-22T15:26:49.201013+0000","last_became_peered":"2023-05-22T15:26:49.201013+0000","last_unstale":"2023-05-22T15:26:53.608324+0000","last_undegraded":"2023-05-22T15:26:53.608324+0000","last_fullsized":"2023-05-22T15:26:53.608324+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_clean_scrub_stamp":"2023-05-22T15:26:48.127413+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:47:17.588354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.607130+0000","last_change":"2023-05-22T15:26:49.577112+0000","last_active":"2023-05-22T15:26:53.607130+0000","last_peered":"2023-05-22T15:26:53.607130+0000","last_clean":"2023-05-22T15:26:53.607130+0000","last_became_active":"2023-05-22T15:26:49.576802+0000","last_became_peered":"2023-05-22T15:26:49.576802+0000","last_unstale":"2023-05-22T15:26:53.607130+0000","last_undegraded":"2023-05-22T15:26:53.607130+0000","last_fullsized":"2023-05-22T15:26:53.607130+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:52:48.123148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-05-22T15:26:54.056338+0000","last_change":"2023-05-22T15:26:53.655225+0000","last_active":"2023-05-22T15:26:54.056338+0000","last_peered":"2023-05-22T15:26:54.056338+0000","last_clean":"2023-05-22T15:26:54.056338+0000","last_became_active":"2023-05-22T15:26:53.648610+0000","last_became_peered":"2023-05-22T15:26:53.648610+0000","last_unstale":"2023-05-22T15:26:54.056338+0000","last_undegraded":"2023-05-22T15:26:54.056338+0000","last_fullsized":"2023-05-22T15:26:54.056338+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_clean_scrub_stamp":"2023-05-22T15:26:52.568670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T15:46:20.544673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3,0],"acting":[5,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.20","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.602276+0000","last_change":"2023-05-22T15:26:49.614417+0000","last_active":"2023-05-22T15:26:53.602276+0000","last_peered":"2023-05-22T15:26:53.602276+0000","last_clean":"2023-05-22T15:26:53.602276+0000","last_became_active":"2023-05-22T15:26:49.610439+0000","last_became_peered":"2023-05-22T15:26:49.610439+0000","last_unstale":"2023-05-22T15:26:53.602276+0000","last_undegraded":"2023-05-22T15:26:53.602276+0000","last_fullsized":"2023-05-22T15:26:53.602276+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:07:42.802772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.21","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.603531+0000","last_change":"2023-05-22T15:26:49.577132+0000","last_active":"2023-05-22T15:26:53.603531+0000","last_peered":"2023-05-22T15:26:53.603531+0000","last_clean":"2023-05-22T15:26:53.603531+0000","last_became_active":"2023-05-22T15:26:49.576859+0000","last_became_peered":"2023-05-22T15:26:49.576859+0000","last_unstale":"2023-05-22T15:26:53.603531+0000","last_undegraded":"2023-05-22T15:26:53.603531+0000","last_fullsized":"2023-05-22T15:26:53.603531+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:08:07.637726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.22","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.619920+0000","last_change":"2023-05-22T15:26:49.622334+0000","last_active":"2023-05-22T15:26:53.619920+0000","last_peered":"2023-05-22T15:26:53.619920+0000","last_clean":"2023-05-22T15:26:53.619920+0000","last_became_active":"2023-05-22T15:26:49.621648+0000","last_became_peered":"2023-05-22T15:26:49.621648+0000","last_unstale":"2023-05-22T15:26:53.619920+0000","last_undegraded":"2023-05-22T15:26:53.619920+0000","last_fullsized":"2023-05-22T15:26:53.619920+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:57:58.007680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.23","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.605756+0000","last_change":"2023-05-22T15:26:49.612753+0000","last_active":"2023-05-22T15:26:53.605756+0000","last_peered":"2023-05-22T15:26:53.605756+0000","last_clean":"2023-05-22T15:26:53.605756+0000","last_became_active":"2023-05-22T15:26:49.612495+0000","last_became_peered":"2023-05-22T15:26:49.612495+0000","last_unstale":"2023-05-22T15:26:53.605756+0000","last_undegraded":"2023-05-22T15:26:53.605756+0000","last_fullsized":"2023-05-22T15:26:53.605756+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:11:51.327926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.24","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.619231+0000","last_change":"2023-05-22T15:26:49.627768+0000","last_active":"2023-05-22T15:26:53.619231+0000","last_peered":"2023-05-22T15:26:53.619231+0000","last_clean":"2023-05-22T15:26:53.619231+0000","last_became_active":"2023-05-22T15:26:49.627340+0000","last_became_peered":"2023-05-22T15:26:49.627340+0000","last_unstale":"2023-05-22T15:26:53.619231+0000","last_undegraded":"2023-05-22T15:26:53.619231+0000","last_fullsized":"2023-05-22T15:26:53.619231+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:44:52.369716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.25","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.634508+0000","last_change":"2023-05-22T15:26:49.613627+0000","last_active":"2023-05-22T15:26:53.634508+0000","last_peered":"2023-05-22T15:26:53.634508+0000","last_clean":"2023-05-22T15:26:53.634508+0000","last_became_active":"2023-05-22T15:26:49.612997+0000","last_became_peered":"2023-05-22T15:26:49.612997+0000","last_unstale":"2023-05-22T15:26:53.634508+0000","last_undegraded":"2023-05-22T15:26:53.634508+0000","last_fullsized":"2023-05-22T15:26:53.634508+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:11:51.327926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.26","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.612444+0000","last_change":"2023-05-22T15:26:49.581772+0000","last_active":"2023-05-22T15:26:53.612444+0000","last_peered":"2023-05-22T15:26:53.612444+0000","last_clean":"2023-05-22T15:26:53.612444+0000","last_became_active":"2023-05-22T15:26:49.581493+0000","last_became_peered":"2023-05-22T15:26:49.581493+0000","last_unstale":"2023-05-22T15:26:53.612444+0000","last_undegraded":"2023-05-22T15:26:53.612444+0000","last_fullsized":"2023-05-22T15:26:53.612444+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:49:00.232177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.27","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.612724+0000","last_change":"2023-05-22T15:26:49.619433+0000","last_active":"2023-05-22T15:26:53.612724+0000","last_peered":"2023-05-22T15:26:53.612724+0000","last_clean":"2023-05-22T15:26:53.612724+0000","last_became_active":"2023-05-22T15:26:49.619107+0000","last_became_peered":"2023-05-22T15:26:49.619107+0000","last_unstale":"2023-05-22T15:26:53.612724+0000","last_undegraded":"2023-05-22T15:26:53.612724+0000","last_fullsized":"2023-05-22T15:26:53.612724+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:11:51.327926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.28","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.609649+0000","last_change":"2023-05-22T15:26:49.624224+0000","last_active":"2023-05-22T15:26:53.609649+0000","last_peered":"2023-05-22T15:26:53.609649+0000","last_clean":"2023-05-22T15:26:53.609649+0000","last_became_active":"2023-05-22T15:26:49.623023+0000","last_became_peered":"2023-05-22T15:26:49.623023+0000","last_unstale":"2023-05-22T15:26:53.609649+0000","last_undegraded":"2023-05-22T15:26:53.609649+0000","last_fullsized":"2023-05-22T15:26:53.609649+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:32:19.582542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.29","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.603353+0000","last_change":"2023-05-22T15:26:49.573154+0000","last_active":"2023-05-22T15:26:53.603353+0000","last_peered":"2023-05-22T15:26:53.603353+0000","last_clean":"2023-05-22T15:26:53.603353+0000","last_became_active":"2023-05-22T15:26:49.572881+0000","last_became_peered":"2023-05-22T15:26:49.572881+0000","last_unstale":"2023-05-22T15:26:53.603353+0000","last_undegraded":"2023-05-22T15:26:53.603353+0000","last_fullsized":"2023-05-22T15:26:53.603353+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:32:19.582542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2a","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.611407+0000","last_change":"2023-05-22T15:26:49.581918+0000","last_active":"2023-05-22T15:26:53.611407+0000","last_peered":"2023-05-22T15:26:53.611407+0000","last_clean":"2023-05-22T15:26:53.611407+0000","last_became_active":"2023-05-22T15:26:49.581462+0000","last_became_peered":"2023-05-22T15:26:49.581462+0000","last_unstale":"2023-05-22T15:26:53.611407+0000","last_undegraded":"2023-05-22T15:26:53.611407+0000","last_fullsized":"2023-05-22T15:26:53.611407+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:19:30.391062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2b","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.608076+0000","last_change":"2023-05-22T15:26:49.624711+0000","last_active":"2023-05-22T15:26:53.608076+0000","last_peered":"2023-05-22T15:26:53.608076+0000","last_clean":"2023-05-22T15:26:53.608076+0000","last_became_active":"2023-05-22T15:26:49.624180+0000","last_became_peered":"2023-05-22T15:26:49.624180+0000","last_unstale":"2023-05-22T15:26:53.608076+0000","last_undegraded":"2023-05-22T15:26:53.608076+0000","last_fullsized":"2023-05-22T15:26:53.608076+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:43:00.518715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2c","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.623768+0000","last_change":"2023-05-22T15:26:49.620689+0000","last_active":"2023-05-22T15:26:53.623768+0000","last_peered":"2023-05-22T15:26:53.623768+0000","last_clean":"2023-05-22T15:26:53.623768+0000","last_became_active":"2023-05-22T15:26:49.620183+0000","last_became_peered":"2023-05-22T15:26:49.620183+0000","last_unstale":"2023-05-22T15:26:53.623768+0000","last_undegraded":"2023-05-22T15:26:53.623768+0000","last_fullsized":"2023-05-22T15:26:53.623768+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:47:22.628767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2d","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.601734+0000","last_change":"2023-05-22T15:26:49.618556+0000","last_active":"2023-05-22T15:26:53.601734+0000","last_peered":"2023-05-22T15:26:53.601734+0000","last_clean":"2023-05-22T15:26:53.601734+0000","last_became_active":"2023-05-22T15:26:49.618215+0000","last_became_peered":"2023-05-22T15:26:49.618215+0000","last_unstale":"2023-05-22T15:26:53.601734+0000","last_undegraded":"2023-05-22T15:26:53.601734+0000","last_fullsized":"2023-05-22T15:26:53.601734+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:07:50.944971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2e","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.621273+0000","last_change":"2023-05-22T15:26:49.597573+0000","last_active":"2023-05-22T15:26:53.621273+0000","last_peered":"2023-05-22T15:26:53.621273+0000","last_clean":"2023-05-22T15:26:53.621273+0000","last_became_active":"2023-05-22T15:26:49.596607+0000","last_became_peered":"2023-05-22T15:26:49.596607+0000","last_unstale":"2023-05-22T15:26:53.621273+0000","last_undegraded":"2023-05-22T15:26:53.621273+0000","last_fullsized":"2023-05-22T15:26:53.621273+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:31:03.890085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2f","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.611892+0000","last_change":"2023-05-22T15:26:49.578237+0000","last_active":"2023-05-22T15:26:53.611892+0000","last_peered":"2023-05-22T15:26:53.611892+0000","last_clean":"2023-05-22T15:26:53.611892+0000","last_became_active":"2023-05-22T15:26:49.577927+0000","last_became_peered":"2023-05-22T15:26:49.577927+0000","last_unstale":"2023-05-22T15:26:53.611892+0000","last_undegraded":"2023-05-22T15:26:53.611892+0000","last_fullsized":"2023-05-22T15:26:53.611892+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:11:51.327926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.30","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.612647+0000","last_change":"2023-05-22T15:26:49.598449+0000","last_active":"2023-05-22T15:26:53.612647+0000","last_peered":"2023-05-22T15:26:53.612647+0000","last_clean":"2023-05-22T15:26:53.612647+0000","last_became_active":"2023-05-22T15:26:49.597652+0000","last_became_peered":"2023-05-22T15:26:49.597652+0000","last_unstale":"2023-05-22T15:26:53.612647+0000","last_undegraded":"2023-05-22T15:26:53.612647+0000","last_fullsized":"2023-05-22T15:26:53.612647+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:48:33.611833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.31","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.628230+0000","last_change":"2023-05-22T15:26:49.625943+0000","last_active":"2023-05-22T15:26:53.628230+0000","last_peered":"2023-05-22T15:26:53.628230+0000","last_clean":"2023-05-22T15:26:53.628230+0000","last_became_active":"2023-05-22T15:26:49.624306+0000","last_became_peered":"2023-05-22T15:26:49.624306+0000","last_unstale":"2023-05-22T15:26:53.628230+0000","last_undegraded":"2023-05-22T15:26:53.628230+0000","last_fullsized":"2023-05-22T15:26:53.628230+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:44:52.369716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.32","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.634913+0000","last_change":"2023-05-22T15:26:49.563010+0000","last_active":"2023-05-22T15:26:53.634913+0000","last_peered":"2023-05-22T15:26:53.634913+0000","last_clean":"2023-05-22T15:26:53.634913+0000","last_became_active":"2023-05-22T15:26:49.562210+0000","last_became_peered":"2023-05-22T15:26:49.562210+0000","last_unstale":"2023-05-22T15:26:53.634913+0000","last_undegraded":"2023-05-22T15:26:53.634913+0000","last_fullsized":"2023-05-22T15:26:53.634913+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:19:30.391062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.33","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.600262+0000","last_change":"2023-05-22T15:26:49.562260+0000","last_active":"2023-05-22T15:26:53.600262+0000","last_peered":"2023-05-22T15:26:53.600262+0000","last_clean":"2023-05-22T15:26:53.600262+0000","last_became_active":"2023-05-22T15:26:49.561965+0000","last_became_peered":"2023-05-22T15:26:49.561965+0000","last_unstale":"2023-05-22T15:26:53.600262+0000","last_undegraded":"2023-05-22T15:26:53.600262+0000","last_fullsized":"2023-05-22T15:26:53.600262+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:59:55.020950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.34","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.598383+0000","last_change":"2023-05-22T15:26:49.601428+0000","last_active":"2023-05-22T15:26:53.598383+0000","last_peered":"2023-05-22T15:26:53.598383+0000","last_clean":"2023-05-22T15:26:53.598383+0000","last_became_active":"2023-05-22T15:26:49.601157+0000","last_became_peered":"2023-05-22T15:26:49.601157+0000","last_unstale":"2023-05-22T15:26:53.598383+0000","last_undegraded":"2023-05-22T15:26:53.598383+0000","last_fullsized":"2023-05-22T15:26:53.598383+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:47:18.599465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.35","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.631671+0000","last_change":"2023-05-22T15:26:49.598414+0000","last_active":"2023-05-22T15:26:53.631671+0000","last_peered":"2023-05-22T15:26:53.631671+0000","last_clean":"2023-05-22T15:26:53.631671+0000","last_became_active":"2023-05-22T15:26:49.595631+0000","last_became_peered":"2023-05-22T15:26:49.595631+0000","last_unstale":"2023-05-22T15:26:53.631671+0000","last_undegraded":"2023-05-22T15:26:53.631671+0000","last_fullsized":"2023-05-22T15:26:53.631671+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:11:51.327926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.36","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.635151+0000","last_change":"2023-05-22T15:26:49.580957+0000","last_active":"2023-05-22T15:26:53.635151+0000","last_peered":"2023-05-22T15:26:53.635151+0000","last_clean":"2023-05-22T15:26:53.635151+0000","last_became_active":"2023-05-22T15:26:49.580172+0000","last_became_peered":"2023-05-22T15:26:49.580172+0000","last_unstale":"2023-05-22T15:26:53.635151+0000","last_undegraded":"2023-05-22T15:26:53.635151+0000","last_fullsized":"2023-05-22T15:26:53.635151+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:43:00.518715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.37","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.615519+0000","last_change":"2023-05-22T15:26:49.587437+0000","last_active":"2023-05-22T15:26:53.615519+0000","last_peered":"2023-05-22T15:26:53.615519+0000","last_clean":"2023-05-22T15:26:53.615519+0000","last_became_active":"2023-05-22T15:26:49.587019+0000","last_became_peered":"2023-05-22T15:26:49.587019+0000","last_unstale":"2023-05-22T15:26:53.615519+0000","last_undegraded":"2023-05-22T15:26:53.615519+0000","last_fullsized":"2023-05-22T15:26:53.615519+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:48:33.611833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.38","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.634076+0000","last_change":"2023-05-22T15:26:49.599720+0000","last_active":"2023-05-22T15:26:53.634076+0000","last_peered":"2023-05-22T15:26:53.634076+0000","last_clean":"2023-05-22T15:26:53.634076+0000","last_became_active":"2023-05-22T15:26:49.599079+0000","last_became_peered":"2023-05-22T15:26:49.599079+0000","last_unstale":"2023-05-22T15:26:53.634076+0000","last_undegraded":"2023-05-22T15:26:53.634076+0000","last_fullsized":"2023-05-22T15:26:53.634076+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:48:33.611833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.39","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.628016+0000","last_change":"2023-05-22T15:26:49.624934+0000","last_active":"2023-05-22T15:26:53.628016+0000","last_peered":"2023-05-22T15:26:53.628016+0000","last_clean":"2023-05-22T15:26:53.628016+0000","last_became_active":"2023-05-22T15:26:49.623557+0000","last_became_peered":"2023-05-22T15:26:49.623557+0000","last_unstale":"2023-05-22T15:26:53.628016+0000","last_undegraded":"2023-05-22T15:26:53.628016+0000","last_fullsized":"2023-05-22T15:26:53.628016+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:19:30.391062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3a","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.619318+0000","last_change":"2023-05-22T15:26:49.596704+0000","last_active":"2023-05-22T15:26:53.619318+0000","last_peered":"2023-05-22T15:26:53.619318+0000","last_clean":"2023-05-22T15:26:53.619318+0000","last_became_active":"2023-05-22T15:26:49.595941+0000","last_became_peered":"2023-05-22T15:26:49.595941+0000","last_unstale":"2023-05-22T15:26:53.619318+0000","last_undegraded":"2023-05-22T15:26:53.619318+0000","last_fullsized":"2023-05-22T15:26:53.619318+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T03:17:59.774769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3b","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.611996+0000","last_change":"2023-05-22T15:26:49.585396+0000","last_active":"2023-05-22T15:26:53.611996+0000","last_peered":"2023-05-22T15:26:53.611996+0000","last_clean":"2023-05-22T15:26:53.611996+0000","last_became_active":"2023-05-22T15:26:49.585110+0000","last_became_peered":"2023-05-22T15:26:49.585110+0000","last_unstale":"2023-05-22T15:26:53.611996+0000","last_undegraded":"2023-05-22T15:26:53.611996+0000","last_fullsized":"2023-05-22T15:26:53.611996+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:22:10.117410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3c","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.602523+0000","last_change":"2023-05-22T15:26:49.607125+0000","last_active":"2023-05-22T15:26:53.602523+0000","last_peered":"2023-05-22T15:26:53.602523+0000","last_clean":"2023-05-22T15:26:53.602523+0000","last_became_active":"2023-05-22T15:26:49.606510+0000","last_became_peered":"2023-05-22T15:26:49.606510+0000","last_unstale":"2023-05-22T15:26:53.602523+0000","last_undegraded":"2023-05-22T15:26:53.602523+0000","last_fullsized":"2023-05-22T15:26:53.602523+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:47:18.599465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3d","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.631452+0000","last_change":"2023-05-22T15:26:49.600011+0000","last_active":"2023-05-22T15:26:53.631452+0000","last_peered":"2023-05-22T15:26:53.631452+0000","last_clean":"2023-05-22T15:26:53.631452+0000","last_became_active":"2023-05-22T15:26:49.596414+0000","last_became_peered":"2023-05-22T15:26:49.596414+0000","last_unstale":"2023-05-22T15:26:53.631452+0000","last_undegraded":"2023-05-22T15:26:53.631452+0000","last_fullsized":"2023-05-22T15:26:53.631452+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T16:31:36.674451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3e","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.634607+0000","last_change":"2023-05-22T15:26:49.580509+0000","last_active":"2023-05-22T15:26:53.634607+0000","last_peered":"2023-05-22T15:26:53.634607+0000","last_clean":"2023-05-22T15:26:53.634607+0000","last_became_active":"2023-05-22T15:26:49.579662+0000","last_became_peered":"2023-05-22T15:26:49.579662+0000","last_unstale":"2023-05-22T15:26:53.634607+0000","last_undegraded":"2023-05-22T15:26:53.634607+0000","last_fullsized":"2023-05-22T15:26:53.634607+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:44:52.369716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3f","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.599109+0000","last_change":"2023-05-22T15:26:49.561088+0000","last_active":"2023-05-22T15:26:53.599109+0000","last_peered":"2023-05-22T15:26:53.599109+0000","last_clean":"2023-05-22T15:26:53.599109+0000","last_became_active":"2023-05-22T15:26:49.560859+0000","last_became_peered":"2023-05-22T15:26:49.560859+0000","last_unstale":"2023-05-22T15:26:53.599109+0000","last_undegraded":"2023-05-22T15:26:53.599109+0000","last_fullsized":"2023-05-22T15:26:53.599109+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:53:35.813397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":4386,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":47,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":8772,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":41,"ondisk_log_size":41,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574855,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9284,"kb_used_data":892,"kb_used_omap":0,"kb_used_meta":8384,"kb_avail":94362556,"statfs":{"total":96636764160,"available":96627257344,"internally_reserved":0,"allocated":913408,"data_stored":714089,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8585216},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97999999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71499999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93100000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1779999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2310000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.149}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3999999999999999}]}]},{"osd":6,"up_from":13,"seq":55834574856,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8100,"kb_used_data":284,"kb_used_omap":0,"kb_used_meta":7808,"kb_avail":94363740,"statfs":{"total":96636764160,"available":96628469760,"internally_reserved":0,"allocated":290816,"data_stored":119901,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7995392},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69899999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91200000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64500000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66500000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84499999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68600000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79600000000000004}]}]},{"osd":5,"up_from":13,"seq":55834574856,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8172,"kb_used_data":292,"kb_used_omap":0,"kb_used_meta":7872,"kb_avail":94363668,"statfs":{"total":96636764160,"available":96628396032,"internally_reserved":0,"allocated":299008,"data_stored":120537,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8060928},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.159}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90900000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77500000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71099999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1279999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94199999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2370000000000001}]}]},{"osd":4,"up_from":13,"seq":55834574854,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8248,"kb_used_data":304,"kb_used_omap":0,"kb_used_meta":7936,"kb_avail":94363592,"statfs":{"total":96636764160,"available":96628318208,"internally_reserved":0,"allocated":311296,"data_stored":123561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8126464},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72299999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78000000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84999999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1040000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1879999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1359999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1699999999999999}]}]},{"osd":3,"up_from":13,"seq":55834574854,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8168,"kb_used_data":288,"kb_used_omap":0,"kb_used_meta":7872,"kb_avail":94363672,"statfs":{"total":96636764160,"available":96628400128,"internally_reserved":0,"allocated":294912,"data_stored":119991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8060928},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34100000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38400000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46600000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85499999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92100000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71299999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79100000000000004}]}]},{"osd":2,"up_from":13,"seq":55834574855,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":12196,"kb_used_data":284,"kb_used_omap":0,"kb_used_meta":11904,"kb_avail":94359644,"statfs":{"total":96636764160,"available":96624275456,"internally_reserved":0,"allocated":290816,"data_stored":119901,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":12189696},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.503}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40400000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61199999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79900000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78900000000000003}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72799999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86699999999999999}]}]},{"osd":1,"up_from":13,"seq":55834574855,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8164,"kb_used_data":284,"kb_used_omap":0,"kb_used_meta":7872,"kb_avail":94363676,"statfs":{"total":96636764160,"available":96628404224,"internally_reserved":0,"allocated":290816,"data_stored":119901,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8060928},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66600000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57099999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77700000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79200000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.034}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0129999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0760000000000001}]}]},{"osd":0,"up_from":13,"seq":55834574856,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":13356,"kb_used_data":868,"kb_used_omap":0,"kb_used_meta":12480,"kb_avail":94358484,"statfs":{"total":96636764160,"available":96623087616,"internally_reserved":0,"allocated":888832,"data_stored":710835,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":12779520},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1839999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.23}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2410000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2969999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.365}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.264}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.28}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":3660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":3820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-05-22T15:27:07.432 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-05-22T15:27:07.433 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2023-05-22T15:27:07.433 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2023-05-22T15:27:07.433 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2023-05-22T15:27:07.434 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2023-05-22T15:27:07.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.578+0000 7f2c75aa0640 1 Processor -- start 2023-05-22T15:27:07.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.578+0000 7f2c75aa0640 1 -- start start 2023-05-22T15:27:07.590 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.578+0000 7f2c75aa0640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f2c700ccab0 0x7f2c7015fe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:07.590 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.578+0000 7f2c75aa0640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c700ccef0 con 0x7f2c700b7dd0 2023-05-22T15:27:07.590 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.578+0000 7f2c75aa0640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c700cd170 con 0x7f2c700d0410 2023-05-22T15:27:07.591 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.578+0000 7f2c75aa0640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c700d3360 con 0x7f2c700ccab0 2023-05-22T15:27:07.591 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.578+0000 7f2c6f7fe640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f2c700b7dd0 0x7f2c7015f700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:51036/0 (socket says 172.21.15.16:51036) 2023-05-22T15:27:07.591 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.578+0000 7f2c6f7fe640 1 -- 172.21.15.16:0/2865040579 learned_addr learned my addr 172.21.15.16:0/2865040579 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:27:07.592 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.578+0000 7f2c6effd640 1 --2- 172.21.15.16:0/2865040579 >> v2:172.21.15.62:3300/0 conn(0x7f2c700ccab0 0x7f2c7015fe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:07.592 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.578+0000 7f2c6e7fc640 1 -- 172.21.15.16:0/2865040579 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3367829330 0 0) 0x7f2c700ccef0 con 0x7f2c700b7dd0 2023-05-22T15:27:07.592 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.578+0000 7f2c6e7fc640 1 -- 172.21.15.16:0/2865040579 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2c58003530 con 0x7f2c700b7dd0 2023-05-22T15:27:07.592 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.578+0000 7f2c6e7fc640 1 -- 172.21.15.16:0/2865040579 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 984306476 0 0) 0x7f2c58003530 con 0x7f2c700b7dd0 2023-05-22T15:27:07.593 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.578+0000 7f2c6e7fc640 1 -- 172.21.15.16:0/2865040579 >> v2:172.21.15.62:3300/0 conn(0x7f2c700ccab0 msgr2=0x7f2c7015fe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:07.593 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.578+0000 7f2c6e7fc640 1 --2- 172.21.15.16:0/2865040579 >> v2:172.21.15.62:3300/0 conn(0x7f2c700ccab0 0x7f2c7015fe10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:07.594 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.578+0000 7f2c6e7fc640 1 -- 172.21.15.16:0/2865040579 >> v1:172.21.15.62:6789/0 conn(0x7f2c700d0410 legacy=0x7f2c700d0810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:07.594 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.578+0000 7f2c6e7fc640 1 -- 172.21.15.16:0/2865040579 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c700d2ac0 con 0x7f2c700b7dd0 2023-05-22T15:27:07.594 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.578+0000 7f2c6e7fc640 1 -- 172.21.15.16:0/2865040579 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7f2c60001680 con 0x7f2c700b7dd0 2023-05-22T15:27:07.594 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.578+0000 7f2c6e7fc640 1 -- 172.21.15.16:0/2865040579 >> v1:172.21.15.16:6789/0 conn(0x7f2c700b7dd0 legacy=0x7f2c7015f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:07.595 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.578+0000 7f2c6e7fc640 1 --2- 172.21.15.16:0/2865040579 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2c58003980 0x7f2c58003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:07.596 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.578+0000 7f2c6e7fc640 1 --2- 172.21.15.16:0/2865040579 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2c580047d0 0x7f2c58004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:07.596 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.578+0000 7f2c6e7fc640 1 --2- 172.21.15.16:0/2865040579 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2c58005180 0x7f2c5800b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:07.596 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.578+0000 7f2c6e7fc640 1 -- 172.21.15.16:0/2865040579 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c700d3360 con 0x7f2c58005180 2023-05-22T15:27:07.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.578+0000 7f2c6e7fc640 1 -- 172.21.15.16:0/2865040579 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c5800c000 con 0x7f2c580047d0 2023-05-22T15:27:07.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.578+0000 7f2c6e7fc640 1 -- 172.21.15.16:0/2865040579 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c5800c1c0 con 0x7f2c58003980 2023-05-22T15:27:07.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.578+0000 7f2c6ffff640 1 --2- 172.21.15.16:0/2865040579 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2c58005180 0x7f2c5800b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:07.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.578+0000 7f2c6effd640 1 --2- 172.21.15.16:0/2865040579 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2c580047d0 0x7f2c58004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:07.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.578+0000 7f2c6f7fe640 1 --2- 172.21.15.16:0/2865040579 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2c58003980 0x7f2c58003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:07.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.578+0000 7f2c6ffff640 1 -- 172.21.15.16:0/2865040579 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2c58003980 msgr2=0x7f2c58003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:07.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.578+0000 7f2c6ffff640 1 --2- 172.21.15.16:0/2865040579 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2c58003980 0x7f2c58003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:07.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.578+0000 7f2c6ffff640 1 -- 172.21.15.16:0/2865040579 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2c580047d0 msgr2=0x7f2c58004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:07.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.578+0000 7f2c6ffff640 1 --2- 172.21.15.16:0/2865040579 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2c580047d0 0x7f2c58004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:07.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.578+0000 7f2c6ffff640 1 -- 172.21.15.16:0/2865040579 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c58003530 con 0x7f2c58005180 2023-05-22T15:27:07.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.578+0000 7f2c6f7fe640 1 --2- 172.21.15.16:0/2865040579 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2c58003980 0x7f2c58003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:27:07.600 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.578+0000 7f2c6effd640 1 --2- 172.21.15.16:0/2865040579 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2c580047d0 0x7f2c58004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:27:07.604 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.582+0000 7f2c6ffff640 1 -- 172.21.15.16:0/2865040579 reap_dead start 2023-05-22T15:27:07.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.582+0000 7f2c6ffff640 1 --2- 172.21.15.16:0/2865040579 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2c58005180 0x7f2c5800b990 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f2c68001e70 tx=0x7f2c6801c760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3794ccb1b5c2fdc4 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:07.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.582+0000 7f2c6e7fc640 1 -- 172.21.15.16:0/2865040579 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c68003b80 con 0x7f2c58005180 2023-05-22T15:27:07.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.582+0000 7f2c6e7fc640 1 -- 172.21.15.16:0/2865040579 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c68003d00 con 0x7f2c58005180 2023-05-22T15:27:07.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.582+0000 7f2c6e7fc640 1 -- 172.21.15.16:0/2865040579 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c68005260 con 0x7f2c58005180 2023-05-22T15:27:07.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.582+0000 7f2c75aa0640 1 -- 172.21.15.16:0/2865040579 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2c58005180 msgr2=0x7f2c5800b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:07.606 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.582+0000 7f2c75aa0640 1 --2- 172.21.15.16:0/2865040579 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2c58005180 0x7f2c5800b990 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f2c68001e70 tx=0x7f2c6801c760 comp rx=0 tx=0).stop 2023-05-22T15:27:07.606 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.582+0000 7f2c75aa0640 1 -- 172.21.15.16:0/2865040579 shutdown_connections 2023-05-22T15:27:07.606 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.582+0000 7f2c75aa0640 1 --2- 172.21.15.16:0/2865040579 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2c58005180 0x7f2c5800b990 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:07.606 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.582+0000 7f2c75aa0640 1 -- 172.21.15.16:0/2865040579 >> 172.21.15.16:0/2865040579 conn(0x7f2c700655c0 msgr2=0x7f2c70070e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:07.606 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.582+0000 7f2c75aa0640 1 -- 172.21.15.16:0/2865040579 shutdown_connections 2023-05-22T15:27:07.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.582+0000 7f2c75aa0640 1 -- 172.21.15.16:0/2865040579 wait complete. 2023-05-22T15:27:07.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.582+0000 7f2c75aa0640 1 Processor -- start 2023-05-22T15:27:07.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.582+0000 7f2c75aa0640 1 -- start start 2023-05-22T15:27:07.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.582+0000 7f2c75aa0640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2c58005180 0x7f2c700cac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:07.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.582+0000 7f2c75aa0640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2c700c7990 0x7f2c700c9e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:07.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.582+0000 7f2c75aa0640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2c700b0160 0x7f2c700b05d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:07.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.582+0000 7f2c75aa0640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c700b0b10 con 0x7f2c58005180 2023-05-22T15:27:07.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.582+0000 7f2c75aa0640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c700c8250 con 0x7f2c700c7990 2023-05-22T15:27:07.609 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.582+0000 7f2c75aa0640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c700c8530 con 0x7f2c700b0160 2023-05-22T15:27:07.609 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.582+0000 7f2c6f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2c58005180 0x7f2c700cac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:07.609 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.582+0000 7f2c6f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2c58005180 0x7f2c700cac30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57724/0 (socket says 172.21.15.16:57724) 2023-05-22T15:27:07.610 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.582+0000 7f2c6f7fe640 1 -- 172.21.15.16:0/3141896827 learned_addr learned my addr 172.21.15.16:0/3141896827 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:07.610 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.582+0000 7f2c6effd640 1 --2- 172.21.15.16:0/3141896827 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2c700c7990 0x7f2c700c9e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:07.610 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.582+0000 7f2c6ffff640 1 --2- 172.21.15.16:0/3141896827 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2c700b0160 0x7f2c700b05d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:07.610 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.582+0000 7f2c6f7fe640 1 -- 172.21.15.16:0/3141896827 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2c700b0160 msgr2=0x7f2c700b05d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:07.611 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.582+0000 7f2c6f7fe640 1 --2- 172.21.15.16:0/3141896827 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2c700b0160 0x7f2c700b05d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:07.611 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.582+0000 7f2c6f7fe640 1 -- 172.21.15.16:0/3141896827 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2c700c7990 msgr2=0x7f2c700c9e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:07.611 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.582+0000 7f2c6f7fe640 1 --2- 172.21.15.16:0/3141896827 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2c700c7990 0x7f2c700c9e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:07.611 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.582+0000 7f2c6f7fe640 1 -- 172.21.15.16:0/3141896827 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c700d3080 con 0x7f2c58005180 2023-05-22T15:27:07.611 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.582+0000 7f2c6ffff640 1 --2- 172.21.15.16:0/3141896827 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2c700b0160 0x7f2c700b05d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:27:07.612 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.582+0000 7f2c6effd640 1 --2- 172.21.15.16:0/3141896827 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2c700c7990 0x7f2c700c9e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:27:07.612 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.586+0000 7f2c6f7fe640 1 --2- 172.21.15.16:0/3141896827 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2c58005180 0x7f2c700cac30 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f2c6000d700 tx=0x7f2c6000dbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:07.612 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.586+0000 7f2c57fff640 1 -- 172.21.15.16:0/3141896827 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c600074e0 con 0x7f2c58005180 2023-05-22T15:27:07.612 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.586+0000 7f2c57fff640 1 -- 172.21.15.16:0/3141896827 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c60007640 con 0x7f2c58005180 2023-05-22T15:27:07.612 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.586+0000 7f2c75aa0640 1 -- 172.21.15.16:0/3141896827 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c700d2da0 con 0x7f2c58005180 2023-05-22T15:27:07.612 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.586+0000 7f2c57fff640 1 -- 172.21.15.16:0/3141896827 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c60004a30 con 0x7f2c58005180 2023-05-22T15:27:07.613 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.586+0000 7f2c75aa0640 1 -- 172.21.15.16:0/3141896827 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c7015f700 con 0x7f2c58005180 2023-05-22T15:27:07.613 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.586+0000 7f2c75aa0640 1 -- 172.21.15.16:0/3141896827 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c7004dbe0 con 0x7f2c58005180 2023-05-22T15:27:07.613 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.586+0000 7f2c57fff640 1 -- 172.21.15.16:0/3141896827 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7f2c60004b90 con 0x7f2c58005180 2023-05-22T15:27:07.613 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.586+0000 7f2c57fff640 1 --2- 172.21.15.16:0/3141896827 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f2c5c0653b0 0x7f2c5c067860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:07.613 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.586+0000 7f2c57fff640 1 -- 172.21.15.16:0/3141896827 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f2c600823f0 con 0x7f2c58005180 2023-05-22T15:27:07.614 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.586+0000 7f2c6effd640 1 --2- 172.21.15.16:0/3141896827 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f2c5c0653b0 0x7f2c5c067860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:07.614 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.586+0000 7f2c6effd640 1 --2- 172.21.15.16:0/3141896827 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f2c5c0653b0 0x7f2c5c067860 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f2c70067bc0 tx=0x7f2c64008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:07.614 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.590+0000 7f2c57fff640 1 -- 172.21.15.16:0/3141896827 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f2c6004fd80 con 0x7f2c58005180 2023-05-22T15:27:07.783 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.774+0000 7f2c75aa0640 1 -- 172.21.15.16:0/3141896827 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7f2c700710a0 con 0x7f2c58005180 2023-05-22T15:27:07.788 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.778+0000 7f2c57fff640 1 -- 172.21.15.16:0/3141896827 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) v1 ==== 74+0+162140 (secure 0 0 0) 0x7f2c600533a0 con 0x7f2c58005180 2023-05-22T15:27:07.790 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:27:07.791 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.782+0000 7f2c75aa0640 1 -- 172.21.15.16:0/3141896827 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f2c5c0653b0 msgr2=0x7f2c5c067860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:07.791 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.782+0000 7f2c75aa0640 1 --2- 172.21.15.16:0/3141896827 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f2c5c0653b0 0x7f2c5c067860 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f2c70067bc0 tx=0x7f2c64008040 comp rx=0 tx=0).stop 2023-05-22T15:27:07.791 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.782+0000 7f2c75aa0640 1 -- 172.21.15.16:0/3141896827 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2c58005180 msgr2=0x7f2c700cac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:07.791 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.782+0000 7f2c75aa0640 1 --2- 172.21.15.16:0/3141896827 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2c58005180 0x7f2c700cac30 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f2c6000d700 tx=0x7f2c6000dbc0 comp rx=0 tx=0).stop 2023-05-22T15:27:07.792 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.782+0000 7f2c75aa0640 1 -- 172.21.15.16:0/3141896827 shutdown_connections 2023-05-22T15:27:07.792 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.782+0000 7f2c75aa0640 1 --2- 172.21.15.16:0/3141896827 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f2c5c0653b0 0x7f2c5c067860 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:07.792 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.782+0000 7f2c75aa0640 1 --2- 172.21.15.16:0/3141896827 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2c700b0160 0x7f2c700b05d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:07.792 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.782+0000 7f2c75aa0640 1 --2- 172.21.15.16:0/3141896827 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2c700c7990 0x7f2c700c9e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:07.792 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.782+0000 7f2c75aa0640 1 --2- 172.21.15.16:0/3141896827 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2c58005180 0x7f2c700cac30 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:07.793 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.782+0000 7f2c75aa0640 1 -- 172.21.15.16:0/3141896827 >> 172.21.15.16:0/3141896827 conn(0x7f2c700655c0 msgr2=0x7f2c700cf230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:07.793 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.782+0000 7f2c75aa0640 1 -- 172.21.15.16:0/3141896827 shutdown_connections 2023-05-22T15:27:07.793 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.782+0000 7f2c75aa0640 1 -- 172.21.15.16:0/3141896827 wait complete. 2023-05-22T15:27:07.807 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":5,"active_gid":4101,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":36921},{"type":"v1","addr":"172.21.15.16:6833","nonce":36921}]},"active_addr":"172.21.15.16:6833/36921","active_change":"2023-05-22T15:26:29.931782+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[{"gid":4114,"name":"x","mgr_features":4540138322906710015,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"alertmanager_web_password":{"name":"alertmanager_web_password","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Alertmanager web password","long_desc":"","tags":[],"see_also":[]},"alertmanager_web_user":{"name":"alertmanager_web_user","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Alertmanager web user","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"prometheus_web_password":{"name":"prometheus_web_password","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Prometheus web password","long_desc":"","tags":[],"see_also":[]},"prometheus_web_user":{"name":"prometheus_web_user","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Prometheus web user","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"alertmanager_web_password":{"name":"alertmanager_web_password","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Alertmanager web password","long_desc":"","tags":[],"see_also":[]},"alertmanager_web_user":{"name":"alertmanager_web_user","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Alertmanager web user","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"prometheus_web_password":{"name":"prometheus_web_password","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Prometheus web password","long_desc":"","tags":[],"see_also":[]},"prometheus_web_user":{"name":"prometheus_web_user","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Prometheus web user","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.16:0","nonce":901306510}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.16:0","nonce":3979017463}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.16:0","nonce":4222124422}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.16:0","nonce":2755009771}]}]} 2023-05-22T15:27:07.811 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2023-05-22T15:27:07.812 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2023-05-22T15:27:07.812 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-05-22T15:27:07.922 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.910+0000 7fb1cd590640 1 Processor -- start 2023-05-22T15:27:07.922 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.910+0000 7fb1cd590640 1 -- start start 2023-05-22T15:27:07.923 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.914+0000 7fb1cd590640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb1c815c920 0x7fb1c80d0030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:07.923 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.914+0000 7fb1cd590640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1c815c490 con 0x7fb1c815cd70 2023-05-22T15:27:07.924 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.914+0000 7fb1cd590640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1c815c140 con 0x7fb1c815fc50 2023-05-22T15:27:07.924 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.914+0000 7fb1cd590640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1c80ccb50 con 0x7fb1c815c920 2023-05-22T15:27:07.924 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.914+0000 7fb1c77fe640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fb1c815cd70 0x7fb1c80ce790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:51038/0 (socket says 172.21.15.16:51038) 2023-05-22T15:27:07.924 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.914+0000 7fb1c77fe640 1 -- 172.21.15.16:0/126223950 learned_addr learned my addr 172.21.15.16:0/126223950 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:27:07.924 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.914+0000 7fb1c67fc640 1 --2- 172.21.15.16:0/126223950 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb1c815c920 0x7fb1c80d0030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:07.925 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.914+0000 7fb1c5ffb640 1 -- 172.21.15.16:0/126223950 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2445038365 0 0) 0x7fb1c815c490 con 0x7fb1c815cd70 2023-05-22T15:27:07.925 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.914+0000 7fb1c5ffb640 1 -- 172.21.15.16:0/126223950 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb1b0003530 con 0x7fb1c815cd70 2023-05-22T15:27:07.925 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.914+0000 7fb1c5ffb640 1 -- 172.21.15.16:0/126223950 <== mon.1 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3021338220 0 0) 0x7fb1c815c140 con 0x7fb1c815fc50 2023-05-22T15:27:07.925 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.914+0000 7fb1c5ffb640 1 -- 172.21.15.16:0/126223950 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb1c815c490 con 0x7fb1c815fc50 2023-05-22T15:27:07.925 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.914+0000 7fb1c5ffb640 1 -- 172.21.15.16:0/126223950 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3297049473 0 0) 0x7fb1b0003530 con 0x7fb1c815cd70 2023-05-22T15:27:07.925 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.914+0000 7fb1c67fc640 1 -- 172.21.15.16:0/126223950 >> v1:172.21.15.62:6789/0 conn(0x7fb1c815fc50 legacy=0x7fb1c8160020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:07.925 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.914+0000 7fb1c67fc640 1 -- 172.21.15.16:0/126223950 >> v1:172.21.15.16:6789/0 conn(0x7fb1c815cd70 legacy=0x7fb1c80ce790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:07.926 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.914+0000 7fb1c67fc640 1 -- 172.21.15.16:0/126223950 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1c80cce30 con 0x7fb1c815c920 2023-05-22T15:27:07.927 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.918+0000 7fb1c67fc640 1 --2- 172.21.15.16:0/126223950 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb1c815c920 0x7fb1c80d0030 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fb1bc0097f0 tx=0x7fb1bc01c3e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=781ff80395293ef9 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:07.927 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.918+0000 7fb1c5ffb640 1 -- 172.21.15.16:0/126223950 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1c0001a00 con 0x7fb1c815c920 2023-05-22T15:27:07.927 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.918+0000 7fb1c5ffb640 1 -- 172.21.15.16:0/126223950 <== mon.2 v2:172.21.15.62:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1bc0a7050 con 0x7fb1c815c920 2023-05-22T15:27:07.927 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.918+0000 7fb1c5ffb640 1 -- 172.21.15.16:0/126223950 <== mon.2 v2:172.21.15.62:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1bc0ab620 con 0x7fb1c815c920 2023-05-22T15:27:07.927 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.918+0000 7fb1cd590640 1 -- 172.21.15.16:0/126223950 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb1c815c920 msgr2=0x7fb1c80d0030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:07.928 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.918+0000 7fb1cd590640 1 --2- 172.21.15.16:0/126223950 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb1c815c920 0x7fb1c80d0030 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fb1bc0097f0 tx=0x7fb1bc01c3e0 comp rx=0 tx=0).stop 2023-05-22T15:27:07.929 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.918+0000 7fb1cd590640 1 -- 172.21.15.16:0/126223950 shutdown_connections 2023-05-22T15:27:07.929 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.918+0000 7fb1cd590640 1 --2- 172.21.15.16:0/126223950 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb1c815c920 0x7fb1c80d0030 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:07.929 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.918+0000 7fb1cd590640 1 -- 172.21.15.16:0/126223950 >> 172.21.15.16:0/126223950 conn(0x7fb1c80657f0 msgr2=0x7fb1c8070e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:07.930 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.918+0000 7fb1cd590640 1 -- 172.21.15.16:0/126223950 shutdown_connections 2023-05-22T15:27:07.930 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.918+0000 7fb1cd590640 1 -- 172.21.15.16:0/126223950 wait complete. 2023-05-22T15:27:07.933 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.918+0000 7fb1cd590640 1 Processor -- start 2023-05-22T15:27:07.934 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.918+0000 7fb1cd590640 1 -- start start 2023-05-22T15:27:07.936 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.918+0000 7fb1cd590640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb1c815c920 0x7fb1c80b4880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:07.936 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.918+0000 7fb1cd590640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb1c815cd70 0x7fb1c8155430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:07.936 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.918+0000 7fb1cd590640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb1c815fc50 0x7fb1c8153c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:07.937 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.918+0000 7fb1cd590640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1c80b29b0 con 0x7fb1c815c920 2023-05-22T15:27:07.937 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.918+0000 7fb1cd590640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1c8154190 con 0x7fb1c815fc50 2023-05-22T15:27:07.937 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.918+0000 7fb1cd590640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1c80b26f0 con 0x7fb1c815cd70 2023-05-22T15:27:07.937 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.918+0000 7fb1c6ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb1c815c920 0x7fb1c80b4880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:07.937 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.918+0000 7fb1c6ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb1c815c920 0x7fb1c80b4880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57738/0 (socket says 172.21.15.16:57738) 2023-05-22T15:27:07.937 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.918+0000 7fb1c6ffd640 1 -- 172.21.15.16:0/3549520850 learned_addr learned my addr 172.21.15.16:0/3549520850 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:07.937 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.918+0000 7fb1c77fe640 1 --2- 172.21.15.16:0/3549520850 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb1c815fc50 0x7fb1c8153c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:07.938 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.918+0000 7fb1c67fc640 1 --2- 172.21.15.16:0/3549520850 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb1c815cd70 0x7fb1c8155430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:07.938 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.918+0000 7fb1c6ffd640 1 -- 172.21.15.16:0/3549520850 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb1c815cd70 msgr2=0x7fb1c8155430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:07.938 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.918+0000 7fb1c6ffd640 1 --2- 172.21.15.16:0/3549520850 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb1c815cd70 0x7fb1c8155430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:07.938 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.918+0000 7fb1c6ffd640 1 -- 172.21.15.16:0/3549520850 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb1c815fc50 msgr2=0x7fb1c8153c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:07.938 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.918+0000 7fb1c6ffd640 1 --2- 172.21.15.16:0/3549520850 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb1c815fc50 0x7fb1c8153c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:07.938 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.918+0000 7fb1c6ffd640 1 -- 172.21.15.16:0/3549520850 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1bc009420 con 0x7fb1c815c920 2023-05-22T15:27:07.938 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.918+0000 7fb1c67fc640 1 --2- 172.21.15.16:0/3549520850 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb1c815cd70 0x7fb1c8155430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T15:27:07.939 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.918+0000 7fb1c77fe640 1 --2- 172.21.15.16:0/3549520850 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb1c815fc50 0x7fb1c8153c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T15:27:07.939 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.918+0000 7fb1c6ffd640 1 --2- 172.21.15.16:0/3549520850 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb1c815c920 0x7fb1c80b4880 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fb1b800c4c0 tx=0x7fb1b800c980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:07.939 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.918+0000 7fb1af7fe640 1 -- 172.21.15.16:0/3549520850 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1b80052d0 con 0x7fb1c815c920 2023-05-22T15:27:07.939 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.918+0000 7fb1af7fe640 1 -- 172.21.15.16:0/3549520850 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1b8005430 con 0x7fb1c815c920 2023-05-22T15:27:07.939 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.918+0000 7fb1cd590640 1 -- 172.21.15.16:0/3549520850 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1c8169c70 con 0x7fb1c815c920 2023-05-22T15:27:07.939 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.918+0000 7fb1af7fe640 1 -- 172.21.15.16:0/3549520850 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1b8012870 con 0x7fb1c815c920 2023-05-22T15:27:07.939 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.918+0000 7fb1cd590640 1 -- 172.21.15.16:0/3549520850 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1c8169ae0 con 0x7fb1c815c920 2023-05-22T15:27:07.940 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.922+0000 7fb1cd590640 1 -- 172.21.15.16:0/3549520850 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1c8169e00 con 0x7fb1c815c920 2023-05-22T15:27:07.940 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.922+0000 7fb1af7fe640 1 -- 172.21.15.16:0/3549520850 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7fb1b80129d0 con 0x7fb1c815c920 2023-05-22T15:27:07.940 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.922+0000 7fb1af7fe640 1 --2- 172.21.15.16:0/3549520850 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fb1b4065380 0x7fb1b4067830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:07.940 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.922+0000 7fb1af7fe640 1 -- 172.21.15.16:0/3549520850 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6813+0+0 (secure 0 0 0) 0x7fb1b80832d0 con 0x7fb1c815c920 2023-05-22T15:27:07.940 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.922+0000 7fb1c67fc640 1 --2- 172.21.15.16:0/3549520850 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fb1b4065380 0x7fb1b4067830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:07.940 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.922+0000 7fb1c67fc640 1 --2- 172.21.15.16:0/3549520850 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fb1b4065380 0x7fb1b4067830 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fb1bc0ad000 tx=0x7fb1bc0a6000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:07.941 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:07.926+0000 7fb1af7fe640 1 -- 172.21.15.16:0/3549520850 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fb1b8050bb0 con 0x7fb1c815c920 2023-05-22T15:27:08.089 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.078+0000 7fb1cd590640 1 -- 172.21.15.16:0/3549520850 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb1c804dbe0 con 0x7fb1c815c920 2023-05-22T15:27:08.091 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.082+0000 7fb1af7fe640 1 -- 172.21.15.16:0/3549520850 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) v1 ==== 74+0+19604 (secure 0 0 0) 0x7fb1b80541d0 con 0x7fb1c815c920 2023-05-22T15:27:08.091 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:27:08.092 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":27,"fsid":"b2719834-7847-4d98-90a6-f65d57f85c49","created":"2023-05-22T15:26:25.259314+0000","modified":"2023-05-22T15:26:53.971378+0000","last_up_change":"2023-05-22T15:26:36.448524+0000","last_in_change":"2023-05-22T15:26:27.675668+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-05-22T15:26:38.019781+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-05-22T15:26:43.570514+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-05-22T15:26:47.879159+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-05-22T15:26:48.522130+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data_ec","create_time":"2023-05-22T15:26:51.616142+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"9e7ff854-b25f-4cbb-a5c2-1799d9c53d2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":37021},{"type":"v1","addr":"172.21.15.16:6825","nonce":37021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":37021},{"type":"v1","addr":"172.21.15.16:6827","nonce":37021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":37021},{"type":"v1","addr":"172.21.15.16:6831","nonce":37021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":37021},{"type":"v1","addr":"172.21.15.16:6829","nonce":37021}]},"public_addr":"172.21.15.16:6825/37021","cluster_addr":"172.21.15.16:6827/37021","heartbeat_back_addr":"172.21.15.16:6831/37021","heartbeat_front_addr":"172.21.15.16:6829/37021","state":["exists","up"]},{"osd":1,"uuid":"4702fbc9-a927-4565-aca5-41dc69cc8264","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":37008},{"type":"v1","addr":"172.21.15.16:6801","nonce":37008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":37008},{"type":"v1","addr":"172.21.15.16:6803","nonce":37008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":37008},{"type":"v1","addr":"172.21.15.16:6807","nonce":37008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":37008},{"type":"v1","addr":"172.21.15.16:6805","nonce":37008}]},"public_addr":"172.21.15.16:6801/37008","cluster_addr":"172.21.15.16:6803/37008","heartbeat_back_addr":"172.21.15.16:6807/37008","heartbeat_front_addr":"172.21.15.16:6805/37008","state":["exists","up"]},{"osd":2,"uuid":"1e7bba46-4eba-47ef-bb81-5e2d371638ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":37009},{"type":"v1","addr":"172.21.15.16:6809","nonce":37009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":37009},{"type":"v1","addr":"172.21.15.16:6811","nonce":37009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":37009},{"type":"v1","addr":"172.21.15.16:6815","nonce":37009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":37009},{"type":"v1","addr":"172.21.15.16:6813","nonce":37009}]},"public_addr":"172.21.15.16:6809/37009","cluster_addr":"172.21.15.16:6811/37009","heartbeat_back_addr":"172.21.15.16:6815/37009","heartbeat_front_addr":"172.21.15.16:6813/37009","state":["exists","up"]},{"osd":3,"uuid":"383cacc0-5a88-4904-990e-498ca59ef48a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":37020},{"type":"v1","addr":"172.21.15.16:6817","nonce":37020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":37020},{"type":"v1","addr":"172.21.15.16:6819","nonce":37020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":37020},{"type":"v1","addr":"172.21.15.16:6823","nonce":37020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":37020},{"type":"v1","addr":"172.21.15.16:6821","nonce":37020}]},"public_addr":"172.21.15.16:6817/37020","cluster_addr":"172.21.15.16:6819/37020","heartbeat_back_addr":"172.21.15.16:6823/37020","heartbeat_front_addr":"172.21.15.16:6821/37020","state":["exists","up"]},{"osd":4,"uuid":"970500ef-95b7-42c1-b080-eed4fae7013d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":37353},{"type":"v1","addr":"172.21.15.62:6817","nonce":37353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6819","nonce":37353},{"type":"v1","addr":"172.21.15.62:6822","nonce":37353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37353},{"type":"v1","addr":"172.21.15.62:6831","nonce":37353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37353},{"type":"v1","addr":"172.21.15.62:6826","nonce":37353}]},"public_addr":"172.21.15.62:6817/37353","cluster_addr":"172.21.15.62:6822/37353","heartbeat_back_addr":"172.21.15.62:6831/37353","heartbeat_front_addr":"172.21.15.62:6826/37353","state":["exists","up"]},{"osd":5,"uuid":"022d67db-d4d8-445b-b1b9-8b446a00f5c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":37352},{"type":"v1","addr":"172.21.15.62:6801","nonce":37352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":37352},{"type":"v1","addr":"172.21.15.62:6803","nonce":37352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":37352},{"type":"v1","addr":"172.21.15.62:6807","nonce":37352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":37352},{"type":"v1","addr":"172.21.15.62:6805","nonce":37352}]},"public_addr":"172.21.15.62:6801/37352","cluster_addr":"172.21.15.62:6803/37352","heartbeat_back_addr":"172.21.15.62:6807/37352","heartbeat_front_addr":"172.21.15.62:6805/37352","state":["exists","up"]},{"osd":6,"uuid":"42bd372b-6688-4a1b-8407-e16e8c266e33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":37354},{"type":"v1","addr":"172.21.15.62:6820","nonce":37354}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":37354},{"type":"v1","addr":"172.21.15.62:6823","nonce":37354}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6829","nonce":37354},{"type":"v1","addr":"172.21.15.62:6830","nonce":37354}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6825","nonce":37354},{"type":"v1","addr":"172.21.15.62:6827","nonce":37354}]},"public_addr":"172.21.15.62:6820/37354","cluster_addr":"172.21.15.62:6823/37354","heartbeat_back_addr":"172.21.15.62:6830/37354","heartbeat_front_addr":"172.21.15.62:6827/37354","state":["exists","up"]},{"osd":7,"uuid":"ac70b2a3-f41b-4bf2-b227-7f2cbb55426d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37355},{"type":"v1","addr":"172.21.15.62:6809","nonce":37355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37355},{"type":"v1","addr":"172.21.15.62:6811","nonce":37355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":37355},{"type":"v1","addr":"172.21.15.62:6815","nonce":37355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":37355},{"type":"v1","addr":"172.21.15.62:6813","nonce":37355}]},"public_addr":"172.21.15.62:6809/37355","cluster_addr":"172.21.15.62:6811/37355","heartbeat_back_addr":"172.21.15.62:6815/37355","heartbeat_front_addr":"172.21.15.62:6813/37355","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.368896+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.107273+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.276888+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.287502+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.124791+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.251937+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.255679+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.232797+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-05-22T15:27:08.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.082+0000 7fb1cd590640 1 -- 172.21.15.16:0/3549520850 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fb1b4065380 msgr2=0x7fb1b4067830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:08.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.082+0000 7fb1cd590640 1 --2- 172.21.15.16:0/3549520850 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fb1b4065380 0x7fb1b4067830 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fb1bc0ad000 tx=0x7fb1bc0a6000 comp rx=0 tx=0).stop 2023-05-22T15:27:08.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.082+0000 7fb1cd590640 1 -- 172.21.15.16:0/3549520850 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb1c815c920 msgr2=0x7fb1c80b4880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:08.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.082+0000 7fb1cd590640 1 --2- 172.21.15.16:0/3549520850 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb1c815c920 0x7fb1c80b4880 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fb1b800c4c0 tx=0x7fb1b800c980 comp rx=0 tx=0).stop 2023-05-22T15:27:08.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.082+0000 7fb1cd590640 1 -- 172.21.15.16:0/3549520850 shutdown_connections 2023-05-22T15:27:08.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.082+0000 7fb1cd590640 1 --2- 172.21.15.16:0/3549520850 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb1c815fc50 0x7fb1c8153c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:08.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.082+0000 7fb1cd590640 1 --2- 172.21.15.16:0/3549520850 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fb1b4065380 0x7fb1b4067830 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:08.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.082+0000 7fb1cd590640 1 --2- 172.21.15.16:0/3549520850 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb1c815cd70 0x7fb1c8155430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:08.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.082+0000 7fb1cd590640 1 --2- 172.21.15.16:0/3549520850 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb1c815c920 0x7fb1c80b4880 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:08.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.082+0000 7fb1cd590640 1 -- 172.21.15.16:0/3549520850 >> 172.21.15.16:0/3549520850 conn(0x7fb1c80657f0 msgr2=0x7fb1c8160020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:08.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.082+0000 7fb1cd590640 1 -- 172.21.15.16:0/3549520850 shutdown_connections 2023-05-22T15:27:08.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.082+0000 7fb1cd590640 1 -- 172.21.15.16:0/3549520850 wait complete. 2023-05-22T15:27:08.110 INFO:tasks.ceph.ceph_manager.ceph:all up! 2023-05-22T15:27:08.110 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-05-22T15:27:08.233 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.222+0000 7f7411a99640 1 Processor -- start 2023-05-22T15:27:08.233 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.222+0000 7f7411a99640 1 -- start start 2023-05-22T15:27:08.234 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.226+0000 7f7411a99640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f740c0ccab0 0x7f740c15fe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:08.234 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.226+0000 7f7411a99640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f740c0d0410 0x7f740c0d0810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:08.234 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.226+0000 7f7411a99640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f740c0ccef0 con 0x7f740c0b7dd0 2023-05-22T15:27:08.235 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.226+0000 7f7411a99640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f740c0d3360 con 0x7f740c0ccab0 2023-05-22T15:27:08.235 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.226+0000 7f7411a99640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f740c0cd170 con 0x7f740c0d0410 2023-05-22T15:27:08.235 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.226+0000 7f740affd640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f740c0ccab0 0x7f740c15fe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:08.235 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.226+0000 7f740affd640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f740c0ccab0 0x7f740c15fe10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57744/0 (socket says 172.21.15.16:57744) 2023-05-22T15:27:08.235 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.226+0000 7f740affd640 1 -- 172.21.15.16:0/3815185801 learned_addr learned my addr 172.21.15.16:0/3815185801 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:08.235 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.226+0000 7f740bfff640 1 --2- 172.21.15.16:0/3815185801 >> v2:172.21.15.62:3300/0 conn(0x7f740c0d0410 0x7f740c0d0810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:08.236 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.226+0000 7f740affd640 1 -- 172.21.15.16:0/3815185801 >> v2:172.21.15.62:3300/0 conn(0x7f740c0d0410 msgr2=0x7f740c0d0810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:08.236 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.226+0000 7f740affd640 1 --2- 172.21.15.16:0/3815185801 >> v2:172.21.15.62:3300/0 conn(0x7f740c0d0410 0x7f740c0d0810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:08.236 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.226+0000 7f740affd640 1 -- 172.21.15.16:0/3815185801 >> v1:172.21.15.16:6789/0 conn(0x7f740c0b7dd0 legacy=0x7f740c15f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:08.236 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.226+0000 7f740affd640 1 -- 172.21.15.16:0/3815185801 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f740c0d2ac0 con 0x7f740c0ccab0 2023-05-22T15:27:08.237 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.226+0000 7f740affd640 1 --2- 172.21.15.16:0/3815185801 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f740c0ccab0 0x7f740c15fe10 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f740c15c190 tx=0x7f740001c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9c5c34d461e9d2f server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:08.237 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.226+0000 7f740a7fc640 1 -- 172.21.15.16:0/3815185801 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74000aa430 con 0x7f740c0ccab0 2023-05-22T15:27:08.237 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.226+0000 7f740a7fc640 1 -- 172.21.15.16:0/3815185801 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74000aa590 con 0x7f740c0ccab0 2023-05-22T15:27:08.237 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.226+0000 7f740a7fc640 1 -- 172.21.15.16:0/3815185801 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74000aa920 con 0x7f740c0ccab0 2023-05-22T15:27:08.238 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.226+0000 7f7411a99640 1 -- 172.21.15.16:0/3815185801 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f740c0ccab0 msgr2=0x7f740c15fe10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:08.239 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.226+0000 7f7411a99640 1 --2- 172.21.15.16:0/3815185801 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f740c0ccab0 0x7f740c15fe10 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f740c15c190 tx=0x7f740001c4d0 comp rx=0 tx=0).stop 2023-05-22T15:27:08.239 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.226+0000 7f7411a99640 1 -- 172.21.15.16:0/3815185801 shutdown_connections 2023-05-22T15:27:08.239 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.226+0000 7f7411a99640 1 --2- 172.21.15.16:0/3815185801 >> v2:172.21.15.62:3300/0 conn(0x7f740c0d0410 0x7f740c0d0810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:08.239 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.226+0000 7f7411a99640 1 --2- 172.21.15.16:0/3815185801 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f740c0ccab0 0x7f740c15fe10 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:08.240 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.226+0000 7f7411a99640 1 -- 172.21.15.16:0/3815185801 >> 172.21.15.16:0/3815185801 conn(0x7f740c0655c0 msgr2=0x7f740c070e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:08.240 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.226+0000 7f7411a99640 1 -- 172.21.15.16:0/3815185801 shutdown_connections 2023-05-22T15:27:08.240 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.226+0000 7f7411a99640 1 -- 172.21.15.16:0/3815185801 wait complete. 2023-05-22T15:27:08.243 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.230+0000 7f7411a99640 1 Processor -- start 2023-05-22T15:27:08.243 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.230+0000 7f7411a99640 1 -- start start 2023-05-22T15:27:08.244 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.230+0000 7f7411a99640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f740c0b7dd0 0x7f740c154350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:08.244 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.230+0000 7f7411a99640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f740c0ccab0 0x7f740c156c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:08.244 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.230+0000 7f7411a99640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f740c0d0410 0x7f740c155ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:08.244 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.230+0000 7f7411a99640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f740c1571d0 con 0x7f740c0b7dd0 2023-05-22T15:27:08.244 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.230+0000 7f7411a99640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f740c155620 con 0x7f740c0d0410 2023-05-22T15:27:08.244 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.230+0000 7f7411a99640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f740c155ff0 con 0x7f740c0ccab0 2023-05-22T15:27:08.245 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.230+0000 7f740b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f740c0b7dd0 0x7f740c154350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:08.245 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.230+0000 7f740b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f740c0b7dd0 0x7f740c154350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57760/0 (socket says 172.21.15.16:57760) 2023-05-22T15:27:08.245 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.230+0000 7f740b7fe640 1 -- 172.21.15.16:0/1905484324 learned_addr learned my addr 172.21.15.16:0/1905484324 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:08.246 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.230+0000 7f740affd640 1 --2- 172.21.15.16:0/1905484324 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f740c0ccab0 0x7f740c156c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:08.246 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.230+0000 7f740b7fe640 1 -- 172.21.15.16:0/1905484324 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f740c0ccab0 msgr2=0x7f740c156c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:08.246 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.230+0000 7f740b7fe640 1 --2- 172.21.15.16:0/1905484324 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f740c0ccab0 0x7f740c156c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:08.246 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.230+0000 7f740bfff640 1 --2- 172.21.15.16:0/1905484324 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f740c0d0410 0x7f740c155ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:08.247 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.230+0000 7f740b7fe640 1 -- 172.21.15.16:0/1905484324 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f740c0d0410 msgr2=0x7f740c155ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:08.247 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.230+0000 7f740b7fe640 1 --2- 172.21.15.16:0/1905484324 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f740c0d0410 0x7f740c155ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:08.247 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.230+0000 7f740b7fe640 1 -- 172.21.15.16:0/1905484324 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74000094d0 con 0x7f740c0b7dd0 2023-05-22T15:27:08.247 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.230+0000 7f740affd640 1 --2- 172.21.15.16:0/1905484324 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f740c0ccab0 0x7f740c156c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:27:08.247 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.230+0000 7f740b7fe640 1 --2- 172.21.15.16:0/1905484324 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f740c0b7dd0 0x7f740c154350 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f73fc00c500 tx=0x7f73fc00c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:08.248 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.230+0000 7f73f3fff640 1 -- 172.21.15.16:0/1905484324 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73fc005220 con 0x7f740c0b7dd0 2023-05-22T15:27:08.248 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.230+0000 7f73f3fff640 1 -- 172.21.15.16:0/1905484324 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73fc005380 con 0x7f740c0b7dd0 2023-05-22T15:27:08.249 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.230+0000 7f7411a99640 1 -- 172.21.15.16:0/1905484324 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f740c160350 con 0x7f740c0b7dd0 2023-05-22T15:27:08.250 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.230+0000 7f73f3fff640 1 -- 172.21.15.16:0/1905484324 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73fc007270 con 0x7f740c0b7dd0 2023-05-22T15:27:08.250 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.230+0000 7f7411a99640 1 -- 172.21.15.16:0/1905484324 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f740c0cf730 con 0x7f740c0b7dd0 2023-05-22T15:27:08.250 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.230+0000 7f7411a99640 1 -- 172.21.15.16:0/1905484324 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f740c1604e0 con 0x7f740c0b7dd0 2023-05-22T15:27:08.250 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.234+0000 7f73f3fff640 1 -- 172.21.15.16:0/1905484324 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7f73fc0073d0 con 0x7f740c0b7dd0 2023-05-22T15:27:08.250 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.234+0000 7f73f3fff640 1 --2- 172.21.15.16:0/1905484324 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f73f8065380 0x7f73f8067830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:08.250 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.234+0000 7f73f3fff640 1 -- 172.21.15.16:0/1905484324 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f73fc0824b0 con 0x7f740c0b7dd0 2023-05-22T15:27:08.250 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.234+0000 7f740affd640 1 --2- 172.21.15.16:0/1905484324 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f73f8065380 0x7f73f8067830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:08.251 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.234+0000 7f740affd640 1 --2- 172.21.15.16:0/1905484324 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f73f8065380 0x7f73f8067830 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f74000ae000 tx=0x7f7400008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:08.251 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.238+0000 7f73f3fff640 1 -- 172.21.15.16:0/1905484324 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f73fc04fdf0 con 0x7f740c0b7dd0 2023-05-22T15:27:08.399 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.390+0000 7f7411a99640 1 -- 172.21.15.16:0/1905484324 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f740c04dbe0 con 0x7f740c0b7dd0 2023-05-22T15:27:08.402 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.394+0000 7f73f3fff640 1 -- 172.21.15.16:0/1905484324 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) v1 ==== 74+0+19604 (secure 0 0 0) 0x7f73fc053410 con 0x7f740c0b7dd0 2023-05-22T15:27:08.403 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:27:08.403 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":27,"fsid":"b2719834-7847-4d98-90a6-f65d57f85c49","created":"2023-05-22T15:26:25.259314+0000","modified":"2023-05-22T15:26:53.971378+0000","last_up_change":"2023-05-22T15:26:36.448524+0000","last_in_change":"2023-05-22T15:26:27.675668+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-05-22T15:26:38.019781+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-05-22T15:26:43.570514+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-05-22T15:26:47.879159+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-05-22T15:26:48.522130+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data_ec","create_time":"2023-05-22T15:26:51.616142+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"9e7ff854-b25f-4cbb-a5c2-1799d9c53d2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":37021},{"type":"v1","addr":"172.21.15.16:6825","nonce":37021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":37021},{"type":"v1","addr":"172.21.15.16:6827","nonce":37021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":37021},{"type":"v1","addr":"172.21.15.16:6831","nonce":37021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":37021},{"type":"v1","addr":"172.21.15.16:6829","nonce":37021}]},"public_addr":"172.21.15.16:6825/37021","cluster_addr":"172.21.15.16:6827/37021","heartbeat_back_addr":"172.21.15.16:6831/37021","heartbeat_front_addr":"172.21.15.16:6829/37021","state":["exists","up"]},{"osd":1,"uuid":"4702fbc9-a927-4565-aca5-41dc69cc8264","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":37008},{"type":"v1","addr":"172.21.15.16:6801","nonce":37008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":37008},{"type":"v1","addr":"172.21.15.16:6803","nonce":37008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":37008},{"type":"v1","addr":"172.21.15.16:6807","nonce":37008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":37008},{"type":"v1","addr":"172.21.15.16:6805","nonce":37008}]},"public_addr":"172.21.15.16:6801/37008","cluster_addr":"172.21.15.16:6803/37008","heartbeat_back_addr":"172.21.15.16:6807/37008","heartbeat_front_addr":"172.21.15.16:6805/37008","state":["exists","up"]},{"osd":2,"uuid":"1e7bba46-4eba-47ef-bb81-5e2d371638ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":37009},{"type":"v1","addr":"172.21.15.16:6809","nonce":37009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":37009},{"type":"v1","addr":"172.21.15.16:6811","nonce":37009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":37009},{"type":"v1","addr":"172.21.15.16:6815","nonce":37009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":37009},{"type":"v1","addr":"172.21.15.16:6813","nonce":37009}]},"public_addr":"172.21.15.16:6809/37009","cluster_addr":"172.21.15.16:6811/37009","heartbeat_back_addr":"172.21.15.16:6815/37009","heartbeat_front_addr":"172.21.15.16:6813/37009","state":["exists","up"]},{"osd":3,"uuid":"383cacc0-5a88-4904-990e-498ca59ef48a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":37020},{"type":"v1","addr":"172.21.15.16:6817","nonce":37020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":37020},{"type":"v1","addr":"172.21.15.16:6819","nonce":37020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":37020},{"type":"v1","addr":"172.21.15.16:6823","nonce":37020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":37020},{"type":"v1","addr":"172.21.15.16:6821","nonce":37020}]},"public_addr":"172.21.15.16:6817/37020","cluster_addr":"172.21.15.16:6819/37020","heartbeat_back_addr":"172.21.15.16:6823/37020","heartbeat_front_addr":"172.21.15.16:6821/37020","state":["exists","up"]},{"osd":4,"uuid":"970500ef-95b7-42c1-b080-eed4fae7013d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":37353},{"type":"v1","addr":"172.21.15.62:6817","nonce":37353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6819","nonce":37353},{"type":"v1","addr":"172.21.15.62:6822","nonce":37353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37353},{"type":"v1","addr":"172.21.15.62:6831","nonce":37353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37353},{"type":"v1","addr":"172.21.15.62:6826","nonce":37353}]},"public_addr":"172.21.15.62:6817/37353","cluster_addr":"172.21.15.62:6822/37353","heartbeat_back_addr":"172.21.15.62:6831/37353","heartbeat_front_addr":"172.21.15.62:6826/37353","state":["exists","up"]},{"osd":5,"uuid":"022d67db-d4d8-445b-b1b9-8b446a00f5c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":37352},{"type":"v1","addr":"172.21.15.62:6801","nonce":37352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":37352},{"type":"v1","addr":"172.21.15.62:6803","nonce":37352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":37352},{"type":"v1","addr":"172.21.15.62:6807","nonce":37352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":37352},{"type":"v1","addr":"172.21.15.62:6805","nonce":37352}]},"public_addr":"172.21.15.62:6801/37352","cluster_addr":"172.21.15.62:6803/37352","heartbeat_back_addr":"172.21.15.62:6807/37352","heartbeat_front_addr":"172.21.15.62:6805/37352","state":["exists","up"]},{"osd":6,"uuid":"42bd372b-6688-4a1b-8407-e16e8c266e33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":37354},{"type":"v1","addr":"172.21.15.62:6820","nonce":37354}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":37354},{"type":"v1","addr":"172.21.15.62:6823","nonce":37354}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6829","nonce":37354},{"type":"v1","addr":"172.21.15.62:6830","nonce":37354}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6825","nonce":37354},{"type":"v1","addr":"172.21.15.62:6827","nonce":37354}]},"public_addr":"172.21.15.62:6820/37354","cluster_addr":"172.21.15.62:6823/37354","heartbeat_back_addr":"172.21.15.62:6830/37354","heartbeat_front_addr":"172.21.15.62:6827/37354","state":["exists","up"]},{"osd":7,"uuid":"ac70b2a3-f41b-4bf2-b227-7f2cbb55426d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37355},{"type":"v1","addr":"172.21.15.62:6809","nonce":37355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37355},{"type":"v1","addr":"172.21.15.62:6811","nonce":37355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":37355},{"type":"v1","addr":"172.21.15.62:6815","nonce":37355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":37355},{"type":"v1","addr":"172.21.15.62:6813","nonce":37355}]},"public_addr":"172.21.15.62:6809/37355","cluster_addr":"172.21.15.62:6811/37355","heartbeat_back_addr":"172.21.15.62:6815/37355","heartbeat_front_addr":"172.21.15.62:6813/37355","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.368896+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.107273+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.276888+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.287502+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.124791+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.251937+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.255679+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.232797+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-05-22T15:27:08.405 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.394+0000 7f7411a99640 1 -- 172.21.15.16:0/1905484324 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f73f8065380 msgr2=0x7f73f8067830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:08.405 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.394+0000 7f7411a99640 1 --2- 172.21.15.16:0/1905484324 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f73f8065380 0x7f73f8067830 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f74000ae000 tx=0x7f7400008040 comp rx=0 tx=0).stop 2023-05-22T15:27:08.405 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.394+0000 7f7411a99640 1 -- 172.21.15.16:0/1905484324 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f740c0b7dd0 msgr2=0x7f740c154350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:08.405 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.394+0000 7f7411a99640 1 --2- 172.21.15.16:0/1905484324 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f740c0b7dd0 0x7f740c154350 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f73fc00c500 tx=0x7f73fc00c9c0 comp rx=0 tx=0).stop 2023-05-22T15:27:08.405 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.394+0000 7f7411a99640 1 -- 172.21.15.16:0/1905484324 shutdown_connections 2023-05-22T15:27:08.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.394+0000 7f7411a99640 1 --2- 172.21.15.16:0/1905484324 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f740c0d0410 0x7f740c155ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:08.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.394+0000 7f7411a99640 1 --2- 172.21.15.16:0/1905484324 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f73f8065380 0x7f73f8067830 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:08.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.394+0000 7f7411a99640 1 --2- 172.21.15.16:0/1905484324 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f740c0ccab0 0x7f740c156c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:08.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.394+0000 7f7411a99640 1 --2- 172.21.15.16:0/1905484324 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f740c0b7dd0 0x7f740c154350 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:08.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.394+0000 7f7411a99640 1 -- 172.21.15.16:0/1905484324 >> 172.21.15.16:0/1905484324 conn(0x7f740c0655c0 msgr2=0x7f740c0cef70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:08.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.394+0000 7f7411a99640 1 -- 172.21.15.16:0/1905484324 shutdown_connections 2023-05-22T15:27:08.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.394+0000 7f7411a99640 1 -- 172.21.15.16:0/1905484324 wait complete. 2023-05-22T15:27:08.421 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2023-05-22T15:27:08.422 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2023-05-22T15:27:08.422 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2023-05-22T15:27:08.423 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2023-05-22T15:27:08.423 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2023-05-22T15:27:08.424 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2023-05-22T15:27:08.424 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2023-05-22T15:27:08.425 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2023-05-22T15:27:08.602 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.594+0000 7f3aebeb8640 1 Processor -- start 2023-05-22T15:27:08.602 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.594+0000 7f3aebeb8640 1 -- start start 2023-05-22T15:27:08.603 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.594+0000 7f7282141640 1 Processor -- start 2023-05-22T15:27:08.603 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.594+0000 7f7282141640 1 -- start start 2023-05-22T15:27:08.603 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.594+0000 7f5d175af640 1 Processor -- start 2023-05-22T15:27:08.604 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.594+0000 7f3aebeb8640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3ae415ff20 0x7f3ae40cc3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:08.604 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.594+0000 7f3aebeb8640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f3ae40cc8f0 0x7f3ae40cde10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:08.604 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.594+0000 7f3aebeb8640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ae40cbbf0 con 0x7f3ae415fb50 2023-05-22T15:27:08.604 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.594+0000 7f3aebeb8640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ae40cdc60 con 0x7f3ae40cc8f0 2023-05-22T15:27:08.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.594+0000 7f3aebeb8640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ae40cd6a0 con 0x7f3ae415ff20 2023-05-22T15:27:08.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.594+0000 7f3ae9c54640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f3ae415fb50 0x7f3ae406bf60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:51056/0 (socket says 172.21.15.16:51056) 2023-05-22T15:27:08.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.594+0000 7f3ae9c54640 1 -- 172.21.15.16:0/4038076565 learned_addr learned my addr 172.21.15.16:0/4038076565 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:27:08.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.594+0000 7f3aea455640 1 --2- 172.21.15.16:0/4038076565 >> v2:172.21.15.16:3300/0 conn(0x7f3ae40cc8f0 0x7f3ae40cde10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:08.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.594+0000 7f5d175af640 1 -- start start 2023-05-22T15:27:08.606 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.594+0000 7f3ae9453640 1 --2- 172.21.15.16:0/4038076565 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3ae415ff20 0x7f3ae40cc3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:08.606 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.594+0000 7f3ae8c52640 1 -- 172.21.15.16:0/4038076565 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1227411855 0 0) 0x7f3ae40cbbf0 con 0x7f3ae415fb50 2023-05-22T15:27:08.606 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.594+0000 7f3ae8c52640 1 -- 172.21.15.16:0/4038076565 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3ad0003530 con 0x7f3ae415fb50 2023-05-22T15:27:08.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.594+0000 7f6ff6b17640 1 Processor -- start 2023-05-22T15:27:08.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.594+0000 7f3aea455640 1 -- 172.21.15.16:0/4038076565 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3ae415ff20 msgr2=0x7f3ae40cc3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:08.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.594+0000 7f3aea455640 1 --2- 172.21.15.16:0/4038076565 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3ae415ff20 0x7f3ae40cc3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:08.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.594+0000 7f3aea455640 1 -- 172.21.15.16:0/4038076565 >> v1:172.21.15.16:6789/0 conn(0x7f3ae415fb50 legacy=0x7f3ae406bf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:08.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.594+0000 7f3aea455640 1 -- 172.21.15.16:0/4038076565 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ae40cd980 con 0x7f3ae40cc8f0 2023-05-22T15:27:08.609 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.594+0000 7f7282141640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f727c15ff20 0x7f727c0cc3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:08.609 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.594+0000 7f7282141640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f727c0cbbf0 con 0x7f727c0cc8f0 2023-05-22T15:27:08.609 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.594+0000 7f7282141640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f727c0cd6a0 con 0x7f727c15fb50 2023-05-22T15:27:08.609 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.594+0000 7f7282141640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f727c0cdc60 con 0x7f727c15ff20 2023-05-22T15:27:08.610 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.594+0000 7f6ff6b17640 1 -- start start 2023-05-22T15:27:08.611 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.594+0000 7f727bfff640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f727c0cc8f0 0x7f727c0cde10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:51070/0 (socket says 172.21.15.16:51070) 2023-05-22T15:27:08.611 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.594+0000 7f727bfff640 1 -- 172.21.15.16:0/1395995082 learned_addr learned my addr 172.21.15.16:0/1395995082 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:27:08.611 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.594+0000 7f727affd640 1 --2- 172.21.15.16:0/1395995082 >> v2:172.21.15.62:3300/0 conn(0x7f727c15ff20 0x7f727c0cc3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:08.612 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.594+0000 7f5d175af640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f5d1015fb50 0x7f5d1015ff20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:08.612 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.594+0000 7f5d175af640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f5d1006c390 0x7f5d100cde10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:08.612 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.594+0000 7f5d175af640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d100cbbf0 con 0x7f5d1006bf60 2023-05-22T15:27:08.613 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.594+0000 7f5d175af640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d10160460 con 0x7f5d1015fb50 2023-05-22T15:27:08.613 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.594+0000 7f5d175af640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d100cdc60 con 0x7f5d1006c390 2023-05-22T15:27:08.614 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.594+0000 7f727a7fc640 1 -- 172.21.15.16:0/1395995082 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1473269852 0 0) 0x7f727c0cbbf0 con 0x7f727c0cc8f0 2023-05-22T15:27:08.614 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.594+0000 7f727a7fc640 1 -- 172.21.15.16:0/1395995082 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7264003530 con 0x7f727c0cc8f0 2023-05-22T15:27:08.615 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.594+0000 7fc5d744c640 1 Processor -- start 2023-05-22T15:27:08.616 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.594+0000 7f5d15b4c640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f5d1006c390 0x7f5d100cde10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:08.616 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.594+0000 7f5d15b4c640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f5d1006c390 0x7f5d100cde10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.16:37774/0 (socket says 172.21.15.16:37774) 2023-05-22T15:27:08.616 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.594+0000 7f5d15b4c640 1 -- 172.21.15.16:0/460058672 learned_addr learned my addr 172.21.15.16:0/460058672 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:08.618 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.598+0000 7f6ff6b17640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6ff00b56a0 0x7f6ff015d7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:08.618 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.598+0000 7f6ff6b17640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6ff006f970 0x7f6ff006de90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:08.619 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.598+0000 7f6ff6b17640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ff006da00 con 0x7f6ff00b5b90 2023-05-22T15:27:08.619 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.598+0000 7f6ff6b17640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ff015a190 con 0x7f6ff006f970 2023-05-22T15:27:08.619 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.598+0000 7f6ff6b17640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ff00cbd10 con 0x7f6ff00b56a0 2023-05-22T15:27:08.619 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.598+0000 7f6ff6316640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6ff006f970 0x7f6ff006de90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:08.620 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.598+0000 7f6ff6316640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6ff006f970 0x7f6ff006de90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57792/0 (socket says 172.21.15.16:57792) 2023-05-22T15:27:08.620 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.598+0000 7f6ff6316640 1 -- 172.21.15.16:0/2386366954 learned_addr learned my addr 172.21.15.16:0/2386366954 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:08.620 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.598+0000 7f6ff5b15640 1 --2- 172.21.15.16:0/2386366954 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6ff00b56a0 0x7f6ff015d7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:08.620 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.598+0000 7f6ff6316640 1 -- 172.21.15.16:0/2386366954 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6ff00b56a0 msgr2=0x7f6ff015d7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:08.621 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.598+0000 7f6ff6316640 1 --2- 172.21.15.16:0/2386366954 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6ff00b56a0 0x7f6ff015d7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:08.621 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.598+0000 7f6ff6316640 1 -- 172.21.15.16:0/2386366954 >> v1:172.21.15.62:6789/0 conn(0x7f6ff00b5b90 legacy=0x7f6ff006f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:08.621 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.598+0000 7f6ff6316640 1 -- 172.21.15.16:0/2386366954 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ff00cbf90 con 0x7f6ff006f970 2023-05-22T15:27:08.621 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.598+0000 7f3aea455640 1 --2- 172.21.15.16:0/4038076565 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3ae40cc8f0 0x7f3ae40cde10 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f3ad40097c0 tx=0x7f3ad401c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=834d4b4f01e38410 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:08.622 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.598+0000 7f3ae8c52640 1 -- 172.21.15.16:0/4038076565 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ad40ab600 con 0x7f3ae40cc8f0 2023-05-22T15:27:08.622 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.598+0000 7f3ae8c52640 1 -- 172.21.15.16:0/4038076565 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ad40a7050 con 0x7f3ae40cc8f0 2023-05-22T15:27:08.622 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.598+0000 7f3ae8c52640 1 -- 172.21.15.16:0/4038076565 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ad40ab980 con 0x7f3ae40cc8f0 2023-05-22T15:27:08.622 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.594+0000 7fd40a7a0640 1 Processor -- start 2023-05-22T15:27:08.623 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.598+0000 7fd40a7a0640 1 -- start start 2023-05-22T15:27:08.623 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.598+0000 7fd40a7a0640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd40415ff30 0x7fd4040c9660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:08.623 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.598+0000 7fd40a7a0640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fd404069e30 0x7fd40406d8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:08.623 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.598+0000 7fd40a7a0640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd404071250 con 0x7fd4040cead0 2023-05-22T15:27:08.623 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.598+0000 7fd40a7a0640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4040c9ba0 con 0x7fd404069e30 2023-05-22T15:27:08.623 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.598+0000 7fd40a7a0640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4040714a0 con 0x7fd40415ff30 2023-05-22T15:27:08.624 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.598+0000 7fc5d744c640 1 -- start start 2023-05-22T15:27:08.624 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.598+0000 7fc5d744c640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc5d006c7c0 0x7fc5d00716d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:08.625 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.598+0000 7fc5d744c640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5d0164a70 con 0x7fc5d0160150 2023-05-22T15:27:08.625 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.598+0000 7fc5d744c640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5d006d220 con 0x7fc5d00cd410 2023-05-22T15:27:08.625 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.598+0000 7fc5d744c640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5d00cc770 con 0x7fc5d006c7c0 2023-05-22T15:27:08.626 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.598+0000 7fd408f9d640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7fd4040cead0 0x7fd404069720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.16:36006/0 (socket says 172.21.15.16:36006) 2023-05-22T15:27:08.626 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.598+0000 7fd408f9d640 1 -- 172.21.15.16:0/3002049849 learned_addr learned my addr 172.21.15.16:0/3002049849 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:27:08.627 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.598+0000 7fc5d5c49640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc5d006c7c0 0x7fc5d00716d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:08.627 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.598+0000 7fc5d5c49640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc5d006c7c0 0x7fc5d00716d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3301/0 says I am v2:172.21.15.16:53218/0 (socket says 172.21.15.16:53218) 2023-05-22T15:27:08.627 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.598+0000 7fc5d5c49640 1 -- 172.21.15.16:0/148263583 learned_addr learned my addr 172.21.15.16:0/148263583 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:08.628 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.598+0000 7fd409f9f640 1 --2- 172.21.15.16:0/3002049849 >> v2:172.21.15.62:3300/0 conn(0x7fd404069e30 0x7fd40406d8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:08.628 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.598+0000 7fd40979e640 1 --2- 172.21.15.16:0/3002049849 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd40415ff30 0x7fd4040c9660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:08.629 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.598+0000 7f3aebeb8640 1 -- 172.21.15.16:0/4038076565 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3ae40cc8f0 msgr2=0x7f3ae40cde10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:08.629 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.598+0000 7f3aebeb8640 1 --2- 172.21.15.16:0/4038076565 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3ae40cc8f0 0x7f3ae40cde10 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f3ad40097c0 tx=0x7f3ad401c4a0 comp rx=0 tx=0).stop 2023-05-22T15:27:08.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.598+0000 7f5d15b4c640 1 -- 172.21.15.16:0/460058672 >> v2:172.21.15.16:3300/0 conn(0x7f5d1015fb50 msgr2=0x7f5d1015ff20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:08.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.598+0000 7f5d15b4c640 1 --2- 172.21.15.16:0/460058672 >> v2:172.21.15.16:3300/0 conn(0x7f5d1015fb50 0x7f5d1015ff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:08.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.598+0000 7f5d15b4c640 1 -- 172.21.15.16:0/460058672 >> v1:172.21.15.62:6789/0 conn(0x7f5d1006bf60 legacy=0x7f5d100cc3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:08.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.598+0000 7f5d15b4c640 1 -- 172.21.15.16:0/460058672 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d100cd6a0 con 0x7f5d1006c390 2023-05-22T15:27:08.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.598+0000 7fd3fbfff640 1 -- 172.21.15.16:0/3002049849 <== mon.1 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 51928825 0 0) 0x7fd404071250 con 0x7fd4040cead0 2023-05-22T15:27:08.632 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.598+0000 7fd3fbfff640 1 -- 172.21.15.16:0/3002049849 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd3f0003530 con 0x7fd4040cead0 2023-05-22T15:27:08.632 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.598+0000 7f3aebeb8640 1 -- 172.21.15.16:0/4038076565 shutdown_connections 2023-05-22T15:27:08.633 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.598+0000 7f3aebeb8640 1 --2- 172.21.15.16:0/4038076565 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3ae40cc8f0 0x7f3ae40cde10 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:08.633 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.598+0000 7f3aebeb8640 1 --2- 172.21.15.16:0/4038076565 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3ae415ff20 0x7f3ae40cc3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:08.633 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.598+0000 7f3aebeb8640 1 -- 172.21.15.16:0/4038076565 >> 172.21.15.16:0/4038076565 conn(0x7f3ae40655c0 msgr2=0x7f3ae4069720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:08.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.598+0000 7f3aebeb8640 1 -- 172.21.15.16:0/4038076565 shutdown_connections 2023-05-22T15:27:08.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.598+0000 7f3aebeb8640 1 -- 172.21.15.16:0/4038076565 wait complete. 2023-05-22T15:27:08.635 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.598+0000 7f727affd640 1 -- 172.21.15.16:0/1395995082 >> v1:172.21.15.62:6789/0 conn(0x7f727c15fb50 legacy=0x7f727c06bf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:08.635 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.598+0000 7f727affd640 1 -- 172.21.15.16:0/1395995082 >> v1:172.21.15.16:6789/0 conn(0x7f727c0cc8f0 legacy=0x7f727c0cde10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:08.635 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.598+0000 7f727affd640 1 -- 172.21.15.16:0/1395995082 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f727c0cd980 con 0x7f727c15ff20 2023-05-22T15:27:08.635 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.598+0000 7f727affd640 1 --2- 172.21.15.16:0/1395995082 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f727c15ff20 0x7f727c0cc3b0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f726c0097c0 tx=0x7f726c01c4a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=570021c2b751eb35 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:08.635 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.598+0000 7f6ff6316640 1 --2- 172.21.15.16:0/2386366954 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6ff006f970 0x7f6ff006de90 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f6fe8009e50 tx=0x7f6fe801c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5a7fd80594a7e1c server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:08.636 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.598+0000 7f727a7fc640 1 -- 172.21.15.16:0/1395995082 <== mon.1 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f726c0a8050 con 0x7f727c15ff20 2023-05-22T15:27:08.636 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.598+0000 7f727a7fc640 1 -- 172.21.15.16:0/1395995082 <== mon.1 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f726c0ac5f0 con 0x7f727c15ff20 2023-05-22T15:27:08.636 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.598+0000 7f727a7fc640 1 -- 172.21.15.16:0/1395995082 <== mon.1 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f726c0ac960 con 0x7f727c15ff20 2023-05-22T15:27:08.637 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.602+0000 7f3aebeb8640 1 Processor -- start 2023-05-22T15:27:08.638 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.602+0000 7f5d15b4c640 1 --2- 172.21.15.16:0/460058672 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5d1006c390 0x7f5d100cde10 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f5d1006f3d0 tx=0x7f5d0c01c4e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e8706ae4b738a7f server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:08.638 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.602+0000 7fc005cbc640 1 Processor -- start 2023-05-22T15:27:08.638 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.598+0000 7fc5d5c49640 1 -- 172.21.15.16:0/148263583 >> v1:172.21.15.62:6789/0 conn(0x7fc5d00cd410 legacy=0x7fc5d0164360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:08.639 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.598+0000 7fc5d5c49640 1 -- 172.21.15.16:0/148263583 >> v1:172.21.15.16:6789/0 conn(0x7fc5d0160150 legacy=0x7fc5d0163c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:08.639 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.598+0000 7fc5d5c49640 1 -- 172.21.15.16:0/148263583 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5d0160520 con 0x7fc5d006c7c0 2023-05-22T15:27:08.639 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.602+0000 7fc005cbc640 1 -- start start 2023-05-22T15:27:08.640 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.602+0000 7f6ff4b13640 1 -- 172.21.15.16:0/2386366954 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fe80a5810 con 0x7f6ff006f970 2023-05-22T15:27:08.641 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.602+0000 7f6ff4b13640 1 -- 172.21.15.16:0/2386366954 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fe80a5970 con 0x7f6ff006f970 2023-05-22T15:27:08.641 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.602+0000 7f6ff4b13640 1 -- 172.21.15.16:0/2386366954 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fe80a5d00 con 0x7f6ff006f970 2023-05-22T15:27:08.641 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.602+0000 7f5d07fff640 1 -- 172.21.15.16:0/460058672 <== mon.1 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d0c0ab600 con 0x7f5d1006c390 2023-05-22T15:27:08.642 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.602+0000 7f5d07fff640 1 -- 172.21.15.16:0/460058672 <== mon.1 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d0c0a7050 con 0x7f5d1006c390 2023-05-22T15:27:08.642 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.602+0000 7f5d07fff640 1 -- 172.21.15.16:0/460058672 <== mon.1 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d0c0ab980 con 0x7f5d1006c390 2023-05-22T15:27:08.643 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.602+0000 7f7282141640 1 -- 172.21.15.16:0/1395995082 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f727c15ff20 msgr2=0x7f727c0cc3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:08.643 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.602+0000 7f7282141640 1 --2- 172.21.15.16:0/1395995082 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f727c15ff20 0x7f727c0cc3b0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f726c0097c0 tx=0x7f726c01c4a0 comp rx=0 tx=0).stop 2023-05-22T15:27:08.644 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.602+0000 7f7282141640 1 -- 172.21.15.16:0/1395995082 shutdown_connections 2023-05-22T15:27:08.644 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.602+0000 7f7282141640 1 --2- 172.21.15.16:0/1395995082 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f727c15ff20 0x7f727c0cc3b0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:08.644 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.602+0000 7f7282141640 1 -- 172.21.15.16:0/1395995082 >> 172.21.15.16:0/1395995082 conn(0x7f727c0655c0 msgr2=0x7f727c069720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:08.645 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.602+0000 7f3aebeb8640 1 -- start start 2023-05-22T15:27:08.645 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.602+0000 7f7282141640 1 -- 172.21.15.16:0/1395995082 shutdown_connections 2023-05-22T15:27:08.646 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.602+0000 7f7282141640 1 -- 172.21.15.16:0/1395995082 wait complete. 2023-05-22T15:27:08.646 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.602+0000 7f3aebeb8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3ae40cc8f0 0x7f3ae40c2d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:08.646 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.602+0000 7f3aebeb8640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3ae415fb50 0x7f3ae40b0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:08.647 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.602+0000 7f3aebeb8640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3ae415ff20 0x7f3ae4157ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:08.647 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.602+0000 7f3aebeb8640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ae40c1a60 con 0x7f3ae40cc8f0 2023-05-22T15:27:08.647 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.602+0000 7f3aebeb8640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ae41620c0 con 0x7f3ae415ff20 2023-05-22T15:27:08.648 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.602+0000 7f3aebeb8640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ae40c32c0 con 0x7f3ae415fb50 2023-05-22T15:27:08.648 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.598+0000 7fd3fbfff640 1 -- 172.21.15.16:0/3002049849 <== mon.1 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1963216970 0 0) 0x7fd3f0003530 con 0x7fd4040cead0 2023-05-22T15:27:08.648 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.598+0000 7fd3fbfff640 1 -- 172.21.15.16:0/3002049849 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd40415ff30 msgr2=0x7fd4040c9660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:08.649 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.598+0000 7fd3fbfff640 1 --2- 172.21.15.16:0/3002049849 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd40415ff30 0x7fd4040c9660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:08.649 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.598+0000 7fd3fbfff640 1 -- 172.21.15.16:0/3002049849 >> v2:172.21.15.62:3300/0 conn(0x7fd404069e30 msgr2=0x7fd40406d8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:08.649 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.598+0000 7fd3fbfff640 1 --2- 172.21.15.16:0/3002049849 >> v2:172.21.15.62:3300/0 conn(0x7fd404069e30 0x7fd40406d8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:08.650 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.602+0000 7f3ae9c54640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3ae40cc8f0 0x7f3ae40c2d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:08.650 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.602+0000 7f3ae9453640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3ae415fb50 0x7f3ae40b0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:08.651 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.598+0000 7fd3fbfff640 1 -- 172.21.15.16:0/3002049849 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd404071780 con 0x7fd4040cead0 2023-05-22T15:27:08.651 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.598+0000 7fd3fbfff640 1 -- 172.21.15.16:0/3002049849 <== mon.1 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7fd3fc001a00 con 0x7fd4040cead0 2023-05-22T15:27:08.651 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.598+0000 7fd3fbfff640 1 -- 172.21.15.16:0/3002049849 >> v1:172.21.15.62:6789/0 conn(0x7fd4040cead0 legacy=0x7fd404069720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:08.652 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.598+0000 7fd3fbfff640 1 --2- 172.21.15.16:0/3002049849 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd3f0003980 0x7fd3f0003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:08.652 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.598+0000 7fd3fbfff640 1 --2- 172.21.15.16:0/3002049849 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd3f00047d0 0x7fd3f0004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:08.652 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.598+0000 7fd3fbfff640 1 --2- 172.21.15.16:0/3002049849 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd3f0005180 0x7fd3f000b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:08.652 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.598+0000 7fd3fbfff640 1 -- 172.21.15.16:0/3002049849 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4040c9ba0 con 0x7fd3f00047d0 2023-05-22T15:27:08.653 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.598+0000 7fd3fbfff640 1 -- 172.21.15.16:0/3002049849 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4040714a0 con 0x7fd3f0003980 2023-05-22T15:27:08.653 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.598+0000 7fd3fbfff640 1 -- 172.21.15.16:0/3002049849 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3f000bf80 con 0x7fd3f0005180 2023-05-22T15:27:08.653 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.598+0000 7fd408f9d640 1 --2- 172.21.15.16:0/3002049849 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd3f00047d0 0x7fd3f0004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:08.653 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.602+0000 7f6ff6b17640 1 -- 172.21.15.16:0/2386366954 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6ff006f970 msgr2=0x7f6ff006de90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:08.653 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.602+0000 7f6ff6b17640 1 --2- 172.21.15.16:0/2386366954 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6ff006f970 0x7f6ff006de90 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f6fe8009e50 tx=0x7f6fe801c970 comp rx=0 tx=0).stop 2023-05-22T15:27:08.654 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.598+0000 7fd408f9d640 1 -- 172.21.15.16:0/3002049849 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd3f0005180 msgr2=0x7fd3f000b990 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-05-22T15:27:08.654 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.598+0000 7fd408f9d640 1 --2- 172.21.15.16:0/3002049849 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd3f0005180 0x7fd3f000b990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:08.655 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.598+0000 7fd408f9d640 1 -- 172.21.15.16:0/3002049849 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd3f0003980 msgr2=0x7fd3f0003d90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-05-22T15:27:08.655 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.598+0000 7fd408f9d640 1 --2- 172.21.15.16:0/3002049849 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd3f0003980 0x7fd3f0003d90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:08.655 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.598+0000 7fd408f9d640 1 -- 172.21.15.16:0/3002049849 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3f0003530 con 0x7fd3f00047d0 2023-05-22T15:27:08.656 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.602+0000 7f5d175af640 1 -- 172.21.15.16:0/460058672 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5d1006c390 msgr2=0x7f5d100cde10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:08.656 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.602+0000 7f5d175af640 1 --2- 172.21.15.16:0/460058672 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5d1006c390 0x7f5d100cde10 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f5d1006f3d0 tx=0x7f5d0c01c4e0 comp rx=0 tx=0).stop 2023-05-22T15:27:08.657 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.602+0000 7f3ae9c54640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3ae40cc8f0 0x7f3ae40c2d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57810/0 (socket says 172.21.15.16:57810) 2023-05-22T15:27:08.657 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.602+0000 7f3ae9c54640 1 -- 172.21.15.16:0/1408321449 learned_addr learned my addr 172.21.15.16:0/1408321449 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:08.657 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.602+0000 7f3aea455640 1 --2- 172.21.15.16:0/1408321449 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3ae415ff20 0x7f3ae4157ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:08.658 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.602+0000 7f5d175af640 1 -- 172.21.15.16:0/460058672 shutdown_connections 2023-05-22T15:27:08.659 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.602+0000 7f5d175af640 1 --2- 172.21.15.16:0/460058672 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5d1006c390 0x7f5d100cde10 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:08.659 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.602+0000 7f5d175af640 1 --2- 172.21.15.16:0/460058672 >> v2:172.21.15.16:3300/0 conn(0x7f5d1015fb50 0x7f5d1015ff20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:08.659 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.602+0000 7f5d175af640 1 -- 172.21.15.16:0/460058672 >> 172.21.15.16:0/460058672 conn(0x7f5d100655c0 msgr2=0x7f5d10069720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:08.660 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.602+0000 7f6ff6b17640 1 -- 172.21.15.16:0/2386366954 shutdown_connections 2023-05-22T15:27:08.660 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.602+0000 7f6ff6b17640 1 --2- 172.21.15.16:0/2386366954 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6ff006f970 0x7f6ff006de90 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:08.661 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.602+0000 7f6ff6b17640 1 --2- 172.21.15.16:0/2386366954 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6ff00b56a0 0x7f6ff015d7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:08.661 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.602+0000 7f6ff6b17640 1 -- 172.21.15.16:0/2386366954 >> 172.21.15.16:0/2386366954 conn(0x7f6ff00cd850 msgr2=0x7f6ff00d0180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:08.661 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.602+0000 7f6ff6b17640 1 -- 172.21.15.16:0/2386366954 shutdown_connections 2023-05-22T15:27:08.662 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.602+0000 7fc5d5c49640 1 --2- 172.21.15.16:0/148263583 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc5d006c7c0 0x7fc5d00716d0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fc5c8009e80 tx=0x7fc5c801c9e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dde55b2e01f85965 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:08.662 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.602+0000 7f6ff6b17640 1 -- 172.21.15.16:0/2386366954 wait complete. 2023-05-22T15:27:08.663 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.602+0000 7fc005cbc640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fc0000d21e0 0x7fc0000b0a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:08.663 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.602+0000 7fc005cbc640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc0000d25b0 0x7fc0000cfe90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:08.664 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.602+0000 7f3ae9c54640 1 -- 172.21.15.16:0/1408321449 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3ae415fb50 msgr2=0x7f3ae40b0cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:08.664 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.602+0000 7f3ae9c54640 1 --2- 172.21.15.16:0/1408321449 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3ae415fb50 0x7f3ae40b0cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:08.664 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.602+0000 7f3ae9c54640 1 -- 172.21.15.16:0/1408321449 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3ae415ff20 msgr2=0x7f3ae4157ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:08.664 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.602+0000 7f3ae9c54640 1 --2- 172.21.15.16:0/1408321449 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3ae415ff20 0x7f3ae4157ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:08.664 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.602+0000 7f3ae9c54640 1 -- 172.21.15.16:0/1408321449 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ad4009420 con 0x7f3ae40cc8f0 2023-05-22T15:27:08.665 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.602+0000 7fc5d5448640 1 -- 172.21.15.16:0/148263583 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5c80a5670 con 0x7fc5d006c7c0 2023-05-22T15:27:08.665 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.602+0000 7fc005cbc640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0000b0600 con 0x7fc0000d03d0 2023-05-22T15:27:08.665 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.602+0000 7fc005cbc640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc00015c000 con 0x7fc0000d21e0 2023-05-22T15:27:08.666 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.602+0000 7fc005cbc640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0000afa30 con 0x7fc0000d25b0 2023-05-22T15:27:08.666 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.602+0000 7fbfff7fe640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fc0000d21e0 0x7fc0000b0a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:08.666 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.602+0000 7fbfff7fe640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fc0000d21e0 0x7fc0000b0a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.16:37796/0 (socket says 172.21.15.16:37796) 2023-05-22T15:27:08.666 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.602+0000 7fbfff7fe640 1 -- 172.21.15.16:0/2150368962 learned_addr learned my addr 172.21.15.16:0/2150368962 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:08.666 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.602+0000 7fbffeffd640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc0000d25b0 0x7fc0000cfe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:08.667 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.602+0000 7f3ae9c54640 1 --2- 172.21.15.16:0/1408321449 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3ae40cc8f0 0x7f3ae40c2d80 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f3ae000c530 tx=0x7f3ae000c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:08.667 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.602+0000 7f5d175af640 1 -- 172.21.15.16:0/460058672 shutdown_connections 2023-05-22T15:27:08.667 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.602+0000 7fc5d5448640 1 -- 172.21.15.16:0/148263583 <== mon.2 v2:172.21.15.62:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5c80a57d0 con 0x7fc5d006c7c0 2023-05-22T15:27:08.667 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.602+0000 7fc5d5448640 1 -- 172.21.15.16:0/148263583 <== mon.2 v2:172.21.15.62:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5c80a5b60 con 0x7fc5d006c7c0 2023-05-22T15:27:08.668 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.602+0000 7f7282141640 1 Processor -- start 2023-05-22T15:27:08.668 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.602+0000 7f3ada7fc640 1 -- 172.21.15.16:0/1408321449 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ae00074e0 con 0x7f3ae40cc8f0 2023-05-22T15:27:08.668 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.602+0000 7f3ada7fc640 1 -- 172.21.15.16:0/1408321449 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ae0007660 con 0x7f3ae40cc8f0 2023-05-22T15:27:08.668 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.602+0000 7f3ada7fc640 1 -- 172.21.15.16:0/1408321449 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ae0004a30 con 0x7f3ae40cc8f0 2023-05-22T15:27:08.669 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.602+0000 7fc5d744c640 1 -- 172.21.15.16:0/148263583 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc5d006c7c0 msgr2=0x7fc5d00716d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:08.669 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.602+0000 7fc5d744c640 1 --2- 172.21.15.16:0/148263583 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc5d006c7c0 0x7fc5d00716d0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fc5c8009e80 tx=0x7fc5c801c9e0 comp rx=0 tx=0).stop 2023-05-22T15:27:08.669 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.602+0000 7fc5d744c640 1 -- 172.21.15.16:0/148263583 shutdown_connections 2023-05-22T15:27:08.669 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.602+0000 7fc5d744c640 1 --2- 172.21.15.16:0/148263583 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc5d006c7c0 0x7fc5d00716d0 secure :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fc5c8009e80 tx=0x7fc5c801c9e0 comp rx=0 tx=0).stop 2023-05-22T15:27:08.669 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.602+0000 7fc5d744c640 1 -- 172.21.15.16:0/148263583 >> 172.21.15.16:0/148263583 conn(0x7fc5d00af9f0 msgr2=0x7fc5d00afdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:08.670 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.602+0000 7f3621d5c640 1 Processor -- start 2023-05-22T15:27:08.670 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.602+0000 7f3621d5c640 1 -- start start 2023-05-22T15:27:08.671 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.602+0000 7f3aebeb8640 1 -- 172.21.15.16:0/1408321449 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ae40cd2e0 con 0x7f3ae40cc8f0 2023-05-22T15:27:08.671 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.602+0000 7f3aebeb8640 1 -- 172.21.15.16:0/1408321449 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ae41602f0 con 0x7f3ae40cc8f0 2023-05-22T15:27:08.671 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.602+0000 7fbfff7fe640 1 -- 172.21.15.16:0/2150368962 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc0000d25b0 msgr2=0x7fc0000cfe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:08.671 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.602+0000 7fbfff7fe640 1 --2- 172.21.15.16:0/2150368962 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc0000d25b0 0x7fc0000cfe90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:08.671 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.602+0000 7fbfff7fe640 1 -- 172.21.15.16:0/2150368962 >> v1:172.21.15.16:6789/0 conn(0x7fc0000d03d0 legacy=0x7fc0000b1ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:08.671 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.602+0000 7fbfff7fe640 1 -- 172.21.15.16:0/2150368962 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0000afcd0 con 0x7fc0000d21e0 2023-05-22T15:27:08.672 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.602+0000 7f6ff6b17640 1 Processor -- start 2023-05-22T15:27:08.672 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.606+0000 7f3aebeb8640 1 -- 172.21.15.16:0/1408321449 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f3ae40cd470 con 0x7f3ae40cc8f0 2023-05-22T15:27:08.673 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.606+0000 7fc5d744c640 1 -- 172.21.15.16:0/148263583 shutdown_connections 2023-05-22T15:27:08.673 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.602+0000 7f5d175af640 1 -- 172.21.15.16:0/460058672 wait complete. 2023-05-22T15:27:08.673 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.606+0000 7f5d175af640 1 Processor -- start 2023-05-22T15:27:08.674 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.606+0000 7f7282141640 1 -- start start 2023-05-22T15:27:08.674 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.606+0000 7f7282141640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f727c0cc8f0 0x7f727c159460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:08.674 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.606+0000 7f7282141640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f727c15fb50 0x7f727c162880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:08.674 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.606+0000 7f7282141640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f727c15ff20 0x7f727c1fbc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:08.674 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.606+0000 7f7282141640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f727c0c44c0 con 0x7f727c15ff20 2023-05-22T15:27:08.674 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.606+0000 7f7282141640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f727c0afe40 con 0x7f727c15fb50 2023-05-22T15:27:08.674 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.606+0000 7f7282141640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f727c0c3eb0 con 0x7f727c0cc8f0 2023-05-22T15:27:08.675 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.606+0000 7fc5d744c640 1 -- 172.21.15.16:0/148263583 wait complete. 2023-05-22T15:27:08.675 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.606+0000 7f727bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f727c15ff20 0x7f727c1fbc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:08.675 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.606+0000 7f727bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f727c15ff20 0x7f727c1fbc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57814/0 (socket says 172.21.15.16:57814) 2023-05-22T15:27:08.676 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.606+0000 7f727bfff640 1 -- 172.21.15.16:0/442555987 learned_addr learned my addr 172.21.15.16:0/442555987 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:08.676 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.606+0000 7f6ff6b17640 1 -- start start 2023-05-22T15:27:08.676 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.606+0000 7f6ff6b17640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6ff006f970 0x7f6ff00c3cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:08.676 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.606+0000 7f6ff6b17640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6ff00b56a0 0x7f6ff00ce710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:08.676 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.606+0000 7f727affd640 1 --2- 172.21.15.16:0/442555987 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f727c15fb50 0x7f727c162880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:08.677 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.606+0000 7f3621d5c640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f36140e6f00 0x7f361404f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:08.677 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.606+0000 7f3621d5c640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f3614051170 0x7f3614033a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:08.677 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.606+0000 7f3621d5c640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3614037320 con 0x7f3614033f40 2023-05-22T15:27:08.677 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.606+0000 7f3621d5c640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36140343c0 con 0x7f3614051170 2023-05-22T15:27:08.677 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.606+0000 7f3621d5c640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36140367a0 con 0x7f36140e6f00 2023-05-22T15:27:08.678 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.606+0000 7f361b7fe640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f36140e6f00 0x7f361404f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:08.678 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.606+0000 7f361b7fe640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f36140e6f00 0x7f361404f2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3301/0 says I am v2:172.21.15.16:53244/0 (socket says 172.21.15.16:53244) 2023-05-22T15:27:08.678 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.606+0000 7f361b7fe640 1 -- 172.21.15.16:0/4215808034 learned_addr learned my addr 172.21.15.16:0/4215808034 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:08.678 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.606+0000 7f361affd640 1 --2- 172.21.15.16:0/4215808034 >> v2:172.21.15.16:3300/0 conn(0x7f3614051170 0x7f3614033a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:08.678 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.606+0000 7f361a7fc640 1 -- 172.21.15.16:0/4215808034 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3720800506 0 0) 0x7f3614037320 con 0x7f3614033f40 2023-05-22T15:27:08.678 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.606+0000 7f361a7fc640 1 -- 172.21.15.16:0/4215808034 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f360c003530 con 0x7f3614033f40 2023-05-22T15:27:08.679 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.606+0000 7fbfff7fe640 1 --2- 172.21.15.16:0/2150368962 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc0000d21e0 0x7fc0000b0a90 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fbff0006fd0 tx=0x7fbff001c500 comp rx=0 tx=0).ready entity=mon.1 client_cookie=126b09ac18e42443 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:08.679 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.606+0000 7f6ff6b17640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6ff00b5b90 0x7f6ff01514b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:08.679 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.606+0000 7fd408f9d640 1 --2- 172.21.15.16:0/3002049849 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd3f00047d0 0x7fd3f0004c40 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fd3fc00a8a0 tx=0x7fd3fc01fac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7b8b04efd9e1d11 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:08.680 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.606+0000 7fc5d744c640 1 Processor -- start 2023-05-22T15:27:08.680 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.606+0000 7f6ff6b17640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ff006a830 con 0x7f6ff00b56a0 2023-05-22T15:27:08.680 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.606+0000 7f6ff6b17640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ff006aad0 con 0x7f6ff006f970 2023-05-22T15:27:08.680 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.606+0000 7fc5d744c640 1 -- start start 2023-05-22T15:27:08.681 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.606+0000 7f5d175af640 1 -- start start 2023-05-22T15:27:08.681 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.606+0000 7f5d175af640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5d1006bf60 0x7f5d1015aa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:08.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.606+0000 7f361a7fc640 1 -- 172.21.15.16:0/4215808034 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3409943522 0 0) 0x7f360c003530 con 0x7f3614033f40 2023-05-22T15:27:08.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.606+0000 7f361a7fc640 1 -- 172.21.15.16:0/4215808034 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f36140e6f00 msgr2=0x7f361404f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:08.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.606+0000 7f361a7fc640 1 --2- 172.21.15.16:0/4215808034 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f36140e6f00 0x7f361404f2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:08.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.606+0000 7f361a7fc640 1 -- 172.21.15.16:0/4215808034 >> v2:172.21.15.16:3300/0 conn(0x7f3614051170 msgr2=0x7f3614033a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:08.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.606+0000 7f361a7fc640 1 --2- 172.21.15.16:0/4215808034 >> v2:172.21.15.16:3300/0 conn(0x7f3614051170 0x7f3614033a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:08.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.606+0000 7f361a7fc640 1 -- 172.21.15.16:0/4215808034 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3614036a80 con 0x7f3614033f40 2023-05-22T15:27:08.683 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.606+0000 7f6ff6b17640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ff006adb0 con 0x7f6ff00b5b90 2023-05-22T15:27:08.683 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.606+0000 7f727b7fe640 1 --2- 172.21.15.16:0/442555987 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f727c0cc8f0 0x7f727c159460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:08.683 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.606+0000 7f5d175af640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5d1006c390 0x7f5d100c1e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:08.684 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.606+0000 7f6ff6316640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6ff00b5b90 0x7f6ff01514b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:08.684 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.606+0000 7f6ff6316640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6ff00b5b90 0x7f6ff01514b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3301/0 says I am v2:172.21.15.16:53266/0 (socket says 172.21.15.16:53266) 2023-05-22T15:27:08.684 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.606+0000 7f6ff6316640 1 -- 172.21.15.16:0/4050012590 learned_addr learned my addr 172.21.15.16:0/4050012590 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:08.684 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.606+0000 7fbffe7fc640 1 -- 172.21.15.16:0/2150368962 <== mon.1 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbff00ac410 con 0x7fc0000d21e0 2023-05-22T15:27:08.685 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.606+0000 7fd3fbfff640 1 -- 172.21.15.16:0/3002049849 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3fc0aa6f0 con 0x7fd3f00047d0 2023-05-22T15:27:08.685 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.606+0000 7fd3fbfff640 1 -- 172.21.15.16:0/3002049849 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3fc0aa870 con 0x7fd3f00047d0 2023-05-22T15:27:08.685 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.606+0000 7fd3fbfff640 1 -- 172.21.15.16:0/3002049849 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3fc0aaca0 con 0x7fd3f00047d0 2023-05-22T15:27:08.685 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.606+0000 7f3ada7fc640 1 -- 172.21.15.16:0/1408321449 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7f3ae001c3b0 con 0x7f3ae40cc8f0 2023-05-22T15:27:08.686 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.606+0000 7f5d175af640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5d1015fb50 0x7f5d100b22f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:08.686 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.606+0000 7f361a7fc640 1 -- 172.21.15.16:0/4215808034 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7f361c072ac0 con 0x7f3614033f40 2023-05-22T15:27:08.686 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.606+0000 7f361a7fc640 1 -- 172.21.15.16:0/4215808034 >> v1:172.21.15.16:6789/0 conn(0x7f3614033f40 legacy=0x7f3614005f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:08.686 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.606+0000 7f361a7fc640 1 --2- 172.21.15.16:0/4215808034 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f36140e6f00 0x7f360c003980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:08.687 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.606+0000 7f361a7fc640 1 --2- 172.21.15.16:0/4215808034 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f360c003f00 0x7f360c0047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:08.687 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.606+0000 7f361a7fc640 1 --2- 172.21.15.16:0/4215808034 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f360c004d10 0x7f360c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:08.687 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.606+0000 7f361a7fc640 1 -- 172.21.15.16:0/4215808034 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36140343c0 con 0x7f360c003f00 2023-05-22T15:27:08.687 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.606+0000 7f361a7fc640 1 -- 172.21.15.16:0/4215808034 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36140367a0 con 0x7f360c004d10 2023-05-22T15:27:08.687 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.606+0000 7f361a7fc640 1 -- 172.21.15.16:0/4215808034 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f360c00ba00 con 0x7f36140e6f00 2023-05-22T15:27:08.687 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.606+0000 7f6ff5b15640 1 --2- 172.21.15.16:0/4050012590 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6ff006f970 0x7f6ff00c3cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:08.688 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.606+0000 7f6ff5314640 1 --2- 172.21.15.16:0/4050012590 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6ff00b56a0 0x7f6ff00ce710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:08.688 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.606+0000 7f5d175af640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d1015f9b0 con 0x7f5d1015fb50 2023-05-22T15:27:08.688 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.606+0000 7f5d175af640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d1015af40 con 0x7f5d1006bf60 2023-05-22T15:27:08.688 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.606+0000 7f5d175af640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d100ae9b0 con 0x7f5d1006c390 2023-05-22T15:27:08.689 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.606+0000 7fbffe7fc640 1 -- 172.21.15.16:0/2150368962 <== mon.1 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbff00ac570 con 0x7fc0000d21e0 2023-05-22T15:27:08.689 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.606+0000 7fbffe7fc640 1 -- 172.21.15.16:0/2150368962 <== mon.1 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbff00ac900 con 0x7fc0000d21e0 2023-05-22T15:27:08.689 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.606+0000 7f5d14b4a640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5d1006c390 0x7f5d100c1e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:08.689 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.606+0000 7f5d14b4a640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5d1006c390 0x7f5d100c1e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3301/0 says I am v2:172.21.15.16:53286/0 (socket says 172.21.15.16:53286) 2023-05-22T15:27:08.690 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.606+0000 7f5d14b4a640 1 -- 172.21.15.16:0/2710789920 learned_addr learned my addr 172.21.15.16:0/2710789920 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:08.690 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.606+0000 7f6ff5314640 1 -- 172.21.15.16:0/4050012590 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6ff00b5b90 msgr2=0x7f6ff01514b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:08.690 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.606+0000 7f6ff5314640 1 --2- 172.21.15.16:0/4050012590 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6ff00b5b90 0x7f6ff01514b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:08.690 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.606+0000 7f6ff5314640 1 -- 172.21.15.16:0/4050012590 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6ff006f970 msgr2=0x7f6ff00c3cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:08.690 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.606+0000 7f6ff5314640 1 --2- 172.21.15.16:0/4050012590 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6ff006f970 0x7f6ff00c3cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:08.691 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.606+0000 7f6ff5314640 1 -- 172.21.15.16:0/4050012590 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fe8009b30 con 0x7f6ff00b56a0 2023-05-22T15:27:08.691 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.606+0000 7f3ada7fc640 1 --2- 172.21.15.16:0/1408321449 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f3ac8065450 0x7f3ac8067900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:08.691 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.606+0000 7f3ada7fc640 1 -- 172.21.15.16:0/1408321449 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f3ae0082840 con 0x7f3ae40cc8f0 2023-05-22T15:27:08.691 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.606+0000 7f3620afa640 1 --2- 172.21.15.16:0/4215808034 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f360c004d10 0x7f360c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:08.692 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.606+0000 7f3ae9453640 1 --2- 172.21.15.16:0/1408321449 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f3ac8065450 0x7f3ac8067900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:08.692 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.606+0000 7f3ada7fc640 1 --2- 172.21.15.16:0/1408321449 >> [v2:172.21.15.16:6800/37008,v1:172.21.15.16:6801/37008] conn(0x7f3ac806f120 0x7f3ac8071550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:08.692 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.606+0000 7f3ada7fc640 1 -- 172.21.15.16:0/1408321449 --> [v2:172.21.15.16:6800/37008,v1:172.21.15.16:6801/37008] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3ac8071c20 con 0x7f3ac806f120 2023-05-22T15:27:08.692 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.606+0000 7f3ada7fc640 1 -- 172.21.15.16:0/1408321449 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3ae0082bf0 con 0x7f3ae40cc8f0 2023-05-22T15:27:08.692 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.606+0000 7f6ff5314640 1 --2- 172.21.15.16:0/4050012590 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6ff00b56a0 0x7f6ff00ce710 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f6fe0000f70 tx=0x7f6fe000c820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:08.693 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.606+0000 7fd40a7a0640 1 -- 172.21.15.16:0/3002049849 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd3f00047d0 msgr2=0x7fd3f0004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:08.693 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.606+0000 7fd40a7a0640 1 --2- 172.21.15.16:0/3002049849 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd3f00047d0 0x7fd3f0004c40 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fd3fc00a8a0 tx=0x7fd3fc01fac0 comp rx=0 tx=0).stop 2023-05-22T15:27:08.693 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.606+0000 7f5d1534b640 1 --2- 172.21.15.16:0/2710789920 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5d1006bf60 0x7f5d1015aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:08.694 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.606+0000 7fd409f9f640 1 -- 172.21.15.16:0/3002049849 reap_dead start 2023-05-22T15:27:08.694 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.606+0000 7fc5d744c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc5d00cd410 0x7fc5d0159220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:08.694 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.606+0000 7f361b7fe640 1 --2- 172.21.15.16:0/4215808034 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f36140e6f00 0x7f360c003980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:08.695 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.606+0000 7f727affd640 1 -- 172.21.15.16:0/442555987 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f727c0cc8f0 msgr2=0x7f727c159460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:08.695 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.606+0000 7f727affd640 1 --2- 172.21.15.16:0/442555987 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f727c0cc8f0 0x7f727c159460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:08.695 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.606+0000 7f727affd640 1 -- 172.21.15.16:0/442555987 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f727c15ff20 msgr2=0x7f727c1fbc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:08.695 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.606+0000 7f727affd640 1 --2- 172.21.15.16:0/442555987 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f727c15ff20 0x7f727c1fbc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:08.695 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.606+0000 7f727affd640 1 -- 172.21.15.16:0/442555987 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f726c009420 con 0x7f727c15fb50 2023-05-22T15:27:08.695 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.606+0000 7f727b7fe640 1 --2- 172.21.15.16:0/442555987 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f727c0cc8f0 0x7f727c159460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:27:08.696 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.606+0000 7f727affd640 1 --2- 172.21.15.16:0/442555987 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f727c15fb50 0x7f727c162880 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f726c0ac000 tx=0x7f726c019240 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:08.696 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.606+0000 7fc5d744c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc5d0160150 0x7fc5d00b3570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:08.696 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.606+0000 7fd40a7a0640 1 -- 172.21.15.16:0/3002049849 shutdown_connections 2023-05-22T15:27:08.696 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.606+0000 7fd40a7a0640 1 -- 172.21.15.16:0/3002049849 >> 172.21.15.16:0/3002049849 conn(0x7fd4040b0bc0 msgr2=0x7fd4040b0fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:08.697 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.610+0000 7fd40a7a0640 1 -- 172.21.15.16:0/3002049849 shutdown_connections 2023-05-22T15:27:08.697 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.610+0000 7fd40a7a0640 1 -- 172.21.15.16:0/3002049849 wait complete. 2023-05-22T15:27:08.697 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.606+0000 7f3aea455640 1 --2- 172.21.15.16:0/1408321449 >> [v2:172.21.15.16:6800/37008,v1:172.21.15.16:6801/37008] conn(0x7f3ac806f120 0x7f3ac8071550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:08.697 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.610+0000 7f3ae9453640 1 --2- 172.21.15.16:0/1408321449 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f3ac8065450 0x7f3ac8067900 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f3ae415fb20 tx=0x7f3adc0098d0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:08.697 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.610+0000 7f3aea455640 1 --2- 172.21.15.16:0/1408321449 >> [v2:172.21.15.16:6800/37008,v1:172.21.15.16:6801/37008] conn(0x7f3ac806f120 0x7f3ac8071550 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:08.698 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.610+0000 7fc005cbc640 1 -- 172.21.15.16:0/2150368962 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc0000d21e0 msgr2=0x7fc0000b0a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:08.698 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.610+0000 7fc005cbc640 1 --2- 172.21.15.16:0/2150368962 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc0000d21e0 0x7fc0000b0a90 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fbff0006fd0 tx=0x7fbff001c500 comp rx=0 tx=0).stop 2023-05-22T15:27:08.698 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.610+0000 7fc005cbc640 1 -- 172.21.15.16:0/2150368962 shutdown_connections 2023-05-22T15:27:08.698 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.610+0000 7fc005cbc640 1 --2- 172.21.15.16:0/2150368962 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc0000d25b0 0x7fc0000cfe90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:08.698 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.610+0000 7fc005cbc640 1 --2- 172.21.15.16:0/2150368962 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc0000d21e0 0x7fc0000b0a90 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:08.698 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.610+0000 7fc005cbc640 1 -- 172.21.15.16:0/2150368962 >> 172.21.15.16:0/2150368962 conn(0x7fc000071770 msgr2=0x7fc00009b590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:08.699 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.610+0000 7fc5d744c640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc5d0155610 0x7fc5d00c4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:08.699 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.610+0000 7fc5d744c640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5d00b3c50 con 0x7fc5d0160150 2023-05-22T15:27:08.699 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.610+0000 7fc5d744c640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5d00c2900 con 0x7fc5d00cd410 2023-05-22T15:27:08.699 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.610+0000 7fc5d744c640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5d00c2be0 con 0x7fc5d0155610 2023-05-22T15:27:08.699 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.610+0000 7fc5d644a640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc5d00cd410 0x7fc5d0159220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:08.699 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.610+0000 7fc5d644a640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc5d00cd410 0x7fc5d0159220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.16:37826/0 (socket says 172.21.15.16:37826) 2023-05-22T15:27:08.700 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.610+0000 7fc5d644a640 1 -- 172.21.15.16:0/3431175871 learned_addr learned my addr 172.21.15.16:0/3431175871 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:08.700 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.610+0000 7fc5d6c4b640 1 --2- 172.21.15.16:0/3431175871 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc5d0155610 0x7fc5d00c4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:08.700 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.610+0000 7f5d14b4a640 1 -- 172.21.15.16:0/2710789920 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5d1006bf60 msgr2=0x7f5d1015aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:08.700 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.610+0000 7f5d14b4a640 1 --2- 172.21.15.16:0/2710789920 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5d1006bf60 0x7f5d1015aa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:08.700 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.610+0000 7f5d14b4a640 1 -- 172.21.15.16:0/2710789920 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5d1015fb50 msgr2=0x7f5d100b22f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:08.700 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.610+0000 7f5d14b4a640 1 --2- 172.21.15.16:0/2710789920 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5d1015fb50 0x7f5d100b22f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:08.701 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.610+0000 7f5d14b4a640 1 -- 172.21.15.16:0/2710789920 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d0c0094d0 con 0x7f5d1006c390 2023-05-22T15:27:08.701 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.610+0000 7f3620afa640 1 -- 172.21.15.16:0/4215808034 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f36140e6f00 msgr2=0x7f360c003980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:08.701 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.610+0000 7f3620afa640 1 --2- 172.21.15.16:0/4215808034 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f36140e6f00 0x7f360c003980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:08.701 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.610+0000 7f361affd640 1 --2- 172.21.15.16:0/4215808034 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f360c003f00 0x7f360c0047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:08.701 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.610+0000 7f3620afa640 1 -- 172.21.15.16:0/4215808034 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f360c003f00 msgr2=0x7f360c0047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:08.701 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.610+0000 7f3620afa640 1 --2- 172.21.15.16:0/4215808034 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f360c003f00 0x7f360c0047d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:08.702 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.610+0000 7f3620afa640 1 -- 172.21.15.16:0/4215808034 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f360c003530 con 0x7f360c004d10 2023-05-22T15:27:08.702 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.610+0000 7fc005cbc640 1 -- 172.21.15.16:0/2150368962 shutdown_connections 2023-05-22T15:27:08.702 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.610+0000 7fc5d5c49640 1 --2- 172.21.15.16:0/3431175871 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc5d0160150 0x7fc5d00b3570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:08.703 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.610+0000 7fc005cbc640 1 -- 172.21.15.16:0/2150368962 wait complete. 2023-05-22T15:27:08.703 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.610+0000 7f5d14b4a640 1 --2- 172.21.15.16:0/2710789920 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5d1006c390 0x7f5d100c1e50 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f5d0800ba80 tx=0x7f5d0800bf40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:08.703 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.610+0000 7f5d05ffb640 1 -- 172.21.15.16:0/2710789920 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d0800cc00 con 0x7f5d1006c390 2023-05-22T15:27:08.703 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.610+0000 7fc5d5c49640 1 -- 172.21.15.16:0/3431175871 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc5d0155610 msgr2=0x7fc5d00c4f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:08.704 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.610+0000 7fc5d5c49640 1 --2- 172.21.15.16:0/3431175871 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc5d0155610 0x7fc5d00c4f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:08.704 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.610+0000 7fc5d5c49640 1 -- 172.21.15.16:0/3431175871 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc5d00cd410 msgr2=0x7fc5d0159220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:08.704 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.610+0000 7fc5d5c49640 1 --2- 172.21.15.16:0/3431175871 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc5d00cd410 0x7fc5d0159220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:08.704 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.610+0000 7fc5d5c49640 1 -- 172.21.15.16:0/3431175871 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5c8009b30 con 0x7fc5d0160150 2023-05-22T15:27:08.704 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.610+0000 7f3ada7fc640 1 -- 172.21.15.16:0/1408321449 <== osd.1 v2:172.21.15.16:6800/37008 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f3ac8071c20 con 0x7f3ac806f120 2023-05-22T15:27:08.705 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.610+0000 7f6fe67fc640 1 -- 172.21.15.16:0/4050012590 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fe00052d0 con 0x7f6ff00b56a0 2023-05-22T15:27:08.705 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.610+0000 7f6fe67fc640 1 -- 172.21.15.16:0/4050012590 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fe0005430 con 0x7f6ff00b56a0 2023-05-22T15:27:08.705 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.610+0000 7f6fe67fc640 1 -- 172.21.15.16:0/4050012590 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fe0003840 con 0x7f6ff00b56a0 2023-05-22T15:27:08.705 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.610+0000 7f6ff6b17640 1 -- 172.21.15.16:0/4050012590 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ff00d0fb0 con 0x7f6ff00b56a0 2023-05-22T15:27:08.705 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.610+0000 7f6ff6b17640 1 -- 172.21.15.16:0/4050012590 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ff00cc830 con 0x7f6ff00b56a0 2023-05-22T15:27:08.706 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.610+0000 7f7263fff640 1 -- 172.21.15.16:0/442555987 <== mon.1 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f726c0acb70 con 0x7f727c15fb50 2023-05-22T15:27:08.706 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.610+0000 7f7263fff640 1 -- 172.21.15.16:0/442555987 <== mon.1 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f726c0accd0 con 0x7f727c15fb50 2023-05-22T15:27:08.706 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.610+0000 7f7263fff640 1 -- 172.21.15.16:0/442555987 <== mon.1 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f726c003820 con 0x7f727c15fb50 2023-05-22T15:27:08.706 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.610+0000 7fd40a7a0640 1 Processor -- start 2023-05-22T15:27:08.706 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.610+0000 7fd40a7a0640 1 -- start start 2023-05-22T15:27:08.707 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.610+0000 7fd40a7a0640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd3f00047d0 0x7fd404161b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:08.707 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.610+0000 7fd40a7a0640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd404158b90 0x7fd40406e1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:08.707 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.610+0000 7fd40a7a0640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd40406e700 0x7fd404160300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:08.707 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.610+0000 7fd40a7a0640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4040c32f0 con 0x7fd3f00047d0 2023-05-22T15:27:08.707 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.610+0000 7fd40a7a0640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd404162130 con 0x7fd404158b90 2023-05-22T15:27:08.708 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.610+0000 7fd40a7a0640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4040c2ca0 con 0x7fd40406e700 2023-05-22T15:27:08.708 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.610+0000 7fd40979e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd3f00047d0 0x7fd404161b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:08.708 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.610+0000 7fd40979e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd3f00047d0 0x7fd404161b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57864/0 (socket says 172.21.15.16:57864) 2023-05-22T15:27:08.708 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.610+0000 7fd40979e640 1 -- 172.21.15.16:0/1361163610 learned_addr learned my addr 172.21.15.16:0/1361163610 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:08.708 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.614+0000 7fd409f9f640 1 --2- 172.21.15.16:0/1361163610 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd40406e700 0x7fd404160300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:08.708 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.610+0000 7f6ff6b17640 1 -- 172.21.15.16:0/4050012590 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f6ff015dd20 con 0x7f6ff00b56a0 2023-05-22T15:27:08.709 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.614+0000 7f6fe67fc640 1 -- 172.21.15.16:0/4050012590 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7f6fe00039a0 con 0x7f6ff00b56a0 2023-05-22T15:27:08.709 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.614+0000 7fc005cbc640 1 Processor -- start 2023-05-22T15:27:08.709 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.614+0000 7fc005cbc640 1 -- start start 2023-05-22T15:27:08.709 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.610+0000 7f7282141640 1 -- 172.21.15.16:0/442555987 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f727c0cd2e0 con 0x7f727c15fb50 2023-05-22T15:27:08.710 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.610+0000 7f7282141640 1 -- 172.21.15.16:0/442555987 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f727c160350 con 0x7f727c15fb50 2023-05-22T15:27:08.710 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.610+0000 7f7282141640 1 -- 172.21.15.16:0/442555987 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f725c000fd0 con 0x7f727c15fb50 2023-05-22T15:27:08.710 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.610+0000 7fc5d5c49640 1 --2- 172.21.15.16:0/3431175871 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc5d0160150 0x7fc5d00b3570 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fc5c80ab000 tx=0x7fc5c8019780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:08.710 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.610+0000 7fc5d644a640 1 --2- 172.21.15.16:0/3431175871 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc5d00cd410 0x7fc5d0159220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T15:27:08.710 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.610+0000 7fc5c6ffd640 1 -- 172.21.15.16:0/3431175871 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5c80b8670 con 0x7fc5d0160150 2023-05-22T15:27:08.710 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.610+0000 7fc5c6ffd640 1 -- 172.21.15.16:0/3431175871 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5c80b87d0 con 0x7fc5d0160150 2023-05-22T15:27:08.711 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.610+0000 7fc5c6ffd640 1 -- 172.21.15.16:0/3431175871 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5c80b8b40 con 0x7fc5d0160150 2023-05-22T15:27:08.711 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.610+0000 7f5d175af640 1 -- 172.21.15.16:0/2710789920 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d100cd2e0 con 0x7f5d1006c390 2023-05-22T15:27:08.711 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.610+0000 7f5d05ffb640 1 -- 172.21.15.16:0/2710789920 <== mon.2 v2:172.21.15.62:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d0800cd60 con 0x7f5d1006c390 2023-05-22T15:27:08.711 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.610+0000 7f5d05ffb640 1 -- 172.21.15.16:0/2710789920 <== mon.2 v2:172.21.15.62:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d08003bc0 con 0x7f5d1006c390 2023-05-22T15:27:08.711 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.610+0000 7f5d175af640 1 -- 172.21.15.16:0/2710789920 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d100ccac0 con 0x7f5d1006c390 2023-05-22T15:27:08.711 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.610+0000 7f5d175af640 1 -- 172.21.15.16:0/2710789920 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f5d100cd470 con 0x7f5d1006c390 2023-05-22T15:27:08.712 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.610+0000 7f3620afa640 1 --2- 172.21.15.16:0/4215808034 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f360c004d10 0x7f360c005180 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f361c063890 tx=0x7f361c096550 comp rx=0 tx=0).ready entity=mon.1 client_cookie=47df3d6802f54862 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:08.712 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.614+0000 7f361a7fc640 1 -- 172.21.15.16:0/4215808034 <== mon.1 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f361c125640 con 0x7f360c004d10 2023-05-22T15:27:08.712 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.614+0000 7f361a7fc640 1 -- 172.21.15.16:0/4215808034 <== mon.1 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f361c121050 con 0x7f360c004d10 2023-05-22T15:27:08.712 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.614+0000 7f361a7fc640 1 -- 172.21.15.16:0/4215808034 <== mon.1 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f361c125ac0 con 0x7f360c004d10 2023-05-22T15:27:08.713 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.614+0000 7fc005cbc640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc0000d03d0 0x7fc00015f050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:08.713 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.614+0000 7f6fe67fc640 1 --2- 172.21.15.16:0/4050012590 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f6fd4065400 0x7f6fd40678b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:08.713 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.614+0000 7fc005cbc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc0000d21e0 0x7fc0000cb5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:08.714 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.614+0000 7f6fe67fc640 1 -- 172.21.15.16:0/4050012590 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f6fe00820a0 con 0x7f6ff00b56a0 2023-05-22T15:27:08.714 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.614+0000 7fc005cbc640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc0000d25b0 0x7fc000156ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:08.714 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.614+0000 7f3621d5c640 1 -- 172.21.15.16:0/4215808034 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f360c004d10 msgr2=0x7f360c005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:08.714 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.614+0000 7f3621d5c640 1 --2- 172.21.15.16:0/4215808034 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f360c004d10 0x7f360c005180 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f361c063890 tx=0x7f361c096550 comp rx=0 tx=0).stop 2023-05-22T15:27:08.715 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.610+0000 7fc5d744c640 1 -- 172.21.15.16:0/3431175871 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5d00c88c0 con 0x7fc5d0160150 2023-05-22T15:27:08.715 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.610+0000 7fc5d744c640 1 -- 172.21.15.16:0/3431175871 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5d00ccc60 con 0x7fc5d0160150 2023-05-22T15:27:08.715 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.614+0000 7fc5d744c640 1 -- 172.21.15.16:0/3431175871 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fc5d016bf90 con 0x7fc5d0160150 2023-05-22T15:27:08.715 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.614+0000 7fd409f9f640 1 -- 172.21.15.16:0/1361163610 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd404158b90 msgr2=0x7fd40406e1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:08.715 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.614+0000 7fd409f9f640 1 --2- 172.21.15.16:0/1361163610 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd404158b90 0x7fd40406e1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:08.716 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.614+0000 7fd409f9f640 1 -- 172.21.15.16:0/1361163610 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd3f00047d0 msgr2=0x7fd404161b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:08.716 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.614+0000 7fd409f9f640 1 --2- 172.21.15.16:0/1361163610 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd3f00047d0 0x7fd404161b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:08.716 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.614+0000 7fd409f9f640 1 -- 172.21.15.16:0/1361163610 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4040cbec0 con 0x7fd40406e700 2023-05-22T15:27:08.716 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.614+0000 7f6fe67fc640 1 --2- 172.21.15.16:0/4050012590 >> [v2:172.21.15.16:6808/37009,v1:172.21.15.16:6809/37009] conn(0x7f6fd406f050 0x7f6fd4071480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:08.716 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.614+0000 7f6fe67fc640 1 -- 172.21.15.16:0/4050012590 --> [v2:172.21.15.16:6808/37009,v1:172.21.15.16:6809/37009] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6fd4071b50 con 0x7f6fd406f050 2023-05-22T15:27:08.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.614+0000 7f6ff5b15640 1 --2- 172.21.15.16:0/4050012590 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f6fd4065400 0x7f6fd40678b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:08.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.614+0000 7f6ff6316640 1 --2- 172.21.15.16:0/4050012590 >> [v2:172.21.15.16:6808/37009,v1:172.21.15.16:6809/37009] conn(0x7f6fd406f050 0x7f6fd4071480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:08.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.614+0000 7fbfff7fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc0000d03d0 0x7fc00015f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:08.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.614+0000 7fbfff7fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc0000d03d0 0x7fc00015f050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.16:37844/0 (socket says 172.21.15.16:37844) 2023-05-22T15:27:08.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.614+0000 7f7263fff640 1 -- 172.21.15.16:0/442555987 <== mon.1 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7f726c0b3050 con 0x7f727c15fb50 2023-05-22T15:27:08.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.614+0000 7f7263fff640 1 --2- 172.21.15.16:0/442555987 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f7268065450 0x7f7268067900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:08.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.614+0000 7f7263fff640 1 -- 172.21.15.16:0/442555987 <== mon.1 v2:172.21.15.62:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f726c11d320 con 0x7f727c15fb50 2023-05-22T15:27:08.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.614+0000 7f7263fff640 1 --2- 172.21.15.16:0/442555987 >> [v2:172.21.15.62:6816/37353,v1:172.21.15.62:6817/37353] conn(0x7f726806f120 0x7f7268071550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:08.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.614+0000 7f7263fff640 1 -- 172.21.15.16:0/442555987 --> [v2:172.21.15.62:6816/37353,v1:172.21.15.62:6817/37353] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7268071c20 con 0x7f726806f120 2023-05-22T15:27:08.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.614+0000 7f7263fff640 1 -- 172.21.15.16:0/442555987 <== mon.1 v2:172.21.15.62:3300/0 6 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f726c0a8030 con 0x7f727c15fb50 2023-05-22T15:27:08.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.614+0000 7f5d05ffb640 1 -- 172.21.15.16:0/2710789920 <== mon.2 v2:172.21.15.62:3301/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7f5d08014440 con 0x7f5d1006c390 2023-05-22T15:27:08.719 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.614+0000 7f5d05ffb640 1 --2- 172.21.15.16:0/2710789920 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f5cf4065380 0x7f5cf4067830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:08.719 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.614+0000 7f5d05ffb640 1 -- 172.21.15.16:0/2710789920 <== mon.2 v2:172.21.15.62:3301/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f5d08083560 con 0x7f5d1006c390 2023-05-22T15:27:08.719 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.614+0000 7f5d05ffb640 1 --2- 172.21.15.16:0/2710789920 >> [v2:172.21.15.62:6818/37354,v1:172.21.15.62:6820/37354] conn(0x7f5cf406f050 0x7f5cf4071480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:08.719 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.614+0000 7f5d05ffb640 1 -- 172.21.15.16:0/2710789920 --> [v2:172.21.15.62:6818/37354,v1:172.21.15.62:6820/37354] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5cf4071b50 con 0x7f5cf406f050 2023-05-22T15:27:08.719 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.614+0000 7f5d05ffb640 1 -- 172.21.15.16:0/2710789920 <== mon.2 v2:172.21.15.62:3301/0 6 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5d08083910 con 0x7f5d1006c390 2023-05-22T15:27:08.720 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.614+0000 7f5d1534b640 1 --2- 172.21.15.16:0/2710789920 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f5cf4065380 0x7f5cf4067830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:08.720 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.614+0000 7f5d15b4c640 1 --2- 172.21.15.16:0/2710789920 >> [v2:172.21.15.62:6818/37354,v1:172.21.15.62:6820/37354] conn(0x7f5cf406f050 0x7f5cf4071480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:08.720 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.614+0000 7f3620afa640 1 -- 172.21.15.16:0/4215808034 reap_dead start 2023-05-22T15:27:08.720 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.614+0000 7f3621d5c640 1 -- 172.21.15.16:0/4215808034 shutdown_connections 2023-05-22T15:27:08.720 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.614+0000 7f3621d5c640 1 -- 172.21.15.16:0/4215808034 >> 172.21.15.16:0/4215808034 conn(0x7f3614051700 msgr2=0x7f3614051b00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:08.720 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.614+0000 7f3621d5c640 1 -- 172.21.15.16:0/4215808034 shutdown_connections 2023-05-22T15:27:08.721 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.614+0000 7f727bfff640 1 --2- 172.21.15.16:0/442555987 >> [v2:172.21.15.62:6816/37353,v1:172.21.15.62:6817/37353] conn(0x7f726806f120 0x7f7268071550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:08.721 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.614+0000 7f727b7fe640 1 --2- 172.21.15.16:0/442555987 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f7268065450 0x7f7268067900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:08.721 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.614+0000 7fd409f9f640 1 --2- 172.21.15.16:0/1361163610 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd40406e700 0x7fd404160300 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fd40406a800 tx=0x7fd40000d4d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:08.722 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.614+0000 7f6ff6316640 1 --2- 172.21.15.16:0/4050012590 >> [v2:172.21.15.16:6808/37009,v1:172.21.15.16:6809/37009] conn(0x7f6fd406f050 0x7f6fd4071480 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:08.722 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.614+0000 7f5d15b4c640 1 --2- 172.21.15.16:0/2710789920 >> [v2:172.21.15.62:6818/37354,v1:172.21.15.62:6820/37354] conn(0x7f5cf406f050 0x7f5cf4071480 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:08.722 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.614+0000 7fd3f9ffb640 1 -- 172.21.15.16:0/1361163610 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd40000db10 con 0x7fd40406e700 2023-05-22T15:27:08.723 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.618+0000 7f727bfff640 1 --2- 172.21.15.16:0/442555987 >> [v2:172.21.15.62:6816/37353,v1:172.21.15.62:6817/37353] conn(0x7f726806f120 0x7f7268071550 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:08.723 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.614+0000 7fc5c6ffd640 1 -- 172.21.15.16:0/3431175871 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7fc5c80b8ca0 con 0x7fc5d0160150 2023-05-22T15:27:08.723 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.614+0000 7fc5c6ffd640 1 --2- 172.21.15.16:0/3431175871 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fc5b4065480 0x7fc5b4067930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:08.723 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.614+0000 7fc5c6ffd640 1 -- 172.21.15.16:0/3431175871 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6813+0+0 (secure 0 0 0) 0x7fc5c811e210 con 0x7fc5d0160150 2023-05-22T15:27:08.723 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.618+0000 7fc5c6ffd640 1 --2- 172.21.15.16:0/3431175871 >> [v2:172.21.15.62:6800/37352,v1:172.21.15.62:6801/37352] conn(0x7fc5b406f150 0x7fc5b4071580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:08.724 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.614+0000 7f3621d5c640 1 -- 172.21.15.16:0/4215808034 wait complete. 2023-05-22T15:27:08.724 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.618+0000 7fd40a7a0640 1 -- 172.21.15.16:0/1361163610 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4040cbbe0 con 0x7fd40406e700 2023-05-22T15:27:08.724 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.618+0000 7fd40a7a0640 1 -- 172.21.15.16:0/1361163610 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4040cc1a0 con 0x7fd40406e700 2023-05-22T15:27:08.725 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.618+0000 7f3621d5c640 1 Processor -- start 2023-05-22T15:27:08.725 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.618+0000 7fc5c6ffd640 1 -- 172.21.15.16:0/3431175871 --> [v2:172.21.15.62:6800/37352,v1:172.21.15.62:6801/37352] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc5b4071c50 con 0x7fc5b406f150 2023-05-22T15:27:08.725 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.618+0000 7f727b7fe640 1 --2- 172.21.15.16:0/442555987 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f7268065450 0x7f7268067900 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f7270000f70 tx=0x7f7270009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:08.726 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.618+0000 7fd3f9ffb640 1 -- 172.21.15.16:0/1361163610 <== mon.2 v2:172.21.15.62:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd40000dc90 con 0x7fd40406e700 2023-05-22T15:27:08.726 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.618+0000 7fc5c6ffd640 1 -- 172.21.15.16:0/3431175871 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc5c811e5c0 con 0x7fc5d0160150 2023-05-22T15:27:08.726 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.618+0000 7fc5d6c4b640 1 --2- 172.21.15.16:0/3431175871 >> [v2:172.21.15.62:6800/37352,v1:172.21.15.62:6801/37352] conn(0x7fc5b406f150 0x7fc5b4071580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:08.726 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.618+0000 7fc5d644a640 1 --2- 172.21.15.16:0/3431175871 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fc5b4065480 0x7fc5b4067930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:08.727 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.618+0000 7f5d1534b640 1 --2- 172.21.15.16:0/2710789920 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f5cf4065380 0x7f5cf4067830 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f5d00009990 tx=0x7f5d00009110 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:08.727 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.618+0000 7fbffeffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc0000d21e0 0x7fc0000cb5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:08.727 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.618+0000 7fbffeffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc0000d21e0 0x7fc0000cb5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57872/0 (socket says 172.21.15.16:57872) 2023-05-22T15:27:08.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.618+0000 7fbfff7fe640 1 -- 172.21.15.16:0/3895870227 learned_addr learned my addr 172.21.15.16:0/3895870227 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:08.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.618+0000 7fc005cbc640 1 -- 172.21.15.16:0/3895870227 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0000ccfc0 con 0x7fc0000d21e0 2023-05-22T15:27:08.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.618+0000 7fc005cbc640 1 -- 172.21.15.16:0/3895870227 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0000cc9b0 con 0x7fc0000d03d0 2023-05-22T15:27:08.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.618+0000 7fc005cbc640 1 -- 172.21.15.16:0/3895870227 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0000cd5c0 con 0x7fc0000d25b0 2023-05-22T15:27:08.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.618+0000 7f6ff5b15640 1 --2- 172.21.15.16:0/4050012590 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f6fd4065400 0x7f6fd40678b0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f6fec007a60 tx=0x7f6fec008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:08.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.618+0000 7f6fe67fc640 1 -- 172.21.15.16:0/4050012590 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6fe004fab0 con 0x7f6ff00b56a0 2023-05-22T15:27:08.729 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.618+0000 7fbffffff640 1 --2- 172.21.15.16:0/3895870227 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc0000d25b0 0x7fc000156ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:08.729 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.618+0000 7fc5d6c4b640 1 --2- 172.21.15.16:0/3431175871 >> [v2:172.21.15.62:6800/37352,v1:172.21.15.62:6801/37352] conn(0x7fc5b406f150 0x7fc5b4071580 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:08.729 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.618+0000 7f6fe67fc640 1 -- 172.21.15.16:0/4050012590 <== osd.2 v2:172.21.15.16:6808/37009 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f6fd4071b50 con 0x7f6fd406f050 2023-05-22T15:27:08.730 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.618+0000 7fc5d644a640 1 --2- 172.21.15.16:0/3431175871 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fc5b4065480 0x7fc5b4067930 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fc5c0001910 tx=0x7fc5c0009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:08.730 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.618+0000 7fd3f9ffb640 1 -- 172.21.15.16:0/1361163610 <== mon.2 v2:172.21.15.62:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd400013480 con 0x7fd40406e700 2023-05-22T15:27:08.730 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.618+0000 7fd40a7a0640 1 -- 172.21.15.16:0/1361163610 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fd3e4000fd0 con 0x7fd40406e700 2023-05-22T15:27:08.731 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.618+0000 7f3621d5c640 1 -- start start 2023-05-22T15:27:08.731 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.618+0000 7f3621d5c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f360c004d10 0x7f36140485d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:08.731 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.618+0000 7f3621d5c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f36141886a0 0x7f3614185130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:08.731 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.618+0000 7fbffeffd640 1 -- 172.21.15.16:0/3895870227 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc0000d25b0 msgr2=0x7fc000156ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:08.732 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.618+0000 7fbffeffd640 1 --2- 172.21.15.16:0/3895870227 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc0000d25b0 0x7fc000156ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:08.732 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.618+0000 7fbffeffd640 1 -- 172.21.15.16:0/3895870227 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc0000d03d0 msgr2=0x7fc00015f050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:08.732 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.618+0000 7fbffeffd640 1 --2- 172.21.15.16:0/3895870227 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc0000d03d0 0x7fc00015f050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:08.732 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.618+0000 7f361affd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f36141886a0 0x7f3614185130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:08.733 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.618+0000 7f361affd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f36141886a0 0x7f3614185130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57878/0 (socket says 172.21.15.16:57878) 2023-05-22T15:27:08.733 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.618+0000 7f3621d5c640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f361404a4e0 0x7f361404a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:08.733 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.618+0000 7f361b7fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f360c004d10 0x7f36140485d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:08.733 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.618+0000 7fbffeffd640 1 -- 172.21.15.16:0/3895870227 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbff0009480 con 0x7fc0000d21e0 2023-05-22T15:27:08.734 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.618+0000 7f361b7fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f360c004d10 0x7f36140485d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.16:37856/0 (socket says 172.21.15.16:37856) 2023-05-22T15:27:08.734 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.618+0000 7f361affd640 1 -- 172.21.15.16:0/3384785626 learned_addr learned my addr 172.21.15.16:0/3384785626 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:08.734 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.618+0000 7fbffeffd640 1 --2- 172.21.15.16:0/3895870227 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc0000d21e0 0x7fc0000cb5f0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fbff400d5d0 tx=0x7fbff400da90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:08.735 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.618+0000 7f3621d5c640 1 -- 172.21.15.16:0/3384785626 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3614185670 con 0x7f36141886a0 2023-05-22T15:27:08.735 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.618+0000 7f3621d5c640 1 -- 172.21.15.16:0/3384785626 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f361404ae90 con 0x7f360c004d10 2023-05-22T15:27:08.735 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.618+0000 7f3621d5c640 1 -- 172.21.15.16:0/3384785626 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3614188af0 con 0x7f361404a4e0 2023-05-22T15:27:08.735 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.618+0000 7f361affd640 1 -- 172.21.15.16:0/3384785626 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f361404a4e0 msgr2=0x7f361404a950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:08.735 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.618+0000 7f361affd640 1 --2- 172.21.15.16:0/3384785626 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f361404a4e0 0x7f361404a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:08.735 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.618+0000 7f361affd640 1 -- 172.21.15.16:0/3384785626 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f360c004d10 msgr2=0x7f36140485d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:08.735 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.618+0000 7f361affd640 1 --2- 172.21.15.16:0/3384785626 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f360c004d10 0x7f36140485d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:08.736 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.618+0000 7f361affd640 1 -- 172.21.15.16:0/3384785626 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3614006830 con 0x7f36141886a0 2023-05-22T15:27:08.736 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.618+0000 7f5d05ffb640 1 -- 172.21.15.16:0/2710789920 <== osd.6 v2:172.21.15.62:6818/37354 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f5cf4071b50 con 0x7f5cf406f050 2023-05-22T15:27:08.736 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.618+0000 7f7263fff640 1 -- 172.21.15.16:0/442555987 <== osd.4 v2:172.21.15.62:6816/37353 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f7268071c20 con 0x7f726806f120 2023-05-22T15:27:08.737 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.618+0000 7f361affd640 1 --2- 172.21.15.16:0/3384785626 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f36141886a0 0x7f3614185130 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f3608010890 tx=0x7f3608010c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:08.737 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.618+0000 7f3607fff640 1 -- 172.21.15.16:0/3384785626 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3608010ee0 con 0x7f36141886a0 2023-05-22T15:27:08.737 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.618+0000 7f3607fff640 1 -- 172.21.15.16:0/3384785626 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f360800eb60 con 0x7f36141886a0 2023-05-22T15:27:08.737 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.618+0000 7f3607fff640 1 -- 172.21.15.16:0/3384785626 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f360801c400 con 0x7f36141886a0 2023-05-22T15:27:08.738 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.618+0000 7fd3f9ffb640 1 -- 172.21.15.16:0/1361163610 <== mon.2 v2:172.21.15.62:3301/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7fd4000135e0 con 0x7fd40406e700 2023-05-22T15:27:08.738 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.618+0000 7f3621d5c640 1 -- 172.21.15.16:0/3384785626 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36140066a0 con 0x7f36141886a0 2023-05-22T15:27:08.738 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.618+0000 7f3621d5c640 1 -- 172.21.15.16:0/3384785626 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36140069c0 con 0x7f36141886a0 2023-05-22T15:27:08.739 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.622+0000 7fd3f9ffb640 1 --2- 172.21.15.16:0/1361163610 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd3e8065450 0x7fd3e8067900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:08.739 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.622+0000 7fd3f9ffb640 1 -- 172.21.15.16:0/1361163610 <== mon.2 v2:172.21.15.62:3301/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6813+0+0 (secure 0 0 0) 0x7fd400082d20 con 0x7fd40406e700 2023-05-22T15:27:08.739 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.622+0000 7fd3f9ffb640 1 --2- 172.21.15.16:0/1361163610 >> [v2:172.21.15.16:6824/37021,v1:172.21.15.16:6825/37021] conn(0x7fd3e806efb0 0x7fd3e80713e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:08.739 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.622+0000 7fd408f9d640 1 --2- 172.21.15.16:0/1361163610 >> [v2:172.21.15.16:6824/37021,v1:172.21.15.16:6825/37021] conn(0x7fd3e806efb0 0x7fd3e80713e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:08.740 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.622+0000 7fd408f9d640 1 --2- 172.21.15.16:0/1361163610 >> [v2:172.21.15.16:6824/37021,v1:172.21.15.16:6825/37021] conn(0x7fd3e806efb0 0x7fd3e80713e0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:08.740 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.622+0000 7fd3f9ffb640 1 -- 172.21.15.16:0/1361163610 --> [v2:172.21.15.16:6824/37021,v1:172.21.15.16:6825/37021] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd3e8071ab0 con 0x7fd3e806efb0 2023-05-22T15:27:08.740 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.622+0000 7fd3f9ffb640 1 -- 172.21.15.16:0/1361163610 <== mon.2 v2:172.21.15.62:3301/0 6 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd4000830d0 con 0x7fd40406e700 2023-05-22T15:27:08.740 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.622+0000 7fd3f9ffb640 1 -- 172.21.15.16:0/1361163610 <== osd.0 v2:172.21.15.16:6824/37021 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7fd3e8071ab0 con 0x7fd3e806efb0 2023-05-22T15:27:08.741 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.622+0000 7f3607fff640 1 -- 172.21.15.16:0/3384785626 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7f360801c560 con 0x7f36141886a0 2023-05-22T15:27:08.741 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.622+0000 7f3607fff640 1 --2- 172.21.15.16:0/3384785626 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f3600065480 0x7f3600067930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:08.741 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.622+0000 7f3607fff640 1 -- 172.21.15.16:0/3384785626 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f3608082f10 con 0x7f36141886a0 2023-05-22T15:27:08.741 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.622+0000 7fd40979e640 1 --2- 172.21.15.16:0/1361163610 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd3e8065450 0x7fd3e8067900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:08.742 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.622+0000 7fbfe7fff640 1 -- 172.21.15.16:0/3895870227 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbff40125f0 con 0x7fc0000d21e0 2023-05-22T15:27:08.742 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.626+0000 7fbfe7fff640 1 -- 172.21.15.16:0/3895870227 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbff4012750 con 0x7fc0000d21e0 2023-05-22T15:27:08.742 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.622+0000 7f3605ffb640 1 --2- 172.21.15.16:0/3384785626 >> [v2:172.21.15.62:6808/37355,v1:172.21.15.62:6809/37355] conn(0x7f35f40015f0 0x7f35f4003aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:08.742 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.622+0000 7f3605ffb640 1 -- 172.21.15.16:0/3384785626 --> [v2:172.21.15.62:6808/37355,v1:172.21.15.62:6809/37355] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f35f4006b40 con 0x7f35f40015f0 2023-05-22T15:27:08.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.626+0000 7f361b7fe640 1 --2- 172.21.15.16:0/3384785626 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f3600065480 0x7f3600067930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:08.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.626+0000 7fd40979e640 1 --2- 172.21.15.16:0/1361163610 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd3e8065450 0x7fd3e8067900 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fd40406f020 tx=0x7fd3f4008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:08.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.626+0000 7f3620afa640 1 --2- 172.21.15.16:0/3384785626 >> [v2:172.21.15.62:6808/37355,v1:172.21.15.62:6809/37355] conn(0x7f35f40015f0 0x7f35f4003aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:08.744 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.626+0000 7fbfe7fff640 1 -- 172.21.15.16:0/3895870227 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbff40125f0 con 0x7fc0000d21e0 2023-05-22T15:27:08.744 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.626+0000 7fc005cbc640 1 -- 172.21.15.16:0/3895870227 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0000b25e0 con 0x7fc0000d21e0 2023-05-22T15:27:08.744 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.626+0000 7fc005cbc640 1 -- 172.21.15.16:0/3895870227 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0000d0850 con 0x7fc0000d21e0 2023-05-22T15:27:08.744 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.626+0000 7fc005cbc640 1 -- 172.21.15.16:0/3895870227 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fc0000b2770 con 0x7fc0000d21e0 2023-05-22T15:27:08.744 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.626+0000 7f361b7fe640 1 --2- 172.21.15.16:0/3384785626 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f3600065480 0x7f3600067930 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f3610009bd0 tx=0x7f3610009240 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:08.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.626+0000 7f3620afa640 1 --2- 172.21.15.16:0/3384785626 >> [v2:172.21.15.62:6808/37355,v1:172.21.15.62:6809/37355] conn(0x7f35f40015f0 0x7f35f4003aa0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:08.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.626+0000 7fc5c6ffd640 1 -- 172.21.15.16:0/3431175871 <== osd.5 v2:172.21.15.62:6800/37352 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7fc5b4071c50 con 0x7fc5b406f150 2023-05-22T15:27:08.746 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.626+0000 7fbfe7fff640 1 -- 172.21.15.16:0/3895870227 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7fbff40128c0 con 0x7fc0000d21e0 2023-05-22T15:27:08.746 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.630+0000 7fbfe7fff640 1 --2- 172.21.15.16:0/3895870227 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fbfec065440 0x7fbfec0678f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:08.746 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.630+0000 7fbfe7fff640 1 -- 172.21.15.16:0/3895870227 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6813+0+0 (secure 0 0 0) 0x7fbff4083570 con 0x7fc0000d21e0 2023-05-22T15:27:08.746 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.630+0000 7fbfe7fff640 1 --2- 172.21.15.16:0/3895870227 >> [v2:172.21.15.16:6816/37020,v1:172.21.15.16:6817/37020] conn(0x7fbfec06f090 0x7fbfec0714c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:08.746 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.630+0000 7fbfe7fff640 1 -- 172.21.15.16:0/3895870227 --> [v2:172.21.15.16:6816/37020,v1:172.21.15.16:6817/37020] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbfec071b90 con 0x7fbfec06f090 2023-05-22T15:27:08.746 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.630+0000 7fbfe7fff640 1 -- 172.21.15.16:0/3895870227 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbff40125f0 con 0x7fc0000d21e0 2023-05-22T15:27:08.747 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.630+0000 7f3607fff640 1 -- 172.21.15.16:0/3384785626 <== osd.7 v2:172.21.15.62:6808/37355 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f35f4006b40 con 0x7f35f40015f0 2023-05-22T15:27:08.747 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.630+0000 7fbfff7fe640 1 --2- 172.21.15.16:0/3895870227 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fbfec065440 0x7fbfec0678f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:08.747 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.630+0000 7fbffffff640 1 --2- 172.21.15.16:0/3895870227 >> [v2:172.21.15.16:6816/37020,v1:172.21.15.16:6817/37020] conn(0x7fbfec06f090 0x7fbfec0714c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:08.748 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.630+0000 7fbfff7fe640 1 --2- 172.21.15.16:0/3895870227 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fbfec065440 0x7fbfec0678f0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fbff001cda0 tx=0x7fbff00a6040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:08.748 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.630+0000 7fbffffff640 1 --2- 172.21.15.16:0/3895870227 >> [v2:172.21.15.16:6816/37020,v1:172.21.15.16:6817/37020] conn(0x7fbfec06f090 0x7fbfec0714c0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:08.748 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.630+0000 7fbfe7fff640 1 -- 172.21.15.16:0/3895870227 <== osd.3 v2:172.21.15.16:6816/37020 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7fbfec071b90 con 0x7fbfec06f090 2023-05-22T15:27:08.748 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.638+0000 7f7282141640 1 -- 172.21.15.16:0/442555987 --> [v2:172.21.15.62:6816/37353,v1:172.21.15.62:6817/37353] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f725c002cc0 con 0x7f726806f120 2023-05-22T15:27:08.748 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.638+0000 7f7263fff640 1 -- 172.21.15.16:0/442555987 <== osd.4 v2:172.21.15.62:6816/37353 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f725c002cc0 con 0x7f726806f120 2023-05-22T15:27:08.749 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.642+0000 7f3aebeb8640 1 -- 172.21.15.16:0/1408321449 --> [v2:172.21.15.16:6800/37008,v1:172.21.15.16:6801/37008] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f3ae40738f0 con 0x7f3ac806f120 2023-05-22T15:27:08.749 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.642+0000 7f7282141640 1 -- 172.21.15.16:0/442555987 >> [v2:172.21.15.62:6816/37353,v1:172.21.15.62:6817/37353] conn(0x7f726806f120 msgr2=0x7f7268071550 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:08.750 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.642+0000 7f7282141640 1 --2- 172.21.15.16:0/442555987 >> [v2:172.21.15.62:6816/37353,v1:172.21.15.62:6817/37353] conn(0x7f726806f120 0x7f7268071550 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:08.750 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.642+0000 7f7282141640 1 -- 172.21.15.16:0/442555987 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f7268065450 msgr2=0x7f7268067900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:08.750 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.642+0000 7f7282141640 1 --2- 172.21.15.16:0/442555987 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f7268065450 0x7f7268067900 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f7270000f70 tx=0x7f7270009040 comp rx=0 tx=0).stop 2023-05-22T15:27:08.750 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.642+0000 7f7282141640 1 -- 172.21.15.16:0/442555987 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f727c15fb50 msgr2=0x7f727c162880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:08.750 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.642+0000 7f7282141640 1 --2- 172.21.15.16:0/442555987 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f727c15fb50 0x7f727c162880 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f726c0ac000 tx=0x7f726c019240 comp rx=0 tx=0).stop 2023-05-22T15:27:08.750 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.642+0000 7f727bfff640 1 -- 172.21.15.16:0/442555987 reap_dead start 2023-05-22T15:27:08.750 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.642+0000 7f7282141640 1 -- 172.21.15.16:0/442555987 shutdown_connections 2023-05-22T15:27:08.751 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.642+0000 7f7282141640 1 -- 172.21.15.16:0/442555987 >> 172.21.15.16:0/442555987 conn(0x7f727c0655c0 msgr2=0x7f727c069720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:08.751 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.642+0000 7f3ada7fc640 1 -- 172.21.15.16:0/1408321449 <== osd.1 v2:172.21.15.16:6800/37008 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f3ae40738f0 con 0x7f3ac806f120 2023-05-22T15:27:08.751 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.642+0000 7f7282141640 1 -- 172.21.15.16:0/442555987 shutdown_connections 2023-05-22T15:27:08.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.642+0000 7f7282141640 1 -- 172.21.15.16:0/442555987 wait complete. 2023-05-22T15:27:08.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.646+0000 7f6ff6b17640 1 -- 172.21.15.16:0/4050012590 --> [v2:172.21.15.16:6808/37009,v1:172.21.15.16:6809/37009] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f6ff0071e70 con 0x7f6fd406f050 2023-05-22T15:27:08.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.646+0000 7f6fe67fc640 1 -- 172.21.15.16:0/4050012590 <== osd.2 v2:172.21.15.16:6808/37009 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f6ff0071e70 con 0x7f6fd406f050 2023-05-22T15:27:08.753 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.646+0000 7f5d175af640 1 -- 172.21.15.16:0/2710789920 --> [v2:172.21.15.62:6818/37354,v1:172.21.15.62:6820/37354] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f5d100738f0 con 0x7f5cf406f050 2023-05-22T15:27:08.753 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.646+0000 7f5d05ffb640 1 -- 172.21.15.16:0/2710789920 <== osd.6 v2:172.21.15.62:6818/37354 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f5d100738f0 con 0x7f5cf406f050 2023-05-22T15:27:08.753 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.646+0000 7f5d175af640 1 -- 172.21.15.16:0/2710789920 >> [v2:172.21.15.62:6818/37354,v1:172.21.15.62:6820/37354] conn(0x7f5cf406f050 msgr2=0x7f5cf4071480 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:08.753 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.646+0000 7f5d175af640 1 --2- 172.21.15.16:0/2710789920 >> [v2:172.21.15.62:6818/37354,v1:172.21.15.62:6820/37354] conn(0x7f5cf406f050 0x7f5cf4071480 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:08.753 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.646+0000 7f5d175af640 1 -- 172.21.15.16:0/2710789920 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f5cf4065380 msgr2=0x7f5cf4067830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:08.754 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.646+0000 7f5d175af640 1 --2- 172.21.15.16:0/2710789920 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f5cf4065380 0x7f5cf4067830 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f5d00009990 tx=0x7f5d00009110 comp rx=0 tx=0).stop 2023-05-22T15:27:08.754 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.646+0000 7f5d175af640 1 -- 172.21.15.16:0/2710789920 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5d1006c390 msgr2=0x7f5d100c1e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:08.754 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.646+0000 7f5d175af640 1 --2- 172.21.15.16:0/2710789920 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5d1006c390 0x7f5d100c1e50 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f5d0800ba80 tx=0x7f5d0800bf40 comp rx=0 tx=0).stop 2023-05-22T15:27:08.754 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.646+0000 7f5d15b4c640 1 -- 172.21.15.16:0/2710789920 reap_dead start 2023-05-22T15:27:08.754 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.646+0000 7f5d175af640 1 -- 172.21.15.16:0/2710789920 shutdown_connections 2023-05-22T15:27:08.754 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.646+0000 7f5d175af640 1 -- 172.21.15.16:0/2710789920 >> 172.21.15.16:0/2710789920 conn(0x7f5d100655c0 msgr2=0x7f5d10162290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:08.755 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.646+0000 7f5d175af640 1 -- 172.21.15.16:0/2710789920 shutdown_connections 2023-05-22T15:27:08.755 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.646+0000 7f5d175af640 1 -- 172.21.15.16:0/2710789920 wait complete. 2023-05-22T15:27:08.755 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.646+0000 7f6ff6b17640 1 -- 172.21.15.16:0/4050012590 >> [v2:172.21.15.16:6808/37009,v1:172.21.15.16:6809/37009] conn(0x7f6fd406f050 msgr2=0x7f6fd4071480 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:08.756 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.646+0000 7f6ff6b17640 1 --2- 172.21.15.16:0/4050012590 >> [v2:172.21.15.16:6808/37009,v1:172.21.15.16:6809/37009] conn(0x7f6fd406f050 0x7f6fd4071480 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:08.756 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.646+0000 7f6ff6b17640 1 -- 172.21.15.16:0/4050012590 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f6fd4065400 msgr2=0x7f6fd40678b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:08.756 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.646+0000 7f6ff6b17640 1 --2- 172.21.15.16:0/4050012590 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f6fd4065400 0x7f6fd40678b0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f6fec007a60 tx=0x7f6fec008040 comp rx=0 tx=0).stop 2023-05-22T15:27:08.756 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.646+0000 7f6ff6b17640 1 -- 172.21.15.16:0/4050012590 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6ff00b56a0 msgr2=0x7f6ff00ce710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:08.756 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.646+0000 7f6ff6b17640 1 --2- 172.21.15.16:0/4050012590 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6ff00b56a0 0x7f6ff00ce710 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f6fe0000f70 tx=0x7f6fe000c820 comp rx=0 tx=0).stop 2023-05-22T15:27:08.756 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.646+0000 7f6ff6316640 1 -- 172.21.15.16:0/4050012590 reap_dead start 2023-05-22T15:27:08.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.646+0000 7f6ff6b17640 1 -- 172.21.15.16:0/4050012590 shutdown_connections 2023-05-22T15:27:08.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.646+0000 7f6ff6b17640 1 -- 172.21.15.16:0/4050012590 >> 172.21.15.16:0/4050012590 conn(0x7f6ff00cd850 msgr2=0x7f6ff006f260 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:08.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.646+0000 7f6ff6b17640 1 -- 172.21.15.16:0/4050012590 shutdown_connections 2023-05-22T15:27:08.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.646+0000 7f6ff6b17640 1 -- 172.21.15.16:0/4050012590 wait complete. 2023-05-22T15:27:08.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.650+0000 7fd40a7a0640 1 -- 172.21.15.16:0/1361163610 --> [v2:172.21.15.16:6824/37021,v1:172.21.15.16:6825/37021] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fd3e4002cc0 con 0x7fd3e806efb0 2023-05-22T15:27:08.758 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.650+0000 7fd3f9ffb640 1 -- 172.21.15.16:0/1361163610 <== osd.0 v2:172.21.15.16:6824/37021 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fd3e4002cc0 con 0x7fd3e806efb0 2023-05-22T15:27:08.758 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.650+0000 7fd40a7a0640 1 -- 172.21.15.16:0/1361163610 >> [v2:172.21.15.16:6824/37021,v1:172.21.15.16:6825/37021] conn(0x7fd3e806efb0 msgr2=0x7fd3e80713e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:08.758 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.650+0000 7fd40a7a0640 1 --2- 172.21.15.16:0/1361163610 >> [v2:172.21.15.16:6824/37021,v1:172.21.15.16:6825/37021] conn(0x7fd3e806efb0 0x7fd3e80713e0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:08.758 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.650+0000 7fd40a7a0640 1 -- 172.21.15.16:0/1361163610 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd3e8065450 msgr2=0x7fd3e8067900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:08.759 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.650+0000 7fd40a7a0640 1 --2- 172.21.15.16:0/1361163610 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd3e8065450 0x7fd3e8067900 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fd40406f020 tx=0x7fd3f4008040 comp rx=0 tx=0).stop 2023-05-22T15:27:08.759 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.650+0000 7fd40a7a0640 1 -- 172.21.15.16:0/1361163610 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd40406e700 msgr2=0x7fd404160300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:08.759 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.650+0000 7fd40a7a0640 1 --2- 172.21.15.16:0/1361163610 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd40406e700 0x7fd404160300 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fd40406a800 tx=0x7fd40000d4d0 comp rx=0 tx=0).stop 2023-05-22T15:27:08.759 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.650+0000 7fd409f9f640 1 -- 172.21.15.16:0/1361163610 reap_dead start 2023-05-22T15:27:08.759 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.650+0000 7fd40a7a0640 1 -- 172.21.15.16:0/1361163610 shutdown_connections 2023-05-22T15:27:08.759 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.650+0000 7fd40a7a0640 1 -- 172.21.15.16:0/1361163610 >> 172.21.15.16:0/1361163610 conn(0x7fd4040b0bc0 msgr2=0x7fd40406c8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:08.760 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.650+0000 7fd40a7a0640 1 -- 172.21.15.16:0/1361163610 shutdown_connections 2023-05-22T15:27:08.760 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.650+0000 7fd40a7a0640 1 -- 172.21.15.16:0/1361163610 wait complete. 2023-05-22T15:27:08.760 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.654+0000 7fc5d744c640 1 -- 172.21.15.16:0/3431175871 --> [v2:172.21.15.62:6800/37352,v1:172.21.15.62:6801/37352] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fc5d00738f0 con 0x7fc5b406f150 2023-05-22T15:27:08.760 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.654+0000 7fc5c6ffd640 1 -- 172.21.15.16:0/3431175871 <== osd.5 v2:172.21.15.62:6800/37352 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fc5d00738f0 con 0x7fc5b406f150 2023-05-22T15:27:08.761 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.654+0000 7fc5d744c640 1 -- 172.21.15.16:0/3431175871 >> [v2:172.21.15.62:6800/37352,v1:172.21.15.62:6801/37352] conn(0x7fc5b406f150 msgr2=0x7fc5b4071580 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:08.761 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.654+0000 7fc5d744c640 1 --2- 172.21.15.16:0/3431175871 >> [v2:172.21.15.62:6800/37352,v1:172.21.15.62:6801/37352] conn(0x7fc5b406f150 0x7fc5b4071580 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:08.761 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.654+0000 7fc5d744c640 1 -- 172.21.15.16:0/3431175871 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fc5b4065480 msgr2=0x7fc5b4067930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:08.761 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.654+0000 7fc5d744c640 1 --2- 172.21.15.16:0/3431175871 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fc5b4065480 0x7fc5b4067930 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fc5c0001910 tx=0x7fc5c0009040 comp rx=0 tx=0).stop 2023-05-22T15:27:08.762 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.654+0000 7fc5d744c640 1 -- 172.21.15.16:0/3431175871 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc5d0160150 msgr2=0x7fc5d00b3570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:08.762 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.654+0000 7fc5d744c640 1 --2- 172.21.15.16:0/3431175871 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc5d0160150 0x7fc5d00b3570 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fc5c80ab000 tx=0x7fc5c8019780 comp rx=0 tx=0).stop 2023-05-22T15:27:08.762 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.654+0000 7f3605ffb640 1 -- 172.21.15.16:0/3384785626 --> [v2:172.21.15.62:6808/37355,v1:172.21.15.62:6809/37355] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f35f4005bc0 con 0x7f35f40015f0 2023-05-22T15:27:08.762 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.654+0000 7fc5d6c4b640 1 -- 172.21.15.16:0/3431175871 reap_dead start 2023-05-22T15:27:08.763 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.654+0000 7fc5d744c640 1 -- 172.21.15.16:0/3431175871 shutdown_connections 2023-05-22T15:27:08.763 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.654+0000 7fc5d744c640 1 -- 172.21.15.16:0/3431175871 >> 172.21.15.16:0/3431175871 conn(0x7fc5d00af9f0 msgr2=0x7fc5d00afdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:08.763 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.654+0000 7fc5d744c640 1 -- 172.21.15.16:0/3431175871 shutdown_connections 2023-05-22T15:27:08.763 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.654+0000 7fc5d744c640 1 -- 172.21.15.16:0/3431175871 wait complete. 2023-05-22T15:27:08.763 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.658+0000 7f3aebeb8640 1 -- 172.21.15.16:0/1408321449 >> [v2:172.21.15.16:6800/37008,v1:172.21.15.16:6801/37008] conn(0x7f3ac806f120 msgr2=0x7f3ac8071550 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:08.764 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.658+0000 7f3aebeb8640 1 --2- 172.21.15.16:0/1408321449 >> [v2:172.21.15.16:6800/37008,v1:172.21.15.16:6801/37008] conn(0x7f3ac806f120 0x7f3ac8071550 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:08.764 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.658+0000 7f3aebeb8640 1 -- 172.21.15.16:0/1408321449 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f3ac8065450 msgr2=0x7f3ac8067900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:08.764 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.658+0000 7f3aebeb8640 1 --2- 172.21.15.16:0/1408321449 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f3ac8065450 0x7f3ac8067900 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f3ae415fb20 tx=0x7f3adc0098d0 comp rx=0 tx=0).stop 2023-05-22T15:27:08.764 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.658+0000 7f3aebeb8640 1 -- 172.21.15.16:0/1408321449 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3ae40cc8f0 msgr2=0x7f3ae40c2d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:08.764 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.658+0000 7f3aebeb8640 1 --2- 172.21.15.16:0/1408321449 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3ae40cc8f0 0x7f3ae40c2d80 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f3ae000c530 tx=0x7f3ae000c9f0 comp rx=0 tx=0).stop 2023-05-22T15:27:08.764 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.658+0000 7f3aea455640 1 -- 172.21.15.16:0/1408321449 reap_dead start 2023-05-22T15:27:08.765 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.658+0000 7f3aebeb8640 1 -- 172.21.15.16:0/1408321449 shutdown_connections 2023-05-22T15:27:08.765 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.658+0000 7f3aebeb8640 1 -- 172.21.15.16:0/1408321449 >> 172.21.15.16:0/1408321449 conn(0x7f3ae40655c0 msgr2=0x7f3ae40af3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:08.765 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.658+0000 7f3aebeb8640 1 -- 172.21.15.16:0/1408321449 shutdown_connections 2023-05-22T15:27:08.765 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.658+0000 7f3607fff640 1 -- 172.21.15.16:0/3384785626 <== osd.7 v2:172.21.15.62:6808/37355 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f35f4005bc0 con 0x7f35f40015f0 2023-05-22T15:27:08.765 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.658+0000 7fc005cbc640 1 -- 172.21.15.16:0/3895870227 --> [v2:172.21.15.16:6816/37020,v1:172.21.15.16:6817/37020] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fc0000b2770 con 0x7fbfec06f090 2023-05-22T15:27:08.766 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.658+0000 7f3aebeb8640 1 -- 172.21.15.16:0/1408321449 wait complete. 2023-05-22T15:27:08.766 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.658+0000 7fbfe7fff640 1 -- 172.21.15.16:0/3895870227 <== osd.3 v2:172.21.15.16:6816/37020 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fc0000b2770 con 0x7fbfec06f090 2023-05-22T15:27:08.767 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.658+0000 7fc005cbc640 1 -- 172.21.15.16:0/3895870227 >> [v2:172.21.15.16:6816/37020,v1:172.21.15.16:6817/37020] conn(0x7fbfec06f090 msgr2=0x7fbfec0714c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:08.767 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.658+0000 7fc005cbc640 1 --2- 172.21.15.16:0/3895870227 >> [v2:172.21.15.16:6816/37020,v1:172.21.15.16:6817/37020] conn(0x7fbfec06f090 0x7fbfec0714c0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:08.767 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.662+0000 7fc005cbc640 1 -- 172.21.15.16:0/3895870227 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fbfec065440 msgr2=0x7fbfec0678f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:08.767 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.662+0000 7fc005cbc640 1 --2- 172.21.15.16:0/3895870227 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fbfec065440 0x7fbfec0678f0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fbff001cda0 tx=0x7fbff00a6040 comp rx=0 tx=0).stop 2023-05-22T15:27:08.767 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.662+0000 7fc005cbc640 1 -- 172.21.15.16:0/3895870227 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc0000d21e0 msgr2=0x7fc0000cb5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:08.767 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.662+0000 7fc005cbc640 1 --2- 172.21.15.16:0/3895870227 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc0000d21e0 0x7fc0000cb5f0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fbff400d5d0 tx=0x7fbff400da90 comp rx=0 tx=0).stop 2023-05-22T15:27:08.767 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.662+0000 7fbffffff640 1 -- 172.21.15.16:0/3895870227 reap_dead start 2023-05-22T15:27:08.768 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.662+0000 7fc005cbc640 1 -- 172.21.15.16:0/3895870227 shutdown_connections 2023-05-22T15:27:08.768 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.662+0000 7fc005cbc640 1 -- 172.21.15.16:0/3895870227 >> 172.21.15.16:0/3895870227 conn(0x7fc000071770 msgr2=0x7fc0000b4610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:08.768 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.662+0000 7fc005cbc640 1 -- 172.21.15.16:0/3895870227 shutdown_connections 2023-05-22T15:27:08.768 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.662+0000 7fc005cbc640 1 -- 172.21.15.16:0/3895870227 wait complete. 2023-05-22T15:27:08.769 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.662+0000 7f3621d5c640 1 -- 172.21.15.16:0/3384785626 >> [v2:172.21.15.62:6808/37355,v1:172.21.15.62:6809/37355] conn(0x7f35f40015f0 msgr2=0x7f35f4003aa0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:08.770 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.662+0000 7f3621d5c640 1 --2- 172.21.15.16:0/3384785626 >> [v2:172.21.15.62:6808/37355,v1:172.21.15.62:6809/37355] conn(0x7f35f40015f0 0x7f35f4003aa0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:08.770 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.662+0000 7f3621d5c640 1 -- 172.21.15.16:0/3384785626 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f3600065480 msgr2=0x7f3600067930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:08.770 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.662+0000 7f3621d5c640 1 --2- 172.21.15.16:0/3384785626 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f3600065480 0x7f3600067930 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f3610009bd0 tx=0x7f3610009240 comp rx=0 tx=0).stop 2023-05-22T15:27:08.770 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.662+0000 7f3621d5c640 1 -- 172.21.15.16:0/3384785626 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f36141886a0 msgr2=0x7f3614185130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:08.770 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.662+0000 7f3621d5c640 1 --2- 172.21.15.16:0/3384785626 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f36141886a0 0x7f3614185130 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f3608010890 tx=0x7f3608010c90 comp rx=0 tx=0).stop 2023-05-22T15:27:08.771 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.662+0000 7f3620afa640 1 -- 172.21.15.16:0/3384785626 reap_dead start 2023-05-22T15:27:08.771 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.662+0000 7f3621d5c640 1 -- 172.21.15.16:0/3384785626 shutdown_connections 2023-05-22T15:27:08.771 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.662+0000 7f3621d5c640 1 -- 172.21.15.16:0/3384785626 >> 172.21.15.16:0/3384785626 conn(0x7f3614051700 msgr2=0x7f3614055460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:08.771 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.662+0000 7f3621d5c640 1 -- 172.21.15.16:0/3384785626 shutdown_connections 2023-05-22T15:27:08.771 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.662+0000 7f3621d5c640 1 -- 172.21.15.16:0/3384785626 wait complete. 2023-05-22T15:27:08.772 INFO:teuthology.orchestra.run.smithi016.stdout:55834574856 2023-05-22T15:27:08.772 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2023-05-22T15:27:08.773 INFO:teuthology.orchestra.run.smithi016.stdout:55834574857 2023-05-22T15:27:08.773 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-05-22T15:27:08.774 INFO:teuthology.orchestra.run.smithi016.stdout:55834574858 2023-05-22T15:27:08.774 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2023-05-22T15:27:08.775 INFO:teuthology.orchestra.run.smithi016.stdout:55834574858 2023-05-22T15:27:08.776 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-05-22T15:27:08.776 INFO:teuthology.orchestra.run.smithi016.stdout:55834574857 2023-05-22T15:27:08.777 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-05-22T15:27:08.777 INFO:teuthology.orchestra.run.smithi016.stdout:55834574858 2023-05-22T15:27:08.778 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2023-05-22T15:27:08.778 INFO:teuthology.orchestra.run.smithi016.stdout:55834574856 2023-05-22T15:27:08.779 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2023-05-22T15:27:08.780 INFO:teuthology.orchestra.run.smithi016.stdout:55834574857 2023-05-22T15:27:08.780 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2023-05-22T15:27:08.986 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.974+0000 7f4b195d0640 1 Processor -- start 2023-05-22T15:27:08.987 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.978+0000 7f4b195d0640 1 -- start start 2023-05-22T15:27:08.988 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.978+0000 7f4b195d0640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4b140caa30 0x7f4b140caea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:08.988 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.978+0000 7f4b195d0640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f4b14158100 0x7f4b141585a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:08.988 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.978+0000 7f4b195d0640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b140ca270 con 0x7f4b14157d30 2023-05-22T15:27:08.988 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.978+0000 7f4b195d0640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b140c8570 con 0x7f4b14158100 2023-05-22T15:27:08.988 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.978+0000 7f4b195d0640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b140c8850 con 0x7f4b140caa30 2023-05-22T15:27:08.989 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.978+0000 7f4b12ffd640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f4b14157d30 0x7f4b1415c140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:51110/0 (socket says 172.21.15.16:51110) 2023-05-22T15:27:08.989 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.978+0000 7f4b12ffd640 1 -- 172.21.15.16:0/3321959455 learned_addr learned my addr 172.21.15.16:0/3321959455 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:27:08.989 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.978+0000 7f4b127fc640 1 --2- 172.21.15.16:0/3321959455 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4b140caa30 0x7f4b140caea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:08.989 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.978+0000 7f4b137fe640 1 --2- 172.21.15.16:0/3321959455 >> v2:172.21.15.62:3300/0 conn(0x7f4b14158100 0x7f4b141585a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:08.990 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.978+0000 7f4b11ffb640 1 -- 172.21.15.16:0/3321959455 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1706995738 0 0) 0x7f4b140ca270 con 0x7f4b14157d30 2023-05-22T15:27:08.990 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.978+0000 7f4b127fc640 1 -- 172.21.15.16:0/3321959455 >> v2:172.21.15.62:3300/0 conn(0x7f4b14158100 msgr2=0x7f4b141585a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:08.990 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.978+0000 7f4b127fc640 1 --2- 172.21.15.16:0/3321959455 >> v2:172.21.15.62:3300/0 conn(0x7f4b14158100 0x7f4b141585a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:08.990 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.978+0000 7f4b127fc640 1 -- 172.21.15.16:0/3321959455 >> v1:172.21.15.16:6789/0 conn(0x7f4b14157d30 legacy=0x7f4b1415c140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:08.990 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.978+0000 7f4b127fc640 1 -- 172.21.15.16:0/3321959455 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b140ca510 con 0x7f4b140caa30 2023-05-22T15:27:08.991 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.982+0000 7f2f57104640 1 Processor -- start 2023-05-22T15:27:08.991 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.982+0000 7f2f57104640 1 -- start start 2023-05-22T15:27:08.992 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.982+0000 7f4b127fc640 1 --2- 172.21.15.16:0/3321959455 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4b140caa30 0x7f4b140caea0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f4b0c009d90 tx=0x7f4b0c01d1d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dd301c308e58ac8a server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:08.992 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.982+0000 7fdbc97ab640 1 Processor -- start 2023-05-22T15:27:08.993 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.982+0000 7f4b11ffb640 1 -- 172.21.15.16:0/3321959455 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b0c0ac600 con 0x7f4b140caa30 2023-05-22T15:27:08.993 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.982+0000 7f4b11ffb640 1 -- 172.21.15.16:0/3321959455 <== mon.2 v2:172.21.15.62:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b0c0a8050 con 0x7f4b140caa30 2023-05-22T15:27:08.993 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.982+0000 7f4b11ffb640 1 -- 172.21.15.16:0/3321959455 <== mon.2 v2:172.21.15.62:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b0c0ac980 con 0x7f4b140caa30 2023-05-22T15:27:08.993 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.982+0000 7fdbc97ab640 1 -- start start 2023-05-22T15:27:08.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.982+0000 7f4b195d0640 1 -- 172.21.15.16:0/3321959455 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4b140caa30 msgr2=0x7f4b140caea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:08.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.982+0000 7f4b195d0640 1 --2- 172.21.15.16:0/3321959455 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4b140caa30 0x7f4b140caea0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f4b0c009d90 tx=0x7f4b0c01d1d0 comp rx=0 tx=0).stop 2023-05-22T15:27:08.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.982+0000 7f4b195d0640 1 -- 172.21.15.16:0/3321959455 shutdown_connections 2023-05-22T15:27:08.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.982+0000 7f4b195d0640 1 --2- 172.21.15.16:0/3321959455 >> v2:172.21.15.62:3300/0 conn(0x7f4b14158100 0x7f4b141585a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:08.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.982+0000 7f4b195d0640 1 --2- 172.21.15.16:0/3321959455 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4b140caa30 0x7f4b140caea0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:08.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.982+0000 7f4b195d0640 1 -- 172.21.15.16:0/3321959455 >> 172.21.15.16:0/3321959455 conn(0x7f4b140af9f0 msgr2=0x7f4b140afdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:08.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.982+0000 7f4b195d0640 1 -- 172.21.15.16:0/3321959455 shutdown_connections 2023-05-22T15:27:08.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.982+0000 7f2f57104640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2f48007310 0x7f2f4803a940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:08.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.982+0000 7f4b195d0640 1 -- 172.21.15.16:0/3321959455 wait complete. 2023-05-22T15:27:08.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.982+0000 7f2f54ea0640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f2f4803e610 0x7f2f48006b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:51120/0 (socket says 172.21.15.16:51120) 2023-05-22T15:27:08.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.982+0000 7f2f54ea0640 1 -- 172.21.15.16:0/3560237028 learned_addr learned my addr 172.21.15.16:0/3560237028 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:27:08.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.982+0000 7f2f57104640 1 -- 172.21.15.16:0/3560237028 --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f480078f0 con 0x7f2f4803e610 2023-05-22T15:27:08.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.982+0000 7f2f57104640 1 -- 172.21.15.16:0/3560237028 --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f48053510 con 0x7f2f4803ae80 2023-05-22T15:27:08.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.982+0000 7f2f4ffff640 1 --2- 172.21.15.16:0/3560237028 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2f48007310 0x7f2f4803a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:08.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.982+0000 7f2f57104640 1 -- 172.21.15.16:0/3560237028 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f48053db0 con 0x7f2f48007310 2023-05-22T15:27:08.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.982+0000 7f2f4f7fe640 1 -- 172.21.15.16:0/3560237028 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 951859963 0 0) 0x7f2f480078f0 con 0x7f2f4803e610 2023-05-22T15:27:08.998 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.982+0000 7f2f4f7fe640 1 -- 172.21.15.16:0/3560237028 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f40003530 con 0x7f2f4803e610 2023-05-22T15:27:08.998 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.982+0000 7fdbc97ab640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fdbc4159b10 0x7fdbc40cc780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:08.998 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.982+0000 7fdbc97ab640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fdbc406e250 0x7fdbc40b3e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:08.998 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.982+0000 7fdbc97ab640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbc4159ee0 con 0x7fdbc40b5e80 2023-05-22T15:27:08.998 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.982+0000 7fdbc97ab640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdbc40b30b0 con 0x7fdbc4159b10 2023-05-22T15:27:08.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.982+0000 7fdbc97ab640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdbc40b2810 con 0x7fdbc406e250 2023-05-22T15:27:08.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.982+0000 7fdbc2ffd640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fdbc4159b10 0x7fdbc40cc780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:08.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.982+0000 7f67e1b46640 1 Processor -- start 2023-05-22T15:27:08.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.986+0000 7f39f0ede640 1 Processor -- start 2023-05-22T15:27:09.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.982+0000 7fdbc2ffd640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fdbc4159b10 0x7fdbc40cc780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.16:37880/0 (socket says 172.21.15.16:37880) 2023-05-22T15:27:09.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.982+0000 7fdbc2ffd640 1 -- 172.21.15.16:0/1708358436 learned_addr learned my addr 172.21.15.16:0/1708358436 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:09.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.986+0000 7fdbc37fe640 1 --2- 172.21.15.16:0/1708358436 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fdbc406e250 0x7fdbc40b3e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:09.001 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.986+0000 7f2f4f7fe640 1 -- 172.21.15.16:0/3560237028 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3496748139 0 0) 0x7f2f40003530 con 0x7f2f4803e610 2023-05-22T15:27:09.001 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.986+0000 7f2f4f7fe640 1 -- 172.21.15.16:0/3560237028 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2f48007310 msgr2=0x7f2f4803a940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:09.001 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.986+0000 7f2f4f7fe640 1 --2- 172.21.15.16:0/3560237028 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2f48007310 0x7f2f4803a940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:09.001 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.986+0000 7f2f4f7fe640 1 -- 172.21.15.16:0/3560237028 >> v1:172.21.15.62:6789/0 conn(0x7f2f4803ae80 legacy=0x7f2f48056ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:09.001 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.986+0000 7f2f4f7fe640 1 -- 172.21.15.16:0/3560237028 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f480537f0 con 0x7f2f4803e610 2023-05-22T15:27:09.002 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.986+0000 7f2f4f7fe640 1 -- 172.21.15.16:0/3560237028 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7f2f440016d0 con 0x7f2f4803e610 2023-05-22T15:27:09.002 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.986+0000 7f2f4f7fe640 1 -- 172.21.15.16:0/3560237028 >> v1:172.21.15.16:6789/0 conn(0x7f2f4803e610 legacy=0x7f2f48006b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:09.002 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.986+0000 7f2f4f7fe640 1 --2- 172.21.15.16:0/3560237028 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2f400039b0 0x7f2f40003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:09.002 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.986+0000 7f2f4f7fe640 1 --2- 172.21.15.16:0/3560237028 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f400047d0 0x7f2f40004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:09.002 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.986+0000 7f2f4f7fe640 1 --2- 172.21.15.16:0/3560237028 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2f48007310 0x7f2f400051a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:09.002 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.986+0000 7f2f4f7fe640 1 -- 172.21.15.16:0/3560237028 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f4000ba60 con 0x7f2f400047d0 2023-05-22T15:27:09.002 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.986+0000 7f2f4f7fe640 1 -- 172.21.15.16:0/3560237028 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f4000bbd0 con 0x7f2f400039b0 2023-05-22T15:27:09.003 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.986+0000 7f2f4f7fe640 1 -- 172.21.15.16:0/3560237028 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f4000bd90 con 0x7f2f48007310 2023-05-22T15:27:09.003 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.986+0000 7f67e1b46640 1 -- start start 2023-05-22T15:27:09.003 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.986+0000 7f2f4ffff640 1 --2- 172.21.15.16:0/3560237028 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f400047d0 0x7f2f40004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:09.004 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.986+0000 7f2f54ea0640 1 --2- 172.21.15.16:0/3560237028 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2f400039b0 0x7f2f40003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:09.004 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.986+0000 7f2f4ffff640 1 -- 172.21.15.16:0/3560237028 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2f48007310 msgr2=0x7f2f400051a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:09.004 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.986+0000 7f2f4ffff640 1 --2- 172.21.15.16:0/3560237028 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2f48007310 0x7f2f400051a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:09.004 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.986+0000 7f2f4ffff640 1 -- 172.21.15.16:0/3560237028 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2f400039b0 msgr2=0x7f2f40003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:09.004 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.986+0000 7f2f4ffff640 1 --2- 172.21.15.16:0/3560237028 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2f400039b0 0x7f2f40003dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:09.004 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.986+0000 7fdbc2ffd640 1 -- 172.21.15.16:0/1708358436 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fdbc406e250 msgr2=0x7fdbc40b3e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:09.005 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.986+0000 7fdbc2ffd640 1 --2- 172.21.15.16:0/1708358436 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fdbc406e250 0x7fdbc40b3e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:09.005 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.986+0000 7fdbc2ffd640 1 -- 172.21.15.16:0/1708358436 >> v1:172.21.15.62:6789/0 conn(0x7fdbc40b5e80 legacy=0x7fdbc406db40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:09.005 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.986+0000 7fdbc2ffd640 1 -- 172.21.15.16:0/1708358436 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbc40b2af0 con 0x7fdbc4159b10 2023-05-22T15:27:09.005 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.986+0000 7f2f4ffff640 1 -- 172.21.15.16:0/3560237028 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f40003530 con 0x7f2f400047d0 2023-05-22T15:27:09.006 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.986+0000 7f4b195d0640 1 Processor -- start 2023-05-22T15:27:09.006 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.986+0000 7f39f0ede640 1 -- start start 2023-05-22T15:27:09.006 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.986+0000 7f39f0ede640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f39e40e6d20 0x7f39e4033f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:09.007 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.986+0000 7f39eaffd640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f39e4050e10 0x7f39e4036640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:51130/0 (socket says 172.21.15.16:51130) 2023-05-22T15:27:09.007 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.986+0000 7f39eaffd640 1 -- 172.21.15.16:0/3101703712 learned_addr learned my addr 172.21.15.16:0/3101703712 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:27:09.007 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.986+0000 7f39f0ede640 1 -- 172.21.15.16:0/3101703712 --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39e4033ac0 con 0x7f39e4050e10 2023-05-22T15:27:09.007 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.986+0000 7f39eb7fe640 1 --2- 172.21.15.16:0/3101703712 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f39e40e6d20 0x7f39e4033f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:09.007 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.986+0000 7f39f0ede640 1 -- 172.21.15.16:0/3101703712 --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39e404f9b0 con 0x7f39e4036d50 2023-05-22T15:27:09.007 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.986+0000 7f39f0ede640 1 -- 172.21.15.16:0/3101703712 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39e404f210 con 0x7f39e40e6d20 2023-05-22T15:27:09.008 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.986+0000 7f67e1b46640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f67d4036530 0x7f67d4036900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:09.008 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.986+0000 7f67e0b44640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f67d4036530 0x7f67d4036900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:09.008 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.986+0000 7f67e0b44640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f67d4036530 0x7f67d4036900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57902/0 (socket says 172.21.15.16:57902) 2023-05-22T15:27:09.008 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.986+0000 7f67e0b44640 1 -- 172.21.15.16:0/2581386126 learned_addr learned my addr 172.21.15.16:0/2581386126 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:09.009 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.986+0000 7f4b195d0640 1 -- start start 2023-05-22T15:27:09.009 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.986+0000 7f4b195d0640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4b140caa30 0x7f4b140cd460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:09.009 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.986+0000 7f4b195d0640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b14157d30 0x7f4b14143820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:09.009 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.986+0000 7f4b195d0640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4b14158100 0x7f4b14145590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:09.009 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.986+0000 7f4b195d0640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b14143d60 con 0x7f4b14157d30 2023-05-22T15:27:09.009 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.986+0000 7f4b195d0640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b140bce20 con 0x7f4b140caa30 2023-05-22T15:27:09.010 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.986+0000 7f4b195d0640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b14144ab0 con 0x7f4b14158100 2023-05-22T15:27:09.010 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.986+0000 7f67e1b46640 1 --2- 172.21.15.16:0/2581386126 >> v2:172.21.15.62:3300/0 conn(0x7f67d404bb80 0x7f67d404c000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:09.010 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.986+0000 7f4b127fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b14157d30 0x7f4b14143820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:09.011 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.986+0000 7f39ea7fc640 1 -- 172.21.15.16:0/3101703712 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1208336890 0 0) 0x7f39e4033ac0 con 0x7f39e4050e10 2023-05-22T15:27:09.011 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.986+0000 7f67e1b46640 1 -- 172.21.15.16:0/2581386126 --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67d4053cc0 con 0x7f67d4054150 2023-05-22T15:27:09.011 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.986+0000 7f39ea7fc640 1 -- 172.21.15.16:0/3101703712 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39dc003530 con 0x7f39e4050e10 2023-05-22T15:27:09.012 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.986+0000 7f39ea7fc640 1 -- 172.21.15.16:0/3101703712 <== mon.1 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 311122984 0 0) 0x7f39e404f9b0 con 0x7f39e4036d50 2023-05-22T15:27:09.012 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.986+0000 7f39ea7fc640 1 -- 172.21.15.16:0/3101703712 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39e4033ac0 con 0x7f39e4036d50 2023-05-22T15:27:09.012 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.986+0000 7f39eb7fe640 1 -- 172.21.15.16:0/3101703712 >> v1:172.21.15.62:6789/0 conn(0x7f39e4036d50 legacy=0x7f39e4005ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:09.012 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.986+0000 7f39ea7fc640 1 -- 172.21.15.16:0/3101703712 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3445914409 0 0) 0x7f39dc003530 con 0x7f39e4050e10 2023-05-22T15:27:09.012 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.986+0000 7f39eb7fe640 1 -- 172.21.15.16:0/3101703712 >> v1:172.21.15.16:6789/0 conn(0x7f39e4050e10 legacy=0x7f39e4036640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:09.012 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.986+0000 7f39eb7fe640 1 -- 172.21.15.16:0/3101703712 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39e404f3f0 con 0x7f39e40e6d20 2023-05-22T15:27:09.013 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.986+0000 7f4b137fe640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4b14158100 0x7f4b14145590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:09.013 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.986+0000 7f4b127fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b14157d30 0x7f4b14143820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57904/0 (socket says 172.21.15.16:57904) 2023-05-22T15:27:09.013 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.986+0000 7f4b127fc640 1 -- 172.21.15.16:0/2152752373 learned_addr learned my addr 172.21.15.16:0/2152752373 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:09.013 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.986+0000 7f4b12ffd640 1 --2- 172.21.15.16:0/2152752373 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4b140caa30 0x7f4b140cd460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:09.014 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.986+0000 7f67e1345640 1 --2- 172.21.15.16:0/2581386126 >> v2:172.21.15.62:3300/0 conn(0x7f67d404bb80 0x7f67d404c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:09.014 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.986+0000 7f67e1b46640 1 -- 172.21.15.16:0/2581386126 --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67d40dec10 con 0x7f67d4036530 2023-05-22T15:27:09.014 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.986+0000 7f67e1b46640 1 -- 172.21.15.16:0/2581386126 --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67d40530b0 con 0x7f67d404bb80 2023-05-22T15:27:09.014 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.986+0000 7f4b127fc640 1 -- 172.21.15.16:0/2152752373 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4b14158100 msgr2=0x7f4b14145590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:09.014 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.986+0000 7f4b127fc640 1 --2- 172.21.15.16:0/2152752373 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4b14158100 0x7f4b14145590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:09.015 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.986+0000 7f4b127fc640 1 -- 172.21.15.16:0/2152752373 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4b140caa30 msgr2=0x7f4b140cd460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:09.015 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.986+0000 7f4b127fc640 1 --2- 172.21.15.16:0/2152752373 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4b140caa30 0x7f4b140cd460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:09.015 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.986+0000 7f4b127fc640 1 -- 172.21.15.16:0/2152752373 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b0c009a70 con 0x7f4b14157d30 2023-05-22T15:27:09.015 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.986+0000 7f67db7fe640 1 -- 172.21.15.16:0/2581386126 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 983372872 0 0) 0x7f67d4053cc0 con 0x7f67d4054150 2023-05-22T15:27:09.015 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.986+0000 7f67db7fe640 1 -- 172.21.15.16:0/2581386126 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f67cc003530 con 0x7f67d4054150 2023-05-22T15:27:09.016 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.986+0000 7f33ad4ce640 1 Processor -- start 2023-05-22T15:27:09.016 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.986+0000 7f67e0b44640 1 -- 172.21.15.16:0/2581386126 >> v2:172.21.15.62:3300/0 conn(0x7f67d404bb80 msgr2=0x7f67d404c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:09.016 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.986+0000 7f67e0b44640 1 --2- 172.21.15.16:0/2581386126 >> v2:172.21.15.62:3300/0 conn(0x7f67d404bb80 0x7f67d404c000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:09.017 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.986+0000 7f67e0b44640 1 -- 172.21.15.16:0/2581386126 >> v1:172.21.15.16:6789/0 conn(0x7f67d4054150 legacy=0x7f67d40525f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:09.017 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.986+0000 7f67e0b44640 1 -- 172.21.15.16:0/2581386126 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67d4053390 con 0x7f67d4036530 2023-05-22T15:27:09.017 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.986+0000 7f4b127fc640 1 --2- 172.21.15.16:0/2152752373 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b14157d30 0x7f4b14143820 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f4b0c0af040 tx=0x7f4b0c019f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:09.017 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.986+0000 7fdbc2ffd640 1 --2- 172.21.15.16:0/1708358436 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdbc4159b10 0x7fdbc40cc780 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fdbb8009870 tx=0x7fdbb801c510 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9457c6064bb6f516 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:09.018 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.990+0000 7f6158604640 1 Processor -- start 2023-05-22T15:27:09.018 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.986+0000 7f4afb7fe640 1 -- 172.21.15.16:0/2152752373 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b0c0ac890 con 0x7f4b14157d30 2023-05-22T15:27:09.018 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.990+0000 7f4afb7fe640 1 -- 172.21.15.16:0/2152752373 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b0c0ac9f0 con 0x7f4b14157d30 2023-05-22T15:27:09.018 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.990+0000 7f4afb7fe640 1 -- 172.21.15.16:0/2152752373 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b0c0acd80 con 0x7f4b14157d30 2023-05-22T15:27:09.019 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.990+0000 7f4b195d0640 1 -- 172.21.15.16:0/2152752373 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b140cc8c0 con 0x7f4b14157d30 2023-05-22T15:27:09.019 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.990+0000 7f4b195d0640 1 -- 172.21.15.16:0/2152752373 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b140cc730 con 0x7f4b14157d30 2023-05-22T15:27:09.019 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.990+0000 7f2f4ffff640 1 --2- 172.21.15.16:0/3560237028 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f400047d0 0x7f2f40004c40 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f2f3c009ae0 tx=0x7f2f3c01ca90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bdad42957d2a8f77 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:09.019 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.990+0000 7f33ad4ce640 1 -- start start 2023-05-22T15:27:09.020 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.990+0000 7fdbc1ffb640 1 -- 172.21.15.16:0/1708358436 <== mon.1 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbb80aa670 con 0x7fdbc4159b10 2023-05-22T15:27:09.020 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.990+0000 7fdbc1ffb640 1 -- 172.21.15.16:0/1708358436 <== mon.1 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbb80aa7d0 con 0x7fdbc4159b10 2023-05-22T15:27:09.021 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.990+0000 7fdbc97ab640 1 -- 172.21.15.16:0/1708358436 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdbc4159b10 msgr2=0x7fdbc40cc780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:09.021 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.990+0000 7fdbc97ab640 1 --2- 172.21.15.16:0/1708358436 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdbc4159b10 0x7fdbc40cc780 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fdbb8009870 tx=0x7fdbb801c510 comp rx=0 tx=0).stop 2023-05-22T15:27:09.021 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.990+0000 7fdbc97ab640 1 -- 172.21.15.16:0/1708358436 shutdown_connections 2023-05-22T15:27:09.021 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.990+0000 7fdbc97ab640 1 --2- 172.21.15.16:0/1708358436 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fdbc406e250 0x7fdbc40b3e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:09.021 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.990+0000 7fdbc97ab640 1 --2- 172.21.15.16:0/1708358436 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdbc4159b10 0x7fdbc40cc780 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:09.021 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.990+0000 7fdbc97ab640 1 -- 172.21.15.16:0/1708358436 >> 172.21.15.16:0/1708358436 conn(0x7fdbc406f560 msgr2=0x7fdbc406f960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:09.022 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.990+0000 7f6158604640 1 -- start start 2023-05-22T15:27:09.022 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.990+0000 7f6158604640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f61480e3340 0x7f6148051e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:09.022 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.990+0000 7f6158604640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61480df870 con 0x7f61480e3b60 2023-05-22T15:27:09.022 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.990+0000 7f6158604640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61480dfad0 con 0x7f61480e3710 2023-05-22T15:27:09.022 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.990+0000 7f6158604640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6148052570 con 0x7f61480e3340 2023-05-22T15:27:09.023 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.990+0000 7f33ad4ce640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f33a80cf9b0 0x7f33a80ce540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:09.023 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.990+0000 7f33ad4ce640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f33a80b4410 0x7f33a80d02a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:09.023 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.990+0000 7f33ad4ce640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33a80cc8d0 con 0x7f33a80b7a40 2023-05-22T15:27:09.023 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.990+0000 7f33ad4ce640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33a80cd150 con 0x7f33a80cf9b0 2023-05-22T15:27:09.023 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.990+0000 7f33ad4ce640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33a80cfe50 con 0x7f33a80b4410 2023-05-22T15:27:09.024 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.990+0000 7fdbc97ab640 1 -- 172.21.15.16:0/1708358436 shutdown_connections 2023-05-22T15:27:09.024 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.990+0000 7f4b195d0640 1 -- 172.21.15.16:0/2152752373 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b140cca50 con 0x7f4b14157d30 2023-05-22T15:27:09.024 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.990+0000 7fdbc97ab640 1 -- 172.21.15.16:0/1708358436 wait complete. 2023-05-22T15:27:09.025 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.990+0000 7f6155b9f640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f61480e3710 0x7f6148053810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.16:36054/0 (socket says 172.21.15.16:36054) 2023-05-22T15:27:09.025 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.990+0000 7f6155b9f640 1 -- 172.21.15.16:0/1753949711 learned_addr learned my addr 172.21.15.16:0/1753949711 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:27:09.025 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.990+0000 7f61563a0640 1 --2- 172.21.15.16:0/1753949711 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f61480e3340 0x7f6148051e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:09.025 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.990+0000 7f33acccd640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f33a80b4410 0x7f33a80d02a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:09.026 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.990+0000 7f39eb7fe640 1 --2- 172.21.15.16:0/3101703712 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f39e40e6d20 0x7f39e4033f50 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f39e00097f0 tx=0x7f39e001c370 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a6d82ffb5b0e07e2 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:09.026 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.990+0000 7f39ea7fc640 1 -- 172.21.15.16:0/3101703712 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39d8001680 con 0x7f39e40e6d20 2023-05-22T15:27:09.026 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.990+0000 7f39ea7fc640 1 -- 172.21.15.16:0/3101703712 <== mon.2 v2:172.21.15.62:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39e00a7050 con 0x7f39e40e6d20 2023-05-22T15:27:09.026 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.990+0000 7f39ea7fc640 1 -- 172.21.15.16:0/3101703712 <== mon.2 v2:172.21.15.62:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39e00ab600 con 0x7f39e40e6d20 2023-05-22T15:27:09.027 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.990+0000 7f33acccd640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f33a80b4410 0x7f33a80d02a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.16:37922/0 (socket says 172.21.15.16:37922) 2023-05-22T15:27:09.027 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.990+0000 7f33acccd640 1 -- 172.21.15.16:0/45405207 learned_addr learned my addr 172.21.15.16:0/45405207 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:09.027 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.990+0000 7f33a77fe640 1 --2- 172.21.15.16:0/45405207 >> v2:172.21.15.16:3300/0 conn(0x7f33a80cf9b0 0x7f33a80ce540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:09.027 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.990+0000 7f67e0b44640 1 --2- 172.21.15.16:0/2581386126 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f67d4036530 0x7f67d4036900 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f67d4034480 tx=0x7f67d001e6e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=efc4a2966e0dcb15 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:09.028 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.990+0000 7f4afb7fe640 1 -- 172.21.15.16:0/2152752373 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7f4b0c0c3460 con 0x7f4b14157d30 2023-05-22T15:27:09.028 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.990+0000 7f4afb7fe640 1 --2- 172.21.15.16:0/2152752373 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f4b00065400 0x7f4b000678b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:09.028 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.990+0000 7f4afb7fe640 1 -- 172.21.15.16:0/2152752373 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f4b0c11df60 con 0x7f4b14157d30 2023-05-22T15:27:09.028 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.990+0000 7fdbc97ab640 1 Processor -- start 2023-05-22T15:27:09.029 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.990+0000 7f2f4f7fe640 1 -- 172.21.15.16:0/3560237028 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f3c0a6470 con 0x7f2f400047d0 2023-05-22T15:27:09.029 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.990+0000 7f2f4f7fe640 1 -- 172.21.15.16:0/3560237028 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f3c0a65d0 con 0x7f2f400047d0 2023-05-22T15:27:09.029 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.990+0000 7f2f4f7fe640 1 -- 172.21.15.16:0/3560237028 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f3c0a69c0 con 0x7f2f400047d0 2023-05-22T15:27:09.029 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.990+0000 7f2f57104640 1 -- 172.21.15.16:0/3560237028 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f400047d0 msgr2=0x7f2f40004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:09.029 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.990+0000 7f2f57104640 1 --2- 172.21.15.16:0/3560237028 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f400047d0 0x7f2f40004c40 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f2f3c009ae0 tx=0x7f2f3c01ca90 comp rx=0 tx=0).stop 2023-05-22T15:27:09.029 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.990+0000 7f2f556a1640 1 -- 172.21.15.16:0/3560237028 reap_dead start 2023-05-22T15:27:09.030 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.990+0000 7fdbc97ab640 1 -- start start 2023-05-22T15:27:09.030 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.990+0000 7f61563a0640 1 -- 172.21.15.16:0/1753949711 >> v1:172.21.15.62:6789/0 conn(0x7f61480e3710 legacy=0x7f6148053810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:09.030 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.990+0000 7f61563a0640 1 -- 172.21.15.16:0/1753949711 >> v1:172.21.15.16:6789/0 conn(0x7f61480e3b60 legacy=0x7f61480dffb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:09.031 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.990+0000 7f61563a0640 1 -- 172.21.15.16:0/1753949711 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61480509f0 con 0x7f61480e3340 2023-05-22T15:27:09.031 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.994+0000 7f61563a0640 1 --2- 172.21.15.16:0/1753949711 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f61480e3340 0x7f6148051e90 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f6148050be0 tx=0x7f6150095cb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2311cf72c04e35de server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:09.031 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.994+0000 7fdbc97ab640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fdbc406e250 0x7fdbc414b130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:09.031 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.994+0000 7fdbc97ab640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdbc40b5e80 0x7fdbc414bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:09.032 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.994+0000 7f615539e640 1 -- 172.21.15.16:0/1753949711 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f615011ebb0 con 0x7f61480e3340 2023-05-22T15:27:09.032 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.994+0000 7f615539e640 1 -- 172.21.15.16:0/1753949711 <== mon.2 v2:172.21.15.62:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f615011ed10 con 0x7f61480e3340 2023-05-22T15:27:09.033 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.994+0000 7fdbc27fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdbc40b5e80 0x7fdbc414bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:09.033 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.994+0000 7fdbc27fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdbc40b5e80 0x7fdbc414bcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57924/0 (socket says 172.21.15.16:57924) 2023-05-22T15:27:09.033 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.994+0000 7fdbc97ab640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdbc4159b10 0x7fdbc414c230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:09.033 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.994+0000 7fdbc2ffd640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fdbc406e250 0x7fdbc414b130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:09.033 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.994+0000 7fdbc97ab640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbc40cebd0 con 0x7fdbc40b5e80 2023-05-22T15:27:09.033 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.994+0000 7fdbc97ab640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbc414ba50 con 0x7fdbc4159b10 2023-05-22T15:27:09.033 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.994+0000 7fdbc2ffd640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fdbc406e250 0x7fdbc414b130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3301/0 says I am v2:172.21.15.16:53386/0 (socket says 172.21.15.16:53386) 2023-05-22T15:27:09.034 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.994+0000 7fdbc2ffd640 1 -- 172.21.15.16:0/3656687334 learned_addr learned my addr 172.21.15.16:0/3656687334 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:09.034 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.994+0000 7fdbc97ab640 1 -- 172.21.15.16:0/3656687334 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdbc40cee70 con 0x7fdbc406e250 2023-05-22T15:27:09.034 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.994+0000 7fdbc37fe640 1 --2- 172.21.15.16:0/3656687334 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdbc4159b10 0x7fdbc414c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:09.034 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.990+0000 7f33acccd640 1 -- 172.21.15.16:0/45405207 >> v2:172.21.15.16:3300/0 conn(0x7f33a80cf9b0 msgr2=0x7f33a80ce540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:09.034 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.990+0000 7f33acccd640 1 --2- 172.21.15.16:0/45405207 >> v2:172.21.15.16:3300/0 conn(0x7f33a80cf9b0 0x7f33a80ce540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:09.035 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.990+0000 7f33acccd640 1 -- 172.21.15.16:0/45405207 >> v1:172.21.15.16:6789/0 conn(0x7f33a80b7a40 legacy=0x7f33a80b3d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:09.035 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.990+0000 7f33acccd640 1 -- 172.21.15.16:0/45405207 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33a8160480 con 0x7f33a80b4410 2023-05-22T15:27:09.035 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.994+0000 7fdbc27fc640 1 -- 172.21.15.16:0/3656687334 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fdbc406e250 msgr2=0x7fdbc414b130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:09.035 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.994+0000 7fdbc27fc640 1 --2- 172.21.15.16:0/3656687334 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fdbc406e250 0x7fdbc414b130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:09.035 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.994+0000 7fdbc27fc640 1 -- 172.21.15.16:0/3656687334 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdbc4159b10 msgr2=0x7fdbc414c230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:09.036 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.994+0000 7fdbc27fc640 1 --2- 172.21.15.16:0/3656687334 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdbc4159b10 0x7fdbc414c230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:09.036 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.994+0000 7fdbc27fc640 1 -- 172.21.15.16:0/3656687334 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbb80094d0 con 0x7fdbc40b5e80 2023-05-22T15:27:09.036 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.994+0000 7f6158604640 1 -- 172.21.15.16:0/1753949711 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f61480e3340 msgr2=0x7f6148051e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:09.036 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.994+0000 7f6158604640 1 --2- 172.21.15.16:0/1753949711 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f61480e3340 0x7f6148051e90 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f6148050be0 tx=0x7f6150095cb0 comp rx=0 tx=0).stop 2023-05-22T15:27:09.037 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.990+0000 7f2f57104640 1 -- 172.21.15.16:0/3560237028 shutdown_connections 2023-05-22T15:27:09.037 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.990+0000 7f2f57104640 1 -- 172.21.15.16:0/3560237028 >> 172.21.15.16:0/3560237028 conn(0x7f2f48055940 msgr2=0x7f2f48055d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:09.037 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.994+0000 7f2f57104640 1 -- 172.21.15.16:0/3560237028 shutdown_connections 2023-05-22T15:27:09.037 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.994+0000 7f2f57104640 1 -- 172.21.15.16:0/3560237028 wait complete. 2023-05-22T15:27:09.037 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.994+0000 7fdbc27fc640 1 --2- 172.21.15.16:0/3656687334 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdbc40b5e80 0x7fdbc414bcf0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fdbb4000f70 tx=0x7fdbb400c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:09.038 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.990+0000 7f39f0ede640 1 -- 172.21.15.16:0/3101703712 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f39e40e6d20 msgr2=0x7f39e4033f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:09.038 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.990+0000 7f39f0ede640 1 --2- 172.21.15.16:0/3101703712 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f39e40e6d20 0x7f39e4033f50 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f39e00097f0 tx=0x7f39e001c370 comp rx=0 tx=0).stop 2023-05-22T15:27:09.038 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.990+0000 7f39f0ede640 1 -- 172.21.15.16:0/3101703712 shutdown_connections 2023-05-22T15:27:09.038 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.990+0000 7f39f0ede640 1 --2- 172.21.15.16:0/3101703712 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f39e40e6d20 0x7f39e4033f50 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:09.038 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.990+0000 7f39f0ede640 1 -- 172.21.15.16:0/3101703712 >> 172.21.15.16:0/3101703712 conn(0x7f39e4051640 msgr2=0x7f39e4051a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:09.039 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.994+0000 7f4b12ffd640 1 --2- 172.21.15.16:0/2152752373 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f4b00065400 0x7f4b000678b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:09.039 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.994+0000 7f33acccd640 1 --2- 172.21.15.16:0/45405207 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f33a80b4410 0x7f33a80d02a0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f33a80cec90 tx=0x7f33a001c990 comp rx=0 tx=0).ready entity=mon.1 client_cookie=47fa29fdd1ce2c34 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:09.040 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.994+0000 7f33a6ffd640 1 -- 172.21.15.16:0/45405207 <== mon.1 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33a00a5740 con 0x7f33a80b4410 2023-05-22T15:27:09.040 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.994+0000 7f33a6ffd640 1 -- 172.21.15.16:0/45405207 <== mon.1 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33a00a58a0 con 0x7f33a80b4410 2023-05-22T15:27:09.040 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.994+0000 7f4b12ffd640 1 --2- 172.21.15.16:0/2152752373 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f4b00065400 0x7f4b000678b0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f4b04007fd0 tx=0x7f4b04007d60 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:09.041 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.994+0000 7f39f0ede640 1 -- 172.21.15.16:0/3101703712 shutdown_connections 2023-05-22T15:27:09.041 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.994+0000 7f39f0ede640 1 -- 172.21.15.16:0/3101703712 wait complete. 2023-05-22T15:27:09.041 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.994+0000 7f6158604640 1 -- 172.21.15.16:0/1753949711 shutdown_connections 2023-05-22T15:27:09.042 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.994+0000 7f6158604640 1 --2- 172.21.15.16:0/1753949711 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f61480e3340 0x7f6148051e90 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:09.042 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.994+0000 7f6158604640 1 -- 172.21.15.16:0/1753949711 >> 172.21.15.16:0/1753949711 conn(0x7f61480526c0 msgr2=0x7f6148052ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:09.042 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.994+0000 7f33ad4ce640 1 -- 172.21.15.16:0/45405207 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f33a80b4410 msgr2=0x7f33a80d02a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:09.042 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.994+0000 7f33ad4ce640 1 --2- 172.21.15.16:0/45405207 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f33a80b4410 0x7f33a80d02a0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f33a80cec90 tx=0x7f33a001c990 comp rx=0 tx=0).stop 2023-05-22T15:27:09.042 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.994+0000 7f33ad4ce640 1 -- 172.21.15.16:0/45405207 shutdown_connections 2023-05-22T15:27:09.043 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.994+0000 7f33ad4ce640 1 --2- 172.21.15.16:0/45405207 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f33a80b4410 0x7f33a80d02a0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:09.043 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.994+0000 7f33ad4ce640 1 --2- 172.21.15.16:0/45405207 >> v2:172.21.15.16:3300/0 conn(0x7f33a80cf9b0 0x7f33a80ce540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:09.043 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.994+0000 7f33ad4ce640 1 -- 172.21.15.16:0/45405207 >> 172.21.15.16:0/45405207 conn(0x7f33a8071a40 msgr2=0x7f33a80ced00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:09.043 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.994+0000 7f6158604640 1 -- 172.21.15.16:0/1753949711 shutdown_connections 2023-05-22T15:27:09.044 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.994+0000 7f2f57104640 1 Processor -- start 2023-05-22T15:27:09.044 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.994+0000 7f2f57104640 1 -- start start 2023-05-22T15:27:09.044 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.994+0000 7f33ad4ce640 1 -- 172.21.15.16:0/45405207 shutdown_connections 2023-05-22T15:27:09.044 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.994+0000 7f33ad4ce640 1 -- 172.21.15.16:0/45405207 wait complete. 2023-05-22T15:27:09.045 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.998+0000 7f67db7fe640 1 -- 172.21.15.16:0/2581386126 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67d00aa630 con 0x7f67d4036530 2023-05-22T15:27:09.045 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.998+0000 7f67db7fe640 1 -- 172.21.15.16:0/2581386126 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67d00aa790 con 0x7f67d4036530 2023-05-22T15:27:09.045 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.994+0000 7f6158604640 1 -- 172.21.15.16:0/1753949711 wait complete. 2023-05-22T15:27:09.046 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.998+0000 7f6158604640 1 Processor -- start 2023-05-22T15:27:09.046 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.998+0000 7f67db7fe640 1 -- 172.21.15.16:0/2581386126 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67d00aab00 con 0x7f67d4036530 2023-05-22T15:27:09.046 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.998+0000 7f39f0ede640 1 Processor -- start 2023-05-22T15:27:09.047 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.998+0000 7f4afb7fe640 1 -- 172.21.15.16:0/2152752373 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f4b0c0eb870 con 0x7f4b14157d30 2023-05-22T15:27:09.048 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.998+0000 7f2f57104640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f48180050 0x7f2f480e4a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:09.048 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.998+0000 7f2f57104640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2f480e2410 0x7f2f48180420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:09.048 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.998+0000 7f2f57104640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2f48180960 0x7f2f48050090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:09.048 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.998+0000 7f2f57104640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f480e3570 con 0x7f2f48180050 2023-05-22T15:27:09.048 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.998+0000 7f2f57104640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f480e2960 con 0x7f2f48180960 2023-05-22T15:27:09.048 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.998+0000 7f2f57104640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f480e2c40 con 0x7f2f480e2410 2023-05-22T15:27:09.049 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.998+0000 7f2f556a1640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2f48180960 0x7f2f48050090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:09.049 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.998+0000 7f2f54ea0640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f48180050 0x7f2f480e4a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:09.049 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.998+0000 7f2f54ea0640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f48180050 0x7f2f480e4a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57936/0 (socket says 172.21.15.16:57936) 2023-05-22T15:27:09.049 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.998+0000 7f2f54ea0640 1 -- 172.21.15.16:0/641126305 learned_addr learned my addr 172.21.15.16:0/641126305 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:09.049 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.998+0000 7f2f4ffff640 1 --2- 172.21.15.16:0/641126305 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2f480e2410 0x7f2f48180420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:09.049 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.998+0000 7f33ad4ce640 1 Processor -- start 2023-05-22T15:27:09.050 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.998+0000 7fdbab7fe640 1 -- 172.21.15.16:0/3656687334 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbb40052d0 con 0x7fdbc40b5e80 2023-05-22T15:27:09.050 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.998+0000 7fdbab7fe640 1 -- 172.21.15.16:0/3656687334 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbb4005430 con 0x7fdbc40b5e80 2023-05-22T15:27:09.050 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.998+0000 7fdbab7fe640 1 -- 172.21.15.16:0/3656687334 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbb4003840 con 0x7fdbc40b5e80 2023-05-22T15:27:09.050 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.998+0000 7f2f4ffff640 1 -- 172.21.15.16:0/641126305 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2f48180960 msgr2=0x7f2f48050090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:09.051 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.998+0000 7f2f4ffff640 1 --2- 172.21.15.16:0/641126305 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2f48180960 0x7f2f48050090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:09.051 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.998+0000 7f2f4ffff640 1 -- 172.21.15.16:0/641126305 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f48180050 msgr2=0x7f2f480e4a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:09.051 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.998+0000 7f2f4ffff640 1 --2- 172.21.15.16:0/641126305 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f48180050 0x7f2f480e4a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:09.051 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.998+0000 7f2f4ffff640 1 -- 172.21.15.16:0/641126305 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f48057e00 con 0x7f2f480e2410 2023-05-22T15:27:09.051 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.998+0000 7f6158604640 1 -- start start 2023-05-22T15:27:09.051 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.998+0000 7f6158604640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f61480e3340 0x7f6148049000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:09.052 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.998+0000 7f6158604640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f61480e3710 0x7f6148034980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:09.052 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.998+0000 7f6158604640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f61480e3b60 0x7f61480d69b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:09.052 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.998+0000 7f6158604640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61480d09c0 con 0x7f61480e3340 2023-05-22T15:27:09.052 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.998+0000 7f6158604640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f614804b460 con 0x7f61480e3710 2023-05-22T15:27:09.052 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.998+0000 7f6158604640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6148034ec0 con 0x7f61480e3b60 2023-05-22T15:27:09.052 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.998+0000 7f67e1b46640 1 -- 172.21.15.16:0/2581386126 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f67d4036530 msgr2=0x7f67d4036900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:09.053 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.998+0000 7f67e1b46640 1 --2- 172.21.15.16:0/2581386126 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f67d4036530 0x7f67d4036900 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f67d4034480 tx=0x7f67d001e6e0 comp rx=0 tx=0).stop 2023-05-22T15:27:09.053 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.998+0000 7f67e1b46640 1 -- 172.21.15.16:0/2581386126 shutdown_connections 2023-05-22T15:27:09.053 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.998+0000 7f67e1b46640 1 --2- 172.21.15.16:0/2581386126 >> v2:172.21.15.62:3300/0 conn(0x7f67d404bb80 0x7f67d404c000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:09.053 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.998+0000 7f67e1b46640 1 --2- 172.21.15.16:0/2581386126 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f67d4036530 0x7f67d4036900 secure :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f67d4034480 tx=0x7f67d001e6e0 comp rx=0 tx=0).stop 2023-05-22T15:27:09.053 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.998+0000 7f67e1b46640 1 -- 172.21.15.16:0/2581386126 >> 172.21.15.16:0/2581386126 conn(0x7f67d4051530 msgr2=0x7f67d4051930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:09.054 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.998+0000 7f2f4ffff640 1 --2- 172.21.15.16:0/641126305 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2f480e2410 0x7f2f48180420 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f2f3c008000 tx=0x7f2f3c019830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:09.054 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.998+0000 7f2f4d7fa640 1 -- 172.21.15.16:0/641126305 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f3c0a6c10 con 0x7f2f480e2410 2023-05-22T15:27:09.054 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.998+0000 7f2f57104640 1 -- 172.21.15.16:0/641126305 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f4803b2a0 con 0x7f2f480e2410 2023-05-22T15:27:09.054 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.998+0000 7f2f57104640 1 -- 172.21.15.16:0/641126305 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f48057f90 con 0x7f2f480e2410 2023-05-22T15:27:09.054 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.998+0000 7f2f4d7fa640 1 -- 172.21.15.16:0/641126305 <== mon.2 v2:172.21.15.62:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f3c0a6d70 con 0x7f2f480e2410 2023-05-22T15:27:09.054 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.998+0000 7f2f4d7fa640 1 -- 172.21.15.16:0/641126305 <== mon.2 v2:172.21.15.62:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f3c0bc690 con 0x7f2f480e2410 2023-05-22T15:27:09.055 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.998+0000 7f33ad4ce640 1 -- start start 2023-05-22T15:27:09.055 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.998+0000 7f33ad4ce640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f33a80b4410 0x7f33a814d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:09.055 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.998+0000 7f33ad4ce640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f33a80b7a40 0x7f33a815a730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:09.055 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.998+0000 7f33ad4ce640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f33a8156f90 0x7f33a81500a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:09.055 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.998+0000 7f33ad4ce640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33a81519b0 con 0x7f33a8156f90 2023-05-22T15:27:09.056 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.998+0000 7f33ad4ce640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33a815a230 con 0x7f33a80b7a40 2023-05-22T15:27:09.056 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.998+0000 7f33ad4ce640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33a80cc1d0 con 0x7f33a80b4410 2023-05-22T15:27:09.056 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.998+0000 7f33a77fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f33a80b7a40 0x7f33a815a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:09.056 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.998+0000 7f33a77fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f33a80b7a40 0x7f33a815a730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.16:37946/0 (socket says 172.21.15.16:37946) 2023-05-22T15:27:09.056 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.998+0000 7f33a77fe640 1 -- 172.21.15.16:0/519042481 learned_addr learned my addr 172.21.15.16:0/519042481 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:09.056 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.998+0000 7f33a7fff640 1 --2- 172.21.15.16:0/519042481 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f33a80b4410 0x7f33a814d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:09.057 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.998+0000 7fdbc97ab640 1 -- 172.21.15.16:0/3656687334 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdbc40b4b80 con 0x7fdbc40b5e80 2023-05-22T15:27:09.057 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.998+0000 7fdbc97ab640 1 -- 172.21.15.16:0/3656687334 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdbc40b3910 con 0x7fdbc40b5e80 2023-05-22T15:27:09.057 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.998+0000 7fdbc97ab640 1 -- 172.21.15.16:0/3656687334 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdba4005260 con 0x7fdbc40b5e80 2023-05-22T15:27:09.057 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.998+0000 7f39f0ede640 1 -- start start 2023-05-22T15:27:09.057 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.998+0000 7f39f0ede640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f39e4036d50 0x7f39e404a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:09.058 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.998+0000 7f39f0ede640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f39e4050e10 0x7f39e40352a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:09.058 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.998+0000 7f39f0ede640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f39e40e6d20 0x7f39e4035ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:09.058 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.998+0000 7f39f0ede640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39e4036420 con 0x7f39e40e6d20 2023-05-22T15:27:09.058 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.998+0000 7f39f0ede640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39e40498f0 con 0x7f39e4050e10 2023-05-22T15:27:09.058 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.998+0000 7f39f0ede640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39e4049b90 con 0x7f39e4036d50 2023-05-22T15:27:09.058 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.998+0000 7f6155b9f640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f61480e3710 0x7f6148034980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:09.059 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.998+0000 7f6156ba1640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f61480e3b60 0x7f61480d69b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:09.059 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.998+0000 7f6156ba1640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f61480e3b60 0x7f61480d69b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3301/0 says I am v2:172.21.15.16:53412/0 (socket says 172.21.15.16:53412) 2023-05-22T15:27:09.059 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.998+0000 7f6156ba1640 1 -- 172.21.15.16:0/3342720135 learned_addr learned my addr 172.21.15.16:0/3342720135 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:09.059 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.998+0000 7f61563a0640 1 --2- 172.21.15.16:0/3342720135 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f61480e3340 0x7f6148049000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:09.059 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.998+0000 7f6156ba1640 1 -- 172.21.15.16:0/3342720135 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f61480e3710 msgr2=0x7f6148034980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:09.059 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.998+0000 7f6156ba1640 1 --2- 172.21.15.16:0/3342720135 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f61480e3710 0x7f6148034980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:09.060 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.998+0000 7f6156ba1640 1 -- 172.21.15.16:0/3342720135 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f61480e3340 msgr2=0x7f6148049000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:09.060 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.998+0000 7f6156ba1640 1 --2- 172.21.15.16:0/3342720135 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f61480e3340 0x7f6148049000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:09.060 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.998+0000 7f6156ba1640 1 -- 172.21.15.16:0/3342720135 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6150061440 con 0x7f61480e3b60 2023-05-22T15:27:09.060 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:08.998+0000 7f6156ba1640 1 --2- 172.21.15.16:0/3342720135 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f61480e3b60 0x7f61480d69b0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f614000b410 tx=0x7f614000b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:09.060 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.002+0000 7f2f57104640 1 -- 172.21.15.16:0/641126305 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f30005260 con 0x7f2f480e2410 2023-05-22T15:27:09.061 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.002+0000 7f33a7fff640 1 -- 172.21.15.16:0/519042481 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f33a80b7a40 msgr2=0x7f33a815a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:09.061 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.002+0000 7f33a7fff640 1 --2- 172.21.15.16:0/519042481 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f33a80b7a40 0x7f33a815a730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:09.061 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.002+0000 7f33a7fff640 1 -- 172.21.15.16:0/519042481 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f33a8156f90 msgr2=0x7f33a81500a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:09.061 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.002+0000 7f33a7fff640 1 --2- 172.21.15.16:0/519042481 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f33a8156f90 0x7f33a81500a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:09.061 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.002+0000 7f33a7fff640 1 -- 172.21.15.16:0/519042481 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33a0009b30 con 0x7f33a80b4410 2023-05-22T15:27:09.062 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.002+0000 7f33a7fff640 1 --2- 172.21.15.16:0/519042481 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f33a80b4410 0x7f33a814d270 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f339c00c410 tx=0x7f339c00c8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:09.062 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.002+0000 7f39ebfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f39e40e6d20 0x7f39e4035ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:09.062 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.002+0000 7f39ebfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f39e40e6d20 0x7f39e4035ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57964/0 (socket says 172.21.15.16:57964) 2023-05-22T15:27:09.062 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.002+0000 7f39ebfff640 1 -- 172.21.15.16:0/395007902 learned_addr learned my addr 172.21.15.16:0/395007902 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:09.063 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.002+0000 7fdbab7fe640 1 -- 172.21.15.16:0/3656687334 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7fdbb40039a0 con 0x7fdbc40b5e80 2023-05-22T15:27:09.063 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.002+0000 7f2f4d7fa640 1 -- 172.21.15.16:0/641126305 <== mon.2 v2:172.21.15.62:3301/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7f2f3c0a7450 con 0x7f2f480e2410 2023-05-22T15:27:09.063 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.002+0000 7f2f4d7fa640 1 --2- 172.21.15.16:0/641126305 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f2f34065450 0x7f2f34067900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:09.064 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.002+0000 7f2f54ea0640 1 --2- 172.21.15.16:0/641126305 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f2f34065450 0x7f2f34067900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:09.064 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.002+0000 7f2f4d7fa640 1 -- 172.21.15.16:0/641126305 <== mon.2 v2:172.21.15.62:3301/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f2f3c11d2d0 con 0x7f2f480e2410 2023-05-22T15:27:09.064 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.002+0000 7f33a4ff9640 1 -- 172.21.15.16:0/519042481 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f339c011790 con 0x7f33a80b4410 2023-05-22T15:27:09.064 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.002+0000 7f33a4ff9640 1 -- 172.21.15.16:0/519042481 <== mon.2 v2:172.21.15.62:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f339c0118f0 con 0x7f33a80b4410 2023-05-22T15:27:09.065 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.002+0000 7f2f54ea0640 1 --2- 172.21.15.16:0/641126305 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f2f34065450 0x7f2f34067900 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f2f44000c10 tx=0x7f2f44009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:09.065 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.002+0000 7f39eaffd640 1 --2- 172.21.15.16:0/395007902 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f39e4050e10 0x7f39e40352a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:09.066 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.002+0000 7f39eb7fe640 1 --2- 172.21.15.16:0/395007902 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f39e4036d50 0x7f39e404a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:09.066 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.002+0000 7f39eaffd640 1 -- 172.21.15.16:0/395007902 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f39e4036d50 msgr2=0x7f39e404a950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:09.066 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.002+0000 7f39eaffd640 1 --2- 172.21.15.16:0/395007902 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f39e4036d50 0x7f39e404a950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:09.066 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.002+0000 7f39eaffd640 1 -- 172.21.15.16:0/395007902 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f39e40e6d20 msgr2=0x7f39e4035ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:09.067 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.002+0000 7f39eaffd640 1 --2- 172.21.15.16:0/395007902 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f39e40e6d20 0x7f39e4035ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:09.067 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.002+0000 7f39eaffd640 1 -- 172.21.15.16:0/395007902 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39e0009420 con 0x7f39e4050e10 2023-05-22T15:27:09.067 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.002+0000 7f33ad4ce640 1 -- 172.21.15.16:0/519042481 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33a81600a0 con 0x7f33a80b4410 2023-05-22T15:27:09.067 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.002+0000 7f33ad4ce640 1 -- 172.21.15.16:0/519042481 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33a80b7e10 con 0x7f33a80b4410 2023-05-22T15:27:09.067 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.006+0000 7f33a4ff9640 1 -- 172.21.15.16:0/519042481 <== mon.2 v2:172.21.15.62:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f339c011ca0 con 0x7f33a80b4410 2023-05-22T15:27:09.068 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.002+0000 7fdbab7fe640 1 --2- 172.21.15.16:0/3656687334 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fdbac069950 0x7fdbac06be00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:09.068 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.002+0000 7f67e1b46640 1 -- 172.21.15.16:0/2581386126 shutdown_connections 2023-05-22T15:27:09.069 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.002+0000 7fdbab7fe640 1 -- 172.21.15.16:0/3656687334 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6813+0+0 (secure 0 0 0) 0x7fdbb4082540 con 0x7fdbc40b5e80 2023-05-22T15:27:09.069 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.002+0000 7fdbc2ffd640 1 --2- 172.21.15.16:0/3656687334 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fdbac069950 0x7fdbac06be00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:09.069 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.002+0000 7fdbc2ffd640 1 --2- 172.21.15.16:0/3656687334 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fdbac069950 0x7fdbac06be00 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fdbb801a140 tx=0x7fdbb80a7000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:09.069 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.006+0000 7f33ad4ce640 1 -- 172.21.15.16:0/519042481 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3388005260 con 0x7f33a80b4410 2023-05-22T15:27:09.069 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.006+0000 7f39eaffd640 1 --2- 172.21.15.16:0/395007902 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f39e4050e10 0x7f39e40352a0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f39d8001390 tx=0x7f39d800cb20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:09.070 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.006+0000 7f6146ffd640 1 -- 172.21.15.16:0/3342720135 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6140011830 con 0x7f61480e3b60 2023-05-22T15:27:09.070 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.006+0000 7f6146ffd640 1 -- 172.21.15.16:0/3342720135 <== mon.2 v2:172.21.15.62:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6140011990 con 0x7f61480e3b60 2023-05-22T15:27:09.070 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.006+0000 7f6158604640 1 -- 172.21.15.16:0/3342720135 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6148050710 con 0x7f61480e3b60 2023-05-22T15:27:09.071 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.006+0000 7f39d7fff640 1 -- 172.21.15.16:0/395007902 <== mon.1 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39d8005ae0 con 0x7f39e4050e10 2023-05-22T15:27:09.071 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.006+0000 7f39f0ede640 1 -- 172.21.15.16:0/395007902 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39e40069f0 con 0x7f39e4050e10 2023-05-22T15:27:09.071 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.006+0000 7f39f0ede640 1 -- 172.21.15.16:0/395007902 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39e40548f0 con 0x7f39e4050e10 2023-05-22T15:27:09.071 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.006+0000 7f39d7fff640 1 -- 172.21.15.16:0/395007902 <== mon.1 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39d8005c60 con 0x7f39e4050e10 2023-05-22T15:27:09.071 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.006+0000 7f39d7fff640 1 -- 172.21.15.16:0/395007902 <== mon.1 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39d800ce60 con 0x7f39e4050e10 2023-05-22T15:27:09.072 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.006+0000 7f6146ffd640 1 -- 172.21.15.16:0/3342720135 <== mon.2 v2:172.21.15.62:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6140011d20 con 0x7f61480e3b60 2023-05-22T15:27:09.072 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.006+0000 7f6158604640 1 -- 172.21.15.16:0/3342720135 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6148050430 con 0x7f61480e3b60 2023-05-22T15:27:09.072 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.006+0000 7f39f0ede640 1 -- 172.21.15.16:0/395007902 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39cc005260 con 0x7f39e4050e10 2023-05-22T15:27:09.073 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.006+0000 7fdbab7fe640 1 -- 172.21.15.16:0/3656687334 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fdbb404ff60 con 0x7fdbc40b5e80 2023-05-22T15:27:09.073 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.006+0000 7f6c14312640 1 Processor -- start 2023-05-22T15:27:09.073 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.006+0000 7f67e1b46640 1 -- 172.21.15.16:0/2581386126 wait complete. 2023-05-22T15:27:09.074 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.006+0000 7f6158604640 1 -- 172.21.15.16:0/3342720135 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6148053f20 con 0x7f61480e3b60 2023-05-22T15:27:09.074 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.006+0000 7f39d7fff640 1 -- 172.21.15.16:0/395007902 <== mon.1 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7f39d8007070 con 0x7f39e4050e10 2023-05-22T15:27:09.074 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.010+0000 7f39d7fff640 1 --2- 172.21.15.16:0/395007902 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f39d0065380 0x7f39d0067830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:09.074 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.010+0000 7f39d7fff640 1 -- 172.21.15.16:0/395007902 <== mon.1 v2:172.21.15.62:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f39d8082220 con 0x7f39e4050e10 2023-05-22T15:27:09.075 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.010+0000 7f67e1b46640 1 Processor -- start 2023-05-22T15:27:09.075 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.010+0000 7f39eb7fe640 1 --2- 172.21.15.16:0/395007902 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f39d0065380 0x7f39d0067830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:09.075 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.010+0000 7f67e1b46640 1 -- start start 2023-05-22T15:27:09.076 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.010+0000 7f39eb7fe640 1 --2- 172.21.15.16:0/395007902 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f39d0065380 0x7f39d0067830 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f39e00ad000 tx=0x7f39e00a6000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:09.077 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.010+0000 7f67e1b46640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f67d404bb80 0x7f67d40d43f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:09.077 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.010+0000 7f67e1b46640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f67d4054150 0x7f67d40e7850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:09.077 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.010+0000 7f67e1b46640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f67d40d4c70 0x7f67d404edc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:09.077 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.010+0000 7f67e1b46640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67d417e4f0 con 0x7f67d404bb80 2023-05-22T15:27:09.077 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.010+0000 7f67e1b46640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67d417e790 con 0x7f67d40d4c70 2023-05-22T15:27:09.078 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.010+0000 7f67e1b46640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67d417ea70 con 0x7f67d4054150 2023-05-22T15:27:09.078 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.010+0000 7f67e0b44640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f67d404bb80 0x7f67d40d43f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:09.078 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.010+0000 7f67e0b44640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f67d404bb80 0x7f67d40d43f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57976/0 (socket says 172.21.15.16:57976) 2023-05-22T15:27:09.078 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.010+0000 7f67e0b44640 1 -- 172.21.15.16:0/2484169114 learned_addr learned my addr 172.21.15.16:0/2484169114 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:09.078 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.010+0000 7f67dbfff640 1 --2- 172.21.15.16:0/2484169114 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f67d4054150 0x7f67d40e7850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:09.078 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.010+0000 7f67e1345640 1 --2- 172.21.15.16:0/2484169114 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f67d40d4c70 0x7f67d404edc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:09.078 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.010+0000 7f67e0b44640 1 -- 172.21.15.16:0/2484169114 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f67d4054150 msgr2=0x7f67d40e7850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:09.079 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.010+0000 7f67e0b44640 1 --2- 172.21.15.16:0/2484169114 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f67d4054150 0x7f67d40e7850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:09.079 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.010+0000 7f67e0b44640 1 -- 172.21.15.16:0/2484169114 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f67d40d4c70 msgr2=0x7f67d404edc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:09.079 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.010+0000 7f67e0b44640 1 --2- 172.21.15.16:0/2484169114 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f67d40d4c70 0x7f67d404edc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:09.079 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.010+0000 7f67e0b44640 1 -- 172.21.15.16:0/2484169114 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67d00094d0 con 0x7f67d404bb80 2023-05-22T15:27:09.079 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.010+0000 7f67dbfff640 1 --2- 172.21.15.16:0/2484169114 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f67d4054150 0x7f67d40e7850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T15:27:09.079 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.010+0000 7f67e0b44640 1 --2- 172.21.15.16:0/2484169114 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f67d404bb80 0x7f67d40d43f0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f67d001e720 tx=0x7f67d001b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:09.079 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.010+0000 7f67d97fa640 1 -- 172.21.15.16:0/2484169114 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67d00aad10 con 0x7f67d404bb80 2023-05-22T15:27:09.080 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.010+0000 7f67d97fa640 1 -- 172.21.15.16:0/2484169114 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67d00aae70 con 0x7f67d404bb80 2023-05-22T15:27:09.080 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.010+0000 7f67d97fa640 1 -- 172.21.15.16:0/2484169114 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67d0003840 con 0x7f67d404bb80 2023-05-22T15:27:09.080 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.010+0000 7f67e1b46640 1 -- 172.21.15.16:0/2484169114 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67d404c950 con 0x7f67d404bb80 2023-05-22T15:27:09.080 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.010+0000 7f67e1b46640 1 -- 172.21.15.16:0/2484169114 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67d4036e40 con 0x7f67d404bb80 2023-05-22T15:27:09.080 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.010+0000 7f6c14312640 1 -- start start 2023-05-22T15:27:09.081 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.010+0000 7f6c14312640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6c0c1648d0 0x7f6c0c0c97b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:09.081 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.010+0000 7f6c14312640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f6c0c0c9cf0 0x7f6c0c0ce380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:09.081 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.010+0000 7f6c14312640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c0c070c30 con 0x7f6c0c0c2700 2023-05-22T15:27:09.081 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.010+0000 7f6c14312640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c0c06e320 con 0x7f6c0c0c9cf0 2023-05-22T15:27:09.081 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.010+0000 7f6c14312640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c0c06a2c0 con 0x7f6c0c1648d0 2023-05-22T15:27:09.081 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.010+0000 7f6c118ad640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f6c0c0c2700 0x7f6c0c06e7b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:51180/0 (socket says 172.21.15.16:51180) 2023-05-22T15:27:09.081 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.010+0000 7f6c118ad640 1 -- 172.21.15.16:0/3629147463 learned_addr learned my addr 172.21.15.16:0/3629147463 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:27:09.082 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.010+0000 7f67e1b46640 1 -- 172.21.15.16:0/2484169114 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67d404e8e0 con 0x7f67d404bb80 2023-05-22T15:27:09.082 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.010+0000 7f6c120ae640 1 --2- 172.21.15.16:0/3629147463 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6c0c1648d0 0x7f6c0c0c97b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:09.082 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.014+0000 7f6c128af640 1 --2- 172.21.15.16:0/3629147463 >> v2:172.21.15.62:3300/0 conn(0x7f6c0c0c9cf0 0x7f6c0c0ce380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:09.083 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.014+0000 7f6c120ae640 1 -- 172.21.15.16:0/3629147463 >> v2:172.21.15.62:3300/0 conn(0x7f6c0c0c9cf0 msgr2=0x7f6c0c0ce380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:09.083 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.014+0000 7f6c120ae640 1 --2- 172.21.15.16:0/3629147463 >> v2:172.21.15.62:3300/0 conn(0x7f6c0c0c9cf0 0x7f6c0c0ce380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:09.083 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.014+0000 7f6c120ae640 1 -- 172.21.15.16:0/3629147463 >> v1:172.21.15.16:6789/0 conn(0x7f6c0c0c2700 legacy=0x7f6c0c06e7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:09.083 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.014+0000 7f6c120ae640 1 -- 172.21.15.16:0/3629147463 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c0c06dd10 con 0x7f6c0c1648d0 2023-05-22T15:27:09.084 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.014+0000 7f67d97fa640 1 -- 172.21.15.16:0/2484169114 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7f67d00039a0 con 0x7f67d404bb80 2023-05-22T15:27:09.084 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.014+0000 7f67d97fa640 1 --2- 172.21.15.16:0/2484169114 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f67c00653b0 0x7f67c0067860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:09.084 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.014+0000 7f6c120ae640 1 --2- 172.21.15.16:0/3629147463 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6c0c1648d0 0x7f6c0c0c97b0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f6bfc0097c0 tx=0x7f6bfc01c450 comp rx=0 tx=0).ready entity=mon.2 client_cookie=248f863b832de880 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:09.085 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.014+0000 7f2f4d7fa640 1 -- 172.21.15.16:0/641126305 <== mon.2 v2:172.21.15.62:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f2f3c0eac70 con 0x7f2f480e2410 2023-05-22T15:27:09.085 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.014+0000 7f6c110ac640 1 -- 172.21.15.16:0/3629147463 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bfc0ab650 con 0x7f6c0c1648d0 2023-05-22T15:27:09.085 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.018+0000 7f6146ffd640 1 -- 172.21.15.16:0/3342720135 <== mon.2 v2:172.21.15.62:3301/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7f614001a400 con 0x7f61480e3b60 2023-05-22T15:27:09.086 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.018+0000 7f67d97fa640 1 -- 172.21.15.16:0/2484169114 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f67d00f6280 con 0x7f67d404bb80 2023-05-22T15:27:09.086 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.018+0000 7f67d97fa640 1 -- 172.21.15.16:0/2484169114 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f67d00ea530 con 0x7f67d404bb80 2023-05-22T15:27:09.086 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.018+0000 7f67dbfff640 1 --2- 172.21.15.16:0/2484169114 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f67c00653b0 0x7f67c0067860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:09.086 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.018+0000 7f33a4ff9640 1 -- 172.21.15.16:0/519042481 <== mon.2 v2:172.21.15.62:3301/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7f339c01c460 con 0x7f33a80b4410 2023-05-22T15:27:09.087 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.018+0000 7f33a4ff9640 1 --2- 172.21.15.16:0/519042481 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f3394065480 0x7f3394067930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:09.087 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.018+0000 7f33a4ff9640 1 -- 172.21.15.16:0/519042481 <== mon.2 v2:172.21.15.62:3301/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f339c083370 con 0x7f33a80b4410 2023-05-22T15:27:09.087 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.018+0000 7f6146ffd640 1 --2- 172.21.15.16:0/3342720135 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f61340653d0 0x7f6134067880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:09.087 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.018+0000 7f33a77fe640 1 --2- 172.21.15.16:0/519042481 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f3394065480 0x7f3394067930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:09.088 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.018+0000 7f33a77fe640 1 --2- 172.21.15.16:0/519042481 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f3394065480 0x7f3394067930 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f33a81f71e0 tx=0x7f3398009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:09.088 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.018+0000 7f67dbfff640 1 --2- 172.21.15.16:0/2484169114 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f67c00653b0 0x7f67c0067860 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f67d4035f60 tx=0x7f67c8009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:09.088 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.018+0000 7f6146ffd640 1 -- 172.21.15.16:0/3342720135 <== mon.2 v2:172.21.15.62:3301/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f61400830b0 con 0x7f61480e3b60 2023-05-22T15:27:09.089 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.022+0000 7f61563a0640 1 --2- 172.21.15.16:0/3342720135 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f61340653d0 0x7f6134067880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:09.089 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.022+0000 7f61563a0640 1 --2- 172.21.15.16:0/3342720135 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f61340653d0 0x7f6134067880 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f61480dfa00 tx=0x7f6150120070 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:09.089 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.022+0000 7f39d7fff640 1 -- 172.21.15.16:0/395007902 <== mon.1 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f39d804fbc0 con 0x7f39e4050e10 2023-05-22T15:27:09.090 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.030+0000 7f6c110ac640 1 -- 172.21.15.16:0/3629147463 <== mon.2 v2:172.21.15.62:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bfc0ab7b0 con 0x7f6c0c1648d0 2023-05-22T15:27:09.090 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.030+0000 7f6c110ac640 1 -- 172.21.15.16:0/3629147463 <== mon.2 v2:172.21.15.62:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bfc0abb40 con 0x7f6c0c1648d0 2023-05-22T15:27:09.090 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.030+0000 7f33a4ff9640 1 -- 172.21.15.16:0/519042481 <== mon.2 v2:172.21.15.62:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f339c050d10 con 0x7f33a80b4410 2023-05-22T15:27:09.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.030+0000 7f6c14312640 1 -- 172.21.15.16:0/3629147463 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6c0c1648d0 msgr2=0x7f6c0c0c97b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:09.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.030+0000 7f6c14312640 1 --2- 172.21.15.16:0/3629147463 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6c0c1648d0 0x7f6c0c0c97b0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f6bfc0097c0 tx=0x7f6bfc01c450 comp rx=0 tx=0).stop 2023-05-22T15:27:09.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.030+0000 7f6c14312640 1 -- 172.21.15.16:0/3629147463 shutdown_connections 2023-05-22T15:27:09.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.030+0000 7f6c14312640 1 --2- 172.21.15.16:0/3629147463 >> v2:172.21.15.62:3300/0 conn(0x7f6c0c0c9cf0 0x7f6c0c0ce380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:09.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.030+0000 7f6c14312640 1 --2- 172.21.15.16:0/3629147463 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6c0c1648d0 0x7f6c0c0c97b0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:09.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.030+0000 7f6c14312640 1 -- 172.21.15.16:0/3629147463 >> 172.21.15.16:0/3629147463 conn(0x7f6c0c0cc870 msgr2=0x7f6c0c0cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:09.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.030+0000 7f6c14312640 1 -- 172.21.15.16:0/3629147463 shutdown_connections 2023-05-22T15:27:09.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.030+0000 7f6c14312640 1 -- 172.21.15.16:0/3629147463 wait complete. 2023-05-22T15:27:09.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.030+0000 7f6c14312640 1 Processor -- start 2023-05-22T15:27:09.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.030+0000 7f6c14312640 1 -- start start 2023-05-22T15:27:09.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.030+0000 7f6c14312640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6c0c0c2700 0x7f6c0c0c3000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:09.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.030+0000 7f6c14312640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c0c0c9cf0 0x7f6c0c0b1f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:09.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.034+0000 7f6c14312640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6c0c1648d0 0x7f6c0c159b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:09.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.034+0000 7f6c14312640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c0c1541d0 con 0x7f6c0c0c9cf0 2023-05-22T15:27:09.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.034+0000 7f6c14312640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c0c164160 con 0x7f6c0c0c2700 2023-05-22T15:27:09.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.034+0000 7f6c14312640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c0c0b49f0 con 0x7f6c0c1648d0 2023-05-22T15:27:09.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.034+0000 7f6c118ad640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c0c0c9cf0 0x7f6c0c0b1f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:09.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.034+0000 7f6c118ad640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c0c0c9cf0 0x7f6c0c0b1f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57980/0 (socket says 172.21.15.16:57980) 2023-05-22T15:27:09.096 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.034+0000 7f6c118ad640 1 -- 172.21.15.16:0/3879614884 learned_addr learned my addr 172.21.15.16:0/3879614884 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:09.096 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.034+0000 7f6c120ae640 1 --2- 172.21.15.16:0/3879614884 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6c0c0c2700 0x7f6c0c0c3000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:09.096 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.034+0000 7f6c128af640 1 --2- 172.21.15.16:0/3879614884 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6c0c1648d0 0x7f6c0c159b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:09.096 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.034+0000 7f6c118ad640 1 -- 172.21.15.16:0/3879614884 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6c0c1648d0 msgr2=0x7f6c0c159b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:09.096 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.034+0000 7f6c118ad640 1 --2- 172.21.15.16:0/3879614884 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6c0c1648d0 0x7f6c0c159b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:09.096 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.034+0000 7f6c118ad640 1 -- 172.21.15.16:0/3879614884 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6c0c0c2700 msgr2=0x7f6c0c0c3000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:09.096 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.034+0000 7f6c118ad640 1 --2- 172.21.15.16:0/3879614884 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6c0c0c2700 0x7f6c0c0c3000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:09.097 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.034+0000 7f6c118ad640 1 -- 172.21.15.16:0/3879614884 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bfc009420 con 0x7f6c0c0c9cf0 2023-05-22T15:27:09.097 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.034+0000 7f6c128af640 1 --2- 172.21.15.16:0/3879614884 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6c0c1648d0 0x7f6c0c159b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T15:27:09.097 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.034+0000 7f6c118ad640 1 --2- 172.21.15.16:0/3879614884 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c0c0c9cf0 0x7f6c0c0b1f90 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f6c0800c370 tx=0x7f6c0800c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:09.097 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.034+0000 7f6c02ffd640 1 -- 172.21.15.16:0/3879614884 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c08005cf0 con 0x7f6c0c0c9cf0 2023-05-22T15:27:09.097 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.034+0000 7f6c02ffd640 1 -- 172.21.15.16:0/3879614884 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c08005e50 con 0x7f6c0c0c9cf0 2023-05-22T15:27:09.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.034+0000 7f6c14312640 1 -- 172.21.15.16:0/3879614884 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c0c0cecd0 con 0x7f6c0c0c9cf0 2023-05-22T15:27:09.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.034+0000 7f6c14312640 1 -- 172.21.15.16:0/3879614884 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c0c06dfb0 con 0x7f6c0c0c9cf0 2023-05-22T15:27:09.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.034+0000 7f6c02ffd640 1 -- 172.21.15.16:0/3879614884 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c080072d0 con 0x7f6c0c0c9cf0 2023-05-22T15:27:09.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.034+0000 7f6c14312640 1 -- 172.21.15.16:0/3879614884 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6bf4005260 con 0x7f6c0c0c9cf0 2023-05-22T15:27:09.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.034+0000 7f6c02ffd640 1 -- 172.21.15.16:0/3879614884 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7f6c08007430 con 0x7f6c0c0c9cf0 2023-05-22T15:27:09.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.038+0000 7f6c02ffd640 1 --2- 172.21.15.16:0/3879614884 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f6bf0065380 0x7f6bf0067830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:09.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.042+0000 7f6146ffd640 1 -- 172.21.15.16:0/3342720135 <== mon.2 v2:172.21.15.62:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f6140050b70 con 0x7f61480e3b60 2023-05-22T15:27:09.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.046+0000 7f6c02ffd640 1 -- 172.21.15.16:0/3879614884 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f6c0805b470 con 0x7f6c0c0c9cf0 2023-05-22T15:27:09.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.046+0000 7f6c02ffd640 1 -- 172.21.15.16:0/3879614884 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f6c0804f7d0 con 0x7f6c0c0c9cf0 2023-05-22T15:27:09.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.046+0000 7f6c120ae640 1 --2- 172.21.15.16:0/3879614884 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f6bf0065380 0x7f6bf0067830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:09.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.046+0000 7f6c120ae640 1 --2- 172.21.15.16:0/3879614884 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f6bf0065380 0x7f6bf0067830 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f6bfc0af000 tx=0x7f6bfc0a7000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:09.311 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.302+0000 7f4b195d0640 1 -- 172.21.15.16:0/2152752373 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f4b140738f0 con 0x7f4b14157d30 2023-05-22T15:27:09.312 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.302+0000 7f4afb7fe640 1 -- 172.21.15.16:0/2152752373 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f4b0c0b1020 con 0x7f4b14157d30 2023-05-22T15:27:09.317 INFO:teuthology.orchestra.run.smithi016.stdout:55834574856 2023-05-22T15:27:09.319 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.306+0000 7f4b195d0640 1 -- 172.21.15.16:0/2152752373 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f4b00065400 msgr2=0x7f4b000678b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:09.319 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.306+0000 7f4b195d0640 1 --2- 172.21.15.16:0/2152752373 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f4b00065400 0x7f4b000678b0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f4b04007fd0 tx=0x7f4b04007d60 comp rx=0 tx=0).stop 2023-05-22T15:27:09.320 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.306+0000 7f4b195d0640 1 -- 172.21.15.16:0/2152752373 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b14157d30 msgr2=0x7f4b14143820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:09.320 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.306+0000 7f4b195d0640 1 --2- 172.21.15.16:0/2152752373 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b14157d30 0x7f4b14143820 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f4b0c0af040 tx=0x7f4b0c019f70 comp rx=0 tx=0).stop 2023-05-22T15:27:09.320 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.306+0000 7f4b195d0640 1 -- 172.21.15.16:0/2152752373 shutdown_connections 2023-05-22T15:27:09.321 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.306+0000 7f4b195d0640 1 --2- 172.21.15.16:0/2152752373 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f4b00065400 0x7f4b000678b0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:09.321 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.306+0000 7f4b195d0640 1 --2- 172.21.15.16:0/2152752373 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4b14158100 0x7f4b14145590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:09.321 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.306+0000 7f4b195d0640 1 --2- 172.21.15.16:0/2152752373 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b14157d30 0x7f4b14143820 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:09.322 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.310+0000 7f4b195d0640 1 --2- 172.21.15.16:0/2152752373 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4b140caa30 0x7f4b140cd460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:09.323 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.310+0000 7f4b195d0640 1 -- 172.21.15.16:0/2152752373 >> 172.21.15.16:0/2152752373 conn(0x7f4b140af9f0 msgr2=0x7f4b14070f80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:09.323 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.310+0000 7f4b195d0640 1 -- 172.21.15.16:0/2152752373 shutdown_connections 2023-05-22T15:27:09.323 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.310+0000 7f4b195d0640 1 -- 172.21.15.16:0/2152752373 wait complete. 2023-05-22T15:27:09.338 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.330+0000 7f2f57104640 1 -- 172.21.15.16:0/641126305 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f2f300057e0 con 0x7f2f480e2410 2023-05-22T15:27:09.339 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.330+0000 7f2f4d7fa640 1 -- 172.21.15.16:0/641126305 <== mon.2 v2:172.21.15.62:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f2f3c0ee290 con 0x7f2f480e2410 2023-05-22T15:27:09.341 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574856 for osd.2 2023-05-22T15:27:09.341 INFO:teuthology.orchestra.run.smithi016.stdout:55834574855 2023-05-22T15:27:09.344 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.330+0000 7f2f3affd640 1 -- 172.21.15.16:0/641126305 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f2f34065450 msgr2=0x7f2f34067900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:09.345 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.330+0000 7f2f3affd640 1 --2- 172.21.15.16:0/641126305 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f2f34065450 0x7f2f34067900 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f2f44000c10 tx=0x7f2f44009040 comp rx=0 tx=0).stop 2023-05-22T15:27:09.345 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.330+0000 7f2f3affd640 1 -- 172.21.15.16:0/641126305 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2f480e2410 msgr2=0x7f2f48180420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:09.345 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.330+0000 7f2f3affd640 1 --2- 172.21.15.16:0/641126305 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2f480e2410 0x7f2f48180420 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f2f3c008000 tx=0x7f2f3c019830 comp rx=0 tx=0).stop 2023-05-22T15:27:09.345 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.330+0000 7f2f3affd640 1 -- 172.21.15.16:0/641126305 shutdown_connections 2023-05-22T15:27:09.346 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.330+0000 7f2f3affd640 1 --2- 172.21.15.16:0/641126305 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2f48180960 0x7f2f48050090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:09.346 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.330+0000 7f2f3affd640 1 --2- 172.21.15.16:0/641126305 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f2f34065450 0x7f2f34067900 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:09.346 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.330+0000 7f2f3affd640 1 --2- 172.21.15.16:0/641126305 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2f480e2410 0x7f2f48180420 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:09.346 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.330+0000 7f2f3affd640 1 --2- 172.21.15.16:0/641126305 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f48180050 0x7f2f480e4a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:09.347 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.330+0000 7f2f3affd640 1 -- 172.21.15.16:0/641126305 >> 172.21.15.16:0/641126305 conn(0x7f2f48055940 msgr2=0x7f2f48058d50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:09.347 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.330+0000 7f2f3affd640 1 -- 172.21.15.16:0/641126305 shutdown_connections 2023-05-22T15:27:09.347 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.330+0000 7f2f3affd640 1 -- 172.21.15.16:0/641126305 wait complete. 2023-05-22T15:27:09.348 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.330+0000 7f67e1b46640 1 -- 172.21.15.16:0/2484169114 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f67d4005280 con 0x7f67d404bb80 2023-05-22T15:27:09.348 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.334+0000 7f67d97fa640 1 -- 172.21.15.16:0/2484169114 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f67d00edb50 con 0x7f67d404bb80 2023-05-22T15:27:09.349 INFO:teuthology.orchestra.run.smithi016.stdout:55834574857 2023-05-22T15:27:09.349 INFO:teuthology.orchestra.run.smithi016.stdout:55834574855 2023-05-22T15:27:09.351 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.334+0000 7f39f0ede640 1 -- 172.21.15.16:0/395007902 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f39cc0057e0 con 0x7f39e4050e10 2023-05-22T15:27:09.351 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.334+0000 7f39d7fff640 1 -- 172.21.15.16:0/395007902 <== mon.1 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f39d80531e0 con 0x7f39e4050e10 2023-05-22T15:27:09.352 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.334+0000 7fdbc97ab640 1 -- 172.21.15.16:0/3656687334 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7fdba40057e0 con 0x7fdbc40b5e80 2023-05-22T15:27:09.352 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.338+0000 7f67e1b46640 1 -- 172.21.15.16:0/2484169114 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f67c00653b0 msgr2=0x7f67c0067860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:09.352 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.338+0000 7f67e1b46640 1 --2- 172.21.15.16:0/2484169114 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f67c00653b0 0x7f67c0067860 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f67d4035f60 tx=0x7f67c8009040 comp rx=0 tx=0).stop 2023-05-22T15:27:09.352 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.338+0000 7f67e1b46640 1 -- 172.21.15.16:0/2484169114 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f67d404bb80 msgr2=0x7f67d40d43f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:09.353 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.338+0000 7fdbab7fe640 1 -- 172.21.15.16:0/3656687334 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fdbb4053580 con 0x7fdbc40b5e80 2023-05-22T15:27:09.353 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.338+0000 7f67e1b46640 1 --2- 172.21.15.16:0/2484169114 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f67d404bb80 0x7f67d40d43f0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f67d001e720 tx=0x7f67d001b480 comp rx=0 tx=0).stop 2023-05-22T15:27:09.354 INFO:teuthology.orchestra.run.smithi016.stdout:55834574857 2023-05-22T15:27:09.355 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.338+0000 7f67e1b46640 1 -- 172.21.15.16:0/2484169114 shutdown_connections 2023-05-22T15:27:09.355 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.338+0000 7f67e1b46640 1 --2- 172.21.15.16:0/2484169114 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f67d40d4c70 0x7f67d404edc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:09.355 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.338+0000 7f67e1b46640 1 --2- 172.21.15.16:0/2484169114 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f67c00653b0 0x7f67c0067860 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:09.355 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.338+0000 7f67e1b46640 1 --2- 172.21.15.16:0/2484169114 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f67d4054150 0x7f67d40e7850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:09.355 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.338+0000 7f67e1b46640 1 --2- 172.21.15.16:0/2484169114 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f67d404bb80 0x7f67d40d43f0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:09.356 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.338+0000 7f67e1b46640 1 -- 172.21.15.16:0/2484169114 >> 172.21.15.16:0/2484169114 conn(0x7f67d4051530 msgr2=0x7f67d4034510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:09.356 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.338+0000 7f67e1b46640 1 -- 172.21.15.16:0/2484169114 shutdown_connections 2023-05-22T15:27:09.356 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.338+0000 7f67e1b46640 1 -- 172.21.15.16:0/2484169114 wait complete. 2023-05-22T15:27:09.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.338+0000 7f39f0ede640 1 -- 172.21.15.16:0/395007902 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f39d0065380 msgr2=0x7f39d0067830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:09.358 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.338+0000 7f39f0ede640 1 --2- 172.21.15.16:0/395007902 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f39d0065380 0x7f39d0067830 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f39e00ad000 tx=0x7f39e00a6000 comp rx=0 tx=0).stop 2023-05-22T15:27:09.358 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.338+0000 7f39f0ede640 1 -- 172.21.15.16:0/395007902 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f39e4050e10 msgr2=0x7f39e40352a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:09.358 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.338+0000 7f39f0ede640 1 --2- 172.21.15.16:0/395007902 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f39e4050e10 0x7f39e40352a0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f39d8001390 tx=0x7f39d800cb20 comp rx=0 tx=0).stop 2023-05-22T15:27:09.359 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.338+0000 7f39f0ede640 1 -- 172.21.15.16:0/395007902 shutdown_connections 2023-05-22T15:27:09.359 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.338+0000 7fdbc97ab640 1 -- 172.21.15.16:0/3656687334 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fdbac069950 msgr2=0x7fdbac06be00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:09.360 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.338+0000 7f39f0ede640 1 --2- 172.21.15.16:0/395007902 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f39e40e6d20 0x7f39e4035ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:09.360 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.338+0000 7fdbc97ab640 1 --2- 172.21.15.16:0/3656687334 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fdbac069950 0x7fdbac06be00 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fdbb801a140 tx=0x7fdbb80a7000 comp rx=0 tx=0).stop 2023-05-22T15:27:09.361 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.338+0000 7fdbc97ab640 1 -- 172.21.15.16:0/3656687334 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdbc40b5e80 msgr2=0x7fdbc414bcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:09.361 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.338+0000 7fdbc97ab640 1 --2- 172.21.15.16:0/3656687334 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdbc40b5e80 0x7fdbc414bcf0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fdbb4000f70 tx=0x7fdbb400c890 comp rx=0 tx=0).stop 2023-05-22T15:27:09.361 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.338+0000 7f39f0ede640 1 --2- 172.21.15.16:0/395007902 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f39e4050e10 0x7f39e40352a0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:09.362 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.342+0000 7f39f0ede640 1 --2- 172.21.15.16:0/395007902 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f39d0065380 0x7f39d0067830 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:09.362 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.342+0000 7f39f0ede640 1 --2- 172.21.15.16:0/395007902 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f39e4036d50 0x7f39e404a950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:09.363 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.342+0000 7fdbc97ab640 1 -- 172.21.15.16:0/3656687334 shutdown_connections 2023-05-22T15:27:09.364 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.342+0000 7fdbc97ab640 1 --2- 172.21.15.16:0/3656687334 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdbc4159b10 0x7fdbc414c230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:09.364 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.342+0000 7fdbc97ab640 1 --2- 172.21.15.16:0/3656687334 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdbc40b5e80 0x7fdbc414bcf0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:09.364 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.342+0000 7fdbc97ab640 1 --2- 172.21.15.16:0/3656687334 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fdbac069950 0x7fdbac06be00 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:09.364 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.342+0000 7fdbc97ab640 1 --2- 172.21.15.16:0/3656687334 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fdbc406e250 0x7fdbc414b130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:09.365 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.342+0000 7fdbc97ab640 1 -- 172.21.15.16:0/3656687334 >> 172.21.15.16:0/3656687334 conn(0x7fdbc406f560 msgr2=0x7fdbc4152680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:09.365 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.342+0000 7f39f0ede640 1 -- 172.21.15.16:0/395007902 >> 172.21.15.16:0/395007902 conn(0x7f39e4051640 msgr2=0x7f39e40e60d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:09.366 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.342+0000 7fdbc97ab640 1 -- 172.21.15.16:0/3656687334 shutdown_connections 2023-05-22T15:27:09.367 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.342+0000 7fdbc97ab640 1 -- 172.21.15.16:0/3656687334 wait complete. 2023-05-22T15:27:09.367 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.342+0000 7f39f0ede640 1 -- 172.21.15.16:0/395007902 shutdown_connections 2023-05-22T15:27:09.367 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.342+0000 7f33ad4ce640 1 -- 172.21.15.16:0/519042481 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f33880057e0 con 0x7f33a80b4410 2023-05-22T15:27:09.368 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.342+0000 7f39f0ede640 1 -- 172.21.15.16:0/395007902 wait complete. 2023-05-22T15:27:09.369 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.342+0000 7f33a4ff9640 1 -- 172.21.15.16:0/519042481 <== mon.2 v2:172.21.15.62:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f339c054330 con 0x7f33a80b4410 2023-05-22T15:27:09.369 INFO:teuthology.orchestra.run.smithi016.stdout:55834574857 2023-05-22T15:27:09.371 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.350+0000 7f33ad4ce640 1 -- 172.21.15.16:0/519042481 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f3394065480 msgr2=0x7f3394067930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:09.371 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.350+0000 7f33ad4ce640 1 --2- 172.21.15.16:0/519042481 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f3394065480 0x7f3394067930 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f33a81f71e0 tx=0x7f3398009190 comp rx=0 tx=0).stop 2023-05-22T15:27:09.372 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.350+0000 7f33ad4ce640 1 -- 172.21.15.16:0/519042481 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f33a80b4410 msgr2=0x7f33a814d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:09.373 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.350+0000 7f33ad4ce640 1 --2- 172.21.15.16:0/519042481 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f33a80b4410 0x7f33a814d270 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f339c00c410 tx=0x7f339c00c8d0 comp rx=0 tx=0).stop 2023-05-22T15:27:09.373 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.350+0000 7f33ad4ce640 1 -- 172.21.15.16:0/519042481 shutdown_connections 2023-05-22T15:27:09.373 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.350+0000 7f33ad4ce640 1 --2- 172.21.15.16:0/519042481 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f33a8156f90 0x7f33a81500a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:09.373 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.350+0000 7f33ad4ce640 1 --2- 172.21.15.16:0/519042481 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f33a80b7a40 0x7f33a815a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:09.374 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.350+0000 7f33ad4ce640 1 --2- 172.21.15.16:0/519042481 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f3394065480 0x7f3394067930 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:09.374 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.350+0000 7f33ad4ce640 1 --2- 172.21.15.16:0/519042481 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f33a80b4410 0x7f33a814d270 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:09.375 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.350+0000 7f33ad4ce640 1 -- 172.21.15.16:0/519042481 >> 172.21.15.16:0/519042481 conn(0x7f33a8071a40 msgr2=0x7f33a80b5790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:09.375 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.350+0000 7f33ad4ce640 1 -- 172.21.15.16:0/519042481 shutdown_connections 2023-05-22T15:27:09.375 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.350+0000 7f33ad4ce640 1 -- 172.21.15.16:0/519042481 wait complete. 2023-05-22T15:27:09.376 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574855 for osd.4 2023-05-22T15:27:09.377 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.354+0000 7f6158604640 1 -- 172.21.15.16:0/3342720135 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f61480068b0 con 0x7f61480e3b60 2023-05-22T15:27:09.377 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.354+0000 7f6146ffd640 1 -- 172.21.15.16:0/3342720135 <== mon.2 v2:172.21.15.62:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f6140054190 con 0x7f61480e3b60 2023-05-22T15:27:09.378 INFO:teuthology.orchestra.run.smithi016.stdout:55834574856 2023-05-22T15:27:09.379 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.358+0000 7f6c14312640 1 -- 172.21.15.16:0/3879614884 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f6bf40057e0 con 0x7f6c0c0c9cf0 2023-05-22T15:27:09.379 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.358+0000 7f6c02ffd640 1 -- 172.21.15.16:0/3879614884 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f6c08017070 con 0x7f6c0c0c9cf0 2023-05-22T15:27:09.380 INFO:teuthology.orchestra.run.smithi016.stdout:55834574856 2023-05-22T15:27:09.381 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.358+0000 7f6c14312640 1 -- 172.21.15.16:0/3879614884 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f6bf0065380 msgr2=0x7f6bf0067830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:09.381 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.358+0000 7f6c14312640 1 --2- 172.21.15.16:0/3879614884 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f6bf0065380 0x7f6bf0067830 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f6bfc0af000 tx=0x7f6bfc0a7000 comp rx=0 tx=0).stop 2023-05-22T15:27:09.381 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.358+0000 7f6c14312640 1 -- 172.21.15.16:0/3879614884 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c0c0c9cf0 msgr2=0x7f6c0c0b1f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:09.382 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.358+0000 7f6c14312640 1 --2- 172.21.15.16:0/3879614884 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c0c0c9cf0 0x7f6c0c0b1f90 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f6c0800c370 tx=0x7f6c0800c830 comp rx=0 tx=0).stop 2023-05-22T15:27:09.383 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.358+0000 7f6c14312640 1 -- 172.21.15.16:0/3879614884 shutdown_connections 2023-05-22T15:27:09.383 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.358+0000 7f6c14312640 1 --2- 172.21.15.16:0/3879614884 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f6bf0065380 0x7f6bf0067830 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:09.383 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.358+0000 7f6c14312640 1 --2- 172.21.15.16:0/3879614884 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6c0c1648d0 0x7f6c0c159b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:09.383 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.358+0000 7f6c14312640 1 --2- 172.21.15.16:0/3879614884 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c0c0c9cf0 0x7f6c0c0b1f90 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:09.384 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.358+0000 7f6c14312640 1 --2- 172.21.15.16:0/3879614884 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6c0c0c2700 0x7f6c0c0c3000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:09.384 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.358+0000 7f6c14312640 1 -- 172.21.15.16:0/3879614884 >> 172.21.15.16:0/3879614884 conn(0x7f6c0c0cc870 msgr2=0x7f6c0c06e7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:09.384 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.358+0000 7f6c14312640 1 -- 172.21.15.16:0/3879614884 shutdown_connections 2023-05-22T15:27:09.385 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.358+0000 7f6c14312640 1 -- 172.21.15.16:0/3879614884 wait complete. 2023-05-22T15:27:09.387 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.358+0000 7f6158604640 1 -- 172.21.15.16:0/3342720135 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f61340653d0 msgr2=0x7f6134067880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:09.387 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.358+0000 7f6158604640 1 --2- 172.21.15.16:0/3342720135 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f61340653d0 0x7f6134067880 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f61480dfa00 tx=0x7f6150120070 comp rx=0 tx=0).stop 2023-05-22T15:27:09.387 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.358+0000 7f6158604640 1 -- 172.21.15.16:0/3342720135 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f61480e3b60 msgr2=0x7f61480d69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:09.388 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.358+0000 7f6158604640 1 --2- 172.21.15.16:0/3342720135 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f61480e3b60 0x7f61480d69b0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f614000b410 tx=0x7f614000b8d0 comp rx=0 tx=0).stop 2023-05-22T15:27:09.388 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.358+0000 7f6158604640 1 -- 172.21.15.16:0/3342720135 shutdown_connections 2023-05-22T15:27:09.388 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.358+0000 7f6158604640 1 --2- 172.21.15.16:0/3342720135 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f61340653d0 0x7f6134067880 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:09.389 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.358+0000 7f6158604640 1 --2- 172.21.15.16:0/3342720135 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f61480e3b60 0x7f61480d69b0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:09.389 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.358+0000 7f6158604640 1 --2- 172.21.15.16:0/3342720135 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f61480e3710 0x7f6148034980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:09.389 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.358+0000 7f6158604640 1 --2- 172.21.15.16:0/3342720135 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f61480e3340 0x7f6148049000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:09.389 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.358+0000 7f6158604640 1 -- 172.21.15.16:0/3342720135 >> 172.21.15.16:0/3342720135 conn(0x7f61480526c0 msgr2=0x7f6148056370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:09.390 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.358+0000 7f6158604640 1 -- 172.21.15.16:0/3342720135 shutdown_connections 2023-05-22T15:27:09.390 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:09.358+0000 7f6158604640 1 -- 172.21.15.16:0/3342720135 wait complete. 2023-05-22T15:27:09.391 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574858 got 55834574857 for osd.5 2023-05-22T15:27:09.393 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574858 got 55834574857 for osd.0 2023-05-22T15:27:09.394 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574855 for osd.3 2023-05-22T15:27:09.395 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574858 got 55834574857 for osd.6 2023-05-22T15:27:09.396 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574856 for osd.7 2023-05-22T15:27:09.398 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574856 for osd.1 2023-05-22T15:27:10.342 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-05-22T15:27:10.377 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2023-05-22T15:27:10.393 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2023-05-22T15:27:10.393 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-05-22T15:27:10.395 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2023-05-22T15:27:10.397 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2023-05-22T15:27:10.397 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2023-05-22T15:27:10.399 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-05-22T15:27:10.498 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.490+0000 7f57bb26e640 1 Processor -- start 2023-05-22T15:27:10.500 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.490+0000 7f57bb26e640 1 -- start start 2023-05-22T15:27:10.500 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.490+0000 7f57bb26e640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f57b40caa30 0x7f57b40caea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:10.500 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.490+0000 7f57bb26e640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57b40ca270 con 0x7f57b4158100 2023-05-22T15:27:10.500 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.490+0000 7f57bb26e640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57b40c8570 con 0x7f57b4157d30 2023-05-22T15:27:10.500 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.490+0000 7f57bb26e640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57b40c8850 con 0x7f57b40caa30 2023-05-22T15:27:10.501 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.490+0000 7f57b8809640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f57b40caa30 0x7f57b40caea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:10.501 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.490+0000 7f57b8809640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f57b40caa30 0x7f57b40caea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37666/0 (socket says 172.21.15.16:37666) 2023-05-22T15:27:10.501 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.490+0000 7f57b8809640 1 -- 172.21.15.16:0/2654677060 learned_addr learned my addr 172.21.15.16:0/2654677060 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:10.504 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.490+0000 7f57b8809640 1 -- 172.21.15.16:0/2654677060 >> v1:172.21.15.62:6789/0 conn(0x7f57b4157d30 legacy=0x7f57b415c140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:10.505 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.490+0000 7f57b8809640 1 -- 172.21.15.16:0/2654677060 >> v1:172.21.15.16:6789/0 conn(0x7f57b4158100 legacy=0x7f57b4158540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:10.505 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.490+0000 7f57b8809640 1 -- 172.21.15.16:0/2654677060 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57b40ca510 con 0x7f57b40caa30 2023-05-22T15:27:10.505 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.494+0000 7f57b8809640 1 --2- 172.21.15.16:0/2654677060 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f57b40caa30 0x7f57b40caea0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f57b0008870 tx=0x7f57b001ea30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d8b0c525180795d server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:10.505 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.494+0000 7f57abfff640 1 -- 172.21.15.16:0/2654677060 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57b00a7900 con 0x7f57b40caa30 2023-05-22T15:27:10.506 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.494+0000 7f57abfff640 1 -- 172.21.15.16:0/2654677060 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57b00ab070 con 0x7f57b40caa30 2023-05-22T15:27:10.506 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.494+0000 7f57abfff640 1 -- 172.21.15.16:0/2654677060 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57b00a7cc0 con 0x7f57b40caa30 2023-05-22T15:27:10.506 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.498+0000 7f57bb26e640 1 -- 172.21.15.16:0/2654677060 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f57b40caa30 msgr2=0x7f57b40caea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:10.507 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.498+0000 7f57bb26e640 1 --2- 172.21.15.16:0/2654677060 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f57b40caa30 0x7f57b40caea0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f57b0008870 tx=0x7f57b001ea30 comp rx=0 tx=0).stop 2023-05-22T15:27:10.507 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.498+0000 7f57bb26e640 1 -- 172.21.15.16:0/2654677060 shutdown_connections 2023-05-22T15:27:10.507 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.498+0000 7f57bb26e640 1 --2- 172.21.15.16:0/2654677060 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f57b40caa30 0x7f57b40caea0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:10.507 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.498+0000 7f57bb26e640 1 -- 172.21.15.16:0/2654677060 >> 172.21.15.16:0/2654677060 conn(0x7f57b40af9f0 msgr2=0x7f57b40afdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:10.510 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.498+0000 7f57bb26e640 1 -- 172.21.15.16:0/2654677060 shutdown_connections 2023-05-22T15:27:10.511 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.498+0000 7f57bb26e640 1 -- 172.21.15.16:0/2654677060 wait complete. 2023-05-22T15:27:10.511 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.498+0000 7f57bb26e640 1 Processor -- start 2023-05-22T15:27:10.511 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.498+0000 7f57bb26e640 1 -- start start 2023-05-22T15:27:10.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.502+0000 7f57bb26e640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f57b40caa30 0x7f57b40c18c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:10.513 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.502+0000 7f57bb26e640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f57b4157d30 0x7f57b40c24e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:10.513 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.502+0000 7f57bb26e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f57b4158100 0x7f57b40c75d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:10.513 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.502+0000 7f57bb26e640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57b40b2110 con 0x7f57b4158100 2023-05-22T15:27:10.513 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.502+0000 7f57bb26e640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57b41502b0 con 0x7f57b4157d30 2023-05-22T15:27:10.514 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.502+0000 7f57bb26e640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57b4150590 con 0x7f57b40caa30 2023-05-22T15:27:10.514 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.502+0000 7f57b900a640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f57b40caa30 0x7f57b40c18c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:10.514 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.502+0000 7f57b900a640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f57b40caa30 0x7f57b40c18c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3301/0 says I am v2:172.21.15.16:57504/0 (socket says 172.21.15.16:57504) 2023-05-22T15:27:10.514 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.502+0000 7f57b900a640 1 -- 172.21.15.16:0/4248254360 learned_addr learned my addr 172.21.15.16:0/4248254360 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:10.514 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.502+0000 7f57b8809640 1 --2- 172.21.15.16:0/4248254360 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f57b4157d30 0x7f57b40c24e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:10.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.502+0000 7f57b980b640 1 --2- 172.21.15.16:0/4248254360 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f57b4158100 0x7f57b40c75d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:10.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.502+0000 7f57b900a640 1 -- 172.21.15.16:0/4248254360 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f57b4157d30 msgr2=0x7f57b40c24e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:10.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.502+0000 7f57b900a640 1 --2- 172.21.15.16:0/4248254360 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f57b4157d30 0x7f57b40c24e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:10.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.502+0000 7f57b900a640 1 -- 172.21.15.16:0/4248254360 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f57b4158100 msgr2=0x7f57b40c75d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:10.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.502+0000 7f57b900a640 1 --2- 172.21.15.16:0/4248254360 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f57b4158100 0x7f57b40c75d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:10.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.502+0000 7f57b900a640 1 -- 172.21.15.16:0/4248254360 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57b0008550 con 0x7f57b40caa30 2023-05-22T15:27:10.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.502+0000 7f57b980b640 1 --2- 172.21.15.16:0/4248254360 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f57b4158100 0x7f57b40c75d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T15:27:10.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.502+0000 7f57b900a640 1 --2- 172.21.15.16:0/4248254360 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f57b40caa30 0x7f57b40c18c0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f57ac00c790 tx=0x7f57ac00cc50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:10.516 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.502+0000 7f57a9ffb640 1 -- 172.21.15.16:0/4248254360 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57ac00cea0 con 0x7f57b40caa30 2023-05-22T15:27:10.516 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.502+0000 7f57bb26e640 1 -- 172.21.15.16:0/4248254360 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57b4165ae0 con 0x7f57b40caa30 2023-05-22T15:27:10.517 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.502+0000 7f57bb26e640 1 -- 172.21.15.16:0/4248254360 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57b40cd740 con 0x7f57b40caa30 2023-05-22T15:27:10.517 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.506+0000 7f57a9ffb640 1 -- 172.21.15.16:0/4248254360 <== mon.2 v2:172.21.15.62:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57ac006d60 con 0x7f57b40caa30 2023-05-22T15:27:10.517 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.506+0000 7f57a9ffb640 1 -- 172.21.15.16:0/4248254360 <== mon.2 v2:172.21.15.62:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57ac003bc0 con 0x7f57b40caa30 2023-05-22T15:27:10.517 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.506+0000 7f57a9ffb640 1 -- 172.21.15.16:0/4248254360 <== mon.2 v2:172.21.15.62:3301/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7f57ac003d20 con 0x7f57b40caa30 2023-05-22T15:27:10.517 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.506+0000 7f57a9ffb640 1 --2- 172.21.15.16:0/4248254360 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f5798065450 0x7f5798067900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:10.517 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.506+0000 7f57a9ffb640 1 -- 172.21.15.16:0/4248254360 <== mon.2 v2:172.21.15.62:3301/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f57ac0826d0 con 0x7f57b40caa30 2023-05-22T15:27:10.518 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.506+0000 7f57b8809640 1 --2- 172.21.15.16:0/4248254360 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f5798065450 0x7f5798067900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:10.518 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.506+0000 7f57bb26e640 1 -- 172.21.15.16:0/4248254360 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5794005260 con 0x7f57b40caa30 2023-05-22T15:27:10.518 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.506+0000 7f57b8809640 1 --2- 172.21.15.16:0/4248254360 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f5798065450 0x7f5798067900 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f57b00b2000 tx=0x7f57b00a9070 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:10.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.522+0000 7f57a9ffb640 1 -- 172.21.15.16:0/4248254360 <== mon.2 v2:172.21.15.62:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f57ac050110 con 0x7f57b40caa30 2023-05-22T15:27:10.559 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.550+0000 7f01f22d5640 1 Processor -- start 2023-05-22T15:27:10.561 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.550+0000 7f01f22d5640 1 -- start start 2023-05-22T15:27:10.561 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.550+0000 7f01f22d5640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f01ec157d30 0x7f01ec15c140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:10.561 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.550+0000 7f01f22d5640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f01ec0caed0 0x7f01ec06be00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:10.561 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.550+0000 7f01f22d5640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01ec15c680 con 0x7f01ec0caa30 2023-05-22T15:27:10.561 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.550+0000 7f01f22d5640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01ec0ca270 con 0x7f01ec157d30 2023-05-22T15:27:10.561 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.550+0000 7f01f22d5640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01ec0c8570 con 0x7f01ec0caed0 2023-05-22T15:27:10.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.550+0000 7f01ebfff640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f01ec157d30 0x7f01ec15c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:10.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.550+0000 7f01eb7fe640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f01ec0caa30 0x7f01ec06b6f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.16:34884/0 (socket says 172.21.15.16:34884) 2023-05-22T15:27:10.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.550+0000 7f01eb7fe640 1 -- 172.21.15.16:0/3096008737 learned_addr learned my addr 172.21.15.16:0/3096008737 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:27:10.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.554+0000 7f01ebfff640 1 -- 172.21.15.16:0/3096008737 >> v2:172.21.15.62:3300/0 conn(0x7f01ec0caed0 msgr2=0x7f01ec06be00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-05-22T15:27:10.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.554+0000 7f01ebfff640 1 --2- 172.21.15.16:0/3096008737 >> v2:172.21.15.62:3300/0 conn(0x7f01ec0caed0 0x7f01ec06be00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:10.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.554+0000 7f01ebfff640 1 -- 172.21.15.16:0/3096008737 >> v1:172.21.15.62:6789/0 conn(0x7f01ec0caa30 legacy=0x7f01ec06b6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:10.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.554+0000 7f01ebfff640 1 -- 172.21.15.16:0/3096008737 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01ec0c8850 con 0x7f01ec157d30 2023-05-22T15:27:10.565 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.554+0000 7f01ebfff640 1 --2- 172.21.15.16:0/3096008737 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f01ec157d30 0x7f01ec15c140 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f01ec06cd20 tx=0x7f01dc01c570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6da8300dc3368520 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:10.567 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.558+0000 7f01eaffd640 1 -- 172.21.15.16:0/3096008737 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01dc0aa670 con 0x7f01ec157d30 2023-05-22T15:27:10.567 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.558+0000 7f01eaffd640 1 -- 172.21.15.16:0/3096008737 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01dc0aa7d0 con 0x7f01ec157d30 2023-05-22T15:27:10.567 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.558+0000 7f01f22d5640 1 -- 172.21.15.16:0/3096008737 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f01ec157d30 msgr2=0x7f01ec15c140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:10.567 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.558+0000 7f01f22d5640 1 --2- 172.21.15.16:0/3096008737 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f01ec157d30 0x7f01ec15c140 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f01ec06cd20 tx=0x7f01dc01c570 comp rx=0 tx=0).stop 2023-05-22T15:27:10.568 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.558+0000 7f01f22d5640 1 -- 172.21.15.16:0/3096008737 shutdown_connections 2023-05-22T15:27:10.568 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.558+0000 7f01f22d5640 1 --2- 172.21.15.16:0/3096008737 >> v2:172.21.15.62:3300/0 conn(0x7f01ec0caed0 0x7f01ec06be00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:10.568 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.558+0000 7f01f22d5640 1 --2- 172.21.15.16:0/3096008737 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f01ec157d30 0x7f01ec15c140 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:10.568 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.558+0000 7f01f22d5640 1 -- 172.21.15.16:0/3096008737 >> 172.21.15.16:0/3096008737 conn(0x7f01ec0af9f0 msgr2=0x7f01ec0afdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:10.569 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.558+0000 7f01f22d5640 1 -- 172.21.15.16:0/3096008737 shutdown_connections 2023-05-22T15:27:10.570 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.558+0000 7f01f22d5640 1 -- 172.21.15.16:0/3096008737 wait complete. 2023-05-22T15:27:10.570 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.562+0000 7f01f22d5640 1 Processor -- start 2023-05-22T15:27:10.571 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.562+0000 7f01f22d5640 1 -- start start 2023-05-22T15:27:10.571 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.562+0000 7f01f22d5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f01ec0caa30 0x7f01ec2008f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:10.572 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.562+0000 7f01f22d5640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f01ec0caed0 0x7f01ec0c4180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:10.572 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.562+0000 7f01f22d5640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f01ec156be0 0x7f01ec0b3000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:10.573 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.562+0000 7f01f22d5640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01ec200e30 con 0x7f01ec0caa30 2023-05-22T15:27:10.573 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.562+0000 7f01f22d5640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01ec0b2b70 con 0x7f01ec0caed0 2023-05-22T15:27:10.573 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.562+0000 7f01f22d5640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01ec15a570 con 0x7f01ec156be0 2023-05-22T15:27:10.574 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.562+0000 7f01ebfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f01ec0caa30 0x7f01ec2008f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:10.574 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.562+0000 7f01ebfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f01ec0caa30 0x7f01ec2008f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37704/0 (socket says 172.21.15.16:37704) 2023-05-22T15:27:10.574 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.562+0000 7f01eb7fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f01ec0caed0 0x7f01ec0c4180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:10.574 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.562+0000 7f01ebfff640 1 -- 172.21.15.16:0/1515889298 learned_addr learned my addr 172.21.15.16:0/1515889298 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:10.574 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.562+0000 7f01eb7fe640 1 -- 172.21.15.16:0/1515889298 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f01ec156be0 msgr2=0x7f01ec0b3000 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-05-22T15:27:10.575 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.562+0000 7f01eb7fe640 1 --2- 172.21.15.16:0/1515889298 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f01ec156be0 0x7f01ec0b3000 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:10.575 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.562+0000 7f01eb7fe640 1 -- 172.21.15.16:0/1515889298 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f01ec0caa30 msgr2=0x7f01ec2008f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:10.575 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.562+0000 7f01eb7fe640 1 --2- 172.21.15.16:0/1515889298 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f01ec0caa30 0x7f01ec2008f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:10.575 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.562+0000 7f01eb7fe640 1 -- 172.21.15.16:0/1515889298 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01dc009420 con 0x7f01ec0caed0 2023-05-22T15:27:10.575 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.562+0000 7f01ebfff640 1 --2- 172.21.15.16:0/1515889298 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f01ec0caa30 0x7f01ec2008f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-05-22T15:27:10.575 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.562+0000 7f01eb7fe640 1 --2- 172.21.15.16:0/1515889298 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f01ec0caed0 0x7f01ec0c4180 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f01e400e570 tx=0x7f01e400ea30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:10.576 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.562+0000 7f01e8ff9640 1 -- 172.21.15.16:0/1515889298 <== mon.1 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01e4012870 con 0x7f01ec0caed0 2023-05-22T15:27:10.576 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.562+0000 7f01f22d5640 1 -- 172.21.15.16:0/1515889298 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01ec0c8b30 con 0x7f01ec0caed0 2023-05-22T15:27:10.576 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.562+0000 7f01f22d5640 1 -- 172.21.15.16:0/1515889298 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01ec0ca510 con 0x7f01ec0caed0 2023-05-22T15:27:10.577 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.566+0000 7f01e8ff9640 1 -- 172.21.15.16:0/1515889298 <== mon.1 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01e40129d0 con 0x7f01ec0caed0 2023-05-22T15:27:10.577 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.566+0000 7f01e8ff9640 1 -- 172.21.15.16:0/1515889298 <== mon.1 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01e4012d80 con 0x7f01ec0caed0 2023-05-22T15:27:10.577 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.566+0000 7f01f22d5640 1 -- 172.21.15.16:0/1515889298 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01cc005260 con 0x7f01ec0caed0 2023-05-22T15:27:10.577 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.566+0000 7f01e8ff9640 1 -- 172.21.15.16:0/1515889298 <== mon.1 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7f01e401c460 con 0x7f01ec0caed0 2023-05-22T15:27:10.577 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.566+0000 7f01e8ff9640 1 --2- 172.21.15.16:0/1515889298 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f01d8065380 0x7f01d8067830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:10.578 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.566+0000 7f01ebfff640 1 --2- 172.21.15.16:0/1515889298 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f01d8065380 0x7f01d8067830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:10.578 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.566+0000 7f01e8ff9640 1 -- 172.21.15.16:0/1515889298 <== mon.1 v2:172.21.15.62:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f01e4085ac0 con 0x7f01ec0caed0 2023-05-22T15:27:10.578 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.570+0000 7f01ebfff640 1 --2- 172.21.15.16:0/1515889298 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f01d8065380 0x7f01d8067830 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f01dc01c5b0 tx=0x7f01dc0b5040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:10.590 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.582+0000 7f01e8ff9640 1 -- 172.21.15.16:0/1515889298 <== mon.1 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f01e40533e0 con 0x7f01ec0caed0 2023-05-22T15:27:10.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.594+0000 7f0c3f59e640 1 Processor -- start 2023-05-22T15:27:10.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.598+0000 7f0c3f59e640 1 -- start start 2023-05-22T15:27:10.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.598+0000 7f0c3f59e640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0c4006d630 0x7f0c4006c090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:10.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.598+0000 7f0c3f59e640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f0c400894b0 0x7f0c4016c330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:10.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.598+0000 7f0c3f59e640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c40089c70 con 0x7f0c400ca890 2023-05-22T15:27:10.609 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.598+0000 7f0c3f59e640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c400cdcf0 con 0x7f0c400894b0 2023-05-22T15:27:10.609 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.598+0000 7f0c3f59e640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c400cd730 con 0x7f0c4006d630 2023-05-22T15:27:10.610 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.598+0000 7f0c3dd9b640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0c4006d630 0x7f0c4006c090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:10.611 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.598+0000 7f0c3ed9d640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f0c400894b0 0x7f0c4016c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:10.611 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.598+0000 7f0c3e59c640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f0c400ca890 0x7f0c400cac60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.16:34890/0 (socket says 172.21.15.16:34890) 2023-05-22T15:27:10.611 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.598+0000 7f0c3e59c640 1 -- 172.21.15.16:0/2815973173 learned_addr learned my addr 172.21.15.16:0/2815973173 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:27:10.611 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.598+0000 7f0c3d59a640 1 -- 172.21.15.16:0/2815973173 <== mon.1 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 912162295 0 0) 0x7f0c40089c70 con 0x7f0c400ca890 2023-05-22T15:27:10.612 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.598+0000 7f0c3d59a640 1 -- 172.21.15.16:0/2815973173 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c28003530 con 0x7f0c400ca890 2023-05-22T15:27:10.612 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.598+0000 7f0c3ed9d640 1 -- 172.21.15.16:0/2815973173 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0c4006d630 msgr2=0x7f0c4006c090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:10.612 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.598+0000 7f0c3ed9d640 1 --2- 172.21.15.16:0/2815973173 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0c4006d630 0x7f0c4006c090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:10.612 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.598+0000 7f0c3ed9d640 1 -- 172.21.15.16:0/2815973173 >> v1:172.21.15.62:6789/0 conn(0x7f0c400ca890 legacy=0x7f0c400cac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:10.613 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.598+0000 7f0c3ed9d640 1 -- 172.21.15.16:0/2815973173 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c400cda10 con 0x7f0c400894b0 2023-05-22T15:27:10.613 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.602+0000 7f0c3dd9b640 1 --2- 172.21.15.16:0/2815973173 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0c4006d630 0x7f0c4006c090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-05-22T15:27:10.614 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.602+0000 7fb01086d640 1 Processor -- start 2023-05-22T15:27:10.615 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.602+0000 7fb01086d640 1 -- start start 2023-05-22T15:27:10.615 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.602+0000 7f6c2b812640 1 Processor -- start 2023-05-22T15:27:10.615 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.602+0000 7f6c2b812640 1 -- start start 2023-05-22T15:27:10.616 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.602+0000 7f0c3ed9d640 1 --2- 172.21.15.16:0/2815973173 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0c400894b0 0x7f0c4016c330 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f0c340097c0 tx=0x7f0c3401c410 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5b614973b2b6fbdd server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:10.616 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.602+0000 7f0c3d59a640 1 -- 172.21.15.16:0/2815973173 <== mon.1 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c4016d080 con 0x7f0c400894b0 2023-05-22T15:27:10.616 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.602+0000 7f0c3d59a640 1 -- 172.21.15.16:0/2815973173 <== mon.1 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c340a7050 con 0x7f0c400894b0 2023-05-22T15:27:10.616 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.602+0000 7f0c3d59a640 1 -- 172.21.15.16:0/2815973173 <== mon.1 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c340ab800 con 0x7f0c400894b0 2023-05-22T15:27:10.617 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.602+0000 7fb01086d640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb00c159920 0x7fb00c0c1da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:10.617 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.602+0000 7f6c2b812640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6c1c056780 0x7f6c1c0548b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:10.617 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.602+0000 7f6c2b812640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6c1c054df0 0x7f6c1c052750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:10.618 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.602+0000 7f6c295ae640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f6c1c037270 0x7f6c1c039b30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:39726/0 (socket says 172.21.15.16:39726) 2023-05-22T15:27:10.618 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.602+0000 7fb00b7fe640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb00c159920 0x7fb00c0c1da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:10.618 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.602+0000 7fb00b7fe640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb00c159920 0x7fb00c0c1da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37710/0 (socket says 172.21.15.16:37710) 2023-05-22T15:27:10.619 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.602+0000 7f6c2b812640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c1c0522c0 con 0x7f6c1c037270 2023-05-22T15:27:10.619 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.602+0000 7f6c295ae640 1 -- 172.21.15.16:0/3503834584 learned_addr learned my addr 172.21.15.16:0/3503834584 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:27:10.619 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.602+0000 7f6c2b812640 1 -- 172.21.15.16:0/3503834584 --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c1c0e0d80 con 0x7f6c1c056780 2023-05-22T15:27:10.619 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.602+0000 7f6c2b812640 1 -- 172.21.15.16:0/3503834584 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c1c051d40 con 0x7f6c1c054df0 2023-05-22T15:27:10.619 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.602+0000 7f0c3f59e640 1 -- 172.21.15.16:0/2815973173 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0c400894b0 msgr2=0x7f0c4016c330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:10.620 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.602+0000 7f0c3f59e640 1 --2- 172.21.15.16:0/2815973173 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0c400894b0 0x7f0c4016c330 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f0c340097c0 tx=0x7f0c3401c410 comp rx=0 tx=0).stop 2023-05-22T15:27:10.620 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.602+0000 7f6c28dad640 1 --2- 172.21.15.16:0/3503834584 >> v2:172.21.15.16:3300/0 conn(0x7f6c1c056780 0x7f6c1c0548b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:10.620 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.602+0000 7f0c3f59e640 1 -- 172.21.15.16:0/2815973173 shutdown_connections 2023-05-22T15:27:10.620 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.602+0000 7f0c3f59e640 1 --2- 172.21.15.16:0/2815973173 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0c400894b0 0x7f0c4016c330 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:10.621 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.602+0000 7f0c3f59e640 1 --2- 172.21.15.16:0/2815973173 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0c4006d630 0x7f0c4006c090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:10.621 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.602+0000 7f0c3f59e640 1 -- 172.21.15.16:0/2815973173 >> 172.21.15.16:0/2815973173 conn(0x7f0c400af9f0 msgr2=0x7f0c400afdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:10.621 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.602+0000 7f6c29daf640 1 --2- 172.21.15.16:0/3503834584 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6c1c054df0 0x7f6c1c052750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:10.622 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.602+0000 7f6c28dad640 1 -- 172.21.15.16:0/3503834584 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6c1c054df0 msgr2=0x7f6c1c052750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:10.622 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.602+0000 7f6c28dad640 1 --2- 172.21.15.16:0/3503834584 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6c1c054df0 0x7f6c1c052750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:10.622 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.602+0000 7f6c28dad640 1 -- 172.21.15.16:0/3503834584 >> v1:172.21.15.16:6789/0 conn(0x7f6c1c037270 legacy=0x7f6c1c039b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:10.622 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.602+0000 7f6c28dad640 1 -- 172.21.15.16:0/3503834584 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c1c00ab70 con 0x7f6c1c056780 2023-05-22T15:27:10.623 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.602+0000 7fb00b7fe640 1 -- 172.21.15.16:0/1383175911 learned_addr learned my addr 172.21.15.16:0/1383175911 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:10.623 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.602+0000 7fb01086d640 1 --2- 172.21.15.16:0/1383175911 >> v2:172.21.15.62:3300/0 conn(0x7fb00c0b03e0 0x7fb00c0716f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:10.623 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.602+0000 7fb01086d640 1 -- 172.21.15.16:0/1383175911 --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb00c0ca290 con 0x7fb00c06c740 2023-05-22T15:27:10.623 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.602+0000 7fb01086d640 1 -- 172.21.15.16:0/1383175911 --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb00c0c88a0 con 0x7fb00c159920 2023-05-22T15:27:10.623 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.602+0000 7fb01086d640 1 -- 172.21.15.16:0/1383175911 --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb00c071c30 con 0x7fb00c0b03e0 2023-05-22T15:27:10.624 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.602+0000 7f584345c640 1 Processor -- start 2023-05-22T15:27:10.625 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.606+0000 7f584345c640 1 -- start start 2023-05-22T15:27:10.625 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.606+0000 7f584345c640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f5834050fa0 0x7f5834036530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:10.625 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.606+0000 7f584345c640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f5834036a70 0x7f5834052c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:10.625 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.606+0000 7f584345c640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5834006fc0 con 0x7f58340e6c10 2023-05-22T15:27:10.625 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.606+0000 7f584345c640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58340513f0 con 0x7f5834036a70 2023-05-22T15:27:10.626 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.606+0000 7f584345c640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58340527b0 con 0x7f5834050fa0 2023-05-22T15:27:10.626 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.606+0000 7f58409f7640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f5834050fa0 0x7f5834036530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:10.626 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.606+0000 7f6c28dad640 1 --2- 172.21.15.16:0/3503834584 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c1c056780 0x7f6c1c0548b0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f6c140097c0 tx=0x7f6c1401c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f939663c6a05f9a server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:10.626 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.606+0000 7f58409f7640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f5834050fa0 0x7f5834036530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.16:48468/0 (socket says 172.21.15.16:48468) 2023-05-22T15:27:10.627 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.606+0000 7f58409f7640 1 -- 172.21.15.16:0/2974174655 learned_addr learned my addr 172.21.15.16:0/2974174655 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:10.627 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.606+0000 7f58409f7640 1 -- 172.21.15.16:0/2974174655 >> v2:172.21.15.16:3300/0 conn(0x7f5834036a70 msgr2=0x7f5834052c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:10.627 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.606+0000 7f58409f7640 1 --2- 172.21.15.16:0/2974174655 >> v2:172.21.15.16:3300/0 conn(0x7f5834036a70 0x7f5834052c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:10.627 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.606+0000 7f58409f7640 1 -- 172.21.15.16:0/2974174655 >> v1:172.21.15.16:6789/0 conn(0x7f58340e6c10 legacy=0x7f583404f070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:10.627 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.606+0000 7f58409f7640 1 -- 172.21.15.16:0/2974174655 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5834036eb0 con 0x7f5834050fa0 2023-05-22T15:27:10.628 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.606+0000 7f6c1bfff640 1 -- 172.21.15.16:0/3503834584 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c140ab600 con 0x7f6c1c056780 2023-05-22T15:27:10.628 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.606+0000 7fb00bfff640 1 --2- 172.21.15.16:0/1383175911 >> v2:172.21.15.62:3300/0 conn(0x7fb00c0b03e0 0x7fb00c0716f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:10.629 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.606+0000 7fb00b7fe640 1 -- 172.21.15.16:0/1383175911 >> v2:172.21.15.62:3300/0 conn(0x7fb00c0b03e0 msgr2=0x7fb00c0716f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:10.629 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.606+0000 7fb00b7fe640 1 --2- 172.21.15.16:0/1383175911 >> v2:172.21.15.62:3300/0 conn(0x7fb00c0b03e0 0x7fb00c0716f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:10.629 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.606+0000 7fb00b7fe640 1 -- 172.21.15.16:0/1383175911 >> v1:172.21.15.62:6789/0 conn(0x7fb00c06c740 legacy=0x7fb00c0afcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:10.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.606+0000 7f6c1bfff640 1 -- 172.21.15.16:0/3503834584 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c140a7050 con 0x7f6c1c056780 2023-05-22T15:27:10.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.606+0000 7fb00b7fe640 1 -- 172.21.15.16:0/1383175911 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb00c0ca570 con 0x7fb00c159920 2023-05-22T15:27:10.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.606+0000 7fb00bfff640 1 --2- 172.21.15.16:0/1383175911 >> v2:172.21.15.62:3300/0 conn(0x7fb00c0b03e0 0x7fb00c0716f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:27:10.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.606+0000 7f6c1bfff640 1 -- 172.21.15.16:0/3503834584 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c140ab980 con 0x7f6c1c056780 2023-05-22T15:27:10.632 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.606+0000 7fb00b7fe640 1 --2- 172.21.15.16:0/1383175911 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb00c159920 0x7fb00c0c1da0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fb00c0ce200 tx=0x7faffc01c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd3c7aef66678b27 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:10.632 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.610+0000 7f0c3f59e640 1 -- 172.21.15.16:0/2815973173 shutdown_connections 2023-05-22T15:27:10.633 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.610+0000 7f0c3f59e640 1 -- 172.21.15.16:0/2815973173 wait complete. 2023-05-22T15:27:10.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.610+0000 7fb00a7fc640 1 -- 172.21.15.16:0/1383175911 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faffc0aa660 con 0x7fb00c159920 2023-05-22T15:27:10.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.610+0000 7fb00a7fc640 1 -- 172.21.15.16:0/1383175911 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faffc0aa7c0 con 0x7fb00c159920 2023-05-22T15:27:10.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.610+0000 7fb00a7fc640 1 -- 172.21.15.16:0/1383175911 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faffc0aab50 con 0x7fb00c159920 2023-05-22T15:27:10.635 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.610+0000 7fb01086d640 1 -- 172.21.15.16:0/1383175911 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb00c159920 msgr2=0x7fb00c0c1da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:10.635 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.610+0000 7fb01086d640 1 --2- 172.21.15.16:0/1383175911 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb00c159920 0x7fb00c0c1da0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fb00c0ce200 tx=0x7faffc01c500 comp rx=0 tx=0).stop 2023-05-22T15:27:10.636 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.610+0000 7f0c3f59e640 1 Processor -- start 2023-05-22T15:27:10.636 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.610+0000 7f6c2b812640 1 -- 172.21.15.16:0/3503834584 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c1c056780 msgr2=0x7f6c1c0548b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:10.636 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.610+0000 7f6c2b812640 1 --2- 172.21.15.16:0/3503834584 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c1c056780 0x7f6c1c0548b0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f6c140097c0 tx=0x7f6c1401c480 comp rx=0 tx=0).stop 2023-05-22T15:27:10.637 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.610+0000 7f6c2b812640 1 -- 172.21.15.16:0/3503834584 shutdown_connections 2023-05-22T15:27:10.637 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.610+0000 7f6c2b812640 1 --2- 172.21.15.16:0/3503834584 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6c1c054df0 0x7f6c1c052750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:10.637 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.610+0000 7f6c2b812640 1 --2- 172.21.15.16:0/3503834584 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c1c056780 0x7f6c1c0548b0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:10.637 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.610+0000 7f6c2b812640 1 -- 172.21.15.16:0/3503834584 >> 172.21.15.16:0/3503834584 conn(0x7f6c1c00a620 msgr2=0x7f6c1c056ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:10.638 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.610+0000 7f6c2b812640 1 -- 172.21.15.16:0/3503834584 shutdown_connections 2023-05-22T15:27:10.638 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.610+0000 7f6c2b812640 1 -- 172.21.15.16:0/3503834584 wait complete. 2023-05-22T15:27:10.639 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.610+0000 7fb01086d640 1 -- 172.21.15.16:0/1383175911 shutdown_connections 2023-05-22T15:27:10.639 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.610+0000 7fb01086d640 1 --2- 172.21.15.16:0/1383175911 >> v2:172.21.15.62:3300/0 conn(0x7fb00c0b03e0 0x7fb00c0716f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:10.640 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.610+0000 7fb01086d640 1 --2- 172.21.15.16:0/1383175911 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb00c159920 0x7fb00c0c1da0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:10.640 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.610+0000 7fb01086d640 1 -- 172.21.15.16:0/1383175911 >> 172.21.15.16:0/1383175911 conn(0x7fb00c0cacd0 msgr2=0x7fb00c0cb0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:10.640 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.610+0000 7fb01086d640 1 -- 172.21.15.16:0/1383175911 shutdown_connections 2023-05-22T15:27:10.640 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.610+0000 7fb01086d640 1 -- 172.21.15.16:0/1383175911 wait complete. 2023-05-22T15:27:10.641 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.610+0000 7f80acd28640 1 Processor -- start 2023-05-22T15:27:10.641 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.610+0000 7f0c3f59e640 1 -- start start 2023-05-22T15:27:10.642 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.610+0000 7f0c3f59e640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0c4006d630 0x7f0c4020cd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:10.642 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.610+0000 7f0c3f59e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0c400894b0 0x7f0c4020a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:10.642 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.610+0000 7f0c3f59e640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0c400ca890 0x7f0c402090d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:10.642 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.610+0000 7f0c3f59e640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c40167e60 con 0x7f0c400894b0 2023-05-22T15:27:10.642 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.610+0000 7f0c3f59e640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c4020a710 con 0x7f0c4006d630 2023-05-22T15:27:10.642 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.610+0000 7f0c3f59e640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c40209610 con 0x7f0c400ca890 2023-05-22T15:27:10.642 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.610+0000 7f0c3dd9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0c400894b0 0x7f0c4020a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:10.642 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.610+0000 7f0c3dd9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0c400894b0 0x7f0c4020a1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37752/0 (socket says 172.21.15.16:37752) 2023-05-22T15:27:10.643 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.610+0000 7f0c3dd9b640 1 -- 172.21.15.16:0/2455868988 learned_addr learned my addr 172.21.15.16:0/2455868988 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:10.644 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.610+0000 7f6c2b812640 1 Processor -- start 2023-05-22T15:27:10.644 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.610+0000 7f6c2b812640 1 -- start start 2023-05-22T15:27:10.644 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.610+0000 7f6c2b812640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6c1c037270 0x7f6c1c188520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:10.644 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.610+0000 7f6c2b812640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c1c054df0 0x7f6c1c184f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:10.644 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.610+0000 7f6c2b812640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6c1c056780 0x7f6c1c0e28f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:10.644 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.610+0000 7f6c2b812640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c1c0e36c0 con 0x7f6c1c054df0 2023-05-22T15:27:10.645 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.610+0000 7f6c2b812640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c1c04fd60 con 0x7f6c1c037270 2023-05-22T15:27:10.645 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.610+0000 7f6c2b812640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c1c0e0ec0 con 0x7f6c1c056780 2023-05-22T15:27:10.645 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.610+0000 7f6c28dad640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c1c054df0 0x7f6c1c184f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:10.645 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.610+0000 7f6c28dad640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c1c054df0 0x7f6c1c184f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37740/0 (socket says 172.21.15.16:37740) 2023-05-22T15:27:10.645 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.610+0000 7f6c28dad640 1 -- 172.21.15.16:0/1014005193 learned_addr learned my addr 172.21.15.16:0/1014005193 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:10.645 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.610+0000 7f6c295ae640 1 --2- 172.21.15.16:0/1014005193 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6c1c037270 0x7f6c1c188520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:10.646 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.610+0000 7f0c3e59c640 1 --2- 172.21.15.16:0/2455868988 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0c4006d630 0x7f0c4020cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:10.646 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.610+0000 7f6c29daf640 1 --2- 172.21.15.16:0/1014005193 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6c1c056780 0x7f6c1c0e28f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:10.647 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.610+0000 7f58409f7640 1 --2- 172.21.15.16:0/2974174655 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5834050fa0 0x7f5834036530 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f583400baf0 tx=0x7f582c01c430 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8dbdd2771b941855 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:10.647 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.610+0000 7f5833fff640 1 -- 172.21.15.16:0/2974174655 <== mon.1 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f582c0ab600 con 0x7f5834050fa0 2023-05-22T15:27:10.648 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.610+0000 7f5833fff640 1 -- 172.21.15.16:0/2974174655 <== mon.1 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f582c0a7050 con 0x7f5834050fa0 2023-05-22T15:27:10.648 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.610+0000 7f5833fff640 1 -- 172.21.15.16:0/2974174655 <== mon.1 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f582c0ab980 con 0x7f5834050fa0 2023-05-22T15:27:10.648 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.610+0000 7f584345c640 1 -- 172.21.15.16:0/2974174655 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5834050fa0 msgr2=0x7f5834036530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:10.648 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.610+0000 7f584345c640 1 --2- 172.21.15.16:0/2974174655 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5834050fa0 0x7f5834036530 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f583400baf0 tx=0x7f582c01c430 comp rx=0 tx=0).stop 2023-05-22T15:27:10.649 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.610+0000 7f584345c640 1 -- 172.21.15.16:0/2974174655 shutdown_connections 2023-05-22T15:27:10.649 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.610+0000 7f584345c640 1 --2- 172.21.15.16:0/2974174655 >> v2:172.21.15.16:3300/0 conn(0x7f5834036a70 0x7f5834052c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:10.649 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.610+0000 7f584345c640 1 --2- 172.21.15.16:0/2974174655 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5834050fa0 0x7f5834036530 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:10.649 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.610+0000 7f584345c640 1 -- 172.21.15.16:0/2974174655 >> 172.21.15.16:0/2974174655 conn(0x7f5834051530 msgr2=0x7f5834051930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:10.649 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.610+0000 7f584345c640 1 -- 172.21.15.16:0/2974174655 shutdown_connections 2023-05-22T15:27:10.649 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.610+0000 7f584345c640 1 -- 172.21.15.16:0/2974174655 wait complete. 2023-05-22T15:27:10.650 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.610+0000 7f0c3ed9d640 1 --2- 172.21.15.16:0/2455868988 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0c400ca890 0x7f0c402090d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:10.650 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.610+0000 7f6c28dad640 1 -- 172.21.15.16:0/1014005193 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6c1c056780 msgr2=0x7f6c1c0e28f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:10.650 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.610+0000 7f6c28dad640 1 --2- 172.21.15.16:0/1014005193 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6c1c056780 0x7f6c1c0e28f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:10.650 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.610+0000 7f6c28dad640 1 -- 172.21.15.16:0/1014005193 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6c1c037270 msgr2=0x7f6c1c188520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:10.650 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.610+0000 7f6c28dad640 1 --2- 172.21.15.16:0/1014005193 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6c1c037270 0x7f6c1c188520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:10.651 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.610+0000 7f6c28dad640 1 -- 172.21.15.16:0/1014005193 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c14009420 con 0x7f6c1c054df0 2023-05-22T15:27:10.651 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.610+0000 7f6c295ae640 1 --2- 172.21.15.16:0/1014005193 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6c1c037270 0x7f6c1c188520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-05-22T15:27:10.651 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.610+0000 7f0c3dd9b640 1 -- 172.21.15.16:0/2455868988 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0c400ca890 msgr2=0x7f0c402090d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:10.651 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.614+0000 7f0c3dd9b640 1 --2- 172.21.15.16:0/2455868988 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0c400ca890 0x7f0c402090d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:10.651 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.614+0000 7f0c3dd9b640 1 -- 172.21.15.16:0/2455868988 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0c4006d630 msgr2=0x7f0c4020cd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:10.651 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.614+0000 7f0c3dd9b640 1 --2- 172.21.15.16:0/2455868988 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0c4006d630 0x7f0c4020cd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:10.652 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.614+0000 7f0c3dd9b640 1 -- 172.21.15.16:0/2455868988 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c30009420 con 0x7f0c400894b0 2023-05-22T15:27:10.652 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.614+0000 7f0c3e59c640 1 --2- 172.21.15.16:0/2455868988 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0c4006d630 0x7f0c4020cd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-05-22T15:27:10.652 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.614+0000 7f6c28dad640 1 --2- 172.21.15.16:0/1014005193 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c1c054df0 0x7f6c1c184f00 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f6c1401ab30 tx=0x7f6c14019220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:10.653 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.610+0000 7f80acd28640 1 -- start start 2023-05-22T15:27:10.653 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.614+0000 7f80acd28640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f80a00ea1a0 0x7f80a0048c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:10.653 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.614+0000 7f80acd28640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f80a0055780 0x7f80a0049170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:10.653 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.614+0000 7f80acd28640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80a0056710 con 0x7f80a0009050 2023-05-22T15:27:10.654 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.614+0000 7f80acd28640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80a00500c0 con 0x7f80a0055780 2023-05-22T15:27:10.654 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.614+0000 7f80acd28640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80a0056410 con 0x7f80a00ea1a0 2023-05-22T15:27:10.654 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.614+0000 7f0c3dd9b640 1 --2- 172.21.15.16:0/2455868988 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0c400894b0 0x7f0c4020a1d0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f0c3000e9b0 tx=0x7f0c3000ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:10.654 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.614+0000 7f80a6d9d640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f80a0009050 0x7f80a00eaad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:39734/0 (socket says 172.21.15.16:39734) 2023-05-22T15:27:10.655 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.614+0000 7f80a6d9d640 1 -- 172.21.15.16:0/3053795072 learned_addr learned my addr 172.21.15.16:0/3053795072 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:27:10.655 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.614+0000 7f584345c640 1 Processor -- start 2023-05-22T15:27:10.655 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.614+0000 7f584345c640 1 -- start start 2023-05-22T15:27:10.656 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.614+0000 7f80a659c640 1 --2- 172.21.15.16:0/3053795072 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f80a00ea1a0 0x7f80a0048c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:10.656 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.614+0000 7f80a559a640 1 -- 172.21.15.16:0/3053795072 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2376459606 0 0) 0x7f80a0056710 con 0x7f80a0009050 2023-05-22T15:27:10.656 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.614+0000 7f80a559a640 1 -- 172.21.15.16:0/3053795072 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8090003530 con 0x7f80a0009050 2023-05-22T15:27:10.656 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.614+0000 7fb01086d640 1 Processor -- start 2023-05-22T15:27:10.657 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.614+0000 7f80a559a640 1 -- 172.21.15.16:0/3053795072 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4219271135 0 0) 0x7f8090003530 con 0x7f80a0009050 2023-05-22T15:27:10.657 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.614+0000 7f80a559a640 1 -- 172.21.15.16:0/3053795072 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f80a00ea1a0 msgr2=0x7f80a0048c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:10.657 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.614+0000 7f80a559a640 1 --2- 172.21.15.16:0/3053795072 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f80a00ea1a0 0x7f80a0048c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:10.657 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.614+0000 7f80a559a640 1 -- 172.21.15.16:0/3053795072 >> v2:172.21.15.62:3300/0 conn(0x7f80a0055780 msgr2=0x7f80a0049170 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-05-22T15:27:10.657 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.614+0000 7f80a559a640 1 --2- 172.21.15.16:0/3053795072 >> v2:172.21.15.62:3300/0 conn(0x7f80a0055780 0x7f80a0049170 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:10.657 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.614+0000 7f80a559a640 1 -- 172.21.15.16:0/3053795072 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80a003b7b0 con 0x7f80a0009050 2023-05-22T15:27:10.658 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.614+0000 7fb01086d640 1 -- start start 2023-05-22T15:27:10.658 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.614+0000 7f80a559a640 1 -- 172.21.15.16:0/3053795072 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7f80a8072ac0 con 0x7f80a0009050 2023-05-22T15:27:10.659 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.614+0000 7f80a559a640 1 -- 172.21.15.16:0/3053795072 >> v1:172.21.15.16:6789/0 conn(0x7f80a0009050 legacy=0x7f80a00eaad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:10.659 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.614+0000 7f80a559a640 1 --2- 172.21.15.16:0/3053795072 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8090003980 0x7f8090003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:10.659 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.614+0000 7f80a559a640 1 --2- 172.21.15.16:0/3053795072 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8090004310 0x7f80900047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:10.659 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.614+0000 7f80a559a640 1 --2- 172.21.15.16:0/3053795072 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f80a00ea1a0 0x7f8090005020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:10.659 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.614+0000 7f80a559a640 1 -- 172.21.15.16:0/3053795072 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80a00500c0 con 0x7f8090004310 2023-05-22T15:27:10.659 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.614+0000 7f80a559a640 1 -- 172.21.15.16:0/3053795072 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80a0056410 con 0x7f8090003980 2023-05-22T15:27:10.659 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.614+0000 7f80a559a640 1 -- 172.21.15.16:0/3053795072 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f809000b980 con 0x7f80a00ea1a0 2023-05-22T15:27:10.660 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.614+0000 7f0c2effd640 1 -- 172.21.15.16:0/2455868988 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c3000c9a0 con 0x7f0c400894b0 2023-05-22T15:27:10.660 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.614+0000 7f0c2effd640 1 -- 172.21.15.16:0/2455868988 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c3000cb00 con 0x7f0c400894b0 2023-05-22T15:27:10.660 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.614+0000 7f0c3f59e640 1 -- 172.21.15.16:0/2455868988 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c34009420 con 0x7f0c400894b0 2023-05-22T15:27:10.660 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.614+0000 7f584345c640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5834036a70 0x7f58340dbea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:10.661 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.614+0000 7f584345c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5834050fa0 0x7f58340e7200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:10.661 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.614+0000 7f584345c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f58340e6c10 0x7f58340cce50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:10.661 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.614+0000 7f584345c640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58340ce830 con 0x7f5834050fa0 2023-05-22T15:27:10.661 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.614+0000 7f584345c640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58340cdf00 con 0x7f58340e6c10 2023-05-22T15:27:10.661 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.614+0000 7f584345c640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58340ce1e0 con 0x7f5834036a70 2023-05-22T15:27:10.661 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.614+0000 7f80a6d9d640 1 --2- 172.21.15.16:0/3053795072 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f80a00ea1a0 0x7f8090005020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:10.662 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.614+0000 7f58409f7640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5834050fa0 0x7f58340e7200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:10.662 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.614+0000 7f58409f7640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5834050fa0 0x7f58340e7200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37768/0 (socket says 172.21.15.16:37768) 2023-05-22T15:27:10.662 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.614+0000 7f58409f7640 1 -- 172.21.15.16:0/1969369175 learned_addr learned my addr 172.21.15.16:0/1969369175 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:10.663 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.614+0000 7f58411f8640 1 --2- 172.21.15.16:0/1969369175 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5834036a70 0x7f58340dbea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:10.663 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.614+0000 7f58419f9640 1 --2- 172.21.15.16:0/1969369175 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f58340e6c10 0x7f58340cce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:10.663 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.614+0000 7f80a659c640 1 --2- 172.21.15.16:0/3053795072 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8090003980 0x7f8090003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:10.663 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.614+0000 7f0c2effd640 1 -- 172.21.15.16:0/2455868988 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c3000ce90 con 0x7f0c400894b0 2023-05-22T15:27:10.663 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.614+0000 7f0c3f59e640 1 -- 172.21.15.16:0/2455868988 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c400ca660 con 0x7f0c400894b0 2023-05-22T15:27:10.664 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.614+0000 7fb01086d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb00c06c740 0x7fb00c14cb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:10.664 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.614+0000 7fb01086d640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb00c0b03e0 0x7fb00c15c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:10.664 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.614+0000 7fb01086d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb00c159920 0x7fb00c154230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:10.664 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.614+0000 7fb01086d640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb00c15ce70 con 0x7fb00c06c740 2023-05-22T15:27:10.664 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.614+0000 7fb01086d640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb00c15cbd0 con 0x7fb00c159920 2023-05-22T15:27:10.664 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.614+0000 7fb01086d640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb00c0ae2c0 con 0x7fb00c0b03e0 2023-05-22T15:27:10.665 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.614+0000 7fb00bfff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb00c159920 0x7fb00c154230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:10.665 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.614+0000 7fb00affd640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb00c0b03e0 0x7fb00c15c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:10.665 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.614+0000 7fb00bfff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb00c159920 0x7fb00c154230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.16:48526/0 (socket says 172.21.15.16:48526) 2023-05-22T15:27:10.665 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.614+0000 7fb00bfff640 1 -- 172.21.15.16:0/703323199 learned_addr learned my addr 172.21.15.16:0/703323199 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:10.665 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.614+0000 7fb00b7fe640 1 --2- 172.21.15.16:0/703323199 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb00c06c740 0x7fb00c14cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:10.665 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.614+0000 7f58409f7640 1 -- 172.21.15.16:0/1969369175 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5834036a70 msgr2=0x7f58340dbea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:10.666 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.614+0000 7f58409f7640 1 --2- 172.21.15.16:0/1969369175 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5834036a70 0x7f58340dbea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:10.666 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.614+0000 7f58409f7640 1 -- 172.21.15.16:0/1969369175 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f58340e6c10 msgr2=0x7f58340cce50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:10.666 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.614+0000 7f58409f7640 1 --2- 172.21.15.16:0/1969369175 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f58340e6c10 0x7f58340cce50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:10.666 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.614+0000 7f58409f7640 1 -- 172.21.15.16:0/1969369175 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f582c0094d0 con 0x7f5834050fa0 2023-05-22T15:27:10.666 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.614+0000 7f58409f7640 1 --2- 172.21.15.16:0/1969369175 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5834050fa0 0x7f58340e7200 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f582c0ae040 tx=0x7f582c01ac70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:10.666 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.614+0000 7f6c19ffb640 1 -- 172.21.15.16:0/1014005193 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c140b4810 con 0x7f6c1c054df0 2023-05-22T15:27:10.667 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.614+0000 7f6c19ffb640 1 -- 172.21.15.16:0/1014005193 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c140b4990 con 0x7f6c1c054df0 2023-05-22T15:27:10.667 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.614+0000 7f6c2b812640 1 -- 172.21.15.16:0/1014005193 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c1c0530a0 con 0x7f6c1c054df0 2023-05-22T15:27:10.667 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.614+0000 7f6c2b812640 1 -- 172.21.15.16:0/1014005193 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c1c037640 con 0x7f6c1c054df0 2023-05-22T15:27:10.667 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.614+0000 7fb00bfff640 1 -- 172.21.15.16:0/703323199 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb00c0b03e0 msgr2=0x7fb00c15c690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:10.667 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.614+0000 7fb00bfff640 1 --2- 172.21.15.16:0/703323199 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb00c0b03e0 0x7fb00c15c690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:10.668 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.614+0000 7fb00bfff640 1 -- 172.21.15.16:0/703323199 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb00c06c740 msgr2=0x7fb00c14cb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:10.668 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.614+0000 7fb00bfff640 1 --2- 172.21.15.16:0/703323199 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb00c06c740 0x7fb00c14cb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:10.668 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.614+0000 7fb00bfff640 1 -- 172.21.15.16:0/703323199 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faffc0094d0 con 0x7fb00c159920 2023-05-22T15:27:10.668 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.618+0000 7f6c2b812640 1 -- 172.21.15.16:0/1014005193 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c1c037e30 con 0x7f6c1c054df0 2023-05-22T15:27:10.669 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.618+0000 7fb00bfff640 1 --2- 172.21.15.16:0/703323199 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb00c159920 0x7fb00c154230 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fb00400cf20 tx=0x7fb00400a7d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:10.669 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.618+0000 7f6c19ffb640 1 -- 172.21.15.16:0/1014005193 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c140b4d40 con 0x7f6c1c054df0 2023-05-22T15:27:10.669 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.618+0000 7f0c3f59e640 1 -- 172.21.15.16:0/2455868988 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c400ca4d0 con 0x7f0c400894b0 2023-05-22T15:27:10.670 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.614+0000 7f80a6d9d640 1 -- 172.21.15.16:0/3053795072 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8090003980 msgr2=0x7f8090003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:10.670 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.614+0000 7f80a6d9d640 1 --2- 172.21.15.16:0/3053795072 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8090003980 0x7f8090003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:10.670 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.614+0000 7f80a6d9d640 1 -- 172.21.15.16:0/3053795072 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8090004310 msgr2=0x7f80900047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:10.670 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.614+0000 7f80a6d9d640 1 --2- 172.21.15.16:0/3053795072 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8090004310 0x7f80900047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:10.670 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.614+0000 7f80a6d9d640 1 -- 172.21.15.16:0/3053795072 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80a003b7b0 con 0x7f80a00ea1a0 2023-05-22T15:27:10.670 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.614+0000 7f80a659c640 1 --2- 172.21.15.16:0/3053795072 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8090003980 0x7f8090003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:27:10.670 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.618+0000 7f80a6d9d640 1 --2- 172.21.15.16:0/3053795072 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f80a00ea1a0 0x7f8090005020 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f80a8000b70 tx=0x7f80a8098d70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e53fccfdea5ab710 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:10.671 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.618+0000 7f80a559a640 1 -- 172.21.15.16:0/3053795072 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80a806b2d0 con 0x7f80a00ea1a0 2023-05-22T15:27:10.671 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.618+0000 7f80a559a640 1 -- 172.21.15.16:0/3053795072 <== mon.2 v2:172.21.15.62:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80a806b430 con 0x7f80a00ea1a0 2023-05-22T15:27:10.671 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.618+0000 7f80a559a640 1 -- 172.21.15.16:0/3053795072 <== mon.2 v2:172.21.15.62:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80a81229b0 con 0x7f80a00ea1a0 2023-05-22T15:27:10.671 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.618+0000 7f80acd28640 1 -- 172.21.15.16:0/3053795072 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f80a00ea1a0 msgr2=0x7f8090005020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:10.671 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.618+0000 7f80acd28640 1 --2- 172.21.15.16:0/3053795072 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f80a00ea1a0 0x7f8090005020 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f80a8000b70 tx=0x7f80a8098d70 comp rx=0 tx=0).stop 2023-05-22T15:27:10.671 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.618+0000 7f80a6d9d640 1 -- 172.21.15.16:0/3053795072 reap_dead start 2023-05-22T15:27:10.671 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.618+0000 7f80acd28640 1 -- 172.21.15.16:0/3053795072 shutdown_connections 2023-05-22T15:27:10.671 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.618+0000 7f80acd28640 1 -- 172.21.15.16:0/3053795072 >> 172.21.15.16:0/3053795072 conn(0x7f80a0039810 msgr2=0x7f80a0055c60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:10.672 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.618+0000 7f80acd28640 1 -- 172.21.15.16:0/3053795072 shutdown_connections 2023-05-22T15:27:10.672 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.618+0000 7f80acd28640 1 -- 172.21.15.16:0/3053795072 wait complete. 2023-05-22T15:27:10.672 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.618+0000 7fb00b7fe640 1 --2- 172.21.15.16:0/703323199 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb00c06c740 0x7fb00c14cb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:27:10.672 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.618+0000 7faff3fff640 1 -- 172.21.15.16:0/703323199 <== mon.1 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb00400d840 con 0x7fb00c159920 2023-05-22T15:27:10.672 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.618+0000 7fb01086d640 1 -- 172.21.15.16:0/703323199 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb00c0c9f10 con 0x7fb00c159920 2023-05-22T15:27:10.672 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.618+0000 7fb01086d640 1 -- 172.21.15.16:0/703323199 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb00c159cf0 con 0x7fb00c159920 2023-05-22T15:27:10.673 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.618+0000 7fb01086d640 1 -- 172.21.15.16:0/703323199 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb00c0c97e0 con 0x7fb00c159920 2023-05-22T15:27:10.673 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.618+0000 7faff3fff640 1 -- 172.21.15.16:0/703323199 <== mon.1 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb00400d9a0 con 0x7fb00c159920 2023-05-22T15:27:10.673 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.618+0000 7faff3fff640 1 -- 172.21.15.16:0/703323199 <== mon.1 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb00400dd30 con 0x7fb00c159920 2023-05-22T15:27:10.673 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.614+0000 7f5831ffb640 1 -- 172.21.15.16:0/1969369175 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f582c0b6830 con 0x7f5834050fa0 2023-05-22T15:27:10.673 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.614+0000 7f5831ffb640 1 -- 172.21.15.16:0/1969369175 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f582c0b69b0 con 0x7f5834050fa0 2023-05-22T15:27:10.674 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.614+0000 7f5831ffb640 1 -- 172.21.15.16:0/1969369175 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f582c0b6d60 con 0x7f5834050fa0 2023-05-22T15:27:10.674 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.614+0000 7f584345c640 1 -- 172.21.15.16:0/1969369175 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5834053950 con 0x7f5834050fa0 2023-05-22T15:27:10.674 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.614+0000 7f584345c640 1 -- 172.21.15.16:0/1969369175 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58340074f0 con 0x7f5834050fa0 2023-05-22T15:27:10.674 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.618+0000 7f584345c640 1 -- 172.21.15.16:0/1969369175 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5834053ae0 con 0x7f5834050fa0 2023-05-22T15:27:10.674 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.618+0000 7f5831ffb640 1 -- 172.21.15.16:0/1969369175 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7f582c0b2040 con 0x7f5834050fa0 2023-05-22T15:27:10.674 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.618+0000 7f5831ffb640 1 --2- 172.21.15.16:0/1969369175 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f58200653d0 0x7f5820067880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:10.674 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.618+0000 7f5831ffb640 1 -- 172.21.15.16:0/1969369175 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f582c11d6a0 con 0x7f5834050fa0 2023-05-22T15:27:10.674 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.618+0000 7f58411f8640 1 --2- 172.21.15.16:0/1969369175 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f58200653d0 0x7f5820067880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:10.675 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.622+0000 7f80acd28640 1 Processor -- start 2023-05-22T15:27:10.675 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.622+0000 7f0c2effd640 1 -- 172.21.15.16:0/2455868988 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7f0c30013470 con 0x7f0c400894b0 2023-05-22T15:27:10.676 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.622+0000 7f0c2effd640 1 --2- 172.21.15.16:0/2455868988 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f0c20065450 0x7f0c20067900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:10.676 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.622+0000 7f0c2effd640 1 -- 172.21.15.16:0/2455868988 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f0c300834d0 con 0x7f0c400894b0 2023-05-22T15:27:10.676 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.622+0000 7f0c3e59c640 1 --2- 172.21.15.16:0/2455868988 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f0c20065450 0x7f0c20067900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:10.676 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.622+0000 7f80acd28640 1 -- start start 2023-05-22T15:27:10.676 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.622+0000 7f80acd28640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f80a0054a80 0x7f80a00d53f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:10.676 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.622+0000 7f80acd28640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f80a004e9f0 0x7f80a00d59a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:10.677 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.622+0000 7f80acd28640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f80a00d2740 0x7f80a00d5ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:10.677 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.622+0000 7f80acd28640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80a0049870 con 0x7f80a004e9f0 2023-05-22T15:27:10.677 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.622+0000 7f80acd28640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80a00d6cd0 con 0x7f80a00d2740 2023-05-22T15:27:10.677 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.622+0000 7f80acd28640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80a00d6f40 con 0x7f80a0054a80 2023-05-22T15:27:10.677 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.622+0000 7f0c3e59c640 1 --2- 172.21.15.16:0/2455868988 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f0c20065450 0x7f0c20067900 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f0c38001570 tx=0x7f0c38001aa0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:10.678 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.622+0000 7f80a659c640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f80a0054a80 0x7f80a00d53f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:10.678 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.622+0000 7f80a659c640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f80a0054a80 0x7f80a00d53f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3301/0 says I am v2:172.21.15.16:57610/0 (socket says 172.21.15.16:57610) 2023-05-22T15:27:10.678 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.622+0000 7f80a659c640 1 -- 172.21.15.16:0/63025921 learned_addr learned my addr 172.21.15.16:0/63025921 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:10.678 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.622+0000 7f6c19ffb640 1 -- 172.21.15.16:0/1014005193 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7f6c140b2040 con 0x7f6c1c054df0 2023-05-22T15:27:10.679 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.622+0000 7f80a6d9d640 1 --2- 172.21.15.16:0/63025921 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f80a00d2740 0x7f80a00d5ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:10.679 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.622+0000 7faff3fff640 1 -- 172.21.15.16:0/703323199 <== mon.1 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7fb004015440 con 0x7fb00c159920 2023-05-22T15:27:10.679 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.622+0000 7faff3fff640 1 --2- 172.21.15.16:0/703323199 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7faff8065380 0x7faff8067830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:10.679 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.622+0000 7faff3fff640 1 -- 172.21.15.16:0/703323199 <== mon.1 v2:172.21.15.62:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6813+0+0 (secure 0 0 0) 0x7fb004083d50 con 0x7fb00c159920 2023-05-22T15:27:10.680 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.622+0000 7f6c19ffb640 1 --2- 172.21.15.16:0/1014005193 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f6c080653d0 0x7f6c08067880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:10.680 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.622+0000 7fb00b7fe640 1 --2- 172.21.15.16:0/703323199 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7faff8065380 0x7faff8067830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:10.681 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.622+0000 7f6c19ffb640 1 -- 172.21.15.16:0/1014005193 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f6c1411d8a0 con 0x7f6c1c054df0 2023-05-22T15:27:10.681 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.622+0000 7f80a659c640 1 -- 172.21.15.16:0/63025921 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f80a00d2740 msgr2=0x7f80a00d5ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:10.681 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.622+0000 7f80a659c640 1 --2- 172.21.15.16:0/63025921 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f80a00d2740 0x7f80a00d5ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:10.681 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.622+0000 7f80a659c640 1 -- 172.21.15.16:0/63025921 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f80a004e9f0 msgr2=0x7f80a00d59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:10.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.622+0000 7f80a659c640 1 --2- 172.21.15.16:0/63025921 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f80a004e9f0 0x7f80a00d59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:10.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.622+0000 7f80a659c640 1 -- 172.21.15.16:0/63025921 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80a0008a70 con 0x7f80a0054a80 2023-05-22T15:27:10.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.622+0000 7f6c295ae640 1 --2- 172.21.15.16:0/1014005193 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f6c080653d0 0x7f6c08067880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:10.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.622+0000 7f58411f8640 1 --2- 172.21.15.16:0/1969369175 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f58200653d0 0x7f5820067880 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f583800b810 tx=0x7f5838008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:10.683 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.622+0000 7fb00b7fe640 1 --2- 172.21.15.16:0/703323199 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7faff8065380 0x7faff8067830 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7faffc0ae000 tx=0x7faffc008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:10.683 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.622+0000 7f80a659c640 1 --2- 172.21.15.16:0/63025921 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f80a0054a80 0x7f80a00d53f0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f809c0108d0 tx=0x7f809c010d90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:10.683 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.622+0000 7f6c295ae640 1 --2- 172.21.15.16:0/1014005193 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f6c080653d0 0x7f6c08067880 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f6c20005890 tx=0x7f6c20009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:10.684 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.614+0000 7fb2199f8640 1 Processor -- start 2023-05-22T15:27:10.684 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.614+0000 7fb2199f8640 1 -- start start 2023-05-22T15:27:10.684 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.618+0000 7fb2199f8640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb2140cd6c0 0x7fb21406c250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:10.684 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.618+0000 7fb2199f8640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fb2140cbd60 0x7fb2140cc1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:10.684 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.618+0000 7fb2199f8640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2140cda90 con 0x7fb2140cc6f0 2023-05-22T15:27:10.684 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.618+0000 7fb2199f8640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2140c9d80 con 0x7fb2140cbd60 2023-05-22T15:27:10.685 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.618+0000 7fb2199f8640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2140ccb70 con 0x7fb2140cd6c0 2023-05-22T15:27:10.685 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.622+0000 7f8096ffd640 1 -- 172.21.15.16:0/63025921 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f809c004220 con 0x7f80a0054a80 2023-05-22T15:27:10.685 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.618+0000 7fb213fff640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fb2140cbd60 0x7fb2140cc1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:10.685 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.618+0000 7fb213fff640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fb2140cbd60 0x7fb2140cc1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.16:48536/0 (socket says 172.21.15.16:48536) 2023-05-22T15:27:10.685 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.618+0000 7fb213fff640 1 -- 172.21.15.16:0/1899855820 learned_addr learned my addr 172.21.15.16:0/1899855820 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:10.686 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.622+0000 7f80acd28640 1 -- 172.21.15.16:0/63025921 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80a003ba90 con 0x7f80a0054a80 2023-05-22T15:27:10.686 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.622+0000 7f8096ffd640 1 -- 172.21.15.16:0/63025921 <== mon.2 v2:172.21.15.62:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f809c004380 con 0x7f80a0054a80 2023-05-22T15:27:10.686 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.622+0000 7fb2189f6640 1 --2- 172.21.15.16:0/1899855820 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb2140cd6c0 0x7fb21406c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:10.687 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.626+0000 7fb213fff640 1 -- 172.21.15.16:0/1899855820 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb2140cd6c0 msgr2=0x7fb21406c250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:10.687 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.626+0000 7fb213fff640 1 --2- 172.21.15.16:0/1899855820 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb2140cd6c0 0x7fb21406c250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:10.687 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.626+0000 7fb213fff640 1 -- 172.21.15.16:0/1899855820 >> v1:172.21.15.16:6789/0 conn(0x7fb2140cc6f0 legacy=0x7fb21416c380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:10.687 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.626+0000 7fb213fff640 1 -- 172.21.15.16:0/1899855820 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2140cdce0 con 0x7fb2140cbd60 2023-05-22T15:27:10.687 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.622+0000 7f8096ffd640 1 -- 172.21.15.16:0/63025921 <== mon.2 v2:172.21.15.62:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f809c00eb60 con 0x7f80a0054a80 2023-05-22T15:27:10.688 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.622+0000 7f80acd28640 1 -- 172.21.15.16:0/63025921 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80a0008c00 con 0x7f80a0054a80 2023-05-22T15:27:10.688 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.626+0000 7f80acd28640 1 -- 172.21.15.16:0/63025921 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80a00094d0 con 0x7f80a0054a80 2023-05-22T15:27:10.688 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.626+0000 7f5831ffb640 1 -- 172.21.15.16:0/1969369175 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f582c0eb250 con 0x7f5834050fa0 2023-05-22T15:27:10.688 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.626+0000 7f8096ffd640 1 -- 172.21.15.16:0/63025921 <== mon.2 v2:172.21.15.62:3301/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7f809c00ecc0 con 0x7f80a0054a80 2023-05-22T15:27:10.689 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.626+0000 7fb213fff640 1 --2- 172.21.15.16:0/1899855820 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb2140cbd60 0x7fb2140cc1b0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fb200007860 tx=0x7fb20001d710 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fd1c4f4326c249c7 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:10.689 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.626+0000 7f8096ffd640 1 --2- 172.21.15.16:0/63025921 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f8088065380 0x7f8088067830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:10.689 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.626+0000 7f8096ffd640 1 -- 172.21.15.16:0/63025921 <== mon.2 v2:172.21.15.62:3301/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f809c083460 con 0x7f80a0054a80 2023-05-22T15:27:10.690 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.626+0000 7f80a5d9b640 1 --2- 172.21.15.16:0/63025921 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f8088065380 0x7f8088067830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:10.690 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.626+0000 7f80a5d9b640 1 --2- 172.21.15.16:0/63025921 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f8088065380 0x7f8088067830 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f80a00d89e0 tx=0x7f8098006c50 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:10.690 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.630+0000 7f6c19ffb640 1 -- 172.21.15.16:0/1014005193 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f6c140eb2b0 con 0x7f6c1c054df0 2023-05-22T15:27:10.690 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.634+0000 7f0c2effd640 1 -- 172.21.15.16:0/2455868988 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f0c30050e90 con 0x7f0c400894b0 2023-05-22T15:27:10.691 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.638+0000 7fb2137fe640 1 -- 172.21.15.16:0/1899855820 <== mon.1 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2000a76c0 con 0x7fb2140cbd60 2023-05-22T15:27:10.691 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.638+0000 7fb2137fe640 1 -- 172.21.15.16:0/1899855820 <== mon.1 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2000a7820 con 0x7fb2140cbd60 2023-05-22T15:27:10.691 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.638+0000 7fb2137fe640 1 -- 172.21.15.16:0/1899855820 <== mon.1 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2000a7bf0 con 0x7fb2140cbd60 2023-05-22T15:27:10.691 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.638+0000 7faff3fff640 1 -- 172.21.15.16:0/703323199 <== mon.1 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fb004051710 con 0x7fb00c159920 2023-05-22T15:27:10.692 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.638+0000 7fb2199f8640 1 -- 172.21.15.16:0/1899855820 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb2140cbd60 msgr2=0x7fb2140cc1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:10.692 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.638+0000 7fb2199f8640 1 --2- 172.21.15.16:0/1899855820 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb2140cbd60 0x7fb2140cc1b0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fb200007860 tx=0x7fb20001d710 comp rx=0 tx=0).stop 2023-05-22T15:27:10.693 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.638+0000 7fb2199f8640 1 -- 172.21.15.16:0/1899855820 shutdown_connections 2023-05-22T15:27:10.693 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.638+0000 7fb2199f8640 1 --2- 172.21.15.16:0/1899855820 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb2140cbd60 0x7fb2140cc1b0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:10.693 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.638+0000 7fb2199f8640 1 --2- 172.21.15.16:0/1899855820 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb2140cd6c0 0x7fb21406c250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:10.693 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.638+0000 7fb2199f8640 1 -- 172.21.15.16:0/1899855820 >> 172.21.15.16:0/1899855820 conn(0x7fb214069720 msgr2=0x7fb214069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:10.693 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.638+0000 7fb2199f8640 1 -- 172.21.15.16:0/1899855820 shutdown_connections 2023-05-22T15:27:10.693 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.638+0000 7fb2199f8640 1 -- 172.21.15.16:0/1899855820 wait complete. 2023-05-22T15:27:10.693 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.642+0000 7fb2199f8640 1 Processor -- start 2023-05-22T15:27:10.694 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.642+0000 7fb2199f8640 1 -- start start 2023-05-22T15:27:10.694 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.642+0000 7fb2199f8640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb2140cc6f0 0x7fb21420a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:10.694 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.642+0000 7fb2199f8640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb2140cd6c0 0x7fb2140888a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:10.694 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.642+0000 7fb2199f8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb214162c20 0x7fb21420aff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:10.694 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.642+0000 7fb2191f7640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb214162c20 0x7fb21420aff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:10.694 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.642+0000 7fb2191f7640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb214162c20 0x7fb21420aff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37802/0 (socket says 172.21.15.16:37802) 2023-05-22T15:27:10.694 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.642+0000 7fb2191f7640 1 -- 172.21.15.16:0/943079964 learned_addr learned my addr 172.21.15.16:0/943079964 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:10.695 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.642+0000 7fb213fff640 1 --2- 172.21.15.16:0/943079964 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb2140cd6c0 0x7fb2140888a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:10.695 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.642+0000 7fb2199f8640 1 -- 172.21.15.16:0/943079964 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb21420adf0 con 0x7fb214162c20 2023-05-22T15:27:10.695 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.642+0000 7fb2189f6640 1 --2- 172.21.15.16:0/943079964 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb2140cc6f0 0x7fb21420a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:10.695 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.642+0000 7fb2199f8640 1 -- 172.21.15.16:0/943079964 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2141631a0 con 0x7fb2140cc6f0 2023-05-22T15:27:10.695 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.642+0000 7fb2199f8640 1 -- 172.21.15.16:0/943079964 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb214163480 con 0x7fb2140cd6c0 2023-05-22T15:27:10.695 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.646+0000 7fb2191f7640 1 -- 172.21.15.16:0/943079964 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb2140cd6c0 msgr2=0x7fb2140888a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:10.695 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.646+0000 7fb2191f7640 1 --2- 172.21.15.16:0/943079964 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb2140cd6c0 0x7fb2140888a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:10.695 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.646+0000 7fb2191f7640 1 -- 172.21.15.16:0/943079964 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb2140cc6f0 msgr2=0x7fb21420a8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:10.696 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.646+0000 7fb2191f7640 1 --2- 172.21.15.16:0/943079964 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb2140cc6f0 0x7fb21420a8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:10.696 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.646+0000 7fb2191f7640 1 -- 172.21.15.16:0/943079964 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb200007540 con 0x7fb214162c20 2023-05-22T15:27:10.696 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.646+0000 7fb2191f7640 1 --2- 172.21.15.16:0/943079964 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb214162c20 0x7fb21420aff0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fb20c00c710 tx=0x7fb20c00cbd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:10.696 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.646+0000 7f8096ffd640 1 -- 172.21.15.16:0/63025921 <== mon.2 v2:172.21.15.62:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f809c050e00 con 0x7f80a0054a80 2023-05-22T15:27:10.697 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.646+0000 7fb2117fa640 1 -- 172.21.15.16:0/943079964 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb20c006d60 con 0x7fb214162c20 2023-05-22T15:27:10.697 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.646+0000 7fb2117fa640 1 -- 172.21.15.16:0/943079964 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb20c00ce20 con 0x7fb214162c20 2023-05-22T15:27:10.697 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.646+0000 7fb2199f8640 1 -- 172.21.15.16:0/943079964 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2141756c0 con 0x7fb214162c20 2023-05-22T15:27:10.698 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.646+0000 7fb2117fa640 1 -- 172.21.15.16:0/943079964 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb20c003b80 con 0x7fb214162c20 2023-05-22T15:27:10.698 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.646+0000 7fb2117fa640 1 -- 172.21.15.16:0/943079964 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7fb20c003ce0 con 0x7fb214162c20 2023-05-22T15:27:10.698 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.646+0000 7fb2199f8640 1 -- 172.21.15.16:0/943079964 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2140ce580 con 0x7fb214162c20 2023-05-22T15:27:10.698 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.650+0000 7fb2117fa640 1 --2- 172.21.15.16:0/943079964 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fb1f8065380 0x7fb1f8067830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:10.698 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.650+0000 7fb2189f6640 1 --2- 172.21.15.16:0/943079964 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fb1f8065380 0x7fb1f8067830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:10.698 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.650+0000 7fb2189f6640 1 --2- 172.21.15.16:0/943079964 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fb1f8065380 0x7fb1f8067830 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fb208009ac0 tx=0x7fb208009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:10.698 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.654+0000 7fb2117fa640 1 -- 172.21.15.16:0/943079964 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6813+0+0 (secure 0 0 0) 0x7fb20c05b550 con 0x7fb214162c20 2023-05-22T15:27:10.699 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.654+0000 7fb2199f8640 1 -- 172.21.15.16:0/943079964 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb214175850 con 0x7fb214162c20 2023-05-22T15:27:10.699 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.662+0000 7fb2117fa640 1 -- 172.21.15.16:0/943079964 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fb20c04f8b0 con 0x7fb214162c20 2023-05-22T15:27:10.864 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.854+0000 7f57bb26e640 1 -- 172.21.15.16:0/4248254360 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f5794005510 con 0x7f57b40caa30 2023-05-22T15:27:10.865 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.854+0000 7f57a9ffb640 1 -- 172.21.15.16:0/4248254360 <== mon.2 v2:172.21.15.62:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f57ac053730 con 0x7f57b40caa30 2023-05-22T15:27:10.865 INFO:teuthology.orchestra.run.smithi016.stdout:55834574857 2023-05-22T15:27:10.869 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.858+0000 7f579f7fe640 1 -- 172.21.15.16:0/4248254360 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f5798065450 msgr2=0x7f5798067900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:10.869 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.858+0000 7f579f7fe640 1 --2- 172.21.15.16:0/4248254360 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f5798065450 0x7f5798067900 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f57b00b2000 tx=0x7f57b00a9070 comp rx=0 tx=0).stop 2023-05-22T15:27:10.869 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.858+0000 7f579f7fe640 1 -- 172.21.15.16:0/4248254360 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f57b40caa30 msgr2=0x7f57b40c18c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:10.869 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.858+0000 7f579f7fe640 1 --2- 172.21.15.16:0/4248254360 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f57b40caa30 0x7f57b40c18c0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f57ac00c790 tx=0x7f57ac00cc50 comp rx=0 tx=0).stop 2023-05-22T15:27:10.870 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.858+0000 7f579f7fe640 1 -- 172.21.15.16:0/4248254360 shutdown_connections 2023-05-22T15:27:10.870 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.858+0000 7f579f7fe640 1 --2- 172.21.15.16:0/4248254360 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f57b4158100 0x7f57b40c75d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:10.870 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.858+0000 7f579f7fe640 1 --2- 172.21.15.16:0/4248254360 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f57b4157d30 0x7f57b40c24e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:10.870 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.858+0000 7f579f7fe640 1 --2- 172.21.15.16:0/4248254360 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f5798065450 0x7f5798067900 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:10.870 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.858+0000 7f579f7fe640 1 --2- 172.21.15.16:0/4248254360 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f57b40caa30 0x7f57b40c18c0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:10.870 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.858+0000 7f579f7fe640 1 -- 172.21.15.16:0/4248254360 >> 172.21.15.16:0/4248254360 conn(0x7f57b40af9f0 msgr2=0x7f57b415b210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:10.870 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.858+0000 7f579f7fe640 1 -- 172.21.15.16:0/4248254360 shutdown_connections 2023-05-22T15:27:10.871 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.858+0000 7f579f7fe640 1 -- 172.21.15.16:0/4248254360 wait complete. 2023-05-22T15:27:10.877 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.866+0000 7f01f22d5640 1 -- 172.21.15.16:0/1515889298 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f01cc005510 con 0x7f01ec0caed0 2023-05-22T15:27:10.878 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.870+0000 7f01e8ff9640 1 -- 172.21.15.16:0/1515889298 <== mon.1 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f01e4056a00 con 0x7f01ec0caed0 2023-05-22T15:27:10.881 INFO:teuthology.orchestra.run.smithi016.stdout:55834574856 2023-05-22T15:27:10.881 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.870+0000 7f01d27fc640 1 -- 172.21.15.16:0/1515889298 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f01d8065380 msgr2=0x7f01d8067830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:10.881 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.870+0000 7f01d27fc640 1 --2- 172.21.15.16:0/1515889298 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f01d8065380 0x7f01d8067830 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f01dc01c5b0 tx=0x7f01dc0b5040 comp rx=0 tx=0).stop 2023-05-22T15:27:10.882 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.874+0000 7f01d27fc640 1 -- 172.21.15.16:0/1515889298 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f01ec0caed0 msgr2=0x7f01ec0c4180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:10.882 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.874+0000 7f01d27fc640 1 --2- 172.21.15.16:0/1515889298 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f01ec0caed0 0x7f01ec0c4180 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f01e400e570 tx=0x7f01e400ea30 comp rx=0 tx=0).stop 2023-05-22T15:27:10.883 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.874+0000 7f01d27fc640 1 -- 172.21.15.16:0/1515889298 shutdown_connections 2023-05-22T15:27:10.883 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.874+0000 7f01d27fc640 1 --2- 172.21.15.16:0/1515889298 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f01d8065380 0x7f01d8067830 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:10.883 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.874+0000 7f01d27fc640 1 --2- 172.21.15.16:0/1515889298 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f01ec156be0 0x7f01ec0b3000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:10.883 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.874+0000 7f01d27fc640 1 --2- 172.21.15.16:0/1515889298 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f01ec0caed0 0x7f01ec0c4180 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:10.883 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.874+0000 7f01d27fc640 1 --2- 172.21.15.16:0/1515889298 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f01ec0caa30 0x7f01ec2008f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:10.883 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.874+0000 7f01d27fc640 1 -- 172.21.15.16:0/1515889298 >> 172.21.15.16:0/1515889298 conn(0x7f01ec0af9f0 msgr2=0x7f01ec0ce550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:10.883 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.874+0000 7f01d27fc640 1 -- 172.21.15.16:0/1515889298 shutdown_connections 2023-05-22T15:27:10.884 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.874+0000 7f01d27fc640 1 -- 172.21.15.16:0/1515889298 wait complete. 2023-05-22T15:27:10.894 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574857 for osd.2 2023-05-22T15:27:10.894 DEBUG:teuthology.parallel:result is None 2023-05-22T15:27:10.902 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574856 for osd.4 2023-05-22T15:27:10.902 DEBUG:teuthology.parallel:result is None 2023-05-22T15:27:10.934 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.926+0000 7f584345c640 1 -- 172.21.15.16:0/1969369175 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f58340ce320 con 0x7f5834050fa0 2023-05-22T15:27:10.934 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.926+0000 7f5831ffb640 1 -- 172.21.15.16:0/1969369175 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f58340ce320 con 0x7f5834050fa0 2023-05-22T15:27:10.935 INFO:teuthology.orchestra.run.smithi016.stdout:55834574858 2023-05-22T15:27:10.936 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.926+0000 7f584345c640 1 -- 172.21.15.16:0/1969369175 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f58200653d0 msgr2=0x7f5820067880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:10.937 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.926+0000 7f584345c640 1 --2- 172.21.15.16:0/1969369175 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f58200653d0 0x7f5820067880 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f583800b810 tx=0x7f5838008040 comp rx=0 tx=0).stop 2023-05-22T15:27:10.937 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.926+0000 7f584345c640 1 -- 172.21.15.16:0/1969369175 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5834050fa0 msgr2=0x7f58340e7200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:10.937 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.926+0000 7f584345c640 1 --2- 172.21.15.16:0/1969369175 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5834050fa0 0x7f58340e7200 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f582c0ae040 tx=0x7f582c01ac70 comp rx=0 tx=0).stop 2023-05-22T15:27:10.937 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.926+0000 7f584345c640 1 -- 172.21.15.16:0/1969369175 shutdown_connections 2023-05-22T15:27:10.937 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.926+0000 7f584345c640 1 --2- 172.21.15.16:0/1969369175 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f58340e6c10 0x7f58340cce50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:10.938 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.926+0000 7f584345c640 1 --2- 172.21.15.16:0/1969369175 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5834050fa0 0x7f58340e7200 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:10.938 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.926+0000 7f584345c640 1 --2- 172.21.15.16:0/1969369175 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f58200653d0 0x7f5820067880 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:10.938 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.926+0000 7f584345c640 1 --2- 172.21.15.16:0/1969369175 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5834036a70 0x7f58340dbea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:10.938 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.926+0000 7f584345c640 1 -- 172.21.15.16:0/1969369175 >> 172.21.15.16:0/1969369175 conn(0x7f5834051530 msgr2=0x7f5834034c90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:10.938 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.926+0000 7f584345c640 1 -- 172.21.15.16:0/1969369175 shutdown_connections 2023-05-22T15:27:10.939 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.926+0000 7f584345c640 1 -- 172.21.15.16:0/1969369175 wait complete. 2023-05-22T15:27:10.951 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574858 got 55834574858 for osd.6 2023-05-22T15:27:10.952 DEBUG:teuthology.parallel:result is None 2023-05-22T15:27:10.955 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.946+0000 7f6c2b812640 1 -- 172.21.15.16:0/1014005193 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f6c1c0385b0 con 0x7f6c1c054df0 2023-05-22T15:27:10.956 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.946+0000 7f6c19ffb640 1 -- 172.21.15.16:0/1014005193 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f6c140b0090 con 0x7f6c1c054df0 2023-05-22T15:27:10.956 INFO:teuthology.orchestra.run.smithi016.stdout:55834574857 2023-05-22T15:27:10.958 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.946+0000 7f0c3f59e640 1 -- 172.21.15.16:0/2455868988 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f0c40177990 con 0x7f0c400894b0 2023-05-22T15:27:10.959 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.946+0000 7f6c2b812640 1 -- 172.21.15.16:0/1014005193 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f6c080653d0 msgr2=0x7f6c08067880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:10.959 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.950+0000 7f6c2b812640 1 --2- 172.21.15.16:0/1014005193 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f6c080653d0 0x7f6c08067880 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f6c20005890 tx=0x7f6c20009040 comp rx=0 tx=0).stop 2023-05-22T15:27:10.959 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.950+0000 7f6c2b812640 1 -- 172.21.15.16:0/1014005193 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c1c054df0 msgr2=0x7f6c1c184f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:10.959 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.950+0000 7f6c2b812640 1 --2- 172.21.15.16:0/1014005193 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c1c054df0 0x7f6c1c184f00 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f6c1401ab30 tx=0x7f6c14019220 comp rx=0 tx=0).stop 2023-05-22T15:27:10.959 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.950+0000 7f6c2b812640 1 -- 172.21.15.16:0/1014005193 shutdown_connections 2023-05-22T15:27:10.959 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.950+0000 7f6c2b812640 1 --2- 172.21.15.16:0/1014005193 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f6c080653d0 0x7f6c08067880 secure :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f6c20005890 tx=0x7f6c20009040 comp rx=0 tx=0).stop 2023-05-22T15:27:10.960 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.950+0000 7f6c2b812640 1 --2- 172.21.15.16:0/1014005193 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6c1c056780 0x7f6c1c0e28f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:10.960 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.950+0000 7f6c2b812640 1 --2- 172.21.15.16:0/1014005193 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c1c054df0 0x7f6c1c184f00 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:10.960 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.950+0000 7f6c2b812640 1 --2- 172.21.15.16:0/1014005193 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6c1c037270 0x7f6c1c188520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:10.960 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.950+0000 7f6c2b812640 1 -- 172.21.15.16:0/1014005193 >> 172.21.15.16:0/1014005193 conn(0x7f6c1c00a620 msgr2=0x7f6c1c039b10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:10.960 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.950+0000 7f6c2b812640 1 -- 172.21.15.16:0/1014005193 shutdown_connections 2023-05-22T15:27:10.960 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.950+0000 7f6c2b812640 1 -- 172.21.15.16:0/1014005193 wait complete. 2023-05-22T15:27:10.961 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.950+0000 7f0c2effd640 1 -- 172.21.15.16:0/2455868988 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f0c300544b0 con 0x7f0c400894b0 2023-05-22T15:27:10.961 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.950+0000 7fb2199f8640 1 -- 172.21.15.16:0/943079964 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7fb2140c8260 con 0x7fb214162c20 2023-05-22T15:27:10.961 INFO:teuthology.orchestra.run.smithi016.stdout:55834574858 2023-05-22T15:27:10.962 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.950+0000 7fb2117fa640 1 -- 172.21.15.16:0/943079964 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fb20c06a020 con 0x7fb214162c20 2023-05-22T15:27:10.962 INFO:teuthology.orchestra.run.smithi016.stdout:55834574857 2023-05-22T15:27:10.962 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.950+0000 7f0c3f59e640 1 -- 172.21.15.16:0/2455868988 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f0c20065450 msgr2=0x7f0c20067900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:10.963 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.950+0000 7f0c3f59e640 1 --2- 172.21.15.16:0/2455868988 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f0c20065450 0x7f0c20067900 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f0c38001570 tx=0x7f0c38001aa0 comp rx=0 tx=0).stop 2023-05-22T15:27:10.963 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.950+0000 7f0c3f59e640 1 -- 172.21.15.16:0/2455868988 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0c400894b0 msgr2=0x7f0c4020a1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:10.963 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.950+0000 7f0c3f59e640 1 --2- 172.21.15.16:0/2455868988 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0c400894b0 0x7f0c4020a1d0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f0c3000e9b0 tx=0x7f0c3000ee70 comp rx=0 tx=0).stop 2023-05-22T15:27:10.964 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.950+0000 7fb2199f8640 1 -- 172.21.15.16:0/943079964 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fb1f8065380 msgr2=0x7fb1f8067830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:10.964 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.950+0000 7fb2199f8640 1 --2- 172.21.15.16:0/943079964 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fb1f8065380 0x7fb1f8067830 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fb208009ac0 tx=0x7fb208009190 comp rx=0 tx=0).stop 2023-05-22T15:27:10.964 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.950+0000 7fb2199f8640 1 -- 172.21.15.16:0/943079964 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb214162c20 msgr2=0x7fb21420aff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:10.964 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.950+0000 7fb2199f8640 1 --2- 172.21.15.16:0/943079964 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb214162c20 0x7fb21420aff0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fb20c00c710 tx=0x7fb20c00cbd0 comp rx=0 tx=0).stop 2023-05-22T15:27:10.965 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.950+0000 7fb2199f8640 1 -- 172.21.15.16:0/943079964 shutdown_connections 2023-05-22T15:27:10.965 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.950+0000 7fb2199f8640 1 --2- 172.21.15.16:0/943079964 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb214162c20 0x7fb21420aff0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:10.965 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.950+0000 7fb2199f8640 1 --2- 172.21.15.16:0/943079964 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fb1f8065380 0x7fb1f8067830 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:10.965 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.950+0000 7fb2199f8640 1 --2- 172.21.15.16:0/943079964 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb2140cd6c0 0x7fb2140888a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:10.966 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.950+0000 7fb2199f8640 1 --2- 172.21.15.16:0/943079964 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb2140cc6f0 0x7fb21420a8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:10.966 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.950+0000 7fb2199f8640 1 -- 172.21.15.16:0/943079964 >> 172.21.15.16:0/943079964 conn(0x7fb214069720 msgr2=0x7fb214165860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:10.966 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.950+0000 7f0c3f59e640 1 -- 172.21.15.16:0/2455868988 shutdown_connections 2023-05-22T15:27:10.967 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.950+0000 7f0c3f59e640 1 --2- 172.21.15.16:0/2455868988 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f0c20065450 0x7f0c20067900 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:10.967 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.950+0000 7f0c3f59e640 1 --2- 172.21.15.16:0/2455868988 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0c400ca890 0x7f0c402090d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:10.967 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.950+0000 7f0c3f59e640 1 --2- 172.21.15.16:0/2455868988 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0c400894b0 0x7f0c4020a1d0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:10.967 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.950+0000 7f0c3f59e640 1 --2- 172.21.15.16:0/2455868988 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0c4006d630 0x7f0c4020cd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:10.968 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.950+0000 7f0c3f59e640 1 -- 172.21.15.16:0/2455868988 >> 172.21.15.16:0/2455868988 conn(0x7f0c400af9f0 msgr2=0x7f0c4016c100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:10.968 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.950+0000 7fb2199f8640 1 -- 172.21.15.16:0/943079964 shutdown_connections 2023-05-22T15:27:10.973 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.950+0000 7fb01086d640 1 -- 172.21.15.16:0/703323199 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7fb00c0738f0 con 0x7fb00c159920 2023-05-22T15:27:10.974 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.950+0000 7fb2199f8640 1 -- 172.21.15.16:0/943079964 wait complete. 2023-05-22T15:27:10.974 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.950+0000 7f0c3f59e640 1 -- 172.21.15.16:0/2455868988 shutdown_connections 2023-05-22T15:27:10.974 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.950+0000 7f0c3f59e640 1 -- 172.21.15.16:0/2455868988 wait complete. 2023-05-22T15:27:10.975 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.950+0000 7faff3fff640 1 -- 172.21.15.16:0/703323199 <== mon.1 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fb004024020 con 0x7fb00c159920 2023-05-22T15:27:10.975 INFO:teuthology.orchestra.run.smithi016.stdout:55834574856 2023-05-22T15:27:10.976 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.954+0000 7f80acd28640 1 -- 172.21.15.16:0/63025921 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f80a00056d0 con 0x7f80a0054a80 2023-05-22T15:27:10.976 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.954+0000 7f8096ffd640 1 -- 172.21.15.16:0/63025921 <== mon.2 v2:172.21.15.62:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f809c054420 con 0x7f80a0054a80 2023-05-22T15:27:10.976 INFO:teuthology.orchestra.run.smithi016.stdout:55834574858 2023-05-22T15:27:10.977 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.954+0000 7fb01086d640 1 -- 172.21.15.16:0/703323199 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7faff8065380 msgr2=0x7faff8067830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:10.977 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.954+0000 7fb01086d640 1 --2- 172.21.15.16:0/703323199 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7faff8065380 0x7faff8067830 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7faffc0ae000 tx=0x7faffc008040 comp rx=0 tx=0).stop 2023-05-22T15:27:10.977 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.954+0000 7fb01086d640 1 -- 172.21.15.16:0/703323199 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb00c159920 msgr2=0x7fb00c154230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:10.978 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.954+0000 7fb01086d640 1 --2- 172.21.15.16:0/703323199 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb00c159920 0x7fb00c154230 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fb00400cf20 tx=0x7fb00400a7d0 comp rx=0 tx=0).stop 2023-05-22T15:27:10.978 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.954+0000 7fb01086d640 1 -- 172.21.15.16:0/703323199 shutdown_connections 2023-05-22T15:27:10.978 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.954+0000 7fb01086d640 1 --2- 172.21.15.16:0/703323199 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb00c159920 0x7fb00c154230 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:10.978 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.954+0000 7fb01086d640 1 --2- 172.21.15.16:0/703323199 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7faff8065380 0x7faff8067830 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:10.978 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.954+0000 7fb01086d640 1 --2- 172.21.15.16:0/703323199 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb00c0b03e0 0x7fb00c15c690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:10.978 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.954+0000 7fb01086d640 1 --2- 172.21.15.16:0/703323199 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb00c06c740 0x7fb00c14cb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:10.978 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.954+0000 7fb01086d640 1 -- 172.21.15.16:0/703323199 >> 172.21.15.16:0/703323199 conn(0x7fb00c0cacd0 msgr2=0x7fb00c158af0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:10.979 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.954+0000 7f80acd28640 1 -- 172.21.15.16:0/63025921 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f8088065380 msgr2=0x7f8088067830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:10.979 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.954+0000 7f80acd28640 1 --2- 172.21.15.16:0/63025921 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f8088065380 0x7f8088067830 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f80a00d89e0 tx=0x7f8098006c50 comp rx=0 tx=0).stop 2023-05-22T15:27:10.979 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.954+0000 7f80acd28640 1 -- 172.21.15.16:0/63025921 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f80a0054a80 msgr2=0x7f80a00d53f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:10.979 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.954+0000 7f80acd28640 1 --2- 172.21.15.16:0/63025921 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f80a0054a80 0x7f80a00d53f0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f809c0108d0 tx=0x7f809c010d90 comp rx=0 tx=0).stop 2023-05-22T15:27:10.980 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.958+0000 7fb01086d640 1 -- 172.21.15.16:0/703323199 shutdown_connections 2023-05-22T15:27:10.980 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.958+0000 7fb01086d640 1 -- 172.21.15.16:0/703323199 wait complete. 2023-05-22T15:27:10.980 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.962+0000 7f80acd28640 1 -- 172.21.15.16:0/63025921 shutdown_connections 2023-05-22T15:27:10.980 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.962+0000 7f80acd28640 1 --2- 172.21.15.16:0/63025921 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f80a00d2740 0x7f80a00d5ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:10.981 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.962+0000 7f80acd28640 1 --2- 172.21.15.16:0/63025921 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f80a004e9f0 0x7f80a00d59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:10.981 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.962+0000 7f80acd28640 1 --2- 172.21.15.16:0/63025921 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f8088065380 0x7f8088067830 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:10.981 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.962+0000 7f80acd28640 1 --2- 172.21.15.16:0/63025921 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f80a0054a80 0x7f80a00d53f0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:10.981 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.962+0000 7f80acd28640 1 -- 172.21.15.16:0/63025921 >> 172.21.15.16:0/63025921 conn(0x7f80a0039810 msgr2=0x7f80a000c590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:10.981 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.962+0000 7f80acd28640 1 -- 172.21.15.16:0/63025921 shutdown_connections 2023-05-22T15:27:10.981 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:10.962+0000 7f80acd28640 1 -- 172.21.15.16:0/63025921 wait complete. 2023-05-22T15:27:10.985 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574857 for osd.7 2023-05-22T15:27:10.986 DEBUG:teuthology.parallel:result is None 2023-05-22T15:27:10.988 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574856 for osd.3 2023-05-22T15:27:10.988 DEBUG:teuthology.parallel:result is None 2023-05-22T15:27:10.993 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574857 for osd.1 2023-05-22T15:27:10.993 DEBUG:teuthology.parallel:result is None 2023-05-22T15:27:10.994 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574858 got 55834574858 for osd.0 2023-05-22T15:27:10.994 DEBUG:teuthology.parallel:result is None 2023-05-22T15:27:10.995 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574858 got 55834574858 for osd.5 2023-05-22T15:27:10.995 DEBUG:teuthology.parallel:result is None 2023-05-22T15:27:10.995 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-05-22T15:27:10.995 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-05-22T15:27:11.141 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:11.130+0000 7ff241b2a640 1 Processor -- start 2023-05-22T15:27:11.142 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:11.130+0000 7ff241b2a640 1 -- start start 2023-05-22T15:27:11.143 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:11.134+0000 7ff241b2a640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff23c0cd730 0x7ff23c0afb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:11.144 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:11.134+0000 7ff241b2a640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff23c0b0080 0x7ff23c0acba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:11.144 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:11.134+0000 7ff241b2a640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff23c0cc300 con 0x7ff23c1531d0 2023-05-22T15:27:11.144 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:11.134+0000 7ff241b2a640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff23c15d330 con 0x7ff23c0b0080 2023-05-22T15:27:11.144 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:11.134+0000 7ff241b2a640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff23c15c4a0 con 0x7ff23c0cd730 2023-05-22T15:27:11.144 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:11.134+0000 7ff23bfff640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff23c0b0080 0x7ff23c0acba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:11.145 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:11.134+0000 7ff23bfff640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff23c0b0080 0x7ff23c0acba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37814/0 (socket says 172.21.15.16:37814) 2023-05-22T15:27:11.145 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:11.134+0000 7ff23bfff640 1 -- 172.21.15.16:0/2253352687 learned_addr learned my addr 172.21.15.16:0/2253352687 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:11.145 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:11.134+0000 7ff233fff640 1 --2- 172.21.15.16:0/2253352687 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff23c0cd730 0x7ff23c0afb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:11.145 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:11.134+0000 7ff23bfff640 1 -- 172.21.15.16:0/2253352687 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff23c0cd730 msgr2=0x7ff23c0afb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:11.145 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:11.134+0000 7ff23bfff640 1 --2- 172.21.15.16:0/2253352687 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff23c0cd730 0x7ff23c0afb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:11.145 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:11.134+0000 7ff23bfff640 1 -- 172.21.15.16:0/2253352687 >> v1:172.21.15.62:6789/0 conn(0x7ff23c1531d0 legacy=0x7ff23c0cbbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:11.146 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:11.134+0000 7ff23bfff640 1 -- 172.21.15.16:0/2253352687 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff23c0cdb60 con 0x7ff23c0b0080 2023-05-22T15:27:11.147 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:11.134+0000 7ff23bfff640 1 --2- 172.21.15.16:0/2253352687 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff23c0b0080 0x7ff23c0acba0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7ff234009e50 tx=0x7ff23401c9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c7d6fa3ac63f2c1 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:11.147 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:11.134+0000 7ff23affd640 1 -- 172.21.15.16:0/2253352687 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2340090d0 con 0x7ff23c0b0080 2023-05-22T15:27:11.147 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:11.134+0000 7ff23affd640 1 -- 172.21.15.16:0/2253352687 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff234009230 con 0x7ff23c0b0080 2023-05-22T15:27:11.147 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:11.134+0000 7ff23affd640 1 -- 172.21.15.16:0/2253352687 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2340a5960 con 0x7ff23c0b0080 2023-05-22T15:27:11.147 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:11.138+0000 7ff241b2a640 1 -- 172.21.15.16:0/2253352687 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff23c0b0080 msgr2=0x7ff23c0acba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:11.147 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:11.138+0000 7ff241b2a640 1 --2- 172.21.15.16:0/2253352687 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff23c0b0080 0x7ff23c0acba0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7ff234009e50 tx=0x7ff23401c9d0 comp rx=0 tx=0).stop 2023-05-22T15:27:11.148 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:11.138+0000 7ff241b2a640 1 -- 172.21.15.16:0/2253352687 shutdown_connections 2023-05-22T15:27:11.148 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:11.138+0000 7ff241b2a640 1 --2- 172.21.15.16:0/2253352687 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff23c0b0080 0x7ff23c0acba0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:11.148 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:11.138+0000 7ff241b2a640 1 --2- 172.21.15.16:0/2253352687 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff23c0cd730 0x7ff23c0afb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:11.148 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:11.138+0000 7ff241b2a640 1 -- 172.21.15.16:0/2253352687 >> 172.21.15.16:0/2253352687 conn(0x7ff23c069720 msgr2=0x7ff23c069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:11.148 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:11.138+0000 7ff241b2a640 1 -- 172.21.15.16:0/2253352687 shutdown_connections 2023-05-22T15:27:11.148 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:11.138+0000 7ff241b2a640 1 -- 172.21.15.16:0/2253352687 wait complete. 2023-05-22T15:27:11.149 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:11.138+0000 7ff241b2a640 1 Processor -- start 2023-05-22T15:27:11.149 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:11.138+0000 7ff241b2a640 1 -- start start 2023-05-22T15:27:11.151 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:11.138+0000 7ff241b2a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff23c0b0080 0x7ff23c15ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:11.151 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:11.138+0000 7ff241b2a640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff23c0cd730 0x7ff23c14b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:11.151 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:11.138+0000 7ff241b2a640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff23c1531d0 0x7ff23c14e1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:11.151 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:11.138+0000 7ff241b2a640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff23c15d5d0 con 0x7ff23c0b0080 2023-05-22T15:27:11.152 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:11.138+0000 7ff241b2a640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff23c14ba30 con 0x7ff23c1531d0 2023-05-22T15:27:11.152 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:11.138+0000 7ff241b2a640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff23c15f1b0 con 0x7ff23c0cd730 2023-05-22T15:27:11.152 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:11.138+0000 7ff23b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff23c0b0080 0x7ff23c15ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:11.152 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:11.138+0000 7ff23b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff23c0b0080 0x7ff23c15ebd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37830/0 (socket says 172.21.15.16:37830) 2023-05-22T15:27:11.152 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:11.138+0000 7ff23b7fe640 1 -- 172.21.15.16:0/3748645370 learned_addr learned my addr 172.21.15.16:0/3748645370 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:11.152 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:11.138+0000 7ff23b7fe640 1 -- 172.21.15.16:0/3748645370 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff23c0cd730 msgr2=0x7ff23c14b4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:11.153 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:11.138+0000 7ff23b7fe640 1 --2- 172.21.15.16:0/3748645370 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff23c0cd730 0x7ff23c14b4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:11.153 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:11.138+0000 7ff23b7fe640 1 -- 172.21.15.16:0/3748645370 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff23c1531d0 msgr2=0x7ff23c14e1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:11.153 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:11.138+0000 7ff23b7fe640 1 --2- 172.21.15.16:0/3748645370 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff23c1531d0 0x7ff23c14e1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:11.153 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:11.138+0000 7ff23b7fe640 1 -- 172.21.15.16:0/3748645370 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff234009b30 con 0x7ff23c0b0080 2023-05-22T15:27:11.153 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:11.138+0000 7ff23b7fe640 1 --2- 172.21.15.16:0/3748645370 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff23c0b0080 0x7ff23c15ebd0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7ff22400b440 tx=0x7ff22400b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:11.153 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:11.138+0000 7ff238ff9640 1 -- 172.21.15.16:0/3748645370 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff224011890 con 0x7ff23c0b0080 2023-05-22T15:27:11.153 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:11.138+0000 7ff238ff9640 1 -- 172.21.15.16:0/3748645370 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2240119f0 con 0x7ff23c0b0080 2023-05-22T15:27:11.154 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:11.138+0000 7ff241b2a640 1 -- 172.21.15.16:0/3748645370 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff23c15cf50 con 0x7ff23c0b0080 2023-05-22T15:27:11.154 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:11.138+0000 7ff238ff9640 1 -- 172.21.15.16:0/3748645370 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff224011da0 con 0x7ff23c0b0080 2023-05-22T15:27:11.154 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:11.138+0000 7ff241b2a640 1 -- 172.21.15.16:0/3748645370 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff23c15cc70 con 0x7ff23c0b0080 2023-05-22T15:27:11.154 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:11.138+0000 7ff241b2a640 1 -- 172.21.15.16:0/3748645370 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff23c160740 con 0x7ff23c0b0080 2023-05-22T15:27:11.154 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:11.142+0000 7ff238ff9640 1 -- 172.21.15.16:0/3748645370 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7ff22401a400 con 0x7ff23c0b0080 2023-05-22T15:27:11.154 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:11.142+0000 7ff238ff9640 1 --2- 172.21.15.16:0/3748645370 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ff21c0653b0 0x7ff21c067860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:11.154 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:11.142+0000 7ff238ff9640 1 -- 172.21.15.16:0/3748645370 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6813+0+0 (secure 0 0 0) 0x7ff224082b60 con 0x7ff23c0b0080 2023-05-22T15:27:11.155 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:11.142+0000 7ff233fff640 1 --2- 172.21.15.16:0/3748645370 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ff21c0653b0 0x7ff21c067860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:11.155 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:11.142+0000 7ff233fff640 1 --2- 172.21.15.16:0/3748645370 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ff21c0653b0 0x7ff21c067860 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7ff23c0c4eb0 tx=0x7ff22c008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:11.157 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:11.146+0000 7ff238ff9640 1 -- 172.21.15.16:0/3748645370 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7ff224050490 con 0x7ff23c0b0080 2023-05-22T15:27:11.298 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:11.290+0000 7ff241b2a640 1 -- 172.21.15.16:0/3748645370 --> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff23c04dbe0 con 0x7ff21c0653b0 2023-05-22T15:27:11.323 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:11.314+0000 7ff238ff9640 1 -- 172.21.15.16:0/3748645370 <== mgr.4101 v2:172.21.15.16:6832/36921 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+367259 (secure 0 0 0) 0x7ff23c04dbe0 con 0x7ff21c0653b0 2023-05-22T15:27:11.324 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:27:11.328 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2023-05-22T15:27:11.329 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:11.318+0000 7ff241b2a640 1 -- 172.21.15.16:0/3748645370 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ff21c0653b0 msgr2=0x7ff21c067860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:11.330 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:11.318+0000 7ff241b2a640 1 --2- 172.21.15.16:0/3748645370 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ff21c0653b0 0x7ff21c067860 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7ff23c0c4eb0 tx=0x7ff22c008040 comp rx=0 tx=0).stop 2023-05-22T15:27:11.330 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:11.318+0000 7ff241b2a640 1 -- 172.21.15.16:0/3748645370 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff23c0b0080 msgr2=0x7ff23c15ebd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:11.330 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:11.318+0000 7ff241b2a640 1 --2- 172.21.15.16:0/3748645370 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff23c0b0080 0x7ff23c15ebd0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7ff22400b440 tx=0x7ff22400b900 comp rx=0 tx=0).stop 2023-05-22T15:27:11.330 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:11.318+0000 7ff241b2a640 1 -- 172.21.15.16:0/3748645370 shutdown_connections 2023-05-22T15:27:11.330 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:11.318+0000 7ff241b2a640 1 --2- 172.21.15.16:0/3748645370 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff23c1531d0 0x7ff23c14e1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:11.331 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:11.318+0000 7ff241b2a640 1 --2- 172.21.15.16:0/3748645370 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ff21c0653b0 0x7ff21c067860 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:11.331 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:11.318+0000 7ff241b2a640 1 --2- 172.21.15.16:0/3748645370 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff23c0cd730 0x7ff23c14b4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:11.331 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:11.318+0000 7ff241b2a640 1 --2- 172.21.15.16:0/3748645370 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff23c0b0080 0x7ff23c15ebd0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:11.331 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:11.318+0000 7ff241b2a640 1 -- 172.21.15.16:0/3748645370 >> 172.21.15.16:0/3748645370 conn(0x7ff23c069720 msgr2=0x7ff23c0ac060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:11.336 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:11.318+0000 7ff241b2a640 1 -- 172.21.15.16:0/3748645370 shutdown_connections 2023-05-22T15:27:11.336 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:11.318+0000 7ff241b2a640 1 -- 172.21.15.16:0/3748645370 wait complete. 2023-05-22T15:27:11.344 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":37,"stamp":"2023-05-22T15:27:09.950396+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":594895,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":207,"num_write_kb":2434,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159,"ondisk_log_size":159,"up":338,"acting":338,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":338,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":75688,"kb_used_data":3496,"kb_used_omap":0,"kb_used_meta":72128,"kb_avail":754899032,"statfs":{"total":773094113280,"available":773016608768,"internally_reserved":0,"allocated":3579904,"data_stored":2148998,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":73859072},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2197,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004514"},"pg_stats":[{"pgid":"5.19","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.979395+0000","last_change":"2023-05-22T15:26:53.651924+0000","last_active":"2023-05-22T15:26:53.979395+0000","last_peered":"2023-05-22T15:26:53.979395+0000","last_clean":"2023-05-22T15:26:53.979395+0000","last_became_active":"2023-05-22T15:26:53.651533+0000","last_became_peered":"2023-05-22T15:26:53.651533+0000","last_unstale":"2023-05-22T15:26:53.979395+0000","last_undegraded":"2023-05-22T15:26:53.979395+0000","last_fullsized":"2023-05-22T15:26:53.979395+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_clean_scrub_stamp":"2023-05-22T15:26:52.568670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:27:41.065402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7,2],"acting":[1,5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"22'4","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.612099+0000","last_change":"2023-05-22T15:26:49.186238+0000","last_active":"2023-05-22T15:26:53.612099+0000","last_peered":"2023-05-22T15:26:53.612099+0000","last_clean":"2023-05-22T15:26:53.612099+0000","last_became_active":"2023-05-22T15:26:49.185455+0000","last_became_peered":"2023-05-22T15:26:49.185455+0000","last_unstale":"2023-05-22T15:26:53.612099+0000","last_undegraded":"2023-05-22T15:26:53.612099+0000","last_fullsized":"2023-05-22T15:26:53.612099+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_clean_scrub_stamp":"2023-05-22T15:26:48.127413+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:21:54.376718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":566,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.612858+0000","last_change":"2023-05-22T15:26:49.622435+0000","last_active":"2023-05-22T15:26:53.612858+0000","last_peered":"2023-05-22T15:26:53.612858+0000","last_clean":"2023-05-22T15:26:53.612858+0000","last_became_active":"2023-05-22T15:26:49.621643+0000","last_became_peered":"2023-05-22T15:26:49.621643+0000","last_unstale":"2023-05-22T15:26:53.612858+0000","last_undegraded":"2023-05-22T15:26:53.612858+0000","last_fullsized":"2023-05-22T15:26:53.612858+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:08:07.637726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-05-22T15:26:54.054128+0000","last_change":"2023-05-22T15:26:53.654741+0000","last_active":"2023-05-22T15:26:54.054128+0000","last_peered":"2023-05-22T15:26:54.054128+0000","last_clean":"2023-05-22T15:26:54.054128+0000","last_became_active":"2023-05-22T15:26:53.653669+0000","last_became_peered":"2023-05-22T15:26:53.653669+0000","last_unstale":"2023-05-22T15:26:54.054128+0000","last_undegraded":"2023-05-22T15:26:54.054128+0000","last_fullsized":"2023-05-22T15:26:54.054128+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_clean_scrub_stamp":"2023-05-22T15:26:52.568670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:01:12.890843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,2],"acting":[4,6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"22'1","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.602985+0000","last_change":"2023-05-22T15:26:49.187902+0000","last_active":"2023-05-22T15:26:53.602985+0000","last_peered":"2023-05-22T15:26:53.602985+0000","last_clean":"2023-05-22T15:26:53.602985+0000","last_became_active":"2023-05-22T15:26:49.187361+0000","last_became_peered":"2023-05-22T15:26:49.187361+0000","last_unstale":"2023-05-22T15:26:53.602985+0000","last_undegraded":"2023-05-22T15:26:53.602985+0000","last_fullsized":"2023-05-22T15:26:53.602985+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_clean_scrub_stamp":"2023-05-22T15:26:48.127413+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T18:13:58.291903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.603896+0000","last_change":"2023-05-22T15:26:49.560606+0000","last_active":"2023-05-22T15:26:53.603896+0000","last_peered":"2023-05-22T15:26:53.603896+0000","last_clean":"2023-05-22T15:26:53.603896+0000","last_became_active":"2023-05-22T15:26:49.560318+0000","last_became_peered":"2023-05-22T15:26:49.560318+0000","last_unstale":"2023-05-22T15:26:53.603896+0000","last_undegraded":"2023-05-22T15:26:53.603896+0000","last_fullsized":"2023-05-22T15:26:53.603896+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:43:00.518715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-05-22T15:26:54.055130+0000","last_change":"2023-05-22T15:26:53.648550+0000","last_active":"2023-05-22T15:26:54.055130+0000","last_peered":"2023-05-22T15:26:54.055130+0000","last_clean":"2023-05-22T15:26:54.055130+0000","last_became_active":"2023-05-22T15:26:53.646940+0000","last_became_peered":"2023-05-22T15:26:53.646940+0000","last_unstale":"2023-05-22T15:26:54.055130+0000","last_undegraded":"2023-05-22T15:26:54.055130+0000","last_fullsized":"2023-05-22T15:26:54.055130+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_clean_scrub_stamp":"2023-05-22T15:26:52.568670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T15:41:55.740195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7,4],"acting":[5,0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"22'2","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.634944+0000","last_change":"2023-05-22T15:26:49.210260+0000","last_active":"2023-05-22T15:26:53.634944+0000","last_peered":"2023-05-22T15:26:53.634944+0000","last_clean":"2023-05-22T15:26:53.634944+0000","last_became_active":"2023-05-22T15:26:49.209487+0000","last_became_peered":"2023-05-22T15:26:49.209487+0000","last_unstale":"2023-05-22T15:26:53.634944+0000","last_undegraded":"2023-05-22T15:26:53.634944+0000","last_fullsized":"2023-05-22T15:26:53.634944+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_clean_scrub_stamp":"2023-05-22T15:26:48.127413+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:07:49.933860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.619548+0000","last_change":"2023-05-22T15:26:49.595133+0000","last_active":"2023-05-22T15:26:53.619548+0000","last_peered":"2023-05-22T15:26:53.619548+0000","last_clean":"2023-05-22T15:26:53.619548+0000","last_became_active":"2023-05-22T15:26:49.594382+0000","last_became_peered":"2023-05-22T15:26:49.594382+0000","last_unstale":"2023-05-22T15:26:53.619548+0000","last_undegraded":"2023-05-22T15:26:53.619548+0000","last_fullsized":"2023-05-22T15:26:53.619548+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:11:42.159966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.982645+0000","last_change":"2023-05-22T15:26:53.657958+0000","last_active":"2023-05-22T15:26:53.982645+0000","last_peered":"2023-05-22T15:26:53.982645+0000","last_clean":"2023-05-22T15:26:53.982645+0000","last_became_active":"2023-05-22T15:26:53.656910+0000","last_became_peered":"2023-05-22T15:26:53.656910+0000","last_unstale":"2023-05-22T15:26:53.982645+0000","last_undegraded":"2023-05-22T15:26:53.982645+0000","last_fullsized":"2023-05-22T15:26:53.982645+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_clean_scrub_stamp":"2023-05-22T15:26:52.568670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:56:11.423640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1,6],"acting":[7,4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"22'2","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.634146+0000","last_change":"2023-05-22T15:26:49.210149+0000","last_active":"2023-05-22T15:26:53.634146+0000","last_peered":"2023-05-22T15:26:53.634146+0000","last_clean":"2023-05-22T15:26:53.634146+0000","last_became_active":"2023-05-22T15:26:49.209497+0000","last_became_peered":"2023-05-22T15:26:49.209497+0000","last_unstale":"2023-05-22T15:26:53.634146+0000","last_undegraded":"2023-05-22T15:26:53.634146+0000","last_fullsized":"2023-05-22T15:26:53.634146+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_clean_scrub_stamp":"2023-05-22T15:26:48.127413+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:47:17.588354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.620540+0000","last_change":"2023-05-22T15:26:49.598141+0000","last_active":"2023-05-22T15:26:53.620540+0000","last_peered":"2023-05-22T15:26:53.620540+0000","last_clean":"2023-05-22T15:26:53.620540+0000","last_became_active":"2023-05-22T15:26:49.597262+0000","last_became_peered":"2023-05-22T15:26:49.597262+0000","last_unstale":"2023-05-22T15:26:53.620540+0000","last_undegraded":"2023-05-22T15:26:53.620540+0000","last_fullsized":"2023-05-22T15:26:53.620540+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:33:23.357846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.980103+0000","last_change":"2023-05-22T15:26:53.656672+0000","last_active":"2023-05-22T15:26:53.980103+0000","last_peered":"2023-05-22T15:26:53.980103+0000","last_clean":"2023-05-22T15:26:53.980103+0000","last_became_active":"2023-05-22T15:26:53.656058+0000","last_became_peered":"2023-05-22T15:26:53.656058+0000","last_unstale":"2023-05-22T15:26:53.980103+0000","last_undegraded":"2023-05-22T15:26:53.980103+0000","last_fullsized":"2023-05-22T15:26:53.980103+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_clean_scrub_stamp":"2023-05-22T15:26:52.568670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:38:53.647917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,0],"acting":[1,3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"22'1","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.612141+0000","last_change":"2023-05-22T15:26:49.206147+0000","last_active":"2023-05-22T15:26:53.612141+0000","last_peered":"2023-05-22T15:26:53.612141+0000","last_clean":"2023-05-22T15:26:53.612141+0000","last_became_active":"2023-05-22T15:26:49.205706+0000","last_became_peered":"2023-05-22T15:26:49.205706+0000","last_unstale":"2023-05-22T15:26:53.612141+0000","last_undegraded":"2023-05-22T15:26:53.612141+0000","last_fullsized":"2023-05-22T15:26:53.612141+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_clean_scrub_stamp":"2023-05-22T15:26:48.127413+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:09:00.816941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.601812+0000","last_change":"2023-05-22T15:26:49.578140+0000","last_active":"2023-05-22T15:26:53.601812+0000","last_peered":"2023-05-22T15:26:53.601812+0000","last_clean":"2023-05-22T15:26:53.601812+0000","last_became_active":"2023-05-22T15:26:49.576922+0000","last_became_peered":"2023-05-22T15:26:49.576922+0000","last_unstale":"2023-05-22T15:26:53.601812+0000","last_undegraded":"2023-05-22T15:26:53.601812+0000","last_fullsized":"2023-05-22T15:26:53.601812+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T16:16:40.288204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-05-22T15:26:54.054907+0000","last_change":"2023-05-22T15:26:53.643098+0000","last_active":"2023-05-22T15:26:54.054907+0000","last_peered":"2023-05-22T15:26:54.054907+0000","last_clean":"2023-05-22T15:26:54.054907+0000","last_became_active":"2023-05-22T15:26:53.642038+0000","last_became_peered":"2023-05-22T15:26:53.642038+0000","last_unstale":"2023-05-22T15:26:54.054907+0000","last_undegraded":"2023-05-22T15:26:54.054907+0000","last_fullsized":"2023-05-22T15:26:54.054907+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_clean_scrub_stamp":"2023-05-22T15:26:52.568670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T18:43:09.309787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2,0],"acting":[4,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"22'1","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.620161+0000","last_change":"2023-05-22T15:26:49.199743+0000","last_active":"2023-05-22T15:26:53.620161+0000","last_peered":"2023-05-22T15:26:53.620161+0000","last_clean":"2023-05-22T15:26:53.620161+0000","last_became_active":"2023-05-22T15:26:49.197695+0000","last_became_peered":"2023-05-22T15:26:49.197695+0000","last_unstale":"2023-05-22T15:26:53.620161+0000","last_undegraded":"2023-05-22T15:26:53.620161+0000","last_fullsized":"2023-05-22T15:26:53.620161+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_clean_scrub_stamp":"2023-05-22T15:26:48.127413+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:21:54.376718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.602293+0000","last_change":"2023-05-22T15:26:49.580932+0000","last_active":"2023-05-22T15:26:53.602293+0000","last_peered":"2023-05-22T15:26:53.602293+0000","last_clean":"2023-05-22T15:26:53.602293+0000","last_became_active":"2023-05-22T15:26:49.580612+0000","last_became_peered":"2023-05-22T15:26:49.580612+0000","last_unstale":"2023-05-22T15:26:53.602293+0000","last_undegraded":"2023-05-22T15:26:53.602293+0000","last_fullsized":"2023-05-22T15:26:53.602293+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:19:30.391062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-05-22T15:26:54.051798+0000","last_change":"2023-05-22T15:26:53.662170+0000","last_active":"2023-05-22T15:26:54.051798+0000","last_peered":"2023-05-22T15:26:54.051798+0000","last_clean":"2023-05-22T15:26:54.051798+0000","last_became_active":"2023-05-22T15:26:53.661164+0000","last_became_peered":"2023-05-22T15:26:53.661164+0000","last_unstale":"2023-05-22T15:26:54.051798+0000","last_undegraded":"2023-05-22T15:26:54.051798+0000","last_fullsized":"2023-05-22T15:26:54.051798+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_clean_scrub_stamp":"2023-05-22T15:26:52.568670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:46:53.737748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3,7],"acting":[6,4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.613865+0000","last_change":"2023-05-22T15:26:49.187546+0000","last_active":"2023-05-22T15:26:53.613865+0000","last_peered":"2023-05-22T15:26:53.613865+0000","last_clean":"2023-05-22T15:26:53.613865+0000","last_became_active":"2023-05-22T15:26:49.187022+0000","last_became_peered":"2023-05-22T15:26:49.187022+0000","last_unstale":"2023-05-22T15:26:53.613865+0000","last_undegraded":"2023-05-22T15:26:53.613865+0000","last_fullsized":"2023-05-22T15:26:53.613865+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_clean_scrub_stamp":"2023-05-22T15:26:48.127413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:53:34.802286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.611910+0000","last_change":"2023-05-22T15:26:49.623535+0000","last_active":"2023-05-22T15:26:53.611910+0000","last_peered":"2023-05-22T15:26:53.611910+0000","last_clean":"2023-05-22T15:26:53.611910+0000","last_became_active":"2023-05-22T15:26:49.622454+0000","last_became_peered":"2023-05-22T15:26:49.622454+0000","last_unstale":"2023-05-22T15:26:53.611910+0000","last_undegraded":"2023-05-22T15:26:53.611910+0000","last_fullsized":"2023-05-22T15:26:53.611910+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:43:00.518715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-05-22T15:26:54.048982+0000","last_change":"2023-05-22T15:26:53.645181+0000","last_active":"2023-05-22T15:26:54.048982+0000","last_peered":"2023-05-22T15:26:54.048982+0000","last_clean":"2023-05-22T15:26:54.048982+0000","last_became_active":"2023-05-22T15:26:53.644764+0000","last_became_peered":"2023-05-22T15:26:53.644764+0000","last_unstale":"2023-05-22T15:26:54.048982+0000","last_undegraded":"2023-05-22T15:26:54.048982+0000","last_fullsized":"2023-05-22T15:26:54.048982+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_clean_scrub_stamp":"2023-05-22T15:26:52.568670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:12:09.126657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1,2],"acting":[0,7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.603942+0000","last_change":"2023-05-22T15:26:49.188920+0000","last_active":"2023-05-22T15:26:53.603942+0000","last_peered":"2023-05-22T15:26:53.603942+0000","last_clean":"2023-05-22T15:26:53.603942+0000","last_became_active":"2023-05-22T15:26:49.188538+0000","last_became_peered":"2023-05-22T15:26:49.188538+0000","last_unstale":"2023-05-22T15:26:53.603942+0000","last_undegraded":"2023-05-22T15:26:53.603942+0000","last_fullsized":"2023-05-22T15:26:53.603942+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_clean_scrub_stamp":"2023-05-22T15:26:48.127413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:09:00.816941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.603003+0000","last_change":"2023-05-22T15:26:49.606092+0000","last_active":"2023-05-22T15:26:53.603003+0000","last_peered":"2023-05-22T15:26:53.603003+0000","last_clean":"2023-05-22T15:26:53.603003+0000","last_became_active":"2023-05-22T15:26:49.605324+0000","last_became_peered":"2023-05-22T15:26:49.605324+0000","last_unstale":"2023-05-22T15:26:53.603003+0000","last_undegraded":"2023-05-22T15:26:53.603003+0000","last_fullsized":"2023-05-22T15:26:53.603003+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:10:51.435508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.603748+0000","last_change":"2023-05-22T15:26:49.578105+0000","last_active":"2023-05-22T15:26:53.603748+0000","last_peered":"2023-05-22T15:26:53.603748+0000","last_clean":"2023-05-22T15:26:53.603748+0000","last_became_active":"2023-05-22T15:26:49.577534+0000","last_became_peered":"2023-05-22T15:26:49.577534+0000","last_unstale":"2023-05-22T15:26:53.603748+0000","last_undegraded":"2023-05-22T15:26:53.603748+0000","last_fullsized":"2023-05-22T15:26:53.603748+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:44:52.369716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.612292+0000","last_change":"2023-05-22T15:26:49.186504+0000","last_active":"2023-05-22T15:26:53.612292+0000","last_peered":"2023-05-22T15:26:53.612292+0000","last_clean":"2023-05-22T15:26:53.612292+0000","last_became_active":"2023-05-22T15:26:49.185880+0000","last_became_peered":"2023-05-22T15:26:49.185880+0000","last_unstale":"2023-05-22T15:26:53.612292+0000","last_undegraded":"2023-05-22T15:26:53.612292+0000","last_fullsized":"2023-05-22T15:26:53.612292+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_clean_scrub_stamp":"2023-05-22T15:26:48.127413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T18:13:58.291903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-05-22T15:26:54.051126+0000","last_change":"2023-05-22T15:26:53.647905+0000","last_active":"2023-05-22T15:26:54.051126+0000","last_peered":"2023-05-22T15:26:54.051126+0000","last_clean":"2023-05-22T15:26:54.051126+0000","last_became_active":"2023-05-22T15:26:53.646790+0000","last_became_peered":"2023-05-22T15:26:53.646790+0000","last_unstale":"2023-05-22T15:26:54.051126+0000","last_undegraded":"2023-05-22T15:26:54.051126+0000","last_fullsized":"2023-05-22T15:26:54.051126+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_clean_scrub_stamp":"2023-05-22T15:26:52.568670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T03:18:03.204915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7,1],"acting":[6,4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.603724+0000","last_change":"2023-05-22T15:26:49.623613+0000","last_active":"2023-05-22T15:26:53.603724+0000","last_peered":"2023-05-22T15:26:53.603724+0000","last_clean":"2023-05-22T15:26:53.603724+0000","last_became_active":"2023-05-22T15:26:49.623385+0000","last_became_peered":"2023-05-22T15:26:49.623385+0000","last_unstale":"2023-05-22T15:26:53.603724+0000","last_undegraded":"2023-05-22T15:26:53.603724+0000","last_fullsized":"2023-05-22T15:26:53.603724+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:22:10.117410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.635389+0000","last_change":"2023-05-22T15:26:49.178387+0000","last_active":"2023-05-22T15:26:53.635389+0000","last_peered":"2023-05-22T15:26:53.635389+0000","last_clean":"2023-05-22T15:26:53.635389+0000","last_became_active":"2023-05-22T15:26:49.177428+0000","last_became_peered":"2023-05-22T15:26:49.177428+0000","last_unstale":"2023-05-22T15:26:53.635389+0000","last_undegraded":"2023-05-22T15:26:53.635389+0000","last_fullsized":"2023-05-22T15:26:53.635389+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_clean_scrub_stamp":"2023-05-22T15:26:48.127413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:07:41.791660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.981431+0000","last_change":"2023-05-22T15:26:53.660574+0000","last_active":"2023-05-22T15:26:53.981431+0000","last_peered":"2023-05-22T15:26:53.981431+0000","last_clean":"2023-05-22T15:26:53.981431+0000","last_became_active":"2023-05-22T15:26:53.659567+0000","last_became_peered":"2023-05-22T15:26:53.659567+0000","last_unstale":"2023-05-22T15:26:53.981431+0000","last_undegraded":"2023-05-22T15:26:53.981431+0000","last_fullsized":"2023-05-22T15:26:53.981431+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_clean_scrub_stamp":"2023-05-22T15:26:52.568670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:07:24.608031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6,0],"acting":[7,4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.15","version":"22'3","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.610146+0000","last_change":"2023-05-22T15:26:49.188354+0000","last_active":"2023-05-22T15:26:53.610146+0000","last_peered":"2023-05-22T15:26:53.610146+0000","last_clean":"2023-05-22T15:26:53.610146+0000","last_became_active":"2023-05-22T15:26:49.187902+0000","last_became_peered":"2023-05-22T15:26:49.187902+0000","last_unstale":"2023-05-22T15:26:53.610146+0000","last_undegraded":"2023-05-22T15:26:53.610146+0000","last_fullsized":"2023-05-22T15:26:53.610146+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_clean_scrub_stamp":"2023-05-22T15:26:48.127413+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:07:49.933860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.612914+0000","last_change":"2023-05-22T15:26:49.619577+0000","last_active":"2023-05-22T15:26:53.612914+0000","last_peered":"2023-05-22T15:26:53.612914+0000","last_clean":"2023-05-22T15:26:53.612914+0000","last_became_active":"2023-05-22T15:26:49.619210+0000","last_became_peered":"2023-05-22T15:26:49.619210+0000","last_unstale":"2023-05-22T15:26:53.612914+0000","last_undegraded":"2023-05-22T15:26:53.612914+0000","last_fullsized":"2023-05-22T15:26:53.612914+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:48:33.611833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.992745+0000","last_change":"2023-05-22T15:26:53.650745+0000","last_active":"2023-05-22T15:26:53.992745+0000","last_peered":"2023-05-22T15:26:53.992745+0000","last_clean":"2023-05-22T15:26:53.992745+0000","last_became_active":"2023-05-22T15:26:53.650209+0000","last_became_peered":"2023-05-22T15:26:53.650209+0000","last_unstale":"2023-05-22T15:26:53.992745+0000","last_undegraded":"2023-05-22T15:26:53.992745+0000","last_fullsized":"2023-05-22T15:26:53.992745+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_clean_scrub_stamp":"2023-05-22T15:26:52.568670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:15:56.755592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,5],"acting":[3,0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.14","version":"27'9","reported_seq":23,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-05-22T15:26:59.032004+0000","last_change":"2023-05-22T15:26:49.201484+0000","last_active":"2023-05-22T15:26:59.032004+0000","last_peered":"2023-05-22T15:26:59.032004+0000","last_clean":"2023-05-22T15:26:59.032004+0000","last_became_active":"2023-05-22T15:26:49.200972+0000","last_became_peered":"2023-05-22T15:26:49.200972+0000","last_unstale":"2023-05-22T15:26:59.032004+0000","last_undegraded":"2023-05-22T15:26:59.032004+0000","last_fullsized":"2023-05-22T15:26:59.032004+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_clean_scrub_stamp":"2023-05-22T15:26:48.127413+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:59:54.009838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3745,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.620650+0000","last_change":"2023-05-22T15:26:49.619268+0000","last_active":"2023-05-22T15:26:53.620650+0000","last_peered":"2023-05-22T15:26:53.620650+0000","last_clean":"2023-05-22T15:26:53.620650+0000","last_became_active":"2023-05-22T15:26:49.618724+0000","last_became_peered":"2023-05-22T15:26:49.618724+0000","last_unstale":"2023-05-22T15:26:53.620650+0000","last_undegraded":"2023-05-22T15:26:53.620650+0000","last_fullsized":"2023-05-22T15:26:53.620650+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:28:29.607131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.979878+0000","last_change":"2023-05-22T15:26:53.653181+0000","last_active":"2023-05-22T15:26:53.979878+0000","last_peered":"2023-05-22T15:26:53.979878+0000","last_clean":"2023-05-22T15:26:53.979878+0000","last_became_active":"2023-05-22T15:26:53.652676+0000","last_became_peered":"2023-05-22T15:26:53.652676+0000","last_unstale":"2023-05-22T15:26:53.979878+0000","last_undegraded":"2023-05-22T15:26:53.979878+0000","last_fullsized":"2023-05-22T15:26:53.979878+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_clean_scrub_stamp":"2023-05-22T15:26:52.568670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T15:46:20.544673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3,7],"acting":[1,5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.616683+0000","last_change":"2023-05-22T15:26:49.597250+0000","last_active":"2023-05-22T15:26:53.616683+0000","last_peered":"2023-05-22T15:26:53.616683+0000","last_clean":"2023-05-22T15:26:53.616683+0000","last_became_active":"2023-05-22T15:26:49.596487+0000","last_became_peered":"2023-05-22T15:26:49.596487+0000","last_unstale":"2023-05-22T15:26:53.616683+0000","last_undegraded":"2023-05-22T15:26:53.616683+0000","last_fullsized":"2023-05-22T15:26:53.616683+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:18:20.756810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.603054+0000","last_change":"2023-05-22T15:26:44.982421+0000","last_active":"2023-05-22T15:26:53.603054+0000","last_peered":"2023-05-22T15:26:53.603054+0000","last_clean":"2023-05-22T15:26:53.603054+0000","last_became_active":"2023-05-22T15:26:44.981586+0000","last_became_peered":"2023-05-22T15:26:44.981586+0000","last_unstale":"2023-05-22T15:26:53.603054+0000","last_undegraded":"2023-05-22T15:26:53.603054+0000","last_fullsized":"2023-05-22T15:26:53.603054+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:14:11.156544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"22'1","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.612191+0000","last_change":"2023-05-22T15:26:49.185412+0000","last_active":"2023-05-22T15:26:53.612191+0000","last_peered":"2023-05-22T15:26:53.612191+0000","last_clean":"2023-05-22T15:26:53.612191+0000","last_became_active":"2023-05-22T15:26:49.184896+0000","last_became_peered":"2023-05-22T15:26:49.184896+0000","last_unstale":"2023-05-22T15:26:53.612191+0000","last_undegraded":"2023-05-22T15:26:53.612191+0000","last_fullsized":"2023-05-22T15:26:53.612191+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_clean_scrub_stamp":"2023-05-22T15:26:48.127413+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:13:03.975127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.992364+0000","last_change":"2023-05-22T15:26:53.669441+0000","last_active":"2023-05-22T15:26:53.992364+0000","last_peered":"2023-05-22T15:26:53.992364+0000","last_clean":"2023-05-22T15:26:53.992364+0000","last_became_active":"2023-05-22T15:26:53.669000+0000","last_became_peered":"2023-05-22T15:26:53.669000+0000","last_unstale":"2023-05-22T15:26:53.992364+0000","last_undegraded":"2023-05-22T15:26:53.992364+0000","last_fullsized":"2023-05-22T15:26:53.992364+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_clean_scrub_stamp":"2023-05-22T15:26:52.568670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T18:55:06.398032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,0],"acting":[3,1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.603843+0000","last_change":"2023-05-22T15:26:49.623319+0000","last_active":"2023-05-22T15:26:53.603843+0000","last_peered":"2023-05-22T15:26:53.603843+0000","last_clean":"2023-05-22T15:26:53.603843+0000","last_became_active":"2023-05-22T15:26:49.622892+0000","last_became_peered":"2023-05-22T15:26:49.622892+0000","last_unstale":"2023-05-22T15:26:53.603843+0000","last_undegraded":"2023-05-22T15:26:53.603843+0000","last_fullsized":"2023-05-22T15:26:53.603843+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:48:33.611833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.612572+0000","last_change":"2023-05-22T15:26:44.978858+0000","last_active":"2023-05-22T15:26:53.612572+0000","last_peered":"2023-05-22T15:26:53.612572+0000","last_clean":"2023-05-22T15:26:53.612572+0000","last_became_active":"2023-05-22T15:26:44.978443+0000","last_became_peered":"2023-05-22T15:26:44.978443+0000","last_unstale":"2023-05-22T15:26:53.612572+0000","last_undegraded":"2023-05-22T15:26:53.612572+0000","last_fullsized":"2023-05-22T15:26:53.612572+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:46:29.290243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"22'2","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.606372+0000","last_change":"2023-05-22T15:26:49.188060+0000","last_active":"2023-05-22T15:26:53.606372+0000","last_peered":"2023-05-22T15:26:53.606372+0000","last_clean":"2023-05-22T15:26:53.606372+0000","last_became_active":"2023-05-22T15:26:49.187586+0000","last_became_peered":"2023-05-22T15:26:49.187586+0000","last_unstale":"2023-05-22T15:26:53.606372+0000","last_undegraded":"2023-05-22T15:26:53.606372+0000","last_fullsized":"2023-05-22T15:26:53.606372+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_clean_scrub_stamp":"2023-05-22T15:26:48.127413+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:59:54.009838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-05-22T15:26:54.053713+0000","last_change":"2023-05-22T15:26:53.659559+0000","last_active":"2023-05-22T15:26:54.053713+0000","last_peered":"2023-05-22T15:26:54.053713+0000","last_clean":"2023-05-22T15:26:54.053713+0000","last_became_active":"2023-05-22T15:26:53.655056+0000","last_became_peered":"2023-05-22T15:26:53.655056+0000","last_unstale":"2023-05-22T15:26:54.053713+0000","last_undegraded":"2023-05-22T15:26:54.053713+0000","last_fullsized":"2023-05-22T15:26:54.053713+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_clean_scrub_stamp":"2023-05-22T15:26:52.568670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:47:56.845713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7,0],"acting":[4,3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.611858+0000","last_change":"2023-05-22T15:26:49.589781+0000","last_active":"2023-05-22T15:26:53.611858+0000","last_peered":"2023-05-22T15:26:53.611858+0000","last_clean":"2023-05-22T15:26:53.611858+0000","last_became_active":"2023-05-22T15:26:49.588830+0000","last_became_peered":"2023-05-22T15:26:49.588830+0000","last_unstale":"2023-05-22T15:26:53.611858+0000","last_undegraded":"2023-05-22T15:26:53.611858+0000","last_fullsized":"2023-05-22T15:26:53.611858+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:52:48.123148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.615526+0000","last_change":"2023-05-22T15:26:45.092367+0000","last_active":"2023-05-22T15:26:53.615526+0000","last_peered":"2023-05-22T15:26:53.615526+0000","last_clean":"2023-05-22T15:26:53.615526+0000","last_became_active":"2023-05-22T15:26:45.091951+0000","last_became_peered":"2023-05-22T15:26:45.091951+0000","last_unstale":"2023-05-22T15:26:53.615526+0000","last_undegraded":"2023-05-22T15:26:53.615526+0000","last_fullsized":"2023-05-22T15:26:53.615526+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:42:57.973030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.619125+0000","last_change":"2023-05-22T15:26:49.180237+0000","last_active":"2023-05-22T15:26:53.619125+0000","last_peered":"2023-05-22T15:26:53.619125+0000","last_clean":"2023-05-22T15:26:53.619125+0000","last_became_active":"2023-05-22T15:26:49.179622+0000","last_became_peered":"2023-05-22T15:26:49.179622+0000","last_unstale":"2023-05-22T15:26:53.619125+0000","last_undegraded":"2023-05-22T15:26:53.619125+0000","last_fullsized":"2023-05-22T15:26:53.619125+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_clean_scrub_stamp":"2023-05-22T15:26:48.127413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T16:16:39.277093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-05-22T15:26:54.051370+0000","last_change":"2023-05-22T15:26:53.623285+0000","last_active":"2023-05-22T15:26:54.051370+0000","last_peered":"2023-05-22T15:26:54.051370+0000","last_clean":"2023-05-22T15:26:54.051370+0000","last_became_active":"2023-05-22T15:26:53.622116+0000","last_became_peered":"2023-05-22T15:26:53.622116+0000","last_unstale":"2023-05-22T15:26:54.051370+0000","last_undegraded":"2023-05-22T15:26:54.051370+0000","last_fullsized":"2023-05-22T15:26:54.051370+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_clean_scrub_stamp":"2023-05-22T15:26:52.568670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T16:31:40.104597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5,2],"acting":[6,0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.613088+0000","last_change":"2023-05-22T15:26:49.576833+0000","last_active":"2023-05-22T15:26:53.613088+0000","last_peered":"2023-05-22T15:26:53.613088+0000","last_clean":"2023-05-22T15:26:53.613088+0000","last_became_active":"2023-05-22T15:26:49.576559+0000","last_became_peered":"2023-05-22T15:26:49.576559+0000","last_unstale":"2023-05-22T15:26:53.613088+0000","last_undegraded":"2023-05-22T15:26:53.613088+0000","last_fullsized":"2023-05-22T15:26:53.613088+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:32:19.582542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.619335+0000","last_change":"2023-05-22T15:26:45.091475+0000","last_active":"2023-05-22T15:26:53.619335+0000","last_peered":"2023-05-22T15:26:53.619335+0000","last_clean":"2023-05-22T15:26:53.619335+0000","last_became_active":"2023-05-22T15:26:45.091143+0000","last_became_peered":"2023-05-22T15:26:45.091143+0000","last_unstale":"2023-05-22T15:26:53.619335+0000","last_undegraded":"2023-05-22T15:26:53.619335+0000","last_fullsized":"2023-05-22T15:26:53.619335+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:12:41.464054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.635151+0000","last_change":"2023-05-22T15:26:49.187061+0000","last_active":"2023-05-22T15:26:53.635151+0000","last_peered":"2023-05-22T15:26:53.635151+0000","last_clean":"2023-05-22T15:26:53.635151+0000","last_became_active":"2023-05-22T15:26:49.185965+0000","last_became_peered":"2023-05-22T15:26:49.185965+0000","last_unstale":"2023-05-22T15:26:53.635151+0000","last_undegraded":"2023-05-22T15:26:53.635151+0000","last_fullsized":"2023-05-22T15:26:53.635151+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_clean_scrub_stamp":"2023-05-22T15:26:48.127413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T16:16:39.277093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-05-22T15:26:54.048601+0000","last_change":"2023-05-22T15:26:53.650446+0000","last_active":"2023-05-22T15:26:54.048601+0000","last_peered":"2023-05-22T15:26:54.048601+0000","last_clean":"2023-05-22T15:26:54.048601+0000","last_became_active":"2023-05-22T15:26:53.649492+0000","last_became_peered":"2023-05-22T15:26:53.649492+0000","last_unstale":"2023-05-22T15:26:54.048601+0000","last_undegraded":"2023-05-22T15:26:54.048601+0000","last_fullsized":"2023-05-22T15:26:54.048601+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_clean_scrub_stamp":"2023-05-22T15:26:52.568670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:56:11.423640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6,5],"acting":[0,1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.619014+0000","last_change":"2023-05-22T15:26:49.581808+0000","last_active":"2023-05-22T15:26:53.619014+0000","last_peered":"2023-05-22T15:26:53.619014+0000","last_clean":"2023-05-22T15:26:53.619014+0000","last_became_active":"2023-05-22T15:26:49.581501+0000","last_became_peered":"2023-05-22T15:26:49.581501+0000","last_unstale":"2023-05-22T15:26:53.619014+0000","last_undegraded":"2023-05-22T15:26:53.619014+0000","last_fullsized":"2023-05-22T15:26:53.619014+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:52:48.123148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.620868+0000","last_change":"2023-05-22T15:26:49.204122+0000","last_active":"2023-05-22T15:26:53.620868+0000","last_peered":"2023-05-22T15:26:53.620868+0000","last_clean":"2023-05-22T15:26:53.620868+0000","last_became_active":"2023-05-22T15:26:49.199596+0000","last_became_peered":"2023-05-22T15:26:49.199596+0000","last_unstale":"2023-05-22T15:26:53.620868+0000","last_undegraded":"2023-05-22T15:26:53.620868+0000","last_fullsized":"2023-05-22T15:26:53.620868+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_clean_scrub_stamp":"2023-05-22T15:26:48.127413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:09:00.816941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.635108+0000","last_change":"2023-05-22T15:26:44.976632+0000","last_active":"2023-05-22T15:26:53.635108+0000","last_peered":"2023-05-22T15:26:53.635108+0000","last_clean":"2023-05-22T15:26:53.635108+0000","last_became_active":"2023-05-22T15:26:44.975546+0000","last_became_peered":"2023-05-22T15:26:44.975546+0000","last_unstale":"2023-05-22T15:26:53.635108+0000","last_undegraded":"2023-05-22T15:26:53.635108+0000","last_fullsized":"2023-05-22T15:26:53.635108+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:42:57.973030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-05-22T15:26:54.048781+0000","last_change":"2023-05-22T15:26:53.669482+0000","last_active":"2023-05-22T15:26:54.048781+0000","last_peered":"2023-05-22T15:26:54.048781+0000","last_clean":"2023-05-22T15:26:54.048781+0000","last_became_active":"2023-05-22T15:26:53.669119+0000","last_became_peered":"2023-05-22T15:26:53.669119+0000","last_unstale":"2023-05-22T15:26:54.048781+0000","last_undegraded":"2023-05-22T15:26:54.048781+0000","last_fullsized":"2023-05-22T15:26:54.048781+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_clean_scrub_stamp":"2023-05-22T15:26:52.568670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:38:53.647917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,6],"acting":[0,1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.613087+0000","last_change":"2023-05-22T15:26:49.589555+0000","last_active":"2023-05-22T15:26:53.613087+0000","last_peered":"2023-05-22T15:26:53.613087+0000","last_clean":"2023-05-22T15:26:53.613087+0000","last_became_active":"2023-05-22T15:26:49.589271+0000","last_became_peered":"2023-05-22T15:26:49.589271+0000","last_unstale":"2023-05-22T15:26:53.613087+0000","last_undegraded":"2023-05-22T15:26:53.613087+0000","last_fullsized":"2023-05-22T15:26:53.613087+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:43:00.518715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.614048+0000","last_change":"2023-05-22T15:26:49.179359+0000","last_active":"2023-05-22T15:26:53.614048+0000","last_peered":"2023-05-22T15:26:53.614048+0000","last_clean":"2023-05-22T15:26:53.614048+0000","last_became_active":"2023-05-22T15:26:49.178930+0000","last_became_peered":"2023-05-22T15:26:49.178930+0000","last_unstale":"2023-05-22T15:26:53.614048+0000","last_undegraded":"2023-05-22T15:26:53.614048+0000","last_fullsized":"2023-05-22T15:26:53.614048+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_clean_scrub_stamp":"2023-05-22T15:26:48.127413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:59:54.009838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.597765+0000","last_change":"2023-05-22T15:26:44.969469+0000","last_active":"2023-05-22T15:26:53.597765+0000","last_peered":"2023-05-22T15:26:53.597765+0000","last_clean":"2023-05-22T15:26:53.597765+0000","last_became_active":"2023-05-22T15:26:44.969083+0000","last_became_peered":"2023-05-22T15:26:44.969083+0000","last_unstale":"2023-05-22T15:26:53.597765+0000","last_undegraded":"2023-05-22T15:26:53.597765+0000","last_fullsized":"2023-05-22T15:26:53.597765+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:29:23.670067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.978603+0000","last_change":"2023-05-22T15:26:53.636902+0000","last_active":"2023-05-22T15:26:53.978603+0000","last_peered":"2023-05-22T15:26:53.978603+0000","last_clean":"2023-05-22T15:26:53.978603+0000","last_became_active":"2023-05-22T15:26:53.636337+0000","last_became_peered":"2023-05-22T15:26:53.636337+0000","last_unstale":"2023-05-22T15:26:53.978603+0000","last_undegraded":"2023-05-22T15:26:53.978603+0000","last_fullsized":"2023-05-22T15:26:53.978603+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_clean_scrub_stamp":"2023-05-22T15:26:52.568670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:58:01.437826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1,7],"acting":[2,5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.612001+0000","last_change":"2023-05-22T15:26:49.581103+0000","last_active":"2023-05-22T15:26:53.612001+0000","last_peered":"2023-05-22T15:26:53.612001+0000","last_clean":"2023-05-22T15:26:53.612001+0000","last_became_active":"2023-05-22T15:26:49.580823+0000","last_became_peered":"2023-05-22T15:26:49.580823+0000","last_unstale":"2023-05-22T15:26:53.612001+0000","last_undegraded":"2023-05-22T15:26:53.612001+0000","last_fullsized":"2023-05-22T15:26:53.612001+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:32:19.582542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.611473+0000","last_change":"2023-05-22T15:26:49.205877+0000","last_active":"2023-05-22T15:26:53.611473+0000","last_peered":"2023-05-22T15:26:53.611473+0000","last_clean":"2023-05-22T15:26:53.611473+0000","last_became_active":"2023-05-22T15:26:49.205432+0000","last_became_peered":"2023-05-22T15:26:49.205432+0000","last_unstale":"2023-05-22T15:26:53.611473+0000","last_undegraded":"2023-05-22T15:26:53.611473+0000","last_fullsized":"2023-05-22T15:26:53.611473+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_clean_scrub_stamp":"2023-05-22T15:26:48.127413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:59:54.009838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.609876+0000","last_change":"2023-05-22T15:26:44.977198+0000","last_active":"2023-05-22T15:26:53.609876+0000","last_peered":"2023-05-22T15:26:53.609876+0000","last_clean":"2023-05-22T15:26:53.609876+0000","last_became_active":"2023-05-22T15:26:44.976445+0000","last_became_peered":"2023-05-22T15:26:44.976445+0000","last_unstale":"2023-05-22T15:26:53.609876+0000","last_undegraded":"2023-05-22T15:26:53.609876+0000","last_fullsized":"2023-05-22T15:26:53.609876+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:12:41.464054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-05-22T15:26:54.056650+0000","last_change":"2023-05-22T15:26:53.667459+0000","last_active":"2023-05-22T15:26:54.056650+0000","last_peered":"2023-05-22T15:26:54.056650+0000","last_clean":"2023-05-22T15:26:54.056650+0000","last_became_active":"2023-05-22T15:26:53.666553+0000","last_became_peered":"2023-05-22T15:26:53.666553+0000","last_unstale":"2023-05-22T15:26:54.056650+0000","last_undegraded":"2023-05-22T15:26:54.056650+0000","last_fullsized":"2023-05-22T15:26:54.056650+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_clean_scrub_stamp":"2023-05-22T15:26:52.568670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:08:31.638326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0,4],"acting":[5,1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.602418+0000","last_change":"2023-05-22T15:26:49.606241+0000","last_active":"2023-05-22T15:26:53.602418+0000","last_peered":"2023-05-22T15:26:53.602418+0000","last_clean":"2023-05-22T15:26:53.602418+0000","last_became_active":"2023-05-22T15:26:49.605628+0000","last_became_peered":"2023-05-22T15:26:49.605628+0000","last_unstale":"2023-05-22T15:26:53.602418+0000","last_undegraded":"2023-05-22T15:26:53.602418+0000","last_fullsized":"2023-05-22T15:26:53.602418+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:59:23.805894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.602217+0000","last_change":"2023-05-22T15:26:49.185421+0000","last_active":"2023-05-22T15:26:53.602217+0000","last_peered":"2023-05-22T15:26:53.602217+0000","last_clean":"2023-05-22T15:26:53.602217+0000","last_became_active":"2023-05-22T15:26:49.184948+0000","last_became_peered":"2023-05-22T15:26:49.184948+0000","last_unstale":"2023-05-22T15:26:53.602217+0000","last_undegraded":"2023-05-22T15:26:53.602217+0000","last_fullsized":"2023-05-22T15:26:53.602217+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_clean_scrub_stamp":"2023-05-22T15:26:48.127413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:21:54.376718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.634293+0000","last_change":"2023-05-22T15:26:44.971951+0000","last_active":"2023-05-22T15:26:53.634293+0000","last_peered":"2023-05-22T15:26:53.634293+0000","last_clean":"2023-05-22T15:26:53.634293+0000","last_became_active":"2023-05-22T15:26:44.970872+0000","last_became_peered":"2023-05-22T15:26:44.970872+0000","last_unstale":"2023-05-22T15:26:53.634293+0000","last_undegraded":"2023-05-22T15:26:53.634293+0000","last_fullsized":"2023-05-22T15:26:53.634293+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:55:34.752949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":169,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.610124+0000","last_change":"2023-05-22T15:26:39.318204+0000","last_active":"2023-05-22T15:26:53.610124+0000","last_peered":"2023-05-22T15:26:53.610124+0000","last_clean":"2023-05-22T15:26:53.610124+0000","last_became_active":"2023-05-22T15:26:39.315370+0000","last_became_peered":"2023-05-22T15:26:39.315370+0000","last_unstale":"2023-05-22T15:26:53.610124+0000","last_undegraded":"2023-05-22T15:26:53.610124+0000","last_fullsized":"2023-05-22T15:26:53.610124+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:38.020076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:38.020076+0000","last_clean_scrub_stamp":"2023-05-22T15:26:38.020076+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:06:39.207543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.982716+0000","last_change":"2023-05-22T15:26:53.649648+0000","last_active":"2023-05-22T15:26:53.982716+0000","last_peered":"2023-05-22T15:26:53.982716+0000","last_clean":"2023-05-22T15:26:53.982716+0000","last_became_active":"2023-05-22T15:26:53.648664+0000","last_became_peered":"2023-05-22T15:26:53.648664+0000","last_unstale":"2023-05-22T15:26:53.982716+0000","last_undegraded":"2023-05-22T15:26:53.982716+0000","last_fullsized":"2023-05-22T15:26:53.982716+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_clean_scrub_stamp":"2023-05-22T15:26:52.568670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T03:14:12.733785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5,3],"acting":[7,2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.612919+0000","last_change":"2023-05-22T15:26:49.578502+0000","last_active":"2023-05-22T15:26:53.612919+0000","last_peered":"2023-05-22T15:26:53.612919+0000","last_clean":"2023-05-22T15:26:53.612919+0000","last_became_active":"2023-05-22T15:26:49.577743+0000","last_became_peered":"2023-05-22T15:26:49.577743+0000","last_unstale":"2023-05-22T15:26:53.612919+0000","last_undegraded":"2023-05-22T15:26:53.612919+0000","last_fullsized":"2023-05-22T15:26:53.612919+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:22:10.117410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.604031+0000","last_change":"2023-05-22T15:26:49.191706+0000","last_active":"2023-05-22T15:26:53.604031+0000","last_peered":"2023-05-22T15:26:53.604031+0000","last_clean":"2023-05-22T15:26:53.604031+0000","last_became_active":"2023-05-22T15:26:49.190396+0000","last_became_peered":"2023-05-22T15:26:49.190396+0000","last_unstale":"2023-05-22T15:26:53.604031+0000","last_undegraded":"2023-05-22T15:26:53.604031+0000","last_fullsized":"2023-05-22T15:26:53.604031+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_clean_scrub_stamp":"2023-05-22T15:26:48.127413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:13:03.975127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-05-22T15:26:54.055266+0000","last_change":"2023-05-22T15:26:53.656007+0000","last_active":"2023-05-22T15:26:54.055266+0000","last_peered":"2023-05-22T15:26:54.055266+0000","last_clean":"2023-05-22T15:26:54.055266+0000","last_became_active":"2023-05-22T15:26:53.652989+0000","last_became_peered":"2023-05-22T15:26:53.652989+0000","last_unstale":"2023-05-22T15:26:54.055266+0000","last_undegraded":"2023-05-22T15:26:54.055266+0000","last_fullsized":"2023-05-22T15:26:54.055266+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_clean_scrub_stamp":"2023-05-22T15:26:52.568670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:36:00.298083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0,2],"acting":[4,5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.621541+0000","last_change":"2023-05-22T15:26:49.622122+0000","last_active":"2023-05-22T15:26:53.621541+0000","last_peered":"2023-05-22T15:26:53.621541+0000","last_clean":"2023-05-22T15:26:53.621541+0000","last_became_active":"2023-05-22T15:26:49.621591+0000","last_became_peered":"2023-05-22T15:26:49.621591+0000","last_unstale":"2023-05-22T15:26:53.621541+0000","last_undegraded":"2023-05-22T15:26:53.621541+0000","last_fullsized":"2023-05-22T15:26:53.621541+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T18:05:09.939239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.617051+0000","last_change":"2023-05-22T15:26:49.200650+0000","last_active":"2023-05-22T15:26:53.617051+0000","last_peered":"2023-05-22T15:26:53.617051+0000","last_clean":"2023-05-22T15:26:53.617051+0000","last_became_active":"2023-05-22T15:26:49.199682+0000","last_became_peered":"2023-05-22T15:26:49.199682+0000","last_unstale":"2023-05-22T15:26:53.617051+0000","last_undegraded":"2023-05-22T15:26:53.617051+0000","last_fullsized":"2023-05-22T15:26:53.617051+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_clean_scrub_stamp":"2023-05-22T15:26:48.127413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:13:03.975127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-05-22T15:26:54.056477+0000","last_change":"2023-05-22T15:26:53.667847+0000","last_active":"2023-05-22T15:26:54.056477+0000","last_peered":"2023-05-22T15:26:54.056477+0000","last_clean":"2023-05-22T15:26:54.056477+0000","last_became_active":"2023-05-22T15:26:53.666866+0000","last_became_peered":"2023-05-22T15:26:53.666866+0000","last_unstale":"2023-05-22T15:26:54.056477+0000","last_undegraded":"2023-05-22T15:26:54.056477+0000","last_fullsized":"2023-05-22T15:26:54.056477+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_clean_scrub_stamp":"2023-05-22T15:26:52.568670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:30:40.980509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6,2],"acting":[5,4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.631889+0000","last_change":"2023-05-22T15:26:49.598177+0000","last_active":"2023-05-22T15:26:53.631889+0000","last_peered":"2023-05-22T15:26:53.631889+0000","last_clean":"2023-05-22T15:26:53.631889+0000","last_became_active":"2023-05-22T15:26:49.594848+0000","last_became_peered":"2023-05-22T15:26:49.594848+0000","last_unstale":"2023-05-22T15:26:53.631889+0000","last_undegraded":"2023-05-22T15:26:53.631889+0000","last_fullsized":"2023-05-22T15:26:53.631889+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T03:14:09.303640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.603009+0000","last_change":"2023-05-22T15:26:49.206461+0000","last_active":"2023-05-22T15:26:53.603009+0000","last_peered":"2023-05-22T15:26:53.603009+0000","last_clean":"2023-05-22T15:26:53.603009+0000","last_became_active":"2023-05-22T15:26:49.205870+0000","last_became_peered":"2023-05-22T15:26:49.205870+0000","last_unstale":"2023-05-22T15:26:53.603009+0000","last_undegraded":"2023-05-22T15:26:53.603009+0000","last_fullsized":"2023-05-22T15:26:53.603009+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_clean_scrub_stamp":"2023-05-22T15:26:48.127413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T18:13:58.291903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.980043+0000","last_change":"2023-05-22T15:26:53.642805+0000","last_active":"2023-05-22T15:26:53.980043+0000","last_peered":"2023-05-22T15:26:53.980043+0000","last_clean":"2023-05-22T15:26:53.980043+0000","last_became_active":"2023-05-22T15:26:53.642142+0000","last_became_peered":"2023-05-22T15:26:53.642142+0000","last_unstale":"2023-05-22T15:26:53.980043+0000","last_undegraded":"2023-05-22T15:26:53.980043+0000","last_fullsized":"2023-05-22T15:26:53.980043+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_clean_scrub_stamp":"2023-05-22T15:26:52.568670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:12:09.126657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0,6],"acting":[1,4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.623794+0000","last_change":"2023-05-22T15:26:49.600035+0000","last_active":"2023-05-22T15:26:53.623794+0000","last_peered":"2023-05-22T15:26:53.623794+0000","last_clean":"2023-05-22T15:26:53.623794+0000","last_became_active":"2023-05-22T15:26:49.597436+0000","last_became_peered":"2023-05-22T15:26:49.597436+0000","last_unstale":"2023-05-22T15:26:53.623794+0000","last_undegraded":"2023-05-22T15:26:53.623794+0000","last_fullsized":"2023-05-22T15:26:53.623794+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:46:50.307602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"22'2","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.606082+0000","last_change":"2023-05-22T15:26:49.191345+0000","last_active":"2023-05-22T15:26:53.606082+0000","last_peered":"2023-05-22T15:26:53.606082+0000","last_clean":"2023-05-22T15:26:53.606082+0000","last_became_active":"2023-05-22T15:26:49.190291+0000","last_became_peered":"2023-05-22T15:26:49.190291+0000","last_unstale":"2023-05-22T15:26:53.606082+0000","last_undegraded":"2023-05-22T15:26:53.606082+0000","last_fullsized":"2023-05-22T15:26:53.606082+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_clean_scrub_stamp":"2023-05-22T15:26:48.127413+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:13:21.177657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.979168+0000","last_change":"2023-05-22T15:26:53.649119+0000","last_active":"2023-05-22T15:26:53.979168+0000","last_peered":"2023-05-22T15:26:53.979168+0000","last_clean":"2023-05-22T15:26:53.979168+0000","last_became_active":"2023-05-22T15:26:53.648706+0000","last_became_peered":"2023-05-22T15:26:53.648706+0000","last_unstale":"2023-05-22T15:26:53.979168+0000","last_undegraded":"2023-05-22T15:26:53.979168+0000","last_fullsized":"2023-05-22T15:26:53.979168+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_clean_scrub_stamp":"2023-05-22T15:26:52.568670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T18:05:13.369385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5,6],"acting":[2,7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.611714+0000","last_change":"2023-05-22T15:26:49.581996+0000","last_active":"2023-05-22T15:26:53.611714+0000","last_peered":"2023-05-22T15:26:53.611714+0000","last_clean":"2023-05-22T15:26:53.611714+0000","last_became_active":"2023-05-22T15:26:49.581611+0000","last_became_peered":"2023-05-22T15:26:49.581611+0000","last_unstale":"2023-05-22T15:26:53.611714+0000","last_undegraded":"2023-05-22T15:26:53.611714+0000","last_fullsized":"2023-05-22T15:26:53.611714+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:44:52.369716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.634393+0000","last_change":"2023-05-22T15:26:49.192121+0000","last_active":"2023-05-22T15:26:53.634393+0000","last_peered":"2023-05-22T15:26:53.634393+0000","last_clean":"2023-05-22T15:26:53.634393+0000","last_became_active":"2023-05-22T15:26:49.191470+0000","last_became_peered":"2023-05-22T15:26:49.191470+0000","last_unstale":"2023-05-22T15:26:53.634393+0000","last_undegraded":"2023-05-22T15:26:53.634393+0000","last_fullsized":"2023-05-22T15:26:53.634393+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_clean_scrub_stamp":"2023-05-22T15:26:48.127413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:53:34.802286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.979064+0000","last_change":"2023-05-22T15:26:53.656554+0000","last_active":"2023-05-22T15:26:53.979064+0000","last_peered":"2023-05-22T15:26:53.979064+0000","last_clean":"2023-05-22T15:26:53.979064+0000","last_became_active":"2023-05-22T15:26:53.655950+0000","last_became_peered":"2023-05-22T15:26:53.655950+0000","last_unstale":"2023-05-22T15:26:53.979064+0000","last_undegraded":"2023-05-22T15:26:53.979064+0000","last_fullsized":"2023-05-22T15:26:53.979064+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_clean_scrub_stamp":"2023-05-22T15:26:52.568670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:11:45.590112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3,6],"acting":[2,4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.602652+0000","last_change":"2023-05-22T15:26:49.618498+0000","last_active":"2023-05-22T15:26:53.602652+0000","last_peered":"2023-05-22T15:26:53.602652+0000","last_clean":"2023-05-22T15:26:53.602652+0000","last_became_active":"2023-05-22T15:26:49.618068+0000","last_became_peered":"2023-05-22T15:26:49.618068+0000","last_unstale":"2023-05-22T15:26:53.602652+0000","last_undegraded":"2023-05-22T15:26:53.602652+0000","last_fullsized":"2023-05-22T15:26:53.602652+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:55:20.098659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"22'1","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.610411+0000","last_change":"2023-05-22T15:26:49.186875+0000","last_active":"2023-05-22T15:26:53.610411+0000","last_peered":"2023-05-22T15:26:53.610411+0000","last_clean":"2023-05-22T15:26:53.610411+0000","last_became_active":"2023-05-22T15:26:49.186431+0000","last_became_peered":"2023-05-22T15:26:49.186431+0000","last_unstale":"2023-05-22T15:26:53.610411+0000","last_undegraded":"2023-05-22T15:26:53.610411+0000","last_fullsized":"2023-05-22T15:26:53.610411+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_clean_scrub_stamp":"2023-05-22T15:26:48.127413+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:59:22.794782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.978460+0000","last_change":"2023-05-22T15:26:53.649077+0000","last_active":"2023-05-22T15:26:53.978460+0000","last_peered":"2023-05-22T15:26:53.978460+0000","last_clean":"2023-05-22T15:26:53.978460+0000","last_became_active":"2023-05-22T15:26:53.648415+0000","last_became_peered":"2023-05-22T15:26:53.648415+0000","last_unstale":"2023-05-22T15:26:53.978460+0000","last_undegraded":"2023-05-22T15:26:53.978460+0000","last_fullsized":"2023-05-22T15:26:53.978460+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_clean_scrub_stamp":"2023-05-22T15:26:52.568670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:33:26.787992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5,6],"acting":[2,0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.616451+0000","last_change":"2023-05-22T15:26:49.597065+0000","last_active":"2023-05-22T15:26:53.616451+0000","last_peered":"2023-05-22T15:26:53.616451+0000","last_clean":"2023-05-22T15:26:53.616451+0000","last_became_active":"2023-05-22T15:26:49.596334+0000","last_became_peered":"2023-05-22T15:26:49.596334+0000","last_unstale":"2023-05-22T15:26:53.616451+0000","last_undegraded":"2023-05-22T15:26:53.616451+0000","last_fullsized":"2023-05-22T15:26:53.616451+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:19:30.391062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"22'2","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.610463+0000","last_change":"2023-05-22T15:26:49.201296+0000","last_active":"2023-05-22T15:26:53.610463+0000","last_peered":"2023-05-22T15:26:53.610463+0000","last_clean":"2023-05-22T15:26:53.610463+0000","last_became_active":"2023-05-22T15:26:49.200707+0000","last_became_peered":"2023-05-22T15:26:49.200707+0000","last_unstale":"2023-05-22T15:26:53.610463+0000","last_undegraded":"2023-05-22T15:26:53.610463+0000","last_fullsized":"2023-05-22T15:26:53.610463+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_clean_scrub_stamp":"2023-05-22T15:26:48.127413+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:07:41.791660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.977970+0000","last_change":"2023-05-22T15:26:53.622351+0000","last_active":"2023-05-22T15:26:53.977970+0000","last_peered":"2023-05-22T15:26:53.977970+0000","last_clean":"2023-05-22T15:26:53.977970+0000","last_became_active":"2023-05-22T15:26:53.621754+0000","last_became_peered":"2023-05-22T15:26:53.621754+0000","last_unstale":"2023-05-22T15:26:53.977970+0000","last_undegraded":"2023-05-22T15:26:53.977970+0000","last_fullsized":"2023-05-22T15:26:53.977970+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_clean_scrub_stamp":"2023-05-22T15:26:52.568670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T16:31:40.104597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1,5],"acting":[2,0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.634315+0000","last_change":"2023-05-22T15:26:49.626518+0000","last_active":"2023-05-22T15:26:53.634315+0000","last_peered":"2023-05-22T15:26:53.634315+0000","last_clean":"2023-05-22T15:26:53.634315+0000","last_became_active":"2023-05-22T15:26:49.626026+0000","last_became_peered":"2023-05-22T15:26:49.626026+0000","last_unstale":"2023-05-22T15:26:53.634315+0000","last_undegraded":"2023-05-22T15:26:53.634315+0000","last_fullsized":"2023-05-22T15:26:53.634315+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:32:19.582542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"22'1","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.622234+0000","last_change":"2023-05-22T15:26:49.201252+0000","last_active":"2023-05-22T15:26:53.622234+0000","last_peered":"2023-05-22T15:26:53.622234+0000","last_clean":"2023-05-22T15:26:53.622234+0000","last_became_active":"2023-05-22T15:26:49.200748+0000","last_became_peered":"2023-05-22T15:26:49.200748+0000","last_unstale":"2023-05-22T15:26:53.622234+0000","last_undegraded":"2023-05-22T15:26:53.622234+0000","last_fullsized":"2023-05-22T15:26:53.622234+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_clean_scrub_stamp":"2023-05-22T15:26:48.127413+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:10:50.424396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.982126+0000","last_change":"2023-05-22T15:26:53.663379+0000","last_active":"2023-05-22T15:26:53.982126+0000","last_peered":"2023-05-22T15:26:53.982126+0000","last_clean":"2023-05-22T15:26:53.982126+0000","last_became_active":"2023-05-22T15:26:53.662170+0000","last_became_peered":"2023-05-22T15:26:53.662170+0000","last_unstale":"2023-05-22T15:26:53.982126+0000","last_undegraded":"2023-05-22T15:26:53.982126+0000","last_fullsized":"2023-05-22T15:26:53.982126+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_clean_scrub_stamp":"2023-05-22T15:26:52.568670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:38:53.647917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3,4],"acting":[7,6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-05-22T15:26:54.056358+0000","last_change":"2023-05-22T15:26:53.647770+0000","last_active":"2023-05-22T15:26:54.056358+0000","last_peered":"2023-05-22T15:26:54.056358+0000","last_clean":"2023-05-22T15:26:54.056358+0000","last_became_active":"2023-05-22T15:26:53.646585+0000","last_became_peered":"2023-05-22T15:26:53.646585+0000","last_unstale":"2023-05-22T15:26:54.056358+0000","last_undegraded":"2023-05-22T15:26:54.056358+0000","last_fullsized":"2023-05-22T15:26:54.056358+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_clean_scrub_stamp":"2023-05-22T15:26:52.568670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:48:44.211681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1,6],"acting":[5,3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"22'8","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.602514+0000","last_change":"2023-05-22T15:26:49.189894+0000","last_active":"2023-05-22T15:26:53.602514+0000","last_peered":"2023-05-22T15:26:53.602514+0000","last_clean":"2023-05-22T15:26:53.602514+0000","last_became_active":"2023-05-22T15:26:49.189156+0000","last_became_peered":"2023-05-22T15:26:49.189156+0000","last_unstale":"2023-05-22T15:26:53.602514+0000","last_undegraded":"2023-05-22T15:26:53.602514+0000","last_fullsized":"2023-05-22T15:26:53.602514+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_clean_scrub_stamp":"2023-05-22T15:26:48.127413+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:13:21.177657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.605587+0000","last_change":"2023-05-22T15:26:49.622540+0000","last_active":"2023-05-22T15:26:53.605587+0000","last_peered":"2023-05-22T15:26:53.605587+0000","last_clean":"2023-05-22T15:26:53.605587+0000","last_became_active":"2023-05-22T15:26:49.622187+0000","last_became_peered":"2023-05-22T15:26:49.622187+0000","last_unstale":"2023-05-22T15:26:53.605587+0000","last_undegraded":"2023-05-22T15:26:53.605587+0000","last_fullsized":"2023-05-22T15:26:53.605587+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T16:31:36.674451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.993571+0000","last_change":"2023-05-22T15:26:53.645178+0000","last_active":"2023-05-22T15:26:53.993571+0000","last_peered":"2023-05-22T15:26:53.993571+0000","last_clean":"2023-05-22T15:26:53.993571+0000","last_became_active":"2023-05-22T15:26:53.644770+0000","last_became_peered":"2023-05-22T15:26:53.644770+0000","last_unstale":"2023-05-22T15:26:53.993571+0000","last_undegraded":"2023-05-22T15:26:53.993571+0000","last_fullsized":"2023-05-22T15:26:53.993571+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_clean_scrub_stamp":"2023-05-22T15:26:52.568670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T15:46:20.544673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,0],"acting":[3,1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.628444+0000","last_change":"2023-05-22T15:26:49.191146+0000","last_active":"2023-05-22T15:26:53.628444+0000","last_peered":"2023-05-22T15:26:53.628444+0000","last_clean":"2023-05-22T15:26:53.628444+0000","last_became_active":"2023-05-22T15:26:49.190714+0000","last_became_peered":"2023-05-22T15:26:49.190714+0000","last_unstale":"2023-05-22T15:26:53.628444+0000","last_undegraded":"2023-05-22T15:26:53.628444+0000","last_fullsized":"2023-05-22T15:26:53.628444+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_clean_scrub_stamp":"2023-05-22T15:26:48.127413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:55:19.087548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.612090+0000","last_change":"2023-05-22T15:26:49.585916+0000","last_active":"2023-05-22T15:26:53.612090+0000","last_peered":"2023-05-22T15:26:53.612090+0000","last_clean":"2023-05-22T15:26:53.612090+0000","last_became_active":"2023-05-22T15:26:49.585261+0000","last_became_peered":"2023-05-22T15:26:49.585261+0000","last_unstale":"2023-05-22T15:26:53.612090+0000","last_undegraded":"2023-05-22T15:26:53.612090+0000","last_fullsized":"2023-05-22T15:26:53.612090+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:49:00.232177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.611499+0000","last_change":"2023-05-22T15:26:49.179340+0000","last_active":"2023-05-22T15:26:53.611499+0000","last_peered":"2023-05-22T15:26:53.611499+0000","last_clean":"2023-05-22T15:26:53.611499+0000","last_became_active":"2023-05-22T15:26:49.178875+0000","last_became_peered":"2023-05-22T15:26:49.178875+0000","last_unstale":"2023-05-22T15:26:53.611499+0000","last_undegraded":"2023-05-22T15:26:53.611499+0000","last_fullsized":"2023-05-22T15:26:53.611499+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_clean_scrub_stamp":"2023-05-22T15:26:48.127413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T16:16:39.277093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.634708+0000","last_change":"2023-05-22T15:26:49.623344+0000","last_active":"2023-05-22T15:26:53.634708+0000","last_peered":"2023-05-22T15:26:53.634708+0000","last_clean":"2023-05-22T15:26:53.634708+0000","last_became_active":"2023-05-22T15:26:49.622700+0000","last_became_peered":"2023-05-22T15:26:49.622700+0000","last_unstale":"2023-05-22T15:26:53.634708+0000","last_undegraded":"2023-05-22T15:26:53.634708+0000","last_fullsized":"2023-05-22T15:26:53.634708+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:22:10.117410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.993134+0000","last_change":"2023-05-22T15:26:53.646185+0000","last_active":"2023-05-22T15:26:53.993134+0000","last_peered":"2023-05-22T15:26:53.993134+0000","last_clean":"2023-05-22T15:26:53.993134+0000","last_became_active":"2023-05-22T15:26:53.645789+0000","last_became_peered":"2023-05-22T15:26:53.645789+0000","last_unstale":"2023-05-22T15:26:53.993134+0000","last_undegraded":"2023-05-22T15:26:53.993134+0000","last_fullsized":"2023-05-22T15:26:53.993134+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_clean_scrub_stamp":"2023-05-22T15:26:52.568670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:30:40.980509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2,1],"acting":[3,7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"22'2","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.608324+0000","last_change":"2023-05-22T15:26:49.201586+0000","last_active":"2023-05-22T15:26:53.608324+0000","last_peered":"2023-05-22T15:26:53.608324+0000","last_clean":"2023-05-22T15:26:53.608324+0000","last_became_active":"2023-05-22T15:26:49.201013+0000","last_became_peered":"2023-05-22T15:26:49.201013+0000","last_unstale":"2023-05-22T15:26:53.608324+0000","last_undegraded":"2023-05-22T15:26:53.608324+0000","last_fullsized":"2023-05-22T15:26:53.608324+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_clean_scrub_stamp":"2023-05-22T15:26:48.127413+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:47:17.588354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.607130+0000","last_change":"2023-05-22T15:26:49.577112+0000","last_active":"2023-05-22T15:26:53.607130+0000","last_peered":"2023-05-22T15:26:53.607130+0000","last_clean":"2023-05-22T15:26:53.607130+0000","last_became_active":"2023-05-22T15:26:49.576802+0000","last_became_peered":"2023-05-22T15:26:49.576802+0000","last_unstale":"2023-05-22T15:26:53.607130+0000","last_undegraded":"2023-05-22T15:26:53.607130+0000","last_fullsized":"2023-05-22T15:26:53.607130+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:52:48.123148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-05-22T15:26:54.056338+0000","last_change":"2023-05-22T15:26:53.655225+0000","last_active":"2023-05-22T15:26:54.056338+0000","last_peered":"2023-05-22T15:26:54.056338+0000","last_clean":"2023-05-22T15:26:54.056338+0000","last_became_active":"2023-05-22T15:26:53.648610+0000","last_became_peered":"2023-05-22T15:26:53.648610+0000","last_unstale":"2023-05-22T15:26:54.056338+0000","last_undegraded":"2023-05-22T15:26:54.056338+0000","last_fullsized":"2023-05-22T15:26:54.056338+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_clean_scrub_stamp":"2023-05-22T15:26:52.568670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T15:46:20.544673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3,0],"acting":[5,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.20","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.602276+0000","last_change":"2023-05-22T15:26:49.614417+0000","last_active":"2023-05-22T15:26:53.602276+0000","last_peered":"2023-05-22T15:26:53.602276+0000","last_clean":"2023-05-22T15:26:53.602276+0000","last_became_active":"2023-05-22T15:26:49.610439+0000","last_became_peered":"2023-05-22T15:26:49.610439+0000","last_unstale":"2023-05-22T15:26:53.602276+0000","last_undegraded":"2023-05-22T15:26:53.602276+0000","last_fullsized":"2023-05-22T15:26:53.602276+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:07:42.802772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.21","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.603531+0000","last_change":"2023-05-22T15:26:49.577132+0000","last_active":"2023-05-22T15:26:53.603531+0000","last_peered":"2023-05-22T15:26:53.603531+0000","last_clean":"2023-05-22T15:26:53.603531+0000","last_became_active":"2023-05-22T15:26:49.576859+0000","last_became_peered":"2023-05-22T15:26:49.576859+0000","last_unstale":"2023-05-22T15:26:53.603531+0000","last_undegraded":"2023-05-22T15:26:53.603531+0000","last_fullsized":"2023-05-22T15:26:53.603531+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:08:07.637726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.22","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.619920+0000","last_change":"2023-05-22T15:26:49.622334+0000","last_active":"2023-05-22T15:26:53.619920+0000","last_peered":"2023-05-22T15:26:53.619920+0000","last_clean":"2023-05-22T15:26:53.619920+0000","last_became_active":"2023-05-22T15:26:49.621648+0000","last_became_peered":"2023-05-22T15:26:49.621648+0000","last_unstale":"2023-05-22T15:26:53.619920+0000","last_undegraded":"2023-05-22T15:26:53.619920+0000","last_fullsized":"2023-05-22T15:26:53.619920+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:57:58.007680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.23","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.605756+0000","last_change":"2023-05-22T15:26:49.612753+0000","last_active":"2023-05-22T15:26:53.605756+0000","last_peered":"2023-05-22T15:26:53.605756+0000","last_clean":"2023-05-22T15:26:53.605756+0000","last_became_active":"2023-05-22T15:26:49.612495+0000","last_became_peered":"2023-05-22T15:26:49.612495+0000","last_unstale":"2023-05-22T15:26:53.605756+0000","last_undegraded":"2023-05-22T15:26:53.605756+0000","last_fullsized":"2023-05-22T15:26:53.605756+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:11:51.327926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.24","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.619231+0000","last_change":"2023-05-22T15:26:49.627768+0000","last_active":"2023-05-22T15:26:53.619231+0000","last_peered":"2023-05-22T15:26:53.619231+0000","last_clean":"2023-05-22T15:26:53.619231+0000","last_became_active":"2023-05-22T15:26:49.627340+0000","last_became_peered":"2023-05-22T15:26:49.627340+0000","last_unstale":"2023-05-22T15:26:53.619231+0000","last_undegraded":"2023-05-22T15:26:53.619231+0000","last_fullsized":"2023-05-22T15:26:53.619231+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:44:52.369716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.25","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.634508+0000","last_change":"2023-05-22T15:26:49.613627+0000","last_active":"2023-05-22T15:26:53.634508+0000","last_peered":"2023-05-22T15:26:53.634508+0000","last_clean":"2023-05-22T15:26:53.634508+0000","last_became_active":"2023-05-22T15:26:49.612997+0000","last_became_peered":"2023-05-22T15:26:49.612997+0000","last_unstale":"2023-05-22T15:26:53.634508+0000","last_undegraded":"2023-05-22T15:26:53.634508+0000","last_fullsized":"2023-05-22T15:26:53.634508+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:11:51.327926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.26","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.612444+0000","last_change":"2023-05-22T15:26:49.581772+0000","last_active":"2023-05-22T15:26:53.612444+0000","last_peered":"2023-05-22T15:26:53.612444+0000","last_clean":"2023-05-22T15:26:53.612444+0000","last_became_active":"2023-05-22T15:26:49.581493+0000","last_became_peered":"2023-05-22T15:26:49.581493+0000","last_unstale":"2023-05-22T15:26:53.612444+0000","last_undegraded":"2023-05-22T15:26:53.612444+0000","last_fullsized":"2023-05-22T15:26:53.612444+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:49:00.232177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.27","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.612724+0000","last_change":"2023-05-22T15:26:49.619433+0000","last_active":"2023-05-22T15:26:53.612724+0000","last_peered":"2023-05-22T15:26:53.612724+0000","last_clean":"2023-05-22T15:26:53.612724+0000","last_became_active":"2023-05-22T15:26:49.619107+0000","last_became_peered":"2023-05-22T15:26:49.619107+0000","last_unstale":"2023-05-22T15:26:53.612724+0000","last_undegraded":"2023-05-22T15:26:53.612724+0000","last_fullsized":"2023-05-22T15:26:53.612724+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:11:51.327926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.28","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.609649+0000","last_change":"2023-05-22T15:26:49.624224+0000","last_active":"2023-05-22T15:26:53.609649+0000","last_peered":"2023-05-22T15:26:53.609649+0000","last_clean":"2023-05-22T15:26:53.609649+0000","last_became_active":"2023-05-22T15:26:49.623023+0000","last_became_peered":"2023-05-22T15:26:49.623023+0000","last_unstale":"2023-05-22T15:26:53.609649+0000","last_undegraded":"2023-05-22T15:26:53.609649+0000","last_fullsized":"2023-05-22T15:26:53.609649+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:32:19.582542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.29","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.603353+0000","last_change":"2023-05-22T15:26:49.573154+0000","last_active":"2023-05-22T15:26:53.603353+0000","last_peered":"2023-05-22T15:26:53.603353+0000","last_clean":"2023-05-22T15:26:53.603353+0000","last_became_active":"2023-05-22T15:26:49.572881+0000","last_became_peered":"2023-05-22T15:26:49.572881+0000","last_unstale":"2023-05-22T15:26:53.603353+0000","last_undegraded":"2023-05-22T15:26:53.603353+0000","last_fullsized":"2023-05-22T15:26:53.603353+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:32:19.582542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2a","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.611407+0000","last_change":"2023-05-22T15:26:49.581918+0000","last_active":"2023-05-22T15:26:53.611407+0000","last_peered":"2023-05-22T15:26:53.611407+0000","last_clean":"2023-05-22T15:26:53.611407+0000","last_became_active":"2023-05-22T15:26:49.581462+0000","last_became_peered":"2023-05-22T15:26:49.581462+0000","last_unstale":"2023-05-22T15:26:53.611407+0000","last_undegraded":"2023-05-22T15:26:53.611407+0000","last_fullsized":"2023-05-22T15:26:53.611407+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:19:30.391062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2b","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.608076+0000","last_change":"2023-05-22T15:26:49.624711+0000","last_active":"2023-05-22T15:26:53.608076+0000","last_peered":"2023-05-22T15:26:53.608076+0000","last_clean":"2023-05-22T15:26:53.608076+0000","last_became_active":"2023-05-22T15:26:49.624180+0000","last_became_peered":"2023-05-22T15:26:49.624180+0000","last_unstale":"2023-05-22T15:26:53.608076+0000","last_undegraded":"2023-05-22T15:26:53.608076+0000","last_fullsized":"2023-05-22T15:26:53.608076+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:43:00.518715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2c","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.623768+0000","last_change":"2023-05-22T15:26:49.620689+0000","last_active":"2023-05-22T15:26:53.623768+0000","last_peered":"2023-05-22T15:26:53.623768+0000","last_clean":"2023-05-22T15:26:53.623768+0000","last_became_active":"2023-05-22T15:26:49.620183+0000","last_became_peered":"2023-05-22T15:26:49.620183+0000","last_unstale":"2023-05-22T15:26:53.623768+0000","last_undegraded":"2023-05-22T15:26:53.623768+0000","last_fullsized":"2023-05-22T15:26:53.623768+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:47:22.628767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2d","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.601734+0000","last_change":"2023-05-22T15:26:49.618556+0000","last_active":"2023-05-22T15:26:53.601734+0000","last_peered":"2023-05-22T15:26:53.601734+0000","last_clean":"2023-05-22T15:26:53.601734+0000","last_became_active":"2023-05-22T15:26:49.618215+0000","last_became_peered":"2023-05-22T15:26:49.618215+0000","last_unstale":"2023-05-22T15:26:53.601734+0000","last_undegraded":"2023-05-22T15:26:53.601734+0000","last_fullsized":"2023-05-22T15:26:53.601734+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:07:50.944971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2e","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.621273+0000","last_change":"2023-05-22T15:26:49.597573+0000","last_active":"2023-05-22T15:26:53.621273+0000","last_peered":"2023-05-22T15:26:53.621273+0000","last_clean":"2023-05-22T15:26:53.621273+0000","last_became_active":"2023-05-22T15:26:49.596607+0000","last_became_peered":"2023-05-22T15:26:49.596607+0000","last_unstale":"2023-05-22T15:26:53.621273+0000","last_undegraded":"2023-05-22T15:26:53.621273+0000","last_fullsized":"2023-05-22T15:26:53.621273+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:31:03.890085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2f","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.611892+0000","last_change":"2023-05-22T15:26:49.578237+0000","last_active":"2023-05-22T15:26:53.611892+0000","last_peered":"2023-05-22T15:26:53.611892+0000","last_clean":"2023-05-22T15:26:53.611892+0000","last_became_active":"2023-05-22T15:26:49.577927+0000","last_became_peered":"2023-05-22T15:26:49.577927+0000","last_unstale":"2023-05-22T15:26:53.611892+0000","last_undegraded":"2023-05-22T15:26:53.611892+0000","last_fullsized":"2023-05-22T15:26:53.611892+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:11:51.327926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.30","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.612647+0000","last_change":"2023-05-22T15:26:49.598449+0000","last_active":"2023-05-22T15:26:53.612647+0000","last_peered":"2023-05-22T15:26:53.612647+0000","last_clean":"2023-05-22T15:26:53.612647+0000","last_became_active":"2023-05-22T15:26:49.597652+0000","last_became_peered":"2023-05-22T15:26:49.597652+0000","last_unstale":"2023-05-22T15:26:53.612647+0000","last_undegraded":"2023-05-22T15:26:53.612647+0000","last_fullsized":"2023-05-22T15:26:53.612647+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:48:33.611833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.31","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.628230+0000","last_change":"2023-05-22T15:26:49.625943+0000","last_active":"2023-05-22T15:26:53.628230+0000","last_peered":"2023-05-22T15:26:53.628230+0000","last_clean":"2023-05-22T15:26:53.628230+0000","last_became_active":"2023-05-22T15:26:49.624306+0000","last_became_peered":"2023-05-22T15:26:49.624306+0000","last_unstale":"2023-05-22T15:26:53.628230+0000","last_undegraded":"2023-05-22T15:26:53.628230+0000","last_fullsized":"2023-05-22T15:26:53.628230+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:44:52.369716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.32","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.634913+0000","last_change":"2023-05-22T15:26:49.563010+0000","last_active":"2023-05-22T15:26:53.634913+0000","last_peered":"2023-05-22T15:26:53.634913+0000","last_clean":"2023-05-22T15:26:53.634913+0000","last_became_active":"2023-05-22T15:26:49.562210+0000","last_became_peered":"2023-05-22T15:26:49.562210+0000","last_unstale":"2023-05-22T15:26:53.634913+0000","last_undegraded":"2023-05-22T15:26:53.634913+0000","last_fullsized":"2023-05-22T15:26:53.634913+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:19:30.391062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.33","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.600262+0000","last_change":"2023-05-22T15:26:49.562260+0000","last_active":"2023-05-22T15:26:53.600262+0000","last_peered":"2023-05-22T15:26:53.600262+0000","last_clean":"2023-05-22T15:26:53.600262+0000","last_became_active":"2023-05-22T15:26:49.561965+0000","last_became_peered":"2023-05-22T15:26:49.561965+0000","last_unstale":"2023-05-22T15:26:53.600262+0000","last_undegraded":"2023-05-22T15:26:53.600262+0000","last_fullsized":"2023-05-22T15:26:53.600262+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:59:55.020950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.34","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.598383+0000","last_change":"2023-05-22T15:26:49.601428+0000","last_active":"2023-05-22T15:26:53.598383+0000","last_peered":"2023-05-22T15:26:53.598383+0000","last_clean":"2023-05-22T15:26:53.598383+0000","last_became_active":"2023-05-22T15:26:49.601157+0000","last_became_peered":"2023-05-22T15:26:49.601157+0000","last_unstale":"2023-05-22T15:26:53.598383+0000","last_undegraded":"2023-05-22T15:26:53.598383+0000","last_fullsized":"2023-05-22T15:26:53.598383+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:47:18.599465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.35","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.631671+0000","last_change":"2023-05-22T15:26:49.598414+0000","last_active":"2023-05-22T15:26:53.631671+0000","last_peered":"2023-05-22T15:26:53.631671+0000","last_clean":"2023-05-22T15:26:53.631671+0000","last_became_active":"2023-05-22T15:26:49.595631+0000","last_became_peered":"2023-05-22T15:26:49.595631+0000","last_unstale":"2023-05-22T15:26:53.631671+0000","last_undegraded":"2023-05-22T15:26:53.631671+0000","last_fullsized":"2023-05-22T15:26:53.631671+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:11:51.327926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.36","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.635151+0000","last_change":"2023-05-22T15:26:49.580957+0000","last_active":"2023-05-22T15:26:53.635151+0000","last_peered":"2023-05-22T15:26:53.635151+0000","last_clean":"2023-05-22T15:26:53.635151+0000","last_became_active":"2023-05-22T15:26:49.580172+0000","last_became_peered":"2023-05-22T15:26:49.580172+0000","last_unstale":"2023-05-22T15:26:53.635151+0000","last_undegraded":"2023-05-22T15:26:53.635151+0000","last_fullsized":"2023-05-22T15:26:53.635151+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:43:00.518715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.37","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.615519+0000","last_change":"2023-05-22T15:26:49.587437+0000","last_active":"2023-05-22T15:26:53.615519+0000","last_peered":"2023-05-22T15:26:53.615519+0000","last_clean":"2023-05-22T15:26:53.615519+0000","last_became_active":"2023-05-22T15:26:49.587019+0000","last_became_peered":"2023-05-22T15:26:49.587019+0000","last_unstale":"2023-05-22T15:26:53.615519+0000","last_undegraded":"2023-05-22T15:26:53.615519+0000","last_fullsized":"2023-05-22T15:26:53.615519+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:48:33.611833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.38","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.634076+0000","last_change":"2023-05-22T15:26:49.599720+0000","last_active":"2023-05-22T15:26:53.634076+0000","last_peered":"2023-05-22T15:26:53.634076+0000","last_clean":"2023-05-22T15:26:53.634076+0000","last_became_active":"2023-05-22T15:26:49.599079+0000","last_became_peered":"2023-05-22T15:26:49.599079+0000","last_unstale":"2023-05-22T15:26:53.634076+0000","last_undegraded":"2023-05-22T15:26:53.634076+0000","last_fullsized":"2023-05-22T15:26:53.634076+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:48:33.611833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.39","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.628016+0000","last_change":"2023-05-22T15:26:49.624934+0000","last_active":"2023-05-22T15:26:53.628016+0000","last_peered":"2023-05-22T15:26:53.628016+0000","last_clean":"2023-05-22T15:26:53.628016+0000","last_became_active":"2023-05-22T15:26:49.623557+0000","last_became_peered":"2023-05-22T15:26:49.623557+0000","last_unstale":"2023-05-22T15:26:53.628016+0000","last_undegraded":"2023-05-22T15:26:53.628016+0000","last_fullsized":"2023-05-22T15:26:53.628016+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:19:30.391062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3a","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.619318+0000","last_change":"2023-05-22T15:26:49.596704+0000","last_active":"2023-05-22T15:26:53.619318+0000","last_peered":"2023-05-22T15:26:53.619318+0000","last_clean":"2023-05-22T15:26:53.619318+0000","last_became_active":"2023-05-22T15:26:49.595941+0000","last_became_peered":"2023-05-22T15:26:49.595941+0000","last_unstale":"2023-05-22T15:26:53.619318+0000","last_undegraded":"2023-05-22T15:26:53.619318+0000","last_fullsized":"2023-05-22T15:26:53.619318+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T03:17:59.774769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3b","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.611996+0000","last_change":"2023-05-22T15:26:49.585396+0000","last_active":"2023-05-22T15:26:53.611996+0000","last_peered":"2023-05-22T15:26:53.611996+0000","last_clean":"2023-05-22T15:26:53.611996+0000","last_became_active":"2023-05-22T15:26:49.585110+0000","last_became_peered":"2023-05-22T15:26:49.585110+0000","last_unstale":"2023-05-22T15:26:53.611996+0000","last_undegraded":"2023-05-22T15:26:53.611996+0000","last_fullsized":"2023-05-22T15:26:53.611996+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:22:10.117410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3c","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.602523+0000","last_change":"2023-05-22T15:26:49.607125+0000","last_active":"2023-05-22T15:26:53.602523+0000","last_peered":"2023-05-22T15:26:53.602523+0000","last_clean":"2023-05-22T15:26:53.602523+0000","last_became_active":"2023-05-22T15:26:49.606510+0000","last_became_peered":"2023-05-22T15:26:49.606510+0000","last_unstale":"2023-05-22T15:26:53.602523+0000","last_undegraded":"2023-05-22T15:26:53.602523+0000","last_fullsized":"2023-05-22T15:26:53.602523+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:47:18.599465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3d","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.631452+0000","last_change":"2023-05-22T15:26:49.600011+0000","last_active":"2023-05-22T15:26:53.631452+0000","last_peered":"2023-05-22T15:26:53.631452+0000","last_clean":"2023-05-22T15:26:53.631452+0000","last_became_active":"2023-05-22T15:26:49.596414+0000","last_became_peered":"2023-05-22T15:26:49.596414+0000","last_unstale":"2023-05-22T15:26:53.631452+0000","last_undegraded":"2023-05-22T15:26:53.631452+0000","last_fullsized":"2023-05-22T15:26:53.631452+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T16:31:36.674451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3e","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.634607+0000","last_change":"2023-05-22T15:26:49.580509+0000","last_active":"2023-05-22T15:26:53.634607+0000","last_peered":"2023-05-22T15:26:53.634607+0000","last_clean":"2023-05-22T15:26:53.634607+0000","last_became_active":"2023-05-22T15:26:49.579662+0000","last_became_peered":"2023-05-22T15:26:49.579662+0000","last_unstale":"2023-05-22T15:26:53.634607+0000","last_undegraded":"2023-05-22T15:26:53.634607+0000","last_fullsized":"2023-05-22T15:26:53.634607+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:44:52.369716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3f","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.599109+0000","last_change":"2023-05-22T15:26:49.561088+0000","last_active":"2023-05-22T15:26:53.599109+0000","last_peered":"2023-05-22T15:26:53.599109+0000","last_clean":"2023-05-22T15:26:53.599109+0000","last_became_active":"2023-05-22T15:26:49.560859+0000","last_became_peered":"2023-05-22T15:26:49.560859+0000","last_unstale":"2023-05-22T15:26:53.599109+0000","last_undegraded":"2023-05-22T15:26:53.599109+0000","last_fullsized":"2023-05-22T15:26:53.599109+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:53:35.813397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":4527,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":9054,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574857,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9284,"kb_used_data":892,"kb_used_omap":0,"kb_used_meta":8384,"kb_avail":94362556,"statfs":{"total":96636764160,"available":96627257344,"internally_reserved":0,"allocated":913408,"data_stored":714230,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8585216},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1399999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88400000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72399999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77000000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98999999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2669999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.21}]}]},{"osd":6,"up_from":13,"seq":55834574858,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8100,"kb_used_data":284,"kb_used_omap":0,"kb_used_meta":7808,"kb_avail":94363740,"statfs":{"total":96636764160,"available":96628469760,"internally_reserved":0,"allocated":290816,"data_stored":119901,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7995392},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.874}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58899999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68799999999999994}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63900000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1659999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95099999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.196}]}]},{"osd":5,"up_from":13,"seq":55834574858,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8172,"kb_used_data":292,"kb_used_omap":0,"kb_used_meta":7872,"kb_avail":94363668,"statfs":{"total":96636764160,"available":96628396032,"internally_reserved":0,"allocated":299008,"data_stored":120537,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8060928},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65000000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0309999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58499999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.75600000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1279999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0089999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97899999999999998}]}]},{"osd":4,"up_from":13,"seq":55834574856,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8248,"kb_used_data":304,"kb_used_omap":0,"kb_used_meta":7936,"kb_avail":94363592,"statfs":{"total":96636764160,"available":96628318208,"internally_reserved":0,"allocated":311296,"data_stored":123702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8126464},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69399999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60399999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82499999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79100000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97599999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0780000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95199999999999996}]}]},{"osd":3,"up_from":13,"seq":55834574856,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8168,"kb_used_data":288,"kb_used_omap":0,"kb_used_meta":7872,"kb_avail":94363672,"statfs":{"total":96636764160,"available":96628400128,"internally_reserved":0,"allocated":294912,"data_stored":119991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8060928},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53400000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46899999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64000000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93100000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.747}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90900000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.133}]}]},{"osd":2,"up_from":13,"seq":55834574857,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":12196,"kb_used_data":284,"kb_used_omap":0,"kb_used_meta":11904,"kb_avail":94359644,"statfs":{"total":96636764160,"available":96624275456,"internally_reserved":0,"allocated":290816,"data_stored":119901,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":12189696},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52300000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72399999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67800000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95199999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.167}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.135}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2430000000000001}]}]},{"osd":1,"up_from":13,"seq":55834574857,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8164,"kb_used_data":284,"kb_used_omap":0,"kb_used_meta":7872,"kb_avail":94363676,"statfs":{"total":96636764160,"available":96628404224,"internally_reserved":0,"allocated":290816,"data_stored":119901,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8060928},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58499999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70299999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53400000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80000000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81899999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97799999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.026}]}]},{"osd":0,"up_from":13,"seq":55834574858,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":13356,"kb_used_data":868,"kb_used_omap":0,"kb_used_meta":12480,"kb_avail":94358484,"statfs":{"total":96636764160,"available":96623087616,"internally_reserved":0,"allocated":888832,"data_stored":710835,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":12779520},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57699999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73099999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.873}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1879999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.129}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0389999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2310000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":3801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":3961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-05-22T15:27:11.353 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-05-22T15:27:11.499 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:11.490+0000 7f618fe5c640 1 Processor -- start 2023-05-22T15:27:11.499 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:11.490+0000 7f618fe5c640 1 -- start start 2023-05-22T15:27:11.500 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:11.490+0000 7f618fe5c640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f61880cb6a0 0x7f6188159340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:11.501 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:11.490+0000 7f618fe5c640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f61880b0250 0x7f6188159880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:11.501 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:11.490+0000 7f618fe5c640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6188156b90 con 0x7f61881567c0 2023-05-22T15:27:11.501 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:11.490+0000 7f618fe5c640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f618815a0b0 con 0x7f61880b0250 2023-05-22T15:27:11.501 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:11.490+0000 7f618fe5c640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61880ca8e0 con 0x7f61880cb6a0 2023-05-22T15:27:11.501 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:11.490+0000 7f618e3f9640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f61880b0250 0x7f6188159880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:11.501 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:11.490+0000 7f618e3f9640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f61880b0250 0x7f6188159880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37832/0 (socket says 172.21.15.16:37832) 2023-05-22T15:27:11.501 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:11.490+0000 7f618e3f9640 1 -- 172.21.15.16:0/2216295623 learned_addr learned my addr 172.21.15.16:0/2216295623 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:11.502 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:11.490+0000 7f618d3f7640 1 --2- 172.21.15.16:0/2216295623 >> v2:172.21.15.62:3300/0 conn(0x7f61880cb6a0 0x7f6188159340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:11.502 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:11.490+0000 7f618e3f9640 1 -- 172.21.15.16:0/2216295623 >> v2:172.21.15.62:3300/0 conn(0x7f61880cb6a0 msgr2=0x7f6188159340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:11.502 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:11.490+0000 7f618e3f9640 1 --2- 172.21.15.16:0/2216295623 >> v2:172.21.15.62:3300/0 conn(0x7f61880cb6a0 0x7f6188159340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:11.502 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:11.490+0000 7f618e3f9640 1 -- 172.21.15.16:0/2216295623 >> v1:172.21.15.16:6789/0 conn(0x7f61881567c0 legacy=0x7f61880afb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:11.503 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:11.490+0000 7f618e3f9640 1 -- 172.21.15.16:0/2216295623 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61880cabc0 con 0x7f61880b0250 2023-05-22T15:27:11.503 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:11.490+0000 7f618d3f7640 1 --2- 172.21.15.16:0/2216295623 >> v2:172.21.15.62:3300/0 conn(0x7f61880cb6a0 0x7f6188159340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T15:27:11.504 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:11.494+0000 7f618e3f9640 1 --2- 172.21.15.16:0/2216295623 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f61880b0250 0x7f6188159880 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f618815d780 tx=0x7f618001c9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c15bc8855c275c45 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:11.504 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:11.494+0000 7f618cbf6640 1 -- 172.21.15.16:0/2216295623 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61800a5670 con 0x7f61880b0250 2023-05-22T15:27:11.504 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:11.494+0000 7f618cbf6640 1 -- 172.21.15.16:0/2216295623 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61800a57d0 con 0x7f61880b0250 2023-05-22T15:27:11.505 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:11.494+0000 7f618cbf6640 1 -- 172.21.15.16:0/2216295623 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61800a5b60 con 0x7f61880b0250 2023-05-22T15:27:11.505 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:11.494+0000 7f618fe5c640 1 -- 172.21.15.16:0/2216295623 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f61880b0250 msgr2=0x7f6188159880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:11.505 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:11.494+0000 7f618fe5c640 1 --2- 172.21.15.16:0/2216295623 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f61880b0250 0x7f6188159880 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f618815d780 tx=0x7f618001c9d0 comp rx=0 tx=0).stop 2023-05-22T15:27:11.505 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:11.494+0000 7f618fe5c640 1 -- 172.21.15.16:0/2216295623 shutdown_connections 2023-05-22T15:27:11.506 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:11.494+0000 7f618fe5c640 1 --2- 172.21.15.16:0/2216295623 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f61880b0250 0x7f6188159880 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:11.506 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:11.494+0000 7f618fe5c640 1 --2- 172.21.15.16:0/2216295623 >> v2:172.21.15.62:3300/0 conn(0x7f61880cb6a0 0x7f6188159340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:11.506 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:11.494+0000 7f618fe5c640 1 -- 172.21.15.16:0/2216295623 >> 172.21.15.16:0/2216295623 conn(0x7f6188069720 msgr2=0x7f6188069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:11.507 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:11.494+0000 7f618fe5c640 1 -- 172.21.15.16:0/2216295623 shutdown_connections 2023-05-22T15:27:11.507 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:11.494+0000 7f618fe5c640 1 -- 172.21.15.16:0/2216295623 wait complete. 2023-05-22T15:27:11.508 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:11.494+0000 7f618fe5c640 1 Processor -- start 2023-05-22T15:27:11.509 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:11.494+0000 7f618fe5c640 1 -- start start 2023-05-22T15:27:11.509 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:11.494+0000 7f618fe5c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f61880b0250 0x7f6188152b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:11.509 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:11.494+0000 7f618fe5c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f61880cb6a0 0x7f61881524f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:11.509 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:11.494+0000 7f618fe5c640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f61881567c0 0x7f61880b2170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:11.509 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:11.494+0000 7f618fe5c640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6188152a30 con 0x7f61880cb6a0 2023-05-22T15:27:11.509 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:11.494+0000 7f618fe5c640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61880b1ce0 con 0x7f61880b0250 2023-05-22T15:27:11.509 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:11.494+0000 7f618fe5c640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61880c1cd0 con 0x7f61881567c0 2023-05-22T15:27:11.510 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:11.498+0000 7f618d3f7640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f61880cb6a0 0x7f61881524f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:11.510 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:11.498+0000 7f618d3f7640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f61880cb6a0 0x7f61881524f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37836/0 (socket says 172.21.15.16:37836) 2023-05-22T15:27:11.510 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:11.498+0000 7f618d3f7640 1 -- 172.21.15.16:0/63438671 learned_addr learned my addr 172.21.15.16:0/63438671 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:11.510 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:11.498+0000 7f618dbf8640 1 --2- 172.21.15.16:0/63438671 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f61880b0250 0x7f6188152b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:11.510 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:11.498+0000 7f618e3f9640 1 --2- 172.21.15.16:0/63438671 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f61881567c0 0x7f61880b2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:11.510 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:11.498+0000 7f618d3f7640 1 -- 172.21.15.16:0/63438671 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f61881567c0 msgr2=0x7f61880b2170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:11.510 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:11.498+0000 7f618d3f7640 1 --2- 172.21.15.16:0/63438671 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f61881567c0 0x7f61880b2170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:11.511 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:11.498+0000 7f618d3f7640 1 -- 172.21.15.16:0/63438671 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f61880b0250 msgr2=0x7f6188152b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:11.511 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:11.498+0000 7f618d3f7640 1 --2- 172.21.15.16:0/63438671 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f61880b0250 0x7f6188152b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:11.511 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:11.498+0000 7f618d3f7640 1 -- 172.21.15.16:0/63438671 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6180009b30 con 0x7f61880cb6a0 2023-05-22T15:27:11.511 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:11.498+0000 7f618dbf8640 1 --2- 172.21.15.16:0/63438671 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f61880b0250 0x7f6188152b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:27:11.511 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:11.498+0000 7f618d3f7640 1 --2- 172.21.15.16:0/63438671 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f61880cb6a0 0x7f61881524f0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f617800e840 tx=0x7f617800ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:11.511 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:11.498+0000 7f617e7fc640 1 -- 172.21.15.16:0/63438671 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f617800c8d0 con 0x7f61880cb6a0 2023-05-22T15:27:11.511 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:11.498+0000 7f617e7fc640 1 -- 172.21.15.16:0/63438671 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f617800ca30 con 0x7f61880cb6a0 2023-05-22T15:27:11.511 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:11.498+0000 7f618fe5c640 1 -- 172.21.15.16:0/63438671 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61880c8e80 con 0x7f61880cb6a0 2023-05-22T15:27:11.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:11.498+0000 7f617e7fc640 1 -- 172.21.15.16:0/63438671 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f617800cde0 con 0x7f61880cb6a0 2023-05-22T15:27:11.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:11.498+0000 7f618fe5c640 1 -- 172.21.15.16:0/63438671 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6188159dc0 con 0x7f61880cb6a0 2023-05-22T15:27:11.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:11.498+0000 7f618fe5c640 1 -- 172.21.15.16:0/63438671 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61880c9010 con 0x7f61880cb6a0 2023-05-22T15:27:11.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:11.498+0000 7f617e7fc640 1 -- 172.21.15.16:0/63438671 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7f6178013410 con 0x7f61880cb6a0 2023-05-22T15:27:11.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:11.498+0000 7f617e7fc640 1 --2- 172.21.15.16:0/63438671 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f616c065380 0x7f616c067830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:11.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:11.498+0000 7f618dbf8640 1 --2- 172.21.15.16:0/63438671 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f616c065380 0x7f616c067830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:11.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:11.498+0000 7f617e7fc640 1 -- 172.21.15.16:0/63438671 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f6178082ae0 con 0x7f61880cb6a0 2023-05-22T15:27:11.513 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:11.498+0000 7f618dbf8640 1 --2- 172.21.15.16:0/63438671 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f616c065380 0x7f616c067830 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f61881553c0 tx=0x7f6184009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:11.513 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:11.502+0000 7f617e7fc640 1 -- 172.21.15.16:0/63438671 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f6178050410 con 0x7f61880cb6a0 2023-05-22T15:27:11.655 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:11.646+0000 7f618fe5c640 1 -- 172.21.15.16:0/63438671 --> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f618804dbe0 con 0x7f616c065380 2023-05-22T15:27:11.667 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:11.658+0000 7f617e7fc640 1 -- 172.21.15.16:0/63438671 <== mgr.4101 v2:172.21.15.16:6832/36921 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+367259 (secure 0 0 0) 0x7f618804dbe0 con 0x7f616c065380 2023-05-22T15:27:11.671 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:27:11.673 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2023-05-22T15:27:11.675 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:11.662+0000 7f618fe5c640 1 -- 172.21.15.16:0/63438671 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f616c065380 msgr2=0x7f616c067830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:11.675 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:11.662+0000 7f618fe5c640 1 --2- 172.21.15.16:0/63438671 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f616c065380 0x7f616c067830 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f61881553c0 tx=0x7f6184009040 comp rx=0 tx=0).stop 2023-05-22T15:27:11.675 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:11.662+0000 7f618fe5c640 1 -- 172.21.15.16:0/63438671 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f61880cb6a0 msgr2=0x7f61881524f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:11.675 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:11.662+0000 7f618fe5c640 1 --2- 172.21.15.16:0/63438671 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f61880cb6a0 0x7f61881524f0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f617800e840 tx=0x7f617800ed00 comp rx=0 tx=0).stop 2023-05-22T15:27:11.675 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:11.662+0000 7f618fe5c640 1 -- 172.21.15.16:0/63438671 shutdown_connections 2023-05-22T15:27:11.676 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:11.662+0000 7f618fe5c640 1 --2- 172.21.15.16:0/63438671 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f616c065380 0x7f616c067830 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:11.676 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:11.662+0000 7f618fe5c640 1 --2- 172.21.15.16:0/63438671 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f61881567c0 0x7f61880b2170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:11.678 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:11.662+0000 7f618fe5c640 1 --2- 172.21.15.16:0/63438671 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f61880cb6a0 0x7f61881524f0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:11.678 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:11.662+0000 7f618fe5c640 1 --2- 172.21.15.16:0/63438671 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f61880b0250 0x7f6188152b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:11.678 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:11.662+0000 7f618fe5c640 1 -- 172.21.15.16:0/63438671 >> 172.21.15.16:0/63438671 conn(0x7f6188069720 msgr2=0x7f6188159010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:11.678 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:11.662+0000 7f618fe5c640 1 -- 172.21.15.16:0/63438671 shutdown_connections 2023-05-22T15:27:11.678 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:11.662+0000 7f618fe5c640 1 -- 172.21.15.16:0/63438671 wait complete. 2023-05-22T15:27:11.687 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":37,"stamp":"2023-05-22T15:27:09.950396+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":594895,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":207,"num_write_kb":2434,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159,"ondisk_log_size":159,"up":338,"acting":338,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":338,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":75688,"kb_used_data":3496,"kb_used_omap":0,"kb_used_meta":72128,"kb_avail":754899032,"statfs":{"total":773094113280,"available":773016608768,"internally_reserved":0,"allocated":3579904,"data_stored":2148998,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":73859072},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2197,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004514"},"pg_stats":[{"pgid":"5.19","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.979395+0000","last_change":"2023-05-22T15:26:53.651924+0000","last_active":"2023-05-22T15:26:53.979395+0000","last_peered":"2023-05-22T15:26:53.979395+0000","last_clean":"2023-05-22T15:26:53.979395+0000","last_became_active":"2023-05-22T15:26:53.651533+0000","last_became_peered":"2023-05-22T15:26:53.651533+0000","last_unstale":"2023-05-22T15:26:53.979395+0000","last_undegraded":"2023-05-22T15:26:53.979395+0000","last_fullsized":"2023-05-22T15:26:53.979395+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_clean_scrub_stamp":"2023-05-22T15:26:52.568670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:27:41.065402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7,2],"acting":[1,5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"22'4","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.612099+0000","last_change":"2023-05-22T15:26:49.186238+0000","last_active":"2023-05-22T15:26:53.612099+0000","last_peered":"2023-05-22T15:26:53.612099+0000","last_clean":"2023-05-22T15:26:53.612099+0000","last_became_active":"2023-05-22T15:26:49.185455+0000","last_became_peered":"2023-05-22T15:26:49.185455+0000","last_unstale":"2023-05-22T15:26:53.612099+0000","last_undegraded":"2023-05-22T15:26:53.612099+0000","last_fullsized":"2023-05-22T15:26:53.612099+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_clean_scrub_stamp":"2023-05-22T15:26:48.127413+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:21:54.376718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":566,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.612858+0000","last_change":"2023-05-22T15:26:49.622435+0000","last_active":"2023-05-22T15:26:53.612858+0000","last_peered":"2023-05-22T15:26:53.612858+0000","last_clean":"2023-05-22T15:26:53.612858+0000","last_became_active":"2023-05-22T15:26:49.621643+0000","last_became_peered":"2023-05-22T15:26:49.621643+0000","last_unstale":"2023-05-22T15:26:53.612858+0000","last_undegraded":"2023-05-22T15:26:53.612858+0000","last_fullsized":"2023-05-22T15:26:53.612858+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:08:07.637726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-05-22T15:26:54.054128+0000","last_change":"2023-05-22T15:26:53.654741+0000","last_active":"2023-05-22T15:26:54.054128+0000","last_peered":"2023-05-22T15:26:54.054128+0000","last_clean":"2023-05-22T15:26:54.054128+0000","last_became_active":"2023-05-22T15:26:53.653669+0000","last_became_peered":"2023-05-22T15:26:53.653669+0000","last_unstale":"2023-05-22T15:26:54.054128+0000","last_undegraded":"2023-05-22T15:26:54.054128+0000","last_fullsized":"2023-05-22T15:26:54.054128+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_clean_scrub_stamp":"2023-05-22T15:26:52.568670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:01:12.890843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,2],"acting":[4,6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"22'1","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.602985+0000","last_change":"2023-05-22T15:26:49.187902+0000","last_active":"2023-05-22T15:26:53.602985+0000","last_peered":"2023-05-22T15:26:53.602985+0000","last_clean":"2023-05-22T15:26:53.602985+0000","last_became_active":"2023-05-22T15:26:49.187361+0000","last_became_peered":"2023-05-22T15:26:49.187361+0000","last_unstale":"2023-05-22T15:26:53.602985+0000","last_undegraded":"2023-05-22T15:26:53.602985+0000","last_fullsized":"2023-05-22T15:26:53.602985+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_clean_scrub_stamp":"2023-05-22T15:26:48.127413+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T18:13:58.291903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.603896+0000","last_change":"2023-05-22T15:26:49.560606+0000","last_active":"2023-05-22T15:26:53.603896+0000","last_peered":"2023-05-22T15:26:53.603896+0000","last_clean":"2023-05-22T15:26:53.603896+0000","last_became_active":"2023-05-22T15:26:49.560318+0000","last_became_peered":"2023-05-22T15:26:49.560318+0000","last_unstale":"2023-05-22T15:26:53.603896+0000","last_undegraded":"2023-05-22T15:26:53.603896+0000","last_fullsized":"2023-05-22T15:26:53.603896+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:43:00.518715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-05-22T15:26:54.055130+0000","last_change":"2023-05-22T15:26:53.648550+0000","last_active":"2023-05-22T15:26:54.055130+0000","last_peered":"2023-05-22T15:26:54.055130+0000","last_clean":"2023-05-22T15:26:54.055130+0000","last_became_active":"2023-05-22T15:26:53.646940+0000","last_became_peered":"2023-05-22T15:26:53.646940+0000","last_unstale":"2023-05-22T15:26:54.055130+0000","last_undegraded":"2023-05-22T15:26:54.055130+0000","last_fullsized":"2023-05-22T15:26:54.055130+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_clean_scrub_stamp":"2023-05-22T15:26:52.568670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T15:41:55.740195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7,4],"acting":[5,0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"22'2","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.634944+0000","last_change":"2023-05-22T15:26:49.210260+0000","last_active":"2023-05-22T15:26:53.634944+0000","last_peered":"2023-05-22T15:26:53.634944+0000","last_clean":"2023-05-22T15:26:53.634944+0000","last_became_active":"2023-05-22T15:26:49.209487+0000","last_became_peered":"2023-05-22T15:26:49.209487+0000","last_unstale":"2023-05-22T15:26:53.634944+0000","last_undegraded":"2023-05-22T15:26:53.634944+0000","last_fullsized":"2023-05-22T15:26:53.634944+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_clean_scrub_stamp":"2023-05-22T15:26:48.127413+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:07:49.933860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.619548+0000","last_change":"2023-05-22T15:26:49.595133+0000","last_active":"2023-05-22T15:26:53.619548+0000","last_peered":"2023-05-22T15:26:53.619548+0000","last_clean":"2023-05-22T15:26:53.619548+0000","last_became_active":"2023-05-22T15:26:49.594382+0000","last_became_peered":"2023-05-22T15:26:49.594382+0000","last_unstale":"2023-05-22T15:26:53.619548+0000","last_undegraded":"2023-05-22T15:26:53.619548+0000","last_fullsized":"2023-05-22T15:26:53.619548+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:11:42.159966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.982645+0000","last_change":"2023-05-22T15:26:53.657958+0000","last_active":"2023-05-22T15:26:53.982645+0000","last_peered":"2023-05-22T15:26:53.982645+0000","last_clean":"2023-05-22T15:26:53.982645+0000","last_became_active":"2023-05-22T15:26:53.656910+0000","last_became_peered":"2023-05-22T15:26:53.656910+0000","last_unstale":"2023-05-22T15:26:53.982645+0000","last_undegraded":"2023-05-22T15:26:53.982645+0000","last_fullsized":"2023-05-22T15:26:53.982645+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_clean_scrub_stamp":"2023-05-22T15:26:52.568670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:56:11.423640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1,6],"acting":[7,4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"22'2","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.634146+0000","last_change":"2023-05-22T15:26:49.210149+0000","last_active":"2023-05-22T15:26:53.634146+0000","last_peered":"2023-05-22T15:26:53.634146+0000","last_clean":"2023-05-22T15:26:53.634146+0000","last_became_active":"2023-05-22T15:26:49.209497+0000","last_became_peered":"2023-05-22T15:26:49.209497+0000","last_unstale":"2023-05-22T15:26:53.634146+0000","last_undegraded":"2023-05-22T15:26:53.634146+0000","last_fullsized":"2023-05-22T15:26:53.634146+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_clean_scrub_stamp":"2023-05-22T15:26:48.127413+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:47:17.588354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.620540+0000","last_change":"2023-05-22T15:26:49.598141+0000","last_active":"2023-05-22T15:26:53.620540+0000","last_peered":"2023-05-22T15:26:53.620540+0000","last_clean":"2023-05-22T15:26:53.620540+0000","last_became_active":"2023-05-22T15:26:49.597262+0000","last_became_peered":"2023-05-22T15:26:49.597262+0000","last_unstale":"2023-05-22T15:26:53.620540+0000","last_undegraded":"2023-05-22T15:26:53.620540+0000","last_fullsized":"2023-05-22T15:26:53.620540+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:33:23.357846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.980103+0000","last_change":"2023-05-22T15:26:53.656672+0000","last_active":"2023-05-22T15:26:53.980103+0000","last_peered":"2023-05-22T15:26:53.980103+0000","last_clean":"2023-05-22T15:26:53.980103+0000","last_became_active":"2023-05-22T15:26:53.656058+0000","last_became_peered":"2023-05-22T15:26:53.656058+0000","last_unstale":"2023-05-22T15:26:53.980103+0000","last_undegraded":"2023-05-22T15:26:53.980103+0000","last_fullsized":"2023-05-22T15:26:53.980103+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_clean_scrub_stamp":"2023-05-22T15:26:52.568670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:38:53.647917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,0],"acting":[1,3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"22'1","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.612141+0000","last_change":"2023-05-22T15:26:49.206147+0000","last_active":"2023-05-22T15:26:53.612141+0000","last_peered":"2023-05-22T15:26:53.612141+0000","last_clean":"2023-05-22T15:26:53.612141+0000","last_became_active":"2023-05-22T15:26:49.205706+0000","last_became_peered":"2023-05-22T15:26:49.205706+0000","last_unstale":"2023-05-22T15:26:53.612141+0000","last_undegraded":"2023-05-22T15:26:53.612141+0000","last_fullsized":"2023-05-22T15:26:53.612141+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_clean_scrub_stamp":"2023-05-22T15:26:48.127413+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:09:00.816941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.601812+0000","last_change":"2023-05-22T15:26:49.578140+0000","last_active":"2023-05-22T15:26:53.601812+0000","last_peered":"2023-05-22T15:26:53.601812+0000","last_clean":"2023-05-22T15:26:53.601812+0000","last_became_active":"2023-05-22T15:26:49.576922+0000","last_became_peered":"2023-05-22T15:26:49.576922+0000","last_unstale":"2023-05-22T15:26:53.601812+0000","last_undegraded":"2023-05-22T15:26:53.601812+0000","last_fullsized":"2023-05-22T15:26:53.601812+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T16:16:40.288204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-05-22T15:26:54.054907+0000","last_change":"2023-05-22T15:26:53.643098+0000","last_active":"2023-05-22T15:26:54.054907+0000","last_peered":"2023-05-22T15:26:54.054907+0000","last_clean":"2023-05-22T15:26:54.054907+0000","last_became_active":"2023-05-22T15:26:53.642038+0000","last_became_peered":"2023-05-22T15:26:53.642038+0000","last_unstale":"2023-05-22T15:26:54.054907+0000","last_undegraded":"2023-05-22T15:26:54.054907+0000","last_fullsized":"2023-05-22T15:26:54.054907+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_clean_scrub_stamp":"2023-05-22T15:26:52.568670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T18:43:09.309787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2,0],"acting":[4,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"22'1","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.620161+0000","last_change":"2023-05-22T15:26:49.199743+0000","last_active":"2023-05-22T15:26:53.620161+0000","last_peered":"2023-05-22T15:26:53.620161+0000","last_clean":"2023-05-22T15:26:53.620161+0000","last_became_active":"2023-05-22T15:26:49.197695+0000","last_became_peered":"2023-05-22T15:26:49.197695+0000","last_unstale":"2023-05-22T15:26:53.620161+0000","last_undegraded":"2023-05-22T15:26:53.620161+0000","last_fullsized":"2023-05-22T15:26:53.620161+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_clean_scrub_stamp":"2023-05-22T15:26:48.127413+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:21:54.376718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.602293+0000","last_change":"2023-05-22T15:26:49.580932+0000","last_active":"2023-05-22T15:26:53.602293+0000","last_peered":"2023-05-22T15:26:53.602293+0000","last_clean":"2023-05-22T15:26:53.602293+0000","last_became_active":"2023-05-22T15:26:49.580612+0000","last_became_peered":"2023-05-22T15:26:49.580612+0000","last_unstale":"2023-05-22T15:26:53.602293+0000","last_undegraded":"2023-05-22T15:26:53.602293+0000","last_fullsized":"2023-05-22T15:26:53.602293+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:19:30.391062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-05-22T15:26:54.051798+0000","last_change":"2023-05-22T15:26:53.662170+0000","last_active":"2023-05-22T15:26:54.051798+0000","last_peered":"2023-05-22T15:26:54.051798+0000","last_clean":"2023-05-22T15:26:54.051798+0000","last_became_active":"2023-05-22T15:26:53.661164+0000","last_became_peered":"2023-05-22T15:26:53.661164+0000","last_unstale":"2023-05-22T15:26:54.051798+0000","last_undegraded":"2023-05-22T15:26:54.051798+0000","last_fullsized":"2023-05-22T15:26:54.051798+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_clean_scrub_stamp":"2023-05-22T15:26:52.568670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:46:53.737748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3,7],"acting":[6,4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.613865+0000","last_change":"2023-05-22T15:26:49.187546+0000","last_active":"2023-05-22T15:26:53.613865+0000","last_peered":"2023-05-22T15:26:53.613865+0000","last_clean":"2023-05-22T15:26:53.613865+0000","last_became_active":"2023-05-22T15:26:49.187022+0000","last_became_peered":"2023-05-22T15:26:49.187022+0000","last_unstale":"2023-05-22T15:26:53.613865+0000","last_undegraded":"2023-05-22T15:26:53.613865+0000","last_fullsized":"2023-05-22T15:26:53.613865+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_clean_scrub_stamp":"2023-05-22T15:26:48.127413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:53:34.802286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.611910+0000","last_change":"2023-05-22T15:26:49.623535+0000","last_active":"2023-05-22T15:26:53.611910+0000","last_peered":"2023-05-22T15:26:53.611910+0000","last_clean":"2023-05-22T15:26:53.611910+0000","last_became_active":"2023-05-22T15:26:49.622454+0000","last_became_peered":"2023-05-22T15:26:49.622454+0000","last_unstale":"2023-05-22T15:26:53.611910+0000","last_undegraded":"2023-05-22T15:26:53.611910+0000","last_fullsized":"2023-05-22T15:26:53.611910+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:43:00.518715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-05-22T15:26:54.048982+0000","last_change":"2023-05-22T15:26:53.645181+0000","last_active":"2023-05-22T15:26:54.048982+0000","last_peered":"2023-05-22T15:26:54.048982+0000","last_clean":"2023-05-22T15:26:54.048982+0000","last_became_active":"2023-05-22T15:26:53.644764+0000","last_became_peered":"2023-05-22T15:26:53.644764+0000","last_unstale":"2023-05-22T15:26:54.048982+0000","last_undegraded":"2023-05-22T15:26:54.048982+0000","last_fullsized":"2023-05-22T15:26:54.048982+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_clean_scrub_stamp":"2023-05-22T15:26:52.568670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:12:09.126657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1,2],"acting":[0,7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.603942+0000","last_change":"2023-05-22T15:26:49.188920+0000","last_active":"2023-05-22T15:26:53.603942+0000","last_peered":"2023-05-22T15:26:53.603942+0000","last_clean":"2023-05-22T15:26:53.603942+0000","last_became_active":"2023-05-22T15:26:49.188538+0000","last_became_peered":"2023-05-22T15:26:49.188538+0000","last_unstale":"2023-05-22T15:26:53.603942+0000","last_undegraded":"2023-05-22T15:26:53.603942+0000","last_fullsized":"2023-05-22T15:26:53.603942+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_clean_scrub_stamp":"2023-05-22T15:26:48.127413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:09:00.816941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.603003+0000","last_change":"2023-05-22T15:26:49.606092+0000","last_active":"2023-05-22T15:26:53.603003+0000","last_peered":"2023-05-22T15:26:53.603003+0000","last_clean":"2023-05-22T15:26:53.603003+0000","last_became_active":"2023-05-22T15:26:49.605324+0000","last_became_peered":"2023-05-22T15:26:49.605324+0000","last_unstale":"2023-05-22T15:26:53.603003+0000","last_undegraded":"2023-05-22T15:26:53.603003+0000","last_fullsized":"2023-05-22T15:26:53.603003+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:10:51.435508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.603748+0000","last_change":"2023-05-22T15:26:49.578105+0000","last_active":"2023-05-22T15:26:53.603748+0000","last_peered":"2023-05-22T15:26:53.603748+0000","last_clean":"2023-05-22T15:26:53.603748+0000","last_became_active":"2023-05-22T15:26:49.577534+0000","last_became_peered":"2023-05-22T15:26:49.577534+0000","last_unstale":"2023-05-22T15:26:53.603748+0000","last_undegraded":"2023-05-22T15:26:53.603748+0000","last_fullsized":"2023-05-22T15:26:53.603748+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:44:52.369716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.612292+0000","last_change":"2023-05-22T15:26:49.186504+0000","last_active":"2023-05-22T15:26:53.612292+0000","last_peered":"2023-05-22T15:26:53.612292+0000","last_clean":"2023-05-22T15:26:53.612292+0000","last_became_active":"2023-05-22T15:26:49.185880+0000","last_became_peered":"2023-05-22T15:26:49.185880+0000","last_unstale":"2023-05-22T15:26:53.612292+0000","last_undegraded":"2023-05-22T15:26:53.612292+0000","last_fullsized":"2023-05-22T15:26:53.612292+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_clean_scrub_stamp":"2023-05-22T15:26:48.127413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T18:13:58.291903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-05-22T15:26:54.051126+0000","last_change":"2023-05-22T15:26:53.647905+0000","last_active":"2023-05-22T15:26:54.051126+0000","last_peered":"2023-05-22T15:26:54.051126+0000","last_clean":"2023-05-22T15:26:54.051126+0000","last_became_active":"2023-05-22T15:26:53.646790+0000","last_became_peered":"2023-05-22T15:26:53.646790+0000","last_unstale":"2023-05-22T15:26:54.051126+0000","last_undegraded":"2023-05-22T15:26:54.051126+0000","last_fullsized":"2023-05-22T15:26:54.051126+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_clean_scrub_stamp":"2023-05-22T15:26:52.568670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T03:18:03.204915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7,1],"acting":[6,4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.603724+0000","last_change":"2023-05-22T15:26:49.623613+0000","last_active":"2023-05-22T15:26:53.603724+0000","last_peered":"2023-05-22T15:26:53.603724+0000","last_clean":"2023-05-22T15:26:53.603724+0000","last_became_active":"2023-05-22T15:26:49.623385+0000","last_became_peered":"2023-05-22T15:26:49.623385+0000","last_unstale":"2023-05-22T15:26:53.603724+0000","last_undegraded":"2023-05-22T15:26:53.603724+0000","last_fullsized":"2023-05-22T15:26:53.603724+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:22:10.117410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.635389+0000","last_change":"2023-05-22T15:26:49.178387+0000","last_active":"2023-05-22T15:26:53.635389+0000","last_peered":"2023-05-22T15:26:53.635389+0000","last_clean":"2023-05-22T15:26:53.635389+0000","last_became_active":"2023-05-22T15:26:49.177428+0000","last_became_peered":"2023-05-22T15:26:49.177428+0000","last_unstale":"2023-05-22T15:26:53.635389+0000","last_undegraded":"2023-05-22T15:26:53.635389+0000","last_fullsized":"2023-05-22T15:26:53.635389+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_clean_scrub_stamp":"2023-05-22T15:26:48.127413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:07:41.791660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.981431+0000","last_change":"2023-05-22T15:26:53.660574+0000","last_active":"2023-05-22T15:26:53.981431+0000","last_peered":"2023-05-22T15:26:53.981431+0000","last_clean":"2023-05-22T15:26:53.981431+0000","last_became_active":"2023-05-22T15:26:53.659567+0000","last_became_peered":"2023-05-22T15:26:53.659567+0000","last_unstale":"2023-05-22T15:26:53.981431+0000","last_undegraded":"2023-05-22T15:26:53.981431+0000","last_fullsized":"2023-05-22T15:26:53.981431+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_clean_scrub_stamp":"2023-05-22T15:26:52.568670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:07:24.608031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6,0],"acting":[7,4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.15","version":"22'3","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.610146+0000","last_change":"2023-05-22T15:26:49.188354+0000","last_active":"2023-05-22T15:26:53.610146+0000","last_peered":"2023-05-22T15:26:53.610146+0000","last_clean":"2023-05-22T15:26:53.610146+0000","last_became_active":"2023-05-22T15:26:49.187902+0000","last_became_peered":"2023-05-22T15:26:49.187902+0000","last_unstale":"2023-05-22T15:26:53.610146+0000","last_undegraded":"2023-05-22T15:26:53.610146+0000","last_fullsized":"2023-05-22T15:26:53.610146+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_clean_scrub_stamp":"2023-05-22T15:26:48.127413+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:07:49.933860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.612914+0000","last_change":"2023-05-22T15:26:49.619577+0000","last_active":"2023-05-22T15:26:53.612914+0000","last_peered":"2023-05-22T15:26:53.612914+0000","last_clean":"2023-05-22T15:26:53.612914+0000","last_became_active":"2023-05-22T15:26:49.619210+0000","last_became_peered":"2023-05-22T15:26:49.619210+0000","last_unstale":"2023-05-22T15:26:53.612914+0000","last_undegraded":"2023-05-22T15:26:53.612914+0000","last_fullsized":"2023-05-22T15:26:53.612914+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:48:33.611833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.992745+0000","last_change":"2023-05-22T15:26:53.650745+0000","last_active":"2023-05-22T15:26:53.992745+0000","last_peered":"2023-05-22T15:26:53.992745+0000","last_clean":"2023-05-22T15:26:53.992745+0000","last_became_active":"2023-05-22T15:26:53.650209+0000","last_became_peered":"2023-05-22T15:26:53.650209+0000","last_unstale":"2023-05-22T15:26:53.992745+0000","last_undegraded":"2023-05-22T15:26:53.992745+0000","last_fullsized":"2023-05-22T15:26:53.992745+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_clean_scrub_stamp":"2023-05-22T15:26:52.568670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:15:56.755592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,5],"acting":[3,0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.14","version":"27'9","reported_seq":23,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-05-22T15:26:59.032004+0000","last_change":"2023-05-22T15:26:49.201484+0000","last_active":"2023-05-22T15:26:59.032004+0000","last_peered":"2023-05-22T15:26:59.032004+0000","last_clean":"2023-05-22T15:26:59.032004+0000","last_became_active":"2023-05-22T15:26:49.200972+0000","last_became_peered":"2023-05-22T15:26:49.200972+0000","last_unstale":"2023-05-22T15:26:59.032004+0000","last_undegraded":"2023-05-22T15:26:59.032004+0000","last_fullsized":"2023-05-22T15:26:59.032004+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_clean_scrub_stamp":"2023-05-22T15:26:48.127413+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:59:54.009838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3745,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.620650+0000","last_change":"2023-05-22T15:26:49.619268+0000","last_active":"2023-05-22T15:26:53.620650+0000","last_peered":"2023-05-22T15:26:53.620650+0000","last_clean":"2023-05-22T15:26:53.620650+0000","last_became_active":"2023-05-22T15:26:49.618724+0000","last_became_peered":"2023-05-22T15:26:49.618724+0000","last_unstale":"2023-05-22T15:26:53.620650+0000","last_undegraded":"2023-05-22T15:26:53.620650+0000","last_fullsized":"2023-05-22T15:26:53.620650+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:28:29.607131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.979878+0000","last_change":"2023-05-22T15:26:53.653181+0000","last_active":"2023-05-22T15:26:53.979878+0000","last_peered":"2023-05-22T15:26:53.979878+0000","last_clean":"2023-05-22T15:26:53.979878+0000","last_became_active":"2023-05-22T15:26:53.652676+0000","last_became_peered":"2023-05-22T15:26:53.652676+0000","last_unstale":"2023-05-22T15:26:53.979878+0000","last_undegraded":"2023-05-22T15:26:53.979878+0000","last_fullsized":"2023-05-22T15:26:53.979878+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_clean_scrub_stamp":"2023-05-22T15:26:52.568670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T15:46:20.544673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3,7],"acting":[1,5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.616683+0000","last_change":"2023-05-22T15:26:49.597250+0000","last_active":"2023-05-22T15:26:53.616683+0000","last_peered":"2023-05-22T15:26:53.616683+0000","last_clean":"2023-05-22T15:26:53.616683+0000","last_became_active":"2023-05-22T15:26:49.596487+0000","last_became_peered":"2023-05-22T15:26:49.596487+0000","last_unstale":"2023-05-22T15:26:53.616683+0000","last_undegraded":"2023-05-22T15:26:53.616683+0000","last_fullsized":"2023-05-22T15:26:53.616683+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:18:20.756810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.603054+0000","last_change":"2023-05-22T15:26:44.982421+0000","last_active":"2023-05-22T15:26:53.603054+0000","last_peered":"2023-05-22T15:26:53.603054+0000","last_clean":"2023-05-22T15:26:53.603054+0000","last_became_active":"2023-05-22T15:26:44.981586+0000","last_became_peered":"2023-05-22T15:26:44.981586+0000","last_unstale":"2023-05-22T15:26:53.603054+0000","last_undegraded":"2023-05-22T15:26:53.603054+0000","last_fullsized":"2023-05-22T15:26:53.603054+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:14:11.156544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"22'1","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.612191+0000","last_change":"2023-05-22T15:26:49.185412+0000","last_active":"2023-05-22T15:26:53.612191+0000","last_peered":"2023-05-22T15:26:53.612191+0000","last_clean":"2023-05-22T15:26:53.612191+0000","last_became_active":"2023-05-22T15:26:49.184896+0000","last_became_peered":"2023-05-22T15:26:49.184896+0000","last_unstale":"2023-05-22T15:26:53.612191+0000","last_undegraded":"2023-05-22T15:26:53.612191+0000","last_fullsized":"2023-05-22T15:26:53.612191+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_clean_scrub_stamp":"2023-05-22T15:26:48.127413+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:13:03.975127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.992364+0000","last_change":"2023-05-22T15:26:53.669441+0000","last_active":"2023-05-22T15:26:53.992364+0000","last_peered":"2023-05-22T15:26:53.992364+0000","last_clean":"2023-05-22T15:26:53.992364+0000","last_became_active":"2023-05-22T15:26:53.669000+0000","last_became_peered":"2023-05-22T15:26:53.669000+0000","last_unstale":"2023-05-22T15:26:53.992364+0000","last_undegraded":"2023-05-22T15:26:53.992364+0000","last_fullsized":"2023-05-22T15:26:53.992364+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_clean_scrub_stamp":"2023-05-22T15:26:52.568670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T18:55:06.398032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,0],"acting":[3,1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.603843+0000","last_change":"2023-05-22T15:26:49.623319+0000","last_active":"2023-05-22T15:26:53.603843+0000","last_peered":"2023-05-22T15:26:53.603843+0000","last_clean":"2023-05-22T15:26:53.603843+0000","last_became_active":"2023-05-22T15:26:49.622892+0000","last_became_peered":"2023-05-22T15:26:49.622892+0000","last_unstale":"2023-05-22T15:26:53.603843+0000","last_undegraded":"2023-05-22T15:26:53.603843+0000","last_fullsized":"2023-05-22T15:26:53.603843+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:48:33.611833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.612572+0000","last_change":"2023-05-22T15:26:44.978858+0000","last_active":"2023-05-22T15:26:53.612572+0000","last_peered":"2023-05-22T15:26:53.612572+0000","last_clean":"2023-05-22T15:26:53.612572+0000","last_became_active":"2023-05-22T15:26:44.978443+0000","last_became_peered":"2023-05-22T15:26:44.978443+0000","last_unstale":"2023-05-22T15:26:53.612572+0000","last_undegraded":"2023-05-22T15:26:53.612572+0000","last_fullsized":"2023-05-22T15:26:53.612572+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:46:29.290243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"22'2","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.606372+0000","last_change":"2023-05-22T15:26:49.188060+0000","last_active":"2023-05-22T15:26:53.606372+0000","last_peered":"2023-05-22T15:26:53.606372+0000","last_clean":"2023-05-22T15:26:53.606372+0000","last_became_active":"2023-05-22T15:26:49.187586+0000","last_became_peered":"2023-05-22T15:26:49.187586+0000","last_unstale":"2023-05-22T15:26:53.606372+0000","last_undegraded":"2023-05-22T15:26:53.606372+0000","last_fullsized":"2023-05-22T15:26:53.606372+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_clean_scrub_stamp":"2023-05-22T15:26:48.127413+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:59:54.009838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-05-22T15:26:54.053713+0000","last_change":"2023-05-22T15:26:53.659559+0000","last_active":"2023-05-22T15:26:54.053713+0000","last_peered":"2023-05-22T15:26:54.053713+0000","last_clean":"2023-05-22T15:26:54.053713+0000","last_became_active":"2023-05-22T15:26:53.655056+0000","last_became_peered":"2023-05-22T15:26:53.655056+0000","last_unstale":"2023-05-22T15:26:54.053713+0000","last_undegraded":"2023-05-22T15:26:54.053713+0000","last_fullsized":"2023-05-22T15:26:54.053713+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_clean_scrub_stamp":"2023-05-22T15:26:52.568670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:47:56.845713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7,0],"acting":[4,3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.611858+0000","last_change":"2023-05-22T15:26:49.589781+0000","last_active":"2023-05-22T15:26:53.611858+0000","last_peered":"2023-05-22T15:26:53.611858+0000","last_clean":"2023-05-22T15:26:53.611858+0000","last_became_active":"2023-05-22T15:26:49.588830+0000","last_became_peered":"2023-05-22T15:26:49.588830+0000","last_unstale":"2023-05-22T15:26:53.611858+0000","last_undegraded":"2023-05-22T15:26:53.611858+0000","last_fullsized":"2023-05-22T15:26:53.611858+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:52:48.123148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.615526+0000","last_change":"2023-05-22T15:26:45.092367+0000","last_active":"2023-05-22T15:26:53.615526+0000","last_peered":"2023-05-22T15:26:53.615526+0000","last_clean":"2023-05-22T15:26:53.615526+0000","last_became_active":"2023-05-22T15:26:45.091951+0000","last_became_peered":"2023-05-22T15:26:45.091951+0000","last_unstale":"2023-05-22T15:26:53.615526+0000","last_undegraded":"2023-05-22T15:26:53.615526+0000","last_fullsized":"2023-05-22T15:26:53.615526+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:42:57.973030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.619125+0000","last_change":"2023-05-22T15:26:49.180237+0000","last_active":"2023-05-22T15:26:53.619125+0000","last_peered":"2023-05-22T15:26:53.619125+0000","last_clean":"2023-05-22T15:26:53.619125+0000","last_became_active":"2023-05-22T15:26:49.179622+0000","last_became_peered":"2023-05-22T15:26:49.179622+0000","last_unstale":"2023-05-22T15:26:53.619125+0000","last_undegraded":"2023-05-22T15:26:53.619125+0000","last_fullsized":"2023-05-22T15:26:53.619125+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_clean_scrub_stamp":"2023-05-22T15:26:48.127413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T16:16:39.277093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-05-22T15:26:54.051370+0000","last_change":"2023-05-22T15:26:53.623285+0000","last_active":"2023-05-22T15:26:54.051370+0000","last_peered":"2023-05-22T15:26:54.051370+0000","last_clean":"2023-05-22T15:26:54.051370+0000","last_became_active":"2023-05-22T15:26:53.622116+0000","last_became_peered":"2023-05-22T15:26:53.622116+0000","last_unstale":"2023-05-22T15:26:54.051370+0000","last_undegraded":"2023-05-22T15:26:54.051370+0000","last_fullsized":"2023-05-22T15:26:54.051370+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_clean_scrub_stamp":"2023-05-22T15:26:52.568670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T16:31:40.104597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5,2],"acting":[6,0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.613088+0000","last_change":"2023-05-22T15:26:49.576833+0000","last_active":"2023-05-22T15:26:53.613088+0000","last_peered":"2023-05-22T15:26:53.613088+0000","last_clean":"2023-05-22T15:26:53.613088+0000","last_became_active":"2023-05-22T15:26:49.576559+0000","last_became_peered":"2023-05-22T15:26:49.576559+0000","last_unstale":"2023-05-22T15:26:53.613088+0000","last_undegraded":"2023-05-22T15:26:53.613088+0000","last_fullsized":"2023-05-22T15:26:53.613088+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:32:19.582542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.619335+0000","last_change":"2023-05-22T15:26:45.091475+0000","last_active":"2023-05-22T15:26:53.619335+0000","last_peered":"2023-05-22T15:26:53.619335+0000","last_clean":"2023-05-22T15:26:53.619335+0000","last_became_active":"2023-05-22T15:26:45.091143+0000","last_became_peered":"2023-05-22T15:26:45.091143+0000","last_unstale":"2023-05-22T15:26:53.619335+0000","last_undegraded":"2023-05-22T15:26:53.619335+0000","last_fullsized":"2023-05-22T15:26:53.619335+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:12:41.464054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.635151+0000","last_change":"2023-05-22T15:26:49.187061+0000","last_active":"2023-05-22T15:26:53.635151+0000","last_peered":"2023-05-22T15:26:53.635151+0000","last_clean":"2023-05-22T15:26:53.635151+0000","last_became_active":"2023-05-22T15:26:49.185965+0000","last_became_peered":"2023-05-22T15:26:49.185965+0000","last_unstale":"2023-05-22T15:26:53.635151+0000","last_undegraded":"2023-05-22T15:26:53.635151+0000","last_fullsized":"2023-05-22T15:26:53.635151+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_clean_scrub_stamp":"2023-05-22T15:26:48.127413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T16:16:39.277093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-05-22T15:26:54.048601+0000","last_change":"2023-05-22T15:26:53.650446+0000","last_active":"2023-05-22T15:26:54.048601+0000","last_peered":"2023-05-22T15:26:54.048601+0000","last_clean":"2023-05-22T15:26:54.048601+0000","last_became_active":"2023-05-22T15:26:53.649492+0000","last_became_peered":"2023-05-22T15:26:53.649492+0000","last_unstale":"2023-05-22T15:26:54.048601+0000","last_undegraded":"2023-05-22T15:26:54.048601+0000","last_fullsized":"2023-05-22T15:26:54.048601+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_clean_scrub_stamp":"2023-05-22T15:26:52.568670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:56:11.423640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6,5],"acting":[0,1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.619014+0000","last_change":"2023-05-22T15:26:49.581808+0000","last_active":"2023-05-22T15:26:53.619014+0000","last_peered":"2023-05-22T15:26:53.619014+0000","last_clean":"2023-05-22T15:26:53.619014+0000","last_became_active":"2023-05-22T15:26:49.581501+0000","last_became_peered":"2023-05-22T15:26:49.581501+0000","last_unstale":"2023-05-22T15:26:53.619014+0000","last_undegraded":"2023-05-22T15:26:53.619014+0000","last_fullsized":"2023-05-22T15:26:53.619014+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:52:48.123148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.620868+0000","last_change":"2023-05-22T15:26:49.204122+0000","last_active":"2023-05-22T15:26:53.620868+0000","last_peered":"2023-05-22T15:26:53.620868+0000","last_clean":"2023-05-22T15:26:53.620868+0000","last_became_active":"2023-05-22T15:26:49.199596+0000","last_became_peered":"2023-05-22T15:26:49.199596+0000","last_unstale":"2023-05-22T15:26:53.620868+0000","last_undegraded":"2023-05-22T15:26:53.620868+0000","last_fullsized":"2023-05-22T15:26:53.620868+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_clean_scrub_stamp":"2023-05-22T15:26:48.127413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:09:00.816941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.635108+0000","last_change":"2023-05-22T15:26:44.976632+0000","last_active":"2023-05-22T15:26:53.635108+0000","last_peered":"2023-05-22T15:26:53.635108+0000","last_clean":"2023-05-22T15:26:53.635108+0000","last_became_active":"2023-05-22T15:26:44.975546+0000","last_became_peered":"2023-05-22T15:26:44.975546+0000","last_unstale":"2023-05-22T15:26:53.635108+0000","last_undegraded":"2023-05-22T15:26:53.635108+0000","last_fullsized":"2023-05-22T15:26:53.635108+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:42:57.973030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-05-22T15:26:54.048781+0000","last_change":"2023-05-22T15:26:53.669482+0000","last_active":"2023-05-22T15:26:54.048781+0000","last_peered":"2023-05-22T15:26:54.048781+0000","last_clean":"2023-05-22T15:26:54.048781+0000","last_became_active":"2023-05-22T15:26:53.669119+0000","last_became_peered":"2023-05-22T15:26:53.669119+0000","last_unstale":"2023-05-22T15:26:54.048781+0000","last_undegraded":"2023-05-22T15:26:54.048781+0000","last_fullsized":"2023-05-22T15:26:54.048781+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_clean_scrub_stamp":"2023-05-22T15:26:52.568670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:38:53.647917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,6],"acting":[0,1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.613087+0000","last_change":"2023-05-22T15:26:49.589555+0000","last_active":"2023-05-22T15:26:53.613087+0000","last_peered":"2023-05-22T15:26:53.613087+0000","last_clean":"2023-05-22T15:26:53.613087+0000","last_became_active":"2023-05-22T15:26:49.589271+0000","last_became_peered":"2023-05-22T15:26:49.589271+0000","last_unstale":"2023-05-22T15:26:53.613087+0000","last_undegraded":"2023-05-22T15:26:53.613087+0000","last_fullsized":"2023-05-22T15:26:53.613087+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:43:00.518715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.614048+0000","last_change":"2023-05-22T15:26:49.179359+0000","last_active":"2023-05-22T15:26:53.614048+0000","last_peered":"2023-05-22T15:26:53.614048+0000","last_clean":"2023-05-22T15:26:53.614048+0000","last_became_active":"2023-05-22T15:26:49.178930+0000","last_became_peered":"2023-05-22T15:26:49.178930+0000","last_unstale":"2023-05-22T15:26:53.614048+0000","last_undegraded":"2023-05-22T15:26:53.614048+0000","last_fullsized":"2023-05-22T15:26:53.614048+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_clean_scrub_stamp":"2023-05-22T15:26:48.127413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:59:54.009838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.597765+0000","last_change":"2023-05-22T15:26:44.969469+0000","last_active":"2023-05-22T15:26:53.597765+0000","last_peered":"2023-05-22T15:26:53.597765+0000","last_clean":"2023-05-22T15:26:53.597765+0000","last_became_active":"2023-05-22T15:26:44.969083+0000","last_became_peered":"2023-05-22T15:26:44.969083+0000","last_unstale":"2023-05-22T15:26:53.597765+0000","last_undegraded":"2023-05-22T15:26:53.597765+0000","last_fullsized":"2023-05-22T15:26:53.597765+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:29:23.670067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.978603+0000","last_change":"2023-05-22T15:26:53.636902+0000","last_active":"2023-05-22T15:26:53.978603+0000","last_peered":"2023-05-22T15:26:53.978603+0000","last_clean":"2023-05-22T15:26:53.978603+0000","last_became_active":"2023-05-22T15:26:53.636337+0000","last_became_peered":"2023-05-22T15:26:53.636337+0000","last_unstale":"2023-05-22T15:26:53.978603+0000","last_undegraded":"2023-05-22T15:26:53.978603+0000","last_fullsized":"2023-05-22T15:26:53.978603+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_clean_scrub_stamp":"2023-05-22T15:26:52.568670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:58:01.437826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1,7],"acting":[2,5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.612001+0000","last_change":"2023-05-22T15:26:49.581103+0000","last_active":"2023-05-22T15:26:53.612001+0000","last_peered":"2023-05-22T15:26:53.612001+0000","last_clean":"2023-05-22T15:26:53.612001+0000","last_became_active":"2023-05-22T15:26:49.580823+0000","last_became_peered":"2023-05-22T15:26:49.580823+0000","last_unstale":"2023-05-22T15:26:53.612001+0000","last_undegraded":"2023-05-22T15:26:53.612001+0000","last_fullsized":"2023-05-22T15:26:53.612001+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:32:19.582542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.611473+0000","last_change":"2023-05-22T15:26:49.205877+0000","last_active":"2023-05-22T15:26:53.611473+0000","last_peered":"2023-05-22T15:26:53.611473+0000","last_clean":"2023-05-22T15:26:53.611473+0000","last_became_active":"2023-05-22T15:26:49.205432+0000","last_became_peered":"2023-05-22T15:26:49.205432+0000","last_unstale":"2023-05-22T15:26:53.611473+0000","last_undegraded":"2023-05-22T15:26:53.611473+0000","last_fullsized":"2023-05-22T15:26:53.611473+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_clean_scrub_stamp":"2023-05-22T15:26:48.127413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:59:54.009838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.609876+0000","last_change":"2023-05-22T15:26:44.977198+0000","last_active":"2023-05-22T15:26:53.609876+0000","last_peered":"2023-05-22T15:26:53.609876+0000","last_clean":"2023-05-22T15:26:53.609876+0000","last_became_active":"2023-05-22T15:26:44.976445+0000","last_became_peered":"2023-05-22T15:26:44.976445+0000","last_unstale":"2023-05-22T15:26:53.609876+0000","last_undegraded":"2023-05-22T15:26:53.609876+0000","last_fullsized":"2023-05-22T15:26:53.609876+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:12:41.464054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-05-22T15:26:54.056650+0000","last_change":"2023-05-22T15:26:53.667459+0000","last_active":"2023-05-22T15:26:54.056650+0000","last_peered":"2023-05-22T15:26:54.056650+0000","last_clean":"2023-05-22T15:26:54.056650+0000","last_became_active":"2023-05-22T15:26:53.666553+0000","last_became_peered":"2023-05-22T15:26:53.666553+0000","last_unstale":"2023-05-22T15:26:54.056650+0000","last_undegraded":"2023-05-22T15:26:54.056650+0000","last_fullsized":"2023-05-22T15:26:54.056650+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_clean_scrub_stamp":"2023-05-22T15:26:52.568670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:08:31.638326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0,4],"acting":[5,1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.602418+0000","last_change":"2023-05-22T15:26:49.606241+0000","last_active":"2023-05-22T15:26:53.602418+0000","last_peered":"2023-05-22T15:26:53.602418+0000","last_clean":"2023-05-22T15:26:53.602418+0000","last_became_active":"2023-05-22T15:26:49.605628+0000","last_became_peered":"2023-05-22T15:26:49.605628+0000","last_unstale":"2023-05-22T15:26:53.602418+0000","last_undegraded":"2023-05-22T15:26:53.602418+0000","last_fullsized":"2023-05-22T15:26:53.602418+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:59:23.805894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.602217+0000","last_change":"2023-05-22T15:26:49.185421+0000","last_active":"2023-05-22T15:26:53.602217+0000","last_peered":"2023-05-22T15:26:53.602217+0000","last_clean":"2023-05-22T15:26:53.602217+0000","last_became_active":"2023-05-22T15:26:49.184948+0000","last_became_peered":"2023-05-22T15:26:49.184948+0000","last_unstale":"2023-05-22T15:26:53.602217+0000","last_undegraded":"2023-05-22T15:26:53.602217+0000","last_fullsized":"2023-05-22T15:26:53.602217+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_clean_scrub_stamp":"2023-05-22T15:26:48.127413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:21:54.376718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.634293+0000","last_change":"2023-05-22T15:26:44.971951+0000","last_active":"2023-05-22T15:26:53.634293+0000","last_peered":"2023-05-22T15:26:53.634293+0000","last_clean":"2023-05-22T15:26:53.634293+0000","last_became_active":"2023-05-22T15:26:44.970872+0000","last_became_peered":"2023-05-22T15:26:44.970872+0000","last_unstale":"2023-05-22T15:26:53.634293+0000","last_undegraded":"2023-05-22T15:26:53.634293+0000","last_fullsized":"2023-05-22T15:26:53.634293+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:55:34.752949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":169,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.610124+0000","last_change":"2023-05-22T15:26:39.318204+0000","last_active":"2023-05-22T15:26:53.610124+0000","last_peered":"2023-05-22T15:26:53.610124+0000","last_clean":"2023-05-22T15:26:53.610124+0000","last_became_active":"2023-05-22T15:26:39.315370+0000","last_became_peered":"2023-05-22T15:26:39.315370+0000","last_unstale":"2023-05-22T15:26:53.610124+0000","last_undegraded":"2023-05-22T15:26:53.610124+0000","last_fullsized":"2023-05-22T15:26:53.610124+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:38.020076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:38.020076+0000","last_clean_scrub_stamp":"2023-05-22T15:26:38.020076+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:06:39.207543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.982716+0000","last_change":"2023-05-22T15:26:53.649648+0000","last_active":"2023-05-22T15:26:53.982716+0000","last_peered":"2023-05-22T15:26:53.982716+0000","last_clean":"2023-05-22T15:26:53.982716+0000","last_became_active":"2023-05-22T15:26:53.648664+0000","last_became_peered":"2023-05-22T15:26:53.648664+0000","last_unstale":"2023-05-22T15:26:53.982716+0000","last_undegraded":"2023-05-22T15:26:53.982716+0000","last_fullsized":"2023-05-22T15:26:53.982716+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_clean_scrub_stamp":"2023-05-22T15:26:52.568670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T03:14:12.733785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5,3],"acting":[7,2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.612919+0000","last_change":"2023-05-22T15:26:49.578502+0000","last_active":"2023-05-22T15:26:53.612919+0000","last_peered":"2023-05-22T15:26:53.612919+0000","last_clean":"2023-05-22T15:26:53.612919+0000","last_became_active":"2023-05-22T15:26:49.577743+0000","last_became_peered":"2023-05-22T15:26:49.577743+0000","last_unstale":"2023-05-22T15:26:53.612919+0000","last_undegraded":"2023-05-22T15:26:53.612919+0000","last_fullsized":"2023-05-22T15:26:53.612919+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:22:10.117410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.604031+0000","last_change":"2023-05-22T15:26:49.191706+0000","last_active":"2023-05-22T15:26:53.604031+0000","last_peered":"2023-05-22T15:26:53.604031+0000","last_clean":"2023-05-22T15:26:53.604031+0000","last_became_active":"2023-05-22T15:26:49.190396+0000","last_became_peered":"2023-05-22T15:26:49.190396+0000","last_unstale":"2023-05-22T15:26:53.604031+0000","last_undegraded":"2023-05-22T15:26:53.604031+0000","last_fullsized":"2023-05-22T15:26:53.604031+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_clean_scrub_stamp":"2023-05-22T15:26:48.127413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:13:03.975127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-05-22T15:26:54.055266+0000","last_change":"2023-05-22T15:26:53.656007+0000","last_active":"2023-05-22T15:26:54.055266+0000","last_peered":"2023-05-22T15:26:54.055266+0000","last_clean":"2023-05-22T15:26:54.055266+0000","last_became_active":"2023-05-22T15:26:53.652989+0000","last_became_peered":"2023-05-22T15:26:53.652989+0000","last_unstale":"2023-05-22T15:26:54.055266+0000","last_undegraded":"2023-05-22T15:26:54.055266+0000","last_fullsized":"2023-05-22T15:26:54.055266+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_clean_scrub_stamp":"2023-05-22T15:26:52.568670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:36:00.298083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0,2],"acting":[4,5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.621541+0000","last_change":"2023-05-22T15:26:49.622122+0000","last_active":"2023-05-22T15:26:53.621541+0000","last_peered":"2023-05-22T15:26:53.621541+0000","last_clean":"2023-05-22T15:26:53.621541+0000","last_became_active":"2023-05-22T15:26:49.621591+0000","last_became_peered":"2023-05-22T15:26:49.621591+0000","last_unstale":"2023-05-22T15:26:53.621541+0000","last_undegraded":"2023-05-22T15:26:53.621541+0000","last_fullsized":"2023-05-22T15:26:53.621541+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T18:05:09.939239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.617051+0000","last_change":"2023-05-22T15:26:49.200650+0000","last_active":"2023-05-22T15:26:53.617051+0000","last_peered":"2023-05-22T15:26:53.617051+0000","last_clean":"2023-05-22T15:26:53.617051+0000","last_became_active":"2023-05-22T15:26:49.199682+0000","last_became_peered":"2023-05-22T15:26:49.199682+0000","last_unstale":"2023-05-22T15:26:53.617051+0000","last_undegraded":"2023-05-22T15:26:53.617051+0000","last_fullsized":"2023-05-22T15:26:53.617051+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_clean_scrub_stamp":"2023-05-22T15:26:48.127413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:13:03.975127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-05-22T15:26:54.056477+0000","last_change":"2023-05-22T15:26:53.667847+0000","last_active":"2023-05-22T15:26:54.056477+0000","last_peered":"2023-05-22T15:26:54.056477+0000","last_clean":"2023-05-22T15:26:54.056477+0000","last_became_active":"2023-05-22T15:26:53.666866+0000","last_became_peered":"2023-05-22T15:26:53.666866+0000","last_unstale":"2023-05-22T15:26:54.056477+0000","last_undegraded":"2023-05-22T15:26:54.056477+0000","last_fullsized":"2023-05-22T15:26:54.056477+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_clean_scrub_stamp":"2023-05-22T15:26:52.568670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:30:40.980509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6,2],"acting":[5,4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.631889+0000","last_change":"2023-05-22T15:26:49.598177+0000","last_active":"2023-05-22T15:26:53.631889+0000","last_peered":"2023-05-22T15:26:53.631889+0000","last_clean":"2023-05-22T15:26:53.631889+0000","last_became_active":"2023-05-22T15:26:49.594848+0000","last_became_peered":"2023-05-22T15:26:49.594848+0000","last_unstale":"2023-05-22T15:26:53.631889+0000","last_undegraded":"2023-05-22T15:26:53.631889+0000","last_fullsized":"2023-05-22T15:26:53.631889+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T03:14:09.303640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.603009+0000","last_change":"2023-05-22T15:26:49.206461+0000","last_active":"2023-05-22T15:26:53.603009+0000","last_peered":"2023-05-22T15:26:53.603009+0000","last_clean":"2023-05-22T15:26:53.603009+0000","last_became_active":"2023-05-22T15:26:49.205870+0000","last_became_peered":"2023-05-22T15:26:49.205870+0000","last_unstale":"2023-05-22T15:26:53.603009+0000","last_undegraded":"2023-05-22T15:26:53.603009+0000","last_fullsized":"2023-05-22T15:26:53.603009+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_clean_scrub_stamp":"2023-05-22T15:26:48.127413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T18:13:58.291903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.980043+0000","last_change":"2023-05-22T15:26:53.642805+0000","last_active":"2023-05-22T15:26:53.980043+0000","last_peered":"2023-05-22T15:26:53.980043+0000","last_clean":"2023-05-22T15:26:53.980043+0000","last_became_active":"2023-05-22T15:26:53.642142+0000","last_became_peered":"2023-05-22T15:26:53.642142+0000","last_unstale":"2023-05-22T15:26:53.980043+0000","last_undegraded":"2023-05-22T15:26:53.980043+0000","last_fullsized":"2023-05-22T15:26:53.980043+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_clean_scrub_stamp":"2023-05-22T15:26:52.568670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:12:09.126657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0,6],"acting":[1,4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.623794+0000","last_change":"2023-05-22T15:26:49.600035+0000","last_active":"2023-05-22T15:26:53.623794+0000","last_peered":"2023-05-22T15:26:53.623794+0000","last_clean":"2023-05-22T15:26:53.623794+0000","last_became_active":"2023-05-22T15:26:49.597436+0000","last_became_peered":"2023-05-22T15:26:49.597436+0000","last_unstale":"2023-05-22T15:26:53.623794+0000","last_undegraded":"2023-05-22T15:26:53.623794+0000","last_fullsized":"2023-05-22T15:26:53.623794+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:46:50.307602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"22'2","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.606082+0000","last_change":"2023-05-22T15:26:49.191345+0000","last_active":"2023-05-22T15:26:53.606082+0000","last_peered":"2023-05-22T15:26:53.606082+0000","last_clean":"2023-05-22T15:26:53.606082+0000","last_became_active":"2023-05-22T15:26:49.190291+0000","last_became_peered":"2023-05-22T15:26:49.190291+0000","last_unstale":"2023-05-22T15:26:53.606082+0000","last_undegraded":"2023-05-22T15:26:53.606082+0000","last_fullsized":"2023-05-22T15:26:53.606082+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_clean_scrub_stamp":"2023-05-22T15:26:48.127413+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:13:21.177657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.979168+0000","last_change":"2023-05-22T15:26:53.649119+0000","last_active":"2023-05-22T15:26:53.979168+0000","last_peered":"2023-05-22T15:26:53.979168+0000","last_clean":"2023-05-22T15:26:53.979168+0000","last_became_active":"2023-05-22T15:26:53.648706+0000","last_became_peered":"2023-05-22T15:26:53.648706+0000","last_unstale":"2023-05-22T15:26:53.979168+0000","last_undegraded":"2023-05-22T15:26:53.979168+0000","last_fullsized":"2023-05-22T15:26:53.979168+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_clean_scrub_stamp":"2023-05-22T15:26:52.568670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T18:05:13.369385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5,6],"acting":[2,7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.611714+0000","last_change":"2023-05-22T15:26:49.581996+0000","last_active":"2023-05-22T15:26:53.611714+0000","last_peered":"2023-05-22T15:26:53.611714+0000","last_clean":"2023-05-22T15:26:53.611714+0000","last_became_active":"2023-05-22T15:26:49.581611+0000","last_became_peered":"2023-05-22T15:26:49.581611+0000","last_unstale":"2023-05-22T15:26:53.611714+0000","last_undegraded":"2023-05-22T15:26:53.611714+0000","last_fullsized":"2023-05-22T15:26:53.611714+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:44:52.369716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.634393+0000","last_change":"2023-05-22T15:26:49.192121+0000","last_active":"2023-05-22T15:26:53.634393+0000","last_peered":"2023-05-22T15:26:53.634393+0000","last_clean":"2023-05-22T15:26:53.634393+0000","last_became_active":"2023-05-22T15:26:49.191470+0000","last_became_peered":"2023-05-22T15:26:49.191470+0000","last_unstale":"2023-05-22T15:26:53.634393+0000","last_undegraded":"2023-05-22T15:26:53.634393+0000","last_fullsized":"2023-05-22T15:26:53.634393+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_clean_scrub_stamp":"2023-05-22T15:26:48.127413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:53:34.802286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.979064+0000","last_change":"2023-05-22T15:26:53.656554+0000","last_active":"2023-05-22T15:26:53.979064+0000","last_peered":"2023-05-22T15:26:53.979064+0000","last_clean":"2023-05-22T15:26:53.979064+0000","last_became_active":"2023-05-22T15:26:53.655950+0000","last_became_peered":"2023-05-22T15:26:53.655950+0000","last_unstale":"2023-05-22T15:26:53.979064+0000","last_undegraded":"2023-05-22T15:26:53.979064+0000","last_fullsized":"2023-05-22T15:26:53.979064+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_clean_scrub_stamp":"2023-05-22T15:26:52.568670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:11:45.590112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3,6],"acting":[2,4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.602652+0000","last_change":"2023-05-22T15:26:49.618498+0000","last_active":"2023-05-22T15:26:53.602652+0000","last_peered":"2023-05-22T15:26:53.602652+0000","last_clean":"2023-05-22T15:26:53.602652+0000","last_became_active":"2023-05-22T15:26:49.618068+0000","last_became_peered":"2023-05-22T15:26:49.618068+0000","last_unstale":"2023-05-22T15:26:53.602652+0000","last_undegraded":"2023-05-22T15:26:53.602652+0000","last_fullsized":"2023-05-22T15:26:53.602652+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:55:20.098659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"22'1","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.610411+0000","last_change":"2023-05-22T15:26:49.186875+0000","last_active":"2023-05-22T15:26:53.610411+0000","last_peered":"2023-05-22T15:26:53.610411+0000","last_clean":"2023-05-22T15:26:53.610411+0000","last_became_active":"2023-05-22T15:26:49.186431+0000","last_became_peered":"2023-05-22T15:26:49.186431+0000","last_unstale":"2023-05-22T15:26:53.610411+0000","last_undegraded":"2023-05-22T15:26:53.610411+0000","last_fullsized":"2023-05-22T15:26:53.610411+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_clean_scrub_stamp":"2023-05-22T15:26:48.127413+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:59:22.794782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.978460+0000","last_change":"2023-05-22T15:26:53.649077+0000","last_active":"2023-05-22T15:26:53.978460+0000","last_peered":"2023-05-22T15:26:53.978460+0000","last_clean":"2023-05-22T15:26:53.978460+0000","last_became_active":"2023-05-22T15:26:53.648415+0000","last_became_peered":"2023-05-22T15:26:53.648415+0000","last_unstale":"2023-05-22T15:26:53.978460+0000","last_undegraded":"2023-05-22T15:26:53.978460+0000","last_fullsized":"2023-05-22T15:26:53.978460+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_clean_scrub_stamp":"2023-05-22T15:26:52.568670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:33:26.787992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5,6],"acting":[2,0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.616451+0000","last_change":"2023-05-22T15:26:49.597065+0000","last_active":"2023-05-22T15:26:53.616451+0000","last_peered":"2023-05-22T15:26:53.616451+0000","last_clean":"2023-05-22T15:26:53.616451+0000","last_became_active":"2023-05-22T15:26:49.596334+0000","last_became_peered":"2023-05-22T15:26:49.596334+0000","last_unstale":"2023-05-22T15:26:53.616451+0000","last_undegraded":"2023-05-22T15:26:53.616451+0000","last_fullsized":"2023-05-22T15:26:53.616451+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:19:30.391062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"22'2","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.610463+0000","last_change":"2023-05-22T15:26:49.201296+0000","last_active":"2023-05-22T15:26:53.610463+0000","last_peered":"2023-05-22T15:26:53.610463+0000","last_clean":"2023-05-22T15:26:53.610463+0000","last_became_active":"2023-05-22T15:26:49.200707+0000","last_became_peered":"2023-05-22T15:26:49.200707+0000","last_unstale":"2023-05-22T15:26:53.610463+0000","last_undegraded":"2023-05-22T15:26:53.610463+0000","last_fullsized":"2023-05-22T15:26:53.610463+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_clean_scrub_stamp":"2023-05-22T15:26:48.127413+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:07:41.791660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.977970+0000","last_change":"2023-05-22T15:26:53.622351+0000","last_active":"2023-05-22T15:26:53.977970+0000","last_peered":"2023-05-22T15:26:53.977970+0000","last_clean":"2023-05-22T15:26:53.977970+0000","last_became_active":"2023-05-22T15:26:53.621754+0000","last_became_peered":"2023-05-22T15:26:53.621754+0000","last_unstale":"2023-05-22T15:26:53.977970+0000","last_undegraded":"2023-05-22T15:26:53.977970+0000","last_fullsized":"2023-05-22T15:26:53.977970+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_clean_scrub_stamp":"2023-05-22T15:26:52.568670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T16:31:40.104597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1,5],"acting":[2,0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.634315+0000","last_change":"2023-05-22T15:26:49.626518+0000","last_active":"2023-05-22T15:26:53.634315+0000","last_peered":"2023-05-22T15:26:53.634315+0000","last_clean":"2023-05-22T15:26:53.634315+0000","last_became_active":"2023-05-22T15:26:49.626026+0000","last_became_peered":"2023-05-22T15:26:49.626026+0000","last_unstale":"2023-05-22T15:26:53.634315+0000","last_undegraded":"2023-05-22T15:26:53.634315+0000","last_fullsized":"2023-05-22T15:26:53.634315+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:32:19.582542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"22'1","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.622234+0000","last_change":"2023-05-22T15:26:49.201252+0000","last_active":"2023-05-22T15:26:53.622234+0000","last_peered":"2023-05-22T15:26:53.622234+0000","last_clean":"2023-05-22T15:26:53.622234+0000","last_became_active":"2023-05-22T15:26:49.200748+0000","last_became_peered":"2023-05-22T15:26:49.200748+0000","last_unstale":"2023-05-22T15:26:53.622234+0000","last_undegraded":"2023-05-22T15:26:53.622234+0000","last_fullsized":"2023-05-22T15:26:53.622234+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_clean_scrub_stamp":"2023-05-22T15:26:48.127413+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:10:50.424396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.982126+0000","last_change":"2023-05-22T15:26:53.663379+0000","last_active":"2023-05-22T15:26:53.982126+0000","last_peered":"2023-05-22T15:26:53.982126+0000","last_clean":"2023-05-22T15:26:53.982126+0000","last_became_active":"2023-05-22T15:26:53.662170+0000","last_became_peered":"2023-05-22T15:26:53.662170+0000","last_unstale":"2023-05-22T15:26:53.982126+0000","last_undegraded":"2023-05-22T15:26:53.982126+0000","last_fullsized":"2023-05-22T15:26:53.982126+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_clean_scrub_stamp":"2023-05-22T15:26:52.568670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:38:53.647917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3,4],"acting":[7,6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-05-22T15:26:54.056358+0000","last_change":"2023-05-22T15:26:53.647770+0000","last_active":"2023-05-22T15:26:54.056358+0000","last_peered":"2023-05-22T15:26:54.056358+0000","last_clean":"2023-05-22T15:26:54.056358+0000","last_became_active":"2023-05-22T15:26:53.646585+0000","last_became_peered":"2023-05-22T15:26:53.646585+0000","last_unstale":"2023-05-22T15:26:54.056358+0000","last_undegraded":"2023-05-22T15:26:54.056358+0000","last_fullsized":"2023-05-22T15:26:54.056358+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_clean_scrub_stamp":"2023-05-22T15:26:52.568670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:48:44.211681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1,6],"acting":[5,3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"22'8","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.602514+0000","last_change":"2023-05-22T15:26:49.189894+0000","last_active":"2023-05-22T15:26:53.602514+0000","last_peered":"2023-05-22T15:26:53.602514+0000","last_clean":"2023-05-22T15:26:53.602514+0000","last_became_active":"2023-05-22T15:26:49.189156+0000","last_became_peered":"2023-05-22T15:26:49.189156+0000","last_unstale":"2023-05-22T15:26:53.602514+0000","last_undegraded":"2023-05-22T15:26:53.602514+0000","last_fullsized":"2023-05-22T15:26:53.602514+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_clean_scrub_stamp":"2023-05-22T15:26:48.127413+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:13:21.177657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.605587+0000","last_change":"2023-05-22T15:26:49.622540+0000","last_active":"2023-05-22T15:26:53.605587+0000","last_peered":"2023-05-22T15:26:53.605587+0000","last_clean":"2023-05-22T15:26:53.605587+0000","last_became_active":"2023-05-22T15:26:49.622187+0000","last_became_peered":"2023-05-22T15:26:49.622187+0000","last_unstale":"2023-05-22T15:26:53.605587+0000","last_undegraded":"2023-05-22T15:26:53.605587+0000","last_fullsized":"2023-05-22T15:26:53.605587+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T16:31:36.674451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.993571+0000","last_change":"2023-05-22T15:26:53.645178+0000","last_active":"2023-05-22T15:26:53.993571+0000","last_peered":"2023-05-22T15:26:53.993571+0000","last_clean":"2023-05-22T15:26:53.993571+0000","last_became_active":"2023-05-22T15:26:53.644770+0000","last_became_peered":"2023-05-22T15:26:53.644770+0000","last_unstale":"2023-05-22T15:26:53.993571+0000","last_undegraded":"2023-05-22T15:26:53.993571+0000","last_fullsized":"2023-05-22T15:26:53.993571+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_clean_scrub_stamp":"2023-05-22T15:26:52.568670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T15:46:20.544673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,0],"acting":[3,1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.628444+0000","last_change":"2023-05-22T15:26:49.191146+0000","last_active":"2023-05-22T15:26:53.628444+0000","last_peered":"2023-05-22T15:26:53.628444+0000","last_clean":"2023-05-22T15:26:53.628444+0000","last_became_active":"2023-05-22T15:26:49.190714+0000","last_became_peered":"2023-05-22T15:26:49.190714+0000","last_unstale":"2023-05-22T15:26:53.628444+0000","last_undegraded":"2023-05-22T15:26:53.628444+0000","last_fullsized":"2023-05-22T15:26:53.628444+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_clean_scrub_stamp":"2023-05-22T15:26:48.127413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:55:19.087548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.612090+0000","last_change":"2023-05-22T15:26:49.585916+0000","last_active":"2023-05-22T15:26:53.612090+0000","last_peered":"2023-05-22T15:26:53.612090+0000","last_clean":"2023-05-22T15:26:53.612090+0000","last_became_active":"2023-05-22T15:26:49.585261+0000","last_became_peered":"2023-05-22T15:26:49.585261+0000","last_unstale":"2023-05-22T15:26:53.612090+0000","last_undegraded":"2023-05-22T15:26:53.612090+0000","last_fullsized":"2023-05-22T15:26:53.612090+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:49:00.232177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.611499+0000","last_change":"2023-05-22T15:26:49.179340+0000","last_active":"2023-05-22T15:26:53.611499+0000","last_peered":"2023-05-22T15:26:53.611499+0000","last_clean":"2023-05-22T15:26:53.611499+0000","last_became_active":"2023-05-22T15:26:49.178875+0000","last_became_peered":"2023-05-22T15:26:49.178875+0000","last_unstale":"2023-05-22T15:26:53.611499+0000","last_undegraded":"2023-05-22T15:26:53.611499+0000","last_fullsized":"2023-05-22T15:26:53.611499+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_clean_scrub_stamp":"2023-05-22T15:26:48.127413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T16:16:39.277093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.634708+0000","last_change":"2023-05-22T15:26:49.623344+0000","last_active":"2023-05-22T15:26:53.634708+0000","last_peered":"2023-05-22T15:26:53.634708+0000","last_clean":"2023-05-22T15:26:53.634708+0000","last_became_active":"2023-05-22T15:26:49.622700+0000","last_became_peered":"2023-05-22T15:26:49.622700+0000","last_unstale":"2023-05-22T15:26:53.634708+0000","last_undegraded":"2023-05-22T15:26:53.634708+0000","last_fullsized":"2023-05-22T15:26:53.634708+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:22:10.117410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.993134+0000","last_change":"2023-05-22T15:26:53.646185+0000","last_active":"2023-05-22T15:26:53.993134+0000","last_peered":"2023-05-22T15:26:53.993134+0000","last_clean":"2023-05-22T15:26:53.993134+0000","last_became_active":"2023-05-22T15:26:53.645789+0000","last_became_peered":"2023-05-22T15:26:53.645789+0000","last_unstale":"2023-05-22T15:26:53.993134+0000","last_undegraded":"2023-05-22T15:26:53.993134+0000","last_fullsized":"2023-05-22T15:26:53.993134+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_clean_scrub_stamp":"2023-05-22T15:26:52.568670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:30:40.980509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2,1],"acting":[3,7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"22'2","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.608324+0000","last_change":"2023-05-22T15:26:49.201586+0000","last_active":"2023-05-22T15:26:53.608324+0000","last_peered":"2023-05-22T15:26:53.608324+0000","last_clean":"2023-05-22T15:26:53.608324+0000","last_became_active":"2023-05-22T15:26:49.201013+0000","last_became_peered":"2023-05-22T15:26:49.201013+0000","last_unstale":"2023-05-22T15:26:53.608324+0000","last_undegraded":"2023-05-22T15:26:53.608324+0000","last_fullsized":"2023-05-22T15:26:53.608324+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:48.127413+0000","last_clean_scrub_stamp":"2023-05-22T15:26:48.127413+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:47:17.588354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.607130+0000","last_change":"2023-05-22T15:26:49.577112+0000","last_active":"2023-05-22T15:26:53.607130+0000","last_peered":"2023-05-22T15:26:53.607130+0000","last_clean":"2023-05-22T15:26:53.607130+0000","last_became_active":"2023-05-22T15:26:49.576802+0000","last_became_peered":"2023-05-22T15:26:49.576802+0000","last_unstale":"2023-05-22T15:26:53.607130+0000","last_undegraded":"2023-05-22T15:26:53.607130+0000","last_fullsized":"2023-05-22T15:26:53.607130+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:52:48.123148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-05-22T15:26:54.056338+0000","last_change":"2023-05-22T15:26:53.655225+0000","last_active":"2023-05-22T15:26:54.056338+0000","last_peered":"2023-05-22T15:26:54.056338+0000","last_clean":"2023-05-22T15:26:54.056338+0000","last_became_active":"2023-05-22T15:26:53.648610+0000","last_became_peered":"2023-05-22T15:26:53.648610+0000","last_unstale":"2023-05-22T15:26:54.056338+0000","last_undegraded":"2023-05-22T15:26:54.056338+0000","last_fullsized":"2023-05-22T15:26:54.056338+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:52.568670+0000","last_clean_scrub_stamp":"2023-05-22T15:26:52.568670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T15:46:20.544673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3,0],"acting":[5,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.20","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.602276+0000","last_change":"2023-05-22T15:26:49.614417+0000","last_active":"2023-05-22T15:26:53.602276+0000","last_peered":"2023-05-22T15:26:53.602276+0000","last_clean":"2023-05-22T15:26:53.602276+0000","last_became_active":"2023-05-22T15:26:49.610439+0000","last_became_peered":"2023-05-22T15:26:49.610439+0000","last_unstale":"2023-05-22T15:26:53.602276+0000","last_undegraded":"2023-05-22T15:26:53.602276+0000","last_fullsized":"2023-05-22T15:26:53.602276+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:07:42.802772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.21","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.603531+0000","last_change":"2023-05-22T15:26:49.577132+0000","last_active":"2023-05-22T15:26:53.603531+0000","last_peered":"2023-05-22T15:26:53.603531+0000","last_clean":"2023-05-22T15:26:53.603531+0000","last_became_active":"2023-05-22T15:26:49.576859+0000","last_became_peered":"2023-05-22T15:26:49.576859+0000","last_unstale":"2023-05-22T15:26:53.603531+0000","last_undegraded":"2023-05-22T15:26:53.603531+0000","last_fullsized":"2023-05-22T15:26:53.603531+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:08:07.637726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.22","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.619920+0000","last_change":"2023-05-22T15:26:49.622334+0000","last_active":"2023-05-22T15:26:53.619920+0000","last_peered":"2023-05-22T15:26:53.619920+0000","last_clean":"2023-05-22T15:26:53.619920+0000","last_became_active":"2023-05-22T15:26:49.621648+0000","last_became_peered":"2023-05-22T15:26:49.621648+0000","last_unstale":"2023-05-22T15:26:53.619920+0000","last_undegraded":"2023-05-22T15:26:53.619920+0000","last_fullsized":"2023-05-22T15:26:53.619920+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:57:58.007680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.23","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.605756+0000","last_change":"2023-05-22T15:26:49.612753+0000","last_active":"2023-05-22T15:26:53.605756+0000","last_peered":"2023-05-22T15:26:53.605756+0000","last_clean":"2023-05-22T15:26:53.605756+0000","last_became_active":"2023-05-22T15:26:49.612495+0000","last_became_peered":"2023-05-22T15:26:49.612495+0000","last_unstale":"2023-05-22T15:26:53.605756+0000","last_undegraded":"2023-05-22T15:26:53.605756+0000","last_fullsized":"2023-05-22T15:26:53.605756+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:11:51.327926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.24","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.619231+0000","last_change":"2023-05-22T15:26:49.627768+0000","last_active":"2023-05-22T15:26:53.619231+0000","last_peered":"2023-05-22T15:26:53.619231+0000","last_clean":"2023-05-22T15:26:53.619231+0000","last_became_active":"2023-05-22T15:26:49.627340+0000","last_became_peered":"2023-05-22T15:26:49.627340+0000","last_unstale":"2023-05-22T15:26:53.619231+0000","last_undegraded":"2023-05-22T15:26:53.619231+0000","last_fullsized":"2023-05-22T15:26:53.619231+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:44:52.369716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.25","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.634508+0000","last_change":"2023-05-22T15:26:49.613627+0000","last_active":"2023-05-22T15:26:53.634508+0000","last_peered":"2023-05-22T15:26:53.634508+0000","last_clean":"2023-05-22T15:26:53.634508+0000","last_became_active":"2023-05-22T15:26:49.612997+0000","last_became_peered":"2023-05-22T15:26:49.612997+0000","last_unstale":"2023-05-22T15:26:53.634508+0000","last_undegraded":"2023-05-22T15:26:53.634508+0000","last_fullsized":"2023-05-22T15:26:53.634508+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:11:51.327926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.26","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.612444+0000","last_change":"2023-05-22T15:26:49.581772+0000","last_active":"2023-05-22T15:26:53.612444+0000","last_peered":"2023-05-22T15:26:53.612444+0000","last_clean":"2023-05-22T15:26:53.612444+0000","last_became_active":"2023-05-22T15:26:49.581493+0000","last_became_peered":"2023-05-22T15:26:49.581493+0000","last_unstale":"2023-05-22T15:26:53.612444+0000","last_undegraded":"2023-05-22T15:26:53.612444+0000","last_fullsized":"2023-05-22T15:26:53.612444+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:49:00.232177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.27","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.612724+0000","last_change":"2023-05-22T15:26:49.619433+0000","last_active":"2023-05-22T15:26:53.612724+0000","last_peered":"2023-05-22T15:26:53.612724+0000","last_clean":"2023-05-22T15:26:53.612724+0000","last_became_active":"2023-05-22T15:26:49.619107+0000","last_became_peered":"2023-05-22T15:26:49.619107+0000","last_unstale":"2023-05-22T15:26:53.612724+0000","last_undegraded":"2023-05-22T15:26:53.612724+0000","last_fullsized":"2023-05-22T15:26:53.612724+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:11:51.327926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.28","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.609649+0000","last_change":"2023-05-22T15:26:49.624224+0000","last_active":"2023-05-22T15:26:53.609649+0000","last_peered":"2023-05-22T15:26:53.609649+0000","last_clean":"2023-05-22T15:26:53.609649+0000","last_became_active":"2023-05-22T15:26:49.623023+0000","last_became_peered":"2023-05-22T15:26:49.623023+0000","last_unstale":"2023-05-22T15:26:53.609649+0000","last_undegraded":"2023-05-22T15:26:53.609649+0000","last_fullsized":"2023-05-22T15:26:53.609649+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:32:19.582542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.29","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.603353+0000","last_change":"2023-05-22T15:26:49.573154+0000","last_active":"2023-05-22T15:26:53.603353+0000","last_peered":"2023-05-22T15:26:53.603353+0000","last_clean":"2023-05-22T15:26:53.603353+0000","last_became_active":"2023-05-22T15:26:49.572881+0000","last_became_peered":"2023-05-22T15:26:49.572881+0000","last_unstale":"2023-05-22T15:26:53.603353+0000","last_undegraded":"2023-05-22T15:26:53.603353+0000","last_fullsized":"2023-05-22T15:26:53.603353+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:32:19.582542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2a","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.611407+0000","last_change":"2023-05-22T15:26:49.581918+0000","last_active":"2023-05-22T15:26:53.611407+0000","last_peered":"2023-05-22T15:26:53.611407+0000","last_clean":"2023-05-22T15:26:53.611407+0000","last_became_active":"2023-05-22T15:26:49.581462+0000","last_became_peered":"2023-05-22T15:26:49.581462+0000","last_unstale":"2023-05-22T15:26:53.611407+0000","last_undegraded":"2023-05-22T15:26:53.611407+0000","last_fullsized":"2023-05-22T15:26:53.611407+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:19:30.391062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2b","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.608076+0000","last_change":"2023-05-22T15:26:49.624711+0000","last_active":"2023-05-22T15:26:53.608076+0000","last_peered":"2023-05-22T15:26:53.608076+0000","last_clean":"2023-05-22T15:26:53.608076+0000","last_became_active":"2023-05-22T15:26:49.624180+0000","last_became_peered":"2023-05-22T15:26:49.624180+0000","last_unstale":"2023-05-22T15:26:53.608076+0000","last_undegraded":"2023-05-22T15:26:53.608076+0000","last_fullsized":"2023-05-22T15:26:53.608076+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:43:00.518715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2c","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.623768+0000","last_change":"2023-05-22T15:26:49.620689+0000","last_active":"2023-05-22T15:26:53.623768+0000","last_peered":"2023-05-22T15:26:53.623768+0000","last_clean":"2023-05-22T15:26:53.623768+0000","last_became_active":"2023-05-22T15:26:49.620183+0000","last_became_peered":"2023-05-22T15:26:49.620183+0000","last_unstale":"2023-05-22T15:26:53.623768+0000","last_undegraded":"2023-05-22T15:26:53.623768+0000","last_fullsized":"2023-05-22T15:26:53.623768+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:47:22.628767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2d","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.601734+0000","last_change":"2023-05-22T15:26:49.618556+0000","last_active":"2023-05-22T15:26:53.601734+0000","last_peered":"2023-05-22T15:26:53.601734+0000","last_clean":"2023-05-22T15:26:53.601734+0000","last_became_active":"2023-05-22T15:26:49.618215+0000","last_became_peered":"2023-05-22T15:26:49.618215+0000","last_unstale":"2023-05-22T15:26:53.601734+0000","last_undegraded":"2023-05-22T15:26:53.601734+0000","last_fullsized":"2023-05-22T15:26:53.601734+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:07:50.944971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2e","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.621273+0000","last_change":"2023-05-22T15:26:49.597573+0000","last_active":"2023-05-22T15:26:53.621273+0000","last_peered":"2023-05-22T15:26:53.621273+0000","last_clean":"2023-05-22T15:26:53.621273+0000","last_became_active":"2023-05-22T15:26:49.596607+0000","last_became_peered":"2023-05-22T15:26:49.596607+0000","last_unstale":"2023-05-22T15:26:53.621273+0000","last_undegraded":"2023-05-22T15:26:53.621273+0000","last_fullsized":"2023-05-22T15:26:53.621273+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:31:03.890085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2f","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.611892+0000","last_change":"2023-05-22T15:26:49.578237+0000","last_active":"2023-05-22T15:26:53.611892+0000","last_peered":"2023-05-22T15:26:53.611892+0000","last_clean":"2023-05-22T15:26:53.611892+0000","last_became_active":"2023-05-22T15:26:49.577927+0000","last_became_peered":"2023-05-22T15:26:49.577927+0000","last_unstale":"2023-05-22T15:26:53.611892+0000","last_undegraded":"2023-05-22T15:26:53.611892+0000","last_fullsized":"2023-05-22T15:26:53.611892+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:11:51.327926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.30","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.612647+0000","last_change":"2023-05-22T15:26:49.598449+0000","last_active":"2023-05-22T15:26:53.612647+0000","last_peered":"2023-05-22T15:26:53.612647+0000","last_clean":"2023-05-22T15:26:53.612647+0000","last_became_active":"2023-05-22T15:26:49.597652+0000","last_became_peered":"2023-05-22T15:26:49.597652+0000","last_unstale":"2023-05-22T15:26:53.612647+0000","last_undegraded":"2023-05-22T15:26:53.612647+0000","last_fullsized":"2023-05-22T15:26:53.612647+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:48:33.611833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.31","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.628230+0000","last_change":"2023-05-22T15:26:49.625943+0000","last_active":"2023-05-22T15:26:53.628230+0000","last_peered":"2023-05-22T15:26:53.628230+0000","last_clean":"2023-05-22T15:26:53.628230+0000","last_became_active":"2023-05-22T15:26:49.624306+0000","last_became_peered":"2023-05-22T15:26:49.624306+0000","last_unstale":"2023-05-22T15:26:53.628230+0000","last_undegraded":"2023-05-22T15:26:53.628230+0000","last_fullsized":"2023-05-22T15:26:53.628230+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:44:52.369716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.32","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.634913+0000","last_change":"2023-05-22T15:26:49.563010+0000","last_active":"2023-05-22T15:26:53.634913+0000","last_peered":"2023-05-22T15:26:53.634913+0000","last_clean":"2023-05-22T15:26:53.634913+0000","last_became_active":"2023-05-22T15:26:49.562210+0000","last_became_peered":"2023-05-22T15:26:49.562210+0000","last_unstale":"2023-05-22T15:26:53.634913+0000","last_undegraded":"2023-05-22T15:26:53.634913+0000","last_fullsized":"2023-05-22T15:26:53.634913+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:19:30.391062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.33","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.600262+0000","last_change":"2023-05-22T15:26:49.562260+0000","last_active":"2023-05-22T15:26:53.600262+0000","last_peered":"2023-05-22T15:26:53.600262+0000","last_clean":"2023-05-22T15:26:53.600262+0000","last_became_active":"2023-05-22T15:26:49.561965+0000","last_became_peered":"2023-05-22T15:26:49.561965+0000","last_unstale":"2023-05-22T15:26:53.600262+0000","last_undegraded":"2023-05-22T15:26:53.600262+0000","last_fullsized":"2023-05-22T15:26:53.600262+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:59:55.020950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.34","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.598383+0000","last_change":"2023-05-22T15:26:49.601428+0000","last_active":"2023-05-22T15:26:53.598383+0000","last_peered":"2023-05-22T15:26:53.598383+0000","last_clean":"2023-05-22T15:26:53.598383+0000","last_became_active":"2023-05-22T15:26:49.601157+0000","last_became_peered":"2023-05-22T15:26:49.601157+0000","last_unstale":"2023-05-22T15:26:53.598383+0000","last_undegraded":"2023-05-22T15:26:53.598383+0000","last_fullsized":"2023-05-22T15:26:53.598383+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:47:18.599465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.35","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.631671+0000","last_change":"2023-05-22T15:26:49.598414+0000","last_active":"2023-05-22T15:26:53.631671+0000","last_peered":"2023-05-22T15:26:53.631671+0000","last_clean":"2023-05-22T15:26:53.631671+0000","last_became_active":"2023-05-22T15:26:49.595631+0000","last_became_peered":"2023-05-22T15:26:49.595631+0000","last_unstale":"2023-05-22T15:26:53.631671+0000","last_undegraded":"2023-05-22T15:26:53.631671+0000","last_fullsized":"2023-05-22T15:26:53.631671+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:11:51.327926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.36","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.635151+0000","last_change":"2023-05-22T15:26:49.580957+0000","last_active":"2023-05-22T15:26:53.635151+0000","last_peered":"2023-05-22T15:26:53.635151+0000","last_clean":"2023-05-22T15:26:53.635151+0000","last_became_active":"2023-05-22T15:26:49.580172+0000","last_became_peered":"2023-05-22T15:26:49.580172+0000","last_unstale":"2023-05-22T15:26:53.635151+0000","last_undegraded":"2023-05-22T15:26:53.635151+0000","last_fullsized":"2023-05-22T15:26:53.635151+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:43:00.518715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.37","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.615519+0000","last_change":"2023-05-22T15:26:49.587437+0000","last_active":"2023-05-22T15:26:53.615519+0000","last_peered":"2023-05-22T15:26:53.615519+0000","last_clean":"2023-05-22T15:26:53.615519+0000","last_became_active":"2023-05-22T15:26:49.587019+0000","last_became_peered":"2023-05-22T15:26:49.587019+0000","last_unstale":"2023-05-22T15:26:53.615519+0000","last_undegraded":"2023-05-22T15:26:53.615519+0000","last_fullsized":"2023-05-22T15:26:53.615519+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:48:33.611833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.38","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.634076+0000","last_change":"2023-05-22T15:26:49.599720+0000","last_active":"2023-05-22T15:26:53.634076+0000","last_peered":"2023-05-22T15:26:53.634076+0000","last_clean":"2023-05-22T15:26:53.634076+0000","last_became_active":"2023-05-22T15:26:49.599079+0000","last_became_peered":"2023-05-22T15:26:49.599079+0000","last_unstale":"2023-05-22T15:26:53.634076+0000","last_undegraded":"2023-05-22T15:26:53.634076+0000","last_fullsized":"2023-05-22T15:26:53.634076+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:48:33.611833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.39","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.628016+0000","last_change":"2023-05-22T15:26:49.624934+0000","last_active":"2023-05-22T15:26:53.628016+0000","last_peered":"2023-05-22T15:26:53.628016+0000","last_clean":"2023-05-22T15:26:53.628016+0000","last_became_active":"2023-05-22T15:26:49.623557+0000","last_became_peered":"2023-05-22T15:26:49.623557+0000","last_unstale":"2023-05-22T15:26:53.628016+0000","last_undegraded":"2023-05-22T15:26:53.628016+0000","last_fullsized":"2023-05-22T15:26:53.628016+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:19:30.391062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3a","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.619318+0000","last_change":"2023-05-22T15:26:49.596704+0000","last_active":"2023-05-22T15:26:53.619318+0000","last_peered":"2023-05-22T15:26:53.619318+0000","last_clean":"2023-05-22T15:26:53.619318+0000","last_became_active":"2023-05-22T15:26:49.595941+0000","last_became_peered":"2023-05-22T15:26:49.595941+0000","last_unstale":"2023-05-22T15:26:53.619318+0000","last_undegraded":"2023-05-22T15:26:53.619318+0000","last_fullsized":"2023-05-22T15:26:53.619318+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T03:17:59.774769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3b","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.611996+0000","last_change":"2023-05-22T15:26:49.585396+0000","last_active":"2023-05-22T15:26:53.611996+0000","last_peered":"2023-05-22T15:26:53.611996+0000","last_clean":"2023-05-22T15:26:53.611996+0000","last_became_active":"2023-05-22T15:26:49.585110+0000","last_became_peered":"2023-05-22T15:26:49.585110+0000","last_unstale":"2023-05-22T15:26:53.611996+0000","last_undegraded":"2023-05-22T15:26:53.611996+0000","last_fullsized":"2023-05-22T15:26:53.611996+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:22:10.117410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3c","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.602523+0000","last_change":"2023-05-22T15:26:49.607125+0000","last_active":"2023-05-22T15:26:53.602523+0000","last_peered":"2023-05-22T15:26:53.602523+0000","last_clean":"2023-05-22T15:26:53.602523+0000","last_became_active":"2023-05-22T15:26:49.606510+0000","last_became_peered":"2023-05-22T15:26:49.606510+0000","last_unstale":"2023-05-22T15:26:53.602523+0000","last_undegraded":"2023-05-22T15:26:53.602523+0000","last_fullsized":"2023-05-22T15:26:53.602523+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:47:18.599465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3d","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.631452+0000","last_change":"2023-05-22T15:26:49.600011+0000","last_active":"2023-05-22T15:26:53.631452+0000","last_peered":"2023-05-22T15:26:53.631452+0000","last_clean":"2023-05-22T15:26:53.631452+0000","last_became_active":"2023-05-22T15:26:49.596414+0000","last_became_peered":"2023-05-22T15:26:49.596414+0000","last_unstale":"2023-05-22T15:26:53.631452+0000","last_undegraded":"2023-05-22T15:26:53.631452+0000","last_fullsized":"2023-05-22T15:26:53.631452+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T16:31:36.674451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3e","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.634607+0000","last_change":"2023-05-22T15:26:49.580509+0000","last_active":"2023-05-22T15:26:53.634607+0000","last_peered":"2023-05-22T15:26:53.634607+0000","last_clean":"2023-05-22T15:26:53.634607+0000","last_became_active":"2023-05-22T15:26:49.579662+0000","last_became_peered":"2023-05-22T15:26:49.579662+0000","last_unstale":"2023-05-22T15:26:53.634607+0000","last_undegraded":"2023-05-22T15:26:53.634607+0000","last_fullsized":"2023-05-22T15:26:53.634607+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:44:52.369716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3f","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-05-22T15:26:53.599109+0000","last_change":"2023-05-22T15:26:49.561088+0000","last_active":"2023-05-22T15:26:53.599109+0000","last_peered":"2023-05-22T15:26:53.599109+0000","last_clean":"2023-05-22T15:26:53.599109+0000","last_became_active":"2023-05-22T15:26:49.560859+0000","last_became_peered":"2023-05-22T15:26:49.560859+0000","last_unstale":"2023-05-22T15:26:53.599109+0000","last_undegraded":"2023-05-22T15:26:53.599109+0000","last_fullsized":"2023-05-22T15:26:53.599109+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:49.138525+0000","last_clean_scrub_stamp":"2023-05-22T15:26:49.138525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:53:35.813397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":4527,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":9054,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574857,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9284,"kb_used_data":892,"kb_used_omap":0,"kb_used_meta":8384,"kb_avail":94362556,"statfs":{"total":96636764160,"available":96627257344,"internally_reserved":0,"allocated":913408,"data_stored":714230,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8585216},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1399999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88400000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72399999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77000000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98999999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2669999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.21}]}]},{"osd":6,"up_from":13,"seq":55834574858,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8100,"kb_used_data":284,"kb_used_omap":0,"kb_used_meta":7808,"kb_avail":94363740,"statfs":{"total":96636764160,"available":96628469760,"internally_reserved":0,"allocated":290816,"data_stored":119901,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7995392},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.874}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58899999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68799999999999994}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63900000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1659999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95099999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.196}]}]},{"osd":5,"up_from":13,"seq":55834574858,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8172,"kb_used_data":292,"kb_used_omap":0,"kb_used_meta":7872,"kb_avail":94363668,"statfs":{"total":96636764160,"available":96628396032,"internally_reserved":0,"allocated":299008,"data_stored":120537,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8060928},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65000000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0309999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58499999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.75600000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1279999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0089999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97899999999999998}]}]},{"osd":4,"up_from":13,"seq":55834574856,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8248,"kb_used_data":304,"kb_used_omap":0,"kb_used_meta":7936,"kb_avail":94363592,"statfs":{"total":96636764160,"available":96628318208,"internally_reserved":0,"allocated":311296,"data_stored":123702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8126464},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69399999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60399999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82499999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79100000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97599999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0780000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95199999999999996}]}]},{"osd":3,"up_from":13,"seq":55834574856,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8168,"kb_used_data":288,"kb_used_omap":0,"kb_used_meta":7872,"kb_avail":94363672,"statfs":{"total":96636764160,"available":96628400128,"internally_reserved":0,"allocated":294912,"data_stored":119991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8060928},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53400000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46899999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64000000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93100000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.747}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90900000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.133}]}]},{"osd":2,"up_from":13,"seq":55834574857,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":12196,"kb_used_data":284,"kb_used_omap":0,"kb_used_meta":11904,"kb_avail":94359644,"statfs":{"total":96636764160,"available":96624275456,"internally_reserved":0,"allocated":290816,"data_stored":119901,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":12189696},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52300000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72399999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67800000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95199999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.167}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.135}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2430000000000001}]}]},{"osd":1,"up_from":13,"seq":55834574857,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8164,"kb_used_data":284,"kb_used_omap":0,"kb_used_meta":7872,"kb_avail":94363676,"statfs":{"total":96636764160,"available":96628404224,"internally_reserved":0,"allocated":290816,"data_stored":119901,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8060928},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58499999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70299999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53400000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80000000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81899999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97799999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.026}]}]},{"osd":0,"up_from":13,"seq":55834574858,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":13356,"kb_used_data":868,"kb_used_omap":0,"kb_used_meta":12480,"kb_avail":94358484,"statfs":{"total":96636764160,"available":96623087616,"internally_reserved":0,"allocated":888832,"data_stored":710835,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":12779520},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57699999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73099999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.873}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1879999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.129}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0389999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2310000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":3801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":3961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-05-22T15:27:11.696 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-05-22T15:27:11.696 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2023-05-22T15:27:11.696 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2023-05-22T15:27:11.697 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-05-22T15:27:11.848 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:11.838+0000 7f83ce1aa640 1 Processor -- start 2023-05-22T15:27:11.848 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:11.838+0000 7f83ce1aa640 1 -- start start 2023-05-22T15:27:11.850 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:11.838+0000 7f83ce1aa640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f83c80ca4e0 0x7f83c815e460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:11.850 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:11.838+0000 7f83ce1aa640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f83c8070df0 0x7f83c8071210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:11.850 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:11.838+0000 7f83ce1aa640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83c80b0650 con 0x7f83c80c9b40 2023-05-22T15:27:11.850 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:11.838+0000 7f83ce1aa640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83c80b0370 con 0x7f83c8070df0 2023-05-22T15:27:11.850 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:11.838+0000 7f83ce1aa640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83c815d980 con 0x7f83c80ca4e0 2023-05-22T15:27:11.851 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:11.838+0000 7f83c7fff640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f83c8070df0 0x7f83c8071210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:11.851 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:11.838+0000 7f83c7fff640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f83c8070df0 0x7f83c8071210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37846/0 (socket says 172.21.15.16:37846) 2023-05-22T15:27:11.851 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:11.838+0000 7f83c7fff640 1 -- 172.21.15.16:0/1035576755 learned_addr learned my addr 172.21.15.16:0/1035576755 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:11.851 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:11.838+0000 7f83c77fe640 1 --2- 172.21.15.16:0/1035576755 >> v2:172.21.15.62:3300/0 conn(0x7f83c80ca4e0 0x7f83c815e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:11.851 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:11.842+0000 7f83c7fff640 1 -- 172.21.15.16:0/1035576755 >> v2:172.21.15.62:3300/0 conn(0x7f83c80ca4e0 msgr2=0x7f83c815e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:11.851 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:11.842+0000 7f83c7fff640 1 --2- 172.21.15.16:0/1035576755 >> v2:172.21.15.62:3300/0 conn(0x7f83c80ca4e0 0x7f83c815e460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:11.851 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:11.842+0000 7f83c7fff640 1 -- 172.21.15.16:0/1035576755 >> v1:172.21.15.16:6789/0 conn(0x7f83c80c9b40 legacy=0x7f83c80afb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:11.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:11.842+0000 7f83c7fff640 1 -- 172.21.15.16:0/1035576755 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83c815dc60 con 0x7f83c8070df0 2023-05-22T15:27:11.853 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:11.842+0000 7f83c7fff640 1 --2- 172.21.15.16:0/1035576755 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83c8070df0 0x7f83c8071210 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f83c80cad40 tx=0x7f83c001c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=280753dbd2083c4e server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:11.853 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:11.842+0000 7f83c67fc640 1 -- 172.21.15.16:0/1035576755 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83c00a7070 con 0x7f83c8070df0 2023-05-22T15:27:11.853 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:11.842+0000 7f83c67fc640 1 -- 172.21.15.16:0/1035576755 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83c00ab430 con 0x7f83c8070df0 2023-05-22T15:27:11.853 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:11.842+0000 7f83c67fc640 1 -- 172.21.15.16:0/1035576755 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83c00ab7c0 con 0x7f83c8070df0 2023-05-22T15:27:11.853 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:11.842+0000 7f83ce1aa640 1 -- 172.21.15.16:0/1035576755 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83c8070df0 msgr2=0x7f83c8071210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:11.853 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:11.842+0000 7f83ce1aa640 1 --2- 172.21.15.16:0/1035576755 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83c8070df0 0x7f83c8071210 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f83c80cad40 tx=0x7f83c001c4d0 comp rx=0 tx=0).stop 2023-05-22T15:27:11.854 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:11.842+0000 7f83ce1aa640 1 -- 172.21.15.16:0/1035576755 shutdown_connections 2023-05-22T15:27:11.854 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:11.842+0000 7f83ce1aa640 1 --2- 172.21.15.16:0/1035576755 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83c8070df0 0x7f83c8071210 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:11.854 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:11.842+0000 7f83ce1aa640 1 --2- 172.21.15.16:0/1035576755 >> v2:172.21.15.62:3300/0 conn(0x7f83c80ca4e0 0x7f83c815e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:11.854 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:11.842+0000 7f83ce1aa640 1 -- 172.21.15.16:0/1035576755 >> 172.21.15.16:0/1035576755 conn(0x7f83c8069720 msgr2=0x7f83c8069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:11.855 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:11.842+0000 7f83ce1aa640 1 -- 172.21.15.16:0/1035576755 shutdown_connections 2023-05-22T15:27:11.855 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:11.842+0000 7f83ce1aa640 1 -- 172.21.15.16:0/1035576755 wait complete. 2023-05-22T15:27:11.857 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:11.842+0000 7f83ce1aa640 1 Processor -- start 2023-05-22T15:27:11.857 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:11.842+0000 7f83ce1aa640 1 -- start start 2023-05-22T15:27:11.857 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:11.846+0000 7f83ce1aa640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83c8070df0 0x7f83c814c400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:11.857 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:11.846+0000 7f83ce1aa640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f83c80c9b40 0x7f83c8156dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:11.857 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:11.846+0000 7f83ce1aa640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f83c80ca4e0 0x7f83c80aef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:11.858 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:11.846+0000 7f83ce1aa640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83c814c940 con 0x7f83c8070df0 2023-05-22T15:27:11.858 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:11.846+0000 7f83ce1aa640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83c80c6860 con 0x7f83c80c9b40 2023-05-22T15:27:11.858 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:11.846+0000 7f83ce1aa640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83c81577b0 con 0x7f83c80ca4e0 2023-05-22T15:27:11.858 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:11.846+0000 7f83c77fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83c8070df0 0x7f83c814c400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:11.858 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:11.846+0000 7f83c77fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83c8070df0 0x7f83c814c400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37848/0 (socket says 172.21.15.16:37848) 2023-05-22T15:27:11.858 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:11.846+0000 7f83c77fe640 1 -- 172.21.15.16:0/2379951645 learned_addr learned my addr 172.21.15.16:0/2379951645 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:11.858 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:11.846+0000 7f83c77fe640 1 -- 172.21.15.16:0/2379951645 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f83c80ca4e0 msgr2=0x7f83c80aef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:11.859 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:11.846+0000 7f83c77fe640 1 --2- 172.21.15.16:0/2379951645 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f83c80ca4e0 0x7f83c80aef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:11.859 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:11.846+0000 7f83c77fe640 1 -- 172.21.15.16:0/2379951645 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f83c80c9b40 msgr2=0x7f83c8156dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:11.859 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:11.846+0000 7f83c77fe640 1 --2- 172.21.15.16:0/2379951645 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f83c80c9b40 0x7f83c8156dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:11.859 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:11.846+0000 7f83c77fe640 1 -- 172.21.15.16:0/2379951645 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83c00094d0 con 0x7f83c8070df0 2023-05-22T15:27:11.859 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:11.846+0000 7f83c77fe640 1 --2- 172.21.15.16:0/2379951645 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83c8070df0 0x7f83c814c400 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f83b000d5f0 tx=0x7f83b000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:11.859 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:11.846+0000 7f83bffff640 1 -- 172.21.15.16:0/2379951645 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83b00125f0 con 0x7f83c8070df0 2023-05-22T15:27:11.859 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:11.846+0000 7f83bffff640 1 -- 172.21.15.16:0/2379951645 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83b0012750 con 0x7f83c8070df0 2023-05-22T15:27:11.860 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:11.846+0000 7f83ce1aa640 1 -- 172.21.15.16:0/2379951645 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83c80c8e80 con 0x7f83c8070df0 2023-05-22T15:27:11.860 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:11.846+0000 7f83bffff640 1 -- 172.21.15.16:0/2379951645 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83b0012ae0 con 0x7f83c8070df0 2023-05-22T15:27:11.860 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:11.846+0000 7f83ce1aa640 1 -- 172.21.15.16:0/2379951645 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83c80ca8b0 con 0x7f83c8070df0 2023-05-22T15:27:11.860 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:11.846+0000 7f83ce1aa640 1 -- 172.21.15.16:0/2379951645 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83c80c9010 con 0x7f83c8070df0 2023-05-22T15:27:11.860 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:11.846+0000 7f83bffff640 1 -- 172.21.15.16:0/2379951645 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7f83b0012c40 con 0x7f83c8070df0 2023-05-22T15:27:11.860 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:11.846+0000 7f83bffff640 1 --2- 172.21.15.16:0/2379951645 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f83a8065380 0x7f83a8067830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:11.860 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:11.846+0000 7f83bffff640 1 -- 172.21.15.16:0/2379951645 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f83b00830c0 con 0x7f83c8070df0 2023-05-22T15:27:11.861 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:11.846+0000 7f83c6ffd640 1 --2- 172.21.15.16:0/2379951645 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f83a8065380 0x7f83a8067830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:11.861 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:11.846+0000 7f83c6ffd640 1 --2- 172.21.15.16:0/2379951645 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f83a8065380 0x7f83a8067830 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f83b8007fd0 tx=0x7f83b8007d40 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:11.861 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:11.850+0000 7f83bffff640 1 -- 172.21.15.16:0/2379951645 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f83b00509b0 con 0x7f83c8070df0 2023-05-22T15:27:12.041 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.030+0000 7f83ce1aa640 1 -- 172.21.15.16:0/2379951645 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f83c80c6630 con 0x7f83c8070df0 2023-05-22T15:27:12.042 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.034+0000 7f83bffff640 1 -- 172.21.15.16:0/2379951645 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7f83c80c6630 con 0x7f83c8070df0 2023-05-22T15:27:12.043 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:27:12.043 INFO:teuthology.orchestra.run.smithi016.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-05-22T15:27:12.045 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.034+0000 7f83ce1aa640 1 -- 172.21.15.16:0/2379951645 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f83a8065380 msgr2=0x7f83a8067830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:12.045 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.034+0000 7f83ce1aa640 1 --2- 172.21.15.16:0/2379951645 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f83a8065380 0x7f83a8067830 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f83b8007fd0 tx=0x7f83b8007d40 comp rx=0 tx=0).stop 2023-05-22T15:27:12.045 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.034+0000 7f83ce1aa640 1 -- 172.21.15.16:0/2379951645 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83c8070df0 msgr2=0x7f83c814c400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:12.046 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.034+0000 7f83ce1aa640 1 --2- 172.21.15.16:0/2379951645 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83c8070df0 0x7f83c814c400 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f83b000d5f0 tx=0x7f83b000dab0 comp rx=0 tx=0).stop 2023-05-22T15:27:12.046 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.034+0000 7f83ce1aa640 1 -- 172.21.15.16:0/2379951645 shutdown_connections 2023-05-22T15:27:12.046 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.034+0000 7f83ce1aa640 1 --2- 172.21.15.16:0/2379951645 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f83a8065380 0x7f83a8067830 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:12.047 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.034+0000 7f83ce1aa640 1 --2- 172.21.15.16:0/2379951645 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f83c80ca4e0 0x7f83c80aef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:12.047 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.034+0000 7f83ce1aa640 1 --2- 172.21.15.16:0/2379951645 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f83c80c9b40 0x7f83c8156dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:12.047 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.034+0000 7f83ce1aa640 1 --2- 172.21.15.16:0/2379951645 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83c8070df0 0x7f83c814c400 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:12.047 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.034+0000 7f83ce1aa640 1 -- 172.21.15.16:0/2379951645 >> 172.21.15.16:0/2379951645 conn(0x7f83c8069720 msgr2=0x7f83c80adbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:12.048 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.034+0000 7f83ce1aa640 1 -- 172.21.15.16:0/2379951645 shutdown_connections 2023-05-22T15:27:12.048 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.034+0000 7f83ce1aa640 1 -- 172.21.15.16:0/2379951645 wait complete. 2023-05-22T15:27:12.060 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2023-05-22T15:27:12.061 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-05-22T15:27:12.178 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.166+0000 7ffb208fd640 1 Processor -- start 2023-05-22T15:27:12.178 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.170+0000 7ffb208fd640 1 -- start start 2023-05-22T15:27:12.180 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.170+0000 7ffb208fd640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ffb1c0ca4e0 0x7ffb1c15e460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:12.180 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.170+0000 7ffb208fd640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ffb1c0c9b40 0x7ffb1c0afb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:12.180 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.170+0000 7ffb208fd640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb1c0b0650 con 0x7ffb1c070df0 2023-05-22T15:27:12.181 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.170+0000 7ffb208fd640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb1c0b0370 con 0x7ffb1c0ca4e0 2023-05-22T15:27:12.181 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.170+0000 7ffb208fd640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb1c15d980 con 0x7ffb1c0c9b40 2023-05-22T15:27:12.181 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.170+0000 7ffb1a59c640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ffb1c0ca4e0 0x7ffb1c15e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:12.182 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.170+0000 7ffb1a59c640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ffb1c0ca4e0 0x7ffb1c15e460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37852/0 (socket says 172.21.15.16:37852) 2023-05-22T15:27:12.183 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.170+0000 7ffb1a59c640 1 -- 172.21.15.16:0/3817622022 learned_addr learned my addr 172.21.15.16:0/3817622022 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:12.183 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.170+0000 7ffb13fff640 1 --2- 172.21.15.16:0/3817622022 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ffb1c0c9b40 0x7ffb1c0afb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:12.183 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.170+0000 7ffb1a59c640 1 -- 172.21.15.16:0/3817622022 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ffb1c0c9b40 msgr2=0x7ffb1c0afb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:12.184 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.170+0000 7ffb1a59c640 1 --2- 172.21.15.16:0/3817622022 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ffb1c0c9b40 0x7ffb1c0afb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:12.184 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.170+0000 7ffb1a59c640 1 -- 172.21.15.16:0/3817622022 >> v1:172.21.15.16:6789/0 conn(0x7ffb1c070df0 legacy=0x7ffb1c071210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:12.184 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.170+0000 7ffb1a59c640 1 -- 172.21.15.16:0/3817622022 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb1c15dc60 con 0x7ffb1c0ca4e0 2023-05-22T15:27:12.189 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.170+0000 7ffb1a59c640 1 --2- 172.21.15.16:0/3817622022 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffb1c0ca4e0 0x7ffb1c15e460 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7ffb0c009870 tx=0x7ffb0c01c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a6a81aa0324cc12 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:12.189 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.170+0000 7ffb19d9b640 1 -- 172.21.15.16:0/3817622022 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb0c0ab660 con 0x7ffb1c0ca4e0 2023-05-22T15:27:12.189 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.170+0000 7ffb19d9b640 1 -- 172.21.15.16:0/3817622022 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb0c0ab7c0 con 0x7ffb1c0ca4e0 2023-05-22T15:27:12.189 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.170+0000 7ffb19d9b640 1 -- 172.21.15.16:0/3817622022 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb0c0abb50 con 0x7ffb1c0ca4e0 2023-05-22T15:27:12.190 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.174+0000 7ffb208fd640 1 -- 172.21.15.16:0/3817622022 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffb1c0ca4e0 msgr2=0x7ffb1c15e460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:12.190 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.174+0000 7ffb208fd640 1 --2- 172.21.15.16:0/3817622022 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffb1c0ca4e0 0x7ffb1c15e460 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7ffb0c009870 tx=0x7ffb0c01c480 comp rx=0 tx=0).stop 2023-05-22T15:27:12.190 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.174+0000 7ffb208fd640 1 -- 172.21.15.16:0/3817622022 shutdown_connections 2023-05-22T15:27:12.190 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.174+0000 7ffb208fd640 1 --2- 172.21.15.16:0/3817622022 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ffb1c0c9b40 0x7ffb1c0afb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:12.190 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.174+0000 7ffb208fd640 1 --2- 172.21.15.16:0/3817622022 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffb1c0ca4e0 0x7ffb1c15e460 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:12.191 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.174+0000 7ffb208fd640 1 -- 172.21.15.16:0/3817622022 >> 172.21.15.16:0/3817622022 conn(0x7ffb1c069720 msgr2=0x7ffb1c069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:12.191 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.174+0000 7ffb208fd640 1 -- 172.21.15.16:0/3817622022 shutdown_connections 2023-05-22T15:27:12.191 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.174+0000 7ffb208fd640 1 -- 172.21.15.16:0/3817622022 wait complete. 2023-05-22T15:27:12.191 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.174+0000 7ffb208fd640 1 Processor -- start 2023-05-22T15:27:12.191 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.174+0000 7ffb208fd640 1 -- start start 2023-05-22T15:27:12.191 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.174+0000 7ffb208fd640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ffb1c070df0 0x7ffb1c069d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:12.192 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.174+0000 7ffb208fd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffb1c0c9b40 0x7ffb1c14b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:12.192 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.174+0000 7ffb208fd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffb1c0ca4e0 0x7ffb1c14be20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:12.192 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.174+0000 7ffb208fd640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb1c14c360 con 0x7ffb1c0ca4e0 2023-05-22T15:27:12.192 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.174+0000 7ffb208fd640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb1c14ab20 con 0x7ffb1c0c9b40 2023-05-22T15:27:12.193 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.174+0000 7ffb208fd640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb1c14ae00 con 0x7ffb1c070df0 2023-05-22T15:27:12.193 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.174+0000 7ffb1ad9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffb1c0ca4e0 0x7ffb1c14be20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:12.194 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.174+0000 7ffb1ad9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffb1c0ca4e0 0x7ffb1c14be20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37864/0 (socket says 172.21.15.16:37864) 2023-05-22T15:27:12.194 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.174+0000 7ffb1ad9d640 1 -- 172.21.15.16:0/1900309061 learned_addr learned my addr 172.21.15.16:0/1900309061 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:12.194 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.174+0000 7ffb1ad9d640 1 -- 172.21.15.16:0/1900309061 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ffb1c070df0 msgr2=0x7ffb1c069d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:12.194 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.174+0000 7ffb1ad9d640 1 --2- 172.21.15.16:0/1900309061 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ffb1c070df0 0x7ffb1c069d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:12.195 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.174+0000 7ffb13fff640 1 --2- 172.21.15.16:0/1900309061 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffb1c0c9b40 0x7ffb1c14b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:12.195 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.174+0000 7ffb1ad9d640 1 -- 172.21.15.16:0/1900309061 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffb1c0c9b40 msgr2=0x7ffb1c14b8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:12.195 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.174+0000 7ffb1ad9d640 1 --2- 172.21.15.16:0/1900309061 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffb1c0c9b40 0x7ffb1c14b8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:12.195 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.174+0000 7ffb1ad9d640 1 -- 172.21.15.16:0/1900309061 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb0c0094d0 con 0x7ffb1c0ca4e0 2023-05-22T15:27:12.195 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.174+0000 7ffb1ad9d640 1 --2- 172.21.15.16:0/1900309061 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffb1c0ca4e0 0x7ffb1c14be20 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7ffb1400c770 tx=0x7ffb1400cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:12.195 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.174+0000 7ffb12ffd640 1 -- 172.21.15.16:0/1900309061 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb14003b80 con 0x7ffb1c0ca4e0 2023-05-22T15:27:12.195 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.174+0000 7ffb12ffd640 1 -- 172.21.15.16:0/1900309061 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb14003ce0 con 0x7ffb1c0ca4e0 2023-05-22T15:27:12.196 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.174+0000 7ffb12ffd640 1 -- 172.21.15.16:0/1900309061 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb1400aa90 con 0x7ffb1c0ca4e0 2023-05-22T15:27:12.196 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.174+0000 7ffb208fd640 1 -- 172.21.15.16:0/1900309061 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb1c0c8e80 con 0x7ffb1c0ca4e0 2023-05-22T15:27:12.196 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.174+0000 7ffb208fd640 1 -- 172.21.15.16:0/1900309061 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb1c0ca8b0 con 0x7ffb1c0ca4e0 2023-05-22T15:27:12.196 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.178+0000 7ffb208fd640 1 -- 172.21.15.16:0/1900309061 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb1c0c9010 con 0x7ffb1c0ca4e0 2023-05-22T15:27:12.196 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.178+0000 7ffb12ffd640 1 -- 172.21.15.16:0/1900309061 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7ffb1400abf0 con 0x7ffb1c0ca4e0 2023-05-22T15:27:12.196 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.178+0000 7ffb12ffd640 1 --2- 172.21.15.16:0/1900309061 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ffafc065380 0x7ffafc067830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:12.196 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.178+0000 7ffb12ffd640 1 -- 172.21.15.16:0/1900309061 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6813+0+0 (secure 0 0 0) 0x7ffb140829e0 con 0x7ffb1c0ca4e0 2023-05-22T15:27:12.197 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.178+0000 7ffb1a59c640 1 --2- 172.21.15.16:0/1900309061 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ffafc065380 0x7ffafc067830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:12.197 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.178+0000 7ffb1a59c640 1 --2- 172.21.15.16:0/1900309061 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ffafc065380 0x7ffafc067830 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7ffb0c0af000 tx=0x7ffb0c008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:12.198 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.182+0000 7ffb12ffd640 1 -- 172.21.15.16:0/1900309061 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7ffb14050370 con 0x7ffb1c0ca4e0 2023-05-22T15:27:12.346 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.334+0000 7ffb208fd640 1 -- 172.21.15.16:0/1900309061 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ffb1c0c8330 con 0x7ffb1c0ca4e0 2023-05-22T15:27:12.348 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.338+0000 7ffb12ffd640 1 -- 172.21.15.16:0/1900309061 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) v1 ==== 74+0+19604 (secure 0 0 0) 0x7ffb14053990 con 0x7ffb1c0ca4e0 2023-05-22T15:27:12.348 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:27:12.348 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":27,"fsid":"b2719834-7847-4d98-90a6-f65d57f85c49","created":"2023-05-22T15:26:25.259314+0000","modified":"2023-05-22T15:26:53.971378+0000","last_up_change":"2023-05-22T15:26:36.448524+0000","last_in_change":"2023-05-22T15:26:27.675668+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-05-22T15:26:38.019781+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-05-22T15:26:43.570514+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-05-22T15:26:47.879159+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-05-22T15:26:48.522130+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data_ec","create_time":"2023-05-22T15:26:51.616142+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"9e7ff854-b25f-4cbb-a5c2-1799d9c53d2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":37021},{"type":"v1","addr":"172.21.15.16:6825","nonce":37021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":37021},{"type":"v1","addr":"172.21.15.16:6827","nonce":37021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":37021},{"type":"v1","addr":"172.21.15.16:6831","nonce":37021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":37021},{"type":"v1","addr":"172.21.15.16:6829","nonce":37021}]},"public_addr":"172.21.15.16:6825/37021","cluster_addr":"172.21.15.16:6827/37021","heartbeat_back_addr":"172.21.15.16:6831/37021","heartbeat_front_addr":"172.21.15.16:6829/37021","state":["exists","up"]},{"osd":1,"uuid":"4702fbc9-a927-4565-aca5-41dc69cc8264","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":37008},{"type":"v1","addr":"172.21.15.16:6801","nonce":37008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":37008},{"type":"v1","addr":"172.21.15.16:6803","nonce":37008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":37008},{"type":"v1","addr":"172.21.15.16:6807","nonce":37008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":37008},{"type":"v1","addr":"172.21.15.16:6805","nonce":37008}]},"public_addr":"172.21.15.16:6801/37008","cluster_addr":"172.21.15.16:6803/37008","heartbeat_back_addr":"172.21.15.16:6807/37008","heartbeat_front_addr":"172.21.15.16:6805/37008","state":["exists","up"]},{"osd":2,"uuid":"1e7bba46-4eba-47ef-bb81-5e2d371638ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":37009},{"type":"v1","addr":"172.21.15.16:6809","nonce":37009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":37009},{"type":"v1","addr":"172.21.15.16:6811","nonce":37009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":37009},{"type":"v1","addr":"172.21.15.16:6815","nonce":37009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":37009},{"type":"v1","addr":"172.21.15.16:6813","nonce":37009}]},"public_addr":"172.21.15.16:6809/37009","cluster_addr":"172.21.15.16:6811/37009","heartbeat_back_addr":"172.21.15.16:6815/37009","heartbeat_front_addr":"172.21.15.16:6813/37009","state":["exists","up"]},{"osd":3,"uuid":"383cacc0-5a88-4904-990e-498ca59ef48a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":37020},{"type":"v1","addr":"172.21.15.16:6817","nonce":37020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":37020},{"type":"v1","addr":"172.21.15.16:6819","nonce":37020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":37020},{"type":"v1","addr":"172.21.15.16:6823","nonce":37020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":37020},{"type":"v1","addr":"172.21.15.16:6821","nonce":37020}]},"public_addr":"172.21.15.16:6817/37020","cluster_addr":"172.21.15.16:6819/37020","heartbeat_back_addr":"172.21.15.16:6823/37020","heartbeat_front_addr":"172.21.15.16:6821/37020","state":["exists","up"]},{"osd":4,"uuid":"970500ef-95b7-42c1-b080-eed4fae7013d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":37353},{"type":"v1","addr":"172.21.15.62:6817","nonce":37353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6819","nonce":37353},{"type":"v1","addr":"172.21.15.62:6822","nonce":37353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37353},{"type":"v1","addr":"172.21.15.62:6831","nonce":37353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37353},{"type":"v1","addr":"172.21.15.62:6826","nonce":37353}]},"public_addr":"172.21.15.62:6817/37353","cluster_addr":"172.21.15.62:6822/37353","heartbeat_back_addr":"172.21.15.62:6831/37353","heartbeat_front_addr":"172.21.15.62:6826/37353","state":["exists","up"]},{"osd":5,"uuid":"022d67db-d4d8-445b-b1b9-8b446a00f5c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":37352},{"type":"v1","addr":"172.21.15.62:6801","nonce":37352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":37352},{"type":"v1","addr":"172.21.15.62:6803","nonce":37352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":37352},{"type":"v1","addr":"172.21.15.62:6807","nonce":37352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":37352},{"type":"v1","addr":"172.21.15.62:6805","nonce":37352}]},"public_addr":"172.21.15.62:6801/37352","cluster_addr":"172.21.15.62:6803/37352","heartbeat_back_addr":"172.21.15.62:6807/37352","heartbeat_front_addr":"172.21.15.62:6805/37352","state":["exists","up"]},{"osd":6,"uuid":"42bd372b-6688-4a1b-8407-e16e8c266e33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":37354},{"type":"v1","addr":"172.21.15.62:6820","nonce":37354}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":37354},{"type":"v1","addr":"172.21.15.62:6823","nonce":37354}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6829","nonce":37354},{"type":"v1","addr":"172.21.15.62:6830","nonce":37354}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6825","nonce":37354},{"type":"v1","addr":"172.21.15.62:6827","nonce":37354}]},"public_addr":"172.21.15.62:6820/37354","cluster_addr":"172.21.15.62:6823/37354","heartbeat_back_addr":"172.21.15.62:6830/37354","heartbeat_front_addr":"172.21.15.62:6827/37354","state":["exists","up"]},{"osd":7,"uuid":"ac70b2a3-f41b-4bf2-b227-7f2cbb55426d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37355},{"type":"v1","addr":"172.21.15.62:6809","nonce":37355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37355},{"type":"v1","addr":"172.21.15.62:6811","nonce":37355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":37355},{"type":"v1","addr":"172.21.15.62:6815","nonce":37355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":37355},{"type":"v1","addr":"172.21.15.62:6813","nonce":37355}]},"public_addr":"172.21.15.62:6809/37355","cluster_addr":"172.21.15.62:6811/37355","heartbeat_back_addr":"172.21.15.62:6815/37355","heartbeat_front_addr":"172.21.15.62:6813/37355","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.368896+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.107273+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.276888+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.287502+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.124791+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.251937+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.255679+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.232797+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-05-22T15:27:12.349 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.338+0000 7ffb208fd640 1 -- 172.21.15.16:0/1900309061 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ffafc065380 msgr2=0x7ffafc067830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:12.349 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.338+0000 7ffb208fd640 1 --2- 172.21.15.16:0/1900309061 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ffafc065380 0x7ffafc067830 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7ffb0c0af000 tx=0x7ffb0c008040 comp rx=0 tx=0).stop 2023-05-22T15:27:12.350 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.338+0000 7ffb208fd640 1 -- 172.21.15.16:0/1900309061 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffb1c0ca4e0 msgr2=0x7ffb1c14be20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:12.350 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.338+0000 7ffb208fd640 1 --2- 172.21.15.16:0/1900309061 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffb1c0ca4e0 0x7ffb1c14be20 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7ffb1400c770 tx=0x7ffb1400cc30 comp rx=0 tx=0).stop 2023-05-22T15:27:12.350 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.338+0000 7ffb208fd640 1 -- 172.21.15.16:0/1900309061 shutdown_connections 2023-05-22T15:27:12.350 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.338+0000 7ffb208fd640 1 --2- 172.21.15.16:0/1900309061 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffb1c0ca4e0 0x7ffb1c14be20 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:12.350 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.338+0000 7ffb208fd640 1 --2- 172.21.15.16:0/1900309061 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffb1c0c9b40 0x7ffb1c14b8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:12.350 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.338+0000 7ffb208fd640 1 --2- 172.21.15.16:0/1900309061 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ffafc065380 0x7ffafc067830 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:12.350 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.338+0000 7ffb208fd640 1 --2- 172.21.15.16:0/1900309061 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ffb1c070df0 0x7ffb1c069d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:12.351 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.338+0000 7ffb208fd640 1 -- 172.21.15.16:0/1900309061 >> 172.21.15.16:0/1900309061 conn(0x7ffb1c069720 msgr2=0x7ffb1c152820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:12.351 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.338+0000 7ffb208fd640 1 -- 172.21.15.16:0/1900309061 shutdown_connections 2023-05-22T15:27:12.351 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.338+0000 7ffb208fd640 1 -- 172.21.15.16:0/1900309061 wait complete. 2023-05-22T15:27:12.365 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-05-22T15:26:38.019781+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-05-22T15:26:43.570514+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2023-05-22T15:26:47.879159+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2023-05-22T15:26:48.522130+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-05-22T15:26:51.616142+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '27', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-05-22T15:27:12.366 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-05-22T15:27:12.493 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.482+0000 7f838f27d640 1 Processor -- start 2023-05-22T15:27:12.493 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.482+0000 7f838f27d640 1 -- start start 2023-05-22T15:27:12.495 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.486+0000 7f838f27d640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8388159970 0x7f8388159d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:12.496 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.486+0000 7f838f27d640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f838815dad0 0x7f838806b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:12.496 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.486+0000 7f838f27d640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8388070c30 con 0x7f838815d680 2023-05-22T15:27:12.496 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.486+0000 7f838f27d640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83880cb970 con 0x7f838815dad0 2023-05-22T15:27:12.496 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.486+0000 7f838f27d640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8388070e60 con 0x7f8388159970 2023-05-22T15:27:12.496 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.486+0000 7f838c818640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f838815d680 0x7f83880cb260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:39786/0 (socket says 172.21.15.16:39786) 2023-05-22T15:27:12.496 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.486+0000 7f838c818640 1 -- 172.21.15.16:0/1021532471 learned_addr learned my addr 172.21.15.16:0/1021532471 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:27:12.497 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.486+0000 7f838d81a640 1 --2- 172.21.15.16:0/1021532471 >> v2:172.21.15.62:3300/0 conn(0x7f838815dad0 0x7f838806b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:12.497 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.486+0000 7f837ffff640 1 -- 172.21.15.16:0/1021532471 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2283409013 0 0) 0x7f8388070c30 con 0x7f838815d680 2023-05-22T15:27:12.497 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.486+0000 7f838d019640 1 --2- 172.21.15.16:0/1021532471 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8388159970 0x7f8388159d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:12.497 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.486+0000 7f837ffff640 1 -- 172.21.15.16:0/1021532471 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8374003530 con 0x7f838815d680 2023-05-22T15:27:12.497 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.486+0000 7f837ffff640 1 -- 172.21.15.16:0/1021532471 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2964870912 0 0) 0x7f8374003530 con 0x7f838815d680 2023-05-22T15:27:12.497 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.486+0000 7f837ffff640 1 -- 172.21.15.16:0/1021532471 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8388159970 msgr2=0x7f8388159d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:12.497 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.486+0000 7f837ffff640 1 --2- 172.21.15.16:0/1021532471 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8388159970 0x7f8388159d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:12.497 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.486+0000 7f837ffff640 1 -- 172.21.15.16:0/1021532471 >> v2:172.21.15.62:3300/0 conn(0x7f838815dad0 msgr2=0x7f838806b6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:12.498 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.486+0000 7f837ffff640 1 --2- 172.21.15.16:0/1021532471 >> v2:172.21.15.62:3300/0 conn(0x7f838815dad0 0x7f838806b6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:12.498 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.486+0000 7f837ffff640 1 -- 172.21.15.16:0/1021532471 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8388071140 con 0x7f838815d680 2023-05-22T15:27:12.498 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.486+0000 7f837ffff640 1 -- 172.21.15.16:0/1021532471 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7f8378001680 con 0x7f838815d680 2023-05-22T15:27:12.498 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.486+0000 7f837ffff640 1 -- 172.21.15.16:0/1021532471 >> v1:172.21.15.16:6789/0 conn(0x7f838815d680 legacy=0x7f83880cb260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:12.498 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.486+0000 7f837ffff640 1 --2- 172.21.15.16:0/1021532471 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8374003980 0x7f8374003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:12.498 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.486+0000 7f837ffff640 1 --2- 172.21.15.16:0/1021532471 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f83740043b0 0x7f8374004870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:12.498 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.486+0000 7f837ffff640 1 --2- 172.21.15.16:0/1021532471 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8374004fe0 0x7f837400b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:12.499 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.486+0000 7f837ffff640 1 -- 172.21.15.16:0/1021532471 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83880cb970 con 0x7f8374003980 2023-05-22T15:27:12.499 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.486+0000 7f837ffff640 1 -- 172.21.15.16:0/1021532471 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8388070e60 con 0x7f8374004fe0 2023-05-22T15:27:12.499 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.486+0000 7f837ffff640 1 -- 172.21.15.16:0/1021532471 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f837400bf80 con 0x7f83740043b0 2023-05-22T15:27:12.499 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.486+0000 7f838d019640 1 --2- 172.21.15.16:0/1021532471 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8388159970 0x7f8388159d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T15:27:12.499 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.486+0000 7f838d019640 1 --2- 172.21.15.16:0/1021532471 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8374003980 0x7f8374003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:12.499 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.486+0000 7f838d81a640 1 --2- 172.21.15.16:0/1021532471 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8374004fe0 0x7f837400b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:12.499 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.486+0000 7f838c818640 1 --2- 172.21.15.16:0/1021532471 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f83740043b0 0x7f8374004870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:12.499 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.486+0000 7f838d019640 1 -- 172.21.15.16:0/1021532471 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f83740043b0 msgr2=0x7f8374004870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:12.500 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.486+0000 7f838d019640 1 --2- 172.21.15.16:0/1021532471 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f83740043b0 0x7f8374004870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:12.500 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.486+0000 7f838d019640 1 -- 172.21.15.16:0/1021532471 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8374004fe0 msgr2=0x7f837400b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:12.500 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.486+0000 7f838d019640 1 --2- 172.21.15.16:0/1021532471 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8374004fe0 0x7f837400b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:12.500 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.486+0000 7f838d019640 1 -- 172.21.15.16:0/1021532471 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8374003530 con 0x7f8374003980 2023-05-22T15:27:12.502 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.490+0000 7f838d019640 1 --2- 172.21.15.16:0/1021532471 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8374003980 0x7f8374003d90 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f83880cf020 tx=0x7f838401c3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6eb7cc8c0ece7e48 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:12.502 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.490+0000 7f837ffff640 1 -- 172.21.15.16:0/1021532471 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83840ac440 con 0x7f8374003980 2023-05-22T15:27:12.503 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.490+0000 7f837ffff640 1 -- 172.21.15.16:0/1021532471 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83840ac5a0 con 0x7f8374003980 2023-05-22T15:27:12.503 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.490+0000 7f837ffff640 1 -- 172.21.15.16:0/1021532471 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83840ac930 con 0x7f8374003980 2023-05-22T15:27:12.503 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.490+0000 7f838f27d640 1 -- 172.21.15.16:0/1021532471 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8374003980 msgr2=0x7f8374003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:12.503 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.490+0000 7f838f27d640 1 --2- 172.21.15.16:0/1021532471 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8374003980 0x7f8374003d90 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f83880cf020 tx=0x7f838401c3c0 comp rx=0 tx=0).stop 2023-05-22T15:27:12.503 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.490+0000 7f838d81a640 1 -- 172.21.15.16:0/1021532471 reap_dead start 2023-05-22T15:27:12.503 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.490+0000 7f838f27d640 1 -- 172.21.15.16:0/1021532471 shutdown_connections 2023-05-22T15:27:12.504 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.490+0000 7f838f27d640 1 -- 172.21.15.16:0/1021532471 >> 172.21.15.16:0/1021532471 conn(0x7f8388069720 msgr2=0x7f8388069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:12.504 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.490+0000 7f838f27d640 1 -- 172.21.15.16:0/1021532471 shutdown_connections 2023-05-22T15:27:12.504 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.490+0000 7f838f27d640 1 -- 172.21.15.16:0/1021532471 wait complete. 2023-05-22T15:27:12.504 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.490+0000 7f838f27d640 1 Processor -- start 2023-05-22T15:27:12.504 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.490+0000 7f838f27d640 1 -- start start 2023-05-22T15:27:12.504 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.490+0000 7f838f27d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8374003980 0x7f8388154570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:12.504 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.490+0000 7f838f27d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f83880c7890 0x7f83880c2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:12.505 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.490+0000 7f838f27d640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f83880c7d00 0x7f83880b2e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:12.505 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.490+0000 7f838f27d640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83880c6e40 con 0x7f8374003980 2023-05-22T15:27:12.505 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.490+0000 7f838f27d640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83880c7400 con 0x7f83880c7890 2023-05-22T15:27:12.505 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.490+0000 7f838f27d640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83880c3780 con 0x7f83880c7d00 2023-05-22T15:27:12.505 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.490+0000 7f838d019640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8374003980 0x7f8388154570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:12.505 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.490+0000 7f838d019640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8374003980 0x7f8388154570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37882/0 (socket says 172.21.15.16:37882) 2023-05-22T15:27:12.505 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.490+0000 7f838d019640 1 -- 172.21.15.16:0/797645862 learned_addr learned my addr 172.21.15.16:0/797645862 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:12.506 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.490+0000 7f838c818640 1 --2- 172.21.15.16:0/797645862 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f83880c7890 0x7f83880c2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:12.506 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.490+0000 7f838d81a640 1 --2- 172.21.15.16:0/797645862 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f83880c7d00 0x7f83880b2e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:12.506 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.490+0000 7f838d019640 1 -- 172.21.15.16:0/797645862 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f83880c7d00 msgr2=0x7f83880b2e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:12.506 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.490+0000 7f838d019640 1 --2- 172.21.15.16:0/797645862 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f83880c7d00 0x7f83880b2e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:12.506 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.490+0000 7f838d019640 1 -- 172.21.15.16:0/797645862 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f83880c7890 msgr2=0x7f83880c2cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:12.506 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.490+0000 7f838d019640 1 --2- 172.21.15.16:0/797645862 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f83880c7890 0x7f83880c2cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:12.506 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.490+0000 7f838d019640 1 -- 172.21.15.16:0/797645862 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83880cac90 con 0x7f8374003980 2023-05-22T15:27:12.506 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.490+0000 7f838c818640 1 --2- 172.21.15.16:0/797645862 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f83880c7890 0x7f83880c2cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:27:12.507 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.490+0000 7f838d019640 1 --2- 172.21.15.16:0/797645862 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8374003980 0x7f8388154570 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f838401c1f0 tx=0x7f8374005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:12.507 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.490+0000 7f837dffb640 1 -- 172.21.15.16:0/797645862 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83840bc070 con 0x7f8374003980 2023-05-22T15:27:12.507 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.490+0000 7f837dffb640 1 -- 172.21.15.16:0/797645862 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83840b7440 con 0x7f8374003980 2023-05-22T15:27:12.507 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.490+0000 7f838f27d640 1 -- 172.21.15.16:0/797645862 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f838806c010 con 0x7f8374003980 2023-05-22T15:27:12.507 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.490+0000 7f837dffb640 1 -- 172.21.15.16:0/797645862 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83840b77f0 con 0x7f8374003980 2023-05-22T15:27:12.507 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.490+0000 7f838f27d640 1 -- 172.21.15.16:0/797645862 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83880cae20 con 0x7f8374003980 2023-05-22T15:27:12.507 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.494+0000 7f838f27d640 1 -- 172.21.15.16:0/797645862 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f838804dbe0 con 0x7f8374003980 2023-05-22T15:27:12.508 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.494+0000 7f837dffb640 1 -- 172.21.15.16:0/797645862 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7f83840b3040 con 0x7f8374003980 2023-05-22T15:27:12.508 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.494+0000 7f837dffb640 1 --2- 172.21.15.16:0/797645862 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f836c0653b0 0x7f836c067860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:12.508 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.494+0000 7f837dffb640 1 -- 172.21.15.16:0/797645862 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f838411c0b0 con 0x7f8374003980 2023-05-22T15:27:12.509 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.494+0000 7f838c818640 1 --2- 172.21.15.16:0/797645862 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f836c0653b0 0x7f836c067860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:12.509 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.494+0000 7f838c818640 1 --2- 172.21.15.16:0/797645862 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f836c0653b0 0x7f836c067860 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f8378000f70 tx=0x7f8378000ee0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:12.509 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.498+0000 7f837dffb640 1 -- 172.21.15.16:0/797645862 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f83840e9a40 con 0x7f8374003980 2023-05-22T15:27:12.654 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.646+0000 7f838f27d640 1 -- 172.21.15.16:0/797645862 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f83880b0a10 con 0x7f8374003980 2023-05-22T15:27:12.654 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.646+0000 7f837dffb640 1 -- 172.21.15.16:0/797645862 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) v1 ==== 93+0+10 (secure 0 0 0) 0x7f838400f040 con 0x7f8374003980 2023-05-22T15:27:12.655 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2023-05-22T15:27:12.656 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.646+0000 7f838f27d640 1 -- 172.21.15.16:0/797645862 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f836c0653b0 msgr2=0x7f836c067860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:12.656 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.646+0000 7f838f27d640 1 --2- 172.21.15.16:0/797645862 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f836c0653b0 0x7f836c067860 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f8378000f70 tx=0x7f8378000ee0 comp rx=0 tx=0).stop 2023-05-22T15:27:12.657 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.646+0000 7f838f27d640 1 -- 172.21.15.16:0/797645862 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8374003980 msgr2=0x7f8388154570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:12.657 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.646+0000 7f838f27d640 1 --2- 172.21.15.16:0/797645862 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8374003980 0x7f8388154570 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f838401c1f0 tx=0x7f8374005800 comp rx=0 tx=0).stop 2023-05-22T15:27:12.657 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.646+0000 7f838f27d640 1 -- 172.21.15.16:0/797645862 shutdown_connections 2023-05-22T15:27:12.657 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.646+0000 7f838f27d640 1 --2- 172.21.15.16:0/797645862 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f836c0653b0 0x7f836c067860 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:12.657 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.646+0000 7f838f27d640 1 --2- 172.21.15.16:0/797645862 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f83880c7d00 0x7f83880b2e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:12.657 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.646+0000 7f838f27d640 1 --2- 172.21.15.16:0/797645862 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f83880c7890 0x7f83880c2cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:12.657 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.646+0000 7f838f27d640 1 --2- 172.21.15.16:0/797645862 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8374003980 0x7f8388154570 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:12.658 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.646+0000 7f838f27d640 1 -- 172.21.15.16:0/797645862 >> 172.21.15.16:0/797645862 conn(0x7f8388069720 msgr2=0x7f838815c770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:12.658 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.646+0000 7f838f27d640 1 -- 172.21.15.16:0/797645862 shutdown_connections 2023-05-22T15:27:12.658 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.646+0000 7f838f27d640 1 -- 172.21.15.16:0/797645862 wait complete. 2023-05-22T15:27:12.672 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-05-22T15:27:12.797 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.786+0000 7f4028508640 1 Processor -- start 2023-05-22T15:27:12.797 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.786+0000 7f4028508640 1 -- start start 2023-05-22T15:27:12.798 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.790+0000 7f4028508640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f402015bca0 0x7f40200cda80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:12.799 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.790+0000 7f4028508640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f40200d2480 0x7f40200d2900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:12.799 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.790+0000 7f4028508640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f40200b5340 0x7f40200b5790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:12.799 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.790+0000 7f4028508640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40200cd660 con 0x7f402015bca0 2023-05-22T15:27:12.799 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.790+0000 7f4028508640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40200d2e40 con 0x7f40200d2480 2023-05-22T15:27:12.799 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.790+0000 7f4028508640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40200cccc0 con 0x7f40200b5340 2023-05-22T15:27:12.799 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.790+0000 7f40262a4640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f402015bca0 0x7f40200cda80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:12.799 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.790+0000 7f40262a4640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f402015bca0 0x7f40200cda80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37886/0 (socket says 172.21.15.16:37886) 2023-05-22T15:27:12.800 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.790+0000 7f40262a4640 1 -- 172.21.15.16:0/830544625 learned_addr learned my addr 172.21.15.16:0/830544625 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:12.800 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.790+0000 7f4025aa3640 1 --2- 172.21.15.16:0/830544625 >> v2:172.21.15.62:3300/0 conn(0x7f40200d2480 0x7f40200d2900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:12.800 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.790+0000 7f4026aa5640 1 --2- 172.21.15.16:0/830544625 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f40200b5340 0x7f40200b5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:12.800 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.790+0000 7f40262a4640 1 -- 172.21.15.16:0/830544625 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f40200b5340 msgr2=0x7f40200b5790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:12.800 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.790+0000 7f40262a4640 1 --2- 172.21.15.16:0/830544625 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f40200b5340 0x7f40200b5790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:12.800 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.790+0000 7f40262a4640 1 -- 172.21.15.16:0/830544625 >> v2:172.21.15.62:3300/0 conn(0x7f40200d2480 msgr2=0x7f40200d2900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:12.800 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.790+0000 7f40262a4640 1 --2- 172.21.15.16:0/830544625 >> v2:172.21.15.62:3300/0 conn(0x7f40200d2480 0x7f40200d2900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:12.801 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.790+0000 7f40262a4640 1 -- 172.21.15.16:0/830544625 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40200ccfa0 con 0x7f402015bca0 2023-05-22T15:27:12.801 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.790+0000 7f4025aa3640 1 --2- 172.21.15.16:0/830544625 >> v2:172.21.15.62:3300/0 conn(0x7f40200d2480 0x7f40200d2900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:27:12.802 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.790+0000 7f40262a4640 1 --2- 172.21.15.16:0/830544625 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f402015bca0 0x7f40200cda80 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f4020065e80 tx=0x7f401c01c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97b5d6fa6794bb12 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:12.802 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.790+0000 7f40252a2640 1 -- 172.21.15.16:0/830544625 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f401c0aa610 con 0x7f402015bca0 2023-05-22T15:27:12.802 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.790+0000 7f40252a2640 1 -- 172.21.15.16:0/830544625 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f401c0a6040 con 0x7f402015bca0 2023-05-22T15:27:12.802 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.790+0000 7f40252a2640 1 -- 172.21.15.16:0/830544625 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f401c0aa990 con 0x7f402015bca0 2023-05-22T15:27:12.802 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.790+0000 7f4028508640 1 -- 172.21.15.16:0/830544625 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f402015bca0 msgr2=0x7f40200cda80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:12.802 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.790+0000 7f4028508640 1 --2- 172.21.15.16:0/830544625 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f402015bca0 0x7f40200cda80 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f4020065e80 tx=0x7f401c01c4d0 comp rx=0 tx=0).stop 2023-05-22T15:27:12.802 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.790+0000 7f4028508640 1 -- 172.21.15.16:0/830544625 shutdown_connections 2023-05-22T15:27:12.803 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.790+0000 7f4028508640 1 --2- 172.21.15.16:0/830544625 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f40200b5340 0x7f40200b5790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:12.803 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.790+0000 7f4028508640 1 --2- 172.21.15.16:0/830544625 >> v2:172.21.15.62:3300/0 conn(0x7f40200d2480 0x7f40200d2900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:12.803 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.790+0000 7f4028508640 1 --2- 172.21.15.16:0/830544625 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f402015bca0 0x7f40200cda80 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:12.803 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.790+0000 7f4028508640 1 -- 172.21.15.16:0/830544625 >> 172.21.15.16:0/830544625 conn(0x7f40200655c0 msgr2=0x7f4020070e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:12.803 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.790+0000 7f4028508640 1 -- 172.21.15.16:0/830544625 shutdown_connections 2023-05-22T15:27:12.803 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.790+0000 7f4028508640 1 -- 172.21.15.16:0/830544625 wait complete. 2023-05-22T15:27:12.805 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.794+0000 7f4028508640 1 Processor -- start 2023-05-22T15:27:12.805 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.794+0000 7f4028508640 1 -- start start 2023-05-22T15:27:12.805 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.794+0000 7f4028508640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f40200b5340 0x7f40200cbe00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:12.805 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.794+0000 7f4028508640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f40200d2480 0x7f4020150960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:12.805 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.794+0000 7f4028508640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f402015bca0 0x7f4020155d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:12.805 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.794+0000 7f4028508640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40200cad60 con 0x7f40200d2480 2023-05-22T15:27:12.806 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.794+0000 7f4028508640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40200cb040 con 0x7f402015bca0 2023-05-22T15:27:12.806 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.794+0000 7f4028508640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40200cb320 con 0x7f40200b5340 2023-05-22T15:27:12.806 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.794+0000 7f4025aa3640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f40200d2480 0x7f4020150960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:12.806 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.794+0000 7f4025aa3640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f40200d2480 0x7f4020150960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37894/0 (socket says 172.21.15.16:37894) 2023-05-22T15:27:12.807 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.794+0000 7f4025aa3640 1 -- 172.21.15.16:0/1414804118 learned_addr learned my addr 172.21.15.16:0/1414804118 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:12.807 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.794+0000 7f40262a4640 1 --2- 172.21.15.16:0/1414804118 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f40200b5340 0x7f40200cbe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:12.807 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.794+0000 7f4025aa3640 1 -- 172.21.15.16:0/1414804118 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f40200b5340 msgr2=0x7f40200cbe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:12.807 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.794+0000 7f4025aa3640 1 --2- 172.21.15.16:0/1414804118 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f40200b5340 0x7f40200cbe00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:12.807 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.794+0000 7f4026aa5640 1 --2- 172.21.15.16:0/1414804118 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f402015bca0 0x7f4020155d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:12.807 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.794+0000 7f4025aa3640 1 -- 172.21.15.16:0/1414804118 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f402015bca0 msgr2=0x7f4020155d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:12.807 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.794+0000 7f4025aa3640 1 --2- 172.21.15.16:0/1414804118 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f402015bca0 0x7f4020155d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:12.807 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.794+0000 7f4025aa3640 1 -- 172.21.15.16:0/1414804118 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f401c0094d0 con 0x7f40200d2480 2023-05-22T15:27:12.808 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.794+0000 7f4025aa3640 1 --2- 172.21.15.16:0/1414804118 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f40200d2480 0x7f4020150960 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f401000c7c0 tx=0x7f401000cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:12.808 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.794+0000 7f4016ffd640 1 -- 172.21.15.16:0/1414804118 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4010007a30 con 0x7f40200d2480 2023-05-22T15:27:12.808 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.794+0000 7f4016ffd640 1 -- 172.21.15.16:0/1414804118 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4010007b90 con 0x7f40200d2480 2023-05-22T15:27:12.808 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.794+0000 7f4028508640 1 -- 172.21.15.16:0/1414804118 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40200cdfc0 con 0x7f40200d2480 2023-05-22T15:27:12.808 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.794+0000 7f4016ffd640 1 -- 172.21.15.16:0/1414804118 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f401000ce70 con 0x7f40200d2480 2023-05-22T15:27:12.808 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.794+0000 7f4028508640 1 -- 172.21.15.16:0/1414804118 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40200b5cd0 con 0x7f40200d2480 2023-05-22T15:27:12.808 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.794+0000 7f4028508640 1 -- 172.21.15.16:0/1414804118 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4020160360 con 0x7f40200d2480 2023-05-22T15:27:12.809 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.798+0000 7f4016ffd640 1 -- 172.21.15.16:0/1414804118 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7f40100113f0 con 0x7f40200d2480 2023-05-22T15:27:12.809 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.798+0000 7f4016ffd640 1 --2- 172.21.15.16:0/1414804118 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f4004065380 0x7f4004067830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:12.809 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.798+0000 7f4016ffd640 1 -- 172.21.15.16:0/1414804118 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f4010083150 con 0x7f40200d2480 2023-05-22T15:27:12.810 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.798+0000 7f40262a4640 1 --2- 172.21.15.16:0/1414804118 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f4004065380 0x7f4004067830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:12.810 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.798+0000 7f40262a4640 1 --2- 172.21.15.16:0/1414804118 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f4004065380 0x7f4004067830 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f401c0ae000 tx=0x7f401c008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:12.810 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.802+0000 7f4016ffd640 1 -- 172.21.15.16:0/1414804118 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f4010050b60 con 0x7f40200d2480 2023-05-22T15:27:12.960 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.950+0000 7f4028508640 1 -- 172.21.15.16:0/1414804118 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f402004dbe0 con 0x7f40200d2480 2023-05-22T15:27:12.961 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.950+0000 7f4016ffd640 1 -- 172.21.15.16:0/1414804118 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v27) v1 ==== 92+0+10 (secure 0 0 0) 0x7f4010054180 con 0x7f40200d2480 2023-05-22T15:27:12.962 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 8 2023-05-22T15:27:12.963 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.954+0000 7f4028508640 1 -- 172.21.15.16:0/1414804118 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f4004065380 msgr2=0x7f4004067830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:12.964 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.954+0000 7f4028508640 1 --2- 172.21.15.16:0/1414804118 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f4004065380 0x7f4004067830 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f401c0ae000 tx=0x7f401c008040 comp rx=0 tx=0).stop 2023-05-22T15:27:12.964 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.954+0000 7f4028508640 1 -- 172.21.15.16:0/1414804118 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f40200d2480 msgr2=0x7f4020150960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:12.964 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.954+0000 7f4028508640 1 --2- 172.21.15.16:0/1414804118 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f40200d2480 0x7f4020150960 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f401000c7c0 tx=0x7f401000cc80 comp rx=0 tx=0).stop 2023-05-22T15:27:12.964 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.954+0000 7f4028508640 1 -- 172.21.15.16:0/1414804118 shutdown_connections 2023-05-22T15:27:12.964 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.954+0000 7f4028508640 1 --2- 172.21.15.16:0/1414804118 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f402015bca0 0x7f4020155d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:12.964 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.954+0000 7f4028508640 1 --2- 172.21.15.16:0/1414804118 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f40200d2480 0x7f4020150960 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:12.965 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.954+0000 7f4028508640 1 --2- 172.21.15.16:0/1414804118 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f4004065380 0x7f4004067830 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:12.965 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.954+0000 7f4028508640 1 --2- 172.21.15.16:0/1414804118 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f40200b5340 0x7f40200cbe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:12.965 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.954+0000 7f4028508640 1 -- 172.21.15.16:0/1414804118 >> 172.21.15.16:0/1414804118 conn(0x7f40200655c0 msgr2=0x7f40200b7f10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:12.965 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.954+0000 7f4028508640 1 -- 172.21.15.16:0/1414804118 shutdown_connections 2023-05-22T15:27:12.965 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:12.954+0000 7f4028508640 1 -- 172.21.15.16:0/1414804118 wait complete. 2023-05-22T15:27:12.979 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-05-22T15:27:13.097 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.086+0000 7fb8d3539640 1 Processor -- start 2023-05-22T15:27:13.097 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.086+0000 7fb8d3539640 1 -- start start 2023-05-22T15:27:13.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.090+0000 7fb8d3539640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fb8cc0cb4c0 0x7fb8cc158fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:13.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.090+0000 7fb8d3539640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8cc159f70 con 0x7fb8cc06b6c0 2023-05-22T15:27:13.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.090+0000 7fb8d3539640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8cc0cb030 con 0x7fb8cc159510 2023-05-22T15:27:13.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.090+0000 7fb8d3539640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8cc071760 con 0x7fb8cc0cb4c0 2023-05-22T15:27:13.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.090+0000 7fb8d12d5640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fb8cc06b6c0 0x7fb8cc06ba90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:39798/0 (socket says 172.21.15.16:39798) 2023-05-22T15:27:13.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.090+0000 7fb8d12d5640 1 -- 172.21.15.16:0/2605396844 learned_addr learned my addr 172.21.15.16:0/2605396844 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:27:13.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.090+0000 7fb8d0ad4640 1 --2- 172.21.15.16:0/2605396844 >> v2:172.21.15.62:3300/0 conn(0x7fb8cc0cb4c0 0x7fb8cc158fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:13.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.090+0000 7fb8cbfff640 1 -- 172.21.15.16:0/2605396844 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3415490732 0 0) 0x7fb8cc159f70 con 0x7fb8cc06b6c0 2023-05-22T15:27:13.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.090+0000 7fb8cbfff640 1 -- 172.21.15.16:0/2605396844 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8b4003530 con 0x7fb8cc06b6c0 2023-05-22T15:27:13.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.090+0000 7fb8cbfff640 1 -- 172.21.15.16:0/2605396844 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 577053177 0 0) 0x7fb8b4003530 con 0x7fb8cc06b6c0 2023-05-22T15:27:13.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.090+0000 7fb8cbfff640 1 -- 172.21.15.16:0/2605396844 >> v2:172.21.15.62:3300/0 conn(0x7fb8cc0cb4c0 msgr2=0x7fb8cc158fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:13.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.090+0000 7fb8cbfff640 1 --2- 172.21.15.16:0/2605396844 >> v2:172.21.15.62:3300/0 conn(0x7fb8cc0cb4c0 0x7fb8cc158fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:13.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.090+0000 7fb8cbfff640 1 -- 172.21.15.16:0/2605396844 >> v1:172.21.15.62:6789/0 conn(0x7fb8cc159510 legacy=0x7fb8cc06fba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:13.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.090+0000 7fb8cbfff640 1 -- 172.21.15.16:0/2605396844 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8cc159c90 con 0x7fb8cc06b6c0 2023-05-22T15:27:13.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.090+0000 7fb8cbfff640 1 -- 172.21.15.16:0/2605396844 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7fb8c4001a00 con 0x7fb8cc06b6c0 2023-05-22T15:27:13.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.090+0000 7fb8cbfff640 1 -- 172.21.15.16:0/2605396844 >> v1:172.21.15.16:6789/0 conn(0x7fb8cc06b6c0 legacy=0x7fb8cc06ba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:13.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.090+0000 7fb8cbfff640 1 --2- 172.21.15.16:0/2605396844 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb8b4003980 0x7fb8b4003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:13.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.090+0000 7fb8cbfff640 1 --2- 172.21.15.16:0/2605396844 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb8b40047d0 0x7fb8b4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:13.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.090+0000 7fb8cbfff640 1 --2- 172.21.15.16:0/2605396844 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb8b4005180 0x7fb8b400b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:13.103 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.090+0000 7fb8cbfff640 1 -- 172.21.15.16:0/2605396844 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8cc071760 con 0x7fb8b4003980 2023-05-22T15:27:13.103 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.090+0000 7fb8cbfff640 1 -- 172.21.15.16:0/2605396844 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8b400c000 con 0x7fb8b4005180 2023-05-22T15:27:13.103 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.090+0000 7fb8cbfff640 1 -- 172.21.15.16:0/2605396844 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8b400c1c0 con 0x7fb8b40047d0 2023-05-22T15:27:13.103 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.090+0000 7fb8d12d5640 1 --2- 172.21.15.16:0/2605396844 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb8b4003980 0x7fb8b4003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:13.103 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.090+0000 7fb8d0ad4640 1 --2- 172.21.15.16:0/2605396844 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb8b40047d0 0x7fb8b4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:13.103 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.090+0000 7fb8d12d5640 1 -- 172.21.15.16:0/2605396844 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb8b40047d0 msgr2=0x7fb8b4004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:13.103 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.090+0000 7fb8d12d5640 1 --2- 172.21.15.16:0/2605396844 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb8b40047d0 0x7fb8b4004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:13.104 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.090+0000 7fb8d1ad6640 1 --2- 172.21.15.16:0/2605396844 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb8b4005180 0x7fb8b400b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:13.104 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.090+0000 7fb8d12d5640 1 -- 172.21.15.16:0/2605396844 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb8b4005180 msgr2=0x7fb8b400b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:13.104 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.090+0000 7fb8d12d5640 1 --2- 172.21.15.16:0/2605396844 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb8b4005180 0x7fb8b400b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:13.104 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.090+0000 7fb8d12d5640 1 -- 172.21.15.16:0/2605396844 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8b4003530 con 0x7fb8b4003980 2023-05-22T15:27:13.104 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.090+0000 7fb8d1ad6640 1 --2- 172.21.15.16:0/2605396844 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb8b4005180 0x7fb8b400b990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:27:13.104 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.090+0000 7fb8d1ad6640 1 -- 172.21.15.16:0/2605396844 reap_dead start 2023-05-22T15:27:13.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.094+0000 7fb8d12d5640 1 --2- 172.21.15.16:0/2605396844 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb8b4003980 0x7fb8b4003d90 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fb8c40013e0 tx=0x7fb8c401d300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad9c364e9ef49d76 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:13.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.094+0000 7fb8cbfff640 1 -- 172.21.15.16:0/2605396844 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8c40a5d00 con 0x7fb8b4003980 2023-05-22T15:27:13.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.094+0000 7fb8cbfff640 1 -- 172.21.15.16:0/2605396844 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8c40a5e60 con 0x7fb8b4003980 2023-05-22T15:27:13.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.094+0000 7fb8cbfff640 1 -- 172.21.15.16:0/2605396844 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8c40072d0 con 0x7fb8b4003980 2023-05-22T15:27:13.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.094+0000 7fb8d3539640 1 -- 172.21.15.16:0/2605396844 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb8b4003980 msgr2=0x7fb8b4003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:13.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.094+0000 7fb8d3539640 1 --2- 172.21.15.16:0/2605396844 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb8b4003980 0x7fb8b4003d90 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fb8c40013e0 tx=0x7fb8c401d300 comp rx=0 tx=0).stop 2023-05-22T15:27:13.107 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.094+0000 7fb8d3539640 1 -- 172.21.15.16:0/2605396844 shutdown_connections 2023-05-22T15:27:13.107 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.094+0000 7fb8d3539640 1 --2- 172.21.15.16:0/2605396844 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb8b4003980 0x7fb8b4003d90 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:13.107 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.094+0000 7fb8d3539640 1 -- 172.21.15.16:0/2605396844 >> 172.21.15.16:0/2605396844 conn(0x7fb8cc069720 msgr2=0x7fb8cc069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:13.107 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.094+0000 7fb8d3539640 1 -- 172.21.15.16:0/2605396844 shutdown_connections 2023-05-22T15:27:13.107 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.094+0000 7fb8d3539640 1 -- 172.21.15.16:0/2605396844 wait complete. 2023-05-22T15:27:13.107 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.098+0000 7fb8d3539640 1 Processor -- start 2023-05-22T15:27:13.107 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.098+0000 7fb8d3539640 1 -- start start 2023-05-22T15:27:13.109 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.098+0000 7fb8d3539640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb8b4003980 0x7fb8cc13d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:13.109 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.098+0000 7fb8d3539640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb8cc0c4430 0x7fb8cc0c4880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:13.109 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.098+0000 7fb8d3539640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb8cc155480 0x7fb8cc1558f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:13.109 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.098+0000 7fb8d3539640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8cc0b2860 con 0x7fb8b4003980 2023-05-22T15:27:13.110 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.098+0000 7fb8d3539640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8cc150a80 con 0x7fb8cc0c4430 2023-05-22T15:27:13.110 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.098+0000 7fb8d3539640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8cc0b1f30 con 0x7fb8cc155480 2023-05-22T15:27:13.110 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.098+0000 7fb8d12d5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb8b4003980 0x7fb8cc13d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:13.110 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.098+0000 7fb8d12d5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb8b4003980 0x7fb8cc13d190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37918/0 (socket says 172.21.15.16:37918) 2023-05-22T15:27:13.110 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.098+0000 7fb8d12d5640 1 -- 172.21.15.16:0/2037437690 learned_addr learned my addr 172.21.15.16:0/2037437690 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:13.110 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.098+0000 7fb8d12d5640 1 -- 172.21.15.16:0/2037437690 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb8cc155480 msgr2=0x7fb8cc1558f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:13.110 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.098+0000 7fb8d1ad6640 1 --2- 172.21.15.16:0/2037437690 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb8cc155480 0x7fb8cc1558f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:13.111 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.098+0000 7fb8d0ad4640 1 --2- 172.21.15.16:0/2037437690 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb8cc0c4430 0x7fb8cc0c4880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:13.111 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.098+0000 7fb8d12d5640 1 --2- 172.21.15.16:0/2037437690 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb8cc155480 0x7fb8cc1558f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:13.111 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.098+0000 7fb8d12d5640 1 -- 172.21.15.16:0/2037437690 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb8cc0c4430 msgr2=0x7fb8cc0c4880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:13.111 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.098+0000 7fb8d12d5640 1 --2- 172.21.15.16:0/2037437690 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb8cc0c4430 0x7fb8cc0c4880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:13.111 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.098+0000 7fb8d12d5640 1 -- 172.21.15.16:0/2037437690 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8cc06bea0 con 0x7fb8b4003980 2023-05-22T15:27:13.111 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.098+0000 7fb8d12d5640 1 --2- 172.21.15.16:0/2037437690 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb8b4003980 0x7fb8cc13d190 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fb8c40ab040 tx=0x7fb8b4005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:13.111 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.098+0000 7fb8c9ffb640 1 -- 172.21.15.16:0/2037437690 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8c4004d70 con 0x7fb8b4003980 2023-05-22T15:27:13.111 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.098+0000 7fb8c9ffb640 1 -- 172.21.15.16:0/2037437690 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8c40b3070 con 0x7fb8b4003980 2023-05-22T15:27:13.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.098+0000 7fb8d3539640 1 -- 172.21.15.16:0/2037437690 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8cc159910 con 0x7fb8b4003980 2023-05-22T15:27:13.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.098+0000 7fb8c9ffb640 1 -- 172.21.15.16:0/2037437690 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8c40ba6a0 con 0x7fb8b4003980 2023-05-22T15:27:13.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.098+0000 7fb8d3539640 1 -- 172.21.15.16:0/2037437690 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8cc06c030 con 0x7fb8b4003980 2023-05-22T15:27:13.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.098+0000 7fb8d3539640 1 -- 172.21.15.16:0/2037437690 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8cc04dbe0 con 0x7fb8b4003980 2023-05-22T15:27:13.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.098+0000 7fb8c9ffb640 1 -- 172.21.15.16:0/2037437690 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7fb8c40070d0 con 0x7fb8b4003980 2023-05-22T15:27:13.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.098+0000 7fb8c9ffb640 1 --2- 172.21.15.16:0/2037437690 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fb8b8065380 0x7fb8b8067830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:13.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.098+0000 7fb8c9ffb640 1 -- 172.21.15.16:0/2037437690 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6813+0+0 (secure 0 0 0) 0x7fb8c40b5070 con 0x7fb8b4003980 2023-05-22T15:27:13.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.098+0000 7fb8d0ad4640 1 --2- 172.21.15.16:0/2037437690 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fb8b8065380 0x7fb8b8067830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:13.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.102+0000 7fb8d0ad4640 1 --2- 172.21.15.16:0/2037437690 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fb8b8065380 0x7fb8b8067830 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fb8cc0b3f40 tx=0x7fb8c0008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:13.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.102+0000 7fb8c9ffb640 1 -- 172.21.15.16:0/2037437690 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fb8c40ea610 con 0x7fb8b4003980 2023-05-22T15:27:13.265 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.254+0000 7fb8d3539640 1 -- 172.21.15.16:0/2037437690 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fb8cc0b2070 con 0x7fb8b4003980 2023-05-22T15:27:13.266 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.258+0000 7fb8c9ffb640 1 -- 172.21.15.16:0/2037437690 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) v1 ==== 104+0+11 (secure 0 0 0) 0x7fb8cc0b2070 con 0x7fb8b4003980 2023-05-22T15:27:13.266 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2023-05-22T15:27:13.268 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.258+0000 7fb8d3539640 1 -- 172.21.15.16:0/2037437690 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fb8b8065380 msgr2=0x7fb8b8067830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:13.268 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.258+0000 7fb8d3539640 1 --2- 172.21.15.16:0/2037437690 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fb8b8065380 0x7fb8b8067830 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fb8cc0b3f40 tx=0x7fb8c0008040 comp rx=0 tx=0).stop 2023-05-22T15:27:13.268 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.258+0000 7fb8d3539640 1 -- 172.21.15.16:0/2037437690 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb8b4003980 msgr2=0x7fb8cc13d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:13.269 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.258+0000 7fb8d3539640 1 --2- 172.21.15.16:0/2037437690 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb8b4003980 0x7fb8cc13d190 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fb8c40ab040 tx=0x7fb8b4005800 comp rx=0 tx=0).stop 2023-05-22T15:27:13.269 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.258+0000 7fb8d3539640 1 -- 172.21.15.16:0/2037437690 shutdown_connections 2023-05-22T15:27:13.269 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.258+0000 7fb8d3539640 1 --2- 172.21.15.16:0/2037437690 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fb8b8065380 0x7fb8b8067830 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:13.269 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.258+0000 7fb8d3539640 1 --2- 172.21.15.16:0/2037437690 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb8cc155480 0x7fb8cc1558f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:13.270 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.258+0000 7fb8d3539640 1 --2- 172.21.15.16:0/2037437690 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb8cc0c4430 0x7fb8cc0c4880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:13.270 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.258+0000 7fb8d3539640 1 --2- 172.21.15.16:0/2037437690 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb8b4003980 0x7fb8cc13d190 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:13.270 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.258+0000 7fb8d3539640 1 -- 172.21.15.16:0/2037437690 >> 172.21.15.16:0/2037437690 conn(0x7fb8cc069720 msgr2=0x7fb8cc15aee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:13.270 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.258+0000 7fb8d3539640 1 -- 172.21.15.16:0/2037437690 shutdown_connections 2023-05-22T15:27:13.271 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.258+0000 7fb8d3539640 1 -- 172.21.15.16:0/2037437690 wait complete. 2023-05-22T15:27:13.283 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-05-22T15:27:13.396 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.386+0000 7f6303471640 1 Processor -- start 2023-05-22T15:27:13.397 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.386+0000 7f6303471640 1 -- start start 2023-05-22T15:27:13.397 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.386+0000 7f6303471640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f62fc159970 0x7f62fc159d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:13.398 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.386+0000 7f6303471640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f62fc15db00 0x7f62fc06b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:13.398 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.386+0000 7f6303471640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62fc070c30 con 0x7f62fc15d680 2023-05-22T15:27:13.398 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.386+0000 7f6303471640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62fc0cb970 con 0x7f62fc15db00 2023-05-22T15:27:13.398 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.386+0000 7f6303471640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62fc070e60 con 0x7f62fc159970 2023-05-22T15:27:13.398 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.386+0000 7f6300a0c640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f62fc15d680 0x7f62fc0cb260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:39804/0 (socket says 172.21.15.16:39804) 2023-05-22T15:27:13.398 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.386+0000 7f6300a0c640 1 -- 172.21.15.16:0/1131363994 learned_addr learned my addr 172.21.15.16:0/1131363994 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:27:13.399 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.386+0000 7f6301a0e640 1 --2- 172.21.15.16:0/1131363994 >> v2:172.21.15.16:3300/0 conn(0x7f62fc15db00 0x7f62fc06b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:13.399 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.390+0000 7f630120d640 1 --2- 172.21.15.16:0/1131363994 >> v2:172.21.15.62:3300/0 conn(0x7f62fc159970 0x7f62fc159d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:13.399 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.390+0000 7f6301a0e640 1 -- 172.21.15.16:0/1131363994 >> v2:172.21.15.62:3300/0 conn(0x7f62fc159970 msgr2=0x7f62fc159d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:13.399 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.390+0000 7f6301a0e640 1 --2- 172.21.15.16:0/1131363994 >> v2:172.21.15.62:3300/0 conn(0x7f62fc159970 0x7f62fc159d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:13.400 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.390+0000 7f6301a0e640 1 -- 172.21.15.16:0/1131363994 >> v1:172.21.15.16:6789/0 conn(0x7f62fc15d680 legacy=0x7f62fc0cb260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:13.400 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.390+0000 7f6301a0e640 1 -- 172.21.15.16:0/1131363994 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62fc071140 con 0x7f62fc15db00 2023-05-22T15:27:13.401 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.390+0000 7f6301a0e640 1 --2- 172.21.15.16:0/1131363994 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f62fc15db00 0x7f62fc06b6c0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f62fc06daf0 tx=0x7f62f401c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1569f7e56a7c7075 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:13.401 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.390+0000 7f62effff640 1 -- 172.21.15.16:0/1131363994 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62f40a5820 con 0x7f62fc15db00 2023-05-22T15:27:13.401 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.390+0000 7f62effff640 1 -- 172.21.15.16:0/1131363994 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62f40a5980 con 0x7f62fc15db00 2023-05-22T15:27:13.401 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.390+0000 7f62effff640 1 -- 172.21.15.16:0/1131363994 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62f40a5d10 con 0x7f62fc15db00 2023-05-22T15:27:13.402 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.390+0000 7f6303471640 1 -- 172.21.15.16:0/1131363994 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f62fc15db00 msgr2=0x7f62fc06b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:13.402 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.390+0000 7f6303471640 1 --2- 172.21.15.16:0/1131363994 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f62fc15db00 0x7f62fc06b6c0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f62fc06daf0 tx=0x7f62f401c850 comp rx=0 tx=0).stop 2023-05-22T15:27:13.403 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.394+0000 7f6303471640 1 -- 172.21.15.16:0/1131363994 shutdown_connections 2023-05-22T15:27:13.403 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.394+0000 7f6303471640 1 --2- 172.21.15.16:0/1131363994 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f62fc15db00 0x7f62fc06b6c0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:13.403 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.394+0000 7f6303471640 1 --2- 172.21.15.16:0/1131363994 >> v2:172.21.15.62:3300/0 conn(0x7f62fc159970 0x7f62fc159d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:13.403 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.394+0000 7f6303471640 1 -- 172.21.15.16:0/1131363994 >> 172.21.15.16:0/1131363994 conn(0x7f62fc069720 msgr2=0x7f62fc069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:13.403 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.394+0000 7f6303471640 1 -- 172.21.15.16:0/1131363994 shutdown_connections 2023-05-22T15:27:13.404 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.394+0000 7f6303471640 1 -- 172.21.15.16:0/1131363994 wait complete. 2023-05-22T15:27:13.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.394+0000 7f6303471640 1 Processor -- start 2023-05-22T15:27:13.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.394+0000 7f6303471640 1 -- start start 2023-05-22T15:27:13.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.394+0000 7f6303471640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f62fc159970 0x7f62fc0ccad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:13.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.394+0000 7f6303471640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f62fc15d680 0x7f62fc0b0e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:13.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.394+0000 7f6303471640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f62fc15db00 0x7f62fc0b1380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:13.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.394+0000 7f6303471640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62fc0cd730 con 0x7f62fc159970 2023-05-22T15:27:13.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.394+0000 7f6303471640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62fc1541c0 con 0x7f62fc15db00 2023-05-22T15:27:13.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.394+0000 7f6303471640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62fc0cd450 con 0x7f62fc15d680 2023-05-22T15:27:13.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.394+0000 7f630120d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f62fc159970 0x7f62fc0ccad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:13.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.394+0000 7f630120d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f62fc159970 0x7f62fc0ccad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37932/0 (socket says 172.21.15.16:37932) 2023-05-22T15:27:13.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.394+0000 7f630120d640 1 -- 172.21.15.16:0/3018893299 learned_addr learned my addr 172.21.15.16:0/3018893299 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:13.408 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.394+0000 7f630120d640 1 -- 172.21.15.16:0/3018893299 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f62fc15d680 msgr2=0x7f62fc0b0e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:13.408 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.394+0000 7f630120d640 1 --2- 172.21.15.16:0/3018893299 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f62fc15d680 0x7f62fc0b0e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:13.408 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.394+0000 7f630120d640 1 -- 172.21.15.16:0/3018893299 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f62fc15db00 msgr2=0x7f62fc0b1380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:13.408 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.394+0000 7f630120d640 1 --2- 172.21.15.16:0/3018893299 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f62fc15db00 0x7f62fc0b1380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:13.409 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.394+0000 7f630120d640 1 -- 172.21.15.16:0/3018893299 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62f4009a70 con 0x7f62fc159970 2023-05-22T15:27:13.409 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.394+0000 7f630120d640 1 --2- 172.21.15.16:0/3018893299 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f62fc159970 0x7f62fc0ccad0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f62f000e930 tx=0x7f62f000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:13.409 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.394+0000 7f62edffb640 1 -- 172.21.15.16:0/3018893299 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62f000ca00 con 0x7f62fc159970 2023-05-22T15:27:13.409 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.394+0000 7f62edffb640 1 -- 172.21.15.16:0/3018893299 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62f000cb60 con 0x7f62fc159970 2023-05-22T15:27:13.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.394+0000 7f6303471640 1 -- 172.21.15.16:0/3018893299 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62fc0cae20 con 0x7f62fc159970 2023-05-22T15:27:13.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.394+0000 7f62edffb640 1 -- 172.21.15.16:0/3018893299 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62f0013470 con 0x7f62fc159970 2023-05-22T15:27:13.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.394+0000 7f6303471640 1 -- 172.21.15.16:0/3018893299 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62fc0cac90 con 0x7f62fc159970 2023-05-22T15:27:13.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.394+0000 7f6303471640 1 -- 172.21.15.16:0/3018893299 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62fc15adc0 con 0x7f62fc159970 2023-05-22T15:27:13.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.398+0000 7f62edffb640 1 -- 172.21.15.16:0/3018893299 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7f62f00135d0 con 0x7f62fc159970 2023-05-22T15:27:13.412 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.398+0000 7f62edffb640 1 --2- 172.21.15.16:0/3018893299 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f62e0065380 0x7f62e0067830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:13.412 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.398+0000 7f62edffb640 1 -- 172.21.15.16:0/3018893299 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f62f0082d60 con 0x7f62fc159970 2023-05-22T15:27:13.413 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.398+0000 7f6300a0c640 1 --2- 172.21.15.16:0/3018893299 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f62e0065380 0x7f62e0067830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:13.413 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.398+0000 7f6300a0c640 1 --2- 172.21.15.16:0/3018893299 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f62e0065380 0x7f62e0067830 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f62f8000f70 tx=0x7f62f8007a90 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:13.413 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.402+0000 7f62edffb640 1 -- 172.21.15.16:0/3018893299 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f62f0050840 con 0x7f62fc159970 2023-05-22T15:27:13.556 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.546+0000 7f6303471640 1 -- 172.21.15.16:0/3018893299 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f62fc154430 con 0x7f62fc159970 2023-05-22T15:27:13.558 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.546+0000 7f62edffb640 1 -- 172.21.15.16:0/3018893299 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) v1 ==== 100+0+11 (secure 0 0 0) 0x7f62f0053e60 con 0x7f62fc159970 2023-05-22T15:27:13.558 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 64 2023-05-22T15:27:13.559 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.550+0000 7f6303471640 1 -- 172.21.15.16:0/3018893299 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f62e0065380 msgr2=0x7f62e0067830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:13.559 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.550+0000 7f6303471640 1 --2- 172.21.15.16:0/3018893299 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f62e0065380 0x7f62e0067830 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f62f8000f70 tx=0x7f62f8007a90 comp rx=0 tx=0).stop 2023-05-22T15:27:13.559 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.550+0000 7f6303471640 1 -- 172.21.15.16:0/3018893299 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f62fc159970 msgr2=0x7f62fc0ccad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:13.560 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.550+0000 7f6303471640 1 --2- 172.21.15.16:0/3018893299 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f62fc159970 0x7f62fc0ccad0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f62f000e930 tx=0x7f62f000edf0 comp rx=0 tx=0).stop 2023-05-22T15:27:13.560 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.550+0000 7f6303471640 1 -- 172.21.15.16:0/3018893299 shutdown_connections 2023-05-22T15:27:13.560 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.550+0000 7f6303471640 1 --2- 172.21.15.16:0/3018893299 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f62fc15db00 0x7f62fc0b1380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:13.560 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.550+0000 7f6303471640 1 --2- 172.21.15.16:0/3018893299 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f62e0065380 0x7f62e0067830 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:13.560 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.550+0000 7f6303471640 1 --2- 172.21.15.16:0/3018893299 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f62fc15d680 0x7f62fc0b0e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:13.560 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.550+0000 7f6303471640 1 --2- 172.21.15.16:0/3018893299 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f62fc159970 0x7f62fc0ccad0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:13.560 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.550+0000 7f6303471640 1 -- 172.21.15.16:0/3018893299 >> 172.21.15.16:0/3018893299 conn(0x7f62fc069720 msgr2=0x7f62fc0cb1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:13.561 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.550+0000 7f6303471640 1 -- 172.21.15.16:0/3018893299 shutdown_connections 2023-05-22T15:27:13.561 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.550+0000 7f6303471640 1 -- 172.21.15.16:0/3018893299 wait complete. 2023-05-22T15:27:13.575 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-05-22T15:27:13.701 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.690+0000 7f5254fba640 1 Processor -- start 2023-05-22T15:27:13.701 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.690+0000 7f5254fba640 1 -- start start 2023-05-22T15:27:13.703 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.694+0000 7f5254fba640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f525015bca0 0x7f52500cda80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:13.703 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.694+0000 7f5254fba640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f52500d2480 0x7f52500d2900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:13.703 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.694+0000 7f5254fba640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f52500b5340 0x7f52500b5790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:13.703 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.694+0000 7f5254fba640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52500cd660 con 0x7f52500d2480 2023-05-22T15:27:13.704 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.694+0000 7f5254fba640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52500d2e40 con 0x7f525015bca0 2023-05-22T15:27:13.704 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.694+0000 7f5254fba640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52500cccc0 con 0x7f52500b5340 2023-05-22T15:27:13.704 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.694+0000 7f524dd9b640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f52500d2480 0x7f52500d2900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:13.704 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.694+0000 7f524dd9b640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f52500d2480 0x7f52500d2900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37942/0 (socket says 172.21.15.16:37942) 2023-05-22T15:27:13.704 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.694+0000 7f524dd9b640 1 -- 172.21.15.16:0/1266751922 learned_addr learned my addr 172.21.15.16:0/1266751922 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:13.704 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.694+0000 7f524e59c640 1 --2- 172.21.15.16:0/1266751922 >> v2:172.21.15.62:3300/0 conn(0x7f525015bca0 0x7f52500cda80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:13.704 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.694+0000 7f524ed9d640 1 --2- 172.21.15.16:0/1266751922 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f52500b5340 0x7f52500b5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:13.705 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.694+0000 7f524dd9b640 1 -- 172.21.15.16:0/1266751922 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f52500b5340 msgr2=0x7f52500b5790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:13.705 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.694+0000 7f524dd9b640 1 --2- 172.21.15.16:0/1266751922 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f52500b5340 0x7f52500b5790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:13.705 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.694+0000 7f524dd9b640 1 -- 172.21.15.16:0/1266751922 >> v2:172.21.15.62:3300/0 conn(0x7f525015bca0 msgr2=0x7f52500cda80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:13.705 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.694+0000 7f524dd9b640 1 --2- 172.21.15.16:0/1266751922 >> v2:172.21.15.62:3300/0 conn(0x7f525015bca0 0x7f52500cda80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:13.705 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.694+0000 7f524dd9b640 1 -- 172.21.15.16:0/1266751922 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52500ccfa0 con 0x7f52500d2480 2023-05-22T15:27:13.706 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.698+0000 7f524dd9b640 1 --2- 172.21.15.16:0/1266751922 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f52500d2480 0x7f52500d2900 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f52500b40d0 tx=0x7f524401c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=794bbf2e52bbd39b server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:13.706 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.698+0000 7f524d59a640 1 -- 172.21.15.16:0/1266751922 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52440aa610 con 0x7f52500d2480 2023-05-22T15:27:13.706 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.698+0000 7f524d59a640 1 -- 172.21.15.16:0/1266751922 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52440a6040 con 0x7f52500d2480 2023-05-22T15:27:13.707 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.698+0000 7f524d59a640 1 -- 172.21.15.16:0/1266751922 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52440aa990 con 0x7f52500d2480 2023-05-22T15:27:13.707 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.698+0000 7f5254fba640 1 -- 172.21.15.16:0/1266751922 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f52500d2480 msgr2=0x7f52500d2900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:13.707 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.698+0000 7f5254fba640 1 --2- 172.21.15.16:0/1266751922 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f52500d2480 0x7f52500d2900 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f52500b40d0 tx=0x7f524401c4d0 comp rx=0 tx=0).stop 2023-05-22T15:27:13.708 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.698+0000 7f5254fba640 1 -- 172.21.15.16:0/1266751922 shutdown_connections 2023-05-22T15:27:13.708 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.698+0000 7f5254fba640 1 --2- 172.21.15.16:0/1266751922 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f52500d2480 0x7f52500d2900 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:13.708 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.698+0000 7f5254fba640 1 --2- 172.21.15.16:0/1266751922 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f52500b5340 0x7f52500b5790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:13.708 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.698+0000 7f5254fba640 1 --2- 172.21.15.16:0/1266751922 >> v2:172.21.15.62:3300/0 conn(0x7f525015bca0 0x7f52500cda80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:13.708 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.698+0000 7f5254fba640 1 -- 172.21.15.16:0/1266751922 >> 172.21.15.16:0/1266751922 conn(0x7f52500655c0 msgr2=0x7f5250070e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:13.709 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.698+0000 7f5254fba640 1 -- 172.21.15.16:0/1266751922 shutdown_connections 2023-05-22T15:27:13.709 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.698+0000 7f5254fba640 1 -- 172.21.15.16:0/1266751922 wait complete. 2023-05-22T15:27:13.710 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.698+0000 7f5254fba640 1 Processor -- start 2023-05-22T15:27:13.711 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.698+0000 7f5254fba640 1 -- start start 2023-05-22T15:27:13.711 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.698+0000 7f5254fba640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f52500b5340 0x7f5250152ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:13.711 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.698+0000 7f5254fba640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f52500d2480 0x7f52501533e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:13.711 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.698+0000 7f5254fba640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f525015bca0 0x7f5250158b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:13.711 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.698+0000 7f5254fba640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5250159740 con 0x7f525015bca0 2023-05-22T15:27:13.711 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.698+0000 7f5254fba640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5250159a20 con 0x7f52500d2480 2023-05-22T15:27:13.711 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.698+0000 7f5254fba640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5250159d00 con 0x7f52500b5340 2023-05-22T15:27:13.712 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.702+0000 7f524ed9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f525015bca0 0x7f5250158b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:13.712 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.702+0000 7f524ed9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f525015bca0 0x7f5250158b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37948/0 (socket says 172.21.15.16:37948) 2023-05-22T15:27:13.712 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.702+0000 7f524ed9d640 1 -- 172.21.15.16:0/1401827846 learned_addr learned my addr 172.21.15.16:0/1401827846 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:13.712 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.702+0000 7f524dd9b640 1 --2- 172.21.15.16:0/1401827846 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f52500d2480 0x7f52501533e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:13.712 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.702+0000 7f524e59c640 1 --2- 172.21.15.16:0/1401827846 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f52500b5340 0x7f5250152ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:13.712 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.702+0000 7f524ed9d640 1 -- 172.21.15.16:0/1401827846 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f52500b5340 msgr2=0x7f5250152ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:13.712 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.702+0000 7f524ed9d640 1 --2- 172.21.15.16:0/1401827846 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f52500b5340 0x7f5250152ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:13.712 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.702+0000 7f524ed9d640 1 -- 172.21.15.16:0/1401827846 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f52500d2480 msgr2=0x7f52501533e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:13.713 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.702+0000 7f524ed9d640 1 --2- 172.21.15.16:0/1401827846 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f52500d2480 0x7f52501533e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:13.713 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.702+0000 7f524ed9d640 1 -- 172.21.15.16:0/1401827846 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52440094d0 con 0x7f525015bca0 2023-05-22T15:27:13.713 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.702+0000 7f524ed9d640 1 --2- 172.21.15.16:0/1401827846 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f525015bca0 0x7f5250158b60 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f5248007e60 tx=0x7f524800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:13.713 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.702+0000 7f523effd640 1 -- 172.21.15.16:0/1401827846 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5248014620 con 0x7f525015bca0 2023-05-22T15:27:13.713 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.702+0000 7f523effd640 1 -- 172.21.15.16:0/1401827846 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f524800ee90 con 0x7f525015bca0 2023-05-22T15:27:13.713 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.702+0000 7f5254fba640 1 -- 172.21.15.16:0/1401827846 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52500cdfc0 con 0x7f525015bca0 2023-05-22T15:27:13.714 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.702+0000 7f523effd640 1 -- 172.21.15.16:0/1401827846 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52480149e0 con 0x7f525015bca0 2023-05-22T15:27:13.714 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.702+0000 7f5254fba640 1 -- 172.21.15.16:0/1401827846 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52500b5cd0 con 0x7f525015bca0 2023-05-22T15:27:13.714 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.702+0000 7f5254fba640 1 -- 172.21.15.16:0/1401827846 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5250160360 con 0x7f525015bca0 2023-05-22T15:27:13.715 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.702+0000 7f523effd640 1 -- 172.21.15.16:0/1401827846 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7f5248014b40 con 0x7f525015bca0 2023-05-22T15:27:13.715 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.702+0000 7f523effd640 1 --2- 172.21.15.16:0/1401827846 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f5230065380 0x7f5230067830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:13.715 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.702+0000 7f524e59c640 1 --2- 172.21.15.16:0/1401827846 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f5230065380 0x7f5230067830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:13.715 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.702+0000 7f523effd640 1 -- 172.21.15.16:0/1401827846 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f5248084440 con 0x7f525015bca0 2023-05-22T15:27:13.715 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.706+0000 7f524e59c640 1 --2- 172.21.15.16:0/1401827846 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f5230065380 0x7f5230067830 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f524000b930 tx=0x7f5240008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:13.716 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.706+0000 7f523effd640 1 -- 172.21.15.16:0/1401827846 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f5248051db0 con 0x7f525015bca0 2023-05-22T15:27:13.865 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.854+0000 7f5254fba640 1 -- 172.21.15.16:0/1401827846 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f525015a1b0 con 0x7f525015bca0 2023-05-22T15:27:13.866 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.858+0000 7f523effd640 1 -- 172.21.15.16:0/1401827846 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) v1 ==== 103+0+11 (secure 0 0 0) 0x7f52480553d0 con 0x7f525015bca0 2023-05-22T15:27:13.866 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2023-05-22T15:27:13.868 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.858+0000 7f5254fba640 1 -- 172.21.15.16:0/1401827846 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f5230065380 msgr2=0x7f5230067830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:13.868 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.858+0000 7f5254fba640 1 --2- 172.21.15.16:0/1401827846 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f5230065380 0x7f5230067830 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f524000b930 tx=0x7f5240008040 comp rx=0 tx=0).stop 2023-05-22T15:27:13.868 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.858+0000 7f5254fba640 1 -- 172.21.15.16:0/1401827846 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f525015bca0 msgr2=0x7f5250158b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:13.868 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.858+0000 7f5254fba640 1 --2- 172.21.15.16:0/1401827846 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f525015bca0 0x7f5250158b60 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f5248007e60 tx=0x7f524800e4b0 comp rx=0 tx=0).stop 2023-05-22T15:27:13.869 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.858+0000 7f5254fba640 1 -- 172.21.15.16:0/1401827846 shutdown_connections 2023-05-22T15:27:13.869 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.858+0000 7f5254fba640 1 --2- 172.21.15.16:0/1401827846 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f525015bca0 0x7f5250158b60 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:13.870 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.858+0000 7f5254fba640 1 --2- 172.21.15.16:0/1401827846 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f52500d2480 0x7f52501533e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:13.870 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.858+0000 7f5254fba640 1 --2- 172.21.15.16:0/1401827846 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f5230065380 0x7f5230067830 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:13.871 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.858+0000 7f5254fba640 1 --2- 172.21.15.16:0/1401827846 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f52500b5340 0x7f5250152ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:13.871 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.858+0000 7f5254fba640 1 -- 172.21.15.16:0/1401827846 >> 172.21.15.16:0/1401827846 conn(0x7f52500655c0 msgr2=0x7f52500b7360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:13.871 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.858+0000 7f5254fba640 1 -- 172.21.15.16:0/1401827846 shutdown_connections 2023-05-22T15:27:13.871 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.858+0000 7f5254fba640 1 -- 172.21.15.16:0/1401827846 wait complete. 2023-05-22T15:27:13.883 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-05-22T15:27:13.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.986+0000 7f60b3e09640 1 Processor -- start 2023-05-22T15:27:13.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.986+0000 7f60b3e09640 1 -- start start 2023-05-22T15:27:13.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.986+0000 7f60b3e09640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f60ac0b7dd0 0x7f60ac0ccab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:13.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.986+0000 7f60b3e09640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f60ac15f700 0x7f60ac15fb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:13.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.986+0000 7f60b3e09640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f60ac160050 0x7f60ac0ce720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:13.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.986+0000 7f60b3e09640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60ac0d3360 con 0x7f60ac15f700 2023-05-22T15:27:13.998 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.986+0000 7f60b3e09640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60ac0cd170 con 0x7f60ac160050 2023-05-22T15:27:13.998 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.986+0000 7f60b3e09640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60ac0d2ac0 con 0x7f60ac0b7dd0 2023-05-22T15:27:13.998 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.986+0000 7f60b13a4640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f60ac15f700 0x7f60ac15fb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:13.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.986+0000 7f60b13a4640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f60ac15f700 0x7f60ac15fb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37954/0 (socket says 172.21.15.16:37954) 2023-05-22T15:27:13.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.986+0000 7f60b13a4640 1 -- 172.21.15.16:0/881766454 learned_addr learned my addr 172.21.15.16:0/881766454 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:13.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.986+0000 7f60b23a6640 1 --2- 172.21.15.16:0/881766454 >> v2:172.21.15.62:3300/0 conn(0x7f60ac160050 0x7f60ac0ce720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:13.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.986+0000 7f60b1ba5640 1 --2- 172.21.15.16:0/881766454 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f60ac0b7dd0 0x7f60ac0ccab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:13.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.986+0000 7f60b13a4640 1 -- 172.21.15.16:0/881766454 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f60ac0b7dd0 msgr2=0x7f60ac0ccab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:14.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.986+0000 7f60b13a4640 1 --2- 172.21.15.16:0/881766454 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f60ac0b7dd0 0x7f60ac0ccab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:14.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.986+0000 7f60b13a4640 1 -- 172.21.15.16:0/881766454 >> v2:172.21.15.62:3300/0 conn(0x7f60ac160050 msgr2=0x7f60ac0ce720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:14.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.986+0000 7f60b13a4640 1 --2- 172.21.15.16:0/881766454 >> v2:172.21.15.62:3300/0 conn(0x7f60ac160050 0x7f60ac0ce720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:14.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.986+0000 7f60b13a4640 1 -- 172.21.15.16:0/881766454 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60ac0d2da0 con 0x7f60ac15f700 2023-05-22T15:27:14.005 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.986+0000 7f60b13a4640 1 --2- 172.21.15.16:0/881766454 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f60ac15f700 0x7f60ac15fb10 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f60ac1607a0 tx=0x7f609c01c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dbc19b1d985aa294 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:14.005 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.990+0000 7f60b0ba3640 1 -- 172.21.15.16:0/881766454 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f609c0aa610 con 0x7f60ac15f700 2023-05-22T15:27:14.005 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.990+0000 7f60b0ba3640 1 -- 172.21.15.16:0/881766454 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f609c0a6040 con 0x7f60ac15f700 2023-05-22T15:27:14.006 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.990+0000 7f60b0ba3640 1 -- 172.21.15.16:0/881766454 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f609c0aa990 con 0x7f60ac15f700 2023-05-22T15:27:14.006 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.990+0000 7f60b3e09640 1 -- 172.21.15.16:0/881766454 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f60ac15f700 msgr2=0x7f60ac15fb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:14.006 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.990+0000 7f60b3e09640 1 --2- 172.21.15.16:0/881766454 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f60ac15f700 0x7f60ac15fb10 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f60ac1607a0 tx=0x7f609c01c4d0 comp rx=0 tx=0).stop 2023-05-22T15:27:14.006 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.990+0000 7f60b3e09640 1 -- 172.21.15.16:0/881766454 shutdown_connections 2023-05-22T15:27:14.007 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.990+0000 7f60b3e09640 1 --2- 172.21.15.16:0/881766454 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f60ac15f700 0x7f60ac15fb10 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:14.007 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.990+0000 7f60b3e09640 1 --2- 172.21.15.16:0/881766454 >> v2:172.21.15.62:3300/0 conn(0x7f60ac160050 0x7f60ac0ce720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:14.007 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.990+0000 7f60b3e09640 1 --2- 172.21.15.16:0/881766454 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f60ac0b7dd0 0x7f60ac0ccab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:14.007 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.990+0000 7f60b3e09640 1 -- 172.21.15.16:0/881766454 >> 172.21.15.16:0/881766454 conn(0x7f60ac0655c0 msgr2=0x7f60ac070e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:14.007 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.990+0000 7f60b3e09640 1 -- 172.21.15.16:0/881766454 shutdown_connections 2023-05-22T15:27:14.008 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.990+0000 7f60b3e09640 1 -- 172.21.15.16:0/881766454 wait complete. 2023-05-22T15:27:14.008 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.990+0000 7f60b3e09640 1 Processor -- start 2023-05-22T15:27:14.008 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.990+0000 7f60b3e09640 1 -- start start 2023-05-22T15:27:14.008 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.990+0000 7f60b3e09640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f60ac0b7dd0 0x7f60ac155610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:14.009 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.990+0000 7f60b3e09640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f60ac15f700 0x7f60ac156710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:14.009 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.990+0000 7f60b3e09640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f60ac160050 0x7f60ac0c8370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:14.009 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.990+0000 7f60b3e09640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60ac155b50 con 0x7f60ac15f700 2023-05-22T15:27:14.009 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.990+0000 7f60b3e09640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60ac157de0 con 0x7f60ac0b7dd0 2023-05-22T15:27:14.010 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.990+0000 7f60b3e09640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60ac0c6e20 con 0x7f60ac160050 2023-05-22T15:27:14.010 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.990+0000 7f60b13a4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f60ac15f700 0x7f60ac156710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:14.010 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.990+0000 7f60b13a4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f60ac15f700 0x7f60ac156710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37970/0 (socket says 172.21.15.16:37970) 2023-05-22T15:27:14.010 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.990+0000 7f60b13a4640 1 -- 172.21.15.16:0/41636980 learned_addr learned my addr 172.21.15.16:0/41636980 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:14.011 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.990+0000 7f60b23a6640 1 --2- 172.21.15.16:0/41636980 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f60ac160050 0x7f60ac0c8370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:14.011 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.990+0000 7f60b13a4640 1 -- 172.21.15.16:0/41636980 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f60ac160050 msgr2=0x7f60ac0c8370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:14.011 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.990+0000 7f60b13a4640 1 --2- 172.21.15.16:0/41636980 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f60ac160050 0x7f60ac0c8370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:14.011 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.990+0000 7f60b1ba5640 1 --2- 172.21.15.16:0/41636980 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f60ac0b7dd0 0x7f60ac155610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:14.012 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.990+0000 7f60b13a4640 1 -- 172.21.15.16:0/41636980 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f60ac0b7dd0 msgr2=0x7f60ac155610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:14.012 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.990+0000 7f60b13a4640 1 --2- 172.21.15.16:0/41636980 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f60ac0b7dd0 0x7f60ac155610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:14.012 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.990+0000 7f60b13a4640 1 -- 172.21.15.16:0/41636980 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f609c0094d0 con 0x7f60ac15f700 2023-05-22T15:27:14.013 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.990+0000 7f60b13a4640 1 --2- 172.21.15.16:0/41636980 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f60ac15f700 0x7f60ac156710 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f609c01ab80 tx=0x7f609c01ad10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:14.013 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.990+0000 7f60a27fc640 1 -- 172.21.15.16:0/41636980 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f609c0bd5f0 con 0x7f60ac15f700 2023-05-22T15:27:14.014 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.990+0000 7f60a27fc640 1 -- 172.21.15.16:0/41636980 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f609c0bd750 con 0x7f60ac15f700 2023-05-22T15:27:14.014 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.990+0000 7f60b3e09640 1 -- 172.21.15.16:0/41636980 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60ac0cf600 con 0x7f60ac15f700 2023-05-22T15:27:14.014 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.990+0000 7f60a27fc640 1 -- 172.21.15.16:0/41636980 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f609c0bdb00 con 0x7f60ac15f700 2023-05-22T15:27:14.014 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.990+0000 7f60b3e09640 1 -- 172.21.15.16:0/41636980 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60ac0cf470 con 0x7f60ac15f700 2023-05-22T15:27:14.014 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.994+0000 7f60b3e09640 1 -- 172.21.15.16:0/41636980 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60ac0cf790 con 0x7f60ac15f700 2023-05-22T15:27:14.015 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.994+0000 7f60a27fc640 1 -- 172.21.15.16:0/41636980 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7f609c0b33f0 con 0x7f60ac15f700 2023-05-22T15:27:14.015 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.994+0000 7f60a27fc640 1 --2- 172.21.15.16:0/41636980 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f6090065380 0x7f6090067830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:14.015 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.994+0000 7f60b1ba5640 1 --2- 172.21.15.16:0/41636980 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f6090065380 0x7f6090067830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:14.015 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.994+0000 7f60a27fc640 1 -- 172.21.15.16:0/41636980 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f609c11cf90 con 0x7f60ac15f700 2023-05-22T15:27:14.015 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.994+0000 7f60b1ba5640 1 --2- 172.21.15.16:0/41636980 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f6090065380 0x7f6090067830 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f60a8009f30 tx=0x7f60a800b040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:14.016 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:13.998+0000 7f60a27fc640 1 -- 172.21.15.16:0/41636980 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f609c0ea8c0 con 0x7f60ac15f700 2023-05-22T15:27:14.187 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.178+0000 7f60b3e09640 1 -- 172.21.15.16:0/41636980 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f60ac04dbe0 con 0x7f60ac15f700 2023-05-22T15:27:14.187 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.178+0000 7f60a27fc640 1 -- 172.21.15.16:0/41636980 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4641 (secure 0 0 0) 0x7f609c0aae60 con 0x7f60ac15f700 2023-05-22T15:27:14.188 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:27:14.188 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4261,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6834/3661808159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":3661808159},{"type":"v1","addr":"172.21.15.62:6834","nonce":3661808159}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4264,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/2227467222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6833","nonce":2227467222},{"type":"v1","addr":"172.21.15.62:6835","nonce":2227467222}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4284,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6836/2808106087","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2808106087},{"type":"v1","addr":"172.21.15.16:6836","nonce":2808106087}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-05-22T15:26:49.546008+0000","modified":"2023-05-22T15:27:06.047204+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4287},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4287":{"gid":4287,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.16:6837/283322289","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6835","nonce":283322289},{"type":"v1","addr":"172.21.15.16:6837","nonce":283322289}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-05-22T15:27:14.188 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 8 2023-05-22T15:27:14.190 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.178+0000 7f60b3e09640 1 -- 172.21.15.16:0/41636980 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f6090065380 msgr2=0x7f6090067830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:14.190 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.178+0000 7f60b3e09640 1 --2- 172.21.15.16:0/41636980 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f6090065380 0x7f6090067830 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f60a8009f30 tx=0x7f60a800b040 comp rx=0 tx=0).stop 2023-05-22T15:27:14.190 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.178+0000 7f60b3e09640 1 -- 172.21.15.16:0/41636980 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f60ac15f700 msgr2=0x7f60ac156710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:14.190 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.178+0000 7f60b3e09640 1 --2- 172.21.15.16:0/41636980 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f60ac15f700 0x7f60ac156710 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f609c01ab80 tx=0x7f609c01ad10 comp rx=0 tx=0).stop 2023-05-22T15:27:14.191 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.182+0000 7f60b3e09640 1 -- 172.21.15.16:0/41636980 shutdown_connections 2023-05-22T15:27:14.191 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.182+0000 7f60b3e09640 1 --2- 172.21.15.16:0/41636980 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f6090065380 0x7f6090067830 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:14.191 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.182+0000 7f60b3e09640 1 --2- 172.21.15.16:0/41636980 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f60ac160050 0x7f60ac0c8370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:14.191 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.182+0000 7f60b3e09640 1 --2- 172.21.15.16:0/41636980 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f60ac15f700 0x7f60ac156710 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:14.191 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.182+0000 7f60b3e09640 1 --2- 172.21.15.16:0/41636980 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f60ac0b7dd0 0x7f60ac155610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:14.191 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.182+0000 7f60b3e09640 1 -- 172.21.15.16:0/41636980 >> 172.21.15.16:0/41636980 conn(0x7f60ac0655c0 msgr2=0x7f60ac0b6980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:14.191 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.182+0000 7f60b3e09640 1 -- 172.21.15.16:0/41636980 shutdown_connections 2023-05-22T15:27:14.192 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.182+0000 7f60b3e09640 1 -- 172.21.15.16:0/41636980 wait complete. 2023-05-22T15:27:14.205 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-05-22T15:27:14.321 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.310+0000 7f7927885640 1 Processor -- start 2023-05-22T15:27:14.322 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.310+0000 7f7927885640 1 -- start start 2023-05-22T15:27:14.323 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.314+0000 7f7927885640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7920158290 0x7f79200c8880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:14.324 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.314+0000 7f7927885640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f7920158660 0x7f79200cdfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:14.324 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.314+0000 7f7927885640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f79200cce20 0x7f79200ce4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:14.324 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.314+0000 7f7927885640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79200c92b0 con 0x7f79200cce20 2023-05-22T15:27:14.325 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.314+0000 7f7927885640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79200cc060 con 0x7f7920158660 2023-05-22T15:27:14.325 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.314+0000 7f7927885640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79200cc340 con 0x7f7920158290 2023-05-22T15:27:14.326 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.314+0000 7f7925e22640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f79200cce20 0x7f79200ce4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:14.326 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.314+0000 7f7925e22640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f79200cce20 0x7f79200ce4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37982/0 (socket says 172.21.15.16:37982) 2023-05-22T15:27:14.326 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.314+0000 7f7925e22640 1 -- 172.21.15.16:0/4049534531 learned_addr learned my addr 172.21.15.16:0/4049534531 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:14.326 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.314+0000 7f7925621640 1 --2- 172.21.15.16:0/4049534531 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7920158290 0x7f79200c8880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:14.327 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.314+0000 7f7924e20640 1 --2- 172.21.15.16:0/4049534531 >> v2:172.21.15.62:3300/0 conn(0x7f7920158660 0x7f79200cdfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:14.327 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.314+0000 7f7925e22640 1 -- 172.21.15.16:0/4049534531 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7920158290 msgr2=0x7f79200c8880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:14.327 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.314+0000 7f7925e22640 1 --2- 172.21.15.16:0/4049534531 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7920158290 0x7f79200c8880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:14.328 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.314+0000 7f7925e22640 1 -- 172.21.15.16:0/4049534531 >> v2:172.21.15.62:3300/0 conn(0x7f7920158660 msgr2=0x7f79200cdfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:14.328 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.314+0000 7f7925e22640 1 --2- 172.21.15.16:0/4049534531 >> v2:172.21.15.62:3300/0 conn(0x7f7920158660 0x7f79200cdfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:14.328 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.314+0000 7f7925e22640 1 -- 172.21.15.16:0/4049534531 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79200cc620 con 0x7f79200cce20 2023-05-22T15:27:14.331 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.318+0000 7f7925e22640 1 --2- 172.21.15.16:0/4049534531 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f79200cce20 0x7f79200ce4f0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f792006d720 tx=0x7f791801c9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c291564e9d4e78 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:14.331 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.318+0000 7f7917fff640 1 -- 172.21.15.16:0/4049534531 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79180a57c0 con 0x7f79200cce20 2023-05-22T15:27:14.331 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.318+0000 7f7917fff640 1 -- 172.21.15.16:0/4049534531 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79180a5920 con 0x7f79200cce20 2023-05-22T15:27:14.332 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.318+0000 7f7917fff640 1 -- 172.21.15.16:0/4049534531 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79180a5cb0 con 0x7f79200cce20 2023-05-22T15:27:14.332 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.318+0000 7f7927885640 1 -- 172.21.15.16:0/4049534531 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f79200cce20 msgr2=0x7f79200ce4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:14.332 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.318+0000 7f7927885640 1 --2- 172.21.15.16:0/4049534531 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f79200cce20 0x7f79200ce4f0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f792006d720 tx=0x7f791801c9a0 comp rx=0 tx=0).stop 2023-05-22T15:27:14.332 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.318+0000 7f7927885640 1 -- 172.21.15.16:0/4049534531 shutdown_connections 2023-05-22T15:27:14.333 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.318+0000 7f7927885640 1 --2- 172.21.15.16:0/4049534531 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f79200cce20 0x7f79200ce4f0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:14.333 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.318+0000 7f7927885640 1 --2- 172.21.15.16:0/4049534531 >> v2:172.21.15.62:3300/0 conn(0x7f7920158660 0x7f79200cdfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:14.333 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.318+0000 7f7927885640 1 --2- 172.21.15.16:0/4049534531 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7920158290 0x7f79200c8880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:14.333 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.318+0000 7f7927885640 1 -- 172.21.15.16:0/4049534531 >> 172.21.15.16:0/4049534531 conn(0x7f7920069720 msgr2=0x7f7920069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:14.333 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.318+0000 7f7927885640 1 -- 172.21.15.16:0/4049534531 shutdown_connections 2023-05-22T15:27:14.334 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.318+0000 7f7927885640 1 -- 172.21.15.16:0/4049534531 wait complete. 2023-05-22T15:27:14.334 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.318+0000 7f7927885640 1 Processor -- start 2023-05-22T15:27:14.334 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.318+0000 7f7927885640 1 -- start start 2023-05-22T15:27:14.334 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.318+0000 7f7927885640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f79200cce20 0x7f79200b2290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:14.334 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.318+0000 7f7927885640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7920158290 0x7f7920150810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:14.334 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.318+0000 7f7927885640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7920158660 0x7f7920150d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:14.334 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.318+0000 7f7927885640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79200c5b70 con 0x7f7920158660 2023-05-22T15:27:14.334 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.318+0000 7f7927885640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f792014ee40 con 0x7f79200cce20 2023-05-22T15:27:14.335 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.318+0000 7f7927885640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f792014f090 con 0x7f7920158290 2023-05-22T15:27:14.335 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.318+0000 7f7925e22640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7920158660 0x7f7920150d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:14.335 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.318+0000 7f7925e22640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7920158660 0x7f7920150d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37990/0 (socket says 172.21.15.16:37990) 2023-05-22T15:27:14.335 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.318+0000 7f7925e22640 1 -- 172.21.15.16:0/4180039593 learned_addr learned my addr 172.21.15.16:0/4180039593 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:14.335 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.318+0000 7f7925621640 1 --2- 172.21.15.16:0/4180039593 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f79200cce20 0x7f79200b2290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:14.336 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.318+0000 7f7924e20640 1 --2- 172.21.15.16:0/4180039593 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7920158290 0x7f7920150810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:14.336 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.318+0000 7f7925e22640 1 -- 172.21.15.16:0/4180039593 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7920158290 msgr2=0x7f7920150810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:14.336 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.318+0000 7f7925e22640 1 --2- 172.21.15.16:0/4180039593 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7920158290 0x7f7920150810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:14.336 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.318+0000 7f7925e22640 1 -- 172.21.15.16:0/4180039593 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f79200cce20 msgr2=0x7f79200b2290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:14.336 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.318+0000 7f7925e22640 1 --2- 172.21.15.16:0/4180039593 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f79200cce20 0x7f79200b2290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:14.336 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.318+0000 7f7925e22640 1 -- 172.21.15.16:0/4180039593 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7918009b30 con 0x7f7920158660 2023-05-22T15:27:14.336 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.318+0000 7f7925e22640 1 --2- 172.21.15.16:0/4180039593 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7920158660 0x7f7920150d50 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f79180af040 tx=0x7f7918019740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:14.337 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.318+0000 7f7915ffb640 1 -- 172.21.15.16:0/4180039593 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79180b8670 con 0x7f7920158660 2023-05-22T15:27:14.337 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.322+0000 7f7915ffb640 1 -- 172.21.15.16:0/4180039593 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79180b87d0 con 0x7f7920158660 2023-05-22T15:27:14.337 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.322+0000 7f7915ffb640 1 -- 172.21.15.16:0/4180039593 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79180b8b60 con 0x7f7920158660 2023-05-22T15:27:14.337 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.322+0000 7f7927885640 1 -- 172.21.15.16:0/4180039593 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79200cabc0 con 0x7f7920158660 2023-05-22T15:27:14.337 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.322+0000 7f7927885640 1 -- 172.21.15.16:0/4180039593 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79200caa30 con 0x7f7920158660 2023-05-22T15:27:14.337 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.322+0000 7f7927885640 1 -- 172.21.15.16:0/4180039593 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79200cad50 con 0x7f7920158660 2023-05-22T15:27:14.337 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.322+0000 7f7915ffb640 1 -- 172.21.15.16:0/4180039593 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7f79180b8cc0 con 0x7f7920158660 2023-05-22T15:27:14.338 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.322+0000 7f7915ffb640 1 --2- 172.21.15.16:0/4180039593 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f7904065380 0x7f7904067830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:14.338 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.322+0000 7f7915ffb640 1 -- 172.21.15.16:0/4180039593 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f791811e210 con 0x7f7920158660 2023-05-22T15:27:14.339 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.322+0000 7f7925621640 1 --2- 172.21.15.16:0/4180039593 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f7904065380 0x7f7904067830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:14.339 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.322+0000 7f7925621640 1 --2- 172.21.15.16:0/4180039593 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f7904065380 0x7f7904067830 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f791c00b9d0 tx=0x7f791c008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:14.339 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.326+0000 7f7915ffb640 1 -- 172.21.15.16:0/4180039593 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f79180ebd00 con 0x7f7920158660 2023-05-22T15:27:14.490 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.482+0000 7f7927885640 1 -- 172.21.15.16:0/4180039593 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f79200c5cb0 con 0x7f7920158660 2023-05-22T15:27:14.492 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.482+0000 7f7915ffb640 1 -- 172.21.15.16:0/4180039593 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) v1 ==== 74+0+19604 (secure 0 0 0) 0x7f79180ef320 con 0x7f7920158660 2023-05-22T15:27:14.493 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:27:14.493 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":27,"fsid":"b2719834-7847-4d98-90a6-f65d57f85c49","created":"2023-05-22T15:26:25.259314+0000","modified":"2023-05-22T15:26:53.971378+0000","last_up_change":"2023-05-22T15:26:36.448524+0000","last_in_change":"2023-05-22T15:26:27.675668+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-05-22T15:26:38.019781+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-05-22T15:26:43.570514+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-05-22T15:26:47.879159+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-05-22T15:26:48.522130+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data_ec","create_time":"2023-05-22T15:26:51.616142+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"9e7ff854-b25f-4cbb-a5c2-1799d9c53d2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":37021},{"type":"v1","addr":"172.21.15.16:6825","nonce":37021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":37021},{"type":"v1","addr":"172.21.15.16:6827","nonce":37021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":37021},{"type":"v1","addr":"172.21.15.16:6831","nonce":37021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":37021},{"type":"v1","addr":"172.21.15.16:6829","nonce":37021}]},"public_addr":"172.21.15.16:6825/37021","cluster_addr":"172.21.15.16:6827/37021","heartbeat_back_addr":"172.21.15.16:6831/37021","heartbeat_front_addr":"172.21.15.16:6829/37021","state":["exists","up"]},{"osd":1,"uuid":"4702fbc9-a927-4565-aca5-41dc69cc8264","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":37008},{"type":"v1","addr":"172.21.15.16:6801","nonce":37008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":37008},{"type":"v1","addr":"172.21.15.16:6803","nonce":37008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":37008},{"type":"v1","addr":"172.21.15.16:6807","nonce":37008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":37008},{"type":"v1","addr":"172.21.15.16:6805","nonce":37008}]},"public_addr":"172.21.15.16:6801/37008","cluster_addr":"172.21.15.16:6803/37008","heartbeat_back_addr":"172.21.15.16:6807/37008","heartbeat_front_addr":"172.21.15.16:6805/37008","state":["exists","up"]},{"osd":2,"uuid":"1e7bba46-4eba-47ef-bb81-5e2d371638ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":37009},{"type":"v1","addr":"172.21.15.16:6809","nonce":37009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":37009},{"type":"v1","addr":"172.21.15.16:6811","nonce":37009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":37009},{"type":"v1","addr":"172.21.15.16:6815","nonce":37009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":37009},{"type":"v1","addr":"172.21.15.16:6813","nonce":37009}]},"public_addr":"172.21.15.16:6809/37009","cluster_addr":"172.21.15.16:6811/37009","heartbeat_back_addr":"172.21.15.16:6815/37009","heartbeat_front_addr":"172.21.15.16:6813/37009","state":["exists","up"]},{"osd":3,"uuid":"383cacc0-5a88-4904-990e-498ca59ef48a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":37020},{"type":"v1","addr":"172.21.15.16:6817","nonce":37020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":37020},{"type":"v1","addr":"172.21.15.16:6819","nonce":37020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":37020},{"type":"v1","addr":"172.21.15.16:6823","nonce":37020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":37020},{"type":"v1","addr":"172.21.15.16:6821","nonce":37020}]},"public_addr":"172.21.15.16:6817/37020","cluster_addr":"172.21.15.16:6819/37020","heartbeat_back_addr":"172.21.15.16:6823/37020","heartbeat_front_addr":"172.21.15.16:6821/37020","state":["exists","up"]},{"osd":4,"uuid":"970500ef-95b7-42c1-b080-eed4fae7013d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":37353},{"type":"v1","addr":"172.21.15.62:6817","nonce":37353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6819","nonce":37353},{"type":"v1","addr":"172.21.15.62:6822","nonce":37353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37353},{"type":"v1","addr":"172.21.15.62:6831","nonce":37353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37353},{"type":"v1","addr":"172.21.15.62:6826","nonce":37353}]},"public_addr":"172.21.15.62:6817/37353","cluster_addr":"172.21.15.62:6822/37353","heartbeat_back_addr":"172.21.15.62:6831/37353","heartbeat_front_addr":"172.21.15.62:6826/37353","state":["exists","up"]},{"osd":5,"uuid":"022d67db-d4d8-445b-b1b9-8b446a00f5c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":37352},{"type":"v1","addr":"172.21.15.62:6801","nonce":37352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":37352},{"type":"v1","addr":"172.21.15.62:6803","nonce":37352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":37352},{"type":"v1","addr":"172.21.15.62:6807","nonce":37352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":37352},{"type":"v1","addr":"172.21.15.62:6805","nonce":37352}]},"public_addr":"172.21.15.62:6801/37352","cluster_addr":"172.21.15.62:6803/37352","heartbeat_back_addr":"172.21.15.62:6807/37352","heartbeat_front_addr":"172.21.15.62:6805/37352","state":["exists","up"]},{"osd":6,"uuid":"42bd372b-6688-4a1b-8407-e16e8c266e33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":37354},{"type":"v1","addr":"172.21.15.62:6820","nonce":37354}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":37354},{"type":"v1","addr":"172.21.15.62:6823","nonce":37354}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6829","nonce":37354},{"type":"v1","addr":"172.21.15.62:6830","nonce":37354}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6825","nonce":37354},{"type":"v1","addr":"172.21.15.62:6827","nonce":37354}]},"public_addr":"172.21.15.62:6820/37354","cluster_addr":"172.21.15.62:6823/37354","heartbeat_back_addr":"172.21.15.62:6830/37354","heartbeat_front_addr":"172.21.15.62:6827/37354","state":["exists","up"]},{"osd":7,"uuid":"ac70b2a3-f41b-4bf2-b227-7f2cbb55426d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37355},{"type":"v1","addr":"172.21.15.62:6809","nonce":37355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37355},{"type":"v1","addr":"172.21.15.62:6811","nonce":37355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":37355},{"type":"v1","addr":"172.21.15.62:6815","nonce":37355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":37355},{"type":"v1","addr":"172.21.15.62:6813","nonce":37355}]},"public_addr":"172.21.15.62:6809/37355","cluster_addr":"172.21.15.62:6811/37355","heartbeat_back_addr":"172.21.15.62:6815/37355","heartbeat_front_addr":"172.21.15.62:6813/37355","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.368896+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.107273+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.276888+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.287502+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.124791+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.251937+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.255679+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.232797+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-05-22T15:27:14.494 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.486+0000 7f7927885640 1 -- 172.21.15.16:0/4180039593 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f7904065380 msgr2=0x7f7904067830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:14.494 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.486+0000 7f7927885640 1 --2- 172.21.15.16:0/4180039593 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f7904065380 0x7f7904067830 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f791c00b9d0 tx=0x7f791c008040 comp rx=0 tx=0).stop 2023-05-22T15:27:14.495 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.486+0000 7f7927885640 1 -- 172.21.15.16:0/4180039593 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7920158660 msgr2=0x7f7920150d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:14.495 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.486+0000 7f7927885640 1 --2- 172.21.15.16:0/4180039593 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7920158660 0x7f7920150d50 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f79180af040 tx=0x7f7918019740 comp rx=0 tx=0).stop 2023-05-22T15:27:14.495 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.486+0000 7f7927885640 1 -- 172.21.15.16:0/4180039593 shutdown_connections 2023-05-22T15:27:14.495 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.486+0000 7f7927885640 1 --2- 172.21.15.16:0/4180039593 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7920158660 0x7f7920150d50 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:14.496 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.486+0000 7f7927885640 1 --2- 172.21.15.16:0/4180039593 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f7904065380 0x7f7904067830 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:14.496 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.486+0000 7f7927885640 1 --2- 172.21.15.16:0/4180039593 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7920158290 0x7f7920150810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:14.496 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.486+0000 7f7927885640 1 --2- 172.21.15.16:0/4180039593 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f79200cce20 0x7f79200b2290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:14.496 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.486+0000 7f7927885640 1 -- 172.21.15.16:0/4180039593 >> 172.21.15.16:0/4180039593 conn(0x7f7920069720 msgr2=0x7f79200b0440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:14.496 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.486+0000 7f7927885640 1 -- 172.21.15.16:0/4180039593 shutdown_connections 2023-05-22T15:27:14.496 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.486+0000 7f7927885640 1 -- 172.21.15.16:0/4180039593 wait complete. 2023-05-22T15:27:14.512 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-05-22T15:26:38.019781+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-05-22T15:26:43.570514+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2023-05-22T15:26:47.879159+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2023-05-22T15:26:48.522130+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-05-22T15:26:51.616142+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '27', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-05-22T15:27:14.513 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-05-22T15:27:14.635 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.627+0000 7f0856f0b640 1 Processor -- start 2023-05-22T15:27:14.794 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.627+0000 7f0856f0b640 1 -- start start 2023-05-22T15:27:14.799 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.627+0000 7f0856f0b640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f085006b6c0 0x7f085006bb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:14.799 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.627+0000 7f0856f0b640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0850070c30 con 0x7f0850159d40 2023-05-22T15:27:14.799 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.627+0000 7f0856f0b640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f085006c040 con 0x7f0850159970 2023-05-22T15:27:14.799 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.627+0000 7f0856f0b640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f085015dd90 con 0x7f085006b6c0 2023-05-22T15:27:14.799 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.627+0000 7f084ffff640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f0850159d40 0x7f08500cb260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:39818/0 (socket says 172.21.15.16:39818) 2023-05-22T15:27:14.800 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.627+0000 7f084ffff640 1 -- 172.21.15.16:0/1377949158 learned_addr learned my addr 172.21.15.16:0/1377949158 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:27:14.800 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.627+0000 7f08554a8640 1 --2- 172.21.15.16:0/1377949158 >> v2:172.21.15.62:3300/0 conn(0x7f085006b6c0 0x7f085006bb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:14.800 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.627+0000 7f084f7fe640 1 -- 172.21.15.16:0/1377949158 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 237347656 0 0) 0x7f0850070c30 con 0x7f0850159d40 2023-05-22T15:27:14.800 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.627+0000 7f084f7fe640 1 -- 172.21.15.16:0/1377949158 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0838003530 con 0x7f0850159d40 2023-05-22T15:27:14.800 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.627+0000 7f084f7fe640 1 -- 172.21.15.16:0/1377949158 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2984295015 0 0) 0x7f0838003530 con 0x7f0850159d40 2023-05-22T15:27:14.800 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.627+0000 7f084f7fe640 1 -- 172.21.15.16:0/1377949158 >> v2:172.21.15.62:3300/0 conn(0x7f085006b6c0 msgr2=0x7f085006bb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:14.800 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.627+0000 7f084f7fe640 1 --2- 172.21.15.16:0/1377949158 >> v2:172.21.15.62:3300/0 conn(0x7f085006b6c0 0x7f085006bb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:14.801 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.627+0000 7f084f7fe640 1 -- 172.21.15.16:0/1377949158 >> v1:172.21.15.62:6789/0 conn(0x7f0850159970 legacy=0x7f085015d680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:14.801 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.627+0000 7f084f7fe640 1 -- 172.21.15.16:0/1377949158 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0850070e60 con 0x7f0850159d40 2023-05-22T15:27:14.801 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.627+0000 7f084f7fe640 1 -- 172.21.15.16:0/1377949158 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7f0844001680 con 0x7f0850159d40 2023-05-22T15:27:14.801 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.627+0000 7f084f7fe640 1 -- 172.21.15.16:0/1377949158 >> v1:172.21.15.16:6789/0 conn(0x7f0850159d40 legacy=0x7f08500cb260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:14.801 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.627+0000 7f084f7fe640 1 --2- 172.21.15.16:0/1377949158 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0838003980 0x7f0838003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:14.801 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.627+0000 7f084f7fe640 1 --2- 172.21.15.16:0/1377949158 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f08380047d0 0x7f0838004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:14.801 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.627+0000 7f084f7fe640 1 --2- 172.21.15.16:0/1377949158 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0838005180 0x7f083800b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:14.802 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.627+0000 7f084f7fe640 1 -- 172.21.15.16:0/1377949158 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f085015dd90 con 0x7f0838005180 2023-05-22T15:27:14.802 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.627+0000 7f084f7fe640 1 -- 172.21.15.16:0/1377949158 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f083800c090 con 0x7f08380047d0 2023-05-22T15:27:14.802 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.627+0000 7f084f7fe640 1 -- 172.21.15.16:0/1377949158 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f083800c250 con 0x7f0838003980 2023-05-22T15:27:14.802 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.627+0000 7f08554a8640 1 --2- 172.21.15.16:0/1377949158 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0838005180 0x7f083800b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:14.802 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.627+0000 7f0854ca7640 1 --2- 172.21.15.16:0/1377949158 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0838003980 0x7f0838003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:14.802 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.627+0000 7f084ffff640 1 --2- 172.21.15.16:0/1377949158 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f08380047d0 0x7f0838004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:14.802 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.627+0000 7f08554a8640 1 -- 172.21.15.16:0/1377949158 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0838003980 msgr2=0x7f0838003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:14.802 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.627+0000 7f08554a8640 1 --2- 172.21.15.16:0/1377949158 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0838003980 0x7f0838003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:14.803 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.627+0000 7f08554a8640 1 -- 172.21.15.16:0/1377949158 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f08380047d0 msgr2=0x7f0838004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:14.803 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.627+0000 7f08554a8640 1 --2- 172.21.15.16:0/1377949158 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f08380047d0 0x7f0838004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:14.803 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.627+0000 7f08554a8640 1 -- 172.21.15.16:0/1377949158 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0838003530 con 0x7f0838005180 2023-05-22T15:27:14.803 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.627+0000 7f0854ca7640 1 --2- 172.21.15.16:0/1377949158 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0838003980 0x7f0838003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:27:14.803 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.631+0000 7f08554a8640 1 -- 172.21.15.16:0/1377949158 reap_dead start 2023-05-22T15:27:14.803 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.631+0000 7f08554a8640 1 --2- 172.21.15.16:0/1377949158 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0838005180 0x7f083800b990 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f0848009fc0 tx=0x7f084801cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=824072d72167884c server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:14.803 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.631+0000 7f084f7fe640 1 -- 172.21.15.16:0/1377949158 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0848003ba0 con 0x7f0838005180 2023-05-22T15:27:14.804 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.631+0000 7f084f7fe640 1 -- 172.21.15.16:0/1377949158 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f084800f040 con 0x7f0838005180 2023-05-22T15:27:14.804 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.631+0000 7f084f7fe640 1 -- 172.21.15.16:0/1377949158 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08480a59a0 con 0x7f0838005180 2023-05-22T15:27:14.804 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.631+0000 7f0856f0b640 1 -- 172.21.15.16:0/1377949158 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0838005180 msgr2=0x7f083800b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:14.804 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.631+0000 7f0856f0b640 1 --2- 172.21.15.16:0/1377949158 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0838005180 0x7f083800b990 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f0848009fc0 tx=0x7f084801cca0 comp rx=0 tx=0).stop 2023-05-22T15:27:14.804 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.631+0000 7f0856f0b640 1 -- 172.21.15.16:0/1377949158 shutdown_connections 2023-05-22T15:27:14.804 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.631+0000 7f0856f0b640 1 --2- 172.21.15.16:0/1377949158 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0838005180 0x7f083800b990 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:14.804 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.631+0000 7f0856f0b640 1 -- 172.21.15.16:0/1377949158 >> 172.21.15.16:0/1377949158 conn(0x7f0850069720 msgr2=0x7f0850069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:14.804 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.631+0000 7f0856f0b640 1 -- 172.21.15.16:0/1377949158 shutdown_connections 2023-05-22T15:27:14.805 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.631+0000 7f0856f0b640 1 -- 172.21.15.16:0/1377949158 wait complete. 2023-05-22T15:27:14.805 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.631+0000 7f0856f0b640 1 Processor -- start 2023-05-22T15:27:14.805 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.631+0000 7f0856f0b640 1 -- start start 2023-05-22T15:27:14.805 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.635+0000 7f0856f0b640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0838005180 0x7f08501516d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:14.805 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.635+0000 7f0856f0b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08500b1630 0x7f08500b1aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:14.805 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.635+0000 7f0856f0b640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f08501f0d50 0x7f08501f11d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:14.805 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.635+0000 7f0856f0b640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08500cba90 con 0x7f08500b1630 2023-05-22T15:27:14.806 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.635+0000 7f0856f0b640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08500b1fe0 con 0x7f08501f0d50 2023-05-22T15:27:14.806 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.635+0000 7f0856f0b640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08501f1b40 con 0x7f0838005180 2023-05-22T15:27:14.806 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.635+0000 7f084ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08500b1630 0x7f08500b1aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:14.806 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.635+0000 7f084ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08500b1630 0x7f08500b1aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37996/0 (socket says 172.21.15.16:37996) 2023-05-22T15:27:14.806 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.635+0000 7f084ffff640 1 -- 172.21.15.16:0/2597870014 learned_addr learned my addr 172.21.15.16:0/2597870014 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:14.806 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.635+0000 7f0854ca7640 1 --2- 172.21.15.16:0/2597870014 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0838005180 0x7f08501516d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:14.806 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.635+0000 7f08554a8640 1 --2- 172.21.15.16:0/2597870014 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f08501f0d50 0x7f08501f11d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:14.806 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.635+0000 7f084ffff640 1 -- 172.21.15.16:0/2597870014 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0838005180 msgr2=0x7f08501516d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:14.807 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.635+0000 7f084ffff640 1 --2- 172.21.15.16:0/2597870014 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0838005180 0x7f08501516d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:14.807 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.635+0000 7f084ffff640 1 -- 172.21.15.16:0/2597870014 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f08501f0d50 msgr2=0x7f08501f11d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:14.807 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.635+0000 7f084ffff640 1 --2- 172.21.15.16:0/2597870014 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f08501f0d50 0x7f08501f11d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:14.807 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.635+0000 7f084ffff640 1 -- 172.21.15.16:0/2597870014 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0850071420 con 0x7f08500b1630 2023-05-22T15:27:14.807 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.635+0000 7f08554a8640 1 --2- 172.21.15.16:0/2597870014 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f08501f0d50 0x7f08501f11d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:27:14.807 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.635+0000 7f0854ca7640 1 --2- 172.21.15.16:0/2597870014 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0838005180 0x7f08501516d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:27:14.807 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.635+0000 7f084ffff640 1 --2- 172.21.15.16:0/2597870014 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08500b1630 0x7f08500b1aa0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f0844011810 tx=0x7f0844011cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:14.807 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.635+0000 7f084d7fa640 1 -- 172.21.15.16:0/2597870014 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f084400fb00 con 0x7f08500b1630 2023-05-22T15:27:14.808 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.635+0000 7f084d7fa640 1 -- 172.21.15.16:0/2597870014 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f084400fc60 con 0x7f08500b1630 2023-05-22T15:27:14.808 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.635+0000 7f084d7fa640 1 -- 172.21.15.16:0/2597870014 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0844015410 con 0x7f08500b1630 2023-05-22T15:27:14.808 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.635+0000 7f0856f0b640 1 -- 172.21.15.16:0/2597870014 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0850071140 con 0x7f08500b1630 2023-05-22T15:27:14.808 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.635+0000 7f0856f0b640 1 -- 172.21.15.16:0/2597870014 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0850071700 con 0x7f08500b1630 2023-05-22T15:27:14.808 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.635+0000 7f0856f0b640 1 -- 172.21.15.16:0/2597870014 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f085004dbe0 con 0x7f08500b1630 2023-05-22T15:27:14.808 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.635+0000 7f084d7fa640 1 -- 172.21.15.16:0/2597870014 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7f0844015570 con 0x7f08500b1630 2023-05-22T15:27:14.808 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.635+0000 7f084d7fa640 1 --2- 172.21.15.16:0/2597870014 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f083c065380 0x7f083c067830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:14.809 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.635+0000 7f084d7fa640 1 -- 172.21.15.16:0/2597870014 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f0844085df0 con 0x7f08500b1630 2023-05-22T15:27:14.809 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.635+0000 7f0854ca7640 1 --2- 172.21.15.16:0/2597870014 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f083c065380 0x7f083c067830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:14.809 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.639+0000 7f0854ca7640 1 --2- 172.21.15.16:0/2597870014 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f083c065380 0x7f083c067830 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f08500c9d70 tx=0x7f0840009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:14.809 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.639+0000 7f084d7fa640 1 -- 172.21.15.16:0/2597870014 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f08440537b0 con 0x7f08500b1630 2023-05-22T15:27:14.809 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.791+0000 7f0856f0b640 1 -- 172.21.15.16:0/2597870014 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f08500caf50 con 0x7f08500b1630 2023-05-22T15:27:14.809 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.791+0000 7f084d7fa640 1 -- 172.21.15.16:0/2597870014 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) v1 ==== 93+0+10 (secure 0 0 0) 0x7f0844056dd0 con 0x7f08500b1630 2023-05-22T15:27:14.810 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2023-05-22T15:27:14.810 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.791+0000 7f0856f0b640 1 -- 172.21.15.16:0/2597870014 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f083c065380 msgr2=0x7f083c067830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:14.811 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.791+0000 7f0856f0b640 1 --2- 172.21.15.16:0/2597870014 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f083c065380 0x7f083c067830 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f08500c9d70 tx=0x7f0840009040 comp rx=0 tx=0).stop 2023-05-22T15:27:14.811 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.791+0000 7f0856f0b640 1 -- 172.21.15.16:0/2597870014 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08500b1630 msgr2=0x7f08500b1aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:14.811 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.791+0000 7f0856f0b640 1 --2- 172.21.15.16:0/2597870014 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08500b1630 0x7f08500b1aa0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f0844011810 tx=0x7f0844011cd0 comp rx=0 tx=0).stop 2023-05-22T15:27:14.811 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.791+0000 7f0856f0b640 1 -- 172.21.15.16:0/2597870014 shutdown_connections 2023-05-22T15:27:14.811 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.791+0000 7f0856f0b640 1 --2- 172.21.15.16:0/2597870014 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f08501f0d50 0x7f08501f11d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:14.811 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.791+0000 7f0856f0b640 1 --2- 172.21.15.16:0/2597870014 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08500b1630 0x7f08500b1aa0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:14.811 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.791+0000 7f0856f0b640 1 --2- 172.21.15.16:0/2597870014 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f083c065380 0x7f083c067830 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:14.812 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.791+0000 7f0856f0b640 1 --2- 172.21.15.16:0/2597870014 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0838005180 0x7f08501516d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:14.812 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.791+0000 7f0856f0b640 1 -- 172.21.15.16:0/2597870014 >> 172.21.15.16:0/2597870014 conn(0x7f0850069720 msgr2=0x7f08501542e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:14.812 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.791+0000 7f0856f0b640 1 -- 172.21.15.16:0/2597870014 shutdown_connections 2023-05-22T15:27:14.812 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.791+0000 7f0856f0b640 1 -- 172.21.15.16:0/2597870014 wait complete. 2023-05-22T15:27:14.816 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-05-22T15:27:14.932 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.923+0000 7f5dfd27b640 1 Processor -- start 2023-05-22T15:27:14.933 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.923+0000 7f5dfd27b640 1 -- start start 2023-05-22T15:27:14.934 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.923+0000 7f5dfd27b640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f5df815c8b0 0x7f5df80cee30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:14.934 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.923+0000 7f5dfd27b640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f5df806bf60 0x7f5df815cc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:14.934 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.923+0000 7f5dfd27b640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5df806c3e0 con 0x7f5df815d1c0 2023-05-22T15:27:14.934 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.923+0000 7f5dfd27b640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5df815a0c0 con 0x7f5df806bf60 2023-05-22T15:27:14.934 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.923+0000 7f5dfd27b640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5df80ce160 con 0x7f5df815c8b0 2023-05-22T15:27:14.934 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.927+0000 7f5deffff640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f5df806bf60 0x7f5df815cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:14.935 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.927+0000 7f5deffff640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f5df806bf60 0x7f5df815cc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38002/0 (socket says 172.21.15.16:38002) 2023-05-22T15:27:14.935 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.927+0000 7f5deffff640 1 -- 172.21.15.16:0/555716753 learned_addr learned my addr 172.21.15.16:0/555716753 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:14.935 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.927+0000 7f5df6ffd640 1 --2- 172.21.15.16:0/555716753 >> v2:172.21.15.62:3300/0 conn(0x7f5df815c8b0 0x7f5df80cee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:14.935 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.927+0000 7f5deffff640 1 -- 172.21.15.16:0/555716753 >> v2:172.21.15.62:3300/0 conn(0x7f5df815c8b0 msgr2=0x7f5df80cee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:14.935 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.927+0000 7f5deffff640 1 --2- 172.21.15.16:0/555716753 >> v2:172.21.15.62:3300/0 conn(0x7f5df815c8b0 0x7f5df80cee30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:14.935 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.927+0000 7f5deffff640 1 -- 172.21.15.16:0/555716753 >> v1:172.21.15.62:6789/0 conn(0x7f5df815d1c0 legacy=0x7f5df806db80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:14.935 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.927+0000 7f5deffff640 1 -- 172.21.15.16:0/555716753 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5df806c640 con 0x7f5df806bf60 2023-05-22T15:27:14.936 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.927+0000 7f5deffff640 1 --2- 172.21.15.16:0/555716753 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5df806bf60 0x7f5df815cc80 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f5de0009870 tx=0x7f5de001c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52a9b46cbc6349e6 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:14.936 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.927+0000 7f5df67fc640 1 -- 172.21.15.16:0/555716753 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5de00ac640 con 0x7f5df806bf60 2023-05-22T15:27:14.936 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.927+0000 7f5df67fc640 1 -- 172.21.15.16:0/555716753 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5de00ac7a0 con 0x7f5df806bf60 2023-05-22T15:27:14.936 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.927+0000 7f5df67fc640 1 -- 172.21.15.16:0/555716753 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5de00acb30 con 0x7f5df806bf60 2023-05-22T15:27:14.937 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.927+0000 7f5dfd27b640 1 -- 172.21.15.16:0/555716753 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5df806bf60 msgr2=0x7f5df815cc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:14.937 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.927+0000 7f5dfd27b640 1 --2- 172.21.15.16:0/555716753 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5df806bf60 0x7f5df815cc80 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f5de0009870 tx=0x7f5de001c4d0 comp rx=0 tx=0).stop 2023-05-22T15:27:14.938 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.927+0000 7f5dfd27b640 1 -- 172.21.15.16:0/555716753 shutdown_connections 2023-05-22T15:27:14.938 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.927+0000 7f5dfd27b640 1 --2- 172.21.15.16:0/555716753 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5df806bf60 0x7f5df815cc80 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:14.938 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.927+0000 7f5dfd27b640 1 --2- 172.21.15.16:0/555716753 >> v2:172.21.15.62:3300/0 conn(0x7f5df815c8b0 0x7f5df80cee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:14.938 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.927+0000 7f5dfd27b640 1 -- 172.21.15.16:0/555716753 >> 172.21.15.16:0/555716753 conn(0x7f5df806f5a0 msgr2=0x7f5df806f9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:14.938 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.927+0000 7f5dfd27b640 1 -- 172.21.15.16:0/555716753 shutdown_connections 2023-05-22T15:27:14.938 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.927+0000 7f5dfd27b640 1 -- 172.21.15.16:0/555716753 wait complete. 2023-05-22T15:27:14.940 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.931+0000 7f5dfd27b640 1 Processor -- start 2023-05-22T15:27:14.940 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.931+0000 7f5dfd27b640 1 -- start start 2023-05-22T15:27:14.940 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.931+0000 7f5dfd27b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5df806bf60 0x7f5df815d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:14.940 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.931+0000 7f5dfd27b640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5df815c8b0 0x7f5df815dcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:14.940 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.931+0000 7f5dfd27b640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5df815d1c0 0x7f5df80b3aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:14.941 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.931+0000 7f5dfd27b640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5df81581e0 con 0x7f5df806bf60 2023-05-22T15:27:14.941 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.931+0000 7f5dfd27b640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5df80b4c40 con 0x7f5df815d1c0 2023-05-22T15:27:14.941 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.931+0000 7f5dfd27b640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5df8142e20 con 0x7f5df815c8b0 2023-05-22T15:27:14.942 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.931+0000 7f5df6ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5df806bf60 0x7f5df815d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:14.942 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.931+0000 7f5df6ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5df806bf60 0x7f5df815d780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38006/0 (socket says 172.21.15.16:38006) 2023-05-22T15:27:14.942 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.931+0000 7f5df6ffd640 1 -- 172.21.15.16:0/1430461975 learned_addr learned my addr 172.21.15.16:0/1430461975 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:14.942 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.931+0000 7f5deffff640 1 --2- 172.21.15.16:0/1430461975 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5df815c8b0 0x7f5df815dcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:14.942 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.931+0000 7f5df77fe640 1 --2- 172.21.15.16:0/1430461975 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5df815d1c0 0x7f5df80b3aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:14.943 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.931+0000 7f5df6ffd640 1 -- 172.21.15.16:0/1430461975 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5df815c8b0 msgr2=0x7f5df815dcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:14.943 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.931+0000 7f5df6ffd640 1 --2- 172.21.15.16:0/1430461975 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5df815c8b0 0x7f5df815dcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:14.943 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.931+0000 7f5df6ffd640 1 -- 172.21.15.16:0/1430461975 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5df815d1c0 msgr2=0x7f5df80b3aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:14.943 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.931+0000 7f5df6ffd640 1 --2- 172.21.15.16:0/1430461975 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5df815d1c0 0x7f5df80b3aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:14.943 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.931+0000 7f5df6ffd640 1 -- 172.21.15.16:0/1430461975 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5de00094d0 con 0x7f5df806bf60 2023-05-22T15:27:14.943 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.931+0000 7f5df6ffd640 1 --2- 172.21.15.16:0/1430461975 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5df806bf60 0x7f5df815d780 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f5de800e8a0 tx=0x7f5de800eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:14.943 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.931+0000 7f5def7fe640 1 -- 172.21.15.16:0/1430461975 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5de8009510 con 0x7f5df806bf60 2023-05-22T15:27:14.944 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.931+0000 7f5def7fe640 1 -- 172.21.15.16:0/1430461975 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5de800c940 con 0x7f5df806bf60 2023-05-22T15:27:14.944 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.931+0000 7f5def7fe640 1 -- 172.21.15.16:0/1430461975 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5de800ccb0 con 0x7f5df806bf60 2023-05-22T15:27:14.944 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.931+0000 7f5dfd27b640 1 -- 172.21.15.16:0/1430461975 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5df806e290 con 0x7f5df806bf60 2023-05-22T15:27:14.944 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.931+0000 7f5dfd27b640 1 -- 172.21.15.16:0/1430461975 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5df80ce930 con 0x7f5df806bf60 2023-05-22T15:27:14.945 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.931+0000 7f5dfd27b640 1 -- 172.21.15.16:0/1430461975 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5df806e420 con 0x7f5df806bf60 2023-05-22T15:27:14.945 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.931+0000 7f5def7fe640 1 -- 172.21.15.16:0/1430461975 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7f5de8013470 con 0x7f5df806bf60 2023-05-22T15:27:14.945 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.935+0000 7f5def7fe640 1 --2- 172.21.15.16:0/1430461975 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f5dd8065380 0x7f5dd8067830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:14.945 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.935+0000 7f5def7fe640 1 -- 172.21.15.16:0/1430461975 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f5de8082c60 con 0x7f5df806bf60 2023-05-22T15:27:14.945 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.935+0000 7f5deffff640 1 --2- 172.21.15.16:0/1430461975 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f5dd8065380 0x7f5dd8067830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:14.945 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.935+0000 7f5deffff640 1 --2- 172.21.15.16:0/1430461975 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f5dd8065380 0x7f5dd8067830 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f5de00ae000 tx=0x7f5de00a7000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:14.946 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:14.939+0000 7f5def7fe640 1 -- 172.21.15.16:0/1430461975 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f5de8050580 con 0x7f5df806bf60 2023-05-22T15:27:15.096 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.087+0000 7f5dfd27b640 1 -- 172.21.15.16:0/1430461975 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f5df804dbe0 con 0x7f5df806bf60 2023-05-22T15:27:15.096 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.087+0000 7f5def7fe640 1 -- 172.21.15.16:0/1430461975 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v27) v1 ==== 92+0+10 (secure 0 0 0) 0x7f5de8053ba0 con 0x7f5df806bf60 2023-05-22T15:27:15.097 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 8 2023-05-22T15:27:15.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.091+0000 7f5dfd27b640 1 -- 172.21.15.16:0/1430461975 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f5dd8065380 msgr2=0x7f5dd8067830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:15.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.091+0000 7f5dfd27b640 1 --2- 172.21.15.16:0/1430461975 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f5dd8065380 0x7f5dd8067830 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f5de00ae000 tx=0x7f5de00a7000 comp rx=0 tx=0).stop 2023-05-22T15:27:15.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.091+0000 7f5dfd27b640 1 -- 172.21.15.16:0/1430461975 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5df806bf60 msgr2=0x7f5df815d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:15.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.091+0000 7f5dfd27b640 1 --2- 172.21.15.16:0/1430461975 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5df806bf60 0x7f5df815d780 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f5de800e8a0 tx=0x7f5de800eca0 comp rx=0 tx=0).stop 2023-05-22T15:27:15.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.091+0000 7f5dfd27b640 1 -- 172.21.15.16:0/1430461975 shutdown_connections 2023-05-22T15:27:15.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.091+0000 7f5dfd27b640 1 --2- 172.21.15.16:0/1430461975 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5df815d1c0 0x7f5df80b3aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:15.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.091+0000 7f5dfd27b640 1 --2- 172.21.15.16:0/1430461975 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f5dd8065380 0x7f5dd8067830 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:15.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.091+0000 7f5dfd27b640 1 --2- 172.21.15.16:0/1430461975 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5df815c8b0 0x7f5df815dcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:15.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.091+0000 7f5dfd27b640 1 --2- 172.21.15.16:0/1430461975 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5df806bf60 0x7f5df815d780 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:15.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.091+0000 7f5dfd27b640 1 -- 172.21.15.16:0/1430461975 >> 172.21.15.16:0/1430461975 conn(0x7f5df806f5a0 msgr2=0x7f5df815b680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:15.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.091+0000 7f5dfd27b640 1 -- 172.21.15.16:0/1430461975 shutdown_connections 2023-05-22T15:27:15.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.091+0000 7f5dfd27b640 1 -- 172.21.15.16:0/1430461975 wait complete. 2023-05-22T15:27:15.115 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-05-22T15:27:15.233 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.223+0000 7ff3063de640 1 Processor -- start 2023-05-22T15:27:15.234 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.223+0000 7ff3063de640 1 -- start start 2023-05-22T15:27:15.235 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.227+0000 7ff3063de640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7ff3000c67d0 0x7ff3000cf730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:15.235 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.227+0000 7ff3063de640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff3000b2880 0x7ff3000cbfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:15.235 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.227+0000 7ff3063de640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff3000cc520 0x7ff3000ce160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:15.235 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.227+0000 7ff3063de640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff30015a300 con 0x7ff3000cc520 2023-05-22T15:27:15.235 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.227+0000 7ff3063de640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3000cfc70 con 0x7ff3000c67d0 2023-05-22T15:27:15.235 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.227+0000 7ff3063de640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3000c6ba0 con 0x7ff3000b2880 2023-05-22T15:27:15.236 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.227+0000 7ff30497b640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff3000cc520 0x7ff3000ce160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:15.236 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.227+0000 7ff30497b640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff3000cc520 0x7ff3000ce160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38008/0 (socket says 172.21.15.16:38008) 2023-05-22T15:27:15.236 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.227+0000 7ff30497b640 1 -- 172.21.15.16:0/1216071871 learned_addr learned my addr 172.21.15.16:0/1216071871 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:15.236 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.227+0000 7ff2f7fff640 1 --2- 172.21.15.16:0/1216071871 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff3000b2880 0x7ff3000cbfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:15.236 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.227+0000 7ff2fffff640 1 --2- 172.21.15.16:0/1216071871 >> v2:172.21.15.62:3300/0 conn(0x7ff3000c67d0 0x7ff3000cf730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:15.236 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.227+0000 7ff30497b640 1 -- 172.21.15.16:0/1216071871 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff3000b2880 msgr2=0x7ff3000cbfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:15.236 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.227+0000 7ff30497b640 1 --2- 172.21.15.16:0/1216071871 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff3000b2880 0x7ff3000cbfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:15.237 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.227+0000 7ff30497b640 1 -- 172.21.15.16:0/1216071871 >> v2:172.21.15.62:3300/0 conn(0x7ff3000c67d0 msgr2=0x7ff3000cf730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:15.237 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.227+0000 7ff30497b640 1 --2- 172.21.15.16:0/1216071871 >> v2:172.21.15.62:3300/0 conn(0x7ff3000c67d0 0x7ff3000cf730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:15.237 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.227+0000 7ff30497b640 1 -- 172.21.15.16:0/1216071871 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3000cfef0 con 0x7ff3000cc520 2023-05-22T15:27:15.238 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.227+0000 7ff30497b640 1 --2- 172.21.15.16:0/1216071871 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff3000cc520 0x7ff3000ce160 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7ff30006eac0 tx=0x7ff2f801c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e12db007784c3022 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:15.238 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.227+0000 7ff2ff7fe640 1 -- 172.21.15.16:0/1216071871 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2f80a64e0 con 0x7ff3000cc520 2023-05-22T15:27:15.238 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.227+0000 7ff2ff7fe640 1 -- 172.21.15.16:0/1216071871 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2f80a6640 con 0x7ff3000cc520 2023-05-22T15:27:15.238 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.227+0000 7ff2ff7fe640 1 -- 172.21.15.16:0/1216071871 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2f80a69d0 con 0x7ff3000cc520 2023-05-22T15:27:15.238 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.227+0000 7ff3063de640 1 -- 172.21.15.16:0/1216071871 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff3000cc520 msgr2=0x7ff3000ce160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:15.239 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.227+0000 7ff3063de640 1 --2- 172.21.15.16:0/1216071871 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff3000cc520 0x7ff3000ce160 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7ff30006eac0 tx=0x7ff2f801c650 comp rx=0 tx=0).stop 2023-05-22T15:27:15.239 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.227+0000 7ff3063de640 1 -- 172.21.15.16:0/1216071871 shutdown_connections 2023-05-22T15:27:15.239 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.227+0000 7ff3063de640 1 --2- 172.21.15.16:0/1216071871 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff3000cc520 0x7ff3000ce160 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:15.239 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.227+0000 7ff3063de640 1 --2- 172.21.15.16:0/1216071871 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff3000b2880 0x7ff3000cbfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:15.239 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.227+0000 7ff3063de640 1 --2- 172.21.15.16:0/1216071871 >> v2:172.21.15.62:3300/0 conn(0x7ff3000c67d0 0x7ff3000cf730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:15.239 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.227+0000 7ff3063de640 1 -- 172.21.15.16:0/1216071871 >> 172.21.15.16:0/1216071871 conn(0x7ff30006f5a0 msgr2=0x7ff30006f9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:15.239 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.227+0000 7ff3063de640 1 -- 172.21.15.16:0/1216071871 shutdown_connections 2023-05-22T15:27:15.239 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.227+0000 7ff3063de640 1 -- 172.21.15.16:0/1216071871 wait complete. 2023-05-22T15:27:15.241 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.231+0000 7ff3063de640 1 Processor -- start 2023-05-22T15:27:15.241 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.231+0000 7ff3063de640 1 -- start start 2023-05-22T15:27:15.241 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.231+0000 7ff3063de640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff3000b2880 0x7ff30006b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:15.242 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.231+0000 7ff3063de640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff3000c67d0 0x7ff30014c260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:15.242 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.231+0000 7ff3063de640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff3000cc520 0x7ff300159500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:15.242 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.231+0000 7ff3063de640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff300158140 con 0x7ff3000cc520 2023-05-22T15:27:15.242 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.231+0000 7ff3063de640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff30015c300 con 0x7ff3000c67d0 2023-05-22T15:27:15.242 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.231+0000 7ff3063de640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff30015c5e0 con 0x7ff3000b2880 2023-05-22T15:27:15.242 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.231+0000 7ff30497b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff3000cc520 0x7ff300159500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:15.242 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.231+0000 7ff30497b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff3000cc520 0x7ff300159500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38020/0 (socket says 172.21.15.16:38020) 2023-05-22T15:27:15.243 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.231+0000 7ff30497b640 1 -- 172.21.15.16:0/2980838823 learned_addr learned my addr 172.21.15.16:0/2980838823 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:15.243 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.231+0000 7ff30497b640 1 -- 172.21.15.16:0/2980838823 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff3000b2880 msgr2=0x7ff30006b310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:15.243 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.231+0000 7ff30497b640 1 --2- 172.21.15.16:0/2980838823 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff3000b2880 0x7ff30006b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:15.243 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.231+0000 7ff30497b640 1 -- 172.21.15.16:0/2980838823 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff3000c67d0 msgr2=0x7ff30014c260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:15.244 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.231+0000 7ff30497b640 1 --2- 172.21.15.16:0/2980838823 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff3000c67d0 0x7ff30014c260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:15.244 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.231+0000 7ff30497b640 1 -- 172.21.15.16:0/2980838823 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2f8009b30 con 0x7ff3000cc520 2023-05-22T15:27:15.244 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.231+0000 7ff30497b640 1 --2- 172.21.15.16:0/2980838823 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff3000cc520 0x7ff300159500 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7ff2f801ad00 tx=0x7ff2f80193f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:15.244 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.231+0000 7ff2fd7fa640 1 -- 172.21.15.16:0/2980838823 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2f80b4070 con 0x7ff3000cc520 2023-05-22T15:27:15.244 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.231+0000 7ff2fd7fa640 1 -- 172.21.15.16:0/2980838823 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2f8009150 con 0x7ff3000cc520 2023-05-22T15:27:15.244 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.231+0000 7ff3063de640 1 -- 172.21.15.16:0/2980838823 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3000d0a70 con 0x7ff3000cc520 2023-05-22T15:27:15.244 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.231+0000 7ff2fd7fa640 1 -- 172.21.15.16:0/2980838823 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2f80a69f0 con 0x7ff3000cc520 2023-05-22T15:27:15.245 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.231+0000 7ff3063de640 1 -- 172.21.15.16:0/2980838823 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3000d0790 con 0x7ff3000cc520 2023-05-22T15:27:15.245 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.231+0000 7ff3063de640 1 -- 172.21.15.16:0/2980838823 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3000cc9a0 con 0x7ff3000cc520 2023-05-22T15:27:15.245 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.235+0000 7ff2fd7fa640 1 -- 172.21.15.16:0/2980838823 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7ff2f80b8070 con 0x7ff3000cc520 2023-05-22T15:27:15.245 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.235+0000 7ff2fd7fa640 1 --2- 172.21.15.16:0/2980838823 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ff2e0065380 0x7ff2e0067830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:15.246 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.235+0000 7ff2fd7fa640 1 -- 172.21.15.16:0/2980838823 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6813+0+0 (secure 0 0 0) 0x7ff2f80aa070 con 0x7ff3000cc520 2023-05-22T15:27:15.246 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.235+0000 7ff2fffff640 1 --2- 172.21.15.16:0/2980838823 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ff2e0065380 0x7ff2e0067830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:15.246 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.235+0000 7ff2fffff640 1 --2- 172.21.15.16:0/2980838823 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ff2e0065380 0x7ff2e0067830 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7ff3001505f0 tx=0x7ff2e8008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:15.246 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.239+0000 7ff2fd7fa640 1 -- 172.21.15.16:0/2980838823 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7ff2f80ea320 con 0x7ff3000cc520 2023-05-22T15:27:15.396 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.387+0000 7ff3063de640 1 -- 172.21.15.16:0/2980838823 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7ff30004dbe0 con 0x7ff3000cc520 2023-05-22T15:27:15.397 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.387+0000 7ff2fd7fa640 1 -- 172.21.15.16:0/2980838823 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) v1 ==== 104+0+11 (secure 0 0 0) 0x7ff2f80ed940 con 0x7ff3000cc520 2023-05-22T15:27:15.397 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2023-05-22T15:27:15.400 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.391+0000 7ff3063de640 1 -- 172.21.15.16:0/2980838823 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ff2e0065380 msgr2=0x7ff2e0067830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:15.400 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.391+0000 7ff3063de640 1 --2- 172.21.15.16:0/2980838823 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ff2e0065380 0x7ff2e0067830 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7ff3001505f0 tx=0x7ff2e8008040 comp rx=0 tx=0).stop 2023-05-22T15:27:15.400 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.391+0000 7ff3063de640 1 -- 172.21.15.16:0/2980838823 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff3000cc520 msgr2=0x7ff300159500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:15.400 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.391+0000 7ff3063de640 1 --2- 172.21.15.16:0/2980838823 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff3000cc520 0x7ff300159500 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7ff2f801ad00 tx=0x7ff2f80193f0 comp rx=0 tx=0).stop 2023-05-22T15:27:15.400 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.391+0000 7ff3063de640 1 -- 172.21.15.16:0/2980838823 shutdown_connections 2023-05-22T15:27:15.400 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.391+0000 7ff3063de640 1 --2- 172.21.15.16:0/2980838823 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff3000cc520 0x7ff300159500 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:15.401 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.391+0000 7ff3063de640 1 --2- 172.21.15.16:0/2980838823 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff3000c67d0 0x7ff30014c260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:15.401 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.391+0000 7ff3063de640 1 --2- 172.21.15.16:0/2980838823 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ff2e0065380 0x7ff2e0067830 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:15.401 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.391+0000 7ff3063de640 1 --2- 172.21.15.16:0/2980838823 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff3000b2880 0x7ff30006b310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:15.401 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.391+0000 7ff3063de640 1 -- 172.21.15.16:0/2980838823 >> 172.21.15.16:0/2980838823 conn(0x7ff30006f5a0 msgr2=0x7ff30015cd20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:15.401 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.391+0000 7ff3063de640 1 -- 172.21.15.16:0/2980838823 shutdown_connections 2023-05-22T15:27:15.401 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.391+0000 7ff3063de640 1 -- 172.21.15.16:0/2980838823 wait complete. 2023-05-22T15:27:15.417 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-05-22T15:27:15.538 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.531+0000 7fb2976a3640 1 Processor -- start 2023-05-22T15:27:15.539 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.531+0000 7fb2976a3640 1 -- start start 2023-05-22T15:27:15.540 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.531+0000 7fb2976a3640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb290159970 0x7fb290159d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:15.540 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.531+0000 7fb2976a3640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb29015d680 0x7fb2900cb260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:15.540 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.531+0000 7fb2976a3640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fb29015dad0 0x7fb29006b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:15.540 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.531+0000 7fb2976a3640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb290070c30 con 0x7fb290159970 2023-05-22T15:27:15.540 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.531+0000 7fb2976a3640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb290070e60 con 0x7fb29015dad0 2023-05-22T15:27:15.541 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.531+0000 7fb2976a3640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb290071140 con 0x7fb29015d680 2023-05-22T15:27:15.541 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.531+0000 7fb29543f640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb290159970 0x7fb290159d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:15.541 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.531+0000 7fb29543f640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb290159970 0x7fb290159d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38032/0 (socket says 172.21.15.16:38032) 2023-05-22T15:27:15.541 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.531+0000 7fb29543f640 1 -- 172.21.15.16:0/2135148987 learned_addr learned my addr 172.21.15.16:0/2135148987 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:15.541 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.531+0000 7fb294c3e640 1 --2- 172.21.15.16:0/2135148987 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb29015d680 0x7fb2900cb260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:15.541 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.531+0000 7fb295c40640 1 --2- 172.21.15.16:0/2135148987 >> v2:172.21.15.62:3300/0 conn(0x7fb29015dad0 0x7fb29006b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:15.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.531+0000 7fb29543f640 1 -- 172.21.15.16:0/2135148987 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb29015d680 msgr2=0x7fb2900cb260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:15.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.531+0000 7fb29543f640 1 --2- 172.21.15.16:0/2135148987 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb29015d680 0x7fb2900cb260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:15.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.531+0000 7fb29543f640 1 -- 172.21.15.16:0/2135148987 >> v2:172.21.15.62:3300/0 conn(0x7fb29015dad0 msgr2=0x7fb29006b6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:15.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.531+0000 7fb29543f640 1 --2- 172.21.15.16:0/2135148987 >> v2:172.21.15.62:3300/0 conn(0x7fb29015dad0 0x7fb29006b6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:15.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.531+0000 7fb29543f640 1 -- 172.21.15.16:0/2135148987 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb290071420 con 0x7fb290159970 2023-05-22T15:27:15.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.531+0000 7fb295c40640 1 --2- 172.21.15.16:0/2135148987 >> v2:172.21.15.62:3300/0 conn(0x7fb29015dad0 0x7fb29006b6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T15:27:15.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.535+0000 7fb29543f640 1 --2- 172.21.15.16:0/2135148987 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb290159970 0x7fb290159d40 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fb2900b40e0 tx=0x7fb28401c440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec34df40d58a9037 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:15.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.535+0000 7fb283fff640 1 -- 172.21.15.16:0/2135148987 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2840ac640 con 0x7fb290159970 2023-05-22T15:27:15.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.535+0000 7fb283fff640 1 -- 172.21.15.16:0/2135148987 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2840ac7a0 con 0x7fb290159970 2023-05-22T15:27:15.544 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.535+0000 7fb283fff640 1 -- 172.21.15.16:0/2135148987 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2840acb30 con 0x7fb290159970 2023-05-22T15:27:15.544 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.535+0000 7fb2976a3640 1 -- 172.21.15.16:0/2135148987 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb290159970 msgr2=0x7fb290159d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:15.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.535+0000 7fb2976a3640 1 --2- 172.21.15.16:0/2135148987 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb290159970 0x7fb290159d40 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fb2900b40e0 tx=0x7fb28401c440 comp rx=0 tx=0).stop 2023-05-22T15:27:15.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.535+0000 7fb2976a3640 1 -- 172.21.15.16:0/2135148987 shutdown_connections 2023-05-22T15:27:15.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.535+0000 7fb2976a3640 1 --2- 172.21.15.16:0/2135148987 >> v2:172.21.15.62:3300/0 conn(0x7fb29015dad0 0x7fb29006b6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:15.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.535+0000 7fb2976a3640 1 --2- 172.21.15.16:0/2135148987 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb29015d680 0x7fb2900cb260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:15.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.535+0000 7fb2976a3640 1 --2- 172.21.15.16:0/2135148987 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb290159970 0x7fb290159d40 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:15.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.535+0000 7fb2976a3640 1 -- 172.21.15.16:0/2135148987 >> 172.21.15.16:0/2135148987 conn(0x7fb290069720 msgr2=0x7fb290069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:15.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.535+0000 7fb2976a3640 1 -- 172.21.15.16:0/2135148987 shutdown_connections 2023-05-22T15:27:15.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.535+0000 7fb2976a3640 1 -- 172.21.15.16:0/2135148987 wait complete. 2023-05-22T15:27:15.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.535+0000 7fb2976a3640 1 Processor -- start 2023-05-22T15:27:15.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.535+0000 7fb2976a3640 1 -- start start 2023-05-22T15:27:15.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.535+0000 7fb2976a3640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb290159970 0x7fb290154570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:15.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.535+0000 7fb2976a3640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb29015d680 0x7fb2900c8300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:15.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.535+0000 7fb2976a3640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb29015dad0 0x7fb2900c1de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:15.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.535+0000 7fb2976a3640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2901550f0 con 0x7fb290159970 2023-05-22T15:27:15.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.535+0000 7fb2976a3640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2901553d0 con 0x7fb29015dad0 2023-05-22T15:27:15.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.535+0000 7fb2976a3640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2901556b0 con 0x7fb29015d680 2023-05-22T15:27:15.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.535+0000 7fb29543f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb290159970 0x7fb290154570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:15.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.535+0000 7fb29543f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb290159970 0x7fb290154570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38048/0 (socket says 172.21.15.16:38048) 2023-05-22T15:27:15.549 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.539+0000 7fb29543f640 1 -- 172.21.15.16:0/3651754265 learned_addr learned my addr 172.21.15.16:0/3651754265 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:15.549 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.539+0000 7fb294c3e640 1 --2- 172.21.15.16:0/3651754265 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb29015d680 0x7fb2900c8300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:15.550 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.539+0000 7fb29543f640 1 -- 172.21.15.16:0/3651754265 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb29015d680 msgr2=0x7fb2900c8300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:15.550 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.539+0000 7fb295c40640 1 --2- 172.21.15.16:0/3651754265 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb29015dad0 0x7fb2900c1de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:15.550 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.539+0000 7fb29543f640 1 --2- 172.21.15.16:0/3651754265 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb29015d680 0x7fb2900c8300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:15.551 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.539+0000 7fb29543f640 1 -- 172.21.15.16:0/3651754265 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb29015dad0 msgr2=0x7fb2900c1de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:15.551 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.539+0000 7fb29543f640 1 --2- 172.21.15.16:0/3651754265 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb29015dad0 0x7fb2900c1de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:15.551 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.539+0000 7fb29543f640 1 -- 172.21.15.16:0/3651754265 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2840094d0 con 0x7fb290159970 2023-05-22T15:27:15.551 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.539+0000 7fb29543f640 1 --2- 172.21.15.16:0/3651754265 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb290159970 0x7fb290154570 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fb2840af040 tx=0x7fb2840191e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:15.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.539+0000 7fb281ffb640 1 -- 172.21.15.16:0/3651754265 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2840b5640 con 0x7fb290159970 2023-05-22T15:27:15.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.539+0000 7fb281ffb640 1 -- 172.21.15.16:0/3651754265 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2840b57a0 con 0x7fb290159970 2023-05-22T15:27:15.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.539+0000 7fb2976a3640 1 -- 172.21.15.16:0/3651754265 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2900cae20 con 0x7fb290159970 2023-05-22T15:27:15.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.539+0000 7fb281ffb640 1 -- 172.21.15.16:0/3651754265 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2840b5b30 con 0x7fb290159970 2023-05-22T15:27:15.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.539+0000 7fb2976a3640 1 -- 172.21.15.16:0/3651754265 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2900cac90 con 0x7fb290159970 2023-05-22T15:27:15.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.539+0000 7fb2976a3640 1 -- 172.21.15.16:0/3651754265 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb29015adc0 con 0x7fb290159970 2023-05-22T15:27:15.554 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.539+0000 7fb281ffb640 1 -- 172.21.15.16:0/3651754265 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7fb2840b3070 con 0x7fb290159970 2023-05-22T15:27:15.555 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.539+0000 7fb281ffb640 1 --2- 172.21.15.16:0/3651754265 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fb2740653b0 0x7fb274067860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:15.555 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.539+0000 7fb281ffb640 1 -- 172.21.15.16:0/3651754265 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6813+0+0 (secure 0 0 0) 0x7fb28411ce30 con 0x7fb290159970 2023-05-22T15:27:15.555 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.539+0000 7fb294c3e640 1 --2- 172.21.15.16:0/3651754265 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fb2740653b0 0x7fb274067860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:15.555 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.539+0000 7fb294c3e640 1 --2- 172.21.15.16:0/3651754265 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fb2740653b0 0x7fb274067860 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fb29006b760 tx=0x7fb28c008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:15.555 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.543+0000 7fb281ffb640 1 -- 172.21.15.16:0/3651754265 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fb2840ea7c0 con 0x7fb290159970 2023-05-22T15:27:15.708 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.699+0000 7fb2976a3640 1 -- 172.21.15.16:0/3651754265 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fb29004dbe0 con 0x7fb290159970 2023-05-22T15:27:15.709 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.699+0000 7fb281ffb640 1 -- 172.21.15.16:0/3651754265 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) v1 ==== 100+0+11 (secure 0 0 0) 0x7fb2840b9020 con 0x7fb290159970 2023-05-22T15:27:15.709 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 64 2023-05-22T15:27:15.711 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.703+0000 7fb2976a3640 1 -- 172.21.15.16:0/3651754265 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fb2740653b0 msgr2=0x7fb274067860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:15.712 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.703+0000 7fb2976a3640 1 --2- 172.21.15.16:0/3651754265 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fb2740653b0 0x7fb274067860 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fb29006b760 tx=0x7fb28c008040 comp rx=0 tx=0).stop 2023-05-22T15:27:15.712 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.703+0000 7fb2976a3640 1 -- 172.21.15.16:0/3651754265 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb290159970 msgr2=0x7fb290154570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:15.712 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.703+0000 7fb2976a3640 1 --2- 172.21.15.16:0/3651754265 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb290159970 0x7fb290154570 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fb2840af040 tx=0x7fb2840191e0 comp rx=0 tx=0).stop 2023-05-22T15:27:15.712 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.703+0000 7fb2976a3640 1 -- 172.21.15.16:0/3651754265 shutdown_connections 2023-05-22T15:27:15.712 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.703+0000 7fb2976a3640 1 --2- 172.21.15.16:0/3651754265 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb29015dad0 0x7fb2900c1de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:15.713 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.703+0000 7fb2976a3640 1 --2- 172.21.15.16:0/3651754265 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fb2740653b0 0x7fb274067860 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:15.713 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.703+0000 7fb2976a3640 1 --2- 172.21.15.16:0/3651754265 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb29015d680 0x7fb2900c8300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:15.713 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.703+0000 7fb2976a3640 1 --2- 172.21.15.16:0/3651754265 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb290159970 0x7fb290154570 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:15.713 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.703+0000 7fb2976a3640 1 -- 172.21.15.16:0/3651754265 >> 172.21.15.16:0/3651754265 conn(0x7fb290069720 msgr2=0x7fb2900c2af0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:15.713 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.703+0000 7fb2976a3640 1 -- 172.21.15.16:0/3651754265 shutdown_connections 2023-05-22T15:27:15.713 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.703+0000 7fb2976a3640 1 -- 172.21.15.16:0/3651754265 wait complete. 2023-05-22T15:27:15.728 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-05-22T15:27:15.857 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.847+0000 7ffbe276b640 1 Processor -- start 2023-05-22T15:27:15.857 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.847+0000 7ffbe276b640 1 -- start start 2023-05-22T15:27:15.858 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.851+0000 7ffbe276b640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7ffbdc070c70 0x7ffbdc071530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:15.858 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.851+0000 7ffbe276b640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ffbdc15c580 0x7ffbdc06fba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:15.858 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.851+0000 7ffbe276b640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbdc15d520 con 0x7ffbdc06b6c0 2023-05-22T15:27:15.859 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.851+0000 7ffbe276b640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffbdc0c8b10 con 0x7ffbdc070c70 2023-05-22T15:27:15.859 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.851+0000 7ffbe276b640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffbdc15c9d0 con 0x7ffbdc15c580 2023-05-22T15:27:15.859 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.851+0000 7ffbe0d08640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7ffbdc06b6c0 0x7ffbdc0cca30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:39830/0 (socket says 172.21.15.16:39830) 2023-05-22T15:27:15.859 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.851+0000 7ffbe0d08640 1 -- 172.21.15.16:0/3226489174 learned_addr learned my addr 172.21.15.16:0/3226489174 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:27:15.860 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.851+0000 7ffbdbfff640 1 --2- 172.21.15.16:0/3226489174 >> v2:172.21.15.62:3300/0 conn(0x7ffbdc070c70 0x7ffbdc071530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:15.861 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.851+0000 7ffbdb7fe640 1 --2- 172.21.15.16:0/3226489174 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ffbdc15c580 0x7ffbdc06fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:15.861 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.851+0000 7ffbdaffd640 1 -- 172.21.15.16:0/3226489174 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1262472244 0 0) 0x7ffbdc15d520 con 0x7ffbdc06b6c0 2023-05-22T15:27:15.861 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.851+0000 7ffbdaffd640 1 -- 172.21.15.16:0/3226489174 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffbc4003530 con 0x7ffbdc06b6c0 2023-05-22T15:27:15.861 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.851+0000 7ffbdaffd640 1 -- 172.21.15.16:0/3226489174 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 958745411 0 0) 0x7ffbc4003530 con 0x7ffbdc06b6c0 2023-05-22T15:27:15.861 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.851+0000 7ffbdaffd640 1 -- 172.21.15.16:0/3226489174 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ffbdc15c580 msgr2=0x7ffbdc06fba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:15.862 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.851+0000 7ffbdaffd640 1 --2- 172.21.15.16:0/3226489174 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ffbdc15c580 0x7ffbdc06fba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:15.862 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.851+0000 7ffbdaffd640 1 -- 172.21.15.16:0/3226489174 >> v2:172.21.15.62:3300/0 conn(0x7ffbdc070c70 msgr2=0x7ffbdc071530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:15.862 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.851+0000 7ffbdaffd640 1 --2- 172.21.15.16:0/3226489174 >> v2:172.21.15.62:3300/0 conn(0x7ffbdc070c70 0x7ffbdc071530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:15.862 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.851+0000 7ffbdaffd640 1 -- 172.21.15.16:0/3226489174 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbdc0700e0 con 0x7ffbdc06b6c0 2023-05-22T15:27:15.862 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.851+0000 7ffbdaffd640 1 -- 172.21.15.16:0/3226489174 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7ffbd4001680 con 0x7ffbdc06b6c0 2023-05-22T15:27:15.862 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.851+0000 7ffbdaffd640 1 -- 172.21.15.16:0/3226489174 >> v1:172.21.15.16:6789/0 conn(0x7ffbdc06b6c0 legacy=0x7ffbdc0cca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:15.862 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.851+0000 7ffbdaffd640 1 --2- 172.21.15.16:0/3226489174 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffbc4003980 0x7ffbc4003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:15.863 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.851+0000 7ffbdaffd640 1 --2- 172.21.15.16:0/3226489174 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffbc40043b0 0x7ffbc4004870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:15.863 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.851+0000 7ffbdaffd640 1 --2- 172.21.15.16:0/3226489174 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ffbdc15c580 0x7ffbc4004fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:15.863 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.851+0000 7ffbdaffd640 1 -- 172.21.15.16:0/3226489174 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbdc0c8b10 con 0x7ffbc4003980 2023-05-22T15:27:15.863 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.851+0000 7ffbdaffd640 1 -- 172.21.15.16:0/3226489174 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbdc15c9d0 con 0x7ffbc40043b0 2023-05-22T15:27:15.864 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.851+0000 7ffbdaffd640 1 -- 172.21.15.16:0/3226489174 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffbc400b990 con 0x7ffbdc15c580 2023-05-22T15:27:15.864 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.851+0000 7ffbdbfff640 1 --2- 172.21.15.16:0/3226489174 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffbc4003980 0x7ffbc4003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:15.864 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.851+0000 7ffbdb7fe640 1 --2- 172.21.15.16:0/3226489174 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffbc40043b0 0x7ffbc4004870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:15.864 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.851+0000 7ffbe0d08640 1 --2- 172.21.15.16:0/3226489174 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ffbdc15c580 0x7ffbc4004fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:15.864 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.851+0000 7ffbdbfff640 1 -- 172.21.15.16:0/3226489174 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ffbdc15c580 msgr2=0x7ffbc4004fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:15.864 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.851+0000 7ffbdbfff640 1 --2- 172.21.15.16:0/3226489174 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ffbdc15c580 0x7ffbc4004fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:15.864 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.851+0000 7ffbdbfff640 1 -- 172.21.15.16:0/3226489174 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffbc40043b0 msgr2=0x7ffbc4004870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:15.865 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.851+0000 7ffbdbfff640 1 --2- 172.21.15.16:0/3226489174 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffbc40043b0 0x7ffbc4004870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:15.865 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.851+0000 7ffbdbfff640 1 -- 172.21.15.16:0/3226489174 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbc4003530 con 0x7ffbc4003980 2023-05-22T15:27:15.865 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.851+0000 7ffbdb7fe640 1 --2- 172.21.15.16:0/3226489174 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffbc40043b0 0x7ffbc4004870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:27:15.867 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.855+0000 7ffbdbfff640 1 --2- 172.21.15.16:0/3226489174 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffbc4003980 0x7ffbc4003d90 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7ffbcc00e170 tx=0x7ffbcc01cc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b06d02acc7f9166 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:15.868 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.855+0000 7ffbdaffd640 1 -- 172.21.15.16:0/3226489174 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbcc0a5a50 con 0x7ffbc4003980 2023-05-22T15:27:15.868 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.855+0000 7ffbdaffd640 1 -- 172.21.15.16:0/3226489174 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffbcc00f040 con 0x7ffbc4003980 2023-05-22T15:27:15.868 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.855+0000 7ffbdaffd640 1 -- 172.21.15.16:0/3226489174 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbcc0a5df0 con 0x7ffbc4003980 2023-05-22T15:27:15.868 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.855+0000 7ffbe276b640 1 -- 172.21.15.16:0/3226489174 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffbc4003980 msgr2=0x7ffbc4003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:15.868 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.855+0000 7ffbe276b640 1 --2- 172.21.15.16:0/3226489174 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffbc4003980 0x7ffbc4003d90 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7ffbcc00e170 tx=0x7ffbcc01cc50 comp rx=0 tx=0).stop 2023-05-22T15:27:15.868 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.855+0000 7ffbe0d08640 1 -- 172.21.15.16:0/3226489174 reap_dead start 2023-05-22T15:27:15.869 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.855+0000 7ffbe276b640 1 -- 172.21.15.16:0/3226489174 shutdown_connections 2023-05-22T15:27:15.869 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.855+0000 7ffbe276b640 1 -- 172.21.15.16:0/3226489174 >> 172.21.15.16:0/3226489174 conn(0x7ffbdc069720 msgr2=0x7ffbdc069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:15.869 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.855+0000 7ffbe276b640 1 -- 172.21.15.16:0/3226489174 shutdown_connections 2023-05-22T15:27:15.869 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.855+0000 7ffbe276b640 1 -- 172.21.15.16:0/3226489174 wait complete. 2023-05-22T15:27:15.869 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.855+0000 7ffbe276b640 1 Processor -- start 2023-05-22T15:27:15.869 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.855+0000 7ffbe276b640 1 -- start start 2023-05-22T15:27:15.869 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.855+0000 7ffbe276b640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffbc4003980 0x7ffbdc14fa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:15.870 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.855+0000 7ffbe276b640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ffbdc15e6f0 0x7ffbdc153450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:15.870 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.855+0000 7ffbe276b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffbdc153990 0x7ffbdc153e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:15.870 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.855+0000 7ffbe276b640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbdc15bd00 con 0x7ffbdc153990 2023-05-22T15:27:15.870 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.855+0000 7ffbe276b640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbdc15b740 con 0x7ffbc4003980 2023-05-22T15:27:15.870 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.855+0000 7ffbe276b640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffbdc15ba20 con 0x7ffbdc15e6f0 2023-05-22T15:27:15.870 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.855+0000 7ffbe0d08640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffbdc153990 0x7ffbdc153e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:15.870 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.855+0000 7ffbe0d08640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffbdc153990 0x7ffbdc153e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38064/0 (socket says 172.21.15.16:38064) 2023-05-22T15:27:15.871 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.855+0000 7ffbe0d08640 1 -- 172.21.15.16:0/3665147477 learned_addr learned my addr 172.21.15.16:0/3665147477 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:15.871 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.855+0000 7ffbdbfff640 1 --2- 172.21.15.16:0/3665147477 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffbc4003980 0x7ffbdc14fa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:15.871 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.855+0000 7ffbdb7fe640 1 --2- 172.21.15.16:0/3665147477 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ffbdc15e6f0 0x7ffbdc153450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:15.871 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.855+0000 7ffbe0d08640 1 -- 172.21.15.16:0/3665147477 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ffbdc15e6f0 msgr2=0x7ffbdc153450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:15.872 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.855+0000 7ffbe0d08640 1 --2- 172.21.15.16:0/3665147477 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ffbdc15e6f0 0x7ffbdc153450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:15.872 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.855+0000 7ffbe0d08640 1 -- 172.21.15.16:0/3665147477 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffbc4003980 msgr2=0x7ffbdc14fa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:15.872 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.855+0000 7ffbe0d08640 1 --2- 172.21.15.16:0/3665147477 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffbc4003980 0x7ffbdc14fa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:15.872 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.855+0000 7ffbe0d08640 1 -- 172.21.15.16:0/3665147477 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbdc0cc510 con 0x7ffbdc153990 2023-05-22T15:27:15.872 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.855+0000 7ffbe0d08640 1 --2- 172.21.15.16:0/3665147477 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffbdc153990 0x7ffbdc153e00 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7ffbdc071a70 tx=0x7ffbd400d530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:15.872 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.855+0000 7ffbd8ff9640 1 -- 172.21.15.16:0/3665147477 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbd400dd70 con 0x7ffbdc153990 2023-05-22T15:27:15.872 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.855+0000 7ffbd8ff9640 1 -- 172.21.15.16:0/3665147477 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffbd400ded0 con 0x7ffbdc153990 2023-05-22T15:27:15.873 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.859+0000 7ffbe276b640 1 -- 172.21.15.16:0/3665147477 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffbdc15d240 con 0x7ffbdc153990 2023-05-22T15:27:15.873 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.859+0000 7ffbd8ff9640 1 -- 172.21.15.16:0/3665147477 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbd4007310 con 0x7ffbdc153990 2023-05-22T15:27:15.873 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.859+0000 7ffbe276b640 1 -- 172.21.15.16:0/3665147477 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffbdc071040 con 0x7ffbdc153990 2023-05-22T15:27:15.873 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.859+0000 7ffbe276b640 1 -- 172.21.15.16:0/3665147477 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffbdc04dbe0 con 0x7ffbdc153990 2023-05-22T15:27:15.873 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.859+0000 7ffbd8ff9640 1 -- 172.21.15.16:0/3665147477 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7ffbd4007470 con 0x7ffbdc153990 2023-05-22T15:27:15.873 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.859+0000 7ffbd8ff9640 1 --2- 172.21.15.16:0/3665147477 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ffbc8065380 0x7ffbc8067830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:15.873 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.859+0000 7ffbdbfff640 1 --2- 172.21.15.16:0/3665147477 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ffbc8065380 0x7ffbc8067830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:15.874 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.859+0000 7ffbd8ff9640 1 -- 172.21.15.16:0/3665147477 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6813+0+0 (secure 0 0 0) 0x7ffbd4081f20 con 0x7ffbdc153990 2023-05-22T15:27:15.874 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.859+0000 7ffbdbfff640 1 --2- 172.21.15.16:0/3665147477 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ffbc8065380 0x7ffbc8067830 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7ffbcc0af000 tx=0x7ffbcc0a7040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:15.874 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:15.863+0000 7ffbd8ff9640 1 -- 172.21.15.16:0/3665147477 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7ffbd404f8a0 con 0x7ffbdc153990 2023-05-22T15:27:16.026 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.015+0000 7ffbe276b640 1 -- 172.21.15.16:0/3665147477 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7ffbdc0b3f90 con 0x7ffbdc153990 2023-05-22T15:27:16.027 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.019+0000 7ffbd8ff9640 1 -- 172.21.15.16:0/3665147477 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) v1 ==== 103+0+11 (secure 0 0 0) 0x7ffbd4052ec0 con 0x7ffbdc153990 2023-05-22T15:27:16.027 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2023-05-22T15:27:16.029 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.019+0000 7ffbe276b640 1 -- 172.21.15.16:0/3665147477 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ffbc8065380 msgr2=0x7ffbc8067830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:16.029 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.019+0000 7ffbe276b640 1 --2- 172.21.15.16:0/3665147477 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ffbc8065380 0x7ffbc8067830 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7ffbcc0af000 tx=0x7ffbcc0a7040 comp rx=0 tx=0).stop 2023-05-22T15:27:16.029 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.019+0000 7ffbe276b640 1 -- 172.21.15.16:0/3665147477 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffbdc153990 msgr2=0x7ffbdc153e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:16.029 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.019+0000 7ffbe276b640 1 --2- 172.21.15.16:0/3665147477 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffbdc153990 0x7ffbdc153e00 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7ffbdc071a70 tx=0x7ffbd400d530 comp rx=0 tx=0).stop 2023-05-22T15:27:16.030 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.019+0000 7ffbe276b640 1 -- 172.21.15.16:0/3665147477 shutdown_connections 2023-05-22T15:27:16.030 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.019+0000 7ffbe276b640 1 --2- 172.21.15.16:0/3665147477 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffbdc153990 0x7ffbdc153e00 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:16.030 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.019+0000 7ffbe276b640 1 --2- 172.21.15.16:0/3665147477 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ffbc8065380 0x7ffbc8067830 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:16.031 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.019+0000 7ffbe276b640 1 --2- 172.21.15.16:0/3665147477 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ffbdc15e6f0 0x7ffbdc153450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:16.031 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.019+0000 7ffbe276b640 1 --2- 172.21.15.16:0/3665147477 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffbc4003980 0x7ffbdc14fa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:16.031 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.019+0000 7ffbe276b640 1 -- 172.21.15.16:0/3665147477 >> 172.21.15.16:0/3665147477 conn(0x7ffbdc069720 msgr2=0x7ffbdc0adab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:16.031 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.019+0000 7ffbe276b640 1 -- 172.21.15.16:0/3665147477 shutdown_connections 2023-05-22T15:27:16.032 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.019+0000 7ffbe276b640 1 -- 172.21.15.16:0/3665147477 wait complete. 2023-05-22T15:27:16.046 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-05-22T15:27:16.173 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.163+0000 7fdabb9fa640 1 Processor -- start 2023-05-22T15:27:16.174 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.163+0000 7fdabb9fa640 1 -- start start 2023-05-22T15:27:16.174 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.167+0000 7fdabb9fa640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fdab4159240 0x7fdab40b2d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:16.175 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.167+0000 7fdabb9fa640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdab406a7d0 con 0x7fdab40b28b0 2023-05-22T15:27:16.175 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.167+0000 7fdabb9fa640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdab40b3260 con 0x7fdab4158760 2023-05-22T15:27:16.175 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.167+0000 7fdabb9fa640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdab4159df0 con 0x7fdab4159240 2023-05-22T15:27:16.175 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.167+0000 7fdab8f95640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fdab40b28b0 0x7fdab4158b30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:39832/0 (socket says 172.21.15.16:39832) 2023-05-22T15:27:16.175 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.167+0000 7fdab8f95640 1 -- 172.21.15.16:0/4173979172 learned_addr learned my addr 172.21.15.16:0/4173979172 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:27:16.175 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.167+0000 7fdab9f97640 1 --2- 172.21.15.16:0/4173979172 >> v2:172.21.15.16:3300/0 conn(0x7fdab4159240 0x7fdab40b2d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:16.175 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.167+0000 7fdab9f97640 1 -- 172.21.15.16:0/4173979172 >> v1:172.21.15.62:6789/0 conn(0x7fdab4158760 legacy=0x7fdab406a0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:16.176 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.167+0000 7fdab9f97640 1 -- 172.21.15.16:0/4173979172 >> v1:172.21.15.16:6789/0 conn(0x7fdab40b28b0 legacy=0x7fdab4158b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:16.176 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.167+0000 7fdab9f97640 1 -- 172.21.15.16:0/4173979172 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdab4159830 con 0x7fdab4159240 2023-05-22T15:27:16.176 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.167+0000 7fdab9f97640 1 --2- 172.21.15.16:0/4173979172 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdab4159240 0x7fdab40b2d20 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fdaac009d90 tx=0x7fdaac01eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db80691f28ecf340 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:16.177 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.167+0000 7fdaabfff640 1 -- 172.21.15.16:0/4173979172 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdaac0a7ae0 con 0x7fdab4159240 2023-05-22T15:27:16.178 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.167+0000 7fdaabfff640 1 -- 172.21.15.16:0/4173979172 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdaac0a7c40 con 0x7fdab4159240 2023-05-22T15:27:16.178 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.167+0000 7fdaabfff640 1 -- 172.21.15.16:0/4173979172 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdaac0af440 con 0x7fdab4159240 2023-05-22T15:27:16.178 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.167+0000 7fdabb9fa640 1 -- 172.21.15.16:0/4173979172 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdab4159240 msgr2=0x7fdab40b2d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:16.178 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.167+0000 7fdabb9fa640 1 --2- 172.21.15.16:0/4173979172 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdab4159240 0x7fdab40b2d20 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fdaac009d90 tx=0x7fdaac01eb20 comp rx=0 tx=0).stop 2023-05-22T15:27:16.178 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.171+0000 7fdabb9fa640 1 -- 172.21.15.16:0/4173979172 shutdown_connections 2023-05-22T15:27:16.178 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.171+0000 7fdabb9fa640 1 --2- 172.21.15.16:0/4173979172 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdab4159240 0x7fdab40b2d20 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:16.178 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.171+0000 7fdabb9fa640 1 -- 172.21.15.16:0/4173979172 >> 172.21.15.16:0/4173979172 conn(0x7fdab406e3c0 msgr2=0x7fdab40b1850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:16.179 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.171+0000 7fdabb9fa640 1 -- 172.21.15.16:0/4173979172 shutdown_connections 2023-05-22T15:27:16.179 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.171+0000 7fdabb9fa640 1 -- 172.21.15.16:0/4173979172 wait complete. 2023-05-22T15:27:16.180 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.171+0000 7fdabb9fa640 1 Processor -- start 2023-05-22T15:27:16.181 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.171+0000 7fdabb9fa640 1 -- start start 2023-05-22T15:27:16.181 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.171+0000 7fdabb9fa640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdab40b28b0 0x7fdab414d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:16.181 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.171+0000 7fdabb9fa640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdab4158760 0x7fdab414b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:16.181 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.171+0000 7fdabb9fa640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fdab4159240 0x7fdab406bef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:16.181 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.171+0000 7fdabb9fa640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdab414ed30 con 0x7fdab40b28b0 2023-05-22T15:27:16.181 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.171+0000 7fdabb9fa640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdab414f7e0 con 0x7fdab4158760 2023-05-22T15:27:16.181 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.171+0000 7fdabb9fa640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdab414e770 con 0x7fdab4159240 2023-05-22T15:27:16.182 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.171+0000 7fdab9796640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdab40b28b0 0x7fdab414d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:16.182 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.171+0000 7fdab9796640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdab40b28b0 0x7fdab414d2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38082/0 (socket says 172.21.15.16:38082) 2023-05-22T15:27:16.182 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.171+0000 7fdab9796640 1 -- 172.21.15.16:0/106462436 learned_addr learned my addr 172.21.15.16:0/106462436 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:16.182 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.171+0000 7fdab9f97640 1 --2- 172.21.15.16:0/106462436 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fdab4159240 0x7fdab406bef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:16.182 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.171+0000 7fdab9796640 1 -- 172.21.15.16:0/106462436 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fdab4159240 msgr2=0x7fdab406bef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:16.182 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.171+0000 7fdab9796640 1 --2- 172.21.15.16:0/106462436 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fdab4159240 0x7fdab406bef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:16.182 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.171+0000 7fdab8f95640 1 --2- 172.21.15.16:0/106462436 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdab4158760 0x7fdab414b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:16.182 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.171+0000 7fdab9796640 1 -- 172.21.15.16:0/106462436 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdab4158760 msgr2=0x7fdab414b2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:16.183 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.171+0000 7fdab9796640 1 --2- 172.21.15.16:0/106462436 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdab4158760 0x7fdab414b2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:16.183 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.171+0000 7fdab9796640 1 -- 172.21.15.16:0/106462436 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdaac009a70 con 0x7fdab40b28b0 2023-05-22T15:27:16.183 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.171+0000 7fdab9796640 1 --2- 172.21.15.16:0/106462436 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdab40b28b0 0x7fdab414d2c0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fdab000c550 tx=0x7fdab000ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:16.183 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.171+0000 7fdaa9ffb640 1 -- 172.21.15.16:0/106462436 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdab0012830 con 0x7fdab40b28b0 2023-05-22T15:27:16.183 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.171+0000 7fdaa9ffb640 1 -- 172.21.15.16:0/106462436 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdab0012990 con 0x7fdab40b28b0 2023-05-22T15:27:16.183 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.171+0000 7fdabb9fa640 1 -- 172.21.15.16:0/106462436 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdab40b39b0 con 0x7fdab40b28b0 2023-05-22T15:27:16.183 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.171+0000 7fdaa9ffb640 1 -- 172.21.15.16:0/106462436 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdab0012d40 con 0x7fdab40b28b0 2023-05-22T15:27:16.183 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.171+0000 7fdabb9fa640 1 -- 172.21.15.16:0/106462436 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdab4069ba0 con 0x7fdab40b28b0 2023-05-22T15:27:16.184 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.175+0000 7fdabb9fa640 1 -- 172.21.15.16:0/106462436 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdab40b3b40 con 0x7fdab40b28b0 2023-05-22T15:27:16.184 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.175+0000 7fdaa9ffb640 1 -- 172.21.15.16:0/106462436 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7fdab001b400 con 0x7fdab40b28b0 2023-05-22T15:27:16.185 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.175+0000 7fdaa9ffb640 1 --2- 172.21.15.16:0/106462436 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fda98065380 0x7fda98067830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:16.185 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.175+0000 7fdaa9ffb640 1 -- 172.21.15.16:0/106462436 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6813+0+0 (secure 0 0 0) 0x7fdab0082d70 con 0x7fdab40b28b0 2023-05-22T15:27:16.185 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.175+0000 7fdab8f95640 1 --2- 172.21.15.16:0/106462436 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fda98065380 0x7fda98067830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:16.185 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.175+0000 7fdab8f95640 1 --2- 172.21.15.16:0/106462436 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fda98065380 0x7fda98067830 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fdab40cdd60 tx=0x7fdaa4009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:16.186 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.179+0000 7fdaa9ffb640 1 -- 172.21.15.16:0/106462436 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fdab00507e0 con 0x7fdab40b28b0 2023-05-22T15:27:16.372 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.363+0000 7fdabb9fa640 1 -- 172.21.15.16:0/106462436 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdab414f920 con 0x7fdab40b28b0 2023-05-22T15:27:16.373 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.363+0000 7fdaa9ffb640 1 -- 172.21.15.16:0/106462436 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4641 (secure 0 0 0) 0x7fdab0016090 con 0x7fdab40b28b0 2023-05-22T15:27:16.374 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:27:16.374 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4261,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6834/3661808159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":3661808159},{"type":"v1","addr":"172.21.15.62:6834","nonce":3661808159}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4264,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/2227467222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6833","nonce":2227467222},{"type":"v1","addr":"172.21.15.62:6835","nonce":2227467222}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4284,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6836/2808106087","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2808106087},{"type":"v1","addr":"172.21.15.16:6836","nonce":2808106087}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-05-22T15:26:49.546008+0000","modified":"2023-05-22T15:27:06.047204+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4287},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4287":{"gid":4287,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.16:6837/283322289","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6835","nonce":283322289},{"type":"v1","addr":"172.21.15.16:6837","nonce":283322289}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-05-22T15:27:16.374 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 8 2023-05-22T15:27:16.375 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.367+0000 7fdabb9fa640 1 -- 172.21.15.16:0/106462436 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fda98065380 msgr2=0x7fda98067830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:16.376 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.367+0000 7fdabb9fa640 1 --2- 172.21.15.16:0/106462436 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fda98065380 0x7fda98067830 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fdab40cdd60 tx=0x7fdaa4009040 comp rx=0 tx=0).stop 2023-05-22T15:27:16.376 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.367+0000 7fdabb9fa640 1 -- 172.21.15.16:0/106462436 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdab40b28b0 msgr2=0x7fdab414d2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:16.376 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.367+0000 7fdabb9fa640 1 --2- 172.21.15.16:0/106462436 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdab40b28b0 0x7fdab414d2c0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fdab000c550 tx=0x7fdab000ca10 comp rx=0 tx=0).stop 2023-05-22T15:27:16.376 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.367+0000 7fdabb9fa640 1 -- 172.21.15.16:0/106462436 shutdown_connections 2023-05-22T15:27:16.376 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.367+0000 7fdabb9fa640 1 --2- 172.21.15.16:0/106462436 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fda98065380 0x7fda98067830 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:16.376 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.367+0000 7fdabb9fa640 1 --2- 172.21.15.16:0/106462436 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fdab4159240 0x7fdab406bef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:16.377 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.367+0000 7fdabb9fa640 1 --2- 172.21.15.16:0/106462436 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdab4158760 0x7fdab414b2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:16.377 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.367+0000 7fdabb9fa640 1 --2- 172.21.15.16:0/106462436 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdab40b28b0 0x7fdab414d2c0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:16.377 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.367+0000 7fdabb9fa640 1 -- 172.21.15.16:0/106462436 >> 172.21.15.16:0/106462436 conn(0x7fdab406e3c0 msgr2=0x7fdab4156840 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:16.377 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.367+0000 7fdabb9fa640 1 -- 172.21.15.16:0/106462436 shutdown_connections 2023-05-22T15:27:16.377 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.367+0000 7fdabb9fa640 1 -- 172.21.15.16:0/106462436 wait complete. 2023-05-22T15:27:16.393 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-05-22T15:27:16.510 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.503+0000 7ff578b2b640 1 Processor -- start 2023-05-22T15:27:16.510 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.503+0000 7ff578b2b640 1 -- start start 2023-05-22T15:27:16.511 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.503+0000 7ff578b2b640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7ff5740d0e40 0x7ff5741648d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:16.511 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.503+0000 7ff578b2b640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5740cf9a0 con 0x7ff5740b2880 2023-05-22T15:27:16.511 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.503+0000 7ff578b2b640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff57415aba0 con 0x7ff574159cd0 2023-05-22T15:27:16.511 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.503+0000 7ff578b2b640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff57415ae80 con 0x7ff5740d0e40 2023-05-22T15:27:16.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.503+0000 7ff571d9b640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7ff5740b2880 0x7ff5740cbfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:39842/0 (socket says 172.21.15.16:39842) 2023-05-22T15:27:16.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.503+0000 7ff571d9b640 1 -- 172.21.15.16:0/1216623847 learned_addr learned my addr 172.21.15.16:0/1216623847 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:27:16.513 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.503+0000 7ff572d9d640 1 --2- 172.21.15.16:0/1216623847 >> v2:172.21.15.62:3300/0 conn(0x7ff5740d0e40 0x7ff5741648d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:16.513 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.503+0000 7ff57159a640 1 -- 172.21.15.16:0/1216623847 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2809134207 0 0) 0x7ff5740cf9a0 con 0x7ff5740b2880 2023-05-22T15:27:16.514 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.503+0000 7ff57159a640 1 -- 172.21.15.16:0/1216623847 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff55c003530 con 0x7ff5740b2880 2023-05-22T15:27:16.514 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.503+0000 7ff57159a640 1 -- 172.21.15.16:0/1216623847 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3886532334 0 0) 0x7ff55c003530 con 0x7ff5740b2880 2023-05-22T15:27:16.514 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.503+0000 7ff57159a640 1 -- 172.21.15.16:0/1216623847 >> v2:172.21.15.62:3300/0 conn(0x7ff5740d0e40 msgr2=0x7ff5741648d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:16.514 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.503+0000 7ff57159a640 1 --2- 172.21.15.16:0/1216623847 >> v2:172.21.15.62:3300/0 conn(0x7ff5740d0e40 0x7ff5741648d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:16.514 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.503+0000 7ff57159a640 1 -- 172.21.15.16:0/1216623847 >> v1:172.21.15.62:6789/0 conn(0x7ff574159cd0 legacy=0x7ff57415a0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:16.514 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.503+0000 7ff57159a640 1 -- 172.21.15.16:0/1216623847 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff57415b160 con 0x7ff5740b2880 2023-05-22T15:27:16.514 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.503+0000 7ff57159a640 1 -- 172.21.15.16:0/1216623847 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7ff568001680 con 0x7ff5740b2880 2023-05-22T15:27:16.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.503+0000 7ff57159a640 1 -- 172.21.15.16:0/1216623847 >> v1:172.21.15.16:6789/0 conn(0x7ff5740b2880 legacy=0x7ff5740cbfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:16.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.503+0000 7ff57159a640 1 --2- 172.21.15.16:0/1216623847 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff55c003980 0x7ff55c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:16.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.503+0000 7ff57159a640 1 --2- 172.21.15.16:0/1216623847 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff55c0047d0 0x7ff55c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:16.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.503+0000 7ff57159a640 1 --2- 172.21.15.16:0/1216623847 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff55c005180 0x7ff55c00b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:16.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.503+0000 7ff57159a640 1 -- 172.21.15.16:0/1216623847 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff57415ae80 con 0x7ff55c003980 2023-05-22T15:27:16.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.503+0000 7ff57159a640 1 -- 172.21.15.16:0/1216623847 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff55c00c000 con 0x7ff55c005180 2023-05-22T15:27:16.516 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.503+0000 7ff57159a640 1 -- 172.21.15.16:0/1216623847 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff55c00c1c0 con 0x7ff55c0047d0 2023-05-22T15:27:16.516 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.503+0000 7ff57259c640 1 --2- 172.21.15.16:0/1216623847 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff55c003980 0x7ff55c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:16.516 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.503+0000 7ff572d9d640 1 --2- 172.21.15.16:0/1216623847 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff55c005180 0x7ff55c00b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:16.516 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.503+0000 7ff571d9b640 1 --2- 172.21.15.16:0/1216623847 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff55c0047d0 0x7ff55c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:16.516 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.503+0000 7ff57259c640 1 -- 172.21.15.16:0/1216623847 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff55c0047d0 msgr2=0x7ff55c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:16.516 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.503+0000 7ff57259c640 1 --2- 172.21.15.16:0/1216623847 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff55c0047d0 0x7ff55c004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:16.516 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.503+0000 7ff57259c640 1 -- 172.21.15.16:0/1216623847 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff55c005180 msgr2=0x7ff55c00b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:16.517 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.503+0000 7ff57259c640 1 --2- 172.21.15.16:0/1216623847 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff55c005180 0x7ff55c00b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:16.517 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.503+0000 7ff57259c640 1 -- 172.21.15.16:0/1216623847 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff55c003530 con 0x7ff55c003980 2023-05-22T15:27:16.517 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.503+0000 7ff572d9d640 1 --2- 172.21.15.16:0/1216623847 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff55c005180 0x7ff55c00b990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:27:16.517 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.503+0000 7ff571d9b640 1 --2- 172.21.15.16:0/1216623847 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff55c0047d0 0x7ff55c004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:27:16.517 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.503+0000 7ff572d9d640 1 -- 172.21.15.16:0/1216623847 reap_dead start 2023-05-22T15:27:16.519 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.507+0000 7ff57259c640 1 --2- 172.21.15.16:0/1216623847 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff55c003980 0x7ff55c003d90 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7ff564001780 tx=0x7ff56401c3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31527929f5d83a08 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:16.519 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.507+0000 7ff57159a640 1 -- 172.21.15.16:0/1216623847 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5640052d0 con 0x7ff55c003980 2023-05-22T15:27:16.519 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.507+0000 7ff57159a640 1 -- 172.21.15.16:0/1216623847 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5640a7050 con 0x7ff55c003980 2023-05-22T15:27:16.519 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.507+0000 7ff57159a640 1 -- 172.21.15.16:0/1216623847 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff564003600 con 0x7ff55c003980 2023-05-22T15:27:16.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.507+0000 7ff578b2b640 1 -- 172.21.15.16:0/1216623847 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff55c003980 msgr2=0x7ff55c003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:16.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.507+0000 7ff578b2b640 1 --2- 172.21.15.16:0/1216623847 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff55c003980 0x7ff55c003d90 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7ff564001780 tx=0x7ff56401c3b0 comp rx=0 tx=0).stop 2023-05-22T15:27:16.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.507+0000 7ff578b2b640 1 -- 172.21.15.16:0/1216623847 shutdown_connections 2023-05-22T15:27:16.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.507+0000 7ff578b2b640 1 --2- 172.21.15.16:0/1216623847 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff55c003980 0x7ff55c003d90 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:16.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.507+0000 7ff578b2b640 1 -- 172.21.15.16:0/1216623847 >> 172.21.15.16:0/1216623847 conn(0x7ff57406f5c0 msgr2=0x7ff57406f9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:16.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.507+0000 7ff578b2b640 1 -- 172.21.15.16:0/1216623847 shutdown_connections 2023-05-22T15:27:16.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.507+0000 7ff578b2b640 1 -- 172.21.15.16:0/1216623847 wait complete. 2023-05-22T15:27:16.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.507+0000 7ff578b2b640 1 Processor -- start 2023-05-22T15:27:16.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.507+0000 7ff578b2b640 1 -- start start 2023-05-22T15:27:16.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.511+0000 7ff578b2b640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff55c003980 0x7ff5740c77c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:16.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.511+0000 7ff578b2b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff57406c310 0x7ff5740c9900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:16.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.511+0000 7ff578b2b640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff5740aea10 0x7ff574152a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:16.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.511+0000 7ff578b2b640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff574154370 con 0x7ff57406c310 2023-05-22T15:27:16.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.511+0000 7ff578b2b640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff574154650 con 0x7ff55c003980 2023-05-22T15:27:16.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.511+0000 7ff578b2b640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff574154930 con 0x7ff5740aea10 2023-05-22T15:27:16.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.511+0000 7ff571d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff57406c310 0x7ff5740c9900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:16.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.511+0000 7ff571d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff57406c310 0x7ff5740c9900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38098/0 (socket says 172.21.15.16:38098) 2023-05-22T15:27:16.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.511+0000 7ff571d9b640 1 -- 172.21.15.16:0/744194549 learned_addr learned my addr 172.21.15.16:0/744194549 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:16.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.511+0000 7ff57259c640 1 --2- 172.21.15.16:0/744194549 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff55c003980 0x7ff5740c77c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:16.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.511+0000 7ff572d9d640 1 --2- 172.21.15.16:0/744194549 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff5740aea10 0x7ff574152a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:16.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.511+0000 7ff571d9b640 1 -- 172.21.15.16:0/744194549 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff5740aea10 msgr2=0x7ff574152a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:16.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.511+0000 7ff571d9b640 1 --2- 172.21.15.16:0/744194549 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff5740aea10 0x7ff574152a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:16.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.511+0000 7ff571d9b640 1 -- 172.21.15.16:0/744194549 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff55c003980 msgr2=0x7ff5740c77c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:16.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.511+0000 7ff571d9b640 1 --2- 172.21.15.16:0/744194549 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff55c003980 0x7ff5740c77c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:16.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.511+0000 7ff571d9b640 1 -- 172.21.15.16:0/744194549 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5740cc8b0 con 0x7ff57406c310 2023-05-22T15:27:16.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.511+0000 7ff571d9b640 1 --2- 172.21.15.16:0/744194549 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff57406c310 0x7ff5740c9900 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7ff568007fd0 tx=0x7ff56800d7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:16.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.511+0000 7ff562ffd640 1 -- 172.21.15.16:0/744194549 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5680074e0 con 0x7ff57406c310 2023-05-22T15:27:16.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.511+0000 7ff562ffd640 1 -- 172.21.15.16:0/744194549 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff568007640 con 0x7ff57406c310 2023-05-22T15:27:16.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.511+0000 7ff578b2b640 1 -- 172.21.15.16:0/744194549 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5740cc720 con 0x7ff57406c310 2023-05-22T15:27:16.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.511+0000 7ff562ffd640 1 -- 172.21.15.16:0/744194549 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff568004a10 con 0x7ff57406c310 2023-05-22T15:27:16.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.511+0000 7ff578b2b640 1 -- 172.21.15.16:0/744194549 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5740cca40 con 0x7ff57406c310 2023-05-22T15:27:16.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.511+0000 7ff578b2b640 1 -- 172.21.15.16:0/744194549 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff57404dbe0 con 0x7ff57406c310 2023-05-22T15:27:16.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.511+0000 7ff562ffd640 1 -- 172.21.15.16:0/744194549 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7ff568004b70 con 0x7ff57406c310 2023-05-22T15:27:16.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.511+0000 7ff562ffd640 1 --2- 172.21.15.16:0/744194549 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ff5540653b0 0x7ff554067860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:16.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.511+0000 7ff562ffd640 1 -- 172.21.15.16:0/744194549 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6813+0+0 (secure 0 0 0) 0x7ff568082240 con 0x7ff57406c310 2023-05-22T15:27:16.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.511+0000 7ff57259c640 1 --2- 172.21.15.16:0/744194549 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ff5540653b0 0x7ff554067860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:16.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.515+0000 7ff57259c640 1 --2- 172.21.15.16:0/744194549 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ff5540653b0 0x7ff554067860 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7ff564001750 tx=0x7ff5640011a0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:16.526 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.515+0000 7ff562ffd640 1 -- 172.21.15.16:0/744194549 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7ff56804b620 con 0x7ff57406c310 2023-05-22T15:27:16.675 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.667+0000 7ff578b2b640 1 -- 172.21.15.16:0/744194549 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff5740b1470 con 0x7ff57406c310 2023-05-22T15:27:16.679 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.671+0000 7ff562ffd640 1 -- 172.21.15.16:0/744194549 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) v1 ==== 74+0+19604 (secure 0 0 0) 0x7ff56804fc60 con 0x7ff57406c310 2023-05-22T15:27:16.679 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:27:16.679 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":27,"fsid":"b2719834-7847-4d98-90a6-f65d57f85c49","created":"2023-05-22T15:26:25.259314+0000","modified":"2023-05-22T15:26:53.971378+0000","last_up_change":"2023-05-22T15:26:36.448524+0000","last_in_change":"2023-05-22T15:26:27.675668+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-05-22T15:26:38.019781+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-05-22T15:26:43.570514+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-05-22T15:26:47.879159+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-05-22T15:26:48.522130+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data_ec","create_time":"2023-05-22T15:26:51.616142+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"9e7ff854-b25f-4cbb-a5c2-1799d9c53d2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":37021},{"type":"v1","addr":"172.21.15.16:6825","nonce":37021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":37021},{"type":"v1","addr":"172.21.15.16:6827","nonce":37021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":37021},{"type":"v1","addr":"172.21.15.16:6831","nonce":37021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":37021},{"type":"v1","addr":"172.21.15.16:6829","nonce":37021}]},"public_addr":"172.21.15.16:6825/37021","cluster_addr":"172.21.15.16:6827/37021","heartbeat_back_addr":"172.21.15.16:6831/37021","heartbeat_front_addr":"172.21.15.16:6829/37021","state":["exists","up"]},{"osd":1,"uuid":"4702fbc9-a927-4565-aca5-41dc69cc8264","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":37008},{"type":"v1","addr":"172.21.15.16:6801","nonce":37008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":37008},{"type":"v1","addr":"172.21.15.16:6803","nonce":37008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":37008},{"type":"v1","addr":"172.21.15.16:6807","nonce":37008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":37008},{"type":"v1","addr":"172.21.15.16:6805","nonce":37008}]},"public_addr":"172.21.15.16:6801/37008","cluster_addr":"172.21.15.16:6803/37008","heartbeat_back_addr":"172.21.15.16:6807/37008","heartbeat_front_addr":"172.21.15.16:6805/37008","state":["exists","up"]},{"osd":2,"uuid":"1e7bba46-4eba-47ef-bb81-5e2d371638ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":37009},{"type":"v1","addr":"172.21.15.16:6809","nonce":37009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":37009},{"type":"v1","addr":"172.21.15.16:6811","nonce":37009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":37009},{"type":"v1","addr":"172.21.15.16:6815","nonce":37009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":37009},{"type":"v1","addr":"172.21.15.16:6813","nonce":37009}]},"public_addr":"172.21.15.16:6809/37009","cluster_addr":"172.21.15.16:6811/37009","heartbeat_back_addr":"172.21.15.16:6815/37009","heartbeat_front_addr":"172.21.15.16:6813/37009","state":["exists","up"]},{"osd":3,"uuid":"383cacc0-5a88-4904-990e-498ca59ef48a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":37020},{"type":"v1","addr":"172.21.15.16:6817","nonce":37020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":37020},{"type":"v1","addr":"172.21.15.16:6819","nonce":37020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":37020},{"type":"v1","addr":"172.21.15.16:6823","nonce":37020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":37020},{"type":"v1","addr":"172.21.15.16:6821","nonce":37020}]},"public_addr":"172.21.15.16:6817/37020","cluster_addr":"172.21.15.16:6819/37020","heartbeat_back_addr":"172.21.15.16:6823/37020","heartbeat_front_addr":"172.21.15.16:6821/37020","state":["exists","up"]},{"osd":4,"uuid":"970500ef-95b7-42c1-b080-eed4fae7013d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":37353},{"type":"v1","addr":"172.21.15.62:6817","nonce":37353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6819","nonce":37353},{"type":"v1","addr":"172.21.15.62:6822","nonce":37353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37353},{"type":"v1","addr":"172.21.15.62:6831","nonce":37353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37353},{"type":"v1","addr":"172.21.15.62:6826","nonce":37353}]},"public_addr":"172.21.15.62:6817/37353","cluster_addr":"172.21.15.62:6822/37353","heartbeat_back_addr":"172.21.15.62:6831/37353","heartbeat_front_addr":"172.21.15.62:6826/37353","state":["exists","up"]},{"osd":5,"uuid":"022d67db-d4d8-445b-b1b9-8b446a00f5c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":37352},{"type":"v1","addr":"172.21.15.62:6801","nonce":37352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":37352},{"type":"v1","addr":"172.21.15.62:6803","nonce":37352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":37352},{"type":"v1","addr":"172.21.15.62:6807","nonce":37352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":37352},{"type":"v1","addr":"172.21.15.62:6805","nonce":37352}]},"public_addr":"172.21.15.62:6801/37352","cluster_addr":"172.21.15.62:6803/37352","heartbeat_back_addr":"172.21.15.62:6807/37352","heartbeat_front_addr":"172.21.15.62:6805/37352","state":["exists","up"]},{"osd":6,"uuid":"42bd372b-6688-4a1b-8407-e16e8c266e33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":37354},{"type":"v1","addr":"172.21.15.62:6820","nonce":37354}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":37354},{"type":"v1","addr":"172.21.15.62:6823","nonce":37354}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6829","nonce":37354},{"type":"v1","addr":"172.21.15.62:6830","nonce":37354}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6825","nonce":37354},{"type":"v1","addr":"172.21.15.62:6827","nonce":37354}]},"public_addr":"172.21.15.62:6820/37354","cluster_addr":"172.21.15.62:6823/37354","heartbeat_back_addr":"172.21.15.62:6830/37354","heartbeat_front_addr":"172.21.15.62:6827/37354","state":["exists","up"]},{"osd":7,"uuid":"ac70b2a3-f41b-4bf2-b227-7f2cbb55426d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37355},{"type":"v1","addr":"172.21.15.62:6809","nonce":37355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37355},{"type":"v1","addr":"172.21.15.62:6811","nonce":37355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":37355},{"type":"v1","addr":"172.21.15.62:6815","nonce":37355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":37355},{"type":"v1","addr":"172.21.15.62:6813","nonce":37355}]},"public_addr":"172.21.15.62:6809/37355","cluster_addr":"172.21.15.62:6811/37355","heartbeat_back_addr":"172.21.15.62:6815/37355","heartbeat_front_addr":"172.21.15.62:6813/37355","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.368896+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.107273+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.276888+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.287502+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.124791+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.251937+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.255679+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.232797+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-05-22T15:27:16.681 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.671+0000 7ff578b2b640 1 -- 172.21.15.16:0/744194549 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ff5540653b0 msgr2=0x7ff554067860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:16.681 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.671+0000 7ff578b2b640 1 --2- 172.21.15.16:0/744194549 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ff5540653b0 0x7ff554067860 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7ff564001750 tx=0x7ff5640011a0 comp rx=0 tx=0).stop 2023-05-22T15:27:16.681 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.671+0000 7ff578b2b640 1 -- 172.21.15.16:0/744194549 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff57406c310 msgr2=0x7ff5740c9900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:16.681 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.671+0000 7ff578b2b640 1 --2- 172.21.15.16:0/744194549 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff57406c310 0x7ff5740c9900 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7ff568007fd0 tx=0x7ff56800d7b0 comp rx=0 tx=0).stop 2023-05-22T15:27:16.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.671+0000 7ff578b2b640 1 -- 172.21.15.16:0/744194549 shutdown_connections 2023-05-22T15:27:16.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.671+0000 7ff578b2b640 1 --2- 172.21.15.16:0/744194549 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ff5540653b0 0x7ff554067860 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:16.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.671+0000 7ff578b2b640 1 --2- 172.21.15.16:0/744194549 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff5740aea10 0x7ff574152a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:16.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.671+0000 7ff578b2b640 1 --2- 172.21.15.16:0/744194549 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff57406c310 0x7ff5740c9900 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:16.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.671+0000 7ff578b2b640 1 --2- 172.21.15.16:0/744194549 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff55c003980 0x7ff5740c77c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:16.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.671+0000 7ff578b2b640 1 -- 172.21.15.16:0/744194549 >> 172.21.15.16:0/744194549 conn(0x7ff57406f5c0 msgr2=0x7ff5740b0b30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:16.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.671+0000 7ff578b2b640 1 -- 172.21.15.16:0/744194549 shutdown_connections 2023-05-22T15:27:16.683 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.671+0000 7ff578b2b640 1 -- 172.21.15.16:0/744194549 wait complete. 2023-05-22T15:27:16.697 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-05-22T15:27:16.832 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.811+0000 7fe04df91640 1 Processor -- start 2023-05-22T15:27:16.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.811+0000 7fe04df91640 1 -- start start 2023-05-22T15:27:16.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.815+0000 7fe04df91640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fe0481531b0 0x7fe0480afb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:16.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.815+0000 7fe04df91640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe04815a0a0 con 0x7fe0480b0080 2023-05-22T15:27:16.834 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.815+0000 7fe04df91640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe048159e50 con 0x7fe048159a40 2023-05-22T15:27:16.834 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.815+0000 7fe04df91640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0480b0520 con 0x7fe0481531b0 2023-05-22T15:27:16.834 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.815+0000 7fe03ffff640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fe0480b0080 0x7fe048159330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:39850/0 (socket says 172.21.15.16:39850) 2023-05-22T15:27:16.834 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.815+0000 7fe03ffff640 1 -- 172.21.15.16:0/2148740914 learned_addr learned my addr 172.21.15.16:0/2148740914 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:27:16.835 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.815+0000 7fe0477fe640 1 --2- 172.21.15.16:0/2148740914 >> v2:172.21.15.62:3300/0 conn(0x7fe0481531b0 0x7fe0480afb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:16.835 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.815+0000 7fe046ffd640 1 -- 172.21.15.16:0/2148740914 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1131094427 0 0) 0x7fe04815a0a0 con 0x7fe0480b0080 2023-05-22T15:27:16.835 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.815+0000 7fe046ffd640 1 -- 172.21.15.16:0/2148740914 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe034003530 con 0x7fe0480b0080 2023-05-22T15:27:16.836 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.815+0000 7fe046ffd640 1 -- 172.21.15.16:0/2148740914 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3480635157 0 0) 0x7fe034003530 con 0x7fe0480b0080 2023-05-22T15:27:16.836 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.815+0000 7fe046ffd640 1 -- 172.21.15.16:0/2148740914 >> v2:172.21.15.62:3300/0 conn(0x7fe0481531b0 msgr2=0x7fe0480afb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:16.837 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.815+0000 7fe046ffd640 1 --2- 172.21.15.16:0/2148740914 >> v2:172.21.15.62:3300/0 conn(0x7fe0481531b0 0x7fe0480afb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:16.837 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.815+0000 7fe046ffd640 1 -- 172.21.15.16:0/2148740914 >> v1:172.21.15.62:6789/0 conn(0x7fe048159a40 legacy=0x7fe0480c8ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:16.837 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.815+0000 7fe046ffd640 1 -- 172.21.15.16:0/2148740914 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe048153580 con 0x7fe0480b0080 2023-05-22T15:27:16.838 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.815+0000 7fe046ffd640 1 -- 172.21.15.16:0/2148740914 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7fe030001680 con 0x7fe0480b0080 2023-05-22T15:27:16.838 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.815+0000 7fe0477fe640 1 --2- 172.21.15.16:0/2148740914 >> v2:172.21.15.62:3300/0 conn(0x7fe0481531b0 0x7fe0480afb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-05-22T15:27:16.838 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.815+0000 7fe046ffd640 1 -- 172.21.15.16:0/2148740914 >> v1:172.21.15.16:6789/0 conn(0x7fe0480b0080 legacy=0x7fe048159330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:16.838 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.815+0000 7fe046ffd640 1 --2- 172.21.15.16:0/2148740914 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe034003980 0x7fe034003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:16.839 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.815+0000 7fe046ffd640 1 --2- 172.21.15.16:0/2148740914 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe0340047d0 0x7fe034004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:16.839 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.815+0000 7fe046ffd640 1 --2- 172.21.15.16:0/2148740914 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe034005180 0x7fe03400b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:16.839 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.815+0000 7fe046ffd640 1 -- 172.21.15.16:0/2148740914 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe03400bfd0 con 0x7fe0340047d0 2023-05-22T15:27:16.840 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.815+0000 7fe046ffd640 1 -- 172.21.15.16:0/2148740914 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe03400c140 con 0x7fe034005180 2023-05-22T15:27:16.840 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.815+0000 7fe046ffd640 1 -- 172.21.15.16:0/2148740914 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe03400c300 con 0x7fe034003980 2023-05-22T15:27:16.840 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.815+0000 7fe03ffff640 1 --2- 172.21.15.16:0/2148740914 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe0340047d0 0x7fe034004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:16.841 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.815+0000 7fe0477fe640 1 --2- 172.21.15.16:0/2148740914 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe034003980 0x7fe034003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:16.841 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.815+0000 7fe03ffff640 1 -- 172.21.15.16:0/2148740914 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe034003980 msgr2=0x7fe034003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:16.842 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.815+0000 7fe047fff640 1 --2- 172.21.15.16:0/2148740914 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe034005180 0x7fe03400b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:16.842 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.815+0000 7fe03ffff640 1 --2- 172.21.15.16:0/2148740914 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe034003980 0x7fe034003d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:16.842 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.815+0000 7fe03ffff640 1 -- 172.21.15.16:0/2148740914 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe034005180 msgr2=0x7fe03400b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:16.843 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.815+0000 7fe03ffff640 1 --2- 172.21.15.16:0/2148740914 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe034005180 0x7fe03400b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:16.843 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.815+0000 7fe03ffff640 1 -- 172.21.15.16:0/2148740914 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe034003530 con 0x7fe0340047d0 2023-05-22T15:27:16.843 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.815+0000 7fe047fff640 1 --2- 172.21.15.16:0/2148740914 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe034005180 0x7fe03400b990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:27:16.843 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.815+0000 7fe047fff640 1 -- 172.21.15.16:0/2148740914 reap_dead start 2023-05-22T15:27:16.844 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.819+0000 7fe03ffff640 1 --2- 172.21.15.16:0/2148740914 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe0340047d0 0x7fe034004c40 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fe03000c240 tx=0x7fe03001f3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49ae842a9a1545f6 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:16.844 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.819+0000 7fe046ffd640 1 -- 172.21.15.16:0/2148740914 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0300058e0 con 0x7fe0340047d0 2023-05-22T15:27:16.844 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.819+0000 7fe046ffd640 1 -- 172.21.15.16:0/2148740914 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe030005a40 con 0x7fe0340047d0 2023-05-22T15:27:16.845 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.819+0000 7fe046ffd640 1 -- 172.21.15.16:0/2148740914 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe030005e70 con 0x7fe0340047d0 2023-05-22T15:27:16.845 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.819+0000 7fe04df91640 1 -- 172.21.15.16:0/2148740914 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe0340047d0 msgr2=0x7fe034004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:16.845 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.819+0000 7fe04df91640 1 --2- 172.21.15.16:0/2148740914 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe0340047d0 0x7fe034004c40 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fe03000c240 tx=0x7fe03001f3b0 comp rx=0 tx=0).stop 2023-05-22T15:27:16.846 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.819+0000 7fe04df91640 1 -- 172.21.15.16:0/2148740914 shutdown_connections 2023-05-22T15:27:16.846 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.819+0000 7fe04df91640 1 --2- 172.21.15.16:0/2148740914 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe0340047d0 0x7fe034004c40 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:16.846 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.819+0000 7fe04df91640 1 -- 172.21.15.16:0/2148740914 >> 172.21.15.16:0/2148740914 conn(0x7fe048069720 msgr2=0x7fe048069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:16.847 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.819+0000 7fe04df91640 1 -- 172.21.15.16:0/2148740914 shutdown_connections 2023-05-22T15:27:16.847 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.819+0000 7fe04df91640 1 -- 172.21.15.16:0/2148740914 wait complete. 2023-05-22T15:27:16.847 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.819+0000 7fe04df91640 1 Processor -- start 2023-05-22T15:27:16.848 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.819+0000 7fe04df91640 1 -- start start 2023-05-22T15:27:16.848 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.819+0000 7fe04df91640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe0340047d0 0x7fe0481493d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:16.848 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.819+0000 7fe04df91640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe0480c1b10 0x7fe0480c8e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:16.848 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.819+0000 7fe04df91640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe0480c1f80 0x7fe0480c93b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:16.849 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.819+0000 7fe04df91640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0480c4a80 con 0x7fe0480c1f80 2023-05-22T15:27:16.849 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.819+0000 7fe04df91640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe048149910 con 0x7fe0340047d0 2023-05-22T15:27:16.849 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.819+0000 7fe04df91640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0480c39e0 con 0x7fe0480c1b10 2023-05-22T15:27:16.850 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.819+0000 7fe047fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe0480c1f80 0x7fe0480c93b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:16.850 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.819+0000 7fe047fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe0480c1f80 0x7fe0480c93b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38126/0 (socket says 172.21.15.16:38126) 2023-05-22T15:27:16.850 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.819+0000 7fe047fff640 1 -- 172.21.15.16:0/221165591 learned_addr learned my addr 172.21.15.16:0/221165591 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:16.851 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.819+0000 7fe0477fe640 1 --2- 172.21.15.16:0/221165591 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe0340047d0 0x7fe0481493d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:16.851 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.819+0000 7fe03ffff640 1 --2- 172.21.15.16:0/221165591 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe0480c1b10 0x7fe0480c8e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:16.851 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.819+0000 7fe047fff640 1 -- 172.21.15.16:0/221165591 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe0480c1b10 msgr2=0x7fe0480c8e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:16.851 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.819+0000 7fe047fff640 1 --2- 172.21.15.16:0/221165591 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe0480c1b10 0x7fe0480c8e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:16.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.819+0000 7fe047fff640 1 -- 172.21.15.16:0/221165591 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe0340047d0 msgr2=0x7fe0481493d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:16.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.819+0000 7fe047fff640 1 --2- 172.21.15.16:0/221165591 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe0340047d0 0x7fe0481493d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:16.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.819+0000 7fe047fff640 1 -- 172.21.15.16:0/221165591 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe04815c820 con 0x7fe0480c1f80 2023-05-22T15:27:16.853 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.819+0000 7fe047fff640 1 --2- 172.21.15.16:0/221165591 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe0480c1f80 0x7fe0480c93b0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fe0480b09d0 tx=0x7fe04000d830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:16.853 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.819+0000 7fe044ff9640 1 -- 172.21.15.16:0/221165591 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe04000b070 con 0x7fe0480c1f80 2023-05-22T15:27:16.853 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.819+0000 7fe044ff9640 1 -- 172.21.15.16:0/221165591 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe040003980 con 0x7fe0480c1f80 2023-05-22T15:27:16.854 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.819+0000 7fe044ff9640 1 -- 172.21.15.16:0/221165591 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe040003d30 con 0x7fe0480c1f80 2023-05-22T15:27:16.854 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.819+0000 7fe04df91640 1 -- 172.21.15.16:0/221165591 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe04815d5e0 con 0x7fe0480c1f80 2023-05-22T15:27:16.854 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.819+0000 7fe04df91640 1 -- 172.21.15.16:0/221165591 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe04815cb00 con 0x7fe0480c1f80 2023-05-22T15:27:16.855 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.823+0000 7fe04df91640 1 -- 172.21.15.16:0/221165591 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe04804dbe0 con 0x7fe0480c1f80 2023-05-22T15:27:16.855 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.823+0000 7fe044ff9640 1 -- 172.21.15.16:0/221165591 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7fe040005070 con 0x7fe0480c1f80 2023-05-22T15:27:16.855 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.823+0000 7fe044ff9640 1 --2- 172.21.15.16:0/221165591 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fe028065380 0x7fe028067830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:16.855 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.823+0000 7fe0477fe640 1 --2- 172.21.15.16:0/221165591 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fe028065380 0x7fe028067830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:16.856 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.823+0000 7fe044ff9640 1 -- 172.21.15.16:0/221165591 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6813+0+0 (secure 0 0 0) 0x7fe040081d60 con 0x7fe0480c1f80 2023-05-22T15:27:16.856 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.823+0000 7fe0477fe640 1 --2- 172.21.15.16:0/221165591 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fe028065380 0x7fe028067830 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fe038009c40 tx=0x7fe038008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:16.857 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:16.827+0000 7fe044ff9640 1 -- 172.21.15.16:0/221165591 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fe04004f6f0 con 0x7fe0480c1f80 2023-05-22T15:27:17.023 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.015+0000 7fe04df91640 1 -- 172.21.15.16:0/221165591 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe0480c1840 con 0x7fe0480c1f80 2023-05-22T15:27:17.025 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.015+0000 7fe044ff9640 1 -- 172.21.15.16:0/221165591 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4641 (secure 0 0 0) 0x7fe040018070 con 0x7fe0480c1f80 2023-05-22T15:27:17.025 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:27:17.025 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4261,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6834/3661808159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":3661808159},{"type":"v1","addr":"172.21.15.62:6834","nonce":3661808159}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4264,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/2227467222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6833","nonce":2227467222},{"type":"v1","addr":"172.21.15.62:6835","nonce":2227467222}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4284,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6836/2808106087","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2808106087},{"type":"v1","addr":"172.21.15.16:6836","nonce":2808106087}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-05-22T15:26:49.546008+0000","modified":"2023-05-22T15:27:06.047204+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4287},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4287":{"gid":4287,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.16:6837/283322289","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6835","nonce":283322289},{"type":"v1","addr":"172.21.15.16:6837","nonce":283322289}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-05-22T15:27:17.026 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 8 2023-05-22T15:27:17.027 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.019+0000 7fe04df91640 1 -- 172.21.15.16:0/221165591 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fe028065380 msgr2=0x7fe028067830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:17.028 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.019+0000 7fe04df91640 1 --2- 172.21.15.16:0/221165591 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fe028065380 0x7fe028067830 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fe038009c40 tx=0x7fe038008040 comp rx=0 tx=0).stop 2023-05-22T15:27:17.028 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.019+0000 7fe04df91640 1 -- 172.21.15.16:0/221165591 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe0480c1f80 msgr2=0x7fe0480c93b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:17.028 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.019+0000 7fe04df91640 1 --2- 172.21.15.16:0/221165591 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe0480c1f80 0x7fe0480c93b0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fe0480b09d0 tx=0x7fe04000d830 comp rx=0 tx=0).stop 2023-05-22T15:27:17.029 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.019+0000 7fe04df91640 1 -- 172.21.15.16:0/221165591 shutdown_connections 2023-05-22T15:27:17.029 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.019+0000 7fe04df91640 1 --2- 172.21.15.16:0/221165591 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe0480c1f80 0x7fe0480c93b0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:17.029 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.019+0000 7fe04df91640 1 --2- 172.21.15.16:0/221165591 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fe028065380 0x7fe028067830 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:17.029 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.019+0000 7fe04df91640 1 --2- 172.21.15.16:0/221165591 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe0480c1b10 0x7fe0480c8e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:17.030 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.019+0000 7fe04df91640 1 --2- 172.21.15.16:0/221165591 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe0340047d0 0x7fe0481493d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:17.031 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.019+0000 7fe04df91640 1 -- 172.21.15.16:0/221165591 >> 172.21.15.16:0/221165591 conn(0x7fe048069720 msgr2=0x7fe048070120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:17.031 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.019+0000 7fe04df91640 1 -- 172.21.15.16:0/221165591 shutdown_connections 2023-05-22T15:27:17.031 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.019+0000 7fe04df91640 1 -- 172.21.15.16:0/221165591 wait complete. 2023-05-22T15:27:17.446 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 8, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-05-22T15:26:49.546008+0000', 'modified': '2023-05-22T15:27:06.047204+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4287}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4287': {'gid': 4287, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.16:6837/283322289', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6835', 'nonce': 283322289}, {'type': 'v1', 'addr': '172.21.15.16:6837', 'nonce': 283322289}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [4, 5], 'metadata_pool': 3, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-05-22T15:27:17.446 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-05-22T15:27:17.446 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2023-05-22T15:27:17.583 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.575+0000 7ff7bb1cc640 1 Processor -- start 2023-05-22T15:27:17.583 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.575+0000 7ff7bb1cc640 1 -- start start 2023-05-22T15:27:17.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.575+0000 7ff7bb1cc640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff7b40c9b40 0x7ff7b415e460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:17.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.575+0000 7ff7bb1cc640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7ff7b4070df0 0x7ff7b4071210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:17.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.575+0000 7ff7bb1cc640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7b40b0650 con 0x7ff7b40ca4e0 2023-05-22T15:27:17.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.575+0000 7ff7bb1cc640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7b40b0370 con 0x7ff7b40c9b40 2023-05-22T15:27:17.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.575+0000 7ff7bb1cc640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7b415d980 con 0x7ff7b4070df0 2023-05-22T15:27:17.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.575+0000 7ff7b3fff640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff7b40c9b40 0x7ff7b415e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:17.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.575+0000 7ff7b3fff640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff7b40c9b40 0x7ff7b415e460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38130/0 (socket says 172.21.15.16:38130) 2023-05-22T15:27:17.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.575+0000 7ff7b3fff640 1 -- 172.21.15.16:0/2488321408 learned_addr learned my addr 172.21.15.16:0/2488321408 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:17.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.575+0000 7ff7b9769640 1 --2- 172.21.15.16:0/2488321408 >> v2:172.21.15.62:3300/0 conn(0x7ff7b4070df0 0x7ff7b4071210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:17.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.575+0000 7ff7b3fff640 1 -- 172.21.15.16:0/2488321408 >> v2:172.21.15.62:3300/0 conn(0x7ff7b4070df0 msgr2=0x7ff7b4071210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:17.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.575+0000 7ff7b3fff640 1 --2- 172.21.15.16:0/2488321408 >> v2:172.21.15.62:3300/0 conn(0x7ff7b4070df0 0x7ff7b4071210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:17.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.575+0000 7ff7b3fff640 1 -- 172.21.15.16:0/2488321408 >> v1:172.21.15.62:6789/0 conn(0x7ff7b40ca4e0 legacy=0x7ff7b40afb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:17.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.575+0000 7ff7b3fff640 1 -- 172.21.15.16:0/2488321408 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7b415dc60 con 0x7ff7b40c9b40 2023-05-22T15:27:17.587 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.579+0000 7ff7b3fff640 1 --2- 172.21.15.16:0/2488321408 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff7b40c9b40 0x7ff7b415e460 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7ff7b40cad10 tx=0x7ff7a001c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a942b68210b7309 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:17.588 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.579+0000 7ff7b37fe640 1 -- 172.21.15.16:0/2488321408 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7a00a7070 con 0x7ff7b40c9b40 2023-05-22T15:27:17.588 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.579+0000 7ff7b37fe640 1 -- 172.21.15.16:0/2488321408 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7a00ab430 con 0x7ff7b40c9b40 2023-05-22T15:27:17.588 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.579+0000 7ff7b37fe640 1 -- 172.21.15.16:0/2488321408 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7a00ab7c0 con 0x7ff7b40c9b40 2023-05-22T15:27:17.589 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.579+0000 7ff7bb1cc640 1 -- 172.21.15.16:0/2488321408 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff7b40c9b40 msgr2=0x7ff7b415e460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:17.589 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.579+0000 7ff7bb1cc640 1 --2- 172.21.15.16:0/2488321408 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff7b40c9b40 0x7ff7b415e460 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7ff7b40cad10 tx=0x7ff7a001c4d0 comp rx=0 tx=0).stop 2023-05-22T15:27:17.589 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.579+0000 7ff7bb1cc640 1 -- 172.21.15.16:0/2488321408 shutdown_connections 2023-05-22T15:27:17.589 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.579+0000 7ff7bb1cc640 1 --2- 172.21.15.16:0/2488321408 >> v2:172.21.15.62:3300/0 conn(0x7ff7b4070df0 0x7ff7b4071210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:17.589 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.579+0000 7ff7bb1cc640 1 --2- 172.21.15.16:0/2488321408 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff7b40c9b40 0x7ff7b415e460 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:17.589 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.579+0000 7ff7bb1cc640 1 -- 172.21.15.16:0/2488321408 >> 172.21.15.16:0/2488321408 conn(0x7ff7b4069720 msgr2=0x7ff7b4069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:17.590 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.579+0000 7ff7bb1cc640 1 -- 172.21.15.16:0/2488321408 shutdown_connections 2023-05-22T15:27:17.590 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.579+0000 7ff7bb1cc640 1 -- 172.21.15.16:0/2488321408 wait complete. 2023-05-22T15:27:17.591 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.583+0000 7ff7bb1cc640 1 Processor -- start 2023-05-22T15:27:17.591 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.583+0000 7ff7bb1cc640 1 -- start start 2023-05-22T15:27:17.591 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.583+0000 7ff7bb1cc640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff7b4070df0 0x7ff7b41575b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:17.591 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.583+0000 7ff7bb1cc640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff7b40c9b40 0x7ff7b40c7ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:17.592 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.583+0000 7ff7bb1cc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff7b40ca4e0 0x7ff7b40b0eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:17.592 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.583+0000 7ff7bb1cc640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7b4157af0 con 0x7ff7b40ca4e0 2023-05-22T15:27:17.592 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.583+0000 7ff7bb1cc640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7b40c38c0 con 0x7ff7b4070df0 2023-05-22T15:27:17.592 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.583+0000 7ff7bb1cc640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7b40c2cb0 con 0x7ff7b40c9b40 2023-05-22T15:27:17.592 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.583+0000 7ff7b9769640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff7b40ca4e0 0x7ff7b40b0eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:17.592 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.583+0000 7ff7b9769640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff7b40ca4e0 0x7ff7b40b0eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38134/0 (socket says 172.21.15.16:38134) 2023-05-22T15:27:17.592 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.583+0000 7ff7b9769640 1 -- 172.21.15.16:0/4139063160 learned_addr learned my addr 172.21.15.16:0/4139063160 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:17.593 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.583+0000 7ff7b8f68640 1 --2- 172.21.15.16:0/4139063160 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff7b4070df0 0x7ff7b41575b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:17.593 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.583+0000 7ff7b3fff640 1 --2- 172.21.15.16:0/4139063160 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff7b40c9b40 0x7ff7b40c7ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:17.593 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.583+0000 7ff7b9769640 1 -- 172.21.15.16:0/4139063160 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff7b40c9b40 msgr2=0x7ff7b40c7ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:17.593 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.583+0000 7ff7b9769640 1 --2- 172.21.15.16:0/4139063160 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff7b40c9b40 0x7ff7b40c7ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:17.593 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.583+0000 7ff7b9769640 1 -- 172.21.15.16:0/4139063160 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff7b4070df0 msgr2=0x7ff7b41575b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:17.593 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.583+0000 7ff7b9769640 1 --2- 172.21.15.16:0/4139063160 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff7b4070df0 0x7ff7b41575b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:17.593 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.583+0000 7ff7b9769640 1 -- 172.21.15.16:0/4139063160 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7a00094d0 con 0x7ff7b40ca4e0 2023-05-22T15:27:17.593 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.583+0000 7ff7b9769640 1 --2- 172.21.15.16:0/4139063160 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff7b40ca4e0 0x7ff7b40b0eb0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7ff7ac007e40 tx=0x7ff7ac00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:17.594 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.583+0000 7ff7b17fa640 1 -- 172.21.15.16:0/4139063160 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7ac014620 con 0x7ff7b40ca4e0 2023-05-22T15:27:17.594 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.583+0000 7ff7b17fa640 1 -- 172.21.15.16:0/4139063160 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7ac00ee90 con 0x7ff7b40ca4e0 2023-05-22T15:27:17.595 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.583+0000 7ff7bb1cc640 1 -- 172.21.15.16:0/4139063160 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7b40c8e80 con 0x7ff7b40ca4e0 2023-05-22T15:27:17.595 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.583+0000 7ff7b17fa640 1 -- 172.21.15.16:0/4139063160 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7ac0149e0 con 0x7ff7b40ca4e0 2023-05-22T15:27:17.595 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.583+0000 7ff7bb1cc640 1 -- 172.21.15.16:0/4139063160 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7b40ca8b0 con 0x7ff7b40ca4e0 2023-05-22T15:27:17.595 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.587+0000 7ff7be2ee000 1 Processor -- start 2023-05-22T15:27:17.596 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.587+0000 7ff7b17fa640 1 -- 172.21.15.16:0/4139063160 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7ff7ac014b40 con 0x7ff7b40ca4e0 2023-05-22T15:27:17.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.587+0000 7ff7be2ee000 1 -- start start 2023-05-22T15:27:17.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.587+0000 7ff7b17fa640 1 --2- 172.21.15.16:0/4139063160 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ff7980653b0 0x7ff798067860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:17.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.587+0000 7ff7b17fa640 1 -- 172.21.15.16:0/4139063160 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6813+0+0 (secure 0 0 0) 0x7ff7ac0834c0 con 0x7ff7b40ca4e0 2023-05-22T15:27:17.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.587+0000 7ff7b3fff640 1 --2- 172.21.15.16:0/4139063160 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ff7980653b0 0x7ff798067860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:17.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.587+0000 7ff7be2ee000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561acceebfd0 0x561acd15ec30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:17.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.587+0000 7ff7be2ee000 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561acd15f170 0x561acd15f540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:17.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.587+0000 7ff7be2ee000 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x561acd112a10 0x561acd112de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:17.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.587+0000 7ff7be2ee000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x561accfdcbd0 con 0x561acceebfd0 2023-05-22T15:27:17.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.587+0000 7ff7be2ee000 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x561accef9dd0 con 0x561acd15f170 2023-05-22T15:27:17.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.587+0000 7ff7be2ee000 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x561accf6a640 con 0x561acd112a10 2023-05-22T15:27:17.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.587+0000 7ff7b8f68640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561acceebfd0 0x561acd15ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:17.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.587+0000 7ff7b8f68640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561acceebfd0 0x561acd15ec30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38144/0 (socket says 172.21.15.16:38144) 2023-05-22T15:27:17.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.587+0000 7ff7b8f68640 1 -- 172.21.15.16:0/611157073 learned_addr learned my addr 172.21.15.16:0/611157073 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:17.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.587+0000 7ff7b3fff640 1 --2- 172.21.15.16:0/4139063160 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ff7980653b0 0x7ff798067860 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7ff7a00af000 tx=0x7ff7a00a7000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:17.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.587+0000 7ff7b3fff640 1 --2- 172.21.15.16:0/611157073 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561acd15f170 0x561acd15f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:17.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.587+0000 7ff7b9769640 1 --2- 172.21.15.16:0/611157073 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x561acd112a10 0x561acd112de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:17.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.587+0000 7ff7b8f68640 1 -- 172.21.15.16:0/611157073 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x561acd112a10 msgr2=0x561acd112de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:17.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.587+0000 7ff7b8f68640 1 --2- 172.21.15.16:0/611157073 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x561acd112a10 0x561acd112de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:17.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.587+0000 7ff7b8f68640 1 -- 172.21.15.16:0/611157073 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561acd15f170 msgr2=0x561acd15f540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:17.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.587+0000 7ff7b8f68640 1 --2- 172.21.15.16:0/611157073 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561acd15f170 0x561acd15f540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:17.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.587+0000 7ff7b8f68640 1 -- 172.21.15.16:0/611157073 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561accf290b0 con 0x561acceebfd0 2023-05-22T15:27:17.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.587+0000 7ff7b3fff640 1 --2- 172.21.15.16:0/611157073 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561acd15f170 0x561acd15f540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:27:17.600 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.587+0000 7ff7b9769640 1 --2- 172.21.15.16:0/611157073 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x561acd112a10 0x561acd112de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:27:17.600 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.587+0000 7ff7b8f68640 1 --2- 172.21.15.16:0/611157073 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561acceebfd0 0x561acd15ec30 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7ff7a8001540 tx=0x7ff7a8001a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e135f303b4f6c2a8 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:17.603 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.587+0000 7ff79e7fc640 1 -- 172.21.15.16:0/611157073 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7a80050d0 con 0x561acceebfd0 2023-05-22T15:27:17.603 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.587+0000 7ff79e7fc640 1 -- 172.21.15.16:0/611157073 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7a8005230 con 0x561acceebfd0 2023-05-22T15:27:17.603 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.587+0000 7ff79e7fc640 1 -- 172.21.15.16:0/611157073 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7a800e6e0 con 0x561acceebfd0 2023-05-22T15:27:17.603 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.591+0000 7ff7be2ee000 1 -- 172.21.15.16:0/611157073 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561acceebfd0 msgr2=0x561acd15ec30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:17.604 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.591+0000 7ff7be2ee000 1 --2- 172.21.15.16:0/611157073 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561acceebfd0 0x561acd15ec30 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7ff7a8001540 tx=0x7ff7a8001a00 comp rx=0 tx=0).stop 2023-05-22T15:27:17.604 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.591+0000 7ff7be2ee000 1 -- 172.21.15.16:0/611157073 shutdown_connections 2023-05-22T15:27:17.604 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.591+0000 7ff7be2ee000 1 --2- 172.21.15.16:0/611157073 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x561acd112a10 0x561acd112de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:17.604 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.591+0000 7ff7be2ee000 1 --2- 172.21.15.16:0/611157073 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561acd15f170 0x561acd15f540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:17.604 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.591+0000 7ff7be2ee000 1 --2- 172.21.15.16:0/611157073 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561acceebfd0 0x561acd15ec30 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:17.604 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.591+0000 7ff7be2ee000 1 -- 172.21.15.16:0/611157073 >> 172.21.15.16:0/611157073 conn(0x561accfbb270 msgr2=0x561acd142220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:17.604 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.591+0000 7ff7be2ee000 1 -- 172.21.15.16:0/611157073 shutdown_connections 2023-05-22T15:27:17.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.591+0000 7ff7be2ee000 1 -- 172.21.15.16:0/611157073 wait complete. 2023-05-22T15:27:17.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.591+0000 7ff7be2ee000 1 -- start start 2023-05-22T15:27:17.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.591+0000 7ff7be2ee000 1 Processor -- start 2023-05-22T15:27:17.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.591+0000 7ff7be2ee000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561acceebfd0 0x561acd11aae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:17.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.591+0000 7ff7be2ee000 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x561acd112a10 0x561acd11b020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:17.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.591+0000 7ff7be2ee000 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561acd15f170 0x561acd1444a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:17.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.591+0000 7ff7be2ee000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x561accf63780 con 0x561acceebfd0 2023-05-22T15:27:17.606 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.591+0000 7ff7be2ee000 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x561accf21250 con 0x561acd15f170 2023-05-22T15:27:17.606 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.591+0000 7ff7be2ee000 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x561accf20ef0 con 0x561acd112a10 2023-05-22T15:27:17.606 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.591+0000 7ff7be2ee000 10 client.0 _ll_register_callbacks cb 0x561acd100b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-05-22T15:27:17.606 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.591+0000 7ff7b8f68640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561acceebfd0 0x561acd11aae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:17.606 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.591+0000 7ff7b8f68640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561acceebfd0 0x561acd11aae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38154/0 (socket says 172.21.15.16:38154) 2023-05-22T15:27:17.606 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.591+0000 7ff7b8f68640 1 -- 172.21.15.16:0/2792444127 learned_addr learned my addr 172.21.15.16:0/2792444127 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:17.606 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.591+0000 7ff7b9769640 1 --2- 172.21.15.16:0/2792444127 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561acd15f170 0x561acd1444a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:17.606 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.591+0000 7ff7b3fff640 1 --2- 172.21.15.16:0/2792444127 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x561acd112a10 0x561acd11b020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:17.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.591+0000 7ff7b8f68640 1 -- 172.21.15.16:0/2792444127 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x561acd112a10 msgr2=0x561acd11b020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:17.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.591+0000 7ff7b8f68640 1 --2- 172.21.15.16:0/2792444127 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x561acd112a10 0x561acd11b020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:17.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.591+0000 7ff7b8f68640 1 -- 172.21.15.16:0/2792444127 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561acd15f170 msgr2=0x561acd1444a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:17.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.591+0000 7ff7b8f68640 1 --2- 172.21.15.16:0/2792444127 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561acd15f170 0x561acd1444a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:17.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.591+0000 7ff7b8f68640 1 -- 172.21.15.16:0/2792444127 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7a800f040 con 0x561acceebfd0 2023-05-22T15:27:17.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.591+0000 7ff7b9769640 1 --2- 172.21.15.16:0/2792444127 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561acd15f170 0x561acd1444a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:27:17.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.591+0000 7ff7b8f68640 1 --2- 172.21.15.16:0/2792444127 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561acceebfd0 0x561acd11aae0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7ff7a8001a40 tx=0x7ff7a8000ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42a165a759ad2237 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:17.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.591+0000 7ff7937fe640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2023-05-22T15:27:17.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.591+0000 7ff7937fe640 1 -- 172.21.15.16:0/2792444127 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7a800e6e0 con 0x561acceebfd0 2023-05-22T15:27:17.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.591+0000 7ff7937fe640 1 -- 172.21.15.16:0/2792444127 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(27..27 src has 1..27) v4 ==== 6813+0+0 (secure 0 0 0) 0x7ff7a800e840 con 0x561acceebfd0 2023-05-22T15:27:17.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.591+0000 7ff7be2ee000 1 -- 172.21.15.16:0/2792444127 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x561accf27c60 con 0x561acceebfd0 2023-05-22T15:27:17.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.591+0000 7ff7937fe640 1 -- 172.21.15.16:0/2792444127 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=8) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff7a800ea40 con 0x561acceebfd0 2023-05-22T15:27:17.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.591+0000 7ff7be2ee000 10 client.4845 fetch_fsmap learned FSMap version 8 2023-05-22T15:27:17.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.591+0000 7ff7be2ee000 1 -- 172.21.15.16:0/2792444127 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=8,monmap=2+}) v3 -- 0x561accc60890 con 0x561acceebfd0 2023-05-22T15:27:17.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.591+0000 7ff7937fe640 1 -- 172.21.15.16:0/2792444127 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x561accf27c60 con 0x561acceebfd0 2023-05-22T15:27:17.609 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.591+0000 7ff7937fe640 1 -- 172.21.15.16:0/2792444127 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 8) v1 ==== 3114+0+0 (secure 0 0 0) 0x7ff7a800eba0 con 0x561acceebfd0 2023-05-22T15:27:17.609 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.591+0000 7ff7be2ee000 10 client.4845 fetch_fsmap finished waiting for FSMap version 8 2023-05-22T15:27:17.609 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.591+0000 7ff7be2ee000 10 client.4845 resolve_mds: resolved name 'c' to daemon mds.c 2023-05-22T15:27:17.609 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.591+0000 7ff7be2ee000 20 client.4845 populate_metadata read hostname 'smithi016' 2023-05-22T15:27:17.609 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.591+0000 7ff7be2ee000 1 --2- 172.21.15.16:0/2792444127 >> [v2:172.21.15.16:6835/283322289,v1:172.21.15.16:6837/283322289] conn(0x561acd16e8e0 0x561acd170cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:17.609 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.591+0000 7ff7be2ee000 4 client.4845 mds_command: new command op to 4287 tid=0[{"prefix": "get_command_descriptions"}] 2023-05-22T15:27:17.609 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.591+0000 7ff7be2ee000 1 -- 172.21.15.16:0/2792444127 --> [v2:172.21.15.16:6835/283322289,v1:172.21.15.16:6837/283322289] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x561accf620d0 con 0x561acd16e8e0 2023-05-22T15:27:17.609 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.591+0000 7ff7b3fff640 1 --2- 172.21.15.16:0/2792444127 >> [v2:172.21.15.16:6835/283322289,v1:172.21.15.16:6837/283322289] conn(0x561acd16e8e0 0x561acd170cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:17.610 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.591+0000 7ff7937fe640 10 client.4845 ms_handle_connect on v2:172.21.15.16:6835/283322289 2023-05-22T15:27:17.610 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.591+0000 7ff7b3fff640 1 --2- 172.21.15.16:0/2792444127 >> [v2:172.21.15.16:6835/283322289,v1:172.21.15.16:6837/283322289] conn(0x561acd16e8e0 0x561acd170cc0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2cfe91e4463e895b server_cookie=887eaec2480e21a1 in_seq=0 out_seq=0 2023-05-22T15:27:17.610 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.591+0000 7ff7937fe640 1 -- 172.21.15.16:0/2792444127 <== mds.0 v2:172.21.15.16:6835/283322289 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x561accf620d0 con 0x561acd16e8e0 2023-05-22T15:27:17.610 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.591+0000 7ff7937fe640 10 client.4845 handle_command_reply: tid=0 2023-05-22T15:27:17.610 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.591+0000 7ff7be2ee000 1 client.4845 shutdown 2023-05-22T15:27:17.610 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.595+0000 7ff7be2ee000 1 -- 172.21.15.16:0/2792444127 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561acceebfd0 msgr2=0x561acd11aae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:17.610 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.595+0000 7ff7be2ee000 1 --2- 172.21.15.16:0/2792444127 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561acceebfd0 0x561acd11aae0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7ff7a8001a40 tx=0x7ff7a8000ee0 comp rx=0 tx=0).stop 2023-05-22T15:27:17.611 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.595+0000 7ff7be2ee000 1 -- 172.21.15.16:0/2792444127 shutdown_connections 2023-05-22T15:27:17.611 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.595+0000 7ff7be2ee000 1 --2- 172.21.15.16:0/2792444127 >> [v2:172.21.15.16:6835/283322289,v1:172.21.15.16:6837/283322289] conn(0x561acd16e8e0 0x561acd170cc0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:17.611 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.595+0000 7ff7be2ee000 1 --2- 172.21.15.16:0/2792444127 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561acd15f170 0x561acd1444a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:17.611 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.595+0000 7ff7be2ee000 1 --2- 172.21.15.16:0/2792444127 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x561acd112a10 0x561acd11b020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:17.611 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.595+0000 7ff7937fe640 0 client.4845 ms_handle_reset on v2:172.21.15.16:6835/283322289 2023-05-22T15:27:17.611 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.595+0000 7ff7be2ee000 1 --2- 172.21.15.16:0/2792444127 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561acceebfd0 0x561acd11aae0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:17.611 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.595+0000 7ff7be2ee000 1 -- 172.21.15.16:0/2792444127 >> 172.21.15.16:0/2792444127 conn(0x561accfbb270 msgr2=0x561acd112100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:17.611 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.595+0000 7ff7be2ee000 1 -- 172.21.15.16:0/2792444127 shutdown_connections 2023-05-22T15:27:17.612 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.595+0000 7ff7be2ee000 1 -- 172.21.15.16:0/2792444127 wait complete. 2023-05-22T15:27:17.612 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.595+0000 7ff7be2ee000 20 client.4845 trim_cache size 0 max 16384 2023-05-22T15:27:17.616 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.607+0000 7ff7be2ee000 1 Processor -- start 2023-05-22T15:27:17.616 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.607+0000 7ff7be2ee000 1 -- start start 2023-05-22T15:27:17.618 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.607+0000 7ff7be2ee000 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561acceebfd0 0x561acd112de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:17.618 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.607+0000 7ff7be2ee000 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x561acd112a10 0x561acd113320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:17.618 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.607+0000 7ff7be2ee000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561acd15f170 0x561acd113860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:17.618 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.607+0000 7ff7be2ee000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x561accc21940 con 0x561acd15f170 2023-05-22T15:27:17.618 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.607+0000 7ff7be2ee000 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x561acd1430a0 con 0x561acceebfd0 2023-05-22T15:27:17.618 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.607+0000 7ff7be2ee000 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x561accf6b8d0 con 0x561acd112a10 2023-05-22T15:27:17.618 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.607+0000 7ff7b9769640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561acd15f170 0x561acd113860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:17.619 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.607+0000 7ff7b9769640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561acd15f170 0x561acd113860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38166/0 (socket says 172.21.15.16:38166) 2023-05-22T15:27:17.619 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.607+0000 7ff7b9769640 1 -- 172.21.15.16:0/437610200 learned_addr learned my addr 172.21.15.16:0/437610200 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:17.619 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.611+0000 7ff7b3fff640 1 --2- 172.21.15.16:0/437610200 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x561acd112a10 0x561acd113320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:17.619 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.611+0000 7ff7b8f68640 1 --2- 172.21.15.16:0/437610200 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561acceebfd0 0x561acd112de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:17.619 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.611+0000 7ff7b9769640 1 -- 172.21.15.16:0/437610200 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x561acd112a10 msgr2=0x561acd113320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:17.619 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.611+0000 7ff7b9769640 1 --2- 172.21.15.16:0/437610200 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x561acd112a10 0x561acd113320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:17.619 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.611+0000 7ff7b9769640 1 -- 172.21.15.16:0/437610200 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561acceebfd0 msgr2=0x561acd112de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:17.620 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.611+0000 7ff7b9769640 1 --2- 172.21.15.16:0/437610200 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561acceebfd0 0x561acd112de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:17.620 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.611+0000 7ff7b9769640 1 -- 172.21.15.16:0/437610200 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561accf29360 con 0x561acd15f170 2023-05-22T15:27:17.620 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.611+0000 7ff7b9769640 1 --2- 172.21.15.16:0/437610200 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561acd15f170 0x561acd113860 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7ff7ac00c040 tx=0x7ff7ac05cd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3ea9309cb1008c1 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:17.620 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.611+0000 7ff79effd640 1 -- 172.21.15.16:0/437610200 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7ac052f60 con 0x561acd15f170 2023-05-22T15:27:17.620 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.611+0000 7ff79effd640 1 -- 172.21.15.16:0/437610200 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7ac055e90 con 0x561acd15f170 2023-05-22T15:27:17.620 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.611+0000 7ff79effd640 1 -- 172.21.15.16:0/437610200 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7ac0832b0 con 0x561acd15f170 2023-05-22T15:27:17.623 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.611+0000 7ff7be2ee000 1 -- 172.21.15.16:0/437610200 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561acd15f170 msgr2=0x561acd113860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:17.624 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.611+0000 7ff7be2ee000 1 --2- 172.21.15.16:0/437610200 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561acd15f170 0x561acd113860 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7ff7ac00c040 tx=0x7ff7ac05cd10 comp rx=0 tx=0).stop 2023-05-22T15:27:17.624 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.611+0000 7ff7be2ee000 1 -- 172.21.15.16:0/437610200 shutdown_connections 2023-05-22T15:27:17.624 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.611+0000 7ff7be2ee000 1 --2- 172.21.15.16:0/437610200 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561acd15f170 0x561acd113860 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:17.624 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.611+0000 7ff7be2ee000 1 --2- 172.21.15.16:0/437610200 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x561acd112a10 0x561acd113320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:17.624 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.611+0000 7ff7be2ee000 1 --2- 172.21.15.16:0/437610200 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561acceebfd0 0x561acd112de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:17.624 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.611+0000 7ff7be2ee000 1 -- 172.21.15.16:0/437610200 >> 172.21.15.16:0/437610200 conn(0x561accfbb270 msgr2=0x561acd141f10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:17.624 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.611+0000 7ff7be2ee000 1 -- 172.21.15.16:0/437610200 shutdown_connections 2023-05-22T15:27:17.625 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.611+0000 7ff7be2ee000 1 -- 172.21.15.16:0/437610200 wait complete. 2023-05-22T15:27:17.625 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.611+0000 7ff7be2ee000 1 -- start start 2023-05-22T15:27:17.625 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.611+0000 7ff7be2ee000 1 Processor -- start 2023-05-22T15:27:17.625 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.611+0000 7ff7be2ee000 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x561acceebfd0 0x561acd1bc080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:17.625 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.611+0000 7ff7be2ee000 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561acd112a10 0x561acd1bc5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:17.625 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.611+0000 7ff7be2ee000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561acd15f170 0x561acd114e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:17.625 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.611+0000 7ff7be2ee000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x561accf662e0 con 0x561acd15f170 2023-05-22T15:27:17.625 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.611+0000 7ff7be2ee000 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x561accf5db20 con 0x561acd112a10 2023-05-22T15:27:17.626 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.611+0000 7ff7be2ee000 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x561accf328e0 con 0x561acceebfd0 2023-05-22T15:27:17.626 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.611+0000 7ff7be2ee000 10 client.0 _ll_register_callbacks cb 0x561accfb3eb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-05-22T15:27:17.626 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.611+0000 7ff7b9769640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561acd15f170 0x561acd114e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:17.626 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.611+0000 7ff7b9769640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561acd15f170 0x561acd114e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38176/0 (socket says 172.21.15.16:38176) 2023-05-22T15:27:17.626 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.611+0000 7ff7b9769640 1 -- 172.21.15.16:0/1684957804 learned_addr learned my addr 172.21.15.16:0/1684957804 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:17.626 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.611+0000 7ff7b8f68640 1 --2- 172.21.15.16:0/1684957804 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x561acceebfd0 0x561acd1bc080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:17.627 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.611+0000 7ff7b3fff640 1 --2- 172.21.15.16:0/1684957804 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561acd112a10 0x561acd1bc5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:17.627 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.611+0000 7ff7b9769640 1 -- 172.21.15.16:0/1684957804 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x561acceebfd0 msgr2=0x561acd1bc080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:17.627 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.611+0000 7ff7b9769640 1 --2- 172.21.15.16:0/1684957804 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x561acceebfd0 0x561acd1bc080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:17.627 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.611+0000 7ff7b9769640 1 -- 172.21.15.16:0/1684957804 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561acd112a10 msgr2=0x561acd1bc5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:17.627 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.611+0000 7ff7b9769640 1 --2- 172.21.15.16:0/1684957804 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561acd112a10 0x561acd1bc5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:17.627 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.611+0000 7ff7b9769640 1 -- 172.21.15.16:0/1684957804 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7b40ca8b0 con 0x561acd15f170 2023-05-22T15:27:17.627 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.611+0000 7ff7b9769640 1 --2- 172.21.15.16:0/1684957804 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561acd15f170 0x561acd114e40 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7ff7ac016010 tx=0x7ff7ac004a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=953e53148889e60e server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:17.628 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.611+0000 7ff7937fe640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2023-05-22T15:27:17.628 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.611+0000 7ff7937fe640 1 -- 172.21.15.16:0/1684957804 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7ac060070 con 0x561acd15f170 2023-05-22T15:27:17.628 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.611+0000 7ff7937fe640 1 -- 172.21.15.16:0/1684957804 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(27..27 src has 1..27) v4 ==== 6813+0+0 (secure 0 0 0) 0x7ff7ac081200 con 0x561acd15f170 2023-05-22T15:27:17.628 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.611+0000 7ff7be2ee000 1 -- 172.21.15.16:0/1684957804 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x561acd119c30 con 0x561acd15f170 2023-05-22T15:27:17.628 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.611+0000 7ff7937fe640 1 -- 172.21.15.16:0/1684957804 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=8) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff7ac0832c0 con 0x561acd15f170 2023-05-22T15:27:17.628 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.611+0000 7ff7be2ee000 10 client.4851 fetch_fsmap learned FSMap version 8 2023-05-22T15:27:17.628 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.611+0000 7ff7be2ee000 1 -- 172.21.15.16:0/1684957804 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=8,monmap=2+}) v3 -- 0x561accf0ebe0 con 0x561acd15f170 2023-05-22T15:27:17.628 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.611+0000 7ff7937fe640 1 -- 172.21.15.16:0/1684957804 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x561acd119c30 con 0x561acd15f170 2023-05-22T15:27:17.629 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.611+0000 7ff7937fe640 1 -- 172.21.15.16:0/1684957804 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 8) v1 ==== 3114+0+0 (secure 0 0 0) 0x7ff7ac005d20 con 0x561acd15f170 2023-05-22T15:27:17.629 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.611+0000 7ff7be2ee000 10 client.4851 fetch_fsmap finished waiting for FSMap version 8 2023-05-22T15:27:17.629 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.611+0000 7ff7be2ee000 10 client.4851 resolve_mds: resolved name 'c' to daemon mds.c 2023-05-22T15:27:17.629 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.611+0000 7ff7be2ee000 20 client.4851 populate_metadata read hostname 'smithi016' 2023-05-22T15:27:17.629 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.611+0000 7ff7be2ee000 1 --2- 172.21.15.16:0/1684957804 >> [v2:172.21.15.16:6835/283322289,v1:172.21.15.16:6837/283322289] conn(0x561acd115800 0x561acd115bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:17.629 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.611+0000 7ff7be2ee000 4 client.4851 mds_command: new command op to 4287 tid=0[{"prefix": "status"}] 2023-05-22T15:27:17.629 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.611+0000 7ff7be2ee000 1 -- 172.21.15.16:0/1684957804 --> [v2:172.21.15.16:6835/283322289,v1:172.21.15.16:6837/283322289] -- command(tid 0: {"prefix": "status"}) v1 -- 0x561accf0a4d0 con 0x561acd115800 2023-05-22T15:27:17.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.611+0000 7ff7b8f68640 1 --2- 172.21.15.16:0/1684957804 >> [v2:172.21.15.16:6835/283322289,v1:172.21.15.16:6837/283322289] conn(0x561acd115800 0x561acd115bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:17.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.615+0000 7ff7b8f68640 1 --2- 172.21.15.16:0/1684957804 >> [v2:172.21.15.16:6835/283322289,v1:172.21.15.16:6837/283322289] conn(0x561acd115800 0x561acd115bd0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a9446133eb846fd9 server_cookie=91c9bfce617a7e47 in_seq=0 out_seq=0 2023-05-22T15:27:17.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.615+0000 7ff7937fe640 10 client.4851 ms_handle_connect on v2:172.21.15.16:6835/283322289 2023-05-22T15:27:17.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.615+0000 7ff7937fe640 1 -- 172.21.15.16:0/1684957804 <== mds.0 v2:172.21.15.16:6835/283322289 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+330 (crc 0 0 0) 0x561accf0a4d0 con 0x561acd115800 2023-05-22T15:27:17.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.615+0000 7ff7937fe640 10 client.4851 handle_command_reply: tid=0 2023-05-22T15:27:17.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.615+0000 7ff7be2ee000 1 client.4851 shutdown 2023-05-22T15:27:17.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.615+0000 7ff7be2ee000 1 -- 172.21.15.16:0/1684957804 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561acd15f170 msgr2=0x561acd114e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:17.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.615+0000 7ff7be2ee000 1 --2- 172.21.15.16:0/1684957804 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561acd15f170 0x561acd114e40 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7ff7ac016010 tx=0x7ff7ac004a50 comp rx=0 tx=0).stop 2023-05-22T15:27:17.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.615+0000 7ff7be2ee000 1 -- 172.21.15.16:0/1684957804 shutdown_connections 2023-05-22T15:27:17.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.615+0000 7ff7be2ee000 1 --2- 172.21.15.16:0/1684957804 >> [v2:172.21.15.16:6835/283322289,v1:172.21.15.16:6837/283322289] conn(0x561acd115800 0x561acd115bd0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:17.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.615+0000 7ff7be2ee000 1 --2- 172.21.15.16:0/1684957804 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561acd15f170 0x561acd114e40 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:17.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.615+0000 7ff7be2ee000 1 --2- 172.21.15.16:0/1684957804 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561acd112a10 0x561acd1bc5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:17.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.615+0000 7ff7be2ee000 1 --2- 172.21.15.16:0/1684957804 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x561acceebfd0 0x561acd1bc080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:17.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.615+0000 7ff7937fe640 0 client.4851 ms_handle_reset on v2:172.21.15.16:6835/283322289 2023-05-22T15:27:17.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.615+0000 7ff7be2ee000 1 -- 172.21.15.16:0/1684957804 >> 172.21.15.16:0/1684957804 conn(0x561accfbb270 msgr2=0x561acd141f10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:17.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.615+0000 7ff7be2ee000 1 -- 172.21.15.16:0/1684957804 shutdown_connections 2023-05-22T15:27:17.632 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.615+0000 7ff7be2ee000 1 -- 172.21.15.16:0/1684957804 wait complete. 2023-05-22T15:27:17.632 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.615+0000 7ff7be2ee000 20 client.4851 trim_cache size 0 max 16384 2023-05-22T15:27:17.632 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-05-22T15:27:17.632 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_fsid": "b2719834-7847-4d98-90a6-f65d57f85c49", 2023-05-22T15:27:17.632 INFO:teuthology.orchestra.run.smithi016.stdout: "whoami": 0, 2023-05-22T15:27:17.633 INFO:teuthology.orchestra.run.smithi016.stdout: "id": 4287, 2023-05-22T15:27:17.633 INFO:teuthology.orchestra.run.smithi016.stdout: "want_state": "up:active", 2023-05-22T15:27:17.633 INFO:teuthology.orchestra.run.smithi016.stdout: "state": "up:active", 2023-05-22T15:27:17.633 INFO:teuthology.orchestra.run.smithi016.stdout: "fs_name": "cephfs", 2023-05-22T15:27:17.633 INFO:teuthology.orchestra.run.smithi016.stdout: "rank_uptime": 28.068087947999999, 2023-05-22T15:27:17.633 INFO:teuthology.orchestra.run.smithi016.stdout: "mdsmap_epoch": 8, 2023-05-22T15:27:17.633 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch": 27, 2023-05-22T15:27:17.634 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch_barrier": 27, 2023-05-22T15:27:17.634 INFO:teuthology.orchestra.run.smithi016.stdout: "uptime": 32.422120732000003 2023-05-22T15:27:17.634 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-05-22T15:27:17.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.615+0000 7ff7bb1cc640 1 -- 172.21.15.16:0/4139063160 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ff7980653b0 msgr2=0x7ff798067860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:17.635 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.615+0000 7ff7bb1cc640 1 --2- 172.21.15.16:0/4139063160 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ff7980653b0 0x7ff798067860 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7ff7a00af000 tx=0x7ff7a00a7000 comp rx=0 tx=0).stop 2023-05-22T15:27:17.635 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.615+0000 7ff7bb1cc640 1 -- 172.21.15.16:0/4139063160 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff7b40ca4e0 msgr2=0x7ff7b40b0eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:17.635 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.615+0000 7ff7bb1cc640 1 --2- 172.21.15.16:0/4139063160 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff7b40ca4e0 0x7ff7b40b0eb0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7ff7ac007e40 tx=0x7ff7ac00e4b0 comp rx=0 tx=0).stop 2023-05-22T15:27:17.635 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.615+0000 7ff7bb1cc640 1 -- 172.21.15.16:0/4139063160 shutdown_connections 2023-05-22T15:27:17.635 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.615+0000 7ff7bb1cc640 1 --2- 172.21.15.16:0/4139063160 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff7b40ca4e0 0x7ff7b40b0eb0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:17.635 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.615+0000 7ff7bb1cc640 1 --2- 172.21.15.16:0/4139063160 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ff7980653b0 0x7ff798067860 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:17.635 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.615+0000 7ff7bb1cc640 1 --2- 172.21.15.16:0/4139063160 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff7b40c9b40 0x7ff7b40c7ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:17.636 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.615+0000 7ff7bb1cc640 1 --2- 172.21.15.16:0/4139063160 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff7b4070df0 0x7ff7b41575b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:17.636 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.615+0000 7ff7bb1cc640 1 -- 172.21.15.16:0/4139063160 >> 172.21.15.16:0/4139063160 conn(0x7ff7b4069720 msgr2=0x7ff7b40b23a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:17.636 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.615+0000 7ff7bb1cc640 1 -- 172.21.15.16:0/4139063160 shutdown_connections 2023-05-22T15:27:17.636 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:17.615+0000 7ff7bb1cc640 1 -- 172.21.15.16:0/4139063160 wait complete. 2023-05-22T15:27:17.638 INFO:teuthology.run_tasks:Running task ceph-fuse... 2023-05-22T15:27:17.656 INFO:tasks.ceph_fuse:Running ceph_fuse task... 2023-05-22T15:27:17.656 INFO:tasks.ceph_fuse:config is {'client.0': {}, 'client.2': {}, 'client.1': {}, 'client.3': {}} 2023-05-22T15:27:17.657 INFO:tasks.ceph_fuse:client.0 config is {} 2023-05-22T15:27:17.657 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2023-05-22T15:27:17.657 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-05-22T15:27:17.657 INFO:tasks.ceph_fuse:client.2 config is {} 2023-05-22T15:27:17.658 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2023-05-22T15:27:17.658 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2023-05-22T15:27:17.658 INFO:tasks.ceph_fuse:client.1 config is {} 2023-05-22T15:27:17.658 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2023-05-22T15:27:17.659 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2023-05-22T15:27:17.659 INFO:tasks.ceph_fuse:client.3 config is {} 2023-05-22T15:27:17.660 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2023-05-22T15:27:17.660 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2023-05-22T15:27:17.661 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:27:17.661 DEBUG:teuthology.orchestra.run.smithi062:> ip netns list 2023-05-22T15:27:17.668 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:27:17.669 DEBUG:teuthology.orchestra.run.smithi062:> sudo ip link delete ceph-brx 2023-05-22T15:27:17.730 INFO:teuthology.orchestra.run.smithi062.stderr:Cannot find device "ceph-brx" 2023-05-22T15:27:17.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-05-22T15:27:17.732 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:27:17.732 DEBUG:teuthology.orchestra.run.smithi016:> ip netns list 2023-05-22T15:27:17.738 INFO:teuthology.orchestra.run.smithi016.stdout:ceph-ns--home-ubuntu-cephtest-mnt.admin (id: 0) 2023-05-22T15:27:17.739 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:27:17.739 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.admin 2023-05-22T15:27:17.795 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:27:17.795 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip link delete ceph-brx 2023-05-22T15:27:17.959 INFO:tasks.ceph_fuse:Mounting ceph-fuse clients... 2023-05-22T15:27:17.960 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-05-22T15:27:18.090 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:18.083+0000 7fba3d4bc640 1 Processor -- start 2023-05-22T15:27:18.091 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:18.083+0000 7fba3d4bc640 1 -- start start 2023-05-22T15:27:18.091 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:18.083+0000 7fba3d4bc640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fba380c9b40 0x7fba3815e460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:18.092 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:18.083+0000 7fba3d4bc640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fba38070df0 0x7fba38071210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:18.092 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:18.083+0000 7fba3d4bc640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba380b0650 con 0x7fba380ca4e0 2023-05-22T15:27:18.092 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:18.083+0000 7fba3d4bc640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba380b0370 con 0x7fba38070df0 2023-05-22T15:27:18.092 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:18.083+0000 7fba3d4bc640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba3815d980 con 0x7fba380c9b40 2023-05-22T15:27:18.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:18.083+0000 7fba377fe640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fba38070df0 0x7fba38071210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:18.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:18.083+0000 7fba36ffd640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7fba380ca4e0 0x7fba380afb40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.16:34986/0 (socket says 172.21.15.16:34986) 2023-05-22T15:27:18.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:18.083+0000 7fba36ffd640 1 -- 172.21.15.16:0/3674192772 learned_addr learned my addr 172.21.15.16:0/3674192772 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:27:18.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:18.083+0000 7fba367fc640 1 --2- 172.21.15.16:0/3674192772 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fba380c9b40 0x7fba3815e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:18.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:18.083+0000 7fba367fc640 1 -- 172.21.15.16:0/3674192772 >> v2:172.21.15.62:3300/0 conn(0x7fba38070df0 msgr2=0x7fba38071210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:18.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:18.083+0000 7fba367fc640 1 --2- 172.21.15.16:0/3674192772 >> v2:172.21.15.62:3300/0 conn(0x7fba38070df0 0x7fba38071210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:18.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:18.083+0000 7fba367fc640 1 -- 172.21.15.16:0/3674192772 >> v1:172.21.15.62:6789/0 conn(0x7fba380ca4e0 legacy=0x7fba380afb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:18.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:18.083+0000 7fba367fc640 1 -- 172.21.15.16:0/3674192772 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba3815dc60 con 0x7fba380c9b40 2023-05-22T15:27:18.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:18.087+0000 7fba367fc640 1 --2- 172.21.15.16:0/3674192772 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fba380c9b40 0x7fba3815e460 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fba200097c0 tx=0x7fba2001c3c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a26dd5b0a81bca52 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:18.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:18.087+0000 7fba35ffb640 1 -- 172.21.15.16:0/3674192772 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba200ab600 con 0x7fba380c9b40 2023-05-22T15:27:18.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:18.087+0000 7fba35ffb640 1 -- 172.21.15.16:0/3674192772 <== mon.2 v2:172.21.15.62:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba200a7050 con 0x7fba380c9b40 2023-05-22T15:27:18.096 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:18.087+0000 7fba35ffb640 1 -- 172.21.15.16:0/3674192772 <== mon.2 v2:172.21.15.62:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba200ab980 con 0x7fba380c9b40 2023-05-22T15:27:18.096 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:18.087+0000 7fba3d4bc640 1 -- 172.21.15.16:0/3674192772 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fba380c9b40 msgr2=0x7fba3815e460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:18.096 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:18.087+0000 7fba3d4bc640 1 --2- 172.21.15.16:0/3674192772 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fba380c9b40 0x7fba3815e460 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fba200097c0 tx=0x7fba2001c3c0 comp rx=0 tx=0).stop 2023-05-22T15:27:18.097 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:18.087+0000 7fba3d4bc640 1 -- 172.21.15.16:0/3674192772 shutdown_connections 2023-05-22T15:27:18.097 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:18.087+0000 7fba3d4bc640 1 --2- 172.21.15.16:0/3674192772 >> v2:172.21.15.62:3300/0 conn(0x7fba38070df0 0x7fba38071210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:18.097 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:18.087+0000 7fba3d4bc640 1 --2- 172.21.15.16:0/3674192772 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fba380c9b40 0x7fba3815e460 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:18.097 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:18.087+0000 7fba3d4bc640 1 -- 172.21.15.16:0/3674192772 >> 172.21.15.16:0/3674192772 conn(0x7fba38069720 msgr2=0x7fba38069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:18.097 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:18.087+0000 7fba3d4bc640 1 -- 172.21.15.16:0/3674192772 shutdown_connections 2023-05-22T15:27:18.097 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:18.087+0000 7fba3d4bc640 1 -- 172.21.15.16:0/3674192772 wait complete. 2023-05-22T15:27:18.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:18.091+0000 7fba3d4bc640 1 Processor -- start 2023-05-22T15:27:18.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:18.091+0000 7fba3d4bc640 1 -- start start 2023-05-22T15:27:18.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:18.091+0000 7fba3d4bc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fba38070df0 0x7fba3814c570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:18.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:18.091+0000 7fba3d4bc640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fba380c9b40 0x7fba38158740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:18.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:18.091+0000 7fba3d4bc640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fba380ca4e0 0x7fba3806b730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:18.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:18.091+0000 7fba3d4bc640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba3814cab0 con 0x7fba38070df0 2023-05-22T15:27:18.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:18.091+0000 7fba3d4bc640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba38156ef0 con 0x7fba380c9b40 2023-05-22T15:27:18.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:18.091+0000 7fba3d4bc640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba38159120 con 0x7fba380ca4e0 2023-05-22T15:27:18.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:18.091+0000 7fba36ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fba38070df0 0x7fba3814c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:18.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:18.091+0000 7fba36ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fba38070df0 0x7fba3814c570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38182/0 (socket says 172.21.15.16:38182) 2023-05-22T15:27:18.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:18.091+0000 7fba36ffd640 1 -- 172.21.15.16:0/438166347 learned_addr learned my addr 172.21.15.16:0/438166347 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:18.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:18.091+0000 7fba377fe640 1 --2- 172.21.15.16:0/438166347 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fba380ca4e0 0x7fba3806b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:18.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:18.091+0000 7fba367fc640 1 --2- 172.21.15.16:0/438166347 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fba380c9b40 0x7fba38158740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:18.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:18.091+0000 7fba36ffd640 1 -- 172.21.15.16:0/438166347 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fba380ca4e0 msgr2=0x7fba3806b730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:18.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:18.091+0000 7fba36ffd640 1 --2- 172.21.15.16:0/438166347 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fba380ca4e0 0x7fba3806b730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:18.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:18.091+0000 7fba36ffd640 1 -- 172.21.15.16:0/438166347 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fba380c9b40 msgr2=0x7fba38158740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:18.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:18.091+0000 7fba36ffd640 1 --2- 172.21.15.16:0/438166347 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fba380c9b40 0x7fba38158740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:18.103 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:18.091+0000 7fba36ffd640 1 -- 172.21.15.16:0/438166347 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba20009420 con 0x7fba38070df0 2023-05-22T15:27:18.103 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:18.091+0000 7fba36ffd640 1 --2- 172.21.15.16:0/438166347 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fba38070df0 0x7fba3814c570 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fba2800c370 tx=0x7fba2800c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:18.103 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:18.091+0000 7fba2f7fe640 1 -- 172.21.15.16:0/438166347 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba28005cf0 con 0x7fba38070df0 2023-05-22T15:27:18.103 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:18.091+0000 7fba2f7fe640 1 -- 172.21.15.16:0/438166347 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba28005e50 con 0x7fba38070df0 2023-05-22T15:27:18.103 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:18.091+0000 7fba2f7fe640 1 -- 172.21.15.16:0/438166347 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba280072f0 con 0x7fba38070df0 2023-05-22T15:27:18.103 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:18.091+0000 7fba3d4bc640 1 -- 172.21.15.16:0/438166347 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba380c8e80 con 0x7fba38070df0 2023-05-22T15:27:18.103 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:18.091+0000 7fba3d4bc640 1 -- 172.21.15.16:0/438166347 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba380ca8b0 con 0x7fba38070df0 2023-05-22T15:27:18.104 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:18.091+0000 7fba3d4bc640 1 -- 172.21.15.16:0/438166347 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba380c9010 con 0x7fba38070df0 2023-05-22T15:27:18.104 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:18.095+0000 7fba2f7fe640 1 -- 172.21.15.16:0/438166347 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7fba28007450 con 0x7fba38070df0 2023-05-22T15:27:18.105 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:18.095+0000 7fba2f7fe640 1 --2- 172.21.15.16:0/438166347 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fba18065380 0x7fba18067830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:18.105 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:18.095+0000 7fba367fc640 1 --2- 172.21.15.16:0/438166347 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fba18065380 0x7fba18067830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:18.105 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:18.095+0000 7fba2f7fe640 1 -- 172.21.15.16:0/438166347 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6813+0+0 (secure 0 0 0) 0x7fba28082020 con 0x7fba38070df0 2023-05-22T15:27:18.105 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:18.095+0000 7fba367fc640 1 --2- 172.21.15.16:0/438166347 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fba18065380 0x7fba18067830 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fba200ad000 tx=0x7fba200a6000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:18.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:18.099+0000 7fba2f7fe640 1 -- 172.21.15.16:0/438166347 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fba2804fa40 con 0x7fba38070df0 2023-05-22T15:27:18.290 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:18.283+0000 7fba3d4bc640 1 -- 172.21.15.16:0/438166347 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fba3804dbe0 con 0x7fba38070df0 2023-05-22T15:27:18.290 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:18.283+0000 7fba2f7fe640 1 -- 172.21.15.16:0/438166347 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) v1 ==== 53+0+88 (secure 0 0 0) 0x7fba28017070 con 0x7fba38070df0 2023-05-22T15:27:18.291 INFO:teuthology.orchestra.run.smithi016.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2023-05-22T15:27:18.292 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:18.283+0000 7fba3d4bc640 1 -- 172.21.15.16:0/438166347 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fba18065380 msgr2=0x7fba18067830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:18.293 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:18.283+0000 7fba3d4bc640 1 --2- 172.21.15.16:0/438166347 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fba18065380 0x7fba18067830 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fba200ad000 tx=0x7fba200a6000 comp rx=0 tx=0).stop 2023-05-22T15:27:18.293 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:18.283+0000 7fba3d4bc640 1 -- 172.21.15.16:0/438166347 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fba38070df0 msgr2=0x7fba3814c570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:18.293 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:18.283+0000 7fba3d4bc640 1 --2- 172.21.15.16:0/438166347 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fba38070df0 0x7fba3814c570 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fba2800c370 tx=0x7fba2800c830 comp rx=0 tx=0).stop 2023-05-22T15:27:18.293 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:18.283+0000 7fba3d4bc640 1 -- 172.21.15.16:0/438166347 shutdown_connections 2023-05-22T15:27:18.293 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:18.283+0000 7fba3d4bc640 1 --2- 172.21.15.16:0/438166347 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fba18065380 0x7fba18067830 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:18.293 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:18.283+0000 7fba3d4bc640 1 --2- 172.21.15.16:0/438166347 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fba380ca4e0 0x7fba3806b730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:18.293 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:18.283+0000 7fba3d4bc640 1 --2- 172.21.15.16:0/438166347 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fba380c9b40 0x7fba38158740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:18.294 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:18.283+0000 7fba3d4bc640 1 --2- 172.21.15.16:0/438166347 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fba38070df0 0x7fba3814c570 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:18.294 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:18.283+0000 7fba3d4bc640 1 -- 172.21.15.16:0/438166347 >> 172.21.15.16:0/438166347 conn(0x7fba38069720 msgr2=0x7fba381562b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:18.294 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:18.283+0000 7fba3d4bc640 1 -- 172.21.15.16:0/438166347 shutdown_connections 2023-05-22T15:27:18.294 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:18.283+0000 7fba3d4bc640 1 -- 172.21.15.16:0/438166347 wait complete. 2023-05-22T15:27:18.308 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-05-22T15:27:18.308 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-05-22T15:27:18.308 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi016.front.sepia.ceph.com 2023-05-22T15:27:18.308 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-05-22T15:27:18.309 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-05-22T15:27:18.309 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-05-22T15:27:18.309 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-05-22T15:27:18.309 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-05-22T15:27:18.309 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2023-05-22T15:27:18.310 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:27:18.310 DEBUG:teuthology.orchestra.run.smithi016:> ip addr 2023-05-22T15:27:18.316 INFO:teuthology.orchestra.run.smithi016.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-05-22T15:27:18.316 INFO:teuthology.orchestra.run.smithi016.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-05-22T15:27:18.316 INFO:teuthology.orchestra.run.smithi016.stdout: inet 127.0.0.1/8 scope host lo 2023-05-22T15:27:18.316 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-05-22T15:27:18.316 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 ::1/128 scope host 2023-05-22T15:27:18.317 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-05-22T15:27:18.317 INFO:teuthology.orchestra.run.smithi016.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-05-22T15:27:18.317 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:6b:fd:e0 brd ff:ff:ff:ff:ff:ff 2023-05-22T15:27:18.317 INFO:teuthology.orchestra.run.smithi016.stdout: altname enp7s0f0 2023-05-22T15:27:18.317 INFO:teuthology.orchestra.run.smithi016.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-05-22T15:27:18.317 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:6b:fd:e1 brd ff:ff:ff:ff:ff:ff 2023-05-22T15:27:18.317 INFO:teuthology.orchestra.run.smithi016.stdout: altname enp7s0f1 2023-05-22T15:27:18.318 INFO:teuthology.orchestra.run.smithi016.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2023-05-22T15:27:18.318 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:bd:15:64 brd ff:ff:ff:ff:ff:ff 2023-05-22T15:27:18.318 INFO:teuthology.orchestra.run.smithi016.stdout: altname enp1s0f0 2023-05-22T15:27:18.318 INFO:teuthology.orchestra.run.smithi016.stdout: inet 172.21.15.16/20 brd 172.21.15.255 scope global dynamic ens1f0 2023-05-22T15:27:18.318 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft 42478sec preferred_lft 42478sec 2023-05-22T15:27:18.318 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::ec4:7aff:febd:1564/64 scope link 2023-05-22T15:27:18.318 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-05-22T15:27:18.318 INFO:teuthology.orchestra.run.smithi016.stdout:5: ens1f1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-05-22T15:27:18.319 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:bd:15:65 brd ff:ff:ff:ff:ff:ff 2023-05-22T15:27:18.319 INFO:teuthology.orchestra.run.smithi016.stdout: altname enp1s0f1 2023-05-22T15:27:18.319 INFO:teuthology.orchestra.run.smithi016.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2023-05-22T15:27:18.319 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 02:42:1f:b8:98:0a brd ff:ff:ff:ff:ff:ff 2023-05-22T15:27:18.319 INFO:teuthology.orchestra.run.smithi016.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2023-05-22T15:27:18.319 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-05-22T15:27:18.319 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2023-05-22T15:27:18.320 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:27:18.320 DEBUG:teuthology.orchestra.run.smithi016:> (cd / && exec bash -c ' 2023-05-22T15:27:18.320 DEBUG:teuthology.orchestra.run.smithi016:> set -e 2023-05-22T15:27:18.321 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip link add name ceph-brx type bridge 2023-05-22T15:27:18.321 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip addr flush dev ceph-brx 2023-05-22T15:27:18.321 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip link set ceph-brx up 2023-05-22T15:27:18.321 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2023-05-22T15:27:18.322 DEBUG:teuthology.orchestra.run.smithi016:> ') 2023-05-22T15:27:18.395 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:27:18.395 DEBUG:teuthology.orchestra.run.smithi016:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-05-22T15:27:18.450 INFO:teuthology.orchestra.run.smithi016.stdout:1 2023-05-22T15:27:18.452 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:27:18.452 DEBUG:teuthology.orchestra.run.smithi016:> route 2023-05-22T15:27:18.498 INFO:teuthology.orchestra.run.smithi016.stdout:Kernel IP routing table 2023-05-22T15:27:18.499 INFO:teuthology.orchestra.run.smithi016.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-05-22T15:27:18.499 INFO:teuthology.orchestra.run.smithi016.stdout:default _gateway 0.0.0.0 UG 0 0 0 ens1f0 2023-05-22T15:27:18.499 INFO:teuthology.orchestra.run.smithi016.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2023-05-22T15:27:18.499 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 0 0 0 ens1f0 2023-05-22T15:27:18.499 INFO:teuthology.orchestra.run.smithi016.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-05-22T15:27:18.500 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:27:18.500 DEBUG:teuthology.orchestra.run.smithi016:> (cd / && exec bash -c ' 2023-05-22T15:27:18.501 DEBUG:teuthology.orchestra.run.smithi016:> set -e 2023-05-22T15:27:18.501 DEBUG:teuthology.orchestra.run.smithi016:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2023-05-22T15:27:18.501 DEBUG:teuthology.orchestra.run.smithi016:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2023-05-22T15:27:18.502 DEBUG:teuthology.orchestra.run.smithi016:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2023-05-22T15:27:18.503 DEBUG:teuthology.orchestra.run.smithi016:> ') 2023-05-22T15:27:18.574 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:27:18.574 DEBUG:teuthology.orchestra.run.smithi016:> ip netns list 2023-05-22T15:27:18.622 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:27:18.623 DEBUG:teuthology.orchestra.run.smithi016:> ip netns list-id 2023-05-22T15:27:18.670 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:27:18.671 DEBUG:teuthology.orchestra.run.smithi016:> (cd / && exec bash -c ' 2023-05-22T15:27:18.671 DEBUG:teuthology.orchestra.run.smithi016:> set -e 2023-05-22T15:27:18.672 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2023-05-22T15:27:18.672 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2023-05-22T15:27:18.672 DEBUG:teuthology.orchestra.run.smithi016:> ') 2023-05-22T15:27:18.740 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2023-05-22T15:27:18.741 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:27:18.742 DEBUG:teuthology.orchestra.run.smithi016:> (cd / && exec bash -c ' 2023-05-22T15:27:18.742 DEBUG:teuthology.orchestra.run.smithi016:> set -e 2023-05-22T15:27:18.743 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2023-05-22T15:27:18.743 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2023-05-22T15:27:18.744 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2023-05-22T15:27:18.744 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2023-05-22T15:27:18.744 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2023-05-22T15:27:18.744 DEBUG:teuthology.orchestra.run.smithi016:> ') 2023-05-22T15:27:18.832 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:27:18.832 DEBUG:teuthology.orchestra.run.smithi016:> (cd / && exec bash -c ' 2023-05-22T15:27:18.833 DEBUG:teuthology.orchestra.run.smithi016:> set -e 2023-05-22T15:27:18.833 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip link set brx.0 up 2023-05-22T15:27:18.834 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip link set dev brx.0 master ceph-brx 2023-05-22T15:27:18.834 DEBUG:teuthology.orchestra.run.smithi016:> ') 2023-05-22T15:27:18.854 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2023-05-22T15:27:18.854 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-05-22T15:27:18.855 DEBUG:teuthology.orchestra.run.smithi016:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-05-22T15:27:18.901 INFO:teuthology.orchestra.run.smithi016.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-05-22T15:27:18.902 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-05-22T15:27:18.902 DEBUG:teuthology.orchestra.run.smithi016:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-05-22T15:27:18.950 DEBUG:teuthology.orchestra.run.smithi016:> sudo modprobe fuse 2023-05-22T15:27:19.012 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-05-22T15:27:19.012 DEBUG:teuthology.orchestra.run.smithi016:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-05-22T15:27:19.066 INFO:teuthology.orchestra.run.smithi016.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-05-22T15:27:19.067 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-05-22T15:27:19.067 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:27:19.068 DEBUG:teuthology.orchestra.run.smithi016:> ls /sys/fs/fuse/connections 2023-05-22T15:27:19.115 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2023-05-22T15:27:19.115 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2023-05-22T15:27:19.162 DEBUG:teuthology.orchestra.run.smithi016:> sudo modprobe fuse 2023-05-22T15:27:19.178 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-05-22T15:27:19.178 DEBUG:teuthology.orchestra.run.smithi016:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-05-22T15:27:19.232 INFO:teuthology.orchestra.run.smithi016.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-05-22T15:27:19.233 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-05-22T15:27:19.234 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:27:19.235 DEBUG:teuthology.orchestra.run.smithi016:> ls /sys/fs/fuse/connections 2023-05-22T15:27:19.240 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi016.stderr:2023-05-22T15:27:19.227+0000 7f8ba1acc580 -1 init, newargv = 0x5581a0338bd0 newargc=15 2023-05-22T15:27:19.241 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi016.stderr:ceph-fuse[45394]: starting ceph client 2023-05-22T15:27:19.245 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi016.stderr:ceph-fuse[45394]: starting fuse 2023-05-22T15:27:20.283 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-05-22T15:27:20.286 DEBUG:teuthology.orchestra.run.smithi062:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-05-22T15:27:20.347 DEBUG:teuthology.orchestra.run.smithi016:> sudo modprobe fuse 2023-05-22T15:27:20.363 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-05-22T15:27:20.363 DEBUG:teuthology.orchestra.run.smithi016:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-05-22T15:27:20.422 INFO:teuthology.orchestra.run.smithi016.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-05-22T15:27:20.423 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-05-22T15:27:20.423 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:27:20.424 DEBUG:teuthology.orchestra.run.smithi016:> ls /sys/fs/fuse/connections 2023-05-22T15:27:20.474 INFO:teuthology.orchestra.run.smithi016.stdout:45 2023-05-22T15:27:20.474 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [45] 2023-05-22T15:27:20.475 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2023-05-22T15:27:20.475 DEBUG:teuthology.orchestra.run.smithi016:> import glob 2023-05-22T15:27:20.475 DEBUG:teuthology.orchestra.run.smithi016:> import re 2023-05-22T15:27:20.475 DEBUG:teuthology.orchestra.run.smithi016:> import os 2023-05-22T15:27:20.476 DEBUG:teuthology.orchestra.run.smithi016:> import subprocess 2023-05-22T15:27:20.476 DEBUG:teuthology.orchestra.run.smithi016:> 2023-05-22T15:27:20.476 DEBUG:teuthology.orchestra.run.smithi016:> def _find_admin_socket(client_name): 2023-05-22T15:27:20.476 DEBUG:teuthology.orchestra.run.smithi016:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2023-05-22T15:27:20.477 DEBUG:teuthology.orchestra.run.smithi016:> files = glob.glob(asok_path) 2023-05-22T15:27:20.477 DEBUG:teuthology.orchestra.run.smithi016:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2023-05-22T15:27:20.477 DEBUG:teuthology.orchestra.run.smithi016:> 2023-05-22T15:27:20.477 DEBUG:teuthology.orchestra.run.smithi016:> # Given a non-glob path, it better be there 2023-05-22T15:27:20.477 DEBUG:teuthology.orchestra.run.smithi016:> if "*" not in asok_path: 2023-05-22T15:27:20.478 DEBUG:teuthology.orchestra.run.smithi016:> assert(len(files) == 1) 2023-05-22T15:27:20.478 DEBUG:teuthology.orchestra.run.smithi016:> return files[0] 2023-05-22T15:27:20.478 DEBUG:teuthology.orchestra.run.smithi016:> 2023-05-22T15:27:20.478 DEBUG:teuthology.orchestra.run.smithi016:> for f in files: 2023-05-22T15:27:20.479 DEBUG:teuthology.orchestra.run.smithi016:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-05-22T15:27:20.479 DEBUG:teuthology.orchestra.run.smithi016:> if os.path.exists("/proc/{0}".format(pid)): 2023-05-22T15:27:20.479 DEBUG:teuthology.orchestra.run.smithi016:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-05-22T15:27:20.479 DEBUG:teuthology.orchestra.run.smithi016:> contents = proc_f.read() 2023-05-22T15:27:20.480 DEBUG:teuthology.orchestra.run.smithi016:> if mountpoint in contents: 2023-05-22T15:27:20.480 DEBUG:teuthology.orchestra.run.smithi016:> return f 2023-05-22T15:27:20.480 DEBUG:teuthology.orchestra.run.smithi016:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-05-22T15:27:20.480 DEBUG:teuthology.orchestra.run.smithi016:> 2023-05-22T15:27:20.481 DEBUG:teuthology.orchestra.run.smithi016:> print(_find_admin_socket("client.0")) 2023-05-22T15:27:20.481 DEBUG:teuthology.orchestra.run.smithi016:> ' 2023-05-22T15:27:20.600 INFO:teuthology.orchestra.run.smithi016.stdout:/var/run/ceph/ceph-client.0.45394.asok 2023-05-22T15:27:20.784 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.45394.asok 2023-05-22T15:27:20.785 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:27:20.785 DEBUG:teuthology.orchestra.run.smithi016:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.45394.asok status 2023-05-22T15:27:20.945 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-05-22T15:27:20.946 INFO:teuthology.orchestra.run.smithi016.stdout: "metadata": { 2023-05-22T15:27:20.946 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph_sha1": "9c9ba4df22024d9d8d6cf9f98a4b96751fc78d0c", 2023-05-22T15:27:20.946 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph_version": "ceph version 18.0.0-3192-g9c9ba4df (9c9ba4df22024d9d8d6cf9f98a4b96751fc78d0c) reef (dev)", 2023-05-22T15:27:20.946 INFO:teuthology.orchestra.run.smithi016.stdout: "entity_id": "0", 2023-05-22T15:27:20.946 INFO:teuthology.orchestra.run.smithi016.stdout: "hostname": "smithi016", 2023-05-22T15:27:20.947 INFO:teuthology.orchestra.run.smithi016.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2023-05-22T15:27:20.947 INFO:teuthology.orchestra.run.smithi016.stdout: "pid": "45394", 2023-05-22T15:27:20.947 INFO:teuthology.orchestra.run.smithi016.stdout: "root": "/" 2023-05-22T15:27:20.947 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:27:20.947 INFO:teuthology.orchestra.run.smithi016.stdout: "dentry_count": 0, 2023-05-22T15:27:20.947 INFO:teuthology.orchestra.run.smithi016.stdout: "dentry_pinned_count": 0, 2023-05-22T15:27:20.947 INFO:teuthology.orchestra.run.smithi016.stdout: "id": 4860, 2023-05-22T15:27:20.947 INFO:teuthology.orchestra.run.smithi016.stdout: "inst": { 2023-05-22T15:27:20.949 INFO:teuthology.orchestra.run.smithi016.stdout: "name": { 2023-05-22T15:27:20.949 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "client", 2023-05-22T15:27:20.949 INFO:teuthology.orchestra.run.smithi016.stdout: "num": 4860 2023-05-22T15:27:20.950 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:27:20.950 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": { 2023-05-22T15:27:20.950 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-05-22T15:27:20.950 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "192.168.0.1:0", 2023-05-22T15:27:20.950 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 890039220 2023-05-22T15:27:20.950 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T15:27:20.950 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:27:20.950 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": { 2023-05-22T15:27:20.951 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-05-22T15:27:20.951 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "192.168.0.1:0", 2023-05-22T15:27:20.951 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 890039220 2023-05-22T15:27:20.951 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:27:20.951 INFO:teuthology.orchestra.run.smithi016.stdout: "inst_str": "client.4860 192.168.0.1:0/890039220", 2023-05-22T15:27:20.951 INFO:teuthology.orchestra.run.smithi016.stdout: "addr_str": "192.168.0.1:0/890039220", 2023-05-22T15:27:20.951 INFO:teuthology.orchestra.run.smithi016.stdout: "inode_count": 1, 2023-05-22T15:27:20.952 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_epoch": 8, 2023-05-22T15:27:20.952 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_epoch": 27, 2023-05-22T15:27:20.952 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_epoch_barrier": 0, 2023-05-22T15:27:20.952 INFO:teuthology.orchestra.run.smithi016.stdout: "blocklisted": false, 2023-05-22T15:27:20.952 INFO:teuthology.orchestra.run.smithi016.stdout: "fs_name": "cephfs" 2023-05-22T15:27:20.952 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-05-22T15:27:20.960 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-05-22T15:27:21.082 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:21.075+0000 7f65c1fbe640 1 Processor -- start 2023-05-22T15:27:21.082 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:21.075+0000 7f65c1fbe640 1 -- start start 2023-05-22T15:27:21.083 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:21.075+0000 7f65c1fbe640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f65bc0b7dd0 0x7f65bc0ccab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:21.083 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:21.075+0000 7f65c1fbe640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f65bc160220 0x7f65bc0d0410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:21.084 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:21.075+0000 7f65c1fbe640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65bc0cd170 con 0x7f65bc15f700 2023-05-22T15:27:21.084 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:21.075+0000 7f65c1fbe640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65bc0d3360 con 0x7f65bc160220 2023-05-22T15:27:21.084 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:21.075+0000 7f65c1fbe640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65bc0d2ac0 con 0x7f65bc0b7dd0 2023-05-22T15:27:21.084 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:21.075+0000 7f65bbfff640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f65bc160220 0x7f65bc0d0410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:21.084 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:21.075+0000 7f65bbfff640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f65bc160220 0x7f65bc0d0410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56524/0 (socket says 172.21.15.16:56524) 2023-05-22T15:27:21.084 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:21.075+0000 7f65bbfff640 1 -- 172.21.15.16:0/3044841996 learned_addr learned my addr 172.21.15.16:0/3044841996 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:21.085 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:21.075+0000 7f65bb7fe640 1 --2- 172.21.15.16:0/3044841996 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f65bc0b7dd0 0x7f65bc0ccab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:21.085 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:21.075+0000 7f65bbfff640 1 -- 172.21.15.16:0/3044841996 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f65bc0b7dd0 msgr2=0x7f65bc0ccab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:21.085 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:21.075+0000 7f65bbfff640 1 --2- 172.21.15.16:0/3044841996 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f65bc0b7dd0 0x7f65bc0ccab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:21.085 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:21.075+0000 7f65bbfff640 1 -- 172.21.15.16:0/3044841996 >> v1:172.21.15.62:6789/0 conn(0x7f65bc15f700 legacy=0x7f65bc15fb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:21.086 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:21.075+0000 7f65bbfff640 1 -- 172.21.15.16:0/3044841996 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65bc0d2da0 con 0x7f65bc160220 2023-05-22T15:27:21.086 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:21.079+0000 7f65bbfff640 1 --2- 172.21.15.16:0/3044841996 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f65bc160220 0x7f65bc0d0410 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f65b4009e50 tx=0x7f65b401c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb15b2b02eb9cc35 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:21.087 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:21.079+0000 7f65baffd640 1 -- 172.21.15.16:0/3044841996 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65b40a57d0 con 0x7f65bc160220 2023-05-22T15:27:21.087 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:21.079+0000 7f65baffd640 1 -- 172.21.15.16:0/3044841996 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65b40a5930 con 0x7f65bc160220 2023-05-22T15:27:21.087 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:21.079+0000 7f65baffd640 1 -- 172.21.15.16:0/3044841996 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65b40a5cc0 con 0x7f65bc160220 2023-05-22T15:27:21.087 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:21.079+0000 7f65c1fbe640 1 -- 172.21.15.16:0/3044841996 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f65bc160220 msgr2=0x7f65bc0d0410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:21.087 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:21.079+0000 7f65c1fbe640 1 --2- 172.21.15.16:0/3044841996 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f65bc160220 0x7f65bc0d0410 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f65b4009e50 tx=0x7f65b401c950 comp rx=0 tx=0).stop 2023-05-22T15:27:21.087 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:21.079+0000 7f65c1fbe640 1 -- 172.21.15.16:0/3044841996 shutdown_connections 2023-05-22T15:27:21.088 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:21.079+0000 7f65c1fbe640 1 --2- 172.21.15.16:0/3044841996 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f65bc160220 0x7f65bc0d0410 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:21.088 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:21.079+0000 7f65c1fbe640 1 --2- 172.21.15.16:0/3044841996 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f65bc0b7dd0 0x7f65bc0ccab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:21.088 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:21.079+0000 7f65c1fbe640 1 -- 172.21.15.16:0/3044841996 >> 172.21.15.16:0/3044841996 conn(0x7f65bc0655c0 msgr2=0x7f65bc070e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:21.088 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:21.079+0000 7f65c1fbe640 1 -- 172.21.15.16:0/3044841996 shutdown_connections 2023-05-22T15:27:21.088 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:21.079+0000 7f65c1fbe640 1 -- 172.21.15.16:0/3044841996 wait complete. 2023-05-22T15:27:21.090 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:21.079+0000 7f65c1fbe640 1 Processor -- start 2023-05-22T15:27:21.090 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:21.079+0000 7f65c1fbe640 1 -- start start 2023-05-22T15:27:21.090 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:21.079+0000 7f65c1fbe640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f65bc0b7dd0 0x7f65bc156db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:21.090 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:21.079+0000 7f65c1fbe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f65bc15f700 0x7f65bc141f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:21.090 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:21.079+0000 7f65c1fbe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f65bc160220 0x7f65bc1424c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:21.090 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:21.079+0000 7f65c1fbe640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65bc15cd60 con 0x7f65bc160220 2023-05-22T15:27:21.091 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:21.079+0000 7f65c1fbe640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65bc15d040 con 0x7f65bc15f700 2023-05-22T15:27:21.091 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:21.079+0000 7f65c1fbe640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65bc15d320 con 0x7f65bc0b7dd0 2023-05-22T15:27:21.091 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:21.079+0000 7f65bbfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f65bc160220 0x7f65bc1424c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:21.091 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:21.079+0000 7f65bbfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f65bc160220 0x7f65bc1424c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56530/0 (socket says 172.21.15.16:56530) 2023-05-22T15:27:21.091 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:21.079+0000 7f65bbfff640 1 -- 172.21.15.16:0/4003972217 learned_addr learned my addr 172.21.15.16:0/4003972217 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:21.091 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:21.079+0000 7f65bb7fe640 1 --2- 172.21.15.16:0/4003972217 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f65bc0b7dd0 0x7f65bc156db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:21.091 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:21.079+0000 7f65b3fff640 1 --2- 172.21.15.16:0/4003972217 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f65bc15f700 0x7f65bc141f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:21.092 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:21.079+0000 7f65bbfff640 1 -- 172.21.15.16:0/4003972217 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f65bc0b7dd0 msgr2=0x7f65bc156db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:21.092 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:21.079+0000 7f65bbfff640 1 --2- 172.21.15.16:0/4003972217 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f65bc0b7dd0 0x7f65bc156db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:21.092 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:21.079+0000 7f65bbfff640 1 -- 172.21.15.16:0/4003972217 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f65bc15f700 msgr2=0x7f65bc141f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:21.092 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:21.079+0000 7f65bbfff640 1 --2- 172.21.15.16:0/4003972217 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f65bc15f700 0x7f65bc141f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:21.092 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:21.079+0000 7f65bbfff640 1 -- 172.21.15.16:0/4003972217 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65b4009b30 con 0x7f65bc160220 2023-05-22T15:27:21.092 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:21.083+0000 7f65bbfff640 1 --2- 172.21.15.16:0/4003972217 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f65bc160220 0x7f65bc1424c0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f65b401a580 tx=0x7f65b40196f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:21.092 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:21.083+0000 7f65b8ff9640 1 -- 172.21.15.16:0/4003972217 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65b40b8670 con 0x7f65bc160220 2023-05-22T15:27:21.092 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:21.083+0000 7f65b8ff9640 1 -- 172.21.15.16:0/4003972217 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65b40b87d0 con 0x7f65bc160220 2023-05-22T15:27:21.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:21.083+0000 7f65c1fbe640 1 -- 172.21.15.16:0/4003972217 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65bc0d12f0 con 0x7f65bc160220 2023-05-22T15:27:21.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:21.083+0000 7f65b8ff9640 1 -- 172.21.15.16:0/4003972217 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65b40b8b60 con 0x7f65bc160220 2023-05-22T15:27:21.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:21.083+0000 7f65c1fbe640 1 -- 172.21.15.16:0/4003972217 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65bc0d1160 con 0x7f65bc160220 2023-05-22T15:27:21.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:21.083+0000 7f65c1fbe640 1 -- 172.21.15.16:0/4003972217 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65bc0d1480 con 0x7f65bc160220 2023-05-22T15:27:21.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:21.083+0000 7f65b8ff9640 1 -- 172.21.15.16:0/4003972217 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7f65b40b8cc0 con 0x7f65bc160220 2023-05-22T15:27:21.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:21.083+0000 7f65b8ff9640 1 --2- 172.21.15.16:0/4003972217 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f659c065380 0x7f659c067830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:21.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:21.083+0000 7f65b8ff9640 1 -- 172.21.15.16:0/4003972217 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f65b411e210 con 0x7f65bc160220 2023-05-22T15:27:21.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:21.083+0000 7f65bb7fe640 1 --2- 172.21.15.16:0/4003972217 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f659c065380 0x7f659c067830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:21.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:21.083+0000 7f65bb7fe640 1 --2- 172.21.15.16:0/4003972217 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f659c065380 0x7f659c067830 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f65ac00b930 tx=0x7f65ac008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:21.096 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:21.087+0000 7f65b8ff9640 1 -- 172.21.15.16:0/4003972217 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f65b40ebbc0 con 0x7f65bc160220 2023-05-22T15:27:21.293 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:21.283+0000 7f65c1fbe640 1 -- 172.21.15.16:0/4003972217 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f65bc04dbe0 con 0x7f65bc160220 2023-05-22T15:27:21.294 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:21.283+0000 7f65b8ff9640 1 -- 172.21.15.16:0/4003972217 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) v1 ==== 53+0+88 (secure 0 0 0) 0x7f65b40ef1e0 con 0x7f65bc160220 2023-05-22T15:27:21.294 INFO:teuthology.orchestra.run.smithi016.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2023-05-22T15:27:21.296 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:21.287+0000 7f65c1fbe640 1 -- 172.21.15.16:0/4003972217 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f659c065380 msgr2=0x7f659c067830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:21.296 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:21.287+0000 7f65c1fbe640 1 --2- 172.21.15.16:0/4003972217 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f659c065380 0x7f659c067830 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f65ac00b930 tx=0x7f65ac008040 comp rx=0 tx=0).stop 2023-05-22T15:27:21.296 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:21.287+0000 7f65c1fbe640 1 -- 172.21.15.16:0/4003972217 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f65bc160220 msgr2=0x7f65bc1424c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:21.296 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:21.287+0000 7f65c1fbe640 1 --2- 172.21.15.16:0/4003972217 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f65bc160220 0x7f65bc1424c0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f65b401a580 tx=0x7f65b40196f0 comp rx=0 tx=0).stop 2023-05-22T15:27:21.297 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:21.287+0000 7f65c1fbe640 1 -- 172.21.15.16:0/4003972217 shutdown_connections 2023-05-22T15:27:21.297 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:21.287+0000 7f65c1fbe640 1 --2- 172.21.15.16:0/4003972217 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f65bc160220 0x7f65bc1424c0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:21.298 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:21.287+0000 7f65c1fbe640 1 --2- 172.21.15.16:0/4003972217 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f65bc15f700 0x7f65bc141f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:21.298 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:21.287+0000 7f65c1fbe640 1 --2- 172.21.15.16:0/4003972217 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f659c065380 0x7f659c067830 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:21.298 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:21.287+0000 7f65c1fbe640 1 --2- 172.21.15.16:0/4003972217 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f65bc0b7dd0 0x7f65bc156db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:21.298 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:21.287+0000 7f65c1fbe640 1 -- 172.21.15.16:0/4003972217 >> 172.21.15.16:0/4003972217 conn(0x7f65bc0655c0 msgr2=0x7f65bc0b16a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:21.298 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:21.287+0000 7f65c1fbe640 1 -- 172.21.15.16:0/4003972217 shutdown_connections 2023-05-22T15:27:21.299 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:21.287+0000 7f65c1fbe640 1 -- 172.21.15.16:0/4003972217 wait complete. 2023-05-22T15:27:21.312 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-05-22T15:27:21.313 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-05-22T15:27:21.313 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi016.front.sepia.ceph.com 2023-05-22T15:27:21.313 INFO:tasks.cephfs.mount:self.client.name = client.2 2023-05-22T15:27:21.313 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2023-05-22T15:27:21.314 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-05-22T15:27:21.314 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-05-22T15:27:21.314 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-05-22T15:27:21.314 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.2' 2023-05-22T15:27:21.315 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:27:21.315 DEBUG:teuthology.orchestra.run.smithi016:> ip addr 2023-05-22T15:27:21.321 INFO:teuthology.orchestra.run.smithi016.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-05-22T15:27:21.321 INFO:teuthology.orchestra.run.smithi016.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-05-22T15:27:21.321 INFO:teuthology.orchestra.run.smithi016.stdout: inet 127.0.0.1/8 scope host lo 2023-05-22T15:27:21.322 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-05-22T15:27:21.322 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 ::1/128 scope host 2023-05-22T15:27:21.322 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-05-22T15:27:21.322 INFO:teuthology.orchestra.run.smithi016.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-05-22T15:27:21.322 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:6b:fd:e0 brd ff:ff:ff:ff:ff:ff 2023-05-22T15:27:21.322 INFO:teuthology.orchestra.run.smithi016.stdout: altname enp7s0f0 2023-05-22T15:27:21.322 INFO:teuthology.orchestra.run.smithi016.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-05-22T15:27:21.322 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:6b:fd:e1 brd ff:ff:ff:ff:ff:ff 2023-05-22T15:27:21.323 INFO:teuthology.orchestra.run.smithi016.stdout: altname enp7s0f1 2023-05-22T15:27:21.323 INFO:teuthology.orchestra.run.smithi016.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2023-05-22T15:27:21.323 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:bd:15:64 brd ff:ff:ff:ff:ff:ff 2023-05-22T15:27:21.323 INFO:teuthology.orchestra.run.smithi016.stdout: altname enp1s0f0 2023-05-22T15:27:21.323 INFO:teuthology.orchestra.run.smithi016.stdout: inet 172.21.15.16/20 brd 172.21.15.255 scope global dynamic ens1f0 2023-05-22T15:27:21.323 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft 42475sec preferred_lft 42475sec 2023-05-22T15:27:21.323 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::ec4:7aff:febd:1564/64 scope link 2023-05-22T15:27:21.324 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-05-22T15:27:21.324 INFO:teuthology.orchestra.run.smithi016.stdout:5: ens1f1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-05-22T15:27:21.324 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:bd:15:65 brd ff:ff:ff:ff:ff:ff 2023-05-22T15:27:21.324 INFO:teuthology.orchestra.run.smithi016.stdout: altname enp1s0f1 2023-05-22T15:27:21.324 INFO:teuthology.orchestra.run.smithi016.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2023-05-22T15:27:21.324 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 02:42:1f:b8:98:0a brd ff:ff:ff:ff:ff:ff 2023-05-22T15:27:21.324 INFO:teuthology.orchestra.run.smithi016.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2023-05-22T15:27:21.324 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-05-22T15:27:21.325 INFO:teuthology.orchestra.run.smithi016.stdout:9: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-05-22T15:27:21.325 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 5e:1f:84:6d:79:ae brd ff:ff:ff:ff:ff:ff 2023-05-22T15:27:21.325 INFO:teuthology.orchestra.run.smithi016.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-05-22T15:27:21.325 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-05-22T15:27:21.325 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::5c1f:84ff:fe6d:79ae/64 scope link 2023-05-22T15:27:21.325 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-05-22T15:27:21.325 INFO:teuthology.orchestra.run.smithi016.stdout:10: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-05-22T15:27:21.326 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0e:5c:f6:1f:82:2f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-05-22T15:27:21.326 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::c5c:f6ff:fe1f:822f/64 scope link 2023-05-22T15:27:21.326 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-05-22T15:27:21.326 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:27:21.326 DEBUG:teuthology.orchestra.run.smithi016:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-05-22T15:27:21.372 INFO:teuthology.orchestra.run.smithi016.stdout:1 2023-05-22T15:27:21.373 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:27:21.374 DEBUG:teuthology.orchestra.run.smithi016:> route 2023-05-22T15:27:21.423 INFO:teuthology.orchestra.run.smithi016.stdout:Kernel IP routing table 2023-05-22T15:27:21.424 INFO:teuthology.orchestra.run.smithi016.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-05-22T15:27:21.424 INFO:teuthology.orchestra.run.smithi016.stdout:default _gateway 0.0.0.0 UG 0 0 0 ens1f0 2023-05-22T15:27:21.424 INFO:teuthology.orchestra.run.smithi016.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2023-05-22T15:27:21.424 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 0 0 0 ens1f0 2023-05-22T15:27:21.424 INFO:teuthology.orchestra.run.smithi016.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-05-22T15:27:21.425 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:27:21.425 DEBUG:teuthology.orchestra.run.smithi016:> (cd / && exec bash -c ' 2023-05-22T15:27:21.425 DEBUG:teuthology.orchestra.run.smithi016:> set -e 2023-05-22T15:27:21.426 DEBUG:teuthology.orchestra.run.smithi016:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2023-05-22T15:27:21.426 DEBUG:teuthology.orchestra.run.smithi016:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2023-05-22T15:27:21.426 DEBUG:teuthology.orchestra.run.smithi016:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2023-05-22T15:27:21.426 DEBUG:teuthology.orchestra.run.smithi016:> ') 2023-05-22T15:27:21.501 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:27:21.502 DEBUG:teuthology.orchestra.run.smithi016:> ip netns list 2023-05-22T15:27:21.551 INFO:teuthology.orchestra.run.smithi016.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-05-22T15:27:21.551 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:27:21.552 DEBUG:teuthology.orchestra.run.smithi016:> ip netns list-id 2023-05-22T15:27:21.598 INFO:teuthology.orchestra.run.smithi016.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.0) 2023-05-22T15:27:21.598 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:27:21.599 DEBUG:teuthology.orchestra.run.smithi016:> (cd / && exec bash -c ' 2023-05-22T15:27:21.599 DEBUG:teuthology.orchestra.run.smithi016:> set -e 2023-05-22T15:27:21.599 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.2 2023-05-22T15:27:21.600 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.2 1 2023-05-22T15:27:21.600 DEBUG:teuthology.orchestra.run.smithi016:> ') 2023-05-22T15:27:21.668 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:27:21.669 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2023-05-22T15:27:21.722 INFO:teuthology.orchestra.run.smithi016.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-05-22T15:27:21.723 INFO:teuthology.orchestra.run.smithi016.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-05-22T15:27:21.723 INFO:teuthology.orchestra.run.smithi016.stdout: inet 127.0.0.1/8 scope host lo 2023-05-22T15:27:21.723 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-05-22T15:27:21.723 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 ::1/128 scope host 2023-05-22T15:27:21.723 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-05-22T15:27:21.723 INFO:teuthology.orchestra.run.smithi016.stdout:2: veth0@if10: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-05-22T15:27:21.724 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether ee:88:d5:b6:4b:47 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2023-05-22T15:27:21.724 INFO:teuthology.orchestra.run.smithi016.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2023-05-22T15:27:21.724 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-05-22T15:27:21.724 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::ec88:d5ff:feb6:4b47/64 scope link 2023-05-22T15:27:21.724 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-05-22T15:27:21.726 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:27:21.726 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2023-05-22T15:27:21.782 INFO:teuthology.orchestra.run.smithi016.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-05-22T15:27:21.782 INFO:teuthology.orchestra.run.smithi016.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-05-22T15:27:21.782 INFO:teuthology.orchestra.run.smithi016.stdout: inet 127.0.0.1/8 scope host lo 2023-05-22T15:27:21.782 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-05-22T15:27:21.782 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 ::1/128 scope host 2023-05-22T15:27:21.782 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-05-22T15:27:21.783 INFO:teuthology.orchestra.run.smithi016.stdout:2: veth0@if10: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-05-22T15:27:21.783 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether ee:88:d5:b6:4b:47 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2023-05-22T15:27:21.783 INFO:teuthology.orchestra.run.smithi016.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2023-05-22T15:27:21.783 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-05-22T15:27:21.783 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::ec88:d5ff:feb6:4b47/64 scope link 2023-05-22T15:27:21.783 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-05-22T15:27:21.784 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.2' with 192.168.0.2/16 2023-05-22T15:27:21.785 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:27:21.785 DEBUG:teuthology.orchestra.run.smithi016:> (cd / && exec bash -c ' 2023-05-22T15:27:21.786 DEBUG:teuthology.orchestra.run.smithi016:> set -e 2023-05-22T15:27:21.786 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.2 type veth peer name brx.1 2023-05-22T15:27:21.786 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2023-05-22T15:27:21.786 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set veth0 up 2023-05-22T15:27:21.787 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set lo up 2023-05-22T15:27:21.787 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip route add default via 192.168.255.254 2023-05-22T15:27:21.787 DEBUG:teuthology.orchestra.run.smithi016:> ') 2023-05-22T15:27:21.883 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:27:21.883 DEBUG:teuthology.orchestra.run.smithi016:> (cd / && exec bash -c ' 2023-05-22T15:27:21.884 DEBUG:teuthology.orchestra.run.smithi016:> set -e 2023-05-22T15:27:21.885 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip link set brx.1 up 2023-05-22T15:27:21.885 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip link set dev brx.1 master ceph-brx 2023-05-22T15:27:21.885 DEBUG:teuthology.orchestra.run.smithi016:> ') 2023-05-22T15:27:21.907 INFO:tasks.cephfs.fuse_mount:Client client.2 config is {} 2023-05-22T15:27:21.907 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-05-22T15:27:21.907 DEBUG:teuthology.orchestra.run.smithi016:> mkdir -p -v /home/ubuntu/cephtest/mnt.2 2023-05-22T15:27:21.958 INFO:teuthology.orchestra.run.smithi016.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.2' 2023-05-22T15:27:21.958 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-05-22T15:27:21.959 DEBUG:teuthology.orchestra.run.smithi016:> chmod 0000 /home/ubuntu/cephtest/mnt.2 2023-05-22T15:27:22.006 DEBUG:teuthology.orchestra.run.smithi016:> sudo modprobe fuse 2023-05-22T15:27:22.064 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-05-22T15:27:22.064 DEBUG:teuthology.orchestra.run.smithi016:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-05-22T15:27:22.117 INFO:teuthology.orchestra.run.smithi016.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-05-22T15:27:22.119 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-05-22T15:27:22.120 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:27:22.120 DEBUG:teuthology.orchestra.run.smithi016:> ls /sys/fs/fuse/connections 2023-05-22T15:27:22.170 INFO:teuthology.orchestra.run.smithi016.stdout:45 2023-05-22T15:27:22.171 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [45] 2023-05-22T15:27:22.171 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.2 --id 2) 2023-05-22T15:27:22.215 DEBUG:teuthology.orchestra.run.smithi016:> sudo modprobe fuse 2023-05-22T15:27:22.231 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-05-22T15:27:22.232 DEBUG:teuthology.orchestra.run.smithi016:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-05-22T15:27:22.284 INFO:teuthology.orchestra.run.smithi016.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-05-22T15:27:22.284 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-05-22T15:27:22.285 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:27:22.285 DEBUG:teuthology.orchestra.run.smithi016:> ls /sys/fs/fuse/connections 2023-05-22T15:27:22.297 INFO:tasks.cephfs.fuse_mount.ceph-fuse.2.smithi016.stderr:2023-05-22T15:27:22.287+0000 7f9849349580 -1 init, newargv = 0x55aad2957c80 newargc=15 2023-05-22T15:27:22.297 INFO:tasks.cephfs.fuse_mount.ceph-fuse.2.smithi016.stderr:ceph-fuse[45535]: starting ceph client 2023-05-22T15:27:22.300 INFO:teuthology.orchestra.run.smithi016.stdout:45 2023-05-22T15:27:22.303 INFO:tasks.cephfs.fuse_mount.ceph-fuse.2.smithi016.stderr:ceph-fuse[45535]: starting fuse 2023-05-22T15:27:23.402 DEBUG:teuthology.orchestra.run.smithi016:> sudo modprobe fuse 2023-05-22T15:27:23.420 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-05-22T15:27:23.420 DEBUG:teuthology.orchestra.run.smithi016:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-05-22T15:27:23.474 INFO:teuthology.orchestra.run.smithi016.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-05-22T15:27:23.475 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-05-22T15:27:23.476 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:27:23.476 DEBUG:teuthology.orchestra.run.smithi016:> ls /sys/fs/fuse/connections 2023-05-22T15:27:23.522 INFO:teuthology.orchestra.run.smithi016.stdout:45 2023-05-22T15:27:23.523 INFO:teuthology.orchestra.run.smithi016.stdout:46 2023-05-22T15:27:23.523 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [45, 46] 2023-05-22T15:27:23.523 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2023-05-22T15:27:23.524 DEBUG:teuthology.orchestra.run.smithi016:> import glob 2023-05-22T15:27:23.524 DEBUG:teuthology.orchestra.run.smithi016:> import re 2023-05-22T15:27:23.524 DEBUG:teuthology.orchestra.run.smithi016:> import os 2023-05-22T15:27:23.524 DEBUG:teuthology.orchestra.run.smithi016:> import subprocess 2023-05-22T15:27:23.525 DEBUG:teuthology.orchestra.run.smithi016:> 2023-05-22T15:27:23.525 DEBUG:teuthology.orchestra.run.smithi016:> def _find_admin_socket(client_name): 2023-05-22T15:27:23.525 DEBUG:teuthology.orchestra.run.smithi016:> asok_path = "/var/run/ceph/ceph-client.2.*.asok" 2023-05-22T15:27:23.525 DEBUG:teuthology.orchestra.run.smithi016:> files = glob.glob(asok_path) 2023-05-22T15:27:23.525 DEBUG:teuthology.orchestra.run.smithi016:> mountpoint = "/home/ubuntu/cephtest/mnt.2" 2023-05-22T15:27:23.526 DEBUG:teuthology.orchestra.run.smithi016:> 2023-05-22T15:27:23.526 DEBUG:teuthology.orchestra.run.smithi016:> # Given a non-glob path, it better be there 2023-05-22T15:27:23.526 DEBUG:teuthology.orchestra.run.smithi016:> if "*" not in asok_path: 2023-05-22T15:27:23.526 DEBUG:teuthology.orchestra.run.smithi016:> assert(len(files) == 1) 2023-05-22T15:27:23.527 DEBUG:teuthology.orchestra.run.smithi016:> return files[0] 2023-05-22T15:27:23.527 DEBUG:teuthology.orchestra.run.smithi016:> 2023-05-22T15:27:23.527 DEBUG:teuthology.orchestra.run.smithi016:> for f in files: 2023-05-22T15:27:23.527 DEBUG:teuthology.orchestra.run.smithi016:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-05-22T15:27:23.528 DEBUG:teuthology.orchestra.run.smithi016:> if os.path.exists("/proc/{0}".format(pid)): 2023-05-22T15:27:23.528 DEBUG:teuthology.orchestra.run.smithi016:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-05-22T15:27:23.528 DEBUG:teuthology.orchestra.run.smithi016:> contents = proc_f.read() 2023-05-22T15:27:23.528 DEBUG:teuthology.orchestra.run.smithi016:> if mountpoint in contents: 2023-05-22T15:27:23.529 DEBUG:teuthology.orchestra.run.smithi016:> return f 2023-05-22T15:27:23.529 DEBUG:teuthology.orchestra.run.smithi016:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-05-22T15:27:23.529 DEBUG:teuthology.orchestra.run.smithi016:> 2023-05-22T15:27:23.529 DEBUG:teuthology.orchestra.run.smithi016:> print(_find_admin_socket("client.2")) 2023-05-22T15:27:23.529 DEBUG:teuthology.orchestra.run.smithi016:> ' 2023-05-22T15:27:23.647 INFO:teuthology.orchestra.run.smithi016.stdout:/var/run/ceph/ceph-client.2.45535.asok 2023-05-22T15:27:23.829 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.2.45535.asok 2023-05-22T15:27:23.829 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:27:23.830 DEBUG:teuthology.orchestra.run.smithi016:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.2.45535.asok status 2023-05-22T15:27:23.985 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-05-22T15:27:23.985 INFO:teuthology.orchestra.run.smithi016.stdout: "metadata": { 2023-05-22T15:27:23.985 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph_sha1": "9c9ba4df22024d9d8d6cf9f98a4b96751fc78d0c", 2023-05-22T15:27:23.986 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph_version": "ceph version 18.0.0-3192-g9c9ba4df (9c9ba4df22024d9d8d6cf9f98a4b96751fc78d0c) reef (dev)", 2023-05-22T15:27:23.986 INFO:teuthology.orchestra.run.smithi016.stdout: "entity_id": "2", 2023-05-22T15:27:23.986 INFO:teuthology.orchestra.run.smithi016.stdout: "hostname": "smithi016", 2023-05-22T15:27:23.986 INFO:teuthology.orchestra.run.smithi016.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.2", 2023-05-22T15:27:23.986 INFO:teuthology.orchestra.run.smithi016.stdout: "pid": "45535", 2023-05-22T15:27:23.986 INFO:teuthology.orchestra.run.smithi016.stdout: "root": "/" 2023-05-22T15:27:23.986 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:27:23.987 INFO:teuthology.orchestra.run.smithi016.stdout: "dentry_count": 0, 2023-05-22T15:27:23.987 INFO:teuthology.orchestra.run.smithi016.stdout: "dentry_pinned_count": 0, 2023-05-22T15:27:23.987 INFO:teuthology.orchestra.run.smithi016.stdout: "id": 4875, 2023-05-22T15:27:23.987 INFO:teuthology.orchestra.run.smithi016.stdout: "inst": { 2023-05-22T15:27:23.987 INFO:teuthology.orchestra.run.smithi016.stdout: "name": { 2023-05-22T15:27:23.987 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "client", 2023-05-22T15:27:23.987 INFO:teuthology.orchestra.run.smithi016.stdout: "num": 4875 2023-05-22T15:27:23.988 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:27:23.988 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": { 2023-05-22T15:27:23.988 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-05-22T15:27:23.988 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "192.168.0.2:0", 2023-05-22T15:27:23.988 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 3429709350 2023-05-22T15:27:23.988 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T15:27:23.988 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:27:23.988 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": { 2023-05-22T15:27:23.989 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-05-22T15:27:23.989 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "192.168.0.2:0", 2023-05-22T15:27:23.989 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 3429709350 2023-05-22T15:27:23.989 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:27:23.989 INFO:teuthology.orchestra.run.smithi016.stdout: "inst_str": "client.4875 192.168.0.2:0/3429709350", 2023-05-22T15:27:23.989 INFO:teuthology.orchestra.run.smithi016.stdout: "addr_str": "192.168.0.2:0/3429709350", 2023-05-22T15:27:23.989 INFO:teuthology.orchestra.run.smithi016.stdout: "inode_count": 1, 2023-05-22T15:27:23.989 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_epoch": 8, 2023-05-22T15:27:23.990 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_epoch": 27, 2023-05-22T15:27:23.990 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_epoch_barrier": 0, 2023-05-22T15:27:23.990 INFO:teuthology.orchestra.run.smithi016.stdout: "blocklisted": false, 2023-05-22T15:27:23.990 INFO:teuthology.orchestra.run.smithi016.stdout: "fs_name": "cephfs" 2023-05-22T15:27:23.990 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-05-22T15:27:23.999 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-05-22T15:27:24.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:24.115+0000 7f53c6b8d640 1 Processor -- start 2023-05-22T15:27:24.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:24.115+0000 7f53c6b8d640 1 -- start start 2023-05-22T15:27:24.126 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:24.119+0000 7f53c6b8d640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f53c015c350 0x7f53c015c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:24.126 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:24.119+0000 7f53c6b8d640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53c00c95f0 con 0x7f53c00ade80 2023-05-22T15:27:24.126 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:24.119+0000 7f53c6b8d640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53c00c98d0 con 0x7f53c0069720 2023-05-22T15:27:24.127 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:24.119+0000 7f53c6b8d640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53c00ca270 con 0x7f53c015c350 2023-05-22T15:27:24.127 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:24.119+0000 7f53c4929640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f53c00ade80 0x7f53c00cc400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:54830/0 (socket says 172.21.15.16:54830) 2023-05-22T15:27:24.127 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:24.119+0000 7f53c4929640 1 -- 172.21.15.16:0/2402761647 learned_addr learned my addr 172.21.15.16:0/2402761647 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:27:24.128 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:24.119+0000 7f53bf7fe640 1 -- 172.21.15.16:0/2402761647 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 966732976 0 0) 0x7f53c00c95f0 con 0x7f53c00ade80 2023-05-22T15:27:24.128 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:24.119+0000 7f53bf7fe640 1 -- 172.21.15.16:0/2402761647 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53b0003530 con 0x7f53c00ade80 2023-05-22T15:27:24.129 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:24.119+0000 7f53bffff640 1 --2- 172.21.15.16:0/2402761647 >> v2:172.21.15.62:3300/0 conn(0x7f53c015c350 0x7f53c015c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:24.129 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:24.119+0000 7f53bf7fe640 1 -- 172.21.15.16:0/2402761647 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 557404717 0 0) 0x7f53b0003530 con 0x7f53c00ade80 2023-05-22T15:27:24.129 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:24.119+0000 7f53bf7fe640 1 -- 172.21.15.16:0/2402761647 >> v2:172.21.15.62:3300/0 conn(0x7f53c015c350 msgr2=0x7f53c015c7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:24.129 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:24.119+0000 7f53bf7fe640 1 --2- 172.21.15.16:0/2402761647 >> v2:172.21.15.62:3300/0 conn(0x7f53c015c350 0x7f53c015c7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:24.129 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:24.119+0000 7f53bf7fe640 1 -- 172.21.15.16:0/2402761647 >> v1:172.21.15.62:6789/0 conn(0x7f53c0069720 legacy=0x7f53c0069b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:24.129 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:24.119+0000 7f53bf7fe640 1 -- 172.21.15.16:0/2402761647 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53c00c9bb0 con 0x7f53c00ade80 2023-05-22T15:27:24.129 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:24.119+0000 7f53bf7fe640 1 -- 172.21.15.16:0/2402761647 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7f53b4001a00 con 0x7f53c00ade80 2023-05-22T15:27:24.130 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:24.119+0000 7f53bf7fe640 1 -- 172.21.15.16:0/2402761647 >> v1:172.21.15.16:6789/0 conn(0x7f53c00ade80 legacy=0x7f53c00cc400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:24.130 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:24.119+0000 7f53bf7fe640 1 --2- 172.21.15.16:0/2402761647 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f53b0003980 0x7f53b0003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:24.130 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:24.119+0000 7f53bf7fe640 1 --2- 172.21.15.16:0/2402761647 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f53b00047d0 0x7f53b0004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:24.130 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:24.119+0000 7f53bf7fe640 1 --2- 172.21.15.16:0/2402761647 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f53b0005180 0x7f53b000b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:24.130 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:24.119+0000 7f53bf7fe640 1 -- 172.21.15.16:0/2402761647 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53c00ca270 con 0x7f53b00047d0 2023-05-22T15:27:24.130 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:24.119+0000 7f53bf7fe640 1 -- 172.21.15.16:0/2402761647 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53b000c090 con 0x7f53b0005180 2023-05-22T15:27:24.130 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:24.119+0000 7f53bf7fe640 1 -- 172.21.15.16:0/2402761647 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53b000c250 con 0x7f53b0003980 2023-05-22T15:27:24.130 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:24.119+0000 7f53bffff640 1 --2- 172.21.15.16:0/2402761647 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f53b00047d0 0x7f53b0004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:24.131 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:24.119+0000 7f53c4929640 1 --2- 172.21.15.16:0/2402761647 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f53b0003980 0x7f53b0003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:24.131 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:24.119+0000 7f53bffff640 1 -- 172.21.15.16:0/2402761647 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f53b0003980 msgr2=0x7f53b0003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:24.131 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:24.119+0000 7f53bffff640 1 --2- 172.21.15.16:0/2402761647 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f53b0003980 0x7f53b0003d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:24.131 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:24.119+0000 7f53bffff640 1 -- 172.21.15.16:0/2402761647 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f53b0005180 msgr2=0x7f53b000b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:24.131 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:24.119+0000 7f53bffff640 1 --2- 172.21.15.16:0/2402761647 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f53b0005180 0x7f53b000b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:24.131 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:24.119+0000 7f53bffff640 1 -- 172.21.15.16:0/2402761647 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53b0003530 con 0x7f53b00047d0 2023-05-22T15:27:24.131 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:24.119+0000 7f53c512a640 1 -- 172.21.15.16:0/2402761647 reap_dead start 2023-05-22T15:27:24.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:24.123+0000 7f53bffff640 1 --2- 172.21.15.16:0/2402761647 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f53b00047d0 0x7f53b0004c40 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f53ac009ae0 tx=0x7f53ac01ca90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f427603cb9bf3f0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:24.134 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:24.123+0000 7f53bf7fe640 1 -- 172.21.15.16:0/2402761647 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53ac0a6490 con 0x7f53b00047d0 2023-05-22T15:27:24.134 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:24.123+0000 7f53bf7fe640 1 -- 172.21.15.16:0/2402761647 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53ac0a65f0 con 0x7f53b00047d0 2023-05-22T15:27:24.134 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:24.123+0000 7f53bf7fe640 1 -- 172.21.15.16:0/2402761647 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53ac0a69a0 con 0x7f53b00047d0 2023-05-22T15:27:24.134 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:24.123+0000 7f53c6b8d640 1 -- 172.21.15.16:0/2402761647 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f53b00047d0 msgr2=0x7f53b0004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:24.134 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:24.123+0000 7f53c6b8d640 1 --2- 172.21.15.16:0/2402761647 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f53b00047d0 0x7f53b0004c40 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f53ac009ae0 tx=0x7f53ac01ca90 comp rx=0 tx=0).stop 2023-05-22T15:27:24.134 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:24.123+0000 7f53c6b8d640 1 -- 172.21.15.16:0/2402761647 shutdown_connections 2023-05-22T15:27:24.134 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:24.123+0000 7f53c6b8d640 1 --2- 172.21.15.16:0/2402761647 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f53b00047d0 0x7f53b0004c40 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:24.135 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:24.123+0000 7f53c6b8d640 1 -- 172.21.15.16:0/2402761647 >> 172.21.15.16:0/2402761647 conn(0x7f53c00b1110 msgr2=0x7f53c00ce830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:24.135 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:24.123+0000 7f53c6b8d640 1 -- 172.21.15.16:0/2402761647 shutdown_connections 2023-05-22T15:27:24.135 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:24.123+0000 7f53c6b8d640 1 -- 172.21.15.16:0/2402761647 wait complete. 2023-05-22T15:27:24.135 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:24.123+0000 7f53c6b8d640 1 Processor -- start 2023-05-22T15:27:24.135 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:24.123+0000 7f53c6b8d640 1 -- start start 2023-05-22T15:27:24.135 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:24.123+0000 7f53c6b8d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f53c0152b60 0x7f53c0146dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:24.135 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:24.123+0000 7f53c6b8d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f53c0147310 0x7f53c00b17c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:24.136 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:24.123+0000 7f53c6b8d640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f53c0152f30 0x7f53c00b1d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:24.136 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:24.123+0000 7f53c6b8d640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53c01469d0 con 0x7f53c0147310 2023-05-22T15:27:24.136 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:24.123+0000 7f53c6b8d640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53c014cb20 con 0x7f53c0152b60 2023-05-22T15:27:24.136 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:24.123+0000 7f53c6b8d640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53c014ed60 con 0x7f53c0152f30 2023-05-22T15:27:24.136 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:24.123+0000 7f53bffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f53c0147310 0x7f53c00b17c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:24.136 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:24.123+0000 7f53bffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f53c0147310 0x7f53c00b17c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56548/0 (socket says 172.21.15.16:56548) 2023-05-22T15:27:24.136 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:24.123+0000 7f53bffff640 1 -- 172.21.15.16:0/4044249599 learned_addr learned my addr 172.21.15.16:0/4044249599 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:24.136 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:24.123+0000 7f53c4929640 1 --2- 172.21.15.16:0/4044249599 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f53c0152b60 0x7f53c0146dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:24.137 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:24.123+0000 7f53c512a640 1 --2- 172.21.15.16:0/4044249599 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f53c0152f30 0x7f53c00b1d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:24.137 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:24.123+0000 7f53bffff640 1 -- 172.21.15.16:0/4044249599 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f53c0152f30 msgr2=0x7f53c00b1d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:24.137 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:24.123+0000 7f53bffff640 1 --2- 172.21.15.16:0/4044249599 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f53c0152f30 0x7f53c00b1d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:24.137 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:24.123+0000 7f53bffff640 1 -- 172.21.15.16:0/4044249599 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f53c0152b60 msgr2=0x7f53c0146dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:24.137 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:24.123+0000 7f53bffff640 1 --2- 172.21.15.16:0/4044249599 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f53c0152b60 0x7f53c0146dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:24.137 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:24.123+0000 7f53bffff640 1 -- 172.21.15.16:0/4044249599 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53c015ccf0 con 0x7f53c0147310 2023-05-22T15:27:24.137 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:24.123+0000 7f53c512a640 1 --2- 172.21.15.16:0/4044249599 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f53c0152f30 0x7f53c00b1d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:27:24.137 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:24.123+0000 7f53bffff640 1 --2- 172.21.15.16:0/4044249599 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f53c0147310 0x7f53c00b17c0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f53ac01d330 tx=0x7f53ac019830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:24.138 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:24.127+0000 7f53bd7fa640 1 -- 172.21.15.16:0/4044249599 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53ac0a6490 con 0x7f53c0147310 2023-05-22T15:27:24.138 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:24.127+0000 7f53bd7fa640 1 -- 172.21.15.16:0/4044249599 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53ac0a6e60 con 0x7f53c0147310 2023-05-22T15:27:24.138 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:24.127+0000 7f53c6b8d640 1 -- 172.21.15.16:0/4044249599 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53c015cb60 con 0x7f53c0147310 2023-05-22T15:27:24.138 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:24.127+0000 7f53bd7fa640 1 -- 172.21.15.16:0/4044249599 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53ac0a7470 con 0x7f53c0147310 2023-05-22T15:27:24.138 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:24.127+0000 7f53c6b8d640 1 -- 172.21.15.16:0/4044249599 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53c00c9210 con 0x7f53c0147310 2023-05-22T15:27:24.139 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:24.127+0000 7f53c6b8d640 1 -- 172.21.15.16:0/4044249599 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53c004dbe0 con 0x7f53c0147310 2023-05-22T15:27:24.139 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:24.127+0000 7f53bd7fa640 1 -- 172.21.15.16:0/4044249599 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7f53ac0a75d0 con 0x7f53c0147310 2023-05-22T15:27:24.139 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:24.127+0000 7f53bd7fa640 1 --2- 172.21.15.16:0/4044249599 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f53a4065380 0x7f53a4067830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:24.139 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:24.127+0000 7f53bd7fa640 1 -- 172.21.15.16:0/4044249599 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f53ac11c4c0 con 0x7f53c0147310 2023-05-22T15:27:24.140 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:24.127+0000 7f53c4929640 1 --2- 172.21.15.16:0/4044249599 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f53a4065380 0x7f53a4067830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:24.140 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:24.127+0000 7f53c4929640 1 --2- 172.21.15.16:0/4044249599 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f53a4065380 0x7f53a4067830 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f53c014ddc0 tx=0x7f53b4009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:24.140 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:24.131+0000 7f53bd7fa640 1 -- 172.21.15.16:0/4044249599 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f53ac0e9e80 con 0x7f53c0147310 2023-05-22T15:27:24.329 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:24.319+0000 7f53c6b8d640 1 -- 172.21.15.16:0/4044249599 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f53c006bf10 con 0x7f53c0147310 2023-05-22T15:27:24.330 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:24.323+0000 7f53bd7fa640 1 -- 172.21.15.16:0/4044249599 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) v1 ==== 53+0+88 (secure 0 0 0) 0x7f53ac0ed4a0 con 0x7f53c0147310 2023-05-22T15:27:24.330 INFO:teuthology.orchestra.run.smithi016.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2023-05-22T15:27:24.332 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:24.323+0000 7f53c6b8d640 1 -- 172.21.15.16:0/4044249599 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f53a4065380 msgr2=0x7f53a4067830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:24.332 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:24.323+0000 7f53c6b8d640 1 --2- 172.21.15.16:0/4044249599 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f53a4065380 0x7f53a4067830 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f53c014ddc0 tx=0x7f53b4009040 comp rx=0 tx=0).stop 2023-05-22T15:27:24.332 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:24.323+0000 7f53c6b8d640 1 -- 172.21.15.16:0/4044249599 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f53c0147310 msgr2=0x7f53c00b17c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:24.332 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:24.323+0000 7f53c6b8d640 1 --2- 172.21.15.16:0/4044249599 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f53c0147310 0x7f53c00b17c0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f53ac01d330 tx=0x7f53ac019830 comp rx=0 tx=0).stop 2023-05-22T15:27:24.332 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:24.323+0000 7f53c6b8d640 1 -- 172.21.15.16:0/4044249599 shutdown_connections 2023-05-22T15:27:24.333 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:24.323+0000 7f53c6b8d640 1 --2- 172.21.15.16:0/4044249599 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f53a4065380 0x7f53a4067830 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:24.333 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:24.323+0000 7f53c6b8d640 1 --2- 172.21.15.16:0/4044249599 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f53c0152f30 0x7f53c00b1d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:24.333 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:24.323+0000 7f53c6b8d640 1 --2- 172.21.15.16:0/4044249599 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f53c0147310 0x7f53c00b17c0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:24.333 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:24.323+0000 7f53c6b8d640 1 --2- 172.21.15.16:0/4044249599 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f53c0152b60 0x7f53c0146dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:24.333 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:24.323+0000 7f53c6b8d640 1 -- 172.21.15.16:0/4044249599 >> 172.21.15.16:0/4044249599 conn(0x7f53c00b1110 msgr2=0x7f53c00708e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:24.333 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:24.323+0000 7f53c6b8d640 1 -- 172.21.15.16:0/4044249599 shutdown_connections 2023-05-22T15:27:24.333 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:24.323+0000 7f53c6b8d640 1 -- 172.21.15.16:0/4044249599 wait complete. 2023-05-22T15:27:24.349 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-05-22T15:27:24.349 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-05-22T15:27:24.350 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi062.front.sepia.ceph.com 2023-05-22T15:27:24.350 INFO:tasks.cephfs.mount:self.client.name = client.1 2023-05-22T15:27:24.350 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2023-05-22T15:27:24.350 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-05-22T15:27:24.350 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-05-22T15:27:24.350 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-05-22T15:27:24.351 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.1' 2023-05-22T15:27:24.351 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:27:24.351 DEBUG:teuthology.orchestra.run.smithi062:> ip addr 2023-05-22T15:27:24.358 INFO:teuthology.orchestra.run.smithi062.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-05-22T15:27:24.359 INFO:teuthology.orchestra.run.smithi062.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-05-22T15:27:24.359 INFO:teuthology.orchestra.run.smithi062.stdout: inet 127.0.0.1/8 scope host lo 2023-05-22T15:27:24.359 INFO:teuthology.orchestra.run.smithi062.stdout: valid_lft forever preferred_lft forever 2023-05-22T15:27:24.359 INFO:teuthology.orchestra.run.smithi062.stdout: inet6 ::1/128 scope host 2023-05-22T15:27:24.359 INFO:teuthology.orchestra.run.smithi062.stdout: valid_lft forever preferred_lft forever 2023-05-22T15:27:24.359 INFO:teuthology.orchestra.run.smithi062.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-05-22T15:27:24.360 INFO:teuthology.orchestra.run.smithi062.stdout: link/ether 0c:c4:7a:a9:aa:14 brd ff:ff:ff:ff:ff:ff 2023-05-22T15:27:24.360 INFO:teuthology.orchestra.run.smithi062.stdout: altname enp7s0f0 2023-05-22T15:27:24.360 INFO:teuthology.orchestra.run.smithi062.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-05-22T15:27:24.360 INFO:teuthology.orchestra.run.smithi062.stdout: link/ether 0c:c4:7a:a9:aa:15 brd ff:ff:ff:ff:ff:ff 2023-05-22T15:27:24.360 INFO:teuthology.orchestra.run.smithi062.stdout: altname enp7s0f1 2023-05-22T15:27:24.360 INFO:teuthology.orchestra.run.smithi062.stdout:4: enp3s0f0: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-05-22T15:27:24.360 INFO:teuthology.orchestra.run.smithi062.stdout: link/ether 0c:c4:7a:bb:d1:c2 brd ff:ff:ff:ff:ff:ff 2023-05-22T15:27:24.361 INFO:teuthology.orchestra.run.smithi062.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-05-22T15:27:24.361 INFO:teuthology.orchestra.run.smithi062.stdout: link/ether 0c:c4:7a:bb:d1:c3 brd ff:ff:ff:ff:ff:ff 2023-05-22T15:27:24.361 INFO:teuthology.orchestra.run.smithi062.stdout: inet 172.21.15.62/20 metric 100 brd 172.21.15.255 scope global dynamic enp3s0f1 2023-05-22T15:27:24.361 INFO:teuthology.orchestra.run.smithi062.stdout: valid_lft 42476sec preferred_lft 42476sec 2023-05-22T15:27:24.361 INFO:teuthology.orchestra.run.smithi062.stdout: inet6 fe80::ec4:7aff:febb:d1c3/64 scope link 2023-05-22T15:27:24.362 INFO:teuthology.orchestra.run.smithi062.stdout: valid_lft forever preferred_lft forever 2023-05-22T15:27:24.362 INFO:teuthology.orchestra.run.smithi062.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2023-05-22T15:27:24.362 INFO:teuthology.orchestra.run.smithi062.stdout: link/ether 02:42:2f:a3:54:48 brd ff:ff:ff:ff:ff:ff 2023-05-22T15:27:24.362 INFO:teuthology.orchestra.run.smithi062.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2023-05-22T15:27:24.362 INFO:teuthology.orchestra.run.smithi062.stdout: valid_lft forever preferred_lft forever 2023-05-22T15:27:24.362 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2023-05-22T15:27:24.363 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:27:24.363 DEBUG:teuthology.orchestra.run.smithi062:> (cd / && exec bash -c ' 2023-05-22T15:27:24.363 DEBUG:teuthology.orchestra.run.smithi062:> set -e 2023-05-22T15:27:24.364 DEBUG:teuthology.orchestra.run.smithi062:> sudo ip link add name ceph-brx type bridge 2023-05-22T15:27:24.364 DEBUG:teuthology.orchestra.run.smithi062:> sudo ip addr flush dev ceph-brx 2023-05-22T15:27:24.364 DEBUG:teuthology.orchestra.run.smithi062:> sudo ip link set ceph-brx up 2023-05-22T15:27:24.364 DEBUG:teuthology.orchestra.run.smithi062:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2023-05-22T15:27:24.364 DEBUG:teuthology.orchestra.run.smithi062:> ') 2023-05-22T15:27:24.442 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:27:24.442 DEBUG:teuthology.orchestra.run.smithi062:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-05-22T15:27:24.494 INFO:teuthology.orchestra.run.smithi062.stdout:1 2023-05-22T15:27:24.495 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:27:24.496 DEBUG:teuthology.orchestra.run.smithi062:> route 2023-05-22T15:27:24.561 INFO:teuthology.orchestra.run.smithi062.stdout:Kernel IP routing table 2023-05-22T15:27:24.561 INFO:teuthology.orchestra.run.smithi062.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-05-22T15:27:24.562 INFO:teuthology.orchestra.run.smithi062.stdout:default _gateway 0.0.0.0 UG 0 0 0 enp3s0f1 2023-05-22T15:27:24.562 INFO:teuthology.orchestra.run.smithi062.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-05-22T15:27:24.562 INFO:teuthology.orchestra.run.smithi062.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2023-05-22T15:27:24.562 INFO:teuthology.orchestra.run.smithi062.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-05-22T15:27:24.563 INFO:teuthology.orchestra.run.smithi062.stdout:ns1.front.sepia 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2023-05-22T15:27:24.563 INFO:teuthology.orchestra.run.smithi062.stdout:ns2.front.sepia 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2023-05-22T15:27:24.563 INFO:teuthology.orchestra.run.smithi062.stdout:_gateway 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2023-05-22T15:27:24.563 INFO:teuthology.orchestra.run.smithi062.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-05-22T15:27:24.564 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:27:24.565 DEBUG:teuthology.orchestra.run.smithi062:> (cd / && exec bash -c ' 2023-05-22T15:27:24.565 DEBUG:teuthology.orchestra.run.smithi062:> set -e 2023-05-22T15:27:24.566 DEBUG:teuthology.orchestra.run.smithi062:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-05-22T15:27:24.567 DEBUG:teuthology.orchestra.run.smithi062:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-05-22T15:27:24.568 DEBUG:teuthology.orchestra.run.smithi062:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-05-22T15:27:24.568 DEBUG:teuthology.orchestra.run.smithi062:> ') 2023-05-22T15:27:24.644 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:27:24.644 DEBUG:teuthology.orchestra.run.smithi062:> ip netns list 2023-05-22T15:27:24.689 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:27:24.690 DEBUG:teuthology.orchestra.run.smithi062:> ip netns list-id 2023-05-22T15:27:24.736 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:27:24.737 DEBUG:teuthology.orchestra.run.smithi062:> (cd / && exec bash -c ' 2023-05-22T15:27:24.737 DEBUG:teuthology.orchestra.run.smithi062:> set -e 2023-05-22T15:27:24.737 DEBUG:teuthology.orchestra.run.smithi062:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.1 2023-05-22T15:27:24.738 DEBUG:teuthology.orchestra.run.smithi062:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.1 0 2023-05-22T15:27:24.738 DEBUG:teuthology.orchestra.run.smithi062:> ') 2023-05-22T15:27:24.811 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.1' with 192.168.0.1/16 2023-05-22T15:27:24.811 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:27:24.812 DEBUG:teuthology.orchestra.run.smithi062:> (cd / && exec bash -c ' 2023-05-22T15:27:24.812 DEBUG:teuthology.orchestra.run.smithi062:> set -e 2023-05-22T15:27:24.812 DEBUG:teuthology.orchestra.run.smithi062:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.1 type veth peer name brx.0 2023-05-22T15:27:24.813 DEBUG:teuthology.orchestra.run.smithi062:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2023-05-22T15:27:24.813 DEBUG:teuthology.orchestra.run.smithi062:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set veth0 up 2023-05-22T15:27:24.813 DEBUG:teuthology.orchestra.run.smithi062:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set lo up 2023-05-22T15:27:24.813 DEBUG:teuthology.orchestra.run.smithi062:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip route add default via 192.168.255.254 2023-05-22T15:27:24.813 DEBUG:teuthology.orchestra.run.smithi062:> ') 2023-05-22T15:27:24.926 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:27:24.927 DEBUG:teuthology.orchestra.run.smithi062:> (cd / && exec bash -c ' 2023-05-22T15:27:24.927 DEBUG:teuthology.orchestra.run.smithi062:> set -e 2023-05-22T15:27:24.927 DEBUG:teuthology.orchestra.run.smithi062:> sudo ip link set brx.0 up 2023-05-22T15:27:24.927 DEBUG:teuthology.orchestra.run.smithi062:> sudo ip link set dev brx.0 master ceph-brx 2023-05-22T15:27:24.928 DEBUG:teuthology.orchestra.run.smithi062:> ') 2023-05-22T15:27:24.949 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2023-05-22T15:27:24.949 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-05-22T15:27:24.950 DEBUG:teuthology.orchestra.run.smithi062:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2023-05-22T15:27:24.997 INFO:teuthology.orchestra.run.smithi062.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2023-05-22T15:27:24.998 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-05-22T15:27:24.998 DEBUG:teuthology.orchestra.run.smithi062:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2023-05-22T15:27:25.045 DEBUG:teuthology.orchestra.run.smithi062:> sudo modprobe fuse 2023-05-22T15:27:25.104 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-05-22T15:27:25.104 DEBUG:teuthology.orchestra.run.smithi062:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-05-22T15:27:25.157 INFO:teuthology.orchestra.run.smithi062.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-05-22T15:27:25.158 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-05-22T15:27:25.158 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:27:25.159 DEBUG:teuthology.orchestra.run.smithi062:> ls /sys/fs/fuse/connections 2023-05-22T15:27:25.209 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2023-05-22T15:27:25.210 DEBUG:teuthology.orchestra.run.smithi062:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.1 --id 1) 2023-05-22T15:27:25.254 DEBUG:teuthology.orchestra.run.smithi062:> sudo modprobe fuse 2023-05-22T15:27:25.270 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-05-22T15:27:25.271 DEBUG:teuthology.orchestra.run.smithi062:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-05-22T15:27:25.323 INFO:teuthology.orchestra.run.smithi062.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-05-22T15:27:25.325 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-05-22T15:27:25.326 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:27:25.326 DEBUG:teuthology.orchestra.run.smithi062:> ls /sys/fs/fuse/connections 2023-05-22T15:27:25.345 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi062.stderr:2023-05-22T15:27:25.338+0000 7f6ff94c6580 -1 init, newargv = 0x56028a1d9e80 newargc=15 2023-05-22T15:27:25.345 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi062.stderr:ceph-fuse[43294]: starting ceph client 2023-05-22T15:27:25.351 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi062.stderr:ceph-fuse[43294]: starting fuse 2023-05-22T15:27:26.452 DEBUG:teuthology.orchestra.run.smithi062:> sudo modprobe fuse 2023-05-22T15:27:26.470 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-05-22T15:27:26.470 DEBUG:teuthology.orchestra.run.smithi062:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-05-22T15:27:26.525 INFO:teuthology.orchestra.run.smithi062.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-05-22T15:27:26.526 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-05-22T15:27:26.527 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:27:26.527 DEBUG:teuthology.orchestra.run.smithi062:> ls /sys/fs/fuse/connections 2023-05-22T15:27:26.573 INFO:teuthology.orchestra.run.smithi062.stdout:45 2023-05-22T15:27:26.573 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [45] 2023-05-22T15:27:26.574 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2023-05-22T15:27:26.574 DEBUG:teuthology.orchestra.run.smithi062:> import glob 2023-05-22T15:27:26.574 DEBUG:teuthology.orchestra.run.smithi062:> import re 2023-05-22T15:27:26.574 DEBUG:teuthology.orchestra.run.smithi062:> import os 2023-05-22T15:27:26.575 DEBUG:teuthology.orchestra.run.smithi062:> import subprocess 2023-05-22T15:27:26.575 DEBUG:teuthology.orchestra.run.smithi062:> 2023-05-22T15:27:26.575 DEBUG:teuthology.orchestra.run.smithi062:> def _find_admin_socket(client_name): 2023-05-22T15:27:26.575 DEBUG:teuthology.orchestra.run.smithi062:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2023-05-22T15:27:26.576 DEBUG:teuthology.orchestra.run.smithi062:> files = glob.glob(asok_path) 2023-05-22T15:27:26.576 DEBUG:teuthology.orchestra.run.smithi062:> mountpoint = "/home/ubuntu/cephtest/mnt.1" 2023-05-22T15:27:26.576 DEBUG:teuthology.orchestra.run.smithi062:> 2023-05-22T15:27:26.576 DEBUG:teuthology.orchestra.run.smithi062:> # Given a non-glob path, it better be there 2023-05-22T15:27:26.577 DEBUG:teuthology.orchestra.run.smithi062:> if "*" not in asok_path: 2023-05-22T15:27:26.577 DEBUG:teuthology.orchestra.run.smithi062:> assert(len(files) == 1) 2023-05-22T15:27:26.577 DEBUG:teuthology.orchestra.run.smithi062:> return files[0] 2023-05-22T15:27:26.577 DEBUG:teuthology.orchestra.run.smithi062:> 2023-05-22T15:27:26.578 DEBUG:teuthology.orchestra.run.smithi062:> for f in files: 2023-05-22T15:27:26.578 DEBUG:teuthology.orchestra.run.smithi062:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-05-22T15:27:26.578 DEBUG:teuthology.orchestra.run.smithi062:> if os.path.exists("/proc/{0}".format(pid)): 2023-05-22T15:27:26.578 DEBUG:teuthology.orchestra.run.smithi062:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-05-22T15:27:26.579 DEBUG:teuthology.orchestra.run.smithi062:> contents = proc_f.read() 2023-05-22T15:27:26.579 DEBUG:teuthology.orchestra.run.smithi062:> if mountpoint in contents: 2023-05-22T15:27:26.579 DEBUG:teuthology.orchestra.run.smithi062:> return f 2023-05-22T15:27:26.579 DEBUG:teuthology.orchestra.run.smithi062:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-05-22T15:27:26.580 DEBUG:teuthology.orchestra.run.smithi062:> 2023-05-22T15:27:26.580 DEBUG:teuthology.orchestra.run.smithi062:> print(_find_admin_socket("client.1")) 2023-05-22T15:27:26.580 DEBUG:teuthology.orchestra.run.smithi062:> ' 2023-05-22T15:27:26.697 INFO:teuthology.orchestra.run.smithi062.stdout:/var/run/ceph/ceph-client.1.43294.asok 2023-05-22T15:27:26.880 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.43294.asok 2023-05-22T15:27:26.881 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:27:26.881 DEBUG:teuthology.orchestra.run.smithi062:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.43294.asok status 2023-05-22T15:27:27.040 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2023-05-22T15:27:27.041 INFO:teuthology.orchestra.run.smithi062.stdout: "metadata": { 2023-05-22T15:27:27.041 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph_sha1": "9c9ba4df22024d9d8d6cf9f98a4b96751fc78d0c", 2023-05-22T15:27:27.041 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph_version": "ceph version 18.0.0-3192-g9c9ba4df (9c9ba4df22024d9d8d6cf9f98a4b96751fc78d0c) reef (dev)", 2023-05-22T15:27:27.041 INFO:teuthology.orchestra.run.smithi062.stdout: "entity_id": "1", 2023-05-22T15:27:27.041 INFO:teuthology.orchestra.run.smithi062.stdout: "hostname": "smithi062", 2023-05-22T15:27:27.041 INFO:teuthology.orchestra.run.smithi062.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2023-05-22T15:27:27.041 INFO:teuthology.orchestra.run.smithi062.stdout: "pid": "43294", 2023-05-22T15:27:27.042 INFO:teuthology.orchestra.run.smithi062.stdout: "root": "/" 2023-05-22T15:27:27.042 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2023-05-22T15:27:27.042 INFO:teuthology.orchestra.run.smithi062.stdout: "dentry_count": 0, 2023-05-22T15:27:27.042 INFO:teuthology.orchestra.run.smithi062.stdout: "dentry_pinned_count": 0, 2023-05-22T15:27:27.042 INFO:teuthology.orchestra.run.smithi062.stdout: "id": 4583, 2023-05-22T15:27:27.042 INFO:teuthology.orchestra.run.smithi062.stdout: "inst": { 2023-05-22T15:27:27.042 INFO:teuthology.orchestra.run.smithi062.stdout: "name": { 2023-05-22T15:27:27.043 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "client", 2023-05-22T15:27:27.043 INFO:teuthology.orchestra.run.smithi062.stdout: "num": 4583 2023-05-22T15:27:27.043 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2023-05-22T15:27:27.043 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": { 2023-05-22T15:27:27.043 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "v1", 2023-05-22T15:27:27.043 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "192.168.0.1:0", 2023-05-22T15:27:27.043 INFO:teuthology.orchestra.run.smithi062.stdout: "nonce": 4176519478 2023-05-22T15:27:27.044 INFO:teuthology.orchestra.run.smithi062.stdout: } 2023-05-22T15:27:27.044 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2023-05-22T15:27:27.044 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": { 2023-05-22T15:27:27.044 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "v1", 2023-05-22T15:27:27.044 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "192.168.0.1:0", 2023-05-22T15:27:27.044 INFO:teuthology.orchestra.run.smithi062.stdout: "nonce": 4176519478 2023-05-22T15:27:27.044 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2023-05-22T15:27:27.045 INFO:teuthology.orchestra.run.smithi062.stdout: "inst_str": "client.4583 192.168.0.1:0/4176519478", 2023-05-22T15:27:27.045 INFO:teuthology.orchestra.run.smithi062.stdout: "addr_str": "192.168.0.1:0/4176519478", 2023-05-22T15:27:27.045 INFO:teuthology.orchestra.run.smithi062.stdout: "inode_count": 1, 2023-05-22T15:27:27.045 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_epoch": 8, 2023-05-22T15:27:27.045 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_epoch": 27, 2023-05-22T15:27:27.045 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_epoch_barrier": 0, 2023-05-22T15:27:27.045 INFO:teuthology.orchestra.run.smithi062.stdout: "blocklisted": false, 2023-05-22T15:27:27.046 INFO:teuthology.orchestra.run.smithi062.stdout: "fs_name": "cephfs" 2023-05-22T15:27:27.046 INFO:teuthology.orchestra.run.smithi062.stdout:} 2023-05-22T15:27:27.054 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-05-22T15:27:27.190 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:27.183+0000 7f846f2e3640 1 Processor -- start 2023-05-22T15:27:27.190 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:27.183+0000 7f846f2e3640 1 -- start start 2023-05-22T15:27:27.191 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:27.183+0000 7f846f2e3640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f846806ee50 0x7f846806db00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:27.192 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:27.183+0000 7f846f2e3640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f846806e040 con 0x7f846815cfa0 2023-05-22T15:27:27.192 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:27.183+0000 7f846f2e3640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f846806e490 con 0x7f84680cf870 2023-05-22T15:27:27.192 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:27.183+0000 7f846f2e3640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84680cb9f0 con 0x7f846806ee50 2023-05-22T15:27:27.192 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:27.183+0000 7f846d880640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f846806ee50 0x7f846806db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:27.192 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:27.183+0000 7f846d07f640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f846815cfa0 0x7f84680b1fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:54834/0 (socket says 172.21.15.16:54834) 2023-05-22T15:27:27.192 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:27.183+0000 7f846d07f640 1 -- 172.21.15.16:0/282533957 learned_addr learned my addr 172.21.15.16:0/282533957 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:27:27.193 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:27.183+0000 7f845ffff640 1 -- 172.21.15.16:0/282533957 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3152082829 0 0) 0x7f8464001120 con 0x7f846815cfa0 2023-05-22T15:27:27.193 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:27.183+0000 7f845ffff640 1 -- 172.21.15.16:0/282533957 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8454003530 con 0x7f846815cfa0 2023-05-22T15:27:27.193 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:27.183+0000 7f846d880640 1 -- 172.21.15.16:0/282533957 >> v1:172.21.15.62:6789/0 conn(0x7f84680cf870 legacy=0x7f846806e740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:27.193 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:27.183+0000 7f846d880640 1 -- 172.21.15.16:0/282533957 >> v1:172.21.15.16:6789/0 conn(0x7f846815cfa0 legacy=0x7f84680b1fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:27.193 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:27.183+0000 7f846d880640 1 -- 172.21.15.16:0/282533957 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84680cad70 con 0x7f846806ee50 2023-05-22T15:27:27.194 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:27.187+0000 7f846d880640 1 --2- 172.21.15.16:0/282533957 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f846806ee50 0x7f846806db00 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f8460009d90 tx=0x7f846001c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38dd4861ac04608 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:27.194 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:27.187+0000 7f845ffff640 1 -- 172.21.15.16:0/282533957 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84600a59a0 con 0x7f846806ee50 2023-05-22T15:27:27.194 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:27.187+0000 7f845ffff640 1 -- 172.21.15.16:0/282533957 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84600a5b00 con 0x7f846806ee50 2023-05-22T15:27:27.195 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:27.187+0000 7f845ffff640 1 -- 172.21.15.16:0/282533957 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84600a5e90 con 0x7f846806ee50 2023-05-22T15:27:27.195 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:27.187+0000 7f846f2e3640 1 -- 172.21.15.16:0/282533957 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f846806ee50 msgr2=0x7f846806db00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:27.195 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:27.187+0000 7f846f2e3640 1 --2- 172.21.15.16:0/282533957 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f846806ee50 0x7f846806db00 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f8460009d90 tx=0x7f846001c8e0 comp rx=0 tx=0).stop 2023-05-22T15:27:27.195 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:27.187+0000 7f846f2e3640 1 -- 172.21.15.16:0/282533957 shutdown_connections 2023-05-22T15:27:27.196 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:27.187+0000 7f846f2e3640 1 --2- 172.21.15.16:0/282533957 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f846806ee50 0x7f846806db00 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:27.196 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:27.187+0000 7f846f2e3640 1 -- 172.21.15.16:0/282533957 >> 172.21.15.16:0/282533957 conn(0x7f84680cc870 msgr2=0x7f84680cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:27.196 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:27.187+0000 7f846f2e3640 1 -- 172.21.15.16:0/282533957 shutdown_connections 2023-05-22T15:27:27.196 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:27.187+0000 7f846f2e3640 1 -- 172.21.15.16:0/282533957 wait complete. 2023-05-22T15:27:27.197 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:27.187+0000 7f846f2e3640 1 Processor -- start 2023-05-22T15:27:27.197 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:27.187+0000 7f846f2e3640 1 -- start start 2023-05-22T15:27:27.197 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:27.187+0000 7f846f2e3640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f846806ee50 0x7f84680c8300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:27.197 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:27.187+0000 7f846f2e3640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f84680cf870 0x7f8468149600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:27.197 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:27.187+0000 7f846f2e3640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f846815cfa0 0x7f846814d490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:27.198 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:27.187+0000 7f846f2e3640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84680c90d0 con 0x7f846815cfa0 2023-05-22T15:27:27.198 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:27.187+0000 7f846f2e3640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84680c8840 con 0x7f846806ee50 2023-05-22T15:27:27.198 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:27.187+0000 7f846f2e3640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84680c8b10 con 0x7f84680cf870 2023-05-22T15:27:27.198 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:27.187+0000 7f846d880640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f846815cfa0 0x7f846814d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:27.198 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:27.187+0000 7f846d880640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f846815cfa0 0x7f846814d490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56580/0 (socket says 172.21.15.16:56580) 2023-05-22T15:27:27.198 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:27.187+0000 7f846d880640 1 -- 172.21.15.16:0/2263382146 learned_addr learned my addr 172.21.15.16:0/2263382146 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:27.199 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:27.191+0000 7f846c87e640 1 --2- 172.21.15.16:0/2263382146 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f84680cf870 0x7f8468149600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:27.200 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:27.191+0000 7f846d880640 1 -- 172.21.15.16:0/2263382146 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f84680cf870 msgr2=0x7f8468149600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:27.200 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:27.191+0000 7f846d880640 1 --2- 172.21.15.16:0/2263382146 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f84680cf870 0x7f8468149600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:27.200 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:27.191+0000 7f846d880640 1 -- 172.21.15.16:0/2263382146 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f846806ee50 msgr2=0x7f84680c8300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:27.200 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:27.191+0000 7f846d880640 1 --2- 172.21.15.16:0/2263382146 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f846806ee50 0x7f84680c8300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:27.200 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:27.191+0000 7f846d880640 1 -- 172.21.15.16:0/2263382146 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8460009a70 con 0x7f846815cfa0 2023-05-22T15:27:27.201 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:27.191+0000 7f846d880640 1 --2- 172.21.15.16:0/2263382146 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f846815cfa0 0x7f846814d490 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f846001a510 tx=0x7f8460019680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:27.201 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:27.191+0000 7f845dffb640 1 -- 172.21.15.16:0/2263382146 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84600add90 con 0x7f846815cfa0 2023-05-22T15:27:27.201 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:27.191+0000 7f845dffb640 1 -- 172.21.15.16:0/2263382146 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8460009450 con 0x7f846815cfa0 2023-05-22T15:27:27.201 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:27.191+0000 7f846f2e3640 1 -- 172.21.15.16:0/2263382146 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f846815d370 con 0x7f846815cfa0 2023-05-22T15:27:27.201 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:27.191+0000 7f845dffb640 1 -- 172.21.15.16:0/2263382146 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84600b9670 con 0x7f846815cfa0 2023-05-22T15:27:27.201 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:27.191+0000 7f846f2e3640 1 -- 172.21.15.16:0/2263382146 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84680cb610 con 0x7f846815cfa0 2023-05-22T15:27:27.201 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:27.191+0000 7f846f2e3640 1 -- 172.21.15.16:0/2263382146 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f846815d500 con 0x7f846815cfa0 2023-05-22T15:27:27.201 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:27.191+0000 7f845dffb640 1 -- 172.21.15.16:0/2263382146 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7f84600b97d0 con 0x7f846815cfa0 2023-05-22T15:27:27.202 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:27.191+0000 7f845dffb640 1 --2- 172.21.15.16:0/2263382146 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f844c065380 0x7f844c067830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:27.202 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:27.191+0000 7f845dffb640 1 -- 172.21.15.16:0/2263382146 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f846011e300 con 0x7f846815cfa0 2023-05-22T15:27:27.202 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:27.191+0000 7f846d07f640 1 --2- 172.21.15.16:0/2263382146 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f844c065380 0x7f844c067830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:27.202 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:27.191+0000 7f846d07f640 1 --2- 172.21.15.16:0/2263382146 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f844c065380 0x7f844c067830 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f8464005890 tx=0x7f8464001160 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:27.204 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:27.195+0000 7f845dffb640 1 -- 172.21.15.16:0/2263382146 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f84600ebc90 con 0x7f846815cfa0 2023-05-22T15:27:27.392 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:27.383+0000 7f846f2e3640 1 -- 172.21.15.16:0/2263382146 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f846815d500 con 0x7f846815cfa0 2023-05-22T15:27:27.393 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:27.383+0000 7f845dffb640 1 -- 172.21.15.16:0/2263382146 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) v1 ==== 53+0+88 (secure 0 0 0) 0x7f84600ef2b0 con 0x7f846815cfa0 2023-05-22T15:27:27.393 INFO:teuthology.orchestra.run.smithi016.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2023-05-22T15:27:27.395 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:27.387+0000 7f846f2e3640 1 -- 172.21.15.16:0/2263382146 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f844c065380 msgr2=0x7f844c067830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:27.395 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:27.387+0000 7f846f2e3640 1 --2- 172.21.15.16:0/2263382146 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f844c065380 0x7f844c067830 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f8464005890 tx=0x7f8464001160 comp rx=0 tx=0).stop 2023-05-22T15:27:27.396 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:27.387+0000 7f846f2e3640 1 -- 172.21.15.16:0/2263382146 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f846815cfa0 msgr2=0x7f846814d490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:27.396 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:27.387+0000 7f846f2e3640 1 --2- 172.21.15.16:0/2263382146 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f846815cfa0 0x7f846814d490 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f846001a510 tx=0x7f8460019680 comp rx=0 tx=0).stop 2023-05-22T15:27:27.396 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:27.387+0000 7f846f2e3640 1 -- 172.21.15.16:0/2263382146 shutdown_connections 2023-05-22T15:27:27.396 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:27.387+0000 7f846f2e3640 1 --2- 172.21.15.16:0/2263382146 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f846815cfa0 0x7f846814d490 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:27.396 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:27.387+0000 7f846f2e3640 1 --2- 172.21.15.16:0/2263382146 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f844c065380 0x7f844c067830 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:27.396 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:27.387+0000 7f846f2e3640 1 --2- 172.21.15.16:0/2263382146 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f84680cf870 0x7f8468149600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:27.396 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:27.387+0000 7f846f2e3640 1 --2- 172.21.15.16:0/2263382146 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f846806ee50 0x7f84680c8300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:27.397 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:27.387+0000 7f846f2e3640 1 -- 172.21.15.16:0/2263382146 >> 172.21.15.16:0/2263382146 conn(0x7f84680cc870 msgr2=0x7f846815b5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:27.397 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:27.387+0000 7f846f2e3640 1 -- 172.21.15.16:0/2263382146 shutdown_connections 2023-05-22T15:27:27.397 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:27.387+0000 7f846f2e3640 1 -- 172.21.15.16:0/2263382146 wait complete. 2023-05-22T15:27:27.412 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-05-22T15:27:27.413 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-05-22T15:27:27.413 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi062.front.sepia.ceph.com 2023-05-22T15:27:27.413 INFO:tasks.cephfs.mount:self.client.name = client.3 2023-05-22T15:27:27.414 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2023-05-22T15:27:27.414 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-05-22T15:27:27.414 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-05-22T15:27:27.414 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-05-22T15:27:27.414 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.3' 2023-05-22T15:27:27.415 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:27:27.415 DEBUG:teuthology.orchestra.run.smithi062:> ip addr 2023-05-22T15:27:27.423 INFO:teuthology.orchestra.run.smithi062.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-05-22T15:27:27.424 INFO:teuthology.orchestra.run.smithi062.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-05-22T15:27:27.424 INFO:teuthology.orchestra.run.smithi062.stdout: inet 127.0.0.1/8 scope host lo 2023-05-22T15:27:27.424 INFO:teuthology.orchestra.run.smithi062.stdout: valid_lft forever preferred_lft forever 2023-05-22T15:27:27.424 INFO:teuthology.orchestra.run.smithi062.stdout: inet6 ::1/128 scope host 2023-05-22T15:27:27.424 INFO:teuthology.orchestra.run.smithi062.stdout: valid_lft forever preferred_lft forever 2023-05-22T15:27:27.424 INFO:teuthology.orchestra.run.smithi062.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-05-22T15:27:27.425 INFO:teuthology.orchestra.run.smithi062.stdout: link/ether 0c:c4:7a:a9:aa:14 brd ff:ff:ff:ff:ff:ff 2023-05-22T15:27:27.425 INFO:teuthology.orchestra.run.smithi062.stdout: altname enp7s0f0 2023-05-22T15:27:27.425 INFO:teuthology.orchestra.run.smithi062.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-05-22T15:27:27.425 INFO:teuthology.orchestra.run.smithi062.stdout: link/ether 0c:c4:7a:a9:aa:15 brd ff:ff:ff:ff:ff:ff 2023-05-22T15:27:27.425 INFO:teuthology.orchestra.run.smithi062.stdout: altname enp7s0f1 2023-05-22T15:27:27.425 INFO:teuthology.orchestra.run.smithi062.stdout:4: enp3s0f0: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-05-22T15:27:27.425 INFO:teuthology.orchestra.run.smithi062.stdout: link/ether 0c:c4:7a:bb:d1:c2 brd ff:ff:ff:ff:ff:ff 2023-05-22T15:27:27.426 INFO:teuthology.orchestra.run.smithi062.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-05-22T15:27:27.426 INFO:teuthology.orchestra.run.smithi062.stdout: link/ether 0c:c4:7a:bb:d1:c3 brd ff:ff:ff:ff:ff:ff 2023-05-22T15:27:27.426 INFO:teuthology.orchestra.run.smithi062.stdout: inet 172.21.15.62/20 metric 100 brd 172.21.15.255 scope global dynamic enp3s0f1 2023-05-22T15:27:27.426 INFO:teuthology.orchestra.run.smithi062.stdout: valid_lft 42473sec preferred_lft 42473sec 2023-05-22T15:27:27.426 INFO:teuthology.orchestra.run.smithi062.stdout: inet6 fe80::ec4:7aff:febb:d1c3/64 scope link 2023-05-22T15:27:27.426 INFO:teuthology.orchestra.run.smithi062.stdout: valid_lft forever preferred_lft forever 2023-05-22T15:27:27.426 INFO:teuthology.orchestra.run.smithi062.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2023-05-22T15:27:27.426 INFO:teuthology.orchestra.run.smithi062.stdout: link/ether 02:42:2f:a3:54:48 brd ff:ff:ff:ff:ff:ff 2023-05-22T15:27:27.427 INFO:teuthology.orchestra.run.smithi062.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2023-05-22T15:27:27.427 INFO:teuthology.orchestra.run.smithi062.stdout: valid_lft forever preferred_lft forever 2023-05-22T15:27:27.427 INFO:teuthology.orchestra.run.smithi062.stdout:7: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-05-22T15:27:27.427 INFO:teuthology.orchestra.run.smithi062.stdout: link/ether 5e:1f:84:6d:79:ae brd ff:ff:ff:ff:ff:ff 2023-05-22T15:27:27.427 INFO:teuthology.orchestra.run.smithi062.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-05-22T15:27:27.427 INFO:teuthology.orchestra.run.smithi062.stdout: valid_lft forever preferred_lft forever 2023-05-22T15:27:27.428 INFO:teuthology.orchestra.run.smithi062.stdout: inet6 fe80::5c1f:84ff:fe6d:79ae/64 scope link 2023-05-22T15:27:27.428 INFO:teuthology.orchestra.run.smithi062.stdout: valid_lft forever preferred_lft forever 2023-05-22T15:27:27.428 INFO:teuthology.orchestra.run.smithi062.stdout:8: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-05-22T15:27:27.428 INFO:teuthology.orchestra.run.smithi062.stdout: link/ether 0e:5c:f6:1f:82:2f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2023-05-22T15:27:27.428 INFO:teuthology.orchestra.run.smithi062.stdout: inet6 fe80::c5c:f6ff:fe1f:822f/64 scope link 2023-05-22T15:27:27.428 INFO:teuthology.orchestra.run.smithi062.stdout: valid_lft forever preferred_lft forever 2023-05-22T15:27:27.429 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:27:27.429 DEBUG:teuthology.orchestra.run.smithi062:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-05-22T15:27:27.481 INFO:teuthology.orchestra.run.smithi062.stdout:1 2023-05-22T15:27:27.483 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:27:27.483 DEBUG:teuthology.orchestra.run.smithi062:> route 2023-05-22T15:27:27.534 INFO:teuthology.orchestra.run.smithi062.stdout:Kernel IP routing table 2023-05-22T15:27:27.534 INFO:teuthology.orchestra.run.smithi062.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-05-22T15:27:27.535 INFO:teuthology.orchestra.run.smithi062.stdout:default _gateway 0.0.0.0 UG 0 0 0 enp3s0f1 2023-05-22T15:27:27.535 INFO:teuthology.orchestra.run.smithi062.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-05-22T15:27:27.535 INFO:teuthology.orchestra.run.smithi062.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2023-05-22T15:27:27.535 INFO:teuthology.orchestra.run.smithi062.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-05-22T15:27:27.535 INFO:teuthology.orchestra.run.smithi062.stdout:ns1.front.sepia 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2023-05-22T15:27:27.535 INFO:teuthology.orchestra.run.smithi062.stdout:ns2.front.sepia 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2023-05-22T15:27:27.536 INFO:teuthology.orchestra.run.smithi062.stdout:_gateway 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2023-05-22T15:27:27.536 INFO:teuthology.orchestra.run.smithi062.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-05-22T15:27:27.536 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:27:27.537 DEBUG:teuthology.orchestra.run.smithi062:> (cd / && exec bash -c ' 2023-05-22T15:27:27.538 DEBUG:teuthology.orchestra.run.smithi062:> set -e 2023-05-22T15:27:27.538 DEBUG:teuthology.orchestra.run.smithi062:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-05-22T15:27:27.539 DEBUG:teuthology.orchestra.run.smithi062:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-05-22T15:27:27.539 DEBUG:teuthology.orchestra.run.smithi062:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-05-22T15:27:27.540 DEBUG:teuthology.orchestra.run.smithi062:> ') 2023-05-22T15:27:27.612 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:27:27.612 DEBUG:teuthology.orchestra.run.smithi062:> ip netns list 2023-05-22T15:27:27.661 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2023-05-22T15:27:27.662 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:27:27.662 DEBUG:teuthology.orchestra.run.smithi062:> ip netns list-id 2023-05-22T15:27:27.709 INFO:teuthology.orchestra.run.smithi062.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.1) 2023-05-22T15:27:27.710 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:27:27.710 DEBUG:teuthology.orchestra.run.smithi062:> (cd / && exec bash -c ' 2023-05-22T15:27:27.710 DEBUG:teuthology.orchestra.run.smithi062:> set -e 2023-05-22T15:27:27.710 DEBUG:teuthology.orchestra.run.smithi062:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.3 2023-05-22T15:27:27.711 DEBUG:teuthology.orchestra.run.smithi062:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.3 1 2023-05-22T15:27:27.711 DEBUG:teuthology.orchestra.run.smithi062:> ') 2023-05-22T15:27:27.780 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:27:27.781 DEBUG:teuthology.orchestra.run.smithi062:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2023-05-22T15:27:27.841 INFO:teuthology.orchestra.run.smithi062.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-05-22T15:27:27.842 INFO:teuthology.orchestra.run.smithi062.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-05-22T15:27:27.842 INFO:teuthology.orchestra.run.smithi062.stdout: inet 127.0.0.1/8 scope host lo 2023-05-22T15:27:27.842 INFO:teuthology.orchestra.run.smithi062.stdout: valid_lft forever preferred_lft forever 2023-05-22T15:27:27.842 INFO:teuthology.orchestra.run.smithi062.stdout: inet6 ::1/128 scope host 2023-05-22T15:27:27.842 INFO:teuthology.orchestra.run.smithi062.stdout: valid_lft forever preferred_lft forever 2023-05-22T15:27:27.843 INFO:teuthology.orchestra.run.smithi062.stdout:2: veth0@if8: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-05-22T15:27:27.843 INFO:teuthology.orchestra.run.smithi062.stdout: link/ether be:3f:d4:e6:7f:c0 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2023-05-22T15:27:27.843 INFO:teuthology.orchestra.run.smithi062.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2023-05-22T15:27:27.843 INFO:teuthology.orchestra.run.smithi062.stdout: valid_lft forever preferred_lft forever 2023-05-22T15:27:27.843 INFO:teuthology.orchestra.run.smithi062.stdout: inet6 fe80::bc3f:d4ff:fee6:7fc0/64 scope link 2023-05-22T15:27:27.843 INFO:teuthology.orchestra.run.smithi062.stdout: valid_lft forever preferred_lft forever 2023-05-22T15:27:27.844 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:27:27.844 DEBUG:teuthology.orchestra.run.smithi062:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2023-05-22T15:27:27.902 INFO:teuthology.orchestra.run.smithi062.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-05-22T15:27:27.902 INFO:teuthology.orchestra.run.smithi062.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-05-22T15:27:27.902 INFO:teuthology.orchestra.run.smithi062.stdout: inet 127.0.0.1/8 scope host lo 2023-05-22T15:27:27.902 INFO:teuthology.orchestra.run.smithi062.stdout: valid_lft forever preferred_lft forever 2023-05-22T15:27:27.902 INFO:teuthology.orchestra.run.smithi062.stdout: inet6 ::1/128 scope host 2023-05-22T15:27:27.903 INFO:teuthology.orchestra.run.smithi062.stdout: valid_lft forever preferred_lft forever 2023-05-22T15:27:27.903 INFO:teuthology.orchestra.run.smithi062.stdout:2: veth0@if8: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-05-22T15:27:27.903 INFO:teuthology.orchestra.run.smithi062.stdout: link/ether be:3f:d4:e6:7f:c0 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2023-05-22T15:27:27.903 INFO:teuthology.orchestra.run.smithi062.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2023-05-22T15:27:27.903 INFO:teuthology.orchestra.run.smithi062.stdout: valid_lft forever preferred_lft forever 2023-05-22T15:27:27.903 INFO:teuthology.orchestra.run.smithi062.stdout: inet6 fe80::bc3f:d4ff:fee6:7fc0/64 scope link 2023-05-22T15:27:27.903 INFO:teuthology.orchestra.run.smithi062.stdout: valid_lft forever preferred_lft forever 2023-05-22T15:27:27.904 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.3' with 192.168.0.2/16 2023-05-22T15:27:27.905 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:27:27.905 DEBUG:teuthology.orchestra.run.smithi062:> (cd / && exec bash -c ' 2023-05-22T15:27:27.905 DEBUG:teuthology.orchestra.run.smithi062:> set -e 2023-05-22T15:27:27.905 DEBUG:teuthology.orchestra.run.smithi062:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.3 type veth peer name brx.1 2023-05-22T15:27:27.906 DEBUG:teuthology.orchestra.run.smithi062:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2023-05-22T15:27:27.906 DEBUG:teuthology.orchestra.run.smithi062:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set veth0 up 2023-05-22T15:27:27.906 DEBUG:teuthology.orchestra.run.smithi062:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set lo up 2023-05-22T15:27:27.906 DEBUG:teuthology.orchestra.run.smithi062:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip route add default via 192.168.255.254 2023-05-22T15:27:27.906 DEBUG:teuthology.orchestra.run.smithi062:> ') 2023-05-22T15:27:28.002 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:27:28.003 DEBUG:teuthology.orchestra.run.smithi062:> (cd / && exec bash -c ' 2023-05-22T15:27:28.003 DEBUG:teuthology.orchestra.run.smithi062:> set -e 2023-05-22T15:27:28.003 DEBUG:teuthology.orchestra.run.smithi062:> sudo ip link set brx.1 up 2023-05-22T15:27:28.003 DEBUG:teuthology.orchestra.run.smithi062:> sudo ip link set dev brx.1 master ceph-brx 2023-05-22T15:27:28.004 DEBUG:teuthology.orchestra.run.smithi062:> ') 2023-05-22T15:27:28.023 INFO:tasks.cephfs.fuse_mount:Client client.3 config is {} 2023-05-22T15:27:28.023 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-05-22T15:27:28.023 DEBUG:teuthology.orchestra.run.smithi062:> mkdir -p -v /home/ubuntu/cephtest/mnt.3 2023-05-22T15:27:28.072 INFO:teuthology.orchestra.run.smithi062.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.3' 2023-05-22T15:27:28.072 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-05-22T15:27:28.073 DEBUG:teuthology.orchestra.run.smithi062:> chmod 0000 /home/ubuntu/cephtest/mnt.3 2023-05-22T15:27:28.120 DEBUG:teuthology.orchestra.run.smithi062:> sudo modprobe fuse 2023-05-22T15:27:28.182 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-05-22T15:27:28.182 DEBUG:teuthology.orchestra.run.smithi062:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-05-22T15:27:28.241 INFO:teuthology.orchestra.run.smithi062.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-05-22T15:27:28.242 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-05-22T15:27:28.243 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:27:28.244 DEBUG:teuthology.orchestra.run.smithi062:> ls /sys/fs/fuse/connections 2023-05-22T15:27:28.289 INFO:teuthology.orchestra.run.smithi062.stdout:45 2023-05-22T15:27:28.289 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [45] 2023-05-22T15:27:28.290 DEBUG:teuthology.orchestra.run.smithi062:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.3 --id 3) 2023-05-22T15:27:28.334 DEBUG:teuthology.orchestra.run.smithi062:> sudo modprobe fuse 2023-05-22T15:27:28.351 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-05-22T15:27:28.351 DEBUG:teuthology.orchestra.run.smithi062:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-05-22T15:27:28.407 INFO:teuthology.orchestra.run.smithi062.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-05-22T15:27:28.408 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-05-22T15:27:28.408 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:27:28.409 DEBUG:teuthology.orchestra.run.smithi062:> ls /sys/fs/fuse/connections 2023-05-22T15:27:28.413 INFO:tasks.cephfs.fuse_mount.ceph-fuse.3.smithi062.stderr:2023-05-22T15:27:28.406+0000 7f5f3c390580 -1 init, newargv = 0x55cab5fe7c30 newargc=15 2023-05-22T15:27:28.414 INFO:tasks.cephfs.fuse_mount.ceph-fuse.3.smithi062.stderr:ceph-fuse[43406]: starting ceph client 2023-05-22T15:27:28.417 INFO:teuthology.orchestra.run.smithi062.stdout:45 2023-05-22T15:27:28.419 INFO:tasks.cephfs.fuse_mount.ceph-fuse.3.smithi062.stderr:ceph-fuse[43406]: starting fuse 2023-05-22T15:27:29.520 DEBUG:teuthology.orchestra.run.smithi062:> sudo modprobe fuse 2023-05-22T15:27:29.536 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-05-22T15:27:29.536 DEBUG:teuthology.orchestra.run.smithi062:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-05-22T15:27:29.588 INFO:teuthology.orchestra.run.smithi062.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-05-22T15:27:29.589 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-05-22T15:27:29.590 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:27:29.590 DEBUG:teuthology.orchestra.run.smithi062:> ls /sys/fs/fuse/connections 2023-05-22T15:27:29.637 INFO:teuthology.orchestra.run.smithi062.stdout:45 2023-05-22T15:27:29.638 INFO:teuthology.orchestra.run.smithi062.stdout:46 2023-05-22T15:27:29.638 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [45, 46] 2023-05-22T15:27:29.638 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2023-05-22T15:27:29.638 DEBUG:teuthology.orchestra.run.smithi062:> import glob 2023-05-22T15:27:29.639 DEBUG:teuthology.orchestra.run.smithi062:> import re 2023-05-22T15:27:29.639 DEBUG:teuthology.orchestra.run.smithi062:> import os 2023-05-22T15:27:29.639 DEBUG:teuthology.orchestra.run.smithi062:> import subprocess 2023-05-22T15:27:29.639 DEBUG:teuthology.orchestra.run.smithi062:> 2023-05-22T15:27:29.640 DEBUG:teuthology.orchestra.run.smithi062:> def _find_admin_socket(client_name): 2023-05-22T15:27:29.640 DEBUG:teuthology.orchestra.run.smithi062:> asok_path = "/var/run/ceph/ceph-client.3.*.asok" 2023-05-22T15:27:29.640 DEBUG:teuthology.orchestra.run.smithi062:> files = glob.glob(asok_path) 2023-05-22T15:27:29.640 DEBUG:teuthology.orchestra.run.smithi062:> mountpoint = "/home/ubuntu/cephtest/mnt.3" 2023-05-22T15:27:29.641 DEBUG:teuthology.orchestra.run.smithi062:> 2023-05-22T15:27:29.641 DEBUG:teuthology.orchestra.run.smithi062:> # Given a non-glob path, it better be there 2023-05-22T15:27:29.641 DEBUG:teuthology.orchestra.run.smithi062:> if "*" not in asok_path: 2023-05-22T15:27:29.641 DEBUG:teuthology.orchestra.run.smithi062:> assert(len(files) == 1) 2023-05-22T15:27:29.642 DEBUG:teuthology.orchestra.run.smithi062:> return files[0] 2023-05-22T15:27:29.642 DEBUG:teuthology.orchestra.run.smithi062:> 2023-05-22T15:27:29.642 DEBUG:teuthology.orchestra.run.smithi062:> for f in files: 2023-05-22T15:27:29.642 DEBUG:teuthology.orchestra.run.smithi062:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-05-22T15:27:29.642 DEBUG:teuthology.orchestra.run.smithi062:> if os.path.exists("/proc/{0}".format(pid)): 2023-05-22T15:27:29.643 DEBUG:teuthology.orchestra.run.smithi062:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-05-22T15:27:29.643 DEBUG:teuthology.orchestra.run.smithi062:> contents = proc_f.read() 2023-05-22T15:27:29.643 DEBUG:teuthology.orchestra.run.smithi062:> if mountpoint in contents: 2023-05-22T15:27:29.643 DEBUG:teuthology.orchestra.run.smithi062:> return f 2023-05-22T15:27:29.644 DEBUG:teuthology.orchestra.run.smithi062:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-05-22T15:27:29.644 DEBUG:teuthology.orchestra.run.smithi062:> 2023-05-22T15:27:29.644 DEBUG:teuthology.orchestra.run.smithi062:> print(_find_admin_socket("client.3")) 2023-05-22T15:27:29.645 DEBUG:teuthology.orchestra.run.smithi062:> ' 2023-05-22T15:27:29.761 INFO:teuthology.orchestra.run.smithi062.stdout:/var/run/ceph/ceph-client.3.43406.asok 2023-05-22T15:27:29.942 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.3.43406.asok 2023-05-22T15:27:29.943 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:27:29.943 DEBUG:teuthology.orchestra.run.smithi062:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.3.43406.asok status 2023-05-22T15:27:30.104 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2023-05-22T15:27:30.104 INFO:teuthology.orchestra.run.smithi062.stdout: "metadata": { 2023-05-22T15:27:30.104 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph_sha1": "9c9ba4df22024d9d8d6cf9f98a4b96751fc78d0c", 2023-05-22T15:27:30.104 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph_version": "ceph version 18.0.0-3192-g9c9ba4df (9c9ba4df22024d9d8d6cf9f98a4b96751fc78d0c) reef (dev)", 2023-05-22T15:27:30.105 INFO:teuthology.orchestra.run.smithi062.stdout: "entity_id": "3", 2023-05-22T15:27:30.105 INFO:teuthology.orchestra.run.smithi062.stdout: "hostname": "smithi062", 2023-05-22T15:27:30.105 INFO:teuthology.orchestra.run.smithi062.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.3", 2023-05-22T15:27:30.105 INFO:teuthology.orchestra.run.smithi062.stdout: "pid": "43406", 2023-05-22T15:27:30.105 INFO:teuthology.orchestra.run.smithi062.stdout: "root": "/" 2023-05-22T15:27:30.105 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2023-05-22T15:27:30.105 INFO:teuthology.orchestra.run.smithi062.stdout: "dentry_count": 0, 2023-05-22T15:27:30.106 INFO:teuthology.orchestra.run.smithi062.stdout: "dentry_pinned_count": 0, 2023-05-22T15:27:30.106 INFO:teuthology.orchestra.run.smithi062.stdout: "id": 4732, 2023-05-22T15:27:30.106 INFO:teuthology.orchestra.run.smithi062.stdout: "inst": { 2023-05-22T15:27:30.106 INFO:teuthology.orchestra.run.smithi062.stdout: "name": { 2023-05-22T15:27:30.106 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "client", 2023-05-22T15:27:30.106 INFO:teuthology.orchestra.run.smithi062.stdout: "num": 4732 2023-05-22T15:27:30.106 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2023-05-22T15:27:30.107 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": { 2023-05-22T15:27:30.107 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "v1", 2023-05-22T15:27:30.107 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "192.168.0.2:0", 2023-05-22T15:27:30.107 INFO:teuthology.orchestra.run.smithi062.stdout: "nonce": 1833917186 2023-05-22T15:27:30.107 INFO:teuthology.orchestra.run.smithi062.stdout: } 2023-05-22T15:27:30.108 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2023-05-22T15:27:30.108 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": { 2023-05-22T15:27:30.108 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "v1", 2023-05-22T15:27:30.108 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "192.168.0.2:0", 2023-05-22T15:27:30.108 INFO:teuthology.orchestra.run.smithi062.stdout: "nonce": 1833917186 2023-05-22T15:27:30.108 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2023-05-22T15:27:30.108 INFO:teuthology.orchestra.run.smithi062.stdout: "inst_str": "client.4732 192.168.0.2:0/1833917186", 2023-05-22T15:27:30.109 INFO:teuthology.orchestra.run.smithi062.stdout: "addr_str": "192.168.0.2:0/1833917186", 2023-05-22T15:27:30.109 INFO:teuthology.orchestra.run.smithi062.stdout: "inode_count": 1, 2023-05-22T15:27:30.109 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_epoch": 8, 2023-05-22T15:27:30.109 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_epoch": 27, 2023-05-22T15:27:30.109 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_epoch_barrier": 0, 2023-05-22T15:27:30.109 INFO:teuthology.orchestra.run.smithi062.stdout: "blocklisted": false, 2023-05-22T15:27:30.110 INFO:teuthology.orchestra.run.smithi062.stdout: "fs_name": "cephfs" 2023-05-22T15:27:30.110 INFO:teuthology.orchestra.run.smithi062.stdout:} 2023-05-22T15:27:30.118 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:27:30.118 DEBUG:teuthology.orchestra.run.smithi016:> stat --file-system '--printf=%T 2023-05-22T15:27:30.119 DEBUG:teuthology.orchestra.run.smithi016:> ' -- /home/ubuntu/cephtest/mnt.0 2023-05-22T15:27:30.126 INFO:teuthology.orchestra.run.smithi016.stdout:fuseblk 2023-05-22T15:27:30.126 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2023-05-22T15:27:30.127 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:27:30.127 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-05-22T15:27:30.189 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:27:30.190 DEBUG:teuthology.orchestra.run.smithi016:> stat --file-system '--printf=%T 2023-05-22T15:27:30.190 DEBUG:teuthology.orchestra.run.smithi016:> ' -- /home/ubuntu/cephtest/mnt.2 2023-05-22T15:27:30.239 INFO:teuthology.orchestra.run.smithi016.stdout:fuseblk 2023-05-22T15:27:30.239 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.2 2023-05-22T15:27:30.240 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:27:30.240 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.2 2023-05-22T15:27:30.299 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:27:30.300 DEBUG:teuthology.orchestra.run.smithi062:> stat --file-system '--printf=%T 2023-05-22T15:27:30.300 DEBUG:teuthology.orchestra.run.smithi062:> ' -- /home/ubuntu/cephtest/mnt.1 2023-05-22T15:27:30.309 INFO:teuthology.orchestra.run.smithi062.stdout:fuseblk 2023-05-22T15:27:30.309 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2023-05-22T15:27:30.310 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:27:30.311 DEBUG:teuthology.orchestra.run.smithi062:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2023-05-22T15:27:30.366 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:27:30.368 DEBUG:teuthology.orchestra.run.smithi062:> stat --file-system '--printf=%T 2023-05-22T15:27:30.368 DEBUG:teuthology.orchestra.run.smithi062:> ' -- /home/ubuntu/cephtest/mnt.3 2023-05-22T15:27:30.416 INFO:teuthology.orchestra.run.smithi062.stdout:fuseblk 2023-05-22T15:27:30.416 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.3 2023-05-22T15:27:30.416 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:27:30.417 DEBUG:teuthology.orchestra.run.smithi062:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.3 2023-05-22T15:27:30.471 INFO:teuthology.run_tasks:Running task cephfs_test_runner... 2023-05-22T15:27:30.491 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-05-22T15:27:30.625 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:30.619+0000 7faabe5fa640 1 Processor -- start 2023-05-22T15:27:30.626 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:30.619+0000 7faabe5fa640 1 -- start start 2023-05-22T15:27:30.627 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:30.619+0000 7faabe5fa640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7faab81531b0 0x7faab80afb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:30.627 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:30.619+0000 7faabe5fa640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7faab80b0080 0x7faab8159330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:30.627 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:30.619+0000 7faabe5fa640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7faab8159870 0x7faab80c8ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:30.627 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:30.619+0000 7faabe5fa640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faab80b0520 con 0x7faab80b0080 2023-05-22T15:27:30.627 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:30.619+0000 7faabe5fa640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faab815a0a0 con 0x7faab81531b0 2023-05-22T15:27:30.627 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:30.619+0000 7faabe5fa640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faab8153580 con 0x7faab8159870 2023-05-22T15:27:30.628 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:30.619+0000 7faab77fe640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7faab80b0080 0x7faab8159330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:30.628 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:30.619+0000 7faab77fe640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7faab80b0080 0x7faab8159330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57218/0 (socket says 172.21.15.16:57218) 2023-05-22T15:27:30.628 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:30.619+0000 7faab77fe640 1 -- 172.21.15.16:0/1320017148 learned_addr learned my addr 172.21.15.16:0/1320017148 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:30.629 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:30.619+0000 7faabcb97640 1 --2- 172.21.15.16:0/1320017148 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7faab8159870 0x7faab80c8ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:30.629 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:30.619+0000 7faab7fff640 1 --2- 172.21.15.16:0/1320017148 >> v2:172.21.15.62:3300/0 conn(0x7faab81531b0 0x7faab80afb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:30.629 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:30.619+0000 7faab77fe640 1 -- 172.21.15.16:0/1320017148 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7faab8159870 msgr2=0x7faab80c8ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:30.629 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:30.619+0000 7faab77fe640 1 --2- 172.21.15.16:0/1320017148 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7faab8159870 0x7faab80c8ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:30.629 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:30.619+0000 7faab77fe640 1 -- 172.21.15.16:0/1320017148 >> v2:172.21.15.62:3300/0 conn(0x7faab81531b0 msgr2=0x7faab80afb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:30.629 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:30.619+0000 7faab77fe640 1 --2- 172.21.15.16:0/1320017148 >> v2:172.21.15.62:3300/0 conn(0x7faab81531b0 0x7faab80afb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:30.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:30.619+0000 7faab77fe640 1 -- 172.21.15.16:0/1320017148 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faab815d5e0 con 0x7faab80b0080 2023-05-22T15:27:30.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:30.623+0000 7faab77fe640 1 --2- 172.21.15.16:0/1320017148 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faab80b0080 0x7faab8159330 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7faab80b0790 tx=0x7faaac01c9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31b11283f9ab266 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:30.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:30.623+0000 7faab6ffd640 1 -- 172.21.15.16:0/1320017148 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faaac0a5820 con 0x7faab80b0080 2023-05-22T15:27:30.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:30.623+0000 7faab6ffd640 1 -- 172.21.15.16:0/1320017148 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faaac0a5980 con 0x7faab80b0080 2023-05-22T15:27:30.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:30.623+0000 7faab6ffd640 1 -- 172.21.15.16:0/1320017148 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faaac0a5d10 con 0x7faab80b0080 2023-05-22T15:27:30.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:30.623+0000 7faabe5fa640 1 -- 172.21.15.16:0/1320017148 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faab80b0080 msgr2=0x7faab8159330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:30.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:30.623+0000 7faabe5fa640 1 --2- 172.21.15.16:0/1320017148 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faab80b0080 0x7faab8159330 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7faab80b0790 tx=0x7faaac01c9a0 comp rx=0 tx=0).stop 2023-05-22T15:27:30.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:30.623+0000 7faabe5fa640 1 -- 172.21.15.16:0/1320017148 shutdown_connections 2023-05-22T15:27:30.632 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:30.623+0000 7faabe5fa640 1 --2- 172.21.15.16:0/1320017148 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faab80b0080 0x7faab8159330 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:30.632 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:30.623+0000 7faabe5fa640 1 --2- 172.21.15.16:0/1320017148 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7faab8159870 0x7faab80c8ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:30.632 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:30.623+0000 7faabe5fa640 1 --2- 172.21.15.16:0/1320017148 >> v2:172.21.15.62:3300/0 conn(0x7faab81531b0 0x7faab80afb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:30.632 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:30.623+0000 7faabe5fa640 1 -- 172.21.15.16:0/1320017148 >> 172.21.15.16:0/1320017148 conn(0x7faab8069720 msgr2=0x7faab8069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:30.632 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:30.623+0000 7faabe5fa640 1 -- 172.21.15.16:0/1320017148 shutdown_connections 2023-05-22T15:27:30.632 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:30.623+0000 7faabe5fa640 1 -- 172.21.15.16:0/1320017148 wait complete. 2023-05-22T15:27:30.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:30.623+0000 7faabe5fa640 1 Processor -- start 2023-05-22T15:27:30.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:30.623+0000 7faabe5fa640 1 -- start start 2023-05-22T15:27:30.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:30.623+0000 7faabe5fa640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faab80b0080 0x7faab80aeba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:30.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:30.623+0000 7faabe5fa640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faab81531b0 0x7faab80b2d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:30.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:30.623+0000 7faabe5fa640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7faab8159870 0x7faab80ce490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:30.635 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:30.623+0000 7faabe5fa640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faab80b3dd0 con 0x7faab81531b0 2023-05-22T15:27:30.635 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:30.623+0000 7faabe5fa640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faab814cb40 con 0x7faab80b0080 2023-05-22T15:27:30.635 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:30.623+0000 7faabe5fa640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faab80b3530 con 0x7faab8159870 2023-05-22T15:27:30.635 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:30.623+0000 7faab77fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faab81531b0 0x7faab80b2d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:30.635 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:30.623+0000 7faab77fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faab81531b0 0x7faab80b2d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57220/0 (socket says 172.21.15.16:57220) 2023-05-22T15:27:30.635 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:30.623+0000 7faab77fe640 1 -- 172.21.15.16:0/4045594897 learned_addr learned my addr 172.21.15.16:0/4045594897 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:30.635 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:30.623+0000 7faabcb97640 1 --2- 172.21.15.16:0/4045594897 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7faab8159870 0x7faab80ce490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:30.636 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:30.623+0000 7faab7fff640 1 --2- 172.21.15.16:0/4045594897 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faab80b0080 0x7faab80aeba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:30.636 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:30.623+0000 7faab77fe640 1 -- 172.21.15.16:0/4045594897 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7faab8159870 msgr2=0x7faab80ce490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:30.636 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:30.623+0000 7faab77fe640 1 --2- 172.21.15.16:0/4045594897 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7faab8159870 0x7faab80ce490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:30.636 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:30.623+0000 7faab77fe640 1 -- 172.21.15.16:0/4045594897 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faab80b0080 msgr2=0x7faab80aeba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:30.636 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:30.623+0000 7faab77fe640 1 --2- 172.21.15.16:0/4045594897 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faab80b0080 0x7faab80aeba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:30.636 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:30.623+0000 7faab77fe640 1 -- 172.21.15.16:0/4045594897 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faaac009b30 con 0x7faab81531b0 2023-05-22T15:27:30.637 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:30.627+0000 7faab77fe640 1 --2- 172.21.15.16:0/4045594897 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faab81531b0 0x7faab80b2d00 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7faaac0af040 tx=0x7faaac019740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:30.637 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:30.627+0000 7faab4ff9640 1 -- 172.21.15.16:0/4045594897 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faaac0b8670 con 0x7faab81531b0 2023-05-22T15:27:30.637 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:30.627+0000 7faab4ff9640 1 -- 172.21.15.16:0/4045594897 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faaac0b87d0 con 0x7faab81531b0 2023-05-22T15:27:30.637 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:30.627+0000 7faabe5fa640 1 -- 172.21.15.16:0/4045594897 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faab815d0c0 con 0x7faab81531b0 2023-05-22T15:27:30.637 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:30.627+0000 7faab4ff9640 1 -- 172.21.15.16:0/4045594897 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faaac0b8b80 con 0x7faab81531b0 2023-05-22T15:27:30.637 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:30.627+0000 7faabe5fa640 1 -- 172.21.15.16:0/4045594897 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faab815cde0 con 0x7faab81531b0 2023-05-22T15:27:30.638 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:30.627+0000 7faabe5fa640 1 -- 172.21.15.16:0/4045594897 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faab8159c40 con 0x7faab81531b0 2023-05-22T15:27:30.638 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:30.627+0000 7faab4ff9640 1 -- 172.21.15.16:0/4045594897 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7faaac0b8ce0 con 0x7faab81531b0 2023-05-22T15:27:30.638 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:30.627+0000 7faab4ff9640 1 --2- 172.21.15.16:0/4045594897 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7faaa4065380 0x7faaa4067830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:30.638 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:30.627+0000 7faab4ff9640 1 -- 172.21.15.16:0/4045594897 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6871+0+0 (secure 0 0 0) 0x7faaac11e220 con 0x7faab81531b0 2023-05-22T15:27:30.639 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:30.627+0000 7faab7fff640 1 --2- 172.21.15.16:0/4045594897 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7faaa4065380 0x7faaa4067830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:30.639 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:30.627+0000 7faab7fff640 1 --2- 172.21.15.16:0/4045594897 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7faaa4065380 0x7faaa4067830 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7faab80cf100 tx=0x7faaa8008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:30.639 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:30.631+0000 7faab4ff9640 1 -- 172.21.15.16:0/4045594897 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7faaac0ebc10 con 0x7faab81531b0 2023-05-22T15:27:30.794 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:30.787+0000 7faabe5fa640 1 -- 172.21.15.16:0/4045594897 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7faab81501c0 con 0x7faab81531b0 2023-05-22T15:27:30.795 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:30.787+0000 7faab4ff9640 1 -- 172.21.15.16:0/4045594897 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v28) v1 ==== 74+0+19700 (secure 0 0 0) 0x7faaac0ef230 con 0x7faab81531b0 2023-05-22T15:27:30.796 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:27:30.796 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":28,"fsid":"b2719834-7847-4d98-90a6-f65d57f85c49","created":"2023-05-22T15:26:25.259314+0000","modified":"2023-05-22T15:27:30.172079+0000","last_up_change":"2023-05-22T15:26:36.448524+0000","last_in_change":"2023-05-22T15:26:27.675668+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-05-22T15:26:38.019781+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-05-22T15:26:43.570514+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-05-22T15:26:47.879159+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-05-22T15:26:48.522130+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data_ec","create_time":"2023-05-22T15:26:51.616142+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"9e7ff854-b25f-4cbb-a5c2-1799d9c53d2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":37021},{"type":"v1","addr":"172.21.15.16:6825","nonce":37021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":37021},{"type":"v1","addr":"172.21.15.16:6827","nonce":37021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":37021},{"type":"v1","addr":"172.21.15.16:6831","nonce":37021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":37021},{"type":"v1","addr":"172.21.15.16:6829","nonce":37021}]},"public_addr":"172.21.15.16:6825/37021","cluster_addr":"172.21.15.16:6827/37021","heartbeat_back_addr":"172.21.15.16:6831/37021","heartbeat_front_addr":"172.21.15.16:6829/37021","state":["exists","up"]},{"osd":1,"uuid":"4702fbc9-a927-4565-aca5-41dc69cc8264","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":37008},{"type":"v1","addr":"172.21.15.16:6801","nonce":37008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":37008},{"type":"v1","addr":"172.21.15.16:6803","nonce":37008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":37008},{"type":"v1","addr":"172.21.15.16:6807","nonce":37008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":37008},{"type":"v1","addr":"172.21.15.16:6805","nonce":37008}]},"public_addr":"172.21.15.16:6801/37008","cluster_addr":"172.21.15.16:6803/37008","heartbeat_back_addr":"172.21.15.16:6807/37008","heartbeat_front_addr":"172.21.15.16:6805/37008","state":["exists","up"]},{"osd":2,"uuid":"1e7bba46-4eba-47ef-bb81-5e2d371638ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":37009},{"type":"v1","addr":"172.21.15.16:6809","nonce":37009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":37009},{"type":"v1","addr":"172.21.15.16:6811","nonce":37009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":37009},{"type":"v1","addr":"172.21.15.16:6815","nonce":37009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":37009},{"type":"v1","addr":"172.21.15.16:6813","nonce":37009}]},"public_addr":"172.21.15.16:6809/37009","cluster_addr":"172.21.15.16:6811/37009","heartbeat_back_addr":"172.21.15.16:6815/37009","heartbeat_front_addr":"172.21.15.16:6813/37009","state":["exists","up"]},{"osd":3,"uuid":"383cacc0-5a88-4904-990e-498ca59ef48a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":37020},{"type":"v1","addr":"172.21.15.16:6817","nonce":37020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":37020},{"type":"v1","addr":"172.21.15.16:6819","nonce":37020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":37020},{"type":"v1","addr":"172.21.15.16:6823","nonce":37020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":37020},{"type":"v1","addr":"172.21.15.16:6821","nonce":37020}]},"public_addr":"172.21.15.16:6817/37020","cluster_addr":"172.21.15.16:6819/37020","heartbeat_back_addr":"172.21.15.16:6823/37020","heartbeat_front_addr":"172.21.15.16:6821/37020","state":["exists","up"]},{"osd":4,"uuid":"970500ef-95b7-42c1-b080-eed4fae7013d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":37353},{"type":"v1","addr":"172.21.15.62:6817","nonce":37353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6819","nonce":37353},{"type":"v1","addr":"172.21.15.62:6822","nonce":37353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37353},{"type":"v1","addr":"172.21.15.62:6831","nonce":37353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37353},{"type":"v1","addr":"172.21.15.62:6826","nonce":37353}]},"public_addr":"172.21.15.62:6817/37353","cluster_addr":"172.21.15.62:6822/37353","heartbeat_back_addr":"172.21.15.62:6831/37353","heartbeat_front_addr":"172.21.15.62:6826/37353","state":["exists","up"]},{"osd":5,"uuid":"022d67db-d4d8-445b-b1b9-8b446a00f5c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":37352},{"type":"v1","addr":"172.21.15.62:6801","nonce":37352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":37352},{"type":"v1","addr":"172.21.15.62:6803","nonce":37352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":37352},{"type":"v1","addr":"172.21.15.62:6807","nonce":37352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":37352},{"type":"v1","addr":"172.21.15.62:6805","nonce":37352}]},"public_addr":"172.21.15.62:6801/37352","cluster_addr":"172.21.15.62:6803/37352","heartbeat_back_addr":"172.21.15.62:6807/37352","heartbeat_front_addr":"172.21.15.62:6805/37352","state":["exists","up"]},{"osd":6,"uuid":"42bd372b-6688-4a1b-8407-e16e8c266e33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":37354},{"type":"v1","addr":"172.21.15.62:6820","nonce":37354}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":37354},{"type":"v1","addr":"172.21.15.62:6823","nonce":37354}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6829","nonce":37354},{"type":"v1","addr":"172.21.15.62:6830","nonce":37354}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6825","nonce":37354},{"type":"v1","addr":"172.21.15.62:6827","nonce":37354}]},"public_addr":"172.21.15.62:6820/37354","cluster_addr":"172.21.15.62:6823/37354","heartbeat_back_addr":"172.21.15.62:6830/37354","heartbeat_front_addr":"172.21.15.62:6827/37354","state":["exists","up"]},{"osd":7,"uuid":"ac70b2a3-f41b-4bf2-b227-7f2cbb55426d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37355},{"type":"v1","addr":"172.21.15.62:6809","nonce":37355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37355},{"type":"v1","addr":"172.21.15.62:6811","nonce":37355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":37355},{"type":"v1","addr":"172.21.15.62:6815","nonce":37355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":37355},{"type":"v1","addr":"172.21.15.62:6813","nonce":37355}]},"public_addr":"172.21.15.62:6809/37355","cluster_addr":"172.21.15.62:6811/37355","heartbeat_back_addr":"172.21.15.62:6815/37355","heartbeat_front_addr":"172.21.15.62:6813/37355","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.368896+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.107273+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.276888+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.287502+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.124791+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.251937+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.255679+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.232797+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"4.23","mappings":[{"from":3,"to":2}]},{"pgid":"4.31","mappings":[{"from":1,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-05-22T15:27:30.797 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:30.787+0000 7faabe5fa640 1 -- 172.21.15.16:0/4045594897 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7faaa4065380 msgr2=0x7faaa4067830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:30.798 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:30.787+0000 7faabe5fa640 1 --2- 172.21.15.16:0/4045594897 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7faaa4065380 0x7faaa4067830 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7faab80cf100 tx=0x7faaa8008040 comp rx=0 tx=0).stop 2023-05-22T15:27:30.798 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:30.787+0000 7faabe5fa640 1 -- 172.21.15.16:0/4045594897 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faab81531b0 msgr2=0x7faab80b2d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:30.798 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:30.787+0000 7faabe5fa640 1 --2- 172.21.15.16:0/4045594897 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faab81531b0 0x7faab80b2d00 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7faaac0af040 tx=0x7faaac019740 comp rx=0 tx=0).stop 2023-05-22T15:27:30.798 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:30.791+0000 7faabe5fa640 1 -- 172.21.15.16:0/4045594897 shutdown_connections 2023-05-22T15:27:30.798 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:30.791+0000 7faabe5fa640 1 --2- 172.21.15.16:0/4045594897 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7faaa4065380 0x7faaa4067830 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:30.798 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:30.791+0000 7faabe5fa640 1 --2- 172.21.15.16:0/4045594897 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7faab8159870 0x7faab80ce490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:30.799 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:30.791+0000 7faabe5fa640 1 --2- 172.21.15.16:0/4045594897 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faab81531b0 0x7faab80b2d00 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:30.799 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:30.791+0000 7faabe5fa640 1 --2- 172.21.15.16:0/4045594897 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faab80b0080 0x7faab80aeba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:30.799 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:30.791+0000 7faabe5fa640 1 -- 172.21.15.16:0/4045594897 >> 172.21.15.16:0/4045594897 conn(0x7faab8069720 msgr2=0x7faab8152590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:30.799 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:30.791+0000 7faabe5fa640 1 -- 172.21.15.16:0/4045594897 shutdown_connections 2023-05-22T15:27:30.799 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:30.791+0000 7faabe5fa640 1 -- 172.21.15.16:0/4045594897 wait complete. 2023-05-22T15:27:30.815 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-05-22T15:26:38.019781+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-05-22T15:26:43.570514+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2023-05-22T15:26:47.879159+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2023-05-22T15:26:48.522130+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-05-22T15:26:51.616142+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '27', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-05-22T15:27:30.816 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-05-22T15:27:30.950 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:30.943+0000 7f49d52e1640 1 Processor -- start 2023-05-22T15:27:30.950 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:30.943+0000 7f49d52e1640 1 -- start start 2023-05-22T15:27:30.951 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:30.943+0000 7f49d52e1640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f49d00c67d0 0x7f49d00cf730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:30.951 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:30.943+0000 7f49d52e1640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49d00cfc70 con 0x7f49d00b2880 2023-05-22T15:27:30.951 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:30.943+0000 7f49d52e1640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49d00c6ba0 con 0x7f49d00cc6f0 2023-05-22T15:27:30.951 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:30.943+0000 7f49d52e1640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49d015a300 con 0x7f49d00c67d0 2023-05-22T15:27:30.951 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:30.943+0000 7f49c7fff640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f49d00b2880 0x7f49d00cbfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:36626/0 (socket says 172.21.15.16:36626) 2023-05-22T15:27:30.951 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:30.943+0000 7f49c7fff640 1 -- 172.21.15.16:0/2495752492 learned_addr learned my addr 172.21.15.16:0/2495752492 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:27:30.953 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:30.943+0000 7f49ceffd640 1 --2- 172.21.15.16:0/2495752492 >> v2:172.21.15.62:3300/0 conn(0x7f49d00c67d0 0x7f49d00cf730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:30.953 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:30.943+0000 7f49ce7fc640 1 -- 172.21.15.16:0/2495752492 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1098054201 0 0) 0x7f49d00cfc70 con 0x7f49d00b2880 2023-05-22T15:27:30.953 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:30.943+0000 7f49ce7fc640 1 -- 172.21.15.16:0/2495752492 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f49bc003530 con 0x7f49d00b2880 2023-05-22T15:27:30.953 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:30.943+0000 7f49ce7fc640 1 -- 172.21.15.16:0/2495752492 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3271949355 0 0) 0x7f49bc003530 con 0x7f49d00b2880 2023-05-22T15:27:30.953 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:30.943+0000 7f49ce7fc640 1 -- 172.21.15.16:0/2495752492 >> v2:172.21.15.62:3300/0 conn(0x7f49d00c67d0 msgr2=0x7f49d00cf730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:30.954 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:30.943+0000 7f49ce7fc640 1 --2- 172.21.15.16:0/2495752492 >> v2:172.21.15.62:3300/0 conn(0x7f49d00c67d0 0x7f49d00cf730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:30.954 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:30.943+0000 7f49ce7fc640 1 -- 172.21.15.16:0/2495752492 >> v1:172.21.15.62:6789/0 conn(0x7f49d00cc6f0 legacy=0x7f49d00c3db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:30.954 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:30.943+0000 7f49ce7fc640 1 -- 172.21.15.16:0/2495752492 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49d00cfef0 con 0x7f49d00b2880 2023-05-22T15:27:30.954 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:30.943+0000 7f49ce7fc640 1 -- 172.21.15.16:0/2495752492 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7f49b8001680 con 0x7f49d00b2880 2023-05-22T15:27:30.954 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:30.943+0000 7f49ce7fc640 1 -- 172.21.15.16:0/2495752492 >> v1:172.21.15.16:6789/0 conn(0x7f49d00b2880 legacy=0x7f49d00cbfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:30.954 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:30.943+0000 7f49ce7fc640 1 --2- 172.21.15.16:0/2495752492 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f49bc003980 0x7f49bc003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:30.954 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:30.943+0000 7f49ce7fc640 1 --2- 172.21.15.16:0/2495752492 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f49bc0047d0 0x7f49bc004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:30.955 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:30.943+0000 7f49ce7fc640 1 --2- 172.21.15.16:0/2495752492 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f49bc005180 0x7f49bc00b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:30.955 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:30.943+0000 7f49ce7fc640 1 -- 172.21.15.16:0/2495752492 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49d015a300 con 0x7f49bc003980 2023-05-22T15:27:30.955 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:30.943+0000 7f49ce7fc640 1 -- 172.21.15.16:0/2495752492 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49bc00c090 con 0x7f49bc0047d0 2023-05-22T15:27:30.955 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:30.943+0000 7f49ce7fc640 1 -- 172.21.15.16:0/2495752492 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49bc00c250 con 0x7f49bc005180 2023-05-22T15:27:30.955 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:30.943+0000 7f49ceffd640 1 --2- 172.21.15.16:0/2495752492 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f49bc003980 0x7f49bc003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:30.955 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:30.943+0000 7f49c7fff640 1 --2- 172.21.15.16:0/2495752492 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f49bc0047d0 0x7f49bc004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:30.955 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:30.943+0000 7f49cf7fe640 1 --2- 172.21.15.16:0/2495752492 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f49bc005180 0x7f49bc00b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:30.956 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:30.943+0000 7f49ceffd640 1 -- 172.21.15.16:0/2495752492 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f49bc005180 msgr2=0x7f49bc00b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:30.956 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:30.943+0000 7f49ceffd640 1 --2- 172.21.15.16:0/2495752492 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f49bc005180 0x7f49bc00b990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:30.956 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:30.943+0000 7f49ceffd640 1 -- 172.21.15.16:0/2495752492 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f49bc0047d0 msgr2=0x7f49bc004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:30.956 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:30.943+0000 7f49ceffd640 1 --2- 172.21.15.16:0/2495752492 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f49bc0047d0 0x7f49bc004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:30.956 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:30.943+0000 7f49ceffd640 1 -- 172.21.15.16:0/2495752492 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49bc003530 con 0x7f49bc003980 2023-05-22T15:27:30.956 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:30.943+0000 7f49cf7fe640 1 -- 172.21.15.16:0/2495752492 reap_dead start 2023-05-22T15:27:30.959 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:30.947+0000 7f49ceffd640 1 --2- 172.21.15.16:0/2495752492 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f49bc003980 0x7f49bc003d90 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f49c0009ae0 tx=0x7f49c001ca20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29871f1725f86f8b server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:30.959 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:30.947+0000 7f49ce7fc640 1 -- 172.21.15.16:0/2495752492 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49c00a6510 con 0x7f49bc003980 2023-05-22T15:27:30.959 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:30.947+0000 7f49ce7fc640 1 -- 172.21.15.16:0/2495752492 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49c00a6670 con 0x7f49bc003980 2023-05-22T15:27:30.960 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:30.947+0000 7f49ce7fc640 1 -- 172.21.15.16:0/2495752492 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49c00a6a20 con 0x7f49bc003980 2023-05-22T15:27:30.960 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:30.947+0000 7f49d52e1640 1 -- 172.21.15.16:0/2495752492 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f49bc003980 msgr2=0x7f49bc003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:30.960 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:30.947+0000 7f49d52e1640 1 --2- 172.21.15.16:0/2495752492 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f49bc003980 0x7f49bc003d90 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f49c0009ae0 tx=0x7f49c001ca20 comp rx=0 tx=0).stop 2023-05-22T15:27:30.960 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:30.947+0000 7f49d52e1640 1 -- 172.21.15.16:0/2495752492 shutdown_connections 2023-05-22T15:27:30.961 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:30.947+0000 7f49d52e1640 1 --2- 172.21.15.16:0/2495752492 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f49bc003980 0x7f49bc003d90 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:30.961 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:30.947+0000 7f49d52e1640 1 -- 172.21.15.16:0/2495752492 >> 172.21.15.16:0/2495752492 conn(0x7f49d006f5a0 msgr2=0x7f49d006f9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:30.961 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:30.947+0000 7f49d52e1640 1 -- 172.21.15.16:0/2495752492 shutdown_connections 2023-05-22T15:27:30.961 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:30.947+0000 7f49d52e1640 1 -- 172.21.15.16:0/2495752492 wait complete. 2023-05-22T15:27:30.962 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:30.947+0000 7f49d52e1640 1 Processor -- start 2023-05-22T15:27:30.962 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:30.947+0000 7f49d52e1640 1 -- start start 2023-05-22T15:27:30.962 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:30.951+0000 7f49d52e1640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f49bc003980 0x7f49d014cde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:30.963 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:30.951+0000 7f49d52e1640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f49d014d6e0 0x7f49d00b51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:30.963 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:30.951+0000 7f49d52e1640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f49d00b5730 0x7f49d01f5eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:30.963 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:30.951+0000 7f49d52e1640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49d00b5bd0 con 0x7f49bc003980 2023-05-22T15:27:30.963 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:30.951+0000 7f49d52e1640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49d01f6c10 con 0x7f49d014d6e0 2023-05-22T15:27:30.964 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:30.951+0000 7f49d52e1640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49d00b3130 con 0x7f49d00b5730 2023-05-22T15:27:30.964 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:30.951+0000 7f49ceffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f49bc003980 0x7f49d014cde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:30.964 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:30.951+0000 7f49ceffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f49bc003980 0x7f49d014cde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57246/0 (socket says 172.21.15.16:57246) 2023-05-22T15:27:30.965 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:30.951+0000 7f49ceffd640 1 -- 172.21.15.16:0/4086878638 learned_addr learned my addr 172.21.15.16:0/4086878638 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:30.965 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:30.951+0000 7f49c7fff640 1 --2- 172.21.15.16:0/4086878638 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f49d014d6e0 0x7f49d00b51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:30.965 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:30.951+0000 7f49cf7fe640 1 --2- 172.21.15.16:0/4086878638 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f49d00b5730 0x7f49d01f5eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:30.965 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:30.951+0000 7f49ceffd640 1 -- 172.21.15.16:0/4086878638 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f49d00b5730 msgr2=0x7f49d01f5eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:30.966 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:30.951+0000 7f49ceffd640 1 --2- 172.21.15.16:0/4086878638 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f49d00b5730 0x7f49d01f5eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:30.966 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:30.951+0000 7f49ceffd640 1 -- 172.21.15.16:0/4086878638 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f49d014d6e0 msgr2=0x7f49d00b51f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:30.966 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:30.951+0000 7f49ceffd640 1 --2- 172.21.15.16:0/4086878638 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f49d014d6e0 0x7f49d00b51f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:30.966 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:30.951+0000 7f49ceffd640 1 -- 172.21.15.16:0/4086878638 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49d00d0a70 con 0x7f49bc003980 2023-05-22T15:27:30.967 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:30.951+0000 7f49ceffd640 1 --2- 172.21.15.16:0/4086878638 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f49bc003980 0x7f49d014cde0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f49c00b0040 tx=0x7f49c00197c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:30.967 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:30.951+0000 7f49c77fe640 1 -- 172.21.15.16:0/4086878638 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49c00a6510 con 0x7f49bc003980 2023-05-22T15:27:30.967 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:30.951+0000 7f49c77fe640 1 -- 172.21.15.16:0/4086878638 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49c00a67e0 con 0x7f49bc003980 2023-05-22T15:27:30.968 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:30.951+0000 7f49d52e1640 1 -- 172.21.15.16:0/4086878638 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49d00d0790 con 0x7f49bc003980 2023-05-22T15:27:30.968 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:30.951+0000 7f49c77fe640 1 -- 172.21.15.16:0/4086878638 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49c00a6b50 con 0x7f49bc003980 2023-05-22T15:27:30.968 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:30.951+0000 7f49d52e1640 1 -- 172.21.15.16:0/4086878638 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49d00d0f90 con 0x7f49bc003980 2023-05-22T15:27:30.969 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:30.951+0000 7f49d52e1640 1 -- 172.21.15.16:0/4086878638 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49d004dbe0 con 0x7f49bc003980 2023-05-22T15:27:30.969 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:30.951+0000 7f49c77fe640 1 -- 172.21.15.16:0/4086878638 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7f49c00a6cb0 con 0x7f49bc003980 2023-05-22T15:27:30.969 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:30.951+0000 7f49c77fe640 1 --2- 172.21.15.16:0/4086878638 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f49b00653b0 0x7f49b0067860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:30.969 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:30.951+0000 7f49c77fe640 1 -- 172.21.15.16:0/4086878638 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f49c00aa070 con 0x7f49bc003980 2023-05-22T15:27:30.970 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:30.951+0000 7f49c7fff640 1 --2- 172.21.15.16:0/4086878638 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f49b00653b0 0x7f49b0067860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:30.970 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:30.951+0000 7f49c7fff640 1 --2- 172.21.15.16:0/4086878638 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f49b00653b0 0x7f49b0067860 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f49d01f5c80 tx=0x7f49b8009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:30.970 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:30.955+0000 7f49c77fe640 1 -- 172.21.15.16:0/4086878638 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f49c00eaef0 con 0x7f49bc003980 2023-05-22T15:27:31.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.111+0000 7f49d52e1640 1 -- 172.21.15.16:0/4086878638 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f49d00b09b0 con 0x7f49bc003980 2023-05-22T15:27:31.118 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.111+0000 7f49c77fe640 1 -- 172.21.15.16:0/4086878638 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v28) v1 ==== 93+0+10 (secure 0 0 0) 0x7f49c00ee510 con 0x7f49bc003980 2023-05-22T15:27:31.119 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2023-05-22T15:27:31.120 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.111+0000 7f49d52e1640 1 -- 172.21.15.16:0/4086878638 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f49b00653b0 msgr2=0x7f49b0067860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:31.121 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.111+0000 7f49d52e1640 1 --2- 172.21.15.16:0/4086878638 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f49b00653b0 0x7f49b0067860 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f49d01f5c80 tx=0x7f49b8009040 comp rx=0 tx=0).stop 2023-05-22T15:27:31.121 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.111+0000 7f49d52e1640 1 -- 172.21.15.16:0/4086878638 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f49bc003980 msgr2=0x7f49d014cde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:31.121 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.111+0000 7f49d52e1640 1 --2- 172.21.15.16:0/4086878638 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f49bc003980 0x7f49d014cde0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f49c00b0040 tx=0x7f49c00197c0 comp rx=0 tx=0).stop 2023-05-22T15:27:31.121 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.111+0000 7f49d52e1640 1 -- 172.21.15.16:0/4086878638 shutdown_connections 2023-05-22T15:27:31.122 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.111+0000 7f49d52e1640 1 --2- 172.21.15.16:0/4086878638 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f49b00653b0 0x7f49b0067860 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:31.122 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.111+0000 7f49d52e1640 1 --2- 172.21.15.16:0/4086878638 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f49d00b5730 0x7f49d01f5eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:31.122 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.111+0000 7f49d52e1640 1 --2- 172.21.15.16:0/4086878638 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f49d014d6e0 0x7f49d00b51f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:31.122 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.111+0000 7f49d52e1640 1 --2- 172.21.15.16:0/4086878638 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f49bc003980 0x7f49d014cde0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:31.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.111+0000 7f49d52e1640 1 -- 172.21.15.16:0/4086878638 >> 172.21.15.16:0/4086878638 conn(0x7f49d006f5a0 msgr2=0x7f49d00b11b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:31.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.111+0000 7f49d52e1640 1 -- 172.21.15.16:0/4086878638 shutdown_connections 2023-05-22T15:27:31.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.111+0000 7f49d52e1640 1 -- 172.21.15.16:0/4086878638 wait complete. 2023-05-22T15:27:31.138 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-05-22T15:27:31.266 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.259+0000 7f743fd1f640 1 Processor -- start 2023-05-22T15:27:31.266 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.259+0000 7f743fd1f640 1 -- start start 2023-05-22T15:27:31.267 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.259+0000 7f743fd1f640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f7438153fe0 0x7f743806d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:31.267 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.259+0000 7f743fd1f640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7438070600 0x7f74380ad410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:31.267 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.259+0000 7f743fd1f640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7438069720 con 0x7f74380cae20 2023-05-22T15:27:31.268 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.259+0000 7f743fd1f640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74381543b0 con 0x7f7438153fe0 2023-05-22T15:27:31.268 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.259+0000 7f743fd1f640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f743815e360 con 0x7f7438070600 2023-05-22T15:27:31.268 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.259+0000 7f743e2bc640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7438070600 0x7f74380ad410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:31.269 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.259+0000 7f743dabb640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f7438153fe0 0x7f743806d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:31.269 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.259+0000 7f743dabb640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f7438153fe0 0x7f743806d3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.16:40042/0 (socket says 172.21.15.16:40042) 2023-05-22T15:27:31.269 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.259+0000 7f743e2bc640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7438070600 0x7f74380ad410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3301/0 says I am v2:172.21.15.16:50322/0 (socket says 172.21.15.16:50322) 2023-05-22T15:27:31.269 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.259+0000 7f743dabb640 1 -- 172.21.15.16:0/4030310291 learned_addr learned my addr 172.21.15.16:0/4030310291 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:31.269 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.259+0000 7f743e2bc640 1 -- 172.21.15.16:0/4030310291 >> v2:172.21.15.62:3300/0 conn(0x7f7438153fe0 msgr2=0x7f743806d3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:31.269 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.259+0000 7f743e2bc640 1 --2- 172.21.15.16:0/4030310291 >> v2:172.21.15.62:3300/0 conn(0x7f7438153fe0 0x7f743806d3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:31.269 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.259+0000 7f743e2bc640 1 -- 172.21.15.16:0/4030310291 >> v1:172.21.15.62:6789/0 conn(0x7f74380cae20 legacy=0x7f743806fef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:31.270 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.259+0000 7f743e2bc640 1 -- 172.21.15.16:0/4030310291 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f743815d5a0 con 0x7f7438070600 2023-05-22T15:27:31.270 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.259+0000 7f743dabb640 1 --2- 172.21.15.16:0/4030310291 >> v2:172.21.15.62:3300/0 conn(0x7f7438153fe0 0x7f743806d3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-05-22T15:27:31.270 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.263+0000 7f743e2bc640 1 --2- 172.21.15.16:0/4030310291 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7438070600 0x7f74380ad410 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f7430009d90 tx=0x7f743001c920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fe130b37b042bcf1 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:31.271 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.263+0000 7f743cab9640 1 -- 172.21.15.16:0/4030310291 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74300a59e0 con 0x7f7438070600 2023-05-22T15:27:31.271 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.263+0000 7f743cab9640 1 -- 172.21.15.16:0/4030310291 <== mon.2 v2:172.21.15.62:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74300a5b40 con 0x7f7438070600 2023-05-22T15:27:31.271 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.263+0000 7f743cab9640 1 -- 172.21.15.16:0/4030310291 <== mon.2 v2:172.21.15.62:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74300ad440 con 0x7f7438070600 2023-05-22T15:27:31.272 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.263+0000 7f743fd1f640 1 -- 172.21.15.16:0/4030310291 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7438070600 msgr2=0x7f74380ad410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:31.272 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.263+0000 7f743fd1f640 1 --2- 172.21.15.16:0/4030310291 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7438070600 0x7f74380ad410 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f7430009d90 tx=0x7f743001c920 comp rx=0 tx=0).stop 2023-05-22T15:27:31.272 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.263+0000 7f743fd1f640 1 -- 172.21.15.16:0/4030310291 shutdown_connections 2023-05-22T15:27:31.272 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.263+0000 7f743fd1f640 1 --2- 172.21.15.16:0/4030310291 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7438070600 0x7f74380ad410 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:31.272 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.263+0000 7f743fd1f640 1 --2- 172.21.15.16:0/4030310291 >> v2:172.21.15.62:3300/0 conn(0x7f7438153fe0 0x7f743806d3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:31.272 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.263+0000 7f743fd1f640 1 -- 172.21.15.16:0/4030310291 >> 172.21.15.16:0/4030310291 conn(0x7f74380b1110 msgr2=0x7f74380ce830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:31.273 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.263+0000 7f743fd1f640 1 -- 172.21.15.16:0/4030310291 shutdown_connections 2023-05-22T15:27:31.273 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.263+0000 7f743fd1f640 1 -- 172.21.15.16:0/4030310291 wait complete. 2023-05-22T15:27:31.274 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.263+0000 7f743fd1f640 1 Processor -- start 2023-05-22T15:27:31.274 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.263+0000 7f743fd1f640 1 -- start start 2023-05-22T15:27:31.274 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.267+0000 7f743fd1f640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7438070600 0x7f743814b9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:31.274 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.267+0000 7f743fd1f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f74380cae20 0x7f743814db30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:31.274 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.267+0000 7f743fd1f640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7438153fe0 0x7f74380c43e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:31.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.267+0000 7f743fd1f640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74380c7050 con 0x7f74380cae20 2023-05-22T15:27:31.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.267+0000 7f743fd1f640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f743814bf30 con 0x7f7438153fe0 2023-05-22T15:27:31.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.267+0000 7f743fd1f640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74380c6720 con 0x7f7438070600 2023-05-22T15:27:31.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.267+0000 7f743d2ba640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f74380cae20 0x7f743814db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:31.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.267+0000 7f743d2ba640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f74380cae20 0x7f743814db30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57260/0 (socket says 172.21.15.16:57260) 2023-05-22T15:27:31.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.267+0000 7f743d2ba640 1 -- 172.21.15.16:0/2894589173 learned_addr learned my addr 172.21.15.16:0/2894589173 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:31.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.267+0000 7f743dabb640 1 --2- 172.21.15.16:0/2894589173 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7438070600 0x7f743814b9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:31.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.267+0000 7f743e2bc640 1 --2- 172.21.15.16:0/2894589173 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7438153fe0 0x7f74380c43e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:31.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.267+0000 7f743d2ba640 1 -- 172.21.15.16:0/2894589173 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7438070600 msgr2=0x7f743814b9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:31.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.267+0000 7f743d2ba640 1 --2- 172.21.15.16:0/2894589173 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7438070600 0x7f743814b9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:31.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.267+0000 7f743d2ba640 1 -- 172.21.15.16:0/2894589173 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7438153fe0 msgr2=0x7f74380c43e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:31.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.267+0000 7f743d2ba640 1 --2- 172.21.15.16:0/2894589173 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7438153fe0 0x7f74380c43e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:31.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.267+0000 7f743d2ba640 1 -- 172.21.15.16:0/2894589173 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74340094d0 con 0x7f74380cae20 2023-05-22T15:27:31.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.267+0000 7f743d2ba640 1 --2- 172.21.15.16:0/2894589173 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f74380cae20 0x7f743814db30 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f742800c4d0 tx=0x7f742800c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:31.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.267+0000 7f742e7fc640 1 -- 172.21.15.16:0/2894589173 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74280052d0 con 0x7f74380cae20 2023-05-22T15:27:31.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.267+0000 7f742e7fc640 1 -- 172.21.15.16:0/2894589173 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7428005430 con 0x7f74380cae20 2023-05-22T15:27:31.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.267+0000 7f742e7fc640 1 -- 172.21.15.16:0/2894589173 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7428012830 con 0x7f74380cae20 2023-05-22T15:27:31.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.267+0000 7f743fd1f640 1 -- 172.21.15.16:0/2894589173 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7430009a70 con 0x7f74380cae20 2023-05-22T15:27:31.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.267+0000 7f743fd1f640 1 -- 172.21.15.16:0/2894589173 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f743806dd20 con 0x7f74380cae20 2023-05-22T15:27:31.282 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.267+0000 7f743fd1f640 1 -- 172.21.15.16:0/2894589173 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f743815de40 con 0x7f74380cae20 2023-05-22T15:27:31.282 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.267+0000 7f742e7fc640 1 -- 172.21.15.16:0/2894589173 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7f7428012990 con 0x7f74380cae20 2023-05-22T15:27:31.282 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.267+0000 7f742e7fc640 1 --2- 172.21.15.16:0/2894589173 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f741c065380 0x7f741c067830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:31.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.267+0000 7f742e7fc640 1 -- 172.21.15.16:0/2894589173 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f7428082620 con 0x7f74380cae20 2023-05-22T15:27:31.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.267+0000 7f743dabb640 1 --2- 172.21.15.16:0/2894589173 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f741c065380 0x7f741c067830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:31.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.271+0000 7f743dabb640 1 --2- 172.21.15.16:0/2894589173 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f741c065380 0x7f741c067830 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f7434006cb0 tx=0x7f7434009240 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:31.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.271+0000 7f742e7fc640 1 -- 172.21.15.16:0/2894589173 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f742804ff20 con 0x7f74380cae20 2023-05-22T15:27:31.436 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.427+0000 7f743fd1f640 1 -- 172.21.15.16:0/2894589173 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f743806b0d0 con 0x7f74380cae20 2023-05-22T15:27:31.437 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.427+0000 7f742e7fc640 1 -- 172.21.15.16:0/2894589173 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v29) v1 ==== 92+0+10 (secure 0 0 0) 0x7f7428053540 con 0x7f74380cae20 2023-05-22T15:27:31.437 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 8 2023-05-22T15:27:31.439 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.431+0000 7f743fd1f640 1 -- 172.21.15.16:0/2894589173 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f741c065380 msgr2=0x7f741c067830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:31.439 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.431+0000 7f743fd1f640 1 --2- 172.21.15.16:0/2894589173 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f741c065380 0x7f741c067830 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f7434006cb0 tx=0x7f7434009240 comp rx=0 tx=0).stop 2023-05-22T15:27:31.439 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.431+0000 7f743fd1f640 1 -- 172.21.15.16:0/2894589173 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f74380cae20 msgr2=0x7f743814db30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:31.440 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.431+0000 7f743fd1f640 1 --2- 172.21.15.16:0/2894589173 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f74380cae20 0x7f743814db30 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f742800c4d0 tx=0x7f742800c990 comp rx=0 tx=0).stop 2023-05-22T15:27:31.440 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.431+0000 7f743fd1f640 1 -- 172.21.15.16:0/2894589173 shutdown_connections 2023-05-22T15:27:31.440 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.431+0000 7f743fd1f640 1 --2- 172.21.15.16:0/2894589173 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7438153fe0 0x7f74380c43e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:31.440 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.431+0000 7f743fd1f640 1 --2- 172.21.15.16:0/2894589173 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f74380cae20 0x7f743814db30 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:31.440 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.431+0000 7f743fd1f640 1 --2- 172.21.15.16:0/2894589173 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f741c065380 0x7f741c067830 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:31.440 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.431+0000 7f743fd1f640 1 --2- 172.21.15.16:0/2894589173 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7438070600 0x7f743814b9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:31.440 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.431+0000 7f743fd1f640 1 -- 172.21.15.16:0/2894589173 >> 172.21.15.16:0/2894589173 conn(0x7f74380b1110 msgr2=0x7f74380ce750 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:31.441 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.431+0000 7f743fd1f640 1 -- 172.21.15.16:0/2894589173 shutdown_connections 2023-05-22T15:27:31.441 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.431+0000 7f743fd1f640 1 -- 172.21.15.16:0/2894589173 wait complete. 2023-05-22T15:27:31.457 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-05-22T15:27:31.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.579+0000 7eff6ca08640 1 Processor -- start 2023-05-22T15:27:31.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.579+0000 7eff6ca08640 1 -- start start 2023-05-22T15:27:31.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.579+0000 7eff6ca08640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7eff68071530 0x7eff6815c580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:31.587 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.579+0000 7eff6ca08640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7eff6806b6c0 0x7eff6806bac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:31.587 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.579+0000 7eff6ca08640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff6815d520 con 0x7eff68070c70 2023-05-22T15:27:31.587 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.579+0000 7eff6ca08640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff68071980 con 0x7eff68071530 2023-05-22T15:27:31.587 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.579+0000 7eff6ca08640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff6806c000 con 0x7eff6806b6c0 2023-05-22T15:27:31.587 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.579+0000 7eff6659c640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7eff68070c70 0x7eff6806fba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:36638/0 (socket says 172.21.15.16:36638) 2023-05-22T15:27:31.587 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.579+0000 7eff6659c640 1 -- 172.21.15.16:0/2561314656 learned_addr learned my addr 172.21.15.16:0/2561314656 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:27:31.587 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.579+0000 7eff65d9b640 1 --2- 172.21.15.16:0/2561314656 >> v2:172.21.15.16:3300/0 conn(0x7eff68071530 0x7eff6815c580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:31.588 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.579+0000 7eff66d9d640 1 --2- 172.21.15.16:0/2561314656 >> v2:172.21.15.62:3300/0 conn(0x7eff6806b6c0 0x7eff6806bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:31.588 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.579+0000 7eff6559a640 1 -- 172.21.15.16:0/2561314656 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1611687823 0 0) 0x7eff6815d520 con 0x7eff68070c70 2023-05-22T15:27:31.588 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.579+0000 7eff6559a640 1 -- 172.21.15.16:0/2561314656 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7eff54003530 con 0x7eff68070c70 2023-05-22T15:27:31.589 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.579+0000 7eff65d9b640 1 -- 172.21.15.16:0/2561314656 >> v2:172.21.15.62:3300/0 conn(0x7eff6806b6c0 msgr2=0x7eff6806bac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:31.589 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.579+0000 7eff65d9b640 1 --2- 172.21.15.16:0/2561314656 >> v2:172.21.15.62:3300/0 conn(0x7eff6806b6c0 0x7eff6806bac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:31.589 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.579+0000 7eff65d9b640 1 -- 172.21.15.16:0/2561314656 >> v1:172.21.15.16:6789/0 conn(0x7eff68070c70 legacy=0x7eff6806fba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:31.589 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.579+0000 7eff65d9b640 1 -- 172.21.15.16:0/2561314656 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff680c8b10 con 0x7eff68071530 2023-05-22T15:27:31.590 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.583+0000 7eff65d9b640 1 --2- 172.21.15.16:0/2561314656 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7eff68071530 0x7eff6815c580 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7eff680702b0 tx=0x7eff5801c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61f85c0168e0c806 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:31.590 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.583+0000 7eff6559a640 1 -- 172.21.15.16:0/2561314656 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff580a5940 con 0x7eff68071530 2023-05-22T15:27:31.590 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.583+0000 7eff6559a640 1 -- 172.21.15.16:0/2561314656 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff580a5aa0 con 0x7eff68071530 2023-05-22T15:27:31.590 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.583+0000 7eff6559a640 1 -- 172.21.15.16:0/2561314656 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff580a5e30 con 0x7eff68071530 2023-05-22T15:27:31.590 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.583+0000 7eff6ca08640 1 -- 172.21.15.16:0/2561314656 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7eff68071530 msgr2=0x7eff6815c580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:31.590 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.583+0000 7eff6ca08640 1 --2- 172.21.15.16:0/2561314656 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7eff68071530 0x7eff6815c580 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7eff680702b0 tx=0x7eff5801c880 comp rx=0 tx=0).stop 2023-05-22T15:27:31.591 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.583+0000 7eff6ca08640 1 -- 172.21.15.16:0/2561314656 shutdown_connections 2023-05-22T15:27:31.591 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.583+0000 7eff6ca08640 1 --2- 172.21.15.16:0/2561314656 >> v2:172.21.15.62:3300/0 conn(0x7eff6806b6c0 0x7eff6806bac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:31.591 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.583+0000 7eff6ca08640 1 --2- 172.21.15.16:0/2561314656 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7eff68071530 0x7eff6815c580 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:31.591 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.583+0000 7eff6ca08640 1 -- 172.21.15.16:0/2561314656 >> 172.21.15.16:0/2561314656 conn(0x7eff68069720 msgr2=0x7eff68069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:31.591 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.583+0000 7eff6ca08640 1 -- 172.21.15.16:0/2561314656 shutdown_connections 2023-05-22T15:27:31.592 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.583+0000 7eff6ca08640 1 -- 172.21.15.16:0/2561314656 wait complete. 2023-05-22T15:27:31.593 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.583+0000 7eff6ca08640 1 Processor -- start 2023-05-22T15:27:31.593 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.583+0000 7eff6ca08640 1 -- start start 2023-05-22T15:27:31.593 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.583+0000 7eff6ca08640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7eff6806b6c0 0x7eff681f7000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:31.593 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.583+0000 7eff6ca08640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7eff68070c70 0x7eff680ac820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:31.593 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.583+0000 7eff6ca08640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7eff68071530 0x7eff680acd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:31.593 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.583+0000 7eff6ca08640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff680ab7d0 con 0x7eff6806b6c0 2023-05-22T15:27:31.594 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.583+0000 7eff6ca08640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff680c45c0 con 0x7eff68071530 2023-05-22T15:27:31.594 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.583+0000 7eff6ca08640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff681f7540 con 0x7eff68070c70 2023-05-22T15:27:31.594 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.583+0000 7eff6659c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7eff6806b6c0 0x7eff681f7000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:31.594 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.583+0000 7eff6659c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7eff6806b6c0 0x7eff681f7000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57276/0 (socket says 172.21.15.16:57276) 2023-05-22T15:27:31.594 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.583+0000 7eff6659c640 1 -- 172.21.15.16:0/716588399 learned_addr learned my addr 172.21.15.16:0/716588399 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:31.594 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.583+0000 7eff66d9d640 1 --2- 172.21.15.16:0/716588399 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7eff68071530 0x7eff680acd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:31.594 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.583+0000 7eff65d9b640 1 --2- 172.21.15.16:0/716588399 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7eff68070c70 0x7eff680ac820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:31.595 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.583+0000 7eff6659c640 1 -- 172.21.15.16:0/716588399 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7eff68070c70 msgr2=0x7eff680ac820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:31.595 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.583+0000 7eff6659c640 1 --2- 172.21.15.16:0/716588399 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7eff68070c70 0x7eff680ac820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:31.595 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.583+0000 7eff6659c640 1 -- 172.21.15.16:0/716588399 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7eff68071530 msgr2=0x7eff680acd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:31.595 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.583+0000 7eff6659c640 1 --2- 172.21.15.16:0/716588399 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7eff68071530 0x7eff680acd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:31.595 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.583+0000 7eff6659c640 1 -- 172.21.15.16:0/716588399 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff58009a70 con 0x7eff6806b6c0 2023-05-22T15:27:31.596 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.587+0000 7eff6659c640 1 --2- 172.21.15.16:0/716588399 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7eff6806b6c0 0x7eff681f7000 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7eff5000c4d0 tx=0x7eff5000c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:31.596 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.587+0000 7eff5effd640 1 -- 172.21.15.16:0/716588399 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff50005cf0 con 0x7eff6806b6c0 2023-05-22T15:27:31.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.587+0000 7eff5effd640 1 -- 172.21.15.16:0/716588399 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff50005e50 con 0x7eff6806b6c0 2023-05-22T15:27:31.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.587+0000 7eff6ca08640 1 -- 172.21.15.16:0/716588399 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff680cc510 con 0x7eff6806b6c0 2023-05-22T15:27:31.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.587+0000 7eff5effd640 1 -- 172.21.15.16:0/716588399 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff50007270 con 0x7eff6806b6c0 2023-05-22T15:27:31.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.587+0000 7eff6ca08640 1 -- 172.21.15.16:0/716588399 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff6815d240 con 0x7eff6806b6c0 2023-05-22T15:27:31.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.587+0000 7eff6ca08640 1 -- 172.21.15.16:0/716588399 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff68071040 con 0x7eff6806b6c0 2023-05-22T15:27:31.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.587+0000 7eff5effd640 1 -- 172.21.15.16:0/716588399 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7eff500073d0 con 0x7eff6806b6c0 2023-05-22T15:27:31.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.587+0000 7eff5effd640 1 --2- 172.21.15.16:0/716588399 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7eff48065380 0x7eff48067830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:31.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.587+0000 7eff5effd640 1 -- 172.21.15.16:0/716588399 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6871+0+0 (secure 0 0 0) 0x7eff50081ff0 con 0x7eff6806b6c0 2023-05-22T15:27:31.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.587+0000 7eff65d9b640 1 --2- 172.21.15.16:0/716588399 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7eff48065380 0x7eff48067830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:31.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.587+0000 7eff65d9b640 1 --2- 172.21.15.16:0/716588399 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7eff48065380 0x7eff48067830 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7eff580af000 tx=0x7eff580a7070 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:31.600 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.591+0000 7eff5effd640 1 -- 172.21.15.16:0/716588399 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7eff5004a620 con 0x7eff6806b6c0 2023-05-22T15:27:31.753 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.747+0000 7eff6ca08640 1 -- 172.21.15.16:0/716588399 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7eff6804dbe0 con 0x7eff6806b6c0 2023-05-22T15:27:31.754 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.747+0000 7eff5effd640 1 -- 172.21.15.16:0/716588399 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v29) v1 ==== 104+0+11 (secure 0 0 0) 0x7eff500534b0 con 0x7eff6806b6c0 2023-05-22T15:27:31.755 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2023-05-22T15:27:31.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.747+0000 7eff6ca08640 1 -- 172.21.15.16:0/716588399 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7eff48065380 msgr2=0x7eff48067830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:31.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.747+0000 7eff6ca08640 1 --2- 172.21.15.16:0/716588399 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7eff48065380 0x7eff48067830 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7eff580af000 tx=0x7eff580a7070 comp rx=0 tx=0).stop 2023-05-22T15:27:31.758 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.747+0000 7eff6ca08640 1 -- 172.21.15.16:0/716588399 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7eff6806b6c0 msgr2=0x7eff681f7000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:31.758 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.747+0000 7eff6ca08640 1 --2- 172.21.15.16:0/716588399 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7eff6806b6c0 0x7eff681f7000 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7eff5000c4d0 tx=0x7eff5000c990 comp rx=0 tx=0).stop 2023-05-22T15:27:31.758 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.747+0000 7eff6ca08640 1 -- 172.21.15.16:0/716588399 shutdown_connections 2023-05-22T15:27:31.758 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.747+0000 7eff6ca08640 1 --2- 172.21.15.16:0/716588399 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7eff68071530 0x7eff680acd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:31.758 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.747+0000 7eff6ca08640 1 --2- 172.21.15.16:0/716588399 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7eff48065380 0x7eff48067830 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:31.758 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.747+0000 7eff6ca08640 1 --2- 172.21.15.16:0/716588399 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7eff68070c70 0x7eff680ac820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:31.758 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.747+0000 7eff6ca08640 1 --2- 172.21.15.16:0/716588399 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7eff6806b6c0 0x7eff681f7000 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:31.759 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.747+0000 7eff6ca08640 1 -- 172.21.15.16:0/716588399 >> 172.21.15.16:0/716588399 conn(0x7eff68069720 msgr2=0x7eff6815b090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:31.759 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.747+0000 7eff6ca08640 1 -- 172.21.15.16:0/716588399 shutdown_connections 2023-05-22T15:27:31.759 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.747+0000 7eff6ca08640 1 -- 172.21.15.16:0/716588399 wait complete. 2023-05-22T15:27:31.775 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-05-22T15:27:31.904 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.895+0000 7f320d2fa640 1 Processor -- start 2023-05-22T15:27:31.904 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.895+0000 7f320d2fa640 1 -- start start 2023-05-22T15:27:31.904 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.895+0000 7f320d2fa640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f3208159980 0x7f3208069720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:31.905 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.895+0000 7f320d2fa640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f32080b4170 0x7f320806fec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:31.905 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.895+0000 7f320d2fa640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f32080b45c0 0x7f3208070400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:31.905 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.895+0000 7f320d2fa640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32080ce720 con 0x7f3208159980 2023-05-22T15:27:31.905 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.895+0000 7f320d2fa640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f320806af20 con 0x7f32080b45c0 2023-05-22T15:27:31.905 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.895+0000 7f320d2fa640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32080cd8c0 con 0x7f32080b4170 2023-05-22T15:27:31.906 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.895+0000 7f3206ffd640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f3208159980 0x7f3208069720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:31.906 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.895+0000 7f3206ffd640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f3208159980 0x7f3208069720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57278/0 (socket says 172.21.15.16:57278) 2023-05-22T15:27:31.906 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.895+0000 7f3206ffd640 1 -- 172.21.15.16:0/2851630207 learned_addr learned my addr 172.21.15.16:0/2851630207 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:31.907 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.899+0000 7f32077fe640 1 --2- 172.21.15.16:0/2851630207 >> v2:172.21.15.62:3300/0 conn(0x7f32080b45c0 0x7f3208070400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:31.907 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.899+0000 7f32067fc640 1 --2- 172.21.15.16:0/2851630207 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f32080b4170 0x7f320806fec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:31.907 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.899+0000 7f3206ffd640 1 -- 172.21.15.16:0/2851630207 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f32080b4170 msgr2=0x7f320806fec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:31.907 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.899+0000 7f3206ffd640 1 --2- 172.21.15.16:0/2851630207 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f32080b4170 0x7f320806fec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:31.907 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.899+0000 7f3206ffd640 1 -- 172.21.15.16:0/2851630207 >> v2:172.21.15.62:3300/0 conn(0x7f32080b45c0 msgr2=0x7f3208070400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:31.907 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.899+0000 7f3206ffd640 1 --2- 172.21.15.16:0/2851630207 >> v2:172.21.15.62:3300/0 conn(0x7f32080b45c0 0x7f3208070400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:31.907 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.899+0000 7f3206ffd640 1 -- 172.21.15.16:0/2851630207 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32080cdba0 con 0x7f3208159980 2023-05-22T15:27:31.908 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.899+0000 7f32077fe640 1 --2- 172.21.15.16:0/2851630207 >> v2:172.21.15.62:3300/0 conn(0x7f32080b45c0 0x7f3208070400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T15:27:31.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.903+0000 7f3206ffd640 1 --2- 172.21.15.16:0/2851630207 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3208159980 0x7f3208069720 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f320815d550 tx=0x7f31f801c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=498a6fdf99450723 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:31.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.903+0000 7f3205ffb640 1 -- 172.21.15.16:0/2851630207 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31f80aa5f0 con 0x7f3208159980 2023-05-22T15:27:31.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.903+0000 7f3205ffb640 1 -- 172.21.15.16:0/2851630207 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31f80aa750 con 0x7f3208159980 2023-05-22T15:27:31.911 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.903+0000 7f3205ffb640 1 -- 172.21.15.16:0/2851630207 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31f80aaae0 con 0x7f3208159980 2023-05-22T15:27:31.911 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.903+0000 7f320d2fa640 1 -- 172.21.15.16:0/2851630207 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3208159980 msgr2=0x7f3208069720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:31.912 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.903+0000 7f320d2fa640 1 --2- 172.21.15.16:0/2851630207 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3208159980 0x7f3208069720 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f320815d550 tx=0x7f31f801c4d0 comp rx=0 tx=0).stop 2023-05-22T15:27:31.912 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.903+0000 7f320d2fa640 1 -- 172.21.15.16:0/2851630207 shutdown_connections 2023-05-22T15:27:31.912 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.903+0000 7f320d2fa640 1 --2- 172.21.15.16:0/2851630207 >> v2:172.21.15.62:3300/0 conn(0x7f32080b45c0 0x7f3208070400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:31.912 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.903+0000 7f320d2fa640 1 --2- 172.21.15.16:0/2851630207 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f32080b4170 0x7f320806fec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:31.913 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.903+0000 7f320d2fa640 1 --2- 172.21.15.16:0/2851630207 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3208159980 0x7f3208069720 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:31.913 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.903+0000 7f320d2fa640 1 -- 172.21.15.16:0/2851630207 >> 172.21.15.16:0/2851630207 conn(0x7f32080ce890 msgr2=0x7f32080cec90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:31.913 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.903+0000 7f320d2fa640 1 -- 172.21.15.16:0/2851630207 shutdown_connections 2023-05-22T15:27:31.913 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.903+0000 7f320d2fa640 1 -- 172.21.15.16:0/2851630207 wait complete. 2023-05-22T15:27:31.914 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.903+0000 7f320d2fa640 1 Processor -- start 2023-05-22T15:27:31.914 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.903+0000 7f320d2fa640 1 -- start start 2023-05-22T15:27:31.916 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.903+0000 7f320d2fa640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f32080b4170 0x7f320806d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:31.916 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.903+0000 7f320d2fa640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f32080b45c0 0x7f32080c4990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:31.916 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.903+0000 7f320d2fa640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3208159980 0x7f32081f73d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:31.917 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.903+0000 7f320d2fa640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f320814d810 con 0x7f32080b4170 2023-05-22T15:27:31.917 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.903+0000 7f320d2fa640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32080ad220 con 0x7f3208159980 2023-05-22T15:27:31.917 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.903+0000 7f320d2fa640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32081f3ef0 con 0x7f32080b45c0 2023-05-22T15:27:31.917 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.903+0000 7f3206ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f32080b4170 0x7f320806d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:31.918 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.903+0000 7f3206ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f32080b4170 0x7f320806d1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57292/0 (socket says 172.21.15.16:57292) 2023-05-22T15:27:31.918 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.903+0000 7f3206ffd640 1 -- 172.21.15.16:0/1638207446 learned_addr learned my addr 172.21.15.16:0/1638207446 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:31.918 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.903+0000 7f32077fe640 1 --2- 172.21.15.16:0/1638207446 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3208159980 0x7f32081f73d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:31.919 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.903+0000 7f32067fc640 1 --2- 172.21.15.16:0/1638207446 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f32080b45c0 0x7f32080c4990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:31.919 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.903+0000 7f3206ffd640 1 -- 172.21.15.16:0/1638207446 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f32080b45c0 msgr2=0x7f32080c4990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:31.920 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.903+0000 7f3206ffd640 1 --2- 172.21.15.16:0/1638207446 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f32080b45c0 0x7f32080c4990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:31.920 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.903+0000 7f3206ffd640 1 -- 172.21.15.16:0/1638207446 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3208159980 msgr2=0x7f32081f73d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:31.920 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.903+0000 7f3206ffd640 1 --2- 172.21.15.16:0/1638207446 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3208159980 0x7f32081f73d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:31.920 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.903+0000 7f3206ffd640 1 -- 172.21.15.16:0/1638207446 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31f80094d0 con 0x7f32080b4170 2023-05-22T15:27:31.921 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.903+0000 7f32067fc640 1 --2- 172.21.15.16:0/1638207446 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f32080b45c0 0x7f32080c4990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:27:31.921 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.907+0000 7f32077fe640 1 --2- 172.21.15.16:0/1638207446 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3208159980 0x7f32081f73d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:27:31.921 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.907+0000 7f3206ffd640 1 --2- 172.21.15.16:0/1638207446 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f32080b4170 0x7f320806d1c0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f31f801ab80 tx=0x7f31f801ad10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:31.922 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.907+0000 7f31ef7fe640 1 -- 172.21.15.16:0/1638207446 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31f80aa5f0 con 0x7f32080b4170 2023-05-22T15:27:31.922 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.907+0000 7f31ef7fe640 1 -- 172.21.15.16:0/1638207446 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31f80b3800 con 0x7f32080b4170 2023-05-22T15:27:31.922 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.907+0000 7f31ef7fe640 1 -- 172.21.15.16:0/1638207446 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31f80b3b90 con 0x7f32080b4170 2023-05-22T15:27:31.922 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.907+0000 7f320d2fa640 1 -- 172.21.15.16:0/1638207446 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3208069c60 con 0x7f32080b4170 2023-05-22T15:27:31.923 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.907+0000 7f320d2fa640 1 -- 172.21.15.16:0/1638207446 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32080ce440 con 0x7f32080b4170 2023-05-22T15:27:31.924 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.907+0000 7f320d2fa640 1 -- 172.21.15.16:0/1638207446 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3208159d50 con 0x7f32080b4170 2023-05-22T15:27:31.924 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.907+0000 7f31ef7fe640 1 -- 172.21.15.16:0/1638207446 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7f31f80b3cf0 con 0x7f32080b4170 2023-05-22T15:27:31.924 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.907+0000 7f31ef7fe640 1 --2- 172.21.15.16:0/1638207446 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f31f4065380 0x7f31f4067830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:31.924 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.907+0000 7f31ef7fe640 1 -- 172.21.15.16:0/1638207446 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f31f80a7070 con 0x7f32080b4170 2023-05-22T15:27:31.925 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.907+0000 7f32067fc640 1 --2- 172.21.15.16:0/1638207446 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f31f4065380 0x7f31f4067830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:31.925 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.907+0000 7f32067fc640 1 --2- 172.21.15.16:0/1638207446 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f31f4065380 0x7f31f4067830 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f3208071250 tx=0x7f31fc008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:31.926 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:31.911+0000 7f31ef7fe640 1 -- 172.21.15.16:0/1638207446 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f31f80eb0c0 con 0x7f32080b4170 2023-05-22T15:27:32.074 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.067+0000 7f320d2fa640 1 -- 172.21.15.16:0/1638207446 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f320804dbe0 con 0x7f32080b4170 2023-05-22T15:27:32.074 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.067+0000 7f31ef7fe640 1 -- 172.21.15.16:0/1638207446 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v29) v1 ==== 100+0+11 (secure 0 0 0) 0x7f31f80ee6e0 con 0x7f32080b4170 2023-05-22T15:27:32.075 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 64 2023-05-22T15:27:32.077 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.067+0000 7f320d2fa640 1 -- 172.21.15.16:0/1638207446 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f31f4065380 msgr2=0x7f31f4067830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:32.078 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.067+0000 7f320d2fa640 1 --2- 172.21.15.16:0/1638207446 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f31f4065380 0x7f31f4067830 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f3208071250 tx=0x7f31fc008040 comp rx=0 tx=0).stop 2023-05-22T15:27:32.078 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.067+0000 7f320d2fa640 1 -- 172.21.15.16:0/1638207446 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f32080b4170 msgr2=0x7f320806d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:32.078 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.067+0000 7f320d2fa640 1 --2- 172.21.15.16:0/1638207446 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f32080b4170 0x7f320806d1c0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f31f801ab80 tx=0x7f31f801ad10 comp rx=0 tx=0).stop 2023-05-22T15:27:32.078 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.067+0000 7f320d2fa640 1 -- 172.21.15.16:0/1638207446 shutdown_connections 2023-05-22T15:27:32.078 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.067+0000 7f320d2fa640 1 --2- 172.21.15.16:0/1638207446 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3208159980 0x7f32081f73d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:32.078 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.067+0000 7f320d2fa640 1 --2- 172.21.15.16:0/1638207446 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f31f4065380 0x7f31f4067830 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:32.079 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.067+0000 7f320d2fa640 1 --2- 172.21.15.16:0/1638207446 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f32080b45c0 0x7f32080c4990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:32.079 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.067+0000 7f320d2fa640 1 --2- 172.21.15.16:0/1638207446 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f32080b4170 0x7f320806d1c0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:32.079 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.067+0000 7f320d2fa640 1 -- 172.21.15.16:0/1638207446 >> 172.21.15.16:0/1638207446 conn(0x7f32080ce890 msgr2=0x7f32080b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:32.079 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.067+0000 7f320d2fa640 1 -- 172.21.15.16:0/1638207446 shutdown_connections 2023-05-22T15:27:32.079 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.067+0000 7f320d2fa640 1 -- 172.21.15.16:0/1638207446 wait complete. 2023-05-22T15:27:32.094 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-05-22T15:27:32.226 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.219+0000 7f69e9a01640 1 Processor -- start 2023-05-22T15:27:32.226 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.219+0000 7f69e9a01640 1 -- start start 2023-05-22T15:27:32.227 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.219+0000 7f69e9a01640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f69e4070c70 0x7f69e4071530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:32.227 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.219+0000 7f69e9a01640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f69e415c580 0x7f69e406fba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:32.227 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.219+0000 7f69e9a01640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f69e406b6c0 0x7f69e406bb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:32.228 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.219+0000 7f69e9a01640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69e415d520 con 0x7f69e4070c70 2023-05-22T15:27:32.228 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.219+0000 7f69e9a01640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69e406c040 con 0x7f69e415c580 2023-05-22T15:27:32.228 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.219+0000 7f69e9a01640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69e415ca00 con 0x7f69e406b6c0 2023-05-22T15:27:32.228 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.219+0000 7f69e2ffd640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f69e4070c70 0x7f69e4071530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:32.228 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.219+0000 7f69e2ffd640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f69e4070c70 0x7f69e4071530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57302/0 (socket says 172.21.15.16:57302) 2023-05-22T15:27:32.228 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.219+0000 7f69e2ffd640 1 -- 172.21.15.16:0/2534405970 learned_addr learned my addr 172.21.15.16:0/2534405970 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:32.228 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.219+0000 7f69e2ffd640 1 -- 172.21.15.16:0/2534405970 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f69e406b6c0 msgr2=0x7f69e406bb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:32.229 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.219+0000 7f69e2ffd640 1 --2- 172.21.15.16:0/2534405970 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f69e406b6c0 0x7f69e406bb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:32.229 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.219+0000 7f69e27fc640 1 --2- 172.21.15.16:0/2534405970 >> v2:172.21.15.62:3300/0 conn(0x7f69e415c580 0x7f69e406fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:32.229 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.219+0000 7f69e2ffd640 1 -- 172.21.15.16:0/2534405970 >> v2:172.21.15.62:3300/0 conn(0x7f69e415c580 msgr2=0x7f69e406fba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:32.229 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.219+0000 7f69e2ffd640 1 --2- 172.21.15.16:0/2534405970 >> v2:172.21.15.62:3300/0 conn(0x7f69e415c580 0x7f69e406fba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:32.229 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.219+0000 7f69e2ffd640 1 -- 172.21.15.16:0/2534405970 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69e40c8b10 con 0x7f69e4070c70 2023-05-22T15:27:32.230 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.223+0000 7f69e2ffd640 1 --2- 172.21.15.16:0/2534405970 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f69e4070c70 0x7f69e4071530 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f69e40c8610 tx=0x7f69d401c470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3356b061335198a5 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:32.230 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.223+0000 7f69e1ffb640 1 -- 172.21.15.16:0/2534405970 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69d40a7070 con 0x7f69e4070c70 2023-05-22T15:27:32.230 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.223+0000 7f69e1ffb640 1 -- 172.21.15.16:0/2534405970 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69d40ab430 con 0x7f69e4070c70 2023-05-22T15:27:32.230 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.223+0000 7f69e1ffb640 1 -- 172.21.15.16:0/2534405970 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69d40ab7c0 con 0x7f69e4070c70 2023-05-22T15:27:32.231 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.223+0000 7f69e9a01640 1 -- 172.21.15.16:0/2534405970 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f69e4070c70 msgr2=0x7f69e4071530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:32.231 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.223+0000 7f69e9a01640 1 --2- 172.21.15.16:0/2534405970 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f69e4070c70 0x7f69e4071530 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f69e40c8610 tx=0x7f69d401c470 comp rx=0 tx=0).stop 2023-05-22T15:27:32.231 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.223+0000 7f69e9a01640 1 -- 172.21.15.16:0/2534405970 shutdown_connections 2023-05-22T15:27:32.231 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.223+0000 7f69e9a01640 1 --2- 172.21.15.16:0/2534405970 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f69e406b6c0 0x7f69e406bb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:32.231 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.223+0000 7f69e9a01640 1 --2- 172.21.15.16:0/2534405970 >> v2:172.21.15.62:3300/0 conn(0x7f69e415c580 0x7f69e406fba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:32.232 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.223+0000 7f69e9a01640 1 --2- 172.21.15.16:0/2534405970 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f69e4070c70 0x7f69e4071530 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:32.232 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.223+0000 7f69e9a01640 1 -- 172.21.15.16:0/2534405970 >> 172.21.15.16:0/2534405970 conn(0x7f69e4069720 msgr2=0x7f69e4069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:32.232 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.223+0000 7f69e9a01640 1 -- 172.21.15.16:0/2534405970 shutdown_connections 2023-05-22T15:27:32.232 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.223+0000 7f69e9a01640 1 -- 172.21.15.16:0/2534405970 wait complete. 2023-05-22T15:27:32.233 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.223+0000 7f69e9a01640 1 Processor -- start 2023-05-22T15:27:32.233 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.223+0000 7f69e9a01640 1 -- start start 2023-05-22T15:27:32.233 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.223+0000 7f69e9a01640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f69e406b6c0 0x7f69e41560d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:32.234 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.223+0000 7f69e9a01640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f69e4070c70 0x7f69e4069d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:32.234 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.223+0000 7f69e9a01640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f69e415c580 0x7f69e4151fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:32.234 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.223+0000 7f69e9a01640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69e40c7ec0 con 0x7f69e415c580 2023-05-22T15:27:32.234 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.223+0000 7f69e9a01640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69e41537c0 con 0x7f69e4070c70 2023-05-22T15:27:32.234 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.223+0000 7f69e9a01640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69e4156610 con 0x7f69e406b6c0 2023-05-22T15:27:32.234 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.223+0000 7f69e37fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f69e415c580 0x7f69e4151fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:32.234 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.223+0000 7f69e37fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f69e415c580 0x7f69e4151fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57308/0 (socket says 172.21.15.16:57308) 2023-05-22T15:27:32.235 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.223+0000 7f69e37fe640 1 -- 172.21.15.16:0/3263117112 learned_addr learned my addr 172.21.15.16:0/3263117112 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:32.235 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.223+0000 7f69e27fc640 1 --2- 172.21.15.16:0/3263117112 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f69e4070c70 0x7f69e4069d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:32.235 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.223+0000 7f69e2ffd640 1 --2- 172.21.15.16:0/3263117112 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f69e406b6c0 0x7f69e41560d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:32.235 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.227+0000 7f69e37fe640 1 -- 172.21.15.16:0/3263117112 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f69e406b6c0 msgr2=0x7f69e41560d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:32.235 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.227+0000 7f69e37fe640 1 --2- 172.21.15.16:0/3263117112 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f69e406b6c0 0x7f69e41560d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:32.235 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.227+0000 7f69e37fe640 1 -- 172.21.15.16:0/3263117112 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f69e4070c70 msgr2=0x7f69e4069d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:32.235 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.227+0000 7f69e37fe640 1 --2- 172.21.15.16:0/3263117112 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f69e4070c70 0x7f69e4069d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:32.235 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.227+0000 7f69e37fe640 1 -- 172.21.15.16:0/3263117112 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69d40094d0 con 0x7f69e415c580 2023-05-22T15:27:32.236 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.227+0000 7f69e27fc640 1 --2- 172.21.15.16:0/3263117112 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f69e4070c70 0x7f69e4069d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T15:27:32.236 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.227+0000 7f69e2ffd640 1 --2- 172.21.15.16:0/3263117112 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f69e406b6c0 0x7f69e41560d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T15:27:32.236 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.227+0000 7f69e37fe640 1 --2- 172.21.15.16:0/3263117112 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f69e415c580 0x7f69e4151fa0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f69dc00b970 tx=0x7f69dc00be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:32.237 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.227+0000 7f69cbfff640 1 -- 172.21.15.16:0/3263117112 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69dc00cb40 con 0x7f69e415c580 2023-05-22T15:27:32.237 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.227+0000 7f69cbfff640 1 -- 172.21.15.16:0/3263117112 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69dc00ccc0 con 0x7f69e415c580 2023-05-22T15:27:32.237 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.227+0000 7f69e9a01640 1 -- 172.21.15.16:0/3263117112 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69e415d240 con 0x7f69e415c580 2023-05-22T15:27:32.237 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.227+0000 7f69cbfff640 1 -- 172.21.15.16:0/3263117112 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69dc014680 con 0x7f69e415c580 2023-05-22T15:27:32.238 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.227+0000 7f69e9a01640 1 -- 172.21.15.16:0/3263117112 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69e415cf60 con 0x7f69e415c580 2023-05-22T15:27:32.238 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.227+0000 7f69e9a01640 1 -- 172.21.15.16:0/3263117112 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69e40cc510 con 0x7f69e415c580 2023-05-22T15:27:32.238 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.227+0000 7f69cbfff640 1 -- 172.21.15.16:0/3263117112 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7f69dc01d3b0 con 0x7f69e415c580 2023-05-22T15:27:32.238 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.227+0000 7f69cbfff640 1 --2- 172.21.15.16:0/3263117112 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f69d0065380 0x7f69d0067830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:32.238 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.227+0000 7f69cbfff640 1 -- 172.21.15.16:0/3263117112 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f69dc082b60 con 0x7f69e415c580 2023-05-22T15:27:32.238 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.227+0000 7f69e2ffd640 1 --2- 172.21.15.16:0/3263117112 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f69d0065380 0x7f69d0067830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:32.238 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.227+0000 7f69e2ffd640 1 --2- 172.21.15.16:0/3263117112 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f69d0065380 0x7f69d0067830 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f69d40af000 tx=0x7f69d4008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:32.240 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.231+0000 7f69cbfff640 1 -- 172.21.15.16:0/3263117112 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f69dc050450 con 0x7f69e415c580 2023-05-22T15:27:32.392 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.383+0000 7f69e9a01640 1 -- 172.21.15.16:0/3263117112 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f69e4158de0 con 0x7f69e415c580 2023-05-22T15:27:32.393 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.387+0000 7f69cbfff640 1 -- 172.21.15.16:0/3263117112 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v29) v1 ==== 103+0+11 (secure 0 0 0) 0x7f69dc019020 con 0x7f69e415c580 2023-05-22T15:27:32.393 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2023-05-22T15:27:32.395 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.387+0000 7f69e9a01640 1 -- 172.21.15.16:0/3263117112 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f69d0065380 msgr2=0x7f69d0067830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:32.396 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.387+0000 7f69e9a01640 1 --2- 172.21.15.16:0/3263117112 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f69d0065380 0x7f69d0067830 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f69d40af000 tx=0x7f69d4008040 comp rx=0 tx=0).stop 2023-05-22T15:27:32.396 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.387+0000 7f69e9a01640 1 -- 172.21.15.16:0/3263117112 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f69e415c580 msgr2=0x7f69e4151fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:32.396 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.387+0000 7f69e9a01640 1 --2- 172.21.15.16:0/3263117112 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f69e415c580 0x7f69e4151fa0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f69dc00b970 tx=0x7f69dc00be30 comp rx=0 tx=0).stop 2023-05-22T15:27:32.396 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.387+0000 7f69e9a01640 1 -- 172.21.15.16:0/3263117112 shutdown_connections 2023-05-22T15:27:32.396 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.387+0000 7f69e9a01640 1 --2- 172.21.15.16:0/3263117112 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f69e415c580 0x7f69e4151fa0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:32.396 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.387+0000 7f69e9a01640 1 --2- 172.21.15.16:0/3263117112 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f69e4070c70 0x7f69e4069d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:32.397 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.387+0000 7f69e9a01640 1 --2- 172.21.15.16:0/3263117112 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f69d0065380 0x7f69d0067830 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:32.397 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.387+0000 7f69e9a01640 1 --2- 172.21.15.16:0/3263117112 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f69e406b6c0 0x7f69e41560d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:32.397 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.387+0000 7f69e9a01640 1 -- 172.21.15.16:0/3263117112 >> 172.21.15.16:0/3263117112 conn(0x7f69e4069720 msgr2=0x7f69e40cb350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:32.397 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.387+0000 7f69e9a01640 1 -- 172.21.15.16:0/3263117112 shutdown_connections 2023-05-22T15:27:32.397 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.387+0000 7f69e9a01640 1 -- 172.21.15.16:0/3263117112 wait complete. 2023-05-22T15:27:32.413 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-05-22T15:27:32.541 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.535+0000 7f5b0e38d640 1 Processor -- start 2023-05-22T15:27:32.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.535+0000 7f5b0e38d640 1 -- start start 2023-05-22T15:27:32.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.535+0000 7f5b0e38d640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f5b08158290 0x7f5b080c8880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:32.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.535+0000 7f5b0e38d640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f5b080ce6c0 0x7f5b0806f630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:32.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.535+0000 7f5b0e38d640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b080c92b0 con 0x7f5b08158660 2023-05-22T15:27:32.544 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.535+0000 7f5b0e38d640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b080cc060 con 0x7f5b080ce6c0 2023-05-22T15:27:32.544 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.535+0000 7f5b0e38d640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b080cc340 con 0x7f5b08158290 2023-05-22T15:27:32.544 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.535+0000 7f5b0c92a640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f5b080ce6c0 0x7f5b0806f630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:32.544 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.535+0000 7f5b0c92a640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f5b080ce6c0 0x7f5b0806f630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57318/0 (socket says 172.21.15.16:57318) 2023-05-22T15:27:32.544 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.535+0000 7f5b0c92a640 1 -- 172.21.15.16:0/1370720427 learned_addr learned my addr 172.21.15.16:0/1370720427 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:32.544 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.535+0000 7f5b07fff640 1 --2- 172.21.15.16:0/1370720427 >> v2:172.21.15.62:3300/0 conn(0x7f5b08158290 0x7f5b080c8880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:32.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.535+0000 7f5b0c92a640 1 -- 172.21.15.16:0/1370720427 >> v2:172.21.15.62:3300/0 conn(0x7f5b08158290 msgr2=0x7f5b080c8880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:32.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.535+0000 7f5b0c92a640 1 --2- 172.21.15.16:0/1370720427 >> v2:172.21.15.62:3300/0 conn(0x7f5b08158290 0x7f5b080c8880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:32.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.535+0000 7f5b0c92a640 1 -- 172.21.15.16:0/1370720427 >> v1:172.21.15.16:6789/0 conn(0x7f5b08158660 legacy=0x7f5b080cdfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:32.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.535+0000 7f5b0c92a640 1 -- 172.21.15.16:0/1370720427 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b080cc620 con 0x7f5b080ce6c0 2023-05-22T15:27:32.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.539+0000 7f5b0c92a640 1 --2- 172.21.15.16:0/1370720427 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b080ce6c0 0x7f5b0806f630 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f5b080caa00 tx=0x7f5b0001ebb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34853227572f47e server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:32.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.539+0000 7f5b06ffd640 1 -- 172.21.15.16:0/1370720427 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b000a7aa0 con 0x7f5b080ce6c0 2023-05-22T15:27:32.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.539+0000 7f5b06ffd640 1 -- 172.21.15.16:0/1370720427 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b000a7c00 con 0x7f5b080ce6c0 2023-05-22T15:27:32.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.539+0000 7f5b06ffd640 1 -- 172.21.15.16:0/1370720427 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b000af3f0 con 0x7f5b080ce6c0 2023-05-22T15:27:32.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.539+0000 7f5b0e38d640 1 -- 172.21.15.16:0/1370720427 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b080ce6c0 msgr2=0x7f5b0806f630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:32.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.539+0000 7f5b0e38d640 1 --2- 172.21.15.16:0/1370720427 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b080ce6c0 0x7f5b0806f630 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f5b080caa00 tx=0x7f5b0001ebb0 comp rx=0 tx=0).stop 2023-05-22T15:27:32.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.539+0000 7f5b0e38d640 1 -- 172.21.15.16:0/1370720427 shutdown_connections 2023-05-22T15:27:32.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.539+0000 7f5b0e38d640 1 --2- 172.21.15.16:0/1370720427 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b080ce6c0 0x7f5b0806f630 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:32.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.539+0000 7f5b0e38d640 1 --2- 172.21.15.16:0/1370720427 >> v2:172.21.15.62:3300/0 conn(0x7f5b08158290 0x7f5b080c8880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:32.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.539+0000 7f5b0e38d640 1 -- 172.21.15.16:0/1370720427 >> 172.21.15.16:0/1370720427 conn(0x7f5b08069720 msgr2=0x7f5b08069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:32.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.539+0000 7f5b0e38d640 1 -- 172.21.15.16:0/1370720427 shutdown_connections 2023-05-22T15:27:32.549 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.539+0000 7f5b0e38d640 1 -- 172.21.15.16:0/1370720427 wait complete. 2023-05-22T15:27:32.551 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.539+0000 7f5b0e38d640 1 Processor -- start 2023-05-22T15:27:32.551 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.539+0000 7f5b0e38d640 1 -- start start 2023-05-22T15:27:32.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.539+0000 7f5b0e38d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b080ce6c0 0x7f5b08152be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:32.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.539+0000 7f5b0e38d640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5b08158290 0x7f5b081f4410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:32.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.539+0000 7f5b0e38d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5b08158660 0x7f5b080c6ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:32.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.539+0000 7f5b0e38d640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b08153350 con 0x7f5b080ce6c0 2023-05-22T15:27:32.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.539+0000 7f5b0e38d640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b080b3170 con 0x7f5b08158660 2023-05-22T15:27:32.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.539+0000 7f5b0e38d640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b08153120 con 0x7f5b08158290 2023-05-22T15:27:32.554 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.539+0000 7f5b07fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b080ce6c0 0x7f5b08152be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:32.554 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.539+0000 7f5b07fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b080ce6c0 0x7f5b08152be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57326/0 (socket says 172.21.15.16:57326) 2023-05-22T15:27:32.554 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.539+0000 7f5b07fff640 1 -- 172.21.15.16:0/701358078 learned_addr learned my addr 172.21.15.16:0/701358078 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:32.554 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.543+0000 7f5b0c92a640 1 --2- 172.21.15.16:0/701358078 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5b08158660 0x7f5b080c6ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:32.554 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.543+0000 7f5b077fe640 1 --2- 172.21.15.16:0/701358078 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5b08158290 0x7f5b081f4410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:32.554 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.543+0000 7f5b07fff640 1 -- 172.21.15.16:0/701358078 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5b08158290 msgr2=0x7f5b081f4410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:32.554 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.543+0000 7f5b07fff640 1 --2- 172.21.15.16:0/701358078 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5b08158290 0x7f5b081f4410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:32.555 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.543+0000 7f5b07fff640 1 -- 172.21.15.16:0/701358078 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5b08158660 msgr2=0x7f5b080c6ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:32.555 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.543+0000 7f5b07fff640 1 --2- 172.21.15.16:0/701358078 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5b08158660 0x7f5b080c6ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:32.555 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.543+0000 7f5b07fff640 1 -- 172.21.15.16:0/701358078 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b00009b30 con 0x7f5b080ce6c0 2023-05-22T15:27:32.555 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.543+0000 7f5b07fff640 1 --2- 172.21.15.16:0/701358078 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b080ce6c0 0x7f5b08152be0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f5afc00d830 tx=0x7f5afc00dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:32.555 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.543+0000 7f5b04ff9640 1 -- 172.21.15.16:0/701358078 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5afc00b6c0 con 0x7f5b080ce6c0 2023-05-22T15:27:32.555 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.543+0000 7f5b04ff9640 1 -- 172.21.15.16:0/701358078 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5afc00b820 con 0x7f5b080ce6c0 2023-05-22T15:27:32.555 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.543+0000 7f5b0e38d640 1 -- 172.21.15.16:0/701358078 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b080ccfb0 con 0x7f5b080ce6c0 2023-05-22T15:27:32.556 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.543+0000 7f5b04ff9640 1 -- 172.21.15.16:0/701358078 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5afc00bb90 con 0x7f5b080ce6c0 2023-05-22T15:27:32.556 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.543+0000 7f5b0e38d640 1 -- 172.21.15.16:0/701358078 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b080cce20 con 0x7f5b080ce6c0 2023-05-22T15:27:32.556 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.543+0000 7f5b0e38d640 1 -- 172.21.15.16:0/701358078 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b080c8dc0 con 0x7f5b080ce6c0 2023-05-22T15:27:32.556 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.543+0000 7f5b04ff9640 1 -- 172.21.15.16:0/701358078 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7f5afc01e460 con 0x7f5b080ce6c0 2023-05-22T15:27:32.557 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.543+0000 7f5b04ff9640 1 --2- 172.21.15.16:0/701358078 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f5af4065380 0x7f5af4067830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:32.557 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.543+0000 7f5b04ff9640 1 -- 172.21.15.16:0/701358078 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f5afc083410 con 0x7f5b080ce6c0 2023-05-22T15:27:32.557 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.543+0000 7f5b077fe640 1 --2- 172.21.15.16:0/701358078 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f5af4065380 0x7f5af4067830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:32.557 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.543+0000 7f5b077fe640 1 --2- 172.21.15.16:0/701358078 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f5af4065380 0x7f5af4067830 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f5b08151740 tx=0x7f5af8009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:32.557 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.547+0000 7f5b04ff9640 1 -- 172.21.15.16:0/701358078 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f5afc050c60 con 0x7f5b080ce6c0 2023-05-22T15:27:32.711 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.703+0000 7f5b0e38d640 1 -- 172.21.15.16:0/701358078 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5b0804dbe0 con 0x7f5b080ce6c0 2023-05-22T15:27:32.714 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.707+0000 7f5b04ff9640 1 -- 172.21.15.16:0/701358078 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v29) v1 ==== 74+0+19700 (secure 0 0 0) 0x7f5afc054280 con 0x7f5b080ce6c0 2023-05-22T15:27:32.714 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:27:32.715 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":29,"fsid":"b2719834-7847-4d98-90a6-f65d57f85c49","created":"2023-05-22T15:26:25.259314+0000","modified":"2023-05-22T15:27:31.182317+0000","last_up_change":"2023-05-22T15:26:36.448524+0000","last_in_change":"2023-05-22T15:26:27.675668+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-05-22T15:26:38.019781+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-05-22T15:26:43.570514+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-05-22T15:26:47.879159+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-05-22T15:26:48.522130+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data_ec","create_time":"2023-05-22T15:26:51.616142+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"9e7ff854-b25f-4cbb-a5c2-1799d9c53d2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":37021},{"type":"v1","addr":"172.21.15.16:6825","nonce":37021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":37021},{"type":"v1","addr":"172.21.15.16:6827","nonce":37021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":37021},{"type":"v1","addr":"172.21.15.16:6831","nonce":37021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":37021},{"type":"v1","addr":"172.21.15.16:6829","nonce":37021}]},"public_addr":"172.21.15.16:6825/37021","cluster_addr":"172.21.15.16:6827/37021","heartbeat_back_addr":"172.21.15.16:6831/37021","heartbeat_front_addr":"172.21.15.16:6829/37021","state":["exists","up"]},{"osd":1,"uuid":"4702fbc9-a927-4565-aca5-41dc69cc8264","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":37008},{"type":"v1","addr":"172.21.15.16:6801","nonce":37008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":37008},{"type":"v1","addr":"172.21.15.16:6803","nonce":37008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":37008},{"type":"v1","addr":"172.21.15.16:6807","nonce":37008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":37008},{"type":"v1","addr":"172.21.15.16:6805","nonce":37008}]},"public_addr":"172.21.15.16:6801/37008","cluster_addr":"172.21.15.16:6803/37008","heartbeat_back_addr":"172.21.15.16:6807/37008","heartbeat_front_addr":"172.21.15.16:6805/37008","state":["exists","up"]},{"osd":2,"uuid":"1e7bba46-4eba-47ef-bb81-5e2d371638ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":37009},{"type":"v1","addr":"172.21.15.16:6809","nonce":37009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":37009},{"type":"v1","addr":"172.21.15.16:6811","nonce":37009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":37009},{"type":"v1","addr":"172.21.15.16:6815","nonce":37009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":37009},{"type":"v1","addr":"172.21.15.16:6813","nonce":37009}]},"public_addr":"172.21.15.16:6809/37009","cluster_addr":"172.21.15.16:6811/37009","heartbeat_back_addr":"172.21.15.16:6815/37009","heartbeat_front_addr":"172.21.15.16:6813/37009","state":["exists","up"]},{"osd":3,"uuid":"383cacc0-5a88-4904-990e-498ca59ef48a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":37020},{"type":"v1","addr":"172.21.15.16:6817","nonce":37020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":37020},{"type":"v1","addr":"172.21.15.16:6819","nonce":37020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":37020},{"type":"v1","addr":"172.21.15.16:6823","nonce":37020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":37020},{"type":"v1","addr":"172.21.15.16:6821","nonce":37020}]},"public_addr":"172.21.15.16:6817/37020","cluster_addr":"172.21.15.16:6819/37020","heartbeat_back_addr":"172.21.15.16:6823/37020","heartbeat_front_addr":"172.21.15.16:6821/37020","state":["exists","up"]},{"osd":4,"uuid":"970500ef-95b7-42c1-b080-eed4fae7013d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":37353},{"type":"v1","addr":"172.21.15.62:6817","nonce":37353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6819","nonce":37353},{"type":"v1","addr":"172.21.15.62:6822","nonce":37353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37353},{"type":"v1","addr":"172.21.15.62:6831","nonce":37353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37353},{"type":"v1","addr":"172.21.15.62:6826","nonce":37353}]},"public_addr":"172.21.15.62:6817/37353","cluster_addr":"172.21.15.62:6822/37353","heartbeat_back_addr":"172.21.15.62:6831/37353","heartbeat_front_addr":"172.21.15.62:6826/37353","state":["exists","up"]},{"osd":5,"uuid":"022d67db-d4d8-445b-b1b9-8b446a00f5c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":37352},{"type":"v1","addr":"172.21.15.62:6801","nonce":37352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":37352},{"type":"v1","addr":"172.21.15.62:6803","nonce":37352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":37352},{"type":"v1","addr":"172.21.15.62:6807","nonce":37352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":37352},{"type":"v1","addr":"172.21.15.62:6805","nonce":37352}]},"public_addr":"172.21.15.62:6801/37352","cluster_addr":"172.21.15.62:6803/37352","heartbeat_back_addr":"172.21.15.62:6807/37352","heartbeat_front_addr":"172.21.15.62:6805/37352","state":["exists","up"]},{"osd":6,"uuid":"42bd372b-6688-4a1b-8407-e16e8c266e33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":37354},{"type":"v1","addr":"172.21.15.62:6820","nonce":37354}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":37354},{"type":"v1","addr":"172.21.15.62:6823","nonce":37354}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6829","nonce":37354},{"type":"v1","addr":"172.21.15.62:6830","nonce":37354}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6825","nonce":37354},{"type":"v1","addr":"172.21.15.62:6827","nonce":37354}]},"public_addr":"172.21.15.62:6820/37354","cluster_addr":"172.21.15.62:6823/37354","heartbeat_back_addr":"172.21.15.62:6830/37354","heartbeat_front_addr":"172.21.15.62:6827/37354","state":["exists","up"]},{"osd":7,"uuid":"ac70b2a3-f41b-4bf2-b227-7f2cbb55426d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37355},{"type":"v1","addr":"172.21.15.62:6809","nonce":37355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37355},{"type":"v1","addr":"172.21.15.62:6811","nonce":37355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":37355},{"type":"v1","addr":"172.21.15.62:6815","nonce":37355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":37355},{"type":"v1","addr":"172.21.15.62:6813","nonce":37355}]},"public_addr":"172.21.15.62:6809/37355","cluster_addr":"172.21.15.62:6811/37355","heartbeat_back_addr":"172.21.15.62:6815/37355","heartbeat_front_addr":"172.21.15.62:6813/37355","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.368896+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.107273+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.276888+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.287502+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.124791+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.251937+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.255679+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.232797+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"4.23","mappings":[{"from":3,"to":2}]},{"pgid":"4.31","mappings":[{"from":1,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-05-22T15:27:32.716 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.707+0000 7f5b0e38d640 1 -- 172.21.15.16:0/701358078 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f5af4065380 msgr2=0x7f5af4067830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:32.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.707+0000 7f5b0e38d640 1 --2- 172.21.15.16:0/701358078 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f5af4065380 0x7f5af4067830 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f5b08151740 tx=0x7f5af8009040 comp rx=0 tx=0).stop 2023-05-22T15:27:32.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.707+0000 7f5b0e38d640 1 -- 172.21.15.16:0/701358078 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b080ce6c0 msgr2=0x7f5b08152be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:32.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.707+0000 7f5b0e38d640 1 --2- 172.21.15.16:0/701358078 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b080ce6c0 0x7f5b08152be0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f5afc00d830 tx=0x7f5afc00dcf0 comp rx=0 tx=0).stop 2023-05-22T15:27:32.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.707+0000 7f5b0e38d640 1 -- 172.21.15.16:0/701358078 shutdown_connections 2023-05-22T15:27:32.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.707+0000 7f5b0e38d640 1 --2- 172.21.15.16:0/701358078 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5b08158660 0x7f5b080c6ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:32.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.707+0000 7f5b0e38d640 1 --2- 172.21.15.16:0/701358078 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f5af4065380 0x7f5af4067830 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:32.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.707+0000 7f5b0e38d640 1 --2- 172.21.15.16:0/701358078 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5b08158290 0x7f5b081f4410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:32.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.707+0000 7f5b0e38d640 1 --2- 172.21.15.16:0/701358078 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b080ce6c0 0x7f5b08152be0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:32.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.707+0000 7f5b0e38d640 1 -- 172.21.15.16:0/701358078 >> 172.21.15.16:0/701358078 conn(0x7f5b08069720 msgr2=0x7f5b080cb3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:32.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.707+0000 7f5b0e38d640 1 -- 172.21.15.16:0/701358078 shutdown_connections 2023-05-22T15:27:32.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.707+0000 7f5b0e38d640 1 -- 172.21.15.16:0/701358078 wait complete. 2023-05-22T15:27:32.733 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-05-22T15:26:38.019781+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-05-22T15:26:43.570514+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2023-05-22T15:26:47.879159+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2023-05-22T15:26:48.522130+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-05-22T15:26:51.616142+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '27', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-05-22T15:27:32.734 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-05-22T15:27:32.857 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.847+0000 7f1f6f558640 1 Processor -- start 2023-05-22T15:27:32.857 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.847+0000 7f1f6f558640 1 -- start start 2023-05-22T15:27:32.858 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.851+0000 7f1f6f558640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f1f6806b6c0 0x7f1f680cb4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:32.858 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.851+0000 7f1f6f558640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f1f6806ba90 0x7f1f68158fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:32.859 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.851+0000 7f1f6f558640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1f68159510 0x7f1f680cc740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:32.859 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.851+0000 7f1f6f558640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f68071760 con 0x7f1f6806b6c0 2023-05-22T15:27:32.859 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.851+0000 7f1f6f558640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f680ca7d0 con 0x7f1f6806ba90 2023-05-22T15:27:32.859 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.851+0000 7f1f6f558640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f68159f70 con 0x7f1f68159510 2023-05-22T15:27:32.860 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.851+0000 7f1f6d2f4640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f1f6806b6c0 0x7f1f680cb4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:32.860 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.851+0000 7f1f6d2f4640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f1f6806b6c0 0x7f1f680cb4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57328/0 (socket says 172.21.15.16:57328) 2023-05-22T15:27:32.860 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.851+0000 7f1f6d2f4640 1 -- 172.21.15.16:0/29194593 learned_addr learned my addr 172.21.15.16:0/29194593 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:32.860 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.851+0000 7f1f6caf3640 1 --2- 172.21.15.16:0/29194593 >> v2:172.21.15.62:3300/0 conn(0x7f1f6806ba90 0x7f1f68158fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:32.860 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.851+0000 7f1f6daf5640 1 --2- 172.21.15.16:0/29194593 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1f68159510 0x7f1f680cc740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:32.860 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.851+0000 7f1f6d2f4640 1 -- 172.21.15.16:0/29194593 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1f68159510 msgr2=0x7f1f680cc740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:32.861 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.851+0000 7f1f6d2f4640 1 --2- 172.21.15.16:0/29194593 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1f68159510 0x7f1f680cc740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:32.861 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.851+0000 7f1f6d2f4640 1 -- 172.21.15.16:0/29194593 >> v2:172.21.15.62:3300/0 conn(0x7f1f6806ba90 msgr2=0x7f1f68158fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:32.861 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.851+0000 7f1f6d2f4640 1 --2- 172.21.15.16:0/29194593 >> v2:172.21.15.62:3300/0 conn(0x7f1f6806ba90 0x7f1f68158fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:32.861 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.851+0000 7f1f6d2f4640 1 -- 172.21.15.16:0/29194593 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f680cb030 con 0x7f1f6806b6c0 2023-05-22T15:27:32.862 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.851+0000 7f1f6d2f4640 1 --2- 172.21.15.16:0/29194593 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f6806b6c0 0x7f1f680cb4c0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f1f6806daf0 tx=0x7f1f6401c6e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f34ce4ff2d6c7a69 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:32.862 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.851+0000 7f1f5ffff640 1 -- 172.21.15.16:0/29194593 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f640a5560 con 0x7f1f6806b6c0 2023-05-22T15:27:32.862 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.851+0000 7f1f5ffff640 1 -- 172.21.15.16:0/29194593 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f640a56c0 con 0x7f1f6806b6c0 2023-05-22T15:27:32.863 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.851+0000 7f1f5ffff640 1 -- 172.21.15.16:0/29194593 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f640a5a50 con 0x7f1f6806b6c0 2023-05-22T15:27:32.863 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.855+0000 7f1f6f558640 1 -- 172.21.15.16:0/29194593 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f6806b6c0 msgr2=0x7f1f680cb4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:32.863 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.855+0000 7f1f6f558640 1 --2- 172.21.15.16:0/29194593 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f6806b6c0 0x7f1f680cb4c0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f1f6806daf0 tx=0x7f1f6401c6e0 comp rx=0 tx=0).stop 2023-05-22T15:27:32.863 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.855+0000 7f1f6f558640 1 -- 172.21.15.16:0/29194593 shutdown_connections 2023-05-22T15:27:32.863 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.855+0000 7f1f6f558640 1 --2- 172.21.15.16:0/29194593 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1f68159510 0x7f1f680cc740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:32.863 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.855+0000 7f1f6f558640 1 --2- 172.21.15.16:0/29194593 >> v2:172.21.15.62:3300/0 conn(0x7f1f6806ba90 0x7f1f68158fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:32.864 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.855+0000 7f1f6f558640 1 --2- 172.21.15.16:0/29194593 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f6806b6c0 0x7f1f680cb4c0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:32.864 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.855+0000 7f1f6f558640 1 -- 172.21.15.16:0/29194593 >> 172.21.15.16:0/29194593 conn(0x7f1f68069720 msgr2=0x7f1f68069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:32.864 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.855+0000 7f1f6f558640 1 -- 172.21.15.16:0/29194593 shutdown_connections 2023-05-22T15:27:32.864 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.855+0000 7f1f6f558640 1 -- 172.21.15.16:0/29194593 wait complete. 2023-05-22T15:27:32.866 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.855+0000 7f1f6f558640 1 Processor -- start 2023-05-22T15:27:32.866 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.855+0000 7f1f6f558640 1 -- start start 2023-05-22T15:27:32.866 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.855+0000 7f1f6f558640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1f6806b6c0 0x7f1f6814d590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:32.867 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.855+0000 7f1f6f558640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f6806ba90 0x7f1f6814e780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:32.867 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.855+0000 7f1f6f558640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1f68159510 0x7f1f68154870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:32.867 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.855+0000 7f1f6f558640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f6814dad0 con 0x7f1f6806ba90 2023-05-22T15:27:32.867 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.855+0000 7f1f6f558640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f680af160 con 0x7f1f6806b6c0 2023-05-22T15:27:32.867 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.855+0000 7f1f6f558640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f680af440 con 0x7f1f68159510 2023-05-22T15:27:32.868 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.855+0000 7f1f6caf3640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f6806ba90 0x7f1f6814e780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:32.868 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.855+0000 7f1f6caf3640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f6806ba90 0x7f1f6814e780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57332/0 (socket says 172.21.15.16:57332) 2023-05-22T15:27:32.868 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.855+0000 7f1f6caf3640 1 -- 172.21.15.16:0/2420676683 learned_addr learned my addr 172.21.15.16:0/2420676683 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:32.868 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.855+0000 7f1f6d2f4640 1 --2- 172.21.15.16:0/2420676683 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1f6806b6c0 0x7f1f6814d590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:32.868 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.855+0000 7f1f6daf5640 1 --2- 172.21.15.16:0/2420676683 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1f68159510 0x7f1f68154870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:32.868 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.855+0000 7f1f6caf3640 1 -- 172.21.15.16:0/2420676683 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1f68159510 msgr2=0x7f1f68154870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:32.868 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.855+0000 7f1f6caf3640 1 --2- 172.21.15.16:0/2420676683 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1f68159510 0x7f1f68154870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:32.869 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.855+0000 7f1f6caf3640 1 -- 172.21.15.16:0/2420676683 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1f6806b6c0 msgr2=0x7f1f6814d590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:32.869 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.855+0000 7f1f6caf3640 1 --2- 172.21.15.16:0/2420676683 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1f6806b6c0 0x7f1f6814d590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:32.869 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.855+0000 7f1f6caf3640 1 -- 172.21.15.16:0/2420676683 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f64009b30 con 0x7f1f6806ba90 2023-05-22T15:27:32.870 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.855+0000 7f1f6daf5640 1 --2- 172.21.15.16:0/2420676683 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1f68159510 0x7f1f68154870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:27:32.870 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.855+0000 7f1f6caf3640 1 --2- 172.21.15.16:0/2420676683 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f6806ba90 0x7f1f6814e780 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f1f5800d800 tx=0x7f1f5800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:32.871 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.855+0000 7f1f5dffb640 1 -- 172.21.15.16:0/2420676683 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f580096a0 con 0x7f1f6806ba90 2023-05-22T15:27:32.871 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.855+0000 7f1f5dffb640 1 -- 172.21.15.16:0/2420676683 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f58009800 con 0x7f1f6806ba90 2023-05-22T15:27:32.871 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.855+0000 7f1f6f558640 1 -- 172.21.15.16:0/2420676683 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f68159910 con 0x7f1f6806ba90 2023-05-22T15:27:32.871 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.855+0000 7f1f5dffb640 1 -- 172.21.15.16:0/2420676683 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f58009b70 con 0x7f1f6806ba90 2023-05-22T15:27:32.871 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.855+0000 7f1f6f558640 1 -- 172.21.15.16:0/2420676683 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f680cacc0 con 0x7f1f6806ba90 2023-05-22T15:27:32.871 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.859+0000 7f1f6f558640 1 -- 172.21.15.16:0/2420676683 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f6806be60 con 0x7f1f6806ba90 2023-05-22T15:27:32.871 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.859+0000 7f1f5dffb640 1 -- 172.21.15.16:0/2420676683 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7f1f58009cd0 con 0x7f1f6806ba90 2023-05-22T15:27:32.872 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.859+0000 7f1f5dffb640 1 --2- 172.21.15.16:0/2420676683 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f1f4c0653b0 0x7f1f4c067860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:32.873 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.859+0000 7f1f5dffb640 1 -- 172.21.15.16:0/2420676683 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f1f58083230 con 0x7f1f6806ba90 2023-05-22T15:27:32.873 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.859+0000 7f1f6d2f4640 1 --2- 172.21.15.16:0/2420676683 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f1f4c0653b0 0x7f1f4c067860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:32.873 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.859+0000 7f1f6d2f4640 1 --2- 172.21.15.16:0/2420676683 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f1f4c0653b0 0x7f1f4c067860 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f1f640ae000 tx=0x7f1f640a7070 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:32.873 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:32.863+0000 7f1f5dffb640 1 -- 172.21.15.16:0/2420676683 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f1f58050c10 con 0x7f1f6806ba90 2023-05-22T15:27:33.022 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.015+0000 7f1f6f558640 1 -- 172.21.15.16:0/2420676683 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f1f6804dbe0 con 0x7f1f6806ba90 2023-05-22T15:27:33.022 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.015+0000 7f1f5dffb640 1 -- 172.21.15.16:0/2420676683 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v29) v1 ==== 93+0+10 (secure 0 0 0) 0x7f1f58054230 con 0x7f1f6806ba90 2023-05-22T15:27:33.023 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2023-05-22T15:27:33.025 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.015+0000 7f1f6f558640 1 -- 172.21.15.16:0/2420676683 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f1f4c0653b0 msgr2=0x7f1f4c067860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:33.025 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.015+0000 7f1f6f558640 1 --2- 172.21.15.16:0/2420676683 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f1f4c0653b0 0x7f1f4c067860 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f1f640ae000 tx=0x7f1f640a7070 comp rx=0 tx=0).stop 2023-05-22T15:27:33.025 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.015+0000 7f1f6f558640 1 -- 172.21.15.16:0/2420676683 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f6806ba90 msgr2=0x7f1f6814e780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:33.025 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.015+0000 7f1f6f558640 1 --2- 172.21.15.16:0/2420676683 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f6806ba90 0x7f1f6814e780 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f1f5800d800 tx=0x7f1f5800dcc0 comp rx=0 tx=0).stop 2023-05-22T15:27:33.026 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.015+0000 7f1f6f558640 1 -- 172.21.15.16:0/2420676683 shutdown_connections 2023-05-22T15:27:33.026 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.015+0000 7f1f6f558640 1 --2- 172.21.15.16:0/2420676683 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f1f4c0653b0 0x7f1f4c067860 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:33.026 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.015+0000 7f1f6f558640 1 --2- 172.21.15.16:0/2420676683 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1f68159510 0x7f1f68154870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:33.026 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.015+0000 7f1f6f558640 1 --2- 172.21.15.16:0/2420676683 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f6806ba90 0x7f1f6814e780 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:33.026 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.015+0000 7f1f6f558640 1 --2- 172.21.15.16:0/2420676683 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1f6806b6c0 0x7f1f6814d590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:33.026 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.015+0000 7f1f6f558640 1 -- 172.21.15.16:0/2420676683 >> 172.21.15.16:0/2420676683 conn(0x7f1f68069720 msgr2=0x7f1f6815bae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:33.026 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.015+0000 7f1f6f558640 1 -- 172.21.15.16:0/2420676683 shutdown_connections 2023-05-22T15:27:33.027 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.015+0000 7f1f6f558640 1 -- 172.21.15.16:0/2420676683 wait complete. 2023-05-22T15:27:33.040 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-05-22T15:27:33.165 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.155+0000 7f4bfccf1640 1 Processor -- start 2023-05-22T15:27:33.166 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.159+0000 7f4bfccf1640 1 -- start start 2023-05-22T15:27:33.167 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.159+0000 7f4bfccf1640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f4bf806db10 0x7f4bf806dee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:33.167 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.159+0000 7f4bfccf1640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4bf806a830 0x7f4bf815fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:33.167 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.159+0000 7f4bfccf1640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4bf815bfc0 con 0x7f4bf806c5f0 2023-05-22T15:27:33.167 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.159+0000 7f4bfccf1640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4bf815f790 con 0x7f4bf806db10 2023-05-22T15:27:33.167 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.159+0000 7f4bfccf1640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4bf806e420 con 0x7f4bf806a830 2023-05-22T15:27:33.168 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.159+0000 7f4bf659c640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f4bf806db10 0x7f4bf806dee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:33.168 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.159+0000 7f4bf659c640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f4bf806db10 0x7f4bf806dee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57348/0 (socket says 172.21.15.16:57348) 2023-05-22T15:27:33.168 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.159+0000 7f4bf659c640 1 -- 172.21.15.16:0/3762000100 learned_addr learned my addr 172.21.15.16:0/3762000100 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:33.168 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.159+0000 7f4bf5d9b640 1 --1- 172.21.15.16:0/3762000100 >> v1:172.21.15.16:6789/0 conn(0x7f4bf806c5f0 0x7f4bf80cc7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:36654/0 (socket says 172.21.15.16:36654) 2023-05-22T15:27:33.168 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.159+0000 7f4bf6d9d640 1 --2- 172.21.15.16:0/3762000100 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4bf806a830 0x7f4bf815fbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:33.168 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.159+0000 7f4bf659c640 1 -- 172.21.15.16:0/3762000100 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4bf806a830 msgr2=0x7f4bf815fbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:33.168 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.159+0000 7f4bf659c640 1 --2- 172.21.15.16:0/3762000100 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4bf806a830 0x7f4bf815fbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:33.169 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.159+0000 7f4bf659c640 1 -- 172.21.15.16:0/3762000100 >> v1:172.21.15.16:6789/0 conn(0x7f4bf806c5f0 legacy=0x7f4bf80cc7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:33.169 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.159+0000 7f4bf659c640 1 -- 172.21.15.16:0/3762000100 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bf80cf620 con 0x7f4bf806db10 2023-05-22T15:27:33.170 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.159+0000 7f4bf659c640 1 --2- 172.21.15.16:0/3762000100 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4bf806db10 0x7f4bf806dee0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f4be8009870 tx=0x7f4be801c510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97858862a1663af0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:33.170 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.163+0000 7f4bf559a640 1 -- 172.21.15.16:0/3762000100 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4be80ab620 con 0x7f4bf806db10 2023-05-22T15:27:33.170 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.163+0000 7f4bf559a640 1 -- 172.21.15.16:0/3762000100 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4be80ab780 con 0x7f4bf806db10 2023-05-22T15:27:33.170 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.163+0000 7f4bf559a640 1 -- 172.21.15.16:0/3762000100 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4be80abb10 con 0x7f4bf806db10 2023-05-22T15:27:33.170 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.163+0000 7f4bfccf1640 1 -- 172.21.15.16:0/3762000100 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4bf806db10 msgr2=0x7f4bf806dee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:33.170 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.163+0000 7f4bfccf1640 1 --2- 172.21.15.16:0/3762000100 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4bf806db10 0x7f4bf806dee0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f4be8009870 tx=0x7f4be801c510 comp rx=0 tx=0).stop 2023-05-22T15:27:33.171 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.163+0000 7f4bfccf1640 1 -- 172.21.15.16:0/3762000100 shutdown_connections 2023-05-22T15:27:33.171 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.163+0000 7f4bfccf1640 1 --2- 172.21.15.16:0/3762000100 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4bf806a830 0x7f4bf815fbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:33.171 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.163+0000 7f4bfccf1640 1 --2- 172.21.15.16:0/3762000100 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4bf806db10 0x7f4bf806dee0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:33.171 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.163+0000 7f4bfccf1640 1 -- 172.21.15.16:0/3762000100 >> 172.21.15.16:0/3762000100 conn(0x7f4bf806f4f0 msgr2=0x7f4bf806f8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:33.171 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.163+0000 7f4bfccf1640 1 -- 172.21.15.16:0/3762000100 shutdown_connections 2023-05-22T15:27:33.171 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.163+0000 7f4bfccf1640 1 -- 172.21.15.16:0/3762000100 wait complete. 2023-05-22T15:27:33.173 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.163+0000 7f4bfccf1640 1 Processor -- start 2023-05-22T15:27:33.173 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.163+0000 7f4bfccf1640 1 -- start start 2023-05-22T15:27:33.173 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.163+0000 7f4bfccf1640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4bf806a830 0x7f4bf8154230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:33.173 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.163+0000 7f4bfccf1640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4bf806c5f0 0x7f4bf8154770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:33.173 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.163+0000 7f4bfccf1640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4bf806db10 0x7f4bf815aad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:33.174 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.163+0000 7f4bfccf1640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bf815c160 con 0x7f4bf806c5f0 2023-05-22T15:27:33.174 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.163+0000 7f4bfccf1640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bf815c440 con 0x7f4bf806a830 2023-05-22T15:27:33.174 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.163+0000 7f4bfccf1640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4bf815c720 con 0x7f4bf806db10 2023-05-22T15:27:33.174 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.163+0000 7f4bf5d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4bf806c5f0 0x7f4bf8154770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:33.174 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.163+0000 7f4bf5d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4bf806c5f0 0x7f4bf8154770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57350/0 (socket says 172.21.15.16:57350) 2023-05-22T15:27:33.174 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.163+0000 7f4bf5d9b640 1 -- 172.21.15.16:0/2366943594 learned_addr learned my addr 172.21.15.16:0/2366943594 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:33.174 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.163+0000 7f4bf6d9d640 1 --2- 172.21.15.16:0/2366943594 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4bf806db10 0x7f4bf815aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:33.175 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.163+0000 7f4bf659c640 1 --2- 172.21.15.16:0/2366943594 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4bf806a830 0x7f4bf8154230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:33.175 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.163+0000 7f4bf5d9b640 1 -- 172.21.15.16:0/2366943594 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4bf806db10 msgr2=0x7f4bf815aad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:33.175 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.163+0000 7f4bf5d9b640 1 --2- 172.21.15.16:0/2366943594 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4bf806db10 0x7f4bf815aad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:33.175 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.163+0000 7f4bf5d9b640 1 -- 172.21.15.16:0/2366943594 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4bf806a830 msgr2=0x7f4bf8154230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:33.175 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.163+0000 7f4bf5d9b640 1 --2- 172.21.15.16:0/2366943594 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4bf806a830 0x7f4bf8154230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:33.175 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.163+0000 7f4bf5d9b640 1 -- 172.21.15.16:0/2366943594 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4be80094d0 con 0x7f4bf806c5f0 2023-05-22T15:27:33.175 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.163+0000 7f4bf5d9b640 1 --2- 172.21.15.16:0/2366943594 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4bf806c5f0 0x7f4bf8154770 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f4bec00c490 tx=0x7f4bec00c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:33.175 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.163+0000 7f4be6ffd640 1 -- 172.21.15.16:0/2366943594 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bec007480 con 0x7f4bf806c5f0 2023-05-22T15:27:33.176 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.163+0000 7f4be6ffd640 1 -- 172.21.15.16:0/2366943594 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bec0075e0 con 0x7f4bf806c5f0 2023-05-22T15:27:33.176 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.163+0000 7f4bfccf1640 1 -- 172.21.15.16:0/2366943594 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4bf80cfec0 con 0x7f4bf806c5f0 2023-05-22T15:27:33.176 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.163+0000 7f4be6ffd640 1 -- 172.21.15.16:0/2366943594 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bec004a10 con 0x7f4bf806c5f0 2023-05-22T15:27:33.176 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.163+0000 7f4bfccf1640 1 -- 172.21.15.16:0/2366943594 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4bf8160530 con 0x7f4bf806c5f0 2023-05-22T15:27:33.177 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.167+0000 7f4bfccf1640 1 -- 172.21.15.16:0/2366943594 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4bf80cf260 con 0x7f4bf806c5f0 2023-05-22T15:27:33.177 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.167+0000 7f4be6ffd640 1 -- 172.21.15.16:0/2366943594 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7f4bec004b70 con 0x7f4bf806c5f0 2023-05-22T15:27:33.177 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.167+0000 7f4be6ffd640 1 --2- 172.21.15.16:0/2366943594 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f4bd8065380 0x7f4bd8067830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:33.177 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.167+0000 7f4bf659c640 1 --2- 172.21.15.16:0/2366943594 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f4bd8065380 0x7f4bd8067830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:33.177 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.167+0000 7f4be6ffd640 1 -- 172.21.15.16:0/2366943594 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f4bec081bf0 con 0x7f4bf806c5f0 2023-05-22T15:27:33.177 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.167+0000 7f4bf659c640 1 --2- 172.21.15.16:0/2366943594 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f4bd8065380 0x7f4bd8067830 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f4be80af000 tx=0x7f4be80a7000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:33.179 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.171+0000 7f4be6ffd640 1 -- 172.21.15.16:0/2366943594 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f4bec04f540 con 0x7f4bf806c5f0 2023-05-22T15:27:33.334 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.327+0000 7f4bfccf1640 1 -- 172.21.15.16:0/2366943594 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f4bf815c890 con 0x7f4bf806c5f0 2023-05-22T15:27:33.335 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.327+0000 7f4be6ffd640 1 -- 172.21.15.16:0/2366943594 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v29) v1 ==== 92+0+10 (secure 0 0 0) 0x7f4bec018020 con 0x7f4bf806c5f0 2023-05-22T15:27:33.336 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 8 2023-05-22T15:27:33.338 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.327+0000 7f4bfccf1640 1 -- 172.21.15.16:0/2366943594 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f4bd8065380 msgr2=0x7f4bd8067830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:33.338 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.327+0000 7f4bfccf1640 1 --2- 172.21.15.16:0/2366943594 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f4bd8065380 0x7f4bd8067830 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f4be80af000 tx=0x7f4be80a7000 comp rx=0 tx=0).stop 2023-05-22T15:27:33.339 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.327+0000 7f4bfccf1640 1 -- 172.21.15.16:0/2366943594 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4bf806c5f0 msgr2=0x7f4bf8154770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:33.339 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.327+0000 7f4bfccf1640 1 --2- 172.21.15.16:0/2366943594 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4bf806c5f0 0x7f4bf8154770 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f4bec00c490 tx=0x7f4bec00c950 comp rx=0 tx=0).stop 2023-05-22T15:27:33.339 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.327+0000 7f4bfccf1640 1 -- 172.21.15.16:0/2366943594 shutdown_connections 2023-05-22T15:27:33.339 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.327+0000 7f4bfccf1640 1 --2- 172.21.15.16:0/2366943594 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f4bd8065380 0x7f4bd8067830 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:33.340 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.327+0000 7f4bfccf1640 1 --2- 172.21.15.16:0/2366943594 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4bf806db10 0x7f4bf815aad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:33.340 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.327+0000 7f4bfccf1640 1 --2- 172.21.15.16:0/2366943594 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4bf806c5f0 0x7f4bf8154770 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:33.340 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.327+0000 7f4bfccf1640 1 --2- 172.21.15.16:0/2366943594 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4bf806a830 0x7f4bf8154230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:33.340 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.327+0000 7f4bfccf1640 1 -- 172.21.15.16:0/2366943594 >> 172.21.15.16:0/2366943594 conn(0x7f4bf806f4f0 msgr2=0x7f4bf80b5130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:33.341 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.327+0000 7f4bfccf1640 1 -- 172.21.15.16:0/2366943594 shutdown_connections 2023-05-22T15:27:33.341 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.331+0000 7f4bfccf1640 1 -- 172.21.15.16:0/2366943594 wait complete. 2023-05-22T15:27:33.354 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-05-22T15:27:33.474 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.467+0000 7f0bf5a57640 1 Processor -- start 2023-05-22T15:27:33.475 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.467+0000 7f0bf5a57640 1 -- start start 2023-05-22T15:27:33.475 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.467+0000 7f0bf5a57640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0bf015bee0 0x7f0bf006b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:33.476 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.467+0000 7f0bf5a57640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f0bf00c9b00 0x7f0bf006bc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:33.476 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.467+0000 7f0bf5a57640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0bf00c8660 con 0x7f0bf00cb730 2023-05-22T15:27:33.476 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.467+0000 7f0bf5a57640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0bf00ce4a0 con 0x7f0bf00c9b00 2023-05-22T15:27:33.477 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.467+0000 7f0bf5a57640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0bf00ce780 con 0x7f0bf015bee0 2023-05-22T15:27:33.477 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.467+0000 7f0bee7fc640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f0bf00cb730 0x7f0bf015f850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:36666/0 (socket says 172.21.15.16:36666) 2023-05-22T15:27:33.477 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.467+0000 7f0bee7fc640 1 -- 172.21.15.16:0/479618718 learned_addr learned my addr 172.21.15.16:0/479618718 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:27:33.478 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.467+0000 7f0beeffd640 1 --2- 172.21.15.16:0/479618718 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0bf015bee0 0x7f0bf006b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:33.478 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.467+0000 7f0bef7fe640 1 --2- 172.21.15.16:0/479618718 >> v2:172.21.15.62:3300/0 conn(0x7f0bf00c9b00 0x7f0bf006bc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:33.478 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.467+0000 7f0bedffb640 1 -- 172.21.15.16:0/479618718 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3198922697 0 0) 0x7f0bf00c8660 con 0x7f0bf00cb730 2023-05-22T15:27:33.478 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.467+0000 7f0bedffb640 1 -- 172.21.15.16:0/479618718 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0bdc003530 con 0x7f0bf00cb730 2023-05-22T15:27:33.479 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.467+0000 7f0bedffb640 1 -- 172.21.15.16:0/479618718 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1326545988 0 0) 0x7f0bdc003530 con 0x7f0bf00cb730 2023-05-22T15:27:33.480 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.467+0000 7f0bedffb640 1 -- 172.21.15.16:0/479618718 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0bf015bee0 msgr2=0x7f0bf006b6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:33.480 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.467+0000 7f0bedffb640 1 --2- 172.21.15.16:0/479618718 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0bf015bee0 0x7f0bf006b6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:33.480 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.467+0000 7f0bedffb640 1 -- 172.21.15.16:0/479618718 >> v2:172.21.15.62:3300/0 conn(0x7f0bf00c9b00 msgr2=0x7f0bf006bc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:33.480 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.467+0000 7f0bedffb640 1 --2- 172.21.15.16:0/479618718 >> v2:172.21.15.62:3300/0 conn(0x7f0bf00c9b00 0x7f0bf006bc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:33.480 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.467+0000 7f0bedffb640 1 -- 172.21.15.16:0/479618718 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bf00cea60 con 0x7f0bf00cb730 2023-05-22T15:27:33.480 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.467+0000 7f0bedffb640 1 -- 172.21.15.16:0/479618718 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7f0bd8001680 con 0x7f0bf00cb730 2023-05-22T15:27:33.481 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.467+0000 7f0bedffb640 1 -- 172.21.15.16:0/479618718 >> v1:172.21.15.16:6789/0 conn(0x7f0bf00cb730 legacy=0x7f0bf015f850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:33.481 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.467+0000 7f0bedffb640 1 --2- 172.21.15.16:0/479618718 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0bdc003980 0x7f0bdc003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:33.481 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.467+0000 7f0bedffb640 1 --2- 172.21.15.16:0/479618718 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0bf015bee0 0x7f0bdc0047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:33.481 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.467+0000 7f0bedffb640 1 --2- 172.21.15.16:0/479618718 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0bdc004d10 0x7f0bdc005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:33.482 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.467+0000 7f0bedffb640 1 -- 172.21.15.16:0/479618718 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bf00ce4a0 con 0x7f0bdc004d10 2023-05-22T15:27:33.482 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.467+0000 7f0bedffb640 1 -- 172.21.15.16:0/479618718 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bf00ce780 con 0x7f0bdc003980 2023-05-22T15:27:33.482 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.467+0000 7f0bedffb640 1 -- 172.21.15.16:0/479618718 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0bdc00b990 con 0x7f0bf015bee0 2023-05-22T15:27:33.482 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.467+0000 7f0bef7fe640 1 --2- 172.21.15.16:0/479618718 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0bdc004d10 0x7f0bdc005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:33.482 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.467+0000 7f0beeffd640 1 --2- 172.21.15.16:0/479618718 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0bdc003980 0x7f0bdc003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:33.483 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.467+0000 7f0bee7fc640 1 --2- 172.21.15.16:0/479618718 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0bf015bee0 0x7f0bdc0047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:33.483 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.467+0000 7f0bef7fe640 1 -- 172.21.15.16:0/479618718 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0bf015bee0 msgr2=0x7f0bdc0047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:33.483 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.467+0000 7f0bef7fe640 1 --2- 172.21.15.16:0/479618718 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0bf015bee0 0x7f0bdc0047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:33.483 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.467+0000 7f0bef7fe640 1 -- 172.21.15.16:0/479618718 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0bdc003980 msgr2=0x7f0bdc003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:33.483 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.467+0000 7f0bef7fe640 1 --2- 172.21.15.16:0/479618718 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0bdc003980 0x7f0bdc003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:33.483 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.467+0000 7f0bef7fe640 1 -- 172.21.15.16:0/479618718 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bdc003530 con 0x7f0bdc004d10 2023-05-22T15:27:33.483 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.467+0000 7f0beeffd640 1 --2- 172.21.15.16:0/479618718 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0bdc003980 0x7f0bdc003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T15:27:33.484 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.471+0000 7f0bef7fe640 1 --2- 172.21.15.16:0/479618718 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0bdc004d10 0x7f0bdc005180 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f0be8009ae0 tx=0x7f0be801ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf811fcf6e9d321b server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:33.484 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.471+0000 7f0bedffb640 1 -- 172.21.15.16:0/479618718 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0be80a5680 con 0x7f0bdc004d10 2023-05-22T15:27:33.484 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.471+0000 7f0bedffb640 1 -- 172.21.15.16:0/479618718 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0be80a57e0 con 0x7f0bdc004d10 2023-05-22T15:27:33.484 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.471+0000 7f0bedffb640 1 -- 172.21.15.16:0/479618718 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0be80a5b90 con 0x7f0bdc004d10 2023-05-22T15:27:33.486 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.471+0000 7f0bf5a57640 1 -- 172.21.15.16:0/479618718 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0bdc004d10 msgr2=0x7f0bdc005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:33.486 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.471+0000 7f0bf5a57640 1 --2- 172.21.15.16:0/479618718 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0bdc004d10 0x7f0bdc005180 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f0be8009ae0 tx=0x7f0be801ca30 comp rx=0 tx=0).stop 2023-05-22T15:27:33.487 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.471+0000 7f0bef7fe640 1 -- 172.21.15.16:0/479618718 reap_dead start 2023-05-22T15:27:33.487 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.471+0000 7f0bf5a57640 1 -- 172.21.15.16:0/479618718 shutdown_connections 2023-05-22T15:27:33.487 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.471+0000 7f0bf5a57640 1 -- 172.21.15.16:0/479618718 >> 172.21.15.16:0/479618718 conn(0x7f0bf0069720 msgr2=0x7f0bf0069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:33.487 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.471+0000 7f0bf5a57640 1 -- 172.21.15.16:0/479618718 shutdown_connections 2023-05-22T15:27:33.487 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.471+0000 7f0bf5a57640 1 -- 172.21.15.16:0/479618718 wait complete. 2023-05-22T15:27:33.487 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.475+0000 7f0bf5a57640 1 Processor -- start 2023-05-22T15:27:33.487 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.475+0000 7f0bf5a57640 1 -- start start 2023-05-22T15:27:33.488 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.475+0000 7f0bf5a57640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0bf01521d0 0x7f0bf0159b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:33.488 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.475+0000 7f0bf5a57640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0bf01525a0 0x7f0bf0154a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:33.488 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.475+0000 7f0bf5a57640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0bf01f0870 0x7f0bf01f0ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:33.488 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.475+0000 7f0bf5a57640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bf01f1220 con 0x7f0bf01525a0 2023-05-22T15:27:33.488 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.475+0000 7f0bf5a57640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bf0158a80 con 0x7f0bf01521d0 2023-05-22T15:27:33.488 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.475+0000 7f0bf5a57640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0bf0158d60 con 0x7f0bf01f0870 2023-05-22T15:27:33.489 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.475+0000 7f0bee7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0bf01525a0 0x7f0bf0154a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:33.489 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.475+0000 7f0bee7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0bf01525a0 0x7f0bf0154a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57368/0 (socket says 172.21.15.16:57368) 2023-05-22T15:27:33.489 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.475+0000 7f0bee7fc640 1 -- 172.21.15.16:0/358913618 learned_addr learned my addr 172.21.15.16:0/358913618 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:33.489 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.475+0000 7f0bee7fc640 1 -- 172.21.15.16:0/358913618 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0bf01f0870 msgr2=0x7f0bf01f0ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:33.489 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.475+0000 7f0bee7fc640 1 --2- 172.21.15.16:0/358913618 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0bf01f0870 0x7f0bf01f0ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:33.490 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.475+0000 7f0bee7fc640 1 -- 172.21.15.16:0/358913618 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0bf01521d0 msgr2=0x7f0bf0159b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:33.490 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.475+0000 7f0bee7fc640 1 --2- 172.21.15.16:0/358913618 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0bf01521d0 0x7f0bf0159b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:33.490 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.475+0000 7f0bee7fc640 1 -- 172.21.15.16:0/358913618 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bf00ce0c0 con 0x7f0bf01525a0 2023-05-22T15:27:33.490 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.475+0000 7f0bee7fc640 1 --2- 172.21.15.16:0/358913618 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0bf01525a0 0x7f0bf0154a70 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f0bd800eb20 tx=0x7f0bd80016b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:33.491 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.475+0000 7f0be7fff640 1 -- 172.21.15.16:0/358913618 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bd800c850 con 0x7f0bf01525a0 2023-05-22T15:27:33.491 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.475+0000 7f0be7fff640 1 -- 172.21.15.16:0/358913618 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bd800c9b0 con 0x7f0bf01525a0 2023-05-22T15:27:33.491 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.475+0000 7f0bf5a57640 1 -- 172.21.15.16:0/358913618 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0bf00ced40 con 0x7f0bf01525a0 2023-05-22T15:27:33.491 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.475+0000 7f0be7fff640 1 -- 172.21.15.16:0/358913618 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bd800cd60 con 0x7f0bf01525a0 2023-05-22T15:27:33.491 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.475+0000 7f0bf5a57640 1 -- 172.21.15.16:0/358913618 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0bf00ce250 con 0x7f0bf01525a0 2023-05-22T15:27:33.491 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.475+0000 7f0bf5a57640 1 -- 172.21.15.16:0/358913618 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0bf004dbe0 con 0x7f0bf01525a0 2023-05-22T15:27:33.491 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.475+0000 7f0be7fff640 1 -- 172.21.15.16:0/358913618 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7f0bd801d460 con 0x7f0bf01525a0 2023-05-22T15:27:33.492 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.475+0000 7f0be7fff640 1 --2- 172.21.15.16:0/358913618 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f0bd00653b0 0x7f0bd0067860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:33.492 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.475+0000 7f0be7fff640 1 -- 172.21.15.16:0/358913618 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f0bd8082ae0 con 0x7f0bf01525a0 2023-05-22T15:27:33.492 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.475+0000 7f0beeffd640 1 --2- 172.21.15.16:0/358913618 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f0bd00653b0 0x7f0bd0067860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:33.492 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.479+0000 7f0beeffd640 1 --2- 172.21.15.16:0/358913618 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f0bd00653b0 0x7f0bd0067860 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f0be0007330 tx=0x7f0be0009750 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:33.493 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.479+0000 7f0be7fff640 1 -- 172.21.15.16:0/358913618 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f0bd8050520 con 0x7f0bf01525a0 2023-05-22T15:27:33.637 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.627+0000 7f0bf5a57640 1 -- 172.21.15.16:0/358913618 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f0bf00cb000 con 0x7f0bf01525a0 2023-05-22T15:27:33.638 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.631+0000 7f0be7fff640 1 -- 172.21.15.16:0/358913618 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v29) v1 ==== 104+0+11 (secure 0 0 0) 0x7f0bd8053b40 con 0x7f0bf01525a0 2023-05-22T15:27:33.638 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2023-05-22T15:27:33.639 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.631+0000 7f0bf5a57640 1 -- 172.21.15.16:0/358913618 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f0bd00653b0 msgr2=0x7f0bd0067860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:33.640 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.631+0000 7f0bf5a57640 1 --2- 172.21.15.16:0/358913618 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f0bd00653b0 0x7f0bd0067860 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f0be0007330 tx=0x7f0be0009750 comp rx=0 tx=0).stop 2023-05-22T15:27:33.640 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.631+0000 7f0bf5a57640 1 -- 172.21.15.16:0/358913618 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0bf01525a0 msgr2=0x7f0bf0154a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:33.640 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.631+0000 7f0bf5a57640 1 --2- 172.21.15.16:0/358913618 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0bf01525a0 0x7f0bf0154a70 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f0bd800eb20 tx=0x7f0bd80016b0 comp rx=0 tx=0).stop 2023-05-22T15:27:33.640 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.631+0000 7f0bf5a57640 1 -- 172.21.15.16:0/358913618 shutdown_connections 2023-05-22T15:27:33.640 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.631+0000 7f0bf5a57640 1 --2- 172.21.15.16:0/358913618 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f0bd00653b0 0x7f0bd0067860 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:33.640 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.631+0000 7f0bf5a57640 1 --2- 172.21.15.16:0/358913618 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0bf01f0870 0x7f0bf01f0ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:33.641 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.631+0000 7f0bf5a57640 1 --2- 172.21.15.16:0/358913618 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0bf01525a0 0x7f0bf0154a70 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:33.641 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.631+0000 7f0bf5a57640 1 --2- 172.21.15.16:0/358913618 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0bf01521d0 0x7f0bf0159b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:33.641 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.631+0000 7f0bf5a57640 1 -- 172.21.15.16:0/358913618 >> 172.21.15.16:0/358913618 conn(0x7f0bf0069720 msgr2=0x7f0bf00b16c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:33.641 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.631+0000 7f0bf5a57640 1 -- 172.21.15.16:0/358913618 shutdown_connections 2023-05-22T15:27:33.641 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.631+0000 7f0bf5a57640 1 -- 172.21.15.16:0/358913618 wait complete. 2023-05-22T15:27:33.656 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-05-22T15:27:33.770 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.763+0000 7f43e7c48640 1 Processor -- start 2023-05-22T15:27:33.771 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.763+0000 7f43e7c48640 1 -- start start 2023-05-22T15:27:33.771 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.763+0000 7f43e7c48640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f43e00d2480 0x7f43e00d2900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:33.771 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.763+0000 7f43e7c48640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f43e00b5340 0x7f43e00b5790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:33.772 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.763+0000 7f43e7c48640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43e00cccc0 con 0x7f43e015bca0 2023-05-22T15:27:33.772 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.763+0000 7f43e7c48640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43e00cd660 con 0x7f43e00d2480 2023-05-22T15:27:33.772 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.763+0000 7f43e7c48640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43e00ce190 con 0x7f43e00b5340 2023-05-22T15:27:33.772 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.763+0000 7f43e59e4640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f43e015bca0 0x7f43e00cda80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:36678/0 (socket says 172.21.15.16:36678) 2023-05-22T15:27:33.772 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.763+0000 7f43e59e4640 1 -- 172.21.15.16:0/2923857747 learned_addr learned my addr 172.21.15.16:0/2923857747 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:27:33.772 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.763+0000 7f43e51e3640 1 --2- 172.21.15.16:0/2923857747 >> v2:172.21.15.16:3300/0 conn(0x7f43e00d2480 0x7f43e00d2900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:33.773 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.763+0000 7f43e61e5640 1 --2- 172.21.15.16:0/2923857747 >> v2:172.21.15.62:3300/0 conn(0x7f43e00b5340 0x7f43e00b5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:33.773 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.763+0000 7f43e49e2640 1 -- 172.21.15.16:0/2923857747 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 987072500 0 0) 0x7f43e00cccc0 con 0x7f43e015bca0 2023-05-22T15:27:33.773 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.763+0000 7f43e49e2640 1 -- 172.21.15.16:0/2923857747 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f43cc003530 con 0x7f43e015bca0 2023-05-22T15:27:33.774 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.763+0000 7f43e51e3640 1 -- 172.21.15.16:0/2923857747 >> v2:172.21.15.62:3300/0 conn(0x7f43e00b5340 msgr2=0x7f43e00b5790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:33.775 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.763+0000 7f43e51e3640 1 --2- 172.21.15.16:0/2923857747 >> v2:172.21.15.62:3300/0 conn(0x7f43e00b5340 0x7f43e00b5790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:33.775 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.763+0000 7f43e51e3640 1 -- 172.21.15.16:0/2923857747 >> v1:172.21.15.16:6789/0 conn(0x7f43e015bca0 legacy=0x7f43e00cda80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:33.776 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.763+0000 7f43e51e3640 1 -- 172.21.15.16:0/2923857747 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43e00ccfa0 con 0x7f43e00d2480 2023-05-22T15:27:33.776 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.763+0000 7f43e61e5640 1 --2- 172.21.15.16:0/2923857747 >> v2:172.21.15.62:3300/0 conn(0x7f43e00b5340 0x7f43e00b5790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T15:27:33.778 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.767+0000 7f43e51e3640 1 --2- 172.21.15.16:0/2923857747 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f43e00d2480 0x7f43e00d2900 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f43e00c5fc0 tx=0x7f43d001c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7f8b6b533671efd server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:33.778 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.767+0000 7f43e49e2640 1 -- 172.21.15.16:0/2923857747 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43d00ab600 con 0x7f43e00d2480 2023-05-22T15:27:33.779 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.767+0000 7f43e49e2640 1 -- 172.21.15.16:0/2923857747 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43d00a7050 con 0x7f43e00d2480 2023-05-22T15:27:33.779 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.767+0000 7f43e49e2640 1 -- 172.21.15.16:0/2923857747 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43d00ab960 con 0x7f43e00d2480 2023-05-22T15:27:33.779 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.767+0000 7f43e7c48640 1 -- 172.21.15.16:0/2923857747 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f43e00d2480 msgr2=0x7f43e00d2900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:33.779 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.767+0000 7f43e7c48640 1 --2- 172.21.15.16:0/2923857747 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f43e00d2480 0x7f43e00d2900 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f43e00c5fc0 tx=0x7f43d001c460 comp rx=0 tx=0).stop 2023-05-22T15:27:33.779 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.767+0000 7f43e7c48640 1 -- 172.21.15.16:0/2923857747 shutdown_connections 2023-05-22T15:27:33.779 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.767+0000 7f43e7c48640 1 --2- 172.21.15.16:0/2923857747 >> v2:172.21.15.62:3300/0 conn(0x7f43e00b5340 0x7f43e00b5790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:33.780 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.767+0000 7f43e7c48640 1 --2- 172.21.15.16:0/2923857747 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f43e00d2480 0x7f43e00d2900 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:33.780 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.767+0000 7f43e7c48640 1 -- 172.21.15.16:0/2923857747 >> 172.21.15.16:0/2923857747 conn(0x7f43e00655c0 msgr2=0x7f43e0070e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:33.780 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.767+0000 7f43e7c48640 1 -- 172.21.15.16:0/2923857747 shutdown_connections 2023-05-22T15:27:33.780 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.767+0000 7f43e7c48640 1 -- 172.21.15.16:0/2923857747 wait complete. 2023-05-22T15:27:33.780 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.767+0000 7f43e7c48640 1 Processor -- start 2023-05-22T15:27:33.781 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.767+0000 7f43e7c48640 1 -- start start 2023-05-22T15:27:33.781 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.771+0000 7f43e7c48640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f43e00b5340 0x7f43e0156470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:33.781 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.771+0000 7f43e7c48640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f43e00d2480 0x7f43e015df10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:33.781 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.771+0000 7f43e7c48640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f43e015bca0 0x7f43e00c7f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:33.782 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.771+0000 7f43e7c48640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43e00c95c0 con 0x7f43e00d2480 2023-05-22T15:27:33.782 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.771+0000 7f43e7c48640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43e00c9e60 con 0x7f43e00b5340 2023-05-22T15:27:33.782 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.771+0000 7f43e7c48640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43e0156b00 con 0x7f43e015bca0 2023-05-22T15:27:33.784 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.771+0000 7f43e51e3640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f43e00d2480 0x7f43e015df10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:33.784 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.771+0000 7f43e51e3640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f43e00d2480 0x7f43e015df10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57382/0 (socket says 172.21.15.16:57382) 2023-05-22T15:27:33.784 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.771+0000 7f43e51e3640 1 -- 172.21.15.16:0/18965887 learned_addr learned my addr 172.21.15.16:0/18965887 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:33.784 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.771+0000 7f43e59e4640 1 --2- 172.21.15.16:0/18965887 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f43e00b5340 0x7f43e0156470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:33.784 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.771+0000 7f43e61e5640 1 --2- 172.21.15.16:0/18965887 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f43e015bca0 0x7f43e00c7f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:33.784 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.771+0000 7f43e51e3640 1 -- 172.21.15.16:0/18965887 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f43e015bca0 msgr2=0x7f43e00c7f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:33.785 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.771+0000 7f43e51e3640 1 --2- 172.21.15.16:0/18965887 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f43e015bca0 0x7f43e00c7f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:33.785 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.771+0000 7f43e51e3640 1 -- 172.21.15.16:0/18965887 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f43e00b5340 msgr2=0x7f43e0156470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:33.785 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.771+0000 7f43e51e3640 1 --2- 172.21.15.16:0/18965887 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f43e00b5340 0x7f43e0156470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:33.785 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.771+0000 7f43e51e3640 1 -- 172.21.15.16:0/18965887 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43d0009420 con 0x7f43e00d2480 2023-05-22T15:27:33.786 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.771+0000 7f43e51e3640 1 --2- 172.21.15.16:0/18965887 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f43e00d2480 0x7f43e015df10 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f43d001a090 tx=0x7f43d0019200 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:33.787 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.771+0000 7f43d67fc640 1 -- 172.21.15.16:0/18965887 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43d00b4400 con 0x7f43e00d2480 2023-05-22T15:27:33.787 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.771+0000 7f43d67fc640 1 -- 172.21.15.16:0/18965887 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43d00b4560 con 0x7f43e00d2480 2023-05-22T15:27:33.787 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.771+0000 7f43d67fc640 1 -- 172.21.15.16:0/18965887 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43d00b4910 con 0x7f43e00d2480 2023-05-22T15:27:33.787 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.771+0000 7f43e7c48640 1 -- 172.21.15.16:0/18965887 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43e0160360 con 0x7f43e00d2480 2023-05-22T15:27:33.787 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.771+0000 7f43e7c48640 1 -- 172.21.15.16:0/18965887 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43e00b5cd0 con 0x7f43e00d2480 2023-05-22T15:27:33.788 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.771+0000 7f43e7c48640 1 -- 172.21.15.16:0/18965887 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43e01604f0 con 0x7f43e00d2480 2023-05-22T15:27:33.788 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.771+0000 7f43d67fc640 1 -- 172.21.15.16:0/18965887 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7f43d00b2040 con 0x7f43e00d2480 2023-05-22T15:27:33.788 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.771+0000 7f43d67fc640 1 --2- 172.21.15.16:0/18965887 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f43c4065380 0x7f43c4067830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:33.789 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.771+0000 7f43d67fc640 1 -- 172.21.15.16:0/18965887 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f43d011c9c0 con 0x7f43e00d2480 2023-05-22T15:27:33.789 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.771+0000 7f43e59e4640 1 --2- 172.21.15.16:0/18965887 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f43c4065380 0x7f43c4067830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:33.789 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.775+0000 7f43e59e4640 1 --2- 172.21.15.16:0/18965887 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f43c4065380 0x7f43c4067830 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f43dc005890 tx=0x7f43dc009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:33.789 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.775+0000 7f43d67fc640 1 -- 172.21.15.16:0/18965887 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f43d00ea2f0 con 0x7f43e00d2480 2023-05-22T15:27:33.933 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.923+0000 7f43e7c48640 1 -- 172.21.15.16:0/18965887 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f43e00c9ff0 con 0x7f43e00d2480 2023-05-22T15:27:33.933 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.923+0000 7f43d67fc640 1 -- 172.21.15.16:0/18965887 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v29) v1 ==== 100+0+11 (secure 0 0 0) 0x7f43d00b00a0 con 0x7f43e00d2480 2023-05-22T15:27:33.934 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 64 2023-05-22T15:27:33.936 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.927+0000 7f43e7c48640 1 -- 172.21.15.16:0/18965887 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f43c4065380 msgr2=0x7f43c4067830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:33.937 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.927+0000 7f43e7c48640 1 --2- 172.21.15.16:0/18965887 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f43c4065380 0x7f43c4067830 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f43dc005890 tx=0x7f43dc009040 comp rx=0 tx=0).stop 2023-05-22T15:27:33.937 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.927+0000 7f43e7c48640 1 -- 172.21.15.16:0/18965887 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f43e00d2480 msgr2=0x7f43e015df10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:33.937 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.927+0000 7f43e7c48640 1 --2- 172.21.15.16:0/18965887 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f43e00d2480 0x7f43e015df10 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f43d001a090 tx=0x7f43d0019200 comp rx=0 tx=0).stop 2023-05-22T15:27:33.938 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.927+0000 7f43e7c48640 1 -- 172.21.15.16:0/18965887 shutdown_connections 2023-05-22T15:27:33.938 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.927+0000 7f43e7c48640 1 --2- 172.21.15.16:0/18965887 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f43c4065380 0x7f43c4067830 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:33.938 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.927+0000 7f43e7c48640 1 --2- 172.21.15.16:0/18965887 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f43e015bca0 0x7f43e00c7f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:33.938 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.927+0000 7f43e7c48640 1 --2- 172.21.15.16:0/18965887 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f43e00d2480 0x7f43e015df10 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:33.939 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.927+0000 7f43e7c48640 1 --2- 172.21.15.16:0/18965887 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f43e00b5340 0x7f43e0156470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:33.939 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.927+0000 7f43e7c48640 1 -- 172.21.15.16:0/18965887 >> 172.21.15.16:0/18965887 conn(0x7f43e00655c0 msgr2=0x7f43e00b73f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:33.939 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.927+0000 7f43e7c48640 1 -- 172.21.15.16:0/18965887 shutdown_connections 2023-05-22T15:27:33.939 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:33.927+0000 7f43e7c48640 1 -- 172.21.15.16:0/18965887 wait complete. 2023-05-22T15:27:33.950 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-05-22T15:27:34.075 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.067+0000 7f6748087640 1 Processor -- start 2023-05-22T15:27:34.076 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.067+0000 7f6748087640 1 -- start start 2023-05-22T15:27:34.078 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.067+0000 7f6748087640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f6740159970 0x7f6740159d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:34.078 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.067+0000 7f6748087640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6740070c30 con 0x7f674015d680 2023-05-22T15:27:34.078 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.067+0000 7f6748087640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6740070e60 con 0x7f674015db00 2023-05-22T15:27:34.078 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.067+0000 7f6748087640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67400cb970 con 0x7f6740159970 2023-05-22T15:27:34.079 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.067+0000 7f6745622640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f674015d680 0x7f67400cb260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:36684/0 (socket says 172.21.15.16:36684) 2023-05-22T15:27:34.079 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.067+0000 7f6745622640 1 -- 172.21.15.16:0/2626015764 learned_addr learned my addr 172.21.15.16:0/2626015764 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:27:34.079 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.067+0000 7f6745e23640 1 --2- 172.21.15.16:0/2626015764 >> v2:172.21.15.62:3300/0 conn(0x7f6740159970 0x7f6740159d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:34.079 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.067+0000 7f6744e21640 1 -- 172.21.15.16:0/2626015764 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2327869183 0 0) 0x7f6740070c30 con 0x7f674015d680 2023-05-22T15:27:34.079 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.071+0000 7f6744e21640 1 -- 172.21.15.16:0/2626015764 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f672c003530 con 0x7f674015d680 2023-05-22T15:27:34.080 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.071+0000 7f6744e21640 1 -- 172.21.15.16:0/2626015764 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2528841747 0 0) 0x7f672c003530 con 0x7f674015d680 2023-05-22T15:27:34.080 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.071+0000 7f6744e21640 1 -- 172.21.15.16:0/2626015764 >> v2:172.21.15.62:3300/0 conn(0x7f6740159970 msgr2=0x7f6740159d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:34.080 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.071+0000 7f6744e21640 1 --2- 172.21.15.16:0/2626015764 >> v2:172.21.15.62:3300/0 conn(0x7f6740159970 0x7f6740159d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:34.080 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.071+0000 7f6744e21640 1 -- 172.21.15.16:0/2626015764 >> v1:172.21.15.62:6789/0 conn(0x7f674015db00 legacy=0x7f674006b6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:34.080 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.071+0000 7f6744e21640 1 -- 172.21.15.16:0/2626015764 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6740071140 con 0x7f674015d680 2023-05-22T15:27:34.081 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.071+0000 7f6744e21640 1 -- 172.21.15.16:0/2626015764 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7f6730001680 con 0x7f674015d680 2023-05-22T15:27:34.081 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.071+0000 7f6744e21640 1 -- 172.21.15.16:0/2626015764 >> v1:172.21.15.16:6789/0 conn(0x7f674015d680 legacy=0x7f67400cb260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:34.081 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.071+0000 7f6744e21640 1 --2- 172.21.15.16:0/2626015764 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f672c0039d0 0x7f672c003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:34.082 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.071+0000 7f6744e21640 1 --2- 172.21.15.16:0/2626015764 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f672c0047d0 0x7f672c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:34.082 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.071+0000 7f6744e21640 1 --2- 172.21.15.16:0/2626015764 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f672c0051d0 0x7f672c00b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:34.082 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.071+0000 7f6744e21640 1 -- 172.21.15.16:0/2626015764 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67400cb970 con 0x7f672c0051d0 2023-05-22T15:27:34.082 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.071+0000 7f6744e21640 1 -- 172.21.15.16:0/2626015764 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f672c00c0e0 con 0x7f672c0047d0 2023-05-22T15:27:34.082 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.071+0000 7f6744e21640 1 -- 172.21.15.16:0/2626015764 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f672c00c2a0 con 0x7f672c0039d0 2023-05-22T15:27:34.082 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.071+0000 7f6746624640 1 --2- 172.21.15.16:0/2626015764 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f672c0051d0 0x7f672c00b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:34.083 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.071+0000 7f6745e23640 1 --2- 172.21.15.16:0/2626015764 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f672c0039d0 0x7f672c003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:34.083 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.071+0000 7f6745622640 1 --2- 172.21.15.16:0/2626015764 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f672c0047d0 0x7f672c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:34.083 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.071+0000 7f6746624640 1 -- 172.21.15.16:0/2626015764 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f672c0039d0 msgr2=0x7f672c003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:34.083 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.071+0000 7f6746624640 1 --2- 172.21.15.16:0/2626015764 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f672c0039d0 0x7f672c003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:34.083 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.071+0000 7f6746624640 1 -- 172.21.15.16:0/2626015764 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f672c0047d0 msgr2=0x7f672c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:34.084 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.071+0000 7f6746624640 1 --2- 172.21.15.16:0/2626015764 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f672c0047d0 0x7f672c004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:34.084 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.071+0000 7f6746624640 1 -- 172.21.15.16:0/2626015764 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f672c003530 con 0x7f672c0051d0 2023-05-22T15:27:34.088 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.071+0000 7f6746624640 1 -- 172.21.15.16:0/2626015764 reap_dead start 2023-05-22T15:27:34.088 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.071+0000 7f6746624640 1 --2- 172.21.15.16:0/2626015764 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f672c0051d0 0x7f672c00b990 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f6738001e70 tx=0x7f673801c760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68a116c2ad37cf4a server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:34.089 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.071+0000 7f6744e21640 1 -- 172.21.15.16:0/2626015764 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6738003b80 con 0x7f672c0051d0 2023-05-22T15:27:34.089 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.071+0000 7f6744e21640 1 -- 172.21.15.16:0/2626015764 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6738003d00 con 0x7f672c0051d0 2023-05-22T15:27:34.089 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.071+0000 7f6744e21640 1 -- 172.21.15.16:0/2626015764 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6738005260 con 0x7f672c0051d0 2023-05-22T15:27:34.089 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.075+0000 7f6748087640 1 -- 172.21.15.16:0/2626015764 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f672c0051d0 msgr2=0x7f672c00b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:34.089 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.075+0000 7f6748087640 1 --2- 172.21.15.16:0/2626015764 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f672c0051d0 0x7f672c00b990 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f6738001e70 tx=0x7f673801c760 comp rx=0 tx=0).stop 2023-05-22T15:27:34.090 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.075+0000 7f6748087640 1 -- 172.21.15.16:0/2626015764 shutdown_connections 2023-05-22T15:27:34.090 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.075+0000 7f6748087640 1 --2- 172.21.15.16:0/2626015764 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f672c0051d0 0x7f672c00b990 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:34.090 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.075+0000 7f6748087640 1 -- 172.21.15.16:0/2626015764 >> 172.21.15.16:0/2626015764 conn(0x7f6740069720 msgr2=0x7f6740069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:34.090 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.075+0000 7f6748087640 1 -- 172.21.15.16:0/2626015764 shutdown_connections 2023-05-22T15:27:34.091 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.075+0000 7f6748087640 1 -- 172.21.15.16:0/2626015764 wait complete. 2023-05-22T15:27:34.091 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.075+0000 7f6748087640 1 Processor -- start 2023-05-22T15:27:34.091 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.075+0000 7f6748087640 1 -- start start 2023-05-22T15:27:34.092 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.075+0000 7f6748087640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f672c0051d0 0x7f6740151080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:34.092 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.075+0000 7f6748087640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f67401e2be0 0x7f67401e3050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:34.092 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.075+0000 7f6748087640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f67401e3590 0x7f67400b2080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:34.092 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.075+0000 7f6748087640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67400b1c60 con 0x7f672c0051d0 2023-05-22T15:27:34.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.075+0000 7f6748087640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67401e3a30 con 0x7f67401e2be0 2023-05-22T15:27:34.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.075+0000 7f6748087640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67400b0fe0 con 0x7f67401e3590 2023-05-22T15:27:34.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.075+0000 7f6745e23640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f672c0051d0 0x7f6740151080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:34.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.075+0000 7f6745e23640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f672c0051d0 0x7f6740151080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57402/0 (socket says 172.21.15.16:57402) 2023-05-22T15:27:34.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.075+0000 7f6745e23640 1 -- 172.21.15.16:0/766160286 learned_addr learned my addr 172.21.15.16:0/766160286 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:34.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.075+0000 7f6745622640 1 --2- 172.21.15.16:0/766160286 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f67401e2be0 0x7f67401e3050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:34.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.075+0000 7f6746624640 1 --2- 172.21.15.16:0/766160286 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f67401e3590 0x7f67400b2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:34.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.075+0000 7f6745e23640 1 -- 172.21.15.16:0/766160286 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f67401e3590 msgr2=0x7f67400b2080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:34.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.075+0000 7f6745e23640 1 --2- 172.21.15.16:0/766160286 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f67401e3590 0x7f67400b2080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:34.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.075+0000 7f6745e23640 1 -- 172.21.15.16:0/766160286 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f67401e2be0 msgr2=0x7f67401e3050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:34.096 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.075+0000 7f6745e23640 1 --2- 172.21.15.16:0/766160286 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f67401e2be0 0x7f67401e3050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:34.096 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.075+0000 7f6745e23640 1 -- 172.21.15.16:0/766160286 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6740071700 con 0x7f672c0051d0 2023-05-22T15:27:34.097 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.075+0000 7f6746624640 1 --2- 172.21.15.16:0/766160286 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f67401e3590 0x7f67400b2080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T15:27:34.097 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.075+0000 7f6745e23640 1 --2- 172.21.15.16:0/766160286 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f672c0051d0 0x7f6740151080 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f673c011960 tx=0x7f673c011d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:34.097 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.075+0000 7f67367fc640 1 -- 172.21.15.16:0/766160286 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f673c00fa30 con 0x7f672c0051d0 2023-05-22T15:27:34.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.075+0000 7f67367fc640 1 -- 172.21.15.16:0/766160286 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f673c00fbb0 con 0x7f672c0051d0 2023-05-22T15:27:34.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.075+0000 7f6748087640 1 -- 172.21.15.16:0/766160286 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6740071420 con 0x7f672c0051d0 2023-05-22T15:27:34.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.075+0000 7f67367fc640 1 -- 172.21.15.16:0/766160286 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f673c014440 con 0x7f672c0051d0 2023-05-22T15:27:34.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.075+0000 7f6748087640 1 -- 172.21.15.16:0/766160286 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67400cac90 con 0x7f672c0051d0 2023-05-22T15:27:34.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.079+0000 7f6748087640 1 -- 172.21.15.16:0/766160286 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f674004dbe0 con 0x7f672c0051d0 2023-05-22T15:27:34.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.079+0000 7f67367fc640 1 -- 172.21.15.16:0/766160286 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7f673c0145a0 con 0x7f672c0051d0 2023-05-22T15:27:34.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.079+0000 7f67367fc640 1 --2- 172.21.15.16:0/766160286 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f67240653b0 0x7f6724067860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:34.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.079+0000 7f67367fc640 1 -- 172.21.15.16:0/766160286 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f673c0831f0 con 0x7f672c0051d0 2023-05-22T15:27:34.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.079+0000 7f6745622640 1 --2- 172.21.15.16:0/766160286 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f67240653b0 0x7f6724067860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:34.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.079+0000 7f6745622640 1 --2- 172.21.15.16:0/766160286 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f67240653b0 0x7f6724067860 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f6730001060 tx=0x7f6730004810 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:34.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.083+0000 7f67367fc640 1 -- 172.21.15.16:0/766160286 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f673c050b70 con 0x7f672c0051d0 2023-05-22T15:27:34.239 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.231+0000 7f6748087640 1 -- 172.21.15.16:0/766160286 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f67400c3b30 con 0x7f672c0051d0 2023-05-22T15:27:34.239 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.231+0000 7f67367fc640 1 -- 172.21.15.16:0/766160286 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v29) v1 ==== 103+0+11 (secure 0 0 0) 0x7f673c054190 con 0x7f672c0051d0 2023-05-22T15:27:34.240 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2023-05-22T15:27:34.241 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.231+0000 7f6748087640 1 -- 172.21.15.16:0/766160286 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f67240653b0 msgr2=0x7f6724067860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:34.241 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.231+0000 7f6748087640 1 --2- 172.21.15.16:0/766160286 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f67240653b0 0x7f6724067860 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f6730001060 tx=0x7f6730004810 comp rx=0 tx=0).stop 2023-05-22T15:27:34.242 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.235+0000 7f6748087640 1 -- 172.21.15.16:0/766160286 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f672c0051d0 msgr2=0x7f6740151080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:34.242 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.235+0000 7f6748087640 1 --2- 172.21.15.16:0/766160286 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f672c0051d0 0x7f6740151080 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f673c011960 tx=0x7f673c011d60 comp rx=0 tx=0).stop 2023-05-22T15:27:34.242 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.235+0000 7f6748087640 1 -- 172.21.15.16:0/766160286 shutdown_connections 2023-05-22T15:27:34.242 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.235+0000 7f6748087640 1 --2- 172.21.15.16:0/766160286 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f67240653b0 0x7f6724067860 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:34.242 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.235+0000 7f6748087640 1 --2- 172.21.15.16:0/766160286 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f67401e3590 0x7f67400b2080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:34.243 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.235+0000 7f6748087640 1 --2- 172.21.15.16:0/766160286 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f67401e2be0 0x7f67401e3050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:34.243 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.235+0000 7f6748087640 1 --2- 172.21.15.16:0/766160286 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f672c0051d0 0x7f6740151080 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:34.243 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.235+0000 7f6748087640 1 -- 172.21.15.16:0/766160286 >> 172.21.15.16:0/766160286 conn(0x7f6740069720 msgr2=0x7f67400c4330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:34.243 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.235+0000 7f6748087640 1 -- 172.21.15.16:0/766160286 shutdown_connections 2023-05-22T15:27:34.243 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.235+0000 7f6748087640 1 -- 172.21.15.16:0/766160286 wait complete. 2023-05-22T15:27:34.257 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-05-22T15:27:34.369 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.363+0000 7ff344bbd640 1 Processor -- start 2023-05-22T15:27:34.370 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.363+0000 7ff344bbd640 1 -- start start 2023-05-22T15:27:34.373 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.363+0000 7ff344bbd640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff3400ca4e0 0x7ff34015e460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:34.373 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.363+0000 7ff344bbd640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3400b0650 con 0x7ff3400c9b40 2023-05-22T15:27:34.373 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.363+0000 7ff344bbd640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3400b0370 con 0x7ff340070df0 2023-05-22T15:27:34.373 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.363+0000 7ff344bbd640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff34015d980 con 0x7ff3400ca4e0 2023-05-22T15:27:34.373 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.363+0000 7ff337fff640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7ff3400c9b40 0x7ff3400afb40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:36698/0 (socket says 172.21.15.16:36698) 2023-05-22T15:27:34.373 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.363+0000 7ff337fff640 1 -- 172.21.15.16:0/3362719019 learned_addr learned my addr 172.21.15.16:0/3362719019 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:27:34.374 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.363+0000 7ff33e59c640 1 --2- 172.21.15.16:0/3362719019 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff3400ca4e0 0x7ff34015e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:34.374 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.363+0000 7ff33dd9b640 1 -- 172.21.15.16:0/3362719019 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4029902827 0 0) 0x7ff3400b0650 con 0x7ff3400c9b40 2023-05-22T15:27:34.374 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.363+0000 7ff33dd9b640 1 -- 172.21.15.16:0/3362719019 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff32c003530 con 0x7ff3400c9b40 2023-05-22T15:27:34.374 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.363+0000 7ff33dd9b640 1 -- 172.21.15.16:0/3362719019 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1670792397 0 0) 0x7ff32c003530 con 0x7ff3400c9b40 2023-05-22T15:27:34.374 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.363+0000 7ff33dd9b640 1 -- 172.21.15.16:0/3362719019 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff3400ca4e0 msgr2=0x7ff34015e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:34.374 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.363+0000 7ff33dd9b640 1 --2- 172.21.15.16:0/3362719019 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff3400ca4e0 0x7ff34015e460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:34.374 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.363+0000 7ff33dd9b640 1 -- 172.21.15.16:0/3362719019 >> v1:172.21.15.62:6789/0 conn(0x7ff340070df0 legacy=0x7ff340071210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:34.375 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.363+0000 7ff33dd9b640 1 -- 172.21.15.16:0/3362719019 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff34015dc60 con 0x7ff3400c9b40 2023-05-22T15:27:34.375 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.363+0000 7ff33dd9b640 1 -- 172.21.15.16:0/3362719019 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7ff330001680 con 0x7ff3400c9b40 2023-05-22T15:27:34.375 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.363+0000 7ff33dd9b640 1 -- 172.21.15.16:0/3362719019 >> v1:172.21.15.16:6789/0 conn(0x7ff3400c9b40 legacy=0x7ff3400afb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:34.375 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.363+0000 7ff33dd9b640 1 --2- 172.21.15.16:0/3362719019 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff32c0039b0 0x7ff32c003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:34.375 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.363+0000 7ff33dd9b640 1 --2- 172.21.15.16:0/3362719019 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff32c0047d0 0x7ff32c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:34.375 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.363+0000 7ff33dd9b640 1 --2- 172.21.15.16:0/3362719019 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff3400ca4e0 0x7ff32c0051a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:34.375 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.363+0000 7ff33dd9b640 1 -- 172.21.15.16:0/3362719019 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff34015d980 con 0x7ff32c0039b0 2023-05-22T15:27:34.375 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.363+0000 7ff33dd9b640 1 -- 172.21.15.16:0/3362719019 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff32c00ba90 con 0x7ff32c0047d0 2023-05-22T15:27:34.376 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.363+0000 7ff33dd9b640 1 -- 172.21.15.16:0/3362719019 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff32c00bc50 con 0x7ff3400ca4e0 2023-05-22T15:27:34.376 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.363+0000 7ff33e59c640 1 --2- 172.21.15.16:0/3362719019 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff32c0039b0 0x7ff32c003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:34.376 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.363+0000 7ff337fff640 1 --2- 172.21.15.16:0/3362719019 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff32c0047d0 0x7ff32c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:34.376 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.363+0000 7ff33ed9d640 1 --2- 172.21.15.16:0/3362719019 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff3400ca4e0 0x7ff32c0051a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:34.376 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.363+0000 7ff33e59c640 1 -- 172.21.15.16:0/3362719019 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff3400ca4e0 msgr2=0x7ff32c0051a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:34.376 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.363+0000 7ff33e59c640 1 --2- 172.21.15.16:0/3362719019 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff3400ca4e0 0x7ff32c0051a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:34.376 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.363+0000 7ff33e59c640 1 -- 172.21.15.16:0/3362719019 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff32c0047d0 msgr2=0x7ff32c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:34.377 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.363+0000 7ff33e59c640 1 --2- 172.21.15.16:0/3362719019 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff32c0047d0 0x7ff32c004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:34.377 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.363+0000 7ff33e59c640 1 -- 172.21.15.16:0/3362719019 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff32c003530 con 0x7ff32c0039b0 2023-05-22T15:27:34.377 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.363+0000 7ff337fff640 1 --2- 172.21.15.16:0/3362719019 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff32c0047d0 0x7ff32c004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T15:27:34.380 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.367+0000 7ff33e59c640 1 --2- 172.21.15.16:0/3362719019 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff32c0039b0 0x7ff32c003dc0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7ff328009ae0 tx=0x7ff32801ca20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc5cc6028c27e656 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:34.380 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.367+0000 7ff33dd9b640 1 -- 172.21.15.16:0/3362719019 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3280a6510 con 0x7ff32c0039b0 2023-05-22T15:27:34.380 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.367+0000 7ff33dd9b640 1 -- 172.21.15.16:0/3362719019 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3280a6670 con 0x7ff32c0039b0 2023-05-22T15:27:34.380 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.367+0000 7ff33dd9b640 1 -- 172.21.15.16:0/3362719019 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3280a6a20 con 0x7ff32c0039b0 2023-05-22T15:27:34.380 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.367+0000 7ff344bbd640 1 -- 172.21.15.16:0/3362719019 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff32c0039b0 msgr2=0x7ff32c003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:34.380 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.367+0000 7ff344bbd640 1 --2- 172.21.15.16:0/3362719019 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff32c0039b0 0x7ff32c003dc0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7ff328009ae0 tx=0x7ff32801ca20 comp rx=0 tx=0).stop 2023-05-22T15:27:34.381 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.367+0000 7ff33ed9d640 1 -- 172.21.15.16:0/3362719019 reap_dead start 2023-05-22T15:27:34.381 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.367+0000 7ff344bbd640 1 -- 172.21.15.16:0/3362719019 shutdown_connections 2023-05-22T15:27:34.381 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.367+0000 7ff344bbd640 1 -- 172.21.15.16:0/3362719019 >> 172.21.15.16:0/3362719019 conn(0x7ff340069720 msgr2=0x7ff340069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:34.381 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.367+0000 7ff344bbd640 1 -- 172.21.15.16:0/3362719019 shutdown_connections 2023-05-22T15:27:34.381 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.367+0000 7ff344bbd640 1 -- 172.21.15.16:0/3362719019 wait complete. 2023-05-22T15:27:34.381 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.367+0000 7ff344bbd640 1 Processor -- start 2023-05-22T15:27:34.381 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.367+0000 7ff344bbd640 1 -- start start 2023-05-22T15:27:34.382 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.367+0000 7ff344bbd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff32c0039b0 0x7ff3401f6b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:34.382 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.367+0000 7ff344bbd640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff3400c0d70 0x7ff340153290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:34.382 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.367+0000 7ff344bbd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff340069b20 0x7ff340069f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:34.382 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.367+0000 7ff344bbd640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff340153ae0 con 0x7ff32c0039b0 2023-05-22T15:27:34.382 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.367+0000 7ff344bbd640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff340142f30 con 0x7ff340069b20 2023-05-22T15:27:34.382 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.367+0000 7ff344bbd640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff34006a4d0 con 0x7ff3400c0d70 2023-05-22T15:27:34.382 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.367+0000 7ff33e59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff32c0039b0 0x7ff3401f6b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:34.382 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.367+0000 7ff33e59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff32c0039b0 0x7ff3401f6b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57416/0 (socket says 172.21.15.16:57416) 2023-05-22T15:27:34.383 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.367+0000 7ff33e59c640 1 -- 172.21.15.16:0/2677499460 learned_addr learned my addr 172.21.15.16:0/2677499460 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:34.383 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.371+0000 7ff337fff640 1 --2- 172.21.15.16:0/2677499460 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff3400c0d70 0x7ff340153290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:34.383 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.371+0000 7ff33e59c640 1 -- 172.21.15.16:0/2677499460 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff3400c0d70 msgr2=0x7ff340153290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:34.383 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.371+0000 7ff33e59c640 1 --2- 172.21.15.16:0/2677499460 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff3400c0d70 0x7ff340153290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:34.383 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.371+0000 7ff33e59c640 1 -- 172.21.15.16:0/2677499460 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff340069b20 msgr2=0x7ff340069f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:34.383 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.371+0000 7ff33e59c640 1 --2- 172.21.15.16:0/2677499460 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff340069b20 0x7ff340069f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:34.383 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.371+0000 7ff33e59c640 1 -- 172.21.15.16:0/2677499460 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3400c9010 con 0x7ff32c0039b0 2023-05-22T15:27:34.384 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.371+0000 7ff33e59c640 1 --2- 172.21.15.16:0/2677499460 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff32c0039b0 0x7ff3401f6b80 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7ff3280b0040 tx=0x7ff32c005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:34.384 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.371+0000 7ff336ffd640 1 -- 172.21.15.16:0/2677499460 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff32800f0c0 con 0x7ff32c0039b0 2023-05-22T15:27:34.384 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.371+0000 7ff336ffd640 1 -- 172.21.15.16:0/2677499460 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3280a67e0 con 0x7ff32c0039b0 2023-05-22T15:27:34.384 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.371+0000 7ff344bbd640 1 -- 172.21.15.16:0/2677499460 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3400c8e80 con 0x7ff32c0039b0 2023-05-22T15:27:34.384 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.371+0000 7ff336ffd640 1 -- 172.21.15.16:0/2677499460 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3280a6b90 con 0x7ff32c0039b0 2023-05-22T15:27:34.384 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.371+0000 7ff344bbd640 1 -- 172.21.15.16:0/2677499460 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3400c91a0 con 0x7ff32c0039b0 2023-05-22T15:27:34.384 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.371+0000 7ff344bbd640 1 -- 172.21.15.16:0/2677499460 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff34004dbe0 con 0x7ff32c0039b0 2023-05-22T15:27:34.384 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.371+0000 7ff336ffd640 1 -- 172.21.15.16:0/2677499460 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7ff3280a6cf0 con 0x7ff32c0039b0 2023-05-22T15:27:34.385 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.371+0000 7ff336ffd640 1 --2- 172.21.15.16:0/2677499460 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ff3200653b0 0x7ff320067860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:34.385 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.371+0000 7ff336ffd640 1 -- 172.21.15.16:0/2677499460 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6871+0+0 (secure 0 0 0) 0x7ff3280aa070 con 0x7ff32c0039b0 2023-05-22T15:27:34.385 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.371+0000 7ff337fff640 1 --2- 172.21.15.16:0/2677499460 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ff3200653b0 0x7ff320067860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:34.385 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.371+0000 7ff337fff640 1 --2- 172.21.15.16:0/2677499460 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ff3200653b0 0x7ff320067860 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7ff330000fc0 tx=0x7ff3300011a0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:34.385 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.375+0000 7ff336ffd640 1 -- 172.21.15.16:0/2677499460 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7ff3280eb130 con 0x7ff32c0039b0 2023-05-22T15:27:34.535 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.527+0000 7ff344bbd640 1 -- 172.21.15.16:0/2677499460 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff3400cb010 con 0x7ff32c0039b0 2023-05-22T15:27:34.538 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.531+0000 7ff336ffd640 1 -- 172.21.15.16:0/2677499460 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v29) v1 ==== 74+0+19700 (secure 0 0 0) 0x7ff3280ee750 con 0x7ff32c0039b0 2023-05-22T15:27:34.539 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:27:34.539 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":29,"fsid":"b2719834-7847-4d98-90a6-f65d57f85c49","created":"2023-05-22T15:26:25.259314+0000","modified":"2023-05-22T15:27:31.182317+0000","last_up_change":"2023-05-22T15:26:36.448524+0000","last_in_change":"2023-05-22T15:26:27.675668+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-05-22T15:26:38.019781+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-05-22T15:26:43.570514+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-05-22T15:26:47.879159+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-05-22T15:26:48.522130+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data_ec","create_time":"2023-05-22T15:26:51.616142+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"9e7ff854-b25f-4cbb-a5c2-1799d9c53d2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":37021},{"type":"v1","addr":"172.21.15.16:6825","nonce":37021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":37021},{"type":"v1","addr":"172.21.15.16:6827","nonce":37021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":37021},{"type":"v1","addr":"172.21.15.16:6831","nonce":37021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":37021},{"type":"v1","addr":"172.21.15.16:6829","nonce":37021}]},"public_addr":"172.21.15.16:6825/37021","cluster_addr":"172.21.15.16:6827/37021","heartbeat_back_addr":"172.21.15.16:6831/37021","heartbeat_front_addr":"172.21.15.16:6829/37021","state":["exists","up"]},{"osd":1,"uuid":"4702fbc9-a927-4565-aca5-41dc69cc8264","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":37008},{"type":"v1","addr":"172.21.15.16:6801","nonce":37008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":37008},{"type":"v1","addr":"172.21.15.16:6803","nonce":37008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":37008},{"type":"v1","addr":"172.21.15.16:6807","nonce":37008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":37008},{"type":"v1","addr":"172.21.15.16:6805","nonce":37008}]},"public_addr":"172.21.15.16:6801/37008","cluster_addr":"172.21.15.16:6803/37008","heartbeat_back_addr":"172.21.15.16:6807/37008","heartbeat_front_addr":"172.21.15.16:6805/37008","state":["exists","up"]},{"osd":2,"uuid":"1e7bba46-4eba-47ef-bb81-5e2d371638ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":37009},{"type":"v1","addr":"172.21.15.16:6809","nonce":37009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":37009},{"type":"v1","addr":"172.21.15.16:6811","nonce":37009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":37009},{"type":"v1","addr":"172.21.15.16:6815","nonce":37009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":37009},{"type":"v1","addr":"172.21.15.16:6813","nonce":37009}]},"public_addr":"172.21.15.16:6809/37009","cluster_addr":"172.21.15.16:6811/37009","heartbeat_back_addr":"172.21.15.16:6815/37009","heartbeat_front_addr":"172.21.15.16:6813/37009","state":["exists","up"]},{"osd":3,"uuid":"383cacc0-5a88-4904-990e-498ca59ef48a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":37020},{"type":"v1","addr":"172.21.15.16:6817","nonce":37020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":37020},{"type":"v1","addr":"172.21.15.16:6819","nonce":37020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":37020},{"type":"v1","addr":"172.21.15.16:6823","nonce":37020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":37020},{"type":"v1","addr":"172.21.15.16:6821","nonce":37020}]},"public_addr":"172.21.15.16:6817/37020","cluster_addr":"172.21.15.16:6819/37020","heartbeat_back_addr":"172.21.15.16:6823/37020","heartbeat_front_addr":"172.21.15.16:6821/37020","state":["exists","up"]},{"osd":4,"uuid":"970500ef-95b7-42c1-b080-eed4fae7013d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":37353},{"type":"v1","addr":"172.21.15.62:6817","nonce":37353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6819","nonce":37353},{"type":"v1","addr":"172.21.15.62:6822","nonce":37353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37353},{"type":"v1","addr":"172.21.15.62:6831","nonce":37353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37353},{"type":"v1","addr":"172.21.15.62:6826","nonce":37353}]},"public_addr":"172.21.15.62:6817/37353","cluster_addr":"172.21.15.62:6822/37353","heartbeat_back_addr":"172.21.15.62:6831/37353","heartbeat_front_addr":"172.21.15.62:6826/37353","state":["exists","up"]},{"osd":5,"uuid":"022d67db-d4d8-445b-b1b9-8b446a00f5c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":37352},{"type":"v1","addr":"172.21.15.62:6801","nonce":37352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":37352},{"type":"v1","addr":"172.21.15.62:6803","nonce":37352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":37352},{"type":"v1","addr":"172.21.15.62:6807","nonce":37352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":37352},{"type":"v1","addr":"172.21.15.62:6805","nonce":37352}]},"public_addr":"172.21.15.62:6801/37352","cluster_addr":"172.21.15.62:6803/37352","heartbeat_back_addr":"172.21.15.62:6807/37352","heartbeat_front_addr":"172.21.15.62:6805/37352","state":["exists","up"]},{"osd":6,"uuid":"42bd372b-6688-4a1b-8407-e16e8c266e33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":37354},{"type":"v1","addr":"172.21.15.62:6820","nonce":37354}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":37354},{"type":"v1","addr":"172.21.15.62:6823","nonce":37354}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6829","nonce":37354},{"type":"v1","addr":"172.21.15.62:6830","nonce":37354}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6825","nonce":37354},{"type":"v1","addr":"172.21.15.62:6827","nonce":37354}]},"public_addr":"172.21.15.62:6820/37354","cluster_addr":"172.21.15.62:6823/37354","heartbeat_back_addr":"172.21.15.62:6830/37354","heartbeat_front_addr":"172.21.15.62:6827/37354","state":["exists","up"]},{"osd":7,"uuid":"ac70b2a3-f41b-4bf2-b227-7f2cbb55426d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37355},{"type":"v1","addr":"172.21.15.62:6809","nonce":37355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37355},{"type":"v1","addr":"172.21.15.62:6811","nonce":37355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":37355},{"type":"v1","addr":"172.21.15.62:6815","nonce":37355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":37355},{"type":"v1","addr":"172.21.15.62:6813","nonce":37355}]},"public_addr":"172.21.15.62:6809/37355","cluster_addr":"172.21.15.62:6811/37355","heartbeat_back_addr":"172.21.15.62:6815/37355","heartbeat_front_addr":"172.21.15.62:6813/37355","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.368896+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.107273+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.276888+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.287502+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.124791+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.251937+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.255679+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.232797+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"4.23","mappings":[{"from":3,"to":2}]},{"pgid":"4.31","mappings":[{"from":1,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-05-22T15:27:34.541 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.531+0000 7ff344bbd640 1 -- 172.21.15.16:0/2677499460 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ff3200653b0 msgr2=0x7ff320067860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:34.541 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.531+0000 7ff344bbd640 1 --2- 172.21.15.16:0/2677499460 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ff3200653b0 0x7ff320067860 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7ff330000fc0 tx=0x7ff3300011a0 comp rx=0 tx=0).stop 2023-05-22T15:27:34.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.531+0000 7ff344bbd640 1 -- 172.21.15.16:0/2677499460 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff32c0039b0 msgr2=0x7ff3401f6b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:34.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.531+0000 7ff344bbd640 1 --2- 172.21.15.16:0/2677499460 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff32c0039b0 0x7ff3401f6b80 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7ff3280b0040 tx=0x7ff32c005800 comp rx=0 tx=0).stop 2023-05-22T15:27:34.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.531+0000 7ff344bbd640 1 -- 172.21.15.16:0/2677499460 shutdown_connections 2023-05-22T15:27:34.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.531+0000 7ff344bbd640 1 --2- 172.21.15.16:0/2677499460 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff340069b20 0x7ff340069f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:34.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.531+0000 7ff344bbd640 1 --2- 172.21.15.16:0/2677499460 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ff3200653b0 0x7ff320067860 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:34.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.531+0000 7ff344bbd640 1 --2- 172.21.15.16:0/2677499460 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff3400c0d70 0x7ff340153290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:34.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.531+0000 7ff344bbd640 1 --2- 172.21.15.16:0/2677499460 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff32c0039b0 0x7ff3401f6b80 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:34.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.531+0000 7ff344bbd640 1 -- 172.21.15.16:0/2677499460 >> 172.21.15.16:0/2677499460 conn(0x7ff340069720 msgr2=0x7ff3400b2a60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:34.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.531+0000 7ff344bbd640 1 -- 172.21.15.16:0/2677499460 shutdown_connections 2023-05-22T15:27:34.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.531+0000 7ff344bbd640 1 -- 172.21.15.16:0/2677499460 wait complete. 2023-05-22T15:27:34.556 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-05-22T15:26:38.019781+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-05-22T15:26:43.570514+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2023-05-22T15:26:47.879159+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2023-05-22T15:26:48.522130+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-05-22T15:26:51.616142+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '27', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-05-22T15:27:34.557 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-05-22T15:27:34.681 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.671+0000 7fba0dda9640 1 Processor -- start 2023-05-22T15:27:34.681 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.671+0000 7fba0dda9640 1 -- start start 2023-05-22T15:27:34.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.675+0000 7fba0dda9640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fba08159d40 0x7fba080cb260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:34.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.675+0000 7fba0dda9640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fba0806b6c0 0x7fba0806bb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:34.683 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.675+0000 7fba0dda9640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba08070c30 con 0x7fba08159970 2023-05-22T15:27:34.683 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.675+0000 7fba0dda9640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba0815dd90 con 0x7fba0806b6c0 2023-05-22T15:27:34.683 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.675+0000 7fba0dda9640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba0806c040 con 0x7fba08159d40 2023-05-22T15:27:34.683 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.675+0000 7fba07fff640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fba0806b6c0 0x7fba0806bb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:34.683 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.675+0000 7fba07fff640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fba0806b6c0 0x7fba0806bb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57426/0 (socket says 172.21.15.16:57426) 2023-05-22T15:27:34.683 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.675+0000 7fba07fff640 1 -- 172.21.15.16:0/3028510580 learned_addr learned my addr 172.21.15.16:0/3028510580 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:34.683 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.675+0000 7fba06ffd640 1 --2- 172.21.15.16:0/3028510580 >> v2:172.21.15.62:3300/0 conn(0x7fba08159d40 0x7fba080cb260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:34.684 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.675+0000 7fba07fff640 1 -- 172.21.15.16:0/3028510580 >> v2:172.21.15.62:3300/0 conn(0x7fba08159d40 msgr2=0x7fba080cb260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:34.684 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.675+0000 7fba07fff640 1 --2- 172.21.15.16:0/3028510580 >> v2:172.21.15.62:3300/0 conn(0x7fba08159d40 0x7fba080cb260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:34.684 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.675+0000 7fba07fff640 1 -- 172.21.15.16:0/3028510580 >> v1:172.21.15.62:6789/0 conn(0x7fba08159970 legacy=0x7fba0815d680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:34.684 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.675+0000 7fba07fff640 1 -- 172.21.15.16:0/3028510580 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba08070e60 con 0x7fba0806b6c0 2023-05-22T15:27:34.685 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.675+0000 7fba07fff640 1 --2- 172.21.15.16:0/3028510580 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fba0806b6c0 0x7fba0806bb00 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fba0815e760 tx=0x7fba0001c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1915cf2b69518075 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:34.685 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.675+0000 7fba067fc640 1 -- 172.21.15.16:0/3028510580 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba000a56f0 con 0x7fba0806b6c0 2023-05-22T15:27:34.685 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.675+0000 7fba067fc640 1 -- 172.21.15.16:0/3028510580 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba000a5850 con 0x7fba0806b6c0 2023-05-22T15:27:34.685 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.675+0000 7fba067fc640 1 -- 172.21.15.16:0/3028510580 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba000a5be0 con 0x7fba0806b6c0 2023-05-22T15:27:34.686 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.679+0000 7fba0dda9640 1 -- 172.21.15.16:0/3028510580 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fba0806b6c0 msgr2=0x7fba0806bb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:34.686 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.679+0000 7fba0dda9640 1 --2- 172.21.15.16:0/3028510580 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fba0806b6c0 0x7fba0806bb00 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fba0815e760 tx=0x7fba0001c940 comp rx=0 tx=0).stop 2023-05-22T15:27:34.686 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.679+0000 7fba0dda9640 1 -- 172.21.15.16:0/3028510580 shutdown_connections 2023-05-22T15:27:34.686 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.679+0000 7fba0dda9640 1 --2- 172.21.15.16:0/3028510580 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fba0806b6c0 0x7fba0806bb00 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:34.686 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.679+0000 7fba0dda9640 1 --2- 172.21.15.16:0/3028510580 >> v2:172.21.15.62:3300/0 conn(0x7fba08159d40 0x7fba080cb260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:34.686 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.679+0000 7fba0dda9640 1 -- 172.21.15.16:0/3028510580 >> 172.21.15.16:0/3028510580 conn(0x7fba08069720 msgr2=0x7fba08069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:34.687 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.679+0000 7fba0dda9640 1 -- 172.21.15.16:0/3028510580 shutdown_connections 2023-05-22T15:27:34.687 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.679+0000 7fba0dda9640 1 -- 172.21.15.16:0/3028510580 wait complete. 2023-05-22T15:27:34.687 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.679+0000 7fba0dda9640 1 Processor -- start 2023-05-22T15:27:34.687 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.679+0000 7fba0dda9640 1 -- start start 2023-05-22T15:27:34.689 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.679+0000 7fba0dda9640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fba0806b6c0 0x7fba080b2010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:34.689 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.679+0000 7fba0dda9640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fba08159970 0x7fba080c2dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:34.689 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.679+0000 7fba0dda9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fba08159d40 0x7fba0814caf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:34.689 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.679+0000 7fba0dda9640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba080b2550 con 0x7fba08159d40 2023-05-22T15:27:34.689 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.679+0000 7fba0dda9640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba080c41d0 con 0x7fba08159970 2023-05-22T15:27:34.689 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.679+0000 7fba0dda9640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba080c3300 con 0x7fba0806b6c0 2023-05-22T15:27:34.690 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.679+0000 7fba07fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fba08159d40 0x7fba0814caf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:34.690 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.679+0000 7fba07fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fba08159d40 0x7fba0814caf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57430/0 (socket says 172.21.15.16:57430) 2023-05-22T15:27:34.690 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.679+0000 7fba07fff640 1 -- 172.21.15.16:0/82323828 learned_addr learned my addr 172.21.15.16:0/82323828 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:34.690 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.679+0000 7fba06ffd640 1 --2- 172.21.15.16:0/82323828 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fba08159970 0x7fba080c2dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:34.690 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.679+0000 7fba077fe640 1 --2- 172.21.15.16:0/82323828 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fba0806b6c0 0x7fba080b2010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:34.690 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.679+0000 7fba07fff640 1 -- 172.21.15.16:0/82323828 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fba0806b6c0 msgr2=0x7fba080b2010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:34.690 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.679+0000 7fba07fff640 1 --2- 172.21.15.16:0/82323828 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fba0806b6c0 0x7fba080b2010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:34.691 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.679+0000 7fba07fff640 1 -- 172.21.15.16:0/82323828 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fba08159970 msgr2=0x7fba080c2dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:34.691 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.679+0000 7fba07fff640 1 --2- 172.21.15.16:0/82323828 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fba08159970 0x7fba080c2dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:34.691 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.679+0000 7fba07fff640 1 -- 172.21.15.16:0/82323828 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba00009b30 con 0x7fba08159d40 2023-05-22T15:27:34.691 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.679+0000 7fba06ffd640 1 --2- 172.21.15.16:0/82323828 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fba08159970 0x7fba080c2dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T15:27:34.691 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.679+0000 7fba07fff640 1 --2- 172.21.15.16:0/82323828 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fba08159d40 0x7fba0814caf0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fba000af040 tx=0x7fba000196e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:34.691 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.679+0000 7fb9fffff640 1 -- 172.21.15.16:0/82323828 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba000b8670 con 0x7fba08159d40 2023-05-22T15:27:34.691 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.679+0000 7fb9fffff640 1 -- 172.21.15.16:0/82323828 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba000b87d0 con 0x7fba08159d40 2023-05-22T15:27:34.691 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.679+0000 7fba0dda9640 1 -- 172.21.15.16:0/82323828 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba080cb7a0 con 0x7fba08159d40 2023-05-22T15:27:34.692 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.679+0000 7fb9fffff640 1 -- 172.21.15.16:0/82323828 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba000b8b60 con 0x7fba08159d40 2023-05-22T15:27:34.692 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.679+0000 7fba0dda9640 1 -- 172.21.15.16:0/82323828 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba08071700 con 0x7fba08159d40 2023-05-22T15:27:34.692 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.683+0000 7fba0dda9640 1 -- 172.21.15.16:0/82323828 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba080cb930 con 0x7fba08159d40 2023-05-22T15:27:34.693 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.683+0000 7fb9fffff640 1 -- 172.21.15.16:0/82323828 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7fba000b8cc0 con 0x7fba08159d40 2023-05-22T15:27:34.693 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.683+0000 7fb9fffff640 1 --2- 172.21.15.16:0/82323828 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fb9e80653b0 0x7fb9e8067860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:34.693 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.683+0000 7fb9fffff640 1 -- 172.21.15.16:0/82323828 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6871+0+0 (secure 0 0 0) 0x7fba0011da30 con 0x7fba08159d40 2023-05-22T15:27:34.693 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.683+0000 7fba077fe640 1 --2- 172.21.15.16:0/82323828 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fb9e80653b0 0x7fb9e8067860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:34.693 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.683+0000 7fba077fe640 1 --2- 172.21.15.16:0/82323828 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fb9e80653b0 0x7fb9e8067860 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fb9f8000f70 tx=0x7fb9f8009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:34.695 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.687+0000 7fb9fffff640 1 -- 172.21.15.16:0/82323828 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fba000eb380 con 0x7fba08159d40 2023-05-22T15:27:34.849 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.839+0000 7fba0dda9640 1 -- 172.21.15.16:0/82323828 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fba0804dbe0 con 0x7fba08159d40 2023-05-22T15:27:34.850 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.843+0000 7fb9fffff640 1 -- 172.21.15.16:0/82323828 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v29) v1 ==== 93+0+10 (secure 0 0 0) 0x7fba000ee9a0 con 0x7fba08159d40 2023-05-22T15:27:34.850 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2023-05-22T15:27:34.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.843+0000 7fba0dda9640 1 -- 172.21.15.16:0/82323828 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fb9e80653b0 msgr2=0x7fb9e8067860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:34.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.843+0000 7fba0dda9640 1 --2- 172.21.15.16:0/82323828 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fb9e80653b0 0x7fb9e8067860 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fb9f8000f70 tx=0x7fb9f8009040 comp rx=0 tx=0).stop 2023-05-22T15:27:34.853 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.843+0000 7fba0dda9640 1 -- 172.21.15.16:0/82323828 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fba08159d40 msgr2=0x7fba0814caf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:34.853 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.843+0000 7fba0dda9640 1 --2- 172.21.15.16:0/82323828 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fba08159d40 0x7fba0814caf0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fba000af040 tx=0x7fba000196e0 comp rx=0 tx=0).stop 2023-05-22T15:27:34.853 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.843+0000 7fba0dda9640 1 -- 172.21.15.16:0/82323828 shutdown_connections 2023-05-22T15:27:34.854 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.843+0000 7fba0dda9640 1 --2- 172.21.15.16:0/82323828 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fba08159d40 0x7fba0814caf0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:34.854 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.843+0000 7fba0dda9640 1 --2- 172.21.15.16:0/82323828 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fba08159970 0x7fba080c2dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:34.854 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.843+0000 7fba0dda9640 1 --2- 172.21.15.16:0/82323828 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fb9e80653b0 0x7fb9e8067860 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:34.855 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.843+0000 7fba0dda9640 1 --2- 172.21.15.16:0/82323828 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fba0806b6c0 0x7fba080b2010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:34.855 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.843+0000 7fba0dda9640 1 -- 172.21.15.16:0/82323828 >> 172.21.15.16:0/82323828 conn(0x7fba08069720 msgr2=0x7fba0815b0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:34.855 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.843+0000 7fba0dda9640 1 -- 172.21.15.16:0/82323828 shutdown_connections 2023-05-22T15:27:34.855 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.843+0000 7fba0dda9640 1 -- 172.21.15.16:0/82323828 wait complete. 2023-05-22T15:27:34.869 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-05-22T15:27:34.989 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.983+0000 7fd7f098b640 1 Processor -- start 2023-05-22T15:27:34.990 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.983+0000 7fd7f098b640 1 -- start start 2023-05-22T15:27:34.992 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.983+0000 7fd7f098b640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd7ec0cf870 0x7fd7ec06db00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:34.992 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.983+0000 7fd7f098b640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7ec06e040 con 0x7fd7ec15cfa0 2023-05-22T15:27:34.993 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.983+0000 7fd7f098b640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7ec06e490 con 0x7fd7ec06e740 2023-05-22T15:27:34.993 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.983+0000 7fd7f098b640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7ec0cb9f0 con 0x7fd7ec0cf870 2023-05-22T15:27:34.993 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.983+0000 7fd7ea59c640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fd7ec15cfa0 0x7fd7ec0b1fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:36714/0 (socket says 172.21.15.16:36714) 2023-05-22T15:27:34.993 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.983+0000 7fd7ea59c640 1 -- 172.21.15.16:0/3688707368 learned_addr learned my addr 172.21.15.16:0/3688707368 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:27:34.993 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.983+0000 7fd7e9d9b640 1 --2- 172.21.15.16:0/3688707368 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd7ec0cf870 0x7fd7ec06db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:34.993 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.983+0000 7fd7e959a640 1 -- 172.21.15.16:0/3688707368 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3054599178 0 0) 0x7fd7dc001120 con 0x7fd7ec15cfa0 2023-05-22T15:27:34.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.983+0000 7fd7e959a640 1 -- 172.21.15.16:0/3688707368 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd7d4003530 con 0x7fd7ec15cfa0 2023-05-22T15:27:34.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.983+0000 7fd7e959a640 1 -- 172.21.15.16:0/3688707368 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2875082616 0 0) 0x7fd7d4003530 con 0x7fd7ec15cfa0 2023-05-22T15:27:34.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.983+0000 7fd7e959a640 1 -- 172.21.15.16:0/3688707368 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd7ec0cf870 msgr2=0x7fd7ec06db00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:34.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.983+0000 7fd7e959a640 1 --2- 172.21.15.16:0/3688707368 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd7ec0cf870 0x7fd7ec06db00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:34.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.983+0000 7fd7e959a640 1 -- 172.21.15.16:0/3688707368 >> v1:172.21.15.62:6789/0 conn(0x7fd7ec06e740 legacy=0x7fd7ec070de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:34.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.983+0000 7fd7e959a640 1 -- 172.21.15.16:0/3688707368 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7ec0cad70 con 0x7fd7ec15cfa0 2023-05-22T15:27:34.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.983+0000 7fd7e959a640 1 -- 172.21.15.16:0/3688707368 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7fd7dc001810 con 0x7fd7ec15cfa0 2023-05-22T15:27:34.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.983+0000 7fd7e959a640 1 -- 172.21.15.16:0/3688707368 >> v1:172.21.15.16:6789/0 conn(0x7fd7ec15cfa0 legacy=0x7fd7ec0b1fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:34.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.983+0000 7fd7e959a640 1 --2- 172.21.15.16:0/3688707368 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd7ec0cf870 0x7fd7d40039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:34.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.983+0000 7fd7e959a640 1 --2- 172.21.15.16:0/3688707368 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd7d4003f30 0x7fd7d40047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:34.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.983+0000 7fd7e959a640 1 --2- 172.21.15.16:0/3688707368 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd7d4004d10 0x7fd7d4005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:34.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.983+0000 7fd7e959a640 1 -- 172.21.15.16:0/3688707368 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7ec0cb9f0 con 0x7fd7d4003f30 2023-05-22T15:27:34.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.983+0000 7fd7e959a640 1 -- 172.21.15.16:0/3688707368 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7d400bb20 con 0x7fd7d4004d10 2023-05-22T15:27:34.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.983+0000 7fd7e959a640 1 -- 172.21.15.16:0/3688707368 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7d400bce0 con 0x7fd7ec0cf870 2023-05-22T15:27:34.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.983+0000 7fd7e9d9b640 1 --2- 172.21.15.16:0/3688707368 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd7d4003f30 0x7fd7d40047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:34.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.983+0000 7fd7ea59c640 1 --2- 172.21.15.16:0/3688707368 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd7ec0cf870 0x7fd7d40039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:34.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.983+0000 7fd7ead9d640 1 --2- 172.21.15.16:0/3688707368 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd7d4004d10 0x7fd7d4005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:34.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.983+0000 7fd7e9d9b640 1 -- 172.21.15.16:0/3688707368 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd7ec0cf870 msgr2=0x7fd7d40039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:34.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.983+0000 7fd7e9d9b640 1 --2- 172.21.15.16:0/3688707368 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd7ec0cf870 0x7fd7d40039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:34.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.983+0000 7fd7e9d9b640 1 -- 172.21.15.16:0/3688707368 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd7d4004d10 msgr2=0x7fd7d4005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:34.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.983+0000 7fd7e9d9b640 1 --2- 172.21.15.16:0/3688707368 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd7d4004d10 0x7fd7d4005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:34.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.983+0000 7fd7e9d9b640 1 -- 172.21.15.16:0/3688707368 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7d4003530 con 0x7fd7d4003f30 2023-05-22T15:27:35.002 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.987+0000 7fd7e9d9b640 1 --2- 172.21.15.16:0/3688707368 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd7d4003f30 0x7fd7d40047d0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fd7e0009ab0 tx=0x7fd7e001ca00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e5c3f887b2e9e58 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:35.003 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.987+0000 7fd7e959a640 1 -- 172.21.15.16:0/3688707368 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7e00a64f0 con 0x7fd7d4003f30 2023-05-22T15:27:35.003 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.987+0000 7fd7e959a640 1 -- 172.21.15.16:0/3688707368 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7e00a6650 con 0x7fd7d4003f30 2023-05-22T15:27:35.003 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.987+0000 7fd7e959a640 1 -- 172.21.15.16:0/3688707368 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7e00a6a00 con 0x7fd7d4003f30 2023-05-22T15:27:35.003 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.987+0000 7fd7f098b640 1 -- 172.21.15.16:0/3688707368 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd7d4003f30 msgr2=0x7fd7d40047d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:35.003 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.987+0000 7fd7f098b640 1 --2- 172.21.15.16:0/3688707368 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd7d4003f30 0x7fd7d40047d0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fd7e0009ab0 tx=0x7fd7e001ca00 comp rx=0 tx=0).stop 2023-05-22T15:27:35.003 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.987+0000 7fd7ead9d640 1 -- 172.21.15.16:0/3688707368 reap_dead start 2023-05-22T15:27:35.003 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.987+0000 7fd7f098b640 1 -- 172.21.15.16:0/3688707368 shutdown_connections 2023-05-22T15:27:35.004 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.987+0000 7fd7f098b640 1 -- 172.21.15.16:0/3688707368 >> 172.21.15.16:0/3688707368 conn(0x7fd7ec0cc870 msgr2=0x7fd7ec0cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:35.004 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.987+0000 7fd7f098b640 1 -- 172.21.15.16:0/3688707368 shutdown_connections 2023-05-22T15:27:35.004 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.987+0000 7fd7f098b640 1 -- 172.21.15.16:0/3688707368 wait complete. 2023-05-22T15:27:35.004 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.987+0000 7fd7f098b640 1 Processor -- start 2023-05-22T15:27:35.005 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.987+0000 7fd7f098b640 1 -- start start 2023-05-22T15:27:35.005 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.987+0000 7fd7f098b640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd7d4003f30 0x7fd7ec153fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:35.005 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.987+0000 7fd7f098b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd7ec0c6230 0x7fd7ec0c66a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:35.005 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.987+0000 7fd7f098b640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd7ec0b28c0 0x7fd7ec0b2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:35.005 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.987+0000 7fd7f098b640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7ec0c7270 con 0x7fd7ec0c6230 2023-05-22T15:27:35.005 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.987+0000 7fd7f098b640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7ec0c6cb0 con 0x7fd7d4003f30 2023-05-22T15:27:35.005 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.987+0000 7fd7f098b640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7ec0c6f90 con 0x7fd7ec0b28c0 2023-05-22T15:27:35.006 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.987+0000 7fd7e9d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd7ec0c6230 0x7fd7ec0c66a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:35.006 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.987+0000 7fd7e9d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd7ec0c6230 0x7fd7ec0c66a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57446/0 (socket says 172.21.15.16:57446) 2023-05-22T15:27:35.006 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.987+0000 7fd7e9d9b640 1 -- 172.21.15.16:0/2033136947 learned_addr learned my addr 172.21.15.16:0/2033136947 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:35.006 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.987+0000 7fd7ea59c640 1 --2- 172.21.15.16:0/2033136947 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd7d4003f30 0x7fd7ec153fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:35.006 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.987+0000 7fd7ead9d640 1 --2- 172.21.15.16:0/2033136947 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd7ec0b28c0 0x7fd7ec0b2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:35.006 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.987+0000 7fd7e9d9b640 1 -- 172.21.15.16:0/2033136947 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd7ec0b28c0 msgr2=0x7fd7ec0b2cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:35.006 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.987+0000 7fd7e9d9b640 1 --2- 172.21.15.16:0/2033136947 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd7ec0b28c0 0x7fd7ec0b2cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:35.006 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.991+0000 7fd7e9d9b640 1 -- 172.21.15.16:0/2033136947 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd7d4003f30 msgr2=0x7fd7ec153fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:35.007 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.991+0000 7fd7e9d9b640 1 --2- 172.21.15.16:0/2033136947 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd7d4003f30 0x7fd7ec153fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:35.007 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.991+0000 7fd7e9d9b640 1 -- 172.21.15.16:0/2033136947 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7ec071900 con 0x7fd7ec0c6230 2023-05-22T15:27:35.007 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.991+0000 7fd7e9d9b640 1 --2- 172.21.15.16:0/2033136947 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd7ec0c6230 0x7fd7ec0c66a0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fd7e001c830 tx=0x7fd7d4005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:35.007 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.991+0000 7fd7daffd640 1 -- 172.21.15.16:0/2033136947 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7e00a67c0 con 0x7fd7ec0c6230 2023-05-22T15:27:35.007 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.991+0000 7fd7daffd640 1 -- 172.21.15.16:0/2033136947 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7e00a6960 con 0x7fd7ec0c6230 2023-05-22T15:27:35.007 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.991+0000 7fd7f098b640 1 -- 172.21.15.16:0/2033136947 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7ec0cb610 con 0x7fd7ec0c6230 2023-05-22T15:27:35.007 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.991+0000 7fd7daffd640 1 -- 172.21.15.16:0/2033136947 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7e00a6d10 con 0x7fd7ec0c6230 2023-05-22T15:27:35.007 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.991+0000 7fd7f098b640 1 -- 172.21.15.16:0/2033136947 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7ec15d370 con 0x7fd7ec0c6230 2023-05-22T15:27:35.008 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.991+0000 7fd7f098b640 1 -- 172.21.15.16:0/2033136947 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7ec04dbe0 con 0x7fd7ec0c6230 2023-05-22T15:27:35.008 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.991+0000 7fd7daffd640 1 -- 172.21.15.16:0/2033136947 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7fd7e0006040 con 0x7fd7ec0c6230 2023-05-22T15:27:35.008 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.991+0000 7fd7daffd640 1 --2- 172.21.15.16:0/2033136947 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd7cc065380 0x7fd7cc067830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:35.008 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.991+0000 7fd7ea59c640 1 --2- 172.21.15.16:0/2033136947 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd7cc065380 0x7fd7cc067830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:35.008 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.991+0000 7fd7daffd640 1 -- 172.21.15.16:0/2033136947 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6871+0+0 (secure 0 0 0) 0x7fd7e00aa070 con 0x7fd7ec0c6230 2023-05-22T15:27:35.008 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.991+0000 7fd7ea59c640 1 --2- 172.21.15.16:0/2033136947 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd7cc065380 0x7fd7cc067830 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fd7ec0aeb60 tx=0x7fd7dc009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:35.009 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:34.995+0000 7fd7daffd640 1 -- 172.21.15.16:0/2033136947 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fd7e00ef040 con 0x7fd7ec0c6230 2023-05-22T15:27:35.151 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.143+0000 7fd7f098b640 1 -- 172.21.15.16:0/2033136947 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fd7ec15b0b0 con 0x7fd7ec0c6230 2023-05-22T15:27:35.152 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.143+0000 7fd7daffd640 1 -- 172.21.15.16:0/2033136947 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v29) v1 ==== 92+0+10 (secure 0 0 0) 0x7fd7e00eebd0 con 0x7fd7ec0c6230 2023-05-22T15:27:35.152 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 8 2023-05-22T15:27:35.154 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.147+0000 7fd7f098b640 1 -- 172.21.15.16:0/2033136947 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd7cc065380 msgr2=0x7fd7cc067830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:35.154 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.147+0000 7fd7f098b640 1 --2- 172.21.15.16:0/2033136947 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd7cc065380 0x7fd7cc067830 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fd7ec0aeb60 tx=0x7fd7dc009040 comp rx=0 tx=0).stop 2023-05-22T15:27:35.154 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.147+0000 7fd7f098b640 1 -- 172.21.15.16:0/2033136947 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd7ec0c6230 msgr2=0x7fd7ec0c66a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:35.154 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.147+0000 7fd7f098b640 1 --2- 172.21.15.16:0/2033136947 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd7ec0c6230 0x7fd7ec0c66a0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fd7e001c830 tx=0x7fd7d4005800 comp rx=0 tx=0).stop 2023-05-22T15:27:35.155 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.147+0000 7fd7f098b640 1 -- 172.21.15.16:0/2033136947 shutdown_connections 2023-05-22T15:27:35.155 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.147+0000 7fd7f098b640 1 --2- 172.21.15.16:0/2033136947 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd7cc065380 0x7fd7cc067830 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:35.155 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.147+0000 7fd7f098b640 1 --2- 172.21.15.16:0/2033136947 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd7ec0b28c0 0x7fd7ec0b2cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:35.155 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.147+0000 7fd7f098b640 1 --2- 172.21.15.16:0/2033136947 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd7ec0c6230 0x7fd7ec0c66a0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:35.155 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.147+0000 7fd7f098b640 1 --2- 172.21.15.16:0/2033136947 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd7d4003f30 0x7fd7ec153fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:35.155 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.147+0000 7fd7f098b640 1 -- 172.21.15.16:0/2033136947 >> 172.21.15.16:0/2033136947 conn(0x7fd7ec0cc870 msgr2=0x7fd7ec164f70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:35.155 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.147+0000 7fd7f098b640 1 -- 172.21.15.16:0/2033136947 shutdown_connections 2023-05-22T15:27:35.156 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.147+0000 7fd7f098b640 1 -- 172.21.15.16:0/2033136947 wait complete. 2023-05-22T15:27:35.169 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-05-22T15:27:35.286 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.279+0000 7f3632f1b640 1 Processor -- start 2023-05-22T15:27:35.286 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.279+0000 7f3632f1b640 1 -- start start 2023-05-22T15:27:35.287 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.279+0000 7f3632f1b640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f362c06b6c0 0x7f362c0cb4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:35.287 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.279+0000 7f3632f1b640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f362c159f70 con 0x7f362c06ba90 2023-05-22T15:27:35.288 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.279+0000 7f3632f1b640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f362c0cb030 con 0x7f362c1596e0 2023-05-22T15:27:35.288 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.279+0000 7f3632f1b640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f362c071760 con 0x7f362c06b6c0 2023-05-22T15:27:35.288 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.279+0000 7f3630cb7640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f362c06b6c0 0x7f362c0cb4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:35.289 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.279+0000 7f3630cb7640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f362c06b6c0 0x7f362c0cb4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57462/0 (socket says 172.21.15.16:57462) 2023-05-22T15:27:35.289 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.279+0000 7f362bfff640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f362c06ba90 0x7f362c158fd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:36718/0 (socket says 172.21.15.16:36718) 2023-05-22T15:27:35.289 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.279+0000 7f3630cb7640 1 -- 172.21.15.16:0/444784753 learned_addr learned my addr 172.21.15.16:0/444784753 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:35.289 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.279+0000 7f3630cb7640 1 -- 172.21.15.16:0/444784753 >> v1:172.21.15.62:6789/0 conn(0x7f362c1596e0 legacy=0x7f362c06fba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:35.290 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.279+0000 7f3630cb7640 1 -- 172.21.15.16:0/444784753 >> v1:172.21.15.16:6789/0 conn(0x7f362c06ba90 legacy=0x7f362c158fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:35.290 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.279+0000 7f3630cb7640 1 -- 172.21.15.16:0/444784753 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f362c159c90 con 0x7f362c06b6c0 2023-05-22T15:27:35.290 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.279+0000 7f362b7fe640 1 -- 172.21.15.16:0/444784753 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 800627867 0 0) 0x7f362c159f70 con 0x7f362c06ba90 2023-05-22T15:27:35.297 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.283+0000 7f3630cb7640 1 --2- 172.21.15.16:0/444784753 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f362c06b6c0 0x7f362c0cb4c0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f361c009870 tx=0x7f361c01c470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad85686aed01af1c server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:35.297 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.283+0000 7f362b7fe640 1 -- 172.21.15.16:0/444784753 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f361c0ab600 con 0x7f362c06b6c0 2023-05-22T15:27:35.297 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.283+0000 7f362b7fe640 1 -- 172.21.15.16:0/444784753 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f361c0a7050 con 0x7f362c06b6c0 2023-05-22T15:27:35.298 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.283+0000 7f362b7fe640 1 -- 172.21.15.16:0/444784753 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f361c0ab980 con 0x7f362c06b6c0 2023-05-22T15:27:35.298 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.283+0000 7f3632f1b640 1 -- 172.21.15.16:0/444784753 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f362c06b6c0 msgr2=0x7f362c0cb4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:35.298 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.283+0000 7f3632f1b640 1 --2- 172.21.15.16:0/444784753 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f362c06b6c0 0x7f362c0cb4c0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f361c009870 tx=0x7f361c01c470 comp rx=0 tx=0).stop 2023-05-22T15:27:35.298 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.283+0000 7f3632f1b640 1 -- 172.21.15.16:0/444784753 shutdown_connections 2023-05-22T15:27:35.299 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.283+0000 7f3632f1b640 1 --2- 172.21.15.16:0/444784753 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f362c06b6c0 0x7f362c0cb4c0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:35.299 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.283+0000 7f3632f1b640 1 -- 172.21.15.16:0/444784753 >> 172.21.15.16:0/444784753 conn(0x7f362c069720 msgr2=0x7f362c069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:35.299 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.283+0000 7f3632f1b640 1 -- 172.21.15.16:0/444784753 shutdown_connections 2023-05-22T15:27:35.299 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.283+0000 7f3632f1b640 1 -- 172.21.15.16:0/444784753 wait complete. 2023-05-22T15:27:35.299 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.283+0000 7f3632f1b640 1 Processor -- start 2023-05-22T15:27:35.299 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.283+0000 7f3632f1b640 1 -- start start 2023-05-22T15:27:35.300 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.283+0000 7f3632f1b640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f362c06b6c0 0x7f362c142ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:35.300 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.283+0000 7f3632f1b640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f362c06ba90 0x7f362c0c9910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:35.300 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.283+0000 7f3632f1b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f362c1596e0 0x7f362c15b610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:35.300 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.283+0000 7f3632f1b640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f362c0b06f0 con 0x7f362c1596e0 2023-05-22T15:27:35.300 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.283+0000 7f3632f1b640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f362c0afb20 con 0x7f362c06b6c0 2023-05-22T15:27:35.300 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.283+0000 7f3632f1b640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f362c0afdc0 con 0x7f362c06ba90 2023-05-22T15:27:35.301 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.283+0000 7f36314b8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f362c1596e0 0x7f362c15b610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:35.301 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.283+0000 7f36314b8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f362c1596e0 0x7f362c15b610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57470/0 (socket says 172.21.15.16:57470) 2023-05-22T15:27:35.301 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.283+0000 7f36314b8640 1 -- 172.21.15.16:0/2858702977 learned_addr learned my addr 172.21.15.16:0/2858702977 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:35.301 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.283+0000 7f362bfff640 1 --2- 172.21.15.16:0/2858702977 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f362c06ba90 0x7f362c0c9910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:35.301 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.283+0000 7f3630cb7640 1 --2- 172.21.15.16:0/2858702977 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f362c06b6c0 0x7f362c142ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:35.301 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.283+0000 7f36314b8640 1 -- 172.21.15.16:0/2858702977 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f362c06ba90 msgr2=0x7f362c0c9910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:35.301 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.283+0000 7f36314b8640 1 --2- 172.21.15.16:0/2858702977 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f362c06ba90 0x7f362c0c9910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:35.302 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.283+0000 7f36314b8640 1 -- 172.21.15.16:0/2858702977 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f362c06b6c0 msgr2=0x7f362c142ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:35.302 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.283+0000 7f36314b8640 1 --2- 172.21.15.16:0/2858702977 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f362c06b6c0 0x7f362c142ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:35.302 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.283+0000 7f36314b8640 1 -- 172.21.15.16:0/2858702977 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f361c0094d0 con 0x7f362c1596e0 2023-05-22T15:27:35.302 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.283+0000 7f36314b8640 1 --2- 172.21.15.16:0/2858702977 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f362c1596e0 0x7f362c15b610 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f362400b970 tx=0x7f362400be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:35.302 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.283+0000 7f36297fa640 1 -- 172.21.15.16:0/2858702977 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f362400cb40 con 0x7f362c1596e0 2023-05-22T15:27:35.302 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.283+0000 7f36297fa640 1 -- 172.21.15.16:0/2858702977 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f362400cca0 con 0x7f362c1596e0 2023-05-22T15:27:35.302 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.283+0000 7f36297fa640 1 -- 172.21.15.16:0/2858702977 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3624014680 con 0x7f362c1596e0 2023-05-22T15:27:35.303 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.283+0000 7f3632f1b640 1 -- 172.21.15.16:0/2858702977 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f362c06c030 con 0x7f362c1596e0 2023-05-22T15:27:35.303 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.283+0000 7f3632f1b640 1 -- 172.21.15.16:0/2858702977 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f362c06bea0 con 0x7f362c1596e0 2023-05-22T15:27:35.303 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.287+0000 7f3632f1b640 1 -- 172.21.15.16:0/2858702977 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f362c156840 con 0x7f362c1596e0 2023-05-22T15:27:35.304 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.287+0000 7f36297fa640 1 -- 172.21.15.16:0/2858702977 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7f362401d3b0 con 0x7f362c1596e0 2023-05-22T15:27:35.304 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.287+0000 7f36297fa640 1 --2- 172.21.15.16:0/2858702977 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f3618065380 0x7f3618067830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:35.304 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.287+0000 7f36297fa640 1 -- 172.21.15.16:0/2858702977 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f36240829f0 con 0x7f362c1596e0 2023-05-22T15:27:35.304 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.287+0000 7f3630cb7640 1 --2- 172.21.15.16:0/2858702977 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f3618065380 0x7f3618067830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:35.305 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.287+0000 7f3630cb7640 1 --2- 172.21.15.16:0/2858702977 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f3618065380 0x7f3618067830 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f361c0ad000 tx=0x7f361c008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:35.306 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.291+0000 7f36297fa640 1 -- 172.21.15.16:0/2858702977 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f36240502e0 con 0x7f362c1596e0 2023-05-22T15:27:35.447 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.439+0000 7f3632f1b640 1 -- 172.21.15.16:0/2858702977 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f362c04dbe0 con 0x7f362c1596e0 2023-05-22T15:27:35.448 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.439+0000 7f36297fa640 1 -- 172.21.15.16:0/2858702977 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v29) v1 ==== 104+0+11 (secure 0 0 0) 0x7f3624053900 con 0x7f362c1596e0 2023-05-22T15:27:35.448 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2023-05-22T15:27:35.451 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.443+0000 7f3632f1b640 1 -- 172.21.15.16:0/2858702977 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f3618065380 msgr2=0x7f3618067830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:35.451 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.443+0000 7f3632f1b640 1 --2- 172.21.15.16:0/2858702977 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f3618065380 0x7f3618067830 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f361c0ad000 tx=0x7f361c008040 comp rx=0 tx=0).stop 2023-05-22T15:27:35.451 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.443+0000 7f3632f1b640 1 -- 172.21.15.16:0/2858702977 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f362c1596e0 msgr2=0x7f362c15b610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:35.452 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.443+0000 7f3632f1b640 1 --2- 172.21.15.16:0/2858702977 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f362c1596e0 0x7f362c15b610 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f362400b970 tx=0x7f362400be30 comp rx=0 tx=0).stop 2023-05-22T15:27:35.452 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.443+0000 7f3632f1b640 1 -- 172.21.15.16:0/2858702977 shutdown_connections 2023-05-22T15:27:35.452 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.443+0000 7f3632f1b640 1 --2- 172.21.15.16:0/2858702977 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f362c1596e0 0x7f362c15b610 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:35.452 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.443+0000 7f3632f1b640 1 --2- 172.21.15.16:0/2858702977 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f3618065380 0x7f3618067830 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:35.452 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.443+0000 7f3632f1b640 1 --2- 172.21.15.16:0/2858702977 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f362c06ba90 0x7f362c0c9910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:35.452 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.443+0000 7f3632f1b640 1 --2- 172.21.15.16:0/2858702977 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f362c06b6c0 0x7f362c142ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:35.453 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.443+0000 7f3632f1b640 1 -- 172.21.15.16:0/2858702977 >> 172.21.15.16:0/2858702977 conn(0x7f362c069720 msgr2=0x7f362c156b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:35.453 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.443+0000 7f3632f1b640 1 -- 172.21.15.16:0/2858702977 shutdown_connections 2023-05-22T15:27:35.453 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.443+0000 7f3632f1b640 1 -- 172.21.15.16:0/2858702977 wait complete. 2023-05-22T15:27:35.466 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-05-22T15:27:35.589 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.579+0000 7fb8e8d59640 1 Processor -- start 2023-05-22T15:27:35.589 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.579+0000 7fb8e8d59640 1 -- start start 2023-05-22T15:27:35.591 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.583+0000 7fb8e8d59640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb8e4158210 0x7fb8e4159160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:35.591 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.583+0000 7fb8e8d59640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb8e415bcd0 0x7fb8e415c120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:35.591 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.583+0000 7fb8e8d59640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fb8e40c8660 0x7fb8e40c8ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:35.591 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.583+0000 7fb8e8d59640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8e40c9020 con 0x7fb8e415bcd0 2023-05-22T15:27:35.592 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.583+0000 7fb8e8d59640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8e4158cd0 con 0x7fb8e40c8660 2023-05-22T15:27:35.592 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.583+0000 7fb8e8d59640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8e41596a0 con 0x7fb8e4158210 2023-05-22T15:27:35.592 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.583+0000 7fb8e1d9b640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb8e415bcd0 0x7fb8e415c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:35.593 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.583+0000 7fb8e1d9b640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb8e415bcd0 0x7fb8e415c120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57480/0 (socket says 172.21.15.16:57480) 2023-05-22T15:27:35.593 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.583+0000 7fb8e1d9b640 1 -- 172.21.15.16:0/1203862896 learned_addr learned my addr 172.21.15.16:0/1203862896 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:35.593 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.583+0000 7fb8e2d9d640 1 --2- 172.21.15.16:0/1203862896 >> v2:172.21.15.62:3300/0 conn(0x7fb8e40c8660 0x7fb8e40c8ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:35.593 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.583+0000 7fb8e259c640 1 --2- 172.21.15.16:0/1203862896 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb8e4158210 0x7fb8e4159160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:35.594 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.583+0000 7fb8e1d9b640 1 -- 172.21.15.16:0/1203862896 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb8e4158210 msgr2=0x7fb8e4159160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:35.594 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.583+0000 7fb8e1d9b640 1 --2- 172.21.15.16:0/1203862896 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb8e4158210 0x7fb8e4159160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:35.594 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.583+0000 7fb8e1d9b640 1 -- 172.21.15.16:0/1203862896 >> v2:172.21.15.62:3300/0 conn(0x7fb8e40c8660 msgr2=0x7fb8e40c8ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:35.594 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.583+0000 7fb8e1d9b640 1 --2- 172.21.15.16:0/1203862896 >> v2:172.21.15.62:3300/0 conn(0x7fb8e40c8660 0x7fb8e40c8ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:35.595 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.583+0000 7fb8e1d9b640 1 -- 172.21.15.16:0/1203862896 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8e415c790 con 0x7fb8e415bcd0 2023-05-22T15:27:35.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.583+0000 7fb8e1d9b640 1 --2- 172.21.15.16:0/1203862896 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb8e415bcd0 0x7fb8e415c120 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fb8e40c9320 tx=0x7fb8cc01c440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6d80c04e89cd130 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:35.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.583+0000 7fb8e159a640 1 -- 172.21.15.16:0/1203862896 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8cc0ab600 con 0x7fb8e415bcd0 2023-05-22T15:27:35.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.583+0000 7fb8e159a640 1 -- 172.21.15.16:0/1203862896 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8cc0ab760 con 0x7fb8e415bcd0 2023-05-22T15:27:35.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.587+0000 7fb8e159a640 1 -- 172.21.15.16:0/1203862896 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8cc0abaf0 con 0x7fb8e415bcd0 2023-05-22T15:27:35.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.587+0000 7fb8e8d59640 1 -- 172.21.15.16:0/1203862896 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb8e415bcd0 msgr2=0x7fb8e415c120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:35.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.587+0000 7fb8e8d59640 1 --2- 172.21.15.16:0/1203862896 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb8e415bcd0 0x7fb8e415c120 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fb8e40c9320 tx=0x7fb8cc01c440 comp rx=0 tx=0).stop 2023-05-22T15:27:35.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.587+0000 7fb8e8d59640 1 -- 172.21.15.16:0/1203862896 shutdown_connections 2023-05-22T15:27:35.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.587+0000 7fb8e8d59640 1 --2- 172.21.15.16:0/1203862896 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb8e415bcd0 0x7fb8e415c120 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:35.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.587+0000 7fb8e8d59640 1 --2- 172.21.15.16:0/1203862896 >> v2:172.21.15.62:3300/0 conn(0x7fb8e40c8660 0x7fb8e40c8ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:35.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.587+0000 7fb8e8d59640 1 --2- 172.21.15.16:0/1203862896 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb8e4158210 0x7fb8e4159160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:35.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.587+0000 7fb8e8d59640 1 -- 172.21.15.16:0/1203862896 >> 172.21.15.16:0/1203862896 conn(0x7fb8e4069720 msgr2=0x7fb8e4069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:35.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.587+0000 7fb8e8d59640 1 -- 172.21.15.16:0/1203862896 shutdown_connections 2023-05-22T15:27:35.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.587+0000 7fb8e8d59640 1 -- 172.21.15.16:0/1203862896 wait complete. 2023-05-22T15:27:35.600 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.587+0000 7fb8e8d59640 1 Processor -- start 2023-05-22T15:27:35.600 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.587+0000 7fb8e8d59640 1 -- start start 2023-05-22T15:27:35.600 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.587+0000 7fb8e8d59640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb8e40c8660 0x7fb8e4150040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:35.600 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.587+0000 7fb8e8d59640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb8e4158210 0x7fb8e4146fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:35.600 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.587+0000 7fb8e8d59640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb8e415bcd0 0x7fb8e41597e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:35.601 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.587+0000 7fb8e8d59640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8e4155c40 con 0x7fb8e4158210 2023-05-22T15:27:35.601 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.587+0000 7fb8e8d59640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8e41610c0 con 0x7fb8e40c8660 2023-05-22T15:27:35.601 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.587+0000 7fb8e8d59640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8e41558f0 con 0x7fb8e415bcd0 2023-05-22T15:27:35.601 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.587+0000 7fb8e1d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb8e4158210 0x7fb8e4146fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:35.601 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.587+0000 7fb8e1d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb8e4158210 0x7fb8e4146fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57484/0 (socket says 172.21.15.16:57484) 2023-05-22T15:27:35.601 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.587+0000 7fb8e1d9b640 1 -- 172.21.15.16:0/3144976859 learned_addr learned my addr 172.21.15.16:0/3144976859 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:35.601 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.587+0000 7fb8e2d9d640 1 --2- 172.21.15.16:0/3144976859 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb8e415bcd0 0x7fb8e41597e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:35.602 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.587+0000 7fb8e259c640 1 --2- 172.21.15.16:0/3144976859 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb8e40c8660 0x7fb8e4150040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:35.602 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.587+0000 7fb8e1d9b640 1 -- 172.21.15.16:0/3144976859 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb8e415bcd0 msgr2=0x7fb8e41597e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:35.602 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.587+0000 7fb8e1d9b640 1 --2- 172.21.15.16:0/3144976859 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb8e415bcd0 0x7fb8e41597e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:35.602 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.587+0000 7fb8e1d9b640 1 -- 172.21.15.16:0/3144976859 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb8e40c8660 msgr2=0x7fb8e4150040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:35.602 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.587+0000 7fb8e1d9b640 1 --2- 172.21.15.16:0/3144976859 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb8e40c8660 0x7fb8e4150040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:35.602 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.587+0000 7fb8e1d9b640 1 -- 172.21.15.16:0/3144976859 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8cc0094d0 con 0x7fb8e4158210 2023-05-22T15:27:35.602 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.587+0000 7fb8e259c640 1 --2- 172.21.15.16:0/3144976859 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb8e40c8660 0x7fb8e4150040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:27:35.603 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.587+0000 7fb8e1d9b640 1 --2- 172.21.15.16:0/3144976859 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb8e4158210 0x7fb8e4146fe0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fb8cc01c920 tx=0x7fb8cc01ac80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:35.603 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.587+0000 7fb8daffd640 1 -- 172.21.15.16:0/3144976859 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8cc0be5f0 con 0x7fb8e4158210 2023-05-22T15:27:35.603 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.587+0000 7fb8daffd640 1 -- 172.21.15.16:0/3144976859 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8cc0be770 con 0x7fb8e4158210 2023-05-22T15:27:35.603 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.587+0000 7fb8e8d59640 1 -- 172.21.15.16:0/3144976859 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8e41585e0 con 0x7fb8e4158210 2023-05-22T15:27:35.603 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.587+0000 7fb8daffd640 1 -- 172.21.15.16:0/3144976859 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8cc0beb20 con 0x7fb8e4158210 2023-05-22T15:27:35.603 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.587+0000 7fb8e8d59640 1 -- 172.21.15.16:0/3144976859 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8e40cd980 con 0x7fb8e4158210 2023-05-22T15:27:35.604 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.591+0000 7fb8e8d59640 1 -- 172.21.15.16:0/3144976859 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8e40caea0 con 0x7fb8e4158210 2023-05-22T15:27:35.604 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.591+0000 7fb8daffd640 1 -- 172.21.15.16:0/3144976859 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7fb8cc0b43f0 con 0x7fb8e4158210 2023-05-22T15:27:35.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.591+0000 7fb8daffd640 1 --2- 172.21.15.16:0/3144976859 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fb8c4065380 0x7fb8c4067830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:35.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.591+0000 7fb8daffd640 1 -- 172.21.15.16:0/3144976859 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6871+0+0 (secure 0 0 0) 0x7fb8cc11da00 con 0x7fb8e4158210 2023-05-22T15:27:35.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.591+0000 7fb8e259c640 1 --2- 172.21.15.16:0/3144976859 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fb8c4065380 0x7fb8c4067830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:35.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.591+0000 7fb8e259c640 1 --2- 172.21.15.16:0/3144976859 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fb8c4065380 0x7fb8c4067830 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fb8dc007520 tx=0x7fb8dc00b040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:35.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.595+0000 7fb8daffd640 1 -- 172.21.15.16:0/3144976859 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fb8cc0eb2e0 con 0x7fb8e4158210 2023-05-22T15:27:35.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.747+0000 7fb8e8d59640 1 -- 172.21.15.16:0/3144976859 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fb8e4155d80 con 0x7fb8e4158210 2023-05-22T15:27:35.758 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.751+0000 7fb8daffd640 1 -- 172.21.15.16:0/3144976859 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v29) v1 ==== 100+0+11 (secure 0 0 0) 0x7fb8e4155d80 con 0x7fb8e4158210 2023-05-22T15:27:35.758 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 64 2023-05-22T15:27:35.760 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.751+0000 7fb8e8d59640 1 -- 172.21.15.16:0/3144976859 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fb8c4065380 msgr2=0x7fb8c4067830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:35.760 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.751+0000 7fb8e8d59640 1 --2- 172.21.15.16:0/3144976859 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fb8c4065380 0x7fb8c4067830 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fb8dc007520 tx=0x7fb8dc00b040 comp rx=0 tx=0).stop 2023-05-22T15:27:35.760 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.751+0000 7fb8e8d59640 1 -- 172.21.15.16:0/3144976859 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb8e4158210 msgr2=0x7fb8e4146fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:35.761 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.751+0000 7fb8e8d59640 1 --2- 172.21.15.16:0/3144976859 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb8e4158210 0x7fb8e4146fe0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fb8cc01c920 tx=0x7fb8cc01ac80 comp rx=0 tx=0).stop 2023-05-22T15:27:35.761 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.751+0000 7fb8e8d59640 1 -- 172.21.15.16:0/3144976859 shutdown_connections 2023-05-22T15:27:35.761 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.751+0000 7fb8e8d59640 1 --2- 172.21.15.16:0/3144976859 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fb8c4065380 0x7fb8c4067830 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:35.761 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.751+0000 7fb8e8d59640 1 --2- 172.21.15.16:0/3144976859 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb8e415bcd0 0x7fb8e41597e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:35.761 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.751+0000 7fb8e8d59640 1 --2- 172.21.15.16:0/3144976859 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb8e4158210 0x7fb8e4146fe0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:35.761 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.751+0000 7fb8e8d59640 1 --2- 172.21.15.16:0/3144976859 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb8e40c8660 0x7fb8e4150040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:35.762 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.751+0000 7fb8e8d59640 1 -- 172.21.15.16:0/3144976859 >> 172.21.15.16:0/3144976859 conn(0x7fb8e4069720 msgr2=0x7fb8e40cb2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:35.762 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.751+0000 7fb8e8d59640 1 -- 172.21.15.16:0/3144976859 shutdown_connections 2023-05-22T15:27:35.762 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.751+0000 7fb8e8d59640 1 -- 172.21.15.16:0/3144976859 wait complete. 2023-05-22T15:27:35.777 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-05-22T15:27:35.900 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.891+0000 7f7931267640 1 Processor -- start 2023-05-22T15:27:35.901 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.891+0000 7f7931267640 1 -- start start 2023-05-22T15:27:35.901 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.895+0000 7f7931267640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f792c06b6c0 0x7f792c06bb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:35.902 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.895+0000 7f7931267640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f792c070c30 con 0x7f792c159970 2023-05-22T15:27:35.902 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.895+0000 7f7931267640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f792c06c040 con 0x7f792c159d40 2023-05-22T15:27:35.902 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.895+0000 7f7931267640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f792c15dd90 con 0x7f792c06b6c0 2023-05-22T15:27:35.902 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.895+0000 7f792b7fe640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f792c06b6c0 0x7f792c06bb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:35.902 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.895+0000 7f792affd640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f792c159970 0x7f792c15d680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:36730/0 (socket says 172.21.15.16:36730) 2023-05-22T15:27:35.902 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.895+0000 7f792affd640 1 -- 172.21.15.16:0/3751198220 learned_addr learned my addr 172.21.15.16:0/3751198220 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:27:35.903 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.895+0000 7f792b7fe640 1 -- 172.21.15.16:0/3751198220 >> v1:172.21.15.62:6789/0 conn(0x7f792c159d40 legacy=0x7f792c0cb260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:35.903 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.895+0000 7f792b7fe640 1 -- 172.21.15.16:0/3751198220 >> v1:172.21.15.16:6789/0 conn(0x7f792c159970 legacy=0x7f792c15d680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:35.903 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.895+0000 7f792b7fe640 1 -- 172.21.15.16:0/3751198220 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f792c070e60 con 0x7f792c06b6c0 2023-05-22T15:27:35.904 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.895+0000 7f792b7fe640 1 --2- 172.21.15.16:0/3751198220 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f792c06b6c0 0x7f792c06bb00 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f791c0097c0 tx=0x7f791c01c3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1b689b2a73cd82a server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:35.905 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.895+0000 7f7929ffb640 1 -- 172.21.15.16:0/3751198220 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f791c0ab600 con 0x7f792c06b6c0 2023-05-22T15:27:35.905 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.895+0000 7f7929ffb640 1 -- 172.21.15.16:0/3751198220 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f791c0a7050 con 0x7f792c06b6c0 2023-05-22T15:27:35.905 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.895+0000 7f7929ffb640 1 -- 172.21.15.16:0/3751198220 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f791c0ab980 con 0x7f792c06b6c0 2023-05-22T15:27:35.906 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.899+0000 7f7931267640 1 -- 172.21.15.16:0/3751198220 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f792c06b6c0 msgr2=0x7f792c06bb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:35.906 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.899+0000 7f7931267640 1 --2- 172.21.15.16:0/3751198220 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f792c06b6c0 0x7f792c06bb00 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f791c0097c0 tx=0x7f791c01c3b0 comp rx=0 tx=0).stop 2023-05-22T15:27:35.906 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.899+0000 7f7931267640 1 -- 172.21.15.16:0/3751198220 shutdown_connections 2023-05-22T15:27:35.906 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.899+0000 7f7931267640 1 --2- 172.21.15.16:0/3751198220 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f792c06b6c0 0x7f792c06bb00 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:35.907 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.899+0000 7f7931267640 1 -- 172.21.15.16:0/3751198220 >> 172.21.15.16:0/3751198220 conn(0x7f792c069720 msgr2=0x7f792c069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:35.907 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.899+0000 7f7931267640 1 -- 172.21.15.16:0/3751198220 shutdown_connections 2023-05-22T15:27:35.907 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.899+0000 7f7931267640 1 -- 172.21.15.16:0/3751198220 wait complete. 2023-05-22T15:27:35.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.899+0000 7f7931267640 1 Processor -- start 2023-05-22T15:27:35.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.899+0000 7f7931267640 1 -- start start 2023-05-22T15:27:35.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.899+0000 7f7931267640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f792c06b6c0 0x7f792c0b0e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:35.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.899+0000 7f7931267640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f792c159970 0x7f792c0ce070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:35.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.899+0000 7f7931267640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f792c159d40 0x7f792c145650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:35.911 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.899+0000 7f7931267640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f792c0ce5b0 con 0x7f792c159d40 2023-05-22T15:27:35.911 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.899+0000 7f7931267640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f792c145b90 con 0x7f792c06b6c0 2023-05-22T15:27:35.911 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.899+0000 7f7931267640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f792c0ce800 con 0x7f792c159970 2023-05-22T15:27:35.912 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.899+0000 7f792b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f792c159d40 0x7f792c145650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:35.912 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.899+0000 7f792b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f792c159d40 0x7f792c145650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57496/0 (socket says 172.21.15.16:57496) 2023-05-22T15:27:35.912 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.899+0000 7f792b7fe640 1 -- 172.21.15.16:0/3330424974 learned_addr learned my addr 172.21.15.16:0/3330424974 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:35.912 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.899+0000 7f792affd640 1 --2- 172.21.15.16:0/3330424974 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f792c06b6c0 0x7f792c0b0e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:35.912 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.899+0000 7f792a7fc640 1 --2- 172.21.15.16:0/3330424974 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f792c159970 0x7f792c0ce070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:35.912 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.899+0000 7f792b7fe640 1 -- 172.21.15.16:0/3330424974 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f792c159970 msgr2=0x7f792c0ce070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:35.913 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.899+0000 7f792b7fe640 1 --2- 172.21.15.16:0/3330424974 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f792c159970 0x7f792c0ce070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:35.913 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.899+0000 7f792b7fe640 1 -- 172.21.15.16:0/3330424974 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f792c06b6c0 msgr2=0x7f792c0b0e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:35.913 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.899+0000 7f792b7fe640 1 --2- 172.21.15.16:0/3330424974 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f792c06b6c0 0x7f792c0b0e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:35.913 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.899+0000 7f792b7fe640 1 -- 172.21.15.16:0/3330424974 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f791c009420 con 0x7f792c159d40 2023-05-22T15:27:35.913 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.899+0000 7f792b7fe640 1 --2- 172.21.15.16:0/3330424974 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f792c159d40 0x7f792c145650 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f791c0ae040 tx=0x7f791c01abf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:35.913 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.899+0000 7f79237fe640 1 -- 172.21.15.16:0/3330424974 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f791c0b7450 con 0x7f792c159d40 2023-05-22T15:27:35.913 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.899+0000 7f79237fe640 1 -- 172.21.15.16:0/3330424974 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f791c0b75b0 con 0x7f792c159d40 2023-05-22T15:27:35.913 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.899+0000 7f7931267640 1 -- 172.21.15.16:0/3330424974 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f792c0cac90 con 0x7f792c159d40 2023-05-22T15:27:35.914 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.899+0000 7f79237fe640 1 -- 172.21.15.16:0/3330424974 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f791c0b7940 con 0x7f792c159d40 2023-05-22T15:27:35.914 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.899+0000 7f7931267640 1 -- 172.21.15.16:0/3330424974 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f792c071700 con 0x7f792c159d40 2023-05-22T15:27:35.914 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.903+0000 7f7931267640 1 -- 172.21.15.16:0/3330424974 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f792c149350 con 0x7f792c159d40 2023-05-22T15:27:35.915 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.903+0000 7f79237fe640 1 -- 172.21.15.16:0/3330424974 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7f791c0b7aa0 con 0x7f792c159d40 2023-05-22T15:27:35.915 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.903+0000 7f79237fe640 1 --2- 172.21.15.16:0/3330424974 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f790c065380 0x7f790c067830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:35.915 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.903+0000 7f79237fe640 1 -- 172.21.15.16:0/3330424974 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f791c0a4e40 con 0x7f792c159d40 2023-05-22T15:27:35.915 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.903+0000 7f792affd640 1 --2- 172.21.15.16:0/3330424974 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f790c065380 0x7f790c067830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:35.916 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.903+0000 7f792affd640 1 --2- 172.21.15.16:0/3330424974 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f790c065380 0x7f790c067830 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f7924005c10 tx=0x7f7924009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:35.916 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:35.907+0000 7f79237fe640 1 -- 172.21.15.16:0/3330424974 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f792c149350 con 0x7f792c159d40 2023-05-22T15:27:36.067 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.059+0000 7f7931267640 1 -- 172.21.15.16:0/3330424974 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f792c149350 con 0x7f792c159d40 2023-05-22T15:27:36.067 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.059+0000 7f79237fe640 1 -- 172.21.15.16:0/3330424974 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v29) v1 ==== 103+0+11 (secure 0 0 0) 0x7f792c149350 con 0x7f792c159d40 2023-05-22T15:27:36.067 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2023-05-22T15:27:36.069 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.063+0000 7f7931267640 1 -- 172.21.15.16:0/3330424974 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f790c065380 msgr2=0x7f790c067830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:36.070 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.063+0000 7f7931267640 1 --2- 172.21.15.16:0/3330424974 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f790c065380 0x7f790c067830 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f7924005c10 tx=0x7f7924009040 comp rx=0 tx=0).stop 2023-05-22T15:27:36.070 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.063+0000 7f7931267640 1 -- 172.21.15.16:0/3330424974 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f792c159d40 msgr2=0x7f792c145650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:36.070 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.063+0000 7f7931267640 1 --2- 172.21.15.16:0/3330424974 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f792c159d40 0x7f792c145650 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f791c0ae040 tx=0x7f791c01abf0 comp rx=0 tx=0).stop 2023-05-22T15:27:36.070 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.063+0000 7f7931267640 1 -- 172.21.15.16:0/3330424974 shutdown_connections 2023-05-22T15:27:36.070 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.063+0000 7f7931267640 1 --2- 172.21.15.16:0/3330424974 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f792c159d40 0x7f792c145650 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:36.071 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.063+0000 7f7931267640 1 --2- 172.21.15.16:0/3330424974 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f790c065380 0x7f790c067830 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:36.071 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.063+0000 7f7931267640 1 --2- 172.21.15.16:0/3330424974 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f792c159970 0x7f792c0ce070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:36.071 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.063+0000 7f7931267640 1 --2- 172.21.15.16:0/3330424974 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f792c06b6c0 0x7f792c0b0e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:36.071 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.063+0000 7f7931267640 1 -- 172.21.15.16:0/3330424974 >> 172.21.15.16:0/3330424974 conn(0x7f792c069720 msgr2=0x7f792c156f70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:36.071 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.063+0000 7f7931267640 1 -- 172.21.15.16:0/3330424974 shutdown_connections 2023-05-22T15:27:36.071 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.063+0000 7f7931267640 1 -- 172.21.15.16:0/3330424974 wait complete. 2023-05-22T15:27:36.087 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-05-22T15:27:36.208 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.199+0000 7f2e81ebd640 1 Processor -- start 2023-05-22T15:27:36.208 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.199+0000 7f2e81ebd640 1 -- start start 2023-05-22T15:27:36.211 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.199+0000 7f2e81ebd640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2e7c1531d0 0x7f2e7c0cd730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:36.211 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.199+0000 7f2e81ebd640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e7c15c6b0 con 0x7f2e7c0b0250 2023-05-22T15:27:36.211 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.199+0000 7f2e81ebd640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e7c15c990 con 0x7f2e7c0cbbf0 2023-05-22T15:27:36.211 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.199+0000 7f2e81ebd640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e7c15d330 con 0x7f2e7c1531d0 2023-05-22T15:27:36.211 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.199+0000 7f2e7bfff640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f2e7c0b0250 0x7f2e7c0acba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:36746/0 (socket says 172.21.15.16:36746) 2023-05-22T15:27:36.211 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.199+0000 7f2e7bfff640 1 -- 172.21.15.16:0/3787537001 learned_addr learned my addr 172.21.15.16:0/3787537001 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:27:36.212 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.199+0000 7f2e7b7fe640 1 --2- 172.21.15.16:0/3787537001 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2e7c1531d0 0x7f2e7c0cd730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:36.212 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.199+0000 7f2e7a7fc640 1 -- 172.21.15.16:0/3787537001 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 243151297 0 0) 0x7f2e7c15c6b0 con 0x7f2e7c0b0250 2023-05-22T15:27:36.212 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.199+0000 7f2e7a7fc640 1 -- 172.21.15.16:0/3787537001 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e68003530 con 0x7f2e7c0b0250 2023-05-22T15:27:36.212 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.203+0000 7f2e7a7fc640 1 -- 172.21.15.16:0/3787537001 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3487495910 0 0) 0x7f2e68003530 con 0x7f2e7c0b0250 2023-05-22T15:27:36.212 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.203+0000 7f2e7a7fc640 1 -- 172.21.15.16:0/3787537001 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2e7c1531d0 msgr2=0x7f2e7c0cd730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:36.212 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.203+0000 7f2e7a7fc640 1 --2- 172.21.15.16:0/3787537001 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2e7c1531d0 0x7f2e7c0cd730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:36.213 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.203+0000 7f2e7a7fc640 1 -- 172.21.15.16:0/3787537001 >> v1:172.21.15.62:6789/0 conn(0x7f2e7c0cbbf0 legacy=0x7f2e7c0afb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:36.213 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.203+0000 7f2e7a7fc640 1 -- 172.21.15.16:0/3787537001 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e7c15cc70 con 0x7f2e7c0b0250 2023-05-22T15:27:36.213 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.203+0000 7f2e7a7fc640 1 -- 172.21.15.16:0/3787537001 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7f2e6c001680 con 0x7f2e7c0b0250 2023-05-22T15:27:36.213 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.203+0000 7f2e7a7fc640 1 -- 172.21.15.16:0/3787537001 >> v1:172.21.15.16:6789/0 conn(0x7f2e7c0b0250 legacy=0x7f2e7c0acba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:36.213 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.203+0000 7f2e7a7fc640 1 --2- 172.21.15.16:0/3787537001 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e680039b0 0x7f2e68003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:36.214 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.203+0000 7f2e7a7fc640 1 --2- 172.21.15.16:0/3787537001 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2e7c1531d0 0x7f2e680047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:36.214 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.203+0000 7f2e7a7fc640 1 --2- 172.21.15.16:0/3787537001 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2e68004d10 0x7f2e68005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:36.214 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.203+0000 7f2e7a7fc640 1 -- 172.21.15.16:0/3787537001 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e7c15d330 con 0x7f2e680039b0 2023-05-22T15:27:36.214 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.203+0000 7f2e7a7fc640 1 -- 172.21.15.16:0/3787537001 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e6800ba40 con 0x7f2e68004d10 2023-05-22T15:27:36.214 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.203+0000 7f2e7a7fc640 1 -- 172.21.15.16:0/3787537001 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e6800bbd0 con 0x7f2e7c1531d0 2023-05-22T15:27:36.214 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.203+0000 7f2e7b7fe640 1 --2- 172.21.15.16:0/3787537001 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e680039b0 0x7f2e68003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:36.214 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.203+0000 7f2e7affd640 1 --2- 172.21.15.16:0/3787537001 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2e7c1531d0 0x7f2e680047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:36.215 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.203+0000 7f2e7b7fe640 1 -- 172.21.15.16:0/3787537001 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2e7c1531d0 msgr2=0x7f2e680047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:36.215 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.203+0000 7f2e7b7fe640 1 --2- 172.21.15.16:0/3787537001 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2e7c1531d0 0x7f2e680047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:36.215 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.203+0000 7f2e7bfff640 1 --2- 172.21.15.16:0/3787537001 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2e68004d10 0x7f2e68005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:36.215 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.203+0000 7f2e7b7fe640 1 -- 172.21.15.16:0/3787537001 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2e68004d10 msgr2=0x7f2e68005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:36.215 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.203+0000 7f2e7b7fe640 1 --2- 172.21.15.16:0/3787537001 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2e68004d10 0x7f2e68005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:36.215 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.203+0000 7f2e7b7fe640 1 -- 172.21.15.16:0/3787537001 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e68003530 con 0x7f2e680039b0 2023-05-22T15:27:36.215 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.203+0000 7f2e7affd640 1 --2- 172.21.15.16:0/3787537001 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2e7c1531d0 0x7f2e680047d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:27:36.219 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.203+0000 7f2e7b7fe640 1 --2- 172.21.15.16:0/3787537001 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e680039b0 0x7f2e68003dc0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f2e74009ae0 tx=0x7f2e7401ca50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93305271462a4bd5 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:36.219 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.203+0000 7f2e7a7fc640 1 -- 172.21.15.16:0/3787537001 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e740a5530 con 0x7f2e680039b0 2023-05-22T15:27:36.219 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.203+0000 7f2e7a7fc640 1 -- 172.21.15.16:0/3787537001 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e740a5690 con 0x7f2e680039b0 2023-05-22T15:27:36.219 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.203+0000 7f2e7a7fc640 1 -- 172.21.15.16:0/3787537001 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e740a5a40 con 0x7f2e680039b0 2023-05-22T15:27:36.220 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.207+0000 7f2e81ebd640 1 -- 172.21.15.16:0/3787537001 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e680039b0 msgr2=0x7f2e68003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:36.220 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.207+0000 7f2e81ebd640 1 --2- 172.21.15.16:0/3787537001 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e680039b0 0x7f2e68003dc0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f2e74009ae0 tx=0x7f2e7401ca50 comp rx=0 tx=0).stop 2023-05-22T15:27:36.220 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.207+0000 7f2e7bfff640 1 -- 172.21.15.16:0/3787537001 reap_dead start 2023-05-22T15:27:36.220 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.207+0000 7f2e81ebd640 1 -- 172.21.15.16:0/3787537001 shutdown_connections 2023-05-22T15:27:36.220 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.207+0000 7f2e81ebd640 1 -- 172.21.15.16:0/3787537001 >> 172.21.15.16:0/3787537001 conn(0x7f2e7c069720 msgr2=0x7f2e7c069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:36.220 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.207+0000 7f2e81ebd640 1 -- 172.21.15.16:0/3787537001 shutdown_connections 2023-05-22T15:27:36.221 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.207+0000 7f2e81ebd640 1 -- 172.21.15.16:0/3787537001 wait complete. 2023-05-22T15:27:36.221 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.207+0000 7f2e81ebd640 1 Processor -- start 2023-05-22T15:27:36.221 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.207+0000 7f2e81ebd640 1 -- start start 2023-05-22T15:27:36.221 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.207+0000 7f2e81ebd640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2e680039b0 0x7f2e7c1516e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:36.221 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.207+0000 7f2e81ebd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e7c0aee10 0x7f2e7c0c5720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:36.221 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.207+0000 7f2e81ebd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2e7c151e10 0x7f2e7c166ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:36.221 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.207+0000 7f2e81ebd640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e7c0af660 con 0x7f2e7c0aee10 2023-05-22T15:27:36.222 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.207+0000 7f2e81ebd640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e7c154760 con 0x7f2e7c151e10 2023-05-22T15:27:36.222 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.207+0000 7f2e81ebd640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e7c151c20 con 0x7f2e680039b0 2023-05-22T15:27:36.222 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.207+0000 7f2e7affd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e7c0aee10 0x7f2e7c0c5720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:36.222 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.207+0000 7f2e7affd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e7c0aee10 0x7f2e7c0c5720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57510/0 (socket says 172.21.15.16:57510) 2023-05-22T15:27:36.222 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.207+0000 7f2e7affd640 1 -- 172.21.15.16:0/880815481 learned_addr learned my addr 172.21.15.16:0/880815481 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:36.222 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.207+0000 7f2e7b7fe640 1 --2- 172.21.15.16:0/880815481 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2e680039b0 0x7f2e7c1516e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:36.222 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.207+0000 7f2e7bfff640 1 --2- 172.21.15.16:0/880815481 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2e7c151e10 0x7f2e7c166ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:36.223 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.207+0000 7f2e7affd640 1 -- 172.21.15.16:0/880815481 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2e680039b0 msgr2=0x7f2e7c1516e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:36.223 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.207+0000 7f2e7affd640 1 --2- 172.21.15.16:0/880815481 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2e680039b0 0x7f2e7c1516e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:36.223 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.207+0000 7f2e7affd640 1 -- 172.21.15.16:0/880815481 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2e7c151e10 msgr2=0x7f2e7c166ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:36.223 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.207+0000 7f2e7affd640 1 --2- 172.21.15.16:0/880815481 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2e7c151e10 0x7f2e7c166ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:36.223 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.207+0000 7f2e7affd640 1 -- 172.21.15.16:0/880815481 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e7c167e30 con 0x7f2e7c0aee10 2023-05-22T15:27:36.223 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.207+0000 7f2e7b7fe640 1 --2- 172.21.15.16:0/880815481 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2e680039b0 0x7f2e7c1516e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:27:36.224 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.207+0000 7f2e7bfff640 1 --2- 172.21.15.16:0/880815481 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2e7c151e10 0x7f2e7c166ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:27:36.224 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.207+0000 7f2e7affd640 1 --2- 172.21.15.16:0/880815481 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e7c0aee10 0x7f2e7c0c5720 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f2e6400db70 tx=0x7f2e6400b4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:36.224 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.207+0000 7f2e73fff640 1 -- 172.21.15.16:0/880815481 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e64017070 con 0x7f2e7c0aee10 2023-05-22T15:27:36.224 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.207+0000 7f2e73fff640 1 -- 172.21.15.16:0/880815481 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e6400de70 con 0x7f2e7c0aee10 2023-05-22T15:27:36.225 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.207+0000 7f2e81ebd640 1 -- 172.21.15.16:0/880815481 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e7c0cdb40 con 0x7f2e7c0aee10 2023-05-22T15:27:36.225 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.207+0000 7f2e73fff640 1 -- 172.21.15.16:0/880815481 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e64006d80 con 0x7f2e7c0aee10 2023-05-22T15:27:36.225 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.207+0000 7f2e81ebd640 1 -- 172.21.15.16:0/880815481 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e7c167fc0 con 0x7f2e7c0aee10 2023-05-22T15:27:36.225 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.207+0000 7f2e81ebd640 1 -- 172.21.15.16:0/880815481 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e7c153fb0 con 0x7f2e7c0aee10 2023-05-22T15:27:36.226 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.211+0000 7f2e73fff640 1 -- 172.21.15.16:0/880815481 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7f2e6400bb50 con 0x7f2e7c0aee10 2023-05-22T15:27:36.226 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.211+0000 7f2e73fff640 1 --2- 172.21.15.16:0/880815481 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f2e5c065380 0x7f2e5c067830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:36.226 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.211+0000 7f2e73fff640 1 -- 172.21.15.16:0/880815481 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f2e640827a0 con 0x7f2e7c0aee10 2023-05-22T15:27:36.227 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.211+0000 7f2e7b7fe640 1 --2- 172.21.15.16:0/880815481 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f2e5c065380 0x7f2e5c067830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:36.227 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.211+0000 7f2e7b7fe640 1 --2- 172.21.15.16:0/880815481 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f2e5c065380 0x7f2e5c067830 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f2e740af000 tx=0x7f2e740046d0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:36.228 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.215+0000 7f2e73fff640 1 -- 172.21.15.16:0/880815481 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f2e7c153fb0 con 0x7f2e7c0aee10 2023-05-22T15:27:36.376 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.367+0000 7f2e81ebd640 1 -- 172.21.15.16:0/880815481 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2e7c153fb0 con 0x7f2e7c0aee10 2023-05-22T15:27:36.378 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.371+0000 7f2e73fff640 1 -- 172.21.15.16:0/880815481 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v29) v1 ==== 74+0+19700 (secure 0 0 0) 0x7f2e7c153fb0 con 0x7f2e7c0aee10 2023-05-22T15:27:36.378 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:27:36.379 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":29,"fsid":"b2719834-7847-4d98-90a6-f65d57f85c49","created":"2023-05-22T15:26:25.259314+0000","modified":"2023-05-22T15:27:31.182317+0000","last_up_change":"2023-05-22T15:26:36.448524+0000","last_in_change":"2023-05-22T15:26:27.675668+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-05-22T15:26:38.019781+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-05-22T15:26:43.570514+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-05-22T15:26:47.879159+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-05-22T15:26:48.522130+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data_ec","create_time":"2023-05-22T15:26:51.616142+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"9e7ff854-b25f-4cbb-a5c2-1799d9c53d2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":37021},{"type":"v1","addr":"172.21.15.16:6825","nonce":37021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":37021},{"type":"v1","addr":"172.21.15.16:6827","nonce":37021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":37021},{"type":"v1","addr":"172.21.15.16:6831","nonce":37021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":37021},{"type":"v1","addr":"172.21.15.16:6829","nonce":37021}]},"public_addr":"172.21.15.16:6825/37021","cluster_addr":"172.21.15.16:6827/37021","heartbeat_back_addr":"172.21.15.16:6831/37021","heartbeat_front_addr":"172.21.15.16:6829/37021","state":["exists","up"]},{"osd":1,"uuid":"4702fbc9-a927-4565-aca5-41dc69cc8264","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":37008},{"type":"v1","addr":"172.21.15.16:6801","nonce":37008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":37008},{"type":"v1","addr":"172.21.15.16:6803","nonce":37008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":37008},{"type":"v1","addr":"172.21.15.16:6807","nonce":37008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":37008},{"type":"v1","addr":"172.21.15.16:6805","nonce":37008}]},"public_addr":"172.21.15.16:6801/37008","cluster_addr":"172.21.15.16:6803/37008","heartbeat_back_addr":"172.21.15.16:6807/37008","heartbeat_front_addr":"172.21.15.16:6805/37008","state":["exists","up"]},{"osd":2,"uuid":"1e7bba46-4eba-47ef-bb81-5e2d371638ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":37009},{"type":"v1","addr":"172.21.15.16:6809","nonce":37009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":37009},{"type":"v1","addr":"172.21.15.16:6811","nonce":37009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":37009},{"type":"v1","addr":"172.21.15.16:6815","nonce":37009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":37009},{"type":"v1","addr":"172.21.15.16:6813","nonce":37009}]},"public_addr":"172.21.15.16:6809/37009","cluster_addr":"172.21.15.16:6811/37009","heartbeat_back_addr":"172.21.15.16:6815/37009","heartbeat_front_addr":"172.21.15.16:6813/37009","state":["exists","up"]},{"osd":3,"uuid":"383cacc0-5a88-4904-990e-498ca59ef48a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":37020},{"type":"v1","addr":"172.21.15.16:6817","nonce":37020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":37020},{"type":"v1","addr":"172.21.15.16:6819","nonce":37020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":37020},{"type":"v1","addr":"172.21.15.16:6823","nonce":37020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":37020},{"type":"v1","addr":"172.21.15.16:6821","nonce":37020}]},"public_addr":"172.21.15.16:6817/37020","cluster_addr":"172.21.15.16:6819/37020","heartbeat_back_addr":"172.21.15.16:6823/37020","heartbeat_front_addr":"172.21.15.16:6821/37020","state":["exists","up"]},{"osd":4,"uuid":"970500ef-95b7-42c1-b080-eed4fae7013d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":37353},{"type":"v1","addr":"172.21.15.62:6817","nonce":37353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6819","nonce":37353},{"type":"v1","addr":"172.21.15.62:6822","nonce":37353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37353},{"type":"v1","addr":"172.21.15.62:6831","nonce":37353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37353},{"type":"v1","addr":"172.21.15.62:6826","nonce":37353}]},"public_addr":"172.21.15.62:6817/37353","cluster_addr":"172.21.15.62:6822/37353","heartbeat_back_addr":"172.21.15.62:6831/37353","heartbeat_front_addr":"172.21.15.62:6826/37353","state":["exists","up"]},{"osd":5,"uuid":"022d67db-d4d8-445b-b1b9-8b446a00f5c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":37352},{"type":"v1","addr":"172.21.15.62:6801","nonce":37352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":37352},{"type":"v1","addr":"172.21.15.62:6803","nonce":37352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":37352},{"type":"v1","addr":"172.21.15.62:6807","nonce":37352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":37352},{"type":"v1","addr":"172.21.15.62:6805","nonce":37352}]},"public_addr":"172.21.15.62:6801/37352","cluster_addr":"172.21.15.62:6803/37352","heartbeat_back_addr":"172.21.15.62:6807/37352","heartbeat_front_addr":"172.21.15.62:6805/37352","state":["exists","up"]},{"osd":6,"uuid":"42bd372b-6688-4a1b-8407-e16e8c266e33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":37354},{"type":"v1","addr":"172.21.15.62:6820","nonce":37354}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":37354},{"type":"v1","addr":"172.21.15.62:6823","nonce":37354}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6829","nonce":37354},{"type":"v1","addr":"172.21.15.62:6830","nonce":37354}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6825","nonce":37354},{"type":"v1","addr":"172.21.15.62:6827","nonce":37354}]},"public_addr":"172.21.15.62:6820/37354","cluster_addr":"172.21.15.62:6823/37354","heartbeat_back_addr":"172.21.15.62:6830/37354","heartbeat_front_addr":"172.21.15.62:6827/37354","state":["exists","up"]},{"osd":7,"uuid":"ac70b2a3-f41b-4bf2-b227-7f2cbb55426d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37355},{"type":"v1","addr":"172.21.15.62:6809","nonce":37355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37355},{"type":"v1","addr":"172.21.15.62:6811","nonce":37355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":37355},{"type":"v1","addr":"172.21.15.62:6815","nonce":37355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":37355},{"type":"v1","addr":"172.21.15.62:6813","nonce":37355}]},"public_addr":"172.21.15.62:6809/37355","cluster_addr":"172.21.15.62:6811/37355","heartbeat_back_addr":"172.21.15.62:6815/37355","heartbeat_front_addr":"172.21.15.62:6813/37355","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.368896+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.107273+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.276888+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.287502+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.124791+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.251937+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.255679+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.232797+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"4.23","mappings":[{"from":3,"to":2}]},{"pgid":"4.31","mappings":[{"from":1,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-05-22T15:27:36.380 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.371+0000 7f2e81ebd640 1 -- 172.21.15.16:0/880815481 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f2e5c065380 msgr2=0x7f2e5c067830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:36.380 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.371+0000 7f2e81ebd640 1 --2- 172.21.15.16:0/880815481 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f2e5c065380 0x7f2e5c067830 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f2e740af000 tx=0x7f2e740046d0 comp rx=0 tx=0).stop 2023-05-22T15:27:36.380 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.371+0000 7f2e81ebd640 1 -- 172.21.15.16:0/880815481 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e7c0aee10 msgr2=0x7f2e7c0c5720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:36.380 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.371+0000 7f2e81ebd640 1 --2- 172.21.15.16:0/880815481 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e7c0aee10 0x7f2e7c0c5720 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f2e6400db70 tx=0x7f2e6400b4a0 comp rx=0 tx=0).stop 2023-05-22T15:27:36.381 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.371+0000 7f2e81ebd640 1 -- 172.21.15.16:0/880815481 shutdown_connections 2023-05-22T15:27:36.381 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.371+0000 7f2e81ebd640 1 --2- 172.21.15.16:0/880815481 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2e7c151e10 0x7f2e7c166ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:36.381 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.371+0000 7f2e81ebd640 1 --2- 172.21.15.16:0/880815481 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e7c0aee10 0x7f2e7c0c5720 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:36.381 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.371+0000 7f2e81ebd640 1 --2- 172.21.15.16:0/880815481 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f2e5c065380 0x7f2e5c067830 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:36.381 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.371+0000 7f2e81ebd640 1 --2- 172.21.15.16:0/880815481 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2e680039b0 0x7f2e7c1516e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:36.381 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.371+0000 7f2e81ebd640 1 -- 172.21.15.16:0/880815481 >> 172.21.15.16:0/880815481 conn(0x7f2e7c069720 msgr2=0x7f2e7c15b590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:36.381 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.371+0000 7f2e81ebd640 1 -- 172.21.15.16:0/880815481 shutdown_connections 2023-05-22T15:27:36.381 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.371+0000 7f2e81ebd640 1 -- 172.21.15.16:0/880815481 wait complete. 2023-05-22T15:27:36.396 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-05-22T15:26:38.019781+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-05-22T15:26:43.570514+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2023-05-22T15:26:47.879159+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2023-05-22T15:26:48.522130+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-05-22T15:26:51.616142+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '27', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-05-22T15:27:36.397 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-05-22T15:27:36.514 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.507+0000 7f42174ac640 1 Processor -- start 2023-05-22T15:27:36.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.507+0000 7f42174ac640 1 -- start start 2023-05-22T15:27:36.516 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.507+0000 7f42174ac640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f4210159d40 0x7f42100cb260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:36.516 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.507+0000 7f42174ac640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f421006b6c0 0x7f421006bb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:36.516 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.507+0000 7f42174ac640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4210070c30 con 0x7f4210159970 2023-05-22T15:27:36.516 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.507+0000 7f42174ac640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f421015dd90 con 0x7f4210159d40 2023-05-22T15:27:36.516 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.507+0000 7f42174ac640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f421006c040 con 0x7f421006b6c0 2023-05-22T15:27:36.517 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.507+0000 7f4214a47640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f4210159d40 0x7f42100cb260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:36.517 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.507+0000 7f4214a47640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f4210159d40 0x7f42100cb260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57520/0 (socket says 172.21.15.16:57520) 2023-05-22T15:27:36.517 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.507+0000 7f4215248640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f4210159970 0x7f421015d680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:36752/0 (socket says 172.21.15.16:36752) 2023-05-22T15:27:36.517 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.507+0000 7f4214a47640 1 -- 172.21.15.16:0/3660666634 learned_addr learned my addr 172.21.15.16:0/3660666634 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:36.517 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.507+0000 7f4215a49640 1 --2- 172.21.15.16:0/3660666634 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f421006b6c0 0x7f421006bb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:36.517 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.507+0000 7f4214a47640 1 -- 172.21.15.16:0/3660666634 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f421006b6c0 msgr2=0x7f421006bb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:36.517 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.507+0000 7f4214a47640 1 --2- 172.21.15.16:0/3660666634 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f421006b6c0 0x7f421006bb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:36.517 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.507+0000 7f4214a47640 1 -- 172.21.15.16:0/3660666634 >> v1:172.21.15.16:6789/0 conn(0x7f4210159970 legacy=0x7f421015d680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:36.518 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.507+0000 7f4207fff640 1 -- 172.21.15.16:0/3660666634 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2829600586 0 0) 0x7f4210070c30 con 0x7f4210159970 2023-05-22T15:27:36.518 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.507+0000 7f4214a47640 1 -- 172.21.15.16:0/3660666634 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4210070e60 con 0x7f4210159d40 2023-05-22T15:27:36.518 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.507+0000 7f4215a49640 1 --2- 172.21.15.16:0/3660666634 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f421006b6c0 0x7f421006bb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:27:36.519 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.511+0000 7f4214a47640 1 --2- 172.21.15.16:0/3660666634 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4210159d40 0x7f42100cb260 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f4200009870 tx=0x7f420001c4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5a7a3a02b33e9f4 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:36.519 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.511+0000 7f4207fff640 1 -- 172.21.15.16:0/3660666634 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42000ac600 con 0x7f4210159d40 2023-05-22T15:27:36.519 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.511+0000 7f4207fff640 1 -- 172.21.15.16:0/3660666634 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42000a8050 con 0x7f4210159d40 2023-05-22T15:27:36.519 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.511+0000 7f4207fff640 1 -- 172.21.15.16:0/3660666634 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42000ac980 con 0x7f4210159d40 2023-05-22T15:27:36.519 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.511+0000 7f42174ac640 1 -- 172.21.15.16:0/3660666634 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4210159d40 msgr2=0x7f42100cb260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:36.519 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.511+0000 7f42174ac640 1 --2- 172.21.15.16:0/3660666634 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4210159d40 0x7f42100cb260 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f4200009870 tx=0x7f420001c4b0 comp rx=0 tx=0).stop 2023-05-22T15:27:36.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.511+0000 7f42174ac640 1 -- 172.21.15.16:0/3660666634 shutdown_connections 2023-05-22T15:27:36.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.511+0000 7f42174ac640 1 --2- 172.21.15.16:0/3660666634 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f421006b6c0 0x7f421006bb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:36.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.511+0000 7f42174ac640 1 --2- 172.21.15.16:0/3660666634 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4210159d40 0x7f42100cb260 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:36.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.511+0000 7f42174ac640 1 -- 172.21.15.16:0/3660666634 >> 172.21.15.16:0/3660666634 conn(0x7f4210069720 msgr2=0x7f4210069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:36.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.511+0000 7f42174ac640 1 -- 172.21.15.16:0/3660666634 shutdown_connections 2023-05-22T15:27:36.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.511+0000 7f42174ac640 1 -- 172.21.15.16:0/3660666634 wait complete. 2023-05-22T15:27:36.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.511+0000 7f42174ac640 1 Processor -- start 2023-05-22T15:27:36.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.511+0000 7f42174ac640 1 -- start start 2023-05-22T15:27:36.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.511+0000 7f42174ac640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f421006b6c0 0x7f42100b3560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:36.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.511+0000 7f42174ac640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4210159970 0x7f42100b23d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:36.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.511+0000 7f42174ac640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f42100b2910 0x7f42100af100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:36.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.515+0000 7f42174ac640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42100b1220 con 0x7f4210159970 2023-05-22T15:27:36.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.515+0000 7f42174ac640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42100b1610 con 0x7f42100b2910 2023-05-22T15:27:36.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.515+0000 7f42174ac640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42100b18f0 con 0x7f421006b6c0 2023-05-22T15:27:36.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.515+0000 7f4214a47640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4210159970 0x7f42100b23d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:36.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.515+0000 7f4214a47640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4210159970 0x7f42100b23d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57524/0 (socket says 172.21.15.16:57524) 2023-05-22T15:27:36.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.515+0000 7f4214a47640 1 -- 172.21.15.16:0/2979279660 learned_addr learned my addr 172.21.15.16:0/2979279660 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:36.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.515+0000 7f4215248640 1 --2- 172.21.15.16:0/2979279660 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f421006b6c0 0x7f42100b3560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:36.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.515+0000 7f4215a49640 1 --2- 172.21.15.16:0/2979279660 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f42100b2910 0x7f42100af100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:36.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.515+0000 7f4214a47640 1 -- 172.21.15.16:0/2979279660 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f421006b6c0 msgr2=0x7f42100b3560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:36.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.515+0000 7f4214a47640 1 --2- 172.21.15.16:0/2979279660 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f421006b6c0 0x7f42100b3560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:36.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.515+0000 7f4214a47640 1 -- 172.21.15.16:0/2979279660 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f42100b2910 msgr2=0x7f42100af100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:36.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.515+0000 7f4214a47640 1 --2- 172.21.15.16:0/2979279660 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f42100b2910 0x7f42100af100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:36.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.515+0000 7f4214a47640 1 -- 172.21.15.16:0/2979279660 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42000094d0 con 0x7f4210159970 2023-05-22T15:27:36.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.515+0000 7f4214a47640 1 --2- 172.21.15.16:0/2979279660 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4210159970 0x7f42100b23d0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f42000af040 tx=0x7f420001acf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:36.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.515+0000 7f4205ffb640 1 -- 172.21.15.16:0/2979279660 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4200004230 con 0x7f4210159970 2023-05-22T15:27:36.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.515+0000 7f4205ffb640 1 -- 172.21.15.16:0/2979279660 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4200004390 con 0x7f4210159970 2023-05-22T15:27:36.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.515+0000 7f4205ffb640 1 -- 172.21.15.16:0/2979279660 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4200003820 con 0x7f4210159970 2023-05-22T15:27:36.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.515+0000 7f42174ac640 1 -- 172.21.15.16:0/2979279660 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42100cb7a0 con 0x7f4210159970 2023-05-22T15:27:36.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.515+0000 7f42174ac640 1 -- 172.21.15.16:0/2979279660 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4210071700 con 0x7f4210159970 2023-05-22T15:27:36.526 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.515+0000 7f42174ac640 1 -- 172.21.15.16:0/2979279660 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42100cb930 con 0x7f4210159970 2023-05-22T15:27:36.526 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.515+0000 7f4205ffb640 1 -- 172.21.15.16:0/2979279660 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7f42000b3070 con 0x7f4210159970 2023-05-22T15:27:36.526 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.515+0000 7f4205ffb640 1 --2- 172.21.15.16:0/2979279660 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f41f40653b0 0x7f41f4067860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:36.526 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.515+0000 7f4205ffb640 1 -- 172.21.15.16:0/2979279660 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f420011d510 con 0x7f4210159970 2023-05-22T15:27:36.526 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.515+0000 7f4215248640 1 --2- 172.21.15.16:0/2979279660 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f41f40653b0 0x7f41f4067860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:36.526 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.515+0000 7f4215248640 1 --2- 172.21.15.16:0/2979279660 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f41f40653b0 0x7f41f4067860 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f420c000f70 tx=0x7f420c007c10 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:36.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.519+0000 7f4205ffb640 1 -- 172.21.15.16:0/2979279660 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f42000eae90 con 0x7f4210159970 2023-05-22T15:27:36.678 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.671+0000 7f42174ac640 1 -- 172.21.15.16:0/2979279660 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f421004dbe0 con 0x7f4210159970 2023-05-22T15:27:36.678 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.671+0000 7f4205ffb640 1 -- 172.21.15.16:0/2979279660 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v29) v1 ==== 93+0+10 (secure 0 0 0) 0x7f42000ee4b0 con 0x7f4210159970 2023-05-22T15:27:36.679 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2023-05-22T15:27:36.681 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.671+0000 7f42174ac640 1 -- 172.21.15.16:0/2979279660 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f41f40653b0 msgr2=0x7f41f4067860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:36.681 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.671+0000 7f42174ac640 1 --2- 172.21.15.16:0/2979279660 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f41f40653b0 0x7f41f4067860 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f420c000f70 tx=0x7f420c007c10 comp rx=0 tx=0).stop 2023-05-22T15:27:36.681 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.671+0000 7f42174ac640 1 -- 172.21.15.16:0/2979279660 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4210159970 msgr2=0x7f42100b23d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:36.681 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.671+0000 7f42174ac640 1 --2- 172.21.15.16:0/2979279660 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4210159970 0x7f42100b23d0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f42000af040 tx=0x7f420001acf0 comp rx=0 tx=0).stop 2023-05-22T15:27:36.681 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.671+0000 7f42174ac640 1 -- 172.21.15.16:0/2979279660 shutdown_connections 2023-05-22T15:27:36.681 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.671+0000 7f42174ac640 1 --2- 172.21.15.16:0/2979279660 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f42100b2910 0x7f42100af100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:36.681 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.671+0000 7f42174ac640 1 --2- 172.21.15.16:0/2979279660 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4210159970 0x7f42100b23d0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:36.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.671+0000 7f42174ac640 1 --2- 172.21.15.16:0/2979279660 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f41f40653b0 0x7f41f4067860 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:36.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.671+0000 7f42174ac640 1 --2- 172.21.15.16:0/2979279660 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f421006b6c0 0x7f42100b3560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:36.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.671+0000 7f42174ac640 1 -- 172.21.15.16:0/2979279660 >> 172.21.15.16:0/2979279660 conn(0x7f4210069720 msgr2=0x7f421015adc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:36.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.671+0000 7f42174ac640 1 -- 172.21.15.16:0/2979279660 shutdown_connections 2023-05-22T15:27:36.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.671+0000 7f42174ac640 1 -- 172.21.15.16:0/2979279660 wait complete. 2023-05-22T15:27:36.696 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-05-22T15:27:36.822 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.815+0000 7f2e34549640 1 Processor -- start 2023-05-22T15:27:36.822 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.815+0000 7f2e34549640 1 -- start start 2023-05-22T15:27:36.823 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.815+0000 7f2e34549640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2e2c15bee0 0x7f2e2c06b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:36.823 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.815+0000 7f2e34549640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f2e2c0cb730 0x7f2e2c06bc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:36.823 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.815+0000 7f2e34549640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2e2c0cbbe0 0x7f2e2c15f850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:36.823 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.815+0000 7f2e34549640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e2c0c8660 con 0x7f2e2c15bee0 2023-05-22T15:27:36.823 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.815+0000 7f2e34549640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e2c0ce4a0 con 0x7f2e2c0cb730 2023-05-22T15:27:36.823 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.815+0000 7f2e34549640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e2c0ce780 con 0x7f2e2c0cbbe0 2023-05-22T15:27:36.824 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.815+0000 7f2e322e5640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2e2c15bee0 0x7f2e2c06b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:36.824 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.815+0000 7f2e322e5640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2e2c15bee0 0x7f2e2c06b6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57540/0 (socket says 172.21.15.16:57540) 2023-05-22T15:27:36.824 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.815+0000 7f2e322e5640 1 -- 172.21.15.16:0/1651201047 learned_addr learned my addr 172.21.15.16:0/1651201047 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:36.824 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.815+0000 7f2e32ae6640 1 --2- 172.21.15.16:0/1651201047 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2e2c0cbbe0 0x7f2e2c15f850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:36.825 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.815+0000 7f2e31ae4640 1 --2- 172.21.15.16:0/1651201047 >> v2:172.21.15.62:3300/0 conn(0x7f2e2c0cb730 0x7f2e2c06bc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:36.825 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.815+0000 7f2e322e5640 1 -- 172.21.15.16:0/1651201047 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2e2c0cbbe0 msgr2=0x7f2e2c15f850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:36.825 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.815+0000 7f2e322e5640 1 --2- 172.21.15.16:0/1651201047 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2e2c0cbbe0 0x7f2e2c15f850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:36.825 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.815+0000 7f2e322e5640 1 -- 172.21.15.16:0/1651201047 >> v2:172.21.15.62:3300/0 conn(0x7f2e2c0cb730 msgr2=0x7f2e2c06bc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:36.825 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.815+0000 7f2e322e5640 1 --2- 172.21.15.16:0/1651201047 >> v2:172.21.15.62:3300/0 conn(0x7f2e2c0cb730 0x7f2e2c06bc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:36.825 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.815+0000 7f2e322e5640 1 -- 172.21.15.16:0/1651201047 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e2c0cea60 con 0x7f2e2c15bee0 2023-05-22T15:27:36.826 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.819+0000 7f2e322e5640 1 --2- 172.21.15.16:0/1651201047 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e2c15bee0 0x7f2e2c06b6c0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f2e2c0c1720 tx=0x7f2e2801c210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2b6549996025916 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:36.827 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.819+0000 7f2e312e3640 1 -- 172.21.15.16:0/1651201047 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e280ab600 con 0x7f2e2c15bee0 2023-05-22T15:27:36.827 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.819+0000 7f2e312e3640 1 -- 172.21.15.16:0/1651201047 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e280a7050 con 0x7f2e2c15bee0 2023-05-22T15:27:36.827 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.819+0000 7f2e312e3640 1 -- 172.21.15.16:0/1651201047 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e280ab980 con 0x7f2e2c15bee0 2023-05-22T15:27:36.827 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.819+0000 7f2e34549640 1 -- 172.21.15.16:0/1651201047 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e2c15bee0 msgr2=0x7f2e2c06b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:36.827 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.819+0000 7f2e34549640 1 --2- 172.21.15.16:0/1651201047 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e2c15bee0 0x7f2e2c06b6c0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f2e2c0c1720 tx=0x7f2e2801c210 comp rx=0 tx=0).stop 2023-05-22T15:27:36.827 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.819+0000 7f2e34549640 1 -- 172.21.15.16:0/1651201047 shutdown_connections 2023-05-22T15:27:36.828 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.819+0000 7f2e34549640 1 --2- 172.21.15.16:0/1651201047 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2e2c0cbbe0 0x7f2e2c15f850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:36.828 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.819+0000 7f2e34549640 1 --2- 172.21.15.16:0/1651201047 >> v2:172.21.15.62:3300/0 conn(0x7f2e2c0cb730 0x7f2e2c06bc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:36.828 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.819+0000 7f2e34549640 1 --2- 172.21.15.16:0/1651201047 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e2c15bee0 0x7f2e2c06b6c0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:36.828 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.819+0000 7f2e34549640 1 -- 172.21.15.16:0/1651201047 >> 172.21.15.16:0/1651201047 conn(0x7f2e2c069720 msgr2=0x7f2e2c069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:36.828 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.819+0000 7f2e34549640 1 -- 172.21.15.16:0/1651201047 shutdown_connections 2023-05-22T15:27:36.828 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.819+0000 7f2e34549640 1 -- 172.21.15.16:0/1651201047 wait complete. 2023-05-22T15:27:36.829 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.819+0000 7f2e34549640 1 Processor -- start 2023-05-22T15:27:36.829 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.819+0000 7f2e34549640 1 -- start start 2023-05-22T15:27:36.829 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.819+0000 7f2e34549640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2e2c0cb730 0x7f2e2c15c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:36.829 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.823+0000 7f2e34549640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e2c0cbbe0 0x7f2e2c15d560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:36.830 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.823+0000 7f2e34549640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2e2c15bee0 0x7f2e2c0ae570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:36.830 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.823+0000 7f2e34549640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e2c0b3c30 con 0x7f2e2c0cbbe0 2023-05-22T15:27:36.830 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.823+0000 7f2e34549640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e2c15c990 con 0x7f2e2c15bee0 2023-05-22T15:27:36.830 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.823+0000 7f2e34549640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e2c15cc70 con 0x7f2e2c0cb730 2023-05-22T15:27:36.831 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.823+0000 7f2e31ae4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e2c0cbbe0 0x7f2e2c15d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:36.832 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.823+0000 7f2e31ae4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e2c0cbbe0 0x7f2e2c15d560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57550/0 (socket says 172.21.15.16:57550) 2023-05-22T15:27:36.832 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.823+0000 7f2e31ae4640 1 -- 172.21.15.16:0/3325886734 learned_addr learned my addr 172.21.15.16:0/3325886734 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:36.832 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.823+0000 7f2e322e5640 1 --2- 172.21.15.16:0/3325886734 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2e2c0cb730 0x7f2e2c15c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:36.832 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.823+0000 7f2e32ae6640 1 --2- 172.21.15.16:0/3325886734 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2e2c15bee0 0x7f2e2c0ae570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:36.832 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.823+0000 7f2e31ae4640 1 -- 172.21.15.16:0/3325886734 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2e2c0cb730 msgr2=0x7f2e2c15c2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:36.832 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.823+0000 7f2e31ae4640 1 --2- 172.21.15.16:0/3325886734 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2e2c0cb730 0x7f2e2c15c2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:36.832 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.823+0000 7f2e31ae4640 1 -- 172.21.15.16:0/3325886734 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2e2c15bee0 msgr2=0x7f2e2c0ae570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:36.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.823+0000 7f2e31ae4640 1 --2- 172.21.15.16:0/3325886734 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2e2c15bee0 0x7f2e2c0ae570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:36.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.823+0000 7f2e31ae4640 1 -- 172.21.15.16:0/3325886734 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e280094d0 con 0x7f2e2c0cbbe0 2023-05-22T15:27:36.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.823+0000 7f2e31ae4640 1 --2- 172.21.15.16:0/3325886734 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e2c0cbbe0 0x7f2e2c15d560 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f2e1c00d800 tx=0x7f2e1c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:36.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.823+0000 7f2e22ffd640 1 -- 172.21.15.16:0/3325886734 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e1c0096a0 con 0x7f2e2c0cbbe0 2023-05-22T15:27:36.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.823+0000 7f2e22ffd640 1 -- 172.21.15.16:0/3325886734 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e1c009800 con 0x7f2e2c0cbbe0 2023-05-22T15:27:36.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.823+0000 7f2e22ffd640 1 -- 172.21.15.16:0/3325886734 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e1c009b70 con 0x7f2e2c0cbbe0 2023-05-22T15:27:36.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.823+0000 7f2e34549640 1 -- 172.21.15.16:0/3325886734 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e2c0c8c80 con 0x7f2e2c0cbbe0 2023-05-22T15:27:36.834 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.823+0000 7f2e34549640 1 -- 172.21.15.16:0/3325886734 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e2c0c8af0 con 0x7f2e2c0cbbe0 2023-05-22T15:27:36.834 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.823+0000 7f2e34549640 1 -- 172.21.15.16:0/3325886734 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e2c0c8e10 con 0x7f2e2c0cbbe0 2023-05-22T15:27:36.834 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.823+0000 7f2e22ffd640 1 -- 172.21.15.16:0/3325886734 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7f2e1c009cd0 con 0x7f2e2c0cbbe0 2023-05-22T15:27:36.834 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.823+0000 7f2e22ffd640 1 --2- 172.21.15.16:0/3325886734 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f2e10065380 0x7f2e10067830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:36.835 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.823+0000 7f2e322e5640 1 --2- 172.21.15.16:0/3325886734 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f2e10065380 0x7f2e10067830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:36.835 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.823+0000 7f2e22ffd640 1 -- 172.21.15.16:0/3325886734 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f2e1c083290 con 0x7f2e2c0cbbe0 2023-05-22T15:27:36.835 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.823+0000 7f2e322e5640 1 --2- 172.21.15.16:0/3325886734 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f2e10065380 0x7f2e10067830 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f2e280ad000 tx=0x7f2e28008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:36.836 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.827+0000 7f2e22ffd640 1 -- 172.21.15.16:0/3325886734 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f2e1c050b80 con 0x7f2e2c0cbbe0 2023-05-22T15:27:36.993 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.983+0000 7f2e34549640 1 -- 172.21.15.16:0/3325886734 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f2e2c15d190 con 0x7f2e2c0cbbe0 2023-05-22T15:27:36.993 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.987+0000 7f2e22ffd640 1 -- 172.21.15.16:0/3325886734 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v29) v1 ==== 92+0+10 (secure 0 0 0) 0x7f2e1c0541a0 con 0x7f2e2c0cbbe0 2023-05-22T15:27:36.994 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 8 2023-05-22T15:27:36.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.987+0000 7f2e34549640 1 -- 172.21.15.16:0/3325886734 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f2e10065380 msgr2=0x7f2e10067830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:36.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.987+0000 7f2e34549640 1 --2- 172.21.15.16:0/3325886734 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f2e10065380 0x7f2e10067830 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f2e280ad000 tx=0x7f2e28008040 comp rx=0 tx=0).stop 2023-05-22T15:27:36.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.987+0000 7f2e34549640 1 -- 172.21.15.16:0/3325886734 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e2c0cbbe0 msgr2=0x7f2e2c15d560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:36.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.987+0000 7f2e34549640 1 --2- 172.21.15.16:0/3325886734 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e2c0cbbe0 0x7f2e2c15d560 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f2e1c00d800 tx=0x7f2e1c00dcc0 comp rx=0 tx=0).stop 2023-05-22T15:27:36.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.987+0000 7f2e34549640 1 -- 172.21.15.16:0/3325886734 shutdown_connections 2023-05-22T15:27:36.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.987+0000 7f2e34549640 1 --2- 172.21.15.16:0/3325886734 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2e2c15bee0 0x7f2e2c0ae570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:36.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.987+0000 7f2e34549640 1 --2- 172.21.15.16:0/3325886734 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e2c0cbbe0 0x7f2e2c15d560 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:36.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.987+0000 7f2e34549640 1 --2- 172.21.15.16:0/3325886734 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f2e10065380 0x7f2e10067830 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:36.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.987+0000 7f2e34549640 1 --2- 172.21.15.16:0/3325886734 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2e2c0cb730 0x7f2e2c15c2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:36.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.987+0000 7f2e34549640 1 -- 172.21.15.16:0/3325886734 >> 172.21.15.16:0/3325886734 conn(0x7f2e2c069720 msgr2=0x7f2e2c0ad9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:36.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.987+0000 7f2e34549640 1 -- 172.21.15.16:0/3325886734 shutdown_connections 2023-05-22T15:27:36.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:36.987+0000 7f2e34549640 1 -- 172.21.15.16:0/3325886734 wait complete. 2023-05-22T15:27:37.013 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-05-22T15:27:37.136 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.127+0000 7f5e8c3af640 1 Processor -- start 2023-05-22T15:27:37.136 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.127+0000 7f5e8c3af640 1 -- start start 2023-05-22T15:27:37.138 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.127+0000 7f5e8c3af640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f5e84158210 0x7f5e84159160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:37.138 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.127+0000 7f5e8c3af640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5e8415bcd0 0x7f5e8415c120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:37.138 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.127+0000 7f5e8c3af640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f5e8406b6c0 0x7f5e840c8660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:37.138 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.127+0000 7f5e8c3af640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e84158cd0 con 0x7f5e84158210 2023-05-22T15:27:37.138 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.127+0000 7f5e8c3af640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e841596a0 con 0x7f5e8406b6c0 2023-05-22T15:27:37.138 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.127+0000 7f5e8c3af640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e8415c790 con 0x7f5e8415bcd0 2023-05-22T15:27:37.138 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.131+0000 7f5e8a14b640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f5e84158210 0x7f5e84159160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:37.139 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.131+0000 7f5e8a14b640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f5e84158210 0x7f5e84159160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57568/0 (socket says 172.21.15.16:57568) 2023-05-22T15:27:37.139 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.131+0000 7f5e8a14b640 1 -- 172.21.15.16:0/881357237 learned_addr learned my addr 172.21.15.16:0/881357237 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:37.139 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.131+0000 7f5e8994a640 1 --2- 172.21.15.16:0/881357237 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5e8415bcd0 0x7f5e8415c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:37.139 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.131+0000 7f5e8a94c640 1 --2- 172.21.15.16:0/881357237 >> v2:172.21.15.62:3300/0 conn(0x7f5e8406b6c0 0x7f5e840c8660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:37.139 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.131+0000 7f5e8a14b640 1 -- 172.21.15.16:0/881357237 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5e8415bcd0 msgr2=0x7f5e8415c120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:37.139 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.131+0000 7f5e8a14b640 1 --2- 172.21.15.16:0/881357237 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5e8415bcd0 0x7f5e8415c120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:37.139 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.131+0000 7f5e8a14b640 1 -- 172.21.15.16:0/881357237 >> v2:172.21.15.62:3300/0 conn(0x7f5e8406b6c0 msgr2=0x7f5e840c8660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:37.140 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.131+0000 7f5e8a14b640 1 --2- 172.21.15.16:0/881357237 >> v2:172.21.15.62:3300/0 conn(0x7f5e8406b6c0 0x7f5e840c8660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:37.140 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.131+0000 7f5e8a14b640 1 -- 172.21.15.16:0/881357237 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e8415ca70 con 0x7f5e84158210 2023-05-22T15:27:37.140 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.131+0000 7f5e8a94c640 1 --2- 172.21.15.16:0/881357237 >> v2:172.21.15.62:3300/0 conn(0x7f5e8406b6c0 0x7f5e840c8660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T15:27:37.141 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.131+0000 7f5e8a14b640 1 --2- 172.21.15.16:0/881357237 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5e84158210 0x7f5e84159160 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f5e840c9320 tx=0x7f5e8001c440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3470320e920c9cc7 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:37.141 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.131+0000 7f5e89149640 1 -- 172.21.15.16:0/881357237 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e800ab620 con 0x7f5e84158210 2023-05-22T15:27:37.141 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.131+0000 7f5e89149640 1 -- 172.21.15.16:0/881357237 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e800ab780 con 0x7f5e84158210 2023-05-22T15:27:37.141 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.131+0000 7f5e89149640 1 -- 172.21.15.16:0/881357237 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e800abb10 con 0x7f5e84158210 2023-05-22T15:27:37.141 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.131+0000 7f5e8c3af640 1 -- 172.21.15.16:0/881357237 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5e84158210 msgr2=0x7f5e84159160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:37.141 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.131+0000 7f5e8c3af640 1 --2- 172.21.15.16:0/881357237 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5e84158210 0x7f5e84159160 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f5e840c9320 tx=0x7f5e8001c440 comp rx=0 tx=0).stop 2023-05-22T15:27:37.142 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.131+0000 7f5e8c3af640 1 -- 172.21.15.16:0/881357237 shutdown_connections 2023-05-22T15:27:37.142 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.131+0000 7f5e8c3af640 1 --2- 172.21.15.16:0/881357237 >> v2:172.21.15.62:3300/0 conn(0x7f5e8406b6c0 0x7f5e840c8660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:37.142 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.131+0000 7f5e8c3af640 1 --2- 172.21.15.16:0/881357237 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5e8415bcd0 0x7f5e8415c120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:37.142 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.131+0000 7f5e8c3af640 1 --2- 172.21.15.16:0/881357237 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5e84158210 0x7f5e84159160 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:37.142 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.131+0000 7f5e8c3af640 1 -- 172.21.15.16:0/881357237 >> 172.21.15.16:0/881357237 conn(0x7f5e84069720 msgr2=0x7f5e84069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:37.142 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.131+0000 7f5e8c3af640 1 -- 172.21.15.16:0/881357237 shutdown_connections 2023-05-22T15:27:37.142 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.131+0000 7f5e8c3af640 1 -- 172.21.15.16:0/881357237 wait complete. 2023-05-22T15:27:37.144 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.135+0000 7f5e8c3af640 1 Processor -- start 2023-05-22T15:27:37.144 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.135+0000 7f5e8c3af640 1 -- start start 2023-05-22T15:27:37.144 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.135+0000 7f5e8c3af640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5e8406b6c0 0x7f5e840c57a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:37.144 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.135+0000 7f5e8c3af640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5e84158210 0x7f5e841571f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:37.144 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.135+0000 7f5e8c3af640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5e8415bcd0 0x7f5e84157730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:37.144 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.135+0000 7f5e8c3af640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e840c5ce0 con 0x7f5e84158210 2023-05-22T15:27:37.145 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.135+0000 7f5e8c3af640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e8414e560 con 0x7f5e8406b6c0 2023-05-22T15:27:37.145 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.135+0000 7f5e8c3af640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e8414e840 con 0x7f5e8415bcd0 2023-05-22T15:27:37.145 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.135+0000 7f5e8994a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5e84158210 0x7f5e841571f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:37.145 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.135+0000 7f5e8994a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5e84158210 0x7f5e841571f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57570/0 (socket says 172.21.15.16:57570) 2023-05-22T15:27:37.145 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.135+0000 7f5e8994a640 1 -- 172.21.15.16:0/3170235707 learned_addr learned my addr 172.21.15.16:0/3170235707 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:37.145 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.135+0000 7f5e8a94c640 1 --2- 172.21.15.16:0/3170235707 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5e8415bcd0 0x7f5e84157730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:37.145 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.135+0000 7f5e8a14b640 1 --2- 172.21.15.16:0/3170235707 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5e8406b6c0 0x7f5e840c57a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:37.146 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.135+0000 7f5e8994a640 1 -- 172.21.15.16:0/3170235707 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5e8415bcd0 msgr2=0x7f5e84157730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:37.146 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.135+0000 7f5e8994a640 1 --2- 172.21.15.16:0/3170235707 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5e8415bcd0 0x7f5e84157730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:37.146 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.135+0000 7f5e8994a640 1 -- 172.21.15.16:0/3170235707 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5e8406b6c0 msgr2=0x7f5e840c57a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:37.146 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.135+0000 7f5e8994a640 1 --2- 172.21.15.16:0/3170235707 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5e8406b6c0 0x7f5e840c57a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:37.146 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.135+0000 7f5e8994a640 1 -- 172.21.15.16:0/3170235707 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e800094d0 con 0x7f5e84158210 2023-05-22T15:27:37.146 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.135+0000 7f5e8994a640 1 --2- 172.21.15.16:0/3170235707 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5e84158210 0x7f5e841571f0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f5e7400e870 tx=0x7f5e7400ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:37.146 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.135+0000 7f5e7affd640 1 -- 172.21.15.16:0/3170235707 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e74009530 con 0x7f5e84158210 2023-05-22T15:27:37.146 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.135+0000 7f5e7affd640 1 -- 172.21.15.16:0/3170235707 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e7400c940 con 0x7f5e84158210 2023-05-22T15:27:37.147 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.135+0000 7f5e8c3af640 1 -- 172.21.15.16:0/3170235707 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e841585e0 con 0x7f5e84158210 2023-05-22T15:27:37.147 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.135+0000 7f5e7affd640 1 -- 172.21.15.16:0/3170235707 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e7400ccd0 con 0x7f5e84158210 2023-05-22T15:27:37.147 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.135+0000 7f5e8c3af640 1 -- 172.21.15.16:0/3170235707 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e840c8fb0 con 0x7f5e84158210 2023-05-22T15:27:37.148 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.135+0000 7f5e8c3af640 1 -- 172.21.15.16:0/3170235707 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e840caea0 con 0x7f5e84158210 2023-05-22T15:27:37.148 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.139+0000 7f5e7affd640 1 -- 172.21.15.16:0/3170235707 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7f5e74013470 con 0x7f5e84158210 2023-05-22T15:27:37.148 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.139+0000 7f5e7affd640 1 --2- 172.21.15.16:0/3170235707 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f5e68065380 0x7f5e68067830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:37.148 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.139+0000 7f5e7affd640 1 -- 172.21.15.16:0/3170235707 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f5e74082cb0 con 0x7f5e84158210 2023-05-22T15:27:37.148 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.139+0000 7f5e8a14b640 1 --2- 172.21.15.16:0/3170235707 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f5e68065380 0x7f5e68067830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:37.148 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.139+0000 7f5e8a14b640 1 --2- 172.21.15.16:0/3170235707 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f5e68065380 0x7f5e68067830 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f5e800af000 tx=0x7f5e80008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:37.149 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.143+0000 7f5e7affd640 1 -- 172.21.15.16:0/3170235707 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f5e740505a0 con 0x7f5e84158210 2023-05-22T15:27:37.306 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.299+0000 7f5e8c3af640 1 -- 172.21.15.16:0/3170235707 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f5e8404dbe0 con 0x7f5e84158210 2023-05-22T15:27:37.306 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.299+0000 7f5e7affd640 1 -- 172.21.15.16:0/3170235707 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v29) v1 ==== 104+0+11 (secure 0 0 0) 0x7f5e740820a0 con 0x7f5e84158210 2023-05-22T15:27:37.307 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2023-05-22T15:27:37.309 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.299+0000 7f5e8c3af640 1 -- 172.21.15.16:0/3170235707 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f5e68065380 msgr2=0x7f5e68067830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:37.309 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.299+0000 7f5e8c3af640 1 --2- 172.21.15.16:0/3170235707 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f5e68065380 0x7f5e68067830 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f5e800af000 tx=0x7f5e80008040 comp rx=0 tx=0).stop 2023-05-22T15:27:37.309 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.299+0000 7f5e8c3af640 1 -- 172.21.15.16:0/3170235707 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5e84158210 msgr2=0x7f5e841571f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:37.309 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.299+0000 7f5e8c3af640 1 --2- 172.21.15.16:0/3170235707 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5e84158210 0x7f5e841571f0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f5e7400e870 tx=0x7f5e7400ed30 comp rx=0 tx=0).stop 2023-05-22T15:27:37.309 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.299+0000 7f5e8c3af640 1 -- 172.21.15.16:0/3170235707 shutdown_connections 2023-05-22T15:27:37.310 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.299+0000 7f5e8c3af640 1 --2- 172.21.15.16:0/3170235707 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f5e68065380 0x7f5e68067830 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:37.310 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.299+0000 7f5e8c3af640 1 --2- 172.21.15.16:0/3170235707 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5e8415bcd0 0x7f5e84157730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:37.310 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.299+0000 7f5e8c3af640 1 --2- 172.21.15.16:0/3170235707 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5e84158210 0x7f5e841571f0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:37.310 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.299+0000 7f5e8c3af640 1 --2- 172.21.15.16:0/3170235707 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5e8406b6c0 0x7f5e840c57a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:37.310 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.299+0000 7f5e8c3af640 1 -- 172.21.15.16:0/3170235707 >> 172.21.15.16:0/3170235707 conn(0x7f5e84069720 msgr2=0x7f5e840af290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:37.310 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.299+0000 7f5e8c3af640 1 -- 172.21.15.16:0/3170235707 shutdown_connections 2023-05-22T15:27:37.310 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.299+0000 7f5e8c3af640 1 -- 172.21.15.16:0/3170235707 wait complete. 2023-05-22T15:27:37.326 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-05-22T15:27:37.457 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.451+0000 7fe01b9cd640 1 Processor -- start 2023-05-22T15:27:37.458 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.451+0000 7fe01b9cd640 1 -- start start 2023-05-22T15:27:37.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.451+0000 7fe01b9cd640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe01415bca0 0x7fe0140cda80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:37.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.451+0000 7fe01b9cd640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fe0140d2480 0x7fe0140d2900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:37.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.451+0000 7fe01b9cd640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0140cccc0 con 0x7fe0140b5340 2023-05-22T15:27:37.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.451+0000 7fe01b9cd640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0140cd660 con 0x7fe01415bca0 2023-05-22T15:27:37.460 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.451+0000 7fe01b9cd640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0140d2e40 con 0x7fe0140d2480 2023-05-22T15:27:37.460 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.451+0000 7fe019769640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe01415bca0 0x7fe0140cda80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:37.460 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.451+0000 7fe019769640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe01415bca0 0x7fe0140cda80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57576/0 (socket says 172.21.15.16:57576) 2023-05-22T15:27:37.460 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.451+0000 7fe019769640 1 -- 172.21.15.16:0/2281249941 learned_addr learned my addr 172.21.15.16:0/2281249941 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:37.460 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.451+0000 7fe018f68640 1 --2- 172.21.15.16:0/2281249941 >> v2:172.21.15.62:3300/0 conn(0x7fe0140d2480 0x7fe0140d2900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:37.460 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.451+0000 7fe019769640 1 -- 172.21.15.16:0/2281249941 >> v2:172.21.15.62:3300/0 conn(0x7fe0140d2480 msgr2=0x7fe0140d2900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:37.461 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.451+0000 7fe019769640 1 --2- 172.21.15.16:0/2281249941 >> v2:172.21.15.62:3300/0 conn(0x7fe0140d2480 0x7fe0140d2900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:37.461 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.451+0000 7fe019769640 1 -- 172.21.15.16:0/2281249941 >> v1:172.21.15.62:6789/0 conn(0x7fe0140b5340 legacy=0x7fe0140b5790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:37.461 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.451+0000 7fe019769640 1 -- 172.21.15.16:0/2281249941 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0140ccfa0 con 0x7fe01415bca0 2023-05-22T15:27:37.462 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.455+0000 7fe019769640 1 --2- 172.21.15.16:0/2281249941 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe01415bca0 0x7fe0140cda80 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fe014065e80 tx=0x7fe01001c4c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26b9ae7135f0eaaf server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:37.462 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.455+0000 7fe00bfff640 1 -- 172.21.15.16:0/2281249941 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0100aa640 con 0x7fe01415bca0 2023-05-22T15:27:37.462 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.455+0000 7fe00bfff640 1 -- 172.21.15.16:0/2281249941 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0100aa7a0 con 0x7fe01415bca0 2023-05-22T15:27:37.462 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.455+0000 7fe00bfff640 1 -- 172.21.15.16:0/2281249941 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0100aab30 con 0x7fe01415bca0 2023-05-22T15:27:37.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.455+0000 7fe01b9cd640 1 -- 172.21.15.16:0/2281249941 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe01415bca0 msgr2=0x7fe0140cda80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:37.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.455+0000 7fe01b9cd640 1 --2- 172.21.15.16:0/2281249941 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe01415bca0 0x7fe0140cda80 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fe014065e80 tx=0x7fe01001c4c0 comp rx=0 tx=0).stop 2023-05-22T15:27:37.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.455+0000 7fe01b9cd640 1 -- 172.21.15.16:0/2281249941 shutdown_connections 2023-05-22T15:27:37.464 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.455+0000 7fe01b9cd640 1 --2- 172.21.15.16:0/2281249941 >> v2:172.21.15.62:3300/0 conn(0x7fe0140d2480 0x7fe0140d2900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:37.464 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.455+0000 7fe01b9cd640 1 --2- 172.21.15.16:0/2281249941 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe01415bca0 0x7fe0140cda80 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:37.464 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.455+0000 7fe01b9cd640 1 -- 172.21.15.16:0/2281249941 >> 172.21.15.16:0/2281249941 conn(0x7fe0140655c0 msgr2=0x7fe014070e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:37.464 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.455+0000 7fe01b9cd640 1 -- 172.21.15.16:0/2281249941 shutdown_connections 2023-05-22T15:27:37.464 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.455+0000 7fe01b9cd640 1 -- 172.21.15.16:0/2281249941 wait complete. 2023-05-22T15:27:37.465 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.455+0000 7fe01b9cd640 1 Processor -- start 2023-05-22T15:27:37.466 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.455+0000 7fe01b9cd640 1 -- start start 2023-05-22T15:27:37.466 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.455+0000 7fe01b9cd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe0140b5340 0x7fe0140c7ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:37.466 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.455+0000 7fe01b9cd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe0140d2480 0x7fe0141517d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:37.466 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.455+0000 7fe01b9cd640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe01415bca0 0x7fe0140c89d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:37.466 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.455+0000 7fe01b9cd640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0141531d0 con 0x7fe0140d2480 2023-05-22T15:27:37.466 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.455+0000 7fe01b9cd640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0140c80b0 con 0x7fe0140b5340 2023-05-22T15:27:37.466 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.455+0000 7fe01b9cd640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe014152b10 con 0x7fe01415bca0 2023-05-22T15:27:37.467 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.455+0000 7fe018f68640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe0140d2480 0x7fe0141517d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:37.467 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.455+0000 7fe018f68640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe0140d2480 0x7fe0141517d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57590/0 (socket says 172.21.15.16:57590) 2023-05-22T15:27:37.467 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.455+0000 7fe018f68640 1 -- 172.21.15.16:0/355397588 learned_addr learned my addr 172.21.15.16:0/355397588 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:37.467 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.459+0000 7fe019769640 1 --2- 172.21.15.16:0/355397588 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe0140b5340 0x7fe0140c7ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:37.467 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.459+0000 7fe019f6a640 1 --2- 172.21.15.16:0/355397588 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe01415bca0 0x7fe0140c89d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:37.467 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.459+0000 7fe018f68640 1 -- 172.21.15.16:0/355397588 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe01415bca0 msgr2=0x7fe0140c89d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:37.467 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.459+0000 7fe018f68640 1 --2- 172.21.15.16:0/355397588 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe01415bca0 0x7fe0140c89d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:37.468 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.459+0000 7fe018f68640 1 -- 172.21.15.16:0/355397588 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe0140b5340 msgr2=0x7fe0140c7ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:37.468 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.459+0000 7fe018f68640 1 --2- 172.21.15.16:0/355397588 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe0140b5340 0x7fe0140c7ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:37.468 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.459+0000 7fe018f68640 1 -- 172.21.15.16:0/355397588 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0100094d0 con 0x7fe0140d2480 2023-05-22T15:27:37.468 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.459+0000 7fe018f68640 1 --2- 172.21.15.16:0/355397588 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe0140d2480 0x7fe0141517d0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fe00400d800 tx=0x7fe00400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:37.468 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.459+0000 7fe009ffb640 1 -- 172.21.15.16:0/355397588 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe00400b6c0 con 0x7fe0140d2480 2023-05-22T15:27:37.468 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.459+0000 7fe009ffb640 1 -- 172.21.15.16:0/355397588 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe00400b820 con 0x7fe0140d2480 2023-05-22T15:27:37.468 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.459+0000 7fe01b9cd640 1 -- 172.21.15.16:0/355397588 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe014160360 con 0x7fe0140d2480 2023-05-22T15:27:37.468 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.459+0000 7fe009ffb640 1 -- 172.21.15.16:0/355397588 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe00400bb90 con 0x7fe0140d2480 2023-05-22T15:27:37.469 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.459+0000 7fe01b9cd640 1 -- 172.21.15.16:0/355397588 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0140cdfc0 con 0x7fe0140d2480 2023-05-22T15:27:37.469 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.459+0000 7fe01b9cd640 1 -- 172.21.15.16:0/355397588 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0141604f0 con 0x7fe0140d2480 2023-05-22T15:27:37.469 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.459+0000 7fe009ffb640 1 -- 172.21.15.16:0/355397588 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7fe00400bcf0 con 0x7fe0140d2480 2023-05-22T15:27:37.470 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.459+0000 7fe009ffb640 1 --2- 172.21.15.16:0/355397588 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fdff8065380 0x7fdff8067830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:37.470 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.459+0000 7fe019769640 1 --2- 172.21.15.16:0/355397588 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fdff8065380 0x7fdff8067830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:37.470 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.459+0000 7fe009ffb640 1 -- 172.21.15.16:0/355397588 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6871+0+0 (secure 0 0 0) 0x7fe004083060 con 0x7fe0140d2480 2023-05-22T15:27:37.470 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.459+0000 7fe019769640 1 --2- 172.21.15.16:0/355397588 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fdff8065380 0x7fdff8067830 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fe0100ae000 tx=0x7fe0100a7000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:37.472 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.463+0000 7fe009ffb640 1 -- 172.21.15.16:0/355397588 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fe004054470 con 0x7fe0140d2480 2023-05-22T15:27:37.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.623+0000 7fe01b9cd640 1 -- 172.21.15.16:0/355397588 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fe014152960 con 0x7fe0140d2480 2023-05-22T15:27:37.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.623+0000 7fe009ffb640 1 -- 172.21.15.16:0/355397588 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v29) v1 ==== 100+0+11 (secure 0 0 0) 0x7fe004054060 con 0x7fe0140d2480 2023-05-22T15:27:37.631 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 64 2023-05-22T15:27:37.633 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.623+0000 7fe01b9cd640 1 -- 172.21.15.16:0/355397588 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fdff8065380 msgr2=0x7fdff8067830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:37.633 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.623+0000 7fe01b9cd640 1 --2- 172.21.15.16:0/355397588 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fdff8065380 0x7fdff8067830 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fe0100ae000 tx=0x7fe0100a7000 comp rx=0 tx=0).stop 2023-05-22T15:27:37.633 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.623+0000 7fe01b9cd640 1 -- 172.21.15.16:0/355397588 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe0140d2480 msgr2=0x7fe0141517d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:37.633 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.623+0000 7fe01b9cd640 1 --2- 172.21.15.16:0/355397588 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe0140d2480 0x7fe0141517d0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fe00400d800 tx=0x7fe00400dcc0 comp rx=0 tx=0).stop 2023-05-22T15:27:37.633 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.623+0000 7fe01b9cd640 1 -- 172.21.15.16:0/355397588 shutdown_connections 2023-05-22T15:27:37.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.623+0000 7fe01b9cd640 1 --2- 172.21.15.16:0/355397588 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fdff8065380 0x7fdff8067830 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:37.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.623+0000 7fe01b9cd640 1 --2- 172.21.15.16:0/355397588 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe01415bca0 0x7fe0140c89d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:37.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.623+0000 7fe01b9cd640 1 --2- 172.21.15.16:0/355397588 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe0140d2480 0x7fe0141517d0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:37.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.623+0000 7fe01b9cd640 1 --2- 172.21.15.16:0/355397588 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe0140b5340 0x7fe0140c7ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:37.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.623+0000 7fe01b9cd640 1 -- 172.21.15.16:0/355397588 >> 172.21.15.16:0/355397588 conn(0x7fe0140655c0 msgr2=0x7fe014157c90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:37.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.623+0000 7fe01b9cd640 1 -- 172.21.15.16:0/355397588 shutdown_connections 2023-05-22T15:27:37.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.627+0000 7fe01b9cd640 1 -- 172.21.15.16:0/355397588 wait complete. 2023-05-22T15:27:37.650 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-05-22T15:27:37.778 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.771+0000 7f695cb67640 1 Processor -- start 2023-05-22T15:27:37.778 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.771+0000 7f695cb67640 1 -- start start 2023-05-22T15:27:37.779 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.771+0000 7f695cb67640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6958159970 0x7f6958159d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:37.779 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.771+0000 7f695cb67640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f695815d680 0x7f69580cb260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:37.779 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.771+0000 7f695cb67640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6958070c30 con 0x7f695815dad0 2023-05-22T15:27:37.779 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.771+0000 7f695cb67640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6958070e60 con 0x7f6958159970 2023-05-22T15:27:37.779 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.771+0000 7f695cb67640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6958071140 con 0x7f695815d680 2023-05-22T15:27:37.780 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.771+0000 7f695659c640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6958159970 0x7f6958159d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:37.780 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.771+0000 7f6956d9d640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f695815dad0 0x7f695806b6c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:36756/0 (socket says 172.21.15.16:36756) 2023-05-22T15:27:37.780 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.771+0000 7f6956d9d640 1 -- 172.21.15.16:0/1246349576 learned_addr learned my addr 172.21.15.16:0/1246349576 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:27:37.780 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.771+0000 7f6955d9b640 1 --2- 172.21.15.16:0/1246349576 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f695815d680 0x7f69580cb260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:37.781 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.771+0000 7f695659c640 1 -- 172.21.15.16:0/1246349576 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f695815d680 msgr2=0x7f69580cb260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:37.781 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.771+0000 7f695659c640 1 --2- 172.21.15.16:0/1246349576 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f695815d680 0x7f69580cb260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:37.781 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.771+0000 7f695659c640 1 -- 172.21.15.16:0/1246349576 >> v1:172.21.15.16:6789/0 conn(0x7f695815dad0 legacy=0x7f695806b6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:37.781 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.771+0000 7f695659c640 1 -- 172.21.15.16:0/1246349576 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6958071420 con 0x7f6958159970 2023-05-22T15:27:37.781 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.775+0000 7f695659c640 1 --2- 172.21.15.16:0/1246349576 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6958159970 0x7f6958159d40 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f69480097c0 tx=0x7f694801c3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db5b26a78d245469 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:37.782 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.775+0000 7f695559a640 1 -- 172.21.15.16:0/1246349576 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69480ac640 con 0x7f6958159970 2023-05-22T15:27:37.782 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.775+0000 7f695559a640 1 -- 172.21.15.16:0/1246349576 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69480ac7a0 con 0x7f6958159970 2023-05-22T15:27:37.782 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.775+0000 7f695559a640 1 -- 172.21.15.16:0/1246349576 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69480acb30 con 0x7f6958159970 2023-05-22T15:27:37.783 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.775+0000 7f695cb67640 1 -- 172.21.15.16:0/1246349576 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6958159970 msgr2=0x7f6958159d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:37.783 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.775+0000 7f695cb67640 1 --2- 172.21.15.16:0/1246349576 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6958159970 0x7f6958159d40 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f69480097c0 tx=0x7f694801c3c0 comp rx=0 tx=0).stop 2023-05-22T15:27:37.783 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.775+0000 7f695cb67640 1 -- 172.21.15.16:0/1246349576 shutdown_connections 2023-05-22T15:27:37.783 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.775+0000 7f695cb67640 1 --2- 172.21.15.16:0/1246349576 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f695815d680 0x7f69580cb260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:37.783 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.775+0000 7f695cb67640 1 --2- 172.21.15.16:0/1246349576 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6958159970 0x7f6958159d40 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:37.783 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.775+0000 7f695cb67640 1 -- 172.21.15.16:0/1246349576 >> 172.21.15.16:0/1246349576 conn(0x7f6958069720 msgr2=0x7f6958069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:37.784 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.775+0000 7f695cb67640 1 -- 172.21.15.16:0/1246349576 shutdown_connections 2023-05-22T15:27:37.784 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.775+0000 7f695cb67640 1 -- 172.21.15.16:0/1246349576 wait complete. 2023-05-22T15:27:37.785 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.775+0000 7f695cb67640 1 Processor -- start 2023-05-22T15:27:37.786 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.775+0000 7f695cb67640 1 -- start start 2023-05-22T15:27:37.786 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.775+0000 7f695cb67640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6958159970 0x7f695815ca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:37.786 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.775+0000 7f695cb67640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f695815d680 0x7f69580ccc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:37.786 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.775+0000 7f695cb67640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f695815dad0 0x7f69580cd190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:37.786 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.775+0000 7f695cb67640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f695815c590 con 0x7f6958159970 2023-05-22T15:27:37.786 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.775+0000 7f695cb67640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6958150af0 con 0x7f695815dad0 2023-05-22T15:27:37.787 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.775+0000 7f695cb67640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f695815b6e0 con 0x7f695815d680 2023-05-22T15:27:37.787 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.775+0000 7f695659c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6958159970 0x7f695815ca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:37.787 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.775+0000 7f695659c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6958159970 0x7f695815ca20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57614/0 (socket says 172.21.15.16:57614) 2023-05-22T15:27:37.787 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.775+0000 7f695659c640 1 -- 172.21.15.16:0/2926265470 learned_addr learned my addr 172.21.15.16:0/2926265470 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:37.787 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.775+0000 7f6955d9b640 1 --2- 172.21.15.16:0/2926265470 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f695815d680 0x7f69580ccc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:37.787 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.775+0000 7f695659c640 1 -- 172.21.15.16:0/2926265470 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f695815d680 msgr2=0x7f69580ccc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:37.787 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.775+0000 7f695659c640 1 --2- 172.21.15.16:0/2926265470 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f695815d680 0x7f69580ccc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:37.788 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.775+0000 7f695659c640 1 -- 172.21.15.16:0/2926265470 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f695815dad0 msgr2=0x7f69580cd190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:37.788 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.775+0000 7f695659c640 1 --2- 172.21.15.16:0/2926265470 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f695815dad0 0x7f69580cd190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:37.788 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.775+0000 7f695659c640 1 -- 172.21.15.16:0/2926265470 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6948009420 con 0x7f6958159970 2023-05-22T15:27:37.788 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.775+0000 7f6955d9b640 1 --2- 172.21.15.16:0/2926265470 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f695815d680 0x7f69580ccc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:27:37.788 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.779+0000 7f695659c640 1 --2- 172.21.15.16:0/2926265470 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6958159970 0x7f695815ca20 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f69480097c0 tx=0x7f6948019160 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:37.789 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.779+0000 7f6946ffd640 1 -- 172.21.15.16:0/2926265470 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69480b5820 con 0x7f6958159970 2023-05-22T15:27:37.789 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.779+0000 7f6946ffd640 1 -- 172.21.15.16:0/2926265470 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69480b59a0 con 0x7f6958159970 2023-05-22T15:27:37.789 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.779+0000 7f695cb67640 1 -- 172.21.15.16:0/2926265470 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69580cae20 con 0x7f6958159970 2023-05-22T15:27:37.789 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.779+0000 7f6946ffd640 1 -- 172.21.15.16:0/2926265470 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69480b5d50 con 0x7f6958159970 2023-05-22T15:27:37.789 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.779+0000 7f695cb67640 1 -- 172.21.15.16:0/2926265470 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69580cac90 con 0x7f6958159970 2023-05-22T15:27:37.790 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.779+0000 7f695cb67640 1 -- 172.21.15.16:0/2926265470 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f695806bdd0 con 0x7f6958159970 2023-05-22T15:27:37.790 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.779+0000 7f6946ffd640 1 -- 172.21.15.16:0/2926265470 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7f69480b3040 con 0x7f6958159970 2023-05-22T15:27:37.790 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.779+0000 7f6946ffd640 1 --2- 172.21.15.16:0/2926265470 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f6938065380 0x7f6938067830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:37.790 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.779+0000 7f6946ffd640 1 -- 172.21.15.16:0/2926265470 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f694811cfc0 con 0x7f6958159970 2023-05-22T15:27:37.790 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.779+0000 7f6955d9b640 1 --2- 172.21.15.16:0/2926265470 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f6938065380 0x7f6938067830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:37.791 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.779+0000 7f6955d9b640 1 --2- 172.21.15.16:0/2926265470 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f6938065380 0x7f6938067830 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f69581f75a0 tx=0x7f694c008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:37.791 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.783+0000 7f6946ffd640 1 -- 172.21.15.16:0/2926265470 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f69480ea910 con 0x7f6958159970 2023-05-22T15:27:37.945 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.935+0000 7f695cb67640 1 -- 172.21.15.16:0/2926265470 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f695804dbe0 con 0x7f6958159970 2023-05-22T15:27:37.946 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.939+0000 7f6946ffd640 1 -- 172.21.15.16:0/2926265470 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v29) v1 ==== 103+0+11 (secure 0 0 0) 0x7f69480b10a0 con 0x7f6958159970 2023-05-22T15:27:37.946 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2023-05-22T15:27:37.949 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.939+0000 7f695cb67640 1 -- 172.21.15.16:0/2926265470 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f6938065380 msgr2=0x7f6938067830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:37.950 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.939+0000 7f695cb67640 1 --2- 172.21.15.16:0/2926265470 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f6938065380 0x7f6938067830 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f69581f75a0 tx=0x7f694c008040 comp rx=0 tx=0).stop 2023-05-22T15:27:37.950 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.939+0000 7f695cb67640 1 -- 172.21.15.16:0/2926265470 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6958159970 msgr2=0x7f695815ca20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:37.950 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.939+0000 7f695cb67640 1 --2- 172.21.15.16:0/2926265470 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6958159970 0x7f695815ca20 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f69480097c0 tx=0x7f6948019160 comp rx=0 tx=0).stop 2023-05-22T15:27:37.951 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.939+0000 7f695cb67640 1 -- 172.21.15.16:0/2926265470 shutdown_connections 2023-05-22T15:27:37.951 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.939+0000 7f695cb67640 1 --2- 172.21.15.16:0/2926265470 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f695815dad0 0x7f69580cd190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:37.951 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.939+0000 7f695cb67640 1 --2- 172.21.15.16:0/2926265470 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f6938065380 0x7f6938067830 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:37.951 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.939+0000 7f695cb67640 1 --2- 172.21.15.16:0/2926265470 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f695815d680 0x7f69580ccc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:37.952 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.939+0000 7f695cb67640 1 --2- 172.21.15.16:0/2926265470 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6958159970 0x7f695815ca20 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:37.952 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.939+0000 7f695cb67640 1 -- 172.21.15.16:0/2926265470 >> 172.21.15.16:0/2926265470 conn(0x7f6958069720 msgr2=0x7f695806b6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:37.953 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.939+0000 7f695cb67640 1 -- 172.21.15.16:0/2926265470 shutdown_connections 2023-05-22T15:27:37.953 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:37.939+0000 7f695cb67640 1 -- 172.21.15.16:0/2926265470 wait complete. 2023-05-22T15:27:38.000 INFO:tasks.cephfs_test_runner:Starting test: test_concurrent_snap_creates (tasks.cephfs.test_snap_schedules.TestSnapSchedules) Test concurrent snap creates in same file-system without db issues 2023-05-22T15:27:38.001 INFO:tasks.cephfs_test_runner:test_concurrent_snap_creates (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2023-05-22T15:27:38.002 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_concurrent_snap_creates' 2023-05-22T15:27:38.126 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:38.119+0000 7fed2a49d640 1 Processor -- start 2023-05-22T15:27:38.126 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:38.119+0000 7fed2a49d640 1 -- start start 2023-05-22T15:27:38.127 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:38.119+0000 7fed2a49d640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fed240cd730 0x7fed240afb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:38.127 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:38.119+0000 7fed2a49d640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed240cc300 con 0x7fed241531d0 2023-05-22T15:27:38.127 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:38.119+0000 7fed2a49d640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed240b04c0 con 0x7fed240b0080 2023-05-22T15:27:38.127 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:38.119+0000 7fed2a49d640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed2415d330 con 0x7fed240cd730 2023-05-22T15:27:38.127 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:38.119+0000 7fed23fff640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fed241531d0 0x7fed240cbbf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:36772/0 (socket says 172.21.15.16:36772) 2023-05-22T15:27:38.127 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:38.119+0000 7fed23fff640 1 -- 172.21.15.16:0/2901654262 learned_addr learned my addr 172.21.15.16:0/2901654262 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:27:38.129 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:38.119+0000 7fed237fe640 1 --2- 172.21.15.16:0/2901654262 >> v2:172.21.15.62:3300/0 conn(0x7fed240cd730 0x7fed240afb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:38.129 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:38.119+0000 7fed22ffd640 1 -- 172.21.15.16:0/2901654262 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 123393911 0 0) 0x7fed240cc300 con 0x7fed241531d0 2023-05-22T15:27:38.129 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:38.119+0000 7fed22ffd640 1 -- 172.21.15.16:0/2901654262 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed0c003530 con 0x7fed241531d0 2023-05-22T15:27:38.129 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:38.119+0000 7fed22ffd640 1 -- 172.21.15.16:0/2901654262 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 811709333 0 0) 0x7fed0c003530 con 0x7fed241531d0 2023-05-22T15:27:38.129 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:38.119+0000 7fed22ffd640 1 -- 172.21.15.16:0/2901654262 >> v2:172.21.15.62:3300/0 conn(0x7fed240cd730 msgr2=0x7fed240afb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:38.129 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:38.119+0000 7fed22ffd640 1 --2- 172.21.15.16:0/2901654262 >> v2:172.21.15.62:3300/0 conn(0x7fed240cd730 0x7fed240afb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:38.130 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:38.119+0000 7fed22ffd640 1 -- 172.21.15.16:0/2901654262 >> v1:172.21.15.62:6789/0 conn(0x7fed240b0080 legacy=0x7fed240acba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:38.130 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:38.119+0000 7fed22ffd640 1 -- 172.21.15.16:0/2901654262 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed240cdb60 con 0x7fed241531d0 2023-05-22T15:27:38.130 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:38.119+0000 7fed22ffd640 1 -- 172.21.15.16:0/2901654262 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7fed14001680 con 0x7fed241531d0 2023-05-22T15:27:38.130 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:38.119+0000 7fed22ffd640 1 -- 172.21.15.16:0/2901654262 >> v1:172.21.15.16:6789/0 conn(0x7fed241531d0 legacy=0x7fed240cbbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:38.130 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:38.119+0000 7fed22ffd640 1 --2- 172.21.15.16:0/2901654262 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fed0c003980 0x7fed0c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:38.130 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:38.119+0000 7fed22ffd640 1 --2- 172.21.15.16:0/2901654262 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fed0c0047d0 0x7fed0c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:38.130 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:38.119+0000 7fed22ffd640 1 --2- 172.21.15.16:0/2901654262 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed0c005180 0x7fed0c00b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:38.131 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:38.119+0000 7fed22ffd640 1 -- 172.21.15.16:0/2901654262 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed2415d330 con 0x7fed0c005180 2023-05-22T15:27:38.131 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:38.119+0000 7fed22ffd640 1 -- 172.21.15.16:0/2901654262 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed0c00c000 con 0x7fed0c0047d0 2023-05-22T15:27:38.131 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:38.119+0000 7fed22ffd640 1 -- 172.21.15.16:0/2901654262 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed0c00c1c0 con 0x7fed0c003980 2023-05-22T15:27:38.131 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:38.119+0000 7fed28a3a640 1 --2- 172.21.15.16:0/2901654262 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed0c005180 0x7fed0c00b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:38.131 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:38.119+0000 7fed23fff640 1 --2- 172.21.15.16:0/2901654262 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fed0c003980 0x7fed0c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:38.131 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:38.119+0000 7fed237fe640 1 --2- 172.21.15.16:0/2901654262 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fed0c0047d0 0x7fed0c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:38.131 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:38.119+0000 7fed28a3a640 1 -- 172.21.15.16:0/2901654262 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fed0c003980 msgr2=0x7fed0c003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:38.131 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:38.119+0000 7fed28a3a640 1 --2- 172.21.15.16:0/2901654262 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fed0c003980 0x7fed0c003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:38.132 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:38.119+0000 7fed28a3a640 1 -- 172.21.15.16:0/2901654262 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fed0c0047d0 msgr2=0x7fed0c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:38.132 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:38.119+0000 7fed28a3a640 1 --2- 172.21.15.16:0/2901654262 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fed0c0047d0 0x7fed0c004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:38.132 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:38.119+0000 7fed28a3a640 1 -- 172.21.15.16:0/2901654262 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed0c003530 con 0x7fed0c005180 2023-05-22T15:27:38.134 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:38.123+0000 7fed28a3a640 1 -- 172.21.15.16:0/2901654262 reap_dead start 2023-05-22T15:27:38.134 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:38.123+0000 7fed28a3a640 1 --2- 172.21.15.16:0/2901654262 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed0c005180 0x7fed0c00b990 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fed1c001780 tx=0x7fed1c01c400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a186e7e5b9ac2bb7 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:38.135 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:38.123+0000 7fed22ffd640 1 -- 172.21.15.16:0/2901654262 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed1c0052d0 con 0x7fed0c005180 2023-05-22T15:27:38.135 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:38.123+0000 7fed22ffd640 1 -- 172.21.15.16:0/2901654262 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed1c0a7050 con 0x7fed0c005180 2023-05-22T15:27:38.135 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:38.123+0000 7fed22ffd640 1 -- 172.21.15.16:0/2901654262 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed1c003840 con 0x7fed0c005180 2023-05-22T15:27:38.135 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:38.123+0000 7fed2a49d640 1 -- 172.21.15.16:0/2901654262 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed0c005180 msgr2=0x7fed0c00b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:38.135 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:38.123+0000 7fed2a49d640 1 --2- 172.21.15.16:0/2901654262 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed0c005180 0x7fed0c00b990 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fed1c001780 tx=0x7fed1c01c400 comp rx=0 tx=0).stop 2023-05-22T15:27:38.135 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:38.123+0000 7fed2a49d640 1 -- 172.21.15.16:0/2901654262 shutdown_connections 2023-05-22T15:27:38.135 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:38.123+0000 7fed2a49d640 1 --2- 172.21.15.16:0/2901654262 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed0c005180 0x7fed0c00b990 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:38.136 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:38.123+0000 7fed2a49d640 1 -- 172.21.15.16:0/2901654262 >> 172.21.15.16:0/2901654262 conn(0x7fed24069720 msgr2=0x7fed24069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:38.136 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:38.123+0000 7fed2a49d640 1 -- 172.21.15.16:0/2901654262 shutdown_connections 2023-05-22T15:27:38.136 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:38.123+0000 7fed2a49d640 1 -- 172.21.15.16:0/2901654262 wait complete. 2023-05-22T15:27:38.136 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:38.123+0000 7fed2a49d640 1 Processor -- start 2023-05-22T15:27:38.136 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:38.123+0000 7fed2a49d640 1 -- start start 2023-05-22T15:27:38.136 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:38.127+0000 7fed2a49d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fed0c005180 0x7fed240c5c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:38.136 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:38.127+0000 7fed2a49d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed240c6220 0x7fed240c99b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:38.137 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:38.127+0000 7fed2a49d640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fed2414f280 0x7fed2414f700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:38.137 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:38.127+0000 7fed2a49d640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed240c7660 con 0x7fed240c6220 2023-05-22T15:27:38.137 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:38.127+0000 7fed2a49d640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed240c69e0 con 0x7fed0c005180 2023-05-22T15:27:38.137 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:38.127+0000 7fed2a49d640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed240c6cc0 con 0x7fed2414f280 2023-05-22T15:27:38.137 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:38.127+0000 7fed237fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed240c6220 0x7fed240c99b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:38.137 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:38.127+0000 7fed237fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed240c6220 0x7fed240c99b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57644/0 (socket says 172.21.15.16:57644) 2023-05-22T15:27:38.137 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:38.127+0000 7fed237fe640 1 -- 172.21.15.16:0/435985412 learned_addr learned my addr 172.21.15.16:0/435985412 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:38.137 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:38.127+0000 7fed28a3a640 1 --2- 172.21.15.16:0/435985412 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fed2414f280 0x7fed2414f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:38.138 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:38.127+0000 7fed23fff640 1 --2- 172.21.15.16:0/435985412 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fed0c005180 0x7fed240c5c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:38.138 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:38.127+0000 7fed237fe640 1 -- 172.21.15.16:0/435985412 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fed2414f280 msgr2=0x7fed2414f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:38.138 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:38.127+0000 7fed237fe640 1 --2- 172.21.15.16:0/435985412 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fed2414f280 0x7fed2414f700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:38.138 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:38.127+0000 7fed237fe640 1 -- 172.21.15.16:0/435985412 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fed0c005180 msgr2=0x7fed240c5c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:38.138 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:38.127+0000 7fed237fe640 1 --2- 172.21.15.16:0/435985412 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fed0c005180 0x7fed240c5c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:38.138 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:38.127+0000 7fed237fe640 1 -- 172.21.15.16:0/435985412 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed2415cc70 con 0x7fed240c6220 2023-05-22T15:27:38.138 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:38.127+0000 7fed237fe640 1 --2- 172.21.15.16:0/435985412 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed240c6220 0x7fed240c99b0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fed1800f350 tx=0x7fed1800f810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:38.138 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:38.127+0000 7fed20ff9640 1 -- 172.21.15.16:0/435985412 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed18017600 con 0x7fed240c6220 2023-05-22T15:27:38.139 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:38.127+0000 7fed20ff9640 1 -- 172.21.15.16:0/435985412 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed18011040 con 0x7fed240c6220 2023-05-22T15:27:38.139 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:38.127+0000 7fed20ff9640 1 -- 172.21.15.16:0/435985412 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed180179c0 con 0x7fed240c6220 2023-05-22T15:27:38.139 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:38.127+0000 7fed2a49d640 1 -- 172.21.15.16:0/435985412 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed2415c990 con 0x7fed240c6220 2023-05-22T15:27:38.139 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:38.127+0000 7fed2a49d640 1 -- 172.21.15.16:0/435985412 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed2415cf50 con 0x7fed240c6220 2023-05-22T15:27:38.140 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:38.127+0000 7fed2a49d640 1 -- 172.21.15.16:0/435985412 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed2404dbe0 con 0x7fed240c6220 2023-05-22T15:27:38.140 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:38.127+0000 7fed20ff9640 1 -- 172.21.15.16:0/435985412 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7fed18017b20 con 0x7fed240c6220 2023-05-22T15:27:38.140 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:38.127+0000 7fed20ff9640 1 --2- 172.21.15.16:0/435985412 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fed10065380 0x7fed10067830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:38.140 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:38.127+0000 7fed20ff9640 1 -- 172.21.15.16:0/435985412 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6871+0+0 (secure 0 0 0) 0x7fed180831f0 con 0x7fed240c6220 2023-05-22T15:27:38.140 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:38.127+0000 7fed23fff640 1 --2- 172.21.15.16:0/435985412 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fed10065380 0x7fed10067830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:38.141 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:38.127+0000 7fed23fff640 1 --2- 172.21.15.16:0/435985412 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fed10065380 0x7fed10067830 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fed14001060 tx=0x7fed14009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:38.141 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:38.131+0000 7fed20ff9640 1 -- 172.21.15.16:0/435985412 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fed18050c90 con 0x7fed240c6220 2023-05-22T15:27:38.348 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:38.339+0000 7fed2a49d640 1 -- 172.21.15.16:0/435985412 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_concurrent_snap_creates"]} v 0) v1 -- 0x7fed240c7340 con 0x7fed240c6220 2023-05-22T15:27:39.258 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:39.251+0000 7fed20ff9640 1 -- 172.21.15.16:0/435985412 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_concurrent_snap_creates"]}]=0 v68) v1 ==== 161+0+0 (secure 0 0 0) 0x7fed180542b0 con 0x7fed240c6220 2023-05-22T15:27:39.261 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:39.251+0000 7fed2a49d640 1 -- 172.21.15.16:0/435985412 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fed10065380 msgr2=0x7fed10067830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:39.261 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:39.251+0000 7fed2a49d640 1 --2- 172.21.15.16:0/435985412 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fed10065380 0x7fed10067830 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fed14001060 tx=0x7fed14009040 comp rx=0 tx=0).stop 2023-05-22T15:27:39.261 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:39.251+0000 7fed2a49d640 1 -- 172.21.15.16:0/435985412 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed240c6220 msgr2=0x7fed240c99b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:39.261 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:39.251+0000 7fed2a49d640 1 --2- 172.21.15.16:0/435985412 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed240c6220 0x7fed240c99b0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fed1800f350 tx=0x7fed1800f810 comp rx=0 tx=0).stop 2023-05-22T15:27:39.261 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:39.251+0000 7fed2a49d640 1 -- 172.21.15.16:0/435985412 shutdown_connections 2023-05-22T15:27:39.262 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:39.251+0000 7fed2a49d640 1 --2- 172.21.15.16:0/435985412 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fed10065380 0x7fed10067830 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:39.262 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:39.251+0000 7fed2a49d640 1 --2- 172.21.15.16:0/435985412 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fed2414f280 0x7fed2414f700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:39.262 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:39.251+0000 7fed2a49d640 1 --2- 172.21.15.16:0/435985412 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed240c6220 0x7fed240c99b0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:39.262 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:39.251+0000 7fed2a49d640 1 --2- 172.21.15.16:0/435985412 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fed0c005180 0x7fed240c5c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:39.262 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:39.251+0000 7fed2a49d640 1 -- 172.21.15.16:0/435985412 >> 172.21.15.16:0/435985412 conn(0x7fed24069720 msgr2=0x7fed24069af0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:39.262 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:39.251+0000 7fed2a49d640 1 -- 172.21.15.16:0/435985412 shutdown_connections 2023-05-22T15:27:39.262 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:39.255+0000 7fed2a49d640 1 -- 172.21.15.16:0/435985412 wait complete. 2023-05-22T15:27:39.281 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-05-22T15:27:39.413 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:39.403+0000 7fd4bdcc3640 1 Processor -- start 2023-05-22T15:27:39.413 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:39.407+0000 7fd4bdcc3640 1 -- start start 2023-05-22T15:27:39.414 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:39.407+0000 7fd4bdcc3640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd4b806b6c0 0x7fd4b80cb4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:39.415 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:39.407+0000 7fd4bdcc3640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd4b81596e0 0x7fd4b806fba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:39.415 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:39.407+0000 7fd4bdcc3640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4b8159f70 con 0x7fd4b806ba90 2023-05-22T15:27:39.415 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:39.407+0000 7fd4bdcc3640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4b8071760 con 0x7fd4b81596e0 2023-05-22T15:27:39.415 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:39.407+0000 7fd4bdcc3640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4b80ca7d0 con 0x7fd4b806b6c0 2023-05-22T15:27:39.415 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:39.407+0000 7fd4b7fff640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd4b81596e0 0x7fd4b806fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:39.415 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:39.407+0000 7fd4b7fff640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd4b81596e0 0x7fd4b806fba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41410/0 (socket says 172.21.15.16:41410) 2023-05-22T15:27:39.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:39.407+0000 7fd4b7fff640 1 -- 172.21.15.16:0/4248004087 learned_addr learned my addr 172.21.15.16:0/4248004087 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:39.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:39.407+0000 7fd4b77fe640 1 --2- 172.21.15.16:0/4248004087 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd4b806b6c0 0x7fd4b80cb4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:39.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:39.407+0000 7fd4b7fff640 1 -- 172.21.15.16:0/4248004087 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd4b806b6c0 msgr2=0x7fd4b80cb4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:39.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:39.407+0000 7fd4b7fff640 1 --2- 172.21.15.16:0/4248004087 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd4b806b6c0 0x7fd4b80cb4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:39.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:39.407+0000 7fd4b7fff640 1 -- 172.21.15.16:0/4248004087 >> v1:172.21.15.62:6789/0 conn(0x7fd4b806ba90 legacy=0x7fd4b8158fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:39.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:39.407+0000 7fd4b7fff640 1 -- 172.21.15.16:0/4248004087 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4b80cb030 con 0x7fd4b81596e0 2023-05-22T15:27:39.417 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:39.407+0000 7fd4b7fff640 1 --2- 172.21.15.16:0/4248004087 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd4b81596e0 0x7fd4b806fba0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fd4b0009e50 tx=0x7fd4b001c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6678bdcd6808b75 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:39.417 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:39.411+0000 7fd4b67fc640 1 -- 172.21.15.16:0/4248004087 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4b00a5810 con 0x7fd4b81596e0 2023-05-22T15:27:39.417 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:39.411+0000 7fd4b67fc640 1 -- 172.21.15.16:0/4248004087 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4b00a5970 con 0x7fd4b81596e0 2023-05-22T15:27:39.417 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:39.411+0000 7fd4b67fc640 1 -- 172.21.15.16:0/4248004087 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4b00a5d00 con 0x7fd4b81596e0 2023-05-22T15:27:39.418 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:39.411+0000 7fd4bdcc3640 1 -- 172.21.15.16:0/4248004087 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd4b81596e0 msgr2=0x7fd4b806fba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:39.418 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:39.411+0000 7fd4bdcc3640 1 --2- 172.21.15.16:0/4248004087 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd4b81596e0 0x7fd4b806fba0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fd4b0009e50 tx=0x7fd4b001c950 comp rx=0 tx=0).stop 2023-05-22T15:27:39.418 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:39.411+0000 7fd4bdcc3640 1 -- 172.21.15.16:0/4248004087 shutdown_connections 2023-05-22T15:27:39.418 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:39.411+0000 7fd4bdcc3640 1 --2- 172.21.15.16:0/4248004087 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd4b81596e0 0x7fd4b806fba0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:39.419 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:39.411+0000 7fd4bdcc3640 1 --2- 172.21.15.16:0/4248004087 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd4b806b6c0 0x7fd4b80cb4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:39.419 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:39.411+0000 7fd4bdcc3640 1 -- 172.21.15.16:0/4248004087 >> 172.21.15.16:0/4248004087 conn(0x7fd4b8069720 msgr2=0x7fd4b8069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:39.419 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:39.411+0000 7fd4bdcc3640 1 -- 172.21.15.16:0/4248004087 shutdown_connections 2023-05-22T15:27:39.419 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:39.411+0000 7fd4bdcc3640 1 -- 172.21.15.16:0/4248004087 wait complete. 2023-05-22T15:27:39.420 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:39.411+0000 7fd4bdcc3640 1 Processor -- start 2023-05-22T15:27:39.420 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:39.411+0000 7fd4bdcc3640 1 -- start start 2023-05-22T15:27:39.420 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:39.411+0000 7fd4bdcc3640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd4b806b6c0 0x7fd4b814be00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:39.420 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:39.411+0000 7fd4bdcc3640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd4b806ba90 0x7fd4b80c1b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:39.421 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:39.411+0000 7fd4bdcc3640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd4b81596e0 0x7fd4b80af120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:39.421 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:39.411+0000 7fd4bdcc3640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4b80c36a0 con 0x7fd4b81596e0 2023-05-22T15:27:39.421 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:39.411+0000 7fd4bdcc3640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4b80c4ac0 con 0x7fd4b806ba90 2023-05-22T15:27:39.421 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:39.411+0000 7fd4bdcc3640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4b80c3210 con 0x7fd4b806b6c0 2023-05-22T15:27:39.421 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:39.411+0000 7fd4b7fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd4b81596e0 0x7fd4b80af120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:39.421 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:39.411+0000 7fd4b7fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd4b81596e0 0x7fd4b80af120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41416/0 (socket says 172.21.15.16:41416) 2023-05-22T15:27:39.421 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:39.411+0000 7fd4b7fff640 1 -- 172.21.15.16:0/906921769 learned_addr learned my addr 172.21.15.16:0/906921769 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:39.421 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:39.411+0000 7fd4b77fe640 1 --2- 172.21.15.16:0/906921769 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd4b806b6c0 0x7fd4b814be00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:39.422 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:39.411+0000 7fd4b6ffd640 1 --2- 172.21.15.16:0/906921769 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd4b806ba90 0x7fd4b80c1b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:39.423 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:39.411+0000 7fd4b7fff640 1 -- 172.21.15.16:0/906921769 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd4b806b6c0 msgr2=0x7fd4b814be00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:39.423 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:39.411+0000 7fd4b7fff640 1 --2- 172.21.15.16:0/906921769 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd4b806b6c0 0x7fd4b814be00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:39.423 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:39.411+0000 7fd4b7fff640 1 -- 172.21.15.16:0/906921769 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd4b806ba90 msgr2=0x7fd4b80c1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:39.423 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:39.411+0000 7fd4b7fff640 1 --2- 172.21.15.16:0/906921769 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd4b806ba90 0x7fd4b80c1b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:39.423 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:39.411+0000 7fd4b7fff640 1 -- 172.21.15.16:0/906921769 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4b0009b30 con 0x7fd4b81596e0 2023-05-22T15:27:39.424 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:39.411+0000 7fd4b7fff640 1 --2- 172.21.15.16:0/906921769 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd4b81596e0 0x7fd4b80af120 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fd4b001a580 tx=0x7fd4b00196f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:39.424 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:39.411+0000 7fd49ffff640 1 -- 172.21.15.16:0/906921769 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4b00b8640 con 0x7fd4b81596e0 2023-05-22T15:27:39.424 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:39.411+0000 7fd49ffff640 1 -- 172.21.15.16:0/906921769 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4b00b87e0 con 0x7fd4b81596e0 2023-05-22T15:27:39.424 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:39.411+0000 7fd4bdcc3640 1 -- 172.21.15.16:0/906921769 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4b806be60 con 0x7fd4b81596e0 2023-05-22T15:27:39.424 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:39.411+0000 7fd49ffff640 1 -- 172.21.15.16:0/906921769 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4b00b8b90 con 0x7fd4b81596e0 2023-05-22T15:27:39.424 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:39.415+0000 7fd4bdcc3640 1 -- 172.21.15.16:0/906921769 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4b80cacc0 con 0x7fd4b81596e0 2023-05-22T15:27:39.424 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:39.415+0000 7fd4bdcc3640 1 -- 172.21.15.16:0/906921769 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4b806bff0 con 0x7fd4b81596e0 2023-05-22T15:27:39.425 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:39.415+0000 7fd49ffff640 1 -- 172.21.15.16:0/906921769 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7fd4b00b8cf0 con 0x7fd4b81596e0 2023-05-22T15:27:39.425 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:39.415+0000 7fd49ffff640 1 --2- 172.21.15.16:0/906921769 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd4a4065380 0x7fd4a4067830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:39.425 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:39.415+0000 7fd49ffff640 1 -- 172.21.15.16:0/906921769 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6871+0+0 (secure 0 0 0) 0x7fd4b011e250 con 0x7fd4b81596e0 2023-05-22T15:27:39.425 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:39.415+0000 7fd4b77fe640 1 --2- 172.21.15.16:0/906921769 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd4a4065380 0x7fd4a4067830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:39.425 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:39.415+0000 7fd4b77fe640 1 --2- 172.21.15.16:0/906921769 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd4a4065380 0x7fd4a4067830 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fd4a8007a60 tx=0x7fd4a8008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:39.427 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:39.419+0000 7fd49ffff640 1 -- 172.21.15.16:0/906921769 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fd4b00ebc40 con 0x7fd4b81596e0 2023-05-22T15:27:39.582 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:39.575+0000 7fd4bdcc3640 1 -- 172.21.15.16:0/906921769 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fd4b804dbe0 con 0x7fd4b81596e0 2023-05-22T15:27:39.588 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:39.579+0000 7fd49ffff640 1 -- 172.21.15.16:0/906921769 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v10)=0 v10) v1 ==== 122+0+0 (secure 0 0 0) 0x7fd4b00ef260 con 0x7fd4b81596e0 2023-05-22T15:27:39.590 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:39.583+0000 7fd4bdcc3640 1 -- 172.21.15.16:0/906921769 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd4a4065380 msgr2=0x7fd4a4067830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:39.590 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:39.583+0000 7fd4bdcc3640 1 --2- 172.21.15.16:0/906921769 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd4a4065380 0x7fd4a4067830 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fd4a8007a60 tx=0x7fd4a8008040 comp rx=0 tx=0).stop 2023-05-22T15:27:39.590 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:39.583+0000 7fd4bdcc3640 1 -- 172.21.15.16:0/906921769 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd4b81596e0 msgr2=0x7fd4b80af120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:39.590 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:39.583+0000 7fd4bdcc3640 1 --2- 172.21.15.16:0/906921769 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd4b81596e0 0x7fd4b80af120 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fd4b001a580 tx=0x7fd4b00196f0 comp rx=0 tx=0).stop 2023-05-22T15:27:39.591 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:39.583+0000 7fd4bdcc3640 1 -- 172.21.15.16:0/906921769 shutdown_connections 2023-05-22T15:27:39.591 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:39.583+0000 7fd4bdcc3640 1 --2- 172.21.15.16:0/906921769 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd4b81596e0 0x7fd4b80af120 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:39.591 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:39.583+0000 7fd4bdcc3640 1 --2- 172.21.15.16:0/906921769 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd4b806ba90 0x7fd4b80c1b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:39.591 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:39.583+0000 7fd4bdcc3640 1 --2- 172.21.15.16:0/906921769 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd4a4065380 0x7fd4a4067830 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:39.591 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:39.583+0000 7fd4bdcc3640 1 --2- 172.21.15.16:0/906921769 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd4b806b6c0 0x7fd4b814be00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:39.591 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:39.583+0000 7fd4bdcc3640 1 -- 172.21.15.16:0/906921769 >> 172.21.15.16:0/906921769 conn(0x7fd4b8069720 msgr2=0x7fd4b80c94f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:39.592 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:39.583+0000 7fd4bdcc3640 1 -- 172.21.15.16:0/906921769 shutdown_connections 2023-05-22T15:27:39.592 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:39.583+0000 7fd4bdcc3640 1 -- 172.21.15.16:0/906921769 wait complete. 2023-05-22T15:27:39.607 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-05-22T15:27:39.608 DEBUG:teuthology.orchestra.run.smithi016:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-05-22T15:27:39.655 DEBUG:teuthology.orchestra.run.smithi062:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-05-22T15:27:39.714 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-05-22T15:27:39.715 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-05-22T15:27:39.715 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2023-05-22T15:27:39.722 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-05-22T15:27:39.723 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2023-05-22T15:27:39.770 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-05-22T15:27:39.935 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:39.927+0000 7f0ba16a0640 1 Processor -- start 2023-05-22T15:27:39.936 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:39.927+0000 7f0ba16a0640 1 -- start start 2023-05-22T15:27:39.936 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:39.927+0000 7f0ba16a0640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f0b9c0ccab0 0x7f0b9c15fe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:39.936 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:39.927+0000 7f0ba16a0640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b9c0ccef0 con 0x7f0b9c0d0410 2023-05-22T15:27:39.937 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:39.927+0000 7f0ba16a0640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b9c0cd170 con 0x7f0b9c0b7dd0 2023-05-22T15:27:39.937 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:39.927+0000 7f0ba16a0640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b9c0d3360 con 0x7f0b9c0ccab0 2023-05-22T15:27:39.937 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:39.927+0000 7f0b9a7fc640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f0b9c0ccab0 0x7f0b9c15fe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:39.937 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:39.927+0000 7f0b9b7fe640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f0b9c0d0410 0x7f0b9c0d0810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:45554/0 (socket says 172.21.15.16:45554) 2023-05-22T15:27:39.937 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:39.927+0000 7f0b9b7fe640 1 -- 172.21.15.16:0/1942721359 learned_addr learned my addr 172.21.15.16:0/1942721359 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:27:39.938 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:39.927+0000 7f0b9a7fc640 1 -- 172.21.15.16:0/1942721359 >> v1:172.21.15.62:6789/0 conn(0x7f0b9c0b7dd0 legacy=0x7f0b9c15f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:39.938 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:39.931+0000 7f0b99ffb640 1 -- 172.21.15.16:0/1942721359 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2228475056 0 0) 0x7f0b9c0ccef0 con 0x7f0b9c0d0410 2023-05-22T15:27:39.938 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:39.931+0000 7f0b9a7fc640 1 -- 172.21.15.16:0/1942721359 >> v1:172.21.15.16:6789/0 conn(0x7f0b9c0d0410 legacy=0x7f0b9c0d0810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:39.938 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:39.931+0000 7f0b9a7fc640 1 -- 172.21.15.16:0/1942721359 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b9c0d2ac0 con 0x7f0b9c0ccab0 2023-05-22T15:27:39.939 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:39.931+0000 7f0b9a7fc640 1 --2- 172.21.15.16:0/1942721359 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0b9c0ccab0 0x7f0b9c15fe10 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f0b8c0097c0 tx=0x7f0b8c01c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b30ff1be6b53216e server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:39.940 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:39.931+0000 7f0b99ffb640 1 -- 172.21.15.16:0/1942721359 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b8c0ab650 con 0x7f0b9c0ccab0 2023-05-22T15:27:39.940 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:39.931+0000 7f0b99ffb640 1 -- 172.21.15.16:0/1942721359 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b8c0ab7b0 con 0x7f0b9c0ccab0 2023-05-22T15:27:39.940 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:39.931+0000 7f0b99ffb640 1 -- 172.21.15.16:0/1942721359 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b8c0abb40 con 0x7f0b9c0ccab0 2023-05-22T15:27:39.941 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:39.931+0000 7f0ba16a0640 1 -- 172.21.15.16:0/1942721359 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0b9c0ccab0 msgr2=0x7f0b9c15fe10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:39.941 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:39.931+0000 7f0ba16a0640 1 --2- 172.21.15.16:0/1942721359 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0b9c0ccab0 0x7f0b9c15fe10 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f0b8c0097c0 tx=0x7f0b8c01c450 comp rx=0 tx=0).stop 2023-05-22T15:27:39.941 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:39.931+0000 7f0ba16a0640 1 -- 172.21.15.16:0/1942721359 shutdown_connections 2023-05-22T15:27:39.942 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:39.931+0000 7f0ba16a0640 1 --2- 172.21.15.16:0/1942721359 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0b9c0ccab0 0x7f0b9c15fe10 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:39.942 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:39.931+0000 7f0ba16a0640 1 -- 172.21.15.16:0/1942721359 >> 172.21.15.16:0/1942721359 conn(0x7f0b9c0655c0 msgr2=0x7f0b9c070e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:39.942 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:39.931+0000 7f0ba16a0640 1 -- 172.21.15.16:0/1942721359 shutdown_connections 2023-05-22T15:27:39.942 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:39.931+0000 7f0ba16a0640 1 -- 172.21.15.16:0/1942721359 wait complete. 2023-05-22T15:27:39.942 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:39.935+0000 7f0ba16a0640 1 Processor -- start 2023-05-22T15:27:39.943 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:39.935+0000 7f0ba16a0640 1 -- start start 2023-05-22T15:27:39.944 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:39.935+0000 7f0ba16a0640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0b9c0b7dd0 0x7f0b9c153b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:39.944 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:39.935+0000 7f0ba16a0640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0b9c0ccab0 0x7f0b9c155d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:39.944 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:39.935+0000 7f0ba16a0640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0b9c0d0410 0x7f0b9c158550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:39.944 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:39.935+0000 7f0ba16a0640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b9c1558f0 con 0x7f0b9c0ccab0 2023-05-22T15:27:39.944 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:39.935+0000 7f0ba16a0640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b9c154040 con 0x7f0b9c0d0410 2023-05-22T15:27:39.944 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:39.935+0000 7f0ba16a0640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b9c154ce0 con 0x7f0b9c0b7dd0 2023-05-22T15:27:39.945 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:39.935+0000 7f0b9a7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0b9c0ccab0 0x7f0b9c155d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:39.945 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:39.935+0000 7f0b9a7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0b9c0ccab0 0x7f0b9c155d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41446/0 (socket says 172.21.15.16:41446) 2023-05-22T15:27:39.945 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:39.935+0000 7f0b9a7fc640 1 -- 172.21.15.16:0/146647325 learned_addr learned my addr 172.21.15.16:0/146647325 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:39.945 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:39.935+0000 7f0b9b7fe640 1 --2- 172.21.15.16:0/146647325 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0b9c0d0410 0x7f0b9c158550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:39.945 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:39.935+0000 7f0b9affd640 1 --2- 172.21.15.16:0/146647325 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0b9c0b7dd0 0x7f0b9c153b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:39.945 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:39.935+0000 7f0b9a7fc640 1 -- 172.21.15.16:0/146647325 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0b9c0b7dd0 msgr2=0x7f0b9c153b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:39.945 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:39.935+0000 7f0b9a7fc640 1 --2- 172.21.15.16:0/146647325 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0b9c0b7dd0 0x7f0b9c153b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:39.946 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:39.935+0000 7f0b9a7fc640 1 -- 172.21.15.16:0/146647325 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0b9c0d0410 msgr2=0x7f0b9c158550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:39.946 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:39.935+0000 7f0b9a7fc640 1 --2- 172.21.15.16:0/146647325 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0b9c0d0410 0x7f0b9c158550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:39.946 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:39.935+0000 7f0b9a7fc640 1 -- 172.21.15.16:0/146647325 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b8c009420 con 0x7f0b9c0ccab0 2023-05-22T15:27:39.947 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:39.935+0000 7f0b9a7fc640 1 --2- 172.21.15.16:0/146647325 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0b9c0ccab0 0x7f0b9c155d80 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f0b8c019060 tx=0x7f0b8c01ac90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:39.947 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:39.935+0000 7f0b837fe640 1 -- 172.21.15.16:0/146647325 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b8c0b5810 con 0x7f0b9c0ccab0 2023-05-22T15:27:39.947 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:39.935+0000 7f0b837fe640 1 -- 172.21.15.16:0/146647325 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b8c0b5990 con 0x7f0b9c0ccab0 2023-05-22T15:27:39.947 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:39.935+0000 7f0b837fe640 1 -- 172.21.15.16:0/146647325 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b8c0b5d40 con 0x7f0b9c0ccab0 2023-05-22T15:27:39.947 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:39.935+0000 7f0ba16a0640 1 -- 172.21.15.16:0/146647325 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b9c160350 con 0x7f0b9c0ccab0 2023-05-22T15:27:39.948 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:39.935+0000 7f0ba16a0640 1 -- 172.21.15.16:0/146647325 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b9c0cf730 con 0x7f0b9c0ccab0 2023-05-22T15:27:39.948 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:39.939+0000 7f0ba16a0640 1 -- 172.21.15.16:0/146647325 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b9c1604e0 con 0x7f0b9c0ccab0 2023-05-22T15:27:39.948 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:39.939+0000 7f0b837fe640 1 -- 172.21.15.16:0/146647325 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7f0b8c0b4390 con 0x7f0b9c0ccab0 2023-05-22T15:27:39.948 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:39.939+0000 7f0b837fe640 1 --2- 172.21.15.16:0/146647325 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f0b880653b0 0x7f0b88067860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:39.948 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:39.939+0000 7f0b837fe640 1 -- 172.21.15.16:0/146647325 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f0b8c11d610 con 0x7f0b9c0ccab0 2023-05-22T15:27:39.948 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:39.939+0000 7f0b9affd640 1 --2- 172.21.15.16:0/146647325 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f0b880653b0 0x7f0b88067860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:39.948 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:39.939+0000 7f0b9affd640 1 --2- 172.21.15.16:0/146647325 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f0b880653b0 0x7f0b88067860 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f0b9c0b7360 tx=0x7f0b90009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:39.950 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:39.943+0000 7f0b837fe640 1 -- 172.21.15.16:0/146647325 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f0b8c0eaef0 con 0x7f0b9c0ccab0 2023-05-22T15:27:40.104 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.095+0000 7f0ba16a0640 1 -- 172.21.15.16:0/146647325 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0b9c155aa0 con 0x7f0b9c0ccab0 2023-05-22T15:27:40.105 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.099+0000 7f0b837fe640 1 -- 172.21.15.16:0/146647325 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v29) v1 ==== 74+0+19700 (secure 0 0 0) 0x7f0b9c155aa0 con 0x7f0b9c0ccab0 2023-05-22T15:27:40.105 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:27:40.106 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":29,"fsid":"b2719834-7847-4d98-90a6-f65d57f85c49","created":"2023-05-22T15:26:25.259314+0000","modified":"2023-05-22T15:27:31.182317+0000","last_up_change":"2023-05-22T15:26:36.448524+0000","last_in_change":"2023-05-22T15:26:27.675668+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-05-22T15:26:38.019781+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-05-22T15:26:43.570514+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-05-22T15:26:47.879159+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-05-22T15:26:48.522130+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data_ec","create_time":"2023-05-22T15:26:51.616142+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"9e7ff854-b25f-4cbb-a5c2-1799d9c53d2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":37021},{"type":"v1","addr":"172.21.15.16:6825","nonce":37021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":37021},{"type":"v1","addr":"172.21.15.16:6827","nonce":37021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":37021},{"type":"v1","addr":"172.21.15.16:6831","nonce":37021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":37021},{"type":"v1","addr":"172.21.15.16:6829","nonce":37021}]},"public_addr":"172.21.15.16:6825/37021","cluster_addr":"172.21.15.16:6827/37021","heartbeat_back_addr":"172.21.15.16:6831/37021","heartbeat_front_addr":"172.21.15.16:6829/37021","state":["exists","up"]},{"osd":1,"uuid":"4702fbc9-a927-4565-aca5-41dc69cc8264","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":37008},{"type":"v1","addr":"172.21.15.16:6801","nonce":37008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":37008},{"type":"v1","addr":"172.21.15.16:6803","nonce":37008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":37008},{"type":"v1","addr":"172.21.15.16:6807","nonce":37008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":37008},{"type":"v1","addr":"172.21.15.16:6805","nonce":37008}]},"public_addr":"172.21.15.16:6801/37008","cluster_addr":"172.21.15.16:6803/37008","heartbeat_back_addr":"172.21.15.16:6807/37008","heartbeat_front_addr":"172.21.15.16:6805/37008","state":["exists","up"]},{"osd":2,"uuid":"1e7bba46-4eba-47ef-bb81-5e2d371638ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":37009},{"type":"v1","addr":"172.21.15.16:6809","nonce":37009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":37009},{"type":"v1","addr":"172.21.15.16:6811","nonce":37009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":37009},{"type":"v1","addr":"172.21.15.16:6815","nonce":37009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":37009},{"type":"v1","addr":"172.21.15.16:6813","nonce":37009}]},"public_addr":"172.21.15.16:6809/37009","cluster_addr":"172.21.15.16:6811/37009","heartbeat_back_addr":"172.21.15.16:6815/37009","heartbeat_front_addr":"172.21.15.16:6813/37009","state":["exists","up"]},{"osd":3,"uuid":"383cacc0-5a88-4904-990e-498ca59ef48a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":37020},{"type":"v1","addr":"172.21.15.16:6817","nonce":37020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":37020},{"type":"v1","addr":"172.21.15.16:6819","nonce":37020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":37020},{"type":"v1","addr":"172.21.15.16:6823","nonce":37020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":37020},{"type":"v1","addr":"172.21.15.16:6821","nonce":37020}]},"public_addr":"172.21.15.16:6817/37020","cluster_addr":"172.21.15.16:6819/37020","heartbeat_back_addr":"172.21.15.16:6823/37020","heartbeat_front_addr":"172.21.15.16:6821/37020","state":["exists","up"]},{"osd":4,"uuid":"970500ef-95b7-42c1-b080-eed4fae7013d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":37353},{"type":"v1","addr":"172.21.15.62:6817","nonce":37353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6819","nonce":37353},{"type":"v1","addr":"172.21.15.62:6822","nonce":37353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37353},{"type":"v1","addr":"172.21.15.62:6831","nonce":37353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37353},{"type":"v1","addr":"172.21.15.62:6826","nonce":37353}]},"public_addr":"172.21.15.62:6817/37353","cluster_addr":"172.21.15.62:6822/37353","heartbeat_back_addr":"172.21.15.62:6831/37353","heartbeat_front_addr":"172.21.15.62:6826/37353","state":["exists","up"]},{"osd":5,"uuid":"022d67db-d4d8-445b-b1b9-8b446a00f5c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":37352},{"type":"v1","addr":"172.21.15.62:6801","nonce":37352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":37352},{"type":"v1","addr":"172.21.15.62:6803","nonce":37352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":37352},{"type":"v1","addr":"172.21.15.62:6807","nonce":37352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":37352},{"type":"v1","addr":"172.21.15.62:6805","nonce":37352}]},"public_addr":"172.21.15.62:6801/37352","cluster_addr":"172.21.15.62:6803/37352","heartbeat_back_addr":"172.21.15.62:6807/37352","heartbeat_front_addr":"172.21.15.62:6805/37352","state":["exists","up"]},{"osd":6,"uuid":"42bd372b-6688-4a1b-8407-e16e8c266e33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":37354},{"type":"v1","addr":"172.21.15.62:6820","nonce":37354}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":37354},{"type":"v1","addr":"172.21.15.62:6823","nonce":37354}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6829","nonce":37354},{"type":"v1","addr":"172.21.15.62:6830","nonce":37354}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6825","nonce":37354},{"type":"v1","addr":"172.21.15.62:6827","nonce":37354}]},"public_addr":"172.21.15.62:6820/37354","cluster_addr":"172.21.15.62:6823/37354","heartbeat_back_addr":"172.21.15.62:6830/37354","heartbeat_front_addr":"172.21.15.62:6827/37354","state":["exists","up"]},{"osd":7,"uuid":"ac70b2a3-f41b-4bf2-b227-7f2cbb55426d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37355},{"type":"v1","addr":"172.21.15.62:6809","nonce":37355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37355},{"type":"v1","addr":"172.21.15.62:6811","nonce":37355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":37355},{"type":"v1","addr":"172.21.15.62:6815","nonce":37355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":37355},{"type":"v1","addr":"172.21.15.62:6813","nonce":37355}]},"public_addr":"172.21.15.62:6809/37355","cluster_addr":"172.21.15.62:6811/37355","heartbeat_back_addr":"172.21.15.62:6815/37355","heartbeat_front_addr":"172.21.15.62:6813/37355","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.368896+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.107273+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.276888+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.287502+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.124791+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.251937+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.255679+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.232797+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"4.23","mappings":[{"from":3,"to":2}]},{"pgid":"4.31","mappings":[{"from":1,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-05-22T15:27:40.107 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.099+0000 7f0ba16a0640 1 -- 172.21.15.16:0/146647325 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f0b880653b0 msgr2=0x7f0b88067860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:40.108 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.099+0000 7f0ba16a0640 1 --2- 172.21.15.16:0/146647325 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f0b880653b0 0x7f0b88067860 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f0b9c0b7360 tx=0x7f0b90009040 comp rx=0 tx=0).stop 2023-05-22T15:27:40.108 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.099+0000 7f0ba16a0640 1 -- 172.21.15.16:0/146647325 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0b9c0ccab0 msgr2=0x7f0b9c155d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:40.108 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.099+0000 7f0ba16a0640 1 --2- 172.21.15.16:0/146647325 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0b9c0ccab0 0x7f0b9c155d80 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f0b8c019060 tx=0x7f0b8c01ac90 comp rx=0 tx=0).stop 2023-05-22T15:27:40.108 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.099+0000 7f0ba16a0640 1 -- 172.21.15.16:0/146647325 shutdown_connections 2023-05-22T15:27:40.108 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.099+0000 7f0ba16a0640 1 --2- 172.21.15.16:0/146647325 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0b9c0d0410 0x7f0b9c158550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:40.108 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.099+0000 7f0ba16a0640 1 --2- 172.21.15.16:0/146647325 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0b9c0ccab0 0x7f0b9c155d80 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:40.109 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.099+0000 7f0ba16a0640 1 --2- 172.21.15.16:0/146647325 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f0b880653b0 0x7f0b88067860 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:40.109 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.099+0000 7f0ba16a0640 1 --2- 172.21.15.16:0/146647325 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0b9c0b7dd0 0x7f0b9c153b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:40.109 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.099+0000 7f0ba16a0640 1 -- 172.21.15.16:0/146647325 >> 172.21.15.16:0/146647325 conn(0x7f0b9c0655c0 msgr2=0x7f0b9c0b5290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:40.109 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.099+0000 7f0ba16a0640 1 -- 172.21.15.16:0/146647325 shutdown_connections 2023-05-22T15:27:40.109 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.099+0000 7f0ba16a0640 1 -- 172.21.15.16:0/146647325 wait complete. 2023-05-22T15:27:40.123 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-05-22T15:26:38.019781+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-05-22T15:26:43.570514+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2023-05-22T15:26:47.879159+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2023-05-22T15:26:48.522130+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-05-22T15:26:51.616142+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '27', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-05-22T15:27:40.124 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-05-22T15:27:40.243 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.235+0000 7fca1e6f2640 1 Processor -- start 2023-05-22T15:27:40.244 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.235+0000 7fca1e6f2640 1 -- start start 2023-05-22T15:27:40.244 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.235+0000 7fca1e6f2640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fca18159d40 0x7fca180cb260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:40.245 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.235+0000 7fca1e6f2640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fca1806b6c0 0x7fca1806bb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:40.245 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.235+0000 7fca1e6f2640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca18070c30 con 0x7fca18159970 2023-05-22T15:27:40.245 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.235+0000 7fca1e6f2640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca1815dd90 con 0x7fca18159d40 2023-05-22T15:27:40.245 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.235+0000 7fca1e6f2640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca1806c040 con 0x7fca1806b6c0 2023-05-22T15:27:40.245 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.235+0000 7fca177fe640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fca18159d40 0x7fca180cb260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:40.245 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.235+0000 7fca177fe640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fca18159d40 0x7fca180cb260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41450/0 (socket says 172.21.15.16:41450) 2023-05-22T15:27:40.246 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.235+0000 7fca177fe640 1 -- 172.21.15.16:0/744375684 learned_addr learned my addr 172.21.15.16:0/744375684 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:40.246 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.239+0000 7fca1cc8f640 1 --2- 172.21.15.16:0/744375684 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fca1806b6c0 0x7fca1806bb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:40.247 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.239+0000 7fca177fe640 1 -- 172.21.15.16:0/744375684 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fca1806b6c0 msgr2=0x7fca1806bb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:40.247 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.239+0000 7fca177fe640 1 --2- 172.21.15.16:0/744375684 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fca1806b6c0 0x7fca1806bb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:40.247 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.239+0000 7fca177fe640 1 -- 172.21.15.16:0/744375684 >> v1:172.21.15.62:6789/0 conn(0x7fca18159970 legacy=0x7fca1815d680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:40.247 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.239+0000 7fca177fe640 1 -- 172.21.15.16:0/744375684 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca18070e60 con 0x7fca18159d40 2023-05-22T15:27:40.248 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.239+0000 7fca177fe640 1 --2- 172.21.15.16:0/744375684 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fca18159d40 0x7fca180cb260 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fca0c009870 tx=0x7fca0c01c4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a2a6c4365aa8490 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:40.248 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.239+0000 7fca16ffd640 1 -- 172.21.15.16:0/744375684 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca0c0ac600 con 0x7fca18159d40 2023-05-22T15:27:40.248 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.239+0000 7fca16ffd640 1 -- 172.21.15.16:0/744375684 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca0c0a8050 con 0x7fca18159d40 2023-05-22T15:27:40.248 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.239+0000 7fca16ffd640 1 -- 172.21.15.16:0/744375684 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca0c0ac980 con 0x7fca18159d40 2023-05-22T15:27:40.249 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.239+0000 7fca1e6f2640 1 -- 172.21.15.16:0/744375684 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fca18159d40 msgr2=0x7fca180cb260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:40.249 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.239+0000 7fca1e6f2640 1 --2- 172.21.15.16:0/744375684 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fca18159d40 0x7fca180cb260 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fca0c009870 tx=0x7fca0c01c4b0 comp rx=0 tx=0).stop 2023-05-22T15:27:40.249 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.239+0000 7fca1e6f2640 1 -- 172.21.15.16:0/744375684 shutdown_connections 2023-05-22T15:27:40.249 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.239+0000 7fca1e6f2640 1 --2- 172.21.15.16:0/744375684 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fca1806b6c0 0x7fca1806bb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:40.249 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.239+0000 7fca1e6f2640 1 --2- 172.21.15.16:0/744375684 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fca18159d40 0x7fca180cb260 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:40.249 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.239+0000 7fca1e6f2640 1 -- 172.21.15.16:0/744375684 >> 172.21.15.16:0/744375684 conn(0x7fca18069720 msgr2=0x7fca18069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:40.249 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.239+0000 7fca1e6f2640 1 -- 172.21.15.16:0/744375684 shutdown_connections 2023-05-22T15:27:40.249 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.239+0000 7fca1e6f2640 1 -- 172.21.15.16:0/744375684 wait complete. 2023-05-22T15:27:40.251 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.243+0000 7fca1e6f2640 1 Processor -- start 2023-05-22T15:27:40.251 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.243+0000 7fca1e6f2640 1 -- start start 2023-05-22T15:27:40.251 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.243+0000 7fca1e6f2640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fca1806b6c0 0x7fca1814e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:40.251 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.243+0000 7fca1e6f2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fca18159970 0x7fca180ada90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:40.251 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.243+0000 7fca1e6f2640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fca18159d40 0x7fca1815c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:40.251 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.243+0000 7fca1e6f2640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca180cbdc0 con 0x7fca18159970 2023-05-22T15:27:40.252 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.243+0000 7fca1e6f2640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca180aeed0 con 0x7fca1806b6c0 2023-05-22T15:27:40.252 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.243+0000 7fca1e6f2640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca180ae2c0 con 0x7fca18159d40 2023-05-22T15:27:40.252 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.243+0000 7fca177fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fca18159970 0x7fca180ada90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:40.252 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.243+0000 7fca177fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fca18159970 0x7fca180ada90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41456/0 (socket says 172.21.15.16:41456) 2023-05-22T15:27:40.253 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.243+0000 7fca177fe640 1 -- 172.21.15.16:0/2071234518 learned_addr learned my addr 172.21.15.16:0/2071234518 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:40.253 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.243+0000 7fca17fff640 1 --2- 172.21.15.16:0/2071234518 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fca1806b6c0 0x7fca1814e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:40.253 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.243+0000 7fca1cc8f640 1 --2- 172.21.15.16:0/2071234518 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fca18159d40 0x7fca1815c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:40.253 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.243+0000 7fca177fe640 1 -- 172.21.15.16:0/2071234518 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fca18159d40 msgr2=0x7fca1815c510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:40.253 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.243+0000 7fca177fe640 1 --2- 172.21.15.16:0/2071234518 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fca18159d40 0x7fca1815c510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:40.253 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.243+0000 7fca177fe640 1 -- 172.21.15.16:0/2071234518 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fca1806b6c0 msgr2=0x7fca1814e050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:40.254 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.243+0000 7fca177fe640 1 --2- 172.21.15.16:0/2071234518 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fca1806b6c0 0x7fca1814e050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:40.254 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.243+0000 7fca177fe640 1 -- 172.21.15.16:0/2071234518 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca0c0094d0 con 0x7fca18159970 2023-05-22T15:27:40.254 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.243+0000 7fca177fe640 1 --2- 172.21.15.16:0/2071234518 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fca18159970 0x7fca180ada90 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fca0c01ab60 tx=0x7fca0c019250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:40.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.243+0000 7fca14ff9640 1 -- 172.21.15.16:0/2071234518 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca0c0b5830 con 0x7fca18159970 2023-05-22T15:27:40.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.243+0000 7fca14ff9640 1 -- 172.21.15.16:0/2071234518 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca0c0b5990 con 0x7fca18159970 2023-05-22T15:27:40.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.243+0000 7fca14ff9640 1 -- 172.21.15.16:0/2071234518 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca0c0b5d20 con 0x7fca18159970 2023-05-22T15:27:40.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.243+0000 7fca1e6f2640 1 -- 172.21.15.16:0/2071234518 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca180cb7a0 con 0x7fca18159970 2023-05-22T15:27:40.256 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.243+0000 7fca1e6f2640 1 -- 172.21.15.16:0/2071234518 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca18071700 con 0x7fca18159970 2023-05-22T15:27:40.256 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.243+0000 7fca1e6f2640 1 -- 172.21.15.16:0/2071234518 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca180cb930 con 0x7fca18159970 2023-05-22T15:27:40.256 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.247+0000 7fca14ff9640 1 -- 172.21.15.16:0/2071234518 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7fca0c0b3070 con 0x7fca18159970 2023-05-22T15:27:40.256 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.247+0000 7fca14ff9640 1 --2- 172.21.15.16:0/2071234518 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fca04065380 0x7fca04067830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:40.256 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.247+0000 7fca14ff9640 1 -- 172.21.15.16:0/2071234518 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6871+0+0 (secure 0 0 0) 0x7fca0c11d080 con 0x7fca18159970 2023-05-22T15:27:40.257 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.247+0000 7fca17fff640 1 --2- 172.21.15.16:0/2071234518 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fca04065380 0x7fca04067830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:40.257 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.247+0000 7fca17fff640 1 --2- 172.21.15.16:0/2071234518 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fca04065380 0x7fca04067830 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fca181647b0 tx=0x7fca08009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:40.260 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.251+0000 7fca14ff9640 1 -- 172.21.15.16:0/2071234518 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fca0c0ee400 con 0x7fca18159970 2023-05-22T15:27:40.414 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.407+0000 7fca1e6f2640 1 -- 172.21.15.16:0/2071234518 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fca180af080 con 0x7fca18159970 2023-05-22T15:27:40.414 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.407+0000 7fca14ff9640 1 -- 172.21.15.16:0/2071234518 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v29) v1 ==== 93+0+10 (secure 0 0 0) 0x7fca0c0edff0 con 0x7fca18159970 2023-05-22T15:27:40.415 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2023-05-22T15:27:40.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.407+0000 7fca1e6f2640 1 -- 172.21.15.16:0/2071234518 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fca04065380 msgr2=0x7fca04067830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:40.417 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.407+0000 7fca1e6f2640 1 --2- 172.21.15.16:0/2071234518 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fca04065380 0x7fca04067830 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fca181647b0 tx=0x7fca08009040 comp rx=0 tx=0).stop 2023-05-22T15:27:40.417 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.407+0000 7fca1e6f2640 1 -- 172.21.15.16:0/2071234518 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fca18159970 msgr2=0x7fca180ada90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:40.417 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.407+0000 7fca1e6f2640 1 --2- 172.21.15.16:0/2071234518 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fca18159970 0x7fca180ada90 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fca0c01ab60 tx=0x7fca0c019250 comp rx=0 tx=0).stop 2023-05-22T15:27:40.417 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.407+0000 7fca1e6f2640 1 -- 172.21.15.16:0/2071234518 shutdown_connections 2023-05-22T15:27:40.417 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.407+0000 7fca1e6f2640 1 --2- 172.21.15.16:0/2071234518 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fca04065380 0x7fca04067830 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:40.417 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.407+0000 7fca1e6f2640 1 --2- 172.21.15.16:0/2071234518 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fca18159d40 0x7fca1815c510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:40.417 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.407+0000 7fca1e6f2640 1 --2- 172.21.15.16:0/2071234518 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fca18159970 0x7fca180ada90 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:40.417 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.407+0000 7fca1e6f2640 1 --2- 172.21.15.16:0/2071234518 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fca1806b6c0 0x7fca1814e050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:40.418 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.407+0000 7fca1e6f2640 1 -- 172.21.15.16:0/2071234518 >> 172.21.15.16:0/2071234518 conn(0x7fca18069720 msgr2=0x7fca180c94f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:40.418 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.407+0000 7fca1e6f2640 1 -- 172.21.15.16:0/2071234518 shutdown_connections 2023-05-22T15:27:40.418 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.407+0000 7fca1e6f2640 1 -- 172.21.15.16:0/2071234518 wait complete. 2023-05-22T15:27:40.433 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-05-22T15:27:40.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.543+0000 7f3648ed5640 1 Processor -- start 2023-05-22T15:27:40.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.543+0000 7f3648ed5640 1 -- start start 2023-05-22T15:27:40.554 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.547+0000 7f3648ed5640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f36440d2480 0x7f36440d2900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:40.554 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.547+0000 7f3648ed5640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36440cccc0 con 0x7f36440b5340 2023-05-22T15:27:40.554 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.547+0000 7f3648ed5640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36440ccfa0 con 0x7f364415bca0 2023-05-22T15:27:40.555 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.547+0000 7f3648ed5640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36440cd660 con 0x7f36440d2480 2023-05-22T15:27:40.555 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.547+0000 7f3642d9d640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f36440b5340 0x7f36440b5790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:45568/0 (socket says 172.21.15.16:45568) 2023-05-22T15:27:40.555 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.547+0000 7f3642d9d640 1 -- 172.21.15.16:0/2304940300 learned_addr learned my addr 172.21.15.16:0/2304940300 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:27:40.555 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.547+0000 7f3639d9b640 1 --2- 172.21.15.16:0/2304940300 >> v2:172.21.15.16:3300/0 conn(0x7f36440d2480 0x7f36440d2900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:40.556 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.547+0000 7f3641d9b640 1 -- 172.21.15.16:0/2304940300 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1222112393 0 0) 0x7f36440cccc0 con 0x7f36440b5340 2023-05-22T15:27:40.556 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.547+0000 7f3641d9b640 1 -- 172.21.15.16:0/2304940300 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3630003530 con 0x7f36440b5340 2023-05-22T15:27:40.556 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.547+0000 7f3639d9b640 1 -- 172.21.15.16:0/2304940300 >> v1:172.21.15.62:6789/0 conn(0x7f364415bca0 legacy=0x7f36440cda80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:40.556 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.547+0000 7f3639d9b640 1 -- 172.21.15.16:0/2304940300 >> v1:172.21.15.16:6789/0 conn(0x7f36440b5340 legacy=0x7f36440b5790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:40.556 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.547+0000 7f3639d9b640 1 -- 172.21.15.16:0/2304940300 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36440cd280 con 0x7f36440d2480 2023-05-22T15:27:40.557 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.547+0000 7f3639d9b640 1 --2- 172.21.15.16:0/2304940300 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f36440d2480 0x7f36440d2900 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f362c0097c0 tx=0x7f362c01c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=374c59090da2139f server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:40.558 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.547+0000 7f3641d9b640 1 -- 172.21.15.16:0/2304940300 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f362c0a8050 con 0x7f36440d2480 2023-05-22T15:27:40.558 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.547+0000 7f3641d9b640 1 -- 172.21.15.16:0/2304940300 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f362c0ac5f0 con 0x7f36440d2480 2023-05-22T15:27:40.558 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.547+0000 7f3641d9b640 1 -- 172.21.15.16:0/2304940300 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f362c0ac960 con 0x7f36440d2480 2023-05-22T15:27:40.558 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.551+0000 7f3648ed5640 1 -- 172.21.15.16:0/2304940300 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f36440d2480 msgr2=0x7f36440d2900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:40.559 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.551+0000 7f3648ed5640 1 --2- 172.21.15.16:0/2304940300 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f36440d2480 0x7f36440d2900 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f362c0097c0 tx=0x7f362c01c480 comp rx=0 tx=0).stop 2023-05-22T15:27:40.559 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.551+0000 7f3648ed5640 1 -- 172.21.15.16:0/2304940300 shutdown_connections 2023-05-22T15:27:40.559 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.551+0000 7f3648ed5640 1 --2- 172.21.15.16:0/2304940300 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f36440d2480 0x7f36440d2900 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:40.559 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.551+0000 7f3648ed5640 1 -- 172.21.15.16:0/2304940300 >> 172.21.15.16:0/2304940300 conn(0x7f36440655c0 msgr2=0x7f3644070e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:40.559 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.551+0000 7f3648ed5640 1 -- 172.21.15.16:0/2304940300 shutdown_connections 2023-05-22T15:27:40.559 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.551+0000 7f3648ed5640 1 -- 172.21.15.16:0/2304940300 wait complete. 2023-05-22T15:27:40.561 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.551+0000 7f3648ed5640 1 Processor -- start 2023-05-22T15:27:40.561 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.551+0000 7f3648ed5640 1 -- start start 2023-05-22T15:27:40.561 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.551+0000 7f3648ed5640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f36440b5340 0x7f3644149270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:40.561 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.551+0000 7f3648ed5640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f36440d2480 0x7f36440c5ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:40.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.551+0000 7f3648ed5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f364415bca0 0x7f36441565a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:40.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.551+0000 7f3648ed5640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36440ce190 con 0x7f364415bca0 2023-05-22T15:27:40.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.551+0000 7f3648ed5640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36441497b0 con 0x7f36440d2480 2023-05-22T15:27:40.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.551+0000 7f3648ed5640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3644150470 con 0x7f36440b5340 2023-05-22T15:27:40.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.551+0000 7f3642d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f364415bca0 0x7f36441565a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:40.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.551+0000 7f3642d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f364415bca0 0x7f36441565a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41468/0 (socket says 172.21.15.16:41468) 2023-05-22T15:27:40.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.551+0000 7f3642d9d640 1 -- 172.21.15.16:0/2878393365 learned_addr learned my addr 172.21.15.16:0/2878393365 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:40.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.551+0000 7f364259c640 1 --2- 172.21.15.16:0/2878393365 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f36440b5340 0x7f3644149270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:40.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.551+0000 7f3639d9b640 1 --2- 172.21.15.16:0/2878393365 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f36440d2480 0x7f36440c5ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:40.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.551+0000 7f3642d9d640 1 -- 172.21.15.16:0/2878393365 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f36440b5340 msgr2=0x7f3644149270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:40.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.551+0000 7f3642d9d640 1 --2- 172.21.15.16:0/2878393365 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f36440b5340 0x7f3644149270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:40.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.551+0000 7f3642d9d640 1 -- 172.21.15.16:0/2878393365 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f36440d2480 msgr2=0x7f36440c5ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:40.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.551+0000 7f3642d9d640 1 --2- 172.21.15.16:0/2878393365 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f36440d2480 0x7f36440c5ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:40.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.551+0000 7f3642d9d640 1 -- 172.21.15.16:0/2878393365 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f362c009420 con 0x7f364415bca0 2023-05-22T15:27:40.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.551+0000 7f3642d9d640 1 --2- 172.21.15.16:0/2878393365 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f364415bca0 0x7f36441565a0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f363c00cb70 tx=0x7f363c00a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:40.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.551+0000 7f363b7fe640 1 -- 172.21.15.16:0/2878393365 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f363c00ad40 con 0x7f364415bca0 2023-05-22T15:27:40.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.551+0000 7f363b7fe640 1 -- 172.21.15.16:0/2878393365 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f363c012400 con 0x7f364415bca0 2023-05-22T15:27:40.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.551+0000 7f3648ed5640 1 -- 172.21.15.16:0/2878393365 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36441604f0 con 0x7f364415bca0 2023-05-22T15:27:40.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.551+0000 7f363b7fe640 1 -- 172.21.15.16:0/2878393365 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f363c00aea0 con 0x7f364415bca0 2023-05-22T15:27:40.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.551+0000 7f3648ed5640 1 -- 172.21.15.16:0/2878393365 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3644160360 con 0x7f364415bca0 2023-05-22T15:27:40.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.555+0000 7f3648ed5640 1 -- 172.21.15.16:0/2878393365 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3644169c60 con 0x7f364415bca0 2023-05-22T15:27:40.565 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.555+0000 7f363b7fe640 1 -- 172.21.15.16:0/2878393365 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7f363c007070 con 0x7f364415bca0 2023-05-22T15:27:40.565 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.555+0000 7f363b7fe640 1 --2- 172.21.15.16:0/2878393365 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f36240653b0 0x7f3624067860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:40.565 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.555+0000 7f363b7fe640 1 -- 172.21.15.16:0/2878393365 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f363c082c80 con 0x7f364415bca0 2023-05-22T15:27:40.565 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.555+0000 7f364259c640 1 --2- 172.21.15.16:0/2878393365 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f36240653b0 0x7f3624067860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:40.566 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.555+0000 7f364259c640 1 --2- 172.21.15.16:0/2878393365 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f36240653b0 0x7f3624067860 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f36340015b0 tx=0x7f3634009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:40.566 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.559+0000 7f363b7fe640 1 -- 172.21.15.16:0/2878393365 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f363c0504e0 con 0x7f364415bca0 2023-05-22T15:27:40.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.711+0000 7f3648ed5640 1 -- 172.21.15.16:0/2878393365 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f364404dbe0 con 0x7f364415bca0 2023-05-22T15:27:40.719 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.711+0000 7f363b7fe640 1 -- 172.21.15.16:0/2878393365 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v29) v1 ==== 92+0+10 (secure 0 0 0) 0x7f363c017070 con 0x7f364415bca0 2023-05-22T15:27:40.719 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 8 2023-05-22T15:27:40.721 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.711+0000 7f3648ed5640 1 -- 172.21.15.16:0/2878393365 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f36240653b0 msgr2=0x7f3624067860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:40.721 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.711+0000 7f3648ed5640 1 --2- 172.21.15.16:0/2878393365 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f36240653b0 0x7f3624067860 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f36340015b0 tx=0x7f3634009040 comp rx=0 tx=0).stop 2023-05-22T15:27:40.721 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.715+0000 7f3648ed5640 1 -- 172.21.15.16:0/2878393365 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f364415bca0 msgr2=0x7f36441565a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:40.721 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.715+0000 7f3648ed5640 1 --2- 172.21.15.16:0/2878393365 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f364415bca0 0x7f36441565a0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f363c00cb70 tx=0x7f363c00a430 comp rx=0 tx=0).stop 2023-05-22T15:27:40.722 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.715+0000 7f3648ed5640 1 -- 172.21.15.16:0/2878393365 shutdown_connections 2023-05-22T15:27:40.722 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.715+0000 7f3648ed5640 1 --2- 172.21.15.16:0/2878393365 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f364415bca0 0x7f36441565a0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:40.722 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.715+0000 7f3648ed5640 1 --2- 172.21.15.16:0/2878393365 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f36440d2480 0x7f36440c5ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:40.722 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.715+0000 7f3648ed5640 1 --2- 172.21.15.16:0/2878393365 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f36240653b0 0x7f3624067860 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:40.722 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.715+0000 7f3648ed5640 1 --2- 172.21.15.16:0/2878393365 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f36440b5340 0x7f3644149270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:40.722 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.715+0000 7f3648ed5640 1 -- 172.21.15.16:0/2878393365 >> 172.21.15.16:0/2878393365 conn(0x7f36440655c0 msgr2=0x7f36440b5790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:40.722 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.715+0000 7f3648ed5640 1 -- 172.21.15.16:0/2878393365 shutdown_connections 2023-05-22T15:27:40.723 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.715+0000 7f3648ed5640 1 -- 172.21.15.16:0/2878393365 wait complete. 2023-05-22T15:27:40.738 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-05-22T15:27:40.854 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.847+0000 7feee406c640 1 Processor -- start 2023-05-22T15:27:40.854 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.847+0000 7feee406c640 1 -- start start 2023-05-22T15:27:40.855 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.847+0000 7feee406c640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7feedc06b6c0 0x7feedc0cb4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:40.855 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.847+0000 7feee406c640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feedc159f70 con 0x7feedc06ba90 2023-05-22T15:27:40.855 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.847+0000 7feee406c640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feedc0cb030 con 0x7feedc1596e0 2023-05-22T15:27:40.855 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.847+0000 7feee406c640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feedc071760 con 0x7feedc06b6c0 2023-05-22T15:27:40.855 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.847+0000 7feee1e08640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7feedc06b6c0 0x7feedc0cb4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:40.856 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.847+0000 7feee1607640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7feedc06ba90 0x7feedc158fd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:45576/0 (socket says 172.21.15.16:45576) 2023-05-22T15:27:40.856 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.847+0000 7feee1e08640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7feedc06b6c0 0x7feedc0cb4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41480/0 (socket says 172.21.15.16:41480) 2023-05-22T15:27:40.856 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.847+0000 7feee1607640 1 -- 172.21.15.16:0/2673311454 learned_addr learned my addr 172.21.15.16:0/2673311454 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:27:40.856 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.847+0000 7feee1e08640 1 -- 172.21.15.16:0/2673311454 >> v1:172.21.15.62:6789/0 conn(0x7feedc1596e0 legacy=0x7feedc06fba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:40.856 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.847+0000 7feee1e08640 1 -- 172.21.15.16:0/2673311454 >> v1:172.21.15.16:6789/0 conn(0x7feedc06ba90 legacy=0x7feedc158fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:40.856 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.847+0000 7feee0e06640 1 -- 172.21.15.16:0/2673311454 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2330235301 0 0) 0x7feedc159f70 con 0x7feedc06ba90 2023-05-22T15:27:40.856 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.847+0000 7feee1e08640 1 -- 172.21.15.16:0/2673311454 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feedc159c90 con 0x7feedc06b6c0 2023-05-22T15:27:40.857 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.851+0000 7feee1e08640 1 --2- 172.21.15.16:0/2673311454 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feedc06b6c0 0x7feedc0cb4c0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7feed800ba60 tx=0x7feed801eac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0983d91138654f5 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:40.857 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.851+0000 7feee0e06640 1 -- 172.21.15.16:0/2673311454 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feed80a7980 con 0x7feedc06b6c0 2023-05-22T15:27:40.857 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.851+0000 7feee0e06640 1 -- 172.21.15.16:0/2673311454 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feed80a7ae0 con 0x7feedc06b6c0 2023-05-22T15:27:40.857 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.851+0000 7feee0e06640 1 -- 172.21.15.16:0/2673311454 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feed80a7e70 con 0x7feedc06b6c0 2023-05-22T15:27:40.858 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.851+0000 7feee406c640 1 -- 172.21.15.16:0/2673311454 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feedc06b6c0 msgr2=0x7feedc0cb4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:40.858 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.851+0000 7feee406c640 1 --2- 172.21.15.16:0/2673311454 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feedc06b6c0 0x7feedc0cb4c0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7feed800ba60 tx=0x7feed801eac0 comp rx=0 tx=0).stop 2023-05-22T15:27:40.858 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.851+0000 7feee406c640 1 -- 172.21.15.16:0/2673311454 shutdown_connections 2023-05-22T15:27:40.859 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.851+0000 7feee406c640 1 --2- 172.21.15.16:0/2673311454 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feedc06b6c0 0x7feedc0cb4c0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:40.859 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.851+0000 7feee406c640 1 -- 172.21.15.16:0/2673311454 >> 172.21.15.16:0/2673311454 conn(0x7feedc069720 msgr2=0x7feedc069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:40.859 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.851+0000 7feee406c640 1 -- 172.21.15.16:0/2673311454 shutdown_connections 2023-05-22T15:27:40.859 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.851+0000 7feee406c640 1 -- 172.21.15.16:0/2673311454 wait complete. 2023-05-22T15:27:40.860 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.851+0000 7feee406c640 1 Processor -- start 2023-05-22T15:27:40.861 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.851+0000 7feee406c640 1 -- start start 2023-05-22T15:27:40.861 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.851+0000 7feee406c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feedc06b6c0 0x7feedc143510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:40.861 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.851+0000 7feee406c640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7feedc06ba90 0x7feedc0aff90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:40.861 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.851+0000 7feee406c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7feedc1596e0 0x7feedc0c94f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:40.861 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.851+0000 7feee406c640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feedc0b3530 con 0x7feedc06b6c0 2023-05-22T15:27:40.861 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.851+0000 7feee406c640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feedc0afb00 con 0x7feedc1596e0 2023-05-22T15:27:40.861 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.851+0000 7feee406c640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feedc0b3780 con 0x7feedc06ba90 2023-05-22T15:27:40.862 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.851+0000 7feee1e08640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feedc06b6c0 0x7feedc143510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:40.862 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.851+0000 7feee1e08640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feedc06b6c0 0x7feedc143510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41494/0 (socket says 172.21.15.16:41494) 2023-05-22T15:27:40.862 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.851+0000 7feee1e08640 1 -- 172.21.15.16:0/2182127086 learned_addr learned my addr 172.21.15.16:0/2182127086 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:40.862 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.851+0000 7feee1e08640 1 -- 172.21.15.16:0/2182127086 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7feedc06ba90 msgr2=0x7feedc0aff90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:40.862 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.851+0000 7feee1e08640 1 --2- 172.21.15.16:0/2182127086 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7feedc06ba90 0x7feedc0aff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:40.862 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.851+0000 7feee1e08640 1 -- 172.21.15.16:0/2182127086 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7feedc1596e0 msgr2=0x7feedc0c94f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:40.862 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.851+0000 7feee1e08640 1 --2- 172.21.15.16:0/2182127086 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7feedc1596e0 0x7feedc0c94f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:40.863 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.851+0000 7feee1e08640 1 -- 172.21.15.16:0/2182127086 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feed800b6c0 con 0x7feedc06b6c0 2023-05-22T15:27:40.863 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.851+0000 7feee1e08640 1 --2- 172.21.15.16:0/2182127086 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feedc06b6c0 0x7feedc143510 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7feed80b2040 tx=0x7feed801b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:40.863 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.851+0000 7feed27fc640 1 -- 172.21.15.16:0/2182127086 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feed80afa10 con 0x7feedc06b6c0 2023-05-22T15:27:40.863 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.851+0000 7feed27fc640 1 -- 172.21.15.16:0/2182127086 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feed80afb90 con 0x7feedc06b6c0 2023-05-22T15:27:40.863 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.851+0000 7feed27fc640 1 -- 172.21.15.16:0/2182127086 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feed80b8420 con 0x7feedc06b6c0 2023-05-22T15:27:40.863 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.851+0000 7feee406c640 1 -- 172.21.15.16:0/2182127086 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feedc06c030 con 0x7feedc06b6c0 2023-05-22T15:27:40.863 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.851+0000 7feee406c640 1 -- 172.21.15.16:0/2182127086 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feedc06bea0 con 0x7feedc06b6c0 2023-05-22T15:27:40.864 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.855+0000 7feee406c640 1 -- 172.21.15.16:0/2182127086 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feedc156840 con 0x7feedc06b6c0 2023-05-22T15:27:40.864 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.855+0000 7feed27fc640 1 -- 172.21.15.16:0/2182127086 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7feed80b6070 con 0x7feedc06b6c0 2023-05-22T15:27:40.864 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.855+0000 7feed27fc640 1 --2- 172.21.15.16:0/2182127086 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7feec0065380 0x7feec0067830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:40.864 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.855+0000 7feed27fc640 1 -- 172.21.15.16:0/2182127086 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6871+0+0 (secure 0 0 0) 0x7feed811c980 con 0x7feedc06b6c0 2023-05-22T15:27:40.865 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.855+0000 7feee1607640 1 --2- 172.21.15.16:0/2182127086 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7feec0065380 0x7feec0067830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:40.865 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.855+0000 7feee1607640 1 --2- 172.21.15.16:0/2182127086 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7feec0065380 0x7feec0067830 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7feecc0011f0 tx=0x7feecc007930 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:40.866 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:40.859+0000 7feed27fc640 1 -- 172.21.15.16:0/2182127086 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7feed80ea1b0 con 0x7feedc06b6c0 2023-05-22T15:27:41.012 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.003+0000 7feee406c640 1 -- 172.21.15.16:0/2182127086 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7feedc04dbe0 con 0x7feedc06b6c0 2023-05-22T15:27:41.013 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.003+0000 7feed27fc640 1 -- 172.21.15.16:0/2182127086 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v29) v1 ==== 104+0+11 (secure 0 0 0) 0x7feed80ed7d0 con 0x7feedc06b6c0 2023-05-22T15:27:41.013 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2023-05-22T15:27:41.015 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.007+0000 7feee406c640 1 -- 172.21.15.16:0/2182127086 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7feec0065380 msgr2=0x7feec0067830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:41.015 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.007+0000 7feee406c640 1 --2- 172.21.15.16:0/2182127086 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7feec0065380 0x7feec0067830 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7feecc0011f0 tx=0x7feecc007930 comp rx=0 tx=0).stop 2023-05-22T15:27:41.015 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.007+0000 7feee406c640 1 -- 172.21.15.16:0/2182127086 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feedc06b6c0 msgr2=0x7feedc143510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:41.016 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.007+0000 7feee406c640 1 --2- 172.21.15.16:0/2182127086 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feedc06b6c0 0x7feedc143510 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7feed80b2040 tx=0x7feed801b860 comp rx=0 tx=0).stop 2023-05-22T15:27:41.016 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.007+0000 7feee406c640 1 -- 172.21.15.16:0/2182127086 shutdown_connections 2023-05-22T15:27:41.016 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.007+0000 7feee406c640 1 --2- 172.21.15.16:0/2182127086 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7feedc1596e0 0x7feedc0c94f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:41.016 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.007+0000 7feee406c640 1 --2- 172.21.15.16:0/2182127086 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7feec0065380 0x7feec0067830 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:41.017 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.007+0000 7feee406c640 1 --2- 172.21.15.16:0/2182127086 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7feedc06ba90 0x7feedc0aff90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:41.017 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.007+0000 7feee406c640 1 --2- 172.21.15.16:0/2182127086 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feedc06b6c0 0x7feedc143510 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:41.017 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.007+0000 7feee406c640 1 -- 172.21.15.16:0/2182127086 >> 172.21.15.16:0/2182127086 conn(0x7feedc069720 msgr2=0x7feedc155760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:41.017 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.007+0000 7feee406c640 1 -- 172.21.15.16:0/2182127086 shutdown_connections 2023-05-22T15:27:41.017 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.007+0000 7feee406c640 1 -- 172.21.15.16:0/2182127086 wait complete. 2023-05-22T15:27:41.031 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-05-22T15:27:41.158 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.151+0000 7fab06f56640 1 Processor -- start 2023-05-22T15:27:41.158 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.151+0000 7fab06f56640 1 -- start start 2023-05-22T15:27:41.159 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.151+0000 7fab06f56640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fab0006b6c0 0x7fab000cb4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:41.160 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.151+0000 7fab06f56640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fab001596e0 0x7fab0006fba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:41.160 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.151+0000 7fab06f56640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab00159f70 con 0x7fab0006ba90 2023-05-22T15:27:41.160 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.151+0000 7fab06f56640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab00071760 con 0x7fab0006b6c0 2023-05-22T15:27:41.160 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.151+0000 7fab06f56640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab000ca7d0 con 0x7fab001596e0 2023-05-22T15:27:41.160 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.151+0000 7faafffff640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fab0006ba90 0x7fab00158fd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:45584/0 (socket says 172.21.15.16:45584) 2023-05-22T15:27:41.160 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.151+0000 7fab04cf2640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fab0006b6c0 0x7fab000cb4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:41.161 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.151+0000 7faafffff640 1 -- 172.21.15.16:0/1522679256 learned_addr learned my addr 172.21.15.16:0/1522679256 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:27:41.161 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.151+0000 7fab054f3640 1 --2- 172.21.15.16:0/1522679256 >> v2:172.21.15.62:3300/0 conn(0x7fab001596e0 0x7fab0006fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:41.161 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.151+0000 7faaff7fe640 1 -- 172.21.15.16:0/1522679256 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2204994321 0 0) 0x7fab00159f70 con 0x7fab0006ba90 2023-05-22T15:27:41.161 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.151+0000 7faaff7fe640 1 -- 172.21.15.16:0/1522679256 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faae8003530 con 0x7fab0006ba90 2023-05-22T15:27:41.161 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.151+0000 7fab04cf2640 1 -- 172.21.15.16:0/1522679256 >> v2:172.21.15.62:3300/0 conn(0x7fab001596e0 msgr2=0x7fab0006fba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:41.161 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.151+0000 7fab04cf2640 1 --2- 172.21.15.16:0/1522679256 >> v2:172.21.15.62:3300/0 conn(0x7fab001596e0 0x7fab0006fba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:41.161 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.151+0000 7fab04cf2640 1 -- 172.21.15.16:0/1522679256 >> v1:172.21.15.16:6789/0 conn(0x7fab0006ba90 legacy=0x7fab00158fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:41.161 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.151+0000 7fab04cf2640 1 -- 172.21.15.16:0/1522679256 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab000cb030 con 0x7fab0006b6c0 2023-05-22T15:27:41.162 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.151+0000 7fab054f3640 1 --2- 172.21.15.16:0/1522679256 >> v2:172.21.15.62:3300/0 conn(0x7fab001596e0 0x7fab0006fba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T15:27:41.162 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.155+0000 7fab04cf2640 1 --2- 172.21.15.16:0/1522679256 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab0006b6c0 0x7fab000cb4c0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fab0006c710 tx=0x7faaf001c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74886e99000a0301 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:41.163 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.155+0000 7faaff7fe640 1 -- 172.21.15.16:0/1522679256 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faaf00ac600 con 0x7fab0006b6c0 2023-05-22T15:27:41.163 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.155+0000 7faaff7fe640 1 -- 172.21.15.16:0/1522679256 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faaf00a8050 con 0x7fab0006b6c0 2023-05-22T15:27:41.163 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.155+0000 7faaff7fe640 1 -- 172.21.15.16:0/1522679256 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faaf00ac960 con 0x7fab0006b6c0 2023-05-22T15:27:41.163 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.155+0000 7fab06f56640 1 -- 172.21.15.16:0/1522679256 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab0006b6c0 msgr2=0x7fab000cb4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:41.163 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.155+0000 7fab06f56640 1 --2- 172.21.15.16:0/1522679256 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab0006b6c0 0x7fab000cb4c0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fab0006c710 tx=0x7faaf001c3a0 comp rx=0 tx=0).stop 2023-05-22T15:27:41.163 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.155+0000 7fab06f56640 1 -- 172.21.15.16:0/1522679256 shutdown_connections 2023-05-22T15:27:41.163 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.155+0000 7fab06f56640 1 --2- 172.21.15.16:0/1522679256 >> v2:172.21.15.62:3300/0 conn(0x7fab001596e0 0x7fab0006fba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:41.164 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.155+0000 7fab06f56640 1 --2- 172.21.15.16:0/1522679256 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab0006b6c0 0x7fab000cb4c0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:41.164 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.155+0000 7fab06f56640 1 -- 172.21.15.16:0/1522679256 >> 172.21.15.16:0/1522679256 conn(0x7fab00069720 msgr2=0x7fab00069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:41.164 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.155+0000 7fab06f56640 1 -- 172.21.15.16:0/1522679256 shutdown_connections 2023-05-22T15:27:41.164 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.155+0000 7fab06f56640 1 -- 172.21.15.16:0/1522679256 wait complete. 2023-05-22T15:27:41.166 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.155+0000 7fab06f56640 1 Processor -- start 2023-05-22T15:27:41.166 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.155+0000 7fab06f56640 1 -- start start 2023-05-22T15:27:41.166 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.155+0000 7fab06f56640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fab0006b6c0 0x7fab001662c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:41.166 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.155+0000 7fab06f56640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fab0006ba90 0x7fab00164960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:41.166 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.155+0000 7fab06f56640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab001596e0 0x7fab00164ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:41.167 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.155+0000 7fab06f56640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab000c3210 con 0x7fab001596e0 2023-05-22T15:27:41.167 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.155+0000 7fab06f56640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab000c34f0 con 0x7fab0006ba90 2023-05-22T15:27:41.167 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.155+0000 7fab06f56640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab000c37d0 con 0x7fab0006b6c0 2023-05-22T15:27:41.167 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.155+0000 7fab054f3640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab001596e0 0x7fab00164ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:41.167 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.155+0000 7fab054f3640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab001596e0 0x7fab00164ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41516/0 (socket says 172.21.15.16:41516) 2023-05-22T15:27:41.167 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.155+0000 7fab054f3640 1 -- 172.21.15.16:0/539230943 learned_addr learned my addr 172.21.15.16:0/539230943 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:41.167 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.155+0000 7fab04cf2640 1 --2- 172.21.15.16:0/539230943 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fab0006b6c0 0x7fab001662c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:41.167 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.155+0000 7faafffff640 1 --2- 172.21.15.16:0/539230943 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fab0006ba90 0x7fab00164960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:41.168 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.155+0000 7fab054f3640 1 -- 172.21.15.16:0/539230943 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fab0006b6c0 msgr2=0x7fab001662c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:41.168 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.155+0000 7fab054f3640 1 --2- 172.21.15.16:0/539230943 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fab0006b6c0 0x7fab001662c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:41.168 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.155+0000 7fab054f3640 1 -- 172.21.15.16:0/539230943 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fab0006ba90 msgr2=0x7fab00164960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:41.168 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.155+0000 7fab054f3640 1 --2- 172.21.15.16:0/539230943 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fab0006ba90 0x7fab00164960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:41.168 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.155+0000 7fab054f3640 1 -- 172.21.15.16:0/539230943 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faaf0009420 con 0x7fab001596e0 2023-05-22T15:27:41.169 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.155+0000 7fab04cf2640 1 --2- 172.21.15.16:0/539230943 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fab0006b6c0 0x7fab001662c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T15:27:41.169 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.155+0000 7fab054f3640 1 --2- 172.21.15.16:0/539230943 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab001596e0 0x7fab00164ea0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7faaf8007df0 tx=0x7faaf800d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:41.169 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.159+0000 7faafd7fa640 1 -- 172.21.15.16:0/539230943 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faaf80039a0 con 0x7fab001596e0 2023-05-22T15:27:41.169 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.159+0000 7faafd7fa640 1 -- 172.21.15.16:0/539230943 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faaf800f040 con 0x7fab001596e0 2023-05-22T15:27:41.169 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.159+0000 7faafd7fa640 1 -- 172.21.15.16:0/539230943 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faaf8003d40 con 0x7fab001596e0 2023-05-22T15:27:41.170 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.159+0000 7fab06f56640 1 -- 172.21.15.16:0/539230943 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab0006be60 con 0x7fab001596e0 2023-05-22T15:27:41.170 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.159+0000 7fab06f56640 1 -- 172.21.15.16:0/539230943 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab000cacc0 con 0x7fab001596e0 2023-05-22T15:27:41.170 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.159+0000 7fab06f56640 1 -- 172.21.15.16:0/539230943 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab0006bff0 con 0x7fab001596e0 2023-05-22T15:27:41.171 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.159+0000 7faafd7fa640 1 -- 172.21.15.16:0/539230943 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7faaf8004030 con 0x7fab001596e0 2023-05-22T15:27:41.171 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.159+0000 7faafd7fa640 1 --2- 172.21.15.16:0/539230943 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7faaec0653b0 0x7faaec067860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:41.171 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.159+0000 7faafd7fa640 1 -- 172.21.15.16:0/539230943 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6871+0+0 (secure 0 0 0) 0x7faaf8082ec0 con 0x7fab001596e0 2023-05-22T15:27:41.171 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.159+0000 7fab04cf2640 1 --2- 172.21.15.16:0/539230943 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7faaec0653b0 0x7faaec067860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:41.171 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.159+0000 7fab04cf2640 1 --2- 172.21.15.16:0/539230943 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7faaec0653b0 0x7faaec067860 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7faaf00ae000 tx=0x7faaf00a7000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:41.172 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.163+0000 7faafd7fa640 1 -- 172.21.15.16:0/539230943 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7faaf80507b0 con 0x7fab001596e0 2023-05-22T15:27:41.327 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.319+0000 7fab06f56640 1 -- 172.21.15.16:0/539230943 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fab0004dbe0 con 0x7fab001596e0 2023-05-22T15:27:41.328 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.319+0000 7faafd7fa640 1 -- 172.21.15.16:0/539230943 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v29) v1 ==== 100+0+11 (secure 0 0 0) 0x7faaf801a020 con 0x7fab001596e0 2023-05-22T15:27:41.328 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 64 2023-05-22T15:27:41.330 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.323+0000 7fab06f56640 1 -- 172.21.15.16:0/539230943 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7faaec0653b0 msgr2=0x7faaec067860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:41.330 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.323+0000 7fab06f56640 1 --2- 172.21.15.16:0/539230943 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7faaec0653b0 0x7faaec067860 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7faaf00ae000 tx=0x7faaf00a7000 comp rx=0 tx=0).stop 2023-05-22T15:27:41.331 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.323+0000 7fab06f56640 1 -- 172.21.15.16:0/539230943 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab001596e0 msgr2=0x7fab00164ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:41.331 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.323+0000 7fab06f56640 1 --2- 172.21.15.16:0/539230943 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab001596e0 0x7fab00164ea0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7faaf8007df0 tx=0x7faaf800d4a0 comp rx=0 tx=0).stop 2023-05-22T15:27:41.331 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.323+0000 7fab06f56640 1 -- 172.21.15.16:0/539230943 shutdown_connections 2023-05-22T15:27:41.331 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.323+0000 7fab06f56640 1 --2- 172.21.15.16:0/539230943 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab001596e0 0x7fab00164ea0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:41.332 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.323+0000 7fab06f56640 1 --2- 172.21.15.16:0/539230943 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fab0006ba90 0x7fab00164960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:41.332 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.323+0000 7fab06f56640 1 --2- 172.21.15.16:0/539230943 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7faaec0653b0 0x7faaec067860 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:41.332 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.323+0000 7fab06f56640 1 --2- 172.21.15.16:0/539230943 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fab0006b6c0 0x7fab001662c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:41.333 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.323+0000 7fab06f56640 1 -- 172.21.15.16:0/539230943 >> 172.21.15.16:0/539230943 conn(0x7fab00069720 msgr2=0x7fab000ccd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:41.333 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.323+0000 7fab06f56640 1 -- 172.21.15.16:0/539230943 shutdown_connections 2023-05-22T15:27:41.333 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.323+0000 7fab06f56640 1 -- 172.21.15.16:0/539230943 wait complete. 2023-05-22T15:27:41.348 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-05-22T15:27:41.478 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.471+0000 7ff1cb4eb640 1 Processor -- start 2023-05-22T15:27:41.478 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.471+0000 7ff1cb4eb640 1 -- start start 2023-05-22T15:27:41.480 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.471+0000 7ff1cb4eb640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff1c415bca0 0x7ff1c40cda80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:41.480 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.471+0000 7ff1cb4eb640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff1c40d2480 0x7ff1c40d28d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:41.480 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.471+0000 7ff1cb4eb640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1c40d2e10 con 0x7ff1c40b5340 2023-05-22T15:27:41.481 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.471+0000 7ff1cb4eb640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1c40cd660 con 0x7ff1c415bca0 2023-05-22T15:27:41.481 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.471+0000 7ff1cb4eb640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1c40cccc0 con 0x7ff1c40d2480 2023-05-22T15:27:41.481 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.471+0000 7ff1c9287640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff1c415bca0 0x7ff1c40cda80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:41.481 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.471+0000 7ff1c9287640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff1c415bca0 0x7ff1c40cda80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41524/0 (socket says 172.21.15.16:41524) 2023-05-22T15:27:41.481 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.471+0000 7ff1c9287640 1 -- 172.21.15.16:0/3099167428 learned_addr learned my addr 172.21.15.16:0/3099167428 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:41.481 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.471+0000 7ff1c8a86640 1 --2- 172.21.15.16:0/3099167428 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff1c40d2480 0x7ff1c40d28d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:41.482 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.471+0000 7ff1c9287640 1 -- 172.21.15.16:0/3099167428 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff1c40d2480 msgr2=0x7ff1c40d28d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:41.482 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.471+0000 7ff1c9287640 1 --2- 172.21.15.16:0/3099167428 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff1c40d2480 0x7ff1c40d28d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:41.482 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.471+0000 7ff1c9287640 1 -- 172.21.15.16:0/3099167428 >> v1:172.21.15.62:6789/0 conn(0x7ff1c40b5340 legacy=0x7ff1c40b57c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:41.482 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.471+0000 7ff1c9287640 1 -- 172.21.15.16:0/3099167428 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1c40ccfa0 con 0x7ff1c415bca0 2023-05-22T15:27:41.483 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.475+0000 7ff1c9287640 1 --2- 172.21.15.16:0/3099167428 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff1c415bca0 0x7ff1c40cda80 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7ff1c0009870 tx=0x7ff1c001c510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9724c5f881f1c9e9 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:41.483 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.475+0000 7ff1bbfff640 1 -- 172.21.15.16:0/3099167428 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1c00a95f0 con 0x7ff1c415bca0 2023-05-22T15:27:41.483 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.475+0000 7ff1bbfff640 1 -- 172.21.15.16:0/3099167428 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1c00a9750 con 0x7ff1c415bca0 2023-05-22T15:27:41.484 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.475+0000 7ff1bbfff640 1 -- 172.21.15.16:0/3099167428 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1c00a9ae0 con 0x7ff1c415bca0 2023-05-22T15:27:41.484 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.475+0000 7ff1cb4eb640 1 -- 172.21.15.16:0/3099167428 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff1c415bca0 msgr2=0x7ff1c40cda80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:41.484 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.475+0000 7ff1cb4eb640 1 --2- 172.21.15.16:0/3099167428 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff1c415bca0 0x7ff1c40cda80 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7ff1c0009870 tx=0x7ff1c001c510 comp rx=0 tx=0).stop 2023-05-22T15:27:41.484 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.475+0000 7ff1cb4eb640 1 -- 172.21.15.16:0/3099167428 shutdown_connections 2023-05-22T15:27:41.484 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.475+0000 7ff1cb4eb640 1 --2- 172.21.15.16:0/3099167428 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff1c40d2480 0x7ff1c40d28d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:41.484 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.475+0000 7ff1cb4eb640 1 --2- 172.21.15.16:0/3099167428 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff1c415bca0 0x7ff1c40cda80 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:41.484 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.475+0000 7ff1cb4eb640 1 -- 172.21.15.16:0/3099167428 >> 172.21.15.16:0/3099167428 conn(0x7ff1c40655c0 msgr2=0x7ff1c4070e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:41.485 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.475+0000 7ff1cb4eb640 1 -- 172.21.15.16:0/3099167428 shutdown_connections 2023-05-22T15:27:41.485 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.475+0000 7ff1cb4eb640 1 -- 172.21.15.16:0/3099167428 wait complete. 2023-05-22T15:27:41.487 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.475+0000 7ff1cb4eb640 1 Processor -- start 2023-05-22T15:27:41.487 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.475+0000 7ff1cb4eb640 1 -- start start 2023-05-22T15:27:41.487 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.479+0000 7ff1cb4eb640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff1c40b5340 0x7ff1c414a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:41.487 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.479+0000 7ff1cb4eb640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff1c40d2480 0x7ff1c4159040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:41.487 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.479+0000 7ff1cb4eb640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff1c415bca0 0x7ff1c4159a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:41.488 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.479+0000 7ff1cb4eb640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1c40b71e0 con 0x7ff1c415bca0 2023-05-22T15:27:41.488 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.479+0000 7ff1cb4eb640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1c414a230 con 0x7ff1c40b5340 2023-05-22T15:27:41.488 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.479+0000 7ff1cb4eb640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1c40b6190 con 0x7ff1c40d2480 2023-05-22T15:27:41.488 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.479+0000 7ff1c9a88640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff1c415bca0 0x7ff1c4159a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:41.488 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.479+0000 7ff1c9a88640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff1c415bca0 0x7ff1c4159a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41526/0 (socket says 172.21.15.16:41526) 2023-05-22T15:27:41.488 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.479+0000 7ff1c9a88640 1 -- 172.21.15.16:0/1944296811 learned_addr learned my addr 172.21.15.16:0/1944296811 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:41.488 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.479+0000 7ff1c9287640 1 --2- 172.21.15.16:0/1944296811 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff1c40b5340 0x7ff1c414a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:41.489 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.479+0000 7ff1c8a86640 1 --2- 172.21.15.16:0/1944296811 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff1c40d2480 0x7ff1c4159040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:41.489 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.479+0000 7ff1c9a88640 1 -- 172.21.15.16:0/1944296811 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff1c40d2480 msgr2=0x7ff1c4159040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:41.489 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.479+0000 7ff1c9a88640 1 --2- 172.21.15.16:0/1944296811 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff1c40d2480 0x7ff1c4159040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:41.489 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.479+0000 7ff1c9a88640 1 -- 172.21.15.16:0/1944296811 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff1c40b5340 msgr2=0x7ff1c414a430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:41.489 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.479+0000 7ff1c9a88640 1 --2- 172.21.15.16:0/1944296811 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff1c40b5340 0x7ff1c414a430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:41.489 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.479+0000 7ff1c9a88640 1 -- 172.21.15.16:0/1944296811 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1c00094d0 con 0x7ff1c415bca0 2023-05-22T15:27:41.489 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.479+0000 7ff1c9a88640 1 --2- 172.21.15.16:0/1944296811 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff1c415bca0 0x7ff1c4159a30 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7ff1bc00c770 tx=0x7ff1bc00cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:41.489 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.479+0000 7ff1b9ffb640 1 -- 172.21.15.16:0/1944296811 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1bc003b80 con 0x7ff1c415bca0 2023-05-22T15:27:41.490 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.479+0000 7ff1b9ffb640 1 -- 172.21.15.16:0/1944296811 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1bc003ce0 con 0x7ff1c415bca0 2023-05-22T15:27:41.490 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.479+0000 7ff1cb4eb640 1 -- 172.21.15.16:0/1944296811 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1c4160360 con 0x7ff1c415bca0 2023-05-22T15:27:41.490 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.479+0000 7ff1b9ffb640 1 -- 172.21.15.16:0/1944296811 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1bc00aa90 con 0x7ff1c415bca0 2023-05-22T15:27:41.490 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.479+0000 7ff1cb4eb640 1 -- 172.21.15.16:0/1944296811 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1c40cdfc0 con 0x7ff1c415bca0 2023-05-22T15:27:41.491 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.479+0000 7ff1cb4eb640 1 -- 172.21.15.16:0/1944296811 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1c41604f0 con 0x7ff1c415bca0 2023-05-22T15:27:41.491 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.479+0000 7ff1b9ffb640 1 -- 172.21.15.16:0/1944296811 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7ff1bc00abf0 con 0x7ff1c415bca0 2023-05-22T15:27:41.492 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.479+0000 7ff1b9ffb640 1 --2- 172.21.15.16:0/1944296811 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ff1a80653b0 0x7ff1a8067860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:41.492 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.479+0000 7ff1b9ffb640 1 -- 172.21.15.16:0/1944296811 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6871+0+0 (secure 0 0 0) 0x7ff1bc082820 con 0x7ff1c415bca0 2023-05-22T15:27:41.492 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.479+0000 7ff1c9287640 1 --2- 172.21.15.16:0/1944296811 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ff1a80653b0 0x7ff1a8067860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:41.492 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.483+0000 7ff1c9287640 1 --2- 172.21.15.16:0/1944296811 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ff1a80653b0 0x7ff1a8067860 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7ff1c00ae000 tx=0x7ff1c00b3040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:41.493 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.483+0000 7ff1b9ffb640 1 -- 172.21.15.16:0/1944296811 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7ff1bc050290 con 0x7ff1c415bca0 2023-05-22T15:27:41.642 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.635+0000 7ff1cb4eb640 1 -- 172.21.15.16:0/1944296811 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7ff1c40d2e10 con 0x7ff1c415bca0 2023-05-22T15:27:41.643 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.635+0000 7ff1b9ffb640 1 -- 172.21.15.16:0/1944296811 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v29) v1 ==== 103+0+11 (secure 0 0 0) 0x7ff1c40d2e10 con 0x7ff1c415bca0 2023-05-22T15:27:41.644 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2023-05-22T15:27:41.645 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.635+0000 7ff1cb4eb640 1 -- 172.21.15.16:0/1944296811 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ff1a80653b0 msgr2=0x7ff1a8067860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:41.646 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.635+0000 7ff1cb4eb640 1 --2- 172.21.15.16:0/1944296811 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ff1a80653b0 0x7ff1a8067860 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7ff1c00ae000 tx=0x7ff1c00b3040 comp rx=0 tx=0).stop 2023-05-22T15:27:41.646 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.639+0000 7ff1cb4eb640 1 -- 172.21.15.16:0/1944296811 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff1c415bca0 msgr2=0x7ff1c4159a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:41.646 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.639+0000 7ff1cb4eb640 1 --2- 172.21.15.16:0/1944296811 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff1c415bca0 0x7ff1c4159a30 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7ff1bc00c770 tx=0x7ff1bc00cc30 comp rx=0 tx=0).stop 2023-05-22T15:27:41.646 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.639+0000 7ff1cb4eb640 1 -- 172.21.15.16:0/1944296811 shutdown_connections 2023-05-22T15:27:41.646 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.639+0000 7ff1cb4eb640 1 --2- 172.21.15.16:0/1944296811 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff1c415bca0 0x7ff1c4159a30 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:41.646 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.639+0000 7ff1cb4eb640 1 --2- 172.21.15.16:0/1944296811 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ff1a80653b0 0x7ff1a8067860 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:41.647 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.639+0000 7ff1cb4eb640 1 --2- 172.21.15.16:0/1944296811 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff1c40d2480 0x7ff1c4159040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:41.647 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.639+0000 7ff1cb4eb640 1 --2- 172.21.15.16:0/1944296811 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff1c40b5340 0x7ff1c414a430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:41.647 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.639+0000 7ff1cb4eb640 1 -- 172.21.15.16:0/1944296811 >> 172.21.15.16:0/1944296811 conn(0x7ff1c40655c0 msgr2=0x7ff1c40c6ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:41.647 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.639+0000 7ff1cb4eb640 1 -- 172.21.15.16:0/1944296811 shutdown_connections 2023-05-22T15:27:41.647 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.639+0000 7ff1cb4eb640 1 -- 172.21.15.16:0/1944296811 wait complete. 2023-05-22T15:27:41.663 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-05-22T15:27:41.782 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.775+0000 7fc096c2b640 1 Processor -- start 2023-05-22T15:27:41.782 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.775+0000 7fc096c2b640 1 -- start start 2023-05-22T15:27:41.783 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.775+0000 7fc096c2b640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc0900d1c70 0x7fc0900ce6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:41.783 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.775+0000 7fc096c2b640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0900b4de0 con 0x7fc090159740 2023-05-22T15:27:41.784 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.775+0000 7fc096c2b640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0900d21e0 con 0x7fc0900cec00 2023-05-22T15:27:41.784 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.775+0000 7fc096c2b640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0900ccbf0 con 0x7fc0900d1c70 2023-05-22T15:27:41.784 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.775+0000 7fc0949c7640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fc090159740 0x7fc09015cc50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:45588/0 (socket says 172.21.15.16:45588) 2023-05-22T15:27:41.784 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.775+0000 7fc0949c7640 1 -- 172.21.15.16:0/1894646098 learned_addr learned my addr 172.21.15.16:0/1894646098 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:27:41.785 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.775+0000 7fc08ffff640 1 --2- 172.21.15.16:0/1894646098 >> v2:172.21.15.16:3300/0 conn(0x7fc0900d1c70 0x7fc0900ce6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:41.785 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.775+0000 7fc08f7fe640 1 -- 172.21.15.16:0/1894646098 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 423017868 0 0) 0x7fc0900b4de0 con 0x7fc090159740 2023-05-22T15:27:41.785 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.775+0000 7fc08f7fe640 1 -- 172.21.15.16:0/1894646098 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc078003530 con 0x7fc090159740 2023-05-22T15:27:41.786 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.775+0000 7fc08ffff640 1 -- 172.21.15.16:0/1894646098 >> v1:172.21.15.62:6789/0 conn(0x7fc0900cec00 legacy=0x7fc0900b3ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:41.786 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.775+0000 7fc08ffff640 1 -- 172.21.15.16:0/1894646098 >> v1:172.21.15.16:6789/0 conn(0x7fc090159740 legacy=0x7fc09015cc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:41.786 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.775+0000 7fc08ffff640 1 -- 172.21.15.16:0/1894646098 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0900cce90 con 0x7fc0900d1c70 2023-05-22T15:27:41.787 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.779+0000 7fc08ffff640 1 --2- 172.21.15.16:0/1894646098 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc0900d1c70 0x7fc0900ce6c0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fc0840097c0 tx=0x7fc08401c3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60308c0ea5b5929a server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:41.787 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.779+0000 7fc08f7fe640 1 -- 172.21.15.16:0/1894646098 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0840ab600 con 0x7fc0900d1c70 2023-05-22T15:27:41.787 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.779+0000 7fc08f7fe640 1 -- 172.21.15.16:0/1894646098 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0840a7050 con 0x7fc0900d1c70 2023-05-22T15:27:41.787 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.779+0000 7fc08f7fe640 1 -- 172.21.15.16:0/1894646098 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0840ab960 con 0x7fc0900d1c70 2023-05-22T15:27:41.787 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.779+0000 7fc096c2b640 1 -- 172.21.15.16:0/1894646098 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc0900d1c70 msgr2=0x7fc0900ce6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:41.788 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.779+0000 7fc096c2b640 1 --2- 172.21.15.16:0/1894646098 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc0900d1c70 0x7fc0900ce6c0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fc0840097c0 tx=0x7fc08401c3c0 comp rx=0 tx=0).stop 2023-05-22T15:27:41.788 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.779+0000 7fc096c2b640 1 -- 172.21.15.16:0/1894646098 shutdown_connections 2023-05-22T15:27:41.788 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.779+0000 7fc096c2b640 1 --2- 172.21.15.16:0/1894646098 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc0900d1c70 0x7fc0900ce6c0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:41.788 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.779+0000 7fc096c2b640 1 -- 172.21.15.16:0/1894646098 >> 172.21.15.16:0/1894646098 conn(0x7fc09006c2f0 msgr2=0x7fc090070e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:41.788 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.779+0000 7fc096c2b640 1 -- 172.21.15.16:0/1894646098 shutdown_connections 2023-05-22T15:27:41.788 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.779+0000 7fc096c2b640 1 -- 172.21.15.16:0/1894646098 wait complete. 2023-05-22T15:27:41.790 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.779+0000 7fc096c2b640 1 Processor -- start 2023-05-22T15:27:41.790 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.779+0000 7fc096c2b640 1 -- start start 2023-05-22T15:27:41.791 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.779+0000 7fc096c2b640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc0900cec00 0x7fc090160330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:41.791 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.783+0000 7fc096c2b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc0900d1c70 0x7fc090152890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:41.791 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.783+0000 7fc096c2b640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc090159740 0x7fc0900caef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:41.791 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.783+0000 7fc096c2b640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0900b37e0 con 0x7fc0900d1c70 2023-05-22T15:27:41.791 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.783+0000 7fc096c2b640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0900c8b40 con 0x7fc090159740 2023-05-22T15:27:41.791 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.783+0000 7fc096c2b640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0900b3220 con 0x7fc0900cec00 2023-05-22T15:27:41.792 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.783+0000 7fc08ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc0900d1c70 0x7fc090152890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:41.792 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.783+0000 7fc08ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc0900d1c70 0x7fc090152890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41546/0 (socket says 172.21.15.16:41546) 2023-05-22T15:27:41.792 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.783+0000 7fc08ffff640 1 -- 172.21.15.16:0/3894766231 learned_addr learned my addr 172.21.15.16:0/3894766231 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:41.792 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.783+0000 7fc0949c7640 1 --2- 172.21.15.16:0/3894766231 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc0900cec00 0x7fc090160330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:41.792 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.783+0000 7fc0951c8640 1 --2- 172.21.15.16:0/3894766231 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc090159740 0x7fc0900caef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:41.792 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.783+0000 7fc08ffff640 1 -- 172.21.15.16:0/3894766231 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc0900cec00 msgr2=0x7fc090160330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:41.792 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.783+0000 7fc08ffff640 1 --2- 172.21.15.16:0/3894766231 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc0900cec00 0x7fc090160330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:41.793 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.783+0000 7fc08ffff640 1 -- 172.21.15.16:0/3894766231 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc090159740 msgr2=0x7fc0900caef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:41.793 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.783+0000 7fc08ffff640 1 --2- 172.21.15.16:0/3894766231 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc090159740 0x7fc0900caef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:41.793 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.783+0000 7fc08ffff640 1 -- 172.21.15.16:0/3894766231 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc084009420 con 0x7fc0900d1c70 2023-05-22T15:27:41.793 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.783+0000 7fc08ffff640 1 --2- 172.21.15.16:0/3894766231 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc0900d1c70 0x7fc090152890 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fc084019ff0 tx=0x7fc084019160 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:41.793 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.783+0000 7fc08d7fa640 1 -- 172.21.15.16:0/3894766231 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc084003af0 con 0x7fc0900d1c70 2023-05-22T15:27:41.793 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.783+0000 7fc08d7fa640 1 -- 172.21.15.16:0/3894766231 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc084003c50 con 0x7fc0900d1c70 2023-05-22T15:27:41.793 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.783+0000 7fc096c2b640 1 -- 172.21.15.16:0/3894766231 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0900b4600 con 0x7fc0900d1c70 2023-05-22T15:27:41.793 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.783+0000 7fc08d7fa640 1 -- 172.21.15.16:0/3894766231 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0840ab400 con 0x7fc0900d1c70 2023-05-22T15:27:41.794 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.783+0000 7fc096c2b640 1 -- 172.21.15.16:0/3894766231 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc090159b10 con 0x7fc0900d1c70 2023-05-22T15:27:41.794 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.783+0000 7fc096c2b640 1 -- 172.21.15.16:0/3894766231 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0900b4790 con 0x7fc0900d1c70 2023-05-22T15:27:41.795 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.783+0000 7fc08d7fa640 1 -- 172.21.15.16:0/3894766231 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7fc0840b2040 con 0x7fc0900d1c70 2023-05-22T15:27:41.795 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.783+0000 7fc08d7fa640 1 --2- 172.21.15.16:0/3894766231 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fc07c065380 0x7fc07c067830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:41.795 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.783+0000 7fc08d7fa640 1 -- 172.21.15.16:0/3894766231 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6871+0+0 (secure 0 0 0) 0x7fc08411cbc0 con 0x7fc0900d1c70 2023-05-22T15:27:41.795 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.783+0000 7fc0949c7640 1 --2- 172.21.15.16:0/3894766231 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fc07c065380 0x7fc07c067830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:41.795 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.783+0000 7fc0949c7640 1 --2- 172.21.15.16:0/3894766231 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fc07c065380 0x7fc07c067830 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fc080001570 tx=0x7fc080009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:41.796 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.787+0000 7fc08d7fa640 1 -- 172.21.15.16:0/3894766231 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fc0840ea500 con 0x7fc0900d1c70 2023-05-22T15:27:41.962 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.955+0000 7fc096c2b640 1 -- 172.21.15.16:0/3894766231 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fc09004dbe0 con 0x7fc0900d1c70 2023-05-22T15:27:41.963 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.955+0000 7fc08d7fa640 1 -- 172.21.15.16:0/3894766231 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v29) v1 ==== 80+0+0 (secure 0 0 0) 0x7fc0840edb20 con 0x7fc0900d1c70 2023-05-22T15:27:41.963 INFO:teuthology.orchestra.run.smithi016.stderr:listed 0 entries 2023-05-22T15:27:41.966 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.959+0000 7fc096c2b640 1 -- 172.21.15.16:0/3894766231 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fc07c065380 msgr2=0x7fc07c067830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:41.966 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.959+0000 7fc096c2b640 1 --2- 172.21.15.16:0/3894766231 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fc07c065380 0x7fc07c067830 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fc080001570 tx=0x7fc080009040 comp rx=0 tx=0).stop 2023-05-22T15:27:41.966 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.959+0000 7fc096c2b640 1 -- 172.21.15.16:0/3894766231 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc0900d1c70 msgr2=0x7fc090152890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:41.966 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.959+0000 7fc096c2b640 1 --2- 172.21.15.16:0/3894766231 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc0900d1c70 0x7fc090152890 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fc084019ff0 tx=0x7fc084019160 comp rx=0 tx=0).stop 2023-05-22T15:27:41.967 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.959+0000 7fc096c2b640 1 -- 172.21.15.16:0/3894766231 shutdown_connections 2023-05-22T15:27:41.967 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.959+0000 7fc096c2b640 1 --2- 172.21.15.16:0/3894766231 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc090159740 0x7fc0900caef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:41.967 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.959+0000 7fc096c2b640 1 --2- 172.21.15.16:0/3894766231 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc0900d1c70 0x7fc090152890 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:41.967 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.959+0000 7fc096c2b640 1 --2- 172.21.15.16:0/3894766231 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fc07c065380 0x7fc07c067830 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:41.967 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.959+0000 7fc096c2b640 1 --2- 172.21.15.16:0/3894766231 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc0900cec00 0x7fc090160330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:41.968 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.959+0000 7fc096c2b640 1 -- 172.21.15.16:0/3894766231 >> 172.21.15.16:0/3894766231 conn(0x7fc09006c2f0 msgr2=0x7fc0901504a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:41.968 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.959+0000 7fc096c2b640 1 -- 172.21.15.16:0/3894766231 shutdown_connections 2023-05-22T15:27:41.968 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:41.959+0000 7fc096c2b640 1 -- 172.21.15.16:0/3894766231 wait complete. 2023-05-22T15:27:41.983 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-05-22T15:27:41.983 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2023-05-22T15:27:41.990 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-05-22T15:27:42.166 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.159+0000 7f4888de0640 1 Processor -- start 2023-05-22T15:27:42.166 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.159+0000 7f4888de0640 1 -- start start 2023-05-22T15:27:42.167 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.159+0000 7f4888de0640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f48840ce160 0x7f488415cf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:42.167 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.159+0000 7f4888de0640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48840cf7f0 con 0x7f488406dd90 2023-05-22T15:27:42.167 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.159+0000 7f4888de0640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f488406c440 con 0x7f488406bfa0 2023-05-22T15:27:42.171 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.159+0000 7f4888de0640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f488406c720 con 0x7f48840ce160 2023-05-22T15:27:42.171 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.159+0000 7f488259c640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f488406dd90 0x7f488406e160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:45602/0 (socket says 172.21.15.16:45602) 2023-05-22T15:27:42.171 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.159+0000 7f488259c640 1 -- 172.21.15.16:0/2380817362 learned_addr learned my addr 172.21.15.16:0/2380817362 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:27:42.175 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.159+0000 7f4882d9d640 1 --2- 172.21.15.16:0/2380817362 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f48840ce160 0x7f488415cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:42.175 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.159+0000 7f488159a640 1 -- 172.21.15.16:0/2380817362 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1175259014 0 0) 0x7f48840cf7f0 con 0x7f488406dd90 2023-05-22T15:27:42.175 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.159+0000 7f488159a640 1 -- 172.21.15.16:0/2380817362 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f486c003530 con 0x7f488406dd90 2023-05-22T15:27:42.175 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.159+0000 7f488159a640 1 -- 172.21.15.16:0/2380817362 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3045488458 0 0) 0x7f486c003530 con 0x7f488406dd90 2023-05-22T15:27:42.175 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.159+0000 7f488159a640 1 -- 172.21.15.16:0/2380817362 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f48840ce160 msgr2=0x7f488415cf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:42.176 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.159+0000 7f488159a640 1 --2- 172.21.15.16:0/2380817362 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f48840ce160 0x7f488415cf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:42.176 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.159+0000 7f488159a640 1 -- 172.21.15.16:0/2380817362 >> v1:172.21.15.62:6789/0 conn(0x7f488406bfa0 legacy=0x7f488415fb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:42.176 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.159+0000 7f488159a640 1 -- 172.21.15.16:0/2380817362 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48840cfa90 con 0x7f488406dd90 2023-05-22T15:27:42.176 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.159+0000 7f488159a640 1 -- 172.21.15.16:0/2380817362 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7f4874001680 con 0x7f488406dd90 2023-05-22T15:27:42.176 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.159+0000 7f488159a640 1 -- 172.21.15.16:0/2380817362 >> v1:172.21.15.16:6789/0 conn(0x7f488406dd90 legacy=0x7f488406e160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:42.177 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.159+0000 7f488159a640 1 --2- 172.21.15.16:0/2380817362 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f486c0039b0 0x7f486c003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:42.177 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.159+0000 7f488159a640 1 --2- 172.21.15.16:0/2380817362 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f48840ce160 0x7f486c0047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:42.177 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.159+0000 7f488159a640 1 --2- 172.21.15.16:0/2380817362 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f486c004d10 0x7f486c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:42.177 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.159+0000 7f488159a640 1 -- 172.21.15.16:0/2380817362 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f488406c720 con 0x7f486c0039b0 2023-05-22T15:27:42.177 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.159+0000 7f488159a640 1 -- 172.21.15.16:0/2380817362 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f486c00ba40 con 0x7f486c004d10 2023-05-22T15:27:42.177 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.159+0000 7f488159a640 1 -- 172.21.15.16:0/2380817362 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f486c00bbd0 con 0x7f48840ce160 2023-05-22T15:27:42.177 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.159+0000 7f488259c640 1 --2- 172.21.15.16:0/2380817362 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f486c0039b0 0x7f486c003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:42.178 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.159+0000 7f4882d9d640 1 --2- 172.21.15.16:0/2380817362 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f486c004d10 0x7f486c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:42.178 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.159+0000 7f488259c640 1 -- 172.21.15.16:0/2380817362 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f48840ce160 msgr2=0x7f486c0047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:42.178 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.159+0000 7f488259c640 1 --2- 172.21.15.16:0/2380817362 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f48840ce160 0x7f486c0047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:42.178 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.159+0000 7f488259c640 1 -- 172.21.15.16:0/2380817362 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f486c004d10 msgr2=0x7f486c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:42.178 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.159+0000 7f488259c640 1 --2- 172.21.15.16:0/2380817362 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f486c004d10 0x7f486c005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:42.178 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.159+0000 7f488259c640 1 -- 172.21.15.16:0/2380817362 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f486c003530 con 0x7f486c0039b0 2023-05-22T15:27:42.178 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.163+0000 7f488259c640 1 --2- 172.21.15.16:0/2380817362 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f486c0039b0 0x7f486c003dc0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f487400c240 tx=0x7f487401cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1cf1313249e56f70 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:42.178 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.163+0000 7f488159a640 1 -- 172.21.15.16:0/2380817362 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48740a5b40 con 0x7f486c0039b0 2023-05-22T15:27:42.179 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.163+0000 7f488159a640 1 -- 172.21.15.16:0/2380817362 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48740a5ca0 con 0x7f486c0039b0 2023-05-22T15:27:42.179 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.163+0000 7f488159a640 1 -- 172.21.15.16:0/2380817362 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4874005b20 con 0x7f486c0039b0 2023-05-22T15:27:42.179 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.163+0000 7f4888de0640 1 -- 172.21.15.16:0/2380817362 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f486c0039b0 msgr2=0x7f486c003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:42.179 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.163+0000 7f4888de0640 1 --2- 172.21.15.16:0/2380817362 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f486c0039b0 0x7f486c003dc0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f487400c240 tx=0x7f487401cc40 comp rx=0 tx=0).stop 2023-05-22T15:27:42.179 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.163+0000 7f4882d9d640 1 -- 172.21.15.16:0/2380817362 reap_dead start 2023-05-22T15:27:42.179 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.163+0000 7f4888de0640 1 -- 172.21.15.16:0/2380817362 shutdown_connections 2023-05-22T15:27:42.179 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.163+0000 7f4888de0640 1 -- 172.21.15.16:0/2380817362 >> 172.21.15.16:0/2380817362 conn(0x7f488406f5c0 msgr2=0x7f488406f9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:42.180 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.163+0000 7f4888de0640 1 -- 172.21.15.16:0/2380817362 shutdown_connections 2023-05-22T15:27:42.180 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.163+0000 7f4888de0640 1 -- 172.21.15.16:0/2380817362 wait complete. 2023-05-22T15:27:42.180 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.163+0000 7f4888de0640 1 Processor -- start 2023-05-22T15:27:42.180 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.163+0000 7f4888de0640 1 -- start start 2023-05-22T15:27:42.180 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.167+0000 7f4888de0640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f486c0039b0 0x7f4884157840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:42.180 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.167+0000 7f4888de0640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4884157f90 0x7f48840ab9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:42.180 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.167+0000 7f4888de0640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f48841584c0 0x7f48840c8780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:42.180 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.167+0000 7f4888de0640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48840c8610 con 0x7f486c0039b0 2023-05-22T15:27:42.181 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.167+0000 7f4888de0640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48840b5c20 con 0x7f48841584c0 2023-05-22T15:27:42.181 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.167+0000 7f4888de0640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48840b4c80 con 0x7f4884157f90 2023-05-22T15:27:42.181 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.167+0000 7f488259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f486c0039b0 0x7f4884157840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:42.181 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.167+0000 7f488259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f486c0039b0 0x7f4884157840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41556/0 (socket says 172.21.15.16:41556) 2023-05-22T15:27:42.181 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.167+0000 7f488259c640 1 -- 172.21.15.16:0/1879539578 learned_addr learned my addr 172.21.15.16:0/1879539578 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:42.181 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.167+0000 7f4882d9d640 1 --2- 172.21.15.16:0/1879539578 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f48841584c0 0x7f48840c8780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:42.181 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.167+0000 7f4881d9b640 1 --2- 172.21.15.16:0/1879539578 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4884157f90 0x7f48840ab9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:42.181 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.167+0000 7f488259c640 1 -- 172.21.15.16:0/1879539578 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4884157f90 msgr2=0x7f48840ab9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:42.182 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.167+0000 7f488259c640 1 --2- 172.21.15.16:0/1879539578 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4884157f90 0x7f48840ab9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:42.182 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.167+0000 7f488259c640 1 -- 172.21.15.16:0/1879539578 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f48841584c0 msgr2=0x7f48840c8780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:42.182 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.167+0000 7f488259c640 1 --2- 172.21.15.16:0/1879539578 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f48841584c0 0x7f48840c8780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:42.182 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.167+0000 7f488259c640 1 -- 172.21.15.16:0/1879539578 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48840b2c10 con 0x7f486c0039b0 2023-05-22T15:27:42.182 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.167+0000 7f4882d9d640 1 --2- 172.21.15.16:0/1879539578 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f48841584c0 0x7f48840c8780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:27:42.182 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.167+0000 7f488259c640 1 --2- 172.21.15.16:0/1879539578 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f486c0039b0 0x7f4884157840 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f4874098660 tx=0x7f486c005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:42.182 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.167+0000 7f4872ffd640 1 -- 172.21.15.16:0/1879539578 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48740b4070 con 0x7f486c0039b0 2023-05-22T15:27:42.183 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.167+0000 7f4872ffd640 1 -- 172.21.15.16:0/1879539578 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4874001440 con 0x7f486c0039b0 2023-05-22T15:27:42.183 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.167+0000 7f4872ffd640 1 -- 172.21.15.16:0/1879539578 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48740047f0 con 0x7f486c0039b0 2023-05-22T15:27:42.183 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.167+0000 7f4888de0640 1 -- 172.21.15.16:0/1879539578 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48840b2a80 con 0x7f486c0039b0 2023-05-22T15:27:42.183 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.167+0000 7f4888de0640 1 -- 172.21.15.16:0/1879539578 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f488415c130 con 0x7f486c0039b0 2023-05-22T15:27:42.184 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.167+0000 7f4888de0640 1 -- 172.21.15.16:0/1879539578 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f488404dbe0 con 0x7f486c0039b0 2023-05-22T15:27:42.184 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.167+0000 7f4872ffd640 1 -- 172.21.15.16:0/1879539578 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7f4874004950 con 0x7f486c0039b0 2023-05-22T15:27:42.184 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.167+0000 7f4872ffd640 1 --2- 172.21.15.16:0/1879539578 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f4864065380 0x7f4864067830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:42.184 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.167+0000 7f4872ffd640 1 -- 172.21.15.16:0/1879539578 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f487411ca40 con 0x7f486c0039b0 2023-05-22T15:27:42.184 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.167+0000 7f4881d9b640 1 --2- 172.21.15.16:0/1879539578 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f4864065380 0x7f4864067830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:42.185 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.167+0000 7f4881d9b640 1 --2- 172.21.15.16:0/1879539578 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f4864065380 0x7f4864067830 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f48840b1520 tx=0x7f4878009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:42.185 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.171+0000 7f4872ffd640 1 -- 172.21.15.16:0/1879539578 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f487411ee60 con 0x7f486c0039b0 2023-05-22T15:27:42.335 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.327+0000 7f4888de0640 1 -- 172.21.15.16:0/1879539578 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f48840afa40 con 0x7f486c0039b0 2023-05-22T15:27:42.337 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.331+0000 7f4872ffd640 1 -- 172.21.15.16:0/1879539578 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v29) v1 ==== 74+0+19700 (secure 0 0 0) 0x7f48740ea420 con 0x7f486c0039b0 2023-05-22T15:27:42.338 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:27:42.338 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":29,"fsid":"b2719834-7847-4d98-90a6-f65d57f85c49","created":"2023-05-22T15:26:25.259314+0000","modified":"2023-05-22T15:27:31.182317+0000","last_up_change":"2023-05-22T15:26:36.448524+0000","last_in_change":"2023-05-22T15:26:27.675668+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-05-22T15:26:38.019781+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-05-22T15:26:43.570514+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-05-22T15:26:47.879159+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-05-22T15:26:48.522130+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data_ec","create_time":"2023-05-22T15:26:51.616142+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"9e7ff854-b25f-4cbb-a5c2-1799d9c53d2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":37021},{"type":"v1","addr":"172.21.15.16:6825","nonce":37021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":37021},{"type":"v1","addr":"172.21.15.16:6827","nonce":37021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":37021},{"type":"v1","addr":"172.21.15.16:6831","nonce":37021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":37021},{"type":"v1","addr":"172.21.15.16:6829","nonce":37021}]},"public_addr":"172.21.15.16:6825/37021","cluster_addr":"172.21.15.16:6827/37021","heartbeat_back_addr":"172.21.15.16:6831/37021","heartbeat_front_addr":"172.21.15.16:6829/37021","state":["exists","up"]},{"osd":1,"uuid":"4702fbc9-a927-4565-aca5-41dc69cc8264","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":37008},{"type":"v1","addr":"172.21.15.16:6801","nonce":37008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":37008},{"type":"v1","addr":"172.21.15.16:6803","nonce":37008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":37008},{"type":"v1","addr":"172.21.15.16:6807","nonce":37008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":37008},{"type":"v1","addr":"172.21.15.16:6805","nonce":37008}]},"public_addr":"172.21.15.16:6801/37008","cluster_addr":"172.21.15.16:6803/37008","heartbeat_back_addr":"172.21.15.16:6807/37008","heartbeat_front_addr":"172.21.15.16:6805/37008","state":["exists","up"]},{"osd":2,"uuid":"1e7bba46-4eba-47ef-bb81-5e2d371638ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":37009},{"type":"v1","addr":"172.21.15.16:6809","nonce":37009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":37009},{"type":"v1","addr":"172.21.15.16:6811","nonce":37009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":37009},{"type":"v1","addr":"172.21.15.16:6815","nonce":37009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":37009},{"type":"v1","addr":"172.21.15.16:6813","nonce":37009}]},"public_addr":"172.21.15.16:6809/37009","cluster_addr":"172.21.15.16:6811/37009","heartbeat_back_addr":"172.21.15.16:6815/37009","heartbeat_front_addr":"172.21.15.16:6813/37009","state":["exists","up"]},{"osd":3,"uuid":"383cacc0-5a88-4904-990e-498ca59ef48a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":37020},{"type":"v1","addr":"172.21.15.16:6817","nonce":37020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":37020},{"type":"v1","addr":"172.21.15.16:6819","nonce":37020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":37020},{"type":"v1","addr":"172.21.15.16:6823","nonce":37020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":37020},{"type":"v1","addr":"172.21.15.16:6821","nonce":37020}]},"public_addr":"172.21.15.16:6817/37020","cluster_addr":"172.21.15.16:6819/37020","heartbeat_back_addr":"172.21.15.16:6823/37020","heartbeat_front_addr":"172.21.15.16:6821/37020","state":["exists","up"]},{"osd":4,"uuid":"970500ef-95b7-42c1-b080-eed4fae7013d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":37353},{"type":"v1","addr":"172.21.15.62:6817","nonce":37353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6819","nonce":37353},{"type":"v1","addr":"172.21.15.62:6822","nonce":37353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37353},{"type":"v1","addr":"172.21.15.62:6831","nonce":37353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37353},{"type":"v1","addr":"172.21.15.62:6826","nonce":37353}]},"public_addr":"172.21.15.62:6817/37353","cluster_addr":"172.21.15.62:6822/37353","heartbeat_back_addr":"172.21.15.62:6831/37353","heartbeat_front_addr":"172.21.15.62:6826/37353","state":["exists","up"]},{"osd":5,"uuid":"022d67db-d4d8-445b-b1b9-8b446a00f5c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":37352},{"type":"v1","addr":"172.21.15.62:6801","nonce":37352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":37352},{"type":"v1","addr":"172.21.15.62:6803","nonce":37352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":37352},{"type":"v1","addr":"172.21.15.62:6807","nonce":37352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":37352},{"type":"v1","addr":"172.21.15.62:6805","nonce":37352}]},"public_addr":"172.21.15.62:6801/37352","cluster_addr":"172.21.15.62:6803/37352","heartbeat_back_addr":"172.21.15.62:6807/37352","heartbeat_front_addr":"172.21.15.62:6805/37352","state":["exists","up"]},{"osd":6,"uuid":"42bd372b-6688-4a1b-8407-e16e8c266e33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":37354},{"type":"v1","addr":"172.21.15.62:6820","nonce":37354}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":37354},{"type":"v1","addr":"172.21.15.62:6823","nonce":37354}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6829","nonce":37354},{"type":"v1","addr":"172.21.15.62:6830","nonce":37354}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6825","nonce":37354},{"type":"v1","addr":"172.21.15.62:6827","nonce":37354}]},"public_addr":"172.21.15.62:6820/37354","cluster_addr":"172.21.15.62:6823/37354","heartbeat_back_addr":"172.21.15.62:6830/37354","heartbeat_front_addr":"172.21.15.62:6827/37354","state":["exists","up"]},{"osd":7,"uuid":"ac70b2a3-f41b-4bf2-b227-7f2cbb55426d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37355},{"type":"v1","addr":"172.21.15.62:6809","nonce":37355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37355},{"type":"v1","addr":"172.21.15.62:6811","nonce":37355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":37355},{"type":"v1","addr":"172.21.15.62:6815","nonce":37355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":37355},{"type":"v1","addr":"172.21.15.62:6813","nonce":37355}]},"public_addr":"172.21.15.62:6809/37355","cluster_addr":"172.21.15.62:6811/37355","heartbeat_back_addr":"172.21.15.62:6815/37355","heartbeat_front_addr":"172.21.15.62:6813/37355","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.368896+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.107273+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.276888+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.287502+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.124791+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.251937+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.255679+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.232797+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"4.23","mappings":[{"from":3,"to":2}]},{"pgid":"4.31","mappings":[{"from":1,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-05-22T15:27:42.339 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.331+0000 7f4888de0640 1 -- 172.21.15.16:0/1879539578 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f4864065380 msgr2=0x7f4864067830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:42.340 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.331+0000 7f4888de0640 1 --2- 172.21.15.16:0/1879539578 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f4864065380 0x7f4864067830 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f48840b1520 tx=0x7f4878009040 comp rx=0 tx=0).stop 2023-05-22T15:27:42.340 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.331+0000 7f4888de0640 1 -- 172.21.15.16:0/1879539578 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f486c0039b0 msgr2=0x7f4884157840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:42.340 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.331+0000 7f4888de0640 1 --2- 172.21.15.16:0/1879539578 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f486c0039b0 0x7f4884157840 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f4874098660 tx=0x7f486c005800 comp rx=0 tx=0).stop 2023-05-22T15:27:42.340 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.331+0000 7f4888de0640 1 -- 172.21.15.16:0/1879539578 shutdown_connections 2023-05-22T15:27:42.340 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.331+0000 7f4888de0640 1 --2- 172.21.15.16:0/1879539578 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f48841584c0 0x7f48840c8780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:42.340 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.331+0000 7f4888de0640 1 --2- 172.21.15.16:0/1879539578 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f4864065380 0x7f4864067830 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:42.340 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.331+0000 7f4888de0640 1 --2- 172.21.15.16:0/1879539578 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4884157f90 0x7f48840ab9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:42.341 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.331+0000 7f4888de0640 1 --2- 172.21.15.16:0/1879539578 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f486c0039b0 0x7f4884157840 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:42.341 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.331+0000 7f4888de0640 1 -- 172.21.15.16:0/1879539578 >> 172.21.15.16:0/1879539578 conn(0x7f488406f5c0 msgr2=0x7f488415d5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:42.341 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.331+0000 7f4888de0640 1 -- 172.21.15.16:0/1879539578 shutdown_connections 2023-05-22T15:27:42.341 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.331+0000 7f4888de0640 1 -- 172.21.15.16:0/1879539578 wait complete. 2023-05-22T15:27:42.356 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-05-22T15:26:38.019781+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-05-22T15:26:43.570514+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2023-05-22T15:26:47.879159+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2023-05-22T15:26:48.522130+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-05-22T15:26:51.616142+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '27', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-05-22T15:27:42.357 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-05-22T15:27:42.474 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.467+0000 7fbd3e3c0640 1 Processor -- start 2023-05-22T15:27:42.474 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.467+0000 7fbd3e3c0640 1 -- start start 2023-05-22T15:27:42.475 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.467+0000 7fbd3e3c0640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbd3815bca0 0x7fbd38160160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:42.475 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.467+0000 7fbd3e3c0640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fbd380b3ee0 0x7fbd380b4330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:42.476 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.467+0000 7fbd3e3c0640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fbd380b4870 0x7fbd380ce900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:42.476 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.467+0000 7fbd3e3c0640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd3815c070 con 0x7fbd380b4870 2023-05-22T15:27:42.476 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.467+0000 7fbd3e3c0640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd380d0400 con 0x7fbd380b3ee0 2023-05-22T15:27:42.476 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.467+0000 7fbd3e3c0640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd381606a0 con 0x7fbd3815bca0 2023-05-22T15:27:42.476 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.467+0000 7fbd3c95d640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fbd380b4870 0x7fbd380ce900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:42.476 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.467+0000 7fbd3c95d640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fbd380b4870 0x7fbd380ce900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41564/0 (socket says 172.21.15.16:41564) 2023-05-22T15:27:42.476 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.467+0000 7fbd3c95d640 1 -- 172.21.15.16:0/240466586 learned_addr learned my addr 172.21.15.16:0/240466586 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:42.477 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.467+0000 7fbd377fe640 1 --2- 172.21.15.16:0/240466586 >> v2:172.21.15.62:3300/0 conn(0x7fbd380b3ee0 0x7fbd380b4330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:42.477 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.467+0000 7fbd37fff640 1 --2- 172.21.15.16:0/240466586 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbd3815bca0 0x7fbd38160160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:42.477 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.467+0000 7fbd3c95d640 1 -- 172.21.15.16:0/240466586 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbd3815bca0 msgr2=0x7fbd38160160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:42.477 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.467+0000 7fbd3c95d640 1 --2- 172.21.15.16:0/240466586 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbd3815bca0 0x7fbd38160160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:42.477 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.467+0000 7fbd3c95d640 1 -- 172.21.15.16:0/240466586 >> v2:172.21.15.62:3300/0 conn(0x7fbd380b3ee0 msgr2=0x7fbd380b4330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:42.477 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.467+0000 7fbd3c95d640 1 --2- 172.21.15.16:0/240466586 >> v2:172.21.15.62:3300/0 conn(0x7fbd380b3ee0 0x7fbd380b4330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:42.477 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.467+0000 7fbd3c95d640 1 -- 172.21.15.16:0/240466586 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd380d00d0 con 0x7fbd380b4870 2023-05-22T15:27:42.478 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.467+0000 7fbd377fe640 1 --2- 172.21.15.16:0/240466586 >> v2:172.21.15.62:3300/0 conn(0x7fbd380b3ee0 0x7fbd380b4330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:27:42.478 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.471+0000 7fbd3c95d640 1 --2- 172.21.15.16:0/240466586 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbd380b4870 0x7fbd380ce900 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fbd38065e50 tx=0x7fbd3001c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c20c5d176b39052e server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:42.478 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.471+0000 7fbd36ffd640 1 -- 172.21.15.16:0/240466586 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd300a5790 con 0x7fbd380b4870 2023-05-22T15:27:42.479 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.471+0000 7fbd36ffd640 1 -- 172.21.15.16:0/240466586 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd300a58f0 con 0x7fbd380b4870 2023-05-22T15:27:42.479 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.471+0000 7fbd36ffd640 1 -- 172.21.15.16:0/240466586 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd300a5c80 con 0x7fbd380b4870 2023-05-22T15:27:42.479 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.471+0000 7fbd3e3c0640 1 -- 172.21.15.16:0/240466586 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbd380b4870 msgr2=0x7fbd380ce900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:42.479 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.471+0000 7fbd3e3c0640 1 --2- 172.21.15.16:0/240466586 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbd380b4870 0x7fbd380ce900 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fbd38065e50 tx=0x7fbd3001c910 comp rx=0 tx=0).stop 2023-05-22T15:27:42.479 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.471+0000 7fbd3e3c0640 1 -- 172.21.15.16:0/240466586 shutdown_connections 2023-05-22T15:27:42.479 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.471+0000 7fbd3e3c0640 1 --2- 172.21.15.16:0/240466586 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbd380b4870 0x7fbd380ce900 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:42.480 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.471+0000 7fbd3e3c0640 1 --2- 172.21.15.16:0/240466586 >> v2:172.21.15.62:3300/0 conn(0x7fbd380b3ee0 0x7fbd380b4330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:42.480 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.471+0000 7fbd3e3c0640 1 --2- 172.21.15.16:0/240466586 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbd3815bca0 0x7fbd38160160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:42.480 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.471+0000 7fbd3e3c0640 1 -- 172.21.15.16:0/240466586 >> 172.21.15.16:0/240466586 conn(0x7fbd380657f0 msgr2=0x7fbd38070e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:42.480 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.471+0000 7fbd3e3c0640 1 -- 172.21.15.16:0/240466586 shutdown_connections 2023-05-22T15:27:42.480 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.471+0000 7fbd3e3c0640 1 -- 172.21.15.16:0/240466586 wait complete. 2023-05-22T15:27:42.482 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.471+0000 7fbd3e3c0640 1 Processor -- start 2023-05-22T15:27:42.482 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.471+0000 7fbd3e3c0640 1 -- start start 2023-05-22T15:27:42.482 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.471+0000 7fbd3e3c0640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbd380b3ee0 0x7fbd3814dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:42.482 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.471+0000 7fbd3e3c0640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbd380b4870 0x7fbd38154680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:42.482 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.471+0000 7fbd3e3c0640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbd3815bca0 0x7fbd38154bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:42.482 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.471+0000 7fbd3e3c0640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd380ca110 con 0x7fbd3815bca0 2023-05-22T15:27:42.483 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.471+0000 7fbd3e3c0640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd380ca3f0 con 0x7fbd380b4870 2023-05-22T15:27:42.483 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.471+0000 7fbd3e3c0640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd380ca6d0 con 0x7fbd380b3ee0 2023-05-22T15:27:42.483 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.471+0000 7fbd3c95d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbd3815bca0 0x7fbd38154bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:42.483 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.471+0000 7fbd3c95d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbd3815bca0 0x7fbd38154bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41566/0 (socket says 172.21.15.16:41566) 2023-05-22T15:27:42.483 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.471+0000 7fbd3c95d640 1 -- 172.21.15.16:0/3608758276 learned_addr learned my addr 172.21.15.16:0/3608758276 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:42.483 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.471+0000 7fbd37fff640 1 --2- 172.21.15.16:0/3608758276 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbd380b3ee0 0x7fbd3814dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:42.483 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.471+0000 7fbd377fe640 1 --2- 172.21.15.16:0/3608758276 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbd380b4870 0x7fbd38154680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:42.483 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.471+0000 7fbd3c95d640 1 -- 172.21.15.16:0/3608758276 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbd380b3ee0 msgr2=0x7fbd3814dfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:42.484 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.471+0000 7fbd3c95d640 1 --2- 172.21.15.16:0/3608758276 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbd380b3ee0 0x7fbd3814dfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:42.484 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.471+0000 7fbd3c95d640 1 -- 172.21.15.16:0/3608758276 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbd380b4870 msgr2=0x7fbd38154680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:42.484 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.471+0000 7fbd3c95d640 1 --2- 172.21.15.16:0/3608758276 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbd380b4870 0x7fbd38154680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:42.484 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.471+0000 7fbd3c95d640 1 -- 172.21.15.16:0/3608758276 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd30009b30 con 0x7fbd3815bca0 2023-05-22T15:27:42.484 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.471+0000 7fbd3c95d640 1 --2- 172.21.15.16:0/3608758276 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbd3815bca0 0x7fbd38154bc0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fbd300af040 tx=0x7fbd300196b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:42.484 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.471+0000 7fbd34ff9640 1 -- 172.21.15.16:0/3608758276 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd300b8670 con 0x7fbd3815bca0 2023-05-22T15:27:42.484 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.471+0000 7fbd34ff9640 1 -- 172.21.15.16:0/3608758276 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd300b87d0 con 0x7fbd3815bca0 2023-05-22T15:27:42.485 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.471+0000 7fbd3e3c0640 1 -- 172.21.15.16:0/3608758276 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd380cd1a0 con 0x7fbd3815bca0 2023-05-22T15:27:42.485 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.475+0000 7fbd34ff9640 1 -- 172.21.15.16:0/3608758276 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd300b8b60 con 0x7fbd3815bca0 2023-05-22T15:27:42.485 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.475+0000 7fbd3e3c0640 1 -- 172.21.15.16:0/3608758276 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd380b4cb0 con 0x7fbd3815bca0 2023-05-22T15:27:42.485 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.475+0000 7fbd3e3c0640 1 -- 172.21.15.16:0/3608758276 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd380d0870 con 0x7fbd3815bca0 2023-05-22T15:27:42.486 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.475+0000 7fbd34ff9640 1 -- 172.21.15.16:0/3608758276 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7fbd300b8cc0 con 0x7fbd3815bca0 2023-05-22T15:27:42.486 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.475+0000 7fbd34ff9640 1 --2- 172.21.15.16:0/3608758276 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fbd18065380 0x7fbd18067830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:42.486 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.475+0000 7fbd34ff9640 1 -- 172.21.15.16:0/3608758276 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6871+0+0 (secure 0 0 0) 0x7fbd3011e280 con 0x7fbd3815bca0 2023-05-22T15:27:42.486 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.475+0000 7fbd37fff640 1 --2- 172.21.15.16:0/3608758276 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fbd18065380 0x7fbd18067830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:42.486 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.475+0000 7fbd37fff640 1 --2- 172.21.15.16:0/3608758276 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fbd18065380 0x7fbd18067830 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fbd200096f0 tx=0x7fbd20008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:42.487 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.479+0000 7fbd34ff9640 1 -- 172.21.15.16:0/3608758276 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fbd300ebc70 con 0x7fbd3815bca0 2023-05-22T15:27:42.639 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.631+0000 7fbd3e3c0640 1 -- 172.21.15.16:0/3608758276 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fbd380ca810 con 0x7fbd3815bca0 2023-05-22T15:27:42.640 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.631+0000 7fbd34ff9640 1 -- 172.21.15.16:0/3608758276 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v29) v1 ==== 93+0+10 (secure 0 0 0) 0x7fbd380ca810 con 0x7fbd3815bca0 2023-05-22T15:27:42.640 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2023-05-22T15:27:42.642 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.635+0000 7fbd3e3c0640 1 -- 172.21.15.16:0/3608758276 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fbd18065380 msgr2=0x7fbd18067830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:42.643 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.635+0000 7fbd3e3c0640 1 --2- 172.21.15.16:0/3608758276 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fbd18065380 0x7fbd18067830 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fbd200096f0 tx=0x7fbd20008040 comp rx=0 tx=0).stop 2023-05-22T15:27:42.643 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.635+0000 7fbd3e3c0640 1 -- 172.21.15.16:0/3608758276 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbd3815bca0 msgr2=0x7fbd38154bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:42.643 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.635+0000 7fbd3e3c0640 1 --2- 172.21.15.16:0/3608758276 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbd3815bca0 0x7fbd38154bc0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fbd300af040 tx=0x7fbd300196b0 comp rx=0 tx=0).stop 2023-05-22T15:27:42.644 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.635+0000 7fbd3e3c0640 1 -- 172.21.15.16:0/3608758276 shutdown_connections 2023-05-22T15:27:42.644 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.635+0000 7fbd3e3c0640 1 --2- 172.21.15.16:0/3608758276 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbd3815bca0 0x7fbd38154bc0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:42.644 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.635+0000 7fbd3e3c0640 1 --2- 172.21.15.16:0/3608758276 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbd380b4870 0x7fbd38154680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:42.644 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.635+0000 7fbd3e3c0640 1 --2- 172.21.15.16:0/3608758276 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fbd18065380 0x7fbd18067830 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:42.645 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.635+0000 7fbd3e3c0640 1 --2- 172.21.15.16:0/3608758276 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbd380b3ee0 0x7fbd3814dfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:42.645 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.635+0000 7fbd3e3c0640 1 -- 172.21.15.16:0/3608758276 >> 172.21.15.16:0/3608758276 conn(0x7fbd380657f0 msgr2=0x7fbd380b3750 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:42.645 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.635+0000 7fbd3e3c0640 1 -- 172.21.15.16:0/3608758276 shutdown_connections 2023-05-22T15:27:42.646 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.635+0000 7fbd3e3c0640 1 -- 172.21.15.16:0/3608758276 wait complete. 2023-05-22T15:27:42.659 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-05-22T15:27:42.782 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.775+0000 7f2efb7a0640 1 Processor -- start 2023-05-22T15:27:42.782 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.775+0000 7f2efb7a0640 1 -- start start 2023-05-22T15:27:42.783 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.775+0000 7f2efb7a0640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2ef415bca0 0x7f2ef40cda80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:42.784 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.775+0000 7f2efb7a0640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f2ef40b5a50 0x7f2ef40d2900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:42.784 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.775+0000 7f2efb7a0640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ef40cccc0 con 0x7f2ef40d2480 2023-05-22T15:27:42.784 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.775+0000 7f2efb7a0640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ef40cd660 con 0x7f2ef415bca0 2023-05-22T15:27:42.784 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.775+0000 7f2efb7a0640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ef40d2e40 con 0x7f2ef40b5a50 2023-05-22T15:27:42.784 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.775+0000 7f2ef953c640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2ef415bca0 0x7f2ef40cda80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:42.784 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.775+0000 7f2ef953c640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2ef415bca0 0x7f2ef40cda80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41574/0 (socket says 172.21.15.16:41574) 2023-05-22T15:27:42.785 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.775+0000 7f2ef953c640 1 -- 172.21.15.16:0/3627267515 learned_addr learned my addr 172.21.15.16:0/3627267515 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:42.785 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.775+0000 7f2ef9d3d640 1 --2- 172.21.15.16:0/3627267515 >> v2:172.21.15.62:3300/0 conn(0x7f2ef40b5a50 0x7f2ef40d2900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:42.785 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.775+0000 7f2ef953c640 1 -- 172.21.15.16:0/3627267515 >> v2:172.21.15.62:3300/0 conn(0x7f2ef40b5a50 msgr2=0x7f2ef40d2900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:42.785 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.775+0000 7f2ef953c640 1 --2- 172.21.15.16:0/3627267515 >> v2:172.21.15.62:3300/0 conn(0x7f2ef40b5a50 0x7f2ef40d2900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:42.785 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.775+0000 7f2ef953c640 1 -- 172.21.15.16:0/3627267515 >> v1:172.21.15.62:6789/0 conn(0x7f2ef40d2480 legacy=0x7f2ef40b5340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:42.785 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.775+0000 7f2ef953c640 1 -- 172.21.15.16:0/3627267515 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ef40ccfa0 con 0x7f2ef415bca0 2023-05-22T15:27:42.786 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.779+0000 7f2ef953c640 1 --2- 172.21.15.16:0/3627267515 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2ef415bca0 0x7f2ef40cda80 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f2ef40c5fc0 tx=0x7f2eec01c4c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a001bd6ca471f82e server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:42.786 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.779+0000 7f2ef3fff640 1 -- 172.21.15.16:0/3627267515 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2eec0aa640 con 0x7f2ef415bca0 2023-05-22T15:27:42.786 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.779+0000 7f2ef3fff640 1 -- 172.21.15.16:0/3627267515 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2eec0aa7a0 con 0x7f2ef415bca0 2023-05-22T15:27:42.786 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.779+0000 7f2ef3fff640 1 -- 172.21.15.16:0/3627267515 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2eec0aab30 con 0x7f2ef415bca0 2023-05-22T15:27:42.787 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.779+0000 7f2efb7a0640 1 -- 172.21.15.16:0/3627267515 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2ef415bca0 msgr2=0x7f2ef40cda80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:42.787 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.779+0000 7f2efb7a0640 1 --2- 172.21.15.16:0/3627267515 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2ef415bca0 0x7f2ef40cda80 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f2ef40c5fc0 tx=0x7f2eec01c4c0 comp rx=0 tx=0).stop 2023-05-22T15:27:42.787 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.779+0000 7f2efb7a0640 1 -- 172.21.15.16:0/3627267515 shutdown_connections 2023-05-22T15:27:42.788 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.779+0000 7f2efb7a0640 1 --2- 172.21.15.16:0/3627267515 >> v2:172.21.15.62:3300/0 conn(0x7f2ef40b5a50 0x7f2ef40d2900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:42.788 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.779+0000 7f2efb7a0640 1 --2- 172.21.15.16:0/3627267515 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2ef415bca0 0x7f2ef40cda80 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:42.788 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.779+0000 7f2efb7a0640 1 -- 172.21.15.16:0/3627267515 >> 172.21.15.16:0/3627267515 conn(0x7f2ef40655c0 msgr2=0x7f2ef4070e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:42.789 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.779+0000 7f2efb7a0640 1 -- 172.21.15.16:0/3627267515 shutdown_connections 2023-05-22T15:27:42.789 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.779+0000 7f2efb7a0640 1 -- 172.21.15.16:0/3627267515 wait complete. 2023-05-22T15:27:42.790 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.779+0000 7f2efb7a0640 1 Processor -- start 2023-05-22T15:27:42.791 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.779+0000 7f2efb7a0640 1 -- start start 2023-05-22T15:27:42.791 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.779+0000 7f2efb7a0640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2ef40b5a50 0x7f2ef41545a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:42.791 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.779+0000 7f2efb7a0640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2ef40d2480 0x7f2ef4156f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:42.791 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.779+0000 7f2efb7a0640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2ef415bca0 0x7f2ef415e1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:42.791 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.779+0000 7f2efb7a0640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ef4149c70 con 0x7f2ef415bca0 2023-05-22T15:27:42.791 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.779+0000 7f2efb7a0640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ef414a650 con 0x7f2ef40b5a50 2023-05-22T15:27:42.791 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.779+0000 7f2efb7a0640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ef4148dc0 con 0x7f2ef40d2480 2023-05-22T15:27:42.792 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.779+0000 7f2ef9d3d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2ef415bca0 0x7f2ef415e1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:42.792 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.779+0000 7f2ef9d3d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2ef415bca0 0x7f2ef415e1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41580/0 (socket says 172.21.15.16:41580) 2023-05-22T15:27:42.792 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.779+0000 7f2ef9d3d640 1 -- 172.21.15.16:0/3823976907 learned_addr learned my addr 172.21.15.16:0/3823976907 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:42.792 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.779+0000 7f2ef953c640 1 --2- 172.21.15.16:0/3823976907 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2ef40b5a50 0x7f2ef41545a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:42.792 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.779+0000 7f2ef8d3b640 1 --2- 172.21.15.16:0/3823976907 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2ef40d2480 0x7f2ef4156f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:42.792 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.783+0000 7f2ef9d3d640 1 -- 172.21.15.16:0/3823976907 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2ef40d2480 msgr2=0x7f2ef4156f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:42.792 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.783+0000 7f2ef9d3d640 1 --2- 172.21.15.16:0/3823976907 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2ef40d2480 0x7f2ef4156f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:42.792 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.783+0000 7f2ef9d3d640 1 -- 172.21.15.16:0/3823976907 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2ef40b5a50 msgr2=0x7f2ef41545a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:42.793 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.783+0000 7f2ef9d3d640 1 --2- 172.21.15.16:0/3823976907 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2ef40b5a50 0x7f2ef41545a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:42.793 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.783+0000 7f2ef9d3d640 1 -- 172.21.15.16:0/3823976907 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2eec0094d0 con 0x7f2ef415bca0 2023-05-22T15:27:42.793 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.783+0000 7f2ef9d3d640 1 --2- 172.21.15.16:0/3823976907 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2ef415bca0 0x7f2ef415e1c0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f2ee400d590 tx=0x7f2ee400da50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:42.793 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.783+0000 7f2ef1ffb640 1 -- 172.21.15.16:0/3823976907 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ee40125f0 con 0x7f2ef415bca0 2023-05-22T15:27:42.793 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.783+0000 7f2ef1ffb640 1 -- 172.21.15.16:0/3823976907 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ee4012770 con 0x7f2ef415bca0 2023-05-22T15:27:42.793 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.783+0000 7f2efb7a0640 1 -- 172.21.15.16:0/3823976907 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ef4160360 con 0x7f2ef415bca0 2023-05-22T15:27:42.793 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.783+0000 7f2ef1ffb640 1 -- 172.21.15.16:0/3823976907 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ee4012b20 con 0x7f2ef415bca0 2023-05-22T15:27:42.794 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.783+0000 7f2efb7a0640 1 -- 172.21.15.16:0/3823976907 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ef40cdfc0 con 0x7f2ef415bca0 2023-05-22T15:27:42.794 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.783+0000 7f2efb7a0640 1 -- 172.21.15.16:0/3823976907 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ef41604f0 con 0x7f2ef415bca0 2023-05-22T15:27:42.794 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.783+0000 7f2ef1ffb640 1 -- 172.21.15.16:0/3823976907 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7f2ee4012c80 con 0x7f2ef415bca0 2023-05-22T15:27:42.795 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.783+0000 7f2ef1ffb640 1 --2- 172.21.15.16:0/3823976907 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f2ee0065380 0x7f2ee0067830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:42.795 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.783+0000 7f2ef1ffb640 1 -- 172.21.15.16:0/3823976907 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f2ee4083a90 con 0x7f2ef415bca0 2023-05-22T15:27:42.795 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.783+0000 7f2ef953c640 1 --2- 172.21.15.16:0/3823976907 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f2ee0065380 0x7f2ee0067830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:42.795 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.783+0000 7f2ef953c640 1 --2- 172.21.15.16:0/3823976907 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f2ee0065380 0x7f2ee0067830 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f2eec0ae000 tx=0x7f2eec0a7000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:42.795 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.787+0000 7f2ef1ffb640 1 -- 172.21.15.16:0/3823976907 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f2ee40514c0 con 0x7f2ef415bca0 2023-05-22T15:27:42.948 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.939+0000 7f2efb7a0640 1 -- 172.21.15.16:0/3823976907 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f2ef404dbe0 con 0x7f2ef415bca0 2023-05-22T15:27:42.949 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.939+0000 7f2ef1ffb640 1 -- 172.21.15.16:0/3823976907 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v29) v1 ==== 92+0+10 (secure 0 0 0) 0x7f2ee4054ae0 con 0x7f2ef415bca0 2023-05-22T15:27:42.949 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 8 2023-05-22T15:27:42.951 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.943+0000 7f2efb7a0640 1 -- 172.21.15.16:0/3823976907 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f2ee0065380 msgr2=0x7f2ee0067830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:42.951 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.943+0000 7f2efb7a0640 1 --2- 172.21.15.16:0/3823976907 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f2ee0065380 0x7f2ee0067830 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f2eec0ae000 tx=0x7f2eec0a7000 comp rx=0 tx=0).stop 2023-05-22T15:27:42.951 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.943+0000 7f2efb7a0640 1 -- 172.21.15.16:0/3823976907 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2ef415bca0 msgr2=0x7f2ef415e1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:42.951 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.943+0000 7f2efb7a0640 1 --2- 172.21.15.16:0/3823976907 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2ef415bca0 0x7f2ef415e1c0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f2ee400d590 tx=0x7f2ee400da50 comp rx=0 tx=0).stop 2023-05-22T15:27:42.951 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.943+0000 7f2efb7a0640 1 -- 172.21.15.16:0/3823976907 shutdown_connections 2023-05-22T15:27:42.952 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.943+0000 7f2efb7a0640 1 --2- 172.21.15.16:0/3823976907 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2ef415bca0 0x7f2ef415e1c0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:42.952 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.943+0000 7f2efb7a0640 1 --2- 172.21.15.16:0/3823976907 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f2ee0065380 0x7f2ee0067830 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:42.952 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.943+0000 7f2efb7a0640 1 --2- 172.21.15.16:0/3823976907 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2ef40d2480 0x7f2ef4156f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:42.952 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.943+0000 7f2efb7a0640 1 --2- 172.21.15.16:0/3823976907 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2ef40b5a50 0x7f2ef41545a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:42.952 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.943+0000 7f2efb7a0640 1 -- 172.21.15.16:0/3823976907 >> 172.21.15.16:0/3823976907 conn(0x7f2ef40655c0 msgr2=0x7f2ef41581f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:42.952 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.943+0000 7f2efb7a0640 1 -- 172.21.15.16:0/3823976907 shutdown_connections 2023-05-22T15:27:42.952 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:42.943+0000 7f2efb7a0640 1 -- 172.21.15.16:0/3823976907 wait complete. 2023-05-22T15:27:42.967 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-05-22T15:27:43.092 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.083+0000 7f0c77d68640 1 Processor -- start 2023-05-22T15:27:43.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.083+0000 7f0c77d68640 1 -- start start 2023-05-22T15:27:43.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.087+0000 7f0c77d68640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f0c700ce480 0x7f0c70071640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:43.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.087+0000 7f0c77d68640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0c700ce850 0x7f0c70160090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:43.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.087+0000 7f0c77d68640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c70069ae0 con 0x7f0c701605d0 2023-05-22T15:27:43.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.087+0000 7f0c77d68640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c7015f990 con 0x7f0c700ce480 2023-05-22T15:27:43.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.087+0000 7f0c77d68640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c70069dc0 con 0x7f0c700ce850 2023-05-22T15:27:43.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.087+0000 7f0c75b04640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f0c700ce480 0x7f0c70071640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:43.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.087+0000 7f0c75b04640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f0c700ce480 0x7f0c70071640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41590/0 (socket says 172.21.15.16:41590) 2023-05-22T15:27:43.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.087+0000 7f0c75b04640 1 -- 172.21.15.16:0/3730439507 learned_addr learned my addr 172.21.15.16:0/3730439507 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:43.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.087+0000 7f0c75303640 1 --2- 172.21.15.16:0/3730439507 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0c700ce850 0x7f0c70160090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:43.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.087+0000 7f0c75b04640 1 -- 172.21.15.16:0/3730439507 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0c700ce850 msgr2=0x7f0c70160090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:43.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.087+0000 7f0c75b04640 1 --2- 172.21.15.16:0/3730439507 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0c700ce850 0x7f0c70160090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:43.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.087+0000 7f0c75b04640 1 -- 172.21.15.16:0/3730439507 >> v1:172.21.15.62:6789/0 conn(0x7f0c701605d0 legacy=0x7f0c7015c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:43.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.087+0000 7f0c75b04640 1 -- 172.21.15.16:0/3730439507 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c7006a0a0 con 0x7f0c700ce480 2023-05-22T15:27:43.096 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.087+0000 7f0c75b04640 1 --2- 172.21.15.16:0/3730439507 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0c700ce480 0x7f0c70071640 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f0c68009e50 tx=0x7f0c6801c9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f83f2eeb1749a1b4 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:43.096 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.087+0000 7f0c74b02640 1 -- 172.21.15.16:0/3730439507 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c680a5730 con 0x7f0c700ce480 2023-05-22T15:27:43.097 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.087+0000 7f0c74b02640 1 -- 172.21.15.16:0/3730439507 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c680a5890 con 0x7f0c700ce480 2023-05-22T15:27:43.097 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.087+0000 7f0c74b02640 1 -- 172.21.15.16:0/3730439507 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c680a5c20 con 0x7f0c700ce480 2023-05-22T15:27:43.097 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.091+0000 7f0c77d68640 1 -- 172.21.15.16:0/3730439507 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0c700ce480 msgr2=0x7f0c70071640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:43.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.091+0000 7f0c77d68640 1 --2- 172.21.15.16:0/3730439507 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0c700ce480 0x7f0c70071640 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f0c68009e50 tx=0x7f0c6801c9a0 comp rx=0 tx=0).stop 2023-05-22T15:27:43.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.091+0000 7f0c77d68640 1 -- 172.21.15.16:0/3730439507 shutdown_connections 2023-05-22T15:27:43.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.091+0000 7f0c77d68640 1 --2- 172.21.15.16:0/3730439507 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0c700ce850 0x7f0c70160090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:43.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.091+0000 7f0c77d68640 1 --2- 172.21.15.16:0/3730439507 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0c700ce480 0x7f0c70071640 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:43.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.091+0000 7f0c77d68640 1 -- 172.21.15.16:0/3730439507 >> 172.21.15.16:0/3730439507 conn(0x7f0c700cc870 msgr2=0x7f0c700cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:43.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.091+0000 7f0c77d68640 1 -- 172.21.15.16:0/3730439507 shutdown_connections 2023-05-22T15:27:43.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.091+0000 7f0c77d68640 1 -- 172.21.15.16:0/3730439507 wait complete. 2023-05-22T15:27:43.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.091+0000 7f0c77d68640 1 Processor -- start 2023-05-22T15:27:43.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.091+0000 7f0c77d68640 1 -- start start 2023-05-22T15:27:43.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.091+0000 7f0c77d68640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0c700ce480 0x7f0c700b1550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:43.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.091+0000 7f0c77d68640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0c700ce850 0x7f0c7014c1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:43.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.091+0000 7f0c77d68640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0c701605d0 0x7f0c70152d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:43.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.091+0000 7f0c77d68640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c700b01e0 con 0x7f0c700ce480 2023-05-22T15:27:43.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.091+0000 7f0c77d68640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c700b0c60 con 0x7f0c701605d0 2023-05-22T15:27:43.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.091+0000 7f0c77d68640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c700b03c0 con 0x7f0c700ce850 2023-05-22T15:27:43.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.091+0000 7f0c75b04640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0c700ce480 0x7f0c700b1550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:43.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.091+0000 7f0c75b04640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0c700ce480 0x7f0c700b1550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41600/0 (socket says 172.21.15.16:41600) 2023-05-22T15:27:43.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.091+0000 7f0c75b04640 1 -- 172.21.15.16:0/462711017 learned_addr learned my addr 172.21.15.16:0/462711017 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:43.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.091+0000 7f0c75303640 1 --2- 172.21.15.16:0/462711017 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0c700ce850 0x7f0c7014c1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:43.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.091+0000 7f0c76305640 1 --2- 172.21.15.16:0/462711017 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0c701605d0 0x7f0c70152d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:43.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.091+0000 7f0c75b04640 1 -- 172.21.15.16:0/462711017 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0c700ce850 msgr2=0x7f0c7014c1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:43.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.091+0000 7f0c75b04640 1 --2- 172.21.15.16:0/462711017 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0c700ce850 0x7f0c7014c1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:43.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.091+0000 7f0c75b04640 1 -- 172.21.15.16:0/462711017 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0c701605d0 msgr2=0x7f0c70152d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:43.103 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.091+0000 7f0c75b04640 1 --2- 172.21.15.16:0/462711017 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0c701605d0 0x7f0c70152d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:43.103 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.091+0000 7f0c75b04640 1 -- 172.21.15.16:0/462711017 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c68009b30 con 0x7f0c700ce480 2023-05-22T15:27:43.103 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.091+0000 7f0c76305640 1 --2- 172.21.15.16:0/462711017 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0c701605d0 0x7f0c70152d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:27:43.103 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.091+0000 7f0c75b04640 1 --2- 172.21.15.16:0/462711017 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0c700ce480 0x7f0c700b1550 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f0c6801b050 tx=0x7f0c68019740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:43.103 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.091+0000 7f0c667fc640 1 -- 172.21.15.16:0/462711017 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c680b8670 con 0x7f0c700ce480 2023-05-22T15:27:43.103 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.091+0000 7f0c667fc640 1 -- 172.21.15.16:0/462711017 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c680b87d0 con 0x7f0c700ce480 2023-05-22T15:27:43.103 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.091+0000 7f0c77d68640 1 -- 172.21.15.16:0/462711017 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c700c6000 con 0x7f0c700ce480 2023-05-22T15:27:43.104 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.091+0000 7f0c667fc640 1 -- 172.21.15.16:0/462711017 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c680b8b80 con 0x7f0c700ce480 2023-05-22T15:27:43.104 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.091+0000 7f0c77d68640 1 -- 172.21.15.16:0/462711017 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c700c5e70 con 0x7f0c700ce480 2023-05-22T15:27:43.104 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.095+0000 7f0c77d68640 1 -- 172.21.15.16:0/462711017 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c700cec20 con 0x7f0c700ce480 2023-05-22T15:27:43.105 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.095+0000 7f0c667fc640 1 -- 172.21.15.16:0/462711017 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7f0c680b8ce0 con 0x7f0c700ce480 2023-05-22T15:27:43.105 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.095+0000 7f0c667fc640 1 --2- 172.21.15.16:0/462711017 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f0c54065380 0x7f0c54067830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:43.105 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.095+0000 7f0c667fc640 1 -- 172.21.15.16:0/462711017 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f0c6811e0a0 con 0x7f0c700ce480 2023-05-22T15:27:43.105 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.095+0000 7f0c75303640 1 --2- 172.21.15.16:0/462711017 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f0c54065380 0x7f0c54067830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:43.105 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.095+0000 7f0c75303640 1 --2- 172.21.15.16:0/462711017 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f0c54065380 0x7f0c54067830 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f0c700aebe0 tx=0x7f0c60008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:43.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.099+0000 7f0c667fc640 1 -- 172.21.15.16:0/462711017 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f0c680eba80 con 0x7f0c700ce480 2023-05-22T15:27:43.252 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.243+0000 7f0c77d68640 1 -- 172.21.15.16:0/462711017 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f0c7004dbe0 con 0x7f0c700ce480 2023-05-22T15:27:43.253 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.243+0000 7f0c667fc640 1 -- 172.21.15.16:0/462711017 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v29) v1 ==== 104+0+11 (secure 0 0 0) 0x7f0c680ef0a0 con 0x7f0c700ce480 2023-05-22T15:27:43.253 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2023-05-22T15:27:43.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.247+0000 7f0c77d68640 1 -- 172.21.15.16:0/462711017 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f0c54065380 msgr2=0x7f0c54067830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:43.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.247+0000 7f0c77d68640 1 --2- 172.21.15.16:0/462711017 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f0c54065380 0x7f0c54067830 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f0c700aebe0 tx=0x7f0c60008040 comp rx=0 tx=0).stop 2023-05-22T15:27:43.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.247+0000 7f0c77d68640 1 -- 172.21.15.16:0/462711017 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0c700ce480 msgr2=0x7f0c700b1550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:43.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.247+0000 7f0c77d68640 1 --2- 172.21.15.16:0/462711017 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0c700ce480 0x7f0c700b1550 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f0c6801b050 tx=0x7f0c68019740 comp rx=0 tx=0).stop 2023-05-22T15:27:43.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.247+0000 7f0c77d68640 1 -- 172.21.15.16:0/462711017 shutdown_connections 2023-05-22T15:27:43.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.247+0000 7f0c77d68640 1 --2- 172.21.15.16:0/462711017 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0c701605d0 0x7f0c70152d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:43.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.247+0000 7f0c77d68640 1 --2- 172.21.15.16:0/462711017 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f0c54065380 0x7f0c54067830 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:43.256 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.247+0000 7f0c77d68640 1 --2- 172.21.15.16:0/462711017 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0c700ce850 0x7f0c7014c1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:43.256 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.247+0000 7f0c77d68640 1 --2- 172.21.15.16:0/462711017 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0c700ce480 0x7f0c700b1550 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:43.256 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.247+0000 7f0c77d68640 1 -- 172.21.15.16:0/462711017 >> 172.21.15.16:0/462711017 conn(0x7f0c700cc870 msgr2=0x7f0c700c9790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:43.256 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.247+0000 7f0c77d68640 1 -- 172.21.15.16:0/462711017 shutdown_connections 2023-05-22T15:27:43.256 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.247+0000 7f0c77d68640 1 -- 172.21.15.16:0/462711017 wait complete. 2023-05-22T15:27:43.271 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-05-22T15:27:43.393 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.387+0000 7f417cc39640 1 Processor -- start 2023-05-22T15:27:43.394 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.387+0000 7f417cc39640 1 -- start start 2023-05-22T15:27:43.395 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.387+0000 7f417cc39640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f4178159970 0x7f4178159d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:43.395 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.387+0000 7f417cc39640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f417815db00 0x7f417806b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:43.395 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.387+0000 7f417cc39640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4178070c30 con 0x7f417815d680 2023-05-22T15:27:43.395 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.387+0000 7f417cc39640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41780cb970 con 0x7f4178159970 2023-05-22T15:27:43.395 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.387+0000 7f417cc39640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4178070e60 con 0x7f417815db00 2023-05-22T15:27:43.395 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.387+0000 7f417659c640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f4178159970 0x7f4178159d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:43.396 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.387+0000 7f417659c640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f4178159970 0x7f4178159d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41606/0 (socket says 172.21.15.16:41606) 2023-05-22T15:27:43.396 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.387+0000 7f417659c640 1 -- 172.21.15.16:0/1983733712 learned_addr learned my addr 172.21.15.16:0/1983733712 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:43.396 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.387+0000 7f4176d9d640 1 --2- 172.21.15.16:0/1983733712 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f417815db00 0x7f417806b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:43.396 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.387+0000 7f417659c640 1 -- 172.21.15.16:0/1983733712 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f417815db00 msgr2=0x7f417806b6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:43.397 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.387+0000 7f417659c640 1 --2- 172.21.15.16:0/1983733712 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f417815db00 0x7f417806b6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:43.397 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.387+0000 7f417659c640 1 -- 172.21.15.16:0/1983733712 >> v1:172.21.15.62:6789/0 conn(0x7f417815d680 legacy=0x7f41780cb260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:43.397 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.387+0000 7f417659c640 1 -- 172.21.15.16:0/1983733712 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4178071140 con 0x7f4178159970 2023-05-22T15:27:43.397 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.387+0000 7f4176d9d640 1 --2- 172.21.15.16:0/1983733712 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f417815db00 0x7f417806b6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:27:43.398 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.391+0000 7f417659c640 1 --2- 172.21.15.16:0/1983733712 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4178159970 0x7f4178159d40 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f4168009870 tx=0x7f416801c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1d10d3780348fd7 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:43.398 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.391+0000 7f417559a640 1 -- 172.21.15.16:0/1983733712 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41680ad640 con 0x7f4178159970 2023-05-22T15:27:43.398 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.391+0000 7f417559a640 1 -- 172.21.15.16:0/1983733712 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41680ad7a0 con 0x7f4178159970 2023-05-22T15:27:43.399 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.391+0000 7f417559a640 1 -- 172.21.15.16:0/1983733712 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41680adb30 con 0x7f4178159970 2023-05-22T15:27:43.399 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.391+0000 7f417cc39640 1 -- 172.21.15.16:0/1983733712 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4178159970 msgr2=0x7f4178159d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:43.400 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.391+0000 7f417cc39640 1 --2- 172.21.15.16:0/1983733712 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4178159970 0x7f4178159d40 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f4168009870 tx=0x7f416801c480 comp rx=0 tx=0).stop 2023-05-22T15:27:43.400 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.391+0000 7f417cc39640 1 -- 172.21.15.16:0/1983733712 shutdown_connections 2023-05-22T15:27:43.400 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.391+0000 7f417cc39640 1 --2- 172.21.15.16:0/1983733712 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f417815db00 0x7f417806b6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:43.401 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.391+0000 7f417cc39640 1 --2- 172.21.15.16:0/1983733712 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4178159970 0x7f4178159d40 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:43.401 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.391+0000 7f417cc39640 1 -- 172.21.15.16:0/1983733712 >> 172.21.15.16:0/1983733712 conn(0x7f4178069720 msgr2=0x7f4178069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:43.401 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.391+0000 7f417cc39640 1 -- 172.21.15.16:0/1983733712 shutdown_connections 2023-05-22T15:27:43.402 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.391+0000 7f417cc39640 1 -- 172.21.15.16:0/1983733712 wait complete. 2023-05-22T15:27:43.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.391+0000 7f417cc39640 1 Processor -- start 2023-05-22T15:27:43.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.391+0000 7f417cc39640 1 -- start start 2023-05-22T15:27:43.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.391+0000 7f417cc39640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4178159970 0x7f41780ae590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:43.408 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.391+0000 7f417cc39640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f417815d680 0x7f41780b3170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:43.408 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.391+0000 7f417cc39640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f417815db00 0x7f41780c4170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:43.408 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.391+0000 7f417cc39640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41780ae170 con 0x7f417815d680 2023-05-22T15:27:43.408 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.391+0000 7f417cc39640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41780b21c0 con 0x7f417815db00 2023-05-22T15:27:43.408 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.391+0000 7f417cc39640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41780b24a0 con 0x7f4178159970 2023-05-22T15:27:43.413 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.391+0000 7f4175d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f417815d680 0x7f41780b3170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:43.413 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.391+0000 7f417659c640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4178159970 0x7f41780ae590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:43.414 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.391+0000 7f417659c640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4178159970 0x7f41780ae590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3301/0 says I am v2:172.21.15.16:36920/0 (socket says 172.21.15.16:36920) 2023-05-22T15:27:43.414 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.391+0000 7f417659c640 1 -- 172.21.15.16:0/677900455 learned_addr learned my addr 172.21.15.16:0/677900455 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:43.414 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.391+0000 7f4175d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f417815d680 0x7f41780b3170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41612/0 (socket says 172.21.15.16:41612) 2023-05-22T15:27:43.414 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.391+0000 7f4176d9d640 1 --2- 172.21.15.16:0/677900455 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f417815db00 0x7f41780c4170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:43.414 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.391+0000 7f4175d9b640 1 -- 172.21.15.16:0/677900455 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4178159970 msgr2=0x7f41780ae590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:43.414 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.391+0000 7f4175d9b640 1 --2- 172.21.15.16:0/677900455 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4178159970 0x7f41780ae590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:43.415 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.391+0000 7f4175d9b640 1 -- 172.21.15.16:0/677900455 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f417815db00 msgr2=0x7f41780c4170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:43.415 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.391+0000 7f4175d9b640 1 --2- 172.21.15.16:0/677900455 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f417815db00 0x7f41780c4170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:43.415 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.391+0000 7f4175d9b640 1 -- 172.21.15.16:0/677900455 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41680094d0 con 0x7f417815d680 2023-05-22T15:27:43.415 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.395+0000 7f4175d9b640 1 --2- 172.21.15.16:0/677900455 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f417815d680 0x7f41780b3170 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f416c000f70 tx=0x7f416c00c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:43.415 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.395+0000 7f4166ffd640 1 -- 172.21.15.16:0/677900455 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f416c0052d0 con 0x7f417815d680 2023-05-22T15:27:43.415 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.395+0000 7f4166ffd640 1 -- 172.21.15.16:0/677900455 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f416c005430 con 0x7f417815d680 2023-05-22T15:27:43.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.395+0000 7f4166ffd640 1 -- 172.21.15.16:0/677900455 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f416c003840 con 0x7f417815d680 2023-05-22T15:27:43.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.395+0000 7f417cc39640 1 -- 172.21.15.16:0/677900455 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41780cae20 con 0x7f417815d680 2023-05-22T15:27:43.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.395+0000 7f417cc39640 1 -- 172.21.15.16:0/677900455 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41780cac90 con 0x7f417815d680 2023-05-22T15:27:43.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.395+0000 7f417cc39640 1 -- 172.21.15.16:0/677900455 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f417815adc0 con 0x7f417815d680 2023-05-22T15:27:43.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.395+0000 7f4166ffd640 1 -- 172.21.15.16:0/677900455 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7f416c0039a0 con 0x7f417815d680 2023-05-22T15:27:43.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.395+0000 7f4166ffd640 1 --2- 172.21.15.16:0/677900455 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f4158065380 0x7f4158067830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:43.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.395+0000 7f4166ffd640 1 -- 172.21.15.16:0/677900455 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f416c082410 con 0x7f417815d680 2023-05-22T15:27:43.417 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.395+0000 7f417659c640 1 --2- 172.21.15.16:0/677900455 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f4158065380 0x7f4158067830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:43.417 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.395+0000 7f417659c640 1 --2- 172.21.15.16:0/677900455 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f4158065380 0x7f4158067830 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f4168009870 tx=0x7f4168009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:43.417 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.399+0000 7f4166ffd640 1 -- 172.21.15.16:0/677900455 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f416c04ff80 con 0x7f417815d680 2023-05-22T15:27:43.559 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.551+0000 7f417cc39640 1 -- 172.21.15.16:0/677900455 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f41780ae2b0 con 0x7f417815d680 2023-05-22T15:27:43.560 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.551+0000 7f4166ffd640 1 -- 172.21.15.16:0/677900455 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v29) v1 ==== 100+0+11 (secure 0 0 0) 0x7f41780ae2b0 con 0x7f417815d680 2023-05-22T15:27:43.560 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 64 2023-05-22T15:27:43.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.555+0000 7f417cc39640 1 -- 172.21.15.16:0/677900455 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f4158065380 msgr2=0x7f4158067830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:43.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.555+0000 7f417cc39640 1 --2- 172.21.15.16:0/677900455 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f4158065380 0x7f4158067830 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f4168009870 tx=0x7f4168009040 comp rx=0 tx=0).stop 2023-05-22T15:27:43.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.555+0000 7f417cc39640 1 -- 172.21.15.16:0/677900455 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f417815d680 msgr2=0x7f41780b3170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:43.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.555+0000 7f417cc39640 1 --2- 172.21.15.16:0/677900455 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f417815d680 0x7f41780b3170 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f416c000f70 tx=0x7f416c00c890 comp rx=0 tx=0).stop 2023-05-22T15:27:43.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.555+0000 7f417cc39640 1 -- 172.21.15.16:0/677900455 shutdown_connections 2023-05-22T15:27:43.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.555+0000 7f417cc39640 1 --2- 172.21.15.16:0/677900455 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f417815db00 0x7f41780c4170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:43.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.555+0000 7f417cc39640 1 --2- 172.21.15.16:0/677900455 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f417815d680 0x7f41780b3170 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:43.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.555+0000 7f417cc39640 1 --2- 172.21.15.16:0/677900455 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f4158065380 0x7f4158067830 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:43.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.555+0000 7f417cc39640 1 --2- 172.21.15.16:0/677900455 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4178159970 0x7f41780ae590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:43.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.555+0000 7f417cc39640 1 -- 172.21.15.16:0/677900455 >> 172.21.15.16:0/677900455 conn(0x7f4178069720 msgr2=0x7f41780c3510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:43.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.555+0000 7f417cc39640 1 -- 172.21.15.16:0/677900455 shutdown_connections 2023-05-22T15:27:43.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.555+0000 7f417cc39640 1 -- 172.21.15.16:0/677900455 wait complete. 2023-05-22T15:27:43.579 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-05-22T15:27:43.701 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.691+0000 7fdef3516640 1 Processor -- start 2023-05-22T15:27:43.701 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.691+0000 7fdef3516640 1 -- start start 2023-05-22T15:27:43.702 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.695+0000 7fdef3516640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fdeec0c9e40 0x7fdeec0cbd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:43.702 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.695+0000 7fdef3516640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdeec0ad5b0 con 0x7fdeec0ab7d0 2023-05-22T15:27:43.702 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.695+0000 7fdef3516640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdeec15c060 con 0x7fdeec1601f0 2023-05-22T15:27:43.702 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.695+0000 7fdef3516640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdeec0cd5b0 con 0x7fdeec0c9e40 2023-05-22T15:27:43.703 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.695+0000 7fdef12b2640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fdeec0c9e40 0x7fdeec0cbd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:43.703 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.695+0000 7fdef1ab3640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fdeec0ab7d0 0x7fdeec0cdd80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:45606/0 (socket says 172.21.15.16:45606) 2023-05-22T15:27:43.703 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.695+0000 7fdef12b2640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fdeec0c9e40 0x7fdeec0cbd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41614/0 (socket says 172.21.15.16:41614) 2023-05-22T15:27:43.703 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.695+0000 7fdef1ab3640 1 -- 172.21.15.16:0/2262455609 learned_addr learned my addr 172.21.15.16:0/2262455609 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:27:43.703 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.695+0000 7fdef12b2640 1 -- 172.21.15.16:0/2262455609 >> v1:172.21.15.62:6789/0 conn(0x7fdeec1601f0 legacy=0x7fdeec0cb1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:43.703 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.695+0000 7fdef12b2640 1 -- 172.21.15.16:0/2262455609 >> v1:172.21.15.16:6789/0 conn(0x7fdeec0ab7d0 legacy=0x7fdeec0cdd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:43.703 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.695+0000 7fdef12b2640 1 -- 172.21.15.16:0/2262455609 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdeec0ad280 con 0x7fdeec0c9e40 2023-05-22T15:27:43.704 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.695+0000 7fdee3fff640 1 -- 172.21.15.16:0/2262455609 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 483024421 0 0) 0x7fdeec0ad5b0 con 0x7fdeec0ab7d0 2023-05-22T15:27:43.704 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.695+0000 7fdef12b2640 1 --2- 172.21.15.16:0/2262455609 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdeec0c9e40 0x7fdeec0cbd70 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fdee80097c0 tx=0x7fdee801c3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e3fac14d377e458 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:43.705 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.695+0000 7fdee3fff640 1 -- 172.21.15.16:0/2262455609 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdee80ac620 con 0x7fdeec0c9e40 2023-05-22T15:27:43.705 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.695+0000 7fdee3fff640 1 -- 172.21.15.16:0/2262455609 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdee80ac780 con 0x7fdeec0c9e40 2023-05-22T15:27:43.705 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.695+0000 7fdee3fff640 1 -- 172.21.15.16:0/2262455609 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdee80acb10 con 0x7fdeec0c9e40 2023-05-22T15:27:43.706 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.699+0000 7fdef3516640 1 -- 172.21.15.16:0/2262455609 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdeec0c9e40 msgr2=0x7fdeec0cbd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:43.706 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.699+0000 7fdef3516640 1 --2- 172.21.15.16:0/2262455609 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdeec0c9e40 0x7fdeec0cbd70 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fdee80097c0 tx=0x7fdee801c3c0 comp rx=0 tx=0).stop 2023-05-22T15:27:43.706 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.699+0000 7fdef3516640 1 -- 172.21.15.16:0/2262455609 shutdown_connections 2023-05-22T15:27:43.706 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.699+0000 7fdef3516640 1 --2- 172.21.15.16:0/2262455609 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdeec0c9e40 0x7fdeec0cbd70 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:43.706 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.699+0000 7fdef3516640 1 -- 172.21.15.16:0/2262455609 >> 172.21.15.16:0/2262455609 conn(0x7fdeec069720 msgr2=0x7fdeec069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:43.706 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.699+0000 7fdef3516640 1 -- 172.21.15.16:0/2262455609 shutdown_connections 2023-05-22T15:27:43.707 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.699+0000 7fdef3516640 1 -- 172.21.15.16:0/2262455609 wait complete. 2023-05-22T15:27:43.708 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.699+0000 7fdef3516640 1 Processor -- start 2023-05-22T15:27:43.708 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.699+0000 7fdef3516640 1 -- start start 2023-05-22T15:27:43.708 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.699+0000 7fdef3516640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fdeec0ab7d0 0x7fdeec1577a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:43.708 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.699+0000 7fdef3516640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdeec0c9e40 0x7fdeec1530f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:43.708 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.699+0000 7fdef3516640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdeec1601f0 0x7fdeec069af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:43.709 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.699+0000 7fdef3516640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdeec15ec00 con 0x7fdeec1601f0 2023-05-22T15:27:43.709 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.699+0000 7fdef3516640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdeec153630 con 0x7fdeec0c9e40 2023-05-22T15:27:43.709 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.699+0000 7fdef3516640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdeec15ee10 con 0x7fdeec0ab7d0 2023-05-22T15:27:43.709 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.699+0000 7fdef1ab3640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdeec1601f0 0x7fdeec069af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:43.709 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.699+0000 7fdef1ab3640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdeec1601f0 0x7fdeec069af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41622/0 (socket says 172.21.15.16:41622) 2023-05-22T15:27:43.709 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.699+0000 7fdef1ab3640 1 -- 172.21.15.16:0/2412889338 learned_addr learned my addr 172.21.15.16:0/2412889338 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:43.709 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.699+0000 7fdef0ab1640 1 --2- 172.21.15.16:0/2412889338 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdeec0c9e40 0x7fdeec1530f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:43.710 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.699+0000 7fdef1ab3640 1 -- 172.21.15.16:0/2412889338 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fdeec0ab7d0 msgr2=0x7fdeec1577a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:43.710 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.699+0000 7fdef12b2640 1 --2- 172.21.15.16:0/2412889338 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fdeec0ab7d0 0x7fdeec1577a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:43.710 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.699+0000 7fdef1ab3640 1 --2- 172.21.15.16:0/2412889338 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fdeec0ab7d0 0x7fdeec1577a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:43.710 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.699+0000 7fdef1ab3640 1 -- 172.21.15.16:0/2412889338 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdeec0c9e40 msgr2=0x7fdeec1530f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:43.710 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.699+0000 7fdef1ab3640 1 --2- 172.21.15.16:0/2412889338 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdeec0c9e40 0x7fdeec1530f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:43.710 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.699+0000 7fdef1ab3640 1 -- 172.21.15.16:0/2412889338 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdee8009420 con 0x7fdeec1601f0 2023-05-22T15:27:43.710 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.699+0000 7fdef0ab1640 1 --2- 172.21.15.16:0/2412889338 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdeec0c9e40 0x7fdeec1530f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:27:43.710 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.699+0000 7fdef1ab3640 1 --2- 172.21.15.16:0/2412889338 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdeec1601f0 0x7fdeec069af0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fdee400c700 tx=0x7fdee400cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:43.711 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.699+0000 7fdee1ffb640 1 -- 172.21.15.16:0/2412889338 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdee400ce10 con 0x7fdeec1601f0 2023-05-22T15:27:43.711 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.699+0000 7fdee1ffb640 1 -- 172.21.15.16:0/2412889338 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdee400a640 con 0x7fdeec1601f0 2023-05-22T15:27:43.711 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.699+0000 7fdee1ffb640 1 -- 172.21.15.16:0/2412889338 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdee400aa70 con 0x7fdeec1601f0 2023-05-22T15:27:43.712 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.699+0000 7fdef3516640 1 -- 172.21.15.16:0/2412889338 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdeec0abfe0 con 0x7fdeec1601f0 2023-05-22T15:27:43.712 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.699+0000 7fdef3516640 1 -- 172.21.15.16:0/2412889338 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdeec0cdaa0 con 0x7fdeec1601f0 2023-05-22T15:27:43.712 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.703+0000 7fdef3516640 1 -- 172.21.15.16:0/2412889338 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdeec0cb8d0 con 0x7fdeec1601f0 2023-05-22T15:27:43.712 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.703+0000 7fdee1ffb640 1 -- 172.21.15.16:0/2412889338 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7fdee400abd0 con 0x7fdeec1601f0 2023-05-22T15:27:43.712 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.703+0000 7fdee1ffb640 1 --2- 172.21.15.16:0/2412889338 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fded0065380 0x7fded0067830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:43.713 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.703+0000 7fdee1ffb640 1 -- 172.21.15.16:0/2412889338 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6871+0+0 (secure 0 0 0) 0x7fdee40820f0 con 0x7fdeec1601f0 2023-05-22T15:27:43.713 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.703+0000 7fdef12b2640 1 --2- 172.21.15.16:0/2412889338 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fded0065380 0x7fded0067830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:43.713 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.703+0000 7fdef12b2640 1 --2- 172.21.15.16:0/2412889338 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fded0065380 0x7fded0067830 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fdee80ae000 tx=0x7fdee80a7000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:43.714 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.707+0000 7fdee1ffb640 1 -- 172.21.15.16:0/2412889338 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fdee404fa40 con 0x7fdeec1601f0 2023-05-22T15:27:43.860 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.851+0000 7fdef3516640 1 -- 172.21.15.16:0/2412889338 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7fdeec157df0 con 0x7fdeec1601f0 2023-05-22T15:27:43.861 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.851+0000 7fdee1ffb640 1 -- 172.21.15.16:0/2412889338 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v29) v1 ==== 103+0+11 (secure 0 0 0) 0x7fdeec157df0 con 0x7fdeec1601f0 2023-05-22T15:27:43.861 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2023-05-22T15:27:43.863 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.855+0000 7fdef3516640 1 -- 172.21.15.16:0/2412889338 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fded0065380 msgr2=0x7fded0067830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:43.863 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.855+0000 7fdef3516640 1 --2- 172.21.15.16:0/2412889338 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fded0065380 0x7fded0067830 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fdee80ae000 tx=0x7fdee80a7000 comp rx=0 tx=0).stop 2023-05-22T15:27:43.863 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.855+0000 7fdef3516640 1 -- 172.21.15.16:0/2412889338 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdeec1601f0 msgr2=0x7fdeec069af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:43.863 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.855+0000 7fdef3516640 1 --2- 172.21.15.16:0/2412889338 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdeec1601f0 0x7fdeec069af0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fdee400c700 tx=0x7fdee400cbc0 comp rx=0 tx=0).stop 2023-05-22T15:27:43.863 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.855+0000 7fdef3516640 1 -- 172.21.15.16:0/2412889338 shutdown_connections 2023-05-22T15:27:43.864 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.855+0000 7fdef3516640 1 --2- 172.21.15.16:0/2412889338 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdeec1601f0 0x7fdeec069af0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:43.864 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.855+0000 7fdef3516640 1 --2- 172.21.15.16:0/2412889338 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdeec0c9e40 0x7fdeec1530f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:43.864 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.855+0000 7fdef3516640 1 --2- 172.21.15.16:0/2412889338 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fded0065380 0x7fded0067830 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:43.864 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.855+0000 7fdef3516640 1 --2- 172.21.15.16:0/2412889338 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fdeec0ab7d0 0x7fdeec1577a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:43.864 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.855+0000 7fdef3516640 1 -- 172.21.15.16:0/2412889338 >> 172.21.15.16:0/2412889338 conn(0x7fdeec069720 msgr2=0x7fdeec06fd50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:43.864 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.855+0000 7fdef3516640 1 -- 172.21.15.16:0/2412889338 shutdown_connections 2023-05-22T15:27:43.865 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.855+0000 7fdef3516640 1 -- 172.21.15.16:0/2412889338 wait complete. 2023-05-22T15:27:43.879 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-05-22T15:27:43.990 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.983+0000 7f4aad08c640 1 Processor -- start 2023-05-22T15:27:43.990 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.983+0000 7f4aad08c640 1 -- start start 2023-05-22T15:27:43.991 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.983+0000 7f4aad08c640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4aa80ce480 0x7f4aa8071640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:43.992 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.983+0000 7f4aad08c640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f4aa815c270 0x7f4aa815c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:43.992 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.983+0000 7f4aad08c640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4aa8069ae0 con 0x7f4aa80ce850 2023-05-22T15:27:43.992 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.983+0000 7f4aad08c640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4aa815f990 con 0x7f4aa815c270 2023-05-22T15:27:43.992 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.983+0000 7f4aad08c640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4aa8069dc0 con 0x7f4aa80ce480 2023-05-22T15:27:43.992 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.983+0000 7f4aa759e640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f4aa815c270 0x7f4aa815c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:43.992 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.983+0000 7f4aa659c640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f4aa80ce850 0x7f4aa8160090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.16:34248/0 (socket says 172.21.15.16:34248) 2023-05-22T15:27:43.992 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.983+0000 7f4aa759e640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f4aa815c270 0x7f4aa815c6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41634/0 (socket says 172.21.15.16:41634) 2023-05-22T15:27:43.993 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.983+0000 7f4aa659c640 1 -- 172.21.15.16:0/3717731126 learned_addr learned my addr 172.21.15.16:0/3717731126 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:27:43.993 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.983+0000 7f4aa6d9d640 1 --2- 172.21.15.16:0/3717731126 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4aa80ce480 0x7f4aa8071640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:43.993 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.983+0000 7f4aa759e640 1 -- 172.21.15.16:0/3717731126 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4aa80ce480 msgr2=0x7f4aa8071640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:43.993 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.983+0000 7f4aa759e640 1 --2- 172.21.15.16:0/3717731126 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4aa80ce480 0x7f4aa8071640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:43.993 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.983+0000 7f4aa759e640 1 -- 172.21.15.16:0/3717731126 >> v1:172.21.15.62:6789/0 conn(0x7f4aa80ce850 legacy=0x7f4aa8160090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:43.993 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.983+0000 7f4aa759e640 1 -- 172.21.15.16:0/3717731126 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4aa806a0a0 con 0x7f4aa815c270 2023-05-22T15:27:43.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.987+0000 7f4aa759e640 1 --2- 172.21.15.16:0/3717731126 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4aa815c270 0x7f4aa815c6f0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f4aa0009d90 tx=0x7f4aa001c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ca4892759c137ba server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:43.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.987+0000 7f4aa5d9b640 1 -- 172.21.15.16:0/3717731126 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4aa00a5a00 con 0x7f4aa815c270 2023-05-22T15:27:43.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.987+0000 7f4aa5d9b640 1 -- 172.21.15.16:0/3717731126 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4aa00a5b60 con 0x7f4aa815c270 2023-05-22T15:27:43.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.987+0000 7f4aa5d9b640 1 -- 172.21.15.16:0/3717731126 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4aa00ad440 con 0x7f4aa815c270 2023-05-22T15:27:43.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.987+0000 7f4aad08c640 1 -- 172.21.15.16:0/3717731126 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4aa815c270 msgr2=0x7f4aa815c6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:43.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.987+0000 7f4aad08c640 1 --2- 172.21.15.16:0/3717731126 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4aa815c270 0x7f4aa815c6f0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f4aa0009d90 tx=0x7f4aa001c920 comp rx=0 tx=0).stop 2023-05-22T15:27:43.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.987+0000 7f4aad08c640 1 -- 172.21.15.16:0/3717731126 shutdown_connections 2023-05-22T15:27:43.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.987+0000 7f4aad08c640 1 --2- 172.21.15.16:0/3717731126 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4aa815c270 0x7f4aa815c6f0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:43.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.987+0000 7f4aad08c640 1 --2- 172.21.15.16:0/3717731126 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4aa80ce480 0x7f4aa8071640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:43.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.987+0000 7f4aad08c640 1 -- 172.21.15.16:0/3717731126 >> 172.21.15.16:0/3717731126 conn(0x7f4aa80cc870 msgr2=0x7f4aa80cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:43.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.987+0000 7f4aad08c640 1 -- 172.21.15.16:0/3717731126 shutdown_connections 2023-05-22T15:27:43.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.987+0000 7f4aad08c640 1 -- 172.21.15.16:0/3717731126 wait complete. 2023-05-22T15:27:43.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.987+0000 7f4aad08c640 1 Processor -- start 2023-05-22T15:27:43.998 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.987+0000 7f4aad08c640 1 -- start start 2023-05-22T15:27:43.998 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.987+0000 7f4aad08c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4aa80ce480 0x7f4aa8151580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:43.998 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.987+0000 7f4aad08c640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4aa80ce850 0x7f4aa80b47c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:43.998 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.987+0000 7f4aad08c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4aa815c270 0x7f4aa815eb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:43.998 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.987+0000 7f4aad08c640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4aa80c3190 con 0x7f4aa815c270 2023-05-22T15:27:43.998 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.987+0000 7f4aad08c640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4aa80c3470 con 0x7f4aa80ce480 2023-05-22T15:27:43.998 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.987+0000 7f4aad08c640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4aa80c3750 con 0x7f4aa80ce850 2023-05-22T15:27:43.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.987+0000 7f4aa759e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4aa815c270 0x7f4aa815eb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:43.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.987+0000 7f4aa759e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4aa815c270 0x7f4aa815eb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41640/0 (socket says 172.21.15.16:41640) 2023-05-22T15:27:43.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.987+0000 7f4aa759e640 1 -- 172.21.15.16:0/3955560706 learned_addr learned my addr 172.21.15.16:0/3955560706 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:43.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.987+0000 7f4aa6d9d640 1 --2- 172.21.15.16:0/3955560706 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4aa80ce480 0x7f4aa8151580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:43.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.987+0000 7f4aa659c640 1 --2- 172.21.15.16:0/3955560706 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4aa80ce850 0x7f4aa80b47c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:43.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.987+0000 7f4aa759e640 1 -- 172.21.15.16:0/3955560706 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4aa80ce850 msgr2=0x7f4aa80b47c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:44.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.987+0000 7f4aa759e640 1 --2- 172.21.15.16:0/3955560706 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4aa80ce850 0x7f4aa80b47c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:44.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.987+0000 7f4aa759e640 1 -- 172.21.15.16:0/3955560706 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4aa80ce480 msgr2=0x7f4aa8151580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:44.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.987+0000 7f4aa759e640 1 --2- 172.21.15.16:0/3955560706 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4aa80ce480 0x7f4aa8151580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:44.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.987+0000 7f4aa759e640 1 -- 172.21.15.16:0/3955560706 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4aa0009a70 con 0x7f4aa815c270 2023-05-22T15:27:44.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.987+0000 7f4aa659c640 1 --2- 172.21.15.16:0/3955560706 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4aa80ce850 0x7f4aa80b47c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:27:44.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.991+0000 7f4aa759e640 1 --2- 172.21.15.16:0/3955560706 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4aa815c270 0x7f4aa815eb10 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f4aa001afd0 tx=0x7f4aa00196c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:44.001 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.991+0000 7f4a9f7fe640 1 -- 172.21.15.16:0/3955560706 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4aa00addc0 con 0x7f4aa815c270 2023-05-22T15:27:44.001 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.991+0000 7f4a9f7fe640 1 -- 172.21.15.16:0/3955560706 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4aa0009450 con 0x7f4aa815c270 2023-05-22T15:27:44.001 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.991+0000 7f4aad08c640 1 -- 172.21.15.16:0/3955560706 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4aa80698b0 con 0x7f4aa815c270 2023-05-22T15:27:44.001 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.991+0000 7f4a9f7fe640 1 -- 172.21.15.16:0/3955560706 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4aa00b7610 con 0x7f4aa815c270 2023-05-22T15:27:44.001 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.991+0000 7f4aad08c640 1 -- 172.21.15.16:0/3955560706 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4aa8069720 con 0x7f4aa815c270 2023-05-22T15:27:44.002 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.991+0000 7f4aad08c640 1 -- 172.21.15.16:0/3955560706 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4aa815cc30 con 0x7f4aa815c270 2023-05-22T15:27:44.002 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.991+0000 7f4a9f7fe640 1 -- 172.21.15.16:0/3955560706 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7f4aa00b4050 con 0x7f4aa815c270 2023-05-22T15:27:44.002 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.991+0000 7f4a9f7fe640 1 --2- 172.21.15.16:0/3955560706 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f4a88065380 0x7f4a88067830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:44.003 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.991+0000 7f4a9f7fe640 1 -- 172.21.15.16:0/3955560706 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f4aa011dd80 con 0x7f4aa815c270 2023-05-22T15:27:44.003 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.991+0000 7f4aa6d9d640 1 --2- 172.21.15.16:0/3955560706 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f4a88065380 0x7f4a88067830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:44.003 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.991+0000 7f4aa6d9d640 1 --2- 172.21.15.16:0/3955560706 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f4a88065380 0x7f4a88067830 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f4a9000b9d0 tx=0x7f4a90008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:44.003 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:43.995+0000 7f4a9f7fe640 1 -- 172.21.15.16:0/3955560706 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f4aa00eb6d0 con 0x7f4aa815c270 2023-05-22T15:27:44.166 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:44.155+0000 7f4aad08c640 1 -- 172.21.15.16:0/3955560706 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f4aa804dbe0 con 0x7f4aa815c270 2023-05-22T15:27:44.166 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:44.155+0000 7f4a9f7fe640 1 -- 172.21.15.16:0/3955560706 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v29) v1 ==== 80+0+0 (secure 0 0 0) 0x7f4aa00b2020 con 0x7f4aa815c270 2023-05-22T15:27:44.167 INFO:teuthology.orchestra.run.smithi016.stderr:listed 0 entries 2023-05-22T15:27:44.167 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:44.155+0000 7f4aad08c640 1 -- 172.21.15.16:0/3955560706 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f4a88065380 msgr2=0x7f4a88067830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:44.167 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:44.155+0000 7f4aad08c640 1 --2- 172.21.15.16:0/3955560706 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f4a88065380 0x7f4a88067830 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f4a9000b9d0 tx=0x7f4a90008040 comp rx=0 tx=0).stop 2023-05-22T15:27:44.167 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:44.155+0000 7f4aad08c640 1 -- 172.21.15.16:0/3955560706 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4aa815c270 msgr2=0x7f4aa815eb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:44.167 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:44.155+0000 7f4aad08c640 1 --2- 172.21.15.16:0/3955560706 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4aa815c270 0x7f4aa815eb10 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f4aa001afd0 tx=0x7f4aa00196c0 comp rx=0 tx=0).stop 2023-05-22T15:27:44.167 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:44.155+0000 7f4aad08c640 1 -- 172.21.15.16:0/3955560706 shutdown_connections 2023-05-22T15:27:44.167 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:44.155+0000 7f4aad08c640 1 --2- 172.21.15.16:0/3955560706 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4aa815c270 0x7f4aa815eb10 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:44.168 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:44.155+0000 7f4aad08c640 1 --2- 172.21.15.16:0/3955560706 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f4a88065380 0x7f4a88067830 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:44.168 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:44.155+0000 7f4aad08c640 1 --2- 172.21.15.16:0/3955560706 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4aa80ce850 0x7f4aa80b47c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:44.168 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:44.155+0000 7f4aad08c640 1 --2- 172.21.15.16:0/3955560706 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4aa80ce480 0x7f4aa8151580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:44.168 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:44.155+0000 7f4aad08c640 1 -- 172.21.15.16:0/3955560706 >> 172.21.15.16:0/3955560706 conn(0x7f4aa80cc870 msgr2=0x7f4aa815d680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:44.168 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:44.155+0000 7f4aad08c640 1 -- 172.21.15.16:0/3955560706 shutdown_connections 2023-05-22T15:27:44.168 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:44.155+0000 7f4aad08c640 1 -- 172.21.15.16:0/3955560706 wait complete. 2023-05-22T15:27:44.181 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi016.front.sepia.ceph.com... 2023-05-22T15:27:44.181 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:27:44.182 DEBUG:teuthology.orchestra.run.smithi016:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2023-05-22T15:27:44.196 INFO:teuthology.orchestra.run:waiting for 300 2023-05-22T15:27:44.197 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi016.front.sepia.ceph.com 2023-05-22T15:27:44.197 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:27:44.198 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-05-22T15:27:44.242 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2023-05-22T15:27:44.242 DEBUG:teuthology.orchestra.run.smithi062:> dd if=/proc/self/mounts of=/dev/stdout 2023-05-22T15:27:44.249 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2023-05-22T15:27:44.249 DEBUG:teuthology.orchestra.run.smithi062:> dd if=/proc/self/mounts of=/dev/stdout 2023-05-22T15:27:44.302 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-05-22T15:27:44.429 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:44.419+0000 7f3eba05f640 1 Processor -- start 2023-05-22T15:27:44.429 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:44.419+0000 7f3eba05f640 1 -- start start 2023-05-22T15:27:44.430 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:44.423+0000 7f3eba05f640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f3eb41531d0 0x7f3eb40cd730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:44.430 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:44.423+0000 7f3eba05f640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f3eb40b0250 0x7f3eb40acba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:44.430 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:44.423+0000 7f3eba05f640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3eb415c6b0 con 0x7f3eb40cbbf0 2023-05-22T15:27:44.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:44.423+0000 7f3eba05f640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3eb415d330 con 0x7f3eb40b0250 2023-05-22T15:27:44.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:44.423+0000 7f3eba05f640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3eb415c4a0 con 0x7f3eb41531d0 2023-05-22T15:27:44.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:44.423+0000 7f3eb3fff640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f3eb40b0250 0x7f3eb40acba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:44.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:44.423+0000 7f3eb2ffd640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f3eb40cbbf0 0x7f3eb40afb40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:45622/0 (socket says 172.21.15.16:45622) 2023-05-22T15:27:44.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:44.423+0000 7f3eb3fff640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f3eb40b0250 0x7f3eb40acba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41642/0 (socket says 172.21.15.16:41642) 2023-05-22T15:27:44.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:44.423+0000 7f3eb2ffd640 1 -- 172.21.15.16:0/3471517101 learned_addr learned my addr 172.21.15.16:0/3471517101 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:27:44.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:44.423+0000 7f3eb37fe640 1 --2- 172.21.15.16:0/3471517101 >> v2:172.21.15.62:3300/0 conn(0x7f3eb41531d0 0x7f3eb40cd730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:44.432 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:44.423+0000 7f3eb27fc640 1 -- 172.21.15.16:0/3471517101 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2042258442 0 0) 0x7f3eb415c6b0 con 0x7f3eb40cbbf0 2023-05-22T15:27:44.432 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:44.423+0000 7f3eb3fff640 1 -- 172.21.15.16:0/3471517101 >> v2:172.21.15.62:3300/0 conn(0x7f3eb41531d0 msgr2=0x7f3eb40cd730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:44.432 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:44.423+0000 7f3eb3fff640 1 --2- 172.21.15.16:0/3471517101 >> v2:172.21.15.62:3300/0 conn(0x7f3eb41531d0 0x7f3eb40cd730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:44.432 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:44.423+0000 7f3eb3fff640 1 -- 172.21.15.16:0/3471517101 >> v1:172.21.15.16:6789/0 conn(0x7f3eb40cbbf0 legacy=0x7f3eb40afb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:44.432 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:44.423+0000 7f3eb3fff640 1 -- 172.21.15.16:0/3471517101 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3eb415c990 con 0x7f3eb40b0250 2023-05-22T15:27:44.434 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:44.423+0000 7f3eb3fff640 1 --2- 172.21.15.16:0/3471517101 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3eb40b0250 0x7f3eb40acba0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f3eb406f430 tx=0x7f3eac01c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=773112cf10d11f57 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:44.434 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:44.423+0000 7f3eb27fc640 1 -- 172.21.15.16:0/3471517101 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3eac0ab460 con 0x7f3eb40b0250 2023-05-22T15:27:44.435 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:44.423+0000 7f3eb27fc640 1 -- 172.21.15.16:0/3471517101 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3eac0ab5c0 con 0x7f3eb40b0250 2023-05-22T15:27:44.435 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:44.423+0000 7f3eb27fc640 1 -- 172.21.15.16:0/3471517101 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3eac0ab950 con 0x7f3eb40b0250 2023-05-22T15:27:44.435 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:44.427+0000 7f3eba05f640 1 -- 172.21.15.16:0/3471517101 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3eb40b0250 msgr2=0x7f3eb40acba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:44.435 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:44.427+0000 7f3eba05f640 1 --2- 172.21.15.16:0/3471517101 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3eb40b0250 0x7f3eb40acba0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f3eb406f430 tx=0x7f3eac01c460 comp rx=0 tx=0).stop 2023-05-22T15:27:44.435 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:44.427+0000 7f3eba05f640 1 -- 172.21.15.16:0/3471517101 shutdown_connections 2023-05-22T15:27:44.435 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:44.427+0000 7f3eba05f640 1 --2- 172.21.15.16:0/3471517101 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3eb40b0250 0x7f3eb40acba0 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:44.435 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:44.427+0000 7f3eba05f640 1 --2- 172.21.15.16:0/3471517101 >> v2:172.21.15.62:3300/0 conn(0x7f3eb41531d0 0x7f3eb40cd730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:44.436 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:44.427+0000 7f3eba05f640 1 -- 172.21.15.16:0/3471517101 >> 172.21.15.16:0/3471517101 conn(0x7f3eb4069720 msgr2=0x7f3eb4069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:44.436 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:44.427+0000 7f3eba05f640 1 -- 172.21.15.16:0/3471517101 shutdown_connections 2023-05-22T15:27:44.436 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:44.427+0000 7f3eba05f640 1 -- 172.21.15.16:0/3471517101 wait complete. 2023-05-22T15:27:44.436 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:44.427+0000 7f3eba05f640 1 Processor -- start 2023-05-22T15:27:44.436 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:44.427+0000 7f3eba05f640 1 -- start start 2023-05-22T15:27:44.438 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:44.427+0000 7f3eba05f640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3eb40b0250 0x7f3eb41502c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:44.438 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:44.427+0000 7f3eba05f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3eb40cbbf0 0x7f3eb4148770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:44.438 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:44.427+0000 7f3eba05f640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3eb41531d0 0x7f3eb4151950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:44.438 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:44.427+0000 7f3eba05f640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3eb414c020 con 0x7f3eb40cbbf0 2023-05-22T15:27:44.438 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:44.427+0000 7f3eba05f640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3eb414b3a0 con 0x7f3eb40b0250 2023-05-22T15:27:44.439 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:44.427+0000 7f3eba05f640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3eb414b680 con 0x7f3eb41531d0 2023-05-22T15:27:44.439 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:44.427+0000 7f3eb2ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3eb40cbbf0 0x7f3eb4148770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:44.439 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:44.427+0000 7f3eb2ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3eb40cbbf0 0x7f3eb4148770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41658/0 (socket says 172.21.15.16:41658) 2023-05-22T15:27:44.439 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:44.427+0000 7f3eb2ffd640 1 -- 172.21.15.16:0/2877320447 learned_addr learned my addr 172.21.15.16:0/2877320447 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:44.439 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:44.427+0000 7f3eb37fe640 1 --2- 172.21.15.16:0/2877320447 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3eb40b0250 0x7f3eb41502c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:44.439 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:44.427+0000 7f3eb3fff640 1 --2- 172.21.15.16:0/2877320447 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3eb41531d0 0x7f3eb4151950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:44.439 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:44.427+0000 7f3eb2ffd640 1 -- 172.21.15.16:0/2877320447 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3eb41531d0 msgr2=0x7f3eb4151950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:44.440 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:44.427+0000 7f3eb2ffd640 1 --2- 172.21.15.16:0/2877320447 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3eb41531d0 0x7f3eb4151950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:44.440 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:44.427+0000 7f3eb2ffd640 1 -- 172.21.15.16:0/2877320447 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3eb40b0250 msgr2=0x7f3eb41502c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:44.440 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:44.427+0000 7f3eb2ffd640 1 --2- 172.21.15.16:0/2877320447 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3eb40b0250 0x7f3eb41502c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:44.440 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:44.427+0000 7f3eb2ffd640 1 -- 172.21.15.16:0/2877320447 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3eac009420 con 0x7f3eb40cbbf0 2023-05-22T15:27:44.440 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:44.427+0000 7f3eb2ffd640 1 --2- 172.21.15.16:0/2877320447 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3eb40cbbf0 0x7f3eb4148770 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f3ea4001570 tx=0x7f3ea400cd20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:44.440 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:44.427+0000 7f3eabfff640 1 -- 172.21.15.16:0/2877320447 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ea400a640 con 0x7f3eb40cbbf0 2023-05-22T15:27:44.440 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:44.427+0000 7f3eabfff640 1 -- 172.21.15.16:0/2877320447 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ea400a7c0 con 0x7f3eb40cbbf0 2023-05-22T15:27:44.441 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:44.427+0000 7f3eba05f640 1 -- 172.21.15.16:0/2877320447 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3eb41608d0 con 0x7f3eb40cbbf0 2023-05-22T15:27:44.441 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:44.427+0000 7f3eabfff640 1 -- 172.21.15.16:0/2877320447 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ea400ab70 con 0x7f3eb40cbbf0 2023-05-22T15:27:44.441 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:44.427+0000 7f3eba05f640 1 -- 172.21.15.16:0/2877320447 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3eb4160740 con 0x7f3eb40cbbf0 2023-05-22T15:27:44.441 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:44.431+0000 7f3eba05f640 1 -- 172.21.15.16:0/2877320447 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3eb4168e30 con 0x7f3eb40cbbf0 2023-05-22T15:27:44.441 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:44.431+0000 7f3eabfff640 1 -- 172.21.15.16:0/2877320447 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7f3ea400acd0 con 0x7f3eb40cbbf0 2023-05-22T15:27:44.441 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:44.431+0000 7f3eabfff640 1 --2- 172.21.15.16:0/2877320447 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f3e940653b0 0x7f3e94067860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:44.441 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:44.431+0000 7f3eabfff640 1 -- 172.21.15.16:0/2877320447 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f3ea4082120 con 0x7f3eb40cbbf0 2023-05-22T15:27:44.441 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:44.431+0000 7f3eb37fe640 1 --2- 172.21.15.16:0/2877320447 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f3e940653b0 0x7f3e94067860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:44.442 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:44.431+0000 7f3eb37fe640 1 --2- 172.21.15.16:0/2877320447 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f3e940653b0 0x7f3e94067860 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f3e9c00b930 tx=0x7f3e9c008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:44.442 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:44.435+0000 7f3eabfff640 1 -- 172.21.15.16:0/2877320447 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f3ea404fa10 con 0x7f3eb40cbbf0 2023-05-22T15:27:44.590 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:44.583+0000 7f3eba05f640 1 -- 172.21.15.16:0/2877320447 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3eb4148d80 con 0x7f3eb40cbbf0 2023-05-22T15:27:44.591 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:44.583+0000 7f3eabfff640 1 -- 172.21.15.16:0/2877320447 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v29) v1 ==== 74+0+19700 (secure 0 0 0) 0x7f3eb4148d80 con 0x7f3eb40cbbf0 2023-05-22T15:27:44.592 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:27:44.592 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":29,"fsid":"b2719834-7847-4d98-90a6-f65d57f85c49","created":"2023-05-22T15:26:25.259314+0000","modified":"2023-05-22T15:27:31.182317+0000","last_up_change":"2023-05-22T15:26:36.448524+0000","last_in_change":"2023-05-22T15:26:27.675668+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-05-22T15:26:38.019781+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-05-22T15:26:43.570514+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-05-22T15:26:47.879159+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-05-22T15:26:48.522130+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data_ec","create_time":"2023-05-22T15:26:51.616142+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"9e7ff854-b25f-4cbb-a5c2-1799d9c53d2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":37021},{"type":"v1","addr":"172.21.15.16:6825","nonce":37021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":37021},{"type":"v1","addr":"172.21.15.16:6827","nonce":37021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":37021},{"type":"v1","addr":"172.21.15.16:6831","nonce":37021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":37021},{"type":"v1","addr":"172.21.15.16:6829","nonce":37021}]},"public_addr":"172.21.15.16:6825/37021","cluster_addr":"172.21.15.16:6827/37021","heartbeat_back_addr":"172.21.15.16:6831/37021","heartbeat_front_addr":"172.21.15.16:6829/37021","state":["exists","up"]},{"osd":1,"uuid":"4702fbc9-a927-4565-aca5-41dc69cc8264","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":37008},{"type":"v1","addr":"172.21.15.16:6801","nonce":37008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":37008},{"type":"v1","addr":"172.21.15.16:6803","nonce":37008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":37008},{"type":"v1","addr":"172.21.15.16:6807","nonce":37008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":37008},{"type":"v1","addr":"172.21.15.16:6805","nonce":37008}]},"public_addr":"172.21.15.16:6801/37008","cluster_addr":"172.21.15.16:6803/37008","heartbeat_back_addr":"172.21.15.16:6807/37008","heartbeat_front_addr":"172.21.15.16:6805/37008","state":["exists","up"]},{"osd":2,"uuid":"1e7bba46-4eba-47ef-bb81-5e2d371638ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":37009},{"type":"v1","addr":"172.21.15.16:6809","nonce":37009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":37009},{"type":"v1","addr":"172.21.15.16:6811","nonce":37009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":37009},{"type":"v1","addr":"172.21.15.16:6815","nonce":37009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":37009},{"type":"v1","addr":"172.21.15.16:6813","nonce":37009}]},"public_addr":"172.21.15.16:6809/37009","cluster_addr":"172.21.15.16:6811/37009","heartbeat_back_addr":"172.21.15.16:6815/37009","heartbeat_front_addr":"172.21.15.16:6813/37009","state":["exists","up"]},{"osd":3,"uuid":"383cacc0-5a88-4904-990e-498ca59ef48a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":37020},{"type":"v1","addr":"172.21.15.16:6817","nonce":37020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":37020},{"type":"v1","addr":"172.21.15.16:6819","nonce":37020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":37020},{"type":"v1","addr":"172.21.15.16:6823","nonce":37020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":37020},{"type":"v1","addr":"172.21.15.16:6821","nonce":37020}]},"public_addr":"172.21.15.16:6817/37020","cluster_addr":"172.21.15.16:6819/37020","heartbeat_back_addr":"172.21.15.16:6823/37020","heartbeat_front_addr":"172.21.15.16:6821/37020","state":["exists","up"]},{"osd":4,"uuid":"970500ef-95b7-42c1-b080-eed4fae7013d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":37353},{"type":"v1","addr":"172.21.15.62:6817","nonce":37353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6819","nonce":37353},{"type":"v1","addr":"172.21.15.62:6822","nonce":37353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37353},{"type":"v1","addr":"172.21.15.62:6831","nonce":37353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37353},{"type":"v1","addr":"172.21.15.62:6826","nonce":37353}]},"public_addr":"172.21.15.62:6817/37353","cluster_addr":"172.21.15.62:6822/37353","heartbeat_back_addr":"172.21.15.62:6831/37353","heartbeat_front_addr":"172.21.15.62:6826/37353","state":["exists","up"]},{"osd":5,"uuid":"022d67db-d4d8-445b-b1b9-8b446a00f5c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":37352},{"type":"v1","addr":"172.21.15.62:6801","nonce":37352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":37352},{"type":"v1","addr":"172.21.15.62:6803","nonce":37352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":37352},{"type":"v1","addr":"172.21.15.62:6807","nonce":37352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":37352},{"type":"v1","addr":"172.21.15.62:6805","nonce":37352}]},"public_addr":"172.21.15.62:6801/37352","cluster_addr":"172.21.15.62:6803/37352","heartbeat_back_addr":"172.21.15.62:6807/37352","heartbeat_front_addr":"172.21.15.62:6805/37352","state":["exists","up"]},{"osd":6,"uuid":"42bd372b-6688-4a1b-8407-e16e8c266e33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":37354},{"type":"v1","addr":"172.21.15.62:6820","nonce":37354}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":37354},{"type":"v1","addr":"172.21.15.62:6823","nonce":37354}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6829","nonce":37354},{"type":"v1","addr":"172.21.15.62:6830","nonce":37354}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6825","nonce":37354},{"type":"v1","addr":"172.21.15.62:6827","nonce":37354}]},"public_addr":"172.21.15.62:6820/37354","cluster_addr":"172.21.15.62:6823/37354","heartbeat_back_addr":"172.21.15.62:6830/37354","heartbeat_front_addr":"172.21.15.62:6827/37354","state":["exists","up"]},{"osd":7,"uuid":"ac70b2a3-f41b-4bf2-b227-7f2cbb55426d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37355},{"type":"v1","addr":"172.21.15.62:6809","nonce":37355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37355},{"type":"v1","addr":"172.21.15.62:6811","nonce":37355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":37355},{"type":"v1","addr":"172.21.15.62:6815","nonce":37355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":37355},{"type":"v1","addr":"172.21.15.62:6813","nonce":37355}]},"public_addr":"172.21.15.62:6809/37355","cluster_addr":"172.21.15.62:6811/37355","heartbeat_back_addr":"172.21.15.62:6815/37355","heartbeat_front_addr":"172.21.15.62:6813/37355","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.368896+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.107273+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.276888+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.287502+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.124791+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.251937+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.255679+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.232797+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"4.23","mappings":[{"from":3,"to":2}]},{"pgid":"4.31","mappings":[{"from":1,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-05-22T15:27:44.593 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:44.587+0000 7f3eba05f640 1 -- 172.21.15.16:0/2877320447 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f3e940653b0 msgr2=0x7f3e94067860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:44.594 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:44.587+0000 7f3eba05f640 1 --2- 172.21.15.16:0/2877320447 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f3e940653b0 0x7f3e94067860 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f3e9c00b930 tx=0x7f3e9c008040 comp rx=0 tx=0).stop 2023-05-22T15:27:44.594 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:44.587+0000 7f3eba05f640 1 -- 172.21.15.16:0/2877320447 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3eb40cbbf0 msgr2=0x7f3eb4148770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:44.594 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:44.587+0000 7f3eba05f640 1 --2- 172.21.15.16:0/2877320447 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3eb40cbbf0 0x7f3eb4148770 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f3ea4001570 tx=0x7f3ea400cd20 comp rx=0 tx=0).stop 2023-05-22T15:27:44.594 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:44.587+0000 7f3eba05f640 1 -- 172.21.15.16:0/2877320447 shutdown_connections 2023-05-22T15:27:44.594 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:44.587+0000 7f3eba05f640 1 --2- 172.21.15.16:0/2877320447 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f3e940653b0 0x7f3e94067860 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:44.594 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:44.587+0000 7f3eba05f640 1 --2- 172.21.15.16:0/2877320447 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3eb41531d0 0x7f3eb4151950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:44.594 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:44.587+0000 7f3eba05f640 1 --2- 172.21.15.16:0/2877320447 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3eb40cbbf0 0x7f3eb4148770 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:44.594 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:44.587+0000 7f3eba05f640 1 --2- 172.21.15.16:0/2877320447 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3eb40b0250 0x7f3eb41502c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:44.595 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:44.587+0000 7f3eba05f640 1 -- 172.21.15.16:0/2877320447 >> 172.21.15.16:0/2877320447 conn(0x7f3eb4069720 msgr2=0x7f3eb406f580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:44.595 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:44.587+0000 7f3eba05f640 1 -- 172.21.15.16:0/2877320447 shutdown_connections 2023-05-22T15:27:44.595 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:44.587+0000 7f3eba05f640 1 -- 172.21.15.16:0/2877320447 wait complete. 2023-05-22T15:27:44.609 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-05-22T15:26:38.019781+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-05-22T15:26:43.570514+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2023-05-22T15:26:47.879159+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2023-05-22T15:26:48.522130+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-05-22T15:26:51.616142+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '27', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-05-22T15:27:44.610 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-05-22T15:27:44.739 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:44.731+0000 7f7a7f44d640 1 Processor -- start 2023-05-22T15:27:44.740 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:44.731+0000 7f7a7f44d640 1 -- start start 2023-05-22T15:27:44.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:44.731+0000 7f7a7f44d640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7a7815bcd0 0x7f7a7815c120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:44.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:44.731+0000 7f7a7f44d640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a78158cd0 con 0x7f7a78158210 2023-05-22T15:27:44.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:44.731+0000 7f7a7f44d640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a7815c790 con 0x7f7a780c8660 2023-05-22T15:27:44.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:44.731+0000 7f7a7f44d640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a7815ca70 con 0x7f7a7815bcd0 2023-05-22T15:27:44.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:44.731+0000 7f7a7d1e9640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f7a78158210 0x7f7a78159160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:45636/0 (socket says 172.21.15.16:45636) 2023-05-22T15:27:44.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:44.731+0000 7f7a7d1e9640 1 -- 172.21.15.16:0/1847892100 learned_addr learned my addr 172.21.15.16:0/1847892100 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:27:44.744 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:44.731+0000 7f7a7c9e8640 1 --2- 172.21.15.16:0/1847892100 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7a7815bcd0 0x7f7a7815c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:44.744 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:44.731+0000 7f7a6ffff640 1 -- 172.21.15.16:0/1847892100 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 627731992 0 0) 0x7f7a78158cd0 con 0x7f7a78158210 2023-05-22T15:27:44.744 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:44.731+0000 7f7a6ffff640 1 -- 172.21.15.16:0/1847892100 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a64003530 con 0x7f7a78158210 2023-05-22T15:27:44.744 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:44.735+0000 7f7a6ffff640 1 -- 172.21.15.16:0/1847892100 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3125757464 0 0) 0x7f7a64003530 con 0x7f7a78158210 2023-05-22T15:27:44.744 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:44.735+0000 7f7a6ffff640 1 -- 172.21.15.16:0/1847892100 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7a7815bcd0 msgr2=0x7f7a7815c120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:44.744 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:44.735+0000 7f7a6ffff640 1 --2- 172.21.15.16:0/1847892100 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7a7815bcd0 0x7f7a7815c120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:44.744 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:44.735+0000 7f7a6ffff640 1 -- 172.21.15.16:0/1847892100 >> v1:172.21.15.62:6789/0 conn(0x7f7a780c8660 legacy=0x7f7a780ca690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:44.744 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:44.735+0000 7f7a6ffff640 1 -- 172.21.15.16:0/1847892100 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a78158960 con 0x7f7a78158210 2023-05-22T15:27:44.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:44.735+0000 7f7a6ffff640 1 -- 172.21.15.16:0/1847892100 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7f7a74001680 con 0x7f7a78158210 2023-05-22T15:27:44.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:44.735+0000 7f7a6ffff640 1 -- 172.21.15.16:0/1847892100 >> v1:172.21.15.16:6789/0 conn(0x7f7a78158210 legacy=0x7f7a78159160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:44.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:44.735+0000 7f7a6ffff640 1 --2- 172.21.15.16:0/1847892100 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7a640039b0 0x7f7a64003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:44.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:44.735+0000 7f7a6ffff640 1 --2- 172.21.15.16:0/1847892100 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7a640047d0 0x7f7a64004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:44.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:44.735+0000 7f7a6ffff640 1 --2- 172.21.15.16:0/1847892100 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7a7815bcd0 0x7f7a640051a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:44.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:44.735+0000 7f7a6ffff640 1 -- 172.21.15.16:0/1847892100 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a7815ca70 con 0x7f7a640047d0 2023-05-22T15:27:44.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:44.735+0000 7f7a6ffff640 1 -- 172.21.15.16:0/1847892100 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a6400ba90 con 0x7f7a640039b0 2023-05-22T15:27:44.746 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:44.735+0000 7f7a6ffff640 1 -- 172.21.15.16:0/1847892100 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a6400bc50 con 0x7f7a7815bcd0 2023-05-22T15:27:44.746 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:44.735+0000 7f7a7c9e8640 1 --2- 172.21.15.16:0/1847892100 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7a640047d0 0x7f7a64004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:44.746 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:44.735+0000 7f7a7d1e9640 1 --2- 172.21.15.16:0/1847892100 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7a640039b0 0x7f7a64003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:44.746 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:44.735+0000 7f7a7d9ea640 1 --2- 172.21.15.16:0/1847892100 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7a7815bcd0 0x7f7a640051a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:44.746 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:44.735+0000 7f7a7c9e8640 1 -- 172.21.15.16:0/1847892100 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7a7815bcd0 msgr2=0x7f7a640051a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:44.746 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:44.735+0000 7f7a7c9e8640 1 --2- 172.21.15.16:0/1847892100 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7a7815bcd0 0x7f7a640051a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:44.746 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:44.735+0000 7f7a7c9e8640 1 -- 172.21.15.16:0/1847892100 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7a640039b0 msgr2=0x7f7a64003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:44.746 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:44.735+0000 7f7a7c9e8640 1 --2- 172.21.15.16:0/1847892100 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7a640039b0 0x7f7a64003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:44.747 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:44.735+0000 7f7a7c9e8640 1 -- 172.21.15.16:0/1847892100 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a64003530 con 0x7f7a640047d0 2023-05-22T15:27:44.747 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:44.739+0000 7f7a7c9e8640 1 --2- 172.21.15.16:0/1847892100 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7a640047d0 0x7f7a64004c40 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f7a68009ae0 tx=0x7f7a6801ca50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60a3be5339bb285d server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:44.747 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:44.739+0000 7f7a6ffff640 1 -- 172.21.15.16:0/1847892100 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a680a5530 con 0x7f7a640047d0 2023-05-22T15:27:44.748 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:44.739+0000 7f7a6ffff640 1 -- 172.21.15.16:0/1847892100 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a680a5690 con 0x7f7a640047d0 2023-05-22T15:27:44.748 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:44.739+0000 7f7a6ffff640 1 -- 172.21.15.16:0/1847892100 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a680a5a40 con 0x7f7a640047d0 2023-05-22T15:27:44.748 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:44.739+0000 7f7a7f44d640 1 -- 172.21.15.16:0/1847892100 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7a640047d0 msgr2=0x7f7a64004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:44.749 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:44.739+0000 7f7a7f44d640 1 --2- 172.21.15.16:0/1847892100 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7a640047d0 0x7f7a64004c40 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f7a68009ae0 tx=0x7f7a6801ca50 comp rx=0 tx=0).stop 2023-05-22T15:27:44.749 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:44.739+0000 7f7a7d9ea640 1 -- 172.21.15.16:0/1847892100 reap_dead start 2023-05-22T15:27:44.749 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:44.739+0000 7f7a7f44d640 1 -- 172.21.15.16:0/1847892100 shutdown_connections 2023-05-22T15:27:44.749 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:44.739+0000 7f7a7f44d640 1 -- 172.21.15.16:0/1847892100 >> 172.21.15.16:0/1847892100 conn(0x7f7a78069720 msgr2=0x7f7a78069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:44.749 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:44.739+0000 7f7a7f44d640 1 -- 172.21.15.16:0/1847892100 shutdown_connections 2023-05-22T15:27:44.749 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:44.739+0000 7f7a7f44d640 1 -- 172.21.15.16:0/1847892100 wait complete. 2023-05-22T15:27:44.751 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:44.743+0000 7f7a7f44d640 1 Processor -- start 2023-05-22T15:27:44.751 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:44.743+0000 7f7a7f44d640 1 -- start start 2023-05-22T15:27:44.751 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:44.743+0000 7f7a7f44d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7a781662f0 0x7f7a7815a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:44.751 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:44.743+0000 7f7a7f44d640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7a780b2f70 0x7f7a78159a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:44.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:44.743+0000 7f7a7f44d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7a781666c0 0x7f7a780c5880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:44.753 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:44.743+0000 7f7a7f44d640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a7815b630 con 0x7f7a781662f0 2023-05-22T15:27:44.753 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:44.743+0000 7f7a7f44d640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a7815b070 con 0x7f7a781666c0 2023-05-22T15:27:44.753 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:44.743+0000 7f7a7f44d640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a7815b350 con 0x7f7a780b2f70 2023-05-22T15:27:44.754 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:44.743+0000 7f7a7d1e9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7a781662f0 0x7f7a7815a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:44.754 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:44.743+0000 7f7a7d1e9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7a781662f0 0x7f7a7815a790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41668/0 (socket says 172.21.15.16:41668) 2023-05-22T15:27:44.754 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:44.743+0000 7f7a7d1e9640 1 -- 172.21.15.16:0/2294639176 learned_addr learned my addr 172.21.15.16:0/2294639176 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:44.754 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:44.743+0000 7f7a7d9ea640 1 --2- 172.21.15.16:0/2294639176 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7a781666c0 0x7f7a780c5880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:44.754 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:44.743+0000 7f7a7d1e9640 1 -- 172.21.15.16:0/2294639176 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7a780b2f70 msgr2=0x7f7a78159a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:44.754 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:44.743+0000 7f7a7d1e9640 1 --2- 172.21.15.16:0/2294639176 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7a780b2f70 0x7f7a78159a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:44.754 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:44.743+0000 7f7a7d1e9640 1 -- 172.21.15.16:0/2294639176 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7a781666c0 msgr2=0x7f7a780c5880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:44.755 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:44.743+0000 7f7a7d1e9640 1 --2- 172.21.15.16:0/2294639176 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7a781666c0 0x7f7a780c5880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:44.755 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:44.743+0000 7f7a7d1e9640 1 -- 172.21.15.16:0/2294639176 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a780c9000 con 0x7f7a781662f0 2023-05-22T15:27:44.755 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:44.743+0000 7f7a7d9ea640 1 --2- 172.21.15.16:0/2294639176 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7a781666c0 0x7f7a780c5880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:27:44.755 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:44.743+0000 7f7a7d1e9640 1 --2- 172.21.15.16:0/2294639176 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7a781662f0 0x7f7a7815a790 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f7a740072c0 tx=0x7f7a74004f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:44.755 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:44.743+0000 7f7a6dffb640 1 -- 172.21.15.16:0/2294639176 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a74001470 con 0x7f7a781662f0 2023-05-22T15:27:44.755 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:44.743+0000 7f7a6dffb640 1 -- 172.21.15.16:0/2294639176 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a7400fa20 con 0x7f7a781662f0 2023-05-22T15:27:44.755 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:44.743+0000 7f7a7f44d640 1 -- 172.21.15.16:0/2294639176 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a780c8e70 con 0x7f7a781662f0 2023-05-22T15:27:44.755 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:44.743+0000 7f7a6dffb640 1 -- 172.21.15.16:0/2294639176 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a7400fdd0 con 0x7f7a781662f0 2023-05-22T15:27:44.756 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:44.743+0000 7f7a7f44d640 1 -- 172.21.15.16:0/2294639176 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a780cada0 con 0x7f7a781662f0 2023-05-22T15:27:44.756 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:44.743+0000 7f7a7f44d640 1 -- 172.21.15.16:0/2294639176 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a7804dbe0 con 0x7f7a781662f0 2023-05-22T15:27:44.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:44.743+0000 7f7a6dffb640 1 -- 172.21.15.16:0/2294639176 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7f7a7401d460 con 0x7f7a781662f0 2023-05-22T15:27:44.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:44.747+0000 7f7a6dffb640 1 --2- 172.21.15.16:0/2294639176 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f7a5c0653b0 0x7f7a5c067860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:44.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:44.747+0000 7f7a6dffb640 1 -- 172.21.15.16:0/2294639176 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f7a74082e30 con 0x7f7a781662f0 2023-05-22T15:27:44.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:44.747+0000 7f7a7c9e8640 1 --2- 172.21.15.16:0/2294639176 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f7a5c0653b0 0x7f7a5c067860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:44.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:44.747+0000 7f7a7c9e8640 1 --2- 172.21.15.16:0/2294639176 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f7a5c0653b0 0x7f7a5c067860 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f7a680af000 tx=0x7f7a68008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:44.758 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:44.751+0000 7f7a6dffb640 1 -- 172.21.15.16:0/2294639176 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f7a74050720 con 0x7f7a781662f0 2023-05-22T15:27:44.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:44.903+0000 7f7a7f44d640 1 -- 172.21.15.16:0/2294639176 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f7a7806c480 con 0x7f7a781662f0 2023-05-22T15:27:44.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:44.903+0000 7f7a6dffb640 1 -- 172.21.15.16:0/2294639176 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v29) v1 ==== 93+0+10 (secure 0 0 0) 0x7f7a74053d40 con 0x7f7a781662f0 2023-05-22T15:27:44.911 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2023-05-22T15:27:44.912 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:44.903+0000 7f7a7f44d640 1 -- 172.21.15.16:0/2294639176 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f7a5c0653b0 msgr2=0x7f7a5c067860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:44.913 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:44.903+0000 7f7a7f44d640 1 --2- 172.21.15.16:0/2294639176 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f7a5c0653b0 0x7f7a5c067860 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f7a680af000 tx=0x7f7a68008040 comp rx=0 tx=0).stop 2023-05-22T15:27:44.913 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:44.903+0000 7f7a7f44d640 1 -- 172.21.15.16:0/2294639176 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7a781662f0 msgr2=0x7f7a7815a790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:44.913 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:44.903+0000 7f7a7f44d640 1 --2- 172.21.15.16:0/2294639176 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7a781662f0 0x7f7a7815a790 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f7a740072c0 tx=0x7f7a74004f90 comp rx=0 tx=0).stop 2023-05-22T15:27:44.913 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:44.903+0000 7f7a7f44d640 1 -- 172.21.15.16:0/2294639176 shutdown_connections 2023-05-22T15:27:44.913 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:44.903+0000 7f7a7f44d640 1 --2- 172.21.15.16:0/2294639176 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7a781666c0 0x7f7a780c5880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:44.913 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:44.903+0000 7f7a7f44d640 1 --2- 172.21.15.16:0/2294639176 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f7a5c0653b0 0x7f7a5c067860 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:44.913 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:44.903+0000 7f7a7f44d640 1 --2- 172.21.15.16:0/2294639176 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7a780b2f70 0x7f7a78159a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:44.914 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:44.903+0000 7f7a7f44d640 1 --2- 172.21.15.16:0/2294639176 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7a781662f0 0x7f7a7815a790 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:44.914 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:44.903+0000 7f7a7f44d640 1 -- 172.21.15.16:0/2294639176 >> 172.21.15.16:0/2294639176 conn(0x7f7a78069720 msgr2=0x7f7a78165670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:44.914 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:44.903+0000 7f7a7f44d640 1 -- 172.21.15.16:0/2294639176 shutdown_connections 2023-05-22T15:27:44.914 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:44.903+0000 7f7a7f44d640 1 -- 172.21.15.16:0/2294639176 wait complete. 2023-05-22T15:27:44.930 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-05-22T15:27:45.053 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.047+0000 7f7068d93640 1 Processor -- start 2023-05-22T15:27:45.054 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.047+0000 7f7068d93640 1 -- start start 2023-05-22T15:27:45.055 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.047+0000 7f7068d93640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f7064159b10 0x7f70640b1f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:45.055 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.047+0000 7f7068d93640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f7064069e30 0x7f7064070c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:45.056 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.047+0000 7f7068d93640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f706406d670 con 0x7f706406c5c0 2023-05-22T15:27:45.056 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.047+0000 7f7068d93640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f706406caf0 con 0x7f7064159b10 2023-05-22T15:27:45.056 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.047+0000 7f7068d93640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f706406cdd0 con 0x7f7064069e30 2023-05-22T15:27:45.056 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.047+0000 7f706259c640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f7064159b10 0x7f70640b1f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:45.056 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.047+0000 7f706259c640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f7064159b10 0x7f70640b1f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41680/0 (socket says 172.21.15.16:41680) 2023-05-22T15:27:45.056 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.047+0000 7f706259c640 1 -- 172.21.15.16:0/1670537649 learned_addr learned my addr 172.21.15.16:0/1670537649 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:45.056 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.047+0000 7f7062d9d640 1 --2- 172.21.15.16:0/1670537649 >> v2:172.21.15.62:3300/0 conn(0x7f7064069e30 0x7f7064070c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:45.057 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.047+0000 7f706259c640 1 -- 172.21.15.16:0/1670537649 >> v2:172.21.15.62:3300/0 conn(0x7f7064069e30 msgr2=0x7f7064070c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:45.057 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.047+0000 7f706259c640 1 --2- 172.21.15.16:0/1670537649 >> v2:172.21.15.62:3300/0 conn(0x7f7064069e30 0x7f7064070c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:45.057 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.047+0000 7f706259c640 1 -- 172.21.15.16:0/1670537649 >> v1:172.21.15.62:6789/0 conn(0x7f706406c5c0 legacy=0x7f7064069720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:45.057 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.047+0000 7f706259c640 1 -- 172.21.15.16:0/1670537649 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f706406d0b0 con 0x7f7064159b10 2023-05-22T15:27:45.058 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.051+0000 7f706259c640 1 --2- 172.21.15.16:0/1670537649 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7064159b10 0x7f70640b1f70 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f706406ff50 tx=0x7f705401c400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9cd07bba03d42b7 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:45.058 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.051+0000 7f706159a640 1 -- 172.21.15.16:0/1670537649 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70540ab400 con 0x7f7064159b10 2023-05-22T15:27:45.059 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.051+0000 7f706159a640 1 -- 172.21.15.16:0/1670537649 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70540a7050 con 0x7f7064159b10 2023-05-22T15:27:45.059 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.051+0000 7f706159a640 1 -- 172.21.15.16:0/1670537649 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70540ab780 con 0x7f7064159b10 2023-05-22T15:27:45.059 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.051+0000 7f7068d93640 1 -- 172.21.15.16:0/1670537649 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7064159b10 msgr2=0x7f70640b1f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:45.059 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.051+0000 7f7068d93640 1 --2- 172.21.15.16:0/1670537649 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7064159b10 0x7f70640b1f70 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f706406ff50 tx=0x7f705401c400 comp rx=0 tx=0).stop 2023-05-22T15:27:45.059 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.051+0000 7f7068d93640 1 -- 172.21.15.16:0/1670537649 shutdown_connections 2023-05-22T15:27:45.060 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.051+0000 7f7068d93640 1 --2- 172.21.15.16:0/1670537649 >> v2:172.21.15.62:3300/0 conn(0x7f7064069e30 0x7f7064070c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:45.060 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.051+0000 7f7068d93640 1 --2- 172.21.15.16:0/1670537649 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7064159b10 0x7f70640b1f70 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:45.060 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.051+0000 7f7068d93640 1 -- 172.21.15.16:0/1670537649 >> 172.21.15.16:0/1670537649 conn(0x7f70640b18e0 msgr2=0x7f70640cf120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:45.060 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.051+0000 7f7068d93640 1 -- 172.21.15.16:0/1670537649 shutdown_connections 2023-05-22T15:27:45.060 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.051+0000 7f7068d93640 1 -- 172.21.15.16:0/1670537649 wait complete. 2023-05-22T15:27:45.064 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.051+0000 7f7068d93640 1 Processor -- start 2023-05-22T15:27:45.064 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.051+0000 7f7068d93640 1 -- start start 2023-05-22T15:27:45.064 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.051+0000 7f7068d93640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7064069e30 0x7f706415c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:45.064 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.051+0000 7f7068d93640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f706406c5c0 0x7f7064158ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:45.065 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.051+0000 7f7068d93640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7064159b10 0x7f706414c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:45.065 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.051+0000 7f7068d93640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f706414d160 con 0x7f706406c5c0 2023-05-22T15:27:45.065 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.051+0000 7f7068d93640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f706414d440 con 0x7f7064069e30 2023-05-22T15:27:45.065 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.051+0000 7f7068d93640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f706414d720 con 0x7f7064159b10 2023-05-22T15:27:45.065 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.051+0000 7f7061d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f706406c5c0 0x7f7064158ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:45.065 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.051+0000 7f7061d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f706406c5c0 0x7f7064158ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41682/0 (socket says 172.21.15.16:41682) 2023-05-22T15:27:45.066 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.051+0000 7f7061d9b640 1 -- 172.21.15.16:0/477473817 learned_addr learned my addr 172.21.15.16:0/477473817 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:45.066 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.051+0000 7f7061d9b640 1 -- 172.21.15.16:0/477473817 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7064159b10 msgr2=0x7f706414c5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:45.066 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.051+0000 7f7061d9b640 1 --2- 172.21.15.16:0/477473817 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7064159b10 0x7f706414c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:45.066 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.051+0000 7f7061d9b640 1 -- 172.21.15.16:0/477473817 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7064069e30 msgr2=0x7f706415c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:45.066 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.051+0000 7f7061d9b640 1 --2- 172.21.15.16:0/477473817 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7064069e30 0x7f706415c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:45.066 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.051+0000 7f7061d9b640 1 -- 172.21.15.16:0/477473817 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70540094d0 con 0x7f706406c5c0 2023-05-22T15:27:45.067 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.051+0000 7f7061d9b640 1 --2- 172.21.15.16:0/477473817 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f706406c5c0 0x7f7064158ad0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f705800c490 tx=0x7f7058001e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:45.067 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.051+0000 7f7052ffd640 1 -- 172.21.15.16:0/477473817 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7058003600 con 0x7f706406c5c0 2023-05-22T15:27:45.067 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.051+0000 7f7052ffd640 1 -- 172.21.15.16:0/477473817 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7058003780 con 0x7f706406c5c0 2023-05-22T15:27:45.067 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.051+0000 7f7068d93640 1 -- 172.21.15.16:0/477473817 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70640cb9c0 con 0x7f706406c5c0 2023-05-22T15:27:45.067 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.051+0000 7f7052ffd640 1 -- 172.21.15.16:0/477473817 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7058003b70 con 0x7f706406c5c0 2023-05-22T15:27:45.067 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.051+0000 7f7068d93640 1 -- 172.21.15.16:0/477473817 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70640cb830 con 0x7f706406c5c0 2023-05-22T15:27:45.067 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.055+0000 7f7068d93640 1 -- 172.21.15.16:0/477473817 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7064166730 con 0x7f706406c5c0 2023-05-22T15:27:45.068 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.055+0000 7f7052ffd640 1 -- 172.21.15.16:0/477473817 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7f7058003cd0 con 0x7f706406c5c0 2023-05-22T15:27:45.068 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.055+0000 7f7052ffd640 1 --2- 172.21.15.16:0/477473817 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f7044065380 0x7f7044067830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:45.068 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.055+0000 7f7052ffd640 1 -- 172.21.15.16:0/477473817 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f7058081e70 con 0x7f706406c5c0 2023-05-22T15:27:45.068 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.055+0000 7f706259c640 1 --2- 172.21.15.16:0/477473817 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f7044065380 0x7f7044067830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:45.068 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.055+0000 7f706259c640 1 --2- 172.21.15.16:0/477473817 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f7044065380 0x7f7044067830 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f70540ad000 tx=0x7f7054004740 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:45.069 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.059+0000 7f7052ffd640 1 -- 172.21.15.16:0/477473817 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f705804f720 con 0x7f706406c5c0 2023-05-22T15:27:45.224 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.215+0000 7f7068d93640 1 -- 172.21.15.16:0/477473817 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f706415c690 con 0x7f706406c5c0 2023-05-22T15:27:45.225 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.215+0000 7f7052ffd640 1 -- 172.21.15.16:0/477473817 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v29) v1 ==== 92+0+10 (secure 0 0 0) 0x7f70580170a0 con 0x7f706406c5c0 2023-05-22T15:27:45.225 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 8 2023-05-22T15:27:45.227 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.219+0000 7f7068d93640 1 -- 172.21.15.16:0/477473817 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f7044065380 msgr2=0x7f7044067830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:45.227 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.219+0000 7f7068d93640 1 --2- 172.21.15.16:0/477473817 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f7044065380 0x7f7044067830 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f70540ad000 tx=0x7f7054004740 comp rx=0 tx=0).stop 2023-05-22T15:27:45.228 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.219+0000 7f7068d93640 1 -- 172.21.15.16:0/477473817 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f706406c5c0 msgr2=0x7f7064158ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:45.228 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.219+0000 7f7068d93640 1 --2- 172.21.15.16:0/477473817 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f706406c5c0 0x7f7064158ad0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f705800c490 tx=0x7f7058001e80 comp rx=0 tx=0).stop 2023-05-22T15:27:45.228 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.219+0000 7f7068d93640 1 -- 172.21.15.16:0/477473817 shutdown_connections 2023-05-22T15:27:45.228 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.219+0000 7f7068d93640 1 --2- 172.21.15.16:0/477473817 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f7044065380 0x7f7044067830 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:45.229 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.219+0000 7f7068d93640 1 --2- 172.21.15.16:0/477473817 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7064159b10 0x7f706414c5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:45.229 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.219+0000 7f7068d93640 1 --2- 172.21.15.16:0/477473817 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f706406c5c0 0x7f7064158ad0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:45.229 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.219+0000 7f7068d93640 1 --2- 172.21.15.16:0/477473817 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7064069e30 0x7f706415c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:45.229 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.219+0000 7f7068d93640 1 -- 172.21.15.16:0/477473817 >> 172.21.15.16:0/477473817 conn(0x7f70640b18e0 msgr2=0x7f7064153e20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:45.229 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.219+0000 7f7068d93640 1 -- 172.21.15.16:0/477473817 shutdown_connections 2023-05-22T15:27:45.230 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.219+0000 7f7068d93640 1 -- 172.21.15.16:0/477473817 wait complete. 2023-05-22T15:27:45.243 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-05-22T15:27:45.373 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.367+0000 7f063fac3640 1 Processor -- start 2023-05-22T15:27:45.374 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.367+0000 7f063fac3640 1 -- start start 2023-05-22T15:27:45.375 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.367+0000 7f063fac3640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f0638153fe0 0x7f063806d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:45.375 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.367+0000 7f063fac3640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0638069720 con 0x7f0638070600 2023-05-22T15:27:45.375 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.367+0000 7f063fac3640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06381543b0 con 0x7f06380cae20 2023-05-22T15:27:45.376 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.367+0000 7f063fac3640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f063815e360 con 0x7f0638153fe0 2023-05-22T15:27:45.376 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.367+0000 7f063e060640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f0638070600 0x7f06380ad410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:45640/0 (socket says 172.21.15.16:45640) 2023-05-22T15:27:45.376 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.367+0000 7f063e060640 1 -- 172.21.15.16:0/3954314437 learned_addr learned my addr 172.21.15.16:0/3954314437 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:27:45.376 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.367+0000 7f063d85f640 1 --2- 172.21.15.16:0/3954314437 >> v2:172.21.15.62:3300/0 conn(0x7f0638153fe0 0x7f063806d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:45.376 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.367+0000 7f063c85d640 1 -- 172.21.15.16:0/3954314437 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4064453741 0 0) 0x7f0638069720 con 0x7f0638070600 2023-05-22T15:27:45.376 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.367+0000 7f063c85d640 1 -- 172.21.15.16:0/3954314437 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0624003530 con 0x7f0638070600 2023-05-22T15:27:45.377 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.367+0000 7f063c85d640 1 -- 172.21.15.16:0/3954314437 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2232642741 0 0) 0x7f0624003530 con 0x7f0638070600 2023-05-22T15:27:45.378 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.367+0000 7f063c85d640 1 -- 172.21.15.16:0/3954314437 >> v2:172.21.15.62:3300/0 conn(0x7f0638153fe0 msgr2=0x7f063806d3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:45.378 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.367+0000 7f063c85d640 1 --2- 172.21.15.16:0/3954314437 >> v2:172.21.15.62:3300/0 conn(0x7f0638153fe0 0x7f063806d3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:45.378 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.367+0000 7f063c85d640 1 -- 172.21.15.16:0/3954314437 >> v1:172.21.15.62:6789/0 conn(0x7f06380cae20 legacy=0x7f063806fef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:45.378 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.367+0000 7f063c85d640 1 -- 172.21.15.16:0/3954314437 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f063815d5a0 con 0x7f0638070600 2023-05-22T15:27:45.378 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.367+0000 7f063c85d640 1 -- 172.21.15.16:0/3954314437 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7f0630001a00 con 0x7f0638070600 2023-05-22T15:27:45.378 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.367+0000 7f063c85d640 1 -- 172.21.15.16:0/3954314437 >> v1:172.21.15.16:6789/0 conn(0x7f0638070600 legacy=0x7f06380ad410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:45.378 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.367+0000 7f063c85d640 1 --2- 172.21.15.16:0/3954314437 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0624003980 0x7f0624003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:45.379 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.367+0000 7f063c85d640 1 --2- 172.21.15.16:0/3954314437 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f06240047d0 0x7f0624004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:45.379 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.367+0000 7f063c85d640 1 --2- 172.21.15.16:0/3954314437 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0624005180 0x7f062400b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:45.379 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.367+0000 7f063c85d640 1 -- 172.21.15.16:0/3954314437 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f063815e360 con 0x7f0624005180 2023-05-22T15:27:45.379 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.367+0000 7f063c85d640 1 -- 172.21.15.16:0/3954314437 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f062400c000 con 0x7f0624003980 2023-05-22T15:27:45.379 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.367+0000 7f063c85d640 1 -- 172.21.15.16:0/3954314437 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f062400c1c0 con 0x7f06240047d0 2023-05-22T15:27:45.379 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.367+0000 7f063e060640 1 --2- 172.21.15.16:0/3954314437 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0624005180 0x7f062400b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:45.379 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.367+0000 7f063d85f640 1 --2- 172.21.15.16:0/3954314437 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0624003980 0x7f0624003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:45.380 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.367+0000 7f063d05e640 1 --2- 172.21.15.16:0/3954314437 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f06240047d0 0x7f0624004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:45.380 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.367+0000 7f063e060640 1 -- 172.21.15.16:0/3954314437 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f06240047d0 msgr2=0x7f0624004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:45.380 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.367+0000 7f063e060640 1 --2- 172.21.15.16:0/3954314437 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f06240047d0 0x7f0624004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:45.380 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.367+0000 7f063e060640 1 -- 172.21.15.16:0/3954314437 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0624003980 msgr2=0x7f0624003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:45.380 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.367+0000 7f063e060640 1 --2- 172.21.15.16:0/3954314437 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0624003980 0x7f0624003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:45.380 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.367+0000 7f063e060640 1 -- 172.21.15.16:0/3954314437 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0624003530 con 0x7f0624005180 2023-05-22T15:27:45.380 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.367+0000 7f063d85f640 1 --2- 172.21.15.16:0/3954314437 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0624003980 0x7f0624003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:27:45.382 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.371+0000 7f063e060640 1 -- 172.21.15.16:0/3954314437 reap_dead start 2023-05-22T15:27:45.382 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.371+0000 7f063e060640 1 --2- 172.21.15.16:0/3954314437 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0624005180 0x7f062400b990 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f0638070f80 tx=0x7f063001d2e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a07e6b691584792b server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:45.383 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.371+0000 7f063c85d640 1 -- 172.21.15.16:0/3954314437 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06300072d0 con 0x7f0624005180 2023-05-22T15:27:45.383 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.371+0000 7f063c85d640 1 -- 172.21.15.16:0/3954314437 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06300a8070 con 0x7f0624005180 2023-05-22T15:27:45.383 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.371+0000 7f063c85d640 1 -- 172.21.15.16:0/3954314437 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0630007690 con 0x7f0624005180 2023-05-22T15:27:45.383 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.371+0000 7f063fac3640 1 -- 172.21.15.16:0/3954314437 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0624005180 msgr2=0x7f062400b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:45.383 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.371+0000 7f063fac3640 1 --2- 172.21.15.16:0/3954314437 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0624005180 0x7f062400b990 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f0638070f80 tx=0x7f063001d2e0 comp rx=0 tx=0).stop 2023-05-22T15:27:45.383 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.371+0000 7f063fac3640 1 -- 172.21.15.16:0/3954314437 shutdown_connections 2023-05-22T15:27:45.383 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.371+0000 7f063fac3640 1 --2- 172.21.15.16:0/3954314437 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0624005180 0x7f062400b990 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:45.384 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.371+0000 7f063fac3640 1 -- 172.21.15.16:0/3954314437 >> 172.21.15.16:0/3954314437 conn(0x7f06380b1110 msgr2=0x7f06380ce830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:45.384 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.371+0000 7f063fac3640 1 -- 172.21.15.16:0/3954314437 shutdown_connections 2023-05-22T15:27:45.384 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.371+0000 7f063fac3640 1 -- 172.21.15.16:0/3954314437 wait complete. 2023-05-22T15:27:45.384 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.371+0000 7f063fac3640 1 Processor -- start 2023-05-22T15:27:45.384 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.371+0000 7f063fac3640 1 -- start start 2023-05-22T15:27:45.384 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.375+0000 7f063fac3640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0624005180 0x7f06381523f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:45.384 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.375+0000 7f063fac3640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f06380c5630 0x7f06380c6c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:45.385 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.375+0000 7f063fac3640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f06380c71a0 0x7f06381ef940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:45.385 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.375+0000 7f063fac3640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06380c7610 con 0x7f0624005180 2023-05-22T15:27:45.385 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.375+0000 7f063fac3640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0638152930 con 0x7f06380c5630 2023-05-22T15:27:45.385 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.375+0000 7f063fac3640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06381ef460 con 0x7f06380c71a0 2023-05-22T15:27:45.385 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.375+0000 7f063d85f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0624005180 0x7f06381523f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:45.386 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.375+0000 7f063d85f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0624005180 0x7f06381523f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41698/0 (socket says 172.21.15.16:41698) 2023-05-22T15:27:45.386 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.375+0000 7f063d85f640 1 -- 172.21.15.16:0/4215597283 learned_addr learned my addr 172.21.15.16:0/4215597283 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:45.386 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.375+0000 7f063d05e640 1 --2- 172.21.15.16:0/4215597283 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f06380c5630 0x7f06380c6c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:45.386 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.375+0000 7f063e060640 1 --2- 172.21.15.16:0/4215597283 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f06380c71a0 0x7f06381ef940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:45.386 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.375+0000 7f063d85f640 1 -- 172.21.15.16:0/4215597283 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f06380c71a0 msgr2=0x7f06381ef940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:45.387 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.375+0000 7f063d85f640 1 --2- 172.21.15.16:0/4215597283 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f06380c71a0 0x7f06381ef940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:45.387 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.375+0000 7f063d85f640 1 -- 172.21.15.16:0/4215597283 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f06380c5630 msgr2=0x7f06380c6c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:45.387 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.375+0000 7f063d85f640 1 --2- 172.21.15.16:0/4215597283 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f06380c5630 0x7f06380c6c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:45.387 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.375+0000 7f063d85f640 1 -- 172.21.15.16:0/4215597283 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f063815db60 con 0x7f0624005180 2023-05-22T15:27:45.387 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.375+0000 7f063d85f640 1 --2- 172.21.15.16:0/4215597283 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0624005180 0x7f06381523f0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f06340108d0 tx=0x7f0634010d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:45.387 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.375+0000 7f062e7fc640 1 -- 172.21.15.16:0/4215597283 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0634004220 con 0x7f0624005180 2023-05-22T15:27:45.387 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.375+0000 7f062e7fc640 1 -- 172.21.15.16:0/4215597283 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0634004380 con 0x7f0624005180 2023-05-22T15:27:45.387 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.375+0000 7f063fac3640 1 -- 172.21.15.16:0/4215597283 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f063815d880 con 0x7f0624005180 2023-05-22T15:27:45.388 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.375+0000 7f062e7fc640 1 -- 172.21.15.16:0/4215597283 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f063400eba0 con 0x7f0624005180 2023-05-22T15:27:45.388 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.375+0000 7f063fac3640 1 -- 172.21.15.16:0/4215597283 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f063815de40 con 0x7f0624005180 2023-05-22T15:27:45.389 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.375+0000 7f063fac3640 1 -- 172.21.15.16:0/4215597283 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f063804dbe0 con 0x7f0624005180 2023-05-22T15:27:45.389 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.375+0000 7f062e7fc640 1 -- 172.21.15.16:0/4215597283 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7f063400ed00 con 0x7f0624005180 2023-05-22T15:27:45.389 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.375+0000 7f062e7fc640 1 --2- 172.21.15.16:0/4215597283 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f061c0653b0 0x7f061c067860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:45.389 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.375+0000 7f062e7fc640 1 -- 172.21.15.16:0/4215597283 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f0634082ea0 con 0x7f0624005180 2023-05-22T15:27:45.390 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.375+0000 7f063d05e640 1 --2- 172.21.15.16:0/4215597283 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f061c0653b0 0x7f061c067860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:45.390 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.375+0000 7f063d05e640 1 --2- 172.21.15.16:0/4215597283 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f061c0653b0 0x7f061c067860 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f0628000c10 tx=0x7f0628009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:45.390 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.379+0000 7f062e7fc640 1 -- 172.21.15.16:0/4215597283 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f06340507f0 con 0x7f0624005180 2023-05-22T15:27:45.534 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.527+0000 7f063fac3640 1 -- 172.21.15.16:0/4215597283 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f06381ef5a0 con 0x7f0624005180 2023-05-22T15:27:45.535 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.527+0000 7f062e7fc640 1 -- 172.21.15.16:0/4215597283 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v29) v1 ==== 104+0+11 (secure 0 0 0) 0x7f0634053e10 con 0x7f0624005180 2023-05-22T15:27:45.535 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2023-05-22T15:27:45.536 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.527+0000 7f063fac3640 1 -- 172.21.15.16:0/4215597283 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f061c0653b0 msgr2=0x7f061c067860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:45.536 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.527+0000 7f063fac3640 1 --2- 172.21.15.16:0/4215597283 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f061c0653b0 0x7f061c067860 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f0628000c10 tx=0x7f0628009040 comp rx=0 tx=0).stop 2023-05-22T15:27:45.537 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.527+0000 7f063fac3640 1 -- 172.21.15.16:0/4215597283 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0624005180 msgr2=0x7f06381523f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:45.537 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.527+0000 7f063fac3640 1 --2- 172.21.15.16:0/4215597283 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0624005180 0x7f06381523f0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f06340108d0 tx=0x7f0634010d90 comp rx=0 tx=0).stop 2023-05-22T15:27:45.537 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.527+0000 7f063fac3640 1 -- 172.21.15.16:0/4215597283 shutdown_connections 2023-05-22T15:27:45.537 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.527+0000 7f063fac3640 1 --2- 172.21.15.16:0/4215597283 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f061c0653b0 0x7f061c067860 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:45.537 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.527+0000 7f063fac3640 1 --2- 172.21.15.16:0/4215597283 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f06380c71a0 0x7f06381ef940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:45.537 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.527+0000 7f063fac3640 1 --2- 172.21.15.16:0/4215597283 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f06380c5630 0x7f06380c6c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:45.537 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.527+0000 7f063fac3640 1 --2- 172.21.15.16:0/4215597283 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0624005180 0x7f06381523f0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:45.538 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.527+0000 7f063fac3640 1 -- 172.21.15.16:0/4215597283 >> 172.21.15.16:0/4215597283 conn(0x7f06380b1110 msgr2=0x7f06380aebb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:45.538 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.527+0000 7f063fac3640 1 -- 172.21.15.16:0/4215597283 shutdown_connections 2023-05-22T15:27:45.538 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.527+0000 7f063fac3640 1 -- 172.21.15.16:0/4215597283 wait complete. 2023-05-22T15:27:45.552 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-05-22T15:27:45.669 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.659+0000 7f8a68135640 1 Processor -- start 2023-05-22T15:27:45.669 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.663+0000 7f8a68135640 1 -- start start 2023-05-22T15:27:45.670 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.663+0000 7f8a68135640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f8a60159970 0x7f8a60159d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:45.671 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.663+0000 7f8a68135640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8a6015d680 0x7f8a600cb260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:45.671 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.663+0000 7f8a68135640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f8a6015dad0 0x7f8a6006b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:45.671 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.663+0000 7f8a68135640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a60070c30 con 0x7f8a6015dad0 2023-05-22T15:27:45.671 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.663+0000 7f8a68135640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a60070e60 con 0x7f8a60159970 2023-05-22T15:27:45.671 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.663+0000 7f8a68135640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a60071140 con 0x7f8a6015d680 2023-05-22T15:27:45.671 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.663+0000 7f8a666d2640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f8a6015dad0 0x7f8a6006b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:45.672 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.663+0000 7f8a666d2640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f8a6015dad0 0x7f8a6006b6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41702/0 (socket says 172.21.15.16:41702) 2023-05-22T15:27:45.672 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.663+0000 7f8a666d2640 1 -- 172.21.15.16:0/783293190 learned_addr learned my addr 172.21.15.16:0/783293190 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:45.672 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.663+0000 7f8a656d0640 1 --2- 172.21.15.16:0/783293190 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8a6015d680 0x7f8a600cb260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:45.672 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.663+0000 7f8a65ed1640 1 --2- 172.21.15.16:0/783293190 >> v2:172.21.15.62:3300/0 conn(0x7f8a60159970 0x7f8a60159d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:45.672 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.663+0000 7f8a666d2640 1 -- 172.21.15.16:0/783293190 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8a6015d680 msgr2=0x7f8a600cb260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:45.672 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.663+0000 7f8a666d2640 1 --2- 172.21.15.16:0/783293190 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8a6015d680 0x7f8a600cb260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:45.672 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.663+0000 7f8a666d2640 1 -- 172.21.15.16:0/783293190 >> v2:172.21.15.62:3300/0 conn(0x7f8a60159970 msgr2=0x7f8a60159d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:45.673 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.663+0000 7f8a666d2640 1 --2- 172.21.15.16:0/783293190 >> v2:172.21.15.62:3300/0 conn(0x7f8a60159970 0x7f8a60159d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:45.673 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.663+0000 7f8a666d2640 1 -- 172.21.15.16:0/783293190 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a60071420 con 0x7f8a6015dad0 2023-05-22T15:27:45.673 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.663+0000 7f8a656d0640 1 --2- 172.21.15.16:0/783293190 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8a6015d680 0x7f8a600cb260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:27:45.673 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.663+0000 7f8a65ed1640 1 --2- 172.21.15.16:0/783293190 >> v2:172.21.15.62:3300/0 conn(0x7f8a60159970 0x7f8a60159d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:27:45.677 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.663+0000 7f8a666d2640 1 --2- 172.21.15.16:0/783293190 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8a6015dad0 0x7f8a6006b6c0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f8a6006daf0 tx=0x7f8a5801c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50f33e18df1817dc server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:45.678 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.663+0000 7f8a64ecf640 1 -- 172.21.15.16:0/783293190 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a580a56a0 con 0x7f8a6015dad0 2023-05-22T15:27:45.678 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.663+0000 7f8a64ecf640 1 -- 172.21.15.16:0/783293190 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a580a5800 con 0x7f8a6015dad0 2023-05-22T15:27:45.679 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.663+0000 7f8a64ecf640 1 -- 172.21.15.16:0/783293190 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a580a5b90 con 0x7f8a6015dad0 2023-05-22T15:27:45.679 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.667+0000 7f8a68135640 1 -- 172.21.15.16:0/783293190 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8a6015dad0 msgr2=0x7f8a6006b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:45.679 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.667+0000 7f8a68135640 1 --2- 172.21.15.16:0/783293190 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8a6015dad0 0x7f8a6006b6c0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f8a6006daf0 tx=0x7f8a5801c910 comp rx=0 tx=0).stop 2023-05-22T15:27:45.680 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.667+0000 7f8a68135640 1 -- 172.21.15.16:0/783293190 shutdown_connections 2023-05-22T15:27:45.680 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.667+0000 7f8a68135640 1 --2- 172.21.15.16:0/783293190 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8a6015dad0 0x7f8a6006b6c0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:45.680 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.667+0000 7f8a68135640 1 --2- 172.21.15.16:0/783293190 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8a6015d680 0x7f8a600cb260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:45.680 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.667+0000 7f8a68135640 1 --2- 172.21.15.16:0/783293190 >> v2:172.21.15.62:3300/0 conn(0x7f8a60159970 0x7f8a60159d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:45.681 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.667+0000 7f8a68135640 1 -- 172.21.15.16:0/783293190 >> 172.21.15.16:0/783293190 conn(0x7f8a60069720 msgr2=0x7f8a60069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:45.681 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.667+0000 7f8a68135640 1 -- 172.21.15.16:0/783293190 shutdown_connections 2023-05-22T15:27:45.681 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.667+0000 7f8a68135640 1 -- 172.21.15.16:0/783293190 wait complete. 2023-05-22T15:27:45.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.667+0000 7f8a68135640 1 Processor -- start 2023-05-22T15:27:45.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.667+0000 7f8a68135640 1 -- start start 2023-05-22T15:27:45.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.667+0000 7f8a68135640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8a60159970 0x7f8a60153f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:45.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.667+0000 7f8a68135640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8a6015d680 0x7f8a6014f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:45.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.667+0000 7f8a68135640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8a6015dad0 0x7f8a6014f650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:45.683 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.667+0000 7f8a68135640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a60162cb0 con 0x7f8a6015dad0 2023-05-22T15:27:45.683 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.667+0000 7f8a68135640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a6013d950 con 0x7f8a60159970 2023-05-22T15:27:45.683 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.667+0000 7f8a68135640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a60162380 con 0x7f8a6015d680 2023-05-22T15:27:45.683 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.667+0000 7f8a666d2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8a6015dad0 0x7f8a6014f650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:45.684 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.667+0000 7f8a666d2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8a6015dad0 0x7f8a6014f650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41706/0 (socket says 172.21.15.16:41706) 2023-05-22T15:27:45.684 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.667+0000 7f8a666d2640 1 -- 172.21.15.16:0/3012226954 learned_addr learned my addr 172.21.15.16:0/3012226954 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:45.684 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.667+0000 7f8a656d0640 1 --2- 172.21.15.16:0/3012226954 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8a6015d680 0x7f8a6014f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:45.685 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.667+0000 7f8a65ed1640 1 --2- 172.21.15.16:0/3012226954 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8a60159970 0x7f8a60153f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:45.685 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.667+0000 7f8a666d2640 1 -- 172.21.15.16:0/3012226954 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8a6015d680 msgr2=0x7f8a6014f110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:45.685 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.667+0000 7f8a666d2640 1 --2- 172.21.15.16:0/3012226954 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8a6015d680 0x7f8a6014f110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:45.685 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.667+0000 7f8a666d2640 1 -- 172.21.15.16:0/3012226954 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8a60159970 msgr2=0x7f8a60153f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:45.686 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.667+0000 7f8a666d2640 1 --2- 172.21.15.16:0/3012226954 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8a60159970 0x7f8a60153f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:45.687 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.667+0000 7f8a666d2640 1 -- 172.21.15.16:0/3012226954 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a58009b30 con 0x7f8a6015dad0 2023-05-22T15:27:45.687 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.667+0000 7f8a666d2640 1 --2- 172.21.15.16:0/3012226954 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8a6015dad0 0x7f8a6014f650 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f8a580af040 tx=0x7f8a580196b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:45.687 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.667+0000 7f8a567fc640 1 -- 172.21.15.16:0/3012226954 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a580b8670 con 0x7f8a6015dad0 2023-05-22T15:27:45.687 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.667+0000 7f8a567fc640 1 -- 172.21.15.16:0/3012226954 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a580b87f0 con 0x7f8a6015dad0 2023-05-22T15:27:45.687 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.667+0000 7f8a68135640 1 -- 172.21.15.16:0/3012226954 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a600cae20 con 0x7f8a6015dad0 2023-05-22T15:27:45.687 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.667+0000 7f8a567fc640 1 -- 172.21.15.16:0/3012226954 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a580b8ba0 con 0x7f8a6015dad0 2023-05-22T15:27:45.687 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.667+0000 7f8a68135640 1 -- 172.21.15.16:0/3012226954 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a600cac90 con 0x7f8a6015dad0 2023-05-22T15:27:45.688 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.671+0000 7f8a68135640 1 -- 172.21.15.16:0/3012226954 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a6015adc0 con 0x7f8a6015dad0 2023-05-22T15:27:45.688 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.671+0000 7f8a567fc640 1 -- 172.21.15.16:0/3012226954 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7f8a580b8d00 con 0x7f8a6015dad0 2023-05-22T15:27:45.688 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.671+0000 7f8a567fc640 1 --2- 172.21.15.16:0/3012226954 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f8a44065380 0x7f8a44067830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:45.689 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.671+0000 7f8a567fc640 1 -- 172.21.15.16:0/3012226954 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f8a5811e1f0 con 0x7f8a6015dad0 2023-05-22T15:27:45.689 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.671+0000 7f8a65ed1640 1 --2- 172.21.15.16:0/3012226954 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f8a44065380 0x7f8a44067830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:45.689 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.671+0000 7f8a65ed1640 1 --2- 172.21.15.16:0/3012226954 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f8a44065380 0x7f8a44067830 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f8a5c00b820 tx=0x7f8a5c008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:45.689 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.675+0000 7f8a567fc640 1 -- 172.21.15.16:0/3012226954 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f8a580ebb50 con 0x7f8a6015dad0 2023-05-22T15:27:45.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.823+0000 7f8a68135640 1 -- 172.21.15.16:0/3012226954 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f8a601624c0 con 0x7f8a6015dad0 2023-05-22T15:27:45.834 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.823+0000 7f8a567fc640 1 -- 172.21.15.16:0/3012226954 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v29) v1 ==== 100+0+11 (secure 0 0 0) 0x7f8a601624c0 con 0x7f8a6015dad0 2023-05-22T15:27:45.834 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 64 2023-05-22T15:27:45.835 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.827+0000 7f8a68135640 1 -- 172.21.15.16:0/3012226954 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f8a44065380 msgr2=0x7f8a44067830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:45.835 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.827+0000 7f8a68135640 1 --2- 172.21.15.16:0/3012226954 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f8a44065380 0x7f8a44067830 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f8a5c00b820 tx=0x7f8a5c008040 comp rx=0 tx=0).stop 2023-05-22T15:27:45.835 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.827+0000 7f8a68135640 1 -- 172.21.15.16:0/3012226954 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8a6015dad0 msgr2=0x7f8a6014f650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:45.835 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.827+0000 7f8a68135640 1 --2- 172.21.15.16:0/3012226954 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8a6015dad0 0x7f8a6014f650 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f8a580af040 tx=0x7f8a580196b0 comp rx=0 tx=0).stop 2023-05-22T15:27:45.835 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.827+0000 7f8a68135640 1 -- 172.21.15.16:0/3012226954 shutdown_connections 2023-05-22T15:27:45.836 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.827+0000 7f8a68135640 1 --2- 172.21.15.16:0/3012226954 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8a6015dad0 0x7f8a6014f650 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:45.836 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.827+0000 7f8a68135640 1 --2- 172.21.15.16:0/3012226954 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f8a44065380 0x7f8a44067830 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:45.836 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.827+0000 7f8a68135640 1 --2- 172.21.15.16:0/3012226954 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8a6015d680 0x7f8a6014f110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:45.836 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.827+0000 7f8a68135640 1 --2- 172.21.15.16:0/3012226954 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8a60159970 0x7f8a60153f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:45.836 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.827+0000 7f8a68135640 1 -- 172.21.15.16:0/3012226954 >> 172.21.15.16:0/3012226954 conn(0x7f8a60069720 msgr2=0x7f8a60158be0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:45.836 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.827+0000 7f8a68135640 1 -- 172.21.15.16:0/3012226954 shutdown_connections 2023-05-22T15:27:45.836 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.827+0000 7f8a68135640 1 -- 172.21.15.16:0/3012226954 wait complete. 2023-05-22T15:27:45.851 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-05-22T15:27:45.964 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.955+0000 7f950773d640 1 Processor -- start 2023-05-22T15:27:45.965 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.955+0000 7f950773d640 1 -- start start 2023-05-22T15:27:45.966 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.959+0000 7f950773d640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9500159b10 0x7f95000b1f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:45.966 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.959+0000 7f950773d640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f950006c5c0 0x7f9500069720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:45.967 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.959+0000 7f950773d640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f950006d670 con 0x7f950006dc20 2023-05-22T15:27:45.967 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.959+0000 7f950773d640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f950006caf0 con 0x7f950006c5c0 2023-05-22T15:27:45.967 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.959+0000 7f950773d640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f950006cdd0 con 0x7f9500159b10 2023-05-22T15:27:45.967 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.959+0000 7f9505cda640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f950006dc20 0x7f9500070c70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:45646/0 (socket says 172.21.15.16:45646) 2023-05-22T15:27:45.967 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.959+0000 7f9505cda640 1 -- 172.21.15.16:0/3956080758 learned_addr learned my addr 172.21.15.16:0/3956080758 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:27:45.967 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.959+0000 7f9504cd8640 1 --2- 172.21.15.16:0/3956080758 >> v2:172.21.15.62:3300/0 conn(0x7f950006c5c0 0x7f9500069720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:45.967 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.959+0000 7f95054d9640 1 --2- 172.21.15.16:0/3956080758 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9500159b10 0x7f95000b1f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:45.968 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.959+0000 7f94f7fff640 1 -- 172.21.15.16:0/3956080758 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1278506552 0 0) 0x7f950006d670 con 0x7f950006dc20 2023-05-22T15:27:45.968 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.959+0000 7f94f7fff640 1 -- 172.21.15.16:0/3956080758 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f94ec003530 con 0x7f950006dc20 2023-05-22T15:27:45.968 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.959+0000 7f94f7fff640 1 -- 172.21.15.16:0/3956080758 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 733273413 0 0) 0x7f94ec003530 con 0x7f950006dc20 2023-05-22T15:27:45.968 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.959+0000 7f94f7fff640 1 -- 172.21.15.16:0/3956080758 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9500159b10 msgr2=0x7f95000b1f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:45.968 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.959+0000 7f94f7fff640 1 --2- 172.21.15.16:0/3956080758 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9500159b10 0x7f95000b1f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:45.968 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.959+0000 7f94f7fff640 1 -- 172.21.15.16:0/3956080758 >> v2:172.21.15.62:3300/0 conn(0x7f950006c5c0 msgr2=0x7f9500069720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:45.968 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.959+0000 7f94f7fff640 1 --2- 172.21.15.16:0/3956080758 >> v2:172.21.15.62:3300/0 conn(0x7f950006c5c0 0x7f9500069720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:45.969 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.959+0000 7f94f7fff640 1 -- 172.21.15.16:0/3956080758 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f950006d0b0 con 0x7f950006dc20 2023-05-22T15:27:45.969 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.959+0000 7f94f7fff640 1 -- 172.21.15.16:0/3956080758 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7f94f8001a00 con 0x7f950006dc20 2023-05-22T15:27:45.969 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.959+0000 7f94f7fff640 1 -- 172.21.15.16:0/3956080758 >> v1:172.21.15.16:6789/0 conn(0x7f950006dc20 legacy=0x7f9500070c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:45.970 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.959+0000 7f94f7fff640 1 --2- 172.21.15.16:0/3956080758 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f94ec003980 0x7f94ec003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:45.970 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.959+0000 7f94f7fff640 1 --2- 172.21.15.16:0/3956080758 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f94ec0043b0 0x7f94ec004870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:45.970 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.959+0000 7f94f7fff640 1 --2- 172.21.15.16:0/3956080758 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9500159b10 0x7f94ec004fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:45.970 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.959+0000 7f94f7fff640 1 -- 172.21.15.16:0/3956080758 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f950006caf0 con 0x7f94ec0043b0 2023-05-22T15:27:45.970 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.959+0000 7f94f7fff640 1 -- 172.21.15.16:0/3956080758 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f950006cdd0 con 0x7f94ec003980 2023-05-22T15:27:45.970 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.959+0000 7f94f7fff640 1 -- 172.21.15.16:0/3956080758 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94ec00b990 con 0x7f9500159b10 2023-05-22T15:27:45.971 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.959+0000 7f9504cd8640 1 --2- 172.21.15.16:0/3956080758 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f94ec0043b0 0x7f94ec004870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:45.971 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.959+0000 7f95054d9640 1 --2- 172.21.15.16:0/3956080758 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f94ec003980 0x7f94ec003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:45.971 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.959+0000 7f9505cda640 1 --2- 172.21.15.16:0/3956080758 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9500159b10 0x7f94ec004fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:45.972 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.959+0000 7f9504cd8640 1 -- 172.21.15.16:0/3956080758 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9500159b10 msgr2=0x7f94ec004fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:45.972 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.959+0000 7f9504cd8640 1 --2- 172.21.15.16:0/3956080758 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9500159b10 0x7f94ec004fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:45.972 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.959+0000 7f9504cd8640 1 -- 172.21.15.16:0/3956080758 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f94ec003980 msgr2=0x7f94ec003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:45.972 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.959+0000 7f9504cd8640 1 --2- 172.21.15.16:0/3956080758 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f94ec003980 0x7f94ec003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:45.972 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.959+0000 7f9504cd8640 1 -- 172.21.15.16:0/3956080758 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94ec003530 con 0x7f94ec0043b0 2023-05-22T15:27:45.972 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.959+0000 7f95054d9640 1 --2- 172.21.15.16:0/3956080758 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f94ec003980 0x7f94ec003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:27:45.978 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.959+0000 7f9504cd8640 1 --2- 172.21.15.16:0/3956080758 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f94ec0043b0 0x7f94ec004870 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f94f0009b10 tx=0x7f94f001ca60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd34bc28ad8bb81 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:45.979 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.959+0000 7f94f7fff640 1 -- 172.21.15.16:0/3956080758 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94f00a5540 con 0x7f94ec0043b0 2023-05-22T15:27:45.979 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.959+0000 7f94f7fff640 1 -- 172.21.15.16:0/3956080758 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94f00a56a0 con 0x7f94ec0043b0 2023-05-22T15:27:45.979 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.959+0000 7f94f7fff640 1 -- 172.21.15.16:0/3956080758 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94f00a5a50 con 0x7f94ec0043b0 2023-05-22T15:27:45.980 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.959+0000 7f950773d640 1 -- 172.21.15.16:0/3956080758 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f94ec0043b0 msgr2=0x7f94ec004870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:45.980 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.959+0000 7f950773d640 1 --2- 172.21.15.16:0/3956080758 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f94ec0043b0 0x7f94ec004870 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f94f0009b10 tx=0x7f94f001ca60 comp rx=0 tx=0).stop 2023-05-22T15:27:45.980 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.959+0000 7f9505cda640 1 -- 172.21.15.16:0/3956080758 reap_dead start 2023-05-22T15:27:45.981 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.959+0000 7f950773d640 1 -- 172.21.15.16:0/3956080758 shutdown_connections 2023-05-22T15:27:45.981 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.959+0000 7f950773d640 1 -- 172.21.15.16:0/3956080758 >> 172.21.15.16:0/3956080758 conn(0x7f95000b18e0 msgr2=0x7f95000cf120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:45.981 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.963+0000 7f950773d640 1 -- 172.21.15.16:0/3956080758 shutdown_connections 2023-05-22T15:27:45.981 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.963+0000 7f950773d640 1 -- 172.21.15.16:0/3956080758 wait complete. 2023-05-22T15:27:45.982 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.963+0000 7f950773d640 1 Processor -- start 2023-05-22T15:27:45.982 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.963+0000 7f950773d640 1 -- start start 2023-05-22T15:27:45.982 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.963+0000 7f950773d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f94ec0043b0 0x7f950015a390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:45.982 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.963+0000 7f950773d640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f950014e1e0 0x7f95000c27d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:45.983 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.963+0000 7f950773d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f95000c2d10 0x7f95000cde30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:45.983 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.963+0000 7f950773d640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95000ccd90 con 0x7f95000c2d10 2023-05-22T15:27:45.983 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.963+0000 7f950773d640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95000cd070 con 0x7f94ec0043b0 2023-05-22T15:27:45.983 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.963+0000 7f950773d640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95000cd350 con 0x7f950014e1e0 2023-05-22T15:27:45.984 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.963+0000 7f9505cda640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f95000c2d10 0x7f95000cde30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:45.984 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.963+0000 7f9505cda640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f95000c2d10 0x7f95000cde30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41718/0 (socket says 172.21.15.16:41718) 2023-05-22T15:27:45.984 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.963+0000 7f9505cda640 1 -- 172.21.15.16:0/2203089572 learned_addr learned my addr 172.21.15.16:0/2203089572 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:45.984 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.963+0000 7f9504cd8640 1 --2- 172.21.15.16:0/2203089572 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f950014e1e0 0x7f95000c27d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:45.985 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.963+0000 7f95054d9640 1 --2- 172.21.15.16:0/2203089572 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f94ec0043b0 0x7f950015a390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:45.985 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.963+0000 7f9505cda640 1 -- 172.21.15.16:0/2203089572 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f950014e1e0 msgr2=0x7f95000c27d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:45.985 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.963+0000 7f9505cda640 1 --2- 172.21.15.16:0/2203089572 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f950014e1e0 0x7f95000c27d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:45.985 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.963+0000 7f9505cda640 1 -- 172.21.15.16:0/2203089572 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f94ec0043b0 msgr2=0x7f950015a390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:45.986 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.963+0000 7f9505cda640 1 --2- 172.21.15.16:0/2203089572 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f94ec0043b0 0x7f950015a390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:45.986 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.963+0000 7f9505cda640 1 -- 172.21.15.16:0/2203089572 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95000cb830 con 0x7f95000c2d10 2023-05-22T15:27:45.986 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.963+0000 7f95054d9640 1 --2- 172.21.15.16:0/2203089572 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f94ec0043b0 0x7f950015a390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:27:45.987 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.963+0000 7f9504cd8640 1 --2- 172.21.15.16:0/2203089572 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f950014e1e0 0x7f95000c27d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:27:45.987 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.963+0000 7f9505cda640 1 --2- 172.21.15.16:0/2203089572 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f95000c2d10 0x7f95000cde30 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f950006ffc0 tx=0x7f94f800dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:45.988 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.963+0000 7f94f5ffb640 1 -- 172.21.15.16:0/2203089572 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94f8007070 con 0x7f95000c2d10 2023-05-22T15:27:45.988 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.963+0000 7f94f5ffb640 1 -- 172.21.15.16:0/2203089572 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94f80071f0 con 0x7f95000c2d10 2023-05-22T15:27:45.989 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.963+0000 7f950773d640 1 -- 172.21.15.16:0/2203089572 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f950006a070 con 0x7f95000c2d10 2023-05-22T15:27:45.989 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.963+0000 7f94f5ffb640 1 -- 172.21.15.16:0/2203089572 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94f80075a0 con 0x7f95000c2d10 2023-05-22T15:27:45.989 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.963+0000 7f950773d640 1 -- 172.21.15.16:0/2203089572 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95000cb9c0 con 0x7f95000c2d10 2023-05-22T15:27:45.989 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.963+0000 7f950773d640 1 -- 172.21.15.16:0/2203089572 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f950004dbe0 con 0x7f95000c2d10 2023-05-22T15:27:45.990 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.967+0000 7f94f5ffb640 1 -- 172.21.15.16:0/2203089572 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7f94f8007700 con 0x7f95000c2d10 2023-05-22T15:27:45.990 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.967+0000 7f94f5ffb640 1 --2- 172.21.15.16:0/2203089572 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f94e40653b0 0x7f94e4067860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:45.990 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.967+0000 7f94f5ffb640 1 -- 172.21.15.16:0/2203089572 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f94f8082220 con 0x7f95000c2d10 2023-05-22T15:27:45.990 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.967+0000 7f95054d9640 1 --2- 172.21.15.16:0/2203089572 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f94e40653b0 0x7f94e4067860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:45.991 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.967+0000 7f95054d9640 1 --2- 172.21.15.16:0/2203089572 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f94e40653b0 0x7f94e4067860 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f950015c9c0 tx=0x7f94fc008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:45.991 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:45.971+0000 7f94f5ffb640 1 -- 172.21.15.16:0/2203089572 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f94f804fb70 con 0x7f95000c2d10 2023-05-22T15:27:46.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.115+0000 7f950773d640 1 -- 172.21.15.16:0/2203089572 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f950014df20 con 0x7f95000c2d10 2023-05-22T15:27:46.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.115+0000 7f94f5ffb640 1 -- 172.21.15.16:0/2203089572 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v29) v1 ==== 103+0+11 (secure 0 0 0) 0x7f94f8053190 con 0x7f95000c2d10 2023-05-22T15:27:46.124 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2023-05-22T15:27:46.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.115+0000 7f950773d640 1 -- 172.21.15.16:0/2203089572 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f94e40653b0 msgr2=0x7f94e4067860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:46.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.115+0000 7f950773d640 1 --2- 172.21.15.16:0/2203089572 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f94e40653b0 0x7f94e4067860 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f950015c9c0 tx=0x7f94fc008040 comp rx=0 tx=0).stop 2023-05-22T15:27:46.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.115+0000 7f950773d640 1 -- 172.21.15.16:0/2203089572 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f95000c2d10 msgr2=0x7f95000cde30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:46.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.115+0000 7f950773d640 1 --2- 172.21.15.16:0/2203089572 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f95000c2d10 0x7f95000cde30 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f950006ffc0 tx=0x7f94f800dcf0 comp rx=0 tx=0).stop 2023-05-22T15:27:46.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.115+0000 7f950773d640 1 -- 172.21.15.16:0/2203089572 shutdown_connections 2023-05-22T15:27:46.126 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.115+0000 7f950773d640 1 --2- 172.21.15.16:0/2203089572 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f95000c2d10 0x7f95000cde30 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:46.126 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.115+0000 7f950773d640 1 --2- 172.21.15.16:0/2203089572 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f94e40653b0 0x7f94e4067860 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:46.126 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.115+0000 7f950773d640 1 --2- 172.21.15.16:0/2203089572 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f950014e1e0 0x7f95000c27d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:46.126 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.115+0000 7f950773d640 1 --2- 172.21.15.16:0/2203089572 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f94ec0043b0 0x7f950015a390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:46.126 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.115+0000 7f950773d640 1 -- 172.21.15.16:0/2203089572 >> 172.21.15.16:0/2203089572 conn(0x7f95000b18e0 msgr2=0x7f950006e150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:46.126 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.119+0000 7f950773d640 1 -- 172.21.15.16:0/2203089572 shutdown_connections 2023-05-22T15:27:46.126 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.119+0000 7f950773d640 1 -- 172.21.15.16:0/2203089572 wait complete. 2023-05-22T15:27:46.141 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-05-22T15:27:46.263 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.255+0000 7f0a04aa5640 1 Processor -- start 2023-05-22T15:27:46.264 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.255+0000 7f0a04aa5640 1 -- start start 2023-05-22T15:27:46.265 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.255+0000 7f0a04aa5640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f0a00158660 0x7f0a000cdfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:46.265 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.255+0000 7f0a04aa5640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a000c92b0 con 0x7f0a000cce20 2023-05-22T15:27:46.265 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.255+0000 7f0a04aa5640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a000cc060 con 0x7f0a00158290 2023-05-22T15:27:46.265 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.255+0000 7f0a04aa5640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a000cc340 con 0x7f0a00158660 2023-05-22T15:27:46.265 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.255+0000 7f09fed9d640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f0a000cce20 0x7f0a0006f630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:45650/0 (socket says 172.21.15.16:45650) 2023-05-22T15:27:46.266 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.255+0000 7f09fed9d640 1 -- 172.21.15.16:0/1937563551 learned_addr learned my addr 172.21.15.16:0/1937563551 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:27:46.266 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.259+0000 7f09fdd9b640 1 --2- 172.21.15.16:0/1937563551 >> v2:172.21.15.62:3300/0 conn(0x7f0a00158660 0x7f0a000cdfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:46.266 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.259+0000 7f09fd59a640 1 -- 172.21.15.16:0/1937563551 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1177503647 0 0) 0x7f0a000c92b0 con 0x7f0a000cce20 2023-05-22T15:27:46.266 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.259+0000 7f09fd59a640 1 -- 172.21.15.16:0/1937563551 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f09e8003530 con 0x7f0a000cce20 2023-05-22T15:27:46.267 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.259+0000 7f09fd59a640 1 -- 172.21.15.16:0/1937563551 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3536487509 0 0) 0x7f09e8003530 con 0x7f0a000cce20 2023-05-22T15:27:46.267 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.259+0000 7f09fd59a640 1 -- 172.21.15.16:0/1937563551 >> v2:172.21.15.62:3300/0 conn(0x7f0a00158660 msgr2=0x7f0a000cdfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:46.268 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.259+0000 7f09fd59a640 1 --2- 172.21.15.16:0/1937563551 >> v2:172.21.15.62:3300/0 conn(0x7f0a00158660 0x7f0a000cdfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:46.268 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.259+0000 7f09fd59a640 1 -- 172.21.15.16:0/1937563551 >> v1:172.21.15.62:6789/0 conn(0x7f0a00158290 legacy=0x7f0a000c8880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:46.268 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.259+0000 7f09fd59a640 1 -- 172.21.15.16:0/1937563551 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a000cc620 con 0x7f0a000cce20 2023-05-22T15:27:46.268 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.259+0000 7f09fd59a640 1 -- 172.21.15.16:0/1937563551 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7f09f8001a00 con 0x7f0a000cce20 2023-05-22T15:27:46.268 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.259+0000 7f09fd59a640 1 -- 172.21.15.16:0/1937563551 >> v1:172.21.15.16:6789/0 conn(0x7f0a000cce20 legacy=0x7f0a0006f630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:46.268 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.259+0000 7f09fd59a640 1 --2- 172.21.15.16:0/1937563551 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f09e8003980 0x7f09e8003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:46.268 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.259+0000 7f09fd59a640 1 --2- 172.21.15.16:0/1937563551 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f09e80047d0 0x7f09e8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:46.269 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.259+0000 7f09fd59a640 1 --2- 172.21.15.16:0/1937563551 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f09e8005180 0x7f09e800b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:46.269 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.259+0000 7f09fd59a640 1 -- 172.21.15.16:0/1937563551 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a000cc340 con 0x7f09e8005180 2023-05-22T15:27:46.269 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.259+0000 7f09fd59a640 1 -- 172.21.15.16:0/1937563551 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09e800c000 con 0x7f09e80047d0 2023-05-22T15:27:46.269 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.259+0000 7f09fd59a640 1 -- 172.21.15.16:0/1937563551 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09e800c1c0 con 0x7f09e8003980 2023-05-22T15:27:46.269 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.259+0000 7f09fed9d640 1 --2- 172.21.15.16:0/1937563551 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f09e8005180 0x7f09e800b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:46.269 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.259+0000 7f09fdd9b640 1 --2- 172.21.15.16:0/1937563551 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f09e80047d0 0x7f09e8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:46.269 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.259+0000 7f09fe59c640 1 --2- 172.21.15.16:0/1937563551 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f09e8003980 0x7f09e8003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:46.269 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.259+0000 7f09fed9d640 1 -- 172.21.15.16:0/1937563551 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f09e8003980 msgr2=0x7f09e8003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:46.270 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.259+0000 7f09fed9d640 1 --2- 172.21.15.16:0/1937563551 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f09e8003980 0x7f09e8003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:46.270 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.259+0000 7f09fed9d640 1 -- 172.21.15.16:0/1937563551 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f09e80047d0 msgr2=0x7f09e8004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:46.270 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.259+0000 7f09fed9d640 1 --2- 172.21.15.16:0/1937563551 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f09e80047d0 0x7f09e8004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:46.270 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.259+0000 7f09fed9d640 1 -- 172.21.15.16:0/1937563551 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09e8003530 con 0x7f09e8005180 2023-05-22T15:27:46.270 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.259+0000 7f09fdd9b640 1 --2- 172.21.15.16:0/1937563551 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f09e80047d0 0x7f09e8004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T15:27:46.272 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.259+0000 7f09fed9d640 1 -- 172.21.15.16:0/1937563551 reap_dead start 2023-05-22T15:27:46.272 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.259+0000 7f09fed9d640 1 --2- 172.21.15.16:0/1937563551 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f09e8005180 0x7f09e800b990 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f09f800a800 tx=0x7f09f801d300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57de478124d950aa server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:46.273 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.259+0000 7f09fd59a640 1 -- 172.21.15.16:0/1937563551 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09f80072d0 con 0x7f09e8005180 2023-05-22T15:27:46.273 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.259+0000 7f09fd59a640 1 -- 172.21.15.16:0/1937563551 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09f80a8070 con 0x7f09e8005180 2023-05-22T15:27:46.273 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.259+0000 7f09fd59a640 1 -- 172.21.15.16:0/1937563551 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09f8007670 con 0x7f09e8005180 2023-05-22T15:27:46.273 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.263+0000 7f0a04aa5640 1 -- 172.21.15.16:0/1937563551 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f09e8005180 msgr2=0x7f09e800b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:46.273 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.263+0000 7f0a04aa5640 1 --2- 172.21.15.16:0/1937563551 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f09e8005180 0x7f09e800b990 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f09f800a800 tx=0x7f09f801d300 comp rx=0 tx=0).stop 2023-05-22T15:27:46.273 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.263+0000 7f0a04aa5640 1 -- 172.21.15.16:0/1937563551 shutdown_connections 2023-05-22T15:27:46.273 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.263+0000 7f0a04aa5640 1 --2- 172.21.15.16:0/1937563551 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f09e8005180 0x7f09e800b990 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:46.274 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.263+0000 7f0a04aa5640 1 -- 172.21.15.16:0/1937563551 >> 172.21.15.16:0/1937563551 conn(0x7f0a00069720 msgr2=0x7f0a00069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:46.274 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.263+0000 7f0a04aa5640 1 -- 172.21.15.16:0/1937563551 shutdown_connections 2023-05-22T15:27:46.274 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.263+0000 7f0a04aa5640 1 -- 172.21.15.16:0/1937563551 wait complete. 2023-05-22T15:27:46.274 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.263+0000 7f0a04aa5640 1 Processor -- start 2023-05-22T15:27:46.274 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.263+0000 7f0a04aa5640 1 -- start start 2023-05-22T15:27:46.274 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.263+0000 7f0a04aa5640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f09e8005180 0x7f0a0014d430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:46.274 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.263+0000 7f0a04aa5640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0a000adf90 0x7f0a000ae3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:46.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.263+0000 7f0a04aa5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0a001f4b10 0x7f0a001f4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:46.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.263+0000 7f0a04aa5640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a000c6020 con 0x7f0a001f4b10 2023-05-22T15:27:46.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.263+0000 7f0a04aa5640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a000c2460 con 0x7f09e8005180 2023-05-22T15:27:46.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.263+0000 7f0a04aa5640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a000c1b50 con 0x7f0a000adf90 2023-05-22T15:27:46.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.263+0000 7f09fed9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0a001f4b10 0x7f0a001f4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:46.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.263+0000 7f09fed9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0a001f4b10 0x7f0a001f4f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41732/0 (socket says 172.21.15.16:41732) 2023-05-22T15:27:46.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.263+0000 7f09fed9d640 1 -- 172.21.15.16:0/3684001872 learned_addr learned my addr 172.21.15.16:0/3684001872 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:46.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.263+0000 7f09fe59c640 1 --2- 172.21.15.16:0/3684001872 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f09e8005180 0x7f0a0014d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:46.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.263+0000 7f09fdd9b640 1 --2- 172.21.15.16:0/3684001872 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0a000adf90 0x7f0a000ae3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:46.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.263+0000 7f09fed9d640 1 -- 172.21.15.16:0/3684001872 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0a000adf90 msgr2=0x7f0a000ae3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:46.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.263+0000 7f09fed9d640 1 --2- 172.21.15.16:0/3684001872 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0a000adf90 0x7f0a000ae3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:46.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.263+0000 7f09fed9d640 1 -- 172.21.15.16:0/3684001872 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f09e8005180 msgr2=0x7f0a0014d430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:46.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.263+0000 7f09fed9d640 1 --2- 172.21.15.16:0/3684001872 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f09e8005180 0x7f0a0014d430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:46.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.263+0000 7f09fed9d640 1 -- 172.21.15.16:0/3684001872 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a0015cd10 con 0x7f0a001f4b10 2023-05-22T15:27:46.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.263+0000 7f09fdd9b640 1 --2- 172.21.15.16:0/3684001872 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0a000adf90 0x7f0a000ae3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:27:46.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.263+0000 7f09fed9d640 1 --2- 172.21.15.16:0/3684001872 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0a001f4b10 0x7f0a001f4f80 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f09f8007040 tx=0x7f09e8005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:46.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.263+0000 7f09eeffd640 1 -- 172.21.15.16:0/3684001872 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09f80b8660 con 0x7f0a001f4b10 2023-05-22T15:27:46.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.263+0000 7f09eeffd640 1 -- 172.21.15.16:0/3684001872 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09f80b87e0 con 0x7f0a001f4b10 2023-05-22T15:27:46.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.263+0000 7f0a04aa5640 1 -- 172.21.15.16:0/3684001872 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a000cc900 con 0x7f0a001f4b10 2023-05-22T15:27:46.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.263+0000 7f09eeffd640 1 -- 172.21.15.16:0/3684001872 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09f80b8b90 con 0x7f0a001f4b10 2023-05-22T15:27:46.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.263+0000 7f0a04aa5640 1 -- 172.21.15.16:0/3684001872 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a0015cea0 con 0x7f0a001f4b10 2023-05-22T15:27:46.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.267+0000 7f0a04aa5640 1 -- 172.21.15.16:0/3684001872 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a000c25a0 con 0x7f0a001f4b10 2023-05-22T15:27:46.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.267+0000 7f09eeffd640 1 -- 172.21.15.16:0/3684001872 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7f09f80aa070 con 0x7f0a001f4b10 2023-05-22T15:27:46.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.267+0000 7f09eeffd640 1 --2- 172.21.15.16:0/3684001872 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f09e00653b0 0x7f09e0067860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:46.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.267+0000 7f09eeffd640 1 -- 172.21.15.16:0/3684001872 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f09f811cc50 con 0x7f0a001f4b10 2023-05-22T15:27:46.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.267+0000 7f09fe59c640 1 --2- 172.21.15.16:0/3684001872 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f09e00653b0 0x7f09e0067860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:46.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.267+0000 7f09fe59c640 1 --2- 172.21.15.16:0/3684001872 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f09e00653b0 0x7f09e0067860 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f09f0005490 tx=0x7f09f0009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:46.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.271+0000 7f09eeffd640 1 -- 172.21.15.16:0/3684001872 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f0a000c25a0 con 0x7f0a001f4b10 2023-05-22T15:27:46.445 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.435+0000 7f0a04aa5640 1 -- 172.21.15.16:0/3684001872 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f0a000c25a0 con 0x7f0a001f4b10 2023-05-22T15:27:46.446 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.439+0000 7f09eeffd640 1 -- 172.21.15.16:0/3684001872 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v29) v1 ==== 80+0+0 (secure 0 0 0) 0x7f0a000c25a0 con 0x7f0a001f4b10 2023-05-22T15:27:46.446 INFO:teuthology.orchestra.run.smithi016.stderr:listed 0 entries 2023-05-22T15:27:46.448 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.439+0000 7f0a04aa5640 1 -- 172.21.15.16:0/3684001872 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f09e00653b0 msgr2=0x7f09e0067860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:46.448 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.439+0000 7f0a04aa5640 1 --2- 172.21.15.16:0/3684001872 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f09e00653b0 0x7f09e0067860 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f09f0005490 tx=0x7f09f0009040 comp rx=0 tx=0).stop 2023-05-22T15:27:46.448 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.439+0000 7f0a04aa5640 1 -- 172.21.15.16:0/3684001872 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0a001f4b10 msgr2=0x7f0a001f4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:46.448 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.439+0000 7f0a04aa5640 1 --2- 172.21.15.16:0/3684001872 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0a001f4b10 0x7f0a001f4f80 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f09f8007040 tx=0x7f09e8005800 comp rx=0 tx=0).stop 2023-05-22T15:27:46.449 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.439+0000 7f0a04aa5640 1 -- 172.21.15.16:0/3684001872 shutdown_connections 2023-05-22T15:27:46.449 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.439+0000 7f0a04aa5640 1 --2- 172.21.15.16:0/3684001872 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0a001f4b10 0x7f0a001f4f80 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:46.449 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.439+0000 7f0a04aa5640 1 --2- 172.21.15.16:0/3684001872 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f09e00653b0 0x7f09e0067860 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:46.449 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.439+0000 7f0a04aa5640 1 --2- 172.21.15.16:0/3684001872 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0a000adf90 0x7f0a000ae3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:46.449 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.439+0000 7f0a04aa5640 1 --2- 172.21.15.16:0/3684001872 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f09e8005180 0x7f0a0014d430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:46.449 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.439+0000 7f0a04aa5640 1 -- 172.21.15.16:0/3684001872 >> 172.21.15.16:0/3684001872 conn(0x7f0a00069720 msgr2=0x7f0a000af420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:46.449 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.439+0000 7f0a04aa5640 1 -- 172.21.15.16:0/3684001872 shutdown_connections 2023-05-22T15:27:46.449 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.439+0000 7f0a04aa5640 1 -- 172.21.15.16:0/3684001872 wait complete. 2023-05-22T15:27:46.465 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2023-05-22T15:27:46.465 DEBUG:teuthology.orchestra.run.smithi062:> dd if=/proc/self/mounts of=/dev/stdout 2023-05-22T15:27:46.472 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-05-22T15:27:46.593 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.583+0000 7fb84466b640 1 Processor -- start 2023-05-22T15:27:46.593 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.583+0000 7fb84466b640 1 -- start start 2023-05-22T15:27:46.594 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.587+0000 7fb84466b640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fb83c0ca4e0 0x7fb83c15e460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:46.594 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.587+0000 7fb84466b640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb83c070df0 0x7fb83c071210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:46.594 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.587+0000 7fb84466b640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb83c0b0650 con 0x7fb83c0c9b40 2023-05-22T15:27:46.594 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.587+0000 7fb84466b640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb83c0b0370 con 0x7fb83c0ca4e0 2023-05-22T15:27:46.595 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.587+0000 7fb84466b640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb83c15d980 con 0x7fb83c070df0 2023-05-22T15:27:46.595 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.587+0000 7fb841c06640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fb83c0c9b40 0x7fb83c0afb40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:45660/0 (socket says 172.21.15.16:45660) 2023-05-22T15:27:46.595 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.587+0000 7fb841c06640 1 -- 172.21.15.16:0/1407861947 learned_addr learned my addr 172.21.15.16:0/1407861947 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:27:46.596 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.587+0000 7fb842407640 1 --2- 172.21.15.16:0/1407861947 >> v2:172.21.15.62:3300/0 conn(0x7fb83c0ca4e0 0x7fb83c15e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:46.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.587+0000 7fb842c08640 1 --2- 172.21.15.16:0/1407861947 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb83c070df0 0x7fb83c071210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:46.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.587+0000 7fb841405640 1 -- 172.21.15.16:0/1407861947 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2454959659 0 0) 0x7fb83c0b0650 con 0x7fb83c0c9b40 2023-05-22T15:27:46.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.587+0000 7fb841405640 1 -- 172.21.15.16:0/1407861947 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb828003530 con 0x7fb83c0c9b40 2023-05-22T15:27:46.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.587+0000 7fb841405640 1 -- 172.21.15.16:0/1407861947 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 275397236 0 0) 0x7fb828003530 con 0x7fb83c0c9b40 2023-05-22T15:27:46.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.587+0000 7fb841405640 1 -- 172.21.15.16:0/1407861947 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb83c070df0 msgr2=0x7fb83c071210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:46.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.587+0000 7fb841405640 1 --2- 172.21.15.16:0/1407861947 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb83c070df0 0x7fb83c071210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:46.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.587+0000 7fb841405640 1 -- 172.21.15.16:0/1407861947 >> v2:172.21.15.62:3300/0 conn(0x7fb83c0ca4e0 msgr2=0x7fb83c15e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:46.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.587+0000 7fb841405640 1 --2- 172.21.15.16:0/1407861947 >> v2:172.21.15.62:3300/0 conn(0x7fb83c0ca4e0 0x7fb83c15e460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:46.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.587+0000 7fb841405640 1 -- 172.21.15.16:0/1407861947 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb83c15dc60 con 0x7fb83c0c9b40 2023-05-22T15:27:46.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.587+0000 7fb841405640 1 -- 172.21.15.16:0/1407861947 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7fb82c0016a0 con 0x7fb83c0c9b40 2023-05-22T15:27:46.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.587+0000 7fb841405640 1 -- 172.21.15.16:0/1407861947 >> v1:172.21.15.16:6789/0 conn(0x7fb83c0c9b40 legacy=0x7fb83c0afb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:46.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.587+0000 7fb841405640 1 --2- 172.21.15.16:0/1407861947 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb83c070df0 0x7fb828003980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:46.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.587+0000 7fb841405640 1 --2- 172.21.15.16:0/1407861947 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb828003f00 0x7fb8280047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:46.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.587+0000 7fb841405640 1 --2- 172.21.15.16:0/1407861947 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb828004d10 0x7fb828005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:46.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.587+0000 7fb841405640 1 -- 172.21.15.16:0/1407861947 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb83c0b0370 con 0x7fb828003f00 2023-05-22T15:27:46.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.587+0000 7fb841405640 1 -- 172.21.15.16:0/1407861947 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb83c15d980 con 0x7fb828004d10 2023-05-22T15:27:46.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.587+0000 7fb841405640 1 -- 172.21.15.16:0/1407861947 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb82800b990 con 0x7fb83c070df0 2023-05-22T15:27:46.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.587+0000 7fb841c06640 1 --2- 172.21.15.16:0/1407861947 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb828003f00 0x7fb8280047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:46.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.587+0000 7fb842407640 1 --2- 172.21.15.16:0/1407861947 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb83c070df0 0x7fb828003980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:46.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.587+0000 7fb841c06640 1 -- 172.21.15.16:0/1407861947 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb83c070df0 msgr2=0x7fb828003980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:46.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.587+0000 7fb842c08640 1 --2- 172.21.15.16:0/1407861947 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb828004d10 0x7fb828005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:46.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.587+0000 7fb841c06640 1 --2- 172.21.15.16:0/1407861947 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb83c070df0 0x7fb828003980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:46.600 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.587+0000 7fb841c06640 1 -- 172.21.15.16:0/1407861947 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb828004d10 msgr2=0x7fb828005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:46.600 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.587+0000 7fb841c06640 1 --2- 172.21.15.16:0/1407861947 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb828004d10 0x7fb828005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:46.601 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.587+0000 7fb841c06640 1 -- 172.21.15.16:0/1407861947 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb828003530 con 0x7fb828003f00 2023-05-22T15:27:46.601 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.587+0000 7fb842407640 1 --2- 172.21.15.16:0/1407861947 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb83c070df0 0x7fb828003980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:27:46.601 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.587+0000 7fb842c08640 1 --2- 172.21.15.16:0/1407861947 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb828004d10 0x7fb828005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:27:46.603 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.591+0000 7fb841c06640 1 --2- 172.21.15.16:0/1407861947 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb828003f00 0x7fb8280047d0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fb82c00c210 tx=0x7fb82c01f390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=448f36249a1ba0f9 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:46.604 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.591+0000 7fb841405640 1 -- 172.21.15.16:0/1407861947 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb82c005ae0 con 0x7fb828003f00 2023-05-22T15:27:46.604 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.591+0000 7fb841405640 1 -- 172.21.15.16:0/1407861947 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb82c0ab070 con 0x7fb828003f00 2023-05-22T15:27:46.604 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.591+0000 7fb841405640 1 -- 172.21.15.16:0/1407861947 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb82c005e80 con 0x7fb828003f00 2023-05-22T15:27:46.604 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.591+0000 7fb84466b640 1 -- 172.21.15.16:0/1407861947 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb828003f00 msgr2=0x7fb8280047d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:46.604 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.591+0000 7fb84466b640 1 --2- 172.21.15.16:0/1407861947 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb828003f00 0x7fb8280047d0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fb82c00c210 tx=0x7fb82c01f390 comp rx=0 tx=0).stop 2023-05-22T15:27:46.604 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.591+0000 7fb842c08640 1 -- 172.21.15.16:0/1407861947 reap_dead start 2023-05-22T15:27:46.604 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.591+0000 7fb84466b640 1 -- 172.21.15.16:0/1407861947 shutdown_connections 2023-05-22T15:27:46.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.591+0000 7fb84466b640 1 -- 172.21.15.16:0/1407861947 >> 172.21.15.16:0/1407861947 conn(0x7fb83c069720 msgr2=0x7fb83c069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:46.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.591+0000 7fb84466b640 1 -- 172.21.15.16:0/1407861947 shutdown_connections 2023-05-22T15:27:46.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.591+0000 7fb84466b640 1 -- 172.21.15.16:0/1407861947 wait complete. 2023-05-22T15:27:46.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.591+0000 7fb84466b640 1 Processor -- start 2023-05-22T15:27:46.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.591+0000 7fb84466b640 1 -- start start 2023-05-22T15:27:46.606 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.591+0000 7fb84466b640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb828003f00 0x7fb83c1f0530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:46.606 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.591+0000 7fb84466b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb83c157300 0x7fb83c1ec430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:46.606 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.591+0000 7fb84466b640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb83c157770 0x7fb83c1ec970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:46.606 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.591+0000 7fb84466b640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb83c14f2e0 con 0x7fb83c157300 2023-05-22T15:27:46.606 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.591+0000 7fb84466b640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb83c14f5c0 con 0x7fb83c157770 2023-05-22T15:27:46.606 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.591+0000 7fb84466b640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb83c14f8a0 con 0x7fb828003f00 2023-05-22T15:27:46.606 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.591+0000 7fb841c06640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb83c157300 0x7fb83c1ec430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:46.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.591+0000 7fb841c06640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb83c157300 0x7fb83c1ec430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41760/0 (socket says 172.21.15.16:41760) 2023-05-22T15:27:46.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.591+0000 7fb841c06640 1 -- 172.21.15.16:0/1149826879 learned_addr learned my addr 172.21.15.16:0/1149826879 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:46.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.591+0000 7fb842c08640 1 --2- 172.21.15.16:0/1149826879 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb83c157770 0x7fb83c1ec970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:46.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.591+0000 7fb842407640 1 --2- 172.21.15.16:0/1149826879 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb828003f00 0x7fb83c1f0530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:46.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.591+0000 7fb841c06640 1 -- 172.21.15.16:0/1149826879 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb828003f00 msgr2=0x7fb83c1f0530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:46.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.591+0000 7fb841c06640 1 --2- 172.21.15.16:0/1149826879 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb828003f00 0x7fb83c1f0530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:46.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.591+0000 7fb841c06640 1 -- 172.21.15.16:0/1149826879 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb83c157770 msgr2=0x7fb83c1ec970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:46.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.591+0000 7fb841c06640 1 --2- 172.21.15.16:0/1149826879 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb83c157770 0x7fb83c1ec970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:46.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.591+0000 7fb841c06640 1 -- 172.21.15.16:0/1149826879 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb83c0ca8b0 con 0x7fb83c157300 2023-05-22T15:27:46.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.591+0000 7fb842c08640 1 --2- 172.21.15.16:0/1149826879 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb83c157770 0x7fb83c1ec970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:27:46.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.591+0000 7fb841c06640 1 --2- 172.21.15.16:0/1149826879 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb83c157300 0x7fb83c1ec430 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fb82c01f1c0 tx=0x7fb828005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:46.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.595+0000 7fb832ffd640 1 -- 172.21.15.16:0/1149826879 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb82c0b7070 con 0x7fb83c157300 2023-05-22T15:27:46.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.595+0000 7fb832ffd640 1 -- 172.21.15.16:0/1149826879 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb82c0047f0 con 0x7fb83c157300 2023-05-22T15:27:46.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.595+0000 7fb84466b640 1 -- 172.21.15.16:0/1149826879 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb83c15df40 con 0x7fb83c157300 2023-05-22T15:27:46.609 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.595+0000 7fb832ffd640 1 -- 172.21.15.16:0/1149826879 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb82c004b80 con 0x7fb83c157300 2023-05-22T15:27:46.609 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.595+0000 7fb84466b640 1 -- 172.21.15.16:0/1149826879 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb83c071210 con 0x7fb83c157300 2023-05-22T15:27:46.609 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.595+0000 7fb84466b640 1 -- 172.21.15.16:0/1149826879 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb83c04dbe0 con 0x7fb83c157300 2023-05-22T15:27:46.609 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.595+0000 7fb832ffd640 1 -- 172.21.15.16:0/1149826879 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7fb82c0ad020 con 0x7fb83c157300 2023-05-22T15:27:46.609 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.595+0000 7fb832ffd640 1 --2- 172.21.15.16:0/1149826879 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fb8200653b0 0x7fb820067860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:46.609 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.595+0000 7fb832ffd640 1 -- 172.21.15.16:0/1149826879 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6871+0+0 (secure 0 0 0) 0x7fb82c11f380 con 0x7fb83c157300 2023-05-22T15:27:46.609 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.595+0000 7fb842407640 1 --2- 172.21.15.16:0/1149826879 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fb8200653b0 0x7fb820067860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:46.609 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.595+0000 7fb842407640 1 --2- 172.21.15.16:0/1149826879 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fb8200653b0 0x7fb820067860 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fb83c157cb0 tx=0x7fb838008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:46.610 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.599+0000 7fb832ffd640 1 -- 172.21.15.16:0/1149826879 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fb82c0f0710 con 0x7fb83c157300 2023-05-22T15:27:46.755 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.747+0000 7fb84466b640 1 -- 172.21.15.16:0/1149826879 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb83c0cb010 con 0x7fb83c157300 2023-05-22T15:27:46.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.751+0000 7fb832ffd640 1 -- 172.21.15.16:0/1149826879 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v29) v1 ==== 74+0+19700 (secure 0 0 0) 0x7fb82c0f0300 con 0x7fb83c157300 2023-05-22T15:27:46.758 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:27:46.758 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":29,"fsid":"b2719834-7847-4d98-90a6-f65d57f85c49","created":"2023-05-22T15:26:25.259314+0000","modified":"2023-05-22T15:27:31.182317+0000","last_up_change":"2023-05-22T15:26:36.448524+0000","last_in_change":"2023-05-22T15:26:27.675668+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-05-22T15:26:38.019781+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-05-22T15:26:43.570514+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-05-22T15:26:47.879159+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-05-22T15:26:48.522130+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data_ec","create_time":"2023-05-22T15:26:51.616142+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"9e7ff854-b25f-4cbb-a5c2-1799d9c53d2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":37021},{"type":"v1","addr":"172.21.15.16:6825","nonce":37021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":37021},{"type":"v1","addr":"172.21.15.16:6827","nonce":37021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":37021},{"type":"v1","addr":"172.21.15.16:6831","nonce":37021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":37021},{"type":"v1","addr":"172.21.15.16:6829","nonce":37021}]},"public_addr":"172.21.15.16:6825/37021","cluster_addr":"172.21.15.16:6827/37021","heartbeat_back_addr":"172.21.15.16:6831/37021","heartbeat_front_addr":"172.21.15.16:6829/37021","state":["exists","up"]},{"osd":1,"uuid":"4702fbc9-a927-4565-aca5-41dc69cc8264","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":37008},{"type":"v1","addr":"172.21.15.16:6801","nonce":37008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":37008},{"type":"v1","addr":"172.21.15.16:6803","nonce":37008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":37008},{"type":"v1","addr":"172.21.15.16:6807","nonce":37008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":37008},{"type":"v1","addr":"172.21.15.16:6805","nonce":37008}]},"public_addr":"172.21.15.16:6801/37008","cluster_addr":"172.21.15.16:6803/37008","heartbeat_back_addr":"172.21.15.16:6807/37008","heartbeat_front_addr":"172.21.15.16:6805/37008","state":["exists","up"]},{"osd":2,"uuid":"1e7bba46-4eba-47ef-bb81-5e2d371638ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":37009},{"type":"v1","addr":"172.21.15.16:6809","nonce":37009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":37009},{"type":"v1","addr":"172.21.15.16:6811","nonce":37009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":37009},{"type":"v1","addr":"172.21.15.16:6815","nonce":37009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":37009},{"type":"v1","addr":"172.21.15.16:6813","nonce":37009}]},"public_addr":"172.21.15.16:6809/37009","cluster_addr":"172.21.15.16:6811/37009","heartbeat_back_addr":"172.21.15.16:6815/37009","heartbeat_front_addr":"172.21.15.16:6813/37009","state":["exists","up"]},{"osd":3,"uuid":"383cacc0-5a88-4904-990e-498ca59ef48a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":37020},{"type":"v1","addr":"172.21.15.16:6817","nonce":37020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":37020},{"type":"v1","addr":"172.21.15.16:6819","nonce":37020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":37020},{"type":"v1","addr":"172.21.15.16:6823","nonce":37020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":37020},{"type":"v1","addr":"172.21.15.16:6821","nonce":37020}]},"public_addr":"172.21.15.16:6817/37020","cluster_addr":"172.21.15.16:6819/37020","heartbeat_back_addr":"172.21.15.16:6823/37020","heartbeat_front_addr":"172.21.15.16:6821/37020","state":["exists","up"]},{"osd":4,"uuid":"970500ef-95b7-42c1-b080-eed4fae7013d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":37353},{"type":"v1","addr":"172.21.15.62:6817","nonce":37353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6819","nonce":37353},{"type":"v1","addr":"172.21.15.62:6822","nonce":37353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37353},{"type":"v1","addr":"172.21.15.62:6831","nonce":37353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37353},{"type":"v1","addr":"172.21.15.62:6826","nonce":37353}]},"public_addr":"172.21.15.62:6817/37353","cluster_addr":"172.21.15.62:6822/37353","heartbeat_back_addr":"172.21.15.62:6831/37353","heartbeat_front_addr":"172.21.15.62:6826/37353","state":["exists","up"]},{"osd":5,"uuid":"022d67db-d4d8-445b-b1b9-8b446a00f5c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":37352},{"type":"v1","addr":"172.21.15.62:6801","nonce":37352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":37352},{"type":"v1","addr":"172.21.15.62:6803","nonce":37352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":37352},{"type":"v1","addr":"172.21.15.62:6807","nonce":37352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":37352},{"type":"v1","addr":"172.21.15.62:6805","nonce":37352}]},"public_addr":"172.21.15.62:6801/37352","cluster_addr":"172.21.15.62:6803/37352","heartbeat_back_addr":"172.21.15.62:6807/37352","heartbeat_front_addr":"172.21.15.62:6805/37352","state":["exists","up"]},{"osd":6,"uuid":"42bd372b-6688-4a1b-8407-e16e8c266e33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":37354},{"type":"v1","addr":"172.21.15.62:6820","nonce":37354}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":37354},{"type":"v1","addr":"172.21.15.62:6823","nonce":37354}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6829","nonce":37354},{"type":"v1","addr":"172.21.15.62:6830","nonce":37354}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6825","nonce":37354},{"type":"v1","addr":"172.21.15.62:6827","nonce":37354}]},"public_addr":"172.21.15.62:6820/37354","cluster_addr":"172.21.15.62:6823/37354","heartbeat_back_addr":"172.21.15.62:6830/37354","heartbeat_front_addr":"172.21.15.62:6827/37354","state":["exists","up"]},{"osd":7,"uuid":"ac70b2a3-f41b-4bf2-b227-7f2cbb55426d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37355},{"type":"v1","addr":"172.21.15.62:6809","nonce":37355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37355},{"type":"v1","addr":"172.21.15.62:6811","nonce":37355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":37355},{"type":"v1","addr":"172.21.15.62:6815","nonce":37355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":37355},{"type":"v1","addr":"172.21.15.62:6813","nonce":37355}]},"public_addr":"172.21.15.62:6809/37355","cluster_addr":"172.21.15.62:6811/37355","heartbeat_back_addr":"172.21.15.62:6815/37355","heartbeat_front_addr":"172.21.15.62:6813/37355","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.368896+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.107273+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.276888+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.287502+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.124791+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.251937+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.255679+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.232797+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"4.23","mappings":[{"from":3,"to":2}]},{"pgid":"4.31","mappings":[{"from":1,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-05-22T15:27:46.760 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.751+0000 7fb84466b640 1 -- 172.21.15.16:0/1149826879 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fb8200653b0 msgr2=0x7fb820067860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:46.761 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.751+0000 7fb84466b640 1 --2- 172.21.15.16:0/1149826879 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fb8200653b0 0x7fb820067860 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fb83c157cb0 tx=0x7fb838008040 comp rx=0 tx=0).stop 2023-05-22T15:27:46.761 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.751+0000 7fb84466b640 1 -- 172.21.15.16:0/1149826879 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb83c157300 msgr2=0x7fb83c1ec430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:46.761 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.751+0000 7fb84466b640 1 --2- 172.21.15.16:0/1149826879 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb83c157300 0x7fb83c1ec430 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fb82c01f1c0 tx=0x7fb828005800 comp rx=0 tx=0).stop 2023-05-22T15:27:46.762 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.751+0000 7fb84466b640 1 -- 172.21.15.16:0/1149826879 shutdown_connections 2023-05-22T15:27:46.762 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.751+0000 7fb84466b640 1 --2- 172.21.15.16:0/1149826879 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb83c157770 0x7fb83c1ec970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:46.762 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.751+0000 7fb84466b640 1 --2- 172.21.15.16:0/1149826879 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb83c157300 0x7fb83c1ec430 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:46.763 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.751+0000 7fb84466b640 1 --2- 172.21.15.16:0/1149826879 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fb8200653b0 0x7fb820067860 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:46.763 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.751+0000 7fb84466b640 1 --2- 172.21.15.16:0/1149826879 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb828003f00 0x7fb83c1f0530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:46.763 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.751+0000 7fb84466b640 1 -- 172.21.15.16:0/1149826879 >> 172.21.15.16:0/1149826879 conn(0x7fb83c069720 msgr2=0x7fb83c0b30e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:46.763 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.751+0000 7fb84466b640 1 -- 172.21.15.16:0/1149826879 shutdown_connections 2023-05-22T15:27:46.763 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.751+0000 7fb84466b640 1 -- 172.21.15.16:0/1149826879 wait complete. 2023-05-22T15:27:46.776 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-05-22T15:26:38.019781+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-05-22T15:26:43.570514+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2023-05-22T15:26:47.879159+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2023-05-22T15:26:48.522130+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-05-22T15:26:51.616142+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '27', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-05-22T15:27:46.777 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-05-22T15:27:46.890 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.883+0000 7fbe3ca27640 1 Processor -- start 2023-05-22T15:27:46.890 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.883+0000 7fbe3ca27640 1 -- start start 2023-05-22T15:27:46.891 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.883+0000 7fbe3ca27640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbe380705d0 0x7fbe380cda20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:46.891 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.883+0000 7fbe3ca27640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe380ce6c0 con 0x7fbe3815bdf0 2023-05-22T15:27:46.891 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.883+0000 7fbe3ca27640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe380ce100 con 0x7fbe3806ac00 2023-05-22T15:27:46.891 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.883+0000 7fbe3ca27640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe3815c1c0 con 0x7fbe380705d0 2023-05-22T15:27:46.892 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.883+0000 7fbe3659c640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fbe3815bdf0 0x7fbe3806c080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:45672/0 (socket says 172.21.15.16:45672) 2023-05-22T15:27:46.892 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.883+0000 7fbe3659c640 1 -- 172.21.15.16:0/4053342197 learned_addr learned my addr 172.21.15.16:0/4053342197 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:27:46.895 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.883+0000 7fbe36d9d640 1 --2- 172.21.15.16:0/4053342197 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbe380705d0 0x7fbe380cda20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:46.895 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.883+0000 7fbe3559a640 1 -- 172.21.15.16:0/4053342197 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 994928981 0 0) 0x7fbe380ce6c0 con 0x7fbe3815bdf0 2023-05-22T15:27:46.895 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.883+0000 7fbe3559a640 1 -- 172.21.15.16:0/4053342197 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe20003530 con 0x7fbe3815bdf0 2023-05-22T15:27:46.895 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.883+0000 7fbe3559a640 1 -- 172.21.15.16:0/4053342197 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 765900796 0 0) 0x7fbe20003530 con 0x7fbe3815bdf0 2023-05-22T15:27:46.896 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.883+0000 7fbe3559a640 1 -- 172.21.15.16:0/4053342197 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbe380705d0 msgr2=0x7fbe380cda20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:46.896 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.883+0000 7fbe3559a640 1 --2- 172.21.15.16:0/4053342197 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbe380705d0 0x7fbe380cda20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:46.896 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.883+0000 7fbe3559a640 1 -- 172.21.15.16:0/4053342197 >> v1:172.21.15.62:6789/0 conn(0x7fbe3806ac00 legacy=0x7fbe3806fec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:46.896 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.883+0000 7fbe3559a640 1 -- 172.21.15.16:0/4053342197 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe380ce3e0 con 0x7fbe3815bdf0 2023-05-22T15:27:46.896 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.883+0000 7fbe3559a640 1 -- 172.21.15.16:0/4053342197 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7fbe28001680 con 0x7fbe3815bdf0 2023-05-22T15:27:46.896 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.883+0000 7fbe3559a640 1 -- 172.21.15.16:0/4053342197 >> v1:172.21.15.16:6789/0 conn(0x7fbe3815bdf0 legacy=0x7fbe3806c080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:46.896 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.883+0000 7fbe3559a640 1 --2- 172.21.15.16:0/4053342197 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbe380705d0 0x7fbe200039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:46.897 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.883+0000 7fbe3559a640 1 --2- 172.21.15.16:0/4053342197 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbe20003f30 0x7fbe200047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:46.897 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.883+0000 7fbe3559a640 1 --2- 172.21.15.16:0/4053342197 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbe20004d10 0x7fbe20005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:46.897 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.883+0000 7fbe3559a640 1 -- 172.21.15.16:0/4053342197 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe2000bb70 con 0x7fbe20004d10 2023-05-22T15:27:46.897 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.883+0000 7fbe3559a640 1 -- 172.21.15.16:0/4053342197 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe2000bce0 con 0x7fbe20003f30 2023-05-22T15:27:46.897 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.883+0000 7fbe3559a640 1 -- 172.21.15.16:0/4053342197 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe2000bea0 con 0x7fbe380705d0 2023-05-22T15:27:46.897 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.883+0000 7fbe36d9d640 1 --2- 172.21.15.16:0/4053342197 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbe20004d10 0x7fbe20005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:46.897 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.883+0000 7fbe3659c640 1 --2- 172.21.15.16:0/4053342197 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbe380705d0 0x7fbe200039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:46.897 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.883+0000 7fbe35d9b640 1 --2- 172.21.15.16:0/4053342197 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbe20003f30 0x7fbe200047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:46.898 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.883+0000 7fbe36d9d640 1 -- 172.21.15.16:0/4053342197 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbe380705d0 msgr2=0x7fbe200039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:46.898 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.883+0000 7fbe36d9d640 1 --2- 172.21.15.16:0/4053342197 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbe380705d0 0x7fbe200039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:46.898 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.883+0000 7fbe36d9d640 1 -- 172.21.15.16:0/4053342197 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbe20003f30 msgr2=0x7fbe200047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:46.898 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.883+0000 7fbe36d9d640 1 --2- 172.21.15.16:0/4053342197 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbe20003f30 0x7fbe200047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:46.898 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.883+0000 7fbe36d9d640 1 -- 172.21.15.16:0/4053342197 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe20003530 con 0x7fbe20004d10 2023-05-22T15:27:46.898 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.883+0000 7fbe35d9b640 1 --2- 172.21.15.16:0/4053342197 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbe20003f30 0x7fbe200047d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:27:46.898 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.883+0000 7fbe3659c640 1 --2- 172.21.15.16:0/4053342197 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbe380705d0 0x7fbe200039b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:27:46.898 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.887+0000 7fbe36d9d640 1 --2- 172.21.15.16:0/4053342197 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbe20004d10 0x7fbe20005180 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fbe381608d0 tx=0x7fbe3001cb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=938c31f26417ea7d server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:46.899 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.887+0000 7fbe3559a640 1 -- 172.21.15.16:0/4053342197 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe300a5b80 con 0x7fbe20004d10 2023-05-22T15:27:46.899 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.887+0000 7fbe3559a640 1 -- 172.21.15.16:0/4053342197 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe3000f040 con 0x7fbe20004d10 2023-05-22T15:27:46.899 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.887+0000 7fbe3559a640 1 -- 172.21.15.16:0/4053342197 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe300ad440 con 0x7fbe20004d10 2023-05-22T15:27:46.899 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.887+0000 7fbe3ca27640 1 -- 172.21.15.16:0/4053342197 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbe20004d10 msgr2=0x7fbe20005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:46.900 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.887+0000 7fbe3ca27640 1 --2- 172.21.15.16:0/4053342197 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbe20004d10 0x7fbe20005180 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fbe381608d0 tx=0x7fbe3001cb90 comp rx=0 tx=0).stop 2023-05-22T15:27:46.900 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.887+0000 7fbe36d9d640 1 -- 172.21.15.16:0/4053342197 reap_dead start 2023-05-22T15:27:46.900 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.887+0000 7fbe3ca27640 1 -- 172.21.15.16:0/4053342197 shutdown_connections 2023-05-22T15:27:46.900 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.887+0000 7fbe3ca27640 1 -- 172.21.15.16:0/4053342197 >> 172.21.15.16:0/4053342197 conn(0x7fbe380ce890 msgr2=0x7fbe380cec90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:46.900 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.887+0000 7fbe3ca27640 1 -- 172.21.15.16:0/4053342197 shutdown_connections 2023-05-22T15:27:46.900 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.887+0000 7fbe3ca27640 1 -- 172.21.15.16:0/4053342197 wait complete. 2023-05-22T15:27:46.902 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.887+0000 7fbe3ca27640 1 Processor -- start 2023-05-22T15:27:46.902 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.887+0000 7fbe3ca27640 1 -- start start 2023-05-22T15:27:46.903 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.891+0000 7fbe3ca27640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbe20004d10 0x7fbe380af390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:46.903 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.891+0000 7fbe3ca27640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbe380b0050 0x7fbe380b2d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:46.903 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.891+0000 7fbe3ca27640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbe380c71d0 0x7fbe380c7640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:46.903 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.891+0000 7fbe3ca27640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe380b07f0 con 0x7fbe380b0050 2023-05-22T15:27:46.903 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.891+0000 7fbe3ca27640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe380c6410 con 0x7fbe380c71d0 2023-05-22T15:27:46.904 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.891+0000 7fbe3ca27640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe380c66f0 con 0x7fbe20004d10 2023-05-22T15:27:46.904 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.891+0000 7fbe35d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbe380b0050 0x7fbe380b2d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:46.904 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.891+0000 7fbe35d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbe380b0050 0x7fbe380b2d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41776/0 (socket says 172.21.15.16:41776) 2023-05-22T15:27:46.904 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.891+0000 7fbe35d9b640 1 -- 172.21.15.16:0/1473316094 learned_addr learned my addr 172.21.15.16:0/1473316094 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:46.904 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.891+0000 7fbe3659c640 1 --2- 172.21.15.16:0/1473316094 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbe20004d10 0x7fbe380af390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:46.904 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.891+0000 7fbe36d9d640 1 --2- 172.21.15.16:0/1473316094 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbe380c71d0 0x7fbe380c7640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:46.904 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.891+0000 7fbe35d9b640 1 -- 172.21.15.16:0/1473316094 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbe20004d10 msgr2=0x7fbe380af390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:46.905 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.891+0000 7fbe35d9b640 1 --2- 172.21.15.16:0/1473316094 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbe20004d10 0x7fbe380af390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:46.905 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.891+0000 7fbe35d9b640 1 -- 172.21.15.16:0/1473316094 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbe380c71d0 msgr2=0x7fbe380c7640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:46.905 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.891+0000 7fbe35d9b640 1 --2- 172.21.15.16:0/1473316094 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbe380c71d0 0x7fbe380c7640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:46.905 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.891+0000 7fbe35d9b640 1 -- 172.21.15.16:0/1473316094 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe380c8be0 con 0x7fbe380b0050 2023-05-22T15:27:46.905 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.891+0000 7fbe36d9d640 1 --2- 172.21.15.16:0/1473316094 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbe380c71d0 0x7fbe380c7640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:27:46.905 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.891+0000 7fbe35d9b640 1 --2- 172.21.15.16:0/1473316094 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbe380b0050 0x7fbe380b2d90 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fbe2c00d770 tx=0x7fbe2c00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:46.905 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.891+0000 7fbe26ffd640 1 -- 172.21.15.16:0/1473316094 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe2c005270 con 0x7fbe380b0050 2023-05-22T15:27:46.905 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.891+0000 7fbe26ffd640 1 -- 172.21.15.16:0/1473316094 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe2c0053d0 con 0x7fbe380b0050 2023-05-22T15:27:46.906 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.891+0000 7fbe3ca27640 1 -- 172.21.15.16:0/1473316094 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe3806bb60 con 0x7fbe380b0050 2023-05-22T15:27:46.906 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.891+0000 7fbe26ffd640 1 -- 172.21.15.16:0/1473316094 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe2c00b8d0 con 0x7fbe380b0050 2023-05-22T15:27:46.906 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.891+0000 7fbe3ca27640 1 -- 172.21.15.16:0/1473316094 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe38069f30 con 0x7fbe380b0050 2023-05-22T15:27:46.906 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.891+0000 7fbe3ca27640 1 -- 172.21.15.16:0/1473316094 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe380cd0f0 con 0x7fbe380b0050 2023-05-22T15:27:46.906 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.891+0000 7fbe26ffd640 1 -- 172.21.15.16:0/1473316094 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7fbe2c00ba30 con 0x7fbe380b0050 2023-05-22T15:27:46.906 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.891+0000 7fbe26ffd640 1 --2- 172.21.15.16:0/1473316094 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fbe18065380 0x7fbe18067830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:46.906 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.891+0000 7fbe26ffd640 1 -- 172.21.15.16:0/1473316094 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6871+0+0 (secure 0 0 0) 0x7fbe2c082be0 con 0x7fbe380b0050 2023-05-22T15:27:46.907 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.891+0000 7fbe3659c640 1 --2- 172.21.15.16:0/1473316094 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fbe18065380 0x7fbe18067830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:46.907 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.891+0000 7fbe3659c640 1 --2- 172.21.15.16:0/1473316094 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fbe18065380 0x7fbe18067830 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fbe28000f70 tx=0x7fbe28009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:46.907 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:46.895+0000 7fbe26ffd640 1 -- 172.21.15.16:0/1473316094 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fbe2c0504d0 con 0x7fbe380b0050 2023-05-22T15:27:47.056 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.047+0000 7fbe3ca27640 1 -- 172.21.15.16:0/1473316094 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fbe380ac950 con 0x7fbe380b0050 2023-05-22T15:27:47.057 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.051+0000 7fbe26ffd640 1 -- 172.21.15.16:0/1473316094 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v29) v1 ==== 93+0+10 (secure 0 0 0) 0x7fbe2c053af0 con 0x7fbe380b0050 2023-05-22T15:27:47.058 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2023-05-22T15:27:47.059 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.051+0000 7fbe3ca27640 1 -- 172.21.15.16:0/1473316094 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fbe18065380 msgr2=0x7fbe18067830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:47.059 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.051+0000 7fbe3ca27640 1 --2- 172.21.15.16:0/1473316094 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fbe18065380 0x7fbe18067830 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fbe28000f70 tx=0x7fbe28009040 comp rx=0 tx=0).stop 2023-05-22T15:27:47.059 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.051+0000 7fbe3ca27640 1 -- 172.21.15.16:0/1473316094 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbe380b0050 msgr2=0x7fbe380b2d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:47.060 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.051+0000 7fbe3ca27640 1 --2- 172.21.15.16:0/1473316094 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbe380b0050 0x7fbe380b2d90 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fbe2c00d770 tx=0x7fbe2c00dc30 comp rx=0 tx=0).stop 2023-05-22T15:27:47.060 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.051+0000 7fbe3ca27640 1 -- 172.21.15.16:0/1473316094 shutdown_connections 2023-05-22T15:27:47.060 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.051+0000 7fbe3ca27640 1 --2- 172.21.15.16:0/1473316094 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbe380c71d0 0x7fbe380c7640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:47.060 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.051+0000 7fbe3ca27640 1 --2- 172.21.15.16:0/1473316094 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbe380b0050 0x7fbe380b2d90 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:47.060 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.051+0000 7fbe3ca27640 1 --2- 172.21.15.16:0/1473316094 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fbe18065380 0x7fbe18067830 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:47.060 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.051+0000 7fbe3ca27640 1 --2- 172.21.15.16:0/1473316094 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbe20004d10 0x7fbe380af390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:47.061 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.051+0000 7fbe3ca27640 1 -- 172.21.15.16:0/1473316094 >> 172.21.15.16:0/1473316094 conn(0x7fbe380ce890 msgr2=0x7fbe3815afa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:47.061 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.051+0000 7fbe3ca27640 1 -- 172.21.15.16:0/1473316094 shutdown_connections 2023-05-22T15:27:47.061 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.051+0000 7fbe3ca27640 1 -- 172.21.15.16:0/1473316094 wait complete. 2023-05-22T15:27:47.075 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-05-22T15:27:47.198 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.191+0000 7fe9dd5ed640 1 Processor -- start 2023-05-22T15:27:47.198 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.191+0000 7fe9dd5ed640 1 -- start start 2023-05-22T15:27:47.199 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.191+0000 7fe9dd5ed640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fe9d8158a10 0x7fe9d806e480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:47.199 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.191+0000 7fe9dd5ed640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe9d806cdf0 0x7fe9d8070be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:47.199 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.191+0000 7fe9dd5ed640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9d8158de0 con 0x7fe9d8070760 2023-05-22T15:27:47.200 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.191+0000 7fe9dd5ed640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9d806e060 con 0x7fe9d806cdf0 2023-05-22T15:27:47.200 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.191+0000 7fe9dd5ed640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9d80711e0 con 0x7fe9d8158a10 2023-05-22T15:27:47.200 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.191+0000 7fe9d77fe640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe9d806cdf0 0x7fe9d8070be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:47.200 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.191+0000 7fe9d6ffd640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fe9d8158a10 0x7fe9d806e480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:47.200 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.191+0000 7fe9d77fe640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe9d806cdf0 0x7fe9d8070be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41790/0 (socket says 172.21.15.16:41790) 2023-05-22T15:27:47.200 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.191+0000 7fe9d77fe640 1 -- 172.21.15.16:0/3606960753 learned_addr learned my addr 172.21.15.16:0/3606960753 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:47.201 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.191+0000 7fe9d77fe640 1 -- 172.21.15.16:0/3606960753 >> v2:172.21.15.62:3300/0 conn(0x7fe9d8158a10 msgr2=0x7fe9d806e480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:47.201 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.191+0000 7fe9d77fe640 1 --2- 172.21.15.16:0/3606960753 >> v2:172.21.15.62:3300/0 conn(0x7fe9d8158a10 0x7fe9d806e480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:47.201 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.191+0000 7fe9d77fe640 1 -- 172.21.15.16:0/3606960753 >> v1:172.21.15.62:6789/0 conn(0x7fe9d8070760 legacy=0x7fe9d806c6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:47.201 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.191+0000 7fe9d77fe640 1 -- 172.21.15.16:0/3606960753 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9d80714c0 con 0x7fe9d806cdf0 2023-05-22T15:27:47.202 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.195+0000 7fe9d77fe640 1 --2- 172.21.15.16:0/3606960753 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe9d806cdf0 0x7fe9d8070be0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fe9d80b1d50 tx=0x7fe9d001c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=244015d345b1f079 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:47.203 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.195+0000 7fe9d5ffb640 1 -- 172.21.15.16:0/3606960753 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9d00a5820 con 0x7fe9d806cdf0 2023-05-22T15:27:47.203 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.195+0000 7fe9d5ffb640 1 -- 172.21.15.16:0/3606960753 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9d00a5980 con 0x7fe9d806cdf0 2023-05-22T15:27:47.203 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.195+0000 7fe9d5ffb640 1 -- 172.21.15.16:0/3606960753 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9d00a5d10 con 0x7fe9d806cdf0 2023-05-22T15:27:47.203 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.195+0000 7fe9dd5ed640 1 -- 172.21.15.16:0/3606960753 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe9d806cdf0 msgr2=0x7fe9d8070be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:47.203 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.195+0000 7fe9dd5ed640 1 --2- 172.21.15.16:0/3606960753 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe9d806cdf0 0x7fe9d8070be0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fe9d80b1d50 tx=0x7fe9d001c940 comp rx=0 tx=0).stop 2023-05-22T15:27:47.203 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.195+0000 7fe9dd5ed640 1 -- 172.21.15.16:0/3606960753 shutdown_connections 2023-05-22T15:27:47.203 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.195+0000 7fe9dd5ed640 1 --2- 172.21.15.16:0/3606960753 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe9d806cdf0 0x7fe9d8070be0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:47.204 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.195+0000 7fe9dd5ed640 1 --2- 172.21.15.16:0/3606960753 >> v2:172.21.15.62:3300/0 conn(0x7fe9d8158a10 0x7fe9d806e480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:47.204 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.195+0000 7fe9dd5ed640 1 -- 172.21.15.16:0/3606960753 >> 172.21.15.16:0/3606960753 conn(0x7fe9d80cc870 msgr2=0x7fe9d80cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:47.204 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.195+0000 7fe9dd5ed640 1 -- 172.21.15.16:0/3606960753 shutdown_connections 2023-05-22T15:27:47.204 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.195+0000 7fe9dd5ed640 1 -- 172.21.15.16:0/3606960753 wait complete. 2023-05-22T15:27:47.206 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.195+0000 7fe9dd5ed640 1 Processor -- start 2023-05-22T15:27:47.206 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.195+0000 7fe9dd5ed640 1 -- start start 2023-05-22T15:27:47.206 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.195+0000 7fe9dd5ed640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe9d806cdf0 0x7fe9d8161560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:47.206 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.195+0000 7fe9dd5ed640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe9d8070760 0x7fe9d8157b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:47.206 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.195+0000 7fe9dd5ed640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe9d8158a10 0x7fe9d80c5530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:47.207 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.195+0000 7fe9dd5ed640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9d81580b0 con 0x7fe9d8158a10 2023-05-22T15:27:47.207 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.195+0000 7fe9dd5ed640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9d8162870 con 0x7fe9d806cdf0 2023-05-22T15:27:47.207 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.195+0000 7fe9dd5ed640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9d8162d00 con 0x7fe9d8070760 2023-05-22T15:27:47.208 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.195+0000 7fe9d77fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe9d8158a10 0x7fe9d80c5530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:47.208 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.195+0000 7fe9d77fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe9d8158a10 0x7fe9d80c5530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41800/0 (socket says 172.21.15.16:41800) 2023-05-22T15:27:47.208 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.195+0000 7fe9d77fe640 1 -- 172.21.15.16:0/3968852140 learned_addr learned my addr 172.21.15.16:0/3968852140 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:47.208 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.195+0000 7fe9d6ffd640 1 --2- 172.21.15.16:0/3968852140 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe9d806cdf0 0x7fe9d8161560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:47.208 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.195+0000 7fe9d67fc640 1 --2- 172.21.15.16:0/3968852140 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe9d8070760 0x7fe9d8157b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:47.208 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.199+0000 7fe9d77fe640 1 -- 172.21.15.16:0/3968852140 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe9d8070760 msgr2=0x7fe9d8157b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:47.208 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.199+0000 7fe9d77fe640 1 --2- 172.21.15.16:0/3968852140 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe9d8070760 0x7fe9d8157b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:47.209 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.199+0000 7fe9d77fe640 1 -- 172.21.15.16:0/3968852140 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe9d806cdf0 msgr2=0x7fe9d8161560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:47.209 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.199+0000 7fe9d77fe640 1 --2- 172.21.15.16:0/3968852140 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe9d806cdf0 0x7fe9d8161560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:47.209 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.199+0000 7fe9d77fe640 1 -- 172.21.15.16:0/3968852140 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9d0009b30 con 0x7fe9d8158a10 2023-05-22T15:27:47.209 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.199+0000 7fe9d6ffd640 1 --2- 172.21.15.16:0/3968852140 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe9d806cdf0 0x7fe9d8161560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T15:27:47.209 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.199+0000 7fe9d77fe640 1 --2- 172.21.15.16:0/3968852140 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe9d8158a10 0x7fe9d80c5530 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fe9d001a570 tx=0x7fe9d00196e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:47.209 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.199+0000 7fe9cf7fe640 1 -- 172.21.15.16:0/3968852140 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9d00b8670 con 0x7fe9d8158a10 2023-05-22T15:27:47.209 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.199+0000 7fe9cf7fe640 1 -- 172.21.15.16:0/3968852140 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9d00b87d0 con 0x7fe9d8158a10 2023-05-22T15:27:47.210 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.199+0000 7fe9dd5ed640 1 -- 172.21.15.16:0/3968852140 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9d806eb50 con 0x7fe9d8158a10 2023-05-22T15:27:47.210 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.199+0000 7fe9cf7fe640 1 -- 172.21.15.16:0/3968852140 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9d00b8b60 con 0x7fe9d8158a10 2023-05-22T15:27:47.210 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.199+0000 7fe9dd5ed640 1 -- 172.21.15.16:0/3968852140 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9d806e9c0 con 0x7fe9d8158a10 2023-05-22T15:27:47.211 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.199+0000 7fe9dd5ed640 1 -- 172.21.15.16:0/3968852140 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9d80ca280 con 0x7fe9d8158a10 2023-05-22T15:27:47.211 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.199+0000 7fe9cf7fe640 1 -- 172.21.15.16:0/3968852140 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7fe9d00b8cc0 con 0x7fe9d8158a10 2023-05-22T15:27:47.211 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.199+0000 7fe9cf7fe640 1 --2- 172.21.15.16:0/3968852140 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fe9b8065380 0x7fe9b8067830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:47.211 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.199+0000 7fe9d6ffd640 1 --2- 172.21.15.16:0/3968852140 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fe9b8065380 0x7fe9b8067830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:47.211 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.199+0000 7fe9cf7fe640 1 -- 172.21.15.16:0/3968852140 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6871+0+0 (secure 0 0 0) 0x7fe9d011e170 con 0x7fe9d8158a10 2023-05-22T15:27:47.212 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.199+0000 7fe9d6ffd640 1 --2- 172.21.15.16:0/3968852140 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fe9b8065380 0x7fe9b8067830 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fe9c0009b10 tx=0x7fe9c0008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:47.214 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.203+0000 7fe9cf7fe640 1 -- 172.21.15.16:0/3968852140 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fe9d00ebb80 con 0x7fe9d8158a10 2023-05-22T15:27:47.362 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.355+0000 7fe9dd5ed640 1 -- 172.21.15.16:0/3968852140 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fe9d804dbe0 con 0x7fe9d8158a10 2023-05-22T15:27:47.363 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.355+0000 7fe9cf7fe640 1 -- 172.21.15.16:0/3968852140 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v29) v1 ==== 92+0+10 (secure 0 0 0) 0x7fe9d00ef1a0 con 0x7fe9d8158a10 2023-05-22T15:27:47.363 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 8 2023-05-22T15:27:47.365 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.355+0000 7fe9dd5ed640 1 -- 172.21.15.16:0/3968852140 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fe9b8065380 msgr2=0x7fe9b8067830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:47.365 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.355+0000 7fe9dd5ed640 1 --2- 172.21.15.16:0/3968852140 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fe9b8065380 0x7fe9b8067830 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fe9c0009b10 tx=0x7fe9c0008040 comp rx=0 tx=0).stop 2023-05-22T15:27:47.365 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.355+0000 7fe9dd5ed640 1 -- 172.21.15.16:0/3968852140 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe9d8158a10 msgr2=0x7fe9d80c5530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:47.365 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.355+0000 7fe9dd5ed640 1 --2- 172.21.15.16:0/3968852140 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe9d8158a10 0x7fe9d80c5530 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fe9d001a570 tx=0x7fe9d00196e0 comp rx=0 tx=0).stop 2023-05-22T15:27:47.365 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.355+0000 7fe9dd5ed640 1 -- 172.21.15.16:0/3968852140 shutdown_connections 2023-05-22T15:27:47.365 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.355+0000 7fe9dd5ed640 1 --2- 172.21.15.16:0/3968852140 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe9d8158a10 0x7fe9d80c5530 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:47.366 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.355+0000 7fe9dd5ed640 1 --2- 172.21.15.16:0/3968852140 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fe9b8065380 0x7fe9b8067830 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:47.366 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.355+0000 7fe9dd5ed640 1 --2- 172.21.15.16:0/3968852140 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe9d8070760 0x7fe9d8157b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:47.366 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.355+0000 7fe9dd5ed640 1 --2- 172.21.15.16:0/3968852140 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe9d806cdf0 0x7fe9d8161560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:47.366 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.355+0000 7fe9dd5ed640 1 -- 172.21.15.16:0/3968852140 >> 172.21.15.16:0/3968852140 conn(0x7fe9d80cc870 msgr2=0x7fe9d806c6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:47.366 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.355+0000 7fe9dd5ed640 1 -- 172.21.15.16:0/3968852140 shutdown_connections 2023-05-22T15:27:47.366 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.359+0000 7fe9dd5ed640 1 -- 172.21.15.16:0/3968852140 wait complete. 2023-05-22T15:27:47.381 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-05-22T15:27:47.503 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.495+0000 7f6d5b856640 1 Processor -- start 2023-05-22T15:27:47.504 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.495+0000 7f6d5b856640 1 -- start start 2023-05-22T15:27:47.504 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.495+0000 7f6d5b856640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6d5415bee0 0x7f6d5406b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:47.505 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.495+0000 7f6d5b856640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6d540cb730 0x7f6d5406bc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:47.505 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.495+0000 7f6d5b856640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f6d540cbbe0 0x7f6d5415f850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:47.505 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.495+0000 7f6d5b856640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d540c8660 con 0x7f6d5415bee0 2023-05-22T15:27:47.505 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.495+0000 7f6d5b856640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d540ce4a0 con 0x7f6d540cbbe0 2023-05-22T15:27:47.505 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.495+0000 7f6d5b856640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d540ce780 con 0x7f6d540cb730 2023-05-22T15:27:47.505 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.495+0000 7f6d595f2640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6d5415bee0 0x7f6d5406b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:47.505 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.495+0000 7f6d595f2640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6d5415bee0 0x7f6d5406b6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41810/0 (socket says 172.21.15.16:41810) 2023-05-22T15:27:47.506 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.495+0000 7f6d595f2640 1 -- 172.21.15.16:0/1503046772 learned_addr learned my addr 172.21.15.16:0/1503046772 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:47.506 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.499+0000 7f6d59df3640 1 --2- 172.21.15.16:0/1503046772 >> v2:172.21.15.62:3300/0 conn(0x7f6d540cbbe0 0x7f6d5415f850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:47.507 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.499+0000 7f6d58df1640 1 --2- 172.21.15.16:0/1503046772 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6d540cb730 0x7f6d5406bc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:47.507 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.499+0000 7f6d595f2640 1 -- 172.21.15.16:0/1503046772 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6d540cb730 msgr2=0x7f6d5406bc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:47.507 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.499+0000 7f6d595f2640 1 --2- 172.21.15.16:0/1503046772 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6d540cb730 0x7f6d5406bc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:47.507 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.499+0000 7f6d595f2640 1 -- 172.21.15.16:0/1503046772 >> v2:172.21.15.62:3300/0 conn(0x7f6d540cbbe0 msgr2=0x7f6d5415f850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:47.507 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.499+0000 7f6d595f2640 1 --2- 172.21.15.16:0/1503046772 >> v2:172.21.15.62:3300/0 conn(0x7f6d540cbbe0 0x7f6d5415f850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:47.507 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.499+0000 7f6d595f2640 1 -- 172.21.15.16:0/1503046772 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d540cea60 con 0x7f6d5415bee0 2023-05-22T15:27:47.507 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.499+0000 7f6d59df3640 1 --2- 172.21.15.16:0/1503046772 >> v2:172.21.15.62:3300/0 conn(0x7f6d540cbbe0 0x7f6d5415f850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T15:27:47.508 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.499+0000 7f6d595f2640 1 --2- 172.21.15.16:0/1503046772 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6d5415bee0 0x7f6d5406b6c0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f6d5406c3a0 tx=0x7f6d5001c210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=491a4256118a8071 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:47.509 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.499+0000 7f6d4bfff640 1 -- 172.21.15.16:0/1503046772 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d500ab600 con 0x7f6d5415bee0 2023-05-22T15:27:47.509 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.499+0000 7f6d4bfff640 1 -- 172.21.15.16:0/1503046772 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d500a7050 con 0x7f6d5415bee0 2023-05-22T15:27:47.509 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.499+0000 7f6d4bfff640 1 -- 172.21.15.16:0/1503046772 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d500ab980 con 0x7f6d5415bee0 2023-05-22T15:27:47.509 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.499+0000 7f6d5b856640 1 -- 172.21.15.16:0/1503046772 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6d5415bee0 msgr2=0x7f6d5406b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:47.509 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.499+0000 7f6d5b856640 1 --2- 172.21.15.16:0/1503046772 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6d5415bee0 0x7f6d5406b6c0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f6d5406c3a0 tx=0x7f6d5001c210 comp rx=0 tx=0).stop 2023-05-22T15:27:47.510 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.499+0000 7f6d5b856640 1 -- 172.21.15.16:0/1503046772 shutdown_connections 2023-05-22T15:27:47.510 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.499+0000 7f6d5b856640 1 --2- 172.21.15.16:0/1503046772 >> v2:172.21.15.62:3300/0 conn(0x7f6d540cbbe0 0x7f6d5415f850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:47.510 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.499+0000 7f6d5b856640 1 --2- 172.21.15.16:0/1503046772 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6d540cb730 0x7f6d5406bc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:47.510 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.499+0000 7f6d5b856640 1 --2- 172.21.15.16:0/1503046772 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6d5415bee0 0x7f6d5406b6c0 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:47.510 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.499+0000 7f6d5b856640 1 -- 172.21.15.16:0/1503046772 >> 172.21.15.16:0/1503046772 conn(0x7f6d54069720 msgr2=0x7f6d54069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:47.510 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.499+0000 7f6d5b856640 1 -- 172.21.15.16:0/1503046772 shutdown_connections 2023-05-22T15:27:47.510 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.499+0000 7f6d5b856640 1 -- 172.21.15.16:0/1503046772 wait complete. 2023-05-22T15:27:47.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.503+0000 7f6d5b856640 1 Processor -- start 2023-05-22T15:27:47.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.503+0000 7f6d5b856640 1 -- start start 2023-05-22T15:27:47.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.503+0000 7f6d5b856640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6d540cb730 0x7f6d540aca80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:47.516 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.503+0000 7f6d5b856640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6d540cbbe0 0x7f6d54150540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:47.516 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.503+0000 7f6d5b856640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6d5415bee0 0x7f6d54150a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:47.516 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.503+0000 7f6d5b856640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d5414fd40 con 0x7f6d540cbbe0 2023-05-22T15:27:47.516 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.503+0000 7f6d5b856640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d54150fc0 con 0x7f6d5415bee0 2023-05-22T15:27:47.516 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.503+0000 7f6d5b856640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d5414f130 con 0x7f6d540cb730 2023-05-22T15:27:47.516 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.503+0000 7f6d58df1640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6d540cbbe0 0x7f6d54150540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:47.517 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.503+0000 7f6d58df1640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6d540cbbe0 0x7f6d54150540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41816/0 (socket says 172.21.15.16:41816) 2023-05-22T15:27:47.517 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.503+0000 7f6d58df1640 1 -- 172.21.15.16:0/3598451001 learned_addr learned my addr 172.21.15.16:0/3598451001 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:47.517 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.503+0000 7f6d59df3640 1 --2- 172.21.15.16:0/3598451001 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6d5415bee0 0x7f6d54150a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:47.517 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.503+0000 7f6d595f2640 1 --2- 172.21.15.16:0/3598451001 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6d540cb730 0x7f6d540aca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:47.518 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.503+0000 7f6d58df1640 1 -- 172.21.15.16:0/3598451001 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6d540cb730 msgr2=0x7f6d540aca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:47.518 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.503+0000 7f6d58df1640 1 --2- 172.21.15.16:0/3598451001 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6d540cb730 0x7f6d540aca80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:47.518 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.503+0000 7f6d58df1640 1 -- 172.21.15.16:0/3598451001 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6d5415bee0 msgr2=0x7f6d54150a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:47.518 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.503+0000 7f6d58df1640 1 --2- 172.21.15.16:0/3598451001 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6d5415bee0 0x7f6d54150a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:47.518 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.503+0000 7f6d58df1640 1 -- 172.21.15.16:0/3598451001 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d500094d0 con 0x7f6d540cbbe0 2023-05-22T15:27:47.519 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.503+0000 7f6d595f2640 1 --2- 172.21.15.16:0/3598451001 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6d540cb730 0x7f6d540aca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:27:47.519 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.503+0000 7f6d58df1640 1 --2- 172.21.15.16:0/3598451001 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6d540cbbe0 0x7f6d54150540 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f6d4400d800 tx=0x7f6d4400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:47.519 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.503+0000 7f6d49ffb640 1 -- 172.21.15.16:0/3598451001 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d440096a0 con 0x7f6d540cbbe0 2023-05-22T15:27:47.519 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.503+0000 7f6d49ffb640 1 -- 172.21.15.16:0/3598451001 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d44009800 con 0x7f6d540cbbe0 2023-05-22T15:27:47.519 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.503+0000 7f6d5b856640 1 -- 172.21.15.16:0/3598451001 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d540c8c80 con 0x7f6d540cbbe0 2023-05-22T15:27:47.519 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.503+0000 7f6d49ffb640 1 -- 172.21.15.16:0/3598451001 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d44009b70 con 0x7f6d540cbbe0 2023-05-22T15:27:47.519 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.503+0000 7f6d5b856640 1 -- 172.21.15.16:0/3598451001 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d540c8af0 con 0x7f6d540cbbe0 2023-05-22T15:27:47.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.503+0000 7f6d5b856640 1 -- 172.21.15.16:0/3598451001 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d540c8e10 con 0x7f6d540cbbe0 2023-05-22T15:27:47.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.503+0000 7f6d49ffb640 1 -- 172.21.15.16:0/3598451001 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7f6d44009cd0 con 0x7f6d540cbbe0 2023-05-22T15:27:47.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.507+0000 7f6d49ffb640 1 --2- 172.21.15.16:0/3598451001 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f6d38065380 0x7f6d38067830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:47.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.507+0000 7f6d49ffb640 1 -- 172.21.15.16:0/3598451001 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f6d440833d0 con 0x7f6d540cbbe0 2023-05-22T15:27:47.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.507+0000 7f6d595f2640 1 --2- 172.21.15.16:0/3598451001 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f6d38065380 0x7f6d38067830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:47.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.507+0000 7f6d595f2640 1 --2- 172.21.15.16:0/3598451001 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f6d38065380 0x7f6d38067830 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f6d500ad000 tx=0x7f6d50008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:47.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.511+0000 7f6d49ffb640 1 -- 172.21.15.16:0/3598451001 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f6d44050c20 con 0x7f6d540cbbe0 2023-05-22T15:27:47.669 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.659+0000 7f6d5b856640 1 -- 172.21.15.16:0/3598451001 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f6d5404dbe0 con 0x7f6d540cbbe0 2023-05-22T15:27:47.669 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.663+0000 7f6d49ffb640 1 -- 172.21.15.16:0/3598451001 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v29) v1 ==== 104+0+11 (secure 0 0 0) 0x7f6d44054240 con 0x7f6d540cbbe0 2023-05-22T15:27:47.670 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2023-05-22T15:27:47.672 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.663+0000 7f6d5b856640 1 -- 172.21.15.16:0/3598451001 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f6d38065380 msgr2=0x7f6d38067830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:47.672 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.663+0000 7f6d5b856640 1 --2- 172.21.15.16:0/3598451001 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f6d38065380 0x7f6d38067830 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f6d500ad000 tx=0x7f6d50008040 comp rx=0 tx=0).stop 2023-05-22T15:27:47.673 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.663+0000 7f6d5b856640 1 -- 172.21.15.16:0/3598451001 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6d540cbbe0 msgr2=0x7f6d54150540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:47.673 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.663+0000 7f6d5b856640 1 --2- 172.21.15.16:0/3598451001 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6d540cbbe0 0x7f6d54150540 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f6d4400d800 tx=0x7f6d4400dcc0 comp rx=0 tx=0).stop 2023-05-22T15:27:47.673 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.663+0000 7f6d5b856640 1 -- 172.21.15.16:0/3598451001 shutdown_connections 2023-05-22T15:27:47.673 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.663+0000 7f6d5b856640 1 --2- 172.21.15.16:0/3598451001 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6d5415bee0 0x7f6d54150a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:47.673 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.663+0000 7f6d5b856640 1 --2- 172.21.15.16:0/3598451001 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6d540cbbe0 0x7f6d54150540 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:47.673 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.663+0000 7f6d5b856640 1 --2- 172.21.15.16:0/3598451001 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f6d38065380 0x7f6d38067830 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:47.673 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.663+0000 7f6d5b856640 1 --2- 172.21.15.16:0/3598451001 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6d540cb730 0x7f6d540aca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:47.674 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.663+0000 7f6d5b856640 1 -- 172.21.15.16:0/3598451001 >> 172.21.15.16:0/3598451001 conn(0x7f6d54069720 msgr2=0x7f6d54154a60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:47.674 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.663+0000 7f6d5b856640 1 -- 172.21.15.16:0/3598451001 shutdown_connections 2023-05-22T15:27:47.674 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.663+0000 7f6d5b856640 1 -- 172.21.15.16:0/3598451001 wait complete. 2023-05-22T15:27:47.689 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-05-22T15:27:47.813 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.803+0000 7f6d17ca3640 1 Processor -- start 2023-05-22T15:27:47.813 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.803+0000 7f6d17ca3640 1 -- start start 2023-05-22T15:27:47.814 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.807+0000 7f6d17ca3640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6d100c8d70 0x7f6d1015c120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:47.814 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.807+0000 7f6d17ca3640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d10158cd0 con 0x7f6d10158210 2023-05-22T15:27:47.814 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.807+0000 7f6d17ca3640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d1015c790 con 0x7f6d1015bcd0 2023-05-22T15:27:47.814 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.807+0000 7f6d17ca3640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d1015ca70 con 0x7f6d100c8d70 2023-05-22T15:27:47.814 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.807+0000 7f6d15a3f640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f6d10158210 0x7f6d10159160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:45678/0 (socket says 172.21.15.16:45678) 2023-05-22T15:27:47.815 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.807+0000 7f6d15a3f640 1 -- 172.21.15.16:0/4214351096 learned_addr learned my addr 172.21.15.16:0/4214351096 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:27:47.815 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.807+0000 7f6d16240640 1 --2- 172.21.15.16:0/4214351096 >> v2:172.21.15.16:3300/0 conn(0x7f6d100c8d70 0x7f6d1015c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:47.815 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.807+0000 7f6d16240640 1 -- 172.21.15.16:0/4214351096 >> v1:172.21.15.62:6789/0 conn(0x7f6d1015bcd0 legacy=0x7f6d100c8660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:47.815 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.807+0000 7f6d16240640 1 -- 172.21.15.16:0/4214351096 >> v1:172.21.15.16:6789/0 conn(0x7f6d10158210 legacy=0x7f6d10159160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:47.816 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.807+0000 7f6d16240640 1 -- 172.21.15.16:0/4214351096 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d10158960 con 0x7f6d100c8d70 2023-05-22T15:27:47.816 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.807+0000 7f6d14a3d640 1 -- 172.21.15.16:0/4214351096 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1856744805 0 0) 0x7f6d10158cd0 con 0x7f6d10158210 2023-05-22T15:27:47.818 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.811+0000 7f6d16240640 1 --2- 172.21.15.16:0/4214351096 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6d100c8d70 0x7f6d1015c120 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f6d08009d90 tx=0x7f6d0801c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5fe848f3167b19dd server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:47.819 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.811+0000 7f6d14a3d640 1 -- 172.21.15.16:0/4214351096 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d080a59a0 con 0x7f6d100c8d70 2023-05-22T15:27:47.819 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.811+0000 7f6d14a3d640 1 -- 172.21.15.16:0/4214351096 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d080a5b00 con 0x7f6d100c8d70 2023-05-22T15:27:47.819 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.811+0000 7f6d14a3d640 1 -- 172.21.15.16:0/4214351096 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d080a5e90 con 0x7f6d100c8d70 2023-05-22T15:27:47.820 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.811+0000 7f6d17ca3640 1 -- 172.21.15.16:0/4214351096 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6d100c8d70 msgr2=0x7f6d1015c120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:47.820 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.811+0000 7f6d17ca3640 1 --2- 172.21.15.16:0/4214351096 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6d100c8d70 0x7f6d1015c120 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f6d08009d90 tx=0x7f6d0801c8c0 comp rx=0 tx=0).stop 2023-05-22T15:27:47.820 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.811+0000 7f6d17ca3640 1 -- 172.21.15.16:0/4214351096 shutdown_connections 2023-05-22T15:27:47.820 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.811+0000 7f6d17ca3640 1 --2- 172.21.15.16:0/4214351096 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6d100c8d70 0x7f6d1015c120 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:47.821 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.811+0000 7f6d17ca3640 1 -- 172.21.15.16:0/4214351096 >> 172.21.15.16:0/4214351096 conn(0x7f6d10069720 msgr2=0x7f6d10069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:47.821 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.811+0000 7f6d17ca3640 1 -- 172.21.15.16:0/4214351096 shutdown_connections 2023-05-22T15:27:47.821 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.811+0000 7f6d17ca3640 1 -- 172.21.15.16:0/4214351096 wait complete. 2023-05-22T15:27:47.823 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.815+0000 7f6d17ca3640 1 Processor -- start 2023-05-22T15:27:47.823 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.815+0000 7f6d17ca3640 1 -- start start 2023-05-22T15:27:47.823 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.815+0000 7f6d17ca3640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6d100c8d70 0x7f6d10147850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:47.823 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.815+0000 7f6d17ca3640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6d10158210 0x7f6d10151360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:47.823 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.815+0000 7f6d17ca3640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6d1015bcd0 0x7f6d101518a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:47.824 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.815+0000 7f6d17ca3640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d10147d90 con 0x7f6d100c8d70 2023-05-22T15:27:47.824 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.815+0000 7f6d17ca3640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d100b3d90 con 0x7f6d1015bcd0 2023-05-22T15:27:47.824 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.815+0000 7f6d17ca3640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d1014e920 con 0x7f6d10158210 2023-05-22T15:27:47.824 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.815+0000 7f6d15a3f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6d100c8d70 0x7f6d10147850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:47.824 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.815+0000 7f6d15a3f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6d100c8d70 0x7f6d10147850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41834/0 (socket says 172.21.15.16:41834) 2023-05-22T15:27:47.825 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.815+0000 7f6d15a3f640 1 -- 172.21.15.16:0/3058903332 learned_addr learned my addr 172.21.15.16:0/3058903332 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:47.825 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.815+0000 7f6d16240640 1 --2- 172.21.15.16:0/3058903332 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6d1015bcd0 0x7f6d101518a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:47.825 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.815+0000 7f6d1523e640 1 --2- 172.21.15.16:0/3058903332 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6d10158210 0x7f6d10151360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:47.825 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.815+0000 7f6d15a3f640 1 -- 172.21.15.16:0/3058903332 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6d10158210 msgr2=0x7f6d10151360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:47.825 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.815+0000 7f6d15a3f640 1 --2- 172.21.15.16:0/3058903332 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6d10158210 0x7f6d10151360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:47.825 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.815+0000 7f6d15a3f640 1 -- 172.21.15.16:0/3058903332 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6d1015bcd0 msgr2=0x7f6d101518a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:47.825 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.815+0000 7f6d15a3f640 1 --2- 172.21.15.16:0/3058903332 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6d1015bcd0 0x7f6d101518a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:47.826 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.815+0000 7f6d15a3f640 1 -- 172.21.15.16:0/3058903332 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d08009a70 con 0x7f6d100c8d70 2023-05-22T15:27:47.826 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.815+0000 7f6d16240640 1 --2- 172.21.15.16:0/3058903332 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6d1015bcd0 0x7f6d101518a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:27:47.826 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.815+0000 7f6d15a3f640 1 --2- 172.21.15.16:0/3058903332 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6d100c8d70 0x7f6d10147850 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f6d0c00c410 tx=0x7f6d0c00c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:47.826 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.815+0000 7f6d067fc640 1 -- 172.21.15.16:0/3058903332 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d0c0074e0 con 0x7f6d100c8d70 2023-05-22T15:27:47.826 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.815+0000 7f6d067fc640 1 -- 172.21.15.16:0/3058903332 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d0c007640 con 0x7f6d100c8d70 2023-05-22T15:27:47.826 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.815+0000 7f6d17ca3640 1 -- 172.21.15.16:0/3058903332 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d100cb030 con 0x7f6d100c8d70 2023-05-22T15:27:47.826 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.815+0000 7f6d067fc640 1 -- 172.21.15.16:0/3058903332 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d0c004a30 con 0x7f6d100c8d70 2023-05-22T15:27:47.827 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.815+0000 7f6d17ca3640 1 -- 172.21.15.16:0/3058903332 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d100caea0 con 0x7f6d100c8d70 2023-05-22T15:27:47.827 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.815+0000 7f6d17ca3640 1 -- 172.21.15.16:0/3058903332 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d100cb1c0 con 0x7f6d100c8d70 2023-05-22T15:27:47.828 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.815+0000 7f6d067fc640 1 -- 172.21.15.16:0/3058903332 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7f6d0c004b90 con 0x7f6d100c8d70 2023-05-22T15:27:47.828 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.815+0000 7f6d067fc640 1 --2- 172.21.15.16:0/3058903332 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f6cf4065380 0x7f6cf4067830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:47.829 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.815+0000 7f6d067fc640 1 -- 172.21.15.16:0/3058903332 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f6d0c082960 con 0x7f6d100c8d70 2023-05-22T15:27:47.829 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.819+0000 7f6d1523e640 1 --2- 172.21.15.16:0/3058903332 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f6cf4065380 0x7f6cf4067830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:47.829 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.819+0000 7f6d1523e640 1 --2- 172.21.15.16:0/3058903332 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f6cf4065380 0x7f6cf4067830 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f6d10150370 tx=0x7f6d00009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:47.829 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.823+0000 7f6d067fc640 1 -- 172.21.15.16:0/3058903332 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f6d0c050340 con 0x7f6d100c8d70 2023-05-22T15:27:47.981 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.971+0000 7f6d17ca3640 1 -- 172.21.15.16:0/3058903332 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f6d1014e270 con 0x7f6d100c8d70 2023-05-22T15:27:47.981 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.975+0000 7f6d067fc640 1 -- 172.21.15.16:0/3058903332 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v29) v1 ==== 100+0+11 (secure 0 0 0) 0x7f6d0c053960 con 0x7f6d100c8d70 2023-05-22T15:27:47.982 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 64 2023-05-22T15:27:47.983 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.975+0000 7f6d17ca3640 1 -- 172.21.15.16:0/3058903332 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f6cf4065380 msgr2=0x7f6cf4067830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:47.983 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.975+0000 7f6d17ca3640 1 --2- 172.21.15.16:0/3058903332 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f6cf4065380 0x7f6cf4067830 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f6d10150370 tx=0x7f6d00009040 comp rx=0 tx=0).stop 2023-05-22T15:27:47.984 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.975+0000 7f6d17ca3640 1 -- 172.21.15.16:0/3058903332 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6d100c8d70 msgr2=0x7f6d10147850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:47.984 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.975+0000 7f6d17ca3640 1 --2- 172.21.15.16:0/3058903332 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6d100c8d70 0x7f6d10147850 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f6d0c00c410 tx=0x7f6d0c00c8d0 comp rx=0 tx=0).stop 2023-05-22T15:27:47.984 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.975+0000 7f6d17ca3640 1 -- 172.21.15.16:0/3058903332 shutdown_connections 2023-05-22T15:27:47.985 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.975+0000 7f6d17ca3640 1 --2- 172.21.15.16:0/3058903332 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6d1015bcd0 0x7f6d101518a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:47.985 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.975+0000 7f6d17ca3640 1 --2- 172.21.15.16:0/3058903332 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f6cf4065380 0x7f6cf4067830 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:47.985 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.975+0000 7f6d17ca3640 1 --2- 172.21.15.16:0/3058903332 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6d10158210 0x7f6d10151360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:47.986 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.975+0000 7f6d17ca3640 1 --2- 172.21.15.16:0/3058903332 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6d100c8d70 0x7f6d10147850 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:47.987 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.975+0000 7f6d17ca3640 1 -- 172.21.15.16:0/3058903332 >> 172.21.15.16:0/3058903332 conn(0x7f6d10069720 msgr2=0x7f6d10069af0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:47.987 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.975+0000 7f6d17ca3640 1 -- 172.21.15.16:0/3058903332 shutdown_connections 2023-05-22T15:27:47.987 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:47.975+0000 7f6d17ca3640 1 -- 172.21.15.16:0/3058903332 wait complete. 2023-05-22T15:27:48.001 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-05-22T15:27:48.129 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.123+0000 7ff00f2b4640 1 Processor -- start 2023-05-22T15:27:48.129 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.123+0000 7ff00f2b4640 1 -- start start 2023-05-22T15:27:48.132 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.123+0000 7ff00f2b4640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7ff008159970 0x7ff008159d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:48.132 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.123+0000 7ff00f2b4640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff00815d680 0x7ff0080cb260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:48.132 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.123+0000 7ff00f2b4640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff00815db00 0x7ff00806b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:48.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.123+0000 7ff00f2b4640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff008070c30 con 0x7ff00815d680 2023-05-22T15:27:48.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.123+0000 7ff00f2b4640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff008070e60 con 0x7ff008159970 2023-05-22T15:27:48.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.123+0000 7ff00f2b4640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff008071140 con 0x7ff00815db00 2023-05-22T15:27:48.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.123+0000 7ff00c84f640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff00815d680 0x7ff0080cb260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:48.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.123+0000 7ff00c84f640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff00815d680 0x7ff0080cb260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41844/0 (socket says 172.21.15.16:41844) 2023-05-22T15:27:48.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.123+0000 7ff00c84f640 1 -- 172.21.15.16:0/834059072 learned_addr learned my addr 172.21.15.16:0/834059072 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:48.134 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.123+0000 7ff00d851640 1 --2- 172.21.15.16:0/834059072 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff00815db00 0x7ff00806b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:48.134 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.123+0000 7ff00c84f640 1 -- 172.21.15.16:0/834059072 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff00815db00 msgr2=0x7ff00806b6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:48.134 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.123+0000 7ff00c84f640 1 --2- 172.21.15.16:0/834059072 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff00815db00 0x7ff00806b6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:48.134 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.123+0000 7ff00c84f640 1 -- 172.21.15.16:0/834059072 >> v2:172.21.15.62:3300/0 conn(0x7ff008159970 msgr2=0x7ff008159d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:48.134 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.123+0000 7ff00d050640 1 --2- 172.21.15.16:0/834059072 >> v2:172.21.15.62:3300/0 conn(0x7ff008159970 0x7ff008159d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:48.135 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.123+0000 7ff00c84f640 1 --2- 172.21.15.16:0/834059072 >> v2:172.21.15.62:3300/0 conn(0x7ff008159970 0x7ff008159d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:48.136 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.123+0000 7ff00c84f640 1 -- 172.21.15.16:0/834059072 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff008071420 con 0x7ff00815d680 2023-05-22T15:27:48.136 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.123+0000 7ff00d050640 1 --2- 172.21.15.16:0/834059072 >> v2:172.21.15.62:3300/0 conn(0x7ff008159970 0x7ff008159d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:27:48.139 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.123+0000 7ff00c84f640 1 --2- 172.21.15.16:0/834059072 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff00815d680 0x7ff0080cb260 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7ff00806daf0 tx=0x7feff801c440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6ddb6e550b1179b server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:48.139 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.127+0000 7feffffff640 1 -- 172.21.15.16:0/834059072 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feff80ac640 con 0x7ff00815d680 2023-05-22T15:27:48.139 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.127+0000 7feffffff640 1 -- 172.21.15.16:0/834059072 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feff80ac7a0 con 0x7ff00815d680 2023-05-22T15:27:48.139 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.127+0000 7feffffff640 1 -- 172.21.15.16:0/834059072 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feff80acb30 con 0x7ff00815d680 2023-05-22T15:27:48.139 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.127+0000 7ff00f2b4640 1 -- 172.21.15.16:0/834059072 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff00815d680 msgr2=0x7ff0080cb260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:48.140 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.127+0000 7ff00f2b4640 1 --2- 172.21.15.16:0/834059072 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff00815d680 0x7ff0080cb260 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7ff00806daf0 tx=0x7feff801c440 comp rx=0 tx=0).stop 2023-05-22T15:27:48.140 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.127+0000 7ff00f2b4640 1 -- 172.21.15.16:0/834059072 shutdown_connections 2023-05-22T15:27:48.140 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.127+0000 7ff00f2b4640 1 --2- 172.21.15.16:0/834059072 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff00815d680 0x7ff0080cb260 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:48.140 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.127+0000 7ff00f2b4640 1 --2- 172.21.15.16:0/834059072 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff00815db00 0x7ff00806b6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:48.140 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.127+0000 7ff00f2b4640 1 --2- 172.21.15.16:0/834059072 >> v2:172.21.15.62:3300/0 conn(0x7ff008159970 0x7ff008159d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:48.140 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.127+0000 7ff00f2b4640 1 -- 172.21.15.16:0/834059072 >> 172.21.15.16:0/834059072 conn(0x7ff008069720 msgr2=0x7ff008069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:48.140 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.127+0000 7ff00f2b4640 1 -- 172.21.15.16:0/834059072 shutdown_connections 2023-05-22T15:27:48.141 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.127+0000 7ff00f2b4640 1 -- 172.21.15.16:0/834059072 wait complete. 2023-05-22T15:27:48.141 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.127+0000 7ff00f2b4640 1 Processor -- start 2023-05-22T15:27:48.141 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.127+0000 7ff00f2b4640 1 -- start start 2023-05-22T15:27:48.141 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.127+0000 7ff00f2b4640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff008159970 0x7ff00814d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:48.141 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.127+0000 7ff00f2b4640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff00815d680 0x7ff008157500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:48.141 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.127+0000 7ff00f2b4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff00815db00 0x7ff00814b0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:48.141 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.127+0000 7ff00f2b4640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff00814bd70 con 0x7ff00815db00 2023-05-22T15:27:48.141 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.127+0000 7ff00f2b4640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff00813a140 con 0x7ff008159970 2023-05-22T15:27:48.142 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.127+0000 7ff00f2b4640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff00813a420 con 0x7ff00815d680 2023-05-22T15:27:48.142 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.127+0000 7ff00d851640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff00815db00 0x7ff00814b0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:48.142 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.127+0000 7ff00d851640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff00815db00 0x7ff00814b0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41848/0 (socket says 172.21.15.16:41848) 2023-05-22T15:27:48.142 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.127+0000 7ff00d851640 1 -- 172.21.15.16:0/1366849679 learned_addr learned my addr 172.21.15.16:0/1366849679 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:48.142 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.127+0000 7ff00d050640 1 --2- 172.21.15.16:0/1366849679 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff008159970 0x7ff00814d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:48.142 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.127+0000 7ff00c84f640 1 --2- 172.21.15.16:0/1366849679 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff00815d680 0x7ff008157500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:48.142 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.127+0000 7ff00d851640 1 -- 172.21.15.16:0/1366849679 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff00815d680 msgr2=0x7ff008157500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:48.143 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.127+0000 7ff00d851640 1 --2- 172.21.15.16:0/1366849679 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff00815d680 0x7ff008157500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:48.143 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.127+0000 7ff00d851640 1 -- 172.21.15.16:0/1366849679 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff008159970 msgr2=0x7ff00814d1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:48.143 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.127+0000 7ff00d851640 1 --2- 172.21.15.16:0/1366849679 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff008159970 0x7ff00814d1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:48.143 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.127+0000 7ff00d851640 1 -- 172.21.15.16:0/1366849679 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feff80094d0 con 0x7ff00815db00 2023-05-22T15:27:48.143 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.127+0000 7ff00d851640 1 --2- 172.21.15.16:0/1366849679 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff00815db00 0x7ff00814b0b0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7ff00000afd0 tx=0x7ff00000ca20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:48.143 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.127+0000 7feffdffb640 1 -- 172.21.15.16:0/1366849679 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff000004220 con 0x7ff00815db00 2023-05-22T15:27:48.143 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.127+0000 7feffdffb640 1 -- 172.21.15.16:0/1366849679 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff000004380 con 0x7ff00815db00 2023-05-22T15:27:48.144 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.127+0000 7ff00f2b4640 1 -- 172.21.15.16:0/1366849679 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0080cae20 con 0x7ff00815db00 2023-05-22T15:27:48.144 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.127+0000 7feffdffb640 1 -- 172.21.15.16:0/1366849679 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff000003840 con 0x7ff00815db00 2023-05-22T15:27:48.144 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.127+0000 7ff00f2b4640 1 -- 172.21.15.16:0/1366849679 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0080cac90 con 0x7ff00815db00 2023-05-22T15:27:48.144 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.131+0000 7ff00f2b4640 1 -- 172.21.15.16:0/1366849679 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff00815adc0 con 0x7ff00815db00 2023-05-22T15:27:48.144 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.131+0000 7feffdffb640 1 -- 172.21.15.16:0/1366849679 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7ff0000039a0 con 0x7ff00815db00 2023-05-22T15:27:48.144 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.131+0000 7feffdffb640 1 --2- 172.21.15.16:0/1366849679 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fefec0653b0 0x7fefec067860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:48.144 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.131+0000 7feffdffb640 1 -- 172.21.15.16:0/1366849679 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6871+0+0 (secure 0 0 0) 0x7ff000082820 con 0x7ff00815db00 2023-05-22T15:27:48.145 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.131+0000 7ff00d050640 1 --2- 172.21.15.16:0/1366849679 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fefec0653b0 0x7fefec067860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:48.145 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.131+0000 7ff00d050640 1 --2- 172.21.15.16:0/1366849679 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fefec0653b0 0x7fefec067860 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7ff0080b0c00 tx=0x7ff0040079c0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:48.145 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.135+0000 7feffdffb640 1 -- 172.21.15.16:0/1366849679 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7ff000050160 con 0x7ff00815db00 2023-05-22T15:27:48.297 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.287+0000 7ff00f2b4640 1 -- 172.21.15.16:0/1366849679 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7ff00814beb0 con 0x7ff00815db00 2023-05-22T15:27:48.297 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.291+0000 7feffdffb640 1 -- 172.21.15.16:0/1366849679 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v29) v1 ==== 103+0+11 (secure 0 0 0) 0x7ff00814beb0 con 0x7ff00815db00 2023-05-22T15:27:48.297 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2023-05-22T15:27:48.300 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.291+0000 7ff00f2b4640 1 -- 172.21.15.16:0/1366849679 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fefec0653b0 msgr2=0x7fefec067860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:48.300 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.291+0000 7ff00f2b4640 1 --2- 172.21.15.16:0/1366849679 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fefec0653b0 0x7fefec067860 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7ff0080b0c00 tx=0x7ff0040079c0 comp rx=0 tx=0).stop 2023-05-22T15:27:48.300 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.291+0000 7ff00f2b4640 1 -- 172.21.15.16:0/1366849679 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff00815db00 msgr2=0x7ff00814b0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:48.300 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.291+0000 7ff00f2b4640 1 --2- 172.21.15.16:0/1366849679 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff00815db00 0x7ff00814b0b0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7ff00000afd0 tx=0x7ff00000ca20 comp rx=0 tx=0).stop 2023-05-22T15:27:48.300 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.291+0000 7ff00f2b4640 1 -- 172.21.15.16:0/1366849679 shutdown_connections 2023-05-22T15:27:48.301 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.291+0000 7ff00f2b4640 1 --2- 172.21.15.16:0/1366849679 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff00815db00 0x7ff00814b0b0 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:48.301 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.291+0000 7ff00f2b4640 1 --2- 172.21.15.16:0/1366849679 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fefec0653b0 0x7fefec067860 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:48.301 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.291+0000 7ff00f2b4640 1 --2- 172.21.15.16:0/1366849679 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff00815d680 0x7ff008157500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:48.301 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.291+0000 7ff00f2b4640 1 --2- 172.21.15.16:0/1366849679 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff008159970 0x7ff00814d1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:48.301 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.291+0000 7ff00f2b4640 1 -- 172.21.15.16:0/1366849679 >> 172.21.15.16:0/1366849679 conn(0x7ff008069720 msgr2=0x7ff00815c940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:48.301 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.291+0000 7ff00f2b4640 1 -- 172.21.15.16:0/1366849679 shutdown_connections 2023-05-22T15:27:48.301 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.291+0000 7ff00f2b4640 1 -- 172.21.15.16:0/1366849679 wait complete. 2023-05-22T15:27:48.318 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-05-22T15:27:48.446 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.439+0000 7fb7a8396640 1 Processor -- start 2023-05-22T15:27:48.446 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.439+0000 7fb7a8396640 1 -- start start 2023-05-22T15:27:48.447 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.439+0000 7fb7a8396640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb7a015fc50 0x7fb7a0160020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:48.447 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.439+0000 7fb7a8396640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7a015c490 con 0x7fb7a00d0740 2023-05-22T15:27:48.447 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.439+0000 7fb7a8396640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7a015c140 con 0x7fb7a015c920 2023-05-22T15:27:48.447 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.439+0000 7fb7a8396640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7a00ccb50 con 0x7fb7a015fc50 2023-05-22T15:27:48.447 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.439+0000 7fb7a6933640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fb7a00d0740 0x7fb7a00ce790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:45682/0 (socket says 172.21.15.16:45682) 2023-05-22T15:27:48.448 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.439+0000 7fb7a6933640 1 -- 172.21.15.16:0/883100372 learned_addr learned my addr 172.21.15.16:0/883100372 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:27:48.448 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.439+0000 7fb7a6132640 1 --2- 172.21.15.16:0/883100372 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb7a015fc50 0x7fb7a0160020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:48.449 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.439+0000 7fb7a5130640 1 -- 172.21.15.16:0/883100372 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1192106587 0 0) 0x7fb7a015c490 con 0x7fb7a00d0740 2023-05-22T15:27:48.449 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.439+0000 7fb7a5130640 1 -- 172.21.15.16:0/883100372 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb78c003530 con 0x7fb7a00d0740 2023-05-22T15:27:48.449 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.439+0000 7fb7a5130640 1 -- 172.21.15.16:0/883100372 <== mon.1 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 897733588 0 0) 0x7fb7a015c140 con 0x7fb7a015c920 2023-05-22T15:27:48.449 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.439+0000 7fb7a5130640 1 -- 172.21.15.16:0/883100372 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb7a015c490 con 0x7fb7a015c920 2023-05-22T15:27:48.449 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.439+0000 7fb7a6132640 1 -- 172.21.15.16:0/883100372 >> v1:172.21.15.62:6789/0 conn(0x7fb7a015c920 legacy=0x7fb7a00d0030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:48.449 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.439+0000 7fb7a6132640 1 -- 172.21.15.16:0/883100372 >> v1:172.21.15.16:6789/0 conn(0x7fb7a00d0740 legacy=0x7fb7a00ce790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:48.450 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.439+0000 7fb7a6132640 1 -- 172.21.15.16:0/883100372 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7a00cce30 con 0x7fb7a015fc50 2023-05-22T15:27:48.450 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.443+0000 7fb7a6132640 1 --2- 172.21.15.16:0/883100372 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb7a015fc50 0x7fb7a0160020 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fb79c009dc0 tx=0x7fb79c01c940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5a548cbc6076873e server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:48.450 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.443+0000 7fb7a5130640 1 -- 172.21.15.16:0/883100372 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb79c0a5910 con 0x7fb7a015fc50 2023-05-22T15:27:48.451 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.443+0000 7fb7a5130640 1 -- 172.21.15.16:0/883100372 <== mon.2 v2:172.21.15.62:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb79c0a5a70 con 0x7fb7a015fc50 2023-05-22T15:27:48.451 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.443+0000 7fb7a5130640 1 -- 172.21.15.16:0/883100372 <== mon.2 v2:172.21.15.62:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb79c0a5de0 con 0x7fb7a015fc50 2023-05-22T15:27:48.451 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.443+0000 7fb7a8396640 1 -- 172.21.15.16:0/883100372 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb7a015fc50 msgr2=0x7fb7a0160020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:48.452 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.443+0000 7fb7a8396640 1 --2- 172.21.15.16:0/883100372 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb7a015fc50 0x7fb7a0160020 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fb79c009dc0 tx=0x7fb79c01c940 comp rx=0 tx=0).stop 2023-05-22T15:27:48.452 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.443+0000 7fb7a8396640 1 -- 172.21.15.16:0/883100372 shutdown_connections 2023-05-22T15:27:48.452 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.443+0000 7fb7a8396640 1 --2- 172.21.15.16:0/883100372 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb7a015fc50 0x7fb7a0160020 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:48.452 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.443+0000 7fb7a8396640 1 -- 172.21.15.16:0/883100372 >> 172.21.15.16:0/883100372 conn(0x7fb7a00657f0 msgr2=0x7fb7a0070e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:48.452 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.443+0000 7fb7a8396640 1 -- 172.21.15.16:0/883100372 shutdown_connections 2023-05-22T15:27:48.452 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.443+0000 7fb7a8396640 1 -- 172.21.15.16:0/883100372 wait complete. 2023-05-22T15:27:48.453 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.443+0000 7fb7a8396640 1 Processor -- start 2023-05-22T15:27:48.453 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.447+0000 7fb7a8396640 1 -- start start 2023-05-22T15:27:48.456 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.447+0000 7fb7a8396640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb7a00d0740 0x7fb7a00b4660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:48.457 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.447+0000 7fb7a8396640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb7a015c920 0x7fb7a014d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:48.457 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.447+0000 7fb7a8396640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb7a015fc50 0x7fb7a00cb350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:48.457 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.447+0000 7fb7a8396640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7a00caec0 con 0x7fb7a00d0740 2023-05-22T15:27:48.457 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.447+0000 7fb7a8396640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7a00ca2b0 con 0x7fb7a015c920 2023-05-22T15:27:48.457 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.447+0000 7fb7a8396640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7a00ca590 con 0x7fb7a015fc50 2023-05-22T15:27:48.457 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.447+0000 7fb7a6132640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb7a00d0740 0x7fb7a00b4660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:48.458 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.447+0000 7fb7a6132640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb7a00d0740 0x7fb7a00b4660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41864/0 (socket says 172.21.15.16:41864) 2023-05-22T15:27:48.458 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.447+0000 7fb7a6132640 1 -- 172.21.15.16:0/2620846130 learned_addr learned my addr 172.21.15.16:0/2620846130 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:48.458 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.447+0000 7fb7a5931640 1 --2- 172.21.15.16:0/2620846130 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb7a015c920 0x7fb7a014d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:48.458 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.447+0000 7fb7a6933640 1 --2- 172.21.15.16:0/2620846130 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb7a015fc50 0x7fb7a00cb350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:48.458 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.447+0000 7fb7a6132640 1 -- 172.21.15.16:0/2620846130 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb7a015fc50 msgr2=0x7fb7a00cb350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:48.458 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.447+0000 7fb7a6132640 1 --2- 172.21.15.16:0/2620846130 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb7a015fc50 0x7fb7a00cb350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:48.458 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.447+0000 7fb7a6132640 1 -- 172.21.15.16:0/2620846130 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb7a015c920 msgr2=0x7fb7a014d600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:48.458 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.447+0000 7fb7a6132640 1 --2- 172.21.15.16:0/2620846130 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb7a015c920 0x7fb7a014d600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:48.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.447+0000 7fb7a6132640 1 -- 172.21.15.16:0/2620846130 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb79c009a70 con 0x7fb7a00d0740 2023-05-22T15:27:48.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.447+0000 7fb7a6933640 1 --2- 172.21.15.16:0/2620846130 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb7a015fc50 0x7fb7a00cb350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:27:48.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.447+0000 7fb7a6132640 1 --2- 172.21.15.16:0/2620846130 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb7a00d0740 0x7fb7a00b4660 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fb79c01a570 tx=0x7fb79c0196e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:48.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.447+0000 7fb796ffd640 1 -- 172.21.15.16:0/2620846130 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb79c0a5910 con 0x7fb7a00d0740 2023-05-22T15:27:48.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.447+0000 7fb796ffd640 1 -- 172.21.15.16:0/2620846130 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb79c003bb0 con 0x7fb7a00d0740 2023-05-22T15:27:48.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.447+0000 7fb7a8396640 1 -- 172.21.15.16:0/2620846130 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7a0169c70 con 0x7fb7a00d0740 2023-05-22T15:27:48.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.447+0000 7fb796ffd640 1 -- 172.21.15.16:0/2620846130 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb79c004040 con 0x7fb7a00d0740 2023-05-22T15:27:48.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.447+0000 7fb7a8396640 1 -- 172.21.15.16:0/2620846130 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7a0169ae0 con 0x7fb7a00d0740 2023-05-22T15:27:48.460 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.447+0000 7fb7a8396640 1 -- 172.21.15.16:0/2620846130 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7a0169e00 con 0x7fb7a00d0740 2023-05-22T15:27:48.460 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.447+0000 7fb796ffd640 1 -- 172.21.15.16:0/2620846130 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7fb79c0041a0 con 0x7fb7a00d0740 2023-05-22T15:27:48.460 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.447+0000 7fb796ffd640 1 --2- 172.21.15.16:0/2620846130 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fb784065380 0x7fb784067830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:48.461 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.451+0000 7fb796ffd640 1 -- 172.21.15.16:0/2620846130 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6871+0+0 (secure 0 0 0) 0x7fb79c11dda0 con 0x7fb7a00d0740 2023-05-22T15:27:48.461 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.451+0000 7fb7a5931640 1 --2- 172.21.15.16:0/2620846130 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fb784065380 0x7fb784067830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:48.461 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.451+0000 7fb7a5931640 1 --2- 172.21.15.16:0/2620846130 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fb784065380 0x7fb784067830 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fb790007fd0 tx=0x7fb790009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:48.462 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.455+0000 7fb796ffd640 1 -- 172.21.15.16:0/2620846130 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fb79c0eb670 con 0x7fb7a00d0740 2023-05-22T15:27:48.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.623+0000 7fb7a8396640 1 -- 172.21.15.16:0/2620846130 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fb7a004dbe0 con 0x7fb7a00d0740 2023-05-22T15:27:48.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.623+0000 7fb796ffd640 1 -- 172.21.15.16:0/2620846130 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v29) v1 ==== 80+0+0 (secure 0 0 0) 0x7fb79c0c6080 con 0x7fb7a00d0740 2023-05-22T15:27:48.631 INFO:teuthology.orchestra.run.smithi016.stderr:listed 0 entries 2023-05-22T15:27:48.633 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.623+0000 7fb7a8396640 1 -- 172.21.15.16:0/2620846130 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fb784065380 msgr2=0x7fb784067830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:48.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.623+0000 7fb7a8396640 1 --2- 172.21.15.16:0/2620846130 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fb784065380 0x7fb784067830 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fb790007fd0 tx=0x7fb790009040 comp rx=0 tx=0).stop 2023-05-22T15:27:48.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.623+0000 7fb7a8396640 1 -- 172.21.15.16:0/2620846130 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb7a00d0740 msgr2=0x7fb7a00b4660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:48.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.623+0000 7fb7a8396640 1 --2- 172.21.15.16:0/2620846130 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb7a00d0740 0x7fb7a00b4660 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fb79c01a570 tx=0x7fb79c0196e0 comp rx=0 tx=0).stop 2023-05-22T15:27:48.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.627+0000 7fb7a8396640 1 -- 172.21.15.16:0/2620846130 shutdown_connections 2023-05-22T15:27:48.635 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.627+0000 7fb7a8396640 1 --2- 172.21.15.16:0/2620846130 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fb784065380 0x7fb784067830 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:48.635 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.627+0000 7fb7a8396640 1 --2- 172.21.15.16:0/2620846130 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb7a015fc50 0x7fb7a00cb350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:48.635 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.627+0000 7fb7a8396640 1 --2- 172.21.15.16:0/2620846130 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb7a015c920 0x7fb7a014d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:48.635 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.627+0000 7fb7a8396640 1 --2- 172.21.15.16:0/2620846130 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb7a00d0740 0x7fb7a00b4660 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:48.637 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.627+0000 7fb7a8396640 1 -- 172.21.15.16:0/2620846130 >> 172.21.15.16:0/2620846130 conn(0x7fb7a00657f0 msgr2=0x7fb7a00ce790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:48.637 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.627+0000 7fb7a8396640 1 -- 172.21.15.16:0/2620846130 shutdown_connections 2023-05-22T15:27:48.637 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.627+0000 7fb7a8396640 1 -- 172.21.15.16:0/2620846130 wait complete. 2023-05-22T15:27:48.650 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi062.front.sepia.ceph.com... 2023-05-22T15:27:48.653 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:27:48.653 DEBUG:teuthology.orchestra.run.smithi062:> sudo fusermount -u /home/ubuntu/cephtest/mnt.1 2023-05-22T15:27:48.675 INFO:teuthology.orchestra.run:waiting for 300 2023-05-22T15:27:48.677 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi062.front.sepia.ceph.com 2023-05-22T15:27:48.677 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:27:48.678 DEBUG:teuthology.orchestra.run.smithi062:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2023-05-22T15:27:48.730 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-05-22T15:27:48.731 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2023-05-22T15:27:48.738 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-05-22T15:27:48.738 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2023-05-22T15:27:48.786 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-05-22T15:27:48.956 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.947+0000 7f863d877640 1 Processor -- start 2023-05-22T15:27:48.957 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.947+0000 7f863d877640 1 -- start start 2023-05-22T15:27:48.958 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.951+0000 7f863d877640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f863815cfa0 0x7f863806db00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:48.958 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.951+0000 7f863d877640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f86380cf870 0x7f86380b1fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:48.958 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.951+0000 7f863d877640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f863806e040 con 0x7f863806e740 2023-05-22T15:27:48.959 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.951+0000 7f863d877640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86380cb9f0 con 0x7f86380cf870 2023-05-22T15:27:48.959 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.951+0000 7f863d877640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86380b24f0 con 0x7f863815cfa0 2023-05-22T15:27:48.959 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.951+0000 7f86367fc640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f86380cf870 0x7f86380b1fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:48.959 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.951+0000 7f86367fc640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f86380cf870 0x7f86380b1fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41880/0 (socket says 172.21.15.16:41880) 2023-05-22T15:27:48.959 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.951+0000 7f86367fc640 1 -- 172.21.15.16:0/2261184479 learned_addr learned my addr 172.21.15.16:0/2261184479 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:48.959 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.951+0000 7f86367fc640 1 -- 172.21.15.16:0/2261184479 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f863815cfa0 msgr2=0x7f863806db00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:48.960 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.951+0000 7f8636ffd640 1 --2- 172.21.15.16:0/2261184479 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f863815cfa0 0x7f863806db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:48.960 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.951+0000 7f86367fc640 1 --2- 172.21.15.16:0/2261184479 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f863815cfa0 0x7f863806db00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:48.960 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.951+0000 7f86367fc640 1 -- 172.21.15.16:0/2261184479 >> v1:172.21.15.62:6789/0 conn(0x7f863806e740 legacy=0x7f8638070de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:48.960 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.951+0000 7f86367fc640 1 -- 172.21.15.16:0/2261184479 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f863806e490 con 0x7f86380cf870 2023-05-22T15:27:48.960 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.951+0000 7f8636ffd640 1 --2- 172.21.15.16:0/2261184479 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f863815cfa0 0x7f863806db00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:27:48.961 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.951+0000 7f86367fc640 1 --2- 172.21.15.16:0/2261184479 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f86380cf870 0x7f86380b1fb0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f862c009870 tx=0x7f862c01c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8eb99aa3bbcabbb4 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:48.961 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.951+0000 7f8635ffb640 1 -- 172.21.15.16:0/2261184479 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f862c0ab680 con 0x7f86380cf870 2023-05-22T15:27:48.962 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.951+0000 7f8635ffb640 1 -- 172.21.15.16:0/2261184479 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f862c0ab7e0 con 0x7f86380cf870 2023-05-22T15:27:48.964 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.951+0000 7f8635ffb640 1 -- 172.21.15.16:0/2261184479 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f862c0abb70 con 0x7f86380cf870 2023-05-22T15:27:48.964 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.951+0000 7f863d877640 1 -- 172.21.15.16:0/2261184479 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f86380cf870 msgr2=0x7f86380b1fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:48.964 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.951+0000 7f863d877640 1 --2- 172.21.15.16:0/2261184479 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f86380cf870 0x7f86380b1fb0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f862c009870 tx=0x7f862c01c480 comp rx=0 tx=0).stop 2023-05-22T15:27:48.964 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.951+0000 7f863d877640 1 -- 172.21.15.16:0/2261184479 shutdown_connections 2023-05-22T15:27:48.965 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.951+0000 7f863d877640 1 --2- 172.21.15.16:0/2261184479 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f86380cf870 0x7f86380b1fb0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:48.965 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.951+0000 7f863d877640 1 --2- 172.21.15.16:0/2261184479 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f863815cfa0 0x7f863806db00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:48.965 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.951+0000 7f863d877640 1 -- 172.21.15.16:0/2261184479 >> 172.21.15.16:0/2261184479 conn(0x7f86380cc870 msgr2=0x7f86380cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:48.965 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.951+0000 7f863d877640 1 -- 172.21.15.16:0/2261184479 shutdown_connections 2023-05-22T15:27:48.965 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.955+0000 7f863d877640 1 -- 172.21.15.16:0/2261184479 wait complete. 2023-05-22T15:27:48.967 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.955+0000 7f863d877640 1 Processor -- start 2023-05-22T15:27:48.967 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.955+0000 7f863d877640 1 -- start start 2023-05-22T15:27:48.967 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.955+0000 7f863d877640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f863806e740 0x7f86380ce680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:48.968 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.955+0000 7f863d877640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f86380cf870 0x7f8638149cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:48.968 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.955+0000 7f863d877640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f863815cfa0 0x7f86380b3cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:48.968 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.955+0000 7f863d877640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8638163890 con 0x7f863806e740 2023-05-22T15:27:48.968 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.955+0000 7f863d877640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8638163b70 con 0x7f86380cf870 2023-05-22T15:27:48.968 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.955+0000 7f863d877640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8638163e50 con 0x7f863815cfa0 2023-05-22T15:27:48.968 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.955+0000 7f8636ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f863806e740 0x7f86380ce680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:48.968 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.955+0000 7f8636ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f863806e740 0x7f86380ce680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41884/0 (socket says 172.21.15.16:41884) 2023-05-22T15:27:48.969 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.955+0000 7f8636ffd640 1 -- 172.21.15.16:0/566015296 learned_addr learned my addr 172.21.15.16:0/566015296 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:48.969 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.955+0000 7f86367fc640 1 --2- 172.21.15.16:0/566015296 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f86380cf870 0x7f8638149cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:48.969 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.955+0000 7f86377fe640 1 --2- 172.21.15.16:0/566015296 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f863815cfa0 0x7f86380b3cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:48.969 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.955+0000 7f8636ffd640 1 -- 172.21.15.16:0/566015296 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f863815cfa0 msgr2=0x7f86380b3cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:48.969 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.955+0000 7f8636ffd640 1 --2- 172.21.15.16:0/566015296 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f863815cfa0 0x7f86380b3cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:48.969 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.955+0000 7f8636ffd640 1 -- 172.21.15.16:0/566015296 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f86380cf870 msgr2=0x7f8638149cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:48.969 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.955+0000 7f8636ffd640 1 --2- 172.21.15.16:0/566015296 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f86380cf870 0x7f8638149cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:48.970 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.955+0000 7f8636ffd640 1 -- 172.21.15.16:0/566015296 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f862c0094d0 con 0x7f863806e740 2023-05-22T15:27:48.970 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.955+0000 7f8636ffd640 1 --2- 172.21.15.16:0/566015296 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f863806e740 0x7f86380ce680 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f862800c7f0 tx=0x7f862800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:48.970 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.955+0000 7f861ffff640 1 -- 172.21.15.16:0/566015296 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8628007820 con 0x7f863806e740 2023-05-22T15:27:48.970 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.955+0000 7f861ffff640 1 -- 172.21.15.16:0/566015296 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8628007980 con 0x7f863806e740 2023-05-22T15:27:48.970 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.955+0000 7f863d877640 1 -- 172.21.15.16:0/566015296 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86380cb610 con 0x7f863806e740 2023-05-22T15:27:48.970 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.955+0000 7f861ffff640 1 -- 172.21.15.16:0/566015296 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8628007d30 con 0x7f863806e740 2023-05-22T15:27:48.970 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.955+0000 7f863d877640 1 -- 172.21.15.16:0/566015296 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86380cb330 con 0x7f863806e740 2023-05-22T15:27:48.970 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.955+0000 7f863d877640 1 -- 172.21.15.16:0/566015296 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8638071900 con 0x7f863806e740 2023-05-22T15:27:48.971 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.959+0000 7f861ffff640 1 -- 172.21.15.16:0/566015296 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7f86280113f0 con 0x7f863806e740 2023-05-22T15:27:48.971 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.959+0000 7f861ffff640 1 --2- 172.21.15.16:0/566015296 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f8624065380 0x7f8624067830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:48.971 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.959+0000 7f861ffff640 1 -- 172.21.15.16:0/566015296 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f8628082f10 con 0x7f863806e740 2023-05-22T15:27:48.971 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.959+0000 7f86367fc640 1 --2- 172.21.15.16:0/566015296 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f8624065380 0x7f8624067830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:48.971 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.959+0000 7f86367fc640 1 --2- 172.21.15.16:0/566015296 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f8624065380 0x7f8624067830 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f862c0af000 tx=0x7f862c0a7000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:48.972 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:48.963+0000 7f861ffff640 1 -- 172.21.15.16:0/566015296 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f8628050800 con 0x7f863806e740 2023-05-22T15:27:49.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.115+0000 7f863d877640 1 -- 172.21.15.16:0/566015296 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f863804dbe0 con 0x7f863806e740 2023-05-22T15:27:49.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.115+0000 7f861ffff640 1 -- 172.21.15.16:0/566015296 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v29) v1 ==== 74+0+19700 (secure 0 0 0) 0x7f8628053e20 con 0x7f863806e740 2023-05-22T15:27:49.125 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:27:49.126 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":29,"fsid":"b2719834-7847-4d98-90a6-f65d57f85c49","created":"2023-05-22T15:26:25.259314+0000","modified":"2023-05-22T15:27:31.182317+0000","last_up_change":"2023-05-22T15:26:36.448524+0000","last_in_change":"2023-05-22T15:26:27.675668+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-05-22T15:26:38.019781+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-05-22T15:26:43.570514+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-05-22T15:26:47.879159+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-05-22T15:26:48.522130+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data_ec","create_time":"2023-05-22T15:26:51.616142+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"9e7ff854-b25f-4cbb-a5c2-1799d9c53d2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":37021},{"type":"v1","addr":"172.21.15.16:6825","nonce":37021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":37021},{"type":"v1","addr":"172.21.15.16:6827","nonce":37021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":37021},{"type":"v1","addr":"172.21.15.16:6831","nonce":37021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":37021},{"type":"v1","addr":"172.21.15.16:6829","nonce":37021}]},"public_addr":"172.21.15.16:6825/37021","cluster_addr":"172.21.15.16:6827/37021","heartbeat_back_addr":"172.21.15.16:6831/37021","heartbeat_front_addr":"172.21.15.16:6829/37021","state":["exists","up"]},{"osd":1,"uuid":"4702fbc9-a927-4565-aca5-41dc69cc8264","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":37008},{"type":"v1","addr":"172.21.15.16:6801","nonce":37008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":37008},{"type":"v1","addr":"172.21.15.16:6803","nonce":37008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":37008},{"type":"v1","addr":"172.21.15.16:6807","nonce":37008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":37008},{"type":"v1","addr":"172.21.15.16:6805","nonce":37008}]},"public_addr":"172.21.15.16:6801/37008","cluster_addr":"172.21.15.16:6803/37008","heartbeat_back_addr":"172.21.15.16:6807/37008","heartbeat_front_addr":"172.21.15.16:6805/37008","state":["exists","up"]},{"osd":2,"uuid":"1e7bba46-4eba-47ef-bb81-5e2d371638ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":37009},{"type":"v1","addr":"172.21.15.16:6809","nonce":37009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":37009},{"type":"v1","addr":"172.21.15.16:6811","nonce":37009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":37009},{"type":"v1","addr":"172.21.15.16:6815","nonce":37009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":37009},{"type":"v1","addr":"172.21.15.16:6813","nonce":37009}]},"public_addr":"172.21.15.16:6809/37009","cluster_addr":"172.21.15.16:6811/37009","heartbeat_back_addr":"172.21.15.16:6815/37009","heartbeat_front_addr":"172.21.15.16:6813/37009","state":["exists","up"]},{"osd":3,"uuid":"383cacc0-5a88-4904-990e-498ca59ef48a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":37020},{"type":"v1","addr":"172.21.15.16:6817","nonce":37020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":37020},{"type":"v1","addr":"172.21.15.16:6819","nonce":37020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":37020},{"type":"v1","addr":"172.21.15.16:6823","nonce":37020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":37020},{"type":"v1","addr":"172.21.15.16:6821","nonce":37020}]},"public_addr":"172.21.15.16:6817/37020","cluster_addr":"172.21.15.16:6819/37020","heartbeat_back_addr":"172.21.15.16:6823/37020","heartbeat_front_addr":"172.21.15.16:6821/37020","state":["exists","up"]},{"osd":4,"uuid":"970500ef-95b7-42c1-b080-eed4fae7013d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":37353},{"type":"v1","addr":"172.21.15.62:6817","nonce":37353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6819","nonce":37353},{"type":"v1","addr":"172.21.15.62:6822","nonce":37353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37353},{"type":"v1","addr":"172.21.15.62:6831","nonce":37353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37353},{"type":"v1","addr":"172.21.15.62:6826","nonce":37353}]},"public_addr":"172.21.15.62:6817/37353","cluster_addr":"172.21.15.62:6822/37353","heartbeat_back_addr":"172.21.15.62:6831/37353","heartbeat_front_addr":"172.21.15.62:6826/37353","state":["exists","up"]},{"osd":5,"uuid":"022d67db-d4d8-445b-b1b9-8b446a00f5c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":37352},{"type":"v1","addr":"172.21.15.62:6801","nonce":37352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":37352},{"type":"v1","addr":"172.21.15.62:6803","nonce":37352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":37352},{"type":"v1","addr":"172.21.15.62:6807","nonce":37352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":37352},{"type":"v1","addr":"172.21.15.62:6805","nonce":37352}]},"public_addr":"172.21.15.62:6801/37352","cluster_addr":"172.21.15.62:6803/37352","heartbeat_back_addr":"172.21.15.62:6807/37352","heartbeat_front_addr":"172.21.15.62:6805/37352","state":["exists","up"]},{"osd":6,"uuid":"42bd372b-6688-4a1b-8407-e16e8c266e33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":37354},{"type":"v1","addr":"172.21.15.62:6820","nonce":37354}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":37354},{"type":"v1","addr":"172.21.15.62:6823","nonce":37354}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6829","nonce":37354},{"type":"v1","addr":"172.21.15.62:6830","nonce":37354}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6825","nonce":37354},{"type":"v1","addr":"172.21.15.62:6827","nonce":37354}]},"public_addr":"172.21.15.62:6820/37354","cluster_addr":"172.21.15.62:6823/37354","heartbeat_back_addr":"172.21.15.62:6830/37354","heartbeat_front_addr":"172.21.15.62:6827/37354","state":["exists","up"]},{"osd":7,"uuid":"ac70b2a3-f41b-4bf2-b227-7f2cbb55426d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37355},{"type":"v1","addr":"172.21.15.62:6809","nonce":37355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37355},{"type":"v1","addr":"172.21.15.62:6811","nonce":37355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":37355},{"type":"v1","addr":"172.21.15.62:6815","nonce":37355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":37355},{"type":"v1","addr":"172.21.15.62:6813","nonce":37355}]},"public_addr":"172.21.15.62:6809/37355","cluster_addr":"172.21.15.62:6811/37355","heartbeat_back_addr":"172.21.15.62:6815/37355","heartbeat_front_addr":"172.21.15.62:6813/37355","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.368896+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.107273+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.276888+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.287502+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.124791+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.251937+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.255679+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.232797+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"4.23","mappings":[{"from":3,"to":2}]},{"pgid":"4.31","mappings":[{"from":1,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-05-22T15:27:49.127 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.119+0000 7f863d877640 1 -- 172.21.15.16:0/566015296 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f8624065380 msgr2=0x7f8624067830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:49.127 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.119+0000 7f863d877640 1 --2- 172.21.15.16:0/566015296 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f8624065380 0x7f8624067830 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f862c0af000 tx=0x7f862c0a7000 comp rx=0 tx=0).stop 2023-05-22T15:27:49.127 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.119+0000 7f863d877640 1 -- 172.21.15.16:0/566015296 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f863806e740 msgr2=0x7f86380ce680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:49.127 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.119+0000 7f863d877640 1 --2- 172.21.15.16:0/566015296 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f863806e740 0x7f86380ce680 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f862800c7f0 tx=0x7f862800ccb0 comp rx=0 tx=0).stop 2023-05-22T15:27:49.127 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.119+0000 7f863d877640 1 -- 172.21.15.16:0/566015296 shutdown_connections 2023-05-22T15:27:49.128 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.119+0000 7f863d877640 1 --2- 172.21.15.16:0/566015296 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f8624065380 0x7f8624067830 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:49.128 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.119+0000 7f863d877640 1 --2- 172.21.15.16:0/566015296 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f863815cfa0 0x7f86380b3cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:49.128 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.119+0000 7f863d877640 1 --2- 172.21.15.16:0/566015296 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f86380cf870 0x7f8638149cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:49.128 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.119+0000 7f863d877640 1 --2- 172.21.15.16:0/566015296 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f863806e740 0x7f86380ce680 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:49.128 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.119+0000 7f863d877640 1 -- 172.21.15.16:0/566015296 >> 172.21.15.16:0/566015296 conn(0x7f86380cc870 msgr2=0x7f86381546d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:49.128 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.119+0000 7f863d877640 1 -- 172.21.15.16:0/566015296 shutdown_connections 2023-05-22T15:27:49.128 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.119+0000 7f863d877640 1 -- 172.21.15.16:0/566015296 wait complete. 2023-05-22T15:27:49.143 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-05-22T15:26:38.019781+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-05-22T15:26:43.570514+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2023-05-22T15:26:47.879159+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2023-05-22T15:26:48.522130+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-05-22T15:26:51.616142+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '27', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-05-22T15:27:49.145 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-05-22T15:27:49.264 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.255+0000 7f84dcd5e640 1 Processor -- start 2023-05-22T15:27:49.264 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.255+0000 7f84dcd5e640 1 -- start start 2023-05-22T15:27:49.265 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.259+0000 7f84dcd5e640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f84d8159970 0x7f84d8159d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:49.265 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.259+0000 7f84dcd5e640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84d8070c30 con 0x7f84d815dad0 2023-05-22T15:27:49.265 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.259+0000 7f84dcd5e640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84d8070e60 con 0x7f84d815d680 2023-05-22T15:27:49.266 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.259+0000 7f84dcd5e640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84d80cb970 con 0x7f84d8159970 2023-05-22T15:27:49.266 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.259+0000 7f84d6d9d640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f84d815dad0 0x7f84d806b6c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:32942/0 (socket says 172.21.15.16:32942) 2023-05-22T15:27:49.266 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.259+0000 7f84d6d9d640 1 -- 172.21.15.16:0/2546108350 learned_addr learned my addr 172.21.15.16:0/2546108350 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:27:49.266 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.259+0000 7f84d659c640 1 --2- 172.21.15.16:0/2546108350 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f84d8159970 0x7f84d8159d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:49.267 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.259+0000 7f84d559a640 1 -- 172.21.15.16:0/2546108350 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3351805348 0 0) 0x7f84d8070c30 con 0x7f84d815dad0 2023-05-22T15:27:49.267 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.259+0000 7f84d559a640 1 -- 172.21.15.16:0/2546108350 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f84c0003530 con 0x7f84d815dad0 2023-05-22T15:27:49.269 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.259+0000 7f84d559a640 1 -- 172.21.15.16:0/2546108350 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1378798075 0 0) 0x7f84c0003530 con 0x7f84d815dad0 2023-05-22T15:27:49.270 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.259+0000 7f84d559a640 1 -- 172.21.15.16:0/2546108350 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f84d8159970 msgr2=0x7f84d8159d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:49.270 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.259+0000 7f84d559a640 1 --2- 172.21.15.16:0/2546108350 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f84d8159970 0x7f84d8159d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:49.270 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.259+0000 7f84d559a640 1 -- 172.21.15.16:0/2546108350 >> v1:172.21.15.62:6789/0 conn(0x7f84d815d680 legacy=0x7f84d80cb260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:49.270 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.259+0000 7f84d559a640 1 -- 172.21.15.16:0/2546108350 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84d8071140 con 0x7f84d815dad0 2023-05-22T15:27:49.270 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.259+0000 7f84d559a640 1 -- 172.21.15.16:0/2546108350 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7f84d0001a00 con 0x7f84d815dad0 2023-05-22T15:27:49.270 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.259+0000 7f84d559a640 1 -- 172.21.15.16:0/2546108350 >> v1:172.21.15.16:6789/0 conn(0x7f84d815dad0 legacy=0x7f84d806b6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:49.271 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.259+0000 7f84d559a640 1 --2- 172.21.15.16:0/2546108350 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f84c00039b0 0x7f84c0003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:49.271 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.259+0000 7f84d559a640 1 --2- 172.21.15.16:0/2546108350 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f84d8159970 0x7f84c00047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:49.271 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.259+0000 7f84d559a640 1 --2- 172.21.15.16:0/2546108350 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f84c0004d10 0x7f84c0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:49.271 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.259+0000 7f84d559a640 1 -- 172.21.15.16:0/2546108350 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84d80cb970 con 0x7f84c00039b0 2023-05-22T15:27:49.271 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.259+0000 7f84d559a640 1 -- 172.21.15.16:0/2546108350 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84c000baa0 con 0x7f84c0004d10 2023-05-22T15:27:49.272 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.259+0000 7f84d559a640 1 -- 172.21.15.16:0/2546108350 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84c000bc60 con 0x7f84d8159970 2023-05-22T15:27:49.272 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.259+0000 7f84d659c640 1 --2- 172.21.15.16:0/2546108350 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f84c00039b0 0x7f84c0003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:49.272 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.259+0000 7f84d5d9b640 1 --2- 172.21.15.16:0/2546108350 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f84d8159970 0x7f84c00047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:49.272 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.259+0000 7f84d6d9d640 1 --2- 172.21.15.16:0/2546108350 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f84c0004d10 0x7f84c0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:49.272 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.259+0000 7f84d659c640 1 -- 172.21.15.16:0/2546108350 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f84d8159970 msgr2=0x7f84c00047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:49.272 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.259+0000 7f84d659c640 1 --2- 172.21.15.16:0/2546108350 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f84d8159970 0x7f84c00047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:49.272 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.259+0000 7f84d659c640 1 -- 172.21.15.16:0/2546108350 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f84c0004d10 msgr2=0x7f84c0005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:49.273 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.259+0000 7f84d659c640 1 --2- 172.21.15.16:0/2546108350 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f84c0004d10 0x7f84c0005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:49.273 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.259+0000 7f84d659c640 1 -- 172.21.15.16:0/2546108350 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84c0003530 con 0x7f84c00039b0 2023-05-22T15:27:49.273 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.259+0000 7f84d5d9b640 1 --2- 172.21.15.16:0/2546108350 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f84d8159970 0x7f84c00047d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:27:49.273 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.259+0000 7f84d6d9d640 1 --2- 172.21.15.16:0/2546108350 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f84c0004d10 0x7f84c0005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:27:49.273 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.259+0000 7f84d659c640 1 --2- 172.21.15.16:0/2546108350 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f84c00039b0 0x7f84c0003dc0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f84c8009ae0 tx=0x7f84c801ca20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8239a5f72499af85 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:49.273 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.259+0000 7f84d559a640 1 -- 172.21.15.16:0/2546108350 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84c80a55f0 con 0x7f84c00039b0 2023-05-22T15:27:49.273 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.259+0000 7f84d559a640 1 -- 172.21.15.16:0/2546108350 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84c80a5750 con 0x7f84c00039b0 2023-05-22T15:27:49.274 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.259+0000 7f84d559a640 1 -- 172.21.15.16:0/2546108350 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84c80a5b00 con 0x7f84c00039b0 2023-05-22T15:27:49.274 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.263+0000 7f84dcd5e640 1 -- 172.21.15.16:0/2546108350 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f84c00039b0 msgr2=0x7f84c0003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:49.274 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.263+0000 7f84dcd5e640 1 --2- 172.21.15.16:0/2546108350 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f84c00039b0 0x7f84c0003dc0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f84c8009ae0 tx=0x7f84c801ca20 comp rx=0 tx=0).stop 2023-05-22T15:27:49.274 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.263+0000 7f84d6d9d640 1 -- 172.21.15.16:0/2546108350 reap_dead start 2023-05-22T15:27:49.274 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.263+0000 7f84dcd5e640 1 -- 172.21.15.16:0/2546108350 shutdown_connections 2023-05-22T15:27:49.274 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.263+0000 7f84dcd5e640 1 -- 172.21.15.16:0/2546108350 >> 172.21.15.16:0/2546108350 conn(0x7f84d8069720 msgr2=0x7f84d8069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:49.274 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.263+0000 7f84dcd5e640 1 -- 172.21.15.16:0/2546108350 shutdown_connections 2023-05-22T15:27:49.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.263+0000 7f84dcd5e640 1 -- 172.21.15.16:0/2546108350 wait complete. 2023-05-22T15:27:49.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.263+0000 7f84dcd5e640 1 Processor -- start 2023-05-22T15:27:49.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.263+0000 7f84dcd5e640 1 -- start start 2023-05-22T15:27:49.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.263+0000 7f84dcd5e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f84c00039b0 0x7f84d80cd080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:49.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.263+0000 7f84dcd5e640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f84d814c8f0 0x7f84d80cbbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:49.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.263+0000 7f84dcd5e640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f84d80cc120 0x7f84d80cc590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:49.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.263+0000 7f84dcd5e640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84d80cd5c0 con 0x7f84c00039b0 2023-05-22T15:27:49.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.263+0000 7f84dcd5e640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84d80c4350 con 0x7f84d814c8f0 2023-05-22T15:27:49.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.263+0000 7f84dcd5e640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84d80a77f0 con 0x7f84d80cc120 2023-05-22T15:27:49.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.263+0000 7f84d659c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f84c00039b0 0x7f84d80cd080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:49.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.263+0000 7f84d659c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f84c00039b0 0x7f84d80cd080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53960/0 (socket says 172.21.15.16:53960) 2023-05-22T15:27:49.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.263+0000 7f84d659c640 1 -- 172.21.15.16:0/1838196982 learned_addr learned my addr 172.21.15.16:0/1838196982 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:49.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.263+0000 7f84d5d9b640 1 --2- 172.21.15.16:0/1838196982 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f84d814c8f0 0x7f84d80cbbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:49.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.263+0000 7f84d659c640 1 -- 172.21.15.16:0/1838196982 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f84d80cc120 msgr2=0x7f84d80cc590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:49.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.263+0000 7f84d659c640 1 --2- 172.21.15.16:0/1838196982 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f84d80cc120 0x7f84d80cc590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:49.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.263+0000 7f84d659c640 1 -- 172.21.15.16:0/1838196982 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f84d814c8f0 msgr2=0x7f84d80cbbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:49.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.263+0000 7f84d659c640 1 --2- 172.21.15.16:0/1838196982 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f84d814c8f0 0x7f84d80cbbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:49.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.263+0000 7f84d659c640 1 -- 172.21.15.16:0/1838196982 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84d80cae20 con 0x7f84c00039b0 2023-05-22T15:27:49.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.263+0000 7f84d5d9b640 1 --2- 172.21.15.16:0/1838196982 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f84d814c8f0 0x7f84d80cbbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:27:49.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.263+0000 7f84d659c640 1 --2- 172.21.15.16:0/1838196982 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f84c00039b0 0x7f84d80cd080 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f84c8009ae0 tx=0x7f84c0005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:49.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.263+0000 7f84c6ffd640 1 -- 172.21.15.16:0/1838196982 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84c80be070 con 0x7f84c00039b0 2023-05-22T15:27:49.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.263+0000 7f84c6ffd640 1 -- 172.21.15.16:0/1838196982 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84c800f000 con 0x7f84c00039b0 2023-05-22T15:27:49.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.263+0000 7f84dcd5e640 1 -- 172.21.15.16:0/1838196982 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84d80cac90 con 0x7f84c00039b0 2023-05-22T15:27:49.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.263+0000 7f84c6ffd640 1 -- 172.21.15.16:0/1838196982 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84c80a55f0 con 0x7f84c00039b0 2023-05-22T15:27:49.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.263+0000 7f84dcd5e640 1 -- 172.21.15.16:0/1838196982 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84d806bdd0 con 0x7f84c00039b0 2023-05-22T15:27:49.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.263+0000 7f84dcd5e640 1 -- 172.21.15.16:0/1838196982 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84d804dbe0 con 0x7f84c00039b0 2023-05-22T15:27:49.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.267+0000 7f84c6ffd640 1 -- 172.21.15.16:0/1838196982 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7f84c80b4040 con 0x7f84c00039b0 2023-05-22T15:27:49.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.267+0000 7f84c6ffd640 1 --2- 172.21.15.16:0/1838196982 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f84b8065380 0x7f84b8067830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:49.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.267+0000 7f84c6ffd640 1 -- 172.21.15.16:0/1838196982 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f84c811c430 con 0x7f84c00039b0 2023-05-22T15:27:49.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.267+0000 7f84d5d9b640 1 --2- 172.21.15.16:0/1838196982 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f84b8065380 0x7f84b8067830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:49.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.267+0000 7f84d5d9b640 1 --2- 172.21.15.16:0/1838196982 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f84b8065380 0x7f84b8067830 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f84d80c1700 tx=0x7f84cc009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:49.282 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.271+0000 7f84c6ffd640 1 -- 172.21.15.16:0/1838196982 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f84c80e9d70 con 0x7f84c00039b0 2023-05-22T15:27:49.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.423+0000 7f84dcd5e640 1 -- 172.21.15.16:0/1838196982 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f84d8157c20 con 0x7f84c00039b0 2023-05-22T15:27:49.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.423+0000 7f84c6ffd640 1 -- 172.21.15.16:0/1838196982 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v29) v1 ==== 93+0+10 (secure 0 0 0) 0x7f84c80ed390 con 0x7f84c00039b0 2023-05-22T15:27:49.431 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2023-05-22T15:27:49.433 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.423+0000 7f84dcd5e640 1 -- 172.21.15.16:0/1838196982 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f84b8065380 msgr2=0x7f84b8067830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:49.433 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.423+0000 7f84dcd5e640 1 --2- 172.21.15.16:0/1838196982 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f84b8065380 0x7f84b8067830 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f84d80c1700 tx=0x7f84cc009040 comp rx=0 tx=0).stop 2023-05-22T15:27:49.433 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.423+0000 7f84dcd5e640 1 -- 172.21.15.16:0/1838196982 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f84c00039b0 msgr2=0x7f84d80cd080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:49.434 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.423+0000 7f84dcd5e640 1 --2- 172.21.15.16:0/1838196982 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f84c00039b0 0x7f84d80cd080 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f84c8009ae0 tx=0x7f84c0005800 comp rx=0 tx=0).stop 2023-05-22T15:27:49.434 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.423+0000 7f84dcd5e640 1 -- 172.21.15.16:0/1838196982 shutdown_connections 2023-05-22T15:27:49.434 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.423+0000 7f84dcd5e640 1 --2- 172.21.15.16:0/1838196982 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f84b8065380 0x7f84b8067830 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:49.434 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.423+0000 7f84dcd5e640 1 --2- 172.21.15.16:0/1838196982 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f84d80cc120 0x7f84d80cc590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:49.434 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.423+0000 7f84dcd5e640 1 --2- 172.21.15.16:0/1838196982 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f84d814c8f0 0x7f84d80cbbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:49.434 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.423+0000 7f84dcd5e640 1 --2- 172.21.15.16:0/1838196982 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f84c00039b0 0x7f84d80cd080 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:49.435 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.423+0000 7f84dcd5e640 1 -- 172.21.15.16:0/1838196982 >> 172.21.15.16:0/1838196982 conn(0x7f84d8069720 msgr2=0x7f84d81554a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:49.435 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.423+0000 7f84dcd5e640 1 -- 172.21.15.16:0/1838196982 shutdown_connections 2023-05-22T15:27:49.435 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.427+0000 7f84dcd5e640 1 -- 172.21.15.16:0/1838196982 wait complete. 2023-05-22T15:27:49.449 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-05-22T15:27:49.565 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.559+0000 7f6df68a7640 1 Processor -- start 2023-05-22T15:27:49.566 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.559+0000 7f6df68a7640 1 -- start start 2023-05-22T15:27:49.567 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.559+0000 7f6df68a7640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6df015bca0 0x7f6df00cda80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:49.567 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.559+0000 7f6df68a7640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f6df00d2480 0x7f6df00d2900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:49.567 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.559+0000 7f6df68a7640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6df00cccc0 con 0x7f6df00b5340 2023-05-22T15:27:49.567 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.559+0000 7f6df68a7640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6df00cd660 con 0x7f6df015bca0 2023-05-22T15:27:49.567 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.559+0000 7f6df68a7640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6df00d2e40 con 0x7f6df00d2480 2023-05-22T15:27:49.568 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.559+0000 7f6deffff640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6df015bca0 0x7f6df00cda80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:49.568 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.559+0000 7f6deffff640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6df015bca0 0x7f6df00cda80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53976/0 (socket says 172.21.15.16:53976) 2023-05-22T15:27:49.568 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.559+0000 7f6deffff640 1 -- 172.21.15.16:0/650791146 learned_addr learned my addr 172.21.15.16:0/650791146 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:49.568 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.559+0000 7f6df4e44640 1 --1- 172.21.15.16:0/650791146 >> v1:172.21.15.16:6789/0 conn(0x7f6df00b5340 0x7f6df00b5790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:32944/0 (socket says 172.21.15.16:32944) 2023-05-22T15:27:49.568 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.559+0000 7f6def7fe640 1 --2- 172.21.15.16:0/650791146 >> v2:172.21.15.62:3300/0 conn(0x7f6df00d2480 0x7f6df00d2900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:49.568 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.559+0000 7f6deffff640 1 -- 172.21.15.16:0/650791146 >> v2:172.21.15.62:3300/0 conn(0x7f6df00d2480 msgr2=0x7f6df00d2900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:49.568 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.559+0000 7f6deffff640 1 --2- 172.21.15.16:0/650791146 >> v2:172.21.15.62:3300/0 conn(0x7f6df00d2480 0x7f6df00d2900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:49.569 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.559+0000 7f6deffff640 1 -- 172.21.15.16:0/650791146 >> v1:172.21.15.16:6789/0 conn(0x7f6df00b5340 legacy=0x7f6df00b5790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:49.569 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.559+0000 7f6deffff640 1 -- 172.21.15.16:0/650791146 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6df00ccfa0 con 0x7f6df015bca0 2023-05-22T15:27:49.569 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.559+0000 7f6def7fe640 1 --2- 172.21.15.16:0/650791146 >> v2:172.21.15.62:3300/0 conn(0x7f6df00d2480 0x7f6df00d2900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T15:27:49.570 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.563+0000 7f6deffff640 1 --2- 172.21.15.16:0/650791146 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6df015bca0 0x7f6df00cda80 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f6df0065e80 tx=0x7f6de001c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65ecbc859ae87fe2 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:49.570 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.563+0000 7f6deeffd640 1 -- 172.21.15.16:0/650791146 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6de00aa680 con 0x7f6df015bca0 2023-05-22T15:27:49.570 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.563+0000 7f6deeffd640 1 -- 172.21.15.16:0/650791146 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6de00aa7e0 con 0x7f6df015bca0 2023-05-22T15:27:49.570 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.563+0000 7f6deeffd640 1 -- 172.21.15.16:0/650791146 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6de00aab70 con 0x7f6df015bca0 2023-05-22T15:27:49.571 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.563+0000 7f6df68a7640 1 -- 172.21.15.16:0/650791146 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6df015bca0 msgr2=0x7f6df00cda80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:49.571 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.563+0000 7f6df68a7640 1 --2- 172.21.15.16:0/650791146 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6df015bca0 0x7f6df00cda80 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f6df0065e80 tx=0x7f6de001c500 comp rx=0 tx=0).stop 2023-05-22T15:27:49.571 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.563+0000 7f6df68a7640 1 -- 172.21.15.16:0/650791146 shutdown_connections 2023-05-22T15:27:49.571 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.563+0000 7f6df68a7640 1 --2- 172.21.15.16:0/650791146 >> v2:172.21.15.62:3300/0 conn(0x7f6df00d2480 0x7f6df00d2900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:49.571 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.563+0000 7f6df68a7640 1 --2- 172.21.15.16:0/650791146 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6df015bca0 0x7f6df00cda80 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:49.572 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.563+0000 7f6df68a7640 1 -- 172.21.15.16:0/650791146 >> 172.21.15.16:0/650791146 conn(0x7f6df00655c0 msgr2=0x7f6df0070e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:49.572 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.563+0000 7f6df68a7640 1 -- 172.21.15.16:0/650791146 shutdown_connections 2023-05-22T15:27:49.572 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.563+0000 7f6df68a7640 1 -- 172.21.15.16:0/650791146 wait complete. 2023-05-22T15:27:49.573 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.563+0000 7f6df68a7640 1 Processor -- start 2023-05-22T15:27:49.574 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.563+0000 7f6df68a7640 1 -- start start 2023-05-22T15:27:49.574 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.563+0000 7f6df68a7640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6df00b5340 0x7f6df015c330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:49.574 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.563+0000 7f6df68a7640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6df00d2480 0x7f6df00b7830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:49.574 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.563+0000 7f6df68a7640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6df015bca0 0x7f6df00c66b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:49.574 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.563+0000 7f6df68a7640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6df015a7d0 con 0x7f6df015bca0 2023-05-22T15:27:49.574 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.563+0000 7f6df68a7640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6df0152ef0 con 0x7f6df00b5340 2023-05-22T15:27:49.574 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.563+0000 7f6df68a7640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6df01531d0 con 0x7f6df00d2480 2023-05-22T15:27:49.575 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.563+0000 7f6df4e44640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6df015bca0 0x7f6df00c66b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:49.575 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.563+0000 7f6df4e44640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6df015bca0 0x7f6df00c66b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53982/0 (socket says 172.21.15.16:53982) 2023-05-22T15:27:49.575 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.563+0000 7f6df4e44640 1 -- 172.21.15.16:0/2241531118 learned_addr learned my addr 172.21.15.16:0/2241531118 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:49.575 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.567+0000 7f6df4e44640 1 -- 172.21.15.16:0/2241531118 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6df00d2480 msgr2=0x7f6df00b7830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:49.575 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.567+0000 7f6deffff640 1 --2- 172.21.15.16:0/2241531118 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6df00b5340 0x7f6df015c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:49.575 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.567+0000 7f6def7fe640 1 --2- 172.21.15.16:0/2241531118 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6df00d2480 0x7f6df00b7830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:49.575 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.567+0000 7f6df4e44640 1 --2- 172.21.15.16:0/2241531118 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6df00d2480 0x7f6df00b7830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:49.576 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.567+0000 7f6df4e44640 1 -- 172.21.15.16:0/2241531118 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6df00b5340 msgr2=0x7f6df015c330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:49.576 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.567+0000 7f6df4e44640 1 --2- 172.21.15.16:0/2241531118 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6df00b5340 0x7f6df015c330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:49.576 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.567+0000 7f6df4e44640 1 -- 172.21.15.16:0/2241531118 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6de00094d0 con 0x7f6df015bca0 2023-05-22T15:27:49.576 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.567+0000 7f6deffff640 1 --2- 172.21.15.16:0/2241531118 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6df00b5340 0x7f6df015c330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:27:49.576 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.567+0000 7f6df4e44640 1 --2- 172.21.15.16:0/2241531118 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6df015bca0 0x7f6df00c66b0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f6de800c700 tx=0x7f6de800cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:49.576 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.567+0000 7f6decff9640 1 -- 172.21.15.16:0/2241531118 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6de800ce10 con 0x7f6df015bca0 2023-05-22T15:27:49.576 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.567+0000 7f6decff9640 1 -- 172.21.15.16:0/2241531118 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6de800a640 con 0x7f6df015bca0 2023-05-22T15:27:49.577 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.567+0000 7f6decff9640 1 -- 172.21.15.16:0/2241531118 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6de800a9b0 con 0x7f6df015bca0 2023-05-22T15:27:49.577 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.567+0000 7f6df68a7640 1 -- 172.21.15.16:0/2241531118 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6df0160360 con 0x7f6df015bca0 2023-05-22T15:27:49.577 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.567+0000 7f6df68a7640 1 -- 172.21.15.16:0/2241531118 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6df00cdfc0 con 0x7f6df015bca0 2023-05-22T15:27:49.578 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.567+0000 7f6df68a7640 1 -- 172.21.15.16:0/2241531118 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6df01604f0 con 0x7f6df015bca0 2023-05-22T15:27:49.578 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.567+0000 7f6decff9640 1 -- 172.21.15.16:0/2241531118 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7f6de800ab10 con 0x7f6df015bca0 2023-05-22T15:27:49.578 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.567+0000 7f6decff9640 1 --2- 172.21.15.16:0/2241531118 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f6ddc065380 0x7f6ddc067830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:49.578 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.567+0000 7f6decff9640 1 -- 172.21.15.16:0/2241531118 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f6de80823b0 con 0x7f6df015bca0 2023-05-22T15:27:49.578 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.567+0000 7f6deffff640 1 --2- 172.21.15.16:0/2241531118 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f6ddc065380 0x7f6ddc067830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:49.578 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.567+0000 7f6deffff640 1 --2- 172.21.15.16:0/2241531118 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f6ddc065380 0x7f6ddc067830 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f6de00ae000 tx=0x7f6de0008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:49.579 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.571+0000 7f6decff9640 1 -- 172.21.15.16:0/2241531118 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f6de804a620 con 0x7f6df015bca0 2023-05-22T15:27:49.736 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.727+0000 7f6df68a7640 1 -- 172.21.15.16:0/2241531118 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f6df004dbe0 con 0x7f6df015bca0 2023-05-22T15:27:49.737 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.727+0000 7f6decff9640 1 -- 172.21.15.16:0/2241531118 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v29) v1 ==== 92+0+10 (secure 0 0 0) 0x7f6de80170a0 con 0x7f6df015bca0 2023-05-22T15:27:49.737 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 8 2023-05-22T15:27:49.739 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.731+0000 7f6df68a7640 1 -- 172.21.15.16:0/2241531118 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f6ddc065380 msgr2=0x7f6ddc067830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:49.739 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.731+0000 7f6df68a7640 1 --2- 172.21.15.16:0/2241531118 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f6ddc065380 0x7f6ddc067830 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f6de00ae000 tx=0x7f6de0008040 comp rx=0 tx=0).stop 2023-05-22T15:27:49.739 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.731+0000 7f6df68a7640 1 -- 172.21.15.16:0/2241531118 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6df015bca0 msgr2=0x7f6df00c66b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:49.739 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.731+0000 7f6df68a7640 1 --2- 172.21.15.16:0/2241531118 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6df015bca0 0x7f6df00c66b0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f6de800c700 tx=0x7f6de800cbc0 comp rx=0 tx=0).stop 2023-05-22T15:27:49.739 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.731+0000 7f6df68a7640 1 -- 172.21.15.16:0/2241531118 shutdown_connections 2023-05-22T15:27:49.740 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.731+0000 7f6df68a7640 1 --2- 172.21.15.16:0/2241531118 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6df015bca0 0x7f6df00c66b0 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:49.740 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.731+0000 7f6df68a7640 1 --2- 172.21.15.16:0/2241531118 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f6ddc065380 0x7f6ddc067830 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:49.740 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.731+0000 7f6df68a7640 1 --2- 172.21.15.16:0/2241531118 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6df00d2480 0x7f6df00b7830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:49.740 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.731+0000 7f6df68a7640 1 --2- 172.21.15.16:0/2241531118 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6df00b5340 0x7f6df015c330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:49.740 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.731+0000 7f6df68a7640 1 -- 172.21.15.16:0/2241531118 >> 172.21.15.16:0/2241531118 conn(0x7f6df00655c0 msgr2=0x7f6df01541c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:49.740 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.731+0000 7f6df68a7640 1 -- 172.21.15.16:0/2241531118 shutdown_connections 2023-05-22T15:27:49.740 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.731+0000 7f6df68a7640 1 -- 172.21.15.16:0/2241531118 wait complete. 2023-05-22T15:27:49.756 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-05-22T15:27:49.884 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.875+0000 7f0eef182640 1 Processor -- start 2023-05-22T15:27:49.884 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.875+0000 7f0eef182640 1 -- start start 2023-05-22T15:27:49.886 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.879+0000 7f0eef182640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0ee80c67d0 0x7f0ee80cf730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:49.886 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.879+0000 7f0eef182640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f0ee80b2880 0x7f0ee80cbfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:49.886 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.879+0000 7f0eef182640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ee80cfc70 con 0x7f0ee80cc520 2023-05-22T15:27:49.886 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.879+0000 7f0eef182640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ee815a300 con 0x7f0ee80b2880 2023-05-22T15:27:49.886 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.879+0000 7f0eef182640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ee80c4dc0 con 0x7f0ee80c67d0 2023-05-22T15:27:49.887 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.879+0000 7f0ee7fff640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f0ee80b2880 0x7f0ee80cbfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:49.887 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.879+0000 7f0eed71f640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f0ee80cc520 0x7f0ee80ce160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:32952/0 (socket says 172.21.15.16:32952) 2023-05-22T15:27:49.887 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.879+0000 7f0eed71f640 1 -- 172.21.15.16:0/1840446798 learned_addr learned my addr 172.21.15.16:0/1840446798 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:27:49.887 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.879+0000 7f0ee7fff640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f0ee80b2880 0x7f0ee80cbfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53994/0 (socket says 172.21.15.16:53994) 2023-05-22T15:27:49.887 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.879+0000 7f0eecf1e640 1 --2- 172.21.15.16:0/1840446798 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0ee80c67d0 0x7f0ee80cf730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:49.887 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.879+0000 7f0ee7fff640 1 -- 172.21.15.16:0/1840446798 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0ee80c67d0 msgr2=0x7f0ee80cf730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:49.887 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.879+0000 7f0ee7fff640 1 --2- 172.21.15.16:0/1840446798 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0ee80c67d0 0x7f0ee80cf730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:49.887 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.879+0000 7f0ee7fff640 1 -- 172.21.15.16:0/1840446798 >> v1:172.21.15.16:6789/0 conn(0x7f0ee80cc520 legacy=0x7f0ee80ce160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:49.888 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.879+0000 7f0ee7fff640 1 -- 172.21.15.16:0/1840446798 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ee80c6ba0 con 0x7f0ee80b2880 2023-05-22T15:27:49.888 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.879+0000 7f0ee77fe640 1 -- 172.21.15.16:0/1840446798 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2875468173 0 0) 0x7f0ee80cfc70 con 0x7f0ee80cc520 2023-05-22T15:27:49.889 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.883+0000 7f0ee7fff640 1 --2- 172.21.15.16:0/1840446798 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ee80b2880 0x7f0ee80cbfe0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f0ed40097c0 tx=0x7f0ed401c3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e0cd97943006a19 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:49.890 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.883+0000 7f0ee77fe640 1 -- 172.21.15.16:0/1840446798 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ed40ab600 con 0x7f0ee80b2880 2023-05-22T15:27:49.890 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.883+0000 7f0ee77fe640 1 -- 172.21.15.16:0/1840446798 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ed40a7050 con 0x7f0ee80b2880 2023-05-22T15:27:49.890 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.883+0000 7f0ee77fe640 1 -- 172.21.15.16:0/1840446798 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ed40ab980 con 0x7f0ee80b2880 2023-05-22T15:27:49.890 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.883+0000 7f0eef182640 1 -- 172.21.15.16:0/1840446798 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ee80b2880 msgr2=0x7f0ee80cbfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:49.890 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.883+0000 7f0eef182640 1 --2- 172.21.15.16:0/1840446798 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ee80b2880 0x7f0ee80cbfe0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f0ed40097c0 tx=0x7f0ed401c3c0 comp rx=0 tx=0).stop 2023-05-22T15:27:49.890 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.883+0000 7f0eef182640 1 -- 172.21.15.16:0/1840446798 shutdown_connections 2023-05-22T15:27:49.890 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.883+0000 7f0eef182640 1 --2- 172.21.15.16:0/1840446798 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ee80b2880 0x7f0ee80cbfe0 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:49.891 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.883+0000 7f0eef182640 1 --2- 172.21.15.16:0/1840446798 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0ee80c67d0 0x7f0ee80cf730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:49.891 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.883+0000 7f0eef182640 1 -- 172.21.15.16:0/1840446798 >> 172.21.15.16:0/1840446798 conn(0x7f0ee806f5a0 msgr2=0x7f0ee806f9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:49.891 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.883+0000 7f0eef182640 1 -- 172.21.15.16:0/1840446798 shutdown_connections 2023-05-22T15:27:49.891 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.883+0000 7f0eef182640 1 -- 172.21.15.16:0/1840446798 wait complete. 2023-05-22T15:27:49.893 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.883+0000 7f0eef182640 1 Processor -- start 2023-05-22T15:27:49.893 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.883+0000 7f0eef182640 1 -- start start 2023-05-22T15:27:49.893 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.883+0000 7f0eef182640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0ee80b2880 0x7f0ee8155320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:49.893 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.883+0000 7f0eef182640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ee80c67d0 0x7f0ee8152380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:49.893 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.883+0000 7f0eef182640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0ee80cc520 0x7f0ee81528c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:49.893 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.883+0000 7f0eef182640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ee81f2b70 con 0x7f0ee80c67d0 2023-05-22T15:27:49.894 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.883+0000 7f0eef182640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ee81f1ef0 con 0x7f0ee80cc520 2023-05-22T15:27:49.894 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.883+0000 7f0eef182640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ee81f21d0 con 0x7f0ee80b2880 2023-05-22T15:27:49.894 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.883+0000 7f0ee7fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ee80c67d0 0x7f0ee8152380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:49.894 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.883+0000 7f0ee7fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ee80c67d0 0x7f0ee8152380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54004/0 (socket says 172.21.15.16:54004) 2023-05-22T15:27:49.894 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.883+0000 7f0ee7fff640 1 -- 172.21.15.16:0/2381555444 learned_addr learned my addr 172.21.15.16:0/2381555444 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:49.894 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.883+0000 7f0eed71f640 1 --2- 172.21.15.16:0/2381555444 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0ee80cc520 0x7f0ee81528c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:49.894 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.883+0000 7f0eecf1e640 1 --2- 172.21.15.16:0/2381555444 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0ee80b2880 0x7f0ee8155320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:49.894 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.883+0000 7f0ee7fff640 1 -- 172.21.15.16:0/2381555444 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0ee80b2880 msgr2=0x7f0ee8155320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:49.895 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.883+0000 7f0ee7fff640 1 --2- 172.21.15.16:0/2381555444 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0ee80b2880 0x7f0ee8155320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:49.895 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.883+0000 7f0ee7fff640 1 -- 172.21.15.16:0/2381555444 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0ee80cc520 msgr2=0x7f0ee81528c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:49.895 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.883+0000 7f0ee7fff640 1 --2- 172.21.15.16:0/2381555444 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0ee80cc520 0x7f0ee81528c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:49.895 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.883+0000 7f0ee7fff640 1 -- 172.21.15.16:0/2381555444 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ed4009420 con 0x7f0ee80c67d0 2023-05-22T15:27:49.895 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.883+0000 7f0ee7fff640 1 --2- 172.21.15.16:0/2381555444 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ee80c67d0 0x7f0ee8152380 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f0ed401aa70 tx=0x7f0ed4019160 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:49.895 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.883+0000 7f0ee57fa640 1 -- 172.21.15.16:0/2381555444 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ed4006e20 con 0x7f0ee80c67d0 2023-05-22T15:27:49.895 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.883+0000 7f0ee57fa640 1 -- 172.21.15.16:0/2381555444 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ed4004030 con 0x7f0ee80c67d0 2023-05-22T15:27:49.895 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.883+0000 7f0eef182640 1 -- 172.21.15.16:0/2381555444 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ee80d0790 con 0x7f0ee80c67d0 2023-05-22T15:27:49.896 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.883+0000 7f0ee57fa640 1 -- 172.21.15.16:0/2381555444 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ed40043a0 con 0x7f0ee80c67d0 2023-05-22T15:27:49.896 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.883+0000 7f0eef182640 1 -- 172.21.15.16:0/2381555444 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ee80d04b0 con 0x7f0ee80c67d0 2023-05-22T15:27:49.896 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.887+0000 7f0eef182640 1 -- 172.21.15.16:0/2381555444 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ee80d0a70 con 0x7f0ee80c67d0 2023-05-22T15:27:49.897 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.887+0000 7f0ee57fa640 1 -- 172.21.15.16:0/2381555444 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7f0ed40b2040 con 0x7f0ee80c67d0 2023-05-22T15:27:49.897 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.887+0000 7f0ee57fa640 1 --2- 172.21.15.16:0/2381555444 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f0ecc065380 0x7f0ecc067830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:49.897 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.887+0000 7f0ee57fa640 1 -- 172.21.15.16:0/2381555444 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f0ed411c970 con 0x7f0ee80c67d0 2023-05-22T15:27:49.897 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.887+0000 7f0eecf1e640 1 --2- 172.21.15.16:0/2381555444 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f0ecc065380 0x7f0ecc067830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:49.897 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.887+0000 7f0eecf1e640 1 --2- 172.21.15.16:0/2381555444 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f0ecc065380 0x7f0ecc067830 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f0ee80c7a10 tx=0x7f0edc008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:49.899 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:49.891+0000 7f0ee57fa640 1 -- 172.21.15.16:0/2381555444 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f0ed40ea350 con 0x7f0ee80c67d0 2023-05-22T15:27:50.053 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.047+0000 7f0eef182640 1 -- 172.21.15.16:0/2381555444 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f0ee81f2cb0 con 0x7f0ee80c67d0 2023-05-22T15:27:50.054 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.047+0000 7f0ee57fa640 1 -- 172.21.15.16:0/2381555444 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v29) v1 ==== 104+0+11 (secure 0 0 0) 0x7f0ed40b0020 con 0x7f0ee80c67d0 2023-05-22T15:27:50.054 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2023-05-22T15:27:50.057 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.047+0000 7f0eef182640 1 -- 172.21.15.16:0/2381555444 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f0ecc065380 msgr2=0x7f0ecc067830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:50.057 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.047+0000 7f0eef182640 1 --2- 172.21.15.16:0/2381555444 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f0ecc065380 0x7f0ecc067830 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f0ee80c7a10 tx=0x7f0edc008040 comp rx=0 tx=0).stop 2023-05-22T15:27:50.057 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.047+0000 7f0eef182640 1 -- 172.21.15.16:0/2381555444 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ee80c67d0 msgr2=0x7f0ee8152380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:50.057 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.047+0000 7f0eef182640 1 --2- 172.21.15.16:0/2381555444 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ee80c67d0 0x7f0ee8152380 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f0ed401aa70 tx=0x7f0ed4019160 comp rx=0 tx=0).stop 2023-05-22T15:27:50.057 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.047+0000 7f0eef182640 1 -- 172.21.15.16:0/2381555444 shutdown_connections 2023-05-22T15:27:50.057 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.047+0000 7f0eef182640 1 --2- 172.21.15.16:0/2381555444 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0ee80cc520 0x7f0ee81528c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:50.058 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.047+0000 7f0eef182640 1 --2- 172.21.15.16:0/2381555444 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ee80c67d0 0x7f0ee8152380 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:50.058 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.047+0000 7f0eef182640 1 --2- 172.21.15.16:0/2381555444 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f0ecc065380 0x7f0ecc067830 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:50.058 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.047+0000 7f0eef182640 1 --2- 172.21.15.16:0/2381555444 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0ee80b2880 0x7f0ee8155320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:50.058 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.047+0000 7f0eef182640 1 -- 172.21.15.16:0/2381555444 >> 172.21.15.16:0/2381555444 conn(0x7f0ee806f5a0 msgr2=0x7f0ee80ce160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:50.058 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.047+0000 7f0eef182640 1 -- 172.21.15.16:0/2381555444 shutdown_connections 2023-05-22T15:27:50.058 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.047+0000 7f0eef182640 1 -- 172.21.15.16:0/2381555444 wait complete. 2023-05-22T15:27:50.073 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-05-22T15:27:50.189 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.183+0000 7f66ee77b640 1 Processor -- start 2023-05-22T15:27:50.189 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.183+0000 7f66ee77b640 1 -- start start 2023-05-22T15:27:50.190 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.183+0000 7f66ee77b640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f66e80c67d0 0x7f66e80cf730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:50.190 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.183+0000 7f66ee77b640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f66e80cc6f0 0x7f66e80c3db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:50.191 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.183+0000 7f66ee77b640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66e80cfc70 con 0x7f66e80b2880 2023-05-22T15:27:50.191 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.183+0000 7f66ee77b640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66e815a300 con 0x7f66e80cc6f0 2023-05-22T15:27:50.191 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.183+0000 7f66ee77b640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66e80c6ba0 con 0x7f66e80c67d0 2023-05-22T15:27:50.191 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.183+0000 7f66ecd18640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f66e80cc6f0 0x7f66e80c3db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:50.191 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.183+0000 7f66ecd18640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f66e80cc6f0 0x7f66e80c3db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54006/0 (socket says 172.21.15.16:54006) 2023-05-22T15:27:50.191 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.183+0000 7f66ecd18640 1 -- 172.21.15.16:0/194271269 learned_addr learned my addr 172.21.15.16:0/194271269 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:50.191 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.183+0000 7f66e7fff640 1 --2- 172.21.15.16:0/194271269 >> v2:172.21.15.62:3300/0 conn(0x7f66e80c67d0 0x7f66e80cf730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:50.192 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.183+0000 7f66ecd18640 1 -- 172.21.15.16:0/194271269 >> v2:172.21.15.62:3300/0 conn(0x7f66e80c67d0 msgr2=0x7f66e80cf730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:50.192 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.183+0000 7f66ecd18640 1 --2- 172.21.15.16:0/194271269 >> v2:172.21.15.62:3300/0 conn(0x7f66e80c67d0 0x7f66e80cf730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:50.192 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.183+0000 7f66ecd18640 1 -- 172.21.15.16:0/194271269 >> v1:172.21.15.16:6789/0 conn(0x7f66e80b2880 legacy=0x7f66e80cbfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:50.192 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.183+0000 7f66ecd18640 1 -- 172.21.15.16:0/194271269 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66e80cfef0 con 0x7f66e80cc6f0 2023-05-22T15:27:50.193 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.183+0000 7f66ecd18640 1 --2- 172.21.15.16:0/194271269 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66e80cc6f0 0x7f66e80c3db0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f66e806eac0 tx=0x7f66e001c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ffc131524a1951d3 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:50.193 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.187+0000 7f66e77fe640 1 -- 172.21.15.16:0/194271269 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66e00a57d0 con 0x7f66e80cc6f0 2023-05-22T15:27:50.193 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.187+0000 7f66e77fe640 1 -- 172.21.15.16:0/194271269 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66e00a5930 con 0x7f66e80cc6f0 2023-05-22T15:27:50.194 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.187+0000 7f66e77fe640 1 -- 172.21.15.16:0/194271269 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66e00a5cc0 con 0x7f66e80cc6f0 2023-05-22T15:27:50.194 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.187+0000 7f66ee77b640 1 -- 172.21.15.16:0/194271269 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66e80cc6f0 msgr2=0x7f66e80c3db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:50.194 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.187+0000 7f66ee77b640 1 --2- 172.21.15.16:0/194271269 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66e80cc6f0 0x7f66e80c3db0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f66e806eac0 tx=0x7f66e001c910 comp rx=0 tx=0).stop 2023-05-22T15:27:50.195 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.187+0000 7f66ee77b640 1 -- 172.21.15.16:0/194271269 shutdown_connections 2023-05-22T15:27:50.195 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.187+0000 7f66ee77b640 1 --2- 172.21.15.16:0/194271269 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66e80cc6f0 0x7f66e80c3db0 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:50.195 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.187+0000 7f66ee77b640 1 --2- 172.21.15.16:0/194271269 >> v2:172.21.15.62:3300/0 conn(0x7f66e80c67d0 0x7f66e80cf730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:50.195 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.187+0000 7f66ee77b640 1 -- 172.21.15.16:0/194271269 >> 172.21.15.16:0/194271269 conn(0x7f66e806f5a0 msgr2=0x7f66e806f9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:50.195 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.187+0000 7f66ee77b640 1 -- 172.21.15.16:0/194271269 shutdown_connections 2023-05-22T15:27:50.195 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.187+0000 7f66ee77b640 1 -- 172.21.15.16:0/194271269 wait complete. 2023-05-22T15:27:50.197 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.187+0000 7f66ee77b640 1 Processor -- start 2023-05-22T15:27:50.197 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.187+0000 7f66ee77b640 1 -- start start 2023-05-22T15:27:50.197 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.187+0000 7f66ee77b640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f66e80b2880 0x7f66e81f5bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:50.197 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.187+0000 7f66ee77b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66e80c67d0 0x7f66e81529d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:50.197 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.187+0000 7f66ee77b640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f66e80cc6f0 0x7f66e80b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:50.198 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.187+0000 7f66ee77b640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66e80ca220 con 0x7f66e80c67d0 2023-05-22T15:27:50.198 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.187+0000 7f66ee77b640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66e81f6130 con 0x7f66e80cc6f0 2023-05-22T15:27:50.198 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.187+0000 7f66ee77b640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66e80c9610 con 0x7f66e80b2880 2023-05-22T15:27:50.198 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.187+0000 7f66df7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66e80c67d0 0x7f66e81529d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:50.198 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.187+0000 7f66df7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66e80c67d0 0x7f66e81529d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54010/0 (socket says 172.21.15.16:54010) 2023-05-22T15:27:50.198 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.187+0000 7f66df7fe640 1 -- 172.21.15.16:0/20054286 learned_addr learned my addr 172.21.15.16:0/20054286 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:50.198 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.187+0000 7f66ecd18640 1 --2- 172.21.15.16:0/20054286 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f66e80cc6f0 0x7f66e80b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:50.198 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.187+0000 7f66df7fe640 1 -- 172.21.15.16:0/20054286 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f66e80b2880 msgr2=0x7f66e81f5bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:50.199 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.187+0000 7f66e7fff640 1 --2- 172.21.15.16:0/20054286 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f66e80b2880 0x7f66e81f5bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:50.199 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.187+0000 7f66df7fe640 1 --2- 172.21.15.16:0/20054286 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f66e80b2880 0x7f66e81f5bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:50.199 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.187+0000 7f66df7fe640 1 -- 172.21.15.16:0/20054286 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f66e80cc6f0 msgr2=0x7f66e80b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:50.199 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.187+0000 7f66df7fe640 1 --2- 172.21.15.16:0/20054286 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f66e80cc6f0 0x7f66e80b04c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:50.199 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.187+0000 7f66df7fe640 1 -- 172.21.15.16:0/20054286 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66e0009b30 con 0x7f66e80c67d0 2023-05-22T15:27:50.199 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.187+0000 7f66ecd18640 1 --2- 172.21.15.16:0/20054286 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f66e80cc6f0 0x7f66e80b04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:27:50.199 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.187+0000 7f66df7fe640 1 --2- 172.21.15.16:0/20054286 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66e80c67d0 0x7f66e81529d0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f66d000c3c0 tx=0x7f66d000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:50.200 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.187+0000 7f66e57fa640 1 -- 172.21.15.16:0/20054286 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66d00052d0 con 0x7f66e80c67d0 2023-05-22T15:27:50.200 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.187+0000 7f66e57fa640 1 -- 172.21.15.16:0/20054286 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66d0005430 con 0x7f66e80c67d0 2023-05-22T15:27:50.200 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.187+0000 7f66ee77b640 1 -- 172.21.15.16:0/20054286 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66e80d0a70 con 0x7f66e80c67d0 2023-05-22T15:27:50.200 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.187+0000 7f66e57fa640 1 -- 172.21.15.16:0/20054286 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66d0003840 con 0x7f66e80c67d0 2023-05-22T15:27:50.200 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.187+0000 7f66ee77b640 1 -- 172.21.15.16:0/20054286 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66e80d0790 con 0x7f66e80c67d0 2023-05-22T15:27:50.201 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.191+0000 7f66ee77b640 1 -- 172.21.15.16:0/20054286 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66e80d0f90 con 0x7f66e80c67d0 2023-05-22T15:27:50.201 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.191+0000 7f66e57fa640 1 -- 172.21.15.16:0/20054286 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7f66d00039a0 con 0x7f66e80c67d0 2023-05-22T15:27:50.201 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.191+0000 7f66e57fa640 1 --2- 172.21.15.16:0/20054286 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f66c8065380 0x7f66c8067830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:50.201 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.191+0000 7f66e57fa640 1 -- 172.21.15.16:0/20054286 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f66d00820d0 con 0x7f66e80c67d0 2023-05-22T15:27:50.201 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.191+0000 7f66e7fff640 1 --2- 172.21.15.16:0/20054286 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f66c8065380 0x7f66c8067830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:50.202 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.191+0000 7f66e7fff640 1 --2- 172.21.15.16:0/20054286 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f66c8065380 0x7f66c8067830 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f66d8009990 tx=0x7f66d8008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:50.206 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.199+0000 7f66e57fa640 1 -- 172.21.15.16:0/20054286 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f66d004fa20 con 0x7f66e80c67d0 2023-05-22T15:27:50.337 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-05-22T15:27:50.339 DEBUG:teuthology.orchestra.run.smithi062:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-05-22T15:27:50.354 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.347+0000 7f66ee77b640 1 -- 172.21.15.16:0/20054286 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f66e804dbe0 con 0x7f66e80c67d0 2023-05-22T15:27:50.355 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.347+0000 7f66e57fa640 1 -- 172.21.15.16:0/20054286 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v29) v1 ==== 100+0+11 (secure 0 0 0) 0x7f66d0053040 con 0x7f66e80c67d0 2023-05-22T15:27:50.355 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 64 2023-05-22T15:27:50.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.347+0000 7f66ee77b640 1 -- 172.21.15.16:0/20054286 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f66c8065380 msgr2=0x7f66c8067830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:50.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.347+0000 7f66ee77b640 1 --2- 172.21.15.16:0/20054286 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f66c8065380 0x7f66c8067830 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f66d8009990 tx=0x7f66d8008040 comp rx=0 tx=0).stop 2023-05-22T15:27:50.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.347+0000 7f66ee77b640 1 -- 172.21.15.16:0/20054286 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66e80c67d0 msgr2=0x7f66e81529d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:50.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.347+0000 7f66ee77b640 1 --2- 172.21.15.16:0/20054286 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66e80c67d0 0x7f66e81529d0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f66d000c3c0 tx=0x7f66d000c880 comp rx=0 tx=0).stop 2023-05-22T15:27:50.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.347+0000 7f66ee77b640 1 -- 172.21.15.16:0/20054286 shutdown_connections 2023-05-22T15:27:50.358 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.347+0000 7f66ee77b640 1 --2- 172.21.15.16:0/20054286 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f66e80cc6f0 0x7f66e80b04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:50.358 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.347+0000 7f66ee77b640 1 --2- 172.21.15.16:0/20054286 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66e80c67d0 0x7f66e81529d0 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:50.358 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.347+0000 7f66ee77b640 1 --2- 172.21.15.16:0/20054286 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f66c8065380 0x7f66c8067830 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:50.358 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.347+0000 7f66ee77b640 1 --2- 172.21.15.16:0/20054286 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f66e80b2880 0x7f66e81f5bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:50.358 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.351+0000 7f66ee77b640 1 -- 172.21.15.16:0/20054286 >> 172.21.15.16:0/20054286 conn(0x7f66e806f5a0 msgr2=0x7f66e80b1870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:50.358 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.351+0000 7f66ee77b640 1 -- 172.21.15.16:0/20054286 shutdown_connections 2023-05-22T15:27:50.358 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.351+0000 7f66ee77b640 1 -- 172.21.15.16:0/20054286 wait complete. 2023-05-22T15:27:50.372 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-05-22T15:27:50.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.535+0000 7f55811e0640 1 Processor -- start 2023-05-22T15:27:50.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.535+0000 7f55811e0640 1 -- start start 2023-05-22T15:27:50.544 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.535+0000 7f55811e0640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f557c153fe0 0x7f557c06d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:50.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.535+0000 7f55811e0640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f557c0cae20 0x7f557c06fef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:50.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.535+0000 7f55811e0640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f557c069720 con 0x7f557c070430 2023-05-22T15:27:50.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.535+0000 7f55811e0640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f557c0708d0 con 0x7f557c153fe0 2023-05-22T15:27:50.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.535+0000 7f55811e0640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f557c1543b0 con 0x7f557c0cae20 2023-05-22T15:27:50.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.535+0000 7f557ad9d640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f557c153fe0 0x7f557c06d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:50.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.535+0000 7f557ad9d640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f557c153fe0 0x7f557c06d3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54014/0 (socket says 172.21.15.16:54014) 2023-05-22T15:27:50.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.535+0000 7f557b59e640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f557c070430 0x7f557c0ad410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:32964/0 (socket says 172.21.15.16:32964) 2023-05-22T15:27:50.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.535+0000 7f557ad9d640 1 -- 172.21.15.16:0/2050844294 learned_addr learned my addr 172.21.15.16:0/2050844294 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:50.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.535+0000 7f557ad9d640 1 -- 172.21.15.16:0/2050844294 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f557c0cae20 msgr2=0x7f557c06fef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:50.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.535+0000 7f557a59c640 1 --2- 172.21.15.16:0/2050844294 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f557c0cae20 0x7f557c06fef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:50.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.535+0000 7f557ad9d640 1 --2- 172.21.15.16:0/2050844294 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f557c0cae20 0x7f557c06fef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:50.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.535+0000 7f557ad9d640 1 -- 172.21.15.16:0/2050844294 >> v1:172.21.15.16:6789/0 conn(0x7f557c070430 legacy=0x7f557c0ad410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:50.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.535+0000 7f557ad9d640 1 -- 172.21.15.16:0/2050844294 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f557c15e360 con 0x7f557c153fe0 2023-05-22T15:27:50.549 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.535+0000 7f5579d9b640 1 -- 172.21.15.16:0/2050844294 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2640015254 0 0) 0x7f557c069720 con 0x7f557c070430 2023-05-22T15:27:50.554 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.539+0000 7f557ad9d640 1 --2- 172.21.15.16:0/2050844294 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f557c153fe0 0x7f557c06d3f0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f556c009870 tx=0x7f556c01c510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a61a7b2190d3b00 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:50.554 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.539+0000 7f5579d9b640 1 -- 172.21.15.16:0/2050844294 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f556c0aa670 con 0x7f557c153fe0 2023-05-22T15:27:50.554 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.539+0000 7f5579d9b640 1 -- 172.21.15.16:0/2050844294 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f556c0aa7d0 con 0x7f557c153fe0 2023-05-22T15:27:50.554 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.539+0000 7f5579d9b640 1 -- 172.21.15.16:0/2050844294 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f556c0aab60 con 0x7f557c153fe0 2023-05-22T15:27:50.554 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.539+0000 7f55811e0640 1 -- 172.21.15.16:0/2050844294 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f557c153fe0 msgr2=0x7f557c06d3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:50.554 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.539+0000 7f55811e0640 1 --2- 172.21.15.16:0/2050844294 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f557c153fe0 0x7f557c06d3f0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f556c009870 tx=0x7f556c01c510 comp rx=0 tx=0).stop 2023-05-22T15:27:50.555 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.539+0000 7f55811e0640 1 -- 172.21.15.16:0/2050844294 shutdown_connections 2023-05-22T15:27:50.555 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.539+0000 7f55811e0640 1 --2- 172.21.15.16:0/2050844294 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f557c0cae20 0x7f557c06fef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:50.555 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.539+0000 7f55811e0640 1 --2- 172.21.15.16:0/2050844294 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f557c153fe0 0x7f557c06d3f0 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:50.555 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.539+0000 7f55811e0640 1 -- 172.21.15.16:0/2050844294 >> 172.21.15.16:0/2050844294 conn(0x7f557c0b1110 msgr2=0x7f557c0ce830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:50.555 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.539+0000 7f55811e0640 1 -- 172.21.15.16:0/2050844294 shutdown_connections 2023-05-22T15:27:50.555 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.539+0000 7f55811e0640 1 -- 172.21.15.16:0/2050844294 wait complete. 2023-05-22T15:27:50.555 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.539+0000 7f55811e0640 1 Processor -- start 2023-05-22T15:27:50.555 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.539+0000 7f55811e0640 1 -- start start 2023-05-22T15:27:50.556 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.539+0000 7f55811e0640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f557c070430 0x7f557c14f370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:50.556 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.539+0000 7f55811e0640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f557c0cae20 0x7f557c150d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:50.556 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.539+0000 7f55811e0640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f557c153fe0 0x7f557c141270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:50.556 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.539+0000 7f55811e0640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f557c1532f0 con 0x7f557c070430 2023-05-22T15:27:50.556 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.539+0000 7f55811e0640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f557c1417b0 con 0x7f557c153fe0 2023-05-22T15:27:50.556 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.539+0000 7f55811e0640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f557c152440 con 0x7f557c0cae20 2023-05-22T15:27:50.557 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.539+0000 7f557ad9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f557c070430 0x7f557c14f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:50.557 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.539+0000 7f557ad9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f557c070430 0x7f557c14f370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54020/0 (socket says 172.21.15.16:54020) 2023-05-22T15:27:50.557 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.539+0000 7f557ad9d640 1 -- 172.21.15.16:0/441599130 learned_addr learned my addr 172.21.15.16:0/441599130 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:50.558 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.539+0000 7f557a59c640 1 --2- 172.21.15.16:0/441599130 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f557c0cae20 0x7f557c150d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:50.558 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.539+0000 7f557ad9d640 1 -- 172.21.15.16:0/441599130 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f557c0cae20 msgr2=0x7f557c150d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:50.558 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.539+0000 7f557b59e640 1 --2- 172.21.15.16:0/441599130 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f557c153fe0 0x7f557c141270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:50.558 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.539+0000 7f557ad9d640 1 --2- 172.21.15.16:0/441599130 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f557c0cae20 0x7f557c150d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:50.559 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.539+0000 7f557ad9d640 1 -- 172.21.15.16:0/441599130 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f557c153fe0 msgr2=0x7f557c141270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:50.559 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.539+0000 7f557ad9d640 1 --2- 172.21.15.16:0/441599130 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f557c153fe0 0x7f557c141270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:50.559 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.539+0000 7f557ad9d640 1 -- 172.21.15.16:0/441599130 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f556c0094d0 con 0x7f557c070430 2023-05-22T15:27:50.559 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.539+0000 7f557b59e640 1 --2- 172.21.15.16:0/441599130 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f557c153fe0 0x7f557c141270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:27:50.560 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.539+0000 7f557ad9d640 1 --2- 172.21.15.16:0/441599130 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f557c070430 0x7f557c14f370 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f556c0ac040 tx=0x7f556c01ad50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:50.560 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.539+0000 7f55637fe640 1 -- 172.21.15.16:0/441599130 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f556c0b3800 con 0x7f557c070430 2023-05-22T15:27:50.560 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.539+0000 7f55637fe640 1 -- 172.21.15.16:0/441599130 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f556c0b3960 con 0x7f557c070430 2023-05-22T15:27:50.560 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.539+0000 7f55811e0640 1 -- 172.21.15.16:0/441599130 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f557c15de40 con 0x7f557c070430 2023-05-22T15:27:50.560 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.543+0000 7f55637fe640 1 -- 172.21.15.16:0/441599130 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f556c0b3d10 con 0x7f557c070430 2023-05-22T15:27:50.560 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.543+0000 7f55811e0640 1 -- 172.21.15.16:0/441599130 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f557c15db60 con 0x7f557c070430 2023-05-22T15:27:50.560 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.543+0000 7f55811e0640 1 -- 172.21.15.16:0/441599130 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f557c06dd20 con 0x7f557c070430 2023-05-22T15:27:50.561 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.543+0000 7f55637fe640 1 -- 172.21.15.16:0/441599130 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7f556c0c73a0 con 0x7f557c070430 2023-05-22T15:27:50.561 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.543+0000 7f55637fe640 1 --2- 172.21.15.16:0/441599130 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f5568065380 0x7f5568067830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:50.561 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.543+0000 7f557a59c640 1 --2- 172.21.15.16:0/441599130 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f5568065380 0x7f5568067830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:50.561 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.543+0000 7f55637fe640 1 -- 172.21.15.16:0/441599130 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f556c11d890 con 0x7f557c070430 2023-05-22T15:27:50.561 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.543+0000 7f557a59c640 1 --2- 172.21.15.16:0/441599130 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f5568065380 0x7f5568067830 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f557c0ce010 tx=0x7f5570008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:50.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.547+0000 7f55637fe640 1 -- 172.21.15.16:0/441599130 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f556c0eb210 con 0x7f557c070430 2023-05-22T15:27:50.702 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.695+0000 7f55811e0640 1 -- 172.21.15.16:0/441599130 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f557c04dbe0 con 0x7f557c070430 2023-05-22T15:27:50.703 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.695+0000 7f55637fe640 1 -- 172.21.15.16:0/441599130 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v29) v1 ==== 103+0+11 (secure 0 0 0) 0x7f556c0ee830 con 0x7f557c070430 2023-05-22T15:27:50.703 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2023-05-22T15:27:50.705 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.695+0000 7f55811e0640 1 -- 172.21.15.16:0/441599130 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f5568065380 msgr2=0x7f5568067830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:50.705 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.695+0000 7f55811e0640 1 --2- 172.21.15.16:0/441599130 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f5568065380 0x7f5568067830 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f557c0ce010 tx=0x7f5570008040 comp rx=0 tx=0).stop 2023-05-22T15:27:50.705 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.695+0000 7f55811e0640 1 -- 172.21.15.16:0/441599130 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f557c070430 msgr2=0x7f557c14f370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:50.705 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.695+0000 7f55811e0640 1 --2- 172.21.15.16:0/441599130 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f557c070430 0x7f557c14f370 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f556c0ac040 tx=0x7f556c01ad50 comp rx=0 tx=0).stop 2023-05-22T15:27:50.705 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.695+0000 7f55811e0640 1 -- 172.21.15.16:0/441599130 shutdown_connections 2023-05-22T15:27:50.705 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.695+0000 7f55811e0640 1 --2- 172.21.15.16:0/441599130 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f557c153fe0 0x7f557c141270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:50.706 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.695+0000 7f55811e0640 1 --2- 172.21.15.16:0/441599130 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f5568065380 0x7f5568067830 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:50.706 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.695+0000 7f55811e0640 1 --2- 172.21.15.16:0/441599130 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f557c0cae20 0x7f557c150d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:50.706 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.695+0000 7f55811e0640 1 --2- 172.21.15.16:0/441599130 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f557c070430 0x7f557c14f370 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:50.706 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.695+0000 7f55811e0640 1 -- 172.21.15.16:0/441599130 >> 172.21.15.16:0/441599130 conn(0x7f557c0b1110 msgr2=0x7f557c0ce780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:50.706 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.699+0000 7f55811e0640 1 -- 172.21.15.16:0/441599130 shutdown_connections 2023-05-22T15:27:50.706 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.699+0000 7f55811e0640 1 -- 172.21.15.16:0/441599130 wait complete. 2023-05-22T15:27:50.722 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-05-22T15:27:50.846 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.839+0000 7f9292494640 1 Processor -- start 2023-05-22T15:27:50.846 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.839+0000 7f9292494640 1 -- start start 2023-05-22T15:27:50.847 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.839+0000 7f9292494640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f928c071640 0x7f928c160090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:50.847 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.839+0000 7f9292494640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f928c1605d0 0x7f928c15c270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:50.847 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.839+0000 7f9292494640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f928c069ae0 con 0x7f928c0ce480 2023-05-22T15:27:50.847 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.839+0000 7f9292494640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f928c15f990 con 0x7f928c071640 2023-05-22T15:27:50.847 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.839+0000 7f9292494640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f928c071ac0 con 0x7f928c1605d0 2023-05-22T15:27:50.848 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.839+0000 7f928b7fe640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f928c071640 0x7f928c160090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:50.848 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.839+0000 7f928b7fe640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f928c071640 0x7f928c160090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54026/0 (socket says 172.21.15.16:54026) 2023-05-22T15:27:50.848 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.839+0000 7f928b7fe640 1 -- 172.21.15.16:0/2876353098 learned_addr learned my addr 172.21.15.16:0/2876353098 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:50.848 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.839+0000 7f9290a31640 1 --2- 172.21.15.16:0/2876353098 >> v2:172.21.15.62:3300/0 conn(0x7f928c1605d0 0x7f928c15c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:50.848 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.839+0000 7f928b7fe640 1 -- 172.21.15.16:0/2876353098 >> v2:172.21.15.62:3300/0 conn(0x7f928c1605d0 msgr2=0x7f928c15c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:50.849 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.839+0000 7f928b7fe640 1 --2- 172.21.15.16:0/2876353098 >> v2:172.21.15.62:3300/0 conn(0x7f928c1605d0 0x7f928c15c270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:50.849 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.839+0000 7f928b7fe640 1 -- 172.21.15.16:0/2876353098 >> v1:172.21.15.62:6789/0 conn(0x7f928c0ce480 legacy=0x7f928c0ce850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:50.849 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.839+0000 7f928b7fe640 1 -- 172.21.15.16:0/2876353098 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f928c069dc0 con 0x7f928c071640 2023-05-22T15:27:50.849 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.843+0000 7f928b7fe640 1 --2- 172.21.15.16:0/2876353098 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f928c071640 0x7f928c160090 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f928c0aeb30 tx=0x7f928001c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=759cab5f2afd28fe server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:50.850 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.843+0000 7f928affd640 1 -- 172.21.15.16:0/2876353098 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92800aa5f0 con 0x7f928c071640 2023-05-22T15:27:50.850 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.843+0000 7f928affd640 1 -- 172.21.15.16:0/2876353098 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92800aa750 con 0x7f928c071640 2023-05-22T15:27:50.850 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.843+0000 7f928affd640 1 -- 172.21.15.16:0/2876353098 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92800aaae0 con 0x7f928c071640 2023-05-22T15:27:50.851 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.843+0000 7f9292494640 1 -- 172.21.15.16:0/2876353098 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f928c071640 msgr2=0x7f928c160090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:50.851 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.843+0000 7f9292494640 1 --2- 172.21.15.16:0/2876353098 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f928c071640 0x7f928c160090 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f928c0aeb30 tx=0x7f928001c490 comp rx=0 tx=0).stop 2023-05-22T15:27:50.851 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.843+0000 7f9292494640 1 -- 172.21.15.16:0/2876353098 shutdown_connections 2023-05-22T15:27:50.851 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.843+0000 7f9292494640 1 --2- 172.21.15.16:0/2876353098 >> v2:172.21.15.62:3300/0 conn(0x7f928c1605d0 0x7f928c15c270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:50.851 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.843+0000 7f9292494640 1 --2- 172.21.15.16:0/2876353098 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f928c071640 0x7f928c160090 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:50.851 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.843+0000 7f9292494640 1 -- 172.21.15.16:0/2876353098 >> 172.21.15.16:0/2876353098 conn(0x7f928c0cc870 msgr2=0x7f928c0cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:50.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.843+0000 7f9292494640 1 -- 172.21.15.16:0/2876353098 shutdown_connections 2023-05-22T15:27:50.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.843+0000 7f9292494640 1 -- 172.21.15.16:0/2876353098 wait complete. 2023-05-22T15:27:50.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.843+0000 7f9292494640 1 Processor -- start 2023-05-22T15:27:50.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.843+0000 7f9292494640 1 -- start start 2023-05-22T15:27:50.854 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.847+0000 7f9292494640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f928c071640 0x7f928c151f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:50.854 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.847+0000 7f9292494640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f928c0ce480 0x7f928c0b1460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:50.854 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.847+0000 7f9292494640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f928c1605d0 0x7f928c1536f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:50.854 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.847+0000 7f9292494640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f928c152450 con 0x7f928c1605d0 2023-05-22T15:27:50.854 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.847+0000 7f9292494640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f928c0b1f90 con 0x7f928c0ce480 2023-05-22T15:27:50.854 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.847+0000 7f9292494640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f928c0afab0 con 0x7f928c071640 2023-05-22T15:27:50.855 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.847+0000 7f9290a31640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f928c1605d0 0x7f928c1536f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:50.855 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.847+0000 7f9290a31640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f928c1605d0 0x7f928c1536f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54038/0 (socket says 172.21.15.16:54038) 2023-05-22T15:27:50.855 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.847+0000 7f9290a31640 1 -- 172.21.15.16:0/2730067685 learned_addr learned my addr 172.21.15.16:0/2730067685 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:50.855 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.847+0000 7f928bfff640 1 --2- 172.21.15.16:0/2730067685 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f928c071640 0x7f928c151f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:50.855 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.847+0000 7f928b7fe640 1 --2- 172.21.15.16:0/2730067685 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f928c0ce480 0x7f928c0b1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:50.855 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.847+0000 7f9290a31640 1 -- 172.21.15.16:0/2730067685 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f928c071640 msgr2=0x7f928c151f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:50.855 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.847+0000 7f9290a31640 1 --2- 172.21.15.16:0/2730067685 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f928c071640 0x7f928c151f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:50.855 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.847+0000 7f9290a31640 1 -- 172.21.15.16:0/2730067685 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f928c0ce480 msgr2=0x7f928c0b1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:50.856 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.847+0000 7f9290a31640 1 --2- 172.21.15.16:0/2730067685 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f928c0ce480 0x7f928c0b1460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:50.856 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.847+0000 7f9290a31640 1 -- 172.21.15.16:0/2730067685 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92800094d0 con 0x7f928c1605d0 2023-05-22T15:27:50.856 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.847+0000 7f928bfff640 1 --2- 172.21.15.16:0/2730067685 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f928c071640 0x7f928c151f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T15:27:50.856 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.847+0000 7f9290a31640 1 --2- 172.21.15.16:0/2730067685 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f928c1605d0 0x7f928c1536f0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f9284007e40 tx=0x7f928400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:50.857 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.847+0000 7f9288ff9640 1 -- 172.21.15.16:0/2730067685 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9284014620 con 0x7f928c1605d0 2023-05-22T15:27:50.857 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.847+0000 7f9288ff9640 1 -- 172.21.15.16:0/2730067685 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f928400ee90 con 0x7f928c1605d0 2023-05-22T15:27:50.857 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.847+0000 7f9288ff9640 1 -- 172.21.15.16:0/2730067685 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92840149e0 con 0x7f928c1605d0 2023-05-22T15:27:50.857 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.847+0000 7f9292494640 1 -- 172.21.15.16:0/2730067685 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f928c0698b0 con 0x7f928c1605d0 2023-05-22T15:27:50.858 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.847+0000 7f9292494640 1 -- 172.21.15.16:0/2730067685 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f928c069720 con 0x7f928c1605d0 2023-05-22T15:27:50.858 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.847+0000 7f9292494640 1 -- 172.21.15.16:0/2730067685 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f928c0c5e70 con 0x7f928c1605d0 2023-05-22T15:27:50.858 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.847+0000 7f9288ff9640 1 -- 172.21.15.16:0/2730067685 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7f9284014b40 con 0x7f928c1605d0 2023-05-22T15:27:50.858 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.847+0000 7f9288ff9640 1 --2- 172.21.15.16:0/2730067685 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f92780653b0 0x7f9278067860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:50.858 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.847+0000 7f928bfff640 1 --2- 172.21.15.16:0/2730067685 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f92780653b0 0x7f9278067860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:50.859 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.847+0000 7f9288ff9640 1 -- 172.21.15.16:0/2730067685 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f9284083400 con 0x7f928c1605d0 2023-05-22T15:27:50.859 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.851+0000 7f928bfff640 1 --2- 172.21.15.16:0/2730067685 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f92780653b0 0x7f9278067860 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f928c15dc40 tx=0x7f927c009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:50.860 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:50.851+0000 7f9288ff9640 1 -- 172.21.15.16:0/2730067685 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f9284050de0 con 0x7f928c1605d0 2023-05-22T15:27:51.025 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.019+0000 7f9292494640 1 -- 172.21.15.16:0/2730067685 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f928c04dbe0 con 0x7f928c1605d0 2023-05-22T15:27:51.026 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.019+0000 7f9288ff9640 1 -- 172.21.15.16:0/2730067685 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v29) v1 ==== 80+0+0 (secure 0 0 0) 0x7f9284054400 con 0x7f928c1605d0 2023-05-22T15:27:51.027 INFO:teuthology.orchestra.run.smithi016.stderr:listed 0 entries 2023-05-22T15:27:51.028 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.019+0000 7f9292494640 1 -- 172.21.15.16:0/2730067685 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f92780653b0 msgr2=0x7f9278067860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:51.028 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.019+0000 7f9292494640 1 --2- 172.21.15.16:0/2730067685 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f92780653b0 0x7f9278067860 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f928c15dc40 tx=0x7f927c009040 comp rx=0 tx=0).stop 2023-05-22T15:27:51.029 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.019+0000 7f9292494640 1 -- 172.21.15.16:0/2730067685 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f928c1605d0 msgr2=0x7f928c1536f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:51.029 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.019+0000 7f9292494640 1 --2- 172.21.15.16:0/2730067685 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f928c1605d0 0x7f928c1536f0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f9284007e40 tx=0x7f928400e4b0 comp rx=0 tx=0).stop 2023-05-22T15:27:51.029 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.019+0000 7f9292494640 1 -- 172.21.15.16:0/2730067685 shutdown_connections 2023-05-22T15:27:51.029 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.019+0000 7f9292494640 1 --2- 172.21.15.16:0/2730067685 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f928c1605d0 0x7f928c1536f0 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:51.029 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.019+0000 7f9292494640 1 --2- 172.21.15.16:0/2730067685 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f928c0ce480 0x7f928c0b1460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:51.029 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.019+0000 7f9292494640 1 --2- 172.21.15.16:0/2730067685 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f92780653b0 0x7f9278067860 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:51.029 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.019+0000 7f9292494640 1 --2- 172.21.15.16:0/2730067685 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f928c071640 0x7f928c151f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:51.029 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.019+0000 7f9292494640 1 -- 172.21.15.16:0/2730067685 >> 172.21.15.16:0/2730067685 conn(0x7f928c0cc870 msgr2=0x7f928c0ce850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:51.030 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.019+0000 7f9292494640 1 -- 172.21.15.16:0/2730067685 shutdown_connections 2023-05-22T15:27:51.030 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.019+0000 7f9292494640 1 -- 172.21.15.16:0/2730067685 wait complete. 2023-05-22T15:27:51.045 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-05-22T15:27:51.046 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2023-05-22T15:27:51.052 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-05-22T15:27:51.213 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.203+0000 7f9de84ef640 1 Processor -- start 2023-05-22T15:27:51.213 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.203+0000 7f9de84ef640 1 -- start start 2023-05-22T15:27:51.214 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.207+0000 7f9de84ef640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f9de00ca080 0x7f9de00c8670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:51.214 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.207+0000 7f9de84ef640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9de015a070 con 0x7f9de015e470 2023-05-22T15:27:51.215 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.207+0000 7f9de84ef640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9de00ca580 con 0x7f9de0070f10 2023-05-22T15:27:51.215 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.207+0000 7f9de84ef640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9de015d990 con 0x7f9de00ca080 2023-05-22T15:27:51.215 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.207+0000 7f9de5a8a640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f9de00ca080 0x7f9de00c8670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:51.215 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.207+0000 7f9de5a8a640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f9de00ca080 0x7f9de00c8670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54048/0 (socket says 172.21.15.16:54048) 2023-05-22T15:27:51.215 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.207+0000 7f9de5a8a640 1 -- 172.21.15.16:0/3607471285 learned_addr learned my addr 172.21.15.16:0/3607471285 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:51.215 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.207+0000 7f9de5a8a640 1 -- 172.21.15.16:0/3607471285 >> v1:172.21.15.62:6789/0 conn(0x7f9de0070f10 legacy=0x7f9de00712e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:51.215 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.207+0000 7f9de5a8a640 1 -- 172.21.15.16:0/3607471285 >> v1:172.21.15.16:6789/0 conn(0x7f9de015e470 legacy=0x7f9de00c8bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:51.216 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.207+0000 7f9de5a8a640 1 -- 172.21.15.16:0/3607471285 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9de015dc70 con 0x7f9de00ca080 2023-05-22T15:27:51.216 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.207+0000 7f9de5a8a640 1 --2- 172.21.15.16:0/3607471285 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9de00ca080 0x7f9de00c8670 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f9dd4009870 tx=0x7f9dd401c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fdade5c350db4b5c server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:51.217 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.207+0000 7f9de5289640 1 -- 172.21.15.16:0/3607471285 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dd40a6040 con 0x7f9de00ca080 2023-05-22T15:27:51.217 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.207+0000 7f9de5289640 1 -- 172.21.15.16:0/3607471285 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9dd40aa600 con 0x7f9de00ca080 2023-05-22T15:27:51.217 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.207+0000 7f9de5289640 1 -- 172.21.15.16:0/3607471285 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dd40aa990 con 0x7f9de00ca080 2023-05-22T15:27:51.218 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.211+0000 7f9de84ef640 1 -- 172.21.15.16:0/3607471285 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9de00ca080 msgr2=0x7f9de00c8670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:51.218 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.211+0000 7f9de84ef640 1 --2- 172.21.15.16:0/3607471285 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9de00ca080 0x7f9de00c8670 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f9dd4009870 tx=0x7f9dd401c480 comp rx=0 tx=0).stop 2023-05-22T15:27:51.218 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.211+0000 7f9de84ef640 1 -- 172.21.15.16:0/3607471285 shutdown_connections 2023-05-22T15:27:51.219 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.211+0000 7f9de84ef640 1 --2- 172.21.15.16:0/3607471285 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9de00ca080 0x7f9de00c8670 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:51.219 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.211+0000 7f9de84ef640 1 -- 172.21.15.16:0/3607471285 >> 172.21.15.16:0/3607471285 conn(0x7f9de0069720 msgr2=0x7f9de0069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:51.219 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.211+0000 7f9de84ef640 1 -- 172.21.15.16:0/3607471285 shutdown_connections 2023-05-22T15:27:51.219 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.211+0000 7f9de84ef640 1 -- 172.21.15.16:0/3607471285 wait complete. 2023-05-22T15:27:51.222 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.211+0000 7f9de84ef640 1 Processor -- start 2023-05-22T15:27:51.223 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.211+0000 7f9de84ef640 1 -- start start 2023-05-22T15:27:51.223 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.211+0000 7f9de84ef640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9de0070f10 0x7f9de0147d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:51.223 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.211+0000 7f9de84ef640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9de00ca080 0x7f9de0147560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:51.223 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.211+0000 7f9de84ef640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9de015e470 0x7f9de00c5f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:51.223 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.211+0000 7f9de84ef640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9de014aa60 con 0x7f9de00ca080 2023-05-22T15:27:51.223 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.211+0000 7f9de84ef640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9de0152d40 con 0x7f9de015e470 2023-05-22T15:27:51.224 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.211+0000 7f9de84ef640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9de0152410 con 0x7f9de0070f10 2023-05-22T15:27:51.224 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.211+0000 7f9de5a8a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9de00ca080 0x7f9de0147560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:51.224 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.211+0000 7f9de5a8a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9de00ca080 0x7f9de0147560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54052/0 (socket says 172.21.15.16:54052) 2023-05-22T15:27:51.224 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.211+0000 7f9de5a8a640 1 -- 172.21.15.16:0/3316770612 learned_addr learned my addr 172.21.15.16:0/3316770612 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:51.224 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.211+0000 7f9de6a8c640 1 --2- 172.21.15.16:0/3316770612 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9de015e470 0x7f9de00c5f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:51.225 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.211+0000 7f9de628b640 1 --2- 172.21.15.16:0/3316770612 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9de0070f10 0x7f9de0147d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:51.225 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.211+0000 7f9de5a8a640 1 -- 172.21.15.16:0/3316770612 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9de0070f10 msgr2=0x7f9de0147d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:51.225 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.211+0000 7f9de5a8a640 1 --2- 172.21.15.16:0/3316770612 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9de0070f10 0x7f9de0147d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:51.225 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.211+0000 7f9de5a8a640 1 -- 172.21.15.16:0/3316770612 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9de015e470 msgr2=0x7f9de00c5f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:51.225 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.211+0000 7f9de5a8a640 1 --2- 172.21.15.16:0/3316770612 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9de015e470 0x7f9de00c5f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:51.225 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.211+0000 7f9de5a8a640 1 -- 172.21.15.16:0/3316770612 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9dd40094d0 con 0x7f9de00ca080 2023-05-22T15:27:51.225 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.211+0000 7f9de628b640 1 --2- 172.21.15.16:0/3316770612 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9de0070f10 0x7f9de0147d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:27:51.225 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.211+0000 7f9de5a8a640 1 --2- 172.21.15.16:0/3316770612 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9de00ca080 0x7f9de0147560 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f9dd401a0b0 tx=0x7f9dd401acc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:51.226 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.211+0000 7f9dd2ffd640 1 -- 172.21.15.16:0/3316770612 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dd40b3800 con 0x7f9de00ca080 2023-05-22T15:27:51.226 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.211+0000 7f9dd2ffd640 1 -- 172.21.15.16:0/3316770612 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9dd40b3960 con 0x7f9de00ca080 2023-05-22T15:27:51.226 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.211+0000 7f9de84ef640 1 -- 172.21.15.16:0/3316770612 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9de0168540 con 0x7f9de00ca080 2023-05-22T15:27:51.226 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.211+0000 7f9dd2ffd640 1 -- 172.21.15.16:0/3316770612 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dd40b3cf0 con 0x7f9de00ca080 2023-05-22T15:27:51.226 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.211+0000 7f9de84ef640 1 -- 172.21.15.16:0/3316770612 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9de01683b0 con 0x7f9de00ca080 2023-05-22T15:27:51.226 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.215+0000 7f9de84ef640 1 -- 172.21.15.16:0/3316770612 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9de01686d0 con 0x7f9de00ca080 2023-05-22T15:27:51.226 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.215+0000 7f9dd2ffd640 1 -- 172.21.15.16:0/3316770612 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7f9dd40c6430 con 0x7f9de00ca080 2023-05-22T15:27:51.227 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.215+0000 7f9dd2ffd640 1 --2- 172.21.15.16:0/3316770612 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f9dc4065380 0x7f9dc4067830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:51.227 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.215+0000 7f9dd2ffd640 1 -- 172.21.15.16:0/3316770612 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f9de00c94d0 con 0x7f9de00ca080 2023-05-22T15:27:51.227 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.215+0000 7f9de628b640 1 --2- 172.21.15.16:0/3316770612 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f9dc4065380 0x7f9dc4067830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:51.227 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.215+0000 7f9de628b640 1 --2- 172.21.15.16:0/3316770612 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f9dc4065380 0x7f9dc4067830 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f9ddc000f70 tx=0x7f9ddc009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:51.227 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.219+0000 7f9dd2ffd640 1 -- 172.21.15.16:0/3316770612 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f9dd40eb270 con 0x7f9de00ca080 2023-05-22T15:27:51.382 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.375+0000 7f9de84ef640 1 -- 172.21.15.16:0/3316770612 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9de004dbe0 con 0x7f9de00ca080 2023-05-22T15:27:51.383 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.375+0000 7f9dd2ffd640 1 -- 172.21.15.16:0/3316770612 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v29) v1 ==== 74+0+19700 (secure 0 0 0) 0x7f9dd40ee890 con 0x7f9de00ca080 2023-05-22T15:27:51.384 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:27:51.384 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":29,"fsid":"b2719834-7847-4d98-90a6-f65d57f85c49","created":"2023-05-22T15:26:25.259314+0000","modified":"2023-05-22T15:27:31.182317+0000","last_up_change":"2023-05-22T15:26:36.448524+0000","last_in_change":"2023-05-22T15:26:27.675668+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-05-22T15:26:38.019781+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-05-22T15:26:43.570514+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-05-22T15:26:47.879159+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-05-22T15:26:48.522130+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data_ec","create_time":"2023-05-22T15:26:51.616142+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"9e7ff854-b25f-4cbb-a5c2-1799d9c53d2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":37021},{"type":"v1","addr":"172.21.15.16:6825","nonce":37021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":37021},{"type":"v1","addr":"172.21.15.16:6827","nonce":37021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":37021},{"type":"v1","addr":"172.21.15.16:6831","nonce":37021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":37021},{"type":"v1","addr":"172.21.15.16:6829","nonce":37021}]},"public_addr":"172.21.15.16:6825/37021","cluster_addr":"172.21.15.16:6827/37021","heartbeat_back_addr":"172.21.15.16:6831/37021","heartbeat_front_addr":"172.21.15.16:6829/37021","state":["exists","up"]},{"osd":1,"uuid":"4702fbc9-a927-4565-aca5-41dc69cc8264","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":37008},{"type":"v1","addr":"172.21.15.16:6801","nonce":37008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":37008},{"type":"v1","addr":"172.21.15.16:6803","nonce":37008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":37008},{"type":"v1","addr":"172.21.15.16:6807","nonce":37008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":37008},{"type":"v1","addr":"172.21.15.16:6805","nonce":37008}]},"public_addr":"172.21.15.16:6801/37008","cluster_addr":"172.21.15.16:6803/37008","heartbeat_back_addr":"172.21.15.16:6807/37008","heartbeat_front_addr":"172.21.15.16:6805/37008","state":["exists","up"]},{"osd":2,"uuid":"1e7bba46-4eba-47ef-bb81-5e2d371638ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":37009},{"type":"v1","addr":"172.21.15.16:6809","nonce":37009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":37009},{"type":"v1","addr":"172.21.15.16:6811","nonce":37009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":37009},{"type":"v1","addr":"172.21.15.16:6815","nonce":37009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":37009},{"type":"v1","addr":"172.21.15.16:6813","nonce":37009}]},"public_addr":"172.21.15.16:6809/37009","cluster_addr":"172.21.15.16:6811/37009","heartbeat_back_addr":"172.21.15.16:6815/37009","heartbeat_front_addr":"172.21.15.16:6813/37009","state":["exists","up"]},{"osd":3,"uuid":"383cacc0-5a88-4904-990e-498ca59ef48a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":37020},{"type":"v1","addr":"172.21.15.16:6817","nonce":37020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":37020},{"type":"v1","addr":"172.21.15.16:6819","nonce":37020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":37020},{"type":"v1","addr":"172.21.15.16:6823","nonce":37020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":37020},{"type":"v1","addr":"172.21.15.16:6821","nonce":37020}]},"public_addr":"172.21.15.16:6817/37020","cluster_addr":"172.21.15.16:6819/37020","heartbeat_back_addr":"172.21.15.16:6823/37020","heartbeat_front_addr":"172.21.15.16:6821/37020","state":["exists","up"]},{"osd":4,"uuid":"970500ef-95b7-42c1-b080-eed4fae7013d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":37353},{"type":"v1","addr":"172.21.15.62:6817","nonce":37353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6819","nonce":37353},{"type":"v1","addr":"172.21.15.62:6822","nonce":37353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37353},{"type":"v1","addr":"172.21.15.62:6831","nonce":37353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37353},{"type":"v1","addr":"172.21.15.62:6826","nonce":37353}]},"public_addr":"172.21.15.62:6817/37353","cluster_addr":"172.21.15.62:6822/37353","heartbeat_back_addr":"172.21.15.62:6831/37353","heartbeat_front_addr":"172.21.15.62:6826/37353","state":["exists","up"]},{"osd":5,"uuid":"022d67db-d4d8-445b-b1b9-8b446a00f5c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":37352},{"type":"v1","addr":"172.21.15.62:6801","nonce":37352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":37352},{"type":"v1","addr":"172.21.15.62:6803","nonce":37352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":37352},{"type":"v1","addr":"172.21.15.62:6807","nonce":37352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":37352},{"type":"v1","addr":"172.21.15.62:6805","nonce":37352}]},"public_addr":"172.21.15.62:6801/37352","cluster_addr":"172.21.15.62:6803/37352","heartbeat_back_addr":"172.21.15.62:6807/37352","heartbeat_front_addr":"172.21.15.62:6805/37352","state":["exists","up"]},{"osd":6,"uuid":"42bd372b-6688-4a1b-8407-e16e8c266e33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":37354},{"type":"v1","addr":"172.21.15.62:6820","nonce":37354}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":37354},{"type":"v1","addr":"172.21.15.62:6823","nonce":37354}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6829","nonce":37354},{"type":"v1","addr":"172.21.15.62:6830","nonce":37354}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6825","nonce":37354},{"type":"v1","addr":"172.21.15.62:6827","nonce":37354}]},"public_addr":"172.21.15.62:6820/37354","cluster_addr":"172.21.15.62:6823/37354","heartbeat_back_addr":"172.21.15.62:6830/37354","heartbeat_front_addr":"172.21.15.62:6827/37354","state":["exists","up"]},{"osd":7,"uuid":"ac70b2a3-f41b-4bf2-b227-7f2cbb55426d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37355},{"type":"v1","addr":"172.21.15.62:6809","nonce":37355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37355},{"type":"v1","addr":"172.21.15.62:6811","nonce":37355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":37355},{"type":"v1","addr":"172.21.15.62:6815","nonce":37355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":37355},{"type":"v1","addr":"172.21.15.62:6813","nonce":37355}]},"public_addr":"172.21.15.62:6809/37355","cluster_addr":"172.21.15.62:6811/37355","heartbeat_back_addr":"172.21.15.62:6815/37355","heartbeat_front_addr":"172.21.15.62:6813/37355","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.368896+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.107273+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.276888+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.287502+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.124791+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.251937+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.255679+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.232797+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"4.23","mappings":[{"from":3,"to":2}]},{"pgid":"4.31","mappings":[{"from":1,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-05-22T15:27:51.386 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.379+0000 7f9de84ef640 1 -- 172.21.15.16:0/3316770612 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f9dc4065380 msgr2=0x7f9dc4067830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:51.386 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.379+0000 7f9de84ef640 1 --2- 172.21.15.16:0/3316770612 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f9dc4065380 0x7f9dc4067830 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f9ddc000f70 tx=0x7f9ddc009040 comp rx=0 tx=0).stop 2023-05-22T15:27:51.386 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.379+0000 7f9de84ef640 1 -- 172.21.15.16:0/3316770612 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9de00ca080 msgr2=0x7f9de0147560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:51.386 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.379+0000 7f9de84ef640 1 --2- 172.21.15.16:0/3316770612 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9de00ca080 0x7f9de0147560 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f9dd401a0b0 tx=0x7f9dd401acc0 comp rx=0 tx=0).stop 2023-05-22T15:27:51.387 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.379+0000 7f9de84ef640 1 -- 172.21.15.16:0/3316770612 shutdown_connections 2023-05-22T15:27:51.387 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.379+0000 7f9de84ef640 1 --2- 172.21.15.16:0/3316770612 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9de015e470 0x7f9de00c5f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:51.387 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.379+0000 7f9de84ef640 1 --2- 172.21.15.16:0/3316770612 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9de00ca080 0x7f9de0147560 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:51.387 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.379+0000 7f9de84ef640 1 --2- 172.21.15.16:0/3316770612 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f9dc4065380 0x7f9dc4067830 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:51.387 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.379+0000 7f9de84ef640 1 --2- 172.21.15.16:0/3316770612 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9de0070f10 0x7f9de0147d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:51.387 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.379+0000 7f9de84ef640 1 -- 172.21.15.16:0/3316770612 >> 172.21.15.16:0/3316770612 conn(0x7f9de0069720 msgr2=0x7f9de00c8650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:51.387 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.379+0000 7f9de84ef640 1 -- 172.21.15.16:0/3316770612 shutdown_connections 2023-05-22T15:27:51.387 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.379+0000 7f9de84ef640 1 -- 172.21.15.16:0/3316770612 wait complete. 2023-05-22T15:27:51.402 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-05-22T15:26:38.019781+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-05-22T15:26:43.570514+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2023-05-22T15:26:47.879159+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2023-05-22T15:26:48.522130+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-05-22T15:26:51.616142+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '27', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-05-22T15:27:51.403 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-05-22T15:27:51.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.511+0000 7fc7b742c640 1 Processor -- start 2023-05-22T15:27:51.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.511+0000 7fc7b742c640 1 -- start start 2023-05-22T15:27:51.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.515+0000 7fc7b742c640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fc7b00ce480 0x7fc7b0071640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:51.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.515+0000 7fc7b742c640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc7b015c270 0x7fc7b015c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:51.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.515+0000 7fc7b742c640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7b0069ae0 con 0x7fc7b00ce850 2023-05-22T15:27:51.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.515+0000 7fc7b742c640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7b015f990 con 0x7fc7b00ce480 2023-05-22T15:27:51.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.515+0000 7fc7b742c640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7b0069dc0 con 0x7fc7b015c270 2023-05-22T15:27:51.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.515+0000 7fc7b49c7640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fc7b00ce850 0x7fc7b0160090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:32990/0 (socket says 172.21.15.16:32990) 2023-05-22T15:27:51.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.515+0000 7fc7b49c7640 1 -- 172.21.15.16:0/93712698 learned_addr learned my addr 172.21.15.16:0/93712698 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:27:51.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.515+0000 7fc7b51c8640 1 --2- 172.21.15.16:0/93712698 >> v2:172.21.15.62:3300/0 conn(0x7fc7b00ce480 0x7fc7b0071640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:51.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.515+0000 7fc7b59c9640 1 --2- 172.21.15.16:0/93712698 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc7b015c270 0x7fc7b015c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:51.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.515+0000 7fc7a7fff640 1 -- 172.21.15.16:0/93712698 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3018888147 0 0) 0x7fc7b0069ae0 con 0x7fc7b00ce850 2023-05-22T15:27:51.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.515+0000 7fc7a7fff640 1 -- 172.21.15.16:0/93712698 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc79c003530 con 0x7fc7b00ce850 2023-05-22T15:27:51.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.515+0000 7fc7a7fff640 1 -- 172.21.15.16:0/93712698 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3527609911 0 0) 0x7fc79c003530 con 0x7fc7b00ce850 2023-05-22T15:27:51.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.515+0000 7fc7a7fff640 1 -- 172.21.15.16:0/93712698 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc7b015c270 msgr2=0x7fc7b015c6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:51.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.515+0000 7fc7a7fff640 1 --2- 172.21.15.16:0/93712698 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc7b015c270 0x7fc7b015c6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:51.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.515+0000 7fc7a7fff640 1 -- 172.21.15.16:0/93712698 >> v2:172.21.15.62:3300/0 conn(0x7fc7b00ce480 msgr2=0x7fc7b0071640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:51.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.515+0000 7fc7a7fff640 1 --2- 172.21.15.16:0/93712698 >> v2:172.21.15.62:3300/0 conn(0x7fc7b00ce480 0x7fc7b0071640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:51.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.515+0000 7fc7a7fff640 1 -- 172.21.15.16:0/93712698 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7b006a0a0 con 0x7fc7b00ce850 2023-05-22T15:27:51.526 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.515+0000 7fc7a7fff640 1 -- 172.21.15.16:0/93712698 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7fc7a0001680 con 0x7fc7b00ce850 2023-05-22T15:27:51.526 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.515+0000 7fc7a7fff640 1 -- 172.21.15.16:0/93712698 >> v1:172.21.15.16:6789/0 conn(0x7fc7b00ce850 legacy=0x7fc7b0160090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:51.526 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.515+0000 7fc7a7fff640 1 --2- 172.21.15.16:0/93712698 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc79c003980 0x7fc79c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:51.526 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.515+0000 7fc7a7fff640 1 --2- 172.21.15.16:0/93712698 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc79c0043b0 0x7fc79c004870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:51.526 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.515+0000 7fc7a7fff640 1 --2- 172.21.15.16:0/93712698 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc7b015c270 0x7fc79c004f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:51.526 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.515+0000 7fc7a7fff640 1 -- 172.21.15.16:0/93712698 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7b015f990 con 0x7fc79c003980 2023-05-22T15:27:51.526 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.515+0000 7fc7a7fff640 1 -- 172.21.15.16:0/93712698 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7b0069dc0 con 0x7fc79c0043b0 2023-05-22T15:27:51.526 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.515+0000 7fc7a7fff640 1 -- 172.21.15.16:0/93712698 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc79c00b990 con 0x7fc7b015c270 2023-05-22T15:27:51.527 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.515+0000 7fc7b51c8640 1 --2- 172.21.15.16:0/93712698 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc79c003980 0x7fc79c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:51.527 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.515+0000 7fc7b49c7640 1 --2- 172.21.15.16:0/93712698 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc79c0043b0 0x7fc79c004870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:51.527 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.515+0000 7fc7b59c9640 1 --2- 172.21.15.16:0/93712698 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc7b015c270 0x7fc79c004f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:51.527 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.515+0000 7fc7b51c8640 1 -- 172.21.15.16:0/93712698 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc7b015c270 msgr2=0x7fc79c004f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:51.527 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.515+0000 7fc7b51c8640 1 --2- 172.21.15.16:0/93712698 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc7b015c270 0x7fc79c004f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:51.527 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.515+0000 7fc7b51c8640 1 -- 172.21.15.16:0/93712698 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc79c0043b0 msgr2=0x7fc79c004870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:51.527 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.515+0000 7fc7b51c8640 1 --2- 172.21.15.16:0/93712698 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc79c0043b0 0x7fc79c004870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:51.527 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.515+0000 7fc7b51c8640 1 -- 172.21.15.16:0/93712698 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc79c003530 con 0x7fc79c003980 2023-05-22T15:27:51.531 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.519+0000 7fc7b51c8640 1 --2- 172.21.15.16:0/93712698 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc79c003980 0x7fc79c003d90 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fc7ac009ae0 tx=0x7fc7ac01ca60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f51b2705bee8e64 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:51.531 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.519+0000 7fc7a7fff640 1 -- 172.21.15.16:0/93712698 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7ac0a5570 con 0x7fc79c003980 2023-05-22T15:27:51.532 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.519+0000 7fc7a7fff640 1 -- 172.21.15.16:0/93712698 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7ac0a56d0 con 0x7fc79c003980 2023-05-22T15:27:51.532 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.519+0000 7fc7a7fff640 1 -- 172.21.15.16:0/93712698 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7ac0a5a80 con 0x7fc79c003980 2023-05-22T15:27:51.532 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.519+0000 7fc7b742c640 1 -- 172.21.15.16:0/93712698 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc79c003980 msgr2=0x7fc79c003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:51.532 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.519+0000 7fc7b742c640 1 --2- 172.21.15.16:0/93712698 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc79c003980 0x7fc79c003d90 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fc7ac009ae0 tx=0x7fc7ac01ca60 comp rx=0 tx=0).stop 2023-05-22T15:27:51.532 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.519+0000 7fc7b59c9640 1 -- 172.21.15.16:0/93712698 reap_dead start 2023-05-22T15:27:51.532 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.519+0000 7fc7b742c640 1 -- 172.21.15.16:0/93712698 shutdown_connections 2023-05-22T15:27:51.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.519+0000 7fc7b742c640 1 -- 172.21.15.16:0/93712698 >> 172.21.15.16:0/93712698 conn(0x7fc7b00cc870 msgr2=0x7fc7b00cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:51.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.519+0000 7fc7b742c640 1 -- 172.21.15.16:0/93712698 shutdown_connections 2023-05-22T15:27:51.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.519+0000 7fc7b742c640 1 -- 172.21.15.16:0/93712698 wait complete. 2023-05-22T15:27:51.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.519+0000 7fc7b742c640 1 Processor -- start 2023-05-22T15:27:51.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.519+0000 7fc7b742c640 1 -- start start 2023-05-22T15:27:51.534 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.519+0000 7fc7b742c640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc79c003980 0x7fc7b014d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:51.534 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.519+0000 7fc7b742c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc7b00b0950 0x7fc7b00b0dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:51.534 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.519+0000 7fc7b742c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc7b00cec50 0x7fc7b00cf0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:51.534 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.519+0000 7fc7b742c640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7b00cf600 con 0x7fc7b00b0950 2023-05-22T15:27:51.534 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.519+0000 7fc7b742c640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7b00b1a60 con 0x7fc7b00cec50 2023-05-22T15:27:51.534 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.519+0000 7fc7b742c640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7b00abc70 con 0x7fc79c003980 2023-05-22T15:27:51.534 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.519+0000 7fc7b49c7640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc7b00b0950 0x7fc7b00b0dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:51.535 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.519+0000 7fc7b49c7640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc7b00b0950 0x7fc7b00b0dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54078/0 (socket says 172.21.15.16:54078) 2023-05-22T15:27:51.535 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.519+0000 7fc7b49c7640 1 -- 172.21.15.16:0/3983249745 learned_addr learned my addr 172.21.15.16:0/3983249745 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:51.535 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.523+0000 7fc7b59c9640 1 --2- 172.21.15.16:0/3983249745 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc7b00cec50 0x7fc7b00cf0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:51.536 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.523+0000 7fc7b51c8640 1 --2- 172.21.15.16:0/3983249745 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc79c003980 0x7fc7b014d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:51.536 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.523+0000 7fc7b49c7640 1 -- 172.21.15.16:0/3983249745 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc79c003980 msgr2=0x7fc7b014d5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:51.536 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.523+0000 7fc7b49c7640 1 --2- 172.21.15.16:0/3983249745 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc79c003980 0x7fc7b014d5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:51.536 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.523+0000 7fc7b49c7640 1 -- 172.21.15.16:0/3983249745 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc7b00cec50 msgr2=0x7fc7b00cf0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:51.536 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.523+0000 7fc7b49c7640 1 --2- 172.21.15.16:0/3983249745 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc7b00cec50 0x7fc7b00cf0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:51.536 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.523+0000 7fc7b49c7640 1 -- 172.21.15.16:0/3983249745 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7b00698b0 con 0x7fc7b00b0950 2023-05-22T15:27:51.536 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.523+0000 7fc7b51c8640 1 --2- 172.21.15.16:0/3983249745 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc79c003980 0x7fc7b014d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:27:51.537 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.523+0000 7fc7b49c7640 1 --2- 172.21.15.16:0/3983249745 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc7b00b0950 0x7fc7b00b0dc0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fc7a000e820 tx=0x7fc7a000ece0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:51.537 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.523+0000 7fc7a5ffb640 1 -- 172.21.15.16:0/3983249745 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7a000eed0 con 0x7fc7b00b0950 2023-05-22T15:27:51.537 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.523+0000 7fc7a5ffb640 1 -- 172.21.15.16:0/3983249745 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7a000c7f0 con 0x7fc7b00b0950 2023-05-22T15:27:51.537 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.523+0000 7fc7b742c640 1 -- 172.21.15.16:0/3983249745 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7b0069720 con 0x7fc7b00b0950 2023-05-22T15:27:51.537 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.523+0000 7fc7a5ffb640 1 -- 172.21.15.16:0/3983249745 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7a000cba0 con 0x7fc7b00b0950 2023-05-22T15:27:51.537 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.523+0000 7fc7b742c640 1 -- 172.21.15.16:0/3983249745 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7b00c5e70 con 0x7fc7b00b0950 2023-05-22T15:27:51.537 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.523+0000 7fc7b742c640 1 -- 172.21.15.16:0/3983249745 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7b004dbe0 con 0x7fc7b00b0950 2023-05-22T15:27:51.537 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.523+0000 7fc7a5ffb640 1 -- 172.21.15.16:0/3983249745 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7fc7a000cd00 con 0x7fc7b00b0950 2023-05-22T15:27:51.538 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.523+0000 7fc7a5ffb640 1 --2- 172.21.15.16:0/3983249745 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fc794065380 0x7fc794067830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:51.538 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.523+0000 7fc7a5ffb640 1 -- 172.21.15.16:0/3983249745 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6871+0+0 (secure 0 0 0) 0x7fc7a0082a10 con 0x7fc7b00b0950 2023-05-22T15:27:51.538 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.523+0000 7fc7b51c8640 1 --2- 172.21.15.16:0/3983249745 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fc794065380 0x7fc794067830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:51.538 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.523+0000 7fc7b51c8640 1 --2- 172.21.15.16:0/3983249745 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fc794065380 0x7fc794067830 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fc7ac0af000 tx=0x7fc7ac0046d0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:51.538 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.527+0000 7fc7a5ffb640 1 -- 172.21.15.16:0/3983249745 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fc7a00503f0 con 0x7fc7b00b0950 2023-05-22T15:27:51.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.675+0000 7fc7b742c640 1 -- 172.21.15.16:0/3983249745 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fc7b0070a10 con 0x7fc7b00b0950 2023-05-22T15:27:51.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.675+0000 7fc7a5ffb640 1 -- 172.21.15.16:0/3983249745 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v29) v1 ==== 93+0+10 (secure 0 0 0) 0x7fc7a0053a10 con 0x7fc7b00b0950 2023-05-22T15:27:51.683 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2023-05-22T15:27:51.684 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.675+0000 7fc7b742c640 1 -- 172.21.15.16:0/3983249745 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fc794065380 msgr2=0x7fc794067830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:51.684 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.675+0000 7fc7b742c640 1 --2- 172.21.15.16:0/3983249745 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fc794065380 0x7fc794067830 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fc7ac0af000 tx=0x7fc7ac0046d0 comp rx=0 tx=0).stop 2023-05-22T15:27:51.685 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.675+0000 7fc7b742c640 1 -- 172.21.15.16:0/3983249745 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc7b00b0950 msgr2=0x7fc7b00b0dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:51.685 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.675+0000 7fc7b742c640 1 --2- 172.21.15.16:0/3983249745 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc7b00b0950 0x7fc7b00b0dc0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fc7a000e820 tx=0x7fc7a000ece0 comp rx=0 tx=0).stop 2023-05-22T15:27:51.685 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.675+0000 7fc7b742c640 1 -- 172.21.15.16:0/3983249745 shutdown_connections 2023-05-22T15:27:51.685 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.675+0000 7fc7b742c640 1 --2- 172.21.15.16:0/3983249745 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc7b00cec50 0x7fc7b00cf0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:51.685 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.675+0000 7fc7b742c640 1 --2- 172.21.15.16:0/3983249745 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc7b00b0950 0x7fc7b00b0dc0 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:51.685 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.675+0000 7fc7b742c640 1 --2- 172.21.15.16:0/3983249745 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fc794065380 0x7fc794067830 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:51.685 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.675+0000 7fc7b742c640 1 --2- 172.21.15.16:0/3983249745 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc79c003980 0x7fc7b014d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:51.685 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.675+0000 7fc7b742c640 1 -- 172.21.15.16:0/3983249745 >> 172.21.15.16:0/3983249745 conn(0x7fc7b00cc870 msgr2=0x7fc7b015d580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:51.686 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.675+0000 7fc7b742c640 1 -- 172.21.15.16:0/3983249745 shutdown_connections 2023-05-22T15:27:51.686 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.675+0000 7fc7b742c640 1 -- 172.21.15.16:0/3983249745 wait complete. 2023-05-22T15:27:51.700 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-05-22T15:27:51.817 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.811+0000 7f8c6c3d7640 1 Processor -- start 2023-05-22T15:27:51.817 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.811+0000 7f8c6c3d7640 1 -- start start 2023-05-22T15:27:51.818 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.811+0000 7f8c6c3d7640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8c640ade80 0x7f8c640cc400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:51.819 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.811+0000 7f8c6c3d7640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f8c6415c350 0x7f8c6415c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:51.819 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.811+0000 7f8c6c3d7640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c640c95f0 con 0x7f8c64158ca0 2023-05-22T15:27:51.819 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.811+0000 7f8c6c3d7640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c640ca270 con 0x7f8c6415c350 2023-05-22T15:27:51.819 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.811+0000 7f8c6c3d7640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c640ca690 con 0x7f8c640ade80 2023-05-22T15:27:51.819 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.811+0000 7f8c69972640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f8c6415c350 0x7f8c6415c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:51.819 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.811+0000 7f8c69972640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f8c6415c350 0x7f8c6415c7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54090/0 (socket says 172.21.15.16:54090) 2023-05-22T15:27:51.819 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.811+0000 7f8c69972640 1 -- 172.21.15.16:0/2313766100 learned_addr learned my addr 172.21.15.16:0/2313766100 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:51.820 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.811+0000 7f8c6a173640 1 --2- 172.21.15.16:0/2313766100 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8c640ade80 0x7f8c640cc400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:51.820 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.811+0000 7f8c69972640 1 -- 172.21.15.16:0/2313766100 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8c640ade80 msgr2=0x7f8c640cc400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:51.820 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.811+0000 7f8c69972640 1 --2- 172.21.15.16:0/2313766100 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8c640ade80 0x7f8c640cc400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:51.821 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.811+0000 7f8c69972640 1 -- 172.21.15.16:0/2313766100 >> v1:172.21.15.62:6789/0 conn(0x7f8c64158ca0 legacy=0x7f8c641590e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:51.821 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.811+0000 7f8c69972640 1 -- 172.21.15.16:0/2313766100 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c640c98d0 con 0x7f8c6415c350 2023-05-22T15:27:51.822 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.811+0000 7f8c69972640 1 --2- 172.21.15.16:0/2313766100 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8c6415c350 0x7f8c6415c7c0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f8c540098a0 tx=0x7f8c5401c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0fe90476d6e1481 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:51.823 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.815+0000 7f8c69171640 1 -- 172.21.15.16:0/2313766100 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c540a4ea0 con 0x7f8c6415c350 2023-05-22T15:27:51.823 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.815+0000 7f8c69171640 1 -- 172.21.15.16:0/2313766100 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c540a7050 con 0x7f8c6415c350 2023-05-22T15:27:51.823 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.815+0000 7f8c69171640 1 -- 172.21.15.16:0/2313766100 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c540ab820 con 0x7f8c6415c350 2023-05-22T15:27:51.824 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.815+0000 7f8c6c3d7640 1 -- 172.21.15.16:0/2313766100 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8c6415c350 msgr2=0x7f8c6415c7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:51.824 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.815+0000 7f8c6c3d7640 1 --2- 172.21.15.16:0/2313766100 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8c6415c350 0x7f8c6415c7c0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f8c540098a0 tx=0x7f8c5401c450 comp rx=0 tx=0).stop 2023-05-22T15:27:51.824 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.815+0000 7f8c6c3d7640 1 -- 172.21.15.16:0/2313766100 shutdown_connections 2023-05-22T15:27:51.824 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.815+0000 7f8c6c3d7640 1 --2- 172.21.15.16:0/2313766100 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8c6415c350 0x7f8c6415c7c0 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:51.824 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.815+0000 7f8c6c3d7640 1 --2- 172.21.15.16:0/2313766100 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8c640ade80 0x7f8c640cc400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:51.825 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.815+0000 7f8c6c3d7640 1 -- 172.21.15.16:0/2313766100 >> 172.21.15.16:0/2313766100 conn(0x7f8c640b1110 msgr2=0x7f8c640ce830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:51.825 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.815+0000 7f8c6c3d7640 1 -- 172.21.15.16:0/2313766100 shutdown_connections 2023-05-22T15:27:51.825 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.815+0000 7f8c6c3d7640 1 -- 172.21.15.16:0/2313766100 wait complete. 2023-05-22T15:27:51.827 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.815+0000 7f8c6c3d7640 1 Processor -- start 2023-05-22T15:27:51.827 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.815+0000 7f8c6c3d7640 1 -- start start 2023-05-22T15:27:51.827 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.815+0000 7f8c6c3d7640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8c640ade80 0x7f8c6414fd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:51.827 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.815+0000 7f8c6c3d7640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8c64158ca0 0x7f8c64159d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:51.827 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.815+0000 7f8c6c3d7640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8c6415c350 0x7f8c6406fef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:51.827 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.815+0000 7f8c6c3d7640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c6415adc0 con 0x7f8c64158ca0 2023-05-22T15:27:51.827 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.815+0000 7f8c6c3d7640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c6415a250 con 0x7f8c6415c350 2023-05-22T15:27:51.828 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.815+0000 7f8c6c3d7640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c6415a520 con 0x7f8c640ade80 2023-05-22T15:27:51.828 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.815+0000 7f8c69972640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8c64158ca0 0x7f8c64159d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:51.828 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.815+0000 7f8c69972640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8c64158ca0 0x7f8c64159d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54096/0 (socket says 172.21.15.16:54096) 2023-05-22T15:27:51.828 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.815+0000 7f8c69972640 1 -- 172.21.15.16:0/2975463596 learned_addr learned my addr 172.21.15.16:0/2975463596 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:51.829 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.815+0000 7f8c6a974640 1 --2- 172.21.15.16:0/2975463596 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8c6415c350 0x7f8c6406fef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:51.829 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.815+0000 7f8c6a173640 1 --2- 172.21.15.16:0/2975463596 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8c640ade80 0x7f8c6414fd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:51.829 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.819+0000 7f8c69972640 1 -- 172.21.15.16:0/2975463596 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8c640ade80 msgr2=0x7f8c6414fd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:51.829 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.819+0000 7f8c69972640 1 --2- 172.21.15.16:0/2975463596 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8c640ade80 0x7f8c6414fd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:51.829 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.819+0000 7f8c69972640 1 -- 172.21.15.16:0/2975463596 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8c6415c350 msgr2=0x7f8c6406fef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:51.830 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.819+0000 7f8c69972640 1 --2- 172.21.15.16:0/2975463596 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8c6415c350 0x7f8c6406fef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:51.830 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.819+0000 7f8c69972640 1 -- 172.21.15.16:0/2975463596 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c540094d0 con 0x7f8c64158ca0 2023-05-22T15:27:51.830 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.819+0000 7f8c6a974640 1 --2- 172.21.15.16:0/2975463596 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8c6415c350 0x7f8c6406fef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T15:27:51.830 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.819+0000 7f8c69972640 1 --2- 172.21.15.16:0/2975463596 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8c64158ca0 0x7f8c64159d10 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f8c5401ccf0 tx=0x7f8c540191f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:51.830 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.819+0000 7f8c5affd640 1 -- 172.21.15.16:0/2975463596 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c540b6620 con 0x7f8c64158ca0 2023-05-22T15:27:51.830 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.819+0000 7f8c5affd640 1 -- 172.21.15.16:0/2975463596 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c540b67a0 con 0x7f8c64158ca0 2023-05-22T15:27:51.830 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.819+0000 7f8c5affd640 1 -- 172.21.15.16:0/2975463596 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c540b6b50 con 0x7f8c64158ca0 2023-05-22T15:27:51.830 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.819+0000 7f8c6c3d7640 1 -- 172.21.15.16:0/2975463596 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c6415d090 con 0x7f8c64158ca0 2023-05-22T15:27:51.831 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.819+0000 7f8c6c3d7640 1 -- 172.21.15.16:0/2975463596 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c640cc940 con 0x7f8c64158ca0 2023-05-22T15:27:51.831 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.819+0000 7f8c6c3d7640 1 -- 172.21.15.16:0/2975463596 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c640c9210 con 0x7f8c64158ca0 2023-05-22T15:27:51.831 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.819+0000 7f8c5affd640 1 -- 172.21.15.16:0/2975463596 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7f8c540b2040 con 0x7f8c64158ca0 2023-05-22T15:27:51.832 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.819+0000 7f8c5affd640 1 --2- 172.21.15.16:0/2975463596 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f8c48065380 0x7f8c48067830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:51.832 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.819+0000 7f8c6a173640 1 --2- 172.21.15.16:0/2975463596 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f8c48065380 0x7f8c48067830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:51.832 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.819+0000 7f8c5affd640 1 -- 172.21.15.16:0/2975463596 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f8c5411d110 con 0x7f8c64158ca0 2023-05-22T15:27:51.832 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.819+0000 7f8c6a173640 1 --2- 172.21.15.16:0/2975463596 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f8c48065380 0x7f8c48067830 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f8c64070a40 tx=0x7f8c60008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:51.832 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.823+0000 7f8c5affd640 1 -- 172.21.15.16:0/2975463596 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f8c540ea9c0 con 0x7f8c64158ca0 2023-05-22T15:27:51.978 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.971+0000 7f8c6c3d7640 1 -- 172.21.15.16:0/2975463596 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f8c6404dbe0 con 0x7f8c64158ca0 2023-05-22T15:27:51.979 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.971+0000 7f8c5affd640 1 -- 172.21.15.16:0/2975463596 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v29) v1 ==== 92+0+10 (secure 0 0 0) 0x7f8c540edfe0 con 0x7f8c64158ca0 2023-05-22T15:27:51.979 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 8 2023-05-22T15:27:51.981 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.971+0000 7f8c6c3d7640 1 -- 172.21.15.16:0/2975463596 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f8c48065380 msgr2=0x7f8c48067830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:51.981 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.971+0000 7f8c6c3d7640 1 --2- 172.21.15.16:0/2975463596 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f8c48065380 0x7f8c48067830 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f8c64070a40 tx=0x7f8c60008040 comp rx=0 tx=0).stop 2023-05-22T15:27:51.981 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.971+0000 7f8c6c3d7640 1 -- 172.21.15.16:0/2975463596 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8c64158ca0 msgr2=0x7f8c64159d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:51.982 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.971+0000 7f8c6c3d7640 1 --2- 172.21.15.16:0/2975463596 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8c64158ca0 0x7f8c64159d10 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f8c5401ccf0 tx=0x7f8c540191f0 comp rx=0 tx=0).stop 2023-05-22T15:27:51.982 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.975+0000 7f8c6c3d7640 1 -- 172.21.15.16:0/2975463596 shutdown_connections 2023-05-22T15:27:51.982 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.975+0000 7f8c6c3d7640 1 --2- 172.21.15.16:0/2975463596 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8c6415c350 0x7f8c6406fef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:51.982 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.975+0000 7f8c6c3d7640 1 --2- 172.21.15.16:0/2975463596 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8c64158ca0 0x7f8c64159d10 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:51.982 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.975+0000 7f8c6c3d7640 1 --2- 172.21.15.16:0/2975463596 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f8c48065380 0x7f8c48067830 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:51.982 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.975+0000 7f8c6c3d7640 1 --2- 172.21.15.16:0/2975463596 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8c640ade80 0x7f8c6414fd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:51.982 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.975+0000 7f8c6c3d7640 1 -- 172.21.15.16:0/2975463596 >> 172.21.15.16:0/2975463596 conn(0x7f8c640b1110 msgr2=0x7f8c64151f10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:51.983 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.975+0000 7f8c6c3d7640 1 -- 172.21.15.16:0/2975463596 shutdown_connections 2023-05-22T15:27:51.983 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:51.975+0000 7f8c6c3d7640 1 -- 172.21.15.16:0/2975463596 wait complete. 2023-05-22T15:27:51.997 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-05-22T15:27:52.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.115+0000 7f57207ec640 1 Processor -- start 2023-05-22T15:27:52.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.115+0000 7f57207ec640 1 -- start start 2023-05-22T15:27:52.126 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.119+0000 7f57207ec640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5718159970 0x7f5718159d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:52.126 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.119+0000 7f57207ec640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f571815dad0 0x7f571806b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:52.126 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.119+0000 7f57207ec640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5718070c30 con 0x7f571815d680 2023-05-22T15:27:52.126 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.119+0000 7f57207ec640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57180cb970 con 0x7f571815dad0 2023-05-22T15:27:52.126 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.119+0000 7f57207ec640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5718070e60 con 0x7f5718159970 2023-05-22T15:27:52.127 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.119+0000 7f571dd87640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f571815d680 0x7f57180cb260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:33006/0 (socket says 172.21.15.16:33006) 2023-05-22T15:27:52.127 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.119+0000 7f571dd87640 1 -- 172.21.15.16:0/2611694592 learned_addr learned my addr 172.21.15.16:0/2611694592 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:27:52.127 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.119+0000 7f571ed89640 1 --2- 172.21.15.16:0/2611694592 >> v2:172.21.15.62:3300/0 conn(0x7f571815dad0 0x7f571806b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:52.127 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.119+0000 7f571e588640 1 --2- 172.21.15.16:0/2611694592 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5718159970 0x7f5718159d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:52.127 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.119+0000 7f571d586640 1 -- 172.21.15.16:0/2611694592 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2243998922 0 0) 0x7f5718070c30 con 0x7f571815d680 2023-05-22T15:27:52.127 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.119+0000 7f571d586640 1 -- 172.21.15.16:0/2611694592 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5704003530 con 0x7f571815d680 2023-05-22T15:27:52.129 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.119+0000 7f571d586640 1 -- 172.21.15.16:0/2611694592 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4151540820 0 0) 0x7f5704003530 con 0x7f571815d680 2023-05-22T15:27:52.129 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.119+0000 7f571d586640 1 -- 172.21.15.16:0/2611694592 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5718159970 msgr2=0x7f5718159d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:52.129 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.119+0000 7f571d586640 1 --2- 172.21.15.16:0/2611694592 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5718159970 0x7f5718159d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:52.129 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.119+0000 7f571d586640 1 -- 172.21.15.16:0/2611694592 >> v2:172.21.15.62:3300/0 conn(0x7f571815dad0 msgr2=0x7f571806b6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:52.129 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.119+0000 7f571d586640 1 --2- 172.21.15.16:0/2611694592 >> v2:172.21.15.62:3300/0 conn(0x7f571815dad0 0x7f571806b6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:52.129 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.119+0000 7f571d586640 1 -- 172.21.15.16:0/2611694592 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5718071140 con 0x7f571815d680 2023-05-22T15:27:52.130 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.119+0000 7f571d586640 1 -- 172.21.15.16:0/2611694592 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7f5708001680 con 0x7f571815d680 2023-05-22T15:27:52.130 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.119+0000 7f571d586640 1 -- 172.21.15.16:0/2611694592 >> v1:172.21.15.16:6789/0 conn(0x7f571815d680 legacy=0x7f57180cb260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:52.130 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.119+0000 7f571d586640 1 --2- 172.21.15.16:0/2611694592 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5718159970 0x7f5704003980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:52.130 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.119+0000 7f571d586640 1 --2- 172.21.15.16:0/2611694592 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5704003f00 0x7f57040047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:52.130 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.119+0000 7f571d586640 1 --2- 172.21.15.16:0/2611694592 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5704004d10 0x7f5704005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:52.130 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.119+0000 7f571d586640 1 -- 172.21.15.16:0/2611694592 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57180cb970 con 0x7f5704004d10 2023-05-22T15:27:52.130 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.119+0000 7f571d586640 1 -- 172.21.15.16:0/2611694592 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5718070e60 con 0x7f5704003f00 2023-05-22T15:27:52.130 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.119+0000 7f571d586640 1 -- 172.21.15.16:0/2611694592 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f570400b9c0 con 0x7f5718159970 2023-05-22T15:27:52.131 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.119+0000 7f571ed89640 1 --2- 172.21.15.16:0/2611694592 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5704004d10 0x7f5704005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:52.131 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.119+0000 7f571dd87640 1 --2- 172.21.15.16:0/2611694592 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5704003f00 0x7f57040047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:52.131 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.119+0000 7f571e588640 1 --2- 172.21.15.16:0/2611694592 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5718159970 0x7f5704003980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:52.131 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.119+0000 7f571ed89640 1 -- 172.21.15.16:0/2611694592 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5718159970 msgr2=0x7f5704003980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:52.131 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.119+0000 7f571ed89640 1 --2- 172.21.15.16:0/2611694592 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5718159970 0x7f5704003980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:52.131 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.119+0000 7f571ed89640 1 -- 172.21.15.16:0/2611694592 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5704003f00 msgr2=0x7f57040047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:52.131 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.119+0000 7f571ed89640 1 --2- 172.21.15.16:0/2611694592 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5704003f00 0x7f57040047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:52.132 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.119+0000 7f571ed89640 1 -- 172.21.15.16:0/2611694592 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5704003530 con 0x7f5704004d10 2023-05-22T15:27:52.134 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.123+0000 7f571ed89640 1 --2- 172.21.15.16:0/2611694592 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5704004d10 0x7f5704005180 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f571815a2a0 tx=0x7f571001cbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=499534ae9f17d804 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:52.134 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.123+0000 7f571d586640 1 -- 172.21.15.16:0/2611694592 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57100a5b40 con 0x7f5704004d10 2023-05-22T15:27:52.135 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.123+0000 7f571d586640 1 -- 172.21.15.16:0/2611694592 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f571000f040 con 0x7f5704004d10 2023-05-22T15:27:52.135 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.123+0000 7f571d586640 1 -- 172.21.15.16:0/2611694592 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57100ad440 con 0x7f5704004d10 2023-05-22T15:27:52.135 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.123+0000 7f57207ec640 1 -- 172.21.15.16:0/2611694592 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5704004d10 msgr2=0x7f5704005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:52.135 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.123+0000 7f57207ec640 1 --2- 172.21.15.16:0/2611694592 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5704004d10 0x7f5704005180 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f571815a2a0 tx=0x7f571001cbe0 comp rx=0 tx=0).stop 2023-05-22T15:27:52.135 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.123+0000 7f571ed89640 1 -- 172.21.15.16:0/2611694592 reap_dead start 2023-05-22T15:27:52.135 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.123+0000 7f57207ec640 1 -- 172.21.15.16:0/2611694592 shutdown_connections 2023-05-22T15:27:52.135 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.123+0000 7f57207ec640 1 -- 172.21.15.16:0/2611694592 >> 172.21.15.16:0/2611694592 conn(0x7f5718069720 msgr2=0x7f5718069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:52.135 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.123+0000 7f57207ec640 1 -- 172.21.15.16:0/2611694592 shutdown_connections 2023-05-22T15:27:52.136 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.123+0000 7f57207ec640 1 -- 172.21.15.16:0/2611694592 wait complete. 2023-05-22T15:27:52.136 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.123+0000 7f57207ec640 1 Processor -- start 2023-05-22T15:27:52.136 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.123+0000 7f57207ec640 1 -- start start 2023-05-22T15:27:52.136 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.127+0000 7f57207ec640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5704004d10 0x7f571814ec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:52.136 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.127+0000 7f57207ec640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f57181f30f0 0x7f5718154f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:52.136 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.127+0000 7f57207ec640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f57181506a0 0x7f5718150d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:52.136 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.127+0000 7f57207ec640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f571814f600 con 0x7f57181f30f0 2023-05-22T15:27:52.137 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.127+0000 7f57207ec640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f571814f8e0 con 0x7f57181506a0 2023-05-22T15:27:52.137 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.127+0000 7f57207ec640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f571814fbc0 con 0x7f5704004d10 2023-05-22T15:27:52.137 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.127+0000 7f571dd87640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f57181f30f0 0x7f5718154f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:52.137 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.127+0000 7f571dd87640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f57181f30f0 0x7f5718154f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54124/0 (socket says 172.21.15.16:54124) 2023-05-22T15:27:52.137 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.127+0000 7f571dd87640 1 -- 172.21.15.16:0/4045221010 learned_addr learned my addr 172.21.15.16:0/4045221010 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:52.137 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.127+0000 7f571ed89640 1 --2- 172.21.15.16:0/4045221010 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f57181506a0 0x7f5718150d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:52.137 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.127+0000 7f571dd87640 1 -- 172.21.15.16:0/4045221010 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5704004d10 msgr2=0x7f571814ec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:52.137 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.127+0000 7f571dd87640 1 --2- 172.21.15.16:0/4045221010 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5704004d10 0x7f571814ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:52.138 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.127+0000 7f571dd87640 1 -- 172.21.15.16:0/4045221010 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f57181506a0 msgr2=0x7f5718150d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:52.138 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.127+0000 7f571dd87640 1 --2- 172.21.15.16:0/4045221010 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f57181506a0 0x7f5718150d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:52.138 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.127+0000 7f571dd87640 1 -- 172.21.15.16:0/4045221010 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5718071700 con 0x7f57181f30f0 2023-05-22T15:27:52.138 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.127+0000 7f571ed89640 1 --2- 172.21.15.16:0/4045221010 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f57181506a0 0x7f5718150d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:27:52.138 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.127+0000 7f571dd87640 1 --2- 172.21.15.16:0/4045221010 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f57181f30f0 0x7f5718154f50 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f5708011980 tx=0x7f5708011e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:52.138 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.127+0000 7f570effd640 1 -- 172.21.15.16:0/4045221010 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f570800fc20 con 0x7f57181f30f0 2023-05-22T15:27:52.139 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.127+0000 7f570effd640 1 -- 172.21.15.16:0/4045221010 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f570800fda0 con 0x7f57181f30f0 2023-05-22T15:27:52.139 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.127+0000 7f57207ec640 1 -- 172.21.15.16:0/4045221010 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5718071420 con 0x7f57181f30f0 2023-05-22T15:27:52.139 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.127+0000 7f570effd640 1 -- 172.21.15.16:0/4045221010 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5708015650 con 0x7f57181f30f0 2023-05-22T15:27:52.139 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.127+0000 7f57207ec640 1 -- 172.21.15.16:0/4045221010 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57180cac90 con 0x7f57181f30f0 2023-05-22T15:27:52.140 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.127+0000 7f57207ec640 1 -- 172.21.15.16:0/4045221010 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f571804dbe0 con 0x7f57181f30f0 2023-05-22T15:27:52.140 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.127+0000 7f570effd640 1 -- 172.21.15.16:0/4045221010 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7f57080157b0 con 0x7f57181f30f0 2023-05-22T15:27:52.140 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.127+0000 7f570effd640 1 --2- 172.21.15.16:0/4045221010 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f56fc065380 0x7f56fc067830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:52.140 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.127+0000 7f570effd640 1 -- 172.21.15.16:0/4045221010 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f5708085fd0 con 0x7f57181f30f0 2023-05-22T15:27:52.141 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.127+0000 7f571e588640 1 --2- 172.21.15.16:0/4045221010 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f56fc065380 0x7f56fc067830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:52.141 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.127+0000 7f571e588640 1 --2- 172.21.15.16:0/4045221010 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f56fc065380 0x7f56fc067830 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f5714009930 tx=0x7f5714008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:52.141 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.131+0000 7f570effd640 1 -- 172.21.15.16:0/4045221010 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f5708053950 con 0x7f57181f30f0 2023-05-22T15:27:52.299 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.283+0000 7f57207ec640 1 -- 172.21.15.16:0/4045221010 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f57180b36e0 con 0x7f57181f30f0 2023-05-22T15:27:52.299 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.283+0000 7f570effd640 1 -- 172.21.15.16:0/4045221010 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v29) v1 ==== 104+0+11 (secure 0 0 0) 0x7f5708056f70 con 0x7f57181f30f0 2023-05-22T15:27:52.299 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2023-05-22T15:27:52.300 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.287+0000 7f57207ec640 1 -- 172.21.15.16:0/4045221010 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f56fc065380 msgr2=0x7f56fc067830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:52.300 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.287+0000 7f57207ec640 1 --2- 172.21.15.16:0/4045221010 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f56fc065380 0x7f56fc067830 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f5714009930 tx=0x7f5714008040 comp rx=0 tx=0).stop 2023-05-22T15:27:52.300 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.287+0000 7f57207ec640 1 -- 172.21.15.16:0/4045221010 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f57181f30f0 msgr2=0x7f5718154f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:52.301 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.287+0000 7f57207ec640 1 --2- 172.21.15.16:0/4045221010 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f57181f30f0 0x7f5718154f50 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f5708011980 tx=0x7f5708011e40 comp rx=0 tx=0).stop 2023-05-22T15:27:52.301 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.287+0000 7f57207ec640 1 -- 172.21.15.16:0/4045221010 shutdown_connections 2023-05-22T15:27:52.301 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.287+0000 7f57207ec640 1 --2- 172.21.15.16:0/4045221010 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f57181506a0 0x7f5718150d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:52.301 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.287+0000 7f57207ec640 1 --2- 172.21.15.16:0/4045221010 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f57181f30f0 0x7f5718154f50 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:52.301 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.287+0000 7f57207ec640 1 --2- 172.21.15.16:0/4045221010 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f56fc065380 0x7f56fc067830 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:52.302 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.287+0000 7f57207ec640 1 --2- 172.21.15.16:0/4045221010 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5704004d10 0x7f571814ec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:52.302 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.287+0000 7f57207ec640 1 -- 172.21.15.16:0/4045221010 >> 172.21.15.16:0/4045221010 conn(0x7f5718069720 msgr2=0x7f5718158bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:52.302 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.287+0000 7f57207ec640 1 -- 172.21.15.16:0/4045221010 shutdown_connections 2023-05-22T15:27:52.302 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.287+0000 7f57207ec640 1 -- 172.21.15.16:0/4045221010 wait complete. 2023-05-22T15:27:52.312 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-05-22T15:27:52.433 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.427+0000 7f8faa890640 1 Processor -- start 2023-05-22T15:27:52.434 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.427+0000 7f8faa890640 1 -- start start 2023-05-22T15:27:52.435 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.427+0000 7f8faa890640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f8fa406ee50 0x7f8fa406db00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:52.435 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.427+0000 7f8faa890640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fa406e040 con 0x7f8fa415cfa0 2023-05-22T15:27:52.435 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.427+0000 7f8faa890640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fa406e490 con 0x7f8fa40cf870 2023-05-22T15:27:52.435 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.427+0000 7f8faa890640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8fa40cb9f0 con 0x7f8fa406ee50 2023-05-22T15:27:52.435 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.427+0000 7f8fa3fff640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f8fa415cfa0 0x7f8fa40b1fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:33022/0 (socket says 172.21.15.16:33022) 2023-05-22T15:27:52.436 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.427+0000 7f8fa3fff640 1 -- 172.21.15.16:0/670877829 learned_addr learned my addr 172.21.15.16:0/670877829 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:27:52.436 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.427+0000 7f8fa8e2d640 1 --2- 172.21.15.16:0/670877829 >> v2:172.21.15.62:3300/0 conn(0x7f8fa406ee50 0x7f8fa406db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:52.436 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.427+0000 7f8fa2ffd640 1 -- 172.21.15.16:0/670877829 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 682951236 0 0) 0x7f8f94001120 con 0x7f8fa415cfa0 2023-05-22T15:27:52.436 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.427+0000 7f8fa2ffd640 1 -- 172.21.15.16:0/670877829 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f8c003530 con 0x7f8fa415cfa0 2023-05-22T15:27:52.438 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.427+0000 7f8fa2ffd640 1 -- 172.21.15.16:0/670877829 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4052951435 0 0) 0x7f8f8c003530 con 0x7f8fa415cfa0 2023-05-22T15:27:52.438 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.427+0000 7f8fa2ffd640 1 -- 172.21.15.16:0/670877829 >> v2:172.21.15.62:3300/0 conn(0x7f8fa406ee50 msgr2=0x7f8fa406db00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:52.438 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.427+0000 7f8fa2ffd640 1 --2- 172.21.15.16:0/670877829 >> v2:172.21.15.62:3300/0 conn(0x7f8fa406ee50 0x7f8fa406db00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:52.438 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.427+0000 7f8fa2ffd640 1 -- 172.21.15.16:0/670877829 >> v1:172.21.15.62:6789/0 conn(0x7f8fa40cf870 legacy=0x7f8fa406e740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:52.438 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.427+0000 7f8fa2ffd640 1 -- 172.21.15.16:0/670877829 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fa40cad70 con 0x7f8fa415cfa0 2023-05-22T15:27:52.438 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.427+0000 7f8fa2ffd640 1 -- 172.21.15.16:0/670877829 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7f8f94001810 con 0x7f8fa415cfa0 2023-05-22T15:27:52.439 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.427+0000 7f8fa2ffd640 1 -- 172.21.15.16:0/670877829 >> v1:172.21.15.16:6789/0 conn(0x7f8fa415cfa0 legacy=0x7f8fa40b1fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:52.439 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.427+0000 7f8fa2ffd640 1 --2- 172.21.15.16:0/670877829 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8f8c003980 0x7f8f8c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:52.439 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.427+0000 7f8fa2ffd640 1 --2- 172.21.15.16:0/670877829 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8f8c0047d0 0x7f8f8c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:52.439 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.427+0000 7f8fa2ffd640 1 --2- 172.21.15.16:0/670877829 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8f8c005180 0x7f8f8c00b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:52.439 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.427+0000 7f8fa2ffd640 1 -- 172.21.15.16:0/670877829 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fa40cb9f0 con 0x7f8f8c0047d0 2023-05-22T15:27:52.440 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.427+0000 7f8fa2ffd640 1 -- 172.21.15.16:0/670877829 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f8c00c000 con 0x7f8f8c005180 2023-05-22T15:27:52.440 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.427+0000 7f8fa2ffd640 1 -- 172.21.15.16:0/670877829 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f8c00c1c0 con 0x7f8f8c003980 2023-05-22T15:27:52.440 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.427+0000 7f8fa37fe640 1 --2- 172.21.15.16:0/670877829 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8f8c0047d0 0x7f8f8c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:52.440 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.427+0000 7f8fa3fff640 1 --2- 172.21.15.16:0/670877829 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8f8c003980 0x7f8f8c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:52.440 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.427+0000 7f8fa8e2d640 1 --2- 172.21.15.16:0/670877829 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8f8c005180 0x7f8f8c00b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:52.440 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.427+0000 7f8fa37fe640 1 -- 172.21.15.16:0/670877829 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8f8c003980 msgr2=0x7f8f8c003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:52.440 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.427+0000 7f8fa37fe640 1 --2- 172.21.15.16:0/670877829 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8f8c003980 0x7f8f8c003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:52.441 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.427+0000 7f8fa37fe640 1 -- 172.21.15.16:0/670877829 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8f8c005180 msgr2=0x7f8f8c00b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:52.441 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.427+0000 7f8fa37fe640 1 --2- 172.21.15.16:0/670877829 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8f8c005180 0x7f8f8c00b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:52.441 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.427+0000 7f8fa37fe640 1 -- 172.21.15.16:0/670877829 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f8c003530 con 0x7f8f8c0047d0 2023-05-22T15:27:52.441 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.427+0000 7f8fa3fff640 1 --2- 172.21.15.16:0/670877829 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8f8c003980 0x7f8f8c003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:27:52.441 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.427+0000 7f8fa8e2d640 1 --2- 172.21.15.16:0/670877829 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8f8c005180 0x7f8f8c00b990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:27:52.441 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.427+0000 7f8fa8e2d640 1 -- 172.21.15.16:0/670877829 reap_dead start 2023-05-22T15:27:52.441 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.431+0000 7f8fa37fe640 1 --2- 172.21.15.16:0/670877829 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8f8c0047d0 0x7f8f8c004c40 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f8f98001780 tx=0x7f8f9801c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=728972eff18dc71b server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:52.444 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.431+0000 7f8fa2ffd640 1 -- 172.21.15.16:0/670877829 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f980052d0 con 0x7f8f8c0047d0 2023-05-22T15:27:52.445 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.431+0000 7f8fa2ffd640 1 -- 172.21.15.16:0/670877829 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f980a7050 con 0x7f8f8c0047d0 2023-05-22T15:27:52.445 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.431+0000 7f8fa2ffd640 1 -- 172.21.15.16:0/670877829 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f98003600 con 0x7f8f8c0047d0 2023-05-22T15:27:52.445 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.431+0000 7f8faa890640 1 -- 172.21.15.16:0/670877829 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8f8c0047d0 msgr2=0x7f8f8c004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:52.445 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.431+0000 7f8faa890640 1 --2- 172.21.15.16:0/670877829 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8f8c0047d0 0x7f8f8c004c40 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f8f98001780 tx=0x7f8f9801c3e0 comp rx=0 tx=0).stop 2023-05-22T15:27:52.445 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.431+0000 7f8faa890640 1 -- 172.21.15.16:0/670877829 shutdown_connections 2023-05-22T15:27:52.445 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.431+0000 7f8faa890640 1 --2- 172.21.15.16:0/670877829 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8f8c0047d0 0x7f8f8c004c40 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:52.445 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.431+0000 7f8faa890640 1 -- 172.21.15.16:0/670877829 >> 172.21.15.16:0/670877829 conn(0x7f8fa40cc870 msgr2=0x7f8fa40cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:52.446 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.431+0000 7f8faa890640 1 -- 172.21.15.16:0/670877829 shutdown_connections 2023-05-22T15:27:52.446 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.431+0000 7f8faa890640 1 -- 172.21.15.16:0/670877829 wait complete. 2023-05-22T15:27:52.446 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.431+0000 7f8faa890640 1 Processor -- start 2023-05-22T15:27:52.446 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.431+0000 7f8faa890640 1 -- start start 2023-05-22T15:27:52.446 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.435+0000 7f8faa890640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8f8c0047d0 0x7f8fa40ab7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:52.446 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.435+0000 7f8faa890640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8fa40ccd30 0x7f8fa40ad610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:52.447 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.435+0000 7f8faa890640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8fa4157ba0 0x7f8fa40ce1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:52.447 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.435+0000 7f8faa890640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fa40cd1c0 con 0x7f8f8c0047d0 2023-05-22T15:27:52.447 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.435+0000 7f8faa890640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fa40cd4a0 con 0x7f8fa4157ba0 2023-05-22T15:27:52.447 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.435+0000 7f8faa890640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fa40cd780 con 0x7f8fa40ccd30 2023-05-22T15:27:52.448 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.435+0000 7f8fa3fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8f8c0047d0 0x7f8fa40ab7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:52.448 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.435+0000 7f8fa3fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8f8c0047d0 0x7f8fa40ab7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54146/0 (socket says 172.21.15.16:54146) 2023-05-22T15:27:52.448 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.435+0000 7f8fa3fff640 1 -- 172.21.15.16:0/4012000763 learned_addr learned my addr 172.21.15.16:0/4012000763 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:52.448 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.435+0000 7f8fa8e2d640 1 --2- 172.21.15.16:0/4012000763 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8fa4157ba0 0x7f8fa40ce1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:52.448 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.435+0000 7f8fa37fe640 1 --2- 172.21.15.16:0/4012000763 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8fa40ccd30 0x7f8fa40ad610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:52.448 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.435+0000 7f8fa3fff640 1 -- 172.21.15.16:0/4012000763 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8fa40ccd30 msgr2=0x7f8fa40ad610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:52.448 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.435+0000 7f8fa3fff640 1 --2- 172.21.15.16:0/4012000763 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8fa40ccd30 0x7f8fa40ad610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:52.449 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.435+0000 7f8fa3fff640 1 -- 172.21.15.16:0/4012000763 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8fa4157ba0 msgr2=0x7f8fa40ce1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:52.449 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.435+0000 7f8fa3fff640 1 --2- 172.21.15.16:0/4012000763 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8fa4157ba0 0x7f8fa40ce1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:52.449 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.435+0000 7f8fa3fff640 1 -- 172.21.15.16:0/4012000763 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fa40cb330 con 0x7f8f8c0047d0 2023-05-22T15:27:52.449 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.435+0000 7f8fa8e2d640 1 --2- 172.21.15.16:0/4012000763 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8fa4157ba0 0x7f8fa40ce1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:27:52.449 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.435+0000 7f8fa37fe640 1 --2- 172.21.15.16:0/4012000763 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8fa40ccd30 0x7f8fa40ad610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:27:52.449 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.435+0000 7f8fa3fff640 1 --2- 172.21.15.16:0/4012000763 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8f8c0047d0 0x7f8fa40ab7d0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f8f94000fc0 tx=0x7f8f94010d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:52.449 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.435+0000 7f8fa0ff9640 1 -- 172.21.15.16:0/4012000763 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f9401a070 con 0x7f8f8c0047d0 2023-05-22T15:27:52.450 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.435+0000 7f8fa0ff9640 1 -- 172.21.15.16:0/4012000763 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f9400e8f0 con 0x7f8f8c0047d0 2023-05-22T15:27:52.450 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.435+0000 7f8faa890640 1 -- 172.21.15.16:0/4012000763 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8fa40cb050 con 0x7f8f8c0047d0 2023-05-22T15:27:52.450 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.435+0000 7f8fa0ff9640 1 -- 172.21.15.16:0/4012000763 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f9400ec80 con 0x7f8f8c0047d0 2023-05-22T15:27:52.450 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.435+0000 7f8faa890640 1 -- 172.21.15.16:0/4012000763 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8fa40cb610 con 0x7f8f8c0047d0 2023-05-22T15:27:52.450 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.435+0000 7f8faa890640 1 -- 172.21.15.16:0/4012000763 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8fa404dbe0 con 0x7f8f8c0047d0 2023-05-22T15:27:52.450 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.435+0000 7f8fa0ff9640 1 -- 172.21.15.16:0/4012000763 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7f8f9401f410 con 0x7f8f8c0047d0 2023-05-22T15:27:52.450 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.435+0000 7f8fa0ff9640 1 --2- 172.21.15.16:0/4012000763 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f8f90065380 0x7f8f90067830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:52.451 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.435+0000 7f8fa37fe640 1 --2- 172.21.15.16:0/4012000763 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f8f90065380 0x7f8f90067830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:52.451 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.435+0000 7f8fa0ff9640 1 -- 172.21.15.16:0/4012000763 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f8f94085920 con 0x7f8f8c0047d0 2023-05-22T15:27:52.451 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.435+0000 7f8fa37fe640 1 --2- 172.21.15.16:0/4012000763 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f8f90065380 0x7f8f90067830 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f8f98001750 tx=0x7f8f980011a0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:52.451 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.439+0000 7f8fa0ff9640 1 -- 172.21.15.16:0/4012000763 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f8f94053210 con 0x7f8f8c0047d0 2023-05-22T15:27:52.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.591+0000 7f8faa890640 1 -- 172.21.15.16:0/4012000763 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f8fa406e040 con 0x7f8f8c0047d0 2023-05-22T15:27:52.600 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.591+0000 7f8fa0ff9640 1 -- 172.21.15.16:0/4012000763 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v29) v1 ==== 100+0+11 (secure 0 0 0) 0x7f8fa406e040 con 0x7f8f8c0047d0 2023-05-22T15:27:52.600 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 64 2023-05-22T15:27:52.602 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.595+0000 7f8faa890640 1 -- 172.21.15.16:0/4012000763 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f8f90065380 msgr2=0x7f8f90067830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:52.603 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.595+0000 7f8faa890640 1 --2- 172.21.15.16:0/4012000763 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f8f90065380 0x7f8f90067830 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f8f98001750 tx=0x7f8f980011a0 comp rx=0 tx=0).stop 2023-05-22T15:27:52.603 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.595+0000 7f8faa890640 1 -- 172.21.15.16:0/4012000763 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8f8c0047d0 msgr2=0x7f8fa40ab7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:52.603 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.595+0000 7f8faa890640 1 --2- 172.21.15.16:0/4012000763 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8f8c0047d0 0x7f8fa40ab7d0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f8f94000fc0 tx=0x7f8f94010d20 comp rx=0 tx=0).stop 2023-05-22T15:27:52.603 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.595+0000 7f8faa890640 1 -- 172.21.15.16:0/4012000763 shutdown_connections 2023-05-22T15:27:52.603 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.595+0000 7f8faa890640 1 --2- 172.21.15.16:0/4012000763 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8fa4157ba0 0x7f8fa40ce1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:52.604 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.595+0000 7f8faa890640 1 --2- 172.21.15.16:0/4012000763 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f8f90065380 0x7f8f90067830 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:52.604 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.595+0000 7f8faa890640 1 --2- 172.21.15.16:0/4012000763 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8fa40ccd30 0x7f8fa40ad610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:52.604 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.595+0000 7f8faa890640 1 --2- 172.21.15.16:0/4012000763 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8f8c0047d0 0x7f8fa40ab7d0 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:52.604 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.595+0000 7f8faa890640 1 -- 172.21.15.16:0/4012000763 >> 172.21.15.16:0/4012000763 conn(0x7f8fa40cc870 msgr2=0x7f8fa406e740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:52.604 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.595+0000 7f8faa890640 1 -- 172.21.15.16:0/4012000763 shutdown_connections 2023-05-22T15:27:52.604 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.595+0000 7f8faa890640 1 -- 172.21.15.16:0/4012000763 wait complete. 2023-05-22T15:27:52.618 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-05-22T15:27:52.737 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.731+0000 7f7e6a44b640 1 Processor -- start 2023-05-22T15:27:52.738 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.731+0000 7f7e6a44b640 1 -- start start 2023-05-22T15:27:52.739 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.731+0000 7f7e6a44b640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f7e640cb4c0 0x7f7e64158fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:52.739 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.731+0000 7f7e6a44b640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e64159f70 con 0x7f7e64159510 2023-05-22T15:27:52.739 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.731+0000 7f7e6a44b640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e640cb030 con 0x7f7e6406b6c0 2023-05-22T15:27:52.739 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.731+0000 7f7e6a44b640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e64071760 con 0x7f7e640cb4c0 2023-05-22T15:27:52.739 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.731+0000 7f7e637fe640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f7e640cb4c0 0x7f7e64158fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:52.739 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.731+0000 7f7e637fe640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f7e640cb4c0 0x7f7e64158fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54162/0 (socket says 172.21.15.16:54162) 2023-05-22T15:27:52.740 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.731+0000 7f7e637fe640 1 -- 172.21.15.16:0/2562003425 learned_addr learned my addr 172.21.15.16:0/2562003425 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:52.740 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.731+0000 7f7e637fe640 1 -- 172.21.15.16:0/2562003425 >> v1:172.21.15.62:6789/0 conn(0x7f7e6406b6c0 legacy=0x7f7e6406ba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:52.740 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.731+0000 7f7e637fe640 1 -- 172.21.15.16:0/2562003425 >> v1:172.21.15.16:6789/0 conn(0x7f7e64159510 legacy=0x7f7e6406fba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:52.740 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.731+0000 7f7e637fe640 1 -- 172.21.15.16:0/2562003425 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e64159c90 con 0x7f7e640cb4c0 2023-05-22T15:27:52.740 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.735+0000 7f7e637fe640 1 --2- 172.21.15.16:0/2562003425 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e640cb4c0 0x7f7e64158fd0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f7e58009870 tx=0x7f7e5801c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5725d7498e1c37bc server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:52.741 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.735+0000 7f7e62ffd640 1 -- 172.21.15.16:0/2562003425 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e580ab600 con 0x7f7e640cb4c0 2023-05-22T15:27:52.741 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.735+0000 7f7e62ffd640 1 -- 172.21.15.16:0/2562003425 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e580a7050 con 0x7f7e640cb4c0 2023-05-22T15:27:52.741 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.735+0000 7f7e62ffd640 1 -- 172.21.15.16:0/2562003425 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e580ab980 con 0x7f7e640cb4c0 2023-05-22T15:27:52.742 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.735+0000 7f7e6a44b640 1 -- 172.21.15.16:0/2562003425 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e640cb4c0 msgr2=0x7f7e64158fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:52.742 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.735+0000 7f7e6a44b640 1 --2- 172.21.15.16:0/2562003425 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e640cb4c0 0x7f7e64158fd0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f7e58009870 tx=0x7f7e5801c500 comp rx=0 tx=0).stop 2023-05-22T15:27:52.742 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.735+0000 7f7e6a44b640 1 -- 172.21.15.16:0/2562003425 shutdown_connections 2023-05-22T15:27:52.742 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.735+0000 7f7e6a44b640 1 --2- 172.21.15.16:0/2562003425 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e640cb4c0 0x7f7e64158fd0 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:52.742 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.735+0000 7f7e6a44b640 1 -- 172.21.15.16:0/2562003425 >> 172.21.15.16:0/2562003425 conn(0x7f7e64069720 msgr2=0x7f7e64069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:52.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.735+0000 7f7e6a44b640 1 -- 172.21.15.16:0/2562003425 shutdown_connections 2023-05-22T15:27:52.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.735+0000 7f7e6a44b640 1 -- 172.21.15.16:0/2562003425 wait complete. 2023-05-22T15:27:52.744 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.735+0000 7f7e6a44b640 1 Processor -- start 2023-05-22T15:27:52.744 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.735+0000 7f7e6a44b640 1 -- start start 2023-05-22T15:27:52.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.735+0000 7f7e6a44b640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7e6406b6c0 0x7f7e6414a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:52.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.735+0000 7f7e6a44b640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7e640cb4c0 0x7f7e640adab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:52.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.735+0000 7f7e6a44b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e64159510 0x7f7e64148310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:52.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.735+0000 7f7e6a44b640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e640afa00 con 0x7f7e64159510 2023-05-22T15:27:52.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.735+0000 7f7e6a44b640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e640afce0 con 0x7f7e6406b6c0 2023-05-22T15:27:52.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.735+0000 7f7e6a44b640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e640affc0 con 0x7f7e640cb4c0 2023-05-22T15:27:52.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.735+0000 7f7e689e8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e64159510 0x7f7e64148310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:52.746 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.735+0000 7f7e689e8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e64159510 0x7f7e64148310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54168/0 (socket says 172.21.15.16:54168) 2023-05-22T15:27:52.746 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.735+0000 7f7e689e8640 1 -- 172.21.15.16:0/4265454689 learned_addr learned my addr 172.21.15.16:0/4265454689 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:52.746 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.735+0000 7f7e63fff640 1 --2- 172.21.15.16:0/4265454689 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7e6406b6c0 0x7f7e6414a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:52.746 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.735+0000 7f7e689e8640 1 -- 172.21.15.16:0/4265454689 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7e640cb4c0 msgr2=0x7f7e640adab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:52.746 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.735+0000 7f7e637fe640 1 --2- 172.21.15.16:0/4265454689 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7e640cb4c0 0x7f7e640adab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:52.746 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.735+0000 7f7e689e8640 1 --2- 172.21.15.16:0/4265454689 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7e640cb4c0 0x7f7e640adab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:52.746 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.735+0000 7f7e689e8640 1 -- 172.21.15.16:0/4265454689 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7e6406b6c0 msgr2=0x7f7e6414a010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:52.747 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.735+0000 7f7e689e8640 1 --2- 172.21.15.16:0/4265454689 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7e6406b6c0 0x7f7e6414a010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:52.747 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.735+0000 7f7e689e8640 1 -- 172.21.15.16:0/4265454689 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e580094d0 con 0x7f7e64159510 2023-05-22T15:27:52.747 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.735+0000 7f7e689e8640 1 --2- 172.21.15.16:0/4265454689 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e64159510 0x7f7e64148310 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f7e5c00c730 tx=0x7f7e5c00cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:52.747 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.735+0000 7f7e60ff9640 1 -- 172.21.15.16:0/4265454689 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e5c003b80 con 0x7f7e64159510 2023-05-22T15:27:52.747 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.735+0000 7f7e60ff9640 1 -- 172.21.15.16:0/4265454689 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e5c003ce0 con 0x7f7e64159510 2023-05-22T15:27:52.748 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.735+0000 7f7e6a44b640 1 -- 172.21.15.16:0/4265454689 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e64156840 con 0x7f7e64159510 2023-05-22T15:27:52.748 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.735+0000 7f7e60ff9640 1 -- 172.21.15.16:0/4265454689 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e5c00aa30 con 0x7f7e64159510 2023-05-22T15:27:52.748 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.735+0000 7f7e6a44b640 1 -- 172.21.15.16:0/4265454689 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e64159910 con 0x7f7e64159510 2023-05-22T15:27:52.749 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.739+0000 7f7e6a44b640 1 -- 172.21.15.16:0/4265454689 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e641569d0 con 0x7f7e64159510 2023-05-22T15:27:52.749 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.739+0000 7f7e60ff9640 1 -- 172.21.15.16:0/4265454689 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7f7e5c00ab90 con 0x7f7e64159510 2023-05-22T15:27:52.749 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.739+0000 7f7e60ff9640 1 --2- 172.21.15.16:0/4265454689 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f7e50065380 0x7f7e50067830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:52.749 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.739+0000 7f7e60ff9640 1 -- 172.21.15.16:0/4265454689 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f7e5c083400 con 0x7f7e64159510 2023-05-22T15:27:52.749 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.739+0000 7f7e63fff640 1 --2- 172.21.15.16:0/4265454689 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f7e50065380 0x7f7e50067830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:52.749 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.739+0000 7f7e63fff640 1 --2- 172.21.15.16:0/4265454689 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f7e50065380 0x7f7e50067830 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f7e54004e30 tx=0x7f7e54009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:52.750 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.743+0000 7f7e60ff9640 1 -- 172.21.15.16:0/4265454689 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f7e5c050e30 con 0x7f7e64159510 2023-05-22T15:27:52.900 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.891+0000 7f7e6a44b640 1 -- 172.21.15.16:0/4265454689 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f7e640b04e0 con 0x7f7e64159510 2023-05-22T15:27:52.900 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.891+0000 7f7e60ff9640 1 -- 172.21.15.16:0/4265454689 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v29) v1 ==== 103+0+11 (secure 0 0 0) 0x7f7e640b04e0 con 0x7f7e64159510 2023-05-22T15:27:52.901 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2023-05-22T15:27:52.903 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.895+0000 7f7e6a44b640 1 -- 172.21.15.16:0/4265454689 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f7e50065380 msgr2=0x7f7e50067830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:52.903 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.895+0000 7f7e6a44b640 1 --2- 172.21.15.16:0/4265454689 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f7e50065380 0x7f7e50067830 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f7e54004e30 tx=0x7f7e54009040 comp rx=0 tx=0).stop 2023-05-22T15:27:52.903 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.895+0000 7f7e6a44b640 1 -- 172.21.15.16:0/4265454689 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e64159510 msgr2=0x7f7e64148310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:52.904 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.895+0000 7f7e6a44b640 1 --2- 172.21.15.16:0/4265454689 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e64159510 0x7f7e64148310 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f7e5c00c730 tx=0x7f7e5c00cbf0 comp rx=0 tx=0).stop 2023-05-22T15:27:52.904 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.895+0000 7f7e6a44b640 1 -- 172.21.15.16:0/4265454689 shutdown_connections 2023-05-22T15:27:52.904 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.895+0000 7f7e6a44b640 1 --2- 172.21.15.16:0/4265454689 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e64159510 0x7f7e64148310 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:52.904 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.895+0000 7f7e6a44b640 1 --2- 172.21.15.16:0/4265454689 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f7e50065380 0x7f7e50067830 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:52.904 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.895+0000 7f7e6a44b640 1 --2- 172.21.15.16:0/4265454689 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7e640cb4c0 0x7f7e640adab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:52.904 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.895+0000 7f7e6a44b640 1 --2- 172.21.15.16:0/4265454689 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7e6406b6c0 0x7f7e6414a010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:52.905 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.895+0000 7f7e6a44b640 1 -- 172.21.15.16:0/4265454689 >> 172.21.15.16:0/4265454689 conn(0x7f7e64069720 msgr2=0x7f7e6406ba90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:52.905 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.895+0000 7f7e6a44b640 1 -- 172.21.15.16:0/4265454689 shutdown_connections 2023-05-22T15:27:52.905 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:52.895+0000 7f7e6a44b640 1 -- 172.21.15.16:0/4265454689 wait complete. 2023-05-22T15:27:52.919 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-05-22T15:27:53.033 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.027+0000 7f6c81696640 1 Processor -- start 2023-05-22T15:27:53.033 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.027+0000 7f6c81696640 1 -- start start 2023-05-22T15:27:53.034 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.027+0000 7f6c81696640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6c7c0b7dd0 0x7f6c7c0ccab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:53.034 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.027+0000 7f6c81696640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f6c7c15f700 0x7f6c7c15fb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:53.034 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.027+0000 7f6c81696640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6c7c160050 0x7f6c7c0ce720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:53.035 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.027+0000 7f6c81696640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c7c0d3360 con 0x7f6c7c160050 2023-05-22T15:27:53.035 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.027+0000 7f6c81696640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c7c0cd170 con 0x7f6c7c15f700 2023-05-22T15:27:53.035 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.027+0000 7f6c81696640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c7c0d2ac0 con 0x7f6c7c0b7dd0 2023-05-22T15:27:53.036 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.027+0000 7f6c7b7fe640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6c7c160050 0x7f6c7c0ce720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:53.036 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.027+0000 7f6c7b7fe640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6c7c160050 0x7f6c7c0ce720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54184/0 (socket says 172.21.15.16:54184) 2023-05-22T15:27:53.036 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.027+0000 7f6c7b7fe640 1 -- 172.21.15.16:0/1949619907 learned_addr learned my addr 172.21.15.16:0/1949619907 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:53.036 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.027+0000 7f6c7a7fc640 1 --2- 172.21.15.16:0/1949619907 >> v2:172.21.15.62:3300/0 conn(0x7f6c7c15f700 0x7f6c7c15fb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:53.036 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.027+0000 7f6c7affd640 1 --2- 172.21.15.16:0/1949619907 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6c7c0b7dd0 0x7f6c7c0ccab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:53.036 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.027+0000 7f6c7b7fe640 1 -- 172.21.15.16:0/1949619907 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6c7c0b7dd0 msgr2=0x7f6c7c0ccab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:53.037 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.027+0000 7f6c7b7fe640 1 --2- 172.21.15.16:0/1949619907 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6c7c0b7dd0 0x7f6c7c0ccab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:53.038 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.027+0000 7f6c7b7fe640 1 -- 172.21.15.16:0/1949619907 >> v2:172.21.15.62:3300/0 conn(0x7f6c7c15f700 msgr2=0x7f6c7c15fb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:53.038 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.027+0000 7f6c7b7fe640 1 --2- 172.21.15.16:0/1949619907 >> v2:172.21.15.62:3300/0 conn(0x7f6c7c15f700 0x7f6c7c15fb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:53.038 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.027+0000 7f6c7b7fe640 1 -- 172.21.15.16:0/1949619907 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c7c0d2da0 con 0x7f6c7c160050 2023-05-22T15:27:53.038 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.027+0000 7f6c7a7fc640 1 --2- 172.21.15.16:0/1949619907 >> v2:172.21.15.62:3300/0 conn(0x7f6c7c15f700 0x7f6c7c15fb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T15:27:53.043 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.031+0000 7f6c7b7fe640 1 --2- 172.21.15.16:0/1949619907 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c7c160050 0x7f6c7c0ce720 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f6c7c0d34c0 tx=0x7f6c7401c9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=419ba77d9db52e6 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:53.043 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.031+0000 7f6c79ffb640 1 -- 172.21.15.16:0/1949619907 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c740a5840 con 0x7f6c7c160050 2023-05-22T15:27:53.043 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.031+0000 7f6c79ffb640 1 -- 172.21.15.16:0/1949619907 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c740a59a0 con 0x7f6c7c160050 2023-05-22T15:27:53.044 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.031+0000 7f6c79ffb640 1 -- 172.21.15.16:0/1949619907 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c740a5d30 con 0x7f6c7c160050 2023-05-22T15:27:53.044 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.031+0000 7f6c81696640 1 -- 172.21.15.16:0/1949619907 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c7c160050 msgr2=0x7f6c7c0ce720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:53.044 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.031+0000 7f6c81696640 1 --2- 172.21.15.16:0/1949619907 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c7c160050 0x7f6c7c0ce720 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f6c7c0d34c0 tx=0x7f6c7401c9a0 comp rx=0 tx=0).stop 2023-05-22T15:27:53.044 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.031+0000 7f6c81696640 1 -- 172.21.15.16:0/1949619907 shutdown_connections 2023-05-22T15:27:53.044 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.031+0000 7f6c81696640 1 --2- 172.21.15.16:0/1949619907 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c7c160050 0x7f6c7c0ce720 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:53.044 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.031+0000 7f6c81696640 1 --2- 172.21.15.16:0/1949619907 >> v2:172.21.15.62:3300/0 conn(0x7f6c7c15f700 0x7f6c7c15fb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:53.044 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.031+0000 7f6c81696640 1 --2- 172.21.15.16:0/1949619907 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6c7c0b7dd0 0x7f6c7c0ccab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:53.045 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.031+0000 7f6c81696640 1 -- 172.21.15.16:0/1949619907 >> 172.21.15.16:0/1949619907 conn(0x7f6c7c0655c0 msgr2=0x7f6c7c070e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:53.045 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.031+0000 7f6c81696640 1 -- 172.21.15.16:0/1949619907 shutdown_connections 2023-05-22T15:27:53.045 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.031+0000 7f6c81696640 1 -- 172.21.15.16:0/1949619907 wait complete. 2023-05-22T15:27:53.045 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.031+0000 7f6c81696640 1 Processor -- start 2023-05-22T15:27:53.045 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.031+0000 7f6c81696640 1 -- start start 2023-05-22T15:27:53.045 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.031+0000 7f6c81696640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6c7c0b7dd0 0x7f6c7c0ca5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:53.045 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.031+0000 7f6c81696640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c7c15f700 0x7f6c7c0b6000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:53.045 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.031+0000 7f6c81696640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6c7c160050 0x7f6c7c15dc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:53.046 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.031+0000 7f6c81696640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c7c157940 con 0x7f6c7c15f700 2023-05-22T15:27:53.046 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.031+0000 7f6c81696640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c7c0c9530 con 0x7f6c7c160050 2023-05-22T15:27:53.046 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.031+0000 7f6c81696640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c7c0c9810 con 0x7f6c7c0b7dd0 2023-05-22T15:27:53.046 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.031+0000 7f6c7a7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c7c15f700 0x7f6c7c0b6000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:53.046 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.031+0000 7f6c7a7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c7c15f700 0x7f6c7c0b6000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54188/0 (socket says 172.21.15.16:54188) 2023-05-22T15:27:53.046 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.031+0000 7f6c7a7fc640 1 -- 172.21.15.16:0/3168343850 learned_addr learned my addr 172.21.15.16:0/3168343850 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:53.046 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.031+0000 7f6c7b7fe640 1 --2- 172.21.15.16:0/3168343850 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6c7c160050 0x7f6c7c15dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:53.047 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.031+0000 7f6c7affd640 1 --2- 172.21.15.16:0/3168343850 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6c7c0b7dd0 0x7f6c7c0ca5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:53.047 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.031+0000 7f6c7a7fc640 1 -- 172.21.15.16:0/3168343850 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6c7c0b7dd0 msgr2=0x7f6c7c0ca5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:53.047 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.031+0000 7f6c7a7fc640 1 --2- 172.21.15.16:0/3168343850 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6c7c0b7dd0 0x7f6c7c0ca5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:53.047 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.031+0000 7f6c7a7fc640 1 -- 172.21.15.16:0/3168343850 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6c7c160050 msgr2=0x7f6c7c15dc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:53.047 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.031+0000 7f6c7a7fc640 1 --2- 172.21.15.16:0/3168343850 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6c7c160050 0x7f6c7c15dc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:53.047 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.031+0000 7f6c7a7fc640 1 -- 172.21.15.16:0/3168343850 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c74009b30 con 0x7f6c7c15f700 2023-05-22T15:27:53.047 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.031+0000 7f6c7a7fc640 1 --2- 172.21.15.16:0/3168343850 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c7c15f700 0x7f6c7c0b6000 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f6c7000e810 tx=0x7f6c7000ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:53.047 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.031+0000 7f6c637fe640 1 -- 172.21.15.16:0/3168343850 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c7000c8d0 con 0x7f6c7c15f700 2023-05-22T15:27:53.048 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.031+0000 7f6c637fe640 1 -- 172.21.15.16:0/3168343850 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c7000ca30 con 0x7f6c7c15f700 2023-05-22T15:27:53.048 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.031+0000 7f6c81696640 1 -- 172.21.15.16:0/3168343850 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c7c0cf600 con 0x7f6c7c15f700 2023-05-22T15:27:53.048 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.031+0000 7f6c637fe640 1 -- 172.21.15.16:0/3168343850 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c7000cde0 con 0x7f6c7c15f700 2023-05-22T15:27:53.048 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.031+0000 7f6c81696640 1 -- 172.21.15.16:0/3168343850 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c7c0cf470 con 0x7f6c7c15f700 2023-05-22T15:27:53.048 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.035+0000 7f6c81696640 1 -- 172.21.15.16:0/3168343850 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c7c0cf790 con 0x7f6c7c15f700 2023-05-22T15:27:53.048 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.035+0000 7f6c637fe640 1 -- 172.21.15.16:0/3168343850 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7f6c70013410 con 0x7f6c7c15f700 2023-05-22T15:27:53.049 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.035+0000 7f6c637fe640 1 --2- 172.21.15.16:0/3168343850 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f6c68065380 0x7f6c68067830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:53.049 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.035+0000 7f6c7affd640 1 --2- 172.21.15.16:0/3168343850 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f6c68065380 0x7f6c68067830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:53.049 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.035+0000 7f6c637fe640 1 -- 172.21.15.16:0/3168343850 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f6c70082a60 con 0x7f6c7c15f700 2023-05-22T15:27:53.049 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.035+0000 7f6c7affd640 1 --2- 172.21.15.16:0/3168343850 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f6c68065380 0x7f6c68067830 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f6c6c00b9d0 tx=0x7f6c6c008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:53.050 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.039+0000 7f6c637fe640 1 -- 172.21.15.16:0/3168343850 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f6c70050350 con 0x7f6c7c15f700 2023-05-22T15:27:53.207 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.199+0000 7f6c81696640 1 -- 172.21.15.16:0/3168343850 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f6c7c04dbe0 con 0x7f6c7c15f700 2023-05-22T15:27:53.207 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.199+0000 7f6c637fe640 1 -- 172.21.15.16:0/3168343850 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v29) v1 ==== 80+0+0 (secure 0 0 0) 0x7f6c70053970 con 0x7f6c7c15f700 2023-05-22T15:27:53.207 INFO:teuthology.orchestra.run.smithi016.stderr:listed 0 entries 2023-05-22T15:27:53.209 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.199+0000 7f6c81696640 1 -- 172.21.15.16:0/3168343850 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f6c68065380 msgr2=0x7f6c68067830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:53.209 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.199+0000 7f6c81696640 1 --2- 172.21.15.16:0/3168343850 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f6c68065380 0x7f6c68067830 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f6c6c00b9d0 tx=0x7f6c6c008040 comp rx=0 tx=0).stop 2023-05-22T15:27:53.210 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.199+0000 7f6c81696640 1 -- 172.21.15.16:0/3168343850 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c7c15f700 msgr2=0x7f6c7c0b6000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:53.210 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.199+0000 7f6c81696640 1 --2- 172.21.15.16:0/3168343850 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c7c15f700 0x7f6c7c0b6000 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f6c7000e810 tx=0x7f6c7000ecd0 comp rx=0 tx=0).stop 2023-05-22T15:27:53.210 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.199+0000 7f6c81696640 1 -- 172.21.15.16:0/3168343850 shutdown_connections 2023-05-22T15:27:53.211 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.199+0000 7f6c81696640 1 --2- 172.21.15.16:0/3168343850 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6c7c160050 0x7f6c7c15dc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:53.211 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.203+0000 7f6c81696640 1 --2- 172.21.15.16:0/3168343850 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c7c15f700 0x7f6c7c0b6000 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:53.211 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.203+0000 7f6c81696640 1 --2- 172.21.15.16:0/3168343850 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f6c68065380 0x7f6c68067830 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:53.211 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.203+0000 7f6c81696640 1 --2- 172.21.15.16:0/3168343850 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6c7c0b7dd0 0x7f6c7c0ca5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:53.211 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.203+0000 7f6c81696640 1 -- 172.21.15.16:0/3168343850 >> 172.21.15.16:0/3168343850 conn(0x7f6c7c0655c0 msgr2=0x7f6c7c0b2b70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:53.211 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.203+0000 7f6c81696640 1 -- 172.21.15.16:0/3168343850 shutdown_connections 2023-05-22T15:27:53.212 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.203+0000 7f6c81696640 1 -- 172.21.15.16:0/3168343850 wait complete. 2023-05-22T15:27:53.224 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi016.front.sepia.ceph.com... 2023-05-22T15:27:53.224 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:27:53.225 DEBUG:teuthology.orchestra.run.smithi016:> sudo fusermount -u /home/ubuntu/cephtest/mnt.2 2023-05-22T15:27:53.237 INFO:teuthology.orchestra.run:waiting for 300 2023-05-22T15:27:53.238 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi016.front.sepia.ceph.com 2023-05-22T15:27:53.238 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:27:53.239 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2023-05-22T15:27:53.285 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2023-05-22T15:27:53.286 DEBUG:teuthology.orchestra.run.smithi062:> dd if=/proc/self/mounts of=/dev/stdout 2023-05-22T15:27:53.293 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2023-05-22T15:27:53.294 DEBUG:teuthology.orchestra.run.smithi062:> dd if=/proc/self/mounts of=/dev/stdout 2023-05-22T15:27:53.342 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-05-22T15:27:53.465 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.459+0000 7f4a6e7d6640 1 Processor -- start 2023-05-22T15:27:53.465 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.459+0000 7f4a6e7d6640 1 -- start start 2023-05-22T15:27:53.466 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.459+0000 7f4a6e7d6640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f4a6806c3a0 0x7f4a6806cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:53.466 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.459+0000 7f4a6e7d6640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f4a680b38e0 0x7f4a6806b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:53.466 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.459+0000 7f4a6e7d6640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4a680afac0 0x7f4a680afef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:53.467 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.459+0000 7f4a6e7d6640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a6806c770 con 0x7f4a680b38e0 2023-05-22T15:27:53.467 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.459+0000 7f4a6e7d6640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a680b0430 con 0x7f4a6806c3a0 2023-05-22T15:27:53.467 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.459+0000 7f4a6e7d6640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a68070cc0 con 0x7f4a680afac0 2023-05-22T15:27:53.467 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.459+0000 7f4a677fe640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f4a680b38e0 0x7f4a6806b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:53.467 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.459+0000 7f4a677fe640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f4a680b38e0 0x7f4a6806b7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54194/0 (socket says 172.21.15.16:54194) 2023-05-22T15:27:53.467 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.459+0000 7f4a677fe640 1 -- 172.21.15.16:0/2962878090 learned_addr learned my addr 172.21.15.16:0/2962878090 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:53.468 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.459+0000 7f4a6cd73640 1 --2- 172.21.15.16:0/2962878090 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4a680afac0 0x7f4a680afef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:53.468 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.459+0000 7f4a67fff640 1 --2- 172.21.15.16:0/2962878090 >> v2:172.21.15.62:3300/0 conn(0x7f4a6806c3a0 0x7f4a6806cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:53.468 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.459+0000 7f4a677fe640 1 -- 172.21.15.16:0/2962878090 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4a680afac0 msgr2=0x7f4a680afef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:53.469 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.459+0000 7f4a677fe640 1 --2- 172.21.15.16:0/2962878090 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4a680afac0 0x7f4a680afef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:53.469 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.459+0000 7f4a677fe640 1 -- 172.21.15.16:0/2962878090 >> v2:172.21.15.62:3300/0 conn(0x7f4a6806c3a0 msgr2=0x7f4a6806cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:53.469 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.459+0000 7f4a677fe640 1 --2- 172.21.15.16:0/2962878090 >> v2:172.21.15.62:3300/0 conn(0x7f4a6806c3a0 0x7f4a6806cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:53.469 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.459+0000 7f4a677fe640 1 -- 172.21.15.16:0/2962878090 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a68070fa0 con 0x7f4a680b38e0 2023-05-22T15:27:53.477 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.459+0000 7f4a677fe640 1 --2- 172.21.15.16:0/2962878090 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4a680b38e0 0x7f4a6806b7b0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f4a68071ab0 tx=0x7f4a5c01c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0416df72e329481 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:53.478 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.463+0000 7f4a66ffd640 1 -- 172.21.15.16:0/2962878090 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a5c0ac630 con 0x7f4a680b38e0 2023-05-22T15:27:53.478 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.463+0000 7f4a66ffd640 1 -- 172.21.15.16:0/2962878090 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a5c0ac790 con 0x7f4a680b38e0 2023-05-22T15:27:53.478 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.463+0000 7f4a66ffd640 1 -- 172.21.15.16:0/2962878090 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a5c0acb20 con 0x7f4a680b38e0 2023-05-22T15:27:53.478 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.463+0000 7f4a6e7d6640 1 -- 172.21.15.16:0/2962878090 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4a680b38e0 msgr2=0x7f4a6806b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:53.478 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.463+0000 7f4a6e7d6640 1 --2- 172.21.15.16:0/2962878090 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4a680b38e0 0x7f4a6806b7b0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f4a68071ab0 tx=0x7f4a5c01c500 comp rx=0 tx=0).stop 2023-05-22T15:27:53.478 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.463+0000 7f4a6e7d6640 1 -- 172.21.15.16:0/2962878090 shutdown_connections 2023-05-22T15:27:53.479 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.463+0000 7f4a6e7d6640 1 --2- 172.21.15.16:0/2962878090 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4a680b38e0 0x7f4a6806b7b0 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:53.479 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.463+0000 7f4a6e7d6640 1 --2- 172.21.15.16:0/2962878090 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4a680afac0 0x7f4a680afef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:53.479 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.463+0000 7f4a6e7d6640 1 --2- 172.21.15.16:0/2962878090 >> v2:172.21.15.62:3300/0 conn(0x7f4a6806c3a0 0x7f4a6806cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:53.479 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.463+0000 7f4a6e7d6640 1 -- 172.21.15.16:0/2962878090 >> 172.21.15.16:0/2962878090 conn(0x7f4a680ce040 msgr2=0x7f4a680ce440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:53.479 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.463+0000 7f4a6e7d6640 1 -- 172.21.15.16:0/2962878090 shutdown_connections 2023-05-22T15:27:53.479 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.463+0000 7f4a6e7d6640 1 -- 172.21.15.16:0/2962878090 wait complete. 2023-05-22T15:27:53.479 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.463+0000 7f4a6e7d6640 1 Processor -- start 2023-05-22T15:27:53.479 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.463+0000 7f4a6e7d6640 1 -- start start 2023-05-22T15:27:53.480 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.463+0000 7f4a6e7d6640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4a6806c3a0 0x7f4a68157350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:53.480 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.463+0000 7f4a6e7d6640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4a680afac0 0x7f4a68157fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:53.480 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.463+0000 7f4a6e7d6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4a680b38e0 0x7f4a6814b2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:53.480 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.463+0000 7f4a6e7d6640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a6815d180 con 0x7f4a680b38e0 2023-05-22T15:27:53.480 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.463+0000 7f4a6e7d6640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a6814ca00 con 0x7f4a680afac0 2023-05-22T15:27:53.480 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.463+0000 7f4a6e7d6640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a6814cca0 con 0x7f4a6806c3a0 2023-05-22T15:27:53.481 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.463+0000 7f4a6cd73640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4a680b38e0 0x7f4a6814b2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:53.481 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.463+0000 7f4a6cd73640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4a680b38e0 0x7f4a6814b2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54206/0 (socket says 172.21.15.16:54206) 2023-05-22T15:27:53.481 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.463+0000 7f4a6cd73640 1 -- 172.21.15.16:0/1662607790 learned_addr learned my addr 172.21.15.16:0/1662607790 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:53.481 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.463+0000 7f4a677fe640 1 --2- 172.21.15.16:0/1662607790 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4a680afac0 0x7f4a68157fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:53.481 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.463+0000 7f4a67fff640 1 --2- 172.21.15.16:0/1662607790 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4a6806c3a0 0x7f4a68157350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:53.482 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.463+0000 7f4a6cd73640 1 -- 172.21.15.16:0/1662607790 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4a6806c3a0 msgr2=0x7f4a68157350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:53.482 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.463+0000 7f4a6cd73640 1 --2- 172.21.15.16:0/1662607790 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4a6806c3a0 0x7f4a68157350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:53.482 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.463+0000 7f4a6cd73640 1 -- 172.21.15.16:0/1662607790 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4a680afac0 msgr2=0x7f4a68157fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:53.482 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.463+0000 7f4a6cd73640 1 --2- 172.21.15.16:0/1662607790 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4a680afac0 0x7f4a68157fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:53.482 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.463+0000 7f4a6cd73640 1 -- 172.21.15.16:0/1662607790 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a5c0094d0 con 0x7f4a680b38e0 2023-05-22T15:27:53.482 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.463+0000 7f4a67fff640 1 --2- 172.21.15.16:0/1662607790 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4a6806c3a0 0x7f4a68157350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T15:27:53.483 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.463+0000 7f4a6cd73640 1 --2- 172.21.15.16:0/1662607790 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4a680b38e0 0x7f4a6814b2b0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f4a60007e40 tx=0x7f4a6000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:53.483 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.463+0000 7f4a64ff9640 1 -- 172.21.15.16:0/1662607790 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a60014620 con 0x7f4a680b38e0 2023-05-22T15:27:53.483 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.463+0000 7f4a64ff9640 1 -- 172.21.15.16:0/1662607790 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a6000ee90 con 0x7f4a680b38e0 2023-05-22T15:27:53.483 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.463+0000 7f4a6e7d6640 1 -- 172.21.15.16:0/1662607790 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a680c9c00 con 0x7f4a680b38e0 2023-05-22T15:27:53.483 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.463+0000 7f4a64ff9640 1 -- 172.21.15.16:0/1662607790 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a600149e0 con 0x7f4a680b38e0 2023-05-22T15:27:53.483 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.463+0000 7f4a6e7d6640 1 -- 172.21.15.16:0/1662607790 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a68071840 con 0x7f4a680b38e0 2023-05-22T15:27:53.483 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.467+0000 7f4a6e7d6640 1 -- 172.21.15.16:0/1662607790 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a6815e2c0 con 0x7f4a680b38e0 2023-05-22T15:27:53.484 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.467+0000 7f4a64ff9640 1 -- 172.21.15.16:0/1662607790 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7f4a60014b40 con 0x7f4a680b38e0 2023-05-22T15:27:53.484 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.467+0000 7f4a64ff9640 1 --2- 172.21.15.16:0/1662607790 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f4a54065380 0x7f4a54067830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:53.484 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.467+0000 7f4a64ff9640 1 -- 172.21.15.16:0/1662607790 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f4a600834b0 con 0x7f4a680b38e0 2023-05-22T15:27:53.484 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.467+0000 7f4a67fff640 1 --2- 172.21.15.16:0/1662607790 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f4a54065380 0x7f4a54067830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:53.484 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.467+0000 7f4a67fff640 1 --2- 172.21.15.16:0/1662607790 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f4a54065380 0x7f4a54067830 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f4a58004660 tx=0x7f4a58008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:53.485 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.471+0000 7f4a64ff9640 1 -- 172.21.15.16:0/1662607790 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f4a60050e90 con 0x7f4a680b38e0 2023-05-22T15:27:53.636 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.627+0000 7f4a6e7d6640 1 -- 172.21.15.16:0/1662607790 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4a680738f0 con 0x7f4a680b38e0 2023-05-22T15:27:53.637 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.631+0000 7f4a64ff9640 1 -- 172.21.15.16:0/1662607790 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v29) v1 ==== 74+0+19700 (secure 0 0 0) 0x7f4a600544b0 con 0x7f4a680b38e0 2023-05-22T15:27:53.637 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:27:53.637 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":29,"fsid":"b2719834-7847-4d98-90a6-f65d57f85c49","created":"2023-05-22T15:26:25.259314+0000","modified":"2023-05-22T15:27:31.182317+0000","last_up_change":"2023-05-22T15:26:36.448524+0000","last_in_change":"2023-05-22T15:26:27.675668+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-05-22T15:26:38.019781+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-05-22T15:26:43.570514+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-05-22T15:26:47.879159+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-05-22T15:26:48.522130+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data_ec","create_time":"2023-05-22T15:26:51.616142+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"9e7ff854-b25f-4cbb-a5c2-1799d9c53d2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":37021},{"type":"v1","addr":"172.21.15.16:6825","nonce":37021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":37021},{"type":"v1","addr":"172.21.15.16:6827","nonce":37021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":37021},{"type":"v1","addr":"172.21.15.16:6831","nonce":37021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":37021},{"type":"v1","addr":"172.21.15.16:6829","nonce":37021}]},"public_addr":"172.21.15.16:6825/37021","cluster_addr":"172.21.15.16:6827/37021","heartbeat_back_addr":"172.21.15.16:6831/37021","heartbeat_front_addr":"172.21.15.16:6829/37021","state":["exists","up"]},{"osd":1,"uuid":"4702fbc9-a927-4565-aca5-41dc69cc8264","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":37008},{"type":"v1","addr":"172.21.15.16:6801","nonce":37008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":37008},{"type":"v1","addr":"172.21.15.16:6803","nonce":37008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":37008},{"type":"v1","addr":"172.21.15.16:6807","nonce":37008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":37008},{"type":"v1","addr":"172.21.15.16:6805","nonce":37008}]},"public_addr":"172.21.15.16:6801/37008","cluster_addr":"172.21.15.16:6803/37008","heartbeat_back_addr":"172.21.15.16:6807/37008","heartbeat_front_addr":"172.21.15.16:6805/37008","state":["exists","up"]},{"osd":2,"uuid":"1e7bba46-4eba-47ef-bb81-5e2d371638ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":37009},{"type":"v1","addr":"172.21.15.16:6809","nonce":37009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":37009},{"type":"v1","addr":"172.21.15.16:6811","nonce":37009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":37009},{"type":"v1","addr":"172.21.15.16:6815","nonce":37009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":37009},{"type":"v1","addr":"172.21.15.16:6813","nonce":37009}]},"public_addr":"172.21.15.16:6809/37009","cluster_addr":"172.21.15.16:6811/37009","heartbeat_back_addr":"172.21.15.16:6815/37009","heartbeat_front_addr":"172.21.15.16:6813/37009","state":["exists","up"]},{"osd":3,"uuid":"383cacc0-5a88-4904-990e-498ca59ef48a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":37020},{"type":"v1","addr":"172.21.15.16:6817","nonce":37020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":37020},{"type":"v1","addr":"172.21.15.16:6819","nonce":37020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":37020},{"type":"v1","addr":"172.21.15.16:6823","nonce":37020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":37020},{"type":"v1","addr":"172.21.15.16:6821","nonce":37020}]},"public_addr":"172.21.15.16:6817/37020","cluster_addr":"172.21.15.16:6819/37020","heartbeat_back_addr":"172.21.15.16:6823/37020","heartbeat_front_addr":"172.21.15.16:6821/37020","state":["exists","up"]},{"osd":4,"uuid":"970500ef-95b7-42c1-b080-eed4fae7013d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":37353},{"type":"v1","addr":"172.21.15.62:6817","nonce":37353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6819","nonce":37353},{"type":"v1","addr":"172.21.15.62:6822","nonce":37353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37353},{"type":"v1","addr":"172.21.15.62:6831","nonce":37353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37353},{"type":"v1","addr":"172.21.15.62:6826","nonce":37353}]},"public_addr":"172.21.15.62:6817/37353","cluster_addr":"172.21.15.62:6822/37353","heartbeat_back_addr":"172.21.15.62:6831/37353","heartbeat_front_addr":"172.21.15.62:6826/37353","state":["exists","up"]},{"osd":5,"uuid":"022d67db-d4d8-445b-b1b9-8b446a00f5c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":37352},{"type":"v1","addr":"172.21.15.62:6801","nonce":37352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":37352},{"type":"v1","addr":"172.21.15.62:6803","nonce":37352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":37352},{"type":"v1","addr":"172.21.15.62:6807","nonce":37352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":37352},{"type":"v1","addr":"172.21.15.62:6805","nonce":37352}]},"public_addr":"172.21.15.62:6801/37352","cluster_addr":"172.21.15.62:6803/37352","heartbeat_back_addr":"172.21.15.62:6807/37352","heartbeat_front_addr":"172.21.15.62:6805/37352","state":["exists","up"]},{"osd":6,"uuid":"42bd372b-6688-4a1b-8407-e16e8c266e33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":37354},{"type":"v1","addr":"172.21.15.62:6820","nonce":37354}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":37354},{"type":"v1","addr":"172.21.15.62:6823","nonce":37354}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6829","nonce":37354},{"type":"v1","addr":"172.21.15.62:6830","nonce":37354}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6825","nonce":37354},{"type":"v1","addr":"172.21.15.62:6827","nonce":37354}]},"public_addr":"172.21.15.62:6820/37354","cluster_addr":"172.21.15.62:6823/37354","heartbeat_back_addr":"172.21.15.62:6830/37354","heartbeat_front_addr":"172.21.15.62:6827/37354","state":["exists","up"]},{"osd":7,"uuid":"ac70b2a3-f41b-4bf2-b227-7f2cbb55426d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37355},{"type":"v1","addr":"172.21.15.62:6809","nonce":37355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37355},{"type":"v1","addr":"172.21.15.62:6811","nonce":37355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":37355},{"type":"v1","addr":"172.21.15.62:6815","nonce":37355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":37355},{"type":"v1","addr":"172.21.15.62:6813","nonce":37355}]},"public_addr":"172.21.15.62:6809/37355","cluster_addr":"172.21.15.62:6811/37355","heartbeat_back_addr":"172.21.15.62:6815/37355","heartbeat_front_addr":"172.21.15.62:6813/37355","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.368896+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.107273+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.276888+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.287502+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.124791+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.251937+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.255679+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.232797+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"4.23","mappings":[{"from":3,"to":2}]},{"pgid":"4.31","mappings":[{"from":1,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-05-22T15:27:53.639 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.631+0000 7f4a6e7d6640 1 -- 172.21.15.16:0/1662607790 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f4a54065380 msgr2=0x7f4a54067830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:53.640 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.631+0000 7f4a6e7d6640 1 --2- 172.21.15.16:0/1662607790 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f4a54065380 0x7f4a54067830 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f4a58004660 tx=0x7f4a58008040 comp rx=0 tx=0).stop 2023-05-22T15:27:53.640 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.631+0000 7f4a6e7d6640 1 -- 172.21.15.16:0/1662607790 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4a680b38e0 msgr2=0x7f4a6814b2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:53.640 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.631+0000 7f4a6e7d6640 1 --2- 172.21.15.16:0/1662607790 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4a680b38e0 0x7f4a6814b2b0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f4a60007e40 tx=0x7f4a6000e4b0 comp rx=0 tx=0).stop 2023-05-22T15:27:53.640 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.631+0000 7f4a6e7d6640 1 -- 172.21.15.16:0/1662607790 shutdown_connections 2023-05-22T15:27:53.640 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.631+0000 7f4a6e7d6640 1 --2- 172.21.15.16:0/1662607790 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4a680b38e0 0x7f4a6814b2b0 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:53.641 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.631+0000 7f4a6e7d6640 1 --2- 172.21.15.16:0/1662607790 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4a680afac0 0x7f4a68157fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:53.641 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.631+0000 7f4a6e7d6640 1 --2- 172.21.15.16:0/1662607790 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f4a54065380 0x7f4a54067830 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:53.641 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.631+0000 7f4a6e7d6640 1 --2- 172.21.15.16:0/1662607790 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4a6806c3a0 0x7f4a68157350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:53.641 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.631+0000 7f4a6e7d6640 1 -- 172.21.15.16:0/1662607790 >> 172.21.15.16:0/1662607790 conn(0x7f4a680ce040 msgr2=0x7f4a680adaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:53.641 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.631+0000 7f4a6e7d6640 1 -- 172.21.15.16:0/1662607790 shutdown_connections 2023-05-22T15:27:53.641 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.631+0000 7f4a6e7d6640 1 -- 172.21.15.16:0/1662607790 wait complete. 2023-05-22T15:27:53.657 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-05-22T15:26:38.019781+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-05-22T15:26:43.570514+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2023-05-22T15:26:47.879159+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2023-05-22T15:26:48.522130+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-05-22T15:26:51.616142+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '27', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-05-22T15:27:53.658 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-05-22T15:27:53.785 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.779+0000 7f9c1ea16640 1 Processor -- start 2023-05-22T15:27:53.786 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.779+0000 7f9c1ea16640 1 -- start start 2023-05-22T15:27:53.787 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.779+0000 7f9c1ea16640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f9c180cdfa0 0x7f9c1815d750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:53.787 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.779+0000 7f9c1ea16640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c180c9e70 con 0x7f9c180ce370 2023-05-22T15:27:53.787 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.779+0000 7f9c1ea16640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c180ca150 con 0x7f9c180b3a70 2023-05-22T15:27:53.787 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.779+0000 7f9c1ea16640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c180cba20 con 0x7f9c180cdfa0 2023-05-22T15:27:53.787 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.779+0000 7f9c17fff640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f9c180cdfa0 0x7f9c1815d750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:53.787 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.779+0000 7f9c17fff640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f9c180cdfa0 0x7f9c1815d750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54208/0 (socket says 172.21.15.16:54208) 2023-05-22T15:27:53.788 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.779+0000 7f9c17fff640 1 -- 172.21.15.16:0/2795362973 learned_addr learned my addr 172.21.15.16:0/2795362973 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:53.788 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.779+0000 7f9c17fff640 1 -- 172.21.15.16:0/2795362973 >> v1:172.21.15.62:6789/0 conn(0x7f9c180b3a70 legacy=0x7f9c180cb260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:53.788 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.779+0000 7f9c17fff640 1 -- 172.21.15.16:0/2795362973 >> v1:172.21.15.16:6789/0 conn(0x7f9c180ce370 legacy=0x7f9c1806b7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:53.788 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.779+0000 7f9c17fff640 1 -- 172.21.15.16:0/2795362973 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c180ca430 con 0x7f9c180cdfa0 2023-05-22T15:27:53.789 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.783+0000 7f9c17fff640 1 --2- 172.21.15.16:0/2795362973 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9c180cdfa0 0x7f9c1815d750 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f9c08009870 tx=0x7f9c0801c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5618d8239a02b00 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:53.789 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.783+0000 7f9c16ffd640 1 -- 172.21.15.16:0/2795362973 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c080ab600 con 0x7f9c180cdfa0 2023-05-22T15:27:53.789 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.783+0000 7f9c16ffd640 1 -- 172.21.15.16:0/2795362973 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c080a7050 con 0x7f9c180cdfa0 2023-05-22T15:27:53.790 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.783+0000 7f9c16ffd640 1 -- 172.21.15.16:0/2795362973 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c080ab980 con 0x7f9c180cdfa0 2023-05-22T15:27:53.790 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.783+0000 7f9c1ea16640 1 -- 172.21.15.16:0/2795362973 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9c180cdfa0 msgr2=0x7f9c1815d750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:53.790 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.783+0000 7f9c1ea16640 1 --2- 172.21.15.16:0/2795362973 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9c180cdfa0 0x7f9c1815d750 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f9c08009870 tx=0x7f9c0801c500 comp rx=0 tx=0).stop 2023-05-22T15:27:53.791 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.783+0000 7f9c1ea16640 1 -- 172.21.15.16:0/2795362973 shutdown_connections 2023-05-22T15:27:53.791 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.783+0000 7f9c1ea16640 1 --2- 172.21.15.16:0/2795362973 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9c180cdfa0 0x7f9c1815d750 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:53.791 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.783+0000 7f9c1ea16640 1 -- 172.21.15.16:0/2795362973 >> 172.21.15.16:0/2795362973 conn(0x7f9c18069720 msgr2=0x7f9c18069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:53.791 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.783+0000 7f9c1ea16640 1 -- 172.21.15.16:0/2795362973 shutdown_connections 2023-05-22T15:27:53.791 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.783+0000 7f9c1ea16640 1 -- 172.21.15.16:0/2795362973 wait complete. 2023-05-22T15:27:53.793 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.783+0000 7f9c1ea16640 1 Processor -- start 2023-05-22T15:27:53.793 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.783+0000 7f9c1ea16640 1 -- start start 2023-05-22T15:27:53.793 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.783+0000 7f9c1ea16640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9c180b3a70 0x7f9c180c9420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:53.793 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.783+0000 7f9c1ea16640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9c180ce370 0x7f9c18151990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:53.793 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.783+0000 7f9c1ea16640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9c18152250 0x7f9c180b29b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:53.794 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.783+0000 7f9c1ea16640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c180c73b0 con 0x7f9c18152250 2023-05-22T15:27:53.794 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.783+0000 7f9c1ea16640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c180c67a0 con 0x7f9c180b3a70 2023-05-22T15:27:53.794 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.783+0000 7f9c1ea16640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c180c6a80 con 0x7f9c180ce370 2023-05-22T15:27:53.794 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.783+0000 7f9c1cfb3640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9c18152250 0x7f9c180b29b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:53.794 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.783+0000 7f9c1cfb3640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9c18152250 0x7f9c180b29b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54214/0 (socket says 172.21.15.16:54214) 2023-05-22T15:27:53.794 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.783+0000 7f9c1cfb3640 1 -- 172.21.15.16:0/3385494473 learned_addr learned my addr 172.21.15.16:0/3385494473 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:53.794 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.783+0000 7f9c177fe640 1 --2- 172.21.15.16:0/3385494473 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9c180ce370 0x7f9c18151990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:53.794 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.783+0000 7f9c17fff640 1 --2- 172.21.15.16:0/3385494473 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9c180b3a70 0x7f9c180c9420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:53.795 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.783+0000 7f9c1cfb3640 1 -- 172.21.15.16:0/3385494473 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9c180ce370 msgr2=0x7f9c18151990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:53.795 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.783+0000 7f9c1cfb3640 1 --2- 172.21.15.16:0/3385494473 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9c180ce370 0x7f9c18151990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:53.795 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.783+0000 7f9c1cfb3640 1 -- 172.21.15.16:0/3385494473 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9c180b3a70 msgr2=0x7f9c180c9420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:53.795 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.783+0000 7f9c1cfb3640 1 --2- 172.21.15.16:0/3385494473 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9c180b3a70 0x7f9c180c9420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:53.795 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.783+0000 7f9c1cfb3640 1 -- 172.21.15.16:0/3385494473 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c080094d0 con 0x7f9c18152250 2023-05-22T15:27:53.795 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.783+0000 7f9c17fff640 1 --2- 172.21.15.16:0/3385494473 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9c180b3a70 0x7f9c180c9420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:27:53.795 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.783+0000 7f9c1cfb3640 1 --2- 172.21.15.16:0/3385494473 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9c18152250 0x7f9c180b29b0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f9c1000c730 tx=0x7f9c1000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:53.796 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.787+0000 7f9c14ff9640 1 -- 172.21.15.16:0/3385494473 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c10003b80 con 0x7f9c18152250 2023-05-22T15:27:53.796 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.787+0000 7f9c14ff9640 1 -- 172.21.15.16:0/3385494473 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c10003ce0 con 0x7f9c18152250 2023-05-22T15:27:53.796 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.787+0000 7f9c1ea16640 1 -- 172.21.15.16:0/3385494473 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c180ce900 con 0x7f9c18152250 2023-05-22T15:27:53.796 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.787+0000 7f9c14ff9640 1 -- 172.21.15.16:0/3385494473 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c1000aa30 con 0x7f9c18152250 2023-05-22T15:27:53.796 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.787+0000 7f9c1ea16640 1 -- 172.21.15.16:0/3385494473 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c180ce770 con 0x7f9c18152250 2023-05-22T15:27:53.797 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.787+0000 7f9c1ea16640 1 -- 172.21.15.16:0/3385494473 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c1806c2e0 con 0x7f9c18152250 2023-05-22T15:27:53.797 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.787+0000 7f9c14ff9640 1 -- 172.21.15.16:0/3385494473 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7f9c1000ab90 con 0x7f9c18152250 2023-05-22T15:27:53.797 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.787+0000 7f9c14ff9640 1 --2- 172.21.15.16:0/3385494473 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f9c040653b0 0x7f9c04067860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:53.798 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.787+0000 7f9c14ff9640 1 -- 172.21.15.16:0/3385494473 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f9c10082960 con 0x7f9c18152250 2023-05-22T15:27:53.798 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.787+0000 7f9c17fff640 1 --2- 172.21.15.16:0/3385494473 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f9c040653b0 0x7f9c04067860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:53.798 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.787+0000 7f9c17fff640 1 --2- 172.21.15.16:0/3385494473 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f9c040653b0 0x7f9c04067860 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f9c080ad000 tx=0x7f9c08008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:53.799 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.791+0000 7f9c14ff9640 1 -- 172.21.15.16:0/3385494473 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f9c100502b0 con 0x7f9c18152250 2023-05-22T15:27:53.951 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.943+0000 7f9c1ea16640 1 -- 172.21.15.16:0/3385494473 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f9c1804dbe0 con 0x7f9c18152250 2023-05-22T15:27:53.952 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.943+0000 7f9c14ff9640 1 -- 172.21.15.16:0/3385494473 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v29) v1 ==== 93+0+10 (secure 0 0 0) 0x7f9c100538d0 con 0x7f9c18152250 2023-05-22T15:27:53.953 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2023-05-22T15:27:53.954 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.947+0000 7f9c1ea16640 1 -- 172.21.15.16:0/3385494473 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f9c040653b0 msgr2=0x7f9c04067860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:53.954 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.947+0000 7f9c1ea16640 1 --2- 172.21.15.16:0/3385494473 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f9c040653b0 0x7f9c04067860 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f9c080ad000 tx=0x7f9c08008040 comp rx=0 tx=0).stop 2023-05-22T15:27:53.955 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.947+0000 7f9c1ea16640 1 -- 172.21.15.16:0/3385494473 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9c18152250 msgr2=0x7f9c180b29b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:53.955 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.947+0000 7f9c1ea16640 1 --2- 172.21.15.16:0/3385494473 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9c18152250 0x7f9c180b29b0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f9c1000c730 tx=0x7f9c1000cbf0 comp rx=0 tx=0).stop 2023-05-22T15:27:53.955 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.947+0000 7f9c1ea16640 1 -- 172.21.15.16:0/3385494473 shutdown_connections 2023-05-22T15:27:53.955 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.947+0000 7f9c1ea16640 1 --2- 172.21.15.16:0/3385494473 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9c18152250 0x7f9c180b29b0 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:53.956 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.947+0000 7f9c1ea16640 1 --2- 172.21.15.16:0/3385494473 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f9c040653b0 0x7f9c04067860 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:53.956 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.947+0000 7f9c1ea16640 1 --2- 172.21.15.16:0/3385494473 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9c180ce370 0x7f9c18151990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:53.956 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.947+0000 7f9c1ea16640 1 --2- 172.21.15.16:0/3385494473 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9c180b3a70 0x7f9c180c9420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:53.957 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.947+0000 7f9c1ea16640 1 -- 172.21.15.16:0/3385494473 >> 172.21.15.16:0/3385494473 conn(0x7f9c18069720 msgr2=0x7f9c180b1cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:53.957 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.947+0000 7f9c1ea16640 1 -- 172.21.15.16:0/3385494473 shutdown_connections 2023-05-22T15:27:53.957 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:53.947+0000 7f9c1ea16640 1 -- 172.21.15.16:0/3385494473 wait complete. 2023-05-22T15:27:53.973 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-05-22T15:27:54.096 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.087+0000 7f02ce5df640 1 Processor -- start 2023-05-22T15:27:54.097 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.087+0000 7f02ce5df640 1 -- start start 2023-05-22T15:27:54.097 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.091+0000 7f02ce5df640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f02c806b6c0 0x7f02c80cb4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:54.097 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.091+0000 7f02ce5df640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02c8159f70 con 0x7f02c81596e0 2023-05-22T15:27:54.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.091+0000 7f02ce5df640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02c80cb030 con 0x7f02c806ba90 2023-05-22T15:27:54.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.091+0000 7f02ce5df640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02c8071760 con 0x7f02c806b6c0 2023-05-22T15:27:54.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.091+0000 7f02c7fff640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f02c806b6c0 0x7f02c80cb4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:54.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.091+0000 7f02c7fff640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f02c806b6c0 0x7f02c80cb4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54230/0 (socket says 172.21.15.16:54230) 2023-05-22T15:27:54.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.091+0000 7f02c7fff640 1 -- 172.21.15.16:0/3500497086 learned_addr learned my addr 172.21.15.16:0/3500497086 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:54.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.091+0000 7f02c7fff640 1 -- 172.21.15.16:0/3500497086 >> v1:172.21.15.62:6789/0 conn(0x7f02c806ba90 legacy=0x7f02c8158fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:54.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.091+0000 7f02c7fff640 1 -- 172.21.15.16:0/3500497086 >> v1:172.21.15.16:6789/0 conn(0x7f02c81596e0 legacy=0x7f02c806fba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:54.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.091+0000 7f02c7fff640 1 -- 172.21.15.16:0/3500497086 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02c8159c90 con 0x7f02c806b6c0 2023-05-22T15:27:54.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.091+0000 7f02c7fff640 1 --2- 172.21.15.16:0/3500497086 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f02c806b6c0 0x7f02c80cb4c0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f02b8009870 tx=0x7f02b801c4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a25d9b77d59fcac7 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:54.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.091+0000 7f02c6ffd640 1 -- 172.21.15.16:0/3500497086 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02b80ab600 con 0x7f02c806b6c0 2023-05-22T15:27:54.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.091+0000 7f02c6ffd640 1 -- 172.21.15.16:0/3500497086 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02b80a7050 con 0x7f02c806b6c0 2023-05-22T15:27:54.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.091+0000 7f02c6ffd640 1 -- 172.21.15.16:0/3500497086 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02b80ab980 con 0x7f02c806b6c0 2023-05-22T15:27:54.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.095+0000 7f02ce5df640 1 -- 172.21.15.16:0/3500497086 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f02c806b6c0 msgr2=0x7f02c80cb4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:54.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.095+0000 7f02ce5df640 1 --2- 172.21.15.16:0/3500497086 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f02c806b6c0 0x7f02c80cb4c0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f02b8009870 tx=0x7f02b801c4b0 comp rx=0 tx=0).stop 2023-05-22T15:27:54.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.095+0000 7f02ce5df640 1 -- 172.21.15.16:0/3500497086 shutdown_connections 2023-05-22T15:27:54.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.095+0000 7f02ce5df640 1 --2- 172.21.15.16:0/3500497086 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f02c806b6c0 0x7f02c80cb4c0 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:54.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.095+0000 7f02ce5df640 1 -- 172.21.15.16:0/3500497086 >> 172.21.15.16:0/3500497086 conn(0x7f02c8069720 msgr2=0x7f02c8069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:54.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.095+0000 7f02ce5df640 1 -- 172.21.15.16:0/3500497086 shutdown_connections 2023-05-22T15:27:54.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.095+0000 7f02ce5df640 1 -- 172.21.15.16:0/3500497086 wait complete. 2023-05-22T15:27:54.105 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.095+0000 7f02ce5df640 1 Processor -- start 2023-05-22T15:27:54.105 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.095+0000 7f02ce5df640 1 -- start start 2023-05-22T15:27:54.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.095+0000 7f02ce5df640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f02c806b6c0 0x7f02c813fb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:54.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.095+0000 7f02ce5df640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f02c806ba90 0x7f02c80b25b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:54.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.095+0000 7f02ce5df640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f02c81596e0 0x7f02c80c37c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:54.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.095+0000 7f02ce5df640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02c80b0530 con 0x7f02c81596e0 2023-05-22T15:27:54.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.095+0000 7f02ce5df640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02c80b09c0 con 0x7f02c806ba90 2023-05-22T15:27:54.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.095+0000 7f02ce5df640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02c80af940 con 0x7f02c806b6c0 2023-05-22T15:27:54.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.095+0000 7f02ccb7c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f02c81596e0 0x7f02c80c37c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:54.107 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.095+0000 7f02ccb7c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f02c81596e0 0x7f02c80c37c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54246/0 (socket says 172.21.15.16:54246) 2023-05-22T15:27:54.107 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.095+0000 7f02ccb7c640 1 -- 172.21.15.16:0/473665495 learned_addr learned my addr 172.21.15.16:0/473665495 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:54.107 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.095+0000 7f02c77fe640 1 --2- 172.21.15.16:0/473665495 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f02c806ba90 0x7f02c80b25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:54.107 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.095+0000 7f02c7fff640 1 --2- 172.21.15.16:0/473665495 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f02c806b6c0 0x7f02c813fb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:54.107 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.095+0000 7f02ccb7c640 1 -- 172.21.15.16:0/473665495 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f02c806b6c0 msgr2=0x7f02c813fb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:54.107 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.095+0000 7f02ccb7c640 1 --2- 172.21.15.16:0/473665495 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f02c806b6c0 0x7f02c813fb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:54.107 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.095+0000 7f02ccb7c640 1 -- 172.21.15.16:0/473665495 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f02c806ba90 msgr2=0x7f02c80b25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:54.108 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.095+0000 7f02ccb7c640 1 --2- 172.21.15.16:0/473665495 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f02c806ba90 0x7f02c80b25b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:54.108 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.095+0000 7f02ccb7c640 1 -- 172.21.15.16:0/473665495 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02b80094d0 con 0x7f02c81596e0 2023-05-22T15:27:54.108 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.095+0000 7f02c77fe640 1 --2- 172.21.15.16:0/473665495 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f02c806ba90 0x7f02c80b25b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T15:27:54.108 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.095+0000 7f02ccb7c640 1 --2- 172.21.15.16:0/473665495 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f02c81596e0 0x7f02c80c37c0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f02c000c430 tx=0x7f02c000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:54.108 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.095+0000 7f02c4ff9640 1 -- 172.21.15.16:0/473665495 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02c0005220 con 0x7f02c81596e0 2023-05-22T15:27:54.108 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.095+0000 7f02c4ff9640 1 -- 172.21.15.16:0/473665495 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02c0005380 con 0x7f02c81596e0 2023-05-22T15:27:54.109 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.095+0000 7f02c4ff9640 1 -- 172.21.15.16:0/473665495 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02c0007270 con 0x7f02c81596e0 2023-05-22T15:27:54.109 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.095+0000 7f02ce5df640 1 -- 172.21.15.16:0/473665495 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02c806bff0 con 0x7f02c81596e0 2023-05-22T15:27:54.109 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.095+0000 7f02ce5df640 1 -- 172.21.15.16:0/473665495 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02c806be60 con 0x7f02c81596e0 2023-05-22T15:27:54.109 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.099+0000 7f02ce5df640 1 -- 172.21.15.16:0/473665495 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02c8156840 con 0x7f02c81596e0 2023-05-22T15:27:54.110 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.099+0000 7f02c4ff9640 1 -- 172.21.15.16:0/473665495 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7f02c00073d0 con 0x7f02c81596e0 2023-05-22T15:27:54.110 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.099+0000 7f02c4ff9640 1 --2- 172.21.15.16:0/473665495 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f02a8065380 0x7f02a8067830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:54.110 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.099+0000 7f02c4ff9640 1 -- 172.21.15.16:0/473665495 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f02c00821a0 con 0x7f02c81596e0 2023-05-22T15:27:54.111 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.099+0000 7f02c7fff640 1 --2- 172.21.15.16:0/473665495 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f02a8065380 0x7f02a8067830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:54.111 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.099+0000 7f02c7fff640 1 --2- 172.21.15.16:0/473665495 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f02a8065380 0x7f02a8067830 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f02b80ad000 tx=0x7f02b8008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:54.111 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.103+0000 7f02c4ff9640 1 -- 172.21.15.16:0/473665495 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f02c004faf0 con 0x7f02c81596e0 2023-05-22T15:27:54.265 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.259+0000 7f02ce5df640 1 -- 172.21.15.16:0/473665495 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f02c804dbe0 con 0x7f02c81596e0 2023-05-22T15:27:54.266 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.259+0000 7f02c4ff9640 1 -- 172.21.15.16:0/473665495 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v29) v1 ==== 92+0+10 (secure 0 0 0) 0x7f02c0053110 con 0x7f02c81596e0 2023-05-22T15:27:54.266 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 8 2023-05-22T15:27:54.268 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.259+0000 7f02ce5df640 1 -- 172.21.15.16:0/473665495 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f02a8065380 msgr2=0x7f02a8067830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:54.268 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.259+0000 7f02ce5df640 1 --2- 172.21.15.16:0/473665495 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f02a8065380 0x7f02a8067830 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f02b80ad000 tx=0x7f02b8008040 comp rx=0 tx=0).stop 2023-05-22T15:27:54.268 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.259+0000 7f02ce5df640 1 -- 172.21.15.16:0/473665495 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f02c81596e0 msgr2=0x7f02c80c37c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:54.268 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.259+0000 7f02ce5df640 1 --2- 172.21.15.16:0/473665495 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f02c81596e0 0x7f02c80c37c0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f02c000c430 tx=0x7f02c000c8f0 comp rx=0 tx=0).stop 2023-05-22T15:27:54.268 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.259+0000 7f02ce5df640 1 -- 172.21.15.16:0/473665495 shutdown_connections 2023-05-22T15:27:54.268 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.259+0000 7f02ce5df640 1 --2- 172.21.15.16:0/473665495 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f02c81596e0 0x7f02c80c37c0 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:54.269 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.259+0000 7f02ce5df640 1 --2- 172.21.15.16:0/473665495 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f02c806ba90 0x7f02c80b25b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:54.269 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.259+0000 7f02ce5df640 1 --2- 172.21.15.16:0/473665495 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f02a8065380 0x7f02a8067830 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:54.269 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.259+0000 7f02ce5df640 1 --2- 172.21.15.16:0/473665495 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f02c806b6c0 0x7f02c813fb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:54.269 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.259+0000 7f02ce5df640 1 -- 172.21.15.16:0/473665495 >> 172.21.15.16:0/473665495 conn(0x7f02c8069720 msgr2=0x7f02c8069af0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:54.270 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.259+0000 7f02ce5df640 1 -- 172.21.15.16:0/473665495 shutdown_connections 2023-05-22T15:27:54.270 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.259+0000 7f02ce5df640 1 -- 172.21.15.16:0/473665495 wait complete. 2023-05-22T15:27:54.285 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-05-22T15:27:54.414 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.407+0000 7fb379e52640 1 Processor -- start 2023-05-22T15:27:54.414 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.407+0000 7fb379e52640 1 -- start start 2023-05-22T15:27:54.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.407+0000 7fb379e52640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb3741563c0 0x7fb3740c7b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:54.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.407+0000 7fb379e52640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb3740cc6c0 0x7fb3740c8080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:54.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.407+0000 7fb379e52640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fb37406c220 0x7fb37406c6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:54.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.407+0000 7fb379e52640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb37415c210 con 0x7fb3741563c0 2023-05-22T15:27:54.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.407+0000 7fb379e52640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb37415c480 con 0x7fb37406c220 2023-05-22T15:27:54.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.407+0000 7fb379e52640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb37415c760 con 0x7fb3740cc6c0 2023-05-22T15:27:54.417 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.407+0000 7fb3737fe640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb3741563c0 0x7fb3740c7b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:54.417 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.407+0000 7fb3737fe640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb3741563c0 0x7fb3740c7b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54260/0 (socket says 172.21.15.16:54260) 2023-05-22T15:27:54.417 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.407+0000 7fb3737fe640 1 -- 172.21.15.16:0/162649768 learned_addr learned my addr 172.21.15.16:0/162649768 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:54.417 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.407+0000 7fb372ffd640 1 --2- 172.21.15.16:0/162649768 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb3740cc6c0 0x7fb3740c8080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:54.417 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.407+0000 7fb373fff640 1 --2- 172.21.15.16:0/162649768 >> v2:172.21.15.62:3300/0 conn(0x7fb37406c220 0x7fb37406c6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:54.417 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.407+0000 7fb3737fe640 1 -- 172.21.15.16:0/162649768 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb3740cc6c0 msgr2=0x7fb3740c8080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:54.417 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.407+0000 7fb3737fe640 1 --2- 172.21.15.16:0/162649768 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb3740cc6c0 0x7fb3740c8080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:54.418 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.407+0000 7fb3737fe640 1 -- 172.21.15.16:0/162649768 >> v2:172.21.15.62:3300/0 conn(0x7fb37406c220 msgr2=0x7fb37406c6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:54.418 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.407+0000 7fb3737fe640 1 --2- 172.21.15.16:0/162649768 >> v2:172.21.15.62:3300/0 conn(0x7fb37406c220 0x7fb37406c6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:54.418 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.407+0000 7fb3737fe640 1 -- 172.21.15.16:0/162649768 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb37415ca40 con 0x7fb3741563c0 2023-05-22T15:27:54.418 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.407+0000 7fb372ffd640 1 --2- 172.21.15.16:0/162649768 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb3740cc6c0 0x7fb3740c8080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:27:54.418 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.407+0000 7fb373fff640 1 --2- 172.21.15.16:0/162649768 >> v2:172.21.15.62:3300/0 conn(0x7fb37406c220 0x7fb37406c6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:27:54.419 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.411+0000 7fb3737fe640 1 --2- 172.21.15.16:0/162649768 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb3741563c0 0x7fb3740c7b40 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fb374159cd0 tx=0x7fb36c01c9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9fb7d4373cf9291 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:54.420 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.411+0000 7fb3727fc640 1 -- 172.21.15.16:0/162649768 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb36c0a5840 con 0x7fb3741563c0 2023-05-22T15:27:54.420 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.411+0000 7fb3727fc640 1 -- 172.21.15.16:0/162649768 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb36c0a59a0 con 0x7fb3741563c0 2023-05-22T15:27:54.420 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.411+0000 7fb3727fc640 1 -- 172.21.15.16:0/162649768 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb36c0a5d30 con 0x7fb3741563c0 2023-05-22T15:27:54.420 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.411+0000 7fb379e52640 1 -- 172.21.15.16:0/162649768 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb3741563c0 msgr2=0x7fb3740c7b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:54.420 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.411+0000 7fb379e52640 1 --2- 172.21.15.16:0/162649768 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb3741563c0 0x7fb3740c7b40 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fb374159cd0 tx=0x7fb36c01c9a0 comp rx=0 tx=0).stop 2023-05-22T15:27:54.420 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.411+0000 7fb379e52640 1 -- 172.21.15.16:0/162649768 shutdown_connections 2023-05-22T15:27:54.420 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.411+0000 7fb379e52640 1 --2- 172.21.15.16:0/162649768 >> v2:172.21.15.62:3300/0 conn(0x7fb37406c220 0x7fb37406c6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:54.421 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.411+0000 7fb379e52640 1 --2- 172.21.15.16:0/162649768 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb3740cc6c0 0x7fb3740c8080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:54.421 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.411+0000 7fb379e52640 1 --2- 172.21.15.16:0/162649768 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb3741563c0 0x7fb3740c7b40 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:54.421 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.411+0000 7fb379e52640 1 -- 172.21.15.16:0/162649768 >> 172.21.15.16:0/162649768 conn(0x7fb3740b1110 msgr2=0x7fb3740ce830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:54.421 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.411+0000 7fb379e52640 1 -- 172.21.15.16:0/162649768 shutdown_connections 2023-05-22T15:27:54.421 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.411+0000 7fb379e52640 1 -- 172.21.15.16:0/162649768 wait complete. 2023-05-22T15:27:54.421 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.411+0000 7fb379e52640 1 Processor -- start 2023-05-22T15:27:54.421 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.411+0000 7fb379e52640 1 -- start start 2023-05-22T15:27:54.424 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.411+0000 7fb379e52640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb37406c220 0x7fb37415ed60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:54.425 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.411+0000 7fb379e52640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb3740cc6c0 0x7fb3741442a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:54.425 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.411+0000 7fb379e52640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb374150200 0x7fb374150680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:54.425 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.411+0000 7fb379e52640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb374150bc0 con 0x7fb374150200 2023-05-22T15:27:54.425 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.411+0000 7fb379e52640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb374145ac0 con 0x7fb37406c220 2023-05-22T15:27:54.425 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.411+0000 7fb379e52640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb374144eb0 con 0x7fb3740cc6c0 2023-05-22T15:27:54.425 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.411+0000 7fb373fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb374150200 0x7fb374150680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:54.426 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.411+0000 7fb373fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb374150200 0x7fb374150680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54264/0 (socket says 172.21.15.16:54264) 2023-05-22T15:27:54.426 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.411+0000 7fb373fff640 1 -- 172.21.15.16:0/3862322763 learned_addr learned my addr 172.21.15.16:0/3862322763 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:54.426 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.415+0000 7fb372ffd640 1 --2- 172.21.15.16:0/3862322763 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb3740cc6c0 0x7fb3741442a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:54.426 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.415+0000 7fb3737fe640 1 --2- 172.21.15.16:0/3862322763 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb37406c220 0x7fb37415ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:54.426 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.415+0000 7fb373fff640 1 -- 172.21.15.16:0/3862322763 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb3740cc6c0 msgr2=0x7fb3741442a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:54.426 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.415+0000 7fb373fff640 1 --2- 172.21.15.16:0/3862322763 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb3740cc6c0 0x7fb3741442a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:54.427 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.415+0000 7fb373fff640 1 -- 172.21.15.16:0/3862322763 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb37406c220 msgr2=0x7fb37415ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:54.427 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.415+0000 7fb373fff640 1 --2- 172.21.15.16:0/3862322763 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb37406c220 0x7fb37415ed60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:54.427 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.415+0000 7fb373fff640 1 -- 172.21.15.16:0/3862322763 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb36c009b30 con 0x7fb374150200 2023-05-22T15:27:54.427 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.415+0000 7fb373fff640 1 --2- 172.21.15.16:0/3862322763 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb374150200 0x7fb374150680 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fb36400c820 tx=0x7fb36400cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:54.427 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.415+0000 7fb35bfff640 1 -- 172.21.15.16:0/3862322763 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3640079d0 con 0x7fb374150200 2023-05-22T15:27:54.428 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.415+0000 7fb35bfff640 1 -- 172.21.15.16:0/3862322763 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb364007b70 con 0x7fb374150200 2023-05-22T15:27:54.428 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.415+0000 7fb35bfff640 1 -- 172.21.15.16:0/3862322763 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb36400ce10 con 0x7fb374150200 2023-05-22T15:27:54.428 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.415+0000 7fb379e52640 1 -- 172.21.15.16:0/3862322763 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb374070890 con 0x7fb374150200 2023-05-22T15:27:54.428 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.415+0000 7fb379e52640 1 -- 172.21.15.16:0/3862322763 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb374070700 con 0x7fb374150200 2023-05-22T15:27:54.429 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.415+0000 7fb379e52640 1 -- 172.21.15.16:0/3862322763 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb374070a20 con 0x7fb374150200 2023-05-22T15:27:54.429 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.415+0000 7fb35bfff640 1 -- 172.21.15.16:0/3862322763 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7fb3640113f0 con 0x7fb374150200 2023-05-22T15:27:54.429 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.415+0000 7fb35bfff640 1 --2- 172.21.15.16:0/3862322763 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fb3600653b0 0x7fb360067860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:54.429 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.415+0000 7fb35bfff640 1 -- 172.21.15.16:0/3862322763 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6871+0+0 (secure 0 0 0) 0x7fb364082d00 con 0x7fb374150200 2023-05-22T15:27:54.429 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.415+0000 7fb3737fe640 1 --2- 172.21.15.16:0/3862322763 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fb3600653b0 0x7fb360067860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:54.429 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.415+0000 7fb3737fe640 1 --2- 172.21.15.16:0/3862322763 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fb3600653b0 0x7fb360067860 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fb36c0ae000 tx=0x7fb36c0a7070 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:54.430 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.419+0000 7fb35bfff640 1 -- 172.21.15.16:0/3862322763 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fb3640505f0 con 0x7fb374150200 2023-05-22T15:27:54.580 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.571+0000 7fb379e52640 1 -- 172.21.15.16:0/3862322763 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fb37404dbe0 con 0x7fb374150200 2023-05-22T15:27:54.581 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.571+0000 7fb35bfff640 1 -- 172.21.15.16:0/3862322763 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v29) v1 ==== 104+0+11 (secure 0 0 0) 0x7fb36401f020 con 0x7fb374150200 2023-05-22T15:27:54.581 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2023-05-22T15:27:54.582 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.575+0000 7fb379e52640 1 -- 172.21.15.16:0/3862322763 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fb3600653b0 msgr2=0x7fb360067860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:54.583 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.575+0000 7fb379e52640 1 --2- 172.21.15.16:0/3862322763 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fb3600653b0 0x7fb360067860 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fb36c0ae000 tx=0x7fb36c0a7070 comp rx=0 tx=0).stop 2023-05-22T15:27:54.584 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.575+0000 7fb379e52640 1 -- 172.21.15.16:0/3862322763 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb374150200 msgr2=0x7fb374150680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:54.584 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.575+0000 7fb379e52640 1 --2- 172.21.15.16:0/3862322763 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb374150200 0x7fb374150680 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fb36400c820 tx=0x7fb36400cce0 comp rx=0 tx=0).stop 2023-05-22T15:27:54.584 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.575+0000 7fb379e52640 1 -- 172.21.15.16:0/3862322763 shutdown_connections 2023-05-22T15:27:54.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.575+0000 7fb379e52640 1 --2- 172.21.15.16:0/3862322763 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb374150200 0x7fb374150680 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:54.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.575+0000 7fb379e52640 1 --2- 172.21.15.16:0/3862322763 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fb3600653b0 0x7fb360067860 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:54.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.575+0000 7fb379e52640 1 --2- 172.21.15.16:0/3862322763 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb3740cc6c0 0x7fb3741442a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:54.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.575+0000 7fb379e52640 1 --2- 172.21.15.16:0/3862322763 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb37406c220 0x7fb37415ed60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:54.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.575+0000 7fb379e52640 1 -- 172.21.15.16:0/3862322763 >> 172.21.15.16:0/3862322763 conn(0x7fb3740b1110 msgr2=0x7fb37406a4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:54.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.575+0000 7fb379e52640 1 -- 172.21.15.16:0/3862322763 shutdown_connections 2023-05-22T15:27:54.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.575+0000 7fb379e52640 1 -- 172.21.15.16:0/3862322763 wait complete. 2023-05-22T15:27:54.599 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-05-22T15:27:54.726 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.719+0000 7f683e249640 1 Processor -- start 2023-05-22T15:27:54.726 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.719+0000 7f683e249640 1 -- start start 2023-05-22T15:27:54.727 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.719+0000 7f683e249640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f683806b6c0 0x7f68380cb4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:54.727 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.719+0000 7f683e249640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f683806ba90 0x7f6838158fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:54.727 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.719+0000 7f683e249640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6838159f70 con 0x7f6838159510 2023-05-22T15:27:54.727 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.719+0000 7f683e249640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6838071760 con 0x7f683806ba90 2023-05-22T15:27:54.727 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.719+0000 7f683e249640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68380ca7d0 con 0x7f683806b6c0 2023-05-22T15:27:54.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.719+0000 7f68377fe640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f683806b6c0 0x7f68380cb4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:54.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.719+0000 7f682ffff640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f683806ba90 0x7f6838158fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:54.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.719+0000 7f6837fff640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f6838159510 0x7f68380cc740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.16:57264/0 (socket says 172.21.15.16:57264) 2023-05-22T15:27:54.729 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.719+0000 7f6837fff640 1 -- 172.21.15.16:0/1442109963 learned_addr learned my addr 172.21.15.16:0/1442109963 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:27:54.729 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.719+0000 7f68377fe640 1 -- 172.21.15.16:0/1442109963 >> v2:172.21.15.62:3300/0 conn(0x7f683806ba90 msgr2=0x7f6838158fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:54.729 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.719+0000 7f68377fe640 1 --2- 172.21.15.16:0/1442109963 >> v2:172.21.15.62:3300/0 conn(0x7f683806ba90 0x7f6838158fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:54.729 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.719+0000 7f68377fe640 1 -- 172.21.15.16:0/1442109963 >> v1:172.21.15.62:6789/0 conn(0x7f6838159510 legacy=0x7f68380cc740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:54.730 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.719+0000 7f68377fe640 1 -- 172.21.15.16:0/1442109963 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68380cb030 con 0x7f683806b6c0 2023-05-22T15:27:54.730 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.719+0000 7f682ffff640 1 --2- 172.21.15.16:0/1442109963 >> v2:172.21.15.62:3300/0 conn(0x7f683806ba90 0x7f6838158fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-05-22T15:27:54.730 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.723+0000 7f68377fe640 1 --2- 172.21.15.16:0/1442109963 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f683806b6c0 0x7f68380cb4c0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f68280097c0 tx=0x7f682801c450 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2183886bfed558e5 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:54.731 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.723+0000 7f6836ffd640 1 -- 172.21.15.16:0/1442109963 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68280ab600 con 0x7f683806b6c0 2023-05-22T15:27:54.732 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.723+0000 7f6836ffd640 1 -- 172.21.15.16:0/1442109963 <== mon.2 v2:172.21.15.62:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68280a7050 con 0x7f683806b6c0 2023-05-22T15:27:54.733 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.723+0000 7f6836ffd640 1 -- 172.21.15.16:0/1442109963 <== mon.2 v2:172.21.15.62:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68280ab980 con 0x7f683806b6c0 2023-05-22T15:27:54.735 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.723+0000 7f683e249640 1 -- 172.21.15.16:0/1442109963 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f683806b6c0 msgr2=0x7f68380cb4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:54.735 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.723+0000 7f683e249640 1 --2- 172.21.15.16:0/1442109963 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f683806b6c0 0x7f68380cb4c0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f68280097c0 tx=0x7f682801c450 comp rx=0 tx=0).stop 2023-05-22T15:27:54.735 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.723+0000 7f683e249640 1 -- 172.21.15.16:0/1442109963 shutdown_connections 2023-05-22T15:27:54.736 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.723+0000 7f683e249640 1 --2- 172.21.15.16:0/1442109963 >> v2:172.21.15.62:3300/0 conn(0x7f683806ba90 0x7f6838158fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:54.736 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.723+0000 7f683e249640 1 --2- 172.21.15.16:0/1442109963 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f683806b6c0 0x7f68380cb4c0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:54.736 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.723+0000 7f683e249640 1 -- 172.21.15.16:0/1442109963 >> 172.21.15.16:0/1442109963 conn(0x7f6838069720 msgr2=0x7f6838069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:54.736 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.723+0000 7f683e249640 1 -- 172.21.15.16:0/1442109963 shutdown_connections 2023-05-22T15:27:54.736 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.723+0000 7f683e249640 1 -- 172.21.15.16:0/1442109963 wait complete. 2023-05-22T15:27:54.736 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.723+0000 7f683e249640 1 Processor -- start 2023-05-22T15:27:54.736 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.727+0000 7f683e249640 1 -- start start 2023-05-22T15:27:54.737 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.727+0000 7f683e249640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f683806b6c0 0x7f683814bc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:54.737 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.727+0000 7f683e249640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f683806ba90 0x7f6838157250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:54.737 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.727+0000 7f683e249640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6838159510 0x7f6838157790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:54.737 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.727+0000 7f683e249640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68380af420 con 0x7f683806b6c0 2023-05-22T15:27:54.737 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.727+0000 7f683e249640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68380ae7a0 con 0x7f683806ba90 2023-05-22T15:27:54.737 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.727+0000 7f683e249640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68380aea80 con 0x7f6838159510 2023-05-22T15:27:54.737 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.727+0000 7f68377fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f683806b6c0 0x7f683814bc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:54.738 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.727+0000 7f68377fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f683806b6c0 0x7f683814bc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54266/0 (socket says 172.21.15.16:54266) 2023-05-22T15:27:54.738 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.727+0000 7f68377fe640 1 -- 172.21.15.16:0/3698121049 learned_addr learned my addr 172.21.15.16:0/3698121049 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:54.738 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.727+0000 7f682ffff640 1 --2- 172.21.15.16:0/3698121049 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f683806ba90 0x7f6838157250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:54.738 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.727+0000 7f6837fff640 1 --2- 172.21.15.16:0/3698121049 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6838159510 0x7f6838157790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:54.738 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.727+0000 7f68377fe640 1 -- 172.21.15.16:0/3698121049 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6838159510 msgr2=0x7f6838157790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:54.738 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.727+0000 7f68377fe640 1 --2- 172.21.15.16:0/3698121049 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6838159510 0x7f6838157790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:54.739 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.727+0000 7f68377fe640 1 -- 172.21.15.16:0/3698121049 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f683806ba90 msgr2=0x7f6838157250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:54.739 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.727+0000 7f68377fe640 1 --2- 172.21.15.16:0/3698121049 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f683806ba90 0x7f6838157250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:54.739 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.727+0000 7f68377fe640 1 -- 172.21.15.16:0/3698121049 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6820009420 con 0x7f683806b6c0 2023-05-22T15:27:54.739 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.727+0000 7f68377fe640 1 --2- 172.21.15.16:0/3698121049 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f683806b6c0 0x7f683814bc50 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f68280ae040 tx=0x7f682801ac90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:54.739 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.727+0000 7f6834ff9640 1 -- 172.21.15.16:0/3698121049 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68280b7450 con 0x7f683806b6c0 2023-05-22T15:27:54.739 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.727+0000 7f6834ff9640 1 -- 172.21.15.16:0/3698121049 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68280b75b0 con 0x7f683806b6c0 2023-05-22T15:27:54.740 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.727+0000 7f683e249640 1 -- 172.21.15.16:0/3698121049 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6828009420 con 0x7f683806b6c0 2023-05-22T15:27:54.741 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.727+0000 7f6834ff9640 1 -- 172.21.15.16:0/3698121049 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68280b7960 con 0x7f683806b6c0 2023-05-22T15:27:54.741 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.727+0000 7f683e249640 1 -- 172.21.15.16:0/3698121049 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6838159910 con 0x7f683806b6c0 2023-05-22T15:27:54.742 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.727+0000 7f683e249640 1 -- 172.21.15.16:0/3698121049 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68380cacc0 con 0x7f683806b6c0 2023-05-22T15:27:54.742 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.727+0000 7f6834ff9640 1 -- 172.21.15.16:0/3698121049 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7f68280b7ac0 con 0x7f683806b6c0 2023-05-22T15:27:54.742 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.727+0000 7f6834ff9640 1 --2- 172.21.15.16:0/3698121049 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f6818065380 0x7f6818067830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:54.742 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.727+0000 7f6834ff9640 1 -- 172.21.15.16:0/3698121049 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f682811d410 con 0x7f683806b6c0 2023-05-22T15:27:54.742 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.727+0000 7f682ffff640 1 --2- 172.21.15.16:0/3698121049 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f6818065380 0x7f6818067830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:54.742 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.731+0000 7f682ffff640 1 --2- 172.21.15.16:0/3698121049 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f6818065380 0x7f6818067830 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f6838163800 tx=0x7f6820008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:54.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.731+0000 7f6834ff9640 1 -- 172.21.15.16:0/3698121049 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f68280eadf0 con 0x7f683806b6c0 2023-05-22T15:27:54.899 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.891+0000 7f683e249640 1 -- 172.21.15.16:0/3698121049 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f683806be60 con 0x7f683806b6c0 2023-05-22T15:27:54.900 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.891+0000 7f6834ff9640 1 -- 172.21.15.16:0/3698121049 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v29) v1 ==== 100+0+11 (secure 0 0 0) 0x7f68280ee410 con 0x7f683806b6c0 2023-05-22T15:27:54.900 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 64 2023-05-22T15:27:54.902 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.895+0000 7f683e249640 1 -- 172.21.15.16:0/3698121049 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f6818065380 msgr2=0x7f6818067830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:54.902 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.895+0000 7f683e249640 1 --2- 172.21.15.16:0/3698121049 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f6818065380 0x7f6818067830 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f6838163800 tx=0x7f6820008040 comp rx=0 tx=0).stop 2023-05-22T15:27:54.903 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.895+0000 7f683e249640 1 -- 172.21.15.16:0/3698121049 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f683806b6c0 msgr2=0x7f683814bc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:54.903 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.895+0000 7f683e249640 1 --2- 172.21.15.16:0/3698121049 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f683806b6c0 0x7f683814bc50 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f68280ae040 tx=0x7f682801ac90 comp rx=0 tx=0).stop 2023-05-22T15:27:54.903 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.895+0000 7f683e249640 1 -- 172.21.15.16:0/3698121049 shutdown_connections 2023-05-22T15:27:54.903 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.895+0000 7f683e249640 1 --2- 172.21.15.16:0/3698121049 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f6818065380 0x7f6818067830 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:54.903 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.895+0000 7f683e249640 1 --2- 172.21.15.16:0/3698121049 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6838159510 0x7f6838157790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:54.903 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.895+0000 7f683e249640 1 --2- 172.21.15.16:0/3698121049 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f683806ba90 0x7f6838157250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:54.903 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.895+0000 7f683e249640 1 --2- 172.21.15.16:0/3698121049 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f683806b6c0 0x7f683814bc50 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:54.904 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.895+0000 7f683e249640 1 -- 172.21.15.16:0/3698121049 >> 172.21.15.16:0/3698121049 conn(0x7f6838069720 msgr2=0x7f68380cc720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:54.904 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.895+0000 7f683e249640 1 -- 172.21.15.16:0/3698121049 shutdown_connections 2023-05-22T15:27:54.904 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:54.895+0000 7f683e249640 1 -- 172.21.15.16:0/3698121049 wait complete. 2023-05-22T15:27:54.919 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-05-22T15:27:55.053 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.043+0000 7f0e4f490640 1 Processor -- start 2023-05-22T15:27:55.053 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.043+0000 7f0e4f490640 1 -- start start 2023-05-22T15:27:55.054 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.047+0000 7f0e4f490640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f0e4815bca0 0x7f0e480cda80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:55.054 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.047+0000 7f0e4f490640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0e480d2480 0x7f0e480d28d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:55.055 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.047+0000 7f0e4f490640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f0e480b5340 0x7f0e480b57c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:55.055 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.047+0000 7f0e4f490640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e480cd660 con 0x7f0e480b5340 2023-05-22T15:27:55.055 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.047+0000 7f0e4f490640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e480d2e10 con 0x7f0e4815bca0 2023-05-22T15:27:55.055 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.047+0000 7f0e4f490640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e480cccc0 con 0x7f0e480d2480 2023-05-22T15:27:55.055 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.047+0000 7f0e4da2d640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f0e480b5340 0x7f0e480b57c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:55.055 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.047+0000 7f0e4da2d640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f0e480b5340 0x7f0e480b57c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54272/0 (socket says 172.21.15.16:54272) 2023-05-22T15:27:55.055 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.047+0000 7f0e4da2d640 1 -- 172.21.15.16:0/647919605 learned_addr learned my addr 172.21.15.16:0/647919605 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:55.056 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.047+0000 7f0e4ca2b640 1 --2- 172.21.15.16:0/647919605 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0e480d2480 0x7f0e480d28d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:55.056 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.047+0000 7f0e4d22c640 1 --2- 172.21.15.16:0/647919605 >> v2:172.21.15.62:3300/0 conn(0x7f0e4815bca0 0x7f0e480cda80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:55.056 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.047+0000 7f0e4da2d640 1 -- 172.21.15.16:0/647919605 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0e480d2480 msgr2=0x7f0e480d28d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:55.056 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.047+0000 7f0e4da2d640 1 --2- 172.21.15.16:0/647919605 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0e480d2480 0x7f0e480d28d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:55.056 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.047+0000 7f0e4da2d640 1 -- 172.21.15.16:0/647919605 >> v2:172.21.15.62:3300/0 conn(0x7f0e4815bca0 msgr2=0x7f0e480cda80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:55.056 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.047+0000 7f0e4da2d640 1 --2- 172.21.15.16:0/647919605 >> v2:172.21.15.62:3300/0 conn(0x7f0e4815bca0 0x7f0e480cda80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:55.056 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.047+0000 7f0e4da2d640 1 -- 172.21.15.16:0/647919605 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e480ccfa0 con 0x7f0e480b5340 2023-05-22T15:27:55.056 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.047+0000 7f0e4d22c640 1 --2- 172.21.15.16:0/647919605 >> v2:172.21.15.62:3300/0 conn(0x7f0e4815bca0 0x7f0e480cda80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:27:55.058 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.047+0000 7f0e4da2d640 1 --2- 172.21.15.16:0/647919605 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e480b5340 0x7f0e480b57c0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f0e480b40d0 tx=0x7f0e4001c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e9497f800b888c4 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:55.058 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.047+0000 7f0e47fff640 1 -- 172.21.15.16:0/647919605 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e400aa610 con 0x7f0e480b5340 2023-05-22T15:27:55.058 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.047+0000 7f0e47fff640 1 -- 172.21.15.16:0/647919605 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e400a6040 con 0x7f0e480b5340 2023-05-22T15:27:55.058 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.047+0000 7f0e47fff640 1 -- 172.21.15.16:0/647919605 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e400aa990 con 0x7f0e480b5340 2023-05-22T15:27:55.058 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.051+0000 7f0e4f490640 1 -- 172.21.15.16:0/647919605 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e480b5340 msgr2=0x7f0e480b57c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:55.058 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.051+0000 7f0e4f490640 1 --2- 172.21.15.16:0/647919605 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e480b5340 0x7f0e480b57c0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f0e480b40d0 tx=0x7f0e4001c4d0 comp rx=0 tx=0).stop 2023-05-22T15:27:55.058 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.051+0000 7f0e4f490640 1 -- 172.21.15.16:0/647919605 shutdown_connections 2023-05-22T15:27:55.059 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.051+0000 7f0e4f490640 1 --2- 172.21.15.16:0/647919605 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e480b5340 0x7f0e480b57c0 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:55.059 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.051+0000 7f0e4f490640 1 --2- 172.21.15.16:0/647919605 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0e480d2480 0x7f0e480d28d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:55.059 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.051+0000 7f0e4f490640 1 --2- 172.21.15.16:0/647919605 >> v2:172.21.15.62:3300/0 conn(0x7f0e4815bca0 0x7f0e480cda80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:55.059 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.051+0000 7f0e4f490640 1 -- 172.21.15.16:0/647919605 >> 172.21.15.16:0/647919605 conn(0x7f0e480655c0 msgr2=0x7f0e48070e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:55.059 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.051+0000 7f0e4f490640 1 -- 172.21.15.16:0/647919605 shutdown_connections 2023-05-22T15:27:55.059 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.051+0000 7f0e4f490640 1 -- 172.21.15.16:0/647919605 wait complete. 2023-05-22T15:27:55.061 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.051+0000 7f0e4f490640 1 Processor -- start 2023-05-22T15:27:55.061 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.051+0000 7f0e4f490640 1 -- start start 2023-05-22T15:27:55.061 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.051+0000 7f0e4f490640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e480b5340 0x7f0e48159cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:55.062 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.051+0000 7f0e4f490640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0e480d2480 0x7f0e48152080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:55.062 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.051+0000 7f0e4f490640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0e4815bca0 0x7f0e481525c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:55.062 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.051+0000 7f0e4f490640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e480ccb50 con 0x7f0e480b5340 2023-05-22T15:27:55.062 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.051+0000 7f0e4f490640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e480c7b50 con 0x7f0e480d2480 2023-05-22T15:27:55.062 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.051+0000 7f0e4f490640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e480c7fe0 con 0x7f0e4815bca0 2023-05-22T15:27:55.062 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.051+0000 7f0e4d22c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e480b5340 0x7f0e48159cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:55.062 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.051+0000 7f0e4d22c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e480b5340 0x7f0e48159cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54280/0 (socket says 172.21.15.16:54280) 2023-05-22T15:27:55.063 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.051+0000 7f0e4d22c640 1 -- 172.21.15.16:0/149538902 learned_addr learned my addr 172.21.15.16:0/149538902 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:55.063 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.051+0000 7f0e4da2d640 1 --2- 172.21.15.16:0/149538902 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0e4815bca0 0x7f0e481525c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:55.063 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.051+0000 7f0e4ca2b640 1 --2- 172.21.15.16:0/149538902 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0e480d2480 0x7f0e48152080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:55.063 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.051+0000 7f0e4d22c640 1 -- 172.21.15.16:0/149538902 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0e4815bca0 msgr2=0x7f0e481525c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:55.063 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.051+0000 7f0e4d22c640 1 --2- 172.21.15.16:0/149538902 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0e4815bca0 0x7f0e481525c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:55.063 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.051+0000 7f0e4d22c640 1 -- 172.21.15.16:0/149538902 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0e480d2480 msgr2=0x7f0e48152080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:55.063 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.051+0000 7f0e4d22c640 1 --2- 172.21.15.16:0/149538902 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0e480d2480 0x7f0e48152080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:55.064 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.051+0000 7f0e4d22c640 1 -- 172.21.15.16:0/149538902 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e400094d0 con 0x7f0e480b5340 2023-05-22T15:27:55.064 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.051+0000 7f0e4da2d640 1 --2- 172.21.15.16:0/149538902 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0e4815bca0 0x7f0e481525c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:27:55.064 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.051+0000 7f0e4ca2b640 1 --2- 172.21.15.16:0/149538902 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0e480d2480 0x7f0e48152080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:27:55.064 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.051+0000 7f0e4d22c640 1 --2- 172.21.15.16:0/149538902 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e480b5340 0x7f0e48159cc0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f0e3800ce00 tx=0x7f0e3800a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:55.064 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.051+0000 7f0e45ffb640 1 -- 172.21.15.16:0/149538902 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e3800f890 con 0x7f0e480b5340 2023-05-22T15:27:55.064 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.051+0000 7f0e45ffb640 1 -- 172.21.15.16:0/149538902 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e3800f9f0 con 0x7f0e480b5340 2023-05-22T15:27:55.064 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.051+0000 7f0e4f490640 1 -- 172.21.15.16:0/149538902 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e480cdfc0 con 0x7f0e480b5340 2023-05-22T15:27:55.064 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.051+0000 7f0e45ffb640 1 -- 172.21.15.16:0/149538902 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e3800fd80 con 0x7f0e480b5340 2023-05-22T15:27:55.065 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.051+0000 7f0e4f490640 1 -- 172.21.15.16:0/149538902 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e480b5d00 con 0x7f0e480b5340 2023-05-22T15:27:55.065 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.055+0000 7f0e4f490640 1 -- 172.21.15.16:0/149538902 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e48160360 con 0x7f0e480b5340 2023-05-22T15:27:55.065 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.055+0000 7f0e45ffb640 1 -- 172.21.15.16:0/149538902 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7f0e3800b040 con 0x7f0e480b5340 2023-05-22T15:27:55.066 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.055+0000 7f0e45ffb640 1 --2- 172.21.15.16:0/149538902 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f0e340653b0 0x7f0e34067860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:55.066 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.055+0000 7f0e45ffb640 1 -- 172.21.15.16:0/149538902 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f0e38083500 con 0x7f0e480b5340 2023-05-22T15:27:55.066 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.055+0000 7f0e4ca2b640 1 --2- 172.21.15.16:0/149538902 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f0e340653b0 0x7f0e34067860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:55.066 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.055+0000 7f0e4ca2b640 1 --2- 172.21.15.16:0/149538902 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f0e340653b0 0x7f0e34067860 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f0e480b75c0 tx=0x7f0e3c008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:55.066 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.059+0000 7f0e45ffb640 1 -- 172.21.15.16:0/149538902 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f0e38050e80 con 0x7f0e480b5340 2023-05-22T15:27:55.219 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.211+0000 7f0e4f490640 1 -- 172.21.15.16:0/149538902 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f0e4804dbe0 con 0x7f0e480b5340 2023-05-22T15:27:55.220 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.211+0000 7f0e45ffb640 1 -- 172.21.15.16:0/149538902 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v29) v1 ==== 103+0+11 (secure 0 0 0) 0x7f0e380544a0 con 0x7f0e480b5340 2023-05-22T15:27:55.220 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2023-05-22T15:27:55.222 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.215+0000 7f0e4f490640 1 -- 172.21.15.16:0/149538902 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f0e340653b0 msgr2=0x7f0e34067860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:55.222 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.215+0000 7f0e4f490640 1 --2- 172.21.15.16:0/149538902 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f0e340653b0 0x7f0e34067860 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f0e480b75c0 tx=0x7f0e3c008040 comp rx=0 tx=0).stop 2023-05-22T15:27:55.223 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.215+0000 7f0e4f490640 1 -- 172.21.15.16:0/149538902 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e480b5340 msgr2=0x7f0e48159cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:55.223 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.215+0000 7f0e4f490640 1 --2- 172.21.15.16:0/149538902 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e480b5340 0x7f0e48159cc0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f0e3800ce00 tx=0x7f0e3800a650 comp rx=0 tx=0).stop 2023-05-22T15:27:55.223 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.215+0000 7f0e4f490640 1 -- 172.21.15.16:0/149538902 shutdown_connections 2023-05-22T15:27:55.223 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.215+0000 7f0e4f490640 1 --2- 172.21.15.16:0/149538902 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f0e340653b0 0x7f0e34067860 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:55.223 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.215+0000 7f0e4f490640 1 --2- 172.21.15.16:0/149538902 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0e4815bca0 0x7f0e481525c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:55.223 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.215+0000 7f0e4f490640 1 --2- 172.21.15.16:0/149538902 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0e480d2480 0x7f0e48152080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:55.223 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.215+0000 7f0e4f490640 1 --2- 172.21.15.16:0/149538902 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e480b5340 0x7f0e48159cc0 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:55.224 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.215+0000 7f0e4f490640 1 -- 172.21.15.16:0/149538902 >> 172.21.15.16:0/149538902 conn(0x7f0e480655c0 msgr2=0x7f0e480b6a50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:55.224 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.215+0000 7f0e4f490640 1 -- 172.21.15.16:0/149538902 shutdown_connections 2023-05-22T15:27:55.224 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.215+0000 7f0e4f490640 1 -- 172.21.15.16:0/149538902 wait complete. 2023-05-22T15:27:55.239 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-05-22T15:27:55.362 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.355+0000 7f45aadb1640 1 Processor -- start 2023-05-22T15:27:55.363 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.355+0000 7f45aadb1640 1 -- start start 2023-05-22T15:27:55.363 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.355+0000 7f45aadb1640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f45a41531b0 0x7f45a40afb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:55.364 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.355+0000 7f45aadb1640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45a415a0a0 con 0x7f45a40b0080 2023-05-22T15:27:55.364 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.355+0000 7f45aadb1640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45a4159e50 con 0x7f45a4159a40 2023-05-22T15:27:55.364 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.355+0000 7f45aadb1640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45a40b0520 con 0x7f45a41531b0 2023-05-22T15:27:55.364 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.355+0000 7f45a8b4d640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f45a41531b0 0x7f45a40afb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:55.364 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.355+0000 7f45a8b4d640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f45a41531b0 0x7f45a40afb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54294/0 (socket says 172.21.15.16:54294) 2023-05-22T15:27:55.364 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.355+0000 7f45a8b4d640 1 -- 172.21.15.16:0/2895889599 learned_addr learned my addr 172.21.15.16:0/2895889599 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:55.365 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.355+0000 7f45a8b4d640 1 -- 172.21.15.16:0/2895889599 >> v1:172.21.15.62:6789/0 conn(0x7f45a4159a40 legacy=0x7f45a40c8ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:55.365 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.355+0000 7f45a8b4d640 1 -- 172.21.15.16:0/2895889599 >> v1:172.21.15.16:6789/0 conn(0x7f45a40b0080 legacy=0x7f45a4159330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:55.365 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.355+0000 7f45a8b4d640 1 -- 172.21.15.16:0/2895889599 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45a4153580 con 0x7f45a41531b0 2023-05-22T15:27:55.366 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.359+0000 7f45a8b4d640 1 --2- 172.21.15.16:0/2895889599 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f45a41531b0 0x7f45a40afb40 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f459c009870 tx=0x7f459c01c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9caa53f67235d4ff server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:55.366 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.359+0000 7f45a37fe640 1 -- 172.21.15.16:0/2895889599 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f459c0ab5f0 con 0x7f45a41531b0 2023-05-22T15:27:55.366 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.359+0000 7f45a37fe640 1 -- 172.21.15.16:0/2895889599 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f459c0ab750 con 0x7f45a41531b0 2023-05-22T15:27:55.366 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.359+0000 7f45a37fe640 1 -- 172.21.15.16:0/2895889599 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f459c0abae0 con 0x7f45a41531b0 2023-05-22T15:27:55.367 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.359+0000 7f45aadb1640 1 -- 172.21.15.16:0/2895889599 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f45a41531b0 msgr2=0x7f45a40afb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:55.367 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.359+0000 7f45aadb1640 1 --2- 172.21.15.16:0/2895889599 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f45a41531b0 0x7f45a40afb40 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f459c009870 tx=0x7f459c01c500 comp rx=0 tx=0).stop 2023-05-22T15:27:55.368 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.359+0000 7f45aadb1640 1 -- 172.21.15.16:0/2895889599 shutdown_connections 2023-05-22T15:27:55.368 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.359+0000 7f45aadb1640 1 --2- 172.21.15.16:0/2895889599 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f45a41531b0 0x7f45a40afb40 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:55.368 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.359+0000 7f45aadb1640 1 -- 172.21.15.16:0/2895889599 >> 172.21.15.16:0/2895889599 conn(0x7f45a4069720 msgr2=0x7f45a4069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:55.368 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.359+0000 7f45aadb1640 1 -- 172.21.15.16:0/2895889599 shutdown_connections 2023-05-22T15:27:55.368 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.359+0000 7f45aadb1640 1 -- 172.21.15.16:0/2895889599 wait complete. 2023-05-22T15:27:55.370 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.359+0000 7f45aadb1640 1 Processor -- start 2023-05-22T15:27:55.371 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.359+0000 7f45aadb1640 1 -- start start 2023-05-22T15:27:55.371 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.363+0000 7f45aadb1640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f45a40b0080 0x7f45a41489a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:55.371 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.363+0000 7f45aadb1640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f45a41531b0 0x7f45a414d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:55.371 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.363+0000 7f45aadb1640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f45a4159a40 0x7f45a414dde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:55.371 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.363+0000 7f45aadb1640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45a4149250 con 0x7f45a4159a40 2023-05-22T15:27:55.372 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.363+0000 7f45aadb1640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45a414a540 con 0x7f45a40b0080 2023-05-22T15:27:55.372 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.363+0000 7f45aadb1640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45a41494a0 con 0x7f45a41531b0 2023-05-22T15:27:55.372 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.363+0000 7f45a934e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f45a4159a40 0x7f45a414dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:55.372 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.363+0000 7f45a934e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f45a4159a40 0x7f45a414dde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54310/0 (socket says 172.21.15.16:54310) 2023-05-22T15:27:55.372 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.363+0000 7f45a934e640 1 -- 172.21.15.16:0/672723097 learned_addr learned my addr 172.21.15.16:0/672723097 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:55.373 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.363+0000 7f45a3fff640 1 --2- 172.21.15.16:0/672723097 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f45a41531b0 0x7f45a414d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:55.373 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.363+0000 7f45a8b4d640 1 --2- 172.21.15.16:0/672723097 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f45a40b0080 0x7f45a41489a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:55.373 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.363+0000 7f45a934e640 1 -- 172.21.15.16:0/672723097 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f45a41531b0 msgr2=0x7f45a414d8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:55.373 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.363+0000 7f45a934e640 1 --2- 172.21.15.16:0/672723097 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f45a41531b0 0x7f45a414d8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:55.373 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.363+0000 7f45a934e640 1 -- 172.21.15.16:0/672723097 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f45a40b0080 msgr2=0x7f45a41489a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:55.373 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.363+0000 7f45a934e640 1 --2- 172.21.15.16:0/672723097 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f45a40b0080 0x7f45a41489a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:55.373 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.363+0000 7f45a934e640 1 -- 172.21.15.16:0/672723097 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f459c0094d0 con 0x7f45a4159a40 2023-05-22T15:27:55.374 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.363+0000 7f45a934e640 1 --2- 172.21.15.16:0/672723097 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f45a4159a40 0x7f45a414dde0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f459400c3c0 tx=0x7f459400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:55.374 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.363+0000 7f45a17fa640 1 -- 172.21.15.16:0/672723097 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4594011790 con 0x7f45a4159a40 2023-05-22T15:27:55.374 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.363+0000 7f45a17fa640 1 -- 172.21.15.16:0/672723097 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45940118f0 con 0x7f45a4159a40 2023-05-22T15:27:55.374 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.363+0000 7f45aadb1640 1 -- 172.21.15.16:0/672723097 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45a415d0c0 con 0x7f45a4159a40 2023-05-22T15:27:55.374 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.363+0000 7f45a17fa640 1 -- 172.21.15.16:0/672723097 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4594011ca0 con 0x7f45a4159a40 2023-05-22T15:27:55.374 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.363+0000 7f45aadb1640 1 -- 172.21.15.16:0/672723097 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45a415cde0 con 0x7f45a4159a40 2023-05-22T15:27:55.374 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.363+0000 7f45aadb1640 1 -- 172.21.15.16:0/672723097 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45a4070d40 con 0x7f45a4159a40 2023-05-22T15:27:55.375 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.363+0000 7f45a17fa640 1 -- 172.21.15.16:0/672723097 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7f459401c460 con 0x7f45a4159a40 2023-05-22T15:27:55.375 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.363+0000 7f45a17fa640 1 --2- 172.21.15.16:0/672723097 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f4584065380 0x7f4584067830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:55.375 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.363+0000 7f45a17fa640 1 -- 172.21.15.16:0/672723097 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f4594082a70 con 0x7f45a4159a40 2023-05-22T15:27:55.376 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.363+0000 7f45a8b4d640 1 --2- 172.21.15.16:0/672723097 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f4584065380 0x7f4584067830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:55.376 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.363+0000 7f45a8b4d640 1 --2- 172.21.15.16:0/672723097 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f4584065380 0x7f4584067830 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f459c0af000 tx=0x7f459c0a7000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:55.376 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.367+0000 7f45a17fa640 1 -- 172.21.15.16:0/672723097 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f4594050360 con 0x7f45a4159a40 2023-05-22T15:27:55.539 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.531+0000 7f45aadb1640 1 -- 172.21.15.16:0/672723097 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f45a404dbe0 con 0x7f45a4159a40 2023-05-22T15:27:55.540 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.531+0000 7f45a17fa640 1 -- 172.21.15.16:0/672723097 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v29) v1 ==== 80+0+0 (secure 0 0 0) 0x7f4594053980 con 0x7f45a4159a40 2023-05-22T15:27:55.540 INFO:teuthology.orchestra.run.smithi016.stderr:listed 0 entries 2023-05-22T15:27:55.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.535+0000 7f45aadb1640 1 -- 172.21.15.16:0/672723097 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f4584065380 msgr2=0x7f4584067830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:55.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.535+0000 7f45aadb1640 1 --2- 172.21.15.16:0/672723097 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f4584065380 0x7f4584067830 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f459c0af000 tx=0x7f459c0a7000 comp rx=0 tx=0).stop 2023-05-22T15:27:55.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.535+0000 7f45aadb1640 1 -- 172.21.15.16:0/672723097 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f45a4159a40 msgr2=0x7f45a414dde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:55.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.535+0000 7f45aadb1640 1 --2- 172.21.15.16:0/672723097 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f45a4159a40 0x7f45a414dde0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f459400c3c0 tx=0x7f459400c880 comp rx=0 tx=0).stop 2023-05-22T15:27:55.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.535+0000 7f45aadb1640 1 -- 172.21.15.16:0/672723097 shutdown_connections 2023-05-22T15:27:55.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.535+0000 7f45aadb1640 1 --2- 172.21.15.16:0/672723097 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f45a4159a40 0x7f45a414dde0 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:55.544 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.535+0000 7f45aadb1640 1 --2- 172.21.15.16:0/672723097 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f4584065380 0x7f4584067830 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:55.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.535+0000 7f45aadb1640 1 --2- 172.21.15.16:0/672723097 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f45a41531b0 0x7f45a414d8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:55.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.535+0000 7f45aadb1640 1 --2- 172.21.15.16:0/672723097 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f45a40b0080 0x7f45a41489a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:55.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.535+0000 7f45aadb1640 1 -- 172.21.15.16:0/672723097 >> 172.21.15.16:0/672723097 conn(0x7f45a4069720 msgr2=0x7f45a40b37f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:55.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.535+0000 7f45aadb1640 1 -- 172.21.15.16:0/672723097 shutdown_connections 2023-05-22T15:27:55.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.535+0000 7f45aadb1640 1 -- 172.21.15.16:0/672723097 wait complete. 2023-05-22T15:27:55.557 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2023-05-22T15:27:55.557 DEBUG:teuthology.orchestra.run.smithi062:> dd if=/proc/self/mounts of=/dev/stdout 2023-05-22T15:27:55.565 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-05-22T15:27:55.680 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.671+0000 7fa549920640 1 Processor -- start 2023-05-22T15:27:55.680 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.671+0000 7fa549920640 1 -- start start 2023-05-22T15:27:55.680 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.671+0000 7fa549920640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fa5441563c0 0x7fa5440c7b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:55.680 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.671+0000 7fa549920640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fa5440cc6c0 0x7fa5440c8080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:55.680 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.671+0000 7fa549920640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa54415c210 con 0x7fa54406c220 2023-05-22T15:27:55.681 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.671+0000 7fa549920640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa544160700 con 0x7fa5441563c0 2023-05-22T15:27:55.681 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.671+0000 7fa549920640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa54406cdd0 con 0x7fa5440cc6c0 2023-05-22T15:27:55.681 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.671+0000 7fa542ffd640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fa5441563c0 0x7fa5440c7b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:55.681 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.671+0000 7fa542ffd640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fa5441563c0 0x7fa5440c7b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54326/0 (socket says 172.21.15.16:54326) 2023-05-22T15:27:55.681 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.671+0000 7fa542ffd640 1 -- 172.21.15.16:0/3602349548 learned_addr learned my addr 172.21.15.16:0/3602349548 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:55.681 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.671+0000 7fa542ffd640 1 -- 172.21.15.16:0/3602349548 >> v2:172.21.15.62:3300/0 conn(0x7fa5440cc6c0 msgr2=0x7fa5440c8080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:55.681 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.671+0000 7fa542ffd640 1 --2- 172.21.15.16:0/3602349548 >> v2:172.21.15.62:3300/0 conn(0x7fa5440cc6c0 0x7fa5440c8080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:55.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.671+0000 7fa542ffd640 1 -- 172.21.15.16:0/3602349548 >> v1:172.21.15.62:6789/0 conn(0x7fa54406c220 legacy=0x7fa54406c6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:55.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.671+0000 7fa542ffd640 1 -- 172.21.15.16:0/3602349548 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa54415c480 con 0x7fa5441563c0 2023-05-22T15:27:55.683 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.675+0000 7fa542ffd640 1 --2- 172.21.15.16:0/3602349548 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa5441563c0 0x7fa5440c7b40 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7fa544070ff0 tx=0x7fa53401c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec80b9303bfcdc9a server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:55.683 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.675+0000 7fa5427fc640 1 -- 172.21.15.16:0/3602349548 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5340aa5f0 con 0x7fa5441563c0 2023-05-22T15:27:55.683 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.675+0000 7fa5427fc640 1 -- 172.21.15.16:0/3602349548 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5340aa750 con 0x7fa5441563c0 2023-05-22T15:27:55.683 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.675+0000 7fa5427fc640 1 -- 172.21.15.16:0/3602349548 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5340aaae0 con 0x7fa5441563c0 2023-05-22T15:27:55.683 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.675+0000 7fa549920640 1 -- 172.21.15.16:0/3602349548 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa5441563c0 msgr2=0x7fa5440c7b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:55.683 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.675+0000 7fa549920640 1 --2- 172.21.15.16:0/3602349548 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa5441563c0 0x7fa5440c7b40 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7fa544070ff0 tx=0x7fa53401c490 comp rx=0 tx=0).stop 2023-05-22T15:27:55.684 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.675+0000 7fa549920640 1 -- 172.21.15.16:0/3602349548 shutdown_connections 2023-05-22T15:27:55.684 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.675+0000 7fa549920640 1 --2- 172.21.15.16:0/3602349548 >> v2:172.21.15.62:3300/0 conn(0x7fa5440cc6c0 0x7fa5440c8080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:55.684 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.675+0000 7fa549920640 1 --2- 172.21.15.16:0/3602349548 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa5441563c0 0x7fa5440c7b40 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:55.684 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.675+0000 7fa549920640 1 -- 172.21.15.16:0/3602349548 >> 172.21.15.16:0/3602349548 conn(0x7fa5440b1110 msgr2=0x7fa5440ce830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:55.684 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.675+0000 7fa549920640 1 -- 172.21.15.16:0/3602349548 shutdown_connections 2023-05-22T15:27:55.684 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.675+0000 7fa549920640 1 -- 172.21.15.16:0/3602349548 wait complete. 2023-05-22T15:27:55.686 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.675+0000 7fa549920640 1 Processor -- start 2023-05-22T15:27:55.686 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.675+0000 7fa549920640 1 -- start start 2023-05-22T15:27:55.686 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.675+0000 7fa549920640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa54406c220 0x7fa544146c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:55.686 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.675+0000 7fa549920640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa5440cc6c0 0x7fa544147490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:55.687 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.679+0000 7fa549920640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa5441479d0 0x7fa544147e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:55.687 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.679+0000 7fa549920640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa544150b70 con 0x7fa5441479d0 2023-05-22T15:27:55.687 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.679+0000 7fa549920640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa544150e50 con 0x7fa5440cc6c0 2023-05-22T15:27:55.687 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.679+0000 7fa549920640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa544151130 con 0x7fa54406c220 2023-05-22T15:27:55.688 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.679+0000 7fa5437fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa5441479d0 0x7fa544147e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:55.688 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.679+0000 7fa5437fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa5441479d0 0x7fa544147e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54328/0 (socket says 172.21.15.16:54328) 2023-05-22T15:27:55.688 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.679+0000 7fa5437fe640 1 -- 172.21.15.16:0/798837210 learned_addr learned my addr 172.21.15.16:0/798837210 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:55.688 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.679+0000 7fa542ffd640 1 --2- 172.21.15.16:0/798837210 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa54406c220 0x7fa544146c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:55.688 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.679+0000 7fa53bfff640 1 --2- 172.21.15.16:0/798837210 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa5440cc6c0 0x7fa544147490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:55.688 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.679+0000 7fa5437fe640 1 -- 172.21.15.16:0/798837210 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa54406c220 msgr2=0x7fa544146c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:55.688 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.679+0000 7fa5437fe640 1 --2- 172.21.15.16:0/798837210 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa54406c220 0x7fa544146c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:55.689 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.679+0000 7fa5437fe640 1 -- 172.21.15.16:0/798837210 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa5440cc6c0 msgr2=0x7fa544147490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:55.689 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.679+0000 7fa5437fe640 1 --2- 172.21.15.16:0/798837210 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa5440cc6c0 0x7fa544147490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:55.689 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.679+0000 7fa5437fe640 1 -- 172.21.15.16:0/798837210 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5340094d0 con 0x7fa5441479d0 2023-05-22T15:27:55.689 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.679+0000 7fa5437fe640 1 --2- 172.21.15.16:0/798837210 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa5441479d0 0x7fa544147e50 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fa53c00c780 tx=0x7fa53c00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:55.689 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.679+0000 7fa540ff9640 1 -- 172.21.15.16:0/798837210 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa53c003d90 con 0x7fa5441479d0 2023-05-22T15:27:55.689 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.679+0000 7fa540ff9640 1 -- 172.21.15.16:0/798837210 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa53c00ce90 con 0x7fa5441479d0 2023-05-22T15:27:55.689 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.679+0000 7fa549920640 1 -- 172.21.15.16:0/798837210 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa54415ff00 con 0x7fa5441479d0 2023-05-22T15:27:55.689 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.679+0000 7fa540ff9640 1 -- 172.21.15.16:0/798837210 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa53c0052b0 con 0x7fa5441479d0 2023-05-22T15:27:55.690 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.679+0000 7fa549920640 1 -- 172.21.15.16:0/798837210 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa54415cd20 con 0x7fa5441479d0 2023-05-22T15:27:55.691 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.679+0000 7fa549920640 1 -- 172.21.15.16:0/798837210 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa544160090 con 0x7fa5441479d0 2023-05-22T15:27:55.691 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.679+0000 7fa540ff9640 1 -- 172.21.15.16:0/798837210 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7fa53c005410 con 0x7fa5441479d0 2023-05-22T15:27:55.692 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.679+0000 7fa540ff9640 1 --2- 172.21.15.16:0/798837210 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fa5240653b0 0x7fa524067860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:55.692 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.679+0000 7fa540ff9640 1 -- 172.21.15.16:0/798837210 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6871+0+0 (secure 0 0 0) 0x7fa53c0824c0 con 0x7fa5441479d0 2023-05-22T15:27:55.692 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.679+0000 7fa542ffd640 1 --2- 172.21.15.16:0/798837210 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fa5240653b0 0x7fa524067860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:55.692 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.679+0000 7fa542ffd640 1 --2- 172.21.15.16:0/798837210 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fa5240653b0 0x7fa524067860 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fa5340ae000 tx=0x7fa534008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:55.692 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.683+0000 7fa540ff9640 1 -- 172.21.15.16:0/798837210 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fa53c04fd70 con 0x7fa5441479d0 2023-05-22T15:27:55.842 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.835+0000 7fa549920640 1 -- 172.21.15.16:0/798837210 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa54404dbe0 con 0x7fa5441479d0 2023-05-22T15:27:55.844 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.835+0000 7fa540ff9640 1 -- 172.21.15.16:0/798837210 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v29) v1 ==== 74+0+19700 (secure 0 0 0) 0x7fa53c053390 con 0x7fa5441479d0 2023-05-22T15:27:55.845 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:27:55.845 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":29,"fsid":"b2719834-7847-4d98-90a6-f65d57f85c49","created":"2023-05-22T15:26:25.259314+0000","modified":"2023-05-22T15:27:31.182317+0000","last_up_change":"2023-05-22T15:26:36.448524+0000","last_in_change":"2023-05-22T15:26:27.675668+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-05-22T15:26:38.019781+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-05-22T15:26:43.570514+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-05-22T15:26:47.879159+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-05-22T15:26:48.522130+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data_ec","create_time":"2023-05-22T15:26:51.616142+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"9e7ff854-b25f-4cbb-a5c2-1799d9c53d2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":37021},{"type":"v1","addr":"172.21.15.16:6825","nonce":37021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":37021},{"type":"v1","addr":"172.21.15.16:6827","nonce":37021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":37021},{"type":"v1","addr":"172.21.15.16:6831","nonce":37021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":37021},{"type":"v1","addr":"172.21.15.16:6829","nonce":37021}]},"public_addr":"172.21.15.16:6825/37021","cluster_addr":"172.21.15.16:6827/37021","heartbeat_back_addr":"172.21.15.16:6831/37021","heartbeat_front_addr":"172.21.15.16:6829/37021","state":["exists","up"]},{"osd":1,"uuid":"4702fbc9-a927-4565-aca5-41dc69cc8264","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":37008},{"type":"v1","addr":"172.21.15.16:6801","nonce":37008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":37008},{"type":"v1","addr":"172.21.15.16:6803","nonce":37008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":37008},{"type":"v1","addr":"172.21.15.16:6807","nonce":37008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":37008},{"type":"v1","addr":"172.21.15.16:6805","nonce":37008}]},"public_addr":"172.21.15.16:6801/37008","cluster_addr":"172.21.15.16:6803/37008","heartbeat_back_addr":"172.21.15.16:6807/37008","heartbeat_front_addr":"172.21.15.16:6805/37008","state":["exists","up"]},{"osd":2,"uuid":"1e7bba46-4eba-47ef-bb81-5e2d371638ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":37009},{"type":"v1","addr":"172.21.15.16:6809","nonce":37009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":37009},{"type":"v1","addr":"172.21.15.16:6811","nonce":37009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":37009},{"type":"v1","addr":"172.21.15.16:6815","nonce":37009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":37009},{"type":"v1","addr":"172.21.15.16:6813","nonce":37009}]},"public_addr":"172.21.15.16:6809/37009","cluster_addr":"172.21.15.16:6811/37009","heartbeat_back_addr":"172.21.15.16:6815/37009","heartbeat_front_addr":"172.21.15.16:6813/37009","state":["exists","up"]},{"osd":3,"uuid":"383cacc0-5a88-4904-990e-498ca59ef48a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":37020},{"type":"v1","addr":"172.21.15.16:6817","nonce":37020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":37020},{"type":"v1","addr":"172.21.15.16:6819","nonce":37020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":37020},{"type":"v1","addr":"172.21.15.16:6823","nonce":37020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":37020},{"type":"v1","addr":"172.21.15.16:6821","nonce":37020}]},"public_addr":"172.21.15.16:6817/37020","cluster_addr":"172.21.15.16:6819/37020","heartbeat_back_addr":"172.21.15.16:6823/37020","heartbeat_front_addr":"172.21.15.16:6821/37020","state":["exists","up"]},{"osd":4,"uuid":"970500ef-95b7-42c1-b080-eed4fae7013d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":37353},{"type":"v1","addr":"172.21.15.62:6817","nonce":37353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6819","nonce":37353},{"type":"v1","addr":"172.21.15.62:6822","nonce":37353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37353},{"type":"v1","addr":"172.21.15.62:6831","nonce":37353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37353},{"type":"v1","addr":"172.21.15.62:6826","nonce":37353}]},"public_addr":"172.21.15.62:6817/37353","cluster_addr":"172.21.15.62:6822/37353","heartbeat_back_addr":"172.21.15.62:6831/37353","heartbeat_front_addr":"172.21.15.62:6826/37353","state":["exists","up"]},{"osd":5,"uuid":"022d67db-d4d8-445b-b1b9-8b446a00f5c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":37352},{"type":"v1","addr":"172.21.15.62:6801","nonce":37352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":37352},{"type":"v1","addr":"172.21.15.62:6803","nonce":37352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":37352},{"type":"v1","addr":"172.21.15.62:6807","nonce":37352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":37352},{"type":"v1","addr":"172.21.15.62:6805","nonce":37352}]},"public_addr":"172.21.15.62:6801/37352","cluster_addr":"172.21.15.62:6803/37352","heartbeat_back_addr":"172.21.15.62:6807/37352","heartbeat_front_addr":"172.21.15.62:6805/37352","state":["exists","up"]},{"osd":6,"uuid":"42bd372b-6688-4a1b-8407-e16e8c266e33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":37354},{"type":"v1","addr":"172.21.15.62:6820","nonce":37354}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":37354},{"type":"v1","addr":"172.21.15.62:6823","nonce":37354}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6829","nonce":37354},{"type":"v1","addr":"172.21.15.62:6830","nonce":37354}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6825","nonce":37354},{"type":"v1","addr":"172.21.15.62:6827","nonce":37354}]},"public_addr":"172.21.15.62:6820/37354","cluster_addr":"172.21.15.62:6823/37354","heartbeat_back_addr":"172.21.15.62:6830/37354","heartbeat_front_addr":"172.21.15.62:6827/37354","state":["exists","up"]},{"osd":7,"uuid":"ac70b2a3-f41b-4bf2-b227-7f2cbb55426d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37355},{"type":"v1","addr":"172.21.15.62:6809","nonce":37355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37355},{"type":"v1","addr":"172.21.15.62:6811","nonce":37355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":37355},{"type":"v1","addr":"172.21.15.62:6815","nonce":37355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":37355},{"type":"v1","addr":"172.21.15.62:6813","nonce":37355}]},"public_addr":"172.21.15.62:6809/37355","cluster_addr":"172.21.15.62:6811/37355","heartbeat_back_addr":"172.21.15.62:6815/37355","heartbeat_front_addr":"172.21.15.62:6813/37355","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.368896+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.107273+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.276888+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.287502+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.124791+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.251937+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.255679+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.232797+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"4.23","mappings":[{"from":3,"to":2}]},{"pgid":"4.31","mappings":[{"from":1,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-05-22T15:27:55.846 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.839+0000 7fa549920640 1 -- 172.21.15.16:0/798837210 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fa5240653b0 msgr2=0x7fa524067860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:55.847 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.839+0000 7fa549920640 1 --2- 172.21.15.16:0/798837210 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fa5240653b0 0x7fa524067860 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fa5340ae000 tx=0x7fa534008040 comp rx=0 tx=0).stop 2023-05-22T15:27:55.847 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.839+0000 7fa549920640 1 -- 172.21.15.16:0/798837210 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa5441479d0 msgr2=0x7fa544147e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:55.847 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.839+0000 7fa549920640 1 --2- 172.21.15.16:0/798837210 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa5441479d0 0x7fa544147e50 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fa53c00c780 tx=0x7fa53c00cc40 comp rx=0 tx=0).stop 2023-05-22T15:27:55.847 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.839+0000 7fa549920640 1 -- 172.21.15.16:0/798837210 shutdown_connections 2023-05-22T15:27:55.847 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.839+0000 7fa549920640 1 --2- 172.21.15.16:0/798837210 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa5441479d0 0x7fa544147e50 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:55.847 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.839+0000 7fa549920640 1 --2- 172.21.15.16:0/798837210 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa5440cc6c0 0x7fa544147490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:55.848 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.839+0000 7fa549920640 1 --2- 172.21.15.16:0/798837210 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fa5240653b0 0x7fa524067860 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:55.848 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.839+0000 7fa549920640 1 --2- 172.21.15.16:0/798837210 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa54406c220 0x7fa544146c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:55.848 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.839+0000 7fa549920640 1 -- 172.21.15.16:0/798837210 >> 172.21.15.16:0/798837210 conn(0x7fa5440b1110 msgr2=0x7fa5441521f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:55.848 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.839+0000 7fa549920640 1 -- 172.21.15.16:0/798837210 shutdown_connections 2023-05-22T15:27:55.848 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.839+0000 7fa549920640 1 -- 172.21.15.16:0/798837210 wait complete. 2023-05-22T15:27:55.863 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-05-22T15:26:38.019781+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-05-22T15:26:43.570514+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2023-05-22T15:26:47.879159+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2023-05-22T15:26:48.522130+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-05-22T15:26:51.616142+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '27', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-05-22T15:27:55.864 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-05-22T15:27:55.993 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.987+0000 7f8660f95640 1 Processor -- start 2023-05-22T15:27:55.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.987+0000 7f8660f95640 1 -- start start 2023-05-22T15:27:55.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.987+0000 7f8660f95640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f865c15c8b0 0x7f865c0cee30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:55.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.987+0000 7f8660f95640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f865c06db80 0x7f865c06dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:55.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.987+0000 7f8660f95640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f865c06e520 con 0x7f865c06bf60 2023-05-22T15:27:55.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.987+0000 7f8660f95640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f865c15a0c0 con 0x7f865c06db80 2023-05-22T15:27:55.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.987+0000 7f8660f95640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f865c0ce160 con 0x7f865c15c8b0 2023-05-22T15:27:55.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.987+0000 7f865ad9d640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f865c06db80 0x7f865c06dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:55.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.987+0000 7f865ad9d640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f865c06db80 0x7f865c06dfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54330/0 (socket says 172.21.15.16:54330) 2023-05-22T15:27:55.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.987+0000 7f865ad9d640 1 -- 172.21.15.16:0/2993132039 learned_addr learned my addr 172.21.15.16:0/2993132039 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:55.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.987+0000 7f865a59c640 1 --2- 172.21.15.16:0/2993132039 >> v2:172.21.15.62:3300/0 conn(0x7f865c15c8b0 0x7f865c0cee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:55.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.987+0000 7f865ad9d640 1 -- 172.21.15.16:0/2993132039 >> v2:172.21.15.62:3300/0 conn(0x7f865c15c8b0 msgr2=0x7f865c0cee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:55.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.987+0000 7f865ad9d640 1 --2- 172.21.15.16:0/2993132039 >> v2:172.21.15.62:3300/0 conn(0x7f865c15c8b0 0x7f865c0cee30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:55.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.987+0000 7f865ad9d640 1 -- 172.21.15.16:0/2993132039 >> v1:172.21.15.62:6789/0 conn(0x7f865c06bf60 legacy=0x7f865c15cc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:55.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.987+0000 7f865ad9d640 1 -- 172.21.15.16:0/2993132039 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f865c06c3e0 con 0x7f865c06db80 2023-05-22T15:27:55.998 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.987+0000 7f865a59c640 1 --2- 172.21.15.16:0/2993132039 >> v2:172.21.15.62:3300/0 conn(0x7f865c15c8b0 0x7f865c0cee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T15:27:55.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.991+0000 7f865ad9d640 1 --2- 172.21.15.16:0/2993132039 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f865c06db80 0x7f865c06dfe0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f8654009e50 tx=0x7f865401c9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82b1f5c9d8397a59 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:55.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.991+0000 7f865959a640 1 -- 172.21.15.16:0/2993132039 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86540a5860 con 0x7f865c06db80 2023-05-22T15:27:55.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.991+0000 7f865959a640 1 -- 172.21.15.16:0/2993132039 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86540a59c0 con 0x7f865c06db80 2023-05-22T15:27:55.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.991+0000 7f865959a640 1 -- 172.21.15.16:0/2993132039 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86540a5d50 con 0x7f865c06db80 2023-05-22T15:27:56.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.991+0000 7f8660f95640 1 -- 172.21.15.16:0/2993132039 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f865c06db80 msgr2=0x7f865c06dfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:56.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.991+0000 7f8660f95640 1 --2- 172.21.15.16:0/2993132039 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f865c06db80 0x7f865c06dfe0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f8654009e50 tx=0x7f865401c9a0 comp rx=0 tx=0).stop 2023-05-22T15:27:56.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.991+0000 7f8660f95640 1 -- 172.21.15.16:0/2993132039 shutdown_connections 2023-05-22T15:27:56.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.991+0000 7f8660f95640 1 --2- 172.21.15.16:0/2993132039 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f865c06db80 0x7f865c06dfe0 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:56.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.991+0000 7f8660f95640 1 --2- 172.21.15.16:0/2993132039 >> v2:172.21.15.62:3300/0 conn(0x7f865c15c8b0 0x7f865c0cee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:56.001 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.991+0000 7f8660f95640 1 -- 172.21.15.16:0/2993132039 >> 172.21.15.16:0/2993132039 conn(0x7f865c06f5a0 msgr2=0x7f865c06f9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:56.001 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.991+0000 7f8660f95640 1 -- 172.21.15.16:0/2993132039 shutdown_connections 2023-05-22T15:27:56.001 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.991+0000 7f8660f95640 1 -- 172.21.15.16:0/2993132039 wait complete. 2023-05-22T15:27:56.006 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.991+0000 7f8660f95640 1 Processor -- start 2023-05-22T15:27:56.006 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.991+0000 7f8660f95640 1 -- start start 2023-05-22T15:27:56.006 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.991+0000 7f8660f95640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f865c06bf60 0x7f865c14a240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:56.006 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.991+0000 7f8660f95640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f865c06db80 0x7f865c1575e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:56.007 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.991+0000 7f8660f95640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f865c15c8b0 0x7f865c157b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:56.007 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.991+0000 7f8660f95640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f865c0cb4b0 con 0x7f865c15c8b0 2023-05-22T15:27:56.007 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.991+0000 7f8660f95640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f865c0cb790 con 0x7f865c06db80 2023-05-22T15:27:56.007 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.991+0000 7f8660f95640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f865c0cba70 con 0x7f865c06bf60 2023-05-22T15:27:56.007 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.991+0000 7f865ad9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f865c15c8b0 0x7f865c157b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:56.007 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.991+0000 7f865ad9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f865c15c8b0 0x7f865c157b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54336/0 (socket says 172.21.15.16:54336) 2023-05-22T15:27:56.007 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.991+0000 7f865ad9d640 1 -- 172.21.15.16:0/1360662598 learned_addr learned my addr 172.21.15.16:0/1360662598 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:56.007 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.991+0000 7f865a59c640 1 --2- 172.21.15.16:0/1360662598 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f865c06bf60 0x7f865c14a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:56.008 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.991+0000 7f8659d9b640 1 --2- 172.21.15.16:0/1360662598 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f865c06db80 0x7f865c1575e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:56.008 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.995+0000 7f865ad9d640 1 -- 172.21.15.16:0/1360662598 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f865c06bf60 msgr2=0x7f865c14a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:56.008 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.995+0000 7f865ad9d640 1 --2- 172.21.15.16:0/1360662598 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f865c06bf60 0x7f865c14a240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:56.008 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.995+0000 7f865ad9d640 1 -- 172.21.15.16:0/1360662598 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f865c06db80 msgr2=0x7f865c1575e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:56.008 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.995+0000 7f865ad9d640 1 --2- 172.21.15.16:0/1360662598 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f865c06db80 0x7f865c1575e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:56.008 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.995+0000 7f865ad9d640 1 -- 172.21.15.16:0/1360662598 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8654009b30 con 0x7f865c15c8b0 2023-05-22T15:27:56.008 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.995+0000 7f8659d9b640 1 --2- 172.21.15.16:0/1360662598 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f865c06db80 0x7f865c1575e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T15:27:56.009 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.995+0000 7f865ad9d640 1 --2- 172.21.15.16:0/1360662598 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f865c15c8b0 0x7f865c157b20 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f86540af040 tx=0x7f8654019740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:56.009 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.995+0000 7f864affd640 1 -- 172.21.15.16:0/1360662598 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86540b8670 con 0x7f865c15c8b0 2023-05-22T15:27:56.009 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.995+0000 7f864affd640 1 -- 172.21.15.16:0/1360662598 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86540b8810 con 0x7f865c15c8b0 2023-05-22T15:27:56.009 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.995+0000 7f8660f95640 1 -- 172.21.15.16:0/1360662598 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f865c0ce930 con 0x7f865c15c8b0 2023-05-22T15:27:56.009 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.995+0000 7f864affd640 1 -- 172.21.15.16:0/1360662598 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86540b8bc0 con 0x7f865c15c8b0 2023-05-22T15:27:56.009 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.995+0000 7f8660f95640 1 -- 172.21.15.16:0/1360662598 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f865c0ce650 con 0x7f865c15c8b0 2023-05-22T15:27:56.009 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.995+0000 7f8660f95640 1 -- 172.21.15.16:0/1360662598 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f865c15d390 con 0x7f865c15c8b0 2023-05-22T15:27:56.009 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.995+0000 7f864affd640 1 -- 172.21.15.16:0/1360662598 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7f86540b8d20 con 0x7f865c15c8b0 2023-05-22T15:27:56.010 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.995+0000 7f864affd640 1 --2- 172.21.15.16:0/1360662598 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f863c065380 0x7f863c067830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:56.010 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.995+0000 7f864affd640 1 -- 172.21.15.16:0/1360662598 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f865411e190 con 0x7f865c15c8b0 2023-05-22T15:27:56.010 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.995+0000 7f865a59c640 1 --2- 172.21.15.16:0/1360662598 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f863c065380 0x7f863c067830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:56.010 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.995+0000 7f865a59c640 1 --2- 172.21.15.16:0/1360662598 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f863c065380 0x7f863c067830 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f865c150ec0 tx=0x7f864c008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:56.010 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:55.999+0000 7f864affd640 1 -- 172.21.15.16:0/1360662598 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f86540eba80 con 0x7f865c15c8b0 2023-05-22T15:27:56.163 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.155+0000 7f8660f95640 1 -- 172.21.15.16:0/1360662598 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f865c04dbe0 con 0x7f865c15c8b0 2023-05-22T15:27:56.163 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.155+0000 7f864affd640 1 -- 172.21.15.16:0/1360662598 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v29) v1 ==== 93+0+10 (secure 0 0 0) 0x7f86540ef0a0 con 0x7f865c15c8b0 2023-05-22T15:27:56.164 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2023-05-22T15:27:56.166 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.159+0000 7f8660f95640 1 -- 172.21.15.16:0/1360662598 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f863c065380 msgr2=0x7f863c067830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:56.166 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.159+0000 7f8660f95640 1 --2- 172.21.15.16:0/1360662598 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f863c065380 0x7f863c067830 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f865c150ec0 tx=0x7f864c008040 comp rx=0 tx=0).stop 2023-05-22T15:27:56.166 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.159+0000 7f8660f95640 1 -- 172.21.15.16:0/1360662598 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f865c15c8b0 msgr2=0x7f865c157b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:56.166 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.159+0000 7f8660f95640 1 --2- 172.21.15.16:0/1360662598 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f865c15c8b0 0x7f865c157b20 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f86540af040 tx=0x7f8654019740 comp rx=0 tx=0).stop 2023-05-22T15:27:56.166 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.159+0000 7f8660f95640 1 -- 172.21.15.16:0/1360662598 shutdown_connections 2023-05-22T15:27:56.166 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.159+0000 7f8660f95640 1 --2- 172.21.15.16:0/1360662598 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f865c15c8b0 0x7f865c157b20 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:56.166 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.159+0000 7f8660f95640 1 --2- 172.21.15.16:0/1360662598 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f865c06db80 0x7f865c1575e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:56.167 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.159+0000 7f8660f95640 1 --2- 172.21.15.16:0/1360662598 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f863c065380 0x7f863c067830 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:56.168 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.159+0000 7f8660f95640 1 --2- 172.21.15.16:0/1360662598 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f865c06bf60 0x7f865c14a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:56.168 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.159+0000 7f8660f95640 1 -- 172.21.15.16:0/1360662598 >> 172.21.15.16:0/1360662598 conn(0x7f865c06f5a0 msgr2=0x7f865c153370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:56.169 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.159+0000 7f8660f95640 1 -- 172.21.15.16:0/1360662598 shutdown_connections 2023-05-22T15:27:56.169 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.159+0000 7f8660f95640 1 -- 172.21.15.16:0/1360662598 wait complete. 2023-05-22T15:27:56.183 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-05-22T15:27:56.313 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.307+0000 7f00306b8640 1 Processor -- start 2023-05-22T15:27:56.313 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.307+0000 7f00306b8640 1 -- start start 2023-05-22T15:27:56.314 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.307+0000 7f00306b8640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f002815bca0 0x7f00280cda80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:56.315 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.307+0000 7f00306b8640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f00280b5a50 0x7f00280d2900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:56.315 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.307+0000 7f00306b8640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00280cccc0 con 0x7f00280d2480 2023-05-22T15:27:56.316 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.307+0000 7f00306b8640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00280cd660 con 0x7f00280b5a50 2023-05-22T15:27:56.316 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.307+0000 7f00306b8640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00280d2e40 con 0x7f002815bca0 2023-05-22T15:27:56.316 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.307+0000 7f002ec55640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f00280b5a50 0x7f00280d2900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:56.317 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.307+0000 7f002ec55640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f00280b5a50 0x7f00280d2900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54342/0 (socket says 172.21.15.16:54342) 2023-05-22T15:27:56.317 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.307+0000 7f002ec55640 1 -- 172.21.15.16:0/4273746262 learned_addr learned my addr 172.21.15.16:0/4273746262 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:56.317 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.307+0000 7f002e454640 1 --2- 172.21.15.16:0/4273746262 >> v2:172.21.15.62:3300/0 conn(0x7f002815bca0 0x7f00280cda80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:56.318 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.307+0000 7f002ec55640 1 -- 172.21.15.16:0/4273746262 >> v2:172.21.15.62:3300/0 conn(0x7f002815bca0 msgr2=0x7f00280cda80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:56.318 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.307+0000 7f002ec55640 1 --2- 172.21.15.16:0/4273746262 >> v2:172.21.15.62:3300/0 conn(0x7f002815bca0 0x7f00280cda80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:56.318 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.307+0000 7f002ec55640 1 -- 172.21.15.16:0/4273746262 >> v1:172.21.15.62:6789/0 conn(0x7f00280d2480 legacy=0x7f00280b5340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:56.318 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.307+0000 7f002ec55640 1 -- 172.21.15.16:0/4273746262 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00280ccfa0 con 0x7f00280b5a50 2023-05-22T15:27:56.322 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.311+0000 7f002ec55640 1 --2- 172.21.15.16:0/4273746262 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f00280b5a50 0x7f00280d2900 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f00280b40d0 tx=0x7f002001c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bbf7198ec7845e55 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:56.323 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.311+0000 7f002d452640 1 -- 172.21.15.16:0/4273746262 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00200aa680 con 0x7f00280b5a50 2023-05-22T15:27:56.323 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.311+0000 7f002d452640 1 -- 172.21.15.16:0/4273746262 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00200aa7e0 con 0x7f00280b5a50 2023-05-22T15:27:56.323 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.311+0000 7f002d452640 1 -- 172.21.15.16:0/4273746262 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00200aab70 con 0x7f00280b5a50 2023-05-22T15:27:56.324 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.311+0000 7f00306b8640 1 -- 172.21.15.16:0/4273746262 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f00280b5a50 msgr2=0x7f00280d2900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:56.324 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.311+0000 7f00306b8640 1 --2- 172.21.15.16:0/4273746262 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f00280b5a50 0x7f00280d2900 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f00280b40d0 tx=0x7f002001c500 comp rx=0 tx=0).stop 2023-05-22T15:27:56.325 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.311+0000 7f00306b8640 1 -- 172.21.15.16:0/4273746262 shutdown_connections 2023-05-22T15:27:56.325 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.311+0000 7f00306b8640 1 --2- 172.21.15.16:0/4273746262 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f00280b5a50 0x7f00280d2900 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:56.325 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.311+0000 7f00306b8640 1 --2- 172.21.15.16:0/4273746262 >> v2:172.21.15.62:3300/0 conn(0x7f002815bca0 0x7f00280cda80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:56.325 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.311+0000 7f00306b8640 1 -- 172.21.15.16:0/4273746262 >> 172.21.15.16:0/4273746262 conn(0x7f00280655c0 msgr2=0x7f0028070e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:56.326 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.311+0000 7f00306b8640 1 -- 172.21.15.16:0/4273746262 shutdown_connections 2023-05-22T15:27:56.326 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.311+0000 7f00306b8640 1 -- 172.21.15.16:0/4273746262 wait complete. 2023-05-22T15:27:56.326 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.311+0000 7f00306b8640 1 Processor -- start 2023-05-22T15:27:56.326 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.311+0000 7f00306b8640 1 -- start start 2023-05-22T15:27:56.327 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.311+0000 7f00306b8640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f00280b5a50 0x7f00280cc4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:56.327 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.311+0000 7f00306b8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f00280d2480 0x7f0028156740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:56.327 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.311+0000 7f00306b8640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f002815bca0 0x7f00281588e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:56.327 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.311+0000 7f00306b8640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f002815a090 con 0x7f00280d2480 2023-05-22T15:27:56.327 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.311+0000 7f00306b8640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00280cca10 con 0x7f00280b5a50 2023-05-22T15:27:56.328 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.311+0000 7f00306b8640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00281537d0 con 0x7f002815bca0 2023-05-22T15:27:56.328 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.311+0000 7f002dc53640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f00280d2480 0x7f0028156740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:56.329 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.311+0000 7f002dc53640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f00280d2480 0x7f0028156740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54344/0 (socket says 172.21.15.16:54344) 2023-05-22T15:27:56.329 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.311+0000 7f002dc53640 1 -- 172.21.15.16:0/4271541764 learned_addr learned my addr 172.21.15.16:0/4271541764 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:56.329 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.315+0000 7f002e454640 1 --2- 172.21.15.16:0/4271541764 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f00280b5a50 0x7f00280cc4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:56.329 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.315+0000 7f002ec55640 1 --2- 172.21.15.16:0/4271541764 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f002815bca0 0x7f00281588e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:56.330 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.315+0000 7f002dc53640 1 -- 172.21.15.16:0/4271541764 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f002815bca0 msgr2=0x7f00281588e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:56.331 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.315+0000 7f002dc53640 1 --2- 172.21.15.16:0/4271541764 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f002815bca0 0x7f00281588e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:56.331 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.315+0000 7f002dc53640 1 -- 172.21.15.16:0/4271541764 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f00280b5a50 msgr2=0x7f00280cc4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:56.332 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.315+0000 7f002dc53640 1 --2- 172.21.15.16:0/4271541764 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f00280b5a50 0x7f00280cc4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:56.332 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.315+0000 7f002dc53640 1 -- 172.21.15.16:0/4271541764 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00200094d0 con 0x7f00280d2480 2023-05-22T15:27:56.332 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.315+0000 7f002dc53640 1 --2- 172.21.15.16:0/4271541764 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f00280d2480 0x7f0028156740 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f001c00c7f0 tx=0x7f001c00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:56.332 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.315+0000 7f0026ffd640 1 -- 172.21.15.16:0/4271541764 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f001c003b80 con 0x7f00280d2480 2023-05-22T15:27:56.333 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.315+0000 7f0026ffd640 1 -- 172.21.15.16:0/4271541764 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f001c003ce0 con 0x7f00280d2480 2023-05-22T15:27:56.333 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.315+0000 7f0026ffd640 1 -- 172.21.15.16:0/4271541764 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f001c00aa90 con 0x7f00280d2480 2023-05-22T15:27:56.333 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.315+0000 7f00306b8640 1 -- 172.21.15.16:0/4271541764 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0028160360 con 0x7f00280d2480 2023-05-22T15:27:56.333 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.315+0000 7f00306b8640 1 -- 172.21.15.16:0/4271541764 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00280cdfc0 con 0x7f00280d2480 2023-05-22T15:27:56.333 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.315+0000 7f00306b8640 1 -- 172.21.15.16:0/4271541764 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00281604f0 con 0x7f00280d2480 2023-05-22T15:27:56.335 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.315+0000 7f0026ffd640 1 -- 172.21.15.16:0/4271541764 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7f001c00abf0 con 0x7f00280d2480 2023-05-22T15:27:56.335 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.315+0000 7f0026ffd640 1 --2- 172.21.15.16:0/4271541764 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f0014065380 0x7f0014067830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:56.336 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.315+0000 7f0026ffd640 1 -- 172.21.15.16:0/4271541764 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f001c082a90 con 0x7f00280d2480 2023-05-22T15:27:56.336 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.315+0000 7f002e454640 1 --2- 172.21.15.16:0/4271541764 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f0014065380 0x7f0014067830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:56.336 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.315+0000 7f002e454640 1 --2- 172.21.15.16:0/4271541764 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f0014065380 0x7f0014067830 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f0018009af0 tx=0x7f0018008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:56.336 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.319+0000 7f0026ffd640 1 -- 172.21.15.16:0/4271541764 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f001c0503e0 con 0x7f00280d2480 2023-05-22T15:27:56.479 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.471+0000 7f00306b8640 1 -- 172.21.15.16:0/4271541764 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f002815a240 con 0x7f00280d2480 2023-05-22T15:27:56.480 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.471+0000 7f0026ffd640 1 -- 172.21.15.16:0/4271541764 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v29) v1 ==== 92+0+10 (secure 0 0 0) 0x7f002815a240 con 0x7f00280d2480 2023-05-22T15:27:56.480 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 8 2023-05-22T15:27:56.482 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.475+0000 7f00306b8640 1 -- 172.21.15.16:0/4271541764 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f0014065380 msgr2=0x7f0014067830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:56.482 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.475+0000 7f00306b8640 1 --2- 172.21.15.16:0/4271541764 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f0014065380 0x7f0014067830 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f0018009af0 tx=0x7f0018008040 comp rx=0 tx=0).stop 2023-05-22T15:27:56.482 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.475+0000 7f00306b8640 1 -- 172.21.15.16:0/4271541764 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f00280d2480 msgr2=0x7f0028156740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:56.482 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.475+0000 7f00306b8640 1 --2- 172.21.15.16:0/4271541764 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f00280d2480 0x7f0028156740 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f001c00c7f0 tx=0x7f001c00ccb0 comp rx=0 tx=0).stop 2023-05-22T15:27:56.482 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.475+0000 7f00306b8640 1 -- 172.21.15.16:0/4271541764 shutdown_connections 2023-05-22T15:27:56.483 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.475+0000 7f00306b8640 1 --2- 172.21.15.16:0/4271541764 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f0014065380 0x7f0014067830 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:56.483 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.475+0000 7f00306b8640 1 --2- 172.21.15.16:0/4271541764 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f002815bca0 0x7f00281588e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:56.483 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.475+0000 7f00306b8640 1 --2- 172.21.15.16:0/4271541764 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f00280d2480 0x7f0028156740 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:56.483 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.475+0000 7f00306b8640 1 --2- 172.21.15.16:0/4271541764 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f00280b5a50 0x7f00280cc4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:56.483 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.475+0000 7f00306b8640 1 -- 172.21.15.16:0/4271541764 >> 172.21.15.16:0/4271541764 conn(0x7f00280655c0 msgr2=0x7f00280b5320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:56.483 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.475+0000 7f00306b8640 1 -- 172.21.15.16:0/4271541764 shutdown_connections 2023-05-22T15:27:56.484 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.475+0000 7f00306b8640 1 -- 172.21.15.16:0/4271541764 wait complete. 2023-05-22T15:27:56.497 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-05-22T15:27:56.616 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.607+0000 7f9f21498640 1 Processor -- start 2023-05-22T15:27:56.616 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.607+0000 7f9f21498640 1 -- start start 2023-05-22T15:27:56.617 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.611+0000 7f9f21498640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f9f1c0c8250 0x7f9f1c06c220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:56.618 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.611+0000 7f9f21498640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f9f1c0cc6c0 0x7f9f1c06c760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:56.618 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.611+0000 7f9f21498640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f1c15c210 con 0x7f9f1c1563c0 2023-05-22T15:27:56.618 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.611+0000 7f9f21498640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f1c160700 con 0x7f9f1c0cc6c0 2023-05-22T15:27:56.619 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.611+0000 7f9f21498640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f1c15c480 con 0x7f9f1c0c8250 2023-05-22T15:27:56.619 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.611+0000 7f9f1b7fe640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f9f1c0cc6c0 0x7f9f1c06c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:56.619 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.611+0000 7f9f1b7fe640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f9f1c0cc6c0 0x7f9f1c06c760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54360/0 (socket says 172.21.15.16:54360) 2023-05-22T15:27:56.619 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.611+0000 7f9f1b7fe640 1 -- 172.21.15.16:0/3600031189 learned_addr learned my addr 172.21.15.16:0/3600031189 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:56.619 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.611+0000 7f9f1a7fc640 1 --2- 172.21.15.16:0/3600031189 >> v2:172.21.15.62:3300/0 conn(0x7f9f1c0c8250 0x7f9f1c06c220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:56.619 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.611+0000 7f9f1b7fe640 1 -- 172.21.15.16:0/3600031189 >> v2:172.21.15.62:3300/0 conn(0x7f9f1c0c8250 msgr2=0x7f9f1c06c220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:56.620 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.611+0000 7f9f1b7fe640 1 --2- 172.21.15.16:0/3600031189 >> v2:172.21.15.62:3300/0 conn(0x7f9f1c0c8250 0x7f9f1c06c220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:56.620 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.611+0000 7f9f1b7fe640 1 -- 172.21.15.16:0/3600031189 >> v1:172.21.15.62:6789/0 conn(0x7f9f1c1563c0 legacy=0x7f9f1c0c7b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:56.620 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.611+0000 7f9f1b7fe640 1 -- 172.21.15.16:0/3600031189 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f1c15c760 con 0x7f9f1c0cc6c0 2023-05-22T15:27:56.621 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.611+0000 7f9f1b7fe640 1 --2- 172.21.15.16:0/3600031189 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f1c0cc6c0 0x7f9f1c06c760 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f9f1c070ff0 tx=0x7f9f1401c9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c52eb3cce584f95c server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:56.622 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.611+0000 7f9f19ffb640 1 -- 172.21.15.16:0/3600031189 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f140a5860 con 0x7f9f1c0cc6c0 2023-05-22T15:27:56.622 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.611+0000 7f9f19ffb640 1 -- 172.21.15.16:0/3600031189 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f140a59c0 con 0x7f9f1c0cc6c0 2023-05-22T15:27:56.622 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.611+0000 7f9f19ffb640 1 -- 172.21.15.16:0/3600031189 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f140a5d50 con 0x7f9f1c0cc6c0 2023-05-22T15:27:56.622 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.611+0000 7f9f21498640 1 -- 172.21.15.16:0/3600031189 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f1c0cc6c0 msgr2=0x7f9f1c06c760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:56.622 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.611+0000 7f9f21498640 1 --2- 172.21.15.16:0/3600031189 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f1c0cc6c0 0x7f9f1c06c760 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f9f1c070ff0 tx=0x7f9f1401c9a0 comp rx=0 tx=0).stop 2023-05-22T15:27:56.623 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.611+0000 7f9f21498640 1 -- 172.21.15.16:0/3600031189 shutdown_connections 2023-05-22T15:27:56.623 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.611+0000 7f9f21498640 1 --2- 172.21.15.16:0/3600031189 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f1c0cc6c0 0x7f9f1c06c760 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:56.624 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.611+0000 7f9f21498640 1 --2- 172.21.15.16:0/3600031189 >> v2:172.21.15.62:3300/0 conn(0x7f9f1c0c8250 0x7f9f1c06c220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:56.624 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.611+0000 7f9f21498640 1 -- 172.21.15.16:0/3600031189 >> 172.21.15.16:0/3600031189 conn(0x7f9f1c0b1110 msgr2=0x7f9f1c0ce830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:56.624 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.611+0000 7f9f21498640 1 -- 172.21.15.16:0/3600031189 shutdown_connections 2023-05-22T15:27:56.624 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.611+0000 7f9f21498640 1 -- 172.21.15.16:0/3600031189 wait complete. 2023-05-22T15:27:56.624 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.615+0000 7f9f21498640 1 Processor -- start 2023-05-22T15:27:56.624 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.615+0000 7f9f21498640 1 -- start start 2023-05-22T15:27:56.627 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.615+0000 7f9f21498640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9f1c0c8250 0x7f9f1c149c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:56.627 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.615+0000 7f9f21498640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9f1c0cc6c0 0x7f9f1c0c9f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:56.627 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.615+0000 7f9f21498640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f1c1f4040 0x7f9f1c150060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:56.627 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.615+0000 7f9f21498640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f1c15d520 con 0x7f9f1c1f4040 2023-05-22T15:27:56.628 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.615+0000 7f9f21498640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f1c0ca4b0 con 0x7f9f1c0c8250 2023-05-22T15:27:56.628 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.615+0000 7f9f21498640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f1c1f4820 con 0x7f9f1c0cc6c0 2023-05-22T15:27:56.628 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.615+0000 7f9f1b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f1c1f4040 0x7f9f1c150060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:56.628 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.615+0000 7f9f1b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f1c1f4040 0x7f9f1c150060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54370/0 (socket says 172.21.15.16:54370) 2023-05-22T15:27:56.629 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.615+0000 7f9f1b7fe640 1 -- 172.21.15.16:0/354023453 learned_addr learned my addr 172.21.15.16:0/354023453 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:56.629 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.615+0000 7f9f1affd640 1 --2- 172.21.15.16:0/354023453 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9f1c0c8250 0x7f9f1c149c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:56.629 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.615+0000 7f9f1b7fe640 1 -- 172.21.15.16:0/354023453 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9f1c0cc6c0 msgr2=0x7f9f1c0c9f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:56.629 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.615+0000 7f9f1b7fe640 1 --2- 172.21.15.16:0/354023453 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9f1c0cc6c0 0x7f9f1c0c9f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:56.629 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.615+0000 7f9f1b7fe640 1 -- 172.21.15.16:0/354023453 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9f1c0c8250 msgr2=0x7f9f1c149c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:56.629 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.615+0000 7f9f1b7fe640 1 --2- 172.21.15.16:0/354023453 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9f1c0c8250 0x7f9f1c149c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:56.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.615+0000 7f9f1b7fe640 1 -- 172.21.15.16:0/354023453 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f14009b30 con 0x7f9f1c1f4040 2023-05-22T15:27:56.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.615+0000 7f9f1b7fe640 1 --2- 172.21.15.16:0/354023453 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f1c1f4040 0x7f9f1c150060 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f9f140195b0 tx=0x7f9f14019740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:56.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.615+0000 7f9f037fe640 1 -- 172.21.15.16:0/354023453 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f140b8670 con 0x7f9f1c1f4040 2023-05-22T15:27:56.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.615+0000 7f9f037fe640 1 -- 172.21.15.16:0/354023453 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f140b87d0 con 0x7f9f1c1f4040 2023-05-22T15:27:56.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.615+0000 7f9f21498640 1 -- 172.21.15.16:0/354023453 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f1c06ce30 con 0x7f9f1c1f4040 2023-05-22T15:27:56.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.615+0000 7f9f037fe640 1 -- 172.21.15.16:0/354023453 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f140b8b80 con 0x7f9f1c1f4040 2023-05-22T15:27:56.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.615+0000 7f9f21498640 1 -- 172.21.15.16:0/354023453 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f1c06cca0 con 0x7f9f1c1f4040 2023-05-22T15:27:56.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.615+0000 7f9f21498640 1 -- 172.21.15.16:0/354023453 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f1c15ff00 con 0x7f9f1c1f4040 2023-05-22T15:27:56.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.619+0000 7f9f037fe640 1 -- 172.21.15.16:0/354023453 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7f9f140b8ce0 con 0x7f9f1c1f4040 2023-05-22T15:27:56.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.619+0000 7f9f037fe640 1 --2- 172.21.15.16:0/354023453 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f9f08065380 0x7f9f08067830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:56.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.619+0000 7f9f037fe640 1 -- 172.21.15.16:0/354023453 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f9f1411e240 con 0x7f9f1c1f4040 2023-05-22T15:27:56.632 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.619+0000 7f9f1affd640 1 --2- 172.21.15.16:0/354023453 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f9f08065380 0x7f9f08067830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:56.632 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.619+0000 7f9f1affd640 1 --2- 172.21.15.16:0/354023453 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f9f08065380 0x7f9f08067830 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f9f0c007fd0 tx=0x7f9f0c009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:56.632 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.623+0000 7f9f037fe640 1 -- 172.21.15.16:0/354023453 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f9f140ebba0 con 0x7f9f1c1f4040 2023-05-22T15:27:56.782 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.775+0000 7f9f21498640 1 -- 172.21.15.16:0/354023453 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f9f1c15d660 con 0x7f9f1c1f4040 2023-05-22T15:27:56.782 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.775+0000 7f9f037fe640 1 -- 172.21.15.16:0/354023453 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v29) v1 ==== 104+0+11 (secure 0 0 0) 0x7f9f1c15d660 con 0x7f9f1c1f4040 2023-05-22T15:27:56.783 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2023-05-22T15:27:56.784 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.775+0000 7f9f21498640 1 -- 172.21.15.16:0/354023453 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f9f08065380 msgr2=0x7f9f08067830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:56.785 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.775+0000 7f9f21498640 1 --2- 172.21.15.16:0/354023453 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f9f08065380 0x7f9f08067830 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f9f0c007fd0 tx=0x7f9f0c009040 comp rx=0 tx=0).stop 2023-05-22T15:27:56.785 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.775+0000 7f9f21498640 1 -- 172.21.15.16:0/354023453 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f1c1f4040 msgr2=0x7f9f1c150060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:56.785 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.775+0000 7f9f21498640 1 --2- 172.21.15.16:0/354023453 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f1c1f4040 0x7f9f1c150060 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f9f140195b0 tx=0x7f9f14019740 comp rx=0 tx=0).stop 2023-05-22T15:27:56.785 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.775+0000 7f9f21498640 1 -- 172.21.15.16:0/354023453 shutdown_connections 2023-05-22T15:27:56.785 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.775+0000 7f9f21498640 1 --2- 172.21.15.16:0/354023453 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f1c1f4040 0x7f9f1c150060 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:56.785 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.775+0000 7f9f21498640 1 --2- 172.21.15.16:0/354023453 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f9f08065380 0x7f9f08067830 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:56.786 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.775+0000 7f9f21498640 1 --2- 172.21.15.16:0/354023453 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9f1c0cc6c0 0x7f9f1c0c9f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:56.786 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.775+0000 7f9f21498640 1 --2- 172.21.15.16:0/354023453 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9f1c0c8250 0x7f9f1c149c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:56.786 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.775+0000 7f9f21498640 1 -- 172.21.15.16:0/354023453 >> 172.21.15.16:0/354023453 conn(0x7f9f1c0b1110 msgr2=0x7f9f1c06a400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:56.786 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.779+0000 7f9f21498640 1 -- 172.21.15.16:0/354023453 shutdown_connections 2023-05-22T15:27:56.786 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.779+0000 7f9f21498640 1 -- 172.21.15.16:0/354023453 wait complete. 2023-05-22T15:27:56.802 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-05-22T15:27:56.929 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.923+0000 7fa3a10d1640 1 Processor -- start 2023-05-22T15:27:56.930 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.923+0000 7fa3a10d1640 1 -- start start 2023-05-22T15:27:56.931 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.923+0000 7fa3a10d1640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa39c159d40 0x7fa39c0cb260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:56.931 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.923+0000 7fa3a10d1640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fa39c06b6c0 0x7fa39c06bb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:56.931 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.923+0000 7fa3a10d1640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa39c070c30 con 0x7fa39c159970 2023-05-22T15:27:56.932 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.923+0000 7fa3a10d1640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa39c15dd90 con 0x7fa39c06b6c0 2023-05-22T15:27:56.932 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.923+0000 7fa3a10d1640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa39c15a190 con 0x7fa39c159d40 2023-05-22T15:27:56.932 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.923+0000 7fa39b59e640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fa39c06b6c0 0x7fa39c06bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:56.933 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.923+0000 7fa39b59e640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fa39c06b6c0 0x7fa39c06bb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54378/0 (socket says 172.21.15.16:54378) 2023-05-22T15:27:56.933 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.923+0000 7fa39b59e640 1 -- 172.21.15.16:0/3955391880 learned_addr learned my addr 172.21.15.16:0/3955391880 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:56.933 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.923+0000 7fa39ad9d640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fa39c159970 0x7fa39c15d680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:33076/0 (socket says 172.21.15.16:33076) 2023-05-22T15:27:56.933 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.923+0000 7fa39a59c640 1 --2- 172.21.15.16:0/3955391880 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa39c159d40 0x7fa39c0cb260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:56.934 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.923+0000 7fa39b59e640 1 -- 172.21.15.16:0/3955391880 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa39c159d40 msgr2=0x7fa39c0cb260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:56.935 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.923+0000 7fa39b59e640 1 --2- 172.21.15.16:0/3955391880 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa39c159d40 0x7fa39c0cb260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:56.936 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.923+0000 7fa39b59e640 1 -- 172.21.15.16:0/3955391880 >> v1:172.21.15.16:6789/0 conn(0x7fa39c159970 legacy=0x7fa39c15d680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:56.936 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.923+0000 7fa39b59e640 1 -- 172.21.15.16:0/3955391880 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa39c06c060 con 0x7fa39c06b6c0 2023-05-22T15:27:56.940 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.927+0000 7fa39b59e640 1 --2- 172.21.15.16:0/3955391880 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa39c06b6c0 0x7fa39c06bb20 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7fa394009e50 tx=0x7fa39401c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3a2aa50aa3038d0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:56.940 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.927+0000 7fa399d9b640 1 -- 172.21.15.16:0/3955391880 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3940a5810 con 0x7fa39c06b6c0 2023-05-22T15:27:56.940 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.927+0000 7fa399d9b640 1 -- 172.21.15.16:0/3955391880 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3940a5970 con 0x7fa39c06b6c0 2023-05-22T15:27:56.940 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.927+0000 7fa399d9b640 1 -- 172.21.15.16:0/3955391880 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3940a5d00 con 0x7fa39c06b6c0 2023-05-22T15:27:56.940 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.927+0000 7fa3a10d1640 1 -- 172.21.15.16:0/3955391880 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa39c06b6c0 msgr2=0x7fa39c06bb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:56.941 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.927+0000 7fa3a10d1640 1 --2- 172.21.15.16:0/3955391880 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa39c06b6c0 0x7fa39c06bb20 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7fa394009e50 tx=0x7fa39401c950 comp rx=0 tx=0).stop 2023-05-22T15:27:56.941 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.927+0000 7fa3a10d1640 1 -- 172.21.15.16:0/3955391880 shutdown_connections 2023-05-22T15:27:56.941 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.927+0000 7fa3a10d1640 1 --2- 172.21.15.16:0/3955391880 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa39c06b6c0 0x7fa39c06bb20 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:56.941 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.927+0000 7fa3a10d1640 1 --2- 172.21.15.16:0/3955391880 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa39c159d40 0x7fa39c0cb260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:56.941 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.927+0000 7fa3a10d1640 1 -- 172.21.15.16:0/3955391880 >> 172.21.15.16:0/3955391880 conn(0x7fa39c069720 msgr2=0x7fa39c069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:56.941 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.927+0000 7fa3a10d1640 1 -- 172.21.15.16:0/3955391880 shutdown_connections 2023-05-22T15:27:56.941 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.927+0000 7fa3a10d1640 1 -- 172.21.15.16:0/3955391880 wait complete. 2023-05-22T15:27:56.942 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.927+0000 7fa3a10d1640 1 Processor -- start 2023-05-22T15:27:56.942 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.927+0000 7fa3a10d1640 1 -- start start 2023-05-22T15:27:56.942 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.931+0000 7fa3a10d1640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa39c06b6c0 0x7fa39c0c1da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:56.942 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.931+0000 7fa3a10d1640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa39c159970 0x7fa39c153400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:56.942 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.931+0000 7fa3a10d1640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa39c159d40 0x7fa39c14d2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:56.942 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.931+0000 7fa3a10d1640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa39c14ce50 con 0x7fa39c159d40 2023-05-22T15:27:56.942 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.931+0000 7fa3a10d1640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa39c14c520 con 0x7fa39c159970 2023-05-22T15:27:56.943 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.931+0000 7fa3a10d1640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa39c14c800 con 0x7fa39c06b6c0 2023-05-22T15:27:56.943 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.931+0000 7fa39b59e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa39c159d40 0x7fa39c14d2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:56.943 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.931+0000 7fa39b59e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa39c159d40 0x7fa39c14d2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54390/0 (socket says 172.21.15.16:54390) 2023-05-22T15:27:56.943 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.931+0000 7fa39b59e640 1 -- 172.21.15.16:0/3862502292 learned_addr learned my addr 172.21.15.16:0/3862502292 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:56.943 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.931+0000 7fa39ad9d640 1 --2- 172.21.15.16:0/3862502292 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa39c06b6c0 0x7fa39c0c1da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:56.943 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.931+0000 7fa39a59c640 1 --2- 172.21.15.16:0/3862502292 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa39c159970 0x7fa39c153400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:56.943 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.931+0000 7fa39b59e640 1 -- 172.21.15.16:0/3862502292 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa39c06b6c0 msgr2=0x7fa39c0c1da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:56.944 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.931+0000 7fa39b59e640 1 --2- 172.21.15.16:0/3862502292 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa39c06b6c0 0x7fa39c0c1da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:56.944 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.931+0000 7fa39b59e640 1 -- 172.21.15.16:0/3862502292 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa39c159970 msgr2=0x7fa39c153400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:56.944 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.931+0000 7fa39b59e640 1 --2- 172.21.15.16:0/3862502292 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa39c159970 0x7fa39c153400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:56.944 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.931+0000 7fa39b59e640 1 -- 172.21.15.16:0/3862502292 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa394009b30 con 0x7fa39c159d40 2023-05-22T15:27:56.944 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.931+0000 7fa39a59c640 1 --2- 172.21.15.16:0/3862502292 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa39c159970 0x7fa39c153400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:27:56.944 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.931+0000 7fa39b59e640 1 --2- 172.21.15.16:0/3862502292 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa39c159d40 0x7fa39c14d2e0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7fa3940af040 tx=0x7fa3940196f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:56.944 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.931+0000 7fa3837fe640 1 -- 172.21.15.16:0/3862502292 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3940b8670 con 0x7fa39c159d40 2023-05-22T15:27:56.944 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.931+0000 7fa3837fe640 1 -- 172.21.15.16:0/3862502292 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3940b87d0 con 0x7fa39c159d40 2023-05-22T15:27:56.945 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.931+0000 7fa3a10d1640 1 -- 172.21.15.16:0/3862502292 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa39c071700 con 0x7fa39c159d40 2023-05-22T15:27:56.945 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.931+0000 7fa3837fe640 1 -- 172.21.15.16:0/3862502292 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3940b8b60 con 0x7fa39c159d40 2023-05-22T15:27:56.945 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.931+0000 7fa3a10d1640 1 -- 172.21.15.16:0/3862502292 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa39c071420 con 0x7fa39c159d40 2023-05-22T15:27:56.945 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.931+0000 7fa3a10d1640 1 -- 172.21.15.16:0/3862502292 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa39c0cb7a0 con 0x7fa39c159d40 2023-05-22T15:27:56.945 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.931+0000 7fa3837fe640 1 -- 172.21.15.16:0/3862502292 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7fa3940b8cc0 con 0x7fa39c159d40 2023-05-22T15:27:56.945 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.931+0000 7fa3837fe640 1 --2- 172.21.15.16:0/3862502292 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fa388065380 0x7fa388067830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:56.946 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.931+0000 7fa3837fe640 1 -- 172.21.15.16:0/3862502292 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6871+0+0 (secure 0 0 0) 0x7fa39411e1c0 con 0x7fa39c159d40 2023-05-22T15:27:56.946 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.931+0000 7fa39ad9d640 1 --2- 172.21.15.16:0/3862502292 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fa388065380 0x7fa388067830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:56.946 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.931+0000 7fa39ad9d640 1 --2- 172.21.15.16:0/3862502292 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fa388065380 0x7fa388067830 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fa38c000f70 tx=0x7fa38c007c90 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:56.947 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:56.935+0000 7fa3837fe640 1 -- 172.21.15.16:0/3862502292 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fa3940ebba0 con 0x7fa39c159d40 2023-05-22T15:27:57.096 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.087+0000 7fa3a10d1640 1 -- 172.21.15.16:0/3862502292 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fa39c0702c0 con 0x7fa39c159d40 2023-05-22T15:27:57.096 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.087+0000 7fa3837fe640 1 -- 172.21.15.16:0/3862502292 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v29) v1 ==== 100+0+11 (secure 0 0 0) 0x7fa3940ef1c0 con 0x7fa39c159d40 2023-05-22T15:27:57.097 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 64 2023-05-22T15:27:57.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.091+0000 7fa3a10d1640 1 -- 172.21.15.16:0/3862502292 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fa388065380 msgr2=0x7fa388067830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:57.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.091+0000 7fa3a10d1640 1 --2- 172.21.15.16:0/3862502292 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fa388065380 0x7fa388067830 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fa38c000f70 tx=0x7fa38c007c90 comp rx=0 tx=0).stop 2023-05-22T15:27:57.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.091+0000 7fa3a10d1640 1 -- 172.21.15.16:0/3862502292 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa39c159d40 msgr2=0x7fa39c14d2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:57.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.091+0000 7fa3a10d1640 1 --2- 172.21.15.16:0/3862502292 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa39c159d40 0x7fa39c14d2e0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7fa3940af040 tx=0x7fa3940196f0 comp rx=0 tx=0).stop 2023-05-22T15:27:57.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.091+0000 7fa3a10d1640 1 -- 172.21.15.16:0/3862502292 shutdown_connections 2023-05-22T15:27:57.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.091+0000 7fa3a10d1640 1 --2- 172.21.15.16:0/3862502292 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa39c159d40 0x7fa39c14d2e0 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:57.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.091+0000 7fa3a10d1640 1 --2- 172.21.15.16:0/3862502292 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa39c159970 0x7fa39c153400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:57.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.091+0000 7fa3a10d1640 1 --2- 172.21.15.16:0/3862502292 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fa388065380 0x7fa388067830 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:57.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.091+0000 7fa3a10d1640 1 --2- 172.21.15.16:0/3862502292 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa39c06b6c0 0x7fa39c0c1da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:57.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.091+0000 7fa3a10d1640 1 -- 172.21.15.16:0/3862502292 >> 172.21.15.16:0/3862502292 conn(0x7fa39c069720 msgr2=0x7fa39c15b260 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:57.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.091+0000 7fa3a10d1640 1 -- 172.21.15.16:0/3862502292 shutdown_connections 2023-05-22T15:27:57.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.091+0000 7fa3a10d1640 1 -- 172.21.15.16:0/3862502292 wait complete. 2023-05-22T15:27:57.115 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-05-22T15:27:57.237 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.231+0000 7f3339a08640 1 Processor -- start 2023-05-22T15:27:57.237 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.231+0000 7f3339a08640 1 -- start start 2023-05-22T15:27:57.239 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.231+0000 7f3339a08640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f3334159970 0x7f3334159d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:57.239 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.231+0000 7f3339a08640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f333415d680 0x7f33340cb260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:57.240 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.231+0000 7f3339a08640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3334070c30 con 0x7f333415dad0 2023-05-22T15:27:57.240 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.231+0000 7f3339a08640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3334070e60 con 0x7f3334159970 2023-05-22T15:27:57.240 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.231+0000 7f3339a08640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3334071140 con 0x7f333415d680 2023-05-22T15:27:57.240 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.231+0000 7f3332ffd640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f3334159970 0x7f3334159d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:57.241 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.231+0000 7f33327fc640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f333415d680 0x7f33340cb260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:57.242 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.231+0000 7f33337fe640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f333415dad0 0x7f333406b6c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.16:57326/0 (socket says 172.21.15.16:57326) 2023-05-22T15:27:57.242 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.231+0000 7f33337fe640 1 -- 172.21.15.16:0/465989469 learned_addr learned my addr 172.21.15.16:0/465989469 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:27:57.242 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.231+0000 7f33327fc640 1 -- 172.21.15.16:0/465989469 >> v2:172.21.15.62:3300/0 conn(0x7f3334159970 msgr2=0x7f3334159d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:57.243 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.231+0000 7f33327fc640 1 --2- 172.21.15.16:0/465989469 >> v2:172.21.15.62:3300/0 conn(0x7f3334159970 0x7f3334159d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:57.243 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.231+0000 7f33327fc640 1 -- 172.21.15.16:0/465989469 >> v1:172.21.15.62:6789/0 conn(0x7f333415dad0 legacy=0x7f333406b6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:57.243 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.231+0000 7f33327fc640 1 -- 172.21.15.16:0/465989469 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3334071420 con 0x7f333415d680 2023-05-22T15:27:57.247 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.235+0000 7f33327fc640 1 --2- 172.21.15.16:0/465989469 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f333415d680 0x7f33340cb260 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f33280097c0 tx=0x7f332801c3c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=17b89c3bd3503754 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:57.247 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.235+0000 7f3331ffb640 1 -- 172.21.15.16:0/465989469 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33280ac640 con 0x7f333415d680 2023-05-22T15:27:57.248 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.235+0000 7f3331ffb640 1 -- 172.21.15.16:0/465989469 <== mon.2 v2:172.21.15.62:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33280ac7a0 con 0x7f333415d680 2023-05-22T15:27:57.248 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.235+0000 7f3331ffb640 1 -- 172.21.15.16:0/465989469 <== mon.2 v2:172.21.15.62:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33280acb30 con 0x7f333415d680 2023-05-22T15:27:57.248 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.235+0000 7f3339a08640 1 -- 172.21.15.16:0/465989469 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f333415d680 msgr2=0x7f33340cb260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:57.249 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.235+0000 7f3339a08640 1 --2- 172.21.15.16:0/465989469 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f333415d680 0x7f33340cb260 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f33280097c0 tx=0x7f332801c3c0 comp rx=0 tx=0).stop 2023-05-22T15:27:57.249 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.235+0000 7f3339a08640 1 -- 172.21.15.16:0/465989469 shutdown_connections 2023-05-22T15:27:57.249 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.235+0000 7f3339a08640 1 --2- 172.21.15.16:0/465989469 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f333415d680 0x7f33340cb260 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:57.249 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.235+0000 7f3339a08640 1 --2- 172.21.15.16:0/465989469 >> v2:172.21.15.62:3300/0 conn(0x7f3334159970 0x7f3334159d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:57.250 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.235+0000 7f3339a08640 1 -- 172.21.15.16:0/465989469 >> 172.21.15.16:0/465989469 conn(0x7f3334069720 msgr2=0x7f3334069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:57.250 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.235+0000 7f3339a08640 1 -- 172.21.15.16:0/465989469 shutdown_connections 2023-05-22T15:27:57.250 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.235+0000 7f3339a08640 1 -- 172.21.15.16:0/465989469 wait complete. 2023-05-22T15:27:57.250 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.235+0000 7f3339a08640 1 Processor -- start 2023-05-22T15:27:57.251 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.235+0000 7f3339a08640 1 -- start start 2023-05-22T15:27:57.251 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.239+0000 7f3339a08640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3334159970 0x7f33340b3560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:57.251 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.239+0000 7f3339a08640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f333415d680 0x7f33340b0e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:57.251 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.239+0000 7f3339a08640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f333415dad0 0x7f33340b1380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:57.252 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.239+0000 7f3339a08640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3334164440 con 0x7f3334159970 2023-05-22T15:27:57.252 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.239+0000 7f3339a08640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33340b1a00 con 0x7f333415d680 2023-05-22T15:27:57.252 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.239+0000 7f3339a08640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33340b1ce0 con 0x7f333415dad0 2023-05-22T15:27:57.253 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.239+0000 7f3332ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3334159970 0x7f33340b3560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:57.253 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.239+0000 7f3332ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3334159970 0x7f33340b3560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54406/0 (socket says 172.21.15.16:54406) 2023-05-22T15:27:57.253 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.239+0000 7f3332ffd640 1 -- 172.21.15.16:0/2903455465 learned_addr learned my addr 172.21.15.16:0/2903455465 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:57.253 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.239+0000 7f33327fc640 1 --2- 172.21.15.16:0/2903455465 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f333415d680 0x7f33340b0e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:57.254 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.239+0000 7f33337fe640 1 --2- 172.21.15.16:0/2903455465 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f333415dad0 0x7f33340b1380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:57.254 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.239+0000 7f3332ffd640 1 -- 172.21.15.16:0/2903455465 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f333415dad0 msgr2=0x7f33340b1380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:57.254 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.239+0000 7f3332ffd640 1 --2- 172.21.15.16:0/2903455465 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f333415dad0 0x7f33340b1380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:57.254 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.239+0000 7f3332ffd640 1 -- 172.21.15.16:0/2903455465 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f333415d680 msgr2=0x7f33340b0e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:57.254 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.239+0000 7f3332ffd640 1 --2- 172.21.15.16:0/2903455465 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f333415d680 0x7f33340b0e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:57.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.239+0000 7f3332ffd640 1 -- 172.21.15.16:0/2903455465 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3328009420 con 0x7f3334159970 2023-05-22T15:27:57.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.239+0000 7f33337fe640 1 --2- 172.21.15.16:0/2903455465 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f333415dad0 0x7f33340b1380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:27:57.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.239+0000 7f33327fc640 1 --2- 172.21.15.16:0/2903455465 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f333415d680 0x7f33340b0e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:27:57.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.239+0000 7f3332ffd640 1 --2- 172.21.15.16:0/2903455465 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3334159970 0x7f33340b3560 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f332400e930 tx=0x7f332400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:57.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.239+0000 7f331bfff640 1 -- 172.21.15.16:0/2903455465 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3324009690 con 0x7f3334159970 2023-05-22T15:27:57.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.239+0000 7f331bfff640 1 -- 172.21.15.16:0/2903455465 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f332400ca00 con 0x7f3334159970 2023-05-22T15:27:57.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.239+0000 7f331bfff640 1 -- 172.21.15.16:0/2903455465 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f332400cd70 con 0x7f3334159970 2023-05-22T15:27:57.256 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.239+0000 7f3339a08640 1 -- 172.21.15.16:0/2903455465 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33340cae20 con 0x7f3334159970 2023-05-22T15:27:57.256 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.239+0000 7f3339a08640 1 -- 172.21.15.16:0/2903455465 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33340cac90 con 0x7f3334159970 2023-05-22T15:27:57.256 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.239+0000 7f3339a08640 1 -- 172.21.15.16:0/2903455465 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f333406bdd0 con 0x7f3334159970 2023-05-22T15:27:57.257 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.239+0000 7f331bfff640 1 -- 172.21.15.16:0/2903455465 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7f3324013470 con 0x7f3334159970 2023-05-22T15:27:57.257 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.239+0000 7f331bfff640 1 --2- 172.21.15.16:0/2903455465 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f3320065380 0x7f3320067830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:57.257 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.239+0000 7f331bfff640 1 -- 172.21.15.16:0/2903455465 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f3324082d60 con 0x7f3334159970 2023-05-22T15:27:57.257 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.239+0000 7f33327fc640 1 --2- 172.21.15.16:0/2903455465 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f3320065380 0x7f3320067830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:57.258 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.243+0000 7f33327fc640 1 --2- 172.21.15.16:0/2903455465 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f3320065380 0x7f3320067830 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f33280ae000 tx=0x7f33280a7000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:57.258 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.243+0000 7f331bfff640 1 -- 172.21.15.16:0/2903455465 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f33240507a0 con 0x7f3334159970 2023-05-22T15:27:57.403 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.395+0000 7f3339a08640 1 -- 172.21.15.16:0/2903455465 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f333404dbe0 con 0x7f3334159970 2023-05-22T15:27:57.403 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.395+0000 7f331bfff640 1 -- 172.21.15.16:0/2903455465 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v29) v1 ==== 103+0+11 (secure 0 0 0) 0x7f3324053dc0 con 0x7f3334159970 2023-05-22T15:27:57.404 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2023-05-22T15:27:57.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.399+0000 7f3339a08640 1 -- 172.21.15.16:0/2903455465 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f3320065380 msgr2=0x7f3320067830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:57.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.399+0000 7f3339a08640 1 --2- 172.21.15.16:0/2903455465 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f3320065380 0x7f3320067830 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f33280ae000 tx=0x7f33280a7000 comp rx=0 tx=0).stop 2023-05-22T15:27:57.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.399+0000 7f3339a08640 1 -- 172.21.15.16:0/2903455465 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3334159970 msgr2=0x7f33340b3560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:57.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.399+0000 7f3339a08640 1 --2- 172.21.15.16:0/2903455465 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3334159970 0x7f33340b3560 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f332400e930 tx=0x7f332400edf0 comp rx=0 tx=0).stop 2023-05-22T15:27:57.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.399+0000 7f3339a08640 1 -- 172.21.15.16:0/2903455465 shutdown_connections 2023-05-22T15:27:57.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.399+0000 7f3339a08640 1 --2- 172.21.15.16:0/2903455465 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f3320065380 0x7f3320067830 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:57.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.399+0000 7f3339a08640 1 --2- 172.21.15.16:0/2903455465 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f333415dad0 0x7f33340b1380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:57.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.399+0000 7f3339a08640 1 --2- 172.21.15.16:0/2903455465 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f333415d680 0x7f33340b0e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:57.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.399+0000 7f3339a08640 1 --2- 172.21.15.16:0/2903455465 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3334159970 0x7f33340b3560 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:57.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.399+0000 7f3339a08640 1 -- 172.21.15.16:0/2903455465 >> 172.21.15.16:0/2903455465 conn(0x7f3334069720 msgr2=0x7f3334157000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:57.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.399+0000 7f3339a08640 1 -- 172.21.15.16:0/2903455465 shutdown_connections 2023-05-22T15:27:57.408 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.399+0000 7f3339a08640 1 -- 172.21.15.16:0/2903455465 wait complete. 2023-05-22T15:27:57.421 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-05-22T15:27:57.534 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.527+0000 7f384af59640 1 Processor -- start 2023-05-22T15:27:57.534 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.527+0000 7f384af59640 1 -- start start 2023-05-22T15:27:57.537 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.527+0000 7f384af59640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f3844159cd0 0x7f384415a0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:57.537 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.527+0000 7f384af59640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f38440cc6f0 0x7f38440cfe30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:57.537 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.527+0000 7f384af59640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38440cf9a0 con 0x7f38440b2880 2023-05-22T15:27:57.538 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.527+0000 7f384af59640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f384415aba0 con 0x7f38440cc6f0 2023-05-22T15:27:57.538 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.527+0000 7f384af59640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f384415ae80 con 0x7f3844159cd0 2023-05-22T15:27:57.538 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.527+0000 7f38494f6640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f38440cc6f0 0x7f38440cfe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:57.539 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.527+0000 7f38494f6640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f38440cc6f0 0x7f38440cfe30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54418/0 (socket says 172.21.15.16:54418) 2023-05-22T15:27:57.539 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.527+0000 7f38494f6640 1 -- 172.21.15.16:0/141935651 learned_addr learned my addr 172.21.15.16:0/141935651 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:57.539 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.527+0000 7f3848cf5640 1 --2- 172.21.15.16:0/141935651 >> v2:172.21.15.62:3300/0 conn(0x7f3844159cd0 0x7f384415a0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:57.539 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.527+0000 7f38494f6640 1 -- 172.21.15.16:0/141935651 >> v2:172.21.15.62:3300/0 conn(0x7f3844159cd0 msgr2=0x7f384415a0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:57.539 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.527+0000 7f38494f6640 1 --2- 172.21.15.16:0/141935651 >> v2:172.21.15.62:3300/0 conn(0x7f3844159cd0 0x7f384415a0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:57.540 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.527+0000 7f38494f6640 1 -- 172.21.15.16:0/141935651 >> v1:172.21.15.62:6789/0 conn(0x7f38440b2880 legacy=0x7f38440cbfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:57.540 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.527+0000 7f38494f6640 1 -- 172.21.15.16:0/141935651 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f384415b160 con 0x7f38440cc6f0 2023-05-22T15:27:57.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.531+0000 7f38494f6640 1 --2- 172.21.15.16:0/141935651 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38440cc6f0 0x7f38440cfe30 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f38440d1330 tx=0x7f383c01c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8a9d147a7331e4 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:57.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.531+0000 7f38437fe640 1 -- 172.21.15.16:0/141935651 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f383c0a5730 con 0x7f38440cc6f0 2023-05-22T15:27:57.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.531+0000 7f38437fe640 1 -- 172.21.15.16:0/141935651 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f383c0a5890 con 0x7f38440cc6f0 2023-05-22T15:27:57.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.531+0000 7f38437fe640 1 -- 172.21.15.16:0/141935651 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f383c0a5c20 con 0x7f38440cc6f0 2023-05-22T15:27:57.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.531+0000 7f384af59640 1 -- 172.21.15.16:0/141935651 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38440cc6f0 msgr2=0x7f38440cfe30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:57.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.531+0000 7f384af59640 1 --2- 172.21.15.16:0/141935651 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38440cc6f0 0x7f38440cfe30 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f38440d1330 tx=0x7f383c01c960 comp rx=0 tx=0).stop 2023-05-22T15:27:57.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.531+0000 7f384af59640 1 -- 172.21.15.16:0/141935651 shutdown_connections 2023-05-22T15:27:57.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.531+0000 7f384af59640 1 --2- 172.21.15.16:0/141935651 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38440cc6f0 0x7f38440cfe30 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:57.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.531+0000 7f384af59640 1 --2- 172.21.15.16:0/141935651 >> v2:172.21.15.62:3300/0 conn(0x7f3844159cd0 0x7f384415a0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:57.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.531+0000 7f384af59640 1 -- 172.21.15.16:0/141935651 >> 172.21.15.16:0/141935651 conn(0x7f384406f5c0 msgr2=0x7f384406f9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:57.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.531+0000 7f384af59640 1 -- 172.21.15.16:0/141935651 shutdown_connections 2023-05-22T15:27:57.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.531+0000 7f384af59640 1 -- 172.21.15.16:0/141935651 wait complete. 2023-05-22T15:27:57.544 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.531+0000 7f384af59640 1 Processor -- start 2023-05-22T15:27:57.544 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.531+0000 7f384af59640 1 -- start start 2023-05-22T15:27:57.544 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.535+0000 7f384af59640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f38440b2880 0x7f384413e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:57.544 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.535+0000 7f384af59640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38440cc6f0 0x7f38441591c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:57.544 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.535+0000 7f384af59640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3844159cd0 0x7f384414eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:57.544 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.535+0000 7f384af59640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38440c8550 con 0x7f38440cc6f0 2023-05-22T15:27:57.544 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.535+0000 7f384af59640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38440c8270 con 0x7f38440b2880 2023-05-22T15:27:57.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.535+0000 7f384af59640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38440c9c50 con 0x7f3844159cd0 2023-05-22T15:27:57.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.535+0000 7f3843fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38440cc6f0 0x7f38441591c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:57.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.535+0000 7f3843fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38440cc6f0 0x7f38441591c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54426/0 (socket says 172.21.15.16:54426) 2023-05-22T15:27:57.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.535+0000 7f3843fff640 1 -- 172.21.15.16:0/1034974599 learned_addr learned my addr 172.21.15.16:0/1034974599 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:57.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.535+0000 7f3848cf5640 1 --2- 172.21.15.16:0/1034974599 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f38440b2880 0x7f384413e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:57.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.535+0000 7f38494f6640 1 --2- 172.21.15.16:0/1034974599 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3844159cd0 0x7f384414eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:57.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.535+0000 7f3843fff640 1 -- 172.21.15.16:0/1034974599 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3844159cd0 msgr2=0x7f384414eed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:57.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.535+0000 7f3843fff640 1 --2- 172.21.15.16:0/1034974599 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3844159cd0 0x7f384414eed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:57.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.535+0000 7f3843fff640 1 -- 172.21.15.16:0/1034974599 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f38440b2880 msgr2=0x7f384413e6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:57.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.535+0000 7f3843fff640 1 --2- 172.21.15.16:0/1034974599 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f38440b2880 0x7f384413e6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:57.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.535+0000 7f3843fff640 1 -- 172.21.15.16:0/1034974599 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f383c009b30 con 0x7f38440cc6f0 2023-05-22T15:27:57.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.535+0000 7f3843fff640 1 --2- 172.21.15.16:0/1034974599 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38440cc6f0 0x7f38441591c0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f383800b410 tx=0x7f383800b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:57.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.535+0000 7f38417fa640 1 -- 172.21.15.16:0/1034974599 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3838004020 con 0x7f38440cc6f0 2023-05-22T15:27:57.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.535+0000 7f38417fa640 1 -- 172.21.15.16:0/1034974599 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3838004180 con 0x7f38440cc6f0 2023-05-22T15:27:57.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.535+0000 7f38417fa640 1 -- 172.21.15.16:0/1034974599 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3838011630 con 0x7f38440cc6f0 2023-05-22T15:27:57.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.535+0000 7f384af59640 1 -- 172.21.15.16:0/1034974599 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38440d0d10 con 0x7f38440cc6f0 2023-05-22T15:27:57.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.535+0000 7f384af59640 1 -- 172.21.15.16:0/1034974599 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38440d0b80 con 0x7f38440cc6f0 2023-05-22T15:27:57.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.535+0000 7f384af59640 1 -- 172.21.15.16:0/1034974599 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38440d0ea0 con 0x7f38440cc6f0 2023-05-22T15:27:57.549 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.535+0000 7f38417fa640 1 -- 172.21.15.16:0/1034974599 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7f3838011790 con 0x7f38440cc6f0 2023-05-22T15:27:57.549 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.535+0000 7f38417fa640 1 --2- 172.21.15.16:0/1034974599 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f3830065380 0x7f3830067830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:57.549 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.535+0000 7f38417fa640 1 -- 172.21.15.16:0/1034974599 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f3838082900 con 0x7f38440cc6f0 2023-05-22T15:27:57.549 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.535+0000 7f3848cf5640 1 --2- 172.21.15.16:0/1034974599 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f3830065380 0x7f3830067830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:57.549 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.535+0000 7f3848cf5640 1 --2- 172.21.15.16:0/1034974599 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f3830065380 0x7f3830067830 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f3834009790 tx=0x7f3834008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:57.549 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.539+0000 7f38417fa640 1 -- 172.21.15.16:0/1034974599 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f3838050250 con 0x7f38440cc6f0 2023-05-22T15:27:57.711 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.703+0000 7f384af59640 1 -- 172.21.15.16:0/1034974599 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f384404dbe0 con 0x7f38440cc6f0 2023-05-22T15:27:57.712 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.703+0000 7f38417fa640 1 -- 172.21.15.16:0/1034974599 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v29) v1 ==== 80+0+0 (secure 0 0 0) 0x7f3838053870 con 0x7f38440cc6f0 2023-05-22T15:27:57.712 INFO:teuthology.orchestra.run.smithi016.stderr:listed 0 entries 2023-05-22T15:27:57.714 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.707+0000 7f384af59640 1 -- 172.21.15.16:0/1034974599 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f3830065380 msgr2=0x7f3830067830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:57.714 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.707+0000 7f384af59640 1 --2- 172.21.15.16:0/1034974599 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f3830065380 0x7f3830067830 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f3834009790 tx=0x7f3834008040 comp rx=0 tx=0).stop 2023-05-22T15:27:57.714 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.707+0000 7f384af59640 1 -- 172.21.15.16:0/1034974599 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38440cc6f0 msgr2=0x7f38441591c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:57.715 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.707+0000 7f384af59640 1 --2- 172.21.15.16:0/1034974599 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38440cc6f0 0x7f38441591c0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f383800b410 tx=0x7f383800b8d0 comp rx=0 tx=0).stop 2023-05-22T15:27:57.715 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.707+0000 7f384af59640 1 -- 172.21.15.16:0/1034974599 shutdown_connections 2023-05-22T15:27:57.715 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.707+0000 7f384af59640 1 --2- 172.21.15.16:0/1034974599 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f3830065380 0x7f3830067830 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:57.715 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.707+0000 7f384af59640 1 --2- 172.21.15.16:0/1034974599 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3844159cd0 0x7f384414eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:57.715 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.707+0000 7f384af59640 1 --2- 172.21.15.16:0/1034974599 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38440cc6f0 0x7f38441591c0 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:57.715 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.707+0000 7f384af59640 1 --2- 172.21.15.16:0/1034974599 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f38440b2880 0x7f384413e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:57.715 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.707+0000 7f384af59640 1 -- 172.21.15.16:0/1034974599 >> 172.21.15.16:0/1034974599 conn(0x7f384406f5c0 msgr2=0x7f384406de00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:57.716 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.707+0000 7f384af59640 1 -- 172.21.15.16:0/1034974599 shutdown_connections 2023-05-22T15:27:57.716 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.707+0000 7f384af59640 1 -- 172.21.15.16:0/1034974599 wait complete. 2023-05-22T15:27:57.729 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi062.front.sepia.ceph.com... 2023-05-22T15:27:57.730 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:27:57.730 DEBUG:teuthology.orchestra.run.smithi062:> sudo fusermount -u /home/ubuntu/cephtest/mnt.3 2023-05-22T15:27:57.745 INFO:teuthology.orchestra.run:waiting for 300 2023-05-22T15:27:57.746 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi062.front.sepia.ceph.com 2023-05-22T15:27:57.746 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:27:57.747 DEBUG:teuthology.orchestra.run.smithi062:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2023-05-22T15:27:57.794 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2023-05-22T15:27:57.794 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-05-22T15:27:57.921 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.915+0000 7f4716596640 1 Processor -- start 2023-05-22T15:27:57.921 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.915+0000 7f4716596640 1 -- start start 2023-05-22T15:27:57.922 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.915+0000 7f4716596640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f47100b7dd0 0x7f47100ccab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:57.923 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.915+0000 7f4716596640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f471015f700 0x7f471015fb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:57.923 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.915+0000 7f4716596640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47100cd170 con 0x7f4710160080 2023-05-22T15:27:57.923 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.915+0000 7f4716596640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47100d3360 con 0x7f47100b7dd0 2023-05-22T15:27:57.923 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.915+0000 7f4716596640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47100d2ac0 con 0x7f471015f700 2023-05-22T15:27:57.923 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.915+0000 7f470ffff640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f47100b7dd0 0x7f47100ccab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:57.923 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.915+0000 7f470ffff640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f47100b7dd0 0x7f47100ccab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54440/0 (socket says 172.21.15.16:54440) 2023-05-22T15:27:57.923 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.915+0000 7f470ffff640 1 -- 172.21.15.16:0/2287055135 learned_addr learned my addr 172.21.15.16:0/2287055135 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:57.924 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.915+0000 7f4707fff640 1 --2- 172.21.15.16:0/2287055135 >> v2:172.21.15.62:3300/0 conn(0x7f471015f700 0x7f471015fb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:57.924 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.915+0000 7f470ffff640 1 -- 172.21.15.16:0/2287055135 >> v2:172.21.15.62:3300/0 conn(0x7f471015f700 msgr2=0x7f471015fb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:57.924 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.915+0000 7f470ffff640 1 --2- 172.21.15.16:0/2287055135 >> v2:172.21.15.62:3300/0 conn(0x7f471015f700 0x7f471015fb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:57.924 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.915+0000 7f470ffff640 1 -- 172.21.15.16:0/2287055135 >> v1:172.21.15.62:6789/0 conn(0x7f4710160080 legacy=0x7f47100ce720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:57.924 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.915+0000 7f470ffff640 1 -- 172.21.15.16:0/2287055135 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47100d2da0 con 0x7f47100b7dd0 2023-05-22T15:27:57.926 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.919+0000 7f470ffff640 1 --2- 172.21.15.16:0/2287055135 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f47100b7dd0 0x7f47100ccab0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f47100b1320 tx=0x7f470001c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42d34ef4727cf9ac server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:57.927 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.919+0000 7f470f7fe640 1 -- 172.21.15.16:0/2287055135 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47000ab670 con 0x7f47100b7dd0 2023-05-22T15:27:57.927 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.919+0000 7f470f7fe640 1 -- 172.21.15.16:0/2287055135 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47000ab7d0 con 0x7f47100b7dd0 2023-05-22T15:27:57.927 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.919+0000 7f470f7fe640 1 -- 172.21.15.16:0/2287055135 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47000abb60 con 0x7f47100b7dd0 2023-05-22T15:27:57.928 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.919+0000 7f4716596640 1 -- 172.21.15.16:0/2287055135 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f47100b7dd0 msgr2=0x7f47100ccab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:57.928 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.919+0000 7f4716596640 1 --2- 172.21.15.16:0/2287055135 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f47100b7dd0 0x7f47100ccab0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f47100b1320 tx=0x7f470001c4d0 comp rx=0 tx=0).stop 2023-05-22T15:27:57.928 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.919+0000 7f4716596640 1 -- 172.21.15.16:0/2287055135 shutdown_connections 2023-05-22T15:27:57.928 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.919+0000 7f4716596640 1 --2- 172.21.15.16:0/2287055135 >> v2:172.21.15.62:3300/0 conn(0x7f471015f700 0x7f471015fb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:57.928 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.919+0000 7f4716596640 1 --2- 172.21.15.16:0/2287055135 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f47100b7dd0 0x7f47100ccab0 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:57.928 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.919+0000 7f4716596640 1 -- 172.21.15.16:0/2287055135 >> 172.21.15.16:0/2287055135 conn(0x7f47100655c0 msgr2=0x7f4710070e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:57.928 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.919+0000 7f4716596640 1 -- 172.21.15.16:0/2287055135 shutdown_connections 2023-05-22T15:27:57.929 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.919+0000 7f4716596640 1 -- 172.21.15.16:0/2287055135 wait complete. 2023-05-22T15:27:57.931 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.919+0000 7f4716596640 1 Processor -- start 2023-05-22T15:27:57.931 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.919+0000 7f4716596640 1 -- start start 2023-05-22T15:27:57.931 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.923+0000 7f4716596640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f47100b7dd0 0x7f471015f070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:57.932 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.923+0000 7f4716596640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f471015f700 0x7f47100b5290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:57.932 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.923+0000 7f4716596640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4710160080 0x7f4710152370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:57.932 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.923+0000 7f4716596640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47100cbc40 con 0x7f47100b7dd0 2023-05-22T15:27:57.932 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.923+0000 7f4716596640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47100b57d0 con 0x7f4710160080 2023-05-22T15:27:57.932 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.923+0000 7f4716596640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47100cc670 con 0x7f471015f700 2023-05-22T15:27:57.932 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.923+0000 7f470ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f47100b7dd0 0x7f471015f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:57.932 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.923+0000 7f470ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f47100b7dd0 0x7f471015f070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54444/0 (socket says 172.21.15.16:54444) 2023-05-22T15:27:57.933 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.923+0000 7f470ffff640 1 -- 172.21.15.16:0/197047020 learned_addr learned my addr 172.21.15.16:0/197047020 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:57.933 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.923+0000 7f4707fff640 1 --2- 172.21.15.16:0/197047020 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f471015f700 0x7f47100b5290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:57.933 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.923+0000 7f4714b33640 1 --2- 172.21.15.16:0/197047020 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4710160080 0x7f4710152370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:57.933 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.923+0000 7f470ffff640 1 -- 172.21.15.16:0/197047020 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f471015f700 msgr2=0x7f47100b5290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:57.933 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.923+0000 7f470ffff640 1 --2- 172.21.15.16:0/197047020 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f471015f700 0x7f47100b5290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:57.933 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.923+0000 7f470ffff640 1 -- 172.21.15.16:0/197047020 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4710160080 msgr2=0x7f4710152370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:57.933 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.923+0000 7f470ffff640 1 --2- 172.21.15.16:0/197047020 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4710160080 0x7f4710152370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:57.934 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.923+0000 7f470ffff640 1 -- 172.21.15.16:0/197047020 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47000094d0 con 0x7f47100b7dd0 2023-05-22T15:27:57.934 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.923+0000 7f4714b33640 1 --2- 172.21.15.16:0/197047020 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4710160080 0x7f4710152370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:27:57.934 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.923+0000 7f4707fff640 1 --2- 172.21.15.16:0/197047020 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f471015f700 0x7f47100b5290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:27:57.934 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.923+0000 7f470ffff640 1 --2- 172.21.15.16:0/197047020 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f47100b7dd0 0x7f471015f070 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f47000190e0 tx=0x7f470001ad10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:57.935 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.923+0000 7f470d7fa640 1 -- 172.21.15.16:0/197047020 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47000b4800 con 0x7f47100b7dd0 2023-05-22T15:27:57.935 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.923+0000 7f470d7fa640 1 -- 172.21.15.16:0/197047020 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47000b4980 con 0x7f47100b7dd0 2023-05-22T15:27:57.935 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.923+0000 7f4716596640 1 -- 172.21.15.16:0/197047020 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4710168a20 con 0x7f47100b7dd0 2023-05-22T15:27:57.935 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.923+0000 7f470d7fa640 1 -- 172.21.15.16:0/197047020 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47000b4d30 con 0x7f47100b7dd0 2023-05-22T15:27:57.935 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.923+0000 7f4716596640 1 -- 172.21.15.16:0/197047020 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4710168890 con 0x7f47100b7dd0 2023-05-22T15:27:57.935 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.923+0000 7f4716596640 1 -- 172.21.15.16:0/197047020 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4710168bb0 con 0x7f47100b7dd0 2023-05-22T15:27:57.936 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.923+0000 7f470d7fa640 1 -- 172.21.15.16:0/197047020 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7f47000b3380 con 0x7f47100b7dd0 2023-05-22T15:27:57.937 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.923+0000 7f470d7fa640 1 --2- 172.21.15.16:0/197047020 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f46f0065380 0x7f46f0067830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:57.937 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.923+0000 7f470d7fa640 1 -- 172.21.15.16:0/197047020 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f470011d030 con 0x7f47100b7dd0 2023-05-22T15:27:57.937 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.923+0000 7f4707fff640 1 --2- 172.21.15.16:0/197047020 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f46f0065380 0x7f46f0067830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:57.937 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.927+0000 7f4707fff640 1 --2- 172.21.15.16:0/197047020 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f46f0065380 0x7f46f0067830 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f47100c9bc0 tx=0x7f46f8008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:57.937 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:57.927+0000 7f470d7fa640 1 -- 172.21.15.16:0/197047020 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f47000ea9d0 con 0x7f47100b7dd0 2023-05-22T15:27:58.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.115+0000 7f4716596640 1 -- 172.21.15.16:0/197047020 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f471004dbe0 con 0x7f47100b7dd0 2023-05-22T15:27:58.126 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.119+0000 7f470d7fa640 1 -- 172.21.15.16:0/197047020 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4641 (secure 0 0 0) 0x7f47000edff0 con 0x7f47100b7dd0 2023-05-22T15:27:58.126 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:27:58.127 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4261,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6834/3661808159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":3661808159},{"type":"v1","addr":"172.21.15.62:6834","nonce":3661808159}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4264,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/2227467222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6833","nonce":2227467222},{"type":"v1","addr":"172.21.15.62:6835","nonce":2227467222}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4284,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6836/2808106087","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2808106087},{"type":"v1","addr":"172.21.15.16:6836","nonce":2808106087}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-05-22T15:26:49.546008+0000","modified":"2023-05-22T15:27:06.047204+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4287},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4287":{"gid":4287,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.16:6837/283322289","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6835","nonce":283322289},{"type":"v1","addr":"172.21.15.16:6837","nonce":283322289}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-05-22T15:27:58.128 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 8 2023-05-22T15:27:58.128 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.119+0000 7f4716596640 1 -- 172.21.15.16:0/197047020 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f46f0065380 msgr2=0x7f46f0067830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:58.130 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.119+0000 7f4716596640 1 --2- 172.21.15.16:0/197047020 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f46f0065380 0x7f46f0067830 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f47100c9bc0 tx=0x7f46f8008040 comp rx=0 tx=0).stop 2023-05-22T15:27:58.130 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.119+0000 7f4716596640 1 -- 172.21.15.16:0/197047020 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f47100b7dd0 msgr2=0x7f471015f070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:58.130 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.119+0000 7f4716596640 1 --2- 172.21.15.16:0/197047020 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f47100b7dd0 0x7f471015f070 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f47000190e0 tx=0x7f470001ad10 comp rx=0 tx=0).stop 2023-05-22T15:27:58.130 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.119+0000 7f4716596640 1 -- 172.21.15.16:0/197047020 shutdown_connections 2023-05-22T15:27:58.131 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.119+0000 7f4716596640 1 --2- 172.21.15.16:0/197047020 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4710160080 0x7f4710152370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:58.131 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.119+0000 7f4716596640 1 --2- 172.21.15.16:0/197047020 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f46f0065380 0x7f46f0067830 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:58.131 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.119+0000 7f4716596640 1 --2- 172.21.15.16:0/197047020 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f471015f700 0x7f47100b5290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:58.132 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.119+0000 7f4716596640 1 --2- 172.21.15.16:0/197047020 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f47100b7dd0 0x7f471015f070 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:58.132 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.119+0000 7f4716596640 1 -- 172.21.15.16:0/197047020 >> 172.21.15.16:0/197047020 conn(0x7f47100655c0 msgr2=0x7f4710166080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:58.132 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.119+0000 7f4716596640 1 -- 172.21.15.16:0/197047020 shutdown_connections 2023-05-22T15:27:58.132 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.119+0000 7f4716596640 1 -- 172.21.15.16:0/197047020 wait complete. 2023-05-22T15:27:58.145 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-05-22T15:27:58.261 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.255+0000 7fe6695eb640 1 Processor -- start 2023-05-22T15:27:58.262 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.255+0000 7fe6695eb640 1 -- start start 2023-05-22T15:27:58.263 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.255+0000 7fe6695eb640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe6640ca4e0 0x7fe66415e460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:58.263 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.255+0000 7fe6695eb640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fe6640c9b40 0x7fe6640afb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:58.263 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.255+0000 7fe6695eb640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6640b0650 con 0x7fe664070df0 2023-05-22T15:27:58.263 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.255+0000 7fe6695eb640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6640b0370 con 0x7fe6640c9b40 2023-05-22T15:27:58.264 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.255+0000 7fe6695eb640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe66415d980 con 0x7fe6640ca4e0 2023-05-22T15:27:58.265 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.255+0000 7fe6637fe640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fe664070df0 0x7fe664071210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:33082/0 (socket says 172.21.15.16:33082) 2023-05-22T15:27:58.265 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.255+0000 7fe6637fe640 1 -- 172.21.15.16:0/287770446 learned_addr learned my addr 172.21.15.16:0/287770446 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:27:58.265 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.255+0000 7fe662ffd640 1 --2- 172.21.15.16:0/287770446 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe6640ca4e0 0x7fe66415e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:58.265 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.255+0000 7fe6627fc640 1 --2- 172.21.15.16:0/287770446 >> v2:172.21.15.62:3300/0 conn(0x7fe6640c9b40 0x7fe6640afb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:58.265 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.255+0000 7fe661ffb640 1 -- 172.21.15.16:0/287770446 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 811622767 0 0) 0x7fe6640b0650 con 0x7fe664070df0 2023-05-22T15:27:58.265 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.255+0000 7fe661ffb640 1 -- 172.21.15.16:0/287770446 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe650003530 con 0x7fe664070df0 2023-05-22T15:27:58.267 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.255+0000 7fe661ffb640 1 -- 172.21.15.16:0/287770446 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 691760278 0 0) 0x7fe650003530 con 0x7fe664070df0 2023-05-22T15:27:58.267 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.255+0000 7fe661ffb640 1 -- 172.21.15.16:0/287770446 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe6640ca4e0 msgr2=0x7fe66415e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:58.268 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.255+0000 7fe661ffb640 1 --2- 172.21.15.16:0/287770446 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe6640ca4e0 0x7fe66415e460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:58.268 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.255+0000 7fe661ffb640 1 -- 172.21.15.16:0/287770446 >> v2:172.21.15.62:3300/0 conn(0x7fe6640c9b40 msgr2=0x7fe6640afb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:58.268 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.255+0000 7fe661ffb640 1 --2- 172.21.15.16:0/287770446 >> v2:172.21.15.62:3300/0 conn(0x7fe6640c9b40 0x7fe6640afb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:58.268 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.255+0000 7fe661ffb640 1 -- 172.21.15.16:0/287770446 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe66415dc60 con 0x7fe664070df0 2023-05-22T15:27:58.268 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.255+0000 7fe661ffb640 1 -- 172.21.15.16:0/287770446 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7fe654001680 con 0x7fe664070df0 2023-05-22T15:27:58.268 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.255+0000 7fe661ffb640 1 -- 172.21.15.16:0/287770446 >> v1:172.21.15.16:6789/0 conn(0x7fe664070df0 legacy=0x7fe664071210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:58.268 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.255+0000 7fe661ffb640 1 --2- 172.21.15.16:0/287770446 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe650003980 0x7fe650003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:58.269 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.255+0000 7fe661ffb640 1 --2- 172.21.15.16:0/287770446 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe6500043b0 0x7fe650004870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:58.269 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.255+0000 7fe661ffb640 1 --2- 172.21.15.16:0/287770446 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe6640ca4e0 0x7fe650004fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:58.269 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.255+0000 7fe661ffb640 1 -- 172.21.15.16:0/287770446 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6640b0370 con 0x7fe6500043b0 2023-05-22T15:27:58.269 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.255+0000 7fe661ffb640 1 -- 172.21.15.16:0/287770446 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe66415d980 con 0x7fe650003980 2023-05-22T15:27:58.269 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.255+0000 7fe661ffb640 1 -- 172.21.15.16:0/287770446 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe65000b990 con 0x7fe6640ca4e0 2023-05-22T15:27:58.269 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.255+0000 7fe6627fc640 1 --2- 172.21.15.16:0/287770446 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe6500043b0 0x7fe650004870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:58.269 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.255+0000 7fe662ffd640 1 --2- 172.21.15.16:0/287770446 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe650003980 0x7fe650003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:58.270 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.255+0000 7fe6627fc640 1 -- 172.21.15.16:0/287770446 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe6640ca4e0 msgr2=0x7fe650004fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:58.270 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.255+0000 7fe6627fc640 1 --2- 172.21.15.16:0/287770446 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe6640ca4e0 0x7fe650004fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:58.270 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.255+0000 7fe6627fc640 1 -- 172.21.15.16:0/287770446 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe650003980 msgr2=0x7fe650003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:58.270 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.255+0000 7fe6627fc640 1 --2- 172.21.15.16:0/287770446 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe650003980 0x7fe650003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:58.270 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.255+0000 7fe6627fc640 1 -- 172.21.15.16:0/287770446 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe650003530 con 0x7fe6500043b0 2023-05-22T15:27:58.270 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.259+0000 7fe6627fc640 1 --2- 172.21.15.16:0/287770446 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe6500043b0 0x7fe650004870 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fe64c00e170 tx=0x7fe64c01cc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85e4f1145b8d40aa server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:58.270 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.259+0000 7fe661ffb640 1 -- 172.21.15.16:0/287770446 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe64c0a5b50 con 0x7fe6500043b0 2023-05-22T15:27:58.270 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.259+0000 7fe661ffb640 1 -- 172.21.15.16:0/287770446 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe64c00f040 con 0x7fe6500043b0 2023-05-22T15:27:58.271 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.259+0000 7fe661ffb640 1 -- 172.21.15.16:0/287770446 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe64c0ad440 con 0x7fe6500043b0 2023-05-22T15:27:58.271 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.259+0000 7fe6695eb640 1 -- 172.21.15.16:0/287770446 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe6500043b0 msgr2=0x7fe650004870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:58.271 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.259+0000 7fe6695eb640 1 --2- 172.21.15.16:0/287770446 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe6500043b0 0x7fe650004870 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fe64c00e170 tx=0x7fe64c01cc50 comp rx=0 tx=0).stop 2023-05-22T15:27:58.271 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.259+0000 7fe6637fe640 1 -- 172.21.15.16:0/287770446 reap_dead start 2023-05-22T15:27:58.271 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.259+0000 7fe6695eb640 1 -- 172.21.15.16:0/287770446 shutdown_connections 2023-05-22T15:27:58.271 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.259+0000 7fe6695eb640 1 -- 172.21.15.16:0/287770446 >> 172.21.15.16:0/287770446 conn(0x7fe664069720 msgr2=0x7fe664069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:58.271 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.259+0000 7fe6695eb640 1 -- 172.21.15.16:0/287770446 shutdown_connections 2023-05-22T15:27:58.271 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.259+0000 7fe6695eb640 1 -- 172.21.15.16:0/287770446 wait complete. 2023-05-22T15:27:58.274 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.259+0000 7fe6695eb640 1 Processor -- start 2023-05-22T15:27:58.274 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.259+0000 7fe6695eb640 1 -- start start 2023-05-22T15:27:58.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.263+0000 7fe6695eb640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe6500043b0 0x7fe664145540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:58.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.263+0000 7fe6695eb640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe664069b20 0x7fe664069f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:58.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.263+0000 7fe6695eb640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe6640c2ef0 0x7fe6640c3360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:58.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.263+0000 7fe6695eb640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe66406a4d0 con 0x7fe664069b20 2023-05-22T15:27:58.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.263+0000 7fe6695eb640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6640c4880 con 0x7fe6640c2ef0 2023-05-22T15:27:58.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.263+0000 7fe6695eb640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6640c3c70 con 0x7fe6500043b0 2023-05-22T15:27:58.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.263+0000 7fe6627fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe664069b20 0x7fe664069f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:58.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.263+0000 7fe6627fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe664069b20 0x7fe664069f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54458/0 (socket says 172.21.15.16:54458) 2023-05-22T15:27:58.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.263+0000 7fe6627fc640 1 -- 172.21.15.16:0/2698055224 learned_addr learned my addr 172.21.15.16:0/2698055224 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:58.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.263+0000 7fe662ffd640 1 --2- 172.21.15.16:0/2698055224 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe6500043b0 0x7fe664145540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:58.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.263+0000 7fe6637fe640 1 --2- 172.21.15.16:0/2698055224 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe6640c2ef0 0x7fe6640c3360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:58.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.263+0000 7fe6627fc640 1 -- 172.21.15.16:0/2698055224 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe6500043b0 msgr2=0x7fe664145540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:58.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.263+0000 7fe6627fc640 1 --2- 172.21.15.16:0/2698055224 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe6500043b0 0x7fe664145540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:58.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.263+0000 7fe6627fc640 1 -- 172.21.15.16:0/2698055224 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe6640c2ef0 msgr2=0x7fe6640c3360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:58.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.263+0000 7fe6627fc640 1 --2- 172.21.15.16:0/2698055224 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe6640c2ef0 0x7fe6640c3360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:58.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.263+0000 7fe6627fc640 1 -- 172.21.15.16:0/2698055224 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6640c8e80 con 0x7fe664069b20 2023-05-22T15:27:58.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.263+0000 7fe6637fe640 1 --2- 172.21.15.16:0/2698055224 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe6640c2ef0 0x7fe6640c3360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:27:58.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.263+0000 7fe6627fc640 1 --2- 172.21.15.16:0/2698055224 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe664069b20 0x7fe664069f90 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fe64c0b0040 tx=0x7fe64c009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:58.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.263+0000 7fe65b7fe640 1 -- 172.21.15.16:0/2698055224 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe64c0ad700 con 0x7fe664069b20 2023-05-22T15:27:58.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.263+0000 7fe65b7fe640 1 -- 172.21.15.16:0/2698055224 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe64c0ad860 con 0x7fe664069b20 2023-05-22T15:27:58.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.263+0000 7fe6695eb640 1 -- 172.21.15.16:0/2698055224 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6640ca8b0 con 0x7fe664069b20 2023-05-22T15:27:58.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.263+0000 7fe65b7fe640 1 -- 172.21.15.16:0/2698055224 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe64c0adc10 con 0x7fe664069b20 2023-05-22T15:27:58.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.263+0000 7fe6695eb640 1 -- 172.21.15.16:0/2698055224 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6640c9010 con 0x7fe664069b20 2023-05-22T15:27:58.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.263+0000 7fe6695eb640 1 -- 172.21.15.16:0/2698055224 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe66404dbe0 con 0x7fe664069b20 2023-05-22T15:27:58.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.263+0000 7fe65b7fe640 1 -- 172.21.15.16:0/2698055224 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7fe64c0b4040 con 0x7fe664069b20 2023-05-22T15:27:58.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.263+0000 7fe65b7fe640 1 --2- 172.21.15.16:0/2698055224 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fe644065380 0x7fe644067830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:58.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.263+0000 7fe65b7fe640 1 -- 172.21.15.16:0/2698055224 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6871+0+0 (secure 0 0 0) 0x7fe64c11d310 con 0x7fe664069b20 2023-05-22T15:27:58.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.263+0000 7fe662ffd640 1 --2- 172.21.15.16:0/2698055224 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fe644065380 0x7fe644067830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:58.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.267+0000 7fe662ffd640 1 --2- 172.21.15.16:0/2698055224 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fe644065380 0x7fe644067830 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fe65c009950 tx=0x7fe65c008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:58.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.267+0000 7fe65b7fe640 1 -- 172.21.15.16:0/2698055224 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fe64c0eabc0 con 0x7fe664069b20 2023-05-22T15:27:58.429 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.423+0000 7fe6695eb640 1 -- 172.21.15.16:0/2698055224 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe6640cb010 con 0x7fe664069b20 2023-05-22T15:27:58.432 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.423+0000 7fe65b7fe640 1 -- 172.21.15.16:0/2698055224 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v29) v1 ==== 74+0+19700 (secure 0 0 0) 0x7fe64c0ee1e0 con 0x7fe664069b20 2023-05-22T15:27:58.432 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:27:58.433 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":29,"fsid":"b2719834-7847-4d98-90a6-f65d57f85c49","created":"2023-05-22T15:26:25.259314+0000","modified":"2023-05-22T15:27:31.182317+0000","last_up_change":"2023-05-22T15:26:36.448524+0000","last_in_change":"2023-05-22T15:26:27.675668+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-05-22T15:26:38.019781+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-05-22T15:26:43.570514+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-05-22T15:26:47.879159+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-05-22T15:26:48.522130+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data_ec","create_time":"2023-05-22T15:26:51.616142+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"9e7ff854-b25f-4cbb-a5c2-1799d9c53d2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":37021},{"type":"v1","addr":"172.21.15.16:6825","nonce":37021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":37021},{"type":"v1","addr":"172.21.15.16:6827","nonce":37021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":37021},{"type":"v1","addr":"172.21.15.16:6831","nonce":37021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":37021},{"type":"v1","addr":"172.21.15.16:6829","nonce":37021}]},"public_addr":"172.21.15.16:6825/37021","cluster_addr":"172.21.15.16:6827/37021","heartbeat_back_addr":"172.21.15.16:6831/37021","heartbeat_front_addr":"172.21.15.16:6829/37021","state":["exists","up"]},{"osd":1,"uuid":"4702fbc9-a927-4565-aca5-41dc69cc8264","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":37008},{"type":"v1","addr":"172.21.15.16:6801","nonce":37008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":37008},{"type":"v1","addr":"172.21.15.16:6803","nonce":37008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":37008},{"type":"v1","addr":"172.21.15.16:6807","nonce":37008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":37008},{"type":"v1","addr":"172.21.15.16:6805","nonce":37008}]},"public_addr":"172.21.15.16:6801/37008","cluster_addr":"172.21.15.16:6803/37008","heartbeat_back_addr":"172.21.15.16:6807/37008","heartbeat_front_addr":"172.21.15.16:6805/37008","state":["exists","up"]},{"osd":2,"uuid":"1e7bba46-4eba-47ef-bb81-5e2d371638ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":37009},{"type":"v1","addr":"172.21.15.16:6809","nonce":37009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":37009},{"type":"v1","addr":"172.21.15.16:6811","nonce":37009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":37009},{"type":"v1","addr":"172.21.15.16:6815","nonce":37009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":37009},{"type":"v1","addr":"172.21.15.16:6813","nonce":37009}]},"public_addr":"172.21.15.16:6809/37009","cluster_addr":"172.21.15.16:6811/37009","heartbeat_back_addr":"172.21.15.16:6815/37009","heartbeat_front_addr":"172.21.15.16:6813/37009","state":["exists","up"]},{"osd":3,"uuid":"383cacc0-5a88-4904-990e-498ca59ef48a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":37020},{"type":"v1","addr":"172.21.15.16:6817","nonce":37020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":37020},{"type":"v1","addr":"172.21.15.16:6819","nonce":37020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":37020},{"type":"v1","addr":"172.21.15.16:6823","nonce":37020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":37020},{"type":"v1","addr":"172.21.15.16:6821","nonce":37020}]},"public_addr":"172.21.15.16:6817/37020","cluster_addr":"172.21.15.16:6819/37020","heartbeat_back_addr":"172.21.15.16:6823/37020","heartbeat_front_addr":"172.21.15.16:6821/37020","state":["exists","up"]},{"osd":4,"uuid":"970500ef-95b7-42c1-b080-eed4fae7013d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":37353},{"type":"v1","addr":"172.21.15.62:6817","nonce":37353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6819","nonce":37353},{"type":"v1","addr":"172.21.15.62:6822","nonce":37353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37353},{"type":"v1","addr":"172.21.15.62:6831","nonce":37353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37353},{"type":"v1","addr":"172.21.15.62:6826","nonce":37353}]},"public_addr":"172.21.15.62:6817/37353","cluster_addr":"172.21.15.62:6822/37353","heartbeat_back_addr":"172.21.15.62:6831/37353","heartbeat_front_addr":"172.21.15.62:6826/37353","state":["exists","up"]},{"osd":5,"uuid":"022d67db-d4d8-445b-b1b9-8b446a00f5c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":37352},{"type":"v1","addr":"172.21.15.62:6801","nonce":37352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":37352},{"type":"v1","addr":"172.21.15.62:6803","nonce":37352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":37352},{"type":"v1","addr":"172.21.15.62:6807","nonce":37352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":37352},{"type":"v1","addr":"172.21.15.62:6805","nonce":37352}]},"public_addr":"172.21.15.62:6801/37352","cluster_addr":"172.21.15.62:6803/37352","heartbeat_back_addr":"172.21.15.62:6807/37352","heartbeat_front_addr":"172.21.15.62:6805/37352","state":["exists","up"]},{"osd":6,"uuid":"42bd372b-6688-4a1b-8407-e16e8c266e33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":37354},{"type":"v1","addr":"172.21.15.62:6820","nonce":37354}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":37354},{"type":"v1","addr":"172.21.15.62:6823","nonce":37354}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6829","nonce":37354},{"type":"v1","addr":"172.21.15.62:6830","nonce":37354}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6825","nonce":37354},{"type":"v1","addr":"172.21.15.62:6827","nonce":37354}]},"public_addr":"172.21.15.62:6820/37354","cluster_addr":"172.21.15.62:6823/37354","heartbeat_back_addr":"172.21.15.62:6830/37354","heartbeat_front_addr":"172.21.15.62:6827/37354","state":["exists","up"]},{"osd":7,"uuid":"ac70b2a3-f41b-4bf2-b227-7f2cbb55426d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37355},{"type":"v1","addr":"172.21.15.62:6809","nonce":37355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37355},{"type":"v1","addr":"172.21.15.62:6811","nonce":37355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":37355},{"type":"v1","addr":"172.21.15.62:6815","nonce":37355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":37355},{"type":"v1","addr":"172.21.15.62:6813","nonce":37355}]},"public_addr":"172.21.15.62:6809/37355","cluster_addr":"172.21.15.62:6811/37355","heartbeat_back_addr":"172.21.15.62:6815/37355","heartbeat_front_addr":"172.21.15.62:6813/37355","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.368896+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.107273+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.276888+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.287502+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.124791+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.251937+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.255679+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.232797+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"4.23","mappings":[{"from":3,"to":2}]},{"pgid":"4.31","mappings":[{"from":1,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-05-22T15:27:58.434 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.427+0000 7fe6695eb640 1 -- 172.21.15.16:0/2698055224 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fe644065380 msgr2=0x7fe644067830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:58.435 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.427+0000 7fe6695eb640 1 --2- 172.21.15.16:0/2698055224 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fe644065380 0x7fe644067830 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fe65c009950 tx=0x7fe65c008040 comp rx=0 tx=0).stop 2023-05-22T15:27:58.435 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.427+0000 7fe6695eb640 1 -- 172.21.15.16:0/2698055224 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe664069b20 msgr2=0x7fe664069f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:58.435 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.427+0000 7fe6695eb640 1 --2- 172.21.15.16:0/2698055224 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe664069b20 0x7fe664069f90 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fe64c0b0040 tx=0x7fe64c009750 comp rx=0 tx=0).stop 2023-05-22T15:27:58.435 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.427+0000 7fe6695eb640 1 -- 172.21.15.16:0/2698055224 shutdown_connections 2023-05-22T15:27:58.435 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.427+0000 7fe6695eb640 1 --2- 172.21.15.16:0/2698055224 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe6640c2ef0 0x7fe6640c3360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:58.436 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.427+0000 7fe6695eb640 1 --2- 172.21.15.16:0/2698055224 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe664069b20 0x7fe664069f90 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:58.436 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.427+0000 7fe6695eb640 1 --2- 172.21.15.16:0/2698055224 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fe644065380 0x7fe644067830 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:58.436 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.427+0000 7fe6695eb640 1 --2- 172.21.15.16:0/2698055224 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe6500043b0 0x7fe664145540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:58.436 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.427+0000 7fe6695eb640 1 -- 172.21.15.16:0/2698055224 >> 172.21.15.16:0/2698055224 conn(0x7fe664069720 msgr2=0x7fe6641594f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:58.436 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.427+0000 7fe6695eb640 1 -- 172.21.15.16:0/2698055224 shutdown_connections 2023-05-22T15:27:58.436 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.427+0000 7fe6695eb640 1 -- 172.21.15.16:0/2698055224 wait complete. 2023-05-22T15:27:58.449 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-05-22T15:26:38.019781+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-05-22T15:26:43.570514+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2023-05-22T15:26:47.879159+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2023-05-22T15:26:48.522130+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-05-22T15:26:51.616142+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '27', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-05-22T15:27:58.451 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-05-22T15:27:58.568 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.559+0000 7fee8eec3640 1 Processor -- start 2023-05-22T15:27:58.568 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.559+0000 7fee8eec3640 1 -- start start 2023-05-22T15:27:58.569 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.563+0000 7fee8eec3640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fee8806b6c0 0x7fee880cb4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:58.569 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.563+0000 7fee8eec3640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fee881596e0 0x7fee8806fba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:58.570 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.563+0000 7fee8eec3640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee88159f70 con 0x7fee8806ba90 2023-05-22T15:27:58.570 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.563+0000 7fee8eec3640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee88071760 con 0x7fee881596e0 2023-05-22T15:27:58.570 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.563+0000 7fee8eec3640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee880ca7d0 con 0x7fee8806b6c0 2023-05-22T15:27:58.570 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.563+0000 7fee8d460640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fee881596e0 0x7fee8806fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:58.571 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.563+0000 7fee8d460640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fee881596e0 0x7fee8806fba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54470/0 (socket says 172.21.15.16:54470) 2023-05-22T15:27:58.571 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.563+0000 7fee8d460640 1 -- 172.21.15.16:0/4228106226 learned_addr learned my addr 172.21.15.16:0/4228106226 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:58.571 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.563+0000 7fee87fff640 1 --1- 172.21.15.16:0/4228106226 >> v1:172.21.15.16:6789/0 conn(0x7fee8806ba90 0x7fee88158fd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:33086/0 (socket says 172.21.15.16:33086) 2023-05-22T15:27:58.572 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.563+0000 7fee8d460640 1 -- 172.21.15.16:0/4228106226 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fee8806b6c0 msgr2=0x7fee880cb4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:58.572 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.563+0000 7fee8d460640 1 --2- 172.21.15.16:0/4228106226 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fee8806b6c0 0x7fee880cb4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:58.572 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.563+0000 7fee8d460640 1 -- 172.21.15.16:0/4228106226 >> v1:172.21.15.16:6789/0 conn(0x7fee8806ba90 legacy=0x7fee88158fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:58.572 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.563+0000 7fee8d460640 1 -- 172.21.15.16:0/4228106226 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee880cb030 con 0x7fee881596e0 2023-05-22T15:27:58.573 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.563+0000 7fee8d460640 1 --2- 172.21.15.16:0/4228106226 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fee881596e0 0x7fee8806fba0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7fee8000d150 tx=0x7fee8001eb50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e527514d5c0ee02 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:58.573 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.563+0000 7fee877fe640 1 -- 172.21.15.16:0/4228106226 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee8000b2d0 con 0x7fee881596e0 2023-05-22T15:27:58.573 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.563+0000 7fee877fe640 1 -- 172.21.15.16:0/4228106226 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee8000b430 con 0x7fee881596e0 2023-05-22T15:27:58.574 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.563+0000 7fee877fe640 1 -- 172.21.15.16:0/4228106226 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee800a7c40 con 0x7fee881596e0 2023-05-22T15:27:58.574 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.563+0000 7fee8eec3640 1 -- 172.21.15.16:0/4228106226 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fee881596e0 msgr2=0x7fee8806fba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:58.574 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.563+0000 7fee8eec3640 1 --2- 172.21.15.16:0/4228106226 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fee881596e0 0x7fee8806fba0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7fee8000d150 tx=0x7fee8001eb50 comp rx=0 tx=0).stop 2023-05-22T15:27:58.574 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.563+0000 7fee8eec3640 1 -- 172.21.15.16:0/4228106226 shutdown_connections 2023-05-22T15:27:58.574 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.563+0000 7fee8eec3640 1 --2- 172.21.15.16:0/4228106226 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fee881596e0 0x7fee8806fba0 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:58.574 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.563+0000 7fee8eec3640 1 --2- 172.21.15.16:0/4228106226 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fee8806b6c0 0x7fee880cb4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:58.574 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.563+0000 7fee8eec3640 1 -- 172.21.15.16:0/4228106226 >> 172.21.15.16:0/4228106226 conn(0x7fee88069720 msgr2=0x7fee88069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:58.575 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.563+0000 7fee8eec3640 1 -- 172.21.15.16:0/4228106226 shutdown_connections 2023-05-22T15:27:58.575 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.563+0000 7fee8eec3640 1 -- 172.21.15.16:0/4228106226 wait complete. 2023-05-22T15:27:58.576 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.567+0000 7fee8eec3640 1 Processor -- start 2023-05-22T15:27:58.576 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.567+0000 7fee8eec3640 1 -- start start 2023-05-22T15:27:58.577 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.567+0000 7fee8eec3640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fee8806b6c0 0x7fee8814abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:58.577 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.567+0000 7fee8eec3640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fee8806ba90 0x7fee880adab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:58.577 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.567+0000 7fee8eec3640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fee881596e0 0x7fee880adff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:58.577 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.567+0000 7fee8eec3640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee880b2180 con 0x7fee881596e0 2023-05-22T15:27:58.577 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.567+0000 7fee8eec3640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee880b35e0 con 0x7fee8806ba90 2023-05-22T15:27:58.577 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.567+0000 7fee8eec3640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee880b1850 con 0x7fee8806b6c0 2023-05-22T15:27:58.578 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.567+0000 7fee8d460640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fee881596e0 0x7fee880adff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:58.578 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.567+0000 7fee8d460640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fee881596e0 0x7fee880adff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54472/0 (socket says 172.21.15.16:54472) 2023-05-22T15:27:58.578 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.567+0000 7fee8d460640 1 -- 172.21.15.16:0/2323299903 learned_addr learned my addr 172.21.15.16:0/2323299903 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:58.578 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.567+0000 7fee8cc5f640 1 --2- 172.21.15.16:0/2323299903 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fee8806b6c0 0x7fee8814abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:58.578 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.567+0000 7fee87fff640 1 --2- 172.21.15.16:0/2323299903 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fee8806ba90 0x7fee880adab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:58.578 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.567+0000 7fee8d460640 1 -- 172.21.15.16:0/2323299903 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fee8806b6c0 msgr2=0x7fee8814abd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:58.578 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.567+0000 7fee8d460640 1 --2- 172.21.15.16:0/2323299903 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fee8806b6c0 0x7fee8814abd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:58.579 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.567+0000 7fee8d460640 1 -- 172.21.15.16:0/2323299903 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fee8806ba90 msgr2=0x7fee880adab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:58.579 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.567+0000 7fee8d460640 1 --2- 172.21.15.16:0/2323299903 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fee8806ba90 0x7fee880adab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:58.579 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.567+0000 7fee8d460640 1 -- 172.21.15.16:0/2323299903 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee8000bd40 con 0x7fee881596e0 2023-05-22T15:27:58.579 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.567+0000 7fee87fff640 1 --2- 172.21.15.16:0/2323299903 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fee8806ba90 0x7fee880adab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:27:58.579 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.567+0000 7fee8d460640 1 --2- 172.21.15.16:0/2323299903 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fee881596e0 0x7fee880adff0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fee8001c780 tx=0x7fee8001b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:58.580 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.567+0000 7fee857fa640 1 -- 172.21.15.16:0/2323299903 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee80009800 con 0x7fee881596e0 2023-05-22T15:27:58.580 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.567+0000 7fee857fa640 1 -- 172.21.15.16:0/2323299903 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee80009960 con 0x7fee881596e0 2023-05-22T15:27:58.580 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.567+0000 7fee857fa640 1 -- 172.21.15.16:0/2323299903 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee80009d10 con 0x7fee881596e0 2023-05-22T15:27:58.580 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.567+0000 7fee8eec3640 1 -- 172.21.15.16:0/2323299903 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee8806be60 con 0x7fee881596e0 2023-05-22T15:27:58.580 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.567+0000 7fee8eec3640 1 -- 172.21.15.16:0/2323299903 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee880cacc0 con 0x7fee881596e0 2023-05-22T15:27:58.580 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.571+0000 7fee8eec3640 1 -- 172.21.15.16:0/2323299903 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee8806bff0 con 0x7fee881596e0 2023-05-22T15:27:58.582 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.571+0000 7fee857fa640 1 -- 172.21.15.16:0/2323299903 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7fee800ba440 con 0x7fee881596e0 2023-05-22T15:27:58.582 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.571+0000 7fee857fa640 1 --2- 172.21.15.16:0/2323299903 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fee74065380 0x7fee74067830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:58.582 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.571+0000 7fee8cc5f640 1 --2- 172.21.15.16:0/2323299903 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fee74065380 0x7fee74067830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:58.582 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.571+0000 7fee857fa640 1 -- 172.21.15.16:0/2323299903 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6871+0+0 (secure 0 0 0) 0x7fee80120300 con 0x7fee881596e0 2023-05-22T15:27:58.582 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.571+0000 7fee8cc5f640 1 --2- 172.21.15.16:0/2323299903 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fee74065380 0x7fee74067830 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fee78006fd0 tx=0x7fee78008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:58.583 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.575+0000 7fee857fa640 1 -- 172.21.15.16:0/2323299903 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fee800edb60 con 0x7fee881596e0 2023-05-22T15:27:58.731 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.723+0000 7fee8eec3640 1 -- 172.21.15.16:0/2323299903 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fee8804dbe0 con 0x7fee881596e0 2023-05-22T15:27:58.732 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.723+0000 7fee857fa640 1 -- 172.21.15.16:0/2323299903 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v29) v1 ==== 93+0+10 (secure 0 0 0) 0x7fee800f1180 con 0x7fee881596e0 2023-05-22T15:27:58.732 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2023-05-22T15:27:58.734 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.727+0000 7fee8eec3640 1 -- 172.21.15.16:0/2323299903 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fee74065380 msgr2=0x7fee74067830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:58.735 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.727+0000 7fee8eec3640 1 --2- 172.21.15.16:0/2323299903 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fee74065380 0x7fee74067830 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fee78006fd0 tx=0x7fee78008040 comp rx=0 tx=0).stop 2023-05-22T15:27:58.735 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.727+0000 7fee8eec3640 1 -- 172.21.15.16:0/2323299903 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fee881596e0 msgr2=0x7fee880adff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:58.735 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.727+0000 7fee8eec3640 1 --2- 172.21.15.16:0/2323299903 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fee881596e0 0x7fee880adff0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fee8001c780 tx=0x7fee8001b8f0 comp rx=0 tx=0).stop 2023-05-22T15:27:58.735 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.727+0000 7fee8eec3640 1 -- 172.21.15.16:0/2323299903 shutdown_connections 2023-05-22T15:27:58.735 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.727+0000 7fee8eec3640 1 --2- 172.21.15.16:0/2323299903 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fee881596e0 0x7fee880adff0 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:58.735 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.727+0000 7fee8eec3640 1 --2- 172.21.15.16:0/2323299903 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fee8806ba90 0x7fee880adab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:58.736 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.727+0000 7fee8eec3640 1 --2- 172.21.15.16:0/2323299903 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fee74065380 0x7fee74067830 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:58.736 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.727+0000 7fee8eec3640 1 --2- 172.21.15.16:0/2323299903 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fee8806b6c0 0x7fee8814abd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:58.736 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.727+0000 7fee8eec3640 1 -- 172.21.15.16:0/2323299903 >> 172.21.15.16:0/2323299903 conn(0x7fee88069720 msgr2=0x7fee880cdfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:58.736 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.727+0000 7fee8eec3640 1 -- 172.21.15.16:0/2323299903 shutdown_connections 2023-05-22T15:27:58.736 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.727+0000 7fee8eec3640 1 -- 172.21.15.16:0/2323299903 wait complete. 2023-05-22T15:27:58.750 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-05-22T15:27:58.877 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.867+0000 7f8c4e468640 1 Processor -- start 2023-05-22T15:27:58.877 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.871+0000 7f8c4e468640 1 -- start start 2023-05-22T15:27:58.882 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.871+0000 7f8c4e468640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8c481596e0 0x7f8c48163950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:58.883 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.871+0000 7f8c4e468640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c48159f70 con 0x7f8c480cb4c0 2023-05-22T15:27:58.883 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.871+0000 7f8c4e468640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c480cb030 con 0x7f8c4806b6c0 2023-05-22T15:27:58.883 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.871+0000 7f8c4e468640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c48071760 con 0x7f8c481596e0 2023-05-22T15:27:58.884 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.871+0000 7f8c477fe640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f8c480cb4c0 0x7f8c48158fd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:33102/0 (socket says 172.21.15.16:33102) 2023-05-22T15:27:58.884 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.871+0000 7f8c477fe640 1 -- 172.21.15.16:0/1907838518 learned_addr learned my addr 172.21.15.16:0/1907838518 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:27:58.885 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.871+0000 7f8c4ca05640 1 --2- 172.21.15.16:0/1907838518 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8c481596e0 0x7f8c48163950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:58.885 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.871+0000 7f8c46ffd640 1 -- 172.21.15.16:0/1907838518 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1298457786 0 0) 0x7f8c48159f70 con 0x7f8c480cb4c0 2023-05-22T15:27:58.885 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.871+0000 7f8c46ffd640 1 -- 172.21.15.16:0/1907838518 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c34003530 con 0x7f8c480cb4c0 2023-05-22T15:27:58.885 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.871+0000 7f8c46ffd640 1 -- 172.21.15.16:0/1907838518 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2023612879 0 0) 0x7f8c34003530 con 0x7f8c480cb4c0 2023-05-22T15:27:58.885 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.871+0000 7f8c46ffd640 1 -- 172.21.15.16:0/1907838518 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8c481596e0 msgr2=0x7f8c48163950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:58.885 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.871+0000 7f8c46ffd640 1 --2- 172.21.15.16:0/1907838518 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8c481596e0 0x7f8c48163950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:58.885 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.871+0000 7f8c46ffd640 1 -- 172.21.15.16:0/1907838518 >> v1:172.21.15.62:6789/0 conn(0x7f8c4806b6c0 legacy=0x7f8c4806ba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:58.886 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.871+0000 7f8c46ffd640 1 -- 172.21.15.16:0/1907838518 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c48159c90 con 0x7f8c480cb4c0 2023-05-22T15:27:58.886 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.871+0000 7f8c46ffd640 1 -- 172.21.15.16:0/1907838518 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7f8c38001a00 con 0x7f8c480cb4c0 2023-05-22T15:27:58.886 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.871+0000 7f8c46ffd640 1 -- 172.21.15.16:0/1907838518 >> v1:172.21.15.16:6789/0 conn(0x7f8c480cb4c0 legacy=0x7f8c48158fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:58.886 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.871+0000 7f8c46ffd640 1 --2- 172.21.15.16:0/1907838518 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8c340039b0 0x7f8c34003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:58.886 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.871+0000 7f8c46ffd640 1 --2- 172.21.15.16:0/1907838518 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8c481596e0 0x7f8c340047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:58.886 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.871+0000 7f8c46ffd640 1 --2- 172.21.15.16:0/1907838518 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8c34004d10 0x7f8c34005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:58.886 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.871+0000 7f8c46ffd640 1 -- 172.21.15.16:0/1907838518 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c48071760 con 0x7f8c340039b0 2023-05-22T15:27:58.886 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.871+0000 7f8c46ffd640 1 -- 172.21.15.16:0/1907838518 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c3400dce0 con 0x7f8c34004d10 2023-05-22T15:27:58.887 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.871+0000 7f8c46ffd640 1 -- 172.21.15.16:0/1907838518 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c3400dea0 con 0x7f8c481596e0 2023-05-22T15:27:58.887 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.871+0000 7f8c47fff640 1 --2- 172.21.15.16:0/1907838518 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8c340039b0 0x7f8c34003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:58.887 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.871+0000 7f8c477fe640 1 --2- 172.21.15.16:0/1907838518 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8c481596e0 0x7f8c340047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:58.887 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.871+0000 7f8c4ca05640 1 --2- 172.21.15.16:0/1907838518 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8c34004d10 0x7f8c34005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:58.887 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.871+0000 7f8c47fff640 1 -- 172.21.15.16:0/1907838518 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8c481596e0 msgr2=0x7f8c340047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:58.887 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.871+0000 7f8c47fff640 1 --2- 172.21.15.16:0/1907838518 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8c481596e0 0x7f8c340047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:58.887 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.871+0000 7f8c47fff640 1 -- 172.21.15.16:0/1907838518 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8c34004d10 msgr2=0x7f8c34005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:58.888 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.871+0000 7f8c47fff640 1 --2- 172.21.15.16:0/1907838518 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8c34004d10 0x7f8c34005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:58.888 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.871+0000 7f8c47fff640 1 -- 172.21.15.16:0/1907838518 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c34003530 con 0x7f8c340039b0 2023-05-22T15:27:58.888 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.871+0000 7f8c4ca05640 1 --2- 172.21.15.16:0/1907838518 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8c34004d10 0x7f8c34005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:27:58.888 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.871+0000 7f8c477fe640 1 --2- 172.21.15.16:0/1907838518 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8c481596e0 0x7f8c340047d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:27:58.888 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.875+0000 7f8c47fff640 1 --2- 172.21.15.16:0/1907838518 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8c340039b0 0x7f8c34003dc0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f8c30001780 tx=0x7f8c3001c400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b87ae1ca352f257 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:58.888 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.875+0000 7f8c46ffd640 1 -- 172.21.15.16:0/1907838518 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c300052d0 con 0x7f8c340039b0 2023-05-22T15:27:58.888 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.875+0000 7f8c46ffd640 1 -- 172.21.15.16:0/1907838518 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c300a7050 con 0x7f8c340039b0 2023-05-22T15:27:58.889 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.875+0000 7f8c46ffd640 1 -- 172.21.15.16:0/1907838518 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c30003600 con 0x7f8c340039b0 2023-05-22T15:27:58.889 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.875+0000 7f8c4e468640 1 -- 172.21.15.16:0/1907838518 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8c340039b0 msgr2=0x7f8c34003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:58.889 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.875+0000 7f8c4e468640 1 --2- 172.21.15.16:0/1907838518 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8c340039b0 0x7f8c34003dc0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f8c30001780 tx=0x7f8c3001c400 comp rx=0 tx=0).stop 2023-05-22T15:27:58.889 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.875+0000 7f8c4ca05640 1 -- 172.21.15.16:0/1907838518 reap_dead start 2023-05-22T15:27:58.889 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.875+0000 7f8c4e468640 1 -- 172.21.15.16:0/1907838518 shutdown_connections 2023-05-22T15:27:58.889 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.875+0000 7f8c4e468640 1 -- 172.21.15.16:0/1907838518 >> 172.21.15.16:0/1907838518 conn(0x7f8c48069720 msgr2=0x7f8c48069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:58.889 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.875+0000 7f8c4e468640 1 -- 172.21.15.16:0/1907838518 shutdown_connections 2023-05-22T15:27:58.890 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.875+0000 7f8c4e468640 1 -- 172.21.15.16:0/1907838518 wait complete. 2023-05-22T15:27:58.892 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.875+0000 7f8c4e468640 1 Processor -- start 2023-05-22T15:27:58.892 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.875+0000 7f8c4e468640 1 -- start start 2023-05-22T15:27:58.892 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.875+0000 7f8c4e468640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8c340039b0 0x7f8c4815d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:58.892 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.875+0000 7f8c4e468640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8c4814cb60 0x7f8c4814c140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:58.892 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.875+0000 7f8c4e468640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8c4814c680 0x7f8c480c2050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:58.892 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.875+0000 7f8c4e468640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c4815fbd0 con 0x7f8c340039b0 2023-05-22T15:27:58.893 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.875+0000 7f8c4e468640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c481609f0 con 0x7f8c4814c680 2023-05-22T15:27:58.893 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.875+0000 7f8c4e468640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c4815dc30 con 0x7f8c4814cb60 2023-05-22T15:27:58.893 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.875+0000 7f8c47fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8c340039b0 0x7f8c4815d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:58.893 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.875+0000 7f8c47fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8c340039b0 0x7f8c4815d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54478/0 (socket says 172.21.15.16:54478) 2023-05-22T15:27:58.893 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.875+0000 7f8c47fff640 1 -- 172.21.15.16:0/3769703246 learned_addr learned my addr 172.21.15.16:0/3769703246 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:58.893 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.875+0000 7f8c477fe640 1 --2- 172.21.15.16:0/3769703246 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8c4814cb60 0x7f8c4814c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:58.893 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.875+0000 7f8c47fff640 1 -- 172.21.15.16:0/3769703246 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8c4814cb60 msgr2=0x7f8c4814c140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:58.894 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.875+0000 7f8c47fff640 1 --2- 172.21.15.16:0/3769703246 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8c4814cb60 0x7f8c4814c140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:58.894 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.875+0000 7f8c47fff640 1 -- 172.21.15.16:0/3769703246 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8c4814c680 msgr2=0x7f8c480c2050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:58.894 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.875+0000 7f8c4ca05640 1 --2- 172.21.15.16:0/3769703246 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8c4814c680 0x7f8c480c2050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:58.894 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.875+0000 7f8c47fff640 1 --2- 172.21.15.16:0/3769703246 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8c4814c680 0x7f8c480c2050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:58.895 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.875+0000 7f8c47fff640 1 -- 172.21.15.16:0/3769703246 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c481569d0 con 0x7f8c340039b0 2023-05-22T15:27:58.895 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.879+0000 7f8c47fff640 1 --2- 172.21.15.16:0/3769703246 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8c340039b0 0x7f8c4815d6f0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f8c30001780 tx=0x7f8c34005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:58.896 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.879+0000 7f8c44ff9640 1 -- 172.21.15.16:0/3769703246 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c30007040 con 0x7f8c340039b0 2023-05-22T15:27:58.896 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.879+0000 7f8c44ff9640 1 -- 172.21.15.16:0/3769703246 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c300071a0 con 0x7f8c340039b0 2023-05-22T15:27:58.896 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.879+0000 7f8c44ff9640 1 -- 172.21.15.16:0/3769703246 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c30007510 con 0x7f8c340039b0 2023-05-22T15:27:58.896 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.879+0000 7f8c4e468640 1 -- 172.21.15.16:0/3769703246 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c48156840 con 0x7f8c340039b0 2023-05-22T15:27:58.897 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.879+0000 7f8c4e468640 1 -- 172.21.15.16:0/3769703246 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c480ce820 con 0x7f8c340039b0 2023-05-22T15:27:58.897 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.879+0000 7f8c4e468640 1 -- 172.21.15.16:0/3769703246 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c4804dbe0 con 0x7f8c340039b0 2023-05-22T15:27:58.897 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.879+0000 7f8c44ff9640 1 -- 172.21.15.16:0/3769703246 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7f8c300067e0 con 0x7f8c340039b0 2023-05-22T15:27:58.897 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.879+0000 7f8c44ff9640 1 --2- 172.21.15.16:0/3769703246 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f8c28065380 0x7f8c28067830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:58.898 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.879+0000 7f8c44ff9640 1 -- 172.21.15.16:0/3769703246 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f8c3011cf00 con 0x7f8c340039b0 2023-05-22T15:27:58.898 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.879+0000 7f8c477fe640 1 --2- 172.21.15.16:0/3769703246 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f8c28065380 0x7f8c28067830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:58.898 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.879+0000 7f8c477fe640 1 --2- 172.21.15.16:0/3769703246 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f8c28065380 0x7f8c28067830 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f8c38001b30 tx=0x7f8c38009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:58.899 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:58.883+0000 7f8c44ff9640 1 -- 172.21.15.16:0/3769703246 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f8c300ee260 con 0x7f8c340039b0 2023-05-22T15:27:59.040 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.031+0000 7f8c4e468640 1 -- 172.21.15.16:0/3769703246 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f8c480cc0e0 con 0x7f8c340039b0 2023-05-22T15:27:59.041 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.031+0000 7f8c44ff9640 1 -- 172.21.15.16:0/3769703246 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v29) v1 ==== 92+0+10 (secure 0 0 0) 0x7f8c300ede50 con 0x7f8c340039b0 2023-05-22T15:27:59.041 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 8 2023-05-22T15:27:59.042 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.035+0000 7f8c4e468640 1 -- 172.21.15.16:0/3769703246 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f8c28065380 msgr2=0x7f8c28067830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:59.043 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.035+0000 7f8c4e468640 1 --2- 172.21.15.16:0/3769703246 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f8c28065380 0x7f8c28067830 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f8c38001b30 tx=0x7f8c38009040 comp rx=0 tx=0).stop 2023-05-22T15:27:59.043 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.035+0000 7f8c4e468640 1 -- 172.21.15.16:0/3769703246 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8c340039b0 msgr2=0x7f8c4815d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:59.043 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.035+0000 7f8c4e468640 1 --2- 172.21.15.16:0/3769703246 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8c340039b0 0x7f8c4815d6f0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f8c30001780 tx=0x7f8c34005800 comp rx=0 tx=0).stop 2023-05-22T15:27:59.043 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.035+0000 7f8c4e468640 1 -- 172.21.15.16:0/3769703246 shutdown_connections 2023-05-22T15:27:59.043 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.035+0000 7f8c4e468640 1 --2- 172.21.15.16:0/3769703246 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8c4814c680 0x7f8c480c2050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:59.043 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.035+0000 7f8c4e468640 1 --2- 172.21.15.16:0/3769703246 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f8c28065380 0x7f8c28067830 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:59.043 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.035+0000 7f8c4e468640 1 --2- 172.21.15.16:0/3769703246 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8c4814cb60 0x7f8c4814c140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:59.044 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.035+0000 7f8c4e468640 1 --2- 172.21.15.16:0/3769703246 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8c340039b0 0x7f8c4815d6f0 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:59.044 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.035+0000 7f8c4e468640 1 -- 172.21.15.16:0/3769703246 >> 172.21.15.16:0/3769703246 conn(0x7f8c48069720 msgr2=0x7f8c48156dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:59.044 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.035+0000 7f8c4e468640 1 -- 172.21.15.16:0/3769703246 shutdown_connections 2023-05-22T15:27:59.044 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.035+0000 7f8c4e468640 1 -- 172.21.15.16:0/3769703246 wait complete. 2023-05-22T15:27:59.058 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-05-22T15:27:59.173 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.167+0000 7f9dc8cb4640 1 Processor -- start 2023-05-22T15:27:59.173 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.167+0000 7f9dc8cb4640 1 -- start start 2023-05-22T15:27:59.175 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.167+0000 7f9dc8cb4640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9dc406bdd0 0x7f9dc40c8af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:59.175 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.167+0000 7f9dc8cb4640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f9dc40cb730 0x7f9dc40c9030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:59.175 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.167+0000 7f9dc8cb4640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9dc40c8660 con 0x7f9dc415bee0 2023-05-22T15:27:59.175 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.167+0000 7f9dc8cb4640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9dc40ce4a0 con 0x7f9dc40cb730 2023-05-22T15:27:59.175 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.167+0000 7f9dc8cb4640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9dc40ce780 con 0x7f9dc406bdd0 2023-05-22T15:27:59.176 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.167+0000 7f9dc2d9d640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f9dc40cb730 0x7f9dc40c9030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:59.176 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.167+0000 7f9dc2d9d640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f9dc40cb730 0x7f9dc40c9030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54488/0 (socket says 172.21.15.16:54488) 2023-05-22T15:27:59.176 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.167+0000 7f9dc2d9d640 1 -- 172.21.15.16:0/2948207848 learned_addr learned my addr 172.21.15.16:0/2948207848 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:59.176 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.167+0000 7f9dc1d9b640 1 --2- 172.21.15.16:0/2948207848 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9dc406bdd0 0x7f9dc40c8af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:59.176 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.167+0000 7f9dc2d9d640 1 -- 172.21.15.16:0/2948207848 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9dc406bdd0 msgr2=0x7f9dc40c8af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:59.176 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.167+0000 7f9dc2d9d640 1 --2- 172.21.15.16:0/2948207848 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9dc406bdd0 0x7f9dc40c8af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:59.176 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.167+0000 7f9dc2d9d640 1 -- 172.21.15.16:0/2948207848 >> v1:172.21.15.16:6789/0 conn(0x7f9dc415bee0 legacy=0x7f9dc406b6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:59.177 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.167+0000 7f9dc2d9d640 1 -- 172.21.15.16:0/2948207848 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9dc40cea60 con 0x7f9dc40cb730 2023-05-22T15:27:59.178 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.167+0000 7f9dc2d9d640 1 --2- 172.21.15.16:0/2948207848 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9dc40cb730 0x7f9dc40c9030 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f9dbc009870 tx=0x7f9dbc01c510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a97d0bcbfb318803 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:59.178 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.167+0000 7f9dc159a640 1 -- 172.21.15.16:0/2948207848 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dbc0a95f0 con 0x7f9dc40cb730 2023-05-22T15:27:59.178 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.167+0000 7f9dc159a640 1 -- 172.21.15.16:0/2948207848 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9dbc0a9750 con 0x7f9dc40cb730 2023-05-22T15:27:59.178 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.167+0000 7f9dc159a640 1 -- 172.21.15.16:0/2948207848 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dbc0a9ae0 con 0x7f9dc40cb730 2023-05-22T15:27:59.178 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.171+0000 7f9dc8cb4640 1 -- 172.21.15.16:0/2948207848 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9dc40cb730 msgr2=0x7f9dc40c9030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:59.178 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.171+0000 7f9dc8cb4640 1 --2- 172.21.15.16:0/2948207848 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9dc40cb730 0x7f9dc40c9030 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f9dbc009870 tx=0x7f9dbc01c510 comp rx=0 tx=0).stop 2023-05-22T15:27:59.179 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.171+0000 7f9dc8cb4640 1 -- 172.21.15.16:0/2948207848 shutdown_connections 2023-05-22T15:27:59.179 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.171+0000 7f9dc8cb4640 1 --2- 172.21.15.16:0/2948207848 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9dc40cb730 0x7f9dc40c9030 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:59.179 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.171+0000 7f9dc8cb4640 1 --2- 172.21.15.16:0/2948207848 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9dc406bdd0 0x7f9dc40c8af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:59.179 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.171+0000 7f9dc8cb4640 1 -- 172.21.15.16:0/2948207848 >> 172.21.15.16:0/2948207848 conn(0x7f9dc4069720 msgr2=0x7f9dc4069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:59.179 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.171+0000 7f9dc8cb4640 1 -- 172.21.15.16:0/2948207848 shutdown_connections 2023-05-22T15:27:59.179 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.171+0000 7f9dc8cb4640 1 -- 172.21.15.16:0/2948207848 wait complete. 2023-05-22T15:27:59.181 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.171+0000 7f9dc8cb4640 1 Processor -- start 2023-05-22T15:27:59.181 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.171+0000 7f9dc8cb4640 1 -- start start 2023-05-22T15:27:59.181 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.171+0000 7f9dc8cb4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9dc406bdd0 0x7f9dc4153880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:59.181 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.171+0000 7f9dc8cb4640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9dc40cb730 0x7f9dc40b3480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:59.182 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.171+0000 7f9dc8cb4640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9dc415bee0 0x7f9dc40c61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:59.182 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.171+0000 7f9dc8cb4640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dc40c6ca0 con 0x7f9dc406bdd0 2023-05-22T15:27:59.182 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.171+0000 7f9dc8cb4640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dc40b2470 con 0x7f9dc415bee0 2023-05-22T15:27:59.182 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.171+0000 7f9dc8cb4640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9dc4153dc0 con 0x7f9dc40cb730 2023-05-22T15:27:59.182 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.171+0000 7f9dc259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9dc406bdd0 0x7f9dc4153880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:59.182 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.171+0000 7f9dc259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9dc406bdd0 0x7f9dc4153880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54490/0 (socket says 172.21.15.16:54490) 2023-05-22T15:27:59.182 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.171+0000 7f9dc259c640 1 -- 172.21.15.16:0/1233314911 learned_addr learned my addr 172.21.15.16:0/1233314911 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:59.183 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.171+0000 7f9dc259c640 1 -- 172.21.15.16:0/1233314911 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9dc40cb730 msgr2=0x7f9dc40b3480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:59.183 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.171+0000 7f9dc259c640 1 --2- 172.21.15.16:0/1233314911 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9dc40cb730 0x7f9dc40b3480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:59.183 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.171+0000 7f9dc259c640 1 -- 172.21.15.16:0/1233314911 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9dc415bee0 msgr2=0x7f9dc40c61e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:59.183 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.171+0000 7f9dc259c640 1 --2- 172.21.15.16:0/1233314911 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9dc415bee0 0x7f9dc40c61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:59.183 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.171+0000 7f9dc259c640 1 -- 172.21.15.16:0/1233314911 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9dbc0094d0 con 0x7f9dc406bdd0 2023-05-22T15:27:59.183 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.171+0000 7f9dc259c640 1 --2- 172.21.15.16:0/1233314911 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9dc406bdd0 0x7f9dc4153880 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f9db400c3c0 tx=0x7f9db400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:59.183 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.171+0000 7f9db2ffd640 1 -- 172.21.15.16:0/1233314911 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9db40052d0 con 0x7f9dc406bdd0 2023-05-22T15:27:59.184 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.171+0000 7f9db2ffd640 1 -- 172.21.15.16:0/1233314911 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9db4005430 con 0x7f9dc406bdd0 2023-05-22T15:27:59.184 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.171+0000 7f9dc8cb4640 1 -- 172.21.15.16:0/1233314911 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9dc4168d40 con 0x7f9dc406bdd0 2023-05-22T15:27:59.184 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.171+0000 7f9db2ffd640 1 -- 172.21.15.16:0/1233314911 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9db4003840 con 0x7f9dc406bdd0 2023-05-22T15:27:59.184 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.171+0000 7f9dc8cb4640 1 -- 172.21.15.16:0/1233314911 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9dc4168bb0 con 0x7f9dc406bdd0 2023-05-22T15:27:59.184 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.175+0000 7f9dc8cb4640 1 -- 172.21.15.16:0/1233314911 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9dc4168ed0 con 0x7f9dc406bdd0 2023-05-22T15:27:59.185 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.175+0000 7f9db2ffd640 1 -- 172.21.15.16:0/1233314911 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7f9db40039a0 con 0x7f9dc406bdd0 2023-05-22T15:27:59.185 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.175+0000 7f9db2ffd640 1 --2- 172.21.15.16:0/1233314911 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f9da4065380 0x7f9da4067830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:59.186 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.175+0000 7f9db2ffd640 1 -- 172.21.15.16:0/1233314911 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f9db4082070 con 0x7f9dc406bdd0 2023-05-22T15:27:59.186 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.175+0000 7f9dc1d9b640 1 --2- 172.21.15.16:0/1233314911 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f9da4065380 0x7f9da4067830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:59.186 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.175+0000 7f9dc1d9b640 1 --2- 172.21.15.16:0/1233314911 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f9da4065380 0x7f9da4067830 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f9db80070b0 tx=0x7f9db8007040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:59.186 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.179+0000 7f9db2ffd640 1 -- 172.21.15.16:0/1233314911 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f9db404f9f0 con 0x7f9dc406bdd0 2023-05-22T15:27:59.336 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.327+0000 7f9dc8cb4640 1 -- 172.21.15.16:0/1233314911 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f9dc404dbe0 con 0x7f9dc406bdd0 2023-05-22T15:27:59.337 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.331+0000 7f9db2ffd640 1 -- 172.21.15.16:0/1233314911 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v29) v1 ==== 104+0+11 (secure 0 0 0) 0x7f9db4053010 con 0x7f9dc406bdd0 2023-05-22T15:27:59.337 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2023-05-22T15:27:59.339 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.331+0000 7f9dc8cb4640 1 -- 172.21.15.16:0/1233314911 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f9da4065380 msgr2=0x7f9da4067830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:59.339 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.331+0000 7f9dc8cb4640 1 --2- 172.21.15.16:0/1233314911 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f9da4065380 0x7f9da4067830 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f9db80070b0 tx=0x7f9db8007040 comp rx=0 tx=0).stop 2023-05-22T15:27:59.339 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.331+0000 7f9dc8cb4640 1 -- 172.21.15.16:0/1233314911 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9dc406bdd0 msgr2=0x7f9dc4153880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:59.340 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.331+0000 7f9dc8cb4640 1 --2- 172.21.15.16:0/1233314911 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9dc406bdd0 0x7f9dc4153880 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f9db400c3c0 tx=0x7f9db400c880 comp rx=0 tx=0).stop 2023-05-22T15:27:59.340 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.331+0000 7f9dc8cb4640 1 -- 172.21.15.16:0/1233314911 shutdown_connections 2023-05-22T15:27:59.340 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.331+0000 7f9dc8cb4640 1 --2- 172.21.15.16:0/1233314911 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9dc415bee0 0x7f9dc40c61e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:59.340 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.331+0000 7f9dc8cb4640 1 --2- 172.21.15.16:0/1233314911 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f9da4065380 0x7f9da4067830 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:59.340 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.331+0000 7f9dc8cb4640 1 --2- 172.21.15.16:0/1233314911 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9dc40cb730 0x7f9dc40b3480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:59.340 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.331+0000 7f9dc8cb4640 1 --2- 172.21.15.16:0/1233314911 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9dc406bdd0 0x7f9dc4153880 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:59.341 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.331+0000 7f9dc8cb4640 1 -- 172.21.15.16:0/1233314911 >> 172.21.15.16:0/1233314911 conn(0x7f9dc4069720 msgr2=0x7f9dc40abd90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:59.341 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.331+0000 7f9dc8cb4640 1 -- 172.21.15.16:0/1233314911 shutdown_connections 2023-05-22T15:27:59.341 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.331+0000 7f9dc8cb4640 1 -- 172.21.15.16:0/1233314911 wait complete. 2023-05-22T15:27:59.355 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-05-22T15:27:59.482 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.475+0000 7ff5717f7640 1 Processor -- start 2023-05-22T15:27:59.482 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.475+0000 7ff5717f7640 1 -- start start 2023-05-22T15:27:59.483 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.475+0000 7ff5717f7640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff56c159d40 0x7ff56c0cb260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:59.483 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.475+0000 7ff5717f7640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff56c070c30 con 0x7ff56c159970 2023-05-22T15:27:59.484 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.475+0000 7ff5717f7640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff56c070e60 con 0x7ff56c06b6c0 2023-05-22T15:27:59.484 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.475+0000 7ff5717f7640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff56c15dd90 con 0x7ff56c159d40 2023-05-22T15:27:59.484 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.475+0000 7ff56affd640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7ff56c159970 0x7ff56c15d680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:50076/0 (socket says 172.21.15.16:50076) 2023-05-22T15:27:59.485 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.475+0000 7ff56affd640 1 -- 172.21.15.16:0/1758927881 learned_addr learned my addr 172.21.15.16:0/1758927881 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:27:59.485 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.475+0000 7ff56a7fc640 1 --2- 172.21.15.16:0/1758927881 >> v2:172.21.15.16:3300/0 conn(0x7ff56c159d40 0x7ff56c0cb260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:59.485 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.475+0000 7ff56a7fc640 1 -- 172.21.15.16:0/1758927881 >> v1:172.21.15.62:6789/0 conn(0x7ff56c06b6c0 legacy=0x7ff56c15a5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:59.485 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.475+0000 7ff56a7fc640 1 -- 172.21.15.16:0/1758927881 >> v1:172.21.15.16:6789/0 conn(0x7ff56c159970 legacy=0x7ff56c15d680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:59.486 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.475+0000 7ff56a7fc640 1 -- 172.21.15.16:0/1758927881 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff56c071140 con 0x7ff56c159d40 2023-05-22T15:27:59.490 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.479+0000 7ff56a7fc640 1 --2- 172.21.15.16:0/1758927881 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff56c159d40 0x7ff56c0cb260 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7ff5600097c0 tx=0x7ff56001c3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=490ee9e6c001e1aa server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:59.490 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.479+0000 7ff569ffb640 1 -- 172.21.15.16:0/1758927881 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5600ab600 con 0x7ff56c159d40 2023-05-22T15:27:59.490 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.479+0000 7ff569ffb640 1 -- 172.21.15.16:0/1758927881 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5600a7050 con 0x7ff56c159d40 2023-05-22T15:27:59.490 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.479+0000 7ff569ffb640 1 -- 172.21.15.16:0/1758927881 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5600ab980 con 0x7ff56c159d40 2023-05-22T15:27:59.491 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.479+0000 7ff5717f7640 1 -- 172.21.15.16:0/1758927881 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff56c159d40 msgr2=0x7ff56c0cb260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:59.491 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.479+0000 7ff5717f7640 1 --2- 172.21.15.16:0/1758927881 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff56c159d40 0x7ff56c0cb260 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7ff5600097c0 tx=0x7ff56001c3b0 comp rx=0 tx=0).stop 2023-05-22T15:27:59.491 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.479+0000 7ff5717f7640 1 -- 172.21.15.16:0/1758927881 shutdown_connections 2023-05-22T15:27:59.491 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.479+0000 7ff5717f7640 1 --2- 172.21.15.16:0/1758927881 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff56c159d40 0x7ff56c0cb260 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:59.491 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.479+0000 7ff5717f7640 1 -- 172.21.15.16:0/1758927881 >> 172.21.15.16:0/1758927881 conn(0x7ff56c069720 msgr2=0x7ff56c069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:59.491 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.479+0000 7ff5717f7640 1 -- 172.21.15.16:0/1758927881 shutdown_connections 2023-05-22T15:27:59.492 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.479+0000 7ff5717f7640 1 -- 172.21.15.16:0/1758927881 wait complete. 2023-05-22T15:27:59.492 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.479+0000 7ff5717f7640 1 Processor -- start 2023-05-22T15:27:59.492 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.479+0000 7ff5717f7640 1 -- start start 2023-05-22T15:27:59.492 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.479+0000 7ff5717f7640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff56c06b6c0 0x7ff56c154700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:59.492 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.479+0000 7ff5717f7640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff56c159970 0x7ff56c154fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:59.492 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.479+0000 7ff5717f7640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff56c159d40 0x7ff56c0b2fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:59.493 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.479+0000 7ff5717f7640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff56c0b28a0 con 0x7ff56c159970 2023-05-22T15:27:59.493 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.479+0000 7ff5717f7640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff56c0aed30 con 0x7ff56c159d40 2023-05-22T15:27:59.493 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.479+0000 7ff5717f7640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff56c0aefd0 con 0x7ff56c06b6c0 2023-05-22T15:27:59.493 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.479+0000 7ff56a7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff56c159970 0x7ff56c154fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:59.493 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.479+0000 7ff56a7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff56c159970 0x7ff56c154fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39616/0 (socket says 172.21.15.16:39616) 2023-05-22T15:27:59.493 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.479+0000 7ff56a7fc640 1 -- 172.21.15.16:0/1470551364 learned_addr learned my addr 172.21.15.16:0/1470551364 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:59.493 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.479+0000 7ff56b7fe640 1 --2- 172.21.15.16:0/1470551364 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff56c159d40 0x7ff56c0b2fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:59.493 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.479+0000 7ff56affd640 1 --2- 172.21.15.16:0/1470551364 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff56c06b6c0 0x7ff56c154700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:59.494 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.479+0000 7ff56a7fc640 1 -- 172.21.15.16:0/1470551364 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff56c06b6c0 msgr2=0x7ff56c154700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:59.494 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.479+0000 7ff56a7fc640 1 --2- 172.21.15.16:0/1470551364 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff56c06b6c0 0x7ff56c154700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:59.494 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.479+0000 7ff56a7fc640 1 -- 172.21.15.16:0/1470551364 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff56c159d40 msgr2=0x7ff56c0b2fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:59.494 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.479+0000 7ff56a7fc640 1 --2- 172.21.15.16:0/1470551364 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff56c159d40 0x7ff56c0b2fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:59.494 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.479+0000 7ff56a7fc640 1 -- 172.21.15.16:0/1470551364 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff560009420 con 0x7ff56c159970 2023-05-22T15:27:59.494 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.479+0000 7ff56affd640 1 --2- 172.21.15.16:0/1470551364 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff56c06b6c0 0x7ff56c154700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T15:27:59.494 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.479+0000 7ff56b7fe640 1 --2- 172.21.15.16:0/1470551364 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff56c159d40 0x7ff56c0b2fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T15:27:59.495 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.483+0000 7ff56a7fc640 1 --2- 172.21.15.16:0/1470551364 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff56c159970 0x7ff56c154fd0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7ff560009710 tx=0x7ff560019150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:59.495 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.483+0000 7ff5537fe640 1 -- 172.21.15.16:0/1470551364 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5600b4600 con 0x7ff56c159970 2023-05-22T15:27:59.495 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.483+0000 7ff5537fe640 1 -- 172.21.15.16:0/1470551364 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5600b4760 con 0x7ff56c159970 2023-05-22T15:27:59.495 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.483+0000 7ff5537fe640 1 -- 172.21.15.16:0/1470551364 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5600b4b10 con 0x7ff56c159970 2023-05-22T15:27:59.495 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.483+0000 7ff5717f7640 1 -- 172.21.15.16:0/1470551364 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff56c0cb7a0 con 0x7ff56c159970 2023-05-22T15:27:59.495 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.483+0000 7ff5717f7640 1 -- 172.21.15.16:0/1470551364 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff56c06bf10 con 0x7ff56c159970 2023-05-22T15:27:59.496 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.483+0000 7ff5717f7640 1 -- 172.21.15.16:0/1470551364 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff56c0cb930 con 0x7ff56c159970 2023-05-22T15:27:59.496 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.483+0000 7ff5537fe640 1 -- 172.21.15.16:0/1470551364 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7ff5600b2040 con 0x7ff56c159970 2023-05-22T15:27:59.496 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.483+0000 7ff5537fe640 1 --2- 172.21.15.16:0/1470551364 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ff5580653b0 0x7ff558067860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:59.497 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.483+0000 7ff5537fe640 1 -- 172.21.15.16:0/1470551364 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6871+0+0 (secure 0 0 0) 0x7ff5600a4e40 con 0x7ff56c159970 2023-05-22T15:27:59.497 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.483+0000 7ff56affd640 1 --2- 172.21.15.16:0/1470551364 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ff5580653b0 0x7ff558067860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:59.497 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.483+0000 7ff56affd640 1 --2- 172.21.15.16:0/1470551364 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ff5580653b0 0x7ff558067860 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7ff55c005890 tx=0x7ff55c009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:59.497 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.487+0000 7ff5537fe640 1 -- 172.21.15.16:0/1470551364 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7ff5600ea580 con 0x7ff56c159970 2023-05-22T15:27:59.647 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.639+0000 7ff5717f7640 1 -- 172.21.15.16:0/1470551364 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7ff56c04dbe0 con 0x7ff56c159970 2023-05-22T15:27:59.648 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.639+0000 7ff5537fe640 1 -- 172.21.15.16:0/1470551364 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v29) v1 ==== 100+0+11 (secure 0 0 0) 0x7ff5600b00a0 con 0x7ff56c159970 2023-05-22T15:27:59.648 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 64 2023-05-22T15:27:59.650 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.643+0000 7ff5717f7640 1 -- 172.21.15.16:0/1470551364 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ff5580653b0 msgr2=0x7ff558067860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:59.650 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.643+0000 7ff5717f7640 1 --2- 172.21.15.16:0/1470551364 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ff5580653b0 0x7ff558067860 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7ff55c005890 tx=0x7ff55c009040 comp rx=0 tx=0).stop 2023-05-22T15:27:59.651 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.643+0000 7ff5717f7640 1 -- 172.21.15.16:0/1470551364 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff56c159970 msgr2=0x7ff56c154fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:59.651 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.643+0000 7ff5717f7640 1 --2- 172.21.15.16:0/1470551364 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff56c159970 0x7ff56c154fd0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7ff560009710 tx=0x7ff560019150 comp rx=0 tx=0).stop 2023-05-22T15:27:59.651 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.643+0000 7ff5717f7640 1 -- 172.21.15.16:0/1470551364 shutdown_connections 2023-05-22T15:27:59.651 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.643+0000 7ff5717f7640 1 --2- 172.21.15.16:0/1470551364 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff56c159d40 0x7ff56c0b2fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:59.651 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.643+0000 7ff5717f7640 1 --2- 172.21.15.16:0/1470551364 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff56c159970 0x7ff56c154fd0 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:59.652 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.643+0000 7ff5717f7640 1 --2- 172.21.15.16:0/1470551364 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ff5580653b0 0x7ff558067860 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:59.652 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.643+0000 7ff5717f7640 1 --2- 172.21.15.16:0/1470551364 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff56c06b6c0 0x7ff56c154700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:59.652 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.643+0000 7ff5717f7640 1 -- 172.21.15.16:0/1470551364 >> 172.21.15.16:0/1470551364 conn(0x7ff56c069720 msgr2=0x7ff56c157170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:59.653 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.643+0000 7ff5717f7640 1 -- 172.21.15.16:0/1470551364 shutdown_connections 2023-05-22T15:27:59.653 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.643+0000 7ff5717f7640 1 -- 172.21.15.16:0/1470551364 wait complete. 2023-05-22T15:27:59.667 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-05-22T15:27:59.792 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.783+0000 7f79b23b3640 1 Processor -- start 2023-05-22T15:27:59.792 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.783+0000 7f79b23b3640 1 -- start start 2023-05-22T15:27:59.793 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.787+0000 7f79b23b3640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f79ac070c70 0x7f79ac071530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:59.793 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.787+0000 7f79b23b3640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79ac15d520 con 0x7f79ac15c580 2023-05-22T15:27:59.793 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.787+0000 7f79b23b3640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79ac15ca00 con 0x7f79ac06b6c0 2023-05-22T15:27:59.794 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.787+0000 7f79b23b3640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79ac0c8b10 con 0x7f79ac070c70 2023-05-22T15:27:59.794 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.787+0000 7f79a3fff640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f79ac15c580 0x7f79ac06fba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:50088/0 (socket says 172.21.15.16:50088) 2023-05-22T15:27:59.794 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.787+0000 7f79a3fff640 1 -- 172.21.15.16:0/2326053519 learned_addr learned my addr 172.21.15.16:0/2326053519 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:27:59.795 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.787+0000 7f79abfff640 1 --2- 172.21.15.16:0/2326053519 >> v2:172.21.15.62:3300/0 conn(0x7f79ac070c70 0x7f79ac071530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:59.796 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.787+0000 7f79ab7fe640 1 -- 172.21.15.16:0/2326053519 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3230748995 0 0) 0x7f79ac15d520 con 0x7f79ac15c580 2023-05-22T15:27:59.796 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.787+0000 7f79ab7fe640 1 -- 172.21.15.16:0/2326053519 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7998003530 con 0x7f79ac15c580 2023-05-22T15:27:59.796 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.787+0000 7f79ab7fe640 1 -- 172.21.15.16:0/2326053519 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3933773617 0 0) 0x7f7998003530 con 0x7f79ac15c580 2023-05-22T15:27:59.796 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.787+0000 7f79ab7fe640 1 -- 172.21.15.16:0/2326053519 >> v2:172.21.15.62:3300/0 conn(0x7f79ac070c70 msgr2=0x7f79ac071530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:59.796 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.787+0000 7f79ab7fe640 1 --2- 172.21.15.16:0/2326053519 >> v2:172.21.15.62:3300/0 conn(0x7f79ac070c70 0x7f79ac071530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:59.797 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.787+0000 7f79ab7fe640 1 -- 172.21.15.16:0/2326053519 >> v1:172.21.15.62:6789/0 conn(0x7f79ac06b6c0 legacy=0x7f79ac06bae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:59.797 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.787+0000 7f79ab7fe640 1 -- 172.21.15.16:0/2326053519 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79ac15cc80 con 0x7f79ac15c580 2023-05-22T15:27:59.797 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.787+0000 7f79ab7fe640 1 -- 172.21.15.16:0/2326053519 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7f7994001680 con 0x7f79ac15c580 2023-05-22T15:27:59.797 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.787+0000 7f79ab7fe640 1 -- 172.21.15.16:0/2326053519 >> v1:172.21.15.16:6789/0 conn(0x7f79ac15c580 legacy=0x7f79ac06fba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:59.797 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.787+0000 7f79ab7fe640 1 --2- 172.21.15.16:0/2326053519 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7998003980 0x7f7998003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:59.798 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.787+0000 7f79ab7fe640 1 --2- 172.21.15.16:0/2326053519 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f79980047d0 0x7f7998004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:59.798 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.787+0000 7f79ab7fe640 1 --2- 172.21.15.16:0/2326053519 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f79980052a0 0x7f799800b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:59.798 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.787+0000 7f79ab7fe640 1 -- 172.21.15.16:0/2326053519 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79ac0c8b10 con 0x7f7998003980 2023-05-22T15:27:59.798 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.787+0000 7f79ab7fe640 1 -- 172.21.15.16:0/2326053519 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f799800c160 con 0x7f79980052a0 2023-05-22T15:27:59.798 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.787+0000 7f79ab7fe640 1 -- 172.21.15.16:0/2326053519 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f799800c320 con 0x7f79980047d0 2023-05-22T15:27:59.798 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.787+0000 7f79abfff640 1 --2- 172.21.15.16:0/2326053519 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7998003980 0x7f7998003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:59.798 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.787+0000 7f79a3fff640 1 --2- 172.21.15.16:0/2326053519 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f79980047d0 0x7f7998004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:59.798 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.787+0000 7f79b0950640 1 --2- 172.21.15.16:0/2326053519 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f79980052a0 0x7f799800b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:59.799 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.787+0000 7f79abfff640 1 -- 172.21.15.16:0/2326053519 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f79980047d0 msgr2=0x7f7998004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:59.799 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.787+0000 7f79abfff640 1 --2- 172.21.15.16:0/2326053519 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f79980047d0 0x7f7998004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:59.799 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.787+0000 7f79abfff640 1 -- 172.21.15.16:0/2326053519 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f79980052a0 msgr2=0x7f799800b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:27:59.799 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.787+0000 7f79abfff640 1 --2- 172.21.15.16:0/2326053519 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f79980052a0 0x7f799800b990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:59.799 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.787+0000 7f79abfff640 1 -- 172.21.15.16:0/2326053519 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7998003530 con 0x7f7998003980 2023-05-22T15:27:59.800 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.787+0000 7f79b0950640 1 -- 172.21.15.16:0/2326053519 reap_dead start 2023-05-22T15:27:59.800 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.787+0000 7f79a3fff640 1 --2- 172.21.15.16:0/2326053519 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f79980047d0 0x7f7998004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:27:59.803 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.787+0000 7f79abfff640 1 --2- 172.21.15.16:0/2326053519 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7998003980 0x7f7998003d90 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f799c00ba30 tx=0x7f799c01c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=602c622d031312e7 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:59.804 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.787+0000 7f79ab7fe640 1 -- 172.21.15.16:0/2326053519 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f799c0ac5f0 con 0x7f7998003980 2023-05-22T15:27:59.804 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.791+0000 7f79ab7fe640 1 -- 172.21.15.16:0/2326053519 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f799c0ac750 con 0x7f7998003980 2023-05-22T15:27:59.804 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.791+0000 7f79ab7fe640 1 -- 172.21.15.16:0/2326053519 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f799c0acb00 con 0x7f7998003980 2023-05-22T15:27:59.804 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.791+0000 7f79b23b3640 1 -- 172.21.15.16:0/2326053519 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7998003980 msgr2=0x7f7998003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:59.804 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.791+0000 7f79b23b3640 1 --2- 172.21.15.16:0/2326053519 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7998003980 0x7f7998003d90 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f799c00ba30 tx=0x7f799c01c650 comp rx=0 tx=0).stop 2023-05-22T15:27:59.805 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.791+0000 7f79b23b3640 1 -- 172.21.15.16:0/2326053519 shutdown_connections 2023-05-22T15:27:59.805 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.791+0000 7f79b23b3640 1 --2- 172.21.15.16:0/2326053519 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7998003980 0x7f7998003d90 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:59.805 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.791+0000 7f79b23b3640 1 -- 172.21.15.16:0/2326053519 >> 172.21.15.16:0/2326053519 conn(0x7f79ac069720 msgr2=0x7f79ac069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:59.805 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.791+0000 7f79b23b3640 1 -- 172.21.15.16:0/2326053519 shutdown_connections 2023-05-22T15:27:59.805 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.791+0000 7f79b23b3640 1 -- 172.21.15.16:0/2326053519 wait complete. 2023-05-22T15:27:59.805 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.791+0000 7f79b23b3640 1 Processor -- start 2023-05-22T15:27:59.805 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.791+0000 7f79b23b3640 1 -- start start 2023-05-22T15:27:59.805 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.791+0000 7f79b23b3640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7998003980 0x7f79ac1e6ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:59.806 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.791+0000 7f79b23b3640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f79ac0c74e0 0x7f79ac13bf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:59.806 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.791+0000 7f79b23b3640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f79ac13c4a0 0x7f79ac13c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:59.806 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.791+0000 7f79b23b3640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79ac0c7f80 con 0x7f7998003980 2023-05-22T15:27:59.806 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.791+0000 7f79b23b3640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79ac15a5e0 con 0x7f79ac0c74e0 2023-05-22T15:27:59.806 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.791+0000 7f79b23b3640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79ac1e7420 con 0x7f79ac13c4a0 2023-05-22T15:27:59.807 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.791+0000 7f79abfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7998003980 0x7f79ac1e6ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:59.807 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.791+0000 7f79abfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7998003980 0x7f79ac1e6ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39632/0 (socket says 172.21.15.16:39632) 2023-05-22T15:27:59.807 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.791+0000 7f79abfff640 1 -- 172.21.15.16:0/2481793592 learned_addr learned my addr 172.21.15.16:0/2481793592 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:27:59.807 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.791+0000 7f79a3fff640 1 --2- 172.21.15.16:0/2481793592 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f79ac0c74e0 0x7f79ac13bf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:59.807 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.791+0000 7f79abfff640 1 -- 172.21.15.16:0/2481793592 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f79ac13c4a0 msgr2=0x7f79ac13c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:59.807 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.791+0000 7f79b0950640 1 --2- 172.21.15.16:0/2481793592 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f79ac13c4a0 0x7f79ac13c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:59.807 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.791+0000 7f79abfff640 1 --2- 172.21.15.16:0/2481793592 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f79ac13c4a0 0x7f79ac13c920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:59.807 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.791+0000 7f79abfff640 1 -- 172.21.15.16:0/2481793592 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f79ac0c74e0 msgr2=0x7f79ac13bf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:59.808 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.791+0000 7f79abfff640 1 --2- 172.21.15.16:0/2481793592 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f79ac0c74e0 0x7f79ac13bf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:59.808 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.791+0000 7f79abfff640 1 -- 172.21.15.16:0/2481793592 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79ac071040 con 0x7f7998003980 2023-05-22T15:27:59.808 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.791+0000 7f79b0950640 1 --2- 172.21.15.16:0/2481793592 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f79ac13c4a0 0x7f79ac13c920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:27:59.808 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.791+0000 7f79a3fff640 1 --2- 172.21.15.16:0/2481793592 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f79ac0c74e0 0x7f79ac13bf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:27:59.808 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.791+0000 7f79abfff640 1 --2- 172.21.15.16:0/2481793592 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7998003980 0x7f79ac1e6ee0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f799c0ae040 tx=0x7f7998005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:59.808 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.791+0000 7f79a97fa640 1 -- 172.21.15.16:0/2481793592 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f799c0acc00 con 0x7f7998003980 2023-05-22T15:27:59.808 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.791+0000 7f79a97fa640 1 -- 172.21.15.16:0/2481793592 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f799c0b9070 con 0x7f7998003980 2023-05-22T15:27:59.809 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.791+0000 7f79b23b3640 1 -- 172.21.15.16:0/2481793592 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79ac0cc510 con 0x7f7998003980 2023-05-22T15:27:59.809 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.791+0000 7f79a97fa640 1 -- 172.21.15.16:0/2481793592 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f799c004020 con 0x7f7998003980 2023-05-22T15:27:59.809 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.791+0000 7f79b23b3640 1 -- 172.21.15.16:0/2481793592 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79ac06bea0 con 0x7f7998003980 2023-05-22T15:27:59.809 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.795+0000 7f79b23b3640 1 -- 172.21.15.16:0/2481793592 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79ac04dbe0 con 0x7f7998003980 2023-05-22T15:27:59.809 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.795+0000 7f79a97fa640 1 -- 172.21.15.16:0/2481793592 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7f799c0a9070 con 0x7f7998003980 2023-05-22T15:27:59.809 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.795+0000 7f79a97fa640 1 --2- 172.21.15.16:0/2481793592 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f798c065380 0x7f798c067830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:27:59.809 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.795+0000 7f79a97fa640 1 -- 172.21.15.16:0/2481793592 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f799c11cde0 con 0x7f7998003980 2023-05-22T15:27:59.810 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.795+0000 7f79a3fff640 1 --2- 172.21.15.16:0/2481793592 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f798c065380 0x7f798c067830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:27:59.810 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.795+0000 7f79a3fff640 1 --2- 172.21.15.16:0/2481793592 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f798c065380 0x7f798c067830 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f79ac0c78f0 tx=0x7f7994009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:27:59.810 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.799+0000 7f79a97fa640 1 -- 172.21.15.16:0/2481793592 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f799c0ee1f0 con 0x7f7998003980 2023-05-22T15:27:59.959 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.951+0000 7f79b23b3640 1 -- 172.21.15.16:0/2481793592 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f79ac0b3f90 con 0x7f7998003980 2023-05-22T15:27:59.960 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.951+0000 7f79a97fa640 1 -- 172.21.15.16:0/2481793592 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v29) v1 ==== 103+0+11 (secure 0 0 0) 0x7f799c0bb020 con 0x7f7998003980 2023-05-22T15:27:59.960 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2023-05-22T15:27:59.962 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.955+0000 7f79b23b3640 1 -- 172.21.15.16:0/2481793592 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f798c065380 msgr2=0x7f798c067830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:59.962 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.955+0000 7f79b23b3640 1 --2- 172.21.15.16:0/2481793592 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f798c065380 0x7f798c067830 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f79ac0c78f0 tx=0x7f7994009040 comp rx=0 tx=0).stop 2023-05-22T15:27:59.963 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.955+0000 7f79b23b3640 1 -- 172.21.15.16:0/2481793592 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7998003980 msgr2=0x7f79ac1e6ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:27:59.963 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.955+0000 7f79b23b3640 1 --2- 172.21.15.16:0/2481793592 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7998003980 0x7f79ac1e6ee0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f799c0ae040 tx=0x7f7998005800 comp rx=0 tx=0).stop 2023-05-22T15:27:59.963 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.955+0000 7f79b23b3640 1 -- 172.21.15.16:0/2481793592 shutdown_connections 2023-05-22T15:27:59.963 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.955+0000 7f79b23b3640 1 --2- 172.21.15.16:0/2481793592 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f798c065380 0x7f798c067830 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:59.963 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.955+0000 7f79b23b3640 1 --2- 172.21.15.16:0/2481793592 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f79ac13c4a0 0x7f79ac13c920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:59.964 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.955+0000 7f79b23b3640 1 --2- 172.21.15.16:0/2481793592 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f79ac0c74e0 0x7f79ac13bf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:59.964 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.955+0000 7f79b23b3640 1 --2- 172.21.15.16:0/2481793592 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7998003980 0x7f79ac1e6ee0 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:27:59.964 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.955+0000 7f79b23b3640 1 -- 172.21.15.16:0/2481793592 >> 172.21.15.16:0/2481793592 conn(0x7f79ac069720 msgr2=0x7f79ac0adc40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:27:59.964 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.955+0000 7f79b23b3640 1 -- 172.21.15.16:0/2481793592 shutdown_connections 2023-05-22T15:27:59.964 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:27:59.955+0000 7f79b23b3640 1 -- 172.21.15.16:0/2481793592 wait complete. 2023-05-22T15:27:59.979 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-05-22T15:28:00.097 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.091+0000 7fe60f579640 1 Processor -- start 2023-05-22T15:28:00.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.091+0000 7fe60f579640 1 -- start start 2023-05-22T15:28:00.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.091+0000 7fe60f579640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe608071640 0x7fe608160090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:00.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.091+0000 7fe60f579640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe6081605d0 0x7fe60815c270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:00.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.091+0000 7fe60f579640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe608069ae0 con 0x7fe6080ce480 2023-05-22T15:28:00.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.091+0000 7fe60f579640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe60815f990 con 0x7fe608071640 2023-05-22T15:28:00.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.091+0000 7fe60f579640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe608071ac0 con 0x7fe6081605d0 2023-05-22T15:28:00.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.091+0000 7fe60d315640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fe6080ce480 0x7fe6080ce850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:50092/0 (socket says 172.21.15.16:50092) 2023-05-22T15:28:00.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.091+0000 7fe60d315640 1 -- 172.21.15.16:0/3618513326 learned_addr learned my addr 172.21.15.16:0/3618513326 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:28:00.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.091+0000 7fe60cb14640 1 --2- 172.21.15.16:0/3618513326 >> v2:172.21.15.16:3300/0 conn(0x7fe608071640 0x7fe608160090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:00.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.091+0000 7fe60db16640 1 --2- 172.21.15.16:0/3618513326 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe6081605d0 0x7fe60815c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:00.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.091+0000 7fe60cb14640 1 -- 172.21.15.16:0/3618513326 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe6081605d0 msgr2=0x7fe60815c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:28:00.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.091+0000 7fe60cb14640 1 --2- 172.21.15.16:0/3618513326 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe6081605d0 0x7fe60815c270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:00.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.091+0000 7fe60cb14640 1 -- 172.21.15.16:0/3618513326 >> v1:172.21.15.16:6789/0 conn(0x7fe6080ce480 legacy=0x7fe6080ce850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:00.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.091+0000 7fe60cb14640 1 -- 172.21.15.16:0/3618513326 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe608069dc0 con 0x7fe608071640 2023-05-22T15:28:00.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.095+0000 7fe60cb14640 1 --2- 172.21.15.16:0/3618513326 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe608071640 0x7fe608160090 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7fe5f80097c0 tx=0x7fe5f801c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67a717a47767b59e server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:28:00.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.095+0000 7fe5fffff640 1 -- 172.21.15.16:0/3618513326 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5f80ab650 con 0x7fe608071640 2023-05-22T15:28:00.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.095+0000 7fe5fffff640 1 -- 172.21.15.16:0/3618513326 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5f80ab7b0 con 0x7fe608071640 2023-05-22T15:28:00.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.095+0000 7fe5fffff640 1 -- 172.21.15.16:0/3618513326 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5f80abb40 con 0x7fe608071640 2023-05-22T15:28:00.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.095+0000 7fe60f579640 1 -- 172.21.15.16:0/3618513326 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe608071640 msgr2=0x7fe608160090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:00.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.095+0000 7fe60f579640 1 --2- 172.21.15.16:0/3618513326 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe608071640 0x7fe608160090 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7fe5f80097c0 tx=0x7fe5f801c450 comp rx=0 tx=0).stop 2023-05-22T15:28:00.103 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.095+0000 7fe60f579640 1 -- 172.21.15.16:0/3618513326 shutdown_connections 2023-05-22T15:28:00.103 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.095+0000 7fe60f579640 1 --2- 172.21.15.16:0/3618513326 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe6081605d0 0x7fe60815c270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:00.103 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.095+0000 7fe60f579640 1 --2- 172.21.15.16:0/3618513326 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe608071640 0x7fe608160090 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:00.103 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.095+0000 7fe60f579640 1 -- 172.21.15.16:0/3618513326 >> 172.21.15.16:0/3618513326 conn(0x7fe6080cc870 msgr2=0x7fe6080cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:28:00.103 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.095+0000 7fe60f579640 1 -- 172.21.15.16:0/3618513326 shutdown_connections 2023-05-22T15:28:00.103 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.095+0000 7fe60f579640 1 -- 172.21.15.16:0/3618513326 wait complete. 2023-05-22T15:28:00.105 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.095+0000 7fe60f579640 1 Processor -- start 2023-05-22T15:28:00.105 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.095+0000 7fe60f579640 1 -- start start 2023-05-22T15:28:00.105 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.095+0000 7fe60f579640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe608071640 0x7fe6080ab9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:00.105 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.095+0000 7fe60f579640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe6080ce480 0x7fe6081f8110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:00.105 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.095+0000 7fe60f579640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe6081605d0 0x7fe60814c450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:00.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.095+0000 7fe60f579640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6080ae680 con 0x7fe6080ce480 2023-05-22T15:28:00.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.095+0000 7fe60f579640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6081f9040 con 0x7fe608071640 2023-05-22T15:28:00.107 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.095+0000 7fe60f579640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6081f87a0 con 0x7fe6081605d0 2023-05-22T15:28:00.107 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.095+0000 7fe60cb14640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe6080ce480 0x7fe6081f8110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:00.107 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.095+0000 7fe60cb14640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe6080ce480 0x7fe6081f8110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39640/0 (socket says 172.21.15.16:39640) 2023-05-22T15:28:00.107 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.095+0000 7fe60cb14640 1 -- 172.21.15.16:0/1361157539 learned_addr learned my addr 172.21.15.16:0/1361157539 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:28:00.107 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.095+0000 7fe60db16640 1 --2- 172.21.15.16:0/1361157539 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe6081605d0 0x7fe60814c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:00.107 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.095+0000 7fe60d315640 1 --2- 172.21.15.16:0/1361157539 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe608071640 0x7fe6080ab9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:00.108 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.095+0000 7fe60cb14640 1 -- 172.21.15.16:0/1361157539 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe6081605d0 msgr2=0x7fe60814c450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:00.108 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.095+0000 7fe60cb14640 1 --2- 172.21.15.16:0/1361157539 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe6081605d0 0x7fe60814c450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:00.108 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.095+0000 7fe60cb14640 1 -- 172.21.15.16:0/1361157539 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe608071640 msgr2=0x7fe6080ab9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:00.108 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.095+0000 7fe60cb14640 1 --2- 172.21.15.16:0/1361157539 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe608071640 0x7fe6080ab9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:00.108 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.095+0000 7fe60cb14640 1 -- 172.21.15.16:0/1361157539 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5f8009420 con 0x7fe6080ce480 2023-05-22T15:28:00.109 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.095+0000 7fe60db16640 1 --2- 172.21.15.16:0/1361157539 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe6081605d0 0x7fe60814c450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:28:00.109 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.099+0000 7fe60cb14640 1 --2- 172.21.15.16:0/1361157539 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe6080ce480 0x7fe6081f8110 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7fe5f80ad040 tx=0x7fe5f801ac90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:28:00.109 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.099+0000 7fe5fdffb640 1 -- 172.21.15.16:0/1361157539 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5f80b5810 con 0x7fe6080ce480 2023-05-22T15:28:00.109 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.099+0000 7fe5fdffb640 1 -- 172.21.15.16:0/1361157539 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5f80b5990 con 0x7fe6080ce480 2023-05-22T15:28:00.110 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.099+0000 7fe60f579640 1 -- 172.21.15.16:0/1361157539 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6080698b0 con 0x7fe6080ce480 2023-05-22T15:28:00.110 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.099+0000 7fe5fdffb640 1 -- 172.21.15.16:0/1361157539 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5f80b5d40 con 0x7fe6080ce480 2023-05-22T15:28:00.110 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.099+0000 7fe60f579640 1 -- 172.21.15.16:0/1361157539 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe608069720 con 0x7fe6080ce480 2023-05-22T15:28:00.111 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.099+0000 7fe60f579640 1 -- 172.21.15.16:0/1361157539 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6080c5e70 con 0x7fe6080ce480 2023-05-22T15:28:00.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.099+0000 7fe5fdffb640 1 -- 172.21.15.16:0/1361157539 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7fe5f80bf410 con 0x7fe6080ce480 2023-05-22T15:28:00.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.099+0000 7fe5fdffb640 1 --2- 172.21.15.16:0/1361157539 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fe5ec065380 0x7fe5ec067830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:00.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.099+0000 7fe5fdffb640 1 -- 172.21.15.16:0/1361157539 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6871+0+0 (secure 0 0 0) 0x7fe5f811daa0 con 0x7fe6080ce480 2023-05-22T15:28:00.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.099+0000 7fe60d315640 1 --2- 172.21.15.16:0/1361157539 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fe5ec065380 0x7fe5ec067830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:00.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.099+0000 7fe60d315640 1 --2- 172.21.15.16:0/1361157539 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fe5ec065380 0x7fe5ec067830 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fe604005890 tx=0x7fe604009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:28:00.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.103+0000 7fe5fdffb640 1 -- 172.21.15.16:0/1361157539 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fe5f80eb390 con 0x7fe6080ce480 2023-05-22T15:28:00.298 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.291+0000 7fe60f579640 1 -- 172.21.15.16:0/1361157539 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe60804dbe0 con 0x7fe6080ce480 2023-05-22T15:28:00.299 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.291+0000 7fe5fdffb640 1 -- 172.21.15.16:0/1361157539 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4641 (secure 0 0 0) 0x7fe5f80ee9b0 con 0x7fe6080ce480 2023-05-22T15:28:00.300 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:28:00.300 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4261,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6834/3661808159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":3661808159},{"type":"v1","addr":"172.21.15.62:6834","nonce":3661808159}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4264,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/2227467222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6833","nonce":2227467222},{"type":"v1","addr":"172.21.15.62:6835","nonce":2227467222}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4284,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6836/2808106087","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2808106087},{"type":"v1","addr":"172.21.15.16:6836","nonce":2808106087}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-05-22T15:26:49.546008+0000","modified":"2023-05-22T15:27:06.047204+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4287},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4287":{"gid":4287,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.16:6837/283322289","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6835","nonce":283322289},{"type":"v1","addr":"172.21.15.16:6837","nonce":283322289}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-05-22T15:28:00.303 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 8 2023-05-22T15:28:00.303 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.295+0000 7fe60f579640 1 -- 172.21.15.16:0/1361157539 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fe5ec065380 msgr2=0x7fe5ec067830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:00.303 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.295+0000 7fe60f579640 1 --2- 172.21.15.16:0/1361157539 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fe5ec065380 0x7fe5ec067830 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fe604005890 tx=0x7fe604009040 comp rx=0 tx=0).stop 2023-05-22T15:28:00.304 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.295+0000 7fe60f579640 1 -- 172.21.15.16:0/1361157539 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe6080ce480 msgr2=0x7fe6081f8110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:00.304 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.295+0000 7fe60f579640 1 --2- 172.21.15.16:0/1361157539 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe6080ce480 0x7fe6081f8110 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7fe5f80ad040 tx=0x7fe5f801ac90 comp rx=0 tx=0).stop 2023-05-22T15:28:00.304 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.295+0000 7fe60f579640 1 -- 172.21.15.16:0/1361157539 shutdown_connections 2023-05-22T15:28:00.304 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.295+0000 7fe60f579640 1 --2- 172.21.15.16:0/1361157539 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fe5ec065380 0x7fe5ec067830 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:00.305 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.295+0000 7fe60f579640 1 --2- 172.21.15.16:0/1361157539 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe6081605d0 0x7fe60814c450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:00.305 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.295+0000 7fe60f579640 1 --2- 172.21.15.16:0/1361157539 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe6080ce480 0x7fe6081f8110 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:00.307 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.295+0000 7fe60f579640 1 --2- 172.21.15.16:0/1361157539 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe608071640 0x7fe6080ab9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:00.308 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.295+0000 7fe60f579640 1 -- 172.21.15.16:0/1361157539 >> 172.21.15.16:0/1361157539 conn(0x7fe6080cc870 msgr2=0x7fe60815d910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:28:00.309 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.295+0000 7fe60f579640 1 -- 172.21.15.16:0/1361157539 shutdown_connections 2023-05-22T15:28:00.311 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.295+0000 7fe60f579640 1 -- 172.21.15.16:0/1361157539 wait complete. 2023-05-22T15:28:00.319 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-05-22T15:28:00.445 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.435+0000 7f98a5ccb640 1 Processor -- start 2023-05-22T15:28:00.445 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.435+0000 7f98a5ccb640 1 -- start start 2023-05-22T15:28:00.450 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.439+0000 7f98a5ccb640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f98a00ca4e0 0x7f98a015e460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:00.450 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.439+0000 7f98a5ccb640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f98a0070df0 0x7f98a0071210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:00.450 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.439+0000 7f98a5ccb640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98a00b0650 con 0x7f98a00c9b40 2023-05-22T15:28:00.451 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.439+0000 7f98a5ccb640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98a00b0370 con 0x7f98a00ca4e0 2023-05-22T15:28:00.452 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.439+0000 7f98a5ccb640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98a015d980 con 0x7f98a0070df0 2023-05-22T15:28:00.452 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.439+0000 7f989effd640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f98a00c9b40 0x7f98a00afb40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:50102/0 (socket says 172.21.15.16:50102) 2023-05-22T15:28:00.452 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.439+0000 7f989effd640 1 -- 172.21.15.16:0/1000788947 learned_addr learned my addr 172.21.15.16:0/1000788947 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:28:00.452 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.439+0000 7f989ffff640 1 --2- 172.21.15.16:0/1000788947 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f98a0070df0 0x7f98a0071210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:00.453 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.439+0000 7f989f7fe640 1 --2- 172.21.15.16:0/1000788947 >> v2:172.21.15.62:3300/0 conn(0x7f98a00ca4e0 0x7f98a015e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:00.453 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.439+0000 7f989e7fc640 1 -- 172.21.15.16:0/1000788947 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2688244706 0 0) 0x7f98a00b0650 con 0x7f98a00c9b40 2023-05-22T15:28:00.453 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.439+0000 7f989e7fc640 1 -- 172.21.15.16:0/1000788947 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f988c003530 con 0x7f98a00c9b40 2023-05-22T15:28:00.453 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.439+0000 7f989e7fc640 1 -- 172.21.15.16:0/1000788947 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3122904069 0 0) 0x7f988c003530 con 0x7f98a00c9b40 2023-05-22T15:28:00.453 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.439+0000 7f989e7fc640 1 -- 172.21.15.16:0/1000788947 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f98a0070df0 msgr2=0x7f98a0071210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:28:00.453 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.439+0000 7f989e7fc640 1 --2- 172.21.15.16:0/1000788947 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f98a0070df0 0x7f98a0071210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:00.454 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.439+0000 7f989e7fc640 1 -- 172.21.15.16:0/1000788947 >> v2:172.21.15.62:3300/0 conn(0x7f98a00ca4e0 msgr2=0x7f98a015e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:28:00.454 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.439+0000 7f989e7fc640 1 --2- 172.21.15.16:0/1000788947 >> v2:172.21.15.62:3300/0 conn(0x7f98a00ca4e0 0x7f98a015e460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:00.454 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.439+0000 7f989e7fc640 1 -- 172.21.15.16:0/1000788947 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98a015dc60 con 0x7f98a00c9b40 2023-05-22T15:28:00.454 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.439+0000 7f989e7fc640 1 -- 172.21.15.16:0/1000788947 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7f98880016a0 con 0x7f98a00c9b40 2023-05-22T15:28:00.455 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.439+0000 7f989e7fc640 1 -- 172.21.15.16:0/1000788947 >> v1:172.21.15.16:6789/0 conn(0x7f98a00c9b40 legacy=0x7f98a00afb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:00.455 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.439+0000 7f989e7fc640 1 --2- 172.21.15.16:0/1000788947 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f988c003980 0x7f988c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:00.455 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.439+0000 7f989e7fc640 1 --2- 172.21.15.16:0/1000788947 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f988c0043b0 0x7f988c004870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:00.456 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.439+0000 7f989e7fc640 1 --2- 172.21.15.16:0/1000788947 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f98a0070df0 0x7f988c004f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:00.456 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.439+0000 7f989e7fc640 1 -- 172.21.15.16:0/1000788947 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98a00b0370 con 0x7f988c0043b0 2023-05-22T15:28:00.456 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.439+0000 7f989e7fc640 1 -- 172.21.15.16:0/1000788947 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98a015d980 con 0x7f988c003980 2023-05-22T15:28:00.457 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.439+0000 7f989e7fc640 1 -- 172.21.15.16:0/1000788947 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f988c00b990 con 0x7f98a0070df0 2023-05-22T15:28:00.457 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.439+0000 7f989effd640 1 --2- 172.21.15.16:0/1000788947 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f988c0043b0 0x7f988c004870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:00.457 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.439+0000 7f989f7fe640 1 --2- 172.21.15.16:0/1000788947 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f988c003980 0x7f988c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:00.458 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.439+0000 7f989effd640 1 -- 172.21.15.16:0/1000788947 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f98a0070df0 msgr2=0x7f988c004f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:28:00.458 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.439+0000 7f989effd640 1 --2- 172.21.15.16:0/1000788947 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f98a0070df0 0x7f988c004f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:00.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.439+0000 7f989effd640 1 -- 172.21.15.16:0/1000788947 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f988c003980 msgr2=0x7f988c003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:28:00.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.439+0000 7f989effd640 1 --2- 172.21.15.16:0/1000788947 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f988c003980 0x7f988c003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:00.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.439+0000 7f989effd640 1 -- 172.21.15.16:0/1000788947 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f988c003530 con 0x7f988c0043b0 2023-05-22T15:28:00.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.443+0000 7f989effd640 1 --2- 172.21.15.16:0/1000788947 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f988c0043b0 0x7f988c004870 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f988800c210 tx=0x7f988801f390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8caca42781ea05c2 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:28:00.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.443+0000 7f989e7fc640 1 -- 172.21.15.16:0/1000788947 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9888005ae0 con 0x7f988c0043b0 2023-05-22T15:28:00.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.443+0000 7f989e7fc640 1 -- 172.21.15.16:0/1000788947 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98880ab070 con 0x7f988c0043b0 2023-05-22T15:28:00.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.443+0000 7f989e7fc640 1 -- 172.21.15.16:0/1000788947 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9888005ea0 con 0x7f988c0043b0 2023-05-22T15:28:00.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.443+0000 7f98a5ccb640 1 -- 172.21.15.16:0/1000788947 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f988c0043b0 msgr2=0x7f988c004870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:00.460 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.443+0000 7f98a5ccb640 1 --2- 172.21.15.16:0/1000788947 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f988c0043b0 0x7f988c004870 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f988800c210 tx=0x7f988801f390 comp rx=0 tx=0).stop 2023-05-22T15:28:00.460 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.443+0000 7f989ffff640 1 -- 172.21.15.16:0/1000788947 reap_dead start 2023-05-22T15:28:00.460 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.443+0000 7f98a5ccb640 1 -- 172.21.15.16:0/1000788947 shutdown_connections 2023-05-22T15:28:00.460 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.443+0000 7f98a5ccb640 1 -- 172.21.15.16:0/1000788947 >> 172.21.15.16:0/1000788947 conn(0x7f98a0069720 msgr2=0x7f98a0069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:28:00.460 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.443+0000 7f98a5ccb640 1 -- 172.21.15.16:0/1000788947 shutdown_connections 2023-05-22T15:28:00.461 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.443+0000 7f98a5ccb640 1 -- 172.21.15.16:0/1000788947 wait complete. 2023-05-22T15:28:00.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.443+0000 7f98a5ccb640 1 Processor -- start 2023-05-22T15:28:00.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.443+0000 7f98a5ccb640 1 -- start start 2023-05-22T15:28:00.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.443+0000 7f98a5ccb640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f988c0043b0 0x7f98a01554c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:00.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.443+0000 7f98a5ccb640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f98a014e280 0x7f98a00c1720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:00.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.443+0000 7f98a5ccb640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f98a00c1c60 0x7f98a00c20d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:00.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.443+0000 7f98a5ccb640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98a0154e20 con 0x7f98a014e280 2023-05-22T15:28:00.464 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.443+0000 7f98a5ccb640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98a014ed40 con 0x7f98a00c1c60 2023-05-22T15:28:00.464 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.443+0000 7f98a5ccb640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98a0155a00 con 0x7f988c0043b0 2023-05-22T15:28:00.464 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.443+0000 7f989effd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f98a014e280 0x7f98a00c1720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:00.464 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.443+0000 7f989effd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f98a014e280 0x7f98a00c1720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39668/0 (socket says 172.21.15.16:39668) 2023-05-22T15:28:00.464 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.443+0000 7f989effd640 1 -- 172.21.15.16:0/2606038204 learned_addr learned my addr 172.21.15.16:0/2606038204 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:28:00.464 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.443+0000 7f989ffff640 1 --2- 172.21.15.16:0/2606038204 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f98a00c1c60 0x7f98a00c20d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:00.464 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.443+0000 7f989f7fe640 1 --2- 172.21.15.16:0/2606038204 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f988c0043b0 0x7f98a01554c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:00.465 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.443+0000 7f989effd640 1 -- 172.21.15.16:0/2606038204 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f988c0043b0 msgr2=0x7f98a01554c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:00.465 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.443+0000 7f989effd640 1 --2- 172.21.15.16:0/2606038204 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f988c0043b0 0x7f98a01554c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:00.465 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.443+0000 7f989effd640 1 -- 172.21.15.16:0/2606038204 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f98a00c1c60 msgr2=0x7f98a00c20d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:00.465 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.443+0000 7f989effd640 1 --2- 172.21.15.16:0/2606038204 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f98a00c1c60 0x7f98a00c20d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:00.465 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.443+0000 7f989effd640 1 -- 172.21.15.16:0/2606038204 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98a00ca8b0 con 0x7f98a014e280 2023-05-22T15:28:00.465 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.443+0000 7f989effd640 1 --2- 172.21.15.16:0/2606038204 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f98a014e280 0x7f98a00c1720 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f988801f3d0 tx=0x7f988c005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:28:00.465 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.443+0000 7f9897fff640 1 -- 172.21.15.16:0/2606038204 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98880b7070 con 0x7f98a014e280 2023-05-22T15:28:00.465 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.447+0000 7f9897fff640 1 -- 172.21.15.16:0/2606038204 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9888001c80 con 0x7f98a014e280 2023-05-22T15:28:00.466 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.447+0000 7f98a5ccb640 1 -- 172.21.15.16:0/2606038204 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98a015df40 con 0x7f98a014e280 2023-05-22T15:28:00.466 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.447+0000 7f9897fff640 1 -- 172.21.15.16:0/2606038204 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9888001680 con 0x7f98a014e280 2023-05-22T15:28:00.466 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.447+0000 7f98a5ccb640 1 -- 172.21.15.16:0/2606038204 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98a0071210 con 0x7f98a014e280 2023-05-22T15:28:00.466 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.447+0000 7f98a5ccb640 1 -- 172.21.15.16:0/2606038204 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98a004dbe0 con 0x7f98a014e280 2023-05-22T15:28:00.466 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.447+0000 7f9897fff640 1 -- 172.21.15.16:0/2606038204 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7f98880ad020 con 0x7f98a014e280 2023-05-22T15:28:00.466 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.447+0000 7f9897fff640 1 --2- 172.21.15.16:0/2606038204 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f98800653b0 0x7f9880067860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:00.466 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.447+0000 7f9897fff640 1 -- 172.21.15.16:0/2606038204 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f988811f320 con 0x7f98a014e280 2023-05-22T15:28:00.467 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.447+0000 7f989f7fe640 1 --2- 172.21.15.16:0/2606038204 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f98800653b0 0x7f9880067860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:00.467 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.447+0000 7f989f7fe640 1 --2- 172.21.15.16:0/2606038204 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f98800653b0 0x7f9880067860 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f9890009a60 tx=0x7f9890008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:28:00.467 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.451+0000 7f9897fff640 1 -- 172.21.15.16:0/2606038204 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f98880ecd20 con 0x7f98a014e280 2023-05-22T15:28:00.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.599+0000 7f98a5ccb640 1 -- 172.21.15.16:0/2606038204 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f98a00cb010 con 0x7f98a014e280 2023-05-22T15:28:00.611 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.603+0000 7f9897fff640 1 -- 172.21.15.16:0/2606038204 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v29) v1 ==== 74+0+19700 (secure 0 0 0) 0x7f9888005e60 con 0x7f98a014e280 2023-05-22T15:28:00.612 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:28:00.612 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":29,"fsid":"b2719834-7847-4d98-90a6-f65d57f85c49","created":"2023-05-22T15:26:25.259314+0000","modified":"2023-05-22T15:27:31.182317+0000","last_up_change":"2023-05-22T15:26:36.448524+0000","last_in_change":"2023-05-22T15:26:27.675668+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-05-22T15:26:38.019781+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-05-22T15:26:43.570514+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-05-22T15:26:47.879159+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-05-22T15:26:48.522130+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data_ec","create_time":"2023-05-22T15:26:51.616142+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"9e7ff854-b25f-4cbb-a5c2-1799d9c53d2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":37021},{"type":"v1","addr":"172.21.15.16:6825","nonce":37021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":37021},{"type":"v1","addr":"172.21.15.16:6827","nonce":37021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":37021},{"type":"v1","addr":"172.21.15.16:6831","nonce":37021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":37021},{"type":"v1","addr":"172.21.15.16:6829","nonce":37021}]},"public_addr":"172.21.15.16:6825/37021","cluster_addr":"172.21.15.16:6827/37021","heartbeat_back_addr":"172.21.15.16:6831/37021","heartbeat_front_addr":"172.21.15.16:6829/37021","state":["exists","up"]},{"osd":1,"uuid":"4702fbc9-a927-4565-aca5-41dc69cc8264","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":37008},{"type":"v1","addr":"172.21.15.16:6801","nonce":37008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":37008},{"type":"v1","addr":"172.21.15.16:6803","nonce":37008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":37008},{"type":"v1","addr":"172.21.15.16:6807","nonce":37008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":37008},{"type":"v1","addr":"172.21.15.16:6805","nonce":37008}]},"public_addr":"172.21.15.16:6801/37008","cluster_addr":"172.21.15.16:6803/37008","heartbeat_back_addr":"172.21.15.16:6807/37008","heartbeat_front_addr":"172.21.15.16:6805/37008","state":["exists","up"]},{"osd":2,"uuid":"1e7bba46-4eba-47ef-bb81-5e2d371638ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":37009},{"type":"v1","addr":"172.21.15.16:6809","nonce":37009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":37009},{"type":"v1","addr":"172.21.15.16:6811","nonce":37009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":37009},{"type":"v1","addr":"172.21.15.16:6815","nonce":37009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":37009},{"type":"v1","addr":"172.21.15.16:6813","nonce":37009}]},"public_addr":"172.21.15.16:6809/37009","cluster_addr":"172.21.15.16:6811/37009","heartbeat_back_addr":"172.21.15.16:6815/37009","heartbeat_front_addr":"172.21.15.16:6813/37009","state":["exists","up"]},{"osd":3,"uuid":"383cacc0-5a88-4904-990e-498ca59ef48a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":37020},{"type":"v1","addr":"172.21.15.16:6817","nonce":37020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":37020},{"type":"v1","addr":"172.21.15.16:6819","nonce":37020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":37020},{"type":"v1","addr":"172.21.15.16:6823","nonce":37020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":37020},{"type":"v1","addr":"172.21.15.16:6821","nonce":37020}]},"public_addr":"172.21.15.16:6817/37020","cluster_addr":"172.21.15.16:6819/37020","heartbeat_back_addr":"172.21.15.16:6823/37020","heartbeat_front_addr":"172.21.15.16:6821/37020","state":["exists","up"]},{"osd":4,"uuid":"970500ef-95b7-42c1-b080-eed4fae7013d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":37353},{"type":"v1","addr":"172.21.15.62:6817","nonce":37353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6819","nonce":37353},{"type":"v1","addr":"172.21.15.62:6822","nonce":37353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37353},{"type":"v1","addr":"172.21.15.62:6831","nonce":37353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37353},{"type":"v1","addr":"172.21.15.62:6826","nonce":37353}]},"public_addr":"172.21.15.62:6817/37353","cluster_addr":"172.21.15.62:6822/37353","heartbeat_back_addr":"172.21.15.62:6831/37353","heartbeat_front_addr":"172.21.15.62:6826/37353","state":["exists","up"]},{"osd":5,"uuid":"022d67db-d4d8-445b-b1b9-8b446a00f5c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":37352},{"type":"v1","addr":"172.21.15.62:6801","nonce":37352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":37352},{"type":"v1","addr":"172.21.15.62:6803","nonce":37352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":37352},{"type":"v1","addr":"172.21.15.62:6807","nonce":37352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":37352},{"type":"v1","addr":"172.21.15.62:6805","nonce":37352}]},"public_addr":"172.21.15.62:6801/37352","cluster_addr":"172.21.15.62:6803/37352","heartbeat_back_addr":"172.21.15.62:6807/37352","heartbeat_front_addr":"172.21.15.62:6805/37352","state":["exists","up"]},{"osd":6,"uuid":"42bd372b-6688-4a1b-8407-e16e8c266e33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":37354},{"type":"v1","addr":"172.21.15.62:6820","nonce":37354}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":37354},{"type":"v1","addr":"172.21.15.62:6823","nonce":37354}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6829","nonce":37354},{"type":"v1","addr":"172.21.15.62:6830","nonce":37354}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6825","nonce":37354},{"type":"v1","addr":"172.21.15.62:6827","nonce":37354}]},"public_addr":"172.21.15.62:6820/37354","cluster_addr":"172.21.15.62:6823/37354","heartbeat_back_addr":"172.21.15.62:6830/37354","heartbeat_front_addr":"172.21.15.62:6827/37354","state":["exists","up"]},{"osd":7,"uuid":"ac70b2a3-f41b-4bf2-b227-7f2cbb55426d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37355},{"type":"v1","addr":"172.21.15.62:6809","nonce":37355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37355},{"type":"v1","addr":"172.21.15.62:6811","nonce":37355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":37355},{"type":"v1","addr":"172.21.15.62:6815","nonce":37355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":37355},{"type":"v1","addr":"172.21.15.62:6813","nonce":37355}]},"public_addr":"172.21.15.62:6809/37355","cluster_addr":"172.21.15.62:6811/37355","heartbeat_back_addr":"172.21.15.62:6815/37355","heartbeat_front_addr":"172.21.15.62:6813/37355","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.368896+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.107273+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.276888+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.287502+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.124791+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.251937+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.255679+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.232797+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"4.23","mappings":[{"from":3,"to":2}]},{"pgid":"4.31","mappings":[{"from":1,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-05-22T15:28:00.614 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.607+0000 7f98a5ccb640 1 -- 172.21.15.16:0/2606038204 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f98800653b0 msgr2=0x7f9880067860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:00.614 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.607+0000 7f98a5ccb640 1 --2- 172.21.15.16:0/2606038204 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f98800653b0 0x7f9880067860 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f9890009a60 tx=0x7f9890008040 comp rx=0 tx=0).stop 2023-05-22T15:28:00.614 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.607+0000 7f98a5ccb640 1 -- 172.21.15.16:0/2606038204 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f98a014e280 msgr2=0x7f98a00c1720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:00.615 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.607+0000 7f98a5ccb640 1 --2- 172.21.15.16:0/2606038204 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f98a014e280 0x7f98a00c1720 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f988801f3d0 tx=0x7f988c005800 comp rx=0 tx=0).stop 2023-05-22T15:28:00.615 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.607+0000 7f98a5ccb640 1 -- 172.21.15.16:0/2606038204 shutdown_connections 2023-05-22T15:28:00.616 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.607+0000 7f98a5ccb640 1 --2- 172.21.15.16:0/2606038204 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f98a00c1c60 0x7f98a00c20d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:00.616 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.607+0000 7f98a5ccb640 1 --2- 172.21.15.16:0/2606038204 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f98a014e280 0x7f98a00c1720 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:00.616 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.607+0000 7f98a5ccb640 1 --2- 172.21.15.16:0/2606038204 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f98800653b0 0x7f9880067860 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:00.616 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.607+0000 7f98a5ccb640 1 --2- 172.21.15.16:0/2606038204 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f988c0043b0 0x7f98a01554c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:00.617 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.607+0000 7f98a5ccb640 1 -- 172.21.15.16:0/2606038204 >> 172.21.15.16:0/2606038204 conn(0x7f98a0069720 msgr2=0x7f98a00cdf20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:28:00.617 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.607+0000 7f98a5ccb640 1 -- 172.21.15.16:0/2606038204 shutdown_connections 2023-05-22T15:28:00.617 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.607+0000 7f98a5ccb640 1 -- 172.21.15.16:0/2606038204 wait complete. 2023-05-22T15:28:00.628 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-05-22T15:28:00.629 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-05-22T15:28:00.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.743+0000 7f4a54ce3640 1 Processor -- start 2023-05-22T15:28:00.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.743+0000 7f4a54ce3640 1 -- start start 2023-05-22T15:28:00.753 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.747+0000 7f4a54ce3640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f4a50158660 0x7f4a500cdfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:00.753 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.747+0000 7f4a54ce3640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f4a500cce20 0x7f4a500ce4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:00.754 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.747+0000 7f4a54ce3640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a500c92b0 con 0x7f4a50158290 2023-05-22T15:28:00.754 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.747+0000 7f4a54ce3640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a500cc060 con 0x7f4a500cce20 2023-05-22T15:28:00.754 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.747+0000 7f4a54ce3640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a500cc340 con 0x7f4a50158660 2023-05-22T15:28:00.754 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.747+0000 7f4a4e59c640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f4a50158290 0x7f4a500c8880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:50110/0 (socket says 172.21.15.16:50110) 2023-05-22T15:28:00.754 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.747+0000 7f4a4e59c640 1 -- 172.21.15.16:0/4182982324 learned_addr learned my addr 172.21.15.16:0/4182982324 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:28:00.754 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.747+0000 7f4a4ed9d640 1 --2- 172.21.15.16:0/4182982324 >> v2:172.21.15.16:3300/0 conn(0x7f4a500cce20 0x7f4a500ce4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:00.754 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.747+0000 7f4a4dd9b640 1 --2- 172.21.15.16:0/4182982324 >> v2:172.21.15.62:3300/0 conn(0x7f4a50158660 0x7f4a500cdfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:00.755 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.747+0000 7f4a4ed9d640 1 -- 172.21.15.16:0/4182982324 >> v2:172.21.15.62:3300/0 conn(0x7f4a50158660 msgr2=0x7f4a500cdfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:28:00.755 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.747+0000 7f4a4ed9d640 1 --2- 172.21.15.16:0/4182982324 >> v2:172.21.15.62:3300/0 conn(0x7f4a50158660 0x7f4a500cdfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:00.755 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.747+0000 7f4a4ed9d640 1 -- 172.21.15.16:0/4182982324 >> v1:172.21.15.16:6789/0 conn(0x7f4a50158290 legacy=0x7f4a500c8880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:00.755 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.747+0000 7f4a4ed9d640 1 -- 172.21.15.16:0/4182982324 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a500cc620 con 0x7f4a500cce20 2023-05-22T15:28:00.756 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.747+0000 7f4a4ed9d640 1 --2- 172.21.15.16:0/4182982324 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4a500cce20 0x7f4a500ce4f0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f4a5006d810 tx=0x7f4a4001e260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a93718c41bb2c08 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:28:00.756 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.747+0000 7f4a4d59a640 1 -- 172.21.15.16:0/4182982324 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a400a6870 con 0x7f4a500cce20 2023-05-22T15:28:00.756 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.747+0000 7f4a4d59a640 1 -- 172.21.15.16:0/4182982324 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a400a69d0 con 0x7f4a500cce20 2023-05-22T15:28:00.756 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.747+0000 7f4a4d59a640 1 -- 172.21.15.16:0/4182982324 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a400a6d60 con 0x7f4a500cce20 2023-05-22T15:28:00.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.751+0000 7f4a54ce3640 1 -- 172.21.15.16:0/4182982324 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4a500cce20 msgr2=0x7f4a500ce4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:00.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.751+0000 7f4a54ce3640 1 --2- 172.21.15.16:0/4182982324 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4a500cce20 0x7f4a500ce4f0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f4a5006d810 tx=0x7f4a4001e260 comp rx=0 tx=0).stop 2023-05-22T15:28:00.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.751+0000 7f4a54ce3640 1 -- 172.21.15.16:0/4182982324 shutdown_connections 2023-05-22T15:28:00.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.751+0000 7f4a54ce3640 1 --2- 172.21.15.16:0/4182982324 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4a500cce20 0x7f4a500ce4f0 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:00.758 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.751+0000 7f4a54ce3640 1 --2- 172.21.15.16:0/4182982324 >> v2:172.21.15.62:3300/0 conn(0x7f4a50158660 0x7f4a500cdfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:00.758 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.751+0000 7f4a54ce3640 1 -- 172.21.15.16:0/4182982324 >> 172.21.15.16:0/4182982324 conn(0x7f4a50069720 msgr2=0x7f4a50069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:28:00.758 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.751+0000 7f4a54ce3640 1 -- 172.21.15.16:0/4182982324 shutdown_connections 2023-05-22T15:28:00.758 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.751+0000 7f4a54ce3640 1 -- 172.21.15.16:0/4182982324 wait complete. 2023-05-22T15:28:00.759 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.751+0000 7f4a54ce3640 1 Processor -- start 2023-05-22T15:28:00.760 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.751+0000 7f4a54ce3640 1 -- start start 2023-05-22T15:28:00.760 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.751+0000 7f4a54ce3640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4a500cce20 0x7f4a50156200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:00.760 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.751+0000 7f4a54ce3640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4a50158290 0x7f4a50151f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:00.760 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.751+0000 7f4a54ce3640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4a50158660 0x7f4a50152470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:00.760 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.751+0000 7f4a54ce3640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a50156ae0 con 0x7f4a50158290 2023-05-22T15:28:00.760 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.751+0000 7f4a54ce3640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a5014a530 con 0x7f4a500cce20 2023-05-22T15:28:00.761 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.751+0000 7f4a54ce3640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a5014a810 con 0x7f4a50158660 2023-05-22T15:28:00.761 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.751+0000 7f4a4dd9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4a50158290 0x7f4a50151f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:00.761 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.751+0000 7f4a4dd9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4a50158290 0x7f4a50151f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39676/0 (socket says 172.21.15.16:39676) 2023-05-22T15:28:00.761 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.751+0000 7f4a4dd9b640 1 -- 172.21.15.16:0/3424989471 learned_addr learned my addr 172.21.15.16:0/3424989471 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:28:00.761 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.751+0000 7f4a4e59c640 1 --2- 172.21.15.16:0/3424989471 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4a500cce20 0x7f4a50156200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:00.761 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.751+0000 7f4a4ed9d640 1 --2- 172.21.15.16:0/3424989471 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4a50158660 0x7f4a50152470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:00.761 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.751+0000 7f4a4dd9b640 1 -- 172.21.15.16:0/3424989471 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4a50158660 msgr2=0x7f4a50152470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:00.761 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.751+0000 7f4a4dd9b640 1 --2- 172.21.15.16:0/3424989471 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4a50158660 0x7f4a50152470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:00.762 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.751+0000 7f4a4dd9b640 1 -- 172.21.15.16:0/3424989471 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4a500cce20 msgr2=0x7f4a50156200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:00.762 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.751+0000 7f4a4dd9b640 1 --2- 172.21.15.16:0/3424989471 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4a500cce20 0x7f4a50156200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:00.762 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.751+0000 7f4a4dd9b640 1 -- 172.21.15.16:0/3424989471 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a4000a5c0 con 0x7f4a50158290 2023-05-22T15:28:00.762 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.751+0000 7f4a4e59c640 1 --2- 172.21.15.16:0/3424989471 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4a500cce20 0x7f4a50156200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:28:00.762 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.751+0000 7f4a4dd9b640 1 --2- 172.21.15.16:0/3424989471 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4a50158290 0x7f4a50151f30 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f4a3800d5f0 tx=0x7f4a3800dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:28:00.762 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.751+0000 7f4a46ffd640 1 -- 172.21.15.16:0/3424989471 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a380125f0 con 0x7f4a50158290 2023-05-22T15:28:00.762 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.751+0000 7f4a46ffd640 1 -- 172.21.15.16:0/3424989471 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a38012750 con 0x7f4a50158290 2023-05-22T15:28:00.763 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.751+0000 7f4a54ce3640 1 -- 172.21.15.16:0/3424989471 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a5015cea0 con 0x7f4a50158290 2023-05-22T15:28:00.763 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.751+0000 7f4a46ffd640 1 -- 172.21.15.16:0/3424989471 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a38012b00 con 0x7f4a50158290 2023-05-22T15:28:00.763 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.751+0000 7f4a54ce3640 1 -- 172.21.15.16:0/3424989471 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a5015cd10 con 0x7f4a50158290 2023-05-22T15:28:00.763 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.751+0000 7f4a54ce3640 1 -- 172.21.15.16:0/3424989471 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a5006f630 con 0x7f4a50158290 2023-05-22T15:28:00.764 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.755+0000 7f4a46ffd640 1 -- 172.21.15.16:0/3424989471 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7f4a38012c60 con 0x7f4a50158290 2023-05-22T15:28:00.764 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.755+0000 7f4a46ffd640 1 --2- 172.21.15.16:0/3424989471 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f4a30065380 0x7f4a30067830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:00.764 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.755+0000 7f4a46ffd640 1 -- 172.21.15.16:0/3424989471 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f4a38083090 con 0x7f4a50158290 2023-05-22T15:28:00.764 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.755+0000 7f4a4e59c640 1 --2- 172.21.15.16:0/3424989471 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f4a30065380 0x7f4a30067830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:00.764 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.755+0000 7f4a4e59c640 1 --2- 172.21.15.16:0/3424989471 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f4a30065380 0x7f4a30067830 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f4a48005c10 tx=0x7f4a48009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:28:00.765 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.759+0000 7f4a46ffd640 1 -- 172.21.15.16:0/3424989471 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f4a38050a70 con 0x7f4a50158290 2023-05-22T15:28:00.956 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.947+0000 7f4a54ce3640 1 -- 172.21.15.16:0/3424989471 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4a5004dbe0 con 0x7f4a50158290 2023-05-22T15:28:00.957 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.947+0000 7f4a46ffd640 1 -- 172.21.15.16:0/3424989471 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4641 (secure 0 0 0) 0x7f4a38054090 con 0x7f4a50158290 2023-05-22T15:28:00.957 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:28:00.957 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4261,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6834/3661808159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":3661808159},{"type":"v1","addr":"172.21.15.62:6834","nonce":3661808159}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4264,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/2227467222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6833","nonce":2227467222},{"type":"v1","addr":"172.21.15.62:6835","nonce":2227467222}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4284,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6836/2808106087","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2808106087},{"type":"v1","addr":"172.21.15.16:6836","nonce":2808106087}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-05-22T15:26:49.546008+0000","modified":"2023-05-22T15:27:06.047204+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4287},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4287":{"gid":4287,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.16:6837/283322289","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6835","nonce":283322289},{"type":"v1","addr":"172.21.15.16:6837","nonce":283322289}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-05-22T15:28:00.958 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 8 2023-05-22T15:28:00.958 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.951+0000 7f4a54ce3640 1 -- 172.21.15.16:0/3424989471 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f4a30065380 msgr2=0x7f4a30067830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:00.958 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.951+0000 7f4a54ce3640 1 --2- 172.21.15.16:0/3424989471 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f4a30065380 0x7f4a30067830 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f4a48005c10 tx=0x7f4a48009040 comp rx=0 tx=0).stop 2023-05-22T15:28:00.959 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.951+0000 7f4a54ce3640 1 -- 172.21.15.16:0/3424989471 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4a50158290 msgr2=0x7f4a50151f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:00.959 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.951+0000 7f4a54ce3640 1 --2- 172.21.15.16:0/3424989471 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4a50158290 0x7f4a50151f30 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f4a3800d5f0 tx=0x7f4a3800dab0 comp rx=0 tx=0).stop 2023-05-22T15:28:00.959 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.951+0000 7f4a54ce3640 1 -- 172.21.15.16:0/3424989471 shutdown_connections 2023-05-22T15:28:00.959 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.951+0000 7f4a54ce3640 1 --2- 172.21.15.16:0/3424989471 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f4a30065380 0x7f4a30067830 secure :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f4a48005c10 tx=0x7f4a48009040 comp rx=0 tx=0).stop 2023-05-22T15:28:00.959 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.951+0000 7f4a54ce3640 1 --2- 172.21.15.16:0/3424989471 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4a50158660 0x7f4a50152470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:00.959 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.951+0000 7f4a54ce3640 1 --2- 172.21.15.16:0/3424989471 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4a50158290 0x7f4a50151f30 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:00.959 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.951+0000 7f4a54ce3640 1 --2- 172.21.15.16:0/3424989471 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4a500cce20 0x7f4a50156200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:00.960 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.951+0000 7f4a54ce3640 1 -- 172.21.15.16:0/3424989471 >> 172.21.15.16:0/3424989471 conn(0x7f4a50069720 msgr2=0x7f4a500cd1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:28:00.960 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.951+0000 7f4a54ce3640 1 -- 172.21.15.16:0/3424989471 shutdown_connections 2023-05-22T15:28:00.960 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:00.951+0000 7f4a54ce3640 1 -- 172.21.15.16:0/3424989471 wait complete. 2023-05-22T15:28:00.978 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-05-22T15:28:01.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.091+0000 7f0962caf640 1 Processor -- start 2023-05-22T15:28:01.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.095+0000 7f0962caf640 1 -- start start 2023-05-22T15:28:01.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.095+0000 7f0962caf640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f095c0ca4e0 0x7f095c15e460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:01.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.095+0000 7f0962caf640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f095c0c9b40 0x7f095c0afb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:01.103 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.095+0000 7f0962caf640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f095c0b0650 con 0x7f095c070df0 2023-05-22T15:28:01.103 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.095+0000 7f0962caf640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f095c0b0370 con 0x7f095c0ca4e0 2023-05-22T15:28:01.103 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.095+0000 7f0962caf640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f095c15d980 con 0x7f095c0c9b40 2023-05-22T15:28:01.103 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.095+0000 7f0960a4b640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f095c0ca4e0 0x7f095c15e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:01.103 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.095+0000 7f0960a4b640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f095c0ca4e0 0x7f095c15e460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39690/0 (socket says 172.21.15.16:39690) 2023-05-22T15:28:01.103 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.095+0000 7f0960a4b640 1 -- 172.21.15.16:0/1110533605 learned_addr learned my addr 172.21.15.16:0/1110533605 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:28:01.103 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.095+0000 7f096124c640 1 --1- 172.21.15.16:0/1110533605 >> v1:172.21.15.16:6789/0 conn(0x7f095c070df0 0x7f095c071210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:50118/0 (socket says 172.21.15.16:50118) 2023-05-22T15:28:01.104 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.095+0000 7f095bfff640 1 --2- 172.21.15.16:0/1110533605 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f095c0c9b40 0x7f095c0afb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:01.104 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.095+0000 7f0960a4b640 1 -- 172.21.15.16:0/1110533605 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f095c0c9b40 msgr2=0x7f095c0afb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:28:01.104 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.095+0000 7f0960a4b640 1 --2- 172.21.15.16:0/1110533605 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f095c0c9b40 0x7f095c0afb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:01.104 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.095+0000 7f0960a4b640 1 -- 172.21.15.16:0/1110533605 >> v1:172.21.15.16:6789/0 conn(0x7f095c070df0 legacy=0x7f095c071210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:01.104 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.095+0000 7f0960a4b640 1 -- 172.21.15.16:0/1110533605 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f095c15dc60 con 0x7f095c0ca4e0 2023-05-22T15:28:01.107 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.099+0000 7f0960a4b640 1 --2- 172.21.15.16:0/1110533605 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f095c0ca4e0 0x7f095c15e460 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f094c009870 tx=0x7f094c01c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ca660482b5daff8 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:28:01.108 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.099+0000 7f095b7fe640 1 -- 172.21.15.16:0/1110533605 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f094c0ab660 con 0x7f095c0ca4e0 2023-05-22T15:28:01.108 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.099+0000 7f095b7fe640 1 -- 172.21.15.16:0/1110533605 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f094c0ab7c0 con 0x7f095c0ca4e0 2023-05-22T15:28:01.108 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.099+0000 7f095b7fe640 1 -- 172.21.15.16:0/1110533605 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f094c0abb50 con 0x7f095c0ca4e0 2023-05-22T15:28:01.109 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.099+0000 7f0962caf640 1 -- 172.21.15.16:0/1110533605 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f095c0ca4e0 msgr2=0x7f095c15e460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:01.109 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.099+0000 7f0962caf640 1 --2- 172.21.15.16:0/1110533605 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f095c0ca4e0 0x7f095c15e460 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f094c009870 tx=0x7f094c01c480 comp rx=0 tx=0).stop 2023-05-22T15:28:01.109 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.099+0000 7f0962caf640 1 -- 172.21.15.16:0/1110533605 shutdown_connections 2023-05-22T15:28:01.109 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.099+0000 7f0962caf640 1 --2- 172.21.15.16:0/1110533605 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f095c0c9b40 0x7f095c0afb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:01.110 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.099+0000 7f0962caf640 1 --2- 172.21.15.16:0/1110533605 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f095c0ca4e0 0x7f095c15e460 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:01.110 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.099+0000 7f0962caf640 1 -- 172.21.15.16:0/1110533605 >> 172.21.15.16:0/1110533605 conn(0x7f095c069720 msgr2=0x7f095c069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:28:01.110 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.099+0000 7f0962caf640 1 -- 172.21.15.16:0/1110533605 shutdown_connections 2023-05-22T15:28:01.110 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.099+0000 7f0962caf640 1 -- 172.21.15.16:0/1110533605 wait complete. 2023-05-22T15:28:01.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.099+0000 7f0962caf640 1 Processor -- start 2023-05-22T15:28:01.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.099+0000 7f0962caf640 1 -- start start 2023-05-22T15:28:01.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.099+0000 7f0962caf640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f095c070df0 0x7f095c0cbe90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:01.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.099+0000 7f0962caf640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f095c0c9b40 0x7f095c0ccd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:01.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.099+0000 7f0962caf640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f095c0ca4e0 0x7f095c069af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:01.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.099+0000 7f0962caf640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f095c0ac7a0 con 0x7f095c070df0 2023-05-22T15:28:01.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.099+0000 7f0962caf640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f095c0abb20 con 0x7f095c0ca4e0 2023-05-22T15:28:01.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.099+0000 7f0962caf640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f095c0abe00 con 0x7f095c0c9b40 2023-05-22T15:28:01.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.099+0000 7f0960a4b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f095c070df0 0x7f095c0cbe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:01.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.099+0000 7f0960a4b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f095c070df0 0x7f095c0cbe90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39696/0 (socket says 172.21.15.16:39696) 2023-05-22T15:28:01.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.099+0000 7f0960a4b640 1 -- 172.21.15.16:0/2416347549 learned_addr learned my addr 172.21.15.16:0/2416347549 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:28:01.118 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.103+0000 7f096124c640 1 --2- 172.21.15.16:0/2416347549 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f095c0ca4e0 0x7f095c069af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:01.118 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.103+0000 7f095bfff640 1 --2- 172.21.15.16:0/2416347549 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f095c0c9b40 0x7f095c0ccd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:01.118 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.103+0000 7f0960a4b640 1 -- 172.21.15.16:0/2416347549 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f095c0c9b40 msgr2=0x7f095c0ccd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:01.118 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.103+0000 7f0960a4b640 1 --2- 172.21.15.16:0/2416347549 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f095c0c9b40 0x7f095c0ccd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:01.119 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.103+0000 7f0960a4b640 1 -- 172.21.15.16:0/2416347549 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f095c0ca4e0 msgr2=0x7f095c069af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:01.119 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.103+0000 7f0960a4b640 1 --2- 172.21.15.16:0/2416347549 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f095c0ca4e0 0x7f095c069af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:01.119 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.103+0000 7f0960a4b640 1 -- 172.21.15.16:0/2416347549 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f094c0094d0 con 0x7f095c070df0 2023-05-22T15:28:01.120 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.103+0000 7f0960a4b640 1 --2- 172.21.15.16:0/2416347549 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f095c070df0 0x7f095c0cbe90 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f094c01a0b0 tx=0x7f094c01acc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:28:01.120 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.103+0000 7f09597fa640 1 -- 172.21.15.16:0/2416347549 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f094c0b4800 con 0x7f095c070df0 2023-05-22T15:28:01.120 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.103+0000 7f09597fa640 1 -- 172.21.15.16:0/2416347549 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f094c0b4980 con 0x7f095c070df0 2023-05-22T15:28:01.121 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.103+0000 7f0962caf640 1 -- 172.21.15.16:0/2416347549 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f095c0c8e80 con 0x7f095c070df0 2023-05-22T15:28:01.121 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.103+0000 7f09597fa640 1 -- 172.21.15.16:0/2416347549 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f094c0b4d30 con 0x7f095c070df0 2023-05-22T15:28:01.121 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.103+0000 7f0962caf640 1 -- 172.21.15.16:0/2416347549 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f095c0ca8b0 con 0x7f095c070df0 2023-05-22T15:28:01.121 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.103+0000 7f0962caf640 1 -- 172.21.15.16:0/2416347549 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f095c0c9010 con 0x7f095c070df0 2023-05-22T15:28:01.122 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.103+0000 7f09597fa640 1 -- 172.21.15.16:0/2416347549 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7f094c0a9070 con 0x7f095c070df0 2023-05-22T15:28:01.122 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.103+0000 7f09597fa640 1 --2- 172.21.15.16:0/2416347549 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f09480653b0 0x7f0948067860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:01.122 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.103+0000 7f09597fa640 1 -- 172.21.15.16:0/2416347549 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f094c11d690 con 0x7f095c070df0 2023-05-22T15:28:01.122 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.103+0000 7f095bfff640 1 --2- 172.21.15.16:0/2416347549 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f09480653b0 0x7f0948067860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:01.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.103+0000 7f095bfff640 1 --2- 172.21.15.16:0/2416347549 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f09480653b0 0x7f0948067860 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f095c1524a0 tx=0x7f0950008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:28:01.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.107+0000 7f09597fa640 1 -- 172.21.15.16:0/2416347549 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f094c0eb010 con 0x7f095c070df0 2023-05-22T15:28:01.303 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.295+0000 7f0962caf640 1 -- 172.21.15.16:0/2416347549 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f095c0c8330 con 0x7f095c070df0 2023-05-22T15:28:01.304 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.295+0000 7f09597fa640 1 -- 172.21.15.16:0/2416347549 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4641 (secure 0 0 0) 0x7f094c0ee630 con 0x7f095c070df0 2023-05-22T15:28:01.305 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:28:01.305 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4261,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6834/3661808159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":3661808159},{"type":"v1","addr":"172.21.15.62:6834","nonce":3661808159}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4264,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/2227467222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6833","nonce":2227467222},{"type":"v1","addr":"172.21.15.62:6835","nonce":2227467222}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4284,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6836/2808106087","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2808106087},{"type":"v1","addr":"172.21.15.16:6836","nonce":2808106087}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-05-22T15:26:49.546008+0000","modified":"2023-05-22T15:27:06.047204+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4287},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4287":{"gid":4287,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.16:6837/283322289","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6835","nonce":283322289},{"type":"v1","addr":"172.21.15.16:6837","nonce":283322289}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-05-22T15:28:01.307 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 8 2023-05-22T15:28:01.307 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.299+0000 7f0962caf640 1 -- 172.21.15.16:0/2416347549 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f09480653b0 msgr2=0x7f0948067860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:01.307 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.299+0000 7f0962caf640 1 --2- 172.21.15.16:0/2416347549 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f09480653b0 0x7f0948067860 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f095c1524a0 tx=0x7f0950008040 comp rx=0 tx=0).stop 2023-05-22T15:28:01.308 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.299+0000 7f0962caf640 1 -- 172.21.15.16:0/2416347549 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f095c070df0 msgr2=0x7f095c0cbe90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:01.308 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.299+0000 7f0962caf640 1 --2- 172.21.15.16:0/2416347549 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f095c070df0 0x7f095c0cbe90 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f094c01a0b0 tx=0x7f094c01acc0 comp rx=0 tx=0).stop 2023-05-22T15:28:01.309 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.299+0000 7f0962caf640 1 -- 172.21.15.16:0/2416347549 shutdown_connections 2023-05-22T15:28:01.309 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.299+0000 7f0962caf640 1 --2- 172.21.15.16:0/2416347549 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f095c0ca4e0 0x7f095c069af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:01.309 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.299+0000 7f0962caf640 1 --2- 172.21.15.16:0/2416347549 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f09480653b0 0x7f0948067860 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:01.309 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.299+0000 7f0962caf640 1 --2- 172.21.15.16:0/2416347549 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f095c0c9b40 0x7f095c0ccd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:01.309 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.299+0000 7f0962caf640 1 --2- 172.21.15.16:0/2416347549 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f095c070df0 0x7f095c0cbe90 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:01.310 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.299+0000 7f0962caf640 1 -- 172.21.15.16:0/2416347549 >> 172.21.15.16:0/2416347549 conn(0x7f095c069720 msgr2=0x7f095c152820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:28:01.310 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.299+0000 7f0962caf640 1 -- 172.21.15.16:0/2416347549 shutdown_connections 2023-05-22T15:28:01.310 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.299+0000 7f0962caf640 1 -- 172.21.15.16:0/2416347549 wait complete. 2023-05-22T15:28:01.322 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-05-22T15:28:01.437 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.431+0000 7f21398ec640 1 Processor -- start 2023-05-22T15:28:01.438 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.431+0000 7f21398ec640 1 -- start start 2023-05-22T15:28:01.440 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.431+0000 7f21398ec640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f21340b7dd0 0x7f21340ccab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:01.440 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.431+0000 7f21398ec640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2134160250 0x7f21340d0410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:01.440 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.431+0000 7f21398ec640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21340cd170 con 0x7f213415f700 2023-05-22T15:28:01.440 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.431+0000 7f21398ec640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21340d3360 con 0x7f21340b7dd0 2023-05-22T15:28:01.440 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.431+0000 7f21398ec640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21340d2ac0 con 0x7f2134160250 2023-05-22T15:28:01.440 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.431+0000 7f21327fc640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f213415f700 0x7f213415fb40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:50122/0 (socket says 172.21.15.16:50122) 2023-05-22T15:28:01.441 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.431+0000 7f21327fc640 1 -- 172.21.15.16:0/1775138055 learned_addr learned my addr 172.21.15.16:0/1775138055 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:28:01.441 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.431+0000 7f21337fe640 1 --2- 172.21.15.16:0/1775138055 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2134160250 0x7f21340d0410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:01.441 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.431+0000 7f2132ffd640 1 --2- 172.21.15.16:0/1775138055 >> v2:172.21.15.62:3300/0 conn(0x7f21340b7dd0 0x7f21340ccab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:01.441 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.431+0000 7f2131ffb640 1 -- 172.21.15.16:0/1775138055 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3366120045 0 0) 0x7f21340cd170 con 0x7f213415f700 2023-05-22T15:28:01.441 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.431+0000 7f2131ffb640 1 -- 172.21.15.16:0/1775138055 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f211c003530 con 0x7f213415f700 2023-05-22T15:28:01.441 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.431+0000 7f2131ffb640 1 -- 172.21.15.16:0/1775138055 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3221526117 0 0) 0x7f211c003530 con 0x7f213415f700 2023-05-22T15:28:01.442 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.431+0000 7f2131ffb640 1 -- 172.21.15.16:0/1775138055 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2134160250 msgr2=0x7f21340d0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:28:01.442 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.431+0000 7f2131ffb640 1 --2- 172.21.15.16:0/1775138055 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2134160250 0x7f21340d0410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:01.442 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.431+0000 7f2131ffb640 1 -- 172.21.15.16:0/1775138055 >> v2:172.21.15.62:3300/0 conn(0x7f21340b7dd0 msgr2=0x7f21340ccab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:28:01.442 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.431+0000 7f2131ffb640 1 --2- 172.21.15.16:0/1775138055 >> v2:172.21.15.62:3300/0 conn(0x7f21340b7dd0 0x7f21340ccab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:01.442 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.431+0000 7f2131ffb640 1 -- 172.21.15.16:0/1775138055 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21340d2da0 con 0x7f213415f700 2023-05-22T15:28:01.443 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.431+0000 7f2131ffb640 1 -- 172.21.15.16:0/1775138055 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7f2124001680 con 0x7f213415f700 2023-05-22T15:28:01.443 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.431+0000 7f2131ffb640 1 -- 172.21.15.16:0/1775138055 >> v1:172.21.15.16:6789/0 conn(0x7f213415f700 legacy=0x7f213415fb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:01.443 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.431+0000 7f2131ffb640 1 --2- 172.21.15.16:0/1775138055 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f211c003980 0x7f211c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:01.443 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.431+0000 7f2131ffb640 1 --2- 172.21.15.16:0/1775138055 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2134160250 0x7f211c0047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:01.444 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.431+0000 7f2131ffb640 1 --2- 172.21.15.16:0/1775138055 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f211c004d10 0x7f211c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:01.444 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.431+0000 7f2131ffb640 1 -- 172.21.15.16:0/1775138055 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21340d3360 con 0x7f211c004d10 2023-05-22T15:28:01.444 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.431+0000 7f2131ffb640 1 -- 172.21.15.16:0/1775138055 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21340d2ac0 con 0x7f211c003980 2023-05-22T15:28:01.445 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.431+0000 7f2131ffb640 1 -- 172.21.15.16:0/1775138055 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f211c00b990 con 0x7f2134160250 2023-05-22T15:28:01.445 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.431+0000 7f21337fe640 1 --2- 172.21.15.16:0/1775138055 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f211c004d10 0x7f211c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:01.445 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.431+0000 7f2132ffd640 1 --2- 172.21.15.16:0/1775138055 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f211c003980 0x7f211c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:01.446 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.431+0000 7f21327fc640 1 --2- 172.21.15.16:0/1775138055 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2134160250 0x7f211c0047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:01.446 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.431+0000 7f21337fe640 1 -- 172.21.15.16:0/1775138055 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2134160250 msgr2=0x7f211c0047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:28:01.446 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.431+0000 7f21337fe640 1 --2- 172.21.15.16:0/1775138055 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2134160250 0x7f211c0047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:01.447 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.431+0000 7f21337fe640 1 -- 172.21.15.16:0/1775138055 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f211c003980 msgr2=0x7f211c003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:28:01.447 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.431+0000 7f21337fe640 1 --2- 172.21.15.16:0/1775138055 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f211c003980 0x7f211c003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:01.447 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.431+0000 7f21337fe640 1 -- 172.21.15.16:0/1775138055 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f211c003530 con 0x7f211c004d10 2023-05-22T15:28:01.447 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.431+0000 7f21327fc640 1 --2- 172.21.15.16:0/1775138055 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2134160250 0x7f211c0047d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:28:01.450 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.435+0000 7f21337fe640 1 --2- 172.21.15.16:0/1775138055 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f211c004d10 0x7f211c005180 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f212c009fc0 tx=0x7f212c01cc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db619bc9dcf2d7e9 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:28:01.451 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.435+0000 7f2131ffb640 1 -- 172.21.15.16:0/1775138055 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f212c0a5c70 con 0x7f211c004d10 2023-05-22T15:28:01.451 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.435+0000 7f2131ffb640 1 -- 172.21.15.16:0/1775138055 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f212c00f040 con 0x7f211c004d10 2023-05-22T15:28:01.451 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.435+0000 7f2131ffb640 1 -- 172.21.15.16:0/1775138055 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f212c0ad440 con 0x7f211c004d10 2023-05-22T15:28:01.451 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.435+0000 7f21398ec640 1 -- 172.21.15.16:0/1775138055 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f211c004d10 msgr2=0x7f211c005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:01.451 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.435+0000 7f21398ec640 1 --2- 172.21.15.16:0/1775138055 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f211c004d10 0x7f211c005180 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f212c009fc0 tx=0x7f212c01cc00 comp rx=0 tx=0).stop 2023-05-22T15:28:01.451 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.435+0000 7f21337fe640 1 -- 172.21.15.16:0/1775138055 reap_dead start 2023-05-22T15:28:01.452 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.435+0000 7f21398ec640 1 -- 172.21.15.16:0/1775138055 shutdown_connections 2023-05-22T15:28:01.452 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.435+0000 7f21398ec640 1 -- 172.21.15.16:0/1775138055 >> 172.21.15.16:0/1775138055 conn(0x7f21340655c0 msgr2=0x7f2134070e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:28:01.452 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.435+0000 7f21398ec640 1 -- 172.21.15.16:0/1775138055 shutdown_connections 2023-05-22T15:28:01.452 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.435+0000 7f21398ec640 1 -- 172.21.15.16:0/1775138055 wait complete. 2023-05-22T15:28:01.452 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.435+0000 7f21398ec640 1 Processor -- start 2023-05-22T15:28:01.452 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.435+0000 7f21398ec640 1 -- start start 2023-05-22T15:28:01.452 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.435+0000 7f21398ec640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f211c004d10 0x7f213414c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:01.453 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.435+0000 7f21398ec640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f213414d2f0 0x7f213414d720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:01.453 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.435+0000 7f21398ec640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f21340ca450 0x7f21340ca8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:01.453 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.435+0000 7f21398ec640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f213415daf0 con 0x7f21340ca450 2023-05-22T15:28:01.453 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.435+0000 7f21398ec640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f213415e670 con 0x7f211c004d10 2023-05-22T15:28:01.453 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.435+0000 7f21398ec640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21340cae00 con 0x7f213414d2f0 2023-05-22T15:28:01.453 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.439+0000 7f21337fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f21340ca450 0x7f21340ca8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:01.453 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.439+0000 7f21337fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f21340ca450 0x7f21340ca8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39712/0 (socket says 172.21.15.16:39712) 2023-05-22T15:28:01.453 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.439+0000 7f21337fe640 1 -- 172.21.15.16:0/3077100097 learned_addr learned my addr 172.21.15.16:0/3077100097 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:28:01.454 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.439+0000 7f21327fc640 1 --2- 172.21.15.16:0/3077100097 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f213414d2f0 0x7f213414d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:01.454 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.439+0000 7f2132ffd640 1 --2- 172.21.15.16:0/3077100097 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f211c004d10 0x7f213414c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:01.454 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.439+0000 7f21337fe640 1 -- 172.21.15.16:0/3077100097 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f213414d2f0 msgr2=0x7f213414d720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:01.454 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.439+0000 7f21337fe640 1 --2- 172.21.15.16:0/3077100097 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f213414d2f0 0x7f213414d720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:01.455 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.439+0000 7f21337fe640 1 -- 172.21.15.16:0/3077100097 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f211c004d10 msgr2=0x7f213414c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:01.455 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.439+0000 7f21337fe640 1 --2- 172.21.15.16:0/3077100097 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f211c004d10 0x7f213414c9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:01.455 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.439+0000 7f21337fe640 1 -- 172.21.15.16:0/3077100097 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21340cf730 con 0x7f21340ca450 2023-05-22T15:28:01.455 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.439+0000 7f21337fe640 1 --2- 172.21.15.16:0/3077100097 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f21340ca450 0x7f21340ca8c0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f212c002240 tx=0x7f212c009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:28:01.455 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.439+0000 7f211bfff640 1 -- 172.21.15.16:0/3077100097 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f212c0bc070 con 0x7f21340ca450 2023-05-22T15:28:01.455 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.439+0000 7f211bfff640 1 -- 172.21.15.16:0/3077100097 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f212c0039a0 con 0x7f21340ca450 2023-05-22T15:28:01.455 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.439+0000 7f21398ec640 1 -- 172.21.15.16:0/3077100097 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21340d3080 con 0x7f21340ca450 2023-05-22T15:28:01.456 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.439+0000 7f211bfff640 1 -- 172.21.15.16:0/3077100097 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f212c003b30 con 0x7f21340ca450 2023-05-22T15:28:01.456 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.439+0000 7f21398ec640 1 -- 172.21.15.16:0/3077100097 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21340d0410 con 0x7f21340ca450 2023-05-22T15:28:01.457 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.439+0000 7f21398ec640 1 -- 172.21.15.16:0/3077100097 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21340d1e80 con 0x7f21340ca450 2023-05-22T15:28:01.457 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.439+0000 7f211bfff640 1 -- 172.21.15.16:0/3077100097 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7f212c0b4040 con 0x7f21340ca450 2023-05-22T15:28:01.457 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.439+0000 7f211bfff640 1 --2- 172.21.15.16:0/3077100097 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f21200653b0 0x7f2120067860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:01.457 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.439+0000 7f211bfff640 1 -- 172.21.15.16:0/3077100097 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f212c11cd90 con 0x7f21340ca450 2023-05-22T15:28:01.457 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.439+0000 7f2132ffd640 1 --2- 172.21.15.16:0/3077100097 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f21200653b0 0x7f2120067860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:01.457 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.439+0000 7f2132ffd640 1 --2- 172.21.15.16:0/3077100097 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f21200653b0 0x7f2120067860 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f2128009a30 tx=0x7f2128008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:28:01.458 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.443+0000 7f211bfff640 1 -- 172.21.15.16:0/3077100097 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f212c0ea6e0 con 0x7f21340ca450 2023-05-22T15:28:01.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.599+0000 7f21398ec640 1 -- 172.21.15.16:0/3077100097 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21340cb010 con 0x7f21340ca450 2023-05-22T15:28:01.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.599+0000 7f211bfff640 1 -- 172.21.15.16:0/3077100097 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v29) v1 ==== 74+0+19700 (secure 0 0 0) 0x7f212c0b2020 con 0x7f21340ca450 2023-05-22T15:28:01.609 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:28:01.609 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":29,"fsid":"b2719834-7847-4d98-90a6-f65d57f85c49","created":"2023-05-22T15:26:25.259314+0000","modified":"2023-05-22T15:27:31.182317+0000","last_up_change":"2023-05-22T15:26:36.448524+0000","last_in_change":"2023-05-22T15:26:27.675668+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-05-22T15:26:38.019781+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-05-22T15:26:43.570514+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-05-22T15:26:47.879159+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-05-22T15:26:48.522130+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data_ec","create_time":"2023-05-22T15:26:51.616142+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"9e7ff854-b25f-4cbb-a5c2-1799d9c53d2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":37021},{"type":"v1","addr":"172.21.15.16:6825","nonce":37021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":37021},{"type":"v1","addr":"172.21.15.16:6827","nonce":37021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":37021},{"type":"v1","addr":"172.21.15.16:6831","nonce":37021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":37021},{"type":"v1","addr":"172.21.15.16:6829","nonce":37021}]},"public_addr":"172.21.15.16:6825/37021","cluster_addr":"172.21.15.16:6827/37021","heartbeat_back_addr":"172.21.15.16:6831/37021","heartbeat_front_addr":"172.21.15.16:6829/37021","state":["exists","up"]},{"osd":1,"uuid":"4702fbc9-a927-4565-aca5-41dc69cc8264","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":37008},{"type":"v1","addr":"172.21.15.16:6801","nonce":37008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":37008},{"type":"v1","addr":"172.21.15.16:6803","nonce":37008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":37008},{"type":"v1","addr":"172.21.15.16:6807","nonce":37008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":37008},{"type":"v1","addr":"172.21.15.16:6805","nonce":37008}]},"public_addr":"172.21.15.16:6801/37008","cluster_addr":"172.21.15.16:6803/37008","heartbeat_back_addr":"172.21.15.16:6807/37008","heartbeat_front_addr":"172.21.15.16:6805/37008","state":["exists","up"]},{"osd":2,"uuid":"1e7bba46-4eba-47ef-bb81-5e2d371638ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":37009},{"type":"v1","addr":"172.21.15.16:6809","nonce":37009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":37009},{"type":"v1","addr":"172.21.15.16:6811","nonce":37009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":37009},{"type":"v1","addr":"172.21.15.16:6815","nonce":37009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":37009},{"type":"v1","addr":"172.21.15.16:6813","nonce":37009}]},"public_addr":"172.21.15.16:6809/37009","cluster_addr":"172.21.15.16:6811/37009","heartbeat_back_addr":"172.21.15.16:6815/37009","heartbeat_front_addr":"172.21.15.16:6813/37009","state":["exists","up"]},{"osd":3,"uuid":"383cacc0-5a88-4904-990e-498ca59ef48a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":37020},{"type":"v1","addr":"172.21.15.16:6817","nonce":37020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":37020},{"type":"v1","addr":"172.21.15.16:6819","nonce":37020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":37020},{"type":"v1","addr":"172.21.15.16:6823","nonce":37020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":37020},{"type":"v1","addr":"172.21.15.16:6821","nonce":37020}]},"public_addr":"172.21.15.16:6817/37020","cluster_addr":"172.21.15.16:6819/37020","heartbeat_back_addr":"172.21.15.16:6823/37020","heartbeat_front_addr":"172.21.15.16:6821/37020","state":["exists","up"]},{"osd":4,"uuid":"970500ef-95b7-42c1-b080-eed4fae7013d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":37353},{"type":"v1","addr":"172.21.15.62:6817","nonce":37353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6819","nonce":37353},{"type":"v1","addr":"172.21.15.62:6822","nonce":37353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37353},{"type":"v1","addr":"172.21.15.62:6831","nonce":37353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37353},{"type":"v1","addr":"172.21.15.62:6826","nonce":37353}]},"public_addr":"172.21.15.62:6817/37353","cluster_addr":"172.21.15.62:6822/37353","heartbeat_back_addr":"172.21.15.62:6831/37353","heartbeat_front_addr":"172.21.15.62:6826/37353","state":["exists","up"]},{"osd":5,"uuid":"022d67db-d4d8-445b-b1b9-8b446a00f5c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":37352},{"type":"v1","addr":"172.21.15.62:6801","nonce":37352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":37352},{"type":"v1","addr":"172.21.15.62:6803","nonce":37352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":37352},{"type":"v1","addr":"172.21.15.62:6807","nonce":37352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":37352},{"type":"v1","addr":"172.21.15.62:6805","nonce":37352}]},"public_addr":"172.21.15.62:6801/37352","cluster_addr":"172.21.15.62:6803/37352","heartbeat_back_addr":"172.21.15.62:6807/37352","heartbeat_front_addr":"172.21.15.62:6805/37352","state":["exists","up"]},{"osd":6,"uuid":"42bd372b-6688-4a1b-8407-e16e8c266e33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":37354},{"type":"v1","addr":"172.21.15.62:6820","nonce":37354}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":37354},{"type":"v1","addr":"172.21.15.62:6823","nonce":37354}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6829","nonce":37354},{"type":"v1","addr":"172.21.15.62:6830","nonce":37354}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6825","nonce":37354},{"type":"v1","addr":"172.21.15.62:6827","nonce":37354}]},"public_addr":"172.21.15.62:6820/37354","cluster_addr":"172.21.15.62:6823/37354","heartbeat_back_addr":"172.21.15.62:6830/37354","heartbeat_front_addr":"172.21.15.62:6827/37354","state":["exists","up"]},{"osd":7,"uuid":"ac70b2a3-f41b-4bf2-b227-7f2cbb55426d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37355},{"type":"v1","addr":"172.21.15.62:6809","nonce":37355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37355},{"type":"v1","addr":"172.21.15.62:6811","nonce":37355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":37355},{"type":"v1","addr":"172.21.15.62:6815","nonce":37355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":37355},{"type":"v1","addr":"172.21.15.62:6813","nonce":37355}]},"public_addr":"172.21.15.62:6809/37355","cluster_addr":"172.21.15.62:6811/37355","heartbeat_back_addr":"172.21.15.62:6815/37355","heartbeat_front_addr":"172.21.15.62:6813/37355","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.368896+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.107273+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.276888+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.287502+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.124791+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.251937+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.255679+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.232797+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"4.23","mappings":[{"from":3,"to":2}]},{"pgid":"4.31","mappings":[{"from":1,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-05-22T15:28:01.611 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.603+0000 7f21398ec640 1 -- 172.21.15.16:0/3077100097 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f21200653b0 msgr2=0x7f2120067860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:01.611 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.603+0000 7f21398ec640 1 --2- 172.21.15.16:0/3077100097 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f21200653b0 0x7f2120067860 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f2128009a30 tx=0x7f2128008040 comp rx=0 tx=0).stop 2023-05-22T15:28:01.611 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.603+0000 7f21398ec640 1 -- 172.21.15.16:0/3077100097 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f21340ca450 msgr2=0x7f21340ca8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:01.611 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.603+0000 7f21398ec640 1 --2- 172.21.15.16:0/3077100097 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f21340ca450 0x7f21340ca8c0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f212c002240 tx=0x7f212c009750 comp rx=0 tx=0).stop 2023-05-22T15:28:01.612 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.603+0000 7f21398ec640 1 -- 172.21.15.16:0/3077100097 shutdown_connections 2023-05-22T15:28:01.612 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.603+0000 7f21398ec640 1 --2- 172.21.15.16:0/3077100097 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f21340ca450 0x7f21340ca8c0 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:01.612 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.603+0000 7f21398ec640 1 --2- 172.21.15.16:0/3077100097 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f21200653b0 0x7f2120067860 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:01.613 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.603+0000 7f21398ec640 1 --2- 172.21.15.16:0/3077100097 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f213414d2f0 0x7f213414d720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:01.613 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.603+0000 7f21398ec640 1 --2- 172.21.15.16:0/3077100097 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f211c004d10 0x7f213414c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:01.613 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.603+0000 7f21398ec640 1 -- 172.21.15.16:0/3077100097 >> 172.21.15.16:0/3077100097 conn(0x7f21340655c0 msgr2=0x7f21340b6930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:28:01.613 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.603+0000 7f21398ec640 1 -- 172.21.15.16:0/3077100097 shutdown_connections 2023-05-22T15:28:01.614 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.603+0000 7f21398ec640 1 -- 172.21.15.16:0/3077100097 wait complete. 2023-05-22T15:28:01.627 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-05-22T15:28:01.738 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.731+0000 7f6860477640 1 Processor -- start 2023-05-22T15:28:01.738 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.731+0000 7f6860477640 1 -- start start 2023-05-22T15:28:01.739 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.731+0000 7f6860477640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6858153fe0 0x7f685806d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:01.739 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.731+0000 7f6860477640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6858070600 0x7f68580ad410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:01.739 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.731+0000 7f6860477640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6858069720 con 0x7f68580cae20 2023-05-22T15:28:01.739 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.731+0000 7f6860477640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68581543b0 con 0x7f6858070600 2023-05-22T15:28:01.740 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.731+0000 7f6860477640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f685815e360 con 0x7f6858153fe0 2023-05-22T15:28:01.740 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.731+0000 7f685ea14640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6858070600 0x7f68580ad410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:01.740 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.731+0000 7f685ea14640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6858070600 0x7f68580ad410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39726/0 (socket says 172.21.15.16:39726) 2023-05-22T15:28:01.740 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.731+0000 7f685ea14640 1 -- 172.21.15.16:0/1851406943 learned_addr learned my addr 172.21.15.16:0/1851406943 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:28:01.740 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.731+0000 7f685da12640 1 --1- 172.21.15.16:0/1851406943 >> v1:172.21.15.16:6789/0 conn(0x7f68580cae20 0x7f685806fef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:50124/0 (socket says 172.21.15.16:50124) 2023-05-22T15:28:01.740 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.731+0000 7f685e213640 1 --2- 172.21.15.16:0/1851406943 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6858153fe0 0x7f685806d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:01.740 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.731+0000 7f685ea14640 1 -- 172.21.15.16:0/1851406943 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6858153fe0 msgr2=0x7f685806d3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:28:01.741 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.731+0000 7f685ea14640 1 --2- 172.21.15.16:0/1851406943 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6858153fe0 0x7f685806d3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:01.741 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.731+0000 7f685ea14640 1 -- 172.21.15.16:0/1851406943 >> v1:172.21.15.16:6789/0 conn(0x7f68580cae20 legacy=0x7f685806fef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:01.741 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.731+0000 7f685ea14640 1 -- 172.21.15.16:0/1851406943 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f685815d5a0 con 0x7f6858070600 2023-05-22T15:28:01.742 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.735+0000 7f685ea14640 1 --2- 172.21.15.16:0/1851406943 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6858070600 0x7f68580ad410 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f6854009870 tx=0x7f685401c510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59ab47ad9131e33a server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:28:01.742 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.735+0000 7f685d211640 1 -- 172.21.15.16:0/1851406943 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68540aa670 con 0x7f6858070600 2023-05-22T15:28:01.742 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.735+0000 7f685d211640 1 -- 172.21.15.16:0/1851406943 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68540aa7d0 con 0x7f6858070600 2023-05-22T15:28:01.742 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.735+0000 7f685d211640 1 -- 172.21.15.16:0/1851406943 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68540aab60 con 0x7f6858070600 2023-05-22T15:28:01.742 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.735+0000 7f6860477640 1 -- 172.21.15.16:0/1851406943 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6858070600 msgr2=0x7f68580ad410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:01.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.735+0000 7f6860477640 1 --2- 172.21.15.16:0/1851406943 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6858070600 0x7f68580ad410 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f6854009870 tx=0x7f685401c510 comp rx=0 tx=0).stop 2023-05-22T15:28:01.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.735+0000 7f6860477640 1 -- 172.21.15.16:0/1851406943 shutdown_connections 2023-05-22T15:28:01.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.735+0000 7f6860477640 1 --2- 172.21.15.16:0/1851406943 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6858070600 0x7f68580ad410 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:01.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.735+0000 7f6860477640 1 --2- 172.21.15.16:0/1851406943 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6858153fe0 0x7f685806d3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:01.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.735+0000 7f6860477640 1 -- 172.21.15.16:0/1851406943 >> 172.21.15.16:0/1851406943 conn(0x7f68580b1110 msgr2=0x7f68580ce830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:28:01.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.735+0000 7f6860477640 1 -- 172.21.15.16:0/1851406943 shutdown_connections 2023-05-22T15:28:01.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.735+0000 7f6860477640 1 -- 172.21.15.16:0/1851406943 wait complete. 2023-05-22T15:28:01.746 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.735+0000 7f6860477640 1 Processor -- start 2023-05-22T15:28:01.747 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.735+0000 7f6860477640 1 -- start start 2023-05-22T15:28:01.747 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.735+0000 7f6860477640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6858070600 0x7f685814f8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:01.747 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.735+0000 7f6860477640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f68580cae20 0x7f685814e060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:01.747 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.735+0000 7f6860477640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6858153fe0 0x7f6858157b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:01.747 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.735+0000 7f6860477640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68581500e0 con 0x7f6858153fe0 2023-05-22T15:28:01.747 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.735+0000 7f6860477640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f685814d470 con 0x7f6858070600 2023-05-22T15:28:01.747 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.735+0000 7f6860477640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f685814cad0 con 0x7f68580cae20 2023-05-22T15:28:01.748 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.735+0000 7f685ea14640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6858153fe0 0x7f6858157b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:01.748 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.735+0000 7f685ea14640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6858153fe0 0x7f6858157b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39736/0 (socket says 172.21.15.16:39736) 2023-05-22T15:28:01.748 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.735+0000 7f685ea14640 1 -- 172.21.15.16:0/2364727417 learned_addr learned my addr 172.21.15.16:0/2364727417 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:28:01.748 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.735+0000 7f685e213640 1 --2- 172.21.15.16:0/2364727417 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6858070600 0x7f685814f8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:01.748 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.735+0000 7f685da12640 1 --2- 172.21.15.16:0/2364727417 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f68580cae20 0x7f685814e060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:01.748 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.735+0000 7f685ea14640 1 -- 172.21.15.16:0/2364727417 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f68580cae20 msgr2=0x7f685814e060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:01.748 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.735+0000 7f685ea14640 1 --2- 172.21.15.16:0/2364727417 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f68580cae20 0x7f685814e060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:01.749 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.735+0000 7f685ea14640 1 -- 172.21.15.16:0/2364727417 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6858070600 msgr2=0x7f685814f8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:01.749 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.735+0000 7f685ea14640 1 --2- 172.21.15.16:0/2364727417 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6858070600 0x7f685814f8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:01.749 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.735+0000 7f685ea14640 1 -- 172.21.15.16:0/2364727417 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68540094d0 con 0x7f6858153fe0 2023-05-22T15:28:01.749 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.735+0000 7f685e213640 1 --2- 172.21.15.16:0/2364727417 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6858070600 0x7f685814f8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:28:01.749 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.739+0000 7f685ea14640 1 --2- 172.21.15.16:0/2364727417 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6858153fe0 0x7f6858157b10 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f68540ac040 tx=0x7f685401ad50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:28:01.749 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.739+0000 7f684effd640 1 -- 172.21.15.16:0/2364727417 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68540b3800 con 0x7f6858153fe0 2023-05-22T15:28:01.749 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.739+0000 7f684effd640 1 -- 172.21.15.16:0/2364727417 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68540b3960 con 0x7f6858153fe0 2023-05-22T15:28:01.750 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.739+0000 7f6860477640 1 -- 172.21.15.16:0/2364727417 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f685806dd20 con 0x7f6858153fe0 2023-05-22T15:28:01.750 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.739+0000 7f684effd640 1 -- 172.21.15.16:0/2364727417 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68540b3d10 con 0x7f6858153fe0 2023-05-22T15:28:01.750 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.739+0000 7f6860477640 1 -- 172.21.15.16:0/2364727417 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f685815de40 con 0x7f6858153fe0 2023-05-22T15:28:01.750 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.739+0000 7f6860477640 1 -- 172.21.15.16:0/2364727417 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f685806d930 con 0x7f6858153fe0 2023-05-22T15:28:01.750 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.739+0000 7f684effd640 1 -- 172.21.15.16:0/2364727417 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7f68540b2380 con 0x7f6858153fe0 2023-05-22T15:28:01.750 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.739+0000 7f684effd640 1 --2- 172.21.15.16:0/2364727417 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f683c065380 0x7f683c067830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:01.750 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.739+0000 7f684effd640 1 -- 172.21.15.16:0/2364727417 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f685411ce70 con 0x7f6858153fe0 2023-05-22T15:28:01.750 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.739+0000 7f685e213640 1 --2- 172.21.15.16:0/2364727417 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f683c065380 0x7f683c067830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:01.751 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.739+0000 7f685e213640 1 --2- 172.21.15.16:0/2364727417 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f683c065380 0x7f683c067830 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f6850007330 tx=0x7f685000b040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:28:01.751 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.743+0000 7f684effd640 1 -- 172.21.15.16:0/2364727417 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f68540ea7f0 con 0x7f6858153fe0 2023-05-22T15:28:01.936 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:01.927+0000 7f6860477640 1 -- 172.21.15.16:0/2364727417 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f685814d5b0 con 0x7f6858153fe0 2023-05-22T15:28:02.370 INFO:tasks.ceph.mon.a.smithi016.stderr:2023-05-22T15:28:02.363+0000 7f95dd48a640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-05-22T15:28:02.386 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:02.379+0000 7f684effd640 1 -- 172.21.15.16:0/2364727417 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v9) v1 ==== 161+0+0 (secure 0 0 0) 0x7f685814d5b0 con 0x7f6858153fe0 2023-05-22T15:28:02.387 INFO:teuthology.orchestra.run.smithi016.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-05-22T15:28:02.390 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:02.383+0000 7f6860477640 1 -- 172.21.15.16:0/2364727417 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f683c065380 msgr2=0x7f683c067830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:02.391 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:02.383+0000 7f6860477640 1 --2- 172.21.15.16:0/2364727417 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f683c065380 0x7f683c067830 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f6850007330 tx=0x7f685000b040 comp rx=0 tx=0).stop 2023-05-22T15:28:02.391 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:02.383+0000 7f6860477640 1 -- 172.21.15.16:0/2364727417 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6858153fe0 msgr2=0x7f6858157b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:02.391 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:02.383+0000 7f6860477640 1 --2- 172.21.15.16:0/2364727417 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6858153fe0 0x7f6858157b10 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f68540ac040 tx=0x7f685401ad50 comp rx=0 tx=0).stop 2023-05-22T15:28:02.391 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:02.383+0000 7f6860477640 1 -- 172.21.15.16:0/2364727417 shutdown_connections 2023-05-22T15:28:02.391 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:02.383+0000 7f6860477640 1 --2- 172.21.15.16:0/2364727417 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6858153fe0 0x7f6858157b10 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:02.391 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:02.383+0000 7f6860477640 1 --2- 172.21.15.16:0/2364727417 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f683c065380 0x7f683c067830 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:02.392 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:02.383+0000 7f6860477640 1 --2- 172.21.15.16:0/2364727417 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f68580cae20 0x7f685814e060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:02.392 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:02.383+0000 7f6860477640 1 --2- 172.21.15.16:0/2364727417 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6858070600 0x7f685814f8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:02.392 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:02.383+0000 7f6860477640 1 -- 172.21.15.16:0/2364727417 >> 172.21.15.16:0/2364727417 conn(0x7f68580b1110 msgr2=0x7f68580c87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:28:02.392 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:02.383+0000 7f6860477640 1 -- 172.21.15.16:0/2364727417 shutdown_connections 2023-05-22T15:28:02.392 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:02.383+0000 7f6860477640 1 -- 172.21.15.16:0/2364727417 wait complete. 2023-05-22T15:28:02.393 INFO:tasks.ceph.mds.c.smithi016.stderr: -4448> 2023-05-22T15:26:51.130+0000 7fcda2cbc640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-05-22T15:28:02.409 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-05-22T15:28:02.434 INFO:tasks.ceph.mds.c.smithi016.stdout:starting mds.c at 2023-05-22T15:28:02.535 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:02.527+0000 7fe6cfb7d640 1 Processor -- start 2023-05-22T15:28:02.535 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:02.527+0000 7fe6cfb7d640 1 -- start start 2023-05-22T15:28:02.537 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:02.527+0000 7fe6cfb7d640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe6c806b6c0 0x7fe6c80cb4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:02.537 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:02.527+0000 7fe6cfb7d640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fe6c806ba90 0x7fe6c8158fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:02.537 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:02.527+0000 7fe6cfb7d640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6c8159f70 con 0x7fe6c8159510 2023-05-22T15:28:02.537 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:02.527+0000 7fe6cfb7d640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6c8071760 con 0x7fe6c806b6c0 2023-05-22T15:28:02.537 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:02.527+0000 7fe6cfb7d640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6c80ca7d0 con 0x7fe6c806ba90 2023-05-22T15:28:02.538 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:02.527+0000 7fe6cd919640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe6c806b6c0 0x7fe6c80cb4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:02.538 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:02.527+0000 7fe6cd919640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe6c806b6c0 0x7fe6c80cb4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39766/0 (socket says 172.21.15.16:39766) 2023-05-22T15:28:02.538 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:02.527+0000 7fe6cd919640 1 -- 172.21.15.16:0/2161494040 learned_addr learned my addr 172.21.15.16:0/2161494040 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:28:02.538 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:02.531+0000 7fe6cd118640 1 --2- 172.21.15.16:0/2161494040 >> v2:172.21.15.62:3300/0 conn(0x7fe6c806ba90 0x7fe6c8158fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:02.538 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:02.531+0000 7fe6cd919640 1 -- 172.21.15.16:0/2161494040 >> v2:172.21.15.62:3300/0 conn(0x7fe6c806ba90 msgr2=0x7fe6c8158fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:28:02.538 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:02.531+0000 7fe6cd919640 1 --2- 172.21.15.16:0/2161494040 >> v2:172.21.15.62:3300/0 conn(0x7fe6c806ba90 0x7fe6c8158fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:02.539 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:02.531+0000 7fe6cd919640 1 -- 172.21.15.16:0/2161494040 >> v1:172.21.15.62:6789/0 conn(0x7fe6c8159510 legacy=0x7fe6c80cc740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:28:02.539 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:02.531+0000 7fe6cd919640 1 -- 172.21.15.16:0/2161494040 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6c80cb030 con 0x7fe6c806b6c0 2023-05-22T15:28:02.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:02.531+0000 7fe6cd919640 1 --2- 172.21.15.16:0/2161494040 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe6c806b6c0 0x7fe6c80cb4c0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7fe6c806daf0 tx=0x7fe6c401c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6af3e42fb845fa2e server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:28:02.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:02.531+0000 7fe6cc917640 1 -- 172.21.15.16:0/2161494040 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6c40ac410 con 0x7fe6c806b6c0 2023-05-22T15:28:02.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:02.531+0000 7fe6cc917640 1 -- 172.21.15.16:0/2161494040 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6c40ac570 con 0x7fe6c806b6c0 2023-05-22T15:28:02.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:02.531+0000 7fe6cc917640 1 -- 172.21.15.16:0/2161494040 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6c40ac900 con 0x7fe6c806b6c0 2023-05-22T15:28:02.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:02.531+0000 7fe6cfb7d640 1 -- 172.21.15.16:0/2161494040 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe6c806b6c0 msgr2=0x7fe6c80cb4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:02.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:02.531+0000 7fe6cfb7d640 1 --2- 172.21.15.16:0/2161494040 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe6c806b6c0 0x7fe6c80cb4c0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7fe6c806daf0 tx=0x7fe6c401c490 comp rx=0 tx=0).stop 2023-05-22T15:28:02.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:02.531+0000 7fe6cfb7d640 1 -- 172.21.15.16:0/2161494040 shutdown_connections 2023-05-22T15:28:02.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:02.531+0000 7fe6cfb7d640 1 --2- 172.21.15.16:0/2161494040 >> v2:172.21.15.62:3300/0 conn(0x7fe6c806ba90 0x7fe6c8158fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:02.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:02.531+0000 7fe6cfb7d640 1 --2- 172.21.15.16:0/2161494040 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe6c806b6c0 0x7fe6c80cb4c0 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:02.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:02.531+0000 7fe6cfb7d640 1 -- 172.21.15.16:0/2161494040 >> 172.21.15.16:0/2161494040 conn(0x7fe6c8069720 msgr2=0x7fe6c8069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:28:02.549 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:02.531+0000 7fe6cfb7d640 1 -- 172.21.15.16:0/2161494040 shutdown_connections 2023-05-22T15:28:02.549 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:02.531+0000 7fe6cfb7d640 1 -- 172.21.15.16:0/2161494040 wait complete. 2023-05-22T15:28:02.549 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:02.535+0000 7fe6cfb7d640 1 Processor -- start 2023-05-22T15:28:02.549 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:02.535+0000 7fe6cfb7d640 1 -- start start 2023-05-22T15:28:02.549 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:02.535+0000 7fe6cfb7d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe6c806b6c0 0x7fe6c80b10a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:02.550 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:02.535+0000 7fe6cfb7d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe6c806ba90 0x7fe6c81f3700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:02.550 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:02.535+0000 7fe6cfb7d640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe6c8159510 0x7fe6c81f3c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:02.550 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:02.535+0000 7fe6cfb7d640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6c81f4180 con 0x7fe6c806b6c0 2023-05-22T15:28:02.550 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:02.535+0000 7fe6cfb7d640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6c8140230 con 0x7fe6c806ba90 2023-05-22T15:28:02.550 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:02.535+0000 7fe6cfb7d640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6c80b32a0 con 0x7fe6c8159510 2023-05-22T15:28:02.550 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:02.535+0000 7fe6cd919640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe6c806b6c0 0x7fe6c80b10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:02.551 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:02.535+0000 7fe6cd919640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe6c806b6c0 0x7fe6c80b10a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39782/0 (socket says 172.21.15.16:39782) 2023-05-22T15:28:02.551 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:02.535+0000 7fe6cd919640 1 -- 172.21.15.16:0/4210766225 learned_addr learned my addr 172.21.15.16:0/4210766225 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:28:02.551 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:02.535+0000 7fe6cd118640 1 --2- 172.21.15.16:0/4210766225 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe6c806ba90 0x7fe6c81f3700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:02.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:02.535+0000 7fe6cd919640 1 -- 172.21.15.16:0/4210766225 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe6c8159510 msgr2=0x7fe6c81f3c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:02.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:02.535+0000 7fe6cd919640 1 --2- 172.21.15.16:0/4210766225 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe6c8159510 0x7fe6c81f3c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:02.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:02.535+0000 7fe6cd919640 1 -- 172.21.15.16:0/4210766225 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe6c806ba90 msgr2=0x7fe6c81f3700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:02.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:02.535+0000 7fe6cd919640 1 --2- 172.21.15.16:0/4210766225 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe6c806ba90 0x7fe6c81f3700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:02.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:02.535+0000 7fe6cd919640 1 -- 172.21.15.16:0/4210766225 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6c40094d0 con 0x7fe6c806b6c0 2023-05-22T15:28:02.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:02.535+0000 7fe6cd919640 1 --2- 172.21.15.16:0/4210766225 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe6c806b6c0 0x7fe6c80b10a0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7fe6c40af040 tx=0x7fe6c4019230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:28:02.554 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:02.535+0000 7fe6be7fc640 1 -- 172.21.15.16:0/4210766225 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6c40acea0 con 0x7fe6c806b6c0 2023-05-22T15:28:02.554 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:02.535+0000 7fe6be7fc640 1 -- 172.21.15.16:0/4210766225 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6c40b5630 con 0x7fe6c806b6c0 2023-05-22T15:28:02.554 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:02.535+0000 7fe6cfb7d640 1 -- 172.21.15.16:0/4210766225 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6c8159990 con 0x7fe6c806b6c0 2023-05-22T15:28:02.555 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:02.535+0000 7fe6be7fc640 1 -- 172.21.15.16:0/4210766225 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6c40ac410 con 0x7fe6c806b6c0 2023-05-22T15:28:02.555 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:02.535+0000 7fe6cfb7d640 1 -- 172.21.15.16:0/4210766225 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6c80cacc0 con 0x7fe6c806b6c0 2023-05-22T15:28:02.555 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:02.535+0000 7fe6cfb7d640 1 -- 172.21.15.16:0/4210766225 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6c806be60 con 0x7fe6c806b6c0 2023-05-22T15:28:02.556 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:02.535+0000 7fe6be7fc640 1 -- 172.21.15.16:0/4210766225 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7fe6c40b3070 con 0x7fe6c806b6c0 2023-05-22T15:28:02.556 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:02.539+0000 7fe6be7fc640 1 --2- 172.21.15.16:0/4210766225 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fe6ac065350 0x7fe6ac067800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:02.556 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:02.539+0000 7fe6cd118640 1 --2- 172.21.15.16:0/4210766225 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fe6ac065350 0x7fe6ac067800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:02.556 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:02.539+0000 7fe6be7fc640 1 -- 172.21.15.16:0/4210766225 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6957+0+0 (secure 0 0 0) 0x7fe6c411d3c0 con 0x7fe6c806b6c0 2023-05-22T15:28:02.556 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:02.539+0000 7fe6cd118640 1 --2- 172.21.15.16:0/4210766225 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fe6ac065350 0x7fe6ac067800 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fe6c8150400 tx=0x7fe6b8008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:28:02.557 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:02.543+0000 7fe6be7fc640 1 -- 172.21.15.16:0/4210766225 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fe6c40ead50 con 0x7fe6c806b6c0 2023-05-22T15:28:02.736 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:02.727+0000 7fe6cfb7d640 1 -- 172.21.15.16:0/4210766225 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fe6c80c4ea0 con 0x7fe6c806b6c0 2023-05-22T15:28:03.391 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:03.383+0000 7fe6be7fc640 1 -- 172.21.15.16:0/4210766225 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v10) v1 ==== 104+0+0 (secure 0 0 0) 0x7fe6c40ee370 con 0x7fe6c806b6c0 2023-05-22T15:28:03.396 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:03.387+0000 7fe6cfb7d640 1 -- 172.21.15.16:0/4210766225 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fe6ac065350 msgr2=0x7fe6ac067800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:03.396 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:03.387+0000 7fe6cfb7d640 1 --2- 172.21.15.16:0/4210766225 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fe6ac065350 0x7fe6ac067800 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fe6c8150400 tx=0x7fe6b8008040 comp rx=0 tx=0).stop 2023-05-22T15:28:03.396 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:03.387+0000 7fe6cfb7d640 1 -- 172.21.15.16:0/4210766225 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe6c806b6c0 msgr2=0x7fe6c80b10a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:03.397 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:03.387+0000 7fe6cfb7d640 1 --2- 172.21.15.16:0/4210766225 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe6c806b6c0 0x7fe6c80b10a0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7fe6c40af040 tx=0x7fe6c4019230 comp rx=0 tx=0).stop 2023-05-22T15:28:03.397 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:03.387+0000 7fe6cfb7d640 1 -- 172.21.15.16:0/4210766225 shutdown_connections 2023-05-22T15:28:03.397 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:03.387+0000 7fe6cfb7d640 1 --2- 172.21.15.16:0/4210766225 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fe6ac065350 0x7fe6ac067800 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:03.397 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:03.387+0000 7fe6cfb7d640 1 --2- 172.21.15.16:0/4210766225 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe6c8159510 0x7fe6c81f3c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:03.397 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:03.387+0000 7fe6cfb7d640 1 --2- 172.21.15.16:0/4210766225 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe6c806ba90 0x7fe6c81f3700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:03.397 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:03.387+0000 7fe6cfb7d640 1 --2- 172.21.15.16:0/4210766225 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe6c806b6c0 0x7fe6c80b10a0 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:03.397 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:03.387+0000 7fe6cfb7d640 1 -- 172.21.15.16:0/4210766225 >> 172.21.15.16:0/4210766225 conn(0x7fe6c8069720 msgr2=0x7fe6c815ade0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:28:03.398 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:03.387+0000 7fe6cfb7d640 1 -- 172.21.15.16:0/4210766225 shutdown_connections 2023-05-22T15:28:03.398 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:03.387+0000 7fe6cfb7d640 1 -- 172.21.15.16:0/4210766225 wait complete. 2023-05-22T15:28:03.417 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-05-22T15:28:03.595 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:03.587+0000 7fc8a8c1d640 1 Processor -- start 2023-05-22T15:28:03.595 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:03.587+0000 7fc8a8c1d640 1 -- start start 2023-05-22T15:28:03.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:03.587+0000 7fc8a8c1d640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc8a415c8e0 0x7fc8a40ccc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:03.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:03.587+0000 7fc8a8c1d640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fc8a40cef10 0x7fc8a40cf370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:03.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:03.587+0000 7fc8a8c1d640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8a40d2a20 con 0x7fc8a40d2eb0 2023-05-22T15:28:03.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:03.587+0000 7fc8a8c1d640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8a40cd230 con 0x7fc8a40cef10 2023-05-22T15:28:03.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:03.587+0000 7fc8a8c1d640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8a40d23d0 con 0x7fc8a415c8e0 2023-05-22T15:28:03.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:03.591+0000 7fc8a2d9d640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fc8a40cef10 0x7fc8a40cf370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:03.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:03.591+0000 7fc8a259c640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc8a415c8e0 0x7fc8a40ccc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:03.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:03.591+0000 7fc8a2d9d640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fc8a40cef10 0x7fc8a40cf370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.16:54438/0 (socket says 172.21.15.16:54438) 2023-05-22T15:28:03.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:03.591+0000 7fc8a259c640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc8a415c8e0 0x7fc8a40ccc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3301/0 says I am v2:172.21.15.16:59064/0 (socket says 172.21.15.16:59064) 2023-05-22T15:28:03.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:03.591+0000 7fc8a2d9d640 1 -- 172.21.15.16:0/994046251 learned_addr learned my addr 172.21.15.16:0/994046251 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:28:03.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:03.591+0000 7fc8a1d9b640 1 --1- 172.21.15.16:0/994046251 >> v1:172.21.15.62:6789/0 conn(0x7fc8a40d2eb0 0x7fc8a415ccb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.16:53996/0 (socket says 172.21.15.16:53996) 2023-05-22T15:28:03.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:03.591+0000 7fc8a2d9d640 1 -- 172.21.15.16:0/994046251 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc8a415c8e0 msgr2=0x7fc8a40ccc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:28:03.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:03.591+0000 7fc8a2d9d640 1 --2- 172.21.15.16:0/994046251 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc8a415c8e0 0x7fc8a40ccc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:03.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:03.591+0000 7fc8a2d9d640 1 -- 172.21.15.16:0/994046251 >> v1:172.21.15.62:6789/0 conn(0x7fc8a40d2eb0 legacy=0x7fc8a415ccb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:03.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:03.591+0000 7fc8a2d9d640 1 -- 172.21.15.16:0/994046251 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8a40d26b0 con 0x7fc8a40cef10 2023-05-22T15:28:03.600 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:03.591+0000 7fc8a2d9d640 1 --2- 172.21.15.16:0/994046251 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc8a40cef10 0x7fc8a40cf370 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fc89c009e50 tx=0x7fc89c01c940 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7071eeb1ce15d10a server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:28:03.600 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:03.591+0000 7fc8a159a640 1 -- 172.21.15.16:0/994046251 <== mon.1 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc89c0a57e0 con 0x7fc8a40cef10 2023-05-22T15:28:03.600 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:03.591+0000 7fc8a159a640 1 -- 172.21.15.16:0/994046251 <== mon.1 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc89c0a5940 con 0x7fc8a40cef10 2023-05-22T15:28:03.600 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:03.591+0000 7fc8a159a640 1 -- 172.21.15.16:0/994046251 <== mon.1 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc89c0a5cd0 con 0x7fc8a40cef10 2023-05-22T15:28:03.601 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:03.595+0000 7fc8a8c1d640 1 -- 172.21.15.16:0/994046251 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc8a40cef10 msgr2=0x7fc8a40cf370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:03.601 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:03.595+0000 7fc8a8c1d640 1 --2- 172.21.15.16:0/994046251 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc8a40cef10 0x7fc8a40cf370 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fc89c009e50 tx=0x7fc89c01c940 comp rx=0 tx=0).stop 2023-05-22T15:28:03.601 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:03.595+0000 7fc8a8c1d640 1 -- 172.21.15.16:0/994046251 shutdown_connections 2023-05-22T15:28:03.601 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:03.595+0000 7fc8a8c1d640 1 --2- 172.21.15.16:0/994046251 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc8a40cef10 0x7fc8a40cf370 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:03.601 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:03.595+0000 7fc8a8c1d640 1 --2- 172.21.15.16:0/994046251 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc8a415c8e0 0x7fc8a40ccc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:03.602 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:03.595+0000 7fc8a8c1d640 1 -- 172.21.15.16:0/994046251 >> 172.21.15.16:0/994046251 conn(0x7fc8a4070e50 msgr2=0x7fc8a4071250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:28:03.602 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:03.595+0000 7fc8a8c1d640 1 -- 172.21.15.16:0/994046251 shutdown_connections 2023-05-22T15:28:03.602 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:03.595+0000 7fc8a8c1d640 1 -- 172.21.15.16:0/994046251 wait complete. 2023-05-22T15:28:03.603 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:03.595+0000 7fc8a8c1d640 1 Processor -- start 2023-05-22T15:28:03.603 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:03.595+0000 7fc8a8c1d640 1 -- start start 2023-05-22T15:28:03.603 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:03.595+0000 7fc8a8c1d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc8a40cef10 0x7fc8a414d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:03.603 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:03.595+0000 7fc8a8c1d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc8a40d2eb0 0x7fc8a415b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:03.604 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:03.595+0000 7fc8a8c1d640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc8a415c8e0 0x7fc8a415b550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:03.604 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:03.595+0000 7fc8a8c1d640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8a414dc30 con 0x7fc8a40cef10 2023-05-22T15:28:03.604 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:03.595+0000 7fc8a8c1d640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8a414ded0 con 0x7fc8a40d2eb0 2023-05-22T15:28:03.604 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:03.595+0000 7fc8a8c1d640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8a414e1b0 con 0x7fc8a415c8e0 2023-05-22T15:28:03.604 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:03.595+0000 7fc8a259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc8a40cef10 0x7fc8a414d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:03.604 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:03.595+0000 7fc8a259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc8a40cef10 0x7fc8a414d3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39790/0 (socket says 172.21.15.16:39790) 2023-05-22T15:28:03.604 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:03.595+0000 7fc8a259c640 1 -- 172.21.15.16:0/4264029061 learned_addr learned my addr 172.21.15.16:0/4264029061 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:28:03.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:03.595+0000 7fc8a1d9b640 1 --2- 172.21.15.16:0/4264029061 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc8a40d2eb0 0x7fc8a415b010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:03.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:03.595+0000 7fc8a2d9d640 1 --2- 172.21.15.16:0/4264029061 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc8a415c8e0 0x7fc8a415b550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:03.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:03.595+0000 7fc8a259c640 1 -- 172.21.15.16:0/4264029061 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc8a415c8e0 msgr2=0x7fc8a415b550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:03.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:03.595+0000 7fc8a259c640 1 --2- 172.21.15.16:0/4264029061 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc8a415c8e0 0x7fc8a415b550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:03.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:03.595+0000 7fc8a259c640 1 -- 172.21.15.16:0/4264029061 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc8a40d2eb0 msgr2=0x7fc8a415b010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:03.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:03.595+0000 7fc8a259c640 1 --2- 172.21.15.16:0/4264029061 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc8a40d2eb0 0x7fc8a415b010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:03.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:03.595+0000 7fc8a259c640 1 -- 172.21.15.16:0/4264029061 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc89c009b30 con 0x7fc8a40cef10 2023-05-22T15:28:03.606 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:03.595+0000 7fc8a259c640 1 --2- 172.21.15.16:0/4264029061 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc8a40cef10 0x7fc8a414d3c0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7fc89400e870 tx=0x7fc89400ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:28:03.606 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:03.595+0000 7fc892ffd640 1 -- 172.21.15.16:0/4264029061 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc894009510 con 0x7fc8a40cef10 2023-05-22T15:28:03.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:03.595+0000 7fc892ffd640 1 -- 172.21.15.16:0/4264029061 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc89400eec0 con 0x7fc8a40cef10 2023-05-22T15:28:03.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:03.595+0000 7fc8a8c1d640 1 -- 172.21.15.16:0/4264029061 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8a40d3410 con 0x7fc8a40cef10 2023-05-22T15:28:03.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:03.595+0000 7fc892ffd640 1 -- 172.21.15.16:0/4264029061 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc89400cb40 con 0x7fc8a40cef10 2023-05-22T15:28:03.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:03.595+0000 7fc8a8c1d640 1 -- 172.21.15.16:0/4264029061 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8a40d3280 con 0x7fc8a40cef10 2023-05-22T15:28:03.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:03.599+0000 7fc8a8c1d640 1 -- 172.21.15.16:0/4264029061 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8a40d0bf0 con 0x7fc8a40cef10 2023-05-22T15:28:03.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:03.599+0000 7fc892ffd640 1 -- 172.21.15.16:0/4264029061 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7fc89400ccc0 con 0x7fc8a40cef10 2023-05-22T15:28:03.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:03.599+0000 7fc892ffd640 1 --2- 172.21.15.16:0/4264029061 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fc884065350 0x7fc884067800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:03.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:03.599+0000 7fc892ffd640 1 -- 172.21.15.16:0/4264029061 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6957+0+0 (secure 0 0 0) 0x7fc894082d30 con 0x7fc8a40cef10 2023-05-22T15:28:03.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:03.599+0000 7fc8a1d9b640 1 --2- 172.21.15.16:0/4264029061 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fc884065350 0x7fc884067800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:03.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:03.599+0000 7fc8a1d9b640 1 --2- 172.21.15.16:0/4264029061 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fc884065350 0x7fc884067800 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fc8a41568e0 tx=0x7fc898007e50 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:28:03.609 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:03.603+0000 7fc892ffd640 1 -- 172.21.15.16:0/4264029061 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fc8940505e0 con 0x7fc8a40cef10 2023-05-22T15:28:03.764 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:03.755+0000 7fc8a8c1d640 1 -- 172.21.15.16:0/4264029061 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fc8a404dbe0 con 0x7fc8a40cef10 2023-05-22T15:28:04.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:04.399+0000 7fc892ffd640 1 -- 172.21.15.16:0/4264029061 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v31) v1 ==== 181+0+0 (secure 0 0 0) 0x7fc894053c00 con 0x7fc8a40cef10 2023-05-22T15:28:04.406 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'cephfs_metadata' removed 2023-05-22T15:28:04.415 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:04.407+0000 7fc8a8c1d640 1 -- 172.21.15.16:0/4264029061 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fc884065350 msgr2=0x7fc884067800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:04.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:04.407+0000 7fc8a8c1d640 1 --2- 172.21.15.16:0/4264029061 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fc884065350 0x7fc884067800 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fc8a41568e0 tx=0x7fc898007e50 comp rx=0 tx=0).stop 2023-05-22T15:28:04.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:04.407+0000 7fc8a8c1d640 1 -- 172.21.15.16:0/4264029061 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc8a40cef10 msgr2=0x7fc8a414d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:04.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:04.407+0000 7fc8a8c1d640 1 --2- 172.21.15.16:0/4264029061 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc8a40cef10 0x7fc8a414d3c0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7fc89400e870 tx=0x7fc89400ec70 comp rx=0 tx=0).stop 2023-05-22T15:28:04.417 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:04.407+0000 7fc8a8c1d640 1 -- 172.21.15.16:0/4264029061 shutdown_connections 2023-05-22T15:28:04.417 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:04.407+0000 7fc8a8c1d640 1 --2- 172.21.15.16:0/4264029061 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fc884065350 0x7fc884067800 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:04.417 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:04.407+0000 7fc8a8c1d640 1 --2- 172.21.15.16:0/4264029061 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc8a415c8e0 0x7fc8a415b550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:04.417 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:04.407+0000 7fc8a8c1d640 1 --2- 172.21.15.16:0/4264029061 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc8a40d2eb0 0x7fc8a415b010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:04.417 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:04.407+0000 7fc8a8c1d640 1 --2- 172.21.15.16:0/4264029061 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc8a40cef10 0x7fc8a414d3c0 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:04.417 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:04.407+0000 7fc8a8c1d640 1 -- 172.21.15.16:0/4264029061 >> 172.21.15.16:0/4264029061 conn(0x7fc8a4070e50 msgr2=0x7fc8a40cca30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:28:04.418 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:04.407+0000 7fc8a8c1d640 1 -- 172.21.15.16:0/4264029061 shutdown_connections 2023-05-22T15:28:04.418 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:04.407+0000 7fc8a8c1d640 1 -- 172.21.15.16:0/4264029061 wait complete. 2023-05-22T15:28:04.439 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-05-22T15:28:04.578 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:04.571+0000 7fc19e1c2640 1 Processor -- start 2023-05-22T15:28:04.578 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:04.571+0000 7fc19e1c2640 1 -- start start 2023-05-22T15:28:04.579 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:04.571+0000 7fc19e1c2640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc198159a30 0x7fc19806e380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:04.580 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:04.571+0000 7fc19e1c2640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fc19815d610 0x7fc1980cbe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:04.580 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:04.571+0000 7fc19e1c2640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc1980cf570 0x7fc1980cf9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:04.580 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:04.571+0000 7fc19e1c2640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1980cff30 con 0x7fc1980cf570 2023-05-22T15:28:04.580 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:04.571+0000 7fc19e1c2640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1980cde70 con 0x7fc19815d610 2023-05-22T15:28:04.581 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:04.571+0000 7fc19e1c2640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc19815db20 con 0x7fc198159a30 2023-05-22T15:28:04.581 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:04.571+0000 7fc197fff640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc1980cf570 0x7fc1980cf9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:04.582 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:04.571+0000 7fc197fff640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc1980cf570 0x7fc1980cf9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39804/0 (socket says 172.21.15.16:39804) 2023-05-22T15:28:04.582 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:04.571+0000 7fc197fff640 1 -- 172.21.15.16:0/2667837136 learned_addr learned my addr 172.21.15.16:0/2667837136 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:28:04.582 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:04.571+0000 7fc196ffd640 1 --2- 172.21.15.16:0/2667837136 >> v2:172.21.15.62:3300/0 conn(0x7fc19815d610 0x7fc1980cbe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:04.582 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:04.571+0000 7fc1977fe640 1 --2- 172.21.15.16:0/2667837136 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc198159a30 0x7fc19806e380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:04.583 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:04.571+0000 7fc197fff640 1 -- 172.21.15.16:0/2667837136 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc198159a30 msgr2=0x7fc19806e380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:28:04.583 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:04.571+0000 7fc197fff640 1 --2- 172.21.15.16:0/2667837136 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc198159a30 0x7fc19806e380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:04.583 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:04.571+0000 7fc197fff640 1 -- 172.21.15.16:0/2667837136 >> v2:172.21.15.62:3300/0 conn(0x7fc19815d610 msgr2=0x7fc1980cbe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:28:04.584 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:04.571+0000 7fc197fff640 1 --2- 172.21.15.16:0/2667837136 >> v2:172.21.15.62:3300/0 conn(0x7fc19815d610 0x7fc1980cbe10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:04.584 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:04.571+0000 7fc197fff640 1 -- 172.21.15.16:0/2667837136 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc19806e8c0 con 0x7fc1980cf570 2023-05-22T15:28:04.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:04.575+0000 7fc197fff640 1 --2- 172.21.15.16:0/2667837136 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc1980cf570 0x7fc1980cf9f0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fc19806d340 tx=0x7fc18801c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12377b1d2809c1b8 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:28:04.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:04.575+0000 7fc1967fc640 1 -- 172.21.15.16:0/2667837136 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1880aa5f0 con 0x7fc1980cf570 2023-05-22T15:28:04.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:04.575+0000 7fc1967fc640 1 -- 172.21.15.16:0/2667837136 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1880aa750 con 0x7fc1980cf570 2023-05-22T15:28:04.587 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:04.575+0000 7fc1967fc640 1 -- 172.21.15.16:0/2667837136 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1880aaae0 con 0x7fc1980cf570 2023-05-22T15:28:04.587 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:04.575+0000 7fc19e1c2640 1 -- 172.21.15.16:0/2667837136 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc1980cf570 msgr2=0x7fc1980cf9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:04.587 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:04.575+0000 7fc19e1c2640 1 --2- 172.21.15.16:0/2667837136 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc1980cf570 0x7fc1980cf9f0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fc19806d340 tx=0x7fc18801c4d0 comp rx=0 tx=0).stop 2023-05-22T15:28:04.588 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:04.575+0000 7fc19e1c2640 1 -- 172.21.15.16:0/2667837136 shutdown_connections 2023-05-22T15:28:04.588 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:04.575+0000 7fc19e1c2640 1 --2- 172.21.15.16:0/2667837136 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc1980cf570 0x7fc1980cf9f0 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:04.588 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:04.575+0000 7fc19e1c2640 1 --2- 172.21.15.16:0/2667837136 >> v2:172.21.15.62:3300/0 conn(0x7fc19815d610 0x7fc1980cbe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:04.588 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:04.575+0000 7fc19e1c2640 1 --2- 172.21.15.16:0/2667837136 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc198159a30 0x7fc19806e380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:04.589 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:04.575+0000 7fc19e1c2640 1 -- 172.21.15.16:0/2667837136 >> 172.21.15.16:0/2667837136 conn(0x7fc1980700b0 msgr2=0x7fc19806d9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:28:04.593 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:04.579+0000 7fc19e1c2640 1 -- 172.21.15.16:0/2667837136 shutdown_connections 2023-05-22T15:28:04.594 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:04.579+0000 7fc19e1c2640 1 -- 172.21.15.16:0/2667837136 wait complete. 2023-05-22T15:28:04.594 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:04.579+0000 7fc19e1c2640 1 Processor -- start 2023-05-22T15:28:04.594 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:04.583+0000 7fc19e1c2640 1 -- start start 2023-05-22T15:28:04.594 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:04.583+0000 7fc19e1c2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc1980cf570 0x7fc1980c61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:04.595 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:04.583+0000 7fc19e1c2640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc198159a30 0x7fc1980c8b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:04.595 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:04.583+0000 7fc19e1c2640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc19815d610 0x7fc1981f71f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:04.595 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:04.583+0000 7fc19e1c2640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1980c8640 con 0x7fc1980cf570 2023-05-22T15:28:04.596 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:04.583+0000 7fc19e1c2640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1980c9090 con 0x7fc198159a30 2023-05-22T15:28:04.596 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:04.583+0000 7fc19e1c2640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1980c7730 con 0x7fc19815d610 2023-05-22T15:28:04.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:04.583+0000 7fc1977fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc1980cf570 0x7fc1980c61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:04.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:04.583+0000 7fc1977fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc1980cf570 0x7fc1980c61b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39812/0 (socket says 172.21.15.16:39812) 2023-05-22T15:28:04.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:04.583+0000 7fc1977fe640 1 -- 172.21.15.16:0/2480561499 learned_addr learned my addr 172.21.15.16:0/2480561499 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:28:04.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:04.583+0000 7fc197fff640 1 --2- 172.21.15.16:0/2480561499 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc19815d610 0x7fc1981f71f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:04.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:04.583+0000 7fc196ffd640 1 --2- 172.21.15.16:0/2480561499 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc198159a30 0x7fc1980c8b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:04.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:04.583+0000 7fc1977fe640 1 -- 172.21.15.16:0/2480561499 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc19815d610 msgr2=0x7fc1981f71f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:04.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:04.583+0000 7fc1977fe640 1 --2- 172.21.15.16:0/2480561499 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc19815d610 0x7fc1981f71f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:04.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:04.583+0000 7fc1977fe640 1 -- 172.21.15.16:0/2480561499 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc198159a30 msgr2=0x7fc1980c8b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:04.600 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:04.583+0000 7fc1977fe640 1 --2- 172.21.15.16:0/2480561499 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc198159a30 0x7fc1980c8b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:04.600 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:04.583+0000 7fc1977fe640 1 -- 172.21.15.16:0/2480561499 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1880094d0 con 0x7fc1980cf570 2023-05-22T15:28:04.600 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:04.583+0000 7fc196ffd640 1 --2- 172.21.15.16:0/2480561499 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc198159a30 0x7fc1980c8b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:28:04.600 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:04.583+0000 7fc1977fe640 1 --2- 172.21.15.16:0/2480561499 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc1980cf570 0x7fc1980c61b0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7fc190007e60 tx=0x7fc19000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:28:04.601 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:04.583+0000 7fc17ffff640 1 -- 172.21.15.16:0/2480561499 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc190014620 con 0x7fc1980cf570 2023-05-22T15:28:04.601 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:04.583+0000 7fc17ffff640 1 -- 172.21.15.16:0/2480561499 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc190014780 con 0x7fc1980cf570 2023-05-22T15:28:04.601 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:04.583+0000 7fc19e1c2640 1 -- 172.21.15.16:0/2480561499 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc198159e00 con 0x7fc1980cf570 2023-05-22T15:28:04.602 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:04.583+0000 7fc17ffff640 1 -- 172.21.15.16:0/2480561499 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc190014b30 con 0x7fc1980cf570 2023-05-22T15:28:04.602 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:04.583+0000 7fc19e1c2640 1 -- 172.21.15.16:0/2480561499 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc19815e3a0 con 0x7fc1980cf570 2023-05-22T15:28:04.602 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:04.583+0000 7fc19e1c2640 1 -- 172.21.15.16:0/2480561499 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1980cc350 con 0x7fc1980cf570 2023-05-22T15:28:04.602 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:04.583+0000 7fc17ffff640 1 -- 172.21.15.16:0/2480561499 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7fc190014c90 con 0x7fc1980cf570 2023-05-22T15:28:04.603 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:04.583+0000 7fc17ffff640 1 --2- 172.21.15.16:0/2480561499 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fc1840653b0 0x7fc184067860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:04.603 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:04.587+0000 7fc17ffff640 1 -- 172.21.15.16:0/2480561499 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6519+0+0 (secure 0 0 0) 0x7fc190083270 con 0x7fc1980cf570 2023-05-22T15:28:04.603 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:04.587+0000 7fc196ffd640 1 --2- 172.21.15.16:0/2480561499 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fc1840653b0 0x7fc184067860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:04.604 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:04.587+0000 7fc196ffd640 1 --2- 172.21.15.16:0/2480561499 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fc1840653b0 0x7fc184067860 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fc1981f7150 tx=0x7fc18c008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:28:04.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:04.591+0000 7fc17ffff640 1 -- 172.21.15.16:0/2480561499 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fc190050d50 con 0x7fc1980cf570 2023-05-22T15:28:04.753 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:04.747+0000 7fc19e1c2640 1 -- 172.21.15.16:0/2480561499 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fc198066910 con 0x7fc1980cf570 2023-05-22T15:28:05.414 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:05.407+0000 7fc17ffff640 1 -- 172.21.15.16:0/2480561499 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v32) v1 ==== 169+0+0 (secure 0 0 0) 0x7fc190054370 con 0x7fc1980cf570 2023-05-22T15:28:05.549 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'cephfs_data' removed 2023-05-22T15:28:05.549 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:05.415+0000 7fc19e1c2640 1 -- 172.21.15.16:0/2480561499 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fc1840653b0 msgr2=0x7fc184067860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:05.550 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:05.415+0000 7fc19e1c2640 1 --2- 172.21.15.16:0/2480561499 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fc1840653b0 0x7fc184067860 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fc1981f7150 tx=0x7fc18c008040 comp rx=0 tx=0).stop 2023-05-22T15:28:05.550 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:05.415+0000 7fc19e1c2640 1 -- 172.21.15.16:0/2480561499 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc1980cf570 msgr2=0x7fc1980c61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:05.550 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:05.415+0000 7fc19e1c2640 1 --2- 172.21.15.16:0/2480561499 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc1980cf570 0x7fc1980c61b0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7fc190007e60 tx=0x7fc19000e4b0 comp rx=0 tx=0).stop 2023-05-22T15:28:05.550 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:05.419+0000 7fc19e1c2640 1 -- 172.21.15.16:0/2480561499 shutdown_connections 2023-05-22T15:28:05.550 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:05.419+0000 7fc19e1c2640 1 --2- 172.21.15.16:0/2480561499 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fc1840653b0 0x7fc184067860 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:05.550 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:05.419+0000 7fc19e1c2640 1 --2- 172.21.15.16:0/2480561499 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc19815d610 0x7fc1981f71f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:05.550 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:05.419+0000 7fc19e1c2640 1 --2- 172.21.15.16:0/2480561499 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc198159a30 0x7fc1980c8b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:05.551 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:05.419+0000 7fc19e1c2640 1 --2- 172.21.15.16:0/2480561499 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc1980cf570 0x7fc1980c61b0 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:05.551 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:05.419+0000 7fc19e1c2640 1 -- 172.21.15.16:0/2480561499 >> 172.21.15.16:0/2480561499 conn(0x7fc1980700b0 msgr2=0x7fc1980ceaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:28:05.551 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:05.419+0000 7fc19e1c2640 1 -- 172.21.15.16:0/2480561499 shutdown_connections 2023-05-22T15:28:05.551 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:05.419+0000 7fc19e1c2640 1 -- 172.21.15.16:0/2480561499 wait complete. 2023-05-22T15:28:05.551 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data_ec cephfs_data_ec --yes-i-really-really-mean-it 2023-05-22T15:28:05.696 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:05.687+0000 7f00c870e640 1 Processor -- start 2023-05-22T15:28:05.697 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:05.687+0000 7f00c870e640 1 -- start start 2023-05-22T15:28:05.697 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:05.691+0000 7f00c870e640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f00c00cef10 0x7f00c00cf310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:05.697 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:05.691+0000 7f00c870e640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00c00d2a20 con 0x7f00c015c8e0 2023-05-22T15:28:05.698 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:05.691+0000 7f00c870e640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00c00cd230 con 0x7f00c00ccc50 2023-05-22T15:28:05.698 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:05.691+0000 7f00c870e640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00c00d23d0 con 0x7f00c00cef10 2023-05-22T15:28:05.698 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:05.691+0000 7f00c64aa640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f00c015c8e0 0x7f00c00d2eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:50136/0 (socket says 172.21.15.16:50136) 2023-05-22T15:28:05.698 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:05.691+0000 7f00c64aa640 1 -- 172.21.15.16:0/2284870969 learned_addr learned my addr 172.21.15.16:0/2284870969 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:28:05.706 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:05.691+0000 7f00c6cab640 1 --2- 172.21.15.16:0/2284870969 >> v2:172.21.15.62:3300/0 conn(0x7f00c00cef10 0x7f00c00cf310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:05.706 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:05.691+0000 7f00c54a8640 1 -- 172.21.15.16:0/2284870969 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1857475858 0 0) 0x7f00c00d2a20 con 0x7f00c015c8e0 2023-05-22T15:28:05.706 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:05.691+0000 7f00c54a8640 1 -- 172.21.15.16:0/2284870969 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f00ac003530 con 0x7f00c015c8e0 2023-05-22T15:28:05.706 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:05.691+0000 7f00c54a8640 1 -- 172.21.15.16:0/2284870969 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1978134001 0 0) 0x7f00ac003530 con 0x7f00c015c8e0 2023-05-22T15:28:05.706 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:05.691+0000 7f00c54a8640 1 -- 172.21.15.16:0/2284870969 >> v2:172.21.15.62:3300/0 conn(0x7f00c00cef10 msgr2=0x7f00c00cf310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:28:05.706 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:05.691+0000 7f00c54a8640 1 --2- 172.21.15.16:0/2284870969 >> v2:172.21.15.62:3300/0 conn(0x7f00c00cef10 0x7f00c00cf310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:05.707 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:05.691+0000 7f00c54a8640 1 -- 172.21.15.16:0/2284870969 >> v1:172.21.15.62:6789/0 conn(0x7f00c00ccc50 legacy=0x7f00c015ccb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:05.707 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:05.691+0000 7f00c54a8640 1 -- 172.21.15.16:0/2284870969 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00c00d26b0 con 0x7f00c015c8e0 2023-05-22T15:28:05.707 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:05.691+0000 7f00c54a8640 1 -- 172.21.15.16:0/2284870969 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7f00bc001680 con 0x7f00c015c8e0 2023-05-22T15:28:05.707 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:05.691+0000 7f00c54a8640 1 -- 172.21.15.16:0/2284870969 >> v1:172.21.15.16:6789/0 conn(0x7f00c015c8e0 legacy=0x7f00c00d2eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:05.707 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:05.691+0000 7f00c54a8640 1 --2- 172.21.15.16:0/2284870969 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f00ac003980 0x7f00ac003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:05.707 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:05.691+0000 7f00c54a8640 1 --2- 172.21.15.16:0/2284870969 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f00ac0047d0 0x7f00ac004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:05.707 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:05.691+0000 7f00c54a8640 1 --2- 172.21.15.16:0/2284870969 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f00ac005180 0x7f00ac00b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:05.708 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:05.691+0000 7f00c54a8640 1 -- 172.21.15.16:0/2284870969 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00c00d23d0 con 0x7f00ac005180 2023-05-22T15:28:05.708 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:05.691+0000 7f00c54a8640 1 -- 172.21.15.16:0/2284870969 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00ac00c090 con 0x7f00ac003980 2023-05-22T15:28:05.708 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:05.691+0000 7f00c54a8640 1 -- 172.21.15.16:0/2284870969 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00ac00c250 con 0x7f00ac0047d0 2023-05-22T15:28:05.708 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:05.691+0000 7f00c6cab640 1 --2- 172.21.15.16:0/2284870969 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f00ac005180 0x7f00ac00b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:05.708 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:05.691+0000 7f00c64aa640 1 --2- 172.21.15.16:0/2284870969 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f00ac003980 0x7f00ac003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:05.708 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:05.691+0000 7f00c5ca9640 1 --2- 172.21.15.16:0/2284870969 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f00ac0047d0 0x7f00ac004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:05.708 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:05.691+0000 7f00c6cab640 1 -- 172.21.15.16:0/2284870969 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f00ac0047d0 msgr2=0x7f00ac004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:28:05.709 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:05.691+0000 7f00c6cab640 1 --2- 172.21.15.16:0/2284870969 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f00ac0047d0 0x7f00ac004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:05.709 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:05.691+0000 7f00c6cab640 1 -- 172.21.15.16:0/2284870969 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f00ac003980 msgr2=0x7f00ac003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:28:05.709 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:05.691+0000 7f00c6cab640 1 --2- 172.21.15.16:0/2284870969 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f00ac003980 0x7f00ac003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:05.709 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:05.691+0000 7f00c6cab640 1 -- 172.21.15.16:0/2284870969 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00ac003530 con 0x7f00ac005180 2023-05-22T15:28:05.709 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:05.691+0000 7f00c64aa640 1 --2- 172.21.15.16:0/2284870969 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f00ac003980 0x7f00ac003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T15:28:05.709 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:05.695+0000 7f00c6cab640 1 -- 172.21.15.16:0/2284870969 reap_dead start 2023-05-22T15:28:05.709 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:05.695+0000 7f00c6cab640 1 --2- 172.21.15.16:0/2284870969 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f00ac005180 0x7f00ac00b990 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f00b8009fc0 tx=0x7f00b801cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51a1d18366fe34d8 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:28:05.710 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:05.695+0000 7f00c54a8640 1 -- 172.21.15.16:0/2284870969 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00b80a5cb0 con 0x7f00ac005180 2023-05-22T15:28:05.710 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:05.695+0000 7f00c54a8640 1 -- 172.21.15.16:0/2284870969 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00b800f040 con 0x7f00ac005180 2023-05-22T15:28:05.710 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:05.695+0000 7f00c54a8640 1 -- 172.21.15.16:0/2284870969 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00b8003be0 con 0x7f00ac005180 2023-05-22T15:28:05.711 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:05.695+0000 7f00c870e640 1 -- 172.21.15.16:0/2284870969 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f00ac005180 msgr2=0x7f00ac00b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:05.711 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:05.695+0000 7f00c870e640 1 --2- 172.21.15.16:0/2284870969 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f00ac005180 0x7f00ac00b990 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f00b8009fc0 tx=0x7f00b801cca0 comp rx=0 tx=0).stop 2023-05-22T15:28:05.711 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:05.695+0000 7f00c870e640 1 -- 172.21.15.16:0/2284870969 shutdown_connections 2023-05-22T15:28:05.711 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:05.695+0000 7f00c870e640 1 --2- 172.21.15.16:0/2284870969 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f00ac005180 0x7f00ac00b990 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:05.711 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:05.695+0000 7f00c870e640 1 -- 172.21.15.16:0/2284870969 >> 172.21.15.16:0/2284870969 conn(0x7f00c0070e50 msgr2=0x7f00c0071250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:28:05.711 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:05.695+0000 7f00c870e640 1 -- 172.21.15.16:0/2284870969 shutdown_connections 2023-05-22T15:28:05.711 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:05.695+0000 7f00c870e640 1 -- 172.21.15.16:0/2284870969 wait complete. 2023-05-22T15:28:05.712 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:05.695+0000 7f00c870e640 1 Processor -- start 2023-05-22T15:28:05.712 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:05.695+0000 7f00c870e640 1 -- start start 2023-05-22T15:28:05.712 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:05.695+0000 7f00c870e640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f00ac005180 0x7f00c01500b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:05.712 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:05.695+0000 7f00c870e640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f00c015d420 0x7f00c015d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:05.712 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:05.695+0000 7f00c870e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f00c00c9f10 0x7f00c00ca380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:05.712 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:05.695+0000 7f00c870e640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00c00cbe70 con 0x7f00c00c9f10 2023-05-22T15:28:05.712 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:05.695+0000 7f00c870e640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00c0154670 con 0x7f00ac005180 2023-05-22T15:28:05.712 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:05.695+0000 7f00c870e640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00c015ddb0 con 0x7f00c015d420 2023-05-22T15:28:05.713 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:05.695+0000 7f00c6cab640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f00c00c9f10 0x7f00c00ca380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:05.713 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:05.699+0000 7f00c6cab640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f00c00c9f10 0x7f00c00ca380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39838/0 (socket says 172.21.15.16:39838) 2023-05-22T15:28:05.713 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:05.699+0000 7f00c6cab640 1 -- 172.21.15.16:0/2516186929 learned_addr learned my addr 172.21.15.16:0/2516186929 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:28:05.713 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:05.699+0000 7f00c5ca9640 1 --2- 172.21.15.16:0/2516186929 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f00c015d420 0x7f00c015d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:05.714 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:05.699+0000 7f00c64aa640 1 --2- 172.21.15.16:0/2516186929 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f00ac005180 0x7f00c01500b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:05.714 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:05.699+0000 7f00c6cab640 1 -- 172.21.15.16:0/2516186929 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f00c015d420 msgr2=0x7f00c015d870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:05.714 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:05.699+0000 7f00c6cab640 1 --2- 172.21.15.16:0/2516186929 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f00c015d420 0x7f00c015d870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:05.714 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:05.699+0000 7f00c6cab640 1 -- 172.21.15.16:0/2516186929 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f00ac005180 msgr2=0x7f00c01500b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:05.714 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:05.699+0000 7f00c6cab640 1 --2- 172.21.15.16:0/2516186929 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f00ac005180 0x7f00c01500b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:05.714 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:05.699+0000 7f00c6cab640 1 -- 172.21.15.16:0/2516186929 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00c00cd510 con 0x7f00c00c9f10 2023-05-22T15:28:05.714 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:05.699+0000 7f00c6cab640 1 --2- 172.21.15.16:0/2516186929 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f00c00c9f10 0x7f00c00ca380 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f00b80987d0 tx=0x7f00b8009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:28:05.715 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:05.699+0000 7f00b6ffd640 1 -- 172.21.15.16:0/2516186929 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00b8003db0 con 0x7f00c00c9f10 2023-05-22T15:28:05.715 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:05.699+0000 7f00b6ffd640 1 -- 172.21.15.16:0/2516186929 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00b8004030 con 0x7f00c00c9f10 2023-05-22T15:28:05.715 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:05.699+0000 7f00c870e640 1 -- 172.21.15.16:0/2516186929 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00c00cd080 con 0x7f00c00c9f10 2023-05-22T15:28:05.715 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:05.699+0000 7f00b6ffd640 1 -- 172.21.15.16:0/2516186929 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00b80043e0 con 0x7f00c00c9f10 2023-05-22T15:28:05.715 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:05.699+0000 7f00c870e640 1 -- 172.21.15.16:0/2516186929 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00c00d2eb0 con 0x7f00c00c9f10 2023-05-22T15:28:05.716 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:05.699+0000 7f00c870e640 1 -- 172.21.15.16:0/2516186929 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00c0154820 con 0x7f00c00c9f10 2023-05-22T15:28:05.716 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:05.699+0000 7f00b6ffd640 1 -- 172.21.15.16:0/2516186929 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7f00b8004540 con 0x7f00c00c9f10 2023-05-22T15:28:05.716 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:05.699+0000 7f00b6ffd640 1 --2- 172.21.15.16:0/2516186929 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f00a4065380 0x7f00a4067830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:05.716 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:05.699+0000 7f00b6ffd640 1 -- 172.21.15.16:0/2516186929 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6047+0+0 (secure 0 0 0) 0x7f00b80a6e40 con 0x7f00c00c9f10 2023-05-22T15:28:05.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:05.699+0000 7f00c64aa640 1 --2- 172.21.15.16:0/2516186929 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f00a4065380 0x7f00a4067830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:05.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:05.699+0000 7f00c64aa640 1 --2- 172.21.15.16:0/2516186929 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f00a4065380 0x7f00a4067830 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f00bc001040 tx=0x7f00bc009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:28:05.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:05.703+0000 7f00b6ffd640 1 -- 172.21.15.16:0/2516186929 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f00c0154820 con 0x7f00c00c9f10 2023-05-22T15:28:05.869 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:05.863+0000 7f00c870e640 1 -- 172.21.15.16:0/2516186929 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f00c0154820 con 0x7f00c00c9f10 2023-05-22T15:28:06.440 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:06.431+0000 7f00b6ffd640 1 -- 172.21.15.16:0/2516186929 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data_ec' removed v33) v1 ==== 178+0+0 (secure 0 0 0) 0x7f00c0154820 con 0x7f00c00c9f10 2023-05-22T15:28:06.443 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'cephfs_data_ec' removed 2023-05-22T15:28:06.446 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:06.439+0000 7f00c870e640 1 -- 172.21.15.16:0/2516186929 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f00a4065380 msgr2=0x7f00a4067830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:06.446 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:06.439+0000 7f00c870e640 1 --2- 172.21.15.16:0/2516186929 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f00a4065380 0x7f00a4067830 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f00bc001040 tx=0x7f00bc009040 comp rx=0 tx=0).stop 2023-05-22T15:28:06.446 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:06.439+0000 7f00c870e640 1 -- 172.21.15.16:0/2516186929 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f00c00c9f10 msgr2=0x7f00c00ca380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:06.446 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:06.439+0000 7f00c870e640 1 --2- 172.21.15.16:0/2516186929 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f00c00c9f10 0x7f00c00ca380 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f00b80987d0 tx=0x7f00b8009750 comp rx=0 tx=0).stop 2023-05-22T15:28:06.449 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:06.439+0000 7f00c870e640 1 -- 172.21.15.16:0/2516186929 shutdown_connections 2023-05-22T15:28:06.450 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:06.439+0000 7f00c870e640 1 --2- 172.21.15.16:0/2516186929 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f00c00c9f10 0x7f00c00ca380 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:06.450 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:06.439+0000 7f00c870e640 1 --2- 172.21.15.16:0/2516186929 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f00a4065380 0x7f00a4067830 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:06.450 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:06.439+0000 7f00c870e640 1 --2- 172.21.15.16:0/2516186929 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f00c015d420 0x7f00c015d870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:06.450 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:06.439+0000 7f00c870e640 1 --2- 172.21.15.16:0/2516186929 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f00ac005180 0x7f00c01500b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:06.450 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:06.439+0000 7f00c870e640 1 -- 172.21.15.16:0/2516186929 >> 172.21.15.16:0/2516186929 conn(0x7f00c0070e50 msgr2=0x7f00c0160150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:28:06.460 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:06.451+0000 7f00c870e640 1 -- 172.21.15.16:0/2516186929 shutdown_connections 2023-05-22T15:28:06.460 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:06.451+0000 7f00c870e640 1 -- 172.21.15.16:0/2516186929 wait complete. 2023-05-22T15:28:06.479 INFO:tasks.ceph.mds.a:Restarting daemon 2023-05-22T15:28:06.479 INFO:tasks.ceph.mds.a:Stopping old one... 2023-05-22T15:28:06.480 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-05-22T15:28:06.480 INFO:teuthology.orchestra.run:waiting for 300 2023-05-22T15:28:06.480 INFO:tasks.ceph.mds.c:Restarting daemon 2023-05-22T15:28:06.480 INFO:tasks.ceph.mds.c:Stopping old one... 2023-05-22T15:28:06.481 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-05-22T15:28:06.481 INFO:teuthology.orchestra.run:waiting for 300 2023-05-22T15:28:06.481 INFO:tasks.ceph.mds.b:Restarting daemon 2023-05-22T15:28:06.481 INFO:tasks.ceph.mds.b:Stopping old one... 2023-05-22T15:28:06.481 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-05-22T15:28:06.482 INFO:teuthology.orchestra.run:waiting for 300 2023-05-22T15:28:06.482 INFO:tasks.ceph.mds.d:Restarting daemon 2023-05-22T15:28:06.482 INFO:tasks.ceph.mds.d:Stopping old one... 2023-05-22T15:28:06.482 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-05-22T15:28:06.482 INFO:teuthology.orchestra.run:waiting for 300 2023-05-22T15:28:06.491 INFO:tasks.ceph.mds.a:Stopped 2023-05-22T15:28:06.491 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-05-22T15:28:06.494 INFO:tasks.ceph.mds.c:Stopped 2023-05-22T15:28:06.495 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-05-22T15:28:06.500 INFO:tasks.ceph.mds.b:Stopped 2023-05-22T15:28:06.501 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-05-22T15:28:06.502 INFO:tasks.ceph.mds.d:Stopped 2023-05-22T15:28:06.502 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-05-22T15:28:06.538 INFO:tasks.ceph.mds.a:Started 2023-05-22T15:28:06.539 DEBUG:teuthology.parallel:result is None 2023-05-22T15:28:06.540 INFO:tasks.ceph.mds.c:Started 2023-05-22T15:28:06.540 DEBUG:teuthology.parallel:result is None 2023-05-22T15:28:06.549 INFO:tasks.ceph.mds.b:Started 2023-05-22T15:28:06.550 DEBUG:teuthology.parallel:result is None 2023-05-22T15:28:06.552 INFO:tasks.ceph.mds.d:Started 2023-05-22T15:28:06.552 DEBUG:teuthology.parallel:result is None 2023-05-22T15:28:06.553 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2023-05-22T15:28:06.608 INFO:tasks.ceph.mds.a.smithi016.stdout:starting mds.a at 2023-05-22T15:28:06.609 INFO:tasks.ceph.mds.c.smithi016.stdout:starting mds.c at 2023-05-22T15:28:06.643 INFO:tasks.ceph.mds.d.smithi062.stdout:starting mds.d at 2023-05-22T15:28:06.648 INFO:tasks.ceph.mds.b.smithi062.stdout:starting mds.b at 2023-05-22T15:28:06.679 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:06.671+0000 7ff474077640 1 Processor -- start 2023-05-22T15:28:06.679 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:06.671+0000 7ff474077640 1 -- start start 2023-05-22T15:28:06.680 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:06.671+0000 7ff474077640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7ff46c0b2880 0x7ff46c0cbfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:06.681 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:06.671+0000 7ff474077640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff46c0cc520 0x7ff46c0cfe30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:06.681 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:06.671+0000 7ff474077640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff46c0cf9a0 con 0x7ff46c159cd0 2023-05-22T15:28:06.681 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:06.671+0000 7ff474077640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff46c15aba0 con 0x7ff46c0b2880 2023-05-22T15:28:06.681 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:06.671+0000 7ff474077640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff46c15ae80 con 0x7ff46c0cc520 2023-05-22T15:28:06.681 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:06.671+0000 7ff471e13640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7ff46c159cd0 0x7ff46c15a0a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:50172/0 (socket says 172.21.15.16:50172) 2023-05-22T15:28:06.681 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:06.671+0000 7ff471e13640 1 -- 172.21.15.16:0/730903200 learned_addr learned my addr 172.21.15.16:0/730903200 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:28:06.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:06.671+0000 7ff471612640 1 --2- 172.21.15.16:0/730903200 >> v2:172.21.15.62:3300/0 conn(0x7ff46c0b2880 0x7ff46c0cbfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:06.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:06.671+0000 7ff472614640 1 --2- 172.21.15.16:0/730903200 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff46c0cc520 0x7ff46c0cfe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:06.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:06.671+0000 7ff470e11640 1 -- 172.21.15.16:0/730903200 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 938261976 0 0) 0x7ff46c0cf9a0 con 0x7ff46c159cd0 2023-05-22T15:28:06.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:06.671+0000 7ff470e11640 1 -- 172.21.15.16:0/730903200 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff458003530 con 0x7ff46c159cd0 2023-05-22T15:28:06.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:06.671+0000 7ff472614640 1 -- 172.21.15.16:0/730903200 >> v2:172.21.15.62:3300/0 conn(0x7ff46c0b2880 msgr2=0x7ff46c0cbfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:28:06.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:06.671+0000 7ff472614640 1 --2- 172.21.15.16:0/730903200 >> v2:172.21.15.62:3300/0 conn(0x7ff46c0b2880 0x7ff46c0cbfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:06.683 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:06.671+0000 7ff472614640 1 -- 172.21.15.16:0/730903200 >> v1:172.21.15.16:6789/0 conn(0x7ff46c159cd0 legacy=0x7ff46c15a0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:06.683 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:06.671+0000 7ff472614640 1 -- 172.21.15.16:0/730903200 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff46c15b160 con 0x7ff46c0cc520 2023-05-22T15:28:06.683 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:06.671+0000 7ff471612640 1 --2- 172.21.15.16:0/730903200 >> v2:172.21.15.62:3300/0 conn(0x7ff46c0b2880 0x7ff46c0cbfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-05-22T15:28:06.685 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:06.675+0000 7ff472614640 1 --2- 172.21.15.16:0/730903200 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff46c0cc520 0x7ff46c0cfe30 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7ff464009d90 tx=0x7ff46401c760 comp rx=0 tx=0).ready entity=mon.2 client_cookie=28ca9a25f772ae20 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:28:06.685 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:06.675+0000 7ff470e11640 1 -- 172.21.15.16:0/730903200 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff468001680 con 0x7ff46c0cc520 2023-05-22T15:28:06.685 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:06.675+0000 7ff470e11640 1 -- 172.21.15.16:0/730903200 <== mon.2 v2:172.21.15.62:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4640a64c0 con 0x7ff46c0cc520 2023-05-22T15:28:06.685 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:06.675+0000 7ff470e11640 1 -- 172.21.15.16:0/730903200 <== mon.2 v2:172.21.15.62:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4640a6850 con 0x7ff46c0cc520 2023-05-22T15:28:06.685 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:06.675+0000 7ff474077640 1 -- 172.21.15.16:0/730903200 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff46c0cc520 msgr2=0x7ff46c0cfe30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:06.686 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:06.675+0000 7ff474077640 1 --2- 172.21.15.16:0/730903200 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff46c0cc520 0x7ff46c0cfe30 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7ff464009d90 tx=0x7ff46401c760 comp rx=0 tx=0).stop 2023-05-22T15:28:06.686 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:06.675+0000 7ff474077640 1 -- 172.21.15.16:0/730903200 shutdown_connections 2023-05-22T15:28:06.686 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:06.675+0000 7ff474077640 1 --2- 172.21.15.16:0/730903200 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff46c0cc520 0x7ff46c0cfe30 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:06.686 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:06.675+0000 7ff474077640 1 --2- 172.21.15.16:0/730903200 >> v2:172.21.15.62:3300/0 conn(0x7ff46c0b2880 0x7ff46c0cbfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:06.686 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:06.675+0000 7ff474077640 1 -- 172.21.15.16:0/730903200 >> 172.21.15.16:0/730903200 conn(0x7ff46c06f5c0 msgr2=0x7ff46c06f9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:28:06.686 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:06.675+0000 7ff474077640 1 -- 172.21.15.16:0/730903200 shutdown_connections 2023-05-22T15:28:06.686 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:06.675+0000 7ff474077640 1 -- 172.21.15.16:0/730903200 wait complete. 2023-05-22T15:28:06.687 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:06.679+0000 7ff474077640 1 Processor -- start 2023-05-22T15:28:06.687 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:06.679+0000 7ff474077640 1 -- start start 2023-05-22T15:28:06.688 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:06.679+0000 7ff474077640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff46c0b2880 0x7ff46c0c9dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:06.689 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:06.679+0000 7ff474077640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff46c0cc520 0x7ff46c144fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:06.689 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:06.679+0000 7ff474077640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff46c159cd0 0x7ff46c14b210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:06.689 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:06.679+0000 7ff474077640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff46c1530a0 con 0x7ff46c0cc520 2023-05-22T15:28:06.689 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:06.679+0000 7ff474077640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff46c145500 con 0x7ff46c159cd0 2023-05-22T15:28:06.689 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:06.679+0000 7ff474077640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff46c1f1fb0 con 0x7ff46c0b2880 2023-05-22T15:28:06.689 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:06.679+0000 7ff471612640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff46c0cc520 0x7ff46c144fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:06.690 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:06.679+0000 7ff471612640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff46c0cc520 0x7ff46c144fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39880/0 (socket says 172.21.15.16:39880) 2023-05-22T15:28:06.690 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:06.679+0000 7ff471e13640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff46c0b2880 0x7ff46c0c9dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:06.690 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:06.679+0000 7ff471612640 1 -- 172.21.15.16:0/849864254 learned_addr learned my addr 172.21.15.16:0/849864254 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:28:06.690 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:06.679+0000 7ff472614640 1 --2- 172.21.15.16:0/849864254 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff46c159cd0 0x7ff46c14b210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:06.690 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:06.679+0000 7ff471612640 1 -- 172.21.15.16:0/849864254 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff46c0b2880 msgr2=0x7ff46c0c9dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:06.690 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:06.679+0000 7ff471612640 1 --2- 172.21.15.16:0/849864254 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff46c0b2880 0x7ff46c0c9dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:06.690 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:06.679+0000 7ff471612640 1 -- 172.21.15.16:0/849864254 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff46c159cd0 msgr2=0x7ff46c14b210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:06.691 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:06.679+0000 7ff471612640 1 --2- 172.21.15.16:0/849864254 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff46c159cd0 0x7ff46c14b210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:06.691 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:06.679+0000 7ff471612640 1 -- 172.21.15.16:0/849864254 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff45c009420 con 0x7ff46c0cc520 2023-05-22T15:28:06.691 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:06.679+0000 7ff471612640 1 --2- 172.21.15.16:0/849864254 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff46c0cc520 0x7ff46c144fc0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7ff45c00ebe0 tx=0x7ff45c00c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:28:06.691 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:06.679+0000 7ff4627fc640 1 -- 172.21.15.16:0/849864254 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff45c00eee0 con 0x7ff46c0cc520 2023-05-22T15:28:06.691 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:06.679+0000 7ff4627fc640 1 -- 172.21.15.16:0/849864254 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff45c00cc30 con 0x7ff46c0cc520 2023-05-22T15:28:06.691 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:06.679+0000 7ff474077640 1 -- 172.21.15.16:0/849864254 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff464009a70 con 0x7ff46c0cc520 2023-05-22T15:28:06.691 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:06.679+0000 7ff4627fc640 1 -- 172.21.15.16:0/849864254 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff45c013470 con 0x7ff46c0cc520 2023-05-22T15:28:06.691 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:06.679+0000 7ff474077640 1 -- 172.21.15.16:0/849864254 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff46c15a940 con 0x7ff46c0cc520 2023-05-22T15:28:06.692 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:06.679+0000 7ff474077640 1 -- 172.21.15.16:0/849864254 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff46c15a7b0 con 0x7ff46c0cc520 2023-05-22T15:28:06.692 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:06.679+0000 7ff4627fc640 1 -- 172.21.15.16:0/849864254 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7ff45c0135d0 con 0x7ff46c0cc520 2023-05-22T15:28:06.692 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:06.679+0000 7ff4627fc640 1 --2- 172.21.15.16:0/849864254 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ff450065380 0x7ff450067830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:06.692 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:06.683+0000 7ff4627fc640 1 -- 172.21.15.16:0/849864254 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 5616+0+0 (secure 0 0 0) 0x7ff45c0829d0 con 0x7ff46c0cc520 2023-05-22T15:28:06.692 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:06.683+0000 7ff471e13640 1 --2- 172.21.15.16:0/849864254 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ff450065380 0x7ff450067830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:06.692 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:06.683+0000 7ff471e13640 1 --2- 172.21.15.16:0/849864254 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ff450065380 0x7ff450067830 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7ff468000f70 tx=0x7ff468001a80 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:28:06.693 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:06.687+0000 7ff4627fc640 1 -- 172.21.15.16:0/849864254 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7ff45c0507b0 con 0x7ff46c0cc520 2023-05-22T15:28:06.859 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:06.851+0000 7ff474077640 1 -- 172.21.15.16:0/849864254 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7ff46c0d0b80 con 0x7ff46c0cc520 2023-05-22T15:28:07.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:07.675+0000 7ff4627fc640 1 -- 172.21.15.16:0/849864254 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v38) v1 ==== 97+0+0 (secure 0 0 0) 0x7ff45c053dd0 con 0x7ff46c0cc520 2023-05-22T15:28:07.683 INFO:teuthology.orchestra.run.smithi016.stderr: removed all blocklist entries 2023-05-22T15:28:07.686 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:07.679+0000 7ff474077640 1 -- 172.21.15.16:0/849864254 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ff450065380 msgr2=0x7ff450067830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:07.686 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:07.679+0000 7ff474077640 1 --2- 172.21.15.16:0/849864254 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ff450065380 0x7ff450067830 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7ff468000f70 tx=0x7ff468001a80 comp rx=0 tx=0).stop 2023-05-22T15:28:07.686 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:07.679+0000 7ff474077640 1 -- 172.21.15.16:0/849864254 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff46c0cc520 msgr2=0x7ff46c144fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:07.687 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:07.679+0000 7ff474077640 1 --2- 172.21.15.16:0/849864254 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff46c0cc520 0x7ff46c144fc0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7ff45c00ebe0 tx=0x7ff45c00c520 comp rx=0 tx=0).stop 2023-05-22T15:28:07.687 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:07.679+0000 7ff474077640 1 -- 172.21.15.16:0/849864254 shutdown_connections 2023-05-22T15:28:07.687 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:07.679+0000 7ff474077640 1 --2- 172.21.15.16:0/849864254 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff46c159cd0 0x7ff46c14b210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:07.687 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:07.679+0000 7ff474077640 1 --2- 172.21.15.16:0/849864254 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff46c0cc520 0x7ff46c144fc0 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:07.688 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:07.679+0000 7ff474077640 1 --2- 172.21.15.16:0/849864254 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ff450065380 0x7ff450067830 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:07.688 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:07.679+0000 7ff474077640 1 --2- 172.21.15.16:0/849864254 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff46c0b2880 0x7ff46c0c9dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:07.689 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:07.679+0000 7ff474077640 1 -- 172.21.15.16:0/849864254 >> 172.21.15.16:0/849864254 conn(0x7ff46c06f5c0 msgr2=0x7ff46c15a0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:28:07.689 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:07.679+0000 7ff474077640 1 -- 172.21.15.16:0/849864254 shutdown_connections 2023-05-22T15:28:07.689 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:07.679+0000 7ff474077640 1 -- 172.21.15.16:0/849864254 wait complete. 2023-05-22T15:28:07.705 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2023-05-22T15:28:07.832 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:07.823+0000 7fd759e1a640 1 Processor -- start 2023-05-22T15:28:07.832 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:07.823+0000 7fd759e1a640 1 -- start start 2023-05-22T15:28:07.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:07.827+0000 7fd759e1a640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fd75406e250 0x7fd7540cc780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:07.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:07.827+0000 7fd759e1a640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd7540b5e80 0x7fd7540b3e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:07.834 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:07.827+0000 7fd759e1a640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd754159ee0 con 0x7fd754159b10 2023-05-22T15:28:07.834 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:07.827+0000 7fd759e1a640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7540b30b0 con 0x7fd7540b5e80 2023-05-22T15:28:07.834 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:07.827+0000 7fd759e1a640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7540b2810 con 0x7fd75406e250 2023-05-22T15:28:07.834 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:07.827+0000 7fd753fff640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd7540b5e80 0x7fd7540b3e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:07.834 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:07.827+0000 7fd753fff640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd7540b5e80 0x7fd7540b3e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39890/0 (socket says 172.21.15.16:39890) 2023-05-22T15:28:07.834 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:07.827+0000 7fd753fff640 1 -- 172.21.15.16:0/1237378372 learned_addr learned my addr 172.21.15.16:0/1237378372 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:28:07.834 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:07.827+0000 7fd752ffd640 1 --2- 172.21.15.16:0/1237378372 >> v2:172.21.15.62:3300/0 conn(0x7fd75406e250 0x7fd7540cc780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:07.835 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:07.827+0000 7fd753fff640 1 -- 172.21.15.16:0/1237378372 >> v2:172.21.15.62:3300/0 conn(0x7fd75406e250 msgr2=0x7fd7540cc780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:28:07.835 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:07.827+0000 7fd753fff640 1 --2- 172.21.15.16:0/1237378372 >> v2:172.21.15.62:3300/0 conn(0x7fd75406e250 0x7fd7540cc780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:07.835 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:07.827+0000 7fd753fff640 1 -- 172.21.15.16:0/1237378372 >> v1:172.21.15.62:6789/0 conn(0x7fd754159b10 legacy=0x7fd75406db40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:28:07.836 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:07.827+0000 7fd753fff640 1 -- 172.21.15.16:0/1237378372 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7540b2af0 con 0x7fd7540b5e80 2023-05-22T15:28:07.836 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:07.827+0000 7fd753fff640 1 --2- 172.21.15.16:0/1237378372 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd7540b5e80 0x7fd7540b3e30 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7fd754071330 tx=0x7fd74c01c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=936d9fc1b672d6d9 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:28:07.837 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:07.831+0000 7fd7527fc640 1 -- 172.21.15.16:0/1237378372 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd74c0aa5f0 con 0x7fd7540b5e80 2023-05-22T15:28:07.837 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:07.831+0000 7fd7527fc640 1 -- 172.21.15.16:0/1237378372 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd74c0aa750 con 0x7fd7540b5e80 2023-05-22T15:28:07.837 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:07.831+0000 7fd7527fc640 1 -- 172.21.15.16:0/1237378372 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd74c0aaae0 con 0x7fd7540b5e80 2023-05-22T15:28:07.839 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:07.831+0000 7fd759e1a640 1 -- 172.21.15.16:0/1237378372 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd7540b5e80 msgr2=0x7fd7540b3e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:07.839 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:07.831+0000 7fd759e1a640 1 --2- 172.21.15.16:0/1237378372 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd7540b5e80 0x7fd7540b3e30 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7fd754071330 tx=0x7fd74c01c490 comp rx=0 tx=0).stop 2023-05-22T15:28:07.839 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:07.831+0000 7fd759e1a640 1 -- 172.21.15.16:0/1237378372 shutdown_connections 2023-05-22T15:28:07.840 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:07.831+0000 7fd759e1a640 1 --2- 172.21.15.16:0/1237378372 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd7540b5e80 0x7fd7540b3e30 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:07.840 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:07.831+0000 7fd759e1a640 1 --2- 172.21.15.16:0/1237378372 >> v2:172.21.15.62:3300/0 conn(0x7fd75406e250 0x7fd7540cc780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:07.841 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:07.831+0000 7fd759e1a640 1 -- 172.21.15.16:0/1237378372 >> 172.21.15.16:0/1237378372 conn(0x7fd75406f560 msgr2=0x7fd75406f960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:28:07.841 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:07.831+0000 7fd759e1a640 1 -- 172.21.15.16:0/1237378372 shutdown_connections 2023-05-22T15:28:07.841 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:07.831+0000 7fd759e1a640 1 -- 172.21.15.16:0/1237378372 wait complete. 2023-05-22T15:28:07.842 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:07.831+0000 7fd759e1a640 1 Processor -- start 2023-05-22T15:28:07.842 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:07.831+0000 7fd759e1a640 1 -- start start 2023-05-22T15:28:07.845 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:07.831+0000 7fd759e1a640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd75406e250 0x7fd754156400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:07.845 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:07.831+0000 7fd759e1a640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd7540b5e80 0x7fd7541550f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:07.846 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:07.831+0000 7fd759e1a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd754159b10 0x7fd754155630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:07.846 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:07.831+0000 7fd759e1a640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd754156940 con 0x7fd754159b10 2023-05-22T15:28:07.846 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:07.831+0000 7fd759e1a640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7540c9b30 con 0x7fd75406e250 2023-05-22T15:28:07.846 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:07.831+0000 7fd759e1a640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7540c9e10 con 0x7fd7540b5e80 2023-05-22T15:28:07.847 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:07.831+0000 7fd753fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd754159b10 0x7fd754155630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:07.847 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:07.831+0000 7fd753fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd754159b10 0x7fd754155630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39902/0 (socket says 172.21.15.16:39902) 2023-05-22T15:28:07.847 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:07.831+0000 7fd753fff640 1 -- 172.21.15.16:0/2761032047 learned_addr learned my addr 172.21.15.16:0/2761032047 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:28:07.847 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:07.831+0000 7fd7537fe640 1 --2- 172.21.15.16:0/2761032047 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd75406e250 0x7fd754156400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:07.847 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:07.831+0000 7fd752ffd640 1 --2- 172.21.15.16:0/2761032047 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd7540b5e80 0x7fd7541550f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:07.848 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:07.831+0000 7fd753fff640 1 -- 172.21.15.16:0/2761032047 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd7540b5e80 msgr2=0x7fd7541550f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:07.848 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:07.831+0000 7fd753fff640 1 --2- 172.21.15.16:0/2761032047 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd7540b5e80 0x7fd7541550f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:07.848 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:07.831+0000 7fd753fff640 1 -- 172.21.15.16:0/2761032047 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd75406e250 msgr2=0x7fd754156400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:07.849 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:07.831+0000 7fd753fff640 1 --2- 172.21.15.16:0/2761032047 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd75406e250 0x7fd754156400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:07.849 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:07.831+0000 7fd753fff640 1 -- 172.21.15.16:0/2761032047 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd74c0094d0 con 0x7fd754159b10 2023-05-22T15:28:07.849 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:07.831+0000 7fd753fff640 1 --2- 172.21.15.16:0/2761032047 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd754159b10 0x7fd754155630 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7fd74c0ac040 tx=0x7fd74c01acd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:28:07.850 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:07.831+0000 7fd73bfff640 1 -- 172.21.15.16:0/2761032047 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd74c0b3800 con 0x7fd754159b10 2023-05-22T15:28:07.850 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:07.831+0000 7fd73bfff640 1 -- 172.21.15.16:0/2761032047 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd74c0b3980 con 0x7fd754159b10 2023-05-22T15:28:07.850 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:07.831+0000 7fd759e1a640 1 -- 172.21.15.16:0/2761032047 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7540b4b80 con 0x7fd754159b10 2023-05-22T15:28:07.851 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:07.831+0000 7fd73bfff640 1 -- 172.21.15.16:0/2761032047 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd74c0b3d30 con 0x7fd754159b10 2023-05-22T15:28:07.851 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:07.835+0000 7fd759e1a640 1 -- 172.21.15.16:0/2761032047 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7540b3910 con 0x7fd754159b10 2023-05-22T15:28:07.851 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:07.835+0000 7fd759e1a640 1 -- 172.21.15.16:0/2761032047 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7540b4d10 con 0x7fd754159b10 2023-05-22T15:28:07.851 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:07.835+0000 7fd73bfff640 1 -- 172.21.15.16:0/2761032047 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7fd74c0b2380 con 0x7fd754159b10 2023-05-22T15:28:07.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:07.835+0000 7fd73bfff640 1 --2- 172.21.15.16:0/2761032047 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd740065380 0x7fd740067830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:07.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:07.835+0000 7fd7537fe640 1 --2- 172.21.15.16:0/2761032047 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd740065380 0x7fd740067830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:07.853 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:07.835+0000 7fd73bfff640 1 -- 172.21.15.16:0/2761032047 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 5530+0+0 (secure 0 0 0) 0x7fd74c0a7070 con 0x7fd754159b10 2023-05-22T15:28:07.853 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:07.835+0000 7fd7537fe640 1 --2- 172.21.15.16:0/2761032047 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd740065380 0x7fd740067830 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fd744009f20 tx=0x7fd7440094d0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:28:07.854 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:07.839+0000 7fd73bfff640 1 -- 172.21.15.16:0/2761032047 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fd74c0eacc0 con 0x7fd754159b10 2023-05-22T15:28:08.053 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.047+0000 7fd759e1a640 1 -- 172.21.15.16:0/2761032047 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fd75404dbe0 con 0x7fd754159b10 2023-05-22T15:28:08.054 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.047+0000 7fd73bfff640 1 -- 172.21.15.16:0/2761032047 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v8) v1 ==== 80+0+6645 (secure 0 0 0) 0x7fd74c0ee2e0 con 0x7fd754159b10 2023-05-22T15:28:08.054 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:28:08.054 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-05-22T15:28:08.054 INFO:teuthology.orchestra.run.smithi016.stdout: "auth_dump": [ 2023-05-22T15:28:08.054 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T15:28:08.055 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "mds.a", 2023-05-22T15:28:08.055 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQCAiWtkw/K0GRAA9hvTMKMmcbgKV68GeoC4ag==", 2023-05-22T15:28:08.055 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-05-22T15:28:08.055 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2023-05-22T15:28:08.055 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow *", 2023-05-22T15:28:08.055 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow *", 2023-05-22T15:28:08.055 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-05-22T15:28:08.056 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T15:28:08.056 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:28:08.056 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T15:28:08.056 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "mds.b", 2023-05-22T15:28:08.056 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQCAiWtkoFzfJRAAxCiB0MGM5apMVA81WDq+dA==", 2023-05-22T15:28:08.056 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-05-22T15:28:08.056 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2023-05-22T15:28:08.057 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow *", 2023-05-22T15:28:08.057 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow *", 2023-05-22T15:28:08.057 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-05-22T15:28:08.057 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T15:28:08.057 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:28:08.057 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T15:28:08.057 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "mds.c", 2023-05-22T15:28:08.057 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQCAiWtkvgOKHxAAMuASPtMqwyoqpZ5I+MRS7A==", 2023-05-22T15:28:08.058 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-05-22T15:28:08.058 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2023-05-22T15:28:08.058 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow *", 2023-05-22T15:28:08.058 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow *", 2023-05-22T15:28:08.058 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-05-22T15:28:08.058 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T15:28:08.058 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:28:08.058 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T15:28:08.059 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "mds.d", 2023-05-22T15:28:08.059 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQCAiWtk6hPQKxAAlCtfC36vja5ftz6uOqUhHg==", 2023-05-22T15:28:08.059 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-05-22T15:28:08.059 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2023-05-22T15:28:08.059 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow *", 2023-05-22T15:28:08.059 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow *", 2023-05-22T15:28:08.059 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-05-22T15:28:08.060 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T15:28:08.060 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:28:08.060 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T15:28:08.060 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.0", 2023-05-22T15:28:08.060 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQCCiWtk6ZLoCRAAKdCQitZuPnWcIpkiXhJeww==", 2023-05-22T15:28:08.060 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-05-22T15:28:08.060 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2023-05-22T15:28:08.060 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2023-05-22T15:28:08.061 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-05-22T15:28:08.061 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T15:28:08.061 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:28:08.061 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T15:28:08.061 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.1", 2023-05-22T15:28:08.061 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQCFiWtkXtVeERAAkgPeDKuFmWXrDiCyqWOYwQ==", 2023-05-22T15:28:08.061 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-05-22T15:28:08.062 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2023-05-22T15:28:08.062 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2023-05-22T15:28:08.062 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-05-22T15:28:08.062 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T15:28:08.062 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:28:08.062 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T15:28:08.062 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.2", 2023-05-22T15:28:08.062 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQCIiWtkdGaRHRAAx3POIp3mnz7p7XcCVMXm5Q==", 2023-05-22T15:28:08.063 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-05-22T15:28:08.063 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2023-05-22T15:28:08.063 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2023-05-22T15:28:08.063 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-05-22T15:28:08.063 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T15:28:08.063 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:28:08.063 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T15:28:08.063 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.3", 2023-05-22T15:28:08.064 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQCLiWtkSBydJhAAsgc19Xaf/GOyy9nm9MLdJQ==", 2023-05-22T15:28:08.064 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-05-22T15:28:08.064 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2023-05-22T15:28:08.064 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2023-05-22T15:28:08.064 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-05-22T15:28:08.064 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T15:28:08.064 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:28:08.065 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T15:28:08.065 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.4", 2023-05-22T15:28:08.065 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQCRiWtkueYvJRAARXbHlKHrTbgTqBArRJfvBA==", 2023-05-22T15:28:08.065 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-05-22T15:28:08.065 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2023-05-22T15:28:08.065 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2023-05-22T15:28:08.065 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-05-22T15:28:08.065 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T15:28:08.066 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:28:08.066 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T15:28:08.066 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.5", 2023-05-22T15:28:08.066 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQCUiWtkH3hwLRAAUekP6y+wkJVPh+OPDiZbzQ==", 2023-05-22T15:28:08.066 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-05-22T15:28:08.066 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2023-05-22T15:28:08.066 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2023-05-22T15:28:08.066 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-05-22T15:28:08.067 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T15:28:08.067 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:28:08.067 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T15:28:08.067 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.6", 2023-05-22T15:28:08.067 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQCXiWtkLsnBNhAAQ8ilX8ZU4ncJ2JJU0I+U4w==", 2023-05-22T15:28:08.067 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-05-22T15:28:08.067 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2023-05-22T15:28:08.067 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2023-05-22T15:28:08.068 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-05-22T15:28:08.068 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T15:28:08.068 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:28:08.068 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T15:28:08.068 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.7", 2023-05-22T15:28:08.068 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQCbiWtkUvXjBBAApMWw7wdfxWq0WEQDL2qprA==", 2023-05-22T15:28:08.068 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-05-22T15:28:08.069 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2023-05-22T15:28:08.069 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2023-05-22T15:28:08.069 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-05-22T15:28:08.069 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T15:28:08.069 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:28:08.069 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T15:28:08.069 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "client.0", 2023-05-22T15:28:08.069 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQCAiWtkZwuUMRAAnYqSFiaT/QRsajdjNgTetA==", 2023-05-22T15:28:08.071 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-05-22T15:28:08.071 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2023-05-22T15:28:08.071 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow r", 2023-05-22T15:28:08.072 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow rw", 2023-05-22T15:28:08.072 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow rwx" 2023-05-22T15:28:08.072 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T15:28:08.072 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:28:08.072 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T15:28:08.072 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "client.1", 2023-05-22T15:28:08.073 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQCAiWtkPimmORAABzNL2rANz5TgCl3p++GVng==", 2023-05-22T15:28:08.073 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-05-22T15:28:08.073 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2023-05-22T15:28:08.073 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow r", 2023-05-22T15:28:08.073 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow rw", 2023-05-22T15:28:08.073 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow rwx" 2023-05-22T15:28:08.074 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T15:28:08.074 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:28:08.074 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T15:28:08.074 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "client.2", 2023-05-22T15:28:08.074 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQCAiWtkqkPMNhAAqugrnD4Mh/4At8QAFmzqcg==", 2023-05-22T15:28:08.074 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-05-22T15:28:08.074 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2023-05-22T15:28:08.075 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow r", 2023-05-22T15:28:08.075 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow rw", 2023-05-22T15:28:08.075 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow rwx" 2023-05-22T15:28:08.075 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T15:28:08.075 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:28:08.075 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T15:28:08.076 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "client.3", 2023-05-22T15:28:08.076 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQCBiWtkpvlWAxAAPuedivXBGZGHIO6hSnz8PQ==", 2023-05-22T15:28:08.076 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-05-22T15:28:08.076 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2023-05-22T15:28:08.076 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow r", 2023-05-22T15:28:08.077 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow rw", 2023-05-22T15:28:08.077 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow rwx" 2023-05-22T15:28:08.077 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T15:28:08.077 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:28:08.077 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T15:28:08.077 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "client.admin", 2023-05-22T15:28:08.077 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQCAiWtkxG9hAhAAfnUnRrBRUFoCEtsb2riNOQ==", 2023-05-22T15:28:08.078 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-05-22T15:28:08.078 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow *", 2023-05-22T15:28:08.078 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow *", 2023-05-22T15:28:08.078 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow *", 2023-05-22T15:28:08.078 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-05-22T15:28:08.078 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T15:28:08.078 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:28:08.078 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T15:28:08.079 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "client.bootstrap-mds", 2023-05-22T15:28:08.079 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQChiWtkkSyMDxAArJSQ84cleEGWWavjLM9inw==", 2023-05-22T15:28:08.079 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-05-22T15:28:08.079 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile bootstrap-mds" 2023-05-22T15:28:08.079 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T15:28:08.079 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:28:08.080 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T15:28:08.080 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "client.bootstrap-mgr", 2023-05-22T15:28:08.080 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQChiWtkSEaMDxAADmJ4ZOTgpns+vaLhmmfJ3A==", 2023-05-22T15:28:08.080 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-05-22T15:28:08.081 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile bootstrap-mgr" 2023-05-22T15:28:08.081 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T15:28:08.081 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:28:08.081 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T15:28:08.081 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "client.bootstrap-osd", 2023-05-22T15:28:08.081 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQChiWtky1uMDxAAPcYKIrGwKZm7gZA9mGaoXA==", 2023-05-22T15:28:08.081 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-05-22T15:28:08.082 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile bootstrap-osd" 2023-05-22T15:28:08.082 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T15:28:08.082 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:28:08.082 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T15:28:08.082 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "client.bootstrap-rbd", 2023-05-22T15:28:08.082 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQChiWtkbnCMDxAALwcNTKSAFI6nCCc0x6cbXQ==", 2023-05-22T15:28:08.083 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-05-22T15:28:08.083 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile bootstrap-rbd" 2023-05-22T15:28:08.083 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T15:28:08.083 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:28:08.083 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T15:28:08.084 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "client.bootstrap-rbd-mirror", 2023-05-22T15:28:08.084 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQChiWtk1oWMDxAAsI85EqceNmB+JEvOhJy6LQ==", 2023-05-22T15:28:08.084 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-05-22T15:28:08.084 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile bootstrap-rbd-mirror" 2023-05-22T15:28:08.084 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T15:28:08.084 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:28:08.084 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T15:28:08.085 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "client.bootstrap-rgw", 2023-05-22T15:28:08.085 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQChiWtklZyMDxAA74mlZHrB6PncMNtdQeVEvQ==", 2023-05-22T15:28:08.085 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-05-22T15:28:08.085 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile bootstrap-rgw" 2023-05-22T15:28:08.085 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T15:28:08.085 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:28:08.085 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T15:28:08.085 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "mgr.x", 2023-05-22T15:28:08.086 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQCAiWtkkaIqFhAAb/4BI6FYWOy3qOSaylXLSw==", 2023-05-22T15:28:08.086 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-05-22T15:28:08.086 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow *", 2023-05-22T15:28:08.086 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile mgr", 2023-05-22T15:28:08.086 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-05-22T15:28:08.086 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T15:28:08.087 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:28:08.087 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T15:28:08.087 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "mgr.y", 2023-05-22T15:28:08.087 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQCAiWtkhgVWExAAjDoh7cexFVfsigsYfGkmVg==", 2023-05-22T15:28:08.088 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-05-22T15:28:08.088 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow *", 2023-05-22T15:28:08.088 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile mgr", 2023-05-22T15:28:08.088 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-05-22T15:28:08.088 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T15:28:08.089 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T15:28:08.089 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-05-22T15:28:08.089 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-05-22T15:28:08.090 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.047+0000 7fd759e1a640 1 -- 172.21.15.16:0/2761032047 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd740065380 msgr2=0x7fd740067830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:08.090 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.047+0000 7fd759e1a640 1 --2- 172.21.15.16:0/2761032047 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd740065380 0x7fd740067830 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fd744009f20 tx=0x7fd7440094d0 comp rx=0 tx=0).stop 2023-05-22T15:28:08.091 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.047+0000 7fd759e1a640 1 -- 172.21.15.16:0/2761032047 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd754159b10 msgr2=0x7fd754155630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:08.091 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.047+0000 7fd759e1a640 1 --2- 172.21.15.16:0/2761032047 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd754159b10 0x7fd754155630 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7fd74c0ac040 tx=0x7fd74c01acd0 comp rx=0 tx=0).stop 2023-05-22T15:28:08.091 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.047+0000 7fd759e1a640 1 -- 172.21.15.16:0/2761032047 shutdown_connections 2023-05-22T15:28:08.091 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.047+0000 7fd759e1a640 1 --2- 172.21.15.16:0/2761032047 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd754159b10 0x7fd754155630 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:08.091 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.047+0000 7fd759e1a640 1 --2- 172.21.15.16:0/2761032047 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd740065380 0x7fd740067830 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:08.091 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.047+0000 7fd759e1a640 1 --2- 172.21.15.16:0/2761032047 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd7540b5e80 0x7fd7541550f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:08.091 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.047+0000 7fd759e1a640 1 --2- 172.21.15.16:0/2761032047 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd75406e250 0x7fd754156400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:08.092 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.047+0000 7fd759e1a640 1 -- 172.21.15.16:0/2761032047 >> 172.21.15.16:0/2761032047 conn(0x7fd75406f560 msgr2=0x7fd7540cbcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:28:08.092 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.047+0000 7fd759e1a640 1 -- 172.21.15.16:0/2761032047 shutdown_connections 2023-05-22T15:28:08.092 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.047+0000 7fd759e1a640 1 -- 172.21.15.16:0/2761032047 wait complete. 2023-05-22T15:28:08.092 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mds 2023-05-22T15:28:08.206 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.199+0000 7f1679a33640 1 Processor -- start 2023-05-22T15:28:08.206 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.199+0000 7f1679a33640 1 -- start start 2023-05-22T15:28:08.207 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.199+0000 7f1679a33640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f16741531b0 0x7f16740afb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:08.207 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.199+0000 7f1679a33640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f16740b0080 0x7f1674159330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:08.208 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.199+0000 7f1679a33640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f1674159870 0x7f16740c8ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:08.208 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.199+0000 7f1679a33640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16740b0520 con 0x7f16741531b0 2023-05-22T15:28:08.208 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.199+0000 7f1679a33640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f167415a0a0 con 0x7f1674159870 2023-05-22T15:28:08.208 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.199+0000 7f1679a33640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1674153580 con 0x7f16740b0080 2023-05-22T15:28:08.208 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.199+0000 7f1672ffd640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f16741531b0 0x7f16740afb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:08.208 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.199+0000 7f1672ffd640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f16741531b0 0x7f16740afb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39918/0 (socket says 172.21.15.16:39918) 2023-05-22T15:28:08.209 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.199+0000 7f1672ffd640 1 -- 172.21.15.16:0/2266595119 learned_addr learned my addr 172.21.15.16:0/2266595119 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:28:08.209 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.199+0000 7f16737fe640 1 --2- 172.21.15.16:0/2266595119 >> v2:172.21.15.62:3300/0 conn(0x7f1674159870 0x7f16740c8ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:08.209 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.199+0000 7f166a7fc640 1 --2- 172.21.15.16:0/2266595119 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f16740b0080 0x7f1674159330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:08.210 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.199+0000 7f1672ffd640 1 -- 172.21.15.16:0/2266595119 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f16740b0080 msgr2=0x7f1674159330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:28:08.210 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.199+0000 7f1672ffd640 1 --2- 172.21.15.16:0/2266595119 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f16740b0080 0x7f1674159330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:08.210 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.199+0000 7f1672ffd640 1 -- 172.21.15.16:0/2266595119 >> v2:172.21.15.62:3300/0 conn(0x7f1674159870 msgr2=0x7f16740c8ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:28:08.210 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.199+0000 7f1672ffd640 1 --2- 172.21.15.16:0/2266595119 >> v2:172.21.15.62:3300/0 conn(0x7f1674159870 0x7f16740c8ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:08.210 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.199+0000 7f1672ffd640 1 -- 172.21.15.16:0/2266595119 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f167415d5e0 con 0x7f16741531b0 2023-05-22T15:28:08.211 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.203+0000 7f1672ffd640 1 --2- 172.21.15.16:0/2266595119 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16741531b0 0x7f16740afb40 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f167415c5c0 tx=0x7f166401c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6c640b6686f74a3 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:28:08.211 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.203+0000 7f16727fc640 1 -- 172.21.15.16:0/2266595119 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16640aa5f0 con 0x7f16741531b0 2023-05-22T15:28:08.211 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.203+0000 7f16727fc640 1 -- 172.21.15.16:0/2266595119 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16640aa750 con 0x7f16741531b0 2023-05-22T15:28:08.211 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.203+0000 7f16727fc640 1 -- 172.21.15.16:0/2266595119 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16640aaae0 con 0x7f16741531b0 2023-05-22T15:28:08.212 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.203+0000 7f1679a33640 1 -- 172.21.15.16:0/2266595119 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16741531b0 msgr2=0x7f16740afb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:08.213 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.203+0000 7f1679a33640 1 --2- 172.21.15.16:0/2266595119 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16741531b0 0x7f16740afb40 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f167415c5c0 tx=0x7f166401c4d0 comp rx=0 tx=0).stop 2023-05-22T15:28:08.213 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.203+0000 7f1679a33640 1 -- 172.21.15.16:0/2266595119 shutdown_connections 2023-05-22T15:28:08.213 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.203+0000 7f1679a33640 1 --2- 172.21.15.16:0/2266595119 >> v2:172.21.15.62:3300/0 conn(0x7f1674159870 0x7f16740c8ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:08.213 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.203+0000 7f1679a33640 1 --2- 172.21.15.16:0/2266595119 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f16740b0080 0x7f1674159330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:08.213 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.203+0000 7f1679a33640 1 --2- 172.21.15.16:0/2266595119 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16741531b0 0x7f16740afb40 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:08.213 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.203+0000 7f1679a33640 1 -- 172.21.15.16:0/2266595119 >> 172.21.15.16:0/2266595119 conn(0x7f1674069720 msgr2=0x7f1674069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:28:08.213 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.203+0000 7f1679a33640 1 -- 172.21.15.16:0/2266595119 shutdown_connections 2023-05-22T15:28:08.214 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.203+0000 7f1679a33640 1 -- 172.21.15.16:0/2266595119 wait complete. 2023-05-22T15:28:08.214 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.203+0000 7f1679a33640 1 Processor -- start 2023-05-22T15:28:08.214 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.207+0000 7f1679a33640 1 -- start start 2023-05-22T15:28:08.217 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.207+0000 7f1679a33640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16740b0080 0x7f16740c5ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:08.218 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.207+0000 7f1679a33640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f16741531b0 0x7f16740c7cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:08.218 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.207+0000 7f1679a33640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1674159870 0x7f1674144860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:08.218 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.207+0000 7f1679a33640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16740c6c20 con 0x7f16740b0080 2023-05-22T15:28:08.219 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.207+0000 7f1679a33640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16740c6f00 con 0x7f1674159870 2023-05-22T15:28:08.219 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.207+0000 7f1679a33640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16740c71e0 con 0x7f16741531b0 2023-05-22T15:28:08.219 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.207+0000 7f1672ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16740b0080 0x7f16740c5ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:08.219 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.207+0000 7f1672ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16740b0080 0x7f16740c5ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39926/0 (socket says 172.21.15.16:39926) 2023-05-22T15:28:08.220 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.207+0000 7f1672ffd640 1 -- 172.21.15.16:0/1334943397 learned_addr learned my addr 172.21.15.16:0/1334943397 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:28:08.220 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.207+0000 7f166a7fc640 1 --2- 172.21.15.16:0/1334943397 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f16741531b0 0x7f16740c7cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:08.220 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.207+0000 7f16737fe640 1 --2- 172.21.15.16:0/1334943397 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1674159870 0x7f1674144860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:08.220 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.207+0000 7f1672ffd640 1 -- 172.21.15.16:0/1334943397 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f16741531b0 msgr2=0x7f16740c7cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:08.220 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.207+0000 7f1672ffd640 1 --2- 172.21.15.16:0/1334943397 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f16741531b0 0x7f16740c7cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:08.220 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.207+0000 7f1672ffd640 1 -- 172.21.15.16:0/1334943397 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1674159870 msgr2=0x7f1674144860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:08.221 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.207+0000 7f1672ffd640 1 --2- 172.21.15.16:0/1334943397 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1674159870 0x7f1674144860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:08.221 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.207+0000 7f1672ffd640 1 -- 172.21.15.16:0/1334943397 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16640094d0 con 0x7f16740b0080 2023-05-22T15:28:08.221 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.207+0000 7f1672ffd640 1 --2- 172.21.15.16:0/1334943397 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16740b0080 0x7f16740c5ad0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f166401ab80 tx=0x7f166401ad10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:28:08.221 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.207+0000 7f1670ff9640 1 -- 172.21.15.16:0/1334943397 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16640b3800 con 0x7f16740b0080 2023-05-22T15:28:08.221 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.207+0000 7f1670ff9640 1 -- 172.21.15.16:0/1334943397 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16640b3980 con 0x7f16740b0080 2023-05-22T15:28:08.221 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.207+0000 7f1670ff9640 1 -- 172.21.15.16:0/1334943397 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16640b3d30 con 0x7f16740b0080 2023-05-22T15:28:08.221 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.207+0000 7f1679a33640 1 -- 172.21.15.16:0/1334943397 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f167415d0c0 con 0x7f16740b0080 2023-05-22T15:28:08.222 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.207+0000 7f1679a33640 1 -- 172.21.15.16:0/1334943397 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f167415cde0 con 0x7f16740b0080 2023-05-22T15:28:08.222 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.207+0000 7f1679a33640 1 -- 172.21.15.16:0/1334943397 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1674159c70 con 0x7f16740b0080 2023-05-22T15:28:08.222 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.207+0000 7f1670ff9640 1 -- 172.21.15.16:0/1334943397 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7f16640b2380 con 0x7f16740b0080 2023-05-22T15:28:08.222 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.211+0000 7f1670ff9640 1 --2- 172.21.15.16:0/1334943397 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f16540653b0 0x7f1654067860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:08.222 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.211+0000 7f1670ff9640 1 -- 172.21.15.16:0/1334943397 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 5530+0+0 (secure 0 0 0) 0x7f16640a7070 con 0x7f16740b0080 2023-05-22T15:28:08.222 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.211+0000 7f166a7fc640 1 --2- 172.21.15.16:0/1334943397 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f16540653b0 0x7f1654067860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:08.223 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.211+0000 7f166a7fc640 1 --2- 172.21.15.16:0/1334943397 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f16540653b0 0x7f1654067860 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f16741f5d80 tx=0x7f165c008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:28:08.223 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.215+0000 7f1670ff9640 1 -- 172.21.15.16:0/1334943397 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f16640eac60 con 0x7f16740b0080 2023-05-22T15:28:08.427 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.419+0000 7f1679a33640 1 -- 172.21.15.16:0/1334943397 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mds"} v 0) v1 -- 0x7f167404dbe0 con 0x7f16740b0080 2023-05-22T15:28:08.434 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.427+0000 7f1670ff9640 1 -- 172.21.15.16:0/1334943397 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mds"}]=0 v9) v1 ==== 90+0+0 (secure 0 0 0) 0x7f16640ee280 con 0x7f16740b0080 2023-05-22T15:28:08.436 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.427+0000 7f1679a33640 1 -- 172.21.15.16:0/1334943397 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f16540653b0 msgr2=0x7f1654067860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:08.437 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.427+0000 7f1679a33640 1 --2- 172.21.15.16:0/1334943397 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f16540653b0 0x7f1654067860 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f16741f5d80 tx=0x7f165c008040 comp rx=0 tx=0).stop 2023-05-22T15:28:08.437 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.427+0000 7f1679a33640 1 -- 172.21.15.16:0/1334943397 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16740b0080 msgr2=0x7f16740c5ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:08.437 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.427+0000 7f1679a33640 1 --2- 172.21.15.16:0/1334943397 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16740b0080 0x7f16740c5ad0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f166401ab80 tx=0x7f166401ad10 comp rx=0 tx=0).stop 2023-05-22T15:28:08.437 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.431+0000 7f1679a33640 1 -- 172.21.15.16:0/1334943397 shutdown_connections 2023-05-22T15:28:08.437 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.431+0000 7f1679a33640 1 --2- 172.21.15.16:0/1334943397 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1674159870 0x7f1674144860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:08.437 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.431+0000 7f1679a33640 1 --2- 172.21.15.16:0/1334943397 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f16540653b0 0x7f1654067860 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:08.437 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.431+0000 7f1679a33640 1 --2- 172.21.15.16:0/1334943397 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f16741531b0 0x7f16740c7cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:08.438 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.431+0000 7f1679a33640 1 --2- 172.21.15.16:0/1334943397 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16740b0080 0x7f16740c5ad0 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:08.438 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.431+0000 7f1679a33640 1 -- 172.21.15.16:0/1334943397 >> 172.21.15.16:0/1334943397 conn(0x7f1674069720 msgr2=0x7f1674071190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:28:08.438 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.431+0000 7f1679a33640 1 -- 172.21.15.16:0/1334943397 shutdown_connections 2023-05-22T15:28:08.438 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.431+0000 7f1679a33640 1 -- 172.21.15.16:0/1334943397 wait complete. 2023-05-22T15:28:08.453 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mgr 2023-05-22T15:28:08.574 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.567+0000 7eff3b468640 1 Processor -- start 2023-05-22T15:28:08.575 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.567+0000 7eff3b468640 1 -- start start 2023-05-22T15:28:08.576 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.567+0000 7eff3b468640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7eff34158a10 0x7eff3406e480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:08.576 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.567+0000 7eff3b468640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7eff34070760 0x7eff34070bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:08.576 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.567+0000 7eff3b468640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff34158de0 con 0x7eff3406c6e0 2023-05-22T15:28:08.576 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.567+0000 7eff3b468640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff3406e060 con 0x7eff34158a10 2023-05-22T15:28:08.577 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.567+0000 7eff3b468640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff340711e0 con 0x7eff34070760 2023-05-22T15:28:08.577 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.567+0000 7eff39204640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7eff34158a10 0x7eff3406e480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:08.577 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.567+0000 7eff39a05640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7eff3406c6e0 0x7eff340c9a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:50174/0 (socket says 172.21.15.16:50174) 2023-05-22T15:28:08.577 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.567+0000 7eff39a05640 1 -- 172.21.15.16:0/1974893761 learned_addr learned my addr 172.21.15.16:0/1974893761 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:28:08.577 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.567+0000 7eff38a03640 1 --2- 172.21.15.16:0/1974893761 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7eff34070760 0x7eff34070bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:08.577 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.567+0000 7eff2bfff640 1 -- 172.21.15.16:0/1974893761 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3739197075 0 0) 0x7eff34158de0 con 0x7eff3406c6e0 2023-05-22T15:28:08.578 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.567+0000 7eff2bfff640 1 -- 172.21.15.16:0/1974893761 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7eff20003530 con 0x7eff3406c6e0 2023-05-22T15:28:08.578 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.567+0000 7eff39204640 1 -- 172.21.15.16:0/1974893761 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7eff34070760 msgr2=0x7eff34070bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:28:08.578 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.567+0000 7eff39204640 1 --2- 172.21.15.16:0/1974893761 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7eff34070760 0x7eff34070bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:08.578 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.567+0000 7eff39204640 1 -- 172.21.15.16:0/1974893761 >> v1:172.21.15.16:6789/0 conn(0x7eff3406c6e0 legacy=0x7eff340c9a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:08.578 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.567+0000 7eff39204640 1 -- 172.21.15.16:0/1974893761 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff340714c0 con 0x7eff34158a10 2023-05-22T15:28:08.579 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.571+0000 7eff39204640 1 --2- 172.21.15.16:0/1974893761 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7eff34158a10 0x7eff3406e480 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7eff300097c0 tx=0x7eff3001c3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d26aba0bf01153cb server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:28:08.579 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.571+0000 7eff2bfff640 1 -- 172.21.15.16:0/1974893761 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff300ab600 con 0x7eff34158a10 2023-05-22T15:28:08.579 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.571+0000 7eff2bfff640 1 -- 172.21.15.16:0/1974893761 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff300a7050 con 0x7eff34158a10 2023-05-22T15:28:08.580 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.571+0000 7eff2bfff640 1 -- 172.21.15.16:0/1974893761 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff300ab960 con 0x7eff34158a10 2023-05-22T15:28:08.580 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.571+0000 7eff3b468640 1 -- 172.21.15.16:0/1974893761 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7eff34158a10 msgr2=0x7eff3406e480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:08.580 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.571+0000 7eff3b468640 1 --2- 172.21.15.16:0/1974893761 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7eff34158a10 0x7eff3406e480 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7eff300097c0 tx=0x7eff3001c3f0 comp rx=0 tx=0).stop 2023-05-22T15:28:08.580 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.571+0000 7eff3b468640 1 -- 172.21.15.16:0/1974893761 shutdown_connections 2023-05-22T15:28:08.581 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.571+0000 7eff3b468640 1 --2- 172.21.15.16:0/1974893761 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7eff34070760 0x7eff34070bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:08.581 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.571+0000 7eff3b468640 1 --2- 172.21.15.16:0/1974893761 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7eff34158a10 0x7eff3406e480 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:08.581 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.571+0000 7eff3b468640 1 -- 172.21.15.16:0/1974893761 >> 172.21.15.16:0/1974893761 conn(0x7eff340cc870 msgr2=0x7eff340cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:28:08.581 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.571+0000 7eff3b468640 1 -- 172.21.15.16:0/1974893761 shutdown_connections 2023-05-22T15:28:08.582 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.571+0000 7eff3b468640 1 -- 172.21.15.16:0/1974893761 wait complete. 2023-05-22T15:28:08.583 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.575+0000 7eff3b468640 1 Processor -- start 2023-05-22T15:28:08.583 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.575+0000 7eff3b468640 1 -- start start 2023-05-22T15:28:08.583 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.575+0000 7eff3b468640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7eff3406c6e0 0x7eff340c4db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:08.583 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.575+0000 7eff3b468640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7eff34070760 0x7eff341515a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:08.584 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.575+0000 7eff3b468640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7eff34158a10 0x7eff34155bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:08.584 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.575+0000 7eff3b468640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff34155450 con 0x7eff3406c6e0 2023-05-22T15:28:08.584 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.575+0000 7eff3b468640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff341547d0 con 0x7eff34158a10 2023-05-22T15:28:08.584 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.575+0000 7eff3b468640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff34154ab0 con 0x7eff34070760 2023-05-22T15:28:08.584 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.575+0000 7eff39204640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7eff3406c6e0 0x7eff340c4db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:08.584 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.575+0000 7eff39204640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7eff3406c6e0 0x7eff340c4db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39944/0 (socket says 172.21.15.16:39944) 2023-05-22T15:28:08.584 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.575+0000 7eff39204640 1 -- 172.21.15.16:0/1357632874 learned_addr learned my addr 172.21.15.16:0/1357632874 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:28:08.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.575+0000 7eff39204640 1 -- 172.21.15.16:0/1357632874 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7eff34070760 msgr2=0x7eff341515a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:08.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.575+0000 7eff39204640 1 --2- 172.21.15.16:0/1357632874 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7eff34070760 0x7eff341515a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:08.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.575+0000 7eff39204640 1 -- 172.21.15.16:0/1357632874 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7eff34158a10 msgr2=0x7eff34155bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:08.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.575+0000 7eff39204640 1 --2- 172.21.15.16:0/1357632874 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7eff34158a10 0x7eff34155bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:08.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.575+0000 7eff39204640 1 -- 172.21.15.16:0/1357632874 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff30009420 con 0x7eff3406c6e0 2023-05-22T15:28:08.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.575+0000 7eff39204640 1 --2- 172.21.15.16:0/1357632874 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7eff3406c6e0 0x7eff340c4db0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7eff3001aaa0 tx=0x7eff30019190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:28:08.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.575+0000 7eff29ffb640 1 -- 172.21.15.16:0/1357632874 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff300b4400 con 0x7eff3406c6e0 2023-05-22T15:28:08.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.575+0000 7eff29ffb640 1 -- 172.21.15.16:0/1357632874 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff300b4560 con 0x7eff3406c6e0 2023-05-22T15:28:08.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.575+0000 7eff29ffb640 1 -- 172.21.15.16:0/1357632874 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff300b48d0 con 0x7eff3406c6e0 2023-05-22T15:28:08.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.575+0000 7eff3b468640 1 -- 172.21.15.16:0/1357632874 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff3406d080 con 0x7eff3406c6e0 2023-05-22T15:28:08.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.575+0000 7eff3b468640 1 -- 172.21.15.16:0/1357632874 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff3406cef0 con 0x7eff3406c6e0 2023-05-22T15:28:08.587 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.575+0000 7eff3b468640 1 -- 172.21.15.16:0/1357632874 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff3406e9c0 con 0x7eff3406c6e0 2023-05-22T15:28:08.587 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.575+0000 7eff29ffb640 1 -- 172.21.15.16:0/1357632874 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7eff300b2040 con 0x7eff3406c6e0 2023-05-22T15:28:08.587 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.579+0000 7eff29ffb640 1 --2- 172.21.15.16:0/1357632874 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7eff18065380 0x7eff18067830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:08.588 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.579+0000 7eff29ffb640 1 -- 172.21.15.16:0/1357632874 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 5530+0+0 (secure 0 0 0) 0x7eff3011c6c0 con 0x7eff3406c6e0 2023-05-22T15:28:08.588 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.579+0000 7eff38a03640 1 --2- 172.21.15.16:0/1357632874 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7eff18065380 0x7eff18067830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:08.588 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.579+0000 7eff38a03640 1 --2- 172.21.15.16:0/1357632874 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7eff18065380 0x7eff18067830 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7eff3413cdf0 tx=0x7eff24008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:28:08.588 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.583+0000 7eff29ffb640 1 -- 172.21.15.16:0/1357632874 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7eff300ea550 con 0x7eff3406c6e0 2023-05-22T15:28:08.803 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.795+0000 7eff3b468640 1 -- 172.21.15.16:0/1357632874 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mgr"} v 0) v1 -- 0x7eff3404dbe0 con 0x7eff3406c6e0 2023-05-22T15:28:08.809 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.803+0000 7eff29ffb640 1 -- 172.21.15.16:0/1357632874 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mgr"}]=0 v10) v1 ==== 90+0+0 (secure 0 0 0) 0x7eff300b90a0 con 0x7eff3406c6e0 2023-05-22T15:28:08.812 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.803+0000 7eff3b468640 1 -- 172.21.15.16:0/1357632874 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7eff18065380 msgr2=0x7eff18067830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:08.812 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.803+0000 7eff3b468640 1 --2- 172.21.15.16:0/1357632874 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7eff18065380 0x7eff18067830 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7eff3413cdf0 tx=0x7eff24008040 comp rx=0 tx=0).stop 2023-05-22T15:28:08.813 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.803+0000 7eff3b468640 1 -- 172.21.15.16:0/1357632874 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7eff3406c6e0 msgr2=0x7eff340c4db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:08.813 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.803+0000 7eff3b468640 1 --2- 172.21.15.16:0/1357632874 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7eff3406c6e0 0x7eff340c4db0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7eff3001aaa0 tx=0x7eff30019190 comp rx=0 tx=0).stop 2023-05-22T15:28:08.813 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.803+0000 7eff3b468640 1 -- 172.21.15.16:0/1357632874 shutdown_connections 2023-05-22T15:28:08.813 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.803+0000 7eff3b468640 1 --2- 172.21.15.16:0/1357632874 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7eff34158a10 0x7eff34155bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:08.814 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.803+0000 7eff3b468640 1 --2- 172.21.15.16:0/1357632874 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7eff18065380 0x7eff18067830 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:08.814 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.803+0000 7eff3b468640 1 --2- 172.21.15.16:0/1357632874 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7eff34070760 0x7eff341515a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:08.814 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.803+0000 7eff3b468640 1 --2- 172.21.15.16:0/1357632874 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7eff3406c6e0 0x7eff340c4db0 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:08.815 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.803+0000 7eff3b468640 1 -- 172.21.15.16:0/1357632874 >> 172.21.15.16:0/1357632874 conn(0x7eff340cc870 msgr2=0x7eff34069d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:28:08.815 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.803+0000 7eff3b468640 1 -- 172.21.15.16:0/1357632874 shutdown_connections 2023-05-22T15:28:08.815 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.803+0000 7eff3b468640 1 -- 172.21.15.16:0/1357632874 wait complete. 2023-05-22T15:28:08.829 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-osd 2023-05-22T15:28:08.955 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.947+0000 7fa6ff4a6640 1 Processor -- start 2023-05-22T15:28:08.955 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.947+0000 7fa6ff4a6640 1 -- start start 2023-05-22T15:28:08.957 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.947+0000 7fa6ff4a6640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa6f815cfa0 0x7fa6f806db00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:08.957 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.947+0000 7fa6ff4a6640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fa6f806e740 0x7fa6f806ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:08.957 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.947+0000 7fa6ff4a6640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6f806e040 con 0x7fa6f80cf870 2023-05-22T15:28:08.957 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.947+0000 7fa6ff4a6640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6f80cb9f0 con 0x7fa6f806e740 2023-05-22T15:28:08.957 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.947+0000 7fa6ff4a6640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6f806f100 con 0x7fa6f815cfa0 2023-05-22T15:28:08.958 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.947+0000 7fa6fca41640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7fa6f80cf870 0x7fa6f80b1fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.16:54020/0 (socket says 172.21.15.16:54020) 2023-05-22T15:28:08.958 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.951+0000 7fa6fda43640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fa6f806e740 0x7fa6f806ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:08.958 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.951+0000 7fa6fca41640 1 -- 172.21.15.16:0/311065738 learned_addr learned my addr 172.21.15.16:0/311065738 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:28:08.958 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.951+0000 7fa6fd242640 1 --2- 172.21.15.16:0/311065738 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa6f815cfa0 0x7fa6f806db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:08.959 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.951+0000 7fa6fda43640 1 -- 172.21.15.16:0/311065738 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa6f815cfa0 msgr2=0x7fa6f806db00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:28:08.959 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.951+0000 7fa6fda43640 1 --2- 172.21.15.16:0/311065738 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa6f815cfa0 0x7fa6f806db00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:08.959 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.951+0000 7fa6fda43640 1 -- 172.21.15.16:0/311065738 >> v1:172.21.15.62:6789/0 conn(0x7fa6f80cf870 legacy=0x7fa6f80b1fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:08.960 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.951+0000 7fa6fda43640 1 -- 172.21.15.16:0/311065738 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6f806e490 con 0x7fa6f806e740 2023-05-22T15:28:08.961 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.951+0000 7fa6fda43640 1 --2- 172.21.15.16:0/311065738 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa6f806e740 0x7fa6f806ebc0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fa6f0009d90 tx=0x7fa6f001c890 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f3a2098e57be7abc server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:28:08.961 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.951+0000 7fa6effff640 1 -- 172.21.15.16:0/311065738 <== mon.1 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6f00a5950 con 0x7fa6f806e740 2023-05-22T15:28:08.961 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.951+0000 7fa6effff640 1 -- 172.21.15.16:0/311065738 <== mon.1 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6f00a5ab0 con 0x7fa6f806e740 2023-05-22T15:28:08.961 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.951+0000 7fa6effff640 1 -- 172.21.15.16:0/311065738 <== mon.1 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6f00a5e40 con 0x7fa6f806e740 2023-05-22T15:28:08.961 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.955+0000 7fa6ff4a6640 1 -- 172.21.15.16:0/311065738 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa6f806e740 msgr2=0x7fa6f806ebc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:08.962 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.955+0000 7fa6ff4a6640 1 --2- 172.21.15.16:0/311065738 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa6f806e740 0x7fa6f806ebc0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fa6f0009d90 tx=0x7fa6f001c890 comp rx=0 tx=0).stop 2023-05-22T15:28:08.962 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.955+0000 7fa6ff4a6640 1 -- 172.21.15.16:0/311065738 shutdown_connections 2023-05-22T15:28:08.962 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.955+0000 7fa6ff4a6640 1 --2- 172.21.15.16:0/311065738 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa6f806e740 0x7fa6f806ebc0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:08.962 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.955+0000 7fa6ff4a6640 1 --2- 172.21.15.16:0/311065738 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa6f815cfa0 0x7fa6f806db00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:08.962 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.955+0000 7fa6ff4a6640 1 -- 172.21.15.16:0/311065738 >> 172.21.15.16:0/311065738 conn(0x7fa6f80cc870 msgr2=0x7fa6f80cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:28:08.962 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.955+0000 7fa6ff4a6640 1 -- 172.21.15.16:0/311065738 shutdown_connections 2023-05-22T15:28:08.963 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.955+0000 7fa6ff4a6640 1 -- 172.21.15.16:0/311065738 wait complete. 2023-05-22T15:28:08.964 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.955+0000 7fa6ff4a6640 1 Processor -- start 2023-05-22T15:28:08.964 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.955+0000 7fa6ff4a6640 1 -- start start 2023-05-22T15:28:08.964 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.955+0000 7fa6ff4a6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa6f806e740 0x7fa6f8163c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:08.965 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.955+0000 7fa6ff4a6640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa6f80cf870 0x7fa6f8157870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:08.965 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.955+0000 7fa6ff4a6640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa6f815cfa0 0x7fa6f80b2fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:08.965 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.955+0000 7fa6ff4a6640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6f8164180 con 0x7fa6f806e740 2023-05-22T15:28:08.965 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.955+0000 7fa6ff4a6640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6f8162ba0 con 0x7fa6f815cfa0 2023-05-22T15:28:08.965 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.955+0000 7fa6ff4a6640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6f8162e80 con 0x7fa6f80cf870 2023-05-22T15:28:08.965 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.955+0000 7fa6fd242640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa6f806e740 0x7fa6f8163c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:08.965 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.955+0000 7fa6fd242640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa6f806e740 0x7fa6f8163c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39946/0 (socket says 172.21.15.16:39946) 2023-05-22T15:28:08.966 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.955+0000 7fa6fd242640 1 -- 172.21.15.16:0/3546985560 learned_addr learned my addr 172.21.15.16:0/3546985560 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:28:08.966 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.955+0000 7fa6fca41640 1 --2- 172.21.15.16:0/3546985560 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa6f80cf870 0x7fa6f8157870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:08.966 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.955+0000 7fa6fda43640 1 --2- 172.21.15.16:0/3546985560 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa6f815cfa0 0x7fa6f80b2fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:08.966 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.955+0000 7fa6fd242640 1 -- 172.21.15.16:0/3546985560 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa6f80cf870 msgr2=0x7fa6f8157870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:08.966 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.955+0000 7fa6fd242640 1 --2- 172.21.15.16:0/3546985560 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa6f80cf870 0x7fa6f8157870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:08.967 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.955+0000 7fa6fd242640 1 -- 172.21.15.16:0/3546985560 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa6f815cfa0 msgr2=0x7fa6f80b2fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:08.967 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.955+0000 7fa6fd242640 1 --2- 172.21.15.16:0/3546985560 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa6f815cfa0 0x7fa6f80b2fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:08.967 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.955+0000 7fa6fd242640 1 -- 172.21.15.16:0/3546985560 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6f0009a70 con 0x7fa6f806e740 2023-05-22T15:28:08.967 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.955+0000 7fa6fd242640 1 --2- 172.21.15.16:0/3546985560 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa6f806e740 0x7fa6f8163c40 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7fa6f400e870 tx=0x7fa6f400ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:28:08.967 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.955+0000 7fa6edffb640 1 -- 172.21.15.16:0/3546985560 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6f4009510 con 0x7fa6f806e740 2023-05-22T15:28:08.967 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.955+0000 7fa6edffb640 1 -- 172.21.15.16:0/3546985560 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6f400eec0 con 0x7fa6f806e740 2023-05-22T15:28:08.967 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.955+0000 7fa6ff4a6640 1 -- 172.21.15.16:0/3546985560 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6f80cb610 con 0x7fa6f806e740 2023-05-22T15:28:08.968 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.955+0000 7fa6edffb640 1 -- 172.21.15.16:0/3546985560 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6f400cb40 con 0x7fa6f806e740 2023-05-22T15:28:08.968 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.955+0000 7fa6ff4a6640 1 -- 172.21.15.16:0/3546985560 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6f80cb330 con 0x7fa6f806e740 2023-05-22T15:28:08.969 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.959+0000 7fa6ff4a6640 1 -- 172.21.15.16:0/3546985560 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6f815d370 con 0x7fa6f806e740 2023-05-22T15:28:08.969 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.959+0000 7fa6edffb640 1 -- 172.21.15.16:0/3546985560 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7fa6f400cca0 con 0x7fa6f806e740 2023-05-22T15:28:08.969 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.959+0000 7fa6edffb640 1 --2- 172.21.15.16:0/3546985560 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fa6dc065380 0x7fa6dc067830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:08.969 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.959+0000 7fa6edffb640 1 -- 172.21.15.16:0/3546985560 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 5530+0+0 (secure 0 0 0) 0x7fa6f4082770 con 0x7fa6f806e740 2023-05-22T15:28:08.969 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.959+0000 7fa6fca41640 1 --2- 172.21.15.16:0/3546985560 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fa6dc065380 0x7fa6dc067830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:08.969 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.959+0000 7fa6fca41640 1 --2- 172.21.15.16:0/3546985560 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fa6dc065380 0x7fa6dc067830 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fa6f80cffc0 tx=0x7fa6e8009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:28:08.970 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:08.963+0000 7fa6edffb640 1 -- 172.21.15.16:0/3546985560 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fa6f4050600 con 0x7fa6f806e740 2023-05-22T15:28:09.177 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:09.171+0000 7fa6ff4a6640 1 -- 172.21.15.16:0/3546985560 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-osd"} v 0) v1 -- 0x7fa6f804dbe0 con 0x7fa6f806e740 2023-05-22T15:28:09.186 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:09.179+0000 7fa6edffb640 1 -- 172.21.15.16:0/3546985560 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-osd"}]=0 v11) v1 ==== 90+0+0 (secure 0 0 0) 0x7fa6f4082070 con 0x7fa6f806e740 2023-05-22T15:28:09.188 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:09.179+0000 7fa6ff4a6640 1 -- 172.21.15.16:0/3546985560 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fa6dc065380 msgr2=0x7fa6dc067830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:09.189 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:09.179+0000 7fa6ff4a6640 1 --2- 172.21.15.16:0/3546985560 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fa6dc065380 0x7fa6dc067830 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fa6f80cffc0 tx=0x7fa6e8009040 comp rx=0 tx=0).stop 2023-05-22T15:28:09.189 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:09.179+0000 7fa6ff4a6640 1 -- 172.21.15.16:0/3546985560 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa6f806e740 msgr2=0x7fa6f8163c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:09.189 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:09.179+0000 7fa6ff4a6640 1 --2- 172.21.15.16:0/3546985560 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa6f806e740 0x7fa6f8163c40 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7fa6f400e870 tx=0x7fa6f400ec70 comp rx=0 tx=0).stop 2023-05-22T15:28:09.189 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:09.179+0000 7fa6ff4a6640 1 -- 172.21.15.16:0/3546985560 shutdown_connections 2023-05-22T15:28:09.190 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:09.179+0000 7fa6ff4a6640 1 --2- 172.21.15.16:0/3546985560 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa6f815cfa0 0x7fa6f80b2fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:09.190 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:09.179+0000 7fa6ff4a6640 1 --2- 172.21.15.16:0/3546985560 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fa6dc065380 0x7fa6dc067830 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:09.190 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:09.179+0000 7fa6ff4a6640 1 --2- 172.21.15.16:0/3546985560 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa6f80cf870 0x7fa6f8157870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:09.191 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:09.179+0000 7fa6ff4a6640 1 --2- 172.21.15.16:0/3546985560 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa6f806e740 0x7fa6f8163c40 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:09.192 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:09.179+0000 7fa6ff4a6640 1 -- 172.21.15.16:0/3546985560 >> 172.21.15.16:0/3546985560 conn(0x7fa6f80cc870 msgr2=0x7fa6f80aee30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:28:09.192 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:09.179+0000 7fa6ff4a6640 1 -- 172.21.15.16:0/3546985560 shutdown_connections 2023-05-22T15:28:09.192 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:09.183+0000 7fa6ff4a6640 1 -- 172.21.15.16:0/3546985560 wait complete. 2023-05-22T15:28:09.205 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd 2023-05-22T15:28:09.333 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:09.327+0000 7fe538011640 1 Processor -- start 2023-05-22T15:28:09.333 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:09.327+0000 7fe538011640 1 -- start start 2023-05-22T15:28:09.334 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:09.327+0000 7fe538011640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe5300d1420 0x7fe530166880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:09.335 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:09.327+0000 7fe538011640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5300ccef0 con 0x7fe5300ccab0 2023-05-22T15:28:09.335 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:09.327+0000 7fe538011640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe530160520 con 0x7fe5300b7dd0 2023-05-22T15:28:09.335 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:09.327+0000 7fe538011640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5300d3360 con 0x7fe5300d1420 2023-05-22T15:28:09.335 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:09.327+0000 7fe5355ac640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fe5300ccab0 0x7fe53015fe10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:39824/0 (socket says 172.21.15.16:39824) 2023-05-22T15:28:09.335 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:09.327+0000 7fe5355ac640 1 -- 172.21.15.16:0/1844927258 learned_addr learned my addr 172.21.15.16:0/1844927258 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:28:09.337 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:09.327+0000 7fe5365ae640 1 --2- 172.21.15.16:0/1844927258 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe5300d1420 0x7fe530166880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:09.337 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:09.327+0000 7fe534dab640 1 -- 172.21.15.16:0/1844927258 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 634648589 0 0) 0x7fe5300ccef0 con 0x7fe5300ccab0 2023-05-22T15:28:09.337 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:09.327+0000 7fe534dab640 1 -- 172.21.15.16:0/1844927258 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe51c003530 con 0x7fe5300ccab0 2023-05-22T15:28:09.337 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:09.327+0000 7fe534dab640 1 -- 172.21.15.16:0/1844927258 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2364750991 0 0) 0x7fe51c003530 con 0x7fe5300ccab0 2023-05-22T15:28:09.337 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:09.327+0000 7fe534dab640 1 -- 172.21.15.16:0/1844927258 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe5300d1420 msgr2=0x7fe530166880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:28:09.338 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:09.327+0000 7fe534dab640 1 --2- 172.21.15.16:0/1844927258 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe5300d1420 0x7fe530166880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:09.338 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:09.327+0000 7fe534dab640 1 -- 172.21.15.16:0/1844927258 >> v1:172.21.15.62:6789/0 conn(0x7fe5300b7dd0 legacy=0x7fe53015f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:09.338 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:09.327+0000 7fe534dab640 1 -- 172.21.15.16:0/1844927258 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5300cd170 con 0x7fe5300ccab0 2023-05-22T15:28:09.338 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:09.327+0000 7fe534dab640 1 -- 172.21.15.16:0/1844927258 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7fe518001a00 con 0x7fe5300ccab0 2023-05-22T15:28:09.338 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:09.327+0000 7fe534dab640 1 -- 172.21.15.16:0/1844927258 >> v1:172.21.15.16:6789/0 conn(0x7fe5300ccab0 legacy=0x7fe53015fe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:09.338 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:09.327+0000 7fe534dab640 1 --2- 172.21.15.16:0/1844927258 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe51c0039b0 0x7fe51c003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:09.338 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:09.327+0000 7fe534dab640 1 --2- 172.21.15.16:0/1844927258 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe5300d1420 0x7fe51c0047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:09.339 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:09.327+0000 7fe534dab640 1 --2- 172.21.15.16:0/1844927258 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe51c004d10 0x7fe51c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:09.339 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:09.327+0000 7fe534dab640 1 -- 172.21.15.16:0/1844927258 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5300d3360 con 0x7fe51c004d10 2023-05-22T15:28:09.339 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:09.327+0000 7fe534dab640 1 -- 172.21.15.16:0/1844927258 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe51c00ba40 con 0x7fe51c0039b0 2023-05-22T15:28:09.339 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:09.327+0000 7fe534dab640 1 -- 172.21.15.16:0/1844927258 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe51c00bbd0 con 0x7fe5300d1420 2023-05-22T15:28:09.339 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:09.327+0000 7fe5365ae640 1 --2- 172.21.15.16:0/1844927258 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe51c004d10 0x7fe51c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:09.339 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:09.327+0000 7fe535dad640 1 --2- 172.21.15.16:0/1844927258 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe51c0039b0 0x7fe51c003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:09.339 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:09.327+0000 7fe5355ac640 1 --2- 172.21.15.16:0/1844927258 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe5300d1420 0x7fe51c0047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:09.340 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:09.327+0000 7fe5365ae640 1 -- 172.21.15.16:0/1844927258 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe5300d1420 msgr2=0x7fe51c0047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:28:09.340 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:09.327+0000 7fe5365ae640 1 --2- 172.21.15.16:0/1844927258 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe5300d1420 0x7fe51c0047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:09.340 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:09.327+0000 7fe5365ae640 1 -- 172.21.15.16:0/1844927258 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe51c0039b0 msgr2=0x7fe51c003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:28:09.340 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:09.327+0000 7fe5365ae640 1 --2- 172.21.15.16:0/1844927258 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe51c0039b0 0x7fe51c003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:09.340 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:09.327+0000 7fe5365ae640 1 -- 172.21.15.16:0/1844927258 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe51c003530 con 0x7fe51c004d10 2023-05-22T15:28:09.340 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:09.327+0000 7fe535dad640 1 --2- 172.21.15.16:0/1844927258 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe51c0039b0 0x7fe51c003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T15:28:09.342 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:09.331+0000 7fe5365ae640 1 --2- 172.21.15.16:0/1844927258 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe51c004d10 0x7fe51c005180 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7fe528009ae0 tx=0x7fe52801ca50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c378af8d42c580b server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:28:09.342 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:09.331+0000 7fe534dab640 1 -- 172.21.15.16:0/1844927258 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5280a5640 con 0x7fe51c004d10 2023-05-22T15:28:09.342 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:09.331+0000 7fe534dab640 1 -- 172.21.15.16:0/1844927258 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5280a57a0 con 0x7fe51c004d10 2023-05-22T15:28:09.343 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:09.331+0000 7fe534dab640 1 -- 172.21.15.16:0/1844927258 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5280a5b50 con 0x7fe51c004d10 2023-05-22T15:28:09.343 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:09.331+0000 7fe538011640 1 -- 172.21.15.16:0/1844927258 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe51c004d10 msgr2=0x7fe51c005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:09.343 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:09.331+0000 7fe538011640 1 --2- 172.21.15.16:0/1844927258 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe51c004d10 0x7fe51c005180 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7fe528009ae0 tx=0x7fe52801ca50 comp rx=0 tx=0).stop 2023-05-22T15:28:09.343 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:09.331+0000 7fe5365ae640 1 -- 172.21.15.16:0/1844927258 reap_dead start 2023-05-22T15:28:09.344 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:09.331+0000 7fe538011640 1 -- 172.21.15.16:0/1844927258 shutdown_connections 2023-05-22T15:28:09.344 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:09.331+0000 7fe538011640 1 -- 172.21.15.16:0/1844927258 >> 172.21.15.16:0/1844927258 conn(0x7fe5300655c0 msgr2=0x7fe530070e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:28:09.344 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:09.331+0000 7fe538011640 1 -- 172.21.15.16:0/1844927258 shutdown_connections 2023-05-22T15:28:09.344 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:09.331+0000 7fe538011640 1 -- 172.21.15.16:0/1844927258 wait complete. 2023-05-22T15:28:09.344 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:09.335+0000 7fe538011640 1 Processor -- start 2023-05-22T15:28:09.344 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:09.335+0000 7fe538011640 1 -- start start 2023-05-22T15:28:09.346 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:09.335+0000 7fe538011640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe51c004d10 0x7fe5301552a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:09.346 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:09.335+0000 7fe538011640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe5300cf3e0 0x7fe53015e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:09.346 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:09.335+0000 7fe538011640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe5301f4a90 0x7fe5300c8520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:09.347 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:09.335+0000 7fe538011640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5301557e0 con 0x7fe5301f4a90 2023-05-22T15:28:09.347 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:09.335+0000 7fe538011640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5301f1350 con 0x7fe5300cf3e0 2023-05-22T15:28:09.347 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:09.335+0000 7fe538011640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5301590a0 con 0x7fe51c004d10 2023-05-22T15:28:09.347 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:09.335+0000 7fe5365ae640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe5301f4a90 0x7fe5300c8520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:09.347 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:09.335+0000 7fe5365ae640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe5301f4a90 0x7fe5300c8520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42558/0 (socket says 172.21.15.16:42558) 2023-05-22T15:28:09.347 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:09.335+0000 7fe5365ae640 1 -- 172.21.15.16:0/3546431745 learned_addr learned my addr 172.21.15.16:0/3546431745 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:28:09.348 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:09.335+0000 7fe5355ac640 1 --2- 172.21.15.16:0/3546431745 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe5300cf3e0 0x7fe53015e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:09.348 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:09.335+0000 7fe535dad640 1 --2- 172.21.15.16:0/3546431745 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe51c004d10 0x7fe5301552a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:09.348 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:09.335+0000 7fe5365ae640 1 -- 172.21.15.16:0/3546431745 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe51c004d10 msgr2=0x7fe5301552a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:09.348 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:09.335+0000 7fe5365ae640 1 --2- 172.21.15.16:0/3546431745 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe51c004d10 0x7fe5301552a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:09.348 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:09.335+0000 7fe5365ae640 1 -- 172.21.15.16:0/3546431745 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe5300cf3e0 msgr2=0x7fe53015e9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:09.348 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:09.335+0000 7fe5365ae640 1 --2- 172.21.15.16:0/3546431745 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe5300cf3e0 0x7fe53015e9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:09.348 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:09.335+0000 7fe5365ae640 1 -- 172.21.15.16:0/3546431745 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5300d2da0 con 0x7fe5301f4a90 2023-05-22T15:28:09.349 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:09.335+0000 7fe5365ae640 1 --2- 172.21.15.16:0/3546431745 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe5301f4a90 0x7fe5300c8520 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7fe528009ae0 tx=0x7fe51c005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:28:09.349 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:09.335+0000 7fe52e7fc640 1 -- 172.21.15.16:0/3546431745 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe528004220 con 0x7fe5301f4a90 2023-05-22T15:28:09.349 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:09.335+0000 7fe52e7fc640 1 -- 172.21.15.16:0/3546431745 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe528004380 con 0x7fe5301f4a90 2023-05-22T15:28:09.349 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:09.335+0000 7fe538011640 1 -- 172.21.15.16:0/3546431745 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5300d2ac0 con 0x7fe5301f4a90 2023-05-22T15:28:09.349 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:09.335+0000 7fe52e7fc640 1 -- 172.21.15.16:0/3546431745 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe528003860 con 0x7fe5301f4a90 2023-05-22T15:28:09.349 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:09.335+0000 7fe538011640 1 -- 172.21.15.16:0/3546431745 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5300d3080 con 0x7fe5301f4a90 2023-05-22T15:28:09.349 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:09.339+0000 7fe538011640 1 -- 172.21.15.16:0/3546431745 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe53015d950 con 0x7fe5301f4a90 2023-05-22T15:28:09.350 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:09.339+0000 7fe52e7fc640 1 -- 172.21.15.16:0/3546431745 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7fe5280b4070 con 0x7fe5301f4a90 2023-05-22T15:28:09.350 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:09.339+0000 7fe52e7fc640 1 --2- 172.21.15.16:0/3546431745 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fe5100653b0 0x7fe510067860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:09.350 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:09.339+0000 7fe535dad640 1 --2- 172.21.15.16:0/3546431745 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fe5100653b0 0x7fe510067860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:09.350 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:09.339+0000 7fe52e7fc640 1 -- 172.21.15.16:0/3546431745 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 5530+0+0 (secure 0 0 0) 0x7fe52811c7a0 con 0x7fe5301f4a90 2023-05-22T15:28:09.350 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:09.339+0000 7fe535dad640 1 --2- 172.21.15.16:0/3546431745 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fe5100653b0 0x7fe510067860 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fe5200054b0 tx=0x7fe520009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:28:09.351 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:09.343+0000 7fe52e7fc640 1 -- 172.21.15.16:0/3546431745 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fe5280ea590 con 0x7fe5301f4a90 2023-05-22T15:28:09.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:09.547+0000 7fe538011640 1 -- 172.21.15.16:0/3546431745 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd"} v 0) v1 -- 0x7fe53015eef0 con 0x7fe5301f4a90 2023-05-22T15:28:09.561 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:09.555+0000 7fe52e7fc640 1 -- 172.21.15.16:0/3546431745 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd"}]=0 v12) v1 ==== 90+0+0 (secure 0 0 0) 0x7fe5280b20a0 con 0x7fe5301f4a90 2023-05-22T15:28:09.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:09.555+0000 7fe538011640 1 -- 172.21.15.16:0/3546431745 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fe5100653b0 msgr2=0x7fe510067860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:09.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:09.555+0000 7fe538011640 1 --2- 172.21.15.16:0/3546431745 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fe5100653b0 0x7fe510067860 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fe5200054b0 tx=0x7fe520009040 comp rx=0 tx=0).stop 2023-05-22T15:28:09.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:09.555+0000 7fe538011640 1 -- 172.21.15.16:0/3546431745 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe5301f4a90 msgr2=0x7fe5300c8520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:09.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:09.555+0000 7fe538011640 1 --2- 172.21.15.16:0/3546431745 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe5301f4a90 0x7fe5300c8520 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7fe528009ae0 tx=0x7fe51c005800 comp rx=0 tx=0).stop 2023-05-22T15:28:09.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:09.555+0000 7fe538011640 1 -- 172.21.15.16:0/3546431745 shutdown_connections 2023-05-22T15:28:09.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:09.555+0000 7fe538011640 1 --2- 172.21.15.16:0/3546431745 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe5301f4a90 0x7fe5300c8520 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:09.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:09.555+0000 7fe538011640 1 --2- 172.21.15.16:0/3546431745 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe5300cf3e0 0x7fe53015e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:09.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:09.555+0000 7fe538011640 1 --2- 172.21.15.16:0/3546431745 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fe5100653b0 0x7fe510067860 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:09.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:09.555+0000 7fe538011640 1 --2- 172.21.15.16:0/3546431745 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe51c004d10 0x7fe5301552a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:09.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:09.555+0000 7fe538011640 1 -- 172.21.15.16:0/3546431745 >> 172.21.15.16:0/3546431745 conn(0x7fe5300655c0 msgr2=0x7fe5300b6590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:28:09.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:09.555+0000 7fe538011640 1 -- 172.21.15.16:0/3546431745 shutdown_connections 2023-05-22T15:28:09.565 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:09.555+0000 7fe538011640 1 -- 172.21.15.16:0/3546431745 wait complete. 2023-05-22T15:28:09.578 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd-mirror 2023-05-22T15:28:09.703 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:09.695+0000 7f76c22b2640 1 Processor -- start 2023-05-22T15:28:09.703 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:09.695+0000 7f76c22b2640 1 -- start start 2023-05-22T15:28:09.704 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:09.695+0000 7f76c22b2640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f76bc15c130 0x7f76bc0b2330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:09.704 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:09.695+0000 7f76c22b2640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f76bc0b2870 0x7f76bc0cf510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:09.705 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:09.695+0000 7f76c22b2640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76bc0b2cc0 con 0x7f76bc0cfa50 2023-05-22T15:28:09.705 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:09.695+0000 7f76c22b2640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76bc06c4a0 con 0x7f76bc15c130 2023-05-22T15:28:09.705 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:09.695+0000 7f76c22b2640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76bc0ce160 con 0x7f76bc0b2870 2023-05-22T15:28:09.705 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:09.695+0000 7f76bbfff640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f76bc15c130 0x7f76bc0b2330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:09.705 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:09.695+0000 7f76bbfff640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f76bc15c130 0x7f76bc0b2330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42570/0 (socket says 172.21.15.16:42570) 2023-05-22T15:28:09.705 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:09.695+0000 7f76bbfff640 1 -- 172.21.15.16:0/1608900175 learned_addr learned my addr 172.21.15.16:0/1608900175 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:28:09.705 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:09.699+0000 7f76b37fe640 1 --2- 172.21.15.16:0/1608900175 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f76bc0b2870 0x7f76bc0cf510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:09.706 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:09.699+0000 7f76bbfff640 1 -- 172.21.15.16:0/1608900175 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f76bc0b2870 msgr2=0x7f76bc0cf510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:28:09.706 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:09.699+0000 7f76bbfff640 1 --2- 172.21.15.16:0/1608900175 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f76bc0b2870 0x7f76bc0cf510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:09.706 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:09.699+0000 7f76bbfff640 1 -- 172.21.15.16:0/1608900175 >> v1:172.21.15.62:6789/0 conn(0x7f76bc0cfa50 legacy=0x7f76bc0d01b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:28:09.706 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:09.699+0000 7f76bbfff640 1 -- 172.21.15.16:0/1608900175 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76bc0cece0 con 0x7f76bc15c130 2023-05-22T15:28:09.707 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:09.699+0000 7f76bbfff640 1 --2- 172.21.15.16:0/1608900175 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f76bc15c130 0x7f76bc0b2330 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f76ac009870 tx=0x7f76ac01c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15e734dfa4e3ff39 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:28:09.707 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:09.699+0000 7f76bb7fe640 1 -- 172.21.15.16:0/1608900175 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76ac0ab600 con 0x7f76bc15c130 2023-05-22T15:28:09.707 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:09.699+0000 7f76bb7fe640 1 -- 172.21.15.16:0/1608900175 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76ac0a7050 con 0x7f76bc15c130 2023-05-22T15:28:09.707 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:09.699+0000 7f76bb7fe640 1 -- 172.21.15.16:0/1608900175 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76ac0ab980 con 0x7f76bc15c130 2023-05-22T15:28:09.708 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:09.699+0000 7f76c22b2640 1 -- 172.21.15.16:0/1608900175 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f76bc15c130 msgr2=0x7f76bc0b2330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:09.708 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:09.699+0000 7f76c22b2640 1 --2- 172.21.15.16:0/1608900175 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f76bc15c130 0x7f76bc0b2330 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f76ac009870 tx=0x7f76ac01c500 comp rx=0 tx=0).stop 2023-05-22T15:28:09.708 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:09.699+0000 7f76c22b2640 1 -- 172.21.15.16:0/1608900175 shutdown_connections 2023-05-22T15:28:09.708 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:09.699+0000 7f76c22b2640 1 --2- 172.21.15.16:0/1608900175 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f76bc0b2870 0x7f76bc0cf510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:09.708 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:09.699+0000 7f76c22b2640 1 --2- 172.21.15.16:0/1608900175 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f76bc15c130 0x7f76bc0b2330 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:09.709 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:09.699+0000 7f76c22b2640 1 -- 172.21.15.16:0/1608900175 >> 172.21.15.16:0/1608900175 conn(0x7f76bc06f5c0 msgr2=0x7f76bc06f9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:28:09.709 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:09.699+0000 7f76c22b2640 1 -- 172.21.15.16:0/1608900175 shutdown_connections 2023-05-22T15:28:09.709 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:09.699+0000 7f76c22b2640 1 -- 172.21.15.16:0/1608900175 wait complete. 2023-05-22T15:28:09.710 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:09.703+0000 7f76c22b2640 1 Processor -- start 2023-05-22T15:28:09.711 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:09.703+0000 7f76c22b2640 1 -- start start 2023-05-22T15:28:09.711 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:09.703+0000 7f76c22b2640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f76bc0b2870 0x7f76bc0c74e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:09.711 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:09.703+0000 7f76c22b2640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f76bc0cfa50 0x7f76bc154570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:09.711 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:09.703+0000 7f76c22b2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f76bc15c130 0x7f76bc154ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:09.711 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:09.703+0000 7f76c22b2640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76bc1520a0 con 0x7f76bc15c130 2023-05-22T15:28:09.711 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:09.703+0000 7f76c22b2640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76bc151490 con 0x7f76bc0b2870 2023-05-22T15:28:09.711 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:09.703+0000 7f76c22b2640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76bc151770 con 0x7f76bc0cfa50 2023-05-22T15:28:09.712 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:09.703+0000 7f76c084f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f76bc15c130 0x7f76bc154ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:09.712 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:09.703+0000 7f76c084f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f76bc15c130 0x7f76bc154ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42580/0 (socket says 172.21.15.16:42580) 2023-05-22T15:28:09.712 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:09.703+0000 7f76c084f640 1 -- 172.21.15.16:0/714024863 learned_addr learned my addr 172.21.15.16:0/714024863 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:28:09.712 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:09.703+0000 7f76bbfff640 1 --2- 172.21.15.16:0/714024863 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f76bc0b2870 0x7f76bc0c74e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:09.712 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:09.703+0000 7f76b37fe640 1 --2- 172.21.15.16:0/714024863 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f76bc0cfa50 0x7f76bc154570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:09.712 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:09.703+0000 7f76c084f640 1 -- 172.21.15.16:0/714024863 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f76bc0cfa50 msgr2=0x7f76bc154570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:09.712 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:09.703+0000 7f76c084f640 1 --2- 172.21.15.16:0/714024863 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f76bc0cfa50 0x7f76bc154570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:09.713 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:09.703+0000 7f76c084f640 1 -- 172.21.15.16:0/714024863 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f76bc0b2870 msgr2=0x7f76bc0c74e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:09.713 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:09.703+0000 7f76c084f640 1 --2- 172.21.15.16:0/714024863 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f76bc0b2870 0x7f76bc0c74e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:09.713 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:09.703+0000 7f76c084f640 1 -- 172.21.15.16:0/714024863 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76ac0094d0 con 0x7f76bc15c130 2023-05-22T15:28:09.713 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:09.703+0000 7f76bbfff640 1 --2- 172.21.15.16:0/714024863 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f76bc0b2870 0x7f76bc0c74e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:28:09.713 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:09.703+0000 7f76b37fe640 1 --2- 172.21.15.16:0/714024863 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f76bc0cfa50 0x7f76bc154570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:28:09.713 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:09.703+0000 7f76c084f640 1 --2- 172.21.15.16:0/714024863 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f76bc15c130 0x7f76bc154ab0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f76b400b970 tx=0x7f76b400be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:28:09.713 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:09.703+0000 7f76b97fa640 1 -- 172.21.15.16:0/714024863 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76b400cb40 con 0x7f76bc15c130 2023-05-22T15:28:09.713 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:09.703+0000 7f76b97fa640 1 -- 172.21.15.16:0/714024863 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76b400cca0 con 0x7f76bc15c130 2023-05-22T15:28:09.714 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:09.703+0000 7f76b97fa640 1 -- 172.21.15.16:0/714024863 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76b4014680 con 0x7f76bc15c130 2023-05-22T15:28:09.714 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:09.703+0000 7f76c22b2640 1 -- 172.21.15.16:0/714024863 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76bc0cb8c0 con 0x7f76bc15c130 2023-05-22T15:28:09.714 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:09.703+0000 7f76c22b2640 1 -- 172.21.15.16:0/714024863 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76bc0ce970 con 0x7f76bc15c130 2023-05-22T15:28:09.715 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:09.703+0000 7f76c22b2640 1 -- 172.21.15.16:0/714024863 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76bc0cc210 con 0x7f76bc15c130 2023-05-22T15:28:09.715 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:09.703+0000 7f76b97fa640 1 -- 172.21.15.16:0/714024863 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7f76b401d3b0 con 0x7f76bc15c130 2023-05-22T15:28:09.715 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:09.703+0000 7f76b97fa640 1 --2- 172.21.15.16:0/714024863 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f769c065380 0x7f769c067830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:09.715 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:09.707+0000 7f76b97fa640 1 -- 172.21.15.16:0/714024863 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 5530+0+0 (secure 0 0 0) 0x7f76b4082570 con 0x7f76bc15c130 2023-05-22T15:28:09.715 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:09.707+0000 7f76bbfff640 1 --2- 172.21.15.16:0/714024863 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f769c065380 0x7f769c067830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:09.715 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:09.707+0000 7f76bbfff640 1 --2- 172.21.15.16:0/714024863 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f769c065380 0x7f769c067830 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f76ac0ad000 tx=0x7f76ac008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:28:09.720 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:09.711+0000 7f76b97fa640 1 -- 172.21.15.16:0/714024863 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f76b40503a0 con 0x7f76bc15c130 2023-05-22T15:28:09.923 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:09.915+0000 7f76c22b2640 1 -- 172.21.15.16:0/714024863 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"} v 0) v1 -- 0x7f76bc04dbe0 con 0x7f76bc15c130 2023-05-22T15:28:09.932 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:09.923+0000 7f76b97fa640 1 -- 172.21.15.16:0/714024863 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"}]=0 v13) v1 ==== 97+0+0 (secure 0 0 0) 0x7f76b40539c0 con 0x7f76bc15c130 2023-05-22T15:28:09.935 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:09.927+0000 7f76c22b2640 1 -- 172.21.15.16:0/714024863 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f769c065380 msgr2=0x7f769c067830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:09.936 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:09.927+0000 7f76c22b2640 1 --2- 172.21.15.16:0/714024863 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f769c065380 0x7f769c067830 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f76ac0ad000 tx=0x7f76ac008040 comp rx=0 tx=0).stop 2023-05-22T15:28:09.936 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:09.927+0000 7f76c22b2640 1 -- 172.21.15.16:0/714024863 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f76bc15c130 msgr2=0x7f76bc154ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:09.936 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:09.927+0000 7f76c22b2640 1 --2- 172.21.15.16:0/714024863 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f76bc15c130 0x7f76bc154ab0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f76b400b970 tx=0x7f76b400be30 comp rx=0 tx=0).stop 2023-05-22T15:28:09.936 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:09.927+0000 7f76c22b2640 1 -- 172.21.15.16:0/714024863 shutdown_connections 2023-05-22T15:28:09.937 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:09.927+0000 7f76c22b2640 1 --2- 172.21.15.16:0/714024863 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f76bc15c130 0x7f76bc154ab0 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:09.937 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:09.927+0000 7f76c22b2640 1 --2- 172.21.15.16:0/714024863 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f769c065380 0x7f769c067830 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:09.938 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:09.927+0000 7f76c22b2640 1 --2- 172.21.15.16:0/714024863 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f76bc0cfa50 0x7f76bc154570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:09.938 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:09.927+0000 7f76c22b2640 1 --2- 172.21.15.16:0/714024863 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f76bc0b2870 0x7f76bc0c74e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:09.938 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:09.927+0000 7f76c22b2640 1 -- 172.21.15.16:0/714024863 >> 172.21.15.16:0/714024863 conn(0x7f76bc06f5c0 msgr2=0x7f76bc0b0d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:28:09.938 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:09.927+0000 7f76c22b2640 1 -- 172.21.15.16:0/714024863 shutdown_connections 2023-05-22T15:28:09.939 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:09.927+0000 7f76c22b2640 1 -- 172.21.15.16:0/714024863 wait complete. 2023-05-22T15:28:09.952 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rgw 2023-05-22T15:28:10.079 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.071+0000 7f5abeba0640 1 Processor -- start 2023-05-22T15:28:10.079 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.071+0000 7f5abeba0640 1 -- start start 2023-05-22T15:28:10.081 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.071+0000 7f5abeba0640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5ab8158660 0x7f5ab80cdfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:10.082 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.071+0000 7f5abeba0640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f5ab80cce20 0x7f5ab80ce4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:10.082 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.071+0000 7f5abeba0640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ab80c92b0 con 0x7f5ab8158290 2023-05-22T15:28:10.082 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.071+0000 7f5abeba0640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ab80cc060 con 0x7f5ab80cce20 2023-05-22T15:28:10.082 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.071+0000 7f5abeba0640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ab80cc340 con 0x7f5ab8158660 2023-05-22T15:28:10.082 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.071+0000 7f5abc93c640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f5ab8158290 0x7f5ab80c8880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:39832/0 (socket says 172.21.15.16:39832) 2023-05-22T15:28:10.082 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.071+0000 7f5abc93c640 1 -- 172.21.15.16:0/1177484419 learned_addr learned my addr 172.21.15.16:0/1177484419 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:28:10.082 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.075+0000 7f5abd13d640 1 --2- 172.21.15.16:0/1177484419 >> v2:172.21.15.62:3300/0 conn(0x7f5ab80cce20 0x7f5ab80ce4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:10.083 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.075+0000 7f5ab7fff640 1 --2- 172.21.15.16:0/1177484419 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5ab8158660 0x7f5ab80cdfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:10.083 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.075+0000 7f5ab77fe640 1 -- 172.21.15.16:0/1177484419 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3445316244 0 0) 0x7f5ab80c92b0 con 0x7f5ab8158290 2023-05-22T15:28:10.083 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.075+0000 7f5ab77fe640 1 -- 172.21.15.16:0/1177484419 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5aa8003450 con 0x7f5ab8158290 2023-05-22T15:28:10.083 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.075+0000 7f5ab77fe640 1 -- 172.21.15.16:0/1177484419 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3219015048 0 0) 0x7f5aa8003450 con 0x7f5ab8158290 2023-05-22T15:28:10.084 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.075+0000 7f5ab77fe640 1 -- 172.21.15.16:0/1177484419 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5ab8158660 msgr2=0x7f5ab80cdfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:28:10.084 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.075+0000 7f5ab77fe640 1 --2- 172.21.15.16:0/1177484419 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5ab8158660 0x7f5ab80cdfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:10.084 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.075+0000 7f5ab77fe640 1 -- 172.21.15.16:0/1177484419 >> v2:172.21.15.62:3300/0 conn(0x7f5ab80cce20 msgr2=0x7f5ab80ce4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:28:10.084 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.075+0000 7f5ab77fe640 1 --2- 172.21.15.16:0/1177484419 >> v2:172.21.15.62:3300/0 conn(0x7f5ab80cce20 0x7f5ab80ce4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:10.084 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.075+0000 7f5ab77fe640 1 -- 172.21.15.16:0/1177484419 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ab80cc620 con 0x7f5ab8158290 2023-05-22T15:28:10.085 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.075+0000 7f5ab77fe640 1 -- 172.21.15.16:0/1177484419 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7f5aac001680 con 0x7f5ab8158290 2023-05-22T15:28:10.085 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.075+0000 7f5ab77fe640 1 -- 172.21.15.16:0/1177484419 >> v1:172.21.15.16:6789/0 conn(0x7f5ab8158290 legacy=0x7f5ab80c8880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:10.085 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.075+0000 7f5ab77fe640 1 --2- 172.21.15.16:0/1177484419 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5ab8158660 0x7f5aa8003a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:10.085 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.075+0000 7f5ab77fe640 1 --2- 172.21.15.16:0/1177484419 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5aa8003ff0 0x7f5aa800a870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:10.085 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.075+0000 7f5ab77fe640 1 --2- 172.21.15.16:0/1177484419 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5aa800ae00 0x7f5aa800d260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:10.085 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.075+0000 7f5ab77fe640 1 -- 172.21.15.16:0/1177484419 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ab80cc060 con 0x7f5aa800ae00 2023-05-22T15:28:10.085 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.075+0000 7f5ab77fe640 1 -- 172.21.15.16:0/1177484419 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ab80cc340 con 0x7f5aa8003ff0 2023-05-22T15:28:10.086 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.075+0000 7f5ab77fe640 1 -- 172.21.15.16:0/1177484419 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5aa800db20 con 0x7f5ab8158660 2023-05-22T15:28:10.086 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.075+0000 7f5abd13d640 1 --2- 172.21.15.16:0/1177484419 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5aa800ae00 0x7f5aa800d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:10.086 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.075+0000 7f5ab7fff640 1 --2- 172.21.15.16:0/1177484419 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5aa8003ff0 0x7f5aa800a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:10.086 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.075+0000 7f5abd13d640 1 -- 172.21.15.16:0/1177484419 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5ab8158660 msgr2=0x7f5aa8003a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:28:10.086 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.075+0000 7f5abd13d640 1 --2- 172.21.15.16:0/1177484419 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5ab8158660 0x7f5aa8003a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:10.086 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.075+0000 7f5abd13d640 1 -- 172.21.15.16:0/1177484419 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5aa8003ff0 msgr2=0x7f5aa800a870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:28:10.086 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.075+0000 7f5abd13d640 1 --2- 172.21.15.16:0/1177484419 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5aa8003ff0 0x7f5aa800a870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:10.086 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.075+0000 7f5abd13d640 1 -- 172.21.15.16:0/1177484419 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5aa8003450 con 0x7f5aa800ae00 2023-05-22T15:28:10.091 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.075+0000 7f5abd13d640 1 --2- 172.21.15.16:0/1177484419 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5aa800ae00 0x7f5aa800d260 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f5ab0009f90 tx=0x7f5ab001cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ecf1b1cc2bfc9c4 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:28:10.091 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.075+0000 7f5ab77fe640 1 -- 172.21.15.16:0/1177484419 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ab00a5c70 con 0x7f5aa800ae00 2023-05-22T15:28:10.091 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.075+0000 7f5ab77fe640 1 -- 172.21.15.16:0/1177484419 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ab000f040 con 0x7f5aa800ae00 2023-05-22T15:28:10.092 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.075+0000 7f5ab77fe640 1 -- 172.21.15.16:0/1177484419 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ab00ad440 con 0x7f5aa800ae00 2023-05-22T15:28:10.092 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.075+0000 7f5abeba0640 1 -- 172.21.15.16:0/1177484419 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5aa800ae00 msgr2=0x7f5aa800d260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:10.092 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.075+0000 7f5abeba0640 1 --2- 172.21.15.16:0/1177484419 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5aa800ae00 0x7f5aa800d260 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f5ab0009f90 tx=0x7f5ab001cc60 comp rx=0 tx=0).stop 2023-05-22T15:28:10.092 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.075+0000 7f5abd13d640 1 -- 172.21.15.16:0/1177484419 reap_dead start 2023-05-22T15:28:10.092 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.079+0000 7f5abeba0640 1 -- 172.21.15.16:0/1177484419 shutdown_connections 2023-05-22T15:28:10.092 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.079+0000 7f5abeba0640 1 -- 172.21.15.16:0/1177484419 >> 172.21.15.16:0/1177484419 conn(0x7f5ab8069720 msgr2=0x7f5ab8069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:28:10.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.079+0000 7f5abeba0640 1 -- 172.21.15.16:0/1177484419 shutdown_connections 2023-05-22T15:28:10.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.079+0000 7f5abeba0640 1 -- 172.21.15.16:0/1177484419 wait complete. 2023-05-22T15:28:10.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.079+0000 7f5abeba0640 1 Processor -- start 2023-05-22T15:28:10.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.079+0000 7f5abeba0640 1 -- start start 2023-05-22T15:28:10.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.079+0000 7f5abeba0640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5ab8070f00 0x7f5ab815a590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:10.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.079+0000 7f5abeba0640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5ab80712d0 0x7f5ab8158a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:10.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.079+0000 7f5abeba0640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5ab806f530 0x7f5ab806f9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:10.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.079+0000 7f5abeba0640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ab806ff10 con 0x7f5ab80712d0 2023-05-22T15:28:10.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.079+0000 7f5abeba0640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ab815b7a0 con 0x7f5ab8070f00 2023-05-22T15:28:10.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.079+0000 7f5abeba0640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ab815ab90 con 0x7f5ab806f530 2023-05-22T15:28:10.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.079+0000 7f5ab7fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5ab80712d0 0x7f5ab8158a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:10.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.079+0000 7f5ab7fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5ab80712d0 0x7f5ab8158a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42598/0 (socket says 172.21.15.16:42598) 2023-05-22T15:28:10.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.079+0000 7f5ab7fff640 1 -- 172.21.15.16:0/3144016300 learned_addr learned my addr 172.21.15.16:0/3144016300 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:28:10.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.079+0000 7f5abc93c640 1 --2- 172.21.15.16:0/3144016300 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5ab8070f00 0x7f5ab815a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:10.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.079+0000 7f5abd13d640 1 --2- 172.21.15.16:0/3144016300 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5ab806f530 0x7f5ab806f9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:10.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.079+0000 7f5ab7fff640 1 -- 172.21.15.16:0/3144016300 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5ab806f530 msgr2=0x7f5ab806f9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:10.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.079+0000 7f5ab7fff640 1 --2- 172.21.15.16:0/3144016300 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5ab806f530 0x7f5ab806f9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:10.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.079+0000 7f5ab7fff640 1 -- 172.21.15.16:0/3144016300 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5ab8070f00 msgr2=0x7f5ab815a590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:10.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.079+0000 7f5ab7fff640 1 --2- 172.21.15.16:0/3144016300 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5ab8070f00 0x7f5ab815a590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:10.096 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.079+0000 7f5ab7fff640 1 -- 172.21.15.16:0/3144016300 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ab80c8f90 con 0x7f5ab80712d0 2023-05-22T15:28:10.096 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.079+0000 7f5ab7fff640 1 --2- 172.21.15.16:0/3144016300 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5ab80712d0 0x7f5ab8158a30 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f5aa40116a0 tx=0x7f5aa4011b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:28:10.096 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.079+0000 7f5ab57fa640 1 -- 172.21.15.16:0/3144016300 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5aa400f9c0 con 0x7f5ab80712d0 2023-05-22T15:28:10.097 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.079+0000 7f5ab57fa640 1 -- 172.21.15.16:0/3144016300 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5aa400fb20 con 0x7f5ab80712d0 2023-05-22T15:28:10.097 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.079+0000 7f5abeba0640 1 -- 172.21.15.16:0/3144016300 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ab80cc900 con 0x7f5ab80712d0 2023-05-22T15:28:10.097 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.079+0000 7f5ab57fa640 1 -- 172.21.15.16:0/3144016300 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5aa400fe90 con 0x7f5ab80712d0 2023-05-22T15:28:10.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.079+0000 7f5abeba0640 1 -- 172.21.15.16:0/3144016300 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ab815cd10 con 0x7f5ab80712d0 2023-05-22T15:28:10.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.083+0000 7f5abeba0640 1 -- 172.21.15.16:0/3144016300 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ab804dbe0 con 0x7f5ab80712d0 2023-05-22T15:28:10.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.083+0000 7f5ab57fa640 1 -- 172.21.15.16:0/3144016300 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7f5aa4015460 con 0x7f5ab80712d0 2023-05-22T15:28:10.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.083+0000 7f5ab57fa640 1 --2- 172.21.15.16:0/3144016300 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f5a9c0653b0 0x7f5a9c067860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:10.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.083+0000 7f5ab57fa640 1 -- 172.21.15.16:0/3144016300 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 5530+0+0 (secure 0 0 0) 0x7f5aa4082e00 con 0x7f5ab80712d0 2023-05-22T15:28:10.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.083+0000 7f5abc93c640 1 --2- 172.21.15.16:0/3144016300 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f5a9c0653b0 0x7f5a9c067860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:10.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.083+0000 7f5abc93c640 1 --2- 172.21.15.16:0/3144016300 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f5a9c0653b0 0x7f5a9c067860 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f5aac0017b0 tx=0x7f5aac001680 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:28:10.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.087+0000 7f5ab57fa640 1 -- 172.21.15.16:0/3144016300 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f5aa4050d10 con 0x7f5ab80712d0 2023-05-22T15:28:10.301 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.295+0000 7f5abeba0640 1 -- 172.21.15.16:0/3144016300 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rgw"} v 0) v1 -- 0x7f5ab80ce810 con 0x7f5ab80712d0 2023-05-22T15:28:10.308 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.303+0000 7f5ab57fa640 1 -- 172.21.15.16:0/3144016300 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rgw"}]=0 v14) v1 ==== 90+0+0 (secure 0 0 0) 0x7f5aa4054330 con 0x7f5ab80712d0 2023-05-22T15:28:10.311 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.303+0000 7f5abeba0640 1 -- 172.21.15.16:0/3144016300 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f5a9c0653b0 msgr2=0x7f5a9c067860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:10.311 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.303+0000 7f5abeba0640 1 --2- 172.21.15.16:0/3144016300 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f5a9c0653b0 0x7f5a9c067860 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f5aac0017b0 tx=0x7f5aac001680 comp rx=0 tx=0).stop 2023-05-22T15:28:10.311 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.303+0000 7f5abeba0640 1 -- 172.21.15.16:0/3144016300 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5ab80712d0 msgr2=0x7f5ab8158a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:10.311 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.303+0000 7f5abeba0640 1 --2- 172.21.15.16:0/3144016300 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5ab80712d0 0x7f5ab8158a30 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f5aa40116a0 tx=0x7f5aa4011b60 comp rx=0 tx=0).stop 2023-05-22T15:28:10.311 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.303+0000 7f5abeba0640 1 -- 172.21.15.16:0/3144016300 shutdown_connections 2023-05-22T15:28:10.312 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.303+0000 7f5abeba0640 1 --2- 172.21.15.16:0/3144016300 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f5a9c0653b0 0x7f5a9c067860 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:10.312 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.303+0000 7f5abeba0640 1 --2- 172.21.15.16:0/3144016300 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5ab806f530 0x7f5ab806f9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:10.312 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.303+0000 7f5abeba0640 1 --2- 172.21.15.16:0/3144016300 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5ab80712d0 0x7f5ab8158a30 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:10.312 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.303+0000 7f5abeba0640 1 --2- 172.21.15.16:0/3144016300 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5ab8070f00 0x7f5ab815a590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:10.312 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.303+0000 7f5abeba0640 1 -- 172.21.15.16:0/3144016300 >> 172.21.15.16:0/3144016300 conn(0x7f5ab8069720 msgr2=0x7f5ab8156030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:28:10.312 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.303+0000 7f5abeba0640 1 -- 172.21.15.16:0/3144016300 shutdown_connections 2023-05-22T15:28:10.312 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.303+0000 7f5abeba0640 1 -- 172.21.15.16:0/3144016300 wait complete. 2023-05-22T15:28:10.328 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-05-22T15:28:10.445 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.439+0000 7f5744d2b640 1 Processor -- start 2023-05-22T15:28:10.446 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.439+0000 7f5744d2b640 1 -- start start 2023-05-22T15:28:10.447 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.439+0000 7f5744d2b640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f57400cbf80 0x7f574006b210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:10.447 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.439+0000 7f5744d2b640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f57400b22e0 0x7f574006bec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:10.447 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.439+0000 7f5744d2b640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f574006c400 0x7f574015f810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:10.447 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.439+0000 7f5744d2b640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f574006c880 con 0x7f574006c400 2023-05-22T15:28:10.448 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.439+0000 7f5744d2b640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f574006db40 con 0x7f57400b22e0 2023-05-22T15:28:10.448 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.439+0000 7f5744d2b640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57400cf3f0 con 0x7f57400cbf80 2023-05-22T15:28:10.448 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.439+0000 7f573ed9d640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f574006c400 0x7f574015f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:10.448 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.439+0000 7f573ed9d640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f574006c400 0x7f574015f810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42604/0 (socket says 172.21.15.16:42604) 2023-05-22T15:28:10.448 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.439+0000 7f573ed9d640 1 -- 172.21.15.16:0/3047181383 learned_addr learned my addr 172.21.15.16:0/3047181383 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:28:10.448 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.439+0000 7f573dd9b640 1 --2- 172.21.15.16:0/3047181383 >> v2:172.21.15.62:3300/0 conn(0x7f57400b22e0 0x7f574006bec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:10.448 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.439+0000 7f573e59c640 1 --2- 172.21.15.16:0/3047181383 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f57400cbf80 0x7f574006b210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:10.449 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.439+0000 7f573ed9d640 1 -- 172.21.15.16:0/3047181383 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f57400cbf80 msgr2=0x7f574006b210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:28:10.449 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.439+0000 7f573ed9d640 1 --2- 172.21.15.16:0/3047181383 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f57400cbf80 0x7f574006b210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:10.449 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.439+0000 7f573ed9d640 1 -- 172.21.15.16:0/3047181383 >> v2:172.21.15.62:3300/0 conn(0x7f57400b22e0 msgr2=0x7f574006bec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:28:10.449 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.439+0000 7f573ed9d640 1 --2- 172.21.15.16:0/3047181383 >> v2:172.21.15.62:3300/0 conn(0x7f57400b22e0 0x7f574006bec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:10.449 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.439+0000 7f573ed9d640 1 -- 172.21.15.16:0/3047181383 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f574015c080 con 0x7f574006c400 2023-05-22T15:28:10.451 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.443+0000 7f573ed9d640 1 --2- 172.21.15.16:0/3047181383 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f574006c400 0x7f574015f810 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f574006e850 tx=0x7f573801c6e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a69c21530e31d315 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:28:10.451 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.443+0000 7f573d59a640 1 -- 172.21.15.16:0/3047181383 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57380a5560 con 0x7f574006c400 2023-05-22T15:28:10.453 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.443+0000 7f573d59a640 1 -- 172.21.15.16:0/3047181383 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57380a56c0 con 0x7f574006c400 2023-05-22T15:28:10.453 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.443+0000 7f573d59a640 1 -- 172.21.15.16:0/3047181383 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57380a5a50 con 0x7f574006c400 2023-05-22T15:28:10.453 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.443+0000 7f5744d2b640 1 -- 172.21.15.16:0/3047181383 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f574006c400 msgr2=0x7f574015f810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:10.453 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.443+0000 7f5744d2b640 1 --2- 172.21.15.16:0/3047181383 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f574006c400 0x7f574015f810 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f574006e850 tx=0x7f573801c6e0 comp rx=0 tx=0).stop 2023-05-22T15:28:10.453 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.443+0000 7f5744d2b640 1 -- 172.21.15.16:0/3047181383 shutdown_connections 2023-05-22T15:28:10.454 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.443+0000 7f5744d2b640 1 --2- 172.21.15.16:0/3047181383 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f574006c400 0x7f574015f810 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:10.454 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.443+0000 7f5744d2b640 1 --2- 172.21.15.16:0/3047181383 >> v2:172.21.15.62:3300/0 conn(0x7f57400b22e0 0x7f574006bec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:10.454 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.443+0000 7f5744d2b640 1 --2- 172.21.15.16:0/3047181383 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f57400cbf80 0x7f574006b210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:10.454 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.443+0000 7f5744d2b640 1 -- 172.21.15.16:0/3047181383 >> 172.21.15.16:0/3047181383 conn(0x7f574006f560 msgr2=0x7f574006f960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:28:10.454 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.443+0000 7f5744d2b640 1 -- 172.21.15.16:0/3047181383 shutdown_connections 2023-05-22T15:28:10.454 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.443+0000 7f5744d2b640 1 -- 172.21.15.16:0/3047181383 wait complete. 2023-05-22T15:28:10.454 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.443+0000 7f5744d2b640 1 Processor -- start 2023-05-22T15:28:10.455 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.443+0000 7f5744d2b640 1 -- start start 2023-05-22T15:28:10.456 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.443+0000 7f5744d2b640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f574006c400 0x7f5740158230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:10.457 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.443+0000 7f5744d2b640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f57400b22e0 0x7f5740158770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:10.457 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.443+0000 7f5744d2b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f57400cbf80 0x7f574015e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:10.457 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.443+0000 7f5744d2b640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5740146f80 con 0x7f57400cbf80 2023-05-22T15:28:10.457 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.443+0000 7f5744d2b640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5740147260 con 0x7f57400b22e0 2023-05-22T15:28:10.457 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.443+0000 7f5744d2b640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5740147540 con 0x7f574006c400 2023-05-22T15:28:10.458 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.443+0000 7f573ed9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f57400cbf80 0x7f574015e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:10.458 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.447+0000 7f573ed9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f57400cbf80 0x7f574015e7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42614/0 (socket says 172.21.15.16:42614) 2023-05-22T15:28:10.458 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.447+0000 7f573ed9d640 1 -- 172.21.15.16:0/2431743325 learned_addr learned my addr 172.21.15.16:0/2431743325 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:28:10.458 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.447+0000 7f573e59c640 1 --2- 172.21.15.16:0/2431743325 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f574006c400 0x7f5740158230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:10.458 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.447+0000 7f573dd9b640 1 --2- 172.21.15.16:0/2431743325 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f57400b22e0 0x7f5740158770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:10.458 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.447+0000 7f573ed9d640 1 -- 172.21.15.16:0/2431743325 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f574006c400 msgr2=0x7f5740158230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:10.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.447+0000 7f573ed9d640 1 --2- 172.21.15.16:0/2431743325 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f574006c400 0x7f5740158230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:10.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.447+0000 7f573ed9d640 1 -- 172.21.15.16:0/2431743325 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f57400b22e0 msgr2=0x7f5740158770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:10.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.447+0000 7f573ed9d640 1 --2- 172.21.15.16:0/2431743325 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f57400b22e0 0x7f5740158770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:10.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.447+0000 7f573ed9d640 1 -- 172.21.15.16:0/2431743325 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5738009b30 con 0x7f57400cbf80 2023-05-22T15:28:10.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.447+0000 7f573dd9b640 1 --2- 172.21.15.16:0/2431743325 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f57400b22e0 0x7f5740158770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T15:28:10.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.447+0000 7f573ed9d640 1 --2- 172.21.15.16:0/2431743325 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f57400cbf80 0x7f574015e7d0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f57380af040 tx=0x7f5738019480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:28:10.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.447+0000 7f572effd640 1 -- 172.21.15.16:0/2431743325 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57380b8670 con 0x7f57400cbf80 2023-05-22T15:28:10.460 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.447+0000 7f572effd640 1 -- 172.21.15.16:0/2431743325 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57380b87d0 con 0x7f57400cbf80 2023-05-22T15:28:10.460 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.447+0000 7f5744d2b640 1 -- 172.21.15.16:0/2431743325 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f574006b750 con 0x7f57400cbf80 2023-05-22T15:28:10.460 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.447+0000 7f572effd640 1 -- 172.21.15.16:0/2431743325 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57380b8b80 con 0x7f57400cbf80 2023-05-22T15:28:10.460 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.447+0000 7f5744d2b640 1 -- 172.21.15.16:0/2431743325 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f574006e380 con 0x7f57400cbf80 2023-05-22T15:28:10.460 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.447+0000 7f5744d2b640 1 -- 172.21.15.16:0/2431743325 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57400cc350 con 0x7f57400cbf80 2023-05-22T15:28:10.460 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.447+0000 7f572effd640 1 -- 172.21.15.16:0/2431743325 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7f57380b8ce0 con 0x7f57400cbf80 2023-05-22T15:28:10.460 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.447+0000 7f572effd640 1 --2- 172.21.15.16:0/2431743325 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f57200653b0 0x7f5720067860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:10.460 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.447+0000 7f572effd640 1 -- 172.21.15.16:0/2431743325 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 5530+0+0 (secure 0 0 0) 0x7f573811d950 con 0x7f57400cbf80 2023-05-22T15:28:10.461 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.447+0000 7f573e59c640 1 --2- 172.21.15.16:0/2431743325 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f57200653b0 0x7f5720067860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:10.461 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.447+0000 7f573e59c640 1 --2- 172.21.15.16:0/2431743325 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f57200653b0 0x7f5720067860 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f5730006fd0 tx=0x7f5730008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:28:10.461 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.451+0000 7f572effd640 1 -- 172.21.15.16:0/2431743325 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f57380eb8a0 con 0x7f57400cbf80 2023-05-22T15:28:10.613 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.607+0000 7f5744d2b640 1 -- 172.21.15.16:0/2431743325 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f574004dbe0 con 0x7f57400cbf80 2023-05-22T15:28:10.614 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.607+0000 7f572effd640 1 -- 172.21.15.16:0/2431743325 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v38) v1 ==== 74+0+14328 (secure 0 0 0) 0x7f5738009510 con 0x7f57400cbf80 2023-05-22T15:28:10.615 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:28:10.615 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":38,"fsid":"b2719834-7847-4d98-90a6-f65d57f85c49","created":"2023-05-22T15:26:25.259314+0000","modified":"2023-05-22T15:28:07.670985+0000","last_up_change":"2023-05-22T15:26:36.448524+0000","last_in_change":"2023-05-22T15:26:27.675668+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-05-22T15:26:38.019781+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-05-22T15:26:43.570514+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9e7ff854-b25f-4cbb-a5c2-1799d9c53d2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":37021},{"type":"v1","addr":"172.21.15.16:6825","nonce":37021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":37021},{"type":"v1","addr":"172.21.15.16:6827","nonce":37021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":37021},{"type":"v1","addr":"172.21.15.16:6831","nonce":37021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":37021},{"type":"v1","addr":"172.21.15.16:6829","nonce":37021}]},"public_addr":"172.21.15.16:6825/37021","cluster_addr":"172.21.15.16:6827/37021","heartbeat_back_addr":"172.21.15.16:6831/37021","heartbeat_front_addr":"172.21.15.16:6829/37021","state":["exists","up"]},{"osd":1,"uuid":"4702fbc9-a927-4565-aca5-41dc69cc8264","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":37008},{"type":"v1","addr":"172.21.15.16:6801","nonce":37008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":37008},{"type":"v1","addr":"172.21.15.16:6803","nonce":37008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":37008},{"type":"v1","addr":"172.21.15.16:6807","nonce":37008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":37008},{"type":"v1","addr":"172.21.15.16:6805","nonce":37008}]},"public_addr":"172.21.15.16:6801/37008","cluster_addr":"172.21.15.16:6803/37008","heartbeat_back_addr":"172.21.15.16:6807/37008","heartbeat_front_addr":"172.21.15.16:6805/37008","state":["exists","up"]},{"osd":2,"uuid":"1e7bba46-4eba-47ef-bb81-5e2d371638ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":37009},{"type":"v1","addr":"172.21.15.16:6809","nonce":37009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":37009},{"type":"v1","addr":"172.21.15.16:6811","nonce":37009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":37009},{"type":"v1","addr":"172.21.15.16:6815","nonce":37009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":37009},{"type":"v1","addr":"172.21.15.16:6813","nonce":37009}]},"public_addr":"172.21.15.16:6809/37009","cluster_addr":"172.21.15.16:6811/37009","heartbeat_back_addr":"172.21.15.16:6815/37009","heartbeat_front_addr":"172.21.15.16:6813/37009","state":["exists","up"]},{"osd":3,"uuid":"383cacc0-5a88-4904-990e-498ca59ef48a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":37020},{"type":"v1","addr":"172.21.15.16:6817","nonce":37020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":37020},{"type":"v1","addr":"172.21.15.16:6819","nonce":37020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":37020},{"type":"v1","addr":"172.21.15.16:6823","nonce":37020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":37020},{"type":"v1","addr":"172.21.15.16:6821","nonce":37020}]},"public_addr":"172.21.15.16:6817/37020","cluster_addr":"172.21.15.16:6819/37020","heartbeat_back_addr":"172.21.15.16:6823/37020","heartbeat_front_addr":"172.21.15.16:6821/37020","state":["exists","up"]},{"osd":4,"uuid":"970500ef-95b7-42c1-b080-eed4fae7013d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":37353},{"type":"v1","addr":"172.21.15.62:6817","nonce":37353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6819","nonce":37353},{"type":"v1","addr":"172.21.15.62:6822","nonce":37353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37353},{"type":"v1","addr":"172.21.15.62:6831","nonce":37353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37353},{"type":"v1","addr":"172.21.15.62:6826","nonce":37353}]},"public_addr":"172.21.15.62:6817/37353","cluster_addr":"172.21.15.62:6822/37353","heartbeat_back_addr":"172.21.15.62:6831/37353","heartbeat_front_addr":"172.21.15.62:6826/37353","state":["exists","up"]},{"osd":5,"uuid":"022d67db-d4d8-445b-b1b9-8b446a00f5c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":37352},{"type":"v1","addr":"172.21.15.62:6801","nonce":37352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":37352},{"type":"v1","addr":"172.21.15.62:6803","nonce":37352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":37352},{"type":"v1","addr":"172.21.15.62:6807","nonce":37352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":37352},{"type":"v1","addr":"172.21.15.62:6805","nonce":37352}]},"public_addr":"172.21.15.62:6801/37352","cluster_addr":"172.21.15.62:6803/37352","heartbeat_back_addr":"172.21.15.62:6807/37352","heartbeat_front_addr":"172.21.15.62:6805/37352","state":["exists","up"]},{"osd":6,"uuid":"42bd372b-6688-4a1b-8407-e16e8c266e33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":37354},{"type":"v1","addr":"172.21.15.62:6820","nonce":37354}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":37354},{"type":"v1","addr":"172.21.15.62:6823","nonce":37354}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6829","nonce":37354},{"type":"v1","addr":"172.21.15.62:6830","nonce":37354}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6825","nonce":37354},{"type":"v1","addr":"172.21.15.62:6827","nonce":37354}]},"public_addr":"172.21.15.62:6820/37354","cluster_addr":"172.21.15.62:6823/37354","heartbeat_back_addr":"172.21.15.62:6830/37354","heartbeat_front_addr":"172.21.15.62:6827/37354","state":["exists","up"]},{"osd":7,"uuid":"ac70b2a3-f41b-4bf2-b227-7f2cbb55426d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37355},{"type":"v1","addr":"172.21.15.62:6809","nonce":37355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37355},{"type":"v1","addr":"172.21.15.62:6811","nonce":37355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":37355},{"type":"v1","addr":"172.21.15.62:6815","nonce":37355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":37355},{"type":"v1","addr":"172.21.15.62:6813","nonce":37355}]},"public_addr":"172.21.15.62:6809/37355","cluster_addr":"172.21.15.62:6811/37355","heartbeat_back_addr":"172.21.15.62:6815/37355","heartbeat_front_addr":"172.21.15.62:6813/37355","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.368896+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.107273+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.276888+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.287502+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.124791+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.251937+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.255679+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.232797+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-05-22T15:28:10.616 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.607+0000 7f5744d2b640 1 -- 172.21.15.16:0/2431743325 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f57200653b0 msgr2=0x7f5720067860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:10.617 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.607+0000 7f5744d2b640 1 --2- 172.21.15.16:0/2431743325 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f57200653b0 0x7f5720067860 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f5730006fd0 tx=0x7f5730008040 comp rx=0 tx=0).stop 2023-05-22T15:28:10.617 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.607+0000 7f5744d2b640 1 -- 172.21.15.16:0/2431743325 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f57400cbf80 msgr2=0x7f574015e7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:10.617 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.607+0000 7f5744d2b640 1 --2- 172.21.15.16:0/2431743325 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f57400cbf80 0x7f574015e7d0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f57380af040 tx=0x7f5738019480 comp rx=0 tx=0).stop 2023-05-22T15:28:10.617 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.607+0000 7f5744d2b640 1 -- 172.21.15.16:0/2431743325 shutdown_connections 2023-05-22T15:28:10.617 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.607+0000 7f5744d2b640 1 --2- 172.21.15.16:0/2431743325 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f57400cbf80 0x7f574015e7d0 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:10.617 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.607+0000 7f5744d2b640 1 --2- 172.21.15.16:0/2431743325 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f57400b22e0 0x7f5740158770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:10.618 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.607+0000 7f5744d2b640 1 --2- 172.21.15.16:0/2431743325 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f57200653b0 0x7f5720067860 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:10.618 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.607+0000 7f5744d2b640 1 --2- 172.21.15.16:0/2431743325 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f574006c400 0x7f5740158230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:10.618 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.607+0000 7f5744d2b640 1 -- 172.21.15.16:0/2431743325 >> 172.21.15.16:0/2431743325 conn(0x7f574006f560 msgr2=0x7f57400b5ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:28:10.618 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.607+0000 7f5744d2b640 1 -- 172.21.15.16:0/2431743325 shutdown_connections 2023-05-22T15:28:10.618 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.607+0000 7f5744d2b640 1 -- 172.21.15.16:0/2431743325 wait complete. 2023-05-22T15:28:10.634 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-05-22T15:26:38.019781+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-05-22T15:26:43.570514+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-05-22T15:28:10.635 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-05-22T15:28:10.765 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.759+0000 7f8892402640 1 Processor -- start 2023-05-22T15:28:10.765 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.759+0000 7f8892402640 1 -- start start 2023-05-22T15:28:10.766 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.759+0000 7f8892402640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f888c070c70 0x7f888c071530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:10.766 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.759+0000 7f8892402640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f888c15c580 0x7f888c06fba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:10.766 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.759+0000 7f8892402640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f888c15d520 con 0x7f888c06b6c0 2023-05-22T15:28:10.767 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.759+0000 7f8892402640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f888c0c8b10 con 0x7f888c15c580 2023-05-22T15:28:10.767 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.759+0000 7f8892402640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f888c15c9d0 con 0x7f888c070c70 2023-05-22T15:28:10.767 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.759+0000 7f889099f640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f888c06b6c0 0x7f888c0cca30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:39848/0 (socket says 172.21.15.16:39848) 2023-05-22T15:28:10.767 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.759+0000 7f889099f640 1 -- 172.21.15.16:0/434410955 learned_addr learned my addr 172.21.15.16:0/434410955 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:28:10.767 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.759+0000 7f888b7fe640 1 --2- 172.21.15.16:0/434410955 >> v2:172.21.15.62:3300/0 conn(0x7f888c15c580 0x7f888c06fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:10.767 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.759+0000 7f888bfff640 1 --2- 172.21.15.16:0/434410955 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f888c070c70 0x7f888c071530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:10.769 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.759+0000 7f888affd640 1 -- 172.21.15.16:0/434410955 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3969780875 0 0) 0x7f888c15d520 con 0x7f888c06b6c0 2023-05-22T15:28:10.769 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.759+0000 7f888affd640 1 -- 172.21.15.16:0/434410955 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8874003530 con 0x7f888c06b6c0 2023-05-22T15:28:10.769 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.759+0000 7f888affd640 1 -- 172.21.15.16:0/434410955 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3404124750 0 0) 0x7f8874003530 con 0x7f888c06b6c0 2023-05-22T15:28:10.770 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.759+0000 7f888affd640 1 -- 172.21.15.16:0/434410955 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f888c070c70 msgr2=0x7f888c071530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:28:10.770 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.759+0000 7f888affd640 1 --2- 172.21.15.16:0/434410955 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f888c070c70 0x7f888c071530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:10.770 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.759+0000 7f888affd640 1 -- 172.21.15.16:0/434410955 >> v2:172.21.15.62:3300/0 conn(0x7f888c15c580 msgr2=0x7f888c06fba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:28:10.770 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.759+0000 7f888affd640 1 --2- 172.21.15.16:0/434410955 >> v2:172.21.15.62:3300/0 conn(0x7f888c15c580 0x7f888c06fba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:10.770 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.759+0000 7f888affd640 1 -- 172.21.15.16:0/434410955 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f888c0700e0 con 0x7f888c06b6c0 2023-05-22T15:28:10.770 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.759+0000 7f888affd640 1 -- 172.21.15.16:0/434410955 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7f8884001680 con 0x7f888c06b6c0 2023-05-22T15:28:10.770 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.759+0000 7f888affd640 1 -- 172.21.15.16:0/434410955 >> v1:172.21.15.16:6789/0 conn(0x7f888c06b6c0 legacy=0x7f888c0cca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:10.771 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.759+0000 7f888affd640 1 --2- 172.21.15.16:0/434410955 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f888c070c70 0x7f8874003980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:10.771 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.759+0000 7f888affd640 1 --2- 172.21.15.16:0/434410955 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8874003f00 0x7f88740047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:10.771 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.759+0000 7f888affd640 1 --2- 172.21.15.16:0/434410955 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8874004d10 0x7f8874005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:10.771 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.759+0000 7f888affd640 1 -- 172.21.15.16:0/434410955 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f888c0c8b10 con 0x7f8874003f00 2023-05-22T15:28:10.771 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.759+0000 7f888affd640 1 -- 172.21.15.16:0/434410955 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f888c15c9d0 con 0x7f8874004d10 2023-05-22T15:28:10.771 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.759+0000 7f888affd640 1 -- 172.21.15.16:0/434410955 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f887400dbd0 con 0x7f888c070c70 2023-05-22T15:28:10.771 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.759+0000 7f888b7fe640 1 --2- 172.21.15.16:0/434410955 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8874003f00 0x7f88740047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:10.771 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.759+0000 7f888bfff640 1 --2- 172.21.15.16:0/434410955 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f888c070c70 0x7f8874003980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:10.772 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.759+0000 7f889099f640 1 --2- 172.21.15.16:0/434410955 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8874004d10 0x7f8874005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:10.772 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.759+0000 7f888b7fe640 1 -- 172.21.15.16:0/434410955 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f888c070c70 msgr2=0x7f8874003980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:28:10.772 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.759+0000 7f888b7fe640 1 --2- 172.21.15.16:0/434410955 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f888c070c70 0x7f8874003980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:10.772 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.759+0000 7f888b7fe640 1 -- 172.21.15.16:0/434410955 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8874004d10 msgr2=0x7f8874005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:28:10.772 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.759+0000 7f888b7fe640 1 --2- 172.21.15.16:0/434410955 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8874004d10 0x7f8874005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:10.772 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.759+0000 7f888b7fe640 1 -- 172.21.15.16:0/434410955 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8874003530 con 0x7f8874003f00 2023-05-22T15:28:10.772 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.759+0000 7f889099f640 1 --2- 172.21.15.16:0/434410955 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8874004d10 0x7f8874005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:28:10.775 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.763+0000 7f888b7fe640 1 --2- 172.21.15.16:0/434410955 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8874003f00 0x7f88740047d0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f8880009fc0 tx=0x7f888001cc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=882775578c7608ff server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:28:10.775 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.763+0000 7f888affd640 1 -- 172.21.15.16:0/434410955 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88800a5a70 con 0x7f8874003f00 2023-05-22T15:28:10.775 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.763+0000 7f888affd640 1 -- 172.21.15.16:0/434410955 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f888000f040 con 0x7f8874003f00 2023-05-22T15:28:10.775 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.763+0000 7f888affd640 1 -- 172.21.15.16:0/434410955 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88800a5e10 con 0x7f8874003f00 2023-05-22T15:28:10.775 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.763+0000 7f8892402640 1 -- 172.21.15.16:0/434410955 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8874003f00 msgr2=0x7f88740047d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:10.775 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.763+0000 7f8892402640 1 --2- 172.21.15.16:0/434410955 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8874003f00 0x7f88740047d0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f8880009fc0 tx=0x7f888001cc10 comp rx=0 tx=0).stop 2023-05-22T15:28:10.776 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.763+0000 7f889099f640 1 -- 172.21.15.16:0/434410955 reap_dead start 2023-05-22T15:28:10.776 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.763+0000 7f8892402640 1 -- 172.21.15.16:0/434410955 shutdown_connections 2023-05-22T15:28:10.776 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.763+0000 7f8892402640 1 -- 172.21.15.16:0/434410955 >> 172.21.15.16:0/434410955 conn(0x7f888c069720 msgr2=0x7f888c069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:28:10.776 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.763+0000 7f8892402640 1 -- 172.21.15.16:0/434410955 shutdown_connections 2023-05-22T15:28:10.776 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.763+0000 7f8892402640 1 -- 172.21.15.16:0/434410955 wait complete. 2023-05-22T15:28:10.776 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.763+0000 7f8892402640 1 Processor -- start 2023-05-22T15:28:10.776 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.763+0000 7f8892402640 1 -- start start 2023-05-22T15:28:10.777 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.763+0000 7f8892402640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f888c14e830 0x7f888c0c29e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:10.777 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.763+0000 7f8892402640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f888c1f58c0 0x7f888c159080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:10.777 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.763+0000 7f8892402640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f888c157230 0x7f888c1576a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:10.777 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.767+0000 7f8892402640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f888c15a130 con 0x7f888c1f58c0 2023-05-22T15:28:10.777 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.767+0000 7f8892402640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f888c157be0 con 0x7f888c157230 2023-05-22T15:28:10.777 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.767+0000 7f8892402640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f888c15a660 con 0x7f888c14e830 2023-05-22T15:28:10.778 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.767+0000 7f888b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f888c1f58c0 0x7f888c159080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:10.778 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.767+0000 7f888b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f888c1f58c0 0x7f888c159080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42628/0 (socket says 172.21.15.16:42628) 2023-05-22T15:28:10.778 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.767+0000 7f888b7fe640 1 -- 172.21.15.16:0/3440326836 learned_addr learned my addr 172.21.15.16:0/3440326836 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:28:10.778 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.767+0000 7f888bfff640 1 --2- 172.21.15.16:0/3440326836 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f888c14e830 0x7f888c0c29e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:10.778 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.767+0000 7f889099f640 1 --2- 172.21.15.16:0/3440326836 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f888c157230 0x7f888c1576a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:10.778 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.767+0000 7f888b7fe640 1 -- 172.21.15.16:0/3440326836 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f888c14e830 msgr2=0x7f888c0c29e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:10.778 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.767+0000 7f888b7fe640 1 --2- 172.21.15.16:0/3440326836 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f888c14e830 0x7f888c0c29e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:10.778 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.767+0000 7f888b7fe640 1 -- 172.21.15.16:0/3440326836 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f888c157230 msgr2=0x7f888c1576a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:10.779 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.767+0000 7f888b7fe640 1 --2- 172.21.15.16:0/3440326836 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f888c157230 0x7f888c1576a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:10.779 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.767+0000 7f888b7fe640 1 -- 172.21.15.16:0/3440326836 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f888c0cc510 con 0x7f888c1f58c0 2023-05-22T15:28:10.779 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.767+0000 7f889099f640 1 --2- 172.21.15.16:0/3440326836 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f888c157230 0x7f888c1576a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:28:10.779 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.767+0000 7f888b7fe640 1 --2- 172.21.15.16:0/3440326836 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f888c1f58c0 0x7f888c159080 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f8880098690 tx=0x7f8880009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:28:10.779 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.767+0000 7f8888ff9640 1 -- 172.21.15.16:0/3440326836 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8880003ba0 con 0x7f888c1f58c0 2023-05-22T15:28:10.779 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.767+0000 7f8888ff9640 1 -- 172.21.15.16:0/3440326836 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8880003d00 con 0x7f888c1f58c0 2023-05-22T15:28:10.779 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.767+0000 7f8892402640 1 -- 172.21.15.16:0/3440326836 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f888c15d240 con 0x7f888c1f58c0 2023-05-22T15:28:10.780 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.767+0000 7f8888ff9640 1 -- 172.21.15.16:0/3440326836 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8880004220 con 0x7f888c1f58c0 2023-05-22T15:28:10.780 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.767+0000 7f8892402640 1 -- 172.21.15.16:0/3440326836 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f888c071040 con 0x7f888c1f58c0 2023-05-22T15:28:10.780 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.767+0000 7f8892402640 1 -- 172.21.15.16:0/3440326836 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f888c15a4c0 con 0x7f888c1f58c0 2023-05-22T15:28:10.780 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.767+0000 7f8888ff9640 1 -- 172.21.15.16:0/3440326836 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7f88800b4040 con 0x7f888c1f58c0 2023-05-22T15:28:10.781 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.767+0000 7f8888ff9640 1 --2- 172.21.15.16:0/3440326836 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f88780653b0 0x7f8878067860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:10.781 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.767+0000 7f8888ff9640 1 -- 172.21.15.16:0/3440326836 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 5530+0+0 (secure 0 0 0) 0x7f888011cc60 con 0x7f888c1f58c0 2023-05-22T15:28:10.781 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.767+0000 7f888bfff640 1 --2- 172.21.15.16:0/3440326836 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f88780653b0 0x7f8878067860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:10.781 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.767+0000 7f888bfff640 1 --2- 172.21.15.16:0/3440326836 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f88780653b0 0x7f8878067860 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f888c1f2870 tx=0x7f887c008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:28:10.781 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.771+0000 7f8888ff9640 1 -- 172.21.15.16:0/3440326836 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f888c15a4c0 con 0x7f888c1f58c0 2023-05-22T15:28:10.935 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.927+0000 7f8892402640 1 -- 172.21.15.16:0/3440326836 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f888c15a4c0 con 0x7f888c1f58c0 2023-05-22T15:28:10.936 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.927+0000 7f8888ff9640 1 -- 172.21.15.16:0/3440326836 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v38) v1 ==== 93+0+10 (secure 0 0 0) 0x7f888c15a4c0 con 0x7f888c1f58c0 2023-05-22T15:28:10.936 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2023-05-22T15:28:10.937 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.927+0000 7f8892402640 1 -- 172.21.15.16:0/3440326836 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f88780653b0 msgr2=0x7f8878067860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:10.938 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.927+0000 7f8892402640 1 --2- 172.21.15.16:0/3440326836 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f88780653b0 0x7f8878067860 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f888c1f2870 tx=0x7f887c008040 comp rx=0 tx=0).stop 2023-05-22T15:28:10.938 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.931+0000 7f8892402640 1 -- 172.21.15.16:0/3440326836 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f888c1f58c0 msgr2=0x7f888c159080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:10.938 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.931+0000 7f8892402640 1 --2- 172.21.15.16:0/3440326836 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f888c1f58c0 0x7f888c159080 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f8880098690 tx=0x7f8880009750 comp rx=0 tx=0).stop 2023-05-22T15:28:10.938 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.931+0000 7f8892402640 1 -- 172.21.15.16:0/3440326836 shutdown_connections 2023-05-22T15:28:10.938 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.931+0000 7f8892402640 1 --2- 172.21.15.16:0/3440326836 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f888c157230 0x7f888c1576a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:10.938 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.931+0000 7f8892402640 1 --2- 172.21.15.16:0/3440326836 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f888c1f58c0 0x7f888c159080 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:10.939 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.931+0000 7f8892402640 1 --2- 172.21.15.16:0/3440326836 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f88780653b0 0x7f8878067860 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:10.939 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.931+0000 7f8892402640 1 --2- 172.21.15.16:0/3440326836 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f888c14e830 0x7f888c0c29e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:10.939 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.931+0000 7f8892402640 1 -- 172.21.15.16:0/3440326836 >> 172.21.15.16:0/3440326836 conn(0x7f888c069720 msgr2=0x7f888c0ae480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:28:10.939 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.931+0000 7f8892402640 1 -- 172.21.15.16:0/3440326836 shutdown_connections 2023-05-22T15:28:10.939 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:10.931+0000 7f8892402640 1 -- 172.21.15.16:0/3440326836 wait complete. 2023-05-22T15:28:10.958 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-05-22T15:28:11.081 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.075+0000 7fcf2454b640 1 Processor -- start 2023-05-22T15:28:11.081 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.075+0000 7fcf2454b640 1 -- start start 2023-05-22T15:28:11.083 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.075+0000 7fcf2454b640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fcf1c159970 0x7fcf1c159d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:11.084 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.075+0000 7fcf2454b640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf1c070c30 con 0x7fcf1c15db00 2023-05-22T15:28:11.084 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.075+0000 7fcf2454b640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf1c070e60 con 0x7fcf1c15d680 2023-05-22T15:28:11.084 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.075+0000 7fcf2454b640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf1c0cb970 con 0x7fcf1c159970 2023-05-22T15:28:11.084 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.075+0000 7fcf22ae8640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fcf1c15db00 0x7fcf1c06b6c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:39854/0 (socket says 172.21.15.16:39854) 2023-05-22T15:28:11.084 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.075+0000 7fcf22ae8640 1 -- 172.21.15.16:0/3860181056 learned_addr learned my addr 172.21.15.16:0/3860181056 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:28:11.084 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.075+0000 7fcf222e7640 1 --2- 172.21.15.16:0/3860181056 >> v2:172.21.15.62:3300/0 conn(0x7fcf1c159970 0x7fcf1c159d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:11.085 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.075+0000 7fcf212e5640 1 -- 172.21.15.16:0/3860181056 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1278511023 0 0) 0x7fcf1c070c30 con 0x7fcf1c15db00 2023-05-22T15:28:11.085 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.075+0000 7fcf212e5640 1 -- 172.21.15.16:0/3860181056 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcf08003530 con 0x7fcf1c15db00 2023-05-22T15:28:11.085 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.075+0000 7fcf212e5640 1 -- 172.21.15.16:0/3860181056 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3337681245 0 0) 0x7fcf08003530 con 0x7fcf1c15db00 2023-05-22T15:28:11.085 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.075+0000 7fcf212e5640 1 -- 172.21.15.16:0/3860181056 >> v2:172.21.15.62:3300/0 conn(0x7fcf1c159970 msgr2=0x7fcf1c159d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:28:11.085 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.075+0000 7fcf212e5640 1 --2- 172.21.15.16:0/3860181056 >> v2:172.21.15.62:3300/0 conn(0x7fcf1c159970 0x7fcf1c159d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:11.085 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.075+0000 7fcf212e5640 1 -- 172.21.15.16:0/3860181056 >> v1:172.21.15.62:6789/0 conn(0x7fcf1c15d680 legacy=0x7fcf1c0cb260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:11.085 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.075+0000 7fcf212e5640 1 -- 172.21.15.16:0/3860181056 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf1c071140 con 0x7fcf1c15db00 2023-05-22T15:28:11.085 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.075+0000 7fcf212e5640 1 -- 172.21.15.16:0/3860181056 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7fcf14001680 con 0x7fcf1c15db00 2023-05-22T15:28:11.086 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.075+0000 7fcf212e5640 1 -- 172.21.15.16:0/3860181056 >> v1:172.21.15.16:6789/0 conn(0x7fcf1c15db00 legacy=0x7fcf1c06b6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:11.086 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.075+0000 7fcf212e5640 1 --2- 172.21.15.16:0/3860181056 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fcf08003980 0x7fcf08003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:11.086 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.075+0000 7fcf212e5640 1 --2- 172.21.15.16:0/3860181056 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcf080047d0 0x7fcf08004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:11.086 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.075+0000 7fcf212e5640 1 --2- 172.21.15.16:0/3860181056 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcf08005180 0x7fcf0800b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:11.086 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.075+0000 7fcf212e5640 1 -- 172.21.15.16:0/3860181056 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf1c0cb970 con 0x7fcf08005180 2023-05-22T15:28:11.086 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.075+0000 7fcf212e5640 1 -- 172.21.15.16:0/3860181056 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf0800c090 con 0x7fcf080047d0 2023-05-22T15:28:11.086 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.075+0000 7fcf212e5640 1 -- 172.21.15.16:0/3860181056 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf0800c250 con 0x7fcf08003980 2023-05-22T15:28:11.087 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.075+0000 7fcf22ae8640 1 --2- 172.21.15.16:0/3860181056 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcf08005180 0x7fcf0800b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:11.087 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.075+0000 7fcf222e7640 1 --2- 172.21.15.16:0/3860181056 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fcf08003980 0x7fcf08003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:11.087 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.075+0000 7fcf21ae6640 1 --2- 172.21.15.16:0/3860181056 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcf080047d0 0x7fcf08004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:11.087 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.075+0000 7fcf22ae8640 1 -- 172.21.15.16:0/3860181056 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fcf08003980 msgr2=0x7fcf08003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:28:11.087 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.075+0000 7fcf22ae8640 1 --2- 172.21.15.16:0/3860181056 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fcf08003980 0x7fcf08003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:11.087 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.075+0000 7fcf22ae8640 1 -- 172.21.15.16:0/3860181056 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcf080047d0 msgr2=0x7fcf08004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:28:11.087 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.075+0000 7fcf22ae8640 1 --2- 172.21.15.16:0/3860181056 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcf080047d0 0x7fcf08004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:11.087 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.075+0000 7fcf22ae8640 1 -- 172.21.15.16:0/3860181056 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf08003530 con 0x7fcf08005180 2023-05-22T15:28:11.089 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.079+0000 7fcf22ae8640 1 -- 172.21.15.16:0/3860181056 reap_dead start 2023-05-22T15:28:11.089 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.079+0000 7fcf22ae8640 1 --2- 172.21.15.16:0/3860181056 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcf08005180 0x7fcf0800b990 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7fcf1400c240 tx=0x7fcf1401f3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e589b310d072f1da server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:28:11.090 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.079+0000 7fcf212e5640 1 -- 172.21.15.16:0/3860181056 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf14005ae0 con 0x7fcf08005180 2023-05-22T15:28:11.090 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.079+0000 7fcf212e5640 1 -- 172.21.15.16:0/3860181056 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf140ab070 con 0x7fcf08005180 2023-05-22T15:28:11.090 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.079+0000 7fcf212e5640 1 -- 172.21.15.16:0/3860181056 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf14007070 con 0x7fcf08005180 2023-05-22T15:28:11.090 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.079+0000 7fcf2454b640 1 -- 172.21.15.16:0/3860181056 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcf08005180 msgr2=0x7fcf0800b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:11.090 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.079+0000 7fcf2454b640 1 --2- 172.21.15.16:0/3860181056 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcf08005180 0x7fcf0800b990 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7fcf1400c240 tx=0x7fcf1401f3b0 comp rx=0 tx=0).stop 2023-05-22T15:28:11.091 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.079+0000 7fcf2454b640 1 -- 172.21.15.16:0/3860181056 shutdown_connections 2023-05-22T15:28:11.091 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.079+0000 7fcf2454b640 1 --2- 172.21.15.16:0/3860181056 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcf08005180 0x7fcf0800b990 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:11.091 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.079+0000 7fcf2454b640 1 -- 172.21.15.16:0/3860181056 >> 172.21.15.16:0/3860181056 conn(0x7fcf1c069720 msgr2=0x7fcf1c069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:28:11.091 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.079+0000 7fcf2454b640 1 -- 172.21.15.16:0/3860181056 shutdown_connections 2023-05-22T15:28:11.091 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.079+0000 7fcf2454b640 1 -- 172.21.15.16:0/3860181056 wait complete. 2023-05-22T15:28:11.091 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.079+0000 7fcf2454b640 1 Processor -- start 2023-05-22T15:28:11.091 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.079+0000 7fcf2454b640 1 -- start start 2023-05-22T15:28:11.091 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.079+0000 7fcf2454b640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fcf08005180 0x7fcf1c14ee30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:11.092 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.079+0000 7fcf2454b640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcf1c0adab0 0x7fcf1c0af900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:11.092 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.079+0000 7fcf2454b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcf1c1f00f0 0x7fcf1c06b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:11.092 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.079+0000 7fcf2454b640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf1c0b3d30 con 0x7fcf1c1f00f0 2023-05-22T15:28:11.092 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.079+0000 7fcf2454b640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf1c0af660 con 0x7fcf1c0adab0 2023-05-22T15:28:11.092 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.079+0000 7fcf2454b640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf1c0adf50 con 0x7fcf08005180 2023-05-22T15:28:11.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.079+0000 7fcf22ae8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcf1c1f00f0 0x7fcf1c06b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:11.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.079+0000 7fcf22ae8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcf1c1f00f0 0x7fcf1c06b6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42640/0 (socket says 172.21.15.16:42640) 2023-05-22T15:28:11.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.079+0000 7fcf22ae8640 1 -- 172.21.15.16:0/560077368 learned_addr learned my addr 172.21.15.16:0/560077368 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:28:11.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.083+0000 7fcf222e7640 1 --2- 172.21.15.16:0/560077368 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fcf08005180 0x7fcf1c14ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:11.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.083+0000 7fcf22ae8640 1 -- 172.21.15.16:0/560077368 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fcf08005180 msgr2=0x7fcf1c14ee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:11.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.083+0000 7fcf21ae6640 1 --2- 172.21.15.16:0/560077368 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcf1c0adab0 0x7fcf1c0af900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:11.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.083+0000 7fcf22ae8640 1 --2- 172.21.15.16:0/560077368 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fcf08005180 0x7fcf1c14ee30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:11.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.083+0000 7fcf22ae8640 1 -- 172.21.15.16:0/560077368 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcf1c0adab0 msgr2=0x7fcf1c0af900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:11.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.083+0000 7fcf22ae8640 1 --2- 172.21.15.16:0/560077368 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcf1c0adab0 0x7fcf1c0af900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:11.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.083+0000 7fcf22ae8640 1 -- 172.21.15.16:0/560077368 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf1c071700 con 0x7fcf1c1f00f0 2023-05-22T15:28:11.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.083+0000 7fcf22ae8640 1 --2- 172.21.15.16:0/560077368 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcf1c1f00f0 0x7fcf1c06b6c0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7fcf1401f1e0 tx=0x7fcf08005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:28:11.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.083+0000 7fcf12ffd640 1 -- 172.21.15.16:0/560077368 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf140b5070 con 0x7fcf1c1f00f0 2023-05-22T15:28:11.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.083+0000 7fcf12ffd640 1 -- 172.21.15.16:0/560077368 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf14001500 con 0x7fcf1c1f00f0 2023-05-22T15:28:11.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.083+0000 7fcf12ffd640 1 -- 172.21.15.16:0/560077368 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf140049f0 con 0x7fcf1c1f00f0 2023-05-22T15:28:11.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.083+0000 7fcf2454b640 1 -- 172.21.15.16:0/560077368 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf1c071420 con 0x7fcf1c1f00f0 2023-05-22T15:28:11.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.083+0000 7fcf2454b640 1 -- 172.21.15.16:0/560077368 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf1c0cac90 con 0x7fcf1c1f00f0 2023-05-22T15:28:11.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.083+0000 7fcf2454b640 1 -- 172.21.15.16:0/560077368 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf1c04dbe0 con 0x7fcf1c1f00f0 2023-05-22T15:28:11.096 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.083+0000 7fcf12ffd640 1 -- 172.21.15.16:0/560077368 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7fcf140ad030 con 0x7fcf1c1f00f0 2023-05-22T15:28:11.096 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.083+0000 7fcf12ffd640 1 --2- 172.21.15.16:0/560077368 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fcf00065380 0x7fcf00067830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:11.096 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.083+0000 7fcf222e7640 1 --2- 172.21.15.16:0/560077368 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fcf00065380 0x7fcf00067830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:11.096 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.083+0000 7fcf12ffd640 1 -- 172.21.15.16:0/560077368 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 5530+0+0 (secure 0 0 0) 0x7fcf1411ef30 con 0x7fcf1c1f00f0 2023-05-22T15:28:11.096 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.083+0000 7fcf222e7640 1 --2- 172.21.15.16:0/560077368 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fcf00065380 0x7fcf00067830 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fcf1c1f2480 tx=0x7fcf1800b040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:28:11.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.087+0000 7fcf12ffd640 1 -- 172.21.15.16:0/560077368 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fcf140ecd20 con 0x7fcf1c1f00f0 2023-05-22T15:28:11.250 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.243+0000 7fcf2454b640 1 -- 172.21.15.16:0/560077368 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fcf1c0c44a0 con 0x7fcf1c1f00f0 2023-05-22T15:28:11.251 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.243+0000 7fcf12ffd640 1 -- 172.21.15.16:0/560077368 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v38) v1 ==== 92+0+10 (secure 0 0 0) 0x7fcf140f0340 con 0x7fcf1c1f00f0 2023-05-22T15:28:11.251 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 8 2023-05-22T15:28:11.252 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.243+0000 7fcf2454b640 1 -- 172.21.15.16:0/560077368 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fcf00065380 msgr2=0x7fcf00067830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:11.252 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.243+0000 7fcf2454b640 1 --2- 172.21.15.16:0/560077368 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fcf00065380 0x7fcf00067830 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fcf1c1f2480 tx=0x7fcf1800b040 comp rx=0 tx=0).stop 2023-05-22T15:28:11.252 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.243+0000 7fcf2454b640 1 -- 172.21.15.16:0/560077368 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcf1c1f00f0 msgr2=0x7fcf1c06b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:11.252 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.243+0000 7fcf2454b640 1 --2- 172.21.15.16:0/560077368 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcf1c1f00f0 0x7fcf1c06b6c0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7fcf1401f1e0 tx=0x7fcf08005800 comp rx=0 tx=0).stop 2023-05-22T15:28:11.254 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.243+0000 7fcf2454b640 1 -- 172.21.15.16:0/560077368 shutdown_connections 2023-05-22T15:28:11.254 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.243+0000 7fcf2454b640 1 --2- 172.21.15.16:0/560077368 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcf1c1f00f0 0x7fcf1c06b6c0 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:11.254 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.243+0000 7fcf2454b640 1 --2- 172.21.15.16:0/560077368 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcf1c0adab0 0x7fcf1c0af900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:11.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.243+0000 7fcf2454b640 1 --2- 172.21.15.16:0/560077368 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fcf00065380 0x7fcf00067830 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:11.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.243+0000 7fcf2454b640 1 --2- 172.21.15.16:0/560077368 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fcf08005180 0x7fcf1c14ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:11.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.243+0000 7fcf2454b640 1 -- 172.21.15.16:0/560077368 >> 172.21.15.16:0/560077368 conn(0x7fcf1c069720 msgr2=0x7fcf1c15c9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:28:11.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.243+0000 7fcf2454b640 1 -- 172.21.15.16:0/560077368 shutdown_connections 2023-05-22T15:28:11.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.243+0000 7fcf2454b640 1 -- 172.21.15.16:0/560077368 wait complete. 2023-05-22T15:28:11.270 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-05-22T15:28:11.393 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.387+0000 7f6853838640 1 Processor -- start 2023-05-22T15:28:11.393 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.387+0000 7f6853838640 1 -- start start 2023-05-22T15:28:11.394 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.387+0000 7f6853838640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f684c159cd0 0x7f684c15a0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:11.394 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.387+0000 7f6853838640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f684c0b2880 0x7f684c0cbfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:11.395 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.387+0000 7f6853838640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f684c15a5e0 0x7f684c0cc520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:11.395 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.387+0000 7f6853838640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f684c0cf9a0 con 0x7f684c0b2880 2023-05-22T15:28:11.395 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.387+0000 7f6853838640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f684c0cca60 con 0x7f684c15a5e0 2023-05-22T15:28:11.395 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.387+0000 7f6853838640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f684c15aba0 con 0x7f684c159cd0 2023-05-22T15:28:11.396 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.387+0000 7f6850dd3640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f684c0b2880 0x7f684c0cbfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:11.396 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.387+0000 7f6850dd3640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f684c0b2880 0x7f684c0cbfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42650/0 (socket says 172.21.15.16:42650) 2023-05-22T15:28:11.396 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.387+0000 7f6850dd3640 1 -- 172.21.15.16:0/2295920266 learned_addr learned my addr 172.21.15.16:0/2295920266 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:28:11.396 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.387+0000 7f68515d4640 1 --2- 172.21.15.16:0/2295920266 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f684c159cd0 0x7f684c15a0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:11.396 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.387+0000 7f6851dd5640 1 --2- 172.21.15.16:0/2295920266 >> v2:172.21.15.62:3300/0 conn(0x7f684c15a5e0 0x7f684c0cc520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:11.397 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.387+0000 7f6850dd3640 1 -- 172.21.15.16:0/2295920266 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f684c159cd0 msgr2=0x7f684c15a0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:28:11.397 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.387+0000 7f6850dd3640 1 --2- 172.21.15.16:0/2295920266 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f684c159cd0 0x7f684c15a0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:11.397 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.387+0000 7f6850dd3640 1 -- 172.21.15.16:0/2295920266 >> v2:172.21.15.62:3300/0 conn(0x7f684c15a5e0 msgr2=0x7f684c0cc520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:28:11.397 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.387+0000 7f6850dd3640 1 --2- 172.21.15.16:0/2295920266 >> v2:172.21.15.62:3300/0 conn(0x7f684c15a5e0 0x7f684c0cc520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:11.397 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.387+0000 7f6850dd3640 1 -- 172.21.15.16:0/2295920266 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f684c15ae80 con 0x7f684c0b2880 2023-05-22T15:28:11.398 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.387+0000 7f6851dd5640 1 --2- 172.21.15.16:0/2295920266 >> v2:172.21.15.62:3300/0 conn(0x7f684c15a5e0 0x7f684c0cc520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:28:11.400 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.391+0000 7f6850dd3640 1 --2- 172.21.15.16:0/2295920266 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f684c0b2880 0x7f684c0cbfe0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f684c070360 tx=0x7f683c01c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3359da93d311b1b server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:28:11.400 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.391+0000 7f6843fff640 1 -- 172.21.15.16:0/2295920266 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f683c0aa610 con 0x7f684c0b2880 2023-05-22T15:28:11.400 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.391+0000 7f6843fff640 1 -- 172.21.15.16:0/2295920266 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f683c0a6040 con 0x7f684c0b2880 2023-05-22T15:28:11.400 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.391+0000 7f6843fff640 1 -- 172.21.15.16:0/2295920266 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f683c0aa990 con 0x7f684c0b2880 2023-05-22T15:28:11.401 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.391+0000 7f6853838640 1 -- 172.21.15.16:0/2295920266 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f684c0b2880 msgr2=0x7f684c0cbfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:11.401 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.391+0000 7f6853838640 1 --2- 172.21.15.16:0/2295920266 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f684c0b2880 0x7f684c0cbfe0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f684c070360 tx=0x7f683c01c4d0 comp rx=0 tx=0).stop 2023-05-22T15:28:11.401 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.391+0000 7f6853838640 1 -- 172.21.15.16:0/2295920266 shutdown_connections 2023-05-22T15:28:11.401 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.391+0000 7f6853838640 1 --2- 172.21.15.16:0/2295920266 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f684c0b2880 0x7f684c0cbfe0 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:11.401 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.391+0000 7f6853838640 1 --2- 172.21.15.16:0/2295920266 >> v2:172.21.15.62:3300/0 conn(0x7f684c15a5e0 0x7f684c0cc520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:11.401 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.391+0000 7f6853838640 1 --2- 172.21.15.16:0/2295920266 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f684c159cd0 0x7f684c15a0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:11.401 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.391+0000 7f6853838640 1 -- 172.21.15.16:0/2295920266 >> 172.21.15.16:0/2295920266 conn(0x7f684c06f5c0 msgr2=0x7f684c06f9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:28:11.402 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.391+0000 7f6853838640 1 -- 172.21.15.16:0/2295920266 shutdown_connections 2023-05-22T15:28:11.402 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.391+0000 7f6853838640 1 -- 172.21.15.16:0/2295920266 wait complete. 2023-05-22T15:28:11.402 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.391+0000 7f6853838640 1 Processor -- start 2023-05-22T15:28:11.402 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.391+0000 7f6853838640 1 -- start start 2023-05-22T15:28:11.402 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.391+0000 7f6853838640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f684c0b2880 0x7f684c0c4630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:11.402 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.391+0000 7f6853838640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f684c159cd0 0x7f684c0b5ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:11.402 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.391+0000 7f6853838640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f684c15a5e0 0x7f684c14e720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:11.402 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.391+0000 7f6853838640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f684c0b3230 con 0x7f684c15a5e0 2023-05-22T15:28:11.403 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.391+0000 7f6853838640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f684c149480 con 0x7f684c159cd0 2023-05-22T15:28:11.403 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.391+0000 7f6853838640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f684c0b60e0 con 0x7f684c0b2880 2023-05-22T15:28:11.403 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.391+0000 7f6851dd5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f684c15a5e0 0x7f684c14e720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:11.403 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.391+0000 7f6851dd5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f684c15a5e0 0x7f684c14e720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42662/0 (socket says 172.21.15.16:42662) 2023-05-22T15:28:11.403 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.391+0000 7f6851dd5640 1 -- 172.21.15.16:0/125027418 learned_addr learned my addr 172.21.15.16:0/125027418 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:28:11.403 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.391+0000 7f68515d4640 1 --2- 172.21.15.16:0/125027418 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f684c0b2880 0x7f684c0c4630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:11.403 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.391+0000 7f6850dd3640 1 --2- 172.21.15.16:0/125027418 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f684c159cd0 0x7f684c0b5ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:11.405 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.391+0000 7f6851dd5640 1 -- 172.21.15.16:0/125027418 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f684c0b2880 msgr2=0x7f684c0c4630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:11.405 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.391+0000 7f6851dd5640 1 --2- 172.21.15.16:0/125027418 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f684c0b2880 0x7f684c0c4630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:11.405 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.391+0000 7f6851dd5640 1 -- 172.21.15.16:0/125027418 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f684c159cd0 msgr2=0x7f684c0b5ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:11.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.391+0000 7f6851dd5640 1 --2- 172.21.15.16:0/125027418 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f684c159cd0 0x7f684c0b5ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:11.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.391+0000 7f6851dd5640 1 -- 172.21.15.16:0/125027418 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f683c0094d0 con 0x7f684c15a5e0 2023-05-22T15:28:11.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.391+0000 7f68515d4640 1 --2- 172.21.15.16:0/125027418 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f684c0b2880 0x7f684c0c4630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T15:28:11.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.391+0000 7f6850dd3640 1 --2- 172.21.15.16:0/125027418 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f684c159cd0 0x7f684c0b5ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T15:28:11.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.391+0000 7f6851dd5640 1 --2- 172.21.15.16:0/125027418 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f684c15a5e0 0x7f684c14e720 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f684400cd50 tx=0x7f684400a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:28:11.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.391+0000 7f6841ffb640 1 -- 172.21.15.16:0/125027418 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f684400f680 con 0x7f684c15a5e0 2023-05-22T15:28:11.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.395+0000 7f6841ffb640 1 -- 172.21.15.16:0/125027418 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f684400f800 con 0x7f684c15a5e0 2023-05-22T15:28:11.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.395+0000 7f6853838640 1 -- 172.21.15.16:0/125027418 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f684c0ceb00 con 0x7f684c15a5e0 2023-05-22T15:28:11.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.395+0000 7f6841ffb640 1 -- 172.21.15.16:0/125027418 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f684400fbb0 con 0x7f684c15a5e0 2023-05-22T15:28:11.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.395+0000 7f6853838640 1 -- 172.21.15.16:0/125027418 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f684c0ce970 con 0x7f684c15a5e0 2023-05-22T15:28:11.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.395+0000 7f6853838640 1 -- 172.21.15.16:0/125027418 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f684c0cec90 con 0x7f684c15a5e0 2023-05-22T15:28:11.408 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.395+0000 7f6841ffb640 1 -- 172.21.15.16:0/125027418 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7f684400b040 con 0x7f684c15a5e0 2023-05-22T15:28:11.408 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.395+0000 7f6841ffb640 1 --2- 172.21.15.16:0/125027418 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f6830065380 0x7f6830067830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:11.408 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.395+0000 7f6841ffb640 1 -- 172.21.15.16:0/125027418 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 5530+0+0 (secure 0 0 0) 0x7f6844083090 con 0x7f684c15a5e0 2023-05-22T15:28:11.408 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.395+0000 7f68515d4640 1 --2- 172.21.15.16:0/125027418 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f6830065380 0x7f6830067830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:11.408 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.395+0000 7f68515d4640 1 --2- 172.21.15.16:0/125027418 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f6830065380 0x7f6830067830 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f684c158920 tx=0x7f6848008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:28:11.409 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.399+0000 7f6841ffb640 1 -- 172.21.15.16:0/125027418 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f684404f620 con 0x7f684c15a5e0 2023-05-22T15:28:11.556 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.547+0000 7f6853838640 1 -- 172.21.15.16:0/125027418 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f684c04dbe0 con 0x7f684c15a5e0 2023-05-22T15:28:11.556 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.551+0000 7f6841ffb640 1 -- 172.21.15.16:0/125027418 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v38) v1 ==== 84+0+126 (secure 0 0 0) 0x7f6844054a30 con 0x7f684c15a5e0 2023-05-22T15:28:11.557 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:28:11.557 INFO:teuthology.orchestra.run.smithi016.stdout:[ 2023-05-22T15:28:11.557 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T15:28:11.557 INFO:teuthology.orchestra.run.smithi016.stdout: "poolnum": 1, 2023-05-22T15:28:11.557 INFO:teuthology.orchestra.run.smithi016.stdout: "poolname": ".mgr" 2023-05-22T15:28:11.557 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:28:11.558 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T15:28:11.558 INFO:teuthology.orchestra.run.smithi016.stdout: "poolnum": 2, 2023-05-22T15:28:11.558 INFO:teuthology.orchestra.run.smithi016.stdout: "poolname": "rbd" 2023-05-22T15:28:11.558 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T15:28:11.558 INFO:teuthology.orchestra.run.smithi016.stdout:] 2023-05-22T15:28:11.559 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.551+0000 7f6853838640 1 -- 172.21.15.16:0/125027418 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f6830065380 msgr2=0x7f6830067830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:11.559 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.551+0000 7f6853838640 1 --2- 172.21.15.16:0/125027418 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f6830065380 0x7f6830067830 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f684c158920 tx=0x7f6848008040 comp rx=0 tx=0).stop 2023-05-22T15:28:11.559 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.551+0000 7f6853838640 1 -- 172.21.15.16:0/125027418 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f684c15a5e0 msgr2=0x7f684c14e720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:11.559 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.551+0000 7f6853838640 1 --2- 172.21.15.16:0/125027418 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f684c15a5e0 0x7f684c14e720 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f684400cd50 tx=0x7f684400a650 comp rx=0 tx=0).stop 2023-05-22T15:28:11.559 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.551+0000 7f6853838640 1 -- 172.21.15.16:0/125027418 shutdown_connections 2023-05-22T15:28:11.560 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.551+0000 7f6853838640 1 --2- 172.21.15.16:0/125027418 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f684c15a5e0 0x7f684c14e720 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:11.560 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.551+0000 7f6853838640 1 --2- 172.21.15.16:0/125027418 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f684c159cd0 0x7f684c0b5ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:11.560 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.551+0000 7f6853838640 1 --2- 172.21.15.16:0/125027418 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f6830065380 0x7f6830067830 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:11.560 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.551+0000 7f6853838640 1 --2- 172.21.15.16:0/125027418 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f684c0b2880 0x7f684c0c4630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:11.560 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.551+0000 7f6853838640 1 -- 172.21.15.16:0/125027418 >> 172.21.15.16:0/125027418 conn(0x7f684c06f5c0 msgr2=0x7f684c06dc40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:28:11.560 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.551+0000 7f6853838640 1 -- 172.21.15.16:0/125027418 shutdown_connections 2023-05-22T15:28:11.560 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.551+0000 7f6853838640 1 -- 172.21.15.16:0/125027418 wait complete. 2023-05-22T15:28:11.574 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-05-22T15:28:11.574 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-05-22T15:28:11.702 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.695+0000 7fd0459c3640 1 Processor -- start 2023-05-22T15:28:11.702 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.695+0000 7fd0459c3640 1 -- start start 2023-05-22T15:28:11.702 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.695+0000 7fd0459c3640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fd04015b9e0 0x7fd0400adca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:11.703 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.695+0000 7fd0459c3640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd04015bdb0 0x7fd0400cad80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:11.703 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.695+0000 7fd0459c3640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0400ce740 con 0x7fd04006fef0 2023-05-22T15:28:11.703 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.695+0000 7fd0459c3640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0400ce460 con 0x7fd04015b9e0 2023-05-22T15:28:11.703 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.695+0000 7fd0459c3640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0400caae0 con 0x7fd04015bdb0 2023-05-22T15:28:11.704 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.695+0000 7fd03e7fc640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd04015bdb0 0x7fd0400cad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:11.704 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.695+0000 7fd03effd640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fd04015b9e0 0x7fd0400adca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:11.704 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.695+0000 7fd03e7fc640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd04015bdb0 0x7fd0400cad80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3301/0 says I am v2:172.21.15.16:49864/0 (socket says 172.21.15.16:49864) 2023-05-22T15:28:11.704 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.695+0000 7fd03e7fc640 1 -- 172.21.15.16:0/3061410357 learned_addr learned my addr 172.21.15.16:0/3061410357 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:28:11.705 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.695+0000 7fd03effd640 1 -- 172.21.15.16:0/3061410357 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd04015bdb0 msgr2=0x7fd0400cad80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:28:11.705 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.695+0000 7fd03effd640 1 --2- 172.21.15.16:0/3061410357 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd04015bdb0 0x7fd0400cad80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:11.705 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.695+0000 7fd03effd640 1 -- 172.21.15.16:0/3061410357 >> v1:172.21.15.62:6789/0 conn(0x7fd04006fef0 legacy=0x7fd040069a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:11.705 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.695+0000 7fd03effd640 1 -- 172.21.15.16:0/3061410357 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0400cde10 con 0x7fd04015b9e0 2023-05-22T15:28:11.706 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.699+0000 7fd03effd640 1 --2- 172.21.15.16:0/3061410357 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd04015b9e0 0x7fd0400adca0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fd0300097c0 tx=0x7fd03001c3f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fa14aa03dcfd03d0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:28:11.706 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.699+0000 7fd03dffb640 1 -- 172.21.15.16:0/3061410357 <== mon.1 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0300ab600 con 0x7fd04015b9e0 2023-05-22T15:28:11.707 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.699+0000 7fd03dffb640 1 -- 172.21.15.16:0/3061410357 <== mon.1 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0300a7050 con 0x7fd04015b9e0 2023-05-22T15:28:11.707 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.699+0000 7fd03dffb640 1 -- 172.21.15.16:0/3061410357 <== mon.1 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0300ab980 con 0x7fd04015b9e0 2023-05-22T15:28:11.708 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.699+0000 7fd0459c3640 1 -- 172.21.15.16:0/3061410357 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd04015b9e0 msgr2=0x7fd0400adca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:11.708 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.699+0000 7fd0459c3640 1 --2- 172.21.15.16:0/3061410357 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd04015b9e0 0x7fd0400adca0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fd0300097c0 tx=0x7fd03001c3f0 comp rx=0 tx=0).stop 2023-05-22T15:28:11.708 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.699+0000 7fd0459c3640 1 -- 172.21.15.16:0/3061410357 shutdown_connections 2023-05-22T15:28:11.708 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.699+0000 7fd0459c3640 1 --2- 172.21.15.16:0/3061410357 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd04015bdb0 0x7fd0400cad80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:11.708 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.699+0000 7fd0459c3640 1 --2- 172.21.15.16:0/3061410357 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd04015b9e0 0x7fd0400adca0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:11.709 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.699+0000 7fd0459c3640 1 -- 172.21.15.16:0/3061410357 >> 172.21.15.16:0/3061410357 conn(0x7fd0400ce8d0 msgr2=0x7fd0400cecd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:28:11.710 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.699+0000 7fd0459c3640 1 -- 172.21.15.16:0/3061410357 shutdown_connections 2023-05-22T15:28:11.710 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.699+0000 7fd0459c3640 1 -- 172.21.15.16:0/3061410357 wait complete. 2023-05-22T15:28:11.711 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.703+0000 7fd0459c3640 1 Processor -- start 2023-05-22T15:28:11.711 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.703+0000 7fd0459c3640 1 -- start start 2023-05-22T15:28:11.712 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.703+0000 7fd0459c3640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd04006fef0 0x7fd04014bfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:11.712 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.703+0000 7fd0459c3640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd04015b9e0 0x7fd04015e580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:11.712 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.703+0000 7fd0459c3640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd04015bdb0 0x7fd04015eac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:11.712 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.703+0000 7fd0459c3640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd04014c660 con 0x7fd04015bdb0 2023-05-22T15:28:11.712 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.703+0000 7fd0459c3640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0401531b0 con 0x7fd04006fef0 2023-05-22T15:28:11.712 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.703+0000 7fd0459c3640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd040158cf0 con 0x7fd04015b9e0 2023-05-22T15:28:11.712 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.703+0000 7fd03f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd04015bdb0 0x7fd04015eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:11.712 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.703+0000 7fd03f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd04015bdb0 0x7fd04015eac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42668/0 (socket says 172.21.15.16:42668) 2023-05-22T15:28:11.713 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.703+0000 7fd03f7fe640 1 -- 172.21.15.16:0/3422492345 learned_addr learned my addr 172.21.15.16:0/3422492345 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:28:11.713 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.703+0000 7fd03e7fc640 1 --2- 172.21.15.16:0/3422492345 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd04015b9e0 0x7fd04015e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:11.713 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.703+0000 7fd03effd640 1 --2- 172.21.15.16:0/3422492345 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd04006fef0 0x7fd04014bfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:11.713 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.703+0000 7fd03f7fe640 1 -- 172.21.15.16:0/3422492345 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd04015b9e0 msgr2=0x7fd04015e580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:11.713 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.703+0000 7fd03f7fe640 1 --2- 172.21.15.16:0/3422492345 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd04015b9e0 0x7fd04015e580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:11.713 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.703+0000 7fd03f7fe640 1 -- 172.21.15.16:0/3422492345 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd04006fef0 msgr2=0x7fd04014bfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:11.713 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.703+0000 7fd03f7fe640 1 --2- 172.21.15.16:0/3422492345 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd04006fef0 0x7fd04014bfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:11.714 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.703+0000 7fd03f7fe640 1 -- 172.21.15.16:0/3422492345 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd030009420 con 0x7fd04015bdb0 2023-05-22T15:28:11.714 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.703+0000 7fd03effd640 1 --2- 172.21.15.16:0/3422492345 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd04006fef0 0x7fd04014bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T15:28:11.714 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.703+0000 7fd03f7fe640 1 --2- 172.21.15.16:0/3422492345 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd04015bdb0 0x7fd04015eac0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7fd02800c370 tx=0x7fd02800c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:28:11.714 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.703+0000 7fd037fff640 1 -- 172.21.15.16:0/3422492345 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd028005220 con 0x7fd04015bdb0 2023-05-22T15:28:11.714 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.703+0000 7fd037fff640 1 -- 172.21.15.16:0/3422492345 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd028005380 con 0x7fd04015bdb0 2023-05-22T15:28:11.714 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.703+0000 7fd037fff640 1 -- 172.21.15.16:0/3422492345 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd028007270 con 0x7fd04015bdb0 2023-05-22T15:28:11.714 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.703+0000 7fd0459c3640 1 -- 172.21.15.16:0/3422492345 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd040070890 con 0x7fd04015bdb0 2023-05-22T15:28:11.714 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.703+0000 7fd0459c3640 1 -- 172.21.15.16:0/3422492345 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd040070700 con 0x7fd04015bdb0 2023-05-22T15:28:11.715 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.703+0000 7fd0459c3640 1 -- 172.21.15.16:0/3422492345 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd04006a520 con 0x7fd04015bdb0 2023-05-22T15:28:11.715 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.707+0000 7fd037fff640 1 -- 172.21.15.16:0/3422492345 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7fd0280073d0 con 0x7fd04015bdb0 2023-05-22T15:28:11.716 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.707+0000 7fd037fff640 1 --2- 172.21.15.16:0/3422492345 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd020065380 0x7fd020067830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:11.716 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.707+0000 7fd037fff640 1 -- 172.21.15.16:0/3422492345 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 5530+0+0 (secure 0 0 0) 0x7fd028081ad0 con 0x7fd04015bdb0 2023-05-22T15:28:11.716 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.707+0000 7fd03effd640 1 --2- 172.21.15.16:0/3422492345 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd020065380 0x7fd020067830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:11.716 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.707+0000 7fd03effd640 1 --2- 172.21.15.16:0/3422492345 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd020065380 0x7fd020067830 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fd0300ad000 tx=0x7fd0300a6000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:28:11.716 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.711+0000 7fd037fff640 1 -- 172.21.15.16:0/3422492345 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fd02804f9f0 con 0x7fd04015bdb0 2023-05-22T15:28:11.866 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.859+0000 7fd0459c3640 1 -- 172.21.15.16:0/3422492345 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fd0400b1a80 con 0x7fd04015bdb0 2023-05-22T15:28:11.937 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.931+0000 7fd037fff640 1 -- 172.21.15.16:0/3422492345 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v39) v1 ==== 138+0+0 (secure 0 0 0) 0x7fd0400b1a80 con 0x7fd04015bdb0 2023-05-22T15:28:11.937 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'cephfs_metadata' created 2023-05-22T15:28:11.940 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.931+0000 7fd0459c3640 1 -- 172.21.15.16:0/3422492345 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd020065380 msgr2=0x7fd020067830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:11.940 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.931+0000 7fd0459c3640 1 --2- 172.21.15.16:0/3422492345 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd020065380 0x7fd020067830 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fd0300ad000 tx=0x7fd0300a6000 comp rx=0 tx=0).stop 2023-05-22T15:28:11.940 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.931+0000 7fd0459c3640 1 -- 172.21.15.16:0/3422492345 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd04015bdb0 msgr2=0x7fd04015eac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:11.940 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.931+0000 7fd0459c3640 1 --2- 172.21.15.16:0/3422492345 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd04015bdb0 0x7fd04015eac0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7fd02800c370 tx=0x7fd02800c830 comp rx=0 tx=0).stop 2023-05-22T15:28:11.940 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.931+0000 7fd0459c3640 1 -- 172.21.15.16:0/3422492345 shutdown_connections 2023-05-22T15:28:11.940 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.931+0000 7fd0459c3640 1 --2- 172.21.15.16:0/3422492345 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd04015bdb0 0x7fd04015eac0 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:11.941 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.931+0000 7fd0459c3640 1 --2- 172.21.15.16:0/3422492345 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd020065380 0x7fd020067830 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:11.941 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.931+0000 7fd0459c3640 1 --2- 172.21.15.16:0/3422492345 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd04015b9e0 0x7fd04015e580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:11.941 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.931+0000 7fd0459c3640 1 --2- 172.21.15.16:0/3422492345 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd04006fef0 0x7fd04014bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:11.941 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.931+0000 7fd0459c3640 1 -- 172.21.15.16:0/3422492345 >> 172.21.15.16:0/3422492345 conn(0x7fd0400ce8d0 msgr2=0x7fd04015cb50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:28:11.941 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.931+0000 7fd0459c3640 1 -- 172.21.15.16:0/3422492345 shutdown_connections 2023-05-22T15:28:11.941 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:11.931+0000 7fd0459c3640 1 -- 172.21.15.16:0/3422492345 wait complete. 2023-05-22T15:28:11.962 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-05-22T15:28:12.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:12.087+0000 7f8d9692a640 1 Processor -- start 2023-05-22T15:28:12.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:12.087+0000 7f8d9692a640 1 -- start start 2023-05-22T15:28:12.096 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:12.087+0000 7f8d9692a640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8d900cb4c0 0x7f8d90158fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:12.096 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:12.087+0000 7f8d9692a640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f8d90159510 0x7f8d9006fba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:12.096 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:12.087+0000 7f8d9692a640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d90159f70 con 0x7f8d9006b6c0 2023-05-22T15:28:12.097 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:12.087+0000 7f8d9692a640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d90071760 con 0x7f8d90159510 2023-05-22T15:28:12.097 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:12.087+0000 7f8d9692a640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d900cb910 con 0x7f8d900cb4c0 2023-05-22T15:28:12.097 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:12.087+0000 7f8d94ec7640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f8d90159510 0x7f8d9006fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:12.097 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:12.087+0000 7f8d94ec7640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f8d90159510 0x7f8d9006fba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42670/0 (socket says 172.21.15.16:42670) 2023-05-22T15:28:12.097 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:12.087+0000 7f8d94ec7640 1 -- 172.21.15.16:0/1154727642 learned_addr learned my addr 172.21.15.16:0/1154727642 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:28:12.097 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:12.087+0000 7f8d8f7fe640 1 --2- 172.21.15.16:0/1154727642 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8d900cb4c0 0x7f8d90158fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:12.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:12.087+0000 7f8d94ec7640 1 -- 172.21.15.16:0/1154727642 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8d900cb4c0 msgr2=0x7f8d90158fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:28:12.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:12.087+0000 7f8d94ec7640 1 --2- 172.21.15.16:0/1154727642 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8d900cb4c0 0x7f8d90158fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:12.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:12.087+0000 7f8d94ec7640 1 -- 172.21.15.16:0/1154727642 >> v1:172.21.15.62:6789/0 conn(0x7f8d9006b6c0 legacy=0x7f8d9006ba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:12.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:12.087+0000 7f8d94ec7640 1 -- 172.21.15.16:0/1154727642 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d900cb030 con 0x7f8d90159510 2023-05-22T15:28:12.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:12.091+0000 7f8d94ec7640 1 --2- 172.21.15.16:0/1154727642 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8d90159510 0x7f8d9006fba0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f8d88009e50 tx=0x7f8d8801ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=723e685e7574443f server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:28:12.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:12.091+0000 7f8d8effd640 1 -- 172.21.15.16:0/1154727642 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d880a57d0 con 0x7f8d90159510 2023-05-22T15:28:12.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:12.091+0000 7f8d8effd640 1 -- 172.21.15.16:0/1154727642 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d880a5930 con 0x7f8d90159510 2023-05-22T15:28:12.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:12.091+0000 7f8d8effd640 1 -- 172.21.15.16:0/1154727642 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d880a5cc0 con 0x7f8d90159510 2023-05-22T15:28:12.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:12.091+0000 7f8d9692a640 1 -- 172.21.15.16:0/1154727642 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8d90159510 msgr2=0x7f8d9006fba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:12.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:12.091+0000 7f8d9692a640 1 --2- 172.21.15.16:0/1154727642 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8d90159510 0x7f8d9006fba0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f8d88009e50 tx=0x7f8d8801ca10 comp rx=0 tx=0).stop 2023-05-22T15:28:12.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:12.091+0000 7f8d9692a640 1 -- 172.21.15.16:0/1154727642 shutdown_connections 2023-05-22T15:28:12.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:12.091+0000 7f8d9692a640 1 --2- 172.21.15.16:0/1154727642 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8d90159510 0x7f8d9006fba0 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:12.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:12.091+0000 7f8d9692a640 1 --2- 172.21.15.16:0/1154727642 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8d900cb4c0 0x7f8d90158fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:12.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:12.091+0000 7f8d9692a640 1 -- 172.21.15.16:0/1154727642 >> 172.21.15.16:0/1154727642 conn(0x7f8d90069720 msgr2=0x7f8d90069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:28:12.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:12.091+0000 7f8d9692a640 1 -- 172.21.15.16:0/1154727642 shutdown_connections 2023-05-22T15:28:12.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:12.091+0000 7f8d9692a640 1 -- 172.21.15.16:0/1154727642 wait complete. 2023-05-22T15:28:12.103 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:12.091+0000 7f8d9692a640 1 Processor -- start 2023-05-22T15:28:12.104 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:12.091+0000 7f8d9692a640 1 -- start start 2023-05-22T15:28:12.104 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:12.095+0000 7f8d9692a640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8d9006b6c0 0x7f8d9014a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:12.104 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:12.095+0000 7f8d9692a640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8d900cb4c0 0x7f8d900b02e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:12.104 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:12.095+0000 7f8d9692a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8d90159510 0x7f8d900b3010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:12.104 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:12.095+0000 7f8d9692a640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d900adab0 con 0x7f8d90159510 2023-05-22T15:28:12.104 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:12.095+0000 7f8d9692a640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d900ae910 con 0x7f8d9006b6c0 2023-05-22T15:28:12.104 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:12.095+0000 7f8d9692a640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d900afda0 con 0x7f8d900cb4c0 2023-05-22T15:28:12.105 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:12.095+0000 7f8d94ec7640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8d90159510 0x7f8d900b3010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:12.105 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:12.095+0000 7f8d94ec7640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8d90159510 0x7f8d900b3010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42674/0 (socket says 172.21.15.16:42674) 2023-05-22T15:28:12.105 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:12.095+0000 7f8d94ec7640 1 -- 172.21.15.16:0/2631345354 learned_addr learned my addr 172.21.15.16:0/2631345354 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:28:12.105 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:12.095+0000 7f8d8ffff640 1 --2- 172.21.15.16:0/2631345354 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8d9006b6c0 0x7f8d9014a5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:12.105 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:12.095+0000 7f8d8f7fe640 1 --2- 172.21.15.16:0/2631345354 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8d900cb4c0 0x7f8d900b02e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:12.105 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:12.095+0000 7f8d94ec7640 1 -- 172.21.15.16:0/2631345354 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8d900cb4c0 msgr2=0x7f8d900b02e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:12.105 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:12.095+0000 7f8d94ec7640 1 --2- 172.21.15.16:0/2631345354 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8d900cb4c0 0x7f8d900b02e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:12.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:12.095+0000 7f8d94ec7640 1 -- 172.21.15.16:0/2631345354 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8d9006b6c0 msgr2=0x7f8d9014a5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:12.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:12.095+0000 7f8d94ec7640 1 --2- 172.21.15.16:0/2631345354 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8d9006b6c0 0x7f8d9014a5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:12.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:12.095+0000 7f8d94ec7640 1 -- 172.21.15.16:0/2631345354 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d88009b30 con 0x7f8d90159510 2023-05-22T15:28:12.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:12.095+0000 7f8d94ec7640 1 --2- 172.21.15.16:0/2631345354 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8d90159510 0x7f8d900b3010 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f8d8801a640 tx=0x7f8d880197b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:28:12.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:12.095+0000 7f8d8cff9640 1 -- 172.21.15.16:0/2631345354 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d880b8640 con 0x7f8d90159510 2023-05-22T15:28:12.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:12.095+0000 7f8d8cff9640 1 -- 172.21.15.16:0/2631345354 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d880b87a0 con 0x7f8d90159510 2023-05-22T15:28:12.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:12.095+0000 7f8d9692a640 1 -- 172.21.15.16:0/2631345354 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d90159990 con 0x7f8d90159510 2023-05-22T15:28:12.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:12.095+0000 7f8d8cff9640 1 -- 172.21.15.16:0/2631345354 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d880b8b50 con 0x7f8d90159510 2023-05-22T15:28:12.107 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:12.095+0000 7f8d9692a640 1 -- 172.21.15.16:0/2631345354 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d900cacc0 con 0x7f8d90159510 2023-05-22T15:28:12.107 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:12.095+0000 7f8d9692a640 1 -- 172.21.15.16:0/2631345354 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d90156840 con 0x7f8d90159510 2023-05-22T15:28:12.107 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:12.095+0000 7f8d8cff9640 1 -- 172.21.15.16:0/2631345354 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7f8d880b8cb0 con 0x7f8d90159510 2023-05-22T15:28:12.108 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:12.095+0000 7f8d8cff9640 1 --2- 172.21.15.16:0/2631345354 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f8d7c065380 0x7f8d7c067830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:12.108 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:12.095+0000 7f8d8cff9640 1 -- 172.21.15.16:0/2631345354 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 5900+0+0 (secure 0 0 0) 0x7f8d8811d880 con 0x7f8d90159510 2023-05-22T15:28:12.108 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:12.099+0000 7f8d8ffff640 1 --2- 172.21.15.16:0/2631345354 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f8d7c065380 0x7f8d7c067830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:12.108 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:12.099+0000 7f8d8ffff640 1 --2- 172.21.15.16:0/2631345354 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f8d7c065380 0x7f8d7c067830 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f8d80001650 tx=0x7f8d80009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:28:12.108 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:12.103+0000 7f8d8cff9640 1 -- 172.21.15.16:0/2631345354 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f8d880ef040 con 0x7f8d90159510 2023-05-22T15:28:12.260 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:12.251+0000 7f8d9692a640 1 -- 172.21.15.16:0/2631345354 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f8d9006bfd0 con 0x7f8d90159510 2023-05-22T15:28:12.953 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:12.947+0000 7f8d8cff9640 1 -- 172.21.15.16:0/2631345354 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v40) v1 ==== 170+0+0 (secure 0 0 0) 0x7f8d880eec30 con 0x7f8d90159510 2023-05-22T15:28:12.966 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'cephfs_data' created 2023-05-22T15:28:12.973 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:12.967+0000 7f8d9692a640 1 -- 172.21.15.16:0/2631345354 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f8d7c065380 msgr2=0x7f8d7c067830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:12.973 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:12.967+0000 7f8d9692a640 1 --2- 172.21.15.16:0/2631345354 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f8d7c065380 0x7f8d7c067830 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f8d80001650 tx=0x7f8d80009040 comp rx=0 tx=0).stop 2023-05-22T15:28:12.974 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:12.967+0000 7f8d9692a640 1 -- 172.21.15.16:0/2631345354 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8d90159510 msgr2=0x7f8d900b3010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:12.974 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:12.967+0000 7f8d9692a640 1 --2- 172.21.15.16:0/2631345354 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8d90159510 0x7f8d900b3010 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f8d8801a640 tx=0x7f8d880197b0 comp rx=0 tx=0).stop 2023-05-22T15:28:12.974 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:12.967+0000 7f8d9692a640 1 -- 172.21.15.16:0/2631345354 shutdown_connections 2023-05-22T15:28:12.974 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:12.967+0000 7f8d9692a640 1 --2- 172.21.15.16:0/2631345354 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8d90159510 0x7f8d900b3010 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:12.975 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:12.967+0000 7f8d9692a640 1 --2- 172.21.15.16:0/2631345354 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f8d7c065380 0x7f8d7c067830 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:12.975 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:12.967+0000 7f8d9692a640 1 --2- 172.21.15.16:0/2631345354 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8d900cb4c0 0x7f8d900b02e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:12.975 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:12.967+0000 7f8d9692a640 1 --2- 172.21.15.16:0/2631345354 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8d9006b6c0 0x7f8d9014a5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:12.975 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:12.967+0000 7f8d9692a640 1 -- 172.21.15.16:0/2631345354 >> 172.21.15.16:0/2631345354 conn(0x7f8d90069720 msgr2=0x7f8d901583f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:28:12.975 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:12.967+0000 7f8d9692a640 1 -- 172.21.15.16:0/2631345354 shutdown_connections 2023-05-22T15:28:12.976 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:12.967+0000 7f8d9692a640 1 -- 172.21.15.16:0/2631345354 wait complete. 2023-05-22T15:28:13.002 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-05-22T15:28:13.121 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.115+0000 7efdb9f11640 1 Processor -- start 2023-05-22T15:28:13.121 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.115+0000 7efdb9f11640 1 -- start start 2023-05-22T15:28:13.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.115+0000 7efdb9f11640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7efdb4159970 0x7efdb4159d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:13.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.115+0000 7efdb9f11640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7efdb415d680 0x7efdb40cb260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:13.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.115+0000 7efdb9f11640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efdb4070c30 con 0x7efdb415dad0 2023-05-22T15:28:13.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.115+0000 7efdb9f11640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efdb4070e60 con 0x7efdb4159970 2023-05-22T15:28:13.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.115+0000 7efdb9f11640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efdb4071140 con 0x7efdb415d680 2023-05-22T15:28:13.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.115+0000 7efdb37fe640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7efdb4159970 0x7efdb4159d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:13.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.115+0000 7efdb2ffd640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7efdb415d680 0x7efdb40cb260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:13.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.115+0000 7efdb37fe640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7efdb4159970 0x7efdb4159d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.16:59968/0 (socket says 172.21.15.16:59968) 2023-05-22T15:28:13.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.115+0000 7efdb37fe640 1 -- 172.21.15.16:0/878808215 learned_addr learned my addr 172.21.15.16:0/878808215 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:28:13.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.115+0000 7efdb3fff640 1 --1- 172.21.15.16:0/878808215 >> v1:172.21.15.62:6789/0 conn(0x7efdb415dad0 0x7efdb406b6c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.16:40470/0 (socket says 172.21.15.16:40470) 2023-05-22T15:28:13.126 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.115+0000 7efdb37fe640 1 -- 172.21.15.16:0/878808215 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7efdb415d680 msgr2=0x7efdb40cb260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:28:13.126 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.115+0000 7efdb37fe640 1 --2- 172.21.15.16:0/878808215 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7efdb415d680 0x7efdb40cb260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:13.126 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.115+0000 7efdb37fe640 1 -- 172.21.15.16:0/878808215 >> v1:172.21.15.62:6789/0 conn(0x7efdb415dad0 legacy=0x7efdb406b6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:13.126 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.115+0000 7efdb37fe640 1 -- 172.21.15.16:0/878808215 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdb4071420 con 0x7efdb4159970 2023-05-22T15:28:13.131 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.119+0000 7efdb37fe640 1 --2- 172.21.15.16:0/878808215 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efdb4159970 0x7efdb4159d40 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7efda4009870 tx=0x7efda401c480 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f0218fc59649680a server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:28:13.131 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.119+0000 7efdb27fc640 1 -- 172.21.15.16:0/878808215 <== mon.1 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efda40a6040 con 0x7efdb4159970 2023-05-22T15:28:13.132 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.119+0000 7efdb27fc640 1 -- 172.21.15.16:0/878808215 <== mon.1 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efda40aa600 con 0x7efdb4159970 2023-05-22T15:28:13.132 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.119+0000 7efdb27fc640 1 -- 172.21.15.16:0/878808215 <== mon.1 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efda40aa990 con 0x7efdb4159970 2023-05-22T15:28:13.132 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.119+0000 7efdb9f11640 1 -- 172.21.15.16:0/878808215 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efdb4159970 msgr2=0x7efdb4159d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:13.132 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.119+0000 7efdb9f11640 1 --2- 172.21.15.16:0/878808215 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efdb4159970 0x7efdb4159d40 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7efda4009870 tx=0x7efda401c480 comp rx=0 tx=0).stop 2023-05-22T15:28:13.132 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.119+0000 7efdb9f11640 1 -- 172.21.15.16:0/878808215 shutdown_connections 2023-05-22T15:28:13.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.119+0000 7efdb9f11640 1 --2- 172.21.15.16:0/878808215 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7efdb415d680 0x7efdb40cb260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:13.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.119+0000 7efdb9f11640 1 --2- 172.21.15.16:0/878808215 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efdb4159970 0x7efdb4159d40 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:13.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.119+0000 7efdb9f11640 1 -- 172.21.15.16:0/878808215 >> 172.21.15.16:0/878808215 conn(0x7efdb4069720 msgr2=0x7efdb4069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:28:13.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.119+0000 7efdb9f11640 1 -- 172.21.15.16:0/878808215 shutdown_connections 2023-05-22T15:28:13.134 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.119+0000 7efdb9f11640 1 -- 172.21.15.16:0/878808215 wait complete. 2023-05-22T15:28:13.134 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.119+0000 7efdb9f11640 1 Processor -- start 2023-05-22T15:28:13.134 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.119+0000 7efdb9f11640 1 -- start start 2023-05-22T15:28:13.134 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.123+0000 7efdb9f11640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7efdb4159970 0x7efdb4155d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:13.134 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.123+0000 7efdb9f11640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efdb415d680 0x7efdb40cd590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:13.134 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.123+0000 7efdb9f11640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efdb415dad0 0x7efdb41541a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:13.134 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.123+0000 7efdb9f11640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdb4157000 con 0x7efdb415dad0 2023-05-22T15:28:13.135 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.123+0000 7efdb9f11640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdb4156380 con 0x7efdb415d680 2023-05-22T15:28:13.135 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.123+0000 7efdb9f11640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efdb4156660 con 0x7efdb4159970 2023-05-22T15:28:13.135 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.123+0000 7efdb3fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efdb415dad0 0x7efdb41541a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:13.135 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.123+0000 7efdb3fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efdb415dad0 0x7efdb41541a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42690/0 (socket says 172.21.15.16:42690) 2023-05-22T15:28:13.135 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.123+0000 7efdb3fff640 1 -- 172.21.15.16:0/1855946684 learned_addr learned my addr 172.21.15.16:0/1855946684 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:28:13.136 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.123+0000 7efdb2ffd640 1 --2- 172.21.15.16:0/1855946684 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efdb415d680 0x7efdb40cd590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:13.136 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.123+0000 7efdb37fe640 1 --2- 172.21.15.16:0/1855946684 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7efdb4159970 0x7efdb4155d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:13.136 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.123+0000 7efdb3fff640 1 -- 172.21.15.16:0/1855946684 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7efdb4159970 msgr2=0x7efdb4155d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:13.136 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.123+0000 7efdb3fff640 1 --2- 172.21.15.16:0/1855946684 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7efdb4159970 0x7efdb4155d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:13.136 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.123+0000 7efdb3fff640 1 -- 172.21.15.16:0/1855946684 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efdb415d680 msgr2=0x7efdb40cd590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:13.136 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.123+0000 7efdb3fff640 1 --2- 172.21.15.16:0/1855946684 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efdb415d680 0x7efdb40cd590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:13.136 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.123+0000 7efdb3fff640 1 -- 172.21.15.16:0/1855946684 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efda40094d0 con 0x7efdb415dad0 2023-05-22T15:28:13.137 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.123+0000 7efdb2ffd640 1 --2- 172.21.15.16:0/1855946684 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efdb415d680 0x7efdb40cd590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:28:13.137 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.123+0000 7efdb3fff640 1 --2- 172.21.15.16:0/1855946684 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efdb415dad0 0x7efdb41541a0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7efdac00c730 tx=0x7efdac00cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:28:13.137 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.123+0000 7efd9bfff640 1 -- 172.21.15.16:0/1855946684 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdac00ce40 con 0x7efdb415dad0 2023-05-22T15:28:13.137 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.123+0000 7efd9bfff640 1 -- 172.21.15.16:0/1855946684 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efdac00a640 con 0x7efdb415dad0 2023-05-22T15:28:13.137 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.123+0000 7efdb9f11640 1 -- 172.21.15.16:0/1855946684 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efdb40cae20 con 0x7efdb415dad0 2023-05-22T15:28:13.137 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.123+0000 7efd9bfff640 1 -- 172.21.15.16:0/1855946684 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdac00a9f0 con 0x7efdb415dad0 2023-05-22T15:28:13.137 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.123+0000 7efdb9f11640 1 -- 172.21.15.16:0/1855946684 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efdb40cac90 con 0x7efdb415dad0 2023-05-22T15:28:13.137 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.123+0000 7efdb9f11640 1 -- 172.21.15.16:0/1855946684 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efdb406bdd0 con 0x7efdb415dad0 2023-05-22T15:28:13.138 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.123+0000 7efd9bfff640 1 -- 172.21.15.16:0/1855946684 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7efdac00ab50 con 0x7efdb415dad0 2023-05-22T15:28:13.139 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.123+0000 7efd9bfff640 1 --2- 172.21.15.16:0/1855946684 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7efda0065380 0x7efda0067830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:13.139 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.123+0000 7efd9bfff640 1 -- 172.21.15.16:0/1855946684 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6282+0+0 (secure 0 0 0) 0x7efdac081ff0 con 0x7efdb415dad0 2023-05-22T15:28:13.139 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.123+0000 7efdb37fe640 1 --2- 172.21.15.16:0/1855946684 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7efda0065380 0x7efda0067830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:13.139 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.123+0000 7efdb37fe640 1 --2- 172.21.15.16:0/1855946684 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7efda0065380 0x7efda0067830 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7efda40ae000 tx=0x7efda40a7000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:28:13.139 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.127+0000 7efd9bfff640 1 -- 172.21.15.16:0/1855946684 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7efdac04a620 con 0x7efdb415dad0 2023-05-22T15:28:13.321 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.315+0000 7efdb9f11640 1 -- 172.21.15.16:0/1855946684 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7efdb4157140 con 0x7efdb415dad0 2023-05-22T15:28:13.331 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.323+0000 7efd9bfff640 1 -- 172.21.15.16:0/1855946684 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 6 and data pool 7 v12) v1 ==== 172+0+0 (secure 0 0 0) 0x7efdb4157140 con 0x7efdb415dad0 2023-05-22T15:28:13.332 INFO:teuthology.orchestra.run.smithi016.stderr:new fs with metadata pool 6 and data pool 7 2023-05-22T15:28:13.346 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.339+0000 7efdb9f11640 1 -- 172.21.15.16:0/1855946684 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7efda0065380 msgr2=0x7efda0067830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:13.346 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.339+0000 7efdb9f11640 1 --2- 172.21.15.16:0/1855946684 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7efda0065380 0x7efda0067830 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7efda40ae000 tx=0x7efda40a7000 comp rx=0 tx=0).stop 2023-05-22T15:28:13.346 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.339+0000 7efdb9f11640 1 -- 172.21.15.16:0/1855946684 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efdb415dad0 msgr2=0x7efdb41541a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:13.346 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.339+0000 7efdb9f11640 1 --2- 172.21.15.16:0/1855946684 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efdb415dad0 0x7efdb41541a0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7efdac00c730 tx=0x7efdac00cbf0 comp rx=0 tx=0).stop 2023-05-22T15:28:13.347 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.339+0000 7efdb9f11640 1 -- 172.21.15.16:0/1855946684 shutdown_connections 2023-05-22T15:28:13.347 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.339+0000 7efdb9f11640 1 --2- 172.21.15.16:0/1855946684 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efdb415dad0 0x7efdb41541a0 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:13.348 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.339+0000 7efdb9f11640 1 --2- 172.21.15.16:0/1855946684 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efdb415d680 0x7efdb40cd590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:13.348 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.339+0000 7efdb9f11640 1 --2- 172.21.15.16:0/1855946684 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7efda0065380 0x7efda0067830 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:13.348 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.339+0000 7efdb9f11640 1 --2- 172.21.15.16:0/1855946684 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7efdb4159970 0x7efdb4155d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:13.348 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.339+0000 7efdb9f11640 1 -- 172.21.15.16:0/1855946684 >> 172.21.15.16:0/1855946684 conn(0x7efdb4069720 msgr2=0x7efdb40c1b40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:28:13.348 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.339+0000 7efdb9f11640 1 -- 172.21.15.16:0/1855946684 shutdown_connections 2023-05-22T15:28:13.348 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.339+0000 7efdb9f11640 1 -- 172.21.15.16:0/1855946684 wait complete. 2023-05-22T15:28:13.371 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-05-22T15:28:13.488 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.479+0000 7f0046df4640 1 Processor -- start 2023-05-22T15:28:13.489 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.479+0000 7f0046df4640 1 -- start start 2023-05-22T15:28:13.491 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.483+0000 7f0046df4640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f0040071640 0x7f0040160090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:13.492 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.483+0000 7f0046df4640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0040069ae0 con 0x7f00400ce480 2023-05-22T15:28:13.492 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.483+0000 7f0046df4640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0040069dc0 con 0x7f00401605d0 2023-05-22T15:28:13.492 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.483+0000 7f0046df4640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f004015f990 con 0x7f0040071640 2023-05-22T15:28:13.493 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.483+0000 7f0044b90640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f00400ce480 0x7f00400ce850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:39868/0 (socket says 172.21.15.16:39868) 2023-05-22T15:28:13.493 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.483+0000 7f0044b90640 1 -- 172.21.15.16:0/3123763085 learned_addr learned my addr 172.21.15.16:0/3123763085 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:28:13.493 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.483+0000 7f003ffff640 1 --2- 172.21.15.16:0/3123763085 >> v2:172.21.15.62:3300/0 conn(0x7f0040071640 0x7f0040160090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:13.493 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.483+0000 7f003f7fe640 1 -- 172.21.15.16:0/3123763085 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 802336128 0 0) 0x7f0040069ae0 con 0x7f00400ce480 2023-05-22T15:28:13.494 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.483+0000 7f003f7fe640 1 -- 172.21.15.16:0/3123763085 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0028003530 con 0x7f00400ce480 2023-05-22T15:28:13.494 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.483+0000 7f003f7fe640 1 -- 172.21.15.16:0/3123763085 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1188073904 0 0) 0x7f0028003530 con 0x7f00400ce480 2023-05-22T15:28:13.494 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.483+0000 7f003f7fe640 1 -- 172.21.15.16:0/3123763085 >> v2:172.21.15.62:3300/0 conn(0x7f0040071640 msgr2=0x7f0040160090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:28:13.495 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.483+0000 7f003f7fe640 1 --2- 172.21.15.16:0/3123763085 >> v2:172.21.15.62:3300/0 conn(0x7f0040071640 0x7f0040160090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:13.495 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.483+0000 7f003f7fe640 1 -- 172.21.15.16:0/3123763085 >> v1:172.21.15.62:6789/0 conn(0x7f00401605d0 legacy=0x7f004015c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:13.495 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.483+0000 7f003f7fe640 1 -- 172.21.15.16:0/3123763085 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f004006a0a0 con 0x7f00400ce480 2023-05-22T15:28:13.495 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.483+0000 7f003f7fe640 1 -- 172.21.15.16:0/3123763085 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7f0030001680 con 0x7f00400ce480 2023-05-22T15:28:13.496 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.483+0000 7f003f7fe640 1 -- 172.21.15.16:0/3123763085 >> v1:172.21.15.16:6789/0 conn(0x7f00400ce480 legacy=0x7f00400ce850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:13.496 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.483+0000 7f003f7fe640 1 --2- 172.21.15.16:0/3123763085 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0028003980 0x7f0028003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:13.496 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.483+0000 7f003f7fe640 1 --2- 172.21.15.16:0/3123763085 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f00280047d0 0x7f0028004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:13.497 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.483+0000 7f003f7fe640 1 --2- 172.21.15.16:0/3123763085 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0028005180 0x7f002800b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:13.497 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.483+0000 7f003f7fe640 1 -- 172.21.15.16:0/3123763085 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f004015f990 con 0x7f0028003980 2023-05-22T15:28:13.497 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.483+0000 7f003f7fe640 1 -- 172.21.15.16:0/3123763085 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f002800c000 con 0x7f00280047d0 2023-05-22T15:28:13.498 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.483+0000 7f003f7fe640 1 -- 172.21.15.16:0/3123763085 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f002800c1c0 con 0x7f0028005180 2023-05-22T15:28:13.498 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.483+0000 7f0044b90640 1 --2- 172.21.15.16:0/3123763085 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0028003980 0x7f0028003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:13.498 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.483+0000 7f003ffff640 1 --2- 172.21.15.16:0/3123763085 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f00280047d0 0x7f0028004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:13.499 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.483+0000 7f0045391640 1 --2- 172.21.15.16:0/3123763085 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0028005180 0x7f002800b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:13.499 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.483+0000 7f0044b90640 1 -- 172.21.15.16:0/3123763085 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0028005180 msgr2=0x7f002800b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:28:13.499 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.483+0000 7f0044b90640 1 --2- 172.21.15.16:0/3123763085 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0028005180 0x7f002800b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:13.499 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.483+0000 7f0044b90640 1 -- 172.21.15.16:0/3123763085 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f00280047d0 msgr2=0x7f0028004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:28:13.500 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.483+0000 7f0044b90640 1 --2- 172.21.15.16:0/3123763085 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f00280047d0 0x7f0028004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:13.500 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.483+0000 7f0045391640 1 -- 172.21.15.16:0/3123763085 reap_dead start 2023-05-22T15:28:13.500 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.483+0000 7f0044b90640 1 -- 172.21.15.16:0/3123763085 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0028003530 con 0x7f0028003980 2023-05-22T15:28:13.505 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.487+0000 7f0044b90640 1 --2- 172.21.15.16:0/3123763085 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0028003980 0x7f0028003d90 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f003000c240 tx=0x7f003001cbd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8c823e3543b9b96 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:28:13.506 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.487+0000 7f003f7fe640 1 -- 172.21.15.16:0/3123763085 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0030005ae0 con 0x7f0028003980 2023-05-22T15:28:13.506 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.487+0000 7f003f7fe640 1 -- 172.21.15.16:0/3123763085 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0030005c40 con 0x7f0028003980 2023-05-22T15:28:13.506 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.487+0000 7f003f7fe640 1 -- 172.21.15.16:0/3123763085 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0030007070 con 0x7f0028003980 2023-05-22T15:28:13.506 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.491+0000 7f0046df4640 1 -- 172.21.15.16:0/3123763085 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0028003980 msgr2=0x7f0028003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:13.507 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.491+0000 7f0046df4640 1 --2- 172.21.15.16:0/3123763085 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0028003980 0x7f0028003d90 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f003000c240 tx=0x7f003001cbd0 comp rx=0 tx=0).stop 2023-05-22T15:28:13.507 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.491+0000 7f0046df4640 1 -- 172.21.15.16:0/3123763085 shutdown_connections 2023-05-22T15:28:13.507 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.491+0000 7f0046df4640 1 --2- 172.21.15.16:0/3123763085 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0028003980 0x7f0028003d90 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:13.508 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.491+0000 7f0046df4640 1 -- 172.21.15.16:0/3123763085 >> 172.21.15.16:0/3123763085 conn(0x7f00400cc870 msgr2=0x7f00400cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:28:13.508 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.491+0000 7f0046df4640 1 -- 172.21.15.16:0/3123763085 shutdown_connections 2023-05-22T15:28:13.508 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.491+0000 7f0046df4640 1 -- 172.21.15.16:0/3123763085 wait complete. 2023-05-22T15:28:13.509 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.491+0000 7f0046df4640 1 Processor -- start 2023-05-22T15:28:13.509 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.491+0000 7f0046df4640 1 -- start start 2023-05-22T15:28:13.509 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.491+0000 7f0046df4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0028003980 0x7f0040152d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:13.510 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.491+0000 7f0046df4640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f00400ad670 0x7f00400adae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:13.510 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.491+0000 7f0046df4640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f004015d7b0 0x7f004015dc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:13.510 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.491+0000 7f0046df4640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0040071ac0 con 0x7f0028003980 2023-05-22T15:28:13.510 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.491+0000 7f0046df4640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00400ae610 con 0x7f004015d7b0 2023-05-22T15:28:13.511 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.491+0000 7f0046df4640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00400ac6c0 con 0x7f00400ad670 2023-05-22T15:28:13.511 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.491+0000 7f0044b90640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0028003980 0x7f0040152d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:13.511 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.491+0000 7f0044b90640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0028003980 0x7f0040152d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42702/0 (socket says 172.21.15.16:42702) 2023-05-22T15:28:13.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.491+0000 7f003ffff640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f00400ad670 0x7f00400adae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:13.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.491+0000 7f0044b90640 1 -- 172.21.15.16:0/3798348888 learned_addr learned my addr 172.21.15.16:0/3798348888 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:28:13.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.491+0000 7f0045391640 1 --2- 172.21.15.16:0/3798348888 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f004015d7b0 0x7f004015dc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:13.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.491+0000 7f0044b90640 1 -- 172.21.15.16:0/3798348888 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f00400ad670 msgr2=0x7f00400adae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:13.513 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.491+0000 7f0044b90640 1 --2- 172.21.15.16:0/3798348888 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f00400ad670 0x7f00400adae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:13.513 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.491+0000 7f0044b90640 1 -- 172.21.15.16:0/3798348888 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f004015d7b0 msgr2=0x7f004015dc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:13.513 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.491+0000 7f0044b90640 1 --2- 172.21.15.16:0/3798348888 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f004015d7b0 0x7f004015dc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:13.513 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.491+0000 7f0044b90640 1 -- 172.21.15.16:0/3798348888 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00400c6000 con 0x7f0028003980 2023-05-22T15:28:13.513 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.491+0000 7f0044b90640 1 --2- 172.21.15.16:0/3798348888 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0028003980 0x7f0040152d00 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f003000c240 tx=0x7f0028005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:28:13.514 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.491+0000 7f003d7fa640 1 -- 172.21.15.16:0/3798348888 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00300b4070 con 0x7f0028003980 2023-05-22T15:28:13.514 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.491+0000 7f003d7fa640 1 -- 172.21.15.16:0/3798348888 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0030005db0 con 0x7f0028003980 2023-05-22T15:28:13.514 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.491+0000 7f003d7fa640 1 -- 172.21.15.16:0/3798348888 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00300070a0 con 0x7f0028003980 2023-05-22T15:28:13.514 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.491+0000 7f0046df4640 1 -- 172.21.15.16:0/3798348888 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00400c5e70 con 0x7f0028003980 2023-05-22T15:28:13.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.491+0000 7f0046df4640 1 -- 172.21.15.16:0/3798348888 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00400cec60 con 0x7f0028003980 2023-05-22T15:28:13.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.495+0000 7f0046df4640 1 -- 172.21.15.16:0/3798348888 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f004004dbe0 con 0x7f0028003980 2023-05-22T15:28:13.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.495+0000 7f003d7fa640 1 -- 172.21.15.16:0/3798348888 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7f0030007200 con 0x7f0028003980 2023-05-22T15:28:13.516 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.495+0000 7f003d7fa640 1 --2- 172.21.15.16:0/3798348888 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f002c0653b0 0x7f002c067860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:13.516 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.495+0000 7f003d7fa640 1 -- 172.21.15.16:0/3798348888 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f003011c240 con 0x7f0028003980 2023-05-22T15:28:13.516 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.495+0000 7f003ffff640 1 --2- 172.21.15.16:0/3798348888 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f002c0653b0 0x7f002c067860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:13.516 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.495+0000 7f003ffff640 1 --2- 172.21.15.16:0/3798348888 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f002c0653b0 0x7f002c067860 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f0034009b00 tx=0x7f0034009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:28:13.517 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.499+0000 7f003d7fa640 1 -- 172.21.15.16:0/3798348888 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f00300e9cf0 con 0x7f0028003980 2023-05-22T15:28:13.660 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.651+0000 7f0046df4640 1 -- 172.21.15.16:0/3798348888 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f00400c9360 con 0x7f0028003980 2023-05-22T15:28:13.662 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.655+0000 7f003d7fa640 1 -- 172.21.15.16:0/3798348888 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v41) v1 ==== 74+0+17997 (secure 0 0 0) 0x7f00300ed310 con 0x7f0028003980 2023-05-22T15:28:13.663 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:28:13.663 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":41,"fsid":"b2719834-7847-4d98-90a6-f65d57f85c49","created":"2023-05-22T15:26:25.259314+0000","modified":"2023-05-22T15:28:13.319512+0000","last_up_change":"2023-05-22T15:26:36.448524+0000","last_in_change":"2023-05-22T15:26:27.675668+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-05-22T15:26:38.019781+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-05-22T15:26:43.570514+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2023-05-22T15:28:11.902605+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"41","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2023-05-22T15:28:12.298278+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"41","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9e7ff854-b25f-4cbb-a5c2-1799d9c53d2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":37021},{"type":"v1","addr":"172.21.15.16:6825","nonce":37021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":37021},{"type":"v1","addr":"172.21.15.16:6827","nonce":37021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":37021},{"type":"v1","addr":"172.21.15.16:6831","nonce":37021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":37021},{"type":"v1","addr":"172.21.15.16:6829","nonce":37021}]},"public_addr":"172.21.15.16:6825/37021","cluster_addr":"172.21.15.16:6827/37021","heartbeat_back_addr":"172.21.15.16:6831/37021","heartbeat_front_addr":"172.21.15.16:6829/37021","state":["exists","up"]},{"osd":1,"uuid":"4702fbc9-a927-4565-aca5-41dc69cc8264","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":37008},{"type":"v1","addr":"172.21.15.16:6801","nonce":37008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":37008},{"type":"v1","addr":"172.21.15.16:6803","nonce":37008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":37008},{"type":"v1","addr":"172.21.15.16:6807","nonce":37008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":37008},{"type":"v1","addr":"172.21.15.16:6805","nonce":37008}]},"public_addr":"172.21.15.16:6801/37008","cluster_addr":"172.21.15.16:6803/37008","heartbeat_back_addr":"172.21.15.16:6807/37008","heartbeat_front_addr":"172.21.15.16:6805/37008","state":["exists","up"]},{"osd":2,"uuid":"1e7bba46-4eba-47ef-bb81-5e2d371638ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":37009},{"type":"v1","addr":"172.21.15.16:6809","nonce":37009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":37009},{"type":"v1","addr":"172.21.15.16:6811","nonce":37009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":37009},{"type":"v1","addr":"172.21.15.16:6815","nonce":37009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":37009},{"type":"v1","addr":"172.21.15.16:6813","nonce":37009}]},"public_addr":"172.21.15.16:6809/37009","cluster_addr":"172.21.15.16:6811/37009","heartbeat_back_addr":"172.21.15.16:6815/37009","heartbeat_front_addr":"172.21.15.16:6813/37009","state":["exists","up"]},{"osd":3,"uuid":"383cacc0-5a88-4904-990e-498ca59ef48a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":37020},{"type":"v1","addr":"172.21.15.16:6817","nonce":37020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":37020},{"type":"v1","addr":"172.21.15.16:6819","nonce":37020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":37020},{"type":"v1","addr":"172.21.15.16:6823","nonce":37020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":37020},{"type":"v1","addr":"172.21.15.16:6821","nonce":37020}]},"public_addr":"172.21.15.16:6817/37020","cluster_addr":"172.21.15.16:6819/37020","heartbeat_back_addr":"172.21.15.16:6823/37020","heartbeat_front_addr":"172.21.15.16:6821/37020","state":["exists","up"]},{"osd":4,"uuid":"970500ef-95b7-42c1-b080-eed4fae7013d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":37353},{"type":"v1","addr":"172.21.15.62:6817","nonce":37353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6819","nonce":37353},{"type":"v1","addr":"172.21.15.62:6822","nonce":37353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37353},{"type":"v1","addr":"172.21.15.62:6831","nonce":37353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37353},{"type":"v1","addr":"172.21.15.62:6826","nonce":37353}]},"public_addr":"172.21.15.62:6817/37353","cluster_addr":"172.21.15.62:6822/37353","heartbeat_back_addr":"172.21.15.62:6831/37353","heartbeat_front_addr":"172.21.15.62:6826/37353","state":["exists","up"]},{"osd":5,"uuid":"022d67db-d4d8-445b-b1b9-8b446a00f5c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":37352},{"type":"v1","addr":"172.21.15.62:6801","nonce":37352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":37352},{"type":"v1","addr":"172.21.15.62:6803","nonce":37352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":37352},{"type":"v1","addr":"172.21.15.62:6807","nonce":37352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":37352},{"type":"v1","addr":"172.21.15.62:6805","nonce":37352}]},"public_addr":"172.21.15.62:6801/37352","cluster_addr":"172.21.15.62:6803/37352","heartbeat_back_addr":"172.21.15.62:6807/37352","heartbeat_front_addr":"172.21.15.62:6805/37352","state":["exists","up"]},{"osd":6,"uuid":"42bd372b-6688-4a1b-8407-e16e8c266e33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":37354},{"type":"v1","addr":"172.21.15.62:6820","nonce":37354}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":37354},{"type":"v1","addr":"172.21.15.62:6823","nonce":37354}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6829","nonce":37354},{"type":"v1","addr":"172.21.15.62:6830","nonce":37354}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6825","nonce":37354},{"type":"v1","addr":"172.21.15.62:6827","nonce":37354}]},"public_addr":"172.21.15.62:6820/37354","cluster_addr":"172.21.15.62:6823/37354","heartbeat_back_addr":"172.21.15.62:6830/37354","heartbeat_front_addr":"172.21.15.62:6827/37354","state":["exists","up"]},{"osd":7,"uuid":"ac70b2a3-f41b-4bf2-b227-7f2cbb55426d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37355},{"type":"v1","addr":"172.21.15.62:6809","nonce":37355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37355},{"type":"v1","addr":"172.21.15.62:6811","nonce":37355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":37355},{"type":"v1","addr":"172.21.15.62:6815","nonce":37355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":37355},{"type":"v1","addr":"172.21.15.62:6813","nonce":37355}]},"public_addr":"172.21.15.62:6809/37355","cluster_addr":"172.21.15.62:6811/37355","heartbeat_back_addr":"172.21.15.62:6815/37355","heartbeat_front_addr":"172.21.15.62:6813/37355","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.368896+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.107273+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.276888+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.287502+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.124791+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.251937+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.255679+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.232797+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-05-22T15:28:13.664 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.655+0000 7f0046df4640 1 -- 172.21.15.16:0/3798348888 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f002c0653b0 msgr2=0x7f002c067860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:13.664 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.655+0000 7f0046df4640 1 --2- 172.21.15.16:0/3798348888 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f002c0653b0 0x7f002c067860 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f0034009b00 tx=0x7f0034009190 comp rx=0 tx=0).stop 2023-05-22T15:28:13.664 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.655+0000 7f0046df4640 1 -- 172.21.15.16:0/3798348888 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0028003980 msgr2=0x7f0040152d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:13.664 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.655+0000 7f0046df4640 1 --2- 172.21.15.16:0/3798348888 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0028003980 0x7f0040152d00 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f003000c240 tx=0x7f0028005800 comp rx=0 tx=0).stop 2023-05-22T15:28:13.664 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.655+0000 7f0046df4640 1 -- 172.21.15.16:0/3798348888 shutdown_connections 2023-05-22T15:28:13.665 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.655+0000 7f0046df4640 1 --2- 172.21.15.16:0/3798348888 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f004015d7b0 0x7f004015dc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:13.665 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.655+0000 7f0046df4640 1 --2- 172.21.15.16:0/3798348888 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f002c0653b0 0x7f002c067860 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:13.665 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.655+0000 7f0046df4640 1 --2- 172.21.15.16:0/3798348888 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f00400ad670 0x7f00400adae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:13.666 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.655+0000 7f0046df4640 1 --2- 172.21.15.16:0/3798348888 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0028003980 0x7f0040152d00 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:13.666 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.655+0000 7f0046df4640 1 -- 172.21.15.16:0/3798348888 >> 172.21.15.16:0/3798348888 conn(0x7f00400cc870 msgr2=0x7f00400ca7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:28:13.666 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.655+0000 7f0046df4640 1 -- 172.21.15.16:0/3798348888 shutdown_connections 2023-05-22T15:28:13.666 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.655+0000 7f0046df4640 1 -- 172.21.15.16:0/3798348888 wait complete. 2023-05-22T15:28:13.683 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-05-22T15:28:13.809 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.803+0000 7f12c8ce8640 1 Processor -- start 2023-05-22T15:28:13.809 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.803+0000 7f12c8ce8640 1 -- start start 2023-05-22T15:28:13.812 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.803+0000 7f12c8ce8640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f12c40d1420 0x7f12c4166880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:13.813 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.803+0000 7f12c8ce8640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12c40ccef0 con 0x7f12c40ccab0 2023-05-22T15:28:13.813 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.803+0000 7f12c8ce8640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12c4160520 con 0x7f12c40b7dd0 2023-05-22T15:28:13.813 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.803+0000 7f12c8ce8640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12c40d3360 con 0x7f12c40d1420 2023-05-22T15:28:13.813 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.803+0000 7f12c1d9b640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f12c40ccab0 0x7f12c415fe10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:39882/0 (socket says 172.21.15.16:39882) 2023-05-22T15:28:13.814 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.803+0000 7f12c1d9b640 1 -- 172.21.15.16:0/681170993 learned_addr learned my addr 172.21.15.16:0/681170993 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:28:13.814 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.803+0000 7f12c2d9d640 1 --2- 172.21.15.16:0/681170993 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f12c40d1420 0x7f12c4166880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:13.814 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.803+0000 7f12c159a640 1 -- 172.21.15.16:0/681170993 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2190282787 0 0) 0x7f12c40ccef0 con 0x7f12c40ccab0 2023-05-22T15:28:13.814 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.803+0000 7f12c159a640 1 -- 172.21.15.16:0/681170993 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f12ac003530 con 0x7f12c40ccab0 2023-05-22T15:28:13.815 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.803+0000 7f12c159a640 1 -- 172.21.15.16:0/681170993 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2696546084 0 0) 0x7f12ac003530 con 0x7f12c40ccab0 2023-05-22T15:28:13.815 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.803+0000 7f12c159a640 1 -- 172.21.15.16:0/681170993 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f12c40d1420 msgr2=0x7f12c4166880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:28:13.815 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.803+0000 7f12c159a640 1 --2- 172.21.15.16:0/681170993 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f12c40d1420 0x7f12c4166880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:13.816 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.803+0000 7f12c159a640 1 -- 172.21.15.16:0/681170993 >> v1:172.21.15.62:6789/0 conn(0x7f12c40b7dd0 legacy=0x7f12c415f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:13.816 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.803+0000 7f12c159a640 1 -- 172.21.15.16:0/681170993 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12c40cd170 con 0x7f12c40ccab0 2023-05-22T15:28:13.816 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.803+0000 7f12c159a640 1 -- 172.21.15.16:0/681170993 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7f12b8001680 con 0x7f12c40ccab0 2023-05-22T15:28:13.816 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.803+0000 7f12c159a640 1 -- 172.21.15.16:0/681170993 >> v1:172.21.15.16:6789/0 conn(0x7f12c40ccab0 legacy=0x7f12c415fe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:13.817 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.803+0000 7f12c159a640 1 --2- 172.21.15.16:0/681170993 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f12c40d1420 0x7f12ac0039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:13.817 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.803+0000 7f12c159a640 1 --2- 172.21.15.16:0/681170993 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f12ac003f30 0x7f12ac0047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:13.817 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.803+0000 7f12c159a640 1 --2- 172.21.15.16:0/681170993 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f12ac004d10 0x7f12ac005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:13.818 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.803+0000 7f12c159a640 1 -- 172.21.15.16:0/681170993 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12c40d3360 con 0x7f12ac003f30 2023-05-22T15:28:13.819 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.803+0000 7f12c159a640 1 -- 172.21.15.16:0/681170993 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12ac00ba70 con 0x7f12ac004d10 2023-05-22T15:28:13.819 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.803+0000 7f12c159a640 1 -- 172.21.15.16:0/681170993 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12ac00bc30 con 0x7f12c40d1420 2023-05-22T15:28:13.819 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.803+0000 7f12c1d9b640 1 --2- 172.21.15.16:0/681170993 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f12ac003f30 0x7f12ac0047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:13.819 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.803+0000 7f12c259c640 1 --2- 172.21.15.16:0/681170993 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f12c40d1420 0x7f12ac0039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:13.819 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.803+0000 7f12c1d9b640 1 -- 172.21.15.16:0/681170993 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f12c40d1420 msgr2=0x7f12ac0039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:28:13.819 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.803+0000 7f12c2d9d640 1 --2- 172.21.15.16:0/681170993 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f12ac004d10 0x7f12ac005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:13.820 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.803+0000 7f12c1d9b640 1 --2- 172.21.15.16:0/681170993 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f12c40d1420 0x7f12ac0039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:13.820 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.803+0000 7f12c1d9b640 1 -- 172.21.15.16:0/681170993 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f12ac004d10 msgr2=0x7f12ac005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:28:13.820 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.803+0000 7f12c1d9b640 1 --2- 172.21.15.16:0/681170993 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f12ac004d10 0x7f12ac005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:13.820 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.803+0000 7f12c1d9b640 1 -- 172.21.15.16:0/681170993 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12ac003530 con 0x7f12ac003f30 2023-05-22T15:28:13.820 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.803+0000 7f12c2d9d640 1 --2- 172.21.15.16:0/681170993 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f12ac004d10 0x7f12ac005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:28:13.823 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.807+0000 7f12c1d9b640 1 --2- 172.21.15.16:0/681170993 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f12ac003f30 0x7f12ac0047d0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f12b800c210 tx=0x7f12b801f370 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13da567bf75ec9dc server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:28:13.823 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.807+0000 7f12c159a640 1 -- 172.21.15.16:0/681170993 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12b8005ae0 con 0x7f12ac003f30 2023-05-22T15:28:13.823 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.807+0000 7f12c159a640 1 -- 172.21.15.16:0/681170993 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12b80ab070 con 0x7f12ac003f30 2023-05-22T15:28:13.824 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.807+0000 7f12c159a640 1 -- 172.21.15.16:0/681170993 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12b8005ea0 con 0x7f12ac003f30 2023-05-22T15:28:13.824 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.807+0000 7f12c8ce8640 1 -- 172.21.15.16:0/681170993 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f12ac003f30 msgr2=0x7f12ac0047d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:13.824 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.807+0000 7f12c8ce8640 1 --2- 172.21.15.16:0/681170993 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f12ac003f30 0x7f12ac0047d0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f12b800c210 tx=0x7f12b801f370 comp rx=0 tx=0).stop 2023-05-22T15:28:13.824 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.807+0000 7f12c2d9d640 1 -- 172.21.15.16:0/681170993 reap_dead start 2023-05-22T15:28:13.824 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.807+0000 7f12c8ce8640 1 -- 172.21.15.16:0/681170993 shutdown_connections 2023-05-22T15:28:13.824 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.807+0000 7f12c8ce8640 1 -- 172.21.15.16:0/681170993 >> 172.21.15.16:0/681170993 conn(0x7f12c40655c0 msgr2=0x7f12c4070e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:28:13.824 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.807+0000 7f12c8ce8640 1 -- 172.21.15.16:0/681170993 shutdown_connections 2023-05-22T15:28:13.825 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.807+0000 7f12c8ce8640 1 -- 172.21.15.16:0/681170993 wait complete. 2023-05-22T15:28:13.825 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.807+0000 7f12c8ce8640 1 Processor -- start 2023-05-22T15:28:13.825 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.807+0000 7f12c8ce8640 1 -- start start 2023-05-22T15:28:13.825 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.811+0000 7f12c8ce8640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f12ac003f30 0x7f12c415da00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:13.826 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.811+0000 7f12c8ce8640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f12c41f4f10 0x7f12c4151970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:13.826 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.811+0000 7f12c8ce8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f12c40c8410 0x7f12c40c8880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:13.826 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.811+0000 7f12c8ce8640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12c40c9f70 con 0x7f12c40c8410 2023-05-22T15:28:13.827 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.811+0000 7f12c8ce8640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12c40c8dc0 con 0x7f12ac003f30 2023-05-22T15:28:13.827 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.811+0000 7f12c8ce8640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12c40c6180 con 0x7f12c41f4f10 2023-05-22T15:28:13.827 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.811+0000 7f12c2d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f12c40c8410 0x7f12c40c8880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:13.827 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.811+0000 7f12c2d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f12c40c8410 0x7f12c40c8880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42728/0 (socket says 172.21.15.16:42728) 2023-05-22T15:28:13.827 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.811+0000 7f12c2d9d640 1 -- 172.21.15.16:0/170988033 learned_addr learned my addr 172.21.15.16:0/170988033 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:28:13.827 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.811+0000 7f12c259c640 1 --2- 172.21.15.16:0/170988033 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f12ac003f30 0x7f12c415da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:13.827 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.811+0000 7f12c1d9b640 1 --2- 172.21.15.16:0/170988033 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f12c41f4f10 0x7f12c4151970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:13.828 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.811+0000 7f12c2d9d640 1 -- 172.21.15.16:0/170988033 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f12c41f4f10 msgr2=0x7f12c4151970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:13.828 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.811+0000 7f12c2d9d640 1 --2- 172.21.15.16:0/170988033 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f12c41f4f10 0x7f12c4151970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:13.828 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.811+0000 7f12c2d9d640 1 -- 172.21.15.16:0/170988033 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f12ac003f30 msgr2=0x7f12c415da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:13.828 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.811+0000 7f12c2d9d640 1 --2- 172.21.15.16:0/170988033 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f12ac003f30 0x7f12c415da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:13.828 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.811+0000 7f12c2d9d640 1 -- 172.21.15.16:0/170988033 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12c40d2da0 con 0x7f12c40c8410 2023-05-22T15:28:13.828 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.811+0000 7f12c1d9b640 1 --2- 172.21.15.16:0/170988033 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f12c41f4f10 0x7f12c4151970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:28:13.828 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.811+0000 7f12c2d9d640 1 --2- 172.21.15.16:0/170988033 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f12c40c8410 0x7f12c40c8880 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f12c41607a0 tx=0x7f12bc010850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:28:13.829 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.811+0000 7f12b2ffd640 1 -- 172.21.15.16:0/170988033 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12bc010e90 con 0x7f12c40c8410 2023-05-22T15:28:13.829 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.811+0000 7f12b2ffd640 1 -- 172.21.15.16:0/170988033 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12bc004020 con 0x7f12c40c8410 2023-05-22T15:28:13.829 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.811+0000 7f12c8ce8640 1 -- 172.21.15.16:0/170988033 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12c40d2ac0 con 0x7f12c40c8410 2023-05-22T15:28:13.829 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.811+0000 7f12b2ffd640 1 -- 172.21.15.16:0/170988033 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12bc0043d0 con 0x7f12c40c8410 2023-05-22T15:28:13.829 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.811+0000 7f12c8ce8640 1 -- 172.21.15.16:0/170988033 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12c40d3080 con 0x7f12c40c8410 2023-05-22T15:28:13.829 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.811+0000 7f12c8ce8640 1 -- 172.21.15.16:0/170988033 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12c404dbe0 con 0x7f12c40c8410 2023-05-22T15:28:13.829 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.811+0000 7f12b2ffd640 1 -- 172.21.15.16:0/170988033 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7f12bc003620 con 0x7f12c40c8410 2023-05-22T15:28:13.830 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.811+0000 7f12b2ffd640 1 --2- 172.21.15.16:0/170988033 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f12a4065380 0x7f12a4067830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:13.830 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.811+0000 7f12b2ffd640 1 -- 172.21.15.16:0/170988033 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f12bc081f80 con 0x7f12c40c8410 2023-05-22T15:28:13.830 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.811+0000 7f12c259c640 1 --2- 172.21.15.16:0/170988033 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f12a4065380 0x7f12a4067830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:13.830 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.811+0000 7f12c259c640 1 --2- 172.21.15.16:0/170988033 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f12a4065380 0x7f12a4067830 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f12b4005210 tx=0x7f12b4009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:28:13.831 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.815+0000 7f12b2ffd640 1 -- 172.21.15.16:0/170988033 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f12bc0b2620 con 0x7f12c40c8410 2023-05-22T15:28:13.978 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.971+0000 7f12c8ce8640 1 -- 172.21.15.16:0/170988033 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f12c40c5f50 con 0x7f12c40c8410 2023-05-22T15:28:13.980 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.971+0000 7f12b2ffd640 1 -- 172.21.15.16:0/170988033 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v41) v1 ==== 74+0+17997 (secure 0 0 0) 0x7f12bc04fb50 con 0x7f12c40c8410 2023-05-22T15:28:13.980 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:28:13.981 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":41,"fsid":"b2719834-7847-4d98-90a6-f65d57f85c49","created":"2023-05-22T15:26:25.259314+0000","modified":"2023-05-22T15:28:13.319512+0000","last_up_change":"2023-05-22T15:26:36.448524+0000","last_in_change":"2023-05-22T15:26:27.675668+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-05-22T15:26:38.019781+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-05-22T15:26:43.570514+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2023-05-22T15:28:11.902605+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"41","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2023-05-22T15:28:12.298278+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"41","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9e7ff854-b25f-4cbb-a5c2-1799d9c53d2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":37021},{"type":"v1","addr":"172.21.15.16:6825","nonce":37021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":37021},{"type":"v1","addr":"172.21.15.16:6827","nonce":37021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":37021},{"type":"v1","addr":"172.21.15.16:6831","nonce":37021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":37021},{"type":"v1","addr":"172.21.15.16:6829","nonce":37021}]},"public_addr":"172.21.15.16:6825/37021","cluster_addr":"172.21.15.16:6827/37021","heartbeat_back_addr":"172.21.15.16:6831/37021","heartbeat_front_addr":"172.21.15.16:6829/37021","state":["exists","up"]},{"osd":1,"uuid":"4702fbc9-a927-4565-aca5-41dc69cc8264","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":37008},{"type":"v1","addr":"172.21.15.16:6801","nonce":37008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":37008},{"type":"v1","addr":"172.21.15.16:6803","nonce":37008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":37008},{"type":"v1","addr":"172.21.15.16:6807","nonce":37008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":37008},{"type":"v1","addr":"172.21.15.16:6805","nonce":37008}]},"public_addr":"172.21.15.16:6801/37008","cluster_addr":"172.21.15.16:6803/37008","heartbeat_back_addr":"172.21.15.16:6807/37008","heartbeat_front_addr":"172.21.15.16:6805/37008","state":["exists","up"]},{"osd":2,"uuid":"1e7bba46-4eba-47ef-bb81-5e2d371638ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":37009},{"type":"v1","addr":"172.21.15.16:6809","nonce":37009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":37009},{"type":"v1","addr":"172.21.15.16:6811","nonce":37009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":37009},{"type":"v1","addr":"172.21.15.16:6815","nonce":37009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":37009},{"type":"v1","addr":"172.21.15.16:6813","nonce":37009}]},"public_addr":"172.21.15.16:6809/37009","cluster_addr":"172.21.15.16:6811/37009","heartbeat_back_addr":"172.21.15.16:6815/37009","heartbeat_front_addr":"172.21.15.16:6813/37009","state":["exists","up"]},{"osd":3,"uuid":"383cacc0-5a88-4904-990e-498ca59ef48a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":37020},{"type":"v1","addr":"172.21.15.16:6817","nonce":37020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":37020},{"type":"v1","addr":"172.21.15.16:6819","nonce":37020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":37020},{"type":"v1","addr":"172.21.15.16:6823","nonce":37020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":37020},{"type":"v1","addr":"172.21.15.16:6821","nonce":37020}]},"public_addr":"172.21.15.16:6817/37020","cluster_addr":"172.21.15.16:6819/37020","heartbeat_back_addr":"172.21.15.16:6823/37020","heartbeat_front_addr":"172.21.15.16:6821/37020","state":["exists","up"]},{"osd":4,"uuid":"970500ef-95b7-42c1-b080-eed4fae7013d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":37353},{"type":"v1","addr":"172.21.15.62:6817","nonce":37353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6819","nonce":37353},{"type":"v1","addr":"172.21.15.62:6822","nonce":37353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37353},{"type":"v1","addr":"172.21.15.62:6831","nonce":37353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37353},{"type":"v1","addr":"172.21.15.62:6826","nonce":37353}]},"public_addr":"172.21.15.62:6817/37353","cluster_addr":"172.21.15.62:6822/37353","heartbeat_back_addr":"172.21.15.62:6831/37353","heartbeat_front_addr":"172.21.15.62:6826/37353","state":["exists","up"]},{"osd":5,"uuid":"022d67db-d4d8-445b-b1b9-8b446a00f5c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":37352},{"type":"v1","addr":"172.21.15.62:6801","nonce":37352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":37352},{"type":"v1","addr":"172.21.15.62:6803","nonce":37352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":37352},{"type":"v1","addr":"172.21.15.62:6807","nonce":37352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":37352},{"type":"v1","addr":"172.21.15.62:6805","nonce":37352}]},"public_addr":"172.21.15.62:6801/37352","cluster_addr":"172.21.15.62:6803/37352","heartbeat_back_addr":"172.21.15.62:6807/37352","heartbeat_front_addr":"172.21.15.62:6805/37352","state":["exists","up"]},{"osd":6,"uuid":"42bd372b-6688-4a1b-8407-e16e8c266e33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":37354},{"type":"v1","addr":"172.21.15.62:6820","nonce":37354}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":37354},{"type":"v1","addr":"172.21.15.62:6823","nonce":37354}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6829","nonce":37354},{"type":"v1","addr":"172.21.15.62:6830","nonce":37354}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6825","nonce":37354},{"type":"v1","addr":"172.21.15.62:6827","nonce":37354}]},"public_addr":"172.21.15.62:6820/37354","cluster_addr":"172.21.15.62:6823/37354","heartbeat_back_addr":"172.21.15.62:6830/37354","heartbeat_front_addr":"172.21.15.62:6827/37354","state":["exists","up"]},{"osd":7,"uuid":"ac70b2a3-f41b-4bf2-b227-7f2cbb55426d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37355},{"type":"v1","addr":"172.21.15.62:6809","nonce":37355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37355},{"type":"v1","addr":"172.21.15.62:6811","nonce":37355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":37355},{"type":"v1","addr":"172.21.15.62:6815","nonce":37355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":37355},{"type":"v1","addr":"172.21.15.62:6813","nonce":37355}]},"public_addr":"172.21.15.62:6809/37355","cluster_addr":"172.21.15.62:6811/37355","heartbeat_back_addr":"172.21.15.62:6815/37355","heartbeat_front_addr":"172.21.15.62:6813/37355","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.368896+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.107273+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.276888+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.287502+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.124791+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.251937+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.255679+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.232797+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-05-22T15:28:13.982 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.975+0000 7f12c8ce8640 1 -- 172.21.15.16:0/170988033 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f12a4065380 msgr2=0x7f12a4067830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:13.982 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.975+0000 7f12c8ce8640 1 --2- 172.21.15.16:0/170988033 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f12a4065380 0x7f12a4067830 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f12b4005210 tx=0x7f12b4009040 comp rx=0 tx=0).stop 2023-05-22T15:28:13.983 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.975+0000 7f12c8ce8640 1 -- 172.21.15.16:0/170988033 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f12c40c8410 msgr2=0x7f12c40c8880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:13.983 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.975+0000 7f12c8ce8640 1 --2- 172.21.15.16:0/170988033 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f12c40c8410 0x7f12c40c8880 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f12c41607a0 tx=0x7f12bc010850 comp rx=0 tx=0).stop 2023-05-22T15:28:13.983 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.975+0000 7f12c8ce8640 1 -- 172.21.15.16:0/170988033 shutdown_connections 2023-05-22T15:28:13.983 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.975+0000 7f12c8ce8640 1 --2- 172.21.15.16:0/170988033 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f12c40c8410 0x7f12c40c8880 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:13.983 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.975+0000 7f12c8ce8640 1 --2- 172.21.15.16:0/170988033 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f12a4065380 0x7f12a4067830 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:13.983 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.975+0000 7f12c8ce8640 1 --2- 172.21.15.16:0/170988033 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f12c41f4f10 0x7f12c4151970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:13.983 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.975+0000 7f12c8ce8640 1 --2- 172.21.15.16:0/170988033 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f12ac003f30 0x7f12c415da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:13.984 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.975+0000 7f12c8ce8640 1 -- 172.21.15.16:0/170988033 >> 172.21.15.16:0/170988033 conn(0x7f12c40655c0 msgr2=0x7f12c40cee20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:28:13.984 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.975+0000 7f12c8ce8640 1 -- 172.21.15.16:0/170988033 shutdown_connections 2023-05-22T15:28:13.984 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:13.975+0000 7f12c8ce8640 1 -- 172.21.15.16:0/170988033 wait complete. 2023-05-22T15:28:14.001 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-05-22T15:28:14.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:14.119+0000 7f129875f640 1 Processor -- start 2023-05-22T15:28:14.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:14.119+0000 7f129875f640 1 -- start start 2023-05-22T15:28:14.126 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:14.119+0000 7f129875f640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f1290159970 0x7f1290159d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:14.126 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:14.119+0000 7f129875f640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1290070c30 con 0x7f129015db00 2023-05-22T15:28:14.126 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:14.119+0000 7f129875f640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1290070e60 con 0x7f129015d680 2023-05-22T15:28:14.126 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:14.119+0000 7f129875f640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12900cb970 con 0x7f1290159970 2023-05-22T15:28:14.126 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:14.119+0000 7f1296cfc640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f129015db00 0x7f129006b6c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:39896/0 (socket says 172.21.15.16:39896) 2023-05-22T15:28:14.127 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:14.119+0000 7f1296cfc640 1 -- 172.21.15.16:0/1992818164 learned_addr learned my addr 172.21.15.16:0/1992818164 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:28:14.128 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:14.119+0000 7f12964fb640 1 --2- 172.21.15.16:0/1992818164 >> v2:172.21.15.62:3300/0 conn(0x7f1290159970 0x7f1290159d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:14.128 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:14.119+0000 7f12954f9640 1 -- 172.21.15.16:0/1992818164 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3526382268 0 0) 0x7f1290070c30 con 0x7f129015db00 2023-05-22T15:28:14.128 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:14.119+0000 7f12954f9640 1 -- 172.21.15.16:0/1992818164 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f127c003530 con 0x7f129015db00 2023-05-22T15:28:14.128 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:14.119+0000 7f12954f9640 1 -- 172.21.15.16:0/1992818164 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3539732897 0 0) 0x7f127c003530 con 0x7f129015db00 2023-05-22T15:28:14.129 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:14.119+0000 7f12954f9640 1 -- 172.21.15.16:0/1992818164 >> v2:172.21.15.62:3300/0 conn(0x7f1290159970 msgr2=0x7f1290159d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:28:14.129 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:14.119+0000 7f12954f9640 1 --2- 172.21.15.16:0/1992818164 >> v2:172.21.15.62:3300/0 conn(0x7f1290159970 0x7f1290159d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:14.129 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:14.119+0000 7f12954f9640 1 -- 172.21.15.16:0/1992818164 >> v1:172.21.15.62:6789/0 conn(0x7f129015d680 legacy=0x7f12900cb260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:14.129 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:14.119+0000 7f12954f9640 1 -- 172.21.15.16:0/1992818164 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1290071140 con 0x7f129015db00 2023-05-22T15:28:14.129 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:14.119+0000 7f12954f9640 1 -- 172.21.15.16:0/1992818164 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7f1288001a00 con 0x7f129015db00 2023-05-22T15:28:14.129 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:14.119+0000 7f12954f9640 1 -- 172.21.15.16:0/1992818164 >> v1:172.21.15.16:6789/0 conn(0x7f129015db00 legacy=0x7f129006b6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:14.129 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:14.119+0000 7f12954f9640 1 --2- 172.21.15.16:0/1992818164 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f127c003980 0x7f127c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:14.130 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:14.119+0000 7f12954f9640 1 --2- 172.21.15.16:0/1992818164 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f127c0047d0 0x7f127c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:14.130 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:14.119+0000 7f12954f9640 1 --2- 172.21.15.16:0/1992818164 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f127c005180 0x7f127c00b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:14.130 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:14.119+0000 7f12954f9640 1 -- 172.21.15.16:0/1992818164 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12900cb970 con 0x7f127c005180 2023-05-22T15:28:14.130 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:14.119+0000 7f12954f9640 1 -- 172.21.15.16:0/1992818164 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f127c00c090 con 0x7f127c0047d0 2023-05-22T15:28:14.130 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:14.119+0000 7f12954f9640 1 -- 172.21.15.16:0/1992818164 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f127c00c250 con 0x7f127c003980 2023-05-22T15:28:14.130 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:14.119+0000 7f1296cfc640 1 --2- 172.21.15.16:0/1992818164 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f127c005180 0x7f127c00b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:14.130 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:14.119+0000 7f12964fb640 1 --2- 172.21.15.16:0/1992818164 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f127c003980 0x7f127c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:14.130 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:14.119+0000 7f1295cfa640 1 --2- 172.21.15.16:0/1992818164 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f127c0047d0 0x7f127c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:14.131 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:14.119+0000 7f1296cfc640 1 -- 172.21.15.16:0/1992818164 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f127c003980 msgr2=0x7f127c003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:28:14.131 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:14.119+0000 7f1296cfc640 1 --2- 172.21.15.16:0/1992818164 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f127c003980 0x7f127c003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:14.131 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:14.119+0000 7f1296cfc640 1 -- 172.21.15.16:0/1992818164 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f127c0047d0 msgr2=0x7f127c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:28:14.131 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:14.119+0000 7f1296cfc640 1 --2- 172.21.15.16:0/1992818164 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f127c0047d0 0x7f127c004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:14.131 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:14.119+0000 7f1296cfc640 1 -- 172.21.15.16:0/1992818164 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f127c003530 con 0x7f127c005180 2023-05-22T15:28:14.131 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:14.119+0000 7f1295cfa640 1 --2- 172.21.15.16:0/1992818164 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f127c0047d0 0x7f127c004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:28:14.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:14.123+0000 7f1296cfc640 1 -- 172.21.15.16:0/1992818164 reap_dead start 2023-05-22T15:28:14.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:14.123+0000 7f1296cfc640 1 --2- 172.21.15.16:0/1992818164 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f127c005180 0x7f127c00b990 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f128800a830 tx=0x7f128801faa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58027fb9b8167aa4 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:28:14.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:14.123+0000 7f12954f9640 1 -- 172.21.15.16:0/1992818164 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12880aa6d0 con 0x7f127c005180 2023-05-22T15:28:14.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:14.123+0000 7f12954f9640 1 -- 172.21.15.16:0/1992818164 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12880aa870 con 0x7f127c005180 2023-05-22T15:28:14.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:14.123+0000 7f12954f9640 1 -- 172.21.15.16:0/1992818164 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12880aac20 con 0x7f127c005180 2023-05-22T15:28:14.134 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:14.123+0000 7f129875f640 1 -- 172.21.15.16:0/1992818164 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f127c005180 msgr2=0x7f127c00b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:14.134 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:14.123+0000 7f129875f640 1 --2- 172.21.15.16:0/1992818164 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f127c005180 0x7f127c00b990 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f128800a830 tx=0x7f128801faa0 comp rx=0 tx=0).stop 2023-05-22T15:28:14.134 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:14.123+0000 7f129875f640 1 -- 172.21.15.16:0/1992818164 shutdown_connections 2023-05-22T15:28:14.134 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:14.123+0000 7f129875f640 1 --2- 172.21.15.16:0/1992818164 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f127c005180 0x7f127c00b990 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:14.134 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:14.123+0000 7f129875f640 1 -- 172.21.15.16:0/1992818164 >> 172.21.15.16:0/1992818164 conn(0x7f1290069720 msgr2=0x7f1290069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:28:14.134 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:14.123+0000 7f129875f640 1 -- 172.21.15.16:0/1992818164 shutdown_connections 2023-05-22T15:28:14.134 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:14.123+0000 7f129875f640 1 -- 172.21.15.16:0/1992818164 wait complete. 2023-05-22T15:28:14.135 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:14.123+0000 7f129875f640 1 Processor -- start 2023-05-22T15:28:14.135 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:14.123+0000 7f129875f640 1 -- start start 2023-05-22T15:28:14.135 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:14.127+0000 7f129875f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f127c005180 0x7f12901e4690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:14.135 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:14.127+0000 7f129875f640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1290149db0 0x7f12900b2f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:14.135 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:14.127+0000 7f129875f640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f12900b3440 0x7f1290152e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:14.135 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:14.127+0000 7f129875f640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12901e4200 con 0x7f127c005180 2023-05-22T15:28:14.135 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:14.127+0000 7f129875f640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12900c3100 con 0x7f1290149db0 2023-05-22T15:28:14.135 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:14.127+0000 7f129875f640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12901e35f0 con 0x7f12900b3440 2023-05-22T15:28:14.136 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:14.127+0000 7f12964fb640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f127c005180 0x7f12901e4690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:14.136 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:14.127+0000 7f12964fb640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f127c005180 0x7f12901e4690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42758/0 (socket says 172.21.15.16:42758) 2023-05-22T15:28:14.136 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:14.127+0000 7f12964fb640 1 -- 172.21.15.16:0/2806460613 learned_addr learned my addr 172.21.15.16:0/2806460613 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:28:14.136 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:14.127+0000 7f1295cfa640 1 --2- 172.21.15.16:0/2806460613 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1290149db0 0x7f12900b2f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:14.136 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:14.127+0000 7f1296cfc640 1 --2- 172.21.15.16:0/2806460613 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f12900b3440 0x7f1290152e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:14.137 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:14.127+0000 7f12964fb640 1 -- 172.21.15.16:0/2806460613 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f12900b3440 msgr2=0x7f1290152e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:14.138 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:14.127+0000 7f12964fb640 1 --2- 172.21.15.16:0/2806460613 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f12900b3440 0x7f1290152e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:14.138 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:14.127+0000 7f12964fb640 1 -- 172.21.15.16:0/2806460613 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1290149db0 msgr2=0x7f12900b2f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:14.138 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:14.127+0000 7f12964fb640 1 --2- 172.21.15.16:0/2806460613 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1290149db0 0x7f12900b2f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:14.138 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:14.127+0000 7f12964fb640 1 -- 172.21.15.16:0/2806460613 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1290071700 con 0x7f127c005180 2023-05-22T15:28:14.138 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:14.127+0000 7f12964fb640 1 --2- 172.21.15.16:0/2806460613 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f127c005180 0x7f12901e4690 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f128c011560 tx=0x7f128c011a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:28:14.138 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:14.127+0000 7f1286ffd640 1 -- 172.21.15.16:0/2806460613 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f128c015870 con 0x7f127c005180 2023-05-22T15:28:14.138 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:14.127+0000 7f1286ffd640 1 -- 172.21.15.16:0/2806460613 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f128c0159d0 con 0x7f127c005180 2023-05-22T15:28:14.139 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:14.127+0000 7f129875f640 1 -- 172.21.15.16:0/2806460613 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1290071420 con 0x7f127c005180 2023-05-22T15:28:14.139 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:14.127+0000 7f1286ffd640 1 -- 172.21.15.16:0/2806460613 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f128c015d80 con 0x7f127c005180 2023-05-22T15:28:14.139 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:14.127+0000 7f129875f640 1 -- 172.21.15.16:0/2806460613 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12900cac90 con 0x7f127c005180 2023-05-22T15:28:14.139 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:14.127+0000 7f129875f640 1 -- 172.21.15.16:0/2806460613 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f129004dbe0 con 0x7f127c005180 2023-05-22T15:28:14.139 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:14.127+0000 7f1286ffd640 1 -- 172.21.15.16:0/2806460613 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7f128c01d400 con 0x7f127c005180 2023-05-22T15:28:14.139 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:14.127+0000 7f1286ffd640 1 --2- 172.21.15.16:0/2806460613 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f1274065380 0x7f1274067830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:14.139 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:14.127+0000 7f1286ffd640 1 -- 172.21.15.16:0/2806460613 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f128c082fd0 con 0x7f127c005180 2023-05-22T15:28:14.139 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:14.127+0000 7f1295cfa640 1 --2- 172.21.15.16:0/2806460613 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f1274065380 0x7f1274067830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:14.140 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:14.131+0000 7f1295cfa640 1 --2- 172.21.15.16:0/2806460613 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f1274065380 0x7f1274067830 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f12901e64c0 tx=0x7f1280009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:28:14.140 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:14.131+0000 7f1286ffd640 1 -- 172.21.15.16:0/2806460613 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f128c0545d0 con 0x7f127c005180 2023-05-22T15:28:14.323 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:14.315+0000 7f129875f640 1 -- 172.21.15.16:0/2806460613 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f129006de30 con 0x7f127c005180 2023-05-22T15:28:14.334 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:14.327+0000 7f1286ffd640 1 -- 172.21.15.16:0/2806460613 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v13) v1 ==== 118+0+0 (secure 0 0 0) 0x7f128c054160 con 0x7f127c005180 2023-05-22T15:28:14.336 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:14.327+0000 7f129875f640 1 -- 172.21.15.16:0/2806460613 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f1274065380 msgr2=0x7f1274067830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:14.337 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:14.327+0000 7f129875f640 1 --2- 172.21.15.16:0/2806460613 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f1274065380 0x7f1274067830 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f12901e64c0 tx=0x7f1280009040 comp rx=0 tx=0).stop 2023-05-22T15:28:14.337 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:14.327+0000 7f129875f640 1 -- 172.21.15.16:0/2806460613 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f127c005180 msgr2=0x7f12901e4690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:14.337 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:14.327+0000 7f129875f640 1 --2- 172.21.15.16:0/2806460613 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f127c005180 0x7f12901e4690 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f128c011560 tx=0x7f128c011a20 comp rx=0 tx=0).stop 2023-05-22T15:28:14.337 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:14.327+0000 7f129875f640 1 -- 172.21.15.16:0/2806460613 shutdown_connections 2023-05-22T15:28:14.337 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:14.327+0000 7f129875f640 1 --2- 172.21.15.16:0/2806460613 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f1274065380 0x7f1274067830 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:14.337 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:14.327+0000 7f129875f640 1 --2- 172.21.15.16:0/2806460613 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f12900b3440 0x7f1290152e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:14.337 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:14.327+0000 7f129875f640 1 --2- 172.21.15.16:0/2806460613 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1290149db0 0x7f12900b2f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:14.338 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:14.327+0000 7f129875f640 1 --2- 172.21.15.16:0/2806460613 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f127c005180 0x7f12901e4690 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:14.338 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:14.327+0000 7f129875f640 1 -- 172.21.15.16:0/2806460613 >> 172.21.15.16:0/2806460613 conn(0x7f1290069720 msgr2=0x7f12901553f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:28:14.338 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:14.327+0000 7f129875f640 1 -- 172.21.15.16:0/2806460613 shutdown_connections 2023-05-22T15:28:14.338 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:14.327+0000 7f129875f640 1 -- 172.21.15.16:0/2806460613 wait complete. 2023-05-22T15:28:14.354 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2023-05-22T15:28:14.355 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-05-22T15:28:14.469 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:14.463+0000 7fa6319f7640 1 Processor -- start 2023-05-22T15:28:14.469 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:14.463+0000 7fa6319f7640 1 -- start start 2023-05-22T15:28:14.471 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:14.463+0000 7fa6319f7640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa62c159d40 0x7fa62c0cb260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:14.471 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:14.463+0000 7fa6319f7640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fa62c06b6c0 0x7fa62c06bb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:14.471 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:14.463+0000 7fa6319f7640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa62c070c30 con 0x7fa62c159970 2023-05-22T15:28:14.471 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:14.463+0000 7fa6319f7640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa62c15dd90 con 0x7fa62c06b6c0 2023-05-22T15:28:14.471 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:14.463+0000 7fa6319f7640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa62c15a190 con 0x7fa62c159d40 2023-05-22T15:28:14.472 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:14.463+0000 7fa62affd640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fa62c159970 0x7fa62c15d680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:39902/0 (socket says 172.21.15.16:39902) 2023-05-22T15:28:14.472 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:14.463+0000 7fa62affd640 1 -- 172.21.15.16:0/3320154062 learned_addr learned my addr 172.21.15.16:0/3320154062 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:28:14.472 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:14.463+0000 7fa629ffb640 1 -- 172.21.15.16:0/3320154062 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3989333717 0 0) 0x7fa62c070c30 con 0x7fa62c159970 2023-05-22T15:28:14.472 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:14.463+0000 7fa629ffb640 1 -- 172.21.15.16:0/3320154062 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa618003530 con 0x7fa62c159970 2023-05-22T15:28:14.472 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:14.463+0000 7fa62b7fe640 1 --2- 172.21.15.16:0/3320154062 >> v2:172.21.15.62:3300/0 conn(0x7fa62c06b6c0 0x7fa62c06bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:14.472 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:14.463+0000 7fa62a7fc640 1 --2- 172.21.15.16:0/3320154062 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa62c159d40 0x7fa62c0cb260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:14.473 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:14.463+0000 7fa629ffb640 1 -- 172.21.15.16:0/3320154062 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2777648430 0 0) 0x7fa618003530 con 0x7fa62c159970 2023-05-22T15:28:14.473 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:14.463+0000 7fa629ffb640 1 -- 172.21.15.16:0/3320154062 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa62c159d40 msgr2=0x7fa62c0cb260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:28:14.474 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:14.463+0000 7fa629ffb640 1 --2- 172.21.15.16:0/3320154062 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa62c159d40 0x7fa62c0cb260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:14.474 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:14.463+0000 7fa629ffb640 1 -- 172.21.15.16:0/3320154062 >> v2:172.21.15.62:3300/0 conn(0x7fa62c06b6c0 msgr2=0x7fa62c06bb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:28:14.474 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:14.463+0000 7fa629ffb640 1 --2- 172.21.15.16:0/3320154062 >> v2:172.21.15.62:3300/0 conn(0x7fa62c06b6c0 0x7fa62c06bb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:14.474 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:14.463+0000 7fa629ffb640 1 -- 172.21.15.16:0/3320154062 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa62c06c060 con 0x7fa62c159970 2023-05-22T15:28:14.474 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:14.463+0000 7fa629ffb640 1 -- 172.21.15.16:0/3320154062 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7fa61c001680 con 0x7fa62c159970 2023-05-22T15:28:14.474 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:14.463+0000 7fa62b7fe640 1 --2- 172.21.15.16:0/3320154062 >> v2:172.21.15.62:3300/0 conn(0x7fa62c06b6c0 0x7fa62c06bb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T15:28:14.474 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:14.463+0000 7fa629ffb640 1 -- 172.21.15.16:0/3320154062 >> v1:172.21.15.16:6789/0 conn(0x7fa62c159970 legacy=0x7fa62c15d680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:14.475 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:14.463+0000 7fa629ffb640 1 --2- 172.21.15.16:0/3320154062 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa618003980 0x7fa618003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:14.475 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:14.463+0000 7fa629ffb640 1 --2- 172.21.15.16:0/3320154062 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa62c159d40 0x7fa6180047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:14.475 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:14.463+0000 7fa629ffb640 1 --2- 172.21.15.16:0/3320154062 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa618004d10 0x7fa618005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:14.475 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:14.463+0000 7fa629ffb640 1 -- 172.21.15.16:0/3320154062 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa62c15dd90 con 0x7fa618003980 2023-05-22T15:28:14.475 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:14.463+0000 7fa629ffb640 1 -- 172.21.15.16:0/3320154062 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa62c15a190 con 0x7fa618004d10 2023-05-22T15:28:14.476 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:14.463+0000 7fa629ffb640 1 -- 172.21.15.16:0/3320154062 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa61800bb20 con 0x7fa62c159d40 2023-05-22T15:28:14.476 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:14.463+0000 7fa62affd640 1 --2- 172.21.15.16:0/3320154062 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa618003980 0x7fa618003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:14.476 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:14.463+0000 7fa62affd640 1 -- 172.21.15.16:0/3320154062 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa62c159d40 msgr2=0x7fa6180047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:28:14.476 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:14.463+0000 7fa62b7fe640 1 --2- 172.21.15.16:0/3320154062 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa618004d10 0x7fa618005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:14.476 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:14.463+0000 7fa62a7fc640 1 --2- 172.21.15.16:0/3320154062 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa62c159d40 0x7fa6180047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:14.476 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:14.463+0000 7fa62affd640 1 --2- 172.21.15.16:0/3320154062 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa62c159d40 0x7fa6180047d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:14.476 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:14.463+0000 7fa62affd640 1 -- 172.21.15.16:0/3320154062 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa618004d10 msgr2=0x7fa618005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:28:14.476 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:14.463+0000 7fa62affd640 1 --2- 172.21.15.16:0/3320154062 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa618004d10 0x7fa618005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:14.477 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:14.463+0000 7fa62affd640 1 -- 172.21.15.16:0/3320154062 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa618003530 con 0x7fa618003980 2023-05-22T15:28:14.481 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:14.467+0000 7fa62affd640 1 --2- 172.21.15.16:0/3320154062 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa618003980 0x7fa618003d90 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7fa61c00c210 tx=0x7fa61c01cc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a66a66728d57a57 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:28:14.481 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:14.467+0000 7fa629ffb640 1 -- 172.21.15.16:0/3320154062 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa61c0058e0 con 0x7fa618003980 2023-05-22T15:28:14.482 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:14.467+0000 7fa629ffb640 1 -- 172.21.15.16:0/3320154062 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa61c005a40 con 0x7fa618003980 2023-05-22T15:28:14.482 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:14.467+0000 7fa629ffb640 1 -- 172.21.15.16:0/3320154062 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa61c005e30 con 0x7fa618003980 2023-05-22T15:28:14.482 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:14.467+0000 7fa6319f7640 1 -- 172.21.15.16:0/3320154062 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa618003980 msgr2=0x7fa618003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:14.482 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:14.467+0000 7fa6319f7640 1 --2- 172.21.15.16:0/3320154062 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa618003980 0x7fa618003d90 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7fa61c00c210 tx=0x7fa61c01cc10 comp rx=0 tx=0).stop 2023-05-22T15:28:14.482 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:14.467+0000 7fa62b7fe640 1 -- 172.21.15.16:0/3320154062 reap_dead start 2023-05-22T15:28:14.482 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:14.467+0000 7fa6319f7640 1 -- 172.21.15.16:0/3320154062 shutdown_connections 2023-05-22T15:28:14.482 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:14.467+0000 7fa6319f7640 1 -- 172.21.15.16:0/3320154062 >> 172.21.15.16:0/3320154062 conn(0x7fa62c069720 msgr2=0x7fa62c069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:28:14.483 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:14.467+0000 7fa6319f7640 1 -- 172.21.15.16:0/3320154062 shutdown_connections 2023-05-22T15:28:14.483 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:14.467+0000 7fa6319f7640 1 -- 172.21.15.16:0/3320154062 wait complete. 2023-05-22T15:28:14.483 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:14.467+0000 7fa6319f7640 1 Processor -- start 2023-05-22T15:28:14.483 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:14.467+0000 7fa6319f7640 1 -- start start 2023-05-22T15:28:14.483 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:14.467+0000 7fa6319f7640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa618003980 0x7fa62c154400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:14.483 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:14.467+0000 7fa6319f7640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa62c0cc4b0 0x7fa62c0cc900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:14.483 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:14.467+0000 7fa6319f7640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa62c1560b0 0x7fa62c156550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:14.484 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:14.467+0000 7fa6319f7640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa62c0cce40 con 0x7fa62c1560b0 2023-05-22T15:28:14.484 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:14.467+0000 7fa6319f7640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa62c156a90 con 0x7fa62c0cc4b0 2023-05-22T15:28:14.484 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:14.467+0000 7fa6319f7640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa62c1f4610 con 0x7fa618003980 2023-05-22T15:28:14.484 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:14.467+0000 7fa62b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa62c1560b0 0x7fa62c156550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:14.484 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:14.467+0000 7fa62b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa62c1560b0 0x7fa62c156550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42780/0 (socket says 172.21.15.16:42780) 2023-05-22T15:28:14.484 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:14.467+0000 7fa62b7fe640 1 -- 172.21.15.16:0/3146967790 learned_addr learned my addr 172.21.15.16:0/3146967790 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:28:14.484 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:14.467+0000 7fa62affd640 1 --2- 172.21.15.16:0/3146967790 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa618003980 0x7fa62c154400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:14.484 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:14.467+0000 7fa62a7fc640 1 --2- 172.21.15.16:0/3146967790 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa62c0cc4b0 0x7fa62c0cc900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:14.485 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:14.467+0000 7fa62b7fe640 1 -- 172.21.15.16:0/3146967790 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa618003980 msgr2=0x7fa62c154400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:14.485 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:14.467+0000 7fa62b7fe640 1 --2- 172.21.15.16:0/3146967790 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa618003980 0x7fa62c154400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:14.485 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:14.467+0000 7fa62b7fe640 1 -- 172.21.15.16:0/3146967790 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa62c0cc4b0 msgr2=0x7fa62c0cc900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:14.485 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:14.467+0000 7fa62b7fe640 1 --2- 172.21.15.16:0/3146967790 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa62c0cc4b0 0x7fa62c0cc900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:14.485 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:14.467+0000 7fa62b7fe640 1 -- 172.21.15.16:0/3146967790 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa62c071700 con 0x7fa62c1560b0 2023-05-22T15:28:14.485 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:14.467+0000 7fa62affd640 1 --2- 172.21.15.16:0/3146967790 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa618003980 0x7fa62c154400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:28:14.485 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:14.471+0000 7fa62b7fe640 1 --2- 172.21.15.16:0/3146967790 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa62c1560b0 0x7fa62c156550 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7fa62c0cf050 tx=0x7fa624004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:28:14.486 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:14.471+0000 7fa623fff640 1 -- 172.21.15.16:0/3146967790 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa624014600 con 0x7fa62c1560b0 2023-05-22T15:28:14.486 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:14.471+0000 7fa623fff640 1 -- 172.21.15.16:0/3146967790 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa624014760 con 0x7fa62c1560b0 2023-05-22T15:28:14.486 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:14.471+0000 7fa6319f7640 1 -- 172.21.15.16:0/3146967790 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa62c071420 con 0x7fa62c1560b0 2023-05-22T15:28:14.486 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:14.471+0000 7fa623fff640 1 -- 172.21.15.16:0/3146967790 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa624014b10 con 0x7fa62c1560b0 2023-05-22T15:28:14.486 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:14.471+0000 7fa6319f7640 1 -- 172.21.15.16:0/3146967790 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa62c0cac90 con 0x7fa62c1560b0 2023-05-22T15:28:14.487 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:14.471+0000 7fa6319f7640 1 -- 172.21.15.16:0/3146967790 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa62c1f8ac0 con 0x7fa62c1560b0 2023-05-22T15:28:14.487 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:14.471+0000 7fa623fff640 1 -- 172.21.15.16:0/3146967790 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7fa62401d3b0 con 0x7fa62c1560b0 2023-05-22T15:28:14.487 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:14.471+0000 7fa623fff640 1 --2- 172.21.15.16:0/3146967790 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fa60c065380 0x7fa60c067830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:14.487 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:14.471+0000 7fa62affd640 1 --2- 172.21.15.16:0/3146967790 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fa60c065380 0x7fa60c067830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:14.487 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:14.471+0000 7fa623fff640 1 -- 172.21.15.16:0/3146967790 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6382+0+0 (secure 0 0 0) 0x7fa6240821f0 con 0x7fa62c1560b0 2023-05-22T15:28:14.487 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:14.471+0000 7fa62affd640 1 --2- 172.21.15.16:0/3146967790 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fa60c065380 0x7fa60c067830 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fa61c0aa000 tx=0x7fa61c001a80 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:28:14.488 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:14.475+0000 7fa623fff640 1 -- 172.21.15.16:0/3146967790 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fa62404fd60 con 0x7fa62c1560b0 2023-05-22T15:28:14.662 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:14.655+0000 7fa6319f7640 1 -- 172.21.15.16:0/3146967790 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fa62c04dbe0 con 0x7fa62c1560b0 2023-05-22T15:28:15.346 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:15.339+0000 7fa623fff640 1 -- 172.21.15.16:0/3146967790 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v14) v1 ==== 122+0+0 (secure 0 0 0) 0x7fa624053380 con 0x7fa62c1560b0 2023-05-22T15:28:15.349 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:15.343+0000 7fa6319f7640 1 -- 172.21.15.16:0/3146967790 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fa60c065380 msgr2=0x7fa60c067830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:15.349 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:15.343+0000 7fa6319f7640 1 --2- 172.21.15.16:0/3146967790 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fa60c065380 0x7fa60c067830 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fa61c0aa000 tx=0x7fa61c001a80 comp rx=0 tx=0).stop 2023-05-22T15:28:15.349 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:15.343+0000 7fa6319f7640 1 -- 172.21.15.16:0/3146967790 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa62c1560b0 msgr2=0x7fa62c156550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:15.349 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:15.343+0000 7fa6319f7640 1 --2- 172.21.15.16:0/3146967790 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa62c1560b0 0x7fa62c156550 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7fa62c0cf050 tx=0x7fa624004470 comp rx=0 tx=0).stop 2023-05-22T15:28:15.349 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:15.343+0000 7fa6319f7640 1 -- 172.21.15.16:0/3146967790 shutdown_connections 2023-05-22T15:28:15.350 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:15.343+0000 7fa6319f7640 1 --2- 172.21.15.16:0/3146967790 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa62c1560b0 0x7fa62c156550 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:15.350 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:15.343+0000 7fa6319f7640 1 --2- 172.21.15.16:0/3146967790 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa62c0cc4b0 0x7fa62c0cc900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:15.350 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:15.343+0000 7fa6319f7640 1 --2- 172.21.15.16:0/3146967790 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fa60c065380 0x7fa60c067830 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:15.350 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:15.343+0000 7fa6319f7640 1 --2- 172.21.15.16:0/3146967790 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa618003980 0x7fa62c154400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:15.350 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:15.343+0000 7fa6319f7640 1 -- 172.21.15.16:0/3146967790 >> 172.21.15.16:0/3146967790 conn(0x7fa62c069720 msgr2=0x7fa62c1585f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:28:15.350 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:15.343+0000 7fa6319f7640 1 -- 172.21.15.16:0/3146967790 shutdown_connections 2023-05-22T15:28:15.350 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:15.343+0000 7fa6319f7640 1 -- 172.21.15.16:0/3146967790 wait complete. 2023-05-22T15:28:15.368 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2023-05-22T15:28:15.369 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-05-22T15:28:15.500 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:15.491+0000 7f0e4704e640 1 Processor -- start 2023-05-22T15:28:15.501 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:15.491+0000 7f0e4704e640 1 -- start start 2023-05-22T15:28:15.502 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:15.495+0000 7f0e4704e640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0e400b7dd0 0x7f0e400ccab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:15.502 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:15.495+0000 7f0e4704e640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f0e40160220 0x7f0e400d0410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:15.502 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:15.495+0000 7f0e4704e640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e400cd170 con 0x7f0e4015f700 2023-05-22T15:28:15.502 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:15.495+0000 7f0e4704e640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e400d3360 con 0x7f0e40160220 2023-05-22T15:28:15.503 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:15.495+0000 7f0e4704e640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e400d2ac0 con 0x7f0e400b7dd0 2023-05-22T15:28:15.503 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:15.495+0000 7f0e3ffff640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f0e4015f700 0x7f0e4015fb10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:39916/0 (socket says 172.21.15.16:39916) 2023-05-22T15:28:15.503 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:15.495+0000 7f0e3ffff640 1 -- 172.21.15.16:0/1154428226 learned_addr learned my addr 172.21.15.16:0/1154428226 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:28:15.503 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:15.495+0000 7f0e455eb640 1 --2- 172.21.15.16:0/1154428226 >> v2:172.21.15.16:3300/0 conn(0x7f0e40160220 0x7f0e400d0410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:15.503 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:15.495+0000 7f0e44dea640 1 --2- 172.21.15.16:0/1154428226 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0e400b7dd0 0x7f0e400ccab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:15.503 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:15.495+0000 7f0e455eb640 1 -- 172.21.15.16:0/1154428226 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0e400b7dd0 msgr2=0x7f0e400ccab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:28:15.503 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:15.495+0000 7f0e455eb640 1 --2- 172.21.15.16:0/1154428226 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0e400b7dd0 0x7f0e400ccab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:15.504 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:15.495+0000 7f0e455eb640 1 -- 172.21.15.16:0/1154428226 >> v1:172.21.15.16:6789/0 conn(0x7f0e4015f700 legacy=0x7f0e4015fb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:15.504 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:15.495+0000 7f0e455eb640 1 -- 172.21.15.16:0/1154428226 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e400d2da0 con 0x7f0e40160220 2023-05-22T15:28:15.504 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:15.495+0000 7f0e44dea640 1 --2- 172.21.15.16:0/1154428226 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0e400b7dd0 0x7f0e400ccab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:28:15.505 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:15.495+0000 7f0e455eb640 1 --2- 172.21.15.16:0/1154428226 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e40160220 0x7f0e400d0410 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f0e38009d90 tx=0x7f0e3801c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e902a37bb46f3558 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:28:15.505 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:15.495+0000 7f0e3f7fe640 1 -- 172.21.15.16:0/1154428226 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e380a5910 con 0x7f0e40160220 2023-05-22T15:28:15.505 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:15.499+0000 7f0e3f7fe640 1 -- 172.21.15.16:0/1154428226 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e380a5a70 con 0x7f0e40160220 2023-05-22T15:28:15.505 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:15.499+0000 7f0e3f7fe640 1 -- 172.21.15.16:0/1154428226 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e380a5e00 con 0x7f0e40160220 2023-05-22T15:28:15.506 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:15.499+0000 7f0e4704e640 1 -- 172.21.15.16:0/1154428226 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e40160220 msgr2=0x7f0e400d0410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:15.506 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:15.499+0000 7f0e4704e640 1 --2- 172.21.15.16:0/1154428226 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e40160220 0x7f0e400d0410 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f0e38009d90 tx=0x7f0e3801c890 comp rx=0 tx=0).stop 2023-05-22T15:28:15.506 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:15.499+0000 7f0e4704e640 1 -- 172.21.15.16:0/1154428226 shutdown_connections 2023-05-22T15:28:15.506 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:15.499+0000 7f0e4704e640 1 --2- 172.21.15.16:0/1154428226 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e40160220 0x7f0e400d0410 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:15.506 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:15.499+0000 7f0e4704e640 1 --2- 172.21.15.16:0/1154428226 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0e400b7dd0 0x7f0e400ccab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:15.506 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:15.499+0000 7f0e4704e640 1 -- 172.21.15.16:0/1154428226 >> 172.21.15.16:0/1154428226 conn(0x7f0e400655c0 msgr2=0x7f0e40070e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:28:15.506 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:15.499+0000 7f0e4704e640 1 -- 172.21.15.16:0/1154428226 shutdown_connections 2023-05-22T15:28:15.507 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:15.499+0000 7f0e4704e640 1 -- 172.21.15.16:0/1154428226 wait complete. 2023-05-22T15:28:15.509 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:15.499+0000 7f0e4704e640 1 Processor -- start 2023-05-22T15:28:15.510 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:15.499+0000 7f0e4704e640 1 -- start start 2023-05-22T15:28:15.510 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:15.499+0000 7f0e4704e640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0e400b7dd0 0x7f0e400ca830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:15.510 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:15.499+0000 7f0e4704e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e4015f700 0x7f0e400c8e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:15.510 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:15.499+0000 7f0e4704e640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0e40160220 0x7f0e40153900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:15.510 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:15.499+0000 7f0e4704e640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e4015f440 con 0x7f0e4015f700 2023-05-22T15:28:15.510 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:15.499+0000 7f0e4704e640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e4015c650 con 0x7f0e40160220 2023-05-22T15:28:15.511 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:15.499+0000 7f0e4704e640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e4015f160 con 0x7f0e400b7dd0 2023-05-22T15:28:15.511 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:15.499+0000 7f0e3ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e4015f700 0x7f0e400c8e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:15.511 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:15.499+0000 7f0e3ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e4015f700 0x7f0e400c8e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42788/0 (socket says 172.21.15.16:42788) 2023-05-22T15:28:15.511 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:15.499+0000 7f0e3ffff640 1 -- 172.21.15.16:0/2890753708 learned_addr learned my addr 172.21.15.16:0/2890753708 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:28:15.511 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:15.499+0000 7f0e44dea640 1 --2- 172.21.15.16:0/2890753708 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0e400b7dd0 0x7f0e400ca830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:15.511 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:15.499+0000 7f0e455eb640 1 --2- 172.21.15.16:0/2890753708 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0e40160220 0x7f0e40153900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:15.511 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:15.499+0000 7f0e3ffff640 1 -- 172.21.15.16:0/2890753708 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0e400b7dd0 msgr2=0x7f0e400ca830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:15.511 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:15.499+0000 7f0e3ffff640 1 --2- 172.21.15.16:0/2890753708 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0e400b7dd0 0x7f0e400ca830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:15.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:15.499+0000 7f0e3ffff640 1 -- 172.21.15.16:0/2890753708 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0e40160220 msgr2=0x7f0e40153900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:15.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:15.499+0000 7f0e3ffff640 1 --2- 172.21.15.16:0/2890753708 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0e40160220 0x7f0e40153900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:15.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:15.499+0000 7f0e3ffff640 1 -- 172.21.15.16:0/2890753708 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e38009a70 con 0x7f0e4015f700 2023-05-22T15:28:15.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:15.499+0000 7f0e3ffff640 1 --2- 172.21.15.16:0/2890753708 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e4015f700 0x7f0e400c8e90 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f0e2c00c500 tx=0x7f0e2c00c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:28:15.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:15.499+0000 7f0e3d7fa640 1 -- 172.21.15.16:0/2890753708 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e2c005cf0 con 0x7f0e4015f700 2023-05-22T15:28:15.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:15.499+0000 7f0e3d7fa640 1 -- 172.21.15.16:0/2890753708 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e2c005e50 con 0x7f0e4015f700 2023-05-22T15:28:15.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:15.499+0000 7f0e4704e640 1 -- 172.21.15.16:0/2890753708 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e400d12f0 con 0x7f0e4015f700 2023-05-22T15:28:15.513 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:15.499+0000 7f0e3d7fa640 1 -- 172.21.15.16:0/2890753708 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e2c0072f0 con 0x7f0e4015f700 2023-05-22T15:28:15.513 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:15.499+0000 7f0e4704e640 1 -- 172.21.15.16:0/2890753708 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e400d1160 con 0x7f0e4015f700 2023-05-22T15:28:15.513 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:15.503+0000 7f0e4704e640 1 -- 172.21.15.16:0/2890753708 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e400d1480 con 0x7f0e4015f700 2023-05-22T15:28:15.513 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:15.503+0000 7f0e3d7fa640 1 -- 172.21.15.16:0/2890753708 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7f0e2c007470 con 0x7f0e4015f700 2023-05-22T15:28:15.514 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:15.503+0000 7f0e3d7fa640 1 --2- 172.21.15.16:0/2890753708 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f0e24065380 0x7f0e24067830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:15.514 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:15.503+0000 7f0e3d7fa640 1 -- 172.21.15.16:0/2890753708 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f0e2c081ec0 con 0x7f0e4015f700 2023-05-22T15:28:15.514 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:15.503+0000 7f0e44dea640 1 --2- 172.21.15.16:0/2890753708 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f0e24065380 0x7f0e24067830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:15.514 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:15.503+0000 7f0e44dea640 1 --2- 172.21.15.16:0/2890753708 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f0e24065380 0x7f0e24067830 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f0e34006fd0 tx=0x7f0e34008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:28:15.514 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:15.507+0000 7f0e3d7fa640 1 -- 172.21.15.16:0/2890753708 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f0e2c04a620 con 0x7f0e4015f700 2023-05-22T15:28:15.699 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:15.691+0000 7f0e4704e640 1 -- 172.21.15.16:0/2890753708 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0e400d1e80 con 0x7f0e4015f700 2023-05-22T15:28:15.702 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:15.691+0000 7f0e3d7fa640 1 -- 172.21.15.16:0/2890753708 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) v1 ==== 94+0+4648 (secure 0 0 0) 0x7f0e2c017090 con 0x7f0e4015f700 2023-05-22T15:28:15.703 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:28:15.703 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":14,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4904,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6833/3153072235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":3153072235},{"type":"v1","addr":"172.21.15.62:6833","nonce":3153072235}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":5164,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/2783822861","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":2783822861},{"type":"v1","addr":"172.21.15.62:6835","nonce":2783822861}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":5571,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/1361634240","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1361634240},{"type":"v1","addr":"172.21.15.16:6835","nonce":1361634240}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":14,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-05-22T15:28:13.320259+0000","modified":"2023-05-22T15:28:15.336631+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5574},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5574":{"gid":5574,"name":"c","rank":0,"incarnation":12,"state":"up:active","state_seq":3,"addr":"172.21.15.16:6837/1033866899","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":1033866899},{"type":"v1","addr":"172.21.15.16:6837","nonce":1033866899}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-05-22T15:28:15.704 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 14 2023-05-22T15:28:15.704 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:15.695+0000 7f0e4704e640 1 -- 172.21.15.16:0/2890753708 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f0e24065380 msgr2=0x7f0e24067830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:15.705 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:15.695+0000 7f0e4704e640 1 --2- 172.21.15.16:0/2890753708 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f0e24065380 0x7f0e24067830 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f0e34006fd0 tx=0x7f0e34008040 comp rx=0 tx=0).stop 2023-05-22T15:28:15.705 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:15.695+0000 7f0e4704e640 1 -- 172.21.15.16:0/2890753708 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e4015f700 msgr2=0x7f0e400c8e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:15.705 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:15.695+0000 7f0e4704e640 1 --2- 172.21.15.16:0/2890753708 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e4015f700 0x7f0e400c8e90 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f0e2c00c500 tx=0x7f0e2c00c9c0 comp rx=0 tx=0).stop 2023-05-22T15:28:15.705 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:15.695+0000 7f0e4704e640 1 -- 172.21.15.16:0/2890753708 shutdown_connections 2023-05-22T15:28:15.706 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:15.695+0000 7f0e4704e640 1 --2- 172.21.15.16:0/2890753708 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0e40160220 0x7f0e40153900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:15.706 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:15.695+0000 7f0e4704e640 1 --2- 172.21.15.16:0/2890753708 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e4015f700 0x7f0e400c8e90 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:15.707 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:15.695+0000 7f0e4704e640 1 --2- 172.21.15.16:0/2890753708 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f0e24065380 0x7f0e24067830 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:15.707 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:15.695+0000 7f0e4704e640 1 --2- 172.21.15.16:0/2890753708 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0e400b7dd0 0x7f0e400ca830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:15.707 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:15.695+0000 7f0e4704e640 1 -- 172.21.15.16:0/2890753708 >> 172.21.15.16:0/2890753708 conn(0x7f0e400655c0 msgr2=0x7f0e400b2180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:28:15.708 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:15.695+0000 7f0e4704e640 1 -- 172.21.15.16:0/2890753708 shutdown_connections 2023-05-22T15:28:15.708 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:15.695+0000 7f0e4704e640 1 -- 172.21.15.16:0/2890753708 wait complete. 2023-05-22T15:28:15.719 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-05-22T15:28:15.831 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:15.823+0000 7fcf48e3f640 1 Processor -- start 2023-05-22T15:28:15.831 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:15.823+0000 7fcf48e3f640 1 -- start start 2023-05-22T15:28:15.832 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:15.823+0000 7fcf48e3f640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fcf440ccab0 0x7fcf4415fe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:15.832 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:15.823+0000 7fcf48e3f640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fcf440d0410 0x7fcf440d0890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:15.832 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:15.823+0000 7fcf48e3f640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf440ccef0 con 0x7fcf440b7dd0 2023-05-22T15:28:15.832 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:15.823+0000 7fcf48e3f640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf440d3360 con 0x7fcf440d0410 2023-05-22T15:28:15.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:15.823+0000 7fcf48e3f640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf440cd170 con 0x7fcf440ccab0 2023-05-22T15:28:15.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:15.827+0000 7fcf4259c640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fcf440b7dd0 0x7fcf4415f700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:39920/0 (socket says 172.21.15.16:39920) 2023-05-22T15:28:15.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:15.827+0000 7fcf4259c640 1 -- 172.21.15.16:0/847723374 learned_addr learned my addr 172.21.15.16:0/847723374 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:28:15.834 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:15.827+0000 7fcf42d9d640 1 --2- 172.21.15.16:0/847723374 >> v2:172.21.15.62:3300/0 conn(0x7fcf440d0410 0x7fcf440d0890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:15.834 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:15.827+0000 7fcf41d9b640 1 --2- 172.21.15.16:0/847723374 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fcf440ccab0 0x7fcf4415fe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:15.834 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:15.827+0000 7fcf4159a640 1 -- 172.21.15.16:0/847723374 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3640579752 0 0) 0x7fcf440ccef0 con 0x7fcf440b7dd0 2023-05-22T15:28:15.835 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:15.827+0000 7fcf4159a640 1 -- 172.21.15.16:0/847723374 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcf30003530 con 0x7fcf440b7dd0 2023-05-22T15:28:15.835 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:15.827+0000 7fcf4159a640 1 -- 172.21.15.16:0/847723374 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3404733291 0 0) 0x7fcf30003530 con 0x7fcf440b7dd0 2023-05-22T15:28:15.835 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:15.827+0000 7fcf4159a640 1 -- 172.21.15.16:0/847723374 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fcf440ccab0 msgr2=0x7fcf4415fe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:28:15.835 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:15.827+0000 7fcf4159a640 1 --2- 172.21.15.16:0/847723374 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fcf440ccab0 0x7fcf4415fe10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:15.835 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:15.827+0000 7fcf4159a640 1 -- 172.21.15.16:0/847723374 >> v2:172.21.15.62:3300/0 conn(0x7fcf440d0410 msgr2=0x7fcf440d0890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:28:15.835 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:15.827+0000 7fcf4159a640 1 --2- 172.21.15.16:0/847723374 >> v2:172.21.15.62:3300/0 conn(0x7fcf440d0410 0x7fcf440d0890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:15.835 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:15.827+0000 7fcf4159a640 1 -- 172.21.15.16:0/847723374 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf440d2ac0 con 0x7fcf440b7dd0 2023-05-22T15:28:15.836 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:15.827+0000 7fcf4159a640 1 -- 172.21.15.16:0/847723374 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7fcf3c001a00 con 0x7fcf440b7dd0 2023-05-22T15:28:15.836 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:15.827+0000 7fcf4159a640 1 -- 172.21.15.16:0/847723374 >> v1:172.21.15.16:6789/0 conn(0x7fcf440b7dd0 legacy=0x7fcf4415f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:15.836 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:15.827+0000 7fcf4159a640 1 --2- 172.21.15.16:0/847723374 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcf30003980 0x7fcf30003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:15.836 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:15.827+0000 7fcf4159a640 1 --2- 172.21.15.16:0/847723374 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcf300043b0 0x7fcf30004870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:15.836 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:15.827+0000 7fcf4159a640 1 --2- 172.21.15.16:0/847723374 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fcf440ccab0 0x7fcf30004f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:15.836 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:15.827+0000 7fcf4159a640 1 -- 172.21.15.16:0/847723374 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf440d3360 con 0x7fcf300043b0 2023-05-22T15:28:15.836 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:15.827+0000 7fcf4159a640 1 -- 172.21.15.16:0/847723374 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf440cd170 con 0x7fcf30003980 2023-05-22T15:28:15.836 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:15.827+0000 7fcf4159a640 1 -- 172.21.15.16:0/847723374 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf3000b990 con 0x7fcf440ccab0 2023-05-22T15:28:15.837 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:15.827+0000 7fcf41d9b640 1 --2- 172.21.15.16:0/847723374 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcf300043b0 0x7fcf30004870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:15.837 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:15.827+0000 7fcf4259c640 1 --2- 172.21.15.16:0/847723374 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcf30003980 0x7fcf30003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:15.837 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:15.827+0000 7fcf41d9b640 1 -- 172.21.15.16:0/847723374 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fcf440ccab0 msgr2=0x7fcf30004f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:28:15.837 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:15.827+0000 7fcf41d9b640 1 --2- 172.21.15.16:0/847723374 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fcf440ccab0 0x7fcf30004f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:15.837 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:15.827+0000 7fcf41d9b640 1 -- 172.21.15.16:0/847723374 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcf30003980 msgr2=0x7fcf30003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:28:15.837 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:15.827+0000 7fcf41d9b640 1 --2- 172.21.15.16:0/847723374 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcf30003980 0x7fcf30003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:15.837 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:15.827+0000 7fcf41d9b640 1 -- 172.21.15.16:0/847723374 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf30003530 con 0x7fcf300043b0 2023-05-22T15:28:15.840 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:15.827+0000 7fcf41d9b640 1 --2- 172.21.15.16:0/847723374 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcf300043b0 0x7fcf30004870 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7fcf2c009ae0 tx=0x7fcf2c01ca90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a88ddb70504e7875 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:28:15.840 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:15.827+0000 7fcf4159a640 1 -- 172.21.15.16:0/847723374 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf2c0a5570 con 0x7fcf300043b0 2023-05-22T15:28:15.840 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:15.827+0000 7fcf4159a640 1 -- 172.21.15.16:0/847723374 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf2c0a56d0 con 0x7fcf300043b0 2023-05-22T15:28:15.840 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:15.827+0000 7fcf4159a640 1 -- 172.21.15.16:0/847723374 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf2c0a5a80 con 0x7fcf300043b0 2023-05-22T15:28:15.840 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:15.827+0000 7fcf48e3f640 1 -- 172.21.15.16:0/847723374 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcf300043b0 msgr2=0x7fcf30004870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:15.840 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:15.827+0000 7fcf48e3f640 1 --2- 172.21.15.16:0/847723374 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcf300043b0 0x7fcf30004870 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7fcf2c009ae0 tx=0x7fcf2c01ca90 comp rx=0 tx=0).stop 2023-05-22T15:28:15.841 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:15.827+0000 7fcf42d9d640 1 -- 172.21.15.16:0/847723374 reap_dead start 2023-05-22T15:28:15.841 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:15.827+0000 7fcf48e3f640 1 -- 172.21.15.16:0/847723374 shutdown_connections 2023-05-22T15:28:15.841 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:15.827+0000 7fcf48e3f640 1 -- 172.21.15.16:0/847723374 >> 172.21.15.16:0/847723374 conn(0x7fcf440655c0 msgr2=0x7fcf44070e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:28:15.841 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:15.827+0000 7fcf48e3f640 1 -- 172.21.15.16:0/847723374 shutdown_connections 2023-05-22T15:28:15.841 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:15.827+0000 7fcf48e3f640 1 -- 172.21.15.16:0/847723374 wait complete. 2023-05-22T15:28:15.841 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:15.831+0000 7fcf48e3f640 1 Processor -- start 2023-05-22T15:28:15.841 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:15.831+0000 7fcf48e3f640 1 -- start start 2023-05-22T15:28:15.841 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:15.831+0000 7fcf48e3f640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fcf300043b0 0x7fcf44154020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:15.842 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:15.831+0000 7fcf48e3f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcf4415cb10 0x7fcf441f0b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:15.842 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:15.831+0000 7fcf48e3f640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcf440b2d50 0x7fcf440b31c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:15.842 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:15.831+0000 7fcf48e3f640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf440cae50 con 0x7fcf4415cb10 2023-05-22T15:28:15.842 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:15.831+0000 7fcf48e3f640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf440b3700 con 0x7fcf440b2d50 2023-05-22T15:28:15.842 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:15.831+0000 7fcf48e3f640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf4415d560 con 0x7fcf300043b0 2023-05-22T15:28:15.842 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:15.831+0000 7fcf41d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcf4415cb10 0x7fcf441f0b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:15.842 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:15.831+0000 7fcf41d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcf4415cb10 0x7fcf441f0b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42794/0 (socket says 172.21.15.16:42794) 2023-05-22T15:28:15.843 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:15.831+0000 7fcf41d9b640 1 -- 172.21.15.16:0/4273854625 learned_addr learned my addr 172.21.15.16:0/4273854625 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:28:15.843 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:15.831+0000 7fcf42d9d640 1 --2- 172.21.15.16:0/4273854625 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcf440b2d50 0x7fcf440b31c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:15.843 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:15.831+0000 7fcf4259c640 1 --2- 172.21.15.16:0/4273854625 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fcf300043b0 0x7fcf44154020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:15.844 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:15.831+0000 7fcf41d9b640 1 -- 172.21.15.16:0/4273854625 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fcf300043b0 msgr2=0x7fcf44154020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:15.844 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:15.831+0000 7fcf41d9b640 1 --2- 172.21.15.16:0/4273854625 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fcf300043b0 0x7fcf44154020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:15.844 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:15.831+0000 7fcf41d9b640 1 -- 172.21.15.16:0/4273854625 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcf440b2d50 msgr2=0x7fcf440b31c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:15.845 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:15.831+0000 7fcf41d9b640 1 --2- 172.21.15.16:0/4273854625 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcf440b2d50 0x7fcf440b31c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:15.845 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:15.831+0000 7fcf41d9b640 1 -- 172.21.15.16:0/4273854625 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf440d3080 con 0x7fcf4415cb10 2023-05-22T15:28:15.845 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:15.831+0000 7fcf4259c640 1 --2- 172.21.15.16:0/4273854625 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fcf300043b0 0x7fcf44154020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:28:15.845 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:15.831+0000 7fcf41d9b640 1 --2- 172.21.15.16:0/4273854625 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcf4415cb10 0x7fcf441f0b20 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7fcf2c01c8c0 tx=0x7fcf30005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:28:15.845 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:15.831+0000 7fcf3affd640 1 -- 172.21.15.16:0/4273854625 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf2c004220 con 0x7fcf4415cb10 2023-05-22T15:28:15.845 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:15.831+0000 7fcf3affd640 1 -- 172.21.15.16:0/4273854625 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf2c004380 con 0x7fcf4415cb10 2023-05-22T15:28:15.845 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:15.831+0000 7fcf48e3f640 1 -- 172.21.15.16:0/4273854625 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf440d2da0 con 0x7fcf4415cb10 2023-05-22T15:28:15.846 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:15.831+0000 7fcf3affd640 1 -- 172.21.15.16:0/4273854625 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf2c003820 con 0x7fcf4415cb10 2023-05-22T15:28:15.846 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:15.831+0000 7fcf48e3f640 1 -- 172.21.15.16:0/4273854625 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf440cf730 con 0x7fcf4415cb10 2023-05-22T15:28:15.846 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:15.831+0000 7fcf48e3f640 1 -- 172.21.15.16:0/4273854625 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf440d1e80 con 0x7fcf4415cb10 2023-05-22T15:28:15.847 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:15.835+0000 7fcf3affd640 1 -- 172.21.15.16:0/4273854625 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7fcf2c0b4040 con 0x7fcf4415cb10 2023-05-22T15:28:15.848 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:15.835+0000 7fcf3affd640 1 --2- 172.21.15.16:0/4273854625 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fcf240653b0 0x7fcf24067860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:15.848 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:15.835+0000 7fcf3affd640 1 -- 172.21.15.16:0/4273854625 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6382+0+0 (secure 0 0 0) 0x7fcf2c11d020 con 0x7fcf4415cb10 2023-05-22T15:28:15.848 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:15.835+0000 7fcf4259c640 1 --2- 172.21.15.16:0/4273854625 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fcf240653b0 0x7fcf24067860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:15.848 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:15.835+0000 7fcf4259c640 1 --2- 172.21.15.16:0/4273854625 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fcf240653b0 0x7fcf24067860 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fcf3c001bb0 tx=0x7fcf3c004b70 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:28:15.848 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:15.839+0000 7fcf3affd640 1 -- 172.21.15.16:0/4273854625 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fcf2c0eaca0 con 0x7fcf4415cb10 2023-05-22T15:28:15.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:15.987+0000 7fcf48e3f640 1 -- 172.21.15.16:0/4273854625 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcf4404dbe0 con 0x7fcf4415cb10 2023-05-22T15:28:15.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:15.991+0000 7fcf3affd640 1 -- 172.21.15.16:0/4273854625 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v42) v1 ==== 74+0+17984 (secure 0 0 0) 0x7fcf2c0044f0 con 0x7fcf4415cb10 2023-05-22T15:28:15.997 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:28:15.997 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":42,"fsid":"b2719834-7847-4d98-90a6-f65d57f85c49","created":"2023-05-22T15:26:25.259314+0000","modified":"2023-05-22T15:28:14.324229+0000","last_up_change":"2023-05-22T15:26:36.448524+0000","last_in_change":"2023-05-22T15:26:27.675668+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-05-22T15:26:38.019781+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-05-22T15:26:43.570514+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2023-05-22T15:28:11.902605+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"41","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2023-05-22T15:28:12.298278+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"42","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9e7ff854-b25f-4cbb-a5c2-1799d9c53d2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":37021},{"type":"v1","addr":"172.21.15.16:6825","nonce":37021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":37021},{"type":"v1","addr":"172.21.15.16:6827","nonce":37021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":37021},{"type":"v1","addr":"172.21.15.16:6831","nonce":37021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":37021},{"type":"v1","addr":"172.21.15.16:6829","nonce":37021}]},"public_addr":"172.21.15.16:6825/37021","cluster_addr":"172.21.15.16:6827/37021","heartbeat_back_addr":"172.21.15.16:6831/37021","heartbeat_front_addr":"172.21.15.16:6829/37021","state":["exists","up"]},{"osd":1,"uuid":"4702fbc9-a927-4565-aca5-41dc69cc8264","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":37008},{"type":"v1","addr":"172.21.15.16:6801","nonce":37008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":37008},{"type":"v1","addr":"172.21.15.16:6803","nonce":37008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":37008},{"type":"v1","addr":"172.21.15.16:6807","nonce":37008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":37008},{"type":"v1","addr":"172.21.15.16:6805","nonce":37008}]},"public_addr":"172.21.15.16:6801/37008","cluster_addr":"172.21.15.16:6803/37008","heartbeat_back_addr":"172.21.15.16:6807/37008","heartbeat_front_addr":"172.21.15.16:6805/37008","state":["exists","up"]},{"osd":2,"uuid":"1e7bba46-4eba-47ef-bb81-5e2d371638ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":37009},{"type":"v1","addr":"172.21.15.16:6809","nonce":37009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":37009},{"type":"v1","addr":"172.21.15.16:6811","nonce":37009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":37009},{"type":"v1","addr":"172.21.15.16:6815","nonce":37009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":37009},{"type":"v1","addr":"172.21.15.16:6813","nonce":37009}]},"public_addr":"172.21.15.16:6809/37009","cluster_addr":"172.21.15.16:6811/37009","heartbeat_back_addr":"172.21.15.16:6815/37009","heartbeat_front_addr":"172.21.15.16:6813/37009","state":["exists","up"]},{"osd":3,"uuid":"383cacc0-5a88-4904-990e-498ca59ef48a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":37020},{"type":"v1","addr":"172.21.15.16:6817","nonce":37020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":37020},{"type":"v1","addr":"172.21.15.16:6819","nonce":37020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":37020},{"type":"v1","addr":"172.21.15.16:6823","nonce":37020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":37020},{"type":"v1","addr":"172.21.15.16:6821","nonce":37020}]},"public_addr":"172.21.15.16:6817/37020","cluster_addr":"172.21.15.16:6819/37020","heartbeat_back_addr":"172.21.15.16:6823/37020","heartbeat_front_addr":"172.21.15.16:6821/37020","state":["exists","up"]},{"osd":4,"uuid":"970500ef-95b7-42c1-b080-eed4fae7013d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":37353},{"type":"v1","addr":"172.21.15.62:6817","nonce":37353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6819","nonce":37353},{"type":"v1","addr":"172.21.15.62:6822","nonce":37353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37353},{"type":"v1","addr":"172.21.15.62:6831","nonce":37353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37353},{"type":"v1","addr":"172.21.15.62:6826","nonce":37353}]},"public_addr":"172.21.15.62:6817/37353","cluster_addr":"172.21.15.62:6822/37353","heartbeat_back_addr":"172.21.15.62:6831/37353","heartbeat_front_addr":"172.21.15.62:6826/37353","state":["exists","up"]},{"osd":5,"uuid":"022d67db-d4d8-445b-b1b9-8b446a00f5c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":37352},{"type":"v1","addr":"172.21.15.62:6801","nonce":37352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":37352},{"type":"v1","addr":"172.21.15.62:6803","nonce":37352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":37352},{"type":"v1","addr":"172.21.15.62:6807","nonce":37352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":37352},{"type":"v1","addr":"172.21.15.62:6805","nonce":37352}]},"public_addr":"172.21.15.62:6801/37352","cluster_addr":"172.21.15.62:6803/37352","heartbeat_back_addr":"172.21.15.62:6807/37352","heartbeat_front_addr":"172.21.15.62:6805/37352","state":["exists","up"]},{"osd":6,"uuid":"42bd372b-6688-4a1b-8407-e16e8c266e33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":37354},{"type":"v1","addr":"172.21.15.62:6820","nonce":37354}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":37354},{"type":"v1","addr":"172.21.15.62:6823","nonce":37354}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6829","nonce":37354},{"type":"v1","addr":"172.21.15.62:6830","nonce":37354}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6825","nonce":37354},{"type":"v1","addr":"172.21.15.62:6827","nonce":37354}]},"public_addr":"172.21.15.62:6820/37354","cluster_addr":"172.21.15.62:6823/37354","heartbeat_back_addr":"172.21.15.62:6830/37354","heartbeat_front_addr":"172.21.15.62:6827/37354","state":["exists","up"]},{"osd":7,"uuid":"ac70b2a3-f41b-4bf2-b227-7f2cbb55426d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37355},{"type":"v1","addr":"172.21.15.62:6809","nonce":37355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37355},{"type":"v1","addr":"172.21.15.62:6811","nonce":37355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":37355},{"type":"v1","addr":"172.21.15.62:6815","nonce":37355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":37355},{"type":"v1","addr":"172.21.15.62:6813","nonce":37355}]},"public_addr":"172.21.15.62:6809/37355","cluster_addr":"172.21.15.62:6811/37355","heartbeat_back_addr":"172.21.15.62:6815/37355","heartbeat_front_addr":"172.21.15.62:6813/37355","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.368896+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.107273+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.276888+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.287502+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.124791+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.251937+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.255679+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.232797+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-05-22T15:28:15.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:15.991+0000 7fcf48e3f640 1 -- 172.21.15.16:0/4273854625 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fcf240653b0 msgr2=0x7fcf24067860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:15.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:15.991+0000 7fcf48e3f640 1 --2- 172.21.15.16:0/4273854625 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fcf240653b0 0x7fcf24067860 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fcf3c001bb0 tx=0x7fcf3c004b70 comp rx=0 tx=0).stop 2023-05-22T15:28:15.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:15.991+0000 7fcf48e3f640 1 -- 172.21.15.16:0/4273854625 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcf4415cb10 msgr2=0x7fcf441f0b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:15.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:15.991+0000 7fcf48e3f640 1 --2- 172.21.15.16:0/4273854625 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcf4415cb10 0x7fcf441f0b20 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7fcf2c01c8c0 tx=0x7fcf30005800 comp rx=0 tx=0).stop 2023-05-22T15:28:15.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:15.991+0000 7fcf48e3f640 1 -- 172.21.15.16:0/4273854625 shutdown_connections 2023-05-22T15:28:15.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:15.991+0000 7fcf48e3f640 1 --2- 172.21.15.16:0/4273854625 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcf440b2d50 0x7fcf440b31c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:15.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:15.991+0000 7fcf48e3f640 1 --2- 172.21.15.16:0/4273854625 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcf4415cb10 0x7fcf441f0b20 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:16.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:15.991+0000 7fcf48e3f640 1 --2- 172.21.15.16:0/4273854625 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fcf240653b0 0x7fcf24067860 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:16.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:15.991+0000 7fcf48e3f640 1 --2- 172.21.15.16:0/4273854625 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fcf300043b0 0x7fcf44154020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:16.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:15.991+0000 7fcf48e3f640 1 -- 172.21.15.16:0/4273854625 >> 172.21.15.16:0/4273854625 conn(0x7fcf440655c0 msgr2=0x7fcf440b6d70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:28:16.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:15.991+0000 7fcf48e3f640 1 -- 172.21.15.16:0/4273854625 shutdown_connections 2023-05-22T15:28:16.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:15.991+0000 7fcf48e3f640 1 -- 172.21.15.16:0/4273854625 wait complete. 2023-05-22T15:28:16.015 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-05-22T15:28:16.015 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-05-22T15:28:16.140 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.131+0000 7fc80dab8640 1 Processor -- start 2023-05-22T15:28:16.140 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.131+0000 7fc80dab8640 1 -- start start 2023-05-22T15:28:16.141 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.135+0000 7fc80dab8640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc8080ca4e0 0x7fc80815e460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:16.141 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.135+0000 7fc80dab8640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8080b0650 con 0x7fc808070df0 2023-05-22T15:28:16.141 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.135+0000 7fc80dab8640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8080b0370 con 0x7fc8080c9b40 2023-05-22T15:28:16.141 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.135+0000 7fc80dab8640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc80815d980 con 0x7fc8080ca4e0 2023-05-22T15:28:16.141 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.135+0000 7fc807fff640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fc808070df0 0x7fc808071210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:39936/0 (socket says 172.21.15.16:39936) 2023-05-22T15:28:16.142 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.135+0000 7fc807fff640 1 -- 172.21.15.16:0/4145980413 learned_addr learned my addr 172.21.15.16:0/4145980413 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:28:16.142 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.135+0000 7fc8077fe640 1 --2- 172.21.15.16:0/4145980413 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc8080ca4e0 0x7fc80815e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:16.143 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.135+0000 7fc8067fc640 1 -- 172.21.15.16:0/4145980413 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3833324733 0 0) 0x7fc8080b0650 con 0x7fc808070df0 2023-05-22T15:28:16.143 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.135+0000 7fc8067fc640 1 -- 172.21.15.16:0/4145980413 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7f0003530 con 0x7fc808070df0 2023-05-22T15:28:16.143 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.135+0000 7fc8067fc640 1 -- 172.21.15.16:0/4145980413 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2602841097 0 0) 0x7fc7f0003530 con 0x7fc808070df0 2023-05-22T15:28:16.143 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.135+0000 7fc8067fc640 1 -- 172.21.15.16:0/4145980413 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc8080ca4e0 msgr2=0x7fc80815e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:28:16.144 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.135+0000 7fc8067fc640 1 --2- 172.21.15.16:0/4145980413 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc8080ca4e0 0x7fc80815e460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:16.144 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.135+0000 7fc8067fc640 1 -- 172.21.15.16:0/4145980413 >> v1:172.21.15.62:6789/0 conn(0x7fc8080c9b40 legacy=0x7fc8080afb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:16.144 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.135+0000 7fc8067fc640 1 -- 172.21.15.16:0/4145980413 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc80815dc60 con 0x7fc808070df0 2023-05-22T15:28:16.144 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.135+0000 7fc8067fc640 1 -- 172.21.15.16:0/4145980413 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7fc800001a00 con 0x7fc808070df0 2023-05-22T15:28:16.144 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.135+0000 7fc8067fc640 1 -- 172.21.15.16:0/4145980413 >> v1:172.21.15.16:6789/0 conn(0x7fc808070df0 legacy=0x7fc808071210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:16.144 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.135+0000 7fc8067fc640 1 --2- 172.21.15.16:0/4145980413 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc7f00039b0 0x7fc7f0003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:16.144 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.135+0000 7fc8067fc640 1 --2- 172.21.15.16:0/4145980413 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc7f00047d0 0x7fc7f0004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:16.145 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.135+0000 7fc8067fc640 1 --2- 172.21.15.16:0/4145980413 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc8080ca4e0 0x7fc7f0005230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:16.145 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.135+0000 7fc8067fc640 1 -- 172.21.15.16:0/4145980413 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc80815d980 con 0x7fc7f00039b0 2023-05-22T15:28:16.145 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.135+0000 7fc8067fc640 1 -- 172.21.15.16:0/4145980413 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7f000bb20 con 0x7fc7f00047d0 2023-05-22T15:28:16.145 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.135+0000 7fc8067fc640 1 -- 172.21.15.16:0/4145980413 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7f000bcb0 con 0x7fc8080ca4e0 2023-05-22T15:28:16.145 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.135+0000 7fc8077fe640 1 --2- 172.21.15.16:0/4145980413 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc7f00039b0 0x7fc7f0003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:16.145 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.135+0000 7fc807fff640 1 --2- 172.21.15.16:0/4145980413 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc8080ca4e0 0x7fc7f0005230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:16.145 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.135+0000 7fc806ffd640 1 --2- 172.21.15.16:0/4145980413 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc7f00047d0 0x7fc7f0004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:16.146 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.135+0000 7fc8077fe640 1 -- 172.21.15.16:0/4145980413 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc8080ca4e0 msgr2=0x7fc7f0005230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:28:16.146 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.135+0000 7fc8077fe640 1 --2- 172.21.15.16:0/4145980413 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc8080ca4e0 0x7fc7f0005230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:16.146 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.135+0000 7fc8077fe640 1 -- 172.21.15.16:0/4145980413 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc7f00047d0 msgr2=0x7fc7f0004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:28:16.146 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.135+0000 7fc8077fe640 1 --2- 172.21.15.16:0/4145980413 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc7f00047d0 0x7fc7f0004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:16.146 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.135+0000 7fc8077fe640 1 -- 172.21.15.16:0/4145980413 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7f0003530 con 0x7fc7f00039b0 2023-05-22T15:28:16.146 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.135+0000 7fc806ffd640 1 --2- 172.21.15.16:0/4145980413 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc7f00047d0 0x7fc7f0004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:28:16.148 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.139+0000 7fc8077fe640 1 --2- 172.21.15.16:0/4145980413 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc7f00039b0 0x7fc7f0003dc0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7fc7f8009b10 tx=0x7fc7f801ca40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ab19c8fda75b121 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:28:16.148 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.139+0000 7fc8067fc640 1 -- 172.21.15.16:0/4145980413 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7f80a5520 con 0x7fc7f00039b0 2023-05-22T15:28:16.149 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.139+0000 7fc8067fc640 1 -- 172.21.15.16:0/4145980413 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7f80a5680 con 0x7fc7f00039b0 2023-05-22T15:28:16.149 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.139+0000 7fc8067fc640 1 -- 172.21.15.16:0/4145980413 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7f80a5a30 con 0x7fc7f00039b0 2023-05-22T15:28:16.149 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.139+0000 7fc80dab8640 1 -- 172.21.15.16:0/4145980413 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc7f00039b0 msgr2=0x7fc7f0003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:16.149 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.139+0000 7fc80dab8640 1 --2- 172.21.15.16:0/4145980413 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc7f00039b0 0x7fc7f0003dc0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7fc7f8009b10 tx=0x7fc7f801ca40 comp rx=0 tx=0).stop 2023-05-22T15:28:16.149 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.139+0000 7fc807fff640 1 -- 172.21.15.16:0/4145980413 reap_dead start 2023-05-22T15:28:16.149 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.139+0000 7fc80dab8640 1 -- 172.21.15.16:0/4145980413 shutdown_connections 2023-05-22T15:28:16.149 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.139+0000 7fc80dab8640 1 -- 172.21.15.16:0/4145980413 >> 172.21.15.16:0/4145980413 conn(0x7fc808069720 msgr2=0x7fc808069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:28:16.149 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.139+0000 7fc80dab8640 1 -- 172.21.15.16:0/4145980413 shutdown_connections 2023-05-22T15:28:16.150 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.139+0000 7fc80dab8640 1 -- 172.21.15.16:0/4145980413 wait complete. 2023-05-22T15:28:16.150 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.139+0000 7fc80dab8640 1 Processor -- start 2023-05-22T15:28:16.150 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.139+0000 7fc80dab8640 1 -- start start 2023-05-22T15:28:16.150 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.139+0000 7fc80dab8640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc7f00039b0 0x7fc80814b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:16.150 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.139+0000 7fc80dab8640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc8080c5c70 0x7fc8080c60c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:16.150 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.139+0000 7fc80dab8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc80814bf70 0x7fc80814c3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:16.150 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.139+0000 7fc80dab8640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc808157e80 con 0x7fc80814bf70 2023-05-22T15:28:16.151 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.139+0000 7fc80dab8640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8080c57e0 con 0x7fc8080c5c70 2023-05-22T15:28:16.151 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.139+0000 7fc80dab8640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc80815cd90 con 0x7fc7f00039b0 2023-05-22T15:28:16.151 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.139+0000 7fc807fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc80814bf70 0x7fc80814c3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:16.151 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.139+0000 7fc807fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc80814bf70 0x7fc80814c3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42812/0 (socket says 172.21.15.16:42812) 2023-05-22T15:28:16.151 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.139+0000 7fc807fff640 1 -- 172.21.15.16:0/1528789455 learned_addr learned my addr 172.21.15.16:0/1528789455 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:28:16.151 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.139+0000 7fc8077fe640 1 --2- 172.21.15.16:0/1528789455 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc7f00039b0 0x7fc80814b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:16.151 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.139+0000 7fc806ffd640 1 --2- 172.21.15.16:0/1528789455 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc8080c5c70 0x7fc8080c60c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:16.151 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.139+0000 7fc807fff640 1 -- 172.21.15.16:0/1528789455 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc7f00039b0 msgr2=0x7fc80814b3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:16.152 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.139+0000 7fc807fff640 1 --2- 172.21.15.16:0/1528789455 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc7f00039b0 0x7fc80814b3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:16.152 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.139+0000 7fc807fff640 1 -- 172.21.15.16:0/1528789455 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc8080c5c70 msgr2=0x7fc8080c60c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:16.152 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.139+0000 7fc807fff640 1 --2- 172.21.15.16:0/1528789455 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc8080c5c70 0x7fc8080c60c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:16.152 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.139+0000 7fc807fff640 1 -- 172.21.15.16:0/1528789455 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8080c9010 con 0x7fc80814bf70 2023-05-22T15:28:16.152 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.139+0000 7fc807fff640 1 --2- 172.21.15.16:0/1528789455 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc80814bf70 0x7fc80814c3e0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7fc8080cad40 tx=0x7fc80000dc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:28:16.152 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.139+0000 7fc7effff640 1 -- 172.21.15.16:0/1528789455 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc800007270 con 0x7fc80814bf70 2023-05-22T15:28:16.152 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.139+0000 7fc7effff640 1 -- 172.21.15.16:0/1528789455 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8000073d0 con 0x7fc80814bf70 2023-05-22T15:28:16.153 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.139+0000 7fc80dab8640 1 -- 172.21.15.16:0/1528789455 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8080c8e80 con 0x7fc80814bf70 2023-05-22T15:28:16.153 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.139+0000 7fc7effff640 1 -- 172.21.15.16:0/1528789455 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc800007780 con 0x7fc80814bf70 2023-05-22T15:28:16.153 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.139+0000 7fc80dab8640 1 -- 172.21.15.16:0/1528789455 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8080c91a0 con 0x7fc80814bf70 2023-05-22T15:28:16.154 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.143+0000 7fc80dab8640 1 -- 172.21.15.16:0/1528789455 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8080c5950 con 0x7fc80814bf70 2023-05-22T15:28:16.154 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.143+0000 7fc7effff640 1 -- 172.21.15.16:0/1528789455 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7fc8000078e0 con 0x7fc80814bf70 2023-05-22T15:28:16.154 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.143+0000 7fc7effff640 1 --2- 172.21.15.16:0/1528789455 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fc7f40653b0 0x7fc7f4067860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:16.154 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.143+0000 7fc7effff640 1 -- 172.21.15.16:0/1528789455 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6382+0+0 (secure 0 0 0) 0x7fc8000820e0 con 0x7fc80814bf70 2023-05-22T15:28:16.154 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.143+0000 7fc8077fe640 1 --2- 172.21.15.16:0/1528789455 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fc7f40653b0 0x7fc7f4067860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:16.154 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.143+0000 7fc8077fe640 1 --2- 172.21.15.16:0/1528789455 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fc7f40653b0 0x7fc7f4067860 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fc7f80af000 tx=0x7fc7f80046d0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:28:16.154 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.147+0000 7fc7effff640 1 -- 172.21.15.16:0/1528789455 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fc8000535b0 con 0x7fc80814bf70 2023-05-22T15:28:16.305 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.299+0000 7fc80dab8640 1 -- 172.21.15.16:0/1528789455 --> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fc80804dbe0 con 0x7fc7f40653b0 2023-05-22T15:28:16.321 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.315+0000 7fc7effff640 1 -- 172.21.15.16:0/1528789455 <== mgr.4101 v2:172.21.15.16:6832/36921 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+299300 (secure 0 0 0) 0x7fc80804dbe0 con 0x7fc7f40653b0 2023-05-22T15:28:16.322 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:28:16.325 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2023-05-22T15:28:16.326 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.319+0000 7fc80dab8640 1 -- 172.21.15.16:0/1528789455 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fc7f40653b0 msgr2=0x7fc7f4067860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:16.327 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.319+0000 7fc80dab8640 1 --2- 172.21.15.16:0/1528789455 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fc7f40653b0 0x7fc7f4067860 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fc7f80af000 tx=0x7fc7f80046d0 comp rx=0 tx=0).stop 2023-05-22T15:28:16.327 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.319+0000 7fc80dab8640 1 -- 172.21.15.16:0/1528789455 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc80814bf70 msgr2=0x7fc80814c3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:16.327 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.319+0000 7fc80dab8640 1 --2- 172.21.15.16:0/1528789455 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc80814bf70 0x7fc80814c3e0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7fc8080cad40 tx=0x7fc80000dc20 comp rx=0 tx=0).stop 2023-05-22T15:28:16.327 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.319+0000 7fc80dab8640 1 -- 172.21.15.16:0/1528789455 shutdown_connections 2023-05-22T15:28:16.327 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.319+0000 7fc80dab8640 1 --2- 172.21.15.16:0/1528789455 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc80814bf70 0x7fc80814c3e0 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:16.327 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.319+0000 7fc80dab8640 1 --2- 172.21.15.16:0/1528789455 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc8080c5c70 0x7fc8080c60c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:16.328 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.319+0000 7fc80dab8640 1 --2- 172.21.15.16:0/1528789455 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fc7f40653b0 0x7fc7f4067860 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:16.328 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.319+0000 7fc80dab8640 1 --2- 172.21.15.16:0/1528789455 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc7f00039b0 0x7fc80814b3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:16.328 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.319+0000 7fc80dab8640 1 -- 172.21.15.16:0/1528789455 >> 172.21.15.16:0/1528789455 conn(0x7fc808069720 msgr2=0x7fc808159570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:28:16.328 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.319+0000 7fc80dab8640 1 -- 172.21.15.16:0/1528789455 shutdown_connections 2023-05-22T15:28:16.328 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.319+0000 7fc80dab8640 1 -- 172.21.15.16:0/1528789455 wait complete. 2023-05-22T15:28:16.344 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":85,"stamp":"2023-05-22T15:28:15.972192+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":82408,"kb_used_data":4584,"kb_used_omap":0,"kb_used_meta":77760,"kb_avail":754892312,"statfs":{"total":773094113280,"available":773009727488,"internally_reserved":0,"allocated":4694016,"data_stored":2708136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":79626240},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.040506"},"pg_stats":[{"pgid":"7.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2023-05-22T15:28:11.938772+0000","last_change":"2023-05-22T15:28:11.938772+0000","last_active":"2023-05-22T15:28:11.927105+0000","last_peered":"2023-05-22T15:28:11.927105+0000","last_clean":"2023-05-22T15:28:11.927105+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:11.938772+0000","last_undegraded":"2023-05-22T15:28:11.938772+0000","last_fullsized":"2023-05-22T15:28:11.938772+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_clean_scrub_stamp":"2023-05-22T15:28:11.927105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T18:33:01.747002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2023-05-22T15:28:11.942902+0000","last_change":"2023-05-22T15:28:11.942902+0000","last_active":"2023-05-22T15:28:11.927105+0000","last_peered":"2023-05-22T15:28:11.927105+0000","last_clean":"2023-05-22T15:28:11.927105+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:11.942902+0000","last_undegraded":"2023-05-22T15:28:11.942902+0000","last_fullsized":"2023-05-22T15:28:11.942902+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_clean_scrub_stamp":"2023-05-22T15:28:11.927105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:21:15.037766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2023-05-22T15:28:11.945096+0000","last_change":"2023-05-22T15:28:11.945096+0000","last_active":"2023-05-22T15:28:11.927105+0000","last_peered":"2023-05-22T15:28:11.927105+0000","last_clean":"2023-05-22T15:28:11.927105+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:11.945096+0000","last_undegraded":"2023-05-22T15:28:11.945096+0000","last_fullsized":"2023-05-22T15:28:11.945096+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_clean_scrub_stamp":"2023-05-22T15:28:11.927105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:02:28.198798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2023-05-22T15:28:11.947443+0000","last_change":"2023-05-22T15:28:11.947443+0000","last_active":"2023-05-22T15:28:11.927105+0000","last_peered":"2023-05-22T15:28:11.927105+0000","last_clean":"2023-05-22T15:28:11.927105+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:11.947443+0000","last_undegraded":"2023-05-22T15:28:11.947443+0000","last_fullsized":"2023-05-22T15:28:11.947443+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_clean_scrub_stamp":"2023-05-22T15:28:11.927105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:42:40.893204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2023-05-22T15:28:11.941543+0000","last_change":"2023-05-22T15:28:11.941543+0000","last_active":"2023-05-22T15:28:11.927105+0000","last_peered":"2023-05-22T15:28:11.927105+0000","last_clean":"2023-05-22T15:28:11.927105+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:11.941543+0000","last_undegraded":"2023-05-22T15:28:11.941543+0000","last_fullsized":"2023-05-22T15:28:11.941543+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_clean_scrub_stamp":"2023-05-22T15:28:11.927105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:05:30.503639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2023-05-22T15:28:11.942842+0000","last_change":"2023-05-22T15:28:11.942842+0000","last_active":"2023-05-22T15:28:11.927105+0000","last_peered":"2023-05-22T15:28:11.927105+0000","last_clean":"2023-05-22T15:28:11.927105+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:11.942842+0000","last_undegraded":"2023-05-22T15:28:11.942842+0000","last_fullsized":"2023-05-22T15:28:11.942842+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_clean_scrub_stamp":"2023-05-22T15:28:11.927105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:17:33.835222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2023-05-22T15:28:11.948423+0000","last_change":"2023-05-22T15:28:11.948423+0000","last_active":"2023-05-22T15:28:11.927105+0000","last_peered":"2023-05-22T15:28:11.927105+0000","last_clean":"2023-05-22T15:28:11.927105+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:11.948423+0000","last_undegraded":"2023-05-22T15:28:11.948423+0000","last_fullsized":"2023-05-22T15:28:11.948423+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_clean_scrub_stamp":"2023-05-22T15:28:11.927105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:45:43.272774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2023-05-22T15:28:11.942591+0000","last_change":"2023-05-22T15:28:11.942591+0000","last_active":"2023-05-22T15:28:11.927105+0000","last_peered":"2023-05-22T15:28:11.927105+0000","last_clean":"2023-05-22T15:28:11.927105+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:11.942591+0000","last_undegraded":"2023-05-22T15:28:11.942591+0000","last_fullsized":"2023-05-22T15:28:11.942591+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_clean_scrub_stamp":"2023-05-22T15:28:11.927105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:02:28.198798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2023-05-22T15:28:11.941674+0000","last_change":"2023-05-22T15:28:11.941674+0000","last_active":"2023-05-22T15:28:11.927105+0000","last_peered":"2023-05-22T15:28:11.927105+0000","last_clean":"2023-05-22T15:28:11.927105+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:11.941674+0000","last_undegraded":"2023-05-22T15:28:11.941674+0000","last_fullsized":"2023-05-22T15:28:11.941674+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_clean_scrub_stamp":"2023-05-22T15:28:11.927105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:50:22.466750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2023-05-22T15:28:11.940470+0000","last_change":"2023-05-22T15:28:11.940470+0000","last_active":"2023-05-22T15:28:11.927105+0000","last_peered":"2023-05-22T15:28:11.927105+0000","last_clean":"2023-05-22T15:28:11.927105+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:11.940470+0000","last_undegraded":"2023-05-22T15:28:11.940470+0000","last_fullsized":"2023-05-22T15:28:11.940470+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_clean_scrub_stamp":"2023-05-22T15:28:11.927105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:15:50.706827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2023-05-22T15:28:11.950095+0000","last_change":"2023-05-22T15:28:11.950095+0000","last_active":"2023-05-22T15:28:11.927105+0000","last_peered":"2023-05-22T15:28:11.927105+0000","last_clean":"2023-05-22T15:28:11.927105+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:11.950095+0000","last_undegraded":"2023-05-22T15:28:11.950095+0000","last_fullsized":"2023-05-22T15:28:11.950095+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_clean_scrub_stamp":"2023-05-22T15:28:11.927105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:57:04.738345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2023-05-22T15:28:11.935772+0000","last_change":"2023-05-22T15:28:11.935772+0000","last_active":"2023-05-22T15:28:11.927105+0000","last_peered":"2023-05-22T15:28:11.927105+0000","last_clean":"2023-05-22T15:28:11.927105+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:11.935772+0000","last_undegraded":"2023-05-22T15:28:11.935772+0000","last_fullsized":"2023-05-22T15:28:11.935772+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_clean_scrub_stamp":"2023-05-22T15:28:11.927105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:30:08.027313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2023-05-22T15:28:11.938777+0000","last_change":"2023-05-22T15:28:11.938777+0000","last_active":"2023-05-22T15:28:11.927105+0000","last_peered":"2023-05-22T15:28:11.927105+0000","last_clean":"2023-05-22T15:28:11.927105+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:11.938777+0000","last_undegraded":"2023-05-22T15:28:11.938777+0000","last_fullsized":"2023-05-22T15:28:11.938777+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_clean_scrub_stamp":"2023-05-22T15:28:11.927105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:47:11.442121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2023-05-22T15:28:11.949598+0000","last_change":"2023-05-22T15:28:11.949598+0000","last_active":"2023-05-22T15:28:11.927105+0000","last_peered":"2023-05-22T15:28:11.927105+0000","last_clean":"2023-05-22T15:28:11.927105+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:11.949598+0000","last_undegraded":"2023-05-22T15:28:11.949598+0000","last_fullsized":"2023-05-22T15:28:11.949598+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_clean_scrub_stamp":"2023-05-22T15:28:11.927105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:26:26.763456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2023-05-22T15:28:11.938244+0000","last_change":"2023-05-22T15:28:11.938244+0000","last_active":"2023-05-22T15:28:11.927105+0000","last_peered":"2023-05-22T15:28:11.927105+0000","last_clean":"2023-05-22T15:28:11.927105+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:11.938244+0000","last_undegraded":"2023-05-22T15:28:11.938244+0000","last_fullsized":"2023-05-22T15:28:11.938244+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_clean_scrub_stamp":"2023-05-22T15:28:11.927105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:09:59.757240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2023-05-22T15:28:11.941202+0000","last_change":"2023-05-22T15:28:11.941202+0000","last_active":"2023-05-22T15:28:11.927105+0000","last_peered":"2023-05-22T15:28:11.927105+0000","last_clean":"2023-05-22T15:28:11.927105+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:11.941202+0000","last_undegraded":"2023-05-22T15:28:11.941202+0000","last_fullsized":"2023-05-22T15:28:11.941202+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_clean_scrub_stamp":"2023-05-22T15:28:11.927105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T15:35:29.715819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2023-05-22T15:28:11.935588+0000","last_change":"2023-05-22T15:28:11.935588+0000","last_active":"2023-05-22T15:28:11.927105+0000","last_peered":"2023-05-22T15:28:11.927105+0000","last_clean":"2023-05-22T15:28:11.927105+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:11.935588+0000","last_undegraded":"2023-05-22T15:28:11.935588+0000","last_fullsized":"2023-05-22T15:28:11.935588+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_clean_scrub_stamp":"2023-05-22T15:28:11.927105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:10:24.156615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2023-05-22T15:28:11.941609+0000","last_change":"2023-05-22T15:28:11.941609+0000","last_active":"2023-05-22T15:28:11.927105+0000","last_peered":"2023-05-22T15:28:11.927105+0000","last_clean":"2023-05-22T15:28:11.927105+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:11.941609+0000","last_undegraded":"2023-05-22T15:28:11.941609+0000","last_fullsized":"2023-05-22T15:28:11.941609+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_clean_scrub_stamp":"2023-05-22T15:28:11.927105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T03:27:50.363592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2023-05-22T15:28:11.947845+0000","last_change":"2023-05-22T15:28:11.947845+0000","last_active":"2023-05-22T15:28:11.927105+0000","last_peered":"2023-05-22T15:28:11.927105+0000","last_clean":"2023-05-22T15:28:11.927105+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:11.947845+0000","last_undegraded":"2023-05-22T15:28:11.947845+0000","last_fullsized":"2023-05-22T15:28:11.947845+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_clean_scrub_stamp":"2023-05-22T15:28:11.927105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:57:04.738345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2023-05-22T15:28:11.938489+0000","last_change":"2023-05-22T15:28:11.938489+0000","last_active":"2023-05-22T15:28:11.927105+0000","last_peered":"2023-05-22T15:28:11.927105+0000","last_clean":"2023-05-22T15:28:11.927105+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:11.938489+0000","last_undegraded":"2023-05-22T15:28:11.938489+0000","last_fullsized":"2023-05-22T15:28:11.938489+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_clean_scrub_stamp":"2023-05-22T15:28:11.927105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:45:43.272774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2023-05-22T15:28:11.940183+0000","last_change":"2023-05-22T15:28:11.940183+0000","last_active":"2023-05-22T15:28:11.927105+0000","last_peered":"2023-05-22T15:28:11.927105+0000","last_clean":"2023-05-22T15:28:11.927105+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:11.940183+0000","last_undegraded":"2023-05-22T15:28:11.940183+0000","last_fullsized":"2023-05-22T15:28:11.940183+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_clean_scrub_stamp":"2023-05-22T15:28:11.927105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:50:22.466750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2023-05-22T15:28:11.947990+0000","last_change":"2023-05-22T15:28:11.947990+0000","last_active":"2023-05-22T15:28:11.927105+0000","last_peered":"2023-05-22T15:28:11.927105+0000","last_clean":"2023-05-22T15:28:11.927105+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:11.947990+0000","last_undegraded":"2023-05-22T15:28:11.947990+0000","last_fullsized":"2023-05-22T15:28:11.947990+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_clean_scrub_stamp":"2023-05-22T15:28:11.927105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T18:18:35.201229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2023-05-22T15:28:11.940527+0000","last_change":"2023-05-22T15:28:11.940527+0000","last_active":"2023-05-22T15:28:11.927105+0000","last_peered":"2023-05-22T15:28:11.927105+0000","last_clean":"2023-05-22T15:28:11.927105+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:11.940527+0000","last_undegraded":"2023-05-22T15:28:11.940527+0000","last_fullsized":"2023-05-22T15:28:11.940527+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_clean_scrub_stamp":"2023-05-22T15:28:11.927105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:24:48.246881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2023-05-22T15:28:11.947499+0000","last_change":"2023-05-22T15:28:11.947499+0000","last_active":"2023-05-22T15:28:11.927105+0000","last_peered":"2023-05-22T15:28:11.927105+0000","last_clean":"2023-05-22T15:28:11.927105+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:11.947499+0000","last_undegraded":"2023-05-22T15:28:11.947499+0000","last_fullsized":"2023-05-22T15:28:11.947499+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_clean_scrub_stamp":"2023-05-22T15:28:11.927105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T18:18:35.201229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2023-05-22T15:28:11.947536+0000","last_change":"2023-05-22T15:28:11.947536+0000","last_active":"2023-05-22T15:28:11.927105+0000","last_peered":"2023-05-22T15:28:11.927105+0000","last_clean":"2023-05-22T15:28:11.927105+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:11.947536+0000","last_undegraded":"2023-05-22T15:28:11.947536+0000","last_fullsized":"2023-05-22T15:28:11.947536+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_clean_scrub_stamp":"2023-05-22T15:28:11.927105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:15:37.287885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":25,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-05-22T15:28:08.260614+0000","last_change":"2023-05-22T15:26:44.982421+0000","last_active":"2023-05-22T15:28:08.260614+0000","last_peered":"2023-05-22T15:28:08.260614+0000","last_clean":"2023-05-22T15:28:08.260614+0000","last_became_active":"2023-05-22T15:26:44.981586+0000","last_became_peered":"2023-05-22T15:26:44.981586+0000","last_unstale":"2023-05-22T15:28:08.260614+0000","last_undegraded":"2023-05-22T15:28:08.260614+0000","last_fullsized":"2023-05-22T15:28:08.260614+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:14:11.156544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2023-05-22T15:28:11.941443+0000","last_change":"2023-05-22T15:28:11.941443+0000","last_active":"2023-05-22T15:28:11.927105+0000","last_peered":"2023-05-22T15:28:11.927105+0000","last_clean":"2023-05-22T15:28:11.927105+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:11.941443+0000","last_undegraded":"2023-05-22T15:28:11.941443+0000","last_fullsized":"2023-05-22T15:28:11.941443+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_clean_scrub_stamp":"2023-05-22T15:28:11.927105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:05:30.503639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":24,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-05-22T15:28:07.758852+0000","last_change":"2023-05-22T15:26:44.978858+0000","last_active":"2023-05-22T15:28:07.758852+0000","last_peered":"2023-05-22T15:28:07.758852+0000","last_clean":"2023-05-22T15:28:07.758852+0000","last_became_active":"2023-05-22T15:26:44.978443+0000","last_became_peered":"2023-05-22T15:26:44.978443+0000","last_unstale":"2023-05-22T15:28:07.758852+0000","last_undegraded":"2023-05-22T15:28:07.758852+0000","last_fullsized":"2023-05-22T15:28:07.758852+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:46:29.290243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2023-05-22T15:28:11.945876+0000","last_change":"2023-05-22T15:28:11.945876+0000","last_active":"2023-05-22T15:28:11.927105+0000","last_peered":"2023-05-22T15:28:11.927105+0000","last_clean":"2023-05-22T15:28:11.927105+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:11.945876+0000","last_undegraded":"2023-05-22T15:28:11.945876+0000","last_fullsized":"2023-05-22T15:28:11.945876+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_clean_scrub_stamp":"2023-05-22T15:28:11.927105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:43:14.376663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":28,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-05-22T15:28:07.681164+0000","last_change":"2023-05-22T15:26:45.092367+0000","last_active":"2023-05-22T15:28:07.681164+0000","last_peered":"2023-05-22T15:28:07.681164+0000","last_clean":"2023-05-22T15:28:07.681164+0000","last_became_active":"2023-05-22T15:26:45.091951+0000","last_became_peered":"2023-05-22T15:26:45.091951+0000","last_unstale":"2023-05-22T15:28:07.681164+0000","last_undegraded":"2023-05-22T15:28:07.681164+0000","last_fullsized":"2023-05-22T15:28:07.681164+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:42:57.973030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2023-05-22T15:28:11.940327+0000","last_change":"2023-05-22T15:28:11.940327+0000","last_active":"2023-05-22T15:28:11.927105+0000","last_peered":"2023-05-22T15:28:11.927105+0000","last_clean":"2023-05-22T15:28:11.927105+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:11.940327+0000","last_undegraded":"2023-05-22T15:28:11.940327+0000","last_fullsized":"2023-05-22T15:28:11.940327+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_clean_scrub_stamp":"2023-05-22T15:28:11.927105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:23:05.013217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":24,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-05-22T15:28:07.758858+0000","last_change":"2023-05-22T15:26:45.091475+0000","last_active":"2023-05-22T15:28:07.758858+0000","last_peered":"2023-05-22T15:28:07.758858+0000","last_clean":"2023-05-22T15:28:07.758858+0000","last_became_active":"2023-05-22T15:26:45.091143+0000","last_became_peered":"2023-05-22T15:26:45.091143+0000","last_unstale":"2023-05-22T15:28:07.758858+0000","last_undegraded":"2023-05-22T15:28:07.758858+0000","last_fullsized":"2023-05-22T15:28:07.758858+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:12:41.464054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2023-05-22T15:28:11.938283+0000","last_change":"2023-05-22T15:28:11.938283+0000","last_active":"2023-05-22T15:28:11.927105+0000","last_peered":"2023-05-22T15:28:11.927105+0000","last_clean":"2023-05-22T15:28:11.927105+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:11.938283+0000","last_undegraded":"2023-05-22T15:28:11.938283+0000","last_fullsized":"2023-05-22T15:28:11.938283+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_clean_scrub_stamp":"2023-05-22T15:28:11.927105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:21:15.037766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":29,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-05-22T15:28:07.680251+0000","last_change":"2023-05-22T15:26:44.976632+0000","last_active":"2023-05-22T15:28:07.680251+0000","last_peered":"2023-05-22T15:28:07.680251+0000","last_clean":"2023-05-22T15:28:07.680251+0000","last_became_active":"2023-05-22T15:26:44.975546+0000","last_became_peered":"2023-05-22T15:26:44.975546+0000","last_unstale":"2023-05-22T15:28:07.680251+0000","last_undegraded":"2023-05-22T15:28:07.680251+0000","last_fullsized":"2023-05-22T15:28:07.680251+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:42:57.973030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2023-05-22T15:28:11.948734+0000","last_change":"2023-05-22T15:28:11.948734+0000","last_active":"2023-05-22T15:28:11.927105+0000","last_peered":"2023-05-22T15:28:11.927105+0000","last_clean":"2023-05-22T15:28:11.927105+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:11.948734+0000","last_undegraded":"2023-05-22T15:28:11.948734+0000","last_fullsized":"2023-05-22T15:28:11.948734+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_clean_scrub_stamp":"2023-05-22T15:28:11.927105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:45:43.272774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":28,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-05-22T15:28:07.678593+0000","last_change":"2023-05-22T15:26:44.969469+0000","last_active":"2023-05-22T15:28:07.678593+0000","last_peered":"2023-05-22T15:28:07.678593+0000","last_clean":"2023-05-22T15:28:07.678593+0000","last_became_active":"2023-05-22T15:26:44.969083+0000","last_became_peered":"2023-05-22T15:26:44.969083+0000","last_unstale":"2023-05-22T15:28:07.678593+0000","last_undegraded":"2023-05-22T15:28:07.678593+0000","last_fullsized":"2023-05-22T15:28:07.678593+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:29:23.670067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2023-05-22T15:28:11.945024+0000","last_change":"2023-05-22T15:28:11.945024+0000","last_active":"2023-05-22T15:28:11.927105+0000","last_peered":"2023-05-22T15:28:11.927105+0000","last_clean":"2023-05-22T15:28:11.927105+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:11.945024+0000","last_undegraded":"2023-05-22T15:28:11.945024+0000","last_fullsized":"2023-05-22T15:28:11.945024+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_clean_scrub_stamp":"2023-05-22T15:28:11.927105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:23:05.013217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":28,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-05-22T15:28:07.681164+0000","last_change":"2023-05-22T15:26:44.977198+0000","last_active":"2023-05-22T15:28:07.681164+0000","last_peered":"2023-05-22T15:28:07.681164+0000","last_clean":"2023-05-22T15:28:07.681164+0000","last_became_active":"2023-05-22T15:26:44.976445+0000","last_became_peered":"2023-05-22T15:26:44.976445+0000","last_unstale":"2023-05-22T15:28:07.681164+0000","last_undegraded":"2023-05-22T15:28:07.681164+0000","last_fullsized":"2023-05-22T15:28:07.681164+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:12:41.464054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2023-05-22T15:28:11.947908+0000","last_change":"2023-05-22T15:28:11.947908+0000","last_active":"2023-05-22T15:28:11.927105+0000","last_peered":"2023-05-22T15:28:11.927105+0000","last_clean":"2023-05-22T15:28:11.927105+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:11.947908+0000","last_undegraded":"2023-05-22T15:28:11.947908+0000","last_fullsized":"2023-05-22T15:28:11.947908+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_clean_scrub_stamp":"2023-05-22T15:28:11.927105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:15:37.287885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":29,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-05-22T15:28:07.680257+0000","last_change":"2023-05-22T15:26:44.971951+0000","last_active":"2023-05-22T15:28:07.680257+0000","last_peered":"2023-05-22T15:28:07.680257+0000","last_clean":"2023-05-22T15:28:07.680257+0000","last_became_active":"2023-05-22T15:26:44.970872+0000","last_became_peered":"2023-05-22T15:26:44.970872+0000","last_unstale":"2023-05-22T15:28:07.680257+0000","last_undegraded":"2023-05-22T15:28:07.680257+0000","last_fullsized":"2023-05-22T15:28:07.680257+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:55:34.752949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":181,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-05-22T15:28:07.681354+0000","last_change":"2023-05-22T15:26:39.318204+0000","last_active":"2023-05-22T15:28:07.681354+0000","last_peered":"2023-05-22T15:28:07.681354+0000","last_clean":"2023-05-22T15:28:07.681354+0000","last_became_active":"2023-05-22T15:26:39.315370+0000","last_became_peered":"2023-05-22T15:26:39.315370+0000","last_unstale":"2023-05-22T15:28:07.681354+0000","last_undegraded":"2023-05-22T15:28:07.681354+0000","last_fullsized":"2023-05-22T15:28:07.681354+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:38.020076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:38.020076+0000","last_clean_scrub_stamp":"2023-05-22T15:26:38.020076+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:06:39.207543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":7},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574870,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":13752,"kb_used_data":1008,"kb_used_omap":0,"kb_used_meta":12736,"kb_avail":94358088,"statfs":{"total":96636764160,"available":96622682112,"internally_reserved":0,"allocated":1032192,"data_stored":781293,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":13041664},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 15:27:38 2023","interfaces":[{"interface":"back","average":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":1.5860000000000001,"5min":1.5860000000000001,"15min":1.5860000000000001},"last":0.77700000000000002},{"interface":"front","average":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"min":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.4239999999999999,"5min":1.4239999999999999,"15min":1.4239999999999999},"last":0.55700000000000005}]},{"osd":1,"last update":"Mon May 22 15:27:38 2023","interfaces":[{"interface":"back","average":{"1min":0.96599999999999997,"5min":0.96599999999999997,"15min":0.96599999999999997},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":1.3600000000000001,"5min":1.3600000000000001,"15min":1.3600000000000001},"last":1.002},{"interface":"front","average":{"1min":1.002,"5min":1.002,"15min":1.002},"min":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"max":{"1min":1.8260000000000001,"5min":1.8260000000000001,"15min":1.8260000000000001},"last":1.208}]},{"osd":2,"last update":"Mon May 22 15:27:38 2023","interfaces":[{"interface":"back","average":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.127,"5min":1.127,"15min":1.127},"last":0.90100000000000002},{"interface":"front","average":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":1.23,"5min":1.23,"15min":1.23},"last":0.83999999999999997}]},{"osd":3,"last update":"Mon May 22 15:27:38 2023","interfaces":[{"interface":"back","average":{"1min":0.99099999999999999,"5min":0.99099999999999999,"15min":0.99099999999999999},"min":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"max":{"1min":1.7829999999999999,"5min":1.7829999999999999,"15min":1.7829999999999999},"last":1.115},{"interface":"front","average":{"1min":1.026,"5min":1.026,"15min":1.026},"min":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":1.6879999999999999,"5min":1.6879999999999999,"15min":1.6879999999999999},"last":0.70199999999999996}]},{"osd":4,"last update":"Mon May 22 15:27:38 2023","interfaces":[{"interface":"back","average":{"1min":0.92700000000000005,"5min":0.92700000000000005,"15min":0.92700000000000005},"min":{"1min":0.62,"5min":0.62,"15min":0.62},"max":{"1min":1.2430000000000001,"5min":1.2430000000000001,"15min":1.2430000000000001},"last":0.65600000000000003},{"interface":"front","average":{"1min":1.1120000000000001,"5min":1.1120000000000001,"15min":1.1120000000000001},"min":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"max":{"1min":1.73,"5min":1.73,"15min":1.73},"last":1.2889999999999999}]},{"osd":5,"last update":"Mon May 22 15:27:38 2023","interfaces":[{"interface":"back","average":{"1min":1.123,"5min":1.123,"15min":1.123},"min":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":1.3779999999999999,"5min":1.3779999999999999,"15min":1.3779999999999999},"last":1.0760000000000001},{"interface":"front","average":{"1min":1.139,"5min":1.139,"15min":1.139},"min":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"max":{"1min":1.6439999999999999,"5min":1.6439999999999999,"15min":1.6439999999999999},"last":1.0429999999999999}]},{"osd":6,"last update":"Mon May 22 15:27:38 2023","interfaces":[{"interface":"back","average":{"1min":1.1020000000000001,"5min":1.1020000000000001,"15min":1.1020000000000001},"min":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"max":{"1min":1.3999999999999999,"5min":1.3999999999999999,"15min":1.3999999999999999},"last":0.96399999999999997},{"interface":"front","average":{"1min":1.1890000000000001,"5min":1.1890000000000001,"15min":1.1890000000000001},"min":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"max":{"1min":2.04,"5min":2.04,"15min":2.04},"last":1.3620000000000001}]}]},{"osd":6,"up_from":13,"seq":55834574871,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8372,"kb_used_data":428,"kb_used_omap":0,"kb_used_meta":7936,"kb_avail":94363468,"statfs":{"total":96636764160,"available":96628191232,"internally_reserved":0,"allocated":438272,"data_stored":190925,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8126464},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 15:27:41 2023","interfaces":[{"interface":"back","average":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"min":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":1.1659999999999999,"5min":1.1659999999999999,"15min":1.1659999999999999},"last":0.97399999999999998},{"interface":"front","average":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"min":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"max":{"1min":1.385,"5min":1.385,"15min":1.385},"last":0.71499999999999997}]},{"osd":1,"last update":"Mon May 22 15:27:41 2023","interfaces":[{"interface":"back","average":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"min":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":1.2869999999999999,"5min":1.2869999999999999,"15min":1.2869999999999999},"last":0.77000000000000002},{"interface":"front","average":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":0.90000000000000002},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":1.204,"5min":1.204,"15min":1.204},"last":0.92000000000000004}]},{"osd":2,"last update":"Mon May 22 15:27:41 2023","interfaces":[{"interface":"back","average":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"min":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"max":{"1min":1.145,"5min":1.145,"15min":1.145},"last":0.84099999999999997},{"interface":"front","average":{"1min":1.0569999999999999,"5min":1.0569999999999999,"15min":1.0569999999999999},"min":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"max":{"1min":1.4670000000000001,"5min":1.4670000000000001,"15min":1.4670000000000001},"last":1.0700000000000001}]},{"osd":3,"last update":"Mon May 22 15:27:41 2023","interfaces":[{"interface":"back","average":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"min":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"max":{"1min":1.415,"5min":1.415,"15min":1.415},"last":1.0169999999999999},{"interface":"front","average":{"1min":0.95799999999999996,"5min":0.95799999999999996,"15min":0.95799999999999996},"min":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":1.486,"5min":1.486,"15min":1.486},"last":1.042}]},{"osd":4,"last update":"Mon May 22 15:27:41 2023","interfaces":[{"interface":"back","average":{"1min":1.0389999999999999,"5min":1.0389999999999999,"15min":1.0389999999999999},"min":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"max":{"1min":1.6459999999999999,"5min":1.6459999999999999,"15min":1.6459999999999999},"last":1.1539999999999999},{"interface":"front","average":{"1min":1.141,"5min":1.141,"15min":1.141},"min":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"max":{"1min":1.5469999999999999,"5min":1.5469999999999999,"15min":1.5469999999999999},"last":1.097}]},{"osd":5,"last update":"Mon May 22 15:27:41 2023","interfaces":[{"interface":"back","average":{"1min":1.075,"5min":1.075,"15min":1.075},"min":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"max":{"1min":1.621,"5min":1.621,"15min":1.621},"last":1.117},{"interface":"front","average":{"1min":1.149,"5min":1.149,"15min":1.149},"min":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"max":{"1min":1.7509999999999999,"5min":1.7509999999999999,"15min":1.7509999999999999},"last":1.232}]},{"osd":7,"last update":"Mon May 22 15:27:41 2023","interfaces":[{"interface":"back","average":{"1min":1.268,"5min":1.268,"15min":1.268},"min":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"max":{"1min":1.8120000000000001,"5min":1.8120000000000001,"15min":1.8120000000000001},"last":1.177},{"interface":"front","average":{"1min":1.179,"5min":1.179,"15min":1.179},"min":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"max":{"1min":1.6819999999999999,"5min":1.6819999999999999,"15min":1.6819999999999999},"last":1.3260000000000001}]}]},{"osd":5,"up_from":13,"seq":55834574871,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8500,"kb_used_data":428,"kb_used_omap":0,"kb_used_meta":8064,"kb_avail":94363340,"statfs":{"total":96636764160,"available":96628060160,"internally_reserved":0,"allocated":438272,"data_stored":190925,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8257536},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 15:27:37 2023","interfaces":[{"interface":"back","average":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"min":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":1.169,"5min":1.169,"15min":1.169},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"min":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.103,"5min":1.103,"15min":1.103},"last":0.68500000000000005}]},{"osd":1,"last update":"Mon May 22 15:27:37 2023","interfaces":[{"interface":"back","average":{"1min":0.90200000000000002,"5min":0.90200000000000002,"15min":0.90200000000000002},"min":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":1.2170000000000001,"5min":1.2170000000000001,"15min":1.2170000000000001},"last":1.0309999999999999},{"interface":"front","average":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"min":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":1.399,"5min":1.399,"15min":1.399},"last":1.0069999999999999}]},{"osd":2,"last update":"Mon May 22 15:27:37 2023","interfaces":[{"interface":"back","average":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"min":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":1.597,"5min":1.597,"15min":1.597},"last":0.83399999999999996},{"interface":"front","average":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"min":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.296,"5min":1.296,"15min":1.296},"last":0.79100000000000004}]},{"osd":3,"last update":"Mon May 22 15:27:37 2023","interfaces":[{"interface":"back","average":{"1min":0.93400000000000005,"5min":0.93400000000000005,"15min":0.93400000000000005},"min":{"1min":0.627,"5min":0.627,"15min":0.627},"max":{"1min":1.6839999999999999,"5min":1.6839999999999999,"15min":1.6839999999999999},"last":0.91000000000000003},{"interface":"front","average":{"1min":1.0209999999999999,"5min":1.0209999999999999,"15min":1.0209999999999999},"min":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"max":{"1min":1.5189999999999999,"5min":1.5189999999999999,"15min":1.5189999999999999},"last":1.111}]},{"osd":4,"last update":"Mon May 22 15:27:37 2023","interfaces":[{"interface":"back","average":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":0.96399999999999997},"min":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"max":{"1min":1.3140000000000001,"5min":1.3140000000000001,"15min":1.3140000000000001},"last":0.876},{"interface":"front","average":{"1min":1.0549999999999999,"5min":1.0549999999999999,"15min":1.0549999999999999},"min":{"1min":0.747,"5min":0.747,"15min":0.747},"max":{"1min":1.319,"5min":1.319,"15min":1.319},"last":1.0700000000000001}]},{"osd":6,"last update":"Mon May 22 15:27:37 2023","interfaces":[{"interface":"back","average":{"1min":1.1000000000000001,"5min":1.1000000000000001,"15min":1.1000000000000001},"min":{"1min":0.94199999999999995,"5min":0.94199999999999995,"15min":0.94199999999999995},"max":{"1min":1.357,"5min":1.357,"15min":1.357},"last":1.2190000000000001},{"interface":"front","average":{"1min":1.109,"5min":1.109,"15min":1.109},"min":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"max":{"1min":1.4350000000000001,"5min":1.4350000000000001,"15min":1.4350000000000001},"last":0.95899999999999996}]},{"osd":7,"last update":"Mon May 22 15:27:37 2023","interfaces":[{"interface":"back","average":{"1min":1.232,"5min":1.232,"15min":1.232},"min":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"max":{"1min":2.8260000000000001,"5min":2.8260000000000001,"15min":2.8260000000000001},"last":1.157},{"interface":"front","average":{"1min":1.1679999999999999,"5min":1.1679999999999999,"15min":1.1679999999999999},"min":{"1min":0.877,"5min":0.877,"15min":0.877},"max":{"1min":1.4490000000000001,"5min":1.4490000000000001,"15min":1.4490000000000001},"last":1.29}]}]},{"osd":4,"up_from":13,"seq":55834574869,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8564,"kb_used_data":428,"kb_used_omap":0,"kb_used_meta":8128,"kb_avail":94363276,"statfs":{"total":96636764160,"available":96627994624,"internally_reserved":0,"allocated":438272,"data_stored":190925,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8323072},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 15:27:37 2023","interfaces":[{"interface":"back","average":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"min":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":1.252,"5min":1.252,"15min":1.252},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"min":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"max":{"1min":1.2989999999999999,"5min":1.2989999999999999,"15min":1.2989999999999999},"last":0.56299999999999994}]},{"osd":1,"last update":"Mon May 22 15:27:37 2023","interfaces":[{"interface":"back","average":{"1min":0.872,"5min":0.872,"15min":0.872},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.3009999999999999,"5min":1.3009999999999999,"15min":1.3009999999999999},"last":1.1220000000000001},{"interface":"front","average":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":0.83299999999999996},"min":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":1.1499999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.71399999999999997}]},{"osd":2,"last update":"Mon May 22 15:27:37 2023","interfaces":[{"interface":"back","average":{"1min":0.96199999999999997,"5min":0.96199999999999997,"15min":0.96199999999999997},"min":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"max":{"1min":1.284,"5min":1.284,"15min":1.284},"last":0.78000000000000003},{"interface":"front","average":{"1min":1.0780000000000001,"5min":1.0780000000000001,"15min":1.0780000000000001},"min":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"max":{"1min":1.9770000000000001,"5min":1.9770000000000001,"15min":1.9770000000000001},"last":0.97499999999999998}]},{"osd":3,"last update":"Mon May 22 15:27:37 2023","interfaces":[{"interface":"back","average":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"min":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.4850000000000001,"5min":1.4850000000000001,"15min":1.4850000000000001},"last":0.81699999999999995},{"interface":"front","average":{"1min":1.1100000000000001,"5min":1.1100000000000001,"15min":1.1100000000000001},"min":{"1min":0.872,"5min":0.872,"15min":0.872},"max":{"1min":1.468,"5min":1.468,"15min":1.468},"last":0.86199999999999999}]},{"osd":5,"last update":"Mon May 22 15:27:37 2023","interfaces":[{"interface":"back","average":{"1min":1.123,"5min":1.123,"15min":1.123},"min":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"max":{"1min":1.7270000000000001,"5min":1.7270000000000001,"15min":1.7270000000000001},"last":1.042},{"interface":"front","average":{"1min":1.024,"5min":1.024,"15min":1.024},"min":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"max":{"1min":1.417,"5min":1.417,"15min":1.417},"last":1.0680000000000001}]},{"osd":6,"last update":"Mon May 22 15:27:37 2023","interfaces":[{"interface":"back","average":{"1min":1.181,"5min":1.181,"15min":1.181},"min":{"1min":0.96899999999999997,"5min":0.96899999999999997,"15min":0.96899999999999997},"max":{"1min":1.6020000000000001,"5min":1.6020000000000001,"15min":1.6020000000000001},"last":1.0089999999999999},{"interface":"front","average":{"1min":1.202,"5min":1.202,"15min":1.202},"min":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"max":{"1min":2.0699999999999998,"5min":2.0699999999999998,"15min":2.0699999999999998},"last":1.1890000000000001}]},{"osd":7,"last update":"Mon May 22 15:27:37 2023","interfaces":[{"interface":"back","average":{"1min":1.1919999999999999,"5min":1.1919999999999999,"15min":1.1919999999999999},"min":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.90800000000000003},"max":{"1min":1.6639999999999999,"5min":1.6639999999999999,"15min":1.6639999999999999},"last":0.90800000000000003},{"interface":"front","average":{"1min":1.2210000000000001,"5min":1.2210000000000001,"15min":1.2210000000000001},"min":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"max":{"1min":1.7410000000000001,"5min":1.7410000000000001,"15min":1.7410000000000001},"last":1.1559999999999999}]}]},{"osd":3,"up_from":13,"seq":55834574869,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8500,"kb_used_data":428,"kb_used_omap":0,"kb_used_meta":8064,"kb_avail":94363340,"statfs":{"total":96636764160,"available":96628060160,"internally_reserved":0,"allocated":438272,"data_stored":190925,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8257536},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 15:27:37 2023","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.42999999999999999}]},{"osd":1,"last update":"Mon May 22 15:27:37 2023","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.505},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":1.6579999999999999,"5min":1.6579999999999999,"15min":1.6579999999999999},"last":0.49099999999999999}]},{"osd":2,"last update":"Mon May 22 15:27:37 2023","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":1.2490000000000001,"5min":1.2490000000000001,"15min":1.2490000000000001},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":1.855,"5min":1.855,"15min":1.855},"last":0.53000000000000003}]},{"osd":4,"last update":"Mon May 22 15:27:37 2023","interfaces":[{"interface":"back","average":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"min":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":1.5820000000000001,"5min":1.5820000000000001,"15min":1.5820000000000001},"last":0.77000000000000002},{"interface":"front","average":{"1min":0.96899999999999997,"5min":0.96899999999999997,"15min":0.96899999999999997},"min":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"max":{"1min":1.8069999999999999,"5min":1.8069999999999999,"15min":1.8069999999999999},"last":0.66800000000000004}]},{"osd":5,"last update":"Mon May 22 15:27:37 2023","interfaces":[{"interface":"back","average":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":0.83799999999999997},"min":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":1.0840000000000001,"5min":1.0840000000000001,"15min":1.0840000000000001},"last":0.69499999999999995},{"interface":"front","average":{"1min":1.0249999999999999,"5min":1.0249999999999999,"15min":1.0249999999999999},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":2.4820000000000002,"5min":2.4820000000000002,"15min":2.4820000000000002},"last":0.78300000000000003}]},{"osd":6,"last update":"Mon May 22 15:27:37 2023","interfaces":[{"interface":"back","average":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":0.90300000000000002},"min":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"max":{"1min":1.452,"5min":1.452,"15min":1.452},"last":0.623},{"interface":"front","average":{"1min":1.085,"5min":1.085,"15min":1.085},"min":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"max":{"1min":2.6589999999999998,"5min":2.6589999999999998,"15min":2.6589999999999998},"last":0.88500000000000001}]},{"osd":7,"last update":"Mon May 22 15:27:37 2023","interfaces":[{"interface":"back","average":{"1min":1.1459999999999999,"5min":1.1459999999999999,"15min":1.1459999999999999},"min":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"max":{"1min":2.762,"5min":2.762,"15min":2.762},"last":0.73499999999999999},{"interface":"front","average":{"1min":1.0369999999999999,"5min":1.0369999999999999,"15min":1.0369999999999999},"min":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"max":{"1min":1.714,"5min":1.714,"15min":1.714},"last":0.64400000000000002}]}]},{"osd":2,"up_from":13,"seq":55834574870,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":12532,"kb_used_data":428,"kb_used_omap":0,"kb_used_meta":12096,"kb_avail":94359308,"statfs":{"total":96636764160,"available":96623931392,"internally_reserved":0,"allocated":438272,"data_stored":190925,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":12386304},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 15:27:36 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":1.4670000000000001,"5min":1.4670000000000001,"15min":1.4670000000000001},"last":0.55700000000000005}]},{"osd":1,"last update":"Mon May 22 15:27:36 2023","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":1.5309999999999999,"5min":1.5309999999999999,"15min":1.5309999999999999},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":1.2210000000000001,"5min":1.2210000000000001,"15min":1.2210000000000001},"last":0.46899999999999997}]},{"osd":3,"last update":"Mon May 22 15:27:36 2023","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":1.3340000000000001,"5min":1.3340000000000001,"15min":1.3340000000000001},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.754,"5min":0.754,"15min":0.754},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":1.6339999999999999,"5min":1.6339999999999999,"15min":1.6339999999999999},"last":0.79500000000000004}]},{"osd":4,"last update":"Mon May 22 15:27:36 2023","interfaces":[{"interface":"back","average":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"min":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"max":{"1min":1.774,"5min":1.774,"15min":1.774},"last":0.91000000000000003},{"interface":"front","average":{"1min":0.97599999999999998,"5min":0.97599999999999998,"15min":0.97599999999999998},"min":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"max":{"1min":1.802,"5min":1.802,"15min":1.802},"last":1.0600000000000001}]},{"osd":5,"last update":"Mon May 22 15:27:36 2023","interfaces":[{"interface":"back","average":{"1min":1.05,"5min":1.05,"15min":1.05},"min":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"max":{"1min":2.4569999999999999,"5min":2.4569999999999999,"15min":2.4569999999999999},"last":0.93500000000000005},{"interface":"front","average":{"1min":1,"5min":1,"15min":1},"min":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"max":{"1min":1.4670000000000001,"5min":1.4670000000000001,"15min":1.4670000000000001},"last":1.091}]},{"osd":6,"last update":"Mon May 22 15:27:36 2023","interfaces":[{"interface":"back","average":{"1min":1.093,"5min":1.093,"15min":1.093},"min":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":3.488,"5min":3.488,"15min":3.488},"last":0.95899999999999996},{"interface":"front","average":{"1min":1.151,"5min":1.151,"15min":1.151},"min":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"max":{"1min":3.7330000000000001,"5min":3.7330000000000001,"15min":3.7330000000000001},"last":1.0029999999999999}]},{"osd":7,"last update":"Mon May 22 15:27:36 2023","interfaces":[{"interface":"back","average":{"1min":1.085,"5min":1.085,"15min":1.085},"min":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"max":{"1min":2.21,"5min":2.21,"15min":2.21},"last":0.875},{"interface":"front","average":{"1min":1.0269999999999999,"5min":1.0269999999999999,"15min":1.0269999999999999},"min":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":1.6120000000000001,"5min":1.6120000000000001,"15min":1.6120000000000001},"last":1.0329999999999999}]}]},{"osd":1,"up_from":13,"seq":55834574870,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8500,"kb_used_data":428,"kb_used_omap":0,"kb_used_meta":8064,"kb_avail":94363340,"statfs":{"total":96636764160,"available":96628060160,"internally_reserved":0,"allocated":438272,"data_stored":190925,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8257536},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 15:27:36 2023","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":1.3089999999999999,"5min":1.3089999999999999,"15min":1.3089999999999999},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":1.3799999999999999,"5min":1.3799999999999999,"15min":1.3799999999999999},"last":0.95899999999999996}]},{"osd":2,"last update":"Mon May 22 15:27:36 2023","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.63,"15min":0.63},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.98299999999999998,"5min":0.98299999999999998,"15min":0.98299999999999998},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.35199999999999998}]},{"osd":3,"last update":"Mon May 22 15:27:36 2023","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":1.145,"5min":1.145,"15min":1.145},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.0760000000000001,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.49299999999999999}]},{"osd":4,"last update":"Mon May 22 15:27:36 2023","interfaces":[{"interface":"back","average":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":0.85299999999999998},"min":{"1min":0.622,"5min":0.622,"15min":0.622},"max":{"1min":1.2250000000000001,"5min":1.2250000000000001,"15min":1.2250000000000001},"last":0.86199999999999999},{"interface":"front","average":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"min":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"max":{"1min":1.272,"5min":1.272,"15min":1.272},"last":1.0109999999999999}]},{"osd":5,"last update":"Mon May 22 15:27:36 2023","interfaces":[{"interface":"back","average":{"1min":0.95999999999999996,"5min":0.95999999999999996,"15min":0.95999999999999996},"min":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"max":{"1min":1.1779999999999999,"5min":1.1779999999999999,"15min":1.1779999999999999},"last":0.68500000000000005},{"interface":"front","average":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":0.96299999999999997},"min":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"max":{"1min":1.488,"5min":1.488,"15min":1.488},"last":0.73299999999999998}]},{"osd":6,"last update":"Mon May 22 15:27:36 2023","interfaces":[{"interface":"back","average":{"1min":0.93400000000000005,"5min":0.93400000000000005,"15min":0.93400000000000005},"min":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"max":{"1min":1.28,"5min":1.28,"15min":1.28},"last":1.151},{"interface":"front","average":{"1min":1.1120000000000001,"5min":1.1120000000000001,"15min":1.1120000000000001},"min":{"1min":0.621,"5min":0.621,"15min":0.621},"max":{"1min":1.8180000000000001,"5min":1.8180000000000001,"15min":1.8180000000000001},"last":0.77400000000000002}]},{"osd":7,"last update":"Mon May 22 15:27:36 2023","interfaces":[{"interface":"back","average":{"1min":1.117,"5min":1.117,"15min":1.117},"min":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"max":{"1min":1.9079999999999999,"5min":1.9079999999999999,"15min":1.9079999999999999},"last":1.0880000000000001},{"interface":"front","average":{"1min":1.0349999999999999,"5min":1.0349999999999999,"15min":1.0349999999999999},"min":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"max":{"1min":1.5780000000000001,"5min":1.5780000000000001,"15min":1.5780000000000001},"last":0.69899999999999995}]}]},{"osd":0,"up_from":13,"seq":55834574871,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":13688,"kb_used_data":1008,"kb_used_omap":0,"kb_used_meta":12672,"kb_avail":94358152,"statfs":{"total":96636764160,"available":96622747648,"internally_reserved":0,"allocated":1032192,"data_stored":781293,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":12976128},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon May 22 15:27:37 2023","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":1.1839999999999999,"5min":1.1839999999999999,"15min":1.1839999999999999},"last":0.317},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":1.1040000000000001,"5min":1.1040000000000001,"15min":1.1040000000000001},"last":0.39800000000000002}]},{"osd":2,"last update":"Mon May 22 15:27:37 2023","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":1.23,"5min":1.23,"15min":1.23},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":1.1639999999999999,"5min":1.1639999999999999,"15min":1.1639999999999999},"last":0.47799999999999998}]},{"osd":3,"last update":"Mon May 22 15:27:37 2023","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":1.2410000000000001,"5min":1.2410000000000001,"15min":1.2410000000000001},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":1.119,"5min":1.119,"15min":1.119},"last":0.45000000000000001}]},{"osd":4,"last update":"Mon May 22 15:27:37 2023","interfaces":[{"interface":"back","average":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"min":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":1.2969999999999999,"5min":1.2969999999999999,"15min":1.2969999999999999},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"min":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"max":{"1min":1.4219999999999999,"5min":1.4219999999999999,"15min":1.4219999999999999},"last":0.79900000000000004}]},{"osd":5,"last update":"Mon May 22 15:27:37 2023","interfaces":[{"interface":"back","average":{"1min":0.999,"5min":0.999,"15min":0.999},"min":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":1.4370000000000001,"5min":1.4370000000000001,"15min":1.4370000000000001},"last":0.81799999999999995},{"interface":"front","average":{"1min":0.95099999999999996,"5min":0.95099999999999996,"15min":0.95099999999999996},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.349,"5min":1.349,"15min":1.349},"last":0.68200000000000005}]},{"osd":6,"last update":"Mon May 22 15:27:37 2023","interfaces":[{"interface":"back","average":{"1min":0.94599999999999995,"5min":0.94599999999999995,"15min":0.94599999999999995},"min":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"max":{"1min":1.2769999999999999,"5min":1.2769999999999999,"15min":1.2769999999999999},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.96899999999999997,"5min":0.96899999999999997,"15min":0.96899999999999997},"min":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":1.492,"5min":1.492,"15min":1.492},"last":0.627}]},{"osd":7,"last update":"Mon May 22 15:27:37 2023","interfaces":[{"interface":"back","average":{"1min":1.04,"5min":1.04,"15min":1.04},"min":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":1.581,"5min":1.581,"15min":1.581},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.98099999999999998,"5min":0.98099999999999998,"15min":0.98099999999999998},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.448,"5min":1.448,"15min":1.448},"last":0.84599999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-05-22T15:28:16.352 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-05-22T15:28:16.517 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.511+0000 7f6a0554e640 1 Processor -- start 2023-05-22T15:28:16.517 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.511+0000 7f6a0554e640 1 -- start start 2023-05-22T15:28:16.518 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.511+0000 7f6a0554e640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f6a000ca4e0 0x7f6a0015e460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:16.518 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.511+0000 7f6a0554e640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6a00070df0 0x7f6a00071210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:16.518 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.511+0000 7f6a0554e640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a000b0650 con 0x7f6a000c9b40 2023-05-22T15:28:16.518 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.511+0000 7f6a0554e640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a000b0370 con 0x7f6a00070df0 2023-05-22T15:28:16.518 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.511+0000 7f6a0554e640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a0015d980 con 0x7f6a000ca4e0 2023-05-22T15:28:16.519 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.511+0000 7f69ff7fe640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6a00070df0 0x7f6a00071210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:16.519 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.511+0000 7f69ff7fe640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6a00070df0 0x7f6a00071210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42824/0 (socket says 172.21.15.16:42824) 2023-05-22T15:28:16.519 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.511+0000 7f69ff7fe640 1 -- 172.21.15.16:0/1783555932 learned_addr learned my addr 172.21.15.16:0/1783555932 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:28:16.519 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.511+0000 7f69feffd640 1 --2- 172.21.15.16:0/1783555932 >> v2:172.21.15.62:3300/0 conn(0x7f6a000ca4e0 0x7f6a0015e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:16.519 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.511+0000 7f69ff7fe640 1 -- 172.21.15.16:0/1783555932 >> v2:172.21.15.62:3300/0 conn(0x7f6a000ca4e0 msgr2=0x7f6a0015e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:28:16.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.511+0000 7f69ff7fe640 1 --2- 172.21.15.16:0/1783555932 >> v2:172.21.15.62:3300/0 conn(0x7f6a000ca4e0 0x7f6a0015e460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:16.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.511+0000 7f69ff7fe640 1 -- 172.21.15.16:0/1783555932 >> v1:172.21.15.62:6789/0 conn(0x7f6a000c9b40 legacy=0x7f6a000afb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:28:16.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.511+0000 7f69ff7fe640 1 -- 172.21.15.16:0/1783555932 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a0015dc60 con 0x7f6a00070df0 2023-05-22T15:28:16.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.515+0000 7f69ff7fe640 1 --2- 172.21.15.16:0/1783555932 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a00070df0 0x7f6a00071210 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f6a000cad40 tx=0x7f69f801c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff8b8562c39a0fe1 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:28:16.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.515+0000 7f69fdffb640 1 -- 172.21.15.16:0/1783555932 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69f80a5780 con 0x7f6a00070df0 2023-05-22T15:28:16.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.515+0000 7f69fdffb640 1 -- 172.21.15.16:0/1783555932 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69f80a58e0 con 0x7f6a00070df0 2023-05-22T15:28:16.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.515+0000 7f69fdffb640 1 -- 172.21.15.16:0/1783555932 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69f80a5c70 con 0x7f6a00070df0 2023-05-22T15:28:16.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.515+0000 7f6a0554e640 1 -- 172.21.15.16:0/1783555932 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a00070df0 msgr2=0x7f6a00071210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:16.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.515+0000 7f6a0554e640 1 --2- 172.21.15.16:0/1783555932 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a00070df0 0x7f6a00071210 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f6a000cad40 tx=0x7f69f801c960 comp rx=0 tx=0).stop 2023-05-22T15:28:16.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.515+0000 7f6a0554e640 1 -- 172.21.15.16:0/1783555932 shutdown_connections 2023-05-22T15:28:16.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.515+0000 7f6a0554e640 1 --2- 172.21.15.16:0/1783555932 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a00070df0 0x7f6a00071210 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:16.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.515+0000 7f6a0554e640 1 --2- 172.21.15.16:0/1783555932 >> v2:172.21.15.62:3300/0 conn(0x7f6a000ca4e0 0x7f6a0015e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:16.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.515+0000 7f6a0554e640 1 -- 172.21.15.16:0/1783555932 >> 172.21.15.16:0/1783555932 conn(0x7f6a00069720 msgr2=0x7f6a00069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:28:16.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.515+0000 7f6a0554e640 1 -- 172.21.15.16:0/1783555932 shutdown_connections 2023-05-22T15:28:16.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.515+0000 7f6a0554e640 1 -- 172.21.15.16:0/1783555932 wait complete. 2023-05-22T15:28:16.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.515+0000 7f6a0554e640 1 Processor -- start 2023-05-22T15:28:16.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.515+0000 7f6a0554e640 1 -- start start 2023-05-22T15:28:16.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.519+0000 7f6a0554e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a00070df0 0x7f6a00151bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:16.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.519+0000 7f6a0554e640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6a000c9b40 0x7f6a001502a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:16.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.519+0000 7f6a0554e640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6a000ca4e0 0x7f6a001533a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:16.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.519+0000 7f6a0554e640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a00153150 con 0x7f6a00070df0 2023-05-22T15:28:16.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.519+0000 7f6a0554e640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a001517b0 con 0x7f6a000ca4e0 2023-05-22T15:28:16.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.519+0000 7f6a0554e640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a001511e0 con 0x7f6a000c9b40 2023-05-22T15:28:16.526 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.519+0000 7f69feffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a00070df0 0x7f6a00151bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:16.526 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.519+0000 7f69feffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a00070df0 0x7f6a00151bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42832/0 (socket says 172.21.15.16:42832) 2023-05-22T15:28:16.526 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.519+0000 7f69feffd640 1 -- 172.21.15.16:0/717376304 learned_addr learned my addr 172.21.15.16:0/717376304 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:28:16.527 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.519+0000 7f69fe7fc640 1 --2- 172.21.15.16:0/717376304 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6a000c9b40 0x7f6a001502a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:16.527 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.519+0000 7f69ff7fe640 1 --2- 172.21.15.16:0/717376304 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6a000ca4e0 0x7f6a001533a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:16.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.519+0000 7f69feffd640 1 -- 172.21.15.16:0/717376304 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6a000c9b40 msgr2=0x7f6a001502a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:16.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.519+0000 7f69feffd640 1 --2- 172.21.15.16:0/717376304 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6a000c9b40 0x7f6a001502a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:16.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.519+0000 7f69feffd640 1 -- 172.21.15.16:0/717376304 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6a000ca4e0 msgr2=0x7f6a001533a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:16.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.519+0000 7f69feffd640 1 --2- 172.21.15.16:0/717376304 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6a000ca4e0 0x7f6a001533a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:16.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.519+0000 7f69feffd640 1 -- 172.21.15.16:0/717376304 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69f8009b30 con 0x7f6a00070df0 2023-05-22T15:28:16.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.519+0000 7f69ff7fe640 1 --2- 172.21.15.16:0/717376304 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6a000ca4e0 0x7f6a001533a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:28:16.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.519+0000 7f69feffd640 1 --2- 172.21.15.16:0/717376304 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a00070df0 0x7f6a00151bd0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f69f000d5f0 tx=0x7f69f000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:28:16.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.519+0000 7f69e77fe640 1 -- 172.21.15.16:0/717376304 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69f00125f0 con 0x7f6a00070df0 2023-05-22T15:28:16.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.519+0000 7f69e77fe640 1 -- 172.21.15.16:0/717376304 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69f0012750 con 0x7f6a00070df0 2023-05-22T15:28:16.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.519+0000 7f69e77fe640 1 -- 172.21.15.16:0/717376304 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69f0012ae0 con 0x7f6a00070df0 2023-05-22T15:28:16.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.519+0000 7f6a0554e640 1 -- 172.21.15.16:0/717376304 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a000c8e80 con 0x7f6a00070df0 2023-05-22T15:28:16.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.519+0000 7f6a0554e640 1 -- 172.21.15.16:0/717376304 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a000ca8b0 con 0x7f6a00070df0 2023-05-22T15:28:16.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.519+0000 7f6a0554e640 1 -- 172.21.15.16:0/717376304 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a000c9010 con 0x7f6a00070df0 2023-05-22T15:28:16.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.519+0000 7f69e77fe640 1 -- 172.21.15.16:0/717376304 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7f69f0012c40 con 0x7f6a00070df0 2023-05-22T15:28:16.530 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.519+0000 7f69e77fe640 1 --2- 172.21.15.16:0/717376304 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f69ec065380 0x7f69ec067830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:16.530 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.519+0000 7f69e77fe640 1 -- 172.21.15.16:0/717376304 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f69f00838d0 con 0x7f6a00070df0 2023-05-22T15:28:16.530 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.519+0000 7f69fe7fc640 1 --2- 172.21.15.16:0/717376304 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f69ec065380 0x7f69ec067830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:16.530 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.519+0000 7f69fe7fc640 1 --2- 172.21.15.16:0/717376304 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f69ec065380 0x7f69ec067830 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f6a00164c40 tx=0x7f69f4009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:28:16.531 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.523+0000 7f69e77fe640 1 -- 172.21.15.16:0/717376304 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f69f00514a0 con 0x7f6a00070df0 2023-05-22T15:28:16.681 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.675+0000 7f6a0554e640 1 -- 172.21.15.16:0/717376304 --> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f6a00151a20 con 0x7f69ec065380 2023-05-22T15:28:16.703 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.695+0000 7f69e77fe640 1 -- 172.21.15.16:0/717376304 <== mgr.4101 v2:172.21.15.16:6832/36921 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+299300 (secure 0 0 0) 0x7f6a00151a20 con 0x7f69ec065380 2023-05-22T15:28:16.705 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:28:16.707 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2023-05-22T15:28:16.708 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.699+0000 7f6a0554e640 1 -- 172.21.15.16:0/717376304 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f69ec065380 msgr2=0x7f69ec067830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:16.708 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.699+0000 7f6a0554e640 1 --2- 172.21.15.16:0/717376304 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f69ec065380 0x7f69ec067830 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f6a00164c40 tx=0x7f69f4009040 comp rx=0 tx=0).stop 2023-05-22T15:28:16.708 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.699+0000 7f6a0554e640 1 -- 172.21.15.16:0/717376304 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a00070df0 msgr2=0x7f6a00151bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:16.708 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.699+0000 7f6a0554e640 1 --2- 172.21.15.16:0/717376304 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a00070df0 0x7f6a00151bd0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f69f000d5f0 tx=0x7f69f000dab0 comp rx=0 tx=0).stop 2023-05-22T15:28:16.709 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.699+0000 7f6a0554e640 1 -- 172.21.15.16:0/717376304 shutdown_connections 2023-05-22T15:28:16.709 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.699+0000 7f6a0554e640 1 --2- 172.21.15.16:0/717376304 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6a000ca4e0 0x7f6a001533a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:16.709 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.699+0000 7f6a0554e640 1 --2- 172.21.15.16:0/717376304 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f69ec065380 0x7f69ec067830 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:16.709 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.699+0000 7f6a0554e640 1 --2- 172.21.15.16:0/717376304 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6a000c9b40 0x7f6a001502a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:16.709 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.699+0000 7f6a0554e640 1 --2- 172.21.15.16:0/717376304 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a00070df0 0x7f6a00151bd0 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:16.709 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.699+0000 7f6a0554e640 1 -- 172.21.15.16:0/717376304 >> 172.21.15.16:0/717376304 conn(0x7f6a00069720 msgr2=0x7f6a0006bf30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:28:16.709 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.699+0000 7f6a0554e640 1 -- 172.21.15.16:0/717376304 shutdown_connections 2023-05-22T15:28:16.710 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.699+0000 7f6a0554e640 1 -- 172.21.15.16:0/717376304 wait complete. 2023-05-22T15:28:16.727 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":85,"stamp":"2023-05-22T15:28:15.972192+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":82408,"kb_used_data":4584,"kb_used_omap":0,"kb_used_meta":77760,"kb_avail":754892312,"statfs":{"total":773094113280,"available":773009727488,"internally_reserved":0,"allocated":4694016,"data_stored":2708136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":79626240},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.040506"},"pg_stats":[{"pgid":"7.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2023-05-22T15:28:11.938772+0000","last_change":"2023-05-22T15:28:11.938772+0000","last_active":"2023-05-22T15:28:11.927105+0000","last_peered":"2023-05-22T15:28:11.927105+0000","last_clean":"2023-05-22T15:28:11.927105+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:11.938772+0000","last_undegraded":"2023-05-22T15:28:11.938772+0000","last_fullsized":"2023-05-22T15:28:11.938772+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_clean_scrub_stamp":"2023-05-22T15:28:11.927105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T18:33:01.747002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2023-05-22T15:28:11.942902+0000","last_change":"2023-05-22T15:28:11.942902+0000","last_active":"2023-05-22T15:28:11.927105+0000","last_peered":"2023-05-22T15:28:11.927105+0000","last_clean":"2023-05-22T15:28:11.927105+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:11.942902+0000","last_undegraded":"2023-05-22T15:28:11.942902+0000","last_fullsized":"2023-05-22T15:28:11.942902+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_clean_scrub_stamp":"2023-05-22T15:28:11.927105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:21:15.037766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2023-05-22T15:28:11.945096+0000","last_change":"2023-05-22T15:28:11.945096+0000","last_active":"2023-05-22T15:28:11.927105+0000","last_peered":"2023-05-22T15:28:11.927105+0000","last_clean":"2023-05-22T15:28:11.927105+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:11.945096+0000","last_undegraded":"2023-05-22T15:28:11.945096+0000","last_fullsized":"2023-05-22T15:28:11.945096+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_clean_scrub_stamp":"2023-05-22T15:28:11.927105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:02:28.198798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2023-05-22T15:28:11.947443+0000","last_change":"2023-05-22T15:28:11.947443+0000","last_active":"2023-05-22T15:28:11.927105+0000","last_peered":"2023-05-22T15:28:11.927105+0000","last_clean":"2023-05-22T15:28:11.927105+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:11.947443+0000","last_undegraded":"2023-05-22T15:28:11.947443+0000","last_fullsized":"2023-05-22T15:28:11.947443+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_clean_scrub_stamp":"2023-05-22T15:28:11.927105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:42:40.893204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2023-05-22T15:28:11.941543+0000","last_change":"2023-05-22T15:28:11.941543+0000","last_active":"2023-05-22T15:28:11.927105+0000","last_peered":"2023-05-22T15:28:11.927105+0000","last_clean":"2023-05-22T15:28:11.927105+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:11.941543+0000","last_undegraded":"2023-05-22T15:28:11.941543+0000","last_fullsized":"2023-05-22T15:28:11.941543+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_clean_scrub_stamp":"2023-05-22T15:28:11.927105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:05:30.503639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2023-05-22T15:28:11.942842+0000","last_change":"2023-05-22T15:28:11.942842+0000","last_active":"2023-05-22T15:28:11.927105+0000","last_peered":"2023-05-22T15:28:11.927105+0000","last_clean":"2023-05-22T15:28:11.927105+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:11.942842+0000","last_undegraded":"2023-05-22T15:28:11.942842+0000","last_fullsized":"2023-05-22T15:28:11.942842+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_clean_scrub_stamp":"2023-05-22T15:28:11.927105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:17:33.835222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2023-05-22T15:28:11.948423+0000","last_change":"2023-05-22T15:28:11.948423+0000","last_active":"2023-05-22T15:28:11.927105+0000","last_peered":"2023-05-22T15:28:11.927105+0000","last_clean":"2023-05-22T15:28:11.927105+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:11.948423+0000","last_undegraded":"2023-05-22T15:28:11.948423+0000","last_fullsized":"2023-05-22T15:28:11.948423+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_clean_scrub_stamp":"2023-05-22T15:28:11.927105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:45:43.272774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2023-05-22T15:28:11.942591+0000","last_change":"2023-05-22T15:28:11.942591+0000","last_active":"2023-05-22T15:28:11.927105+0000","last_peered":"2023-05-22T15:28:11.927105+0000","last_clean":"2023-05-22T15:28:11.927105+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:11.942591+0000","last_undegraded":"2023-05-22T15:28:11.942591+0000","last_fullsized":"2023-05-22T15:28:11.942591+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_clean_scrub_stamp":"2023-05-22T15:28:11.927105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:02:28.198798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2023-05-22T15:28:11.941674+0000","last_change":"2023-05-22T15:28:11.941674+0000","last_active":"2023-05-22T15:28:11.927105+0000","last_peered":"2023-05-22T15:28:11.927105+0000","last_clean":"2023-05-22T15:28:11.927105+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:11.941674+0000","last_undegraded":"2023-05-22T15:28:11.941674+0000","last_fullsized":"2023-05-22T15:28:11.941674+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_clean_scrub_stamp":"2023-05-22T15:28:11.927105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:50:22.466750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2023-05-22T15:28:11.940470+0000","last_change":"2023-05-22T15:28:11.940470+0000","last_active":"2023-05-22T15:28:11.927105+0000","last_peered":"2023-05-22T15:28:11.927105+0000","last_clean":"2023-05-22T15:28:11.927105+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:11.940470+0000","last_undegraded":"2023-05-22T15:28:11.940470+0000","last_fullsized":"2023-05-22T15:28:11.940470+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_clean_scrub_stamp":"2023-05-22T15:28:11.927105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:15:50.706827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2023-05-22T15:28:11.950095+0000","last_change":"2023-05-22T15:28:11.950095+0000","last_active":"2023-05-22T15:28:11.927105+0000","last_peered":"2023-05-22T15:28:11.927105+0000","last_clean":"2023-05-22T15:28:11.927105+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:11.950095+0000","last_undegraded":"2023-05-22T15:28:11.950095+0000","last_fullsized":"2023-05-22T15:28:11.950095+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_clean_scrub_stamp":"2023-05-22T15:28:11.927105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:57:04.738345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2023-05-22T15:28:11.935772+0000","last_change":"2023-05-22T15:28:11.935772+0000","last_active":"2023-05-22T15:28:11.927105+0000","last_peered":"2023-05-22T15:28:11.927105+0000","last_clean":"2023-05-22T15:28:11.927105+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:11.935772+0000","last_undegraded":"2023-05-22T15:28:11.935772+0000","last_fullsized":"2023-05-22T15:28:11.935772+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_clean_scrub_stamp":"2023-05-22T15:28:11.927105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:30:08.027313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2023-05-22T15:28:11.938777+0000","last_change":"2023-05-22T15:28:11.938777+0000","last_active":"2023-05-22T15:28:11.927105+0000","last_peered":"2023-05-22T15:28:11.927105+0000","last_clean":"2023-05-22T15:28:11.927105+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:11.938777+0000","last_undegraded":"2023-05-22T15:28:11.938777+0000","last_fullsized":"2023-05-22T15:28:11.938777+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_clean_scrub_stamp":"2023-05-22T15:28:11.927105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:47:11.442121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2023-05-22T15:28:11.949598+0000","last_change":"2023-05-22T15:28:11.949598+0000","last_active":"2023-05-22T15:28:11.927105+0000","last_peered":"2023-05-22T15:28:11.927105+0000","last_clean":"2023-05-22T15:28:11.927105+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:11.949598+0000","last_undegraded":"2023-05-22T15:28:11.949598+0000","last_fullsized":"2023-05-22T15:28:11.949598+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_clean_scrub_stamp":"2023-05-22T15:28:11.927105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:26:26.763456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2023-05-22T15:28:11.938244+0000","last_change":"2023-05-22T15:28:11.938244+0000","last_active":"2023-05-22T15:28:11.927105+0000","last_peered":"2023-05-22T15:28:11.927105+0000","last_clean":"2023-05-22T15:28:11.927105+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:11.938244+0000","last_undegraded":"2023-05-22T15:28:11.938244+0000","last_fullsized":"2023-05-22T15:28:11.938244+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_clean_scrub_stamp":"2023-05-22T15:28:11.927105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:09:59.757240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2023-05-22T15:28:11.941202+0000","last_change":"2023-05-22T15:28:11.941202+0000","last_active":"2023-05-22T15:28:11.927105+0000","last_peered":"2023-05-22T15:28:11.927105+0000","last_clean":"2023-05-22T15:28:11.927105+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:11.941202+0000","last_undegraded":"2023-05-22T15:28:11.941202+0000","last_fullsized":"2023-05-22T15:28:11.941202+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_clean_scrub_stamp":"2023-05-22T15:28:11.927105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T15:35:29.715819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2023-05-22T15:28:11.935588+0000","last_change":"2023-05-22T15:28:11.935588+0000","last_active":"2023-05-22T15:28:11.927105+0000","last_peered":"2023-05-22T15:28:11.927105+0000","last_clean":"2023-05-22T15:28:11.927105+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:11.935588+0000","last_undegraded":"2023-05-22T15:28:11.935588+0000","last_fullsized":"2023-05-22T15:28:11.935588+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_clean_scrub_stamp":"2023-05-22T15:28:11.927105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:10:24.156615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2023-05-22T15:28:11.941609+0000","last_change":"2023-05-22T15:28:11.941609+0000","last_active":"2023-05-22T15:28:11.927105+0000","last_peered":"2023-05-22T15:28:11.927105+0000","last_clean":"2023-05-22T15:28:11.927105+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:11.941609+0000","last_undegraded":"2023-05-22T15:28:11.941609+0000","last_fullsized":"2023-05-22T15:28:11.941609+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_clean_scrub_stamp":"2023-05-22T15:28:11.927105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T03:27:50.363592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2023-05-22T15:28:11.947845+0000","last_change":"2023-05-22T15:28:11.947845+0000","last_active":"2023-05-22T15:28:11.927105+0000","last_peered":"2023-05-22T15:28:11.927105+0000","last_clean":"2023-05-22T15:28:11.927105+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:11.947845+0000","last_undegraded":"2023-05-22T15:28:11.947845+0000","last_fullsized":"2023-05-22T15:28:11.947845+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_clean_scrub_stamp":"2023-05-22T15:28:11.927105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:57:04.738345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2023-05-22T15:28:11.938489+0000","last_change":"2023-05-22T15:28:11.938489+0000","last_active":"2023-05-22T15:28:11.927105+0000","last_peered":"2023-05-22T15:28:11.927105+0000","last_clean":"2023-05-22T15:28:11.927105+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:11.938489+0000","last_undegraded":"2023-05-22T15:28:11.938489+0000","last_fullsized":"2023-05-22T15:28:11.938489+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_clean_scrub_stamp":"2023-05-22T15:28:11.927105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:45:43.272774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2023-05-22T15:28:11.940183+0000","last_change":"2023-05-22T15:28:11.940183+0000","last_active":"2023-05-22T15:28:11.927105+0000","last_peered":"2023-05-22T15:28:11.927105+0000","last_clean":"2023-05-22T15:28:11.927105+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:11.940183+0000","last_undegraded":"2023-05-22T15:28:11.940183+0000","last_fullsized":"2023-05-22T15:28:11.940183+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_clean_scrub_stamp":"2023-05-22T15:28:11.927105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:50:22.466750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2023-05-22T15:28:11.947990+0000","last_change":"2023-05-22T15:28:11.947990+0000","last_active":"2023-05-22T15:28:11.927105+0000","last_peered":"2023-05-22T15:28:11.927105+0000","last_clean":"2023-05-22T15:28:11.927105+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:11.947990+0000","last_undegraded":"2023-05-22T15:28:11.947990+0000","last_fullsized":"2023-05-22T15:28:11.947990+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_clean_scrub_stamp":"2023-05-22T15:28:11.927105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T18:18:35.201229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2023-05-22T15:28:11.940527+0000","last_change":"2023-05-22T15:28:11.940527+0000","last_active":"2023-05-22T15:28:11.927105+0000","last_peered":"2023-05-22T15:28:11.927105+0000","last_clean":"2023-05-22T15:28:11.927105+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:11.940527+0000","last_undegraded":"2023-05-22T15:28:11.940527+0000","last_fullsized":"2023-05-22T15:28:11.940527+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_clean_scrub_stamp":"2023-05-22T15:28:11.927105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:24:48.246881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2023-05-22T15:28:11.947499+0000","last_change":"2023-05-22T15:28:11.947499+0000","last_active":"2023-05-22T15:28:11.927105+0000","last_peered":"2023-05-22T15:28:11.927105+0000","last_clean":"2023-05-22T15:28:11.927105+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:11.947499+0000","last_undegraded":"2023-05-22T15:28:11.947499+0000","last_fullsized":"2023-05-22T15:28:11.947499+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_clean_scrub_stamp":"2023-05-22T15:28:11.927105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T18:18:35.201229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2023-05-22T15:28:11.947536+0000","last_change":"2023-05-22T15:28:11.947536+0000","last_active":"2023-05-22T15:28:11.927105+0000","last_peered":"2023-05-22T15:28:11.927105+0000","last_clean":"2023-05-22T15:28:11.927105+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:11.947536+0000","last_undegraded":"2023-05-22T15:28:11.947536+0000","last_fullsized":"2023-05-22T15:28:11.947536+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_clean_scrub_stamp":"2023-05-22T15:28:11.927105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:15:37.287885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":25,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-05-22T15:28:08.260614+0000","last_change":"2023-05-22T15:26:44.982421+0000","last_active":"2023-05-22T15:28:08.260614+0000","last_peered":"2023-05-22T15:28:08.260614+0000","last_clean":"2023-05-22T15:28:08.260614+0000","last_became_active":"2023-05-22T15:26:44.981586+0000","last_became_peered":"2023-05-22T15:26:44.981586+0000","last_unstale":"2023-05-22T15:28:08.260614+0000","last_undegraded":"2023-05-22T15:28:08.260614+0000","last_fullsized":"2023-05-22T15:28:08.260614+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:14:11.156544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2023-05-22T15:28:11.941443+0000","last_change":"2023-05-22T15:28:11.941443+0000","last_active":"2023-05-22T15:28:11.927105+0000","last_peered":"2023-05-22T15:28:11.927105+0000","last_clean":"2023-05-22T15:28:11.927105+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:11.941443+0000","last_undegraded":"2023-05-22T15:28:11.941443+0000","last_fullsized":"2023-05-22T15:28:11.941443+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_clean_scrub_stamp":"2023-05-22T15:28:11.927105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:05:30.503639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":24,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-05-22T15:28:07.758852+0000","last_change":"2023-05-22T15:26:44.978858+0000","last_active":"2023-05-22T15:28:07.758852+0000","last_peered":"2023-05-22T15:28:07.758852+0000","last_clean":"2023-05-22T15:28:07.758852+0000","last_became_active":"2023-05-22T15:26:44.978443+0000","last_became_peered":"2023-05-22T15:26:44.978443+0000","last_unstale":"2023-05-22T15:28:07.758852+0000","last_undegraded":"2023-05-22T15:28:07.758852+0000","last_fullsized":"2023-05-22T15:28:07.758852+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:46:29.290243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2023-05-22T15:28:11.945876+0000","last_change":"2023-05-22T15:28:11.945876+0000","last_active":"2023-05-22T15:28:11.927105+0000","last_peered":"2023-05-22T15:28:11.927105+0000","last_clean":"2023-05-22T15:28:11.927105+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:11.945876+0000","last_undegraded":"2023-05-22T15:28:11.945876+0000","last_fullsized":"2023-05-22T15:28:11.945876+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_clean_scrub_stamp":"2023-05-22T15:28:11.927105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:43:14.376663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":28,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-05-22T15:28:07.681164+0000","last_change":"2023-05-22T15:26:45.092367+0000","last_active":"2023-05-22T15:28:07.681164+0000","last_peered":"2023-05-22T15:28:07.681164+0000","last_clean":"2023-05-22T15:28:07.681164+0000","last_became_active":"2023-05-22T15:26:45.091951+0000","last_became_peered":"2023-05-22T15:26:45.091951+0000","last_unstale":"2023-05-22T15:28:07.681164+0000","last_undegraded":"2023-05-22T15:28:07.681164+0000","last_fullsized":"2023-05-22T15:28:07.681164+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:42:57.973030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2023-05-22T15:28:11.940327+0000","last_change":"2023-05-22T15:28:11.940327+0000","last_active":"2023-05-22T15:28:11.927105+0000","last_peered":"2023-05-22T15:28:11.927105+0000","last_clean":"2023-05-22T15:28:11.927105+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:11.940327+0000","last_undegraded":"2023-05-22T15:28:11.940327+0000","last_fullsized":"2023-05-22T15:28:11.940327+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_clean_scrub_stamp":"2023-05-22T15:28:11.927105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:23:05.013217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":24,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-05-22T15:28:07.758858+0000","last_change":"2023-05-22T15:26:45.091475+0000","last_active":"2023-05-22T15:28:07.758858+0000","last_peered":"2023-05-22T15:28:07.758858+0000","last_clean":"2023-05-22T15:28:07.758858+0000","last_became_active":"2023-05-22T15:26:45.091143+0000","last_became_peered":"2023-05-22T15:26:45.091143+0000","last_unstale":"2023-05-22T15:28:07.758858+0000","last_undegraded":"2023-05-22T15:28:07.758858+0000","last_fullsized":"2023-05-22T15:28:07.758858+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:12:41.464054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2023-05-22T15:28:11.938283+0000","last_change":"2023-05-22T15:28:11.938283+0000","last_active":"2023-05-22T15:28:11.927105+0000","last_peered":"2023-05-22T15:28:11.927105+0000","last_clean":"2023-05-22T15:28:11.927105+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:11.938283+0000","last_undegraded":"2023-05-22T15:28:11.938283+0000","last_fullsized":"2023-05-22T15:28:11.938283+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_clean_scrub_stamp":"2023-05-22T15:28:11.927105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:21:15.037766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":29,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-05-22T15:28:07.680251+0000","last_change":"2023-05-22T15:26:44.976632+0000","last_active":"2023-05-22T15:28:07.680251+0000","last_peered":"2023-05-22T15:28:07.680251+0000","last_clean":"2023-05-22T15:28:07.680251+0000","last_became_active":"2023-05-22T15:26:44.975546+0000","last_became_peered":"2023-05-22T15:26:44.975546+0000","last_unstale":"2023-05-22T15:28:07.680251+0000","last_undegraded":"2023-05-22T15:28:07.680251+0000","last_fullsized":"2023-05-22T15:28:07.680251+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:42:57.973030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2023-05-22T15:28:11.948734+0000","last_change":"2023-05-22T15:28:11.948734+0000","last_active":"2023-05-22T15:28:11.927105+0000","last_peered":"2023-05-22T15:28:11.927105+0000","last_clean":"2023-05-22T15:28:11.927105+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:11.948734+0000","last_undegraded":"2023-05-22T15:28:11.948734+0000","last_fullsized":"2023-05-22T15:28:11.948734+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_clean_scrub_stamp":"2023-05-22T15:28:11.927105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:45:43.272774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":28,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-05-22T15:28:07.678593+0000","last_change":"2023-05-22T15:26:44.969469+0000","last_active":"2023-05-22T15:28:07.678593+0000","last_peered":"2023-05-22T15:28:07.678593+0000","last_clean":"2023-05-22T15:28:07.678593+0000","last_became_active":"2023-05-22T15:26:44.969083+0000","last_became_peered":"2023-05-22T15:26:44.969083+0000","last_unstale":"2023-05-22T15:28:07.678593+0000","last_undegraded":"2023-05-22T15:28:07.678593+0000","last_fullsized":"2023-05-22T15:28:07.678593+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:29:23.670067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2023-05-22T15:28:11.945024+0000","last_change":"2023-05-22T15:28:11.945024+0000","last_active":"2023-05-22T15:28:11.927105+0000","last_peered":"2023-05-22T15:28:11.927105+0000","last_clean":"2023-05-22T15:28:11.927105+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:11.945024+0000","last_undegraded":"2023-05-22T15:28:11.945024+0000","last_fullsized":"2023-05-22T15:28:11.945024+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_clean_scrub_stamp":"2023-05-22T15:28:11.927105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:23:05.013217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":28,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-05-22T15:28:07.681164+0000","last_change":"2023-05-22T15:26:44.977198+0000","last_active":"2023-05-22T15:28:07.681164+0000","last_peered":"2023-05-22T15:28:07.681164+0000","last_clean":"2023-05-22T15:28:07.681164+0000","last_became_active":"2023-05-22T15:26:44.976445+0000","last_became_peered":"2023-05-22T15:26:44.976445+0000","last_unstale":"2023-05-22T15:28:07.681164+0000","last_undegraded":"2023-05-22T15:28:07.681164+0000","last_fullsized":"2023-05-22T15:28:07.681164+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:12:41.464054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2023-05-22T15:28:11.947908+0000","last_change":"2023-05-22T15:28:11.947908+0000","last_active":"2023-05-22T15:28:11.927105+0000","last_peered":"2023-05-22T15:28:11.927105+0000","last_clean":"2023-05-22T15:28:11.927105+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:11.947908+0000","last_undegraded":"2023-05-22T15:28:11.947908+0000","last_fullsized":"2023-05-22T15:28:11.947908+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_clean_scrub_stamp":"2023-05-22T15:28:11.927105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:15:37.287885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":29,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-05-22T15:28:07.680257+0000","last_change":"2023-05-22T15:26:44.971951+0000","last_active":"2023-05-22T15:28:07.680257+0000","last_peered":"2023-05-22T15:28:07.680257+0000","last_clean":"2023-05-22T15:28:07.680257+0000","last_became_active":"2023-05-22T15:26:44.970872+0000","last_became_peered":"2023-05-22T15:26:44.970872+0000","last_unstale":"2023-05-22T15:28:07.680257+0000","last_undegraded":"2023-05-22T15:28:07.680257+0000","last_fullsized":"2023-05-22T15:28:07.680257+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:55:34.752949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":181,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-05-22T15:28:07.681354+0000","last_change":"2023-05-22T15:26:39.318204+0000","last_active":"2023-05-22T15:28:07.681354+0000","last_peered":"2023-05-22T15:28:07.681354+0000","last_clean":"2023-05-22T15:28:07.681354+0000","last_became_active":"2023-05-22T15:26:39.315370+0000","last_became_peered":"2023-05-22T15:26:39.315370+0000","last_unstale":"2023-05-22T15:28:07.681354+0000","last_undegraded":"2023-05-22T15:28:07.681354+0000","last_fullsized":"2023-05-22T15:28:07.681354+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:38.020076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:38.020076+0000","last_clean_scrub_stamp":"2023-05-22T15:26:38.020076+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:06:39.207543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":7},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574870,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":13752,"kb_used_data":1008,"kb_used_omap":0,"kb_used_meta":12736,"kb_avail":94358088,"statfs":{"total":96636764160,"available":96622682112,"internally_reserved":0,"allocated":1032192,"data_stored":781293,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":13041664},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 15:27:38 2023","interfaces":[{"interface":"back","average":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":1.5860000000000001,"5min":1.5860000000000001,"15min":1.5860000000000001},"last":0.77700000000000002},{"interface":"front","average":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"min":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.4239999999999999,"5min":1.4239999999999999,"15min":1.4239999999999999},"last":0.55700000000000005}]},{"osd":1,"last update":"Mon May 22 15:27:38 2023","interfaces":[{"interface":"back","average":{"1min":0.96599999999999997,"5min":0.96599999999999997,"15min":0.96599999999999997},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":1.3600000000000001,"5min":1.3600000000000001,"15min":1.3600000000000001},"last":1.002},{"interface":"front","average":{"1min":1.002,"5min":1.002,"15min":1.002},"min":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"max":{"1min":1.8260000000000001,"5min":1.8260000000000001,"15min":1.8260000000000001},"last":1.208}]},{"osd":2,"last update":"Mon May 22 15:27:38 2023","interfaces":[{"interface":"back","average":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.127,"5min":1.127,"15min":1.127},"last":0.90100000000000002},{"interface":"front","average":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":1.23,"5min":1.23,"15min":1.23},"last":0.83999999999999997}]},{"osd":3,"last update":"Mon May 22 15:27:38 2023","interfaces":[{"interface":"back","average":{"1min":0.99099999999999999,"5min":0.99099999999999999,"15min":0.99099999999999999},"min":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"max":{"1min":1.7829999999999999,"5min":1.7829999999999999,"15min":1.7829999999999999},"last":1.115},{"interface":"front","average":{"1min":1.026,"5min":1.026,"15min":1.026},"min":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":1.6879999999999999,"5min":1.6879999999999999,"15min":1.6879999999999999},"last":0.70199999999999996}]},{"osd":4,"last update":"Mon May 22 15:27:38 2023","interfaces":[{"interface":"back","average":{"1min":0.92700000000000005,"5min":0.92700000000000005,"15min":0.92700000000000005},"min":{"1min":0.62,"5min":0.62,"15min":0.62},"max":{"1min":1.2430000000000001,"5min":1.2430000000000001,"15min":1.2430000000000001},"last":0.65600000000000003},{"interface":"front","average":{"1min":1.1120000000000001,"5min":1.1120000000000001,"15min":1.1120000000000001},"min":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"max":{"1min":1.73,"5min":1.73,"15min":1.73},"last":1.2889999999999999}]},{"osd":5,"last update":"Mon May 22 15:27:38 2023","interfaces":[{"interface":"back","average":{"1min":1.123,"5min":1.123,"15min":1.123},"min":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":1.3779999999999999,"5min":1.3779999999999999,"15min":1.3779999999999999},"last":1.0760000000000001},{"interface":"front","average":{"1min":1.139,"5min":1.139,"15min":1.139},"min":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"max":{"1min":1.6439999999999999,"5min":1.6439999999999999,"15min":1.6439999999999999},"last":1.0429999999999999}]},{"osd":6,"last update":"Mon May 22 15:27:38 2023","interfaces":[{"interface":"back","average":{"1min":1.1020000000000001,"5min":1.1020000000000001,"15min":1.1020000000000001},"min":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"max":{"1min":1.3999999999999999,"5min":1.3999999999999999,"15min":1.3999999999999999},"last":0.96399999999999997},{"interface":"front","average":{"1min":1.1890000000000001,"5min":1.1890000000000001,"15min":1.1890000000000001},"min":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"max":{"1min":2.04,"5min":2.04,"15min":2.04},"last":1.3620000000000001}]}]},{"osd":6,"up_from":13,"seq":55834574871,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8372,"kb_used_data":428,"kb_used_omap":0,"kb_used_meta":7936,"kb_avail":94363468,"statfs":{"total":96636764160,"available":96628191232,"internally_reserved":0,"allocated":438272,"data_stored":190925,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8126464},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 15:27:41 2023","interfaces":[{"interface":"back","average":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"min":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":1.1659999999999999,"5min":1.1659999999999999,"15min":1.1659999999999999},"last":0.97399999999999998},{"interface":"front","average":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"min":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"max":{"1min":1.385,"5min":1.385,"15min":1.385},"last":0.71499999999999997}]},{"osd":1,"last update":"Mon May 22 15:27:41 2023","interfaces":[{"interface":"back","average":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"min":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":1.2869999999999999,"5min":1.2869999999999999,"15min":1.2869999999999999},"last":0.77000000000000002},{"interface":"front","average":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":0.90000000000000002},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":1.204,"5min":1.204,"15min":1.204},"last":0.92000000000000004}]},{"osd":2,"last update":"Mon May 22 15:27:41 2023","interfaces":[{"interface":"back","average":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"min":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"max":{"1min":1.145,"5min":1.145,"15min":1.145},"last":0.84099999999999997},{"interface":"front","average":{"1min":1.0569999999999999,"5min":1.0569999999999999,"15min":1.0569999999999999},"min":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"max":{"1min":1.4670000000000001,"5min":1.4670000000000001,"15min":1.4670000000000001},"last":1.0700000000000001}]},{"osd":3,"last update":"Mon May 22 15:27:41 2023","interfaces":[{"interface":"back","average":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"min":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"max":{"1min":1.415,"5min":1.415,"15min":1.415},"last":1.0169999999999999},{"interface":"front","average":{"1min":0.95799999999999996,"5min":0.95799999999999996,"15min":0.95799999999999996},"min":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":1.486,"5min":1.486,"15min":1.486},"last":1.042}]},{"osd":4,"last update":"Mon May 22 15:27:41 2023","interfaces":[{"interface":"back","average":{"1min":1.0389999999999999,"5min":1.0389999999999999,"15min":1.0389999999999999},"min":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"max":{"1min":1.6459999999999999,"5min":1.6459999999999999,"15min":1.6459999999999999},"last":1.1539999999999999},{"interface":"front","average":{"1min":1.141,"5min":1.141,"15min":1.141},"min":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"max":{"1min":1.5469999999999999,"5min":1.5469999999999999,"15min":1.5469999999999999},"last":1.097}]},{"osd":5,"last update":"Mon May 22 15:27:41 2023","interfaces":[{"interface":"back","average":{"1min":1.075,"5min":1.075,"15min":1.075},"min":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"max":{"1min":1.621,"5min":1.621,"15min":1.621},"last":1.117},{"interface":"front","average":{"1min":1.149,"5min":1.149,"15min":1.149},"min":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"max":{"1min":1.7509999999999999,"5min":1.7509999999999999,"15min":1.7509999999999999},"last":1.232}]},{"osd":7,"last update":"Mon May 22 15:27:41 2023","interfaces":[{"interface":"back","average":{"1min":1.268,"5min":1.268,"15min":1.268},"min":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"max":{"1min":1.8120000000000001,"5min":1.8120000000000001,"15min":1.8120000000000001},"last":1.177},{"interface":"front","average":{"1min":1.179,"5min":1.179,"15min":1.179},"min":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"max":{"1min":1.6819999999999999,"5min":1.6819999999999999,"15min":1.6819999999999999},"last":1.3260000000000001}]}]},{"osd":5,"up_from":13,"seq":55834574871,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8500,"kb_used_data":428,"kb_used_omap":0,"kb_used_meta":8064,"kb_avail":94363340,"statfs":{"total":96636764160,"available":96628060160,"internally_reserved":0,"allocated":438272,"data_stored":190925,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8257536},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 15:27:37 2023","interfaces":[{"interface":"back","average":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"min":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":1.169,"5min":1.169,"15min":1.169},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"min":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.103,"5min":1.103,"15min":1.103},"last":0.68500000000000005}]},{"osd":1,"last update":"Mon May 22 15:27:37 2023","interfaces":[{"interface":"back","average":{"1min":0.90200000000000002,"5min":0.90200000000000002,"15min":0.90200000000000002},"min":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":1.2170000000000001,"5min":1.2170000000000001,"15min":1.2170000000000001},"last":1.0309999999999999},{"interface":"front","average":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"min":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":1.399,"5min":1.399,"15min":1.399},"last":1.0069999999999999}]},{"osd":2,"last update":"Mon May 22 15:27:37 2023","interfaces":[{"interface":"back","average":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"min":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":1.597,"5min":1.597,"15min":1.597},"last":0.83399999999999996},{"interface":"front","average":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"min":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.296,"5min":1.296,"15min":1.296},"last":0.79100000000000004}]},{"osd":3,"last update":"Mon May 22 15:27:37 2023","interfaces":[{"interface":"back","average":{"1min":0.93400000000000005,"5min":0.93400000000000005,"15min":0.93400000000000005},"min":{"1min":0.627,"5min":0.627,"15min":0.627},"max":{"1min":1.6839999999999999,"5min":1.6839999999999999,"15min":1.6839999999999999},"last":0.91000000000000003},{"interface":"front","average":{"1min":1.0209999999999999,"5min":1.0209999999999999,"15min":1.0209999999999999},"min":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"max":{"1min":1.5189999999999999,"5min":1.5189999999999999,"15min":1.5189999999999999},"last":1.111}]},{"osd":4,"last update":"Mon May 22 15:27:37 2023","interfaces":[{"interface":"back","average":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":0.96399999999999997},"min":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"max":{"1min":1.3140000000000001,"5min":1.3140000000000001,"15min":1.3140000000000001},"last":0.876},{"interface":"front","average":{"1min":1.0549999999999999,"5min":1.0549999999999999,"15min":1.0549999999999999},"min":{"1min":0.747,"5min":0.747,"15min":0.747},"max":{"1min":1.319,"5min":1.319,"15min":1.319},"last":1.0700000000000001}]},{"osd":6,"last update":"Mon May 22 15:27:37 2023","interfaces":[{"interface":"back","average":{"1min":1.1000000000000001,"5min":1.1000000000000001,"15min":1.1000000000000001},"min":{"1min":0.94199999999999995,"5min":0.94199999999999995,"15min":0.94199999999999995},"max":{"1min":1.357,"5min":1.357,"15min":1.357},"last":1.2190000000000001},{"interface":"front","average":{"1min":1.109,"5min":1.109,"15min":1.109},"min":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"max":{"1min":1.4350000000000001,"5min":1.4350000000000001,"15min":1.4350000000000001},"last":0.95899999999999996}]},{"osd":7,"last update":"Mon May 22 15:27:37 2023","interfaces":[{"interface":"back","average":{"1min":1.232,"5min":1.232,"15min":1.232},"min":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"max":{"1min":2.8260000000000001,"5min":2.8260000000000001,"15min":2.8260000000000001},"last":1.157},{"interface":"front","average":{"1min":1.1679999999999999,"5min":1.1679999999999999,"15min":1.1679999999999999},"min":{"1min":0.877,"5min":0.877,"15min":0.877},"max":{"1min":1.4490000000000001,"5min":1.4490000000000001,"15min":1.4490000000000001},"last":1.29}]}]},{"osd":4,"up_from":13,"seq":55834574869,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8564,"kb_used_data":428,"kb_used_omap":0,"kb_used_meta":8128,"kb_avail":94363276,"statfs":{"total":96636764160,"available":96627994624,"internally_reserved":0,"allocated":438272,"data_stored":190925,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8323072},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 15:27:37 2023","interfaces":[{"interface":"back","average":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"min":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":1.252,"5min":1.252,"15min":1.252},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"min":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"max":{"1min":1.2989999999999999,"5min":1.2989999999999999,"15min":1.2989999999999999},"last":0.56299999999999994}]},{"osd":1,"last update":"Mon May 22 15:27:37 2023","interfaces":[{"interface":"back","average":{"1min":0.872,"5min":0.872,"15min":0.872},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.3009999999999999,"5min":1.3009999999999999,"15min":1.3009999999999999},"last":1.1220000000000001},{"interface":"front","average":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":0.83299999999999996},"min":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":1.1499999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.71399999999999997}]},{"osd":2,"last update":"Mon May 22 15:27:37 2023","interfaces":[{"interface":"back","average":{"1min":0.96199999999999997,"5min":0.96199999999999997,"15min":0.96199999999999997},"min":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"max":{"1min":1.284,"5min":1.284,"15min":1.284},"last":0.78000000000000003},{"interface":"front","average":{"1min":1.0780000000000001,"5min":1.0780000000000001,"15min":1.0780000000000001},"min":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"max":{"1min":1.9770000000000001,"5min":1.9770000000000001,"15min":1.9770000000000001},"last":0.97499999999999998}]},{"osd":3,"last update":"Mon May 22 15:27:37 2023","interfaces":[{"interface":"back","average":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"min":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.4850000000000001,"5min":1.4850000000000001,"15min":1.4850000000000001},"last":0.81699999999999995},{"interface":"front","average":{"1min":1.1100000000000001,"5min":1.1100000000000001,"15min":1.1100000000000001},"min":{"1min":0.872,"5min":0.872,"15min":0.872},"max":{"1min":1.468,"5min":1.468,"15min":1.468},"last":0.86199999999999999}]},{"osd":5,"last update":"Mon May 22 15:27:37 2023","interfaces":[{"interface":"back","average":{"1min":1.123,"5min":1.123,"15min":1.123},"min":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"max":{"1min":1.7270000000000001,"5min":1.7270000000000001,"15min":1.7270000000000001},"last":1.042},{"interface":"front","average":{"1min":1.024,"5min":1.024,"15min":1.024},"min":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"max":{"1min":1.417,"5min":1.417,"15min":1.417},"last":1.0680000000000001}]},{"osd":6,"last update":"Mon May 22 15:27:37 2023","interfaces":[{"interface":"back","average":{"1min":1.181,"5min":1.181,"15min":1.181},"min":{"1min":0.96899999999999997,"5min":0.96899999999999997,"15min":0.96899999999999997},"max":{"1min":1.6020000000000001,"5min":1.6020000000000001,"15min":1.6020000000000001},"last":1.0089999999999999},{"interface":"front","average":{"1min":1.202,"5min":1.202,"15min":1.202},"min":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"max":{"1min":2.0699999999999998,"5min":2.0699999999999998,"15min":2.0699999999999998},"last":1.1890000000000001}]},{"osd":7,"last update":"Mon May 22 15:27:37 2023","interfaces":[{"interface":"back","average":{"1min":1.1919999999999999,"5min":1.1919999999999999,"15min":1.1919999999999999},"min":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.90800000000000003},"max":{"1min":1.6639999999999999,"5min":1.6639999999999999,"15min":1.6639999999999999},"last":0.90800000000000003},{"interface":"front","average":{"1min":1.2210000000000001,"5min":1.2210000000000001,"15min":1.2210000000000001},"min":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"max":{"1min":1.7410000000000001,"5min":1.7410000000000001,"15min":1.7410000000000001},"last":1.1559999999999999}]}]},{"osd":3,"up_from":13,"seq":55834574869,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8500,"kb_used_data":428,"kb_used_omap":0,"kb_used_meta":8064,"kb_avail":94363340,"statfs":{"total":96636764160,"available":96628060160,"internally_reserved":0,"allocated":438272,"data_stored":190925,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8257536},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 15:27:37 2023","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.42999999999999999}]},{"osd":1,"last update":"Mon May 22 15:27:37 2023","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.505},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":1.6579999999999999,"5min":1.6579999999999999,"15min":1.6579999999999999},"last":0.49099999999999999}]},{"osd":2,"last update":"Mon May 22 15:27:37 2023","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":1.2490000000000001,"5min":1.2490000000000001,"15min":1.2490000000000001},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":1.855,"5min":1.855,"15min":1.855},"last":0.53000000000000003}]},{"osd":4,"last update":"Mon May 22 15:27:37 2023","interfaces":[{"interface":"back","average":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"min":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":1.5820000000000001,"5min":1.5820000000000001,"15min":1.5820000000000001},"last":0.77000000000000002},{"interface":"front","average":{"1min":0.96899999999999997,"5min":0.96899999999999997,"15min":0.96899999999999997},"min":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"max":{"1min":1.8069999999999999,"5min":1.8069999999999999,"15min":1.8069999999999999},"last":0.66800000000000004}]},{"osd":5,"last update":"Mon May 22 15:27:37 2023","interfaces":[{"interface":"back","average":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":0.83799999999999997},"min":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":1.0840000000000001,"5min":1.0840000000000001,"15min":1.0840000000000001},"last":0.69499999999999995},{"interface":"front","average":{"1min":1.0249999999999999,"5min":1.0249999999999999,"15min":1.0249999999999999},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":2.4820000000000002,"5min":2.4820000000000002,"15min":2.4820000000000002},"last":0.78300000000000003}]},{"osd":6,"last update":"Mon May 22 15:27:37 2023","interfaces":[{"interface":"back","average":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":0.90300000000000002},"min":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"max":{"1min":1.452,"5min":1.452,"15min":1.452},"last":0.623},{"interface":"front","average":{"1min":1.085,"5min":1.085,"15min":1.085},"min":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"max":{"1min":2.6589999999999998,"5min":2.6589999999999998,"15min":2.6589999999999998},"last":0.88500000000000001}]},{"osd":7,"last update":"Mon May 22 15:27:37 2023","interfaces":[{"interface":"back","average":{"1min":1.1459999999999999,"5min":1.1459999999999999,"15min":1.1459999999999999},"min":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"max":{"1min":2.762,"5min":2.762,"15min":2.762},"last":0.73499999999999999},{"interface":"front","average":{"1min":1.0369999999999999,"5min":1.0369999999999999,"15min":1.0369999999999999},"min":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"max":{"1min":1.714,"5min":1.714,"15min":1.714},"last":0.64400000000000002}]}]},{"osd":2,"up_from":13,"seq":55834574870,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":12532,"kb_used_data":428,"kb_used_omap":0,"kb_used_meta":12096,"kb_avail":94359308,"statfs":{"total":96636764160,"available":96623931392,"internally_reserved":0,"allocated":438272,"data_stored":190925,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":12386304},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 15:27:36 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":1.4670000000000001,"5min":1.4670000000000001,"15min":1.4670000000000001},"last":0.55700000000000005}]},{"osd":1,"last update":"Mon May 22 15:27:36 2023","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":1.5309999999999999,"5min":1.5309999999999999,"15min":1.5309999999999999},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":1.2210000000000001,"5min":1.2210000000000001,"15min":1.2210000000000001},"last":0.46899999999999997}]},{"osd":3,"last update":"Mon May 22 15:27:36 2023","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":1.3340000000000001,"5min":1.3340000000000001,"15min":1.3340000000000001},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.754,"5min":0.754,"15min":0.754},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":1.6339999999999999,"5min":1.6339999999999999,"15min":1.6339999999999999},"last":0.79500000000000004}]},{"osd":4,"last update":"Mon May 22 15:27:36 2023","interfaces":[{"interface":"back","average":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"min":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"max":{"1min":1.774,"5min":1.774,"15min":1.774},"last":0.91000000000000003},{"interface":"front","average":{"1min":0.97599999999999998,"5min":0.97599999999999998,"15min":0.97599999999999998},"min":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"max":{"1min":1.802,"5min":1.802,"15min":1.802},"last":1.0600000000000001}]},{"osd":5,"last update":"Mon May 22 15:27:36 2023","interfaces":[{"interface":"back","average":{"1min":1.05,"5min":1.05,"15min":1.05},"min":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"max":{"1min":2.4569999999999999,"5min":2.4569999999999999,"15min":2.4569999999999999},"last":0.93500000000000005},{"interface":"front","average":{"1min":1,"5min":1,"15min":1},"min":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"max":{"1min":1.4670000000000001,"5min":1.4670000000000001,"15min":1.4670000000000001},"last":1.091}]},{"osd":6,"last update":"Mon May 22 15:27:36 2023","interfaces":[{"interface":"back","average":{"1min":1.093,"5min":1.093,"15min":1.093},"min":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":3.488,"5min":3.488,"15min":3.488},"last":0.95899999999999996},{"interface":"front","average":{"1min":1.151,"5min":1.151,"15min":1.151},"min":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"max":{"1min":3.7330000000000001,"5min":3.7330000000000001,"15min":3.7330000000000001},"last":1.0029999999999999}]},{"osd":7,"last update":"Mon May 22 15:27:36 2023","interfaces":[{"interface":"back","average":{"1min":1.085,"5min":1.085,"15min":1.085},"min":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"max":{"1min":2.21,"5min":2.21,"15min":2.21},"last":0.875},{"interface":"front","average":{"1min":1.0269999999999999,"5min":1.0269999999999999,"15min":1.0269999999999999},"min":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":1.6120000000000001,"5min":1.6120000000000001,"15min":1.6120000000000001},"last":1.0329999999999999}]}]},{"osd":1,"up_from":13,"seq":55834574870,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8500,"kb_used_data":428,"kb_used_omap":0,"kb_used_meta":8064,"kb_avail":94363340,"statfs":{"total":96636764160,"available":96628060160,"internally_reserved":0,"allocated":438272,"data_stored":190925,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8257536},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 15:27:36 2023","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":1.3089999999999999,"5min":1.3089999999999999,"15min":1.3089999999999999},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":1.3799999999999999,"5min":1.3799999999999999,"15min":1.3799999999999999},"last":0.95899999999999996}]},{"osd":2,"last update":"Mon May 22 15:27:36 2023","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.63,"15min":0.63},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.98299999999999998,"5min":0.98299999999999998,"15min":0.98299999999999998},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.35199999999999998}]},{"osd":3,"last update":"Mon May 22 15:27:36 2023","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":1.145,"5min":1.145,"15min":1.145},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.0760000000000001,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.49299999999999999}]},{"osd":4,"last update":"Mon May 22 15:27:36 2023","interfaces":[{"interface":"back","average":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":0.85299999999999998},"min":{"1min":0.622,"5min":0.622,"15min":0.622},"max":{"1min":1.2250000000000001,"5min":1.2250000000000001,"15min":1.2250000000000001},"last":0.86199999999999999},{"interface":"front","average":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"min":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"max":{"1min":1.272,"5min":1.272,"15min":1.272},"last":1.0109999999999999}]},{"osd":5,"last update":"Mon May 22 15:27:36 2023","interfaces":[{"interface":"back","average":{"1min":0.95999999999999996,"5min":0.95999999999999996,"15min":0.95999999999999996},"min":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"max":{"1min":1.1779999999999999,"5min":1.1779999999999999,"15min":1.1779999999999999},"last":0.68500000000000005},{"interface":"front","average":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":0.96299999999999997},"min":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"max":{"1min":1.488,"5min":1.488,"15min":1.488},"last":0.73299999999999998}]},{"osd":6,"last update":"Mon May 22 15:27:36 2023","interfaces":[{"interface":"back","average":{"1min":0.93400000000000005,"5min":0.93400000000000005,"15min":0.93400000000000005},"min":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"max":{"1min":1.28,"5min":1.28,"15min":1.28},"last":1.151},{"interface":"front","average":{"1min":1.1120000000000001,"5min":1.1120000000000001,"15min":1.1120000000000001},"min":{"1min":0.621,"5min":0.621,"15min":0.621},"max":{"1min":1.8180000000000001,"5min":1.8180000000000001,"15min":1.8180000000000001},"last":0.77400000000000002}]},{"osd":7,"last update":"Mon May 22 15:27:36 2023","interfaces":[{"interface":"back","average":{"1min":1.117,"5min":1.117,"15min":1.117},"min":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"max":{"1min":1.9079999999999999,"5min":1.9079999999999999,"15min":1.9079999999999999},"last":1.0880000000000001},{"interface":"front","average":{"1min":1.0349999999999999,"5min":1.0349999999999999,"15min":1.0349999999999999},"min":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"max":{"1min":1.5780000000000001,"5min":1.5780000000000001,"15min":1.5780000000000001},"last":0.69899999999999995}]}]},{"osd":0,"up_from":13,"seq":55834574871,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":13688,"kb_used_data":1008,"kb_used_omap":0,"kb_used_meta":12672,"kb_avail":94358152,"statfs":{"total":96636764160,"available":96622747648,"internally_reserved":0,"allocated":1032192,"data_stored":781293,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":12976128},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon May 22 15:27:37 2023","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":1.1839999999999999,"5min":1.1839999999999999,"15min":1.1839999999999999},"last":0.317},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":1.1040000000000001,"5min":1.1040000000000001,"15min":1.1040000000000001},"last":0.39800000000000002}]},{"osd":2,"last update":"Mon May 22 15:27:37 2023","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":1.23,"5min":1.23,"15min":1.23},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":1.1639999999999999,"5min":1.1639999999999999,"15min":1.1639999999999999},"last":0.47799999999999998}]},{"osd":3,"last update":"Mon May 22 15:27:37 2023","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":1.2410000000000001,"5min":1.2410000000000001,"15min":1.2410000000000001},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":1.119,"5min":1.119,"15min":1.119},"last":0.45000000000000001}]},{"osd":4,"last update":"Mon May 22 15:27:37 2023","interfaces":[{"interface":"back","average":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"min":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":1.2969999999999999,"5min":1.2969999999999999,"15min":1.2969999999999999},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"min":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"max":{"1min":1.4219999999999999,"5min":1.4219999999999999,"15min":1.4219999999999999},"last":0.79900000000000004}]},{"osd":5,"last update":"Mon May 22 15:27:37 2023","interfaces":[{"interface":"back","average":{"1min":0.999,"5min":0.999,"15min":0.999},"min":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":1.4370000000000001,"5min":1.4370000000000001,"15min":1.4370000000000001},"last":0.81799999999999995},{"interface":"front","average":{"1min":0.95099999999999996,"5min":0.95099999999999996,"15min":0.95099999999999996},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.349,"5min":1.349,"15min":1.349},"last":0.68200000000000005}]},{"osd":6,"last update":"Mon May 22 15:27:37 2023","interfaces":[{"interface":"back","average":{"1min":0.94599999999999995,"5min":0.94599999999999995,"15min":0.94599999999999995},"min":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"max":{"1min":1.2769999999999999,"5min":1.2769999999999999,"15min":1.2769999999999999},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.96899999999999997,"5min":0.96899999999999997,"15min":0.96899999999999997},"min":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":1.492,"5min":1.492,"15min":1.492},"last":0.627}]},{"osd":7,"last update":"Mon May 22 15:27:37 2023","interfaces":[{"interface":"back","average":{"1min":1.04,"5min":1.04,"15min":1.04},"min":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":1.581,"5min":1.581,"15min":1.581},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.98099999999999998,"5min":0.98099999999999998,"15min":0.98099999999999998},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.448,"5min":1.448,"15min":1.448},"last":0.84599999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-05-22T15:28:16.734 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-05-22T15:28:16.899 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.891+0000 7f131e44e640 1 Processor -- start 2023-05-22T15:28:16.899 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.891+0000 7f131e44e640 1 -- start start 2023-05-22T15:28:16.900 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.891+0000 7f131e44e640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f13181531d0 0x7f13180cd730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:16.900 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.891+0000 7f131e44e640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f13180b0250 0x7f13180acba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:16.900 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.891+0000 7f131e44e640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f131815c6b0 con 0x7f13180cbbf0 2023-05-22T15:28:16.900 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.891+0000 7f131e44e640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f131815d330 con 0x7f13180b0250 2023-05-22T15:28:16.900 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.891+0000 7f131e44e640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f131815c4a0 con 0x7f13181531d0 2023-05-22T15:28:16.900 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.891+0000 7f131c9eb640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f13180b0250 0x7f13180acba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:16.901 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.891+0000 7f13177fe640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f13180cbbf0 0x7f13180afb40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:39946/0 (socket says 172.21.15.16:39946) 2023-05-22T15:28:16.901 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.891+0000 7f13177fe640 1 -- 172.21.15.16:0/2448690780 learned_addr learned my addr 172.21.15.16:0/2448690780 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:28:16.901 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.895+0000 7f131c9eb640 1 -- 172.21.15.16:0/2448690780 >> v2:172.21.15.62:3300/0 conn(0x7f13181531d0 msgr2=0x7f13180cd730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:28:16.901 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.895+0000 7f1317fff640 1 --2- 172.21.15.16:0/2448690780 >> v2:172.21.15.62:3300/0 conn(0x7f13181531d0 0x7f13180cd730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:16.902 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.895+0000 7f131c9eb640 1 --2- 172.21.15.16:0/2448690780 >> v2:172.21.15.62:3300/0 conn(0x7f13181531d0 0x7f13180cd730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:16.902 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.895+0000 7f131c9eb640 1 -- 172.21.15.16:0/2448690780 >> v1:172.21.15.16:6789/0 conn(0x7f13180cbbf0 legacy=0x7f13180afb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:16.902 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.895+0000 7f131c9eb640 1 -- 172.21.15.16:0/2448690780 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f131815c990 con 0x7f13180b0250 2023-05-22T15:28:16.902 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.895+0000 7f1316ffd640 1 -- 172.21.15.16:0/2448690780 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3687873239 0 0) 0x7f131815c6b0 con 0x7f13180cbbf0 2023-05-22T15:28:16.902 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.895+0000 7f1317fff640 1 --2- 172.21.15.16:0/2448690780 >> v2:172.21.15.62:3300/0 conn(0x7f13181531d0 0x7f13180cd730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:28:16.903 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.895+0000 7f131c9eb640 1 --2- 172.21.15.16:0/2448690780 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f13180b0250 0x7f13180acba0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f131806f430 tx=0x7f131001c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6bb8c8a4b217e3ca server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:28:16.903 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.895+0000 7f1316ffd640 1 -- 172.21.15.16:0/2448690780 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13100a57d0 con 0x7f13180b0250 2023-05-22T15:28:16.903 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.895+0000 7f1316ffd640 1 -- 172.21.15.16:0/2448690780 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13100a5930 con 0x7f13180b0250 2023-05-22T15:28:16.904 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.895+0000 7f1316ffd640 1 -- 172.21.15.16:0/2448690780 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13100a5cc0 con 0x7f13180b0250 2023-05-22T15:28:16.904 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.895+0000 7f131e44e640 1 -- 172.21.15.16:0/2448690780 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f13180b0250 msgr2=0x7f13180acba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:16.904 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.895+0000 7f131e44e640 1 --2- 172.21.15.16:0/2448690780 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f13180b0250 0x7f13180acba0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f131806f430 tx=0x7f131001c910 comp rx=0 tx=0).stop 2023-05-22T15:28:16.904 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.895+0000 7f131e44e640 1 -- 172.21.15.16:0/2448690780 shutdown_connections 2023-05-22T15:28:16.904 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.895+0000 7f131e44e640 1 --2- 172.21.15.16:0/2448690780 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f13180b0250 0x7f13180acba0 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:16.904 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.895+0000 7f131e44e640 1 --2- 172.21.15.16:0/2448690780 >> v2:172.21.15.62:3300/0 conn(0x7f13181531d0 0x7f13180cd730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:16.904 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.895+0000 7f131e44e640 1 -- 172.21.15.16:0/2448690780 >> 172.21.15.16:0/2448690780 conn(0x7f1318069720 msgr2=0x7f1318069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:28:16.905 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.895+0000 7f131e44e640 1 -- 172.21.15.16:0/2448690780 shutdown_connections 2023-05-22T15:28:16.905 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.895+0000 7f131e44e640 1 -- 172.21.15.16:0/2448690780 wait complete. 2023-05-22T15:28:16.906 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.899+0000 7f131e44e640 1 Processor -- start 2023-05-22T15:28:16.906 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.899+0000 7f131e44e640 1 -- start start 2023-05-22T15:28:16.906 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.899+0000 7f131e44e640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f13180b0250 0x7f13180c89f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:16.907 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.899+0000 7f131e44e640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f13180cbbf0 0x7f13181494d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:16.907 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.899+0000 7f131e44e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f13181531d0 0x7f13180c9060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:16.907 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.899+0000 7f131e44e640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f131814abf0 con 0x7f13181531d0 2023-05-22T15:28:16.907 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.899+0000 7f131e44e640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1318149a10 con 0x7f13180b0250 2023-05-22T15:28:16.907 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.899+0000 7f131e44e640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1318149fe0 con 0x7f13180cbbf0 2023-05-22T15:28:16.908 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.899+0000 7f131c9eb640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f13181531d0 0x7f13180c9060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:16.908 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.899+0000 7f131c9eb640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f13181531d0 0x7f13180c9060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42854/0 (socket says 172.21.15.16:42854) 2023-05-22T15:28:16.908 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.899+0000 7f131c9eb640 1 -- 172.21.15.16:0/3196894419 learned_addr learned my addr 172.21.15.16:0/3196894419 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:28:16.908 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.899+0000 7f13177fe640 1 --2- 172.21.15.16:0/3196894419 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f13180cbbf0 0x7f13181494d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:16.908 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.899+0000 7f1317fff640 1 --2- 172.21.15.16:0/3196894419 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f13180b0250 0x7f13180c89f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:16.908 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.899+0000 7f131c9eb640 1 -- 172.21.15.16:0/3196894419 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f13180cbbf0 msgr2=0x7f13181494d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:16.908 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.899+0000 7f131c9eb640 1 --2- 172.21.15.16:0/3196894419 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f13180cbbf0 0x7f13181494d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:16.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.899+0000 7f131c9eb640 1 -- 172.21.15.16:0/3196894419 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f13180b0250 msgr2=0x7f13180c89f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:16.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.899+0000 7f131c9eb640 1 --2- 172.21.15.16:0/3196894419 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f13180b0250 0x7f13180c89f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:16.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.899+0000 7f131c9eb640 1 -- 172.21.15.16:0/3196894419 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1310009a70 con 0x7f13181531d0 2023-05-22T15:28:16.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.899+0000 7f13177fe640 1 --2- 172.21.15.16:0/3196894419 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f13180cbbf0 0x7f13181494d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:28:16.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.899+0000 7f1317fff640 1 --2- 172.21.15.16:0/3196894419 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f13180b0250 0x7f13180c89f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:28:16.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.899+0000 7f131c9eb640 1 --2- 172.21.15.16:0/3196894419 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f13181531d0 0x7f13180c9060 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f1310019520 tx=0x7f13100196b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:28:16.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.899+0000 7f1314ff9640 1 -- 172.21.15.16:0/3196894419 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13100ad9d0 con 0x7f13181531d0 2023-05-22T15:28:16.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.899+0000 7f1314ff9640 1 -- 172.21.15.16:0/3196894419 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13100adb50 con 0x7f13181531d0 2023-05-22T15:28:16.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.899+0000 7f131e44e640 1 -- 172.21.15.16:0/3196894419 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13181608d0 con 0x7f13181531d0 2023-05-22T15:28:16.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.899+0000 7f1314ff9640 1 -- 172.21.15.16:0/3196894419 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13100b9450 con 0x7f13181531d0 2023-05-22T15:28:16.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.899+0000 7f131e44e640 1 -- 172.21.15.16:0/3196894419 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1318160740 con 0x7f13181531d0 2023-05-22T15:28:16.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.899+0000 7f131e44e640 1 -- 172.21.15.16:0/3196894419 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1318168e30 con 0x7f13181531d0 2023-05-22T15:28:16.911 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.899+0000 7f1314ff9640 1 -- 172.21.15.16:0/3196894419 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7f13100b95b0 con 0x7f13181531d0 2023-05-22T15:28:16.911 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.903+0000 7f1314ff9640 1 --2- 172.21.15.16:0/3196894419 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f1304065380 0x7f1304067830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:16.911 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.903+0000 7f1314ff9640 1 -- 172.21.15.16:0/3196894419 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f131011df90 con 0x7f13181531d0 2023-05-22T15:28:16.911 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.903+0000 7f1317fff640 1 --2- 172.21.15.16:0/3196894419 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f1304065380 0x7f1304067830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:16.911 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.903+0000 7f1317fff640 1 --2- 172.21.15.16:0/3196894419 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f1304065380 0x7f1304067830 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f13181522c0 tx=0x7f1308008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:28:16.912 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:16.907+0000 7f1314ff9640 1 -- 172.21.15.16:0/3196894419 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f13100ebb40 con 0x7f13181531d0 2023-05-22T15:28:17.060 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:17.051+0000 7f131e44e640 1 -- 172.21.15.16:0/3196894419 --> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f131804dbe0 con 0x7f1304065380 2023-05-22T15:28:17.072 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:17.063+0000 7f1314ff9640 1 -- 172.21.15.16:0/3196894419 <== mgr.4101 v2:172.21.15.16:6832/36921 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+299300 (secure 0 0 0) 0x7f131804dbe0 con 0x7f1304065380 2023-05-22T15:28:17.074 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:28:17.076 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2023-05-22T15:28:17.077 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:17.071+0000 7f131e44e640 1 -- 172.21.15.16:0/3196894419 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f1304065380 msgr2=0x7f1304067830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:17.077 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:17.071+0000 7f131e44e640 1 --2- 172.21.15.16:0/3196894419 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f1304065380 0x7f1304067830 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f13181522c0 tx=0x7f1308008040 comp rx=0 tx=0).stop 2023-05-22T15:28:17.077 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:17.071+0000 7f131e44e640 1 -- 172.21.15.16:0/3196894419 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f13181531d0 msgr2=0x7f13180c9060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:17.077 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:17.071+0000 7f131e44e640 1 --2- 172.21.15.16:0/3196894419 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f13181531d0 0x7f13180c9060 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f1310019520 tx=0x7f13100196b0 comp rx=0 tx=0).stop 2023-05-22T15:28:17.077 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:17.071+0000 7f131e44e640 1 -- 172.21.15.16:0/3196894419 shutdown_connections 2023-05-22T15:28:17.078 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:17.071+0000 7f131e44e640 1 --2- 172.21.15.16:0/3196894419 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f13181531d0 0x7f13180c9060 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:17.078 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:17.071+0000 7f131e44e640 1 --2- 172.21.15.16:0/3196894419 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f1304065380 0x7f1304067830 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:17.078 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:17.071+0000 7f131e44e640 1 --2- 172.21.15.16:0/3196894419 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f13180cbbf0 0x7f13181494d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:17.078 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:17.071+0000 7f131e44e640 1 --2- 172.21.15.16:0/3196894419 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f13180b0250 0x7f13180c89f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:17.078 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:17.071+0000 7f131e44e640 1 -- 172.21.15.16:0/3196894419 >> 172.21.15.16:0/3196894419 conn(0x7f1318069720 msgr2=0x7f1318070090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:28:17.078 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:17.071+0000 7f131e44e640 1 -- 172.21.15.16:0/3196894419 shutdown_connections 2023-05-22T15:28:17.078 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:17.071+0000 7f131e44e640 1 -- 172.21.15.16:0/3196894419 wait complete. 2023-05-22T15:28:17.093 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":85,"stamp":"2023-05-22T15:28:15.972192+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":82408,"kb_used_data":4584,"kb_used_omap":0,"kb_used_meta":77760,"kb_avail":754892312,"statfs":{"total":773094113280,"available":773009727488,"internally_reserved":0,"allocated":4694016,"data_stored":2708136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":79626240},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.040506"},"pg_stats":[{"pgid":"7.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2023-05-22T15:28:11.938772+0000","last_change":"2023-05-22T15:28:11.938772+0000","last_active":"2023-05-22T15:28:11.927105+0000","last_peered":"2023-05-22T15:28:11.927105+0000","last_clean":"2023-05-22T15:28:11.927105+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:11.938772+0000","last_undegraded":"2023-05-22T15:28:11.938772+0000","last_fullsized":"2023-05-22T15:28:11.938772+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_clean_scrub_stamp":"2023-05-22T15:28:11.927105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T18:33:01.747002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2023-05-22T15:28:11.942902+0000","last_change":"2023-05-22T15:28:11.942902+0000","last_active":"2023-05-22T15:28:11.927105+0000","last_peered":"2023-05-22T15:28:11.927105+0000","last_clean":"2023-05-22T15:28:11.927105+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:11.942902+0000","last_undegraded":"2023-05-22T15:28:11.942902+0000","last_fullsized":"2023-05-22T15:28:11.942902+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_clean_scrub_stamp":"2023-05-22T15:28:11.927105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:21:15.037766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2023-05-22T15:28:11.945096+0000","last_change":"2023-05-22T15:28:11.945096+0000","last_active":"2023-05-22T15:28:11.927105+0000","last_peered":"2023-05-22T15:28:11.927105+0000","last_clean":"2023-05-22T15:28:11.927105+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:11.945096+0000","last_undegraded":"2023-05-22T15:28:11.945096+0000","last_fullsized":"2023-05-22T15:28:11.945096+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_clean_scrub_stamp":"2023-05-22T15:28:11.927105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:02:28.198798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2023-05-22T15:28:11.947443+0000","last_change":"2023-05-22T15:28:11.947443+0000","last_active":"2023-05-22T15:28:11.927105+0000","last_peered":"2023-05-22T15:28:11.927105+0000","last_clean":"2023-05-22T15:28:11.927105+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:11.947443+0000","last_undegraded":"2023-05-22T15:28:11.947443+0000","last_fullsized":"2023-05-22T15:28:11.947443+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_clean_scrub_stamp":"2023-05-22T15:28:11.927105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:42:40.893204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2023-05-22T15:28:11.941543+0000","last_change":"2023-05-22T15:28:11.941543+0000","last_active":"2023-05-22T15:28:11.927105+0000","last_peered":"2023-05-22T15:28:11.927105+0000","last_clean":"2023-05-22T15:28:11.927105+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:11.941543+0000","last_undegraded":"2023-05-22T15:28:11.941543+0000","last_fullsized":"2023-05-22T15:28:11.941543+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_clean_scrub_stamp":"2023-05-22T15:28:11.927105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:05:30.503639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2023-05-22T15:28:11.942842+0000","last_change":"2023-05-22T15:28:11.942842+0000","last_active":"2023-05-22T15:28:11.927105+0000","last_peered":"2023-05-22T15:28:11.927105+0000","last_clean":"2023-05-22T15:28:11.927105+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:11.942842+0000","last_undegraded":"2023-05-22T15:28:11.942842+0000","last_fullsized":"2023-05-22T15:28:11.942842+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_clean_scrub_stamp":"2023-05-22T15:28:11.927105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:17:33.835222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2023-05-22T15:28:11.948423+0000","last_change":"2023-05-22T15:28:11.948423+0000","last_active":"2023-05-22T15:28:11.927105+0000","last_peered":"2023-05-22T15:28:11.927105+0000","last_clean":"2023-05-22T15:28:11.927105+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:11.948423+0000","last_undegraded":"2023-05-22T15:28:11.948423+0000","last_fullsized":"2023-05-22T15:28:11.948423+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_clean_scrub_stamp":"2023-05-22T15:28:11.927105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:45:43.272774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2023-05-22T15:28:11.942591+0000","last_change":"2023-05-22T15:28:11.942591+0000","last_active":"2023-05-22T15:28:11.927105+0000","last_peered":"2023-05-22T15:28:11.927105+0000","last_clean":"2023-05-22T15:28:11.927105+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:11.942591+0000","last_undegraded":"2023-05-22T15:28:11.942591+0000","last_fullsized":"2023-05-22T15:28:11.942591+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_clean_scrub_stamp":"2023-05-22T15:28:11.927105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:02:28.198798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2023-05-22T15:28:11.941674+0000","last_change":"2023-05-22T15:28:11.941674+0000","last_active":"2023-05-22T15:28:11.927105+0000","last_peered":"2023-05-22T15:28:11.927105+0000","last_clean":"2023-05-22T15:28:11.927105+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:11.941674+0000","last_undegraded":"2023-05-22T15:28:11.941674+0000","last_fullsized":"2023-05-22T15:28:11.941674+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_clean_scrub_stamp":"2023-05-22T15:28:11.927105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:50:22.466750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2023-05-22T15:28:11.940470+0000","last_change":"2023-05-22T15:28:11.940470+0000","last_active":"2023-05-22T15:28:11.927105+0000","last_peered":"2023-05-22T15:28:11.927105+0000","last_clean":"2023-05-22T15:28:11.927105+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:11.940470+0000","last_undegraded":"2023-05-22T15:28:11.940470+0000","last_fullsized":"2023-05-22T15:28:11.940470+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_clean_scrub_stamp":"2023-05-22T15:28:11.927105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:15:50.706827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2023-05-22T15:28:11.950095+0000","last_change":"2023-05-22T15:28:11.950095+0000","last_active":"2023-05-22T15:28:11.927105+0000","last_peered":"2023-05-22T15:28:11.927105+0000","last_clean":"2023-05-22T15:28:11.927105+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:11.950095+0000","last_undegraded":"2023-05-22T15:28:11.950095+0000","last_fullsized":"2023-05-22T15:28:11.950095+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_clean_scrub_stamp":"2023-05-22T15:28:11.927105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:57:04.738345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2023-05-22T15:28:11.935772+0000","last_change":"2023-05-22T15:28:11.935772+0000","last_active":"2023-05-22T15:28:11.927105+0000","last_peered":"2023-05-22T15:28:11.927105+0000","last_clean":"2023-05-22T15:28:11.927105+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:11.935772+0000","last_undegraded":"2023-05-22T15:28:11.935772+0000","last_fullsized":"2023-05-22T15:28:11.935772+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_clean_scrub_stamp":"2023-05-22T15:28:11.927105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:30:08.027313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2023-05-22T15:28:11.938777+0000","last_change":"2023-05-22T15:28:11.938777+0000","last_active":"2023-05-22T15:28:11.927105+0000","last_peered":"2023-05-22T15:28:11.927105+0000","last_clean":"2023-05-22T15:28:11.927105+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:11.938777+0000","last_undegraded":"2023-05-22T15:28:11.938777+0000","last_fullsized":"2023-05-22T15:28:11.938777+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_clean_scrub_stamp":"2023-05-22T15:28:11.927105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:47:11.442121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2023-05-22T15:28:11.949598+0000","last_change":"2023-05-22T15:28:11.949598+0000","last_active":"2023-05-22T15:28:11.927105+0000","last_peered":"2023-05-22T15:28:11.927105+0000","last_clean":"2023-05-22T15:28:11.927105+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:11.949598+0000","last_undegraded":"2023-05-22T15:28:11.949598+0000","last_fullsized":"2023-05-22T15:28:11.949598+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_clean_scrub_stamp":"2023-05-22T15:28:11.927105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:26:26.763456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2023-05-22T15:28:11.938244+0000","last_change":"2023-05-22T15:28:11.938244+0000","last_active":"2023-05-22T15:28:11.927105+0000","last_peered":"2023-05-22T15:28:11.927105+0000","last_clean":"2023-05-22T15:28:11.927105+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:11.938244+0000","last_undegraded":"2023-05-22T15:28:11.938244+0000","last_fullsized":"2023-05-22T15:28:11.938244+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_clean_scrub_stamp":"2023-05-22T15:28:11.927105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:09:59.757240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2023-05-22T15:28:11.941202+0000","last_change":"2023-05-22T15:28:11.941202+0000","last_active":"2023-05-22T15:28:11.927105+0000","last_peered":"2023-05-22T15:28:11.927105+0000","last_clean":"2023-05-22T15:28:11.927105+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:11.941202+0000","last_undegraded":"2023-05-22T15:28:11.941202+0000","last_fullsized":"2023-05-22T15:28:11.941202+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_clean_scrub_stamp":"2023-05-22T15:28:11.927105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T15:35:29.715819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2023-05-22T15:28:11.935588+0000","last_change":"2023-05-22T15:28:11.935588+0000","last_active":"2023-05-22T15:28:11.927105+0000","last_peered":"2023-05-22T15:28:11.927105+0000","last_clean":"2023-05-22T15:28:11.927105+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:11.935588+0000","last_undegraded":"2023-05-22T15:28:11.935588+0000","last_fullsized":"2023-05-22T15:28:11.935588+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_clean_scrub_stamp":"2023-05-22T15:28:11.927105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:10:24.156615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2023-05-22T15:28:11.941609+0000","last_change":"2023-05-22T15:28:11.941609+0000","last_active":"2023-05-22T15:28:11.927105+0000","last_peered":"2023-05-22T15:28:11.927105+0000","last_clean":"2023-05-22T15:28:11.927105+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:11.941609+0000","last_undegraded":"2023-05-22T15:28:11.941609+0000","last_fullsized":"2023-05-22T15:28:11.941609+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_clean_scrub_stamp":"2023-05-22T15:28:11.927105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T03:27:50.363592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2023-05-22T15:28:11.947845+0000","last_change":"2023-05-22T15:28:11.947845+0000","last_active":"2023-05-22T15:28:11.927105+0000","last_peered":"2023-05-22T15:28:11.927105+0000","last_clean":"2023-05-22T15:28:11.927105+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:11.947845+0000","last_undegraded":"2023-05-22T15:28:11.947845+0000","last_fullsized":"2023-05-22T15:28:11.947845+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_clean_scrub_stamp":"2023-05-22T15:28:11.927105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:57:04.738345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2023-05-22T15:28:11.938489+0000","last_change":"2023-05-22T15:28:11.938489+0000","last_active":"2023-05-22T15:28:11.927105+0000","last_peered":"2023-05-22T15:28:11.927105+0000","last_clean":"2023-05-22T15:28:11.927105+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:11.938489+0000","last_undegraded":"2023-05-22T15:28:11.938489+0000","last_fullsized":"2023-05-22T15:28:11.938489+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_clean_scrub_stamp":"2023-05-22T15:28:11.927105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:45:43.272774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2023-05-22T15:28:11.940183+0000","last_change":"2023-05-22T15:28:11.940183+0000","last_active":"2023-05-22T15:28:11.927105+0000","last_peered":"2023-05-22T15:28:11.927105+0000","last_clean":"2023-05-22T15:28:11.927105+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:11.940183+0000","last_undegraded":"2023-05-22T15:28:11.940183+0000","last_fullsized":"2023-05-22T15:28:11.940183+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_clean_scrub_stamp":"2023-05-22T15:28:11.927105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:50:22.466750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2023-05-22T15:28:11.947990+0000","last_change":"2023-05-22T15:28:11.947990+0000","last_active":"2023-05-22T15:28:11.927105+0000","last_peered":"2023-05-22T15:28:11.927105+0000","last_clean":"2023-05-22T15:28:11.927105+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:11.947990+0000","last_undegraded":"2023-05-22T15:28:11.947990+0000","last_fullsized":"2023-05-22T15:28:11.947990+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_clean_scrub_stamp":"2023-05-22T15:28:11.927105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T18:18:35.201229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2023-05-22T15:28:11.940527+0000","last_change":"2023-05-22T15:28:11.940527+0000","last_active":"2023-05-22T15:28:11.927105+0000","last_peered":"2023-05-22T15:28:11.927105+0000","last_clean":"2023-05-22T15:28:11.927105+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:11.940527+0000","last_undegraded":"2023-05-22T15:28:11.940527+0000","last_fullsized":"2023-05-22T15:28:11.940527+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_clean_scrub_stamp":"2023-05-22T15:28:11.927105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:24:48.246881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2023-05-22T15:28:11.947499+0000","last_change":"2023-05-22T15:28:11.947499+0000","last_active":"2023-05-22T15:28:11.927105+0000","last_peered":"2023-05-22T15:28:11.927105+0000","last_clean":"2023-05-22T15:28:11.927105+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:11.947499+0000","last_undegraded":"2023-05-22T15:28:11.947499+0000","last_fullsized":"2023-05-22T15:28:11.947499+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_clean_scrub_stamp":"2023-05-22T15:28:11.927105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T18:18:35.201229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2023-05-22T15:28:11.947536+0000","last_change":"2023-05-22T15:28:11.947536+0000","last_active":"2023-05-22T15:28:11.927105+0000","last_peered":"2023-05-22T15:28:11.927105+0000","last_clean":"2023-05-22T15:28:11.927105+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:11.947536+0000","last_undegraded":"2023-05-22T15:28:11.947536+0000","last_fullsized":"2023-05-22T15:28:11.947536+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_clean_scrub_stamp":"2023-05-22T15:28:11.927105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:15:37.287885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":25,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-05-22T15:28:08.260614+0000","last_change":"2023-05-22T15:26:44.982421+0000","last_active":"2023-05-22T15:28:08.260614+0000","last_peered":"2023-05-22T15:28:08.260614+0000","last_clean":"2023-05-22T15:28:08.260614+0000","last_became_active":"2023-05-22T15:26:44.981586+0000","last_became_peered":"2023-05-22T15:26:44.981586+0000","last_unstale":"2023-05-22T15:28:08.260614+0000","last_undegraded":"2023-05-22T15:28:08.260614+0000","last_fullsized":"2023-05-22T15:28:08.260614+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:14:11.156544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2023-05-22T15:28:11.941443+0000","last_change":"2023-05-22T15:28:11.941443+0000","last_active":"2023-05-22T15:28:11.927105+0000","last_peered":"2023-05-22T15:28:11.927105+0000","last_clean":"2023-05-22T15:28:11.927105+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:11.941443+0000","last_undegraded":"2023-05-22T15:28:11.941443+0000","last_fullsized":"2023-05-22T15:28:11.941443+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_clean_scrub_stamp":"2023-05-22T15:28:11.927105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:05:30.503639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":24,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-05-22T15:28:07.758852+0000","last_change":"2023-05-22T15:26:44.978858+0000","last_active":"2023-05-22T15:28:07.758852+0000","last_peered":"2023-05-22T15:28:07.758852+0000","last_clean":"2023-05-22T15:28:07.758852+0000","last_became_active":"2023-05-22T15:26:44.978443+0000","last_became_peered":"2023-05-22T15:26:44.978443+0000","last_unstale":"2023-05-22T15:28:07.758852+0000","last_undegraded":"2023-05-22T15:28:07.758852+0000","last_fullsized":"2023-05-22T15:28:07.758852+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:46:29.290243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2023-05-22T15:28:11.945876+0000","last_change":"2023-05-22T15:28:11.945876+0000","last_active":"2023-05-22T15:28:11.927105+0000","last_peered":"2023-05-22T15:28:11.927105+0000","last_clean":"2023-05-22T15:28:11.927105+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:11.945876+0000","last_undegraded":"2023-05-22T15:28:11.945876+0000","last_fullsized":"2023-05-22T15:28:11.945876+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_clean_scrub_stamp":"2023-05-22T15:28:11.927105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:43:14.376663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":28,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-05-22T15:28:07.681164+0000","last_change":"2023-05-22T15:26:45.092367+0000","last_active":"2023-05-22T15:28:07.681164+0000","last_peered":"2023-05-22T15:28:07.681164+0000","last_clean":"2023-05-22T15:28:07.681164+0000","last_became_active":"2023-05-22T15:26:45.091951+0000","last_became_peered":"2023-05-22T15:26:45.091951+0000","last_unstale":"2023-05-22T15:28:07.681164+0000","last_undegraded":"2023-05-22T15:28:07.681164+0000","last_fullsized":"2023-05-22T15:28:07.681164+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:42:57.973030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2023-05-22T15:28:11.940327+0000","last_change":"2023-05-22T15:28:11.940327+0000","last_active":"2023-05-22T15:28:11.927105+0000","last_peered":"2023-05-22T15:28:11.927105+0000","last_clean":"2023-05-22T15:28:11.927105+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:11.940327+0000","last_undegraded":"2023-05-22T15:28:11.940327+0000","last_fullsized":"2023-05-22T15:28:11.940327+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_clean_scrub_stamp":"2023-05-22T15:28:11.927105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:23:05.013217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":24,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-05-22T15:28:07.758858+0000","last_change":"2023-05-22T15:26:45.091475+0000","last_active":"2023-05-22T15:28:07.758858+0000","last_peered":"2023-05-22T15:28:07.758858+0000","last_clean":"2023-05-22T15:28:07.758858+0000","last_became_active":"2023-05-22T15:26:45.091143+0000","last_became_peered":"2023-05-22T15:26:45.091143+0000","last_unstale":"2023-05-22T15:28:07.758858+0000","last_undegraded":"2023-05-22T15:28:07.758858+0000","last_fullsized":"2023-05-22T15:28:07.758858+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:12:41.464054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2023-05-22T15:28:11.938283+0000","last_change":"2023-05-22T15:28:11.938283+0000","last_active":"2023-05-22T15:28:11.927105+0000","last_peered":"2023-05-22T15:28:11.927105+0000","last_clean":"2023-05-22T15:28:11.927105+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:11.938283+0000","last_undegraded":"2023-05-22T15:28:11.938283+0000","last_fullsized":"2023-05-22T15:28:11.938283+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_clean_scrub_stamp":"2023-05-22T15:28:11.927105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:21:15.037766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":29,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-05-22T15:28:07.680251+0000","last_change":"2023-05-22T15:26:44.976632+0000","last_active":"2023-05-22T15:28:07.680251+0000","last_peered":"2023-05-22T15:28:07.680251+0000","last_clean":"2023-05-22T15:28:07.680251+0000","last_became_active":"2023-05-22T15:26:44.975546+0000","last_became_peered":"2023-05-22T15:26:44.975546+0000","last_unstale":"2023-05-22T15:28:07.680251+0000","last_undegraded":"2023-05-22T15:28:07.680251+0000","last_fullsized":"2023-05-22T15:28:07.680251+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:42:57.973030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2023-05-22T15:28:11.948734+0000","last_change":"2023-05-22T15:28:11.948734+0000","last_active":"2023-05-22T15:28:11.927105+0000","last_peered":"2023-05-22T15:28:11.927105+0000","last_clean":"2023-05-22T15:28:11.927105+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:11.948734+0000","last_undegraded":"2023-05-22T15:28:11.948734+0000","last_fullsized":"2023-05-22T15:28:11.948734+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_clean_scrub_stamp":"2023-05-22T15:28:11.927105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:45:43.272774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":28,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-05-22T15:28:07.678593+0000","last_change":"2023-05-22T15:26:44.969469+0000","last_active":"2023-05-22T15:28:07.678593+0000","last_peered":"2023-05-22T15:28:07.678593+0000","last_clean":"2023-05-22T15:28:07.678593+0000","last_became_active":"2023-05-22T15:26:44.969083+0000","last_became_peered":"2023-05-22T15:26:44.969083+0000","last_unstale":"2023-05-22T15:28:07.678593+0000","last_undegraded":"2023-05-22T15:28:07.678593+0000","last_fullsized":"2023-05-22T15:28:07.678593+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:29:23.670067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2023-05-22T15:28:11.945024+0000","last_change":"2023-05-22T15:28:11.945024+0000","last_active":"2023-05-22T15:28:11.927105+0000","last_peered":"2023-05-22T15:28:11.927105+0000","last_clean":"2023-05-22T15:28:11.927105+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:11.945024+0000","last_undegraded":"2023-05-22T15:28:11.945024+0000","last_fullsized":"2023-05-22T15:28:11.945024+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_clean_scrub_stamp":"2023-05-22T15:28:11.927105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:23:05.013217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":28,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-05-22T15:28:07.681164+0000","last_change":"2023-05-22T15:26:44.977198+0000","last_active":"2023-05-22T15:28:07.681164+0000","last_peered":"2023-05-22T15:28:07.681164+0000","last_clean":"2023-05-22T15:28:07.681164+0000","last_became_active":"2023-05-22T15:26:44.976445+0000","last_became_peered":"2023-05-22T15:26:44.976445+0000","last_unstale":"2023-05-22T15:28:07.681164+0000","last_undegraded":"2023-05-22T15:28:07.681164+0000","last_fullsized":"2023-05-22T15:28:07.681164+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:12:41.464054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2023-05-22T15:28:11.947908+0000","last_change":"2023-05-22T15:28:11.947908+0000","last_active":"2023-05-22T15:28:11.927105+0000","last_peered":"2023-05-22T15:28:11.927105+0000","last_clean":"2023-05-22T15:28:11.927105+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:11.947908+0000","last_undegraded":"2023-05-22T15:28:11.947908+0000","last_fullsized":"2023-05-22T15:28:11.947908+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_clean_scrub_stamp":"2023-05-22T15:28:11.927105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:15:37.287885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":29,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-05-22T15:28:07.680257+0000","last_change":"2023-05-22T15:26:44.971951+0000","last_active":"2023-05-22T15:28:07.680257+0000","last_peered":"2023-05-22T15:28:07.680257+0000","last_clean":"2023-05-22T15:28:07.680257+0000","last_became_active":"2023-05-22T15:26:44.970872+0000","last_became_peered":"2023-05-22T15:26:44.970872+0000","last_unstale":"2023-05-22T15:28:07.680257+0000","last_undegraded":"2023-05-22T15:28:07.680257+0000","last_fullsized":"2023-05-22T15:28:07.680257+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:55:34.752949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":181,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-05-22T15:28:07.681354+0000","last_change":"2023-05-22T15:26:39.318204+0000","last_active":"2023-05-22T15:28:07.681354+0000","last_peered":"2023-05-22T15:28:07.681354+0000","last_clean":"2023-05-22T15:28:07.681354+0000","last_became_active":"2023-05-22T15:26:39.315370+0000","last_became_peered":"2023-05-22T15:26:39.315370+0000","last_unstale":"2023-05-22T15:28:07.681354+0000","last_undegraded":"2023-05-22T15:28:07.681354+0000","last_fullsized":"2023-05-22T15:28:07.681354+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:38.020076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:38.020076+0000","last_clean_scrub_stamp":"2023-05-22T15:26:38.020076+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:06:39.207543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":7},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574870,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":13752,"kb_used_data":1008,"kb_used_omap":0,"kb_used_meta":12736,"kb_avail":94358088,"statfs":{"total":96636764160,"available":96622682112,"internally_reserved":0,"allocated":1032192,"data_stored":781293,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":13041664},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 15:27:38 2023","interfaces":[{"interface":"back","average":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":1.5860000000000001,"5min":1.5860000000000001,"15min":1.5860000000000001},"last":0.77700000000000002},{"interface":"front","average":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"min":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.4239999999999999,"5min":1.4239999999999999,"15min":1.4239999999999999},"last":0.55700000000000005}]},{"osd":1,"last update":"Mon May 22 15:27:38 2023","interfaces":[{"interface":"back","average":{"1min":0.96599999999999997,"5min":0.96599999999999997,"15min":0.96599999999999997},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":1.3600000000000001,"5min":1.3600000000000001,"15min":1.3600000000000001},"last":1.002},{"interface":"front","average":{"1min":1.002,"5min":1.002,"15min":1.002},"min":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"max":{"1min":1.8260000000000001,"5min":1.8260000000000001,"15min":1.8260000000000001},"last":1.208}]},{"osd":2,"last update":"Mon May 22 15:27:38 2023","interfaces":[{"interface":"back","average":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.127,"5min":1.127,"15min":1.127},"last":0.90100000000000002},{"interface":"front","average":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":1.23,"5min":1.23,"15min":1.23},"last":0.83999999999999997}]},{"osd":3,"last update":"Mon May 22 15:27:38 2023","interfaces":[{"interface":"back","average":{"1min":0.99099999999999999,"5min":0.99099999999999999,"15min":0.99099999999999999},"min":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"max":{"1min":1.7829999999999999,"5min":1.7829999999999999,"15min":1.7829999999999999},"last":1.115},{"interface":"front","average":{"1min":1.026,"5min":1.026,"15min":1.026},"min":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":1.6879999999999999,"5min":1.6879999999999999,"15min":1.6879999999999999},"last":0.70199999999999996}]},{"osd":4,"last update":"Mon May 22 15:27:38 2023","interfaces":[{"interface":"back","average":{"1min":0.92700000000000005,"5min":0.92700000000000005,"15min":0.92700000000000005},"min":{"1min":0.62,"5min":0.62,"15min":0.62},"max":{"1min":1.2430000000000001,"5min":1.2430000000000001,"15min":1.2430000000000001},"last":0.65600000000000003},{"interface":"front","average":{"1min":1.1120000000000001,"5min":1.1120000000000001,"15min":1.1120000000000001},"min":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"max":{"1min":1.73,"5min":1.73,"15min":1.73},"last":1.2889999999999999}]},{"osd":5,"last update":"Mon May 22 15:27:38 2023","interfaces":[{"interface":"back","average":{"1min":1.123,"5min":1.123,"15min":1.123},"min":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":1.3779999999999999,"5min":1.3779999999999999,"15min":1.3779999999999999},"last":1.0760000000000001},{"interface":"front","average":{"1min":1.139,"5min":1.139,"15min":1.139},"min":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"max":{"1min":1.6439999999999999,"5min":1.6439999999999999,"15min":1.6439999999999999},"last":1.0429999999999999}]},{"osd":6,"last update":"Mon May 22 15:27:38 2023","interfaces":[{"interface":"back","average":{"1min":1.1020000000000001,"5min":1.1020000000000001,"15min":1.1020000000000001},"min":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"max":{"1min":1.3999999999999999,"5min":1.3999999999999999,"15min":1.3999999999999999},"last":0.96399999999999997},{"interface":"front","average":{"1min":1.1890000000000001,"5min":1.1890000000000001,"15min":1.1890000000000001},"min":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"max":{"1min":2.04,"5min":2.04,"15min":2.04},"last":1.3620000000000001}]}]},{"osd":6,"up_from":13,"seq":55834574871,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8372,"kb_used_data":428,"kb_used_omap":0,"kb_used_meta":7936,"kb_avail":94363468,"statfs":{"total":96636764160,"available":96628191232,"internally_reserved":0,"allocated":438272,"data_stored":190925,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8126464},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 15:27:41 2023","interfaces":[{"interface":"back","average":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"min":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":1.1659999999999999,"5min":1.1659999999999999,"15min":1.1659999999999999},"last":0.97399999999999998},{"interface":"front","average":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"min":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"max":{"1min":1.385,"5min":1.385,"15min":1.385},"last":0.71499999999999997}]},{"osd":1,"last update":"Mon May 22 15:27:41 2023","interfaces":[{"interface":"back","average":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"min":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":1.2869999999999999,"5min":1.2869999999999999,"15min":1.2869999999999999},"last":0.77000000000000002},{"interface":"front","average":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":0.90000000000000002},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":1.204,"5min":1.204,"15min":1.204},"last":0.92000000000000004}]},{"osd":2,"last update":"Mon May 22 15:27:41 2023","interfaces":[{"interface":"back","average":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"min":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"max":{"1min":1.145,"5min":1.145,"15min":1.145},"last":0.84099999999999997},{"interface":"front","average":{"1min":1.0569999999999999,"5min":1.0569999999999999,"15min":1.0569999999999999},"min":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"max":{"1min":1.4670000000000001,"5min":1.4670000000000001,"15min":1.4670000000000001},"last":1.0700000000000001}]},{"osd":3,"last update":"Mon May 22 15:27:41 2023","interfaces":[{"interface":"back","average":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"min":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"max":{"1min":1.415,"5min":1.415,"15min":1.415},"last":1.0169999999999999},{"interface":"front","average":{"1min":0.95799999999999996,"5min":0.95799999999999996,"15min":0.95799999999999996},"min":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":1.486,"5min":1.486,"15min":1.486},"last":1.042}]},{"osd":4,"last update":"Mon May 22 15:27:41 2023","interfaces":[{"interface":"back","average":{"1min":1.0389999999999999,"5min":1.0389999999999999,"15min":1.0389999999999999},"min":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"max":{"1min":1.6459999999999999,"5min":1.6459999999999999,"15min":1.6459999999999999},"last":1.1539999999999999},{"interface":"front","average":{"1min":1.141,"5min":1.141,"15min":1.141},"min":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"max":{"1min":1.5469999999999999,"5min":1.5469999999999999,"15min":1.5469999999999999},"last":1.097}]},{"osd":5,"last update":"Mon May 22 15:27:41 2023","interfaces":[{"interface":"back","average":{"1min":1.075,"5min":1.075,"15min":1.075},"min":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"max":{"1min":1.621,"5min":1.621,"15min":1.621},"last":1.117},{"interface":"front","average":{"1min":1.149,"5min":1.149,"15min":1.149},"min":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"max":{"1min":1.7509999999999999,"5min":1.7509999999999999,"15min":1.7509999999999999},"last":1.232}]},{"osd":7,"last update":"Mon May 22 15:27:41 2023","interfaces":[{"interface":"back","average":{"1min":1.268,"5min":1.268,"15min":1.268},"min":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"max":{"1min":1.8120000000000001,"5min":1.8120000000000001,"15min":1.8120000000000001},"last":1.177},{"interface":"front","average":{"1min":1.179,"5min":1.179,"15min":1.179},"min":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"max":{"1min":1.6819999999999999,"5min":1.6819999999999999,"15min":1.6819999999999999},"last":1.3260000000000001}]}]},{"osd":5,"up_from":13,"seq":55834574871,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8500,"kb_used_data":428,"kb_used_omap":0,"kb_used_meta":8064,"kb_avail":94363340,"statfs":{"total":96636764160,"available":96628060160,"internally_reserved":0,"allocated":438272,"data_stored":190925,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8257536},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 15:27:37 2023","interfaces":[{"interface":"back","average":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"min":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":1.169,"5min":1.169,"15min":1.169},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"min":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.103,"5min":1.103,"15min":1.103},"last":0.68500000000000005}]},{"osd":1,"last update":"Mon May 22 15:27:37 2023","interfaces":[{"interface":"back","average":{"1min":0.90200000000000002,"5min":0.90200000000000002,"15min":0.90200000000000002},"min":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":1.2170000000000001,"5min":1.2170000000000001,"15min":1.2170000000000001},"last":1.0309999999999999},{"interface":"front","average":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"min":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":1.399,"5min":1.399,"15min":1.399},"last":1.0069999999999999}]},{"osd":2,"last update":"Mon May 22 15:27:37 2023","interfaces":[{"interface":"back","average":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"min":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":1.597,"5min":1.597,"15min":1.597},"last":0.83399999999999996},{"interface":"front","average":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"min":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.296,"5min":1.296,"15min":1.296},"last":0.79100000000000004}]},{"osd":3,"last update":"Mon May 22 15:27:37 2023","interfaces":[{"interface":"back","average":{"1min":0.93400000000000005,"5min":0.93400000000000005,"15min":0.93400000000000005},"min":{"1min":0.627,"5min":0.627,"15min":0.627},"max":{"1min":1.6839999999999999,"5min":1.6839999999999999,"15min":1.6839999999999999},"last":0.91000000000000003},{"interface":"front","average":{"1min":1.0209999999999999,"5min":1.0209999999999999,"15min":1.0209999999999999},"min":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"max":{"1min":1.5189999999999999,"5min":1.5189999999999999,"15min":1.5189999999999999},"last":1.111}]},{"osd":4,"last update":"Mon May 22 15:27:37 2023","interfaces":[{"interface":"back","average":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":0.96399999999999997},"min":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"max":{"1min":1.3140000000000001,"5min":1.3140000000000001,"15min":1.3140000000000001},"last":0.876},{"interface":"front","average":{"1min":1.0549999999999999,"5min":1.0549999999999999,"15min":1.0549999999999999},"min":{"1min":0.747,"5min":0.747,"15min":0.747},"max":{"1min":1.319,"5min":1.319,"15min":1.319},"last":1.0700000000000001}]},{"osd":6,"last update":"Mon May 22 15:27:37 2023","interfaces":[{"interface":"back","average":{"1min":1.1000000000000001,"5min":1.1000000000000001,"15min":1.1000000000000001},"min":{"1min":0.94199999999999995,"5min":0.94199999999999995,"15min":0.94199999999999995},"max":{"1min":1.357,"5min":1.357,"15min":1.357},"last":1.2190000000000001},{"interface":"front","average":{"1min":1.109,"5min":1.109,"15min":1.109},"min":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"max":{"1min":1.4350000000000001,"5min":1.4350000000000001,"15min":1.4350000000000001},"last":0.95899999999999996}]},{"osd":7,"last update":"Mon May 22 15:27:37 2023","interfaces":[{"interface":"back","average":{"1min":1.232,"5min":1.232,"15min":1.232},"min":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"max":{"1min":2.8260000000000001,"5min":2.8260000000000001,"15min":2.8260000000000001},"last":1.157},{"interface":"front","average":{"1min":1.1679999999999999,"5min":1.1679999999999999,"15min":1.1679999999999999},"min":{"1min":0.877,"5min":0.877,"15min":0.877},"max":{"1min":1.4490000000000001,"5min":1.4490000000000001,"15min":1.4490000000000001},"last":1.29}]}]},{"osd":4,"up_from":13,"seq":55834574869,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8564,"kb_used_data":428,"kb_used_omap":0,"kb_used_meta":8128,"kb_avail":94363276,"statfs":{"total":96636764160,"available":96627994624,"internally_reserved":0,"allocated":438272,"data_stored":190925,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8323072},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 15:27:37 2023","interfaces":[{"interface":"back","average":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"min":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":1.252,"5min":1.252,"15min":1.252},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"min":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"max":{"1min":1.2989999999999999,"5min":1.2989999999999999,"15min":1.2989999999999999},"last":0.56299999999999994}]},{"osd":1,"last update":"Mon May 22 15:27:37 2023","interfaces":[{"interface":"back","average":{"1min":0.872,"5min":0.872,"15min":0.872},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.3009999999999999,"5min":1.3009999999999999,"15min":1.3009999999999999},"last":1.1220000000000001},{"interface":"front","average":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":0.83299999999999996},"min":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":1.1499999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.71399999999999997}]},{"osd":2,"last update":"Mon May 22 15:27:37 2023","interfaces":[{"interface":"back","average":{"1min":0.96199999999999997,"5min":0.96199999999999997,"15min":0.96199999999999997},"min":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"max":{"1min":1.284,"5min":1.284,"15min":1.284},"last":0.78000000000000003},{"interface":"front","average":{"1min":1.0780000000000001,"5min":1.0780000000000001,"15min":1.0780000000000001},"min":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"max":{"1min":1.9770000000000001,"5min":1.9770000000000001,"15min":1.9770000000000001},"last":0.97499999999999998}]},{"osd":3,"last update":"Mon May 22 15:27:37 2023","interfaces":[{"interface":"back","average":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"min":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.4850000000000001,"5min":1.4850000000000001,"15min":1.4850000000000001},"last":0.81699999999999995},{"interface":"front","average":{"1min":1.1100000000000001,"5min":1.1100000000000001,"15min":1.1100000000000001},"min":{"1min":0.872,"5min":0.872,"15min":0.872},"max":{"1min":1.468,"5min":1.468,"15min":1.468},"last":0.86199999999999999}]},{"osd":5,"last update":"Mon May 22 15:27:37 2023","interfaces":[{"interface":"back","average":{"1min":1.123,"5min":1.123,"15min":1.123},"min":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"max":{"1min":1.7270000000000001,"5min":1.7270000000000001,"15min":1.7270000000000001},"last":1.042},{"interface":"front","average":{"1min":1.024,"5min":1.024,"15min":1.024},"min":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"max":{"1min":1.417,"5min":1.417,"15min":1.417},"last":1.0680000000000001}]},{"osd":6,"last update":"Mon May 22 15:27:37 2023","interfaces":[{"interface":"back","average":{"1min":1.181,"5min":1.181,"15min":1.181},"min":{"1min":0.96899999999999997,"5min":0.96899999999999997,"15min":0.96899999999999997},"max":{"1min":1.6020000000000001,"5min":1.6020000000000001,"15min":1.6020000000000001},"last":1.0089999999999999},{"interface":"front","average":{"1min":1.202,"5min":1.202,"15min":1.202},"min":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"max":{"1min":2.0699999999999998,"5min":2.0699999999999998,"15min":2.0699999999999998},"last":1.1890000000000001}]},{"osd":7,"last update":"Mon May 22 15:27:37 2023","interfaces":[{"interface":"back","average":{"1min":1.1919999999999999,"5min":1.1919999999999999,"15min":1.1919999999999999},"min":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.90800000000000003},"max":{"1min":1.6639999999999999,"5min":1.6639999999999999,"15min":1.6639999999999999},"last":0.90800000000000003},{"interface":"front","average":{"1min":1.2210000000000001,"5min":1.2210000000000001,"15min":1.2210000000000001},"min":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"max":{"1min":1.7410000000000001,"5min":1.7410000000000001,"15min":1.7410000000000001},"last":1.1559999999999999}]}]},{"osd":3,"up_from":13,"seq":55834574869,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8500,"kb_used_data":428,"kb_used_omap":0,"kb_used_meta":8064,"kb_avail":94363340,"statfs":{"total":96636764160,"available":96628060160,"internally_reserved":0,"allocated":438272,"data_stored":190925,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8257536},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 15:27:37 2023","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.42999999999999999}]},{"osd":1,"last update":"Mon May 22 15:27:37 2023","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.505},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":1.6579999999999999,"5min":1.6579999999999999,"15min":1.6579999999999999},"last":0.49099999999999999}]},{"osd":2,"last update":"Mon May 22 15:27:37 2023","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":1.2490000000000001,"5min":1.2490000000000001,"15min":1.2490000000000001},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":1.855,"5min":1.855,"15min":1.855},"last":0.53000000000000003}]},{"osd":4,"last update":"Mon May 22 15:27:37 2023","interfaces":[{"interface":"back","average":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"min":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":1.5820000000000001,"5min":1.5820000000000001,"15min":1.5820000000000001},"last":0.77000000000000002},{"interface":"front","average":{"1min":0.96899999999999997,"5min":0.96899999999999997,"15min":0.96899999999999997},"min":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"max":{"1min":1.8069999999999999,"5min":1.8069999999999999,"15min":1.8069999999999999},"last":0.66800000000000004}]},{"osd":5,"last update":"Mon May 22 15:27:37 2023","interfaces":[{"interface":"back","average":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":0.83799999999999997},"min":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":1.0840000000000001,"5min":1.0840000000000001,"15min":1.0840000000000001},"last":0.69499999999999995},{"interface":"front","average":{"1min":1.0249999999999999,"5min":1.0249999999999999,"15min":1.0249999999999999},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":2.4820000000000002,"5min":2.4820000000000002,"15min":2.4820000000000002},"last":0.78300000000000003}]},{"osd":6,"last update":"Mon May 22 15:27:37 2023","interfaces":[{"interface":"back","average":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":0.90300000000000002},"min":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"max":{"1min":1.452,"5min":1.452,"15min":1.452},"last":0.623},{"interface":"front","average":{"1min":1.085,"5min":1.085,"15min":1.085},"min":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"max":{"1min":2.6589999999999998,"5min":2.6589999999999998,"15min":2.6589999999999998},"last":0.88500000000000001}]},{"osd":7,"last update":"Mon May 22 15:27:37 2023","interfaces":[{"interface":"back","average":{"1min":1.1459999999999999,"5min":1.1459999999999999,"15min":1.1459999999999999},"min":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"max":{"1min":2.762,"5min":2.762,"15min":2.762},"last":0.73499999999999999},{"interface":"front","average":{"1min":1.0369999999999999,"5min":1.0369999999999999,"15min":1.0369999999999999},"min":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"max":{"1min":1.714,"5min":1.714,"15min":1.714},"last":0.64400000000000002}]}]},{"osd":2,"up_from":13,"seq":55834574870,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":12532,"kb_used_data":428,"kb_used_omap":0,"kb_used_meta":12096,"kb_avail":94359308,"statfs":{"total":96636764160,"available":96623931392,"internally_reserved":0,"allocated":438272,"data_stored":190925,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":12386304},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 15:27:36 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":1.4670000000000001,"5min":1.4670000000000001,"15min":1.4670000000000001},"last":0.55700000000000005}]},{"osd":1,"last update":"Mon May 22 15:27:36 2023","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":1.5309999999999999,"5min":1.5309999999999999,"15min":1.5309999999999999},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":1.2210000000000001,"5min":1.2210000000000001,"15min":1.2210000000000001},"last":0.46899999999999997}]},{"osd":3,"last update":"Mon May 22 15:27:36 2023","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":1.3340000000000001,"5min":1.3340000000000001,"15min":1.3340000000000001},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.754,"5min":0.754,"15min":0.754},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":1.6339999999999999,"5min":1.6339999999999999,"15min":1.6339999999999999},"last":0.79500000000000004}]},{"osd":4,"last update":"Mon May 22 15:27:36 2023","interfaces":[{"interface":"back","average":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"min":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"max":{"1min":1.774,"5min":1.774,"15min":1.774},"last":0.91000000000000003},{"interface":"front","average":{"1min":0.97599999999999998,"5min":0.97599999999999998,"15min":0.97599999999999998},"min":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"max":{"1min":1.802,"5min":1.802,"15min":1.802},"last":1.0600000000000001}]},{"osd":5,"last update":"Mon May 22 15:27:36 2023","interfaces":[{"interface":"back","average":{"1min":1.05,"5min":1.05,"15min":1.05},"min":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"max":{"1min":2.4569999999999999,"5min":2.4569999999999999,"15min":2.4569999999999999},"last":0.93500000000000005},{"interface":"front","average":{"1min":1,"5min":1,"15min":1},"min":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"max":{"1min":1.4670000000000001,"5min":1.4670000000000001,"15min":1.4670000000000001},"last":1.091}]},{"osd":6,"last update":"Mon May 22 15:27:36 2023","interfaces":[{"interface":"back","average":{"1min":1.093,"5min":1.093,"15min":1.093},"min":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":3.488,"5min":3.488,"15min":3.488},"last":0.95899999999999996},{"interface":"front","average":{"1min":1.151,"5min":1.151,"15min":1.151},"min":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"max":{"1min":3.7330000000000001,"5min":3.7330000000000001,"15min":3.7330000000000001},"last":1.0029999999999999}]},{"osd":7,"last update":"Mon May 22 15:27:36 2023","interfaces":[{"interface":"back","average":{"1min":1.085,"5min":1.085,"15min":1.085},"min":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"max":{"1min":2.21,"5min":2.21,"15min":2.21},"last":0.875},{"interface":"front","average":{"1min":1.0269999999999999,"5min":1.0269999999999999,"15min":1.0269999999999999},"min":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":1.6120000000000001,"5min":1.6120000000000001,"15min":1.6120000000000001},"last":1.0329999999999999}]}]},{"osd":1,"up_from":13,"seq":55834574870,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8500,"kb_used_data":428,"kb_used_omap":0,"kb_used_meta":8064,"kb_avail":94363340,"statfs":{"total":96636764160,"available":96628060160,"internally_reserved":0,"allocated":438272,"data_stored":190925,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8257536},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 15:27:36 2023","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":1.3089999999999999,"5min":1.3089999999999999,"15min":1.3089999999999999},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":1.3799999999999999,"5min":1.3799999999999999,"15min":1.3799999999999999},"last":0.95899999999999996}]},{"osd":2,"last update":"Mon May 22 15:27:36 2023","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.63,"15min":0.63},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.98299999999999998,"5min":0.98299999999999998,"15min":0.98299999999999998},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.35199999999999998}]},{"osd":3,"last update":"Mon May 22 15:27:36 2023","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":1.145,"5min":1.145,"15min":1.145},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.0760000000000001,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.49299999999999999}]},{"osd":4,"last update":"Mon May 22 15:27:36 2023","interfaces":[{"interface":"back","average":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":0.85299999999999998},"min":{"1min":0.622,"5min":0.622,"15min":0.622},"max":{"1min":1.2250000000000001,"5min":1.2250000000000001,"15min":1.2250000000000001},"last":0.86199999999999999},{"interface":"front","average":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"min":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"max":{"1min":1.272,"5min":1.272,"15min":1.272},"last":1.0109999999999999}]},{"osd":5,"last update":"Mon May 22 15:27:36 2023","interfaces":[{"interface":"back","average":{"1min":0.95999999999999996,"5min":0.95999999999999996,"15min":0.95999999999999996},"min":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"max":{"1min":1.1779999999999999,"5min":1.1779999999999999,"15min":1.1779999999999999},"last":0.68500000000000005},{"interface":"front","average":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":0.96299999999999997},"min":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"max":{"1min":1.488,"5min":1.488,"15min":1.488},"last":0.73299999999999998}]},{"osd":6,"last update":"Mon May 22 15:27:36 2023","interfaces":[{"interface":"back","average":{"1min":0.93400000000000005,"5min":0.93400000000000005,"15min":0.93400000000000005},"min":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"max":{"1min":1.28,"5min":1.28,"15min":1.28},"last":1.151},{"interface":"front","average":{"1min":1.1120000000000001,"5min":1.1120000000000001,"15min":1.1120000000000001},"min":{"1min":0.621,"5min":0.621,"15min":0.621},"max":{"1min":1.8180000000000001,"5min":1.8180000000000001,"15min":1.8180000000000001},"last":0.77400000000000002}]},{"osd":7,"last update":"Mon May 22 15:27:36 2023","interfaces":[{"interface":"back","average":{"1min":1.117,"5min":1.117,"15min":1.117},"min":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"max":{"1min":1.9079999999999999,"5min":1.9079999999999999,"15min":1.9079999999999999},"last":1.0880000000000001},{"interface":"front","average":{"1min":1.0349999999999999,"5min":1.0349999999999999,"15min":1.0349999999999999},"min":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"max":{"1min":1.5780000000000001,"5min":1.5780000000000001,"15min":1.5780000000000001},"last":0.69899999999999995}]}]},{"osd":0,"up_from":13,"seq":55834574871,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":13688,"kb_used_data":1008,"kb_used_omap":0,"kb_used_meta":12672,"kb_avail":94358152,"statfs":{"total":96636764160,"available":96622747648,"internally_reserved":0,"allocated":1032192,"data_stored":781293,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":12976128},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon May 22 15:27:37 2023","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":1.1839999999999999,"5min":1.1839999999999999,"15min":1.1839999999999999},"last":0.317},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":1.1040000000000001,"5min":1.1040000000000001,"15min":1.1040000000000001},"last":0.39800000000000002}]},{"osd":2,"last update":"Mon May 22 15:27:37 2023","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":1.23,"5min":1.23,"15min":1.23},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":1.1639999999999999,"5min":1.1639999999999999,"15min":1.1639999999999999},"last":0.47799999999999998}]},{"osd":3,"last update":"Mon May 22 15:27:37 2023","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":1.2410000000000001,"5min":1.2410000000000001,"15min":1.2410000000000001},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":1.119,"5min":1.119,"15min":1.119},"last":0.45000000000000001}]},{"osd":4,"last update":"Mon May 22 15:27:37 2023","interfaces":[{"interface":"back","average":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"min":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":1.2969999999999999,"5min":1.2969999999999999,"15min":1.2969999999999999},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"min":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"max":{"1min":1.4219999999999999,"5min":1.4219999999999999,"15min":1.4219999999999999},"last":0.79900000000000004}]},{"osd":5,"last update":"Mon May 22 15:27:37 2023","interfaces":[{"interface":"back","average":{"1min":0.999,"5min":0.999,"15min":0.999},"min":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":1.4370000000000001,"5min":1.4370000000000001,"15min":1.4370000000000001},"last":0.81799999999999995},{"interface":"front","average":{"1min":0.95099999999999996,"5min":0.95099999999999996,"15min":0.95099999999999996},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.349,"5min":1.349,"15min":1.349},"last":0.68200000000000005}]},{"osd":6,"last update":"Mon May 22 15:27:37 2023","interfaces":[{"interface":"back","average":{"1min":0.94599999999999995,"5min":0.94599999999999995,"15min":0.94599999999999995},"min":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"max":{"1min":1.2769999999999999,"5min":1.2769999999999999,"15min":1.2769999999999999},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.96899999999999997,"5min":0.96899999999999997,"15min":0.96899999999999997},"min":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":1.492,"5min":1.492,"15min":1.492},"last":0.627}]},{"osd":7,"last update":"Mon May 22 15:27:37 2023","interfaces":[{"interface":"back","average":{"1min":1.04,"5min":1.04,"15min":1.04},"min":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":1.581,"5min":1.581,"15min":1.581},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.98099999999999998,"5min":0.98099999999999998,"15min":0.98099999999999998},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.448,"5min":1.448,"15min":1.448},"last":0.84599999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-05-22T15:28:17.100 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.3f is not active+clean 2023-05-22T15:28:17.101 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.3f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-05-22T15:28:12.931051+0000', 'last_change': '2023-05-22T15:28:12.931051+0000', 'last_active': '2023-05-22T15:28:12.931051+0000', 'last_peered': '2023-05-22T15:28:12.931051+0000', 'last_clean': '2023-05-22T15:28:12.931051+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-05-22T15:28:12.931051+0000', 'last_undegraded': '2023-05-22T15:28:12.931051+0000', 'last_fullsized': '2023-05-22T15:28:12.931051+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-05-22T15:28:12.931051+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-05-22T15:28:12.931051+0000', 'last_clean_scrub_stamp': '2023-05-22T15:28:12.931051+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-05-22T15:28:17.101 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.3e is not active+clean 2023-05-22T15:28:17.101 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.3e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-05-22T15:28:12.931051+0000', 'last_change': '2023-05-22T15:28:12.931051+0000', 'last_active': '2023-05-22T15:28:12.931051+0000', 'last_peered': '2023-05-22T15:28:12.931051+0000', 'last_clean': '2023-05-22T15:28:12.931051+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-05-22T15:28:12.931051+0000', 'last_undegraded': '2023-05-22T15:28:12.931051+0000', 'last_fullsized': '2023-05-22T15:28:12.931051+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-05-22T15:28:12.931051+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-05-22T15:28:12.931051+0000', 'last_clean_scrub_stamp': '2023-05-22T15:28:12.931051+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-05-22T15:28:17.102 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.3d is not active+clean 2023-05-22T15:28:17.102 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.3d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-05-22T15:28:12.931051+0000', 'last_change': '2023-05-22T15:28:12.931051+0000', 'last_active': '2023-05-22T15:28:12.931051+0000', 'last_peered': '2023-05-22T15:28:12.931051+0000', 'last_clean': '2023-05-22T15:28:12.931051+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-05-22T15:28:12.931051+0000', 'last_undegraded': '2023-05-22T15:28:12.931051+0000', 'last_fullsized': '2023-05-22T15:28:12.931051+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-05-22T15:28:12.931051+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-05-22T15:28:12.931051+0000', 'last_clean_scrub_stamp': '2023-05-22T15:28:12.931051+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-05-22T15:28:17.102 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.3c is not active+clean 2023-05-22T15:28:17.102 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.3c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-05-22T15:28:12.931051+0000', 'last_change': '2023-05-22T15:28:12.931051+0000', 'last_active': '2023-05-22T15:28:12.931051+0000', 'last_peered': '2023-05-22T15:28:12.931051+0000', 'last_clean': '2023-05-22T15:28:12.931051+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-05-22T15:28:12.931051+0000', 'last_undegraded': '2023-05-22T15:28:12.931051+0000', 'last_fullsized': '2023-05-22T15:28:12.931051+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-05-22T15:28:12.931051+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-05-22T15:28:12.931051+0000', 'last_clean_scrub_stamp': '2023-05-22T15:28:12.931051+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-05-22T15:28:17.103 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.3b is not active+clean 2023-05-22T15:28:17.103 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.3b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-05-22T15:28:12.931051+0000', 'last_change': '2023-05-22T15:28:12.931051+0000', 'last_active': '2023-05-22T15:28:12.931051+0000', 'last_peered': '2023-05-22T15:28:12.931051+0000', 'last_clean': '2023-05-22T15:28:12.931051+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-05-22T15:28:12.931051+0000', 'last_undegraded': '2023-05-22T15:28:12.931051+0000', 'last_fullsized': '2023-05-22T15:28:12.931051+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-05-22T15:28:12.931051+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-05-22T15:28:12.931051+0000', 'last_clean_scrub_stamp': '2023-05-22T15:28:12.931051+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-05-22T15:28:17.103 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.3a is not active+clean 2023-05-22T15:28:17.104 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.3a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-05-22T15:28:12.931051+0000', 'last_change': '2023-05-22T15:28:12.931051+0000', 'last_active': '2023-05-22T15:28:12.931051+0000', 'last_peered': '2023-05-22T15:28:12.931051+0000', 'last_clean': '2023-05-22T15:28:12.931051+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-05-22T15:28:12.931051+0000', 'last_undegraded': '2023-05-22T15:28:12.931051+0000', 'last_fullsized': '2023-05-22T15:28:12.931051+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-05-22T15:28:12.931051+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-05-22T15:28:12.931051+0000', 'last_clean_scrub_stamp': '2023-05-22T15:28:12.931051+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-05-22T15:28:17.104 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.39 is not active+clean 2023-05-22T15:28:17.104 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.39', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-05-22T15:28:12.931051+0000', 'last_change': '2023-05-22T15:28:12.931051+0000', 'last_active': '2023-05-22T15:28:12.931051+0000', 'last_peered': '2023-05-22T15:28:12.931051+0000', 'last_clean': '2023-05-22T15:28:12.931051+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-05-22T15:28:12.931051+0000', 'last_undegraded': '2023-05-22T15:28:12.931051+0000', 'last_fullsized': '2023-05-22T15:28:12.931051+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-05-22T15:28:12.931051+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-05-22T15:28:12.931051+0000', 'last_clean_scrub_stamp': '2023-05-22T15:28:12.931051+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-05-22T15:28:17.105 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.38 is not active+clean 2023-05-22T15:28:17.105 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.38', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-05-22T15:28:12.931051+0000', 'last_change': '2023-05-22T15:28:12.931051+0000', 'last_active': '2023-05-22T15:28:12.931051+0000', 'last_peered': '2023-05-22T15:28:12.931051+0000', 'last_clean': '2023-05-22T15:28:12.931051+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-05-22T15:28:12.931051+0000', 'last_undegraded': '2023-05-22T15:28:12.931051+0000', 'last_fullsized': '2023-05-22T15:28:12.931051+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-05-22T15:28:12.931051+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-05-22T15:28:12.931051+0000', 'last_clean_scrub_stamp': '2023-05-22T15:28:12.931051+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-05-22T15:28:17.105 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.37 is not active+clean 2023-05-22T15:28:17.106 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.37', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-05-22T15:28:12.931051+0000', 'last_change': '2023-05-22T15:28:12.931051+0000', 'last_active': '2023-05-22T15:28:12.931051+0000', 'last_peered': '2023-05-22T15:28:12.931051+0000', 'last_clean': '2023-05-22T15:28:12.931051+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-05-22T15:28:12.931051+0000', 'last_undegraded': '2023-05-22T15:28:12.931051+0000', 'last_fullsized': '2023-05-22T15:28:12.931051+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-05-22T15:28:12.931051+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-05-22T15:28:12.931051+0000', 'last_clean_scrub_stamp': '2023-05-22T15:28:12.931051+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-05-22T15:28:17.106 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.36 is not active+clean 2023-05-22T15:28:17.106 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.36', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-05-22T15:28:12.931051+0000', 'last_change': '2023-05-22T15:28:12.931051+0000', 'last_active': '2023-05-22T15:28:12.931051+0000', 'last_peered': '2023-05-22T15:28:12.931051+0000', 'last_clean': '2023-05-22T15:28:12.931051+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-05-22T15:28:12.931051+0000', 'last_undegraded': '2023-05-22T15:28:12.931051+0000', 'last_fullsized': '2023-05-22T15:28:12.931051+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-05-22T15:28:12.931051+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-05-22T15:28:12.931051+0000', 'last_clean_scrub_stamp': '2023-05-22T15:28:12.931051+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-05-22T15:28:17.107 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.35 is not active+clean 2023-05-22T15:28:17.107 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.35', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-05-22T15:28:12.931051+0000', 'last_change': '2023-05-22T15:28:12.931051+0000', 'last_active': '2023-05-22T15:28:12.931051+0000', 'last_peered': '2023-05-22T15:28:12.931051+0000', 'last_clean': '2023-05-22T15:28:12.931051+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-05-22T15:28:12.931051+0000', 'last_undegraded': '2023-05-22T15:28:12.931051+0000', 'last_fullsized': '2023-05-22T15:28:12.931051+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-05-22T15:28:12.931051+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-05-22T15:28:12.931051+0000', 'last_clean_scrub_stamp': '2023-05-22T15:28:12.931051+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-05-22T15:28:17.107 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.34 is not active+clean 2023-05-22T15:28:17.108 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.34', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-05-22T15:28:12.931051+0000', 'last_change': '2023-05-22T15:28:12.931051+0000', 'last_active': '2023-05-22T15:28:12.931051+0000', 'last_peered': '2023-05-22T15:28:12.931051+0000', 'last_clean': '2023-05-22T15:28:12.931051+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-05-22T15:28:12.931051+0000', 'last_undegraded': '2023-05-22T15:28:12.931051+0000', 'last_fullsized': '2023-05-22T15:28:12.931051+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-05-22T15:28:12.931051+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-05-22T15:28:12.931051+0000', 'last_clean_scrub_stamp': '2023-05-22T15:28:12.931051+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-05-22T15:28:17.108 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.33 is not active+clean 2023-05-22T15:28:17.108 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.33', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-05-22T15:28:12.931051+0000', 'last_change': '2023-05-22T15:28:12.931051+0000', 'last_active': '2023-05-22T15:28:12.931051+0000', 'last_peered': '2023-05-22T15:28:12.931051+0000', 'last_clean': '2023-05-22T15:28:12.931051+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-05-22T15:28:12.931051+0000', 'last_undegraded': '2023-05-22T15:28:12.931051+0000', 'last_fullsized': '2023-05-22T15:28:12.931051+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-05-22T15:28:12.931051+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-05-22T15:28:12.931051+0000', 'last_clean_scrub_stamp': '2023-05-22T15:28:12.931051+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-05-22T15:28:17.109 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.32 is not active+clean 2023-05-22T15:28:17.109 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.32', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-05-22T15:28:12.931051+0000', 'last_change': '2023-05-22T15:28:12.931051+0000', 'last_active': '2023-05-22T15:28:12.931051+0000', 'last_peered': '2023-05-22T15:28:12.931051+0000', 'last_clean': '2023-05-22T15:28:12.931051+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-05-22T15:28:12.931051+0000', 'last_undegraded': '2023-05-22T15:28:12.931051+0000', 'last_fullsized': '2023-05-22T15:28:12.931051+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-05-22T15:28:12.931051+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-05-22T15:28:12.931051+0000', 'last_clean_scrub_stamp': '2023-05-22T15:28:12.931051+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-05-22T15:28:17.109 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.31 is not active+clean 2023-05-22T15:28:17.109 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.31', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-05-22T15:28:12.931051+0000', 'last_change': '2023-05-22T15:28:12.931051+0000', 'last_active': '2023-05-22T15:28:12.931051+0000', 'last_peered': '2023-05-22T15:28:12.931051+0000', 'last_clean': '2023-05-22T15:28:12.931051+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-05-22T15:28:12.931051+0000', 'last_undegraded': '2023-05-22T15:28:12.931051+0000', 'last_fullsized': '2023-05-22T15:28:12.931051+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-05-22T15:28:12.931051+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-05-22T15:28:12.931051+0000', 'last_clean_scrub_stamp': '2023-05-22T15:28:12.931051+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-05-22T15:28:17.110 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.30 is not active+clean 2023-05-22T15:28:17.110 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.30', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-05-22T15:28:12.931051+0000', 'last_change': '2023-05-22T15:28:12.931051+0000', 'last_active': '2023-05-22T15:28:12.931051+0000', 'last_peered': '2023-05-22T15:28:12.931051+0000', 'last_clean': '2023-05-22T15:28:12.931051+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-05-22T15:28:12.931051+0000', 'last_undegraded': '2023-05-22T15:28:12.931051+0000', 'last_fullsized': '2023-05-22T15:28:12.931051+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-05-22T15:28:12.931051+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-05-22T15:28:12.931051+0000', 'last_clean_scrub_stamp': '2023-05-22T15:28:12.931051+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-05-22T15:28:17.110 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.2f is not active+clean 2023-05-22T15:28:17.111 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.2f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-05-22T15:28:12.931051+0000', 'last_change': '2023-05-22T15:28:12.931051+0000', 'last_active': '2023-05-22T15:28:12.931051+0000', 'last_peered': '2023-05-22T15:28:12.931051+0000', 'last_clean': '2023-05-22T15:28:12.931051+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-05-22T15:28:12.931051+0000', 'last_undegraded': '2023-05-22T15:28:12.931051+0000', 'last_fullsized': '2023-05-22T15:28:12.931051+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-05-22T15:28:12.931051+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-05-22T15:28:12.931051+0000', 'last_clean_scrub_stamp': '2023-05-22T15:28:12.931051+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-05-22T15:28:17.111 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.2e is not active+clean 2023-05-22T15:28:17.111 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.2e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-05-22T15:28:12.931051+0000', 'last_change': '2023-05-22T15:28:12.931051+0000', 'last_active': '2023-05-22T15:28:12.931051+0000', 'last_peered': '2023-05-22T15:28:12.931051+0000', 'last_clean': '2023-05-22T15:28:12.931051+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-05-22T15:28:12.931051+0000', 'last_undegraded': '2023-05-22T15:28:12.931051+0000', 'last_fullsized': '2023-05-22T15:28:12.931051+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-05-22T15:28:12.931051+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-05-22T15:28:12.931051+0000', 'last_clean_scrub_stamp': '2023-05-22T15:28:12.931051+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-05-22T15:28:17.112 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.2d is not active+clean 2023-05-22T15:28:17.112 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.2d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-05-22T15:28:12.931051+0000', 'last_change': '2023-05-22T15:28:12.931051+0000', 'last_active': '2023-05-22T15:28:12.931051+0000', 'last_peered': '2023-05-22T15:28:12.931051+0000', 'last_clean': '2023-05-22T15:28:12.931051+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-05-22T15:28:12.931051+0000', 'last_undegraded': '2023-05-22T15:28:12.931051+0000', 'last_fullsized': '2023-05-22T15:28:12.931051+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-05-22T15:28:12.931051+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-05-22T15:28:12.931051+0000', 'last_clean_scrub_stamp': '2023-05-22T15:28:12.931051+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-05-22T15:28:17.112 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.2c is not active+clean 2023-05-22T15:28:17.113 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.2c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-05-22T15:28:12.931051+0000', 'last_change': '2023-05-22T15:28:12.931051+0000', 'last_active': '2023-05-22T15:28:12.931051+0000', 'last_peered': '2023-05-22T15:28:12.931051+0000', 'last_clean': '2023-05-22T15:28:12.931051+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-05-22T15:28:12.931051+0000', 'last_undegraded': '2023-05-22T15:28:12.931051+0000', 'last_fullsized': '2023-05-22T15:28:12.931051+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-05-22T15:28:12.931051+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-05-22T15:28:12.931051+0000', 'last_clean_scrub_stamp': '2023-05-22T15:28:12.931051+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-05-22T15:28:17.113 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.2b is not active+clean 2023-05-22T15:28:17.113 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.2b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-05-22T15:28:12.931051+0000', 'last_change': '2023-05-22T15:28:12.931051+0000', 'last_active': '2023-05-22T15:28:12.931051+0000', 'last_peered': '2023-05-22T15:28:12.931051+0000', 'last_clean': '2023-05-22T15:28:12.931051+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-05-22T15:28:12.931051+0000', 'last_undegraded': '2023-05-22T15:28:12.931051+0000', 'last_fullsized': '2023-05-22T15:28:12.931051+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-05-22T15:28:12.931051+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-05-22T15:28:12.931051+0000', 'last_clean_scrub_stamp': '2023-05-22T15:28:12.931051+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-05-22T15:28:17.114 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.2a is not active+clean 2023-05-22T15:28:17.114 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.2a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-05-22T15:28:12.931051+0000', 'last_change': '2023-05-22T15:28:12.931051+0000', 'last_active': '2023-05-22T15:28:12.931051+0000', 'last_peered': '2023-05-22T15:28:12.931051+0000', 'last_clean': '2023-05-22T15:28:12.931051+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-05-22T15:28:12.931051+0000', 'last_undegraded': '2023-05-22T15:28:12.931051+0000', 'last_fullsized': '2023-05-22T15:28:12.931051+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-05-22T15:28:12.931051+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-05-22T15:28:12.931051+0000', 'last_clean_scrub_stamp': '2023-05-22T15:28:12.931051+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-05-22T15:28:17.114 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.29 is not active+clean 2023-05-22T15:28:17.114 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.29', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-05-22T15:28:12.931051+0000', 'last_change': '2023-05-22T15:28:12.931051+0000', 'last_active': '2023-05-22T15:28:12.931051+0000', 'last_peered': '2023-05-22T15:28:12.931051+0000', 'last_clean': '2023-05-22T15:28:12.931051+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-05-22T15:28:12.931051+0000', 'last_undegraded': '2023-05-22T15:28:12.931051+0000', 'last_fullsized': '2023-05-22T15:28:12.931051+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-05-22T15:28:12.931051+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-05-22T15:28:12.931051+0000', 'last_clean_scrub_stamp': '2023-05-22T15:28:12.931051+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-05-22T15:28:17.115 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.28 is not active+clean 2023-05-22T15:28:17.115 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.28', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-05-22T15:28:12.931051+0000', 'last_change': '2023-05-22T15:28:12.931051+0000', 'last_active': '2023-05-22T15:28:12.931051+0000', 'last_peered': '2023-05-22T15:28:12.931051+0000', 'last_clean': '2023-05-22T15:28:12.931051+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-05-22T15:28:12.931051+0000', 'last_undegraded': '2023-05-22T15:28:12.931051+0000', 'last_fullsized': '2023-05-22T15:28:12.931051+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-05-22T15:28:12.931051+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-05-22T15:28:12.931051+0000', 'last_clean_scrub_stamp': '2023-05-22T15:28:12.931051+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-05-22T15:28:17.115 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.27 is not active+clean 2023-05-22T15:28:17.116 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.27', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-05-22T15:28:12.931051+0000', 'last_change': '2023-05-22T15:28:12.931051+0000', 'last_active': '2023-05-22T15:28:12.931051+0000', 'last_peered': '2023-05-22T15:28:12.931051+0000', 'last_clean': '2023-05-22T15:28:12.931051+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-05-22T15:28:12.931051+0000', 'last_undegraded': '2023-05-22T15:28:12.931051+0000', 'last_fullsized': '2023-05-22T15:28:12.931051+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-05-22T15:28:12.931051+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-05-22T15:28:12.931051+0000', 'last_clean_scrub_stamp': '2023-05-22T15:28:12.931051+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-05-22T15:28:17.116 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.26 is not active+clean 2023-05-22T15:28:17.116 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.26', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-05-22T15:28:12.931051+0000', 'last_change': '2023-05-22T15:28:12.931051+0000', 'last_active': '2023-05-22T15:28:12.931051+0000', 'last_peered': '2023-05-22T15:28:12.931051+0000', 'last_clean': '2023-05-22T15:28:12.931051+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-05-22T15:28:12.931051+0000', 'last_undegraded': '2023-05-22T15:28:12.931051+0000', 'last_fullsized': '2023-05-22T15:28:12.931051+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-05-22T15:28:12.931051+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-05-22T15:28:12.931051+0000', 'last_clean_scrub_stamp': '2023-05-22T15:28:12.931051+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-05-22T15:28:17.117 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.25 is not active+clean 2023-05-22T15:28:17.117 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.25', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-05-22T15:28:12.931051+0000', 'last_change': '2023-05-22T15:28:12.931051+0000', 'last_active': '2023-05-22T15:28:12.931051+0000', 'last_peered': '2023-05-22T15:28:12.931051+0000', 'last_clean': '2023-05-22T15:28:12.931051+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-05-22T15:28:12.931051+0000', 'last_undegraded': '2023-05-22T15:28:12.931051+0000', 'last_fullsized': '2023-05-22T15:28:12.931051+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-05-22T15:28:12.931051+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-05-22T15:28:12.931051+0000', 'last_clean_scrub_stamp': '2023-05-22T15:28:12.931051+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-05-22T15:28:17.117 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.24 is not active+clean 2023-05-22T15:28:17.118 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.24', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-05-22T15:28:12.931051+0000', 'last_change': '2023-05-22T15:28:12.931051+0000', 'last_active': '2023-05-22T15:28:12.931051+0000', 'last_peered': '2023-05-22T15:28:12.931051+0000', 'last_clean': '2023-05-22T15:28:12.931051+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-05-22T15:28:12.931051+0000', 'last_undegraded': '2023-05-22T15:28:12.931051+0000', 'last_fullsized': '2023-05-22T15:28:12.931051+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-05-22T15:28:12.931051+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-05-22T15:28:12.931051+0000', 'last_clean_scrub_stamp': '2023-05-22T15:28:12.931051+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-05-22T15:28:17.118 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.23 is not active+clean 2023-05-22T15:28:17.118 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.23', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-05-22T15:28:12.931051+0000', 'last_change': '2023-05-22T15:28:12.931051+0000', 'last_active': '2023-05-22T15:28:12.931051+0000', 'last_peered': '2023-05-22T15:28:12.931051+0000', 'last_clean': '2023-05-22T15:28:12.931051+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-05-22T15:28:12.931051+0000', 'last_undegraded': '2023-05-22T15:28:12.931051+0000', 'last_fullsized': '2023-05-22T15:28:12.931051+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-05-22T15:28:12.931051+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-05-22T15:28:12.931051+0000', 'last_clean_scrub_stamp': '2023-05-22T15:28:12.931051+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-05-22T15:28:17.119 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.22 is not active+clean 2023-05-22T15:28:17.119 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.22', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-05-22T15:28:12.931051+0000', 'last_change': '2023-05-22T15:28:12.931051+0000', 'last_active': '2023-05-22T15:28:12.931051+0000', 'last_peered': '2023-05-22T15:28:12.931051+0000', 'last_clean': '2023-05-22T15:28:12.931051+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-05-22T15:28:12.931051+0000', 'last_undegraded': '2023-05-22T15:28:12.931051+0000', 'last_fullsized': '2023-05-22T15:28:12.931051+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-05-22T15:28:12.931051+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-05-22T15:28:12.931051+0000', 'last_clean_scrub_stamp': '2023-05-22T15:28:12.931051+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-05-22T15:28:17.119 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.21 is not active+clean 2023-05-22T15:28:17.120 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.21', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-05-22T15:28:12.931051+0000', 'last_change': '2023-05-22T15:28:12.931051+0000', 'last_active': '2023-05-22T15:28:12.931051+0000', 'last_peered': '2023-05-22T15:28:12.931051+0000', 'last_clean': '2023-05-22T15:28:12.931051+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-05-22T15:28:12.931051+0000', 'last_undegraded': '2023-05-22T15:28:12.931051+0000', 'last_fullsized': '2023-05-22T15:28:12.931051+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-05-22T15:28:12.931051+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-05-22T15:28:12.931051+0000', 'last_clean_scrub_stamp': '2023-05-22T15:28:12.931051+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-05-22T15:28:17.120 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.20 is not active+clean 2023-05-22T15:28:17.120 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.20', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-05-22T15:28:12.931051+0000', 'last_change': '2023-05-22T15:28:12.931051+0000', 'last_active': '2023-05-22T15:28:12.931051+0000', 'last_peered': '2023-05-22T15:28:12.931051+0000', 'last_clean': '2023-05-22T15:28:12.931051+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-05-22T15:28:12.931051+0000', 'last_undegraded': '2023-05-22T15:28:12.931051+0000', 'last_fullsized': '2023-05-22T15:28:12.931051+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-05-22T15:28:12.931051+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-05-22T15:28:12.931051+0000', 'last_clean_scrub_stamp': '2023-05-22T15:28:12.931051+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-05-22T15:28:17.121 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.1e is not active+clean 2023-05-22T15:28:17.121 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.1e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-05-22T15:28:12.931051+0000', 'last_change': '2023-05-22T15:28:12.931051+0000', 'last_active': '2023-05-22T15:28:12.931051+0000', 'last_peered': '2023-05-22T15:28:12.931051+0000', 'last_clean': '2023-05-22T15:28:12.931051+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-05-22T15:28:12.931051+0000', 'last_undegraded': '2023-05-22T15:28:12.931051+0000', 'last_fullsized': '2023-05-22T15:28:12.931051+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-05-22T15:28:12.931051+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-05-22T15:28:12.931051+0000', 'last_clean_scrub_stamp': '2023-05-22T15:28:12.931051+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-05-22T15:28:17.121 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.1f is not active+clean 2023-05-22T15:28:17.121 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.1f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 39, 'state': 'creating+peering', 'last_fresh': '2023-05-22T15:28:11.938772+0000', 'last_change': '2023-05-22T15:28:11.938772+0000', 'last_active': '2023-05-22T15:28:11.927105+0000', 'last_peered': '2023-05-22T15:28:11.927105+0000', 'last_clean': '2023-05-22T15:28:11.927105+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-05-22T15:28:11.938772+0000', 'last_undegraded': '2023-05-22T15:28:11.938772+0000', 'last_fullsized': '2023-05-22T15:28:11.938772+0000', 'mapping_epoch': 39, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 39, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-05-22T15:28:11.927105+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-05-22T15:28:11.927105+0000', 'last_clean_scrub_stamp': '2023-05-22T15:28:11.927105+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-05-23T18:33:01.747002+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 6], 'acting': [3, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2023-05-22T15:28:17.122 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.1f is not active+clean 2023-05-22T15:28:17.122 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.1f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-05-22T15:28:12.931051+0000', 'last_change': '2023-05-22T15:28:12.931051+0000', 'last_active': '2023-05-22T15:28:12.931051+0000', 'last_peered': '2023-05-22T15:28:12.931051+0000', 'last_clean': '2023-05-22T15:28:12.931051+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-05-22T15:28:12.931051+0000', 'last_undegraded': '2023-05-22T15:28:12.931051+0000', 'last_fullsized': '2023-05-22T15:28:12.931051+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-05-22T15:28:12.931051+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-05-22T15:28:12.931051+0000', 'last_clean_scrub_stamp': '2023-05-22T15:28:12.931051+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-05-22T15:28:17.122 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.1e is not active+clean 2023-05-22T15:28:17.123 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.1e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 39, 'state': 'creating+peering', 'last_fresh': '2023-05-22T15:28:11.942902+0000', 'last_change': '2023-05-22T15:28:11.942902+0000', 'last_active': '2023-05-22T15:28:11.927105+0000', 'last_peered': '2023-05-22T15:28:11.927105+0000', 'last_clean': '2023-05-22T15:28:11.927105+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-05-22T15:28:11.942902+0000', 'last_undegraded': '2023-05-22T15:28:11.942902+0000', 'last_fullsized': '2023-05-22T15:28:11.942902+0000', 'mapping_epoch': 39, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 39, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-05-22T15:28:11.927105+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-05-22T15:28:11.927105+0000', 'last_clean_scrub_stamp': '2023-05-22T15:28:11.927105+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-05-24T02:21:15.037766+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 6], 'acting': [4, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2023-05-22T15:28:17.123 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.1c is not active+clean 2023-05-22T15:28:17.123 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.1c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-05-22T15:28:12.931051+0000', 'last_change': '2023-05-22T15:28:12.931051+0000', 'last_active': '2023-05-22T15:28:12.931051+0000', 'last_peered': '2023-05-22T15:28:12.931051+0000', 'last_clean': '2023-05-22T15:28:12.931051+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-05-22T15:28:12.931051+0000', 'last_undegraded': '2023-05-22T15:28:12.931051+0000', 'last_fullsized': '2023-05-22T15:28:12.931051+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-05-22T15:28:12.931051+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-05-22T15:28:12.931051+0000', 'last_clean_scrub_stamp': '2023-05-22T15:28:12.931051+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-05-22T15:28:17.124 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.1d is not active+clean 2023-05-22T15:28:17.124 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.1d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 39, 'state': 'creating+peering', 'last_fresh': '2023-05-22T15:28:11.945096+0000', 'last_change': '2023-05-22T15:28:11.945096+0000', 'last_active': '2023-05-22T15:28:11.927105+0000', 'last_peered': '2023-05-22T15:28:11.927105+0000', 'last_clean': '2023-05-22T15:28:11.927105+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-05-22T15:28:11.945096+0000', 'last_undegraded': '2023-05-22T15:28:11.945096+0000', 'last_fullsized': '2023-05-22T15:28:11.945096+0000', 'mapping_epoch': 39, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 39, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-05-22T15:28:11.927105+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-05-22T15:28:11.927105+0000', 'last_clean_scrub_stamp': '2023-05-22T15:28:11.927105+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-05-23T19:02:28.198798+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 5], 'acting': [1, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-05-22T15:28:17.124 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.1d is not active+clean 2023-05-22T15:28:17.125 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.1d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-05-22T15:28:12.931051+0000', 'last_change': '2023-05-22T15:28:12.931051+0000', 'last_active': '2023-05-22T15:28:12.931051+0000', 'last_peered': '2023-05-22T15:28:12.931051+0000', 'last_clean': '2023-05-22T15:28:12.931051+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-05-22T15:28:12.931051+0000', 'last_undegraded': '2023-05-22T15:28:12.931051+0000', 'last_fullsized': '2023-05-22T15:28:12.931051+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-05-22T15:28:12.931051+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-05-22T15:28:12.931051+0000', 'last_clean_scrub_stamp': '2023-05-22T15:28:12.931051+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-05-22T15:28:17.125 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.1c is not active+clean 2023-05-22T15:28:17.125 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.1c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 39, 'state': 'creating+peering', 'last_fresh': '2023-05-22T15:28:11.947443+0000', 'last_change': '2023-05-22T15:28:11.947443+0000', 'last_active': '2023-05-22T15:28:11.927105+0000', 'last_peered': '2023-05-22T15:28:11.927105+0000', 'last_clean': '2023-05-22T15:28:11.927105+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-05-22T15:28:11.947443+0000', 'last_undegraded': '2023-05-22T15:28:11.947443+0000', 'last_fullsized': '2023-05-22T15:28:11.947443+0000', 'mapping_epoch': 39, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 39, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-05-22T15:28:11.927105+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-05-22T15:28:11.927105+0000', 'last_clean_scrub_stamp': '2023-05-22T15:28:11.927105+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-05-23T20:42:40.893204+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 5], 'acting': [7, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2023-05-22T15:28:17.126 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.1a is not active+clean 2023-05-22T15:28:17.126 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.1a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-05-22T15:28:12.931051+0000', 'last_change': '2023-05-22T15:28:12.931051+0000', 'last_active': '2023-05-22T15:28:12.931051+0000', 'last_peered': '2023-05-22T15:28:12.931051+0000', 'last_clean': '2023-05-22T15:28:12.931051+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-05-22T15:28:12.931051+0000', 'last_undegraded': '2023-05-22T15:28:12.931051+0000', 'last_fullsized': '2023-05-22T15:28:12.931051+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-05-22T15:28:12.931051+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-05-22T15:28:12.931051+0000', 'last_clean_scrub_stamp': '2023-05-22T15:28:12.931051+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-05-22T15:28:17.126 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.1b is not active+clean 2023-05-22T15:28:17.126 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.1b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 39, 'state': 'creating+peering', 'last_fresh': '2023-05-22T15:28:11.941543+0000', 'last_change': '2023-05-22T15:28:11.941543+0000', 'last_active': '2023-05-22T15:28:11.927105+0000', 'last_peered': '2023-05-22T15:28:11.927105+0000', 'last_clean': '2023-05-22T15:28:11.927105+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-05-22T15:28:11.941543+0000', 'last_undegraded': '2023-05-22T15:28:11.941543+0000', 'last_fullsized': '2023-05-22T15:28:11.941543+0000', 'mapping_epoch': 39, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 39, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-05-22T15:28:11.927105+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-05-22T15:28:11.927105+0000', 'last_clean_scrub_stamp': '2023-05-22T15:28:11.927105+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-05-23T21:05:30.503639+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 7], 'acting': [3, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2023-05-22T15:28:17.127 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.1b is not active+clean 2023-05-22T15:28:17.127 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.1b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-05-22T15:28:12.931051+0000', 'last_change': '2023-05-22T15:28:12.931051+0000', 'last_active': '2023-05-22T15:28:12.931051+0000', 'last_peered': '2023-05-22T15:28:12.931051+0000', 'last_clean': '2023-05-22T15:28:12.931051+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-05-22T15:28:12.931051+0000', 'last_undegraded': '2023-05-22T15:28:12.931051+0000', 'last_fullsized': '2023-05-22T15:28:12.931051+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-05-22T15:28:12.931051+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-05-22T15:28:12.931051+0000', 'last_clean_scrub_stamp': '2023-05-22T15:28:12.931051+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-05-22T15:28:17.127 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.1a is not active+clean 2023-05-22T15:28:17.128 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.1a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 39, 'state': 'creating+peering', 'last_fresh': '2023-05-22T15:28:11.942842+0000', 'last_change': '2023-05-22T15:28:11.942842+0000', 'last_active': '2023-05-22T15:28:11.927105+0000', 'last_peered': '2023-05-22T15:28:11.927105+0000', 'last_clean': '2023-05-22T15:28:11.927105+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-05-22T15:28:11.942842+0000', 'last_undegraded': '2023-05-22T15:28:11.942842+0000', 'last_fullsized': '2023-05-22T15:28:11.942842+0000', 'mapping_epoch': 39, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 39, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-05-22T15:28:11.927105+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-05-22T15:28:11.927105+0000', 'last_clean_scrub_stamp': '2023-05-22T15:28:11.927105+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-05-23T21:17:33.835222+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 5], 'acting': [4, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2023-05-22T15:28:17.128 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.18 is not active+clean 2023-05-22T15:28:17.128 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.18', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-05-22T15:28:12.931051+0000', 'last_change': '2023-05-22T15:28:12.931051+0000', 'last_active': '2023-05-22T15:28:12.931051+0000', 'last_peered': '2023-05-22T15:28:12.931051+0000', 'last_clean': '2023-05-22T15:28:12.931051+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-05-22T15:28:12.931051+0000', 'last_undegraded': '2023-05-22T15:28:12.931051+0000', 'last_fullsized': '2023-05-22T15:28:12.931051+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-05-22T15:28:12.931051+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-05-22T15:28:12.931051+0000', 'last_clean_scrub_stamp': '2023-05-22T15:28:12.931051+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-05-22T15:28:17.129 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.19 is not active+clean 2023-05-22T15:28:17.129 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.19', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 39, 'state': 'creating+peering', 'last_fresh': '2023-05-22T15:28:11.948423+0000', 'last_change': '2023-05-22T15:28:11.948423+0000', 'last_active': '2023-05-22T15:28:11.927105+0000', 'last_peered': '2023-05-22T15:28:11.927105+0000', 'last_clean': '2023-05-22T15:28:11.927105+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-05-22T15:28:11.948423+0000', 'last_undegraded': '2023-05-22T15:28:11.948423+0000', 'last_fullsized': '2023-05-22T15:28:11.948423+0000', 'mapping_epoch': 39, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 39, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-05-22T15:28:11.927105+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-05-22T15:28:11.927105+0000', 'last_clean_scrub_stamp': '2023-05-22T15:28:11.927105+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-05-23T19:45:43.272774+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 1], 'acting': [5, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2023-05-22T15:28:17.129 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.19 is not active+clean 2023-05-22T15:28:17.130 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.19', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-05-22T15:28:12.931051+0000', 'last_change': '2023-05-22T15:28:12.931051+0000', 'last_active': '2023-05-22T15:28:12.931051+0000', 'last_peered': '2023-05-22T15:28:12.931051+0000', 'last_clean': '2023-05-22T15:28:12.931051+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-05-22T15:28:12.931051+0000', 'last_undegraded': '2023-05-22T15:28:12.931051+0000', 'last_fullsized': '2023-05-22T15:28:12.931051+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-05-22T15:28:12.931051+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-05-22T15:28:12.931051+0000', 'last_clean_scrub_stamp': '2023-05-22T15:28:12.931051+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-05-22T15:28:17.130 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.18 is not active+clean 2023-05-22T15:28:17.130 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.18', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 39, 'state': 'creating+peering', 'last_fresh': '2023-05-22T15:28:11.942591+0000', 'last_change': '2023-05-22T15:28:11.942591+0000', 'last_active': '2023-05-22T15:28:11.927105+0000', 'last_peered': '2023-05-22T15:28:11.927105+0000', 'last_clean': '2023-05-22T15:28:11.927105+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-05-22T15:28:11.942591+0000', 'last_undegraded': '2023-05-22T15:28:11.942591+0000', 'last_fullsized': '2023-05-22T15:28:11.942591+0000', 'mapping_epoch': 39, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 39, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-05-22T15:28:11.927105+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-05-22T15:28:11.927105+0000', 'last_clean_scrub_stamp': '2023-05-22T15:28:11.927105+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-05-23T19:02:28.198798+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2023-05-22T15:28:17.131 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.16 is not active+clean 2023-05-22T15:28:17.131 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.16', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-05-22T15:28:12.931051+0000', 'last_change': '2023-05-22T15:28:12.931051+0000', 'last_active': '2023-05-22T15:28:12.931051+0000', 'last_peered': '2023-05-22T15:28:12.931051+0000', 'last_clean': '2023-05-22T15:28:12.931051+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-05-22T15:28:12.931051+0000', 'last_undegraded': '2023-05-22T15:28:12.931051+0000', 'last_fullsized': '2023-05-22T15:28:12.931051+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-05-22T15:28:12.931051+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-05-22T15:28:12.931051+0000', 'last_clean_scrub_stamp': '2023-05-22T15:28:12.931051+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-05-22T15:28:17.131 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.17 is not active+clean 2023-05-22T15:28:17.131 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.17', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 39, 'state': 'creating+peering', 'last_fresh': '2023-05-22T15:28:11.941674+0000', 'last_change': '2023-05-22T15:28:11.941674+0000', 'last_active': '2023-05-22T15:28:11.927105+0000', 'last_peered': '2023-05-22T15:28:11.927105+0000', 'last_clean': '2023-05-22T15:28:11.927105+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-05-22T15:28:11.941674+0000', 'last_undegraded': '2023-05-22T15:28:11.941674+0000', 'last_fullsized': '2023-05-22T15:28:11.941674+0000', 'mapping_epoch': 39, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 39, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-05-22T15:28:11.927105+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-05-22T15:28:11.927105+0000', 'last_clean_scrub_stamp': '2023-05-22T15:28:11.927105+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-05-23T21:50:22.466750+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 2], 'acting': [4, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2023-05-22T15:28:17.132 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.17 is not active+clean 2023-05-22T15:28:17.132 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.17', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-05-22T15:28:12.931051+0000', 'last_change': '2023-05-22T15:28:12.931051+0000', 'last_active': '2023-05-22T15:28:12.931051+0000', 'last_peered': '2023-05-22T15:28:12.931051+0000', 'last_clean': '2023-05-22T15:28:12.931051+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-05-22T15:28:12.931051+0000', 'last_undegraded': '2023-05-22T15:28:12.931051+0000', 'last_fullsized': '2023-05-22T15:28:12.931051+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-05-22T15:28:12.931051+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-05-22T15:28:12.931051+0000', 'last_clean_scrub_stamp': '2023-05-22T15:28:12.931051+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-05-22T15:28:17.132 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.16 is not active+clean 2023-05-22T15:28:17.133 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.16', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 39, 'state': 'creating+peering', 'last_fresh': '2023-05-22T15:28:11.940470+0000', 'last_change': '2023-05-22T15:28:11.940470+0000', 'last_active': '2023-05-22T15:28:11.927105+0000', 'last_peered': '2023-05-22T15:28:11.927105+0000', 'last_clean': '2023-05-22T15:28:11.927105+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-05-22T15:28:11.940470+0000', 'last_undegraded': '2023-05-22T15:28:11.940470+0000', 'last_fullsized': '2023-05-22T15:28:11.940470+0000', 'mapping_epoch': 39, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 39, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-05-22T15:28:11.927105+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-05-22T15:28:11.927105+0000', 'last_clean_scrub_stamp': '2023-05-22T15:28:11.927105+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-05-24T01:15:50.706827+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 7], 'acting': [0, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2023-05-22T15:28:17.133 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.14 is not active+clean 2023-05-22T15:28:17.133 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.14', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-05-22T15:28:12.931051+0000', 'last_change': '2023-05-22T15:28:12.931051+0000', 'last_active': '2023-05-22T15:28:12.931051+0000', 'last_peered': '2023-05-22T15:28:12.931051+0000', 'last_clean': '2023-05-22T15:28:12.931051+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-05-22T15:28:12.931051+0000', 'last_undegraded': '2023-05-22T15:28:12.931051+0000', 'last_fullsized': '2023-05-22T15:28:12.931051+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-05-22T15:28:12.931051+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-05-22T15:28:12.931051+0000', 'last_clean_scrub_stamp': '2023-05-22T15:28:12.931051+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-05-22T15:28:17.134 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.15 is not active+clean 2023-05-22T15:28:17.134 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.15', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 39, 'state': 'creating+peering', 'last_fresh': '2023-05-22T15:28:11.950095+0000', 'last_change': '2023-05-22T15:28:11.950095+0000', 'last_active': '2023-05-22T15:28:11.927105+0000', 'last_peered': '2023-05-22T15:28:11.927105+0000', 'last_clean': '2023-05-22T15:28:11.927105+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-05-22T15:28:11.950095+0000', 'last_undegraded': '2023-05-22T15:28:11.950095+0000', 'last_fullsized': '2023-05-22T15:28:11.950095+0000', 'mapping_epoch': 39, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 39, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-05-22T15:28:11.927105+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-05-22T15:28:11.927105+0000', 'last_clean_scrub_stamp': '2023-05-22T15:28:11.927105+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-05-23T22:57:04.738345+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2023-05-22T15:28:17.134 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.15 is not active+clean 2023-05-22T15:28:17.135 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.15', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-05-22T15:28:12.931051+0000', 'last_change': '2023-05-22T15:28:12.931051+0000', 'last_active': '2023-05-22T15:28:12.931051+0000', 'last_peered': '2023-05-22T15:28:12.931051+0000', 'last_clean': '2023-05-22T15:28:12.931051+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-05-22T15:28:12.931051+0000', 'last_undegraded': '2023-05-22T15:28:12.931051+0000', 'last_fullsized': '2023-05-22T15:28:12.931051+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-05-22T15:28:12.931051+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-05-22T15:28:12.931051+0000', 'last_clean_scrub_stamp': '2023-05-22T15:28:12.931051+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-05-22T15:28:17.135 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.14 is not active+clean 2023-05-22T15:28:17.135 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.14', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 39, 'state': 'creating+peering', 'last_fresh': '2023-05-22T15:28:11.935772+0000', 'last_change': '2023-05-22T15:28:11.935772+0000', 'last_active': '2023-05-22T15:28:11.927105+0000', 'last_peered': '2023-05-22T15:28:11.927105+0000', 'last_clean': '2023-05-22T15:28:11.927105+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-05-22T15:28:11.935772+0000', 'last_undegraded': '2023-05-22T15:28:11.935772+0000', 'last_fullsized': '2023-05-22T15:28:11.935772+0000', 'mapping_epoch': 39, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 39, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-05-22T15:28:11.927105+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-05-22T15:28:11.927105+0000', 'last_clean_scrub_stamp': '2023-05-22T15:28:11.927105+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-05-23T19:30:08.027313+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 4], 'acting': [2, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-05-22T15:28:17.136 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.12 is not active+clean 2023-05-22T15:28:17.136 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.12', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-05-22T15:28:12.931051+0000', 'last_change': '2023-05-22T15:28:12.931051+0000', 'last_active': '2023-05-22T15:28:12.931051+0000', 'last_peered': '2023-05-22T15:28:12.931051+0000', 'last_clean': '2023-05-22T15:28:12.931051+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-05-22T15:28:12.931051+0000', 'last_undegraded': '2023-05-22T15:28:12.931051+0000', 'last_fullsized': '2023-05-22T15:28:12.931051+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-05-22T15:28:12.931051+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-05-22T15:28:12.931051+0000', 'last_clean_scrub_stamp': '2023-05-22T15:28:12.931051+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-05-22T15:28:17.136 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.13 is not active+clean 2023-05-22T15:28:17.137 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.13', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 39, 'state': 'creating+peering', 'last_fresh': '2023-05-22T15:28:11.938777+0000', 'last_change': '2023-05-22T15:28:11.938777+0000', 'last_active': '2023-05-22T15:28:11.927105+0000', 'last_peered': '2023-05-22T15:28:11.927105+0000', 'last_clean': '2023-05-22T15:28:11.927105+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-05-22T15:28:11.938777+0000', 'last_undegraded': '2023-05-22T15:28:11.938777+0000', 'last_fullsized': '2023-05-22T15:28:11.938777+0000', 'mapping_epoch': 39, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 39, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-05-22T15:28:11.927105+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-05-22T15:28:11.927105+0000', 'last_clean_scrub_stamp': '2023-05-22T15:28:11.927105+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-05-23T19:47:11.442121+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0], 'acting': [3, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2023-05-22T15:28:17.137 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.13 is not active+clean 2023-05-22T15:28:17.137 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.13', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-05-22T15:28:12.931051+0000', 'last_change': '2023-05-22T15:28:12.931051+0000', 'last_active': '2023-05-22T15:28:12.931051+0000', 'last_peered': '2023-05-22T15:28:12.931051+0000', 'last_clean': '2023-05-22T15:28:12.931051+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-05-22T15:28:12.931051+0000', 'last_undegraded': '2023-05-22T15:28:12.931051+0000', 'last_fullsized': '2023-05-22T15:28:12.931051+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-05-22T15:28:12.931051+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-05-22T15:28:12.931051+0000', 'last_clean_scrub_stamp': '2023-05-22T15:28:12.931051+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-05-22T15:28:17.137 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.12 is not active+clean 2023-05-22T15:28:17.138 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.12', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 39, 'state': 'creating+peering', 'last_fresh': '2023-05-22T15:28:11.949598+0000', 'last_change': '2023-05-22T15:28:11.949598+0000', 'last_active': '2023-05-22T15:28:11.927105+0000', 'last_peered': '2023-05-22T15:28:11.927105+0000', 'last_clean': '2023-05-22T15:28:11.927105+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-05-22T15:28:11.949598+0000', 'last_undegraded': '2023-05-22T15:28:11.949598+0000', 'last_fullsized': '2023-05-22T15:28:11.949598+0000', 'mapping_epoch': 39, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 39, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-05-22T15:28:11.927105+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-05-22T15:28:11.927105+0000', 'last_clean_scrub_stamp': '2023-05-22T15:28:11.927105+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-05-23T19:26:26.763456+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 2], 'acting': [7, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2023-05-22T15:28:17.138 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.10 is not active+clean 2023-05-22T15:28:17.138 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.10', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-05-22T15:28:12.931051+0000', 'last_change': '2023-05-22T15:28:12.931051+0000', 'last_active': '2023-05-22T15:28:12.931051+0000', 'last_peered': '2023-05-22T15:28:12.931051+0000', 'last_clean': '2023-05-22T15:28:12.931051+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-05-22T15:28:12.931051+0000', 'last_undegraded': '2023-05-22T15:28:12.931051+0000', 'last_fullsized': '2023-05-22T15:28:12.931051+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-05-22T15:28:12.931051+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-05-22T15:28:12.931051+0000', 'last_clean_scrub_stamp': '2023-05-22T15:28:12.931051+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-05-22T15:28:17.139 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.11 is not active+clean 2023-05-22T15:28:17.139 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.11', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 39, 'state': 'creating+peering', 'last_fresh': '2023-05-22T15:28:11.938244+0000', 'last_change': '2023-05-22T15:28:11.938244+0000', 'last_active': '2023-05-22T15:28:11.927105+0000', 'last_peered': '2023-05-22T15:28:11.927105+0000', 'last_clean': '2023-05-22T15:28:11.927105+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-05-22T15:28:11.938244+0000', 'last_undegraded': '2023-05-22T15:28:11.938244+0000', 'last_fullsized': '2023-05-22T15:28:11.938244+0000', 'mapping_epoch': 39, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 39, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-05-22T15:28:11.927105+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-05-22T15:28:11.927105+0000', 'last_clean_scrub_stamp': '2023-05-22T15:28:11.927105+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-05-24T02:09:59.757240+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0], 'acting': [3, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2023-05-22T15:28:17.139 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.11 is not active+clean 2023-05-22T15:28:17.140 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.11', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-05-22T15:28:12.931051+0000', 'last_change': '2023-05-22T15:28:12.931051+0000', 'last_active': '2023-05-22T15:28:12.931051+0000', 'last_peered': '2023-05-22T15:28:12.931051+0000', 'last_clean': '2023-05-22T15:28:12.931051+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-05-22T15:28:12.931051+0000', 'last_undegraded': '2023-05-22T15:28:12.931051+0000', 'last_fullsized': '2023-05-22T15:28:12.931051+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-05-22T15:28:12.931051+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-05-22T15:28:12.931051+0000', 'last_clean_scrub_stamp': '2023-05-22T15:28:12.931051+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-05-22T15:28:17.140 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.10 is not active+clean 2023-05-22T15:28:17.140 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.10', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 39, 'state': 'creating+peering', 'last_fresh': '2023-05-22T15:28:11.941202+0000', 'last_change': '2023-05-22T15:28:11.941202+0000', 'last_active': '2023-05-22T15:28:11.927105+0000', 'last_peered': '2023-05-22T15:28:11.927105+0000', 'last_clean': '2023-05-22T15:28:11.927105+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-05-22T15:28:11.941202+0000', 'last_undegraded': '2023-05-22T15:28:11.941202+0000', 'last_fullsized': '2023-05-22T15:28:11.941202+0000', 'mapping_epoch': 39, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 39, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-05-22T15:28:11.927105+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-05-22T15:28:11.927105+0000', 'last_clean_scrub_stamp': '2023-05-22T15:28:11.927105+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-05-23T15:35:29.715819+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 5], 'acting': [0, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2023-05-22T15:28:17.141 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.e is not active+clean 2023-05-22T15:28:17.141 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-05-22T15:28:12.931051+0000', 'last_change': '2023-05-22T15:28:12.931051+0000', 'last_active': '2023-05-22T15:28:12.931051+0000', 'last_peered': '2023-05-22T15:28:12.931051+0000', 'last_clean': '2023-05-22T15:28:12.931051+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-05-22T15:28:12.931051+0000', 'last_undegraded': '2023-05-22T15:28:12.931051+0000', 'last_fullsized': '2023-05-22T15:28:12.931051+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-05-22T15:28:12.931051+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-05-22T15:28:12.931051+0000', 'last_clean_scrub_stamp': '2023-05-22T15:28:12.931051+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-05-22T15:28:17.141 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.f is not active+clean 2023-05-22T15:28:17.142 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 39, 'state': 'creating+peering', 'last_fresh': '2023-05-22T15:28:11.935588+0000', 'last_change': '2023-05-22T15:28:11.935588+0000', 'last_active': '2023-05-22T15:28:11.927105+0000', 'last_peered': '2023-05-22T15:28:11.927105+0000', 'last_clean': '2023-05-22T15:28:11.927105+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-05-22T15:28:11.935588+0000', 'last_undegraded': '2023-05-22T15:28:11.935588+0000', 'last_fullsized': '2023-05-22T15:28:11.935588+0000', 'mapping_epoch': 39, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 39, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-05-22T15:28:11.927105+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-05-22T15:28:11.927105+0000', 'last_clean_scrub_stamp': '2023-05-22T15:28:11.927105+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-05-23T20:10:24.156615+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-05-22T15:28:17.142 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.f is not active+clean 2023-05-22T15:28:17.142 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-05-22T15:28:12.931051+0000', 'last_change': '2023-05-22T15:28:12.931051+0000', 'last_active': '2023-05-22T15:28:12.931051+0000', 'last_peered': '2023-05-22T15:28:12.931051+0000', 'last_clean': '2023-05-22T15:28:12.931051+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-05-22T15:28:12.931051+0000', 'last_undegraded': '2023-05-22T15:28:12.931051+0000', 'last_fullsized': '2023-05-22T15:28:12.931051+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-05-22T15:28:12.931051+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-05-22T15:28:12.931051+0000', 'last_clean_scrub_stamp': '2023-05-22T15:28:12.931051+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-05-22T15:28:17.142 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.e is not active+clean 2023-05-22T15:28:17.143 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 39, 'state': 'creating+peering', 'last_fresh': '2023-05-22T15:28:11.941609+0000', 'last_change': '2023-05-22T15:28:11.941609+0000', 'last_active': '2023-05-22T15:28:11.927105+0000', 'last_peered': '2023-05-22T15:28:11.927105+0000', 'last_clean': '2023-05-22T15:28:11.927105+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-05-22T15:28:11.941609+0000', 'last_undegraded': '2023-05-22T15:28:11.941609+0000', 'last_fullsized': '2023-05-22T15:28:11.941609+0000', 'mapping_epoch': 39, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 39, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-05-22T15:28:11.927105+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-05-22T15:28:11.927105+0000', 'last_clean_scrub_stamp': '2023-05-22T15:28:11.927105+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-05-24T03:27:50.363592+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 1], 'acting': [4, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2023-05-22T15:28:17.143 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.c is not active+clean 2023-05-22T15:28:17.143 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-05-22T15:28:12.931051+0000', 'last_change': '2023-05-22T15:28:12.931051+0000', 'last_active': '2023-05-22T15:28:12.931051+0000', 'last_peered': '2023-05-22T15:28:12.931051+0000', 'last_clean': '2023-05-22T15:28:12.931051+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-05-22T15:28:12.931051+0000', 'last_undegraded': '2023-05-22T15:28:12.931051+0000', 'last_fullsized': '2023-05-22T15:28:12.931051+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-05-22T15:28:12.931051+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-05-22T15:28:12.931051+0000', 'last_clean_scrub_stamp': '2023-05-22T15:28:12.931051+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-05-22T15:28:17.144 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.d is not active+clean 2023-05-22T15:28:17.144 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 39, 'state': 'creating+peering', 'last_fresh': '2023-05-22T15:28:11.947845+0000', 'last_change': '2023-05-22T15:28:11.947845+0000', 'last_active': '2023-05-22T15:28:11.927105+0000', 'last_peered': '2023-05-22T15:28:11.927105+0000', 'last_clean': '2023-05-22T15:28:11.927105+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-05-22T15:28:11.947845+0000', 'last_undegraded': '2023-05-22T15:28:11.947845+0000', 'last_fullsized': '2023-05-22T15:28:11.947845+0000', 'mapping_epoch': 39, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 39, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-05-22T15:28:11.927105+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-05-22T15:28:11.927105+0000', 'last_clean_scrub_stamp': '2023-05-22T15:28:11.927105+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-05-23T22:57:04.738345+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 1], 'acting': [5, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2023-05-22T15:28:17.144 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.d is not active+clean 2023-05-22T15:28:17.145 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-05-22T15:28:12.931051+0000', 'last_change': '2023-05-22T15:28:12.931051+0000', 'last_active': '2023-05-22T15:28:12.931051+0000', 'last_peered': '2023-05-22T15:28:12.931051+0000', 'last_clean': '2023-05-22T15:28:12.931051+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-05-22T15:28:12.931051+0000', 'last_undegraded': '2023-05-22T15:28:12.931051+0000', 'last_fullsized': '2023-05-22T15:28:12.931051+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-05-22T15:28:12.931051+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-05-22T15:28:12.931051+0000', 'last_clean_scrub_stamp': '2023-05-22T15:28:12.931051+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-05-22T15:28:17.145 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.c is not active+clean 2023-05-22T15:28:17.145 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 39, 'state': 'creating+peering', 'last_fresh': '2023-05-22T15:28:11.938489+0000', 'last_change': '2023-05-22T15:28:11.938489+0000', 'last_active': '2023-05-22T15:28:11.927105+0000', 'last_peered': '2023-05-22T15:28:11.927105+0000', 'last_clean': '2023-05-22T15:28:11.927105+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-05-22T15:28:11.938489+0000', 'last_undegraded': '2023-05-22T15:28:11.938489+0000', 'last_fullsized': '2023-05-22T15:28:11.938489+0000', 'mapping_epoch': 39, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 39, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-05-22T15:28:11.927105+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-05-22T15:28:11.927105+0000', 'last_clean_scrub_stamp': '2023-05-22T15:28:11.927105+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-05-23T19:45:43.272774+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 6], 'acting': [3, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2023-05-22T15:28:17.146 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.a is not active+clean 2023-05-22T15:28:17.146 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-05-22T15:28:12.931051+0000', 'last_change': '2023-05-22T15:28:12.931051+0000', 'last_active': '2023-05-22T15:28:12.931051+0000', 'last_peered': '2023-05-22T15:28:12.931051+0000', 'last_clean': '2023-05-22T15:28:12.931051+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-05-22T15:28:12.931051+0000', 'last_undegraded': '2023-05-22T15:28:12.931051+0000', 'last_fullsized': '2023-05-22T15:28:12.931051+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-05-22T15:28:12.931051+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-05-22T15:28:12.931051+0000', 'last_clean_scrub_stamp': '2023-05-22T15:28:12.931051+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-05-22T15:28:17.146 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.b is not active+clean 2023-05-22T15:28:17.147 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 39, 'state': 'creating+peering', 'last_fresh': '2023-05-22T15:28:11.940183+0000', 'last_change': '2023-05-22T15:28:11.940183+0000', 'last_active': '2023-05-22T15:28:11.927105+0000', 'last_peered': '2023-05-22T15:28:11.927105+0000', 'last_clean': '2023-05-22T15:28:11.927105+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-05-22T15:28:11.940183+0000', 'last_undegraded': '2023-05-22T15:28:11.940183+0000', 'last_fullsized': '2023-05-22T15:28:11.940183+0000', 'mapping_epoch': 39, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 39, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-05-22T15:28:11.927105+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-05-22T15:28:11.927105+0000', 'last_clean_scrub_stamp': '2023-05-22T15:28:11.927105+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-05-23T21:50:22.466750+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 7], 'acting': [3, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2023-05-22T15:28:17.147 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.b is not active+clean 2023-05-22T15:28:17.147 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-05-22T15:28:12.931051+0000', 'last_change': '2023-05-22T15:28:12.931051+0000', 'last_active': '2023-05-22T15:28:12.931051+0000', 'last_peered': '2023-05-22T15:28:12.931051+0000', 'last_clean': '2023-05-22T15:28:12.931051+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-05-22T15:28:12.931051+0000', 'last_undegraded': '2023-05-22T15:28:12.931051+0000', 'last_fullsized': '2023-05-22T15:28:12.931051+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-05-22T15:28:12.931051+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-05-22T15:28:12.931051+0000', 'last_clean_scrub_stamp': '2023-05-22T15:28:12.931051+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-05-22T15:28:17.147 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.a is not active+clean 2023-05-22T15:28:17.148 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 39, 'state': 'creating+peering', 'last_fresh': '2023-05-22T15:28:11.947990+0000', 'last_change': '2023-05-22T15:28:11.947990+0000', 'last_active': '2023-05-22T15:28:11.927105+0000', 'last_peered': '2023-05-22T15:28:11.927105+0000', 'last_clean': '2023-05-22T15:28:11.927105+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-05-22T15:28:11.947990+0000', 'last_undegraded': '2023-05-22T15:28:11.947990+0000', 'last_fullsized': '2023-05-22T15:28:11.947990+0000', 'mapping_epoch': 39, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 39, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-05-22T15:28:11.927105+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-05-22T15:28:11.927105+0000', 'last_clean_scrub_stamp': '2023-05-22T15:28:11.927105+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-05-23T18:18:35.201229+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 6], 'acting': [5, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2023-05-22T15:28:17.148 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.8 is not active+clean 2023-05-22T15:28:17.148 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.8', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-05-22T15:28:12.931051+0000', 'last_change': '2023-05-22T15:28:12.931051+0000', 'last_active': '2023-05-22T15:28:12.931051+0000', 'last_peered': '2023-05-22T15:28:12.931051+0000', 'last_clean': '2023-05-22T15:28:12.931051+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-05-22T15:28:12.931051+0000', 'last_undegraded': '2023-05-22T15:28:12.931051+0000', 'last_fullsized': '2023-05-22T15:28:12.931051+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-05-22T15:28:12.931051+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-05-22T15:28:12.931051+0000', 'last_clean_scrub_stamp': '2023-05-22T15:28:12.931051+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-05-22T15:28:17.149 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.9 is not active+clean 2023-05-22T15:28:17.149 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.9', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 39, 'state': 'creating+peering', 'last_fresh': '2023-05-22T15:28:11.940527+0000', 'last_change': '2023-05-22T15:28:11.940527+0000', 'last_active': '2023-05-22T15:28:11.927105+0000', 'last_peered': '2023-05-22T15:28:11.927105+0000', 'last_clean': '2023-05-22T15:28:11.927105+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-05-22T15:28:11.940527+0000', 'last_undegraded': '2023-05-22T15:28:11.940527+0000', 'last_fullsized': '2023-05-22T15:28:11.940527+0000', 'mapping_epoch': 39, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 39, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-05-22T15:28:11.927105+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-05-22T15:28:11.927105+0000', 'last_clean_scrub_stamp': '2023-05-22T15:28:11.927105+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-05-24T01:24:48.246881+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 7], 'acting': [0, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2023-05-22T15:28:17.149 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.9 is not active+clean 2023-05-22T15:28:17.150 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.9', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-05-22T15:28:12.931051+0000', 'last_change': '2023-05-22T15:28:12.931051+0000', 'last_active': '2023-05-22T15:28:12.931051+0000', 'last_peered': '2023-05-22T15:28:12.931051+0000', 'last_clean': '2023-05-22T15:28:12.931051+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-05-22T15:28:12.931051+0000', 'last_undegraded': '2023-05-22T15:28:12.931051+0000', 'last_fullsized': '2023-05-22T15:28:12.931051+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-05-22T15:28:12.931051+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-05-22T15:28:12.931051+0000', 'last_clean_scrub_stamp': '2023-05-22T15:28:12.931051+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-05-22T15:28:17.150 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.8 is not active+clean 2023-05-22T15:28:17.150 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.8', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 39, 'state': 'creating+peering', 'last_fresh': '2023-05-22T15:28:11.947499+0000', 'last_change': '2023-05-22T15:28:11.947499+0000', 'last_active': '2023-05-22T15:28:11.927105+0000', 'last_peered': '2023-05-22T15:28:11.927105+0000', 'last_clean': '2023-05-22T15:28:11.927105+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-05-22T15:28:11.947499+0000', 'last_undegraded': '2023-05-22T15:28:11.947499+0000', 'last_fullsized': '2023-05-22T15:28:11.947499+0000', 'mapping_epoch': 39, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 39, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-05-22T15:28:11.927105+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-05-22T15:28:11.927105+0000', 'last_clean_scrub_stamp': '2023-05-22T15:28:11.927105+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-05-23T18:18:35.201229+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 2], 'acting': [7, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2023-05-22T15:28:17.151 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.2 is not active+clean 2023-05-22T15:28:17.151 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.2', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-05-22T15:28:12.931051+0000', 'last_change': '2023-05-22T15:28:12.931051+0000', 'last_active': '2023-05-22T15:28:12.931051+0000', 'last_peered': '2023-05-22T15:28:12.931051+0000', 'last_clean': '2023-05-22T15:28:12.931051+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-05-22T15:28:12.931051+0000', 'last_undegraded': '2023-05-22T15:28:12.931051+0000', 'last_fullsized': '2023-05-22T15:28:12.931051+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-05-22T15:28:12.931051+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-05-22T15:28:12.931051+0000', 'last_clean_scrub_stamp': '2023-05-22T15:28:12.931051+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-05-22T15:28:17.151 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.3 is not active+clean 2023-05-22T15:28:17.152 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.3', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 39, 'state': 'creating+peering', 'last_fresh': '2023-05-22T15:28:11.947536+0000', 'last_change': '2023-05-22T15:28:11.947536+0000', 'last_active': '2023-05-22T15:28:11.927105+0000', 'last_peered': '2023-05-22T15:28:11.927105+0000', 'last_clean': '2023-05-22T15:28:11.927105+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-05-22T15:28:11.947536+0000', 'last_undegraded': '2023-05-22T15:28:11.947536+0000', 'last_fullsized': '2023-05-22T15:28:11.947536+0000', 'mapping_epoch': 39, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 39, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-05-22T15:28:11.927105+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-05-22T15:28:11.927105+0000', 'last_clean_scrub_stamp': '2023-05-22T15:28:11.927105+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-05-23T22:15:37.287885+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2023-05-22T15:28:17.152 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.3 is not active+clean 2023-05-22T15:28:17.152 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.3', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-05-22T15:28:12.931051+0000', 'last_change': '2023-05-22T15:28:12.931051+0000', 'last_active': '2023-05-22T15:28:12.931051+0000', 'last_peered': '2023-05-22T15:28:12.931051+0000', 'last_clean': '2023-05-22T15:28:12.931051+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-05-22T15:28:12.931051+0000', 'last_undegraded': '2023-05-22T15:28:12.931051+0000', 'last_fullsized': '2023-05-22T15:28:12.931051+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-05-22T15:28:12.931051+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-05-22T15:28:12.931051+0000', 'last_clean_scrub_stamp': '2023-05-22T15:28:12.931051+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-05-22T15:28:17.153 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.2 is not active+clean 2023-05-22T15:28:17.153 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.2', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 39, 'state': 'creating+peering', 'last_fresh': '2023-05-22T15:28:11.941443+0000', 'last_change': '2023-05-22T15:28:11.941443+0000', 'last_active': '2023-05-22T15:28:11.927105+0000', 'last_peered': '2023-05-22T15:28:11.927105+0000', 'last_clean': '2023-05-22T15:28:11.927105+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-05-22T15:28:11.941443+0000', 'last_undegraded': '2023-05-22T15:28:11.941443+0000', 'last_fullsized': '2023-05-22T15:28:11.941443+0000', 'mapping_epoch': 39, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 39, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-05-22T15:28:11.927105+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-05-22T15:28:11.927105+0000', 'last_clean_scrub_stamp': '2023-05-22T15:28:11.927105+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-05-23T21:05:30.503639+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 3], 'acting': [4, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2023-05-22T15:28:17.153 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.0 is not active+clean 2023-05-22T15:28:17.153 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.0', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-05-22T15:28:12.931051+0000', 'last_change': '2023-05-22T15:28:12.931051+0000', 'last_active': '2023-05-22T15:28:12.931051+0000', 'last_peered': '2023-05-22T15:28:12.931051+0000', 'last_clean': '2023-05-22T15:28:12.931051+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-05-22T15:28:12.931051+0000', 'last_undegraded': '2023-05-22T15:28:12.931051+0000', 'last_fullsized': '2023-05-22T15:28:12.931051+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-05-22T15:28:12.931051+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-05-22T15:28:12.931051+0000', 'last_clean_scrub_stamp': '2023-05-22T15:28:12.931051+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-05-22T15:28:17.154 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.1 is not active+clean 2023-05-22T15:28:17.154 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.1', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 39, 'state': 'creating+peering', 'last_fresh': '2023-05-22T15:28:11.945876+0000', 'last_change': '2023-05-22T15:28:11.945876+0000', 'last_active': '2023-05-22T15:28:11.927105+0000', 'last_peered': '2023-05-22T15:28:11.927105+0000', 'last_clean': '2023-05-22T15:28:11.927105+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-05-22T15:28:11.945876+0000', 'last_undegraded': '2023-05-22T15:28:11.945876+0000', 'last_fullsized': '2023-05-22T15:28:11.945876+0000', 'mapping_epoch': 39, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 39, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-05-22T15:28:11.927105+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-05-22T15:28:11.927105+0000', 'last_clean_scrub_stamp': '2023-05-22T15:28:11.927105+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-05-23T17:43:14.376663+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 6], 'acting': [1, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-05-22T15:28:17.154 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.1 is not active+clean 2023-05-22T15:28:17.155 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.1', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-05-22T15:28:12.931051+0000', 'last_change': '2023-05-22T15:28:12.931051+0000', 'last_active': '2023-05-22T15:28:12.931051+0000', 'last_peered': '2023-05-22T15:28:12.931051+0000', 'last_clean': '2023-05-22T15:28:12.931051+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-05-22T15:28:12.931051+0000', 'last_undegraded': '2023-05-22T15:28:12.931051+0000', 'last_fullsized': '2023-05-22T15:28:12.931051+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-05-22T15:28:12.931051+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-05-22T15:28:12.931051+0000', 'last_clean_scrub_stamp': '2023-05-22T15:28:12.931051+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-05-22T15:28:17.155 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.0 is not active+clean 2023-05-22T15:28:17.155 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.0', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 39, 'state': 'creating+peering', 'last_fresh': '2023-05-22T15:28:11.940327+0000', 'last_change': '2023-05-22T15:28:11.940327+0000', 'last_active': '2023-05-22T15:28:11.927105+0000', 'last_peered': '2023-05-22T15:28:11.927105+0000', 'last_clean': '2023-05-22T15:28:11.927105+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-05-22T15:28:11.940327+0000', 'last_undegraded': '2023-05-22T15:28:11.940327+0000', 'last_fullsized': '2023-05-22T15:28:11.940327+0000', 'mapping_epoch': 39, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 39, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-05-22T15:28:11.927105+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-05-22T15:28:11.927105+0000', 'last_clean_scrub_stamp': '2023-05-22T15:28:11.927105+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-05-23T22:23:05.013217+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3], 'acting': [0, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2023-05-22T15:28:17.156 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.7 is not active+clean 2023-05-22T15:28:17.156 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.7', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-05-22T15:28:12.931051+0000', 'last_change': '2023-05-22T15:28:12.931051+0000', 'last_active': '2023-05-22T15:28:12.931051+0000', 'last_peered': '2023-05-22T15:28:12.931051+0000', 'last_clean': '2023-05-22T15:28:12.931051+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-05-22T15:28:12.931051+0000', 'last_undegraded': '2023-05-22T15:28:12.931051+0000', 'last_fullsized': '2023-05-22T15:28:12.931051+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-05-22T15:28:12.931051+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-05-22T15:28:12.931051+0000', 'last_clean_scrub_stamp': '2023-05-22T15:28:12.931051+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-05-22T15:28:17.156 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.6 is not active+clean 2023-05-22T15:28:17.157 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.6', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 39, 'state': 'creating+peering', 'last_fresh': '2023-05-22T15:28:11.938283+0000', 'last_change': '2023-05-22T15:28:11.938283+0000', 'last_active': '2023-05-22T15:28:11.927105+0000', 'last_peered': '2023-05-22T15:28:11.927105+0000', 'last_clean': '2023-05-22T15:28:11.927105+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-05-22T15:28:11.938283+0000', 'last_undegraded': '2023-05-22T15:28:11.938283+0000', 'last_fullsized': '2023-05-22T15:28:11.938283+0000', 'mapping_epoch': 39, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 39, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-05-22T15:28:11.927105+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-05-22T15:28:11.927105+0000', 'last_clean_scrub_stamp': '2023-05-22T15:28:11.927105+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-05-24T02:21:15.037766+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 4], 'acting': [3, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2023-05-22T15:28:17.157 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.4 is not active+clean 2023-05-22T15:28:17.157 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.4', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-05-22T15:28:12.931051+0000', 'last_change': '2023-05-22T15:28:12.931051+0000', 'last_active': '2023-05-22T15:28:12.931051+0000', 'last_peered': '2023-05-22T15:28:12.931051+0000', 'last_clean': '2023-05-22T15:28:12.931051+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-05-22T15:28:12.931051+0000', 'last_undegraded': '2023-05-22T15:28:12.931051+0000', 'last_fullsized': '2023-05-22T15:28:12.931051+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-05-22T15:28:12.931051+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-05-22T15:28:12.931051+0000', 'last_clean_scrub_stamp': '2023-05-22T15:28:12.931051+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-05-22T15:28:17.158 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.5 is not active+clean 2023-05-22T15:28:17.158 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.5', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 39, 'state': 'creating+peering', 'last_fresh': '2023-05-22T15:28:11.948734+0000', 'last_change': '2023-05-22T15:28:11.948734+0000', 'last_active': '2023-05-22T15:28:11.927105+0000', 'last_peered': '2023-05-22T15:28:11.927105+0000', 'last_clean': '2023-05-22T15:28:11.927105+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-05-22T15:28:11.948734+0000', 'last_undegraded': '2023-05-22T15:28:11.948734+0000', 'last_fullsized': '2023-05-22T15:28:11.948734+0000', 'mapping_epoch': 39, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 39, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-05-22T15:28:11.927105+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-05-22T15:28:11.927105+0000', 'last_clean_scrub_stamp': '2023-05-22T15:28:11.927105+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-05-23T19:45:43.272774+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2023-05-22T15:28:17.158 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.5 is not active+clean 2023-05-22T15:28:17.158 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.5', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-05-22T15:28:12.931051+0000', 'last_change': '2023-05-22T15:28:12.931051+0000', 'last_active': '2023-05-22T15:28:12.931051+0000', 'last_peered': '2023-05-22T15:28:12.931051+0000', 'last_clean': '2023-05-22T15:28:12.931051+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-05-22T15:28:12.931051+0000', 'last_undegraded': '2023-05-22T15:28:12.931051+0000', 'last_fullsized': '2023-05-22T15:28:12.931051+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-05-22T15:28:12.931051+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-05-22T15:28:12.931051+0000', 'last_clean_scrub_stamp': '2023-05-22T15:28:12.931051+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-05-22T15:28:17.159 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.4 is not active+clean 2023-05-22T15:28:17.159 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.4', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 39, 'state': 'creating+peering', 'last_fresh': '2023-05-22T15:28:11.945024+0000', 'last_change': '2023-05-22T15:28:11.945024+0000', 'last_active': '2023-05-22T15:28:11.927105+0000', 'last_peered': '2023-05-22T15:28:11.927105+0000', 'last_clean': '2023-05-22T15:28:11.927105+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-05-22T15:28:11.945024+0000', 'last_undegraded': '2023-05-22T15:28:11.945024+0000', 'last_fullsized': '2023-05-22T15:28:11.945024+0000', 'mapping_epoch': 39, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 39, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-05-22T15:28:11.927105+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-05-22T15:28:11.927105+0000', 'last_clean_scrub_stamp': '2023-05-22T15:28:11.927105+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-05-23T22:23:05.013217+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 5], 'acting': [1, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-05-22T15:28:17.159 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.6 is not active+clean 2023-05-22T15:28:17.160 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.6', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-05-22T15:28:12.931051+0000', 'last_change': '2023-05-22T15:28:12.931051+0000', 'last_active': '2023-05-22T15:28:12.931051+0000', 'last_peered': '2023-05-22T15:28:12.931051+0000', 'last_clean': '2023-05-22T15:28:12.931051+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-05-22T15:28:12.931051+0000', 'last_undegraded': '2023-05-22T15:28:12.931051+0000', 'last_fullsized': '2023-05-22T15:28:12.931051+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-05-22T15:28:12.931051+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-05-22T15:28:12.931051+0000', 'last_clean_scrub_stamp': '2023-05-22T15:28:12.931051+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-05-22T15:28:17.160 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.7 is not active+clean 2023-05-22T15:28:17.160 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.7', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 39, 'state': 'creating+peering', 'last_fresh': '2023-05-22T15:28:11.947908+0000', 'last_change': '2023-05-22T15:28:11.947908+0000', 'last_active': '2023-05-22T15:28:11.927105+0000', 'last_peered': '2023-05-22T15:28:11.927105+0000', 'last_clean': '2023-05-22T15:28:11.927105+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-05-22T15:28:11.947908+0000', 'last_undegraded': '2023-05-22T15:28:11.947908+0000', 'last_fullsized': '2023-05-22T15:28:11.947908+0000', 'mapping_epoch': 39, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 39, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-05-22T15:28:11.927105+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-05-22T15:28:11.927105+0000', 'last_clean_scrub_stamp': '2023-05-22T15:28:11.927105+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-05-23T22:15:37.287885+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 3], 'acting': [5, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2023-05-22T15:28:17.161 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2023-05-22T15:28:17.287 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:17.279+0000 7fa8502ae640 1 Processor -- start 2023-05-22T15:28:17.287 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:17.279+0000 7fa8502ae640 1 -- start start 2023-05-22T15:28:17.288 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:17.283+0000 7fa8502ae640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fa8480c9b40 0x7fa84815e460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:17.288 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:17.283+0000 7fa8502ae640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8480b0650 con 0x7fa848070df0 2023-05-22T15:28:17.289 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:17.283+0000 7fa8502ae640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8480b0370 con 0x7fa8480ca4e0 2023-05-22T15:28:17.289 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:17.283+0000 7fa8502ae640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa84815d980 con 0x7fa8480c9b40 2023-05-22T15:28:17.289 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:17.283+0000 7fa84d849640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fa8480c9b40 0x7fa84815e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:17.289 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:17.283+0000 7fa84e84b640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fa848070df0 0x7fa848071210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:39958/0 (socket says 172.21.15.16:39958) 2023-05-22T15:28:17.289 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:17.283+0000 7fa84e84b640 1 -- 172.21.15.16:0/2631985740 learned_addr learned my addr 172.21.15.16:0/2631985740 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:28:17.290 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:17.283+0000 7fa84d849640 1 -- 172.21.15.16:0/2631985740 >> v1:172.21.15.62:6789/0 conn(0x7fa8480ca4e0 legacy=0x7fa8480afb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:28:17.290 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:17.283+0000 7fa84d849640 1 -- 172.21.15.16:0/2631985740 >> v1:172.21.15.16:6789/0 conn(0x7fa848070df0 legacy=0x7fa848071210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:17.290 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:17.283+0000 7fa84d849640 1 -- 172.21.15.16:0/2631985740 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa84815dc60 con 0x7fa8480c9b40 2023-05-22T15:28:17.292 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:17.287+0000 7fa84d849640 1 --2- 172.21.15.16:0/2631985740 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa8480c9b40 0x7fa84815e460 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7fa8380097c0 tx=0x7fa83801c410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd7f7a6dc283217d server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:28:17.292 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:17.287+0000 7fa84d048640 1 -- 172.21.15.16:0/2631985740 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8380ab600 con 0x7fa8480c9b40 2023-05-22T15:28:17.293 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:17.287+0000 7fa84d048640 1 -- 172.21.15.16:0/2631985740 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8380a7050 con 0x7fa8480c9b40 2023-05-22T15:28:17.293 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:17.287+0000 7fa84d048640 1 -- 172.21.15.16:0/2631985740 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8380ab980 con 0x7fa8480c9b40 2023-05-22T15:28:17.293 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:17.287+0000 7fa8502ae640 1 -- 172.21.15.16:0/2631985740 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa8480c9b40 msgr2=0x7fa84815e460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:17.294 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:17.287+0000 7fa8502ae640 1 --2- 172.21.15.16:0/2631985740 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa8480c9b40 0x7fa84815e460 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7fa8380097c0 tx=0x7fa83801c410 comp rx=0 tx=0).stop 2023-05-22T15:28:17.294 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:17.287+0000 7fa8502ae640 1 -- 172.21.15.16:0/2631985740 shutdown_connections 2023-05-22T15:28:17.294 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:17.287+0000 7fa8502ae640 1 --2- 172.21.15.16:0/2631985740 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa8480c9b40 0x7fa84815e460 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:17.294 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:17.287+0000 7fa8502ae640 1 -- 172.21.15.16:0/2631985740 >> 172.21.15.16:0/2631985740 conn(0x7fa848069720 msgr2=0x7fa848069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:28:17.294 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:17.287+0000 7fa8502ae640 1 -- 172.21.15.16:0/2631985740 shutdown_connections 2023-05-22T15:28:17.294 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:17.287+0000 7fa8502ae640 1 -- 172.21.15.16:0/2631985740 wait complete. 2023-05-22T15:28:17.295 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:17.287+0000 7fa8502ae640 1 Processor -- start 2023-05-22T15:28:17.295 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:17.287+0000 7fa8502ae640 1 -- start start 2023-05-22T15:28:17.295 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:17.287+0000 7fa8502ae640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa848070df0 0x7fa8481481c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:17.295 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:17.287+0000 7fa8502ae640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa8480c9b40 0x7fa848150c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:17.296 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:17.287+0000 7fa8502ae640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa8480ca4e0 0x7fa848151160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:17.296 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:17.287+0000 7fa8502ae640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa84814ac10 con 0x7fa848070df0 2023-05-22T15:28:17.296 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:17.287+0000 7fa8502ae640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa84814ae00 con 0x7fa8480ca4e0 2023-05-22T15:28:17.296 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:17.287+0000 7fa8502ae640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa84814b0e0 con 0x7fa8480c9b40 2023-05-22T15:28:17.297 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:17.287+0000 7fa84e04a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa848070df0 0x7fa8481481c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:17.297 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:17.287+0000 7fa84e04a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa848070df0 0x7fa8481481c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42886/0 (socket says 172.21.15.16:42886) 2023-05-22T15:28:17.298 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:17.287+0000 7fa84e04a640 1 -- 172.21.15.16:0/3905450254 learned_addr learned my addr 172.21.15.16:0/3905450254 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:28:17.298 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:17.287+0000 7fa84d849640 1 --2- 172.21.15.16:0/3905450254 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa8480c9b40 0x7fa848150c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:17.298 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:17.287+0000 7fa84e84b640 1 --2- 172.21.15.16:0/3905450254 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa8480ca4e0 0x7fa848151160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:17.298 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:17.287+0000 7fa84e04a640 1 -- 172.21.15.16:0/3905450254 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa8480c9b40 msgr2=0x7fa848150c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:17.298 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:17.287+0000 7fa84e04a640 1 --2- 172.21.15.16:0/3905450254 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa8480c9b40 0x7fa848150c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:17.298 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:17.287+0000 7fa84e04a640 1 -- 172.21.15.16:0/3905450254 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa8480ca4e0 msgr2=0x7fa848151160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:17.298 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:17.287+0000 7fa84e04a640 1 --2- 172.21.15.16:0/3905450254 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa8480ca4e0 0x7fa848151160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:17.299 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:17.287+0000 7fa84e04a640 1 -- 172.21.15.16:0/3905450254 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa838009420 con 0x7fa848070df0 2023-05-22T15:28:17.299 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:17.287+0000 7fa84e84b640 1 --2- 172.21.15.16:0/3905450254 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa8480ca4e0 0x7fa848151160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:28:17.299 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:17.287+0000 7fa84e04a640 1 --2- 172.21.15.16:0/3905450254 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa848070df0 0x7fa8481481c0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7fa84400c450 tx=0x7fa84400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:28:17.299 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:17.291+0000 7fa83effd640 1 -- 172.21.15.16:0/3905450254 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa844012830 con 0x7fa848070df0 2023-05-22T15:28:17.299 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:17.291+0000 7fa83effd640 1 -- 172.21.15.16:0/3905450254 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8440129b0 con 0x7fa848070df0 2023-05-22T15:28:17.299 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:17.291+0000 7fa8502ae640 1 -- 172.21.15.16:0/3905450254 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8480c9010 con 0x7fa848070df0 2023-05-22T15:28:17.299 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:17.291+0000 7fa83effd640 1 -- 172.21.15.16:0/3905450254 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa844012d60 con 0x7fa848070df0 2023-05-22T15:28:17.299 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:17.291+0000 7fa8502ae640 1 -- 172.21.15.16:0/3905450254 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8480c8e80 con 0x7fa848070df0 2023-05-22T15:28:17.300 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:17.291+0000 7fa8502ae640 1 -- 172.21.15.16:0/3905450254 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8480c91a0 con 0x7fa848070df0 2023-05-22T15:28:17.300 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:17.291+0000 7fa83effd640 1 -- 172.21.15.16:0/3905450254 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7fa84401c460 con 0x7fa848070df0 2023-05-22T15:28:17.300 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:17.291+0000 7fa83effd640 1 --2- 172.21.15.16:0/3905450254 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fa82c065380 0x7fa82c067830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:17.301 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:17.291+0000 7fa83effd640 1 -- 172.21.15.16:0/3905450254 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6382+0+0 (secure 0 0 0) 0x7fa8440827f0 con 0x7fa848070df0 2023-05-22T15:28:17.301 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:17.291+0000 7fa84d849640 1 --2- 172.21.15.16:0/3905450254 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fa82c065380 0x7fa82c067830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:17.301 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:17.291+0000 7fa84d849640 1 --2- 172.21.15.16:0/3905450254 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fa82c065380 0x7fa82c067830 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fa8380ad000 tx=0x7fa838008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:28:17.302 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:17.295+0000 7fa83effd640 1 -- 172.21.15.16:0/3905450254 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fa844050360 con 0x7fa848070df0 2023-05-22T15:28:17.448 INFO:tasks.ceph.mds.c.smithi016.stderr:2023-05-22T15:28:17.439+0000 7fb9c3ccb640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-05-22T15:28:17.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:17.507+0000 7fa8502ae640 1 -- 172.21.15.16:0/3905450254 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7fa84804dbe0 con 0x7fa848070df0 2023-05-22T15:28:17.513 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:17.507+0000 7fa83effd640 1 -- 172.21.15.16:0/3905450254 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1108 (secure 0 0 0) 0x7fa844053980 con 0x7fa848070df0 2023-05-22T15:28:17.513 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:28:17.513 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"b2719834-7847-4d98-90a6-f65d57f85c49","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":112,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":42,"num_osds":8,"num_up_osds":8,"osd_up_since":1684769196,"num_in_osds":8,"osd_in_since":1684769187,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"unknown","count":64},{"state_name":"creating+peering","count":32},{"state_name":"active+clean","count":9}],"num_pgs":105,"num_pools":4,"num_objects":2,"data_bytes":590368,"bytes_used":84385792,"bytes_avail":773009727488,"bytes_total":773094113280,"unknown_pgs_ratio":0.60952383279800415,"inactive_pgs_ratio":0.30476191639900208},"fsmap":{"epoch":14,"id":2,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":2,"rank":0,"name":"c","status":"up:active","gid":5574}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2023-05-22T15:27:57.967749+0000","services":{}},"progress_events":{}} 2023-05-22T15:28:17.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:17.507+0000 7fa8502ae640 1 -- 172.21.15.16:0/3905450254 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fa82c065380 msgr2=0x7fa82c067830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:17.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:17.507+0000 7fa8502ae640 1 --2- 172.21.15.16:0/3905450254 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fa82c065380 0x7fa82c067830 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fa8380ad000 tx=0x7fa838008040 comp rx=0 tx=0).stop 2023-05-22T15:28:17.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:17.507+0000 7fa8502ae640 1 -- 172.21.15.16:0/3905450254 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa848070df0 msgr2=0x7fa8481481c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:17.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:17.507+0000 7fa8502ae640 1 --2- 172.21.15.16:0/3905450254 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa848070df0 0x7fa8481481c0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7fa84400c450 tx=0x7fa84400c910 comp rx=0 tx=0).stop 2023-05-22T15:28:17.516 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:17.507+0000 7fa8502ae640 1 -- 172.21.15.16:0/3905450254 shutdown_connections 2023-05-22T15:28:17.516 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:17.507+0000 7fa8502ae640 1 --2- 172.21.15.16:0/3905450254 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa8480ca4e0 0x7fa848151160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:17.516 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:17.507+0000 7fa8502ae640 1 --2- 172.21.15.16:0/3905450254 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fa82c065380 0x7fa82c067830 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:17.516 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:17.507+0000 7fa8502ae640 1 --2- 172.21.15.16:0/3905450254 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa8480c9b40 0x7fa848150c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:17.516 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:17.507+0000 7fa8502ae640 1 --2- 172.21.15.16:0/3905450254 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa848070df0 0x7fa8481481c0 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:17.516 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:17.507+0000 7fa8502ae640 1 -- 172.21.15.16:0/3905450254 >> 172.21.15.16:0/3905450254 conn(0x7fa848069720 msgr2=0x7fa848151b60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:28:17.516 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:17.507+0000 7fa8502ae640 1 -- 172.21.15.16:0/3905450254 shutdown_connections 2023-05-22T15:28:17.517 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:17.507+0000 7fa8502ae640 1 -- 172.21.15.16:0/3905450254 wait complete. 2023-05-22T15:28:17.533 INFO:tasks.cephfs.filesystem.ceph_manager:no progress seen, keeping timeout for now 2023-05-22T15:28:17.534 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-05-22T15:28:17.660 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:17.651+0000 7faea3592640 1 Processor -- start 2023-05-22T15:28:17.660 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:17.651+0000 7faea3592640 1 -- start start 2023-05-22T15:28:17.661 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:17.655+0000 7faea3592640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fae9c0b7dd0 0x7fae9c0ccab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:17.661 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:17.655+0000 7faea3592640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fae9c15f700 0x7fae9c15fb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:17.661 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:17.655+0000 7faea3592640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae9c0cd170 con 0x7fae9c160050 2023-05-22T15:28:17.661 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:17.655+0000 7faea3592640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae9c0d3360 con 0x7fae9c15f700 2023-05-22T15:28:17.662 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:17.655+0000 7faea3592640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae9c0d2ac0 con 0x7fae9c0b7dd0 2023-05-22T15:28:17.662 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:17.655+0000 7faea132e640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fae9c0b7dd0 0x7fae9c0ccab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:17.662 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:17.655+0000 7faea0b2d640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fae9c15f700 0x7fae9c15fb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:17.663 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:17.655+0000 7faea1b2f640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7fae9c160050 0x7fae9c0ce720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.16:40528/0 (socket says 172.21.15.16:40528) 2023-05-22T15:28:17.663 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:17.655+0000 7faea1b2f640 1 -- 172.21.15.16:0/3803532001 learned_addr learned my addr 172.21.15.16:0/3803532001 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:28:17.663 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:17.655+0000 7faea132e640 1 -- 172.21.15.16:0/3803532001 >> v2:172.21.15.62:3300/0 conn(0x7fae9c15f700 msgr2=0x7fae9c15fb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:28:17.663 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:17.655+0000 7faea132e640 1 --2- 172.21.15.16:0/3803532001 >> v2:172.21.15.62:3300/0 conn(0x7fae9c15f700 0x7fae9c15fb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:17.663 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:17.655+0000 7faea132e640 1 -- 172.21.15.16:0/3803532001 >> v1:172.21.15.62:6789/0 conn(0x7fae9c160050 legacy=0x7fae9c0ce720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:17.663 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:17.655+0000 7faea132e640 1 -- 172.21.15.16:0/3803532001 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae9c0d2da0 con 0x7fae9c0b7dd0 2023-05-22T15:28:17.664 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:17.659+0000 7faea132e640 1 --2- 172.21.15.16:0/3803532001 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fae9c0b7dd0 0x7fae9c0ccab0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fae980097c0 tx=0x7fae9801c450 comp rx=0 tx=0).ready entity=mon.2 client_cookie=74e3e3e967821e24 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:28:17.666 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:17.659+0000 7fae93fff640 1 -- 172.21.15.16:0/3803532001 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae980ab670 con 0x7fae9c0b7dd0 2023-05-22T15:28:17.666 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:17.659+0000 7fae93fff640 1 -- 172.21.15.16:0/3803532001 <== mon.2 v2:172.21.15.62:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae980ab7d0 con 0x7fae9c0b7dd0 2023-05-22T15:28:17.666 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:17.659+0000 7fae93fff640 1 -- 172.21.15.16:0/3803532001 <== mon.2 v2:172.21.15.62:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae980abb60 con 0x7fae9c0b7dd0 2023-05-22T15:28:17.667 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:17.659+0000 7faea3592640 1 -- 172.21.15.16:0/3803532001 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fae9c0b7dd0 msgr2=0x7fae9c0ccab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:17.667 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:17.659+0000 7faea3592640 1 --2- 172.21.15.16:0/3803532001 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fae9c0b7dd0 0x7fae9c0ccab0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fae980097c0 tx=0x7fae9801c450 comp rx=0 tx=0).stop 2023-05-22T15:28:17.667 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:17.659+0000 7faea3592640 1 -- 172.21.15.16:0/3803532001 shutdown_connections 2023-05-22T15:28:17.667 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:17.659+0000 7faea3592640 1 --2- 172.21.15.16:0/3803532001 >> v2:172.21.15.62:3300/0 conn(0x7fae9c15f700 0x7fae9c15fb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:17.668 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:17.659+0000 7faea3592640 1 --2- 172.21.15.16:0/3803532001 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fae9c0b7dd0 0x7fae9c0ccab0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:17.668 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:17.659+0000 7faea3592640 1 -- 172.21.15.16:0/3803532001 >> 172.21.15.16:0/3803532001 conn(0x7fae9c0655c0 msgr2=0x7fae9c070e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:28:17.668 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:17.659+0000 7faea3592640 1 -- 172.21.15.16:0/3803532001 shutdown_connections 2023-05-22T15:28:17.668 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:17.659+0000 7faea3592640 1 -- 172.21.15.16:0/3803532001 wait complete. 2023-05-22T15:28:17.672 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:17.659+0000 7faea3592640 1 Processor -- start 2023-05-22T15:28:17.672 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:17.659+0000 7faea3592640 1 -- start start 2023-05-22T15:28:17.672 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:17.659+0000 7faea3592640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fae9c0b7dd0 0x7fae9c15ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:17.672 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:17.659+0000 7faea3592640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fae9c15f700 0x7fae9c1450e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:17.672 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:17.659+0000 7faea3592640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fae9c160050 0x7fae9c0cbd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:17.673 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:17.659+0000 7faea3592640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae9c15f110 con 0x7fae9c0b7dd0 2023-05-22T15:28:17.673 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:17.659+0000 7faea3592640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae9c145620 con 0x7fae9c15f700 2023-05-22T15:28:17.673 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:17.659+0000 7faea3592640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae9c0cafb0 con 0x7fae9c160050 2023-05-22T15:28:17.673 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:17.659+0000 7faea132e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fae9c0b7dd0 0x7fae9c15ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:17.673 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:17.659+0000 7faea132e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fae9c0b7dd0 0x7fae9c15ea20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42890/0 (socket says 172.21.15.16:42890) 2023-05-22T15:28:17.673 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:17.659+0000 7faea132e640 1 -- 172.21.15.16:0/2272860181 learned_addr learned my addr 172.21.15.16:0/2272860181 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:28:17.673 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:17.659+0000 7faea0b2d640 1 --2- 172.21.15.16:0/2272860181 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fae9c15f700 0x7fae9c1450e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:17.674 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:17.659+0000 7faea1b2f640 1 --2- 172.21.15.16:0/2272860181 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fae9c160050 0x7fae9c0cbd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:17.674 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:17.659+0000 7faea132e640 1 -- 172.21.15.16:0/2272860181 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fae9c160050 msgr2=0x7fae9c0cbd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:17.674 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:17.659+0000 7faea132e640 1 --2- 172.21.15.16:0/2272860181 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fae9c160050 0x7fae9c0cbd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:17.674 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:17.663+0000 7faea132e640 1 -- 172.21.15.16:0/2272860181 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fae9c15f700 msgr2=0x7fae9c1450e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:17.674 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:17.663+0000 7faea132e640 1 --2- 172.21.15.16:0/2272860181 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fae9c15f700 0x7fae9c1450e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:17.674 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:17.663+0000 7faea132e640 1 -- 172.21.15.16:0/2272860181 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae98009420 con 0x7fae9c0b7dd0 2023-05-22T15:28:17.674 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:17.663+0000 7faea132e640 1 --2- 172.21.15.16:0/2272860181 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fae9c0b7dd0 0x7fae9c15ea20 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7fae980ad040 tx=0x7fae9801ac90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:28:17.675 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:17.663+0000 7fae91ffb640 1 -- 172.21.15.16:0/2272860181 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae980bf610 con 0x7fae9c0b7dd0 2023-05-22T15:28:17.675 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:17.663+0000 7fae91ffb640 1 -- 172.21.15.16:0/2272860181 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae980bf770 con 0x7fae9c0b7dd0 2023-05-22T15:28:17.675 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:17.663+0000 7faea3592640 1 -- 172.21.15.16:0/2272860181 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae9c168a20 con 0x7fae9c0b7dd0 2023-05-22T15:28:17.675 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:17.663+0000 7fae91ffb640 1 -- 172.21.15.16:0/2272860181 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae980bfae0 con 0x7fae9c0b7dd0 2023-05-22T15:28:17.675 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:17.663+0000 7faea3592640 1 -- 172.21.15.16:0/2272860181 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae9c168890 con 0x7fae9c0b7dd0 2023-05-22T15:28:17.675 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:17.663+0000 7faea3592640 1 -- 172.21.15.16:0/2272860181 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae9c168bb0 con 0x7fae9c0b7dd0 2023-05-22T15:28:17.676 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:17.663+0000 7fae91ffb640 1 -- 172.21.15.16:0/2272860181 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7fae980bfc40 con 0x7fae9c0b7dd0 2023-05-22T15:28:17.676 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:17.663+0000 7fae91ffb640 1 --2- 172.21.15.16:0/2272860181 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fae80065380 0x7fae80067830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:17.676 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:17.663+0000 7fae91ffb640 1 -- 172.21.15.16:0/2272860181 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6382+0+0 (secure 0 0 0) 0x7fae9811d6d0 con 0x7fae9c0b7dd0 2023-05-22T15:28:17.676 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:17.663+0000 7faea0b2d640 1 --2- 172.21.15.16:0/2272860181 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fae80065380 0x7fae80067830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:17.677 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:17.663+0000 7faea0b2d640 1 --2- 172.21.15.16:0/2272860181 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fae80065380 0x7fae80067830 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fae9c0b2bf0 tx=0x7fae8c008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:28:17.677 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:17.667+0000 7fae91ffb640 1 -- 172.21.15.16:0/2272860181 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fae980eb2a0 con 0x7fae9c0b7dd0 2023-05-22T15:28:17.823 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:17.815+0000 7faea3592640 1 -- 172.21.15.16:0/2272860181 --> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fae9c0d1e80 con 0x7fae80065380 2023-05-22T15:28:17.844 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:17.835+0000 7fae91ffb640 1 -- 172.21.15.16:0/2272860181 <== mgr.4101 v2:172.21.15.16:6832/36921 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+299300 (secure 0 0 0) 0x7fae9c0d1e80 con 0x7fae80065380 2023-05-22T15:28:17.848 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:28:17.849 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2023-05-22T15:28:17.851 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:17.843+0000 7faea3592640 1 -- 172.21.15.16:0/2272860181 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fae80065380 msgr2=0x7fae80067830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:17.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:17.843+0000 7faea3592640 1 --2- 172.21.15.16:0/2272860181 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fae80065380 0x7fae80067830 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fae9c0b2bf0 tx=0x7fae8c008040 comp rx=0 tx=0).stop 2023-05-22T15:28:17.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:17.843+0000 7faea3592640 1 -- 172.21.15.16:0/2272860181 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fae9c0b7dd0 msgr2=0x7fae9c15ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:17.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:17.843+0000 7faea3592640 1 --2- 172.21.15.16:0/2272860181 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fae9c0b7dd0 0x7fae9c15ea20 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7fae980ad040 tx=0x7fae9801ac90 comp rx=0 tx=0).stop 2023-05-22T15:28:17.853 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:17.843+0000 7faea3592640 1 -- 172.21.15.16:0/2272860181 shutdown_connections 2023-05-22T15:28:17.853 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:17.843+0000 7faea3592640 1 --2- 172.21.15.16:0/2272860181 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fae80065380 0x7fae80067830 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:17.853 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:17.843+0000 7faea3592640 1 --2- 172.21.15.16:0/2272860181 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fae9c160050 0x7fae9c0cbd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:17.853 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:17.843+0000 7faea3592640 1 --2- 172.21.15.16:0/2272860181 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fae9c15f700 0x7fae9c1450e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:17.854 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:17.843+0000 7faea3592640 1 --2- 172.21.15.16:0/2272860181 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fae9c0b7dd0 0x7fae9c15ea20 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:17.854 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:17.843+0000 7faea3592640 1 -- 172.21.15.16:0/2272860181 >> 172.21.15.16:0/2272860181 conn(0x7fae9c0655c0 msgr2=0x7fae9c158750 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:28:17.854 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:17.843+0000 7faea3592640 1 -- 172.21.15.16:0/2272860181 shutdown_connections 2023-05-22T15:28:17.855 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:17.843+0000 7faea3592640 1 -- 172.21.15.16:0/2272860181 wait complete. 2023-05-22T15:28:17.867 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":85,"stamp":"2023-05-22T15:28:15.972192+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":82408,"kb_used_data":4584,"kb_used_omap":0,"kb_used_meta":77760,"kb_avail":754892312,"statfs":{"total":773094113280,"available":773009727488,"internally_reserved":0,"allocated":4694016,"data_stored":2708136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":79626240},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.040506"},"pg_stats":[{"pgid":"7.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2023-05-22T15:28:11.938772+0000","last_change":"2023-05-22T15:28:11.938772+0000","last_active":"2023-05-22T15:28:11.927105+0000","last_peered":"2023-05-22T15:28:11.927105+0000","last_clean":"2023-05-22T15:28:11.927105+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:11.938772+0000","last_undegraded":"2023-05-22T15:28:11.938772+0000","last_fullsized":"2023-05-22T15:28:11.938772+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_clean_scrub_stamp":"2023-05-22T15:28:11.927105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T18:33:01.747002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2023-05-22T15:28:11.942902+0000","last_change":"2023-05-22T15:28:11.942902+0000","last_active":"2023-05-22T15:28:11.927105+0000","last_peered":"2023-05-22T15:28:11.927105+0000","last_clean":"2023-05-22T15:28:11.927105+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:11.942902+0000","last_undegraded":"2023-05-22T15:28:11.942902+0000","last_fullsized":"2023-05-22T15:28:11.942902+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_clean_scrub_stamp":"2023-05-22T15:28:11.927105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:21:15.037766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2023-05-22T15:28:11.945096+0000","last_change":"2023-05-22T15:28:11.945096+0000","last_active":"2023-05-22T15:28:11.927105+0000","last_peered":"2023-05-22T15:28:11.927105+0000","last_clean":"2023-05-22T15:28:11.927105+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:11.945096+0000","last_undegraded":"2023-05-22T15:28:11.945096+0000","last_fullsized":"2023-05-22T15:28:11.945096+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_clean_scrub_stamp":"2023-05-22T15:28:11.927105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:02:28.198798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2023-05-22T15:28:11.947443+0000","last_change":"2023-05-22T15:28:11.947443+0000","last_active":"2023-05-22T15:28:11.927105+0000","last_peered":"2023-05-22T15:28:11.927105+0000","last_clean":"2023-05-22T15:28:11.927105+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:11.947443+0000","last_undegraded":"2023-05-22T15:28:11.947443+0000","last_fullsized":"2023-05-22T15:28:11.947443+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_clean_scrub_stamp":"2023-05-22T15:28:11.927105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:42:40.893204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2023-05-22T15:28:11.941543+0000","last_change":"2023-05-22T15:28:11.941543+0000","last_active":"2023-05-22T15:28:11.927105+0000","last_peered":"2023-05-22T15:28:11.927105+0000","last_clean":"2023-05-22T15:28:11.927105+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:11.941543+0000","last_undegraded":"2023-05-22T15:28:11.941543+0000","last_fullsized":"2023-05-22T15:28:11.941543+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_clean_scrub_stamp":"2023-05-22T15:28:11.927105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:05:30.503639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2023-05-22T15:28:11.942842+0000","last_change":"2023-05-22T15:28:11.942842+0000","last_active":"2023-05-22T15:28:11.927105+0000","last_peered":"2023-05-22T15:28:11.927105+0000","last_clean":"2023-05-22T15:28:11.927105+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:11.942842+0000","last_undegraded":"2023-05-22T15:28:11.942842+0000","last_fullsized":"2023-05-22T15:28:11.942842+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_clean_scrub_stamp":"2023-05-22T15:28:11.927105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:17:33.835222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2023-05-22T15:28:11.948423+0000","last_change":"2023-05-22T15:28:11.948423+0000","last_active":"2023-05-22T15:28:11.927105+0000","last_peered":"2023-05-22T15:28:11.927105+0000","last_clean":"2023-05-22T15:28:11.927105+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:11.948423+0000","last_undegraded":"2023-05-22T15:28:11.948423+0000","last_fullsized":"2023-05-22T15:28:11.948423+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_clean_scrub_stamp":"2023-05-22T15:28:11.927105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:45:43.272774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2023-05-22T15:28:11.942591+0000","last_change":"2023-05-22T15:28:11.942591+0000","last_active":"2023-05-22T15:28:11.927105+0000","last_peered":"2023-05-22T15:28:11.927105+0000","last_clean":"2023-05-22T15:28:11.927105+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:11.942591+0000","last_undegraded":"2023-05-22T15:28:11.942591+0000","last_fullsized":"2023-05-22T15:28:11.942591+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_clean_scrub_stamp":"2023-05-22T15:28:11.927105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:02:28.198798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2023-05-22T15:28:11.941674+0000","last_change":"2023-05-22T15:28:11.941674+0000","last_active":"2023-05-22T15:28:11.927105+0000","last_peered":"2023-05-22T15:28:11.927105+0000","last_clean":"2023-05-22T15:28:11.927105+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:11.941674+0000","last_undegraded":"2023-05-22T15:28:11.941674+0000","last_fullsized":"2023-05-22T15:28:11.941674+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_clean_scrub_stamp":"2023-05-22T15:28:11.927105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:50:22.466750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2023-05-22T15:28:11.940470+0000","last_change":"2023-05-22T15:28:11.940470+0000","last_active":"2023-05-22T15:28:11.927105+0000","last_peered":"2023-05-22T15:28:11.927105+0000","last_clean":"2023-05-22T15:28:11.927105+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:11.940470+0000","last_undegraded":"2023-05-22T15:28:11.940470+0000","last_fullsized":"2023-05-22T15:28:11.940470+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_clean_scrub_stamp":"2023-05-22T15:28:11.927105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:15:50.706827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2023-05-22T15:28:11.950095+0000","last_change":"2023-05-22T15:28:11.950095+0000","last_active":"2023-05-22T15:28:11.927105+0000","last_peered":"2023-05-22T15:28:11.927105+0000","last_clean":"2023-05-22T15:28:11.927105+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:11.950095+0000","last_undegraded":"2023-05-22T15:28:11.950095+0000","last_fullsized":"2023-05-22T15:28:11.950095+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_clean_scrub_stamp":"2023-05-22T15:28:11.927105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:57:04.738345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2023-05-22T15:28:11.935772+0000","last_change":"2023-05-22T15:28:11.935772+0000","last_active":"2023-05-22T15:28:11.927105+0000","last_peered":"2023-05-22T15:28:11.927105+0000","last_clean":"2023-05-22T15:28:11.927105+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:11.935772+0000","last_undegraded":"2023-05-22T15:28:11.935772+0000","last_fullsized":"2023-05-22T15:28:11.935772+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_clean_scrub_stamp":"2023-05-22T15:28:11.927105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:30:08.027313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2023-05-22T15:28:11.938777+0000","last_change":"2023-05-22T15:28:11.938777+0000","last_active":"2023-05-22T15:28:11.927105+0000","last_peered":"2023-05-22T15:28:11.927105+0000","last_clean":"2023-05-22T15:28:11.927105+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:11.938777+0000","last_undegraded":"2023-05-22T15:28:11.938777+0000","last_fullsized":"2023-05-22T15:28:11.938777+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_clean_scrub_stamp":"2023-05-22T15:28:11.927105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:47:11.442121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2023-05-22T15:28:11.949598+0000","last_change":"2023-05-22T15:28:11.949598+0000","last_active":"2023-05-22T15:28:11.927105+0000","last_peered":"2023-05-22T15:28:11.927105+0000","last_clean":"2023-05-22T15:28:11.927105+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:11.949598+0000","last_undegraded":"2023-05-22T15:28:11.949598+0000","last_fullsized":"2023-05-22T15:28:11.949598+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_clean_scrub_stamp":"2023-05-22T15:28:11.927105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:26:26.763456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2023-05-22T15:28:11.938244+0000","last_change":"2023-05-22T15:28:11.938244+0000","last_active":"2023-05-22T15:28:11.927105+0000","last_peered":"2023-05-22T15:28:11.927105+0000","last_clean":"2023-05-22T15:28:11.927105+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:11.938244+0000","last_undegraded":"2023-05-22T15:28:11.938244+0000","last_fullsized":"2023-05-22T15:28:11.938244+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_clean_scrub_stamp":"2023-05-22T15:28:11.927105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:09:59.757240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2023-05-22T15:28:11.941202+0000","last_change":"2023-05-22T15:28:11.941202+0000","last_active":"2023-05-22T15:28:11.927105+0000","last_peered":"2023-05-22T15:28:11.927105+0000","last_clean":"2023-05-22T15:28:11.927105+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:11.941202+0000","last_undegraded":"2023-05-22T15:28:11.941202+0000","last_fullsized":"2023-05-22T15:28:11.941202+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_clean_scrub_stamp":"2023-05-22T15:28:11.927105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T15:35:29.715819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2023-05-22T15:28:11.935588+0000","last_change":"2023-05-22T15:28:11.935588+0000","last_active":"2023-05-22T15:28:11.927105+0000","last_peered":"2023-05-22T15:28:11.927105+0000","last_clean":"2023-05-22T15:28:11.927105+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:11.935588+0000","last_undegraded":"2023-05-22T15:28:11.935588+0000","last_fullsized":"2023-05-22T15:28:11.935588+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_clean_scrub_stamp":"2023-05-22T15:28:11.927105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:10:24.156615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2023-05-22T15:28:11.941609+0000","last_change":"2023-05-22T15:28:11.941609+0000","last_active":"2023-05-22T15:28:11.927105+0000","last_peered":"2023-05-22T15:28:11.927105+0000","last_clean":"2023-05-22T15:28:11.927105+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:11.941609+0000","last_undegraded":"2023-05-22T15:28:11.941609+0000","last_fullsized":"2023-05-22T15:28:11.941609+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_clean_scrub_stamp":"2023-05-22T15:28:11.927105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T03:27:50.363592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2023-05-22T15:28:11.947845+0000","last_change":"2023-05-22T15:28:11.947845+0000","last_active":"2023-05-22T15:28:11.927105+0000","last_peered":"2023-05-22T15:28:11.927105+0000","last_clean":"2023-05-22T15:28:11.927105+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:11.947845+0000","last_undegraded":"2023-05-22T15:28:11.947845+0000","last_fullsized":"2023-05-22T15:28:11.947845+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_clean_scrub_stamp":"2023-05-22T15:28:11.927105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:57:04.738345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2023-05-22T15:28:11.938489+0000","last_change":"2023-05-22T15:28:11.938489+0000","last_active":"2023-05-22T15:28:11.927105+0000","last_peered":"2023-05-22T15:28:11.927105+0000","last_clean":"2023-05-22T15:28:11.927105+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:11.938489+0000","last_undegraded":"2023-05-22T15:28:11.938489+0000","last_fullsized":"2023-05-22T15:28:11.938489+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_clean_scrub_stamp":"2023-05-22T15:28:11.927105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:45:43.272774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2023-05-22T15:28:11.940183+0000","last_change":"2023-05-22T15:28:11.940183+0000","last_active":"2023-05-22T15:28:11.927105+0000","last_peered":"2023-05-22T15:28:11.927105+0000","last_clean":"2023-05-22T15:28:11.927105+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:11.940183+0000","last_undegraded":"2023-05-22T15:28:11.940183+0000","last_fullsized":"2023-05-22T15:28:11.940183+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_clean_scrub_stamp":"2023-05-22T15:28:11.927105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:50:22.466750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2023-05-22T15:28:11.947990+0000","last_change":"2023-05-22T15:28:11.947990+0000","last_active":"2023-05-22T15:28:11.927105+0000","last_peered":"2023-05-22T15:28:11.927105+0000","last_clean":"2023-05-22T15:28:11.927105+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:11.947990+0000","last_undegraded":"2023-05-22T15:28:11.947990+0000","last_fullsized":"2023-05-22T15:28:11.947990+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_clean_scrub_stamp":"2023-05-22T15:28:11.927105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T18:18:35.201229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2023-05-22T15:28:11.940527+0000","last_change":"2023-05-22T15:28:11.940527+0000","last_active":"2023-05-22T15:28:11.927105+0000","last_peered":"2023-05-22T15:28:11.927105+0000","last_clean":"2023-05-22T15:28:11.927105+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:11.940527+0000","last_undegraded":"2023-05-22T15:28:11.940527+0000","last_fullsized":"2023-05-22T15:28:11.940527+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_clean_scrub_stamp":"2023-05-22T15:28:11.927105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:24:48.246881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2023-05-22T15:28:11.947499+0000","last_change":"2023-05-22T15:28:11.947499+0000","last_active":"2023-05-22T15:28:11.927105+0000","last_peered":"2023-05-22T15:28:11.927105+0000","last_clean":"2023-05-22T15:28:11.927105+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:11.947499+0000","last_undegraded":"2023-05-22T15:28:11.947499+0000","last_fullsized":"2023-05-22T15:28:11.947499+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_clean_scrub_stamp":"2023-05-22T15:28:11.927105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T18:18:35.201229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2023-05-22T15:28:11.947536+0000","last_change":"2023-05-22T15:28:11.947536+0000","last_active":"2023-05-22T15:28:11.927105+0000","last_peered":"2023-05-22T15:28:11.927105+0000","last_clean":"2023-05-22T15:28:11.927105+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:11.947536+0000","last_undegraded":"2023-05-22T15:28:11.947536+0000","last_fullsized":"2023-05-22T15:28:11.947536+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_clean_scrub_stamp":"2023-05-22T15:28:11.927105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:15:37.287885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":25,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-05-22T15:28:08.260614+0000","last_change":"2023-05-22T15:26:44.982421+0000","last_active":"2023-05-22T15:28:08.260614+0000","last_peered":"2023-05-22T15:28:08.260614+0000","last_clean":"2023-05-22T15:28:08.260614+0000","last_became_active":"2023-05-22T15:26:44.981586+0000","last_became_peered":"2023-05-22T15:26:44.981586+0000","last_unstale":"2023-05-22T15:28:08.260614+0000","last_undegraded":"2023-05-22T15:28:08.260614+0000","last_fullsized":"2023-05-22T15:28:08.260614+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:14:11.156544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2023-05-22T15:28:11.941443+0000","last_change":"2023-05-22T15:28:11.941443+0000","last_active":"2023-05-22T15:28:11.927105+0000","last_peered":"2023-05-22T15:28:11.927105+0000","last_clean":"2023-05-22T15:28:11.927105+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:11.941443+0000","last_undegraded":"2023-05-22T15:28:11.941443+0000","last_fullsized":"2023-05-22T15:28:11.941443+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_clean_scrub_stamp":"2023-05-22T15:28:11.927105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:05:30.503639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":24,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-05-22T15:28:07.758852+0000","last_change":"2023-05-22T15:26:44.978858+0000","last_active":"2023-05-22T15:28:07.758852+0000","last_peered":"2023-05-22T15:28:07.758852+0000","last_clean":"2023-05-22T15:28:07.758852+0000","last_became_active":"2023-05-22T15:26:44.978443+0000","last_became_peered":"2023-05-22T15:26:44.978443+0000","last_unstale":"2023-05-22T15:28:07.758852+0000","last_undegraded":"2023-05-22T15:28:07.758852+0000","last_fullsized":"2023-05-22T15:28:07.758852+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:46:29.290243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2023-05-22T15:28:11.945876+0000","last_change":"2023-05-22T15:28:11.945876+0000","last_active":"2023-05-22T15:28:11.927105+0000","last_peered":"2023-05-22T15:28:11.927105+0000","last_clean":"2023-05-22T15:28:11.927105+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:11.945876+0000","last_undegraded":"2023-05-22T15:28:11.945876+0000","last_fullsized":"2023-05-22T15:28:11.945876+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_clean_scrub_stamp":"2023-05-22T15:28:11.927105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:43:14.376663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":28,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-05-22T15:28:07.681164+0000","last_change":"2023-05-22T15:26:45.092367+0000","last_active":"2023-05-22T15:28:07.681164+0000","last_peered":"2023-05-22T15:28:07.681164+0000","last_clean":"2023-05-22T15:28:07.681164+0000","last_became_active":"2023-05-22T15:26:45.091951+0000","last_became_peered":"2023-05-22T15:26:45.091951+0000","last_unstale":"2023-05-22T15:28:07.681164+0000","last_undegraded":"2023-05-22T15:28:07.681164+0000","last_fullsized":"2023-05-22T15:28:07.681164+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:42:57.973030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2023-05-22T15:28:11.940327+0000","last_change":"2023-05-22T15:28:11.940327+0000","last_active":"2023-05-22T15:28:11.927105+0000","last_peered":"2023-05-22T15:28:11.927105+0000","last_clean":"2023-05-22T15:28:11.927105+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:11.940327+0000","last_undegraded":"2023-05-22T15:28:11.940327+0000","last_fullsized":"2023-05-22T15:28:11.940327+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_clean_scrub_stamp":"2023-05-22T15:28:11.927105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:23:05.013217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":24,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-05-22T15:28:07.758858+0000","last_change":"2023-05-22T15:26:45.091475+0000","last_active":"2023-05-22T15:28:07.758858+0000","last_peered":"2023-05-22T15:28:07.758858+0000","last_clean":"2023-05-22T15:28:07.758858+0000","last_became_active":"2023-05-22T15:26:45.091143+0000","last_became_peered":"2023-05-22T15:26:45.091143+0000","last_unstale":"2023-05-22T15:28:07.758858+0000","last_undegraded":"2023-05-22T15:28:07.758858+0000","last_fullsized":"2023-05-22T15:28:07.758858+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:12:41.464054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2023-05-22T15:28:11.938283+0000","last_change":"2023-05-22T15:28:11.938283+0000","last_active":"2023-05-22T15:28:11.927105+0000","last_peered":"2023-05-22T15:28:11.927105+0000","last_clean":"2023-05-22T15:28:11.927105+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:11.938283+0000","last_undegraded":"2023-05-22T15:28:11.938283+0000","last_fullsized":"2023-05-22T15:28:11.938283+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_clean_scrub_stamp":"2023-05-22T15:28:11.927105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:21:15.037766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":29,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-05-22T15:28:07.680251+0000","last_change":"2023-05-22T15:26:44.976632+0000","last_active":"2023-05-22T15:28:07.680251+0000","last_peered":"2023-05-22T15:28:07.680251+0000","last_clean":"2023-05-22T15:28:07.680251+0000","last_became_active":"2023-05-22T15:26:44.975546+0000","last_became_peered":"2023-05-22T15:26:44.975546+0000","last_unstale":"2023-05-22T15:28:07.680251+0000","last_undegraded":"2023-05-22T15:28:07.680251+0000","last_fullsized":"2023-05-22T15:28:07.680251+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:42:57.973030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2023-05-22T15:28:11.948734+0000","last_change":"2023-05-22T15:28:11.948734+0000","last_active":"2023-05-22T15:28:11.927105+0000","last_peered":"2023-05-22T15:28:11.927105+0000","last_clean":"2023-05-22T15:28:11.927105+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:11.948734+0000","last_undegraded":"2023-05-22T15:28:11.948734+0000","last_fullsized":"2023-05-22T15:28:11.948734+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_clean_scrub_stamp":"2023-05-22T15:28:11.927105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:45:43.272774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":28,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-05-22T15:28:07.678593+0000","last_change":"2023-05-22T15:26:44.969469+0000","last_active":"2023-05-22T15:28:07.678593+0000","last_peered":"2023-05-22T15:28:07.678593+0000","last_clean":"2023-05-22T15:28:07.678593+0000","last_became_active":"2023-05-22T15:26:44.969083+0000","last_became_peered":"2023-05-22T15:26:44.969083+0000","last_unstale":"2023-05-22T15:28:07.678593+0000","last_undegraded":"2023-05-22T15:28:07.678593+0000","last_fullsized":"2023-05-22T15:28:07.678593+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:29:23.670067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2023-05-22T15:28:11.945024+0000","last_change":"2023-05-22T15:28:11.945024+0000","last_active":"2023-05-22T15:28:11.927105+0000","last_peered":"2023-05-22T15:28:11.927105+0000","last_clean":"2023-05-22T15:28:11.927105+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:11.945024+0000","last_undegraded":"2023-05-22T15:28:11.945024+0000","last_fullsized":"2023-05-22T15:28:11.945024+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_clean_scrub_stamp":"2023-05-22T15:28:11.927105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:23:05.013217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":28,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-05-22T15:28:07.681164+0000","last_change":"2023-05-22T15:26:44.977198+0000","last_active":"2023-05-22T15:28:07.681164+0000","last_peered":"2023-05-22T15:28:07.681164+0000","last_clean":"2023-05-22T15:28:07.681164+0000","last_became_active":"2023-05-22T15:26:44.976445+0000","last_became_peered":"2023-05-22T15:26:44.976445+0000","last_unstale":"2023-05-22T15:28:07.681164+0000","last_undegraded":"2023-05-22T15:28:07.681164+0000","last_fullsized":"2023-05-22T15:28:07.681164+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:12:41.464054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-05-22T15:28:12.931051+0000","last_change":"2023-05-22T15:28:12.931051+0000","last_active":"2023-05-22T15:28:12.931051+0000","last_peered":"2023-05-22T15:28:12.931051+0000","last_clean":"2023-05-22T15:28:12.931051+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:12.931051+0000","last_undegraded":"2023-05-22T15:28:12.931051+0000","last_fullsized":"2023-05-22T15:28:12.931051+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2023-05-22T15:28:11.947908+0000","last_change":"2023-05-22T15:28:11.947908+0000","last_active":"2023-05-22T15:28:11.927105+0000","last_peered":"2023-05-22T15:28:11.927105+0000","last_clean":"2023-05-22T15:28:11.927105+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:28:11.947908+0000","last_undegraded":"2023-05-22T15:28:11.947908+0000","last_fullsized":"2023-05-22T15:28:11.947908+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_clean_scrub_stamp":"2023-05-22T15:28:11.927105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:15:37.287885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":29,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-05-22T15:28:07.680257+0000","last_change":"2023-05-22T15:26:44.971951+0000","last_active":"2023-05-22T15:28:07.680257+0000","last_peered":"2023-05-22T15:28:07.680257+0000","last_clean":"2023-05-22T15:28:07.680257+0000","last_became_active":"2023-05-22T15:26:44.970872+0000","last_became_peered":"2023-05-22T15:26:44.970872+0000","last_unstale":"2023-05-22T15:28:07.680257+0000","last_undegraded":"2023-05-22T15:28:07.680257+0000","last_fullsized":"2023-05-22T15:28:07.680257+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:55:34.752949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":181,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-05-22T15:28:07.681354+0000","last_change":"2023-05-22T15:26:39.318204+0000","last_active":"2023-05-22T15:28:07.681354+0000","last_peered":"2023-05-22T15:28:07.681354+0000","last_clean":"2023-05-22T15:28:07.681354+0000","last_became_active":"2023-05-22T15:26:39.315370+0000","last_became_peered":"2023-05-22T15:26:39.315370+0000","last_unstale":"2023-05-22T15:28:07.681354+0000","last_undegraded":"2023-05-22T15:28:07.681354+0000","last_fullsized":"2023-05-22T15:28:07.681354+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:38.020076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:38.020076+0000","last_clean_scrub_stamp":"2023-05-22T15:26:38.020076+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:06:39.207543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":7},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574870,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":13752,"kb_used_data":1008,"kb_used_omap":0,"kb_used_meta":12736,"kb_avail":94358088,"statfs":{"total":96636764160,"available":96622682112,"internally_reserved":0,"allocated":1032192,"data_stored":781293,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":13041664},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 15:27:38 2023","interfaces":[{"interface":"back","average":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":1.5860000000000001,"5min":1.5860000000000001,"15min":1.5860000000000001},"last":0.77700000000000002},{"interface":"front","average":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"min":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.4239999999999999,"5min":1.4239999999999999,"15min":1.4239999999999999},"last":0.55700000000000005}]},{"osd":1,"last update":"Mon May 22 15:27:38 2023","interfaces":[{"interface":"back","average":{"1min":0.96599999999999997,"5min":0.96599999999999997,"15min":0.96599999999999997},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":1.3600000000000001,"5min":1.3600000000000001,"15min":1.3600000000000001},"last":1.002},{"interface":"front","average":{"1min":1.002,"5min":1.002,"15min":1.002},"min":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"max":{"1min":1.8260000000000001,"5min":1.8260000000000001,"15min":1.8260000000000001},"last":1.208}]},{"osd":2,"last update":"Mon May 22 15:27:38 2023","interfaces":[{"interface":"back","average":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.127,"5min":1.127,"15min":1.127},"last":0.90100000000000002},{"interface":"front","average":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":1.23,"5min":1.23,"15min":1.23},"last":0.83999999999999997}]},{"osd":3,"last update":"Mon May 22 15:27:38 2023","interfaces":[{"interface":"back","average":{"1min":0.99099999999999999,"5min":0.99099999999999999,"15min":0.99099999999999999},"min":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"max":{"1min":1.7829999999999999,"5min":1.7829999999999999,"15min":1.7829999999999999},"last":1.115},{"interface":"front","average":{"1min":1.026,"5min":1.026,"15min":1.026},"min":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":1.6879999999999999,"5min":1.6879999999999999,"15min":1.6879999999999999},"last":0.70199999999999996}]},{"osd":4,"last update":"Mon May 22 15:27:38 2023","interfaces":[{"interface":"back","average":{"1min":0.92700000000000005,"5min":0.92700000000000005,"15min":0.92700000000000005},"min":{"1min":0.62,"5min":0.62,"15min":0.62},"max":{"1min":1.2430000000000001,"5min":1.2430000000000001,"15min":1.2430000000000001},"last":0.65600000000000003},{"interface":"front","average":{"1min":1.1120000000000001,"5min":1.1120000000000001,"15min":1.1120000000000001},"min":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"max":{"1min":1.73,"5min":1.73,"15min":1.73},"last":1.2889999999999999}]},{"osd":5,"last update":"Mon May 22 15:27:38 2023","interfaces":[{"interface":"back","average":{"1min":1.123,"5min":1.123,"15min":1.123},"min":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":1.3779999999999999,"5min":1.3779999999999999,"15min":1.3779999999999999},"last":1.0760000000000001},{"interface":"front","average":{"1min":1.139,"5min":1.139,"15min":1.139},"min":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"max":{"1min":1.6439999999999999,"5min":1.6439999999999999,"15min":1.6439999999999999},"last":1.0429999999999999}]},{"osd":6,"last update":"Mon May 22 15:27:38 2023","interfaces":[{"interface":"back","average":{"1min":1.1020000000000001,"5min":1.1020000000000001,"15min":1.1020000000000001},"min":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"max":{"1min":1.3999999999999999,"5min":1.3999999999999999,"15min":1.3999999999999999},"last":0.96399999999999997},{"interface":"front","average":{"1min":1.1890000000000001,"5min":1.1890000000000001,"15min":1.1890000000000001},"min":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"max":{"1min":2.04,"5min":2.04,"15min":2.04},"last":1.3620000000000001}]}]},{"osd":6,"up_from":13,"seq":55834574871,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8372,"kb_used_data":428,"kb_used_omap":0,"kb_used_meta":7936,"kb_avail":94363468,"statfs":{"total":96636764160,"available":96628191232,"internally_reserved":0,"allocated":438272,"data_stored":190925,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8126464},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 15:27:41 2023","interfaces":[{"interface":"back","average":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"min":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":1.1659999999999999,"5min":1.1659999999999999,"15min":1.1659999999999999},"last":0.97399999999999998},{"interface":"front","average":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"min":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"max":{"1min":1.385,"5min":1.385,"15min":1.385},"last":0.71499999999999997}]},{"osd":1,"last update":"Mon May 22 15:27:41 2023","interfaces":[{"interface":"back","average":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"min":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":1.2869999999999999,"5min":1.2869999999999999,"15min":1.2869999999999999},"last":0.77000000000000002},{"interface":"front","average":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":0.90000000000000002},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":1.204,"5min":1.204,"15min":1.204},"last":0.92000000000000004}]},{"osd":2,"last update":"Mon May 22 15:27:41 2023","interfaces":[{"interface":"back","average":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"min":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"max":{"1min":1.145,"5min":1.145,"15min":1.145},"last":0.84099999999999997},{"interface":"front","average":{"1min":1.0569999999999999,"5min":1.0569999999999999,"15min":1.0569999999999999},"min":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"max":{"1min":1.4670000000000001,"5min":1.4670000000000001,"15min":1.4670000000000001},"last":1.0700000000000001}]},{"osd":3,"last update":"Mon May 22 15:27:41 2023","interfaces":[{"interface":"back","average":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"min":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"max":{"1min":1.415,"5min":1.415,"15min":1.415},"last":1.0169999999999999},{"interface":"front","average":{"1min":0.95799999999999996,"5min":0.95799999999999996,"15min":0.95799999999999996},"min":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":1.486,"5min":1.486,"15min":1.486},"last":1.042}]},{"osd":4,"last update":"Mon May 22 15:27:41 2023","interfaces":[{"interface":"back","average":{"1min":1.0389999999999999,"5min":1.0389999999999999,"15min":1.0389999999999999},"min":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"max":{"1min":1.6459999999999999,"5min":1.6459999999999999,"15min":1.6459999999999999},"last":1.1539999999999999},{"interface":"front","average":{"1min":1.141,"5min":1.141,"15min":1.141},"min":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"max":{"1min":1.5469999999999999,"5min":1.5469999999999999,"15min":1.5469999999999999},"last":1.097}]},{"osd":5,"last update":"Mon May 22 15:27:41 2023","interfaces":[{"interface":"back","average":{"1min":1.075,"5min":1.075,"15min":1.075},"min":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"max":{"1min":1.621,"5min":1.621,"15min":1.621},"last":1.117},{"interface":"front","average":{"1min":1.149,"5min":1.149,"15min":1.149},"min":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"max":{"1min":1.7509999999999999,"5min":1.7509999999999999,"15min":1.7509999999999999},"last":1.232}]},{"osd":7,"last update":"Mon May 22 15:27:41 2023","interfaces":[{"interface":"back","average":{"1min":1.268,"5min":1.268,"15min":1.268},"min":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"max":{"1min":1.8120000000000001,"5min":1.8120000000000001,"15min":1.8120000000000001},"last":1.177},{"interface":"front","average":{"1min":1.179,"5min":1.179,"15min":1.179},"min":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"max":{"1min":1.6819999999999999,"5min":1.6819999999999999,"15min":1.6819999999999999},"last":1.3260000000000001}]}]},{"osd":5,"up_from":13,"seq":55834574871,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8500,"kb_used_data":428,"kb_used_omap":0,"kb_used_meta":8064,"kb_avail":94363340,"statfs":{"total":96636764160,"available":96628060160,"internally_reserved":0,"allocated":438272,"data_stored":190925,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8257536},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 15:27:37 2023","interfaces":[{"interface":"back","average":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"min":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":1.169,"5min":1.169,"15min":1.169},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"min":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.103,"5min":1.103,"15min":1.103},"last":0.68500000000000005}]},{"osd":1,"last update":"Mon May 22 15:27:37 2023","interfaces":[{"interface":"back","average":{"1min":0.90200000000000002,"5min":0.90200000000000002,"15min":0.90200000000000002},"min":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":1.2170000000000001,"5min":1.2170000000000001,"15min":1.2170000000000001},"last":1.0309999999999999},{"interface":"front","average":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"min":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":1.399,"5min":1.399,"15min":1.399},"last":1.0069999999999999}]},{"osd":2,"last update":"Mon May 22 15:27:37 2023","interfaces":[{"interface":"back","average":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"min":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":1.597,"5min":1.597,"15min":1.597},"last":0.83399999999999996},{"interface":"front","average":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"min":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.296,"5min":1.296,"15min":1.296},"last":0.79100000000000004}]},{"osd":3,"last update":"Mon May 22 15:27:37 2023","interfaces":[{"interface":"back","average":{"1min":0.93400000000000005,"5min":0.93400000000000005,"15min":0.93400000000000005},"min":{"1min":0.627,"5min":0.627,"15min":0.627},"max":{"1min":1.6839999999999999,"5min":1.6839999999999999,"15min":1.6839999999999999},"last":0.91000000000000003},{"interface":"front","average":{"1min":1.0209999999999999,"5min":1.0209999999999999,"15min":1.0209999999999999},"min":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"max":{"1min":1.5189999999999999,"5min":1.5189999999999999,"15min":1.5189999999999999},"last":1.111}]},{"osd":4,"last update":"Mon May 22 15:27:37 2023","interfaces":[{"interface":"back","average":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":0.96399999999999997},"min":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"max":{"1min":1.3140000000000001,"5min":1.3140000000000001,"15min":1.3140000000000001},"last":0.876},{"interface":"front","average":{"1min":1.0549999999999999,"5min":1.0549999999999999,"15min":1.0549999999999999},"min":{"1min":0.747,"5min":0.747,"15min":0.747},"max":{"1min":1.319,"5min":1.319,"15min":1.319},"last":1.0700000000000001}]},{"osd":6,"last update":"Mon May 22 15:27:37 2023","interfaces":[{"interface":"back","average":{"1min":1.1000000000000001,"5min":1.1000000000000001,"15min":1.1000000000000001},"min":{"1min":0.94199999999999995,"5min":0.94199999999999995,"15min":0.94199999999999995},"max":{"1min":1.357,"5min":1.357,"15min":1.357},"last":1.2190000000000001},{"interface":"front","average":{"1min":1.109,"5min":1.109,"15min":1.109},"min":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"max":{"1min":1.4350000000000001,"5min":1.4350000000000001,"15min":1.4350000000000001},"last":0.95899999999999996}]},{"osd":7,"last update":"Mon May 22 15:27:37 2023","interfaces":[{"interface":"back","average":{"1min":1.232,"5min":1.232,"15min":1.232},"min":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"max":{"1min":2.8260000000000001,"5min":2.8260000000000001,"15min":2.8260000000000001},"last":1.157},{"interface":"front","average":{"1min":1.1679999999999999,"5min":1.1679999999999999,"15min":1.1679999999999999},"min":{"1min":0.877,"5min":0.877,"15min":0.877},"max":{"1min":1.4490000000000001,"5min":1.4490000000000001,"15min":1.4490000000000001},"last":1.29}]}]},{"osd":4,"up_from":13,"seq":55834574869,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8564,"kb_used_data":428,"kb_used_omap":0,"kb_used_meta":8128,"kb_avail":94363276,"statfs":{"total":96636764160,"available":96627994624,"internally_reserved":0,"allocated":438272,"data_stored":190925,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8323072},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 15:27:37 2023","interfaces":[{"interface":"back","average":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"min":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":1.252,"5min":1.252,"15min":1.252},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"min":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"max":{"1min":1.2989999999999999,"5min":1.2989999999999999,"15min":1.2989999999999999},"last":0.56299999999999994}]},{"osd":1,"last update":"Mon May 22 15:27:37 2023","interfaces":[{"interface":"back","average":{"1min":0.872,"5min":0.872,"15min":0.872},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.3009999999999999,"5min":1.3009999999999999,"15min":1.3009999999999999},"last":1.1220000000000001},{"interface":"front","average":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":0.83299999999999996},"min":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":1.1499999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.71399999999999997}]},{"osd":2,"last update":"Mon May 22 15:27:37 2023","interfaces":[{"interface":"back","average":{"1min":0.96199999999999997,"5min":0.96199999999999997,"15min":0.96199999999999997},"min":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"max":{"1min":1.284,"5min":1.284,"15min":1.284},"last":0.78000000000000003},{"interface":"front","average":{"1min":1.0780000000000001,"5min":1.0780000000000001,"15min":1.0780000000000001},"min":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"max":{"1min":1.9770000000000001,"5min":1.9770000000000001,"15min":1.9770000000000001},"last":0.97499999999999998}]},{"osd":3,"last update":"Mon May 22 15:27:37 2023","interfaces":[{"interface":"back","average":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"min":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.4850000000000001,"5min":1.4850000000000001,"15min":1.4850000000000001},"last":0.81699999999999995},{"interface":"front","average":{"1min":1.1100000000000001,"5min":1.1100000000000001,"15min":1.1100000000000001},"min":{"1min":0.872,"5min":0.872,"15min":0.872},"max":{"1min":1.468,"5min":1.468,"15min":1.468},"last":0.86199999999999999}]},{"osd":5,"last update":"Mon May 22 15:27:37 2023","interfaces":[{"interface":"back","average":{"1min":1.123,"5min":1.123,"15min":1.123},"min":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"max":{"1min":1.7270000000000001,"5min":1.7270000000000001,"15min":1.7270000000000001},"last":1.042},{"interface":"front","average":{"1min":1.024,"5min":1.024,"15min":1.024},"min":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"max":{"1min":1.417,"5min":1.417,"15min":1.417},"last":1.0680000000000001}]},{"osd":6,"last update":"Mon May 22 15:27:37 2023","interfaces":[{"interface":"back","average":{"1min":1.181,"5min":1.181,"15min":1.181},"min":{"1min":0.96899999999999997,"5min":0.96899999999999997,"15min":0.96899999999999997},"max":{"1min":1.6020000000000001,"5min":1.6020000000000001,"15min":1.6020000000000001},"last":1.0089999999999999},{"interface":"front","average":{"1min":1.202,"5min":1.202,"15min":1.202},"min":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"max":{"1min":2.0699999999999998,"5min":2.0699999999999998,"15min":2.0699999999999998},"last":1.1890000000000001}]},{"osd":7,"last update":"Mon May 22 15:27:37 2023","interfaces":[{"interface":"back","average":{"1min":1.1919999999999999,"5min":1.1919999999999999,"15min":1.1919999999999999},"min":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.90800000000000003},"max":{"1min":1.6639999999999999,"5min":1.6639999999999999,"15min":1.6639999999999999},"last":0.90800000000000003},{"interface":"front","average":{"1min":1.2210000000000001,"5min":1.2210000000000001,"15min":1.2210000000000001},"min":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"max":{"1min":1.7410000000000001,"5min":1.7410000000000001,"15min":1.7410000000000001},"last":1.1559999999999999}]}]},{"osd":3,"up_from":13,"seq":55834574869,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8500,"kb_used_data":428,"kb_used_omap":0,"kb_used_meta":8064,"kb_avail":94363340,"statfs":{"total":96636764160,"available":96628060160,"internally_reserved":0,"allocated":438272,"data_stored":190925,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8257536},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 15:27:37 2023","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.42999999999999999}]},{"osd":1,"last update":"Mon May 22 15:27:37 2023","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.505},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":1.6579999999999999,"5min":1.6579999999999999,"15min":1.6579999999999999},"last":0.49099999999999999}]},{"osd":2,"last update":"Mon May 22 15:27:37 2023","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":1.2490000000000001,"5min":1.2490000000000001,"15min":1.2490000000000001},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":1.855,"5min":1.855,"15min":1.855},"last":0.53000000000000003}]},{"osd":4,"last update":"Mon May 22 15:27:37 2023","interfaces":[{"interface":"back","average":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"min":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":1.5820000000000001,"5min":1.5820000000000001,"15min":1.5820000000000001},"last":0.77000000000000002},{"interface":"front","average":{"1min":0.96899999999999997,"5min":0.96899999999999997,"15min":0.96899999999999997},"min":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"max":{"1min":1.8069999999999999,"5min":1.8069999999999999,"15min":1.8069999999999999},"last":0.66800000000000004}]},{"osd":5,"last update":"Mon May 22 15:27:37 2023","interfaces":[{"interface":"back","average":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":0.83799999999999997},"min":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":1.0840000000000001,"5min":1.0840000000000001,"15min":1.0840000000000001},"last":0.69499999999999995},{"interface":"front","average":{"1min":1.0249999999999999,"5min":1.0249999999999999,"15min":1.0249999999999999},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":2.4820000000000002,"5min":2.4820000000000002,"15min":2.4820000000000002},"last":0.78300000000000003}]},{"osd":6,"last update":"Mon May 22 15:27:37 2023","interfaces":[{"interface":"back","average":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":0.90300000000000002},"min":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"max":{"1min":1.452,"5min":1.452,"15min":1.452},"last":0.623},{"interface":"front","average":{"1min":1.085,"5min":1.085,"15min":1.085},"min":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"max":{"1min":2.6589999999999998,"5min":2.6589999999999998,"15min":2.6589999999999998},"last":0.88500000000000001}]},{"osd":7,"last update":"Mon May 22 15:27:37 2023","interfaces":[{"interface":"back","average":{"1min":1.1459999999999999,"5min":1.1459999999999999,"15min":1.1459999999999999},"min":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"max":{"1min":2.762,"5min":2.762,"15min":2.762},"last":0.73499999999999999},{"interface":"front","average":{"1min":1.0369999999999999,"5min":1.0369999999999999,"15min":1.0369999999999999},"min":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"max":{"1min":1.714,"5min":1.714,"15min":1.714},"last":0.64400000000000002}]}]},{"osd":2,"up_from":13,"seq":55834574870,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":12532,"kb_used_data":428,"kb_used_omap":0,"kb_used_meta":12096,"kb_avail":94359308,"statfs":{"total":96636764160,"available":96623931392,"internally_reserved":0,"allocated":438272,"data_stored":190925,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":12386304},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 15:27:36 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":1.4670000000000001,"5min":1.4670000000000001,"15min":1.4670000000000001},"last":0.55700000000000005}]},{"osd":1,"last update":"Mon May 22 15:27:36 2023","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":1.5309999999999999,"5min":1.5309999999999999,"15min":1.5309999999999999},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":1.2210000000000001,"5min":1.2210000000000001,"15min":1.2210000000000001},"last":0.46899999999999997}]},{"osd":3,"last update":"Mon May 22 15:27:36 2023","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":1.3340000000000001,"5min":1.3340000000000001,"15min":1.3340000000000001},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.754,"5min":0.754,"15min":0.754},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":1.6339999999999999,"5min":1.6339999999999999,"15min":1.6339999999999999},"last":0.79500000000000004}]},{"osd":4,"last update":"Mon May 22 15:27:36 2023","interfaces":[{"interface":"back","average":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"min":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"max":{"1min":1.774,"5min":1.774,"15min":1.774},"last":0.91000000000000003},{"interface":"front","average":{"1min":0.97599999999999998,"5min":0.97599999999999998,"15min":0.97599999999999998},"min":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"max":{"1min":1.802,"5min":1.802,"15min":1.802},"last":1.0600000000000001}]},{"osd":5,"last update":"Mon May 22 15:27:36 2023","interfaces":[{"interface":"back","average":{"1min":1.05,"5min":1.05,"15min":1.05},"min":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"max":{"1min":2.4569999999999999,"5min":2.4569999999999999,"15min":2.4569999999999999},"last":0.93500000000000005},{"interface":"front","average":{"1min":1,"5min":1,"15min":1},"min":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"max":{"1min":1.4670000000000001,"5min":1.4670000000000001,"15min":1.4670000000000001},"last":1.091}]},{"osd":6,"last update":"Mon May 22 15:27:36 2023","interfaces":[{"interface":"back","average":{"1min":1.093,"5min":1.093,"15min":1.093},"min":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":3.488,"5min":3.488,"15min":3.488},"last":0.95899999999999996},{"interface":"front","average":{"1min":1.151,"5min":1.151,"15min":1.151},"min":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"max":{"1min":3.7330000000000001,"5min":3.7330000000000001,"15min":3.7330000000000001},"last":1.0029999999999999}]},{"osd":7,"last update":"Mon May 22 15:27:36 2023","interfaces":[{"interface":"back","average":{"1min":1.085,"5min":1.085,"15min":1.085},"min":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"max":{"1min":2.21,"5min":2.21,"15min":2.21},"last":0.875},{"interface":"front","average":{"1min":1.0269999999999999,"5min":1.0269999999999999,"15min":1.0269999999999999},"min":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":1.6120000000000001,"5min":1.6120000000000001,"15min":1.6120000000000001},"last":1.0329999999999999}]}]},{"osd":1,"up_from":13,"seq":55834574870,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8500,"kb_used_data":428,"kb_used_omap":0,"kb_used_meta":8064,"kb_avail":94363340,"statfs":{"total":96636764160,"available":96628060160,"internally_reserved":0,"allocated":438272,"data_stored":190925,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8257536},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 15:27:36 2023","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":1.3089999999999999,"5min":1.3089999999999999,"15min":1.3089999999999999},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":1.3799999999999999,"5min":1.3799999999999999,"15min":1.3799999999999999},"last":0.95899999999999996}]},{"osd":2,"last update":"Mon May 22 15:27:36 2023","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.63,"15min":0.63},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.98299999999999998,"5min":0.98299999999999998,"15min":0.98299999999999998},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.35199999999999998}]},{"osd":3,"last update":"Mon May 22 15:27:36 2023","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":1.145,"5min":1.145,"15min":1.145},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.0760000000000001,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.49299999999999999}]},{"osd":4,"last update":"Mon May 22 15:27:36 2023","interfaces":[{"interface":"back","average":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":0.85299999999999998},"min":{"1min":0.622,"5min":0.622,"15min":0.622},"max":{"1min":1.2250000000000001,"5min":1.2250000000000001,"15min":1.2250000000000001},"last":0.86199999999999999},{"interface":"front","average":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"min":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"max":{"1min":1.272,"5min":1.272,"15min":1.272},"last":1.0109999999999999}]},{"osd":5,"last update":"Mon May 22 15:27:36 2023","interfaces":[{"interface":"back","average":{"1min":0.95999999999999996,"5min":0.95999999999999996,"15min":0.95999999999999996},"min":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"max":{"1min":1.1779999999999999,"5min":1.1779999999999999,"15min":1.1779999999999999},"last":0.68500000000000005},{"interface":"front","average":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":0.96299999999999997},"min":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"max":{"1min":1.488,"5min":1.488,"15min":1.488},"last":0.73299999999999998}]},{"osd":6,"last update":"Mon May 22 15:27:36 2023","interfaces":[{"interface":"back","average":{"1min":0.93400000000000005,"5min":0.93400000000000005,"15min":0.93400000000000005},"min":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"max":{"1min":1.28,"5min":1.28,"15min":1.28},"last":1.151},{"interface":"front","average":{"1min":1.1120000000000001,"5min":1.1120000000000001,"15min":1.1120000000000001},"min":{"1min":0.621,"5min":0.621,"15min":0.621},"max":{"1min":1.8180000000000001,"5min":1.8180000000000001,"15min":1.8180000000000001},"last":0.77400000000000002}]},{"osd":7,"last update":"Mon May 22 15:27:36 2023","interfaces":[{"interface":"back","average":{"1min":1.117,"5min":1.117,"15min":1.117},"min":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"max":{"1min":1.9079999999999999,"5min":1.9079999999999999,"15min":1.9079999999999999},"last":1.0880000000000001},{"interface":"front","average":{"1min":1.0349999999999999,"5min":1.0349999999999999,"15min":1.0349999999999999},"min":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"max":{"1min":1.5780000000000001,"5min":1.5780000000000001,"15min":1.5780000000000001},"last":0.69899999999999995}]}]},{"osd":0,"up_from":13,"seq":55834574871,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":13688,"kb_used_data":1008,"kb_used_omap":0,"kb_used_meta":12672,"kb_avail":94358152,"statfs":{"total":96636764160,"available":96622747648,"internally_reserved":0,"allocated":1032192,"data_stored":781293,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":12976128},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon May 22 15:27:37 2023","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":1.1839999999999999,"5min":1.1839999999999999,"15min":1.1839999999999999},"last":0.317},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":1.1040000000000001,"5min":1.1040000000000001,"15min":1.1040000000000001},"last":0.39800000000000002}]},{"osd":2,"last update":"Mon May 22 15:27:37 2023","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":1.23,"5min":1.23,"15min":1.23},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":1.1639999999999999,"5min":1.1639999999999999,"15min":1.1639999999999999},"last":0.47799999999999998}]},{"osd":3,"last update":"Mon May 22 15:27:37 2023","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":1.2410000000000001,"5min":1.2410000000000001,"15min":1.2410000000000001},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":1.119,"5min":1.119,"15min":1.119},"last":0.45000000000000001}]},{"osd":4,"last update":"Mon May 22 15:27:37 2023","interfaces":[{"interface":"back","average":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"min":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":1.2969999999999999,"5min":1.2969999999999999,"15min":1.2969999999999999},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"min":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"max":{"1min":1.4219999999999999,"5min":1.4219999999999999,"15min":1.4219999999999999},"last":0.79900000000000004}]},{"osd":5,"last update":"Mon May 22 15:27:37 2023","interfaces":[{"interface":"back","average":{"1min":0.999,"5min":0.999,"15min":0.999},"min":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":1.4370000000000001,"5min":1.4370000000000001,"15min":1.4370000000000001},"last":0.81799999999999995},{"interface":"front","average":{"1min":0.95099999999999996,"5min":0.95099999999999996,"15min":0.95099999999999996},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.349,"5min":1.349,"15min":1.349},"last":0.68200000000000005}]},{"osd":6,"last update":"Mon May 22 15:27:37 2023","interfaces":[{"interface":"back","average":{"1min":0.94599999999999995,"5min":0.94599999999999995,"15min":0.94599999999999995},"min":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"max":{"1min":1.2769999999999999,"5min":1.2769999999999999,"15min":1.2769999999999999},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.96899999999999997,"5min":0.96899999999999997,"15min":0.96899999999999997},"min":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":1.492,"5min":1.492,"15min":1.492},"last":0.627}]},{"osd":7,"last update":"Mon May 22 15:27:37 2023","interfaces":[{"interface":"back","average":{"1min":1.04,"5min":1.04,"15min":1.04},"min":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":1.581,"5min":1.581,"15min":1.581},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.98099999999999998,"5min":0.98099999999999998,"15min":0.98099999999999998},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.448,"5min":1.448,"15min":1.448},"last":0.84599999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-05-22T15:28:20.387 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-05-22T15:28:20.390 DEBUG:teuthology.orchestra.run.smithi062:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-05-22T15:28:20.878 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-05-22T15:28:21.015 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.007+0000 7f3c433b8640 1 Processor -- start 2023-05-22T15:28:21.016 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.007+0000 7f3c433b8640 1 -- start start 2023-05-22T15:28:21.016 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.011+0000 7f3c433b8640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f3c3c0ca4e0 0x7f3c3c15e460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:21.017 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.011+0000 7f3c433b8640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3c3c070df0 0x7f3c3c071210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:21.017 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.011+0000 7f3c433b8640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c3c0b0650 con 0x7f3c3c0c9b40 2023-05-22T15:28:21.017 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.011+0000 7f3c433b8640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c3c0b0370 con 0x7f3c3c0ca4e0 2023-05-22T15:28:21.017 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.011+0000 7f3c433b8640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c3c15d980 con 0x7f3c3c070df0 2023-05-22T15:28:21.018 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.011+0000 7f3c41154640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f3c3c0ca4e0 0x7f3c3c15e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:21.018 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.011+0000 7f3c41955640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3c3c070df0 0x7f3c3c071210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:21.018 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.011+0000 7f3c41154640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f3c3c0ca4e0 0x7f3c3c15e460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.16:50714/0 (socket says 172.21.15.16:50714) 2023-05-22T15:28:21.018 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.011+0000 7f3c41154640 1 -- 172.21.15.16:0/1145765321 learned_addr learned my addr 172.21.15.16:0/1145765321 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:28:21.018 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.011+0000 7f3c41955640 1 -- 172.21.15.16:0/1145765321 >> v2:172.21.15.62:3300/0 conn(0x7f3c3c0ca4e0 msgr2=0x7f3c3c15e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:28:21.019 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.011+0000 7f3c41955640 1 --2- 172.21.15.16:0/1145765321 >> v2:172.21.15.62:3300/0 conn(0x7f3c3c0ca4e0 0x7f3c3c15e460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:21.019 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.011+0000 7f3c41955640 1 -- 172.21.15.16:0/1145765321 >> v1:172.21.15.62:6789/0 conn(0x7f3c3c0c9b40 legacy=0x7f3c3c0afb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:21.019 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.011+0000 7f3c41955640 1 -- 172.21.15.16:0/1145765321 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c3c15dc60 con 0x7f3c3c070df0 2023-05-22T15:28:21.019 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.011+0000 7f3c41955640 1 --2- 172.21.15.16:0/1145765321 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3c3c070df0 0x7f3c3c071210 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f3c34009d90 tx=0x7f3c3401c920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b3460f47ac14c63f server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:28:21.021 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.011+0000 7f3c33fff640 1 -- 172.21.15.16:0/1145765321 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c340a5940 con 0x7f3c3c070df0 2023-05-22T15:28:21.021 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.011+0000 7f3c33fff640 1 -- 172.21.15.16:0/1145765321 <== mon.2 v2:172.21.15.62:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c340a5aa0 con 0x7f3c3c070df0 2023-05-22T15:28:21.021 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.011+0000 7f3c33fff640 1 -- 172.21.15.16:0/1145765321 <== mon.2 v2:172.21.15.62:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c340a5e30 con 0x7f3c3c070df0 2023-05-22T15:28:21.021 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.015+0000 7f3c433b8640 1 -- 172.21.15.16:0/1145765321 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3c3c070df0 msgr2=0x7f3c3c071210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:21.021 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.015+0000 7f3c433b8640 1 --2- 172.21.15.16:0/1145765321 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3c3c070df0 0x7f3c3c071210 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f3c34009d90 tx=0x7f3c3401c920 comp rx=0 tx=0).stop 2023-05-22T15:28:21.021 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.015+0000 7f3c433b8640 1 -- 172.21.15.16:0/1145765321 shutdown_connections 2023-05-22T15:28:21.022 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.015+0000 7f3c433b8640 1 --2- 172.21.15.16:0/1145765321 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3c3c070df0 0x7f3c3c071210 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:21.022 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.015+0000 7f3c433b8640 1 --2- 172.21.15.16:0/1145765321 >> v2:172.21.15.62:3300/0 conn(0x7f3c3c0ca4e0 0x7f3c3c15e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:21.022 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.015+0000 7f3c433b8640 1 -- 172.21.15.16:0/1145765321 >> 172.21.15.16:0/1145765321 conn(0x7f3c3c069720 msgr2=0x7f3c3c069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:28:21.022 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.015+0000 7f3c433b8640 1 -- 172.21.15.16:0/1145765321 shutdown_connections 2023-05-22T15:28:21.022 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.015+0000 7f3c433b8640 1 -- 172.21.15.16:0/1145765321 wait complete. 2023-05-22T15:28:21.023 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.015+0000 7f3c433b8640 1 Processor -- start 2023-05-22T15:28:21.024 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.015+0000 7f3c433b8640 1 -- start start 2023-05-22T15:28:21.024 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.015+0000 7f3c433b8640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3c3c070df0 0x7f3c3c159500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:21.024 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.015+0000 7f3c433b8640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3c3c0c9b40 0x7f3c3c15a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:21.024 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.015+0000 7f3c433b8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3c3c0ca4e0 0x7f3c3c14a330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:21.024 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.015+0000 7f3c433b8640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c3c156f60 con 0x7f3c3c0ca4e0 2023-05-22T15:28:21.024 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.015+0000 7f3c433b8640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c3c14a870 con 0x7f3c3c070df0 2023-05-22T15:28:21.025 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.015+0000 7f3c433b8640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c3c14bdc0 con 0x7f3c3c0c9b40 2023-05-22T15:28:21.025 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.015+0000 7f3c41955640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3c3c0ca4e0 0x7f3c3c14a330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:21.025 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.015+0000 7f3c41955640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3c3c0ca4e0 0x7f3c3c14a330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51994/0 (socket says 172.21.15.16:51994) 2023-05-22T15:28:21.025 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.015+0000 7f3c41955640 1 -- 172.21.15.16:0/2052191560 learned_addr learned my addr 172.21.15.16:0/2052191560 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:28:21.025 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.015+0000 7f3c40953640 1 --2- 172.21.15.16:0/2052191560 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3c3c0c9b40 0x7f3c3c15a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:21.025 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.015+0000 7f3c41154640 1 --2- 172.21.15.16:0/2052191560 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3c3c070df0 0x7f3c3c159500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:21.025 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.015+0000 7f3c41955640 1 -- 172.21.15.16:0/2052191560 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3c3c0c9b40 msgr2=0x7f3c3c15a2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:21.026 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.015+0000 7f3c41955640 1 --2- 172.21.15.16:0/2052191560 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3c3c0c9b40 0x7f3c3c15a2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:21.026 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.015+0000 7f3c41955640 1 -- 172.21.15.16:0/2052191560 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3c3c070df0 msgr2=0x7f3c3c159500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:21.026 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.015+0000 7f3c41955640 1 --2- 172.21.15.16:0/2052191560 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3c3c070df0 0x7f3c3c159500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:21.026 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.015+0000 7f3c41955640 1 -- 172.21.15.16:0/2052191560 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c34009a70 con 0x7f3c3c0ca4e0 2023-05-22T15:28:21.026 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.015+0000 7f3c41955640 1 --2- 172.21.15.16:0/2052191560 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3c3c0ca4e0 0x7f3c3c14a330 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f3c3401afd0 tx=0x7f3c340196c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:28:21.027 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.019+0000 7f3c31ffb640 1 -- 172.21.15.16:0/2052191560 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c340aedc0 con 0x7f3c3c0ca4e0 2023-05-22T15:28:21.027 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.019+0000 7f3c31ffb640 1 -- 172.21.15.16:0/2052191560 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c34009450 con 0x7f3c3c0ca4e0 2023-05-22T15:28:21.027 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.019+0000 7f3c31ffb640 1 -- 172.21.15.16:0/2052191560 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c340b7650 con 0x7f3c3c0ca4e0 2023-05-22T15:28:21.027 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.019+0000 7f3c433b8640 1 -- 172.21.15.16:0/2052191560 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c3c0c8e80 con 0x7f3c3c0ca4e0 2023-05-22T15:28:21.028 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.019+0000 7f3c433b8640 1 -- 172.21.15.16:0/2052191560 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c3c0ca8b0 con 0x7f3c3c0ca4e0 2023-05-22T15:28:21.028 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.019+0000 7f3c433b8640 1 -- 172.21.15.16:0/2052191560 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c3c0c9010 con 0x7f3c3c0ca4e0 2023-05-22T15:28:21.028 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.019+0000 7f3c31ffb640 1 -- 172.21.15.16:0/2052191560 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7f3c340b4050 con 0x7f3c3c0ca4e0 2023-05-22T15:28:21.029 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.019+0000 7f3c31ffb640 1 --2- 172.21.15.16:0/2052191560 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f3c20065380 0x7f3c20067830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:21.029 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.019+0000 7f3c31ffb640 1 -- 172.21.15.16:0/2052191560 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f3c3411dce0 con 0x7f3c3c0ca4e0 2023-05-22T15:28:21.029 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.019+0000 7f3c41154640 1 --2- 172.21.15.16:0/2052191560 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f3c20065380 0x7f3c20067830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:21.029 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.023+0000 7f3c41154640 1 --2- 172.21.15.16:0/2052191560 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f3c20065380 0x7f3c20067830 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f3c3c0ad620 tx=0x7f3c38008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:28:21.032 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.023+0000 7f3c31ffb640 1 -- 172.21.15.16:0/2052191560 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f3c340eb810 con 0x7f3c3c0ca4e0 2023-05-22T15:28:21.182 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.175+0000 7f3c433b8640 1 -- 172.21.15.16:0/2052191560 --> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f3c3c14c270 con 0x7f3c20065380 2023-05-22T15:28:21.195 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.187+0000 7f3c31ffb640 1 -- 172.21.15.16:0/2052191560 <== mgr.4101 v2:172.21.15.16:6832/36921 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+310185 (secure 0 0 0) 0x7f3c3c14c270 con 0x7f3c20065380 2023-05-22T15:28:21.199 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:28:21.201 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2023-05-22T15:28:21.202 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.191+0000 7f3c433b8640 1 -- 172.21.15.16:0/2052191560 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f3c20065380 msgr2=0x7f3c20067830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:21.202 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.191+0000 7f3c433b8640 1 --2- 172.21.15.16:0/2052191560 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f3c20065380 0x7f3c20067830 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f3c3c0ad620 tx=0x7f3c38008040 comp rx=0 tx=0).stop 2023-05-22T15:28:21.203 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.191+0000 7f3c433b8640 1 -- 172.21.15.16:0/2052191560 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3c3c0ca4e0 msgr2=0x7f3c3c14a330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:21.203 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.191+0000 7f3c433b8640 1 --2- 172.21.15.16:0/2052191560 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3c3c0ca4e0 0x7f3c3c14a330 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f3c3401afd0 tx=0x7f3c340196c0 comp rx=0 tx=0).stop 2023-05-22T15:28:21.203 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.191+0000 7f3c433b8640 1 -- 172.21.15.16:0/2052191560 shutdown_connections 2023-05-22T15:28:21.203 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.191+0000 7f3c433b8640 1 --2- 172.21.15.16:0/2052191560 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3c3c0ca4e0 0x7f3c3c14a330 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:21.203 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.191+0000 7f3c433b8640 1 --2- 172.21.15.16:0/2052191560 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f3c20065380 0x7f3c20067830 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:21.203 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.191+0000 7f3c433b8640 1 --2- 172.21.15.16:0/2052191560 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3c3c0c9b40 0x7f3c3c15a2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:21.203 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.191+0000 7f3c433b8640 1 --2- 172.21.15.16:0/2052191560 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3c3c070df0 0x7f3c3c159500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:21.204 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.191+0000 7f3c433b8640 1 -- 172.21.15.16:0/2052191560 >> 172.21.15.16:0/2052191560 conn(0x7f3c3c069720 msgr2=0x7f3c3c0ae460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:28:21.204 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.191+0000 7f3c433b8640 1 -- 172.21.15.16:0/2052191560 shutdown_connections 2023-05-22T15:28:21.204 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.191+0000 7f3c433b8640 1 -- 172.21.15.16:0/2052191560 wait complete. 2023-05-22T15:28:21.218 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":87,"stamp":"2023-05-22T15:28:19.973707+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592698,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":210,"acting":210,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":195,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":84072,"kb_used_data":5032,"kb_used_omap":0,"kb_used_meta":78976,"kb_avail":754890648,"statfs":{"total":773094113280,"available":773008023552,"internally_reserved":0,"allocated":5152768,"data_stored":2935604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":80871424},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":64,"apply_latency_ms":64,"commit_latency_ns":64000000,"apply_latency_ns":64000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2330,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.035853"},"pg_stats":[{"pgid":"7.3f","version":"0'0","reported_seq":10,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-05-22T15:28:15.388858+0000","last_change":"2023-05-22T15:28:13.374810+0000","last_active":"2023-05-22T15:28:15.388858+0000","last_peered":"2023-05-22T15:28:15.388858+0000","last_clean":"2023-05-22T15:28:15.388858+0000","last_became_active":"2023-05-22T15:28:13.373987+0000","last_became_peered":"2023-05-22T15:28:13.373987+0000","last_unstale":"2023-05-22T15:28:15.388858+0000","last_undegraded":"2023-05-22T15:28:15.388858+0000","last_fullsized":"2023-05-22T15:28:15.388858+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:53:53.090998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3e","version":"0'0","reported_seq":10,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-05-22T15:28:14.344342+0000","last_change":"2023-05-22T15:28:13.359524+0000","last_active":"2023-05-22T15:28:14.344342+0000","last_peered":"2023-05-22T15:28:14.344342+0000","last_clean":"2023-05-22T15:28:14.344342+0000","last_became_active":"2023-05-22T15:28:13.356167+0000","last_became_peered":"2023-05-22T15:28:13.356167+0000","last_unstale":"2023-05-22T15:28:14.344342+0000","last_undegraded":"2023-05-22T15:28:14.344342+0000","last_fullsized":"2023-05-22T15:28:14.344342+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:13:07.018500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3d","version":"0'0","reported_seq":10,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-05-22T15:28:14.683876+0000","last_change":"2023-05-22T15:28:13.373409+0000","last_active":"2023-05-22T15:28:14.683876+0000","last_peered":"2023-05-22T15:28:14.683876+0000","last_clean":"2023-05-22T15:28:14.683876+0000","last_became_active":"2023-05-22T15:28:13.372577+0000","last_became_peered":"2023-05-22T15:28:13.372577+0000","last_unstale":"2023-05-22T15:28:14.683876+0000","last_undegraded":"2023-05-22T15:28:14.683876+0000","last_fullsized":"2023-05-22T15:28:14.683876+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T15:50:36.712116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3c","version":"0'0","reported_seq":10,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-05-22T15:28:15.387458+0000","last_change":"2023-05-22T15:28:13.382458+0000","last_active":"2023-05-22T15:28:15.387458+0000","last_peered":"2023-05-22T15:28:15.387458+0000","last_clean":"2023-05-22T15:28:15.387458+0000","last_became_active":"2023-05-22T15:28:13.382152+0000","last_became_peered":"2023-05-22T15:28:13.382152+0000","last_unstale":"2023-05-22T15:28:15.387458+0000","last_undegraded":"2023-05-22T15:28:15.387458+0000","last_fullsized":"2023-05-22T15:28:15.387458+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T18:56:04.263372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3b","version":"0'0","reported_seq":10,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-05-22T15:28:14.343881+0000","last_change":"2023-05-22T15:28:13.357339+0000","last_active":"2023-05-22T15:28:14.343881+0000","last_peered":"2023-05-22T15:28:14.343881+0000","last_clean":"2023-05-22T15:28:14.343881+0000","last_became_active":"2023-05-22T15:28:13.356575+0000","last_became_peered":"2023-05-22T15:28:13.356575+0000","last_unstale":"2023-05-22T15:28:14.343881+0000","last_undegraded":"2023-05-22T15:28:14.343881+0000","last_fullsized":"2023-05-22T15:28:14.343881+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:14:04.358855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3a","version":"0'0","reported_seq":10,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-05-22T15:28:14.341583+0000","last_change":"2023-05-22T15:28:13.360726+0000","last_active":"2023-05-22T15:28:14.341583+0000","last_peered":"2023-05-22T15:28:14.341583+0000","last_clean":"2023-05-22T15:28:14.341583+0000","last_became_active":"2023-05-22T15:28:13.360451+0000","last_became_peered":"2023-05-22T15:28:13.360451+0000","last_unstale":"2023-05-22T15:28:14.341583+0000","last_undegraded":"2023-05-22T15:28:14.341583+0000","last_fullsized":"2023-05-22T15:28:14.341583+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:16:53.287861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.39","version":"0'0","reported_seq":10,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-05-22T15:28:15.388299+0000","last_change":"2023-05-22T15:28:13.351945+0000","last_active":"2023-05-22T15:28:15.388299+0000","last_peered":"2023-05-22T15:28:15.388299+0000","last_clean":"2023-05-22T15:28:15.388299+0000","last_became_active":"2023-05-22T15:28:13.351249+0000","last_became_peered":"2023-05-22T15:28:13.351249+0000","last_unstale":"2023-05-22T15:28:15.388299+0000","last_undegraded":"2023-05-22T15:28:15.388299+0000","last_fullsized":"2023-05-22T15:28:15.388299+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:43:14.133670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.38","version":"0'0","reported_seq":10,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-05-22T15:28:15.387058+0000","last_change":"2023-05-22T15:28:13.383717+0000","last_active":"2023-05-22T15:28:15.387058+0000","last_peered":"2023-05-22T15:28:15.387058+0000","last_clean":"2023-05-22T15:28:15.387058+0000","last_became_active":"2023-05-22T15:28:13.383305+0000","last_became_peered":"2023-05-22T15:28:13.383305+0000","last_unstale":"2023-05-22T15:28:15.387058+0000","last_undegraded":"2023-05-22T15:28:15.387058+0000","last_fullsized":"2023-05-22T15:28:15.387058+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:30:20.535065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.37","version":"0'0","reported_seq":10,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-05-22T15:28:15.387888+0000","last_change":"2023-05-22T15:28:13.347099+0000","last_active":"2023-05-22T15:28:15.387888+0000","last_peered":"2023-05-22T15:28:15.387888+0000","last_clean":"2023-05-22T15:28:15.387888+0000","last_became_active":"2023-05-22T15:28:13.345804+0000","last_became_peered":"2023-05-22T15:28:13.345804+0000","last_unstale":"2023-05-22T15:28:15.387888+0000","last_undegraded":"2023-05-22T15:28:15.387888+0000","last_fullsized":"2023-05-22T15:28:15.387888+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:57:05.742291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.36","version":"0'0","reported_seq":10,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-05-22T15:28:15.387873+0000","last_change":"2023-05-22T15:28:13.381834+0000","last_active":"2023-05-22T15:28:15.387873+0000","last_peered":"2023-05-22T15:28:15.387873+0000","last_clean":"2023-05-22T15:28:15.387873+0000","last_became_active":"2023-05-22T15:28:13.381521+0000","last_became_peered":"2023-05-22T15:28:13.381521+0000","last_unstale":"2023-05-22T15:28:15.387873+0000","last_undegraded":"2023-05-22T15:28:15.387873+0000","last_fullsized":"2023-05-22T15:28:15.387873+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:45:29.798248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.35","version":"0'0","reported_seq":10,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-05-22T15:28:15.387746+0000","last_change":"2023-05-22T15:28:13.346421+0000","last_active":"2023-05-22T15:28:15.387746+0000","last_peered":"2023-05-22T15:28:15.387746+0000","last_clean":"2023-05-22T15:28:15.387746+0000","last_became_active":"2023-05-22T15:28:13.346158+0000","last_became_peered":"2023-05-22T15:28:13.346158+0000","last_unstale":"2023-05-22T15:28:15.387746+0000","last_undegraded":"2023-05-22T15:28:15.387746+0000","last_fullsized":"2023-05-22T15:28:15.387746+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:17:34.839169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.34","version":"0'0","reported_seq":10,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-05-22T15:28:14.344230+0000","last_change":"2023-05-22T15:28:13.369182+0000","last_active":"2023-05-22T15:28:14.344230+0000","last_peered":"2023-05-22T15:28:14.344230+0000","last_clean":"2023-05-22T15:28:14.344230+0000","last_became_active":"2023-05-22T15:28:13.367856+0000","last_became_peered":"2023-05-22T15:28:13.367856+0000","last_unstale":"2023-05-22T15:28:14.344230+0000","last_undegraded":"2023-05-22T15:28:14.344230+0000","last_fullsized":"2023-05-22T15:28:14.344230+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:57:10.157295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.33","version":"0'0","reported_seq":10,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-05-22T15:28:14.339128+0000","last_change":"2023-05-22T15:28:13.341080+0000","last_active":"2023-05-22T15:28:14.339128+0000","last_peered":"2023-05-22T15:28:14.339128+0000","last_clean":"2023-05-22T15:28:14.339128+0000","last_became_active":"2023-05-22T15:28:13.340755+0000","last_became_peered":"2023-05-22T15:28:13.340755+0000","last_unstale":"2023-05-22T15:28:14.339128+0000","last_undegraded":"2023-05-22T15:28:14.339128+0000","last_fullsized":"2023-05-22T15:28:14.339128+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:45:44.276721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.32","version":"0'0","reported_seq":10,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-05-22T15:28:15.387401+0000","last_change":"2023-05-22T15:28:13.346913+0000","last_active":"2023-05-22T15:28:15.387401+0000","last_peered":"2023-05-22T15:28:15.387401+0000","last_clean":"2023-05-22T15:28:15.387401+0000","last_became_active":"2023-05-22T15:28:13.346676+0000","last_became_peered":"2023-05-22T15:28:13.346676+0000","last_unstale":"2023-05-22T15:28:15.387401+0000","last_undegraded":"2023-05-22T15:28:15.387401+0000","last_fullsized":"2023-05-22T15:28:15.387401+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T03:27:51.367539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.31","version":"0'0","reported_seq":10,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-05-22T15:28:14.341380+0000","last_change":"2023-05-22T15:28:13.373666+0000","last_active":"2023-05-22T15:28:14.341380+0000","last_peered":"2023-05-22T15:28:14.341380+0000","last_clean":"2023-05-22T15:28:14.341380+0000","last_became_active":"2023-05-22T15:28:13.373437+0000","last_became_peered":"2023-05-22T15:28:13.373437+0000","last_unstale":"2023-05-22T15:28:14.341380+0000","last_undegraded":"2023-05-22T15:28:14.341380+0000","last_fullsized":"2023-05-22T15:28:14.341380+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:17:59.314767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.30","version":"0'0","reported_seq":10,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-05-22T15:28:15.003421+0000","last_change":"2023-05-22T15:28:13.383031+0000","last_active":"2023-05-22T15:28:15.003421+0000","last_peered":"2023-05-22T15:28:15.003421+0000","last_clean":"2023-05-22T15:28:15.003421+0000","last_became_active":"2023-05-22T15:28:13.382005+0000","last_became_peered":"2023-05-22T15:28:13.382005+0000","last_unstale":"2023-05-22T15:28:15.003421+0000","last_undegraded":"2023-05-22T15:28:15.003421+0000","last_fullsized":"2023-05-22T15:28:15.003421+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:24:56.552189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2f","version":"0'0","reported_seq":10,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-05-22T15:28:14.684177+0000","last_change":"2023-05-22T15:28:13.390608+0000","last_active":"2023-05-22T15:28:14.684177+0000","last_peered":"2023-05-22T15:28:14.684177+0000","last_clean":"2023-05-22T15:28:14.684177+0000","last_became_active":"2023-05-22T15:28:13.389876+0000","last_became_peered":"2023-05-22T15:28:13.389876+0000","last_unstale":"2023-05-22T15:28:14.684177+0000","last_undegraded":"2023-05-22T15:28:14.684177+0000","last_fullsized":"2023-05-22T15:28:14.684177+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:45:29.798248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2e","version":"0'0","reported_seq":10,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-05-22T15:28:14.683979+0000","last_change":"2023-05-22T15:28:13.377162+0000","last_active":"2023-05-22T15:28:14.683979+0000","last_peered":"2023-05-22T15:28:14.683979+0000","last_clean":"2023-05-22T15:28:14.683979+0000","last_became_active":"2023-05-22T15:28:13.376192+0000","last_became_peered":"2023-05-22T15:28:13.376192+0000","last_unstale":"2023-05-22T15:28:14.683979+0000","last_undegraded":"2023-05-22T15:28:14.683979+0000","last_fullsized":"2023-05-22T15:28:14.683979+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:19:29.076699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2d","version":"0'0","reported_seq":10,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-05-22T15:28:15.004641+0000","last_change":"2023-05-22T15:28:13.390432+0000","last_active":"2023-05-22T15:28:15.004641+0000","last_peered":"2023-05-22T15:28:15.004641+0000","last_clean":"2023-05-22T15:28:15.004641+0000","last_became_active":"2023-05-22T15:28:13.386708+0000","last_became_peered":"2023-05-22T15:28:13.386708+0000","last_unstale":"2023-05-22T15:28:15.004641+0000","last_undegraded":"2023-05-22T15:28:15.004641+0000","last_fullsized":"2023-05-22T15:28:15.004641+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:09:14.075735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2c","version":"0'0","reported_seq":10,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-05-22T15:28:14.339262+0000","last_change":"2023-05-22T15:28:13.347980+0000","last_active":"2023-05-22T15:28:14.339262+0000","last_peered":"2023-05-22T15:28:14.339262+0000","last_clean":"2023-05-22T15:28:14.339262+0000","last_became_active":"2023-05-22T15:28:13.347730+0000","last_became_peered":"2023-05-22T15:28:13.347730+0000","last_unstale":"2023-05-22T15:28:14.339262+0000","last_undegraded":"2023-05-22T15:28:14.339262+0000","last_fullsized":"2023-05-22T15:28:14.339262+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:42:41.897151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.2b","version":"0'0","reported_seq":10,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-05-22T15:28:14.341484+0000","last_change":"2023-05-22T15:28:13.386959+0000","last_active":"2023-05-22T15:28:14.341484+0000","last_peered":"2023-05-22T15:28:14.341484+0000","last_clean":"2023-05-22T15:28:14.341484+0000","last_became_active":"2023-05-22T15:28:13.386615+0000","last_became_peered":"2023-05-22T15:28:13.386615+0000","last_unstale":"2023-05-22T15:28:14.341484+0000","last_undegraded":"2023-05-22T15:28:14.341484+0000","last_fullsized":"2023-05-22T15:28:14.341484+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:19:29.076699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2a","version":"0'0","reported_seq":10,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-05-22T15:28:15.387310+0000","last_change":"2023-05-22T15:28:13.383647+0000","last_active":"2023-05-22T15:28:15.387310+0000","last_peered":"2023-05-22T15:28:15.387310+0000","last_clean":"2023-05-22T15:28:15.387310+0000","last_became_active":"2023-05-22T15:28:13.383360+0000","last_became_peered":"2023-05-22T15:28:13.383360+0000","last_unstale":"2023-05-22T15:28:15.387310+0000","last_undegraded":"2023-05-22T15:28:15.387310+0000","last_fullsized":"2023-05-22T15:28:15.387310+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T15:50:36.712116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.29","version":"0'0","reported_seq":10,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-05-22T15:28:15.003414+0000","last_change":"2023-05-22T15:28:13.381448+0000","last_active":"2023-05-22T15:28:15.003414+0000","last_peered":"2023-05-22T15:28:15.003414+0000","last_clean":"2023-05-22T15:28:15.003414+0000","last_became_active":"2023-05-22T15:28:13.380808+0000","last_became_peered":"2023-05-22T15:28:13.380808+0000","last_unstale":"2023-05-22T15:28:15.003414+0000","last_undegraded":"2023-05-22T15:28:15.003414+0000","last_fullsized":"2023-05-22T15:28:15.003414+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:10:25.160562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.28","version":"0'0","reported_seq":10,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-05-22T15:28:15.003690+0000","last_change":"2023-05-22T15:28:13.385531+0000","last_active":"2023-05-22T15:28:15.003690+0000","last_peered":"2023-05-22T15:28:15.003690+0000","last_clean":"2023-05-22T15:28:15.003690+0000","last_became_active":"2023-05-22T15:28:13.383213+0000","last_became_peered":"2023-05-22T15:28:13.383213+0000","last_unstale":"2023-05-22T15:28:15.003690+0000","last_undegraded":"2023-05-22T15:28:15.003690+0000","last_fullsized":"2023-05-22T15:28:15.003690+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T15:38:51.888378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.27","version":"0'0","reported_seq":10,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-05-22T15:28:15.004368+0000","last_change":"2023-05-22T15:28:13.391496+0000","last_active":"2023-05-22T15:28:15.004368+0000","last_peered":"2023-05-22T15:28:15.004368+0000","last_clean":"2023-05-22T15:28:15.004368+0000","last_became_active":"2023-05-22T15:28:13.387059+0000","last_became_peered":"2023-05-22T15:28:13.387059+0000","last_unstale":"2023-05-22T15:28:15.004368+0000","last_undegraded":"2023-05-22T15:28:15.004368+0000","last_fullsized":"2023-05-22T15:28:15.004368+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:15:51.710774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.26","version":"0'0","reported_seq":10,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-05-22T15:28:14.339451+0000","last_change":"2023-05-22T15:28:13.341013+0000","last_active":"2023-05-22T15:28:14.339451+0000","last_peered":"2023-05-22T15:28:14.339451+0000","last_clean":"2023-05-22T15:28:14.339451+0000","last_became_active":"2023-05-22T15:28:13.340743+0000","last_became_peered":"2023-05-22T15:28:13.340743+0000","last_unstale":"2023-05-22T15:28:14.339451+0000","last_undegraded":"2023-05-22T15:28:14.339451+0000","last_fullsized":"2023-05-22T15:28:14.339451+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:30:48.719910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.25","version":"0'0","reported_seq":10,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-05-22T15:28:14.684020+0000","last_change":"2023-05-22T15:28:13.390256+0000","last_active":"2023-05-22T15:28:14.684020+0000","last_peered":"2023-05-22T15:28:14.684020+0000","last_clean":"2023-05-22T15:28:14.684020+0000","last_became_active":"2023-05-22T15:28:13.389776+0000","last_became_peered":"2023-05-22T15:28:13.389776+0000","last_unstale":"2023-05-22T15:28:14.684020+0000","last_undegraded":"2023-05-22T15:28:14.684020+0000","last_fullsized":"2023-05-22T15:28:14.684020+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:30:20.535065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.24","version":"0'0","reported_seq":10,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-05-22T15:28:14.344014+0000","last_change":"2023-05-22T15:28:13.368654+0000","last_active":"2023-05-22T15:28:14.344014+0000","last_peered":"2023-05-22T15:28:14.344014+0000","last_clean":"2023-05-22T15:28:14.344014+0000","last_became_active":"2023-05-22T15:28:13.367042+0000","last_became_peered":"2023-05-22T15:28:13.367042+0000","last_unstale":"2023-05-22T15:28:14.344014+0000","last_undegraded":"2023-05-22T15:28:14.344014+0000","last_fullsized":"2023-05-22T15:28:14.344014+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:17:34.839169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.23","version":"0'0","reported_seq":10,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-05-22T15:28:15.003917+0000","last_change":"2023-05-22T15:28:13.381490+0000","last_active":"2023-05-22T15:28:15.003917+0000","last_peered":"2023-05-22T15:28:15.003917+0000","last_clean":"2023-05-22T15:28:15.003917+0000","last_became_active":"2023-05-22T15:28:13.380757+0000","last_became_peered":"2023-05-22T15:28:13.380757+0000","last_unstale":"2023-05-22T15:28:15.003917+0000","last_undegraded":"2023-05-22T15:28:15.003917+0000","last_fullsized":"2023-05-22T15:28:15.003917+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:02:29.202744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.22","version":"0'0","reported_seq":10,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-05-22T15:28:14.683563+0000","last_change":"2023-05-22T15:28:13.373264+0000","last_active":"2023-05-22T15:28:14.683563+0000","last_peered":"2023-05-22T15:28:14.683563+0000","last_clean":"2023-05-22T15:28:14.683563+0000","last_became_active":"2023-05-22T15:28:13.372739+0000","last_became_peered":"2023-05-22T15:28:13.372739+0000","last_unstale":"2023-05-22T15:28:14.683563+0000","last_undegraded":"2023-05-22T15:28:14.683563+0000","last_fullsized":"2023-05-22T15:28:14.683563+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T16:07:16.553042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.21","version":"0'0","reported_seq":10,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-05-22T15:28:14.341507+0000","last_change":"2023-05-22T15:28:13.378926+0000","last_active":"2023-05-22T15:28:14.341507+0000","last_peered":"2023-05-22T15:28:14.341507+0000","last_clean":"2023-05-22T15:28:14.341507+0000","last_became_active":"2023-05-22T15:28:13.378725+0000","last_became_peered":"2023-05-22T15:28:13.378725+0000","last_unstale":"2023-05-22T15:28:14.341507+0000","last_undegraded":"2023-05-22T15:28:14.341507+0000","last_fullsized":"2023-05-22T15:28:14.341507+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:52:47.586962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.20","version":"0'0","reported_seq":10,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-05-22T15:28:14.339213+0000","last_change":"2023-05-22T15:28:13.349317+0000","last_active":"2023-05-22T15:28:14.339213+0000","last_peered":"2023-05-22T15:28:14.339213+0000","last_clean":"2023-05-22T15:28:14.339213+0000","last_became_active":"2023-05-22T15:28:13.348991+0000","last_became_peered":"2023-05-22T15:28:13.348991+0000","last_unstale":"2023-05-22T15:28:14.339213+0000","last_undegraded":"2023-05-22T15:28:14.339213+0000","last_fullsized":"2023-05-22T15:28:14.339213+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:24:49.250828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":10,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-05-22T15:28:15.386259+0000","last_change":"2023-05-22T15:28:13.352847+0000","last_active":"2023-05-22T15:28:15.386259+0000","last_peered":"2023-05-22T15:28:15.386259+0000","last_clean":"2023-05-22T15:28:15.386259+0000","last_became_active":"2023-05-22T15:28:13.352455+0000","last_became_peered":"2023-05-22T15:28:13.352455+0000","last_unstale":"2023-05-22T15:28:15.386259+0000","last_undegraded":"2023-05-22T15:28:15.386259+0000","last_fullsized":"2023-05-22T15:28:15.386259+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:42:41.897151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"41'4","reported_seq":15,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-05-22T15:28:13.349488+0000","last_change":"2023-05-22T15:28:12.972435+0000","last_active":"2023-05-22T15:28:13.349488+0000","last_peered":"2023-05-22T15:28:13.349488+0000","last_clean":"2023-05-22T15:28:13.349488+0000","last_became_active":"2023-05-22T15:28:12.971922+0000","last_became_peered":"2023-05-22T15:28:12.971922+0000","last_unstale":"2023-05-22T15:28:13.349488+0000","last_undegraded":"2023-05-22T15:28:13.349488+0000","last_fullsized":"2023-05-22T15:28:13.349488+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_clean_scrub_stamp":"2023-05-22T15:28:11.927105+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T18:33:01.747002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":566,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":10,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-05-22T15:28:15.004527+0000","last_change":"2023-05-22T15:28:13.385691+0000","last_active":"2023-05-22T15:28:15.004527+0000","last_peered":"2023-05-22T15:28:15.004527+0000","last_clean":"2023-05-22T15:28:15.004527+0000","last_became_active":"2023-05-22T15:28:13.383416+0000","last_became_peered":"2023-05-22T15:28:13.383416+0000","last_unstale":"2023-05-22T15:28:15.004527+0000","last_undegraded":"2023-05-22T15:28:15.004527+0000","last_fullsized":"2023-05-22T15:28:15.004527+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:30:48.719910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1e","version":"41'1","reported_seq":12,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-05-22T15:28:13.388847+0000","last_change":"2023-05-22T15:28:12.996966+0000","last_active":"2023-05-22T15:28:13.388847+0000","last_peered":"2023-05-22T15:28:13.388847+0000","last_clean":"2023-05-22T15:28:13.388847+0000","last_became_active":"2023-05-22T15:28:12.995429+0000","last_became_peered":"2023-05-22T15:28:12.995429+0000","last_unstale":"2023-05-22T15:28:13.388847+0000","last_undegraded":"2023-05-22T15:28:13.388847+0000","last_fullsized":"2023-05-22T15:28:13.388847+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_clean_scrub_stamp":"2023-05-22T15:28:11.927105+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:21:15.037766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":10,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-05-22T15:28:14.344295+0000","last_change":"2023-05-22T15:28:13.342478+0000","last_active":"2023-05-22T15:28:14.344295+0000","last_peered":"2023-05-22T15:28:14.344295+0000","last_clean":"2023-05-22T15:28:14.344295+0000","last_became_active":"2023-05-22T15:28:13.341721+0000","last_became_peered":"2023-05-22T15:28:13.341721+0000","last_unstale":"2023-05-22T15:28:14.344295+0000","last_undegraded":"2023-05-22T15:28:14.344295+0000","last_fullsized":"2023-05-22T15:28:14.344295+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:23:02.289036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1d","version":"41'2","reported_seq":13,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-05-22T15:28:13.381118+0000","last_change":"2023-05-22T15:28:12.962971+0000","last_active":"2023-05-22T15:28:13.381118+0000","last_peered":"2023-05-22T15:28:13.381118+0000","last_clean":"2023-05-22T15:28:13.381118+0000","last_became_active":"2023-05-22T15:28:12.962562+0000","last_became_peered":"2023-05-22T15:28:12.962562+0000","last_unstale":"2023-05-22T15:28:13.381118+0000","last_undegraded":"2023-05-22T15:28:13.381118+0000","last_fullsized":"2023-05-22T15:28:13.381118+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_clean_scrub_stamp":"2023-05-22T15:28:11.927105+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:02:28.198798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":10,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-05-22T15:28:15.386027+0000","last_change":"2023-05-22T15:28:13.387836+0000","last_active":"2023-05-22T15:28:15.386027+0000","last_peered":"2023-05-22T15:28:15.386027+0000","last_clean":"2023-05-22T15:28:15.386027+0000","last_became_active":"2023-05-22T15:28:13.387495+0000","last_became_peered":"2023-05-22T15:28:13.387495+0000","last_unstale":"2023-05-22T15:28:15.386027+0000","last_undegraded":"2023-05-22T15:28:15.386027+0000","last_fullsized":"2023-05-22T15:28:15.386027+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:45:44.276721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"41'2","reported_seq":13,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-05-22T15:28:13.384328+0000","last_change":"2023-05-22T15:28:13.002633+0000","last_active":"2023-05-22T15:28:13.384328+0000","last_peered":"2023-05-22T15:28:13.384328+0000","last_clean":"2023-05-22T15:28:13.384328+0000","last_became_active":"2023-05-22T15:28:13.002087+0000","last_became_peered":"2023-05-22T15:28:13.002087+0000","last_unstale":"2023-05-22T15:28:13.384328+0000","last_undegraded":"2023-05-22T15:28:13.384328+0000","last_fullsized":"2023-05-22T15:28:13.384328+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_clean_scrub_stamp":"2023-05-22T15:28:11.927105+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:42:40.893204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":10,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-05-22T15:28:14.683728+0000","last_change":"2023-05-22T15:28:13.387058+0000","last_active":"2023-05-22T15:28:14.683728+0000","last_peered":"2023-05-22T15:28:14.683728+0000","last_clean":"2023-05-22T15:28:14.683728+0000","last_became_active":"2023-05-22T15:28:13.386582+0000","last_became_peered":"2023-05-22T15:28:13.386582+0000","last_unstale":"2023-05-22T15:28:14.683728+0000","last_undegraded":"2023-05-22T15:28:14.683728+0000","last_fullsized":"2023-05-22T15:28:14.683728+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:20:16.968104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"41'1","reported_seq":12,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-05-22T15:28:13.360937+0000","last_change":"2023-05-22T15:28:13.000382+0000","last_active":"2023-05-22T15:28:13.360937+0000","last_peered":"2023-05-22T15:28:13.360937+0000","last_clean":"2023-05-22T15:28:13.360937+0000","last_became_active":"2023-05-22T15:28:12.999838+0000","last_became_peered":"2023-05-22T15:28:12.999838+0000","last_unstale":"2023-05-22T15:28:13.360937+0000","last_undegraded":"2023-05-22T15:28:13.360937+0000","last_fullsized":"2023-05-22T15:28:13.360937+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_clean_scrub_stamp":"2023-05-22T15:28:11.927105+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:05:30.503639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":10,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-05-22T15:28:15.387248+0000","last_change":"2023-05-22T15:28:13.344369+0000","last_active":"2023-05-22T15:28:15.387248+0000","last_peered":"2023-05-22T15:28:15.387248+0000","last_clean":"2023-05-22T15:28:15.387248+0000","last_became_active":"2023-05-22T15:28:13.343578+0000","last_became_peered":"2023-05-22T15:28:13.343578+0000","last_unstale":"2023-05-22T15:28:15.387248+0000","last_undegraded":"2023-05-22T15:28:15.387248+0000","last_fullsized":"2023-05-22T15:28:15.387248+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:53:53.090998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1a","version":"41'1","reported_seq":12,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-05-22T15:28:13.389342+0000","last_change":"2023-05-22T15:28:12.994044+0000","last_active":"2023-05-22T15:28:13.389342+0000","last_peered":"2023-05-22T15:28:13.389342+0000","last_clean":"2023-05-22T15:28:13.389342+0000","last_became_active":"2023-05-22T15:28:12.989779+0000","last_became_peered":"2023-05-22T15:28:12.989779+0000","last_unstale":"2023-05-22T15:28:13.389342+0000","last_undegraded":"2023-05-22T15:28:13.389342+0000","last_fullsized":"2023-05-22T15:28:13.389342+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_clean_scrub_stamp":"2023-05-22T15:28:11.927105+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:17:33.835222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":10,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-05-22T15:28:15.387122+0000","last_change":"2023-05-22T15:28:13.383546+0000","last_active":"2023-05-22T15:28:15.387122+0000","last_peered":"2023-05-22T15:28:15.387122+0000","last_clean":"2023-05-22T15:28:15.387122+0000","last_became_active":"2023-05-22T15:28:13.383260+0000","last_became_peered":"2023-05-22T15:28:13.383260+0000","last_unstale":"2023-05-22T15:28:15.387122+0000","last_undegraded":"2023-05-22T15:28:15.387122+0000","last_fullsized":"2023-05-22T15:28:15.387122+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:20:16.968104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":11,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-05-22T15:28:13.331463+0000","last_change":"2023-05-22T15:28:12.977530+0000","last_active":"2023-05-22T15:28:13.331463+0000","last_peered":"2023-05-22T15:28:13.331463+0000","last_clean":"2023-05-22T15:28:13.331463+0000","last_became_active":"2023-05-22T15:28:12.976542+0000","last_became_peered":"2023-05-22T15:28:12.976542+0000","last_unstale":"2023-05-22T15:28:13.331463+0000","last_undegraded":"2023-05-22T15:28:13.331463+0000","last_fullsized":"2023-05-22T15:28:13.331463+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_clean_scrub_stamp":"2023-05-22T15:28:11.927105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:45:43.272774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":10,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-05-22T15:28:15.003242+0000","last_change":"2023-05-22T15:28:13.387651+0000","last_active":"2023-05-22T15:28:15.003242+0000","last_peered":"2023-05-22T15:28:15.003242+0000","last_clean":"2023-05-22T15:28:15.003242+0000","last_became_active":"2023-05-22T15:28:13.386571+0000","last_became_peered":"2023-05-22T15:28:13.386571+0000","last_unstale":"2023-05-22T15:28:15.003242+0000","last_undegraded":"2023-05-22T15:28:15.003242+0000","last_fullsized":"2023-05-22T15:28:15.003242+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:23:06.017164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":11,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-05-22T15:28:13.334697+0000","last_change":"2023-05-22T15:28:12.966806+0000","last_active":"2023-05-22T15:28:13.334697+0000","last_peered":"2023-05-22T15:28:13.334697+0000","last_clean":"2023-05-22T15:28:13.334697+0000","last_became_active":"2023-05-22T15:28:12.966081+0000","last_became_peered":"2023-05-22T15:28:12.966081+0000","last_unstale":"2023-05-22T15:28:13.334697+0000","last_undegraded":"2023-05-22T15:28:13.334697+0000","last_fullsized":"2023-05-22T15:28:13.334697+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_clean_scrub_stamp":"2023-05-22T15:28:11.927105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:02:28.198798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":10,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-05-22T15:28:14.684114+0000","last_change":"2023-05-22T15:28:13.377678+0000","last_active":"2023-05-22T15:28:14.684114+0000","last_peered":"2023-05-22T15:28:14.684114+0000","last_clean":"2023-05-22T15:28:14.684114+0000","last_became_active":"2023-05-22T15:28:13.373627+0000","last_became_peered":"2023-05-22T15:28:13.373627+0000","last_unstale":"2023-05-22T15:28:14.684114+0000","last_undegraded":"2023-05-22T15:28:14.684114+0000","last_fullsized":"2023-05-22T15:28:14.684114+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:17:59.314767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":11,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-05-22T15:28:13.355545+0000","last_change":"2023-05-22T15:28:12.989880+0000","last_active":"2023-05-22T15:28:13.355545+0000","last_peered":"2023-05-22T15:28:13.355545+0000","last_clean":"2023-05-22T15:28:13.355545+0000","last_became_active":"2023-05-22T15:28:12.987670+0000","last_became_peered":"2023-05-22T15:28:12.987670+0000","last_unstale":"2023-05-22T15:28:13.355545+0000","last_undegraded":"2023-05-22T15:28:13.355545+0000","last_fullsized":"2023-05-22T15:28:13.355545+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_clean_scrub_stamp":"2023-05-22T15:28:11.927105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:50:22.466750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":10,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-05-22T15:28:15.389009+0000","last_change":"2023-05-22T15:28:13.379880+0000","last_active":"2023-05-22T15:28:15.389009+0000","last_peered":"2023-05-22T15:28:15.389009+0000","last_clean":"2023-05-22T15:28:15.389009+0000","last_became_active":"2023-05-22T15:28:13.379178+0000","last_became_peered":"2023-05-22T15:28:13.379178+0000","last_unstale":"2023-05-22T15:28:15.389009+0000","last_undegraded":"2023-05-22T15:28:15.389009+0000","last_fullsized":"2023-05-22T15:28:15.389009+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:23:02.289036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":11,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-05-22T15:28:13.333064+0000","last_change":"2023-05-22T15:28:13.000074+0000","last_active":"2023-05-22T15:28:13.333064+0000","last_peered":"2023-05-22T15:28:13.333064+0000","last_clean":"2023-05-22T15:28:13.333064+0000","last_became_active":"2023-05-22T15:28:12.999740+0000","last_became_peered":"2023-05-22T15:28:12.999740+0000","last_unstale":"2023-05-22T15:28:13.333064+0000","last_undegraded":"2023-05-22T15:28:13.333064+0000","last_fullsized":"2023-05-22T15:28:13.333064+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_clean_scrub_stamp":"2023-05-22T15:28:11.927105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:15:50.706827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":10,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-05-22T15:28:14.342126+0000","last_change":"2023-05-22T15:28:13.348970+0000","last_active":"2023-05-22T15:28:14.342126+0000","last_peered":"2023-05-22T15:28:14.342126+0000","last_clean":"2023-05-22T15:28:14.342126+0000","last_became_active":"2023-05-22T15:28:13.348728+0000","last_became_peered":"2023-05-22T15:28:13.348728+0000","last_unstale":"2023-05-22T15:28:14.342126+0000","last_undegraded":"2023-05-22T15:28:14.342126+0000","last_fullsized":"2023-05-22T15:28:14.342126+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:45:29.798248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"41'3","reported_seq":14,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-05-22T15:28:13.383947+0000","last_change":"2023-05-22T15:28:13.004069+0000","last_active":"2023-05-22T15:28:13.383947+0000","last_peered":"2023-05-22T15:28:13.383947+0000","last_clean":"2023-05-22T15:28:13.383947+0000","last_became_active":"2023-05-22T15:28:13.003643+0000","last_became_peered":"2023-05-22T15:28:13.003643+0000","last_unstale":"2023-05-22T15:28:13.383947+0000","last_undegraded":"2023-05-22T15:28:13.383947+0000","last_fullsized":"2023-05-22T15:28:13.383947+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_clean_scrub_stamp":"2023-05-22T15:28:11.927105+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:57:04.738345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":10,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-05-22T15:28:15.388554+0000","last_change":"2023-05-22T15:28:13.347075+0000","last_active":"2023-05-22T15:28:15.388554+0000","last_peered":"2023-05-22T15:28:15.388554+0000","last_clean":"2023-05-22T15:28:15.388554+0000","last_became_active":"2023-05-22T15:28:13.346385+0000","last_became_peered":"2023-05-22T15:28:13.346385+0000","last_unstale":"2023-05-22T15:28:15.388554+0000","last_undegraded":"2023-05-22T15:28:15.388554+0000","last_fullsized":"2023-05-22T15:28:15.388554+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:57:10.157295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.14","version":"41'6","reported_seq":17,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-05-22T15:28:13.377302+0000","last_change":"2023-05-22T15:28:12.988477+0000","last_active":"2023-05-22T15:28:13.377302+0000","last_peered":"2023-05-22T15:28:13.377302+0000","last_clean":"2023-05-22T15:28:13.377302+0000","last_became_active":"2023-05-22T15:28:12.988051+0000","last_became_peered":"2023-05-22T15:28:12.988051+0000","last_unstale":"2023-05-22T15:28:13.377302+0000","last_undegraded":"2023-05-22T15:28:13.377302+0000","last_fullsized":"2023-05-22T15:28:13.377302+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_clean_scrub_stamp":"2023-05-22T15:28:11.927105+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:30:08.027313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1548,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":10,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-05-22T15:28:15.003557+0000","last_change":"2023-05-22T15:28:13.383691+0000","last_active":"2023-05-22T15:28:15.003557+0000","last_peered":"2023-05-22T15:28:15.003557+0000","last_clean":"2023-05-22T15:28:15.003557+0000","last_became_active":"2023-05-22T15:28:13.382340+0000","last_became_peered":"2023-05-22T15:28:13.382340+0000","last_unstale":"2023-05-22T15:28:15.003557+0000","last_undegraded":"2023-05-22T15:28:15.003557+0000","last_fullsized":"2023-05-22T15:28:15.003557+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:44:09.792277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"41'2","reported_seq":13,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-05-22T15:28:13.347595+0000","last_change":"2023-05-22T15:28:12.966476+0000","last_active":"2023-05-22T15:28:13.347595+0000","last_peered":"2023-05-22T15:28:13.347595+0000","last_clean":"2023-05-22T15:28:13.347595+0000","last_became_active":"2023-05-22T15:28:12.966145+0000","last_became_peered":"2023-05-22T15:28:12.966145+0000","last_unstale":"2023-05-22T15:28:13.347595+0000","last_undegraded":"2023-05-22T15:28:13.347595+0000","last_fullsized":"2023-05-22T15:28:13.347595+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_clean_scrub_stamp":"2023-05-22T15:28:11.927105+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:47:11.442121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":10,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-05-22T15:28:15.387422+0000","last_change":"2023-05-22T15:28:13.360865+0000","last_active":"2023-05-22T15:28:15.387422+0000","last_peered":"2023-05-22T15:28:15.387422+0000","last_clean":"2023-05-22T15:28:15.387422+0000","last_became_active":"2023-05-22T15:28:13.360473+0000","last_became_peered":"2023-05-22T15:28:13.360473+0000","last_unstale":"2023-05-22T15:28:15.387422+0000","last_undegraded":"2023-05-22T15:28:15.387422+0000","last_fullsized":"2023-05-22T15:28:15.387422+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:15:03.118709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":11,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-05-22T15:28:13.335087+0000","last_change":"2023-05-22T15:28:13.003266+0000","last_active":"2023-05-22T15:28:13.335087+0000","last_peered":"2023-05-22T15:28:13.335087+0000","last_clean":"2023-05-22T15:28:13.335087+0000","last_became_active":"2023-05-22T15:28:13.002780+0000","last_became_peered":"2023-05-22T15:28:13.002780+0000","last_unstale":"2023-05-22T15:28:13.335087+0000","last_undegraded":"2023-05-22T15:28:13.335087+0000","last_fullsized":"2023-05-22T15:28:13.335087+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_clean_scrub_stamp":"2023-05-22T15:28:11.927105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:26:26.763456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":10,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-05-22T15:28:14.341307+0000","last_change":"2023-05-22T15:28:13.348415+0000","last_active":"2023-05-22T15:28:14.341307+0000","last_peered":"2023-05-22T15:28:14.341307+0000","last_clean":"2023-05-22T15:28:14.341307+0000","last_became_active":"2023-05-22T15:28:13.348086+0000","last_became_peered":"2023-05-22T15:28:13.348086+0000","last_unstale":"2023-05-22T15:28:14.341307+0000","last_undegraded":"2023-05-22T15:28:14.341307+0000","last_fullsized":"2023-05-22T15:28:14.341307+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:53:53.090998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":11,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-05-22T15:28:13.336199+0000","last_change":"2023-05-22T15:28:12.966826+0000","last_active":"2023-05-22T15:28:13.336199+0000","last_peered":"2023-05-22T15:28:13.336199+0000","last_clean":"2023-05-22T15:28:13.336199+0000","last_became_active":"2023-05-22T15:28:12.966393+0000","last_became_peered":"2023-05-22T15:28:12.966393+0000","last_unstale":"2023-05-22T15:28:13.336199+0000","last_undegraded":"2023-05-22T15:28:13.336199+0000","last_fullsized":"2023-05-22T15:28:13.336199+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_clean_scrub_stamp":"2023-05-22T15:28:11.927105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:09:59.757240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":10,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-05-22T15:28:15.388586+0000","last_change":"2023-05-22T15:28:13.385135+0000","last_active":"2023-05-22T15:28:15.388586+0000","last_peered":"2023-05-22T15:28:15.388586+0000","last_clean":"2023-05-22T15:28:15.388586+0000","last_became_active":"2023-05-22T15:28:13.384425+0000","last_became_peered":"2023-05-22T15:28:13.384425+0000","last_unstale":"2023-05-22T15:28:15.388586+0000","last_undegraded":"2023-05-22T15:28:15.388586+0000","last_fullsized":"2023-05-22T15:28:15.388586+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:13:07.018500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.10","version":"41'8","reported_seq":19,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-05-22T15:28:13.380187+0000","last_change":"2023-05-22T15:28:12.978289+0000","last_active":"2023-05-22T15:28:13.380187+0000","last_peered":"2023-05-22T15:28:13.380187+0000","last_clean":"2023-05-22T15:28:13.380187+0000","last_became_active":"2023-05-22T15:28:12.977677+0000","last_became_peered":"2023-05-22T15:28:12.977677+0000","last_unstale":"2023-05-22T15:28:13.380187+0000","last_undegraded":"2023-05-22T15:28:13.380187+0000","last_fullsized":"2023-05-22T15:28:13.380187+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_clean_scrub_stamp":"2023-05-22T15:28:11.927105+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T15:35:29.715819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":10,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-05-22T15:28:14.339594+0000","last_change":"2023-05-22T15:28:13.370432+0000","last_active":"2023-05-22T15:28:14.339594+0000","last_peered":"2023-05-22T15:28:14.339594+0000","last_clean":"2023-05-22T15:28:14.339594+0000","last_became_active":"2023-05-22T15:28:13.370181+0000","last_became_peered":"2023-05-22T15:28:13.370181+0000","last_unstale":"2023-05-22T15:28:14.339594+0000","last_undegraded":"2023-05-22T15:28:14.339594+0000","last_fullsized":"2023-05-22T15:28:14.339594+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:26:27.767403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"41'1","reported_seq":12,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-05-22T15:28:13.335487+0000","last_change":"2023-05-22T15:28:12.966282+0000","last_active":"2023-05-22T15:28:13.335487+0000","last_peered":"2023-05-22T15:28:13.335487+0000","last_clean":"2023-05-22T15:28:13.335487+0000","last_became_active":"2023-05-22T15:28:12.965990+0000","last_became_peered":"2023-05-22T15:28:12.965990+0000","last_unstale":"2023-05-22T15:28:13.335487+0000","last_undegraded":"2023-05-22T15:28:13.335487+0000","last_fullsized":"2023-05-22T15:28:13.335487+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_clean_scrub_stamp":"2023-05-22T15:28:11.927105+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:10:24.156615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":10,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-05-22T15:28:15.387921+0000","last_change":"2023-05-22T15:28:13.343851+0000","last_active":"2023-05-22T15:28:15.387921+0000","last_peered":"2023-05-22T15:28:15.387921+0000","last_clean":"2023-05-22T15:28:15.387921+0000","last_became_active":"2023-05-22T15:28:13.343616+0000","last_became_peered":"2023-05-22T15:28:13.343616+0000","last_unstale":"2023-05-22T15:28:15.387921+0000","last_undegraded":"2023-05-22T15:28:15.387921+0000","last_fullsized":"2023-05-22T15:28:15.387921+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:47:12.446067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"41'2","reported_seq":13,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-05-22T15:28:13.373220+0000","last_change":"2023-05-22T15:28:12.993444+0000","last_active":"2023-05-22T15:28:13.373220+0000","last_peered":"2023-05-22T15:28:13.373220+0000","last_clean":"2023-05-22T15:28:13.373220+0000","last_became_active":"2023-05-22T15:28:12.989301+0000","last_became_peered":"2023-05-22T15:28:12.989301+0000","last_unstale":"2023-05-22T15:28:13.373220+0000","last_undegraded":"2023-05-22T15:28:13.373220+0000","last_fullsized":"2023-05-22T15:28:13.373220+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_clean_scrub_stamp":"2023-05-22T15:28:11.927105+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T03:27:50.363592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":10,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-05-22T15:28:15.004221+0000","last_change":"2023-05-22T15:28:13.382241+0000","last_active":"2023-05-22T15:28:15.004221+0000","last_peered":"2023-05-22T15:28:15.004221+0000","last_clean":"2023-05-22T15:28:15.004221+0000","last_became_active":"2023-05-22T15:28:13.380712+0000","last_became_peered":"2023-05-22T15:28:13.380712+0000","last_unstale":"2023-05-22T15:28:15.004221+0000","last_undegraded":"2023-05-22T15:28:15.004221+0000","last_fullsized":"2023-05-22T15:28:15.004221+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T15:35:30.719765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"41'1","reported_seq":12,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-05-22T15:28:13.379399+0000","last_change":"2023-05-22T15:28:12.977974+0000","last_active":"2023-05-22T15:28:13.379399+0000","last_peered":"2023-05-22T15:28:13.379399+0000","last_clean":"2023-05-22T15:28:13.379399+0000","last_became_active":"2023-05-22T15:28:12.977154+0000","last_became_peered":"2023-05-22T15:28:12.977154+0000","last_unstale":"2023-05-22T15:28:13.379399+0000","last_undegraded":"2023-05-22T15:28:13.379399+0000","last_fullsized":"2023-05-22T15:28:13.379399+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_clean_scrub_stamp":"2023-05-22T15:28:11.927105+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:57:04.738345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":10,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-05-22T15:28:15.004412+0000","last_change":"2023-05-22T15:28:13.390799+0000","last_active":"2023-05-22T15:28:15.004412+0000","last_peered":"2023-05-22T15:28:15.004412+0000","last_clean":"2023-05-22T15:28:15.004412+0000","last_became_active":"2023-05-22T15:28:13.387968+0000","last_became_peered":"2023-05-22T15:28:13.387968+0000","last_unstale":"2023-05-22T15:28:15.004412+0000","last_undegraded":"2023-05-22T15:28:15.004412+0000","last_fullsized":"2023-05-22T15:28:15.004412+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:43:15.380609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":11,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-05-22T15:28:13.336965+0000","last_change":"2023-05-22T15:28:12.983659+0000","last_active":"2023-05-22T15:28:13.336965+0000","last_peered":"2023-05-22T15:28:13.336965+0000","last_clean":"2023-05-22T15:28:13.336965+0000","last_became_active":"2023-05-22T15:28:12.983185+0000","last_became_peered":"2023-05-22T15:28:12.983185+0000","last_unstale":"2023-05-22T15:28:13.336965+0000","last_undegraded":"2023-05-22T15:28:13.336965+0000","last_fullsized":"2023-05-22T15:28:13.336965+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_clean_scrub_stamp":"2023-05-22T15:28:11.927105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:45:43.272774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":10,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-05-22T15:28:14.341498+0000","last_change":"2023-05-22T15:28:13.348470+0000","last_active":"2023-05-22T15:28:14.341498+0000","last_peered":"2023-05-22T15:28:14.341498+0000","last_clean":"2023-05-22T15:28:14.341498+0000","last_became_active":"2023-05-22T15:28:13.348241+0000","last_became_peered":"2023-05-22T15:28:13.348241+0000","last_unstale":"2023-05-22T15:28:14.341498+0000","last_undegraded":"2023-05-22T15:28:14.341498+0000","last_fullsized":"2023-05-22T15:28:14.341498+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T15:50:36.712116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"41'2","reported_seq":13,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-05-22T15:28:13.360823+0000","last_change":"2023-05-22T15:28:13.000058+0000","last_active":"2023-05-22T15:28:13.360823+0000","last_peered":"2023-05-22T15:28:13.360823+0000","last_clean":"2023-05-22T15:28:13.360823+0000","last_became_active":"2023-05-22T15:28:12.999699+0000","last_became_peered":"2023-05-22T15:28:12.999699+0000","last_unstale":"2023-05-22T15:28:13.360823+0000","last_undegraded":"2023-05-22T15:28:13.360823+0000","last_fullsized":"2023-05-22T15:28:13.360823+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_clean_scrub_stamp":"2023-05-22T15:28:11.927105+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:50:22.466750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":10,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-05-22T15:28:15.385798+0000","last_change":"2023-05-22T15:28:13.380477+0000","last_active":"2023-05-22T15:28:15.385798+0000","last_peered":"2023-05-22T15:28:15.385798+0000","last_clean":"2023-05-22T15:28:15.385798+0000","last_became_active":"2023-05-22T15:28:13.380268+0000","last_became_peered":"2023-05-22T15:28:13.380268+0000","last_unstale":"2023-05-22T15:28:15.385798+0000","last_undegraded":"2023-05-22T15:28:15.385798+0000","last_fullsized":"2023-05-22T15:28:15.385798+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T18:18:36.205176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":11,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-05-22T15:28:13.332299+0000","last_change":"2023-05-22T15:28:12.987140+0000","last_active":"2023-05-22T15:28:13.332299+0000","last_peered":"2023-05-22T15:28:13.332299+0000","last_clean":"2023-05-22T15:28:13.332299+0000","last_became_active":"2023-05-22T15:28:12.986375+0000","last_became_peered":"2023-05-22T15:28:12.986375+0000","last_unstale":"2023-05-22T15:28:13.332299+0000","last_undegraded":"2023-05-22T15:28:13.332299+0000","last_fullsized":"2023-05-22T15:28:13.332299+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_clean_scrub_stamp":"2023-05-22T15:28:11.927105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T18:18:35.201229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":10,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-05-22T15:28:14.339278+0000","last_change":"2023-05-22T15:28:13.360186+0000","last_active":"2023-05-22T15:28:14.339278+0000","last_peered":"2023-05-22T15:28:14.339278+0000","last_clean":"2023-05-22T15:28:14.339278+0000","last_became_active":"2023-05-22T15:28:13.359892+0000","last_became_peered":"2023-05-22T15:28:13.359892+0000","last_unstale":"2023-05-22T15:28:14.339278+0000","last_undegraded":"2023-05-22T15:28:14.339278+0000","last_fullsized":"2023-05-22T15:28:14.339278+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T18:18:36.205176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":11,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-05-22T15:28:13.331300+0000","last_change":"2023-05-22T15:28:13.002443+0000","last_active":"2023-05-22T15:28:13.331300+0000","last_peered":"2023-05-22T15:28:13.331300+0000","last_clean":"2023-05-22T15:28:13.331300+0000","last_became_active":"2023-05-22T15:28:13.002233+0000","last_became_peered":"2023-05-22T15:28:13.002233+0000","last_unstale":"2023-05-22T15:28:13.331300+0000","last_undegraded":"2023-05-22T15:28:13.331300+0000","last_fullsized":"2023-05-22T15:28:13.331300+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_clean_scrub_stamp":"2023-05-22T15:28:11.927105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:24:48.246881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":10,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-05-22T15:28:15.387176+0000","last_change":"2023-05-22T15:28:13.349627+0000","last_active":"2023-05-22T15:28:15.387176+0000","last_peered":"2023-05-22T15:28:15.387176+0000","last_clean":"2023-05-22T15:28:15.387176+0000","last_became_active":"2023-05-22T15:28:13.349381+0000","last_became_peered":"2023-05-22T15:28:13.349381+0000","last_unstale":"2023-05-22T15:28:15.387176+0000","last_undegraded":"2023-05-22T15:28:15.387176+0000","last_fullsized":"2023-05-22T15:28:15.387176+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:14:04.358855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":11,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-05-22T15:28:13.337691+0000","last_change":"2023-05-22T15:28:13.002516+0000","last_active":"2023-05-22T15:28:13.337691+0000","last_peered":"2023-05-22T15:28:13.337691+0000","last_clean":"2023-05-22T15:28:13.337691+0000","last_became_active":"2023-05-22T15:28:13.002011+0000","last_became_peered":"2023-05-22T15:28:13.002011+0000","last_unstale":"2023-05-22T15:28:13.337691+0000","last_undegraded":"2023-05-22T15:28:13.337691+0000","last_fullsized":"2023-05-22T15:28:13.337691+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_clean_scrub_stamp":"2023-05-22T15:28:11.927105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T18:18:35.201229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":10,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-05-22T15:28:14.341606+0000","last_change":"2023-05-22T15:28:13.344193+0000","last_active":"2023-05-22T15:28:14.341606+0000","last_peered":"2023-05-22T15:28:14.341606+0000","last_clean":"2023-05-22T15:28:14.341606+0000","last_became_active":"2023-05-22T15:28:13.344010+0000","last_became_peered":"2023-05-22T15:28:13.344010+0000","last_unstale":"2023-05-22T15:28:14.341606+0000","last_undegraded":"2023-05-22T15:28:14.341606+0000","last_fullsized":"2023-05-22T15:28:14.341606+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:05:14.518546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":11,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-05-22T15:28:13.335391+0000","last_change":"2023-05-22T15:28:13.003224+0000","last_active":"2023-05-22T15:28:13.335391+0000","last_peered":"2023-05-22T15:28:13.335391+0000","last_clean":"2023-05-22T15:28:13.335391+0000","last_became_active":"2023-05-22T15:28:13.002695+0000","last_became_peered":"2023-05-22T15:28:13.002695+0000","last_unstale":"2023-05-22T15:28:13.335391+0000","last_undegraded":"2023-05-22T15:28:13.335391+0000","last_fullsized":"2023-05-22T15:28:13.335391+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_clean_scrub_stamp":"2023-05-22T15:28:11.927105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:15:37.287885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":28,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-05-22T15:28:13.358832+0000","last_change":"2023-05-22T15:26:44.982421+0000","last_active":"2023-05-22T15:28:13.358832+0000","last_peered":"2023-05-22T15:28:13.358832+0000","last_clean":"2023-05-22T15:28:13.358832+0000","last_became_active":"2023-05-22T15:26:44.981586+0000","last_became_peered":"2023-05-22T15:26:44.981586+0000","last_unstale":"2023-05-22T15:28:13.358832+0000","last_undegraded":"2023-05-22T15:28:13.358832+0000","last_fullsized":"2023-05-22T15:28:13.358832+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:14:11.156544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":10,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-05-22T15:28:14.341581+0000","last_change":"2023-05-22T15:28:13.348288+0000","last_active":"2023-05-22T15:28:14.341581+0000","last_peered":"2023-05-22T15:28:14.341581+0000","last_clean":"2023-05-22T15:28:14.341581+0000","last_became_active":"2023-05-22T15:28:13.348009+0000","last_became_peered":"2023-05-22T15:28:13.348009+0000","last_unstale":"2023-05-22T15:28:14.341581+0000","last_undegraded":"2023-05-22T15:28:14.341581+0000","last_fullsized":"2023-05-22T15:28:14.341581+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:30:20.535065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":11,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-05-22T15:28:13.353887+0000","last_change":"2023-05-22T15:28:12.985752+0000","last_active":"2023-05-22T15:28:13.353887+0000","last_peered":"2023-05-22T15:28:13.353887+0000","last_clean":"2023-05-22T15:28:13.353887+0000","last_became_active":"2023-05-22T15:28:12.980963+0000","last_became_peered":"2023-05-22T15:28:12.980963+0000","last_unstale":"2023-05-22T15:28:13.353887+0000","last_undegraded":"2023-05-22T15:28:13.353887+0000","last_fullsized":"2023-05-22T15:28:13.353887+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_clean_scrub_stamp":"2023-05-22T15:28:11.927105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:05:30.503639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":27,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-05-22T15:28:13.345314+0000","last_change":"2023-05-22T15:26:44.978858+0000","last_active":"2023-05-22T15:28:13.345314+0000","last_peered":"2023-05-22T15:28:13.345314+0000","last_clean":"2023-05-22T15:28:13.345314+0000","last_became_active":"2023-05-22T15:26:44.978443+0000","last_became_peered":"2023-05-22T15:26:44.978443+0000","last_unstale":"2023-05-22T15:28:13.345314+0000","last_undegraded":"2023-05-22T15:28:13.345314+0000","last_fullsized":"2023-05-22T15:28:13.345314+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:46:29.290243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":10,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-05-22T15:28:14.343230+0000","last_change":"2023-05-22T15:28:13.358572+0000","last_active":"2023-05-22T15:28:14.343230+0000","last_peered":"2023-05-22T15:28:14.343230+0000","last_clean":"2023-05-22T15:28:14.343230+0000","last_became_active":"2023-05-22T15:28:13.357213+0000","last_became_peered":"2023-05-22T15:28:13.357213+0000","last_unstale":"2023-05-22T15:28:14.343230+0000","last_undegraded":"2023-05-22T15:28:14.343230+0000","last_fullsized":"2023-05-22T15:28:14.343230+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:43:14.133670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":11,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-05-22T15:28:13.348125+0000","last_change":"2023-05-22T15:28:12.983493+0000","last_active":"2023-05-22T15:28:13.348125+0000","last_peered":"2023-05-22T15:28:13.348125+0000","last_clean":"2023-05-22T15:28:13.348125+0000","last_became_active":"2023-05-22T15:28:12.983056+0000","last_became_peered":"2023-05-22T15:28:12.983056+0000","last_unstale":"2023-05-22T15:28:13.348125+0000","last_undegraded":"2023-05-22T15:28:13.348125+0000","last_fullsized":"2023-05-22T15:28:13.348125+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_clean_scrub_stamp":"2023-05-22T15:28:11.927105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:43:14.376663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":31,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-05-22T15:28:13.337022+0000","last_change":"2023-05-22T15:26:45.092367+0000","last_active":"2023-05-22T15:28:13.337022+0000","last_peered":"2023-05-22T15:28:13.337022+0000","last_clean":"2023-05-22T15:28:13.337022+0000","last_became_active":"2023-05-22T15:26:45.091951+0000","last_became_peered":"2023-05-22T15:26:45.091951+0000","last_unstale":"2023-05-22T15:28:13.337022+0000","last_undegraded":"2023-05-22T15:28:13.337022+0000","last_fullsized":"2023-05-22T15:28:13.337022+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:42:57.973030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":10,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-05-22T15:28:14.683663+0000","last_change":"2023-05-22T15:28:13.373706+0000","last_active":"2023-05-22T15:28:14.683663+0000","last_peered":"2023-05-22T15:28:14.683663+0000","last_clean":"2023-05-22T15:28:14.683663+0000","last_became_active":"2023-05-22T15:28:13.372681+0000","last_became_peered":"2023-05-22T15:28:13.372681+0000","last_unstale":"2023-05-22T15:28:14.683663+0000","last_undegraded":"2023-05-22T15:28:14.683663+0000","last_fullsized":"2023-05-22T15:28:14.683663+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T18:56:04.263372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":11,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-05-22T15:28:13.334848+0000","last_change":"2023-05-22T15:28:12.969674+0000","last_active":"2023-05-22T15:28:13.334848+0000","last_peered":"2023-05-22T15:28:13.334848+0000","last_clean":"2023-05-22T15:28:13.334848+0000","last_became_active":"2023-05-22T15:28:12.969055+0000","last_became_peered":"2023-05-22T15:28:12.969055+0000","last_unstale":"2023-05-22T15:28:13.334848+0000","last_undegraded":"2023-05-22T15:28:13.334848+0000","last_fullsized":"2023-05-22T15:28:13.334848+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_clean_scrub_stamp":"2023-05-22T15:28:11.927105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:23:05.013217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":27,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-05-22T15:28:13.344896+0000","last_change":"2023-05-22T15:26:45.091475+0000","last_active":"2023-05-22T15:28:13.344896+0000","last_peered":"2023-05-22T15:28:13.344896+0000","last_clean":"2023-05-22T15:28:13.344896+0000","last_became_active":"2023-05-22T15:26:45.091143+0000","last_became_peered":"2023-05-22T15:26:45.091143+0000","last_unstale":"2023-05-22T15:28:13.344896+0000","last_undegraded":"2023-05-22T15:28:13.344896+0000","last_fullsized":"2023-05-22T15:28:13.344896+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:12:41.464054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":10,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-05-22T15:28:15.004652+0000","last_change":"2023-05-22T15:28:13.384023+0000","last_active":"2023-05-22T15:28:15.004652+0000","last_peered":"2023-05-22T15:28:15.004652+0000","last_clean":"2023-05-22T15:28:15.004652+0000","last_became_active":"2023-05-22T15:28:13.382720+0000","last_became_peered":"2023-05-22T15:28:13.382720+0000","last_unstale":"2023-05-22T15:28:15.004652+0000","last_undegraded":"2023-05-22T15:28:15.004652+0000","last_fullsized":"2023-05-22T15:28:15.004652+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:24:49.250828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"42'1","reported_seq":12,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-05-22T15:28:14.341897+0000","last_change":"2023-05-22T15:28:12.998080+0000","last_active":"2023-05-22T15:28:14.341897+0000","last_peered":"2023-05-22T15:28:14.341897+0000","last_clean":"2023-05-22T15:28:14.341897+0000","last_became_active":"2023-05-22T15:28:12.997779+0000","last_became_peered":"2023-05-22T15:28:12.997779+0000","last_unstale":"2023-05-22T15:28:14.341897+0000","last_undegraded":"2023-05-22T15:28:14.341897+0000","last_fullsized":"2023-05-22T15:28:14.341897+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_clean_scrub_stamp":"2023-05-22T15:28:11.927105+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:21:15.037766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":32,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-05-22T15:28:13.332480+0000","last_change":"2023-05-22T15:26:44.976632+0000","last_active":"2023-05-22T15:28:13.332480+0000","last_peered":"2023-05-22T15:28:13.332480+0000","last_clean":"2023-05-22T15:28:13.332480+0000","last_became_active":"2023-05-22T15:26:44.975546+0000","last_became_peered":"2023-05-22T15:26:44.975546+0000","last_unstale":"2023-05-22T15:28:13.332480+0000","last_undegraded":"2023-05-22T15:28:13.332480+0000","last_fullsized":"2023-05-22T15:28:13.332480+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:42:57.973030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":10,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-05-22T15:28:15.387562+0000","last_change":"2023-05-22T15:28:13.372938+0000","last_active":"2023-05-22T15:28:15.387562+0000","last_peered":"2023-05-22T15:28:15.387562+0000","last_clean":"2023-05-22T15:28:15.387562+0000","last_became_active":"2023-05-22T15:28:13.372626+0000","last_became_peered":"2023-05-22T15:28:13.372626+0000","last_unstale":"2023-05-22T15:28:15.387562+0000","last_undegraded":"2023-05-22T15:28:15.387562+0000","last_fullsized":"2023-05-22T15:28:15.387562+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:16:53.287861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":11,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-05-22T15:28:13.336797+0000","last_change":"2023-05-22T15:28:13.004749+0000","last_active":"2023-05-22T15:28:13.336797+0000","last_peered":"2023-05-22T15:28:13.336797+0000","last_clean":"2023-05-22T15:28:13.336797+0000","last_became_active":"2023-05-22T15:28:13.003949+0000","last_became_peered":"2023-05-22T15:28:13.003949+0000","last_unstale":"2023-05-22T15:28:13.336797+0000","last_undegraded":"2023-05-22T15:28:13.336797+0000","last_fullsized":"2023-05-22T15:28:13.336797+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_clean_scrub_stamp":"2023-05-22T15:28:11.927105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:45:43.272774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":31,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-05-22T15:28:13.326617+0000","last_change":"2023-05-22T15:26:44.969469+0000","last_active":"2023-05-22T15:28:13.326617+0000","last_peered":"2023-05-22T15:28:13.326617+0000","last_clean":"2023-05-22T15:28:13.326617+0000","last_became_active":"2023-05-22T15:26:44.969083+0000","last_became_peered":"2023-05-22T15:26:44.969083+0000","last_unstale":"2023-05-22T15:28:13.326617+0000","last_undegraded":"2023-05-22T15:28:13.326617+0000","last_fullsized":"2023-05-22T15:28:13.326617+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:29:23.670067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":10,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-05-22T15:28:15.388684+0000","last_change":"2023-05-22T15:28:13.360211+0000","last_active":"2023-05-22T15:28:15.388684+0000","last_peered":"2023-05-22T15:28:15.388684+0000","last_clean":"2023-05-22T15:28:15.388684+0000","last_became_active":"2023-05-22T15:28:13.359411+0000","last_became_peered":"2023-05-22T15:28:13.359411+0000","last_unstale":"2023-05-22T15:28:15.388684+0000","last_undegraded":"2023-05-22T15:28:15.388684+0000","last_fullsized":"2023-05-22T15:28:15.388684+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:14:04.358855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":11,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-05-22T15:28:13.344756+0000","last_change":"2023-05-22T15:28:12.963131+0000","last_active":"2023-05-22T15:28:13.344756+0000","last_peered":"2023-05-22T15:28:13.344756+0000","last_clean":"2023-05-22T15:28:13.344756+0000","last_became_active":"2023-05-22T15:28:12.962689+0000","last_became_peered":"2023-05-22T15:28:12.962689+0000","last_unstale":"2023-05-22T15:28:13.344756+0000","last_undegraded":"2023-05-22T15:28:13.344756+0000","last_fullsized":"2023-05-22T15:28:13.344756+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_clean_scrub_stamp":"2023-05-22T15:28:11.927105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:23:05.013217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":31,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-05-22T15:28:13.337392+0000","last_change":"2023-05-22T15:26:44.977198+0000","last_active":"2023-05-22T15:28:13.337392+0000","last_peered":"2023-05-22T15:28:13.337392+0000","last_clean":"2023-05-22T15:28:13.337392+0000","last_became_active":"2023-05-22T15:26:44.976445+0000","last_became_peered":"2023-05-22T15:26:44.976445+0000","last_unstale":"2023-05-22T15:28:13.337392+0000","last_undegraded":"2023-05-22T15:28:13.337392+0000","last_fullsized":"2023-05-22T15:28:13.337392+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:12:41.464054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":10,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-05-22T15:28:15.388010+0000","last_change":"2023-05-22T15:28:13.360506+0000","last_active":"2023-05-22T15:28:15.388010+0000","last_peered":"2023-05-22T15:28:15.388010+0000","last_clean":"2023-05-22T15:28:15.388010+0000","last_became_active":"2023-05-22T15:28:13.360257+0000","last_became_peered":"2023-05-22T15:28:13.360257+0000","last_unstale":"2023-05-22T15:28:15.388010+0000","last_undegraded":"2023-05-22T15:28:15.388010+0000","last_fullsized":"2023-05-22T15:28:15.388010+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:12.931051+0000","last_clean_scrub_stamp":"2023-05-22T15:28:12.931051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:05:14.518546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"41'2","reported_seq":13,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-05-22T15:28:13.379621+0000","last_change":"2023-05-22T15:28:12.978763+0000","last_active":"2023-05-22T15:28:13.379621+0000","last_peered":"2023-05-22T15:28:13.379621+0000","last_clean":"2023-05-22T15:28:13.379621+0000","last_became_active":"2023-05-22T15:28:12.977772+0000","last_became_peered":"2023-05-22T15:28:12.977772+0000","last_unstale":"2023-05-22T15:28:13.379621+0000","last_undegraded":"2023-05-22T15:28:13.379621+0000","last_fullsized":"2023-05-22T15:28:13.379621+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:28:11.927105+0000","last_clean_scrub_stamp":"2023-05-22T15:28:11.927105+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:15:37.287885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":32,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-05-22T15:28:13.332262+0000","last_change":"2023-05-22T15:26:44.971951+0000","last_active":"2023-05-22T15:28:13.332262+0000","last_peered":"2023-05-22T15:28:13.332262+0000","last_clean":"2023-05-22T15:28:13.332262+0000","last_became_active":"2023-05-22T15:26:44.970872+0000","last_became_peered":"2023-05-22T15:26:44.970872+0000","last_unstale":"2023-05-22T15:28:13.332262+0000","last_undegraded":"2023-05-22T15:28:13.332262+0000","last_fullsized":"2023-05-22T15:28:13.332262+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:55:34.752949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":184,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-05-22T15:28:13.337936+0000","last_change":"2023-05-22T15:26:39.318204+0000","last_active":"2023-05-22T15:28:13.337936+0000","last_peered":"2023-05-22T15:28:13.337936+0000","last_clean":"2023-05-22T15:28:13.337936+0000","last_became_active":"2023-05-22T15:26:39.315370+0000","last_became_peered":"2023-05-22T15:26:39.315370+0000","last_unstale":"2023-05-22T15:28:13.337936+0000","last_undegraded":"2023-05-22T15:28:13.337936+0000","last_fullsized":"2023-05-22T15:28:13.337936+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:38.020076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:38.020076+0000","last_clean_scrub_stamp":"2023-05-22T15:26:38.020076+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:06:39.207543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":2330,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574871,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":13932,"kb_used_data":1060,"kb_used_omap":0,"kb_used_meta":12864,"kb_avail":94357908,"statfs":{"total":96636764160,"available":96622497792,"internally_reserved":0,"allocated":1085440,"data_stored":809234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":13172736},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 15:27:38 2023","interfaces":[{"interface":"back","average":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":1.5860000000000001,"5min":1.5860000000000001,"15min":1.5860000000000001},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"min":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.4239999999999999,"5min":1.4239999999999999,"15min":1.4239999999999999},"last":1.0369999999999999}]},{"osd":1,"last update":"Mon May 22 15:27:38 2023","interfaces":[{"interface":"back","average":{"1min":0.96599999999999997,"5min":0.96599999999999997,"15min":0.96599999999999997},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":1.3600000000000001,"5min":1.3600000000000001,"15min":1.3600000000000001},"last":1.0029999999999999},{"interface":"front","average":{"1min":1.002,"5min":1.002,"15min":1.002},"min":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"max":{"1min":1.8260000000000001,"5min":1.8260000000000001,"15min":1.8260000000000001},"last":0.82699999999999996}]},{"osd":2,"last update":"Mon May 22 15:27:38 2023","interfaces":[{"interface":"back","average":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.127,"5min":1.127,"15min":1.127},"last":0.90200000000000002},{"interface":"front","average":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":1.23,"5min":1.23,"15min":1.23},"last":0.73299999999999998}]},{"osd":3,"last update":"Mon May 22 15:27:38 2023","interfaces":[{"interface":"back","average":{"1min":0.99099999999999999,"5min":0.99099999999999999,"15min":0.99099999999999999},"min":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"max":{"1min":1.7829999999999999,"5min":1.7829999999999999,"15min":1.7829999999999999},"last":0.95099999999999996},{"interface":"front","average":{"1min":1.026,"5min":1.026,"15min":1.026},"min":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":1.6879999999999999,"5min":1.6879999999999999,"15min":1.6879999999999999},"last":0.78100000000000003}]},{"osd":4,"last update":"Mon May 22 15:27:38 2023","interfaces":[{"interface":"back","average":{"1min":0.92700000000000005,"5min":0.92700000000000005,"15min":0.92700000000000005},"min":{"1min":0.62,"5min":0.62,"15min":0.62},"max":{"1min":1.2430000000000001,"5min":1.2430000000000001,"15min":1.2430000000000001},"last":1.1160000000000001},{"interface":"front","average":{"1min":1.1120000000000001,"5min":1.1120000000000001,"15min":1.1120000000000001},"min":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"max":{"1min":1.73,"5min":1.73,"15min":1.73},"last":1.0800000000000001}]},{"osd":5,"last update":"Mon May 22 15:27:38 2023","interfaces":[{"interface":"back","average":{"1min":1.123,"5min":1.123,"15min":1.123},"min":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":1.3779999999999999,"5min":1.3779999999999999,"15min":1.3779999999999999},"last":1.1399999999999999},{"interface":"front","average":{"1min":1.139,"5min":1.139,"15min":1.139},"min":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"max":{"1min":1.6439999999999999,"5min":1.6439999999999999,"15min":1.6439999999999999},"last":1.1950000000000001}]},{"osd":6,"last update":"Mon May 22 15:27:38 2023","interfaces":[{"interface":"back","average":{"1min":1.1020000000000001,"5min":1.1020000000000001,"15min":1.1020000000000001},"min":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"max":{"1min":1.3999999999999999,"5min":1.3999999999999999,"15min":1.3999999999999999},"last":1.2410000000000001},{"interface":"front","average":{"1min":1.1890000000000001,"5min":1.1890000000000001,"15min":1.1890000000000001},"min":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"max":{"1min":2.04,"5min":2.04,"15min":2.04},"last":1.169}]}]},{"osd":6,"up_from":13,"seq":55834574872,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8620,"kb_used_data":484,"kb_used_omap":0,"kb_used_meta":8128,"kb_avail":94363220,"statfs":{"total":96636764160,"available":96627937280,"internally_reserved":0,"allocated":495616,"data_stored":219432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8323072},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 15:27:41 2023","interfaces":[{"interface":"back","average":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"min":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":1.1659999999999999,"5min":1.1659999999999999,"15min":1.1659999999999999},"last":0.76900000000000002},{"interface":"front","average":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"min":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"max":{"1min":1.385,"5min":1.385,"15min":1.385},"last":0.92400000000000004}]},{"osd":1,"last update":"Mon May 22 15:27:41 2023","interfaces":[{"interface":"back","average":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"min":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":1.2869999999999999,"5min":1.2869999999999999,"15min":1.2869999999999999},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":0.90000000000000002},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":1.204,"5min":1.204,"15min":1.204},"last":1.0589999999999999}]},{"osd":2,"last update":"Mon May 22 15:27:41 2023","interfaces":[{"interface":"back","average":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"min":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"max":{"1min":1.145,"5min":1.145,"15min":1.145},"last":1.3220000000000001},{"interface":"front","average":{"1min":1.0569999999999999,"5min":1.0569999999999999,"15min":1.0569999999999999},"min":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"max":{"1min":1.4670000000000001,"5min":1.4670000000000001,"15min":1.4670000000000001},"last":0.97099999999999997}]},{"osd":3,"last update":"Mon May 22 15:27:41 2023","interfaces":[{"interface":"back","average":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"min":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"max":{"1min":1.415,"5min":1.415,"15min":1.415},"last":1.3799999999999999},{"interface":"front","average":{"1min":0.95799999999999996,"5min":0.95799999999999996,"15min":0.95799999999999996},"min":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":1.486,"5min":1.486,"15min":1.486},"last":0.79600000000000004}]},{"osd":4,"last update":"Mon May 22 15:27:41 2023","interfaces":[{"interface":"back","average":{"1min":1.0389999999999999,"5min":1.0389999999999999,"15min":1.0389999999999999},"min":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"max":{"1min":1.6459999999999999,"5min":1.6459999999999999,"15min":1.6459999999999999},"last":0.878},{"interface":"front","average":{"1min":1.141,"5min":1.141,"15min":1.141},"min":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"max":{"1min":1.5469999999999999,"5min":1.5469999999999999,"15min":1.5469999999999999},"last":1.4730000000000001}]},{"osd":5,"last update":"Mon May 22 15:27:41 2023","interfaces":[{"interface":"back","average":{"1min":1.075,"5min":1.075,"15min":1.075},"min":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"max":{"1min":1.621,"5min":1.621,"15min":1.621},"last":1.01},{"interface":"front","average":{"1min":1.149,"5min":1.149,"15min":1.149},"min":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"max":{"1min":1.7509999999999999,"5min":1.7509999999999999,"15min":1.7509999999999999},"last":1.1439999999999999}]},{"osd":7,"last update":"Mon May 22 15:27:41 2023","interfaces":[{"interface":"back","average":{"1min":1.268,"5min":1.268,"15min":1.268},"min":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"max":{"1min":1.8120000000000001,"5min":1.8120000000000001,"15min":1.8120000000000001},"last":1.1890000000000001},{"interface":"front","average":{"1min":1.179,"5min":1.179,"15min":1.179},"min":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"max":{"1min":1.6819999999999999,"5min":1.6819999999999999,"15min":1.6819999999999999},"last":1.218}]}]},{"osd":5,"up_from":13,"seq":55834574872,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8680,"kb_used_data":480,"kb_used_omap":0,"kb_used_meta":8192,"kb_avail":94363160,"statfs":{"total":96636764160,"available":96627875840,"internally_reserved":0,"allocated":491520,"data_stored":218846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8388608},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 15:27:37 2023","interfaces":[{"interface":"back","average":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"min":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":1.169,"5min":1.169,"15min":1.169},"last":0.77200000000000002},{"interface":"front","average":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"min":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.103,"5min":1.103,"15min":1.103},"last":0.84399999999999997}]},{"osd":1,"last update":"Mon May 22 15:27:37 2023","interfaces":[{"interface":"back","average":{"1min":0.90200000000000002,"5min":0.90200000000000002,"15min":0.90200000000000002},"min":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":1.2170000000000001,"5min":1.2170000000000001,"15min":1.2170000000000001},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"min":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":1.399,"5min":1.399,"15min":1.399},"last":0.68600000000000005}]},{"osd":2,"last update":"Mon May 22 15:27:37 2023","interfaces":[{"interface":"back","average":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"min":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":1.597,"5min":1.597,"15min":1.597},"last":1.1319999999999999},{"interface":"front","average":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"min":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.296,"5min":1.296,"15min":1.296},"last":0.74099999999999999}]},{"osd":3,"last update":"Mon May 22 15:27:37 2023","interfaces":[{"interface":"back","average":{"1min":0.93400000000000005,"5min":0.93400000000000005,"15min":0.93400000000000005},"min":{"1min":0.627,"5min":0.627,"15min":0.627},"max":{"1min":1.6839999999999999,"5min":1.6839999999999999,"15min":1.6839999999999999},"last":0.56000000000000005},{"interface":"front","average":{"1min":1.0209999999999999,"5min":1.0209999999999999,"15min":1.0209999999999999},"min":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"max":{"1min":1.5189999999999999,"5min":1.5189999999999999,"15min":1.5189999999999999},"last":1.3109999999999999}]},{"osd":4,"last update":"Mon May 22 15:27:37 2023","interfaces":[{"interface":"back","average":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":0.96399999999999997},"min":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"max":{"1min":1.3140000000000001,"5min":1.3140000000000001,"15min":1.3140000000000001},"last":1.0900000000000001},{"interface":"front","average":{"1min":1.0549999999999999,"5min":1.0549999999999999,"15min":1.0549999999999999},"min":{"1min":0.747,"5min":0.747,"15min":0.747},"max":{"1min":1.319,"5min":1.319,"15min":1.319},"last":1.1830000000000001}]},{"osd":6,"last update":"Mon May 22 15:27:37 2023","interfaces":[{"interface":"back","average":{"1min":1.1000000000000001,"5min":1.1000000000000001,"15min":1.1000000000000001},"min":{"1min":0.94199999999999995,"5min":0.94199999999999995,"15min":0.94199999999999995},"max":{"1min":1.357,"5min":1.357,"15min":1.357},"last":1.399},{"interface":"front","average":{"1min":1.109,"5min":1.109,"15min":1.109},"min":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"max":{"1min":1.4350000000000001,"5min":1.4350000000000001,"15min":1.4350000000000001},"last":0.92400000000000004}]},{"osd":7,"last update":"Mon May 22 15:27:37 2023","interfaces":[{"interface":"back","average":{"1min":1.232,"5min":1.232,"15min":1.232},"min":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"max":{"1min":2.8260000000000001,"5min":2.8260000000000001,"15min":2.8260000000000001},"last":1.0089999999999999},{"interface":"front","average":{"1min":1.1679999999999999,"5min":1.1679999999999999,"15min":1.1679999999999999},"min":{"1min":0.877,"5min":0.877,"15min":0.877},"max":{"1min":1.4490000000000001,"5min":1.4490000000000001,"15min":1.4490000000000001},"last":1.476}]}]},{"osd":4,"up_from":13,"seq":55834574870,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8752,"kb_used_data":488,"kb_used_omap":0,"kb_used_meta":8256,"kb_avail":94363088,"statfs":{"total":96636764160,"available":96627802112,"internally_reserved":0,"allocated":499712,"data_stored":220324,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8454144},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 15:27:37 2023","interfaces":[{"interface":"back","average":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"min":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":1.252,"5min":1.252,"15min":1.252},"last":0.754},{"interface":"front","average":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"min":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"max":{"1min":1.2989999999999999,"5min":1.2989999999999999,"15min":1.2989999999999999},"last":0.80500000000000005}]},{"osd":1,"last update":"Mon May 22 15:27:37 2023","interfaces":[{"interface":"back","average":{"1min":0.872,"5min":0.872,"15min":0.872},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.3009999999999999,"5min":1.3009999999999999,"15min":1.3009999999999999},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":0.83299999999999996},"min":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":1.1499999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.51600000000000001}]},{"osd":2,"last update":"Mon May 22 15:27:37 2023","interfaces":[{"interface":"back","average":{"1min":0.96199999999999997,"5min":0.96199999999999997,"15min":0.96199999999999997},"min":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"max":{"1min":1.284,"5min":1.284,"15min":1.284},"last":0.91500000000000004},{"interface":"front","average":{"1min":1.0780000000000001,"5min":1.0780000000000001,"15min":1.0780000000000001},"min":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"max":{"1min":1.9770000000000001,"5min":1.9770000000000001,"15min":1.9770000000000001},"last":0.95099999999999996}]},{"osd":3,"last update":"Mon May 22 15:27:37 2023","interfaces":[{"interface":"back","average":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"min":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.4850000000000001,"5min":1.4850000000000001,"15min":1.4850000000000001},"last":0.85799999999999998},{"interface":"front","average":{"1min":1.1100000000000001,"5min":1.1100000000000001,"15min":1.1100000000000001},"min":{"1min":0.872,"5min":0.872,"15min":0.872},"max":{"1min":1.468,"5min":1.468,"15min":1.468},"last":1.1930000000000001}]},{"osd":5,"last update":"Mon May 22 15:27:37 2023","interfaces":[{"interface":"back","average":{"1min":1.123,"5min":1.123,"15min":1.123},"min":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"max":{"1min":1.7270000000000001,"5min":1.7270000000000001,"15min":1.7270000000000001},"last":1.236},{"interface":"front","average":{"1min":1.024,"5min":1.024,"15min":1.024},"min":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"max":{"1min":1.417,"5min":1.417,"15min":1.417},"last":1.0069999999999999}]},{"osd":6,"last update":"Mon May 22 15:27:37 2023","interfaces":[{"interface":"back","average":{"1min":1.181,"5min":1.181,"15min":1.181},"min":{"1min":0.96899999999999997,"5min":0.96899999999999997,"15min":0.96899999999999997},"max":{"1min":1.6020000000000001,"5min":1.6020000000000001,"15min":1.6020000000000001},"last":1.0760000000000001},{"interface":"front","average":{"1min":1.202,"5min":1.202,"15min":1.202},"min":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"max":{"1min":2.0699999999999998,"5min":2.0699999999999998,"15min":2.0699999999999998},"last":1.119}]},{"osd":7,"last update":"Mon May 22 15:27:37 2023","interfaces":[{"interface":"back","average":{"1min":1.1919999999999999,"5min":1.1919999999999999,"15min":1.1919999999999999},"min":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.90800000000000003},"max":{"1min":1.6639999999999999,"5min":1.6639999999999999,"15min":1.6639999999999999},"last":1.1599999999999999},{"interface":"front","average":{"1min":1.2210000000000001,"5min":1.2210000000000001,"15min":1.2210000000000001},"min":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"max":{"1min":1.7410000000000001,"5min":1.7410000000000001,"15min":1.7410000000000001},"last":1.341}]}]},{"osd":3,"up_from":13,"seq":55834574870,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8752,"kb_used_data":488,"kb_used_omap":0,"kb_used_meta":8256,"kb_avail":94363088,"statfs":{"total":96636764160,"available":96627802112,"internally_reserved":0,"allocated":499712,"data_stored":219398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8454144},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 15:27:37 2023","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.82699999999999996}]},{"osd":1,"last update":"Mon May 22 15:27:37 2023","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":1.6579999999999999,"5min":1.6579999999999999,"15min":1.6579999999999999},"last":0.624}]},{"osd":2,"last update":"Mon May 22 15:27:37 2023","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":1.2490000000000001,"5min":1.2490000000000001,"15min":1.2490000000000001},"last":0.96599999999999997},{"interface":"front","average":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":1.855,"5min":1.855,"15min":1.855},"last":0.75600000000000001}]},{"osd":4,"last update":"Mon May 22 15:27:37 2023","interfaces":[{"interface":"back","average":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"min":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":1.5820000000000001,"5min":1.5820000000000001,"15min":1.5820000000000001},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.96899999999999997,"5min":0.96899999999999997,"15min":0.96899999999999997},"min":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"max":{"1min":1.8069999999999999,"5min":1.8069999999999999,"15min":1.8069999999999999},"last":1.01}]},{"osd":5,"last update":"Mon May 22 15:27:37 2023","interfaces":[{"interface":"back","average":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":0.83799999999999997},"min":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":1.0840000000000001,"5min":1.0840000000000001,"15min":1.0840000000000001},"last":1.0780000000000001},{"interface":"front","average":{"1min":1.0249999999999999,"5min":1.0249999999999999,"15min":1.0249999999999999},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":2.4820000000000002,"5min":2.4820000000000002,"15min":2.4820000000000002},"last":0.997}]},{"osd":6,"last update":"Mon May 22 15:27:37 2023","interfaces":[{"interface":"back","average":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":0.90300000000000002},"min":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"max":{"1min":1.452,"5min":1.452,"15min":1.452},"last":0.93700000000000006},{"interface":"front","average":{"1min":1.085,"5min":1.085,"15min":1.085},"min":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"max":{"1min":2.6589999999999998,"5min":2.6589999999999998,"15min":2.6589999999999998},"last":0.91400000000000003}]},{"osd":7,"last update":"Mon May 22 15:27:37 2023","interfaces":[{"interface":"back","average":{"1min":1.1459999999999999,"5min":1.1459999999999999,"15min":1.1459999999999999},"min":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"max":{"1min":2.762,"5min":2.762,"15min":2.762},"last":1.036},{"interface":"front","average":{"1min":1.0369999999999999,"5min":1.0369999999999999,"15min":1.0369999999999999},"min":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"max":{"1min":1.714,"5min":1.714,"15min":1.714},"last":0.79800000000000004}]}]},{"osd":2,"up_from":13,"seq":55834574871,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":12724,"kb_used_data":492,"kb_used_omap":0,"kb_used_meta":12224,"kb_avail":94359116,"statfs":{"total":96636764160,"available":96623734784,"internally_reserved":0,"allocated":503808,"data_stored":220346,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":12517376},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 15:27:36 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":1.4670000000000001,"5min":1.4670000000000001,"15min":1.4670000000000001},"last":0.624}]},{"osd":1,"last update":"Mon May 22 15:27:36 2023","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":1.5309999999999999,"5min":1.5309999999999999,"15min":1.5309999999999999},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":1.2210000000000001,"5min":1.2210000000000001,"15min":1.2210000000000001},"last":0.59199999999999997}]},{"osd":3,"last update":"Mon May 22 15:27:36 2023","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":1.3340000000000001,"5min":1.3340000000000001,"15min":1.3340000000000001},"last":0.442},{"interface":"front","average":{"1min":0.754,"5min":0.754,"15min":0.754},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":1.6339999999999999,"5min":1.6339999999999999,"15min":1.6339999999999999},"last":0.72199999999999998}]},{"osd":4,"last update":"Mon May 22 15:27:36 2023","interfaces":[{"interface":"back","average":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"min":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"max":{"1min":1.774,"5min":1.774,"15min":1.774},"last":0.749},{"interface":"front","average":{"1min":0.97599999999999998,"5min":0.97599999999999998,"15min":0.97599999999999998},"min":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"max":{"1min":1.802,"5min":1.802,"15min":1.802},"last":0.86499999999999999}]},{"osd":5,"last update":"Mon May 22 15:27:36 2023","interfaces":[{"interface":"back","average":{"1min":1.05,"5min":1.05,"15min":1.05},"min":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"max":{"1min":2.4569999999999999,"5min":2.4569999999999999,"15min":2.4569999999999999},"last":0.91500000000000004},{"interface":"front","average":{"1min":1,"5min":1,"15min":1},"min":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"max":{"1min":1.4670000000000001,"5min":1.4670000000000001,"15min":1.4670000000000001},"last":1.002}]},{"osd":6,"last update":"Mon May 22 15:27:36 2023","interfaces":[{"interface":"back","average":{"1min":1.093,"5min":1.093,"15min":1.093},"min":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":3.488,"5min":3.488,"15min":3.488},"last":0.93500000000000005},{"interface":"front","average":{"1min":1.151,"5min":1.151,"15min":1.151},"min":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"max":{"1min":3.7330000000000001,"5min":3.7330000000000001,"15min":3.7330000000000001},"last":0.69099999999999995}]},{"osd":7,"last update":"Mon May 22 15:27:36 2023","interfaces":[{"interface":"back","average":{"1min":1.085,"5min":1.085,"15min":1.085},"min":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"max":{"1min":2.21,"5min":2.21,"15min":2.21},"last":0.80200000000000005},{"interface":"front","average":{"1min":1.0269999999999999,"5min":1.0269999999999999,"15min":1.0269999999999999},"min":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":1.6120000000000001,"5min":1.6120000000000001,"15min":1.6120000000000001},"last":1.052}]}]},{"osd":1,"up_from":13,"seq":55834574871,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8680,"kb_used_data":480,"kb_used_omap":0,"kb_used_meta":8192,"kb_avail":94363160,"statfs":{"total":96636764160,"available":96627875840,"internally_reserved":0,"allocated":491520,"data_stored":218846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8388608},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 15:27:36 2023","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":1.3089999999999999,"5min":1.3089999999999999,"15min":1.3089999999999999},"last":1.7689999999999999},{"interface":"front","average":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":1.3799999999999999,"5min":1.3799999999999999,"15min":1.3799999999999999},"last":1.2410000000000001}]},{"osd":2,"last update":"Mon May 22 15:27:36 2023","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.63,"15min":0.63},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.98299999999999998,"5min":0.98299999999999998,"15min":0.98299999999999998},"last":1.7290000000000001},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":2.4049999999999998}]},{"osd":3,"last update":"Mon May 22 15:27:36 2023","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":1.145,"5min":1.145,"15min":1.145},"last":1.5069999999999999},{"interface":"front","average":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.0760000000000001,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":7.5620000000000003}]},{"osd":4,"last update":"Mon May 22 15:27:36 2023","interfaces":[{"interface":"back","average":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":0.85299999999999998},"min":{"1min":0.622,"5min":0.622,"15min":0.622},"max":{"1min":1.2250000000000001,"5min":1.2250000000000001,"15min":1.2250000000000001},"last":1.8169999999999999},{"interface":"front","average":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"min":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"max":{"1min":1.272,"5min":1.272,"15min":1.272},"last":1.1439999999999999}]},{"osd":5,"last update":"Mon May 22 15:27:36 2023","interfaces":[{"interface":"back","average":{"1min":0.95999999999999996,"5min":0.95999999999999996,"15min":0.95999999999999996},"min":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"max":{"1min":1.1779999999999999,"5min":1.1779999999999999,"15min":1.1779999999999999},"last":1.0669999999999999},{"interface":"front","average":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":0.96299999999999997},"min":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"max":{"1min":1.488,"5min":1.488,"15min":1.488},"last":4.4710000000000001}]},{"osd":6,"last update":"Mon May 22 15:27:36 2023","interfaces":[{"interface":"back","average":{"1min":0.93400000000000005,"5min":0.93400000000000005,"15min":0.93400000000000005},"min":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"max":{"1min":1.28,"5min":1.28,"15min":1.28},"last":14.968999999999999},{"interface":"front","average":{"1min":1.1120000000000001,"5min":1.1120000000000001,"15min":1.1120000000000001},"min":{"1min":0.621,"5min":0.621,"15min":0.621},"max":{"1min":1.8180000000000001,"5min":1.8180000000000001,"15min":1.8180000000000001},"last":13.242000000000001}]},{"osd":7,"last update":"Mon May 22 15:27:36 2023","interfaces":[{"interface":"back","average":{"1min":1.117,"5min":1.117,"15min":1.117},"min":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"max":{"1min":1.9079999999999999,"5min":1.9079999999999999,"15min":1.9079999999999999},"last":1.101},{"interface":"front","average":{"1min":1.0349999999999999,"5min":1.0349999999999999,"15min":1.0349999999999999},"min":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"max":{"1min":1.5780000000000001,"5min":1.5780000000000001,"15min":1.5780000000000001},"last":5.681}]}]},{"osd":0,"up_from":13,"seq":55834574872,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":13932,"kb_used_data":1060,"kb_used_omap":0,"kb_used_meta":12864,"kb_avail":94357908,"statfs":{"total":96636764160,"available":96622497792,"internally_reserved":0,"allocated":1085440,"data_stored":809178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":13172736},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon May 22 15:27:37 2023","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":1.1839999999999999,"5min":1.1839999999999999,"15min":1.1839999999999999},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":1.1040000000000001,"5min":1.1040000000000001,"15min":1.1040000000000001},"last":0.45300000000000001}]},{"osd":2,"last update":"Mon May 22 15:27:37 2023","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":1.23,"5min":1.23,"15min":1.23},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":1.1639999999999999,"5min":1.1639999999999999,"15min":1.1639999999999999},"last":0.51200000000000001}]},{"osd":3,"last update":"Mon May 22 15:27:37 2023","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":1.2410000000000001,"5min":1.2410000000000001,"15min":1.2410000000000001},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":1.119,"5min":1.119,"15min":1.119},"last":0.56999999999999995}]},{"osd":4,"last update":"Mon May 22 15:27:37 2023","interfaces":[{"interface":"back","average":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"min":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":1.2969999999999999,"5min":1.2969999999999999,"15min":1.2969999999999999},"last":0.83399999999999996},{"interface":"front","average":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"min":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"max":{"1min":1.4219999999999999,"5min":1.4219999999999999,"15min":1.4219999999999999},"last":0.72199999999999998}]},{"osd":5,"last update":"Mon May 22 15:27:37 2023","interfaces":[{"interface":"back","average":{"1min":0.999,"5min":0.999,"15min":0.999},"min":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":1.4370000000000001,"5min":1.4370000000000001,"15min":1.4370000000000001},"last":0.73399999999999999},{"interface":"front","average":{"1min":0.95099999999999996,"5min":0.95099999999999996,"15min":0.95099999999999996},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.349,"5min":1.349,"15min":1.349},"last":0.87}]},{"osd":6,"last update":"Mon May 22 15:27:37 2023","interfaces":[{"interface":"back","average":{"1min":0.94599999999999995,"5min":0.94599999999999995,"15min":0.94599999999999995},"min":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"max":{"1min":1.2769999999999999,"5min":1.2769999999999999,"15min":1.2769999999999999},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.96899999999999997,"5min":0.96899999999999997,"15min":0.96899999999999997},"min":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":1.492,"5min":1.492,"15min":1.492},"last":0.80000000000000004}]},{"osd":7,"last update":"Mon May 22 15:27:37 2023","interfaces":[{"interface":"back","average":{"1min":1.04,"5min":1.04,"15min":1.04},"min":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":1.581,"5min":1.581,"15min":1.581},"last":0.69299999999999995},{"interface":"front","average":{"1min":0.98099999999999998,"5min":0.98099999999999998,"15min":0.98099999999999998},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.448,"5min":1.448,"15min":1.448},"last":0.96299999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1570,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-05-22T15:28:21.225 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-05-22T15:28:21.225 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2023-05-22T15:28:21.394 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.387+0000 7f13f2f28640 1 Processor -- start 2023-05-22T15:28:21.395 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.387+0000 7f13f2f28640 1 -- start start 2023-05-22T15:28:21.395 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.387+0000 7f13f2f28640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f13ec158210 0x7f13ec159160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:21.396 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.387+0000 7f13f2f28640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f13ec15bcd0 0x7f13ec15c120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:21.396 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.387+0000 7f13f2f28640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f13ec0c8660 0x7f13ec0c8ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:21.396 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.387+0000 7f13f2f28640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13ec0c9020 con 0x7f13ec158210 2023-05-22T15:28:21.396 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.387+0000 7f13f2f28640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13ec158cd0 con 0x7f13ec0c8660 2023-05-22T15:28:21.396 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.387+0000 7f13f2f28640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13ec1596a0 con 0x7f13ec15bcd0 2023-05-22T15:28:21.397 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.387+0000 7f13f0cc4640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f13ec158210 0x7f13ec159160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:21.397 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.387+0000 7f13ebfff640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f13ec15bcd0 0x7f13ec15c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:21.397 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.387+0000 7f13f0cc4640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f13ec158210 0x7f13ec159160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52008/0 (socket says 172.21.15.16:52008) 2023-05-22T15:28:21.397 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.387+0000 7f13f0cc4640 1 -- 172.21.15.16:0/1243730089 learned_addr learned my addr 172.21.15.16:0/1243730089 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:28:21.398 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.387+0000 7f13f14c5640 1 --2- 172.21.15.16:0/1243730089 >> v2:172.21.15.62:3300/0 conn(0x7f13ec0c8660 0x7f13ec0c8ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:21.398 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.391+0000 7f13f0cc4640 1 -- 172.21.15.16:0/1243730089 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f13ec15bcd0 msgr2=0x7f13ec15c120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:28:21.398 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.391+0000 7f13f0cc4640 1 --2- 172.21.15.16:0/1243730089 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f13ec15bcd0 0x7f13ec15c120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:21.398 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.391+0000 7f13f0cc4640 1 -- 172.21.15.16:0/1243730089 >> v2:172.21.15.62:3300/0 conn(0x7f13ec0c8660 msgr2=0x7f13ec0c8ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:28:21.398 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.391+0000 7f13f0cc4640 1 --2- 172.21.15.16:0/1243730089 >> v2:172.21.15.62:3300/0 conn(0x7f13ec0c8660 0x7f13ec0c8ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:21.398 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.391+0000 7f13f0cc4640 1 -- 172.21.15.16:0/1243730089 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13ec15c790 con 0x7f13ec158210 2023-05-22T15:28:21.399 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.391+0000 7f13f0cc4640 1 --2- 172.21.15.16:0/1243730089 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f13ec158210 0x7f13ec159160 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f13ec0c9320 tx=0x7f13dc01c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f876b4e510e54a31 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:28:21.399 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.391+0000 7f13eb7fe640 1 -- 172.21.15.16:0/1243730089 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13dc0a5710 con 0x7f13ec158210 2023-05-22T15:28:21.399 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.391+0000 7f13eb7fe640 1 -- 172.21.15.16:0/1243730089 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13dc0a5870 con 0x7f13ec158210 2023-05-22T15:28:21.399 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.391+0000 7f13eb7fe640 1 -- 172.21.15.16:0/1243730089 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13dc0a5c00 con 0x7f13ec158210 2023-05-22T15:28:21.400 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.391+0000 7f13f2f28640 1 -- 172.21.15.16:0/1243730089 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f13ec158210 msgr2=0x7f13ec159160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:21.400 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.391+0000 7f13f2f28640 1 --2- 172.21.15.16:0/1243730089 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f13ec158210 0x7f13ec159160 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f13ec0c9320 tx=0x7f13dc01c910 comp rx=0 tx=0).stop 2023-05-22T15:28:21.401 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.391+0000 7f13f2f28640 1 -- 172.21.15.16:0/1243730089 shutdown_connections 2023-05-22T15:28:21.401 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.391+0000 7f13f2f28640 1 --2- 172.21.15.16:0/1243730089 >> v2:172.21.15.62:3300/0 conn(0x7f13ec0c8660 0x7f13ec0c8ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:21.401 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.391+0000 7f13f2f28640 1 --2- 172.21.15.16:0/1243730089 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f13ec15bcd0 0x7f13ec15c120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:21.401 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.391+0000 7f13f2f28640 1 --2- 172.21.15.16:0/1243730089 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f13ec158210 0x7f13ec159160 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:21.401 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.391+0000 7f13f2f28640 1 -- 172.21.15.16:0/1243730089 >> 172.21.15.16:0/1243730089 conn(0x7f13ec069720 msgr2=0x7f13ec069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:28:21.402 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.391+0000 7f13f2f28640 1 -- 172.21.15.16:0/1243730089 shutdown_connections 2023-05-22T15:28:21.402 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.391+0000 7f13f2f28640 1 -- 172.21.15.16:0/1243730089 wait complete. 2023-05-22T15:28:21.403 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.395+0000 7f13f2f28640 1 Processor -- start 2023-05-22T15:28:21.404 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.395+0000 7f13f2f28640 1 -- start start 2023-05-22T15:28:21.404 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.395+0000 7f13f2f28640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f13ec0c8660 0x7f13ec0c5400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:21.404 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.395+0000 7f13f2f28640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f13ec158210 0x7f13ec0c7250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:21.404 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.395+0000 7f13f2f28640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f13ec15bcd0 0x7f13ec150730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:21.404 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.395+0000 7f13f2f28640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13ec0c5940 con 0x7f13ec15bcd0 2023-05-22T15:28:21.404 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.395+0000 7f13f2f28640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13ec14cc80 con 0x7f13ec158210 2023-05-22T15:28:21.405 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.395+0000 7f13f2f28640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13ec14cf60 con 0x7f13ec0c8660 2023-05-22T15:28:21.405 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.395+0000 7f13f14c5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f13ec15bcd0 0x7f13ec150730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:21.405 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.395+0000 7f13f14c5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f13ec15bcd0 0x7f13ec150730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52020/0 (socket says 172.21.15.16:52020) 2023-05-22T15:28:21.405 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.395+0000 7f13f14c5640 1 -- 172.21.15.16:0/619382642 learned_addr learned my addr 172.21.15.16:0/619382642 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:28:21.405 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.395+0000 7f13f0cc4640 1 --2- 172.21.15.16:0/619382642 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f13ec0c8660 0x7f13ec0c5400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:21.405 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.395+0000 7f13ebfff640 1 --2- 172.21.15.16:0/619382642 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f13ec158210 0x7f13ec0c7250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:21.405 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.395+0000 7f13f14c5640 1 -- 172.21.15.16:0/619382642 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f13ec0c8660 msgr2=0x7f13ec0c5400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:21.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.395+0000 7f13f14c5640 1 --2- 172.21.15.16:0/619382642 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f13ec0c8660 0x7f13ec0c5400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:21.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.395+0000 7f13f14c5640 1 -- 172.21.15.16:0/619382642 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f13ec158210 msgr2=0x7f13ec0c7250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:21.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.395+0000 7f13f14c5640 1 --2- 172.21.15.16:0/619382642 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f13ec158210 0x7f13ec0c7250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:21.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.395+0000 7f13f14c5640 1 -- 172.21.15.16:0/619382642 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13dc009b30 con 0x7f13ec15bcd0 2023-05-22T15:28:21.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.395+0000 7f13f0cc4640 1 --2- 172.21.15.16:0/619382642 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f13ec0c8660 0x7f13ec0c5400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T15:28:21.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.395+0000 7f13f14c5640 1 --2- 172.21.15.16:0/619382642 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f13ec15bcd0 0x7f13ec150730 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f13e400e930 tx=0x7f13e400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:28:21.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.395+0000 7f13e97fa640 1 -- 172.21.15.16:0/619382642 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13e40096b0 con 0x7f13ec15bcd0 2023-05-22T15:28:21.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.395+0000 7f13e97fa640 1 -- 172.21.15.16:0/619382642 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13e400c9a0 con 0x7f13ec15bcd0 2023-05-22T15:28:21.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.395+0000 7f13e97fa640 1 -- 172.21.15.16:0/619382642 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13e400cd30 con 0x7f13ec15bcd0 2023-05-22T15:28:21.408 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.395+0000 7f13f2f28640 1 -- 172.21.15.16:0/619382642 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13ec1585e0 con 0x7f13ec15bcd0 2023-05-22T15:28:21.408 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.395+0000 7f13f2f28640 1 -- 172.21.15.16:0/619382642 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13ec0cd980 con 0x7f13ec15bcd0 2023-05-22T15:28:21.408 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.399+0000 7f13f2f28640 1 -- 172.21.15.16:0/619382642 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13ec0ca690 con 0x7f13ec15bcd0 2023-05-22T15:28:21.408 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.399+0000 7f13e97fa640 1 -- 172.21.15.16:0/619382642 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7f13e4013470 con 0x7f13ec15bcd0 2023-05-22T15:28:21.408 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.399+0000 7f13e97fa640 1 --2- 172.21.15.16:0/619382642 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f13d8065380 0x7f13d8067830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:21.409 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.399+0000 7f13e97fa640 1 -- 172.21.15.16:0/619382642 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f13e4082f10 con 0x7f13ec15bcd0 2023-05-22T15:28:21.409 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.399+0000 7f13f0cc4640 1 --2- 172.21.15.16:0/619382642 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f13d8065380 0x7f13d8067830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:21.409 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.399+0000 7f13f0cc4640 1 --2- 172.21.15.16:0/619382642 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f13d8065380 0x7f13d8067830 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f13dc0ae000 tx=0x7f13dc0a7070 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:28:21.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.403+0000 7f13e97fa640 1 -- 172.21.15.16:0/619382642 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f13e4050b40 con 0x7f13ec15bcd0 2023-05-22T15:28:21.610 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.603+0000 7f13f2f28640 1 -- 172.21.15.16:0/619382642 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f13ec04dbe0 con 0x7f13ec15bcd0 2023-05-22T15:28:21.619 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.611+0000 7f13e97fa640 1 -- 172.21.15.16:0/619382642 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v15) v1 ==== 190+0+0 (secure 0 0 0) 0x7f13e4054160 con 0x7f13ec15bcd0 2023-05-22T15:28:21.620 INFO:teuthology.orchestra.run.smithi016.stderr:updated caps for client.0 2023-05-22T15:28:21.621 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.615+0000 7f13f2f28640 1 -- 172.21.15.16:0/619382642 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f13d8065380 msgr2=0x7f13d8067830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:21.621 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.615+0000 7f13f2f28640 1 --2- 172.21.15.16:0/619382642 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f13d8065380 0x7f13d8067830 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f13dc0ae000 tx=0x7f13dc0a7070 comp rx=0 tx=0).stop 2023-05-22T15:28:21.622 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.615+0000 7f13f2f28640 1 -- 172.21.15.16:0/619382642 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f13ec15bcd0 msgr2=0x7f13ec150730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:21.622 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.615+0000 7f13f2f28640 1 --2- 172.21.15.16:0/619382642 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f13ec15bcd0 0x7f13ec150730 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f13e400e930 tx=0x7f13e400edf0 comp rx=0 tx=0).stop 2023-05-22T15:28:21.622 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.615+0000 7f13f2f28640 1 -- 172.21.15.16:0/619382642 shutdown_connections 2023-05-22T15:28:21.622 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.615+0000 7f13f2f28640 1 --2- 172.21.15.16:0/619382642 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f13ec15bcd0 0x7f13ec150730 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:21.622 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.615+0000 7f13f2f28640 1 --2- 172.21.15.16:0/619382642 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f13ec158210 0x7f13ec0c7250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:21.622 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.615+0000 7f13f2f28640 1 --2- 172.21.15.16:0/619382642 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f13d8065380 0x7f13d8067830 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:21.623 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.615+0000 7f13f2f28640 1 --2- 172.21.15.16:0/619382642 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f13ec0c8660 0x7f13ec0c5400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:21.623 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.615+0000 7f13f2f28640 1 -- 172.21.15.16:0/619382642 >> 172.21.15.16:0/619382642 conn(0x7f13ec069720 msgr2=0x7f13ec0cc970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:28:21.623 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.615+0000 7f13f2f28640 1 -- 172.21.15.16:0/619382642 shutdown_connections 2023-05-22T15:28:21.623 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.615+0000 7f13f2f28640 1 -- 172.21.15.16:0/619382642 wait complete. 2023-05-22T15:28:21.636 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-05-22T15:28:21.756 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.751+0000 7f90aee71640 1 Processor -- start 2023-05-22T15:28:21.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.751+0000 7f90aee71640 1 -- start start 2023-05-22T15:28:21.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.751+0000 7f90aee71640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f90a8158290 0x7f90a80c8880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:21.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.751+0000 7f90aee71640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90a80c92b0 con 0x7f90a80ce6c0 2023-05-22T15:28:21.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.751+0000 7f90aee71640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90a80cc060 con 0x7f90a8158660 2023-05-22T15:28:21.758 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.751+0000 7f90aee71640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90a80cc340 con 0x7f90a8158290 2023-05-22T15:28:21.758 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.751+0000 7f90ad40e640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f90a80ce6c0 0x7f90a806f630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:33794/0 (socket says 172.21.15.16:33794) 2023-05-22T15:28:21.759 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.751+0000 7f90acc0d640 1 --2- 172.21.15.16:0/3629568635 >> v2:172.21.15.62:3300/0 conn(0x7f90a8158290 0x7f90a80c8880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:21.759 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.751+0000 7f90ad40e640 1 -- 172.21.15.16:0/3629568635 learned_addr learned my addr 172.21.15.16:0/3629568635 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:28:21.759 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.751+0000 7f90a77fe640 1 -- 172.21.15.16:0/3629568635 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1248138013 0 0) 0x7f90a80c92b0 con 0x7f90a80ce6c0 2023-05-22T15:28:21.759 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.751+0000 7f90a77fe640 1 -- 172.21.15.16:0/3629568635 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9090003530 con 0x7f90a80ce6c0 2023-05-22T15:28:21.760 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.751+0000 7f90acc0d640 1 -- 172.21.15.16:0/3629568635 >> v1:172.21.15.62:6789/0 conn(0x7f90a8158660 legacy=0x7f90a80cdfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:21.760 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.751+0000 7f90acc0d640 1 -- 172.21.15.16:0/3629568635 >> v1:172.21.15.16:6789/0 conn(0x7f90a80ce6c0 legacy=0x7f90a806f630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:21.760 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.751+0000 7f90acc0d640 1 -- 172.21.15.16:0/3629568635 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90a80cc620 con 0x7f90a8158290 2023-05-22T15:28:21.762 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.755+0000 7f90acc0d640 1 --2- 172.21.15.16:0/3629568635 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f90a8158290 0x7f90a80c8880 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f90980097c0 tx=0x7f909801c450 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1d84438bb6cc8a66 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:28:21.762 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.755+0000 7f90a77fe640 1 -- 172.21.15.16:0/3629568635 <== mon.1 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90980ab670 con 0x7f90a8158290 2023-05-22T15:28:21.762 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.755+0000 7f90a77fe640 1 -- 172.21.15.16:0/3629568635 <== mon.1 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90980ab7d0 con 0x7f90a8158290 2023-05-22T15:28:21.762 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.755+0000 7f90a77fe640 1 -- 172.21.15.16:0/3629568635 <== mon.1 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90980abb40 con 0x7f90a8158290 2023-05-22T15:28:21.763 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.755+0000 7f90aee71640 1 -- 172.21.15.16:0/3629568635 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f90a8158290 msgr2=0x7f90a80c8880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:21.763 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.755+0000 7f90aee71640 1 --2- 172.21.15.16:0/3629568635 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f90a8158290 0x7f90a80c8880 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f90980097c0 tx=0x7f909801c450 comp rx=0 tx=0).stop 2023-05-22T15:28:21.763 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.755+0000 7f90aee71640 1 -- 172.21.15.16:0/3629568635 shutdown_connections 2023-05-22T15:28:21.763 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.755+0000 7f90aee71640 1 --2- 172.21.15.16:0/3629568635 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f90a8158290 0x7f90a80c8880 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:21.763 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.755+0000 7f90aee71640 1 -- 172.21.15.16:0/3629568635 >> 172.21.15.16:0/3629568635 conn(0x7f90a8069720 msgr2=0x7f90a8069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:28:21.764 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.755+0000 7f90aee71640 1 -- 172.21.15.16:0/3629568635 shutdown_connections 2023-05-22T15:28:21.764 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.755+0000 7f90aee71640 1 -- 172.21.15.16:0/3629568635 wait complete. 2023-05-22T15:28:21.765 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.755+0000 7f90aee71640 1 Processor -- start 2023-05-22T15:28:21.766 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.755+0000 7f90aee71640 1 -- start start 2023-05-22T15:28:21.766 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.755+0000 7f90aee71640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f90a80ce6c0 0x7f90a80b0eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:21.766 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.755+0000 7f90aee71640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f90a8158290 0x7f90a80b13f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:21.766 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.755+0000 7f90aee71640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f90a8158660 0x7f90a80b1930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:21.766 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.755+0000 7f90aee71640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90a81f2630 con 0x7f90a80ce6c0 2023-05-22T15:28:21.766 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.755+0000 7f90aee71640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90a8158a30 con 0x7f90a8158290 2023-05-22T15:28:21.766 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.755+0000 7f90aee71640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90a8159890 con 0x7f90a8158660 2023-05-22T15:28:21.767 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.755+0000 7f90acc0d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f90a80ce6c0 0x7f90a80b0eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:21.767 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.759+0000 7f90acc0d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f90a80ce6c0 0x7f90a80b0eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52032/0 (socket says 172.21.15.16:52032) 2023-05-22T15:28:21.767 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.759+0000 7f90acc0d640 1 -- 172.21.15.16:0/3937023392 learned_addr learned my addr 172.21.15.16:0/3937023392 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:28:21.767 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.759+0000 7f90ad40e640 1 --2- 172.21.15.16:0/3937023392 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f90a8158660 0x7f90a80b1930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:21.767 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.759+0000 7f90a7fff640 1 --2- 172.21.15.16:0/3937023392 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f90a8158290 0x7f90a80b13f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:21.767 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.759+0000 7f90acc0d640 1 -- 172.21.15.16:0/3937023392 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f90a8158660 msgr2=0x7f90a80b1930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:21.767 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.759+0000 7f90acc0d640 1 --2- 172.21.15.16:0/3937023392 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f90a8158660 0x7f90a80b1930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:21.767 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.759+0000 7f90acc0d640 1 -- 172.21.15.16:0/3937023392 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f90a8158290 msgr2=0x7f90a80b13f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:21.768 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.759+0000 7f90acc0d640 1 --2- 172.21.15.16:0/3937023392 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f90a8158290 0x7f90a80b13f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:21.768 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.759+0000 7f90acc0d640 1 -- 172.21.15.16:0/3937023392 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9098009420 con 0x7f90a80ce6c0 2023-05-22T15:28:21.768 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.759+0000 7f90acc0d640 1 --2- 172.21.15.16:0/3937023392 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f90a80ce6c0 0x7f90a80b0eb0 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f9098000c10 tx=0x7f90980191f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:28:21.768 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.759+0000 7f90a57fa640 1 -- 172.21.15.16:0/3937023392 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90980ab940 con 0x7f90a80ce6c0 2023-05-22T15:28:21.768 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.759+0000 7f90a57fa640 1 -- 172.21.15.16:0/3937023392 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90980abaa0 con 0x7f90a80ce6c0 2023-05-22T15:28:21.768 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.759+0000 7f90aee71640 1 -- 172.21.15.16:0/3937023392 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90a80ccfb0 con 0x7f90a80ce6c0 2023-05-22T15:28:21.768 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.759+0000 7f90a57fa640 1 -- 172.21.15.16:0/3937023392 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90980abe10 con 0x7f90a80ce6c0 2023-05-22T15:28:21.769 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.759+0000 7f90aee71640 1 -- 172.21.15.16:0/3937023392 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90a80cce20 con 0x7f90a80ce6c0 2023-05-22T15:28:21.769 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.759+0000 7f90aee71640 1 -- 172.21.15.16:0/3937023392 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90a80c8dc0 con 0x7f90a80ce6c0 2023-05-22T15:28:21.770 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.759+0000 7f90a57fa640 1 -- 172.21.15.16:0/3937023392 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7f9098003620 con 0x7f90a80ce6c0 2023-05-22T15:28:21.770 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.759+0000 7f90a57fa640 1 --2- 172.21.15.16:0/3937023392 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f9094065380 0x7f9094067830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:21.770 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.759+0000 7f90a7fff640 1 --2- 172.21.15.16:0/3937023392 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f9094065380 0x7f9094067830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:21.770 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.759+0000 7f90a57fa640 1 -- 172.21.15.16:0/3937023392 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f909811cc00 con 0x7f90a80ce6c0 2023-05-22T15:28:21.770 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.759+0000 7f90a7fff640 1 --2- 172.21.15.16:0/3937023392 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f9094065380 0x7f9094067830 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f90a80b4190 tx=0x7f909c009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:28:21.771 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.763+0000 7f90a57fa640 1 -- 172.21.15.16:0/3937023392 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f9098123080 con 0x7f90a80ce6c0 2023-05-22T15:28:21.951 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.943+0000 7f90aee71640 1 -- 172.21.15.16:0/3937023392 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f90a8158ce0 con 0x7f90a80ce6c0 2023-05-22T15:28:21.952 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.943+0000 7f90a57fa640 1 -- 172.21.15.16:0/3937023392 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) v1 ==== 94+0+4648 (secure 0 0 0) 0x7f90980ea7f0 con 0x7f90a80ce6c0 2023-05-22T15:28:21.952 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:28:21.952 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":14,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4904,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6833/3153072235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":3153072235},{"type":"v1","addr":"172.21.15.62:6833","nonce":3153072235}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":5164,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/2783822861","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":2783822861},{"type":"v1","addr":"172.21.15.62:6835","nonce":2783822861}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":5571,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/1361634240","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1361634240},{"type":"v1","addr":"172.21.15.16:6835","nonce":1361634240}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":14,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-05-22T15:28:13.320259+0000","modified":"2023-05-22T15:28:15.336631+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5574},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5574":{"gid":5574,"name":"c","rank":0,"incarnation":12,"state":"up:active","state_seq":3,"addr":"172.21.15.16:6837/1033866899","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":1033866899},{"type":"v1","addr":"172.21.15.16:6837","nonce":1033866899}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-05-22T15:28:21.953 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 14 2023-05-22T15:28:21.954 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.947+0000 7f90aee71640 1 -- 172.21.15.16:0/3937023392 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f9094065380 msgr2=0x7f9094067830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:21.954 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.947+0000 7f90aee71640 1 --2- 172.21.15.16:0/3937023392 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f9094065380 0x7f9094067830 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f90a80b4190 tx=0x7f909c009040 comp rx=0 tx=0).stop 2023-05-22T15:28:21.954 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.947+0000 7f90aee71640 1 -- 172.21.15.16:0/3937023392 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f90a80ce6c0 msgr2=0x7f90a80b0eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:21.955 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.947+0000 7f90aee71640 1 --2- 172.21.15.16:0/3937023392 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f90a80ce6c0 0x7f90a80b0eb0 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f9098000c10 tx=0x7f90980191f0 comp rx=0 tx=0).stop 2023-05-22T15:28:21.955 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.947+0000 7f90aee71640 1 -- 172.21.15.16:0/3937023392 shutdown_connections 2023-05-22T15:28:21.955 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.947+0000 7f90aee71640 1 --2- 172.21.15.16:0/3937023392 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f9094065380 0x7f9094067830 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:21.955 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.947+0000 7f90aee71640 1 --2- 172.21.15.16:0/3937023392 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f90a8158660 0x7f90a80b1930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:21.955 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.947+0000 7f90aee71640 1 --2- 172.21.15.16:0/3937023392 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f90a8158290 0x7f90a80b13f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:21.955 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.947+0000 7f90aee71640 1 --2- 172.21.15.16:0/3937023392 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f90a80ce6c0 0x7f90a80b0eb0 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:21.955 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.947+0000 7f90aee71640 1 -- 172.21.15.16:0/3937023392 >> 172.21.15.16:0/3937023392 conn(0x7f90a8069720 msgr2=0x7f90a80cdfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:28:21.955 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.947+0000 7f90aee71640 1 -- 172.21.15.16:0/3937023392 shutdown_connections 2023-05-22T15:28:21.956 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:21.947+0000 7f90aee71640 1 -- 172.21.15.16:0/3937023392 wait complete. 2023-05-22T15:28:22.372 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 14, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-05-22T15:28:13.320259+0000', 'modified': '2023-05-22T15:28:15.336631+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5574}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5574': {'gid': 5574, 'name': 'c', 'rank': 0, 'incarnation': 12, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.16:6837/1033866899', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6836', 'nonce': 1033866899}, {'type': 'v1', 'addr': '172.21.15.16:6837', 'nonce': 1033866899}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [7], 'metadata_pool': 6, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-05-22T15:28:22.373 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-05-22T15:28:22.373 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2023-05-22T15:28:22.509 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.503+0000 7fe010ba6640 1 Processor -- start 2023-05-22T15:28:22.509 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.503+0000 7fe010ba6640 1 -- start start 2023-05-22T15:28:22.510 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.503+0000 7fe010ba6640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe00c1531b0 0x7fe00c0afb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:22.510 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.503+0000 7fe010ba6640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe00c159a40 0x7fe00c0c8ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:22.511 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.503+0000 7fe010ba6640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe00c15a0a0 con 0x7fe00c0b0080 2023-05-22T15:28:22.511 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.503+0000 7fe010ba6640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe00c0b0520 con 0x7fe00c1531b0 2023-05-22T15:28:22.511 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.503+0000 7fe010ba6640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe00c159e50 con 0x7fe00c159a40 2023-05-22T15:28:22.511 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.503+0000 7fe00a59c640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe00c1531b0 0x7fe00c0afb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:22.511 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.503+0000 7fe00a59c640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe00c1531b0 0x7fe00c0afb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52038/0 (socket says 172.21.15.16:52038) 2023-05-22T15:28:22.511 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.503+0000 7fe00a59c640 1 -- 172.21.15.16:0/201543454 learned_addr learned my addr 172.21.15.16:0/201543454 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:28:22.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.503+0000 7fe00ad9d640 1 --2- 172.21.15.16:0/201543454 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe00c159a40 0x7fe00c0c8ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:22.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.503+0000 7fe00a59c640 1 -- 172.21.15.16:0/201543454 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe00c159a40 msgr2=0x7fe00c0c8ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:28:22.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.503+0000 7fe00a59c640 1 --2- 172.21.15.16:0/201543454 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe00c159a40 0x7fe00c0c8ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:22.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.503+0000 7fe00a59c640 1 -- 172.21.15.16:0/201543454 >> v1:172.21.15.62:6789/0 conn(0x7fe00c0b0080 legacy=0x7fe00c159330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:28:22.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.503+0000 7fe00a59c640 1 -- 172.21.15.16:0/201543454 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe00c153580 con 0x7fe00c1531b0 2023-05-22T15:28:22.513 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.507+0000 7fe00a59c640 1 --2- 172.21.15.16:0/201543454 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe00c1531b0 0x7fe00c0afb40 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7fe00c0b0790 tx=0x7fdffc01c4c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27f75d5722452c71 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:28:22.513 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.507+0000 7fe00959a640 1 -- 172.21.15.16:0/201543454 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdffc0ab600 con 0x7fe00c1531b0 2023-05-22T15:28:22.513 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.507+0000 7fe00959a640 1 -- 172.21.15.16:0/201543454 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdffc0a7050 con 0x7fe00c1531b0 2023-05-22T15:28:22.513 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.507+0000 7fe00959a640 1 -- 172.21.15.16:0/201543454 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdffc0ab980 con 0x7fe00c1531b0 2023-05-22T15:28:22.514 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.507+0000 7fe010ba6640 1 -- 172.21.15.16:0/201543454 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe00c1531b0 msgr2=0x7fe00c0afb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:22.514 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.507+0000 7fe010ba6640 1 --2- 172.21.15.16:0/201543454 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe00c1531b0 0x7fe00c0afb40 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7fe00c0b0790 tx=0x7fdffc01c4c0 comp rx=0 tx=0).stop 2023-05-22T15:28:22.514 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.507+0000 7fe010ba6640 1 -- 172.21.15.16:0/201543454 shutdown_connections 2023-05-22T15:28:22.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.507+0000 7fe010ba6640 1 --2- 172.21.15.16:0/201543454 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe00c159a40 0x7fe00c0c8ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:22.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.507+0000 7fe010ba6640 1 --2- 172.21.15.16:0/201543454 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe00c1531b0 0x7fe00c0afb40 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:22.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.507+0000 7fe010ba6640 1 -- 172.21.15.16:0/201543454 >> 172.21.15.16:0/201543454 conn(0x7fe00c069720 msgr2=0x7fe00c069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:28:22.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.507+0000 7fe010ba6640 1 -- 172.21.15.16:0/201543454 shutdown_connections 2023-05-22T15:28:22.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.507+0000 7fe010ba6640 1 -- 172.21.15.16:0/201543454 wait complete. 2023-05-22T15:28:22.517 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.507+0000 7fe010ba6640 1 Processor -- start 2023-05-22T15:28:22.517 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.507+0000 7fe010ba6640 1 -- start start 2023-05-22T15:28:22.517 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.507+0000 7fe010ba6640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe00c0b0080 0x7fe00c144480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:22.517 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.507+0000 7fe010ba6640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe00c1531b0 0x7fe00c14d6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:22.518 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.507+0000 7fe010ba6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe00c159a40 0x7fe00c15b3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:22.518 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.507+0000 7fe010ba6640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe00c1449c0 con 0x7fe00c159a40 2023-05-22T15:28:22.518 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.507+0000 7fe010ba6640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe00c150450 con 0x7fe00c1531b0 2023-05-22T15:28:22.518 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.507+0000 7fe010ba6640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe00c14f840 con 0x7fe00c0b0080 2023-05-22T15:28:22.518 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.507+0000 7fe00ad9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe00c159a40 0x7fe00c15b3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:22.519 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.507+0000 7fe00ad9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe00c159a40 0x7fe00c15b3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52048/0 (socket says 172.21.15.16:52048) 2023-05-22T15:28:22.519 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.507+0000 7fe00ad9d640 1 -- 172.21.15.16:0/4214088620 learned_addr learned my addr 172.21.15.16:0/4214088620 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:28:22.519 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.511+0000 7fe00a59c640 1 --2- 172.21.15.16:0/4214088620 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe00c0b0080 0x7fe00c144480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:22.519 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.511+0000 7fe009d9b640 1 --2- 172.21.15.16:0/4214088620 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe00c1531b0 0x7fe00c14d6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:22.519 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.511+0000 7fe00ad9d640 1 -- 172.21.15.16:0/4214088620 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe00c0b0080 msgr2=0x7fe00c144480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:22.519 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.511+0000 7fe00ad9d640 1 --2- 172.21.15.16:0/4214088620 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe00c0b0080 0x7fe00c144480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:22.519 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.511+0000 7fe00ad9d640 1 -- 172.21.15.16:0/4214088620 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe00c1531b0 msgr2=0x7fe00c14d6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:22.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.511+0000 7fe00ad9d640 1 --2- 172.21.15.16:0/4214088620 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe00c1531b0 0x7fe00c14d6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:22.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.511+0000 7fe00ad9d640 1 -- 172.21.15.16:0/4214088620 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdffc0094d0 con 0x7fe00c159a40 2023-05-22T15:28:22.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.511+0000 7fe009d9b640 1 --2- 172.21.15.16:0/4214088620 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe00c1531b0 0x7fe00c14d6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:28:22.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.511+0000 7fe00ad9d640 1 --2- 172.21.15.16:0/4214088620 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe00c159a40 0x7fe00c15b3b0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7fe004007e60 tx=0x7fe00400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:28:22.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.511+0000 7fdffaffd640 1 -- 172.21.15.16:0/4214088620 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe004014420 con 0x7fe00c159a40 2023-05-22T15:28:22.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.511+0000 7fdffaffd640 1 -- 172.21.15.16:0/4214088620 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0040145a0 con 0x7fe00c159a40 2023-05-22T15:28:22.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.511+0000 7fe010ba6640 1 -- 172.21.15.16:0/4214088620 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe00c15d0c0 con 0x7fe00c159a40 2023-05-22T15:28:22.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.511+0000 7fdffaffd640 1 -- 172.21.15.16:0/4214088620 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe004014950 con 0x7fe00c159a40 2023-05-22T15:28:22.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.511+0000 7fe010ba6640 1 -- 172.21.15.16:0/4214088620 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe00c15cde0 con 0x7fe00c159a40 2023-05-22T15:28:22.527 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.511+0000 7fdffaffd640 1 -- 172.21.15.16:0/4214088620 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7fe004014ab0 con 0x7fe00c159a40 2023-05-22T15:28:22.527 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.511+0000 7fdffaffd640 1 --2- 172.21.15.16:0/4214088620 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fdfec065380 0x7fdfec067830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:22.527 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.511+0000 7fe00a59c640 1 --2- 172.21.15.16:0/4214088620 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fdfec065380 0x7fdfec067830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:22.527 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.511+0000 7fdffaffd640 1 -- 172.21.15.16:0/4214088620 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6382+0+0 (secure 0 0 0) 0x7fe004083290 con 0x7fe00c159a40 2023-05-22T15:28:22.527 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.511+0000 7fe00a59c640 1 --2- 172.21.15.16:0/4214088620 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fdfec065380 0x7fdfec067830 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fdffc0ad000 tx=0x7fdffc0a6000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:28:22.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.511+0000 7fe013cc8000 1 Processor -- start 2023-05-22T15:28:22.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.511+0000 7fe013cc8000 1 -- start start 2023-05-22T15:28:22.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.515+0000 7fe013cc8000 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x55f64bb86fd0 0x55f64bdf9c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:22.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.515+0000 7fe013cc8000 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f64bdfa170 0x55f64bdfa540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:22.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.515+0000 7fe013cc8000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f64bdada10 0x55f64bdadde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:22.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.515+0000 7fe013cc8000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f64bc77bd0 con 0x55f64bdada10 2023-05-22T15:28:22.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.515+0000 7fe013cc8000 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f64bb94dd0 con 0x55f64bdfa170 2023-05-22T15:28:22.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.515+0000 7fe013cc8000 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x55f64bc05640 con 0x55f64bb86fd0 2023-05-22T15:28:22.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.515+0000 7fe00ad9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f64bdada10 0x55f64bdadde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:22.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.515+0000 7fe00ad9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f64bdada10 0x55f64bdadde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52050/0 (socket says 172.21.15.16:52050) 2023-05-22T15:28:22.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.515+0000 7fe00ad9d640 1 -- 172.21.15.16:0/739356195 learned_addr learned my addr 172.21.15.16:0/739356195 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:28:22.530 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.515+0000 7fe00a59c640 1 --2- 172.21.15.16:0/739356195 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x55f64bb86fd0 0x55f64bdf9c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:22.530 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.515+0000 7fe009d9b640 1 --2- 172.21.15.16:0/739356195 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f64bdfa170 0x55f64bdfa540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:22.530 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.515+0000 7fe00ad9d640 1 -- 172.21.15.16:0/739356195 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x55f64bb86fd0 msgr2=0x55f64bdf9c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:28:22.530 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.515+0000 7fe00ad9d640 1 --2- 172.21.15.16:0/739356195 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x55f64bb86fd0 0x55f64bdf9c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:22.530 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.515+0000 7fe00ad9d640 1 -- 172.21.15.16:0/739356195 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f64bdfa170 msgr2=0x55f64bdfa540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:28:22.530 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.515+0000 7fe00ad9d640 1 --2- 172.21.15.16:0/739356195 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f64bdfa170 0x55f64bdfa540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:22.530 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.515+0000 7fe00ad9d640 1 -- 172.21.15.16:0/739356195 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f64bbc40b0 con 0x55f64bdada10 2023-05-22T15:28:22.530 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.515+0000 7fe009d9b640 1 --2- 172.21.15.16:0/739356195 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f64bdfa170 0x55f64bdfa540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:28:22.531 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.515+0000 7fe00ad9d640 1 --2- 172.21.15.16:0/739356195 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f64bdada10 0x55f64bdadde0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7fe00405c5e0 tx=0x7fe004081590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c89a7b4c89c327df server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:28:22.531 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.515+0000 7fdfebfff640 1 -- 172.21.15.16:0/739356195 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe004083070 con 0x55f64bdada10 2023-05-22T15:28:22.531 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.515+0000 7fdfebfff640 1 -- 172.21.15.16:0/739356195 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe00405c680 con 0x55f64bdada10 2023-05-22T15:28:22.531 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.515+0000 7fdfebfff640 1 -- 172.21.15.16:0/739356195 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe004050870 con 0x55f64bdada10 2023-05-22T15:28:22.532 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.515+0000 7fe013cc8000 1 -- 172.21.15.16:0/739356195 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f64bdada10 msgr2=0x55f64bdadde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:22.532 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.515+0000 7fe013cc8000 1 --2- 172.21.15.16:0/739356195 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f64bdada10 0x55f64bdadde0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7fe00405c5e0 tx=0x7fe004081590 comp rx=0 tx=0).stop 2023-05-22T15:28:22.532 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.515+0000 7fe013cc8000 1 -- 172.21.15.16:0/739356195 shutdown_connections 2023-05-22T15:28:22.532 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.515+0000 7fe013cc8000 1 --2- 172.21.15.16:0/739356195 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f64bdada10 0x55f64bdadde0 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:22.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.515+0000 7fe013cc8000 1 --2- 172.21.15.16:0/739356195 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f64bdfa170 0x55f64bdfa540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:22.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.515+0000 7fe013cc8000 1 --2- 172.21.15.16:0/739356195 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x55f64bb86fd0 0x55f64bdf9c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:22.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.515+0000 7fe013cc8000 1 -- 172.21.15.16:0/739356195 >> 172.21.15.16:0/739356195 conn(0x55f64bc56270 msgr2=0x55f64bddd220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:28:22.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.515+0000 7fe013cc8000 1 -- 172.21.15.16:0/739356195 shutdown_connections 2023-05-22T15:28:22.534 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.515+0000 7fe013cc8000 1 -- 172.21.15.16:0/739356195 wait complete. 2023-05-22T15:28:22.534 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.515+0000 7fe013cc8000 1 -- start start 2023-05-22T15:28:22.534 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.515+0000 7fe013cc8000 1 Processor -- start 2023-05-22T15:28:22.535 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.515+0000 7fe013cc8000 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x55f64bb86fd0 0x55f64be00ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:22.535 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.515+0000 7fe013cc8000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f64bdada10 0x55f64be010e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:22.535 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.515+0000 7fe013cc8000 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f64bdfa170 0x55f64bdde450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:22.535 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.515+0000 7fe013cc8000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f64bbfe780 con 0x55f64bdada10 2023-05-22T15:28:22.536 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.515+0000 7fe013cc8000 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f64bbbc250 con 0x55f64bdfa170 2023-05-22T15:28:22.536 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.515+0000 7fe013cc8000 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x55f64bbbbef0 con 0x55f64bb86fd0 2023-05-22T15:28:22.536 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.515+0000 7fe013cc8000 10 client.0 _ll_register_callbacks cb 0x55f64bd9bb50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-05-22T15:28:22.536 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.515+0000 7fe009d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f64bdada10 0x55f64be010e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:22.537 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.515+0000 7fe009d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f64bdada10 0x55f64be010e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52066/0 (socket says 172.21.15.16:52066) 2023-05-22T15:28:22.537 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.515+0000 7fe009d9b640 1 -- 172.21.15.16:0/3068020068 learned_addr learned my addr 172.21.15.16:0/3068020068 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:28:22.537 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.515+0000 7fe00a59c640 1 --2- 172.21.15.16:0/3068020068 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x55f64bb86fd0 0x55f64be00ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:22.537 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.515+0000 7fe00ad9d640 1 --2- 172.21.15.16:0/3068020068 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f64bdfa170 0x55f64bdde450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:22.537 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.515+0000 7fe009d9b640 1 -- 172.21.15.16:0/3068020068 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x55f64bb86fd0 msgr2=0x55f64be00ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:28:22.537 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.515+0000 7fe009d9b640 1 --2- 172.21.15.16:0/3068020068 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x55f64bb86fd0 0x55f64be00ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:22.537 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.515+0000 7fe009d9b640 1 -- 172.21.15.16:0/3068020068 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f64bdfa170 msgr2=0x55f64bdde450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:28:22.538 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.515+0000 7fe009d9b640 1 --2- 172.21.15.16:0/3068020068 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f64bdfa170 0x55f64bdde450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:22.538 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.515+0000 7fe009d9b640 1 -- 172.21.15.16:0/3068020068 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe00c15cde0 con 0x55f64bdada10 2023-05-22T15:28:22.538 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.515+0000 7fe009d9b640 1 --2- 172.21.15.16:0/3068020068 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f64bdada10 0x55f64be010e0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7fe000005590 tx=0x7fe00000e650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb7758b2a2e6bec3 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:28:22.538 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.515+0000 7fdfe97fa640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2023-05-22T15:28:22.538 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.519+0000 7fdfe97fa640 1 -- 172.21.15.16:0/3068020068 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe000007230 con 0x55f64bdada10 2023-05-22T15:28:22.538 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.519+0000 7fe013cc8000 1 -- 172.21.15.16:0/3068020068 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55f64bbc2c60 con 0x55f64bdada10 2023-05-22T15:28:22.538 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.519+0000 7fdfe97fa640 1 -- 172.21.15.16:0/3068020068 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(42..42 src has 1..42) v4 ==== 6382+0+0 (secure 0 0 0) 0x7fe000007390 con 0x55f64bdada10 2023-05-22T15:28:22.538 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.519+0000 7fdfe97fa640 1 -- 172.21.15.16:0/3068020068 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=14) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe000007570 con 0x55f64bdada10 2023-05-22T15:28:22.539 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.519+0000 7fe013cc8000 10 client.5769 fetch_fsmap learned FSMap version 14 2023-05-22T15:28:22.539 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.519+0000 7fe013cc8000 1 -- 172.21.15.16:0/3068020068 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=14,monmap=2+}) v3 -- 0x55f64b8fb890 con 0x55f64bdada10 2023-05-22T15:28:22.539 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.519+0000 7fdfe97fa640 1 -- 172.21.15.16:0/3068020068 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55f64bbc2c60 con 0x55f64bdada10 2023-05-22T15:28:22.539 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.519+0000 7fdfe97fa640 1 -- 172.21.15.16:0/3068020068 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 14) v1 ==== 3106+0+0 (secure 0 0 0) 0x7fe00001a060 con 0x55f64bdada10 2023-05-22T15:28:22.539 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.519+0000 7fe013cc8000 10 client.5769 fetch_fsmap finished waiting for FSMap version 14 2023-05-22T15:28:22.539 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.519+0000 7fe013cc8000 10 client.5769 resolve_mds: resolved name 'c' to daemon mds.c 2023-05-22T15:28:22.539 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.519+0000 7fe013cc8000 20 client.5769 populate_metadata read hostname 'smithi016' 2023-05-22T15:28:22.540 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.519+0000 7fe013cc8000 1 --2- 172.21.15.16:0/3068020068 >> [v2:172.21.15.16:6836/1033866899,v1:172.21.15.16:6837/1033866899] conn(0x55f64bddf1c0 0x55f64bddf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:22.540 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.519+0000 7fe013cc8000 4 client.5769 mds_command: new command op to 5574 tid=0[{"prefix": "get_command_descriptions"}] 2023-05-22T15:28:22.540 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.519+0000 7fe013cc8000 1 -- 172.21.15.16:0/3068020068 --> [v2:172.21.15.16:6836/1033866899,v1:172.21.15.16:6837/1033866899] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f64bbfd0d0 con 0x55f64bddf1c0 2023-05-22T15:28:22.540 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.519+0000 7fe00a59c640 1 --2- 172.21.15.16:0/3068020068 >> [v2:172.21.15.16:6836/1033866899,v1:172.21.15.16:6837/1033866899] conn(0x55f64bddf1c0 0x55f64bddf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:22.540 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.519+0000 7fe00a59c640 1 --2- 172.21.15.16:0/3068020068 >> [v2:172.21.15.16:6836/1033866899,v1:172.21.15.16:6837/1033866899] conn(0x55f64bddf1c0 0x55f64bddf590 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f27a3e3da5334122 server_cookie=3d18e3ab65b3540b in_seq=0 out_seq=0 2023-05-22T15:28:22.540 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.519+0000 7fdfe97fa640 10 client.5769 ms_handle_connect on v2:172.21.15.16:6836/1033866899 2023-05-22T15:28:22.540 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.519+0000 7fdfe97fa640 1 -- 172.21.15.16:0/3068020068 <== mds.0 v2:172.21.15.16:6836/1033866899 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x55f64bbfd0d0 con 0x55f64bddf1c0 2023-05-22T15:28:22.541 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.519+0000 7fdfe97fa640 10 client.5769 handle_command_reply: tid=0 2023-05-22T15:28:22.541 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.519+0000 7fe013cc8000 1 client.5769 shutdown 2023-05-22T15:28:22.541 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.519+0000 7fe013cc8000 1 -- 172.21.15.16:0/3068020068 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f64bdada10 msgr2=0x55f64be010e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:22.541 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.519+0000 7fe013cc8000 1 --2- 172.21.15.16:0/3068020068 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f64bdada10 0x55f64be010e0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7fe000005590 tx=0x7fe00000e650 comp rx=0 tx=0).stop 2023-05-22T15:28:22.541 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.519+0000 7fe013cc8000 1 -- 172.21.15.16:0/3068020068 shutdown_connections 2023-05-22T15:28:22.541 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.519+0000 7fe013cc8000 1 --2- 172.21.15.16:0/3068020068 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f64bdfa170 0x55f64bdde450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:22.541 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.519+0000 7fe013cc8000 1 --2- 172.21.15.16:0/3068020068 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f64bdada10 0x55f64be010e0 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:22.541 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.519+0000 7fe013cc8000 1 --2- 172.21.15.16:0/3068020068 >> [v2:172.21.15.16:6836/1033866899,v1:172.21.15.16:6837/1033866899] conn(0x55f64bddf1c0 0x55f64bddf590 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:22.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.519+0000 7fe013cc8000 1 --2- 172.21.15.16:0/3068020068 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x55f64bb86fd0 0x55f64be00ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:22.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.519+0000 7fdfe97fa640 0 client.5769 ms_handle_reset on v2:172.21.15.16:6836/1033866899 2023-05-22T15:28:22.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.519+0000 7fe013cc8000 1 -- 172.21.15.16:0/3068020068 >> 172.21.15.16:0/3068020068 conn(0x55f64bc56270 msgr2=0x55f64bdad100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:28:22.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.519+0000 7fe013cc8000 1 -- 172.21.15.16:0/3068020068 shutdown_connections 2023-05-22T15:28:22.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.519+0000 7fe013cc8000 1 -- 172.21.15.16:0/3068020068 wait complete. 2023-05-22T15:28:22.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.519+0000 7fe013cc8000 20 client.5769 trim_cache size 0 max 16384 2023-05-22T15:28:22.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.535+0000 7fe013cc8000 1 Processor -- start 2023-05-22T15:28:22.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.535+0000 7fe013cc8000 1 -- start start 2023-05-22T15:28:22.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.535+0000 7fe013cc8000 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x55f64bb86fd0 0x55f64bdb0e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:22.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.535+0000 7fe013cc8000 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f64bdada10 0x55f64bdb1340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:22.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.535+0000 7fe013cc8000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f64bdfa170 0x55f64bdd6ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:22.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.535+0000 7fe013cc8000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f64b8bc940 con 0x55f64bdfa170 2023-05-22T15:28:22.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.535+0000 7fe013cc8000 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f64bda7e80 con 0x55f64bdada10 2023-05-22T15:28:22.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.535+0000 7fe013cc8000 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x55f64bdf9d00 con 0x55f64bb86fd0 2023-05-22T15:28:22.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.535+0000 7fe00ad9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f64bdfa170 0x55f64bdd6ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:22.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.535+0000 7fe00ad9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f64bdfa170 0x55f64bdd6ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52068/0 (socket says 172.21.15.16:52068) 2023-05-22T15:28:22.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.535+0000 7fe00ad9d640 1 -- 172.21.15.16:0/1343859152 learned_addr learned my addr 172.21.15.16:0/1343859152 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:28:22.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.535+0000 7fe00ad9d640 1 -- 172.21.15.16:0/1343859152 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x55f64bb86fd0 msgr2=0x55f64bdb0e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:28:22.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.535+0000 7fe00ad9d640 1 --2- 172.21.15.16:0/1343859152 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x55f64bb86fd0 0x55f64bdb0e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:22.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.535+0000 7fe00ad9d640 1 -- 172.21.15.16:0/1343859152 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f64bdada10 msgr2=0x55f64bdb1340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:28:22.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.535+0000 7fe00ad9d640 1 --2- 172.21.15.16:0/1343859152 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f64bdada10 0x55f64bdb1340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:22.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.535+0000 7fe00ad9d640 1 -- 172.21.15.16:0/1343859152 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f64bbc4360 con 0x55f64bdfa170 2023-05-22T15:28:22.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.535+0000 7fe00ad9d640 1 --2- 172.21.15.16:0/1343859152 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f64bdfa170 0x55f64bdd6ee0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7fe00400e9b0 tx=0x7fe004015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70b2773ee69d710e server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:28:22.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.535+0000 7fdff8ff9640 1 -- 172.21.15.16:0/1343859152 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe004050dc0 con 0x55f64bdfa170 2023-05-22T15:28:22.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.535+0000 7fdff8ff9640 1 -- 172.21.15.16:0/1343859152 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe004083070 con 0x55f64bdfa170 2023-05-22T15:28:22.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.535+0000 7fdff8ff9640 1 -- 172.21.15.16:0/1343859152 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe00405c880 con 0x55f64bdfa170 2023-05-22T15:28:22.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.535+0000 7fe013cc8000 1 -- 172.21.15.16:0/1343859152 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f64bdfa170 msgr2=0x55f64bdd6ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:22.549 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.535+0000 7fe013cc8000 1 --2- 172.21.15.16:0/1343859152 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f64bdfa170 0x55f64bdd6ee0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7fe00400e9b0 tx=0x7fe004015000 comp rx=0 tx=0).stop 2023-05-22T15:28:22.549 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.535+0000 7fe013cc8000 1 -- 172.21.15.16:0/1343859152 shutdown_connections 2023-05-22T15:28:22.549 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.535+0000 7fe013cc8000 1 --2- 172.21.15.16:0/1343859152 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f64bdfa170 0x55f64bdd6ee0 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:22.549 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.535+0000 7fe013cc8000 1 --2- 172.21.15.16:0/1343859152 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f64bdada10 0x55f64bdb1340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:22.549 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.535+0000 7fe013cc8000 1 --2- 172.21.15.16:0/1343859152 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x55f64bb86fd0 0x55f64bdb0e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:22.549 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.535+0000 7fe013cc8000 1 -- 172.21.15.16:0/1343859152 >> 172.21.15.16:0/1343859152 conn(0x55f64bc56270 msgr2=0x55f64bdad080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:28:22.549 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.535+0000 7fe013cc8000 1 -- 172.21.15.16:0/1343859152 shutdown_connections 2023-05-22T15:28:22.550 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.535+0000 7fe013cc8000 1 -- 172.21.15.16:0/1343859152 wait complete. 2023-05-22T15:28:22.550 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.535+0000 7fe013cc8000 1 -- start start 2023-05-22T15:28:22.550 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.535+0000 7fe013cc8000 1 Processor -- start 2023-05-22T15:28:22.550 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.539+0000 7fe013cc8000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f64bb86fd0 0x55f64be56a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:22.550 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.539+0000 7fe013cc8000 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x55f64bdada10 0x55f64be58f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:22.551 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.539+0000 7fe013cc8000 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f64bdfa170 0x55f64be5b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:22.551 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.539+0000 7fe013cc8000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f64bbfbb80 con 0x55f64bb86fd0 2023-05-22T15:28:22.551 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.539+0000 7fe013cc8000 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f64bdf9890 con 0x55f64bdfa170 2023-05-22T15:28:22.551 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.539+0000 7fe013cc8000 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x55f64be07620 con 0x55f64bdada10 2023-05-22T15:28:22.551 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.539+0000 7fe013cc8000 10 client.0 _ll_register_callbacks cb 0x55f64bc4eeb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-05-22T15:28:22.551 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.539+0000 7fe00a59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f64bb86fd0 0x55f64be56a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:22.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.539+0000 7fe00a59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f64bb86fd0 0x55f64be56a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52080/0 (socket says 172.21.15.16:52080) 2023-05-22T15:28:22.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.539+0000 7fe00a59c640 1 -- 172.21.15.16:0/44009434 learned_addr learned my addr 172.21.15.16:0/44009434 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:28:22.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.539+0000 7fe00a59c640 1 -- 172.21.15.16:0/44009434 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x55f64bdada10 msgr2=0x55f64be58f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:28:22.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.539+0000 7fe00a59c640 1 --2- 172.21.15.16:0/44009434 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x55f64bdada10 0x55f64be58f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:22.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.539+0000 7fe00a59c640 1 -- 172.21.15.16:0/44009434 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f64bdfa170 msgr2=0x55f64be5b4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:28:22.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.539+0000 7fe00a59c640 1 --2- 172.21.15.16:0/44009434 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f64bdfa170 0x55f64be5b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:22.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.539+0000 7fe00a59c640 1 -- 172.21.15.16:0/44009434 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f64bbc40b0 con 0x55f64bb86fd0 2023-05-22T15:28:22.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.539+0000 7fe00a59c640 1 --2- 172.21.15.16:0/44009434 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f64bb86fd0 0x55f64be56a40 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7fdffc0ae040 tx=0x7fdffc019260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c24e20ca25603030 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:28:22.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.539+0000 7fdfe97fa640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2023-05-22T15:28:22.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.539+0000 7fdfe97fa640 1 -- 172.21.15.16:0/44009434 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdffc0c7070 con 0x55f64bb86fd0 2023-05-22T15:28:22.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.539+0000 7fdfe97fa640 1 -- 172.21.15.16:0/44009434 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(42..42 src has 1..42) v4 ==== 6382+0+0 (secure 0 0 0) 0x7fdffc0abc50 con 0x55f64bb86fd0 2023-05-22T15:28:22.554 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.539+0000 7fe013cc8000 1 -- 172.21.15.16:0/44009434 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55f64bdb4160 con 0x55f64bb86fd0 2023-05-22T15:28:22.554 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.539+0000 7fdfe97fa640 1 -- 172.21.15.16:0/44009434 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=14) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdffc0bb040 con 0x55f64bb86fd0 2023-05-22T15:28:22.554 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.539+0000 7fe013cc8000 10 client.5775 fetch_fsmap learned FSMap version 14 2023-05-22T15:28:22.554 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.539+0000 7fe013cc8000 1 -- 172.21.15.16:0/44009434 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=14,monmap=2+}) v3 -- 0x55f64bba9be0 con 0x55f64bb86fd0 2023-05-22T15:28:22.554 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.539+0000 7fdfe97fa640 1 -- 172.21.15.16:0/44009434 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55f64bdb4160 con 0x55f64bb86fd0 2023-05-22T15:28:22.555 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.539+0000 7fdfe97fa640 1 -- 172.21.15.16:0/44009434 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 14) v1 ==== 3106+0+0 (secure 0 0 0) 0x7fdffc0b5470 con 0x55f64bb86fd0 2023-05-22T15:28:22.555 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.539+0000 7fe013cc8000 10 client.5775 fetch_fsmap finished waiting for FSMap version 14 2023-05-22T15:28:22.555 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.539+0000 7fe013cc8000 10 client.5775 resolve_mds: resolved name 'c' to daemon mds.c 2023-05-22T15:28:22.555 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.539+0000 7fe013cc8000 20 client.5775 populate_metadata read hostname 'smithi016' 2023-05-22T15:28:22.555 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.539+0000 7fe013cc8000 1 --2- 172.21.15.16:0/44009434 >> [v2:172.21.15.16:6836/1033866899,v1:172.21.15.16:6837/1033866899] conn(0x55f64be5ba20 0x55f64be5de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:22.555 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.539+0000 7fe013cc8000 4 client.5775 mds_command: new command op to 5574 tid=0[{"prefix": "status"}] 2023-05-22T15:28:22.555 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.539+0000 7fe013cc8000 1 -- 172.21.15.16:0/44009434 --> [v2:172.21.15.16:6836/1033866899,v1:172.21.15.16:6837/1033866899] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55f64bba54d0 con 0x55f64be5ba20 2023-05-22T15:28:22.556 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.539+0000 7fe009d9b640 1 --2- 172.21.15.16:0/44009434 >> [v2:172.21.15.16:6836/1033866899,v1:172.21.15.16:6837/1033866899] conn(0x55f64be5ba20 0x55f64be5de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:22.557 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.539+0000 7fe009d9b640 1 --2- 172.21.15.16:0/44009434 >> [v2:172.21.15.16:6836/1033866899,v1:172.21.15.16:6837/1033866899] conn(0x55f64be5ba20 0x55f64be5de00 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7bb171d30e8ba71d server_cookie=218aba0faf505caf in_seq=0 out_seq=0 2023-05-22T15:28:22.557 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.539+0000 7fdfe97fa640 10 client.5775 ms_handle_connect on v2:172.21.15.16:6836/1033866899 2023-05-22T15:28:22.557 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.539+0000 7fdfe97fa640 1 -- 172.21.15.16:0/44009434 <== mds.0 v2:172.21.15.16:6836/1033866899 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+324 (crc 0 0 0) 0x55f64bba54d0 con 0x55f64be5ba20 2023-05-22T15:28:22.557 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.539+0000 7fdfe97fa640 10 client.5775 handle_command_reply: tid=0 2023-05-22T15:28:22.558 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.539+0000 7fe013cc8000 1 client.5775 shutdown 2023-05-22T15:28:22.558 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.539+0000 7fe013cc8000 1 -- 172.21.15.16:0/44009434 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f64bb86fd0 msgr2=0x55f64be56a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:22.558 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.539+0000 7fe013cc8000 1 --2- 172.21.15.16:0/44009434 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f64bb86fd0 0x55f64be56a40 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7fdffc0ae040 tx=0x7fdffc019260 comp rx=0 tx=0).stop 2023-05-22T15:28:22.558 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.539+0000 7fe013cc8000 1 -- 172.21.15.16:0/44009434 shutdown_connections 2023-05-22T15:28:22.558 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.539+0000 7fe013cc8000 1 --2- 172.21.15.16:0/44009434 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f64bdfa170 0x55f64be5b4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:22.558 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.539+0000 7fe013cc8000 1 --2- 172.21.15.16:0/44009434 >> [v2:172.21.15.16:6836/1033866899,v1:172.21.15.16:6837/1033866899] conn(0x55f64be5ba20 0x55f64be5de00 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:22.558 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.539+0000 7fe013cc8000 1 --2- 172.21.15.16:0/44009434 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x55f64bdada10 0x55f64be58f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:22.559 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.539+0000 7fe013cc8000 1 --2- 172.21.15.16:0/44009434 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f64bb86fd0 0x55f64be56a40 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:22.559 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.539+0000 7fe013cc8000 1 -- 172.21.15.16:0/44009434 >> 172.21.15.16:0/44009434 conn(0x55f64bc56270 msgr2=0x55f64bdafdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:28:22.559 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.539+0000 7fdfe97fa640 0 client.5775 ms_handle_reset on v2:172.21.15.16:6836/1033866899 2023-05-22T15:28:22.559 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.539+0000 7fe013cc8000 1 -- 172.21.15.16:0/44009434 shutdown_connections 2023-05-22T15:28:22.559 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.539+0000 7fe013cc8000 1 -- 172.21.15.16:0/44009434 wait complete. 2023-05-22T15:28:22.559 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.539+0000 7fe013cc8000 20 client.5775 trim_cache size 0 max 16384 2023-05-22T15:28:22.560 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-05-22T15:28:22.560 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_fsid": "b2719834-7847-4d98-90a6-f65d57f85c49", 2023-05-22T15:28:22.560 INFO:teuthology.orchestra.run.smithi016.stdout: "whoami": 0, 2023-05-22T15:28:22.560 INFO:teuthology.orchestra.run.smithi016.stdout: "id": 5574, 2023-05-22T15:28:22.561 INFO:teuthology.orchestra.run.smithi016.stdout: "want_state": "up:active", 2023-05-22T15:28:22.561 INFO:teuthology.orchestra.run.smithi016.stdout: "state": "up:active", 2023-05-22T15:28:22.561 INFO:teuthology.orchestra.run.smithi016.stdout: "fs_name": "cephfs", 2023-05-22T15:28:22.561 INFO:teuthology.orchestra.run.smithi016.stdout: "rank_uptime": 9.2204212590000001, 2023-05-22T15:28:22.561 INFO:teuthology.orchestra.run.smithi016.stdout: "mdsmap_epoch": 14, 2023-05-22T15:28:22.561 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch": 42, 2023-05-22T15:28:22.561 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch_barrier": 42, 2023-05-22T15:28:22.561 INFO:teuthology.orchestra.run.smithi016.stdout: "uptime": 15.93565098 2023-05-22T15:28:22.562 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-05-22T15:28:22.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.539+0000 7fe010ba6640 1 -- 172.21.15.16:0/4214088620 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fdfec065380 msgr2=0x7fdfec067830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:22.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.539+0000 7fe010ba6640 1 --2- 172.21.15.16:0/4214088620 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fdfec065380 0x7fdfec067830 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fdffc0ad000 tx=0x7fdffc0a6000 comp rx=0 tx=0).stop 2023-05-22T15:28:22.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.539+0000 7fe010ba6640 1 -- 172.21.15.16:0/4214088620 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe00c159a40 msgr2=0x7fe00c15b3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:22.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.539+0000 7fe010ba6640 1 --2- 172.21.15.16:0/4214088620 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe00c159a40 0x7fe00c15b3b0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7fe004007e60 tx=0x7fe00400e4b0 comp rx=0 tx=0).stop 2023-05-22T15:28:22.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.539+0000 7fe010ba6640 1 -- 172.21.15.16:0/4214088620 shutdown_connections 2023-05-22T15:28:22.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.539+0000 7fe010ba6640 1 --2- 172.21.15.16:0/4214088620 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe00c159a40 0x7fe00c15b3b0 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:22.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.539+0000 7fe010ba6640 1 --2- 172.21.15.16:0/4214088620 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe00c1531b0 0x7fe00c14d6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:22.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.539+0000 7fe010ba6640 1 --2- 172.21.15.16:0/4214088620 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fdfec065380 0x7fdfec067830 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:22.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.539+0000 7fe010ba6640 1 --2- 172.21.15.16:0/4214088620 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe00c0b0080 0x7fe00c144480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:22.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.539+0000 7fe010ba6640 1 -- 172.21.15.16:0/4214088620 >> 172.21.15.16:0/4214088620 conn(0x7fe00c069720 msgr2=0x7fe00c0cc4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:28:22.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.539+0000 7fe010ba6640 1 -- 172.21.15.16:0/4214088620 shutdown_connections 2023-05-22T15:28:22.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.539+0000 7fe010ba6640 1 -- 172.21.15.16:0/4214088620 wait complete. 2023-05-22T15:28:22.565 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-05-22T15:28:22.676 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.671+0000 7f8e0d168640 1 Processor -- start 2023-05-22T15:28:22.677 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.671+0000 7f8e0d168640 1 -- start start 2023-05-22T15:28:22.678 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.671+0000 7f8e0d168640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8e0815fc50 0x7f8e08160020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:22.678 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.671+0000 7f8e0d168640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f8e0815c920 0x7f8e080d0030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:22.678 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.671+0000 7f8e0d168640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e0815c490 con 0x7f8e0815cd70 2023-05-22T15:28:22.678 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.671+0000 7f8e0d168640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e080cf7a0 con 0x7f8e0815c920 2023-05-22T15:28:22.679 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.671+0000 7f8e0d168640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e0815c140 con 0x7f8e0815fc50 2023-05-22T15:28:22.679 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.671+0000 7f8e0659c640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f8e0815c920 0x7f8e080d0030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:22.679 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.671+0000 7f8e0659c640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f8e0815c920 0x7f8e080d0030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52088/0 (socket says 172.21.15.16:52088) 2023-05-22T15:28:22.679 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.671+0000 7f8e0759e640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f8e0815cd70 0x7f8e080b3ee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:33804/0 (socket says 172.21.15.16:33804) 2023-05-22T15:28:22.679 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.671+0000 7f8e0659c640 1 -- 172.21.15.16:0/1158448445 learned_addr learned my addr 172.21.15.16:0/1158448445 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:28:22.679 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.671+0000 7f8e06d9d640 1 --2- 172.21.15.16:0/1158448445 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8e0815fc50 0x7f8e08160020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:22.679 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.671+0000 7f8e0659c640 1 -- 172.21.15.16:0/1158448445 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8e0815fc50 msgr2=0x7f8e08160020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:28:22.680 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.671+0000 7f8e0659c640 1 --2- 172.21.15.16:0/1158448445 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8e0815fc50 0x7f8e08160020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:22.680 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.671+0000 7f8e0659c640 1 -- 172.21.15.16:0/1158448445 >> v1:172.21.15.16:6789/0 conn(0x7f8e0815cd70 legacy=0x7f8e080b3ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:22.680 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.671+0000 7f8e0659c640 1 -- 172.21.15.16:0/1158448445 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e080ccb50 con 0x7f8e0815c920 2023-05-22T15:28:22.680 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.671+0000 7f8e05d9b640 1 -- 172.21.15.16:0/1158448445 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3601049169 0 0) 0x7f8e0815c490 con 0x7f8e0815cd70 2023-05-22T15:28:22.681 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.675+0000 7f8e0659c640 1 --2- 172.21.15.16:0/1158448445 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8e0815c920 0x7f8e080d0030 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f8dfc009870 tx=0x7f8dfc01c510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c68d03fd8df5e7b server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:28:22.681 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.675+0000 7f8e05d9b640 1 -- 172.21.15.16:0/1158448445 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dfc0aa670 con 0x7f8e0815c920 2023-05-22T15:28:22.681 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.675+0000 7f8e05d9b640 1 -- 172.21.15.16:0/1158448445 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8dfc0aa7d0 con 0x7f8e0815c920 2023-05-22T15:28:22.681 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.675+0000 7f8e05d9b640 1 -- 172.21.15.16:0/1158448445 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dfc0aab60 con 0x7f8e0815c920 2023-05-22T15:28:22.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.675+0000 7f8e0d168640 1 -- 172.21.15.16:0/1158448445 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8e0815c920 msgr2=0x7f8e080d0030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:22.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.675+0000 7f8e0d168640 1 --2- 172.21.15.16:0/1158448445 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8e0815c920 0x7f8e080d0030 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f8dfc009870 tx=0x7f8dfc01c510 comp rx=0 tx=0).stop 2023-05-22T15:28:22.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.675+0000 7f8e0d168640 1 -- 172.21.15.16:0/1158448445 shutdown_connections 2023-05-22T15:28:22.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.675+0000 7f8e0d168640 1 --2- 172.21.15.16:0/1158448445 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8e0815c920 0x7f8e080d0030 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:22.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.675+0000 7f8e0d168640 1 --2- 172.21.15.16:0/1158448445 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8e0815fc50 0x7f8e08160020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:22.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.675+0000 7f8e0d168640 1 -- 172.21.15.16:0/1158448445 >> 172.21.15.16:0/1158448445 conn(0x7f8e080657f0 msgr2=0x7f8e08070e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:28:22.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.675+0000 7f8e0d168640 1 -- 172.21.15.16:0/1158448445 shutdown_connections 2023-05-22T15:28:22.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.675+0000 7f8e0d168640 1 -- 172.21.15.16:0/1158448445 wait complete. 2023-05-22T15:28:22.683 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.675+0000 7f8e0d168640 1 Processor -- start 2023-05-22T15:28:22.683 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.675+0000 7f8e0d168640 1 -- start start 2023-05-22T15:28:22.685 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.675+0000 7f8e0d168640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8e0815c920 0x7f8e081561d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:22.685 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.675+0000 7f8e0d168640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8e0815cd70 0x7f8e0814e460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:22.685 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.675+0000 7f8e0d168640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8e0815fc50 0x7f8e080c5f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:22.685 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.675+0000 7f8e0d168640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e080b76c0 con 0x7f8e0815fc50 2023-05-22T15:28:22.685 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.675+0000 7f8e0d168640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e08152c90 con 0x7f8e0815cd70 2023-05-22T15:28:22.685 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.675+0000 7f8e0d168640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e080b6d90 con 0x7f8e0815c920 2023-05-22T15:28:22.686 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.675+0000 7f8e0759e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8e0815fc50 0x7f8e080c5f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:22.686 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.675+0000 7f8e0759e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8e0815fc50 0x7f8e080c5f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52098/0 (socket says 172.21.15.16:52098) 2023-05-22T15:28:22.686 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.675+0000 7f8e0759e640 1 -- 172.21.15.16:0/470087793 learned_addr learned my addr 172.21.15.16:0/470087793 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:28:22.686 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.679+0000 7f8e06d9d640 1 --2- 172.21.15.16:0/470087793 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8e0815c920 0x7f8e081561d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:22.686 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.679+0000 7f8e0659c640 1 --2- 172.21.15.16:0/470087793 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8e0815cd70 0x7f8e0814e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:22.686 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.679+0000 7f8e0759e640 1 -- 172.21.15.16:0/470087793 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8e0815c920 msgr2=0x7f8e081561d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:22.686 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.679+0000 7f8e0759e640 1 --2- 172.21.15.16:0/470087793 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8e0815c920 0x7f8e081561d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:22.687 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.679+0000 7f8e0759e640 1 -- 172.21.15.16:0/470087793 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8e0815cd70 msgr2=0x7f8e0814e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:22.687 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.679+0000 7f8e0759e640 1 --2- 172.21.15.16:0/470087793 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8e0815cd70 0x7f8e0814e460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:22.687 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.679+0000 7f8e0759e640 1 -- 172.21.15.16:0/470087793 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8dfc0094d0 con 0x7f8e0815fc50 2023-05-22T15:28:22.687 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.679+0000 7f8e0659c640 1 --2- 172.21.15.16:0/470087793 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8e0815cd70 0x7f8e0814e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:28:22.687 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.679+0000 7f8e06d9d640 1 --2- 172.21.15.16:0/470087793 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8e0815c920 0x7f8e081561d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:28:22.687 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.679+0000 7f8e0759e640 1 --2- 172.21.15.16:0/470087793 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8e0815fc50 0x7f8e080c5f90 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f8e00001570 tx=0x7f8e0000cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:28:22.687 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.679+0000 7f8def7fe640 1 -- 172.21.15.16:0/470087793 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e0000a640 con 0x7f8e0815fc50 2023-05-22T15:28:22.687 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.679+0000 7f8def7fe640 1 -- 172.21.15.16:0/470087793 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e0000a7c0 con 0x7f8e0815fc50 2023-05-22T15:28:22.688 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.679+0000 7f8e0d168640 1 -- 172.21.15.16:0/470087793 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e08160560 con 0x7f8e0815fc50 2023-05-22T15:28:22.688 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.679+0000 7f8def7fe640 1 -- 172.21.15.16:0/470087793 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e0000ab70 con 0x7f8e0815fc50 2023-05-22T15:28:22.688 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.679+0000 7f8e0d168640 1 -- 172.21.15.16:0/470087793 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e080d0980 con 0x7f8e0815fc50 2023-05-22T15:28:22.689 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.679+0000 7f8e0d168640 1 -- 172.21.15.16:0/470087793 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e08168ad0 con 0x7f8e0815fc50 2023-05-22T15:28:22.689 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.679+0000 7f8def7fe640 1 -- 172.21.15.16:0/470087793 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7f8e0000acd0 con 0x7f8e0815fc50 2023-05-22T15:28:22.689 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.679+0000 7f8def7fe640 1 --2- 172.21.15.16:0/470087793 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f8df4065380 0x7f8df4067830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:22.689 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.679+0000 7f8def7fe640 1 -- 172.21.15.16:0/470087793 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f8e00082230 con 0x7f8e0815fc50 2023-05-22T15:28:22.689 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.679+0000 7f8e06d9d640 1 --2- 172.21.15.16:0/470087793 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f8df4065380 0x7f8df4067830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:22.689 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.679+0000 7f8e06d9d640 1 --2- 172.21.15.16:0/470087793 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f8df4065380 0x7f8df4067830 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f8df8006fd0 tx=0x7f8df8008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:28:22.691 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.683+0000 7f8def7fe640 1 -- 172.21.15.16:0/470087793 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f8e0004fd70 con 0x7f8e0815fc50 2023-05-22T15:28:22.876 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.867+0000 7f8e0d168640 1 -- 172.21.15.16:0/470087793 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f8e0804dbe0 con 0x7f8e0815fc50 2023-05-22T15:28:22.877 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.871+0000 7f8def7fe640 1 -- 172.21.15.16:0/470087793 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v14) v1 ==== 53+0+73 (secure 0 0 0) 0x7f8e00053390 con 0x7f8e0815fc50 2023-05-22T15:28:22.878 INFO:teuthology.orchestra.run.smithi016.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-05-22T15:28:22.879 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.871+0000 7f8e0d168640 1 -- 172.21.15.16:0/470087793 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f8df4065380 msgr2=0x7f8df4067830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:22.880 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.871+0000 7f8e0d168640 1 --2- 172.21.15.16:0/470087793 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f8df4065380 0x7f8df4067830 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f8df8006fd0 tx=0x7f8df8008040 comp rx=0 tx=0).stop 2023-05-22T15:28:22.880 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.871+0000 7f8e0d168640 1 -- 172.21.15.16:0/470087793 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8e0815fc50 msgr2=0x7f8e080c5f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:22.880 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.871+0000 7f8e0d168640 1 --2- 172.21.15.16:0/470087793 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8e0815fc50 0x7f8e080c5f90 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f8e00001570 tx=0x7f8e0000cca0 comp rx=0 tx=0).stop 2023-05-22T15:28:22.880 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.871+0000 7f8e0d168640 1 -- 172.21.15.16:0/470087793 shutdown_connections 2023-05-22T15:28:22.881 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.871+0000 7f8e0d168640 1 --2- 172.21.15.16:0/470087793 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8e0815fc50 0x7f8e080c5f90 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:22.881 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.871+0000 7f8e0d168640 1 --2- 172.21.15.16:0/470087793 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8e0815cd70 0x7f8e0814e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:22.882 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.871+0000 7f8e0d168640 1 --2- 172.21.15.16:0/470087793 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f8df4065380 0x7f8df4067830 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:22.882 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.871+0000 7f8e0d168640 1 --2- 172.21.15.16:0/470087793 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8e0815c920 0x7f8e081561d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:22.882 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.871+0000 7f8e0d168640 1 -- 172.21.15.16:0/470087793 >> 172.21.15.16:0/470087793 conn(0x7f8e080657f0 msgr2=0x7f8e0815d170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:28:22.883 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.871+0000 7f8e0d168640 1 -- 172.21.15.16:0/470087793 shutdown_connections 2023-05-22T15:28:22.884 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:22.871+0000 7f8e0d168640 1 -- 172.21.15.16:0/470087793 wait complete. 2023-05-22T15:28:22.895 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-05-22T15:28:22.895 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-05-22T15:28:22.896 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi016.front.sepia.ceph.com 2023-05-22T15:28:22.896 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-05-22T15:28:22.897 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-05-22T15:28:22.897 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-05-22T15:28:22.898 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-05-22T15:28:22.898 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-05-22T15:28:22.898 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-05-22T15:28:22.899 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:28:22.900 DEBUG:teuthology.orchestra.run.smithi016:> ip addr 2023-05-22T15:28:22.906 INFO:teuthology.orchestra.run.smithi016.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-05-22T15:28:22.907 INFO:teuthology.orchestra.run.smithi016.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-05-22T15:28:22.907 INFO:teuthology.orchestra.run.smithi016.stdout: inet 127.0.0.1/8 scope host lo 2023-05-22T15:28:22.907 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-05-22T15:28:22.907 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 ::1/128 scope host 2023-05-22T15:28:22.907 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-05-22T15:28:22.907 INFO:teuthology.orchestra.run.smithi016.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-05-22T15:28:22.908 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:6b:fd:e0 brd ff:ff:ff:ff:ff:ff 2023-05-22T15:28:22.908 INFO:teuthology.orchestra.run.smithi016.stdout: altname enp7s0f0 2023-05-22T15:28:22.908 INFO:teuthology.orchestra.run.smithi016.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-05-22T15:28:22.908 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:6b:fd:e1 brd ff:ff:ff:ff:ff:ff 2023-05-22T15:28:22.908 INFO:teuthology.orchestra.run.smithi016.stdout: altname enp7s0f1 2023-05-22T15:28:22.908 INFO:teuthology.orchestra.run.smithi016.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2023-05-22T15:28:22.908 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:bd:15:64 brd ff:ff:ff:ff:ff:ff 2023-05-22T15:28:22.908 INFO:teuthology.orchestra.run.smithi016.stdout: altname enp1s0f0 2023-05-22T15:28:22.909 INFO:teuthology.orchestra.run.smithi016.stdout: inet 172.21.15.16/20 brd 172.21.15.255 scope global dynamic ens1f0 2023-05-22T15:28:22.909 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft 42413sec preferred_lft 42413sec 2023-05-22T15:28:22.909 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::ec4:7aff:febd:1564/64 scope link 2023-05-22T15:28:22.909 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-05-22T15:28:22.909 INFO:teuthology.orchestra.run.smithi016.stdout:5: ens1f1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-05-22T15:28:22.910 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:bd:15:65 brd ff:ff:ff:ff:ff:ff 2023-05-22T15:28:22.910 INFO:teuthology.orchestra.run.smithi016.stdout: altname enp1s0f1 2023-05-22T15:28:22.910 INFO:teuthology.orchestra.run.smithi016.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2023-05-22T15:28:22.910 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 02:42:1f:b8:98:0a brd ff:ff:ff:ff:ff:ff 2023-05-22T15:28:22.910 INFO:teuthology.orchestra.run.smithi016.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2023-05-22T15:28:22.910 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-05-22T15:28:22.911 INFO:teuthology.orchestra.run.smithi016.stdout:9: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-05-22T15:28:22.911 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 5e:1f:84:6d:79:ae brd ff:ff:ff:ff:ff:ff 2023-05-22T15:28:22.911 INFO:teuthology.orchestra.run.smithi016.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-05-22T15:28:22.911 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-05-22T15:28:22.911 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::5c1f:84ff:fe6d:79ae/64 scope link 2023-05-22T15:28:22.911 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-05-22T15:28:22.912 INFO:teuthology.orchestra.run.smithi016.stdout:10: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-05-22T15:28:22.912 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0e:5c:f6:1f:82:2f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-05-22T15:28:22.912 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::c5c:f6ff:fe1f:822f/64 scope link 2023-05-22T15:28:22.913 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-05-22T15:28:22.913 INFO:teuthology.orchestra.run.smithi016.stdout:11: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-05-22T15:28:22.913 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether c6:a6:3d:69:d0:db brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-05-22T15:28:22.913 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::c4a6:3dff:fe69:d0db/64 scope link 2023-05-22T15:28:22.913 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-05-22T15:28:22.914 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:28:22.914 DEBUG:teuthology.orchestra.run.smithi016:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-05-22T15:28:22.959 INFO:teuthology.orchestra.run.smithi016.stdout:1 2023-05-22T15:28:22.960 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:28:22.961 DEBUG:teuthology.orchestra.run.smithi016:> route 2023-05-22T15:28:23.010 INFO:teuthology.orchestra.run.smithi016.stdout:Kernel IP routing table 2023-05-22T15:28:23.010 INFO:teuthology.orchestra.run.smithi016.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-05-22T15:28:23.010 INFO:teuthology.orchestra.run.smithi016.stdout:default _gateway 0.0.0.0 UG 0 0 0 ens1f0 2023-05-22T15:28:23.010 INFO:teuthology.orchestra.run.smithi016.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2023-05-22T15:28:23.011 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 0 0 0 ens1f0 2023-05-22T15:28:23.011 INFO:teuthology.orchestra.run.smithi016.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-05-22T15:28:23.011 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:28:23.011 DEBUG:teuthology.orchestra.run.smithi016:> (cd / && exec bash -c ' 2023-05-22T15:28:23.012 DEBUG:teuthology.orchestra.run.smithi016:> set -e 2023-05-22T15:28:23.012 DEBUG:teuthology.orchestra.run.smithi016:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2023-05-22T15:28:23.012 DEBUG:teuthology.orchestra.run.smithi016:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2023-05-22T15:28:23.013 DEBUG:teuthology.orchestra.run.smithi016:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2023-05-22T15:28:23.013 DEBUG:teuthology.orchestra.run.smithi016:> ') 2023-05-22T15:28:23.084 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:28:23.084 DEBUG:teuthology.orchestra.run.smithi016:> ip netns list 2023-05-22T15:28:23.132 INFO:teuthology.orchestra.run.smithi016.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-05-22T15:28:23.132 INFO:teuthology.orchestra.run.smithi016.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-05-22T15:28:23.133 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-05-22T15:28:23.133 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:28:23.133 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip link set brx.0 up 2023-05-22T15:28:23.187 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2023-05-22T15:28:23.187 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-05-22T15:28:23.188 DEBUG:teuthology.orchestra.run.smithi016:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-05-22T15:28:23.232 INFO:teuthology.orchestra.run.smithi016.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-05-22T15:28:23.233 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-05-22T15:28:23.233 DEBUG:teuthology.orchestra.run.smithi016:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-05-22T15:28:23.281 DEBUG:teuthology.orchestra.run.smithi016:> sudo modprobe fuse 2023-05-22T15:28:23.343 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-05-22T15:28:23.343 DEBUG:teuthology.orchestra.run.smithi016:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-05-22T15:28:23.397 INFO:teuthology.orchestra.run.smithi016.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-05-22T15:28:23.398 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-05-22T15:28:23.399 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:28:23.399 DEBUG:teuthology.orchestra.run.smithi016:> ls /sys/fs/fuse/connections 2023-05-22T15:28:23.449 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2023-05-22T15:28:23.451 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2023-05-22T15:28:23.494 DEBUG:teuthology.orchestra.run.smithi016:> sudo modprobe fuse 2023-05-22T15:28:23.509 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-05-22T15:28:23.510 DEBUG:teuthology.orchestra.run.smithi016:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-05-22T15:28:23.564 INFO:teuthology.orchestra.run.smithi016.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-05-22T15:28:23.565 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-05-22T15:28:23.566 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:28:23.567 DEBUG:teuthology.orchestra.run.smithi016:> ls /sys/fs/fuse/connections 2023-05-22T15:28:23.570 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi016.stderr:2023-05-22T15:28:23.563+0000 7f7d1ac71580 -1 init, newargv = 0x55b280dbfdc0 newargc=15 2023-05-22T15:28:23.570 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi016.stderr:ceph-fuse[49179]: starting ceph client 2023-05-22T15:28:23.577 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi016.stderr:ceph-fuse[49179]: starting fuse 2023-05-22T15:28:24.676 DEBUG:teuthology.orchestra.run.smithi016:> sudo modprobe fuse 2023-05-22T15:28:24.694 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-05-22T15:28:24.695 DEBUG:teuthology.orchestra.run.smithi016:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-05-22T15:28:24.748 INFO:teuthology.orchestra.run.smithi016.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-05-22T15:28:24.749 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-05-22T15:28:24.750 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:28:24.751 DEBUG:teuthology.orchestra.run.smithi016:> ls /sys/fs/fuse/connections 2023-05-22T15:28:24.801 INFO:teuthology.orchestra.run.smithi016.stdout:45 2023-05-22T15:28:24.802 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [45] 2023-05-22T15:28:24.802 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2023-05-22T15:28:24.802 DEBUG:teuthology.orchestra.run.smithi016:> import glob 2023-05-22T15:28:24.803 DEBUG:teuthology.orchestra.run.smithi016:> import re 2023-05-22T15:28:24.803 DEBUG:teuthology.orchestra.run.smithi016:> import os 2023-05-22T15:28:24.804 DEBUG:teuthology.orchestra.run.smithi016:> import subprocess 2023-05-22T15:28:24.804 DEBUG:teuthology.orchestra.run.smithi016:> 2023-05-22T15:28:24.804 DEBUG:teuthology.orchestra.run.smithi016:> def _find_admin_socket(client_name): 2023-05-22T15:28:24.804 DEBUG:teuthology.orchestra.run.smithi016:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2023-05-22T15:28:24.805 DEBUG:teuthology.orchestra.run.smithi016:> files = glob.glob(asok_path) 2023-05-22T15:28:24.805 DEBUG:teuthology.orchestra.run.smithi016:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2023-05-22T15:28:24.805 DEBUG:teuthology.orchestra.run.smithi016:> 2023-05-22T15:28:24.806 DEBUG:teuthology.orchestra.run.smithi016:> # Given a non-glob path, it better be there 2023-05-22T15:28:24.806 DEBUG:teuthology.orchestra.run.smithi016:> if "*" not in asok_path: 2023-05-22T15:28:24.806 DEBUG:teuthology.orchestra.run.smithi016:> assert(len(files) == 1) 2023-05-22T15:28:24.807 DEBUG:teuthology.orchestra.run.smithi016:> return files[0] 2023-05-22T15:28:24.807 DEBUG:teuthology.orchestra.run.smithi016:> 2023-05-22T15:28:24.807 DEBUG:teuthology.orchestra.run.smithi016:> for f in files: 2023-05-22T15:28:24.808 DEBUG:teuthology.orchestra.run.smithi016:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-05-22T15:28:24.808 DEBUG:teuthology.orchestra.run.smithi016:> if os.path.exists("/proc/{0}".format(pid)): 2023-05-22T15:28:24.809 DEBUG:teuthology.orchestra.run.smithi016:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-05-22T15:28:24.810 DEBUG:teuthology.orchestra.run.smithi016:> contents = proc_f.read() 2023-05-22T15:28:24.810 DEBUG:teuthology.orchestra.run.smithi016:> if mountpoint in contents: 2023-05-22T15:28:24.811 DEBUG:teuthology.orchestra.run.smithi016:> return f 2023-05-22T15:28:24.811 DEBUG:teuthology.orchestra.run.smithi016:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-05-22T15:28:24.811 DEBUG:teuthology.orchestra.run.smithi016:> 2023-05-22T15:28:24.812 DEBUG:teuthology.orchestra.run.smithi016:> print(_find_admin_socket("client.0")) 2023-05-22T15:28:24.812 DEBUG:teuthology.orchestra.run.smithi016:> ' 2023-05-22T15:28:24.925 INFO:teuthology.orchestra.run.smithi016.stdout:/var/run/ceph/ceph-client.0.49179.asok 2023-05-22T15:28:25.107 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.49179.asok 2023-05-22T15:28:25.107 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:28:25.108 DEBUG:teuthology.orchestra.run.smithi016:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.49179.asok status 2023-05-22T15:28:25.264 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-05-22T15:28:25.265 INFO:teuthology.orchestra.run.smithi016.stdout: "metadata": { 2023-05-22T15:28:25.265 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph_sha1": "9c9ba4df22024d9d8d6cf9f98a4b96751fc78d0c", 2023-05-22T15:28:25.265 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph_version": "ceph version 18.0.0-3192-g9c9ba4df (9c9ba4df22024d9d8d6cf9f98a4b96751fc78d0c) reef (dev)", 2023-05-22T15:28:25.265 INFO:teuthology.orchestra.run.smithi016.stdout: "entity_id": "0", 2023-05-22T15:28:25.265 INFO:teuthology.orchestra.run.smithi016.stdout: "hostname": "smithi016", 2023-05-22T15:28:25.265 INFO:teuthology.orchestra.run.smithi016.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2023-05-22T15:28:25.265 INFO:teuthology.orchestra.run.smithi016.stdout: "pid": "49179", 2023-05-22T15:28:25.266 INFO:teuthology.orchestra.run.smithi016.stdout: "root": "/" 2023-05-22T15:28:25.266 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:28:25.266 INFO:teuthology.orchestra.run.smithi016.stdout: "dentry_count": 0, 2023-05-22T15:28:25.266 INFO:teuthology.orchestra.run.smithi016.stdout: "dentry_pinned_count": 0, 2023-05-22T15:28:25.266 INFO:teuthology.orchestra.run.smithi016.stdout: "id": 5793, 2023-05-22T15:28:25.266 INFO:teuthology.orchestra.run.smithi016.stdout: "inst": { 2023-05-22T15:28:25.266 INFO:teuthology.orchestra.run.smithi016.stdout: "name": { 2023-05-22T15:28:25.267 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "client", 2023-05-22T15:28:25.267 INFO:teuthology.orchestra.run.smithi016.stdout: "num": 5793 2023-05-22T15:28:25.267 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:28:25.267 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": { 2023-05-22T15:28:25.267 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-05-22T15:28:25.267 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "192.168.0.1:0", 2023-05-22T15:28:25.267 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 3250928193 2023-05-22T15:28:25.267 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T15:28:25.268 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:28:25.268 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": { 2023-05-22T15:28:25.268 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-05-22T15:28:25.268 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "192.168.0.1:0", 2023-05-22T15:28:25.268 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 3250928193 2023-05-22T15:28:25.268 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:28:25.268 INFO:teuthology.orchestra.run.smithi016.stdout: "inst_str": "client.5793 192.168.0.1:0/3250928193", 2023-05-22T15:28:25.269 INFO:teuthology.orchestra.run.smithi016.stdout: "addr_str": "192.168.0.1:0/3250928193", 2023-05-22T15:28:25.269 INFO:teuthology.orchestra.run.smithi016.stdout: "inode_count": 1, 2023-05-22T15:28:25.269 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_epoch": 14, 2023-05-22T15:28:25.269 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_epoch": 42, 2023-05-22T15:28:25.269 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_epoch_barrier": 0, 2023-05-22T15:28:25.269 INFO:teuthology.orchestra.run.smithi016.stdout: "blocklisted": false, 2023-05-22T15:28:25.269 INFO:teuthology.orchestra.run.smithi016.stdout: "fs_name": "cephfs" 2023-05-22T15:28:25.269 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-05-22T15:28:25.279 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:28:25.279 DEBUG:teuthology.orchestra.run.smithi016:> stat --file-system '--printf=%T 2023-05-22T15:28:25.280 DEBUG:teuthology.orchestra.run.smithi016:> ' -- /home/ubuntu/cephtest/mnt.0 2023-05-22T15:28:25.286 INFO:teuthology.orchestra.run.smithi016.stdout:fuseblk 2023-05-22T15:28:25.287 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2023-05-22T15:28:25.287 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:28:25.287 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-05-22T15:28:25.344 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2023-05-22T15:28:25.516 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:25.507+0000 7f7089447640 1 Processor -- start 2023-05-22T15:28:25.516 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:25.507+0000 7f7089447640 1 -- start start 2023-05-22T15:28:25.519 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:25.511+0000 7f7089447640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f70840ca4e0 0x7f708415e460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:25.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:25.511+0000 7f7089447640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f70840c9b40 0x7f70840afb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:25.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:25.511+0000 7f7089447640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f7084070df0 0x7f7084071210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:25.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:25.511+0000 7f7089447640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70840b0650 con 0x7f70840ca4e0 2023-05-22T15:28:25.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:25.511+0000 7f7089447640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70840b0370 con 0x7f7084070df0 2023-05-22T15:28:25.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:25.511+0000 7f7089447640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f708415d980 con 0x7f70840c9b40 2023-05-22T15:28:25.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:25.511+0000 7f7082ffd640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f70840ca4e0 0x7f708415e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:25.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:25.511+0000 7f7082ffd640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f70840ca4e0 0x7f708415e460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52100/0 (socket says 172.21.15.16:52100) 2023-05-22T15:28:25.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:25.511+0000 7f7082ffd640 1 -- 172.21.15.16:0/32184924 learned_addr learned my addr 172.21.15.16:0/32184924 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:28:25.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:25.511+0000 7f70837fe640 1 --2- 172.21.15.16:0/32184924 >> v2:172.21.15.62:3300/0 conn(0x7f7084070df0 0x7f7084071210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:25.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:25.511+0000 7f70827fc640 1 --2- 172.21.15.16:0/32184924 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f70840c9b40 0x7f70840afb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:25.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:25.511+0000 7f7082ffd640 1 -- 172.21.15.16:0/32184924 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f70840c9b40 msgr2=0x7f70840afb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:28:25.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:25.511+0000 7f7082ffd640 1 --2- 172.21.15.16:0/32184924 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f70840c9b40 0x7f70840afb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:25.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:25.511+0000 7f7082ffd640 1 -- 172.21.15.16:0/32184924 >> v2:172.21.15.62:3300/0 conn(0x7f7084070df0 msgr2=0x7f7084071210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:28:25.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:25.511+0000 7f7082ffd640 1 --2- 172.21.15.16:0/32184924 >> v2:172.21.15.62:3300/0 conn(0x7f7084070df0 0x7f7084071210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:25.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:25.511+0000 7f7082ffd640 1 -- 172.21.15.16:0/32184924 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f708415dc60 con 0x7f70840ca4e0 2023-05-22T15:28:25.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:25.511+0000 7f7082ffd640 1 --2- 172.21.15.16:0/32184924 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f70840ca4e0 0x7f708415e460 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f70840b0a40 tx=0x7f706c01c440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9761d340d163ba9d server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:28:25.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:25.511+0000 7f7081ffb640 1 -- 172.21.15.16:0/32184924 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f706c0ac640 con 0x7f70840ca4e0 2023-05-22T15:28:25.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:25.511+0000 7f7081ffb640 1 -- 172.21.15.16:0/32184924 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f706c0ac7a0 con 0x7f70840ca4e0 2023-05-22T15:28:25.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:25.511+0000 7f7081ffb640 1 -- 172.21.15.16:0/32184924 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f706c0acb30 con 0x7f70840ca4e0 2023-05-22T15:28:25.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:25.511+0000 7f7089447640 1 -- 172.21.15.16:0/32184924 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f70840ca4e0 msgr2=0x7f708415e460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:25.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:25.511+0000 7f7089447640 1 --2- 172.21.15.16:0/32184924 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f70840ca4e0 0x7f708415e460 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f70840b0a40 tx=0x7f706c01c440 comp rx=0 tx=0).stop 2023-05-22T15:28:25.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:25.515+0000 7f7089447640 1 -- 172.21.15.16:0/32184924 shutdown_connections 2023-05-22T15:28:25.530 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:25.515+0000 7f7089447640 1 --2- 172.21.15.16:0/32184924 >> v2:172.21.15.62:3300/0 conn(0x7f7084070df0 0x7f7084071210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:25.530 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:25.515+0000 7f7089447640 1 --2- 172.21.15.16:0/32184924 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f70840c9b40 0x7f70840afb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:25.530 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:25.515+0000 7f7089447640 1 --2- 172.21.15.16:0/32184924 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f70840ca4e0 0x7f708415e460 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:25.531 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:25.515+0000 7f7089447640 1 -- 172.21.15.16:0/32184924 >> 172.21.15.16:0/32184924 conn(0x7f7084069720 msgr2=0x7f7084069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:28:25.531 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:25.515+0000 7f7089447640 1 -- 172.21.15.16:0/32184924 shutdown_connections 2023-05-22T15:28:25.531 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:25.515+0000 7f7089447640 1 -- 172.21.15.16:0/32184924 wait complete. 2023-05-22T15:28:25.531 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:25.515+0000 7f7089447640 1 Processor -- start 2023-05-22T15:28:25.532 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:25.515+0000 7f7089447640 1 -- start start 2023-05-22T15:28:25.532 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:25.515+0000 7f7089447640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7084070df0 0x7f7084151dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:25.532 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:25.515+0000 7f7089447640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f70840c9b40 0x7f708414c170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:25.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:25.515+0000 7f7089447640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f70840ca4e0 0x7f708414b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:25.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:25.515+0000 7f7089447640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f708415bd30 con 0x7f70840c9b40 2023-05-22T15:28:25.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:25.515+0000 7f7089447640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f708415bfd0 con 0x7f7084070df0 2023-05-22T15:28:25.534 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:25.515+0000 7f7089447640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f708415c2b0 con 0x7f70840ca4e0 2023-05-22T15:28:25.534 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:25.515+0000 7f70827fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f70840c9b40 0x7f708414c170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:25.534 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:25.515+0000 7f70827fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f70840c9b40 0x7f708414c170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52110/0 (socket says 172.21.15.16:52110) 2023-05-22T15:28:25.534 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:25.515+0000 7f70827fc640 1 -- 172.21.15.16:0/2915492550 learned_addr learned my addr 172.21.15.16:0/2915492550 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:28:25.534 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:25.515+0000 7f70837fe640 1 --2- 172.21.15.16:0/2915492550 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f70840ca4e0 0x7f708414b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:25.535 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:25.515+0000 7f7082ffd640 1 --2- 172.21.15.16:0/2915492550 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7084070df0 0x7f7084151dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:25.535 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:25.515+0000 7f70827fc640 1 -- 172.21.15.16:0/2915492550 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f70840ca4e0 msgr2=0x7f708414b5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:25.535 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:25.515+0000 7f70827fc640 1 --2- 172.21.15.16:0/2915492550 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f70840ca4e0 0x7f708414b5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:25.535 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:25.515+0000 7f70827fc640 1 -- 172.21.15.16:0/2915492550 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7084070df0 msgr2=0x7f7084151dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:25.536 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:25.515+0000 7f70827fc640 1 --2- 172.21.15.16:0/2915492550 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7084070df0 0x7f7084151dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:25.536 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:25.515+0000 7f70827fc640 1 -- 172.21.15.16:0/2915492550 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f706c0094d0 con 0x7f70840c9b40 2023-05-22T15:28:25.536 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:25.515+0000 7f7082ffd640 1 --2- 172.21.15.16:0/2915492550 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7084070df0 0x7f7084151dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:28:25.536 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:25.515+0000 7f70827fc640 1 --2- 172.21.15.16:0/2915492550 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f70840c9b40 0x7f708414c170 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f707400d800 tx=0x7f707400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:28:25.537 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:25.515+0000 7f707b7fe640 1 -- 172.21.15.16:0/2915492550 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f707400b6c0 con 0x7f70840c9b40 2023-05-22T15:28:25.537 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:25.515+0000 7f707b7fe640 1 -- 172.21.15.16:0/2915492550 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f707400b820 con 0x7f70840c9b40 2023-05-22T15:28:25.537 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:25.515+0000 7f7089447640 1 -- 172.21.15.16:0/2915492550 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70840ca8b0 con 0x7f70840c9b40 2023-05-22T15:28:25.537 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:25.515+0000 7f707b7fe640 1 -- 172.21.15.16:0/2915492550 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f707400bb90 con 0x7f70840c9b40 2023-05-22T15:28:25.537 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:25.515+0000 7f7089447640 1 -- 172.21.15.16:0/2915492550 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7084071750 con 0x7f70840c9b40 2023-05-22T15:28:25.538 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:25.515+0000 7f7089447640 1 -- 172.21.15.16:0/2915492550 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70840c8e80 con 0x7f70840c9b40 2023-05-22T15:28:25.538 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:25.519+0000 7f707b7fe640 1 -- 172.21.15.16:0/2915492550 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7f707400bcf0 con 0x7f70840c9b40 2023-05-22T15:28:25.538 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:25.519+0000 7f707b7fe640 1 --2- 172.21.15.16:0/2915492550 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f7064065380 0x7f7064067830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:25.538 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:25.519+0000 7f7082ffd640 1 --2- 172.21.15.16:0/2915492550 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f7064065380 0x7f7064067830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:25.538 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:25.519+0000 7f707b7fe640 1 -- 172.21.15.16:0/2915492550 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f7074082f00 con 0x7f70840c9b40 2023-05-22T15:28:25.539 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:25.519+0000 7f7082ffd640 1 --2- 172.21.15.16:0/2915492550 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f7064065380 0x7f7064067830 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f706c0ae000 tx=0x7f706c008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:28:25.539 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:25.523+0000 7f707b7fe640 1 -- 172.21.15.16:0/2915492550 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f7074054470 con 0x7f70840c9b40 2023-05-22T15:28:25.702 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:25.695+0000 7f7089447640 1 -- 172.21.15.16:0/2915492550 --> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f708404dbe0 con 0x7f7064065380 2023-05-22T15:28:25.704 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:25.699+0000 7f707b7fe640 1 -- 172.21.15.16:0/2915492550 <== mgr.4101 v2:172.21.15.16:6832/36921 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f708404dbe0 con 0x7f7064065380 2023-05-22T15:28:25.705 INFO:teuthology.orchestra.run.smithi016.stdout:[ 2023-05-22T15:28:25.705 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T15:28:25.705 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "cephfs" 2023-05-22T15:28:25.705 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T15:28:25.706 INFO:teuthology.orchestra.run.smithi016.stdout:] 2023-05-22T15:28:25.707 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:25.699+0000 7f7089447640 1 -- 172.21.15.16:0/2915492550 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f7064065380 msgr2=0x7f7064067830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:25.707 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:25.699+0000 7f7089447640 1 --2- 172.21.15.16:0/2915492550 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f7064065380 0x7f7064067830 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f706c0ae000 tx=0x7f706c008040 comp rx=0 tx=0).stop 2023-05-22T15:28:25.707 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:25.699+0000 7f7089447640 1 -- 172.21.15.16:0/2915492550 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f70840c9b40 msgr2=0x7f708414c170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:25.708 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:25.699+0000 7f7089447640 1 --2- 172.21.15.16:0/2915492550 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f70840c9b40 0x7f708414c170 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f707400d800 tx=0x7f707400dcc0 comp rx=0 tx=0).stop 2023-05-22T15:28:25.708 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:25.699+0000 7f7089447640 1 -- 172.21.15.16:0/2915492550 shutdown_connections 2023-05-22T15:28:25.708 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:25.699+0000 7f7089447640 1 --2- 172.21.15.16:0/2915492550 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f7064065380 0x7f7064067830 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:25.708 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:25.699+0000 7f7089447640 1 --2- 172.21.15.16:0/2915492550 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f70840ca4e0 0x7f708414b5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:25.709 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:25.699+0000 7f7089447640 1 --2- 172.21.15.16:0/2915492550 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f70840c9b40 0x7f708414c170 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:25.709 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:25.699+0000 7f7089447640 1 --2- 172.21.15.16:0/2915492550 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7084070df0 0x7f7084151dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:25.709 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:25.699+0000 7f7089447640 1 -- 172.21.15.16:0/2915492550 >> 172.21.15.16:0/2915492550 conn(0x7f7084069720 msgr2=0x7f70840b1040 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:28:25.709 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:25.699+0000 7f7089447640 1 -- 172.21.15.16:0/2915492550 shutdown_connections 2023-05-22T15:28:25.710 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:25.699+0000 7f7089447640 1 -- 172.21.15.16:0/2915492550 wait complete. 2023-05-22T15:28:25.722 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2023-05-22T15:28:25.847 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:25.839+0000 7f20a9990640 1 Processor -- start 2023-05-22T15:28:25.848 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:25.839+0000 7f20a9990640 1 -- start start 2023-05-22T15:28:25.850 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:25.843+0000 7f20a9990640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f20a406b6c0 0x7f20a40cb4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:25.851 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:25.843+0000 7f20a9990640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20a4159f70 con 0x7f20a41596e0 2023-05-22T15:28:25.851 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:25.843+0000 7f20a9990640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20a40cb030 con 0x7f20a406ba90 2023-05-22T15:28:25.851 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:25.843+0000 7f20a9990640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20a4071760 con 0x7f20a406b6c0 2023-05-22T15:28:25.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:25.843+0000 7f20a2ffd640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f20a406b6c0 0x7f20a40cb4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:25.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:25.843+0000 7f20a37fe640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f20a41596e0 0x7f20a406fba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:33806/0 (socket says 172.21.15.16:33806) 2023-05-22T15:28:25.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:25.843+0000 7f20a37fe640 1 -- 172.21.15.16:0/4113072954 learned_addr learned my addr 172.21.15.16:0/4113072954 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:28:25.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:25.843+0000 7f20a2ffd640 1 -- 172.21.15.16:0/4113072954 >> v1:172.21.15.62:6789/0 conn(0x7f20a406ba90 legacy=0x7f20a4158fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:28:25.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:25.843+0000 7f20a2ffd640 1 -- 172.21.15.16:0/4113072954 >> v1:172.21.15.16:6789/0 conn(0x7f20a41596e0 legacy=0x7f20a406fba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:25.853 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:25.843+0000 7f20a2ffd640 1 -- 172.21.15.16:0/4113072954 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20a4159c90 con 0x7f20a406b6c0 2023-05-22T15:28:25.855 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:25.843+0000 7f20a2ffd640 1 --2- 172.21.15.16:0/4113072954 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f20a406b6c0 0x7f20a40cb4c0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f20940097c0 tx=0x7f209401c3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b73d14453913b2c server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:28:25.855 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:25.843+0000 7f20a1ffb640 1 -- 172.21.15.16:0/4113072954 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20940ab600 con 0x7f20a406b6c0 2023-05-22T15:28:25.856 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:25.843+0000 7f20a1ffb640 1 -- 172.21.15.16:0/4113072954 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20940a7050 con 0x7f20a406b6c0 2023-05-22T15:28:25.856 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:25.843+0000 7f20a1ffb640 1 -- 172.21.15.16:0/4113072954 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20940ab980 con 0x7f20a406b6c0 2023-05-22T15:28:25.856 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:25.843+0000 7f20a9990640 1 -- 172.21.15.16:0/4113072954 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f20a406b6c0 msgr2=0x7f20a40cb4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:25.857 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:25.843+0000 7f20a9990640 1 --2- 172.21.15.16:0/4113072954 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f20a406b6c0 0x7f20a40cb4c0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f20940097c0 tx=0x7f209401c3b0 comp rx=0 tx=0).stop 2023-05-22T15:28:25.857 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:25.843+0000 7f20a9990640 1 -- 172.21.15.16:0/4113072954 shutdown_connections 2023-05-22T15:28:25.857 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:25.843+0000 7f20a9990640 1 --2- 172.21.15.16:0/4113072954 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f20a406b6c0 0x7f20a40cb4c0 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:25.857 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:25.843+0000 7f20a9990640 1 -- 172.21.15.16:0/4113072954 >> 172.21.15.16:0/4113072954 conn(0x7f20a4069720 msgr2=0x7f20a4069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:28:25.858 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:25.843+0000 7f20a9990640 1 -- 172.21.15.16:0/4113072954 shutdown_connections 2023-05-22T15:28:25.858 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:25.843+0000 7f20a9990640 1 -- 172.21.15.16:0/4113072954 wait complete. 2023-05-22T15:28:25.858 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:25.847+0000 7f20a9990640 1 Processor -- start 2023-05-22T15:28:25.858 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:25.847+0000 7f20a9990640 1 -- start start 2023-05-22T15:28:25.859 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:25.847+0000 7f20a9990640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f20a406b6c0 0x7f20a40af370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:25.859 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:25.847+0000 7f20a9990640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f20a406ba90 0x7f20a40b02f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:25.859 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:25.847+0000 7f20a9990640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f20a41596e0 0x7f20a414a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:25.859 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:25.847+0000 7f20a9990640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20a40aeee0 con 0x7f20a41596e0 2023-05-22T15:28:25.860 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:25.847+0000 7f20a9990640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20a40b09f0 con 0x7f20a406b6c0 2023-05-22T15:28:25.861 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:25.847+0000 7f20a9990640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20a40ae5f0 con 0x7f20a406ba90 2023-05-22T15:28:25.861 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:25.847+0000 7f20a37fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f20a41596e0 0x7f20a414a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:25.861 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:25.847+0000 7f20a37fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f20a41596e0 0x7f20a414a9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52124/0 (socket says 172.21.15.16:52124) 2023-05-22T15:28:25.862 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:25.847+0000 7f20a37fe640 1 -- 172.21.15.16:0/611533012 learned_addr learned my addr 172.21.15.16:0/611533012 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:28:25.862 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:25.847+0000 7f20a2ffd640 1 --2- 172.21.15.16:0/611533012 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f20a406b6c0 0x7f20a40af370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:25.862 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:25.847+0000 7f20a27fc640 1 --2- 172.21.15.16:0/611533012 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f20a406ba90 0x7f20a40b02f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:25.862 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:25.847+0000 7f20a37fe640 1 -- 172.21.15.16:0/611533012 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f20a406ba90 msgr2=0x7f20a40b02f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:25.863 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:25.847+0000 7f20a37fe640 1 --2- 172.21.15.16:0/611533012 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f20a406ba90 0x7f20a40b02f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:25.863 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:25.847+0000 7f20a37fe640 1 -- 172.21.15.16:0/611533012 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f20a406b6c0 msgr2=0x7f20a40af370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:25.863 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:25.847+0000 7f20a37fe640 1 --2- 172.21.15.16:0/611533012 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f20a406b6c0 0x7f20a40af370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:25.864 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:25.847+0000 7f20a37fe640 1 -- 172.21.15.16:0/611533012 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2094009420 con 0x7f20a41596e0 2023-05-22T15:28:25.864 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:25.847+0000 7f20a27fc640 1 --2- 172.21.15.16:0/611533012 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f20a406ba90 0x7f20a40b02f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:28:25.864 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:25.847+0000 7f20a37fe640 1 --2- 172.21.15.16:0/611533012 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f20a41596e0 0x7f20a414a9b0 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f209c00cbc0 tx=0x7f209c00a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:28:25.865 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:25.847+0000 7f208bfff640 1 -- 172.21.15.16:0/611533012 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f209c00ad40 con 0x7f20a41596e0 2023-05-22T15:28:25.865 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:25.847+0000 7f208bfff640 1 -- 172.21.15.16:0/611533012 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f209c012400 con 0x7f20a41596e0 2023-05-22T15:28:25.865 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:25.847+0000 7f208bfff640 1 -- 172.21.15.16:0/611533012 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f209c00aea0 con 0x7f20a41596e0 2023-05-22T15:28:25.865 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:25.847+0000 7f20a9990640 1 -- 172.21.15.16:0/611533012 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20a4156840 con 0x7f20a41596e0 2023-05-22T15:28:25.865 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:25.847+0000 7f20a9990640 1 -- 172.21.15.16:0/611533012 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20a406bee0 con 0x7f20a41596e0 2023-05-22T15:28:25.867 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:25.847+0000 7f20a9990640 1 -- 172.21.15.16:0/611533012 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20a41569d0 con 0x7f20a41596e0 2023-05-22T15:28:25.868 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:25.851+0000 7f208bfff640 1 -- 172.21.15.16:0/611533012 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 84230+0+0 (secure 0 0 0) 0x7f209c007070 con 0x7f20a41596e0 2023-05-22T15:28:25.868 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:25.851+0000 7f208bfff640 1 --2- 172.21.15.16:0/611533012 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f20900653b0 0x7f2090067860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:25.868 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:25.851+0000 7f208bfff640 1 -- 172.21.15.16:0/611533012 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f209c082770 con 0x7f20a41596e0 2023-05-22T15:28:25.868 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:25.851+0000 7f20a2ffd640 1 --2- 172.21.15.16:0/611533012 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f20900653b0 0x7f2090067860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:25.868 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:25.851+0000 7f20a2ffd640 1 --2- 172.21.15.16:0/611533012 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f20900653b0 0x7f2090067860 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f20940ad000 tx=0x7f2094008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:28:25.868 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:25.855+0000 7f208bfff640 1 -- 172.21.15.16:0/611533012 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f209c04a620 con 0x7f20a41596e0 2023-05-22T15:28:26.053 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:26.047+0000 7f20a9990640 1 -- 172.21.15.16:0/611533012 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7f20a404dbe0 con 0x7f20a41596e0 2023-05-22T15:28:27.058 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.051+0000 7f208bfff640 1 -- 172.21.15.16:0/611533012 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mgrmap(e 6) v1 ==== 84247+0+0 (secure 0 0 0) 0x7f209c050760 con 0x7f20a41596e0 2023-05-22T15:28:27.059 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.051+0000 7f208bfff640 1 -- 172.21.15.16:0/611533012 <== mon.0 v2:172.21.15.16:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v6) v1 ==== 92+0+0 (secure 0 0 0) 0x7f209c053e20 con 0x7f20a41596e0 2023-05-22T15:28:27.062 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.055+0000 7f20a9990640 1 -- 172.21.15.16:0/611533012 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f20900653b0 msgr2=0x7f2090067860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:27.062 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.055+0000 7f20a9990640 1 --2- 172.21.15.16:0/611533012 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f20900653b0 0x7f2090067860 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f20940ad000 tx=0x7f2094008040 comp rx=0 tx=0).stop 2023-05-22T15:28:27.063 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.055+0000 7f20a9990640 1 -- 172.21.15.16:0/611533012 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f20a41596e0 msgr2=0x7f20a414a9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:27.063 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.055+0000 7f20a9990640 1 --2- 172.21.15.16:0/611533012 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f20a41596e0 0x7f20a414a9b0 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f209c00cbc0 tx=0x7f209c00a430 comp rx=0 tx=0).stop 2023-05-22T15:28:27.063 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.055+0000 7f20a9990640 1 -- 172.21.15.16:0/611533012 shutdown_connections 2023-05-22T15:28:27.063 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.055+0000 7f20a9990640 1 --2- 172.21.15.16:0/611533012 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f20a41596e0 0x7f20a414a9b0 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:27.063 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.055+0000 7f20a9990640 1 --2- 172.21.15.16:0/611533012 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f20900653b0 0x7f2090067860 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:27.063 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.055+0000 7f20a9990640 1 --2- 172.21.15.16:0/611533012 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f20a406ba90 0x7f20a40b02f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:27.063 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.055+0000 7f20a9990640 1 --2- 172.21.15.16:0/611533012 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f20a406b6c0 0x7f20a40af370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:27.064 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.055+0000 7f20a9990640 1 -- 172.21.15.16:0/611533012 >> 172.21.15.16:0/611533012 conn(0x7f20a4069720 msgr2=0x7f20a4069af0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:28:27.064 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.055+0000 7f20a9990640 1 -- 172.21.15.16:0/611533012 shutdown_connections 2023-05-22T15:28:27.064 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.055+0000 7f20a9990640 1 -- 172.21.15.16:0/611533012 wait complete. 2023-05-22T15:28:27.082 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2023-05-22T15:28:27.184 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:28:27.175+0000 7f89dc3cf180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-05-22T15:28:27.184 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:28:27.174+0000 7f057ebf7180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-05-22T15:28:27.200 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.195+0000 7f0a5ea22640 1 Processor -- start 2023-05-22T15:28:27.201 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.195+0000 7f0a5ea22640 1 -- start start 2023-05-22T15:28:27.202 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.195+0000 7f0a5ea22640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f0a580ade80 0x7f0a580cc400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:27.202 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.195+0000 7f0a5ea22640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f0a5815c350 0x7f0a5815c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:27.202 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.195+0000 7f0a5ea22640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a580c95f0 con 0x7f0a58158ca0 2023-05-22T15:28:27.202 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.195+0000 7f0a5ea22640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a580ca270 con 0x7f0a5815c350 2023-05-22T15:28:27.203 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.195+0000 7f0a5ea22640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a5815cd30 con 0x7f0a580ade80 2023-05-22T15:28:27.203 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.195+0000 7f0a577fe640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f0a5815c350 0x7f0a5815c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:27.203 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.195+0000 7f0a5cfbf640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f0a58158ca0 0x7f0a581590c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:33820/0 (socket says 172.21.15.16:33820) 2023-05-22T15:28:27.203 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.195+0000 7f0a5cfbf640 1 -- 172.21.15.16:0/3703425443 learned_addr learned my addr 172.21.15.16:0/3703425443 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:28:27.203 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.195+0000 7f0a57fff640 1 --2- 172.21.15.16:0/3703425443 >> v2:172.21.15.62:3300/0 conn(0x7f0a580ade80 0x7f0a580cc400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:27.204 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.195+0000 7f0a577fe640 1 -- 172.21.15.16:0/3703425443 >> v2:172.21.15.62:3300/0 conn(0x7f0a580ade80 msgr2=0x7f0a580cc400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:28:27.204 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.195+0000 7f0a577fe640 1 --2- 172.21.15.16:0/3703425443 >> v2:172.21.15.62:3300/0 conn(0x7f0a580ade80 0x7f0a580cc400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:27.204 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.195+0000 7f0a577fe640 1 -- 172.21.15.16:0/3703425443 >> v1:172.21.15.16:6789/0 conn(0x7f0a58158ca0 legacy=0x7f0a581590c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:27.204 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.195+0000 7f0a56ffd640 1 -- 172.21.15.16:0/3703425443 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2954186334 0 0) 0x7f0a580c95f0 con 0x7f0a58158ca0 2023-05-22T15:28:27.204 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.195+0000 7f0a577fe640 1 -- 172.21.15.16:0/3703425443 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a580c98d0 con 0x7f0a5815c350 2023-05-22T15:28:27.205 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.195+0000 7f0a577fe640 1 --2- 172.21.15.16:0/3703425443 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0a5815c350 0x7f0a5815c7f0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f0a580cf660 tx=0x7f0a4c01c3d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c54365f002a8a235 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:28:27.205 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.199+0000 7f0a56ffd640 1 -- 172.21.15.16:0/3703425443 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a4c0ab600 con 0x7f0a5815c350 2023-05-22T15:28:27.205 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.199+0000 7f0a56ffd640 1 -- 172.21.15.16:0/3703425443 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a4c0a7050 con 0x7f0a5815c350 2023-05-22T15:28:27.205 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.199+0000 7f0a56ffd640 1 -- 172.21.15.16:0/3703425443 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a4c0ab980 con 0x7f0a5815c350 2023-05-22T15:28:27.210 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.199+0000 7f0a5ea22640 1 -- 172.21.15.16:0/3703425443 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0a5815c350 msgr2=0x7f0a5815c7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:27.210 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.199+0000 7f0a5ea22640 1 --2- 172.21.15.16:0/3703425443 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0a5815c350 0x7f0a5815c7f0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f0a580cf660 tx=0x7f0a4c01c3d0 comp rx=0 tx=0).stop 2023-05-22T15:28:27.210 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.199+0000 7f0a5ea22640 1 -- 172.21.15.16:0/3703425443 shutdown_connections 2023-05-22T15:28:27.210 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.199+0000 7f0a5ea22640 1 --2- 172.21.15.16:0/3703425443 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0a5815c350 0x7f0a5815c7f0 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:27.211 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.199+0000 7f0a5ea22640 1 --2- 172.21.15.16:0/3703425443 >> v2:172.21.15.62:3300/0 conn(0x7f0a580ade80 0x7f0a580cc400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:27.211 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.199+0000 7f0a5ea22640 1 -- 172.21.15.16:0/3703425443 >> 172.21.15.16:0/3703425443 conn(0x7f0a580b1110 msgr2=0x7f0a580ce830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:28:27.211 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.199+0000 7f0a5ea22640 1 -- 172.21.15.16:0/3703425443 shutdown_connections 2023-05-22T15:28:27.211 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.199+0000 7f0a5ea22640 1 -- 172.21.15.16:0/3703425443 wait complete. 2023-05-22T15:28:27.211 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.199+0000 7f0a5ea22640 1 Processor -- start 2023-05-22T15:28:27.211 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.199+0000 7f0a5ea22640 1 -- start start 2023-05-22T15:28:27.211 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.199+0000 7f0a5ea22640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0a580ade80 0x7f0a5815b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:27.211 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.199+0000 7f0a5ea22640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0a58158ca0 0x7f0a58158500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:27.212 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.199+0000 7f0a5ea22640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0a5815c350 0x7f0a580cea90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:27.212 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.199+0000 7f0a5ea22640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a58158a40 con 0x7f0a58158ca0 2023-05-22T15:28:27.212 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.199+0000 7f0a5ea22640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a5815ad40 con 0x7f0a5815c350 2023-05-22T15:28:27.212 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.199+0000 7f0a5ea22640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a58070990 con 0x7f0a580ade80 2023-05-22T15:28:27.212 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.199+0000 7f0a577fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0a58158ca0 0x7f0a58158500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:27.212 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.199+0000 7f0a577fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0a58158ca0 0x7f0a58158500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52160/0 (socket says 172.21.15.16:52160) 2023-05-22T15:28:27.212 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.199+0000 7f0a577fe640 1 -- 172.21.15.16:0/2133768759 learned_addr learned my addr 172.21.15.16:0/2133768759 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:28:27.213 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.199+0000 7f0a5cfbf640 1 --2- 172.21.15.16:0/2133768759 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0a5815c350 0x7f0a580cea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:27.213 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.199+0000 7f0a57fff640 1 --2- 172.21.15.16:0/2133768759 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0a580ade80 0x7f0a5815b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:27.213 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.199+0000 7f0a577fe640 1 -- 172.21.15.16:0/2133768759 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0a580ade80 msgr2=0x7f0a5815b750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:27.213 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.199+0000 7f0a577fe640 1 --2- 172.21.15.16:0/2133768759 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0a580ade80 0x7f0a5815b750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:27.213 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.199+0000 7f0a577fe640 1 -- 172.21.15.16:0/2133768759 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0a5815c350 msgr2=0x7f0a580cea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:27.213 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.199+0000 7f0a577fe640 1 --2- 172.21.15.16:0/2133768759 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0a5815c350 0x7f0a580cea90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:27.213 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.199+0000 7f0a577fe640 1 -- 172.21.15.16:0/2133768759 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a4c009420 con 0x7f0a58158ca0 2023-05-22T15:28:27.213 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.199+0000 7f0a577fe640 1 --2- 172.21.15.16:0/2133768759 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0a58158ca0 0x7f0a58158500 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f0a4c0ae040 tx=0x7f0a4c019170 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:28:27.214 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.199+0000 7f0a54ff9640 1 -- 172.21.15.16:0/2133768759 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a4c0b4610 con 0x7f0a58158ca0 2023-05-22T15:28:27.214 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.199+0000 7f0a54ff9640 1 -- 172.21.15.16:0/2133768759 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a4c0b4770 con 0x7f0a58158ca0 2023-05-22T15:28:27.214 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.199+0000 7f0a5ea22640 1 -- 172.21.15.16:0/2133768759 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a5815d090 con 0x7f0a58158ca0 2023-05-22T15:28:27.214 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.199+0000 7f0a54ff9640 1 -- 172.21.15.16:0/2133768759 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a4c0b4b20 con 0x7f0a58158ca0 2023-05-22T15:28:27.214 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.203+0000 7f0a5ea22640 1 -- 172.21.15.16:0/2133768759 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a580cc940 con 0x7f0a58158ca0 2023-05-22T15:28:27.214 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.203+0000 7f0a5ea22640 1 -- 172.21.15.16:0/2133768759 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a580c9210 con 0x7f0a58158ca0 2023-05-22T15:28:27.214 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.203+0000 7f0a54ff9640 1 -- 172.21.15.16:0/2133768759 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 6) v1 ==== 84247+0+0 (secure 0 0 0) 0x7f0a4c0b2040 con 0x7f0a58158ca0 2023-05-22T15:28:27.214 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.203+0000 7f0a54ff9640 1 --2- 172.21.15.16:0/2133768759 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f0a440653d0 0x7f0a44067880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:27.215 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.203+0000 7f0a57fff640 1 -- 172.21.15.16:0/2133768759 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f0a440653d0 msgr2=0x7f0a44067880 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/36921 2023-05-22T15:28:27.215 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.203+0000 7f0a54ff9640 1 -- 172.21.15.16:0/2133768759 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f0a4c11ca20 con 0x7f0a58158ca0 2023-05-22T15:28:27.215 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.203+0000 7f0a57fff640 1 --2- 172.21.15.16:0/2133768759 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f0a440653d0 0x7f0a44067880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-05-22T15:28:27.215 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.207+0000 7f0a54ff9640 1 -- 172.21.15.16:0/2133768759 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f0a4c0ea560 con 0x7f0a58158ca0 2023-05-22T15:28:27.365 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.359+0000 7f0a5ea22640 1 -- 172.21.15.16:0/2133768759 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7f0a5804dbe0 con 0x7f0a58158ca0 2023-05-22T15:28:27.371 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.363+0000 7f0a54ff9640 1 -- 172.21.15.16:0/2133768759 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v11)=0 v11) v1 ==== 138+0+0 (secure 0 0 0) 0x7f0a4c0c20d0 con 0x7f0a58158ca0 2023-05-22T15:28:27.374 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.367+0000 7f0a5ea22640 1 -- 172.21.15.16:0/2133768759 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f0a440653d0 msgr2=0x7f0a44067880 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-05-22T15:28:27.374 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.367+0000 7f0a5ea22640 1 --2- 172.21.15.16:0/2133768759 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f0a440653d0 0x7f0a44067880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:27.374 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.367+0000 7f0a5ea22640 1 -- 172.21.15.16:0/2133768759 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0a58158ca0 msgr2=0x7f0a58158500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:27.375 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.367+0000 7f0a5ea22640 1 --2- 172.21.15.16:0/2133768759 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0a58158ca0 0x7f0a58158500 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f0a4c0ae040 tx=0x7f0a4c019170 comp rx=0 tx=0).stop 2023-05-22T15:28:27.375 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.367+0000 7f0a5ea22640 1 -- 172.21.15.16:0/2133768759 shutdown_connections 2023-05-22T15:28:27.375 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.367+0000 7f0a5ea22640 1 --2- 172.21.15.16:0/2133768759 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0a5815c350 0x7f0a580cea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:27.375 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.367+0000 7f0a5ea22640 1 --2- 172.21.15.16:0/2133768759 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0a58158ca0 0x7f0a58158500 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:27.375 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.367+0000 7f0a5ea22640 1 --2- 172.21.15.16:0/2133768759 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f0a440653d0 0x7f0a44067880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:27.375 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.367+0000 7f0a5ea22640 1 --2- 172.21.15.16:0/2133768759 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0a580ade80 0x7f0a5815b750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:27.375 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.367+0000 7f0a5ea22640 1 -- 172.21.15.16:0/2133768759 >> 172.21.15.16:0/2133768759 conn(0x7f0a580b1110 msgr2=0x7f0a58147c10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:28:27.375 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.367+0000 7f0a5ea22640 1 -- 172.21.15.16:0/2133768759 shutdown_connections 2023-05-22T15:28:27.376 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.367+0000 7f0a5ea22640 1 -- 172.21.15.16:0/2133768759 wait complete. 2023-05-22T15:28:27.390 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2023-05-22T15:28:27.509 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.503+0000 7fb374722640 1 Processor -- start 2023-05-22T15:28:27.509 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.503+0000 7fb374722640 1 -- start start 2023-05-22T15:28:27.510 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.503+0000 7fb374722640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb36c158a10 0x7fb36c06e480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:27.510 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.503+0000 7fb374722640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fb36c070760 0x7fb36c070be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:27.511 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.503+0000 7fb374722640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb36c158de0 con 0x7fb36c06c6e0 2023-05-22T15:28:27.511 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.503+0000 7fb374722640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb36c06e060 con 0x7fb36c158a10 2023-05-22T15:28:27.511 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.503+0000 7fb374722640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb36c0711e0 con 0x7fb36c070760 2023-05-22T15:28:27.511 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.503+0000 7fb3724be640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb36c158a10 0x7fb36c06e480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:27.511 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.503+0000 7fb3724be640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb36c158a10 0x7fb36c06e480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52176/0 (socket says 172.21.15.16:52176) 2023-05-22T15:28:27.511 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.503+0000 7fb372cbf640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fb36c06c6e0 0x7fb36c0c9a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:33822/0 (socket says 172.21.15.16:33822) 2023-05-22T15:28:27.511 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.503+0000 7fb3724be640 1 -- 172.21.15.16:0/4101249692 learned_addr learned my addr 172.21.15.16:0/4101249692 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:28:27.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.503+0000 7fb371cbd640 1 --2- 172.21.15.16:0/4101249692 >> v2:172.21.15.62:3300/0 conn(0x7fb36c070760 0x7fb36c070be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:27.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.503+0000 7fb3724be640 1 -- 172.21.15.16:0/4101249692 >> v2:172.21.15.62:3300/0 conn(0x7fb36c070760 msgr2=0x7fb36c070be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:28:27.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.503+0000 7fb3724be640 1 --2- 172.21.15.16:0/4101249692 >> v2:172.21.15.62:3300/0 conn(0x7fb36c070760 0x7fb36c070be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:27.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.503+0000 7fb3724be640 1 -- 172.21.15.16:0/4101249692 >> v1:172.21.15.16:6789/0 conn(0x7fb36c06c6e0 legacy=0x7fb36c0c9a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:27.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.503+0000 7fb3724be640 1 -- 172.21.15.16:0/4101249692 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb36c0714c0 con 0x7fb36c158a10 2023-05-22T15:28:27.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.503+0000 7fb3714bc640 1 -- 172.21.15.16:0/4101249692 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2023462284 0 0) 0x7fb36c158de0 con 0x7fb36c06c6e0 2023-05-22T15:28:27.513 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.507+0000 7fb3724be640 1 --2- 172.21.15.16:0/4101249692 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb36c158a10 0x7fb36c06e480 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7fb36c06d780 tx=0x7fb36801eb40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd105afe05502a6 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:28:27.513 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.507+0000 7fb3714bc640 1 -- 172.21.15.16:0/4101249692 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3680a7a20 con 0x7fb36c158a10 2023-05-22T15:28:27.513 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.507+0000 7fb3714bc640 1 -- 172.21.15.16:0/4101249692 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3680a7b80 con 0x7fb36c158a10 2023-05-22T15:28:27.514 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.507+0000 7fb3714bc640 1 -- 172.21.15.16:0/4101249692 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3680af440 con 0x7fb36c158a10 2023-05-22T15:28:27.514 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.507+0000 7fb374722640 1 -- 172.21.15.16:0/4101249692 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb36c158a10 msgr2=0x7fb36c06e480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:27.514 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.507+0000 7fb374722640 1 --2- 172.21.15.16:0/4101249692 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb36c158a10 0x7fb36c06e480 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7fb36c06d780 tx=0x7fb36801eb40 comp rx=0 tx=0).stop 2023-05-22T15:28:27.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.507+0000 7fb374722640 1 -- 172.21.15.16:0/4101249692 shutdown_connections 2023-05-22T15:28:27.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.507+0000 7fb374722640 1 --2- 172.21.15.16:0/4101249692 >> v2:172.21.15.62:3300/0 conn(0x7fb36c070760 0x7fb36c070be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:27.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.507+0000 7fb374722640 1 --2- 172.21.15.16:0/4101249692 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb36c158a10 0x7fb36c06e480 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:27.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.507+0000 7fb374722640 1 -- 172.21.15.16:0/4101249692 >> 172.21.15.16:0/4101249692 conn(0x7fb36c0cc870 msgr2=0x7fb36c0cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:28:27.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.507+0000 7fb374722640 1 -- 172.21.15.16:0/4101249692 shutdown_connections 2023-05-22T15:28:27.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.507+0000 7fb374722640 1 -- 172.21.15.16:0/4101249692 wait complete. 2023-05-22T15:28:27.516 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.511+0000 7fb374722640 1 Processor -- start 2023-05-22T15:28:27.516 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.511+0000 7fb374722640 1 -- start start 2023-05-22T15:28:27.518 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.511+0000 7fb374722640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb36c06c6e0 0x7fb36c157b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:27.519 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.511+0000 7fb374722640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb36c070760 0x7fb36c14a150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:27.519 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.511+0000 7fb374722640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb36c158a10 0x7fb36c0b21a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:27.519 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.511+0000 7fb374722640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb36c1526c0 con 0x7fb36c158a10 2023-05-22T15:28:27.519 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.511+0000 7fb374722640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb36c156010 con 0x7fb36c070760 2023-05-22T15:28:27.519 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.511+0000 7fb374722640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb36c1562f0 con 0x7fb36c06c6e0 2023-05-22T15:28:27.519 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.511+0000 7fb372cbf640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb36c158a10 0x7fb36c0b21a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:27.519 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.511+0000 7fb372cbf640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb36c158a10 0x7fb36c0b21a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52184/0 (socket says 172.21.15.16:52184) 2023-05-22T15:28:27.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.511+0000 7fb372cbf640 1 -- 172.21.15.16:0/1892143998 learned_addr learned my addr 172.21.15.16:0/1892143998 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:28:27.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.511+0000 7fb3724be640 1 --2- 172.21.15.16:0/1892143998 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb36c06c6e0 0x7fb36c157b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:27.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.511+0000 7fb371cbd640 1 --2- 172.21.15.16:0/1892143998 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb36c070760 0x7fb36c14a150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:27.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.511+0000 7fb372cbf640 1 -- 172.21.15.16:0/1892143998 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb36c06c6e0 msgr2=0x7fb36c157b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:27.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.511+0000 7fb372cbf640 1 --2- 172.21.15.16:0/1892143998 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb36c06c6e0 0x7fb36c157b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:27.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.511+0000 7fb372cbf640 1 -- 172.21.15.16:0/1892143998 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb36c070760 msgr2=0x7fb36c14a150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:27.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.511+0000 7fb372cbf640 1 --2- 172.21.15.16:0/1892143998 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb36c070760 0x7fb36c14a150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:27.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.511+0000 7fb372cbf640 1 -- 172.21.15.16:0/1892143998 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb36800b770 con 0x7fb36c158a10 2023-05-22T15:28:27.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.511+0000 7fb3724be640 1 --2- 172.21.15.16:0/1892143998 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb36c06c6e0 0x7fb36c157b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:28:27.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.511+0000 7fb371cbd640 1 --2- 172.21.15.16:0/1892143998 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb36c070760 0x7fb36c14a150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:28:27.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.511+0000 7fb372cbf640 1 --2- 172.21.15.16:0/1892143998 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb36c158a10 0x7fb36c0b21a0 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7fb36400c700 tx=0x7fb36400cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:28:27.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.511+0000 7fb362ffd640 1 -- 172.21.15.16:0/1892143998 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb36400ce10 con 0x7fb36c158a10 2023-05-22T15:28:27.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.511+0000 7fb362ffd640 1 -- 172.21.15.16:0/1892143998 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb36400a640 con 0x7fb36c158a10 2023-05-22T15:28:27.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.511+0000 7fb374722640 1 -- 172.21.15.16:0/1892143998 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb36c06e9c0 con 0x7fb36c158a10 2023-05-22T15:28:27.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.511+0000 7fb362ffd640 1 -- 172.21.15.16:0/1892143998 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb36400aa70 con 0x7fb36c158a10 2023-05-22T15:28:27.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.511+0000 7fb374722640 1 -- 172.21.15.16:0/1892143998 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb36c06cf50 con 0x7fb36c158a10 2023-05-22T15:28:27.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.511+0000 7fb374722640 1 -- 172.21.15.16:0/1892143998 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb36c06eb50 con 0x7fb36c158a10 2023-05-22T15:28:27.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.511+0000 7fb362ffd640 1 -- 172.21.15.16:0/1892143998 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 6) v1 ==== 84247+0+0 (secure 0 0 0) 0x7fb36400abd0 con 0x7fb36c158a10 2023-05-22T15:28:27.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.515+0000 7fb362ffd640 1 --2- 172.21.15.16:0/1892143998 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fb3500653d0 0x7fb350067880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:27.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.515+0000 7fb3724be640 1 -- 172.21.15.16:0/1892143998 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fb3500653d0 msgr2=0x7fb350067880 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/36921 2023-05-22T15:28:27.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.515+0000 7fb362ffd640 1 -- 172.21.15.16:0/1892143998 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6382+0+0 (secure 0 0 0) 0x7fb364082230 con 0x7fb36c158a10 2023-05-22T15:28:27.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.515+0000 7fb3724be640 1 --2- 172.21.15.16:0/1892143998 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fb3500653d0 0x7fb350067880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-05-22T15:28:27.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.515+0000 7fb362ffd640 1 -- 172.21.15.16:0/1892143998 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fb36404fcd0 con 0x7fb36c158a10 2023-05-22T15:28:27.566 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:28:27.558+0000 7f057ebf7180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-05-22T15:28:27.581 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:28:27.575+0000 7f89dc3cf180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-05-22T15:28:27.679 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.671+0000 7fb374722640 1 -- 172.21.15.16:0/1892143998 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7fb36c04dbe0 con 0x7fb36c158a10 2023-05-22T15:28:27.689 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.683+0000 7fb362ffd640 1 -- 172.21.15.16:0/1892143998 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v12)=0 v12) v1 ==== 133+0+0 (secure 0 0 0) 0x7fb3640532f0 con 0x7fb36c158a10 2023-05-22T15:28:27.693 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.687+0000 7fb374722640 1 -- 172.21.15.16:0/1892143998 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fb3500653d0 msgr2=0x7fb350067880 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-05-22T15:28:27.693 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.687+0000 7fb374722640 1 --2- 172.21.15.16:0/1892143998 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fb3500653d0 0x7fb350067880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:27.693 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.687+0000 7fb374722640 1 -- 172.21.15.16:0/1892143998 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb36c158a10 msgr2=0x7fb36c0b21a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:27.693 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.687+0000 7fb374722640 1 --2- 172.21.15.16:0/1892143998 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb36c158a10 0x7fb36c0b21a0 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7fb36400c700 tx=0x7fb36400cbc0 comp rx=0 tx=0).stop 2023-05-22T15:28:27.694 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.687+0000 7fb374722640 1 -- 172.21.15.16:0/1892143998 shutdown_connections 2023-05-22T15:28:27.694 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.687+0000 7fb374722640 1 --2- 172.21.15.16:0/1892143998 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb36c158a10 0x7fb36c0b21a0 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:27.694 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.687+0000 7fb374722640 1 --2- 172.21.15.16:0/1892143998 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb36c070760 0x7fb36c14a150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:27.694 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.687+0000 7fb374722640 1 --2- 172.21.15.16:0/1892143998 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fb3500653d0 0x7fb350067880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:27.694 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.687+0000 7fb374722640 1 --2- 172.21.15.16:0/1892143998 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb36c06c6e0 0x7fb36c157b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:27.696 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.687+0000 7fb374722640 1 -- 172.21.15.16:0/1892143998 >> 172.21.15.16:0/1892143998 conn(0x7fb36c0cc870 msgr2=0x7fb36c0cef90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:28:27.696 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.687+0000 7fb374722640 1 -- 172.21.15.16:0/1892143998 shutdown_connections 2023-05-22T15:28:27.696 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.687+0000 7fb374722640 1 -- 172.21.15.16:0/1892143998 wait complete. 2023-05-22T15:28:27.703 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:28:27.698+0000 7f057ebf7180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-05-22T15:28:27.713 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:28:27.714 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1) 2023-05-22T15:28:27.724 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:28:27.725 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/dir0) 2023-05-22T15:28:27.729 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:28:27.723+0000 7f89dc3cf180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-05-22T15:28:27.736 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add --fs cephfs /snap_test_dir1/dir0 1M 2023-05-22T15:28:27.903 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.895+0000 7f1ca1b80640 1 Processor -- start 2023-05-22T15:28:27.903 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.895+0000 7f1ca1b80640 1 -- start start 2023-05-22T15:28:27.904 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.899+0000 7f1ca1b80640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f1c9c158ad0 0x7f1c9c06b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:27.905 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.899+0000 7f1ca1b80640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1c9c0ccd20 0x7f1c9c0cd150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:27.905 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.899+0000 7f1ca1b80640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c9c158ea0 con 0x7f1c9c06a610 2023-05-22T15:28:27.905 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.899+0000 7f1ca1b80640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c9c0cd690 con 0x7f1c9c158ad0 2023-05-22T15:28:27.905 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.899+0000 7f1ca1b80640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c9c0c96e0 con 0x7f1c9c0ccd20 2023-05-22T15:28:27.905 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.899+0000 7f1c9b7fe640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f1c9c158ad0 0x7f1c9c06b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:27.905 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.899+0000 7f1c9b7fe640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f1c9c158ad0 0x7f1c9c06b9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52190/0 (socket says 172.21.15.16:52190) 2023-05-22T15:28:27.906 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.899+0000 7f1c9b7fe640 1 -- 172.21.15.16:0/3272504980 learned_addr learned my addr 172.21.15.16:0/3272504980 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:28:27.906 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.899+0000 7f1c93fff640 1 --2- 172.21.15.16:0/3272504980 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1c9c0ccd20 0x7f1c9c0cd150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:27.906 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.899+0000 7f1c9b7fe640 1 -- 172.21.15.16:0/3272504980 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1c9c0ccd20 msgr2=0x7f1c9c0cd150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:28:27.906 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.899+0000 7f1c9b7fe640 1 --2- 172.21.15.16:0/3272504980 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1c9c0ccd20 0x7f1c9c0cd150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:27.906 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.899+0000 7f1c9b7fe640 1 -- 172.21.15.16:0/3272504980 >> v1:172.21.15.62:6789/0 conn(0x7f1c9c06a610 legacy=0x7f1c9c0ca4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:27.906 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.899+0000 7f1c9b7fe640 1 -- 172.21.15.16:0/3272504980 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c9c069a70 con 0x7f1c9c158ad0 2023-05-22T15:28:27.907 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.899+0000 7f1c9b7fe640 1 --2- 172.21.15.16:0/3272504980 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c9c158ad0 0x7f1c9c06b9c0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f1c8c009870 tx=0x7f1c8c01c510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1124eee95aeedf3c server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:28:27.907 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.899+0000 7f1c9affd640 1 -- 172.21.15.16:0/3272504980 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c8c0aa670 con 0x7f1c9c158ad0 2023-05-22T15:28:27.907 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.899+0000 7f1c9affd640 1 -- 172.21.15.16:0/3272504980 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c8c0aa7d0 con 0x7f1c9c158ad0 2023-05-22T15:28:27.907 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.899+0000 7f1c9affd640 1 -- 172.21.15.16:0/3272504980 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c8c0aab60 con 0x7f1c9c158ad0 2023-05-22T15:28:27.908 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.899+0000 7f1ca1b80640 1 -- 172.21.15.16:0/3272504980 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c9c158ad0 msgr2=0x7f1c9c06b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:27.908 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.899+0000 7f1ca1b80640 1 --2- 172.21.15.16:0/3272504980 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c9c158ad0 0x7f1c9c06b9c0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f1c8c009870 tx=0x7f1c8c01c510 comp rx=0 tx=0).stop 2023-05-22T15:28:27.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.899+0000 7f1ca1b80640 1 -- 172.21.15.16:0/3272504980 shutdown_connections 2023-05-22T15:28:27.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.899+0000 7f1ca1b80640 1 --2- 172.21.15.16:0/3272504980 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1c9c0ccd20 0x7f1c9c0cd150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:27.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.899+0000 7f1ca1b80640 1 --2- 172.21.15.16:0/3272504980 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c9c158ad0 0x7f1c9c06b9c0 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:27.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.899+0000 7f1ca1b80640 1 -- 172.21.15.16:0/3272504980 >> 172.21.15.16:0/3272504980 conn(0x7f1c9c0ce8d0 msgr2=0x7f1c9c0cecd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:28:27.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.899+0000 7f1ca1b80640 1 -- 172.21.15.16:0/3272504980 shutdown_connections 2023-05-22T15:28:27.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.899+0000 7f1ca1b80640 1 -- 172.21.15.16:0/3272504980 wait complete. 2023-05-22T15:28:27.912 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.903+0000 7f1ca1b80640 1 Processor -- start 2023-05-22T15:28:27.912 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.903+0000 7f1ca1b80640 1 -- start start 2023-05-22T15:28:27.912 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.903+0000 7f1ca1b80640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1c9c06a610 0x7f1c9c13bc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:27.912 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.903+0000 7f1ca1b80640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1c9c0ccd20 0x7f1c9c0b3d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:27.912 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.903+0000 7f1ca1b80640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c9c158ad0 0x7f1c9c165a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:27.913 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.903+0000 7f1ca1b80640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c9c0b3930 con 0x7f1c9c158ad0 2023-05-22T15:28:27.913 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.903+0000 7f1ca1b80640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c9c13c170 con 0x7f1c9c06a610 2023-05-22T15:28:27.913 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.903+0000 7f1ca1b80640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c9c0b2cb0 con 0x7f1c9c0ccd20 2023-05-22T15:28:27.913 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.903+0000 7f1c9bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c9c158ad0 0x7f1c9c165a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:27.913 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.903+0000 7f1c9bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c9c158ad0 0x7f1c9c165a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52196/0 (socket says 172.21.15.16:52196) 2023-05-22T15:28:27.913 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.903+0000 7f1c9bfff640 1 -- 172.21.15.16:0/1043875102 learned_addr learned my addr 172.21.15.16:0/1043875102 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:28:27.913 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.903+0000 7f1c93fff640 1 --2- 172.21.15.16:0/1043875102 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1c9c0ccd20 0x7f1c9c0b3d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:27.914 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.903+0000 7f1c9b7fe640 1 --2- 172.21.15.16:0/1043875102 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1c9c06a610 0x7f1c9c13bc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:27.914 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.903+0000 7f1c9bfff640 1 -- 172.21.15.16:0/1043875102 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1c9c0ccd20 msgr2=0x7f1c9c0b3d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:27.914 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.903+0000 7f1c9bfff640 1 --2- 172.21.15.16:0/1043875102 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1c9c0ccd20 0x7f1c9c0b3d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:27.914 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.903+0000 7f1c9bfff640 1 -- 172.21.15.16:0/1043875102 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1c9c06a610 msgr2=0x7f1c9c13bc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:27.914 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.903+0000 7f1c9bfff640 1 --2- 172.21.15.16:0/1043875102 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1c9c06a610 0x7f1c9c13bc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:27.914 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.903+0000 7f1c9bfff640 1 -- 172.21.15.16:0/1043875102 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c8c0094d0 con 0x7f1c9c158ad0 2023-05-22T15:28:27.914 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.903+0000 7f1c9bfff640 1 --2- 172.21.15.16:0/1043875102 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c9c158ad0 0x7f1c9c165a40 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f1c9400c770 tx=0x7f1c9400cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:28:27.914 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.903+0000 7f1c98ff9640 1 -- 172.21.15.16:0/1043875102 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c94003b80 con 0x7f1c9c158ad0 2023-05-22T15:28:27.915 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.903+0000 7f1c98ff9640 1 -- 172.21.15.16:0/1043875102 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c94003ce0 con 0x7f1c9c158ad0 2023-05-22T15:28:27.915 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.903+0000 7f1ca1b80640 1 -- 172.21.15.16:0/1043875102 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c9c0c9300 con 0x7f1c9c158ad0 2023-05-22T15:28:27.915 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.903+0000 7f1c98ff9640 1 -- 172.21.15.16:0/1043875102 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c9400aa30 con 0x7f1c9c158ad0 2023-05-22T15:28:27.915 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.903+0000 7f1ca1b80640 1 -- 172.21.15.16:0/1043875102 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c9c0c9f80 con 0x7f1c9c158ad0 2023-05-22T15:28:27.916 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.903+0000 7f1ca1b80640 1 -- 172.21.15.16:0/1043875102 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c9c0c9490 con 0x7f1c9c158ad0 2023-05-22T15:28:27.916 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.907+0000 7f1c98ff9640 1 -- 172.21.15.16:0/1043875102 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 6) v1 ==== 84247+0+0 (secure 0 0 0) 0x7f1c9400ab90 con 0x7f1c9c158ad0 2023-05-22T15:28:27.916 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.907+0000 7f1c98ff9640 1 --2- 172.21.15.16:0/1043875102 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f1c7c0653d0 0x7f1c7c067880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:27.916 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.907+0000 7f1c9b7fe640 1 -- 172.21.15.16:0/1043875102 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f1c7c0653d0 msgr2=0x7f1c7c067880 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/36921 2023-05-22T15:28:27.917 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.907+0000 7f1c98ff9640 1 -- 172.21.15.16:0/1043875102 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f1c940831e0 con 0x7f1c9c158ad0 2023-05-22T15:28:27.917 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.907+0000 7f1c9b7fe640 1 --2- 172.21.15.16:0/1043875102 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f1c7c0653d0 0x7f1c7c067880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-05-22T15:28:27.920 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:27.915+0000 7f1c98ff9640 1 -- 172.21.15.16:0/1043875102 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f1c94050db0 con 0x7f1c9c158ad0 2023-05-22T15:28:28.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:28.087+0000 7f1ca1b80640 1 -- 172.21.15.16:0/1043875102 --> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/snap_test_dir1/dir0", "snap_schedule": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f1c9c0b3aa0 con 0x7f1c7c0653d0 2023-05-22T15:28:28.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:28.111+0000 7f1c9b7fe640 1 -- 172.21.15.16:0/1043875102 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f1c7c0653d0 msgr2=0x7f1c7c067880 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/36921 2023-05-22T15:28:28.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:28.111+0000 7f1c9b7fe640 1 --2- 172.21.15.16:0/1043875102 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f1c7c0653d0 0x7f1c7c067880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-05-22T15:28:28.144 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:28:28.138+0000 7f057ebf7180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-05-22T15:28:28.181 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:28:28.175+0000 7f89dc3cf180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-05-22T15:28:28.196 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:28:28.190+0000 7f057ebf7180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-05-22T15:28:28.234 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:28:28.227+0000 7f89dc3cf180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-05-22T15:28:28.246 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:28:28.238+0000 7f057ebf7180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-05-22T15:28:28.285 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:28:28.279+0000 7f89dc3cf180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-05-22T15:28:28.324 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:28:28.318+0000 7f057ebf7180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-05-22T15:28:28.362 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:28:28.355+0000 7f89dc3cf180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-05-22T15:28:28.375 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:28:28.366+0000 7f057ebf7180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-05-22T15:28:28.413 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:28:28.407+0000 7f89dc3cf180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-05-22T15:28:28.450 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:28:28.442+0000 7f057ebf7180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-05-22T15:28:28.490 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:28:28.483+0000 7f89dc3cf180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-05-22T15:28:28.503 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:28:28.494+0000 7f057ebf7180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-05-22T15:28:28.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:28.515+0000 7f1c9b7fe640 1 -- 172.21.15.16:0/1043875102 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f1c7c0653d0 msgr2=0x7f1c7c067880 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/36921 2023-05-22T15:28:28.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:28.515+0000 7f1c9b7fe640 1 --2- 172.21.15.16:0/1043875102 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f1c7c0653d0 0x7f1c7c067880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-05-22T15:28:28.541 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:28:28.535+0000 7f89dc3cf180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-05-22T15:28:28.634 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:28:28.626+0000 7f057ebf7180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-05-22T15:28:28.673 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:28:28.667+0000 7f89dc3cf180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-05-22T15:28:28.687 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:28:28.678+0000 7f057ebf7180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-05-22T15:28:28.726 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:28:28.719+0000 7f89dc3cf180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-05-22T15:28:28.910 INFO:tasks.ceph.mgr.x.smithi062.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2023-05-22T15:28:28.911 INFO:tasks.ceph.mgr.x.smithi062.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2023-05-22T15:28:28.911 INFO:tasks.ceph.mgr.x.smithi062.stderr: from numpy import show_config as show_numpy_config 2023-05-22T15:28:28.918 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:28:28.910+0000 7f057ebf7180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-05-22T15:28:28.950 INFO:tasks.ceph.mgr.y.smithi016.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2023-05-22T15:28:28.951 INFO:tasks.ceph.mgr.y.smithi016.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2023-05-22T15:28:28.951 INFO:tasks.ceph.mgr.y.smithi016.stderr: from numpy import show_config as show_numpy_config 2023-05-22T15:28:28.958 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:28:28.951+0000 7f89dc3cf180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-05-22T15:28:29.045 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:28:29.038+0000 7f057ebf7180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-05-22T15:28:29.086 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:28:29.079+0000 7f89dc3cf180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-05-22T15:28:29.157 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:28:29.150+0000 7f057ebf7180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-05-22T15:28:29.199 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:28:29.191+0000 7f89dc3cf180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-05-22T15:28:29.220 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:28:29.214+0000 7f057ebf7180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-05-22T15:28:29.263 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:28:29.255+0000 7f89dc3cf180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-05-22T15:28:29.324 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:29.319+0000 7f1c9b7fe640 1 -- 172.21.15.16:0/1043875102 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f1c7c0653d0 msgr2=0x7f1c7c067880 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/36921 2023-05-22T15:28:29.325 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:29.319+0000 7f1c9b7fe640 1 --2- 172.21.15.16:0/1043875102 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f1c7c0653d0 0x7f1c7c067880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-05-22T15:28:29.345 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:28:29.338+0000 7f057ebf7180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-05-22T15:28:29.393 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:28:29.387+0000 7f89dc3cf180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-05-22T15:28:29.839 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:28:29.830+0000 7f057ebf7180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-05-22T15:28:29.889 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:28:29.883+0000 7f89dc3cf180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-05-22T15:28:29.915 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:28:29.906+0000 7f057ebf7180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-05-22T15:28:29.966 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:28:29.959+0000 7f89dc3cf180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-05-22T15:28:30.051 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:28:30.042+0000 7f057ebf7180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-05-22T15:28:30.107 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:28:30.099+0000 7f89dc3cf180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-05-22T15:28:30.128 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:28:30.122+0000 7f057ebf7180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-05-22T15:28:30.183 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:28:30.175+0000 7f89dc3cf180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-05-22T15:28:30.373 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:28:30.366+0000 7f057ebf7180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-05-22T15:28:30.428 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:28:30.422+0000 7f057ebf7180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-05-22T15:28:30.431 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:28:30.423+0000 7f89dc3cf180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-05-22T15:28:30.486 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:28:30.479+0000 7f89dc3cf180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-05-22T15:28:30.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:30.511+0000 7f1c98ff9640 1 -- 172.21.15.16:0/1043875102 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mgrmap(e 7) v1 ==== 83958+0+0 (secure 0 0 0) 0x7f1c94051180 con 0x7f1c9c158ad0 2023-05-22T15:28:30.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:30.515+0000 7f1c98ff9640 1 -- 172.21.15.16:0/1043875102 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f1c7c0653d0 msgr2=0x7f1c7c067880 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-05-22T15:28:30.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:30.515+0000 7f1c98ff9640 1 --2- 172.21.15.16:0/1043875102 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f1c7c0653d0 0x7f1c7c067880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:30.674 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:28:30.667+0000 7f895e549640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T15:28:30.674 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:28:30.667+0000 7f895e549640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T15:28:30.674 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:28:30.667+0000 7f895e549640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T15:28:30.674 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:28:30.667+0000 7f895e549640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T15:28:30.674 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:28:30.667+0000 7f895e549640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T15:28:30.675 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:28:30.667+0000 7f895a541640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T15:28:30.675 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:28:30.667+0000 7f895a541640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T15:28:30.675 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:28:30.667+0000 7f895a541640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T15:28:30.675 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:28:30.667+0000 7f895a541640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T15:28:30.675 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:28:30.667+0000 7f895a541640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T15:28:31.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:31.523+0000 7f1c98ff9640 1 -- 172.21.15.16:0/1043875102 <== mon.0 v2:172.21.15.16:3300/0 8 ==== mgrmap(e 8) v1 ==== 84085+0+0 (secure 0 0 0) 0x7f1c9406df20 con 0x7f1c9c158ad0 2023-05-22T15:28:31.530 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:31.523+0000 7f1c98ff9640 1 --2- 172.21.15.16:0/1043875102 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f1c7c0653d0 0x7f1c7c067800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:31.530 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:31.523+0000 7f1c98ff9640 1 -- 172.21.15.16:0/1043875102 --> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/snap_test_dir1/dir0", "snap_schedule": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f1c9c0b3aa0 con 0x7f1c7c0653d0 2023-05-22T15:28:31.536 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:31.527+0000 7f1c9b7fe640 1 --2- 172.21.15.16:0/1043875102 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f1c7c0653d0 0x7f1c7c067800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:31.539 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:31.531+0000 7f1c9b7fe640 1 --2- 172.21.15.16:0/1043875102 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f1c7c0653d0 0x7f1c7c067800 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f1c8c0ae000 tx=0x7f1c8c0a7000 comp rx=0 tx=0).ready entity=mgr.5823 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:28:31.685 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:31.679+0000 7f1c98ff9640 1 -- 172.21.15.16:0/1043875102 <== mgr.5823 v2:172.21.15.16:6832/36921 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7f1c9c0b3aa0 con 0x7f1c7c0653d0 2023-05-22T15:28:31.686 INFO:teuthology.orchestra.run.smithi016.stdout:Schedule set for path /snap_test_dir1/dir0 2023-05-22T15:28:31.688 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:31.679+0000 7f1ca1b80640 1 -- 172.21.15.16:0/1043875102 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f1c7c0653d0 msgr2=0x7f1c7c067800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:31.688 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:31.679+0000 7f1ca1b80640 1 --2- 172.21.15.16:0/1043875102 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f1c7c0653d0 0x7f1c7c067800 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f1c8c0ae000 tx=0x7f1c8c0a7000 comp rx=0 tx=0).stop 2023-05-22T15:28:31.688 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:31.679+0000 7f1ca1b80640 1 -- 172.21.15.16:0/1043875102 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c9c158ad0 msgr2=0x7f1c9c165a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:31.688 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:31.679+0000 7f1ca1b80640 1 --2- 172.21.15.16:0/1043875102 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c9c158ad0 0x7f1c9c165a40 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f1c9400c770 tx=0x7f1c9400cc30 comp rx=0 tx=0).stop 2023-05-22T15:28:31.688 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:31.679+0000 7f1ca1b80640 1 -- 172.21.15.16:0/1043875102 shutdown_connections 2023-05-22T15:28:31.688 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:31.679+0000 7f1ca1b80640 1 --2- 172.21.15.16:0/1043875102 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c9c158ad0 0x7f1c9c165a40 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:31.689 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:31.679+0000 7f1ca1b80640 1 --2- 172.21.15.16:0/1043875102 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f1c7c0653d0 0x7f1c7c067800 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:31.689 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:31.679+0000 7f1ca1b80640 1 --2- 172.21.15.16:0/1043875102 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1c9c0ccd20 0x7f1c9c0b3d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:31.689 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:31.679+0000 7f1ca1b80640 1 --2- 172.21.15.16:0/1043875102 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1c9c06a610 0x7f1c9c13bc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:31.689 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:31.679+0000 7f1ca1b80640 1 -- 172.21.15.16:0/1043875102 >> 172.21.15.16:0/1043875102 conn(0x7f1c9c0ce8d0 msgr2=0x7f1c9c152e40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:28:31.689 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:31.683+0000 7f1ca1b80640 1 -- 172.21.15.16:0/1043875102 shutdown_connections 2023-05-22T15:28:31.689 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:31.683+0000 7f1ca1b80640 1 -- 172.21.15.16:0/1043875102 wait complete. 2023-05-22T15:28:31.709 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/dir0 2023-05-22T15:28:31.710 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:28:31.710 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/dir1) 2023-05-22T15:28:31.720 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add --fs cephfs /snap_test_dir1/dir1 1M 2023-05-22T15:28:31.902 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:31.895+0000 7fd1a9598640 1 Processor -- start 2023-05-22T15:28:31.903 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:31.895+0000 7fd1a9598640 1 -- start start 2023-05-22T15:28:31.903 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:31.895+0000 7fd1a9598640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd1a415bd90 0x7fd1a40ce9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:31.903 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:31.895+0000 7fd1a9598640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fd1a40cb1c0 0x7fd1a415f8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:31.904 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:31.895+0000 7fd1a9598640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1a40c4f90 con 0x7fd1a40c4a40 2023-05-22T15:28:31.904 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:31.895+0000 7fd1a9598640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1a406c700 con 0x7fd1a40cb1c0 2023-05-22T15:28:31.904 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:31.895+0000 7fd1a9598640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1a40ad450 con 0x7fd1a415bd90 2023-05-22T15:28:31.904 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:31.895+0000 7fd1a37fe640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fd1a40cb1c0 0x7fd1a415f8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:31.905 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:31.895+0000 7fd1a2ffd640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd1a415bd90 0x7fd1a40ce9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:31.905 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:31.899+0000 7fd1a37fe640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fd1a40cb1c0 0x7fd1a415f8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.16:38668/0 (socket says 172.21.15.16:38668) 2023-05-22T15:28:31.905 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:31.899+0000 7fd1a37fe640 1 -- 172.21.15.16:0/2862304185 learned_addr learned my addr 172.21.15.16:0/2862304185 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:28:31.905 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:31.899+0000 7fd1a2ffd640 1 -- 172.21.15.16:0/2862304185 >> v2:172.21.15.62:3300/0 conn(0x7fd1a40cb1c0 msgr2=0x7fd1a415f8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:28:31.905 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:31.899+0000 7fd1a2ffd640 1 --2- 172.21.15.16:0/2862304185 >> v2:172.21.15.62:3300/0 conn(0x7fd1a40cb1c0 0x7fd1a415f8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:31.905 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:31.899+0000 7fd1a2ffd640 1 -- 172.21.15.16:0/2862304185 >> v1:172.21.15.62:6789/0 conn(0x7fd1a40c4a40 legacy=0x7fd1a40caab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:31.906 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:31.899+0000 7fd1a2ffd640 1 -- 172.21.15.16:0/2862304185 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1a40abb10 con 0x7fd1a415bd90 2023-05-22T15:28:31.906 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:31.899+0000 7fd1a2ffd640 1 --2- 172.21.15.16:0/2862304185 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd1a415bd90 0x7fd1a40ce9a0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fd194009d90 tx=0x7fd19401c950 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fa233fc4d7d610a9 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:28:31.907 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:31.899+0000 7fd1a1ffb640 1 -- 172.21.15.16:0/2862304185 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1940a5810 con 0x7fd1a415bd90 2023-05-22T15:28:31.907 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:31.899+0000 7fd1a1ffb640 1 -- 172.21.15.16:0/2862304185 <== mon.2 v2:172.21.15.62:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1940a5970 con 0x7fd1a415bd90 2023-05-22T15:28:31.907 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:31.899+0000 7fd1a1ffb640 1 -- 172.21.15.16:0/2862304185 <== mon.2 v2:172.21.15.62:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1940a5d00 con 0x7fd1a415bd90 2023-05-22T15:28:31.908 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:31.899+0000 7fd1a9598640 1 -- 172.21.15.16:0/2862304185 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd1a415bd90 msgr2=0x7fd1a40ce9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:31.908 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:31.899+0000 7fd1a9598640 1 --2- 172.21.15.16:0/2862304185 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd1a415bd90 0x7fd1a40ce9a0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fd194009d90 tx=0x7fd19401c950 comp rx=0 tx=0).stop 2023-05-22T15:28:31.908 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:31.899+0000 7fd1a9598640 1 -- 172.21.15.16:0/2862304185 shutdown_connections 2023-05-22T15:28:31.908 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:31.899+0000 7fd1a9598640 1 --2- 172.21.15.16:0/2862304185 >> v2:172.21.15.62:3300/0 conn(0x7fd1a40cb1c0 0x7fd1a415f8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:31.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:31.899+0000 7fd1a9598640 1 --2- 172.21.15.16:0/2862304185 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd1a415bd90 0x7fd1a40ce9a0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:31.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:31.899+0000 7fd1a9598640 1 -- 172.21.15.16:0/2862304185 >> 172.21.15.16:0/2862304185 conn(0x7fd1a4069720 msgr2=0x7fd1a4069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:28:31.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:31.899+0000 7fd1a9598640 1 -- 172.21.15.16:0/2862304185 shutdown_connections 2023-05-22T15:28:31.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:31.899+0000 7fd1a9598640 1 -- 172.21.15.16:0/2862304185 wait complete. 2023-05-22T15:28:31.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:31.903+0000 7fd1a9598640 1 Processor -- start 2023-05-22T15:28:31.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:31.903+0000 7fd1a9598640 1 -- start start 2023-05-22T15:28:31.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:31.903+0000 7fd1a9598640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd1a40c4a40 0x7fd1a4150a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:31.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:31.903+0000 7fd1a9598640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd1a40cb1c0 0x7fd1a415a490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:31.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:31.903+0000 7fd1a9598640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd1a4151330 0x7fd1a415a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:31.911 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:31.903+0000 7fd1a9598640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1a40ada70 con 0x7fd1a4151330 2023-05-22T15:28:31.911 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:31.903+0000 7fd1a9598640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1a40adcc0 con 0x7fd1a40cb1c0 2023-05-22T15:28:31.911 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:31.903+0000 7fd1a9598640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1a40adfa0 con 0x7fd1a40c4a40 2023-05-22T15:28:31.911 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:31.903+0000 7fd1a37fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd1a4151330 0x7fd1a415a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:31.911 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:31.903+0000 7fd1a37fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd1a4151330 0x7fd1a415a9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42360/0 (socket says 172.21.15.16:42360) 2023-05-22T15:28:31.911 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:31.903+0000 7fd1a37fe640 1 -- 172.21.15.16:0/29211433 learned_addr learned my addr 172.21.15.16:0/29211433 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:28:31.911 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:31.903+0000 7fd1a2ffd640 1 --2- 172.21.15.16:0/29211433 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd1a40c4a40 0x7fd1a4150a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:31.911 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:31.903+0000 7fd1a27fc640 1 --2- 172.21.15.16:0/29211433 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd1a40cb1c0 0x7fd1a415a490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:31.913 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:31.903+0000 7fd1a37fe640 1 -- 172.21.15.16:0/29211433 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd1a40c4a40 msgr2=0x7fd1a4150a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:31.913 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:31.903+0000 7fd1a37fe640 1 --2- 172.21.15.16:0/29211433 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd1a40c4a40 0x7fd1a4150a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:31.913 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:31.903+0000 7fd1a37fe640 1 -- 172.21.15.16:0/29211433 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd1a40cb1c0 msgr2=0x7fd1a415a490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:31.913 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:31.903+0000 7fd1a37fe640 1 --2- 172.21.15.16:0/29211433 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd1a40cb1c0 0x7fd1a415a490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:31.913 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:31.903+0000 7fd1a37fe640 1 -- 172.21.15.16:0/29211433 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd194009a70 con 0x7fd1a4151330 2023-05-22T15:28:31.914 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:31.903+0000 7fd1a37fe640 1 --2- 172.21.15.16:0/29211433 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd1a4151330 0x7fd1a415a9d0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7fd19c00e7c0 tx=0x7fd19c00ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:28:31.914 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:31.903+0000 7fd19b7fe640 1 -- 172.21.15.16:0/29211433 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd19c00ee70 con 0x7fd1a4151330 2023-05-22T15:28:31.914 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:31.903+0000 7fd19b7fe640 1 -- 172.21.15.16:0/29211433 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd19c00c940 con 0x7fd1a4151330 2023-05-22T15:28:31.914 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:31.903+0000 7fd1a9598640 1 -- 172.21.15.16:0/29211433 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1a40ad8e0 con 0x7fd1a4151330 2023-05-22T15:28:31.914 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:31.903+0000 7fd19b7fe640 1 -- 172.21.15.16:0/29211433 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd19c00ccf0 con 0x7fd1a4151330 2023-05-22T15:28:31.914 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:31.903+0000 7fd1a9598640 1 -- 172.21.15.16:0/29211433 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1a40ad120 con 0x7fd1a4151330 2023-05-22T15:28:31.914 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:31.907+0000 7fd1a9598640 1 -- 172.21.15.16:0/29211433 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1a4168c70 con 0x7fd1a4151330 2023-05-22T15:28:31.914 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:31.907+0000 7fd19b7fe640 1 -- 172.21.15.16:0/29211433 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 8) v1 ==== 84085+0+0 (secure 0 0 0) 0x7fd19c013470 con 0x7fd1a4151330 2023-05-22T15:28:31.915 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:31.907+0000 7fd19b7fe640 1 --2- 172.21.15.16:0/29211433 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd184065250 0x7fd184067700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:31.915 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:31.907+0000 7fd19b7fe640 1 -- 172.21.15.16:0/29211433 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 6640+0+0 (secure 0 0 0) 0x7fd19c082720 con 0x7fd1a4151330 2023-05-22T15:28:31.915 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:31.907+0000 7fd1a2ffd640 1 --2- 172.21.15.16:0/29211433 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd184065250 0x7fd184067700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:31.915 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:31.907+0000 7fd1a2ffd640 1 --2- 172.21.15.16:0/29211433 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd184065250 0x7fd184067700 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fd1940af000 tx=0x7fd1940a7070 comp rx=0 tx=0).ready entity=mgr.5823 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:28:31.917 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:31.911+0000 7fd19b7fe640 1 -- 172.21.15.16:0/29211433 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fd19c050440 con 0x7fd1a4151330 2023-05-22T15:28:32.081 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:32.075+0000 7fd1a9598640 1 -- 172.21.15.16:0/29211433 --> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/snap_test_dir1/dir1", "snap_schedule": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7fd1a404dbe0 con 0x7fd184065250 2023-05-22T15:28:32.199 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:32.191+0000 7fd19b7fe640 1 -- 172.21.15.16:0/29211433 <== mgr.5823 v2:172.21.15.16:6832/36921 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7fd1a404dbe0 con 0x7fd184065250 2023-05-22T15:28:32.199 INFO:teuthology.orchestra.run.smithi016.stdout:Schedule set for path /snap_test_dir1/dir1 2023-05-22T15:28:32.202 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:32.195+0000 7fd1a9598640 1 -- 172.21.15.16:0/29211433 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd184065250 msgr2=0x7fd184067700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:32.203 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:32.195+0000 7fd1a9598640 1 --2- 172.21.15.16:0/29211433 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd184065250 0x7fd184067700 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fd1940af000 tx=0x7fd1940a7070 comp rx=0 tx=0).stop 2023-05-22T15:28:32.203 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:32.195+0000 7fd1a9598640 1 -- 172.21.15.16:0/29211433 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd1a4151330 msgr2=0x7fd1a415a9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:32.203 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:32.195+0000 7fd1a9598640 1 --2- 172.21.15.16:0/29211433 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd1a4151330 0x7fd1a415a9d0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7fd19c00e7c0 tx=0x7fd19c00ec80 comp rx=0 tx=0).stop 2023-05-22T15:28:32.203 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:32.195+0000 7fd1a9598640 1 -- 172.21.15.16:0/29211433 shutdown_connections 2023-05-22T15:28:32.203 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:32.195+0000 7fd1a9598640 1 --2- 172.21.15.16:0/29211433 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd1a4151330 0x7fd1a415a9d0 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:32.203 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:32.195+0000 7fd1a9598640 1 --2- 172.21.15.16:0/29211433 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd1a40cb1c0 0x7fd1a415a490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:32.203 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:32.195+0000 7fd1a9598640 1 --2- 172.21.15.16:0/29211433 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd184065250 0x7fd184067700 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:32.204 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:32.195+0000 7fd1a9598640 1 --2- 172.21.15.16:0/29211433 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd1a40c4a40 0x7fd1a4150a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:32.204 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:32.195+0000 7fd1a9598640 1 -- 172.21.15.16:0/29211433 >> 172.21.15.16:0/29211433 conn(0x7fd1a4069720 msgr2=0x7fd1a40c4070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:28:32.204 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:32.195+0000 7fd1a9598640 1 -- 172.21.15.16:0/29211433 shutdown_connections 2023-05-22T15:28:32.204 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:32.195+0000 7fd1a9598640 1 -- 172.21.15.16:0/29211433 wait complete. 2023-05-22T15:28:32.219 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/dir1 2023-05-22T15:28:32.220 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:28:32.220 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/dir2) 2023-05-22T15:28:32.228 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add --fs cephfs /snap_test_dir1/dir2 1M 2023-05-22T15:28:32.405 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:32.399+0000 7f8284772640 1 Processor -- start 2023-05-22T15:28:32.405 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:32.399+0000 7f8284772640 1 -- start start 2023-05-22T15:28:32.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:32.399+0000 7f8284772640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f827c0ccc50 0x7f827c15ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:32.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:32.399+0000 7f8284772640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f827c0cef10 0x7f827c0cf370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:32.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:32.399+0000 7f8284772640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f827c0d2a20 con 0x7f827c15c8e0 2023-05-22T15:28:32.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:32.399+0000 7f8284772640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f827c0cd230 con 0x7f827c0cef10 2023-05-22T15:28:32.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:32.399+0000 7f8284772640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f827c0d23d0 con 0x7f827c0ccc50 2023-05-22T15:28:32.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:32.399+0000 7f8282d0f640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f827c0cef10 0x7f827c0cf370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:32.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:32.399+0000 7f828250e640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f827c15c8e0 0x7f827c0d2eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:47938/0 (socket says 172.21.15.16:47938) 2023-05-22T15:28:32.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:32.399+0000 7f828250e640 1 -- 172.21.15.16:0/2550187168 learned_addr learned my addr 172.21.15.16:0/2550187168 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:28:32.408 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:32.399+0000 7f8281d0d640 1 --2- 172.21.15.16:0/2550187168 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f827c0ccc50 0x7f827c15ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:32.409 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:32.399+0000 7f828150c640 1 -- 172.21.15.16:0/2550187168 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1904691420 0 0) 0x7f827c0d2a20 con 0x7f827c15c8e0 2023-05-22T15:28:32.409 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:32.399+0000 7f828150c640 1 -- 172.21.15.16:0/2550187168 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8268003530 con 0x7f827c15c8e0 2023-05-22T15:28:32.409 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:32.399+0000 7f8282d0f640 1 -- 172.21.15.16:0/2550187168 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f827c0ccc50 msgr2=0x7f827c15ccb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:28:32.409 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:32.399+0000 7f8282d0f640 1 --2- 172.21.15.16:0/2550187168 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f827c0ccc50 0x7f827c15ccb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:32.409 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:32.399+0000 7f8282d0f640 1 -- 172.21.15.16:0/2550187168 >> v1:172.21.15.16:6789/0 conn(0x7f827c15c8e0 legacy=0x7f827c0d2eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:32.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:32.399+0000 7f8282d0f640 1 -- 172.21.15.16:0/2550187168 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f827c0d26b0 con 0x7f827c0cef10 2023-05-22T15:28:32.412 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:32.403+0000 7f8282d0f640 1 --2- 172.21.15.16:0/2550187168 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f827c0cef10 0x7f827c0cf370 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f8274009d90 tx=0x7f827401c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1db89bfd0f302feb server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:28:32.412 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:32.403+0000 7f828150c640 1 -- 172.21.15.16:0/2550187168 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82740a5810 con 0x7f827c0cef10 2023-05-22T15:28:32.413 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:32.403+0000 7f828150c640 1 -- 172.21.15.16:0/2550187168 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82740a5970 con 0x7f827c0cef10 2023-05-22T15:28:32.413 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:32.403+0000 7f828150c640 1 -- 172.21.15.16:0/2550187168 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82740a5d00 con 0x7f827c0cef10 2023-05-22T15:28:32.413 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:32.403+0000 7f8284772640 1 -- 172.21.15.16:0/2550187168 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f827c0cef10 msgr2=0x7f827c0cf370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:32.413 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:32.403+0000 7f8284772640 1 --2- 172.21.15.16:0/2550187168 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f827c0cef10 0x7f827c0cf370 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f8274009d90 tx=0x7f827401c8c0 comp rx=0 tx=0).stop 2023-05-22T15:28:32.413 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:32.403+0000 7f8284772640 1 -- 172.21.15.16:0/2550187168 shutdown_connections 2023-05-22T15:28:32.414 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:32.403+0000 7f8284772640 1 --2- 172.21.15.16:0/2550187168 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f827c0cef10 0x7f827c0cf370 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:32.414 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:32.403+0000 7f8284772640 1 --2- 172.21.15.16:0/2550187168 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f827c0ccc50 0x7f827c15ccb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:32.414 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:32.403+0000 7f8284772640 1 -- 172.21.15.16:0/2550187168 >> 172.21.15.16:0/2550187168 conn(0x7f827c070e50 msgr2=0x7f827c071250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:28:32.414 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:32.403+0000 7f8284772640 1 -- 172.21.15.16:0/2550187168 shutdown_connections 2023-05-22T15:28:32.414 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:32.403+0000 7f8284772640 1 -- 172.21.15.16:0/2550187168 wait complete. 2023-05-22T15:28:32.414 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:32.403+0000 7f8284772640 1 Processor -- start 2023-05-22T15:28:32.414 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:32.403+0000 7f8284772640 1 -- start start 2023-05-22T15:28:32.415 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:32.403+0000 7f8284772640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f827c0ccc50 0x7f827c0b5430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:32.415 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:32.403+0000 7f8284772640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f827c0cef10 0x7f827c15fa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:32.415 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:32.403+0000 7f8284772640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f827c15c8e0 0x7f827c15ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:32.415 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:32.403+0000 7f8284772640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f827c0c6d70 con 0x7f827c0ccc50 2023-05-22T15:28:32.415 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:32.403+0000 7f8284772640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f827c0b5b10 con 0x7f827c0cef10 2023-05-22T15:28:32.415 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:32.403+0000 7f8284772640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f827c0c6440 con 0x7f827c15c8e0 2023-05-22T15:28:32.415 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:32.403+0000 7f828250e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f827c0ccc50 0x7f827c0b5430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:32.415 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:32.403+0000 7f828250e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f827c0ccc50 0x7f827c0b5430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42368/0 (socket says 172.21.15.16:42368) 2023-05-22T15:28:32.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:32.403+0000 7f828250e640 1 -- 172.21.15.16:0/3103204229 learned_addr learned my addr 172.21.15.16:0/3103204229 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:28:32.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:32.403+0000 7f8281d0d640 1 --2- 172.21.15.16:0/3103204229 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f827c0cef10 0x7f827c15fa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:32.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:32.403+0000 7f8282d0f640 1 --2- 172.21.15.16:0/3103204229 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f827c15c8e0 0x7f827c15ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:32.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:32.403+0000 7f828250e640 1 -- 172.21.15.16:0/3103204229 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f827c15c8e0 msgr2=0x7f827c15ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:32.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:32.407+0000 7f828250e640 1 --2- 172.21.15.16:0/3103204229 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f827c15c8e0 0x7f827c15ffa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:32.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:32.407+0000 7f828250e640 1 -- 172.21.15.16:0/3103204229 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f827c0cef10 msgr2=0x7f827c15fa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:32.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:32.407+0000 7f828250e640 1 --2- 172.21.15.16:0/3103204229 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f827c0cef10 0x7f827c15fa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:32.417 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:32.407+0000 7f828250e640 1 -- 172.21.15.16:0/3103204229 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8274009a70 con 0x7f827c0ccc50 2023-05-22T15:28:32.417 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:32.407+0000 7f8281d0d640 1 --2- 172.21.15.16:0/3103204229 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f827c0cef10 0x7f827c15fa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:28:32.417 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:32.407+0000 7f8282d0f640 1 --2- 172.21.15.16:0/3103204229 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f827c15c8e0 0x7f827c15ffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:28:32.417 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:32.407+0000 7f828250e640 1 --2- 172.21.15.16:0/3103204229 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f827c0ccc50 0x7f827c0b5430 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f827800c420 tx=0x7f827800c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:28:32.417 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:32.407+0000 7f8272ffd640 1 -- 172.21.15.16:0/3103204229 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8278005cf0 con 0x7f827c0ccc50 2023-05-22T15:28:32.417 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:32.407+0000 7f8272ffd640 1 -- 172.21.15.16:0/3103204229 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8278005e50 con 0x7f827c0ccc50 2023-05-22T15:28:32.417 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:32.407+0000 7f8284772640 1 -- 172.21.15.16:0/3103204229 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f827c0d0bf0 con 0x7f827c0ccc50 2023-05-22T15:28:32.418 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:32.407+0000 7f8272ffd640 1 -- 172.21.15.16:0/3103204229 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82780072d0 con 0x7f827c0ccc50 2023-05-22T15:28:32.418 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:32.407+0000 7f8284772640 1 -- 172.21.15.16:0/3103204229 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f827c0cf8b0 con 0x7f827c0ccc50 2023-05-22T15:28:32.419 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:32.407+0000 7f8284772640 1 -- 172.21.15.16:0/3103204229 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f827c0d0d80 con 0x7f827c0ccc50 2023-05-22T15:28:32.419 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:32.407+0000 7f8272ffd640 1 -- 172.21.15.16:0/3103204229 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 8) v1 ==== 84085+0+0 (secure 0 0 0) 0x7f8278007430 con 0x7f827c0ccc50 2023-05-22T15:28:32.419 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:32.407+0000 7f8272ffd640 1 --2- 172.21.15.16:0/3103204229 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f8260065220 0x7f82600676d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:32.420 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:32.407+0000 7f8272ffd640 1 -- 172.21.15.16:0/3103204229 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 6640+0+0 (secure 0 0 0) 0x7f8278081c70 con 0x7f827c0ccc50 2023-05-22T15:28:32.420 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:32.407+0000 7f8281d0d640 1 --2- 172.21.15.16:0/3103204229 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f8260065220 0x7f82600676d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:32.420 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:32.407+0000 7f8281d0d640 1 --2- 172.21.15.16:0/3103204229 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f8260065220 0x7f82600676d0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f827c0cbfd0 tx=0x7f826c008040 comp rx=0 tx=0).ready entity=mgr.5823 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:28:32.420 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:32.411+0000 7f8272ffd640 1 -- 172.21.15.16:0/3103204229 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f827804a620 con 0x7f827c0ccc50 2023-05-22T15:28:32.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:32.591+0000 7f8284772640 1 -- 172.21.15.16:0/3103204229 --> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/snap_test_dir1/dir2", "snap_schedule": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f827c0c6eb0 con 0x7f8260065220 2023-05-22T15:28:32.719 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:32.711+0000 7f8272ffd640 1 -- 172.21.15.16:0/3103204229 <== mgr.5823 v2:172.21.15.16:6832/36921 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7f826c009df0 con 0x7f8260065220 2023-05-22T15:28:32.719 INFO:teuthology.orchestra.run.smithi016.stdout:Schedule set for path /snap_test_dir1/dir2 2023-05-22T15:28:32.722 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:32.715+0000 7f8284772640 1 -- 172.21.15.16:0/3103204229 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f8260065220 msgr2=0x7f82600676d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:32.723 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:32.715+0000 7f8284772640 1 --2- 172.21.15.16:0/3103204229 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f8260065220 0x7f82600676d0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f827c0cbfd0 tx=0x7f826c008040 comp rx=0 tx=0).stop 2023-05-22T15:28:32.723 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:32.715+0000 7f8284772640 1 -- 172.21.15.16:0/3103204229 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f827c0ccc50 msgr2=0x7f827c0b5430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:32.723 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:32.715+0000 7f8284772640 1 --2- 172.21.15.16:0/3103204229 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f827c0ccc50 0x7f827c0b5430 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f827800c420 tx=0x7f827800c8e0 comp rx=0 tx=0).stop 2023-05-22T15:28:32.723 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:32.715+0000 7f8284772640 1 -- 172.21.15.16:0/3103204229 shutdown_connections 2023-05-22T15:28:32.724 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:32.715+0000 7f8284772640 1 --2- 172.21.15.16:0/3103204229 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f8260065220 0x7f82600676d0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:32.724 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:32.715+0000 7f8284772640 1 --2- 172.21.15.16:0/3103204229 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f827c15c8e0 0x7f827c15ffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:32.724 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:32.715+0000 7f8284772640 1 --2- 172.21.15.16:0/3103204229 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f827c0cef10 0x7f827c15fa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:32.724 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:32.715+0000 7f8284772640 1 --2- 172.21.15.16:0/3103204229 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f827c0ccc50 0x7f827c0b5430 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:32.725 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:32.715+0000 7f8284772640 1 -- 172.21.15.16:0/3103204229 >> 172.21.15.16:0/3103204229 conn(0x7f827c070e50 msgr2=0x7f827c0d19d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:28:32.725 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:32.715+0000 7f8284772640 1 -- 172.21.15.16:0/3103204229 shutdown_connections 2023-05-22T15:28:32.725 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:32.715+0000 7f8284772640 1 -- 172.21.15.16:0/3103204229 wait complete. 2023-05-22T15:28:32.743 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/dir2 2023-05-22T15:28:32.744 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:28:32.744 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/dir3) 2023-05-22T15:28:32.753 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add --fs cephfs /snap_test_dir1/dir3 1M 2023-05-22T15:28:32.937 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:32.931+0000 7f6988ab3640 1 Processor -- start 2023-05-22T15:28:32.937 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:32.931+0000 7f6988ab3640 1 -- start start 2023-05-22T15:28:32.939 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:32.931+0000 7f6988ab3640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6984070c10 0x7f69840cc4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:32.939 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:32.931+0000 7f6988ab3640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69840ccbb0 con 0x7f698415d650 2023-05-22T15:28:32.940 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:32.931+0000 7f6988ab3640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69840cd7c0 con 0x7f69840c9e40 2023-05-22T15:28:32.940 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:32.931+0000 7f6988ab3640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69840cdaa0 con 0x7f6984070c10 2023-05-22T15:28:32.940 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:32.931+0000 7f698259c640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6984070c10 0x7f69840cc4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:32.941 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:32.931+0000 7f698259c640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6984070c10 0x7f69840cc4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42380/0 (socket says 172.21.15.16:42380) 2023-05-22T15:28:32.941 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:32.931+0000 7f698259c640 1 -- 172.21.15.16:0/3424306243 learned_addr learned my addr 172.21.15.16:0/3424306243 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:28:32.941 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:32.931+0000 7f698259c640 1 -- 172.21.15.16:0/3424306243 >> v1:172.21.15.62:6789/0 conn(0x7f69840c9e40 legacy=0x7f698406bfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:28:32.941 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:32.931+0000 7f698259c640 1 -- 172.21.15.16:0/3424306243 >> v1:172.21.15.16:6789/0 conn(0x7f698415d650 legacy=0x7f698415da70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:32.942 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:32.931+0000 7f698259c640 1 -- 172.21.15.16:0/3424306243 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69840cdd80 con 0x7f6984070c10 2023-05-22T15:28:32.943 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:32.935+0000 7f698259c640 1 --2- 172.21.15.16:0/3424306243 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6984070c10 0x7f69840cc4e0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f696c009870 tx=0x7f696c01c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75affdba13b746ba server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:28:32.943 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:32.935+0000 7f698159a640 1 -- 172.21.15.16:0/3424306243 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f696c0aa680 con 0x7f6984070c10 2023-05-22T15:28:32.943 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:32.935+0000 7f698159a640 1 -- 172.21.15.16:0/3424306243 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f696c0aa7e0 con 0x7f6984070c10 2023-05-22T15:28:32.944 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:32.935+0000 7f698159a640 1 -- 172.21.15.16:0/3424306243 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f696c0aab70 con 0x7f6984070c10 2023-05-22T15:28:32.944 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:32.935+0000 7f6988ab3640 1 -- 172.21.15.16:0/3424306243 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6984070c10 msgr2=0x7f69840cc4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:32.944 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:32.935+0000 7f6988ab3640 1 --2- 172.21.15.16:0/3424306243 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6984070c10 0x7f69840cc4e0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f696c009870 tx=0x7f696c01c500 comp rx=0 tx=0).stop 2023-05-22T15:28:32.945 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:32.935+0000 7f6988ab3640 1 -- 172.21.15.16:0/3424306243 shutdown_connections 2023-05-22T15:28:32.945 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:32.935+0000 7f6988ab3640 1 --2- 172.21.15.16:0/3424306243 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6984070c10 0x7f69840cc4e0 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:32.945 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:32.935+0000 7f6988ab3640 1 -- 172.21.15.16:0/3424306243 >> 172.21.15.16:0/3424306243 conn(0x7f6984069720 msgr2=0x7f6984069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:28:32.945 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:32.935+0000 7f6988ab3640 1 -- 172.21.15.16:0/3424306243 shutdown_connections 2023-05-22T15:28:32.945 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:32.935+0000 7f6988ab3640 1 -- 172.21.15.16:0/3424306243 wait complete. 2023-05-22T15:28:32.945 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:32.935+0000 7f6988ab3640 1 Processor -- start 2023-05-22T15:28:32.946 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:32.935+0000 7f6988ab3640 1 -- start start 2023-05-22T15:28:32.948 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:32.939+0000 7f6988ab3640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6984070c10 0x7f69840b1e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:32.948 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:32.939+0000 7f6988ab3640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f69840c9e40 0x7f698414d630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:32.949 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:32.939+0000 7f6988ab3640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f698415d650 0x7f698414db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:32.949 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:32.939+0000 7f6988ab3640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69840b2340 con 0x7f698415d650 2023-05-22T15:28:32.949 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:32.939+0000 7f6988ab3640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69840b1340 con 0x7f69840c9e40 2023-05-22T15:28:32.949 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:32.939+0000 7f6988ab3640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69840b3ce0 con 0x7f6984070c10 2023-05-22T15:28:32.949 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:32.939+0000 7f6982d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f698415d650 0x7f698414db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:32.949 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:32.939+0000 7f6982d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f698415d650 0x7f698414db70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42384/0 (socket says 172.21.15.16:42384) 2023-05-22T15:28:32.949 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:32.939+0000 7f6982d9d640 1 -- 172.21.15.16:0/11350047 learned_addr learned my addr 172.21.15.16:0/11350047 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:28:32.950 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:32.939+0000 7f6981d9b640 1 --2- 172.21.15.16:0/11350047 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f69840c9e40 0x7f698414d630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:32.950 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:32.939+0000 7f698259c640 1 --2- 172.21.15.16:0/11350047 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6984070c10 0x7f69840b1e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:32.950 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:32.939+0000 7f6982d9d640 1 -- 172.21.15.16:0/11350047 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6984070c10 msgr2=0x7f69840b1e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:32.950 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:32.939+0000 7f6982d9d640 1 --2- 172.21.15.16:0/11350047 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6984070c10 0x7f69840b1e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:32.950 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:32.939+0000 7f6982d9d640 1 -- 172.21.15.16:0/11350047 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f69840c9e40 msgr2=0x7f698414d630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:32.950 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:32.939+0000 7f6982d9d640 1 --2- 172.21.15.16:0/11350047 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f69840c9e40 0x7f698414d630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:32.951 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:32.939+0000 7f6982d9d640 1 -- 172.21.15.16:0/11350047 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f696c0094d0 con 0x7f698415d650 2023-05-22T15:28:32.951 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:32.939+0000 7f6982d9d640 1 --2- 172.21.15.16:0/11350047 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f698415d650 0x7f698414db70 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f697c00c730 tx=0x7f697c00cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:28:32.951 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:32.939+0000 7f697affd640 1 -- 172.21.15.16:0/11350047 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f697c003b80 con 0x7f698415d650 2023-05-22T15:28:32.951 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:32.939+0000 7f697affd640 1 -- 172.21.15.16:0/11350047 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f697c003ce0 con 0x7f698415d650 2023-05-22T15:28:32.951 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:32.939+0000 7f6988ab3640 1 -- 172.21.15.16:0/11350047 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69840ceab0 con 0x7f698415d650 2023-05-22T15:28:32.951 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:32.939+0000 7f697affd640 1 -- 172.21.15.16:0/11350047 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f697c00aa30 con 0x7f698415d650 2023-05-22T15:28:32.951 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:32.939+0000 7f6988ab3640 1 -- 172.21.15.16:0/11350047 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69840cce90 con 0x7f698415d650 2023-05-22T15:28:32.952 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:32.939+0000 7f6988ab3640 1 -- 172.21.15.16:0/11350047 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69840cec40 con 0x7f698415d650 2023-05-22T15:28:32.952 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:32.939+0000 7f697affd640 1 -- 172.21.15.16:0/11350047 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 8) v1 ==== 84085+0+0 (secure 0 0 0) 0x7f697c00ab90 con 0x7f698415d650 2023-05-22T15:28:32.952 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:32.939+0000 7f697affd640 1 --2- 172.21.15.16:0/11350047 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f6964065220 0x7f69640676d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:32.952 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:32.939+0000 7f698259c640 1 --2- 172.21.15.16:0/11350047 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f6964065220 0x7f69640676d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:32.952 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:32.939+0000 7f697affd640 1 -- 172.21.15.16:0/11350047 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 6640+0+0 (secure 0 0 0) 0x7f697c0824d0 con 0x7f698415d650 2023-05-22T15:28:32.952 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:32.939+0000 7f698259c640 1 --2- 172.21.15.16:0/11350047 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f6964065220 0x7f69640676d0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f696c0ae000 tx=0x7f696c0a7000 comp rx=0 tx=0).ready entity=mgr.5823 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:28:32.953 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:32.943+0000 7f697affd640 1 -- 172.21.15.16:0/11350047 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f697c050250 con 0x7f698415d650 2023-05-22T15:28:33.122 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:33.115+0000 7f6988ab3640 1 -- 172.21.15.16:0/11350047 --> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/snap_test_dir1/dir3", "snap_schedule": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f698404dbe0 con 0x7f6964065220 2023-05-22T15:28:33.251 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:33.243+0000 7f697affd640 1 -- 172.21.15.16:0/11350047 <== mgr.5823 v2:172.21.15.16:6832/36921 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7f698404dbe0 con 0x7f6964065220 2023-05-22T15:28:33.251 INFO:teuthology.orchestra.run.smithi016.stdout:Schedule set for path /snap_test_dir1/dir3 2023-05-22T15:28:33.254 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:33.247+0000 7f6988ab3640 1 -- 172.21.15.16:0/11350047 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f6964065220 msgr2=0x7f69640676d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:33.254 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:33.247+0000 7f6988ab3640 1 --2- 172.21.15.16:0/11350047 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f6964065220 0x7f69640676d0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f696c0ae000 tx=0x7f696c0a7000 comp rx=0 tx=0).stop 2023-05-22T15:28:33.254 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:33.247+0000 7f6988ab3640 1 -- 172.21.15.16:0/11350047 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f698415d650 msgr2=0x7f698414db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:33.254 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:33.247+0000 7f6988ab3640 1 --2- 172.21.15.16:0/11350047 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f698415d650 0x7f698414db70 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f697c00c730 tx=0x7f697c00cbf0 comp rx=0 tx=0).stop 2023-05-22T15:28:33.254 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:33.247+0000 7f6988ab3640 1 -- 172.21.15.16:0/11350047 shutdown_connections 2023-05-22T15:28:33.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:33.247+0000 7f6988ab3640 1 --2- 172.21.15.16:0/11350047 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f698415d650 0x7f698414db70 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:33.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:33.247+0000 7f6988ab3640 1 --2- 172.21.15.16:0/11350047 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f69840c9e40 0x7f698414d630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:33.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:33.247+0000 7f6988ab3640 1 --2- 172.21.15.16:0/11350047 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f6964065220 0x7f69640676d0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:33.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:33.247+0000 7f6988ab3640 1 --2- 172.21.15.16:0/11350047 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6984070c10 0x7f69840b1e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:33.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:33.247+0000 7f6988ab3640 1 -- 172.21.15.16:0/11350047 >> 172.21.15.16:0/11350047 conn(0x7f6984069720 msgr2=0x7f6984069af0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:28:33.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:33.247+0000 7f6988ab3640 1 -- 172.21.15.16:0/11350047 shutdown_connections 2023-05-22T15:28:33.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:33.247+0000 7f6988ab3640 1 -- 172.21.15.16:0/11350047 wait complete. 2023-05-22T15:28:33.270 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/dir3 2023-05-22T15:28:33.271 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:28:33.271 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/dir4) 2023-05-22T15:28:33.280 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add --fs cephfs /snap_test_dir1/dir4 1M 2023-05-22T15:28:33.455 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:33.447+0000 7f22637e4640 1 Processor -- start 2023-05-22T15:28:33.455 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:33.447+0000 7f22637e4640 1 -- start start 2023-05-22T15:28:33.456 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:33.451+0000 7f22637e4640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f225c070c10 0x7f225c0cc4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:33.456 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:33.451+0000 7f22637e4640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f225c15d650 0x7f225c15da70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:33.456 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:33.451+0000 7f22637e4640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f225c15dfb0 con 0x7f225c0c9e40 2023-05-22T15:28:33.457 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:33.451+0000 7f22637e4640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f225c0ccbb0 con 0x7f225c15d650 2023-05-22T15:28:33.457 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:33.451+0000 7f22637e4640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f225c0cd7c0 con 0x7f225c070c10 2023-05-22T15:28:33.457 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:33.451+0000 7f2261d81640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f225c15d650 0x7f225c15da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:33.457 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:33.451+0000 7f2261d81640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f225c15d650 0x7f225c15da70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42396/0 (socket says 172.21.15.16:42396) 2023-05-22T15:28:33.457 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:33.451+0000 7f2261d81640 1 -- 172.21.15.16:0/2616080274 learned_addr learned my addr 172.21.15.16:0/2616080274 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:28:33.458 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:33.451+0000 7f2261580640 1 --2- 172.21.15.16:0/2616080274 >> v2:172.21.15.62:3300/0 conn(0x7f225c070c10 0x7f225c0cc4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:33.458 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:33.451+0000 7f2261d81640 1 -- 172.21.15.16:0/2616080274 >> v2:172.21.15.62:3300/0 conn(0x7f225c070c10 msgr2=0x7f225c0cc4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:28:33.458 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:33.451+0000 7f2261d81640 1 --2- 172.21.15.16:0/2616080274 >> v2:172.21.15.62:3300/0 conn(0x7f225c070c10 0x7f225c0cc4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:33.458 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:33.451+0000 7f2261d81640 1 -- 172.21.15.16:0/2616080274 >> v1:172.21.15.62:6789/0 conn(0x7f225c0c9e40 legacy=0x7f225c06bfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:28:33.458 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:33.451+0000 7f2261d81640 1 -- 172.21.15.16:0/2616080274 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f225c0cdaa0 con 0x7f225c15d650 2023-05-22T15:28:33.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:33.451+0000 7f2261d81640 1 --2- 172.21.15.16:0/2616080274 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f225c15d650 0x7f225c15da70 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f225c0cf110 tx=0x7f225401c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93ca95c5e3360060 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:28:33.460 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:33.451+0000 7f2253fff640 1 -- 172.21.15.16:0/2616080274 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22540a5840 con 0x7f225c15d650 2023-05-22T15:28:33.460 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:33.451+0000 7f2253fff640 1 -- 172.21.15.16:0/2616080274 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22540a59a0 con 0x7f225c15d650 2023-05-22T15:28:33.460 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:33.451+0000 7f2253fff640 1 -- 172.21.15.16:0/2616080274 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22540a5d30 con 0x7f225c15d650 2023-05-22T15:28:33.460 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:33.451+0000 7f22637e4640 1 -- 172.21.15.16:0/2616080274 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f225c15d650 msgr2=0x7f225c15da70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:33.460 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:33.451+0000 7f22637e4640 1 --2- 172.21.15.16:0/2616080274 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f225c15d650 0x7f225c15da70 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f225c0cf110 tx=0x7f225401c960 comp rx=0 tx=0).stop 2023-05-22T15:28:33.460 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:33.451+0000 7f22637e4640 1 -- 172.21.15.16:0/2616080274 shutdown_connections 2023-05-22T15:28:33.460 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:33.451+0000 7f22637e4640 1 --2- 172.21.15.16:0/2616080274 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f225c15d650 0x7f225c15da70 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:33.461 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:33.451+0000 7f22637e4640 1 --2- 172.21.15.16:0/2616080274 >> v2:172.21.15.62:3300/0 conn(0x7f225c070c10 0x7f225c0cc4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:33.461 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:33.451+0000 7f22637e4640 1 -- 172.21.15.16:0/2616080274 >> 172.21.15.16:0/2616080274 conn(0x7f225c069720 msgr2=0x7f225c069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:28:33.461 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:33.451+0000 7f22637e4640 1 -- 172.21.15.16:0/2616080274 shutdown_connections 2023-05-22T15:28:33.461 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:33.451+0000 7f22637e4640 1 -- 172.21.15.16:0/2616080274 wait complete. 2023-05-22T15:28:33.462 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:33.455+0000 7f22637e4640 1 Processor -- start 2023-05-22T15:28:33.462 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:33.455+0000 7f22637e4640 1 -- start start 2023-05-22T15:28:33.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:33.455+0000 7f22637e4640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f225c070c10 0x7f225c14b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:33.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:33.455+0000 7f22637e4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f225c0c9e40 0x7f225c144ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:33.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:33.455+0000 7f22637e4640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f225c15d650 0x7f225c15c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:33.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:33.455+0000 7f22637e4640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f225c144ff0 con 0x7f225c0c9e40 2023-05-22T15:28:33.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:33.455+0000 7f22637e4640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f225c15cf40 con 0x7f225c070c10 2023-05-22T15:28:33.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:33.455+0000 7f22637e4640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f225c15d220 con 0x7f225c15d650 2023-05-22T15:28:33.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:33.455+0000 7f2260d7f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f225c0c9e40 0x7f225c144ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:33.464 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:33.455+0000 7f2260d7f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f225c0c9e40 0x7f225c144ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42412/0 (socket says 172.21.15.16:42412) 2023-05-22T15:28:33.464 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:33.455+0000 7f2260d7f640 1 -- 172.21.15.16:0/3796294935 learned_addr learned my addr 172.21.15.16:0/3796294935 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:28:33.464 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:33.455+0000 7f2261580640 1 --2- 172.21.15.16:0/3796294935 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f225c070c10 0x7f225c14b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:33.464 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:33.455+0000 7f2261d81640 1 --2- 172.21.15.16:0/3796294935 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f225c15d650 0x7f225c15c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:33.464 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:33.455+0000 7f2260d7f640 1 -- 172.21.15.16:0/3796294935 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f225c15d650 msgr2=0x7f225c15c6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:33.464 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:33.455+0000 7f2260d7f640 1 --2- 172.21.15.16:0/3796294935 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f225c15d650 0x7f225c15c6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:33.464 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:33.455+0000 7f2260d7f640 1 -- 172.21.15.16:0/3796294935 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f225c070c10 msgr2=0x7f225c14b2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:33.465 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:33.455+0000 7f2260d7f640 1 --2- 172.21.15.16:0/3796294935 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f225c070c10 0x7f225c14b2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:33.465 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:33.455+0000 7f2260d7f640 1 -- 172.21.15.16:0/3796294935 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2254009b30 con 0x7f225c0c9e40 2023-05-22T15:28:33.465 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:33.455+0000 7f2260d7f640 1 --2- 172.21.15.16:0/3796294935 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f225c0c9e40 0x7f225c144ab0 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f224c00c490 tx=0x7f224c001e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:28:33.466 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:33.455+0000 7f2251ffb640 1 -- 172.21.15.16:0/3796294935 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f224c003600 con 0x7f225c0c9e40 2023-05-22T15:28:33.466 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:33.455+0000 7f2251ffb640 1 -- 172.21.15.16:0/3796294935 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f224c003780 con 0x7f225c0c9e40 2023-05-22T15:28:33.466 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:33.455+0000 7f22637e4640 1 -- 172.21.15.16:0/3796294935 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f225c0cce90 con 0x7f225c0c9e40 2023-05-22T15:28:33.467 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:33.455+0000 7f2251ffb640 1 -- 172.21.15.16:0/3796294935 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f224c003b30 con 0x7f225c0c9e40 2023-05-22T15:28:33.467 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:33.455+0000 7f22637e4640 1 -- 172.21.15.16:0/3796294935 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f225c0ca2c0 con 0x7f225c0c9e40 2023-05-22T15:28:33.467 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:33.455+0000 7f22637e4640 1 -- 172.21.15.16:0/3796294935 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f225c0ceab0 con 0x7f225c0c9e40 2023-05-22T15:28:33.467 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:33.459+0000 7f2251ffb640 1 -- 172.21.15.16:0/3796294935 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 8) v1 ==== 84085+0+0 (secure 0 0 0) 0x7f224c003c90 con 0x7f225c0c9e40 2023-05-22T15:28:33.467 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:33.459+0000 7f2251ffb640 1 --2- 172.21.15.16:0/3796294935 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f2240065220 0x7f22400676d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:33.467 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:33.459+0000 7f2261580640 1 --2- 172.21.15.16:0/3796294935 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f2240065220 0x7f22400676d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:33.467 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:33.459+0000 7f2251ffb640 1 -- 172.21.15.16:0/3796294935 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 6640+0+0 (secure 0 0 0) 0x7f224c081930 con 0x7f225c0c9e40 2023-05-22T15:28:33.468 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:33.459+0000 7f2261580640 1 --2- 172.21.15.16:0/3796294935 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f2240065220 0x7f22400676d0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f225800a970 tx=0x7f2258008040 comp rx=0 tx=0).ready entity=mgr.5823 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:28:33.469 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:33.463+0000 7f2251ffb640 1 -- 172.21.15.16:0/3796294935 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f224c04f650 con 0x7f225c0c9e40 2023-05-22T15:28:33.539 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:33.531+0000 7f2251ffb640 1 -- 172.21.15.16:0/3796294935 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mgrmap(e 9) v1 ==== 84406+0+0 (secure 0 0 0) 0x7f224c083d50 con 0x7f225c0c9e40 2023-05-22T15:28:33.647 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:33.639+0000 7f22637e4640 1 -- 172.21.15.16:0/3796294935 --> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/snap_test_dir1/dir4", "snap_schedule": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f225c0ab7c0 con 0x7f2240065220 2023-05-22T15:28:33.778 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:33.771+0000 7f2251ffb640 1 -- 172.21.15.16:0/3796294935 <== mgr.5823 v2:172.21.15.16:6832/36921 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7f225c0ab7c0 con 0x7f2240065220 2023-05-22T15:28:33.778 INFO:teuthology.orchestra.run.smithi016.stdout:Schedule set for path /snap_test_dir1/dir4 2023-05-22T15:28:33.781 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:33.775+0000 7f22637e4640 1 -- 172.21.15.16:0/3796294935 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f2240065220 msgr2=0x7f22400676d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:33.781 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:33.775+0000 7f22637e4640 1 --2- 172.21.15.16:0/3796294935 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f2240065220 0x7f22400676d0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f225800a970 tx=0x7f2258008040 comp rx=0 tx=0).stop 2023-05-22T15:28:33.781 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:33.775+0000 7f22637e4640 1 -- 172.21.15.16:0/3796294935 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f225c0c9e40 msgr2=0x7f225c144ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:33.782 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:33.775+0000 7f22637e4640 1 --2- 172.21.15.16:0/3796294935 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f225c0c9e40 0x7f225c144ab0 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f224c00c490 tx=0x7f224c001e80 comp rx=0 tx=0).stop 2023-05-22T15:28:33.782 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:33.775+0000 7f22637e4640 1 -- 172.21.15.16:0/3796294935 shutdown_connections 2023-05-22T15:28:33.782 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:33.775+0000 7f22637e4640 1 --2- 172.21.15.16:0/3796294935 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f2240065220 0x7f22400676d0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:33.782 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:33.775+0000 7f22637e4640 1 --2- 172.21.15.16:0/3796294935 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f225c15d650 0x7f225c15c6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:33.783 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:33.775+0000 7f22637e4640 1 --2- 172.21.15.16:0/3796294935 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f225c0c9e40 0x7f225c144ab0 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:33.783 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:33.775+0000 7f22637e4640 1 --2- 172.21.15.16:0/3796294935 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f225c070c10 0x7f225c14b2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:33.783 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:33.775+0000 7f22637e4640 1 -- 172.21.15.16:0/3796294935 >> 172.21.15.16:0/3796294935 conn(0x7f225c069720 msgr2=0x7f225c0712a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:28:33.783 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:33.775+0000 7f22637e4640 1 -- 172.21.15.16:0/3796294935 shutdown_connections 2023-05-22T15:28:33.785 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:33.775+0000 7f22637e4640 1 -- 172.21.15.16:0/3796294935 wait complete. 2023-05-22T15:28:33.802 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/dir4 2023-05-22T15:28:33.802 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:28:33.802 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/dir5) 2023-05-22T15:28:33.812 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add --fs cephfs /snap_test_dir1/dir5 1M 2023-05-22T15:28:33.992 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:33.983+0000 7f421c208640 1 Processor -- start 2023-05-22T15:28:33.992 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:33.983+0000 7f421c208640 1 -- start start 2023-05-22T15:28:33.993 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:33.987+0000 7f421c208640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f42140cfa40 0x7f42140cd260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:33.993 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:33.987+0000 7f421c208640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f421415c0a0 con 0x7f42140cf670 2023-05-22T15:28:33.993 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:33.987+0000 7f421c208640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f421406b3c0 con 0x7f421406a830 2023-05-22T15:28:33.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:33.987+0000 7f421c208640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42140cc850 con 0x7f42140cfa40 2023-05-22T15:28:33.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:33.987+0000 7f4219fa4640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f42140cf670 0x7f42141601d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:47956/0 (socket says 172.21.15.16:47956) 2023-05-22T15:28:33.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:33.987+0000 7f4219fa4640 1 -- 172.21.15.16:0/2677554633 learned_addr learned my addr 172.21.15.16:0/2677554633 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:28:33.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:33.987+0000 7f42197a3640 1 --2- 172.21.15.16:0/2677554633 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f42140cfa40 0x7f42140cd260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:33.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:33.987+0000 7f4218fa2640 1 -- 172.21.15.16:0/2677554633 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1493677453 0 0) 0x7f4210001120 con 0x7f42140cf670 2023-05-22T15:28:33.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:33.987+0000 7f4218fa2640 1 -- 172.21.15.16:0/2677554633 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4200003530 con 0x7f42140cf670 2023-05-22T15:28:33.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:33.987+0000 7f4218fa2640 1 -- 172.21.15.16:0/2677554633 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1555934784 0 0) 0x7f4200003530 con 0x7f42140cf670 2023-05-22T15:28:33.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:33.987+0000 7f4218fa2640 1 -- 172.21.15.16:0/2677554633 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f42140cfa40 msgr2=0x7f42140cd260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:28:33.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:33.987+0000 7f4218fa2640 1 --2- 172.21.15.16:0/2677554633 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f42140cfa40 0x7f42140cd260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:33.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:33.987+0000 7f4218fa2640 1 -- 172.21.15.16:0/2677554633 >> v1:172.21.15.62:6789/0 conn(0x7f421406a830 legacy=0x7f421406acb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:33.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:33.987+0000 7f4218fa2640 1 -- 172.21.15.16:0/2677554633 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42140ccdd0 con 0x7f42140cf670 2023-05-22T15:28:33.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:33.987+0000 7f4218fa2640 1 -- 172.21.15.16:0/2677554633 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7f4210001810 con 0x7f42140cf670 2023-05-22T15:28:33.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:33.987+0000 7f4218fa2640 1 -- 172.21.15.16:0/2677554633 >> v1:172.21.15.16:6789/0 conn(0x7f42140cf670 legacy=0x7f42141601d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:33.998 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:33.987+0000 7f4218fa2640 1 --2- 172.21.15.16:0/2677554633 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f42000039b0 0x7f4200003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:33.998 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:33.987+0000 7f4218fa2640 1 --2- 172.21.15.16:0/2677554633 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f42000047d0 0x7f4200004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:33.998 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:33.987+0000 7f4218fa2640 1 --2- 172.21.15.16:0/2677554633 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4200005180 0x7f420000b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:33.998 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:33.987+0000 7f4218fa2640 1 -- 172.21.15.16:0/2677554633 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42140cc850 con 0x7f42000047d0 2023-05-22T15:28:33.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:33.987+0000 7f4218fa2640 1 -- 172.21.15.16:0/2677554633 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f420000c000 con 0x7f42000039b0 2023-05-22T15:28:33.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:33.987+0000 7f4218fa2640 1 -- 172.21.15.16:0/2677554633 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f420000c1c0 con 0x7f4200005180 2023-05-22T15:28:33.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:33.987+0000 7f42197a3640 1 --2- 172.21.15.16:0/2677554633 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f42140cfa40 0x7f42140cd260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T15:28:33.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:33.987+0000 7f42197a3640 1 --2- 172.21.15.16:0/2677554633 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f42000047d0 0x7f4200004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:33.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:33.987+0000 7f4219fa4640 1 --2- 172.21.15.16:0/2677554633 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f42000039b0 0x7f4200003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:33.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:33.987+0000 7f421a7a5640 1 --2- 172.21.15.16:0/2677554633 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4200005180 0x7f420000b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:34.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:33.987+0000 7f42197a3640 1 -- 172.21.15.16:0/2677554633 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4200005180 msgr2=0x7f420000b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:28:34.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:33.987+0000 7f42197a3640 1 --2- 172.21.15.16:0/2677554633 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4200005180 0x7f420000b990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:34.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:33.987+0000 7f42197a3640 1 -- 172.21.15.16:0/2677554633 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f42000039b0 msgr2=0x7f4200003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:28:34.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:33.987+0000 7f42197a3640 1 --2- 172.21.15.16:0/2677554633 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f42000039b0 0x7f4200003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:34.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:33.987+0000 7f42197a3640 1 -- 172.21.15.16:0/2677554633 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4200003530 con 0x7f42000047d0 2023-05-22T15:28:34.004 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:33.991+0000 7f42197a3640 1 --2- 172.21.15.16:0/2677554633 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f42000047d0 0x7f4200004c40 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f421415f720 tx=0x7f420401c3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc0983f179482a80 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:28:34.005 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:33.991+0000 7f4218fa2640 1 -- 172.21.15.16:0/2677554633 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42040ab600 con 0x7f42000047d0 2023-05-22T15:28:34.005 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:33.991+0000 7f4218fa2640 1 -- 172.21.15.16:0/2677554633 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42040a7050 con 0x7f42000047d0 2023-05-22T15:28:34.005 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:33.991+0000 7f4218fa2640 1 -- 172.21.15.16:0/2677554633 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42040ab980 con 0x7f42000047d0 2023-05-22T15:28:34.005 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:33.991+0000 7f421c208640 1 -- 172.21.15.16:0/2677554633 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f42000047d0 msgr2=0x7f4200004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:34.005 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:33.991+0000 7f421c208640 1 --2- 172.21.15.16:0/2677554633 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f42000047d0 0x7f4200004c40 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f421415f720 tx=0x7f420401c3f0 comp rx=0 tx=0).stop 2023-05-22T15:28:34.005 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:33.991+0000 7f421a7a5640 1 -- 172.21.15.16:0/2677554633 reap_dead start 2023-05-22T15:28:34.006 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:33.991+0000 7f421c208640 1 -- 172.21.15.16:0/2677554633 shutdown_connections 2023-05-22T15:28:34.006 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:33.991+0000 7f421c208640 1 -- 172.21.15.16:0/2677554633 >> 172.21.15.16:0/2677554633 conn(0x7f42140700b0 msgr2=0x7f421406d9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:28:34.006 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:33.991+0000 7f421c208640 1 -- 172.21.15.16:0/2677554633 shutdown_connections 2023-05-22T15:28:34.006 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:33.991+0000 7f421c208640 1 -- 172.21.15.16:0/2677554633 wait complete. 2023-05-22T15:28:34.006 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:33.991+0000 7f421c208640 1 Processor -- start 2023-05-22T15:28:34.006 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:33.991+0000 7f421c208640 1 -- start start 2023-05-22T15:28:34.006 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:33.991+0000 7f421c208640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f42000047d0 0x7f4214155e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:34.006 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:33.991+0000 7f421c208640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f42140acd60 0x7f4214158a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:34.007 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:33.991+0000 7f421c208640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f42140ae4e0 0x7f42140b3d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:34.007 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:33.991+0000 7f421c208640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42141574a0 con 0x7f42140acd60 2023-05-22T15:28:34.007 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:33.991+0000 7f421c208640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4214157170 con 0x7f42000047d0 2023-05-22T15:28:34.007 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:33.991+0000 7f421c208640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42140aef30 con 0x7f42140ae4e0 2023-05-22T15:28:34.007 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:33.991+0000 7f42197a3640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f42140acd60 0x7f4214158a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:34.007 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:33.991+0000 7f42197a3640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f42140acd60 0x7f4214158a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42428/0 (socket says 172.21.15.16:42428) 2023-05-22T15:28:34.007 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:33.991+0000 7f42197a3640 1 -- 172.21.15.16:0/1302912108 learned_addr learned my addr 172.21.15.16:0/1302912108 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:28:34.007 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:33.995+0000 7f421a7a5640 1 --2- 172.21.15.16:0/1302912108 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f42140ae4e0 0x7f42140b3d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:34.008 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:33.995+0000 7f4219fa4640 1 --2- 172.21.15.16:0/1302912108 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f42000047d0 0x7f4214155e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:34.008 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:33.995+0000 7f42197a3640 1 -- 172.21.15.16:0/1302912108 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f42140ae4e0 msgr2=0x7f42140b3d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:34.008 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:33.995+0000 7f42197a3640 1 --2- 172.21.15.16:0/1302912108 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f42140ae4e0 0x7f42140b3d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:34.008 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:33.995+0000 7f42197a3640 1 -- 172.21.15.16:0/1302912108 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f42000047d0 msgr2=0x7f4214155e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:34.008 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:33.995+0000 7f42197a3640 1 --2- 172.21.15.16:0/1302912108 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f42000047d0 0x7f4214155e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:34.009 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:33.995+0000 7f42197a3640 1 -- 172.21.15.16:0/1302912108 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f421406c6b0 con 0x7f42140acd60 2023-05-22T15:28:34.009 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:33.995+0000 7f42197a3640 1 --2- 172.21.15.16:0/1302912108 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f42140acd60 0x7f4214158a10 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f420401c220 tx=0x7f4200005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:28:34.009 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:33.995+0000 7f420a7fc640 1 -- 172.21.15.16:0/1302912108 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42040b5620 con 0x7f42140acd60 2023-05-22T15:28:34.009 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:33.995+0000 7f420a7fc640 1 -- 172.21.15.16:0/1302912108 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42040b5780 con 0x7f42140acd60 2023-05-22T15:28:34.009 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:33.995+0000 7f421c208640 1 -- 172.21.15.16:0/1302912108 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f421406c3d0 con 0x7f42140acd60 2023-05-22T15:28:34.009 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:33.995+0000 7f420a7fc640 1 -- 172.21.15.16:0/1302912108 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42040b5b10 con 0x7f42140acd60 2023-05-22T15:28:34.009 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:33.995+0000 7f421c208640 1 -- 172.21.15.16:0/1302912108 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42140cca60 con 0x7f42140acd60 2023-05-22T15:28:34.009 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:33.995+0000 7f421c208640 1 -- 172.21.15.16:0/1302912108 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42140ce680 con 0x7f42140acd60 2023-05-22T15:28:34.010 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:33.995+0000 7f420a7fc640 1 -- 172.21.15.16:0/1302912108 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 9) v1 ==== 84406+0+0 (secure 0 0 0) 0x7f42040b2070 con 0x7f42140acd60 2023-05-22T15:28:34.010 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:33.995+0000 7f420a7fc640 1 --2- 172.21.15.16:0/1302912108 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f41f80655b0 0x7f41f8067a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:34.010 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:33.995+0000 7f420a7fc640 1 -- 172.21.15.16:0/1302912108 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 6640+0+0 (secure 0 0 0) 0x7f420411d270 con 0x7f42140acd60 2023-05-22T15:28:34.010 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:33.995+0000 7f4219fa4640 1 --2- 172.21.15.16:0/1302912108 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f41f80655b0 0x7f41f8067a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:34.010 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:33.995+0000 7f4219fa4640 1 --2- 172.21.15.16:0/1302912108 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f41f80655b0 0x7f41f8067a60 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f42100012c0 tx=0x7f4210009040 comp rx=0 tx=0).ready entity=mgr.5823 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:28:34.011 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:33.999+0000 7f420a7fc640 1 -- 172.21.15.16:0/1302912108 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f42040eaa60 con 0x7f42140acd60 2023-05-22T15:28:34.179 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:34.171+0000 7f421c208640 1 -- 172.21.15.16:0/1302912108 --> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/snap_test_dir1/dir5", "snap_schedule": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f421404dbe0 con 0x7f41f80655b0 2023-05-22T15:28:34.321 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:34.315+0000 7f420a7fc640 1 -- 172.21.15.16:0/1302912108 <== mgr.5823 v2:172.21.15.16:6832/36921 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7f421404dbe0 con 0x7f41f80655b0 2023-05-22T15:28:34.322 INFO:teuthology.orchestra.run.smithi016.stdout:Schedule set for path /snap_test_dir1/dir5 2023-05-22T15:28:34.325 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:34.319+0000 7f421c208640 1 -- 172.21.15.16:0/1302912108 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f41f80655b0 msgr2=0x7f41f8067a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:34.325 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:34.319+0000 7f421c208640 1 --2- 172.21.15.16:0/1302912108 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f41f80655b0 0x7f41f8067a60 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f42100012c0 tx=0x7f4210009040 comp rx=0 tx=0).stop 2023-05-22T15:28:34.325 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:34.319+0000 7f421c208640 1 -- 172.21.15.16:0/1302912108 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f42140acd60 msgr2=0x7f4214158a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:34.325 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:34.319+0000 7f421c208640 1 --2- 172.21.15.16:0/1302912108 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f42140acd60 0x7f4214158a10 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f420401c220 tx=0x7f4200005800 comp rx=0 tx=0).stop 2023-05-22T15:28:34.325 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:34.319+0000 7f421c208640 1 -- 172.21.15.16:0/1302912108 shutdown_connections 2023-05-22T15:28:34.325 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:34.319+0000 7f421c208640 1 --2- 172.21.15.16:0/1302912108 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f41f80655b0 0x7f41f8067a60 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:34.326 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:34.319+0000 7f421c208640 1 --2- 172.21.15.16:0/1302912108 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f42140ae4e0 0x7f42140b3d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:34.326 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:34.319+0000 7f421c208640 1 --2- 172.21.15.16:0/1302912108 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f42140acd60 0x7f4214158a10 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:34.326 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:34.319+0000 7f421c208640 1 --2- 172.21.15.16:0/1302912108 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f42000047d0 0x7f4214155e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:34.326 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:34.319+0000 7f421c208640 1 -- 172.21.15.16:0/1302912108 >> 172.21.15.16:0/1302912108 conn(0x7f42140700b0 msgr2=0x7f42140cba30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:28:34.326 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:34.319+0000 7f421c208640 1 -- 172.21.15.16:0/1302912108 shutdown_connections 2023-05-22T15:28:34.326 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:34.319+0000 7f421c208640 1 -- 172.21.15.16:0/1302912108 wait complete. 2023-05-22T15:28:34.348 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/dir5 2023-05-22T15:28:34.348 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:28:34.349 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/dir6) 2023-05-22T15:28:34.358 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add --fs cephfs /snap_test_dir1/dir6 1M 2023-05-22T15:28:34.539 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:34.531+0000 7fc6f3431640 1 Processor -- start 2023-05-22T15:28:34.540 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:34.531+0000 7fc6f3431640 1 -- start start 2023-05-22T15:28:34.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:34.535+0000 7fc6f3431640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc6ec070c10 0x7fc6ec071580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:34.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:34.535+0000 7fc6f3431640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6ec0cdda0 con 0x7fc6ec0cd620 2023-05-22T15:28:34.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:34.535+0000 7fc6f3431640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6ec15a190 con 0x7fc6ec0ce740 2023-05-22T15:28:34.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:34.535+0000 7fc6f3431640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6ec0cb880 con 0x7fc6ec070c10 2023-05-22T15:28:34.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:34.535+0000 7fc6f09cc640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fc6ec0cd620 0x7fc6ec0ce030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:47960/0 (socket says 172.21.15.16:47960) 2023-05-22T15:28:34.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:34.535+0000 7fc6f09cc640 1 -- 172.21.15.16:0/1155947065 learned_addr learned my addr 172.21.15.16:0/1155947065 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:28:34.544 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:34.535+0000 7fc6f11cd640 1 --2- 172.21.15.16:0/1155947065 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc6ec070c10 0x7fc6ec071580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:34.544 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:34.535+0000 7fc6e3fff640 1 -- 172.21.15.16:0/1155947065 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1620717940 0 0) 0x7fc6ec0cdda0 con 0x7fc6ec0cd620 2023-05-22T15:28:34.544 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:34.535+0000 7fc6e3fff640 1 -- 172.21.15.16:0/1155947065 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc6d8003530 con 0x7fc6ec0cd620 2023-05-22T15:28:34.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:34.535+0000 7fc6e3fff640 1 -- 172.21.15.16:0/1155947065 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 632319802 0 0) 0x7fc6d8003530 con 0x7fc6ec0cd620 2023-05-22T15:28:34.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:34.535+0000 7fc6e3fff640 1 -- 172.21.15.16:0/1155947065 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc6ec070c10 msgr2=0x7fc6ec071580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:28:34.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:34.535+0000 7fc6e3fff640 1 --2- 172.21.15.16:0/1155947065 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc6ec070c10 0x7fc6ec071580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:34.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:34.535+0000 7fc6e3fff640 1 -- 172.21.15.16:0/1155947065 >> v1:172.21.15.62:6789/0 conn(0x7fc6ec0ce740 legacy=0x7fc6ec150300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:34.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:34.535+0000 7fc6e3fff640 1 -- 172.21.15.16:0/1155947065 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6ec15a470 con 0x7fc6ec0cd620 2023-05-22T15:28:34.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:34.535+0000 7fc6e3fff640 1 -- 172.21.15.16:0/1155947065 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7fc6dc001680 con 0x7fc6ec0cd620 2023-05-22T15:28:34.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:34.535+0000 7fc6e3fff640 1 -- 172.21.15.16:0/1155947065 >> v1:172.21.15.16:6789/0 conn(0x7fc6ec0cd620 legacy=0x7fc6ec0ce030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:34.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:34.535+0000 7fc6e3fff640 1 --2- 172.21.15.16:0/1155947065 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc6d80039b0 0x7fc6d8003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:34.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:34.535+0000 7fc6e3fff640 1 --2- 172.21.15.16:0/1155947065 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc6ec070c10 0x7fc6d80047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:34.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:34.535+0000 7fc6e3fff640 1 --2- 172.21.15.16:0/1155947065 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc6d8004d10 0x7fc6d8005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:34.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:34.535+0000 7fc6e3fff640 1 -- 172.21.15.16:0/1155947065 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6ec0cb880 con 0x7fc6d8004d10 2023-05-22T15:28:34.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:34.535+0000 7fc6e3fff640 1 -- 172.21.15.16:0/1155947065 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6d800ba40 con 0x7fc6d80039b0 2023-05-22T15:28:34.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:34.535+0000 7fc6e3fff640 1 -- 172.21.15.16:0/1155947065 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6d800bbd0 con 0x7fc6ec070c10 2023-05-22T15:28:34.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:34.535+0000 7fc6f19ce640 1 --2- 172.21.15.16:0/1155947065 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc6d8004d10 0x7fc6d8005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:34.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:34.535+0000 7fc6f09cc640 1 --2- 172.21.15.16:0/1155947065 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc6ec070c10 0x7fc6d80047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:34.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:34.535+0000 7fc6f11cd640 1 --2- 172.21.15.16:0/1155947065 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc6d80039b0 0x7fc6d8003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:34.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:34.535+0000 7fc6f19ce640 1 -- 172.21.15.16:0/1155947065 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc6ec070c10 msgr2=0x7fc6d80047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:28:34.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:34.535+0000 7fc6f19ce640 1 --2- 172.21.15.16:0/1155947065 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc6ec070c10 0x7fc6d80047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:34.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:34.535+0000 7fc6f19ce640 1 -- 172.21.15.16:0/1155947065 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc6d80039b0 msgr2=0x7fc6d8003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:28:34.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:34.535+0000 7fc6f19ce640 1 --2- 172.21.15.16:0/1155947065 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc6d80039b0 0x7fc6d8003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:34.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:34.535+0000 7fc6f19ce640 1 -- 172.21.15.16:0/1155947065 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6d8003530 con 0x7fc6d8004d10 2023-05-22T15:28:34.550 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:34.535+0000 7fc6f19ce640 1 --2- 172.21.15.16:0/1155947065 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc6d8004d10 0x7fc6d8005180 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7fc6ec157b50 tx=0x7fc6e401c700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e45096edbc8ae57b server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:28:34.550 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:34.539+0000 7fc6e3fff640 1 -- 172.21.15.16:0/1155947065 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6e4003b80 con 0x7fc6d8004d10 2023-05-22T15:28:34.550 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:34.539+0000 7fc6e3fff640 1 -- 172.21.15.16:0/1155947065 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6e4003ce0 con 0x7fc6d8004d10 2023-05-22T15:28:34.550 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:34.539+0000 7fc6e3fff640 1 -- 172.21.15.16:0/1155947065 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6e4005310 con 0x7fc6d8004d10 2023-05-22T15:28:34.551 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:34.539+0000 7fc6f3431640 1 -- 172.21.15.16:0/1155947065 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc6d8004d10 msgr2=0x7fc6d8005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:34.551 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:34.539+0000 7fc6f3431640 1 --2- 172.21.15.16:0/1155947065 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc6d8004d10 0x7fc6d8005180 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7fc6ec157b50 tx=0x7fc6e401c700 comp rx=0 tx=0).stop 2023-05-22T15:28:34.551 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:34.539+0000 7fc6f19ce640 1 -- 172.21.15.16:0/1155947065 reap_dead start 2023-05-22T15:28:34.551 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:34.539+0000 7fc6f3431640 1 -- 172.21.15.16:0/1155947065 shutdown_connections 2023-05-22T15:28:34.551 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:34.539+0000 7fc6f3431640 1 -- 172.21.15.16:0/1155947065 >> 172.21.15.16:0/1155947065 conn(0x7fc6ec069720 msgr2=0x7fc6ec069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:28:34.551 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:34.539+0000 7fc6f3431640 1 -- 172.21.15.16:0/1155947065 shutdown_connections 2023-05-22T15:28:34.551 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:34.539+0000 7fc6f3431640 1 -- 172.21.15.16:0/1155947065 wait complete. 2023-05-22T15:28:34.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:34.539+0000 7fc6f3431640 1 Processor -- start 2023-05-22T15:28:34.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:34.539+0000 7fc6f3431640 1 -- start start 2023-05-22T15:28:34.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:34.539+0000 7fc6f3431640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc6ec1f1ff0 0x7fc6ec1f01e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:34.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:34.539+0000 7fc6f3431640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc6ec0c5170 0x7fc6ec1efae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:34.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:34.539+0000 7fc6f3431640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc6ec15c550 0x7fc6ec15c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:34.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:34.539+0000 7fc6f3431640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6ec15cf10 con 0x7fc6ec0c5170 2023-05-22T15:28:34.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:34.539+0000 7fc6f3431640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6ec1f0020 con 0x7fc6ec1f1ff0 2023-05-22T15:28:34.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:34.539+0000 7fc6f3431640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6ec1f23c0 con 0x7fc6ec15c550 2023-05-22T15:28:34.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:34.539+0000 7fc6f09cc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc6ec0c5170 0x7fc6ec1efae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:34.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:34.539+0000 7fc6f09cc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc6ec0c5170 0x7fc6ec1efae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42448/0 (socket says 172.21.15.16:42448) 2023-05-22T15:28:34.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:34.539+0000 7fc6f09cc640 1 -- 172.21.15.16:0/382401285 learned_addr learned my addr 172.21.15.16:0/382401285 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:28:34.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:34.539+0000 7fc6f11cd640 1 --2- 172.21.15.16:0/382401285 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc6ec1f1ff0 0x7fc6ec1f01e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:34.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:34.539+0000 7fc6f19ce640 1 --2- 172.21.15.16:0/382401285 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc6ec15c550 0x7fc6ec15c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:34.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:34.539+0000 7fc6f09cc640 1 -- 172.21.15.16:0/382401285 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc6ec15c550 msgr2=0x7fc6ec15c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:34.554 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:34.539+0000 7fc6f09cc640 1 --2- 172.21.15.16:0/382401285 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc6ec15c550 0x7fc6ec15c9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:34.554 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:34.539+0000 7fc6f09cc640 1 -- 172.21.15.16:0/382401285 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc6ec1f1ff0 msgr2=0x7fc6ec1f01e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:34.554 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:34.539+0000 7fc6f09cc640 1 --2- 172.21.15.16:0/382401285 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc6ec1f1ff0 0x7fc6ec1f01e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:34.554 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:34.539+0000 7fc6f09cc640 1 -- 172.21.15.16:0/382401285 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6ec15aa30 con 0x7fc6ec0c5170 2023-05-22T15:28:34.555 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:34.539+0000 7fc6f11cd640 1 --2- 172.21.15.16:0/382401285 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc6ec1f1ff0 0x7fc6ec1f01e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:28:34.555 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:34.539+0000 7fc6f09cc640 1 --2- 172.21.15.16:0/382401285 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc6ec0c5170 0x7fc6ec1efae0 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7fc6dc001210 tx=0x7fc6dc00ef80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:28:34.555 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:34.539+0000 7fc6e1ffb640 1 -- 172.21.15.16:0/382401285 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6dc00c840 con 0x7fc6ec0c5170 2023-05-22T15:28:34.555 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:34.539+0000 7fc6e1ffb640 1 -- 172.21.15.16:0/382401285 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6dc00c9c0 con 0x7fc6ec0c5170 2023-05-22T15:28:34.555 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:34.539+0000 7fc6e1ffb640 1 -- 172.21.15.16:0/382401285 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6dc00cd70 con 0x7fc6ec0c5170 2023-05-22T15:28:34.555 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:34.539+0000 7fc6f3431640 1 -- 172.21.15.16:0/382401285 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6ec15a750 con 0x7fc6ec0c5170 2023-05-22T15:28:34.556 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:34.543+0000 7fc6f3431640 1 -- 172.21.15.16:0/382401285 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6ec070fe0 con 0x7fc6ec0c5170 2023-05-22T15:28:34.556 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:34.543+0000 7fc6f3431640 1 -- 172.21.15.16:0/382401285 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6ec04dbe0 con 0x7fc6ec0c5170 2023-05-22T15:28:34.556 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:34.543+0000 7fc6e1ffb640 1 -- 172.21.15.16:0/382401285 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 9) v1 ==== 84406+0+0 (secure 0 0 0) 0x7fc6dc01d460 con 0x7fc6ec0c5170 2023-05-22T15:28:34.556 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:34.543+0000 7fc6e1ffb640 1 --2- 172.21.15.16:0/382401285 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fc6d0065580 0x7fc6d0067a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:34.556 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:34.543+0000 7fc6e1ffb640 1 -- 172.21.15.16:0/382401285 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 6640+0+0 (secure 0 0 0) 0x7fc6dc082ee0 con 0x7fc6ec0c5170 2023-05-22T15:28:34.556 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:34.543+0000 7fc6f11cd640 1 --2- 172.21.15.16:0/382401285 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fc6d0065580 0x7fc6d0067a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:34.557 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:34.543+0000 7fc6f11cd640 1 --2- 172.21.15.16:0/382401285 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fc6d0065580 0x7fc6d0067a30 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fc6ec0aed70 tx=0x7fc6e8008040 comp rx=0 tx=0).ready entity=mgr.5823 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:28:34.557 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:34.547+0000 7fc6e1ffb640 1 -- 172.21.15.16:0/382401285 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fc6dc050640 con 0x7fc6ec0c5170 2023-05-22T15:28:34.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:34.723+0000 7fc6f3431640 1 -- 172.21.15.16:0/382401285 --> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/snap_test_dir1/dir6", "snap_schedule": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7fc6ec071ae0 con 0x7fc6d0065580 2023-05-22T15:28:34.850 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:34.843+0000 7fc6e1ffb640 1 -- 172.21.15.16:0/382401285 <== mgr.5823 v2:172.21.15.16:6832/36921 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7fc6ec071ae0 con 0x7fc6d0065580 2023-05-22T15:28:34.850 INFO:teuthology.orchestra.run.smithi016.stdout:Schedule set for path /snap_test_dir1/dir6 2023-05-22T15:28:34.854 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:34.847+0000 7fc6f3431640 1 -- 172.21.15.16:0/382401285 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fc6d0065580 msgr2=0x7fc6d0067a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:34.854 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:34.847+0000 7fc6f3431640 1 --2- 172.21.15.16:0/382401285 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fc6d0065580 0x7fc6d0067a30 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fc6ec0aed70 tx=0x7fc6e8008040 comp rx=0 tx=0).stop 2023-05-22T15:28:34.855 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:34.847+0000 7fc6f3431640 1 -- 172.21.15.16:0/382401285 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc6ec0c5170 msgr2=0x7fc6ec1efae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:34.855 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:34.847+0000 7fc6f3431640 1 --2- 172.21.15.16:0/382401285 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc6ec0c5170 0x7fc6ec1efae0 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7fc6dc001210 tx=0x7fc6dc00ef80 comp rx=0 tx=0).stop 2023-05-22T15:28:34.855 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:34.847+0000 7fc6f3431640 1 -- 172.21.15.16:0/382401285 shutdown_connections 2023-05-22T15:28:34.855 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:34.847+0000 7fc6f3431640 1 --2- 172.21.15.16:0/382401285 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fc6d0065580 0x7fc6d0067a30 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:34.855 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:34.847+0000 7fc6f3431640 1 --2- 172.21.15.16:0/382401285 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc6ec15c550 0x7fc6ec15c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:34.855 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:34.847+0000 7fc6f3431640 1 --2- 172.21.15.16:0/382401285 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc6ec0c5170 0x7fc6ec1efae0 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:34.856 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:34.847+0000 7fc6f3431640 1 --2- 172.21.15.16:0/382401285 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc6ec1f1ff0 0x7fc6ec1f01e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:34.856 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:34.847+0000 7fc6f3431640 1 -- 172.21.15.16:0/382401285 >> 172.21.15.16:0/382401285 conn(0x7fc6ec069720 msgr2=0x7fc6ec06f670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:28:34.856 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:34.847+0000 7fc6f3431640 1 -- 172.21.15.16:0/382401285 shutdown_connections 2023-05-22T15:28:34.856 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:34.847+0000 7fc6f3431640 1 -- 172.21.15.16:0/382401285 wait complete. 2023-05-22T15:28:34.875 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/dir6 2023-05-22T15:28:34.876 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:28:34.876 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/dir7) 2023-05-22T15:28:34.885 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add --fs cephfs /snap_test_dir1/dir7 1M 2023-05-22T15:28:35.066 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:35.059+0000 7f606b5e5640 1 Processor -- start 2023-05-22T15:28:35.067 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:35.059+0000 7f606b5e5640 1 -- start start 2023-05-22T15:28:35.070 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:35.059+0000 7f606b5e5640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f60640cc4e0 0x7f606406bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:35.070 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:35.059+0000 7f606b5e5640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60640cc960 con 0x7f6064070c10 2023-05-22T15:28:35.070 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:35.059+0000 7f606b5e5640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60640ccbb0 con 0x7f606415d650 2023-05-22T15:28:35.070 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:35.059+0000 7f606b5e5640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60640cd7c0 con 0x7f60640cc4e0 2023-05-22T15:28:35.070 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:35.063+0000 7f6069381640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f6064070c10 0x7f60640c9e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:47976/0 (socket says 172.21.15.16:47976) 2023-05-22T15:28:35.071 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:35.063+0000 7f6069381640 1 -- 172.21.15.16:0/2461395777 learned_addr learned my addr 172.21.15.16:0/2461395777 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:28:35.071 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:35.063+0000 7f6068b80640 1 --2- 172.21.15.16:0/2461395777 >> v2:172.21.15.62:3300/0 conn(0x7f60640cc4e0 0x7f606406bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:35.071 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:35.063+0000 7f6057fff640 1 -- 172.21.15.16:0/2461395777 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3157887200 0 0) 0x7f60640cc960 con 0x7f6064070c10 2023-05-22T15:28:35.071 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:35.063+0000 7f6057fff640 1 -- 172.21.15.16:0/2461395777 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6050003530 con 0x7f6064070c10 2023-05-22T15:28:35.071 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:35.063+0000 7f6057fff640 1 -- 172.21.15.16:0/2461395777 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1041608464 0 0) 0x7f6050003530 con 0x7f6064070c10 2023-05-22T15:28:35.071 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:35.063+0000 7f6057fff640 1 -- 172.21.15.16:0/2461395777 >> v2:172.21.15.62:3300/0 conn(0x7f60640cc4e0 msgr2=0x7f606406bfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:28:35.071 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:35.063+0000 7f6057fff640 1 --2- 172.21.15.16:0/2461395777 >> v2:172.21.15.62:3300/0 conn(0x7f60640cc4e0 0x7f606406bfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:35.071 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:35.063+0000 7f6057fff640 1 -- 172.21.15.16:0/2461395777 >> v1:172.21.15.62:6789/0 conn(0x7f606415d650 legacy=0x7f60640ce2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:35.072 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:35.063+0000 7f6057fff640 1 -- 172.21.15.16:0/2461395777 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60640cdaa0 con 0x7f6064070c10 2023-05-22T15:28:35.072 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:35.063+0000 7f6057fff640 1 -- 172.21.15.16:0/2461395777 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7f6058001680 con 0x7f6064070c10 2023-05-22T15:28:35.072 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:35.063+0000 7f6057fff640 1 -- 172.21.15.16:0/2461395777 >> v1:172.21.15.16:6789/0 conn(0x7f6064070c10 legacy=0x7f60640c9e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:35.072 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:35.063+0000 7f6057fff640 1 --2- 172.21.15.16:0/2461395777 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6050003980 0x7f6050003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:35.072 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:35.063+0000 7f6057fff640 1 --2- 172.21.15.16:0/2461395777 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f60500047d0 0x7f6050004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:35.073 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:35.063+0000 7f6057fff640 1 --2- 172.21.15.16:0/2461395777 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6050005180 0x7f605000b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:35.073 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:35.063+0000 7f6057fff640 1 -- 172.21.15.16:0/2461395777 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60640cd7c0 con 0x7f6050005180 2023-05-22T15:28:35.073 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:35.063+0000 7f6057fff640 1 -- 172.21.15.16:0/2461395777 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f605000c000 con 0x7f60500047d0 2023-05-22T15:28:35.073 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:35.063+0000 7f6057fff640 1 -- 172.21.15.16:0/2461395777 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f605000c1c0 con 0x7f6050003980 2023-05-22T15:28:35.073 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:35.063+0000 7f6069b82640 1 --2- 172.21.15.16:0/2461395777 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6050005180 0x7f605000b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:35.073 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:35.063+0000 7f6068b80640 1 --2- 172.21.15.16:0/2461395777 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f60500047d0 0x7f6050004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:35.073 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:35.063+0000 7f6069381640 1 --2- 172.21.15.16:0/2461395777 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6050003980 0x7f6050003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:35.074 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:35.063+0000 7f6069b82640 1 -- 172.21.15.16:0/2461395777 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6050003980 msgr2=0x7f6050003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:28:35.074 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:35.063+0000 7f6069b82640 1 --2- 172.21.15.16:0/2461395777 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6050003980 0x7f6050003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:35.074 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:35.063+0000 7f6069b82640 1 -- 172.21.15.16:0/2461395777 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f60500047d0 msgr2=0x7f6050004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:28:35.075 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:35.063+0000 7f6069b82640 1 --2- 172.21.15.16:0/2461395777 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f60500047d0 0x7f6050004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:35.075 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:35.063+0000 7f6069b82640 1 -- 172.21.15.16:0/2461395777 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6050003530 con 0x7f6050005180 2023-05-22T15:28:35.075 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:35.063+0000 7f6068b80640 1 --2- 172.21.15.16:0/2461395777 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f60500047d0 0x7f6050004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T15:28:35.078 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:35.063+0000 7f6069b82640 1 -- 172.21.15.16:0/2461395777 reap_dead start 2023-05-22T15:28:35.079 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:35.063+0000 7f6069b82640 1 --2- 172.21.15.16:0/2461395777 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6050005180 0x7f605000b990 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f605c001e70 tx=0x7f605c01c760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=494369f62cb6d4db server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:28:35.079 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:35.063+0000 7f6057fff640 1 -- 172.21.15.16:0/2461395777 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f605c003b80 con 0x7f6050005180 2023-05-22T15:28:35.079 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:35.063+0000 7f6057fff640 1 -- 172.21.15.16:0/2461395777 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f605c003d00 con 0x7f6050005180 2023-05-22T15:28:35.079 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:35.063+0000 7f6057fff640 1 -- 172.21.15.16:0/2461395777 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f605c005260 con 0x7f6050005180 2023-05-22T15:28:35.079 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:35.067+0000 7f606b5e5640 1 -- 172.21.15.16:0/2461395777 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6050005180 msgr2=0x7f605000b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:35.079 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:35.067+0000 7f606b5e5640 1 --2- 172.21.15.16:0/2461395777 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6050005180 0x7f605000b990 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f605c001e70 tx=0x7f605c01c760 comp rx=0 tx=0).stop 2023-05-22T15:28:35.079 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:35.067+0000 7f606b5e5640 1 -- 172.21.15.16:0/2461395777 shutdown_connections 2023-05-22T15:28:35.080 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:35.067+0000 7f606b5e5640 1 --2- 172.21.15.16:0/2461395777 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6050005180 0x7f605000b990 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:35.080 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:35.067+0000 7f606b5e5640 1 -- 172.21.15.16:0/2461395777 >> 172.21.15.16:0/2461395777 conn(0x7f6064069720 msgr2=0x7f6064069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:28:35.080 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:35.067+0000 7f606b5e5640 1 -- 172.21.15.16:0/2461395777 shutdown_connections 2023-05-22T15:28:35.080 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:35.067+0000 7f606b5e5640 1 -- 172.21.15.16:0/2461395777 wait complete. 2023-05-22T15:28:35.080 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:35.067+0000 7f606b5e5640 1 Processor -- start 2023-05-22T15:28:35.080 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:35.067+0000 7f606b5e5640 1 -- start start 2023-05-22T15:28:35.080 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:35.067+0000 7f606b5e5640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6050005180 0x7f606415ce70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:35.081 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:35.067+0000 7f606b5e5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f606414fbd0 0x7f6064151780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:35.081 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:35.067+0000 7f606b5e5640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f60641502d0 0x7f60640c4500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:35.081 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:35.067+0000 7f606b5e5640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6064150090 con 0x7f606414fbd0 2023-05-22T15:28:35.081 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:35.067+0000 7f606b5e5640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f606415be10 con 0x7f6050005180 2023-05-22T15:28:35.081 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:35.067+0000 7f606b5e5640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f606415c0b0 con 0x7f60641502d0 2023-05-22T15:28:35.081 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:35.067+0000 7f6068b80640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f606414fbd0 0x7f6064151780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:35.082 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:35.067+0000 7f6069381640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6050005180 0x7f606415ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:35.082 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:35.067+0000 7f6068b80640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f606414fbd0 0x7f6064151780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42464/0 (socket says 172.21.15.16:42464) 2023-05-22T15:28:35.082 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:35.067+0000 7f6068b80640 1 -- 172.21.15.16:0/2281304715 learned_addr learned my addr 172.21.15.16:0/2281304715 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:28:35.082 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:35.067+0000 7f6069b82640 1 --2- 172.21.15.16:0/2281304715 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f60641502d0 0x7f60640c4500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:35.082 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:35.067+0000 7f6068b80640 1 -- 172.21.15.16:0/2281304715 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f60641502d0 msgr2=0x7f60640c4500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:35.082 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:35.067+0000 7f6068b80640 1 --2- 172.21.15.16:0/2281304715 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f60641502d0 0x7f60640c4500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:35.082 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:35.067+0000 7f6068b80640 1 -- 172.21.15.16:0/2281304715 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6050005180 msgr2=0x7f606415ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:35.083 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:35.067+0000 7f6068b80640 1 --2- 172.21.15.16:0/2281304715 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6050005180 0x7f606415ce70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:35.083 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:35.067+0000 7f6068b80640 1 -- 172.21.15.16:0/2281304715 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6064070fe0 con 0x7f606414fbd0 2023-05-22T15:28:35.083 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:35.067+0000 7f6068b80640 1 --2- 172.21.15.16:0/2281304715 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f606414fbd0 0x7f6064151780 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f6060009160 tx=0x7f6060011a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:28:35.083 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:35.067+0000 7f6055ffb640 1 -- 172.21.15.16:0/2281304715 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6060004220 con 0x7f606414fbd0 2023-05-22T15:28:35.083 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:35.067+0000 7f6055ffb640 1 -- 172.21.15.16:0/2281304715 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6060004380 con 0x7f606414fbd0 2023-05-22T15:28:35.083 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:35.067+0000 7f6055ffb640 1 -- 172.21.15.16:0/2281304715 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6060015890 con 0x7f606414fbd0 2023-05-22T15:28:35.084 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:35.067+0000 7f606b5e5640 1 -- 172.21.15.16:0/2281304715 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60640cdd80 con 0x7f606414fbd0 2023-05-22T15:28:35.084 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:35.067+0000 7f606b5e5640 1 -- 172.21.15.16:0/2281304715 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60640cce90 con 0x7f606414fbd0 2023-05-22T15:28:35.084 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:35.071+0000 7f606b5e5640 1 -- 172.21.15.16:0/2281304715 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f606404dbe0 con 0x7f606414fbd0 2023-05-22T15:28:35.084 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:35.071+0000 7f6055ffb640 1 -- 172.21.15.16:0/2281304715 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 9) v1 ==== 84406+0+0 (secure 0 0 0) 0x7f60600159f0 con 0x7f606414fbd0 2023-05-22T15:28:35.084 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:35.071+0000 7f6055ffb640 1 --2- 172.21.15.16:0/2281304715 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f60480655b0 0x7f6048067a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:35.085 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:35.071+0000 7f6055ffb640 1 -- 172.21.15.16:0/2281304715 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 6640+0+0 (secure 0 0 0) 0x7f60600831f0 con 0x7f606414fbd0 2023-05-22T15:28:35.085 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:35.071+0000 7f6069381640 1 --2- 172.21.15.16:0/2281304715 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f60480655b0 0x7f6048067a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:35.086 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:35.071+0000 7f6069381640 1 --2- 172.21.15.16:0/2281304715 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f60480655b0 0x7f6048067a60 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f60580019e0 tx=0x7f6058007040 comp rx=0 tx=0).ready entity=mgr.5823 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:28:35.086 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:35.075+0000 7f6055ffb640 1 -- 172.21.15.16:0/2281304715 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f6060050b10 con 0x7f606414fbd0 2023-05-22T15:28:35.256 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:35.251+0000 7f606b5e5640 1 -- 172.21.15.16:0/2281304715 --> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/snap_test_dir1/dir7", "snap_schedule": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f60640ca7d0 con 0x7f60480655b0 2023-05-22T15:28:35.390 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:35.383+0000 7f6055ffb640 1 -- 172.21.15.16:0/2281304715 <== mgr.5823 v2:172.21.15.16:6832/36921 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7f60640ca7d0 con 0x7f60480655b0 2023-05-22T15:28:35.390 INFO:teuthology.orchestra.run.smithi016.stdout:Schedule set for path /snap_test_dir1/dir7 2023-05-22T15:28:35.393 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:35.387+0000 7f606b5e5640 1 -- 172.21.15.16:0/2281304715 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f60480655b0 msgr2=0x7f6048067a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:35.393 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:35.387+0000 7f606b5e5640 1 --2- 172.21.15.16:0/2281304715 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f60480655b0 0x7f6048067a60 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f60580019e0 tx=0x7f6058007040 comp rx=0 tx=0).stop 2023-05-22T15:28:35.393 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:35.387+0000 7f606b5e5640 1 -- 172.21.15.16:0/2281304715 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f606414fbd0 msgr2=0x7f6064151780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:35.393 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:35.387+0000 7f606b5e5640 1 --2- 172.21.15.16:0/2281304715 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f606414fbd0 0x7f6064151780 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f6060009160 tx=0x7f6060011a60 comp rx=0 tx=0).stop 2023-05-22T15:28:35.393 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:35.387+0000 7f606b5e5640 1 -- 172.21.15.16:0/2281304715 shutdown_connections 2023-05-22T15:28:35.393 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:35.387+0000 7f606b5e5640 1 --2- 172.21.15.16:0/2281304715 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f60480655b0 0x7f6048067a60 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:35.394 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:35.387+0000 7f606b5e5640 1 --2- 172.21.15.16:0/2281304715 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f60641502d0 0x7f60640c4500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:35.394 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:35.387+0000 7f606b5e5640 1 --2- 172.21.15.16:0/2281304715 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f606414fbd0 0x7f6064151780 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:35.394 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:35.387+0000 7f606b5e5640 1 --2- 172.21.15.16:0/2281304715 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6050005180 0x7f606415ce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:35.394 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:35.387+0000 7f606b5e5640 1 -- 172.21.15.16:0/2281304715 >> 172.21.15.16:0/2281304715 conn(0x7f6064069720 msgr2=0x7f6064157250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:28:35.394 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:35.387+0000 7f606b5e5640 1 -- 172.21.15.16:0/2281304715 shutdown_connections 2023-05-22T15:28:35.394 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:35.387+0000 7f606b5e5640 1 -- 172.21.15.16:0/2281304715 wait complete. 2023-05-22T15:28:35.412 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/dir7 2023-05-22T15:28:35.412 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:28:35.413 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/dir8) 2023-05-22T15:28:35.421 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add --fs cephfs /snap_test_dir1/dir8 1M 2023-05-22T15:28:35.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:35.591+0000 7f8d7a069640 1 Processor -- start 2023-05-22T15:28:35.600 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:35.591+0000 7f8d7a069640 1 -- start start 2023-05-22T15:28:35.601 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:35.595+0000 7f8d7a069640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f8d740cb440 0x7f8d740cd410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:35.601 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:35.595+0000 7f8d7a069640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d74070bb0 con 0x7f8d74158950 2023-05-22T15:28:35.601 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:35.595+0000 7f8d7a069640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d74070e90 con 0x7f8d740cc010 2023-05-22T15:28:35.601 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:35.595+0000 7f8d7a069640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d74071170 con 0x7f8d740cb440 2023-05-22T15:28:35.601 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:35.595+0000 7f8d737fe640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f8d74158950 0x7f8d740cb900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:47978/0 (socket says 172.21.15.16:47978) 2023-05-22T15:28:35.601 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:35.595+0000 7f8d737fe640 1 -- 172.21.15.16:0/2235064358 learned_addr learned my addr 172.21.15.16:0/2235064358 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:28:35.602 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:35.595+0000 7f8d73fff640 1 --2- 172.21.15.16:0/2235064358 >> v2:172.21.15.16:3300/0 conn(0x7f8d740cb440 0x7f8d740cd410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:35.602 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:35.595+0000 7f8d727fc640 1 -- 172.21.15.16:0/2235064358 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1080083878 0 0) 0x7f8d74070bb0 con 0x7f8d74158950 2023-05-22T15:28:35.602 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:35.595+0000 7f8d727fc640 1 -- 172.21.15.16:0/2235064358 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d60003530 con 0x7f8d74158950 2023-05-22T15:28:35.602 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:35.595+0000 7f8d73fff640 1 -- 172.21.15.16:0/2235064358 >> v1:172.21.15.62:6789/0 conn(0x7f8d740cc010 legacy=0x7f8d740ccd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:35.603 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:35.595+0000 7f8d73fff640 1 -- 172.21.15.16:0/2235064358 >> v1:172.21.15.16:6789/0 conn(0x7f8d74158950 legacy=0x7f8d740cb900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:35.603 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:35.595+0000 7f8d73fff640 1 -- 172.21.15.16:0/2235064358 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d74071450 con 0x7f8d740cb440 2023-05-22T15:28:35.603 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:35.595+0000 7f8d73fff640 1 --2- 172.21.15.16:0/2235064358 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8d740cb440 0x7f8d740cd410 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f8d640097c0 tx=0x7f8d6401c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45b81bae5d635868 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:28:35.604 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:35.595+0000 7f8d727fc640 1 -- 172.21.15.16:0/2235064358 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d640ad660 con 0x7f8d740cb440 2023-05-22T15:28:35.604 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:35.595+0000 7f8d727fc640 1 -- 172.21.15.16:0/2235064358 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d640ad7c0 con 0x7f8d740cb440 2023-05-22T15:28:35.604 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:35.595+0000 7f8d727fc640 1 -- 172.21.15.16:0/2235064358 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d640adb50 con 0x7f8d740cb440 2023-05-22T15:28:35.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:35.599+0000 7f8d7a069640 1 -- 172.21.15.16:0/2235064358 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8d740cb440 msgr2=0x7f8d740cd410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:35.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:35.599+0000 7f8d7a069640 1 --2- 172.21.15.16:0/2235064358 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8d740cb440 0x7f8d740cd410 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f8d640097c0 tx=0x7f8d6401c450 comp rx=0 tx=0).stop 2023-05-22T15:28:35.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:35.599+0000 7f8d7a069640 1 -- 172.21.15.16:0/2235064358 shutdown_connections 2023-05-22T15:28:35.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:35.599+0000 7f8d7a069640 1 --2- 172.21.15.16:0/2235064358 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8d740cb440 0x7f8d740cd410 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:35.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:35.599+0000 7f8d7a069640 1 -- 172.21.15.16:0/2235064358 >> 172.21.15.16:0/2235064358 conn(0x7f8d740b0cc0 msgr2=0x7f8d740b10c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:28:35.606 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:35.599+0000 7f8d7a069640 1 -- 172.21.15.16:0/2235064358 shutdown_connections 2023-05-22T15:28:35.606 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:35.599+0000 7f8d7a069640 1 -- 172.21.15.16:0/2235064358 wait complete. 2023-05-22T15:28:35.606 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:35.599+0000 7f8d7a069640 1 Processor -- start 2023-05-22T15:28:35.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:35.599+0000 7f8d7a069640 1 -- start start 2023-05-22T15:28:35.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:35.599+0000 7f8d7a069640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8d740cb440 0x7f8d741622b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:35.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:35.599+0000 7f8d7a069640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8d740cc010 0x7f8d74163aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:35.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:35.599+0000 7f8d7a069640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8d74158950 0x7f8d740ccce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:35.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:35.599+0000 7f8d7a069640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d741f81e0 con 0x7f8d740cc010 2023-05-22T15:28:35.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:35.599+0000 7f8d7a069640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d741f78f0 con 0x7f8d74158950 2023-05-22T15:28:35.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:35.599+0000 7f8d7a069640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d741f7b90 con 0x7f8d740cb440 2023-05-22T15:28:35.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:35.599+0000 7f8d72ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8d740cc010 0x7f8d74163aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:35.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:35.599+0000 7f8d72ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8d740cc010 0x7f8d74163aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42492/0 (socket says 172.21.15.16:42492) 2023-05-22T15:28:35.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:35.599+0000 7f8d72ffd640 1 -- 172.21.15.16:0/1711913563 learned_addr learned my addr 172.21.15.16:0/1711913563 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:28:35.610 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:35.599+0000 7f8d73fff640 1 --2- 172.21.15.16:0/1711913563 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8d74158950 0x7f8d740ccce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:35.610 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:35.599+0000 7f8d737fe640 1 --2- 172.21.15.16:0/1711913563 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8d740cb440 0x7f8d741622b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:35.611 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:35.599+0000 7f8d72ffd640 1 -- 172.21.15.16:0/1711913563 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8d740cb440 msgr2=0x7f8d741622b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:35.611 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:35.599+0000 7f8d72ffd640 1 --2- 172.21.15.16:0/1711913563 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8d740cb440 0x7f8d741622b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:35.611 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:35.599+0000 7f8d72ffd640 1 -- 172.21.15.16:0/1711913563 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8d74158950 msgr2=0x7f8d740ccce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:35.611 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:35.599+0000 7f8d72ffd640 1 --2- 172.21.15.16:0/1711913563 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8d74158950 0x7f8d740ccce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:35.612 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:35.599+0000 7f8d72ffd640 1 -- 172.21.15.16:0/1711913563 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d64009420 con 0x7f8d740cc010 2023-05-22T15:28:35.612 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:35.599+0000 7f8d72ffd640 1 --2- 172.21.15.16:0/1711913563 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8d740cc010 0x7f8d74163aa0 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f8d6c00c7c0 tx=0x7f8d6c00cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:28:35.612 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:35.599+0000 7f8d6bfff640 1 -- 172.21.15.16:0/1711913563 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d6c003b80 con 0x7f8d740cc010 2023-05-22T15:28:35.612 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:35.599+0000 7f8d6bfff640 1 -- 172.21.15.16:0/1711913563 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d6c003ce0 con 0x7f8d740cc010 2023-05-22T15:28:35.612 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:35.599+0000 7f8d7a069640 1 -- 172.21.15.16:0/1711913563 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d74070980 con 0x7f8d740cc010 2023-05-22T15:28:35.612 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:35.599+0000 7f8d6bfff640 1 -- 172.21.15.16:0/1711913563 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d6c00aa90 con 0x7f8d740cc010 2023-05-22T15:28:35.613 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:35.599+0000 7f8d7a069640 1 -- 172.21.15.16:0/1711913563 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d740707f0 con 0x7f8d740cc010 2023-05-22T15:28:35.613 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:35.603+0000 7f8d7a069640 1 -- 172.21.15.16:0/1711913563 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d74166890 con 0x7f8d740cc010 2023-05-22T15:28:35.613 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:35.603+0000 7f8d6bfff640 1 -- 172.21.15.16:0/1711913563 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 9) v1 ==== 84406+0+0 (secure 0 0 0) 0x7f8d6c00abf0 con 0x7f8d740cc010 2023-05-22T15:28:35.613 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:35.603+0000 7f8d6bfff640 1 --2- 172.21.15.16:0/1711913563 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f8d540655b0 0x7f8d54067a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:35.613 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:35.603+0000 7f8d6bfff640 1 -- 172.21.15.16:0/1711913563 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 6640+0+0 (secure 0 0 0) 0x7f8d6c082cb0 con 0x7f8d740cc010 2023-05-22T15:28:35.613 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:35.603+0000 7f8d737fe640 1 --2- 172.21.15.16:0/1711913563 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f8d540655b0 0x7f8d54067a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:35.613 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:35.603+0000 7f8d737fe640 1 --2- 172.21.15.16:0/1711913563 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f8d540655b0 0x7f8d54067a60 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f8d5c0011f0 tx=0x7f8d5c009040 comp rx=0 tx=0).ready entity=mgr.5823 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:28:35.614 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:35.607+0000 7f8d6bfff640 1 -- 172.21.15.16:0/1711913563 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f8d6c050470 con 0x7f8d740cc010 2023-05-22T15:28:35.785 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:35.779+0000 7f8d7a069640 1 -- 172.21.15.16:0/1711913563 --> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/snap_test_dir1/dir8", "snap_schedule": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f8d741f8040 con 0x7f8d540655b0 2023-05-22T15:28:35.919 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:35.911+0000 7f8d6bfff640 1 -- 172.21.15.16:0/1711913563 <== mgr.5823 v2:172.21.15.16:6832/36921 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7f8d5c001cb0 con 0x7f8d540655b0 2023-05-22T15:28:35.919 INFO:teuthology.orchestra.run.smithi016.stdout:Schedule set for path /snap_test_dir1/dir8 2023-05-22T15:28:35.921 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:35.915+0000 7f8d7a069640 1 -- 172.21.15.16:0/1711913563 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f8d540655b0 msgr2=0x7f8d54067a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:35.922 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:35.915+0000 7f8d7a069640 1 --2- 172.21.15.16:0/1711913563 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f8d540655b0 0x7f8d54067a60 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f8d5c0011f0 tx=0x7f8d5c009040 comp rx=0 tx=0).stop 2023-05-22T15:28:35.922 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:35.915+0000 7f8d7a069640 1 -- 172.21.15.16:0/1711913563 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8d740cc010 msgr2=0x7f8d74163aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:35.922 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:35.915+0000 7f8d7a069640 1 --2- 172.21.15.16:0/1711913563 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8d740cc010 0x7f8d74163aa0 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f8d6c00c7c0 tx=0x7f8d6c00cc80 comp rx=0 tx=0).stop 2023-05-22T15:28:35.922 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:35.915+0000 7f8d7a069640 1 -- 172.21.15.16:0/1711913563 shutdown_connections 2023-05-22T15:28:35.922 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:35.915+0000 7f8d7a069640 1 --2- 172.21.15.16:0/1711913563 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8d74158950 0x7f8d740ccce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:35.922 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:35.915+0000 7f8d7a069640 1 --2- 172.21.15.16:0/1711913563 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8d740cc010 0x7f8d74163aa0 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:35.923 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:35.915+0000 7f8d7a069640 1 --2- 172.21.15.16:0/1711913563 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f8d540655b0 0x7f8d54067a60 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:35.923 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:35.915+0000 7f8d7a069640 1 --2- 172.21.15.16:0/1711913563 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8d740cb440 0x7f8d741622b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:35.923 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:35.915+0000 7f8d7a069640 1 -- 172.21.15.16:0/1711913563 >> 172.21.15.16:0/1711913563 conn(0x7f8d740b0cc0 msgr2=0x7f8d740cf980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:28:35.923 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:35.915+0000 7f8d7a069640 1 -- 172.21.15.16:0/1711913563 shutdown_connections 2023-05-22T15:28:35.923 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:35.915+0000 7f8d7a069640 1 -- 172.21.15.16:0/1711913563 wait complete. 2023-05-22T15:28:35.941 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/dir8 2023-05-22T15:28:35.942 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:28:35.943 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/dir9) 2023-05-22T15:28:35.951 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add --fs cephfs /snap_test_dir1/dir9 1M 2023-05-22T15:28:36.130 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:36.123+0000 7f0ea41bb640 1 Processor -- start 2023-05-22T15:28:36.131 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:36.123+0000 7f0ea41bb640 1 -- start start 2023-05-22T15:28:36.131 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:36.123+0000 7f0ea41bb640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0e9c0ce530 0x7f0e9c15c540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:36.132 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:36.123+0000 7f0ea41bb640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f0e9c0cbe80 0x7f0e9c0cc680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:36.132 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:36.123+0000 7f0ea41bb640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e9c0c3930 con 0x7f0e9c0ce900 2023-05-22T15:28:36.132 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:36.123+0000 7f0ea41bb640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e9c158f00 con 0x7f0e9c0cbe80 2023-05-22T15:28:36.132 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:36.123+0000 7f0ea41bb640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e9c159d90 con 0x7f0e9c0ce530 2023-05-22T15:28:36.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:36.123+0000 7f0ea1756640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f0e9c0ce900 0x7f0e9c159680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:47992/0 (socket says 172.21.15.16:47992) 2023-05-22T15:28:36.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:36.127+0000 7f0ea1756640 1 -- 172.21.15.16:0/2619127200 learned_addr learned my addr 172.21.15.16:0/2619127200 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:28:36.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:36.127+0000 7f0ea1f57640 1 --2- 172.21.15.16:0/2619127200 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0e9c0ce530 0x7f0e9c15c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:36.134 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:36.127+0000 7f0ea2758640 1 --2- 172.21.15.16:0/2619127200 >> v2:172.21.15.62:3300/0 conn(0x7f0e9c0cbe80 0x7f0e9c0cc680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:36.134 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:36.127+0000 7f0ea0f55640 1 -- 172.21.15.16:0/2619127200 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2947866242 0 0) 0x7f0e8c001120 con 0x7f0e9c0ce900 2023-05-22T15:28:36.134 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:36.127+0000 7f0ea0f55640 1 -- 172.21.15.16:0/2619127200 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0e88005740 con 0x7f0e9c0ce900 2023-05-22T15:28:36.134 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:36.127+0000 7f0ea1f57640 1 -- 172.21.15.16:0/2619127200 >> v2:172.21.15.62:3300/0 conn(0x7f0e9c0cbe80 msgr2=0x7f0e9c0cc680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:28:36.134 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:36.127+0000 7f0ea1f57640 1 --2- 172.21.15.16:0/2619127200 >> v2:172.21.15.62:3300/0 conn(0x7f0e9c0cbe80 0x7f0e9c0cc680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:36.134 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:36.127+0000 7f0ea1f57640 1 -- 172.21.15.16:0/2619127200 >> v1:172.21.15.16:6789/0 conn(0x7f0e9c0ce900 legacy=0x7f0e9c159680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:36.135 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:36.127+0000 7f0ea1f57640 1 -- 172.21.15.16:0/2619127200 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e9c15cc40 con 0x7f0e9c0ce530 2023-05-22T15:28:36.135 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:36.127+0000 7f0ea2758640 1 --2- 172.21.15.16:0/2619127200 >> v2:172.21.15.62:3300/0 conn(0x7f0e9c0cbe80 0x7f0e9c0cc680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-05-22T15:28:36.136 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:36.127+0000 7f0ea1f57640 1 --2- 172.21.15.16:0/2619127200 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0e9c0ce530 0x7f0e9c15c540 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f0e980097c0 tx=0x7f0e9801c480 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ca0dfa198f08b79e server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:28:36.136 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:36.127+0000 7f0ea0f55640 1 -- 172.21.15.16:0/2619127200 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e980ab600 con 0x7f0e9c0ce530 2023-05-22T15:28:36.136 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:36.127+0000 7f0ea0f55640 1 -- 172.21.15.16:0/2619127200 <== mon.2 v2:172.21.15.62:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e980a7050 con 0x7f0e9c0ce530 2023-05-22T15:28:36.137 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:36.127+0000 7f0ea0f55640 1 -- 172.21.15.16:0/2619127200 <== mon.2 v2:172.21.15.62:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e980ab960 con 0x7f0e9c0ce530 2023-05-22T15:28:36.137 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:36.127+0000 7f0ea41bb640 1 -- 172.21.15.16:0/2619127200 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0e9c0ce530 msgr2=0x7f0e9c15c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:36.137 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:36.127+0000 7f0ea41bb640 1 --2- 172.21.15.16:0/2619127200 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0e9c0ce530 0x7f0e9c15c540 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f0e980097c0 tx=0x7f0e9801c480 comp rx=0 tx=0).stop 2023-05-22T15:28:36.137 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:36.127+0000 7f0ea41bb640 1 -- 172.21.15.16:0/2619127200 shutdown_connections 2023-05-22T15:28:36.137 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:36.127+0000 7f0ea41bb640 1 --2- 172.21.15.16:0/2619127200 >> v2:172.21.15.62:3300/0 conn(0x7f0e9c0cbe80 0x7f0e9c0cc680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:36.137 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:36.127+0000 7f0ea41bb640 1 --2- 172.21.15.16:0/2619127200 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0e9c0ce530 0x7f0e9c15c540 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:36.137 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:36.127+0000 7f0ea41bb640 1 -- 172.21.15.16:0/2619127200 >> 172.21.15.16:0/2619127200 conn(0x7f0e9c069720 msgr2=0x7f0e9c069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:28:36.138 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:36.127+0000 7f0ea41bb640 1 -- 172.21.15.16:0/2619127200 shutdown_connections 2023-05-22T15:28:36.138 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:36.127+0000 7f0ea41bb640 1 -- 172.21.15.16:0/2619127200 wait complete. 2023-05-22T15:28:36.139 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:36.131+0000 7f0ea41bb640 1 Processor -- start 2023-05-22T15:28:36.139 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:36.131+0000 7f0ea41bb640 1 -- start start 2023-05-22T15:28:36.139 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:36.131+0000 7f0ea41bb640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0e9c0cbe80 0x7f0e9c0b0350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:36.140 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:36.131+0000 7f0ea41bb640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0e9c0ce530 0x7f0e9c0c3b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:36.140 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:36.131+0000 7f0ea41bb640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e9c0ce900 0x7f0e9c0c4090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:36.140 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:36.131+0000 7f0ea41bb640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e9c1537b0 con 0x7f0e9c0ce900 2023-05-22T15:28:36.140 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:36.131+0000 7f0ea41bb640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e9c153a90 con 0x7f0e9c0cbe80 2023-05-22T15:28:36.140 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:36.131+0000 7f0ea41bb640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e9c153d70 con 0x7f0e9c0ce530 2023-05-22T15:28:36.140 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:36.131+0000 7f0ea2758640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e9c0ce900 0x7f0e9c0c4090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:36.140 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:36.131+0000 7f0ea2758640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e9c0ce900 0x7f0e9c0c4090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42504/0 (socket says 172.21.15.16:42504) 2023-05-22T15:28:36.141 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:36.131+0000 7f0ea2758640 1 -- 172.21.15.16:0/379607045 learned_addr learned my addr 172.21.15.16:0/379607045 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:28:36.141 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:36.131+0000 7f0ea1756640 1 --2- 172.21.15.16:0/379607045 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0e9c0ce530 0x7f0e9c0c3b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:36.141 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:36.131+0000 7f0ea1f57640 1 --2- 172.21.15.16:0/379607045 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0e9c0cbe80 0x7f0e9c0b0350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:36.141 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:36.131+0000 7f0ea2758640 1 -- 172.21.15.16:0/379607045 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0e9c0ce530 msgr2=0x7f0e9c0c3b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:36.141 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:36.131+0000 7f0ea2758640 1 --2- 172.21.15.16:0/379607045 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0e9c0ce530 0x7f0e9c0c3b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:36.142 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:36.131+0000 7f0ea2758640 1 -- 172.21.15.16:0/379607045 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0e9c0cbe80 msgr2=0x7f0e9c0b0350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:36.142 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:36.131+0000 7f0ea2758640 1 --2- 172.21.15.16:0/379607045 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0e9c0cbe80 0x7f0e9c0b0350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:36.142 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:36.131+0000 7f0ea2758640 1 -- 172.21.15.16:0/379607045 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e94009a70 con 0x7f0e9c0ce900 2023-05-22T15:28:36.142 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:36.131+0000 7f0ea2758640 1 --2- 172.21.15.16:0/379607045 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e9c0ce900 0x7f0e9c0c4090 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f0e9400efc0 tx=0x7f0e94009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:28:36.142 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:36.131+0000 7f0e927fc640 1 -- 172.21.15.16:0/379607045 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e94009d70 con 0x7f0e9c0ce900 2023-05-22T15:28:36.142 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:36.131+0000 7f0e927fc640 1 -- 172.21.15.16:0/379607045 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e9400f040 con 0x7f0e9c0ce900 2023-05-22T15:28:36.142 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:36.131+0000 7f0ea41bb640 1 -- 172.21.15.16:0/379607045 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e98009420 con 0x7f0e9c0ce900 2023-05-22T15:28:36.142 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:36.131+0000 7f0e927fc640 1 -- 172.21.15.16:0/379607045 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e940039a0 con 0x7f0e9c0ce900 2023-05-22T15:28:36.143 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:36.131+0000 7f0ea41bb640 1 -- 172.21.15.16:0/379607045 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e9c159160 con 0x7f0e9c0ce900 2023-05-22T15:28:36.143 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:36.131+0000 7f0ea41bb640 1 -- 172.21.15.16:0/379607045 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e9c0ccfd0 con 0x7f0e9c0ce900 2023-05-22T15:28:36.144 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:36.135+0000 7f0e927fc640 1 -- 172.21.15.16:0/379607045 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 9) v1 ==== 84406+0+0 (secure 0 0 0) 0x7f0e940144a0 con 0x7f0e9c0ce900 2023-05-22T15:28:36.144 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:36.135+0000 7f0e927fc640 1 --2- 172.21.15.16:0/379607045 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f0e80065580 0x7f0e80067a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:36.144 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:36.135+0000 7f0e927fc640 1 -- 172.21.15.16:0/379607045 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 6640+0+0 (secure 0 0 0) 0x7f0e94083c10 con 0x7f0e9c0ce900 2023-05-22T15:28:36.144 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:36.135+0000 7f0ea1f57640 1 --2- 172.21.15.16:0/379607045 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f0e80065580 0x7f0e80067a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:36.144 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:36.135+0000 7f0ea1f57640 1 --2- 172.21.15.16:0/379607045 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f0e80065580 0x7f0e80067a30 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f0e980ad000 tx=0x7f0e980a6000 comp rx=0 tx=0).ready entity=mgr.5823 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:28:36.145 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:36.139+0000 7f0e927fc640 1 -- 172.21.15.16:0/379607045 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f0e94051260 con 0x7f0e9c0ce900 2023-05-22T15:28:36.318 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:36.311+0000 7f0ea41bb640 1 -- 172.21.15.16:0/379607045 --> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/snap_test_dir1/dir9", "snap_schedule": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f0e9c14f2f0 con 0x7f0e80065580 2023-05-22T15:28:36.444 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:36.439+0000 7f0e927fc640 1 -- 172.21.15.16:0/379607045 <== mgr.5823 v2:172.21.15.16:6832/36921 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7f0e9c14f2f0 con 0x7f0e80065580 2023-05-22T15:28:36.445 INFO:teuthology.orchestra.run.smithi016.stdout:Schedule set for path /snap_test_dir1/dir9 2023-05-22T15:28:36.449 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:36.443+0000 7f0ea41bb640 1 -- 172.21.15.16:0/379607045 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f0e80065580 msgr2=0x7f0e80067a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:36.449 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:36.443+0000 7f0ea41bb640 1 --2- 172.21.15.16:0/379607045 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f0e80065580 0x7f0e80067a30 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f0e980ad000 tx=0x7f0e980a6000 comp rx=0 tx=0).stop 2023-05-22T15:28:36.449 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:36.443+0000 7f0ea41bb640 1 -- 172.21.15.16:0/379607045 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e9c0ce900 msgr2=0x7f0e9c0c4090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:36.449 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:36.443+0000 7f0ea41bb640 1 --2- 172.21.15.16:0/379607045 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e9c0ce900 0x7f0e9c0c4090 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f0e9400efc0 tx=0x7f0e94009f90 comp rx=0 tx=0).stop 2023-05-22T15:28:36.449 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:36.443+0000 7f0ea41bb640 1 -- 172.21.15.16:0/379607045 shutdown_connections 2023-05-22T15:28:36.449 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:36.443+0000 7f0ea41bb640 1 --2- 172.21.15.16:0/379607045 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e9c0ce900 0x7f0e9c0c4090 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:36.450 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:36.443+0000 7f0ea41bb640 1 --2- 172.21.15.16:0/379607045 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f0e80065580 0x7f0e80067a30 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:36.450 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:36.443+0000 7f0ea41bb640 1 --2- 172.21.15.16:0/379607045 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0e9c0ce530 0x7f0e9c0c3b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:36.450 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:36.443+0000 7f0ea41bb640 1 --2- 172.21.15.16:0/379607045 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0e9c0cbe80 0x7f0e9c0b0350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:36.450 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:36.443+0000 7f0ea41bb640 1 -- 172.21.15.16:0/379607045 >> 172.21.15.16:0/379607045 conn(0x7f0e9c069720 msgr2=0x7f0e9c070860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:28:36.450 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:36.443+0000 7f0ea41bb640 1 -- 172.21.15.16:0/379607045 shutdown_connections 2023-05-22T15:28:36.450 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:36.443+0000 7f0ea41bb640 1 -- 172.21.15.16:0/379607045 wait complete. 2023-05-22T15:28:36.470 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/dir9 2023-05-22T15:28:36.573 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate --fs cephfs /snap_test_dir1/dir0 1M 2023-05-22T15:28:36.715 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:36.707+0000 7fe990835640 1 Processor -- start 2023-05-22T15:28:36.715 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:36.707+0000 7fe990835640 1 -- start start 2023-05-22T15:28:36.716 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:36.711+0000 7fe990835640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe98c15d650 0x7fe98c15dad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:36.716 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:36.711+0000 7fe990835640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe98c0cc960 con 0x7fe98c0cc4e0 2023-05-22T15:28:36.716 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:36.711+0000 7fe990835640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe98c0ccbb0 con 0x7fe98c070c10 2023-05-22T15:28:36.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:36.711+0000 7fe990835640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe98c15e010 con 0x7fe98c15d650 2023-05-22T15:28:36.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:36.711+0000 7fe989d9b640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fe98c0cc4e0 0x7fe98c06bfa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:47998/0 (socket says 172.21.15.16:47998) 2023-05-22T15:28:36.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:36.711+0000 7fe989d9b640 1 -- 172.21.15.16:0/1911893116 learned_addr learned my addr 172.21.15.16:0/1911893116 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:28:36.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:36.711+0000 7fe98ad9d640 1 --2- 172.21.15.16:0/1911893116 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe98c15d650 0x7fe98c15dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:36.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:36.711+0000 7fe98959a640 1 -- 172.21.15.16:0/1911893116 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1030688073 0 0) 0x7fe98c0cc960 con 0x7fe98c0cc4e0 2023-05-22T15:28:36.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:36.711+0000 7fe98959a640 1 -- 172.21.15.16:0/1911893116 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe978003530 con 0x7fe98c0cc4e0 2023-05-22T15:28:36.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:36.711+0000 7fe98959a640 1 -- 172.21.15.16:0/1911893116 <== mon.1 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1850247281 0 0) 0x7fe98c0ccbb0 con 0x7fe98c070c10 2023-05-22T15:28:36.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:36.711+0000 7fe98ad9d640 1 -- 172.21.15.16:0/1911893116 >> v1:172.21.15.62:6789/0 conn(0x7fe98c070c10 legacy=0x7fe98c0c9e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:36.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:36.711+0000 7fe98ad9d640 1 -- 172.21.15.16:0/1911893116 >> v1:172.21.15.16:6789/0 conn(0x7fe98c0cc4e0 legacy=0x7fe98c06bfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:36.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:36.711+0000 7fe98ad9d640 1 -- 172.21.15.16:0/1911893116 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe98c0cd7c0 con 0x7fe98c15d650 2023-05-22T15:28:36.719 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:36.711+0000 7fe98ad9d640 1 --2- 172.21.15.16:0/1911893116 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe98c15d650 0x7fe98c15dad0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fe984009dc0 tx=0x7fe98401c940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2567d83473a80658 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:28:36.719 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:36.711+0000 7fe98959a640 1 -- 172.21.15.16:0/1911893116 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9840a5a00 con 0x7fe98c15d650 2023-05-22T15:28:36.720 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:36.711+0000 7fe98959a640 1 -- 172.21.15.16:0/1911893116 <== mon.2 v2:172.21.15.62:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9840a5b60 con 0x7fe98c15d650 2023-05-22T15:28:36.720 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:36.711+0000 7fe98959a640 1 -- 172.21.15.16:0/1911893116 <== mon.2 v2:172.21.15.62:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9840ad440 con 0x7fe98c15d650 2023-05-22T15:28:36.721 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:36.715+0000 7fe990835640 1 -- 172.21.15.16:0/1911893116 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe98c15d650 msgr2=0x7fe98c15dad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:36.721 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:36.715+0000 7fe990835640 1 --2- 172.21.15.16:0/1911893116 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe98c15d650 0x7fe98c15dad0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fe984009dc0 tx=0x7fe98401c940 comp rx=0 tx=0).stop 2023-05-22T15:28:36.721 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:36.715+0000 7fe990835640 1 -- 172.21.15.16:0/1911893116 shutdown_connections 2023-05-22T15:28:36.721 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:36.715+0000 7fe990835640 1 --2- 172.21.15.16:0/1911893116 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe98c15d650 0x7fe98c15dad0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:36.721 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:36.715+0000 7fe990835640 1 -- 172.21.15.16:0/1911893116 >> 172.21.15.16:0/1911893116 conn(0x7fe98c069720 msgr2=0x7fe98c069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:28:36.721 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:36.715+0000 7fe990835640 1 -- 172.21.15.16:0/1911893116 shutdown_connections 2023-05-22T15:28:36.722 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:36.715+0000 7fe990835640 1 -- 172.21.15.16:0/1911893116 wait complete. 2023-05-22T15:28:36.722 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:36.715+0000 7fe990835640 1 Processor -- start 2023-05-22T15:28:36.722 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:36.715+0000 7fe990835640 1 -- start start 2023-05-22T15:28:36.723 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:36.715+0000 7fe990835640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe98c070c10 0x7fe98c13b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:36.723 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:36.715+0000 7fe990835640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe98c0cc4e0 0x7fe98c15b200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:36.723 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:36.715+0000 7fe990835640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe98c15d650 0x7fe98c14a0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:36.723 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:36.715+0000 7fe990835640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe98c15d4d0 con 0x7fe98c15d650 2023-05-22T15:28:36.724 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:36.715+0000 7fe990835640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe98c13fac0 con 0x7fe98c070c10 2023-05-22T15:28:36.724 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:36.715+0000 7fe990835640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe98c13fda0 con 0x7fe98c0cc4e0 2023-05-22T15:28:36.724 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:36.715+0000 7fe98ad9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe98c15d650 0x7fe98c14a0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:36.724 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:36.715+0000 7fe98ad9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe98c15d650 0x7fe98c14a0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42508/0 (socket says 172.21.15.16:42508) 2023-05-22T15:28:36.724 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:36.715+0000 7fe98ad9d640 1 -- 172.21.15.16:0/824664008 learned_addr learned my addr 172.21.15.16:0/824664008 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:28:36.726 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:36.715+0000 7fe989d9b640 1 --2- 172.21.15.16:0/824664008 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe98c0cc4e0 0x7fe98c15b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:36.726 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:36.715+0000 7fe98a59c640 1 --2- 172.21.15.16:0/824664008 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe98c070c10 0x7fe98c13b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:36.726 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:36.715+0000 7fe98ad9d640 1 -- 172.21.15.16:0/824664008 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe98c0cc4e0 msgr2=0x7fe98c15b200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:36.726 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:36.715+0000 7fe98ad9d640 1 --2- 172.21.15.16:0/824664008 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe98c0cc4e0 0x7fe98c15b200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:36.726 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:36.715+0000 7fe98ad9d640 1 -- 172.21.15.16:0/824664008 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe98c070c10 msgr2=0x7fe98c13b410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:36.727 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:36.715+0000 7fe98ad9d640 1 --2- 172.21.15.16:0/824664008 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe98c070c10 0x7fe98c13b410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:36.727 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:36.715+0000 7fe98ad9d640 1 -- 172.21.15.16:0/824664008 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe984009a70 con 0x7fe98c15d650 2023-05-22T15:28:36.727 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:36.715+0000 7fe98ad9d640 1 --2- 172.21.15.16:0/824664008 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe98c15d650 0x7fe98c14a0d0 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7fe98401a570 tx=0x7fe9840196e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:28:36.727 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:36.719+0000 7fe982ffd640 1 -- 172.21.15.16:0/824664008 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe984004240 con 0x7fe98c15d650 2023-05-22T15:28:36.727 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:36.719+0000 7fe982ffd640 1 -- 172.21.15.16:0/824664008 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9840043a0 con 0x7fe98c15d650 2023-05-22T15:28:36.727 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:36.719+0000 7fe990835640 1 -- 172.21.15.16:0/824664008 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe98c0cce90 con 0x7fe98c15d650 2023-05-22T15:28:36.727 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:36.719+0000 7fe982ffd640 1 -- 172.21.15.16:0/824664008 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe984004710 con 0x7fe98c15d650 2023-05-22T15:28:36.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:36.719+0000 7fe990835640 1 -- 172.21.15.16:0/824664008 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe98c070fe0 con 0x7fe98c15d650 2023-05-22T15:28:36.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:36.719+0000 7fe990835640 1 -- 172.21.15.16:0/824664008 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe98c1402c0 con 0x7fe98c15d650 2023-05-22T15:28:36.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:36.719+0000 7fe982ffd640 1 -- 172.21.15.16:0/824664008 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 9) v1 ==== 84406+0+0 (secure 0 0 0) 0x7fe9840b4030 con 0x7fe98c15d650 2023-05-22T15:28:36.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:36.719+0000 7fe982ffd640 1 --2- 172.21.15.16:0/824664008 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fe96c0655b0 0x7fe96c067a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:36.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:36.719+0000 7fe982ffd640 1 -- 172.21.15.16:0/824664008 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 6640+0+0 (secure 0 0 0) 0x7fe98411dc50 con 0x7fe98c15d650 2023-05-22T15:28:36.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:36.719+0000 7fe98a59c640 1 --2- 172.21.15.16:0/824664008 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fe96c0655b0 0x7fe96c067a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:36.729 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:36.719+0000 7fe98a59c640 1 --2- 172.21.15.16:0/824664008 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fe96c0655b0 0x7fe96c067a60 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fe974000f70 tx=0x7fe974009040 comp rx=0 tx=0).ready entity=mgr.5823 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:28:36.730 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:36.723+0000 7fe982ffd640 1 -- 172.21.15.16:0/824664008 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fe9840eb4a0 con 0x7fe98c15d650 2023-05-22T15:28:36.906 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:36.899+0000 7fe990835640 1 -- 172.21.15.16:0/824664008 --> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "fs": "cephfs", "path": "/snap_test_dir1/dir0", "repeat": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7fe98c13ff10 con 0x7fe96c0655b0 2023-05-22T15:28:36.998 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:36.991+0000 7fe982ffd640 1 -- 172.21.15.16:0/824664008 <== mgr.5823 v2:172.21.15.16:6832/36921 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7fe98c13ff10 con 0x7fe96c0655b0 2023-05-22T15:28:36.998 INFO:teuthology.orchestra.run.smithi016.stdout:Schedule activated for path /snap_test_dir1/dir0 2023-05-22T15:28:37.002 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:36.995+0000 7fe990835640 1 -- 172.21.15.16:0/824664008 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fe96c0655b0 msgr2=0x7fe96c067a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:37.002 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:36.995+0000 7fe990835640 1 --2- 172.21.15.16:0/824664008 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fe96c0655b0 0x7fe96c067a60 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fe974000f70 tx=0x7fe974009040 comp rx=0 tx=0).stop 2023-05-22T15:28:37.002 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:36.995+0000 7fe990835640 1 -- 172.21.15.16:0/824664008 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe98c15d650 msgr2=0x7fe98c14a0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:37.003 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:36.995+0000 7fe990835640 1 --2- 172.21.15.16:0/824664008 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe98c15d650 0x7fe98c14a0d0 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7fe98401a570 tx=0x7fe9840196e0 comp rx=0 tx=0).stop 2023-05-22T15:28:37.003 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:36.995+0000 7fe990835640 1 -- 172.21.15.16:0/824664008 shutdown_connections 2023-05-22T15:28:37.003 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:36.995+0000 7fe990835640 1 --2- 172.21.15.16:0/824664008 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe98c15d650 0x7fe98c14a0d0 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:37.003 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:36.995+0000 7fe990835640 1 --2- 172.21.15.16:0/824664008 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fe96c0655b0 0x7fe96c067a60 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:37.003 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:36.995+0000 7fe990835640 1 --2- 172.21.15.16:0/824664008 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe98c0cc4e0 0x7fe98c15b200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:37.003 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:36.995+0000 7fe990835640 1 --2- 172.21.15.16:0/824664008 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe98c070c10 0x7fe98c13b410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:37.003 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:36.995+0000 7fe990835640 1 -- 172.21.15.16:0/824664008 >> 172.21.15.16:0/824664008 conn(0x7fe98c069720 msgr2=0x7fe98c14ec30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:28:37.004 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:36.995+0000 7fe990835640 1 -- 172.21.15.16:0/824664008 shutdown_connections 2023-05-22T15:28:37.004 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:36.995+0000 7fe990835640 1 -- 172.21.15.16:0/824664008 wait complete. 2023-05-22T15:28:37.021 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/dir0 2023-05-22T15:28:37.021 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate --fs cephfs /snap_test_dir1/dir1 1M 2023-05-22T15:28:37.147 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:37.139+0000 7f6e2e6f4640 1 Processor -- start 2023-05-22T15:28:37.147 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:37.139+0000 7f6e2e6f4640 1 -- start start 2023-05-22T15:28:37.149 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:37.139+0000 7f6e2e6f4640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6e280cae40 0x7f6e280b17d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:37.149 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:37.139+0000 7f6e2e6f4640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6e280cb340 0x7f6e2806c700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:37.149 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:37.139+0000 7f6e2e6f4640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e280ce350 con 0x7f6e28069720 2023-05-22T15:28:37.150 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:37.139+0000 7f6e2e6f4640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e2815f7e0 con 0x7f6e280cb340 2023-05-22T15:28:37.150 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:37.139+0000 7f6e2e6f4640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e28071090 con 0x7f6e280cae40 2023-05-22T15:28:37.150 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:37.143+0000 7f6e277fe640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6e280cb340 0x7f6e2806c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:37.150 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:37.143+0000 7f6e277fe640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6e280cb340 0x7f6e2806c700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42516/0 (socket says 172.21.15.16:42516) 2023-05-22T15:28:37.151 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:37.143+0000 7f6e277fe640 1 -- 172.21.15.16:0/2586661263 learned_addr learned my addr 172.21.15.16:0/2586661263 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:28:37.151 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:37.143+0000 7f6e2cc91640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f6e28069720 0x7f6e2815fc70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:48002/0 (socket says 172.21.15.16:48002) 2023-05-22T15:28:37.151 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:37.143+0000 7f6e27fff640 1 --2- 172.21.15.16:0/2586661263 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6e280cae40 0x7f6e280b17d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:37.151 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:37.143+0000 7f6e277fe640 1 -- 172.21.15.16:0/2586661263 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6e280cae40 msgr2=0x7f6e280b17d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:28:37.152 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:37.143+0000 7f6e277fe640 1 --2- 172.21.15.16:0/2586661263 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6e280cae40 0x7f6e280b17d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:37.152 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:37.143+0000 7f6e277fe640 1 -- 172.21.15.16:0/2586661263 >> v1:172.21.15.16:6789/0 conn(0x7f6e28069720 legacy=0x7f6e2815fc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:37.152 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:37.143+0000 7f6e277fe640 1 -- 172.21.15.16:0/2586661263 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e280ce580 con 0x7f6e280cb340 2023-05-22T15:28:37.152 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:37.143+0000 7f6e26ffd640 1 -- 172.21.15.16:0/2586661263 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 107376353 0 0) 0x7f6e280ce350 con 0x7f6e28069720 2023-05-22T15:28:37.156 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:37.143+0000 7f6e277fe640 1 --2- 172.21.15.16:0/2586661263 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6e280cb340 0x7f6e2806c700 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f6e1c009870 tx=0x7f6e1c01c540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4510baaf865f12ac server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:28:37.156 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:37.143+0000 7f6e26ffd640 1 -- 172.21.15.16:0/2586661263 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e1c0a7070 con 0x7f6e280cb340 2023-05-22T15:28:37.156 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:37.143+0000 7f6e26ffd640 1 -- 172.21.15.16:0/2586661263 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e1c0ab430 con 0x7f6e280cb340 2023-05-22T15:28:37.156 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:37.143+0000 7f6e26ffd640 1 -- 172.21.15.16:0/2586661263 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e1c0ab7c0 con 0x7f6e280cb340 2023-05-22T15:28:37.156 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:37.143+0000 7f6e2e6f4640 1 -- 172.21.15.16:0/2586661263 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6e280cb340 msgr2=0x7f6e2806c700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:37.156 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:37.143+0000 7f6e2e6f4640 1 --2- 172.21.15.16:0/2586661263 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6e280cb340 0x7f6e2806c700 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f6e1c009870 tx=0x7f6e1c01c540 comp rx=0 tx=0).stop 2023-05-22T15:28:37.157 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:37.143+0000 7f6e2e6f4640 1 -- 172.21.15.16:0/2586661263 shutdown_connections 2023-05-22T15:28:37.157 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:37.143+0000 7f6e2e6f4640 1 --2- 172.21.15.16:0/2586661263 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6e280cb340 0x7f6e2806c700 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:37.157 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:37.143+0000 7f6e2e6f4640 1 --2- 172.21.15.16:0/2586661263 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6e280cae40 0x7f6e280b17d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:37.157 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:37.143+0000 7f6e2e6f4640 1 -- 172.21.15.16:0/2586661263 >> 172.21.15.16:0/2586661263 conn(0x7f6e280b0cc0 msgr2=0x7f6e280b10c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:28:37.157 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:37.143+0000 7f6e2e6f4640 1 -- 172.21.15.16:0/2586661263 shutdown_connections 2023-05-22T15:28:37.158 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:37.143+0000 7f6e2e6f4640 1 -- 172.21.15.16:0/2586661263 wait complete. 2023-05-22T15:28:37.158 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:37.147+0000 7f6e2e6f4640 1 Processor -- start 2023-05-22T15:28:37.158 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:37.147+0000 7f6e2e6f4640 1 -- start start 2023-05-22T15:28:37.158 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:37.147+0000 7f6e2e6f4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6e28069720 0x7f6e280ade20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:37.158 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:37.147+0000 7f6e2e6f4640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6e280cae40 0x7f6e280ab7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:37.158 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:37.147+0000 7f6e2e6f4640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6e280cb340 0x7f6e280abd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:37.159 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:37.147+0000 7f6e2e6f4640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e28141280 con 0x7f6e28069720 2023-05-22T15:28:37.159 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:37.147+0000 7f6e2e6f4640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e28140670 con 0x7f6e280cae40 2023-05-22T15:28:37.159 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:37.147+0000 7f6e2e6f4640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e28140950 con 0x7f6e280cb340 2023-05-22T15:28:37.160 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:37.147+0000 7f6e27fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6e28069720 0x7f6e280ade20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:37.160 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:37.147+0000 7f6e27fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6e28069720 0x7f6e280ade20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42526/0 (socket says 172.21.15.16:42526) 2023-05-22T15:28:37.160 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:37.147+0000 7f6e27fff640 1 -- 172.21.15.16:0/3756570229 learned_addr learned my addr 172.21.15.16:0/3756570229 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:28:37.160 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:37.147+0000 7f6e277fe640 1 --2- 172.21.15.16:0/3756570229 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6e280cae40 0x7f6e280ab7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:37.161 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:37.147+0000 7f6e2cc91640 1 --2- 172.21.15.16:0/3756570229 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6e280cb340 0x7f6e280abd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:37.161 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:37.147+0000 7f6e27fff640 1 -- 172.21.15.16:0/3756570229 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6e280cb340 msgr2=0x7f6e280abd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:37.161 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:37.147+0000 7f6e27fff640 1 --2- 172.21.15.16:0/3756570229 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6e280cb340 0x7f6e280abd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:37.161 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:37.147+0000 7f6e27fff640 1 -- 172.21.15.16:0/3756570229 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6e280cae40 msgr2=0x7f6e280ab7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:37.163 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:37.147+0000 7f6e27fff640 1 --2- 172.21.15.16:0/3756570229 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6e280cae40 0x7f6e280ab7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:37.163 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:37.147+0000 7f6e27fff640 1 -- 172.21.15.16:0/3756570229 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e1c0094d0 con 0x7f6e28069720 2023-05-22T15:28:37.163 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:37.147+0000 7f6e277fe640 1 --2- 172.21.15.16:0/3756570229 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6e280cae40 0x7f6e280ab7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:28:37.164 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:37.147+0000 7f6e27fff640 1 --2- 172.21.15.16:0/3756570229 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6e28069720 0x7f6e280ade20 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f6e1800d800 tx=0x7f6e1800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:28:37.164 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:37.147+0000 7f6e24ff9640 1 -- 172.21.15.16:0/3756570229 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e1800b6c0 con 0x7f6e28069720 2023-05-22T15:28:37.164 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:37.147+0000 7f6e24ff9640 1 -- 172.21.15.16:0/3756570229 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e1800b820 con 0x7f6e28069720 2023-05-22T15:28:37.164 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:37.147+0000 7f6e2e6f4640 1 -- 172.21.15.16:0/3756570229 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e28069f70 con 0x7f6e28069720 2023-05-22T15:28:37.165 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:37.147+0000 7f6e24ff9640 1 -- 172.21.15.16:0/3756570229 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e1800bb90 con 0x7f6e28069720 2023-05-22T15:28:37.165 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:37.147+0000 7f6e2e6f4640 1 -- 172.21.15.16:0/3756570229 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e280cee20 con 0x7f6e28069720 2023-05-22T15:28:37.165 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:37.147+0000 7f6e2e6f4640 1 -- 172.21.15.16:0/3756570229 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e2806cc40 con 0x7f6e28069720 2023-05-22T15:28:37.166 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:37.151+0000 7f6e24ff9640 1 -- 172.21.15.16:0/3756570229 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 9) v1 ==== 84406+0+0 (secure 0 0 0) 0x7f6e1800bcf0 con 0x7f6e28069720 2023-05-22T15:28:37.167 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:37.151+0000 7f6e24ff9640 1 --2- 172.21.15.16:0/3756570229 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f6e14065580 0x7f6e14067a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:37.167 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:37.151+0000 7f6e24ff9640 1 -- 172.21.15.16:0/3756570229 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 6640+0+0 (secure 0 0 0) 0x7f6e18083380 con 0x7f6e28069720 2023-05-22T15:28:37.167 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:37.151+0000 7f6e277fe640 1 --2- 172.21.15.16:0/3756570229 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f6e14065580 0x7f6e14067a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:37.167 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:37.151+0000 7f6e277fe640 1 --2- 172.21.15.16:0/3756570229 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f6e14065580 0x7f6e14067a30 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f6e1c0af000 tx=0x7f6e1c008040 comp rx=0 tx=0).ready entity=mgr.5823 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:28:37.167 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:37.155+0000 7f6e24ff9640 1 -- 172.21.15.16:0/3756570229 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f6e18050b40 con 0x7f6e28069720 2023-05-22T15:28:37.335 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:37.327+0000 7f6e2e6f4640 1 -- 172.21.15.16:0/3756570229 --> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "fs": "cephfs", "path": "/snap_test_dir1/dir1", "repeat": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f6e281413c0 con 0x7f6e14065580 2023-05-22T15:28:37.429 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:37.423+0000 7f6e24ff9640 1 -- 172.21.15.16:0/3756570229 <== mgr.5823 v2:172.21.15.16:6832/36921 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f6e1c0bb0d0 con 0x7f6e14065580 2023-05-22T15:28:37.429 INFO:teuthology.orchestra.run.smithi016.stdout:Schedule activated for path /snap_test_dir1/dir1 2023-05-22T15:28:37.433 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:37.427+0000 7f6e2e6f4640 1 -- 172.21.15.16:0/3756570229 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f6e14065580 msgr2=0x7f6e14067a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:37.434 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:37.427+0000 7f6e2e6f4640 1 --2- 172.21.15.16:0/3756570229 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f6e14065580 0x7f6e14067a30 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f6e1c0af000 tx=0x7f6e1c008040 comp rx=0 tx=0).stop 2023-05-22T15:28:37.434 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:37.427+0000 7f6e2e6f4640 1 -- 172.21.15.16:0/3756570229 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6e28069720 msgr2=0x7f6e280ade20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:37.434 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:37.427+0000 7f6e2e6f4640 1 --2- 172.21.15.16:0/3756570229 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6e28069720 0x7f6e280ade20 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f6e1800d800 tx=0x7f6e1800dcc0 comp rx=0 tx=0).stop 2023-05-22T15:28:37.434 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:37.427+0000 7f6e2e6f4640 1 -- 172.21.15.16:0/3756570229 shutdown_connections 2023-05-22T15:28:37.434 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:37.427+0000 7f6e2e6f4640 1 --2- 172.21.15.16:0/3756570229 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f6e14065580 0x7f6e14067a30 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:37.435 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:37.427+0000 7f6e2e6f4640 1 --2- 172.21.15.16:0/3756570229 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6e280cb340 0x7f6e280abd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:37.435 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:37.427+0000 7f6e2e6f4640 1 --2- 172.21.15.16:0/3756570229 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6e280cae40 0x7f6e280ab7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:37.435 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:37.427+0000 7f6e2e6f4640 1 --2- 172.21.15.16:0/3756570229 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6e28069720 0x7f6e280ade20 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:37.436 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:37.427+0000 7f6e2e6f4640 1 -- 172.21.15.16:0/3756570229 >> 172.21.15.16:0/3756570229 conn(0x7f6e280b0cc0 msgr2=0x7f6e280b1090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:28:37.436 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:37.427+0000 7f6e2e6f4640 1 -- 172.21.15.16:0/3756570229 shutdown_connections 2023-05-22T15:28:37.436 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:37.427+0000 7f6e2e6f4640 1 -- 172.21.15.16:0/3756570229 wait complete. 2023-05-22T15:28:37.453 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/dir1 2023-05-22T15:28:37.453 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate --fs cephfs /snap_test_dir1/dir2 1M 2023-05-22T15:28:37.584 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:37.575+0000 7fadf9b43640 1 Processor -- start 2023-05-22T15:28:37.584 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:37.579+0000 7fadf9b43640 1 -- start start 2023-05-22T15:28:37.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:37.579+0000 7fadf9b43640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fadf40ce530 0x7fadf415c540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:37.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:37.579+0000 7fadf9b43640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fadf40c3930 con 0x7fadf40cbe80 2023-05-22T15:28:37.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:37.579+0000 7fadf9b43640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fadf4158f00 con 0x7fadf40ce900 2023-05-22T15:28:37.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:37.579+0000 7fadf9b43640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fadf4159d90 con 0x7fadf40ce530 2023-05-22T15:28:37.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:37.579+0000 7fadf3fff640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fadf40cbe80 0x7fadf40cc680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:48006/0 (socket says 172.21.15.16:48006) 2023-05-22T15:28:37.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:37.579+0000 7fadf3fff640 1 -- 172.21.15.16:0/2333240063 learned_addr learned my addr 172.21.15.16:0/2333240063 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:28:37.589 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:37.579+0000 7fadf37fe640 1 --2- 172.21.15.16:0/2333240063 >> v2:172.21.15.62:3300/0 conn(0x7fadf40ce530 0x7fadf415c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:37.589 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:37.579+0000 7fadf27fc640 1 -- 172.21.15.16:0/2333240063 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1896827550 0 0) 0x7fadec0014a0 con 0x7fadf40cbe80 2023-05-22T15:28:37.590 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:37.579+0000 7fadf27fc640 1 -- 172.21.15.16:0/2333240063 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faddc005740 con 0x7fadf40cbe80 2023-05-22T15:28:37.590 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:37.579+0000 7fadf27fc640 1 -- 172.21.15.16:0/2333240063 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2832777586 0 0) 0x7faddc005740 con 0x7fadf40cbe80 2023-05-22T15:28:37.590 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:37.579+0000 7fadf27fc640 1 -- 172.21.15.16:0/2333240063 >> v2:172.21.15.62:3300/0 conn(0x7fadf40ce530 msgr2=0x7fadf415c540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:28:37.591 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:37.579+0000 7fadf27fc640 1 --2- 172.21.15.16:0/2333240063 >> v2:172.21.15.62:3300/0 conn(0x7fadf40ce530 0x7fadf415c540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:37.591 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:37.579+0000 7fadf27fc640 1 -- 172.21.15.16:0/2333240063 >> v1:172.21.15.62:6789/0 conn(0x7fadf40ce900 legacy=0x7fadf4159680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:37.591 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:37.579+0000 7fadf27fc640 1 -- 172.21.15.16:0/2333240063 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadf415cc40 con 0x7fadf40cbe80 2023-05-22T15:28:37.591 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:37.579+0000 7fadf27fc640 1 -- 172.21.15.16:0/2333240063 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7fadec001b90 con 0x7fadf40cbe80 2023-05-22T15:28:37.591 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:37.579+0000 7fadf27fc640 1 -- 172.21.15.16:0/2333240063 >> v1:172.21.15.16:6789/0 conn(0x7fadf40cbe80 legacy=0x7fadf40cc680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:37.592 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:37.579+0000 7fadf27fc640 1 --2- 172.21.15.16:0/2333240063 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faddc005b90 0x7faddc005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:37.592 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:37.579+0000 7fadf27fc640 1 --2- 172.21.15.16:0/2333240063 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7faddc0069e0 0x7faddc006e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:37.592 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:37.579+0000 7fadf27fc640 1 --2- 172.21.15.16:0/2333240063 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faddc007390 0x7faddc00dba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:37.592 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:37.579+0000 7fadf27fc640 1 -- 172.21.15.16:0/2333240063 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadf4159d90 con 0x7faddc007390 2023-05-22T15:28:37.593 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:37.579+0000 7fadf27fc640 1 -- 172.21.15.16:0/2333240063 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faddc00e210 con 0x7faddc005b90 2023-05-22T15:28:37.593 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:37.579+0000 7fadf27fc640 1 -- 172.21.15.16:0/2333240063 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faddc00e3d0 con 0x7faddc0069e0 2023-05-22T15:28:37.593 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:37.579+0000 7fadf3fff640 1 --2- 172.21.15.16:0/2333240063 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faddc007390 0x7faddc00dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:37.593 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:37.579+0000 7fadf37fe640 1 --2- 172.21.15.16:0/2333240063 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faddc005b90 0x7faddc005fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:37.594 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:37.579+0000 7fadf2ffd640 1 --2- 172.21.15.16:0/2333240063 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7faddc0069e0 0x7faddc006e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:37.594 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:37.579+0000 7fadf3fff640 1 -- 172.21.15.16:0/2333240063 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7faddc0069e0 msgr2=0x7faddc006e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:28:37.594 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:37.579+0000 7fadf3fff640 1 --2- 172.21.15.16:0/2333240063 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7faddc0069e0 0x7faddc006e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:37.594 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:37.579+0000 7fadf3fff640 1 -- 172.21.15.16:0/2333240063 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faddc005b90 msgr2=0x7faddc005fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:28:37.594 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:37.579+0000 7fadf3fff640 1 --2- 172.21.15.16:0/2333240063 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faddc005b90 0x7faddc005fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:37.595 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:37.579+0000 7fadf3fff640 1 -- 172.21.15.16:0/2333240063 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faddc005740 con 0x7faddc007390 2023-05-22T15:28:37.595 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:37.579+0000 7fadf37fe640 1 --2- 172.21.15.16:0/2333240063 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faddc005b90 0x7faddc005fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T15:28:37.595 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:37.583+0000 7fadf3fff640 1 -- 172.21.15.16:0/2333240063 reap_dead start 2023-05-22T15:28:37.595 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:37.583+0000 7fadf3fff640 1 --2- 172.21.15.16:0/2333240063 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faddc007390 0x7faddc00dba0 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7fadec00a830 tx=0x7fadec01d340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=614fba32ed555628 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:28:37.595 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:37.583+0000 7fadf27fc640 1 -- 172.21.15.16:0/2333240063 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadec0ad630 con 0x7faddc007390 2023-05-22T15:28:37.595 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:37.583+0000 7fadf27fc640 1 -- 172.21.15.16:0/2333240063 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fadec0a8070 con 0x7faddc007390 2023-05-22T15:28:37.595 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:37.583+0000 7fadf27fc640 1 -- 172.21.15.16:0/2333240063 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadec0ad9f0 con 0x7faddc007390 2023-05-22T15:28:37.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:37.583+0000 7fadf9b43640 1 -- 172.21.15.16:0/2333240063 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faddc007390 msgr2=0x7faddc00dba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:37.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:37.583+0000 7fadf9b43640 1 --2- 172.21.15.16:0/2333240063 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faddc007390 0x7faddc00dba0 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7fadec00a830 tx=0x7fadec01d340 comp rx=0 tx=0).stop 2023-05-22T15:28:37.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:37.583+0000 7fadf9b43640 1 -- 172.21.15.16:0/2333240063 shutdown_connections 2023-05-22T15:28:37.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:37.583+0000 7fadf9b43640 1 --2- 172.21.15.16:0/2333240063 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faddc007390 0x7faddc00dba0 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:37.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:37.583+0000 7fadf9b43640 1 -- 172.21.15.16:0/2333240063 >> 172.21.15.16:0/2333240063 conn(0x7fadf4069720 msgr2=0x7fadf4069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:28:37.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:37.583+0000 7fadf9b43640 1 -- 172.21.15.16:0/2333240063 shutdown_connections 2023-05-22T15:28:37.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:37.583+0000 7fadf9b43640 1 -- 172.21.15.16:0/2333240063 wait complete. 2023-05-22T15:28:37.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:37.583+0000 7fadf9b43640 1 Processor -- start 2023-05-22T15:28:37.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:37.583+0000 7fadf9b43640 1 -- start start 2023-05-22T15:28:37.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:37.583+0000 7fadf9b43640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faddc007390 0x7fadf40c5b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:37.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:37.583+0000 7fadf9b43640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fadf4150d00 0x7fadf4148030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:37.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:37.583+0000 7fadf9b43640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fadf4157ae0 0x7fadf4157f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:37.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:37.583+0000 7fadf9b43640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadf41584c0 con 0x7faddc007390 2023-05-22T15:28:37.600 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:37.583+0000 7fadf9b43640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadf40c6090 con 0x7fadf4150d00 2023-05-22T15:28:37.600 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:37.583+0000 7fadf9b43640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fadf40c6f10 con 0x7fadf4157ae0 2023-05-22T15:28:37.600 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:37.587+0000 7fadf37fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faddc007390 0x7fadf40c5b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:37.600 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:37.587+0000 7fadf37fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faddc007390 0x7fadf40c5b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42548/0 (socket says 172.21.15.16:42548) 2023-05-22T15:28:37.600 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:37.587+0000 7fadf37fe640 1 -- 172.21.15.16:0/1553631538 learned_addr learned my addr 172.21.15.16:0/1553631538 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:28:37.600 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:37.587+0000 7fadf2ffd640 1 --2- 172.21.15.16:0/1553631538 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fadf4150d00 0x7fadf4148030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:37.600 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:37.587+0000 7fadf3fff640 1 --2- 172.21.15.16:0/1553631538 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fadf4157ae0 0x7fadf4157f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:37.601 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:37.587+0000 7fadf37fe640 1 -- 172.21.15.16:0/1553631538 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fadf4157ae0 msgr2=0x7fadf4157f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:37.601 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:37.587+0000 7fadf37fe640 1 --2- 172.21.15.16:0/1553631538 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fadf4157ae0 0x7fadf4157f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:37.601 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:37.587+0000 7fadf37fe640 1 -- 172.21.15.16:0/1553631538 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fadf4150d00 msgr2=0x7fadf4148030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:37.601 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:37.587+0000 7fadf37fe640 1 --2- 172.21.15.16:0/1553631538 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fadf4150d00 0x7fadf4148030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:37.601 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:37.587+0000 7fadf37fe640 1 -- 172.21.15.16:0/1553631538 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadf415d200 con 0x7faddc007390 2023-05-22T15:28:37.601 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:37.587+0000 7fadf2ffd640 1 --2- 172.21.15.16:0/1553631538 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fadf4150d00 0x7fadf4148030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:28:37.601 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:37.587+0000 7fadf37fe640 1 --2- 172.21.15.16:0/1553631538 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faddc007390 0x7fadf40c5b50 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7fade4009fd0 tx=0x7fade4011a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:28:37.601 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:37.587+0000 7faddbfff640 1 -- 172.21.15.16:0/1553631538 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fade4015870 con 0x7faddc007390 2023-05-22T15:28:37.602 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:37.587+0000 7faddbfff640 1 -- 172.21.15.16:0/1553631538 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fade40159d0 con 0x7faddc007390 2023-05-22T15:28:37.602 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:37.587+0000 7fadf9b43640 1 -- 172.21.15.16:0/1553631538 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fadf415cf20 con 0x7faddc007390 2023-05-22T15:28:37.602 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:37.587+0000 7faddbfff640 1 -- 172.21.15.16:0/1553631538 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fade4015d80 con 0x7faddc007390 2023-05-22T15:28:37.602 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:37.587+0000 7fadf9b43640 1 -- 172.21.15.16:0/1553631538 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fadf4159160 con 0x7faddc007390 2023-05-22T15:28:37.602 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:37.587+0000 7fadf9b43640 1 -- 172.21.15.16:0/1553631538 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fadf404dbe0 con 0x7faddc007390 2023-05-22T15:28:37.602 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:37.587+0000 7faddbfff640 1 -- 172.21.15.16:0/1553631538 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 9) v1 ==== 84406+0+0 (secure 0 0 0) 0x7fade401d400 con 0x7faddc007390 2023-05-22T15:28:37.603 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:37.587+0000 7faddbfff640 1 --2- 172.21.15.16:0/1553631538 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fade00655b0 0x7fade0067a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:37.603 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:37.587+0000 7fadf2ffd640 1 --2- 172.21.15.16:0/1553631538 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fade00655b0 0x7fade0067a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:37.603 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:37.587+0000 7faddbfff640 1 -- 172.21.15.16:0/1553631538 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 6640+0+0 (secure 0 0 0) 0x7fade4083590 con 0x7faddc007390 2023-05-22T15:28:37.603 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:37.587+0000 7fadf2ffd640 1 --2- 172.21.15.16:0/1553631538 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fade00655b0 0x7fade0067a60 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fade8000c10 tx=0x7fade8009040 comp rx=0 tx=0).ready entity=mgr.5823 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:28:37.603 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:37.591+0000 7faddbfff640 1 -- 172.21.15.16:0/1553631538 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fade4050cf0 con 0x7faddc007390 2023-05-22T15:28:37.773 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:37.767+0000 7fadf9b43640 1 -- 172.21.15.16:0/1553631538 --> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "fs": "cephfs", "path": "/snap_test_dir1/dir2", "repeat": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7fadf4148b00 con 0x7fade00655b0 2023-05-22T15:28:37.883 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:37.875+0000 7faddbfff640 1 -- 172.21.15.16:0/1553631538 <== mgr.5823 v2:172.21.15.16:6832/36921 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7fadf4148b00 con 0x7fade00655b0 2023-05-22T15:28:37.883 INFO:teuthology.orchestra.run.smithi016.stdout:Schedule activated for path /snap_test_dir1/dir2 2023-05-22T15:28:37.886 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:37.879+0000 7fadf9b43640 1 -- 172.21.15.16:0/1553631538 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fade00655b0 msgr2=0x7fade0067a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:37.886 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:37.879+0000 7fadf9b43640 1 --2- 172.21.15.16:0/1553631538 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fade00655b0 0x7fade0067a60 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fade8000c10 tx=0x7fade8009040 comp rx=0 tx=0).stop 2023-05-22T15:28:37.886 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:37.879+0000 7fadf9b43640 1 -- 172.21.15.16:0/1553631538 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faddc007390 msgr2=0x7fadf40c5b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:37.886 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:37.879+0000 7fadf9b43640 1 --2- 172.21.15.16:0/1553631538 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faddc007390 0x7fadf40c5b50 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7fade4009fd0 tx=0x7fade4011a60 comp rx=0 tx=0).stop 2023-05-22T15:28:37.886 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:37.879+0000 7fadf9b43640 1 -- 172.21.15.16:0/1553631538 shutdown_connections 2023-05-22T15:28:37.887 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:37.879+0000 7fadf9b43640 1 --2- 172.21.15.16:0/1553631538 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fade00655b0 0x7fade0067a60 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:37.887 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:37.879+0000 7fadf9b43640 1 --2- 172.21.15.16:0/1553631538 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fadf4157ae0 0x7fadf4157f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:37.887 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:37.879+0000 7fadf9b43640 1 --2- 172.21.15.16:0/1553631538 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fadf4150d00 0x7fadf4148030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:37.887 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:37.879+0000 7fadf9b43640 1 --2- 172.21.15.16:0/1553631538 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faddc007390 0x7fadf40c5b50 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:37.887 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:37.879+0000 7fadf9b43640 1 -- 172.21.15.16:0/1553631538 >> 172.21.15.16:0/1553631538 conn(0x7fadf4069720 msgr2=0x7fadf40cc680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:28:37.887 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:37.879+0000 7fadf9b43640 1 -- 172.21.15.16:0/1553631538 shutdown_connections 2023-05-22T15:28:37.887 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:37.879+0000 7fadf9b43640 1 -- 172.21.15.16:0/1553631538 wait complete. 2023-05-22T15:28:37.905 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/dir2 2023-05-22T15:28:37.906 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate --fs cephfs /snap_test_dir1/dir3 1M 2023-05-22T15:28:38.028 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.023+0000 7f6ac8a0c640 1 Processor -- start 2023-05-22T15:28:38.029 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.023+0000 7f6ac8a0c640 1 -- start start 2023-05-22T15:28:38.029 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.023+0000 7f6ac8a0c640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6ac40ce900 0x7f6ac415c540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:38.030 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.023+0000 7f6ac8a0c640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f6ac40cbe80 0x7f6ac40cc680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:38.030 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.023+0000 7f6ac8a0c640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ac40c3930 con 0x7f6ac40ce530 2023-05-22T15:28:38.030 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.023+0000 7f6ac8a0c640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ac4158f00 con 0x7f6ac40cbe80 2023-05-22T15:28:38.030 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.023+0000 7f6ac8a0c640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ac4159d90 con 0x7f6ac40ce900 2023-05-22T15:28:38.031 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.023+0000 7f6ac2d9d640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f6ac40cbe80 0x7f6ac40cc680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:38.031 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.023+0000 7f6ac1d9b640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6ac40ce900 0x7f6ac415c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:38.032 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.023+0000 7f6ac2d9d640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f6ac40cbe80 0x7f6ac40cc680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.16:38824/0 (socket says 172.21.15.16:38824) 2023-05-22T15:28:38.032 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.023+0000 7f6ac1d9b640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6ac40ce900 0x7f6ac415c540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3301/0 says I am v2:172.21.15.16:51878/0 (socket says 172.21.15.16:51878) 2023-05-22T15:28:38.032 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.023+0000 7f6ac2d9d640 1 -- 172.21.15.16:0/1327230731 learned_addr learned my addr 172.21.15.16:0/1327230731 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:28:38.032 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.023+0000 7f6ac259c640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f6ac40ce530 0x7f6ac4159680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.16:58104/0 (socket says 172.21.15.16:58104) 2023-05-22T15:28:38.033 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.023+0000 7f6ac2d9d640 1 -- 172.21.15.16:0/1327230731 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6ac40ce900 msgr2=0x7f6ac415c540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:28:38.033 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.023+0000 7f6ac2d9d640 1 --2- 172.21.15.16:0/1327230731 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6ac40ce900 0x7f6ac415c540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:38.033 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.023+0000 7f6ac2d9d640 1 -- 172.21.15.16:0/1327230731 >> v1:172.21.15.62:6789/0 conn(0x7f6ac40ce530 legacy=0x7f6ac4159680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:38.033 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.023+0000 7f6ac2d9d640 1 -- 172.21.15.16:0/1327230731 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ac415cc40 con 0x7f6ac40cbe80 2023-05-22T15:28:38.033 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.023+0000 7f6ac1d9b640 1 --2- 172.21.15.16:0/1327230731 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6ac40ce900 0x7f6ac415c540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-05-22T15:28:38.034 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.027+0000 7f6ac2d9d640 1 --2- 172.21.15.16:0/1327230731 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6ac40cbe80 0x7f6ac40cc680 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f6abc00d150 tx=0x7f6abc01ec10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=80385e94e8b3a37d server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:28:38.034 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.027+0000 7f6ac159a640 1 -- 172.21.15.16:0/1327230731 <== mon.1 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6abc00b2d0 con 0x7f6ac40cbe80 2023-05-22T15:28:38.034 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.027+0000 7f6ac159a640 1 -- 172.21.15.16:0/1327230731 <== mon.1 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6abc00b430 con 0x7f6ac40cbe80 2023-05-22T15:28:38.034 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.027+0000 7f6ac159a640 1 -- 172.21.15.16:0/1327230731 <== mon.1 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6abc0a7840 con 0x7f6ac40cbe80 2023-05-22T15:28:38.035 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.027+0000 7f6ac8a0c640 1 -- 172.21.15.16:0/1327230731 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6ac40cbe80 msgr2=0x7f6ac40cc680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:38.035 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.027+0000 7f6ac8a0c640 1 --2- 172.21.15.16:0/1327230731 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6ac40cbe80 0x7f6ac40cc680 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f6abc00d150 tx=0x7f6abc01ec10 comp rx=0 tx=0).stop 2023-05-22T15:28:38.035 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.027+0000 7f6ac8a0c640 1 -- 172.21.15.16:0/1327230731 shutdown_connections 2023-05-22T15:28:38.035 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.027+0000 7f6ac8a0c640 1 --2- 172.21.15.16:0/1327230731 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6ac40cbe80 0x7f6ac40cc680 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:38.036 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.027+0000 7f6ac8a0c640 1 --2- 172.21.15.16:0/1327230731 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6ac40ce900 0x7f6ac415c540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:38.036 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.027+0000 7f6ac8a0c640 1 -- 172.21.15.16:0/1327230731 >> 172.21.15.16:0/1327230731 conn(0x7f6ac4069720 msgr2=0x7f6ac4069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:28:38.036 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.027+0000 7f6ac8a0c640 1 -- 172.21.15.16:0/1327230731 shutdown_connections 2023-05-22T15:28:38.036 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.027+0000 7f6ac8a0c640 1 -- 172.21.15.16:0/1327230731 wait complete. 2023-05-22T15:28:38.038 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.027+0000 7f6ac8a0c640 1 Processor -- start 2023-05-22T15:28:38.038 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.027+0000 7f6ac8a0c640 1 -- start start 2023-05-22T15:28:38.038 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.031+0000 7f6ac8a0c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6ac40cbe80 0x7f6ac4147820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:38.038 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.031+0000 7f6ac8a0c640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6ac40ce530 0x7f6ac40adc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:38.039 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.031+0000 7f6ac8a0c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6ac40ce900 0x7f6ac40ae140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:38.039 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.031+0000 7f6ac8a0c640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ac4155940 con 0x7f6ac40cbe80 2023-05-22T15:28:38.039 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.031+0000 7f6ac8a0c640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ac4155c20 con 0x7f6ac40ce900 2023-05-22T15:28:38.039 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.031+0000 7f6ac8a0c640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ac4155f00 con 0x7f6ac40ce530 2023-05-22T15:28:38.039 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.031+0000 7f6ac259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6ac40cbe80 0x7f6ac4147820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:38.039 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.031+0000 7f6ac259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6ac40cbe80 0x7f6ac4147820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42552/0 (socket says 172.21.15.16:42552) 2023-05-22T15:28:38.039 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.031+0000 7f6ac259c640 1 -- 172.21.15.16:0/1172375673 learned_addr learned my addr 172.21.15.16:0/1172375673 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:28:38.039 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.031+0000 7f6ac1d9b640 1 --2- 172.21.15.16:0/1172375673 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6ac40ce530 0x7f6ac40adc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:38.040 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.031+0000 7f6ac2d9d640 1 --2- 172.21.15.16:0/1172375673 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6ac40ce900 0x7f6ac40ae140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:38.040 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.031+0000 7f6ac259c640 1 -- 172.21.15.16:0/1172375673 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6ac40ce530 msgr2=0x7f6ac40adc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:38.040 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.031+0000 7f6ac259c640 1 --2- 172.21.15.16:0/1172375673 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6ac40ce530 0x7f6ac40adc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:38.040 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.031+0000 7f6ac259c640 1 -- 172.21.15.16:0/1172375673 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6ac40ce900 msgr2=0x7f6ac40ae140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:38.040 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.031+0000 7f6ac259c640 1 --2- 172.21.15.16:0/1172375673 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6ac40ce900 0x7f6ac40ae140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:38.040 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.031+0000 7f6ac259c640 1 -- 172.21.15.16:0/1172375673 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ab8009420 con 0x7f6ac40cbe80 2023-05-22T15:28:38.041 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.031+0000 7f6ac259c640 1 --2- 172.21.15.16:0/1172375673 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6ac40cbe80 0x7f6ac4147820 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f6ab400c420 tx=0x7f6ab400c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:28:38.041 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.031+0000 7f6ab2ffd640 1 -- 172.21.15.16:0/1172375673 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ab40074e0 con 0x7f6ac40cbe80 2023-05-22T15:28:38.041 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.031+0000 7f6ab2ffd640 1 -- 172.21.15.16:0/1172375673 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ab4007640 con 0x7f6ac40cbe80 2023-05-22T15:28:38.041 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.031+0000 7f6ac8a0c640 1 -- 172.21.15.16:0/1172375673 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6abc00bd40 con 0x7f6ac40cbe80 2023-05-22T15:28:38.041 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.031+0000 7f6ab2ffd640 1 -- 172.21.15.16:0/1172375673 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ab4004a30 con 0x7f6ac40cbe80 2023-05-22T15:28:38.042 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.031+0000 7f6ac8a0c640 1 -- 172.21.15.16:0/1172375673 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ac414f2f0 con 0x7f6ac40cbe80 2023-05-22T15:28:38.043 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.031+0000 7f6ac8a0c640 1 -- 172.21.15.16:0/1172375673 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ac4159160 con 0x7f6ac40cbe80 2023-05-22T15:28:38.043 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.031+0000 7f6ab2ffd640 1 -- 172.21.15.16:0/1172375673 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 9) v1 ==== 84406+0+0 (secure 0 0 0) 0x7f6ab4004b90 con 0x7f6ac40cbe80 2023-05-22T15:28:38.043 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.031+0000 7f6ab2ffd640 1 --2- 172.21.15.16:0/1172375673 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f6aa4065580 0x7f6aa4067a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:38.044 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.031+0000 7f6ab2ffd640 1 -- 172.21.15.16:0/1172375673 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 6640+0+0 (secure 0 0 0) 0x7f6ab4082710 con 0x7f6ac40cbe80 2023-05-22T15:28:38.044 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.031+0000 7f6ac1d9b640 1 --2- 172.21.15.16:0/1172375673 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f6aa4065580 0x7f6aa4067a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:38.044 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.035+0000 7f6ac1d9b640 1 --2- 172.21.15.16:0/1172375673 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f6aa4065580 0x7f6aa4067a30 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f6ac4159c10 tx=0x7f6ab8008040 comp rx=0 tx=0).ready entity=mgr.5823 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:28:38.044 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.039+0000 7f6ab2ffd640 1 -- 172.21.15.16:0/1172375673 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f6ab404fed0 con 0x7f6ac40cbe80 2023-05-22T15:28:38.217 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.211+0000 7f6ac8a0c640 1 -- 172.21.15.16:0/1172375673 --> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "fs": "cephfs", "path": "/snap_test_dir1/dir3", "repeat": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f6ac40cda40 con 0x7f6aa4065580 2023-05-22T15:28:38.319 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.311+0000 7f6ab2ffd640 1 -- 172.21.15.16:0/1172375673 <== mgr.5823 v2:172.21.15.16:6832/36921 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f6ac40cda40 con 0x7f6aa4065580 2023-05-22T15:28:38.319 INFO:teuthology.orchestra.run.smithi016.stdout:Schedule activated for path /snap_test_dir1/dir3 2023-05-22T15:28:38.322 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.315+0000 7f6ac8a0c640 1 -- 172.21.15.16:0/1172375673 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f6aa4065580 msgr2=0x7f6aa4067a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:38.322 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.315+0000 7f6ac8a0c640 1 --2- 172.21.15.16:0/1172375673 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f6aa4065580 0x7f6aa4067a30 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f6ac4159c10 tx=0x7f6ab8008040 comp rx=0 tx=0).stop 2023-05-22T15:28:38.322 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.315+0000 7f6ac8a0c640 1 -- 172.21.15.16:0/1172375673 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6ac40cbe80 msgr2=0x7f6ac4147820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:38.322 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.315+0000 7f6ac8a0c640 1 --2- 172.21.15.16:0/1172375673 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6ac40cbe80 0x7f6ac4147820 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f6ab400c420 tx=0x7f6ab400c8e0 comp rx=0 tx=0).stop 2023-05-22T15:28:38.323 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.315+0000 7f6ac8a0c640 1 -- 172.21.15.16:0/1172375673 shutdown_connections 2023-05-22T15:28:38.323 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.315+0000 7f6ac8a0c640 1 --2- 172.21.15.16:0/1172375673 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6ac40ce900 0x7f6ac40ae140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:38.323 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.315+0000 7f6ac8a0c640 1 --2- 172.21.15.16:0/1172375673 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f6aa4065580 0x7f6aa4067a30 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:38.323 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.315+0000 7f6ac8a0c640 1 --2- 172.21.15.16:0/1172375673 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6ac40ce530 0x7f6ac40adc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:38.323 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.315+0000 7f6ac8a0c640 1 --2- 172.21.15.16:0/1172375673 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6ac40cbe80 0x7f6ac4147820 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:38.323 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.315+0000 7f6ac8a0c640 1 -- 172.21.15.16:0/1172375673 >> 172.21.15.16:0/1172375673 conn(0x7f6ac4069720 msgr2=0x7f6ac4157d50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:28:38.323 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.315+0000 7f6ac8a0c640 1 -- 172.21.15.16:0/1172375673 shutdown_connections 2023-05-22T15:28:38.324 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.315+0000 7f6ac8a0c640 1 -- 172.21.15.16:0/1172375673 wait complete. 2023-05-22T15:28:38.341 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/dir3 2023-05-22T15:28:38.342 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate --fs cephfs /snap_test_dir1/dir4 1M 2023-05-22T15:28:38.475 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.467+0000 7f1fbfe26640 1 Processor -- start 2023-05-22T15:28:38.475 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.467+0000 7f1fbfe26640 1 -- start start 2023-05-22T15:28:38.477 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.471+0000 7f1fbfe26640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1fb8158820 0x7f1fb815c5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:38.477 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.471+0000 7f1fbfe26640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f1fb8158bf0 0x7f1fb806fbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:38.477 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.471+0000 7f1fbfe26640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fb80cc3c0 con 0x7f1fb815cb00 2023-05-22T15:28:38.477 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.471+0000 7f1fbfe26640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1fb80cc980 con 0x7f1fb8158bf0 2023-05-22T15:28:38.477 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.471+0000 7f1fbfe26640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1fb815d040 con 0x7f1fb8158820 2023-05-22T15:28:38.478 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.471+0000 7f1fbd3c1640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f1fb8158bf0 0x7f1fb806fbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:38.478 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.471+0000 7f1fbd3c1640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f1fb8158bf0 0x7f1fb806fbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42562/0 (socket says 172.21.15.16:42562) 2023-05-22T15:28:38.478 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.471+0000 7f1fbe3c3640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f1fb815cb00 0x7f1fb80cce10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:48018/0 (socket says 172.21.15.16:48018) 2023-05-22T15:28:38.478 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.471+0000 7f1fbd3c1640 1 -- 172.21.15.16:0/2919702789 learned_addr learned my addr 172.21.15.16:0/2919702789 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:28:38.479 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.471+0000 7f1fbdbc2640 1 --2- 172.21.15.16:0/2919702789 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1fb8158820 0x7f1fb815c5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:38.479 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.471+0000 7f1fbd3c1640 1 -- 172.21.15.16:0/2919702789 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1fb8158820 msgr2=0x7f1fb815c5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:28:38.479 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.471+0000 7f1fbd3c1640 1 --2- 172.21.15.16:0/2919702789 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1fb8158820 0x7f1fb815c5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:38.479 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.471+0000 7f1fbd3c1640 1 -- 172.21.15.16:0/2919702789 >> v1:172.21.15.16:6789/0 conn(0x7f1fb815cb00 legacy=0x7f1fb80cce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:38.479 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.471+0000 7f1fbd3c1640 1 -- 172.21.15.16:0/2919702789 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fb815d320 con 0x7f1fb8158bf0 2023-05-22T15:28:38.480 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.471+0000 7f1fbd3c1640 1 --2- 172.21.15.16:0/2919702789 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1fb8158bf0 0x7f1fb806fbc0 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f1fa8009870 tx=0x7f1fa801c510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=915f0dc1a9865f17 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:28:38.480 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.471+0000 7f1fbcbc0640 1 -- 172.21.15.16:0/2919702789 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fa80ab620 con 0x7f1fb8158bf0 2023-05-22T15:28:38.481 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.471+0000 7f1fbcbc0640 1 -- 172.21.15.16:0/2919702789 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fa80ab780 con 0x7f1fb8158bf0 2023-05-22T15:28:38.481 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.471+0000 7f1fbcbc0640 1 -- 172.21.15.16:0/2919702789 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fa80abb10 con 0x7f1fb8158bf0 2023-05-22T15:28:38.482 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.471+0000 7f1fbfe26640 1 -- 172.21.15.16:0/2919702789 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1fb8158bf0 msgr2=0x7f1fb806fbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:38.482 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.471+0000 7f1fbfe26640 1 --2- 172.21.15.16:0/2919702789 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1fb8158bf0 0x7f1fb806fbc0 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f1fa8009870 tx=0x7f1fa801c510 comp rx=0 tx=0).stop 2023-05-22T15:28:38.482 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.471+0000 7f1fbfe26640 1 -- 172.21.15.16:0/2919702789 shutdown_connections 2023-05-22T15:28:38.482 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.471+0000 7f1fbfe26640 1 --2- 172.21.15.16:0/2919702789 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1fb8158bf0 0x7f1fb806fbc0 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:38.482 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.471+0000 7f1fbfe26640 1 --2- 172.21.15.16:0/2919702789 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1fb8158820 0x7f1fb815c5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:38.482 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.471+0000 7f1fbfe26640 1 -- 172.21.15.16:0/2919702789 >> 172.21.15.16:0/2919702789 conn(0x7f1fb8069720 msgr2=0x7f1fb8069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:28:38.483 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.475+0000 7f1fbfe26640 1 -- 172.21.15.16:0/2919702789 shutdown_connections 2023-05-22T15:28:38.483 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.475+0000 7f1fbfe26640 1 -- 172.21.15.16:0/2919702789 wait complete. 2023-05-22T15:28:38.484 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.475+0000 7f1fbfe26640 1 Processor -- start 2023-05-22T15:28:38.485 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.475+0000 7f1fbfe26640 1 -- start start 2023-05-22T15:28:38.485 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.475+0000 7f1fbfe26640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1fb8158820 0x7f1fb8143a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:38.485 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.475+0000 7f1fbfe26640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1fb8158bf0 0x7f1fb80c31a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:38.486 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.475+0000 7f1fbfe26640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1fb815cb00 0x7f1fb80c8700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:38.486 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.475+0000 7f1fbfe26640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fb814e0b0 con 0x7f1fb8158bf0 2023-05-22T15:28:38.486 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.475+0000 7f1fbfe26640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fb80caee0 con 0x7f1fb8158820 2023-05-22T15:28:38.486 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.475+0000 7f1fbfe26640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1fb80cb1c0 con 0x7f1fb815cb00 2023-05-22T15:28:38.486 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.475+0000 7f1fbd3c1640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1fb8158bf0 0x7f1fb80c31a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:38.486 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.475+0000 7f1fbd3c1640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1fb8158bf0 0x7f1fb80c31a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42564/0 (socket says 172.21.15.16:42564) 2023-05-22T15:28:38.486 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.475+0000 7f1fbd3c1640 1 -- 172.21.15.16:0/3083485492 learned_addr learned my addr 172.21.15.16:0/3083485492 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:28:38.487 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.475+0000 7f1fbe3c3640 1 --2- 172.21.15.16:0/3083485492 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1fb815cb00 0x7f1fb80c8700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:38.487 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.475+0000 7f1fbdbc2640 1 --2- 172.21.15.16:0/3083485492 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1fb8158820 0x7f1fb8143a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:38.487 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.475+0000 7f1fbd3c1640 1 -- 172.21.15.16:0/3083485492 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1fb815cb00 msgr2=0x7f1fb80c8700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:38.487 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.475+0000 7f1fbd3c1640 1 --2- 172.21.15.16:0/3083485492 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1fb815cb00 0x7f1fb80c8700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:38.487 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.475+0000 7f1fbd3c1640 1 -- 172.21.15.16:0/3083485492 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1fb8158820 msgr2=0x7f1fb8143a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:38.487 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.475+0000 7f1fbd3c1640 1 --2- 172.21.15.16:0/3083485492 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1fb8158820 0x7f1fb8143a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:38.487 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.475+0000 7f1fbd3c1640 1 -- 172.21.15.16:0/3083485492 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fa80094d0 con 0x7f1fb8158bf0 2023-05-22T15:28:38.487 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.475+0000 7f1fbd3c1640 1 --2- 172.21.15.16:0/3083485492 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1fb8158bf0 0x7f1fb80c31a0 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f1fa80ad040 tx=0x7f1fa801ad50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:28:38.488 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.475+0000 7f1fae7fc640 1 -- 172.21.15.16:0/3083485492 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fa80be430 con 0x7f1fb8158bf0 2023-05-22T15:28:38.488 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.475+0000 7f1fae7fc640 1 -- 172.21.15.16:0/3083485492 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fa80be5b0 con 0x7f1fb8158bf0 2023-05-22T15:28:38.488 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.475+0000 7f1fbfe26640 1 -- 172.21.15.16:0/3083485492 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1fb80cd6b0 con 0x7f1fb8158bf0 2023-05-22T15:28:38.488 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.475+0000 7f1fae7fc640 1 -- 172.21.15.16:0/3083485492 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fa80be960 con 0x7f1fb8158bf0 2023-05-22T15:28:38.488 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.475+0000 7f1fbfe26640 1 -- 172.21.15.16:0/3083485492 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fb80cd520 con 0x7f1fb8158bf0 2023-05-22T15:28:38.488 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.475+0000 7f1fbfe26640 1 -- 172.21.15.16:0/3083485492 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1fb80ce0d0 con 0x7f1fb8158bf0 2023-05-22T15:28:38.489 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.479+0000 7f1fae7fc640 1 -- 172.21.15.16:0/3083485492 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 9) v1 ==== 84406+0+0 (secure 0 0 0) 0x7f1fa80b43f0 con 0x7f1fb8158bf0 2023-05-22T15:28:38.489 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.479+0000 7f1fae7fc640 1 --2- 172.21.15.16:0/3083485492 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f1f9c0655b0 0x7f1f9c067a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:38.490 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.479+0000 7f1fae7fc640 1 -- 172.21.15.16:0/3083485492 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 6640+0+0 (secure 0 0 0) 0x7f1fa811cfb0 con 0x7f1fb8158bf0 2023-05-22T15:28:38.490 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.479+0000 7f1fbdbc2640 1 --2- 172.21.15.16:0/3083485492 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f1f9c0655b0 0x7f1f9c067a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:38.490 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.479+0000 7f1fbdbc2640 1 --2- 172.21.15.16:0/3083485492 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f1f9c0655b0 0x7f1f9c067a60 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f1fb0003fd0 tx=0x7f1fb00039a0 comp rx=0 tx=0).ready entity=mgr.5823 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:28:38.490 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.483+0000 7f1fae7fc640 1 -- 172.21.15.16:0/3083485492 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f1fa80ea740 con 0x7f1fb8158bf0 2023-05-22T15:28:38.664 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.659+0000 7f1fbfe26640 1 -- 172.21.15.16:0/3083485492 --> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "fs": "cephfs", "path": "/snap_test_dir1/dir4", "repeat": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f1fb80cb330 con 0x7f1f9c0655b0 2023-05-22T15:28:38.759 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.751+0000 7f1fae7fc640 1 -- 172.21.15.16:0/3083485492 <== mgr.5823 v2:172.21.15.16:6832/36921 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f1fb80cb330 con 0x7f1f9c0655b0 2023-05-22T15:28:38.759 INFO:teuthology.orchestra.run.smithi016.stdout:Schedule activated for path /snap_test_dir1/dir4 2023-05-22T15:28:38.762 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.755+0000 7f1fbfe26640 1 -- 172.21.15.16:0/3083485492 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f1f9c0655b0 msgr2=0x7f1f9c067a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:38.762 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.755+0000 7f1fbfe26640 1 --2- 172.21.15.16:0/3083485492 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f1f9c0655b0 0x7f1f9c067a60 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f1fb0003fd0 tx=0x7f1fb00039a0 comp rx=0 tx=0).stop 2023-05-22T15:28:38.762 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.755+0000 7f1fbfe26640 1 -- 172.21.15.16:0/3083485492 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1fb8158bf0 msgr2=0x7f1fb80c31a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:38.762 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.755+0000 7f1fbfe26640 1 --2- 172.21.15.16:0/3083485492 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1fb8158bf0 0x7f1fb80c31a0 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f1fa80ad040 tx=0x7f1fa801ad50 comp rx=0 tx=0).stop 2023-05-22T15:28:38.762 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.755+0000 7f1fbfe26640 1 -- 172.21.15.16:0/3083485492 shutdown_connections 2023-05-22T15:28:38.762 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.755+0000 7f1fbfe26640 1 --2- 172.21.15.16:0/3083485492 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f1f9c0655b0 0x7f1f9c067a60 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:38.763 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.755+0000 7f1fbfe26640 1 --2- 172.21.15.16:0/3083485492 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1fb815cb00 0x7f1fb80c8700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:38.763 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.755+0000 7f1fbfe26640 1 --2- 172.21.15.16:0/3083485492 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1fb8158bf0 0x7f1fb80c31a0 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:38.763 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.755+0000 7f1fbfe26640 1 --2- 172.21.15.16:0/3083485492 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1fb8158820 0x7f1fb8143a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:38.763 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.755+0000 7f1fbfe26640 1 -- 172.21.15.16:0/3083485492 >> 172.21.15.16:0/3083485492 conn(0x7f1fb8069720 msgr2=0x7f1fb815a320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:28:38.763 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.755+0000 7f1fbfe26640 1 -- 172.21.15.16:0/3083485492 shutdown_connections 2023-05-22T15:28:38.763 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.755+0000 7f1fbfe26640 1 -- 172.21.15.16:0/3083485492 wait complete. 2023-05-22T15:28:38.781 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/dir4 2023-05-22T15:28:38.782 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate --fs cephfs /snap_test_dir1/dir5 1M 2023-05-22T15:28:38.908 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.899+0000 7fa7c8e7a640 1 Processor -- start 2023-05-22T15:28:38.908 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.899+0000 7fa7c8e7a640 1 -- start start 2023-05-22T15:28:38.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.903+0000 7fa7c8e7a640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fa7c40ce530 0x7fa7c415c540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:38.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.903+0000 7fa7c8e7a640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fa7c40cbe80 0x7fa7c40cc680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:38.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.903+0000 7fa7c8e7a640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7c40c3930 con 0x7fa7c40ce900 2023-05-22T15:28:38.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.903+0000 7fa7c8e7a640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7c4158f00 con 0x7fa7c40cbe80 2023-05-22T15:28:38.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.903+0000 7fa7c8e7a640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7c4159d90 con 0x7fa7c40ce530 2023-05-22T15:28:38.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.903+0000 7fa7c2d9d640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fa7c40cbe80 0x7fa7c40cc680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:38.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.903+0000 7fa7c1d9b640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fa7c40ce900 0x7fa7c4159680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:48024/0 (socket says 172.21.15.16:48024) 2023-05-22T15:28:38.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.903+0000 7fa7c1d9b640 1 -- 172.21.15.16:0/605406331 learned_addr learned my addr 172.21.15.16:0/605406331 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:28:38.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.903+0000 7fa7c259c640 1 --2- 172.21.15.16:0/605406331 >> v2:172.21.15.62:3300/0 conn(0x7fa7c40ce530 0x7fa7c415c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:38.911 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.903+0000 7fa7c2d9d640 1 -- 172.21.15.16:0/605406331 >> v2:172.21.15.62:3300/0 conn(0x7fa7c40ce530 msgr2=0x7fa7c415c540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:28:38.911 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.903+0000 7fa7c2d9d640 1 --2- 172.21.15.16:0/605406331 >> v2:172.21.15.62:3300/0 conn(0x7fa7c40ce530 0x7fa7c415c540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:38.911 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.903+0000 7fa7c2d9d640 1 -- 172.21.15.16:0/605406331 >> v1:172.21.15.16:6789/0 conn(0x7fa7c40ce900 legacy=0x7fa7c4159680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:38.911 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.903+0000 7fa7c159a640 1 -- 172.21.15.16:0/605406331 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1869834549 0 0) 0x7fa7b8001120 con 0x7fa7c40ce900 2023-05-22T15:28:38.911 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.903+0000 7fa7c2d9d640 1 -- 172.21.15.16:0/605406331 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7c415cc40 con 0x7fa7c40cbe80 2023-05-22T15:28:38.911 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.903+0000 7fa7c259c640 1 --2- 172.21.15.16:0/605406331 >> v2:172.21.15.62:3300/0 conn(0x7fa7c40ce530 0x7fa7c415c540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:28:38.912 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.903+0000 7fa7c2d9d640 1 --2- 172.21.15.16:0/605406331 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa7c40cbe80 0x7fa7c40cc680 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7fa7c40cf160 tx=0x7fa7bc01c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20a0fd26501443c0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:28:38.913 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.903+0000 7fa7c159a640 1 -- 172.21.15.16:0/605406331 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7bc0a9620 con 0x7fa7c40cbe80 2023-05-22T15:28:38.913 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.903+0000 7fa7c159a640 1 -- 172.21.15.16:0/605406331 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7bc0a9780 con 0x7fa7c40cbe80 2023-05-22T15:28:38.913 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.903+0000 7fa7c159a640 1 -- 172.21.15.16:0/605406331 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7bc0a9b10 con 0x7fa7c40cbe80 2023-05-22T15:28:38.913 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.907+0000 7fa7c8e7a640 1 -- 172.21.15.16:0/605406331 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa7c40cbe80 msgr2=0x7fa7c40cc680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:38.913 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.907+0000 7fa7c8e7a640 1 --2- 172.21.15.16:0/605406331 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa7c40cbe80 0x7fa7c40cc680 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7fa7c40cf160 tx=0x7fa7bc01c910 comp rx=0 tx=0).stop 2023-05-22T15:28:38.913 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.907+0000 7fa7c8e7a640 1 -- 172.21.15.16:0/605406331 shutdown_connections 2023-05-22T15:28:38.913 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.907+0000 7fa7c8e7a640 1 --2- 172.21.15.16:0/605406331 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa7c40cbe80 0x7fa7c40cc680 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:38.914 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.907+0000 7fa7c8e7a640 1 --2- 172.21.15.16:0/605406331 >> v2:172.21.15.62:3300/0 conn(0x7fa7c40ce530 0x7fa7c415c540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:38.914 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.907+0000 7fa7c8e7a640 1 -- 172.21.15.16:0/605406331 >> 172.21.15.16:0/605406331 conn(0x7fa7c4069720 msgr2=0x7fa7c4069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:28:38.914 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.907+0000 7fa7c8e7a640 1 -- 172.21.15.16:0/605406331 shutdown_connections 2023-05-22T15:28:38.914 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.907+0000 7fa7c8e7a640 1 -- 172.21.15.16:0/605406331 wait complete. 2023-05-22T15:28:38.915 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.907+0000 7fa7c8e7a640 1 Processor -- start 2023-05-22T15:28:38.916 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.907+0000 7fa7c8e7a640 1 -- start start 2023-05-22T15:28:38.916 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.907+0000 7fa7c8e7a640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa7c40cbe80 0x7fa7c414e780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:38.916 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.907+0000 7fa7c8e7a640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa7c40ce530 0x7fa7c40ac2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:38.916 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.907+0000 7fa7c8e7a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa7c40ce900 0x7fa7c40cb1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:38.916 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.907+0000 7fa7c8e7a640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7c414ecc0 con 0x7fa7c40ce900 2023-05-22T15:28:38.916 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.907+0000 7fa7c8e7a640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7c41573b0 con 0x7fa7c40cbe80 2023-05-22T15:28:38.916 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.907+0000 7fa7c8e7a640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7c4157690 con 0x7fa7c40ce530 2023-05-22T15:28:38.917 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.907+0000 7fa7c2d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa7c40ce900 0x7fa7c40cb1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:38.917 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.907+0000 7fa7c2d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa7c40ce900 0x7fa7c40cb1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42586/0 (socket says 172.21.15.16:42586) 2023-05-22T15:28:38.917 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.907+0000 7fa7c2d9d640 1 -- 172.21.15.16:0/988031680 learned_addr learned my addr 172.21.15.16:0/988031680 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:28:38.917 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.907+0000 7fa7c259c640 1 --2- 172.21.15.16:0/988031680 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa7c40cbe80 0x7fa7c414e780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:38.917 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.907+0000 7fa7c2d9d640 1 -- 172.21.15.16:0/988031680 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa7c40ce530 msgr2=0x7fa7c40ac2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:38.917 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.907+0000 7fa7c1d9b640 1 --2- 172.21.15.16:0/988031680 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa7c40ce530 0x7fa7c40ac2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:38.917 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.907+0000 7fa7c2d9d640 1 --2- 172.21.15.16:0/988031680 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa7c40ce530 0x7fa7c40ac2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:38.918 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.907+0000 7fa7c2d9d640 1 -- 172.21.15.16:0/988031680 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa7c40cbe80 msgr2=0x7fa7c414e780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:38.918 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.907+0000 7fa7c2d9d640 1 --2- 172.21.15.16:0/988031680 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa7c40cbe80 0x7fa7c414e780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:38.918 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.907+0000 7fa7c2d9d640 1 -- 172.21.15.16:0/988031680 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7bc009a70 con 0x7fa7c40ce900 2023-05-22T15:28:38.918 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.907+0000 7fa7c259c640 1 --2- 172.21.15.16:0/988031680 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa7c40cbe80 0x7fa7c414e780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:28:38.918 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.907+0000 7fa7c2d9d640 1 --2- 172.21.15.16:0/988031680 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa7c40ce900 0x7fa7c40cb1e0 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7fa7bc019520 tx=0x7fa7bc0196b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:28:38.918 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.907+0000 7fa7b2ffd640 1 -- 172.21.15.16:0/988031680 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7bc0b7070 con 0x7fa7c40ce900 2023-05-22T15:28:38.919 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.907+0000 7fa7b2ffd640 1 -- 172.21.15.16:0/988031680 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7bc0a99b0 con 0x7fa7c40ce900 2023-05-22T15:28:38.919 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.907+0000 7fa7b2ffd640 1 -- 172.21.15.16:0/988031680 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7bc0a9d20 con 0x7fa7c40ce900 2023-05-22T15:28:38.919 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.907+0000 7fa7c8e7a640 1 -- 172.21.15.16:0/988031680 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7c414f2f0 con 0x7fa7c40ce900 2023-05-22T15:28:38.919 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.907+0000 7fa7c8e7a640 1 -- 172.21.15.16:0/988031680 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7c4159160 con 0x7fa7c40ce900 2023-05-22T15:28:38.920 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.911+0000 7fa7c8e7a640 1 -- 172.21.15.16:0/988031680 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7c414f480 con 0x7fa7c40ce900 2023-05-22T15:28:38.920 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.911+0000 7fa7b2ffd640 1 -- 172.21.15.16:0/988031680 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 9) v1 ==== 84406+0+0 (secure 0 0 0) 0x7fa7bc0bb070 con 0x7fa7c40ce900 2023-05-22T15:28:38.920 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.911+0000 7fa7b2ffd640 1 --2- 172.21.15.16:0/988031680 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fa7a4065580 0x7fa7a4067a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:38.920 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.911+0000 7fa7b2ffd640 1 -- 172.21.15.16:0/988031680 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 6640+0+0 (secure 0 0 0) 0x7fa7bc11d230 con 0x7fa7c40ce900 2023-05-22T15:28:38.921 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.911+0000 7fa7c259c640 1 --2- 172.21.15.16:0/988031680 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fa7a4065580 0x7fa7a4067a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:38.921 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.911+0000 7fa7c259c640 1 --2- 172.21.15.16:0/988031680 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fa7a4065580 0x7fa7a4067a30 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fa7c4070510 tx=0x7fa7b4008040 comp rx=0 tx=0).ready entity=mgr.5823 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:28:38.921 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:38.915+0000 7fa7b2ffd640 1 -- 172.21.15.16:0/988031680 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fa7bc0ea8e0 con 0x7fa7c40ce900 2023-05-22T15:28:39.096 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:39.087+0000 7fa7c8e7a640 1 -- 172.21.15.16:0/988031680 --> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "fs": "cephfs", "path": "/snap_test_dir1/dir5", "repeat": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7fa7c404dbe0 con 0x7fa7a4065580 2023-05-22T15:28:39.200 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:39.191+0000 7fa7b2ffd640 1 -- 172.21.15.16:0/988031680 <== mgr.5823 v2:172.21.15.16:6832/36921 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7fa7c404dbe0 con 0x7fa7a4065580 2023-05-22T15:28:39.200 INFO:teuthology.orchestra.run.smithi016.stdout:Schedule activated for path /snap_test_dir1/dir5 2023-05-22T15:28:39.203 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:39.195+0000 7fa7c8e7a640 1 -- 172.21.15.16:0/988031680 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fa7a4065580 msgr2=0x7fa7a4067a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:39.204 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:39.195+0000 7fa7c8e7a640 1 --2- 172.21.15.16:0/988031680 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fa7a4065580 0x7fa7a4067a30 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fa7c4070510 tx=0x7fa7b4008040 comp rx=0 tx=0).stop 2023-05-22T15:28:39.204 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:39.195+0000 7fa7c8e7a640 1 -- 172.21.15.16:0/988031680 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa7c40ce900 msgr2=0x7fa7c40cb1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:39.204 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:39.195+0000 7fa7c8e7a640 1 --2- 172.21.15.16:0/988031680 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa7c40ce900 0x7fa7c40cb1e0 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7fa7bc019520 tx=0x7fa7bc0196b0 comp rx=0 tx=0).stop 2023-05-22T15:28:39.204 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:39.195+0000 7fa7c8e7a640 1 -- 172.21.15.16:0/988031680 shutdown_connections 2023-05-22T15:28:39.205 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:39.195+0000 7fa7c8e7a640 1 --2- 172.21.15.16:0/988031680 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa7c40ce900 0x7fa7c40cb1e0 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:39.205 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:39.195+0000 7fa7c8e7a640 1 --2- 172.21.15.16:0/988031680 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fa7a4065580 0x7fa7a4067a30 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:39.205 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:39.195+0000 7fa7c8e7a640 1 --2- 172.21.15.16:0/988031680 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa7c40ce530 0x7fa7c40ac2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:39.206 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:39.195+0000 7fa7c8e7a640 1 --2- 172.21.15.16:0/988031680 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa7c40cbe80 0x7fa7c414e780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:39.206 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:39.195+0000 7fa7c8e7a640 1 -- 172.21.15.16:0/988031680 >> 172.21.15.16:0/988031680 conn(0x7fa7c4069720 msgr2=0x7fa7c40c9fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:28:39.206 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:39.195+0000 7fa7c8e7a640 1 -- 172.21.15.16:0/988031680 shutdown_connections 2023-05-22T15:28:39.206 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:39.195+0000 7fa7c8e7a640 1 -- 172.21.15.16:0/988031680 wait complete. 2023-05-22T15:28:39.222 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/dir5 2023-05-22T15:28:39.223 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate --fs cephfs /snap_test_dir1/dir6 1M 2023-05-22T15:28:39.355 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:39.347+0000 7f1fa08ba640 1 Processor -- start 2023-05-22T15:28:39.356 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:39.347+0000 7f1fa08ba640 1 -- start start 2023-05-22T15:28:39.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:39.351+0000 7f1fa08ba640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f1f9c0cc4e0 0x7f1f9c06bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:39.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:39.351+0000 7f1fa08ba640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1f9c15d650 0x7f1f9c15dad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:39.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:39.351+0000 7f1fa08ba640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f9c0cc960 con 0x7f1f9c070c10 2023-05-22T15:28:39.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:39.351+0000 7f1fa08ba640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f9c15e010 con 0x7f1f9c0cc4e0 2023-05-22T15:28:39.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:39.351+0000 7f1fa08ba640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f9c0ccbb0 con 0x7f1f9c15d650 2023-05-22T15:28:39.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:39.351+0000 7f1f9a59c640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f1f9c070c10 0x7f1f9c0c9e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:39942/0 (socket says 172.21.15.16:39942) 2023-05-22T15:28:39.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:39.351+0000 7f1f99d9b640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f1f9c0cc4e0 0x7f1f9c06bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:39.358 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:39.351+0000 7f1f9a59c640 1 -- 172.21.15.16:0/3636858733 learned_addr learned my addr 172.21.15.16:0/3636858733 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:28:39.358 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:39.351+0000 7f1f9ad9d640 1 --2- 172.21.15.16:0/3636858733 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1f9c15d650 0x7f1f9c15dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:39.358 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:39.351+0000 7f1f99d9b640 1 -- 172.21.15.16:0/3636858733 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1f9c15d650 msgr2=0x7f1f9c15dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:28:39.359 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:39.351+0000 7f1f99d9b640 1 --2- 172.21.15.16:0/3636858733 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1f9c15d650 0x7f1f9c15dad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:39.359 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:39.351+0000 7f1f99d9b640 1 -- 172.21.15.16:0/3636858733 >> v1:172.21.15.16:6789/0 conn(0x7f1f9c070c10 legacy=0x7f1f9c0c9e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:39.359 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:39.351+0000 7f1f99d9b640 1 -- 172.21.15.16:0/3636858733 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f9c0cd7c0 con 0x7f1f9c0cc4e0 2023-05-22T15:28:39.359 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:39.351+0000 7f1f9ad9d640 1 --2- 172.21.15.16:0/3636858733 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1f9c15d650 0x7f1f9c15dad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:28:39.360 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:39.351+0000 7f1f99d9b640 1 --2- 172.21.15.16:0/3636858733 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f9c0cc4e0 0x7f1f9c06bfa0 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f1f840097c0 tx=0x7f1f8401c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73b6076f9871078 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:28:39.360 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:39.355+0000 7f1f9959a640 1 -- 172.21.15.16:0/3636858733 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f840ab650 con 0x7f1f9c0cc4e0 2023-05-22T15:28:39.360 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:39.355+0000 7f1f9959a640 1 -- 172.21.15.16:0/3636858733 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f840ab7b0 con 0x7f1f9c0cc4e0 2023-05-22T15:28:39.360 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:39.355+0000 7f1f9959a640 1 -- 172.21.15.16:0/3636858733 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f840abb40 con 0x7f1f9c0cc4e0 2023-05-22T15:28:39.361 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:39.355+0000 7f1fa08ba640 1 -- 172.21.15.16:0/3636858733 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f9c0cc4e0 msgr2=0x7f1f9c06bfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:39.361 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:39.355+0000 7f1fa08ba640 1 --2- 172.21.15.16:0/3636858733 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f9c0cc4e0 0x7f1f9c06bfa0 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f1f840097c0 tx=0x7f1f8401c450 comp rx=0 tx=0).stop 2023-05-22T15:28:39.361 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:39.355+0000 7f1fa08ba640 1 -- 172.21.15.16:0/3636858733 shutdown_connections 2023-05-22T15:28:39.361 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:39.355+0000 7f1fa08ba640 1 --2- 172.21.15.16:0/3636858733 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1f9c15d650 0x7f1f9c15dad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:39.361 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:39.355+0000 7f1fa08ba640 1 --2- 172.21.15.16:0/3636858733 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f9c0cc4e0 0x7f1f9c06bfa0 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:39.362 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:39.355+0000 7f1fa08ba640 1 -- 172.21.15.16:0/3636858733 >> 172.21.15.16:0/3636858733 conn(0x7f1f9c069720 msgr2=0x7f1f9c069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:28:39.362 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:39.355+0000 7f1fa08ba640 1 -- 172.21.15.16:0/3636858733 shutdown_connections 2023-05-22T15:28:39.362 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:39.355+0000 7f1fa08ba640 1 -- 172.21.15.16:0/3636858733 wait complete. 2023-05-22T15:28:39.364 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:39.355+0000 7f1fa08ba640 1 Processor -- start 2023-05-22T15:28:39.364 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:39.355+0000 7f1fa08ba640 1 -- start start 2023-05-22T15:28:39.364 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:39.355+0000 7f1fa08ba640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1f9c070c10 0x7f1f9c149d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:39.364 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:39.355+0000 7f1fa08ba640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1f9c0cc4e0 0x7f1f9c143a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:39.364 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:39.355+0000 7f1fa08ba640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f9c15d650 0x7f1f9c144190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:39.364 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:39.355+0000 7f1fa08ba640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f9c143fc0 con 0x7f1f9c15d650 2023-05-22T15:28:39.365 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:39.355+0000 7f1fa08ba640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f9c0c2f20 con 0x7f1f9c070c10 2023-05-22T15:28:39.365 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:39.355+0000 7f1fa08ba640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f9c144bf0 con 0x7f1f9c0cc4e0 2023-05-22T15:28:39.365 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:39.355+0000 7f1f9ad9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f9c15d650 0x7f1f9c144190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:39.365 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:39.355+0000 7f1f9ad9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f9c15d650 0x7f1f9c144190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46340/0 (socket says 172.21.15.16:46340) 2023-05-22T15:28:39.365 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:39.355+0000 7f1f9ad9d640 1 -- 172.21.15.16:0/2989738523 learned_addr learned my addr 172.21.15.16:0/2989738523 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:28:39.365 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:39.355+0000 7f1f99d9b640 1 --2- 172.21.15.16:0/2989738523 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1f9c0cc4e0 0x7f1f9c143a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:39.365 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:39.355+0000 7f1f9a59c640 1 --2- 172.21.15.16:0/2989738523 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1f9c070c10 0x7f1f9c149d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:39.366 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:39.355+0000 7f1f9ad9d640 1 -- 172.21.15.16:0/2989738523 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1f9c0cc4e0 msgr2=0x7f1f9c143a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:39.366 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:39.355+0000 7f1f9ad9d640 1 --2- 172.21.15.16:0/2989738523 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1f9c0cc4e0 0x7f1f9c143a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:39.366 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:39.355+0000 7f1f9ad9d640 1 -- 172.21.15.16:0/2989738523 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1f9c070c10 msgr2=0x7f1f9c149d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:39.366 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:39.355+0000 7f1f9ad9d640 1 --2- 172.21.15.16:0/2989738523 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1f9c070c10 0x7f1f9c149d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:39.366 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:39.355+0000 7f1f9ad9d640 1 -- 172.21.15.16:0/2989738523 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f84009420 con 0x7f1f9c15d650 2023-05-22T15:28:39.366 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:39.355+0000 7f1f9ad9d640 1 --2- 172.21.15.16:0/2989738523 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f9c15d650 0x7f1f9c144190 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f1f9400cd50 tx=0x7f1f9400a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:28:39.366 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:39.355+0000 7f1f92ffd640 1 -- 172.21.15.16:0/2989738523 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f9400f680 con 0x7f1f9c15d650 2023-05-22T15:28:39.367 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:39.355+0000 7f1f92ffd640 1 -- 172.21.15.16:0/2989738523 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f9400f800 con 0x7f1f9c15d650 2023-05-22T15:28:39.367 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:39.355+0000 7f1f92ffd640 1 -- 172.21.15.16:0/2989738523 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f9400fbb0 con 0x7f1f9c15d650 2023-05-22T15:28:39.367 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:39.355+0000 7f1fa08ba640 1 -- 172.21.15.16:0/2989738523 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f9c06c4e0 con 0x7f1f9c15d650 2023-05-22T15:28:39.367 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:39.355+0000 7f1fa08ba640 1 -- 172.21.15.16:0/2989738523 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f9c070fe0 con 0x7f1f9c15d650 2023-05-22T15:28:39.368 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:39.359+0000 7f1fa08ba640 1 -- 172.21.15.16:0/2989738523 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f9c0cce90 con 0x7f1f9c15d650 2023-05-22T15:28:39.368 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:39.359+0000 7f1f92ffd640 1 -- 172.21.15.16:0/2989738523 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 9) v1 ==== 84406+0+0 (secure 0 0 0) 0x7f1f9400b040 con 0x7f1f9c15d650 2023-05-22T15:28:39.368 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:39.359+0000 7f1f92ffd640 1 --2- 172.21.15.16:0/2989738523 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f1f7c0655b0 0x7f1f7c067a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:39.368 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:39.359+0000 7f1f92ffd640 1 -- 172.21.15.16:0/2989738523 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 6640+0+0 (secure 0 0 0) 0x7f1f94083a70 con 0x7f1f9c15d650 2023-05-22T15:28:39.368 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:39.359+0000 7f1f9a59c640 1 --2- 172.21.15.16:0/2989738523 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f1f7c0655b0 0x7f1f7c067a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:39.369 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:39.359+0000 7f1f9a59c640 1 --2- 172.21.15.16:0/2989738523 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f1f7c0655b0 0x7f1f7c067a60 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f1f8c005890 tx=0x7f1f8c009040 comp rx=0 tx=0).ready entity=mgr.5823 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:28:39.370 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:39.363+0000 7f1f92ffd640 1 -- 172.21.15.16:0/2989738523 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f1f94050fb0 con 0x7f1f9c15d650 2023-05-22T15:28:39.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:39.539+0000 7f1fa08ba640 1 -- 172.21.15.16:0/2989738523 --> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "fs": "cephfs", "path": "/snap_test_dir1/dir6", "repeat": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f1f9c04dbe0 con 0x7f1f7c0655b0 2023-05-22T15:28:39.641 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:39.635+0000 7f1f92ffd640 1 -- 172.21.15.16:0/2989738523 <== mgr.5823 v2:172.21.15.16:6832/36921 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f1f9c04dbe0 con 0x7f1f7c0655b0 2023-05-22T15:28:39.641 INFO:teuthology.orchestra.run.smithi016.stdout:Schedule activated for path /snap_test_dir1/dir6 2023-05-22T15:28:39.643 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:39.635+0000 7f1fa08ba640 1 -- 172.21.15.16:0/2989738523 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f1f7c0655b0 msgr2=0x7f1f7c067a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:39.643 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:39.635+0000 7f1fa08ba640 1 --2- 172.21.15.16:0/2989738523 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f1f7c0655b0 0x7f1f7c067a60 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f1f8c005890 tx=0x7f1f8c009040 comp rx=0 tx=0).stop 2023-05-22T15:28:39.643 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:39.635+0000 7f1fa08ba640 1 -- 172.21.15.16:0/2989738523 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f9c15d650 msgr2=0x7f1f9c144190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:39.644 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:39.635+0000 7f1fa08ba640 1 --2- 172.21.15.16:0/2989738523 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f9c15d650 0x7f1f9c144190 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f1f9400cd50 tx=0x7f1f9400a650 comp rx=0 tx=0).stop 2023-05-22T15:28:39.644 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:39.635+0000 7f1fa08ba640 1 -- 172.21.15.16:0/2989738523 shutdown_connections 2023-05-22T15:28:39.644 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:39.635+0000 7f1fa08ba640 1 --2- 172.21.15.16:0/2989738523 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f9c15d650 0x7f1f9c144190 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:39.645 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:39.635+0000 7f1fa08ba640 1 --2- 172.21.15.16:0/2989738523 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f1f7c0655b0 0x7f1f7c067a60 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:39.645 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:39.635+0000 7f1fa08ba640 1 --2- 172.21.15.16:0/2989738523 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1f9c0cc4e0 0x7f1f9c143a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:39.645 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:39.635+0000 7f1fa08ba640 1 --2- 172.21.15.16:0/2989738523 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1f9c070c10 0x7f1f9c149d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:39.645 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:39.635+0000 7f1fa08ba640 1 -- 172.21.15.16:0/2989738523 >> 172.21.15.16:0/2989738523 conn(0x7f1f9c069720 msgr2=0x7f1f9c158f80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:28:39.645 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:39.639+0000 7f1fa08ba640 1 -- 172.21.15.16:0/2989738523 shutdown_connections 2023-05-22T15:28:39.645 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:39.639+0000 7f1fa08ba640 1 -- 172.21.15.16:0/2989738523 wait complete. 2023-05-22T15:28:39.664 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/dir6 2023-05-22T15:28:39.664 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate --fs cephfs /snap_test_dir1/dir7 1M 2023-05-22T15:28:39.798 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:39.791+0000 7f9d0d588640 1 Processor -- start 2023-05-22T15:28:39.798 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:39.791+0000 7f9d0d588640 1 -- start start 2023-05-22T15:28:39.799 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:39.791+0000 7f9d0d588640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f9d08070c10 0x7f9d080cc4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:39.799 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:39.791+0000 7f9d0d588640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f9d080c9e40 0x7f9d0806bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:39.800 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:39.791+0000 7f9d0d588640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d080ccbb0 con 0x7f9d0815d650 2023-05-22T15:28:39.800 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:39.791+0000 7f9d0d588640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d080cd7c0 con 0x7f9d080c9e40 2023-05-22T15:28:39.800 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:39.791+0000 7f9d0d588640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d080cdaa0 con 0x7f9d08070c10 2023-05-22T15:28:39.800 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:39.791+0000 7f9d067fc640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f9d080c9e40 0x7f9d0806bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:39.800 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:39.791+0000 7f9d067fc640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f9d080c9e40 0x7f9d0806bfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46346/0 (socket says 172.21.15.16:46346) 2023-05-22T15:28:39.800 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:39.791+0000 7f9d067fc640 1 -- 172.21.15.16:0/262779152 learned_addr learned my addr 172.21.15.16:0/262779152 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:28:39.801 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:39.791+0000 7f9d06ffd640 1 --2- 172.21.15.16:0/262779152 >> v2:172.21.15.62:3300/0 conn(0x7f9d08070c10 0x7f9d080cc4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:39.801 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:39.791+0000 7f9d067fc640 1 -- 172.21.15.16:0/262779152 >> v2:172.21.15.62:3300/0 conn(0x7f9d08070c10 msgr2=0x7f9d080cc4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:28:39.801 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:39.791+0000 7f9d067fc640 1 --2- 172.21.15.16:0/262779152 >> v2:172.21.15.62:3300/0 conn(0x7f9d08070c10 0x7f9d080cc4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:39.801 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:39.791+0000 7f9d067fc640 1 -- 172.21.15.16:0/262779152 >> v1:172.21.15.62:6789/0 conn(0x7f9d0815d650 legacy=0x7f9d0815da70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:28:39.801 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:39.791+0000 7f9d067fc640 1 -- 172.21.15.16:0/262779152 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d080cdd80 con 0x7f9d080c9e40 2023-05-22T15:28:39.801 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:39.791+0000 7f9d06ffd640 1 --2- 172.21.15.16:0/262779152 >> v2:172.21.15.62:3300/0 conn(0x7f9d08070c10 0x7f9d080cc4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:28:39.802 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:39.795+0000 7f9d067fc640 1 --2- 172.21.15.16:0/262779152 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9d080c9e40 0x7f9d0806bfa0 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f9d080cf110 tx=0x7f9cf801c4c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da9f7416ca82c464 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:28:39.802 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:39.795+0000 7f9d05ffb640 1 -- 172.21.15.16:0/262779152 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cf80aa640 con 0x7f9d080c9e40 2023-05-22T15:28:39.802 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:39.795+0000 7f9d05ffb640 1 -- 172.21.15.16:0/262779152 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9cf80aa7a0 con 0x7f9d080c9e40 2023-05-22T15:28:39.802 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:39.795+0000 7f9d05ffb640 1 -- 172.21.15.16:0/262779152 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cf80aab30 con 0x7f9d080c9e40 2023-05-22T15:28:39.803 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:39.795+0000 7f9d0d588640 1 -- 172.21.15.16:0/262779152 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9d080c9e40 msgr2=0x7f9d0806bfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:39.803 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:39.795+0000 7f9d0d588640 1 --2- 172.21.15.16:0/262779152 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9d080c9e40 0x7f9d0806bfa0 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f9d080cf110 tx=0x7f9cf801c4c0 comp rx=0 tx=0).stop 2023-05-22T15:28:39.803 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:39.795+0000 7f9d0d588640 1 -- 172.21.15.16:0/262779152 shutdown_connections 2023-05-22T15:28:39.803 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:39.795+0000 7f9d0d588640 1 --2- 172.21.15.16:0/262779152 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9d080c9e40 0x7f9d0806bfa0 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:39.803 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:39.795+0000 7f9d0d588640 1 --2- 172.21.15.16:0/262779152 >> v2:172.21.15.62:3300/0 conn(0x7f9d08070c10 0x7f9d080cc4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:39.804 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:39.795+0000 7f9d0d588640 1 -- 172.21.15.16:0/262779152 >> 172.21.15.16:0/262779152 conn(0x7f9d08069720 msgr2=0x7f9d08069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:28:39.804 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:39.795+0000 7f9d0d588640 1 -- 172.21.15.16:0/262779152 shutdown_connections 2023-05-22T15:28:39.804 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:39.795+0000 7f9d0d588640 1 -- 172.21.15.16:0/262779152 wait complete. 2023-05-22T15:28:39.804 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:39.799+0000 7f9d0d588640 1 Processor -- start 2023-05-22T15:28:39.805 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:39.799+0000 7f9d0d588640 1 -- start start 2023-05-22T15:28:39.806 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:39.799+0000 7f9d0d588640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9d08070c10 0x7f9d0815c290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:39.806 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:39.799+0000 7f9d0d588640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9d080c9e40 0x7f9d080c7a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:39.806 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:39.799+0000 7f9d0d588640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9d0815d650 0x7f9d080b0fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:39.806 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:39.799+0000 7f9d0d588640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d080c7fb0 con 0x7f9d080c9e40 2023-05-22T15:28:39.806 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:39.799+0000 7f9d0d588640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d080b14f0 con 0x7f9d0815d650 2023-05-22T15:28:39.807 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:39.799+0000 7f9d0d588640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d080b17c0 con 0x7f9d08070c10 2023-05-22T15:28:39.807 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:39.799+0000 7f9d067fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9d080c9e40 0x7f9d080c7a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:39.807 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:39.799+0000 7f9d067fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9d080c9e40 0x7f9d080c7a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46348/0 (socket says 172.21.15.16:46348) 2023-05-22T15:28:39.807 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:39.799+0000 7f9d067fc640 1 -- 172.21.15.16:0/75314402 learned_addr learned my addr 172.21.15.16:0/75314402 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:28:39.807 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:39.799+0000 7f9d077fe640 1 --2- 172.21.15.16:0/75314402 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9d0815d650 0x7f9d080b0fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:39.807 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:39.799+0000 7f9d06ffd640 1 --2- 172.21.15.16:0/75314402 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9d08070c10 0x7f9d0815c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:39.807 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:39.799+0000 7f9d067fc640 1 -- 172.21.15.16:0/75314402 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9d08070c10 msgr2=0x7f9d0815c290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:39.808 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:39.799+0000 7f9d067fc640 1 --2- 172.21.15.16:0/75314402 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9d08070c10 0x7f9d0815c290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:39.808 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:39.799+0000 7f9d067fc640 1 -- 172.21.15.16:0/75314402 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9d0815d650 msgr2=0x7f9d080b0fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:39.808 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:39.799+0000 7f9d067fc640 1 --2- 172.21.15.16:0/75314402 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9d0815d650 0x7f9d080b0fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:39.808 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:39.799+0000 7f9d067fc640 1 -- 172.21.15.16:0/75314402 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cf80094d0 con 0x7f9d080c9e40 2023-05-22T15:28:39.808 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:39.799+0000 7f9d067fc640 1 --2- 172.21.15.16:0/75314402 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9d080c9e40 0x7f9d080c7a70 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f9cf801a0f0 tx=0x7f9cf801ad00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:28:39.808 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:39.799+0000 7f9cff7fe640 1 -- 172.21.15.16:0/75314402 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cf80bd5f0 con 0x7f9d080c9e40 2023-05-22T15:28:39.808 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:39.799+0000 7f9cff7fe640 1 -- 172.21.15.16:0/75314402 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9cf80bd750 con 0x7f9d080c9e40 2023-05-22T15:28:39.809 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:39.799+0000 7f9d0d588640 1 -- 172.21.15.16:0/75314402 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d080cce90 con 0x7f9d080c9e40 2023-05-22T15:28:39.809 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:39.799+0000 7f9cff7fe640 1 -- 172.21.15.16:0/75314402 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cf80bdb00 con 0x7f9d080c9e40 2023-05-22T15:28:39.809 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:39.799+0000 7f9d0d588640 1 -- 172.21.15.16:0/75314402 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d0806c4e0 con 0x7f9d080c9e40 2023-05-22T15:28:39.809 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:39.799+0000 7f9d0d588640 1 -- 172.21.15.16:0/75314402 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d080ceab0 con 0x7f9d080c9e40 2023-05-22T15:28:39.810 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:39.799+0000 7f9cff7fe640 1 -- 172.21.15.16:0/75314402 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 9) v1 ==== 84406+0+0 (secure 0 0 0) 0x7f9cf80a8050 con 0x7f9d080c9e40 2023-05-22T15:28:39.810 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:39.799+0000 7f9cff7fe640 1 --2- 172.21.15.16:0/75314402 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f9ce8065580 0x7f9ce8067a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:39.810 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:39.803+0000 7f9cff7fe640 1 -- 172.21.15.16:0/75314402 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 6640+0+0 (secure 0 0 0) 0x7f9cf811dc30 con 0x7f9d080c9e40 2023-05-22T15:28:39.810 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:39.803+0000 7f9d06ffd640 1 --2- 172.21.15.16:0/75314402 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f9ce8065580 0x7f9ce8067a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:39.810 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:39.803+0000 7f9d06ffd640 1 --2- 172.21.15.16:0/75314402 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f9ce8065580 0x7f9ce8067a30 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f9cf000a9a0 tx=0x7f9cf0008040 comp rx=0 tx=0).ready entity=mgr.5823 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:28:39.812 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:39.807+0000 7f9cff7fe640 1 -- 172.21.15.16:0/75314402 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f9cf80eb200 con 0x7f9d080c9e40 2023-05-22T15:28:39.985 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:39.979+0000 7f9d0d588640 1 -- 172.21.15.16:0/75314402 --> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "fs": "cephfs", "path": "/snap_test_dir1/dir7", "repeat": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f9d0804dbe0 con 0x7f9ce8065580 2023-05-22T15:28:40.077 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:40.071+0000 7f9cff7fe640 1 -- 172.21.15.16:0/75314402 <== mgr.5823 v2:172.21.15.16:6832/36921 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f9d0804dbe0 con 0x7f9ce8065580 2023-05-22T15:28:40.077 INFO:teuthology.orchestra.run.smithi016.stdout:Schedule activated for path /snap_test_dir1/dir7 2023-05-22T15:28:40.079 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:40.071+0000 7f9d0d588640 1 -- 172.21.15.16:0/75314402 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f9ce8065580 msgr2=0x7f9ce8067a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:40.080 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:40.071+0000 7f9d0d588640 1 --2- 172.21.15.16:0/75314402 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f9ce8065580 0x7f9ce8067a30 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f9cf000a9a0 tx=0x7f9cf0008040 comp rx=0 tx=0).stop 2023-05-22T15:28:40.080 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:40.071+0000 7f9d0d588640 1 -- 172.21.15.16:0/75314402 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9d080c9e40 msgr2=0x7f9d080c7a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:40.080 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:40.071+0000 7f9d0d588640 1 --2- 172.21.15.16:0/75314402 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9d080c9e40 0x7f9d080c7a70 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f9cf801a0f0 tx=0x7f9cf801ad00 comp rx=0 tx=0).stop 2023-05-22T15:28:40.080 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:40.071+0000 7f9d0d588640 1 -- 172.21.15.16:0/75314402 shutdown_connections 2023-05-22T15:28:40.080 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:40.071+0000 7f9d0d588640 1 --2- 172.21.15.16:0/75314402 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9d0815d650 0x7f9d080b0fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:40.080 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:40.071+0000 7f9d0d588640 1 --2- 172.21.15.16:0/75314402 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9d080c9e40 0x7f9d080c7a70 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:40.081 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:40.071+0000 7f9d0d588640 1 --2- 172.21.15.16:0/75314402 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f9ce8065580 0x7f9ce8067a30 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:40.081 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:40.071+0000 7f9d0d588640 1 --2- 172.21.15.16:0/75314402 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9d08070c10 0x7f9d0815c290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:40.081 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:40.071+0000 7f9d0d588640 1 -- 172.21.15.16:0/75314402 >> 172.21.15.16:0/75314402 conn(0x7f9d08069720 msgr2=0x7f9d08153fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:28:40.081 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:40.071+0000 7f9d0d588640 1 -- 172.21.15.16:0/75314402 shutdown_connections 2023-05-22T15:28:40.081 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:40.071+0000 7f9d0d588640 1 -- 172.21.15.16:0/75314402 wait complete. 2023-05-22T15:28:40.101 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/dir7 2023-05-22T15:28:40.102 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate --fs cephfs /snap_test_dir1/dir8 1M 2023-05-22T15:28:40.235 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:40.227+0000 7f03088af640 1 Processor -- start 2023-05-22T15:28:40.235 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:40.227+0000 7f03088af640 1 -- start start 2023-05-22T15:28:40.236 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:40.231+0000 7f03088af640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f03040ce530 0x7f030415c540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:40.237 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:40.231+0000 7f03088af640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f03040ce900 0x7f0304159680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:40.237 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:40.231+0000 7f03088af640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03040c3930 con 0x7f0304159bc0 2023-05-22T15:28:40.237 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:40.231+0000 7f03088af640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0304158f00 con 0x7f03040ce900 2023-05-22T15:28:40.237 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:40.231+0000 7f03088af640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f030415cc40 con 0x7f03040ce530 2023-05-22T15:28:40.237 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:40.231+0000 7f0301d9b640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f03040ce900 0x7f0304159680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:40.237 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:40.231+0000 7f0301d9b640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f03040ce900 0x7f0304159680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46352/0 (socket says 172.21.15.16:46352) 2023-05-22T15:28:40.238 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:40.231+0000 7f0301d9b640 1 -- 172.21.15.16:0/72731851 learned_addr learned my addr 172.21.15.16:0/72731851 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:28:40.238 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:40.231+0000 7f0301d9b640 1 -- 172.21.15.16:0/72731851 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f03040ce530 msgr2=0x7f030415c540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:28:40.238 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:40.231+0000 7f030259c640 1 --2- 172.21.15.16:0/72731851 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f03040ce530 0x7f030415c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:40.238 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:40.231+0000 7f0301d9b640 1 --2- 172.21.15.16:0/72731851 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f03040ce530 0x7f030415c540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:40.238 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:40.231+0000 7f0301d9b640 1 -- 172.21.15.16:0/72731851 >> v1:172.21.15.62:6789/0 conn(0x7f0304159bc0 legacy=0x7f03040cc680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:28:40.238 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:40.231+0000 7f0301d9b640 1 -- 172.21.15.16:0/72731851 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f030415cf20 con 0x7f03040ce900 2023-05-22T15:28:40.238 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:40.231+0000 7f030259c640 1 --2- 172.21.15.16:0/72731851 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f03040ce530 0x7f030415c540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:28:40.239 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:40.231+0000 7f0301d9b640 1 --2- 172.21.15.16:0/72731851 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f03040ce900 0x7f0304159680 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f02f4009870 tx=0x7f02f401c540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=583da20b8bbe1aef server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:28:40.239 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:40.231+0000 7f030159a640 1 -- 172.21.15.16:0/72731851 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02f40a9400 con 0x7f03040ce900 2023-05-22T15:28:40.240 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:40.231+0000 7f030159a640 1 -- 172.21.15.16:0/72731851 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02f40a9560 con 0x7f03040ce900 2023-05-22T15:28:40.240 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:40.231+0000 7f030159a640 1 -- 172.21.15.16:0/72731851 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02f40a98f0 con 0x7f03040ce900 2023-05-22T15:28:40.240 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:40.235+0000 7f03088af640 1 -- 172.21.15.16:0/72731851 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f03040ce900 msgr2=0x7f0304159680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:40.240 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:40.235+0000 7f03088af640 1 --2- 172.21.15.16:0/72731851 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f03040ce900 0x7f0304159680 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f02f4009870 tx=0x7f02f401c540 comp rx=0 tx=0).stop 2023-05-22T15:28:40.241 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:40.235+0000 7f03088af640 1 -- 172.21.15.16:0/72731851 shutdown_connections 2023-05-22T15:28:40.241 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:40.235+0000 7f03088af640 1 --2- 172.21.15.16:0/72731851 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f03040ce900 0x7f0304159680 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:40.241 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:40.235+0000 7f03088af640 1 --2- 172.21.15.16:0/72731851 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f03040ce530 0x7f030415c540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:40.241 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:40.235+0000 7f03088af640 1 -- 172.21.15.16:0/72731851 >> 172.21.15.16:0/72731851 conn(0x7f0304069720 msgr2=0x7f0304069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:28:40.241 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:40.235+0000 7f03088af640 1 -- 172.21.15.16:0/72731851 shutdown_connections 2023-05-22T15:28:40.241 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:40.235+0000 7f03088af640 1 -- 172.21.15.16:0/72731851 wait complete. 2023-05-22T15:28:40.243 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:40.235+0000 7f03088af640 1 Processor -- start 2023-05-22T15:28:40.243 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:40.235+0000 7f03088af640 1 -- start start 2023-05-22T15:28:40.243 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:40.235+0000 7f03088af640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f03040ce530 0x7f03040b2110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:40.243 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:40.235+0000 7f03088af640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f03040ce900 0x7f0304149320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:40.244 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:40.235+0000 7f03088af640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0304159bc0 0x7f030415ac00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:40.244 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:40.235+0000 7f03088af640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0304149860 con 0x7f03040ce900 2023-05-22T15:28:40.244 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:40.235+0000 7f03088af640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0304149a90 con 0x7f03040ce530 2023-05-22T15:28:40.244 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:40.235+0000 7f03088af640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0304149d70 con 0x7f0304159bc0 2023-05-22T15:28:40.244 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:40.235+0000 7f0301d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f03040ce900 0x7f0304149320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:40.244 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:40.235+0000 7f0301d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f03040ce900 0x7f0304149320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46366/0 (socket says 172.21.15.16:46366) 2023-05-22T15:28:40.244 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:40.235+0000 7f0301d9b640 1 -- 172.21.15.16:0/2571580556 learned_addr learned my addr 172.21.15.16:0/2571580556 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:28:40.245 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:40.235+0000 7f030259c640 1 --2- 172.21.15.16:0/2571580556 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f03040ce530 0x7f03040b2110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:40.245 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:40.235+0000 7f0302d9d640 1 --2- 172.21.15.16:0/2571580556 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0304159bc0 0x7f030415ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:40.245 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:40.235+0000 7f0301d9b640 1 -- 172.21.15.16:0/2571580556 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0304159bc0 msgr2=0x7f030415ac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:40.245 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:40.235+0000 7f0301d9b640 1 --2- 172.21.15.16:0/2571580556 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0304159bc0 0x7f030415ac00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:40.245 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:40.235+0000 7f0301d9b640 1 -- 172.21.15.16:0/2571580556 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f03040ce530 msgr2=0x7f03040b2110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:40.245 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:40.235+0000 7f0301d9b640 1 --2- 172.21.15.16:0/2571580556 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f03040ce530 0x7f03040b2110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:40.245 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:40.235+0000 7f0301d9b640 1 -- 172.21.15.16:0/2571580556 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02f40094d0 con 0x7f03040ce900 2023-05-22T15:28:40.245 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:40.235+0000 7f0302d9d640 1 --2- 172.21.15.16:0/2571580556 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0304159bc0 0x7f030415ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T15:28:40.246 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:40.235+0000 7f030259c640 1 --2- 172.21.15.16:0/2571580556 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f03040ce530 0x7f03040b2110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T15:28:40.246 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:40.235+0000 7f0301d9b640 1 --2- 172.21.15.16:0/2571580556 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f03040ce900 0x7f0304149320 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f02f401a170 tx=0x7f02f401ad80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:28:40.246 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:40.235+0000 7f02f2ffd640 1 -- 172.21.15.16:0/2571580556 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02f40b1800 con 0x7f03040ce900 2023-05-22T15:28:40.246 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:40.235+0000 7f02f2ffd640 1 -- 172.21.15.16:0/2571580556 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02f40b1960 con 0x7f03040ce900 2023-05-22T15:28:40.246 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:40.235+0000 7f03088af640 1 -- 172.21.15.16:0/2571580556 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03040c9700 con 0x7f03040ce900 2023-05-22T15:28:40.246 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:40.235+0000 7f02f2ffd640 1 -- 172.21.15.16:0/2571580556 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02f40b1cf0 con 0x7f03040ce900 2023-05-22T15:28:40.246 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:40.235+0000 7f03088af640 1 -- 172.21.15.16:0/2571580556 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03040c9570 con 0x7f03040ce900 2023-05-22T15:28:40.247 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:40.239+0000 7f03088af640 1 -- 172.21.15.16:0/2571580556 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03040ccd90 con 0x7f03040ce900 2023-05-22T15:28:40.247 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:40.239+0000 7f02f2ffd640 1 -- 172.21.15.16:0/2571580556 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 9) v1 ==== 84406+0+0 (secure 0 0 0) 0x7f02f40ba3a0 con 0x7f03040ce900 2023-05-22T15:28:40.247 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:40.239+0000 7f02f2ffd640 1 --2- 172.21.15.16:0/2571580556 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f02e4065580 0x7f02e4067a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:40.248 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:40.239+0000 7f02f2ffd640 1 -- 172.21.15.16:0/2571580556 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 6640+0+0 (secure 0 0 0) 0x7f02f411d420 con 0x7f03040ce900 2023-05-22T15:28:40.248 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:40.239+0000 7f030259c640 1 --2- 172.21.15.16:0/2571580556 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f02e4065580 0x7f02e4067a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:40.248 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:40.239+0000 7f030259c640 1 --2- 172.21.15.16:0/2571580556 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f02e4065580 0x7f02e4067a30 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f03040b4200 tx=0x7f02f80076e0 comp rx=0 tx=0).ready entity=mgr.5823 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:28:40.249 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:40.243+0000 7f02f2ffd640 1 -- 172.21.15.16:0/2571580556 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f02f40eaa70 con 0x7f03040ce900 2023-05-22T15:28:40.424 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:40.419+0000 7f03088af640 1 -- 172.21.15.16:0/2571580556 --> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "fs": "cephfs", "path": "/snap_test_dir1/dir8", "repeat": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f030404dbe0 con 0x7f02e4065580 2023-05-22T15:28:40.519 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:40.511+0000 7f02f2ffd640 1 -- 172.21.15.16:0/2571580556 <== mgr.5823 v2:172.21.15.16:6832/36921 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f030404dbe0 con 0x7f02e4065580 2023-05-22T15:28:40.519 INFO:teuthology.orchestra.run.smithi016.stdout:Schedule activated for path /snap_test_dir1/dir8 2023-05-22T15:28:40.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:40.515+0000 7f03088af640 1 -- 172.21.15.16:0/2571580556 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f02e4065580 msgr2=0x7f02e4067a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:40.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:40.515+0000 7f03088af640 1 --2- 172.21.15.16:0/2571580556 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f02e4065580 0x7f02e4067a30 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f03040b4200 tx=0x7f02f80076e0 comp rx=0 tx=0).stop 2023-05-22T15:28:40.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:40.515+0000 7f03088af640 1 -- 172.21.15.16:0/2571580556 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f03040ce900 msgr2=0x7f0304149320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:40.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:40.515+0000 7f03088af640 1 --2- 172.21.15.16:0/2571580556 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f03040ce900 0x7f0304149320 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f02f401a170 tx=0x7f02f401ad80 comp rx=0 tx=0).stop 2023-05-22T15:28:40.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:40.515+0000 7f03088af640 1 -- 172.21.15.16:0/2571580556 shutdown_connections 2023-05-22T15:28:40.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:40.515+0000 7f03088af640 1 --2- 172.21.15.16:0/2571580556 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f02e4065580 0x7f02e4067a30 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:40.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:40.515+0000 7f03088af640 1 --2- 172.21.15.16:0/2571580556 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0304159bc0 0x7f030415ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:40.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:40.515+0000 7f03088af640 1 --2- 172.21.15.16:0/2571580556 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f03040ce900 0x7f0304149320 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:40.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:40.515+0000 7f03088af640 1 --2- 172.21.15.16:0/2571580556 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f03040ce530 0x7f03040b2110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:40.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:40.515+0000 7f03088af640 1 -- 172.21.15.16:0/2571580556 >> 172.21.15.16:0/2571580556 conn(0x7f0304069720 msgr2=0x7f030414ecf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:28:40.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:40.515+0000 7f03088af640 1 -- 172.21.15.16:0/2571580556 shutdown_connections 2023-05-22T15:28:40.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:40.515+0000 7f03088af640 1 -- 172.21.15.16:0/2571580556 wait complete. 2023-05-22T15:28:40.543 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/dir8 2023-05-22T15:28:40.543 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate --fs cephfs /snap_test_dir1/dir9 1M 2023-05-22T15:28:40.673 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:40.667+0000 7f748c39a640 1 Processor -- start 2023-05-22T15:28:40.674 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:40.667+0000 7f748c39a640 1 -- start start 2023-05-22T15:28:40.674 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:40.667+0000 7f748c39a640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7484158ad0 0x7f748406b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:40.675 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:40.667+0000 7f748c39a640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f74840ccd20 0x7f74840cd150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:40.675 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:40.667+0000 7f748c39a640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7484158ea0 con 0x7f748406a610 2023-05-22T15:28:40.675 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:40.667+0000 7f748c39a640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74840cd690 con 0x7f74840ccd20 2023-05-22T15:28:40.675 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:40.667+0000 7f748c39a640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74840c96e0 con 0x7f7484158ad0 2023-05-22T15:28:40.675 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:40.667+0000 7f748a937640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f748406a610 0x7f74840ca4a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:39948/0 (socket says 172.21.15.16:39948) 2023-05-22T15:28:40.675 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:40.667+0000 7f748a937640 1 -- 172.21.15.16:0/1448376950 learned_addr learned my addr 172.21.15.16:0/1448376950 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:28:40.676 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:40.667+0000 7f7489935640 1 --2- 172.21.15.16:0/1448376950 >> v2:172.21.15.62:3300/0 conn(0x7f74840ccd20 0x7f74840cd150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:40.676 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:40.667+0000 7f748a136640 1 --2- 172.21.15.16:0/1448376950 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7484158ad0 0x7f748406b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:40.678 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:40.667+0000 7f7489134640 1 -- 172.21.15.16:0/1448376950 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3693580323 0 0) 0x7f7484158ea0 con 0x7f748406a610 2023-05-22T15:28:40.678 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:40.667+0000 7f7489134640 1 -- 172.21.15.16:0/1448376950 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7470005740 con 0x7f748406a610 2023-05-22T15:28:40.678 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:40.667+0000 7f7489134640 1 -- 172.21.15.16:0/1448376950 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2819102970 0 0) 0x7f7470005740 con 0x7f748406a610 2023-05-22T15:28:40.678 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:40.667+0000 7f7489134640 1 -- 172.21.15.16:0/1448376950 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7484158ad0 msgr2=0x7f748406b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:28:40.678 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:40.667+0000 7f7489134640 1 --2- 172.21.15.16:0/1448376950 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7484158ad0 0x7f748406b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:40.679 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:40.667+0000 7f7489134640 1 -- 172.21.15.16:0/1448376950 >> v2:172.21.15.62:3300/0 conn(0x7f74840ccd20 msgr2=0x7f74840cd150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:28:40.679 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:40.667+0000 7f7489134640 1 --2- 172.21.15.16:0/1448376950 >> v2:172.21.15.62:3300/0 conn(0x7f74840ccd20 0x7f74840cd150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:40.679 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:40.667+0000 7f7489134640 1 -- 172.21.15.16:0/1448376950 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7484069a70 con 0x7f748406a610 2023-05-22T15:28:40.679 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:40.667+0000 7f7489134640 1 -- 172.21.15.16:0/1448376950 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7f7480001680 con 0x7f748406a610 2023-05-22T15:28:40.679 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:40.667+0000 7f7489134640 1 -- 172.21.15.16:0/1448376950 >> v1:172.21.15.16:6789/0 conn(0x7f748406a610 legacy=0x7f74840ca4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:40.680 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:40.667+0000 7f7489134640 1 --2- 172.21.15.16:0/1448376950 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7470005b90 0x7f7470005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:40.680 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:40.667+0000 7f7489134640 1 --2- 172.21.15.16:0/1448376950 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7484158ad0 0x7f74700069e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:40.680 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:40.667+0000 7f7489134640 1 --2- 172.21.15.16:0/1448376950 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7470006f20 0x7f7470007390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:40.680 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:40.667+0000 7f7489134640 1 -- 172.21.15.16:0/1448376950 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74840cd690 con 0x7f7470006f20 2023-05-22T15:28:40.680 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:40.667+0000 7f7489134640 1 -- 172.21.15.16:0/1448376950 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74840c96e0 con 0x7f7470005b90 2023-05-22T15:28:40.681 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:40.667+0000 7f7489134640 1 -- 172.21.15.16:0/1448376950 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f747000dba0 con 0x7f7484158ad0 2023-05-22T15:28:40.681 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:40.667+0000 7f748a937640 1 --2- 172.21.15.16:0/1448376950 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7470006f20 0x7f7470007390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:40.681 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:40.671+0000 7f7489935640 1 --2- 172.21.15.16:0/1448376950 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7484158ad0 0x7f74700069e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:40.681 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:40.671+0000 7f748a136640 1 --2- 172.21.15.16:0/1448376950 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7470005b90 0x7f7470005fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:40.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:40.671+0000 7f748a937640 1 -- 172.21.15.16:0/1448376950 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7484158ad0 msgr2=0x7f74700069e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:28:40.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:40.671+0000 7f748a937640 1 --2- 172.21.15.16:0/1448376950 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7484158ad0 0x7f74700069e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:40.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:40.671+0000 7f748a937640 1 -- 172.21.15.16:0/1448376950 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7470005b90 msgr2=0x7f7470005fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:28:40.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:40.671+0000 7f748a937640 1 --2- 172.21.15.16:0/1448376950 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7470005b90 0x7f7470005fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:40.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:40.671+0000 7f748a937640 1 -- 172.21.15.16:0/1448376950 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7470005740 con 0x7f7470006f20 2023-05-22T15:28:40.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:40.671+0000 7f748a937640 1 --2- 172.21.15.16:0/1448376950 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7470006f20 0x7f7470007390 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f748000c240 tx=0x7f748001cc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2092679b5556e883 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:28:40.685 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:40.671+0000 7f7489134640 1 -- 172.21.15.16:0/1448376950 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74800a5960 con 0x7f7470006f20 2023-05-22T15:28:40.685 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:40.671+0000 7f7489134640 1 -- 172.21.15.16:0/1448376950 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74800a5ac0 con 0x7f7470006f20 2023-05-22T15:28:40.685 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:40.671+0000 7f7489134640 1 -- 172.21.15.16:0/1448376950 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74800058e0 con 0x7f7470006f20 2023-05-22T15:28:40.685 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:40.671+0000 7f748c39a640 1 -- 172.21.15.16:0/1448376950 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7470006f20 msgr2=0x7f7470007390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:40.685 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:40.671+0000 7f748c39a640 1 --2- 172.21.15.16:0/1448376950 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7470006f20 0x7f7470007390 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f748000c240 tx=0x7f748001cc00 comp rx=0 tx=0).stop 2023-05-22T15:28:40.686 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:40.671+0000 7f748a937640 1 -- 172.21.15.16:0/1448376950 reap_dead start 2023-05-22T15:28:40.686 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:40.671+0000 7f748c39a640 1 -- 172.21.15.16:0/1448376950 shutdown_connections 2023-05-22T15:28:40.686 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:40.671+0000 7f748c39a640 1 -- 172.21.15.16:0/1448376950 >> 172.21.15.16:0/1448376950 conn(0x7f74840ce8d0 msgr2=0x7f74840cecd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:28:40.686 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:40.671+0000 7f748c39a640 1 -- 172.21.15.16:0/1448376950 shutdown_connections 2023-05-22T15:28:40.686 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:40.671+0000 7f748c39a640 1 -- 172.21.15.16:0/1448376950 wait complete. 2023-05-22T15:28:40.686 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:40.675+0000 7f748c39a640 1 Processor -- start 2023-05-22T15:28:40.686 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:40.675+0000 7f748c39a640 1 -- start start 2023-05-22T15:28:40.686 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:40.675+0000 7f748c39a640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7470006f20 0x7f74841f0f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:40.687 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:40.675+0000 7f748c39a640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f74840b0610 0x7f74840ae5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:40.687 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:40.675+0000 7f748c39a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f74840b0a60 0x7f74840aeb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:40.687 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:40.675+0000 7f748c39a640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f748415bf50 con 0x7f74840b0a60 2023-05-22T15:28:40.687 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:40.675+0000 7f748c39a640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74841f1450 con 0x7f7470006f20 2023-05-22T15:28:40.687 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:40.675+0000 7f748c39a640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f748415b990 con 0x7f74840b0610 2023-05-22T15:28:40.687 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:40.675+0000 7f748a937640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f74840b0a60 0x7f74840aeb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:40.687 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:40.675+0000 7f748a937640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f74840b0a60 0x7f74840aeb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46382/0 (socket says 172.21.15.16:46382) 2023-05-22T15:28:40.688 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:40.675+0000 7f748a937640 1 -- 172.21.15.16:0/3597834578 learned_addr learned my addr 172.21.15.16:0/3597834578 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:28:40.688 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:40.675+0000 7f748a136640 1 --2- 172.21.15.16:0/3597834578 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7470006f20 0x7f74841f0f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:40.688 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:40.675+0000 7f7489935640 1 --2- 172.21.15.16:0/3597834578 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f74840b0610 0x7f74840ae5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:40.688 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:40.675+0000 7f748a937640 1 -- 172.21.15.16:0/3597834578 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f74840b0610 msgr2=0x7f74840ae5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:40.688 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:40.675+0000 7f748a937640 1 --2- 172.21.15.16:0/3597834578 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f74840b0610 0x7f74840ae5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:40.688 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:40.675+0000 7f748a937640 1 -- 172.21.15.16:0/3597834578 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7470006f20 msgr2=0x7f74841f0f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:40.688 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:40.675+0000 7f748a937640 1 --2- 172.21.15.16:0/3597834578 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7470006f20 0x7f74841f0f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:40.688 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:40.675+0000 7f748a937640 1 -- 172.21.15.16:0/3597834578 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74840c9ca0 con 0x7f74840b0a60 2023-05-22T15:28:40.689 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:40.675+0000 7f748a937640 1 --2- 172.21.15.16:0/3597834578 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f74840b0a60 0x7f74840aeb00 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f74800ab040 tx=0x7f7470007a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:28:40.689 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:40.675+0000 7f7476ffd640 1 -- 172.21.15.16:0/3597834578 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74800072d0 con 0x7f74840b0a60 2023-05-22T15:28:40.689 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:40.675+0000 7f7476ffd640 1 -- 172.21.15.16:0/3597834578 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7480007430 con 0x7f74840b0a60 2023-05-22T15:28:40.689 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:40.675+0000 7f7476ffd640 1 -- 172.21.15.16:0/3597834578 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74800077a0 con 0x7f74840b0a60 2023-05-22T15:28:40.689 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:40.675+0000 7f748c39a640 1 -- 172.21.15.16:0/3597834578 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74840c99c0 con 0x7f74840b0a60 2023-05-22T15:28:40.689 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:40.675+0000 7f748c39a640 1 -- 172.21.15.16:0/3597834578 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74840c9f80 con 0x7f74840b0a60 2023-05-22T15:28:40.689 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:40.675+0000 7f748c39a640 1 -- 172.21.15.16:0/3597834578 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f748404dbe0 con 0x7f74840b0a60 2023-05-22T15:28:40.690 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:40.679+0000 7f7476ffd640 1 -- 172.21.15.16:0/3597834578 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 9) v1 ==== 84406+0+0 (secure 0 0 0) 0x7f74800af440 con 0x7f74840b0a60 2023-05-22T15:28:40.690 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:40.679+0000 7f7476ffd640 1 --2- 172.21.15.16:0/3597834578 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f7468065580 0x7f7468067a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:28:40.690 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:40.679+0000 7f748a136640 1 --2- 172.21.15.16:0/3597834578 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f7468065580 0x7f7468067a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:28:40.690 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:40.679+0000 7f7476ffd640 1 -- 172.21.15.16:0/3597834578 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 6640+0+0 (secure 0 0 0) 0x7f74800b5030 con 0x7f74840b0a60 2023-05-22T15:28:40.690 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:40.679+0000 7f748a136640 1 --2- 172.21.15.16:0/3597834578 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f7468065580 0x7f7468067a30 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f747c003b90 tx=0x7f747c00b040 comp rx=0 tx=0).ready entity=mgr.5823 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:28:40.691 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:40.683+0000 7f7476ffd640 1 -- 172.21.15.16:0/3597834578 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f74800ea8f0 con 0x7f74840b0a60 2023-05-22T15:28:40.862 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:40.855+0000 7f748c39a640 1 -- 172.21.15.16:0/3597834578 --> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "fs": "cephfs", "path": "/snap_test_dir1/dir9", "repeat": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f74840b31d0 con 0x7f7468065580 2023-05-22T15:28:40.964 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:40.955+0000 7f7476ffd640 1 -- 172.21.15.16:0/3597834578 <== mgr.5823 v2:172.21.15.16:6832/36921 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f74840b31d0 con 0x7f7468065580 2023-05-22T15:28:40.964 INFO:teuthology.orchestra.run.smithi016.stdout:Schedule activated for path /snap_test_dir1/dir9 2023-05-22T15:28:40.966 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:40.959+0000 7f748c39a640 1 -- 172.21.15.16:0/3597834578 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f7468065580 msgr2=0x7f7468067a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:40.967 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:40.959+0000 7f748c39a640 1 --2- 172.21.15.16:0/3597834578 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f7468065580 0x7f7468067a30 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f747c003b90 tx=0x7f747c00b040 comp rx=0 tx=0).stop 2023-05-22T15:28:40.967 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:40.959+0000 7f748c39a640 1 -- 172.21.15.16:0/3597834578 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f74840b0a60 msgr2=0x7f74840aeb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:28:40.967 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:40.959+0000 7f748c39a640 1 --2- 172.21.15.16:0/3597834578 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f74840b0a60 0x7f74840aeb00 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f74800ab040 tx=0x7f7470007a10 comp rx=0 tx=0).stop 2023-05-22T15:28:40.967 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:40.959+0000 7f748c39a640 1 -- 172.21.15.16:0/3597834578 shutdown_connections 2023-05-22T15:28:40.967 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:40.959+0000 7f748c39a640 1 --2- 172.21.15.16:0/3597834578 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f74840b0a60 0x7f74840aeb00 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:40.968 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:40.959+0000 7f748c39a640 1 --2- 172.21.15.16:0/3597834578 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f7468065580 0x7f7468067a30 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:40.968 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:40.959+0000 7f748c39a640 1 --2- 172.21.15.16:0/3597834578 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f74840b0610 0x7f74840ae5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:40.968 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:40.959+0000 7f748c39a640 1 --2- 172.21.15.16:0/3597834578 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7470006f20 0x7f74841f0f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:28:40.969 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:40.959+0000 7f748c39a640 1 -- 172.21.15.16:0/3597834578 >> 172.21.15.16:0/3597834578 conn(0x7f74840ce8d0 msgr2=0x7f74840cecd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:28:40.969 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:40.959+0000 7f748c39a640 1 -- 172.21.15.16:0/3597834578 shutdown_connections 2023-05-22T15:28:40.969 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:28:40.959+0000 7f748c39a640 1 -- 172.21.15.16:0/3597834578 wait complete. 2023-05-22T15:28:40.985 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/dir9 2023-05-22T15:28:50.530 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-05-22T15:28:50.533 DEBUG:teuthology.orchestra.run.smithi062:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-05-22T15:29:00.094 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:29:00.087+0000 7f894ad22640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T15:29:00.094 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:29:00.087+0000 7f894ad22640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T15:29:00.094 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:29:00.087+0000 7f894ad22640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T15:29:00.095 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:29:00.087+0000 7f894ad22640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T15:29:00.095 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:29:00.087+0000 7f894ad22640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T15:29:20.580 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-05-22T15:29:20.583 DEBUG:teuthology.orchestra.run.smithi062:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-05-22T15:29:50.627 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-05-22T15:29:50.633 DEBUG:teuthology.orchestra.run.smithi062:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-05-22T15:30:20.682 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-05-22T15:30:20.685 DEBUG:teuthology.orchestra.run.smithi062:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-05-22T15:30:50.731 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-05-22T15:30:50.734 DEBUG:teuthology.orchestra.run.smithi062:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-05-22T15:31:20.814 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-05-22T15:31:20.817 DEBUG:teuthology.orchestra.run.smithi062:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-05-22T15:31:50.863 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-05-22T15:31:50.866 DEBUG:teuthology.orchestra.run.smithi062:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-05-22T15:32:20.910 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-05-22T15:32:20.913 DEBUG:teuthology.orchestra.run.smithi062:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-05-22T15:32:50.958 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-05-22T15:32:50.961 DEBUG:teuthology.orchestra.run.smithi062:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-05-22T15:33:21.074 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-05-22T15:33:21.077 DEBUG:teuthology.orchestra.run.smithi062:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-05-22T15:33:51.121 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-05-22T15:33:51.125 DEBUG:teuthology.orchestra.run.smithi062:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-05-22T15:34:00.734 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:34:00.726+0000 7f8949d20640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T15:34:00.734 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:34:00.726+0000 7f8949d20640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T15:34:00.734 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:34:00.726+0000 7f8949d20640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T15:34:00.734 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:34:00.726+0000 7f8949d20640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T15:34:00.735 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:34:00.726+0000 7f8949d20640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T15:34:21.178 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-05-22T15:34:21.180 DEBUG:teuthology.orchestra.run.smithi062:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-05-22T15:34:51.222 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-05-22T15:34:51.225 DEBUG:teuthology.orchestra.run.smithi062:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-05-22T15:35:21.270 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-05-22T15:35:21.272 DEBUG:teuthology.orchestra.run.smithi062:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-05-22T15:35:51.334 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-05-22T15:35:51.337 DEBUG:teuthology.orchestra.run.smithi062:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-05-22T15:36:21.413 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-05-22T15:36:21.416 DEBUG:teuthology.orchestra.run.smithi062:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-05-22T15:36:51.454 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-05-22T15:36:51.457 DEBUG:teuthology.orchestra.run.smithi062:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-05-22T15:37:21.502 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-05-22T15:37:21.505 DEBUG:teuthology.orchestra.run.smithi062:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-05-22T15:37:51.556 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-05-22T15:37:51.560 DEBUG:teuthology.orchestra.run.smithi062:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-05-22T15:38:21.613 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-05-22T15:38:21.616 DEBUG:teuthology.orchestra.run.smithi062:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-05-22T15:38:51.661 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-05-22T15:38:51.665 DEBUG:teuthology.orchestra.run.smithi062:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-05-22T15:39:19.987 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate --fs cephfs /snap_test_dir1/dir0 1M 2023-05-22T15:39:20.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:20.119+0000 7f0066c22640 1 Processor -- start 2023-05-22T15:39:20.126 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:20.119+0000 7f0066c22640 1 -- start start 2023-05-22T15:39:20.127 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:20.123+0000 7f0066c22640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f0060070c10 0x7f00600cc4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:20.127 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:20.123+0000 7f0066c22640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f00600c9e40 0x7f006006bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:20.127 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:20.123+0000 7f0066c22640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00600ccbb0 con 0x7f006015d650 2023-05-22T15:39:20.127 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:20.123+0000 7f0066c22640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00600cd7c0 con 0x7f0060070c10 2023-05-22T15:39:20.127 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:20.123+0000 7f0066c22640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00600cdaa0 con 0x7f00600c9e40 2023-05-22T15:39:20.127 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:20.123+0000 7f00649be640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f0060070c10 0x7f00600cc4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:20.127 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:20.123+0000 7f00649be640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f0060070c10 0x7f00600cc4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52230/0 (socket says 172.21.15.16:52230) 2023-05-22T15:39:20.128 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:20.123+0000 7f00649be640 1 -- 172.21.15.16:0/4232934677 learned_addr learned my addr 172.21.15.16:0/4232934677 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:39:20.128 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:20.123+0000 7f0057fff640 1 --2- 172.21.15.16:0/4232934677 >> v2:172.21.15.62:3300/0 conn(0x7f00600c9e40 0x7f006006bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:20.128 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:20.123+0000 7f00649be640 1 -- 172.21.15.16:0/4232934677 >> v2:172.21.15.62:3300/0 conn(0x7f00600c9e40 msgr2=0x7f006006bfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:39:20.128 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:20.123+0000 7f00649be640 1 --2- 172.21.15.16:0/4232934677 >> v2:172.21.15.62:3300/0 conn(0x7f00600c9e40 0x7f006006bfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:20.129 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:20.123+0000 7f00649be640 1 -- 172.21.15.16:0/4232934677 >> v1:172.21.15.62:6789/0 conn(0x7f006015d650 legacy=0x7f006015da70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:39:20.129 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:20.123+0000 7f00649be640 1 -- 172.21.15.16:0/4232934677 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00600cdd80 con 0x7f0060070c10 2023-05-22T15:39:20.129 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:20.123+0000 7f0057fff640 1 --2- 172.21.15.16:0/4232934677 >> v2:172.21.15.62:3300/0 conn(0x7f00600c9e40 0x7f006006bfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:39:20.129 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:20.123+0000 7f00649be640 1 --2- 172.21.15.16:0/4232934677 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0060070c10 0x7f00600cc4e0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f006015b170 tx=0x7f005c01c4c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a901e5b0d3187111 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:39:20.130 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:20.123+0000 7f00577fe640 1 -- 172.21.15.16:0/4232934677 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f005c0aa640 con 0x7f0060070c10 2023-05-22T15:39:20.130 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:20.123+0000 7f00577fe640 1 -- 172.21.15.16:0/4232934677 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f005c0aa7a0 con 0x7f0060070c10 2023-05-22T15:39:20.130 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:20.123+0000 7f00577fe640 1 -- 172.21.15.16:0/4232934677 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f005c0aab30 con 0x7f0060070c10 2023-05-22T15:39:20.131 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:20.127+0000 7f0066c22640 1 -- 172.21.15.16:0/4232934677 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0060070c10 msgr2=0x7f00600cc4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:20.131 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:20.127+0000 7f0066c22640 1 --2- 172.21.15.16:0/4232934677 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0060070c10 0x7f00600cc4e0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f006015b170 tx=0x7f005c01c4c0 comp rx=0 tx=0).stop 2023-05-22T15:39:20.131 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:20.127+0000 7f0066c22640 1 -- 172.21.15.16:0/4232934677 shutdown_connections 2023-05-22T15:39:20.131 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:20.127+0000 7f0066c22640 1 --2- 172.21.15.16:0/4232934677 >> v2:172.21.15.62:3300/0 conn(0x7f00600c9e40 0x7f006006bfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:20.131 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:20.127+0000 7f0066c22640 1 --2- 172.21.15.16:0/4232934677 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0060070c10 0x7f00600cc4e0 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:20.131 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:20.127+0000 7f0066c22640 1 -- 172.21.15.16:0/4232934677 >> 172.21.15.16:0/4232934677 conn(0x7f0060069720 msgr2=0x7f0060069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:39:20.132 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:20.127+0000 7f0066c22640 1 -- 172.21.15.16:0/4232934677 shutdown_connections 2023-05-22T15:39:20.132 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:20.127+0000 7f0066c22640 1 -- 172.21.15.16:0/4232934677 wait complete. 2023-05-22T15:39:20.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:20.127+0000 7f0066c22640 1 Processor -- start 2023-05-22T15:39:20.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:20.127+0000 7f0066c22640 1 -- start start 2023-05-22T15:39:20.134 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:20.127+0000 7f0066c22640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0060070c10 0x7f00601566c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:20.134 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:20.127+0000 7f0066c22640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f00600c9e40 0x7f006014ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:20.134 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:20.127+0000 7f0066c22640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f006015d650 0x7f006015b200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:20.134 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:20.127+0000 7f0066c22640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0060154aa0 con 0x7f006015d650 2023-05-22T15:39:20.134 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:20.127+0000 7f0066c22640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0060154100 con 0x7f00600c9e40 2023-05-22T15:39:20.135 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:20.127+0000 7f0066c22640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00601543e0 con 0x7f0060070c10 2023-05-22T15:39:20.135 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:20.127+0000 7f00651bf640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f006015d650 0x7f006015b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:20.135 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:20.127+0000 7f00651bf640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f006015d650 0x7f006015b200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52236/0 (socket says 172.21.15.16:52236) 2023-05-22T15:39:20.135 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:20.127+0000 7f00651bf640 1 -- 172.21.15.16:0/382891908 learned_addr learned my addr 172.21.15.16:0/382891908 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:39:20.136 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:20.127+0000 7f00649be640 1 --2- 172.21.15.16:0/382891908 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0060070c10 0x7f00601566c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:20.136 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:20.127+0000 7f0057fff640 1 --2- 172.21.15.16:0/382891908 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f00600c9e40 0x7f006014ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:20.136 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:20.127+0000 7f00651bf640 1 -- 172.21.15.16:0/382891908 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0060070c10 msgr2=0x7f00601566c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:20.136 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:20.127+0000 7f00651bf640 1 --2- 172.21.15.16:0/382891908 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0060070c10 0x7f00601566c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:20.136 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:20.127+0000 7f00651bf640 1 -- 172.21.15.16:0/382891908 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f00600c9e40 msgr2=0x7f006014ecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:20.136 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:20.127+0000 7f00651bf640 1 --2- 172.21.15.16:0/382891908 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f00600c9e40 0x7f006014ecc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:20.136 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:20.127+0000 7f00651bf640 1 -- 172.21.15.16:0/382891908 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f005c0094d0 con 0x7f006015d650 2023-05-22T15:39:20.137 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:20.127+0000 7f00651bf640 1 --2- 172.21.15.16:0/382891908 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f006015d650 0x7f006015b200 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f005800c780 tx=0x7f005800cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:39:20.137 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:20.127+0000 7f00557fa640 1 -- 172.21.15.16:0/382891908 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0058004b70 con 0x7f006015d650 2023-05-22T15:39:20.137 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:20.127+0000 7f00557fa640 1 -- 172.21.15.16:0/382891908 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f005800ce90 con 0x7f006015d650 2023-05-22T15:39:20.137 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:20.127+0000 7f0066c22640 1 -- 172.21.15.16:0/382891908 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00600ca240 con 0x7f006015d650 2023-05-22T15:39:20.137 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:20.127+0000 7f00557fa640 1 -- 172.21.15.16:0/382891908 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f005800a850 con 0x7f006015d650 2023-05-22T15:39:20.137 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:20.127+0000 7f0066c22640 1 -- 172.21.15.16:0/382891908 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00600cce90 con 0x7f006015d650 2023-05-22T15:39:20.138 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:20.127+0000 7f0066c22640 1 -- 172.21.15.16:0/382891908 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00600ca3d0 con 0x7f006015d650 2023-05-22T15:39:20.138 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:20.131+0000 7f00557fa640 1 -- 172.21.15.16:0/382891908 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 12) v1 ==== 84418+0+0 (secure 0 0 0) 0x7f005800a9b0 con 0x7f006015d650 2023-05-22T15:39:20.138 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:20.131+0000 7f00557fa640 1 --2- 172.21.15.16:0/382891908 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f004c065580 0x7f004c067a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:20.138 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:20.131+0000 7f00557fa640 1 -- 172.21.15.16:0/382891908 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 6698+0+0 (secure 0 0 0) 0x7f0058082f00 con 0x7f006015d650 2023-05-22T15:39:20.139 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:20.131+0000 7f00649be640 1 --2- 172.21.15.16:0/382891908 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f004c065580 0x7f004c067a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:20.139 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:20.131+0000 7f00649be640 1 --2- 172.21.15.16:0/382891908 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f004c065580 0x7f004c067a30 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f005c0ae000 tx=0x7f005c0a7000 comp rx=0 tx=0).ready entity=mgr.5823 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:39:20.139 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:20.135+0000 7f00557fa640 1 -- 172.21.15.16:0/382891908 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f0058050710 con 0x7f006015d650 2023-05-22T15:39:20.312 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:20.307+0000 7f0066c22640 1 -- 172.21.15.16:0/382891908 --> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "fs": "cephfs", "path": "/snap_test_dir1/dir0", "repeat": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f0060154be0 con 0x7f004c065580 2023-05-22T15:39:20.464 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:20.459+0000 7f00557fa640 1 -- 172.21.15.16:0/382891908 <== mgr.5823 v2:172.21.15.16:6832/36921 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f0060154be0 con 0x7f004c065580 2023-05-22T15:39:20.464 INFO:teuthology.orchestra.run.smithi016.stdout:Schedule deactivated for path /snap_test_dir1/dir0 2023-05-22T15:39:20.468 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:20.463+0000 7f0066c22640 1 -- 172.21.15.16:0/382891908 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f004c065580 msgr2=0x7f004c067a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:20.468 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:20.463+0000 7f0066c22640 1 --2- 172.21.15.16:0/382891908 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f004c065580 0x7f004c067a30 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f005c0ae000 tx=0x7f005c0a7000 comp rx=0 tx=0).stop 2023-05-22T15:39:20.468 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:20.463+0000 7f0066c22640 1 -- 172.21.15.16:0/382891908 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f006015d650 msgr2=0x7f006015b200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:20.468 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:20.463+0000 7f0066c22640 1 --2- 172.21.15.16:0/382891908 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f006015d650 0x7f006015b200 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f005800c780 tx=0x7f005800cc40 comp rx=0 tx=0).stop 2023-05-22T15:39:20.468 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:20.463+0000 7f0066c22640 1 -- 172.21.15.16:0/382891908 shutdown_connections 2023-05-22T15:39:20.468 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:20.463+0000 7f0066c22640 1 --2- 172.21.15.16:0/382891908 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f006015d650 0x7f006015b200 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:20.469 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:20.463+0000 7f0066c22640 1 --2- 172.21.15.16:0/382891908 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f00600c9e40 0x7f006014ecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:20.469 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:20.463+0000 7f0066c22640 1 --2- 172.21.15.16:0/382891908 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f004c065580 0x7f004c067a30 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:20.469 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:20.463+0000 7f0066c22640 1 --2- 172.21.15.16:0/382891908 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0060070c10 0x7f00601566c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:20.469 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:20.463+0000 7f0066c22640 1 -- 172.21.15.16:0/382891908 >> 172.21.15.16:0/382891908 conn(0x7f0060069720 msgr2=0x7f0060069af0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:39:20.469 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:20.463+0000 7f0066c22640 1 -- 172.21.15.16:0/382891908 shutdown_connections 2023-05-22T15:39:20.469 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:20.463+0000 7f0066c22640 1 -- 172.21.15.16:0/382891908 wait complete. 2023-05-22T15:39:20.488 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/dir0 2023-05-22T15:39:20.488 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate --fs cephfs /snap_test_dir1/dir1 1M 2023-05-22T15:39:20.625 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:20.619+0000 7feb6adc8640 1 Processor -- start 2023-05-22T15:39:20.625 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:20.619+0000 7feb6adc8640 1 -- start start 2023-05-22T15:39:20.627 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:20.623+0000 7feb6adc8640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7feb64070c10 0x7feb640cc4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:20.627 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:20.623+0000 7feb6adc8640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7feb6415d650 0x7feb6415da70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:20.627 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:20.623+0000 7feb6adc8640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb6415dfb0 con 0x7feb640c9e40 2023-05-22T15:39:20.628 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:20.623+0000 7feb6adc8640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb640ccbb0 con 0x7feb6415d650 2023-05-22T15:39:20.628 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:20.623+0000 7feb6adc8640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb640cd7c0 con 0x7feb64070c10 2023-05-22T15:39:20.628 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:20.623+0000 7feb69365640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7feb6415d650 0x7feb6415da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:20.629 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:20.623+0000 7feb69365640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7feb6415d650 0x7feb6415da70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52244/0 (socket says 172.21.15.16:52244) 2023-05-22T15:39:20.629 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:20.623+0000 7feb69365640 1 -- 172.21.15.16:0/1952131252 learned_addr learned my addr 172.21.15.16:0/1952131252 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:39:20.629 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:20.623+0000 7feb68b64640 1 --2- 172.21.15.16:0/1952131252 >> v2:172.21.15.62:3300/0 conn(0x7feb64070c10 0x7feb640cc4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:20.629 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:20.623+0000 7feb69365640 1 -- 172.21.15.16:0/1952131252 >> v2:172.21.15.62:3300/0 conn(0x7feb64070c10 msgr2=0x7feb640cc4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:39:20.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:20.623+0000 7feb69365640 1 --2- 172.21.15.16:0/1952131252 >> v2:172.21.15.62:3300/0 conn(0x7feb64070c10 0x7feb640cc4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:20.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:20.623+0000 7feb69365640 1 -- 172.21.15.16:0/1952131252 >> v1:172.21.15.16:6789/0 conn(0x7feb640c9e40 legacy=0x7feb6406bfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:20.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:20.623+0000 7feb69365640 1 -- 172.21.15.16:0/1952131252 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb640cdaa0 con 0x7feb6415d650 2023-05-22T15:39:20.637 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:20.623+0000 7feb69365640 1 --2- 172.21.15.16:0/1952131252 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feb6415d650 0x7feb6415da70 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7feb640cf110 tx=0x7feb5c01c9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c54cb669499e615 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:39:20.638 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:20.623+0000 7feb637fe640 1 -- 172.21.15.16:0/1952131252 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb5c0a5880 con 0x7feb6415d650 2023-05-22T15:39:20.638 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:20.623+0000 7feb637fe640 1 -- 172.21.15.16:0/1952131252 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb5c0a59e0 con 0x7feb6415d650 2023-05-22T15:39:20.638 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:20.623+0000 7feb637fe640 1 -- 172.21.15.16:0/1952131252 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb5c0a5d70 con 0x7feb6415d650 2023-05-22T15:39:20.638 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:20.623+0000 7feb6adc8640 1 -- 172.21.15.16:0/1952131252 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feb6415d650 msgr2=0x7feb6415da70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:20.639 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:20.623+0000 7feb6adc8640 1 --2- 172.21.15.16:0/1952131252 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feb6415d650 0x7feb6415da70 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7feb640cf110 tx=0x7feb5c01c9a0 comp rx=0 tx=0).stop 2023-05-22T15:39:20.639 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:20.623+0000 7feb6adc8640 1 -- 172.21.15.16:0/1952131252 shutdown_connections 2023-05-22T15:39:20.639 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:20.623+0000 7feb6adc8640 1 --2- 172.21.15.16:0/1952131252 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feb6415d650 0x7feb6415da70 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:20.639 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:20.623+0000 7feb6adc8640 1 --2- 172.21.15.16:0/1952131252 >> v2:172.21.15.62:3300/0 conn(0x7feb64070c10 0x7feb640cc4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:20.640 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:20.623+0000 7feb6adc8640 1 -- 172.21.15.16:0/1952131252 >> 172.21.15.16:0/1952131252 conn(0x7feb64069720 msgr2=0x7feb64069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:39:20.640 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:20.623+0000 7feb6adc8640 1 -- 172.21.15.16:0/1952131252 shutdown_connections 2023-05-22T15:39:20.640 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:20.623+0000 7feb6adc8640 1 -- 172.21.15.16:0/1952131252 wait complete. 2023-05-22T15:39:20.640 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:20.627+0000 7feb6adc8640 1 Processor -- start 2023-05-22T15:39:20.641 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:20.627+0000 7feb6adc8640 1 -- start start 2023-05-22T15:39:20.641 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:20.627+0000 7feb6adc8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feb64070c10 0x7feb641508a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:20.641 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:20.627+0000 7feb6adc8640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7feb640c9e40 0x7feb64145c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:20.641 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:20.627+0000 7feb6adc8640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7feb6415d650 0x7feb6414af20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:20.641 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:20.627+0000 7feb6adc8640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb6415d1d0 con 0x7feb64070c10 2023-05-22T15:39:20.642 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:20.627+0000 7feb6adc8640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb6415b2a0 con 0x7feb640c9e40 2023-05-22T15:39:20.642 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:20.627+0000 7feb6adc8640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb640c5d80 con 0x7feb6415d650 2023-05-22T15:39:20.642 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:20.627+0000 7feb68b64640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feb64070c10 0x7feb641508a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:20.642 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:20.627+0000 7feb68b64640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feb64070c10 0x7feb641508a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52258/0 (socket says 172.21.15.16:52258) 2023-05-22T15:39:20.642 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:20.627+0000 7feb68b64640 1 -- 172.21.15.16:0/2412145897 learned_addr learned my addr 172.21.15.16:0/2412145897 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:39:20.643 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:20.627+0000 7feb63fff640 1 --2- 172.21.15.16:0/2412145897 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7feb640c9e40 0x7feb64145c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:20.643 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:20.627+0000 7feb69365640 1 --2- 172.21.15.16:0/2412145897 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7feb6415d650 0x7feb6414af20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:20.643 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:20.627+0000 7feb68b64640 1 -- 172.21.15.16:0/2412145897 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7feb6415d650 msgr2=0x7feb6414af20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:20.644 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:20.627+0000 7feb68b64640 1 --2- 172.21.15.16:0/2412145897 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7feb6415d650 0x7feb6414af20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:20.644 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:20.627+0000 7feb68b64640 1 -- 172.21.15.16:0/2412145897 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7feb640c9e40 msgr2=0x7feb64145c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:20.644 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:20.627+0000 7feb68b64640 1 --2- 172.21.15.16:0/2412145897 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7feb640c9e40 0x7feb64145c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:20.644 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:20.627+0000 7feb68b64640 1 -- 172.21.15.16:0/2412145897 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb5c009b30 con 0x7feb64070c10 2023-05-22T15:39:20.644 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:20.627+0000 7feb69365640 1 --2- 172.21.15.16:0/2412145897 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7feb6415d650 0x7feb6414af20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:39:20.644 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:20.627+0000 7feb68b64640 1 --2- 172.21.15.16:0/2412145897 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feb64070c10 0x7feb641508a0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7feb5400d5f0 tx=0x7feb5400dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:39:20.644 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:20.627+0000 7feb617fa640 1 -- 172.21.15.16:0/2412145897 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb540125f0 con 0x7feb64070c10 2023-05-22T15:39:20.645 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:20.627+0000 7feb617fa640 1 -- 172.21.15.16:0/2412145897 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb54012750 con 0x7feb64070c10 2023-05-22T15:39:20.645 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:20.627+0000 7feb6adc8640 1 -- 172.21.15.16:0/2412145897 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb640ca240 con 0x7feb64070c10 2023-05-22T15:39:20.645 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:20.627+0000 7feb617fa640 1 -- 172.21.15.16:0/2412145897 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb54012ae0 con 0x7feb64070c10 2023-05-22T15:39:20.645 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:20.627+0000 7feb6adc8640 1 -- 172.21.15.16:0/2412145897 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb640cce90 con 0x7feb64070c10 2023-05-22T15:39:20.645 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:20.627+0000 7feb6adc8640 1 -- 172.21.15.16:0/2412145897 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb640ca3d0 con 0x7feb64070c10 2023-05-22T15:39:20.645 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:20.631+0000 7feb617fa640 1 -- 172.21.15.16:0/2412145897 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 12) v1 ==== 84418+0+0 (secure 0 0 0) 0x7feb54012c40 con 0x7feb64070c10 2023-05-22T15:39:20.645 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:20.631+0000 7feb617fa640 1 --2- 172.21.15.16:0/2412145897 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7feb50065580 0x7feb50067a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:20.645 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:20.631+0000 7feb617fa640 1 -- 172.21.15.16:0/2412145897 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 6698+0+0 (secure 0 0 0) 0x7feb54083360 con 0x7feb64070c10 2023-05-22T15:39:20.646 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:20.631+0000 7feb63fff640 1 --2- 172.21.15.16:0/2412145897 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7feb50065580 0x7feb50067a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:20.646 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:20.631+0000 7feb63fff640 1 --2- 172.21.15.16:0/2412145897 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7feb50065580 0x7feb50067a30 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7feb640ad870 tx=0x7feb58009040 comp rx=0 tx=0).ready entity=mgr.5823 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:39:20.646 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:20.635+0000 7feb617fa640 1 -- 172.21.15.16:0/2412145897 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7feb54050a80 con 0x7feb64070c10 2023-05-22T15:39:20.813 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:20.807+0000 7feb6adc8640 1 -- 172.21.15.16:0/2412145897 --> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "fs": "cephfs", "path": "/snap_test_dir1/dir1", "repeat": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7feb6404dbe0 con 0x7feb50065580 2023-05-22T15:39:20.962 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:20.955+0000 7feb617fa640 1 -- 172.21.15.16:0/2412145897 <== mgr.5823 v2:172.21.15.16:6832/36921 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7feb6404dbe0 con 0x7feb50065580 2023-05-22T15:39:20.963 INFO:teuthology.orchestra.run.smithi016.stdout:Schedule deactivated for path /snap_test_dir1/dir1 2023-05-22T15:39:20.965 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:20.959+0000 7feb6adc8640 1 -- 172.21.15.16:0/2412145897 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7feb50065580 msgr2=0x7feb50067a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:20.965 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:20.959+0000 7feb6adc8640 1 --2- 172.21.15.16:0/2412145897 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7feb50065580 0x7feb50067a30 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7feb640ad870 tx=0x7feb58009040 comp rx=0 tx=0).stop 2023-05-22T15:39:20.965 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:20.959+0000 7feb6adc8640 1 -- 172.21.15.16:0/2412145897 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feb64070c10 msgr2=0x7feb641508a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:20.965 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:20.959+0000 7feb6adc8640 1 --2- 172.21.15.16:0/2412145897 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feb64070c10 0x7feb641508a0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7feb5400d5f0 tx=0x7feb5400dab0 comp rx=0 tx=0).stop 2023-05-22T15:39:20.966 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:20.959+0000 7feb6adc8640 1 -- 172.21.15.16:0/2412145897 shutdown_connections 2023-05-22T15:39:20.966 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:20.959+0000 7feb6adc8640 1 --2- 172.21.15.16:0/2412145897 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7feb50065580 0x7feb50067a30 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:20.966 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:20.959+0000 7feb6adc8640 1 --2- 172.21.15.16:0/2412145897 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7feb6415d650 0x7feb6414af20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:20.966 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:20.959+0000 7feb6adc8640 1 --2- 172.21.15.16:0/2412145897 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7feb640c9e40 0x7feb64145c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:20.966 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:20.959+0000 7feb6adc8640 1 --2- 172.21.15.16:0/2412145897 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feb64070c10 0x7feb641508a0 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:20.966 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:20.959+0000 7feb6adc8640 1 -- 172.21.15.16:0/2412145897 >> 172.21.15.16:0/2412145897 conn(0x7feb64069720 msgr2=0x7feb64069af0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:39:20.967 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:20.959+0000 7feb6adc8640 1 -- 172.21.15.16:0/2412145897 shutdown_connections 2023-05-22T15:39:20.967 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:20.959+0000 7feb6adc8640 1 -- 172.21.15.16:0/2412145897 wait complete. 2023-05-22T15:39:20.986 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/dir1 2023-05-22T15:39:20.987 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate --fs cephfs /snap_test_dir1/dir2 1M 2023-05-22T15:39:21.122 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:21.115+0000 7f6066a61640 1 Processor -- start 2023-05-22T15:39:21.122 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:21.115+0000 7f6066a61640 1 -- start start 2023-05-22T15:39:21.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:21.119+0000 7f6066a61640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f60600ce530 0x7f606015c540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:21.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:21.119+0000 7f6066a61640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60600c3930 con 0x7f60600ce900 2023-05-22T15:39:21.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:21.119+0000 7f6066a61640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6060158f00 con 0x7f60600cbe80 2023-05-22T15:39:21.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:21.119+0000 7f6066a61640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6060159d90 con 0x7f60600ce530 2023-05-22T15:39:21.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:21.119+0000 7f605f7fe640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f60600ce900 0x7f6060159680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:52988/0 (socket says 172.21.15.16:52988) 2023-05-22T15:39:21.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:21.119+0000 7f605f7fe640 1 -- 172.21.15.16:0/2578957182 learned_addr learned my addr 172.21.15.16:0/2578957182 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:39:21.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:21.119+0000 7f605ffff640 1 --2- 172.21.15.16:0/2578957182 >> v2:172.21.15.62:3300/0 conn(0x7f60600ce530 0x7f606015c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:21.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:21.119+0000 7f605effd640 1 -- 172.21.15.16:0/2578957182 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2472939378 0 0) 0x7f6048001120 con 0x7f60600ce900 2023-05-22T15:39:21.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:21.119+0000 7f605effd640 1 -- 172.21.15.16:0/2578957182 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f604c003530 con 0x7f60600ce900 2023-05-22T15:39:21.126 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:21.119+0000 7f605effd640 1 -- 172.21.15.16:0/2578957182 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 173275574 0 0) 0x7f604c003530 con 0x7f60600ce900 2023-05-22T15:39:21.126 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:21.119+0000 7f605effd640 1 -- 172.21.15.16:0/2578957182 >> v2:172.21.15.62:3300/0 conn(0x7f60600ce530 msgr2=0x7f606015c540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:39:21.126 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:21.119+0000 7f605effd640 1 --2- 172.21.15.16:0/2578957182 >> v2:172.21.15.62:3300/0 conn(0x7f60600ce530 0x7f606015c540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:21.126 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:21.119+0000 7f605effd640 1 -- 172.21.15.16:0/2578957182 >> v1:172.21.15.62:6789/0 conn(0x7f60600cbe80 legacy=0x7f60600cc680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:21.126 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:21.119+0000 7f605effd640 1 -- 172.21.15.16:0/2578957182 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f606015cc40 con 0x7f60600ce900 2023-05-22T15:39:21.127 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:21.119+0000 7f605effd640 1 -- 172.21.15.16:0/2578957182 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7f6048001810 con 0x7f60600ce900 2023-05-22T15:39:21.127 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:21.119+0000 7f605effd640 1 -- 172.21.15.16:0/2578957182 >> v1:172.21.15.16:6789/0 conn(0x7f60600ce900 legacy=0x7f6060159680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:21.127 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:21.119+0000 7f605effd640 1 --2- 172.21.15.16:0/2578957182 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f604c003980 0x7f604c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:21.127 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:21.119+0000 7f605effd640 1 --2- 172.21.15.16:0/2578957182 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f604c0047d0 0x7f604c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:21.127 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:21.119+0000 7f605effd640 1 --2- 172.21.15.16:0/2578957182 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f604c005180 0x7f604c00b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:21.127 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:21.119+0000 7f605effd640 1 -- 172.21.15.16:0/2578957182 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6060159d90 con 0x7f604c0047d0 2023-05-22T15:39:21.127 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:21.119+0000 7f605effd640 1 -- 172.21.15.16:0/2578957182 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f604c00c000 con 0x7f604c003980 2023-05-22T15:39:21.128 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:21.119+0000 7f605effd640 1 -- 172.21.15.16:0/2578957182 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f604c00c1c0 con 0x7f604c005180 2023-05-22T15:39:21.128 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:21.119+0000 7f605f7fe640 1 --2- 172.21.15.16:0/2578957182 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f604c0047d0 0x7f604c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:21.128 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:21.119+0000 7f605ffff640 1 --2- 172.21.15.16:0/2578957182 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f604c003980 0x7f604c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:21.128 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:21.119+0000 7f6064ffe640 1 --2- 172.21.15.16:0/2578957182 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f604c005180 0x7f604c00b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:21.128 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:21.119+0000 7f605f7fe640 1 -- 172.21.15.16:0/2578957182 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f604c005180 msgr2=0x7f604c00b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:39:21.128 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:21.119+0000 7f605f7fe640 1 --2- 172.21.15.16:0/2578957182 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f604c005180 0x7f604c00b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:21.128 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:21.119+0000 7f605f7fe640 1 -- 172.21.15.16:0/2578957182 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f604c003980 msgr2=0x7f604c003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:39:21.129 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:21.119+0000 7f605f7fe640 1 --2- 172.21.15.16:0/2578957182 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f604c003980 0x7f604c003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:21.129 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:21.119+0000 7f605f7fe640 1 -- 172.21.15.16:0/2578957182 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f604c003530 con 0x7f604c0047d0 2023-05-22T15:39:21.129 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:21.119+0000 7f6064ffe640 1 -- 172.21.15.16:0/2578957182 reap_dead start 2023-05-22T15:39:21.129 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:21.119+0000 7f605ffff640 1 --2- 172.21.15.16:0/2578957182 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f604c003980 0x7f604c003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T15:39:21.132 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:21.123+0000 7f605f7fe640 1 --2- 172.21.15.16:0/2578957182 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f604c0047d0 0x7f604c004c40 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f604800c240 tx=0x7f604801f3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da916f7a9a199b3e server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:39:21.132 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:21.123+0000 7f605effd640 1 -- 172.21.15.16:0/2578957182 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60480058e0 con 0x7f604c0047d0 2023-05-22T15:39:21.132 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:21.123+0000 7f605effd640 1 -- 172.21.15.16:0/2578957182 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6048005a60 con 0x7f604c0047d0 2023-05-22T15:39:21.132 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:21.123+0000 7f605effd640 1 -- 172.21.15.16:0/2578957182 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6048005e50 con 0x7f604c0047d0 2023-05-22T15:39:21.132 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:21.123+0000 7f6066a61640 1 -- 172.21.15.16:0/2578957182 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f604c0047d0 msgr2=0x7f604c004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:21.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:21.123+0000 7f6066a61640 1 --2- 172.21.15.16:0/2578957182 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f604c0047d0 0x7f604c004c40 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f604800c240 tx=0x7f604801f3b0 comp rx=0 tx=0).stop 2023-05-22T15:39:21.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:21.123+0000 7f6066a61640 1 -- 172.21.15.16:0/2578957182 shutdown_connections 2023-05-22T15:39:21.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:21.123+0000 7f6066a61640 1 --2- 172.21.15.16:0/2578957182 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f604c0047d0 0x7f604c004c40 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:21.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:21.123+0000 7f6066a61640 1 -- 172.21.15.16:0/2578957182 >> 172.21.15.16:0/2578957182 conn(0x7f6060069720 msgr2=0x7f6060069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:39:21.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:21.123+0000 7f6066a61640 1 -- 172.21.15.16:0/2578957182 shutdown_connections 2023-05-22T15:39:21.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:21.123+0000 7f6066a61640 1 -- 172.21.15.16:0/2578957182 wait complete. 2023-05-22T15:39:21.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:21.123+0000 7f6066a61640 1 Processor -- start 2023-05-22T15:39:21.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:21.123+0000 7f6066a61640 1 -- start start 2023-05-22T15:39:21.134 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:21.123+0000 7f6066a61640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f604c0047d0 0x7f606014dd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:21.134 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:21.123+0000 7f6066a61640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f606006bd40 0x7f60600adc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:21.134 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:21.123+0000 7f6066a61640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f60600ae140 0x7f606014f6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:21.134 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:21.123+0000 7f6066a61640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f606014e280 con 0x7f60600ae140 2023-05-22T15:39:21.134 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:21.123+0000 7f6066a61640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f606014e630 con 0x7f604c0047d0 2023-05-22T15:39:21.134 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:21.123+0000 7f6066a61640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f606014e910 con 0x7f606006bd40 2023-05-22T15:39:21.134 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:21.123+0000 7f6064ffe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f60600ae140 0x7f606014f6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:21.135 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:21.123+0000 7f6064ffe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f60600ae140 0x7f606014f6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52274/0 (socket says 172.21.15.16:52274) 2023-05-22T15:39:21.135 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:21.123+0000 7f6064ffe640 1 -- 172.21.15.16:0/3002124802 learned_addr learned my addr 172.21.15.16:0/3002124802 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:39:21.135 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:21.123+0000 7f605ffff640 1 --2- 172.21.15.16:0/3002124802 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f604c0047d0 0x7f606014dd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:21.135 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:21.123+0000 7f605f7fe640 1 --2- 172.21.15.16:0/3002124802 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f606006bd40 0x7f60600adc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:21.135 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:21.123+0000 7f6064ffe640 1 -- 172.21.15.16:0/3002124802 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f606006bd40 msgr2=0x7f60600adc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:21.135 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:21.123+0000 7f6064ffe640 1 --2- 172.21.15.16:0/3002124802 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f606006bd40 0x7f60600adc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:21.135 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:21.123+0000 7f6064ffe640 1 -- 172.21.15.16:0/3002124802 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f604c0047d0 msgr2=0x7f606014dd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:21.135 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:21.123+0000 7f6064ffe640 1 --2- 172.21.15.16:0/3002124802 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f604c0047d0 0x7f606014dd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:21.136 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:21.123+0000 7f6064ffe640 1 -- 172.21.15.16:0/3002124802 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f606015d200 con 0x7f60600ae140 2023-05-22T15:39:21.136 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:21.123+0000 7f605ffff640 1 --2- 172.21.15.16:0/3002124802 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f604c0047d0 0x7f606014dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:39:21.136 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:21.123+0000 7f6064ffe640 1 --2- 172.21.15.16:0/3002124802 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f60600ae140 0x7f606014f6d0 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f60600caad0 tx=0x7f605800e840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:39:21.136 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:21.127+0000 7f605cff9640 1 -- 172.21.15.16:0/3002124802 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f605800ed60 con 0x7f60600ae140 2023-05-22T15:39:21.136 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:21.127+0000 7f605cff9640 1 -- 172.21.15.16:0/3002124802 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f605800eec0 con 0x7f60600ae140 2023-05-22T15:39:21.136 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:21.127+0000 7f6066a61640 1 -- 172.21.15.16:0/3002124802 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f606015cf20 con 0x7f60600ae140 2023-05-22T15:39:21.136 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:21.127+0000 7f605cff9640 1 -- 172.21.15.16:0/3002124802 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6058003bc0 con 0x7f60600ae140 2023-05-22T15:39:21.137 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:21.127+0000 7f6066a61640 1 -- 172.21.15.16:0/3002124802 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6060159160 con 0x7f60600ae140 2023-05-22T15:39:21.137 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:21.127+0000 7f6066a61640 1 -- 172.21.15.16:0/3002124802 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f606004dbe0 con 0x7f60600ae140 2023-05-22T15:39:21.137 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:21.127+0000 7f605cff9640 1 -- 172.21.15.16:0/3002124802 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 12) v1 ==== 84418+0+0 (secure 0 0 0) 0x7f6058003d20 con 0x7f60600ae140 2023-05-22T15:39:21.137 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:21.127+0000 7f605cff9640 1 --2- 172.21.15.16:0/3002124802 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f60400655b0 0x7f6040067a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:21.137 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:21.127+0000 7f605cff9640 1 -- 172.21.15.16:0/3002124802 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 6698+0+0 (secure 0 0 0) 0x7f6058081fb0 con 0x7f60600ae140 2023-05-22T15:39:21.137 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:21.127+0000 7f605ffff640 1 --2- 172.21.15.16:0/3002124802 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f60400655b0 0x7f6040067a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:21.138 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:21.127+0000 7f605ffff640 1 --2- 172.21.15.16:0/3002124802 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f60400655b0 0x7f6040067a60 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f60601f0c70 tx=0x7f6050008040 comp rx=0 tx=0).ready entity=mgr.5823 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:39:21.138 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:21.131+0000 7f605cff9640 1 -- 172.21.15.16:0/3002124802 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f605804f7c0 con 0x7f60600ae140 2023-05-22T15:39:21.308 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:21.303+0000 7f6066a61640 1 -- 172.21.15.16:0/3002124802 --> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "fs": "cephfs", "path": "/snap_test_dir1/dir2", "repeat": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f60600cc460 con 0x7f60400655b0 2023-05-22T15:39:21.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:21.455+0000 7f605cff9640 1 -- 172.21.15.16:0/3002124802 <== mgr.5823 v2:172.21.15.16:6832/36921 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f60600cc460 con 0x7f60400655b0 2023-05-22T15:39:21.459 INFO:teuthology.orchestra.run.smithi016.stdout:Schedule deactivated for path /snap_test_dir1/dir2 2023-05-22T15:39:21.461 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:21.455+0000 7f6066a61640 1 -- 172.21.15.16:0/3002124802 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f60400655b0 msgr2=0x7f6040067a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:21.461 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:21.455+0000 7f6066a61640 1 --2- 172.21.15.16:0/3002124802 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f60400655b0 0x7f6040067a60 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f60601f0c70 tx=0x7f6050008040 comp rx=0 tx=0).stop 2023-05-22T15:39:21.462 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:21.455+0000 7f6066a61640 1 -- 172.21.15.16:0/3002124802 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f60600ae140 msgr2=0x7f606014f6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:21.462 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:21.455+0000 7f6066a61640 1 --2- 172.21.15.16:0/3002124802 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f60600ae140 0x7f606014f6d0 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f60600caad0 tx=0x7f605800e840 comp rx=0 tx=0).stop 2023-05-22T15:39:21.462 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:21.455+0000 7f6066a61640 1 -- 172.21.15.16:0/3002124802 shutdown_connections 2023-05-22T15:39:21.462 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:21.455+0000 7f6066a61640 1 --2- 172.21.15.16:0/3002124802 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f60600ae140 0x7f606014f6d0 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:21.462 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:21.455+0000 7f6066a61640 1 --2- 172.21.15.16:0/3002124802 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f60400655b0 0x7f6040067a60 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:21.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:21.455+0000 7f6066a61640 1 --2- 172.21.15.16:0/3002124802 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f606006bd40 0x7f60600adc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:21.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:21.455+0000 7f6066a61640 1 --2- 172.21.15.16:0/3002124802 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f604c0047d0 0x7f606014dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:21.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:21.455+0000 7f6066a61640 1 -- 172.21.15.16:0/3002124802 >> 172.21.15.16:0/3002124802 conn(0x7f6060069720 msgr2=0x7f6060070920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:39:21.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:21.455+0000 7f6066a61640 1 -- 172.21.15.16:0/3002124802 shutdown_connections 2023-05-22T15:39:21.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:21.455+0000 7f6066a61640 1 -- 172.21.15.16:0/3002124802 wait complete. 2023-05-22T15:39:21.480 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/dir2 2023-05-22T15:39:21.480 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate --fs cephfs /snap_test_dir1/dir3 1M 2023-05-22T15:39:21.612 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:21.607+0000 7fc988a5e640 1 Processor -- start 2023-05-22T15:39:21.612 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:21.607+0000 7fc988a5e640 1 -- start start 2023-05-22T15:39:21.613 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:21.607+0000 7fc988a5e640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fc9840cc010 0x7fc9840ccd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:21.613 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:21.607+0000 7fc988a5e640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc984070bb0 con 0x7fc984158950 2023-05-22T15:39:21.613 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:21.607+0000 7fc988a5e640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc984070e90 con 0x7fc9840cb440 2023-05-22T15:39:21.613 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:21.607+0000 7fc988a5e640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc984071170 con 0x7fc9840cc010 2023-05-22T15:39:21.615 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:21.607+0000 7fc98259c640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fc984158950 0x7fc9840cb900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:52994/0 (socket says 172.21.15.16:52994) 2023-05-22T15:39:21.616 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:21.607+0000 7fc98259c640 1 -- 172.21.15.16:0/1436559821 learned_addr learned my addr 172.21.15.16:0/1436559821 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:39:21.616 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:21.607+0000 7fc981d9b640 1 --2- 172.21.15.16:0/1436559821 >> v2:172.21.15.62:3300/0 conn(0x7fc9840cc010 0x7fc9840ccd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:21.616 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:21.607+0000 7fc98159a640 1 -- 172.21.15.16:0/1436559821 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 973761336 0 0) 0x7fc984070bb0 con 0x7fc984158950 2023-05-22T15:39:21.616 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:21.607+0000 7fc98159a640 1 -- 172.21.15.16:0/1436559821 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc964005740 con 0x7fc984158950 2023-05-22T15:39:21.616 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:21.607+0000 7fc98159a640 1 -- 172.21.15.16:0/1436559821 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2954574235 0 0) 0x7fc964005740 con 0x7fc984158950 2023-05-22T15:39:21.616 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:21.607+0000 7fc98159a640 1 -- 172.21.15.16:0/1436559821 >> v2:172.21.15.62:3300/0 conn(0x7fc9840cc010 msgr2=0x7fc9840ccd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:39:21.616 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:21.607+0000 7fc98159a640 1 --2- 172.21.15.16:0/1436559821 >> v2:172.21.15.62:3300/0 conn(0x7fc9840cc010 0x7fc9840ccd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:21.617 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:21.607+0000 7fc98159a640 1 -- 172.21.15.16:0/1436559821 >> v1:172.21.15.62:6789/0 conn(0x7fc9840cb440 legacy=0x7fc9840cd240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:21.617 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:21.607+0000 7fc98159a640 1 -- 172.21.15.16:0/1436559821 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc984071450 con 0x7fc984158950 2023-05-22T15:39:21.617 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:21.611+0000 7fc98159a640 1 -- 172.21.15.16:0/1436559821 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7fc96c001680 con 0x7fc984158950 2023-05-22T15:39:21.617 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:21.611+0000 7fc98159a640 1 -- 172.21.15.16:0/1436559821 >> v1:172.21.15.16:6789/0 conn(0x7fc984158950 legacy=0x7fc9840cb900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:21.617 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:21.611+0000 7fc98159a640 1 --2- 172.21.15.16:0/1436559821 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc964005b90 0x7fc964005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:21.617 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:21.611+0000 7fc98159a640 1 --2- 172.21.15.16:0/1436559821 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc9640069e0 0x7fc964006e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:21.617 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:21.611+0000 7fc98159a640 1 --2- 172.21.15.16:0/1436559821 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc964007390 0x7fc96400dba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:21.617 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:21.611+0000 7fc98159a640 1 -- 172.21.15.16:0/1436559821 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc984071170 con 0x7fc964005b90 2023-05-22T15:39:21.618 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:21.611+0000 7fc98159a640 1 -- 172.21.15.16:0/1436559821 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc96400e210 con 0x7fc964007390 2023-05-22T15:39:21.618 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:21.611+0000 7fc98159a640 1 -- 172.21.15.16:0/1436559821 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc96400e3d0 con 0x7fc9640069e0 2023-05-22T15:39:21.618 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:21.611+0000 7fc98259c640 1 --2- 172.21.15.16:0/1436559821 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc964005b90 0x7fc964005fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:21.618 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:21.611+0000 7fc982d9d640 1 --2- 172.21.15.16:0/1436559821 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc964007390 0x7fc96400dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:21.618 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:21.611+0000 7fc981d9b640 1 --2- 172.21.15.16:0/1436559821 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc9640069e0 0x7fc964006e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:21.618 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:21.611+0000 7fc98259c640 1 -- 172.21.15.16:0/1436559821 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc9640069e0 msgr2=0x7fc964006e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:39:21.618 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:21.611+0000 7fc98259c640 1 --2- 172.21.15.16:0/1436559821 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc9640069e0 0x7fc964006e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:21.619 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:21.611+0000 7fc98259c640 1 -- 172.21.15.16:0/1436559821 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc964007390 msgr2=0x7fc96400dba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:39:21.619 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:21.611+0000 7fc98259c640 1 --2- 172.21.15.16:0/1436559821 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc964007390 0x7fc96400dba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:21.619 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:21.611+0000 7fc98259c640 1 -- 172.21.15.16:0/1436559821 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc964005740 con 0x7fc964005b90 2023-05-22T15:39:21.619 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:21.611+0000 7fc982d9d640 1 -- 172.21.15.16:0/1436559821 reap_dead start 2023-05-22T15:39:21.619 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:21.611+0000 7fc981d9b640 1 --2- 172.21.15.16:0/1436559821 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc9640069e0 0x7fc964006e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:39:21.622 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:21.611+0000 7fc98259c640 1 --2- 172.21.15.16:0/1436559821 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc964005b90 0x7fc964005fa0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7fc96c00c240 tx=0x7fc96c01cc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=648019d31a401f53 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:39:21.622 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:21.611+0000 7fc98159a640 1 -- 172.21.15.16:0/1436559821 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc96c005ae0 con 0x7fc964005b90 2023-05-22T15:39:21.622 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:21.611+0000 7fc98159a640 1 -- 172.21.15.16:0/1436559821 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc96c005c60 con 0x7fc964005b90 2023-05-22T15:39:21.622 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:21.611+0000 7fc98159a640 1 -- 172.21.15.16:0/1436559821 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc96c007070 con 0x7fc964005b90 2023-05-22T15:39:21.623 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:21.611+0000 7fc988a5e640 1 -- 172.21.15.16:0/1436559821 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc964005b90 msgr2=0x7fc964005fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:21.623 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:21.611+0000 7fc988a5e640 1 --2- 172.21.15.16:0/1436559821 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc964005b90 0x7fc964005fa0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7fc96c00c240 tx=0x7fc96c01cc10 comp rx=0 tx=0).stop 2023-05-22T15:39:21.623 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:21.611+0000 7fc988a5e640 1 -- 172.21.15.16:0/1436559821 shutdown_connections 2023-05-22T15:39:21.623 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:21.611+0000 7fc988a5e640 1 --2- 172.21.15.16:0/1436559821 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc964005b90 0x7fc964005fa0 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:21.623 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:21.611+0000 7fc988a5e640 1 -- 172.21.15.16:0/1436559821 >> 172.21.15.16:0/1436559821 conn(0x7fc9840b0cc0 msgr2=0x7fc9840b10c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:39:21.623 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:21.611+0000 7fc988a5e640 1 -- 172.21.15.16:0/1436559821 shutdown_connections 2023-05-22T15:39:21.624 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:21.611+0000 7fc988a5e640 1 -- 172.21.15.16:0/1436559821 wait complete. 2023-05-22T15:39:21.624 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:21.615+0000 7fc988a5e640 1 Processor -- start 2023-05-22T15:39:21.624 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:21.615+0000 7fc988a5e640 1 -- start start 2023-05-22T15:39:21.624 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:21.615+0000 7fc988a5e640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc964005b90 0x7fc9841578d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:21.624 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:21.615+0000 7fc988a5e640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc9840b4d70 0x7fc9840c6200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:21.624 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:21.615+0000 7fc988a5e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc9841f46d0 0x7fc9840b2af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:21.624 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:21.615+0000 7fc988a5e640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9840c6b70 con 0x7fc9841f46d0 2023-05-22T15:39:21.625 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:21.615+0000 7fc988a5e640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9840c6e50 con 0x7fc9840b4d70 2023-05-22T15:39:21.625 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:21.615+0000 7fc988a5e640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9840c7130 con 0x7fc964005b90 2023-05-22T15:39:21.625 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:21.615+0000 7fc982d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc9841f46d0 0x7fc9840b2af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:21.625 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:21.615+0000 7fc982d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc9841f46d0 0x7fc9840b2af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52288/0 (socket says 172.21.15.16:52288) 2023-05-22T15:39:21.626 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:21.615+0000 7fc982d9d640 1 -- 172.21.15.16:0/3529472995 learned_addr learned my addr 172.21.15.16:0/3529472995 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:39:21.626 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:21.615+0000 7fc981d9b640 1 --2- 172.21.15.16:0/3529472995 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc9840b4d70 0x7fc9840c6200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:21.626 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:21.615+0000 7fc98259c640 1 --2- 172.21.15.16:0/3529472995 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc964005b90 0x7fc9841578d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:21.626 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:21.615+0000 7fc982d9d640 1 -- 172.21.15.16:0/3529472995 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc964005b90 msgr2=0x7fc9841578d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:21.626 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:21.615+0000 7fc982d9d640 1 --2- 172.21.15.16:0/3529472995 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc964005b90 0x7fc9841578d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:21.626 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:21.615+0000 7fc982d9d640 1 -- 172.21.15.16:0/3529472995 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc9840b4d70 msgr2=0x7fc9840c6200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:21.626 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:21.615+0000 7fc982d9d640 1 --2- 172.21.15.16:0/3529472995 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc9840b4d70 0x7fc9840c6200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:21.627 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:21.615+0000 7fc982d9d640 1 -- 172.21.15.16:0/3529472995 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9840707f0 con 0x7fc9841f46d0 2023-05-22T15:39:21.627 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:21.615+0000 7fc98259c640 1 --2- 172.21.15.16:0/3529472995 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc964005b90 0x7fc9841578d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T15:39:21.627 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:21.615+0000 7fc982d9d640 1 --2- 172.21.15.16:0/3529472995 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc9841f46d0 0x7fc9840b2af0 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7fc9840ae5e0 tx=0x7fc97400e4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:39:21.627 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:21.615+0000 7fc97affd640 1 -- 172.21.15.16:0/3529472995 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc974005320 con 0x7fc9841f46d0 2023-05-22T15:39:21.627 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:21.615+0000 7fc97affd640 1 -- 172.21.15.16:0/3529472995 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9740054a0 con 0x7fc9841f46d0 2023-05-22T15:39:21.627 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:21.615+0000 7fc97affd640 1 -- 172.21.15.16:0/3529472995 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc974007270 con 0x7fc9841f46d0 2023-05-22T15:39:21.627 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:21.615+0000 7fc988a5e640 1 -- 172.21.15.16:0/3529472995 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9840cbd10 con 0x7fc9841f46d0 2023-05-22T15:39:21.627 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:21.615+0000 7fc988a5e640 1 -- 172.21.15.16:0/3529472995 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc984070980 con 0x7fc9841f46d0 2023-05-22T15:39:21.628 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:21.615+0000 7fc988a5e640 1 -- 172.21.15.16:0/3529472995 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc98404dbe0 con 0x7fc9841f46d0 2023-05-22T15:39:21.628 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:21.619+0000 7fc97affd640 1 -- 172.21.15.16:0/3529472995 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 12) v1 ==== 84418+0+0 (secure 0 0 0) 0x7fc9740073d0 con 0x7fc9841f46d0 2023-05-22T15:39:21.628 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:21.619+0000 7fc97affd640 1 --2- 172.21.15.16:0/3529472995 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fc968065580 0x7fc968067a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:21.629 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:21.619+0000 7fc97affd640 1 -- 172.21.15.16:0/3529472995 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 6698+0+0 (secure 0 0 0) 0x7fc974081780 con 0x7fc9841f46d0 2023-05-22T15:39:21.629 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:21.619+0000 7fc98259c640 1 --2- 172.21.15.16:0/3529472995 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fc968065580 0x7fc968067a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:21.629 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:21.619+0000 7fc98259c640 1 --2- 172.21.15.16:0/3529472995 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fc968065580 0x7fc968067a30 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fc96c0ab000 tx=0x7fc96c0b1040 comp rx=0 tx=0).ready entity=mgr.5823 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:39:21.629 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:21.623+0000 7fc97affd640 1 -- 172.21.15.16:0/3529472995 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fc97404b620 con 0x7fc9841f46d0 2023-05-22T15:39:21.734 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-05-22T15:39:21.737 DEBUG:teuthology.orchestra.run.smithi062:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-05-22T15:39:21.803 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:21.799+0000 7fc988a5e640 1 -- 172.21.15.16:0/3529472995 --> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "fs": "cephfs", "path": "/snap_test_dir1/dir3", "repeat": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7fc9840b18a0 con 0x7fc968065580 2023-05-22T15:39:21.943 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:21.939+0000 7fc97affd640 1 -- 172.21.15.16:0/3529472995 <== mgr.5823 v2:172.21.15.16:6832/36921 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7fc9840b18a0 con 0x7fc968065580 2023-05-22T15:39:21.943 INFO:teuthology.orchestra.run.smithi016.stdout:Schedule deactivated for path /snap_test_dir1/dir3 2023-05-22T15:39:21.945 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:21.939+0000 7fc988a5e640 1 -- 172.21.15.16:0/3529472995 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fc968065580 msgr2=0x7fc968067a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:21.946 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:21.939+0000 7fc988a5e640 1 --2- 172.21.15.16:0/3529472995 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fc968065580 0x7fc968067a30 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fc96c0ab000 tx=0x7fc96c0b1040 comp rx=0 tx=0).stop 2023-05-22T15:39:21.946 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:21.939+0000 7fc988a5e640 1 -- 172.21.15.16:0/3529472995 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc9841f46d0 msgr2=0x7fc9840b2af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:21.946 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:21.939+0000 7fc988a5e640 1 --2- 172.21.15.16:0/3529472995 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc9841f46d0 0x7fc9840b2af0 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7fc9840ae5e0 tx=0x7fc97400e4d0 comp rx=0 tx=0).stop 2023-05-22T15:39:21.946 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:21.939+0000 7fc988a5e640 1 -- 172.21.15.16:0/3529472995 shutdown_connections 2023-05-22T15:39:21.946 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:21.939+0000 7fc988a5e640 1 --2- 172.21.15.16:0/3529472995 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc9841f46d0 0x7fc9840b2af0 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:21.946 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:21.939+0000 7fc988a5e640 1 --2- 172.21.15.16:0/3529472995 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc9840b4d70 0x7fc9840c6200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:21.946 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:21.939+0000 7fc988a5e640 1 --2- 172.21.15.16:0/3529472995 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fc968065580 0x7fc968067a30 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:21.947 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:21.939+0000 7fc988a5e640 1 --2- 172.21.15.16:0/3529472995 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc964005b90 0x7fc9841578d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:21.947 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:21.939+0000 7fc988a5e640 1 -- 172.21.15.16:0/3529472995 >> 172.21.15.16:0/3529472995 conn(0x7fc9840b0cc0 msgr2=0x7fc9840caa80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:39:21.947 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:21.939+0000 7fc988a5e640 1 -- 172.21.15.16:0/3529472995 shutdown_connections 2023-05-22T15:39:21.947 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:21.939+0000 7fc988a5e640 1 -- 172.21.15.16:0/3529472995 wait complete. 2023-05-22T15:39:21.965 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/dir3 2023-05-22T15:39:21.966 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate --fs cephfs /snap_test_dir1/dir4 1M 2023-05-22T15:39:22.141 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:22.135+0000 7fd2d2933640 1 Processor -- start 2023-05-22T15:39:22.141 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:22.135+0000 7fd2d2933640 1 -- start start 2023-05-22T15:39:22.142 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:22.135+0000 7fd2d2933640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd2cc15bd90 0x7fd2cc0ce9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:22.142 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:22.135+0000 7fd2d2933640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fd2cc0cb1c0 0x7fd2cc15f8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:22.142 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:22.135+0000 7fd2d2933640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2cc0c4f90 con 0x7fd2cc0c4a40 2023-05-22T15:39:22.142 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:22.135+0000 7fd2d2933640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2cc06c700 con 0x7fd2cc0cb1c0 2023-05-22T15:39:22.142 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:22.135+0000 7fd2d2933640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2cc0ad450 con 0x7fd2cc15bd90 2023-05-22T15:39:22.143 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:22.139+0000 7fd2d0ed0640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fd2cc0cb1c0 0x7fd2cc15f8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:22.143 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:22.139+0000 7fd2cbfff640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd2cc15bd90 0x7fd2cc0ce9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:22.143 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:22.139+0000 7fd2d0ed0640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fd2cc0cb1c0 0x7fd2cc15f8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.16:34846/0 (socket says 172.21.15.16:34846) 2023-05-22T15:39:22.143 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:22.139+0000 7fd2d0ed0640 1 -- 172.21.15.16:0/146729938 learned_addr learned my addr 172.21.15.16:0/146729938 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:39:22.143 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:22.139+0000 7fd2d0ed0640 1 -- 172.21.15.16:0/146729938 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd2cc15bd90 msgr2=0x7fd2cc0ce9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:39:22.144 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:22.139+0000 7fd2d0ed0640 1 --2- 172.21.15.16:0/146729938 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd2cc15bd90 0x7fd2cc0ce9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:22.144 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:22.139+0000 7fd2d0ed0640 1 -- 172.21.15.16:0/146729938 >> v1:172.21.15.62:6789/0 conn(0x7fd2cc0c4a40 legacy=0x7fd2cc0caab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:22.144 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:22.139+0000 7fd2d0ed0640 1 -- 172.21.15.16:0/146729938 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2cc0abb10 con 0x7fd2cc0cb1c0 2023-05-22T15:39:22.145 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:22.139+0000 7fd2d0ed0640 1 --2- 172.21.15.16:0/146729938 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd2cc0cb1c0 0x7fd2cc15f8d0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fd2c4009870 tx=0x7fd2c401c540 comp rx=0 tx=0).ready entity=mon.1 client_cookie=54fc7aa943c18f45 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:39:22.145 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:22.139+0000 7fd2caffd640 1 -- 172.21.15.16:0/146729938 <== mon.1 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2c40ab600 con 0x7fd2cc0cb1c0 2023-05-22T15:39:22.146 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:22.139+0000 7fd2caffd640 1 -- 172.21.15.16:0/146729938 <== mon.1 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2c40a7050 con 0x7fd2cc0cb1c0 2023-05-22T15:39:22.146 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:22.139+0000 7fd2caffd640 1 -- 172.21.15.16:0/146729938 <== mon.1 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2c40ab980 con 0x7fd2cc0cb1c0 2023-05-22T15:39:22.146 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:22.143+0000 7fd2d2933640 1 -- 172.21.15.16:0/146729938 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd2cc0cb1c0 msgr2=0x7fd2cc15f8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:22.147 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:22.143+0000 7fd2d2933640 1 --2- 172.21.15.16:0/146729938 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd2cc0cb1c0 0x7fd2cc15f8d0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fd2c4009870 tx=0x7fd2c401c540 comp rx=0 tx=0).stop 2023-05-22T15:39:22.147 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:22.143+0000 7fd2d2933640 1 -- 172.21.15.16:0/146729938 shutdown_connections 2023-05-22T15:39:22.147 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:22.143+0000 7fd2d2933640 1 --2- 172.21.15.16:0/146729938 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd2cc0cb1c0 0x7fd2cc15f8d0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:22.147 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:22.143+0000 7fd2d2933640 1 --2- 172.21.15.16:0/146729938 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd2cc15bd90 0x7fd2cc0ce9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:22.147 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:22.143+0000 7fd2d2933640 1 -- 172.21.15.16:0/146729938 >> 172.21.15.16:0/146729938 conn(0x7fd2cc069720 msgr2=0x7fd2cc069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:39:22.147 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:22.143+0000 7fd2d2933640 1 -- 172.21.15.16:0/146729938 shutdown_connections 2023-05-22T15:39:22.147 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:22.143+0000 7fd2d2933640 1 -- 172.21.15.16:0/146729938 wait complete. 2023-05-22T15:39:22.148 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:22.143+0000 7fd2d2933640 1 Processor -- start 2023-05-22T15:39:22.148 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:22.143+0000 7fd2d2933640 1 -- start start 2023-05-22T15:39:22.149 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:22.143+0000 7fd2d2933640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd2cc0c4a40 0x7fd2cc159020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:22.149 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:22.143+0000 7fd2d2933640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd2cc0cb1c0 0x7fd2cc14da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:22.149 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:22.143+0000 7fd2d2933640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd2cc15bd90 0x7fd2cc15a9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:22.149 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:22.143+0000 7fd2d2933640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2cc15b1f0 con 0x7fd2cc15bd90 2023-05-22T15:39:22.149 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:22.143+0000 7fd2d2933640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2cc14dfa0 con 0x7fd2cc0cb1c0 2023-05-22T15:39:22.150 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:22.143+0000 7fd2d2933640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2cc15af30 con 0x7fd2cc0c4a40 2023-05-22T15:39:22.151 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:22.143+0000 7fd2d0ed0640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd2cc15bd90 0x7fd2cc15a9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:22.151 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:22.143+0000 7fd2cbfff640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd2cc0c4a40 0x7fd2cc159020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:22.151 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:22.143+0000 7fd2cbfff640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd2cc0c4a40 0x7fd2cc159020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3301/0 says I am v2:172.21.15.16:58194/0 (socket says 172.21.15.16:58194) 2023-05-22T15:39:22.151 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:22.143+0000 7fd2d0ed0640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd2cc15bd90 0x7fd2cc15a9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52300/0 (socket says 172.21.15.16:52300) 2023-05-22T15:39:22.151 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:22.143+0000 7fd2cbfff640 1 -- 172.21.15.16:0/1073039038 learned_addr learned my addr 172.21.15.16:0/1073039038 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:39:22.151 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:22.143+0000 7fd2cb7fe640 1 --2- 172.21.15.16:0/1073039038 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd2cc0cb1c0 0x7fd2cc14da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:22.152 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:22.143+0000 7fd2d0ed0640 1 -- 172.21.15.16:0/1073039038 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd2cc0c4a40 msgr2=0x7fd2cc159020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:22.152 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:22.143+0000 7fd2d0ed0640 1 --2- 172.21.15.16:0/1073039038 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd2cc0c4a40 0x7fd2cc159020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:22.152 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:22.143+0000 7fd2d0ed0640 1 -- 172.21.15.16:0/1073039038 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd2cc0cb1c0 msgr2=0x7fd2cc14da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:22.152 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:22.143+0000 7fd2d0ed0640 1 --2- 172.21.15.16:0/1073039038 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd2cc0cb1c0 0x7fd2cc14da60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:22.152 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:22.143+0000 7fd2d0ed0640 1 -- 172.21.15.16:0/1073039038 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2c40094d0 con 0x7fd2cc15bd90 2023-05-22T15:39:22.152 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:22.147+0000 7fd2d0ed0640 1 --2- 172.21.15.16:0/1073039038 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd2cc15bd90 0x7fd2cc15a9f0 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7fd2cc06dc10 tx=0x7fd2c40192e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:39:22.152 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:22.147+0000 7fd2c8ff9640 1 -- 172.21.15.16:0/1073039038 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2c40b6640 con 0x7fd2cc15bd90 2023-05-22T15:39:22.153 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:22.147+0000 7fd2c8ff9640 1 -- 172.21.15.16:0/1073039038 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2c40b67a0 con 0x7fd2cc15bd90 2023-05-22T15:39:22.153 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:22.147+0000 7fd2d2933640 1 -- 172.21.15.16:0/1073039038 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2cc0ad8c0 con 0x7fd2cc15bd90 2023-05-22T15:39:22.153 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:22.147+0000 7fd2c8ff9640 1 -- 172.21.15.16:0/1073039038 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2c40b6b30 con 0x7fd2cc15bd90 2023-05-22T15:39:22.153 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:22.147+0000 7fd2d2933640 1 -- 172.21.15.16:0/1073039038 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2cc0ad120 con 0x7fd2cc15bd90 2023-05-22T15:39:22.154 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:22.147+0000 7fd2d2933640 1 -- 172.21.15.16:0/1073039038 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2cc0ada50 con 0x7fd2cc15bd90 2023-05-22T15:39:22.154 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:22.147+0000 7fd2c8ff9640 1 -- 172.21.15.16:0/1073039038 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 12) v1 ==== 84418+0+0 (secure 0 0 0) 0x7fd2c40b2040 con 0x7fd2cc15bd90 2023-05-22T15:39:22.154 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:22.147+0000 7fd2c8ff9640 1 --2- 172.21.15.16:0/1073039038 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd2b8065580 0x7fd2b8067a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:22.154 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:22.147+0000 7fd2c8ff9640 1 -- 172.21.15.16:0/1073039038 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 6698+0+0 (secure 0 0 0) 0x7fd2c411c9a0 con 0x7fd2cc15bd90 2023-05-22T15:39:22.154 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:22.147+0000 7fd2cbfff640 1 --2- 172.21.15.16:0/1073039038 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd2b8065580 0x7fd2b8067a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:22.154 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:22.147+0000 7fd2cbfff640 1 --2- 172.21.15.16:0/1073039038 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd2b8065580 0x7fd2b8067a30 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fd2bc009b80 tx=0x7fd2bc009190 comp rx=0 tx=0).ready entity=mgr.5823 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:39:22.156 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:22.151+0000 7fd2c8ff9640 1 -- 172.21.15.16:0/1073039038 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fd2cc0ada50 con 0x7fd2cc15bd90 2023-05-22T15:39:22.331 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:22.327+0000 7fd2d2933640 1 -- 172.21.15.16:0/1073039038 --> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "fs": "cephfs", "path": "/snap_test_dir1/dir4", "repeat": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7fd2cc0ada50 con 0x7fd2b8065580 2023-05-22T15:39:22.484 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:22.479+0000 7fd2c8ff9640 1 -- 172.21.15.16:0/1073039038 <== mgr.5823 v2:172.21.15.16:6832/36921 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7fd2cc0ada50 con 0x7fd2b8065580 2023-05-22T15:39:22.485 INFO:teuthology.orchestra.run.smithi016.stdout:Schedule deactivated for path /snap_test_dir1/dir4 2023-05-22T15:39:22.487 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:22.483+0000 7fd2d2933640 1 -- 172.21.15.16:0/1073039038 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd2b8065580 msgr2=0x7fd2b8067a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:22.487 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:22.483+0000 7fd2d2933640 1 --2- 172.21.15.16:0/1073039038 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd2b8065580 0x7fd2b8067a30 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fd2bc009b80 tx=0x7fd2bc009190 comp rx=0 tx=0).stop 2023-05-22T15:39:22.487 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:22.483+0000 7fd2d2933640 1 -- 172.21.15.16:0/1073039038 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd2cc15bd90 msgr2=0x7fd2cc15a9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:22.487 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:22.483+0000 7fd2d2933640 1 --2- 172.21.15.16:0/1073039038 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd2cc15bd90 0x7fd2cc15a9f0 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7fd2cc06dc10 tx=0x7fd2c40192e0 comp rx=0 tx=0).stop 2023-05-22T15:39:22.487 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:22.483+0000 7fd2d2933640 1 -- 172.21.15.16:0/1073039038 shutdown_connections 2023-05-22T15:39:22.488 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:22.483+0000 7fd2d2933640 1 --2- 172.21.15.16:0/1073039038 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd2cc15bd90 0x7fd2cc15a9f0 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:22.488 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:22.483+0000 7fd2d2933640 1 --2- 172.21.15.16:0/1073039038 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd2cc0cb1c0 0x7fd2cc14da60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:22.488 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:22.483+0000 7fd2d2933640 1 --2- 172.21.15.16:0/1073039038 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd2b8065580 0x7fd2b8067a30 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:22.488 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:22.483+0000 7fd2d2933640 1 --2- 172.21.15.16:0/1073039038 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd2cc0c4a40 0x7fd2cc159020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:22.488 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:22.483+0000 7fd2d2933640 1 -- 172.21.15.16:0/1073039038 >> 172.21.15.16:0/1073039038 conn(0x7fd2cc069720 msgr2=0x7fd2cc06f670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:39:22.488 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:22.483+0000 7fd2d2933640 1 -- 172.21.15.16:0/1073039038 shutdown_connections 2023-05-22T15:39:22.488 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:22.483+0000 7fd2d2933640 1 -- 172.21.15.16:0/1073039038 wait complete. 2023-05-22T15:39:22.507 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/dir4 2023-05-22T15:39:22.508 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate --fs cephfs /snap_test_dir1/dir5 1M 2023-05-22T15:39:22.638 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:22.631+0000 7f471e024640 1 Processor -- start 2023-05-22T15:39:22.638 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:22.635+0000 7f471e024640 1 -- start start 2023-05-22T15:39:22.639 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:22.635+0000 7f471e024640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f4718159a30 0x7f471806e380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:22.639 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:22.635+0000 7f471e024640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f47180cf570 0x7f47180cf990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:22.639 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:22.635+0000 7f471e024640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47180cde70 con 0x7f471815d610 2023-05-22T15:39:22.640 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:22.635+0000 7f471e024640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47180cc520 con 0x7f47180cf570 2023-05-22T15:39:22.640 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:22.635+0000 7f471e024640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47180cfed0 con 0x7f4718159a30 2023-05-22T15:39:22.640 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:22.635+0000 7f4717fff640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f47180cf570 0x7f47180cf990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:22.640 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:22.635+0000 7f4717fff640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f47180cf570 0x7f47180cf990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52314/0 (socket says 172.21.15.16:52314) 2023-05-22T15:39:22.640 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:22.635+0000 7f4717fff640 1 -- 172.21.15.16:0/1729912269 learned_addr learned my addr 172.21.15.16:0/1729912269 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:39:22.641 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:22.635+0000 7f47177fe640 1 --2- 172.21.15.16:0/1729912269 >> v2:172.21.15.62:3300/0 conn(0x7f4718159a30 0x7f471806e380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:22.641 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:22.635+0000 7f4717fff640 1 -- 172.21.15.16:0/1729912269 >> v2:172.21.15.62:3300/0 conn(0x7f4718159a30 msgr2=0x7f471806e380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:39:22.641 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:22.635+0000 7f4717fff640 1 --2- 172.21.15.16:0/1729912269 >> v2:172.21.15.62:3300/0 conn(0x7f4718159a30 0x7f471806e380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:22.641 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:22.635+0000 7f4717fff640 1 -- 172.21.15.16:0/1729912269 >> v1:172.21.15.62:6789/0 conn(0x7f471815d610 legacy=0x7f47180cbe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:39:22.641 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:22.635+0000 7f4717fff640 1 -- 172.21.15.16:0/1729912269 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f471815db20 con 0x7f47180cf570 2023-05-22T15:39:22.642 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:22.639+0000 7f4717fff640 1 --2- 172.21.15.16:0/1729912269 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f47180cf570 0x7f47180cf990 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f4710009e50 tx=0x7f471001c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=292819636a95eec7 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:39:22.642 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:22.639+0000 7f47167fc640 1 -- 172.21.15.16:0/1729912269 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47100a5830 con 0x7f47180cf570 2023-05-22T15:39:22.642 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:22.639+0000 7f47167fc640 1 -- 172.21.15.16:0/1729912269 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47100a5990 con 0x7f47180cf570 2023-05-22T15:39:22.643 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:22.639+0000 7f47167fc640 1 -- 172.21.15.16:0/1729912269 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47100a5d20 con 0x7f47180cf570 2023-05-22T15:39:22.643 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:22.639+0000 7f471e024640 1 -- 172.21.15.16:0/1729912269 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f47180cf570 msgr2=0x7f47180cf990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:22.643 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:22.639+0000 7f471e024640 1 --2- 172.21.15.16:0/1729912269 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f47180cf570 0x7f47180cf990 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f4710009e50 tx=0x7f471001c970 comp rx=0 tx=0).stop 2023-05-22T15:39:22.644 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:22.639+0000 7f471e024640 1 -- 172.21.15.16:0/1729912269 shutdown_connections 2023-05-22T15:39:22.644 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:22.639+0000 7f471e024640 1 --2- 172.21.15.16:0/1729912269 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f47180cf570 0x7f47180cf990 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:22.644 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:22.639+0000 7f471e024640 1 --2- 172.21.15.16:0/1729912269 >> v2:172.21.15.62:3300/0 conn(0x7f4718159a30 0x7f471806e380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:22.644 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:22.639+0000 7f471e024640 1 -- 172.21.15.16:0/1729912269 >> 172.21.15.16:0/1729912269 conn(0x7f47180700b0 msgr2=0x7f471806d9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:39:22.644 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:22.639+0000 7f471e024640 1 -- 172.21.15.16:0/1729912269 shutdown_connections 2023-05-22T15:39:22.644 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:22.639+0000 7f471e024640 1 -- 172.21.15.16:0/1729912269 wait complete. 2023-05-22T15:39:22.645 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:22.639+0000 7f471e024640 1 Processor -- start 2023-05-22T15:39:22.645 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:22.639+0000 7f471e024640 1 -- start start 2023-05-22T15:39:22.646 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:22.639+0000 7f471e024640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f47180cf570 0x7f4718151190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:22.646 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:22.639+0000 7f471e024640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4718159a30 0x7f47180c3db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:22.646 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:22.639+0000 7f471e024640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f471815d610 0x7f47180c93e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:22.646 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:22.639+0000 7f471e024640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47180c6900 con 0x7f4718159a30 2023-05-22T15:39:22.646 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:22.639+0000 7f471e024640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47180c6d20 con 0x7f471815d610 2023-05-22T15:39:22.646 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:22.639+0000 7f471e024640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47180c5c80 con 0x7f47180cf570 2023-05-22T15:39:22.646 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:22.639+0000 7f4716ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4718159a30 0x7f47180c3db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:22.646 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:22.639+0000 7f4716ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4718159a30 0x7f47180c3db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52320/0 (socket says 172.21.15.16:52320) 2023-05-22T15:39:22.647 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:22.639+0000 7f4716ffd640 1 -- 172.21.15.16:0/563381841 learned_addr learned my addr 172.21.15.16:0/563381841 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:39:22.648 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:22.639+0000 7f47177fe640 1 --2- 172.21.15.16:0/563381841 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f47180cf570 0x7f4718151190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:22.648 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:22.639+0000 7f4717fff640 1 --2- 172.21.15.16:0/563381841 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f471815d610 0x7f47180c93e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:22.648 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:22.639+0000 7f4716ffd640 1 -- 172.21.15.16:0/563381841 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f47180cf570 msgr2=0x7f4718151190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:22.648 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:22.639+0000 7f4716ffd640 1 --2- 172.21.15.16:0/563381841 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f47180cf570 0x7f4718151190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:22.649 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:22.639+0000 7f4716ffd640 1 -- 172.21.15.16:0/563381841 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f471815d610 msgr2=0x7f47180c93e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:22.649 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:22.639+0000 7f4716ffd640 1 --2- 172.21.15.16:0/563381841 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f471815d610 0x7f47180c93e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:22.649 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:22.639+0000 7f4716ffd640 1 -- 172.21.15.16:0/563381841 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4710009b30 con 0x7f4718159a30 2023-05-22T15:39:22.649 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:22.639+0000 7f4717fff640 1 --2- 172.21.15.16:0/563381841 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f471815d610 0x7f47180c93e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:39:22.649 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:22.639+0000 7f4716ffd640 1 --2- 172.21.15.16:0/563381841 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4718159a30 0x7f47180c3db0 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f470800c410 tx=0x7f470800c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:39:22.649 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:22.643+0000 7f46fffff640 1 -- 172.21.15.16:0/563381841 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4708011790 con 0x7f4718159a30 2023-05-22T15:39:22.649 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:22.643+0000 7f46fffff640 1 -- 172.21.15.16:0/563381841 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47080118f0 con 0x7f4718159a30 2023-05-22T15:39:22.649 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:22.643+0000 7f471e024640 1 -- 172.21.15.16:0/563381841 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f471815e3a0 con 0x7f4718159a30 2023-05-22T15:39:22.650 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:22.643+0000 7f46fffff640 1 -- 172.21.15.16:0/563381841 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4708011ce0 con 0x7f4718159a30 2023-05-22T15:39:22.650 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:22.643+0000 7f471e024640 1 -- 172.21.15.16:0/563381841 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f471815e0c0 con 0x7f4718159a30 2023-05-22T15:39:22.650 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:22.643+0000 7f471e024640 1 -- 172.21.15.16:0/563381841 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4718159e00 con 0x7f4718159a30 2023-05-22T15:39:22.650 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:22.643+0000 7f46fffff640 1 -- 172.21.15.16:0/563381841 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 12) v1 ==== 84418+0+0 (secure 0 0 0) 0x7f470801c460 con 0x7f4718159a30 2023-05-22T15:39:22.650 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:22.643+0000 7f46fffff640 1 --2- 172.21.15.16:0/563381841 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f4704065580 0x7f4704067a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:22.650 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:22.643+0000 7f46fffff640 1 -- 172.21.15.16:0/563381841 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 6698+0+0 (secure 0 0 0) 0x7f4708082b60 con 0x7f4718159a30 2023-05-22T15:39:22.650 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:22.643+0000 7f47177fe640 1 --2- 172.21.15.16:0/563381841 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f4704065580 0x7f4704067a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:22.651 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:22.643+0000 7f47177fe640 1 --2- 172.21.15.16:0/563381841 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f4704065580 0x7f4704067a30 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f471806d9a0 tx=0x7f470c008040 comp rx=0 tx=0).ready entity=mgr.5823 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:39:22.652 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:22.647+0000 7f46fffff640 1 -- 172.21.15.16:0/563381841 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f4708050310 con 0x7f4718159a30 2023-05-22T15:39:22.827 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:22.823+0000 7f471e024640 1 -- 172.21.15.16:0/563381841 --> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "fs": "cephfs", "path": "/snap_test_dir1/dir5", "repeat": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f47180c6a40 con 0x7f4704065580 2023-05-22T15:39:22.979 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:22.975+0000 7f46fffff640 1 -- 172.21.15.16:0/563381841 <== mgr.5823 v2:172.21.15.16:6832/36921 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f47180c6a40 con 0x7f4704065580 2023-05-22T15:39:22.980 INFO:teuthology.orchestra.run.smithi016.stdout:Schedule deactivated for path /snap_test_dir1/dir5 2023-05-22T15:39:22.983 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:22.979+0000 7f471e024640 1 -- 172.21.15.16:0/563381841 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f4704065580 msgr2=0x7f4704067a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:22.984 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:22.979+0000 7f471e024640 1 --2- 172.21.15.16:0/563381841 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f4704065580 0x7f4704067a30 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f471806d9a0 tx=0x7f470c008040 comp rx=0 tx=0).stop 2023-05-22T15:39:22.984 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:22.979+0000 7f471e024640 1 -- 172.21.15.16:0/563381841 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4718159a30 msgr2=0x7f47180c3db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:22.984 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:22.979+0000 7f471e024640 1 --2- 172.21.15.16:0/563381841 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4718159a30 0x7f47180c3db0 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f470800c410 tx=0x7f470800c8d0 comp rx=0 tx=0).stop 2023-05-22T15:39:22.985 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:22.979+0000 7f471e024640 1 -- 172.21.15.16:0/563381841 shutdown_connections 2023-05-22T15:39:22.985 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:22.979+0000 7f471e024640 1 --2- 172.21.15.16:0/563381841 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f471815d610 0x7f47180c93e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:22.985 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:22.979+0000 7f471e024640 1 --2- 172.21.15.16:0/563381841 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4718159a30 0x7f47180c3db0 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:22.986 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:22.979+0000 7f471e024640 1 --2- 172.21.15.16:0/563381841 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f4704065580 0x7f4704067a30 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:22.986 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:22.979+0000 7f471e024640 1 --2- 172.21.15.16:0/563381841 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f47180cf570 0x7f4718151190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:22.986 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:22.979+0000 7f471e024640 1 -- 172.21.15.16:0/563381841 >> 172.21.15.16:0/563381841 conn(0x7f47180700b0 msgr2=0x7f47180b30d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:39:22.987 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:22.979+0000 7f471e024640 1 -- 172.21.15.16:0/563381841 shutdown_connections 2023-05-22T15:39:22.987 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:22.979+0000 7f471e024640 1 -- 172.21.15.16:0/563381841 wait complete. 2023-05-22T15:39:23.006 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/dir5 2023-05-22T15:39:23.007 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate --fs cephfs /snap_test_dir1/dir6 1M 2023-05-22T15:39:23.142 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:23.135+0000 7f1b0b7d6640 1 Processor -- start 2023-05-22T15:39:23.143 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:23.135+0000 7f1b0b7d6640 1 -- start start 2023-05-22T15:39:23.151 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:23.139+0000 7f1b0b7d6640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1b04159a30 0x7f1b0406e380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:23.151 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:23.139+0000 7f1b0b7d6640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b040cde70 con 0x7f1b040cf570 2023-05-22T15:39:23.151 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:23.139+0000 7f1b0b7d6640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b0415db20 con 0x7f1b0415d610 2023-05-22T15:39:23.151 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:23.139+0000 7f1b0b7d6640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b040cc520 con 0x7f1b04159a30 2023-05-22T15:39:23.151 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:23.139+0000 7f1b09d73640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f1b040cf570 0x7f1b040ce300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:53010/0 (socket says 172.21.15.16:53010) 2023-05-22T15:39:23.151 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:23.139+0000 7f1b09d73640 1 -- 172.21.15.16:0/3716067709 learned_addr learned my addr 172.21.15.16:0/3716067709 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:39:23.152 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:23.139+0000 7f1b09572640 1 --2- 172.21.15.16:0/3716067709 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1b04159a30 0x7f1b0406e380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:23.152 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:23.139+0000 7f1afbfff640 1 -- 172.21.15.16:0/3716067709 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1102977623 0 0) 0x7f1b040cde70 con 0x7f1b040cf570 2023-05-22T15:39:23.152 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:23.139+0000 7f1afbfff640 1 -- 172.21.15.16:0/3716067709 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1af0003530 con 0x7f1b040cf570 2023-05-22T15:39:23.152 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:23.139+0000 7f1afbfff640 1 -- 172.21.15.16:0/3716067709 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1892818985 0 0) 0x7f1af0003530 con 0x7f1b040cf570 2023-05-22T15:39:23.152 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:23.139+0000 7f1afbfff640 1 -- 172.21.15.16:0/3716067709 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1b04159a30 msgr2=0x7f1b0406e380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:39:23.152 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:23.139+0000 7f1afbfff640 1 --2- 172.21.15.16:0/3716067709 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1b04159a30 0x7f1b0406e380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:23.152 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:23.139+0000 7f1afbfff640 1 -- 172.21.15.16:0/3716067709 >> v1:172.21.15.62:6789/0 conn(0x7f1b0415d610 legacy=0x7f1b040cbe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:23.153 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:23.139+0000 7f1afbfff640 1 -- 172.21.15.16:0/3716067709 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b0406e8c0 con 0x7f1b040cf570 2023-05-22T15:39:23.153 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:23.139+0000 7f1afbfff640 1 -- 172.21.15.16:0/3716067709 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7f1afc001a00 con 0x7f1b040cf570 2023-05-22T15:39:23.153 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:23.139+0000 7f1afbfff640 1 -- 172.21.15.16:0/3716067709 >> v1:172.21.15.16:6789/0 conn(0x7f1b040cf570 legacy=0x7f1b040ce300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:23.153 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:23.139+0000 7f1afbfff640 1 --2- 172.21.15.16:0/3716067709 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1af00039b0 0x7f1af0003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:23.153 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:23.139+0000 7f1afbfff640 1 --2- 172.21.15.16:0/3716067709 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1b04159a30 0x7f1af00047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:23.153 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:23.139+0000 7f1afbfff640 1 --2- 172.21.15.16:0/3716067709 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1af0004d10 0x7f1af0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:23.153 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:23.139+0000 7f1afbfff640 1 -- 172.21.15.16:0/3716067709 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b040cc520 con 0x7f1af0004d10 2023-05-22T15:39:23.154 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:23.139+0000 7f1afbfff640 1 -- 172.21.15.16:0/3716067709 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1af000ba70 con 0x7f1af00039b0 2023-05-22T15:39:23.154 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:23.139+0000 7f1afbfff640 1 -- 172.21.15.16:0/3716067709 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1af000bc30 con 0x7f1b04159a30 2023-05-22T15:39:23.154 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:23.139+0000 7f1b09d73640 1 --2- 172.21.15.16:0/3716067709 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1af0004d10 0x7f1af0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:23.154 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:23.139+0000 7f1b09572640 1 --2- 172.21.15.16:0/3716067709 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1af00039b0 0x7f1af0003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:23.154 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:23.139+0000 7f1b08d71640 1 --2- 172.21.15.16:0/3716067709 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1b04159a30 0x7f1af00047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:23.154 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:23.139+0000 7f1b09d73640 1 -- 172.21.15.16:0/3716067709 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1b04159a30 msgr2=0x7f1af00047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:39:23.154 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:23.139+0000 7f1b09d73640 1 --2- 172.21.15.16:0/3716067709 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1b04159a30 0x7f1af00047d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:23.154 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:23.139+0000 7f1b09d73640 1 -- 172.21.15.16:0/3716067709 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1af00039b0 msgr2=0x7f1af0003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:39:23.155 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:23.139+0000 7f1b09d73640 1 --2- 172.21.15.16:0/3716067709 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1af00039b0 0x7f1af0003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:23.155 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:23.139+0000 7f1b09d73640 1 -- 172.21.15.16:0/3716067709 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1af0003530 con 0x7f1af0004d10 2023-05-22T15:39:23.155 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:23.139+0000 7f1b09572640 1 --2- 172.21.15.16:0/3716067709 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1af00039b0 0x7f1af0003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:39:23.155 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:23.143+0000 7f1b09d73640 1 --2- 172.21.15.16:0/3716067709 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1af0004d10 0x7f1af0005180 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f1afc00a760 tx=0x7f1afc01d260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=486b06afb35d0e1c server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:39:23.155 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:23.143+0000 7f1afbfff640 1 -- 172.21.15.16:0/3716067709 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1afc0ad630 con 0x7f1af0004d10 2023-05-22T15:39:23.155 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:23.143+0000 7f1afbfff640 1 -- 172.21.15.16:0/3716067709 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1afc0a8070 con 0x7f1af0004d10 2023-05-22T15:39:23.156 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:23.143+0000 7f1afbfff640 1 -- 172.21.15.16:0/3716067709 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1afc0ada10 con 0x7f1af0004d10 2023-05-22T15:39:23.156 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:23.143+0000 7f1b0b7d6640 1 -- 172.21.15.16:0/3716067709 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1af0004d10 msgr2=0x7f1af0005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:23.156 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:23.143+0000 7f1b0b7d6640 1 --2- 172.21.15.16:0/3716067709 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1af0004d10 0x7f1af0005180 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f1afc00a760 tx=0x7f1afc01d260 comp rx=0 tx=0).stop 2023-05-22T15:39:23.156 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:23.143+0000 7f1b09d73640 1 -- 172.21.15.16:0/3716067709 reap_dead start 2023-05-22T15:39:23.156 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:23.143+0000 7f1b0b7d6640 1 -- 172.21.15.16:0/3716067709 shutdown_connections 2023-05-22T15:39:23.156 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:23.143+0000 7f1b0b7d6640 1 -- 172.21.15.16:0/3716067709 >> 172.21.15.16:0/3716067709 conn(0x7f1b040700b0 msgr2=0x7f1b0406d9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:39:23.156 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:23.143+0000 7f1b0b7d6640 1 -- 172.21.15.16:0/3716067709 shutdown_connections 2023-05-22T15:39:23.157 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:23.143+0000 7f1b0b7d6640 1 -- 172.21.15.16:0/3716067709 wait complete. 2023-05-22T15:39:23.157 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:23.143+0000 7f1b0b7d6640 1 Processor -- start 2023-05-22T15:39:23.157 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:23.143+0000 7f1b0b7d6640 1 -- start start 2023-05-22T15:39:23.157 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:23.143+0000 7f1b0b7d6640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1af0004d10 0x7f1b041f1a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:23.157 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:23.143+0000 7f1b0b7d6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1b040b4120 0x7f1b040b4590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:23.157 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:23.143+0000 7f1b0b7d6640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1b0414ee90 0x7f1b0414f300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:23.157 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:23.143+0000 7f1b0b7d6640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b0414f840 con 0x7f1b040b4120 2023-05-22T15:39:23.157 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:23.143+0000 7f1b0b7d6640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b040b4be0 con 0x7f1af0004d10 2023-05-22T15:39:23.158 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:23.143+0000 7f1b0b7d6640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b040b4ec0 con 0x7f1b0414ee90 2023-05-22T15:39:23.158 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:23.143+0000 7f1b08d71640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1b040b4120 0x7f1b040b4590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:23.158 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:23.143+0000 7f1b08d71640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1b040b4120 0x7f1b040b4590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52350/0 (socket says 172.21.15.16:52350) 2023-05-22T15:39:23.158 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:23.143+0000 7f1b08d71640 1 -- 172.21.15.16:0/3152116926 learned_addr learned my addr 172.21.15.16:0/3152116926 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:39:23.158 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:23.143+0000 7f1b09d73640 1 --2- 172.21.15.16:0/3152116926 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1b0414ee90 0x7f1b0414f300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:23.158 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:23.143+0000 7f1b09572640 1 --2- 172.21.15.16:0/3152116926 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1af0004d10 0x7f1b041f1a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:23.159 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:23.143+0000 7f1b08d71640 1 -- 172.21.15.16:0/3152116926 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1b0414ee90 msgr2=0x7f1b0414f300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:23.159 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:23.143+0000 7f1b08d71640 1 --2- 172.21.15.16:0/3152116926 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1b0414ee90 0x7f1b0414f300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:23.159 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:23.143+0000 7f1b08d71640 1 -- 172.21.15.16:0/3152116926 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1af0004d10 msgr2=0x7f1b041f1a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:23.159 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:23.143+0000 7f1b08d71640 1 --2- 172.21.15.16:0/3152116926 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1af0004d10 0x7f1b041f1a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:23.160 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:23.143+0000 7f1b08d71640 1 -- 172.21.15.16:0/3152116926 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b0415e0c0 con 0x7f1b040b4120 2023-05-22T15:39:23.160 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:23.143+0000 7f1b09572640 1 --2- 172.21.15.16:0/3152116926 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1af0004d10 0x7f1b041f1a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:39:23.160 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:23.147+0000 7f1b08d71640 1 --2- 172.21.15.16:0/3152116926 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1b040b4120 0x7f1b040b4590 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f1b0000d770 tx=0x7f1b0000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:39:23.160 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:23.147+0000 7f1af9ffb640 1 -- 172.21.15.16:0/3152116926 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b00005270 con 0x7f1b040b4120 2023-05-22T15:39:23.160 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:23.147+0000 7f1af9ffb640 1 -- 172.21.15.16:0/3152116926 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b000053d0 con 0x7f1b040b4120 2023-05-22T15:39:23.161 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:23.147+0000 7f1b0b7d6640 1 -- 172.21.15.16:0/3152116926 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b0415dde0 con 0x7f1b040b4120 2023-05-22T15:39:23.161 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:23.147+0000 7f1af9ffb640 1 -- 172.21.15.16:0/3152116926 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b0000b8b0 con 0x7f1b040b4120 2023-05-22T15:39:23.161 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:23.147+0000 7f1b0b7d6640 1 -- 172.21.15.16:0/3152116926 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b0415e3a0 con 0x7f1b040b4120 2023-05-22T15:39:23.163 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:23.147+0000 7f1b0b7d6640 1 -- 172.21.15.16:0/3152116926 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b04066910 con 0x7f1b040b4120 2023-05-22T15:39:23.163 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:23.147+0000 7f1af9ffb640 1 -- 172.21.15.16:0/3152116926 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 12) v1 ==== 84418+0+0 (secure 0 0 0) 0x7f1b0000ba10 con 0x7f1b040b4120 2023-05-22T15:39:23.163 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:23.147+0000 7f1af9ffb640 1 --2- 172.21.15.16:0/3152116926 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f1ae8065580 0x7f1ae8067a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:23.164 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:23.147+0000 7f1af9ffb640 1 -- 172.21.15.16:0/3152116926 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 6698+0+0 (secure 0 0 0) 0x7f1b00082f00 con 0x7f1b040b4120 2023-05-22T15:39:23.164 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:23.147+0000 7f1b09572640 1 --2- 172.21.15.16:0/3152116926 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f1ae8065580 0x7f1ae8067a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:23.164 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:23.147+0000 7f1b09572640 1 --2- 172.21.15.16:0/3152116926 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f1ae8065580 0x7f1ae8067a30 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f1af4009ac0 tx=0x7f1af4008040 comp rx=0 tx=0).ready entity=mgr.5823 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:39:23.164 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:23.151+0000 7f1af9ffb640 1 -- 172.21.15.16:0/3152116926 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f1b00050510 con 0x7f1b040b4120 2023-05-22T15:39:23.337 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:23.331+0000 7f1b0b7d6640 1 -- 172.21.15.16:0/3152116926 --> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "fs": "cephfs", "path": "/snap_test_dir1/dir6", "repeat": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f1b0404dbe0 con 0x7f1ae8065580 2023-05-22T15:39:23.485 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:23.479+0000 7f1af9ffb640 1 -- 172.21.15.16:0/3152116926 <== mgr.5823 v2:172.21.15.16:6832/36921 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f1b0404dbe0 con 0x7f1ae8065580 2023-05-22T15:39:23.486 INFO:teuthology.orchestra.run.smithi016.stdout:Schedule deactivated for path /snap_test_dir1/dir6 2023-05-22T15:39:23.489 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:23.483+0000 7f1b0b7d6640 1 -- 172.21.15.16:0/3152116926 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f1ae8065580 msgr2=0x7f1ae8067a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:23.489 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:23.483+0000 7f1b0b7d6640 1 --2- 172.21.15.16:0/3152116926 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f1ae8065580 0x7f1ae8067a30 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f1af4009ac0 tx=0x7f1af4008040 comp rx=0 tx=0).stop 2023-05-22T15:39:23.489 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:23.483+0000 7f1b0b7d6640 1 -- 172.21.15.16:0/3152116926 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1b040b4120 msgr2=0x7f1b040b4590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:23.489 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:23.483+0000 7f1b0b7d6640 1 --2- 172.21.15.16:0/3152116926 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1b040b4120 0x7f1b040b4590 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f1b0000d770 tx=0x7f1b0000dc30 comp rx=0 tx=0).stop 2023-05-22T15:39:23.490 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:23.483+0000 7f1b0b7d6640 1 -- 172.21.15.16:0/3152116926 shutdown_connections 2023-05-22T15:39:23.490 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:23.483+0000 7f1b0b7d6640 1 --2- 172.21.15.16:0/3152116926 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f1ae8065580 0x7f1ae8067a30 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:23.490 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:23.483+0000 7f1b0b7d6640 1 --2- 172.21.15.16:0/3152116926 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1b0414ee90 0x7f1b0414f300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:23.490 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:23.483+0000 7f1b0b7d6640 1 --2- 172.21.15.16:0/3152116926 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1b040b4120 0x7f1b040b4590 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:23.490 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:23.483+0000 7f1b0b7d6640 1 --2- 172.21.15.16:0/3152116926 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1af0004d10 0x7f1b041f1a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:23.490 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:23.483+0000 7f1b0b7d6640 1 -- 172.21.15.16:0/3152116926 >> 172.21.15.16:0/3152116926 conn(0x7f1b040700b0 msgr2=0x7f1b0415a250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:39:23.490 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:23.483+0000 7f1b0b7d6640 1 -- 172.21.15.16:0/3152116926 shutdown_connections 2023-05-22T15:39:23.490 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:23.483+0000 7f1b0b7d6640 1 -- 172.21.15.16:0/3152116926 wait complete. 2023-05-22T15:39:23.508 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/dir6 2023-05-22T15:39:23.509 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate --fs cephfs /snap_test_dir1/dir7 1M 2023-05-22T15:39:23.646 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:23.639+0000 7f087343d640 1 Processor -- start 2023-05-22T15:39:23.685 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:23.639+0000 7f087343d640 1 -- start start 2023-05-22T15:39:23.688 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:23.643+0000 7f087343d640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f086c158950 0x7f086c0cb900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:23.689 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:23.643+0000 7f087343d640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f086c0cb440 0x7f086c0cbe40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:23.689 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:23.643+0000 7f087343d640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f086c070bb0 con 0x7f086c06c700 2023-05-22T15:39:23.689 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:23.643+0000 7f087343d640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f086c070e90 con 0x7f086c158950 2023-05-22T15:39:23.689 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:23.643+0000 7f087343d640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f086c071170 con 0x7f086c0cb440 2023-05-22T15:39:23.689 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:23.643+0000 7f08709d8640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f086c0cb440 0x7f086c0cbe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:23.689 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:23.643+0000 7f08711d9640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f086c158950 0x7f086c0cb900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:23.690 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:23.643+0000 7f08709d8640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f086c0cb440 0x7f086c0cbe40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3301/0 says I am v2:172.21.15.16:58262/0 (socket says 172.21.15.16:58262) 2023-05-22T15:39:23.690 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:23.643+0000 7f08709d8640 1 -- 172.21.15.16:0/860096920 learned_addr learned my addr 172.21.15.16:0/860096920 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:39:23.690 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:23.643+0000 7f08711d9640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f086c158950 0x7f086c0cb900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.16:34900/0 (socket says 172.21.15.16:34900) 2023-05-22T15:39:23.690 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:23.643+0000 7f08719da640 1 --1- 172.21.15.16:0/860096920 >> v1:172.21.15.62:6789/0 conn(0x7f086c06c700 0x7f086c15c540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.16:53882/0 (socket says 172.21.15.16:53882) 2023-05-22T15:39:23.690 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:23.643+0000 7f08709d8640 1 -- 172.21.15.16:0/860096920 >> v2:172.21.15.62:3300/0 conn(0x7f086c158950 msgr2=0x7f086c0cb900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:39:23.690 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:23.643+0000 7f08709d8640 1 --2- 172.21.15.16:0/860096920 >> v2:172.21.15.62:3300/0 conn(0x7f086c158950 0x7f086c0cb900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:23.691 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:23.643+0000 7f08709d8640 1 -- 172.21.15.16:0/860096920 >> v1:172.21.15.62:6789/0 conn(0x7f086c06c700 legacy=0x7f086c15c540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:23.691 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:23.643+0000 7f08709d8640 1 -- 172.21.15.16:0/860096920 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f086c071450 con 0x7f086c0cb440 2023-05-22T15:39:23.691 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:23.643+0000 7f08711d9640 1 --2- 172.21.15.16:0/860096920 >> v2:172.21.15.62:3300/0 conn(0x7f086c158950 0x7f086c0cb900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-05-22T15:39:23.691 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:23.647+0000 7f08709d8640 1 --2- 172.21.15.16:0/860096920 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f086c0cb440 0x7f086c0cbe40 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f086000d150 tx=0x7f086001ebe0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ed2de2be8a4ecbfd server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:39:23.691 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:23.647+0000 7f086bfff640 1 -- 172.21.15.16:0/860096920 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f086000b2d0 con 0x7f086c0cb440 2023-05-22T15:39:23.691 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:23.647+0000 7f086bfff640 1 -- 172.21.15.16:0/860096920 <== mon.2 v2:172.21.15.62:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f086000b430 con 0x7f086c0cb440 2023-05-22T15:39:23.691 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:23.647+0000 7f086bfff640 1 -- 172.21.15.16:0/860096920 <== mon.2 v2:172.21.15.62:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08600a7be0 con 0x7f086c0cb440 2023-05-22T15:39:23.692 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:23.647+0000 7f087343d640 1 -- 172.21.15.16:0/860096920 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f086c0cb440 msgr2=0x7f086c0cbe40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:23.692 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:23.647+0000 7f087343d640 1 --2- 172.21.15.16:0/860096920 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f086c0cb440 0x7f086c0cbe40 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f086000d150 tx=0x7f086001ebe0 comp rx=0 tx=0).stop 2023-05-22T15:39:23.692 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:23.647+0000 7f087343d640 1 -- 172.21.15.16:0/860096920 shutdown_connections 2023-05-22T15:39:23.692 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:23.647+0000 7f087343d640 1 --2- 172.21.15.16:0/860096920 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f086c0cb440 0x7f086c0cbe40 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:23.692 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:23.647+0000 7f087343d640 1 --2- 172.21.15.16:0/860096920 >> v2:172.21.15.62:3300/0 conn(0x7f086c158950 0x7f086c0cb900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:23.692 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:23.647+0000 7f087343d640 1 -- 172.21.15.16:0/860096920 >> 172.21.15.16:0/860096920 conn(0x7f086c0b0cc0 msgr2=0x7f086c0b10c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:39:23.692 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:23.647+0000 7f087343d640 1 -- 172.21.15.16:0/860096920 shutdown_connections 2023-05-22T15:39:23.693 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:23.647+0000 7f087343d640 1 -- 172.21.15.16:0/860096920 wait complete. 2023-05-22T15:39:23.693 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:23.647+0000 7f087343d640 1 Processor -- start 2023-05-22T15:39:23.693 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:23.647+0000 7f087343d640 1 -- start start 2023-05-22T15:39:23.693 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:23.647+0000 7f087343d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f086c06c700 0x7f086c152950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:23.693 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:23.647+0000 7f087343d640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f086c0cb440 0x7f086c155f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:23.693 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:23.647+0000 7f087343d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f086c158950 0x7f086c163060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:23.693 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:23.647+0000 7f087343d640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f086c1534f0 con 0x7f086c158950 2023-05-22T15:39:23.694 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:23.647+0000 7f087343d640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f086c1564c0 con 0x7f086c06c700 2023-05-22T15:39:23.694 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:23.647+0000 7f087343d640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f086c159f80 con 0x7f086c0cb440 2023-05-22T15:39:23.694 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:23.647+0000 7f08719da640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f086c158950 0x7f086c163060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:23.694 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:23.647+0000 7f08719da640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f086c158950 0x7f086c163060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52366/0 (socket says 172.21.15.16:52366) 2023-05-22T15:39:23.694 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:23.647+0000 7f08719da640 1 -- 172.21.15.16:0/835910032 learned_addr learned my addr 172.21.15.16:0/835910032 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:39:23.694 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:23.647+0000 7f08711d9640 1 --2- 172.21.15.16:0/835910032 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f086c06c700 0x7f086c152950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:23.695 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:23.647+0000 7f08709d8640 1 --2- 172.21.15.16:0/835910032 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f086c0cb440 0x7f086c155f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:23.695 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:23.651+0000 7f08719da640 1 -- 172.21.15.16:0/835910032 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f086c0cb440 msgr2=0x7f086c155f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:23.695 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:23.651+0000 7f08719da640 1 --2- 172.21.15.16:0/835910032 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f086c0cb440 0x7f086c155f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:23.695 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:23.651+0000 7f08719da640 1 -- 172.21.15.16:0/835910032 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f086c06c700 msgr2=0x7f086c152950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:23.695 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:23.651+0000 7f08719da640 1 --2- 172.21.15.16:0/835910032 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f086c06c700 0x7f086c152950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:23.696 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:23.651+0000 7f08719da640 1 -- 172.21.15.16:0/835910032 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0864009420 con 0x7f086c158950 2023-05-22T15:39:23.696 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:23.651+0000 7f08709d8640 1 --2- 172.21.15.16:0/835910032 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f086c0cb440 0x7f086c155f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T15:39:23.696 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:23.651+0000 7f08711d9640 1 --2- 172.21.15.16:0/835910032 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f086c06c700 0x7f086c152950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T15:39:23.696 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:23.651+0000 7f08719da640 1 --2- 172.21.15.16:0/835910032 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f086c158950 0x7f086c163060 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f085c00c3e0 tx=0x7f085c00c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:39:23.697 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:23.651+0000 7f0869ffb640 1 -- 172.21.15.16:0/835910032 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f085c0074e0 con 0x7f086c158950 2023-05-22T15:39:23.697 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:23.651+0000 7f0869ffb640 1 -- 172.21.15.16:0/835910032 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f085c007640 con 0x7f086c158950 2023-05-22T15:39:23.697 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:23.651+0000 7f087343d640 1 -- 172.21.15.16:0/835910032 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f086000bd40 con 0x7f086c158950 2023-05-22T15:39:23.697 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:23.651+0000 7f0869ffb640 1 -- 172.21.15.16:0/835910032 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f085c0049f0 con 0x7f086c158950 2023-05-22T15:39:23.697 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:23.651+0000 7f087343d640 1 -- 172.21.15.16:0/835910032 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f086c070980 con 0x7f086c158950 2023-05-22T15:39:23.697 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:23.651+0000 7f087343d640 1 -- 172.21.15.16:0/835910032 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f086c0707f0 con 0x7f086c158950 2023-05-22T15:39:23.698 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:23.651+0000 7f0869ffb640 1 -- 172.21.15.16:0/835910032 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 12) v1 ==== 84418+0+0 (secure 0 0 0) 0x7f085c004b50 con 0x7f086c158950 2023-05-22T15:39:23.698 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:23.651+0000 7f0869ffb640 1 --2- 172.21.15.16:0/835910032 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f0858065580 0x7f0858067a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:23.698 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:23.651+0000 7f0869ffb640 1 -- 172.21.15.16:0/835910032 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 6698+0+0 (secure 0 0 0) 0x7f085c082820 con 0x7f086c158950 2023-05-22T15:39:23.698 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:23.651+0000 7f08711d9640 1 --2- 172.21.15.16:0/835910032 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f0858065580 0x7f0858067a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:23.698 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:23.651+0000 7f08711d9640 1 --2- 172.21.15.16:0/835910032 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f0858065580 0x7f0858067a30 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f0864009ca0 tx=0x7f0864008040 comp rx=0 tx=0).ready entity=mgr.5823 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:39:23.698 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:23.655+0000 7f0869ffb640 1 -- 172.21.15.16:0/835910032 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f085c04ffa0 con 0x7f086c158950 2023-05-22T15:39:23.836 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:23.831+0000 7f087343d640 1 -- 172.21.15.16:0/835910032 --> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "fs": "cephfs", "path": "/snap_test_dir1/dir7", "repeat": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f086c069720 con 0x7f0858065580 2023-05-22T15:39:23.988 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:23.983+0000 7f0869ffb640 1 -- 172.21.15.16:0/835910032 <== mgr.5823 v2:172.21.15.16:6832/36921 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f086c069720 con 0x7f0858065580 2023-05-22T15:39:23.989 INFO:teuthology.orchestra.run.smithi016.stdout:Schedule deactivated for path /snap_test_dir1/dir7 2023-05-22T15:39:23.991 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:23.987+0000 7f087343d640 1 -- 172.21.15.16:0/835910032 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f0858065580 msgr2=0x7f0858067a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:23.991 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:23.987+0000 7f087343d640 1 --2- 172.21.15.16:0/835910032 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f0858065580 0x7f0858067a30 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f0864009ca0 tx=0x7f0864008040 comp rx=0 tx=0).stop 2023-05-22T15:39:23.991 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:23.987+0000 7f087343d640 1 -- 172.21.15.16:0/835910032 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f086c158950 msgr2=0x7f086c163060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:23.991 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:23.987+0000 7f087343d640 1 --2- 172.21.15.16:0/835910032 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f086c158950 0x7f086c163060 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f085c00c3e0 tx=0x7f085c00c8a0 comp rx=0 tx=0).stop 2023-05-22T15:39:23.991 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:23.987+0000 7f087343d640 1 -- 172.21.15.16:0/835910032 shutdown_connections 2023-05-22T15:39:23.991 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:23.987+0000 7f087343d640 1 --2- 172.21.15.16:0/835910032 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f086c158950 0x7f086c163060 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:23.992 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:23.987+0000 7f087343d640 1 --2- 172.21.15.16:0/835910032 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f0858065580 0x7f0858067a30 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:23.992 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:23.987+0000 7f087343d640 1 --2- 172.21.15.16:0/835910032 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f086c0cb440 0x7f086c155f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:23.992 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:23.987+0000 7f087343d640 1 --2- 172.21.15.16:0/835910032 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f086c06c700 0x7f086c152950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:23.992 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:23.987+0000 7f087343d640 1 -- 172.21.15.16:0/835910032 >> 172.21.15.16:0/835910032 conn(0x7f086c0b0cc0 msgr2=0x7f086c1512e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:39:23.992 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:23.987+0000 7f087343d640 1 -- 172.21.15.16:0/835910032 shutdown_connections 2023-05-22T15:39:23.992 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:23.987+0000 7f087343d640 1 -- 172.21.15.16:0/835910032 wait complete. 2023-05-22T15:39:24.010 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/dir7 2023-05-22T15:39:24.011 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate --fs cephfs /snap_test_dir1/dir8 1M 2023-05-22T15:39:24.145 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:24.139+0000 7f82f504a640 1 Processor -- start 2023-05-22T15:39:24.145 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:24.139+0000 7f82f504a640 1 -- start start 2023-05-22T15:39:24.146 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:24.143+0000 7f82f504a640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f82f0157f70 0x7f82f00c33f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:24.147 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:24.143+0000 7f82f504a640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f82f0158340 0x7f82f00cbc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:24.147 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:24.143+0000 7f82f504a640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82f00c30e0 con 0x7f82f015bcf0 2023-05-22T15:39:24.147 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:24.143+0000 7f82f504a640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82f006f770 con 0x7f82f0157f70 2023-05-22T15:39:24.147 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:24.143+0000 7f82f504a640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82f00c3a40 con 0x7f82f0158340 2023-05-22T15:39:24.147 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:24.143+0000 7f82eed9d640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f82f0157f70 0x7f82f00c33f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:24.147 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:24.143+0000 7f82ef59e640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f82f015bcf0 0x7f82f006b680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:53026/0 (socket says 172.21.15.16:53026) 2023-05-22T15:39:24.147 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:24.143+0000 7f82ef59e640 1 -- 172.21.15.16:0/4060328838 learned_addr learned my addr 172.21.15.16:0/4060328838 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:39:24.148 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:24.143+0000 7f82edd9b640 1 -- 172.21.15.16:0/4060328838 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2429220852 0 0) 0x7f82f00c30e0 con 0x7f82f015bcf0 2023-05-22T15:39:24.148 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:24.143+0000 7f82edd9b640 1 -- 172.21.15.16:0/4060328838 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82d8003450 con 0x7f82f015bcf0 2023-05-22T15:39:24.148 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:24.143+0000 7f82eed9d640 1 -- 172.21.15.16:0/4060328838 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f82f0158340 msgr2=0x7f82f00cbc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:39:24.148 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:24.143+0000 7f82eed9d640 1 --2- 172.21.15.16:0/4060328838 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f82f0158340 0x7f82f00cbc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:24.149 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:24.143+0000 7f82eed9d640 1 -- 172.21.15.16:0/4060328838 >> v1:172.21.15.16:6789/0 conn(0x7f82f015bcf0 legacy=0x7f82f006b680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:24.149 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:24.143+0000 7f82eed9d640 1 -- 172.21.15.16:0/4060328838 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82f00c3d20 con 0x7f82f0157f70 2023-05-22T15:39:24.149 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:24.143+0000 7f82eed9d640 1 --2- 172.21.15.16:0/4060328838 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82f0157f70 0x7f82f00c33f0 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f82e00097f0 tx=0x7f82e001c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f95069a63776765 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:39:24.149 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:24.143+0000 7f82edd9b640 1 -- 172.21.15.16:0/4060328838 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82e00ab650 con 0x7f82f0157f70 2023-05-22T15:39:24.150 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:24.143+0000 7f82edd9b640 1 -- 172.21.15.16:0/4060328838 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82e00ab7b0 con 0x7f82f0157f70 2023-05-22T15:39:24.150 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:24.143+0000 7f82edd9b640 1 -- 172.21.15.16:0/4060328838 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82e00abb40 con 0x7f82f0157f70 2023-05-22T15:39:24.150 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:24.143+0000 7f82f504a640 1 -- 172.21.15.16:0/4060328838 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82f0157f70 msgr2=0x7f82f00c33f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:24.151 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:24.143+0000 7f82f504a640 1 --2- 172.21.15.16:0/4060328838 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82f0157f70 0x7f82f00c33f0 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f82e00097f0 tx=0x7f82e001c450 comp rx=0 tx=0).stop 2023-05-22T15:39:24.151 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:24.143+0000 7f82f504a640 1 -- 172.21.15.16:0/4060328838 shutdown_connections 2023-05-22T15:39:24.151 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:24.143+0000 7f82f504a640 1 --2- 172.21.15.16:0/4060328838 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f82f0158340 0x7f82f00cbc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:24.151 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:24.143+0000 7f82f504a640 1 --2- 172.21.15.16:0/4060328838 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82f0157f70 0x7f82f00c33f0 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:24.151 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:24.143+0000 7f82f504a640 1 -- 172.21.15.16:0/4060328838 >> 172.21.15.16:0/4060328838 conn(0x7f82f0069720 msgr2=0x7f82f0069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:39:24.151 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:24.147+0000 7f82f504a640 1 -- 172.21.15.16:0/4060328838 shutdown_connections 2023-05-22T15:39:24.151 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:24.147+0000 7f82f504a640 1 -- 172.21.15.16:0/4060328838 wait complete. 2023-05-22T15:39:24.153 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:24.147+0000 7f82f504a640 1 Processor -- start 2023-05-22T15:39:24.153 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:24.147+0000 7f82f504a640 1 -- start start 2023-05-22T15:39:24.153 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:24.147+0000 7f82f504a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82f0157f70 0x7f82f0158770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:24.153 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:24.147+0000 7f82f504a640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f82f0158340 0x7f82f015b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:24.154 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:24.147+0000 7f82f504a640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f82f015bcf0 0x7f82f0143aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:24.154 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:24.147+0000 7f82f504a640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82f0069eb0 con 0x7f82f0157f70 2023-05-22T15:39:24.154 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:24.147+0000 7f82f504a640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82f0158dc0 con 0x7f82f0158340 2023-05-22T15:39:24.154 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:24.147+0000 7f82f504a640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82f00c7660 con 0x7f82f015bcf0 2023-05-22T15:39:24.154 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:24.147+0000 7f82eed9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82f0157f70 0x7f82f0158770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:24.154 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:24.147+0000 7f82eed9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82f0157f70 0x7f82f0158770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52388/0 (socket says 172.21.15.16:52388) 2023-05-22T15:39:24.154 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:24.147+0000 7f82eed9d640 1 -- 172.21.15.16:0/3838650662 learned_addr learned my addr 172.21.15.16:0/3838650662 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:39:24.155 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:24.147+0000 7f82eed9d640 1 -- 172.21.15.16:0/3838650662 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f82f015bcf0 msgr2=0x7f82f0143aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:24.155 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:24.147+0000 7f82eed9d640 1 --2- 172.21.15.16:0/3838650662 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f82f015bcf0 0x7f82f0143aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:24.155 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:24.147+0000 7f82eed9d640 1 -- 172.21.15.16:0/3838650662 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f82f0158340 msgr2=0x7f82f015b410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:24.155 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:24.147+0000 7f82eed9d640 1 --2- 172.21.15.16:0/3838650662 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f82f0158340 0x7f82f015b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:24.155 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:24.147+0000 7f82eed9d640 1 -- 172.21.15.16:0/3838650662 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82e0009420 con 0x7f82f0157f70 2023-05-22T15:39:24.155 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:24.147+0000 7f82eed9d640 1 --2- 172.21.15.16:0/3838650662 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82f0157f70 0x7f82f0158770 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f82e00097f0 tx=0x7f82e001ac90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:39:24.155 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:24.147+0000 7f82d77fe640 1 -- 172.21.15.16:0/3838650662 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82e00b5810 con 0x7f82f0157f70 2023-05-22T15:39:24.155 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:24.147+0000 7f82d77fe640 1 -- 172.21.15.16:0/3838650662 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82e00b5990 con 0x7f82f0157f70 2023-05-22T15:39:24.156 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:24.147+0000 7f82f504a640 1 -- 172.21.15.16:0/3838650662 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82f006bd90 con 0x7f82f0157f70 2023-05-22T15:39:24.156 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:24.147+0000 7f82d77fe640 1 -- 172.21.15.16:0/3838650662 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82e00b5d40 con 0x7f82f0157f70 2023-05-22T15:39:24.156 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:24.147+0000 7f82f504a640 1 -- 172.21.15.16:0/3838650662 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82f00cc5c0 con 0x7f82f0157f70 2023-05-22T15:39:24.157 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:24.147+0000 7f82f504a640 1 -- 172.21.15.16:0/3838650662 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82f006bf20 con 0x7f82f0157f70 2023-05-22T15:39:24.157 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:24.151+0000 7f82d77fe640 1 -- 172.21.15.16:0/3838650662 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 12) v1 ==== 84418+0+0 (secure 0 0 0) 0x7f82e00b4390 con 0x7f82f0157f70 2023-05-22T15:39:24.157 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:24.151+0000 7f82d77fe640 1 --2- 172.21.15.16:0/3838650662 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f82dc0655b0 0x7f82dc067a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:24.157 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:24.151+0000 7f82d77fe640 1 -- 172.21.15.16:0/3838650662 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 6698+0+0 (secure 0 0 0) 0x7f82e011d610 con 0x7f82f0157f70 2023-05-22T15:39:24.157 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:24.151+0000 7f82ee59c640 1 --2- 172.21.15.16:0/3838650662 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f82dc0655b0 0x7f82dc067a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:24.157 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:24.151+0000 7f82ee59c640 1 --2- 172.21.15.16:0/3838650662 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f82dc0655b0 0x7f82dc067a60 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f82f015b200 tx=0x7f82e4006ca0 comp rx=0 tx=0).ready entity=mgr.5823 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:39:24.159 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:24.155+0000 7f82d77fe640 1 -- 172.21.15.16:0/3838650662 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f82e00ead30 con 0x7f82f0157f70 2023-05-22T15:39:24.336 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:24.331+0000 7f82f504a640 1 -- 172.21.15.16:0/3838650662 --> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "fs": "cephfs", "path": "/snap_test_dir1/dir8", "repeat": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f82f00c77a0 con 0x7f82dc0655b0 2023-05-22T15:39:24.487 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:24.483+0000 7f82d77fe640 1 -- 172.21.15.16:0/3838650662 <== mgr.5823 v2:172.21.15.16:6832/36921 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f82e4013400 con 0x7f82dc0655b0 2023-05-22T15:39:24.487 INFO:teuthology.orchestra.run.smithi016.stdout:Schedule deactivated for path /snap_test_dir1/dir8 2023-05-22T15:39:24.491 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:24.487+0000 7f82f504a640 1 -- 172.21.15.16:0/3838650662 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f82dc0655b0 msgr2=0x7f82dc067a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:24.491 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:24.487+0000 7f82f504a640 1 --2- 172.21.15.16:0/3838650662 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f82dc0655b0 0x7f82dc067a60 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f82f015b200 tx=0x7f82e4006ca0 comp rx=0 tx=0).stop 2023-05-22T15:39:24.491 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:24.487+0000 7f82f504a640 1 -- 172.21.15.16:0/3838650662 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82f0157f70 msgr2=0x7f82f0158770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:24.492 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:24.487+0000 7f82f504a640 1 --2- 172.21.15.16:0/3838650662 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82f0157f70 0x7f82f0158770 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f82e00097f0 tx=0x7f82e001ac90 comp rx=0 tx=0).stop 2023-05-22T15:39:24.492 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:24.487+0000 7f82f504a640 1 -- 172.21.15.16:0/3838650662 shutdown_connections 2023-05-22T15:39:24.492 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:24.487+0000 7f82f504a640 1 --2- 172.21.15.16:0/3838650662 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f82dc0655b0 0x7f82dc067a60 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:24.492 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:24.487+0000 7f82f504a640 1 --2- 172.21.15.16:0/3838650662 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f82f015bcf0 0x7f82f0143aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:24.492 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:24.487+0000 7f82f504a640 1 --2- 172.21.15.16:0/3838650662 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f82f0158340 0x7f82f015b410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:24.492 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:24.487+0000 7f82f504a640 1 --2- 172.21.15.16:0/3838650662 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82f0157f70 0x7f82f0158770 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:24.492 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:24.487+0000 7f82f504a640 1 -- 172.21.15.16:0/3838650662 >> 172.21.15.16:0/3838650662 conn(0x7f82f0069720 msgr2=0x7f82f01627c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:39:24.493 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:24.487+0000 7f82f504a640 1 -- 172.21.15.16:0/3838650662 shutdown_connections 2023-05-22T15:39:24.493 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:24.487+0000 7f82f504a640 1 -- 172.21.15.16:0/3838650662 wait complete. 2023-05-22T15:39:24.515 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/dir8 2023-05-22T15:39:24.516 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate --fs cephfs /snap_test_dir1/dir9 1M 2023-05-22T15:39:24.644 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:24.639+0000 7f11076da640 1 Processor -- start 2023-05-22T15:39:24.644 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:24.639+0000 7f11076da640 1 -- start start 2023-05-22T15:39:24.645 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:24.639+0000 7f11076da640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f11000cf210 0x7f11000d2b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:24.645 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:24.639+0000 7f11076da640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f11000b46e0 0x7f11000d0460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:24.645 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:24.639+0000 7f11076da640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11000cd2d0 con 0x7f11000b7c90 2023-05-22T15:39:24.645 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:24.639+0000 7f11076da640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11000b8060 con 0x7f11000cf210 2023-05-22T15:39:24.645 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:24.639+0000 7f11076da640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11000cf680 con 0x7f11000b46e0 2023-05-22T15:39:24.646 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:24.639+0000 7f1105c77640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f11000b46e0 0x7f11000d0460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:24.646 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:24.639+0000 7f1104c75640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f11000cf210 0x7f11000d2b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:24.646 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:24.639+0000 7f1105476640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f11000b7c90 0x7f11000d2440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.16:53892/0 (socket says 172.21.15.16:53892) 2023-05-22T15:39:24.647 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:24.639+0000 7f1105476640 1 -- 172.21.15.16:0/1677994118 learned_addr learned my addr 172.21.15.16:0/1677994118 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:39:24.647 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:24.643+0000 7f10fffff640 1 -- 172.21.15.16:0/1677994118 <== mon.1 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3129142379 0 0) 0x7f11000cd2d0 con 0x7f11000b7c90 2023-05-22T15:39:24.647 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:24.643+0000 7f10fffff640 1 -- 172.21.15.16:0/1677994118 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f10e8003530 con 0x7f11000b7c90 2023-05-22T15:39:24.647 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:24.643+0000 7f1105c77640 1 -- 172.21.15.16:0/1677994118 >> v2:172.21.15.62:3300/0 conn(0x7f11000cf210 msgr2=0x7f11000d2b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:39:24.647 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:24.643+0000 7f1105c77640 1 --2- 172.21.15.16:0/1677994118 >> v2:172.21.15.62:3300/0 conn(0x7f11000cf210 0x7f11000d2b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:24.647 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:24.643+0000 7f1105c77640 1 -- 172.21.15.16:0/1677994118 >> v1:172.21.15.62:6789/0 conn(0x7f11000b7c90 legacy=0x7f11000d2440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:24.647 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:24.643+0000 7f1105c77640 1 -- 172.21.15.16:0/1677994118 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11000ccc80 con 0x7f11000b46e0 2023-05-22T15:39:24.648 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:24.643+0000 7f1104c75640 1 --2- 172.21.15.16:0/1677994118 >> v2:172.21.15.62:3300/0 conn(0x7f11000cf210 0x7f11000d2b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-05-22T15:39:24.649 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:24.643+0000 7f1105c77640 1 --2- 172.21.15.16:0/1677994118 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f11000b46e0 0x7f11000d0460 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f10f00097c0 tx=0x7f10f001c450 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6084cf195c5f5379 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:39:24.649 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:24.643+0000 7f10fffff640 1 -- 172.21.15.16:0/1677994118 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10f00ab670 con 0x7f11000b46e0 2023-05-22T15:39:24.650 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:24.643+0000 7f10fffff640 1 -- 172.21.15.16:0/1677994118 <== mon.2 v2:172.21.15.62:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10f00ab7d0 con 0x7f11000b46e0 2023-05-22T15:39:24.650 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:24.643+0000 7f10fffff640 1 -- 172.21.15.16:0/1677994118 <== mon.2 v2:172.21.15.62:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10f00abb40 con 0x7f11000b46e0 2023-05-22T15:39:24.650 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:24.647+0000 7f11076da640 1 -- 172.21.15.16:0/1677994118 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f11000b46e0 msgr2=0x7f11000d0460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:24.651 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:24.647+0000 7f11076da640 1 --2- 172.21.15.16:0/1677994118 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f11000b46e0 0x7f11000d0460 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f10f00097c0 tx=0x7f10f001c450 comp rx=0 tx=0).stop 2023-05-22T15:39:24.651 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:24.647+0000 7f11076da640 1 -- 172.21.15.16:0/1677994118 shutdown_connections 2023-05-22T15:39:24.651 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:24.647+0000 7f11076da640 1 --2- 172.21.15.16:0/1677994118 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f11000b46e0 0x7f11000d0460 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:24.651 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:24.647+0000 7f11076da640 1 --2- 172.21.15.16:0/1677994118 >> v2:172.21.15.62:3300/0 conn(0x7f11000cf210 0x7f11000d2b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:24.651 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:24.647+0000 7f11076da640 1 -- 172.21.15.16:0/1677994118 >> 172.21.15.16:0/1677994118 conn(0x7f11000655c0 msgr2=0x7f1100070e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:39:24.651 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:24.647+0000 7f11076da640 1 -- 172.21.15.16:0/1677994118 shutdown_connections 2023-05-22T15:39:24.651 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:24.647+0000 7f11076da640 1 -- 172.21.15.16:0/1677994118 wait complete. 2023-05-22T15:39:24.652 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:24.647+0000 7f11076da640 1 Processor -- start 2023-05-22T15:39:24.652 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:24.647+0000 7f11076da640 1 -- start start 2023-05-22T15:39:24.653 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:24.647+0000 7f11076da640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f11000b46e0 0x7f1100153a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:24.653 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:24.647+0000 7f11076da640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f11000b7c90 0x7f110014e500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:24.653 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:24.647+0000 7f11076da640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f11000cf210 0x7f110014ea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:24.654 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:24.647+0000 7f11076da640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11000c6e10 con 0x7f11000cf210 2023-05-22T15:39:24.654 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:24.647+0000 7f11076da640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11000c82c0 con 0x7f11000b46e0 2023-05-22T15:39:24.654 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:24.647+0000 7f11076da640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11000c8560 con 0x7f11000b7c90 2023-05-22T15:39:24.654 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:24.647+0000 7f1105c77640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f11000cf210 0x7f110014ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:24.654 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:24.647+0000 7f1105c77640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f11000cf210 0x7f110014ea40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52396/0 (socket says 172.21.15.16:52396) 2023-05-22T15:39:24.654 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:24.647+0000 7f1105c77640 1 -- 172.21.15.16:0/1801053243 learned_addr learned my addr 172.21.15.16:0/1801053243 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:39:24.654 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:24.647+0000 7f1104c75640 1 --2- 172.21.15.16:0/1801053243 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f11000b7c90 0x7f110014e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:24.655 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:24.647+0000 7f1105476640 1 --2- 172.21.15.16:0/1801053243 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f11000b46e0 0x7f1100153a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:24.656 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:24.647+0000 7f1105c77640 1 -- 172.21.15.16:0/1801053243 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f11000b7c90 msgr2=0x7f110014e500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:24.656 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:24.647+0000 7f1105c77640 1 --2- 172.21.15.16:0/1801053243 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f11000b7c90 0x7f110014e500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:24.656 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:24.647+0000 7f1105c77640 1 -- 172.21.15.16:0/1801053243 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f11000b46e0 msgr2=0x7f1100153a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:24.656 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:24.647+0000 7f1105c77640 1 --2- 172.21.15.16:0/1801053243 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f11000b46e0 0x7f1100153a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:24.656 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:24.647+0000 7f1105c77640 1 -- 172.21.15.16:0/1801053243 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10f4009420 con 0x7f11000cf210 2023-05-22T15:39:24.656 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:24.651+0000 7f1105c77640 1 --2- 172.21.15.16:0/1801053243 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f11000cf210 0x7f110014ea40 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f10f001a080 tx=0x7f10f00191f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:39:24.656 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:24.651+0000 7f10fdffb640 1 -- 172.21.15.16:0/1801053243 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10f0004450 con 0x7f11000cf210 2023-05-22T15:39:24.657 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:24.651+0000 7f10fdffb640 1 -- 172.21.15.16:0/1801053243 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10f00ab940 con 0x7f11000cf210 2023-05-22T15:39:24.657 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:24.651+0000 7f11076da640 1 -- 172.21.15.16:0/1801053243 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10f0009420 con 0x7f11000cf210 2023-05-22T15:39:24.657 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:24.651+0000 7f10fdffb640 1 -- 172.21.15.16:0/1801053243 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10f00abcb0 con 0x7f11000cf210 2023-05-22T15:39:24.657 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:24.651+0000 7f11076da640 1 -- 172.21.15.16:0/1801053243 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11000d1340 con 0x7f11000cf210 2023-05-22T15:39:24.657 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:24.651+0000 7f11076da640 1 -- 172.21.15.16:0/1801053243 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11000d11b0 con 0x7f11000cf210 2023-05-22T15:39:24.657 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:24.651+0000 7f10fdffb640 1 -- 172.21.15.16:0/1801053243 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 12) v1 ==== 84418+0+0 (secure 0 0 0) 0x7f10f0003620 con 0x7f11000cf210 2023-05-22T15:39:24.658 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:24.651+0000 7f10fdffb640 1 --2- 172.21.15.16:0/1801053243 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f10ec065580 0x7f10ec067a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:24.658 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:24.651+0000 7f10fdffb640 1 -- 172.21.15.16:0/1801053243 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 6698+0+0 (secure 0 0 0) 0x7f10f011d0f0 con 0x7f11000cf210 2023-05-22T15:39:24.658 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:24.651+0000 7f1105476640 1 --2- 172.21.15.16:0/1801053243 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f10ec065580 0x7f10ec067a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:24.658 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:24.651+0000 7f1105476640 1 --2- 172.21.15.16:0/1801053243 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f10ec065580 0x7f10ec067a30 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f10f8005c10 tx=0x7f10f8009040 comp rx=0 tx=0).ready entity=mgr.5823 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:39:24.660 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:24.655+0000 7f10fdffb640 1 -- 172.21.15.16:0/1801053243 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f10f00ea6f0 con 0x7f11000cf210 2023-05-22T15:39:24.837 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:24.831+0000 7f11076da640 1 -- 172.21.15.16:0/1801053243 --> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "fs": "cephfs", "path": "/snap_test_dir1/dir9", "repeat": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f11000d14d0 con 0x7f10ec065580 2023-05-22T15:39:24.986 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:24.979+0000 7f10fdffb640 1 -- 172.21.15.16:0/1801053243 <== mgr.5823 v2:172.21.15.16:6832/36921 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f11000d14d0 con 0x7f10ec065580 2023-05-22T15:39:24.986 INFO:teuthology.orchestra.run.smithi016.stdout:Schedule deactivated for path /snap_test_dir1/dir9 2023-05-22T15:39:24.990 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:24.983+0000 7f11076da640 1 -- 172.21.15.16:0/1801053243 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f10ec065580 msgr2=0x7f10ec067a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:24.990 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:24.983+0000 7f11076da640 1 --2- 172.21.15.16:0/1801053243 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f10ec065580 0x7f10ec067a30 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f10f8005c10 tx=0x7f10f8009040 comp rx=0 tx=0).stop 2023-05-22T15:39:24.990 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:24.983+0000 7f11076da640 1 -- 172.21.15.16:0/1801053243 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f11000cf210 msgr2=0x7f110014ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:24.990 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:24.983+0000 7f11076da640 1 --2- 172.21.15.16:0/1801053243 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f11000cf210 0x7f110014ea40 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f10f001a080 tx=0x7f10f00191f0 comp rx=0 tx=0).stop 2023-05-22T15:39:24.990 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:24.983+0000 7f11076da640 1 -- 172.21.15.16:0/1801053243 shutdown_connections 2023-05-22T15:39:24.991 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:24.983+0000 7f11076da640 1 --2- 172.21.15.16:0/1801053243 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f11000cf210 0x7f110014ea40 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:24.991 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:24.983+0000 7f11076da640 1 --2- 172.21.15.16:0/1801053243 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f10ec065580 0x7f10ec067a30 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:24.991 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:24.983+0000 7f11076da640 1 --2- 172.21.15.16:0/1801053243 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f11000b7c90 0x7f110014e500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:24.991 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:24.983+0000 7f11076da640 1 --2- 172.21.15.16:0/1801053243 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f11000b46e0 0x7f1100153a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:24.991 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:24.983+0000 7f11076da640 1 -- 172.21.15.16:0/1801053243 >> 172.21.15.16:0/1801053243 conn(0x7f11000655c0 msgr2=0x7f11000b6b40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:39:24.991 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:24.983+0000 7f11076da640 1 -- 172.21.15.16:0/1801053243 shutdown_connections 2023-05-22T15:39:24.992 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:24.987+0000 7f11076da640 1 -- 172.21.15.16:0/1801053243 wait complete. 2023-05-22T15:39:25.015 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/dir9 2023-05-22T15:39:25.015 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:39:25.016 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir0/.snap) 2023-05-22T15:39:25.028 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_29_01_UTC 2023-05-22T15:39:25.029 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_30_00_UTC 2023-05-22T15:39:25.029 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_31_01_UTC 2023-05-22T15:39:25.029 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_32_00_UTC 2023-05-22T15:39:25.029 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_33_00_UTC 2023-05-22T15:39:25.029 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_34_00_UTC 2023-05-22T15:39:25.029 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_35_00_UTC 2023-05-22T15:39:25.030 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_36_00_UTC 2023-05-22T15:39:25.030 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_37_01_UTC 2023-05-22T15:39:25.030 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_38_01_UTC 2023-05-22T15:39:25.030 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_39_01_UTC 2023-05-22T15:39:25.030 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2023-05-22-15_29_01_UTC', 'scheduled-2023-05-22-15_30_00_UTC', 'scheduled-2023-05-22-15_31_01_UTC', 'scheduled-2023-05-22-15_32_00_UTC', 'scheduled-2023-05-22-15_33_00_UTC', 'scheduled-2023-05-22-15_34_00_UTC', 'scheduled-2023-05-22-15_35_00_UTC', 'scheduled-2023-05-22-15_36_00_UTC', 'scheduled-2023-05-22-15_37_01_UTC', 'scheduled-2023-05-22-15_38_01_UTC', 'scheduled-2023-05-22-15_39_01_UTC'] 2023-05-22T15:39:25.031 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --fs cephfs --format json /snap_test_dir1/dir0 2023-05-22T15:39:25.203 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:25.199+0000 7f3c77e93640 1 Processor -- start 2023-05-22T15:39:25.204 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:25.199+0000 7f3c77e93640 1 -- start start 2023-05-22T15:39:25.204 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:25.199+0000 7f3c77e93640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3c7015c8e0 0x7f3c700ccc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:25.205 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:25.199+0000 7f3c77e93640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f3c700d2eb0 0x7f3c7015ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:25.205 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:25.199+0000 7f3c77e93640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c700d2a20 con 0x7f3c700cef10 2023-05-22T15:39:25.205 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:25.199+0000 7f3c77e93640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c700cd230 con 0x7f3c700d2eb0 2023-05-22T15:39:25.205 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:25.199+0000 7f3c77e93640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c700d23d0 con 0x7f3c7015c8e0 2023-05-22T15:39:25.206 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:25.199+0000 7f3c76430640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f3c700cef10 0x7f3c700cf370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:53040/0 (socket says 172.21.15.16:53040) 2023-05-22T15:39:25.206 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:25.199+0000 7f3c76430640 1 -- 172.21.15.16:0/4262512572 learned_addr learned my addr 172.21.15.16:0/4262512572 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:39:25.207 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:25.199+0000 7f3c74c2d640 1 -- 172.21.15.16:0/4262512572 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 717172753 0 0) 0x7f3c700d2a20 con 0x7f3c700cef10 2023-05-22T15:39:25.207 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:25.199+0000 7f3c7542e640 1 --2- 172.21.15.16:0/4262512572 >> v2:172.21.15.62:3300/0 conn(0x7f3c700d2eb0 0x7f3c7015ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:25.207 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:25.199+0000 7f3c75c2f640 1 --2- 172.21.15.16:0/4262512572 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3c7015c8e0 0x7f3c700ccc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:25.207 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:25.199+0000 7f3c74c2d640 1 -- 172.21.15.16:0/4262512572 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c5c003530 con 0x7f3c700cef10 2023-05-22T15:39:25.207 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:25.199+0000 7f3c74c2d640 1 -- 172.21.15.16:0/4262512572 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3033000041 0 0) 0x7f3c5c003530 con 0x7f3c700cef10 2023-05-22T15:39:25.207 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:25.199+0000 7f3c74c2d640 1 -- 172.21.15.16:0/4262512572 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3c7015c8e0 msgr2=0x7f3c700ccc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:39:25.207 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:25.199+0000 7f3c74c2d640 1 --2- 172.21.15.16:0/4262512572 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3c7015c8e0 0x7f3c700ccc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:25.208 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:25.199+0000 7f3c74c2d640 1 -- 172.21.15.16:0/4262512572 >> v2:172.21.15.62:3300/0 conn(0x7f3c700d2eb0 msgr2=0x7f3c7015ccb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:39:25.208 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:25.199+0000 7f3c74c2d640 1 --2- 172.21.15.16:0/4262512572 >> v2:172.21.15.62:3300/0 conn(0x7f3c700d2eb0 0x7f3c7015ccb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:25.208 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:25.199+0000 7f3c74c2d640 1 -- 172.21.15.16:0/4262512572 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c700d26b0 con 0x7f3c700cef10 2023-05-22T15:39:25.208 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:25.199+0000 7f3c74c2d640 1 -- 172.21.15.16:0/4262512572 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7f3c68001a00 con 0x7f3c700cef10 2023-05-22T15:39:25.208 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:25.199+0000 7f3c74c2d640 1 -- 172.21.15.16:0/4262512572 >> v1:172.21.15.16:6789/0 conn(0x7f3c700cef10 legacy=0x7f3c700cf370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:25.208 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:25.199+0000 7f3c74c2d640 1 --2- 172.21.15.16:0/4262512572 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3c5c003980 0x7f3c5c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:25.208 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:25.199+0000 7f3c74c2d640 1 --2- 172.21.15.16:0/4262512572 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3c7015c8e0 0x7f3c5c0047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:25.209 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:25.199+0000 7f3c74c2d640 1 --2- 172.21.15.16:0/4262512572 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3c5c004d10 0x7f3c5c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:25.209 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:25.199+0000 7f3c74c2d640 1 -- 172.21.15.16:0/4262512572 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c700cd230 con 0x7f3c5c004d10 2023-05-22T15:39:25.209 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:25.199+0000 7f3c74c2d640 1 -- 172.21.15.16:0/4262512572 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c700d23d0 con 0x7f3c5c003980 2023-05-22T15:39:25.209 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:25.199+0000 7f3c74c2d640 1 -- 172.21.15.16:0/4262512572 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c5c00ba10 con 0x7f3c7015c8e0 2023-05-22T15:39:25.209 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:25.199+0000 7f3c76430640 1 --2- 172.21.15.16:0/4262512572 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3c5c004d10 0x7f3c5c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:25.209 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:25.199+0000 7f3c75c2f640 1 --2- 172.21.15.16:0/4262512572 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3c5c003980 0x7f3c5c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:25.209 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:25.199+0000 7f3c7542e640 1 --2- 172.21.15.16:0/4262512572 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3c7015c8e0 0x7f3c5c0047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:25.210 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:25.203+0000 7f3c76430640 1 -- 172.21.15.16:0/4262512572 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3c7015c8e0 msgr2=0x7f3c5c0047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:39:25.210 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:25.203+0000 7f3c76430640 1 --2- 172.21.15.16:0/4262512572 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3c7015c8e0 0x7f3c5c0047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:25.210 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:25.203+0000 7f3c76430640 1 -- 172.21.15.16:0/4262512572 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3c5c003980 msgr2=0x7f3c5c003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:39:25.210 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:25.203+0000 7f3c76430640 1 --2- 172.21.15.16:0/4262512572 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3c5c003980 0x7f3c5c003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:25.210 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:25.203+0000 7f3c76430640 1 -- 172.21.15.16:0/4262512572 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c5c003530 con 0x7f3c5c004d10 2023-05-22T15:39:25.212 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:25.203+0000 7f3c76430640 1 --2- 172.21.15.16:0/4262512572 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3c5c004d10 0x7f3c5c005180 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f3c6800a860 tx=0x7f3c6801fac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16008fc3a5baf13b server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:39:25.212 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:25.203+0000 7f3c74c2d640 1 -- 172.21.15.16:0/4262512572 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c680aa6f0 con 0x7f3c5c004d10 2023-05-22T15:39:25.212 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:25.203+0000 7f3c74c2d640 1 -- 172.21.15.16:0/4262512572 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c680aa850 con 0x7f3c5c004d10 2023-05-22T15:39:25.212 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:25.203+0000 7f3c74c2d640 1 -- 172.21.15.16:0/4262512572 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c680aac40 con 0x7f3c5c004d10 2023-05-22T15:39:25.212 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:25.203+0000 7f3c77e93640 1 -- 172.21.15.16:0/4262512572 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3c5c004d10 msgr2=0x7f3c5c005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:25.213 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:25.203+0000 7f3c77e93640 1 --2- 172.21.15.16:0/4262512572 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3c5c004d10 0x7f3c5c005180 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f3c6800a860 tx=0x7f3c6801fac0 comp rx=0 tx=0).stop 2023-05-22T15:39:25.213 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:25.203+0000 7f3c76430640 1 -- 172.21.15.16:0/4262512572 reap_dead start 2023-05-22T15:39:25.213 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:25.203+0000 7f3c77e93640 1 -- 172.21.15.16:0/4262512572 shutdown_connections 2023-05-22T15:39:25.213 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:25.203+0000 7f3c77e93640 1 -- 172.21.15.16:0/4262512572 >> 172.21.15.16:0/4262512572 conn(0x7f3c70070e50 msgr2=0x7f3c70071250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:39:25.213 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:25.203+0000 7f3c77e93640 1 -- 172.21.15.16:0/4262512572 shutdown_connections 2023-05-22T15:39:25.213 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:25.203+0000 7f3c77e93640 1 -- 172.21.15.16:0/4262512572 wait complete. 2023-05-22T15:39:25.213 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:25.207+0000 7f3c77e93640 1 Processor -- start 2023-05-22T15:39:25.213 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:25.207+0000 7f3c77e93640 1 -- start start 2023-05-22T15:39:25.214 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:25.207+0000 7f3c77e93640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3c5c004d10 0x7f3c701f0c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:25.214 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:25.207+0000 7f3c77e93640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3c700cb840 0x7f3c700cbcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:25.214 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:25.207+0000 7f3c77e93640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3c700ca200 0x7f3c700ca670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:25.214 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:25.207+0000 7f3c77e93640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c70144fc0 con 0x7f3c700cb840 2023-05-22T15:39:25.214 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:25.207+0000 7f3c77e93640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c700cc1f0 con 0x7f3c700ca200 2023-05-22T15:39:25.214 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:25.207+0000 7f3c77e93640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c700cb520 con 0x7f3c5c004d10 2023-05-22T15:39:25.214 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:25.207+0000 7f3c7542e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3c700cb840 0x7f3c700cbcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:25.215 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:25.207+0000 7f3c7542e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3c700cb840 0x7f3c700cbcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52416/0 (socket says 172.21.15.16:52416) 2023-05-22T15:39:25.215 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:25.207+0000 7f3c7542e640 1 -- 172.21.15.16:0/2752209017 learned_addr learned my addr 172.21.15.16:0/2752209017 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:39:25.215 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:25.207+0000 7f3c76430640 1 --2- 172.21.15.16:0/2752209017 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3c700ca200 0x7f3c700ca670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:25.215 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:25.207+0000 7f3c75c2f640 1 --2- 172.21.15.16:0/2752209017 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3c5c004d10 0x7f3c701f0c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:25.215 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:25.207+0000 7f3c7542e640 1 -- 172.21.15.16:0/2752209017 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3c5c004d10 msgr2=0x7f3c701f0c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:25.215 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:25.207+0000 7f3c7542e640 1 --2- 172.21.15.16:0/2752209017 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3c5c004d10 0x7f3c701f0c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:25.215 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:25.207+0000 7f3c7542e640 1 -- 172.21.15.16:0/2752209017 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3c700ca200 msgr2=0x7f3c700ca670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:25.215 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:25.207+0000 7f3c7542e640 1 --2- 172.21.15.16:0/2752209017 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3c700ca200 0x7f3c700ca670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:25.216 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:25.207+0000 7f3c7542e640 1 -- 172.21.15.16:0/2752209017 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c7015d1f0 con 0x7f3c700cb840 2023-05-22T15:39:25.216 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:25.207+0000 7f3c7542e640 1 --2- 172.21.15.16:0/2752209017 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3c700cb840 0x7f3c700cbcb0 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f3c6c011690 tx=0x7f3c6c011b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:39:25.216 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:25.207+0000 7f3c667fc640 1 -- 172.21.15.16:0/2752209017 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c6c00f960 con 0x7f3c700cb840 2023-05-22T15:39:25.216 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:25.207+0000 7f3c667fc640 1 -- 172.21.15.16:0/2752209017 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c6c00fac0 con 0x7f3c700cb840 2023-05-22T15:39:25.216 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:25.207+0000 7f3c667fc640 1 -- 172.21.15.16:0/2752209017 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c6c00fe30 con 0x7f3c700cb840 2023-05-22T15:39:25.217 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:25.207+0000 7f3c77e93640 1 -- 172.21.15.16:0/2752209017 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c700cd510 con 0x7f3c700cb840 2023-05-22T15:39:25.217 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:25.207+0000 7f3c77e93640 1 -- 172.21.15.16:0/2752209017 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c700d3300 con 0x7f3c700cb840 2023-05-22T15:39:25.217 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:25.207+0000 7f3c77e93640 1 -- 172.21.15.16:0/2752209017 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c7004dbe0 con 0x7f3c700cb840 2023-05-22T15:39:25.217 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:25.211+0000 7f3c667fc640 1 -- 172.21.15.16:0/2752209017 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 12) v1 ==== 84418+0+0 (secure 0 0 0) 0x7f3c6c015460 con 0x7f3c700cb840 2023-05-22T15:39:25.218 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:25.211+0000 7f3c667fc640 1 --2- 172.21.15.16:0/2752209017 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f3c540655b0 0x7f3c54067a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:25.218 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:25.211+0000 7f3c75c2f640 1 --2- 172.21.15.16:0/2752209017 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f3c540655b0 0x7f3c54067a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:25.218 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:25.211+0000 7f3c667fc640 1 -- 172.21.15.16:0/2752209017 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 6698+0+0 (secure 0 0 0) 0x7f3c6c0832c0 con 0x7f3c700cb840 2023-05-22T15:39:25.218 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:25.211+0000 7f3c75c2f640 1 --2- 172.21.15.16:0/2752209017 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f3c540655b0 0x7f3c54067a60 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f3c6000d9b0 tx=0x7f3c60008040 comp rx=0 tx=0).ready entity=mgr.5823 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:39:25.219 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:25.215+0000 7f3c667fc640 1 -- 172.21.15.16:0/2752209017 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f3c6c050b40 con 0x7f3c700cb840 2023-05-22T15:39:25.396 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:25.391+0000 7f3c77e93640 1 -- 172.21.15.16:0/2752209017 --> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/snap_test_dir1/dir0", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f3c700ccf70 con 0x7f3c540655b0 2023-05-22T15:39:25.413 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:25.407+0000 7f3c667fc640 1 -- 172.21.15.16:0/2752209017 <== mgr.5823 v2:172.21.15.16:6832/36921 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+345 (secure 0 0 0) 0x7f3c700ccf70 con 0x7f3c540655b0 2023-05-22T15:39:25.413 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:39:25.414 INFO:teuthology.orchestra.run.smithi016.stdout:[{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/dir0", "rel_path": "/snap_test_dir1/dir0", "schedule": "1M", "retention": {}, "start": "2023-05-22T00:00:00", "created": "2023-05-22T15:28:31", "first": "2023-05-22T15:29:01", "last": "2023-05-22T15:39:01", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2023-05-22T15:39:25.415 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:25.411+0000 7f3c77e93640 1 -- 172.21.15.16:0/2752209017 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f3c540655b0 msgr2=0x7f3c54067a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:25.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:25.411+0000 7f3c77e93640 1 --2- 172.21.15.16:0/2752209017 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f3c540655b0 0x7f3c54067a60 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f3c6000d9b0 tx=0x7f3c60008040 comp rx=0 tx=0).stop 2023-05-22T15:39:25.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:25.411+0000 7f3c77e93640 1 -- 172.21.15.16:0/2752209017 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3c700cb840 msgr2=0x7f3c700cbcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:25.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:25.411+0000 7f3c77e93640 1 --2- 172.21.15.16:0/2752209017 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3c700cb840 0x7f3c700cbcb0 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f3c6c011690 tx=0x7f3c6c011b50 comp rx=0 tx=0).stop 2023-05-22T15:39:25.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:25.411+0000 7f3c77e93640 1 -- 172.21.15.16:0/2752209017 shutdown_connections 2023-05-22T15:39:25.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:25.411+0000 7f3c77e93640 1 --2- 172.21.15.16:0/2752209017 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3c700ca200 0x7f3c700ca670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:25.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:25.411+0000 7f3c77e93640 1 --2- 172.21.15.16:0/2752209017 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3c700cb840 0x7f3c700cbcb0 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:25.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:25.411+0000 7f3c77e93640 1 --2- 172.21.15.16:0/2752209017 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f3c540655b0 0x7f3c54067a60 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:25.417 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:25.411+0000 7f3c77e93640 1 --2- 172.21.15.16:0/2752209017 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3c5c004d10 0x7f3c701f0c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:25.417 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:25.411+0000 7f3c77e93640 1 -- 172.21.15.16:0/2752209017 >> 172.21.15.16:0/2752209017 conn(0x7f3c70070e50 msgr2=0x7f3c700cd730 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:39:25.417 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:25.411+0000 7f3c77e93640 1 -- 172.21.15.16:0/2752209017 shutdown_connections 2023-05-22T15:39:25.417 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:25.411+0000 7f3c77e93640 1 -- 172.21.15.16:0/2752209017 wait complete. 2023-05-22T15:39:25.432 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/dir0", "rel_path": "/snap_test_dir1/dir0", "schedule": "1M", "retention": {}, "start": "2023-05-22T00:00:00", "created": "2023-05-22T15:28:31", "first": "2023-05-22T15:29:01", "last": "2023-05-22T15:39:01", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2023-05-22T15:39:25.433 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'path': '/snap_test_dir1/dir0', 'rel_path': '/snap_test_dir1/dir0', 'schedule': '1M', 'retention': {}, 'start': '2023-05-22T00:00:00', 'created': '2023-05-22T15:28:31', 'first': '2023-05-22T15:29:01', 'last': '2023-05-22T15:39:01', 'last_pruned': None, 'created_count': 11, 'pruned_count': 0, 'active': False} 2023-05-22T15:39:25.433 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:39:25.434 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir1/.snap) 2023-05-22T15:39:25.444 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_29_00_UTC 2023-05-22T15:39:25.444 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_30_00_UTC 2023-05-22T15:39:25.444 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_31_00_UTC 2023-05-22T15:39:25.444 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_32_00_UTC 2023-05-22T15:39:25.445 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_33_01_UTC 2023-05-22T15:39:25.445 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_34_00_UTC 2023-05-22T15:39:25.445 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_35_00_UTC 2023-05-22T15:39:25.445 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_36_00_UTC 2023-05-22T15:39:25.445 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_37_00_UTC 2023-05-22T15:39:25.445 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_38_00_UTC 2023-05-22T15:39:25.445 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_39_00_UTC 2023-05-22T15:39:25.446 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2023-05-22-15_29_00_UTC', 'scheduled-2023-05-22-15_30_00_UTC', 'scheduled-2023-05-22-15_31_00_UTC', 'scheduled-2023-05-22-15_32_00_UTC', 'scheduled-2023-05-22-15_33_01_UTC', 'scheduled-2023-05-22-15_34_00_UTC', 'scheduled-2023-05-22-15_35_00_UTC', 'scheduled-2023-05-22-15_36_00_UTC', 'scheduled-2023-05-22-15_37_00_UTC', 'scheduled-2023-05-22-15_38_00_UTC', 'scheduled-2023-05-22-15_39_00_UTC'] 2023-05-22T15:39:25.447 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --fs cephfs --format json /snap_test_dir1/dir1 2023-05-22T15:39:25.619 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:25.615+0000 7f530be08640 1 Processor -- start 2023-05-22T15:39:25.620 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:25.615+0000 7f530be08640 1 -- start start 2023-05-22T15:39:25.621 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:25.615+0000 7f530be08640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f5304159ce0 0x7f53040c3e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:25.621 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:25.615+0000 7f530be08640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5304070300 con 0x7f53040b1d80 2023-05-22T15:39:25.621 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:25.615+0000 7f530be08640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53040cf0f0 con 0x7f53040ccce0 2023-05-22T15:39:25.621 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:25.615+0000 7f530be08640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53040b2240 con 0x7f5304159ce0 2023-05-22T15:39:25.621 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:25.615+0000 7f5309ba4640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f5304159ce0 0x7f53040c3e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:25.621 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:25.615+0000 7f53093a3640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f53040b1d80 0x7f530406fbf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:53042/0 (socket says 172.21.15.16:53042) 2023-05-22T15:39:25.621 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:25.615+0000 7f53093a3640 1 -- 172.21.15.16:0/3164127543 learned_addr learned my addr 172.21.15.16:0/3164127543 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:39:25.622 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:25.615+0000 7f5309ba4640 1 -- 172.21.15.16:0/3164127543 >> v1:172.21.15.62:6789/0 conn(0x7f53040ccce0 legacy=0x7f53040cd120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:39:25.622 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:25.615+0000 7f5309ba4640 1 -- 172.21.15.16:0/3164127543 >> v1:172.21.15.16:6789/0 conn(0x7f53040b1d80 legacy=0x7f530406fbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:25.622 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:25.615+0000 7f5309ba4640 1 -- 172.21.15.16:0/3164127543 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53040b2520 con 0x7f5304159ce0 2023-05-22T15:39:25.622 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:25.615+0000 7f5308ba2640 1 -- 172.21.15.16:0/3164127543 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 40976219 0 0) 0x7f5304070300 con 0x7f53040b1d80 2023-05-22T15:39:25.625 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:25.619+0000 7f5309ba4640 1 --2- 172.21.15.16:0/3164127543 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5304159ce0 0x7f53040c3e50 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f53000097c0 tx=0x7f530001c410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed8d2c31e8702242 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:39:25.625 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:25.619+0000 7f5308ba2640 1 -- 172.21.15.16:0/3164127543 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53000ab610 con 0x7f5304159ce0 2023-05-22T15:39:25.625 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:25.619+0000 7f5308ba2640 1 -- 172.21.15.16:0/3164127543 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53000ab770 con 0x7f5304159ce0 2023-05-22T15:39:25.625 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:25.619+0000 7f5308ba2640 1 -- 172.21.15.16:0/3164127543 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53000abb00 con 0x7f5304159ce0 2023-05-22T15:39:25.626 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:25.619+0000 7f530be08640 1 -- 172.21.15.16:0/3164127543 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5304159ce0 msgr2=0x7f53040c3e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:25.626 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:25.619+0000 7f530be08640 1 --2- 172.21.15.16:0/3164127543 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5304159ce0 0x7f53040c3e50 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f53000097c0 tx=0x7f530001c410 comp rx=0 tx=0).stop 2023-05-22T15:39:25.627 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:25.619+0000 7f530be08640 1 -- 172.21.15.16:0/3164127543 shutdown_connections 2023-05-22T15:39:25.627 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:25.619+0000 7f530be08640 1 --2- 172.21.15.16:0/3164127543 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5304159ce0 0x7f53040c3e50 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:25.627 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:25.619+0000 7f530be08640 1 -- 172.21.15.16:0/3164127543 >> 172.21.15.16:0/3164127543 conn(0x7f530406d9d0 msgr2=0x7f530406ddd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:39:25.627 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:25.619+0000 7f530be08640 1 -- 172.21.15.16:0/3164127543 shutdown_connections 2023-05-22T15:39:25.627 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:25.619+0000 7f530be08640 1 -- 172.21.15.16:0/3164127543 wait complete. 2023-05-22T15:39:25.628 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:25.623+0000 7f530be08640 1 Processor -- start 2023-05-22T15:39:25.629 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:25.623+0000 7f530be08640 1 -- start start 2023-05-22T15:39:25.629 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:25.623+0000 7f530be08640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f53040b1d80 0x7f5304145bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:25.629 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:25.623+0000 7f530be08640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f53040ccce0 0x7f53040c6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:25.629 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:25.623+0000 7f530be08640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5304159ce0 0x7f5304150cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:25.629 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:25.623+0000 7f530be08640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5304146100 con 0x7f53040b1d80 2023-05-22T15:39:25.629 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:25.623+0000 7f530be08640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f530415a0b0 con 0x7f53040ccce0 2023-05-22T15:39:25.629 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:25.623+0000 7f530be08640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53040c6ae0 con 0x7f5304159ce0 2023-05-22T15:39:25.629 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:25.623+0000 7f5309ba4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f53040b1d80 0x7f5304145bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:25.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:25.623+0000 7f5309ba4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f53040b1d80 0x7f5304145bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52432/0 (socket says 172.21.15.16:52432) 2023-05-22T15:39:25.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:25.623+0000 7f5309ba4640 1 -- 172.21.15.16:0/3807458664 learned_addr learned my addr 172.21.15.16:0/3807458664 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:39:25.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:25.623+0000 7f53093a3640 1 --2- 172.21.15.16:0/3807458664 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f53040ccce0 0x7f53040c6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:25.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:25.623+0000 7f530a3a5640 1 --2- 172.21.15.16:0/3807458664 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5304159ce0 0x7f5304150cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:25.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:25.623+0000 7f5309ba4640 1 -- 172.21.15.16:0/3807458664 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5304159ce0 msgr2=0x7f5304150cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:25.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:25.623+0000 7f5309ba4640 1 --2- 172.21.15.16:0/3807458664 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5304159ce0 0x7f5304150cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:25.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:25.623+0000 7f5309ba4640 1 -- 172.21.15.16:0/3807458664 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f53040ccce0 msgr2=0x7f53040c6260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:25.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:25.623+0000 7f5309ba4640 1 --2- 172.21.15.16:0/3807458664 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f53040ccce0 0x7f53040c6260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:25.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:25.623+0000 7f5309ba4640 1 -- 172.21.15.16:0/3807458664 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5300009420 con 0x7f53040b1d80 2023-05-22T15:39:25.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:25.623+0000 7f53093a3640 1 --2- 172.21.15.16:0/3807458664 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f53040ccce0 0x7f53040c6260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T15:39:25.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:25.623+0000 7f5309ba4640 1 --2- 172.21.15.16:0/3807458664 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f53040b1d80 0x7f5304145bc0 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f53000ad040 tx=0x7f530001ac50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:39:25.632 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:25.623+0000 7f52fa7fc640 1 -- 172.21.15.16:0/3807458664 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53000b5810 con 0x7f53040b1d80 2023-05-22T15:39:25.632 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:25.623+0000 7f52fa7fc640 1 -- 172.21.15.16:0/3807458664 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53000b59b0 con 0x7f53040b1d80 2023-05-22T15:39:25.632 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:25.623+0000 7f530be08640 1 -- 172.21.15.16:0/3807458664 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53040c4520 con 0x7f53040b1d80 2023-05-22T15:39:25.632 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:25.623+0000 7f52fa7fc640 1 -- 172.21.15.16:0/3807458664 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53000b5d60 con 0x7f53040b1d80 2023-05-22T15:39:25.633 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:25.623+0000 7f530be08640 1 -- 172.21.15.16:0/3807458664 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53040c4390 con 0x7f53040b1d80 2023-05-22T15:39:25.633 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:25.623+0000 7f530be08640 1 -- 172.21.15.16:0/3807458664 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53040cfde0 con 0x7f53040b1d80 2023-05-22T15:39:25.633 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:25.627+0000 7f52fa7fc640 1 -- 172.21.15.16:0/3807458664 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 12) v1 ==== 84418+0+0 (secure 0 0 0) 0x7f53000bf410 con 0x7f53040b1d80 2023-05-22T15:39:25.633 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:25.627+0000 7f52fa7fc640 1 --2- 172.21.15.16:0/3807458664 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f52e8065580 0x7f52e8067a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:25.633 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:25.627+0000 7f52fa7fc640 1 -- 172.21.15.16:0/3807458664 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 6698+0+0 (secure 0 0 0) 0x7f530011de50 con 0x7f53040b1d80 2023-05-22T15:39:25.633 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:25.627+0000 7f53093a3640 1 --2- 172.21.15.16:0/3807458664 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f52e8065580 0x7f52e8067a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:25.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:25.627+0000 7f53093a3640 1 --2- 172.21.15.16:0/3807458664 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f52e8065580 0x7f52e8067a30 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f52f4007fd0 tx=0x7f52f4007c20 comp rx=0 tx=0).ready entity=mgr.5823 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:39:25.635 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:25.631+0000 7f52fa7fc640 1 -- 172.21.15.16:0/3807458664 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f53000eb4f0 con 0x7f53040b1d80 2023-05-22T15:39:25.808 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:25.803+0000 7f530be08640 1 -- 172.21.15.16:0/3807458664 --> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/snap_test_dir1/dir1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f5304067a70 con 0x7f52e8065580 2023-05-22T15:39:25.824 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:25.819+0000 7f52fa7fc640 1 -- 172.21.15.16:0/3807458664 <== mgr.5823 v2:172.21.15.16:6832/36921 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+345 (secure 0 0 0) 0x7f5304067a70 con 0x7f52e8065580 2023-05-22T15:39:25.825 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:39:25.825 INFO:teuthology.orchestra.run.smithi016.stdout:[{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/dir1", "rel_path": "/snap_test_dir1/dir1", "schedule": "1M", "retention": {}, "start": "2023-05-22T00:00:00", "created": "2023-05-22T15:28:32", "first": "2023-05-22T15:29:00", "last": "2023-05-22T15:39:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2023-05-22T15:39:25.826 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:25.823+0000 7f530be08640 1 -- 172.21.15.16:0/3807458664 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f52e8065580 msgr2=0x7f52e8067a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:25.827 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:25.823+0000 7f530be08640 1 --2- 172.21.15.16:0/3807458664 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f52e8065580 0x7f52e8067a30 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f52f4007fd0 tx=0x7f52f4007c20 comp rx=0 tx=0).stop 2023-05-22T15:39:25.827 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:25.823+0000 7f530be08640 1 -- 172.21.15.16:0/3807458664 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f53040b1d80 msgr2=0x7f5304145bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:25.827 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:25.823+0000 7f530be08640 1 --2- 172.21.15.16:0/3807458664 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f53040b1d80 0x7f5304145bc0 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f53000ad040 tx=0x7f530001ac50 comp rx=0 tx=0).stop 2023-05-22T15:39:25.827 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:25.823+0000 7f530be08640 1 -- 172.21.15.16:0/3807458664 shutdown_connections 2023-05-22T15:39:25.827 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:25.823+0000 7f530be08640 1 --2- 172.21.15.16:0/3807458664 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f52e8065580 0x7f52e8067a30 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:25.827 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:25.823+0000 7f530be08640 1 --2- 172.21.15.16:0/3807458664 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5304159ce0 0x7f5304150cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:25.827 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:25.823+0000 7f530be08640 1 --2- 172.21.15.16:0/3807458664 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f53040ccce0 0x7f53040c6260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:25.828 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:25.823+0000 7f530be08640 1 --2- 172.21.15.16:0/3807458664 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f53040b1d80 0x7f5304145bc0 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:25.828 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:25.823+0000 7f530be08640 1 -- 172.21.15.16:0/3807458664 >> 172.21.15.16:0/3807458664 conn(0x7f530406d9d0 msgr2=0x7f53040ce800 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:39:25.828 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:25.823+0000 7f530be08640 1 -- 172.21.15.16:0/3807458664 shutdown_connections 2023-05-22T15:39:25.828 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:25.823+0000 7f530be08640 1 -- 172.21.15.16:0/3807458664 wait complete. 2023-05-22T15:39:25.844 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/dir1", "rel_path": "/snap_test_dir1/dir1", "schedule": "1M", "retention": {}, "start": "2023-05-22T00:00:00", "created": "2023-05-22T15:28:32", "first": "2023-05-22T15:29:00", "last": "2023-05-22T15:39:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2023-05-22T15:39:25.844 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'path': '/snap_test_dir1/dir1', 'rel_path': '/snap_test_dir1/dir1', 'schedule': '1M', 'retention': {}, 'start': '2023-05-22T00:00:00', 'created': '2023-05-22T15:28:32', 'first': '2023-05-22T15:29:00', 'last': '2023-05-22T15:39:00', 'last_pruned': None, 'created_count': 11, 'pruned_count': 0, 'active': False} 2023-05-22T15:39:25.845 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:39:25.845 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir2/.snap) 2023-05-22T15:39:25.856 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_29_00_UTC 2023-05-22T15:39:25.856 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_30_00_UTC 2023-05-22T15:39:25.856 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_31_00_UTC 2023-05-22T15:39:25.856 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_32_00_UTC 2023-05-22T15:39:25.856 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_33_00_UTC 2023-05-22T15:39:25.857 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_34_00_UTC 2023-05-22T15:39:25.857 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_35_00_UTC 2023-05-22T15:39:25.857 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_36_00_UTC 2023-05-22T15:39:25.857 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_37_00_UTC 2023-05-22T15:39:25.857 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_38_00_UTC 2023-05-22T15:39:25.857 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_39_00_UTC 2023-05-22T15:39:25.858 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2023-05-22-15_29_00_UTC', 'scheduled-2023-05-22-15_30_00_UTC', 'scheduled-2023-05-22-15_31_00_UTC', 'scheduled-2023-05-22-15_32_00_UTC', 'scheduled-2023-05-22-15_33_00_UTC', 'scheduled-2023-05-22-15_34_00_UTC', 'scheduled-2023-05-22-15_35_00_UTC', 'scheduled-2023-05-22-15_36_00_UTC', 'scheduled-2023-05-22-15_37_00_UTC', 'scheduled-2023-05-22-15_38_00_UTC', 'scheduled-2023-05-22-15_39_00_UTC'] 2023-05-22T15:39:25.859 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --fs cephfs --format json /snap_test_dir1/dir2 2023-05-22T15:39:26.032 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.027+0000 7f2f5e6bc640 1 Processor -- start 2023-05-22T15:39:26.032 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.027+0000 7f2f5e6bc640 1 -- start start 2023-05-22T15:39:26.034 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.027+0000 7f2f5e6bc640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f2f580cc380 0x7f2f580c33f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:26.034 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.027+0000 7f2f5e6bc640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f580c30e0 con 0x7f2f58158340 2023-05-22T15:39:26.034 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.027+0000 7f2f5e6bc640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f580c3a40 con 0x7f2f58157f70 2023-05-22T15:39:26.034 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.027+0000 7f2f5e6bc640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f5806f770 con 0x7f2f580cc380 2023-05-22T15:39:26.035 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.027+0000 7f2f5cc59640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f2f58158340 0x7f2f5806fbc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:53054/0 (socket says 172.21.15.16:53054) 2023-05-22T15:39:26.035 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.027+0000 7f2f5cc59640 1 -- 172.21.15.16:0/2106106292 learned_addr learned my addr 172.21.15.16:0/2106106292 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:39:26.035 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.027+0000 7f2f577fe640 1 --2- 172.21.15.16:0/2106106292 >> v2:172.21.15.62:3300/0 conn(0x7f2f580cc380 0x7f2f580c33f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:26.035 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.027+0000 7f2f56ffd640 1 -- 172.21.15.16:0/2106106292 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1160009199 0 0) 0x7f2f580c30e0 con 0x7f2f58158340 2023-05-22T15:39:26.035 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.027+0000 7f2f56ffd640 1 -- 172.21.15.16:0/2106106292 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f40003450 con 0x7f2f58158340 2023-05-22T15:39:26.035 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.027+0000 7f2f56ffd640 1 -- 172.21.15.16:0/2106106292 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3128526839 0 0) 0x7f2f40003450 con 0x7f2f58158340 2023-05-22T15:39:26.035 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.027+0000 7f2f56ffd640 1 -- 172.21.15.16:0/2106106292 >> v2:172.21.15.62:3300/0 conn(0x7f2f580cc380 msgr2=0x7f2f580c33f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:39:26.036 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.027+0000 7f2f56ffd640 1 --2- 172.21.15.16:0/2106106292 >> v2:172.21.15.62:3300/0 conn(0x7f2f580cc380 0x7f2f580c33f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:26.036 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.027+0000 7f2f56ffd640 1 -- 172.21.15.16:0/2106106292 >> v1:172.21.15.62:6789/0 conn(0x7f2f58157f70 legacy=0x7f2f580cbc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:26.036 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.027+0000 7f2f56ffd640 1 -- 172.21.15.16:0/2106106292 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f580c3d20 con 0x7f2f58158340 2023-05-22T15:39:26.036 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.027+0000 7f2f56ffd640 1 -- 172.21.15.16:0/2106106292 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7f2f50001a00 con 0x7f2f58158340 2023-05-22T15:39:26.036 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.027+0000 7f2f56ffd640 1 -- 172.21.15.16:0/2106106292 >> v1:172.21.15.16:6789/0 conn(0x7f2f58158340 legacy=0x7f2f5806fbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:26.036 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.027+0000 7f2f56ffd640 1 --2- 172.21.15.16:0/2106106292 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2f40003a70 0x7f2f40003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:26.036 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.027+0000 7f2f56ffd640 1 --2- 172.21.15.16:0/2106106292 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2f40008860 0x7f2f4000ac80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:26.036 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.027+0000 7f2f56ffd640 1 --2- 172.21.15.16:0/2106106292 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f4000b480 0x7f2f4000d930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:26.037 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.027+0000 7f2f56ffd640 1 -- 172.21.15.16:0/2106106292 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f4000e1e0 con 0x7f2f4000b480 2023-05-22T15:39:26.037 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.027+0000 7f2f56ffd640 1 -- 172.21.15.16:0/2106106292 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f4000e350 con 0x7f2f40003a70 2023-05-22T15:39:26.037 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.027+0000 7f2f56ffd640 1 -- 172.21.15.16:0/2106106292 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f4000e510 con 0x7f2f40008860 2023-05-22T15:39:26.037 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.027+0000 7f2f5cc59640 1 --2- 172.21.15.16:0/2106106292 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f4000b480 0x7f2f4000d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:26.037 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.027+0000 7f2f57fff640 1 --2- 172.21.15.16:0/2106106292 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2f40003a70 0x7f2f40003e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:26.037 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.027+0000 7f2f5cc59640 1 -- 172.21.15.16:0/2106106292 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2f40008860 msgr2=0x7f2f4000ac80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:39:26.037 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.027+0000 7f2f577fe640 1 --2- 172.21.15.16:0/2106106292 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2f40008860 0x7f2f4000ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:26.038 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.027+0000 7f2f5cc59640 1 --2- 172.21.15.16:0/2106106292 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2f40008860 0x7f2f4000ac80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:26.038 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.027+0000 7f2f5cc59640 1 -- 172.21.15.16:0/2106106292 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2f40003a70 msgr2=0x7f2f40003e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:39:26.038 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.027+0000 7f2f5cc59640 1 --2- 172.21.15.16:0/2106106292 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2f40003a70 0x7f2f40003e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:26.038 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.027+0000 7f2f5cc59640 1 -- 172.21.15.16:0/2106106292 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f40003450 con 0x7f2f4000b480 2023-05-22T15:39:26.038 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.027+0000 7f2f57fff640 1 --2- 172.21.15.16:0/2106106292 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2f40003a70 0x7f2f40003e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:39:26.038 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.027+0000 7f2f577fe640 1 --2- 172.21.15.16:0/2106106292 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2f40008860 0x7f2f4000ac80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:39:26.040 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.031+0000 7f2f5cc59640 1 -- 172.21.15.16:0/2106106292 reap_dead start 2023-05-22T15:39:26.041 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.031+0000 7f2f5cc59640 1 --2- 172.21.15.16:0/2106106292 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f4000b480 0x7f2f4000d930 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f2f580c86a0 tx=0x7f2f5001d2e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8bb23f3b17e3d2e4 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:39:26.041 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.031+0000 7f2f56ffd640 1 -- 172.21.15.16:0/2106106292 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f500072d0 con 0x7f2f4000b480 2023-05-22T15:39:26.041 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.031+0000 7f2f56ffd640 1 -- 172.21.15.16:0/2106106292 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f500a8070 con 0x7f2f4000b480 2023-05-22T15:39:26.041 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.031+0000 7f2f56ffd640 1 -- 172.21.15.16:0/2106106292 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f50007670 con 0x7f2f4000b480 2023-05-22T15:39:26.041 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.031+0000 7f2f5e6bc640 1 -- 172.21.15.16:0/2106106292 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f4000b480 msgr2=0x7f2f4000d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:26.041 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.031+0000 7f2f5e6bc640 1 --2- 172.21.15.16:0/2106106292 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f4000b480 0x7f2f4000d930 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f2f580c86a0 tx=0x7f2f5001d2e0 comp rx=0 tx=0).stop 2023-05-22T15:39:26.042 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.031+0000 7f2f5e6bc640 1 -- 172.21.15.16:0/2106106292 shutdown_connections 2023-05-22T15:39:26.042 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.031+0000 7f2f5e6bc640 1 --2- 172.21.15.16:0/2106106292 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f4000b480 0x7f2f4000d930 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:26.042 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.031+0000 7f2f5e6bc640 1 -- 172.21.15.16:0/2106106292 >> 172.21.15.16:0/2106106292 conn(0x7f2f58069720 msgr2=0x7f2f58069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:39:26.042 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.031+0000 7f2f5e6bc640 1 -- 172.21.15.16:0/2106106292 shutdown_connections 2023-05-22T15:39:26.042 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.031+0000 7f2f5e6bc640 1 -- 172.21.15.16:0/2106106292 wait complete. 2023-05-22T15:39:26.042 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.035+0000 7f2f5e6bc640 1 Processor -- start 2023-05-22T15:39:26.042 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.035+0000 7f2f5e6bc640 1 -- start start 2023-05-22T15:39:26.043 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.035+0000 7f2f5e6bc640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2f4000b480 0x7f2f58146260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:26.043 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.035+0000 7f2f5e6bc640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2f581467a0 0x7f2f58146c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:26.043 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.035+0000 7f2f5e6bc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f581498c0 0x7f2f580c9750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:26.043 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.035+0000 7f2f5e6bc640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f5815ae40 con 0x7f2f581498c0 2023-05-22T15:39:26.043 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.035+0000 7f2f5e6bc640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f5815b0b0 con 0x7f2f581467a0 2023-05-22T15:39:26.043 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.035+0000 7f2f5e6bc640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f5815b390 con 0x7f2f4000b480 2023-05-22T15:39:26.043 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.035+0000 7f2f5cc59640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f581498c0 0x7f2f580c9750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:26.043 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.035+0000 7f2f5cc59640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f581498c0 0x7f2f580c9750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52452/0 (socket says 172.21.15.16:52452) 2023-05-22T15:39:26.044 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.035+0000 7f2f5cc59640 1 -- 172.21.15.16:0/164992816 learned_addr learned my addr 172.21.15.16:0/164992816 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:39:26.044 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.035+0000 7f2f57fff640 1 --2- 172.21.15.16:0/164992816 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2f4000b480 0x7f2f58146260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:26.044 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.035+0000 7f2f577fe640 1 --2- 172.21.15.16:0/164992816 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2f581467a0 0x7f2f58146c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:26.044 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.035+0000 7f2f5cc59640 1 -- 172.21.15.16:0/164992816 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2f4000b480 msgr2=0x7f2f58146260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:26.044 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.035+0000 7f2f5cc59640 1 --2- 172.21.15.16:0/164992816 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2f4000b480 0x7f2f58146260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:26.044 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.035+0000 7f2f5cc59640 1 -- 172.21.15.16:0/164992816 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2f581467a0 msgr2=0x7f2f58146c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:26.044 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.035+0000 7f2f5cc59640 1 --2- 172.21.15.16:0/164992816 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2f581467a0 0x7f2f58146c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:26.045 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.035+0000 7f2f5cc59640 1 -- 172.21.15.16:0/164992816 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f580c42e0 con 0x7f2f581498c0 2023-05-22T15:39:26.045 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.035+0000 7f2f577fe640 1 --2- 172.21.15.16:0/164992816 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2f581467a0 0x7f2f58146c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T15:39:26.046 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.035+0000 7f2f5cc59640 1 --2- 172.21.15.16:0/164992816 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f581498c0 0x7f2f580c9750 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f2f50007040 tx=0x7f2f40003700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:39:26.046 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.035+0000 7f2f54ff9640 1 -- 172.21.15.16:0/164992816 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f50001b50 con 0x7f2f581498c0 2023-05-22T15:39:26.046 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.035+0000 7f2f54ff9640 1 -- 172.21.15.16:0/164992816 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f500070d0 con 0x7f2f581498c0 2023-05-22T15:39:26.046 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.035+0000 7f2f5e6bc640 1 -- 172.21.15.16:0/164992816 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f580c4000 con 0x7f2f581498c0 2023-05-22T15:39:26.046 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.035+0000 7f2f54ff9640 1 -- 172.21.15.16:0/164992816 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f50007480 con 0x7f2f581498c0 2023-05-22T15:39:26.046 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.035+0000 7f2f5e6bc640 1 -- 172.21.15.16:0/164992816 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f5815bcf0 con 0x7f2f581498c0 2023-05-22T15:39:26.047 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.035+0000 7f2f5e6bc640 1 -- 172.21.15.16:0/164992816 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f580c9470 con 0x7f2f581498c0 2023-05-22T15:39:26.047 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.035+0000 7f2f54ff9640 1 -- 172.21.15.16:0/164992816 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 12) v1 ==== 84418+0+0 (secure 0 0 0) 0x7f2f500aa070 con 0x7f2f581498c0 2023-05-22T15:39:26.048 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.039+0000 7f2f54ff9640 1 --2- 172.21.15.16:0/164992816 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f2f44065580 0x7f2f44067a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:26.048 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.039+0000 7f2f54ff9640 1 -- 172.21.15.16:0/164992816 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 6698+0+0 (secure 0 0 0) 0x7f2f5011c5e0 con 0x7f2f581498c0 2023-05-22T15:39:26.048 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.039+0000 7f2f57fff640 1 --2- 172.21.15.16:0/164992816 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f2f44065580 0x7f2f44067a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:26.048 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.039+0000 7f2f57fff640 1 --2- 172.21.15.16:0/164992816 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f2f44065580 0x7f2f44067a30 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f2f4c007fd0 tx=0x7f2f4c009040 comp rx=0 tx=0).ready entity=mgr.5823 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:39:26.048 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.043+0000 7f2f54ff9640 1 -- 172.21.15.16:0/164992816 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f2f580c9470 con 0x7f2f581498c0 2023-05-22T15:39:26.212 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.207+0000 7f2f5e6bc640 1 -- 172.21.15.16:0/164992816 --> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/snap_test_dir1/dir2", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f2f5815ae40 con 0x7f2f44065580 2023-05-22T15:39:26.227 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.223+0000 7f2f54ff9640 1 -- 172.21.15.16:0/164992816 <== mgr.5823 v2:172.21.15.16:6832/36921 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+345 (secure 0 0 0) 0x7f2f5815ae40 con 0x7f2f44065580 2023-05-22T15:39:26.228 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:39:26.228 INFO:teuthology.orchestra.run.smithi016.stdout:[{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/dir2", "rel_path": "/snap_test_dir1/dir2", "schedule": "1M", "retention": {}, "start": "2023-05-22T00:00:00", "created": "2023-05-22T15:28:32", "first": "2023-05-22T15:29:00", "last": "2023-05-22T15:39:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2023-05-22T15:39:26.229 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.223+0000 7f2f5e6bc640 1 -- 172.21.15.16:0/164992816 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f2f44065580 msgr2=0x7f2f44067a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:26.229 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.223+0000 7f2f5e6bc640 1 --2- 172.21.15.16:0/164992816 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f2f44065580 0x7f2f44067a30 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f2f4c007fd0 tx=0x7f2f4c009040 comp rx=0 tx=0).stop 2023-05-22T15:39:26.230 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.223+0000 7f2f5e6bc640 1 -- 172.21.15.16:0/164992816 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f581498c0 msgr2=0x7f2f580c9750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:26.230 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.223+0000 7f2f5e6bc640 1 --2- 172.21.15.16:0/164992816 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f581498c0 0x7f2f580c9750 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f2f50007040 tx=0x7f2f40003700 comp rx=0 tx=0).stop 2023-05-22T15:39:26.230 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.223+0000 7f2f5e6bc640 1 -- 172.21.15.16:0/164992816 shutdown_connections 2023-05-22T15:39:26.230 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.223+0000 7f2f5e6bc640 1 --2- 172.21.15.16:0/164992816 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f581498c0 0x7f2f580c9750 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:26.230 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.223+0000 7f2f5e6bc640 1 --2- 172.21.15.16:0/164992816 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2f581467a0 0x7f2f58146c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:26.230 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.223+0000 7f2f5e6bc640 1 --2- 172.21.15.16:0/164992816 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f2f44065580 0x7f2f44067a30 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:26.230 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.223+0000 7f2f5e6bc640 1 --2- 172.21.15.16:0/164992816 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2f4000b480 0x7f2f58146260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:26.231 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.223+0000 7f2f5e6bc640 1 -- 172.21.15.16:0/164992816 >> 172.21.15.16:0/164992816 conn(0x7f2f58069720 msgr2=0x7f2f58071070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:39:26.231 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.223+0000 7f2f5e6bc640 1 -- 172.21.15.16:0/164992816 shutdown_connections 2023-05-22T15:39:26.231 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.223+0000 7f2f5e6bc640 1 -- 172.21.15.16:0/164992816 wait complete. 2023-05-22T15:39:26.245 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/dir2", "rel_path": "/snap_test_dir1/dir2", "schedule": "1M", "retention": {}, "start": "2023-05-22T00:00:00", "created": "2023-05-22T15:28:32", "first": "2023-05-22T15:29:00", "last": "2023-05-22T15:39:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2023-05-22T15:39:26.246 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'path': '/snap_test_dir1/dir2', 'rel_path': '/snap_test_dir1/dir2', 'schedule': '1M', 'retention': {}, 'start': '2023-05-22T00:00:00', 'created': '2023-05-22T15:28:32', 'first': '2023-05-22T15:29:00', 'last': '2023-05-22T15:39:00', 'last_pruned': None, 'created_count': 11, 'pruned_count': 0, 'active': False} 2023-05-22T15:39:26.246 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:39:26.247 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir3/.snap) 2023-05-22T15:39:26.257 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_29_00_UTC 2023-05-22T15:39:26.257 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_30_00_UTC 2023-05-22T15:39:26.257 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_31_00_UTC 2023-05-22T15:39:26.257 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_32_00_UTC 2023-05-22T15:39:26.258 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_33_00_UTC 2023-05-22T15:39:26.258 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_34_00_UTC 2023-05-22T15:39:26.258 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_35_00_UTC 2023-05-22T15:39:26.258 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_36_00_UTC 2023-05-22T15:39:26.258 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_37_00_UTC 2023-05-22T15:39:26.258 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_38_00_UTC 2023-05-22T15:39:26.258 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_39_01_UTC 2023-05-22T15:39:26.259 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2023-05-22-15_29_00_UTC', 'scheduled-2023-05-22-15_30_00_UTC', 'scheduled-2023-05-22-15_31_00_UTC', 'scheduled-2023-05-22-15_32_00_UTC', 'scheduled-2023-05-22-15_33_00_UTC', 'scheduled-2023-05-22-15_34_00_UTC', 'scheduled-2023-05-22-15_35_00_UTC', 'scheduled-2023-05-22-15_36_00_UTC', 'scheduled-2023-05-22-15_37_00_UTC', 'scheduled-2023-05-22-15_38_00_UTC', 'scheduled-2023-05-22-15_39_01_UTC'] 2023-05-22T15:39:26.260 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --fs cephfs --format json /snap_test_dir1/dir3 2023-05-22T15:39:26.430 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.423+0000 7f382cd0d640 1 Processor -- start 2023-05-22T15:39:26.430 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.423+0000 7f382cd0d640 1 -- start start 2023-05-22T15:39:26.432 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.427+0000 7f382cd0d640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3828158950 0x7f38280cb900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:26.432 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.427+0000 7f382cd0d640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f382815d360 con 0x7f38280cb440 2023-05-22T15:39:26.432 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.427+0000 7f382cd0d640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3828070bb0 con 0x7f38280cdd10 2023-05-22T15:39:26.433 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.427+0000 7f382cd0d640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3828070e90 con 0x7f3828158950 2023-05-22T15:39:26.433 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.427+0000 7f3825d9b640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f38280cb440 0x7f382815c540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:53064/0 (socket says 172.21.15.16:53064) 2023-05-22T15:39:26.433 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.427+0000 7f3825d9b640 1 -- 172.21.15.16:0/3681025652 learned_addr learned my addr 172.21.15.16:0/3681025652 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:39:26.433 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.427+0000 7f382659c640 1 --2- 172.21.15.16:0/3681025652 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3828158950 0x7f38280cb900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:26.433 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.427+0000 7f382559a640 1 -- 172.21.15.16:0/3681025652 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3740451296 0 0) 0x7f382815d360 con 0x7f38280cb440 2023-05-22T15:39:26.433 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.427+0000 7f382559a640 1 -- 172.21.15.16:0/3681025652 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3810003530 con 0x7f38280cb440 2023-05-22T15:39:26.433 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.427+0000 7f382559a640 1 -- 172.21.15.16:0/3681025652 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3383716298 0 0) 0x7f3810003530 con 0x7f38280cb440 2023-05-22T15:39:26.433 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.427+0000 7f382559a640 1 -- 172.21.15.16:0/3681025652 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3828158950 msgr2=0x7f38280cb900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:39:26.434 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.427+0000 7f382559a640 1 --2- 172.21.15.16:0/3681025652 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3828158950 0x7f38280cb900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:26.434 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.427+0000 7f382559a640 1 -- 172.21.15.16:0/3681025652 >> v1:172.21.15.62:6789/0 conn(0x7f38280cdd10 legacy=0x7f382815cc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:26.434 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.427+0000 7f382559a640 1 -- 172.21.15.16:0/3681025652 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3828071170 con 0x7f38280cb440 2023-05-22T15:39:26.434 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.427+0000 7f382559a640 1 -- 172.21.15.16:0/3681025652 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7f381c0016a0 con 0x7f38280cb440 2023-05-22T15:39:26.434 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.427+0000 7f382559a640 1 -- 172.21.15.16:0/3681025652 >> v1:172.21.15.16:6789/0 conn(0x7f38280cb440 legacy=0x7f382815c540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:26.434 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.427+0000 7f382559a640 1 --2- 172.21.15.16:0/3681025652 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38100039b0 0x7f3810003d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:26.434 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.427+0000 7f382559a640 1 --2- 172.21.15.16:0/3681025652 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3828158950 0x7f38100047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:26.435 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.427+0000 7f382559a640 1 --2- 172.21.15.16:0/3681025652 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3810004d10 0x7f3810005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:26.435 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.427+0000 7f382559a640 1 -- 172.21.15.16:0/3681025652 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3828070e90 con 0x7f38100039b0 2023-05-22T15:39:26.435 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.427+0000 7f382559a640 1 -- 172.21.15.16:0/3681025652 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f381000b990 con 0x7f3810004d10 2023-05-22T15:39:26.435 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.427+0000 7f382559a640 1 -- 172.21.15.16:0/3681025652 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f381000dd30 con 0x7f3828158950 2023-05-22T15:39:26.435 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.427+0000 7f382659c640 1 --2- 172.21.15.16:0/3681025652 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38100039b0 0x7f3810003d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:26.435 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.427+0000 7f3825d9b640 1 --2- 172.21.15.16:0/3681025652 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3828158950 0x7f38100047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:26.435 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.427+0000 7f3826d9d640 1 --2- 172.21.15.16:0/3681025652 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3810004d10 0x7f3810005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:26.435 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.427+0000 7f382659c640 1 -- 172.21.15.16:0/3681025652 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3828158950 msgr2=0x7f38100047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:39:26.436 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.427+0000 7f382659c640 1 --2- 172.21.15.16:0/3681025652 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3828158950 0x7f38100047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:26.436 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.427+0000 7f382659c640 1 -- 172.21.15.16:0/3681025652 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3810004d10 msgr2=0x7f3810005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:39:26.436 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.427+0000 7f382659c640 1 --2- 172.21.15.16:0/3681025652 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3810004d10 0x7f3810005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:26.437 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.427+0000 7f382659c640 1 -- 172.21.15.16:0/3681025652 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3810003530 con 0x7f38100039b0 2023-05-22T15:39:26.439 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.431+0000 7f382659c640 1 --2- 172.21.15.16:0/3681025652 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38100039b0 0x7f3810003d80 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f3818009ae0 tx=0x7f381801ca50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa737b407d77752e server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:39:26.439 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.431+0000 7f382559a640 1 -- 172.21.15.16:0/3681025652 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38180a5640 con 0x7f38100039b0 2023-05-22T15:39:26.440 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.431+0000 7f382559a640 1 -- 172.21.15.16:0/3681025652 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38180a57a0 con 0x7f38100039b0 2023-05-22T15:39:26.440 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.431+0000 7f382559a640 1 -- 172.21.15.16:0/3681025652 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38180a5b50 con 0x7f38100039b0 2023-05-22T15:39:26.441 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.431+0000 7f382cd0d640 1 -- 172.21.15.16:0/3681025652 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38100039b0 msgr2=0x7f3810003d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:26.441 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.431+0000 7f382cd0d640 1 --2- 172.21.15.16:0/3681025652 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38100039b0 0x7f3810003d80 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f3818009ae0 tx=0x7f381801ca50 comp rx=0 tx=0).stop 2023-05-22T15:39:26.441 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.431+0000 7f3826d9d640 1 -- 172.21.15.16:0/3681025652 reap_dead start 2023-05-22T15:39:26.441 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.431+0000 7f382cd0d640 1 -- 172.21.15.16:0/3681025652 shutdown_connections 2023-05-22T15:39:26.441 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.431+0000 7f382cd0d640 1 -- 172.21.15.16:0/3681025652 >> 172.21.15.16:0/3681025652 conn(0x7f38280b0cc0 msgr2=0x7f38280b10c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:39:26.441 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.431+0000 7f382cd0d640 1 -- 172.21.15.16:0/3681025652 shutdown_connections 2023-05-22T15:39:26.441 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.431+0000 7f382cd0d640 1 -- 172.21.15.16:0/3681025652 wait complete. 2023-05-22T15:39:26.442 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.431+0000 7f382cd0d640 1 Processor -- start 2023-05-22T15:39:26.442 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.431+0000 7f382cd0d640 1 -- start start 2023-05-22T15:39:26.442 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.431+0000 7f382cd0d640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f38100039b0 0x7f38280c8ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:26.442 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.431+0000 7f382cd0d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3828150820 0x7f382814ff50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:26.442 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.431+0000 7f382cd0d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38280ccce0 0x7f38280cd150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:26.442 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.431+0000 7f382cd0d640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38280cd690 con 0x7f38280ccce0 2023-05-22T15:39:26.442 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.431+0000 7f382cd0d640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38281eceb0 con 0x7f3828150820 2023-05-22T15:39:26.442 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.431+0000 7f382cd0d640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3828150d20 con 0x7f38100039b0 2023-05-22T15:39:26.443 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.431+0000 7f3826d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38280ccce0 0x7f38280cd150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:26.443 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.431+0000 7f3826d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38280ccce0 0x7f38280cd150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52472/0 (socket says 172.21.15.16:52472) 2023-05-22T15:39:26.443 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.431+0000 7f3826d9d640 1 -- 172.21.15.16:0/1070316604 learned_addr learned my addr 172.21.15.16:0/1070316604 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:39:26.443 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.431+0000 7f3825d9b640 1 --2- 172.21.15.16:0/1070316604 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3828150820 0x7f382814ff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:26.443 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.431+0000 7f382659c640 1 --2- 172.21.15.16:0/1070316604 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f38100039b0 0x7f38280c8ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:26.443 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.431+0000 7f3826d9d640 1 -- 172.21.15.16:0/1070316604 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f38100039b0 msgr2=0x7f38280c8ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:26.443 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.431+0000 7f3826d9d640 1 --2- 172.21.15.16:0/1070316604 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f38100039b0 0x7f38280c8ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:26.444 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.431+0000 7f3826d9d640 1 -- 172.21.15.16:0/1070316604 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3828150820 msgr2=0x7f382814ff50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:26.444 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.431+0000 7f3826d9d640 1 --2- 172.21.15.16:0/1070316604 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3828150820 0x7f382814ff50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:26.444 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.431+0000 7f3826d9d640 1 -- 172.21.15.16:0/1070316604 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3828071730 con 0x7f38280ccce0 2023-05-22T15:39:26.444 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.431+0000 7f382659c640 1 --2- 172.21.15.16:0/1070316604 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f38100039b0 0x7f38280c8ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:39:26.444 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.431+0000 7f3826d9d640 1 --2- 172.21.15.16:0/1070316604 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38280ccce0 0x7f38280cd150 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f382815c4a0 tx=0x7f3820010b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:39:26.444 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.431+0000 7f3816ffd640 1 -- 172.21.15.16:0/1070316604 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3820003d90 con 0x7f38280ccce0 2023-05-22T15:39:26.444 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.431+0000 7f3816ffd640 1 -- 172.21.15.16:0/1070316604 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f382000d050 con 0x7f38280ccce0 2023-05-22T15:39:26.445 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.431+0000 7f382cd0d640 1 -- 172.21.15.16:0/1070316604 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3828071450 con 0x7f38280ccce0 2023-05-22T15:39:26.445 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.431+0000 7f3816ffd640 1 -- 172.21.15.16:0/1070316604 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3820005310 con 0x7f38280ccce0 2023-05-22T15:39:26.445 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.431+0000 7f382cd0d640 1 -- 172.21.15.16:0/1070316604 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3828158d20 con 0x7f38280ccce0 2023-05-22T15:39:26.445 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.435+0000 7f382cd0d640 1 -- 172.21.15.16:0/1070316604 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f382804dbe0 con 0x7f38280ccce0 2023-05-22T15:39:26.445 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.435+0000 7f3816ffd640 1 -- 172.21.15.16:0/1070316604 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 12) v1 ==== 84418+0+0 (secure 0 0 0) 0x7f3820005470 con 0x7f38280ccce0 2023-05-22T15:39:26.446 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.435+0000 7f3816ffd640 1 --2- 172.21.15.16:0/1070316604 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f3808065580 0x7f3808067a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:26.446 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.435+0000 7f3816ffd640 1 -- 172.21.15.16:0/1070316604 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 6698+0+0 (secure 0 0 0) 0x7f3820085d00 con 0x7f38280ccce0 2023-05-22T15:39:26.446 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.435+0000 7f382659c640 1 --2- 172.21.15.16:0/1070316604 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f3808065580 0x7f3808067a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:26.446 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.435+0000 7f382659c640 1 --2- 172.21.15.16:0/1070316604 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f3808065580 0x7f3808067a30 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f38180af000 tx=0x7f38180046d0 comp rx=0 tx=0).ready entity=mgr.5823 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:39:26.447 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.439+0000 7f3816ffd640 1 -- 172.21.15.16:0/1070316604 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f38200532d0 con 0x7f38280ccce0 2023-05-22T15:39:26.615 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.611+0000 7f382cd0d640 1 -- 172.21.15.16:0/1070316604 --> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/snap_test_dir1/dir3", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f38280b18a0 con 0x7f3808065580 2023-05-22T15:39:26.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.627+0000 7f3816ffd640 1 -- 172.21.15.16:0/1070316604 <== mgr.5823 v2:172.21.15.16:6832/36921 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+345 (secure 0 0 0) 0x7f38280b18a0 con 0x7f3808065580 2023-05-22T15:39:26.630 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:39:26.631 INFO:teuthology.orchestra.run.smithi016.stdout:[{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/dir3", "rel_path": "/snap_test_dir1/dir3", "schedule": "1M", "retention": {}, "start": "2023-05-22T00:00:00", "created": "2023-05-22T15:28:33", "first": "2023-05-22T15:29:00", "last": "2023-05-22T15:39:01", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2023-05-22T15:39:26.632 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.627+0000 7f382cd0d640 1 -- 172.21.15.16:0/1070316604 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f3808065580 msgr2=0x7f3808067a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:26.633 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.627+0000 7f382cd0d640 1 --2- 172.21.15.16:0/1070316604 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f3808065580 0x7f3808067a30 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f38180af000 tx=0x7f38180046d0 comp rx=0 tx=0).stop 2023-05-22T15:39:26.633 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.627+0000 7f382cd0d640 1 -- 172.21.15.16:0/1070316604 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38280ccce0 msgr2=0x7f38280cd150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:26.633 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.627+0000 7f382cd0d640 1 --2- 172.21.15.16:0/1070316604 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38280ccce0 0x7f38280cd150 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f382815c4a0 tx=0x7f3820010b50 comp rx=0 tx=0).stop 2023-05-22T15:39:26.633 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.627+0000 7f382cd0d640 1 -- 172.21.15.16:0/1070316604 shutdown_connections 2023-05-22T15:39:26.633 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.627+0000 7f382cd0d640 1 --2- 172.21.15.16:0/1070316604 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38280ccce0 0x7f38280cd150 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:26.633 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.627+0000 7f382cd0d640 1 --2- 172.21.15.16:0/1070316604 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3828150820 0x7f382814ff50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:26.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.627+0000 7f382cd0d640 1 --2- 172.21.15.16:0/1070316604 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f3808065580 0x7f3808067a30 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:26.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.627+0000 7f382cd0d640 1 --2- 172.21.15.16:0/1070316604 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f38100039b0 0x7f38280c8ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:26.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.627+0000 7f382cd0d640 1 -- 172.21.15.16:0/1070316604 >> 172.21.15.16:0/1070316604 conn(0x7f38280b0cc0 msgr2=0x7f3828153be0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:39:26.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.627+0000 7f382cd0d640 1 -- 172.21.15.16:0/1070316604 shutdown_connections 2023-05-22T15:39:26.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.627+0000 7f382cd0d640 1 -- 172.21.15.16:0/1070316604 wait complete. 2023-05-22T15:39:26.651 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/dir3", "rel_path": "/snap_test_dir1/dir3", "schedule": "1M", "retention": {}, "start": "2023-05-22T00:00:00", "created": "2023-05-22T15:28:33", "first": "2023-05-22T15:29:00", "last": "2023-05-22T15:39:01", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2023-05-22T15:39:26.651 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'path': '/snap_test_dir1/dir3', 'rel_path': '/snap_test_dir1/dir3', 'schedule': '1M', 'retention': {}, 'start': '2023-05-22T00:00:00', 'created': '2023-05-22T15:28:33', 'first': '2023-05-22T15:29:00', 'last': '2023-05-22T15:39:01', 'last_pruned': None, 'created_count': 11, 'pruned_count': 0, 'active': False} 2023-05-22T15:39:26.652 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:39:26.653 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir4/.snap) 2023-05-22T15:39:26.664 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_29_00_UTC 2023-05-22T15:39:26.665 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_30_01_UTC 2023-05-22T15:39:26.665 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_31_00_UTC 2023-05-22T15:39:26.665 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_32_00_UTC 2023-05-22T15:39:26.665 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_33_00_UTC 2023-05-22T15:39:26.665 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_34_00_UTC 2023-05-22T15:39:26.665 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_35_00_UTC 2023-05-22T15:39:26.666 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_36_01_UTC 2023-05-22T15:39:26.666 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_37_00_UTC 2023-05-22T15:39:26.666 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_38_00_UTC 2023-05-22T15:39:26.666 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_39_00_UTC 2023-05-22T15:39:26.667 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2023-05-22-15_29_00_UTC', 'scheduled-2023-05-22-15_30_01_UTC', 'scheduled-2023-05-22-15_31_00_UTC', 'scheduled-2023-05-22-15_32_00_UTC', 'scheduled-2023-05-22-15_33_00_UTC', 'scheduled-2023-05-22-15_34_00_UTC', 'scheduled-2023-05-22-15_35_00_UTC', 'scheduled-2023-05-22-15_36_01_UTC', 'scheduled-2023-05-22-15_37_00_UTC', 'scheduled-2023-05-22-15_38_00_UTC', 'scheduled-2023-05-22-15_39_00_UTC'] 2023-05-22T15:39:26.668 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --fs cephfs --format json /snap_test_dir1/dir4 2023-05-22T15:39:26.841 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.835+0000 7fc6a29e4640 1 Processor -- start 2023-05-22T15:39:26.842 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.835+0000 7fc6a29e4640 1 -- start start 2023-05-22T15:39:26.842 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.839+0000 7fc6a29e4640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fc69c15d610 0x7fc69c0cbe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:26.843 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.839+0000 7fc6a29e4640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc69c0cf570 0x7fc69c0cf9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:26.843 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.839+0000 7fc6a29e4640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc69c0cff10 con 0x7fc69c159a30 2023-05-22T15:39:26.843 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.839+0000 7fc6a29e4640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc69c0cde70 con 0x7fc69c15d610 2023-05-22T15:39:26.843 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.839+0000 7fc6a29e4640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc69c15db20 con 0x7fc69c0cf570 2023-05-22T15:39:26.844 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.839+0000 7fc69b7fe640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fc69c15d610 0x7fc69c0cbe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:26.844 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.839+0000 7fc6a0f81640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc69c0cf570 0x7fc69c0cf9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:26.844 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.839+0000 7fc69b7fe640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fc69c15d610 0x7fc69c0cbe10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.16:35018/0 (socket says 172.21.15.16:35018) 2023-05-22T15:39:26.844 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.839+0000 7fc69b7fe640 1 -- 172.21.15.16:0/17415989 learned_addr learned my addr 172.21.15.16:0/17415989 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:39:26.844 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.839+0000 7fc69b7fe640 1 -- 172.21.15.16:0/17415989 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc69c0cf570 msgr2=0x7fc69c0cf9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:39:26.845 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.839+0000 7fc69b7fe640 1 --2- 172.21.15.16:0/17415989 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc69c0cf570 0x7fc69c0cf9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:26.845 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.839+0000 7fc69b7fe640 1 -- 172.21.15.16:0/17415989 >> v1:172.21.15.62:6789/0 conn(0x7fc69c159a30 legacy=0x7fc69c06e380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:26.845 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.839+0000 7fc69b7fe640 1 -- 172.21.15.16:0/17415989 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc69c15dde0 con 0x7fc69c15d610 2023-05-22T15:39:26.845 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.839+0000 7fc6a0f81640 1 --2- 172.21.15.16:0/17415989 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc69c0cf570 0x7fc69c0cf9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-05-22T15:39:26.846 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.843+0000 7fc69b7fe640 1 --2- 172.21.15.16:0/17415989 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc69c15d610 0x7fc69c0cbe10 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fc6900098a0 tx=0x7fc69001c4f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e571c0d088902b42 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:39:26.846 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.843+0000 7fc69affd640 1 -- 172.21.15.16:0/17415989 <== mon.1 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6900aa650 con 0x7fc69c15d610 2023-05-22T15:39:26.846 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.843+0000 7fc69affd640 1 -- 172.21.15.16:0/17415989 <== mon.1 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6900aa7b0 con 0x7fc69c15d610 2023-05-22T15:39:26.847 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.843+0000 7fc69affd640 1 -- 172.21.15.16:0/17415989 <== mon.1 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6900aab40 con 0x7fc69c15d610 2023-05-22T15:39:26.847 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.843+0000 7fc6a29e4640 1 -- 172.21.15.16:0/17415989 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc69c15d610 msgr2=0x7fc69c0cbe10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:26.847 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.843+0000 7fc6a29e4640 1 --2- 172.21.15.16:0/17415989 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc69c15d610 0x7fc69c0cbe10 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fc6900098a0 tx=0x7fc69001c4f0 comp rx=0 tx=0).stop 2023-05-22T15:39:26.847 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.843+0000 7fc6a29e4640 1 -- 172.21.15.16:0/17415989 shutdown_connections 2023-05-22T15:39:26.848 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.843+0000 7fc6a29e4640 1 --2- 172.21.15.16:0/17415989 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc69c0cf570 0x7fc69c0cf9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:26.848 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.843+0000 7fc6a29e4640 1 --2- 172.21.15.16:0/17415989 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc69c15d610 0x7fc69c0cbe10 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:26.848 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.843+0000 7fc6a29e4640 1 -- 172.21.15.16:0/17415989 >> 172.21.15.16:0/17415989 conn(0x7fc69c0700b0 msgr2=0x7fc69c06d9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:39:26.848 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.843+0000 7fc6a29e4640 1 -- 172.21.15.16:0/17415989 shutdown_connections 2023-05-22T15:39:26.848 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.843+0000 7fc6a29e4640 1 -- 172.21.15.16:0/17415989 wait complete. 2023-05-22T15:39:26.849 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.843+0000 7fc6a29e4640 1 Processor -- start 2023-05-22T15:39:26.850 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.843+0000 7fc6a29e4640 1 -- start start 2023-05-22T15:39:26.850 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.843+0000 7fc6a29e4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc69c0cf570 0x7fc69c14c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:26.850 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.843+0000 7fc6a29e4640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc69c159a30 0x7fc69c1492a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:26.850 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.843+0000 7fc6a29e4640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc69c15d610 0x7fc69c1497e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:26.850 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.843+0000 7fc6a29e4640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc69c149d20 con 0x7fc69c0cf570 2023-05-22T15:39:26.850 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.843+0000 7fc6a29e4640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc69c14ae30 con 0x7fc69c159a30 2023-05-22T15:39:26.850 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.843+0000 7fc6a29e4640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc69c14b110 con 0x7fc69c15d610 2023-05-22T15:39:26.851 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.843+0000 7fc69bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc69c0cf570 0x7fc69c14c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:26.851 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.843+0000 7fc69bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc69c0cf570 0x7fc69c14c690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52474/0 (socket says 172.21.15.16:52474) 2023-05-22T15:39:26.851 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.843+0000 7fc69bfff640 1 -- 172.21.15.16:0/861604040 learned_addr learned my addr 172.21.15.16:0/861604040 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:39:26.851 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.843+0000 7fc69b7fe640 1 --2- 172.21.15.16:0/861604040 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc69c159a30 0x7fc69c1492a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:26.851 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.843+0000 7fc6a0f81640 1 --2- 172.21.15.16:0/861604040 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc69c15d610 0x7fc69c1497e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:26.851 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.843+0000 7fc69bfff640 1 -- 172.21.15.16:0/861604040 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc69c15d610 msgr2=0x7fc69c1497e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:26.851 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.843+0000 7fc69bfff640 1 --2- 172.21.15.16:0/861604040 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc69c15d610 0x7fc69c1497e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:26.851 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.843+0000 7fc69bfff640 1 -- 172.21.15.16:0/861604040 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc69c159a30 msgr2=0x7fc69c1492a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:26.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.843+0000 7fc69bfff640 1 --2- 172.21.15.16:0/861604040 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc69c159a30 0x7fc69c1492a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:26.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.843+0000 7fc69bfff640 1 -- 172.21.15.16:0/861604040 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc694009a70 con 0x7fc69c0cf570 2023-05-22T15:39:26.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.843+0000 7fc69bfff640 1 --2- 172.21.15.16:0/861604040 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc69c0cf570 0x7fc69c14c690 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7fc68c00c420 tx=0x7fc68c00c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:39:26.853 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.843+0000 7fc698ff9640 1 -- 172.21.15.16:0/861604040 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc68c0052d0 con 0x7fc69c0cf570 2023-05-22T15:39:26.853 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.847+0000 7fc6a29e4640 1 -- 172.21.15.16:0/861604040 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6900094d0 con 0x7fc69c0cf570 2023-05-22T15:39:26.853 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.847+0000 7fc698ff9640 1 -- 172.21.15.16:0/861604040 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc68c005430 con 0x7fc69c0cf570 2023-05-22T15:39:26.853 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.847+0000 7fc698ff9640 1 -- 172.21.15.16:0/861604040 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc68c007260 con 0x7fc69c0cf570 2023-05-22T15:39:26.853 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.847+0000 7fc6a29e4640 1 -- 172.21.15.16:0/861604040 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc69c0cc350 con 0x7fc69c0cf570 2023-05-22T15:39:26.854 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.847+0000 7fc6a29e4640 1 -- 172.21.15.16:0/861604040 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc69c159e00 con 0x7fc69c0cf570 2023-05-22T15:39:26.854 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.847+0000 7fc698ff9640 1 -- 172.21.15.16:0/861604040 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 12) v1 ==== 84418+0+0 (secure 0 0 0) 0x7fc68c0073c0 con 0x7fc69c0cf570 2023-05-22T15:39:26.854 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.847+0000 7fc698ff9640 1 --2- 172.21.15.16:0/861604040 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fc688065580 0x7fc688067a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:26.854 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.847+0000 7fc698ff9640 1 -- 172.21.15.16:0/861604040 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 6698+0+0 (secure 0 0 0) 0x7fc68c082230 con 0x7fc69c0cf570 2023-05-22T15:39:26.854 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.847+0000 7fc69b7fe640 1 --2- 172.21.15.16:0/861604040 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fc688065580 0x7fc688067a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:26.854 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.847+0000 7fc69b7fe640 1 --2- 172.21.15.16:0/861604040 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fc688065580 0x7fc688067a30 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fc6900ae000 tx=0x7fc6900a7000 comp rx=0 tx=0).ready entity=mgr.5823 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:39:26.856 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:26.851+0000 7fc698ff9640 1 -- 172.21.15.16:0/861604040 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fc68c049620 con 0x7fc69c0cf570 2023-05-22T15:39:27.029 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:27.023+0000 7fc6a29e4640 1 -- 172.21.15.16:0/861604040 --> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/snap_test_dir1/dir4", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fc69c0cc4e0 con 0x7fc688065580 2023-05-22T15:39:27.045 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:27.039+0000 7fc698ff9640 1 -- 172.21.15.16:0/861604040 <== mgr.5823 v2:172.21.15.16:6832/36921 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+345 (secure 0 0 0) 0x7fc69c0cc4e0 con 0x7fc688065580 2023-05-22T15:39:27.045 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:39:27.046 INFO:teuthology.orchestra.run.smithi016.stdout:[{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/dir4", "rel_path": "/snap_test_dir1/dir4", "schedule": "1M", "retention": {}, "start": "2023-05-22T00:00:00", "created": "2023-05-22T15:28:33", "first": "2023-05-22T15:29:00", "last": "2023-05-22T15:39:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2023-05-22T15:39:27.047 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:27.043+0000 7fc6a29e4640 1 -- 172.21.15.16:0/861604040 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fc688065580 msgr2=0x7fc688067a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:27.048 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:27.043+0000 7fc6a29e4640 1 --2- 172.21.15.16:0/861604040 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fc688065580 0x7fc688067a30 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fc6900ae000 tx=0x7fc6900a7000 comp rx=0 tx=0).stop 2023-05-22T15:39:27.048 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:27.043+0000 7fc6a29e4640 1 -- 172.21.15.16:0/861604040 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc69c0cf570 msgr2=0x7fc69c14c690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:27.048 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:27.043+0000 7fc6a29e4640 1 --2- 172.21.15.16:0/861604040 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc69c0cf570 0x7fc69c14c690 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7fc68c00c420 tx=0x7fc68c00c8e0 comp rx=0 tx=0).stop 2023-05-22T15:39:27.048 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:27.043+0000 7fc6a29e4640 1 -- 172.21.15.16:0/861604040 shutdown_connections 2023-05-22T15:39:27.049 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:27.043+0000 7fc6a29e4640 1 --2- 172.21.15.16:0/861604040 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fc688065580 0x7fc688067a30 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:27.049 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:27.043+0000 7fc6a29e4640 1 --2- 172.21.15.16:0/861604040 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc69c15d610 0x7fc69c1497e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:27.049 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:27.043+0000 7fc6a29e4640 1 --2- 172.21.15.16:0/861604040 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc69c159a30 0x7fc69c1492a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:27.049 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:27.043+0000 7fc6a29e4640 1 --2- 172.21.15.16:0/861604040 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc69c0cf570 0x7fc69c14c690 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:27.049 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:27.043+0000 7fc6a29e4640 1 -- 172.21.15.16:0/861604040 >> 172.21.15.16:0/861604040 conn(0x7fc69c0700b0 msgr2=0x7fc69c152b40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:39:27.049 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:27.043+0000 7fc6a29e4640 1 -- 172.21.15.16:0/861604040 shutdown_connections 2023-05-22T15:39:27.050 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:27.043+0000 7fc6a29e4640 1 -- 172.21.15.16:0/861604040 wait complete. 2023-05-22T15:39:27.065 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/dir4", "rel_path": "/snap_test_dir1/dir4", "schedule": "1M", "retention": {}, "start": "2023-05-22T00:00:00", "created": "2023-05-22T15:28:33", "first": "2023-05-22T15:29:00", "last": "2023-05-22T15:39:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2023-05-22T15:39:27.066 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'path': '/snap_test_dir1/dir4', 'rel_path': '/snap_test_dir1/dir4', 'schedule': '1M', 'retention': {}, 'start': '2023-05-22T00:00:00', 'created': '2023-05-22T15:28:33', 'first': '2023-05-22T15:29:00', 'last': '2023-05-22T15:39:00', 'last_pruned': None, 'created_count': 11, 'pruned_count': 0, 'active': False} 2023-05-22T15:39:27.067 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:39:27.067 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir5/.snap) 2023-05-22T15:39:27.080 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_29_00_UTC 2023-05-22T15:39:27.080 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_30_00_UTC 2023-05-22T15:39:27.080 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_31_01_UTC 2023-05-22T15:39:27.081 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_32_00_UTC 2023-05-22T15:39:27.081 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_33_00_UTC 2023-05-22T15:39:27.081 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_34_00_UTC 2023-05-22T15:39:27.081 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_35_01_UTC 2023-05-22T15:39:27.081 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_36_00_UTC 2023-05-22T15:39:27.081 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_37_00_UTC 2023-05-22T15:39:27.082 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_38_00_UTC 2023-05-22T15:39:27.082 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_39_01_UTC 2023-05-22T15:39:27.082 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2023-05-22-15_29_00_UTC', 'scheduled-2023-05-22-15_30_00_UTC', 'scheduled-2023-05-22-15_31_01_UTC', 'scheduled-2023-05-22-15_32_00_UTC', 'scheduled-2023-05-22-15_33_00_UTC', 'scheduled-2023-05-22-15_34_00_UTC', 'scheduled-2023-05-22-15_35_01_UTC', 'scheduled-2023-05-22-15_36_00_UTC', 'scheduled-2023-05-22-15_37_00_UTC', 'scheduled-2023-05-22-15_38_00_UTC', 'scheduled-2023-05-22-15_39_01_UTC'] 2023-05-22T15:39:27.083 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --fs cephfs --format json /snap_test_dir1/dir5 2023-05-22T15:39:27.264 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:27.259+0000 7fb53feae640 1 Processor -- start 2023-05-22T15:39:27.264 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:27.259+0000 7fb53feae640 1 -- start start 2023-05-22T15:39:27.265 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:27.259+0000 7fb53feae640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb53815be10 0x7fb5380ceeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:27.265 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:27.259+0000 7fb53feae640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fb5380cd440 0x7fb5380b5bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:27.265 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:27.259+0000 7fb53feae640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5380cf840 con 0x7fb5380cf3f0 2023-05-22T15:39:27.266 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:27.259+0000 7fb53feae640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5380cda40 con 0x7fb5380cd440 2023-05-22T15:39:27.266 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:27.259+0000 7fb53feae640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5380cdd20 con 0x7fb53815be10 2023-05-22T15:39:27.266 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:27.259+0000 7fb53d449640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fb5380cf3f0 0x7fb5380ccd30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:53074/0 (socket says 172.21.15.16:53074) 2023-05-22T15:39:27.266 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:27.259+0000 7fb53d449640 1 -- 172.21.15.16:0/3639400920 learned_addr learned my addr 172.21.15.16:0/3639400920 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:39:27.267 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:27.259+0000 7fb53dc4a640 1 --2- 172.21.15.16:0/3639400920 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb53815be10 0x7fb5380ceeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:27.268 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:27.259+0000 7fb53e44b640 1 --2- 172.21.15.16:0/3639400920 >> v2:172.21.15.62:3300/0 conn(0x7fb5380cd440 0x7fb5380b5bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:27.268 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:27.259+0000 7fb53cc48640 1 -- 172.21.15.16:0/3639400920 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1054311449 0 0) 0x7fb534001120 con 0x7fb5380cf3f0 2023-05-22T15:39:27.268 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:27.259+0000 7fb53cc48640 1 -- 172.21.15.16:0/3639400920 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb524003530 con 0x7fb5380cf3f0 2023-05-22T15:39:27.268 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:27.259+0000 7fb53cc48640 1 -- 172.21.15.16:0/3639400920 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 203896657 0 0) 0x7fb524003530 con 0x7fb5380cf3f0 2023-05-22T15:39:27.268 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:27.259+0000 7fb53cc48640 1 -- 172.21.15.16:0/3639400920 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb53815be10 msgr2=0x7fb5380ceeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:39:27.268 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:27.259+0000 7fb53cc48640 1 --2- 172.21.15.16:0/3639400920 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb53815be10 0x7fb5380ceeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:27.269 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:27.259+0000 7fb53cc48640 1 -- 172.21.15.16:0/3639400920 >> v2:172.21.15.62:3300/0 conn(0x7fb5380cd440 msgr2=0x7fb5380b5bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:39:27.269 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:27.259+0000 7fb53cc48640 1 --2- 172.21.15.16:0/3639400920 >> v2:172.21.15.62:3300/0 conn(0x7fb5380cd440 0x7fb5380b5bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:27.269 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:27.259+0000 7fb53cc48640 1 -- 172.21.15.16:0/3639400920 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5380ce000 con 0x7fb5380cf3f0 2023-05-22T15:39:27.269 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:27.259+0000 7fb53cc48640 1 -- 172.21.15.16:0/3639400920 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7fb534001810 con 0x7fb5380cf3f0 2023-05-22T15:39:27.269 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:27.259+0000 7fb53cc48640 1 -- 172.21.15.16:0/3639400920 >> v1:172.21.15.16:6789/0 conn(0x7fb5380cf3f0 legacy=0x7fb5380ccd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:27.269 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:27.263+0000 7fb53cc48640 1 --2- 172.21.15.16:0/3639400920 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb53815be10 0x7fb524003980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:27.270 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:27.263+0000 7fb53cc48640 1 --2- 172.21.15.16:0/3639400920 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb524003f00 0x7fb5240047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:27.270 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:27.263+0000 7fb53cc48640 1 --2- 172.21.15.16:0/3639400920 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb524004d10 0x7fb524005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:27.270 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:27.263+0000 7fb53cc48640 1 -- 172.21.15.16:0/3639400920 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5380cda40 con 0x7fb524003f00 2023-05-22T15:39:27.270 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:27.263+0000 7fb53cc48640 1 -- 172.21.15.16:0/3639400920 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5380cdd20 con 0x7fb524004d10 2023-05-22T15:39:27.270 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:27.263+0000 7fb53cc48640 1 -- 172.21.15.16:0/3639400920 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb52400b990 con 0x7fb53815be10 2023-05-22T15:39:27.270 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:27.263+0000 7fb53d449640 1 --2- 172.21.15.16:0/3639400920 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb524003f00 0x7fb5240047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:27.271 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:27.263+0000 7fb53dc4a640 1 --2- 172.21.15.16:0/3639400920 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb53815be10 0x7fb524003980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:27.271 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:27.263+0000 7fb53d449640 1 -- 172.21.15.16:0/3639400920 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb53815be10 msgr2=0x7fb524003980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:39:27.271 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:27.263+0000 7fb53d449640 1 --2- 172.21.15.16:0/3639400920 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb53815be10 0x7fb524003980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:27.271 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:27.263+0000 7fb53d449640 1 -- 172.21.15.16:0/3639400920 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb524004d10 msgr2=0x7fb524005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:39:27.271 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:27.263+0000 7fb53d449640 1 --2- 172.21.15.16:0/3639400920 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb524004d10 0x7fb524005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:27.271 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:27.263+0000 7fb53d449640 1 -- 172.21.15.16:0/3639400920 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb524003530 con 0x7fb524003f00 2023-05-22T15:39:27.274 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:27.263+0000 7fb53d449640 1 --2- 172.21.15.16:0/3639400920 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb524003f00 0x7fb5240047d0 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7fb53400c210 tx=0x7fb53401f3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1ea0478edf29dc9 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:39:27.274 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:27.263+0000 7fb53cc48640 1 -- 172.21.15.16:0/3639400920 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb534005ae0 con 0x7fb524003f00 2023-05-22T15:39:27.274 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:27.263+0000 7fb53cc48640 1 -- 172.21.15.16:0/3639400920 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5340ab070 con 0x7fb524003f00 2023-05-22T15:39:27.274 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:27.263+0000 7fb53cc48640 1 -- 172.21.15.16:0/3639400920 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb534007070 con 0x7fb524003f00 2023-05-22T15:39:27.274 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:27.263+0000 7fb53feae640 1 -- 172.21.15.16:0/3639400920 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb524003f00 msgr2=0x7fb5240047d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:27.274 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:27.263+0000 7fb53feae640 1 --2- 172.21.15.16:0/3639400920 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb524003f00 0x7fb5240047d0 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7fb53400c210 tx=0x7fb53401f3b0 comp rx=0 tx=0).stop 2023-05-22T15:39:27.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:27.263+0000 7fb53e44b640 1 -- 172.21.15.16:0/3639400920 reap_dead start 2023-05-22T15:39:27.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:27.263+0000 7fb53feae640 1 -- 172.21.15.16:0/3639400920 shutdown_connections 2023-05-22T15:39:27.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:27.263+0000 7fb53feae640 1 -- 172.21.15.16:0/3639400920 >> 172.21.15.16:0/3639400920 conn(0x7fb538070e50 msgr2=0x7fb538071250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:39:27.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:27.263+0000 7fb53feae640 1 -- 172.21.15.16:0/3639400920 shutdown_connections 2023-05-22T15:39:27.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:27.263+0000 7fb53feae640 1 -- 172.21.15.16:0/3639400920 wait complete. 2023-05-22T15:39:27.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:27.267+0000 7fb53feae640 1 Processor -- start 2023-05-22T15:39:27.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:27.267+0000 7fb53feae640 1 -- start start 2023-05-22T15:39:27.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:27.267+0000 7fb53feae640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb524003f00 0x7fb53814df50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:27.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:27.267+0000 7fb53feae640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb538148310 0x7fb538148780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:27.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:27.267+0000 7fb53feae640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb53814c180 0x7fb53814c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:27.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:27.267+0000 7fb53feae640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb538148cc0 con 0x7fb53814c180 2023-05-22T15:39:27.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:27.267+0000 7fb53feae640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb53814ceb0 con 0x7fb524003f00 2023-05-22T15:39:27.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:27.267+0000 7fb53feae640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb53814d190 con 0x7fb538148310 2023-05-22T15:39:27.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:27.267+0000 7fb53e44b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb53814c180 0x7fb53814c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:27.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:27.267+0000 7fb53e44b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb53814c180 0x7fb53814c5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52500/0 (socket says 172.21.15.16:52500) 2023-05-22T15:39:27.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:27.267+0000 7fb53e44b640 1 -- 172.21.15.16:0/3144481053 learned_addr learned my addr 172.21.15.16:0/3144481053 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:39:27.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:27.267+0000 7fb53d449640 1 --2- 172.21.15.16:0/3144481053 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb538148310 0x7fb538148780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:27.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:27.267+0000 7fb53dc4a640 1 --2- 172.21.15.16:0/3144481053 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb524003f00 0x7fb53814df50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:27.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:27.267+0000 7fb53e44b640 1 -- 172.21.15.16:0/3144481053 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb538148310 msgr2=0x7fb538148780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:27.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:27.267+0000 7fb53e44b640 1 --2- 172.21.15.16:0/3144481053 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb538148310 0x7fb538148780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:27.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:27.267+0000 7fb53e44b640 1 -- 172.21.15.16:0/3144481053 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb524003f00 msgr2=0x7fb53814df50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:27.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:27.267+0000 7fb53e44b640 1 --2- 172.21.15.16:0/3144481053 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb524003f00 0x7fb53814df50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:27.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:27.267+0000 7fb53e44b640 1 -- 172.21.15.16:0/3144481053 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5380b56a0 con 0x7fb53814c180 2023-05-22T15:39:27.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:27.267+0000 7fb53e44b640 1 --2- 172.21.15.16:0/3144481053 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb53814c180 0x7fb53814c5f0 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7fb5380cde60 tx=0x7fb53000ef10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:39:27.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:27.267+0000 7fb52a7fc640 1 -- 172.21.15.16:0/3144481053 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb530010040 con 0x7fb53814c180 2023-05-22T15:39:27.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:27.267+0000 7fb52a7fc640 1 -- 172.21.15.16:0/3144481053 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb530014810 con 0x7fb53814c180 2023-05-22T15:39:27.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:27.267+0000 7fb53feae640 1 -- 172.21.15.16:0/3144481053 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5380b53c0 con 0x7fb53814c180 2023-05-22T15:39:27.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:27.267+0000 7fb52a7fc640 1 -- 172.21.15.16:0/3144481053 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb530009b10 con 0x7fb53814c180 2023-05-22T15:39:27.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:27.267+0000 7fb53feae640 1 -- 172.21.15.16:0/3144481053 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5380ceeb0 con 0x7fb53814c180 2023-05-22T15:39:27.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:27.267+0000 7fb53feae640 1 -- 172.21.15.16:0/3144481053 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb53804dbe0 con 0x7fb53814c180 2023-05-22T15:39:27.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:27.271+0000 7fb52a7fc640 1 -- 172.21.15.16:0/3144481053 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 12) v1 ==== 84418+0+0 (secure 0 0 0) 0x7fb530014bc0 con 0x7fb53814c180 2023-05-22T15:39:27.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:27.271+0000 7fb52a7fc640 1 --2- 172.21.15.16:0/3144481053 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fb51c065580 0x7fb51c067a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:27.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:27.271+0000 7fb52a7fc640 1 -- 172.21.15.16:0/3144481053 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 6698+0+0 (secure 0 0 0) 0x7fb530083400 con 0x7fb53814c180 2023-05-22T15:39:27.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:27.271+0000 7fb53dc4a640 1 --2- 172.21.15.16:0/3144481053 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fb51c065580 0x7fb51c067a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:27.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:27.271+0000 7fb53dc4a640 1 --2- 172.21.15.16:0/3144481053 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fb51c065580 0x7fb51c067a30 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fb52c009950 tx=0x7fb52c008040 comp rx=0 tx=0).ready entity=mgr.5823 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:39:27.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:27.275+0000 7fb52a7fc640 1 -- 172.21.15.16:0/3144481053 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fb530050ae0 con 0x7fb53814c180 2023-05-22T15:39:27.450 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:27.447+0000 7fb53feae640 1 -- 172.21.15.16:0/3144481053 --> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/snap_test_dir1/dir5", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb53815afd0 con 0x7fb51c065580 2023-05-22T15:39:27.465 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:27.459+0000 7fb52a7fc640 1 -- 172.21.15.16:0/3144481053 <== mgr.5823 v2:172.21.15.16:6832/36921 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+345 (secure 0 0 0) 0x7fb53815afd0 con 0x7fb51c065580 2023-05-22T15:39:27.465 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:39:27.466 INFO:teuthology.orchestra.run.smithi016.stdout:[{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/dir5", "rel_path": "/snap_test_dir1/dir5", "schedule": "1M", "retention": {}, "start": "2023-05-22T00:00:00", "created": "2023-05-22T15:28:34", "first": "2023-05-22T15:29:00", "last": "2023-05-22T15:39:01", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2023-05-22T15:39:27.467 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:27.463+0000 7fb53feae640 1 -- 172.21.15.16:0/3144481053 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fb51c065580 msgr2=0x7fb51c067a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:27.468 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:27.463+0000 7fb53feae640 1 --2- 172.21.15.16:0/3144481053 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fb51c065580 0x7fb51c067a30 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fb52c009950 tx=0x7fb52c008040 comp rx=0 tx=0).stop 2023-05-22T15:39:27.468 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:27.463+0000 7fb53feae640 1 -- 172.21.15.16:0/3144481053 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb53814c180 msgr2=0x7fb53814c5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:27.468 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:27.463+0000 7fb53feae640 1 --2- 172.21.15.16:0/3144481053 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb53814c180 0x7fb53814c5f0 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7fb5380cde60 tx=0x7fb53000ef10 comp rx=0 tx=0).stop 2023-05-22T15:39:27.468 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:27.463+0000 7fb53feae640 1 -- 172.21.15.16:0/3144481053 shutdown_connections 2023-05-22T15:39:27.468 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:27.463+0000 7fb53feae640 1 --2- 172.21.15.16:0/3144481053 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb53814c180 0x7fb53814c5f0 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:27.468 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:27.463+0000 7fb53feae640 1 --2- 172.21.15.16:0/3144481053 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fb51c065580 0x7fb51c067a30 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:27.468 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:27.463+0000 7fb53feae640 1 --2- 172.21.15.16:0/3144481053 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb538148310 0x7fb538148780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:27.469 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:27.463+0000 7fb53feae640 1 --2- 172.21.15.16:0/3144481053 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb524003f00 0x7fb53814df50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:27.469 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:27.463+0000 7fb53feae640 1 -- 172.21.15.16:0/3144481053 >> 172.21.15.16:0/3144481053 conn(0x7fb538070e50 msgr2=0x7fb5380d0010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:39:27.469 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:27.463+0000 7fb53feae640 1 -- 172.21.15.16:0/3144481053 shutdown_connections 2023-05-22T15:39:27.469 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:27.463+0000 7fb53feae640 1 -- 172.21.15.16:0/3144481053 wait complete. 2023-05-22T15:39:27.485 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/dir5", "rel_path": "/snap_test_dir1/dir5", "schedule": "1M", "retention": {}, "start": "2023-05-22T00:00:00", "created": "2023-05-22T15:28:34", "first": "2023-05-22T15:29:00", "last": "2023-05-22T15:39:01", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2023-05-22T15:39:27.485 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'path': '/snap_test_dir1/dir5', 'rel_path': '/snap_test_dir1/dir5', 'schedule': '1M', 'retention': {}, 'start': '2023-05-22T00:00:00', 'created': '2023-05-22T15:28:34', 'first': '2023-05-22T15:29:00', 'last': '2023-05-22T15:39:01', 'last_pruned': None, 'created_count': 11, 'pruned_count': 0, 'active': False} 2023-05-22T15:39:27.485 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:39:27.486 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir6/.snap) 2023-05-22T15:39:27.498 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_29_00_UTC 2023-05-22T15:39:27.499 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_30_01_UTC 2023-05-22T15:39:27.499 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_31_00_UTC 2023-05-22T15:39:27.499 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_32_01_UTC 2023-05-22T15:39:27.499 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_33_00_UTC 2023-05-22T15:39:27.500 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_34_00_UTC 2023-05-22T15:39:27.500 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_35_00_UTC 2023-05-22T15:39:27.500 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_36_00_UTC 2023-05-22T15:39:27.500 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_37_00_UTC 2023-05-22T15:39:27.500 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_38_00_UTC 2023-05-22T15:39:27.500 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_39_00_UTC 2023-05-22T15:39:27.501 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2023-05-22-15_29_00_UTC', 'scheduled-2023-05-22-15_30_01_UTC', 'scheduled-2023-05-22-15_31_00_UTC', 'scheduled-2023-05-22-15_32_01_UTC', 'scheduled-2023-05-22-15_33_00_UTC', 'scheduled-2023-05-22-15_34_00_UTC', 'scheduled-2023-05-22-15_35_00_UTC', 'scheduled-2023-05-22-15_36_00_UTC', 'scheduled-2023-05-22-15_37_00_UTC', 'scheduled-2023-05-22-15_38_00_UTC', 'scheduled-2023-05-22-15_39_00_UTC'] 2023-05-22T15:39:27.502 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --fs cephfs --format json /snap_test_dir1/dir6 2023-05-22T15:39:27.679 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:27.675+0000 7fecc355e640 1 Processor -- start 2023-05-22T15:39:27.679 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:27.675+0000 7fecc355e640 1 -- start start 2023-05-22T15:39:27.680 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:27.675+0000 7fecc355e640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fecbc0cf670 0x7fecbc1601d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:27.681 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:27.675+0000 7fecc355e640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fecbc0cfa40 0x7fecbc0cd260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:27.681 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:27.675+0000 7fecc355e640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fecbc15c0a0 con 0x7fecbc06a830 2023-05-22T15:39:27.681 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:27.675+0000 7fecc355e640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fecbc0cc850 con 0x7fecbc0cfa40 2023-05-22T15:39:27.681 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:27.675+0000 7fecc355e640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fecbc06b3c0 con 0x7fecbc0cf670 2023-05-22T15:39:27.681 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:27.675+0000 7fecc1afb640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fecbc06a830 0x7fecbc06acb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:53088/0 (socket says 172.21.15.16:53088) 2023-05-22T15:39:27.681 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:27.675+0000 7fecc1afb640 1 -- 172.21.15.16:0/3229643102 learned_addr learned my addr 172.21.15.16:0/3229643102 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:39:27.681 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:27.675+0000 7fecc0af9640 1 --2- 172.21.15.16:0/3229643102 >> v2:172.21.15.16:3300/0 conn(0x7fecbc0cfa40 0x7fecbc0cd260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:27.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:27.675+0000 7fecc12fa640 1 --2- 172.21.15.16:0/3229643102 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fecbc0cf670 0x7fecbc1601d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:27.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:27.675+0000 7fecb3fff640 1 -- 172.21.15.16:0/3229643102 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3620215645 0 0) 0x7fecb8001120 con 0x7fecbc06a830 2023-05-22T15:39:27.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:27.675+0000 7fecb3fff640 1 -- 172.21.15.16:0/3229643102 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feca8003530 con 0x7fecbc06a830 2023-05-22T15:39:27.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:27.675+0000 7fecc0af9640 1 -- 172.21.15.16:0/3229643102 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fecbc0cf670 msgr2=0x7fecbc1601d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:39:27.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:27.675+0000 7fecc0af9640 1 --2- 172.21.15.16:0/3229643102 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fecbc0cf670 0x7fecbc1601d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:27.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:27.675+0000 7fecc0af9640 1 -- 172.21.15.16:0/3229643102 >> v1:172.21.15.16:6789/0 conn(0x7fecbc06a830 legacy=0x7fecbc06acb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:27.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:27.675+0000 7fecc0af9640 1 -- 172.21.15.16:0/3229643102 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecbc0ccdd0 con 0x7fecbc0cfa40 2023-05-22T15:39:27.683 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:27.679+0000 7fecc0af9640 1 --2- 172.21.15.16:0/3229643102 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fecbc0cfa40 0x7fecbc0cd260 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7fecac0097f0 tx=0x7fecac01c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57f85f4d869195bc server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:39:27.683 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:27.679+0000 7fecb3fff640 1 -- 172.21.15.16:0/3229643102 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecac0ab600 con 0x7fecbc0cfa40 2023-05-22T15:39:27.684 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:27.679+0000 7fecb3fff640 1 -- 172.21.15.16:0/3229643102 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fecac0a7050 con 0x7fecbc0cfa40 2023-05-22T15:39:27.684 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:27.679+0000 7fecb3fff640 1 -- 172.21.15.16:0/3229643102 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecac0ab960 con 0x7fecbc0cfa40 2023-05-22T15:39:27.684 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:27.679+0000 7fecc355e640 1 -- 172.21.15.16:0/3229643102 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fecbc0cfa40 msgr2=0x7fecbc0cd260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:27.685 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:27.679+0000 7fecc355e640 1 --2- 172.21.15.16:0/3229643102 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fecbc0cfa40 0x7fecbc0cd260 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7fecac0097f0 tx=0x7fecac01c480 comp rx=0 tx=0).stop 2023-05-22T15:39:27.685 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:27.679+0000 7fecc355e640 1 -- 172.21.15.16:0/3229643102 shutdown_connections 2023-05-22T15:39:27.685 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:27.679+0000 7fecc355e640 1 --2- 172.21.15.16:0/3229643102 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fecbc0cfa40 0x7fecbc0cd260 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:27.685 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:27.679+0000 7fecc355e640 1 --2- 172.21.15.16:0/3229643102 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fecbc0cf670 0x7fecbc1601d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:27.685 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:27.679+0000 7fecc355e640 1 -- 172.21.15.16:0/3229643102 >> 172.21.15.16:0/3229643102 conn(0x7fecbc0700b0 msgr2=0x7fecbc06d9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:39:27.685 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:27.679+0000 7fecc355e640 1 -- 172.21.15.16:0/3229643102 shutdown_connections 2023-05-22T15:39:27.685 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:27.679+0000 7fecc355e640 1 -- 172.21.15.16:0/3229643102 wait complete. 2023-05-22T15:39:27.686 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:27.679+0000 7fecc355e640 1 Processor -- start 2023-05-22T15:39:27.686 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:27.679+0000 7fecc355e640 1 -- start start 2023-05-22T15:39:27.687 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:27.683+0000 7fecc355e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fecbc06a830 0x7fecbc0aff60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:27.687 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:27.683+0000 7fecc355e640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fecbc0cf670 0x7fecbc15a210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:27.688 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:27.683+0000 7fecc355e640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fecbc0cfa40 0x7fecbc15a750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:27.688 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:27.683+0000 7fecc355e640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecbc158e10 con 0x7fecbc06a830 2023-05-22T15:39:27.688 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:27.683+0000 7fecc355e640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecbc15adf0 con 0x7fecbc0cf670 2023-05-22T15:39:27.688 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:27.683+0000 7fecc355e640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fecbc15b0d0 con 0x7fecbc0cfa40 2023-05-22T15:39:27.688 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:27.683+0000 7fecc12fa640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fecbc06a830 0x7fecbc0aff60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:27.688 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:27.683+0000 7fecc12fa640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fecbc06a830 0x7fecbc0aff60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52526/0 (socket says 172.21.15.16:52526) 2023-05-22T15:39:27.688 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:27.683+0000 7fecc12fa640 1 -- 172.21.15.16:0/847956680 learned_addr learned my addr 172.21.15.16:0/847956680 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:39:27.689 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:27.683+0000 7fecc1afb640 1 --2- 172.21.15.16:0/847956680 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fecbc0cfa40 0x7fecbc15a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:27.689 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:27.683+0000 7fecc0af9640 1 --2- 172.21.15.16:0/847956680 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fecbc0cf670 0x7fecbc15a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:27.689 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:27.683+0000 7fecc12fa640 1 -- 172.21.15.16:0/847956680 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fecbc0cfa40 msgr2=0x7fecbc15a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:27.689 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:27.683+0000 7fecc12fa640 1 --2- 172.21.15.16:0/847956680 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fecbc0cfa40 0x7fecbc15a750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:27.689 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:27.683+0000 7fecc12fa640 1 -- 172.21.15.16:0/847956680 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fecbc0cf670 msgr2=0x7fecbc15a210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:27.689 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:27.683+0000 7fecc12fa640 1 --2- 172.21.15.16:0/847956680 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fecbc0cf670 0x7fecbc15a210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:27.690 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:27.683+0000 7fecc12fa640 1 -- 172.21.15.16:0/847956680 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecac009420 con 0x7fecbc06a830 2023-05-22T15:39:27.690 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:27.683+0000 7fecc12fa640 1 --2- 172.21.15.16:0/847956680 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fecbc06a830 0x7fecbc0aff60 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7fecb4007ef0 tx=0x7fecb400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:39:27.690 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:27.683+0000 7fecb1ffb640 1 -- 172.21.15.16:0/847956680 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecb4014620 con 0x7fecbc06a830 2023-05-22T15:39:27.691 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:27.683+0000 7fecb1ffb640 1 -- 172.21.15.16:0/847956680 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fecb4014780 con 0x7fecbc06a830 2023-05-22T15:39:27.691 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:27.683+0000 7fecc355e640 1 -- 172.21.15.16:0/847956680 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fecbc06c0e0 con 0x7fecbc06a830 2023-05-22T15:39:27.691 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:27.683+0000 7fecb1ffb640 1 -- 172.21.15.16:0/847956680 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecb4014b30 con 0x7fecbc06a830 2023-05-22T15:39:27.691 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:27.683+0000 7fecc355e640 1 -- 172.21.15.16:0/847956680 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fecbc0cca60 con 0x7fecbc06a830 2023-05-22T15:39:27.692 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:27.683+0000 7fecc355e640 1 -- 172.21.15.16:0/847956680 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fecbc0cd7a0 con 0x7fecbc06a830 2023-05-22T15:39:27.692 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:27.683+0000 7fecb1ffb640 1 -- 172.21.15.16:0/847956680 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 12) v1 ==== 84418+0+0 (secure 0 0 0) 0x7fecb4014c90 con 0x7fecbc06a830 2023-05-22T15:39:27.692 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:27.683+0000 7fecb1ffb640 1 --2- 172.21.15.16:0/847956680 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7feca0065580 0x7feca0067a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:27.692 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:27.683+0000 7fecb1ffb640 1 -- 172.21.15.16:0/847956680 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 6698+0+0 (secure 0 0 0) 0x7fecb4083930 con 0x7fecbc06a830 2023-05-22T15:39:27.692 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:27.683+0000 7fecc0af9640 1 --2- 172.21.15.16:0/847956680 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7feca0065580 0x7feca0067a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:27.692 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:27.687+0000 7fecc0af9640 1 --2- 172.21.15.16:0/847956680 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7feca0065580 0x7feca0067a30 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fecac0ad000 tx=0x7fecac0a6000 comp rx=0 tx=0).ready entity=mgr.5823 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:39:27.694 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:27.687+0000 7fecb1ffb640 1 -- 172.21.15.16:0/847956680 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fecb4051010 con 0x7fecbc06a830 2023-05-22T15:39:27.866 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:27.863+0000 7fecc355e640 1 -- 172.21.15.16:0/847956680 --> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/snap_test_dir1/dir6", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fecbc0ce680 con 0x7feca0065580 2023-05-22T15:39:27.880 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:27.875+0000 7fecb1ffb640 1 -- 172.21.15.16:0/847956680 <== mgr.5823 v2:172.21.15.16:6832/36921 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+345 (secure 0 0 0) 0x7fecbc0ce680 con 0x7feca0065580 2023-05-22T15:39:27.880 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:39:27.881 INFO:teuthology.orchestra.run.smithi016.stdout:[{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/dir6", "rel_path": "/snap_test_dir1/dir6", "schedule": "1M", "retention": {}, "start": "2023-05-22T00:00:00", "created": "2023-05-22T15:28:34", "first": "2023-05-22T15:29:00", "last": "2023-05-22T15:39:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2023-05-22T15:39:27.882 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:27.879+0000 7fecc355e640 1 -- 172.21.15.16:0/847956680 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7feca0065580 msgr2=0x7feca0067a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:27.882 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:27.879+0000 7fecc355e640 1 --2- 172.21.15.16:0/847956680 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7feca0065580 0x7feca0067a30 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fecac0ad000 tx=0x7fecac0a6000 comp rx=0 tx=0).stop 2023-05-22T15:39:27.882 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:27.879+0000 7fecc355e640 1 -- 172.21.15.16:0/847956680 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fecbc06a830 msgr2=0x7fecbc0aff60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:27.883 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:27.879+0000 7fecc355e640 1 --2- 172.21.15.16:0/847956680 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fecbc06a830 0x7fecbc0aff60 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7fecb4007ef0 tx=0x7fecb400e4b0 comp rx=0 tx=0).stop 2023-05-22T15:39:27.883 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:27.879+0000 7fecc355e640 1 -- 172.21.15.16:0/847956680 shutdown_connections 2023-05-22T15:39:27.883 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:27.879+0000 7fecc355e640 1 --2- 172.21.15.16:0/847956680 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7feca0065580 0x7feca0067a30 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:27.883 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:27.879+0000 7fecc355e640 1 --2- 172.21.15.16:0/847956680 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fecbc0cfa40 0x7fecbc15a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:27.883 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:27.879+0000 7fecc355e640 1 --2- 172.21.15.16:0/847956680 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fecbc0cf670 0x7fecbc15a210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:27.883 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:27.879+0000 7fecc355e640 1 --2- 172.21.15.16:0/847956680 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fecbc06a830 0x7fecbc0aff60 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:27.883 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:27.879+0000 7fecc355e640 1 -- 172.21.15.16:0/847956680 >> 172.21.15.16:0/847956680 conn(0x7fecbc0700b0 msgr2=0x7fecbc0b1480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:39:27.884 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:27.879+0000 7fecc355e640 1 -- 172.21.15.16:0/847956680 shutdown_connections 2023-05-22T15:39:27.884 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:27.879+0000 7fecc355e640 1 -- 172.21.15.16:0/847956680 wait complete. 2023-05-22T15:39:27.899 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/dir6", "rel_path": "/snap_test_dir1/dir6", "schedule": "1M", "retention": {}, "start": "2023-05-22T00:00:00", "created": "2023-05-22T15:28:34", "first": "2023-05-22T15:29:00", "last": "2023-05-22T15:39:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2023-05-22T15:39:27.900 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'path': '/snap_test_dir1/dir6', 'rel_path': '/snap_test_dir1/dir6', 'schedule': '1M', 'retention': {}, 'start': '2023-05-22T00:00:00', 'created': '2023-05-22T15:28:34', 'first': '2023-05-22T15:29:00', 'last': '2023-05-22T15:39:00', 'last_pruned': None, 'created_count': 11, 'pruned_count': 0, 'active': False} 2023-05-22T15:39:27.900 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:39:27.901 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir7/.snap) 2023-05-22T15:39:27.912 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_29_00_UTC 2023-05-22T15:39:27.913 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_30_00_UTC 2023-05-22T15:39:27.913 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_31_00_UTC 2023-05-22T15:39:27.913 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_32_01_UTC 2023-05-22T15:39:27.913 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_33_00_UTC 2023-05-22T15:39:27.913 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_34_00_UTC 2023-05-22T15:39:27.913 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_35_00_UTC 2023-05-22T15:39:27.913 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_36_00_UTC 2023-05-22T15:39:27.914 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_37_01_UTC 2023-05-22T15:39:27.914 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_38_00_UTC 2023-05-22T15:39:27.914 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_39_00_UTC 2023-05-22T15:39:27.915 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2023-05-22-15_29_00_UTC', 'scheduled-2023-05-22-15_30_00_UTC', 'scheduled-2023-05-22-15_31_00_UTC', 'scheduled-2023-05-22-15_32_01_UTC', 'scheduled-2023-05-22-15_33_00_UTC', 'scheduled-2023-05-22-15_34_00_UTC', 'scheduled-2023-05-22-15_35_00_UTC', 'scheduled-2023-05-22-15_36_00_UTC', 'scheduled-2023-05-22-15_37_01_UTC', 'scheduled-2023-05-22-15_38_00_UTC', 'scheduled-2023-05-22-15_39_00_UTC'] 2023-05-22T15:39:27.915 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --fs cephfs --format json /snap_test_dir1/dir7 2023-05-22T15:39:28.091 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.087+0000 7f1939ad2640 1 Processor -- start 2023-05-22T15:39:28.091 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.087+0000 7f1939ad2640 1 -- start start 2023-05-22T15:39:28.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.087+0000 7f1939ad2640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1934069720 0x7f1934069af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:28.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.087+0000 7f1939ad2640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f19340718e0 0x7f193406ce10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:28.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.087+0000 7f1939ad2640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1934070d20 con 0x7f19340cc880 2023-05-22T15:39:28.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.087+0000 7f1939ad2640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19340b1c70 con 0x7f19340718e0 2023-05-22T15:39:28.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.087+0000 7f1939ad2640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f193415e5d0 con 0x7f1934069720 2023-05-22T15:39:28.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.087+0000 7f1933fff640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f19340718e0 0x7f193406ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:28.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.087+0000 7f19337fe640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1934069720 0x7f1934069af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:28.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.087+0000 7f19337fe640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1934069720 0x7f1934069af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3301/0 says I am v2:172.21.15.16:58448/0 (socket says 172.21.15.16:58448) 2023-05-22T15:39:28.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.087+0000 7f1933fff640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f19340718e0 0x7f193406ce10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.16:35060/0 (socket says 172.21.15.16:35060) 2023-05-22T15:39:28.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.087+0000 7f19337fe640 1 -- 172.21.15.16:0/2468049298 learned_addr learned my addr 172.21.15.16:0/2468049298 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:39:28.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.087+0000 7f19337fe640 1 -- 172.21.15.16:0/2468049298 >> v2:172.21.15.62:3300/0 conn(0x7f19340718e0 msgr2=0x7f193406ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:39:28.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.087+0000 7f19337fe640 1 --2- 172.21.15.16:0/2468049298 >> v2:172.21.15.62:3300/0 conn(0x7f19340718e0 0x7f193406ce10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:28.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.087+0000 7f19337fe640 1 -- 172.21.15.16:0/2468049298 >> v1:172.21.15.62:6789/0 conn(0x7f19340cc880 legacy=0x7f193406c700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:28.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.087+0000 7f19337fe640 1 -- 172.21.15.16:0/2468049298 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1934159f40 con 0x7f1934069720 2023-05-22T15:39:28.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.087+0000 7f1933fff640 1 --2- 172.21.15.16:0/2468049298 >> v2:172.21.15.62:3300/0 conn(0x7f19340718e0 0x7f193406ce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-05-22T15:39:28.096 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.091+0000 7f19337fe640 1 --2- 172.21.15.16:0/2468049298 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1934069720 0x7f1934069af0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f192c009e50 tx=0x7f192c01c980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ea6cd2f5f36c9fb5 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:39:28.096 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.091+0000 7f19327fc640 1 -- 172.21.15.16:0/2468049298 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f192c0a5840 con 0x7f1934069720 2023-05-22T15:39:28.097 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.091+0000 7f19327fc640 1 -- 172.21.15.16:0/2468049298 <== mon.2 v2:172.21.15.62:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f192c0a59a0 con 0x7f1934069720 2023-05-22T15:39:28.097 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.091+0000 7f19327fc640 1 -- 172.21.15.16:0/2468049298 <== mon.2 v2:172.21.15.62:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f192c0a5d30 con 0x7f1934069720 2023-05-22T15:39:28.097 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.091+0000 7f1939ad2640 1 -- 172.21.15.16:0/2468049298 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1934069720 msgr2=0x7f1934069af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:28.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.091+0000 7f1939ad2640 1 --2- 172.21.15.16:0/2468049298 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1934069720 0x7f1934069af0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f192c009e50 tx=0x7f192c01c980 comp rx=0 tx=0).stop 2023-05-22T15:39:28.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.091+0000 7f1939ad2640 1 -- 172.21.15.16:0/2468049298 shutdown_connections 2023-05-22T15:39:28.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.091+0000 7f1939ad2640 1 --2- 172.21.15.16:0/2468049298 >> v2:172.21.15.62:3300/0 conn(0x7f19340718e0 0x7f193406ce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:28.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.091+0000 7f1939ad2640 1 --2- 172.21.15.16:0/2468049298 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1934069720 0x7f1934069af0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:28.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.091+0000 7f1939ad2640 1 -- 172.21.15.16:0/2468049298 >> 172.21.15.16:0/2468049298 conn(0x7f19340b0cc0 msgr2=0x7f19340b10c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:39:28.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.091+0000 7f1939ad2640 1 -- 172.21.15.16:0/2468049298 shutdown_connections 2023-05-22T15:39:28.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.091+0000 7f1939ad2640 1 -- 172.21.15.16:0/2468049298 wait complete. 2023-05-22T15:39:28.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.095+0000 7f1939ad2640 1 Processor -- start 2023-05-22T15:39:28.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.095+0000 7f1939ad2640 1 -- start start 2023-05-22T15:39:28.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.095+0000 7f1939ad2640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1934069720 0x7f19340c7d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:28.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.095+0000 7f1939ad2640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f19340c94a0 0x7f1934159510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:28.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.095+0000 7f1939ad2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f19340c6270 0x7f193413f1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:28.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.095+0000 7f1939ad2640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19340c5670 con 0x7f19340c6270 2023-05-22T15:39:28.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.095+0000 7f1939ad2640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1934159a50 con 0x7f1934069720 2023-05-22T15:39:28.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.095+0000 7f1939ad2640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19340c8400 con 0x7f19340c94a0 2023-05-22T15:39:28.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.095+0000 7f1933fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f19340c6270 0x7f193413f1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:28.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.095+0000 7f1933fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f19340c6270 0x7f193413f1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52528/0 (socket says 172.21.15.16:52528) 2023-05-22T15:39:28.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.095+0000 7f1933fff640 1 -- 172.21.15.16:0/3167288513 learned_addr learned my addr 172.21.15.16:0/3167288513 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:39:28.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.095+0000 7f19337fe640 1 --2- 172.21.15.16:0/3167288513 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1934069720 0x7f19340c7d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:28.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.095+0000 7f1932ffd640 1 --2- 172.21.15.16:0/3167288513 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f19340c94a0 0x7f1934159510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:28.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.095+0000 7f1933fff640 1 -- 172.21.15.16:0/3167288513 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f19340c94a0 msgr2=0x7f1934159510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:28.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.095+0000 7f1933fff640 1 --2- 172.21.15.16:0/3167288513 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f19340c94a0 0x7f1934159510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:28.103 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.095+0000 7f1933fff640 1 -- 172.21.15.16:0/3167288513 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1934069720 msgr2=0x7f19340c7d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:28.103 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.095+0000 7f1933fff640 1 --2- 172.21.15.16:0/3167288513 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1934069720 0x7f19340c7d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:28.103 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.095+0000 7f1933fff640 1 -- 172.21.15.16:0/3167288513 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1924009420 con 0x7f19340c6270 2023-05-22T15:39:28.103 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.095+0000 7f1932ffd640 1 --2- 172.21.15.16:0/3167288513 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f19340c94a0 0x7f1934159510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:39:28.103 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.095+0000 7f1933fff640 1 --2- 172.21.15.16:0/3167288513 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f19340c6270 0x7f193413f1c0 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f192400e8e0 tx=0x7f192400eda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:39:28.104 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.095+0000 7f192bfff640 1 -- 172.21.15.16:0/3167288513 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f192400c940 con 0x7f19340c6270 2023-05-22T15:39:28.104 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.095+0000 7f192bfff640 1 -- 172.21.15.16:0/3167288513 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f192400cac0 con 0x7f19340c6270 2023-05-22T15:39:28.104 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.095+0000 7f1939ad2640 1 -- 172.21.15.16:0/3167288513 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f192c009b30 con 0x7f19340c6270 2023-05-22T15:39:28.105 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.095+0000 7f192bfff640 1 -- 172.21.15.16:0/3167288513 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f192400ce70 con 0x7f19340c6270 2023-05-22T15:39:28.105 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.095+0000 7f1939ad2640 1 -- 172.21.15.16:0/3167288513 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f193415ddd0 con 0x7f19340c6270 2023-05-22T15:39:28.105 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.099+0000 7f1939ad2640 1 -- 172.21.15.16:0/3167288513 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f193415daf0 con 0x7f19340c6270 2023-05-22T15:39:28.105 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.099+0000 7f192bfff640 1 -- 172.21.15.16:0/3167288513 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 12) v1 ==== 84418+0+0 (secure 0 0 0) 0x7f1924013470 con 0x7f19340c6270 2023-05-22T15:39:28.105 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.099+0000 7f192bfff640 1 --2- 172.21.15.16:0/3167288513 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f1914065580 0x7f1914067a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:28.105 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.099+0000 7f192bfff640 1 -- 172.21.15.16:0/3167288513 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 6698+0+0 (secure 0 0 0) 0x7f1924083950 con 0x7f19340c6270 2023-05-22T15:39:28.105 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.099+0000 7f19337fe640 1 --2- 172.21.15.16:0/3167288513 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f1914065580 0x7f1914067a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:28.105 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.099+0000 7f19337fe640 1 --2- 172.21.15.16:0/3167288513 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f1914065580 0x7f1914067a30 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f192c0ae000 tx=0x7f192c0a7070 comp rx=0 tx=0).ready entity=mgr.5823 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:39:28.107 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.103+0000 7f192bfff640 1 -- 172.21.15.16:0/3167288513 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f19240510c0 con 0x7f19340c6270 2023-05-22T15:39:28.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.271+0000 7f1939ad2640 1 -- 172.21.15.16:0/3167288513 --> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/snap_test_dir1/dir7", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f193415e0b0 con 0x7f1914065580 2023-05-22T15:39:28.291 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.287+0000 7f192bfff640 1 -- 172.21.15.16:0/3167288513 <== mgr.5823 v2:172.21.15.16:6832/36921 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+345 (secure 0 0 0) 0x7f193415e0b0 con 0x7f1914065580 2023-05-22T15:39:28.291 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:39:28.291 INFO:teuthology.orchestra.run.smithi016.stdout:[{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/dir7", "rel_path": "/snap_test_dir1/dir7", "schedule": "1M", "retention": {}, "start": "2023-05-22T00:00:00", "created": "2023-05-22T15:28:35", "first": "2023-05-22T15:29:00", "last": "2023-05-22T15:39:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2023-05-22T15:39:28.293 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.287+0000 7f1939ad2640 1 -- 172.21.15.16:0/3167288513 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f1914065580 msgr2=0x7f1914067a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:28.294 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.287+0000 7f1939ad2640 1 --2- 172.21.15.16:0/3167288513 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f1914065580 0x7f1914067a30 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f192c0ae000 tx=0x7f192c0a7070 comp rx=0 tx=0).stop 2023-05-22T15:39:28.294 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.287+0000 7f1939ad2640 1 -- 172.21.15.16:0/3167288513 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f19340c6270 msgr2=0x7f193413f1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:28.294 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.287+0000 7f1939ad2640 1 --2- 172.21.15.16:0/3167288513 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f19340c6270 0x7f193413f1c0 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f192400e8e0 tx=0x7f192400eda0 comp rx=0 tx=0).stop 2023-05-22T15:39:28.294 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.287+0000 7f1939ad2640 1 -- 172.21.15.16:0/3167288513 shutdown_connections 2023-05-22T15:39:28.294 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.287+0000 7f1939ad2640 1 --2- 172.21.15.16:0/3167288513 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f19340c6270 0x7f193413f1c0 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:28.294 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.287+0000 7f1939ad2640 1 --2- 172.21.15.16:0/3167288513 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f1914065580 0x7f1914067a30 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:28.294 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.287+0000 7f1939ad2640 1 --2- 172.21.15.16:0/3167288513 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f19340c94a0 0x7f1934159510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:28.295 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.287+0000 7f1939ad2640 1 --2- 172.21.15.16:0/3167288513 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1934069720 0x7f19340c7d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:28.295 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.287+0000 7f1939ad2640 1 -- 172.21.15.16:0/3167288513 >> 172.21.15.16:0/3167288513 conn(0x7f19340b0cc0 msgr2=0x7f19340aeb30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:39:28.295 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.287+0000 7f1939ad2640 1 -- 172.21.15.16:0/3167288513 shutdown_connections 2023-05-22T15:39:28.295 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.287+0000 7f1939ad2640 1 -- 172.21.15.16:0/3167288513 wait complete. 2023-05-22T15:39:28.312 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/dir7", "rel_path": "/snap_test_dir1/dir7", "schedule": "1M", "retention": {}, "start": "2023-05-22T00:00:00", "created": "2023-05-22T15:28:35", "first": "2023-05-22T15:29:00", "last": "2023-05-22T15:39:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2023-05-22T15:39:28.312 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'path': '/snap_test_dir1/dir7', 'rel_path': '/snap_test_dir1/dir7', 'schedule': '1M', 'retention': {}, 'start': '2023-05-22T00:00:00', 'created': '2023-05-22T15:28:35', 'first': '2023-05-22T15:29:00', 'last': '2023-05-22T15:39:00', 'last_pruned': None, 'created_count': 11, 'pruned_count': 0, 'active': False} 2023-05-22T15:39:28.313 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:39:28.313 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir8/.snap) 2023-05-22T15:39:28.328 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_29_00_UTC 2023-05-22T15:39:28.328 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_30_00_UTC 2023-05-22T15:39:28.328 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_31_00_UTC 2023-05-22T15:39:28.328 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_32_00_UTC 2023-05-22T15:39:28.328 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_33_00_UTC 2023-05-22T15:39:28.329 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_34_01_UTC 2023-05-22T15:39:28.329 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_35_00_UTC 2023-05-22T15:39:28.329 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_36_00_UTC 2023-05-22T15:39:28.329 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_37_00_UTC 2023-05-22T15:39:28.329 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_38_01_UTC 2023-05-22T15:39:28.329 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_39_01_UTC 2023-05-22T15:39:28.329 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2023-05-22-15_29_00_UTC', 'scheduled-2023-05-22-15_30_00_UTC', 'scheduled-2023-05-22-15_31_00_UTC', 'scheduled-2023-05-22-15_32_00_UTC', 'scheduled-2023-05-22-15_33_00_UTC', 'scheduled-2023-05-22-15_34_01_UTC', 'scheduled-2023-05-22-15_35_00_UTC', 'scheduled-2023-05-22-15_36_00_UTC', 'scheduled-2023-05-22-15_37_00_UTC', 'scheduled-2023-05-22-15_38_01_UTC', 'scheduled-2023-05-22-15_39_01_UTC'] 2023-05-22T15:39:28.330 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --fs cephfs --format json /snap_test_dir1/dir8 2023-05-22T15:39:28.504 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.499+0000 7fa4233e8640 1 Processor -- start 2023-05-22T15:39:28.505 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.499+0000 7fa4233e8640 1 -- start start 2023-05-22T15:39:28.505 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.499+0000 7fa4233e8640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fa41c0cc380 0x7fa41c0c33f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:28.505 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.499+0000 7fa4233e8640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa41c158340 0x7fa41c06fbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:28.506 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.499+0000 7fa4233e8640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa41c0c30e0 con 0x7fa41c157f70 2023-05-22T15:39:28.506 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.499+0000 7fa4233e8640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa41c06f770 con 0x7fa41c0cc380 2023-05-22T15:39:28.506 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.499+0000 7fa4233e8640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa41c0c3a40 con 0x7fa41c158340 2023-05-22T15:39:28.506 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.499+0000 7fa421184640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fa41c157f70 0x7fa41c0cbc70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:53100/0 (socket says 172.21.15.16:53100) 2023-05-22T15:39:28.506 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.499+0000 7fa421184640 1 -- 172.21.15.16:0/2731734912 learned_addr learned my addr 172.21.15.16:0/2731734912 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:39:28.507 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.499+0000 7fa421985640 1 --2- 172.21.15.16:0/2731734912 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa41c158340 0x7fa41c06fbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:28.508 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.499+0000 7fa420983640 1 --2- 172.21.15.16:0/2731734912 >> v2:172.21.15.62:3300/0 conn(0x7fa41c0cc380 0x7fa41c0c33f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:28.508 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.499+0000 7fa413fff640 1 -- 172.21.15.16:0/2731734912 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3646118911 0 0) 0x7fa41c0c30e0 con 0x7fa41c157f70 2023-05-22T15:39:28.508 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.499+0000 7fa413fff640 1 -- 172.21.15.16:0/2731734912 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa408003530 con 0x7fa41c157f70 2023-05-22T15:39:28.508 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.503+0000 7fa413fff640 1 -- 172.21.15.16:0/2731734912 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2902088721 0 0) 0x7fa408003530 con 0x7fa41c157f70 2023-05-22T15:39:28.508 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.503+0000 7fa413fff640 1 -- 172.21.15.16:0/2731734912 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa41c158340 msgr2=0x7fa41c06fbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:39:28.508 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.503+0000 7fa413fff640 1 --2- 172.21.15.16:0/2731734912 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa41c158340 0x7fa41c06fbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:28.509 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.503+0000 7fa413fff640 1 -- 172.21.15.16:0/2731734912 >> v2:172.21.15.62:3300/0 conn(0x7fa41c0cc380 msgr2=0x7fa41c0c33f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:39:28.509 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.503+0000 7fa413fff640 1 --2- 172.21.15.16:0/2731734912 >> v2:172.21.15.62:3300/0 conn(0x7fa41c0cc380 0x7fa41c0c33f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:28.509 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.503+0000 7fa413fff640 1 -- 172.21.15.16:0/2731734912 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa41c0c3d20 con 0x7fa41c157f70 2023-05-22T15:39:28.509 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.503+0000 7fa413fff640 1 -- 172.21.15.16:0/2731734912 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7fa418001680 con 0x7fa41c157f70 2023-05-22T15:39:28.509 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.503+0000 7fa413fff640 1 -- 172.21.15.16:0/2731734912 >> v1:172.21.15.16:6789/0 conn(0x7fa41c157f70 legacy=0x7fa41c0cbc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:28.509 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.503+0000 7fa413fff640 1 --2- 172.21.15.16:0/2731734912 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa41c158340 0x7fa408003980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:28.509 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.503+0000 7fa413fff640 1 --2- 172.21.15.16:0/2731734912 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa408003f00 0x7fa4080047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:28.509 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.503+0000 7fa413fff640 1 --2- 172.21.15.16:0/2731734912 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa408004d10 0x7fa408005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:28.510 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.503+0000 7fa413fff640 1 -- 172.21.15.16:0/2731734912 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa41c0c3a40 con 0x7fa408003f00 2023-05-22T15:39:28.510 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.503+0000 7fa413fff640 1 -- 172.21.15.16:0/2731734912 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa40800b990 con 0x7fa408004d10 2023-05-22T15:39:28.510 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.503+0000 7fa413fff640 1 -- 172.21.15.16:0/2731734912 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa40800bad0 con 0x7fa41c158340 2023-05-22T15:39:28.510 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.503+0000 7fa420983640 1 --2- 172.21.15.16:0/2731734912 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa408003f00 0x7fa4080047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:28.510 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.503+0000 7fa421184640 1 --2- 172.21.15.16:0/2731734912 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa41c158340 0x7fa408003980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:28.510 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.503+0000 7fa421985640 1 --2- 172.21.15.16:0/2731734912 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa408004d10 0x7fa408005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:28.510 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.503+0000 7fa420983640 1 -- 172.21.15.16:0/2731734912 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa41c158340 msgr2=0x7fa408003980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:39:28.511 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.503+0000 7fa420983640 1 --2- 172.21.15.16:0/2731734912 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa41c158340 0x7fa408003980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:28.511 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.503+0000 7fa420983640 1 -- 172.21.15.16:0/2731734912 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa408004d10 msgr2=0x7fa408005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:39:28.511 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.503+0000 7fa420983640 1 --2- 172.21.15.16:0/2731734912 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa408004d10 0x7fa408005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:28.511 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.503+0000 7fa420983640 1 -- 172.21.15.16:0/2731734912 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa408003530 con 0x7fa408003f00 2023-05-22T15:39:28.511 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.503+0000 7fa421985640 1 --2- 172.21.15.16:0/2731734912 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa408004d10 0x7fa408005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:39:28.514 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.503+0000 7fa420983640 1 --2- 172.21.15.16:0/2731734912 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa408003f00 0x7fa4080047d0 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7fa40c009ab0 tx=0x7fa40c01ca70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84150e49f468526b server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:39:28.514 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.503+0000 7fa413fff640 1 -- 172.21.15.16:0/2731734912 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa40c0a5550 con 0x7fa408003f00 2023-05-22T15:39:28.514 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.503+0000 7fa413fff640 1 -- 172.21.15.16:0/2731734912 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa40c0a56b0 con 0x7fa408003f00 2023-05-22T15:39:28.514 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.503+0000 7fa413fff640 1 -- 172.21.15.16:0/2731734912 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa40c0a5a60 con 0x7fa408003f00 2023-05-22T15:39:28.514 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.503+0000 7fa4233e8640 1 -- 172.21.15.16:0/2731734912 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa408003f00 msgr2=0x7fa4080047d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:28.514 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.503+0000 7fa4233e8640 1 --2- 172.21.15.16:0/2731734912 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa408003f00 0x7fa4080047d0 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7fa40c009ab0 tx=0x7fa40c01ca70 comp rx=0 tx=0).stop 2023-05-22T15:39:28.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.503+0000 7fa421985640 1 -- 172.21.15.16:0/2731734912 reap_dead start 2023-05-22T15:39:28.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.503+0000 7fa4233e8640 1 -- 172.21.15.16:0/2731734912 shutdown_connections 2023-05-22T15:39:28.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.503+0000 7fa4233e8640 1 -- 172.21.15.16:0/2731734912 >> 172.21.15.16:0/2731734912 conn(0x7fa41c069720 msgr2=0x7fa41c069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:39:28.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.503+0000 7fa4233e8640 1 -- 172.21.15.16:0/2731734912 shutdown_connections 2023-05-22T15:39:28.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.503+0000 7fa4233e8640 1 -- 172.21.15.16:0/2731734912 wait complete. 2023-05-22T15:39:28.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.507+0000 7fa4233e8640 1 Processor -- start 2023-05-22T15:39:28.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.507+0000 7fa4233e8640 1 -- start start 2023-05-22T15:39:28.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.507+0000 7fa4233e8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa408003f00 0x7fa41c1537c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:28.516 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.507+0000 7fa4233e8640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa41c155170 0x7fa41c1555c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:28.516 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.507+0000 7fa4233e8640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa41c0c5170 0x7fa41c0c55e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:28.516 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.507+0000 7fa4233e8640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa41c155b00 con 0x7fa408003f00 2023-05-22T15:39:28.516 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.507+0000 7fa4233e8640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa41c0c5b20 con 0x7fa41c0c5170 2023-05-22T15:39:28.516 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.507+0000 7fa4233e8640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa41c153d00 con 0x7fa41c155170 2023-05-22T15:39:28.516 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.507+0000 7fa421184640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa408003f00 0x7fa41c1537c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:28.516 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.507+0000 7fa421184640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa408003f00 0x7fa41c1537c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52548/0 (socket says 172.21.15.16:52548) 2023-05-22T15:39:28.517 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.507+0000 7fa421184640 1 -- 172.21.15.16:0/3826899986 learned_addr learned my addr 172.21.15.16:0/3826899986 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:39:28.517 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.507+0000 7fa420983640 1 --2- 172.21.15.16:0/3826899986 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa41c155170 0x7fa41c1555c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:28.517 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.507+0000 7fa421985640 1 --2- 172.21.15.16:0/3826899986 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa41c0c5170 0x7fa41c0c55e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:28.517 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.507+0000 7fa421184640 1 -- 172.21.15.16:0/3826899986 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa41c155170 msgr2=0x7fa41c1555c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:28.517 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.507+0000 7fa421184640 1 --2- 172.21.15.16:0/3826899986 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa41c155170 0x7fa41c1555c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:28.517 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.507+0000 7fa421184640 1 -- 172.21.15.16:0/3826899986 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa41c0c5170 msgr2=0x7fa41c0c55e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:28.517 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.507+0000 7fa421184640 1 --2- 172.21.15.16:0/3826899986 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa41c0c5170 0x7fa41c0c55e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:28.518 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.507+0000 7fa421184640 1 -- 172.21.15.16:0/3826899986 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa41c0c42e0 con 0x7fa408003f00 2023-05-22T15:39:28.518 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.507+0000 7fa421184640 1 --2- 172.21.15.16:0/3826899986 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa408003f00 0x7fa41c1537c0 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7fa4180118e0 tx=0x7fa418011da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:39:28.518 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.507+0000 7fa411ffb640 1 -- 172.21.15.16:0/3826899986 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa41800fba0 con 0x7fa408003f00 2023-05-22T15:39:28.518 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.507+0000 7fa411ffb640 1 -- 172.21.15.16:0/3826899986 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa41800fd20 con 0x7fa408003f00 2023-05-22T15:39:28.518 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.507+0000 7fa4233e8640 1 -- 172.21.15.16:0/3826899986 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa41c0c4000 con 0x7fa408003f00 2023-05-22T15:39:28.518 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.507+0000 7fa411ffb640 1 -- 172.21.15.16:0/3826899986 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa418015650 con 0x7fa408003f00 2023-05-22T15:39:28.518 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.507+0000 7fa4233e8640 1 -- 172.21.15.16:0/3826899986 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa41c15bcf0 con 0x7fa408003f00 2023-05-22T15:39:28.519 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.507+0000 7fa4233e8640 1 -- 172.21.15.16:0/3826899986 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa41c04dbe0 con 0x7fa408003f00 2023-05-22T15:39:28.519 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.511+0000 7fa411ffb640 1 -- 172.21.15.16:0/3826899986 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 12) v1 ==== 84418+0+0 (secure 0 0 0) 0x7fa4180157b0 con 0x7fa408003f00 2023-05-22T15:39:28.519 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.511+0000 7fa411ffb640 1 --2- 172.21.15.16:0/3826899986 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fa400065580 0x7fa400067a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:28.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.511+0000 7fa411ffb640 1 -- 172.21.15.16:0/3826899986 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 6698+0+0 (secure 0 0 0) 0x7fa418086d50 con 0x7fa408003f00 2023-05-22T15:39:28.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.511+0000 7fa420983640 1 --2- 172.21.15.16:0/3826899986 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fa400065580 0x7fa400067a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:28.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.511+0000 7fa420983640 1 --2- 172.21.15.16:0/3826899986 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fa400065580 0x7fa400067a30 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fa40c0af000 tx=0x7fa40c0046d0 comp rx=0 tx=0).ready entity=mgr.5823 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:39:28.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.515+0000 7fa411ffb640 1 -- 172.21.15.16:0/3826899986 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fa418054520 con 0x7fa408003f00 2023-05-22T15:39:28.689 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.683+0000 7fa4233e8640 1 -- 172.21.15.16:0/3826899986 --> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/snap_test_dir1/dir8", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fa41c0cee10 con 0x7fa400065580 2023-05-22T15:39:28.705 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.699+0000 7fa411ffb640 1 -- 172.21.15.16:0/3826899986 <== mgr.5823 v2:172.21.15.16:6832/36921 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+345 (secure 0 0 0) 0x7fa41c0cee10 con 0x7fa400065580 2023-05-22T15:39:28.706 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:39:28.706 INFO:teuthology.orchestra.run.smithi016.stdout:[{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/dir8", "rel_path": "/snap_test_dir1/dir8", "schedule": "1M", "retention": {}, "start": "2023-05-22T00:00:00", "created": "2023-05-22T15:28:35", "first": "2023-05-22T15:29:00", "last": "2023-05-22T15:39:01", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2023-05-22T15:39:28.707 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.703+0000 7fa4233e8640 1 -- 172.21.15.16:0/3826899986 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fa400065580 msgr2=0x7fa400067a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:28.707 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.703+0000 7fa4233e8640 1 --2- 172.21.15.16:0/3826899986 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fa400065580 0x7fa400067a30 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fa40c0af000 tx=0x7fa40c0046d0 comp rx=0 tx=0).stop 2023-05-22T15:39:28.708 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.703+0000 7fa4233e8640 1 -- 172.21.15.16:0/3826899986 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa408003f00 msgr2=0x7fa41c1537c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:28.708 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.703+0000 7fa4233e8640 1 --2- 172.21.15.16:0/3826899986 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa408003f00 0x7fa41c1537c0 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7fa4180118e0 tx=0x7fa418011da0 comp rx=0 tx=0).stop 2023-05-22T15:39:28.708 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.703+0000 7fa4233e8640 1 -- 172.21.15.16:0/3826899986 shutdown_connections 2023-05-22T15:39:28.708 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.703+0000 7fa4233e8640 1 --2- 172.21.15.16:0/3826899986 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa41c0c5170 0x7fa41c0c55e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:28.708 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.703+0000 7fa4233e8640 1 --2- 172.21.15.16:0/3826899986 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fa400065580 0x7fa400067a30 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:28.708 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.703+0000 7fa4233e8640 1 --2- 172.21.15.16:0/3826899986 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa41c155170 0x7fa41c1555c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:28.709 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.703+0000 7fa4233e8640 1 --2- 172.21.15.16:0/3826899986 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa408003f00 0x7fa41c1537c0 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:28.709 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.703+0000 7fa4233e8640 1 -- 172.21.15.16:0/3826899986 >> 172.21.15.16:0/3826899986 conn(0x7fa41c069720 msgr2=0x7fa41c0b0fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:39:28.709 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.703+0000 7fa4233e8640 1 -- 172.21.15.16:0/3826899986 shutdown_connections 2023-05-22T15:39:28.709 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.703+0000 7fa4233e8640 1 -- 172.21.15.16:0/3826899986 wait complete. 2023-05-22T15:39:28.725 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/dir8", "rel_path": "/snap_test_dir1/dir8", "schedule": "1M", "retention": {}, "start": "2023-05-22T00:00:00", "created": "2023-05-22T15:28:35", "first": "2023-05-22T15:29:00", "last": "2023-05-22T15:39:01", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2023-05-22T15:39:28.725 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'path': '/snap_test_dir1/dir8', 'rel_path': '/snap_test_dir1/dir8', 'schedule': '1M', 'retention': {}, 'start': '2023-05-22T00:00:00', 'created': '2023-05-22T15:28:35', 'first': '2023-05-22T15:29:00', 'last': '2023-05-22T15:39:01', 'last_pruned': None, 'created_count': 11, 'pruned_count': 0, 'active': False} 2023-05-22T15:39:28.725 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:39:28.726 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir9/.snap) 2023-05-22T15:39:28.738 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_29_01_UTC 2023-05-22T15:39:28.739 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_30_00_UTC 2023-05-22T15:39:28.739 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_31_00_UTC 2023-05-22T15:39:28.739 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_32_00_UTC 2023-05-22T15:39:28.739 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_33_01_UTC 2023-05-22T15:39:28.739 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_34_00_UTC 2023-05-22T15:39:28.739 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_35_00_UTC 2023-05-22T15:39:28.740 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_36_01_UTC 2023-05-22T15:39:28.740 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_37_01_UTC 2023-05-22T15:39:28.740 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_38_01_UTC 2023-05-22T15:39:28.740 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_39_00_UTC 2023-05-22T15:39:28.740 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2023-05-22-15_29_01_UTC', 'scheduled-2023-05-22-15_30_00_UTC', 'scheduled-2023-05-22-15_31_00_UTC', 'scheduled-2023-05-22-15_32_00_UTC', 'scheduled-2023-05-22-15_33_01_UTC', 'scheduled-2023-05-22-15_34_00_UTC', 'scheduled-2023-05-22-15_35_00_UTC', 'scheduled-2023-05-22-15_36_01_UTC', 'scheduled-2023-05-22-15_37_01_UTC', 'scheduled-2023-05-22-15_38_01_UTC', 'scheduled-2023-05-22-15_39_00_UTC'] 2023-05-22T15:39:28.741 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --fs cephfs --format json /snap_test_dir1/dir9 2023-05-22T15:39:28.912 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.907+0000 7f3e50d98640 1 Processor -- start 2023-05-22T15:39:28.912 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.907+0000 7f3e50d98640 1 -- start start 2023-05-22T15:39:28.913 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.907+0000 7f3e50d98640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f3e4c0cdf40 0x7f3e4c069870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:28.913 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.907+0000 7f3e50d98640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3e4c069db0 0x7f3e4c0b0020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:28.913 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.907+0000 7f3e50d98640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e4c159440 con 0x7f3e4c0ccf50 2023-05-22T15:39:28.913 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.907+0000 7f3e50d98640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e4c158b10 con 0x7f3e4c0cdf40 2023-05-22T15:39:28.914 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.907+0000 7f3e50d98640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e4c158df0 con 0x7f3e4c069db0 2023-05-22T15:39:28.914 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.907+0000 7f3e49d9b640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f3e4c0cdf40 0x7f3e4c069870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:28.914 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.907+0000 7f3e49d9b640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f3e4c0cdf40 0x7f3e4c069870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52556/0 (socket says 172.21.15.16:52556) 2023-05-22T15:39:28.914 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.907+0000 7f3e49d9b640 1 -- 172.21.15.16:0/2208806866 learned_addr learned my addr 172.21.15.16:0/2208806866 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:39:28.914 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.907+0000 7f3e4ad9d640 1 --2- 172.21.15.16:0/2208806866 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3e4c069db0 0x7f3e4c0b0020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:28.914 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.907+0000 7f3e49d9b640 1 -- 172.21.15.16:0/2208806866 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3e4c069db0 msgr2=0x7f3e4c0b0020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:39:28.914 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.907+0000 7f3e49d9b640 1 --2- 172.21.15.16:0/2208806866 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3e4c069db0 0x7f3e4c0b0020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:28.915 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.907+0000 7f3e49d9b640 1 -- 172.21.15.16:0/2208806866 >> v1:172.21.15.62:6789/0 conn(0x7f3e4c0ccf50 legacy=0x7f3e4c0cd830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:39:28.915 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.907+0000 7f3e49d9b640 1 -- 172.21.15.16:0/2208806866 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e4c1590d0 con 0x7f3e4c0cdf40 2023-05-22T15:39:28.918 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.915+0000 7f3e49d9b640 1 --2- 172.21.15.16:0/2208806866 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3e4c0cdf40 0x7f3e4c069870 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f3e34009870 tx=0x7f3e3401c540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9eea0a4332e9f090 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:39:28.919 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.915+0000 7f3e4959a640 1 -- 172.21.15.16:0/2208806866 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e340ab680 con 0x7f3e4c0cdf40 2023-05-22T15:39:28.919 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.915+0000 7f3e4959a640 1 -- 172.21.15.16:0/2208806866 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e340ab7e0 con 0x7f3e4c0cdf40 2023-05-22T15:39:28.919 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.915+0000 7f3e4959a640 1 -- 172.21.15.16:0/2208806866 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e340abb70 con 0x7f3e4c0cdf40 2023-05-22T15:39:28.920 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.915+0000 7f3e50d98640 1 -- 172.21.15.16:0/2208806866 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3e4c0cdf40 msgr2=0x7f3e4c069870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:28.920 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.915+0000 7f3e50d98640 1 --2- 172.21.15.16:0/2208806866 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3e4c0cdf40 0x7f3e4c069870 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f3e34009870 tx=0x7f3e3401c540 comp rx=0 tx=0).stop 2023-05-22T15:39:28.920 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.915+0000 7f3e50d98640 1 -- 172.21.15.16:0/2208806866 shutdown_connections 2023-05-22T15:39:28.920 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.915+0000 7f3e50d98640 1 --2- 172.21.15.16:0/2208806866 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3e4c069db0 0x7f3e4c0b0020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:28.920 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.915+0000 7f3e50d98640 1 --2- 172.21.15.16:0/2208806866 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3e4c0cdf40 0x7f3e4c069870 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:28.920 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.915+0000 7f3e50d98640 1 -- 172.21.15.16:0/2208806866 >> 172.21.15.16:0/2208806866 conn(0x7f3e4c0cafb0 msgr2=0x7f3e4c0cb3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:39:28.921 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.915+0000 7f3e50d98640 1 -- 172.21.15.16:0/2208806866 shutdown_connections 2023-05-22T15:39:28.921 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.915+0000 7f3e50d98640 1 -- 172.21.15.16:0/2208806866 wait complete. 2023-05-22T15:39:28.922 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.915+0000 7f3e50d98640 1 Processor -- start 2023-05-22T15:39:28.922 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.915+0000 7f3e50d98640 1 -- start start 2023-05-22T15:39:28.922 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.915+0000 7f3e50d98640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3e4c069db0 0x7f3e4c14baa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:28.922 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.915+0000 7f3e50d98640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3e4c0ccf50 0x7f3e4c148e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:28.922 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.915+0000 7f3e50d98640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3e4c0cdf40 0x7f3e4c0ae240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:28.922 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.915+0000 7f3e50d98640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e4c150810 con 0x7f3e4c069db0 2023-05-22T15:39:28.923 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.915+0000 7f3e50d98640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e4c0c3190 con 0x7f3e4c0ccf50 2023-05-22T15:39:28.923 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.915+0000 7f3e50d98640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e4c149360 con 0x7f3e4c0cdf40 2023-05-22T15:39:28.923 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.915+0000 7f3e4a59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3e4c069db0 0x7f3e4c14baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:28.923 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.915+0000 7f3e4a59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3e4c069db0 0x7f3e4c14baa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52564/0 (socket says 172.21.15.16:52564) 2023-05-22T15:39:28.923 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.915+0000 7f3e4a59c640 1 -- 172.21.15.16:0/2302225034 learned_addr learned my addr 172.21.15.16:0/2302225034 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:39:28.923 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.919+0000 7f3e4ad9d640 1 --2- 172.21.15.16:0/2302225034 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3e4c0cdf40 0x7f3e4c0ae240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:28.923 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.919+0000 7f3e49d9b640 1 --2- 172.21.15.16:0/2302225034 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3e4c0ccf50 0x7f3e4c148e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:28.926 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.919+0000 7f3e4a59c640 1 -- 172.21.15.16:0/2302225034 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3e4c0cdf40 msgr2=0x7f3e4c0ae240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:28.926 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.919+0000 7f3e4a59c640 1 --2- 172.21.15.16:0/2302225034 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3e4c0cdf40 0x7f3e4c0ae240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:28.926 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.919+0000 7f3e4a59c640 1 -- 172.21.15.16:0/2302225034 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3e4c0ccf50 msgr2=0x7f3e4c148e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:28.927 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.919+0000 7f3e4a59c640 1 --2- 172.21.15.16:0/2302225034 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3e4c0ccf50 0x7f3e4c148e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:28.927 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.919+0000 7f3e4a59c640 1 -- 172.21.15.16:0/2302225034 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e340094d0 con 0x7f3e4c069db0 2023-05-22T15:39:28.927 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.919+0000 7f3e49d9b640 1 --2- 172.21.15.16:0/2302225034 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3e4c0ccf50 0x7f3e4c148e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T15:39:28.927 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.919+0000 7f3e4a59c640 1 --2- 172.21.15.16:0/2302225034 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3e4c069db0 0x7f3e4c14baa0 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f3e4400c480 tx=0x7f3e4400c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:39:28.927 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.919+0000 7f3e42ffd640 1 -- 172.21.15.16:0/2302225034 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e440052d0 con 0x7f3e4c069db0 2023-05-22T15:39:28.927 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.919+0000 7f3e42ffd640 1 -- 172.21.15.16:0/2302225034 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e44005430 con 0x7f3e4c069db0 2023-05-22T15:39:28.927 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.919+0000 7f3e50d98640 1 -- 172.21.15.16:0/2302225034 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e4c0b06f0 con 0x7f3e4c069db0 2023-05-22T15:39:28.928 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.919+0000 7f3e42ffd640 1 -- 172.21.15.16:0/2302225034 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e44007260 con 0x7f3e4c069db0 2023-05-22T15:39:28.928 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.919+0000 7f3e50d98640 1 -- 172.21.15.16:0/2302225034 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e4c0b0560 con 0x7f3e4c069db0 2023-05-22T15:39:28.928 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.919+0000 7f3e50d98640 1 -- 172.21.15.16:0/2302225034 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e4c06a1b0 con 0x7f3e4c069db0 2023-05-22T15:39:28.928 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.919+0000 7f3e42ffd640 1 -- 172.21.15.16:0/2302225034 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 12) v1 ==== 84418+0+0 (secure 0 0 0) 0x7f3e440073c0 con 0x7f3e4c069db0 2023-05-22T15:39:28.928 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.919+0000 7f3e42ffd640 1 --2- 172.21.15.16:0/2302225034 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f3e2c065580 0x7f3e2c067a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:28.928 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.919+0000 7f3e42ffd640 1 -- 172.21.15.16:0/2302225034 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 6698+0+0 (secure 0 0 0) 0x7f3e44082770 con 0x7f3e4c069db0 2023-05-22T15:39:28.929 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.919+0000 7f3e49d9b640 1 --2- 172.21.15.16:0/2302225034 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f3e2c065580 0x7f3e2c067a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:28.929 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.919+0000 7f3e49d9b640 1 --2- 172.21.15.16:0/2302225034 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f3e2c065580 0x7f3e2c067a30 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f3e340af000 tx=0x7f3e340a7000 comp rx=0 tx=0).ready entity=mgr.5823 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:39:28.929 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:28.923+0000 7f3e42ffd640 1 -- 172.21.15.16:0/2302225034 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f3e4404ff80 con 0x7f3e4c069db0 2023-05-22T15:39:29.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:29.099+0000 7f3e50d98640 1 -- 172.21.15.16:0/2302225034 --> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/snap_test_dir1/dir9", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f3e4c0c32d0 con 0x7f3e2c065580 2023-05-22T15:39:29.120 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:29.115+0000 7f3e42ffd640 1 -- 172.21.15.16:0/2302225034 <== mgr.5823 v2:172.21.15.16:6832/36921 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+345 (secure 0 0 0) 0x7f3e340ab950 con 0x7f3e2c065580 2023-05-22T15:39:29.120 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:39:29.121 INFO:teuthology.orchestra.run.smithi016.stdout:[{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/dir9", "rel_path": "/snap_test_dir1/dir9", "schedule": "1M", "retention": {}, "start": "2023-05-22T00:00:00", "created": "2023-05-22T15:28:36", "first": "2023-05-22T15:29:01", "last": "2023-05-22T15:39:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2023-05-22T15:39:29.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:29.119+0000 7f3e50d98640 1 -- 172.21.15.16:0/2302225034 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f3e2c065580 msgr2=0x7f3e2c067a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:29.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:29.119+0000 7f3e50d98640 1 --2- 172.21.15.16:0/2302225034 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f3e2c065580 0x7f3e2c067a30 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f3e340af000 tx=0x7f3e340a7000 comp rx=0 tx=0).stop 2023-05-22T15:39:29.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:29.119+0000 7f3e50d98640 1 -- 172.21.15.16:0/2302225034 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3e4c069db0 msgr2=0x7f3e4c14baa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:29.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:29.119+0000 7f3e50d98640 1 --2- 172.21.15.16:0/2302225034 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3e4c069db0 0x7f3e4c14baa0 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f3e4400c480 tx=0x7f3e4400c940 comp rx=0 tx=0).stop 2023-05-22T15:39:29.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:29.119+0000 7f3e50d98640 1 -- 172.21.15.16:0/2302225034 shutdown_connections 2023-05-22T15:39:29.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:29.119+0000 7f3e50d98640 1 --2- 172.21.15.16:0/2302225034 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f3e2c065580 0x7f3e2c067a30 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:29.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:29.119+0000 7f3e50d98640 1 --2- 172.21.15.16:0/2302225034 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3e4c0cdf40 0x7f3e4c0ae240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:29.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:29.119+0000 7f3e50d98640 1 --2- 172.21.15.16:0/2302225034 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3e4c0ccf50 0x7f3e4c148e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:29.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:29.119+0000 7f3e50d98640 1 --2- 172.21.15.16:0/2302225034 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3e4c069db0 0x7f3e4c14baa0 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:29.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:29.119+0000 7f3e50d98640 1 -- 172.21.15.16:0/2302225034 >> 172.21.15.16:0/2302225034 conn(0x7f3e4c0cafb0 msgr2=0x7f3e4c0cb380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:39:29.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:29.119+0000 7f3e50d98640 1 -- 172.21.15.16:0/2302225034 shutdown_connections 2023-05-22T15:39:29.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:29.119+0000 7f3e50d98640 1 -- 172.21.15.16:0/2302225034 wait complete. 2023-05-22T15:39:29.140 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/dir9", "rel_path": "/snap_test_dir1/dir9", "schedule": "1M", "retention": {}, "start": "2023-05-22T00:00:00", "created": "2023-05-22T15:28:36", "first": "2023-05-22T15:29:01", "last": "2023-05-22T15:39:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2023-05-22T15:39:29.140 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'path': '/snap_test_dir1/dir9', 'rel_path': '/snap_test_dir1/dir9', 'schedule': '1M', 'retention': {}, 'start': '2023-05-22T00:00:00', 'created': '2023-05-22T15:28:36', 'first': '2023-05-22T15:29:01', 'last': '2023-05-22T15:39:00', 'last_pruned': None, 'created_count': 11, 'pruned_count': 0, 'active': False} 2023-05-22T15:39:29.141 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove --fs cephfs /snap_test_dir1/dir0 1M 2023-05-22T15:39:29.273 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:29.267+0000 7f2640b19640 1 Processor -- start 2023-05-22T15:39:29.273 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:29.267+0000 7f2640b19640 1 -- start start 2023-05-22T15:39:29.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:29.271+0000 7f2640b19640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f263c0ce530 0x7f263c15c540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:29.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:29.271+0000 7f2640b19640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f263c0cbe80 0x7f263c0cc680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:29.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:29.271+0000 7f2640b19640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f263c0c3930 con 0x7f263c0ce900 2023-05-22T15:39:29.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:29.271+0000 7f2640b19640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f263c158f00 con 0x7f263c0cbe80 2023-05-22T15:39:29.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:29.271+0000 7f2640b19640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f263c159d90 con 0x7f263c0ce530 2023-05-22T15:39:29.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:29.271+0000 7f263ad9d640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f263c0cbe80 0x7f263c0cc680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:29.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:29.271+0000 7f2639d9b640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f263c0ce900 0x7f263c159680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:46132/0 (socket says 172.21.15.16:46132) 2023-05-22T15:39:29.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:29.271+0000 7f2639d9b640 1 -- 172.21.15.16:0/553242644 learned_addr learned my addr 172.21.15.16:0/553242644 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:39:29.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:29.271+0000 7f263ad9d640 1 -- 172.21.15.16:0/553242644 >> v2:172.21.15.62:3300/0 conn(0x7f263c0ce530 msgr2=0x7f263c15c540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:39:29.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:29.271+0000 7f263ad9d640 1 --2- 172.21.15.16:0/553242644 >> v2:172.21.15.62:3300/0 conn(0x7f263c0ce530 0x7f263c15c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:29.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:29.271+0000 7f263ad9d640 1 -- 172.21.15.16:0/553242644 >> v1:172.21.15.16:6789/0 conn(0x7f263c0ce900 legacy=0x7f263c159680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:29.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:29.271+0000 7f263ad9d640 1 -- 172.21.15.16:0/553242644 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f263c15cc40 con 0x7f263c0cbe80 2023-05-22T15:39:29.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:29.271+0000 7f263ad9d640 1 --2- 172.21.15.16:0/553242644 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f263c0cbe80 0x7f263c0cc680 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f263c0cf160 tx=0x7f263401eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de3618c9ea55fcd0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:39:29.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:29.271+0000 7f263959a640 1 -- 172.21.15.16:0/553242644 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26340a7730 con 0x7f263c0cbe80 2023-05-22T15:39:29.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:29.271+0000 7f263959a640 1 -- 172.21.15.16:0/553242644 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26340a7890 con 0x7f263c0cbe80 2023-05-22T15:39:29.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:29.271+0000 7f263959a640 1 -- 172.21.15.16:0/553242644 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26340a7c20 con 0x7f263c0cbe80 2023-05-22T15:39:29.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:29.271+0000 7f2640b19640 1 -- 172.21.15.16:0/553242644 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f263c0cbe80 msgr2=0x7f263c0cc680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:29.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:29.271+0000 7f2640b19640 1 --2- 172.21.15.16:0/553242644 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f263c0cbe80 0x7f263c0cc680 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f263c0cf160 tx=0x7f263401eb20 comp rx=0 tx=0).stop 2023-05-22T15:39:29.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:29.275+0000 7f2640b19640 1 -- 172.21.15.16:0/553242644 shutdown_connections 2023-05-22T15:39:29.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:29.275+0000 7f2640b19640 1 --2- 172.21.15.16:0/553242644 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f263c0cbe80 0x7f263c0cc680 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:29.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:29.275+0000 7f2640b19640 1 --2- 172.21.15.16:0/553242644 >> v2:172.21.15.62:3300/0 conn(0x7f263c0ce530 0x7f263c15c540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:29.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:29.275+0000 7f2640b19640 1 -- 172.21.15.16:0/553242644 >> 172.21.15.16:0/553242644 conn(0x7f263c069720 msgr2=0x7f263c069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:39:29.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:29.275+0000 7f2640b19640 1 -- 172.21.15.16:0/553242644 shutdown_connections 2023-05-22T15:39:29.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:29.275+0000 7f2640b19640 1 -- 172.21.15.16:0/553242644 wait complete. 2023-05-22T15:39:29.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:29.275+0000 7f2640b19640 1 Processor -- start 2023-05-22T15:39:29.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:29.275+0000 7f2640b19640 1 -- start start 2023-05-22T15:39:29.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:29.275+0000 7f2640b19640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f263c0cbe80 0x7f263c0ad0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:29.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:29.275+0000 7f2640b19640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f263c0ce530 0x7f263c14a730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:29.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:29.275+0000 7f2640b19640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f263c0ce900 0x7f263c0ab7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:29.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:29.275+0000 7f2640b19640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f263c0abd00 con 0x7f263c0ce530 2023-05-22T15:39:29.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:29.275+0000 7f2640b19640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f263c0b0fb0 con 0x7f263c0ce900 2023-05-22T15:39:29.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:29.275+0000 7f2640b19640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f263c0abf30 con 0x7f263c0cbe80 2023-05-22T15:39:29.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:29.275+0000 7f2639d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f263c0ce530 0x7f263c14a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:29.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:29.275+0000 7f2639d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f263c0ce530 0x7f263c14a730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53348/0 (socket says 172.21.15.16:53348) 2023-05-22T15:39:29.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:29.275+0000 7f2639d9b640 1 -- 172.21.15.16:0/1584886178 learned_addr learned my addr 172.21.15.16:0/1584886178 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:39:29.282 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:29.275+0000 7f263ad9d640 1 --2- 172.21.15.16:0/1584886178 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f263c0ce900 0x7f263c0ab7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:29.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:29.275+0000 7f263a59c640 1 --2- 172.21.15.16:0/1584886178 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f263c0cbe80 0x7f263c0ad0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:29.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:29.275+0000 7f2639d9b640 1 -- 172.21.15.16:0/1584886178 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f263c0cbe80 msgr2=0x7f263c0ad0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:29.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:29.275+0000 7f2639d9b640 1 --2- 172.21.15.16:0/1584886178 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f263c0cbe80 0x7f263c0ad0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:29.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:29.275+0000 7f2639d9b640 1 -- 172.21.15.16:0/1584886178 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f263c0ce900 msgr2=0x7f263c0ab7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:29.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:29.275+0000 7f2639d9b640 1 --2- 172.21.15.16:0/1584886178 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f263c0ce900 0x7f263c0ab7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:29.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:29.275+0000 7f2639d9b640 1 -- 172.21.15.16:0/1584886178 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2634009a70 con 0x7f263c0ce530 2023-05-22T15:39:29.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:29.275+0000 7f2639d9b640 1 --2- 172.21.15.16:0/1584886178 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f263c0ce530 0x7f263c14a730 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f263000c400 tx=0x7f263000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:39:29.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:29.275+0000 7f262affd640 1 -- 172.21.15.16:0/1584886178 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2630005cf0 con 0x7f263c0ce530 2023-05-22T15:39:29.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:29.275+0000 7f262affd640 1 -- 172.21.15.16:0/1584886178 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2630005e50 con 0x7f263c0ce530 2023-05-22T15:39:29.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:29.275+0000 7f262affd640 1 -- 172.21.15.16:0/1584886178 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26300072d0 con 0x7f263c0ce530 2023-05-22T15:39:29.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:29.275+0000 7f2640b19640 1 -- 172.21.15.16:0/1584886178 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f263c14f2f0 con 0x7f263c0ce530 2023-05-22T15:39:29.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:29.275+0000 7f2640b19640 1 -- 172.21.15.16:0/1584886178 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f263c159160 con 0x7f263c0ce530 2023-05-22T15:39:29.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:29.275+0000 7f2640b19640 1 -- 172.21.15.16:0/1584886178 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f263c14f480 con 0x7f263c0ce530 2023-05-22T15:39:29.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:29.279+0000 7f262affd640 1 -- 172.21.15.16:0/1584886178 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 12) v1 ==== 84418+0+0 (secure 0 0 0) 0x7f2630007430 con 0x7f263c0ce530 2023-05-22T15:39:29.285 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:29.279+0000 7f262affd640 1 --2- 172.21.15.16:0/1584886178 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f261c065580 0x7f261c067a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:29.285 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:29.279+0000 7f262affd640 1 -- 172.21.15.16:0/1584886178 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 6698+0+0 (secure 0 0 0) 0x7f2630082190 con 0x7f263c0ce530 2023-05-22T15:39:29.285 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:29.279+0000 7f263a59c640 1 --2- 172.21.15.16:0/1584886178 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f261c065580 0x7f261c067a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:29.286 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:29.279+0000 7f263a59c640 1 --2- 172.21.15.16:0/1584886178 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f261c065580 0x7f261c067a30 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f262c00a830 tx=0x7f262c008040 comp rx=0 tx=0).ready entity=mgr.5823 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:39:29.287 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:29.283+0000 7f262affd640 1 -- 172.21.15.16:0/1584886178 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f263004a620 con 0x7f263c0ce530 2023-05-22T15:39:29.460 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:29.455+0000 7f2640b19640 1 -- 172.21.15.16:0/1584886178 --> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "fs": "cephfs", "path": "/snap_test_dir1/dir0", "repeat": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f263c0c3930 con 0x7f261c065580 2023-05-22T15:39:29.637 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:29.631+0000 7f262affd640 1 -- 172.21.15.16:0/1584886178 <== mgr.5823 v2:172.21.15.16:6832/36921 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+47 (secure 0 0 0) 0x7f263c0c3930 con 0x7f261c065580 2023-05-22T15:39:29.637 INFO:teuthology.orchestra.run.smithi016.stdout:Schedule removed for path /snap_test_dir1/dir0 2023-05-22T15:39:29.639 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:29.635+0000 7f2640b19640 1 -- 172.21.15.16:0/1584886178 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f261c065580 msgr2=0x7f261c067a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:29.640 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:29.635+0000 7f2640b19640 1 --2- 172.21.15.16:0/1584886178 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f261c065580 0x7f261c067a30 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f262c00a830 tx=0x7f262c008040 comp rx=0 tx=0).stop 2023-05-22T15:39:29.640 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:29.635+0000 7f2640b19640 1 -- 172.21.15.16:0/1584886178 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f263c0ce530 msgr2=0x7f263c14a730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:29.640 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:29.635+0000 7f2640b19640 1 --2- 172.21.15.16:0/1584886178 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f263c0ce530 0x7f263c14a730 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f263000c400 tx=0x7f263000c8c0 comp rx=0 tx=0).stop 2023-05-22T15:39:29.640 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:29.635+0000 7f2640b19640 1 -- 172.21.15.16:0/1584886178 shutdown_connections 2023-05-22T15:39:29.640 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:29.635+0000 7f2640b19640 1 --2- 172.21.15.16:0/1584886178 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f263c0ce900 0x7f263c0ab7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:29.640 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:29.635+0000 7f2640b19640 1 --2- 172.21.15.16:0/1584886178 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f263c0ce530 0x7f263c14a730 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:29.641 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:29.635+0000 7f2640b19640 1 --2- 172.21.15.16:0/1584886178 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f261c065580 0x7f261c067a30 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:29.641 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:29.635+0000 7f2640b19640 1 --2- 172.21.15.16:0/1584886178 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f263c0cbe80 0x7f263c0ad0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:29.641 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:29.635+0000 7f2640b19640 1 -- 172.21.15.16:0/1584886178 >> 172.21.15.16:0/1584886178 conn(0x7f263c069720 msgr2=0x7f263c0c9d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:39:29.641 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:29.635+0000 7f2640b19640 1 -- 172.21.15.16:0/1584886178 shutdown_connections 2023-05-22T15:39:29.641 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:29.635+0000 7f2640b19640 1 -- 172.21.15.16:0/1584886178 wait complete. 2023-05-22T15:39:29.658 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/dir0 2023-05-22T15:39:29.659 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:39:29.659 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir0/.snap) 2023-05-22T15:39:29.671 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_29_01_UTC 2023-05-22T15:39:29.671 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_30_00_UTC 2023-05-22T15:39:29.671 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_31_01_UTC 2023-05-22T15:39:29.672 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_32_00_UTC 2023-05-22T15:39:29.672 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_33_00_UTC 2023-05-22T15:39:29.672 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_34_00_UTC 2023-05-22T15:39:29.672 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_35_00_UTC 2023-05-22T15:39:29.672 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_36_00_UTC 2023-05-22T15:39:29.673 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_37_01_UTC 2023-05-22T15:39:29.673 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_38_01_UTC 2023-05-22T15:39:29.673 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_39_01_UTC 2023-05-22T15:39:29.673 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2023-05-22-15_29_01_UTC 2023-05-22T15:39:29.674 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:39:29.674 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2023-05-22-15_29_01_UTC) 2023-05-22T15:39:29.737 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2023-05-22-15_30_00_UTC 2023-05-22T15:39:29.737 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:39:29.738 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2023-05-22-15_30_00_UTC) 2023-05-22T15:39:29.798 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2023-05-22-15_31_01_UTC 2023-05-22T15:39:29.798 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:39:29.799 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2023-05-22-15_31_01_UTC) 2023-05-22T15:39:29.863 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2023-05-22-15_32_00_UTC 2023-05-22T15:39:29.864 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:39:29.864 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2023-05-22-15_32_00_UTC) 2023-05-22T15:39:29.930 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2023-05-22-15_33_00_UTC 2023-05-22T15:39:29.931 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:39:29.931 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2023-05-22-15_33_00_UTC) 2023-05-22T15:39:29.992 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2023-05-22-15_34_00_UTC 2023-05-22T15:39:29.993 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:39:29.993 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2023-05-22-15_34_00_UTC) 2023-05-22T15:39:30.058 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2023-05-22-15_35_00_UTC 2023-05-22T15:39:30.059 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:39:30.059 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2023-05-22-15_35_00_UTC) 2023-05-22T15:39:30.127 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2023-05-22-15_36_00_UTC 2023-05-22T15:39:30.128 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:39:30.129 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2023-05-22-15_36_00_UTC) 2023-05-22T15:39:30.194 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2023-05-22-15_37_01_UTC 2023-05-22T15:39:30.194 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:39:30.195 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2023-05-22-15_37_01_UTC) 2023-05-22T15:39:30.265 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2023-05-22-15_38_01_UTC 2023-05-22T15:39:30.266 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:39:30.266 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2023-05-22-15_38_01_UTC) 2023-05-22T15:39:30.327 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2023-05-22-15_39_01_UTC 2023-05-22T15:39:30.327 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:39:30.328 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2023-05-22-15_39_01_UTC) 2023-05-22T15:39:30.392 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:39:30.393 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0) 2023-05-22T15:39:30.446 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove --fs cephfs /snap_test_dir1/dir1 1M 2023-05-22T15:39:30.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:30.623+0000 7f562938d640 1 Processor -- start 2023-05-22T15:39:30.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:30.623+0000 7f562938d640 1 -- start start 2023-05-22T15:39:30.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:30.627+0000 7f562938d640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f5624158950 0x7f56240cb900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:30.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:30.627+0000 7f562938d640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f56240cdd10 0x7f56240cbe40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:30.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:30.627+0000 7f562938d640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5624070bb0 con 0x7f56240cb440 2023-05-22T15:39:30.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:30.627+0000 7f562938d640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5624070e90 con 0x7f5624158950 2023-05-22T15:39:30.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:30.627+0000 7f562938d640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5624071170 con 0x7f56240cdd10 2023-05-22T15:39:30.632 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:30.627+0000 7f5622ffd640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f5624158950 0x7f56240cb900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:30.632 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:30.627+0000 7f5622ffd640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f5624158950 0x7f56240cb900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53350/0 (socket says 172.21.15.16:53350) 2023-05-22T15:39:30.632 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:30.627+0000 7f5622ffd640 1 -- 172.21.15.16:0/2891225906 learned_addr learned my addr 172.21.15.16:0/2891225906 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:39:30.632 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:30.627+0000 7f56227fc640 1 --1- 172.21.15.16:0/2891225906 >> v1:172.21.15.16:6789/0 conn(0x7f56240cb440 0x7f562415c540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:46144/0 (socket says 172.21.15.16:46144) 2023-05-22T15:39:30.633 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:30.627+0000 7f56237fe640 1 --2- 172.21.15.16:0/2891225906 >> v2:172.21.15.62:3300/0 conn(0x7f56240cdd10 0x7f56240cbe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:30.633 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:30.627+0000 7f5622ffd640 1 -- 172.21.15.16:0/2891225906 >> v2:172.21.15.62:3300/0 conn(0x7f56240cdd10 msgr2=0x7f56240cbe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:39:30.633 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:30.627+0000 7f5622ffd640 1 --2- 172.21.15.16:0/2891225906 >> v2:172.21.15.62:3300/0 conn(0x7f56240cdd10 0x7f56240cbe40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:30.633 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:30.627+0000 7f5622ffd640 1 -- 172.21.15.16:0/2891225906 >> v1:172.21.15.16:6789/0 conn(0x7f56240cb440 legacy=0x7f562415c540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:30.633 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:30.627+0000 7f5622ffd640 1 -- 172.21.15.16:0/2891225906 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5624071450 con 0x7f5624158950 2023-05-22T15:39:30.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:30.631+0000 7f5622ffd640 1 --2- 172.21.15.16:0/2891225906 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5624158950 0x7f56240cb900 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f56240b1840 tx=0x7f561c01eba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43891e60b02f7544 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:39:30.635 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:30.631+0000 7f5621ffb640 1 -- 172.21.15.16:0/2891225906 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f561c00b2d0 con 0x7f5624158950 2023-05-22T15:39:30.635 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:30.631+0000 7f5621ffb640 1 -- 172.21.15.16:0/2891225906 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f561c00b430 con 0x7f5624158950 2023-05-22T15:39:30.635 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:30.631+0000 7f5621ffb640 1 -- 172.21.15.16:0/2891225906 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f561c0a7ba0 con 0x7f5624158950 2023-05-22T15:39:30.636 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:30.631+0000 7f562938d640 1 -- 172.21.15.16:0/2891225906 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5624158950 msgr2=0x7f56240cb900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:30.636 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:30.631+0000 7f562938d640 1 --2- 172.21.15.16:0/2891225906 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5624158950 0x7f56240cb900 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f56240b1840 tx=0x7f561c01eba0 comp rx=0 tx=0).stop 2023-05-22T15:39:30.636 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:30.631+0000 7f562938d640 1 -- 172.21.15.16:0/2891225906 shutdown_connections 2023-05-22T15:39:30.636 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:30.631+0000 7f562938d640 1 --2- 172.21.15.16:0/2891225906 >> v2:172.21.15.62:3300/0 conn(0x7f56240cdd10 0x7f56240cbe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:30.637 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:30.631+0000 7f562938d640 1 --2- 172.21.15.16:0/2891225906 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5624158950 0x7f56240cb900 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:30.637 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:30.631+0000 7f562938d640 1 -- 172.21.15.16:0/2891225906 >> 172.21.15.16:0/2891225906 conn(0x7f56240b0cc0 msgr2=0x7f56240b10c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:39:30.637 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:30.631+0000 7f562938d640 1 -- 172.21.15.16:0/2891225906 shutdown_connections 2023-05-22T15:39:30.637 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:30.631+0000 7f562938d640 1 -- 172.21.15.16:0/2891225906 wait complete. 2023-05-22T15:39:30.638 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:30.631+0000 7f562938d640 1 Processor -- start 2023-05-22T15:39:30.638 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:30.631+0000 7f562938d640 1 -- start start 2023-05-22T15:39:30.638 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:30.631+0000 7f562938d640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f56240cb440 0x7f5624156340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:30.638 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:30.631+0000 7f562938d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f56240cdd10 0x7f56241f79b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:30.639 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:30.631+0000 7f562938d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5624158950 0x7f56241f3cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:30.639 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:30.631+0000 7f562938d640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56240c8f30 con 0x7f5624158950 2023-05-22T15:39:30.639 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:30.631+0000 7f562938d640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56241f88c0 con 0x7f56240cdd10 2023-05-22T15:39:30.639 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:30.631+0000 7f562938d640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56241f8020 con 0x7f56240cb440 2023-05-22T15:39:30.639 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:30.631+0000 7f56237fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5624158950 0x7f56241f3cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:30.639 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:30.631+0000 7f56237fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5624158950 0x7f56241f3cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53358/0 (socket says 172.21.15.16:53358) 2023-05-22T15:39:30.639 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:30.631+0000 7f56237fe640 1 -- 172.21.15.16:0/4144854562 learned_addr learned my addr 172.21.15.16:0/4144854562 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:39:30.640 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:30.635+0000 7f56227fc640 1 --2- 172.21.15.16:0/4144854562 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f56240cdd10 0x7f56241f79b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:30.640 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:30.635+0000 7f5622ffd640 1 --2- 172.21.15.16:0/4144854562 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f56240cb440 0x7f5624156340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:30.640 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:30.635+0000 7f56237fe640 1 -- 172.21.15.16:0/4144854562 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f56240cb440 msgr2=0x7f5624156340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:30.640 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:30.635+0000 7f56237fe640 1 --2- 172.21.15.16:0/4144854562 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f56240cb440 0x7f5624156340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:30.640 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:30.635+0000 7f56237fe640 1 -- 172.21.15.16:0/4144854562 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f56240cdd10 msgr2=0x7f56241f79b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:30.640 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:30.635+0000 7f56237fe640 1 --2- 172.21.15.16:0/4144854562 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f56240cdd10 0x7f56241f79b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:30.640 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:30.635+0000 7f56237fe640 1 -- 172.21.15.16:0/4144854562 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f561c00bd40 con 0x7f5624158950 2023-05-22T15:39:30.641 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:30.635+0000 7f56237fe640 1 --2- 172.21.15.16:0/4144854562 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5624158950 0x7f56241f3cd0 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f561400d830 tx=0x7f561400dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:39:30.641 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:30.635+0000 7f561b7fe640 1 -- 172.21.15.16:0/4144854562 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f561400b6c0 con 0x7f5624158950 2023-05-22T15:39:30.642 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:30.635+0000 7f561b7fe640 1 -- 172.21.15.16:0/4144854562 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f561400b820 con 0x7f5624158950 2023-05-22T15:39:30.642 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:30.635+0000 7f562938d640 1 -- 172.21.15.16:0/4144854562 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5624070980 con 0x7f5624158950 2023-05-22T15:39:30.642 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:30.635+0000 7f561b7fe640 1 -- 172.21.15.16:0/4144854562 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f561400bbd0 con 0x7f5624158950 2023-05-22T15:39:30.642 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:30.635+0000 7f562938d640 1 -- 172.21.15.16:0/4144854562 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56240707f0 con 0x7f5624158950 2023-05-22T15:39:30.642 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:30.635+0000 7f562938d640 1 -- 172.21.15.16:0/4144854562 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5624069720 con 0x7f5624158950 2023-05-22T15:39:30.642 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:30.635+0000 7f561b7fe640 1 -- 172.21.15.16:0/4144854562 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 12) v1 ==== 84418+0+0 (secure 0 0 0) 0x7f561401e460 con 0x7f5624158950 2023-05-22T15:39:30.643 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:30.635+0000 7f561b7fe640 1 --2- 172.21.15.16:0/4144854562 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f5604065580 0x7f5604067a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:30.643 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:30.635+0000 7f561b7fe640 1 -- 172.21.15.16:0/4144854562 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 6698+0+0 (secure 0 0 0) 0x7f5614083600 con 0x7f5624158950 2023-05-22T15:39:30.643 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:30.635+0000 7f5622ffd640 1 --2- 172.21.15.16:0/4144854562 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f5604065580 0x7f5604067a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:30.643 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:30.635+0000 7f5622ffd640 1 --2- 172.21.15.16:0/4144854562 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f5604065580 0x7f5604067a30 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f561c0b0000 tx=0x7f561c0a9070 comp rx=0 tx=0).ready entity=mgr.5823 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:39:30.645 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:30.639+0000 7f561b7fe640 1 -- 172.21.15.16:0/4144854562 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f5614050d10 con 0x7f5624158950 2023-05-22T15:39:30.825 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:30.819+0000 7f562938d640 1 -- 172.21.15.16:0/4144854562 --> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "fs": "cephfs", "path": "/snap_test_dir1/dir1", "repeat": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f56240b18a0 con 0x7f5604065580 2023-05-22T15:39:31.008 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:31.003+0000 7f561b7fe640 1 -- 172.21.15.16:0/4144854562 <== mgr.5823 v2:172.21.15.16:6832/36921 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+47 (secure 0 0 0) 0x7f56240b18a0 con 0x7f5604065580 2023-05-22T15:39:31.009 INFO:teuthology.orchestra.run.smithi016.stdout:Schedule removed for path /snap_test_dir1/dir1 2023-05-22T15:39:31.011 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:31.007+0000 7f562938d640 1 -- 172.21.15.16:0/4144854562 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f5604065580 msgr2=0x7f5604067a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:31.011 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:31.007+0000 7f562938d640 1 --2- 172.21.15.16:0/4144854562 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f5604065580 0x7f5604067a30 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f561c0b0000 tx=0x7f561c0a9070 comp rx=0 tx=0).stop 2023-05-22T15:39:31.011 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:31.007+0000 7f562938d640 1 -- 172.21.15.16:0/4144854562 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5624158950 msgr2=0x7f56241f3cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:31.012 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:31.007+0000 7f562938d640 1 --2- 172.21.15.16:0/4144854562 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5624158950 0x7f56241f3cd0 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f561400d830 tx=0x7f561400dcf0 comp rx=0 tx=0).stop 2023-05-22T15:39:31.012 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:31.007+0000 7f562938d640 1 -- 172.21.15.16:0/4144854562 shutdown_connections 2023-05-22T15:39:31.012 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:31.007+0000 7f562938d640 1 --2- 172.21.15.16:0/4144854562 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5624158950 0x7f56241f3cd0 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:31.012 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:31.007+0000 7f562938d640 1 --2- 172.21.15.16:0/4144854562 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f56240cdd10 0x7f56241f79b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:31.012 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:31.007+0000 7f562938d640 1 --2- 172.21.15.16:0/4144854562 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f5604065580 0x7f5604067a30 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:31.012 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:31.007+0000 7f562938d640 1 --2- 172.21.15.16:0/4144854562 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f56240cb440 0x7f5624156340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:31.013 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:31.007+0000 7f562938d640 1 -- 172.21.15.16:0/4144854562 >> 172.21.15.16:0/4144854562 conn(0x7f56240b0cc0 msgr2=0x7f56240aec30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:39:31.013 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:31.007+0000 7f562938d640 1 -- 172.21.15.16:0/4144854562 shutdown_connections 2023-05-22T15:39:31.013 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:31.007+0000 7f562938d640 1 -- 172.21.15.16:0/4144854562 wait complete. 2023-05-22T15:39:31.031 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/dir1 2023-05-22T15:39:31.031 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:39:31.032 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir1/.snap) 2023-05-22T15:39:31.050 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_29_00_UTC 2023-05-22T15:39:31.051 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_30_00_UTC 2023-05-22T15:39:31.051 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_31_00_UTC 2023-05-22T15:39:31.051 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_32_00_UTC 2023-05-22T15:39:31.051 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_33_01_UTC 2023-05-22T15:39:31.051 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_34_00_UTC 2023-05-22T15:39:31.051 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_35_00_UTC 2023-05-22T15:39:31.051 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_36_00_UTC 2023-05-22T15:39:31.052 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_37_00_UTC 2023-05-22T15:39:31.052 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_38_00_UTC 2023-05-22T15:39:31.052 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_39_00_UTC 2023-05-22T15:39:31.053 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2023-05-22-15_29_00_UTC 2023-05-22T15:39:31.054 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:39:31.054 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2023-05-22-15_29_00_UTC) 2023-05-22T15:39:31.119 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2023-05-22-15_30_00_UTC 2023-05-22T15:39:31.120 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:39:31.120 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2023-05-22-15_30_00_UTC) 2023-05-22T15:39:31.182 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2023-05-22-15_31_00_UTC 2023-05-22T15:39:31.183 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:39:31.183 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2023-05-22-15_31_00_UTC) 2023-05-22T15:39:31.252 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2023-05-22-15_32_00_UTC 2023-05-22T15:39:31.253 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:39:31.253 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2023-05-22-15_32_00_UTC) 2023-05-22T15:39:31.316 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2023-05-22-15_33_01_UTC 2023-05-22T15:39:31.316 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:39:31.317 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2023-05-22-15_33_01_UTC) 2023-05-22T15:39:31.379 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2023-05-22-15_34_00_UTC 2023-05-22T15:39:31.380 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:39:31.381 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2023-05-22-15_34_00_UTC) 2023-05-22T15:39:31.443 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2023-05-22-15_35_00_UTC 2023-05-22T15:39:31.444 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:39:31.444 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2023-05-22-15_35_00_UTC) 2023-05-22T15:39:31.509 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2023-05-22-15_36_00_UTC 2023-05-22T15:39:31.509 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:39:31.509 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2023-05-22-15_36_00_UTC) 2023-05-22T15:39:31.570 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2023-05-22-15_37_00_UTC 2023-05-22T15:39:31.571 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:39:31.571 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2023-05-22-15_37_00_UTC) 2023-05-22T15:39:31.636 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2023-05-22-15_38_00_UTC 2023-05-22T15:39:31.637 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:39:31.637 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2023-05-22-15_38_00_UTC) 2023-05-22T15:39:31.699 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2023-05-22-15_39_00_UTC 2023-05-22T15:39:31.699 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:39:31.700 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2023-05-22-15_39_00_UTC) 2023-05-22T15:39:31.765 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:39:31.766 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1) 2023-05-22T15:39:31.817 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove --fs cephfs /snap_test_dir1/dir2 1M 2023-05-22T15:39:31.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:31.987+0000 7f2935a1d640 1 Processor -- start 2023-05-22T15:39:31.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:31.991+0000 7f2935a1d640 1 -- start start 2023-05-22T15:39:31.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:31.991+0000 7f2935a1d640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f29301603d0 0x7f29300cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:31.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:31.991+0000 7f2935a1d640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f293015f980 con 0x7f293015be20 2023-05-22T15:39:31.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:31.991+0000 7f2935a1d640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2930160820 con 0x7f29300cfad0 2023-05-22T15:39:31.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:31.991+0000 7f2935a1d640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29300b2490 con 0x7f29301603d0 2023-05-22T15:39:31.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:31.991+0000 7f292effd640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f293015be20 0x7f29300b1d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:46154/0 (socket says 172.21.15.16:46154) 2023-05-22T15:39:31.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:31.991+0000 7f292effd640 1 -- 172.21.15.16:0/3612791273 learned_addr learned my addr 172.21.15.16:0/3612791273 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:39:31.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:31.991+0000 7f292e7fc640 1 --2- 172.21.15.16:0/3612791273 >> v2:172.21.15.62:3300/0 conn(0x7f29301603d0 0x7f29300cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:31.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:31.991+0000 7f292dffb640 1 -- 172.21.15.16:0/3612791273 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4203638988 0 0) 0x7f293015f980 con 0x7f293015be20 2023-05-22T15:39:31.998 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:31.991+0000 7f292dffb640 1 -- 172.21.15.16:0/3612791273 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2918003530 con 0x7f293015be20 2023-05-22T15:39:31.998 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:31.991+0000 7f292dffb640 1 -- 172.21.15.16:0/3612791273 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3208437828 0 0) 0x7f2918003530 con 0x7f293015be20 2023-05-22T15:39:31.998 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:31.991+0000 7f292dffb640 1 -- 172.21.15.16:0/3612791273 >> v2:172.21.15.62:3300/0 conn(0x7f29301603d0 msgr2=0x7f29300cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:39:31.998 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:31.991+0000 7f292dffb640 1 --2- 172.21.15.16:0/3612791273 >> v2:172.21.15.62:3300/0 conn(0x7f29301603d0 0x7f29300cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:31.998 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:31.991+0000 7f292dffb640 1 -- 172.21.15.16:0/3612791273 >> v1:172.21.15.62:6789/0 conn(0x7f29300cfad0 legacy=0x7f29300ca950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:31.998 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:31.991+0000 7f292dffb640 1 -- 172.21.15.16:0/3612791273 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29300b2760 con 0x7f293015be20 2023-05-22T15:39:31.998 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:31.991+0000 7f292dffb640 1 -- 172.21.15.16:0/3612791273 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7f2920001680 con 0x7f293015be20 2023-05-22T15:39:31.998 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:31.991+0000 7f292dffb640 1 -- 172.21.15.16:0/3612791273 >> v1:172.21.15.16:6789/0 conn(0x7f293015be20 legacy=0x7f29300b1d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:31.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:31.991+0000 7f292dffb640 1 --2- 172.21.15.16:0/3612791273 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2918003980 0x7f2918003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:31.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:31.991+0000 7f292dffb640 1 --2- 172.21.15.16:0/3612791273 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f29180047d0 0x7f2918004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:31.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:31.991+0000 7f292dffb640 1 --2- 172.21.15.16:0/3612791273 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2918005180 0x7f291800b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:31.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:31.991+0000 7f292dffb640 1 -- 172.21.15.16:0/3612791273 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29300b2490 con 0x7f29180047d0 2023-05-22T15:39:31.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:31.991+0000 7f292dffb640 1 -- 172.21.15.16:0/3612791273 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f291800c090 con 0x7f2918003980 2023-05-22T15:39:31.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:31.991+0000 7f292dffb640 1 -- 172.21.15.16:0/3612791273 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f291800c250 con 0x7f2918005180 2023-05-22T15:39:31.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:31.991+0000 7f292e7fc640 1 --2- 172.21.15.16:0/3612791273 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f29180047d0 0x7f2918004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:32.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:31.991+0000 7f292effd640 1 --2- 172.21.15.16:0/3612791273 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2918003980 0x7f2918003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:32.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:31.991+0000 7f292e7fc640 1 -- 172.21.15.16:0/3612791273 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2918005180 msgr2=0x7f291800b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:39:32.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:31.991+0000 7f292e7fc640 1 --2- 172.21.15.16:0/3612791273 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2918005180 0x7f291800b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:32.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:31.991+0000 7f292e7fc640 1 -- 172.21.15.16:0/3612791273 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2918003980 msgr2=0x7f2918003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:39:32.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:31.991+0000 7f292e7fc640 1 --2- 172.21.15.16:0/3612791273 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2918003980 0x7f2918003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:32.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:31.991+0000 7f292e7fc640 1 -- 172.21.15.16:0/3612791273 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2918003530 con 0x7f29180047d0 2023-05-22T15:39:32.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:31.991+0000 7f292f7fe640 1 -- 172.21.15.16:0/3612791273 reap_dead start 2023-05-22T15:39:32.001 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:31.995+0000 7f292e7fc640 1 --2- 172.21.15.16:0/3612791273 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f29180047d0 0x7f2918004c40 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f2924009ae0 tx=0x7f292401ca60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9e10ea3658fb836 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:39:32.001 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:31.995+0000 7f292dffb640 1 -- 172.21.15.16:0/3612791273 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29240a5540 con 0x7f29180047d0 2023-05-22T15:39:32.002 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:31.995+0000 7f292dffb640 1 -- 172.21.15.16:0/3612791273 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29240a56a0 con 0x7f29180047d0 2023-05-22T15:39:32.002 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:31.995+0000 7f292dffb640 1 -- 172.21.15.16:0/3612791273 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29240a5a50 con 0x7f29180047d0 2023-05-22T15:39:32.002 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:31.995+0000 7f2935a1d640 1 -- 172.21.15.16:0/3612791273 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f29180047d0 msgr2=0x7f2918004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:32.002 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:31.995+0000 7f2935a1d640 1 --2- 172.21.15.16:0/3612791273 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f29180047d0 0x7f2918004c40 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f2924009ae0 tx=0x7f292401ca60 comp rx=0 tx=0).stop 2023-05-22T15:39:32.002 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:31.995+0000 7f2935a1d640 1 -- 172.21.15.16:0/3612791273 shutdown_connections 2023-05-22T15:39:32.002 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:31.995+0000 7f2935a1d640 1 --2- 172.21.15.16:0/3612791273 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f29180047d0 0x7f2918004c40 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:32.002 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:31.995+0000 7f2935a1d640 1 -- 172.21.15.16:0/3612791273 >> 172.21.15.16:0/3612791273 conn(0x7f293006d9d0 msgr2=0x7f293006ddd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:39:32.003 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:31.995+0000 7f2935a1d640 1 -- 172.21.15.16:0/3612791273 shutdown_connections 2023-05-22T15:39:32.003 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:31.995+0000 7f2935a1d640 1 -- 172.21.15.16:0/3612791273 wait complete. 2023-05-22T15:39:32.004 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:31.995+0000 7f2935a1d640 1 Processor -- start 2023-05-22T15:39:32.004 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:31.995+0000 7f2935a1d640 1 -- start start 2023-05-22T15:39:32.005 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:31.999+0000 7f2935a1d640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f29180047d0 0x7f29300c5f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:32.005 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:31.999+0000 7f2935a1d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f293014c7c0 0x7f2930153c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:32.005 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:31.999+0000 7f2935a1d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f293015c380 0x7f293015a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:32.005 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:31.999+0000 7f2935a1d640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2930154160 con 0x7f293014c7c0 2023-05-22T15:39:32.005 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:31.999+0000 7f2935a1d640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2930154400 con 0x7f293015c380 2023-05-22T15:39:32.006 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:31.999+0000 7f2935a1d640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29301546e0 con 0x7f29180047d0 2023-05-22T15:39:32.006 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:31.999+0000 7f292e7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f293014c7c0 0x7f2930153c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:32.006 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:31.999+0000 7f292e7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f293014c7c0 0x7f2930153c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53374/0 (socket says 172.21.15.16:53374) 2023-05-22T15:39:32.006 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:31.999+0000 7f292e7fc640 1 -- 172.21.15.16:0/2845281225 learned_addr learned my addr 172.21.15.16:0/2845281225 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:39:32.006 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:31.999+0000 7f292effd640 1 --2- 172.21.15.16:0/2845281225 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f29180047d0 0x7f29300c5f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:32.006 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:31.999+0000 7f292f7fe640 1 --2- 172.21.15.16:0/2845281225 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f293015c380 0x7f293015a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:32.006 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:31.999+0000 7f292e7fc640 1 -- 172.21.15.16:0/2845281225 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f29180047d0 msgr2=0x7f29300c5f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:32.007 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:31.999+0000 7f292e7fc640 1 --2- 172.21.15.16:0/2845281225 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f29180047d0 0x7f29300c5f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:32.007 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:31.999+0000 7f292e7fc640 1 -- 172.21.15.16:0/2845281225 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f293015c380 msgr2=0x7f293015a8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:32.007 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:31.999+0000 7f292e7fc640 1 --2- 172.21.15.16:0/2845281225 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f293015c380 0x7f293015a8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:32.007 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:31.999+0000 7f292e7fc640 1 -- 172.21.15.16:0/2845281225 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f293015c1f0 con 0x7f293014c7c0 2023-05-22T15:39:32.007 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:31.999+0000 7f292f7fe640 1 --2- 172.21.15.16:0/2845281225 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f293015c380 0x7f293015a8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:39:32.007 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:31.999+0000 7f292e7fc640 1 --2- 172.21.15.16:0/2845281225 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f293014c7c0 0x7f2930153c20 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f2924009ae0 tx=0x7f2918005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:39:32.007 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:31.999+0000 7f2917fff640 1 -- 172.21.15.16:0/2845281225 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29240a5e10 con 0x7f293014c7c0 2023-05-22T15:39:32.007 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:31.999+0000 7f2917fff640 1 -- 172.21.15.16:0/2845281225 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2924004020 con 0x7f293014c7c0 2023-05-22T15:39:32.008 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:31.999+0000 7f2917fff640 1 -- 172.21.15.16:0/2845281225 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2924004390 con 0x7f293014c7c0 2023-05-22T15:39:32.008 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:31.999+0000 7f2935a1d640 1 -- 172.21.15.16:0/2845281225 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f293015feb0 con 0x7f293014c7c0 2023-05-22T15:39:32.008 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:31.999+0000 7f2935a1d640 1 -- 172.21.15.16:0/2845281225 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29300b2190 con 0x7f293014c7c0 2023-05-22T15:39:32.008 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:31.999+0000 7f2935a1d640 1 -- 172.21.15.16:0/2845281225 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f293004dbe0 con 0x7f293014c7c0 2023-05-22T15:39:32.009 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:31.999+0000 7f2917fff640 1 -- 172.21.15.16:0/2845281225 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 12) v1 ==== 84418+0+0 (secure 0 0 0) 0x7f29240b4040 con 0x7f293014c7c0 2023-05-22T15:39:32.009 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:31.999+0000 7f2917fff640 1 --2- 172.21.15.16:0/2845281225 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f291c065580 0x7f291c067a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:32.009 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:31.999+0000 7f2917fff640 1 -- 172.21.15.16:0/2845281225 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 6698+0+0 (secure 0 0 0) 0x7f292411d000 con 0x7f293014c7c0 2023-05-22T15:39:32.009 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:31.999+0000 7f292effd640 1 --2- 172.21.15.16:0/2845281225 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f291c065580 0x7f291c067a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:32.009 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:32.003+0000 7f292effd640 1 --2- 172.21.15.16:0/2845281225 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f291c065580 0x7f291c067a30 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f2920001ec0 tx=0x7f2920001d50 comp rx=0 tx=0).ready entity=mgr.5823 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:39:32.010 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:32.003+0000 7f2917fff640 1 -- 172.21.15.16:0/2845281225 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f29240ea830 con 0x7f293014c7c0 2023-05-22T15:39:32.185 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:32.179+0000 7f2935a1d640 1 -- 172.21.15.16:0/2845281225 --> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "fs": "cephfs", "path": "/snap_test_dir1/dir2", "repeat": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f29300b6280 con 0x7f291c065580 2023-05-22T15:39:32.366 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:32.359+0000 7f2917fff640 1 -- 172.21.15.16:0/2845281225 <== mgr.5823 v2:172.21.15.16:6832/36921 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+47 (secure 0 0 0) 0x7f29300b6280 con 0x7f291c065580 2023-05-22T15:39:32.366 INFO:teuthology.orchestra.run.smithi016.stdout:Schedule removed for path /snap_test_dir1/dir2 2023-05-22T15:39:32.369 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:32.363+0000 7f2935a1d640 1 -- 172.21.15.16:0/2845281225 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f291c065580 msgr2=0x7f291c067a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:32.369 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:32.363+0000 7f2935a1d640 1 --2- 172.21.15.16:0/2845281225 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f291c065580 0x7f291c067a30 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f2920001ec0 tx=0x7f2920001d50 comp rx=0 tx=0).stop 2023-05-22T15:39:32.369 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:32.363+0000 7f2935a1d640 1 -- 172.21.15.16:0/2845281225 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f293014c7c0 msgr2=0x7f2930153c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:32.369 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:32.363+0000 7f2935a1d640 1 --2- 172.21.15.16:0/2845281225 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f293014c7c0 0x7f2930153c20 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f2924009ae0 tx=0x7f2918005800 comp rx=0 tx=0).stop 2023-05-22T15:39:32.369 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:32.363+0000 7f2935a1d640 1 -- 172.21.15.16:0/2845281225 shutdown_connections 2023-05-22T15:39:32.370 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:32.363+0000 7f2935a1d640 1 --2- 172.21.15.16:0/2845281225 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f293015c380 0x7f293015a8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:32.370 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:32.363+0000 7f2935a1d640 1 --2- 172.21.15.16:0/2845281225 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f293014c7c0 0x7f2930153c20 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:32.370 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:32.363+0000 7f2935a1d640 1 --2- 172.21.15.16:0/2845281225 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f291c065580 0x7f291c067a30 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:32.370 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:32.363+0000 7f2935a1d640 1 --2- 172.21.15.16:0/2845281225 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f29180047d0 0x7f29300c5f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:32.370 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:32.363+0000 7f2935a1d640 1 -- 172.21.15.16:0/2845281225 >> 172.21.15.16:0/2845281225 conn(0x7f293006d9d0 msgr2=0x7f29300b1590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:39:32.370 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:32.363+0000 7f2935a1d640 1 -- 172.21.15.16:0/2845281225 shutdown_connections 2023-05-22T15:39:32.370 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:32.363+0000 7f2935a1d640 1 -- 172.21.15.16:0/2845281225 wait complete. 2023-05-22T15:39:32.394 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/dir2 2023-05-22T15:39:32.396 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:39:32.396 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir2/.snap) 2023-05-22T15:39:32.422 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_29_00_UTC 2023-05-22T15:39:32.423 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_30_00_UTC 2023-05-22T15:39:32.423 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_31_00_UTC 2023-05-22T15:39:32.423 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_32_00_UTC 2023-05-22T15:39:32.423 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_33_00_UTC 2023-05-22T15:39:32.423 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_34_00_UTC 2023-05-22T15:39:32.423 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_35_00_UTC 2023-05-22T15:39:32.423 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_36_00_UTC 2023-05-22T15:39:32.424 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_37_00_UTC 2023-05-22T15:39:32.424 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_38_00_UTC 2023-05-22T15:39:32.424 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_39_00_UTC 2023-05-22T15:39:32.424 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2023-05-22-15_29_00_UTC 2023-05-22T15:39:32.425 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:39:32.425 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2023-05-22-15_29_00_UTC) 2023-05-22T15:39:32.484 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2023-05-22-15_30_00_UTC 2023-05-22T15:39:32.484 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:39:32.485 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2023-05-22-15_30_00_UTC) 2023-05-22T15:39:32.545 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2023-05-22-15_31_00_UTC 2023-05-22T15:39:32.545 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:39:32.545 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2023-05-22-15_31_00_UTC) 2023-05-22T15:39:32.606 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2023-05-22-15_32_00_UTC 2023-05-22T15:39:32.606 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:39:32.607 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2023-05-22-15_32_00_UTC) 2023-05-22T15:39:32.672 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2023-05-22-15_33_00_UTC 2023-05-22T15:39:32.673 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:39:32.673 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2023-05-22-15_33_00_UTC) 2023-05-22T15:39:32.736 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2023-05-22-15_34_00_UTC 2023-05-22T15:39:32.737 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:39:32.737 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2023-05-22-15_34_00_UTC) 2023-05-22T15:39:32.795 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2023-05-22-15_35_00_UTC 2023-05-22T15:39:32.796 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:39:32.796 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2023-05-22-15_35_00_UTC) 2023-05-22T15:39:32.863 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2023-05-22-15_36_00_UTC 2023-05-22T15:39:32.864 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:39:32.864 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2023-05-22-15_36_00_UTC) 2023-05-22T15:39:32.925 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2023-05-22-15_37_00_UTC 2023-05-22T15:39:32.926 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:39:32.926 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2023-05-22-15_37_00_UTC) 2023-05-22T15:39:32.986 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2023-05-22-15_38_00_UTC 2023-05-22T15:39:32.986 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:39:32.987 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2023-05-22-15_38_00_UTC) 2023-05-22T15:39:33.051 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2023-05-22-15_39_00_UTC 2023-05-22T15:39:33.052 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:39:33.052 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2023-05-22-15_39_00_UTC) 2023-05-22T15:39:33.114 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:39:33.115 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2) 2023-05-22T15:39:33.169 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove --fs cephfs /snap_test_dir1/dir3 1M 2023-05-22T15:39:33.348 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:33.343+0000 7f16e38c1640 1 Processor -- start 2023-05-22T15:39:33.349 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:33.343+0000 7f16e38c1640 1 -- start start 2023-05-22T15:39:33.350 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:33.343+0000 7f16e38c1640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f16dc159080 0x7f16dc0c8f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:33.350 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:33.343+0000 7f16e38c1640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f16dc0c9460 0x7f16dc069f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:33.350 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:33.343+0000 7f16e38c1640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16dc069a90 con 0x7f16dc06fef0 2023-05-22T15:39:33.350 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:33.343+0000 7f16e38c1640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16dc0cc7f0 con 0x7f16dc159080 2023-05-22T15:39:33.350 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:33.343+0000 7f16e38c1640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16dc0ccad0 con 0x7f16dc0c9460 2023-05-22T15:39:33.351 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:33.343+0000 7f16e0e5c640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f16dc159080 0x7f16dc0c8f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:33.351 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:33.343+0000 7f16e0e5c640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f16dc159080 0x7f16dc0c8f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53390/0 (socket says 172.21.15.16:53390) 2023-05-22T15:39:33.351 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:33.343+0000 7f16e0e5c640 1 -- 172.21.15.16:0/1406858440 learned_addr learned my addr 172.21.15.16:0/1406858440 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:39:33.351 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:33.343+0000 7f16e1e5e640 1 --2- 172.21.15.16:0/1406858440 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f16dc0c9460 0x7f16dc069f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:33.352 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:33.347+0000 7f16e0e5c640 1 -- 172.21.15.16:0/1406858440 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f16dc0c9460 msgr2=0x7f16dc069f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:39:33.352 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:33.347+0000 7f16e0e5c640 1 --2- 172.21.15.16:0/1406858440 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f16dc0c9460 0x7f16dc069f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:33.352 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:33.347+0000 7f16e0e5c640 1 -- 172.21.15.16:0/1406858440 >> v1:172.21.15.62:6789/0 conn(0x7f16dc06fef0 legacy=0x7f16dc0702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:39:33.352 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:33.347+0000 7f16e0e5c640 1 -- 172.21.15.16:0/1406858440 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16dc0ccdb0 con 0x7f16dc159080 2023-05-22T15:39:33.353 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:33.347+0000 7f16e0e5c640 1 --2- 172.21.15.16:0/1406858440 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16dc159080 0x7f16dc0c8f20 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f16d000aa10 tx=0x7f16d001d8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a0e3eb21234fae5 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:39:33.353 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:33.347+0000 7f16cffff640 1 -- 172.21.15.16:0/1406858440 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16d00a6850 con 0x7f16dc159080 2023-05-22T15:39:33.354 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:33.347+0000 7f16cffff640 1 -- 172.21.15.16:0/1406858440 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16d00a69b0 con 0x7f16dc159080 2023-05-22T15:39:33.354 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:33.347+0000 7f16cffff640 1 -- 172.21.15.16:0/1406858440 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16d00a6d40 con 0x7f16dc159080 2023-05-22T15:39:33.354 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:33.351+0000 7f16e38c1640 1 -- 172.21.15.16:0/1406858440 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16dc159080 msgr2=0x7f16dc0c8f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:33.355 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:33.351+0000 7f16e38c1640 1 --2- 172.21.15.16:0/1406858440 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16dc159080 0x7f16dc0c8f20 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f16d000aa10 tx=0x7f16d001d8e0 comp rx=0 tx=0).stop 2023-05-22T15:39:33.355 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:33.351+0000 7f16e38c1640 1 -- 172.21.15.16:0/1406858440 shutdown_connections 2023-05-22T15:39:33.355 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:33.351+0000 7f16e38c1640 1 --2- 172.21.15.16:0/1406858440 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f16dc0c9460 0x7f16dc069f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:33.355 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:33.351+0000 7f16e38c1640 1 --2- 172.21.15.16:0/1406858440 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16dc159080 0x7f16dc0c8f20 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:33.355 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:33.351+0000 7f16e38c1640 1 -- 172.21.15.16:0/1406858440 >> 172.21.15.16:0/1406858440 conn(0x7f16dc0ce8d0 msgr2=0x7f16dc0cecd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:39:33.355 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:33.351+0000 7f16e38c1640 1 -- 172.21.15.16:0/1406858440 shutdown_connections 2023-05-22T15:39:33.355 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:33.351+0000 7f16e38c1640 1 -- 172.21.15.16:0/1406858440 wait complete. 2023-05-22T15:39:33.356 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:33.351+0000 7f16e38c1640 1 Processor -- start 2023-05-22T15:39:33.356 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:33.351+0000 7f16e38c1640 1 -- start start 2023-05-22T15:39:33.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:33.351+0000 7f16e38c1640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f16dc06fef0 0x7f16dc14fde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:33.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:33.351+0000 7f16e38c1640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f16dc0c9460 0x7f16dc0ae1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:33.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:33.351+0000 7f16e38c1640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16dc159080 0x7f16dc0af7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:33.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:33.351+0000 7f16e38c1640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16dc154c80 con 0x7f16dc159080 2023-05-22T15:39:33.358 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:33.351+0000 7f16e38c1640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16dc0aec80 con 0x7f16dc06fef0 2023-05-22T15:39:33.358 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:33.351+0000 7f16e38c1640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16dc0aef60 con 0x7f16dc0c9460 2023-05-22T15:39:33.358 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:33.351+0000 7f16e1e5e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16dc159080 0x7f16dc0af7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:33.358 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:33.351+0000 7f16e1e5e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16dc159080 0x7f16dc0af7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53402/0 (socket says 172.21.15.16:53402) 2023-05-22T15:39:33.358 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:33.351+0000 7f16e1e5e640 1 -- 172.21.15.16:0/607627500 learned_addr learned my addr 172.21.15.16:0/607627500 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:39:33.358 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:33.351+0000 7f16e0e5c640 1 --2- 172.21.15.16:0/607627500 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f16dc0c9460 0x7f16dc0ae1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:33.358 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:33.351+0000 7f16e165d640 1 --2- 172.21.15.16:0/607627500 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f16dc06fef0 0x7f16dc14fde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:33.359 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:33.351+0000 7f16e1e5e640 1 -- 172.21.15.16:0/607627500 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f16dc0c9460 msgr2=0x7f16dc0ae1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:33.359 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:33.351+0000 7f16e1e5e640 1 --2- 172.21.15.16:0/607627500 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f16dc0c9460 0x7f16dc0ae1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:33.359 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:33.351+0000 7f16e1e5e640 1 -- 172.21.15.16:0/607627500 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f16dc06fef0 msgr2=0x7f16dc14fde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:33.359 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:33.351+0000 7f16e1e5e640 1 --2- 172.21.15.16:0/607627500 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f16dc06fef0 0x7f16dc14fde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:33.359 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:33.351+0000 7f16e1e5e640 1 -- 172.21.15.16:0/607627500 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16d000a670 con 0x7f16dc159080 2023-05-22T15:39:33.360 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:33.351+0000 7f16e1e5e640 1 --2- 172.21.15.16:0/607627500 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16dc159080 0x7f16dc0af7e0 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f16d800e930 tx=0x7f16d800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:39:33.360 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:33.355+0000 7f16cdffb640 1 -- 172.21.15.16:0/607627500 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16d80096b0 con 0x7f16dc159080 2023-05-22T15:39:33.360 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:33.355+0000 7f16cdffb640 1 -- 172.21.15.16:0/607627500 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16d800c9a0 con 0x7f16dc159080 2023-05-22T15:39:33.360 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:33.355+0000 7f16cdffb640 1 -- 172.21.15.16:0/607627500 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16d800cd30 con 0x7f16dc159080 2023-05-22T15:39:33.361 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:33.355+0000 7f16e38c1640 1 -- 172.21.15.16:0/607627500 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16dc158340 con 0x7f16dc159080 2023-05-22T15:39:33.361 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:33.355+0000 7f16e38c1640 1 -- 172.21.15.16:0/607627500 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16dc0cc4a0 con 0x7f16dc159080 2023-05-22T15:39:33.361 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:33.355+0000 7f16e38c1640 1 -- 172.21.15.16:0/607627500 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16dc1584d0 con 0x7f16dc159080 2023-05-22T15:39:33.361 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:33.355+0000 7f16cdffb640 1 -- 172.21.15.16:0/607627500 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 12) v1 ==== 84418+0+0 (secure 0 0 0) 0x7f16d8013470 con 0x7f16dc159080 2023-05-22T15:39:33.362 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:33.355+0000 7f16cdffb640 1 --2- 172.21.15.16:0/607627500 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f16c0065580 0x7f16c0067a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:33.362 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:33.355+0000 7f16cdffb640 1 -- 172.21.15.16:0/607627500 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 7298+0+0 (secure 0 0 0) 0x7f16d8083090 con 0x7f16dc159080 2023-05-22T15:39:33.362 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:33.355+0000 7f16e165d640 1 --2- 172.21.15.16:0/607627500 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f16c0065580 0x7f16c0067a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:33.362 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:33.359+0000 7f16e165d640 1 --2- 172.21.15.16:0/607627500 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f16c0065580 0x7f16c0067a30 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f16d40012f0 tx=0x7f16d4009040 comp rx=0 tx=0).ready entity=mgr.5823 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:39:33.365 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:33.359+0000 7f16cdffb640 1 -- 172.21.15.16:0/607627500 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f16dc1584d0 con 0x7f16dc159080 2023-05-22T15:39:33.574 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:33.571+0000 7f16e38c1640 1 -- 172.21.15.16:0/607627500 --> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "fs": "cephfs", "path": "/snap_test_dir1/dir3", "repeat": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f16dc1584d0 con 0x7f16c0065580 2023-05-22T15:39:33.742 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:33.735+0000 7f16cdffb640 1 -- 172.21.15.16:0/607627500 <== mgr.5823 v2:172.21.15.16:6832/36921 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+47 (secure 0 0 0) 0x7f16dc1584d0 con 0x7f16c0065580 2023-05-22T15:39:33.742 INFO:teuthology.orchestra.run.smithi016.stdout:Schedule removed for path /snap_test_dir1/dir3 2023-05-22T15:39:33.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:33.739+0000 7f16e38c1640 1 -- 172.21.15.16:0/607627500 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f16c0065580 msgr2=0x7f16c0067a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:33.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:33.739+0000 7f16e38c1640 1 --2- 172.21.15.16:0/607627500 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f16c0065580 0x7f16c0067a30 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f16d40012f0 tx=0x7f16d4009040 comp rx=0 tx=0).stop 2023-05-22T15:39:33.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:33.739+0000 7f16e38c1640 1 -- 172.21.15.16:0/607627500 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16dc159080 msgr2=0x7f16dc0af7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:33.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:33.739+0000 7f16e38c1640 1 --2- 172.21.15.16:0/607627500 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16dc159080 0x7f16dc0af7e0 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f16d800e930 tx=0x7f16d800edf0 comp rx=0 tx=0).stop 2023-05-22T15:39:33.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:33.739+0000 7f16e38c1640 1 -- 172.21.15.16:0/607627500 shutdown_connections 2023-05-22T15:39:33.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:33.739+0000 7f16e38c1640 1 --2- 172.21.15.16:0/607627500 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16dc159080 0x7f16dc0af7e0 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:33.746 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:33.739+0000 7f16e38c1640 1 --2- 172.21.15.16:0/607627500 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f16c0065580 0x7f16c0067a30 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:33.746 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:33.739+0000 7f16e38c1640 1 --2- 172.21.15.16:0/607627500 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f16dc0c9460 0x7f16dc0ae1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:33.746 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:33.739+0000 7f16e38c1640 1 --2- 172.21.15.16:0/607627500 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f16dc06fef0 0x7f16dc14fde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:33.746 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:33.739+0000 7f16e38c1640 1 -- 172.21.15.16:0/607627500 >> 172.21.15.16:0/607627500 conn(0x7f16dc0ce8d0 msgr2=0x7f16dc06b9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:39:33.746 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:33.739+0000 7f16e38c1640 1 -- 172.21.15.16:0/607627500 shutdown_connections 2023-05-22T15:39:33.746 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:33.739+0000 7f16e38c1640 1 -- 172.21.15.16:0/607627500 wait complete. 2023-05-22T15:39:33.766 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/dir3 2023-05-22T15:39:33.766 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:39:33.766 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir3/.snap) 2023-05-22T15:39:33.781 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_29_00_UTC 2023-05-22T15:39:33.781 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_30_00_UTC 2023-05-22T15:39:33.781 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_31_00_UTC 2023-05-22T15:39:33.782 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_32_00_UTC 2023-05-22T15:39:33.782 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_33_00_UTC 2023-05-22T15:39:33.782 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_34_00_UTC 2023-05-22T15:39:33.782 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_35_00_UTC 2023-05-22T15:39:33.782 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_36_00_UTC 2023-05-22T15:39:33.782 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_37_00_UTC 2023-05-22T15:39:33.782 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_38_00_UTC 2023-05-22T15:39:33.782 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_39_01_UTC 2023-05-22T15:39:33.783 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2023-05-22-15_29_00_UTC 2023-05-22T15:39:33.784 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:39:33.784 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2023-05-22-15_29_00_UTC) 2023-05-22T15:39:33.845 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2023-05-22-15_30_00_UTC 2023-05-22T15:39:33.845 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:39:33.846 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2023-05-22-15_30_00_UTC) 2023-05-22T15:39:33.903 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2023-05-22-15_31_00_UTC 2023-05-22T15:39:33.904 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:39:33.904 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2023-05-22-15_31_00_UTC) 2023-05-22T15:39:33.966 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2023-05-22-15_32_00_UTC 2023-05-22T15:39:33.966 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:39:33.967 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2023-05-22-15_32_00_UTC) 2023-05-22T15:39:34.030 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2023-05-22-15_33_00_UTC 2023-05-22T15:39:34.031 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:39:34.031 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2023-05-22-15_33_00_UTC) 2023-05-22T15:39:34.094 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2023-05-22-15_34_00_UTC 2023-05-22T15:39:34.095 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:39:34.095 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2023-05-22-15_34_00_UTC) 2023-05-22T15:39:34.157 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2023-05-22-15_35_00_UTC 2023-05-22T15:39:34.157 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:39:34.158 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2023-05-22-15_35_00_UTC) 2023-05-22T15:39:34.217 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2023-05-22-15_36_00_UTC 2023-05-22T15:39:34.219 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:39:34.220 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2023-05-22-15_36_00_UTC) 2023-05-22T15:39:34.277 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2023-05-22-15_37_00_UTC 2023-05-22T15:39:34.278 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:39:34.279 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2023-05-22-15_37_00_UTC) 2023-05-22T15:39:34.336 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2023-05-22-15_38_00_UTC 2023-05-22T15:39:34.337 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:39:34.338 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2023-05-22-15_38_00_UTC) 2023-05-22T15:39:34.403 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2023-05-22-15_39_01_UTC 2023-05-22T15:39:34.403 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:39:34.403 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2023-05-22-15_39_01_UTC) 2023-05-22T15:39:34.466 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:39:34.466 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3) 2023-05-22T15:39:34.519 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove --fs cephfs /snap_test_dir1/dir4 1M 2023-05-22T15:39:34.692 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:34.686+0000 7f20012d2640 1 Processor -- start 2023-05-22T15:39:34.693 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:34.686+0000 7f20012d2640 1 -- start start 2023-05-22T15:39:34.694 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:34.686+0000 7f20012d2640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1ffc0cfa40 0x7f1ffc0cd260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:34.694 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:34.686+0000 7f20012d2640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f1ffc06a830 0x7f1ffc06acb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:34.694 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:34.686+0000 7f20012d2640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ffc15c0a0 con 0x7f1ffc0cf670 2023-05-22T15:39:34.694 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:34.686+0000 7f20012d2640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ffc0cc850 con 0x7f1ffc06a830 2023-05-22T15:39:34.694 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:34.686+0000 7f20012d2640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ffc0ccdd0 con 0x7f1ffc0cfa40 2023-05-22T15:39:34.694 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:34.686+0000 7f1ffb7fe640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f1ffc06a830 0x7f1ffc06acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:34.694 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:34.686+0000 7f1ffaffd640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f1ffc0cf670 0x7f1ffc1601d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:46164/0 (socket says 172.21.15.16:46164) 2023-05-22T15:39:34.695 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:34.686+0000 7f1ffaffd640 1 -- 172.21.15.16:0/1741334060 learned_addr learned my addr 172.21.15.16:0/1741334060 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:39:34.695 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:34.686+0000 7f1ff3fff640 1 --2- 172.21.15.16:0/1741334060 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1ffc0cfa40 0x7f1ffc0cd260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:34.695 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:34.686+0000 7f1ffb7fe640 1 -- 172.21.15.16:0/1741334060 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1ffc0cfa40 msgr2=0x7f1ffc0cd260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:39:34.695 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:34.686+0000 7f1ffb7fe640 1 --2- 172.21.15.16:0/1741334060 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1ffc0cfa40 0x7f1ffc0cd260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:34.695 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:34.686+0000 7f1ffb7fe640 1 -- 172.21.15.16:0/1741334060 >> v1:172.21.15.16:6789/0 conn(0x7f1ffc0cf670 legacy=0x7f1ffc1601d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:34.695 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:34.686+0000 7f1ffb7fe640 1 -- 172.21.15.16:0/1741334060 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ffc06c3d0 con 0x7f1ffc06a830 2023-05-22T15:39:34.695 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:34.686+0000 7f1ffa7fc640 1 -- 172.21.15.16:0/1741334060 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2605616376 0 0) 0x7f1fec001120 con 0x7f1ffc0cf670 2023-05-22T15:39:34.696 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:34.690+0000 7f1ffb7fe640 1 --2- 172.21.15.16:0/1741334060 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1ffc06a830 0x7f1ffc06acb0 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f1ff4009dc0 tx=0x7f1ff401c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3f16c9a6e5fee76 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:39:34.696 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:34.690+0000 7f1ffa7fc640 1 -- 172.21.15.16:0/1741334060 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ff40a59c0 con 0x7f1ffc06a830 2023-05-22T15:39:34.696 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:34.690+0000 7f1ffa7fc640 1 -- 172.21.15.16:0/1741334060 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ff40a5b20 con 0x7f1ffc06a830 2023-05-22T15:39:34.697 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:34.690+0000 7f1ffa7fc640 1 -- 172.21.15.16:0/1741334060 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ff40ad440 con 0x7f1ffc06a830 2023-05-22T15:39:34.697 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:34.690+0000 7f20012d2640 1 -- 172.21.15.16:0/1741334060 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1ffc06a830 msgr2=0x7f1ffc06acb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:34.697 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:34.690+0000 7f20012d2640 1 --2- 172.21.15.16:0/1741334060 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1ffc06a830 0x7f1ffc06acb0 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f1ff4009dc0 tx=0x7f1ff401c900 comp rx=0 tx=0).stop 2023-05-22T15:39:34.698 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:34.690+0000 7f20012d2640 1 -- 172.21.15.16:0/1741334060 shutdown_connections 2023-05-22T15:39:34.698 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:34.690+0000 7f20012d2640 1 --2- 172.21.15.16:0/1741334060 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1ffc06a830 0x7f1ffc06acb0 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:34.698 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:34.690+0000 7f20012d2640 1 --2- 172.21.15.16:0/1741334060 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1ffc0cfa40 0x7f1ffc0cd260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:34.698 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:34.690+0000 7f20012d2640 1 -- 172.21.15.16:0/1741334060 >> 172.21.15.16:0/1741334060 conn(0x7f1ffc0700b0 msgr2=0x7f1ffc06d9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:39:34.698 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:34.690+0000 7f20012d2640 1 -- 172.21.15.16:0/1741334060 shutdown_connections 2023-05-22T15:39:34.698 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:34.690+0000 7f20012d2640 1 -- 172.21.15.16:0/1741334060 wait complete. 2023-05-22T15:39:34.699 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:34.694+0000 7f20012d2640 1 Processor -- start 2023-05-22T15:39:34.699 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:34.694+0000 7f20012d2640 1 -- start start 2023-05-22T15:39:34.699 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:34.694+0000 7f20012d2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1ffc06a830 0x7f1ffc158670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:34.699 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:34.694+0000 7f20012d2640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1ffc0cf670 0x7f1ffc159680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:34.699 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:34.694+0000 7f20012d2640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1ffc0cfa40 0x7f1ffc159bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:34.700 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:34.694+0000 7f20012d2640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ffc142e70 con 0x7f1ffc06a830 2023-05-22T15:39:34.700 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:34.694+0000 7f20012d2640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ffc0acfb0 con 0x7f1ffc0cf670 2023-05-22T15:39:34.700 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:34.694+0000 7f20012d2640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ffc158bb0 con 0x7f1ffc0cfa40 2023-05-22T15:39:34.701 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:34.694+0000 7f1ffaffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1ffc06a830 0x7f1ffc158670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:34.701 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:34.694+0000 7f1ffaffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1ffc06a830 0x7f1ffc158670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53420/0 (socket says 172.21.15.16:53420) 2023-05-22T15:39:34.701 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:34.694+0000 7f1ffaffd640 1 -- 172.21.15.16:0/2504498661 learned_addr learned my addr 172.21.15.16:0/2504498661 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:39:34.701 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:34.694+0000 7f1ffb7fe640 1 --2- 172.21.15.16:0/2504498661 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1ffc0cfa40 0x7f1ffc159bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:34.701 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:34.694+0000 7f1ff3fff640 1 --2- 172.21.15.16:0/2504498661 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1ffc0cf670 0x7f1ffc159680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:34.702 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:34.694+0000 7f1ffaffd640 1 -- 172.21.15.16:0/2504498661 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1ffc0cfa40 msgr2=0x7f1ffc159bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:34.702 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:34.694+0000 7f1ffaffd640 1 --2- 172.21.15.16:0/2504498661 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1ffc0cfa40 0x7f1ffc159bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:34.702 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:34.694+0000 7f1ffaffd640 1 -- 172.21.15.16:0/2504498661 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1ffc0cf670 msgr2=0x7f1ffc159680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:34.702 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:34.694+0000 7f1ffaffd640 1 --2- 172.21.15.16:0/2504498661 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1ffc0cf670 0x7f1ffc159680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:34.702 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:34.694+0000 7f1ffaffd640 1 -- 172.21.15.16:0/2504498661 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ff4009a70 con 0x7f1ffc06a830 2023-05-22T15:39:34.702 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:34.694+0000 7f1ffaffd640 1 --2- 172.21.15.16:0/2504498661 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1ffc06a830 0x7f1ffc158670 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f1fec00c780 tx=0x7f1fec00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:39:34.702 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:34.694+0000 7f1ff37fe640 1 -- 172.21.15.16:0/2504498661 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fec0072d0 con 0x7f1ffc06a830 2023-05-22T15:39:34.702 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:34.694+0000 7f1ff37fe640 1 -- 172.21.15.16:0/2504498661 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fec007430 con 0x7f1ffc06a830 2023-05-22T15:39:34.703 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:34.694+0000 7f1ff37fe640 1 -- 172.21.15.16:0/2504498661 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fec0047f0 con 0x7f1ffc06a830 2023-05-22T15:39:34.703 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:34.694+0000 7f20012d2640 1 -- 172.21.15.16:0/2504498661 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ffc0cd7a0 con 0x7f1ffc06a830 2023-05-22T15:39:34.703 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:34.694+0000 7f20012d2640 1 -- 172.21.15.16:0/2504498661 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ffc06c0e0 con 0x7f1ffc06a830 2023-05-22T15:39:34.704 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:34.694+0000 7f20012d2640 1 -- 172.21.15.16:0/2504498661 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ffc0cc5a0 con 0x7f1ffc06a830 2023-05-22T15:39:34.704 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:34.698+0000 7f1ff37fe640 1 -- 172.21.15.16:0/2504498661 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 12) v1 ==== 84418+0+0 (secure 0 0 0) 0x7f1fec004950 con 0x7f1ffc06a830 2023-05-22T15:39:34.704 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:34.698+0000 7f1ff37fe640 1 --2- 172.21.15.16:0/2504498661 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f1fdc065580 0x7f1fdc067a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:34.704 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:34.698+0000 7f1ff37fe640 1 -- 172.21.15.16:0/2504498661 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 7378+0+0 (secure 0 0 0) 0x7f1fec0824c0 con 0x7f1ffc06a830 2023-05-22T15:39:34.704 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:34.698+0000 7f1ff3fff640 1 --2- 172.21.15.16:0/2504498661 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f1fdc065580 0x7f1fdc067a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:34.704 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:34.698+0000 7f1ff3fff640 1 --2- 172.21.15.16:0/2504498661 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f1fdc065580 0x7f1fdc067a30 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f1ffc159000 tx=0x7f1fe4008040 comp rx=0 tx=0).ready entity=mgr.5823 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:39:34.706 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:34.702+0000 7f1ff37fe640 1 -- 172.21.15.16:0/2504498661 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f1fec04fa40 con 0x7f1ffc06a830 2023-05-22T15:39:34.901 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:34.894+0000 7f20012d2640 1 -- 172.21.15.16:0/2504498661 --> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "fs": "cephfs", "path": "/snap_test_dir1/dir4", "repeat": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f1ffc0ce680 con 0x7f1fdc065580 2023-05-22T15:39:35.056 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:35.050+0000 7f1ff37fe640 1 -- 172.21.15.16:0/2504498661 <== mgr.5823 v2:172.21.15.16:6832/36921 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+47 (secure 0 0 0) 0x7f1ffc0ce680 con 0x7f1fdc065580 2023-05-22T15:39:35.056 INFO:teuthology.orchestra.run.smithi016.stdout:Schedule removed for path /snap_test_dir1/dir4 2023-05-22T15:39:35.058 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:35.054+0000 7f20012d2640 1 -- 172.21.15.16:0/2504498661 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f1fdc065580 msgr2=0x7f1fdc067a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:35.058 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:35.054+0000 7f20012d2640 1 --2- 172.21.15.16:0/2504498661 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f1fdc065580 0x7f1fdc067a30 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f1ffc159000 tx=0x7f1fe4008040 comp rx=0 tx=0).stop 2023-05-22T15:39:35.058 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:35.054+0000 7f20012d2640 1 -- 172.21.15.16:0/2504498661 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1ffc06a830 msgr2=0x7f1ffc158670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:35.058 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:35.054+0000 7f20012d2640 1 --2- 172.21.15.16:0/2504498661 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1ffc06a830 0x7f1ffc158670 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f1fec00c780 tx=0x7f1fec00cc40 comp rx=0 tx=0).stop 2023-05-22T15:39:35.059 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:35.054+0000 7f20012d2640 1 -- 172.21.15.16:0/2504498661 shutdown_connections 2023-05-22T15:39:35.059 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:35.054+0000 7f20012d2640 1 --2- 172.21.15.16:0/2504498661 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f1fdc065580 0x7f1fdc067a30 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:35.059 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:35.054+0000 7f20012d2640 1 --2- 172.21.15.16:0/2504498661 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1ffc0cfa40 0x7f1ffc159bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:35.059 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:35.054+0000 7f20012d2640 1 --2- 172.21.15.16:0/2504498661 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1ffc0cf670 0x7f1ffc159680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:35.060 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:35.054+0000 7f20012d2640 1 --2- 172.21.15.16:0/2504498661 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1ffc06a830 0x7f1ffc158670 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:35.060 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:35.054+0000 7f20012d2640 1 -- 172.21.15.16:0/2504498661 >> 172.21.15.16:0/2504498661 conn(0x7f1ffc0700b0 msgr2=0x7f1ffc0afde0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:39:35.060 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:35.054+0000 7f20012d2640 1 -- 172.21.15.16:0/2504498661 shutdown_connections 2023-05-22T15:39:35.060 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:35.054+0000 7f20012d2640 1 -- 172.21.15.16:0/2504498661 wait complete. 2023-05-22T15:39:35.076 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/dir4 2023-05-22T15:39:35.077 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:39:35.079 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir4/.snap) 2023-05-22T15:39:35.095 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_29_00_UTC 2023-05-22T15:39:35.095 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_30_01_UTC 2023-05-22T15:39:35.095 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_31_00_UTC 2023-05-22T15:39:35.096 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_32_00_UTC 2023-05-22T15:39:35.096 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_33_00_UTC 2023-05-22T15:39:35.096 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_34_00_UTC 2023-05-22T15:39:35.096 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_35_00_UTC 2023-05-22T15:39:35.096 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_36_01_UTC 2023-05-22T15:39:35.096 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_37_00_UTC 2023-05-22T15:39:35.096 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_38_00_UTC 2023-05-22T15:39:35.097 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_39_00_UTC 2023-05-22T15:39:35.097 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2023-05-22-15_29_00_UTC 2023-05-22T15:39:35.098 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:39:35.098 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2023-05-22-15_29_00_UTC) 2023-05-22T15:39:35.158 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2023-05-22-15_30_01_UTC 2023-05-22T15:39:35.159 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:39:35.159 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2023-05-22-15_30_01_UTC) 2023-05-22T15:39:35.220 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2023-05-22-15_31_00_UTC 2023-05-22T15:39:35.220 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:39:35.221 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2023-05-22-15_31_00_UTC) 2023-05-22T15:39:35.280 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2023-05-22-15_32_00_UTC 2023-05-22T15:39:35.280 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:39:35.281 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2023-05-22-15_32_00_UTC) 2023-05-22T15:39:35.336 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2023-05-22-15_33_00_UTC 2023-05-22T15:39:35.336 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:39:35.337 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2023-05-22-15_33_00_UTC) 2023-05-22T15:39:35.402 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2023-05-22-15_34_00_UTC 2023-05-22T15:39:35.403 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:39:35.403 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2023-05-22-15_34_00_UTC) 2023-05-22T15:39:35.464 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2023-05-22-15_35_00_UTC 2023-05-22T15:39:35.465 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:39:35.465 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2023-05-22-15_35_00_UTC) 2023-05-22T15:39:35.523 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2023-05-22-15_36_01_UTC 2023-05-22T15:39:35.523 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:39:35.524 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2023-05-22-15_36_01_UTC) 2023-05-22T15:39:35.581 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2023-05-22-15_37_00_UTC 2023-05-22T15:39:35.581 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:39:35.582 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2023-05-22-15_37_00_UTC) 2023-05-22T15:39:35.645 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2023-05-22-15_38_00_UTC 2023-05-22T15:39:35.646 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:39:35.646 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2023-05-22-15_38_00_UTC) 2023-05-22T15:39:35.704 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2023-05-22-15_39_00_UTC 2023-05-22T15:39:35.704 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:39:35.705 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2023-05-22-15_39_00_UTC) 2023-05-22T15:39:35.770 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:39:35.770 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4) 2023-05-22T15:39:35.826 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove --fs cephfs /snap_test_dir1/dir5 1M 2023-05-22T15:39:35.998 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:35.994+0000 7f1795cb2640 1 Processor -- start 2023-05-22T15:39:35.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:35.994+0000 7f1795cb2640 1 -- start start 2023-05-22T15:39:36.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:35.994+0000 7f1795cb2640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f1790069720 0x7f1790069af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:36.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:35.994+0000 7f1795cb2640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f17900718e0 0x7f179006ce10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:36.001 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:35.994+0000 7f1795cb2640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1790070d20 con 0x7f17900cc880 2023-05-22T15:39:36.001 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:35.994+0000 7f1795cb2640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17900b1c70 con 0x7f17900718e0 2023-05-22T15:39:36.001 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:35.994+0000 7f1795cb2640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f179015e5d0 con 0x7f1790069720 2023-05-22T15:39:36.001 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:35.994+0000 7f178ffff640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f17900718e0 0x7f179006ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:36.001 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:35.994+0000 7f178ffff640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f17900718e0 0x7f179006ce10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53424/0 (socket says 172.21.15.16:53424) 2023-05-22T15:39:36.001 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:35.994+0000 7f178ffff640 1 -- 172.21.15.16:0/3599577926 learned_addr learned my addr 172.21.15.16:0/3599577926 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:39:36.001 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:35.994+0000 7f178f7fe640 1 --2- 172.21.15.16:0/3599577926 >> v2:172.21.15.62:3300/0 conn(0x7f1790069720 0x7f1790069af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:36.002 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:35.994+0000 7f178ffff640 1 -- 172.21.15.16:0/3599577926 >> v2:172.21.15.62:3300/0 conn(0x7f1790069720 msgr2=0x7f1790069af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:39:36.002 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:35.994+0000 7f178ffff640 1 --2- 172.21.15.16:0/3599577926 >> v2:172.21.15.62:3300/0 conn(0x7f1790069720 0x7f1790069af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:36.002 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:35.994+0000 7f178ffff640 1 -- 172.21.15.16:0/3599577926 >> v1:172.21.15.62:6789/0 conn(0x7f17900cc880 legacy=0x7f179006c700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:39:36.002 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:35.994+0000 7f178ffff640 1 -- 172.21.15.16:0/3599577926 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1790159f40 con 0x7f17900718e0 2023-05-22T15:39:36.003 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:35.998+0000 7f178ffff640 1 --2- 172.21.15.16:0/3599577926 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f17900718e0 0x7f179006ce10 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f179006f680 tx=0x7f178801c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92299b250a6d292a server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:39:36.003 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:35.998+0000 7f178effd640 1 -- 172.21.15.16:0/3599577926 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17880a57c0 con 0x7f17900718e0 2023-05-22T15:39:36.003 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:35.998+0000 7f178effd640 1 -- 172.21.15.16:0/3599577926 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17880a5920 con 0x7f17900718e0 2023-05-22T15:39:36.003 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:35.998+0000 7f178effd640 1 -- 172.21.15.16:0/3599577926 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17880a5cb0 con 0x7f17900718e0 2023-05-22T15:39:36.004 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:35.998+0000 7f1795cb2640 1 -- 172.21.15.16:0/3599577926 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f17900718e0 msgr2=0x7f179006ce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:36.004 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:35.998+0000 7f1795cb2640 1 --2- 172.21.15.16:0/3599577926 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f17900718e0 0x7f179006ce10 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f179006f680 tx=0x7f178801c900 comp rx=0 tx=0).stop 2023-05-22T15:39:36.004 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:35.998+0000 7f1795cb2640 1 -- 172.21.15.16:0/3599577926 shutdown_connections 2023-05-22T15:39:36.004 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:35.998+0000 7f1795cb2640 1 --2- 172.21.15.16:0/3599577926 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f17900718e0 0x7f179006ce10 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:36.004 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:35.998+0000 7f1795cb2640 1 --2- 172.21.15.16:0/3599577926 >> v2:172.21.15.62:3300/0 conn(0x7f1790069720 0x7f1790069af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:36.004 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:35.998+0000 7f1795cb2640 1 -- 172.21.15.16:0/3599577926 >> 172.21.15.16:0/3599577926 conn(0x7f17900b0cc0 msgr2=0x7f17900b10c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:39:36.004 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:35.998+0000 7f1795cb2640 1 -- 172.21.15.16:0/3599577926 shutdown_connections 2023-05-22T15:39:36.004 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:35.998+0000 7f1795cb2640 1 -- 172.21.15.16:0/3599577926 wait complete. 2023-05-22T15:39:36.006 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:35.998+0000 7f1795cb2640 1 Processor -- start 2023-05-22T15:39:36.006 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:35.998+0000 7f1795cb2640 1 -- start start 2023-05-22T15:39:36.006 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:35.998+0000 7f1795cb2640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1790069720 0x7f179014cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:36.006 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:35.998+0000 7f1795cb2640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1790151570 0x7f17901519c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:36.006 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:35.998+0000 7f1795cb2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f179015c4f0 0x7f179015c960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:36.007 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:35.998+0000 7f1795cb2640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1790152bd0 con 0x7f179015c4f0 2023-05-22T15:39:36.007 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:35.998+0000 7f1795cb2640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1790151fc0 con 0x7f1790151570 2023-05-22T15:39:36.007 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:35.998+0000 7f1795cb2640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17901522a0 con 0x7f1790069720 2023-05-22T15:39:36.007 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:35.998+0000 7f178ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f179015c4f0 0x7f179015c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:36.007 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:35.998+0000 7f178ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f179015c4f0 0x7f179015c960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53430/0 (socket says 172.21.15.16:53430) 2023-05-22T15:39:36.007 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:35.998+0000 7f178ffff640 1 -- 172.21.15.16:0/1340901846 learned_addr learned my addr 172.21.15.16:0/1340901846 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:39:36.007 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:35.998+0000 7f178ffff640 1 -- 172.21.15.16:0/1340901846 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1790069720 msgr2=0x7f179014cfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:36.007 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:35.998+0000 7f178ffff640 1 --2- 172.21.15.16:0/1340901846 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1790069720 0x7f179014cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:36.008 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:35.998+0000 7f178ffff640 1 -- 172.21.15.16:0/1340901846 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1790151570 msgr2=0x7f17901519c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:36.008 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:35.998+0000 7f178ffff640 1 --2- 172.21.15.16:0/1340901846 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1790151570 0x7f17901519c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:36.008 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:35.998+0000 7f178ffff640 1 -- 172.21.15.16:0/1340901846 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1788009b30 con 0x7f179015c4f0 2023-05-22T15:39:36.008 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:35.998+0000 7f178ffff640 1 --2- 172.21.15.16:0/1340901846 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f179015c4f0 0x7f179015c960 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f17880af040 tx=0x7f17880196a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:39:36.008 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:35.998+0000 7f178cff9640 1 -- 172.21.15.16:0/1340901846 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17880b8880 con 0x7f179015c4f0 2023-05-22T15:39:36.008 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:35.998+0000 7f178cff9640 1 -- 172.21.15.16:0/1340901846 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17880b8a00 con 0x7f179015c4f0 2023-05-22T15:39:36.008 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:36.002+0000 7f1795cb2640 1 -- 172.21.15.16:0/1340901846 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f179015ddd0 con 0x7f179015c4f0 2023-05-22T15:39:36.009 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:36.002+0000 7f178cff9640 1 -- 172.21.15.16:0/1340901846 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17880b8db0 con 0x7f179015c4f0 2023-05-22T15:39:36.009 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:36.002+0000 7f1795cb2640 1 -- 172.21.15.16:0/1340901846 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f179015daf0 con 0x7f179015c4f0 2023-05-22T15:39:36.009 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:36.002+0000 7f1795cb2640 1 -- 172.21.15.16:0/1340901846 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f179015e0b0 con 0x7f179015c4f0 2023-05-22T15:39:36.010 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:36.002+0000 7f178cff9640 1 -- 172.21.15.16:0/1340901846 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 12) v1 ==== 84418+0+0 (secure 0 0 0) 0x7f17880c3460 con 0x7f179015c4f0 2023-05-22T15:39:36.010 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:36.002+0000 7f178cff9640 1 --2- 172.21.15.16:0/1340901846 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f1770065580 0x7f1770067a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:36.010 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:36.002+0000 7f178cff9640 1 -- 172.21.15.16:0/1340901846 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 7378+0+0 (secure 0 0 0) 0x7f178811e9b0 con 0x7f179015c4f0 2023-05-22T15:39:36.010 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:36.002+0000 7f178f7fe640 1 --2- 172.21.15.16:0/1340901846 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f1770065580 0x7f1770067a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:36.010 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:36.002+0000 7f178f7fe640 1 --2- 172.21.15.16:0/1340901846 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f1770065580 0x7f1770067a30 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f17800098f0 tx=0x7f1780008040 comp rx=0 tx=0).ready entity=mgr.5823 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:39:36.012 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:36.006+0000 7f178cff9640 1 -- 172.21.15.16:0/1340901846 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f17880ebef0 con 0x7f179015c4f0 2023-05-22T15:39:36.188 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:36.182+0000 7f1795cb2640 1 -- 172.21.15.16:0/1340901846 --> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "fs": "cephfs", "path": "/snap_test_dir1/dir5", "repeat": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f179015e0b0 con 0x7f1770065580 2023-05-22T15:39:36.368 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:36.362+0000 7f178cff9640 1 -- 172.21.15.16:0/1340901846 <== mgr.5823 v2:172.21.15.16:6832/36921 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+47 (secure 0 0 0) 0x7f179015e0b0 con 0x7f1770065580 2023-05-22T15:39:36.368 INFO:teuthology.orchestra.run.smithi016.stdout:Schedule removed for path /snap_test_dir1/dir5 2023-05-22T15:39:36.372 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:36.366+0000 7f1795cb2640 1 -- 172.21.15.16:0/1340901846 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f1770065580 msgr2=0x7f1770067a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:36.372 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:36.366+0000 7f1795cb2640 1 --2- 172.21.15.16:0/1340901846 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f1770065580 0x7f1770067a30 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f17800098f0 tx=0x7f1780008040 comp rx=0 tx=0).stop 2023-05-22T15:39:36.373 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:36.366+0000 7f1795cb2640 1 -- 172.21.15.16:0/1340901846 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f179015c4f0 msgr2=0x7f179015c960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:36.373 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:36.366+0000 7f1795cb2640 1 --2- 172.21.15.16:0/1340901846 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f179015c4f0 0x7f179015c960 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f17880af040 tx=0x7f17880196a0 comp rx=0 tx=0).stop 2023-05-22T15:39:36.373 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:36.366+0000 7f1795cb2640 1 -- 172.21.15.16:0/1340901846 shutdown_connections 2023-05-22T15:39:36.373 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:36.366+0000 7f1795cb2640 1 --2- 172.21.15.16:0/1340901846 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f179015c4f0 0x7f179015c960 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:36.374 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:36.366+0000 7f1795cb2640 1 --2- 172.21.15.16:0/1340901846 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1790151570 0x7f17901519c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:36.374 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:36.366+0000 7f1795cb2640 1 --2- 172.21.15.16:0/1340901846 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f1770065580 0x7f1770067a30 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:36.374 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:36.366+0000 7f1795cb2640 1 --2- 172.21.15.16:0/1340901846 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1790069720 0x7f179014cfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:36.374 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:36.366+0000 7f1795cb2640 1 -- 172.21.15.16:0/1340901846 >> 172.21.15.16:0/1340901846 conn(0x7f17900b0cc0 msgr2=0x7f17900b2200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:39:36.375 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:36.366+0000 7f1795cb2640 1 -- 172.21.15.16:0/1340901846 shutdown_connections 2023-05-22T15:39:36.375 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:36.366+0000 7f1795cb2640 1 -- 172.21.15.16:0/1340901846 wait complete. 2023-05-22T15:39:36.389 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/dir5 2023-05-22T15:39:36.390 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:39:36.390 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir5/.snap) 2023-05-22T15:39:36.409 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_29_00_UTC 2023-05-22T15:39:36.409 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_30_00_UTC 2023-05-22T15:39:36.410 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_31_01_UTC 2023-05-22T15:39:36.410 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_32_00_UTC 2023-05-22T15:39:36.410 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_33_00_UTC 2023-05-22T15:39:36.410 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_34_00_UTC 2023-05-22T15:39:36.410 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_35_01_UTC 2023-05-22T15:39:36.410 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_36_00_UTC 2023-05-22T15:39:36.410 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_37_00_UTC 2023-05-22T15:39:36.410 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_38_00_UTC 2023-05-22T15:39:36.411 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_39_01_UTC 2023-05-22T15:39:36.411 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2023-05-22-15_29_00_UTC 2023-05-22T15:39:36.411 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:39:36.411 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2023-05-22-15_29_00_UTC) 2023-05-22T15:39:36.469 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2023-05-22-15_30_00_UTC 2023-05-22T15:39:36.469 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:39:36.469 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2023-05-22-15_30_00_UTC) 2023-05-22T15:39:36.529 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2023-05-22-15_31_01_UTC 2023-05-22T15:39:36.529 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:39:36.529 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2023-05-22-15_31_01_UTC) 2023-05-22T15:39:36.585 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2023-05-22-15_32_00_UTC 2023-05-22T15:39:36.585 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:39:36.586 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2023-05-22-15_32_00_UTC) 2023-05-22T15:39:36.653 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2023-05-22-15_33_00_UTC 2023-05-22T15:39:36.653 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:39:36.654 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2023-05-22-15_33_00_UTC) 2023-05-22T15:39:36.735 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2023-05-22-15_34_00_UTC 2023-05-22T15:39:36.735 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:39:36.736 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2023-05-22-15_34_00_UTC) 2023-05-22T15:39:36.795 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2023-05-22-15_35_01_UTC 2023-05-22T15:39:36.796 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:39:36.796 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2023-05-22-15_35_01_UTC) 2023-05-22T15:39:36.858 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2023-05-22-15_36_00_UTC 2023-05-22T15:39:36.859 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:39:36.859 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2023-05-22-15_36_00_UTC) 2023-05-22T15:39:36.919 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2023-05-22-15_37_00_UTC 2023-05-22T15:39:36.920 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:39:36.921 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2023-05-22-15_37_00_UTC) 2023-05-22T15:39:36.979 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2023-05-22-15_38_00_UTC 2023-05-22T15:39:36.980 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:39:36.980 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2023-05-22-15_38_00_UTC) 2023-05-22T15:39:37.040 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2023-05-22-15_39_01_UTC 2023-05-22T15:39:37.040 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:39:37.041 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2023-05-22-15_39_01_UTC) 2023-05-22T15:39:37.100 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:39:37.101 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5) 2023-05-22T15:39:37.153 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove --fs cephfs /snap_test_dir1/dir6 1M 2023-05-22T15:39:37.329 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:37.322+0000 7faf3387d640 1 Processor -- start 2023-05-22T15:39:37.330 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:37.322+0000 7faf3387d640 1 -- start start 2023-05-22T15:39:37.330 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:37.326+0000 7faf3387d640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7faf2c157f70 0x7faf2c0c33f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:37.331 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:37.326+0000 7faf3387d640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7faf2c158340 0x7faf2c0cbc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:37.331 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:37.326+0000 7faf3387d640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf2c0c30e0 con 0x7faf2c15bcf0 2023-05-22T15:39:37.331 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:37.326+0000 7faf3387d640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf2c06f770 con 0x7faf2c158340 2023-05-22T15:39:37.331 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:37.326+0000 7faf3387d640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf2c0c3a40 con 0x7faf2c157f70 2023-05-22T15:39:37.332 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:37.326+0000 7faf30e18640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7faf2c158340 0x7faf2c0cbc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:37.332 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:37.326+0000 7faf31e1a640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7faf2c15bcf0 0x7faf2c06b680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:46172/0 (socket says 172.21.15.16:46172) 2023-05-22T15:39:37.332 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:37.326+0000 7faf31e1a640 1 -- 172.21.15.16:0/1651098908 learned_addr learned my addr 172.21.15.16:0/1651098908 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:39:37.332 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:37.326+0000 7faf31619640 1 --2- 172.21.15.16:0/1651098908 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7faf2c157f70 0x7faf2c0c33f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:37.333 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:37.326+0000 7faf30e18640 1 -- 172.21.15.16:0/1651098908 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7faf2c157f70 msgr2=0x7faf2c0c33f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:39:37.333 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:37.326+0000 7faf30e18640 1 --2- 172.21.15.16:0/1651098908 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7faf2c157f70 0x7faf2c0c33f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:37.333 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:37.326+0000 7faf30e18640 1 -- 172.21.15.16:0/1651098908 >> v1:172.21.15.16:6789/0 conn(0x7faf2c15bcf0 legacy=0x7faf2c06b680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:37.333 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:37.326+0000 7faf2bfff640 1 -- 172.21.15.16:0/1651098908 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2609107394 0 0) 0x7faf2c0c30e0 con 0x7faf2c15bcf0 2023-05-22T15:39:37.333 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:37.326+0000 7faf30e18640 1 -- 172.21.15.16:0/1651098908 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf2c0c3d20 con 0x7faf2c158340 2023-05-22T15:39:37.334 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:37.326+0000 7faf30e18640 1 --2- 172.21.15.16:0/1651098908 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faf2c158340 0x7faf2c0cbc70 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7faf20009dc0 tx=0x7faf2001c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1535fd09c2179a4d server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:39:37.334 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:37.326+0000 7faf2bfff640 1 -- 172.21.15.16:0/1651098908 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf200a59c0 con 0x7faf2c158340 2023-05-22T15:39:37.335 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:37.326+0000 7faf2bfff640 1 -- 172.21.15.16:0/1651098908 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf200a5b20 con 0x7faf2c158340 2023-05-22T15:39:37.335 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:37.326+0000 7faf2bfff640 1 -- 172.21.15.16:0/1651098908 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf200ad440 con 0x7faf2c158340 2023-05-22T15:39:37.335 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:37.330+0000 7faf3387d640 1 -- 172.21.15.16:0/1651098908 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faf2c158340 msgr2=0x7faf2c0cbc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:37.335 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:37.330+0000 7faf3387d640 1 --2- 172.21.15.16:0/1651098908 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faf2c158340 0x7faf2c0cbc70 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7faf20009dc0 tx=0x7faf2001c900 comp rx=0 tx=0).stop 2023-05-22T15:39:37.335 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:37.330+0000 7faf3387d640 1 -- 172.21.15.16:0/1651098908 shutdown_connections 2023-05-22T15:39:37.335 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:37.330+0000 7faf3387d640 1 --2- 172.21.15.16:0/1651098908 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faf2c158340 0x7faf2c0cbc70 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:37.335 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:37.330+0000 7faf3387d640 1 --2- 172.21.15.16:0/1651098908 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7faf2c157f70 0x7faf2c0c33f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:37.335 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:37.330+0000 7faf3387d640 1 -- 172.21.15.16:0/1651098908 >> 172.21.15.16:0/1651098908 conn(0x7faf2c069720 msgr2=0x7faf2c069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:39:37.336 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:37.330+0000 7faf3387d640 1 -- 172.21.15.16:0/1651098908 shutdown_connections 2023-05-22T15:39:37.336 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:37.330+0000 7faf3387d640 1 -- 172.21.15.16:0/1651098908 wait complete. 2023-05-22T15:39:37.337 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:37.330+0000 7faf3387d640 1 Processor -- start 2023-05-22T15:39:37.337 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:37.330+0000 7faf3387d640 1 -- start start 2023-05-22T15:39:37.338 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:37.330+0000 7faf3387d640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7faf2c157f70 0x7faf2c0cb0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:37.338 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:37.330+0000 7faf3387d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faf2c158340 0x7faf2c0ce040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:37.338 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:37.330+0000 7faf3387d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faf2c15bcf0 0x7faf2c1426e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:37.338 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:37.330+0000 7faf3387d640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf2c159830 con 0x7faf2c158340 2023-05-22T15:39:37.338 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:37.330+0000 7faf3387d640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf2c0cb610 con 0x7faf2c15bcf0 2023-05-22T15:39:37.338 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:37.330+0000 7faf3387d640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf2c152780 con 0x7faf2c157f70 2023-05-22T15:39:37.339 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:37.330+0000 7faf30e18640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faf2c158340 0x7faf2c0ce040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:37.339 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:37.330+0000 7faf30e18640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faf2c158340 0x7faf2c0ce040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53452/0 (socket says 172.21.15.16:53452) 2023-05-22T15:39:37.339 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:37.330+0000 7faf30e18640 1 -- 172.21.15.16:0/1716170810 learned_addr learned my addr 172.21.15.16:0/1716170810 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:39:37.339 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:37.330+0000 7faf31619640 1 --2- 172.21.15.16:0/1716170810 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7faf2c157f70 0x7faf2c0cb0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:37.339 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:37.330+0000 7faf31e1a640 1 --2- 172.21.15.16:0/1716170810 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faf2c15bcf0 0x7faf2c1426e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:37.339 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:37.330+0000 7faf30e18640 1 -- 172.21.15.16:0/1716170810 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7faf2c157f70 msgr2=0x7faf2c0cb0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:37.339 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:37.330+0000 7faf30e18640 1 --2- 172.21.15.16:0/1716170810 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7faf2c157f70 0x7faf2c0cb0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:37.340 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:37.330+0000 7faf30e18640 1 -- 172.21.15.16:0/1716170810 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faf2c15bcf0 msgr2=0x7faf2c1426e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:37.340 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:37.330+0000 7faf30e18640 1 --2- 172.21.15.16:0/1716170810 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faf2c15bcf0 0x7faf2c1426e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:37.340 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:37.330+0000 7faf30e18640 1 -- 172.21.15.16:0/1716170810 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf20009a70 con 0x7faf2c158340 2023-05-22T15:39:37.340 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:37.330+0000 7faf30e18640 1 --2- 172.21.15.16:0/1716170810 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faf2c158340 0x7faf2c0ce040 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7faf200b0040 tx=0x7faf200196a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:39:37.340 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:37.330+0000 7faf29ffb640 1 -- 172.21.15.16:0/1716170810 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf200a59c0 con 0x7faf2c158340 2023-05-22T15:39:37.341 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:37.330+0000 7faf29ffb640 1 -- 172.21.15.16:0/1716170810 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf20009450 con 0x7faf2c158340 2023-05-22T15:39:37.341 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:37.330+0000 7faf3387d640 1 -- 172.21.15.16:0/1716170810 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf2c165090 con 0x7faf2c158340 2023-05-22T15:39:37.341 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:37.330+0000 7faf29ffb640 1 -- 172.21.15.16:0/1716170810 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf200b9690 con 0x7faf2c158340 2023-05-22T15:39:37.341 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:37.330+0000 7faf3387d640 1 -- 172.21.15.16:0/1716170810 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf2c0cc5c0 con 0x7faf2c158340 2023-05-22T15:39:37.342 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:37.334+0000 7faf3387d640 1 -- 172.21.15.16:0/1716170810 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf2c165220 con 0x7faf2c158340 2023-05-22T15:39:37.342 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:37.334+0000 7faf29ffb640 1 -- 172.21.15.16:0/1716170810 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 12) v1 ==== 84418+0+0 (secure 0 0 0) 0x7faf200b97f0 con 0x7faf2c158340 2023-05-22T15:39:37.342 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:37.334+0000 7faf29ffb640 1 --2- 172.21.15.16:0/1716170810 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7faf18065580 0x7faf18067a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:37.342 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:37.334+0000 7faf29ffb640 1 -- 172.21.15.16:0/1716170810 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 7346+0+0 (secure 0 0 0) 0x7faf2011eab0 con 0x7faf2c158340 2023-05-22T15:39:37.342 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:37.334+0000 7faf31619640 1 --2- 172.21.15.16:0/1716170810 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7faf18065580 0x7faf18067a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:37.342 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:37.334+0000 7faf31619640 1 --2- 172.21.15.16:0/1716170810 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7faf18065580 0x7faf18067a30 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7faf1c00b9d0 tx=0x7faf1c008040 comp rx=0 tx=0).ready entity=mgr.5823 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:39:37.343 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:37.338+0000 7faf29ffb640 1 -- 172.21.15.16:0/1716170810 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7faf200ebf00 con 0x7faf2c158340 2023-05-22T15:39:37.581 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:37.574+0000 7faf3387d640 1 -- 172.21.15.16:0/1716170810 --> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "fs": "cephfs", "path": "/snap_test_dir1/dir6", "repeat": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7faf2c04dbe0 con 0x7faf18065580 2023-05-22T15:39:37.725 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:37.718+0000 7faf29ffb640 1 -- 172.21.15.16:0/1716170810 <== mgr.5823 v2:172.21.15.16:6832/36921 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+47 (secure 0 0 0) 0x7faf2c04dbe0 con 0x7faf18065580 2023-05-22T15:39:37.726 INFO:teuthology.orchestra.run.smithi016.stdout:Schedule removed for path /snap_test_dir1/dir6 2023-05-22T15:39:37.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:37.722+0000 7faf3387d640 1 -- 172.21.15.16:0/1716170810 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7faf18065580 msgr2=0x7faf18067a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:37.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:37.722+0000 7faf3387d640 1 --2- 172.21.15.16:0/1716170810 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7faf18065580 0x7faf18067a30 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7faf1c00b9d0 tx=0x7faf1c008040 comp rx=0 tx=0).stop 2023-05-22T15:39:37.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:37.722+0000 7faf3387d640 1 -- 172.21.15.16:0/1716170810 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faf2c158340 msgr2=0x7faf2c0ce040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:37.729 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:37.722+0000 7faf3387d640 1 --2- 172.21.15.16:0/1716170810 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faf2c158340 0x7faf2c0ce040 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7faf200b0040 tx=0x7faf200196a0 comp rx=0 tx=0).stop 2023-05-22T15:39:37.729 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:37.722+0000 7faf3387d640 1 -- 172.21.15.16:0/1716170810 shutdown_connections 2023-05-22T15:39:37.729 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:37.722+0000 7faf3387d640 1 --2- 172.21.15.16:0/1716170810 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faf2c15bcf0 0x7faf2c1426e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:37.729 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:37.722+0000 7faf3387d640 1 --2- 172.21.15.16:0/1716170810 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faf2c158340 0x7faf2c0ce040 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:37.729 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:37.722+0000 7faf3387d640 1 --2- 172.21.15.16:0/1716170810 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7faf18065580 0x7faf18067a30 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:37.730 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:37.722+0000 7faf3387d640 1 --2- 172.21.15.16:0/1716170810 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7faf2c157f70 0x7faf2c0cb0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:37.730 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:37.722+0000 7faf3387d640 1 -- 172.21.15.16:0/1716170810 >> 172.21.15.16:0/1716170810 conn(0x7faf2c069720 msgr2=0x7faf2c152d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:39:37.730 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:37.722+0000 7faf3387d640 1 -- 172.21.15.16:0/1716170810 shutdown_connections 2023-05-22T15:39:37.730 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:37.722+0000 7faf3387d640 1 -- 172.21.15.16:0/1716170810 wait complete. 2023-05-22T15:39:37.750 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/dir6 2023-05-22T15:39:37.751 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:39:37.751 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir6/.snap) 2023-05-22T15:39:37.766 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_29_00_UTC 2023-05-22T15:39:37.766 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_30_01_UTC 2023-05-22T15:39:37.767 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_31_00_UTC 2023-05-22T15:39:37.767 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_32_01_UTC 2023-05-22T15:39:37.767 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_33_00_UTC 2023-05-22T15:39:37.767 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_34_00_UTC 2023-05-22T15:39:37.768 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_35_00_UTC 2023-05-22T15:39:37.768 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_36_00_UTC 2023-05-22T15:39:37.768 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_37_00_UTC 2023-05-22T15:39:37.768 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_38_00_UTC 2023-05-22T15:39:37.768 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_39_00_UTC 2023-05-22T15:39:37.769 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2023-05-22-15_29_00_UTC 2023-05-22T15:39:37.769 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:39:37.769 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2023-05-22-15_29_00_UTC) 2023-05-22T15:39:37.834 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2023-05-22-15_30_01_UTC 2023-05-22T15:39:37.834 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:39:37.835 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2023-05-22-15_30_01_UTC) 2023-05-22T15:39:37.893 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2023-05-22-15_31_00_UTC 2023-05-22T15:39:37.893 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:39:37.894 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2023-05-22-15_31_00_UTC) 2023-05-22T15:39:37.952 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2023-05-22-15_32_01_UTC 2023-05-22T15:39:37.953 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:39:37.953 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2023-05-22-15_32_01_UTC) 2023-05-22T15:39:38.012 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2023-05-22-15_33_00_UTC 2023-05-22T15:39:38.012 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:39:38.013 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2023-05-22-15_33_00_UTC) 2023-05-22T15:39:38.074 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2023-05-22-15_34_00_UTC 2023-05-22T15:39:38.074 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:39:38.075 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2023-05-22-15_34_00_UTC) 2023-05-22T15:39:38.141 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2023-05-22-15_35_00_UTC 2023-05-22T15:39:38.142 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:39:38.142 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2023-05-22-15_35_00_UTC) 2023-05-22T15:39:38.204 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2023-05-22-15_36_00_UTC 2023-05-22T15:39:38.205 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:39:38.205 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2023-05-22-15_36_00_UTC) 2023-05-22T15:39:38.263 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2023-05-22-15_37_00_UTC 2023-05-22T15:39:38.263 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:39:38.264 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2023-05-22-15_37_00_UTC) 2023-05-22T15:39:38.325 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2023-05-22-15_38_00_UTC 2023-05-22T15:39:38.326 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:39:38.326 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2023-05-22-15_38_00_UTC) 2023-05-22T15:39:38.385 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2023-05-22-15_39_00_UTC 2023-05-22T15:39:38.386 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:39:38.386 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2023-05-22-15_39_00_UTC) 2023-05-22T15:39:38.460 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:39:38.461 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6) 2023-05-22T15:39:38.515 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove --fs cephfs /snap_test_dir1/dir7 1M 2023-05-22T15:39:38.714 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:38.706+0000 7f2581dba640 1 Processor -- start 2023-05-22T15:39:38.714 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:38.706+0000 7f2581dba640 1 -- start start 2023-05-22T15:39:38.716 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:38.710+0000 7f2581dba640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f257c15c8e0 0x7f257c0ccc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:38.716 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:38.710+0000 7f2581dba640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f257c0d2a20 con 0x7f257c0d2eb0 2023-05-22T15:39:38.716 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:38.710+0000 7f2581dba640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f257c0cd230 con 0x7f257c0cef10 2023-05-22T15:39:38.716 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:38.710+0000 7f2581dba640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f257c0d23d0 con 0x7f257c15c8e0 2023-05-22T15:39:38.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:38.710+0000 7f2573fff640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f257c0d2eb0 0x7f257c15ccb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:46174/0 (socket says 172.21.15.16:46174) 2023-05-22T15:39:38.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:38.710+0000 7f2573fff640 1 -- 172.21.15.16:0/2563887371 learned_addr learned my addr 172.21.15.16:0/2563887371 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:39:38.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:38.710+0000 7f257b7fe640 1 --2- 172.21.15.16:0/2563887371 >> v2:172.21.15.62:3300/0 conn(0x7f257c15c8e0 0x7f257c0ccc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:38.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:38.710+0000 7f257affd640 1 -- 172.21.15.16:0/2563887371 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2085392008 0 0) 0x7f257c0d2a20 con 0x7f257c0d2eb0 2023-05-22T15:39:38.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:38.710+0000 7f257affd640 1 -- 172.21.15.16:0/2563887371 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2568003530 con 0x7f257c0d2eb0 2023-05-22T15:39:38.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:38.710+0000 7f257b7fe640 1 -- 172.21.15.16:0/2563887371 >> v1:172.21.15.62:6789/0 conn(0x7f257c0cef10 legacy=0x7f257c0cf310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:38.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:38.710+0000 7f257b7fe640 1 -- 172.21.15.16:0/2563887371 >> v1:172.21.15.16:6789/0 conn(0x7f257c0d2eb0 legacy=0x7f257c15ccb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:38.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:38.710+0000 7f257b7fe640 1 -- 172.21.15.16:0/2563887371 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f257c0d26b0 con 0x7f257c15c8e0 2023-05-22T15:39:38.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:38.710+0000 7f257b7fe640 1 --2- 172.21.15.16:0/2563887371 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f257c15c8e0 0x7f257c0ccc50 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f256c0097c0 tx=0x7f256c01c3f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=743e38baf105d689 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:39:38.719 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:38.714+0000 7f257affd640 1 -- 172.21.15.16:0/2563887371 <== mon.1 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f256c0ab610 con 0x7f257c15c8e0 2023-05-22T15:39:38.719 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:38.714+0000 7f257affd640 1 -- 172.21.15.16:0/2563887371 <== mon.1 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f256c0ab770 con 0x7f257c15c8e0 2023-05-22T15:39:38.719 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:38.714+0000 7f257affd640 1 -- 172.21.15.16:0/2563887371 <== mon.1 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f256c0abb00 con 0x7f257c15c8e0 2023-05-22T15:39:38.719 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:38.714+0000 7f2581dba640 1 -- 172.21.15.16:0/2563887371 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f257c15c8e0 msgr2=0x7f257c0ccc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:38.720 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:38.714+0000 7f2581dba640 1 --2- 172.21.15.16:0/2563887371 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f257c15c8e0 0x7f257c0ccc50 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f256c0097c0 tx=0x7f256c01c3f0 comp rx=0 tx=0).stop 2023-05-22T15:39:38.720 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:38.714+0000 7f2581dba640 1 -- 172.21.15.16:0/2563887371 shutdown_connections 2023-05-22T15:39:38.720 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:38.714+0000 7f2581dba640 1 --2- 172.21.15.16:0/2563887371 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f257c15c8e0 0x7f257c0ccc50 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:38.720 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:38.714+0000 7f2581dba640 1 -- 172.21.15.16:0/2563887371 >> 172.21.15.16:0/2563887371 conn(0x7f257c070e50 msgr2=0x7f257c071250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:39:38.720 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:38.714+0000 7f2581dba640 1 -- 172.21.15.16:0/2563887371 shutdown_connections 2023-05-22T15:39:38.720 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:38.714+0000 7f2581dba640 1 -- 172.21.15.16:0/2563887371 wait complete. 2023-05-22T15:39:38.721 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:38.714+0000 7f2581dba640 1 Processor -- start 2023-05-22T15:39:38.722 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:38.714+0000 7f2581dba640 1 -- start start 2023-05-22T15:39:38.722 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:38.718+0000 7f2581dba640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f257c0cef10 0x7f257c0c96d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:38.723 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:38.718+0000 7f2581dba640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f257c0d2eb0 0x7f257c0cb0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:38.723 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:38.718+0000 7f2581dba640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f257c0cb620 0x7f257c14fb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:38.723 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:38.718+0000 7f2581dba640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f257c0ca040 con 0x7f257c0cb620 2023-05-22T15:39:38.723 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:38.718+0000 7f2581dba640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f257c0ca320 con 0x7f257c0cef10 2023-05-22T15:39:38.723 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:38.718+0000 7f2581dba640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f257c0ca600 con 0x7f257c0d2eb0 2023-05-22T15:39:38.724 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:38.718+0000 7f257bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f257c0cb620 0x7f257c14fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:38.724 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:38.718+0000 7f2573fff640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f257c0d2eb0 0x7f257c0cb0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:38.725 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:38.718+0000 7f257bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f257c0cb620 0x7f257c14fb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53468/0 (socket says 172.21.15.16:53468) 2023-05-22T15:39:38.725 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:38.718+0000 7f257bfff640 1 -- 172.21.15.16:0/1507859633 learned_addr learned my addr 172.21.15.16:0/1507859633 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:39:38.725 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:38.718+0000 7f257b7fe640 1 --2- 172.21.15.16:0/1507859633 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f257c0cef10 0x7f257c0c96d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:38.725 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:38.718+0000 7f2573fff640 1 -- 172.21.15.16:0/1507859633 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f257c0cef10 msgr2=0x7f257c0c96d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:38.726 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:38.718+0000 7f2573fff640 1 --2- 172.21.15.16:0/1507859633 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f257c0cef10 0x7f257c0c96d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:38.726 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:38.718+0000 7f2573fff640 1 -- 172.21.15.16:0/1507859633 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f257c0cb620 msgr2=0x7f257c14fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:38.726 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:38.718+0000 7f2573fff640 1 --2- 172.21.15.16:0/1507859633 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f257c0cb620 0x7f257c14fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:38.726 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:38.718+0000 7f2573fff640 1 -- 172.21.15.16:0/1507859633 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f256c009420 con 0x7f257c0d2eb0 2023-05-22T15:39:38.726 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:38.718+0000 7f257b7fe640 1 --2- 172.21.15.16:0/1507859633 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f257c0cef10 0x7f257c0c96d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-05-22T15:39:38.726 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:38.718+0000 7f257bfff640 1 --2- 172.21.15.16:0/1507859633 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f257c0cb620 0x7f257c14fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-05-22T15:39:38.726 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:38.718+0000 7f2573fff640 1 --2- 172.21.15.16:0/1507859633 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f257c0d2eb0 0x7f257c0cb0e0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f2564001ec0 tx=0x7f256400c820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:39:38.727 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:38.718+0000 7f2578ff9640 1 -- 172.21.15.16:0/1507859633 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2564005cf0 con 0x7f257c0d2eb0 2023-05-22T15:39:38.727 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:38.718+0000 7f2578ff9640 1 -- 172.21.15.16:0/1507859633 <== mon.2 v2:172.21.15.62:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2564005e50 con 0x7f257c0d2eb0 2023-05-22T15:39:38.727 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:38.718+0000 7f2581dba640 1 -- 172.21.15.16:0/1507859633 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f257c0d0d80 con 0x7f257c0d2eb0 2023-05-22T15:39:38.727 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:38.718+0000 7f2578ff9640 1 -- 172.21.15.16:0/1507859633 <== mon.2 v2:172.21.15.62:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25640072d0 con 0x7f257c0d2eb0 2023-05-22T15:39:38.727 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:38.718+0000 7f2581dba640 1 -- 172.21.15.16:0/1507859633 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f257c0d0bf0 con 0x7f257c0d2eb0 2023-05-22T15:39:38.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:38.718+0000 7f2581dba640 1 -- 172.21.15.16:0/1507859633 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f257c0d0f10 con 0x7f257c0d2eb0 2023-05-22T15:39:38.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:38.722+0000 7f2578ff9640 1 -- 172.21.15.16:0/1507859633 <== mon.2 v2:172.21.15.62:3301/0 4 ==== mgrmap(e 12) v1 ==== 84418+0+0 (secure 0 0 0) 0x7f2564007430 con 0x7f257c0d2eb0 2023-05-22T15:39:38.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:38.722+0000 7f2578ff9640 1 --2- 172.21.15.16:0/1507859633 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f255c065580 0x7f255c067a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:38.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:38.722+0000 7f2578ff9640 1 -- 172.21.15.16:0/1507859633 <== mon.2 v2:172.21.15.62:3301/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 7346+0+0 (secure 0 0 0) 0x7f2564082c60 con 0x7f257c0d2eb0 2023-05-22T15:39:38.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:38.722+0000 7f257b7fe640 1 --2- 172.21.15.16:0/1507859633 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f255c065580 0x7f255c067a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:38.729 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:38.722+0000 7f257b7fe640 1 --2- 172.21.15.16:0/1507859633 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f255c065580 0x7f255c067a30 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f256c0af000 tx=0x7f256c0a7000 comp rx=0 tx=0).ready entity=mgr.5823 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:39:38.733 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:38.726+0000 7f2578ff9640 1 -- 172.21.15.16:0/1507859633 <== mon.2 v2:172.21.15.62:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f2564050390 con 0x7f257c0d2eb0 2023-05-22T15:39:38.930 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:38.922+0000 7f2581dba640 1 -- 172.21.15.16:0/1507859633 --> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "fs": "cephfs", "path": "/snap_test_dir1/dir7", "repeat": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f257c0ca740 con 0x7f255c065580 2023-05-22T15:39:39.086 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:39.082+0000 7f2578ff9640 1 -- 172.21.15.16:0/1507859633 <== mgr.5823 v2:172.21.15.16:6832/36921 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+47 (secure 0 0 0) 0x7f257c0ca740 con 0x7f255c065580 2023-05-22T15:39:39.087 INFO:teuthology.orchestra.run.smithi016.stdout:Schedule removed for path /snap_test_dir1/dir7 2023-05-22T15:39:39.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:39.086+0000 7f2581dba640 1 -- 172.21.15.16:0/1507859633 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f255c065580 msgr2=0x7f255c067a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:39.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:39.086+0000 7f2581dba640 1 --2- 172.21.15.16:0/1507859633 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f255c065580 0x7f255c067a30 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f256c0af000 tx=0x7f256c0a7000 comp rx=0 tx=0).stop 2023-05-22T15:39:39.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:39.086+0000 7f2581dba640 1 -- 172.21.15.16:0/1507859633 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f257c0d2eb0 msgr2=0x7f257c0cb0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:39.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:39.086+0000 7f2581dba640 1 --2- 172.21.15.16:0/1507859633 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f257c0d2eb0 0x7f257c0cb0e0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f2564001ec0 tx=0x7f256400c820 comp rx=0 tx=0).stop 2023-05-22T15:39:39.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:39.086+0000 7f2581dba640 1 -- 172.21.15.16:0/1507859633 shutdown_connections 2023-05-22T15:39:39.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:39.086+0000 7f2581dba640 1 --2- 172.21.15.16:0/1507859633 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f257c0cb620 0x7f257c14fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:39.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:39.086+0000 7f2581dba640 1 --2- 172.21.15.16:0/1507859633 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f255c065580 0x7f255c067a30 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:39.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:39.086+0000 7f2581dba640 1 --2- 172.21.15.16:0/1507859633 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f257c0d2eb0 0x7f257c0cb0e0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:39.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:39.086+0000 7f2581dba640 1 --2- 172.21.15.16:0/1507859633 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f257c0cef10 0x7f257c0c96d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:39.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:39.086+0000 7f2581dba640 1 -- 172.21.15.16:0/1507859633 >> 172.21.15.16:0/1507859633 conn(0x7f257c070e50 msgr2=0x7f257c0675a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:39:39.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:39.086+0000 7f2581dba640 1 -- 172.21.15.16:0/1507859633 shutdown_connections 2023-05-22T15:39:39.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:39.086+0000 7f2581dba640 1 -- 172.21.15.16:0/1507859633 wait complete. 2023-05-22T15:39:39.113 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/dir7 2023-05-22T15:39:39.113 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:39:39.114 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir7/.snap) 2023-05-22T15:39:39.133 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_29_00_UTC 2023-05-22T15:39:39.134 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_30_00_UTC 2023-05-22T15:39:39.134 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_31_00_UTC 2023-05-22T15:39:39.134 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_32_01_UTC 2023-05-22T15:39:39.134 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_33_00_UTC 2023-05-22T15:39:39.134 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_34_00_UTC 2023-05-22T15:39:39.135 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_35_00_UTC 2023-05-22T15:39:39.135 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_36_00_UTC 2023-05-22T15:39:39.135 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_37_01_UTC 2023-05-22T15:39:39.135 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_38_00_UTC 2023-05-22T15:39:39.136 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_39_00_UTC 2023-05-22T15:39:39.137 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2023-05-22-15_29_00_UTC 2023-05-22T15:39:39.138 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:39:39.138 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2023-05-22-15_29_00_UTC) 2023-05-22T15:39:39.200 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2023-05-22-15_30_00_UTC 2023-05-22T15:39:39.201 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:39:39.201 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2023-05-22-15_30_00_UTC) 2023-05-22T15:39:39.261 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2023-05-22-15_31_00_UTC 2023-05-22T15:39:39.261 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:39:39.262 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2023-05-22-15_31_00_UTC) 2023-05-22T15:39:39.323 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2023-05-22-15_32_01_UTC 2023-05-22T15:39:39.323 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:39:39.324 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2023-05-22-15_32_01_UTC) 2023-05-22T15:39:39.386 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2023-05-22-15_33_00_UTC 2023-05-22T15:39:39.387 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:39:39.387 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2023-05-22-15_33_00_UTC) 2023-05-22T15:39:39.473 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2023-05-22-15_34_00_UTC 2023-05-22T15:39:39.473 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:39:39.474 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2023-05-22-15_34_00_UTC) 2023-05-22T15:39:39.536 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2023-05-22-15_35_00_UTC 2023-05-22T15:39:39.537 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:39:39.538 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2023-05-22-15_35_00_UTC) 2023-05-22T15:39:39.595 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2023-05-22-15_36_00_UTC 2023-05-22T15:39:39.595 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:39:39.595 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2023-05-22-15_36_00_UTC) 2023-05-22T15:39:39.660 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2023-05-22-15_37_01_UTC 2023-05-22T15:39:39.660 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:39:39.661 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2023-05-22-15_37_01_UTC) 2023-05-22T15:39:39.718 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2023-05-22-15_38_00_UTC 2023-05-22T15:39:39.719 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:39:39.719 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2023-05-22-15_38_00_UTC) 2023-05-22T15:39:39.779 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2023-05-22-15_39_00_UTC 2023-05-22T15:39:39.780 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:39:39.780 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2023-05-22-15_39_00_UTC) 2023-05-22T15:39:39.838 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:39:39.838 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7) 2023-05-22T15:39:39.893 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove --fs cephfs /snap_test_dir1/dir8 1M 2023-05-22T15:39:40.047 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:40.042+0000 7f9da3539640 1 Processor -- start 2023-05-22T15:39:40.047 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:40.042+0000 7f9da3539640 1 -- start start 2023-05-22T15:39:40.048 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:40.042+0000 7f9da3539640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f9d9c0cc4e0 0x7f9d9c06bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:40.049 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:40.042+0000 7f9da3539640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9d9c15d650 0x7f9d9c15dad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:40.049 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:40.042+0000 7f9da3539640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d9c0cc960 con 0x7f9d9c070c10 2023-05-22T15:39:40.049 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:40.042+0000 7f9da3539640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d9c15e010 con 0x7f9d9c0cc4e0 2023-05-22T15:39:40.049 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:40.042+0000 7f9da3539640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d9c0ccbb0 con 0x7f9d9c15d650 2023-05-22T15:39:40.049 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:40.042+0000 7f9da0ad4640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f9d9c0cc4e0 0x7f9d9c06bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:40.049 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:40.042+0000 7f9da0ad4640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f9d9c0cc4e0 0x7f9d9c06bfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52586/0 (socket says 172.21.15.16:52586) 2023-05-22T15:39:40.049 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:40.042+0000 7f9da0ad4640 1 -- 172.21.15.16:0/234982620 learned_addr learned my addr 172.21.15.16:0/234982620 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:39:40.050 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:40.042+0000 7f9da0ad4640 1 -- 172.21.15.16:0/234982620 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9d9c15d650 msgr2=0x7f9d9c15dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:39:40.050 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:40.042+0000 7f9da0ad4640 1 --2- 172.21.15.16:0/234982620 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9d9c15d650 0x7f9d9c15dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:40.050 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:40.042+0000 7f9da0ad4640 1 -- 172.21.15.16:0/234982620 >> v1:172.21.15.62:6789/0 conn(0x7f9d9c070c10 legacy=0x7f9d9c0c9e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:39:40.050 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:40.042+0000 7f9da0ad4640 1 -- 172.21.15.16:0/234982620 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d9c0cd7c0 con 0x7f9d9c0cc4e0 2023-05-22T15:39:40.050 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:40.046+0000 7f9da0ad4640 1 --2- 172.21.15.16:0/234982620 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9d9c0cc4e0 0x7f9d9c06bfa0 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f9d90009870 tx=0x7f9d9001c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=baf7e9436809c4fc server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:39:40.051 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:40.046+0000 7f9d9bfff640 1 -- 172.21.15.16:0/234982620 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d900aa5f0 con 0x7f9d9c0cc4e0 2023-05-22T15:39:40.051 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:40.046+0000 7f9d9bfff640 1 -- 172.21.15.16:0/234982620 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d900aa750 con 0x7f9d9c0cc4e0 2023-05-22T15:39:40.051 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:40.046+0000 7f9d9bfff640 1 -- 172.21.15.16:0/234982620 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d900aaae0 con 0x7f9d9c0cc4e0 2023-05-22T15:39:40.052 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:40.046+0000 7f9da3539640 1 -- 172.21.15.16:0/234982620 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9d9c0cc4e0 msgr2=0x7f9d9c06bfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:40.052 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:40.046+0000 7f9da3539640 1 --2- 172.21.15.16:0/234982620 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9d9c0cc4e0 0x7f9d9c06bfa0 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f9d90009870 tx=0x7f9d9001c4d0 comp rx=0 tx=0).stop 2023-05-22T15:39:40.052 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:40.046+0000 7f9da3539640 1 -- 172.21.15.16:0/234982620 shutdown_connections 2023-05-22T15:39:40.053 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:40.046+0000 7f9da3539640 1 --2- 172.21.15.16:0/234982620 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9d9c15d650 0x7f9d9c15dad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:40.053 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:40.046+0000 7f9da3539640 1 --2- 172.21.15.16:0/234982620 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9d9c0cc4e0 0x7f9d9c06bfa0 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:40.053 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:40.046+0000 7f9da3539640 1 -- 172.21.15.16:0/234982620 >> 172.21.15.16:0/234982620 conn(0x7f9d9c069720 msgr2=0x7f9d9c069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:39:40.053 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:40.046+0000 7f9da3539640 1 -- 172.21.15.16:0/234982620 shutdown_connections 2023-05-22T15:39:40.054 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:40.046+0000 7f9da3539640 1 -- 172.21.15.16:0/234982620 wait complete. 2023-05-22T15:39:40.056 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:40.046+0000 7f9da3539640 1 Processor -- start 2023-05-22T15:39:40.056 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:40.046+0000 7f9da3539640 1 -- start start 2023-05-22T15:39:40.056 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:40.046+0000 7f9da3539640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9d9c070c10 0x7f9d9c13e440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:40.056 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:40.046+0000 7f9da3539640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9d9c0cc4e0 0x7f9d9c147150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:40.056 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:40.046+0000 7f9da3539640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9d9c15d650 0x7f9d9c0c77a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:40.056 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:40.046+0000 7f9da3539640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d9c0c7380 con 0x7f9d9c070c10 2023-05-22T15:39:40.057 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:40.046+0000 7f9da3539640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d9c1464b0 con 0x7f9d9c0cc4e0 2023-05-22T15:39:40.057 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:40.046+0000 7f9da3539640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d9c13e980 con 0x7f9d9c15d650 2023-05-22T15:39:40.057 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:40.046+0000 7f9da12d5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9d9c070c10 0x7f9d9c13e440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:40.057 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:40.046+0000 7f9da12d5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9d9c070c10 0x7f9d9c13e440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52596/0 (socket says 172.21.15.16:52596) 2023-05-22T15:39:40.057 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:40.046+0000 7f9da12d5640 1 -- 172.21.15.16:0/1635740721 learned_addr learned my addr 172.21.15.16:0/1635740721 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:39:40.057 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:40.046+0000 7f9da1ad6640 1 --2- 172.21.15.16:0/1635740721 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9d9c15d650 0x7f9d9c0c77a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:40.057 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:40.046+0000 7f9da0ad4640 1 --2- 172.21.15.16:0/1635740721 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9d9c0cc4e0 0x7f9d9c147150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:40.057 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:40.046+0000 7f9da12d5640 1 -- 172.21.15.16:0/1635740721 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9d9c15d650 msgr2=0x7f9d9c0c77a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:40.058 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:40.046+0000 7f9da12d5640 1 --2- 172.21.15.16:0/1635740721 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9d9c15d650 0x7f9d9c0c77a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:40.058 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:40.046+0000 7f9da12d5640 1 -- 172.21.15.16:0/1635740721 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9d9c0cc4e0 msgr2=0x7f9d9c147150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:40.058 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:40.046+0000 7f9da12d5640 1 --2- 172.21.15.16:0/1635740721 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9d9c0cc4e0 0x7f9d9c147150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:40.058 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:40.046+0000 7f9da12d5640 1 -- 172.21.15.16:0/1635740721 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d900094d0 con 0x7f9d9c070c10 2023-05-22T15:39:40.058 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:40.046+0000 7f9da12d5640 1 --2- 172.21.15.16:0/1635740721 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9d9c070c10 0x7f9d9c13e440 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f9d8c00b3e0 tx=0x7f9d8c00b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:39:40.058 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:40.050+0000 7f9d99ffb640 1 -- 172.21.15.16:0/1635740721 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d8c011830 con 0x7f9d9c070c10 2023-05-22T15:39:40.058 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:40.050+0000 7f9d99ffb640 1 -- 172.21.15.16:0/1635740721 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d8c0119b0 con 0x7f9d9c070c10 2023-05-22T15:39:40.059 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:40.050+0000 7f9da3539640 1 -- 172.21.15.16:0/1635740721 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d9c06c4e0 con 0x7f9d9c070c10 2023-05-22T15:39:40.059 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:40.050+0000 7f9d99ffb640 1 -- 172.21.15.16:0/1635740721 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d8c011d60 con 0x7f9d9c070c10 2023-05-22T15:39:40.059 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:40.050+0000 7f9da3539640 1 -- 172.21.15.16:0/1635740721 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d9c070fe0 con 0x7f9d9c070c10 2023-05-22T15:39:40.059 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:40.050+0000 7f9da3539640 1 -- 172.21.15.16:0/1635740721 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d9c0cce90 con 0x7f9d9c070c10 2023-05-22T15:39:40.059 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:40.050+0000 7f9d99ffb640 1 -- 172.21.15.16:0/1635740721 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 12) v1 ==== 84418+0+0 (secure 0 0 0) 0x7f9d8c01a400 con 0x7f9d9c070c10 2023-05-22T15:39:40.059 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:40.050+0000 7f9d99ffb640 1 --2- 172.21.15.16:0/1635740721 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f9d88065580 0x7f9d88067a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:40.059 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:40.050+0000 7f9d99ffb640 1 -- 172.21.15.16:0/1635740721 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 7330+0+0 (secure 0 0 0) 0x7f9d8c083210 con 0x7f9d9c070c10 2023-05-22T15:39:40.059 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:40.050+0000 7f9da0ad4640 1 --2- 172.21.15.16:0/1635740721 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f9d88065580 0x7f9d88067a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:40.060 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:40.050+0000 7f9da0ad4640 1 --2- 172.21.15.16:0/1635740721 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f9d88065580 0x7f9d88067a30 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f9d900ae000 tx=0x7f9d90008040 comp rx=0 tx=0).ready entity=mgr.5823 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:39:40.060 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:40.054+0000 7f9d99ffb640 1 -- 172.21.15.16:0/1635740721 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f9d8c050740 con 0x7f9d9c070c10 2023-05-22T15:39:40.235 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:40.230+0000 7f9da3539640 1 -- 172.21.15.16:0/1635740721 --> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "fs": "cephfs", "path": "/snap_test_dir1/dir8", "repeat": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f9d9c0c75f0 con 0x7f9d88065580 2023-05-22T15:39:40.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:40.458+0000 7f9d99ffb640 1 -- 172.21.15.16:0/1635740721 <== mgr.5823 v2:172.21.15.16:6832/36921 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+47 (secure 0 0 0) 0x7f9d9c0c75f0 con 0x7f9d88065580 2023-05-22T15:39:40.464 INFO:teuthology.orchestra.run.smithi016.stdout:Schedule removed for path /snap_test_dir1/dir8 2023-05-22T15:39:40.467 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:40.462+0000 7f9da3539640 1 -- 172.21.15.16:0/1635740721 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f9d88065580 msgr2=0x7f9d88067a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:40.467 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:40.462+0000 7f9da3539640 1 --2- 172.21.15.16:0/1635740721 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f9d88065580 0x7f9d88067a30 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f9d900ae000 tx=0x7f9d90008040 comp rx=0 tx=0).stop 2023-05-22T15:39:40.467 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:40.462+0000 7f9da3539640 1 -- 172.21.15.16:0/1635740721 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9d9c070c10 msgr2=0x7f9d9c13e440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:40.467 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:40.462+0000 7f9da3539640 1 --2- 172.21.15.16:0/1635740721 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9d9c070c10 0x7f9d9c13e440 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f9d8c00b3e0 tx=0x7f9d8c00b8a0 comp rx=0 tx=0).stop 2023-05-22T15:39:40.468 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:40.462+0000 7f9da3539640 1 -- 172.21.15.16:0/1635740721 shutdown_connections 2023-05-22T15:39:40.468 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:40.462+0000 7f9da3539640 1 --2- 172.21.15.16:0/1635740721 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f9d88065580 0x7f9d88067a30 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:40.468 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:40.462+0000 7f9da3539640 1 --2- 172.21.15.16:0/1635740721 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9d9c15d650 0x7f9d9c0c77a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:40.468 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:40.462+0000 7f9da3539640 1 --2- 172.21.15.16:0/1635740721 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9d9c0cc4e0 0x7f9d9c147150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:40.468 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:40.462+0000 7f9da3539640 1 --2- 172.21.15.16:0/1635740721 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9d9c070c10 0x7f9d9c13e440 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:40.469 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:40.462+0000 7f9da3539640 1 -- 172.21.15.16:0/1635740721 >> 172.21.15.16:0/1635740721 conn(0x7f9d9c069720 msgr2=0x7f9d9c15a860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:39:40.469 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:40.462+0000 7f9da3539640 1 -- 172.21.15.16:0/1635740721 shutdown_connections 2023-05-22T15:39:40.469 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:40.462+0000 7f9da3539640 1 -- 172.21.15.16:0/1635740721 wait complete. 2023-05-22T15:39:40.492 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/dir8 2023-05-22T15:39:40.492 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:39:40.493 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir8/.snap) 2023-05-22T15:39:40.511 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_29_00_UTC 2023-05-22T15:39:40.511 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_30_00_UTC 2023-05-22T15:39:40.511 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_31_00_UTC 2023-05-22T15:39:40.511 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_32_00_UTC 2023-05-22T15:39:40.512 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_33_00_UTC 2023-05-22T15:39:40.512 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_34_01_UTC 2023-05-22T15:39:40.512 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_35_00_UTC 2023-05-22T15:39:40.512 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_36_00_UTC 2023-05-22T15:39:40.512 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_37_00_UTC 2023-05-22T15:39:40.512 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_38_01_UTC 2023-05-22T15:39:40.513 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_39_01_UTC 2023-05-22T15:39:40.515 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2023-05-22-15_29_00_UTC 2023-05-22T15:39:40.515 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:39:40.516 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2023-05-22-15_29_00_UTC) 2023-05-22T15:39:40.577 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2023-05-22-15_30_00_UTC 2023-05-22T15:39:40.577 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:39:40.577 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2023-05-22-15_30_00_UTC) 2023-05-22T15:39:40.640 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2023-05-22-15_31_00_UTC 2023-05-22T15:39:40.640 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:39:40.641 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2023-05-22-15_31_00_UTC) 2023-05-22T15:39:40.710 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2023-05-22-15_32_00_UTC 2023-05-22T15:39:40.711 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:39:40.711 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2023-05-22-15_32_00_UTC) 2023-05-22T15:39:40.778 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2023-05-22-15_33_00_UTC 2023-05-22T15:39:40.778 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:39:40.778 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2023-05-22-15_33_00_UTC) 2023-05-22T15:39:40.841 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2023-05-22-15_34_01_UTC 2023-05-22T15:39:40.842 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:39:40.842 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2023-05-22-15_34_01_UTC) 2023-05-22T15:39:40.901 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2023-05-22-15_35_00_UTC 2023-05-22T15:39:40.902 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:39:40.902 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2023-05-22-15_35_00_UTC) 2023-05-22T15:39:40.961 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2023-05-22-15_36_00_UTC 2023-05-22T15:39:40.961 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:39:40.962 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2023-05-22-15_36_00_UTC) 2023-05-22T15:39:41.024 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2023-05-22-15_37_00_UTC 2023-05-22T15:39:41.025 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:39:41.025 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2023-05-22-15_37_00_UTC) 2023-05-22T15:39:41.088 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2023-05-22-15_38_01_UTC 2023-05-22T15:39:41.088 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:39:41.088 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2023-05-22-15_38_01_UTC) 2023-05-22T15:39:41.154 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2023-05-22-15_39_01_UTC 2023-05-22T15:39:41.155 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:39:41.156 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2023-05-22-15_39_01_UTC) 2023-05-22T15:39:41.219 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:39:41.219 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8) 2023-05-22T15:39:41.273 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove --fs cephfs /snap_test_dir1/dir9 1M 2023-05-22T15:39:41.467 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:41.462+0000 7f49598cd640 1 Processor -- start 2023-05-22T15:39:41.467 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:41.462+0000 7f49598cd640 1 -- start start 2023-05-22T15:39:41.468 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:41.462+0000 7f49598cd640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f49540cb440 0x7f49540cd410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:41.468 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:41.462+0000 7f49598cd640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4954070bb0 con 0x7f49540cc010 2023-05-22T15:39:41.469 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:41.462+0000 7f49598cd640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4954070e90 con 0x7f4954158950 2023-05-22T15:39:41.469 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:41.462+0000 7f49598cd640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4954071170 con 0x7f49540cb440 2023-05-22T15:39:41.469 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:41.462+0000 7f49537fe640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f49540cb440 0x7f49540cd410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:41.469 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:41.462+0000 7f49537fe640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f49540cb440 0x7f49540cd410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52602/0 (socket says 172.21.15.16:52602) 2023-05-22T15:39:41.469 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:41.462+0000 7f49537fe640 1 -- 172.21.15.16:0/3839502054 learned_addr learned my addr 172.21.15.16:0/3839502054 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:39:41.470 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:41.462+0000 7f49537fe640 1 -- 172.21.15.16:0/3839502054 >> v1:172.21.15.62:6789/0 conn(0x7f4954158950 legacy=0x7f49540cb900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:39:41.470 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:41.462+0000 7f49537fe640 1 -- 172.21.15.16:0/3839502054 >> v1:172.21.15.16:6789/0 conn(0x7f49540cc010 legacy=0x7f49540ccd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:41.470 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:41.462+0000 7f49537fe640 1 -- 172.21.15.16:0/3839502054 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4954071450 con 0x7f49540cb440 2023-05-22T15:39:41.470 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:41.462+0000 7f4951ffb640 1 -- 172.21.15.16:0/3839502054 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2092292592 0 0) 0x7f4954070bb0 con 0x7f49540cc010 2023-05-22T15:39:41.471 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:41.466+0000 7f49537fe640 1 --2- 172.21.15.16:0/3839502054 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f49540cb440 0x7f49540cd410 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f494c009e50 tx=0x7f494c01c9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6a17b4e0ccc7b5 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:39:41.471 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:41.466+0000 7f4951ffb640 1 -- 172.21.15.16:0/3839502054 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f494c0a5770 con 0x7f49540cb440 2023-05-22T15:39:41.472 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:41.466+0000 7f4951ffb640 1 -- 172.21.15.16:0/3839502054 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f494c0a58d0 con 0x7f49540cb440 2023-05-22T15:39:41.472 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:41.466+0000 7f4951ffb640 1 -- 172.21.15.16:0/3839502054 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f494c0a5c60 con 0x7f49540cb440 2023-05-22T15:39:41.473 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:41.466+0000 7f49598cd640 1 -- 172.21.15.16:0/3839502054 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f49540cb440 msgr2=0x7f49540cd410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:41.473 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:41.466+0000 7f49598cd640 1 --2- 172.21.15.16:0/3839502054 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f49540cb440 0x7f49540cd410 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f494c009e50 tx=0x7f494c01c9a0 comp rx=0 tx=0).stop 2023-05-22T15:39:41.473 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:41.466+0000 7f49598cd640 1 -- 172.21.15.16:0/3839502054 shutdown_connections 2023-05-22T15:39:41.473 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:41.466+0000 7f49598cd640 1 --2- 172.21.15.16:0/3839502054 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f49540cb440 0x7f49540cd410 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:41.473 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:41.466+0000 7f49598cd640 1 -- 172.21.15.16:0/3839502054 >> 172.21.15.16:0/3839502054 conn(0x7f49540b0cc0 msgr2=0x7f49540b10c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:39:41.473 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:41.466+0000 7f49598cd640 1 -- 172.21.15.16:0/3839502054 shutdown_connections 2023-05-22T15:39:41.473 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:41.466+0000 7f49598cd640 1 -- 172.21.15.16:0/3839502054 wait complete. 2023-05-22T15:39:41.475 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:41.466+0000 7f49598cd640 1 Processor -- start 2023-05-22T15:39:41.475 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:41.466+0000 7f49598cd640 1 -- start start 2023-05-22T15:39:41.475 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:41.470+0000 7f49598cd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f49540cb440 0x7f495414e940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:41.475 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:41.470+0000 7f49598cd640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f49540cc010 0x7f49540b2720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:41.475 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:41.470+0000 7f49598cd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4954158950 0x7f4954153570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:41.475 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:41.470+0000 7f49598cd640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49540b3ec0 con 0x7f49540cb440 2023-05-22T15:39:41.475 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:41.470+0000 7f49598cd640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4954167950 con 0x7f4954158950 2023-05-22T15:39:41.476 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:41.470+0000 7f49598cd640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4954167c30 con 0x7f49540cc010 2023-05-22T15:39:41.476 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:41.470+0000 7f4952ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f49540cb440 0x7f495414e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:41.476 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:41.470+0000 7f4952ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f49540cb440 0x7f495414e940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52608/0 (socket says 172.21.15.16:52608) 2023-05-22T15:39:41.476 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:41.470+0000 7f4952ffd640 1 -- 172.21.15.16:0/2912891506 learned_addr learned my addr 172.21.15.16:0/2912891506 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:39:41.476 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:41.470+0000 7f49537fe640 1 --2- 172.21.15.16:0/2912891506 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4954158950 0x7f4954153570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:41.476 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:41.470+0000 7f49527fc640 1 --2- 172.21.15.16:0/2912891506 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f49540cc010 0x7f49540b2720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:41.476 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:41.470+0000 7f4952ffd640 1 -- 172.21.15.16:0/2912891506 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f49540cc010 msgr2=0x7f49540b2720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:41.477 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:41.470+0000 7f4952ffd640 1 --2- 172.21.15.16:0/2912891506 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f49540cc010 0x7f49540b2720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:41.477 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:41.470+0000 7f4952ffd640 1 -- 172.21.15.16:0/2912891506 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4954158950 msgr2=0x7f4954153570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:41.477 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:41.470+0000 7f4952ffd640 1 --2- 172.21.15.16:0/2912891506 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4954158950 0x7f4954153570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:41.477 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:41.470+0000 7f4952ffd640 1 -- 172.21.15.16:0/2912891506 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f494c009b30 con 0x7f49540cb440 2023-05-22T15:39:41.477 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:41.470+0000 7f4952ffd640 1 --2- 172.21.15.16:0/2912891506 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f49540cb440 0x7f495414e940 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f494400b410 tx=0x7f494400b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:39:41.478 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:41.470+0000 7f493bfff640 1 -- 172.21.15.16:0/2912891506 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4944004020 con 0x7f49540cb440 2023-05-22T15:39:41.478 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:41.470+0000 7f493bfff640 1 -- 172.21.15.16:0/2912891506 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4944004180 con 0x7f49540cb440 2023-05-22T15:39:41.478 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:41.470+0000 7f49598cd640 1 -- 172.21.15.16:0/2912891506 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4954070980 con 0x7f49540cb440 2023-05-22T15:39:41.479 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:41.470+0000 7f493bfff640 1 -- 172.21.15.16:0/2912891506 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4944011630 con 0x7f49540cb440 2023-05-22T15:39:41.479 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:41.470+0000 7f49598cd640 1 -- 172.21.15.16:0/2912891506 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49540707f0 con 0x7f49540cb440 2023-05-22T15:39:41.479 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:41.470+0000 7f49598cd640 1 -- 172.21.15.16:0/2912891506 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4954069720 con 0x7f49540cb440 2023-05-22T15:39:41.479 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:41.470+0000 7f493bfff640 1 -- 172.21.15.16:0/2912891506 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 12) v1 ==== 84418+0+0 (secure 0 0 0) 0x7f4944011790 con 0x7f49540cb440 2023-05-22T15:39:41.479 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:41.470+0000 7f493bfff640 1 --2- 172.21.15.16:0/2912891506 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f4940065580 0x7f4940067a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:41.479 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:41.470+0000 7f493bfff640 1 -- 172.21.15.16:0/2912891506 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 7314+0+0 (secure 0 0 0) 0x7f4944082df0 con 0x7f49540cb440 2023-05-22T15:39:41.479 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:41.470+0000 7f49527fc640 1 --2- 172.21.15.16:0/2912891506 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f4940065580 0x7f4940067a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:41.480 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:41.474+0000 7f49527fc640 1 --2- 172.21.15.16:0/2912891506 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f4940065580 0x7f4940067a30 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f49540b0040 tx=0x7f4948009040 comp rx=0 tx=0).ready entity=mgr.5823 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:39:41.482 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:41.478+0000 7f493bfff640 1 -- 172.21.15.16:0/2912891506 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f4944050330 con 0x7f49540cb440 2023-05-22T15:39:41.671 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:41.666+0000 7f49598cd640 1 -- 172.21.15.16:0/2912891506 --> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "fs": "cephfs", "path": "/snap_test_dir1/dir9", "repeat": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f49541646d0 con 0x7f4940065580 2023-05-22T15:39:41.865 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:41.858+0000 7f493bfff640 1 -- 172.21.15.16:0/2912891506 <== mgr.5823 v2:172.21.15.16:6832/36921 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+47 (secure 0 0 0) 0x7f49541646d0 con 0x7f4940065580 2023-05-22T15:39:41.865 INFO:teuthology.orchestra.run.smithi016.stdout:Schedule removed for path /snap_test_dir1/dir9 2023-05-22T15:39:41.868 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:41.862+0000 7f49598cd640 1 -- 172.21.15.16:0/2912891506 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f4940065580 msgr2=0x7f4940067a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:41.869 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:41.862+0000 7f49598cd640 1 --2- 172.21.15.16:0/2912891506 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f4940065580 0x7f4940067a30 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f49540b0040 tx=0x7f4948009040 comp rx=0 tx=0).stop 2023-05-22T15:39:41.869 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:41.862+0000 7f49598cd640 1 -- 172.21.15.16:0/2912891506 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f49540cb440 msgr2=0x7f495414e940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:41.869 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:41.862+0000 7f49598cd640 1 --2- 172.21.15.16:0/2912891506 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f49540cb440 0x7f495414e940 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f494400b410 tx=0x7f494400b8d0 comp rx=0 tx=0).stop 2023-05-22T15:39:41.869 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:41.862+0000 7f49598cd640 1 -- 172.21.15.16:0/2912891506 shutdown_connections 2023-05-22T15:39:41.870 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:41.862+0000 7f49598cd640 1 --2- 172.21.15.16:0/2912891506 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4954158950 0x7f4954153570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:41.870 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:41.862+0000 7f49598cd640 1 --2- 172.21.15.16:0/2912891506 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f4940065580 0x7f4940067a30 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:41.870 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:41.862+0000 7f49598cd640 1 --2- 172.21.15.16:0/2912891506 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f49540cc010 0x7f49540b2720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:41.871 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:41.862+0000 7f49598cd640 1 --2- 172.21.15.16:0/2912891506 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f49540cb440 0x7f495414e940 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:41.871 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:41.862+0000 7f49598cd640 1 -- 172.21.15.16:0/2912891506 >> 172.21.15.16:0/2912891506 conn(0x7f49540b0cc0 msgr2=0x7f49540b00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:39:41.871 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:41.862+0000 7f49598cd640 1 -- 172.21.15.16:0/2912891506 shutdown_connections 2023-05-22T15:39:41.871 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:41.862+0000 7f49598cd640 1 -- 172.21.15.16:0/2912891506 wait complete. 2023-05-22T15:39:41.894 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/dir9 2023-05-22T15:39:41.895 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:39:41.896 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir9/.snap) 2023-05-22T15:39:41.909 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_29_01_UTC 2023-05-22T15:39:41.909 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_30_00_UTC 2023-05-22T15:39:41.909 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_31_00_UTC 2023-05-22T15:39:41.909 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_32_00_UTC 2023-05-22T15:39:41.910 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_33_01_UTC 2023-05-22T15:39:41.910 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_34_00_UTC 2023-05-22T15:39:41.910 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_35_00_UTC 2023-05-22T15:39:41.910 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_36_01_UTC 2023-05-22T15:39:41.910 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_37_01_UTC 2023-05-22T15:39:41.910 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_38_01_UTC 2023-05-22T15:39:41.910 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_39_00_UTC 2023-05-22T15:39:41.911 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2023-05-22-15_29_01_UTC 2023-05-22T15:39:41.911 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:39:41.911 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2023-05-22-15_29_01_UTC) 2023-05-22T15:39:41.971 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2023-05-22-15_30_00_UTC 2023-05-22T15:39:41.971 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:39:41.972 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2023-05-22-15_30_00_UTC) 2023-05-22T15:39:42.033 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2023-05-22-15_31_00_UTC 2023-05-22T15:39:42.034 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:39:42.034 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2023-05-22-15_31_00_UTC) 2023-05-22T15:39:42.089 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2023-05-22-15_32_00_UTC 2023-05-22T15:39:42.089 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:39:42.090 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2023-05-22-15_32_00_UTC) 2023-05-22T15:39:42.147 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2023-05-22-15_33_01_UTC 2023-05-22T15:39:42.148 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:39:42.148 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2023-05-22-15_33_01_UTC) 2023-05-22T15:39:42.208 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2023-05-22-15_34_00_UTC 2023-05-22T15:39:42.208 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:39:42.209 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2023-05-22-15_34_00_UTC) 2023-05-22T15:39:42.266 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2023-05-22-15_35_00_UTC 2023-05-22T15:39:42.267 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:39:42.267 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2023-05-22-15_35_00_UTC) 2023-05-22T15:39:42.329 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2023-05-22-15_36_01_UTC 2023-05-22T15:39:42.330 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:39:42.330 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2023-05-22-15_36_01_UTC) 2023-05-22T15:39:42.393 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2023-05-22-15_37_01_UTC 2023-05-22T15:39:42.394 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:39:42.394 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2023-05-22-15_37_01_UTC) 2023-05-22T15:39:42.452 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2023-05-22-15_38_01_UTC 2023-05-22T15:39:42.452 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:39:42.453 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2023-05-22-15_38_01_UTC) 2023-05-22T15:39:42.512 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2023-05-22-15_39_00_UTC 2023-05-22T15:39:42.512 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:39:42.512 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2023-05-22-15_39_00_UTC) 2023-05-22T15:39:42.575 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:39:42.576 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9) 2023-05-22T15:39:42.629 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2023-05-22T15:39:42.814 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:42.810+0000 7f4d2ddbd640 1 Processor -- start 2023-05-22T15:39:42.815 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:42.810+0000 7f4d2ddbd640 1 -- start start 2023-05-22T15:39:42.816 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:42.810+0000 7f4d2ddbd640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4d2806bdd0 0x7f4d280c8af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:42.816 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:42.810+0000 7f4d2ddbd640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f4d280cb730 0x7f4d280c9030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:42.816 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:42.810+0000 7f4d2ddbd640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d280c8660 con 0x7f4d2815bee0 2023-05-22T15:39:42.817 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:42.810+0000 7f4d2ddbd640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d280ce4a0 con 0x7f4d280cb730 2023-05-22T15:39:42.817 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:42.810+0000 7f4d2ddbd640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d280ce780 con 0x7f4d2806bdd0 2023-05-22T15:39:42.817 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:42.810+0000 7f4d27fff640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f4d280cb730 0x7f4d280c9030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:42.817 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:42.810+0000 7f4d27fff640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f4d280cb730 0x7f4d280c9030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52620/0 (socket says 172.21.15.16:52620) 2023-05-22T15:39:42.817 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:42.810+0000 7f4d27fff640 1 -- 172.21.15.16:0/4291937783 learned_addr learned my addr 172.21.15.16:0/4291937783 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:39:42.817 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:42.810+0000 7f4d1ffff640 1 --2- 172.21.15.16:0/4291937783 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4d2806bdd0 0x7f4d280c8af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:42.818 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:42.810+0000 7f4d27fff640 1 -- 172.21.15.16:0/4291937783 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4d2806bdd0 msgr2=0x7f4d280c8af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:39:42.818 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:42.810+0000 7f4d27fff640 1 --2- 172.21.15.16:0/4291937783 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4d2806bdd0 0x7f4d280c8af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:42.818 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:42.810+0000 7f4d27fff640 1 -- 172.21.15.16:0/4291937783 >> v1:172.21.15.62:6789/0 conn(0x7f4d2815bee0 legacy=0x7f4d2806b6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:42.818 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:42.810+0000 7f4d27fff640 1 -- 172.21.15.16:0/4291937783 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d280cea60 con 0x7f4d280cb730 2023-05-22T15:39:42.819 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:42.814+0000 7f4d27fff640 1 --2- 172.21.15.16:0/4291937783 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4d280cb730 0x7f4d280c9030 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f4d20009e50 tx=0x7f4d2001c9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb8537155fb5c0d9 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:39:42.819 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:42.814+0000 7f4d26ffd640 1 -- 172.21.15.16:0/4291937783 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d200a58d0 con 0x7f4d280cb730 2023-05-22T15:39:42.819 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:42.814+0000 7f4d26ffd640 1 -- 172.21.15.16:0/4291937783 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d200a5a30 con 0x7f4d280cb730 2023-05-22T15:39:42.820 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:42.814+0000 7f4d26ffd640 1 -- 172.21.15.16:0/4291937783 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d200a5dc0 con 0x7f4d280cb730 2023-05-22T15:39:42.820 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:42.814+0000 7f4d2ddbd640 1 -- 172.21.15.16:0/4291937783 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4d280cb730 msgr2=0x7f4d280c9030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:42.820 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:42.814+0000 7f4d2ddbd640 1 --2- 172.21.15.16:0/4291937783 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4d280cb730 0x7f4d280c9030 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f4d20009e50 tx=0x7f4d2001c9e0 comp rx=0 tx=0).stop 2023-05-22T15:39:42.821 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:42.814+0000 7f4d2ddbd640 1 -- 172.21.15.16:0/4291937783 shutdown_connections 2023-05-22T15:39:42.821 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:42.814+0000 7f4d2ddbd640 1 --2- 172.21.15.16:0/4291937783 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4d280cb730 0x7f4d280c9030 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:42.821 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:42.814+0000 7f4d2ddbd640 1 --2- 172.21.15.16:0/4291937783 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4d2806bdd0 0x7f4d280c8af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:42.821 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:42.814+0000 7f4d2ddbd640 1 -- 172.21.15.16:0/4291937783 >> 172.21.15.16:0/4291937783 conn(0x7f4d28069720 msgr2=0x7f4d28069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:39:42.821 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:42.814+0000 7f4d2ddbd640 1 -- 172.21.15.16:0/4291937783 shutdown_connections 2023-05-22T15:39:42.821 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:42.814+0000 7f4d2ddbd640 1 -- 172.21.15.16:0/4291937783 wait complete. 2023-05-22T15:39:42.822 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:42.814+0000 7f4d2ddbd640 1 Processor -- start 2023-05-22T15:39:42.823 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:42.814+0000 7f4d2ddbd640 1 -- start start 2023-05-22T15:39:42.823 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:42.814+0000 7f4d2ddbd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4d2806bdd0 0x7f4d280aee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:42.823 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:42.814+0000 7f4d2ddbd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4d280cb730 0x7f4d2814f880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:42.823 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:42.814+0000 7f4d2ddbd640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4d2815bee0 0x7f4d2814a320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:42.823 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:42.814+0000 7f4d2ddbd640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d2814bbb0 con 0x7f4d280cb730 2023-05-22T15:39:42.823 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:42.818+0000 7f4d2ddbd640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d2814afa0 con 0x7f4d2806bdd0 2023-05-22T15:39:42.824 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:42.818+0000 7f4d2ddbd640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d2814b280 con 0x7f4d2815bee0 2023-05-22T15:39:42.824 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:42.818+0000 7f4d1ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4d280cb730 0x7f4d2814f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:42.824 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:42.818+0000 7f4d277fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4d2806bdd0 0x7f4d280aee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:42.824 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:42.818+0000 7f4d1ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4d280cb730 0x7f4d2814f880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52630/0 (socket says 172.21.15.16:52630) 2023-05-22T15:39:42.824 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:42.818+0000 7f4d1ffff640 1 -- 172.21.15.16:0/1141779104 learned_addr learned my addr 172.21.15.16:0/1141779104 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:39:42.824 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:42.818+0000 7f4d27fff640 1 --2- 172.21.15.16:0/1141779104 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4d2815bee0 0x7f4d2814a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:42.824 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:42.818+0000 7f4d1ffff640 1 -- 172.21.15.16:0/1141779104 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4d2815bee0 msgr2=0x7f4d2814a320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:42.825 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:42.818+0000 7f4d1ffff640 1 --2- 172.21.15.16:0/1141779104 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4d2815bee0 0x7f4d2814a320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:42.825 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:42.818+0000 7f4d1ffff640 1 -- 172.21.15.16:0/1141779104 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4d2806bdd0 msgr2=0x7f4d280aee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:42.825 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:42.818+0000 7f4d1ffff640 1 --2- 172.21.15.16:0/1141779104 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4d2806bdd0 0x7f4d280aee00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:42.825 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:42.818+0000 7f4d1ffff640 1 -- 172.21.15.16:0/1141779104 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d20009b30 con 0x7f4d280cb730 2023-05-22T15:39:42.826 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:42.818+0000 7f4d1ffff640 1 --2- 172.21.15.16:0/1141779104 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4d280cb730 0x7f4d2814f880 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f4d1000b190 tx=0x7f4d1000b650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:39:42.826 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:42.818+0000 7f4d24ff9640 1 -- 172.21.15.16:0/1141779104 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d10019070 con 0x7f4d280cb730 2023-05-22T15:39:42.827 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:42.818+0000 7f4d24ff9640 1 -- 172.21.15.16:0/1141779104 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d1000be80 con 0x7f4d280cb730 2023-05-22T15:39:42.827 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:42.818+0000 7f4d2ddbd640 1 -- 172.21.15.16:0/1141779104 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d28168d40 con 0x7f4d280cb730 2023-05-22T15:39:42.827 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:42.818+0000 7f4d24ff9640 1 -- 172.21.15.16:0/1141779104 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d10014b10 con 0x7f4d280cb730 2023-05-22T15:39:42.827 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:42.818+0000 7f4d2ddbd640 1 -- 172.21.15.16:0/1141779104 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d28168bb0 con 0x7f4d280cb730 2023-05-22T15:39:42.827 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:42.818+0000 7f4d2ddbd640 1 -- 172.21.15.16:0/1141779104 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d28168ed0 con 0x7f4d280cb730 2023-05-22T15:39:42.827 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:42.818+0000 7f4d24ff9640 1 -- 172.21.15.16:0/1141779104 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 12) v1 ==== 84418+0+0 (secure 0 0 0) 0x7f4d1001b760 con 0x7f4d280cb730 2023-05-22T15:39:42.827 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:42.822+0000 7f4d24ff9640 1 --2- 172.21.15.16:0/1141779104 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f4d080655d0 0x7f4d08067a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:42.828 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:42.822+0000 7f4d24ff9640 1 -- 172.21.15.16:0/1141779104 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 7426+0+0 (secure 0 0 0) 0x7f4d1008af70 con 0x7f4d280cb730 2023-05-22T15:39:42.828 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:42.822+0000 7f4d277fe640 1 --2- 172.21.15.16:0/1141779104 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f4d080655d0 0x7f4d08067a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:42.828 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:42.822+0000 7f4d277fe640 1 --2- 172.21.15.16:0/1141779104 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f4d080655d0 0x7f4d08067a80 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f4d280c3f30 tx=0x7f4d18009040 comp rx=0 tx=0).ready entity=mgr.5823 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:39:42.831 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:42.826+0000 7f4d24ff9640 1 -- 172.21.15.16:0/1141779104 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f4d10058550 con 0x7f4d280cb730 2023-05-22T15:39:43.059 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:43.054+0000 7f4d2ddbd640 1 -- 172.21.15.16:0/1141779104 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7f4d2804dbe0 con 0x7f4d280cb730 2023-05-22T15:39:43.404 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:43.398+0000 7f4d24ff9640 1 -- 172.21.15.16:0/1141779104 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mgrmap(e 13) v1 ==== 84401+0+0 (secure 0 0 0) 0x7f4d10058920 con 0x7f4d280cb730 2023-05-22T15:39:43.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:43.402+0000 7f4d24ff9640 1 -- 172.21.15.16:0/1141779104 <== mon.0 v2:172.21.15.16:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v13) v1 ==== 93+0+0 (secure 0 0 0) 0x7f4d10064440 con 0x7f4d280cb730 2023-05-22T15:39:43.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:43.406+0000 7f4d2ddbd640 1 -- 172.21.15.16:0/1141779104 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f4d080655d0 msgr2=0x7f4d08067a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:43.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:43.406+0000 7f4d2ddbd640 1 --2- 172.21.15.16:0/1141779104 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f4d080655d0 0x7f4d08067a80 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f4d280c3f30 tx=0x7f4d18009040 comp rx=0 tx=0).stop 2023-05-22T15:39:43.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:43.406+0000 7f4d2ddbd640 1 -- 172.21.15.16:0/1141779104 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4d280cb730 msgr2=0x7f4d2814f880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:43.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:43.406+0000 7f4d2ddbd640 1 --2- 172.21.15.16:0/1141779104 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4d280cb730 0x7f4d2814f880 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f4d1000b190 tx=0x7f4d1000b650 comp rx=0 tx=0).stop 2023-05-22T15:39:43.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:43.406+0000 7f4d2ddbd640 1 -- 172.21.15.16:0/1141779104 shutdown_connections 2023-05-22T15:39:43.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:43.406+0000 7f4d2ddbd640 1 --2- 172.21.15.16:0/1141779104 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f4d080655d0 0x7f4d08067a80 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:43.412 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:43.406+0000 7f4d2ddbd640 1 --2- 172.21.15.16:0/1141779104 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4d2815bee0 0x7f4d2814a320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:43.412 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:43.406+0000 7f4d2ddbd640 1 --2- 172.21.15.16:0/1141779104 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4d280cb730 0x7f4d2814f880 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:43.412 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:43.406+0000 7f4d2ddbd640 1 --2- 172.21.15.16:0/1141779104 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4d2806bdd0 0x7f4d280aee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:43.412 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:43.406+0000 7f4d2ddbd640 1 -- 172.21.15.16:0/1141779104 >> 172.21.15.16:0/1141779104 conn(0x7f4d28069720 msgr2=0x7f4d28069af0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:39:43.412 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:43.406+0000 7f4d2ddbd640 1 -- 172.21.15.16:0/1141779104 shutdown_connections 2023-05-22T15:39:43.412 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:43.406+0000 7f4d2ddbd640 1 -- 172.21.15.16:0/1141779104 wait complete. 2023-05-22T15:39:43.444 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-05-22T15:39:43.445 DEBUG:teuthology.orchestra.run.smithi016:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-05-22T15:39:43.496 DEBUG:teuthology.orchestra.run.smithi062:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-05-22T15:39:43.531 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-05-22T15:39:43.531 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-05-22T15:39:43.532 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2023-05-22T15:39:43.538 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-05-22T15:39:43.570 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:39:43.569+0000 7f6e633d5180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-05-22T15:39:43.581 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:39:43.574+0000 7f3646e83180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-05-22T15:39:43.716 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:43.710+0000 7f22f25f3640 1 Processor -- start 2023-05-22T15:39:43.716 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:43.710+0000 7f22f25f3640 1 -- start start 2023-05-22T15:39:43.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:43.710+0000 7f22f25f3640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f22ec0ce480 0x7f22ec071640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:43.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:43.710+0000 7f22f25f3640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f22ec0ce850 0x7f22ec160090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:43.719 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:43.710+0000 7f22f25f3640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22ec069ae0 con 0x7f22ec1605d0 2023-05-22T15:39:43.719 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:43.710+0000 7f22f25f3640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22ec15f990 con 0x7f22ec0ce850 2023-05-22T15:39:43.719 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:43.710+0000 7f22f25f3640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22ec069dc0 con 0x7f22ec0ce480 2023-05-22T15:39:43.719 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:43.710+0000 7f22eb7fe640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f22ec0ce850 0x7f22ec160090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:43.719 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:43.710+0000 7f22f0b90640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f22ec1605d0 0x7f22ec15c270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:56518/0 (socket says 172.21.15.16:56518) 2023-05-22T15:39:43.719 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:43.710+0000 7f22f0b90640 1 -- 172.21.15.16:0/3550216580 learned_addr learned my addr 172.21.15.16:0/3550216580 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:39:43.720 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:43.710+0000 7f22ebfff640 1 --2- 172.21.15.16:0/3550216580 >> v2:172.21.15.62:3300/0 conn(0x7f22ec0ce480 0x7f22ec071640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:43.720 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:43.710+0000 7f22eb7fe640 1 -- 172.21.15.16:0/3550216580 >> v2:172.21.15.62:3300/0 conn(0x7f22ec0ce480 msgr2=0x7f22ec071640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:39:43.720 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:43.710+0000 7f22eb7fe640 1 --2- 172.21.15.16:0/3550216580 >> v2:172.21.15.62:3300/0 conn(0x7f22ec0ce480 0x7f22ec071640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:43.720 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:43.710+0000 7f22eb7fe640 1 -- 172.21.15.16:0/3550216580 >> v1:172.21.15.16:6789/0 conn(0x7f22ec1605d0 legacy=0x7f22ec15c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:43.720 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:43.710+0000 7f22eb7fe640 1 -- 172.21.15.16:0/3550216580 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22ec06a0a0 con 0x7f22ec0ce850 2023-05-22T15:39:43.721 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:43.714+0000 7f22eb7fe640 1 --2- 172.21.15.16:0/3550216580 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f22ec0ce850 0x7f22ec160090 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f22ec06f310 tx=0x7f22e001c430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8d01ea0abd8ac6d server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:39:43.721 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:43.714+0000 7f22eaffd640 1 -- 172.21.15.16:0/3550216580 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22e00ab600 con 0x7f22ec0ce850 2023-05-22T15:39:43.721 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:43.714+0000 7f22eaffd640 1 -- 172.21.15.16:0/3550216580 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22e00a7050 con 0x7f22ec0ce850 2023-05-22T15:39:43.722 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:43.714+0000 7f22eaffd640 1 -- 172.21.15.16:0/3550216580 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22e00ab980 con 0x7f22ec0ce850 2023-05-22T15:39:43.722 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:43.714+0000 7f22f25f3640 1 -- 172.21.15.16:0/3550216580 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f22ec0ce850 msgr2=0x7f22ec160090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:43.722 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:43.714+0000 7f22f25f3640 1 --2- 172.21.15.16:0/3550216580 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f22ec0ce850 0x7f22ec160090 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f22ec06f310 tx=0x7f22e001c430 comp rx=0 tx=0).stop 2023-05-22T15:39:43.722 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:43.714+0000 7f22f25f3640 1 -- 172.21.15.16:0/3550216580 shutdown_connections 2023-05-22T15:39:43.722 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:43.714+0000 7f22f25f3640 1 --2- 172.21.15.16:0/3550216580 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f22ec0ce850 0x7f22ec160090 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:43.722 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:43.714+0000 7f22f25f3640 1 --2- 172.21.15.16:0/3550216580 >> v2:172.21.15.62:3300/0 conn(0x7f22ec0ce480 0x7f22ec071640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:43.723 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:43.714+0000 7f22f25f3640 1 -- 172.21.15.16:0/3550216580 >> 172.21.15.16:0/3550216580 conn(0x7f22ec0cc870 msgr2=0x7f22ec0cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:39:43.723 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:43.714+0000 7f22f25f3640 1 -- 172.21.15.16:0/3550216580 shutdown_connections 2023-05-22T15:39:43.723 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:43.714+0000 7f22f25f3640 1 -- 172.21.15.16:0/3550216580 wait complete. 2023-05-22T15:39:43.725 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:43.718+0000 7f22f25f3640 1 Processor -- start 2023-05-22T15:39:43.725 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:43.718+0000 7f22f25f3640 1 -- start start 2023-05-22T15:39:43.725 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:43.718+0000 7f22f25f3640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f22ec0ce480 0x7f22ec153ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:43.725 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:43.718+0000 7f22f25f3640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f22ec0ce850 0x7f22ec159590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:43.726 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:43.718+0000 7f22f25f3640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f22ec1605d0 0x7f22ec159ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:43.726 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:43.718+0000 7f22f25f3640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22ec14d790 con 0x7f22ec0ce480 2023-05-22T15:39:43.726 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:43.718+0000 7f22f25f3640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22ec14ee80 con 0x7f22ec1605d0 2023-05-22T15:39:43.726 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:43.718+0000 7f22f25f3640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22ec14d9e0 con 0x7f22ec0ce850 2023-05-22T15:39:43.726 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:43.718+0000 7f22ebfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f22ec0ce480 0x7f22ec153ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:43.726 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:43.718+0000 7f22ebfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f22ec0ce480 0x7f22ec153ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52668/0 (socket says 172.21.15.16:52668) 2023-05-22T15:39:43.727 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:43.718+0000 7f22ebfff640 1 -- 172.21.15.16:0/1033778045 learned_addr learned my addr 172.21.15.16:0/1033778045 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:39:43.727 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:43.718+0000 7f22f0b90640 1 --2- 172.21.15.16:0/1033778045 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f22ec1605d0 0x7f22ec159ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:43.727 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:43.718+0000 7f22eb7fe640 1 --2- 172.21.15.16:0/1033778045 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f22ec0ce850 0x7f22ec159590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:43.727 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:43.718+0000 7f22ebfff640 1 -- 172.21.15.16:0/1033778045 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f22ec0ce850 msgr2=0x7f22ec159590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:43.727 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:43.718+0000 7f22ebfff640 1 --2- 172.21.15.16:0/1033778045 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f22ec0ce850 0x7f22ec159590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:43.727 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:43.718+0000 7f22ebfff640 1 -- 172.21.15.16:0/1033778045 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f22ec1605d0 msgr2=0x7f22ec159ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:43.727 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:43.718+0000 7f22ebfff640 1 --2- 172.21.15.16:0/1033778045 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f22ec1605d0 0x7f22ec159ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:43.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:43.718+0000 7f22ebfff640 1 -- 172.21.15.16:0/1033778045 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22e0009420 con 0x7f22ec0ce480 2023-05-22T15:39:43.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:43.718+0000 7f22f0b90640 1 --2- 172.21.15.16:0/1033778045 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f22ec1605d0 0x7f22ec159ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T15:39:43.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:43.718+0000 7f22ebfff640 1 --2- 172.21.15.16:0/1033778045 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f22ec0ce480 0x7f22ec153ae0 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f22dc00e870 tx=0x7f22dc00ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:39:43.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:43.718+0000 7f22e8ff9640 1 -- 172.21.15.16:0/1033778045 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22dc009510 con 0x7f22ec0ce480 2023-05-22T15:39:43.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:43.718+0000 7f22e8ff9640 1 -- 172.21.15.16:0/1033778045 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22dc00eec0 con 0x7f22ec0ce480 2023-05-22T15:39:43.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:43.718+0000 7f22e8ff9640 1 -- 172.21.15.16:0/1033778045 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22dc00cb40 con 0x7f22ec0ce480 2023-05-22T15:39:43.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:43.718+0000 7f22f25f3640 1 -- 172.21.15.16:0/1033778045 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22ec0698b0 con 0x7f22ec0ce480 2023-05-22T15:39:43.729 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:43.718+0000 7f22f25f3640 1 -- 172.21.15.16:0/1033778045 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22ec069720 con 0x7f22ec0ce480 2023-05-22T15:39:43.729 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:43.722+0000 7f22f25f3640 1 -- 172.21.15.16:0/1033778045 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22ec0c5e70 con 0x7f22ec0ce480 2023-05-22T15:39:43.730 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:43.722+0000 7f22e8ff9640 1 -- 172.21.15.16:0/1033778045 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 13) v1 ==== 84401+0+0 (secure 0 0 0) 0x7f22dc00cca0 con 0x7f22ec0ce480 2023-05-22T15:39:43.730 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:43.722+0000 7f22e8ff9640 1 --2- 172.21.15.16:0/1033778045 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f22d8065530 0x7f22d80679e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:43.730 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:43.722+0000 7f22eb7fe640 1 -- 172.21.15.16:0/1033778045 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f22d8065530 msgr2=0x7f22d80679e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/36921 2023-05-22T15:39:43.730 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:43.722+0000 7f22e8ff9640 1 -- 172.21.15.16:0/1033778045 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 7410+0+0 (secure 0 0 0) 0x7f22dc083200 con 0x7f22ec0ce480 2023-05-22T15:39:43.730 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:43.722+0000 7f22eb7fe640 1 --2- 172.21.15.16:0/1033778045 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f22d8065530 0x7f22d80679e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-05-22T15:39:43.731 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:43.726+0000 7f22e8ff9640 1 -- 172.21.15.16:0/1033778045 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f22dc0506b0 con 0x7f22ec0ce480 2023-05-22T15:39:43.890 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:43.882+0000 7f22f25f3640 1 -- 172.21.15.16:0/1033778045 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f22ec15dd10 con 0x7f22ec0ce480 2023-05-22T15:39:43.893 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:43.886+0000 7f22e8ff9640 1 -- 172.21.15.16:0/1033778045 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v57) v1 ==== 74+0+19604 (secure 0 0 0) 0x7f22dc053cd0 con 0x7f22ec0ce480 2023-05-22T15:39:43.894 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:39:43.894 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":57,"fsid":"b2719834-7847-4d98-90a6-f65d57f85c49","created":"2023-05-22T15:26:25.259314+0000","modified":"2023-05-22T15:39:43.338840+0000","last_up_change":"2023-05-22T15:26:36.448524+0000","last_in_change":"2023-05-22T15:26:27.675668+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-05-22T15:26:38.019781+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-05-22T15:26:43.570514+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2023-05-22T15:28:11.902605+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"41","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2023-05-22T15:28:12.298278+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"57","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":216,"snap_epoch":57,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9e7ff854-b25f-4cbb-a5c2-1799d9c53d2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":37021},{"type":"v1","addr":"172.21.15.16:6825","nonce":37021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":37021},{"type":"v1","addr":"172.21.15.16:6827","nonce":37021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":37021},{"type":"v1","addr":"172.21.15.16:6831","nonce":37021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":37021},{"type":"v1","addr":"172.21.15.16:6829","nonce":37021}]},"public_addr":"172.21.15.16:6825/37021","cluster_addr":"172.21.15.16:6827/37021","heartbeat_back_addr":"172.21.15.16:6831/37021","heartbeat_front_addr":"172.21.15.16:6829/37021","state":["exists","up"]},{"osd":1,"uuid":"4702fbc9-a927-4565-aca5-41dc69cc8264","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":37008},{"type":"v1","addr":"172.21.15.16:6801","nonce":37008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":37008},{"type":"v1","addr":"172.21.15.16:6803","nonce":37008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":37008},{"type":"v1","addr":"172.21.15.16:6807","nonce":37008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":37008},{"type":"v1","addr":"172.21.15.16:6805","nonce":37008}]},"public_addr":"172.21.15.16:6801/37008","cluster_addr":"172.21.15.16:6803/37008","heartbeat_back_addr":"172.21.15.16:6807/37008","heartbeat_front_addr":"172.21.15.16:6805/37008","state":["exists","up"]},{"osd":2,"uuid":"1e7bba46-4eba-47ef-bb81-5e2d371638ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":37009},{"type":"v1","addr":"172.21.15.16:6809","nonce":37009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":37009},{"type":"v1","addr":"172.21.15.16:6811","nonce":37009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":37009},{"type":"v1","addr":"172.21.15.16:6815","nonce":37009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":37009},{"type":"v1","addr":"172.21.15.16:6813","nonce":37009}]},"public_addr":"172.21.15.16:6809/37009","cluster_addr":"172.21.15.16:6811/37009","heartbeat_back_addr":"172.21.15.16:6815/37009","heartbeat_front_addr":"172.21.15.16:6813/37009","state":["exists","up"]},{"osd":3,"uuid":"383cacc0-5a88-4904-990e-498ca59ef48a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":37020},{"type":"v1","addr":"172.21.15.16:6817","nonce":37020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":37020},{"type":"v1","addr":"172.21.15.16:6819","nonce":37020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":37020},{"type":"v1","addr":"172.21.15.16:6823","nonce":37020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":37020},{"type":"v1","addr":"172.21.15.16:6821","nonce":37020}]},"public_addr":"172.21.15.16:6817/37020","cluster_addr":"172.21.15.16:6819/37020","heartbeat_back_addr":"172.21.15.16:6823/37020","heartbeat_front_addr":"172.21.15.16:6821/37020","state":["exists","up"]},{"osd":4,"uuid":"970500ef-95b7-42c1-b080-eed4fae7013d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":37353},{"type":"v1","addr":"172.21.15.62:6817","nonce":37353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6819","nonce":37353},{"type":"v1","addr":"172.21.15.62:6822","nonce":37353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37353},{"type":"v1","addr":"172.21.15.62:6831","nonce":37353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37353},{"type":"v1","addr":"172.21.15.62:6826","nonce":37353}]},"public_addr":"172.21.15.62:6817/37353","cluster_addr":"172.21.15.62:6822/37353","heartbeat_back_addr":"172.21.15.62:6831/37353","heartbeat_front_addr":"172.21.15.62:6826/37353","state":["exists","up"]},{"osd":5,"uuid":"022d67db-d4d8-445b-b1b9-8b446a00f5c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":37352},{"type":"v1","addr":"172.21.15.62:6801","nonce":37352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":37352},{"type":"v1","addr":"172.21.15.62:6803","nonce":37352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":37352},{"type":"v1","addr":"172.21.15.62:6807","nonce":37352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":37352},{"type":"v1","addr":"172.21.15.62:6805","nonce":37352}]},"public_addr":"172.21.15.62:6801/37352","cluster_addr":"172.21.15.62:6803/37352","heartbeat_back_addr":"172.21.15.62:6807/37352","heartbeat_front_addr":"172.21.15.62:6805/37352","state":["exists","up"]},{"osd":6,"uuid":"42bd372b-6688-4a1b-8407-e16e8c266e33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":37354},{"type":"v1","addr":"172.21.15.62:6820","nonce":37354}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":37354},{"type":"v1","addr":"172.21.15.62:6823","nonce":37354}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6829","nonce":37354},{"type":"v1","addr":"172.21.15.62:6830","nonce":37354}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6825","nonce":37354},{"type":"v1","addr":"172.21.15.62:6827","nonce":37354}]},"public_addr":"172.21.15.62:6820/37354","cluster_addr":"172.21.15.62:6823/37354","heartbeat_back_addr":"172.21.15.62:6830/37354","heartbeat_front_addr":"172.21.15.62:6827/37354","state":["exists","up"]},{"osd":7,"uuid":"ac70b2a3-f41b-4bf2-b227-7f2cbb55426d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37355},{"type":"v1","addr":"172.21.15.62:6809","nonce":37355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37355},{"type":"v1","addr":"172.21.15.62:6811","nonce":37355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":37355},{"type":"v1","addr":"172.21.15.62:6815","nonce":37355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":37355},{"type":"v1","addr":"172.21.15.62:6813","nonce":37355}]},"public_addr":"172.21.15.62:6809/37355","cluster_addr":"172.21.15.62:6811/37355","heartbeat_back_addr":"172.21.15.62:6815/37355","heartbeat_front_addr":"172.21.15.62:6813/37355","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.368896+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.107273+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.276888+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.287502+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.124791+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.251937+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.255679+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.232797+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"7.4","mappings":[{"from":0,"to":5}]},{"pgid":"7.2e","mappings":[{"from":0,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:0/3979017463":"2023-05-23T15:28:30.504372+0000","172.21.15.16:0/901306510":"2023-05-23T15:28:30.504372+0000","172.21.15.16:0/2755009771":"2023-05-23T15:28:30.504372+0000","172.21.15.16:0/4222124422":"2023-05-23T15:28:30.504372+0000","172.21.15.16:6833/36921":"2023-05-23T15:28:30.504372+0000","172.21.15.16:6832/36921":"2023-05-23T15:28:30.504372+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":7,"snaps":[{"begin":2,"length":2},{"begin":6,"length":3},{"begin":10,"length":1},{"begin":12,"length":1},{"begin":14,"length":2},{"begin":19,"length":3},{"begin":24,"length":3},{"begin":28,"length":3},{"begin":33,"length":1},{"begin":35,"length":1},{"begin":37,"length":1},{"begin":39,"length":4},{"begin":45,"length":3},{"begin":49,"length":2},{"begin":52,"length":2},{"begin":55,"length":2},{"begin":58,"length":1},{"begin":61,"length":2},{"begin":64,"length":1},{"begin":67,"length":1},{"begin":70,"length":2},{"begin":73,"length":2},{"begin":77,"length":1},{"begin":79,"length":2},{"begin":82,"length":1},{"begin":84,"length":1},{"begin":86,"length":4},{"begin":93,"length":4},{"begin":98,"length":1},{"begin":100,"length":1},{"begin":102,"length":2},{"begin":105,"length":2},{"begin":108,"length":1},{"begin":110,"length":2},{"begin":144,"length":73}]}],"new_removed_snaps":[{"pool":7,"snaps":[{"begin":10,"length":1},{"begin":14,"length":1},{"begin":29,"length":1},{"begin":35,"length":1},{"begin":50,"length":1},{"begin":58,"length":1},{"begin":108,"length":1},{"begin":210,"length":7}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-05-22T15:39:43.896 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:43.890+0000 7f22f25f3640 1 -- 172.21.15.16:0/1033778045 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f22d8065530 msgr2=0x7f22d80679e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-05-22T15:39:43.896 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:43.890+0000 7f22f25f3640 1 --2- 172.21.15.16:0/1033778045 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f22d8065530 0x7f22d80679e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:43.896 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:43.890+0000 7f22f25f3640 1 -- 172.21.15.16:0/1033778045 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f22ec0ce480 msgr2=0x7f22ec153ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:43.896 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:43.890+0000 7f22f25f3640 1 --2- 172.21.15.16:0/1033778045 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f22ec0ce480 0x7f22ec153ae0 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f22dc00e870 tx=0x7f22dc00ec70 comp rx=0 tx=0).stop 2023-05-22T15:39:43.897 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:43.890+0000 7f22f25f3640 1 -- 172.21.15.16:0/1033778045 shutdown_connections 2023-05-22T15:39:43.897 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:43.890+0000 7f22f25f3640 1 --2- 172.21.15.16:0/1033778045 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f22ec1605d0 0x7f22ec159ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:43.897 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:43.890+0000 7f22f25f3640 1 --2- 172.21.15.16:0/1033778045 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f22d8065530 0x7f22d80679e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:43.897 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:43.890+0000 7f22f25f3640 1 --2- 172.21.15.16:0/1033778045 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f22ec0ce850 0x7f22ec159590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:43.897 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:43.890+0000 7f22f25f3640 1 --2- 172.21.15.16:0/1033778045 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f22ec0ce480 0x7f22ec153ae0 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:43.897 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:43.890+0000 7f22f25f3640 1 -- 172.21.15.16:0/1033778045 >> 172.21.15.16:0/1033778045 conn(0x7f22ec0cc870 msgr2=0x7f22ec15d600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:39:43.897 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:43.890+0000 7f22f25f3640 1 -- 172.21.15.16:0/1033778045 shutdown_connections 2023-05-22T15:39:43.898 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:43.890+0000 7f22f25f3640 1 -- 172.21.15.16:0/1033778045 wait complete. 2023-05-22T15:39:43.914 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-05-22T15:26:38.019781+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-05-22T15:26:43.570514+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_metadata', 'create_time': '2023-05-22T15:28:11.902605+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '41', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'cephfs_data', 'create_time': '2023-05-22T15:28:12.298278+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '57', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 216, 'snap_epoch': 57, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-05-22T15:39:43.915 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-05-22T15:39:44.019 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:39:44.017+0000 7f6e633d5180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-05-22T15:39:44.021 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:39:44.014+0000 7f3646e83180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-05-22T15:39:44.034 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.030+0000 7fdd04f5c640 1 Processor -- start 2023-05-22T15:39:44.035 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.030+0000 7fdd04f5c640 1 -- start start 2023-05-22T15:39:44.036 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.030+0000 7fdd04f5c640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fdd0015cfa0 0x7fdd0006db00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:44.036 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.030+0000 7fdd04f5c640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fdd0006e740 0x7fdd0006ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:44.036 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.030+0000 7fdd04f5c640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd0006e040 con 0x7fdd000cf870 2023-05-22T15:39:44.036 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.030+0000 7fdd04f5c640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd000cb9f0 con 0x7fdd0015cfa0 2023-05-22T15:39:44.037 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.030+0000 7fdd04f5c640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd0006f100 con 0x7fdd0006e740 2023-05-22T15:39:44.037 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.030+0000 7fdcfe59c640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fdd0015cfa0 0x7fdd0006db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:44.037 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.030+0000 7fdcfe59c640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fdd0015cfa0 0x7fdd0006db00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52674/0 (socket says 172.21.15.16:52674) 2023-05-22T15:39:44.037 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.030+0000 7fdcfe59c640 1 -- 172.21.15.16:0/2228942079 learned_addr learned my addr 172.21.15.16:0/2228942079 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:39:44.037 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.030+0000 7fdcfed9d640 1 --2- 172.21.15.16:0/2228942079 >> v2:172.21.15.62:3300/0 conn(0x7fdd0006e740 0x7fdd0006ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:44.037 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.030+0000 7fdcfe59c640 1 -- 172.21.15.16:0/2228942079 >> v2:172.21.15.62:3300/0 conn(0x7fdd0006e740 msgr2=0x7fdd0006ebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:39:44.037 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.030+0000 7fdcfe59c640 1 --2- 172.21.15.16:0/2228942079 >> v2:172.21.15.62:3300/0 conn(0x7fdd0006e740 0x7fdd0006ebc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:44.038 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.030+0000 7fdcfe59c640 1 -- 172.21.15.16:0/2228942079 >> v1:172.21.15.62:6789/0 conn(0x7fdd000cf870 legacy=0x7fdd000b1fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:39:44.038 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.030+0000 7fdcfe59c640 1 -- 172.21.15.16:0/2228942079 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd0006e490 con 0x7fdd0015cfa0 2023-05-22T15:39:44.039 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.034+0000 7fdcfe59c640 1 --2- 172.21.15.16:0/2228942079 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdd0015cfa0 0x7fdd0006db00 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7fdd000c9760 tx=0x7fdcf001c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=880f4597250b8bf server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:39:44.039 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.034+0000 7fdcfd59a640 1 -- 172.21.15.16:0/2228942079 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcf00aa5f0 con 0x7fdd0015cfa0 2023-05-22T15:39:44.039 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.034+0000 7fdcfd59a640 1 -- 172.21.15.16:0/2228942079 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdcf00aa750 con 0x7fdd0015cfa0 2023-05-22T15:39:44.039 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.034+0000 7fdcfd59a640 1 -- 172.21.15.16:0/2228942079 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcf00aaae0 con 0x7fdd0015cfa0 2023-05-22T15:39:44.043 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.038+0000 7fdd04f5c640 1 -- 172.21.15.16:0/2228942079 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdd0015cfa0 msgr2=0x7fdd0006db00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:44.043 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.038+0000 7fdd04f5c640 1 --2- 172.21.15.16:0/2228942079 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdd0015cfa0 0x7fdd0006db00 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7fdd000c9760 tx=0x7fdcf001c490 comp rx=0 tx=0).stop 2023-05-22T15:39:44.043 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.038+0000 7fdd04f5c640 1 -- 172.21.15.16:0/2228942079 shutdown_connections 2023-05-22T15:39:44.043 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.038+0000 7fdd04f5c640 1 --2- 172.21.15.16:0/2228942079 >> v2:172.21.15.62:3300/0 conn(0x7fdd0006e740 0x7fdd0006ebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:44.043 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.038+0000 7fdd04f5c640 1 --2- 172.21.15.16:0/2228942079 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdd0015cfa0 0x7fdd0006db00 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:44.044 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.038+0000 7fdd04f5c640 1 -- 172.21.15.16:0/2228942079 >> 172.21.15.16:0/2228942079 conn(0x7fdd000cc870 msgr2=0x7fdd000cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:39:44.044 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.038+0000 7fdd04f5c640 1 -- 172.21.15.16:0/2228942079 shutdown_connections 2023-05-22T15:39:44.044 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.038+0000 7fdd04f5c640 1 -- 172.21.15.16:0/2228942079 wait complete. 2023-05-22T15:39:44.044 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.038+0000 7fdd04f5c640 1 Processor -- start 2023-05-22T15:39:44.044 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.038+0000 7fdd04f5c640 1 -- start start 2023-05-22T15:39:44.045 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.038+0000 7fdd04f5c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdd0006e740 0x7fdd00149810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:44.046 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.038+0000 7fdd04f5c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdd000cf870 0x7fdd00151d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:44.046 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.038+0000 7fdd04f5c640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fdd0015cfa0 0x7fdd000b29e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:44.046 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.038+0000 7fdd04f5c640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd000c60c0 con 0x7fdd0006e740 2023-05-22T15:39:44.046 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.038+0000 7fdd04f5c640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd001569c0 con 0x7fdd000cf870 2023-05-22T15:39:44.046 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.038+0000 7fdd04f5c640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd000b1f90 con 0x7fdd0015cfa0 2023-05-22T15:39:44.046 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.038+0000 7fdcfe59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdd0006e740 0x7fdd00149810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:44.047 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.038+0000 7fdcfe59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdd0006e740 0x7fdd00149810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52688/0 (socket says 172.21.15.16:52688) 2023-05-22T15:39:44.047 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.038+0000 7fdcfe59c640 1 -- 172.21.15.16:0/2257427040 learned_addr learned my addr 172.21.15.16:0/2257427040 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:39:44.048 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.038+0000 7fdcfdd9b640 1 --2- 172.21.15.16:0/2257427040 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdd000cf870 0x7fdd00151d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:44.048 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.038+0000 7fdcfed9d640 1 --2- 172.21.15.16:0/2257427040 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fdd0015cfa0 0x7fdd000b29e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:44.048 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.042+0000 7fdcfe59c640 1 -- 172.21.15.16:0/2257427040 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fdd0015cfa0 msgr2=0x7fdd000b29e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:44.048 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.042+0000 7fdcfe59c640 1 --2- 172.21.15.16:0/2257427040 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fdd0015cfa0 0x7fdd000b29e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:44.048 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.042+0000 7fdcfe59c640 1 -- 172.21.15.16:0/2257427040 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdd000cf870 msgr2=0x7fdd00151d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:44.048 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.042+0000 7fdcfe59c640 1 --2- 172.21.15.16:0/2257427040 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdd000cf870 0x7fdd00151d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:44.049 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.042+0000 7fdcfe59c640 1 -- 172.21.15.16:0/2257427040 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcf00094d0 con 0x7fdd0006e740 2023-05-22T15:39:44.049 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.042+0000 7fdcfdd9b640 1 --2- 172.21.15.16:0/2257427040 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdd000cf870 0x7fdd00151d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T15:39:44.049 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.042+0000 7fdcfe59c640 1 --2- 172.21.15.16:0/2257427040 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdd0006e740 0x7fdd00149810 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7fdcf00ac040 tx=0x7fdcf001acd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:39:44.049 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.042+0000 7fdceeffd640 1 -- 172.21.15.16:0/2257427040 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcf00b3800 con 0x7fdd0006e740 2023-05-22T15:39:44.049 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.042+0000 7fdceeffd640 1 -- 172.21.15.16:0/2257427040 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdcf00b3980 con 0x7fdd0006e740 2023-05-22T15:39:44.049 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.042+0000 7fdd04f5c640 1 -- 172.21.15.16:0/2257427040 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd000cb610 con 0x7fdd0006e740 2023-05-22T15:39:44.049 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.042+0000 7fdceeffd640 1 -- 172.21.15.16:0/2257427040 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcf00b3d30 con 0x7fdd0006e740 2023-05-22T15:39:44.049 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.042+0000 7fdd04f5c640 1 -- 172.21.15.16:0/2257427040 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd000cb330 con 0x7fdd0006e740 2023-05-22T15:39:44.050 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.042+0000 7fdd04f5c640 1 -- 172.21.15.16:0/2257427040 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd0015d370 con 0x7fdd0006e740 2023-05-22T15:39:44.050 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.042+0000 7fdceeffd640 1 -- 172.21.15.16:0/2257427040 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 13) v1 ==== 84401+0+0 (secure 0 0 0) 0x7fdcf00c6440 con 0x7fdd0006e740 2023-05-22T15:39:44.050 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.042+0000 7fdceeffd640 1 --2- 172.21.15.16:0/2257427040 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fdce0065530 0x7fdce00679e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:44.051 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.042+0000 7fdcfdd9b640 1 -- 172.21.15.16:0/2257427040 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fdce0065530 msgr2=0x7fdce00679e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/36921 2023-05-22T15:39:44.051 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.042+0000 7fdceeffd640 1 -- 172.21.15.16:0/2257427040 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 7410+0+0 (secure 0 0 0) 0x7fdcf00a7070 con 0x7fdd0006e740 2023-05-22T15:39:44.051 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.042+0000 7fdcfdd9b640 1 --2- 172.21.15.16:0/2257427040 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fdce0065530 0x7fdce00679e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-05-22T15:39:44.052 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.046+0000 7fdceeffd640 1 -- 172.21.15.16:0/2257427040 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fdcf00eb390 con 0x7fdd0006e740 2023-05-22T15:39:44.162 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:39:44.158+0000 7f3646e83180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-05-22T15:39:44.165 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:39:44.161+0000 7f6e633d5180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-05-22T15:39:44.202 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.198+0000 7fdd04f5c640 1 -- 172.21.15.16:0/2257427040 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fdd0006e040 con 0x7fdd0006e740 2023-05-22T15:39:44.203 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.198+0000 7fdceeffd640 1 -- 172.21.15.16:0/2257427040 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v57) v1 ==== 93+0+10 (secure 0 0 0) 0x7fdd0006e040 con 0x7fdd0006e740 2023-05-22T15:39:44.203 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2023-05-22T15:39:44.206 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.198+0000 7fdd04f5c640 1 -- 172.21.15.16:0/2257427040 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fdce0065530 msgr2=0x7fdce00679e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-05-22T15:39:44.206 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.198+0000 7fdd04f5c640 1 --2- 172.21.15.16:0/2257427040 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fdce0065530 0x7fdce00679e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:44.206 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.198+0000 7fdd04f5c640 1 -- 172.21.15.16:0/2257427040 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdd0006e740 msgr2=0x7fdd00149810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:44.206 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.198+0000 7fdd04f5c640 1 --2- 172.21.15.16:0/2257427040 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdd0006e740 0x7fdd00149810 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7fdcf00ac040 tx=0x7fdcf001acd0 comp rx=0 tx=0).stop 2023-05-22T15:39:44.206 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.198+0000 7fdd04f5c640 1 -- 172.21.15.16:0/2257427040 shutdown_connections 2023-05-22T15:39:44.206 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.198+0000 7fdd04f5c640 1 --2- 172.21.15.16:0/2257427040 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fdce0065530 0x7fdce00679e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:44.207 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.198+0000 7fdd04f5c640 1 --2- 172.21.15.16:0/2257427040 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fdd0015cfa0 0x7fdd000b29e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:44.207 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.198+0000 7fdd04f5c640 1 --2- 172.21.15.16:0/2257427040 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdd000cf870 0x7fdd00151d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:44.207 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.198+0000 7fdd04f5c640 1 --2- 172.21.15.16:0/2257427040 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdd0006e740 0x7fdd00149810 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:44.207 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.198+0000 7fdd04f5c640 1 -- 172.21.15.16:0/2257427040 >> 172.21.15.16:0/2257427040 conn(0x7fdd000cc870 msgr2=0x7fdd0015bec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:39:44.207 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.198+0000 7fdd04f5c640 1 -- 172.21.15.16:0/2257427040 shutdown_connections 2023-05-22T15:39:44.208 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.198+0000 7fdd04f5c640 1 -- 172.21.15.16:0/2257427040 wait complete. 2023-05-22T15:39:44.222 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-05-22T15:39:44.338 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.330+0000 7f5432544640 1 Processor -- start 2023-05-22T15:39:44.338 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.334+0000 7f5432544640 1 -- start start 2023-05-22T15:39:44.340 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.334+0000 7f5432544640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f542c0c67d0 0x7f542c0cf730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:44.341 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.334+0000 7f5432544640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f542c0b2880 0x7f542c0cbfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:44.341 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.334+0000 7f5432544640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f542c0cfc70 con 0x7f542c0cc520 2023-05-22T15:39:44.341 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.334+0000 7f5432544640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f542c15a300 con 0x7f542c0c67d0 2023-05-22T15:39:44.341 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.334+0000 7f5432544640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f542c0c4dc0 con 0x7f542c0b2880 2023-05-22T15:39:44.341 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.334+0000 7f5430ae1640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f542c0cc520 0x7f542c0ce160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:56526/0 (socket says 172.21.15.16:56526) 2023-05-22T15:39:44.341 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.334+0000 7f5430ae1640 1 -- 172.21.15.16:0/251914990 learned_addr learned my addr 172.21.15.16:0/251914990 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:39:44.341 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.334+0000 7f542bfff640 1 --2- 172.21.15.16:0/251914990 >> v2:172.21.15.62:3300/0 conn(0x7f542c0c67d0 0x7f542c0cf730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:44.341 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.334+0000 7f542b7fe640 1 --2- 172.21.15.16:0/251914990 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f542c0b2880 0x7f542c0cbfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:44.342 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.334+0000 7f542affd640 1 -- 172.21.15.16:0/251914990 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3776557906 0 0) 0x7f542c0cfc70 con 0x7f542c0cc520 2023-05-22T15:39:44.342 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.334+0000 7f542affd640 1 -- 172.21.15.16:0/251914990 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5418003530 con 0x7f542c0cc520 2023-05-22T15:39:44.342 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.334+0000 7f542affd640 1 -- 172.21.15.16:0/251914990 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1219332101 0 0) 0x7f5418003530 con 0x7f542c0cc520 2023-05-22T15:39:44.342 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.334+0000 7f542affd640 1 -- 172.21.15.16:0/251914990 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f542c0b2880 msgr2=0x7f542c0cbfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:39:44.342 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.334+0000 7f542affd640 1 --2- 172.21.15.16:0/251914990 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f542c0b2880 0x7f542c0cbfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:44.342 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.334+0000 7f542affd640 1 -- 172.21.15.16:0/251914990 >> v2:172.21.15.62:3300/0 conn(0x7f542c0c67d0 msgr2=0x7f542c0cf730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:39:44.343 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.334+0000 7f542affd640 1 --2- 172.21.15.16:0/251914990 >> v2:172.21.15.62:3300/0 conn(0x7f542c0c67d0 0x7f542c0cf730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:44.343 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.334+0000 7f542affd640 1 -- 172.21.15.16:0/251914990 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f542c0c6ba0 con 0x7f542c0cc520 2023-05-22T15:39:44.343 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.334+0000 7f542affd640 1 -- 172.21.15.16:0/251914990 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7f5424001a00 con 0x7f542c0cc520 2023-05-22T15:39:44.343 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.334+0000 7f542affd640 1 -- 172.21.15.16:0/251914990 >> v1:172.21.15.16:6789/0 conn(0x7f542c0cc520 legacy=0x7f542c0ce160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:44.343 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.334+0000 7f542affd640 1 --2- 172.21.15.16:0/251914990 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f542c0b2880 0x7f5418003980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:44.343 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.334+0000 7f542affd640 1 --2- 172.21.15.16:0/251914990 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5418003f00 0x7f54180047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:44.343 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.334+0000 7f542affd640 1 --2- 172.21.15.16:0/251914990 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5418004d10 0x7f5418005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:44.344 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.334+0000 7f542affd640 1 -- 172.21.15.16:0/251914990 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f542c15a300 con 0x7f5418004d10 2023-05-22T15:39:44.344 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.334+0000 7f542affd640 1 -- 172.21.15.16:0/251914990 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f542c0c4dc0 con 0x7f5418003f00 2023-05-22T15:39:44.344 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.334+0000 7f542affd640 1 -- 172.21.15.16:0/251914990 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f541800ba70 con 0x7f542c0b2880 2023-05-22T15:39:44.344 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.334+0000 7f5430ae1640 1 --2- 172.21.15.16:0/251914990 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5418004d10 0x7f5418005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:44.344 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.334+0000 7f542b7fe640 1 --2- 172.21.15.16:0/251914990 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5418003f00 0x7f54180047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:44.344 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.334+0000 7f542bfff640 1 --2- 172.21.15.16:0/251914990 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f542c0b2880 0x7f5418003980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:44.345 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.334+0000 7f5430ae1640 1 -- 172.21.15.16:0/251914990 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f542c0b2880 msgr2=0x7f5418003980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:39:44.345 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.334+0000 7f5430ae1640 1 --2- 172.21.15.16:0/251914990 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f542c0b2880 0x7f5418003980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:44.345 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.334+0000 7f5430ae1640 1 -- 172.21.15.16:0/251914990 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5418003f00 msgr2=0x7f54180047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:39:44.345 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.334+0000 7f5430ae1640 1 --2- 172.21.15.16:0/251914990 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5418003f00 0x7f54180047d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:44.345 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.334+0000 7f5430ae1640 1 -- 172.21.15.16:0/251914990 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5418003530 con 0x7f5418004d10 2023-05-22T15:39:44.347 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.338+0000 7f5430ae1640 1 --2- 172.21.15.16:0/251914990 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5418004d10 0x7f5418005180 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7f54240013e0 tx=0x7f542401fb00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2dcbdbc80e659216 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:39:44.347 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.338+0000 7f542affd640 1 -- 172.21.15.16:0/251914990 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54240aa6a0 con 0x7f5418004d10 2023-05-22T15:39:44.347 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.338+0000 7f542affd640 1 -- 172.21.15.16:0/251914990 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54240aa800 con 0x7f5418004d10 2023-05-22T15:39:44.347 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.338+0000 7f542affd640 1 -- 172.21.15.16:0/251914990 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54240aabb0 con 0x7f5418004d10 2023-05-22T15:39:44.347 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.338+0000 7f5432544640 1 -- 172.21.15.16:0/251914990 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5418004d10 msgr2=0x7f5418005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:44.347 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.338+0000 7f5432544640 1 --2- 172.21.15.16:0/251914990 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5418004d10 0x7f5418005180 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7f54240013e0 tx=0x7f542401fb00 comp rx=0 tx=0).stop 2023-05-22T15:39:44.348 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.338+0000 7f5430ae1640 1 -- 172.21.15.16:0/251914990 reap_dead start 2023-05-22T15:39:44.348 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.338+0000 7f5432544640 1 -- 172.21.15.16:0/251914990 shutdown_connections 2023-05-22T15:39:44.348 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.338+0000 7f5432544640 1 -- 172.21.15.16:0/251914990 >> 172.21.15.16:0/251914990 conn(0x7f542c06f5a0 msgr2=0x7f542c06f9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:39:44.348 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.338+0000 7f5432544640 1 -- 172.21.15.16:0/251914990 shutdown_connections 2023-05-22T15:39:44.348 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.338+0000 7f5432544640 1 -- 172.21.15.16:0/251914990 wait complete. 2023-05-22T15:39:44.348 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.338+0000 7f5432544640 1 Processor -- start 2023-05-22T15:39:44.348 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.338+0000 7f5432544640 1 -- start start 2023-05-22T15:39:44.349 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.338+0000 7f5432544640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5418004d10 0x7f542c0b5560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:44.349 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.338+0000 7f5432544640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f542c1eec70 0x7f542c1509f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:44.349 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.338+0000 7f5432544640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f542c150f30 0x7f542c152640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:44.349 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.338+0000 7f5432544640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f542c0b5140 con 0x7f5418004d10 2023-05-22T15:39:44.349 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.338+0000 7f5432544640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f542c14f8c0 con 0x7f542c150f30 2023-05-22T15:39:44.349 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.338+0000 7f5432544640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f542c0b5aa0 con 0x7f542c1eec70 2023-05-22T15:39:44.349 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.338+0000 7f542bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5418004d10 0x7f542c0b5560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:44.350 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.338+0000 7f542bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5418004d10 0x7f542c0b5560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52698/0 (socket says 172.21.15.16:52698) 2023-05-22T15:39:44.350 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.338+0000 7f542bfff640 1 -- 172.21.15.16:0/189860772 learned_addr learned my addr 172.21.15.16:0/189860772 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:39:44.350 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.342+0000 7f5430ae1640 1 --2- 172.21.15.16:0/189860772 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f542c150f30 0x7f542c152640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:44.350 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.342+0000 7f542bfff640 1 -- 172.21.15.16:0/189860772 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f542c1eec70 msgr2=0x7f542c1509f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-05-22T15:39:44.350 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.342+0000 7f542bfff640 1 --2- 172.21.15.16:0/189860772 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f542c1eec70 0x7f542c1509f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:44.350 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.342+0000 7f542bfff640 1 -- 172.21.15.16:0/189860772 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f542c150f30 msgr2=0x7f542c152640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:44.350 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.342+0000 7f542bfff640 1 --2- 172.21.15.16:0/189860772 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f542c150f30 0x7f542c152640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:44.351 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.342+0000 7f542bfff640 1 -- 172.21.15.16:0/189860772 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f542c0d01d0 con 0x7f5418004d10 2023-05-22T15:39:44.351 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.342+0000 7f5430ae1640 1 --2- 172.21.15.16:0/189860772 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f542c150f30 0x7f542c152640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:39:44.351 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.342+0000 7f542bfff640 1 --2- 172.21.15.16:0/189860772 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5418004d10 0x7f542c0b5560 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f541c0108c0 tx=0x7f541c010d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:39:44.351 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.342+0000 7f5428ff9640 1 -- 172.21.15.16:0/189860772 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f541c009530 con 0x7f5418004d10 2023-05-22T15:39:44.351 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.342+0000 7f5428ff9640 1 -- 172.21.15.16:0/189860772 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f541c00eb60 con 0x7f5418004d10 2023-05-22T15:39:44.351 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.342+0000 7f5428ff9640 1 -- 172.21.15.16:0/189860772 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f541c01c400 con 0x7f5418004d10 2023-05-22T15:39:44.351 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.342+0000 7f5432544640 1 -- 172.21.15.16:0/189860772 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f542c0cfef0 con 0x7f5418004d10 2023-05-22T15:39:44.352 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.342+0000 7f5432544640 1 -- 172.21.15.16:0/189860772 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f542c0d04b0 con 0x7f5418004d10 2023-05-22T15:39:44.352 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.342+0000 7f5432544640 1 -- 172.21.15.16:0/189860772 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f542c0b5280 con 0x7f5418004d10 2023-05-22T15:39:44.353 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.342+0000 7f5428ff9640 1 -- 172.21.15.16:0/189860772 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 13) v1 ==== 84401+0+0 (secure 0 0 0) 0x7f541c01c560 con 0x7f5418004d10 2023-05-22T15:39:44.353 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.342+0000 7f5428ff9640 1 --2- 172.21.15.16:0/189860772 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f540c065530 0x7f540c0679e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:44.353 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.342+0000 7f542b7fe640 1 -- 172.21.15.16:0/189860772 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f540c065530 msgr2=0x7f540c0679e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/36921 2023-05-22T15:39:44.353 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.342+0000 7f5428ff9640 1 -- 172.21.15.16:0/189860772 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 7330+0+0 (secure 0 0 0) 0x7f541c083490 con 0x7f5418004d10 2023-05-22T15:39:44.353 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.342+0000 7f542b7fe640 1 --2- 172.21.15.16:0/189860772 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f540c065530 0x7f540c0679e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-05-22T15:39:44.354 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.346+0000 7f5428ff9640 1 -- 172.21.15.16:0/189860772 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f542c0b5280 con 0x7f5418004d10 2023-05-22T15:39:44.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.506+0000 7f5432544640 1 -- 172.21.15.16:0/189860772 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f542c0b5280 con 0x7f5418004d10 2023-05-22T15:39:44.513 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.506+0000 7f5428ff9640 1 -- 172.21.15.16:0/189860772 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v58) v1 ==== 92+0+10 (secure 0 0 0) 0x7f542c0b5280 con 0x7f5418004d10 2023-05-22T15:39:44.513 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 8 2023-05-22T15:39:44.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.510+0000 7f5432544640 1 -- 172.21.15.16:0/189860772 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f540c065530 msgr2=0x7f540c0679e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-05-22T15:39:44.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.510+0000 7f5432544640 1 --2- 172.21.15.16:0/189860772 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f540c065530 0x7f540c0679e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:44.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.510+0000 7f5432544640 1 -- 172.21.15.16:0/189860772 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5418004d10 msgr2=0x7f542c0b5560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:44.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.510+0000 7f5432544640 1 --2- 172.21.15.16:0/189860772 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5418004d10 0x7f542c0b5560 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f541c0108c0 tx=0x7f541c010d80 comp rx=0 tx=0).stop 2023-05-22T15:39:44.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.510+0000 7f5432544640 1 -- 172.21.15.16:0/189860772 shutdown_connections 2023-05-22T15:39:44.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.510+0000 7f5432544640 1 --2- 172.21.15.16:0/189860772 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f542c150f30 0x7f542c152640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:44.516 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.510+0000 7f5432544640 1 --2- 172.21.15.16:0/189860772 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f540c065530 0x7f540c0679e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:44.516 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.510+0000 7f5432544640 1 --2- 172.21.15.16:0/189860772 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f542c1eec70 0x7f542c1509f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:44.516 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.510+0000 7f5432544640 1 --2- 172.21.15.16:0/189860772 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5418004d10 0x7f542c0b5560 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:44.516 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.510+0000 7f5432544640 1 -- 172.21.15.16:0/189860772 >> 172.21.15.16:0/189860772 conn(0x7f542c06f5a0 msgr2=0x7f542c0ce160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:39:44.516 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.510+0000 7f5432544640 1 -- 172.21.15.16:0/189860772 shutdown_connections 2023-05-22T15:39:44.517 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.510+0000 7f5432544640 1 -- 172.21.15.16:0/189860772 wait complete. 2023-05-22T15:39:44.531 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-05-22T15:39:44.628 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:39:44.622+0000 7f3646e83180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-05-22T15:39:44.642 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:39:44.641+0000 7f6e633d5180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-05-22T15:39:44.662 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.658+0000 7ff7d3d8a640 1 Processor -- start 2023-05-22T15:39:44.663 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.658+0000 7ff7d3d8a640 1 -- start start 2023-05-22T15:39:44.663 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.658+0000 7ff7d3d8a640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7ff7cc0cc6f0 0x7ff7cc164a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:44.664 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.658+0000 7ff7d3d8a640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7cc0c4dc0 con 0x7ff7cc0c67d0 2023-05-22T15:39:44.664 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.658+0000 7ff7d3d8a640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7cc0c6ba0 con 0x7ff7cc0b2880 2023-05-22T15:39:44.664 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.658+0000 7ff7d3d8a640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7cc15a300 con 0x7ff7cc0cc6f0 2023-05-22T15:39:44.664 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.658+0000 7ff7d1b26640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7ff7cc0c67d0 0x7ff7cc0cf730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:56532/0 (socket says 172.21.15.16:56532) 2023-05-22T15:39:44.664 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.658+0000 7ff7d1b26640 1 -- 172.21.15.16:0/3721660203 learned_addr learned my addr 172.21.15.16:0/3721660203 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:39:44.665 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.658+0000 7ff7d2327640 1 --2- 172.21.15.16:0/3721660203 >> v2:172.21.15.62:3300/0 conn(0x7ff7cc0cc6f0 0x7ff7cc164a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:44.666 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.658+0000 7ff7d0b24640 1 -- 172.21.15.16:0/3721660203 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1114077427 0 0) 0x7ff7cc0c4dc0 con 0x7ff7cc0c67d0 2023-05-22T15:39:44.666 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.658+0000 7ff7d0b24640 1 -- 172.21.15.16:0/3721660203 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff7b4003530 con 0x7ff7cc0c67d0 2023-05-22T15:39:44.666 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.658+0000 7ff7d0b24640 1 -- 172.21.15.16:0/3721660203 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3687820058 0 0) 0x7ff7b4003530 con 0x7ff7cc0c67d0 2023-05-22T15:39:44.666 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.658+0000 7ff7d0b24640 1 -- 172.21.15.16:0/3721660203 >> v2:172.21.15.62:3300/0 conn(0x7ff7cc0cc6f0 msgr2=0x7ff7cc164a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:39:44.666 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.658+0000 7ff7d0b24640 1 --2- 172.21.15.16:0/3721660203 >> v2:172.21.15.62:3300/0 conn(0x7ff7cc0cc6f0 0x7ff7cc164a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:44.666 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.658+0000 7ff7d0b24640 1 -- 172.21.15.16:0/3721660203 >> v1:172.21.15.62:6789/0 conn(0x7ff7cc0b2880 legacy=0x7ff7cc0cbfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:44.666 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.658+0000 7ff7d0b24640 1 -- 172.21.15.16:0/3721660203 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7cc0cfef0 con 0x7ff7cc0c67d0 2023-05-22T15:39:44.667 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.658+0000 7ff7d0b24640 1 -- 172.21.15.16:0/3721660203 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7ff7c4001a00 con 0x7ff7cc0c67d0 2023-05-22T15:39:44.667 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.658+0000 7ff7d0b24640 1 -- 172.21.15.16:0/3721660203 >> v1:172.21.15.16:6789/0 conn(0x7ff7cc0c67d0 legacy=0x7ff7cc0cf730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:44.667 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.658+0000 7ff7d0b24640 1 --2- 172.21.15.16:0/3721660203 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff7b4003980 0x7ff7b4003d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:44.667 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.658+0000 7ff7d0b24640 1 --2- 172.21.15.16:0/3721660203 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff7b4004370 0x7ff7b4004830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:44.668 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.658+0000 7ff7d0b24640 1 --2- 172.21.15.16:0/3721660203 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff7b40050d0 0x7ff7b400b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:44.668 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.658+0000 7ff7d0b24640 1 -- 172.21.15.16:0/3721660203 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7cc15a300 con 0x7ff7b4004370 2023-05-22T15:39:44.668 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.658+0000 7ff7d0b24640 1 -- 172.21.15.16:0/3721660203 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7b400c000 con 0x7ff7b40050d0 2023-05-22T15:39:44.668 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.658+0000 7ff7d0b24640 1 -- 172.21.15.16:0/3721660203 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7b400c1c0 con 0x7ff7b4003980 2023-05-22T15:39:44.669 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.658+0000 7ff7d1325640 1 --2- 172.21.15.16:0/3721660203 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff7b4004370 0x7ff7b4004830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:44.669 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.658+0000 7ff7d1b26640 1 --2- 172.21.15.16:0/3721660203 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff7b4003980 0x7ff7b4003d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:44.669 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.658+0000 7ff7d2327640 1 --2- 172.21.15.16:0/3721660203 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff7b40050d0 0x7ff7b400b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:44.669 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.658+0000 7ff7d1325640 1 -- 172.21.15.16:0/3721660203 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff7b4003980 msgr2=0x7ff7b4003d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:39:44.669 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.658+0000 7ff7d1325640 1 --2- 172.21.15.16:0/3721660203 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff7b4003980 0x7ff7b4003d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:44.669 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.658+0000 7ff7d1325640 1 -- 172.21.15.16:0/3721660203 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff7b40050d0 msgr2=0x7ff7b400b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:39:44.669 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.658+0000 7ff7d1325640 1 --2- 172.21.15.16:0/3721660203 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff7b40050d0 0x7ff7b400b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:44.670 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.658+0000 7ff7d1325640 1 -- 172.21.15.16:0/3721660203 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7b4003530 con 0x7ff7b4004370 2023-05-22T15:39:44.670 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.658+0000 7ff7d2327640 1 -- 172.21.15.16:0/3721660203 reap_dead start 2023-05-22T15:39:44.672 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.662+0000 7ff7d1325640 1 --2- 172.21.15.16:0/3721660203 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff7b4004370 0x7ff7b4004830 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7ff7c0001910 tx=0x7ff7c001c4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a183ecb2b24475f6 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:39:44.672 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.662+0000 7ff7d0b24640 1 -- 172.21.15.16:0/3721660203 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7c00ab600 con 0x7ff7b4004370 2023-05-22T15:39:44.672 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.662+0000 7ff7d0b24640 1 -- 172.21.15.16:0/3721660203 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7c00a7050 con 0x7ff7b4004370 2023-05-22T15:39:44.672 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.662+0000 7ff7d0b24640 1 -- 172.21.15.16:0/3721660203 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7c00ab9c0 con 0x7ff7b4004370 2023-05-22T15:39:44.672 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.662+0000 7ff7d3d8a640 1 -- 172.21.15.16:0/3721660203 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff7b4004370 msgr2=0x7ff7b4004830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:44.673 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.662+0000 7ff7d3d8a640 1 --2- 172.21.15.16:0/3721660203 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff7b4004370 0x7ff7b4004830 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7ff7c0001910 tx=0x7ff7c001c4f0 comp rx=0 tx=0).stop 2023-05-22T15:39:44.673 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.662+0000 7ff7d3d8a640 1 -- 172.21.15.16:0/3721660203 shutdown_connections 2023-05-22T15:39:44.673 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.662+0000 7ff7d3d8a640 1 --2- 172.21.15.16:0/3721660203 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff7b4004370 0x7ff7b4004830 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:44.673 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.662+0000 7ff7d3d8a640 1 -- 172.21.15.16:0/3721660203 >> 172.21.15.16:0/3721660203 conn(0x7ff7cc06f5a0 msgr2=0x7ff7cc06f9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:39:44.673 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.662+0000 7ff7d3d8a640 1 -- 172.21.15.16:0/3721660203 shutdown_connections 2023-05-22T15:39:44.673 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.662+0000 7ff7d3d8a640 1 -- 172.21.15.16:0/3721660203 wait complete. 2023-05-22T15:39:44.673 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.662+0000 7ff7d3d8a640 1 Processor -- start 2023-05-22T15:39:44.674 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.662+0000 7ff7d3d8a640 1 -- start start 2023-05-22T15:39:44.674 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.662+0000 7ff7d3d8a640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff7b4004370 0x7ff7cc15cfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:44.674 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.662+0000 7ff7d3d8a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff7cc150a70 0x7ff7cc15b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:44.674 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.662+0000 7ff7d3d8a640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff7cc1503a0 0x7ff7cc0c9740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:44.674 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.662+0000 7ff7d3d8a640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7cc150840 con 0x7ff7cc150a70 2023-05-22T15:39:44.674 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.662+0000 7ff7d3d8a640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7cc0c8bd0 con 0x7ff7b4004370 2023-05-22T15:39:44.674 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.662+0000 7ff7d3d8a640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7cc15aef0 con 0x7ff7cc1503a0 2023-05-22T15:39:44.675 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.666+0000 7ff7d1325640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff7cc150a70 0x7ff7cc15b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:44.675 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.666+0000 7ff7d1325640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff7cc150a70 0x7ff7cc15b9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52724/0 (socket says 172.21.15.16:52724) 2023-05-22T15:39:44.675 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.666+0000 7ff7d1325640 1 -- 172.21.15.16:0/1023467573 learned_addr learned my addr 172.21.15.16:0/1023467573 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:39:44.675 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.666+0000 7ff7d2327640 1 --2- 172.21.15.16:0/1023467573 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff7cc1503a0 0x7ff7cc0c9740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:44.675 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.666+0000 7ff7d1b26640 1 --2- 172.21.15.16:0/1023467573 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff7b4004370 0x7ff7cc15cfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:44.675 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.666+0000 7ff7d1325640 1 -- 172.21.15.16:0/1023467573 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff7cc1503a0 msgr2=0x7ff7cc0c9740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:44.675 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.666+0000 7ff7d1325640 1 --2- 172.21.15.16:0/1023467573 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff7cc1503a0 0x7ff7cc0c9740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:44.676 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.666+0000 7ff7d1325640 1 -- 172.21.15.16:0/1023467573 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff7b4004370 msgr2=0x7ff7cc15cfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:44.676 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.666+0000 7ff7d1325640 1 --2- 172.21.15.16:0/1023467573 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff7b4004370 0x7ff7cc15cfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:44.676 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.666+0000 7ff7d1325640 1 -- 172.21.15.16:0/1023467573 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7cc0d0a70 con 0x7ff7cc150a70 2023-05-22T15:39:44.676 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.666+0000 7ff7d2327640 1 --2- 172.21.15.16:0/1023467573 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff7cc1503a0 0x7ff7cc0c9740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T15:39:44.676 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.666+0000 7ff7d1325640 1 --2- 172.21.15.16:0/1023467573 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff7cc150a70 0x7ff7cc15b9d0 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7ff7c001c320 tx=0x7ff7c0019290 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:39:44.676 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.666+0000 7ff7ca7fc640 1 -- 172.21.15.16:0/1023467573 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7c00abb20 con 0x7ff7cc150a70 2023-05-22T15:39:44.676 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.666+0000 7ff7ca7fc640 1 -- 172.21.15.16:0/1023467573 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7c00abc80 con 0x7ff7cc150a70 2023-05-22T15:39:44.677 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.666+0000 7ff7d3d8a640 1 -- 172.21.15.16:0/1023467573 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7cc0d0790 con 0x7ff7cc150a70 2023-05-22T15:39:44.677 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.666+0000 7ff7ca7fc640 1 -- 172.21.15.16:0/1023467573 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7c0007020 con 0x7ff7cc150a70 2023-05-22T15:39:44.677 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.666+0000 7ff7d3d8a640 1 -- 172.21.15.16:0/1023467573 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7cc0d0f90 con 0x7ff7cc150a70 2023-05-22T15:39:44.677 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.666+0000 7ff7d3d8a640 1 -- 172.21.15.16:0/1023467573 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7cc04dbe0 con 0x7ff7cc150a70 2023-05-22T15:39:44.678 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.666+0000 7ff7ca7fc640 1 -- 172.21.15.16:0/1023467573 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 13) v1 ==== 84401+0+0 (secure 0 0 0) 0x7ff7c0007180 con 0x7ff7cc150a70 2023-05-22T15:39:44.678 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.666+0000 7ff7ca7fc640 1 --2- 172.21.15.16:0/1023467573 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ff7b8065560 0x7ff7b8067a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:44.678 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.666+0000 7ff7d1b26640 1 -- 172.21.15.16:0/1023467573 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ff7b8065560 msgr2=0x7ff7b8067a10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/36921 2023-05-22T15:39:44.678 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.666+0000 7ff7ca7fc640 1 -- 172.21.15.16:0/1023467573 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 7330+0+0 (secure 0 0 0) 0x7ff7c011d320 con 0x7ff7cc150a70 2023-05-22T15:39:44.679 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.666+0000 7ff7d1b26640 1 --2- 172.21.15.16:0/1023467573 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ff7b8065560 0x7ff7b8067a10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-05-22T15:39:44.679 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.670+0000 7ff7ca7fc640 1 -- 172.21.15.16:0/1023467573 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7ff7c00ea7a0 con 0x7ff7cc150a70 2023-05-22T15:39:44.683 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:39:44.678+0000 7f3646e83180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-05-22T15:39:44.697 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:39:44.693+0000 7f6e633d5180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-05-22T15:39:44.735 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:39:44.730+0000 7f3646e83180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-05-22T15:39:44.747 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:39:44.745+0000 7f6e633d5180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-05-22T15:39:44.813 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:39:44.806+0000 7f3646e83180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-05-22T15:39:44.826 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:39:44.825+0000 7f6e633d5180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-05-22T15:39:44.831 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.826+0000 7ff7d3d8a640 1 -- 172.21.15.16:0/1023467573 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7ff7cc0cd490 con 0x7ff7cc150a70 2023-05-22T15:39:44.831 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.826+0000 7ff7ca7fc640 1 -- 172.21.15.16:0/1023467573 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v58) v1 ==== 104+0+11 (secure 0 0 0) 0x7ff7c00eddc0 con 0x7ff7cc150a70 2023-05-22T15:39:44.832 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2023-05-22T15:39:44.834 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.826+0000 7ff7d3d8a640 1 -- 172.21.15.16:0/1023467573 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ff7b8065560 msgr2=0x7ff7b8067a10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-05-22T15:39:44.834 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.826+0000 7ff7d3d8a640 1 --2- 172.21.15.16:0/1023467573 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ff7b8065560 0x7ff7b8067a10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:44.834 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.826+0000 7ff7d3d8a640 1 -- 172.21.15.16:0/1023467573 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff7cc150a70 msgr2=0x7ff7cc15b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:44.834 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.826+0000 7ff7d3d8a640 1 --2- 172.21.15.16:0/1023467573 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff7cc150a70 0x7ff7cc15b9d0 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7ff7c001c320 tx=0x7ff7c0019290 comp rx=0 tx=0).stop 2023-05-22T15:39:44.834 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.826+0000 7ff7d3d8a640 1 -- 172.21.15.16:0/1023467573 shutdown_connections 2023-05-22T15:39:44.834 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.826+0000 7ff7d3d8a640 1 --2- 172.21.15.16:0/1023467573 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ff7b8065560 0x7ff7b8067a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:44.835 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.826+0000 7ff7d3d8a640 1 --2- 172.21.15.16:0/1023467573 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff7cc1503a0 0x7ff7cc0c9740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:44.835 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.826+0000 7ff7d3d8a640 1 --2- 172.21.15.16:0/1023467573 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff7cc150a70 0x7ff7cc15b9d0 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:44.835 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.826+0000 7ff7d3d8a640 1 --2- 172.21.15.16:0/1023467573 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff7b4004370 0x7ff7cc15cfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:44.835 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.826+0000 7ff7d3d8a640 1 -- 172.21.15.16:0/1023467573 >> 172.21.15.16:0/1023467573 conn(0x7ff7cc06f5a0 msgr2=0x7ff7cc0b1cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:39:44.835 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.826+0000 7ff7d3d8a640 1 -- 172.21.15.16:0/1023467573 shutdown_connections 2023-05-22T15:39:44.835 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.826+0000 7ff7d3d8a640 1 -- 172.21.15.16:0/1023467573 wait complete. 2023-05-22T15:39:44.852 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-05-22T15:39:44.867 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:39:44.862+0000 7f3646e83180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-05-22T15:39:44.879 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:39:44.877+0000 7f6e633d5180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-05-22T15:39:44.954 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:39:44.946+0000 7f3646e83180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-05-22T15:39:44.960 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:39:44.957+0000 7f6e633d5180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-05-22T15:39:44.974 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.970+0000 7fbc36d8d640 1 Processor -- start 2023-05-22T15:39:44.974 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.970+0000 7fbc36d8d640 1 -- start start 2023-05-22T15:39:44.977 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.970+0000 7fbc36d8d640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fbc300c67d0 0x7fbc300cf730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:44.977 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.970+0000 7fbc36d8d640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fbc300b2880 0x7fbc300cbfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:44.978 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.970+0000 7fbc36d8d640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbc300cc520 0x7fbc300ce160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:44.978 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.970+0000 7fbc36d8d640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc3015a300 con 0x7fbc300c67d0 2023-05-22T15:39:44.978 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.970+0000 7fbc36d8d640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc300cfc70 con 0x7fbc300b2880 2023-05-22T15:39:44.979 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.970+0000 7fbc36d8d640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc300c6ba0 con 0x7fbc300cc520 2023-05-22T15:39:44.979 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.970+0000 7fbc34b29640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fbc300c67d0 0x7fbc300cf730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:44.979 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.970+0000 7fbc34b29640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fbc300c67d0 0x7fbc300cf730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52734/0 (socket says 172.21.15.16:52734) 2023-05-22T15:39:44.979 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.970+0000 7fbc34b29640 1 -- 172.21.15.16:0/1436324948 learned_addr learned my addr 172.21.15.16:0/1436324948 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:39:44.980 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.970+0000 7fbc3532a640 1 --2- 172.21.15.16:0/1436324948 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbc300cc520 0x7fbc300ce160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:44.980 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.970+0000 7fbc2ffff640 1 --2- 172.21.15.16:0/1436324948 >> v2:172.21.15.62:3300/0 conn(0x7fbc300b2880 0x7fbc300cbfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:44.980 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.970+0000 7fbc34b29640 1 -- 172.21.15.16:0/1436324948 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbc300cc520 msgr2=0x7fbc300ce160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:39:44.981 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.970+0000 7fbc34b29640 1 --2- 172.21.15.16:0/1436324948 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbc300cc520 0x7fbc300ce160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:44.981 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.970+0000 7fbc34b29640 1 -- 172.21.15.16:0/1436324948 >> v2:172.21.15.62:3300/0 conn(0x7fbc300b2880 msgr2=0x7fbc300cbfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:39:44.981 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.970+0000 7fbc34b29640 1 --2- 172.21.15.16:0/1436324948 >> v2:172.21.15.62:3300/0 conn(0x7fbc300b2880 0x7fbc300cbfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:44.981 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.970+0000 7fbc34b29640 1 -- 172.21.15.16:0/1436324948 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc300cfef0 con 0x7fbc300c67d0 2023-05-22T15:39:44.985 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.974+0000 7fbc34b29640 1 --2- 172.21.15.16:0/1436324948 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc300c67d0 0x7fbc300cf730 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7fbc300ccc40 tx=0x7fbc2401c180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ef87292367287ae server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:39:44.986 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.974+0000 7fbc2f7fe640 1 -- 172.21.15.16:0/1436324948 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc240ab600 con 0x7fbc300c67d0 2023-05-22T15:39:44.986 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.974+0000 7fbc2f7fe640 1 -- 172.21.15.16:0/1436324948 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc240a7050 con 0x7fbc300c67d0 2023-05-22T15:39:44.986 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.974+0000 7fbc2f7fe640 1 -- 172.21.15.16:0/1436324948 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc240ab980 con 0x7fbc300c67d0 2023-05-22T15:39:44.987 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.974+0000 7fbc36d8d640 1 -- 172.21.15.16:0/1436324948 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc300c67d0 msgr2=0x7fbc300cf730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:44.987 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.974+0000 7fbc36d8d640 1 --2- 172.21.15.16:0/1436324948 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc300c67d0 0x7fbc300cf730 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7fbc300ccc40 tx=0x7fbc2401c180 comp rx=0 tx=0).stop 2023-05-22T15:39:44.988 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.974+0000 7fbc36d8d640 1 -- 172.21.15.16:0/1436324948 shutdown_connections 2023-05-22T15:39:44.988 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.974+0000 7fbc36d8d640 1 --2- 172.21.15.16:0/1436324948 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbc300cc520 0x7fbc300ce160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:44.988 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.974+0000 7fbc36d8d640 1 --2- 172.21.15.16:0/1436324948 >> v2:172.21.15.62:3300/0 conn(0x7fbc300b2880 0x7fbc300cbfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:44.988 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.974+0000 7fbc36d8d640 1 --2- 172.21.15.16:0/1436324948 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc300c67d0 0x7fbc300cf730 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:44.989 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.974+0000 7fbc36d8d640 1 -- 172.21.15.16:0/1436324948 >> 172.21.15.16:0/1436324948 conn(0x7fbc3006f5a0 msgr2=0x7fbc3006f9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:39:44.989 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.974+0000 7fbc36d8d640 1 -- 172.21.15.16:0/1436324948 shutdown_connections 2023-05-22T15:39:44.989 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.974+0000 7fbc36d8d640 1 -- 172.21.15.16:0/1436324948 wait complete. 2023-05-22T15:39:44.989 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.974+0000 7fbc36d8d640 1 Processor -- start 2023-05-22T15:39:44.990 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.974+0000 7fbc36d8d640 1 -- start start 2023-05-22T15:39:44.990 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.974+0000 7fbc36d8d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbc300b2880 0x7fbc3015be80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:44.990 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.974+0000 7fbc36d8d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc300c67d0 0x7fbc3014a770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:44.990 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.974+0000 7fbc36d8d640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbc300cc520 0x7fbc300c8220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:44.991 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.974+0000 7fbc36d8d640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc3014b920 con 0x7fbc300c67d0 2023-05-22T15:39:44.991 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.974+0000 7fbc36d8d640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc3015c3c0 con 0x7fbc300b2880 2023-05-22T15:39:44.991 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.974+0000 7fbc36d8d640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc3015c660 con 0x7fbc300cc520 2023-05-22T15:39:44.991 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.974+0000 7fbc2ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc300c67d0 0x7fbc3014a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:44.992 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.974+0000 7fbc2ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc300c67d0 0x7fbc3014a770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52746/0 (socket says 172.21.15.16:52746) 2023-05-22T15:39:44.992 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.974+0000 7fbc2ffff640 1 -- 172.21.15.16:0/2764745949 learned_addr learned my addr 172.21.15.16:0/2764745949 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:39:44.992 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.974+0000 7fbc3532a640 1 --2- 172.21.15.16:0/2764745949 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbc300cc520 0x7fbc300c8220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:44.993 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.974+0000 7fbc34b29640 1 --2- 172.21.15.16:0/2764745949 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbc300b2880 0x7fbc3015be80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:44.993 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.974+0000 7fbc2ffff640 1 -- 172.21.15.16:0/2764745949 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbc300cc520 msgr2=0x7fbc300c8220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:44.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.974+0000 7fbc2ffff640 1 --2- 172.21.15.16:0/2764745949 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbc300cc520 0x7fbc300c8220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:44.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.974+0000 7fbc2ffff640 1 -- 172.21.15.16:0/2764745949 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbc300b2880 msgr2=0x7fbc3015be80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:44.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.974+0000 7fbc2ffff640 1 --2- 172.21.15.16:0/2764745949 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbc300b2880 0x7fbc3015be80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:44.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.974+0000 7fbc2ffff640 1 -- 172.21.15.16:0/2764745949 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc240094d0 con 0x7fbc300c67d0 2023-05-22T15:39:44.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.974+0000 7fbc2ffff640 1 --2- 172.21.15.16:0/2764745949 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc300c67d0 0x7fbc3014a770 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7fbc1c00d5f0 tx=0x7fbc1c00dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:39:44.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.974+0000 7fbc2d7fa640 1 -- 172.21.15.16:0/2764745949 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc1c0125f0 con 0x7fbc300c67d0 2023-05-22T15:39:44.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.978+0000 7fbc2d7fa640 1 -- 172.21.15.16:0/2764745949 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc1c012750 con 0x7fbc300c67d0 2023-05-22T15:39:44.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.978+0000 7fbc36d8d640 1 -- 172.21.15.16:0/2764745949 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc300d0a70 con 0x7fbc300c67d0 2023-05-22T15:39:44.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.978+0000 7fbc2d7fa640 1 -- 172.21.15.16:0/2764745949 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc1c012ae0 con 0x7fbc300c67d0 2023-05-22T15:39:44.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.978+0000 7fbc36d8d640 1 -- 172.21.15.16:0/2764745949 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc300d0790 con 0x7fbc300c67d0 2023-05-22T15:39:44.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.978+0000 7fbc36d8d640 1 -- 172.21.15.16:0/2764745949 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc300d0f90 con 0x7fbc300c67d0 2023-05-22T15:39:44.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.978+0000 7fbc2d7fa640 1 -- 172.21.15.16:0/2764745949 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 13) v1 ==== 84401+0+0 (secure 0 0 0) 0x7fbc1c012c40 con 0x7fbc300c67d0 2023-05-22T15:39:44.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.978+0000 7fbc2d7fa640 1 --2- 172.21.15.16:0/2764745949 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fbc14065530 0x7fbc140679e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:44.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.978+0000 7fbc34b29640 1 -- 172.21.15.16:0/2764745949 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fbc14065530 msgr2=0x7fbc140679e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/36921 2023-05-22T15:39:44.998 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.978+0000 7fbc2d7fa640 1 -- 172.21.15.16:0/2764745949 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 7330+0+0 (secure 0 0 0) 0x7fbc1c083f50 con 0x7fbc300c67d0 2023-05-22T15:39:44.998 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.978+0000 7fbc34b29640 1 --2- 172.21.15.16:0/2764745949 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fbc14065530 0x7fbc140679e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-05-22T15:39:44.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:44.982+0000 7fbc2d7fa640 1 -- 172.21.15.16:0/2764745949 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fbc1c051440 con 0x7fbc300c67d0 2023-05-22T15:39:45.014 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:39:45.010+0000 7f3646e83180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-05-22T15:39:45.015 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:39:45.013+0000 7f6e633d5180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-05-22T15:39:45.172 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:45.166+0000 7fbc36d8d640 1 -- 172.21.15.16:0/2764745949 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fbc3004dbe0 con 0x7fbc300c67d0 2023-05-22T15:39:45.173 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:45.166+0000 7fbc2d7fa640 1 -- 172.21.15.16:0/2764745949 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v58) v1 ==== 100+0+11 (secure 0 0 0) 0x7fbc1c081e60 con 0x7fbc300c67d0 2023-05-22T15:39:45.173 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 64 2023-05-22T15:39:45.174 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:45.170+0000 7fbc36d8d640 1 -- 172.21.15.16:0/2764745949 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fbc14065530 msgr2=0x7fbc140679e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-05-22T15:39:45.174 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:45.170+0000 7fbc36d8d640 1 --2- 172.21.15.16:0/2764745949 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fbc14065530 0x7fbc140679e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:45.174 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:45.170+0000 7fbc36d8d640 1 -- 172.21.15.16:0/2764745949 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc300c67d0 msgr2=0x7fbc3014a770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:45.175 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:45.170+0000 7fbc36d8d640 1 --2- 172.21.15.16:0/2764745949 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc300c67d0 0x7fbc3014a770 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7fbc1c00d5f0 tx=0x7fbc1c00dab0 comp rx=0 tx=0).stop 2023-05-22T15:39:45.175 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:45.170+0000 7fbc36d8d640 1 -- 172.21.15.16:0/2764745949 shutdown_connections 2023-05-22T15:39:45.175 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:45.170+0000 7fbc36d8d640 1 --2- 172.21.15.16:0/2764745949 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fbc14065530 0x7fbc140679e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:45.175 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:45.170+0000 7fbc36d8d640 1 --2- 172.21.15.16:0/2764745949 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbc300cc520 0x7fbc300c8220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:45.175 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:45.170+0000 7fbc36d8d640 1 --2- 172.21.15.16:0/2764745949 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc300c67d0 0x7fbc3014a770 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:45.175 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:45.170+0000 7fbc36d8d640 1 --2- 172.21.15.16:0/2764745949 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbc300b2880 0x7fbc3015be80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:45.175 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:45.170+0000 7fbc36d8d640 1 -- 172.21.15.16:0/2764745949 >> 172.21.15.16:0/2764745949 conn(0x7fbc3006f5a0 msgr2=0x7fbc300c7530 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:39:45.176 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:45.170+0000 7fbc36d8d640 1 -- 172.21.15.16:0/2764745949 shutdown_connections 2023-05-22T15:39:45.176 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:45.170+0000 7fbc36d8d640 1 -- 172.21.15.16:0/2764745949 wait complete. 2023-05-22T15:39:45.176 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:39:45.170+0000 7f3646e83180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-05-22T15:39:45.178 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:39:45.177+0000 7f6e633d5180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-05-22T15:39:45.191 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-05-22T15:39:45.233 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:39:45.226+0000 7f3646e83180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-05-22T15:39:45.236 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:39:45.233+0000 7f6e633d5180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-05-22T15:39:45.343 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:45.338+0000 7f8cd5226640 1 Processor -- start 2023-05-22T15:39:45.343 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:45.338+0000 7f8cd5226640 1 -- start start 2023-05-22T15:39:45.344 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:45.338+0000 7f8cd5226640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f8cd015c350 0x7f8cd015c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:45.344 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:45.338+0000 7f8cd5226640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8cd0069720 0x7f8cd0069b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:45.344 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:45.338+0000 7f8cd5226640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cd00c95f0 con 0x7f8cd00ade80 2023-05-22T15:39:45.344 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:45.338+0000 7f8cd5226640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8cd00ca270 con 0x7f8cd015c350 2023-05-22T15:39:45.344 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:45.338+0000 7f8cd5226640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8cd015cd30 con 0x7f8cd0069720 2023-05-22T15:39:45.345 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:45.338+0000 7f8cce59c640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f8cd015c350 0x7f8cd015c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:45.345 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:45.338+0000 7f8ccf59e640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8cd0069720 0x7f8cd0069b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:45.345 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:45.338+0000 7f8cce59c640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f8cd015c350 0x7f8cd015c7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.16:51566/0 (socket says 172.21.15.16:51566) 2023-05-22T15:39:45.346 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:45.338+0000 7f8cce59c640 1 -- 172.21.15.16:0/2574344794 learned_addr learned my addr 172.21.15.16:0/2574344794 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:39:45.346 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:45.338+0000 7f8ccf59e640 1 -- 172.21.15.16:0/2574344794 >> v2:172.21.15.62:3300/0 conn(0x7f8cd015c350 msgr2=0x7f8cd015c7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:39:45.346 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:45.338+0000 7f8ccf59e640 1 --2- 172.21.15.16:0/2574344794 >> v2:172.21.15.62:3300/0 conn(0x7f8cd015c350 0x7f8cd015c7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:45.346 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:45.338+0000 7f8ccf59e640 1 -- 172.21.15.16:0/2574344794 >> v1:172.21.15.16:6789/0 conn(0x7f8cd00ade80 legacy=0x7f8cd00cc400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:39:45.346 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:45.338+0000 7f8ccf59e640 1 -- 172.21.15.16:0/2574344794 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cd00c98d0 con 0x7f8cd0069720 2023-05-22T15:39:45.348 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:45.342+0000 7f8ccf59e640 1 --2- 172.21.15.16:0/2574344794 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8cd0069720 0x7f8cd0069b40 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f8cd006fe80 tx=0x7f8cc801c680 comp rx=0 tx=0).ready entity=mon.2 client_cookie=43263bb3799b67dd server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:39:45.352 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:45.346+0000 7f8ccdd9b640 1 -- 172.21.15.16:0/2574344794 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cc80a9610 con 0x7f8cd0069720 2023-05-22T15:39:45.353 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:45.346+0000 7f8ccdd9b640 1 -- 172.21.15.16:0/2574344794 <== mon.2 v2:172.21.15.62:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8cc80a9770 con 0x7f8cd0069720 2023-05-22T15:39:45.353 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:45.346+0000 7f8ccdd9b640 1 -- 172.21.15.16:0/2574344794 <== mon.2 v2:172.21.15.62:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cc80a9b00 con 0x7f8cd0069720 2023-05-22T15:39:45.353 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:45.346+0000 7f8cd5226640 1 -- 172.21.15.16:0/2574344794 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8cd0069720 msgr2=0x7f8cd0069b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:45.354 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:45.346+0000 7f8cd5226640 1 --2- 172.21.15.16:0/2574344794 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8cd0069720 0x7f8cd0069b40 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f8cd006fe80 tx=0x7f8cc801c680 comp rx=0 tx=0).stop 2023-05-22T15:39:45.354 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:45.346+0000 7f8cd5226640 1 -- 172.21.15.16:0/2574344794 shutdown_connections 2023-05-22T15:39:45.354 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:45.346+0000 7f8cd5226640 1 --2- 172.21.15.16:0/2574344794 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8cd0069720 0x7f8cd0069b40 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:45.354 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:45.346+0000 7f8cd5226640 1 --2- 172.21.15.16:0/2574344794 >> v2:172.21.15.62:3300/0 conn(0x7f8cd015c350 0x7f8cd015c7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:45.354 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:45.346+0000 7f8cd5226640 1 -- 172.21.15.16:0/2574344794 >> 172.21.15.16:0/2574344794 conn(0x7f8cd00b1110 msgr2=0x7f8cd00ce830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:39:45.355 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:45.346+0000 7f8cd5226640 1 -- 172.21.15.16:0/2574344794 shutdown_connections 2023-05-22T15:39:45.355 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:45.346+0000 7f8cd5226640 1 -- 172.21.15.16:0/2574344794 wait complete. 2023-05-22T15:39:45.356 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:45.350+0000 7f8cd5226640 1 Processor -- start 2023-05-22T15:39:45.356 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:45.350+0000 7f8cd5226640 1 -- start start 2023-05-22T15:39:45.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:45.350+0000 7f8cd5226640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8cd0069720 0x7f8cd01421d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:45.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:45.350+0000 7f8cd5226640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8cd00ade80 0x7f8cd00af980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:45.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:45.350+0000 7f8cd5226640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8cd015c350 0x7f8cd0145de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:45.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:45.350+0000 7f8cd5226640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cd00af4f0 con 0x7f8cd0069720 2023-05-22T15:39:45.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:45.350+0000 7f8cd5226640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cd01451e0 con 0x7f8cd015c350 2023-05-22T15:39:45.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:45.350+0000 7f8cd5226640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8cd00ae8e0 con 0x7f8cd00ade80 2023-05-22T15:39:45.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:45.350+0000 7f8cced9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8cd0069720 0x7f8cd01421d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:45.358 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:45.350+0000 7f8cced9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8cd0069720 0x7f8cd01421d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52750/0 (socket says 172.21.15.16:52750) 2023-05-22T15:39:45.358 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:45.350+0000 7f8cced9d640 1 -- 172.21.15.16:0/2051755750 learned_addr learned my addr 172.21.15.16:0/2051755750 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:39:45.358 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:45.350+0000 7f8ccf59e640 1 --2- 172.21.15.16:0/2051755750 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8cd015c350 0x7f8cd0145de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:45.358 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:45.350+0000 7f8cced9d640 1 -- 172.21.15.16:0/2051755750 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8cd00ade80 msgr2=0x7f8cd00af980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:45.359 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:45.350+0000 7f8cced9d640 1 --2- 172.21.15.16:0/2051755750 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8cd00ade80 0x7f8cd00af980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:45.359 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:45.350+0000 7f8cced9d640 1 -- 172.21.15.16:0/2051755750 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8cd015c350 msgr2=0x7f8cd0145de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:45.359 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:45.350+0000 7f8cced9d640 1 --2- 172.21.15.16:0/2051755750 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8cd015c350 0x7f8cd0145de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:45.359 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:45.350+0000 7f8cced9d640 1 -- 172.21.15.16:0/2051755750 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cc8009a70 con 0x7f8cd0069720 2023-05-22T15:39:45.359 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:45.350+0000 7f8cced9d640 1 --2- 172.21.15.16:0/2051755750 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8cd0069720 0x7f8cd01421d0 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f8cc000b3e0 tx=0x7f8cc000b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:39:45.359 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:45.350+0000 7f8cb77fe640 1 -- 172.21.15.16:0/2051755750 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cc0011830 con 0x7f8cd0069720 2023-05-22T15:39:45.360 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:45.350+0000 7f8cb77fe640 1 -- 172.21.15.16:0/2051755750 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8cc00119b0 con 0x7f8cd0069720 2023-05-22T15:39:45.360 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:45.350+0000 7f8cd5226640 1 -- 172.21.15.16:0/2051755750 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8cd00c9210 con 0x7f8cd0069720 2023-05-22T15:39:45.361 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:45.350+0000 7f8cb77fe640 1 -- 172.21.15.16:0/2051755750 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cc0011d60 con 0x7f8cd0069720 2023-05-22T15:39:45.361 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:45.350+0000 7f8cd5226640 1 -- 172.21.15.16:0/2051755750 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8cd015d090 con 0x7f8cd0069720 2023-05-22T15:39:45.361 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:45.350+0000 7f8cd5226640 1 -- 172.21.15.16:0/2051755750 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8cd00c93a0 con 0x7f8cd0069720 2023-05-22T15:39:45.362 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:45.350+0000 7f8cb77fe640 1 -- 172.21.15.16:0/2051755750 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 13) v1 ==== 84401+0+0 (secure 0 0 0) 0x7f8cc001a400 con 0x7f8cd0069720 2023-05-22T15:39:45.362 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:45.354+0000 7f8cb77fe640 1 --2- 172.21.15.16:0/2051755750 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f8cbc065530 0x7f8cbc0679e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:45.362 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:45.354+0000 7f8cce59c640 1 -- 172.21.15.16:0/2051755750 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f8cbc065530 msgr2=0x7f8cbc0679e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/36921 2023-05-22T15:39:45.362 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:45.354+0000 7f8cb77fe640 1 -- 172.21.15.16:0/2051755750 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 7570+0+0 (secure 0 0 0) 0x7f8cc0083330 con 0x7f8cd0069720 2023-05-22T15:39:45.362 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:45.354+0000 7f8cce59c640 1 --2- 172.21.15.16:0/2051755750 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f8cbc065530 0x7f8cbc0679e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-05-22T15:39:45.363 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:45.358+0000 7f8cb77fe640 1 -- 172.21.15.16:0/2051755750 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f8cc00506e0 con 0x7f8cd0069720 2023-05-22T15:39:45.470 INFO:tasks.ceph.mgr.y.smithi016.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2023-05-22T15:39:45.470 INFO:tasks.ceph.mgr.y.smithi016.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2023-05-22T15:39:45.470 INFO:tasks.ceph.mgr.y.smithi016.stderr: from numpy import show_config as show_numpy_config 2023-05-22T15:39:45.472 INFO:tasks.ceph.mgr.x.smithi062.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2023-05-22T15:39:45.473 INFO:tasks.ceph.mgr.x.smithi062.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2023-05-22T15:39:45.473 INFO:tasks.ceph.mgr.x.smithi062.stderr: from numpy import show_config as show_numpy_config 2023-05-22T15:39:45.478 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:39:45.474+0000 7f3646e83180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-05-22T15:39:45.480 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:39:45.477+0000 7f6e633d5180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-05-22T15:39:45.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:45.558+0000 7f8cce59c640 1 -- 172.21.15.16:0/2051755750 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f8cbc065530 msgr2=0x7f8cbc0679e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/36921 2023-05-22T15:39:45.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:45.558+0000 7f8cce59c640 1 --2- 172.21.15.16:0/2051755750 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f8cbc065530 0x7f8cbc0679e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-05-22T15:39:45.576 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:45.570+0000 7f8cd5226640 1 -- 172.21.15.16:0/2051755750 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f8cd004dbe0 con 0x7f8cd0069720 2023-05-22T15:39:45.577 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:45.570+0000 7f8cb77fe640 1 -- 172.21.15.16:0/2051755750 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 6 entries v59) v1 ==== 80+0+343 (secure 0 0 0) 0x7f8cc0053d00 con 0x7f8cd0069720 2023-05-22T15:39:45.578 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/3979017463 2023-05-23T15:28:30.504372+0000 2023-05-22T15:39:45.578 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/901306510 2023-05-23T15:28:30.504372+0000 2023-05-22T15:39:45.578 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/2755009771 2023-05-23T15:28:30.504372+0000 2023-05-22T15:39:45.578 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/4222124422 2023-05-23T15:28:30.504372+0000 2023-05-22T15:39:45.579 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:6833/36921 2023-05-23T15:28:30.504372+0000 2023-05-22T15:39:45.579 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:6832/36921 2023-05-23T15:28:30.504372+0000 2023-05-22T15:39:45.579 INFO:teuthology.orchestra.run.smithi016.stderr:listed 6 entries 2023-05-22T15:39:45.580 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:45.574+0000 7f8cd5226640 1 -- 172.21.15.16:0/2051755750 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f8cbc065530 msgr2=0x7f8cbc0679e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-05-22T15:39:45.580 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:45.574+0000 7f8cd5226640 1 --2- 172.21.15.16:0/2051755750 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f8cbc065530 0x7f8cbc0679e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:45.580 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:45.574+0000 7f8cd5226640 1 -- 172.21.15.16:0/2051755750 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8cd0069720 msgr2=0x7f8cd01421d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:45.580 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:45.574+0000 7f8cd5226640 1 --2- 172.21.15.16:0/2051755750 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8cd0069720 0x7f8cd01421d0 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f8cc000b3e0 tx=0x7f8cc000b8a0 comp rx=0 tx=0).stop 2023-05-22T15:39:45.580 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:45.574+0000 7f8cd5226640 1 -- 172.21.15.16:0/2051755750 shutdown_connections 2023-05-22T15:39:45.580 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:45.574+0000 7f8cd5226640 1 --2- 172.21.15.16:0/2051755750 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8cd015c350 0x7f8cd0145de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:45.581 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:45.574+0000 7f8cd5226640 1 --2- 172.21.15.16:0/2051755750 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f8cbc065530 0x7f8cbc0679e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:45.581 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:45.574+0000 7f8cd5226640 1 --2- 172.21.15.16:0/2051755750 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8cd00ade80 0x7f8cd00af980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:45.581 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:45.574+0000 7f8cd5226640 1 --2- 172.21.15.16:0/2051755750 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8cd0069720 0x7f8cd01421d0 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:45.581 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:45.574+0000 7f8cd5226640 1 -- 172.21.15.16:0/2051755750 >> 172.21.15.16:0/2051755750 conn(0x7f8cd00b1110 msgr2=0x7f8cd006c3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:39:45.581 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:45.574+0000 7f8cd5226640 1 -- 172.21.15.16:0/2051755750 shutdown_connections 2023-05-22T15:39:45.581 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:45.574+0000 7f8cd5226640 1 -- 172.21.15.16:0/2051755750 wait complete. 2023-05-22T15:39:45.595 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi016.front.sepia.ceph.com... 2023-05-22T15:39:45.595 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:39:45.595 DEBUG:teuthology.orchestra.run.smithi016:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2023-05-22T15:39:45.604 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi016.stderr:ceph-fuse[49179]: fuse finished with error 0 and tester_r 0 2023-05-22T15:39:45.605 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi016.front.sepia.ceph.com 2023-05-22T15:39:45.605 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:39:45.606 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-05-22T15:39:45.606 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:39:45.605+0000 7f6e633d5180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-05-22T15:39:45.608 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:39:45.602+0000 7f3646e83180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-05-22T15:39:45.723 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:39:45.721+0000 7f6e633d5180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-05-22T15:39:45.723 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:39:45.718+0000 7f3646e83180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-05-22T15:39:45.731 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2023-05-22T15:39:45.732 DEBUG:teuthology.orchestra.run.smithi062:> dd if=/proc/self/mounts of=/dev/stdout 2023-05-22T15:39:45.737 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16342116k,nr_inodes=4085529,mode=755,inode64 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275932k,mode=755,inode64 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev,inode64 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=13250 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275928k,nr_inodes=818982,mode=700,uid=1000,gid=1271,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275932k,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-05-22T15:39:45.738 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi062.front.sepia.ceph.com 2023-05-22T15:39:45.738 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:39:45.738 DEBUG:teuthology.orchestra.run.smithi062:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2023-05-22T15:39:45.786 INFO:teuthology.orchestra.run.smithi062.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2023-05-22T15:39:45.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-05-22T15:39:45.788 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:39:45.785+0000 7f6e633d5180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-05-22T15:39:45.788 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-05-22T15:39:45.788 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2023-05-22T15:39:45.789 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:39:45.782+0000 7f3646e83180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-05-22T15:39:45.794 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16345192k,nr_inodes=4086298,mode=755,inode64 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3276544k,mode=755,inode64 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev,inode64 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=15732 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3276540k,nr_inodes=819135,mode=700,uid=1000,gid=1271,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3276544k,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-05-22T15:39:45.794 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi016.front.sepia.ceph.com 2023-05-22T15:39:45.795 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:39:45.795 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2023-05-22T15:39:45.842 INFO:teuthology.orchestra.run.smithi016.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2023-05-22T15:39:45.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-05-22T15:39:45.843 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2023-05-22T15:39:45.844 DEBUG:teuthology.orchestra.run.smithi062:> dd if=/proc/self/mounts of=/dev/stdout 2023-05-22T15:39:45.849 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16342116k,nr_inodes=4085529,mode=755,inode64 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275932k,mode=755,inode64 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev,inode64 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=13250 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275928k,nr_inodes=818982,mode=700,uid=1000,gid=1271,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275932k,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-05-22T15:39:45.850 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi062.front.sepia.ceph.com 2023-05-22T15:39:45.850 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:39:45.850 DEBUG:teuthology.orchestra.run.smithi062:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2023-05-22T15:39:45.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-05-22T15:39:45.899 INFO:teuthology.orchestra.run.smithi062.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2023-05-22T15:39:45.900 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-05-22T15:39:45.915 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:39:45.913+0000 7f6e633d5180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-05-22T15:39:45.919 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:39:45.914+0000 7f3646e83180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-05-22T15:39:46.015 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.010+0000 7f974a2be640 1 Processor -- start 2023-05-22T15:39:46.015 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.010+0000 7f974a2be640 1 -- start start 2023-05-22T15:39:46.017 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.010+0000 7f974a2be640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f97441563c0 0x7f97440c7b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:46.017 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.010+0000 7f974a2be640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f97440cc6c0 0x7f97440c8080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:46.017 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.010+0000 7f974a2be640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f974415c210 con 0x7f974406c220 2023-05-22T15:39:46.017 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.010+0000 7f974a2be640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9744160700 con 0x7f97440cc6c0 2023-05-22T15:39:46.017 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.010+0000 7f974a2be640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f974406cdd0 con 0x7f97441563c0 2023-05-22T15:39:46.017 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.010+0000 7f973b7fe640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f97440cc6c0 0x7f97440c8080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:46.017 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.010+0000 7f973b7fe640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f97440cc6c0 0x7f97440c8080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52762/0 (socket says 172.21.15.16:52762) 2023-05-22T15:39:46.018 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.010+0000 7f973b7fe640 1 -- 172.21.15.16:0/2547717292 learned_addr learned my addr 172.21.15.16:0/2547717292 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:39:46.018 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.010+0000 7f9743fff640 1 --2- 172.21.15.16:0/2547717292 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f97441563c0 0x7f97440c7b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:46.018 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.010+0000 7f973b7fe640 1 -- 172.21.15.16:0/2547717292 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f97441563c0 msgr2=0x7f97440c7b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:39:46.018 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.010+0000 7f973b7fe640 1 --2- 172.21.15.16:0/2547717292 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f97441563c0 0x7f97440c7b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:46.018 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.010+0000 7f973b7fe640 1 -- 172.21.15.16:0/2547717292 >> v1:172.21.15.62:6789/0 conn(0x7f974406c220 legacy=0x7f974406c6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:46.018 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.010+0000 7f973b7fe640 1 -- 172.21.15.16:0/2547717292 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f974415c480 con 0x7f97440cc6c0 2023-05-22T15:39:46.019 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.014+0000 7f973b7fe640 1 --2- 172.21.15.16:0/2547717292 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f97440cc6c0 0x7f97440c8080 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f972c009870 tx=0x7f972c01c510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79824d384d3000b7 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:39:46.019 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.014+0000 7f97437fe640 1 -- 172.21.15.16:0/2547717292 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f972c0aa670 con 0x7f97440cc6c0 2023-05-22T15:39:46.019 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.014+0000 7f97437fe640 1 -- 172.21.15.16:0/2547717292 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f972c0aa7d0 con 0x7f97440cc6c0 2023-05-22T15:39:46.019 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.014+0000 7f97437fe640 1 -- 172.21.15.16:0/2547717292 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f972c0aab60 con 0x7f97440cc6c0 2023-05-22T15:39:46.020 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.014+0000 7f974a2be640 1 -- 172.21.15.16:0/2547717292 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f97440cc6c0 msgr2=0x7f97440c8080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:46.020 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.014+0000 7f974a2be640 1 --2- 172.21.15.16:0/2547717292 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f97440cc6c0 0x7f97440c8080 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f972c009870 tx=0x7f972c01c510 comp rx=0 tx=0).stop 2023-05-22T15:39:46.021 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.014+0000 7f974a2be640 1 -- 172.21.15.16:0/2547717292 shutdown_connections 2023-05-22T15:39:46.021 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.014+0000 7f974a2be640 1 --2- 172.21.15.16:0/2547717292 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f97440cc6c0 0x7f97440c8080 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:46.021 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.014+0000 7f974a2be640 1 --2- 172.21.15.16:0/2547717292 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f97441563c0 0x7f97440c7b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:46.022 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.014+0000 7f974a2be640 1 -- 172.21.15.16:0/2547717292 >> 172.21.15.16:0/2547717292 conn(0x7f97440b1110 msgr2=0x7f97440ce830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:39:46.022 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.014+0000 7f974a2be640 1 -- 172.21.15.16:0/2547717292 shutdown_connections 2023-05-22T15:39:46.022 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.014+0000 7f974a2be640 1 -- 172.21.15.16:0/2547717292 wait complete. 2023-05-22T15:39:46.025 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.014+0000 7f974a2be640 1 Processor -- start 2023-05-22T15:39:46.025 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.014+0000 7f974a2be640 1 -- start start 2023-05-22T15:39:46.025 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.014+0000 7f974a2be640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f974406c220 0x7f9744151a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:46.025 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.014+0000 7f974a2be640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f97440cc6c0 0x7f97441447f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:46.025 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.014+0000 7f974a2be640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f97441507e0 0x7f97440c8e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:46.026 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.014+0000 7f974a2be640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97440b02f0 con 0x7f974406c220 2023-05-22T15:39:46.026 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.014+0000 7f974a2be640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97440b0010 con 0x7f97440cc6c0 2023-05-22T15:39:46.026 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.014+0000 7f974a2be640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f974414fd00 con 0x7f97441507e0 2023-05-22T15:39:46.026 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.014+0000 7f9743fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f974406c220 0x7f9744151a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:46.026 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.014+0000 7f9743fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f974406c220 0x7f9744151a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52764/0 (socket says 172.21.15.16:52764) 2023-05-22T15:39:46.026 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.014+0000 7f9743fff640 1 -- 172.21.15.16:0/416251911 learned_addr learned my addr 172.21.15.16:0/416251911 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:39:46.026 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.018+0000 7f973b7fe640 1 --2- 172.21.15.16:0/416251911 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f97440cc6c0 0x7f97441447f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:46.027 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.018+0000 7f974885b640 1 --2- 172.21.15.16:0/416251911 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f97441507e0 0x7f97440c8e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:46.027 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.018+0000 7f9743fff640 1 -- 172.21.15.16:0/416251911 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f97441507e0 msgr2=0x7f97440c8e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:46.027 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.018+0000 7f9743fff640 1 --2- 172.21.15.16:0/416251911 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f97441507e0 0x7f97440c8e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:46.027 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.018+0000 7f9743fff640 1 -- 172.21.15.16:0/416251911 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f97440cc6c0 msgr2=0x7f97441447f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:46.027 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.018+0000 7f9743fff640 1 --2- 172.21.15.16:0/416251911 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f97440cc6c0 0x7f97441447f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:46.027 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.018+0000 7f9743fff640 1 -- 172.21.15.16:0/416251911 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f972c0094d0 con 0x7f974406c220 2023-05-22T15:39:46.027 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.018+0000 7f9743fff640 1 --2- 172.21.15.16:0/416251911 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f974406c220 0x7f9744151a40 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f973400d5f0 tx=0x7f973400dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:39:46.027 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.018+0000 7f97417fa640 1 -- 172.21.15.16:0/416251911 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97340125f0 con 0x7f974406c220 2023-05-22T15:39:46.028 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.018+0000 7f97417fa640 1 -- 172.21.15.16:0/416251911 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9734012750 con 0x7f974406c220 2023-05-22T15:39:46.028 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.018+0000 7f97417fa640 1 -- 172.21.15.16:0/416251911 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9734012ae0 con 0x7f974406c220 2023-05-22T15:39:46.028 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.018+0000 7f974a2be640 1 -- 172.21.15.16:0/416251911 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f974415ff00 con 0x7f974406c220 2023-05-22T15:39:46.028 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.018+0000 7f974a2be640 1 -- 172.21.15.16:0/416251911 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f974415cd20 con 0x7f974406c220 2023-05-22T15:39:46.028 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.018+0000 7f974a2be640 1 -- 172.21.15.16:0/416251911 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9744160090 con 0x7f974406c220 2023-05-22T15:39:46.028 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.018+0000 7f97417fa640 1 -- 172.21.15.16:0/416251911 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 13) v1 ==== 84401+0+0 (secure 0 0 0) 0x7f9734012c40 con 0x7f974406c220 2023-05-22T15:39:46.028 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.018+0000 7f97417fa640 1 --2- 172.21.15.16:0/416251911 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f9728065560 0x7f9728067a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:46.029 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.018+0000 7f973b7fe640 1 -- 172.21.15.16:0/416251911 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f9728065560 msgr2=0x7f9728067a10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/36921 2023-05-22T15:39:46.029 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.018+0000 7f97417fa640 1 -- 172.21.15.16:0/416251911 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 7570+0+0 (secure 0 0 0) 0x7f9734084160 con 0x7f974406c220 2023-05-22T15:39:46.029 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.018+0000 7f973b7fe640 1 --2- 172.21.15.16:0/416251911 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f9728065560 0x7f9728067a10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-05-22T15:39:46.029 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.022+0000 7f97417fa640 1 -- 172.21.15.16:0/416251911 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f9734051510 con 0x7f974406c220 2023-05-22T15:39:46.211 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.206+0000 7f974a2be640 1 -- 172.21.15.16:0/416251911 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f974404dbe0 con 0x7f974406c220 2023-05-22T15:39:46.211 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.206+0000 7f97417fa640 1 -- 172.21.15.16:0/416251911 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) v1 ==== 94+0+4648 (secure 0 0 0) 0x7f9734054b30 con 0x7f974406c220 2023-05-22T15:39:46.212 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:39:46.212 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":14,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4904,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6833/3153072235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":3153072235},{"type":"v1","addr":"172.21.15.62:6833","nonce":3153072235}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":5164,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/2783822861","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":2783822861},{"type":"v1","addr":"172.21.15.62:6835","nonce":2783822861}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":5571,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/1361634240","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1361634240},{"type":"v1","addr":"172.21.15.16:6835","nonce":1361634240}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":14,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-05-22T15:28:13.320259+0000","modified":"2023-05-22T15:28:15.336631+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5574},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5574":{"gid":5574,"name":"c","rank":0,"incarnation":12,"state":"up:active","state_seq":3,"addr":"172.21.15.16:6837/1033866899","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":1033866899},{"type":"v1","addr":"172.21.15.16:6837","nonce":1033866899}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-05-22T15:39:46.213 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 14 2023-05-22T15:39:46.214 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.210+0000 7f974a2be640 1 -- 172.21.15.16:0/416251911 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f9728065560 msgr2=0x7f9728067a10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-05-22T15:39:46.214 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.210+0000 7f974a2be640 1 --2- 172.21.15.16:0/416251911 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f9728065560 0x7f9728067a10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:46.214 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.210+0000 7f974a2be640 1 -- 172.21.15.16:0/416251911 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f974406c220 msgr2=0x7f9744151a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:46.215 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.210+0000 7f974a2be640 1 --2- 172.21.15.16:0/416251911 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f974406c220 0x7f9744151a40 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f973400d5f0 tx=0x7f973400dab0 comp rx=0 tx=0).stop 2023-05-22T15:39:46.215 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.210+0000 7f974a2be640 1 -- 172.21.15.16:0/416251911 shutdown_connections 2023-05-22T15:39:46.215 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.210+0000 7f974a2be640 1 --2- 172.21.15.16:0/416251911 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f9728065560 0x7f9728067a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:46.215 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.210+0000 7f974a2be640 1 --2- 172.21.15.16:0/416251911 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f97441507e0 0x7f97440c8e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:46.215 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.210+0000 7f974a2be640 1 --2- 172.21.15.16:0/416251911 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f97440cc6c0 0x7f97441447f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:46.216 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.210+0000 7f974a2be640 1 --2- 172.21.15.16:0/416251911 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f974406c220 0x7f9744151a40 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:46.216 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.210+0000 7f974a2be640 1 -- 172.21.15.16:0/416251911 >> 172.21.15.16:0/416251911 conn(0x7f97440b1110 msgr2=0x7f97440705f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:39:46.216 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.210+0000 7f974a2be640 1 -- 172.21.15.16:0/416251911 shutdown_connections 2023-05-22T15:39:46.216 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.210+0000 7f974a2be640 1 -- 172.21.15.16:0/416251911 wait complete. 2023-05-22T15:39:46.231 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-05-22T15:39:46.386 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.378+0000 7f4ed6e88640 1 Processor -- start 2023-05-22T15:39:46.386 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.378+0000 7f4ed6e88640 1 -- start start 2023-05-22T15:39:46.387 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.382+0000 7f4ed6e88640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4ed00cae20 0x7f4ed006fef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:46.388 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.382+0000 7f4ed6e88640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f4ed0070430 0x7f4ed00ad410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:46.388 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.382+0000 7f4ed6e88640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ed0069720 con 0x7f4ed0153fe0 2023-05-22T15:39:46.388 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.382+0000 7f4ed6e88640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ed00708d0 con 0x7f4ed0070430 2023-05-22T15:39:46.388 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.382+0000 7f4ed6e88640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ed006db00 con 0x7f4ed00cae20 2023-05-22T15:39:46.388 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.382+0000 7f4ec7fff640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4ed00cae20 0x7f4ed006fef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:46.389 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.382+0000 7f4ed5425640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f4ed0070430 0x7f4ed00ad410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:46.389 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.382+0000 7f4ec7fff640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4ed00cae20 0x7f4ed006fef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3301/0 says I am v2:172.21.15.16:50402/0 (socket says 172.21.15.16:50402) 2023-05-22T15:39:46.389 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.382+0000 7f4ec7fff640 1 -- 172.21.15.16:0/1536787219 learned_addr learned my addr 172.21.15.16:0/1536787219 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:39:46.389 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.382+0000 7f4ec7fff640 1 -- 172.21.15.16:0/1536787219 >> v2:172.21.15.62:3300/0 conn(0x7f4ed0070430 msgr2=0x7f4ed00ad410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:39:46.389 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.382+0000 7f4ec7fff640 1 --2- 172.21.15.16:0/1536787219 >> v2:172.21.15.62:3300/0 conn(0x7f4ed0070430 0x7f4ed00ad410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:46.389 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.382+0000 7f4ec7fff640 1 -- 172.21.15.16:0/1536787219 >> v1:172.21.15.62:6789/0 conn(0x7f4ed0153fe0 legacy=0x7f4ed006d3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:46.389 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.382+0000 7f4ec7fff640 1 -- 172.21.15.16:0/1536787219 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ed01543b0 con 0x7f4ed00cae20 2023-05-22T15:39:46.390 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.382+0000 7f4ec77fe640 1 -- 172.21.15.16:0/1536787219 <== mon.1 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2295424123 0 0) 0x7f4ed0069720 con 0x7f4ed0153fe0 2023-05-22T15:39:46.391 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.382+0000 7f4ec7fff640 1 --2- 172.21.15.16:0/1536787219 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4ed00cae20 0x7f4ed006fef0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f4ec0009870 tx=0x7f4ec001c510 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4171588eeb15ce6a server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:39:46.391 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.382+0000 7f4ec77fe640 1 -- 172.21.15.16:0/1536787219 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ec00aa670 con 0x7f4ed00cae20 2023-05-22T15:39:46.391 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.382+0000 7f4ec77fe640 1 -- 172.21.15.16:0/1536787219 <== mon.2 v2:172.21.15.62:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ec00aa7d0 con 0x7f4ed00cae20 2023-05-22T15:39:46.391 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.382+0000 7f4ec77fe640 1 -- 172.21.15.16:0/1536787219 <== mon.2 v2:172.21.15.62:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ec00aab60 con 0x7f4ed00cae20 2023-05-22T15:39:46.391 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.386+0000 7f4ed6e88640 1 -- 172.21.15.16:0/1536787219 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4ed00cae20 msgr2=0x7f4ed006fef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:46.392 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.386+0000 7f4ed6e88640 1 --2- 172.21.15.16:0/1536787219 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4ed00cae20 0x7f4ed006fef0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f4ec0009870 tx=0x7f4ec001c510 comp rx=0 tx=0).stop 2023-05-22T15:39:46.392 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.386+0000 7f4ed6e88640 1 -- 172.21.15.16:0/1536787219 shutdown_connections 2023-05-22T15:39:46.392 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.386+0000 7f4ed6e88640 1 --2- 172.21.15.16:0/1536787219 >> v2:172.21.15.62:3300/0 conn(0x7f4ed0070430 0x7f4ed00ad410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:46.392 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.386+0000 7f4ed6e88640 1 --2- 172.21.15.16:0/1536787219 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4ed00cae20 0x7f4ed006fef0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:46.392 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.386+0000 7f4ed6e88640 1 -- 172.21.15.16:0/1536787219 >> 172.21.15.16:0/1536787219 conn(0x7f4ed00b1110 msgr2=0x7f4ed00ce830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:39:46.392 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.386+0000 7f4ed6e88640 1 -- 172.21.15.16:0/1536787219 shutdown_connections 2023-05-22T15:39:46.392 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.386+0000 7f4ed6e88640 1 -- 172.21.15.16:0/1536787219 wait complete. 2023-05-22T15:39:46.394 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.386+0000 7f4ed6e88640 1 Processor -- start 2023-05-22T15:39:46.394 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.386+0000 7f4ed6e88640 1 -- start start 2023-05-22T15:39:46.394 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.386+0000 7f4ed6e88640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4ed0070430 0x7f4ed014d8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:46.395 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.386+0000 7f4ed6e88640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4ed00cae20 0x7f4ed0158c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:46.395 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.386+0000 7f4ed6e88640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ed0153fe0 0x7f4ed01591d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:46.395 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.386+0000 7f4ed6e88640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ed00cd790 con 0x7f4ed0153fe0 2023-05-22T15:39:46.395 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.386+0000 7f4ed6e88640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ed00cd4b0 con 0x7f4ed0070430 2023-05-22T15:39:46.395 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.386+0000 7f4ed6e88640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ed015af00 con 0x7f4ed00cae20 2023-05-22T15:39:46.395 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.386+0000 7f4ed5425640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ed0153fe0 0x7f4ed01591d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:46.395 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.386+0000 7f4ed5425640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ed0153fe0 0x7f4ed01591d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52780/0 (socket says 172.21.15.16:52780) 2023-05-22T15:39:46.396 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.386+0000 7f4ed5425640 1 -- 172.21.15.16:0/529340643 learned_addr learned my addr 172.21.15.16:0/529340643 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:39:46.396 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.386+0000 7f4ec7fff640 1 --2- 172.21.15.16:0/529340643 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4ed00cae20 0x7f4ed0158c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:46.396 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.386+0000 7f4ed4c24640 1 --2- 172.21.15.16:0/529340643 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4ed0070430 0x7f4ed014d8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:46.396 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.386+0000 7f4ed5425640 1 -- 172.21.15.16:0/529340643 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4ed00cae20 msgr2=0x7f4ed0158c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:46.396 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.386+0000 7f4ed5425640 1 --2- 172.21.15.16:0/529340643 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4ed00cae20 0x7f4ed0158c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:46.396 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.386+0000 7f4ed5425640 1 -- 172.21.15.16:0/529340643 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4ed0070430 msgr2=0x7f4ed014d8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:46.397 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.386+0000 7f4ed5425640 1 --2- 172.21.15.16:0/529340643 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4ed0070430 0x7f4ed014d8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:46.399 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.386+0000 7f4ed5425640 1 -- 172.21.15.16:0/529340643 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ec00094d0 con 0x7f4ed0153fe0 2023-05-22T15:39:46.399 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.386+0000 7f4ed4c24640 1 --2- 172.21.15.16:0/529340643 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4ed0070430 0x7f4ed014d8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T15:39:46.399 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.386+0000 7f4ed5425640 1 --2- 172.21.15.16:0/529340643 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ed0153fe0 0x7f4ed01591d0 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f4ec800efc0 tx=0x7f4ec800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:39:46.400 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.386+0000 7f4ec57fa640 1 -- 172.21.15.16:0/529340643 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ec8014480 con 0x7f4ed0153fe0 2023-05-22T15:39:46.400 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.386+0000 7f4ec57fa640 1 -- 172.21.15.16:0/529340643 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ec800f040 con 0x7f4ed0153fe0 2023-05-22T15:39:46.400 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.386+0000 7f4ed6e88640 1 -- 172.21.15.16:0/529340643 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ed015db60 con 0x7f4ed0153fe0 2023-05-22T15:39:46.400 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.386+0000 7f4ec57fa640 1 -- 172.21.15.16:0/529340643 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ec8014820 con 0x7f4ed0153fe0 2023-05-22T15:39:46.401 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.386+0000 7f4ed6e88640 1 -- 172.21.15.16:0/529340643 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ed015d880 con 0x7f4ed0153fe0 2023-05-22T15:39:46.401 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.390+0000 7f4ed6e88640 1 -- 172.21.15.16:0/529340643 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ed015de40 con 0x7f4ed0153fe0 2023-05-22T15:39:46.402 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.390+0000 7f4ec57fa640 1 -- 172.21.15.16:0/529340643 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 13) v1 ==== 84401+0+0 (secure 0 0 0) 0x7f4ec8014980 con 0x7f4ed0153fe0 2023-05-22T15:39:46.403 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.390+0000 7f4ec57fa640 1 --2- 172.21.15.16:0/529340643 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f4ebc065530 0x7f4ebc0679e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:46.403 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.390+0000 7f4ed4c24640 1 -- 172.21.15.16:0/529340643 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f4ebc065530 msgr2=0x7f4ebc0679e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/36921 2023-05-22T15:39:46.403 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.390+0000 7f4ec57fa640 1 -- 172.21.15.16:0/529340643 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 7570+0+0 (secure 0 0 0) 0x7f4ec8083db0 con 0x7f4ed0153fe0 2023-05-22T15:39:46.403 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.390+0000 7f4ed4c24640 1 --2- 172.21.15.16:0/529340643 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f4ebc065530 0x7f4ebc0679e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-05-22T15:39:46.404 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.394+0000 7f4ec57fa640 1 -- 172.21.15.16:0/529340643 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f4ec80510b0 con 0x7f4ed0153fe0 2023-05-22T15:39:46.410 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:39:46.409+0000 7f6e633d5180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-05-22T15:39:46.422 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:39:46.418+0000 7f3646e83180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-05-22T15:39:46.487 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:39:46.485+0000 7f6e633d5180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-05-22T15:39:46.500 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:39:46.494+0000 7f3646e83180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-05-22T15:39:46.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.546+0000 7f4ed6e88640 1 -- 172.21.15.16:0/529340643 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4ed015ad60 con 0x7f4ed0153fe0 2023-05-22T15:39:46.555 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.550+0000 7f4ec57fa640 1 -- 172.21.15.16:0/529340643 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v59) v1 ==== 74+0+19843 (secure 0 0 0) 0x7f4ed015ad60 con 0x7f4ed0153fe0 2023-05-22T15:39:46.556 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:39:46.556 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":59,"fsid":"b2719834-7847-4d98-90a6-f65d57f85c49","created":"2023-05-22T15:26:25.259314+0000","modified":"2023-05-22T15:39:45.328876+0000","last_up_change":"2023-05-22T15:26:36.448524+0000","last_in_change":"2023-05-22T15:26:27.675668+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-05-22T15:26:38.019781+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-05-22T15:26:43.570514+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2023-05-22T15:28:11.902605+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"41","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2023-05-22T15:28:12.298278+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"58","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":220,"snap_epoch":58,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9e7ff854-b25f-4cbb-a5c2-1799d9c53d2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":37021},{"type":"v1","addr":"172.21.15.16:6825","nonce":37021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":37021},{"type":"v1","addr":"172.21.15.16:6827","nonce":37021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":37021},{"type":"v1","addr":"172.21.15.16:6831","nonce":37021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":37021},{"type":"v1","addr":"172.21.15.16:6829","nonce":37021}]},"public_addr":"172.21.15.16:6825/37021","cluster_addr":"172.21.15.16:6827/37021","heartbeat_back_addr":"172.21.15.16:6831/37021","heartbeat_front_addr":"172.21.15.16:6829/37021","state":["exists","up"]},{"osd":1,"uuid":"4702fbc9-a927-4565-aca5-41dc69cc8264","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":37008},{"type":"v1","addr":"172.21.15.16:6801","nonce":37008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":37008},{"type":"v1","addr":"172.21.15.16:6803","nonce":37008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":37008},{"type":"v1","addr":"172.21.15.16:6807","nonce":37008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":37008},{"type":"v1","addr":"172.21.15.16:6805","nonce":37008}]},"public_addr":"172.21.15.16:6801/37008","cluster_addr":"172.21.15.16:6803/37008","heartbeat_back_addr":"172.21.15.16:6807/37008","heartbeat_front_addr":"172.21.15.16:6805/37008","state":["exists","up"]},{"osd":2,"uuid":"1e7bba46-4eba-47ef-bb81-5e2d371638ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":37009},{"type":"v1","addr":"172.21.15.16:6809","nonce":37009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":37009},{"type":"v1","addr":"172.21.15.16:6811","nonce":37009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":37009},{"type":"v1","addr":"172.21.15.16:6815","nonce":37009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":37009},{"type":"v1","addr":"172.21.15.16:6813","nonce":37009}]},"public_addr":"172.21.15.16:6809/37009","cluster_addr":"172.21.15.16:6811/37009","heartbeat_back_addr":"172.21.15.16:6815/37009","heartbeat_front_addr":"172.21.15.16:6813/37009","state":["exists","up"]},{"osd":3,"uuid":"383cacc0-5a88-4904-990e-498ca59ef48a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":37020},{"type":"v1","addr":"172.21.15.16:6817","nonce":37020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":37020},{"type":"v1","addr":"172.21.15.16:6819","nonce":37020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":37020},{"type":"v1","addr":"172.21.15.16:6823","nonce":37020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":37020},{"type":"v1","addr":"172.21.15.16:6821","nonce":37020}]},"public_addr":"172.21.15.16:6817/37020","cluster_addr":"172.21.15.16:6819/37020","heartbeat_back_addr":"172.21.15.16:6823/37020","heartbeat_front_addr":"172.21.15.16:6821/37020","state":["exists","up"]},{"osd":4,"uuid":"970500ef-95b7-42c1-b080-eed4fae7013d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":37353},{"type":"v1","addr":"172.21.15.62:6817","nonce":37353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6819","nonce":37353},{"type":"v1","addr":"172.21.15.62:6822","nonce":37353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37353},{"type":"v1","addr":"172.21.15.62:6831","nonce":37353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37353},{"type":"v1","addr":"172.21.15.62:6826","nonce":37353}]},"public_addr":"172.21.15.62:6817/37353","cluster_addr":"172.21.15.62:6822/37353","heartbeat_back_addr":"172.21.15.62:6831/37353","heartbeat_front_addr":"172.21.15.62:6826/37353","state":["exists","up"]},{"osd":5,"uuid":"022d67db-d4d8-445b-b1b9-8b446a00f5c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":37352},{"type":"v1","addr":"172.21.15.62:6801","nonce":37352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":37352},{"type":"v1","addr":"172.21.15.62:6803","nonce":37352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":37352},{"type":"v1","addr":"172.21.15.62:6807","nonce":37352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":37352},{"type":"v1","addr":"172.21.15.62:6805","nonce":37352}]},"public_addr":"172.21.15.62:6801/37352","cluster_addr":"172.21.15.62:6803/37352","heartbeat_back_addr":"172.21.15.62:6807/37352","heartbeat_front_addr":"172.21.15.62:6805/37352","state":["exists","up"]},{"osd":6,"uuid":"42bd372b-6688-4a1b-8407-e16e8c266e33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":37354},{"type":"v1","addr":"172.21.15.62:6820","nonce":37354}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":37354},{"type":"v1","addr":"172.21.15.62:6823","nonce":37354}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6829","nonce":37354},{"type":"v1","addr":"172.21.15.62:6830","nonce":37354}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6825","nonce":37354},{"type":"v1","addr":"172.21.15.62:6827","nonce":37354}]},"public_addr":"172.21.15.62:6820/37354","cluster_addr":"172.21.15.62:6823/37354","heartbeat_back_addr":"172.21.15.62:6830/37354","heartbeat_front_addr":"172.21.15.62:6827/37354","state":["exists","up"]},{"osd":7,"uuid":"ac70b2a3-f41b-4bf2-b227-7f2cbb55426d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37355},{"type":"v1","addr":"172.21.15.62:6809","nonce":37355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37355},{"type":"v1","addr":"172.21.15.62:6811","nonce":37355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":37355},{"type":"v1","addr":"172.21.15.62:6815","nonce":37355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":37355},{"type":"v1","addr":"172.21.15.62:6813","nonce":37355}]},"public_addr":"172.21.15.62:6809/37355","cluster_addr":"172.21.15.62:6811/37355","heartbeat_back_addr":"172.21.15.62:6815/37355","heartbeat_front_addr":"172.21.15.62:6813/37355","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.368896+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.107273+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.276888+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.287502+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.124791+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.251937+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.255679+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.232797+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"7.4","mappings":[{"from":0,"to":5}]},{"pgid":"7.2e","mappings":[{"from":0,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:0/3979017463":"2023-05-23T15:28:30.504372+0000","172.21.15.16:0/901306510":"2023-05-23T15:28:30.504372+0000","172.21.15.16:0/2755009771":"2023-05-23T15:28:30.504372+0000","172.21.15.16:0/4222124422":"2023-05-23T15:28:30.504372+0000","172.21.15.16:6833/36921":"2023-05-23T15:28:30.504372+0000","172.21.15.16:6832/36921":"2023-05-23T15:28:30.504372+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":7,"snaps":[{"begin":6,"length":1},{"begin":10,"length":1},{"begin":14,"length":1},{"begin":19,"length":1},{"begin":24,"length":1},{"begin":29,"length":1},{"begin":35,"length":1},{"begin":39,"length":2},{"begin":45,"length":1},{"begin":47,"length":1},{"begin":50,"length":1},{"begin":53,"length":1},{"begin":58,"length":1},{"begin":61,"length":1},{"begin":65,"length":1},{"begin":67,"length":1},{"begin":70,"length":1},{"begin":74,"length":1},{"begin":77,"length":1},{"begin":81,"length":1},{"begin":84,"length":1},{"begin":89,"length":2},{"begin":98,"length":3},{"begin":106,"length":1},{"begin":108,"length":1},{"begin":192,"length":29}]}],"new_removed_snaps":[],"new_purged_snaps":[{"pool":7,"snaps":[{"begin":2,"length":2},{"begin":7,"length":2},{"begin":12,"length":1},{"begin":15,"length":1},{"begin":20,"length":2},{"begin":25,"length":2},{"begin":28,"length":1},{"begin":30,"length":1},{"begin":33,"length":1},{"begin":37,"length":1},{"begin":41,"length":2},{"begin":46,"length":1},{"begin":49,"length":1},{"begin":52,"length":1},{"begin":55,"length":2},{"begin":62,"length":1},{"begin":64,"length":1},{"begin":71,"length":1},{"begin":73,"length":1},{"begin":79,"length":2},{"begin":82,"length":1},{"begin":86,"length":3},{"begin":93,"length":4},{"begin":102,"length":2},{"begin":105,"length":1},{"begin":110,"length":2},{"begin":144,"length":48}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-05-22T15:39:46.558 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.550+0000 7f4ed6e88640 1 -- 172.21.15.16:0/529340643 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f4ebc065530 msgr2=0x7f4ebc0679e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-05-22T15:39:46.558 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.550+0000 7f4ed6e88640 1 --2- 172.21.15.16:0/529340643 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f4ebc065530 0x7f4ebc0679e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:46.558 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.550+0000 7f4ed6e88640 1 -- 172.21.15.16:0/529340643 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ed0153fe0 msgr2=0x7f4ed01591d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:46.558 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.550+0000 7f4ed6e88640 1 --2- 172.21.15.16:0/529340643 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ed0153fe0 0x7f4ed01591d0 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f4ec800efc0 tx=0x7f4ec800c490 comp rx=0 tx=0).stop 2023-05-22T15:39:46.558 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.550+0000 7f4ed6e88640 1 -- 172.21.15.16:0/529340643 shutdown_connections 2023-05-22T15:39:46.558 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.550+0000 7f4ed6e88640 1 --2- 172.21.15.16:0/529340643 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ed0153fe0 0x7f4ed01591d0 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:46.558 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.550+0000 7f4ed6e88640 1 --2- 172.21.15.16:0/529340643 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f4ebc065530 0x7f4ebc0679e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:46.559 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.550+0000 7f4ed6e88640 1 --2- 172.21.15.16:0/529340643 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4ed00cae20 0x7f4ed0158c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:46.559 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.550+0000 7f4ed6e88640 1 --2- 172.21.15.16:0/529340643 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4ed0070430 0x7f4ed014d8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:46.559 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.550+0000 7f4ed6e88640 1 -- 172.21.15.16:0/529340643 >> 172.21.15.16:0/529340643 conn(0x7f4ed00b1110 msgr2=0x7f4ed00afcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:39:46.559 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.550+0000 7f4ed6e88640 1 -- 172.21.15.16:0/529340643 shutdown_connections 2023-05-22T15:39:46.559 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.550+0000 7f4ed6e88640 1 -- 172.21.15.16:0/529340643 wait complete. 2023-05-22T15:39:46.574 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-05-22T15:26:38.019781+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-05-22T15:26:43.570514+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_metadata', 'create_time': '2023-05-22T15:28:11.902605+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '41', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'cephfs_data', 'create_time': '2023-05-22T15:28:12.298278+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '58', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 220, 'snap_epoch': 58, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-05-22T15:39:46.575 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-05-22T15:39:46.625 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:39:46.621+0000 7f6e633d5180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-05-22T15:39:46.643 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:39:46.638+0000 7f3646e83180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-05-22T15:39:46.694 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.686+0000 7f5183d76640 1 Processor -- start 2023-05-22T15:39:46.694 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.686+0000 7f5183d76640 1 -- start start 2023-05-22T15:39:46.697 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.690+0000 7f5183d76640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f517c0c67d0 0x7f517c0cf730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:46.698 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.690+0000 7f5183d76640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f517c0b2880 0x7f517c0cbfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:46.698 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.690+0000 7f5183d76640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f517c0cfc70 con 0x7f517c0cc520 2023-05-22T15:39:46.698 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.690+0000 7f5183d76640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f517c15a300 con 0x7f517c0c67d0 2023-05-22T15:39:46.698 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.690+0000 7f5183d76640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f517c0c4dc0 con 0x7f517c0b2880 2023-05-22T15:39:46.699 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.690+0000 7f5182313640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f517c0cc520 0x7f517c0ce160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:56546/0 (socket says 172.21.15.16:56546) 2023-05-22T15:39:46.699 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.690+0000 7f5182313640 1 -- 172.21.15.16:0/3989478416 learned_addr learned my addr 172.21.15.16:0/3989478416 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:39:46.699 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.690+0000 7f5181b12640 1 --2- 172.21.15.16:0/3989478416 >> v2:172.21.15.62:3300/0 conn(0x7f517c0c67d0 0x7f517c0cf730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:46.700 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.690+0000 7f5181311640 1 --2- 172.21.15.16:0/3989478416 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f517c0b2880 0x7f517c0cbfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:46.700 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.690+0000 7f5180b10640 1 -- 172.21.15.16:0/3989478416 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3570569536 0 0) 0x7f517c0cfc70 con 0x7f517c0cc520 2023-05-22T15:39:46.700 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.690+0000 7f5180b10640 1 -- 172.21.15.16:0/3989478416 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5168003530 con 0x7f517c0cc520 2023-05-22T15:39:46.700 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.690+0000 7f5180b10640 1 -- 172.21.15.16:0/3989478416 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1962212732 0 0) 0x7f5168003530 con 0x7f517c0cc520 2023-05-22T15:39:46.700 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.690+0000 7f5180b10640 1 -- 172.21.15.16:0/3989478416 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f517c0b2880 msgr2=0x7f517c0cbfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:39:46.701 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.690+0000 7f5180b10640 1 --2- 172.21.15.16:0/3989478416 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f517c0b2880 0x7f517c0cbfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:46.701 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.690+0000 7f5180b10640 1 -- 172.21.15.16:0/3989478416 >> v2:172.21.15.62:3300/0 conn(0x7f517c0c67d0 msgr2=0x7f517c0cf730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:39:46.701 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.690+0000 7f5180b10640 1 --2- 172.21.15.16:0/3989478416 >> v2:172.21.15.62:3300/0 conn(0x7f517c0c67d0 0x7f517c0cf730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:46.701 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.690+0000 7f5180b10640 1 -- 172.21.15.16:0/3989478416 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f517c0c6ba0 con 0x7f517c0cc520 2023-05-22T15:39:46.702 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.690+0000 7f5180b10640 1 -- 172.21.15.16:0/3989478416 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7f5174001a00 con 0x7f517c0cc520 2023-05-22T15:39:46.702 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.690+0000 7f5180b10640 1 -- 172.21.15.16:0/3989478416 >> v1:172.21.15.16:6789/0 conn(0x7f517c0cc520 legacy=0x7f517c0ce160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:46.702 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.690+0000 7f5180b10640 1 --2- 172.21.15.16:0/3989478416 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5168003980 0x7f5168003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:46.702 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.690+0000 7f5180b10640 1 --2- 172.21.15.16:0/3989478416 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f517c0b2880 0x7f51680047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:46.703 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.690+0000 7f5180b10640 1 --2- 172.21.15.16:0/3989478416 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5168004d10 0x7f5168005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:46.703 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.690+0000 7f5180b10640 1 -- 172.21.15.16:0/3989478416 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f517c15a300 con 0x7f5168004d10 2023-05-22T15:39:46.703 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.690+0000 7f5180b10640 1 -- 172.21.15.16:0/3989478416 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f517c0c4dc0 con 0x7f5168003980 2023-05-22T15:39:46.703 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.690+0000 7f5180b10640 1 -- 172.21.15.16:0/3989478416 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f516800b990 con 0x7f517c0b2880 2023-05-22T15:39:46.704 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.690+0000 7f5182313640 1 --2- 172.21.15.16:0/3989478416 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5168004d10 0x7f5168005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:46.704 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.690+0000 7f5181311640 1 --2- 172.21.15.16:0/3989478416 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f517c0b2880 0x7f51680047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:46.704 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.690+0000 7f5181b12640 1 --2- 172.21.15.16:0/3989478416 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5168003980 0x7f5168003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:46.705 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.690+0000 7f5182313640 1 -- 172.21.15.16:0/3989478416 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f517c0b2880 msgr2=0x7f51680047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:39:46.705 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.690+0000 7f5182313640 1 --2- 172.21.15.16:0/3989478416 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f517c0b2880 0x7f51680047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:46.705 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.690+0000 7f5182313640 1 -- 172.21.15.16:0/3989478416 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5168003980 msgr2=0x7f5168003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:39:46.705 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.690+0000 7f5182313640 1 --2- 172.21.15.16:0/3989478416 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5168003980 0x7f5168003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:46.706 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.690+0000 7f5182313640 1 -- 172.21.15.16:0/3989478416 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5168003530 con 0x7f5168004d10 2023-05-22T15:39:46.707 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:39:46.701+0000 7f6e633d5180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-05-22T15:39:46.711 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.694+0000 7f5182313640 1 --2- 172.21.15.16:0/3989478416 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5168004d10 0x7f5168005180 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f517400a8d0 tx=0x7f517401fb40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f2ee294217fee7a server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:39:46.712 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.694+0000 7f5180b10640 1 -- 172.21.15.16:0/3989478416 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51740aa6e0 con 0x7f5168004d10 2023-05-22T15:39:46.712 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.694+0000 7f5180b10640 1 -- 172.21.15.16:0/3989478416 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51740aa840 con 0x7f5168004d10 2023-05-22T15:39:46.712 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.694+0000 7f5180b10640 1 -- 172.21.15.16:0/3989478416 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51740aabf0 con 0x7f5168004d10 2023-05-22T15:39:46.712 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.694+0000 7f5183d76640 1 -- 172.21.15.16:0/3989478416 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5168004d10 msgr2=0x7f5168005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:46.713 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.694+0000 7f5183d76640 1 --2- 172.21.15.16:0/3989478416 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5168004d10 0x7f5168005180 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f517400a8d0 tx=0x7f517401fb40 comp rx=0 tx=0).stop 2023-05-22T15:39:46.713 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.694+0000 7f5182313640 1 -- 172.21.15.16:0/3989478416 reap_dead start 2023-05-22T15:39:46.713 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.694+0000 7f5183d76640 1 -- 172.21.15.16:0/3989478416 shutdown_connections 2023-05-22T15:39:46.713 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.694+0000 7f5183d76640 1 -- 172.21.15.16:0/3989478416 >> 172.21.15.16:0/3989478416 conn(0x7f517c06f5a0 msgr2=0x7f517c06f9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:39:46.714 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.694+0000 7f5183d76640 1 -- 172.21.15.16:0/3989478416 shutdown_connections 2023-05-22T15:39:46.714 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.694+0000 7f5183d76640 1 -- 172.21.15.16:0/3989478416 wait complete. 2023-05-22T15:39:46.714 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.694+0000 7f5183d76640 1 Processor -- start 2023-05-22T15:39:46.714 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.694+0000 7f5183d76640 1 -- start start 2023-05-22T15:39:46.715 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.694+0000 7f5183d76640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5168004d10 0x7f517c0b3aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:46.715 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.694+0000 7f5183d76640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f517c0b5320 0x7f517c14e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:46.715 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.694+0000 7f5183d76640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f517c15bd20 0x7f517c15c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:46.715 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.694+0000 7f5183d76640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f517c0c7f60 con 0x7f517c0b5320 2023-05-22T15:39:46.716 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.694+0000 7f5183d76640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f517c15c6d0 con 0x7f517c15bd20 2023-05-22T15:39:46.716 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.694+0000 7f5183d76640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f517c152c40 con 0x7f5168004d10 2023-05-22T15:39:46.716 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.694+0000 7f5181311640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f517c0b5320 0x7f517c14e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:46.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.694+0000 7f5181311640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f517c0b5320 0x7f517c14e760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52806/0 (socket says 172.21.15.16:52806) 2023-05-22T15:39:46.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.694+0000 7f5181311640 1 -- 172.21.15.16:0/1288817175 learned_addr learned my addr 172.21.15.16:0/1288817175 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:39:46.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.694+0000 7f5182313640 1 --2- 172.21.15.16:0/1288817175 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f517c15bd20 0x7f517c15c190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:46.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.694+0000 7f5181b12640 1 --2- 172.21.15.16:0/1288817175 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5168004d10 0x7f517c0b3aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:46.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.694+0000 7f5181311640 1 -- 172.21.15.16:0/1288817175 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5168004d10 msgr2=0x7f517c0b3aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:46.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.694+0000 7f5181311640 1 --2- 172.21.15.16:0/1288817175 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5168004d10 0x7f517c0b3aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:46.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.694+0000 7f5181311640 1 -- 172.21.15.16:0/1288817175 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f517c15bd20 msgr2=0x7f517c15c190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:46.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.694+0000 7f5181311640 1 --2- 172.21.15.16:0/1288817175 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f517c15bd20 0x7f517c15c190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:46.719 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.694+0000 7f5181311640 1 -- 172.21.15.16:0/1288817175 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f517c0d01d0 con 0x7f517c0b5320 2023-05-22T15:39:46.719 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.694+0000 7f5181311640 1 --2- 172.21.15.16:0/1288817175 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f517c0b5320 0x7f517c14e760 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f516c0115e0 tx=0x7f516c011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:39:46.719 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.694+0000 7f51727fc640 1 -- 172.21.15.16:0/1288817175 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f516c00f8f0 con 0x7f517c0b5320 2023-05-22T15:39:46.719 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.694+0000 7f51727fc640 1 -- 172.21.15.16:0/1288817175 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f516c00fa50 con 0x7f517c0b5320 2023-05-22T15:39:46.720 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.694+0000 7f5183d76640 1 -- 172.21.15.16:0/1288817175 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f517c0cfef0 con 0x7f517c0b5320 2023-05-22T15:39:46.720 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.694+0000 7f51727fc640 1 -- 172.21.15.16:0/1288817175 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f516c00fdc0 con 0x7f517c0b5320 2023-05-22T15:39:46.720 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.694+0000 7f5183d76640 1 -- 172.21.15.16:0/1288817175 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f517c0d04b0 con 0x7f517c0b5320 2023-05-22T15:39:46.720 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.698+0000 7f5183d76640 1 -- 172.21.15.16:0/1288817175 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f517c04dbe0 con 0x7f517c0b5320 2023-05-22T15:39:46.721 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.698+0000 7f51727fc640 1 -- 172.21.15.16:0/1288817175 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 13) v1 ==== 84401+0+0 (secure 0 0 0) 0x7f516c015460 con 0x7f517c0b5320 2023-05-22T15:39:46.721 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.698+0000 7f51727fc640 1 --2- 172.21.15.16:0/1288817175 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f5160065530 0x7f51600679e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:46.721 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.698+0000 7f5181b12640 1 -- 172.21.15.16:0/1288817175 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f5160065530 msgr2=0x7f51600679e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/36921 2023-05-22T15:39:46.722 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.698+0000 7f51727fc640 1 -- 172.21.15.16:0/1288817175 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 7570+0+0 (secure 0 0 0) 0x7f516c0836f0 con 0x7f517c0b5320 2023-05-22T15:39:46.722 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.698+0000 7f5181b12640 1 --2- 172.21.15.16:0/1288817175 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f5160065530 0x7f51600679e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-05-22T15:39:46.722 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.706+0000 7f51727fc640 1 -- 172.21.15.16:0/1288817175 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f516c050aa0 con 0x7f517c0b5320 2023-05-22T15:39:46.725 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:39:46.718+0000 7f3646e83180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-05-22T15:39:46.863 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.858+0000 7f5183d76640 1 -- 172.21.15.16:0/1288817175 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f517c0b5a90 con 0x7f517c0b5320 2023-05-22T15:39:46.864 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.858+0000 7f51727fc640 1 -- 172.21.15.16:0/1288817175 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v59) v1 ==== 93+0+10 (secure 0 0 0) 0x7f516c0540c0 con 0x7f517c0b5320 2023-05-22T15:39:46.864 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2023-05-22T15:39:46.866 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.862+0000 7f5183d76640 1 -- 172.21.15.16:0/1288817175 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f5160065530 msgr2=0x7f51600679e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-05-22T15:39:46.866 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.862+0000 7f5183d76640 1 --2- 172.21.15.16:0/1288817175 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f5160065530 0x7f51600679e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:46.866 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.862+0000 7f5183d76640 1 -- 172.21.15.16:0/1288817175 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f517c0b5320 msgr2=0x7f517c14e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:46.867 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.862+0000 7f5183d76640 1 --2- 172.21.15.16:0/1288817175 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f517c0b5320 0x7f517c14e760 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f516c0115e0 tx=0x7f516c011aa0 comp rx=0 tx=0).stop 2023-05-22T15:39:46.867 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.862+0000 7f5183d76640 1 -- 172.21.15.16:0/1288817175 shutdown_connections 2023-05-22T15:39:46.867 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.862+0000 7f5183d76640 1 --2- 172.21.15.16:0/1288817175 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f517c15bd20 0x7f517c15c190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:46.867 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.862+0000 7f5183d76640 1 --2- 172.21.15.16:0/1288817175 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f517c0b5320 0x7f517c14e760 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:46.867 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.862+0000 7f5183d76640 1 --2- 172.21.15.16:0/1288817175 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f5160065530 0x7f51600679e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:46.867 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.862+0000 7f5183d76640 1 --2- 172.21.15.16:0/1288817175 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5168004d10 0x7f517c0b3aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:46.868 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.862+0000 7f5183d76640 1 -- 172.21.15.16:0/1288817175 >> 172.21.15.16:0/1288817175 conn(0x7f517c06f5a0 msgr2=0x7f517c0c5960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:39:46.868 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.862+0000 7f5183d76640 1 -- 172.21.15.16:0/1288817175 shutdown_connections 2023-05-22T15:39:46.868 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.862+0000 7f5183d76640 1 -- 172.21.15.16:0/1288817175 wait complete. 2023-05-22T15:39:46.881 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-05-22T15:39:46.949 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:39:46.945+0000 7f6e633d5180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-05-22T15:39:46.978 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:39:46.974+0000 7f3646e83180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-05-22T15:39:46.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.990+0000 7fd5735b4640 1 Processor -- start 2023-05-22T15:39:46.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.990+0000 7fd5735b4640 1 -- start start 2023-05-22T15:39:47.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.990+0000 7fd5735b4640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fd56c0c2700 0x7fd56c06e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:47.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.990+0000 7fd5735b4640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd56c070c30 con 0x7fd56c06ecf0 2023-05-22T15:39:47.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.990+0000 7fd5735b4640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd56c06e320 con 0x7fd56c1648d0 2023-05-22T15:39:47.001 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.990+0000 7fd5735b4640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd56c06f190 con 0x7fd56c0c2700 2023-05-22T15:39:47.001 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.990+0000 7fd571b51640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fd56c06ecf0 0x7fd56c0ce380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:56548/0 (socket says 172.21.15.16:56548) 2023-05-22T15:39:47.002 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.990+0000 7fd571b51640 1 -- 172.21.15.16:0/704961842 learned_addr learned my addr 172.21.15.16:0/704961842 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:39:47.002 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.990+0000 7fd570b4f640 1 --2- 172.21.15.16:0/704961842 >> v2:172.21.15.62:3300/0 conn(0x7fd56c0c2700 0x7fd56c06e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:47.002 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.990+0000 7fd563fff640 1 -- 172.21.15.16:0/704961842 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4129413014 0 0) 0x7fd56c070c30 con 0x7fd56c06ecf0 2023-05-22T15:39:47.003 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.990+0000 7fd563fff640 1 -- 172.21.15.16:0/704961842 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd558003530 con 0x7fd56c06ecf0 2023-05-22T15:39:47.003 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.990+0000 7fd563fff640 1 -- 172.21.15.16:0/704961842 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3354528479 0 0) 0x7fd558003530 con 0x7fd56c06ecf0 2023-05-22T15:39:47.003 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.990+0000 7fd563fff640 1 -- 172.21.15.16:0/704961842 >> v2:172.21.15.62:3300/0 conn(0x7fd56c0c2700 msgr2=0x7fd56c06e7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:39:47.004 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.990+0000 7fd563fff640 1 --2- 172.21.15.16:0/704961842 >> v2:172.21.15.62:3300/0 conn(0x7fd56c0c2700 0x7fd56c06e7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:47.004 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.990+0000 7fd563fff640 1 -- 172.21.15.16:0/704961842 >> v1:172.21.15.62:6789/0 conn(0x7fd56c1648d0 legacy=0x7fd56c0c97b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:47.004 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.990+0000 7fd563fff640 1 -- 172.21.15.16:0/704961842 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd56c06a2c0 con 0x7fd56c06ecf0 2023-05-22T15:39:47.004 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.990+0000 7fd563fff640 1 -- 172.21.15.16:0/704961842 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7fd564001a00 con 0x7fd56c06ecf0 2023-05-22T15:39:47.004 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.990+0000 7fd563fff640 1 -- 172.21.15.16:0/704961842 >> v1:172.21.15.16:6789/0 conn(0x7fd56c06ecf0 legacy=0x7fd56c0ce380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:47.004 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.990+0000 7fd563fff640 1 --2- 172.21.15.16:0/704961842 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd558003980 0x7fd558003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:47.005 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.990+0000 7fd563fff640 1 --2- 172.21.15.16:0/704961842 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd5580047d0 0x7fd558004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:47.005 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.990+0000 7fd563fff640 1 --2- 172.21.15.16:0/704961842 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd558005180 0x7fd55800b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:47.005 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.990+0000 7fd563fff640 1 -- 172.21.15.16:0/704961842 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd56c06f190 con 0x7fd558003980 2023-05-22T15:39:47.005 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.990+0000 7fd563fff640 1 -- 172.21.15.16:0/704961842 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd55800c000 con 0x7fd558005180 2023-05-22T15:39:47.005 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.990+0000 7fd563fff640 1 -- 172.21.15.16:0/704961842 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd55800c1c0 con 0x7fd5580047d0 2023-05-22T15:39:47.005 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.990+0000 7fd571350640 1 --2- 172.21.15.16:0/704961842 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd558003980 0x7fd558003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:47.005 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.990+0000 7fd571350640 1 -- 172.21.15.16:0/704961842 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd5580047d0 msgr2=0x7fd558004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:39:47.005 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.990+0000 7fd571350640 1 --2- 172.21.15.16:0/704961842 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd5580047d0 0x7fd558004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:47.006 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.990+0000 7fd571350640 1 -- 172.21.15.16:0/704961842 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd558005180 msgr2=0x7fd55800b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:39:47.006 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.990+0000 7fd571350640 1 --2- 172.21.15.16:0/704961842 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd558005180 0x7fd55800b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:47.006 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.990+0000 7fd571350640 1 -- 172.21.15.16:0/704961842 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd558003530 con 0x7fd558003980 2023-05-22T15:39:47.006 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.990+0000 7fd571b51640 1 -- 172.21.15.16:0/704961842 reap_dead start 2023-05-22T15:39:47.006 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.994+0000 7fd571350640 1 --2- 172.21.15.16:0/704961842 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd558003980 0x7fd558003d90 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7fd55c001780 tx=0x7fd55c01c400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5518395dfe5d0d0e server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:39:47.006 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.994+0000 7fd563fff640 1 -- 172.21.15.16:0/704961842 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd56c0c9fc0 con 0x7fd558003980 2023-05-22T15:39:47.007 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.994+0000 7fd563fff640 1 -- 172.21.15.16:0/704961842 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd55c0a7050 con 0x7fd558003980 2023-05-22T15:39:47.007 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.994+0000 7fd563fff640 1 -- 172.21.15.16:0/704961842 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd55c003600 con 0x7fd558003980 2023-05-22T15:39:47.007 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:39:47.001+0000 7f6e633d5180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-05-22T15:39:47.010 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.994+0000 7fd5735b4640 1 -- 172.21.15.16:0/704961842 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd558003980 msgr2=0x7fd558003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:47.010 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.994+0000 7fd5735b4640 1 --2- 172.21.15.16:0/704961842 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd558003980 0x7fd558003d90 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7fd55c001780 tx=0x7fd55c01c400 comp rx=0 tx=0).stop 2023-05-22T15:39:47.010 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.994+0000 7fd5735b4640 1 -- 172.21.15.16:0/704961842 shutdown_connections 2023-05-22T15:39:47.010 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.994+0000 7fd5735b4640 1 --2- 172.21.15.16:0/704961842 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd558003980 0x7fd558003d90 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:47.011 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.994+0000 7fd5735b4640 1 -- 172.21.15.16:0/704961842 >> 172.21.15.16:0/704961842 conn(0x7fd56c0cc870 msgr2=0x7fd56c0cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:39:47.011 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.994+0000 7fd5735b4640 1 -- 172.21.15.16:0/704961842 shutdown_connections 2023-05-22T15:39:47.011 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.994+0000 7fd5735b4640 1 -- 172.21.15.16:0/704961842 wait complete. 2023-05-22T15:39:47.011 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.998+0000 7fd5735b4640 1 Processor -- start 2023-05-22T15:39:47.011 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.998+0000 7fd5735b4640 1 -- start start 2023-05-22T15:39:47.011 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.998+0000 7fd5735b4640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd558003980 0x7fd56c0c2050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:47.011 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.998+0000 7fd5735b4640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd56c0c5620 0x7fd56c157540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:47.012 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.998+0000 7fd5735b4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd56c0c3d30 0x7fd56c0c41a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:47.012 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.998+0000 7fd5735b4640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd56c0c2590 con 0x7fd56c0c3d30 2023-05-22T15:39:47.012 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.998+0000 7fd5735b4640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd56c0c38a0 con 0x7fd558003980 2023-05-22T15:39:47.012 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.998+0000 7fd5735b4640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd56c157a80 con 0x7fd56c0c5620 2023-05-22T15:39:47.012 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.998+0000 7fd571b51640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd56c0c3d30 0x7fd56c0c41a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:47.013 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.998+0000 7fd571b51640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd56c0c3d30 0x7fd56c0c41a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52826/0 (socket says 172.21.15.16:52826) 2023-05-22T15:39:47.013 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.998+0000 7fd571b51640 1 -- 172.21.15.16:0/8608806 learned_addr learned my addr 172.21.15.16:0/8608806 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:39:47.013 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.998+0000 7fd570b4f640 1 --2- 172.21.15.16:0/8608806 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd56c0c5620 0x7fd56c157540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:47.013 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.998+0000 7fd571350640 1 --2- 172.21.15.16:0/8608806 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd558003980 0x7fd56c0c2050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:47.014 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.998+0000 7fd571b51640 1 -- 172.21.15.16:0/8608806 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd56c0c5620 msgr2=0x7fd56c157540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:47.014 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.998+0000 7fd571b51640 1 --2- 172.21.15.16:0/8608806 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd56c0c5620 0x7fd56c157540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:47.014 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.998+0000 7fd571b51640 1 -- 172.21.15.16:0/8608806 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd558003980 msgr2=0x7fd56c0c2050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:47.015 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.998+0000 7fd571b51640 1 --2- 172.21.15.16:0/8608806 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd558003980 0x7fd56c0c2050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:47.015 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.998+0000 7fd571b51640 1 -- 172.21.15.16:0/8608806 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd56c0711c0 con 0x7fd56c0c3d30 2023-05-22T15:39:47.015 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.998+0000 7fd571b51640 1 --2- 172.21.15.16:0/8608806 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd56c0c3d30 0x7fd56c0c41a0 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7fd56c06d830 tx=0x7fd56400c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:39:47.015 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.998+0000 7fd561ffb640 1 -- 172.21.15.16:0/8608806 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd56400cee0 con 0x7fd56c0c3d30 2023-05-22T15:39:47.016 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.998+0000 7fd561ffb640 1 -- 172.21.15.16:0/8608806 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd564004d70 con 0x7fd56c0c3d30 2023-05-22T15:39:47.016 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.998+0000 7fd5735b4640 1 -- 172.21.15.16:0/8608806 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd56c06dfb0 con 0x7fd56c0c3d30 2023-05-22T15:39:47.016 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.998+0000 7fd561ffb640 1 -- 172.21.15.16:0/8608806 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd564007310 con 0x7fd56c0c3d30 2023-05-22T15:39:47.016 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.998+0000 7fd5735b4640 1 -- 172.21.15.16:0/8608806 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd56c071350 con 0x7fd56c0c3d30 2023-05-22T15:39:47.017 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.998+0000 7fd5735b4640 1 -- 172.21.15.16:0/8608806 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd56c04dbe0 con 0x7fd56c0c3d30 2023-05-22T15:39:47.017 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.998+0000 7fd561ffb640 1 -- 172.21.15.16:0/8608806 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 13) v1 ==== 84401+0+0 (secure 0 0 0) 0x7fd564007470 con 0x7fd56c0c3d30 2023-05-22T15:39:47.017 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.998+0000 7fd561ffb640 1 --2- 172.21.15.16:0/8608806 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd550065530 0x7fd5500679e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:47.018 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.998+0000 7fd571350640 1 -- 172.21.15.16:0/8608806 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd550065530 msgr2=0x7fd5500679e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/36921 2023-05-22T15:39:47.018 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.998+0000 7fd561ffb640 1 -- 172.21.15.16:0/8608806 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 7570+0+0 (secure 0 0 0) 0x7fd5640824a0 con 0x7fd56c0c3d30 2023-05-22T15:39:47.018 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:46.998+0000 7fd571350640 1 --2- 172.21.15.16:0/8608806 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd550065530 0x7fd5500679e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-05-22T15:39:47.018 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.002+0000 7fd561ffb640 1 -- 172.21.15.16:0/8608806 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fd56404f930 con 0x7fd56c0c3d30 2023-05-22T15:39:47.035 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:39:47.030+0000 7f3646e83180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-05-22T15:39:47.065 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.058+0000 7fd561ffb640 1 -- 172.21.15.16:0/8608806 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mgrmap(e 14) v1 ==== 83941+0+0 (secure 0 0 0) 0x7fd56404fd00 con 0x7fd56c0c3d30 2023-05-22T15:39:47.066 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.058+0000 7fd561ffb640 1 -- 172.21.15.16:0/8608806 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd550065530 msgr2=0x7fd5500679e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-05-22T15:39:47.066 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.058+0000 7fd561ffb640 1 --2- 172.21.15.16:0/8608806 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd550065530 0x7fd5500679e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:47.165 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:39:47.158+0000 7f35ce008640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T15:39:47.165 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:39:47.158+0000 7f35ce008640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T15:39:47.165 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:39:47.158+0000 7f35ce008640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T15:39:47.165 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:39:47.158+0000 7f35ce008640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T15:39:47.166 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:39:47.158+0000 7f35ce008640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T15:39:47.170 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:39:47.166+0000 7f35d3012640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T15:39:47.171 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:39:47.166+0000 7f35d3012640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T15:39:47.171 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:39:47.166+0000 7f35d3012640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T15:39:47.171 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:39:47.166+0000 7f35d3012640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T15:39:47.171 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:39:47.166+0000 7f35d3012640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T15:39:47.203 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.198+0000 7fd5735b4640 1 -- 172.21.15.16:0/8608806 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fd56c0b50e0 con 0x7fd56c0c3d30 2023-05-22T15:39:47.204 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.198+0000 7fd561ffb640 1 -- 172.21.15.16:0/8608806 <== mon.0 v2:172.21.15.16:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v60) v1 ==== 92+0+10 (secure 0 0 0) 0x7fd56407fd30 con 0x7fd56c0c3d30 2023-05-22T15:39:47.204 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 8 2023-05-22T15:39:47.206 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.202+0000 7fd5735b4640 1 -- 172.21.15.16:0/8608806 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd56c0c3d30 msgr2=0x7fd56c0c41a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:47.206 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.202+0000 7fd5735b4640 1 --2- 172.21.15.16:0/8608806 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd56c0c3d30 0x7fd56c0c41a0 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7fd56c06d830 tx=0x7fd56400c900 comp rx=0 tx=0).stop 2023-05-22T15:39:47.206 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.202+0000 7fd5735b4640 1 -- 172.21.15.16:0/8608806 shutdown_connections 2023-05-22T15:39:47.207 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.202+0000 7fd5735b4640 1 --2- 172.21.15.16:0/8608806 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd56c0c3d30 0x7fd56c0c41a0 unknown :-1 s=CLOSED pgs=813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:47.207 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.202+0000 7fd5735b4640 1 --2- 172.21.15.16:0/8608806 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd550065530 0x7fd5500679e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:47.207 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.202+0000 7fd5735b4640 1 --2- 172.21.15.16:0/8608806 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd56c0c5620 0x7fd56c157540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:47.207 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.202+0000 7fd5735b4640 1 --2- 172.21.15.16:0/8608806 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd558003980 0x7fd56c0c2050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:47.207 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.202+0000 7fd5735b4640 1 -- 172.21.15.16:0/8608806 >> 172.21.15.16:0/8608806 conn(0x7fd56c0cc870 msgr2=0x7fd56c0ce380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:39:47.207 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.202+0000 7fd5735b4640 1 -- 172.21.15.16:0/8608806 shutdown_connections 2023-05-22T15:39:47.207 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.202+0000 7fd5735b4640 1 -- 172.21.15.16:0/8608806 wait complete. 2023-05-22T15:39:47.223 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-05-22T15:39:47.331 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.326+0000 7f3cd62dc640 1 Processor -- start 2023-05-22T15:39:47.331 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.326+0000 7f3cd62dc640 1 -- start start 2023-05-22T15:39:47.332 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.326+0000 7f3cd62dc640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3cd015bee0 0x7f3cd006b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:47.332 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.326+0000 7f3cd62dc640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f3cd00c9b00 0x7f3cd006bc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:47.332 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.326+0000 7f3cd62dc640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3cd00c8660 con 0x7f3cd00cb730 2023-05-22T15:39:47.332 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.326+0000 7f3cd62dc640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3cd00ce4a0 con 0x7f3cd00c9b00 2023-05-22T15:39:47.333 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.326+0000 7f3cd62dc640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3cd00ce780 con 0x7f3cd015bee0 2023-05-22T15:39:47.333 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.326+0000 7f3cd4879640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f3cd00c9b00 0x7f3cd006bc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:47.333 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.326+0000 7f3ccffff640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3cd015bee0 0x7f3cd006b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:47.333 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.326+0000 7f3ccf7fe640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f3cd00cb730 0x7f3cd015f850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.16:56988/0 (socket says 172.21.15.16:56988) 2023-05-22T15:39:47.333 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.326+0000 7f3ccffff640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3cd015bee0 0x7f3cd006b6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3301/0 says I am v2:172.21.15.16:50548/0 (socket says 172.21.15.16:50548) 2023-05-22T15:39:47.334 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.326+0000 7f3cd4879640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f3cd00c9b00 0x7f3cd006bc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.16:51762/0 (socket says 172.21.15.16:51762) 2023-05-22T15:39:47.334 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.326+0000 7f3ccf7fe640 1 -- 172.21.15.16:0/510776653 learned_addr learned my addr 172.21.15.16:0/510776653 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:39:47.334 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.326+0000 7f3ccffff640 1 -- 172.21.15.16:0/510776653 >> v2:172.21.15.62:3300/0 conn(0x7f3cd00c9b00 msgr2=0x7f3cd006bc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:39:47.335 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.326+0000 7f3ccffff640 1 --2- 172.21.15.16:0/510776653 >> v2:172.21.15.62:3300/0 conn(0x7f3cd00c9b00 0x7f3cd006bc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:47.335 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.326+0000 7f3ccffff640 1 -- 172.21.15.16:0/510776653 >> v1:172.21.15.62:6789/0 conn(0x7f3cd00cb730 legacy=0x7f3cd015f850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:47.335 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.326+0000 7f3ccffff640 1 -- 172.21.15.16:0/510776653 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cd00cea60 con 0x7f3cd015bee0 2023-05-22T15:39:47.335 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.326+0000 7f3cd4879640 1 --2- 172.21.15.16:0/510776653 >> v2:172.21.15.62:3300/0 conn(0x7f3cd00c9b00 0x7f3cd006bc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-05-22T15:39:47.340 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.330+0000 7f3ccffff640 1 --2- 172.21.15.16:0/510776653 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3cd015bee0 0x7f3cd006b6c0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f3cc80097c0 tx=0x7f3cc801c450 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1fecff4f812bf85 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:39:47.340 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.330+0000 7f3cceffd640 1 -- 172.21.15.16:0/510776653 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cc80ab670 con 0x7f3cd015bee0 2023-05-22T15:39:47.340 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.330+0000 7f3cceffd640 1 -- 172.21.15.16:0/510776653 <== mon.2 v2:172.21.15.62:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3cc80ab7d0 con 0x7f3cd015bee0 2023-05-22T15:39:47.341 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.330+0000 7f3cceffd640 1 -- 172.21.15.16:0/510776653 <== mon.2 v2:172.21.15.62:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cc80abb60 con 0x7f3cd015bee0 2023-05-22T15:39:47.341 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.330+0000 7f3cd62dc640 1 -- 172.21.15.16:0/510776653 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3cd015bee0 msgr2=0x7f3cd006b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:47.341 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.330+0000 7f3cd62dc640 1 --2- 172.21.15.16:0/510776653 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3cd015bee0 0x7f3cd006b6c0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f3cc80097c0 tx=0x7f3cc801c450 comp rx=0 tx=0).stop 2023-05-22T15:39:47.341 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.330+0000 7f3cd62dc640 1 -- 172.21.15.16:0/510776653 shutdown_connections 2023-05-22T15:39:47.342 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.330+0000 7f3cd62dc640 1 --2- 172.21.15.16:0/510776653 >> v2:172.21.15.62:3300/0 conn(0x7f3cd00c9b00 0x7f3cd006bc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:47.342 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.330+0000 7f3cd62dc640 1 --2- 172.21.15.16:0/510776653 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3cd015bee0 0x7f3cd006b6c0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:47.342 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.330+0000 7f3cd62dc640 1 -- 172.21.15.16:0/510776653 >> 172.21.15.16:0/510776653 conn(0x7f3cd0069720 msgr2=0x7f3cd0069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:39:47.343 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.330+0000 7f3cd62dc640 1 -- 172.21.15.16:0/510776653 shutdown_connections 2023-05-22T15:39:47.343 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.330+0000 7f3cd62dc640 1 -- 172.21.15.16:0/510776653 wait complete. 2023-05-22T15:39:47.343 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.330+0000 7f3cd62dc640 1 Processor -- start 2023-05-22T15:39:47.343 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.330+0000 7f3cd62dc640 1 -- start start 2023-05-22T15:39:47.343 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.330+0000 7f3cd62dc640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3cd00c9b00 0x7f3cd014d500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:47.343 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.330+0000 7f3cd62dc640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3cd00cb730 0x7f3cd014dc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:47.343 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.330+0000 7f3cd62dc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3cd015bee0 0x7f3cd014e190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:47.344 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.330+0000 7f3cd62dc640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cd00abde0 con 0x7f3cd015bee0 2023-05-22T15:39:47.344 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.330+0000 7f3cd62dc640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cd014da40 con 0x7f3cd00cb730 2023-05-22T15:39:47.344 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.330+0000 7f3cd62dc640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3cd00ab7d0 con 0x7f3cd00c9b00 2023-05-22T15:39:47.344 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.334+0000 7f3cd4879640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3cd015bee0 0x7f3cd014e190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:47.344 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.334+0000 7f3cd4879640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3cd015bee0 0x7f3cd014e190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52948/0 (socket says 172.21.15.16:52948) 2023-05-22T15:39:47.344 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.334+0000 7f3cd4879640 1 -- 172.21.15.16:0/4136340750 learned_addr learned my addr 172.21.15.16:0/4136340750 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:39:47.344 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.334+0000 7f3ccf7fe640 1 --2- 172.21.15.16:0/4136340750 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3cd00cb730 0x7f3cd014dc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:47.345 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.334+0000 7f3ccffff640 1 --2- 172.21.15.16:0/4136340750 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3cd00c9b00 0x7f3cd014d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:47.345 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.334+0000 7f3cd4879640 1 -- 172.21.15.16:0/4136340750 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3cd00c9b00 msgr2=0x7f3cd014d500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:47.345 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.334+0000 7f3cd4879640 1 --2- 172.21.15.16:0/4136340750 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3cd00c9b00 0x7f3cd014d500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:47.345 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.334+0000 7f3cd4879640 1 -- 172.21.15.16:0/4136340750 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3cd00cb730 msgr2=0x7f3cd014dc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:47.345 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.334+0000 7f3cd4879640 1 --2- 172.21.15.16:0/4136340750 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3cd00cb730 0x7f3cd014dc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:47.345 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.334+0000 7f3cd4879640 1 -- 172.21.15.16:0/4136340750 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cc0009a70 con 0x7f3cd015bee0 2023-05-22T15:39:47.346 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.334+0000 7f3cd4879640 1 --2- 172.21.15.16:0/4136340750 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3cd015bee0 0x7f3cd014e190 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f3cc000efc0 tx=0x7f3cc000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:39:47.346 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.334+0000 7f3cccff9640 1 -- 172.21.15.16:0/4136340750 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cc0009cd0 con 0x7f3cd015bee0 2023-05-22T15:39:47.346 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.334+0000 7f3cd62dc640 1 -- 172.21.15.16:0/4136340750 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3cc8009420 con 0x7f3cd015bee0 2023-05-22T15:39:47.346 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.334+0000 7f3cd62dc640 1 -- 172.21.15.16:0/4136340750 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3cd0168d40 con 0x7f3cd015bee0 2023-05-22T15:39:47.346 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.334+0000 7f3cccff9640 1 -- 172.21.15.16:0/4136340750 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3cc000f040 con 0x7f3cd015bee0 2023-05-22T15:39:47.346 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.334+0000 7f3cccff9640 1 -- 172.21.15.16:0/4136340750 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cc00039a0 con 0x7f3cd015bee0 2023-05-22T15:39:47.346 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.334+0000 7f3cd62dc640 1 -- 172.21.15.16:0/4136340750 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3cb0005260 con 0x7f3cd015bee0 2023-05-22T15:39:47.347 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.334+0000 7f3cccff9640 1 -- 172.21.15.16:0/4136340750 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 14) v1 ==== 83941+0+0 (secure 0 0 0) 0x7f3cc0014480 con 0x7f3cd015bee0 2023-05-22T15:39:47.347 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.334+0000 7f3cccff9640 1 -- 172.21.15.16:0/4136340750 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 7455+0+0 (secure 0 0 0) 0x7f3cc00833e0 con 0x7f3cd015bee0 2023-05-22T15:39:47.347 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.338+0000 7f3cccff9640 1 -- 172.21.15.16:0/4136340750 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f3cc0050d10 con 0x7f3cd015bee0 2023-05-22T15:39:47.492 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.486+0000 7f3cd62dc640 1 -- 172.21.15.16:0/4136340750 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f3cb0005a90 con 0x7f3cd015bee0 2023-05-22T15:39:47.493 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.486+0000 7f3cccff9640 1 -- 172.21.15.16:0/4136340750 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v60) v1 ==== 104+0+11 (secure 0 0 0) 0x7f3cc0054500 con 0x7f3cd015bee0 2023-05-22T15:39:47.493 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2023-05-22T15:39:47.495 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.490+0000 7f3cd62dc640 1 -- 172.21.15.16:0/4136340750 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3cd015bee0 msgr2=0x7f3cd014e190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:47.495 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.490+0000 7f3cd62dc640 1 --2- 172.21.15.16:0/4136340750 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3cd015bee0 0x7f3cd014e190 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f3cc000efc0 tx=0x7f3cc000c490 comp rx=0 tx=0).stop 2023-05-22T15:39:47.495 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.490+0000 7f3cd62dc640 1 -- 172.21.15.16:0/4136340750 shutdown_connections 2023-05-22T15:39:47.496 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.490+0000 7f3cd62dc640 1 --2- 172.21.15.16:0/4136340750 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3cd015bee0 0x7f3cd014e190 unknown :-1 s=CLOSED pgs=827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:47.496 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.490+0000 7f3cd62dc640 1 --2- 172.21.15.16:0/4136340750 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3cd00cb730 0x7f3cd014dc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:47.496 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.490+0000 7f3cd62dc640 1 --2- 172.21.15.16:0/4136340750 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3cd00c9b00 0x7f3cd014d500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:47.496 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.490+0000 7f3cd62dc640 1 -- 172.21.15.16:0/4136340750 >> 172.21.15.16:0/4136340750 conn(0x7f3cd0069720 msgr2=0x7f3cd00cd5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:39:47.497 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.490+0000 7f3cd62dc640 1 -- 172.21.15.16:0/4136340750 shutdown_connections 2023-05-22T15:39:47.497 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.490+0000 7f3cd62dc640 1 -- 172.21.15.16:0/4136340750 wait complete. 2023-05-22T15:39:47.510 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-05-22T15:39:47.621 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.614+0000 7f49ce1d2640 1 Processor -- start 2023-05-22T15:39:47.622 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.614+0000 7f49ce1d2640 1 -- start start 2023-05-22T15:39:47.623 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.618+0000 7f49ce1d2640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f49c8159970 0x7f49c8159d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:47.623 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.618+0000 7f49ce1d2640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f49c815d680 0x7f49c80cb260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:47.623 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.618+0000 7f49ce1d2640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f49c815db00 0x7f49c806b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:47.623 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.618+0000 7f49ce1d2640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49c8070c30 con 0x7f49c8159970 2023-05-22T15:39:47.624 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.618+0000 7f49ce1d2640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49c8070e60 con 0x7f49c815d680 2023-05-22T15:39:47.624 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.618+0000 7f49ce1d2640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49c8071140 con 0x7f49c815db00 2023-05-22T15:39:47.624 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.618+0000 7f49c77fe640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f49c8159970 0x7f49c8159d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:47.624 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.618+0000 7f49c77fe640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f49c8159970 0x7f49c8159d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52952/0 (socket says 172.21.15.16:52952) 2023-05-22T15:39:47.624 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.618+0000 7f49c77fe640 1 -- 172.21.15.16:0/1443701300 learned_addr learned my addr 172.21.15.16:0/1443701300 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:39:47.624 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.618+0000 7f49beffd640 1 --2- 172.21.15.16:0/1443701300 >> v2:172.21.15.62:3300/0 conn(0x7f49c815d680 0x7f49c80cb260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:47.624 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.618+0000 7f49c7fff640 1 --2- 172.21.15.16:0/1443701300 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f49c815db00 0x7f49c806b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:47.625 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.618+0000 7f49c77fe640 1 -- 172.21.15.16:0/1443701300 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f49c815db00 msgr2=0x7f49c806b6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:39:47.625 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.618+0000 7f49c77fe640 1 --2- 172.21.15.16:0/1443701300 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f49c815db00 0x7f49c806b6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:47.625 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.618+0000 7f49c77fe640 1 -- 172.21.15.16:0/1443701300 >> v2:172.21.15.62:3300/0 conn(0x7f49c815d680 msgr2=0x7f49c80cb260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:39:47.625 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.618+0000 7f49c77fe640 1 --2- 172.21.15.16:0/1443701300 >> v2:172.21.15.62:3300/0 conn(0x7f49c815d680 0x7f49c80cb260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:47.625 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.618+0000 7f49c77fe640 1 -- 172.21.15.16:0/1443701300 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49c8071420 con 0x7f49c8159970 2023-05-22T15:39:47.625 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.618+0000 7f49beffd640 1 --2- 172.21.15.16:0/1443701300 >> v2:172.21.15.62:3300/0 conn(0x7f49c815d680 0x7f49c80cb260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:39:47.626 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.618+0000 7f49c77fe640 1 --2- 172.21.15.16:0/1443701300 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f49c8159970 0x7f49c8159d40 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f49c815e760 tx=0x7f49b801c440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5a95149278da839 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:39:47.626 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.618+0000 7f49c6ffd640 1 -- 172.21.15.16:0/1443701300 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49b80ac640 con 0x7f49c8159970 2023-05-22T15:39:47.626 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.622+0000 7f49c6ffd640 1 -- 172.21.15.16:0/1443701300 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49b80ac7a0 con 0x7f49c8159970 2023-05-22T15:39:47.626 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.622+0000 7f49c6ffd640 1 -- 172.21.15.16:0/1443701300 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49b80acb30 con 0x7f49c8159970 2023-05-22T15:39:47.627 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.622+0000 7f49ce1d2640 1 -- 172.21.15.16:0/1443701300 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f49c8159970 msgr2=0x7f49c8159d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:47.627 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.622+0000 7f49ce1d2640 1 --2- 172.21.15.16:0/1443701300 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f49c8159970 0x7f49c8159d40 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f49c815e760 tx=0x7f49b801c440 comp rx=0 tx=0).stop 2023-05-22T15:39:47.627 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.622+0000 7f49ce1d2640 1 -- 172.21.15.16:0/1443701300 shutdown_connections 2023-05-22T15:39:47.627 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.622+0000 7f49ce1d2640 1 --2- 172.21.15.16:0/1443701300 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f49c815db00 0x7f49c806b6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:47.628 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.622+0000 7f49ce1d2640 1 --2- 172.21.15.16:0/1443701300 >> v2:172.21.15.62:3300/0 conn(0x7f49c815d680 0x7f49c80cb260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:47.628 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.622+0000 7f49ce1d2640 1 --2- 172.21.15.16:0/1443701300 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f49c8159970 0x7f49c8159d40 unknown :-1 s=CLOSED pgs=828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:47.628 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.622+0000 7f49ce1d2640 1 -- 172.21.15.16:0/1443701300 >> 172.21.15.16:0/1443701300 conn(0x7f49c8069720 msgr2=0x7f49c8069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:39:47.628 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.622+0000 7f49ce1d2640 1 -- 172.21.15.16:0/1443701300 shutdown_connections 2023-05-22T15:39:47.628 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.622+0000 7f49ce1d2640 1 -- 172.21.15.16:0/1443701300 wait complete. 2023-05-22T15:39:47.629 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.622+0000 7f49ce1d2640 1 Processor -- start 2023-05-22T15:39:47.629 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.622+0000 7f49ce1d2640 1 -- start start 2023-05-22T15:39:47.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.622+0000 7f49ce1d2640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f49c8159970 0x7f49c81508e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:47.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.622+0000 7f49ce1d2640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f49c815d680 0x7f49c80af9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:47.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.622+0000 7f49ce1d2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f49c815db00 0x7f49c80b2b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:47.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.622+0000 7f49ce1d2640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49c80b0480 con 0x7f49c815db00 2023-05-22T15:39:47.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.622+0000 7f49ce1d2640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49c80b0760 con 0x7f49c815d680 2023-05-22T15:39:47.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.622+0000 7f49ce1d2640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49c815af40 con 0x7f49c8159970 2023-05-22T15:39:47.632 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.622+0000 7f49c7fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f49c815db00 0x7f49c80b2b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:47.632 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.622+0000 7f49c7fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f49c815db00 0x7f49c80b2b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52960/0 (socket says 172.21.15.16:52960) 2023-05-22T15:39:47.632 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.622+0000 7f49c7fff640 1 -- 172.21.15.16:0/2398780337 learned_addr learned my addr 172.21.15.16:0/2398780337 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:39:47.632 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.622+0000 7f49c77fe640 1 --2- 172.21.15.16:0/2398780337 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f49c8159970 0x7f49c81508e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:47.633 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.622+0000 7f49beffd640 1 --2- 172.21.15.16:0/2398780337 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f49c815d680 0x7f49c80af9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:47.633 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.622+0000 7f49c7fff640 1 -- 172.21.15.16:0/2398780337 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f49c8159970 msgr2=0x7f49c81508e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:47.633 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.622+0000 7f49c7fff640 1 --2- 172.21.15.16:0/2398780337 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f49c8159970 0x7f49c81508e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:47.633 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.622+0000 7f49c7fff640 1 -- 172.21.15.16:0/2398780337 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f49c815d680 msgr2=0x7f49c80af9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:47.633 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.622+0000 7f49c7fff640 1 --2- 172.21.15.16:0/2398780337 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f49c815d680 0x7f49c80af9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:47.633 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.622+0000 7f49c7fff640 1 -- 172.21.15.16:0/2398780337 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49b80094d0 con 0x7f49c815db00 2023-05-22T15:39:47.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.626+0000 7f49c7fff640 1 --2- 172.21.15.16:0/2398780337 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f49c815db00 0x7f49c80b2b90 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f49c000cc30 tx=0x7f49c000a5c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:39:47.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.626+0000 7f49c4ff9640 1 -- 172.21.15.16:0/2398780337 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49c000ce40 con 0x7f49c815db00 2023-05-22T15:39:47.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.626+0000 7f49c4ff9640 1 -- 172.21.15.16:0/2398780337 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49c00039a0 con 0x7f49c815db00 2023-05-22T15:39:47.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.626+0000 7f49ce1d2640 1 -- 172.21.15.16:0/2398780337 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49c80cac90 con 0x7f49c815db00 2023-05-22T15:39:47.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.626+0000 7f49c4ff9640 1 -- 172.21.15.16:0/2398780337 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49c0003d30 con 0x7f49c815db00 2023-05-22T15:39:47.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.626+0000 7f49ce1d2640 1 -- 172.21.15.16:0/2398780337 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49c80cdbf0 con 0x7f49c815db00 2023-05-22T15:39:47.635 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.626+0000 7f49ce1d2640 1 -- 172.21.15.16:0/2398780337 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49c80cae20 con 0x7f49c815db00 2023-05-22T15:39:47.635 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.626+0000 7f49c4ff9640 1 -- 172.21.15.16:0/2398780337 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 14) v1 ==== 83941+0+0 (secure 0 0 0) 0x7f49c0003e90 con 0x7f49c815db00 2023-05-22T15:39:47.635 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.626+0000 7f49c4ff9640 1 -- 172.21.15.16:0/2398780337 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 7455+0+0 (secure 0 0 0) 0x7f49c0083000 con 0x7f49c815db00 2023-05-22T15:39:47.636 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.630+0000 7f49c4ff9640 1 -- 172.21.15.16:0/2398780337 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f49c0050ba0 con 0x7f49c815db00 2023-05-22T15:39:47.787 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.782+0000 7f49ce1d2640 1 -- 172.21.15.16:0/2398780337 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f49c804dbe0 con 0x7f49c815db00 2023-05-22T15:39:47.788 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.782+0000 7f49c4ff9640 1 -- 172.21.15.16:0/2398780337 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v60) v1 ==== 100+0+11 (secure 0 0 0) 0x7f49c001d020 con 0x7f49c815db00 2023-05-22T15:39:47.788 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 64 2023-05-22T15:39:47.789 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.782+0000 7f49ce1d2640 1 -- 172.21.15.16:0/2398780337 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f49c815db00 msgr2=0x7f49c80b2b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:47.790 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.782+0000 7f49ce1d2640 1 --2- 172.21.15.16:0/2398780337 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f49c815db00 0x7f49c80b2b90 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f49c000cc30 tx=0x7f49c000a5c0 comp rx=0 tx=0).stop 2023-05-22T15:39:47.790 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.782+0000 7f49ce1d2640 1 -- 172.21.15.16:0/2398780337 shutdown_connections 2023-05-22T15:39:47.790 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.782+0000 7f49ce1d2640 1 --2- 172.21.15.16:0/2398780337 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f49c815db00 0x7f49c80b2b90 unknown :-1 s=CLOSED pgs=829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:47.790 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.782+0000 7f49ce1d2640 1 --2- 172.21.15.16:0/2398780337 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f49c815d680 0x7f49c80af9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:47.790 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.782+0000 7f49ce1d2640 1 --2- 172.21.15.16:0/2398780337 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f49c8159970 0x7f49c81508e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:47.790 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.782+0000 7f49ce1d2640 1 -- 172.21.15.16:0/2398780337 >> 172.21.15.16:0/2398780337 conn(0x7f49c8069720 msgr2=0x7f49c80c4230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:39:47.791 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.782+0000 7f49ce1d2640 1 -- 172.21.15.16:0/2398780337 shutdown_connections 2023-05-22T15:39:47.791 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.782+0000 7f49ce1d2640 1 -- 172.21.15.16:0/2398780337 wait complete. 2023-05-22T15:39:47.805 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-05-22T15:39:47.923 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.918+0000 7f011c4f1640 1 Processor -- start 2023-05-22T15:39:47.923 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.918+0000 7f011c4f1640 1 -- start start 2023-05-22T15:39:47.924 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.918+0000 7f011c4f1640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f01140b2880 0x7f01140cbfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:47.924 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.918+0000 7f011c4f1640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01140cf9a0 con 0x7f0114159cd0 2023-05-22T15:39:47.924 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.918+0000 7f011c4f1640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f011415aba0 con 0x7f01140cc520 2023-05-22T15:39:47.924 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.918+0000 7f011c4f1640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f011415ae80 con 0x7f01140b2880 2023-05-22T15:39:47.924 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.918+0000 7f011a28d640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f0114159cd0 0x7f011415a0a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:56564/0 (socket says 172.21.15.16:56564) 2023-05-22T15:39:47.924 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.918+0000 7f011a28d640 1 -- 172.21.15.16:0/1271367844 learned_addr learned my addr 172.21.15.16:0/1271367844 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:39:47.926 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.918+0000 7f0119a8c640 1 --2- 172.21.15.16:0/1271367844 >> v2:172.21.15.62:3300/0 conn(0x7f01140b2880 0x7f01140cbfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:47.926 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.918+0000 7f011928b640 1 -- 172.21.15.16:0/1271367844 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2338355036 0 0) 0x7f01140cf9a0 con 0x7f0114159cd0 2023-05-22T15:39:47.926 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.918+0000 7f011928b640 1 -- 172.21.15.16:0/1271367844 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0100003530 con 0x7f0114159cd0 2023-05-22T15:39:47.926 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.918+0000 7f011928b640 1 -- 172.21.15.16:0/1271367844 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1685661834 0 0) 0x7f0100003530 con 0x7f0114159cd0 2023-05-22T15:39:47.927 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.918+0000 7f011928b640 1 -- 172.21.15.16:0/1271367844 >> v2:172.21.15.62:3300/0 conn(0x7f01140b2880 msgr2=0x7f01140cbfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:39:47.927 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.918+0000 7f011928b640 1 --2- 172.21.15.16:0/1271367844 >> v2:172.21.15.62:3300/0 conn(0x7f01140b2880 0x7f01140cbfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:47.927 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.918+0000 7f011928b640 1 -- 172.21.15.16:0/1271367844 >> v1:172.21.15.62:6789/0 conn(0x7f01140cc520 legacy=0x7f01140cfe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:47.927 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.918+0000 7f011928b640 1 -- 172.21.15.16:0/1271367844 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f011415b160 con 0x7f0114159cd0 2023-05-22T15:39:47.927 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.918+0000 7f011928b640 1 -- 172.21.15.16:0/1271367844 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7f0110001680 con 0x7f0114159cd0 2023-05-22T15:39:47.927 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.918+0000 7f011928b640 1 -- 172.21.15.16:0/1271367844 >> v1:172.21.15.16:6789/0 conn(0x7f0114159cd0 legacy=0x7f011415a0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:47.927 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.918+0000 7f011928b640 1 --2- 172.21.15.16:0/1271367844 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0100003980 0x7f0100003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:47.927 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.918+0000 7f011928b640 1 --2- 172.21.15.16:0/1271367844 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f01000047d0 0x7f0100004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:47.928 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.918+0000 7f011928b640 1 --2- 172.21.15.16:0/1271367844 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0100005180 0x7f010000b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:47.928 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.918+0000 7f011928b640 1 -- 172.21.15.16:0/1271367844 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f011415ae80 con 0x7f0100005180 2023-05-22T15:39:47.928 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.918+0000 7f011928b640 1 -- 172.21.15.16:0/1271367844 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f010000c000 con 0x7f0100003980 2023-05-22T15:39:47.928 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.918+0000 7f011928b640 1 -- 172.21.15.16:0/1271367844 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f010000c1c0 con 0x7f01000047d0 2023-05-22T15:39:47.928 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.918+0000 7f011aa8e640 1 --2- 172.21.15.16:0/1271367844 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0100005180 0x7f010000b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:47.928 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.918+0000 7f011a28d640 1 --2- 172.21.15.16:0/1271367844 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0100003980 0x7f0100003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:47.928 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.918+0000 7f0119a8c640 1 --2- 172.21.15.16:0/1271367844 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f01000047d0 0x7f0100004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:47.929 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.918+0000 7f011aa8e640 1 -- 172.21.15.16:0/1271367844 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f01000047d0 msgr2=0x7f0100004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:39:47.929 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.918+0000 7f011aa8e640 1 --2- 172.21.15.16:0/1271367844 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f01000047d0 0x7f0100004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:47.929 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.918+0000 7f011aa8e640 1 -- 172.21.15.16:0/1271367844 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0100003980 msgr2=0x7f0100003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:39:47.929 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.918+0000 7f011aa8e640 1 --2- 172.21.15.16:0/1271367844 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0100003980 0x7f0100003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:47.929 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.918+0000 7f011aa8e640 1 -- 172.21.15.16:0/1271367844 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0100003530 con 0x7f0100005180 2023-05-22T15:39:47.929 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.918+0000 7f0119a8c640 1 --2- 172.21.15.16:0/1271367844 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f01000047d0 0x7f0100004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:39:47.934 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.922+0000 7f011aa8e640 1 -- 172.21.15.16:0/1271367844 reap_dead start 2023-05-22T15:39:47.934 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.922+0000 7f011aa8e640 1 --2- 172.21.15.16:0/1271367844 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0100005180 0x7f010000b990 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f010c001e70 tx=0x7f010c01c760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30895d10d2554f00 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:39:47.934 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.922+0000 7f011928b640 1 -- 172.21.15.16:0/1271367844 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f010c003b80 con 0x7f0100005180 2023-05-22T15:39:47.935 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.922+0000 7f011928b640 1 -- 172.21.15.16:0/1271367844 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f010c003d20 con 0x7f0100005180 2023-05-22T15:39:47.935 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.922+0000 7f011928b640 1 -- 172.21.15.16:0/1271367844 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f010c005260 con 0x7f0100005180 2023-05-22T15:39:47.935 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.922+0000 7f011c4f1640 1 -- 172.21.15.16:0/1271367844 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0100005180 msgr2=0x7f010000b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:47.935 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.922+0000 7f011c4f1640 1 --2- 172.21.15.16:0/1271367844 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0100005180 0x7f010000b990 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f010c001e70 tx=0x7f010c01c760 comp rx=0 tx=0).stop 2023-05-22T15:39:47.936 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.922+0000 7f011c4f1640 1 -- 172.21.15.16:0/1271367844 shutdown_connections 2023-05-22T15:39:47.936 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.922+0000 7f011c4f1640 1 --2- 172.21.15.16:0/1271367844 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0100005180 0x7f010000b990 unknown :-1 s=CLOSED pgs=831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:47.936 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.922+0000 7f011c4f1640 1 -- 172.21.15.16:0/1271367844 >> 172.21.15.16:0/1271367844 conn(0x7f011406f5c0 msgr2=0x7f011406f9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:39:47.936 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.922+0000 7f011c4f1640 1 -- 172.21.15.16:0/1271367844 shutdown_connections 2023-05-22T15:39:47.936 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.922+0000 7f011c4f1640 1 -- 172.21.15.16:0/1271367844 wait complete. 2023-05-22T15:39:47.936 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.922+0000 7f011c4f1640 1 Processor -- start 2023-05-22T15:39:47.937 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.922+0000 7f011c4f1640 1 -- start start 2023-05-22T15:39:47.937 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.926+0000 7f011c4f1640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0100005180 0x7f01140b58d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:47.937 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.926+0000 7f011c4f1640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f01140c46f0 0x7f01140c4b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:47.937 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.926+0000 7f011c4f1640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f01140b3090 0x7f01140b3500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:47.937 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.926+0000 7f011c4f1640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01140c50a0 con 0x7f01140b3090 2023-05-22T15:39:47.937 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.926+0000 7f011c4f1640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01140c9bd0 con 0x7f0100005180 2023-05-22T15:39:47.937 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.926+0000 7f011c4f1640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01140c8fc0 con 0x7f01140c46f0 2023-05-22T15:39:47.937 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.926+0000 7f011aa8e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f01140b3090 0x7f01140b3500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:47.938 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.926+0000 7f011aa8e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f01140b3090 0x7f01140b3500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52986/0 (socket says 172.21.15.16:52986) 2023-05-22T15:39:47.938 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.926+0000 7f011aa8e640 1 -- 172.21.15.16:0/225029422 learned_addr learned my addr 172.21.15.16:0/225029422 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:39:47.938 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.926+0000 7f011a28d640 1 --2- 172.21.15.16:0/225029422 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0100005180 0x7f01140b58d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:47.938 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.926+0000 7f0119a8c640 1 --2- 172.21.15.16:0/225029422 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f01140c46f0 0x7f01140c4b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:47.938 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.926+0000 7f011aa8e640 1 -- 172.21.15.16:0/225029422 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f01140c46f0 msgr2=0x7f01140c4b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:47.938 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.926+0000 7f011aa8e640 1 --2- 172.21.15.16:0/225029422 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f01140c46f0 0x7f01140c4b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:47.938 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.926+0000 7f011aa8e640 1 -- 172.21.15.16:0/225029422 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0100005180 msgr2=0x7f01140b58d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:47.939 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.926+0000 7f011aa8e640 1 --2- 172.21.15.16:0/225029422 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0100005180 0x7f01140b58d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:47.939 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.926+0000 7f011aa8e640 1 -- 172.21.15.16:0/225029422 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01140cc9a0 con 0x7f01140b3090 2023-05-22T15:39:47.939 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.926+0000 7f011aa8e640 1 --2- 172.21.15.16:0/225029422 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f01140b3090 0x7f01140b3500 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f010c01cc40 tx=0x7f0100005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:39:47.939 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.926+0000 7f010affd640 1 -- 172.21.15.16:0/225029422 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f010c0ac070 con 0x7f01140b3090 2023-05-22T15:39:47.939 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.926+0000 7f010affd640 1 -- 172.21.15.16:0/225029422 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f010c003e90 con 0x7f01140b3090 2023-05-22T15:39:47.939 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.926+0000 7f011c4f1640 1 -- 172.21.15.16:0/225029422 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01140cf630 con 0x7f01140b3090 2023-05-22T15:39:47.939 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.926+0000 7f010affd640 1 -- 172.21.15.16:0/225029422 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f010c005020 con 0x7f01140b3090 2023-05-22T15:39:47.939 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.926+0000 7f011c4f1640 1 -- 172.21.15.16:0/225029422 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01140cbfe0 con 0x7f01140b3090 2023-05-22T15:39:47.940 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.926+0000 7f011c4f1640 1 -- 172.21.15.16:0/225029422 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f011404dbe0 con 0x7f01140b3090 2023-05-22T15:39:47.940 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.926+0000 7f010affd640 1 -- 172.21.15.16:0/225029422 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 14) v1 ==== 83941+0+0 (secure 0 0 0) 0x7f010c005180 con 0x7f01140b3090 2023-05-22T15:39:47.940 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.926+0000 7f010affd640 1 -- 172.21.15.16:0/225029422 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 7455+0+0 (secure 0 0 0) 0x7f010c11bb70 con 0x7f01140b3090 2023-05-22T15:39:47.940 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:47.930+0000 7f010affd640 1 -- 172.21.15.16:0/225029422 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f010c0e95e0 con 0x7f01140b3090 2023-05-22T15:39:48.081 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.074+0000 7f010affd640 1 -- 172.21.15.16:0/225029422 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mgrmap(e 15) v1 ==== 84068+0+0 (secure 0 0 0) 0x7f010c0e9090 con 0x7f01140b3090 2023-05-22T15:39:48.081 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.074+0000 7f010affd640 1 --2- 172.21.15.16:0/225029422 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f00f806bc70 0x7f00f806e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:48.085 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.078+0000 7f011a28d640 1 --2- 172.21.15.16:0/225029422 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f00f806bc70 0x7f00f806e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:48.086 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.082+0000 7f011a28d640 1 --2- 172.21.15.16:0/225029422 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f00f806bc70 0x7f00f806e050 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f0110001ec0 tx=0x7f0110009040 comp rx=0 tx=0).ready entity=mgr.6240 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:39:48.141 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.134+0000 7f011c4f1640 1 -- 172.21.15.16:0/225029422 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f01140caf00 con 0x7f01140b3090 2023-05-22T15:39:48.141 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.134+0000 7f010affd640 1 -- 172.21.15.16:0/225029422 <== mon.0 v2:172.21.15.16:3300/0 8 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) v1 ==== 94+0+4648 (secure 0 0 0) 0x7f010c1186c0 con 0x7f01140b3090 2023-05-22T15:39:48.142 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:39:48.142 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":14,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4904,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6833/3153072235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":3153072235},{"type":"v1","addr":"172.21.15.62:6833","nonce":3153072235}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":5164,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/2783822861","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":2783822861},{"type":"v1","addr":"172.21.15.62:6835","nonce":2783822861}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":5571,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/1361634240","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1361634240},{"type":"v1","addr":"172.21.15.16:6835","nonce":1361634240}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":14,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-05-22T15:28:13.320259+0000","modified":"2023-05-22T15:28:15.336631+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5574},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5574":{"gid":5574,"name":"c","rank":0,"incarnation":12,"state":"up:active","state_seq":3,"addr":"172.21.15.16:6837/1033866899","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":1033866899},{"type":"v1","addr":"172.21.15.16:6837","nonce":1033866899}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-05-22T15:39:48.142 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 14 2023-05-22T15:39:48.143 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.138+0000 7f011c4f1640 1 -- 172.21.15.16:0/225029422 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f00f806bc70 msgr2=0x7f00f806e050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:48.144 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.138+0000 7f011c4f1640 1 --2- 172.21.15.16:0/225029422 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f00f806bc70 0x7f00f806e050 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f0110001ec0 tx=0x7f0110009040 comp rx=0 tx=0).stop 2023-05-22T15:39:48.144 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.138+0000 7f011c4f1640 1 -- 172.21.15.16:0/225029422 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f01140b3090 msgr2=0x7f01140b3500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:48.144 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.138+0000 7f011c4f1640 1 --2- 172.21.15.16:0/225029422 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f01140b3090 0x7f01140b3500 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f010c01cc40 tx=0x7f0100005800 comp rx=0 tx=0).stop 2023-05-22T15:39:48.144 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.138+0000 7f011c4f1640 1 -- 172.21.15.16:0/225029422 shutdown_connections 2023-05-22T15:39:48.144 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.138+0000 7f011c4f1640 1 --2- 172.21.15.16:0/225029422 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f01140b3090 0x7f01140b3500 unknown :-1 s=CLOSED pgs=832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:48.144 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.138+0000 7f011c4f1640 1 --2- 172.21.15.16:0/225029422 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f00f806bc70 0x7f00f806e050 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:48.145 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.138+0000 7f011c4f1640 1 --2- 172.21.15.16:0/225029422 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f01140c46f0 0x7f01140c4b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:48.145 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.138+0000 7f011c4f1640 1 --2- 172.21.15.16:0/225029422 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0100005180 0x7f01140b58d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:48.145 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.138+0000 7f011c4f1640 1 -- 172.21.15.16:0/225029422 >> 172.21.15.16:0/225029422 conn(0x7f011406f5c0 msgr2=0x7f01140cd070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:39:48.145 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.138+0000 7f011c4f1640 1 -- 172.21.15.16:0/225029422 shutdown_connections 2023-05-22T15:39:48.145 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.138+0000 7f011c4f1640 1 -- 172.21.15.16:0/225029422 wait complete. 2023-05-22T15:39:48.159 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-05-22T15:39:48.273 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.266+0000 7f756936a640 1 Processor -- start 2023-05-22T15:39:48.274 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.266+0000 7f756936a640 1 -- start start 2023-05-22T15:39:48.274 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.270+0000 7f756936a640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f75641567c0 0x7f75640afb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:48.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.270+0000 7f756936a640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f7564159a50 0x7f75640b0080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:48.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.270+0000 7f756936a640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7564156b90 con 0x7f75640cb6a0 2023-05-22T15:39:48.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.270+0000 7f756936a640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f756415a0b0 con 0x7f7564159a50 2023-05-22T15:39:48.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.270+0000 7f756936a640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75640ca8e0 con 0x7f75641567c0 2023-05-22T15:39:48.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.270+0000 7f75637fe640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f7564159a50 0x7f75640b0080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:48.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.270+0000 7f75637fe640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f7564159a50 0x7f75640b0080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52994/0 (socket says 172.21.15.16:52994) 2023-05-22T15:39:48.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.270+0000 7f75637fe640 1 -- 172.21.15.16:0/3277085573 learned_addr learned my addr 172.21.15.16:0/3277085573 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:39:48.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.270+0000 7f7562ffd640 1 --2- 172.21.15.16:0/3277085573 >> v2:172.21.15.62:3300/0 conn(0x7f75641567c0 0x7f75640afb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:48.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.270+0000 7f75637fe640 1 -- 172.21.15.16:0/3277085573 >> v2:172.21.15.62:3300/0 conn(0x7f75641567c0 msgr2=0x7f75640afb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:39:48.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.270+0000 7f75637fe640 1 --2- 172.21.15.16:0/3277085573 >> v2:172.21.15.62:3300/0 conn(0x7f75641567c0 0x7f75640afb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:48.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.270+0000 7f75637fe640 1 -- 172.21.15.16:0/3277085573 >> v1:172.21.15.16:6789/0 conn(0x7f75640cb6a0 legacy=0x7f7564159340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:48.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.270+0000 7f75637fe640 1 -- 172.21.15.16:0/3277085573 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75640cabc0 con 0x7f7564159a50 2023-05-22T15:39:48.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.270+0000 7f75637fe640 1 --2- 172.21.15.16:0/3277085573 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7564159a50 0x7f75640b0080 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7f75640c50a0 tx=0x7f755c01c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36653bed86ef5b4 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:39:48.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.270+0000 7f7561ffb640 1 -- 172.21.15.16:0/3277085573 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f755c0ab600 con 0x7f7564159a50 2023-05-22T15:39:48.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.270+0000 7f7561ffb640 1 -- 172.21.15.16:0/3277085573 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f755c0a7050 con 0x7f7564159a50 2023-05-22T15:39:48.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.270+0000 7f7561ffb640 1 -- 172.21.15.16:0/3277085573 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f755c0ab980 con 0x7f7564159a50 2023-05-22T15:39:48.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.274+0000 7f756936a640 1 -- 172.21.15.16:0/3277085573 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7564159a50 msgr2=0x7f75640b0080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:48.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.274+0000 7f756936a640 1 --2- 172.21.15.16:0/3277085573 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7564159a50 0x7f75640b0080 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7f75640c50a0 tx=0x7f755c01c500 comp rx=0 tx=0).stop 2023-05-22T15:39:48.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.274+0000 7f756936a640 1 -- 172.21.15.16:0/3277085573 shutdown_connections 2023-05-22T15:39:48.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.274+0000 7f756936a640 1 --2- 172.21.15.16:0/3277085573 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7564159a50 0x7f75640b0080 unknown :-1 s=CLOSED pgs=834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:48.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.274+0000 7f756936a640 1 --2- 172.21.15.16:0/3277085573 >> v2:172.21.15.62:3300/0 conn(0x7f75641567c0 0x7f75640afb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:48.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.274+0000 7f756936a640 1 -- 172.21.15.16:0/3277085573 >> 172.21.15.16:0/3277085573 conn(0x7f7564069720 msgr2=0x7f7564069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:39:48.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.274+0000 7f756936a640 1 -- 172.21.15.16:0/3277085573 shutdown_connections 2023-05-22T15:39:48.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.274+0000 7f756936a640 1 -- 172.21.15.16:0/3277085573 wait complete. 2023-05-22T15:39:48.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.274+0000 7f756936a640 1 Processor -- start 2023-05-22T15:39:48.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.274+0000 7f756936a640 1 -- start start 2023-05-22T15:39:48.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.274+0000 7f756936a640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f75640cb6a0 0x7f756415c940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:48.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.274+0000 7f756936a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f75641567c0 0x7f75641464f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:48.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.274+0000 7f756936a640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7564159a50 0x7f7564146a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:48.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.274+0000 7f756936a640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f756414c870 con 0x7f75641567c0 2023-05-22T15:39:48.282 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.274+0000 7f756936a640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f756415ad10 con 0x7f7564159a50 2023-05-22T15:39:48.282 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.274+0000 7f756936a640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f756414bfd0 con 0x7f75640cb6a0 2023-05-22T15:39:48.282 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.274+0000 7f75627fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f75641567c0 0x7f75641464f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:48.282 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.274+0000 7f75627fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f75641567c0 0x7f75641464f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53006/0 (socket says 172.21.15.16:53006) 2023-05-22T15:39:48.282 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.274+0000 7f75627fc640 1 -- 172.21.15.16:0/1532719528 learned_addr learned my addr 172.21.15.16:0/1532719528 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:39:48.282 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.274+0000 7f75627fc640 1 -- 172.21.15.16:0/1532719528 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f75640cb6a0 msgr2=0x7f756415c940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:48.282 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.274+0000 7f75627fc640 1 --2- 172.21.15.16:0/1532719528 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f75640cb6a0 0x7f756415c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:48.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.274+0000 7f75627fc640 1 -- 172.21.15.16:0/1532719528 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7564159a50 msgr2=0x7f7564146a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:48.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.274+0000 7f75637fe640 1 --2- 172.21.15.16:0/1532719528 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7564159a50 0x7f7564146a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:48.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.274+0000 7f75627fc640 1 --2- 172.21.15.16:0/1532719528 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7564159a50 0x7f7564146a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:48.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.274+0000 7f75627fc640 1 -- 172.21.15.16:0/1532719528 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f755c0094d0 con 0x7f75641567c0 2023-05-22T15:39:48.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.274+0000 7f75627fc640 1 --2- 172.21.15.16:0/1532719528 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f75641567c0 0x7f75641464f0 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f755800c500 tx=0x7f755800c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:39:48.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.274+0000 7f754b7fe640 1 -- 172.21.15.16:0/1532719528 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7558005220 con 0x7f75641567c0 2023-05-22T15:39:48.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.274+0000 7f754b7fe640 1 -- 172.21.15.16:0/1532719528 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7558005380 con 0x7f75641567c0 2023-05-22T15:39:48.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.274+0000 7f756936a640 1 -- 172.21.15.16:0/1532719528 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75640c9010 con 0x7f75641567c0 2023-05-22T15:39:48.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.274+0000 7f754b7fe640 1 -- 172.21.15.16:0/1532719528 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7558007270 con 0x7f75641567c0 2023-05-22T15:39:48.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.274+0000 7f756936a640 1 -- 172.21.15.16:0/1532719528 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75640c8e80 con 0x7f75641567c0 2023-05-22T15:39:48.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.274+0000 7f756936a640 1 -- 172.21.15.16:0/1532719528 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75640c91a0 con 0x7f75641567c0 2023-05-22T15:39:48.285 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.278+0000 7f754b7fe640 1 -- 172.21.15.16:0/1532719528 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 15) v1 ==== 84068+0+0 (secure 0 0 0) 0x7f75580073d0 con 0x7f75641567c0 2023-05-22T15:39:48.285 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.278+0000 7f754b7fe640 1 --2- 172.21.15.16:0/1532719528 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f75500651d0 0x7f7550067680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:48.285 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.278+0000 7f754b7fe640 1 -- 172.21.15.16:0/1532719528 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 7455+0+0 (secure 0 0 0) 0x7f75580830b0 con 0x7f75641567c0 2023-05-22T15:39:48.285 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.278+0000 7f7562ffd640 1 --2- 172.21.15.16:0/1532719528 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f75500651d0 0x7f7550067680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:48.285 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.278+0000 7f7562ffd640 1 --2- 172.21.15.16:0/1532719528 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f75500651d0 0x7f7550067680 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f75540070b0 tx=0x7f7554007040 comp rx=0 tx=0).ready entity=mgr.6240 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:39:48.286 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.282+0000 7f754b7fe640 1 -- 172.21.15.16:0/1532719528 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f7558050a30 con 0x7f75641567c0 2023-05-22T15:39:48.435 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.430+0000 7f756936a640 1 -- 172.21.15.16:0/1532719528 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7564156b90 con 0x7f75641567c0 2023-05-22T15:39:48.437 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.430+0000 7f754b7fe640 1 -- 172.21.15.16:0/1532719528 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v60) v1 ==== 74+0+19649 (secure 0 0 0) 0x7f7564156b90 con 0x7f75641567c0 2023-05-22T15:39:48.437 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:39:48.438 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":60,"fsid":"b2719834-7847-4d98-90a6-f65d57f85c49","created":"2023-05-22T15:26:25.259314+0000","modified":"2023-05-22T15:39:47.055848+0000","last_up_change":"2023-05-22T15:26:36.448524+0000","last_in_change":"2023-05-22T15:26:27.675668+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-05-22T15:26:38.019781+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-05-22T15:26:43.570514+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2023-05-22T15:28:11.902605+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"41","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2023-05-22T15:28:12.298278+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"60","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":221,"snap_epoch":60,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9e7ff854-b25f-4cbb-a5c2-1799d9c53d2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":37021},{"type":"v1","addr":"172.21.15.16:6825","nonce":37021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":37021},{"type":"v1","addr":"172.21.15.16:6827","nonce":37021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":37021},{"type":"v1","addr":"172.21.15.16:6831","nonce":37021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":37021},{"type":"v1","addr":"172.21.15.16:6829","nonce":37021}]},"public_addr":"172.21.15.16:6825/37021","cluster_addr":"172.21.15.16:6827/37021","heartbeat_back_addr":"172.21.15.16:6831/37021","heartbeat_front_addr":"172.21.15.16:6829/37021","state":["exists","up"]},{"osd":1,"uuid":"4702fbc9-a927-4565-aca5-41dc69cc8264","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":37008},{"type":"v1","addr":"172.21.15.16:6801","nonce":37008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":37008},{"type":"v1","addr":"172.21.15.16:6803","nonce":37008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":37008},{"type":"v1","addr":"172.21.15.16:6807","nonce":37008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":37008},{"type":"v1","addr":"172.21.15.16:6805","nonce":37008}]},"public_addr":"172.21.15.16:6801/37008","cluster_addr":"172.21.15.16:6803/37008","heartbeat_back_addr":"172.21.15.16:6807/37008","heartbeat_front_addr":"172.21.15.16:6805/37008","state":["exists","up"]},{"osd":2,"uuid":"1e7bba46-4eba-47ef-bb81-5e2d371638ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":37009},{"type":"v1","addr":"172.21.15.16:6809","nonce":37009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":37009},{"type":"v1","addr":"172.21.15.16:6811","nonce":37009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":37009},{"type":"v1","addr":"172.21.15.16:6815","nonce":37009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":37009},{"type":"v1","addr":"172.21.15.16:6813","nonce":37009}]},"public_addr":"172.21.15.16:6809/37009","cluster_addr":"172.21.15.16:6811/37009","heartbeat_back_addr":"172.21.15.16:6815/37009","heartbeat_front_addr":"172.21.15.16:6813/37009","state":["exists","up"]},{"osd":3,"uuid":"383cacc0-5a88-4904-990e-498ca59ef48a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":37020},{"type":"v1","addr":"172.21.15.16:6817","nonce":37020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":37020},{"type":"v1","addr":"172.21.15.16:6819","nonce":37020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":37020},{"type":"v1","addr":"172.21.15.16:6823","nonce":37020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":37020},{"type":"v1","addr":"172.21.15.16:6821","nonce":37020}]},"public_addr":"172.21.15.16:6817/37020","cluster_addr":"172.21.15.16:6819/37020","heartbeat_back_addr":"172.21.15.16:6823/37020","heartbeat_front_addr":"172.21.15.16:6821/37020","state":["exists","up"]},{"osd":4,"uuid":"970500ef-95b7-42c1-b080-eed4fae7013d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":37353},{"type":"v1","addr":"172.21.15.62:6817","nonce":37353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6819","nonce":37353},{"type":"v1","addr":"172.21.15.62:6822","nonce":37353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37353},{"type":"v1","addr":"172.21.15.62:6831","nonce":37353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37353},{"type":"v1","addr":"172.21.15.62:6826","nonce":37353}]},"public_addr":"172.21.15.62:6817/37353","cluster_addr":"172.21.15.62:6822/37353","heartbeat_back_addr":"172.21.15.62:6831/37353","heartbeat_front_addr":"172.21.15.62:6826/37353","state":["exists","up"]},{"osd":5,"uuid":"022d67db-d4d8-445b-b1b9-8b446a00f5c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":37352},{"type":"v1","addr":"172.21.15.62:6801","nonce":37352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":37352},{"type":"v1","addr":"172.21.15.62:6803","nonce":37352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":37352},{"type":"v1","addr":"172.21.15.62:6807","nonce":37352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":37352},{"type":"v1","addr":"172.21.15.62:6805","nonce":37352}]},"public_addr":"172.21.15.62:6801/37352","cluster_addr":"172.21.15.62:6803/37352","heartbeat_back_addr":"172.21.15.62:6807/37352","heartbeat_front_addr":"172.21.15.62:6805/37352","state":["exists","up"]},{"osd":6,"uuid":"42bd372b-6688-4a1b-8407-e16e8c266e33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":37354},{"type":"v1","addr":"172.21.15.62:6820","nonce":37354}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":37354},{"type":"v1","addr":"172.21.15.62:6823","nonce":37354}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6829","nonce":37354},{"type":"v1","addr":"172.21.15.62:6830","nonce":37354}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6825","nonce":37354},{"type":"v1","addr":"172.21.15.62:6827","nonce":37354}]},"public_addr":"172.21.15.62:6820/37354","cluster_addr":"172.21.15.62:6823/37354","heartbeat_back_addr":"172.21.15.62:6830/37354","heartbeat_front_addr":"172.21.15.62:6827/37354","state":["exists","up"]},{"osd":7,"uuid":"ac70b2a3-f41b-4bf2-b227-7f2cbb55426d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37355},{"type":"v1","addr":"172.21.15.62:6809","nonce":37355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37355},{"type":"v1","addr":"172.21.15.62:6811","nonce":37355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":37355},{"type":"v1","addr":"172.21.15.62:6815","nonce":37355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":37355},{"type":"v1","addr":"172.21.15.62:6813","nonce":37355}]},"public_addr":"172.21.15.62:6809/37355","cluster_addr":"172.21.15.62:6811/37355","heartbeat_back_addr":"172.21.15.62:6815/37355","heartbeat_front_addr":"172.21.15.62:6813/37355","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.368896+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.107273+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.276888+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.287502+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.124791+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.251937+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.255679+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.232797+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"7.4","mappings":[{"from":0,"to":5}]},{"pgid":"7.2e","mappings":[{"from":0,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:0/2741846205":"2023-05-23T15:39:47.055770+0000","172.21.15.16:0/651011632":"2023-05-23T15:39:47.055770+0000","172.21.15.16:0/2307989846":"2023-05-23T15:39:47.055770+0000","172.21.15.16:0/2901257227":"2023-05-23T15:39:47.055770+0000","172.21.15.16:0/3979017463":"2023-05-23T15:28:30.504372+0000","172.21.15.16:0/901306510":"2023-05-23T15:28:30.504372+0000","172.21.15.16:0/1579456307":"2023-05-23T15:39:47.055770+0000","172.21.15.16:0/2755009771":"2023-05-23T15:28:30.504372+0000","172.21.15.16:0/4222124422":"2023-05-23T15:28:30.504372+0000","172.21.15.16:0/593205291":"2023-05-23T15:39:47.055770+0000","172.21.15.16:6833/36921":"2023-05-23T15:28:30.504372+0000","172.21.15.16:0/4184273947":"2023-05-23T15:39:47.055770+0000","172.21.15.16:6832/36921":"2023-05-23T15:28:30.504372+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":7,"snaps":[{"begin":6,"length":1},{"begin":10,"length":1},{"begin":14,"length":1},{"begin":19,"length":1},{"begin":24,"length":1},{"begin":29,"length":1},{"begin":35,"length":1},{"begin":39,"length":2},{"begin":45,"length":1},{"begin":47,"length":1},{"begin":50,"length":1},{"begin":53,"length":1},{"begin":58,"length":1},{"begin":61,"length":1},{"begin":65,"length":1},{"begin":67,"length":1},{"begin":70,"length":1},{"begin":74,"length":1},{"begin":77,"length":1},{"begin":81,"length":1},{"begin":84,"length":1},{"begin":89,"length":2},{"begin":98,"length":3},{"begin":106,"length":3},{"begin":192,"length":30}]}],"new_removed_snaps":[{"pool":7,"snaps":[{"begin":107,"length":1},{"begin":221,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-05-22T15:39:48.439 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.434+0000 7f756936a640 1 -- 172.21.15.16:0/1532719528 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f75500651d0 msgr2=0x7f7550067680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:48.439 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.434+0000 7f756936a640 1 --2- 172.21.15.16:0/1532719528 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f75500651d0 0x7f7550067680 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f75540070b0 tx=0x7f7554007040 comp rx=0 tx=0).stop 2023-05-22T15:39:48.439 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.434+0000 7f756936a640 1 -- 172.21.15.16:0/1532719528 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f75641567c0 msgr2=0x7f75641464f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:48.440 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.434+0000 7f756936a640 1 --2- 172.21.15.16:0/1532719528 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f75641567c0 0x7f75641464f0 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f755800c500 tx=0x7f755800c9c0 comp rx=0 tx=0).stop 2023-05-22T15:39:48.440 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.434+0000 7f756936a640 1 -- 172.21.15.16:0/1532719528 shutdown_connections 2023-05-22T15:39:48.440 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.434+0000 7f756936a640 1 --2- 172.21.15.16:0/1532719528 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7564159a50 0x7f7564146a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:48.440 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.434+0000 7f756936a640 1 --2- 172.21.15.16:0/1532719528 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f75641567c0 0x7f75641464f0 unknown :-1 s=CLOSED pgs=835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:48.440 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.434+0000 7f756936a640 1 --2- 172.21.15.16:0/1532719528 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f75500651d0 0x7f7550067680 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:48.440 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.434+0000 7f756936a640 1 --2- 172.21.15.16:0/1532719528 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f75640cb6a0 0x7f756415c940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:48.440 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.434+0000 7f756936a640 1 -- 172.21.15.16:0/1532719528 >> 172.21.15.16:0/1532719528 conn(0x7f7564069720 msgr2=0x7f75640cd050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:39:48.441 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.434+0000 7f756936a640 1 -- 172.21.15.16:0/1532719528 shutdown_connections 2023-05-22T15:39:48.441 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.434+0000 7f756936a640 1 -- 172.21.15.16:0/1532719528 wait complete. 2023-05-22T15:39:48.455 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-05-22T15:39:48.457 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-05-22T15:39:48.568 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.562+0000 7f5081194640 1 Processor -- start 2023-05-22T15:39:48.568 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.562+0000 7f5081194640 1 -- start start 2023-05-22T15:39:48.571 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.562+0000 7f5081194640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f507c0b7dd0 0x7f507c0ccab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:48.572 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.562+0000 7f5081194640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f507c160220 0x7f507c0d0410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:48.572 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.562+0000 7f5081194640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f507c0cd170 con 0x7f507c15f700 2023-05-22T15:39:48.573 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.562+0000 7f5081194640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f507c0d3360 con 0x7f507c160220 2023-05-22T15:39:48.573 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.562+0000 7f5081194640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f507c0d2ac0 con 0x7f507c0b7dd0 2023-05-22T15:39:48.573 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.562+0000 7f507a59c640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f507c15f700 0x7f507c15fb10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:56578/0 (socket says 172.21.15.16:56578) 2023-05-22T15:39:48.573 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.562+0000 7f507a59c640 1 -- 172.21.15.16:0/2759175114 learned_addr learned my addr 172.21.15.16:0/2759175114 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:39:48.574 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.562+0000 7f5079d9b640 1 -- 172.21.15.16:0/2759175114 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2131634230 0 0) 0x7f507c0cd170 con 0x7f507c15f700 2023-05-22T15:39:48.574 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.562+0000 7f507ad9d640 1 --2- 172.21.15.16:0/2759175114 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f507c0b7dd0 0x7f507c0ccab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:48.574 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.562+0000 7f507b59e640 1 --2- 172.21.15.16:0/2759175114 >> v2:172.21.15.62:3300/0 conn(0x7f507c160220 0x7f507c0d0410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:48.574 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.562+0000 7f5079d9b640 1 -- 172.21.15.16:0/2759175114 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5064003530 con 0x7f507c15f700 2023-05-22T15:39:48.574 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.566+0000 7f5079d9b640 1 -- 172.21.15.16:0/2759175114 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3061731392 0 0) 0x7f5064003530 con 0x7f507c15f700 2023-05-22T15:39:48.575 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.566+0000 7f5079d9b640 1 -- 172.21.15.16:0/2759175114 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f507c0b7dd0 msgr2=0x7f507c0ccab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:39:48.575 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.566+0000 7f5079d9b640 1 --2- 172.21.15.16:0/2759175114 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f507c0b7dd0 0x7f507c0ccab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:48.575 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.566+0000 7f5079d9b640 1 -- 172.21.15.16:0/2759175114 >> v2:172.21.15.62:3300/0 conn(0x7f507c160220 msgr2=0x7f507c0d0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:39:48.575 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.566+0000 7f5079d9b640 1 --2- 172.21.15.16:0/2759175114 >> v2:172.21.15.62:3300/0 conn(0x7f507c160220 0x7f507c0d0410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:48.575 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.566+0000 7f5079d9b640 1 -- 172.21.15.16:0/2759175114 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f507c0d2da0 con 0x7f507c15f700 2023-05-22T15:39:48.575 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.566+0000 7f5079d9b640 1 -- 172.21.15.16:0/2759175114 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7f5070001680 con 0x7f507c15f700 2023-05-22T15:39:48.575 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.566+0000 7f5079d9b640 1 -- 172.21.15.16:0/2759175114 >> v1:172.21.15.16:6789/0 conn(0x7f507c15f700 legacy=0x7f507c15fb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:48.576 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.566+0000 7f5079d9b640 1 --2- 172.21.15.16:0/2759175114 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f507c0b7dd0 0x7f5064003980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:48.576 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.566+0000 7f5079d9b640 1 --2- 172.21.15.16:0/2759175114 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5064003f00 0x7f50640047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:48.576 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.566+0000 7f5079d9b640 1 --2- 172.21.15.16:0/2759175114 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5064004d10 0x7f5064005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:48.576 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.566+0000 7f5079d9b640 1 -- 172.21.15.16:0/2759175114 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f507c0d3360 con 0x7f5064003f00 2023-05-22T15:39:48.576 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.566+0000 7f5079d9b640 1 -- 172.21.15.16:0/2759175114 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f507c0d2ac0 con 0x7f5064004d10 2023-05-22T15:39:48.576 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.566+0000 7f5079d9b640 1 -- 172.21.15.16:0/2759175114 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f506400b9c0 con 0x7f507c0b7dd0 2023-05-22T15:39:48.576 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.566+0000 7f507a59c640 1 --2- 172.21.15.16:0/2759175114 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5064003f00 0x7f50640047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:48.577 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.566+0000 7f507ad9d640 1 --2- 172.21.15.16:0/2759175114 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f507c0b7dd0 0x7f5064003980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:48.577 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.566+0000 7f507b59e640 1 --2- 172.21.15.16:0/2759175114 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5064004d10 0x7f5064005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:48.577 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.566+0000 7f507a59c640 1 -- 172.21.15.16:0/2759175114 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f507c0b7dd0 msgr2=0x7f5064003980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:39:48.577 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.566+0000 7f507a59c640 1 --2- 172.21.15.16:0/2759175114 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f507c0b7dd0 0x7f5064003980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:48.577 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.566+0000 7f507a59c640 1 -- 172.21.15.16:0/2759175114 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5064004d10 msgr2=0x7f5064005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:39:48.578 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.566+0000 7f507a59c640 1 --2- 172.21.15.16:0/2759175114 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5064004d10 0x7f5064005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:48.578 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.566+0000 7f507a59c640 1 -- 172.21.15.16:0/2759175114 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5064003530 con 0x7f5064003f00 2023-05-22T15:39:48.581 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.566+0000 7f507a59c640 1 --2- 172.21.15.16:0/2759175114 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5064003f00 0x7f50640047d0 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f507000c290 tx=0x7f507001cc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c4a42662d730a65 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:39:48.582 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.566+0000 7f5079d9b640 1 -- 172.21.15.16:0/2759175114 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50700a5920 con 0x7f5064003f00 2023-05-22T15:39:48.582 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.566+0000 7f5079d9b640 1 -- 172.21.15.16:0/2759175114 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50700a5a80 con 0x7f5064003f00 2023-05-22T15:39:48.582 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.566+0000 7f5079d9b640 1 -- 172.21.15.16:0/2759175114 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50700a5e30 con 0x7f5064003f00 2023-05-22T15:39:48.582 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.566+0000 7f5081194640 1 -- 172.21.15.16:0/2759175114 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5064003f00 msgr2=0x7f50640047d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:48.583 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.566+0000 7f5081194640 1 --2- 172.21.15.16:0/2759175114 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5064003f00 0x7f50640047d0 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f507000c290 tx=0x7f507001cc50 comp rx=0 tx=0).stop 2023-05-22T15:39:48.583 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.566+0000 7f507b59e640 1 -- 172.21.15.16:0/2759175114 reap_dead start 2023-05-22T15:39:48.583 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.566+0000 7f5081194640 1 -- 172.21.15.16:0/2759175114 shutdown_connections 2023-05-22T15:39:48.583 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.566+0000 7f5081194640 1 -- 172.21.15.16:0/2759175114 >> 172.21.15.16:0/2759175114 conn(0x7f507c0655c0 msgr2=0x7f507c070e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:39:48.584 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.566+0000 7f5081194640 1 -- 172.21.15.16:0/2759175114 shutdown_connections 2023-05-22T15:39:48.584 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.566+0000 7f5081194640 1 -- 172.21.15.16:0/2759175114 wait complete. 2023-05-22T15:39:48.584 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.570+0000 7f5081194640 1 Processor -- start 2023-05-22T15:39:48.584 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.570+0000 7f5081194640 1 -- start start 2023-05-22T15:39:48.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.570+0000 7f5081194640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5064003f00 0x7f507c158ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:48.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.570+0000 7f5081194640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f507c1f5250 0x7f507c06a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:48.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.570+0000 7f5081194640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f507c06ac20 0x7f507c1f4090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:48.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.570+0000 7f5081194640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f507c0cc800 con 0x7f5064003f00 2023-05-22T15:39:48.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.570+0000 7f5081194640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f507c1f50a0 con 0x7f507c1f5250 2023-05-22T15:39:48.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.570+0000 7f5081194640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f507c06b070 con 0x7f507c06ac20 2023-05-22T15:39:48.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.570+0000 7f507ad9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5064003f00 0x7f507c158ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:48.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.570+0000 7f507ad9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5064003f00 0x7f507c158ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53018/0 (socket says 172.21.15.16:53018) 2023-05-22T15:39:48.587 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.570+0000 7f507ad9d640 1 -- 172.21.15.16:0/2054395827 learned_addr learned my addr 172.21.15.16:0/2054395827 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:39:48.587 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.570+0000 7f507b59e640 1 --2- 172.21.15.16:0/2054395827 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f507c06ac20 0x7f507c1f4090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:48.587 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.570+0000 7f507a59c640 1 --2- 172.21.15.16:0/2054395827 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f507c1f5250 0x7f507c06a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:48.587 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.570+0000 7f507ad9d640 1 -- 172.21.15.16:0/2054395827 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f507c06ac20 msgr2=0x7f507c1f4090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:48.588 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.570+0000 7f507ad9d640 1 --2- 172.21.15.16:0/2054395827 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f507c06ac20 0x7f507c1f4090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:48.588 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.570+0000 7f507ad9d640 1 -- 172.21.15.16:0/2054395827 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f507c1f5250 msgr2=0x7f507c06a6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:48.588 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.570+0000 7f507ad9d640 1 --2- 172.21.15.16:0/2054395827 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f507c1f5250 0x7f507c06a6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:48.588 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.570+0000 7f507ad9d640 1 -- 172.21.15.16:0/2054395827 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f507c0d1160 con 0x7f5064003f00 2023-05-22T15:39:48.589 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.570+0000 7f507ad9d640 1 --2- 172.21.15.16:0/2054395827 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5064003f00 0x7f507c158ae0 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7f506c00f380 tx=0x7f506c00f840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:39:48.589 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.570+0000 7f50637fe640 1 -- 172.21.15.16:0/2054395827 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f506c017600 con 0x7f5064003f00 2023-05-22T15:39:48.589 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.570+0000 7f50637fe640 1 -- 172.21.15.16:0/2054395827 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f506c011040 con 0x7f5064003f00 2023-05-22T15:39:48.589 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.570+0000 7f5081194640 1 -- 172.21.15.16:0/2054395827 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f507c0cf730 con 0x7f5064003f00 2023-05-22T15:39:48.590 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.570+0000 7f50637fe640 1 -- 172.21.15.16:0/2054395827 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f506c0179c0 con 0x7f5064003f00 2023-05-22T15:39:48.590 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.570+0000 7f5081194640 1 -- 172.21.15.16:0/2054395827 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f507c0d12f0 con 0x7f5064003f00 2023-05-22T15:39:48.590 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.570+0000 7f5081194640 1 -- 172.21.15.16:0/2054395827 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f507c0ca920 con 0x7f5064003f00 2023-05-22T15:39:48.590 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.570+0000 7f50637fe640 1 -- 172.21.15.16:0/2054395827 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 15) v1 ==== 84068+0+0 (secure 0 0 0) 0x7f506c017b20 con 0x7f5064003f00 2023-05-22T15:39:48.591 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.574+0000 7f50637fe640 1 --2- 172.21.15.16:0/2054395827 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f5068065200 0x7f50680676b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:48.591 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.574+0000 7f50637fe640 1 -- 172.21.15.16:0/2054395827 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 7455+0+0 (secure 0 0 0) 0x7f506c083840 con 0x7f5064003f00 2023-05-22T15:39:48.591 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.574+0000 7f507a59c640 1 --2- 172.21.15.16:0/2054395827 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f5068065200 0x7f50680676b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:48.591 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.574+0000 7f507a59c640 1 --2- 172.21.15.16:0/2054395827 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f5068065200 0x7f50680676b0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f50700aa000 tx=0x7f5070001a10 comp rx=0 tx=0).ready entity=mgr.6240 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:39:48.592 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.574+0000 7f50637fe640 1 -- 172.21.15.16:0/2054395827 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f506c051390 con 0x7f5064003f00 2023-05-22T15:39:48.768 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.762+0000 7f5081194640 1 -- 172.21.15.16:0/2054395827 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f507c04dbe0 con 0x7f5064003f00 2023-05-22T15:39:48.769 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.762+0000 7f50637fe640 1 -- 172.21.15.16:0/2054395827 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) v1 ==== 94+0+4648 (secure 0 0 0) 0x7f506c01c090 con 0x7f5064003f00 2023-05-22T15:39:48.769 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:39:48.769 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":14,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4904,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6833/3153072235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":3153072235},{"type":"v1","addr":"172.21.15.62:6833","nonce":3153072235}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":5164,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/2783822861","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":2783822861},{"type":"v1","addr":"172.21.15.62:6835","nonce":2783822861}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":5571,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/1361634240","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1361634240},{"type":"v1","addr":"172.21.15.16:6835","nonce":1361634240}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":14,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-05-22T15:28:13.320259+0000","modified":"2023-05-22T15:28:15.336631+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5574},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5574":{"gid":5574,"name":"c","rank":0,"incarnation":12,"state":"up:active","state_seq":3,"addr":"172.21.15.16:6837/1033866899","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":1033866899},{"type":"v1","addr":"172.21.15.16:6837","nonce":1033866899}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-05-22T15:39:48.770 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 14 2023-05-22T15:39:48.771 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.766+0000 7f5081194640 1 -- 172.21.15.16:0/2054395827 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f5068065200 msgr2=0x7f50680676b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:48.771 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.766+0000 7f5081194640 1 --2- 172.21.15.16:0/2054395827 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f5068065200 0x7f50680676b0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f50700aa000 tx=0x7f5070001a10 comp rx=0 tx=0).stop 2023-05-22T15:39:48.771 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.766+0000 7f5081194640 1 -- 172.21.15.16:0/2054395827 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5064003f00 msgr2=0x7f507c158ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:48.771 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.766+0000 7f5081194640 1 --2- 172.21.15.16:0/2054395827 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5064003f00 0x7f507c158ae0 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7f506c00f380 tx=0x7f506c00f840 comp rx=0 tx=0).stop 2023-05-22T15:39:48.771 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.766+0000 7f5081194640 1 -- 172.21.15.16:0/2054395827 shutdown_connections 2023-05-22T15:39:48.771 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.766+0000 7f5081194640 1 --2- 172.21.15.16:0/2054395827 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f5068065200 0x7f50680676b0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:48.772 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.766+0000 7f5081194640 1 --2- 172.21.15.16:0/2054395827 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f507c06ac20 0x7f507c1f4090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:48.772 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.766+0000 7f5081194640 1 --2- 172.21.15.16:0/2054395827 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f507c1f5250 0x7f507c06a6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:48.772 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.766+0000 7f5081194640 1 --2- 172.21.15.16:0/2054395827 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5064003f00 0x7f507c158ae0 unknown :-1 s=CLOSED pgs=838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:48.772 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.766+0000 7f5081194640 1 -- 172.21.15.16:0/2054395827 >> 172.21.15.16:0/2054395827 conn(0x7f507c0655c0 msgr2=0x7f507c0d2220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:39:48.772 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.766+0000 7f5081194640 1 -- 172.21.15.16:0/2054395827 shutdown_connections 2023-05-22T15:39:48.772 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.766+0000 7f5081194640 1 -- 172.21.15.16:0/2054395827 wait complete. 2023-05-22T15:39:48.787 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-05-22T15:39:48.898 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.894+0000 7f54b3d32640 1 Processor -- start 2023-05-22T15:39:48.899 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.894+0000 7f54b3d32640 1 -- start start 2023-05-22T15:39:48.901 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.894+0000 7f54b3d32640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f54ac159b10 0x7f54ac0cc780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:48.901 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.894+0000 7f54b3d32640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54ac159ee0 con 0x7f54ac06e250 2023-05-22T15:39:48.901 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.894+0000 7f54b3d32640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54ac0b30b0 con 0x7f54ac0b5e80 2023-05-22T15:39:48.901 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.894+0000 7f54b3d32640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54ac0b2810 con 0x7f54ac159b10 2023-05-22T15:39:48.902 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.894+0000 7f54b1ace640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f54ac159b10 0x7f54ac0cc780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:48.902 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.894+0000 7f54b1ace640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f54ac159b10 0x7f54ac0cc780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53022/0 (socket says 172.21.15.16:53022) 2023-05-22T15:39:48.902 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.894+0000 7f54b1ace640 1 -- 172.21.15.16:0/1738246832 learned_addr learned my addr 172.21.15.16:0/1738246832 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:39:48.903 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.894+0000 7f54b22cf640 1 --1- 172.21.15.16:0/1738246832 >> v1:172.21.15.16:6789/0 conn(0x7f54ac06e250 0x7f54ac0b3e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:56582/0 (socket says 172.21.15.16:56582) 2023-05-22T15:39:48.903 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.894+0000 7f54b1ace640 1 -- 172.21.15.16:0/1738246832 >> v1:172.21.15.62:6789/0 conn(0x7f54ac0b5e80 legacy=0x7f54ac06db40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:39:48.903 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.894+0000 7f54b1ace640 1 -- 172.21.15.16:0/1738246832 >> v1:172.21.15.16:6789/0 conn(0x7f54ac06e250 legacy=0x7f54ac0b3e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:48.904 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.894+0000 7f54b1ace640 1 -- 172.21.15.16:0/1738246832 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54ac0b2af0 con 0x7f54ac159b10 2023-05-22T15:39:48.908 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.898+0000 7f54b1ace640 1 --2- 172.21.15.16:0/1738246832 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f54ac159b10 0x7f54ac0cc780 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f54a8009870 tx=0x7f54a801c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bcf8dad5f7ed92e3 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:39:48.908 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.898+0000 7f54b0acc640 1 -- 172.21.15.16:0/1738246832 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54a80aa430 con 0x7f54ac159b10 2023-05-22T15:39:48.908 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.898+0000 7f54b0acc640 1 -- 172.21.15.16:0/1738246832 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54a80aa590 con 0x7f54ac159b10 2023-05-22T15:39:48.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.898+0000 7f54b0acc640 1 -- 172.21.15.16:0/1738246832 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54a80aa920 con 0x7f54ac159b10 2023-05-22T15:39:48.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.898+0000 7f54b3d32640 1 -- 172.21.15.16:0/1738246832 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f54ac159b10 msgr2=0x7f54ac0cc780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:48.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.898+0000 7f54b3d32640 1 --2- 172.21.15.16:0/1738246832 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f54ac159b10 0x7f54ac0cc780 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f54a8009870 tx=0x7f54a801c4d0 comp rx=0 tx=0).stop 2023-05-22T15:39:48.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.898+0000 7f54b3d32640 1 -- 172.21.15.16:0/1738246832 shutdown_connections 2023-05-22T15:39:48.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.898+0000 7f54b3d32640 1 --2- 172.21.15.16:0/1738246832 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f54ac159b10 0x7f54ac0cc780 unknown :-1 s=CLOSED pgs=840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:48.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.898+0000 7f54b3d32640 1 -- 172.21.15.16:0/1738246832 >> 172.21.15.16:0/1738246832 conn(0x7f54ac06f560 msgr2=0x7f54ac06f960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:39:48.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.898+0000 7f54b3d32640 1 -- 172.21.15.16:0/1738246832 shutdown_connections 2023-05-22T15:39:48.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.898+0000 7f54b3d32640 1 -- 172.21.15.16:0/1738246832 wait complete. 2023-05-22T15:39:48.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.898+0000 7f54b3d32640 1 Processor -- start 2023-05-22T15:39:48.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.898+0000 7f54b3d32640 1 -- start start 2023-05-22T15:39:48.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.898+0000 7f54b3d32640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f54ac06e250 0x7f54ac147470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:48.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.898+0000 7f54b3d32640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f54ac0b5e80 0x7f54ac14da30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:48.911 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.898+0000 7f54b3d32640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f54ac159b10 0x7f54ac0cf740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:48.911 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.898+0000 7f54b3d32640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54ac14e280 con 0x7f54ac159b10 2023-05-22T15:39:48.911 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.898+0000 7f54b3d32640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54ac0ce9c0 con 0x7f54ac06e250 2023-05-22T15:39:48.911 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.898+0000 7f54b3d32640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54ac0cec60 con 0x7f54ac0b5e80 2023-05-22T15:39:48.911 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.898+0000 7f54b22cf640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f54ac159b10 0x7f54ac0cf740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:48.911 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.898+0000 7f54b22cf640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f54ac159b10 0x7f54ac0cf740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53024/0 (socket says 172.21.15.16:53024) 2023-05-22T15:39:48.911 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.898+0000 7f54b22cf640 1 -- 172.21.15.16:0/2702862734 learned_addr learned my addr 172.21.15.16:0/2702862734 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:39:48.912 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.902+0000 7f54b1ace640 1 --2- 172.21.15.16:0/2702862734 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f54ac06e250 0x7f54ac147470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:48.912 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.902+0000 7f54b12cd640 1 --2- 172.21.15.16:0/2702862734 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f54ac0b5e80 0x7f54ac14da30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:48.912 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.902+0000 7f54b22cf640 1 -- 172.21.15.16:0/2702862734 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f54ac0b5e80 msgr2=0x7f54ac14da30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:48.912 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.902+0000 7f54b22cf640 1 --2- 172.21.15.16:0/2702862734 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f54ac0b5e80 0x7f54ac14da30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:48.912 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.902+0000 7f54b22cf640 1 -- 172.21.15.16:0/2702862734 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f54ac06e250 msgr2=0x7f54ac147470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:48.912 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.902+0000 7f54b22cf640 1 --2- 172.21.15.16:0/2702862734 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f54ac06e250 0x7f54ac147470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:48.912 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.902+0000 7f54b22cf640 1 -- 172.21.15.16:0/2702862734 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54a80094d0 con 0x7f54ac159b10 2023-05-22T15:39:48.913 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.902+0000 7f54b22cf640 1 --2- 172.21.15.16:0/2702862734 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f54ac159b10 0x7f54ac0cf740 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f54a400c6c0 tx=0x7f54a400cb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:39:48.913 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.902+0000 7f54a27fc640 1 -- 172.21.15.16:0/2702862734 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54a4004d70 con 0x7f54ac159b10 2023-05-22T15:39:48.913 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.902+0000 7f54a27fc640 1 -- 172.21.15.16:0/2702862734 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54a400a640 con 0x7f54ac159b10 2023-05-22T15:39:48.913 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.902+0000 7f54b3d32640 1 -- 172.21.15.16:0/2702862734 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54ac0b4d50 con 0x7f54ac159b10 2023-05-22T15:39:48.913 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.902+0000 7f54a27fc640 1 -- 172.21.15.16:0/2702862734 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54a400a9b0 con 0x7f54ac159b10 2023-05-22T15:39:48.913 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.902+0000 7f54b3d32640 1 -- 172.21.15.16:0/2702862734 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54ac0b3910 con 0x7f54ac159b10 2023-05-22T15:39:48.913 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.902+0000 7f54b3d32640 1 -- 172.21.15.16:0/2702862734 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54ac0b4ee0 con 0x7f54ac159b10 2023-05-22T15:39:48.914 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.902+0000 7f54a27fc640 1 -- 172.21.15.16:0/2702862734 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 15) v1 ==== 84068+0+0 (secure 0 0 0) 0x7f54a400ab10 con 0x7f54ac159b10 2023-05-22T15:39:48.914 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.902+0000 7f54a27fc640 1 --2- 172.21.15.16:0/2702862734 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f54900651d0 0x7f5490067680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:48.914 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.902+0000 7f54a27fc640 1 -- 172.21.15.16:0/2702862734 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 7455+0+0 (secure 0 0 0) 0x7f54a40821e0 con 0x7f54ac159b10 2023-05-22T15:39:48.915 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.902+0000 7f54b1ace640 1 --2- 172.21.15.16:0/2702862734 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f54900651d0 0x7f5490067680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:48.915 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.902+0000 7f54b1ace640 1 --2- 172.21.15.16:0/2702862734 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f54900651d0 0x7f5490067680 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f54a80ae000 tx=0x7f54a8008040 comp rx=0 tx=0).ready entity=mgr.6240 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:39:48.915 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:48.906+0000 7f54a27fc640 1 -- 172.21.15.16:0/2702862734 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f54a404fc30 con 0x7f54ac159b10 2023-05-22T15:39:49.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:49.086+0000 7f54b3d32640 1 -- 172.21.15.16:0/2702862734 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f54ac04dbe0 con 0x7f54ac159b10 2023-05-22T15:39:49.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:49.090+0000 7f54a27fc640 1 -- 172.21.15.16:0/2702862734 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) v1 ==== 94+0+4648 (secure 0 0 0) 0x7f54a4053250 con 0x7f54ac159b10 2023-05-22T15:39:49.095 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:39:49.095 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":14,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4904,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6833/3153072235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":3153072235},{"type":"v1","addr":"172.21.15.62:6833","nonce":3153072235}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":5164,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/2783822861","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":2783822861},{"type":"v1","addr":"172.21.15.62:6835","nonce":2783822861}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":5571,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/1361634240","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1361634240},{"type":"v1","addr":"172.21.15.16:6835","nonce":1361634240}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":14,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-05-22T15:28:13.320259+0000","modified":"2023-05-22T15:28:15.336631+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5574},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5574":{"gid":5574,"name":"c","rank":0,"incarnation":12,"state":"up:active","state_seq":3,"addr":"172.21.15.16:6837/1033866899","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":1033866899},{"type":"v1","addr":"172.21.15.16:6837","nonce":1033866899}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-05-22T15:39:49.095 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 14 2023-05-22T15:39:49.096 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:49.090+0000 7f54b3d32640 1 -- 172.21.15.16:0/2702862734 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f54900651d0 msgr2=0x7f5490067680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:49.097 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:49.090+0000 7f54b3d32640 1 --2- 172.21.15.16:0/2702862734 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f54900651d0 0x7f5490067680 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f54a80ae000 tx=0x7f54a8008040 comp rx=0 tx=0).stop 2023-05-22T15:39:49.097 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:49.090+0000 7f54b3d32640 1 -- 172.21.15.16:0/2702862734 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f54ac159b10 msgr2=0x7f54ac0cf740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:49.097 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:49.090+0000 7f54b3d32640 1 --2- 172.21.15.16:0/2702862734 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f54ac159b10 0x7f54ac0cf740 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f54a400c6c0 tx=0x7f54a400cb80 comp rx=0 tx=0).stop 2023-05-22T15:39:49.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:49.090+0000 7f54b3d32640 1 -- 172.21.15.16:0/2702862734 shutdown_connections 2023-05-22T15:39:49.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:49.090+0000 7f54b3d32640 1 --2- 172.21.15.16:0/2702862734 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f54ac159b10 0x7f54ac0cf740 unknown :-1 s=CLOSED pgs=841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:49.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:49.090+0000 7f54b3d32640 1 --2- 172.21.15.16:0/2702862734 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f54900651d0 0x7f5490067680 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:49.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:49.090+0000 7f54b3d32640 1 --2- 172.21.15.16:0/2702862734 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f54ac0b5e80 0x7f54ac14da30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:49.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:49.090+0000 7f54b3d32640 1 --2- 172.21.15.16:0/2702862734 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f54ac06e250 0x7f54ac147470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:49.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:49.090+0000 7f54b3d32640 1 -- 172.21.15.16:0/2702862734 >> 172.21.15.16:0/2702862734 conn(0x7f54ac06f560 msgr2=0x7f54ac0c4580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:39:49.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:49.090+0000 7f54b3d32640 1 -- 172.21.15.16:0/2702862734 shutdown_connections 2023-05-22T15:39:49.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:49.090+0000 7f54b3d32640 1 -- 172.21.15.16:0/2702862734 wait complete. 2023-05-22T15:39:49.112 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-05-22T15:39:49.226 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:49.218+0000 7f6d4302f640 1 Processor -- start 2023-05-22T15:39:49.226 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:49.218+0000 7f6d4302f640 1 -- start start 2023-05-22T15:39:49.227 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:49.222+0000 7f6d4302f640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6d3c0ca4e0 0x7f6d3c15e460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:49.228 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:49.222+0000 7f6d4302f640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f6d3c0c9b40 0x7f6d3c0afb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:49.228 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:49.222+0000 7f6d4302f640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d3c0b0650 con 0x7f6d3c070df0 2023-05-22T15:39:49.228 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:49.222+0000 7f6d4302f640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d3c0b0370 con 0x7f6d3c0ca4e0 2023-05-22T15:39:49.229 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:49.222+0000 7f6d4302f640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d3c15d980 con 0x7f6d3c0c9b40 2023-05-22T15:39:49.229 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:49.222+0000 7f6d40dcb640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6d3c0ca4e0 0x7f6d3c15e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:49.229 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:49.222+0000 7f6d40dcb640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6d3c0ca4e0 0x7f6d3c15e460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57360/0 (socket says 172.21.15.16:57360) 2023-05-22T15:39:49.229 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:49.222+0000 7f6d40dcb640 1 -- 172.21.15.16:0/3060798131 learned_addr learned my addr 172.21.15.16:0/3060798131 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:39:49.229 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:49.222+0000 7f6d3bfff640 1 --2- 172.21.15.16:0/3060798131 >> v2:172.21.15.62:3300/0 conn(0x7f6d3c0c9b40 0x7f6d3c0afb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:49.230 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:49.222+0000 7f6d40dcb640 1 -- 172.21.15.16:0/3060798131 >> v2:172.21.15.62:3300/0 conn(0x7f6d3c0c9b40 msgr2=0x7f6d3c0afb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:39:49.230 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:49.222+0000 7f6d40dcb640 1 --2- 172.21.15.16:0/3060798131 >> v2:172.21.15.62:3300/0 conn(0x7f6d3c0c9b40 0x7f6d3c0afb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:49.230 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:49.222+0000 7f6d40dcb640 1 -- 172.21.15.16:0/3060798131 >> v1:172.21.15.16:6789/0 conn(0x7f6d3c070df0 legacy=0x7f6d3c071210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:49.230 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:49.222+0000 7f6d40dcb640 1 -- 172.21.15.16:0/3060798131 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d3c15dc60 con 0x7f6d3c0ca4e0 2023-05-22T15:39:49.230 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:49.222+0000 7f6d3bfff640 1 --2- 172.21.15.16:0/3060798131 >> v2:172.21.15.62:3300/0 conn(0x7f6d3c0c9b40 0x7f6d3c0afb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:39:49.233 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:49.222+0000 7f6d40dcb640 1 --2- 172.21.15.16:0/3060798131 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6d3c0ca4e0 0x7f6d3c15e460 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f6d3c06f0e0 tx=0x7f6d2401c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=daa2a578edb430bb server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:39:49.233 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:49.222+0000 7f6d3b7fe640 1 -- 172.21.15.16:0/3060798131 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d240aa610 con 0x7f6d3c0ca4e0 2023-05-22T15:39:49.233 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:49.222+0000 7f6d3b7fe640 1 -- 172.21.15.16:0/3060798131 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d240a6040 con 0x7f6d3c0ca4e0 2023-05-22T15:39:49.233 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:49.222+0000 7f6d3b7fe640 1 -- 172.21.15.16:0/3060798131 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d240aa990 con 0x7f6d3c0ca4e0 2023-05-22T15:39:49.233 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:49.222+0000 7f6d4302f640 1 -- 172.21.15.16:0/3060798131 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6d3c0ca4e0 msgr2=0x7f6d3c15e460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:49.233 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:49.222+0000 7f6d4302f640 1 --2- 172.21.15.16:0/3060798131 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6d3c0ca4e0 0x7f6d3c15e460 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f6d3c06f0e0 tx=0x7f6d2401c4d0 comp rx=0 tx=0).stop 2023-05-22T15:39:49.234 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:49.222+0000 7f6d4302f640 1 -- 172.21.15.16:0/3060798131 shutdown_connections 2023-05-22T15:39:49.234 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:49.222+0000 7f6d4302f640 1 --2- 172.21.15.16:0/3060798131 >> v2:172.21.15.62:3300/0 conn(0x7f6d3c0c9b40 0x7f6d3c0afb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:49.234 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:49.222+0000 7f6d4302f640 1 --2- 172.21.15.16:0/3060798131 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6d3c0ca4e0 0x7f6d3c15e460 unknown :-1 s=CLOSED pgs=843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:49.234 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:49.222+0000 7f6d4302f640 1 -- 172.21.15.16:0/3060798131 >> 172.21.15.16:0/3060798131 conn(0x7f6d3c069720 msgr2=0x7f6d3c069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:39:49.234 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:49.222+0000 7f6d4302f640 1 -- 172.21.15.16:0/3060798131 shutdown_connections 2023-05-22T15:39:49.234 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:49.222+0000 7f6d4302f640 1 -- 172.21.15.16:0/3060798131 wait complete. 2023-05-22T15:39:49.234 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:49.226+0000 7f6d4302f640 1 Processor -- start 2023-05-22T15:39:49.234 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:49.226+0000 7f6d4302f640 1 -- start start 2023-05-22T15:39:49.235 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:49.226+0000 7f6d4302f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6d3c070df0 0x7f6d3c14a230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:49.235 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:49.226+0000 7f6d4302f640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6d3c0c9b40 0x7f6d3c1571e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:49.235 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:49.226+0000 7f6d4302f640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6d3c0ca4e0 0x7f6d3c14b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:49.235 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:49.226+0000 7f6d4302f640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d3c14c320 con 0x7f6d3c070df0 2023-05-22T15:39:49.235 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:49.226+0000 7f6d4302f640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d3c0ae7b0 con 0x7f6d3c0c9b40 2023-05-22T15:39:49.235 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:49.226+0000 7f6d4302f640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d3c0adba0 con 0x7f6d3c0ca4e0 2023-05-22T15:39:49.235 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:49.226+0000 7f6d40dcb640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6d3c070df0 0x7f6d3c14a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:49.236 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:49.226+0000 7f6d40dcb640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6d3c070df0 0x7f6d3c14a230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57374/0 (socket says 172.21.15.16:57374) 2023-05-22T15:39:49.236 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:49.226+0000 7f6d40dcb640 1 -- 172.21.15.16:0/4135600887 learned_addr learned my addr 172.21.15.16:0/4135600887 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:39:49.236 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:49.226+0000 7f6d40dcb640 1 -- 172.21.15.16:0/4135600887 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6d3c0ca4e0 msgr2=0x7f6d3c14b480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:49.236 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:49.226+0000 7f6d40dcb640 1 --2- 172.21.15.16:0/4135600887 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6d3c0ca4e0 0x7f6d3c14b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:49.236 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:49.226+0000 7f6d40dcb640 1 -- 172.21.15.16:0/4135600887 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6d3c0c9b40 msgr2=0x7f6d3c1571e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:49.236 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:49.226+0000 7f6d3bfff640 1 --2- 172.21.15.16:0/4135600887 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6d3c0c9b40 0x7f6d3c1571e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:49.236 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:49.226+0000 7f6d40dcb640 1 --2- 172.21.15.16:0/4135600887 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6d3c0c9b40 0x7f6d3c1571e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:49.237 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:49.226+0000 7f6d40dcb640 1 -- 172.21.15.16:0/4135600887 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d240094d0 con 0x7f6d3c070df0 2023-05-22T15:39:49.237 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:49.226+0000 7f6d3bfff640 1 --2- 172.21.15.16:0/4135600887 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6d3c0c9b40 0x7f6d3c1571e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:39:49.237 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:49.226+0000 7f6d40dcb640 1 --2- 172.21.15.16:0/4135600887 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6d3c070df0 0x7f6d3c14a230 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f6d240190e0 tx=0x7f6d2401ad10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:39:49.237 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:49.226+0000 7f6d397fa640 1 -- 172.21.15.16:0/4135600887 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d240bd5f0 con 0x7f6d3c070df0 2023-05-22T15:39:49.237 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:49.226+0000 7f6d397fa640 1 -- 172.21.15.16:0/4135600887 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d240bd770 con 0x7f6d3c070df0 2023-05-22T15:39:49.237 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:49.226+0000 7f6d4302f640 1 -- 172.21.15.16:0/4135600887 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d3c0c8e80 con 0x7f6d3c070df0 2023-05-22T15:39:49.238 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:49.226+0000 7f6d397fa640 1 -- 172.21.15.16:0/4135600887 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d240bdb20 con 0x7f6d3c070df0 2023-05-22T15:39:49.238 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:49.226+0000 7f6d4302f640 1 -- 172.21.15.16:0/4135600887 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d3c0ca8b0 con 0x7f6d3c070df0 2023-05-22T15:39:49.239 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:49.226+0000 7f6d4302f640 1 -- 172.21.15.16:0/4135600887 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d3c0c9010 con 0x7f6d3c070df0 2023-05-22T15:39:49.239 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:49.230+0000 7f6d397fa640 1 -- 172.21.15.16:0/4135600887 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 15) v1 ==== 84068+0+0 (secure 0 0 0) 0x7f6d240b33f0 con 0x7f6d3c070df0 2023-05-22T15:39:49.239 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:49.230+0000 7f6d397fa640 1 --2- 172.21.15.16:0/4135600887 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f6d1c0651d0 0x7f6d1c067680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:49.239 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:49.230+0000 7f6d397fa640 1 -- 172.21.15.16:0/4135600887 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 7455+0+0 (secure 0 0 0) 0x7f6d2411cf20 con 0x7f6d3c070df0 2023-05-22T15:39:49.239 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:49.230+0000 7f6d3bfff640 1 --2- 172.21.15.16:0/4135600887 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f6d1c0651d0 0x7f6d1c067680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:49.239 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:49.230+0000 7f6d3bfff640 1 --2- 172.21.15.16:0/4135600887 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f6d1c0651d0 0x7f6d1c067680 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f6d3c0b2850 tx=0x7f6d2c008040 comp rx=0 tx=0).ready entity=mgr.6240 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:39:49.240 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:49.234+0000 7f6d397fa640 1 -- 172.21.15.16:0/4135600887 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f6d240ea910 con 0x7f6d3c070df0 2023-05-22T15:39:49.387 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:49.382+0000 7f6d4302f640 1 -- 172.21.15.16:0/4135600887 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6d3c04dbe0 con 0x7f6d3c070df0 2023-05-22T15:39:49.390 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:49.386+0000 7f6d397fa640 1 -- 172.21.15.16:0/4135600887 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v60) v1 ==== 74+0+19649 (secure 0 0 0) 0x7f6d240edf30 con 0x7f6d3c070df0 2023-05-22T15:39:49.391 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:39:49.391 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":60,"fsid":"b2719834-7847-4d98-90a6-f65d57f85c49","created":"2023-05-22T15:26:25.259314+0000","modified":"2023-05-22T15:39:47.055848+0000","last_up_change":"2023-05-22T15:26:36.448524+0000","last_in_change":"2023-05-22T15:26:27.675668+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-05-22T15:26:38.019781+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-05-22T15:26:43.570514+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2023-05-22T15:28:11.902605+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"41","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2023-05-22T15:28:12.298278+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"60","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":221,"snap_epoch":60,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9e7ff854-b25f-4cbb-a5c2-1799d9c53d2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":37021},{"type":"v1","addr":"172.21.15.16:6825","nonce":37021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":37021},{"type":"v1","addr":"172.21.15.16:6827","nonce":37021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":37021},{"type":"v1","addr":"172.21.15.16:6831","nonce":37021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":37021},{"type":"v1","addr":"172.21.15.16:6829","nonce":37021}]},"public_addr":"172.21.15.16:6825/37021","cluster_addr":"172.21.15.16:6827/37021","heartbeat_back_addr":"172.21.15.16:6831/37021","heartbeat_front_addr":"172.21.15.16:6829/37021","state":["exists","up"]},{"osd":1,"uuid":"4702fbc9-a927-4565-aca5-41dc69cc8264","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":37008},{"type":"v1","addr":"172.21.15.16:6801","nonce":37008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":37008},{"type":"v1","addr":"172.21.15.16:6803","nonce":37008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":37008},{"type":"v1","addr":"172.21.15.16:6807","nonce":37008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":37008},{"type":"v1","addr":"172.21.15.16:6805","nonce":37008}]},"public_addr":"172.21.15.16:6801/37008","cluster_addr":"172.21.15.16:6803/37008","heartbeat_back_addr":"172.21.15.16:6807/37008","heartbeat_front_addr":"172.21.15.16:6805/37008","state":["exists","up"]},{"osd":2,"uuid":"1e7bba46-4eba-47ef-bb81-5e2d371638ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":37009},{"type":"v1","addr":"172.21.15.16:6809","nonce":37009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":37009},{"type":"v1","addr":"172.21.15.16:6811","nonce":37009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":37009},{"type":"v1","addr":"172.21.15.16:6815","nonce":37009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":37009},{"type":"v1","addr":"172.21.15.16:6813","nonce":37009}]},"public_addr":"172.21.15.16:6809/37009","cluster_addr":"172.21.15.16:6811/37009","heartbeat_back_addr":"172.21.15.16:6815/37009","heartbeat_front_addr":"172.21.15.16:6813/37009","state":["exists","up"]},{"osd":3,"uuid":"383cacc0-5a88-4904-990e-498ca59ef48a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":37020},{"type":"v1","addr":"172.21.15.16:6817","nonce":37020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":37020},{"type":"v1","addr":"172.21.15.16:6819","nonce":37020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":37020},{"type":"v1","addr":"172.21.15.16:6823","nonce":37020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":37020},{"type":"v1","addr":"172.21.15.16:6821","nonce":37020}]},"public_addr":"172.21.15.16:6817/37020","cluster_addr":"172.21.15.16:6819/37020","heartbeat_back_addr":"172.21.15.16:6823/37020","heartbeat_front_addr":"172.21.15.16:6821/37020","state":["exists","up"]},{"osd":4,"uuid":"970500ef-95b7-42c1-b080-eed4fae7013d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":37353},{"type":"v1","addr":"172.21.15.62:6817","nonce":37353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6819","nonce":37353},{"type":"v1","addr":"172.21.15.62:6822","nonce":37353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37353},{"type":"v1","addr":"172.21.15.62:6831","nonce":37353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37353},{"type":"v1","addr":"172.21.15.62:6826","nonce":37353}]},"public_addr":"172.21.15.62:6817/37353","cluster_addr":"172.21.15.62:6822/37353","heartbeat_back_addr":"172.21.15.62:6831/37353","heartbeat_front_addr":"172.21.15.62:6826/37353","state":["exists","up"]},{"osd":5,"uuid":"022d67db-d4d8-445b-b1b9-8b446a00f5c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":37352},{"type":"v1","addr":"172.21.15.62:6801","nonce":37352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":37352},{"type":"v1","addr":"172.21.15.62:6803","nonce":37352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":37352},{"type":"v1","addr":"172.21.15.62:6807","nonce":37352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":37352},{"type":"v1","addr":"172.21.15.62:6805","nonce":37352}]},"public_addr":"172.21.15.62:6801/37352","cluster_addr":"172.21.15.62:6803/37352","heartbeat_back_addr":"172.21.15.62:6807/37352","heartbeat_front_addr":"172.21.15.62:6805/37352","state":["exists","up"]},{"osd":6,"uuid":"42bd372b-6688-4a1b-8407-e16e8c266e33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":37354},{"type":"v1","addr":"172.21.15.62:6820","nonce":37354}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":37354},{"type":"v1","addr":"172.21.15.62:6823","nonce":37354}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6829","nonce":37354},{"type":"v1","addr":"172.21.15.62:6830","nonce":37354}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6825","nonce":37354},{"type":"v1","addr":"172.21.15.62:6827","nonce":37354}]},"public_addr":"172.21.15.62:6820/37354","cluster_addr":"172.21.15.62:6823/37354","heartbeat_back_addr":"172.21.15.62:6830/37354","heartbeat_front_addr":"172.21.15.62:6827/37354","state":["exists","up"]},{"osd":7,"uuid":"ac70b2a3-f41b-4bf2-b227-7f2cbb55426d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37355},{"type":"v1","addr":"172.21.15.62:6809","nonce":37355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37355},{"type":"v1","addr":"172.21.15.62:6811","nonce":37355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":37355},{"type":"v1","addr":"172.21.15.62:6815","nonce":37355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":37355},{"type":"v1","addr":"172.21.15.62:6813","nonce":37355}]},"public_addr":"172.21.15.62:6809/37355","cluster_addr":"172.21.15.62:6811/37355","heartbeat_back_addr":"172.21.15.62:6815/37355","heartbeat_front_addr":"172.21.15.62:6813/37355","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.368896+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.107273+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.276888+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.287502+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.124791+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.251937+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.255679+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.232797+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"7.4","mappings":[{"from":0,"to":5}]},{"pgid":"7.2e","mappings":[{"from":0,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:0/2741846205":"2023-05-23T15:39:47.055770+0000","172.21.15.16:0/651011632":"2023-05-23T15:39:47.055770+0000","172.21.15.16:0/2307989846":"2023-05-23T15:39:47.055770+0000","172.21.15.16:0/2901257227":"2023-05-23T15:39:47.055770+0000","172.21.15.16:0/3979017463":"2023-05-23T15:28:30.504372+0000","172.21.15.16:0/901306510":"2023-05-23T15:28:30.504372+0000","172.21.15.16:0/1579456307":"2023-05-23T15:39:47.055770+0000","172.21.15.16:0/2755009771":"2023-05-23T15:28:30.504372+0000","172.21.15.16:0/4222124422":"2023-05-23T15:28:30.504372+0000","172.21.15.16:0/593205291":"2023-05-23T15:39:47.055770+0000","172.21.15.16:6833/36921":"2023-05-23T15:28:30.504372+0000","172.21.15.16:0/4184273947":"2023-05-23T15:39:47.055770+0000","172.21.15.16:6832/36921":"2023-05-23T15:28:30.504372+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":7,"snaps":[{"begin":6,"length":1},{"begin":10,"length":1},{"begin":14,"length":1},{"begin":19,"length":1},{"begin":24,"length":1},{"begin":29,"length":1},{"begin":35,"length":1},{"begin":39,"length":2},{"begin":45,"length":1},{"begin":47,"length":1},{"begin":50,"length":1},{"begin":53,"length":1},{"begin":58,"length":1},{"begin":61,"length":1},{"begin":65,"length":1},{"begin":67,"length":1},{"begin":70,"length":1},{"begin":74,"length":1},{"begin":77,"length":1},{"begin":81,"length":1},{"begin":84,"length":1},{"begin":89,"length":2},{"begin":98,"length":3},{"begin":106,"length":3},{"begin":192,"length":30}]}],"new_removed_snaps":[{"pool":7,"snaps":[{"begin":107,"length":1},{"begin":221,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-05-22T15:39:49.394 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:49.386+0000 7f6d4302f640 1 -- 172.21.15.16:0/4135600887 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f6d1c0651d0 msgr2=0x7f6d1c067680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:49.394 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:49.386+0000 7f6d4302f640 1 --2- 172.21.15.16:0/4135600887 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f6d1c0651d0 0x7f6d1c067680 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f6d3c0b2850 tx=0x7f6d2c008040 comp rx=0 tx=0).stop 2023-05-22T15:39:49.394 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:49.386+0000 7f6d4302f640 1 -- 172.21.15.16:0/4135600887 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6d3c070df0 msgr2=0x7f6d3c14a230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:49.394 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:49.386+0000 7f6d4302f640 1 --2- 172.21.15.16:0/4135600887 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6d3c070df0 0x7f6d3c14a230 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f6d240190e0 tx=0x7f6d2401ad10 comp rx=0 tx=0).stop 2023-05-22T15:39:49.394 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:49.386+0000 7f6d4302f640 1 -- 172.21.15.16:0/4135600887 shutdown_connections 2023-05-22T15:39:49.395 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:49.386+0000 7f6d4302f640 1 --2- 172.21.15.16:0/4135600887 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f6d1c0651d0 0x7f6d1c067680 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:49.395 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:49.386+0000 7f6d4302f640 1 --2- 172.21.15.16:0/4135600887 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6d3c0ca4e0 0x7f6d3c14b480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:49.395 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:49.386+0000 7f6d4302f640 1 --2- 172.21.15.16:0/4135600887 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6d3c0c9b40 0x7f6d3c1571e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:49.395 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:49.386+0000 7f6d4302f640 1 --2- 172.21.15.16:0/4135600887 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6d3c070df0 0x7f6d3c14a230 unknown :-1 s=CLOSED pgs=844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:49.395 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:49.386+0000 7f6d4302f640 1 -- 172.21.15.16:0/4135600887 >> 172.21.15.16:0/4135600887 conn(0x7f6d3c069720 msgr2=0x7f6d3c0b13d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:39:49.395 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:49.390+0000 7f6d4302f640 1 -- 172.21.15.16:0/4135600887 shutdown_connections 2023-05-22T15:39:49.395 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:49.390+0000 7f6d4302f640 1 -- 172.21.15.16:0/4135600887 wait complete. 2023-05-22T15:39:49.410 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-05-22T15:39:49.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:49.518+0000 7fa0634b5640 1 Processor -- start 2023-05-22T15:39:49.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:49.518+0000 7fa0634b5640 1 -- start start 2023-05-22T15:39:49.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:49.518+0000 7fa0634b5640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fa05c15cfa0 0x7fa05c06db00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:49.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:49.518+0000 7fa0634b5640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fa05c06e740 0x7fa05c06ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:49.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:49.518+0000 7fa0634b5640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa05c06e040 con 0x7fa05c0cf870 2023-05-22T15:39:49.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:49.518+0000 7fa0634b5640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa05c0cb9f0 con 0x7fa05c15cfa0 2023-05-22T15:39:49.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:49.518+0000 7fa0634b5640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa05c06f100 con 0x7fa05c06e740 2023-05-22T15:39:49.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:49.518+0000 7fa061251640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fa05c15cfa0 0x7fa05c06db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:49.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:49.518+0000 7fa061251640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fa05c15cfa0 0x7fa05c06db00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57376/0 (socket says 172.21.15.16:57376) 2023-05-22T15:39:49.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:49.518+0000 7fa061251640 1 -- 172.21.15.16:0/4155031863 learned_addr learned my addr 172.21.15.16:0/4155031863 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:39:49.526 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:49.518+0000 7fa061a52640 1 --2- 172.21.15.16:0/4155031863 >> v2:172.21.15.62:3300/0 conn(0x7fa05c06e740 0x7fa05c06ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:49.526 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:49.518+0000 7fa061251640 1 -- 172.21.15.16:0/4155031863 >> v2:172.21.15.62:3300/0 conn(0x7fa05c06e740 msgr2=0x7fa05c06ebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:39:49.526 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:49.518+0000 7fa061251640 1 --2- 172.21.15.16:0/4155031863 >> v2:172.21.15.62:3300/0 conn(0x7fa05c06e740 0x7fa05c06ebc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:49.526 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:49.518+0000 7fa061251640 1 -- 172.21.15.16:0/4155031863 >> v1:172.21.15.16:6789/0 conn(0x7fa05c0cf870 legacy=0x7fa05c0b1fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:49.526 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:49.518+0000 7fa061251640 1 -- 172.21.15.16:0/4155031863 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa05c06e490 con 0x7fa05c15cfa0 2023-05-22T15:39:49.526 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:49.518+0000 7fa053fff640 1 -- 172.21.15.16:0/4155031863 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 705646733 0 0) 0x7fa04c001120 con 0x7fa05c0cf870 2023-05-22T15:39:49.527 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:49.522+0000 7fa061251640 1 --2- 172.21.15.16:0/4155031863 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa05c15cfa0 0x7fa05c06db00 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7fa05c0c9760 tx=0x7fa05801c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1f852ebe43a347f server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:39:49.527 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:49.522+0000 7fa053fff640 1 -- 172.21.15.16:0/4155031863 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0580aa430 con 0x7fa05c15cfa0 2023-05-22T15:39:49.527 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:49.522+0000 7fa053fff640 1 -- 172.21.15.16:0/4155031863 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0580aa590 con 0x7fa05c15cfa0 2023-05-22T15:39:49.527 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:49.522+0000 7fa053fff640 1 -- 172.21.15.16:0/4155031863 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0580aa920 con 0x7fa05c15cfa0 2023-05-22T15:39:49.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:49.522+0000 7fa0634b5640 1 -- 172.21.15.16:0/4155031863 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa05c15cfa0 msgr2=0x7fa05c06db00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:49.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:49.522+0000 7fa0634b5640 1 --2- 172.21.15.16:0/4155031863 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa05c15cfa0 0x7fa05c06db00 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7fa05c0c9760 tx=0x7fa05801c4d0 comp rx=0 tx=0).stop 2023-05-22T15:39:49.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:49.522+0000 7fa0634b5640 1 -- 172.21.15.16:0/4155031863 shutdown_connections 2023-05-22T15:39:49.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:49.522+0000 7fa0634b5640 1 --2- 172.21.15.16:0/4155031863 >> v2:172.21.15.62:3300/0 conn(0x7fa05c06e740 0x7fa05c06ebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:49.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:49.522+0000 7fa0634b5640 1 --2- 172.21.15.16:0/4155031863 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa05c15cfa0 0x7fa05c06db00 unknown :-1 s=CLOSED pgs=846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:49.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:49.522+0000 7fa0634b5640 1 -- 172.21.15.16:0/4155031863 >> 172.21.15.16:0/4155031863 conn(0x7fa05c0cc870 msgr2=0x7fa05c0cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:39:49.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:49.522+0000 7fa0634b5640 1 -- 172.21.15.16:0/4155031863 shutdown_connections 2023-05-22T15:39:49.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:49.522+0000 7fa0634b5640 1 -- 172.21.15.16:0/4155031863 wait complete. 2023-05-22T15:39:49.530 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:49.522+0000 7fa0634b5640 1 Processor -- start 2023-05-22T15:39:49.530 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:49.522+0000 7fa0634b5640 1 -- start start 2023-05-22T15:39:49.531 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:49.522+0000 7fa0634b5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa05c06e740 0x7fa05c1f5ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:49.531 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:49.522+0000 7fa0634b5640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa05c0cf870 0x7fa05c14c0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:49.531 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:49.522+0000 7fa0634b5640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa05c15cfa0 0x7fa05c0c8640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:49.531 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:49.522+0000 7fa0634b5640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa05c1f60e0 con 0x7fa05c06e740 2023-05-22T15:39:49.531 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:49.522+0000 7fa0634b5640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa05c1f6310 con 0x7fa05c15cfa0 2023-05-22T15:39:49.531 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:49.522+0000 7fa0634b5640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa05c14b330 con 0x7fa05c0cf870 2023-05-22T15:39:49.531 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:49.522+0000 7fa061251640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa05c06e740 0x7fa05c1f5ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:49.531 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:49.522+0000 7fa061251640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa05c06e740 0x7fa05c1f5ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57386/0 (socket says 172.21.15.16:57386) 2023-05-22T15:39:49.532 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:49.522+0000 7fa061251640 1 -- 172.21.15.16:0/1606126378 learned_addr learned my addr 172.21.15.16:0/1606126378 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:39:49.532 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:49.526+0000 7fa061251640 1 -- 172.21.15.16:0/1606126378 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa05c0cf870 msgr2=0x7fa05c14c0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:49.532 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:49.526+0000 7fa060a50640 1 --2- 172.21.15.16:0/1606126378 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa05c0cf870 0x7fa05c14c0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:49.532 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:49.526+0000 7fa061a52640 1 --2- 172.21.15.16:0/1606126378 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa05c15cfa0 0x7fa05c0c8640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:49.532 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:49.526+0000 7fa061251640 1 --2- 172.21.15.16:0/1606126378 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa05c0cf870 0x7fa05c14c0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:49.532 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:49.526+0000 7fa061251640 1 -- 172.21.15.16:0/1606126378 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa05c15cfa0 msgr2=0x7fa05c0c8640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:49.532 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:49.526+0000 7fa061251640 1 --2- 172.21.15.16:0/1606126378 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa05c15cfa0 0x7fa05c0c8640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:49.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:49.526+0000 7fa061251640 1 -- 172.21.15.16:0/1606126378 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0580094d0 con 0x7fa05c06e740 2023-05-22T15:39:49.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:49.526+0000 7fa061251640 1 --2- 172.21.15.16:0/1606126378 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa05c06e740 0x7fa05c1f5ba0 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7fa0580ac040 tx=0x7fa05801ad10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:39:49.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:49.526+0000 7fa051ffb640 1 -- 172.21.15.16:0/1606126378 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0580b3800 con 0x7fa05c06e740 2023-05-22T15:39:49.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:49.526+0000 7fa051ffb640 1 -- 172.21.15.16:0/1606126378 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0580b3980 con 0x7fa05c06e740 2023-05-22T15:39:49.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:49.526+0000 7fa051ffb640 1 -- 172.21.15.16:0/1606126378 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0580b3d30 con 0x7fa05c06e740 2023-05-22T15:39:49.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:49.526+0000 7fa0634b5640 1 -- 172.21.15.16:0/1606126378 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa05c0cb610 con 0x7fa05c06e740 2023-05-22T15:39:49.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:49.526+0000 7fa0634b5640 1 -- 172.21.15.16:0/1606126378 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa05c0cb330 con 0x7fa05c06e740 2023-05-22T15:39:49.534 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:49.526+0000 7fa0634b5640 1 -- 172.21.15.16:0/1606126378 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa05c15d370 con 0x7fa05c06e740 2023-05-22T15:39:49.534 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:49.526+0000 7fa051ffb640 1 -- 172.21.15.16:0/1606126378 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 15) v1 ==== 84068+0+0 (secure 0 0 0) 0x7fa0580c6440 con 0x7fa05c06e740 2023-05-22T15:39:49.534 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:49.526+0000 7fa051ffb640 1 --2- 172.21.15.16:0/1606126378 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fa0400651d0 0x7fa040067680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:49.534 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:49.526+0000 7fa051ffb640 1 -- 172.21.15.16:0/1606126378 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 7455+0+0 (secure 0 0 0) 0x7fa05811d6f0 con 0x7fa05c06e740 2023-05-22T15:39:49.535 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:49.526+0000 7fa060a50640 1 --2- 172.21.15.16:0/1606126378 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fa0400651d0 0x7fa040067680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:49.535 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:49.526+0000 7fa060a50640 1 --2- 172.21.15.16:0/1606126378 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fa0400651d0 0x7fa040067680 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fa04c001380 tx=0x7fa04c009040 comp rx=0 tx=0).ready entity=mgr.6240 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:39:49.536 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:49.530+0000 7fa051ffb640 1 -- 172.21.15.16:0/1606126378 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fa0580eb0e0 con 0x7fa05c06e740 2023-05-22T15:39:49.714 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:49.710+0000 7fa0634b5640 1 -- 172.21.15.16:0/1606126378 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fa05c069fb0 con 0x7fa05c06e740 2023-05-22T15:39:50.096 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:50.090+0000 7fa051ffb640 1 -- 172.21.15.16:0/1606126378 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mgrmap(e 16) v1 ==== 84332+0+0 (secure 0 0 0) 0x7fa0580eb4b0 con 0x7fa05c06e740 2023-05-22T15:39:50.394 INFO:tasks.ceph.mon.a.smithi016.stderr:2023-05-22T15:39:50.386+0000 7f95dd48a640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-05-22T15:39:50.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:50.410+0000 7fa051ffb640 1 -- 172.21.15.16:0/1606126378 <== mon.0 v2:172.21.15.16:3300/0 8 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v15) v1 ==== 161+0+0 (secure 0 0 0) 0x7fa0580eb0e0 con 0x7fa05c06e740 2023-05-22T15:39:50.417 INFO:teuthology.orchestra.run.smithi016.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-05-22T15:39:50.419 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:50.414+0000 7fa0634b5640 1 -- 172.21.15.16:0/1606126378 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fa0400651d0 msgr2=0x7fa040067680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:50.419 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:50.414+0000 7fa0634b5640 1 --2- 172.21.15.16:0/1606126378 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fa0400651d0 0x7fa040067680 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fa04c001380 tx=0x7fa04c009040 comp rx=0 tx=0).stop 2023-05-22T15:39:50.420 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:50.414+0000 7fa0634b5640 1 -- 172.21.15.16:0/1606126378 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa05c06e740 msgr2=0x7fa05c1f5ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:50.420 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:50.414+0000 7fa0634b5640 1 --2- 172.21.15.16:0/1606126378 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa05c06e740 0x7fa05c1f5ba0 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7fa0580ac040 tx=0x7fa05801ad10 comp rx=0 tx=0).stop 2023-05-22T15:39:50.420 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:50.414+0000 7fa0634b5640 1 -- 172.21.15.16:0/1606126378 shutdown_connections 2023-05-22T15:39:50.420 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:50.414+0000 7fa0634b5640 1 --2- 172.21.15.16:0/1606126378 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa05c15cfa0 0x7fa05c0c8640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:50.420 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:50.414+0000 7fa0634b5640 1 --2- 172.21.15.16:0/1606126378 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fa0400651d0 0x7fa040067680 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:50.420 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:50.414+0000 7fa0634b5640 1 --2- 172.21.15.16:0/1606126378 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa05c0cf870 0x7fa05c14c0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:50.420 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:50.414+0000 7fa0634b5640 1 --2- 172.21.15.16:0/1606126378 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa05c06e740 0x7fa05c1f5ba0 unknown :-1 s=CLOSED pgs=847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:50.421 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:50.414+0000 7fa0634b5640 1 -- 172.21.15.16:0/1606126378 >> 172.21.15.16:0/1606126378 conn(0x7fa05c0cc870 msgr2=0x7fa05c0c5920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:39:50.421 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:50.414+0000 7fa0634b5640 1 -- 172.21.15.16:0/1606126378 shutdown_connections 2023-05-22T15:39:50.421 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:50.414+0000 7fa0634b5640 1 -- 172.21.15.16:0/1606126378 wait complete. 2023-05-22T15:39:50.444 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-05-22T15:39:50.472 INFO:tasks.ceph.mds.c.smithi016.stdout:starting mds.c at 2023-05-22T15:39:50.566 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:50.558+0000 7f6b08387640 1 Processor -- start 2023-05-22T15:39:50.566 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:50.558+0000 7f6b08387640 1 -- start start 2023-05-22T15:39:50.567 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:50.562+0000 7f6b08387640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f6b0015c8b0 0x7f6b000cee30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:50.567 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:50.562+0000 7f6b08387640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6b0006bf60 0x7f6b0015cc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:50.567 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:50.562+0000 7f6b08387640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b0006c3b0 con 0x7f6b0015d1c0 2023-05-22T15:39:50.567 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:50.562+0000 7f6b08387640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b0015a0c0 con 0x7f6b0015c8b0 2023-05-22T15:39:50.567 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:50.562+0000 7f6b08387640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b000ce160 con 0x7f6b0006bf60 2023-05-22T15:39:50.568 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:50.562+0000 7f6b06123640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f6b0015c8b0 0x7f6b000cee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:50.568 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:50.562+0000 7f6b05922640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6b0006bf60 0x7f6b0015cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:50.568 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:50.562+0000 7f6b06123640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f6b0015c8b0 0x7f6b000cee30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.16:36860/0 (socket says 172.21.15.16:36860) 2023-05-22T15:39:50.568 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:50.562+0000 7f6b06924640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f6b0015d1c0 0x7f6b0006db80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.16:35624/0 (socket says 172.21.15.16:35624) 2023-05-22T15:39:50.569 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:50.562+0000 7f6b05922640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6b0006bf60 0x7f6b0015cc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3301/0 says I am v2:172.21.15.16:44200/0 (socket says 172.21.15.16:44200) 2023-05-22T15:39:50.569 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:50.562+0000 7f6b06123640 1 -- 172.21.15.16:0/2601561086 learned_addr learned my addr 172.21.15.16:0/2601561086 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:39:50.569 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:50.562+0000 7f6b05922640 1 -- 172.21.15.16:0/2601561086 >> v2:172.21.15.62:3300/0 conn(0x7f6b0015c8b0 msgr2=0x7f6b000cee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:39:50.569 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:50.562+0000 7f6b05922640 1 --2- 172.21.15.16:0/2601561086 >> v2:172.21.15.62:3300/0 conn(0x7f6b0015c8b0 0x7f6b000cee30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:50.569 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:50.562+0000 7f6b05922640 1 -- 172.21.15.16:0/2601561086 >> v1:172.21.15.62:6789/0 conn(0x7f6b0015d1c0 legacy=0x7f6b0006db80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:50.569 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:50.562+0000 7f6b05922640 1 -- 172.21.15.16:0/2601561086 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b0006c640 con 0x7f6b0006bf60 2023-05-22T15:39:50.569 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:50.562+0000 7f6b06123640 1 --2- 172.21.15.16:0/2601561086 >> v2:172.21.15.62:3300/0 conn(0x7f6b0015c8b0 0x7f6b000cee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-05-22T15:39:50.570 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:50.566+0000 7f6b05922640 1 --2- 172.21.15.16:0/2601561086 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6b0006bf60 0x7f6b0015cc80 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f6af00097c0 tx=0x7f6af001c480 comp rx=0 tx=0).ready entity=mon.2 client_cookie=79ffc2ce249fc094 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:39:50.570 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:50.566+0000 7f6b05121640 1 -- 172.21.15.16:0/2601561086 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6af00ab600 con 0x7f6b0006bf60 2023-05-22T15:39:50.570 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:50.566+0000 7f6b05121640 1 -- 172.21.15.16:0/2601561086 <== mon.2 v2:172.21.15.62:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6af00a7050 con 0x7f6b0006bf60 2023-05-22T15:39:50.570 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:50.566+0000 7f6b05121640 1 -- 172.21.15.16:0/2601561086 <== mon.2 v2:172.21.15.62:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6af00ab980 con 0x7f6b0006bf60 2023-05-22T15:39:50.571 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:50.566+0000 7f6b08387640 1 -- 172.21.15.16:0/2601561086 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6b0006bf60 msgr2=0x7f6b0015cc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:50.571 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:50.566+0000 7f6b08387640 1 --2- 172.21.15.16:0/2601561086 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6b0006bf60 0x7f6b0015cc80 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f6af00097c0 tx=0x7f6af001c480 comp rx=0 tx=0).stop 2023-05-22T15:39:50.571 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:50.566+0000 7f6b08387640 1 -- 172.21.15.16:0/2601561086 shutdown_connections 2023-05-22T15:39:50.572 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:50.566+0000 7f6b08387640 1 --2- 172.21.15.16:0/2601561086 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6b0006bf60 0x7f6b0015cc80 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:50.572 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:50.566+0000 7f6b08387640 1 --2- 172.21.15.16:0/2601561086 >> v2:172.21.15.62:3300/0 conn(0x7f6b0015c8b0 0x7f6b000cee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:50.572 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:50.566+0000 7f6b08387640 1 -- 172.21.15.16:0/2601561086 >> 172.21.15.16:0/2601561086 conn(0x7f6b0006f5a0 msgr2=0x7f6b0006f9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:39:50.572 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:50.566+0000 7f6b08387640 1 -- 172.21.15.16:0/2601561086 shutdown_connections 2023-05-22T15:39:50.572 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:50.566+0000 7f6b08387640 1 -- 172.21.15.16:0/2601561086 wait complete. 2023-05-22T15:39:50.573 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:50.566+0000 7f6b08387640 1 Processor -- start 2023-05-22T15:39:50.573 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:50.566+0000 7f6b08387640 1 -- start start 2023-05-22T15:39:50.573 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:50.566+0000 7f6b08387640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6b0006bf60 0x7f6b000c5ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:50.573 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:50.566+0000 7f6b08387640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6b0015c8b0 0x7f6b000c6420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:50.574 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:50.566+0000 7f6b08387640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6b0015d1c0 0x7f6b00152be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:50.574 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:50.566+0000 7f6b08387640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b000b5c70 con 0x7f6b0006bf60 2023-05-22T15:39:50.574 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:50.566+0000 7f6b08387640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b000b6070 con 0x7f6b0015d1c0 2023-05-22T15:39:50.574 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:50.566+0000 7f6b08387640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b00153c10 con 0x7f6b0015c8b0 2023-05-22T15:39:50.574 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:50.566+0000 7f6b06123640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6b0006bf60 0x7f6b000c5ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:50.574 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:50.566+0000 7f6b06123640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6b0006bf60 0x7f6b000c5ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57412/0 (socket says 172.21.15.16:57412) 2023-05-22T15:39:50.574 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:50.566+0000 7f6b06123640 1 -- 172.21.15.16:0/612960562 learned_addr learned my addr 172.21.15.16:0/612960562 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:39:50.575 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:50.566+0000 7f6b05922640 1 --2- 172.21.15.16:0/612960562 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6b0015c8b0 0x7f6b000c6420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:50.576 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:50.566+0000 7f6b06924640 1 --2- 172.21.15.16:0/612960562 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6b0015d1c0 0x7f6b00152be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:50.576 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:50.566+0000 7f6b06123640 1 -- 172.21.15.16:0/612960562 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6b0015c8b0 msgr2=0x7f6b000c6420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:50.576 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:50.566+0000 7f6b06123640 1 --2- 172.21.15.16:0/612960562 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6b0015c8b0 0x7f6b000c6420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:50.576 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:50.566+0000 7f6b06123640 1 -- 172.21.15.16:0/612960562 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6b0015d1c0 msgr2=0x7f6b00152be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:50.576 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:50.566+0000 7f6b06123640 1 --2- 172.21.15.16:0/612960562 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6b0015d1c0 0x7f6b00152be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:50.576 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:50.566+0000 7f6b06123640 1 -- 172.21.15.16:0/612960562 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6afc009b30 con 0x7f6b0006bf60 2023-05-22T15:39:50.577 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:50.570+0000 7f6b06123640 1 --2- 172.21.15.16:0/612960562 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6b0006bf60 0x7f6b000c5ee0 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f6afc00efc0 tx=0x7f6afc00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:39:50.577 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:50.570+0000 7f6af6ffd640 1 -- 172.21.15.16:0/612960562 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6afc00ce60 con 0x7f6b0006bf60 2023-05-22T15:39:50.577 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:50.570+0000 7f6af6ffd640 1 -- 172.21.15.16:0/612960562 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6afc00f040 con 0x7f6b0006bf60 2023-05-22T15:39:50.577 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:50.570+0000 7f6af6ffd640 1 -- 172.21.15.16:0/612960562 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6afc014680 con 0x7f6b0006bf60 2023-05-22T15:39:50.577 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:50.570+0000 7f6b08387640 1 -- 172.21.15.16:0/612960562 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6af0009420 con 0x7f6b0006bf60 2023-05-22T15:39:50.577 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:50.570+0000 7f6b08387640 1 -- 172.21.15.16:0/612960562 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b0006e290 con 0x7f6b0006bf60 2023-05-22T15:39:50.577 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:50.570+0000 7f6b08387640 1 -- 172.21.15.16:0/612960562 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b000ce930 con 0x7f6b0006bf60 2023-05-22T15:39:50.577 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:50.570+0000 7f6af6ffd640 1 -- 172.21.15.16:0/612960562 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f6afc0147e0 con 0x7f6b0006bf60 2023-05-22T15:39:50.578 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:50.570+0000 7f6af6ffd640 1 --2- 172.21.15.16:0/612960562 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f6ae40654a0 0x7f6ae4067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:50.578 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:50.570+0000 7f6af6ffd640 1 -- 172.21.15.16:0/612960562 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 7497+0+0 (secure 0 0 0) 0x7f6afc0842b0 con 0x7f6b0006bf60 2023-05-22T15:39:50.578 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:50.570+0000 7f6b05922640 1 --2- 172.21.15.16:0/612960562 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f6ae40654a0 0x7f6ae4067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:50.579 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:50.570+0000 7f6b05922640 1 --2- 172.21.15.16:0/612960562 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f6ae40654a0 0x7f6ae4067950 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f6af00ad000 tx=0x7f6af00a6000 comp rx=0 tx=0).ready entity=mgr.6240 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:39:50.580 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:50.574+0000 7f6af6ffd640 1 -- 172.21.15.16:0/612960562 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f6afc0518e0 con 0x7f6b0006bf60 2023-05-22T15:39:50.767 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:50.762+0000 7f6b08387640 1 -- 172.21.15.16:0/612960562 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f6b000b5ee0 con 0x7f6b0006bf60 2023-05-22T15:39:51.415 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:51.410+0000 7f6af6ffd640 1 -- 172.21.15.16:0/612960562 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v16) v1 ==== 104+0+0 (secure 0 0 0) 0x7f6afc054f00 con 0x7f6b0006bf60 2023-05-22T15:39:51.419 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:51.414+0000 7f6b08387640 1 -- 172.21.15.16:0/612960562 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f6ae40654a0 msgr2=0x7f6ae4067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:51.420 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:51.414+0000 7f6b08387640 1 --2- 172.21.15.16:0/612960562 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f6ae40654a0 0x7f6ae4067950 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f6af00ad000 tx=0x7f6af00a6000 comp rx=0 tx=0).stop 2023-05-22T15:39:51.420 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:51.414+0000 7f6b08387640 1 -- 172.21.15.16:0/612960562 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6b0006bf60 msgr2=0x7f6b000c5ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:51.420 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:51.414+0000 7f6b08387640 1 --2- 172.21.15.16:0/612960562 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6b0006bf60 0x7f6b000c5ee0 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f6afc00efc0 tx=0x7f6afc00c490 comp rx=0 tx=0).stop 2023-05-22T15:39:51.420 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:51.414+0000 7f6b08387640 1 -- 172.21.15.16:0/612960562 shutdown_connections 2023-05-22T15:39:51.420 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:51.414+0000 7f6b08387640 1 --2- 172.21.15.16:0/612960562 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6b0015d1c0 0x7f6b00152be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:51.421 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:51.414+0000 7f6b08387640 1 --2- 172.21.15.16:0/612960562 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f6ae40654a0 0x7f6ae4067950 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:51.421 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:51.414+0000 7f6b08387640 1 --2- 172.21.15.16:0/612960562 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6b0015c8b0 0x7f6b000c6420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:51.421 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:51.414+0000 7f6b08387640 1 --2- 172.21.15.16:0/612960562 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6b0006bf60 0x7f6b000c5ee0 unknown :-1 s=CLOSED pgs=850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:51.421 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:51.414+0000 7f6b08387640 1 -- 172.21.15.16:0/612960562 >> 172.21.15.16:0/612960562 conn(0x7f6b0006f5a0 msgr2=0x7f6b000caee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:39:51.421 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:51.414+0000 7f6b08387640 1 -- 172.21.15.16:0/612960562 shutdown_connections 2023-05-22T15:39:51.421 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:51.414+0000 7f6b08387640 1 -- 172.21.15.16:0/612960562 wait complete. 2023-05-22T15:39:51.442 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-05-22T15:39:51.621 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:51.614+0000 7fa00258b640 1 Processor -- start 2023-05-22T15:39:51.621 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:51.614+0000 7fa00258b640 1 -- start start 2023-05-22T15:39:51.622 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:51.614+0000 7fa00258b640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9ffc070c10 0x7f9ffc071580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:51.622 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:51.618+0000 7fa00258b640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ffc0cdda0 con 0x7f9ffc0cd620 2023-05-22T15:39:51.622 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:51.618+0000 7fa00258b640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ffc15a190 con 0x7f9ffc0ce740 2023-05-22T15:39:51.622 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:51.618+0000 7fa00258b640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ffc0cb880 con 0x7f9ffc070c10 2023-05-22T15:39:51.622 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:51.618+0000 7f9ffb7fe640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f9ffc0cd620 0x7f9ffc0ce030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:51258/0 (socket says 172.21.15.16:51258) 2023-05-22T15:39:51.623 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:51.618+0000 7f9ffb7fe640 1 -- 172.21.15.16:0/3041600040 learned_addr learned my addr 172.21.15.16:0/3041600040 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:39:51.624 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:51.618+0000 7f9ffbfff640 1 --2- 172.21.15.16:0/3041600040 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9ffc070c10 0x7f9ffc071580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:51.624 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:51.618+0000 7f9ffaffd640 1 -- 172.21.15.16:0/3041600040 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4146109098 0 0) 0x7f9ffc0cdda0 con 0x7f9ffc0cd620 2023-05-22T15:39:51.624 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:51.618+0000 7f9ffaffd640 1 -- 172.21.15.16:0/3041600040 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9fe4003530 con 0x7f9ffc0cd620 2023-05-22T15:39:51.624 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:51.618+0000 7f9ffaffd640 1 -- 172.21.15.16:0/3041600040 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4213365273 0 0) 0x7f9fe4003530 con 0x7f9ffc0cd620 2023-05-22T15:39:51.625 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:51.618+0000 7f9ffaffd640 1 -- 172.21.15.16:0/3041600040 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9ffc070c10 msgr2=0x7f9ffc071580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:39:51.625 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:51.618+0000 7f9ffaffd640 1 --2- 172.21.15.16:0/3041600040 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9ffc070c10 0x7f9ffc071580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:51.625 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:51.618+0000 7f9ffaffd640 1 -- 172.21.15.16:0/3041600040 >> v1:172.21.15.62:6789/0 conn(0x7f9ffc0ce740 legacy=0x7f9ffc150300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:51.625 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:51.618+0000 7f9ffaffd640 1 -- 172.21.15.16:0/3041600040 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ffc15a470 con 0x7f9ffc0cd620 2023-05-22T15:39:51.625 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:51.618+0000 7f9ffaffd640 1 -- 172.21.15.16:0/3041600040 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7f9ff0001680 con 0x7f9ffc0cd620 2023-05-22T15:39:51.625 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:51.618+0000 7f9ffaffd640 1 -- 172.21.15.16:0/3041600040 >> v1:172.21.15.16:6789/0 conn(0x7f9ffc0cd620 legacy=0x7f9ffc0ce030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:51.626 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:51.618+0000 7f9ffaffd640 1 --2- 172.21.15.16:0/3041600040 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9fe40039b0 0x7f9fe4003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:51.626 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:51.618+0000 7f9ffaffd640 1 --2- 172.21.15.16:0/3041600040 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9fe40047d0 0x7f9fe4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:51.626 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:51.618+0000 7f9ffaffd640 1 --2- 172.21.15.16:0/3041600040 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9ffc070c10 0x7f9fe4005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:51.626 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:51.618+0000 7f9ffaffd640 1 -- 172.21.15.16:0/3041600040 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ffc0cb880 con 0x7f9fe40039b0 2023-05-22T15:39:51.627 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:51.618+0000 7f9ffaffd640 1 -- 172.21.15.16:0/3041600040 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fe400ba40 con 0x7f9fe40047d0 2023-05-22T15:39:51.627 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:51.618+0000 7f9ffaffd640 1 -- 172.21.15.16:0/3041600040 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9fe400bbd0 con 0x7f9ffc070c10 2023-05-22T15:39:51.627 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:51.618+0000 7f9ffbfff640 1 --2- 172.21.15.16:0/3041600040 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9fe40039b0 0x7f9fe4003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:51.627 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:51.618+0000 7f9ffb7fe640 1 --2- 172.21.15.16:0/3041600040 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9fe40047d0 0x7f9fe4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:51.627 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:51.618+0000 7fa000b28640 1 --2- 172.21.15.16:0/3041600040 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9ffc070c10 0x7f9fe4005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:51.627 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:51.618+0000 7f9ffbfff640 1 -- 172.21.15.16:0/3041600040 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9ffc070c10 msgr2=0x7f9fe4005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:39:51.627 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:51.618+0000 7f9ffbfff640 1 --2- 172.21.15.16:0/3041600040 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9ffc070c10 0x7f9fe4005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:51.628 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:51.618+0000 7f9ffbfff640 1 -- 172.21.15.16:0/3041600040 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9fe40047d0 msgr2=0x7f9fe4004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:39:51.628 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:51.618+0000 7f9ffbfff640 1 --2- 172.21.15.16:0/3041600040 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9fe40047d0 0x7f9fe4004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:51.628 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:51.618+0000 7f9ffbfff640 1 -- 172.21.15.16:0/3041600040 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fe4003530 con 0x7f9fe40039b0 2023-05-22T15:39:51.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:51.618+0000 7f9ffbfff640 1 --2- 172.21.15.16:0/3041600040 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9fe40039b0 0x7f9fe4003dc0 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7f9fec009ae0 tx=0x7f9fec01ca50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d361f96e3d028bc server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:39:51.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:51.618+0000 7f9ffaffd640 1 -- 172.21.15.16:0/3041600040 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fec0a6430 con 0x7f9fe40039b0 2023-05-22T15:39:51.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:51.618+0000 7f9ffaffd640 1 -- 172.21.15.16:0/3041600040 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9fec0a6590 con 0x7f9fe40039b0 2023-05-22T15:39:51.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:51.618+0000 7f9ffaffd640 1 -- 172.21.15.16:0/3041600040 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fec0a6940 con 0x7f9fe40039b0 2023-05-22T15:39:51.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:51.622+0000 7fa00258b640 1 -- 172.21.15.16:0/3041600040 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9fe40039b0 msgr2=0x7f9fe4003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:51.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:51.622+0000 7fa00258b640 1 --2- 172.21.15.16:0/3041600040 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9fe40039b0 0x7f9fe4003dc0 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7f9fec009ae0 tx=0x7f9fec01ca50 comp rx=0 tx=0).stop 2023-05-22T15:39:51.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:51.622+0000 7fa000b28640 1 -- 172.21.15.16:0/3041600040 reap_dead start 2023-05-22T15:39:51.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:51.622+0000 7fa00258b640 1 -- 172.21.15.16:0/3041600040 shutdown_connections 2023-05-22T15:39:51.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:51.622+0000 7fa00258b640 1 -- 172.21.15.16:0/3041600040 >> 172.21.15.16:0/3041600040 conn(0x7f9ffc069720 msgr2=0x7f9ffc069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:39:51.632 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:51.622+0000 7fa00258b640 1 -- 172.21.15.16:0/3041600040 shutdown_connections 2023-05-22T15:39:51.632 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:51.622+0000 7fa00258b640 1 -- 172.21.15.16:0/3041600040 wait complete. 2023-05-22T15:39:51.632 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:51.622+0000 7fa00258b640 1 Processor -- start 2023-05-22T15:39:51.632 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:51.622+0000 7fa00258b640 1 -- start start 2023-05-22T15:39:51.632 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:51.622+0000 7fa00258b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9fe40039b0 0x7f9ffc1537f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:51.632 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:51.622+0000 7fa00258b640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9ffc14e490 0x7f9ffc14dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:51.632 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:51.622+0000 7fa00258b640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9ffc150fb0 0x7f9ffc151420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:51.633 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:51.622+0000 7fa00258b640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ffc0b0980 con 0x7f9fe40039b0 2023-05-22T15:39:51.633 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:51.622+0000 7fa00258b640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ffc153e80 con 0x7f9ffc14e490 2023-05-22T15:39:51.633 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:51.622+0000 7fa00258b640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ffc0b03c0 con 0x7f9ffc150fb0 2023-05-22T15:39:51.633 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:51.622+0000 7f9ffbfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9fe40039b0 0x7f9ffc1537f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:51.633 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:51.622+0000 7f9ffbfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9fe40039b0 0x7f9ffc1537f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57438/0 (socket says 172.21.15.16:57438) 2023-05-22T15:39:51.633 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:51.622+0000 7f9ffbfff640 1 -- 172.21.15.16:0/4130445787 learned_addr learned my addr 172.21.15.16:0/4130445787 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:39:51.633 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:51.622+0000 7fa000b28640 1 --2- 172.21.15.16:0/4130445787 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9ffc150fb0 0x7f9ffc151420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:51.633 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:51.622+0000 7f9ffb7fe640 1 --2- 172.21.15.16:0/4130445787 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9ffc14e490 0x7f9ffc14dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:51.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:51.622+0000 7f9ffbfff640 1 -- 172.21.15.16:0/4130445787 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9ffc150fb0 msgr2=0x7f9ffc151420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:51.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:51.622+0000 7f9ffbfff640 1 --2- 172.21.15.16:0/4130445787 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9ffc150fb0 0x7f9ffc151420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:51.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:51.622+0000 7f9ffbfff640 1 -- 172.21.15.16:0/4130445787 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9ffc14e490 msgr2=0x7f9ffc14dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:51.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:51.622+0000 7f9ffbfff640 1 --2- 172.21.15.16:0/4130445787 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9ffc14e490 0x7f9ffc14dcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:51.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:51.622+0000 7f9ffbfff640 1 -- 172.21.15.16:0/4130445787 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ffc070fe0 con 0x7f9fe40039b0 2023-05-22T15:39:51.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:51.622+0000 7f9ffbfff640 1 --2- 172.21.15.16:0/4130445787 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9fe40039b0 0x7f9ffc1537f0 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f9fec009ae0 tx=0x7f9fe4005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:39:51.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:51.622+0000 7f9ff8ff9640 1 -- 172.21.15.16:0/4130445787 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fec00f000 con 0x7f9fe40039b0 2023-05-22T15:39:51.635 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:51.622+0000 7f9ff8ff9640 1 -- 172.21.15.16:0/4130445787 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9fec0a6d80 con 0x7f9fe40039b0 2023-05-22T15:39:51.635 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:51.622+0000 7fa00258b640 1 -- 172.21.15.16:0/4130445787 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ffc150e20 con 0x7f9fe40039b0 2023-05-22T15:39:51.635 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:51.622+0000 7f9ff8ff9640 1 -- 172.21.15.16:0/4130445787 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fec0b9630 con 0x7f9fe40039b0 2023-05-22T15:39:51.635 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:51.622+0000 7fa00258b640 1 -- 172.21.15.16:0/4130445787 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ffc0cd9f0 con 0x7f9fe40039b0 2023-05-22T15:39:51.635 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:51.626+0000 7fa00258b640 1 -- 172.21.15.16:0/4130445787 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ffc0b0500 con 0x7f9fe40039b0 2023-05-22T15:39:51.636 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:51.626+0000 7f9ff8ff9640 1 -- 172.21.15.16:0/4130445787 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f9fec0a7410 con 0x7f9fe40039b0 2023-05-22T15:39:51.636 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:51.626+0000 7f9ff8ff9640 1 --2- 172.21.15.16:0/4130445787 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f9fe80654d0 0x7f9fe8067980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:51.636 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:51.626+0000 7f9ff8ff9640 1 -- 172.21.15.16:0/4130445787 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 7497+0+0 (secure 0 0 0) 0x7f9fec11caf0 con 0x7f9fe40039b0 2023-05-22T15:39:51.636 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:51.626+0000 7f9ffb7fe640 1 --2- 172.21.15.16:0/4130445787 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f9fe80654d0 0x7f9fe8067980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:51.637 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:51.626+0000 7f9ffb7fe640 1 --2- 172.21.15.16:0/4130445787 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f9fe80654d0 0x7f9fe8067980 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f9ff0001ca0 tx=0x7f9ff0009040 comp rx=0 tx=0).ready entity=mgr.6240 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:39:51.637 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:51.630+0000 7f9ff8ff9640 1 -- 172.21.15.16:0/4130445787 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f9ffc0b0500 con 0x7f9fe40039b0 2023-05-22T15:39:51.781 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-05-22T15:39:51.784 DEBUG:teuthology.orchestra.run.smithi062:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-05-22T15:39:51.792 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:51.786+0000 7fa00258b640 1 -- 172.21.15.16:0/4130445787 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f9ffc0b0500 con 0x7f9fe40039b0 2023-05-22T15:39:52.513 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:52.506+0000 7f9ff8ff9640 1 -- 172.21.15.16:0/4130445787 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v63) v1 ==== 181+0+0 (secure 0 0 0) 0x7f9ffc0b0500 con 0x7f9fe40039b0 2023-05-22T15:39:52.514 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'cephfs_metadata' removed 2023-05-22T15:39:52.517 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:52.510+0000 7fa00258b640 1 -- 172.21.15.16:0/4130445787 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f9fe80654d0 msgr2=0x7f9fe8067980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:52.517 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:52.510+0000 7fa00258b640 1 --2- 172.21.15.16:0/4130445787 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f9fe80654d0 0x7f9fe8067980 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f9ff0001ca0 tx=0x7f9ff0009040 comp rx=0 tx=0).stop 2023-05-22T15:39:52.517 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:52.510+0000 7fa00258b640 1 -- 172.21.15.16:0/4130445787 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9fe40039b0 msgr2=0x7f9ffc1537f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:52.518 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:52.510+0000 7fa00258b640 1 --2- 172.21.15.16:0/4130445787 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9fe40039b0 0x7f9ffc1537f0 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f9fec009ae0 tx=0x7f9fe4005800 comp rx=0 tx=0).stop 2023-05-22T15:39:52.518 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:52.510+0000 7fa00258b640 1 -- 172.21.15.16:0/4130445787 shutdown_connections 2023-05-22T15:39:52.518 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:52.510+0000 7fa00258b640 1 --2- 172.21.15.16:0/4130445787 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f9fe80654d0 0x7f9fe8067980 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:52.518 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:52.510+0000 7fa00258b640 1 --2- 172.21.15.16:0/4130445787 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9ffc150fb0 0x7f9ffc151420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:52.518 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:52.510+0000 7fa00258b640 1 --2- 172.21.15.16:0/4130445787 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9ffc14e490 0x7f9ffc14dcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:52.518 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:52.510+0000 7fa00258b640 1 --2- 172.21.15.16:0/4130445787 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9fe40039b0 0x7f9ffc1537f0 unknown :-1 s=CLOSED pgs=853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:52.518 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:52.510+0000 7fa00258b640 1 -- 172.21.15.16:0/4130445787 >> 172.21.15.16:0/4130445787 conn(0x7f9ffc069720 msgr2=0x7f9ffc0cb9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:39:52.519 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:52.510+0000 7fa00258b640 1 -- 172.21.15.16:0/4130445787 shutdown_connections 2023-05-22T15:39:52.519 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:52.510+0000 7fa00258b640 1 -- 172.21.15.16:0/4130445787 wait complete. 2023-05-22T15:39:52.541 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-05-22T15:39:52.675 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:52.670+0000 7fe4e5a36640 1 Processor -- start 2023-05-22T15:39:52.676 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:52.670+0000 7fe4e5a36640 1 -- start start 2023-05-22T15:39:52.677 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:52.670+0000 7fe4e5a36640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe4e00cc010 0x7fe4e00ccd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:52.677 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:52.670+0000 7fe4e5a36640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe4e00cb440 0x7fe4e00cd240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:52.677 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:52.670+0000 7fe4e5a36640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4e0070bb0 con 0x7fe4e0158950 2023-05-22T15:39:52.677 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:52.670+0000 7fe4e5a36640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4e0070e90 con 0x7fe4e00cb440 2023-05-22T15:39:52.677 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:52.670+0000 7fe4e5a36640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4e0071170 con 0x7fe4e00cc010 2023-05-22T15:39:52.677 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:52.670+0000 7fe4deffd640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fe4e0158950 0x7fe4e00cb900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:51262/0 (socket says 172.21.15.16:51262) 2023-05-22T15:39:52.678 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:52.670+0000 7fe4deffd640 1 -- 172.21.15.16:0/1541931275 learned_addr learned my addr 172.21.15.16:0/1541931275 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:39:52.678 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:52.670+0000 7fe4df7fe640 1 --2- 172.21.15.16:0/1541931275 >> v2:172.21.15.16:3300/0 conn(0x7fe4e00cb440 0x7fe4e00cd240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:52.678 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:52.670+0000 7fe4de7fc640 1 --2- 172.21.15.16:0/1541931275 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe4e00cc010 0x7fe4e00ccd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:52.678 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:52.670+0000 7fe4df7fe640 1 -- 172.21.15.16:0/1541931275 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe4e00cc010 msgr2=0x7fe4e00ccd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:39:52.678 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:52.670+0000 7fe4df7fe640 1 --2- 172.21.15.16:0/1541931275 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe4e00cc010 0x7fe4e00ccd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:52.678 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:52.670+0000 7fe4df7fe640 1 -- 172.21.15.16:0/1541931275 >> v1:172.21.15.16:6789/0 conn(0x7fe4e0158950 legacy=0x7fe4e00cb900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:52.678 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:52.670+0000 7fe4df7fe640 1 -- 172.21.15.16:0/1541931275 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4e0071450 con 0x7fe4e00cb440 2023-05-22T15:39:52.678 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:52.670+0000 7fe4ddffb640 1 -- 172.21.15.16:0/1541931275 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2852974641 0 0) 0x7fe4e0070bb0 con 0x7fe4e0158950 2023-05-22T15:39:52.680 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:52.674+0000 7fe4df7fe640 1 --2- 172.21.15.16:0/1541931275 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe4e00cb440 0x7fe4e00cd240 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7fe4d8009d90 tx=0x7fe4d801c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f51127859e328a9a server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:39:52.680 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:52.674+0000 7fe4ddffb640 1 -- 172.21.15.16:0/1541931275 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4d80a58f0 con 0x7fe4e00cb440 2023-05-22T15:39:52.680 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:52.674+0000 7fe4ddffb640 1 -- 172.21.15.16:0/1541931275 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4d80a5a50 con 0x7fe4e00cb440 2023-05-22T15:39:52.680 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:52.674+0000 7fe4ddffb640 1 -- 172.21.15.16:0/1541931275 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4d80a5de0 con 0x7fe4e00cb440 2023-05-22T15:39:52.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:52.674+0000 7fe4e5a36640 1 -- 172.21.15.16:0/1541931275 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe4e00cb440 msgr2=0x7fe4e00cd240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:52.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:52.674+0000 7fe4e5a36640 1 --2- 172.21.15.16:0/1541931275 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe4e00cb440 0x7fe4e00cd240 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7fe4d8009d90 tx=0x7fe4d801c920 comp rx=0 tx=0).stop 2023-05-22T15:39:52.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:52.674+0000 7fe4e5a36640 1 -- 172.21.15.16:0/1541931275 shutdown_connections 2023-05-22T15:39:52.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:52.674+0000 7fe4e5a36640 1 --2- 172.21.15.16:0/1541931275 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe4e00cb440 0x7fe4e00cd240 unknown :-1 s=CLOSED pgs=855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:52.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:52.674+0000 7fe4e5a36640 1 --2- 172.21.15.16:0/1541931275 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe4e00cc010 0x7fe4e00ccd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:52.683 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:52.674+0000 7fe4e5a36640 1 -- 172.21.15.16:0/1541931275 >> 172.21.15.16:0/1541931275 conn(0x7fe4e00b0cc0 msgr2=0x7fe4e00b10c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:39:52.683 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:52.674+0000 7fe4e5a36640 1 -- 172.21.15.16:0/1541931275 shutdown_connections 2023-05-22T15:39:52.683 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:52.674+0000 7fe4e5a36640 1 -- 172.21.15.16:0/1541931275 wait complete. 2023-05-22T15:39:52.683 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:52.674+0000 7fe4e5a36640 1 Processor -- start 2023-05-22T15:39:52.683 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:52.674+0000 7fe4e5a36640 1 -- start start 2023-05-22T15:39:52.684 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:52.678+0000 7fe4e5a36640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe4e00cb440 0x7fe4e00c6df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:52.685 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:52.678+0000 7fe4e5a36640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe4e00cc010 0x7fe4e00b2a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:52.685 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:52.678+0000 7fe4e5a36640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe4e0158950 0x7fe4e0153520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:52.685 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:52.678+0000 7fe4e5a36640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4e00b1fc0 con 0x7fe4e00cc010 2023-05-22T15:39:52.685 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:52.678+0000 7fe4e5a36640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4e00c8c80 con 0x7fe4e0158950 2023-05-22T15:39:52.685 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:52.678+0000 7fe4e5a36640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4e00c8f60 con 0x7fe4e00cb440 2023-05-22T15:39:52.685 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:52.678+0000 7fe4de7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe4e00cc010 0x7fe4e00b2a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:52.685 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:52.678+0000 7fe4de7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe4e00cc010 0x7fe4e00b2a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57452/0 (socket says 172.21.15.16:57452) 2023-05-22T15:39:52.686 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:52.678+0000 7fe4de7fc640 1 -- 172.21.15.16:0/2824674259 learned_addr learned my addr 172.21.15.16:0/2824674259 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:39:52.686 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:52.678+0000 7fe4df7fe640 1 --2- 172.21.15.16:0/2824674259 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe4e0158950 0x7fe4e0153520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:52.686 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:52.678+0000 7fe4deffd640 1 --2- 172.21.15.16:0/2824674259 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe4e00cb440 0x7fe4e00c6df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:52.686 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:52.678+0000 7fe4de7fc640 1 -- 172.21.15.16:0/2824674259 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe4e00cb440 msgr2=0x7fe4e00c6df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:52.686 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:52.678+0000 7fe4de7fc640 1 --2- 172.21.15.16:0/2824674259 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe4e00cb440 0x7fe4e00c6df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:52.686 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:52.678+0000 7fe4de7fc640 1 -- 172.21.15.16:0/2824674259 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe4e0158950 msgr2=0x7fe4e0153520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:52.686 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:52.678+0000 7fe4de7fc640 1 --2- 172.21.15.16:0/2824674259 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe4e0158950 0x7fe4e0153520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:52.687 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:52.678+0000 7fe4de7fc640 1 -- 172.21.15.16:0/2824674259 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4d8009a70 con 0x7fe4e00cc010 2023-05-22T15:39:52.687 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:52.678+0000 7fe4de7fc640 1 --2- 172.21.15.16:0/2824674259 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe4e00cc010 0x7fe4e00b2a10 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7fe4c800c7f0 tx=0x7fe4c800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:39:52.687 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:52.678+0000 7fe4d7fff640 1 -- 172.21.15.16:0/2824674259 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4c8004220 con 0x7fe4e00cc010 2023-05-22T15:39:52.687 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:52.678+0000 7fe4d7fff640 1 -- 172.21.15.16:0/2824674259 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4c8004380 con 0x7fe4e00cc010 2023-05-22T15:39:52.687 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:52.678+0000 7fe4d7fff640 1 -- 172.21.15.16:0/2824674259 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4c8003840 con 0x7fe4e00cc010 2023-05-22T15:39:52.687 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:52.678+0000 7fe4e5a36640 1 -- 172.21.15.16:0/2824674259 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4e0070980 con 0x7fe4e00cc010 2023-05-22T15:39:52.688 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:52.678+0000 7fe4e5a36640 1 -- 172.21.15.16:0/2824674259 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4e00707f0 con 0x7fe4e00cc010 2023-05-22T15:39:52.688 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:52.678+0000 7fe4e5a36640 1 -- 172.21.15.16:0/2824674259 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4e0165880 con 0x7fe4e00cc010 2023-05-22T15:39:52.688 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:52.678+0000 7fe4d7fff640 1 -- 172.21.15.16:0/2824674259 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) v1 ==== 84332+0+0 (secure 0 0 0) 0x7fe4c8003bb0 con 0x7fe4e00cc010 2023-05-22T15:39:52.689 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:52.678+0000 7fe4d7fff640 1 --2- 172.21.15.16:0/2824674259 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fe4c00654a0 0x7fe4c0067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:52.689 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:52.678+0000 7fe4d7fff640 1 -- 172.21.15.16:0/2824674259 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 6647+0+0 (secure 0 0 0) 0x7fe4c8082db0 con 0x7fe4e00cc010 2023-05-22T15:39:52.689 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:52.678+0000 7fe4deffd640 1 --2- 172.21.15.16:0/2824674259 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fe4c00654a0 0x7fe4c0067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:52.689 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:52.678+0000 7fe4deffd640 1 --2- 172.21.15.16:0/2824674259 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fe4c00654a0 0x7fe4c0067950 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fe4d0000f70 tx=0x7fe4d0009040 comp rx=0 tx=0).ready entity=mgr.6240 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:39:52.689 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:52.682+0000 7fe4d7fff640 1 -- 172.21.15.16:0/2824674259 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fe4c8087070 con 0x7fe4e00cc010 2023-05-22T15:39:52.853 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:52.846+0000 7fe4e5a36640 1 -- 172.21.15.16:0/2824674259 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fe4e00c9120 con 0x7fe4e00cc010 2023-05-22T15:39:53.530 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:53.522+0000 7fe4d7fff640 1 -- 172.21.15.16:0/2824674259 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v64) v1 ==== 169+0+0 (secure 0 0 0) 0x7fe4e00c9120 con 0x7fe4e00cc010 2023-05-22T15:39:53.530 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'cephfs_data' removed 2023-05-22T15:39:53.534 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:53.530+0000 7fe4e5a36640 1 -- 172.21.15.16:0/2824674259 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fe4c00654a0 msgr2=0x7fe4c0067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:53.535 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:53.530+0000 7fe4e5a36640 1 --2- 172.21.15.16:0/2824674259 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fe4c00654a0 0x7fe4c0067950 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fe4d0000f70 tx=0x7fe4d0009040 comp rx=0 tx=0).stop 2023-05-22T15:39:53.535 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:53.530+0000 7fe4e5a36640 1 -- 172.21.15.16:0/2824674259 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe4e00cc010 msgr2=0x7fe4e00b2a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:53.535 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:53.530+0000 7fe4e5a36640 1 --2- 172.21.15.16:0/2824674259 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe4e00cc010 0x7fe4e00b2a10 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7fe4c800c7f0 tx=0x7fe4c800ccb0 comp rx=0 tx=0).stop 2023-05-22T15:39:53.536 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:53.530+0000 7fe4e5a36640 1 -- 172.21.15.16:0/2824674259 shutdown_connections 2023-05-22T15:39:53.536 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:53.530+0000 7fe4e5a36640 1 --2- 172.21.15.16:0/2824674259 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe4e0158950 0x7fe4e0153520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:53.536 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:53.530+0000 7fe4e5a36640 1 --2- 172.21.15.16:0/2824674259 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe4e00cc010 0x7fe4e00b2a10 unknown :-1 s=CLOSED pgs=856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:53.536 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:53.530+0000 7fe4e5a36640 1 --2- 172.21.15.16:0/2824674259 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fe4c00654a0 0x7fe4c0067950 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:53.536 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:53.530+0000 7fe4e5a36640 1 --2- 172.21.15.16:0/2824674259 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe4e00cb440 0x7fe4e00c6df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:53.536 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:53.530+0000 7fe4e5a36640 1 -- 172.21.15.16:0/2824674259 >> 172.21.15.16:0/2824674259 conn(0x7fe4e00b0cc0 msgr2=0x7fe4e014f5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:39:53.537 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:53.530+0000 7fe4e5a36640 1 -- 172.21.15.16:0/2824674259 shutdown_connections 2023-05-22T15:39:53.538 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:53.530+0000 7fe4e5a36640 1 -- 172.21.15.16:0/2824674259 wait complete. 2023-05-22T15:39:53.564 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2023-05-22T15:39:53.692 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:53.686+0000 7f19e4b3b640 1 Processor -- start 2023-05-22T15:39:53.693 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:53.686+0000 7f19e4b3b640 1 -- start start 2023-05-22T15:39:53.693 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:53.686+0000 7f19e4b3b640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f19e006b6c0 0x7f19e00cb4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:53.694 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:53.686+0000 7f19e4b3b640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f19e006ba90 0x7f19e0158fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:53.694 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:53.686+0000 7f19e4b3b640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f19e0159510 0x7f19e00cc740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:53.694 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:53.686+0000 7f19e4b3b640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19e0071760 con 0x7f19e0159510 2023-05-22T15:39:53.694 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:53.686+0000 7f19e4b3b640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19e00ca7d0 con 0x7f19e006ba90 2023-05-22T15:39:53.694 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:53.686+0000 7f19e4b3b640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19e0159f70 con 0x7f19e006b6c0 2023-05-22T15:39:53.695 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:53.686+0000 7f19ded9d640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f19e0159510 0x7f19e00cc740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:53.695 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:53.690+0000 7f19ded9d640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f19e0159510 0x7f19e00cc740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57464/0 (socket says 172.21.15.16:57464) 2023-05-22T15:39:53.695 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:53.690+0000 7f19ded9d640 1 -- 172.21.15.16:0/63116471 learned_addr learned my addr 172.21.15.16:0/63116471 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:39:53.695 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:53.690+0000 7f19de59c640 1 --2- 172.21.15.16:0/63116471 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f19e006b6c0 0x7f19e00cb4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:53.695 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:53.690+0000 7f19ddd9b640 1 --2- 172.21.15.16:0/63116471 >> v2:172.21.15.62:3300/0 conn(0x7f19e006ba90 0x7f19e0158fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:53.696 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:53.690+0000 7f19ded9d640 1 -- 172.21.15.16:0/63116471 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f19e006b6c0 msgr2=0x7f19e00cb4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:39:53.696 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:53.690+0000 7f19ded9d640 1 --2- 172.21.15.16:0/63116471 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f19e006b6c0 0x7f19e00cb4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:53.696 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:53.690+0000 7f19ded9d640 1 -- 172.21.15.16:0/63116471 >> v2:172.21.15.62:3300/0 conn(0x7f19e006ba90 msgr2=0x7f19e0158fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:39:53.696 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:53.690+0000 7f19ded9d640 1 --2- 172.21.15.16:0/63116471 >> v2:172.21.15.62:3300/0 conn(0x7f19e006ba90 0x7f19e0158fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:53.696 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:53.690+0000 7f19ded9d640 1 -- 172.21.15.16:0/63116471 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19e00cb030 con 0x7f19e0159510 2023-05-22T15:39:53.697 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:53.690+0000 7f19ded9d640 1 --2- 172.21.15.16:0/63116471 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f19e0159510 0x7f19e00cc740 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f19e00cbb60 tx=0x7f19d801c6e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f021f504cbbdd server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:39:53.697 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:53.690+0000 7f19dd59a640 1 -- 172.21.15.16:0/63116471 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19d80a5560 con 0x7f19e0159510 2023-05-22T15:39:53.697 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:53.690+0000 7f19dd59a640 1 -- 172.21.15.16:0/63116471 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19d80a56c0 con 0x7f19e0159510 2023-05-22T15:39:53.697 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:53.690+0000 7f19dd59a640 1 -- 172.21.15.16:0/63116471 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19d80a5a50 con 0x7f19e0159510 2023-05-22T15:39:53.698 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:53.690+0000 7f19e4b3b640 1 -- 172.21.15.16:0/63116471 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f19e0159510 msgr2=0x7f19e00cc740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:53.698 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:53.690+0000 7f19e4b3b640 1 --2- 172.21.15.16:0/63116471 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f19e0159510 0x7f19e00cc740 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f19e00cbb60 tx=0x7f19d801c6e0 comp rx=0 tx=0).stop 2023-05-22T15:39:53.698 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:53.690+0000 7f19e4b3b640 1 -- 172.21.15.16:0/63116471 shutdown_connections 2023-05-22T15:39:53.698 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:53.690+0000 7f19e4b3b640 1 --2- 172.21.15.16:0/63116471 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f19e0159510 0x7f19e00cc740 unknown :-1 s=CLOSED pgs=857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:53.698 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:53.690+0000 7f19e4b3b640 1 --2- 172.21.15.16:0/63116471 >> v2:172.21.15.62:3300/0 conn(0x7f19e006ba90 0x7f19e0158fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:53.699 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:53.690+0000 7f19e4b3b640 1 --2- 172.21.15.16:0/63116471 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f19e006b6c0 0x7f19e00cb4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:53.699 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:53.690+0000 7f19e4b3b640 1 -- 172.21.15.16:0/63116471 >> 172.21.15.16:0/63116471 conn(0x7f19e0069720 msgr2=0x7f19e0069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:39:53.699 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:53.690+0000 7f19e4b3b640 1 -- 172.21.15.16:0/63116471 shutdown_connections 2023-05-22T15:39:53.699 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:53.690+0000 7f19e4b3b640 1 -- 172.21.15.16:0/63116471 wait complete. 2023-05-22T15:39:53.699 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:53.694+0000 7f19e4b3b640 1 Processor -- start 2023-05-22T15:39:53.700 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:53.694+0000 7f19e4b3b640 1 -- start start 2023-05-22T15:39:53.701 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:53.694+0000 7f19e4b3b640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f19e006b6c0 0x7f19e0147dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:53.701 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:53.694+0000 7f19e4b3b640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f19e006ba90 0x7f19e0156840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:53.701 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:53.694+0000 7f19e4b3b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f19e0159510 0x7f19e00b2fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:53.701 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:53.694+0000 7f19e4b3b640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19e00c22f0 con 0x7f19e0159510 2023-05-22T15:39:53.701 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:53.694+0000 7f19e4b3b640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19e0148300 con 0x7f19e006ba90 2023-05-22T15:39:53.702 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:53.694+0000 7f19e4b3b640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19e00b3d80 con 0x7f19e006b6c0 2023-05-22T15:39:53.702 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:53.694+0000 7f19ded9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f19e0159510 0x7f19e00b2fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:53.702 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:53.694+0000 7f19ded9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f19e0159510 0x7f19e00b2fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57476/0 (socket says 172.21.15.16:57476) 2023-05-22T15:39:53.702 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:53.694+0000 7f19ded9d640 1 -- 172.21.15.16:0/1921048310 learned_addr learned my addr 172.21.15.16:0/1921048310 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:39:53.702 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:53.694+0000 7f19de59c640 1 --2- 172.21.15.16:0/1921048310 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f19e006b6c0 0x7f19e0147dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:53.703 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:53.694+0000 7f19ddd9b640 1 --2- 172.21.15.16:0/1921048310 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f19e006ba90 0x7f19e0156840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:53.703 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:53.694+0000 7f19ded9d640 1 -- 172.21.15.16:0/1921048310 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f19e006b6c0 msgr2=0x7f19e0147dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:53.703 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:53.694+0000 7f19ded9d640 1 --2- 172.21.15.16:0/1921048310 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f19e006b6c0 0x7f19e0147dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:53.703 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:53.694+0000 7f19ded9d640 1 -- 172.21.15.16:0/1921048310 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f19e006ba90 msgr2=0x7f19e0156840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:53.703 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:53.694+0000 7f19ded9d640 1 --2- 172.21.15.16:0/1921048310 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f19e006ba90 0x7f19e0156840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:53.703 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:53.694+0000 7f19ded9d640 1 -- 172.21.15.16:0/1921048310 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19d8009b30 con 0x7f19e0159510 2023-05-22T15:39:53.703 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:53.694+0000 7f19ded9d640 1 --2- 172.21.15.16:0/1921048310 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f19e0159510 0x7f19e00b2fa0 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f19d80af040 tx=0x7f19d8019480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:39:53.704 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:53.694+0000 7f19d6ffd640 1 -- 172.21.15.16:0/1921048310 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19d80a5560 con 0x7f19e0159510 2023-05-22T15:39:53.705 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:53.694+0000 7f19d6ffd640 1 -- 172.21.15.16:0/1921048310 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19d80b8640 con 0x7f19e0159510 2023-05-22T15:39:53.705 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:53.694+0000 7f19e4b3b640 1 -- 172.21.15.16:0/1921048310 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19e0159990 con 0x7f19e0159510 2023-05-22T15:39:53.705 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:53.694+0000 7f19d6ffd640 1 -- 172.21.15.16:0/1921048310 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19d80b89d0 con 0x7f19e0159510 2023-05-22T15:39:53.705 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:53.694+0000 7f19e4b3b640 1 -- 172.21.15.16:0/1921048310 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19e00cacc0 con 0x7f19e0159510 2023-05-22T15:39:53.705 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:53.698+0000 7f19e4b3b640 1 -- 172.21.15.16:0/1921048310 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19e006be60 con 0x7f19e0159510 2023-05-22T15:39:53.705 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:53.698+0000 7f19d6ffd640 1 -- 172.21.15.16:0/1921048310 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f19d80b8b30 con 0x7f19e0159510 2023-05-22T15:39:53.705 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:53.698+0000 7f19d6ffd640 1 --2- 172.21.15.16:0/1921048310 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f19c00654d0 0x7f19c0067980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:53.706 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:53.698+0000 7f19d6ffd640 1 -- 172.21.15.16:0/1921048310 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 6175+0+0 (secure 0 0 0) 0x7f19d811d6f0 con 0x7f19e0159510 2023-05-22T15:39:53.706 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:53.698+0000 7f19de59c640 1 --2- 172.21.15.16:0/1921048310 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f19c00654d0 0x7f19c0067980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:53.706 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:53.698+0000 7f19de59c640 1 --2- 172.21.15.16:0/1921048310 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f19c00654d0 0x7f19c0067980 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f19c8009a70 tx=0x7f19c8008040 comp rx=0 tx=0).ready entity=mgr.6240 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:39:53.707 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:53.702+0000 7f19d6ffd640 1 -- 172.21.15.16:0/1921048310 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f19d80eb2a0 con 0x7f19e0159510 2023-05-22T15:39:53.863 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:53.858+0000 7f19e4b3b640 1 -- 172.21.15.16:0/1921048310 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7f19e00c2430 con 0x7f19e0159510 2023-05-22T15:39:53.873 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:53.866+0000 7f19d6ffd640 1 -- 172.21.15.16:0/1921048310 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v13) v1 ==== 120+0+0 (secure 0 0 0) 0x7f19e00c2430 con 0x7f19e0159510 2023-05-22T15:39:53.881 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:53.874+0000 7f19e4b3b640 1 -- 172.21.15.16:0/1921048310 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f19c00654d0 msgr2=0x7f19c0067980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:53.881 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:53.874+0000 7f19e4b3b640 1 --2- 172.21.15.16:0/1921048310 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f19c00654d0 0x7f19c0067980 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f19c8009a70 tx=0x7f19c8008040 comp rx=0 tx=0).stop 2023-05-22T15:39:53.882 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:53.874+0000 7f19e4b3b640 1 -- 172.21.15.16:0/1921048310 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f19e0159510 msgr2=0x7f19e00b2fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:53.882 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:53.874+0000 7f19e4b3b640 1 --2- 172.21.15.16:0/1921048310 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f19e0159510 0x7f19e00b2fa0 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f19d80af040 tx=0x7f19d8019480 comp rx=0 tx=0).stop 2023-05-22T15:39:53.882 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:53.874+0000 7f19e4b3b640 1 -- 172.21.15.16:0/1921048310 shutdown_connections 2023-05-22T15:39:53.882 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:53.874+0000 7f19e4b3b640 1 --2- 172.21.15.16:0/1921048310 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f19e0159510 0x7f19e00b2fa0 unknown :-1 s=CLOSED pgs=858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:53.882 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:53.874+0000 7f19e4b3b640 1 --2- 172.21.15.16:0/1921048310 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f19e006ba90 0x7f19e0156840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:53.883 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:53.874+0000 7f19e4b3b640 1 --2- 172.21.15.16:0/1921048310 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f19c00654d0 0x7f19c0067980 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:53.883 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:53.874+0000 7f19e4b3b640 1 --2- 172.21.15.16:0/1921048310 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f19e006b6c0 0x7f19e0147dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:53.883 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:53.874+0000 7f19e4b3b640 1 -- 172.21.15.16:0/1921048310 >> 172.21.15.16:0/1921048310 conn(0x7f19e0069720 msgr2=0x7f19e00ce010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:39:53.883 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:53.874+0000 7f19e4b3b640 1 -- 172.21.15.16:0/1921048310 shutdown_connections 2023-05-22T15:39:53.884 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:53.878+0000 7f19e4b3b640 1 -- 172.21.15.16:0/1921048310 wait complete. 2023-05-22T15:39:53.900 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2023-05-22T15:39:54.022 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.014+0000 7f9e22d0b640 1 Processor -- start 2023-05-22T15:39:54.022 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.018+0000 7f9e22d0b640 1 -- start start 2023-05-22T15:39:54.023 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.018+0000 7f9e22d0b640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f9e1c071530 0x7f9e1c15c580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:54.023 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.018+0000 7f9e22d0b640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e1c15d520 con 0x7f9e1c06b6c0 2023-05-22T15:39:54.023 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.018+0000 7f9e22d0b640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e1c0c8b10 con 0x7f9e1c070c70 2023-05-22T15:39:54.023 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.018+0000 7f9e22d0b640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e1c071980 con 0x7f9e1c071530 2023-05-22T15:39:54.024 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.018+0000 7f9e212a8640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f9e1c06b6c0 0x7f9e1c06bac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:51264/0 (socket says 172.21.15.16:51264) 2023-05-22T15:39:54.024 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.018+0000 7f9e212a8640 1 -- 172.21.15.16:0/2339311230 learned_addr learned my addr 172.21.15.16:0/2339311230 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:39:54.026 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.018+0000 7f9e1bfff640 1 --2- 172.21.15.16:0/2339311230 >> v2:172.21.15.62:3300/0 conn(0x7f9e1c071530 0x7f9e1c15c580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:54.026 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.018+0000 7f9e1b7fe640 1 -- 172.21.15.16:0/2339311230 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3829058185 0 0) 0x7f9e1c15d520 con 0x7f9e1c06b6c0 2023-05-22T15:39:54.026 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.018+0000 7f9e1b7fe640 1 -- 172.21.15.16:0/2339311230 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e04003530 con 0x7f9e1c06b6c0 2023-05-22T15:39:54.026 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.018+0000 7f9e1b7fe640 1 -- 172.21.15.16:0/2339311230 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2839306814 0 0) 0x7f9e04003530 con 0x7f9e1c06b6c0 2023-05-22T15:39:54.026 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.018+0000 7f9e1b7fe640 1 -- 172.21.15.16:0/2339311230 >> v2:172.21.15.62:3300/0 conn(0x7f9e1c071530 msgr2=0x7f9e1c15c580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:39:54.027 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.018+0000 7f9e1b7fe640 1 --2- 172.21.15.16:0/2339311230 >> v2:172.21.15.62:3300/0 conn(0x7f9e1c071530 0x7f9e1c15c580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:54.027 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.018+0000 7f9e1b7fe640 1 -- 172.21.15.16:0/2339311230 >> v1:172.21.15.62:6789/0 conn(0x7f9e1c070c70 legacy=0x7f9e1c06fba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:54.027 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.018+0000 7f9e1b7fe640 1 -- 172.21.15.16:0/2339311230 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e1c15cc80 con 0x7f9e1c06b6c0 2023-05-22T15:39:54.027 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.018+0000 7f9e1b7fe640 1 -- 172.21.15.16:0/2339311230 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7f9e14001a00 con 0x7f9e1c06b6c0 2023-05-22T15:39:54.027 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.018+0000 7f9e1b7fe640 1 -- 172.21.15.16:0/2339311230 >> v1:172.21.15.16:6789/0 conn(0x7f9e1c06b6c0 legacy=0x7f9e1c06bac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:54.027 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.018+0000 7f9e1b7fe640 1 --2- 172.21.15.16:0/2339311230 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9e04003980 0x7f9e04003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:54.028 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.018+0000 7f9e1b7fe640 1 --2- 172.21.15.16:0/2339311230 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e040047d0 0x7f9e04004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:54.028 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.018+0000 7f9e1b7fe640 1 --2- 172.21.15.16:0/2339311230 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9e04005180 0x7f9e0400b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:54.028 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.018+0000 7f9e1b7fe640 1 -- 172.21.15.16:0/2339311230 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e1c071980 con 0x7f9e040047d0 2023-05-22T15:39:54.028 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.018+0000 7f9e1b7fe640 1 -- 172.21.15.16:0/2339311230 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e0400c090 con 0x7f9e04005180 2023-05-22T15:39:54.028 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.018+0000 7f9e1b7fe640 1 -- 172.21.15.16:0/2339311230 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e0400c250 con 0x7f9e04003980 2023-05-22T15:39:54.028 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.018+0000 7f9e1bfff640 1 --2- 172.21.15.16:0/2339311230 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e040047d0 0x7f9e04004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:54.028 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.018+0000 7f9e20aa7640 1 --2- 172.21.15.16:0/2339311230 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9e04003980 0x7f9e04003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:54.029 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.018+0000 7f9e1bfff640 1 -- 172.21.15.16:0/2339311230 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9e04003980 msgr2=0x7f9e04003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:39:54.029 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.018+0000 7f9e1bfff640 1 --2- 172.21.15.16:0/2339311230 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9e04003980 0x7f9e04003d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:54.029 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.018+0000 7f9e1bfff640 1 -- 172.21.15.16:0/2339311230 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9e04005180 msgr2=0x7f9e0400b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:39:54.029 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.018+0000 7f9e212a8640 1 --2- 172.21.15.16:0/2339311230 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9e04005180 0x7f9e0400b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:54.029 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.018+0000 7f9e1bfff640 1 --2- 172.21.15.16:0/2339311230 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9e04005180 0x7f9e0400b990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:54.029 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.018+0000 7f9e1bfff640 1 -- 172.21.15.16:0/2339311230 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e04003530 con 0x7f9e040047d0 2023-05-22T15:39:54.029 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.018+0000 7f9e212a8640 1 -- 172.21.15.16:0/2339311230 reap_dead start 2023-05-22T15:39:54.033 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.022+0000 7f9e1bfff640 1 --2- 172.21.15.16:0/2339311230 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e040047d0 0x7f9e04004c40 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f9e10009ae0 tx=0x7f9e1001ca20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df41476df06f654a server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:39:54.033 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.022+0000 7f9e1b7fe640 1 -- 172.21.15.16:0/2339311230 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e100a55f0 con 0x7f9e040047d0 2023-05-22T15:39:54.033 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.022+0000 7f9e1b7fe640 1 -- 172.21.15.16:0/2339311230 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e100a5750 con 0x7f9e040047d0 2023-05-22T15:39:54.033 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.022+0000 7f9e1b7fe640 1 -- 172.21.15.16:0/2339311230 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e100a5b00 con 0x7f9e040047d0 2023-05-22T15:39:54.033 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.022+0000 7f9e22d0b640 1 -- 172.21.15.16:0/2339311230 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e040047d0 msgr2=0x7f9e04004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:54.033 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.022+0000 7f9e22d0b640 1 --2- 172.21.15.16:0/2339311230 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e040047d0 0x7f9e04004c40 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f9e10009ae0 tx=0x7f9e1001ca20 comp rx=0 tx=0).stop 2023-05-22T15:39:54.034 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.022+0000 7f9e22d0b640 1 -- 172.21.15.16:0/2339311230 shutdown_connections 2023-05-22T15:39:54.034 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.022+0000 7f9e22d0b640 1 --2- 172.21.15.16:0/2339311230 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e040047d0 0x7f9e04004c40 unknown :-1 s=CLOSED pgs=860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:54.034 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.022+0000 7f9e22d0b640 1 -- 172.21.15.16:0/2339311230 >> 172.21.15.16:0/2339311230 conn(0x7f9e1c069720 msgr2=0x7f9e1c069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:39:54.034 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.022+0000 7f9e22d0b640 1 -- 172.21.15.16:0/2339311230 shutdown_connections 2023-05-22T15:39:54.034 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.022+0000 7f9e22d0b640 1 -- 172.21.15.16:0/2339311230 wait complete. 2023-05-22T15:39:54.034 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.022+0000 7f9e22d0b640 1 Processor -- start 2023-05-22T15:39:54.034 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.022+0000 7f9e22d0b640 1 -- start start 2023-05-22T15:39:54.035 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.022+0000 7f9e22d0b640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9e040047d0 0x7f9e1c15a440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:54.035 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.022+0000 7f9e22d0b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e1c14ce10 0x7f9e1c15add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:54.035 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.022+0000 7f9e22d0b640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9e1c15b310 0x7f9e1c15b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:54.035 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.022+0000 7f9e22d0b640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e1c14d280 con 0x7f9e1c14ce10 2023-05-22T15:39:54.035 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.022+0000 7f9e22d0b640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e1c14d720 con 0x7f9e040047d0 2023-05-22T15:39:54.035 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.022+0000 7f9e22d0b640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e1c0c25f0 con 0x7f9e1c15b310 2023-05-22T15:39:54.035 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.022+0000 7f9e1bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e1c14ce10 0x7f9e1c15add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:54.036 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.022+0000 7f9e1bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e1c14ce10 0x7f9e1c15add0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57490/0 (socket says 172.21.15.16:57490) 2023-05-22T15:39:54.036 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.022+0000 7f9e1bfff640 1 -- 172.21.15.16:0/2367721537 learned_addr learned my addr 172.21.15.16:0/2367721537 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:39:54.036 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.026+0000 7f9e1bfff640 1 -- 172.21.15.16:0/2367721537 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9e1c15b310 msgr2=0x7f9e1c15b7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:54.036 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.026+0000 7f9e1bfff640 1 --2- 172.21.15.16:0/2367721537 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9e1c15b310 0x7f9e1c15b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:54.036 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.026+0000 7f9e1bfff640 1 -- 172.21.15.16:0/2367721537 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9e040047d0 msgr2=0x7f9e1c15a440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:54.036 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.026+0000 7f9e1bfff640 1 --2- 172.21.15.16:0/2367721537 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9e040047d0 0x7f9e1c15a440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:54.037 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.026+0000 7f9e1bfff640 1 -- 172.21.15.16:0/2367721537 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e1c071040 con 0x7f9e1c14ce10 2023-05-22T15:39:54.037 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.026+0000 7f9e1bfff640 1 --2- 172.21.15.16:0/2367721537 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e1c14ce10 0x7f9e1c15add0 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f9e1001c850 tx=0x7f9e100197c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:39:54.037 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.026+0000 7f9e197fa640 1 -- 172.21.15.16:0/2367721537 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e1000f000 con 0x7f9e1c14ce10 2023-05-22T15:39:54.037 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.026+0000 7f9e197fa640 1 -- 172.21.15.16:0/2367721537 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e10004020 con 0x7f9e1c14ce10 2023-05-22T15:39:54.037 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.026+0000 7f9e197fa640 1 -- 172.21.15.16:0/2367721537 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e100043d0 con 0x7f9e1c14ce10 2023-05-22T15:39:54.037 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.026+0000 7f9e22d0b640 1 -- 172.21.15.16:0/2367721537 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e1c0cc510 con 0x7f9e1c14ce10 2023-05-22T15:39:54.037 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.026+0000 7f9e22d0b640 1 -- 172.21.15.16:0/2367721537 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e1c15c990 con 0x7f9e1c14ce10 2023-05-22T15:39:54.038 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.026+0000 7f9e22d0b640 1 -- 172.21.15.16:0/2367721537 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e1c04dbe0 con 0x7f9e1c14ce10 2023-05-22T15:39:54.038 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.026+0000 7f9e197fa640 1 -- 172.21.15.16:0/2367721537 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f9e100b4070 con 0x7f9e1c14ce10 2023-05-22T15:39:54.038 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.026+0000 7f9e197fa640 1 --2- 172.21.15.16:0/2367721537 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f9e080654d0 0x7f9e08067980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:54.038 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.026+0000 7f9e197fa640 1 -- 172.21.15.16:0/2367721537 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 6175+0+0 (secure 0 0 0) 0x7f9e1011d080 con 0x7f9e1c14ce10 2023-05-22T15:39:54.038 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.026+0000 7f9e20aa7640 1 --2- 172.21.15.16:0/2367721537 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f9e080654d0 0x7f9e08067980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:54.038 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.026+0000 7f9e20aa7640 1 --2- 172.21.15.16:0/2367721537 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f9e080654d0 0x7f9e08067980 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f9e0c007fd0 tx=0x7f9e0c009040 comp rx=0 tx=0).ready entity=mgr.6240 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:39:54.039 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.030+0000 7f9e197fa640 1 -- 172.21.15.16:0/2367721537 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f9e100eab70 con 0x7f9e1c14ce10 2023-05-22T15:39:54.194 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.190+0000 7f9e22d0b640 1 -- 172.21.15.16:0/2367721537 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f9e1c0c9a70 con 0x7f9e1c14ce10 2023-05-22T15:39:54.204 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.198+0000 7f9e197fa640 1 -- 172.21.15.16:0/2367721537 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v14) v1 ==== 104+0+0 (secure 0 0 0) 0x7f9e100ee190 con 0x7f9e1c14ce10 2023-05-22T15:39:54.206 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.202+0000 7f9e22d0b640 1 -- 172.21.15.16:0/2367721537 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f9e080654d0 msgr2=0x7f9e08067980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:54.207 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.202+0000 7f9e22d0b640 1 --2- 172.21.15.16:0/2367721537 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f9e080654d0 0x7f9e08067980 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f9e0c007fd0 tx=0x7f9e0c009040 comp rx=0 tx=0).stop 2023-05-22T15:39:54.207 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.202+0000 7f9e22d0b640 1 -- 172.21.15.16:0/2367721537 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e1c14ce10 msgr2=0x7f9e1c15add0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:54.207 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.202+0000 7f9e22d0b640 1 --2- 172.21.15.16:0/2367721537 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e1c14ce10 0x7f9e1c15add0 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f9e1001c850 tx=0x7f9e100197c0 comp rx=0 tx=0).stop 2023-05-22T15:39:54.207 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.202+0000 7f9e22d0b640 1 -- 172.21.15.16:0/2367721537 shutdown_connections 2023-05-22T15:39:54.207 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.202+0000 7f9e22d0b640 1 --2- 172.21.15.16:0/2367721537 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f9e080654d0 0x7f9e08067980 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:54.207 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.202+0000 7f9e22d0b640 1 --2- 172.21.15.16:0/2367721537 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9e1c15b310 0x7f9e1c15b7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:54.208 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.202+0000 7f9e22d0b640 1 --2- 172.21.15.16:0/2367721537 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e1c14ce10 0x7f9e1c15add0 unknown :-1 s=CLOSED pgs=861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:54.208 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.202+0000 7f9e22d0b640 1 --2- 172.21.15.16:0/2367721537 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9e040047d0 0x7f9e1c15a440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:54.208 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.202+0000 7f9e22d0b640 1 -- 172.21.15.16:0/2367721537 >> 172.21.15.16:0/2367721537 conn(0x7f9e1c069720 msgr2=0x7f9e1c069af0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:39:54.208 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.202+0000 7f9e22d0b640 1 -- 172.21.15.16:0/2367721537 shutdown_connections 2023-05-22T15:39:54.208 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.202+0000 7f9e22d0b640 1 -- 172.21.15.16:0/2367721537 wait complete. 2023-05-22T15:39:54.224 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2023-05-22T15:39:54.345 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.338+0000 7f310fa89640 1 Processor -- start 2023-05-22T15:39:54.345 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.338+0000 7f310fa89640 1 -- start start 2023-05-22T15:39:54.346 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.342+0000 7f310fa89640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f31080d2480 0x7f31080d2900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:54.347 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.342+0000 7f310fa89640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f31080b5340 0x7f31080b5790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:54.347 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.342+0000 7f310fa89640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31080cccc0 con 0x7f310815bca0 2023-05-22T15:39:54.347 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.342+0000 7f310fa89640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31080cd660 con 0x7f31080d2480 2023-05-22T15:39:54.347 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.342+0000 7f310fa89640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31080ce190 con 0x7f31080b5340 2023-05-22T15:39:54.347 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.342+0000 7f310d825640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f310815bca0 0x7f31080cda80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:51276/0 (socket says 172.21.15.16:51276) 2023-05-22T15:39:54.347 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.342+0000 7f310d825640 1 -- 172.21.15.16:0/1031470350 learned_addr learned my addr 172.21.15.16:0/1031470350 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:39:54.347 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.342+0000 7f310e026640 1 --2- 172.21.15.16:0/1031470350 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f31080b5340 0x7f31080b5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:54.348 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.342+0000 7f310d024640 1 --2- 172.21.15.16:0/1031470350 >> v2:172.21.15.62:3300/0 conn(0x7f31080d2480 0x7f31080d2900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:54.348 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.342+0000 7f310c823640 1 -- 172.21.15.16:0/1031470350 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3164534600 0 0) 0x7f31080cccc0 con 0x7f310815bca0 2023-05-22T15:39:54.348 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.342+0000 7f310c823640 1 -- 172.21.15.16:0/1031470350 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f30f4003530 con 0x7f310815bca0 2023-05-22T15:39:54.351 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.342+0000 7f310c823640 1 -- 172.21.15.16:0/1031470350 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1305544045 0 0) 0x7f30f4003530 con 0x7f310815bca0 2023-05-22T15:39:54.351 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.342+0000 7f310c823640 1 -- 172.21.15.16:0/1031470350 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f31080b5340 msgr2=0x7f31080b5790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:39:54.352 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.342+0000 7f310c823640 1 --2- 172.21.15.16:0/1031470350 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f31080b5340 0x7f31080b5790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:54.352 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.342+0000 7f310c823640 1 -- 172.21.15.16:0/1031470350 >> v2:172.21.15.62:3300/0 conn(0x7f31080d2480 msgr2=0x7f31080d2900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:39:54.352 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.342+0000 7f310c823640 1 --2- 172.21.15.16:0/1031470350 >> v2:172.21.15.62:3300/0 conn(0x7f31080d2480 0x7f31080d2900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:54.352 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.342+0000 7f310c823640 1 -- 172.21.15.16:0/1031470350 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31080ccfa0 con 0x7f310815bca0 2023-05-22T15:39:54.422 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.342+0000 7f310c823640 1 -- 172.21.15.16:0/1031470350 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7f31040016a0 con 0x7f310815bca0 2023-05-22T15:39:54.423 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.342+0000 7f310c823640 1 -- 172.21.15.16:0/1031470350 >> v1:172.21.15.16:6789/0 conn(0x7f310815bca0 legacy=0x7f31080cda80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:54.423 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.342+0000 7f310c823640 1 --2- 172.21.15.16:0/1031470350 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f31080b5340 0x7f30f40039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:54.423 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.342+0000 7f310c823640 1 --2- 172.21.15.16:0/1031470350 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f30f4003f80 0x7f30f40047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:54.423 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.342+0000 7f310c823640 1 --2- 172.21.15.16:0/1031470350 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f30f4004d10 0x7f30f4005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:54.424 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.342+0000 7f310c823640 1 -- 172.21.15.16:0/1031470350 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31080cd660 con 0x7f30f4004d10 2023-05-22T15:39:54.424 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.342+0000 7f310c823640 1 -- 172.21.15.16:0/1031470350 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31080ce190 con 0x7f30f4003f80 2023-05-22T15:39:54.424 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.342+0000 7f310c823640 1 -- 172.21.15.16:0/1031470350 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30f400b990 con 0x7f31080b5340 2023-05-22T15:39:54.424 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.342+0000 7f310e026640 1 --2- 172.21.15.16:0/1031470350 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f30f4004d10 0x7f30f4005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:54.424 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.342+0000 7f310d024640 1 --2- 172.21.15.16:0/1031470350 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f30f4003f80 0x7f30f40047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:54.424 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.342+0000 7f310d825640 1 --2- 172.21.15.16:0/1031470350 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f31080b5340 0x7f30f40039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:54.425 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.342+0000 7f310e026640 1 -- 172.21.15.16:0/1031470350 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f31080b5340 msgr2=0x7f30f40039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:39:54.425 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.342+0000 7f310e026640 1 --2- 172.21.15.16:0/1031470350 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f31080b5340 0x7f30f40039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:54.425 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.342+0000 7f310e026640 1 -- 172.21.15.16:0/1031470350 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f30f4003f80 msgr2=0x7f30f40047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:39:54.425 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.342+0000 7f310e026640 1 --2- 172.21.15.16:0/1031470350 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f30f4003f80 0x7f30f40047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:54.425 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.342+0000 7f310e026640 1 -- 172.21.15.16:0/1031470350 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30f4003530 con 0x7f30f4004d10 2023-05-22T15:39:54.425 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.346+0000 7f310e026640 1 --2- 172.21.15.16:0/1031470350 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f30f4004d10 0x7f30f4005180 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f3100009fa0 tx=0x7f310001cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7cb15ce74dd11991 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:39:54.425 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.346+0000 7f310c823640 1 -- 172.21.15.16:0/1031470350 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31000a5cf0 con 0x7f30f4004d10 2023-05-22T15:39:54.426 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.346+0000 7f310c823640 1 -- 172.21.15.16:0/1031470350 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f310000f040 con 0x7f30f4004d10 2023-05-22T15:39:54.426 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.346+0000 7f310c823640 1 -- 172.21.15.16:0/1031470350 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31000ad660 con 0x7f30f4004d10 2023-05-22T15:39:54.426 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.346+0000 7f310fa89640 1 -- 172.21.15.16:0/1031470350 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f30f4004d10 msgr2=0x7f30f4005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:54.426 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.346+0000 7f310fa89640 1 --2- 172.21.15.16:0/1031470350 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f30f4004d10 0x7f30f4005180 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f3100009fa0 tx=0x7f310001cc20 comp rx=0 tx=0).stop 2023-05-22T15:39:54.426 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.346+0000 7f310e026640 1 -- 172.21.15.16:0/1031470350 reap_dead start 2023-05-22T15:39:54.426 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.346+0000 7f310fa89640 1 -- 172.21.15.16:0/1031470350 shutdown_connections 2023-05-22T15:39:54.427 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.346+0000 7f310fa89640 1 -- 172.21.15.16:0/1031470350 >> 172.21.15.16:0/1031470350 conn(0x7f31080655c0 msgr2=0x7f3108070e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:39:54.427 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.346+0000 7f310fa89640 1 -- 172.21.15.16:0/1031470350 shutdown_connections 2023-05-22T15:39:54.427 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.346+0000 7f310fa89640 1 -- 172.21.15.16:0/1031470350 wait complete. 2023-05-22T15:39:54.430 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.346+0000 7f310fa89640 1 Processor -- start 2023-05-22T15:39:54.430 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.346+0000 7f310fa89640 1 -- start start 2023-05-22T15:39:54.430 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.346+0000 7f310fa89640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3108155d60 0x7f310815dca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:54.430 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.346+0000 7f310fa89640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f310815c200 0x7f31081575f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:54.430 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.346+0000 7f310fa89640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3108157b30 0x7f31081f4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:54.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.346+0000 7f310fa89640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3108157fa0 con 0x7f310815c200 2023-05-22T15:39:54.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.346+0000 7f310fa89640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31080c9740 con 0x7f3108155d60 2023-05-22T15:39:54.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.346+0000 7f310fa89640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f310815c9e0 con 0x7f3108157b30 2023-05-22T15:39:54.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.346+0000 7f310d024640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f310815c200 0x7f31081575f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:54.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.346+0000 7f310d024640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f310815c200 0x7f31081575f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57502/0 (socket says 172.21.15.16:57502) 2023-05-22T15:39:54.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.346+0000 7f310d024640 1 -- 172.21.15.16:0/742869982 learned_addr learned my addr 172.21.15.16:0/742869982 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:39:54.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.346+0000 7f310d825640 1 --2- 172.21.15.16:0/742869982 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3108155d60 0x7f310815dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:54.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.346+0000 7f310e026640 1 --2- 172.21.15.16:0/742869982 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3108157b30 0x7f31081f4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:54.432 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.346+0000 7f310d024640 1 -- 172.21.15.16:0/742869982 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3108157b30 msgr2=0x7f31081f4f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:54.432 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.346+0000 7f310d024640 1 --2- 172.21.15.16:0/742869982 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3108157b30 0x7f31081f4f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:54.432 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.346+0000 7f310d024640 1 -- 172.21.15.16:0/742869982 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3108155d60 msgr2=0x7f310815dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:54.432 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.346+0000 7f310d024640 1 --2- 172.21.15.16:0/742869982 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3108155d60 0x7f310815dca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:54.432 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.346+0000 7f310d024640 1 -- 172.21.15.16:0/742869982 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f310815c070 con 0x7f310815c200 2023-05-22T15:39:54.432 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.346+0000 7f310d825640 1 --2- 172.21.15.16:0/742869982 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3108155d60 0x7f310815dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T15:39:54.433 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.350+0000 7f310d024640 1 --2- 172.21.15.16:0/742869982 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f310815c200 0x7f31081575f0 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f30f80115e0 tx=0x7f30f8011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:39:54.433 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.350+0000 7f30fe7fc640 1 -- 172.21.15.16:0/742869982 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30f800f8f0 con 0x7f310815c200 2023-05-22T15:39:54.433 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.350+0000 7f30fe7fc640 1 -- 172.21.15.16:0/742869982 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30f800fa50 con 0x7f310815c200 2023-05-22T15:39:54.433 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.350+0000 7f30fe7fc640 1 -- 172.21.15.16:0/742869982 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30f800fdc0 con 0x7f310815c200 2023-05-22T15:39:54.433 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.350+0000 7f310fa89640 1 -- 172.21.15.16:0/742869982 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31080cd280 con 0x7f310815c200 2023-05-22T15:39:54.434 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.350+0000 7f310fa89640 1 -- 172.21.15.16:0/742869982 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31080b5790 con 0x7f310815c200 2023-05-22T15:39:54.434 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.350+0000 7f310fa89640 1 -- 172.21.15.16:0/742869982 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31080c9900 con 0x7f310815c200 2023-05-22T15:39:54.434 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.350+0000 7f30fe7fc640 1 -- 172.21.15.16:0/742869982 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f30f8015460 con 0x7f310815c200 2023-05-22T15:39:54.434 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.350+0000 7f30fe7fc640 1 --2- 172.21.15.16:0/742869982 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f30ec0654a0 0x7f30ec067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:54.434 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.350+0000 7f30fe7fc640 1 -- 172.21.15.16:0/742869982 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 6175+0+0 (secure 0 0 0) 0x7f30f8082f80 con 0x7f310815c200 2023-05-22T15:39:54.434 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.350+0000 7f310d825640 1 --2- 172.21.15.16:0/742869982 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f30ec0654a0 0x7f30ec067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:54.434 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.350+0000 7f310d825640 1 --2- 172.21.15.16:0/742869982 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f30ec0654a0 0x7f30ec067950 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f3104001ec0 tx=0x7f3104009040 comp rx=0 tx=0).ready entity=mgr.6240 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:39:54.435 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.354+0000 7f30fe7fc640 1 -- 172.21.15.16:0/742869982 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f30f80b3620 con 0x7f310815c200 2023-05-22T15:39:54.514 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.510+0000 7f310fa89640 1 -- 172.21.15.16:0/742869982 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7f310804dbe0 con 0x7f310815c200 2023-05-22T15:39:54.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.518+0000 7f30fe7fc640 1 -- 172.21.15.16:0/742869982 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v15) v1 ==== 115+0+0 (secure 0 0 0) 0x7f30f8050a20 con 0x7f310815c200 2023-05-22T15:39:54.527 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.522+0000 7f310fa89640 1 -- 172.21.15.16:0/742869982 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f30ec0654a0 msgr2=0x7f30ec067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:54.527 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.522+0000 7f310fa89640 1 --2- 172.21.15.16:0/742869982 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f30ec0654a0 0x7f30ec067950 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f3104001ec0 tx=0x7f3104009040 comp rx=0 tx=0).stop 2023-05-22T15:39:54.527 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.522+0000 7f310fa89640 1 -- 172.21.15.16:0/742869982 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f310815c200 msgr2=0x7f31081575f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:54.527 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.522+0000 7f310fa89640 1 --2- 172.21.15.16:0/742869982 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f310815c200 0x7f31081575f0 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f30f80115e0 tx=0x7f30f8011aa0 comp rx=0 tx=0).stop 2023-05-22T15:39:54.527 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.522+0000 7f310fa89640 1 -- 172.21.15.16:0/742869982 shutdown_connections 2023-05-22T15:39:54.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.522+0000 7f310fa89640 1 --2- 172.21.15.16:0/742869982 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f30ec0654a0 0x7f30ec067950 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:54.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.522+0000 7f310fa89640 1 --2- 172.21.15.16:0/742869982 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3108157b30 0x7f31081f4f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:54.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.522+0000 7f310fa89640 1 --2- 172.21.15.16:0/742869982 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f310815c200 0x7f31081575f0 unknown :-1 s=CLOSED pgs=864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:54.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.522+0000 7f310fa89640 1 --2- 172.21.15.16:0/742869982 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3108155d60 0x7f310815dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:54.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.522+0000 7f310fa89640 1 -- 172.21.15.16:0/742869982 >> 172.21.15.16:0/742869982 conn(0x7f31080655c0 msgr2=0x7f31080b1f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:39:54.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.522+0000 7f310fa89640 1 -- 172.21.15.16:0/742869982 shutdown_connections 2023-05-22T15:39:54.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.522+0000 7f310fa89640 1 -- 172.21.15.16:0/742869982 wait complete. 2023-05-22T15:39:54.547 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_concurrent_snap_creates' 2023-05-22T15:39:54.662 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.658+0000 7f30400ac640 1 Processor -- start 2023-05-22T15:39:54.662 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.658+0000 7f30400ac640 1 -- start start 2023-05-22T15:39:54.663 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.658+0000 7f30400ac640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f3038070f10 0x7f30380712e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:54.663 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.658+0000 7f30400ac640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30380cedd0 con 0x7f303815e470 2023-05-22T15:39:54.663 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.658+0000 7f30400ac640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3038071820 con 0x7f30380ca080 2023-05-22T15:39:54.663 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.658+0000 7f30400ac640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f303815a070 con 0x7f3038070f10 2023-05-22T15:39:54.665 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.658+0000 7f303e649640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f303815e470 0x7f30380ce6c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:51288/0 (socket says 172.21.15.16:51288) 2023-05-22T15:39:54.665 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.658+0000 7f303e649640 1 -- 172.21.15.16:0/1512116142 learned_addr learned my addr 172.21.15.16:0/1512116142 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:39:54.665 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.658+0000 7f303de48640 1 --2- 172.21.15.16:0/1512116142 >> v2:172.21.15.62:3300/0 conn(0x7f3038070f10 0x7f30380712e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:54.665 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.658+0000 7f303ce46640 1 -- 172.21.15.16:0/1512116142 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1147365840 0 0) 0x7f30380cedd0 con 0x7f303815e470 2023-05-22T15:39:54.665 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.658+0000 7f303ce46640 1 -- 172.21.15.16:0/1512116142 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3024003530 con 0x7f303815e470 2023-05-22T15:39:54.666 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.658+0000 7f303ce46640 1 -- 172.21.15.16:0/1512116142 <== mon.1 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1173376391 0 0) 0x7f3038071820 con 0x7f30380ca080 2023-05-22T15:39:54.666 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.658+0000 7f303ce46640 1 -- 172.21.15.16:0/1512116142 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f30380cedd0 con 0x7f30380ca080 2023-05-22T15:39:54.666 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.658+0000 7f303ce46640 1 -- 172.21.15.16:0/1512116142 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3074462389 0 0) 0x7f3024003530 con 0x7f303815e470 2023-05-22T15:39:54.666 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.658+0000 7f303ce46640 1 -- 172.21.15.16:0/1512116142 >> v2:172.21.15.62:3300/0 conn(0x7f3038070f10 msgr2=0x7f30380712e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:39:54.666 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.658+0000 7f303ce46640 1 --2- 172.21.15.16:0/1512116142 >> v2:172.21.15.62:3300/0 conn(0x7f3038070f10 0x7f30380712e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:54.666 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.658+0000 7f303ce46640 1 -- 172.21.15.16:0/1512116142 >> v1:172.21.15.62:6789/0 conn(0x7f30380ca080 legacy=0x7f30380cdfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:54.666 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.658+0000 7f303ce46640 1 -- 172.21.15.16:0/1512116142 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30380ca580 con 0x7f303815e470 2023-05-22T15:39:54.666 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.658+0000 7f303ce46640 1 -- 172.21.15.16:0/1512116142 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7f3030001a00 con 0x7f303815e470 2023-05-22T15:39:54.667 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.658+0000 7f303ce46640 1 -- 172.21.15.16:0/1512116142 >> v1:172.21.15.16:6789/0 conn(0x7f303815e470 legacy=0x7f30380ce6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:54.667 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.658+0000 7f303ce46640 1 --2- 172.21.15.16:0/1512116142 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3024003b10 0x7f3024003f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:54.667 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.658+0000 7f303ce46640 1 --2- 172.21.15.16:0/1512116142 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f30240047d0 0x7f3024004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:54.667 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.658+0000 7f303ce46640 1 --2- 172.21.15.16:0/1512116142 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3024005180 0x7f302400b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:54.667 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.658+0000 7f303ce46640 1 -- 172.21.15.16:0/1512116142 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f303815a070 con 0x7f3024003b10 2023-05-22T15:39:54.667 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.658+0000 7f303ce46640 1 -- 172.21.15.16:0/1512116142 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f302400bfe0 con 0x7f3024005180 2023-05-22T15:39:54.667 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.658+0000 7f303ce46640 1 -- 172.21.15.16:0/1512116142 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f302400c1a0 con 0x7f30240047d0 2023-05-22T15:39:54.668 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.658+0000 7f303de48640 1 --2- 172.21.15.16:0/1512116142 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3024003b10 0x7f3024003f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:54.668 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.658+0000 7f303e649640 1 --2- 172.21.15.16:0/1512116142 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3024005180 0x7f302400b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:54.668 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.658+0000 7f303d647640 1 --2- 172.21.15.16:0/1512116142 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f30240047d0 0x7f3024004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:54.668 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.658+0000 7f303de48640 1 -- 172.21.15.16:0/1512116142 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f30240047d0 msgr2=0x7f3024004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:39:54.668 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.658+0000 7f303de48640 1 --2- 172.21.15.16:0/1512116142 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f30240047d0 0x7f3024004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:54.668 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.658+0000 7f303de48640 1 -- 172.21.15.16:0/1512116142 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3024005180 msgr2=0x7f302400b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:39:54.668 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.658+0000 7f303de48640 1 --2- 172.21.15.16:0/1512116142 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3024005180 0x7f302400b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:54.668 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.658+0000 7f303e649640 1 -- 172.21.15.16:0/1512116142 reap_dead start 2023-05-22T15:39:54.669 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.658+0000 7f303de48640 1 -- 172.21.15.16:0/1512116142 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3024003530 con 0x7f3024003b10 2023-05-22T15:39:54.671 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.662+0000 7f303de48640 1 --2- 172.21.15.16:0/1512116142 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3024003b10 0x7f3024003f00 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f3034009ae0 tx=0x7f303401ca20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eacb3a8d5db91861 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:39:54.671 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.662+0000 7f303ce46640 1 -- 172.21.15.16:0/1512116142 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30340a5510 con 0x7f3024003b10 2023-05-22T15:39:54.671 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.662+0000 7f303ce46640 1 -- 172.21.15.16:0/1512116142 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30340a5670 con 0x7f3024003b10 2023-05-22T15:39:54.671 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.662+0000 7f303ce46640 1 -- 172.21.15.16:0/1512116142 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30340a5a20 con 0x7f3024003b10 2023-05-22T15:39:54.671 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.662+0000 7f30400ac640 1 -- 172.21.15.16:0/1512116142 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3024003b10 msgr2=0x7f3024003f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:54.672 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.662+0000 7f30400ac640 1 --2- 172.21.15.16:0/1512116142 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3024003b10 0x7f3024003f00 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f3034009ae0 tx=0x7f303401ca20 comp rx=0 tx=0).stop 2023-05-22T15:39:54.672 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.662+0000 7f30400ac640 1 -- 172.21.15.16:0/1512116142 shutdown_connections 2023-05-22T15:39:54.672 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.662+0000 7f30400ac640 1 --2- 172.21.15.16:0/1512116142 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3024003b10 0x7f3024003f00 unknown :-1 s=CLOSED pgs=866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:54.672 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.662+0000 7f30400ac640 1 -- 172.21.15.16:0/1512116142 >> 172.21.15.16:0/1512116142 conn(0x7f3038069720 msgr2=0x7f3038069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:39:54.672 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.662+0000 7f30400ac640 1 -- 172.21.15.16:0/1512116142 shutdown_connections 2023-05-22T15:39:54.672 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.662+0000 7f30400ac640 1 -- 172.21.15.16:0/1512116142 wait complete. 2023-05-22T15:39:54.672 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.662+0000 7f30400ac640 1 Processor -- start 2023-05-22T15:39:54.672 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.662+0000 7f30400ac640 1 -- start start 2023-05-22T15:39:54.673 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.662+0000 7f30400ac640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3024003b10 0x7f30380c33c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:54.673 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.662+0000 7f30400ac640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f303814c070 0x7f30380c4300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:54.673 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.662+0000 7f30400ac640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f303814b8a0 0x7f30380b2d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:54.673 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.662+0000 7f30400ac640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30380c4840 con 0x7f303814b8a0 2023-05-22T15:39:54.673 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.662+0000 7f30400ac640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30380c2f30 con 0x7f303814c070 2023-05-22T15:39:54.673 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.662+0000 7f30400ac640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30380c4a50 con 0x7f3024003b10 2023-05-22T15:39:54.673 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.662+0000 7f303e649640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f303814b8a0 0x7f30380b2d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:54.674 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.662+0000 7f303e649640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f303814b8a0 0x7f30380b2d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57530/0 (socket says 172.21.15.16:57530) 2023-05-22T15:39:54.674 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.662+0000 7f303e649640 1 -- 172.21.15.16:0/2445100577 learned_addr learned my addr 172.21.15.16:0/2445100577 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:39:54.674 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.666+0000 7f303d647640 1 --2- 172.21.15.16:0/2445100577 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f303814c070 0x7f30380c4300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:54.674 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.666+0000 7f303de48640 1 --2- 172.21.15.16:0/2445100577 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3024003b10 0x7f30380c33c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:54.674 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.666+0000 7f303e649640 1 -- 172.21.15.16:0/2445100577 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3024003b10 msgr2=0x7f30380c33c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:54.674 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.666+0000 7f303e649640 1 --2- 172.21.15.16:0/2445100577 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3024003b10 0x7f30380c33c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:54.674 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.666+0000 7f303e649640 1 -- 172.21.15.16:0/2445100577 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f303814c070 msgr2=0x7f30380c4300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:54.674 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.666+0000 7f303e649640 1 --2- 172.21.15.16:0/2445100577 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f303814c070 0x7f30380c4300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:54.675 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.666+0000 7f303e649640 1 -- 172.21.15.16:0/2445100577 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30380712e0 con 0x7f303814b8a0 2023-05-22T15:39:54.675 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.666+0000 7f303e649640 1 --2- 172.21.15.16:0/2445100577 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f303814b8a0 0x7f30380b2d50 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f30380cb730 tx=0x7f3030007aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:39:54.675 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.666+0000 7f302e7fc640 1 -- 172.21.15.16:0/2445100577 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3030010510 con 0x7f303814b8a0 2023-05-22T15:39:54.675 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.666+0000 7f302e7fc640 1 -- 172.21.15.16:0/2445100577 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3030010670 con 0x7f303814b8a0 2023-05-22T15:39:54.675 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.666+0000 7f30400ac640 1 -- 172.21.15.16:0/2445100577 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f303815df50 con 0x7f303814b8a0 2023-05-22T15:39:54.675 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.666+0000 7f302e7fc640 1 -- 172.21.15.16:0/2445100577 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3030010a20 con 0x7f303814b8a0 2023-05-22T15:39:54.675 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.666+0000 7f30400ac640 1 -- 172.21.15.16:0/2445100577 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3038071470 con 0x7f303814b8a0 2023-05-22T15:39:54.676 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.666+0000 7f30400ac640 1 -- 172.21.15.16:0/2445100577 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30380c30c0 con 0x7f303814b8a0 2023-05-22T15:39:54.676 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.666+0000 7f302e7fc640 1 -- 172.21.15.16:0/2445100577 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f3030010b80 con 0x7f303814b8a0 2023-05-22T15:39:54.677 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.666+0000 7f302e7fc640 1 --2- 172.21.15.16:0/2445100577 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f301c0654d0 0x7f301c067980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:54.677 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.666+0000 7f302e7fc640 1 -- 172.21.15.16:0/2445100577 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 6175+0+0 (secure 0 0 0) 0x7f3030082120 con 0x7f303814b8a0 2023-05-22T15:39:54.677 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.666+0000 7f303de48640 1 --2- 172.21.15.16:0/2445100577 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f301c0654d0 0x7f301c067980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:54.677 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.666+0000 7f303de48640 1 --2- 172.21.15.16:0/2445100577 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f301c0654d0 0x7f301c067980 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f30340af000 tx=0x7f3034008040 comp rx=0 tx=0).ready entity=mgr.6240 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:39:54.677 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.670+0000 7f302e7fc640 1 -- 172.21.15.16:0/2445100577 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f303004fc70 con 0x7f303814b8a0 2023-05-22T15:39:54.885 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:54.878+0000 7f30400ac640 1 -- 172.21.15.16:0/2445100577 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_concurrent_snap_creates"]} v 0) v1 -- 0x7f303804dbe0 con 0x7f303814b8a0 2023-05-22T15:39:55.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:55.538+0000 7f302e7fc640 1 -- 172.21.15.16:0/2445100577 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_concurrent_snap_creates"]}]=0 v479) v1 ==== 158+0+0 (secure 0 0 0) 0x7f303001a0e0 con 0x7f303814b8a0 2023-05-22T15:39:55.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:55.542+0000 7f30400ac640 1 -- 172.21.15.16:0/2445100577 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f301c0654d0 msgr2=0x7f301c067980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:55.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:55.542+0000 7f30400ac640 1 --2- 172.21.15.16:0/2445100577 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f301c0654d0 0x7f301c067980 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f30340af000 tx=0x7f3034008040 comp rx=0 tx=0).stop 2023-05-22T15:39:55.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:55.542+0000 7f30400ac640 1 -- 172.21.15.16:0/2445100577 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f303814b8a0 msgr2=0x7f30380b2d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:55.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:55.542+0000 7f30400ac640 1 --2- 172.21.15.16:0/2445100577 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f303814b8a0 0x7f30380b2d50 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f30380cb730 tx=0x7f3030007aa0 comp rx=0 tx=0).stop 2023-05-22T15:39:55.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:55.542+0000 7f30400ac640 1 -- 172.21.15.16:0/2445100577 shutdown_connections 2023-05-22T15:39:55.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:55.542+0000 7f30400ac640 1 --2- 172.21.15.16:0/2445100577 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f303814b8a0 0x7f30380b2d50 unknown :-1 s=CLOSED pgs=867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:55.549 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:55.542+0000 7f30400ac640 1 --2- 172.21.15.16:0/2445100577 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f303814c070 0x7f30380c4300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:55.549 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:55.542+0000 7f30400ac640 1 --2- 172.21.15.16:0/2445100577 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f301c0654d0 0x7f301c067980 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:55.549 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:55.542+0000 7f30400ac640 1 --2- 172.21.15.16:0/2445100577 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3024003b10 0x7f30380c33c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:55.549 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:55.542+0000 7f30400ac640 1 -- 172.21.15.16:0/2445100577 >> 172.21.15.16:0/2445100577 conn(0x7f3038069720 msgr2=0x7f303815b780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:39:55.549 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:55.542+0000 7f30400ac640 1 -- 172.21.15.16:0/2445100577 shutdown_connections 2023-05-22T15:39:55.550 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:55.542+0000 7f30400ac640 1 -- 172.21.15.16:0/2445100577 wait complete. 2023-05-22T15:39:55.572 INFO:tasks.cephfs_test_runner:Test concurrent snap creates in same file-system without db issues ... ok 2023-05-22T15:39:55.572 INFO:tasks.cephfs_test_runner:Starting test: test_multi_snap_schedule (tasks.cephfs.test_snap_schedules.TestSnapSchedules) Test exisitence of multiple scheduled snapshots 2023-05-22T15:39:55.573 INFO:tasks.cephfs_test_runner:test_multi_snap_schedule (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2023-05-22T15:39:55.573 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_multi_snap_schedule' 2023-05-22T15:39:55.702 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:55.698+0000 7fd18051e640 1 Processor -- start 2023-05-22T15:39:55.703 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:55.698+0000 7fd18051e640 1 -- start start 2023-05-22T15:39:55.703 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:55.698+0000 7fd18051e640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fd178071500 0x7fd17815e460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:55.704 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:55.698+0000 7fd18051e640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1780b0650 con 0x7fd1780c9b40 2023-05-22T15:39:55.704 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:55.698+0000 7fd18051e640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1780b0370 con 0x7fd1780ca4e0 2023-05-22T15:39:55.704 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:55.698+0000 7fd18051e640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd17815d980 con 0x7fd178071500 2023-05-22T15:39:55.704 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:55.698+0000 7fd17dab9640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fd1780c9b40 0x7fd178070df0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:51304/0 (socket says 172.21.15.16:51304) 2023-05-22T15:39:55.704 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:55.698+0000 7fd17dab9640 1 -- 172.21.15.16:0/3446174981 learned_addr learned my addr 172.21.15.16:0/3446174981 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:39:55.706 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:55.698+0000 7fd17eabb640 1 --2- 172.21.15.16:0/3446174981 >> v2:172.21.15.62:3300/0 conn(0x7fd178071500 0x7fd17815e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:55.706 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:55.698+0000 7fd17d2b8640 1 -- 172.21.15.16:0/3446174981 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3375462467 0 0) 0x7fd1780b0650 con 0x7fd1780c9b40 2023-05-22T15:39:55.706 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:55.698+0000 7fd17d2b8640 1 -- 172.21.15.16:0/3446174981 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd164003530 con 0x7fd1780c9b40 2023-05-22T15:39:55.706 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:55.698+0000 7fd17d2b8640 1 -- 172.21.15.16:0/3446174981 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4137969421 0 0) 0x7fd164003530 con 0x7fd1780c9b40 2023-05-22T15:39:55.706 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:55.698+0000 7fd17d2b8640 1 -- 172.21.15.16:0/3446174981 >> v2:172.21.15.62:3300/0 conn(0x7fd178071500 msgr2=0x7fd17815e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:39:55.706 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:55.698+0000 7fd17d2b8640 1 --2- 172.21.15.16:0/3446174981 >> v2:172.21.15.62:3300/0 conn(0x7fd178071500 0x7fd17815e460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:55.707 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:55.698+0000 7fd17d2b8640 1 -- 172.21.15.16:0/3446174981 >> v1:172.21.15.62:6789/0 conn(0x7fd1780ca4e0 legacy=0x7fd1780afb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:55.707 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:55.698+0000 7fd17d2b8640 1 -- 172.21.15.16:0/3446174981 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd17815dc60 con 0x7fd1780c9b40 2023-05-22T15:39:55.707 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:55.698+0000 7fd17d2b8640 1 -- 172.21.15.16:0/3446174981 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7fd168001680 con 0x7fd1780c9b40 2023-05-22T15:39:55.707 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:55.698+0000 7fd17d2b8640 1 -- 172.21.15.16:0/3446174981 >> v1:172.21.15.16:6789/0 conn(0x7fd1780c9b40 legacy=0x7fd178070df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:55.707 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:55.698+0000 7fd17d2b8640 1 --2- 172.21.15.16:0/3446174981 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd164003980 0x7fd164003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:55.707 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:55.698+0000 7fd17d2b8640 1 --2- 172.21.15.16:0/3446174981 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd1640047d0 0x7fd164004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:55.707 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:55.698+0000 7fd17d2b8640 1 --2- 172.21.15.16:0/3446174981 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd164005180 0x7fd16400b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:55.707 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:55.698+0000 7fd17d2b8640 1 -- 172.21.15.16:0/3446174981 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd17815d980 con 0x7fd164003980 2023-05-22T15:39:55.708 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:55.698+0000 7fd17d2b8640 1 -- 172.21.15.16:0/3446174981 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd16400c000 con 0x7fd164005180 2023-05-22T15:39:55.708 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:55.698+0000 7fd17d2b8640 1 -- 172.21.15.16:0/3446174981 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd16400c1c0 con 0x7fd1640047d0 2023-05-22T15:39:55.708 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:55.698+0000 7fd17e2ba640 1 --2- 172.21.15.16:0/3446174981 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd164003980 0x7fd164003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:55.708 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:55.698+0000 7fd17dab9640 1 --2- 172.21.15.16:0/3446174981 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd1640047d0 0x7fd164004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:55.708 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:55.698+0000 7fd17eabb640 1 --2- 172.21.15.16:0/3446174981 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd164005180 0x7fd16400b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:55.708 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:55.698+0000 7fd17e2ba640 1 -- 172.21.15.16:0/3446174981 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd1640047d0 msgr2=0x7fd164004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:39:55.708 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:55.698+0000 7fd17e2ba640 1 --2- 172.21.15.16:0/3446174981 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd1640047d0 0x7fd164004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:55.709 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:55.698+0000 7fd17e2ba640 1 -- 172.21.15.16:0/3446174981 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd164005180 msgr2=0x7fd16400b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:39:55.709 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:55.698+0000 7fd17e2ba640 1 --2- 172.21.15.16:0/3446174981 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd164005180 0x7fd16400b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:55.709 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:55.698+0000 7fd17e2ba640 1 -- 172.21.15.16:0/3446174981 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd164003530 con 0x7fd164003980 2023-05-22T15:39:55.709 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:55.698+0000 7fd17eabb640 1 --2- 172.21.15.16:0/3446174981 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd164005180 0x7fd16400b990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:39:55.709 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:55.698+0000 7fd17eabb640 1 -- 172.21.15.16:0/3446174981 reap_dead start 2023-05-22T15:39:55.711 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:55.702+0000 7fd17e2ba640 1 --2- 172.21.15.16:0/3446174981 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd164003980 0x7fd164003d90 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7fd174001e70 tx=0x7fd17401c710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4bf8d6434b938ce4 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:39:55.711 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:55.702+0000 7fd17d2b8640 1 -- 172.21.15.16:0/3446174981 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd174003b80 con 0x7fd164003980 2023-05-22T15:39:55.712 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:55.702+0000 7fd17d2b8640 1 -- 172.21.15.16:0/3446174981 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1740a7050 con 0x7fd164003980 2023-05-22T15:39:55.712 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:55.702+0000 7fd17d2b8640 1 -- 172.21.15.16:0/3446174981 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1740050d0 con 0x7fd164003980 2023-05-22T15:39:55.712 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:55.702+0000 7fd18051e640 1 -- 172.21.15.16:0/3446174981 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd164003980 msgr2=0x7fd164003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:55.712 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:55.702+0000 7fd18051e640 1 --2- 172.21.15.16:0/3446174981 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd164003980 0x7fd164003d90 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7fd174001e70 tx=0x7fd17401c710 comp rx=0 tx=0).stop 2023-05-22T15:39:55.712 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:55.702+0000 7fd18051e640 1 -- 172.21.15.16:0/3446174981 shutdown_connections 2023-05-22T15:39:55.712 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:55.702+0000 7fd18051e640 1 --2- 172.21.15.16:0/3446174981 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd164003980 0x7fd164003d90 unknown :-1 s=CLOSED pgs=869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:55.712 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:55.702+0000 7fd18051e640 1 -- 172.21.15.16:0/3446174981 >> 172.21.15.16:0/3446174981 conn(0x7fd178069720 msgr2=0x7fd178069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:39:55.713 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:55.702+0000 7fd18051e640 1 -- 172.21.15.16:0/3446174981 shutdown_connections 2023-05-22T15:39:55.713 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:55.702+0000 7fd18051e640 1 -- 172.21.15.16:0/3446174981 wait complete. 2023-05-22T15:39:55.713 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:55.702+0000 7fd18051e640 1 Processor -- start 2023-05-22T15:39:55.713 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:55.702+0000 7fd18051e640 1 -- start start 2023-05-22T15:39:55.713 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:55.706+0000 7fd18051e640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd164003980 0x7fd1781eff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:55.713 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:55.706+0000 7fd18051e640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd178153550 0x7fd178142620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:55.713 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:55.706+0000 7fd18051e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd178142b60 0x7fd1781503f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:55.713 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:55.706+0000 7fd18051e640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1781f0460 con 0x7fd178142b60 2023-05-22T15:39:55.714 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:55.706+0000 7fd18051e640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1781530c0 con 0x7fd164003980 2023-05-22T15:39:55.714 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:55.706+0000 7fd18051e640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1781524b0 con 0x7fd178153550 2023-05-22T15:39:55.714 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:55.706+0000 7fd17eabb640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd178142b60 0x7fd1781503f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:55.714 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:55.706+0000 7fd17eabb640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd178142b60 0x7fd1781503f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57542/0 (socket says 172.21.15.16:57542) 2023-05-22T15:39:55.714 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:55.706+0000 7fd17eabb640 1 -- 172.21.15.16:0/1652021280 learned_addr learned my addr 172.21.15.16:0/1652021280 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:39:55.714 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:55.706+0000 7fd17dab9640 1 --2- 172.21.15.16:0/1652021280 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd178153550 0x7fd178142620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:55.714 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:55.706+0000 7fd17e2ba640 1 --2- 172.21.15.16:0/1652021280 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd164003980 0x7fd1781eff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:55.714 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:55.706+0000 7fd17eabb640 1 -- 172.21.15.16:0/1652021280 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd178153550 msgr2=0x7fd178142620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:55.715 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:55.706+0000 7fd17eabb640 1 --2- 172.21.15.16:0/1652021280 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd178153550 0x7fd178142620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:55.715 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:55.706+0000 7fd17eabb640 1 -- 172.21.15.16:0/1652021280 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd164003980 msgr2=0x7fd1781eff20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:55.715 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:55.706+0000 7fd17eabb640 1 --2- 172.21.15.16:0/1652021280 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd164003980 0x7fd1781eff20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:55.715 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:55.706+0000 7fd17eabb640 1 -- 172.21.15.16:0/1652021280 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1780ca8b0 con 0x7fd178142b60 2023-05-22T15:39:55.715 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:55.706+0000 7fd17eabb640 1 --2- 172.21.15.16:0/1652021280 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd178142b60 0x7fd1781503f0 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7fd1780b0a40 tx=0x7fd170010850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:39:55.715 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:55.706+0000 7fd16effd640 1 -- 172.21.15.16:0/1652021280 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1700143f0 con 0x7fd178142b60 2023-05-22T15:39:55.715 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:55.706+0000 7fd16effd640 1 -- 172.21.15.16:0/1652021280 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd170014550 con 0x7fd178142b60 2023-05-22T15:39:55.715 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:55.706+0000 7fd18051e640 1 -- 172.21.15.16:0/1652021280 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1780c8e80 con 0x7fd178142b60 2023-05-22T15:39:55.716 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:55.706+0000 7fd16effd640 1 -- 172.21.15.16:0/1652021280 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1700148c0 con 0x7fd178142b60 2023-05-22T15:39:55.716 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:55.706+0000 7fd18051e640 1 -- 172.21.15.16:0/1652021280 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1780c9010 con 0x7fd178142b60 2023-05-22T15:39:55.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:55.706+0000 7fd18051e640 1 -- 172.21.15.16:0/1652021280 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd17815e870 con 0x7fd178142b60 2023-05-22T15:39:55.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:55.706+0000 7fd16effd640 1 -- 172.21.15.16:0/1652021280 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) v1 ==== 84332+0+0 (secure 0 0 0) 0x7fd170014a20 con 0x7fd178142b60 2023-05-22T15:39:55.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:55.706+0000 7fd16effd640 1 --2- 172.21.15.16:0/1652021280 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd15c0654d0 0x7fd15c067980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:55.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:55.706+0000 7fd16effd640 1 -- 172.21.15.16:0/1652021280 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 6175+0+0 (secure 0 0 0) 0x7fd1700821c0 con 0x7fd178142b60 2023-05-22T15:39:55.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:55.706+0000 7fd17e2ba640 1 --2- 172.21.15.16:0/1652021280 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd15c0654d0 0x7fd15c067980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:55.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:55.706+0000 7fd17e2ba640 1 --2- 172.21.15.16:0/1652021280 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd15c0654d0 0x7fd15c067980 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fd1740ac000 tx=0x7fd174001520 comp rx=0 tx=0).ready entity=mgr.6240 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:39:55.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:55.710+0000 7fd16effd640 1 -- 172.21.15.16:0/1652021280 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fd17815e870 con 0x7fd178142b60 2023-05-22T15:39:55.927 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:55.922+0000 7fd18051e640 1 -- 172.21.15.16:0/1652021280 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_multi_snap_schedule"]} v 0) v1 -- 0x7fd1780721a0 con 0x7fd178142b60 2023-05-22T15:39:56.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:56.542+0000 7fd16effd640 1 -- 172.21.15.16:0/1652021280 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_multi_snap_schedule"]}]=0 v480) v1 ==== 157+0+0 (secure 0 0 0) 0x7fd1780721a0 con 0x7fd178142b60 2023-05-22T15:39:56.551 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:56.546+0000 7fd18051e640 1 -- 172.21.15.16:0/1652021280 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd15c0654d0 msgr2=0x7fd15c067980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:56.551 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:56.546+0000 7fd18051e640 1 --2- 172.21.15.16:0/1652021280 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd15c0654d0 0x7fd15c067980 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fd1740ac000 tx=0x7fd174001520 comp rx=0 tx=0).stop 2023-05-22T15:39:56.551 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:56.546+0000 7fd18051e640 1 -- 172.21.15.16:0/1652021280 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd178142b60 msgr2=0x7fd1781503f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:56.551 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:56.546+0000 7fd18051e640 1 --2- 172.21.15.16:0/1652021280 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd178142b60 0x7fd1781503f0 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7fd1780b0a40 tx=0x7fd170010850 comp rx=0 tx=0).stop 2023-05-22T15:39:56.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:56.546+0000 7fd18051e640 1 -- 172.21.15.16:0/1652021280 shutdown_connections 2023-05-22T15:39:56.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:56.546+0000 7fd18051e640 1 --2- 172.21.15.16:0/1652021280 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd178142b60 0x7fd1781503f0 unknown :-1 s=CLOSED pgs=870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:56.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:56.546+0000 7fd18051e640 1 --2- 172.21.15.16:0/1652021280 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd15c0654d0 0x7fd15c067980 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:56.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:56.546+0000 7fd18051e640 1 --2- 172.21.15.16:0/1652021280 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd178153550 0x7fd178142620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:56.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:56.546+0000 7fd18051e640 1 --2- 172.21.15.16:0/1652021280 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd164003980 0x7fd1781eff20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:56.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:56.546+0000 7fd18051e640 1 -- 172.21.15.16:0/1652021280 >> 172.21.15.16:0/1652021280 conn(0x7fd178069720 msgr2=0x7fd1780b20d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:39:56.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:56.546+0000 7fd18051e640 1 -- 172.21.15.16:0/1652021280 shutdown_connections 2023-05-22T15:39:56.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:56.546+0000 7fd18051e640 1 -- 172.21.15.16:0/1652021280 wait complete. 2023-05-22T15:39:56.576 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-05-22T15:39:56.711 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:56.706+0000 7f78cb18c640 1 Processor -- start 2023-05-22T15:39:56.711 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:56.706+0000 7f78cb18c640 1 -- start start 2023-05-22T15:39:56.712 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:56.706+0000 7f78cb18c640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f78c415bca0 0x7f78c40cda80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:56.712 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:56.706+0000 7f78cb18c640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f78c40b5a50 0x7f78c40d2900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:56.712 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:56.706+0000 7f78cb18c640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78c40cccc0 con 0x7f78c40d2480 2023-05-22T15:39:56.713 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:56.706+0000 7f78cb18c640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78c40cd660 con 0x7f78c415bca0 2023-05-22T15:39:56.713 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:56.706+0000 7f78cb18c640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78c40d2e40 con 0x7f78c40b5a50 2023-05-22T15:39:56.713 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:56.706+0000 7f78c8f28640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f78c415bca0 0x7f78c40cda80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:56.713 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:56.706+0000 7f78c3fff640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f78c40d2480 0x7f78c40b5340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:51316/0 (socket says 172.21.15.16:51316) 2023-05-22T15:39:56.713 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:56.706+0000 7f78c3fff640 1 -- 172.21.15.16:0/1258783861 learned_addr learned my addr 172.21.15.16:0/1258783861 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:39:56.713 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:56.706+0000 7f78c9729640 1 --2- 172.21.15.16:0/1258783861 >> v2:172.21.15.62:3300/0 conn(0x7f78c40b5a50 0x7f78c40d2900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:56.713 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:56.706+0000 7f78c8f28640 1 -- 172.21.15.16:0/1258783861 >> v2:172.21.15.62:3300/0 conn(0x7f78c40b5a50 msgr2=0x7f78c40d2900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:39:56.714 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:56.706+0000 7f78c8f28640 1 --2- 172.21.15.16:0/1258783861 >> v2:172.21.15.62:3300/0 conn(0x7f78c40b5a50 0x7f78c40d2900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:56.714 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:56.706+0000 7f78c8f28640 1 -- 172.21.15.16:0/1258783861 >> v1:172.21.15.16:6789/0 conn(0x7f78c40d2480 legacy=0x7f78c40b5340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:56.714 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:56.706+0000 7f78c8f28640 1 -- 172.21.15.16:0/1258783861 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78c40ccfa0 con 0x7f78c415bca0 2023-05-22T15:39:56.714 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:56.706+0000 7f78c9729640 1 --2- 172.21.15.16:0/1258783861 >> v2:172.21.15.62:3300/0 conn(0x7f78c40b5a50 0x7f78c40d2900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:39:56.715 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:56.710+0000 7f78c8f28640 1 --2- 172.21.15.16:0/1258783861 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f78c415bca0 0x7f78c40cda80 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f78c40c5fc0 tx=0x7f78b401c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ded927e28a3225b2 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:39:56.715 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:56.710+0000 7f78c37fe640 1 -- 172.21.15.16:0/1258783861 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78b40ab680 con 0x7f78c415bca0 2023-05-22T15:39:56.715 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:56.710+0000 7f78c37fe640 1 -- 172.21.15.16:0/1258783861 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78b40ab7e0 con 0x7f78c415bca0 2023-05-22T15:39:56.715 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:56.710+0000 7f78c37fe640 1 -- 172.21.15.16:0/1258783861 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78b40abb70 con 0x7f78c415bca0 2023-05-22T15:39:56.716 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:56.710+0000 7f78cb18c640 1 -- 172.21.15.16:0/1258783861 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f78c415bca0 msgr2=0x7f78c40cda80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:56.716 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:56.710+0000 7f78cb18c640 1 --2- 172.21.15.16:0/1258783861 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f78c415bca0 0x7f78c40cda80 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f78c40c5fc0 tx=0x7f78b401c460 comp rx=0 tx=0).stop 2023-05-22T15:39:56.716 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:56.710+0000 7f78cb18c640 1 -- 172.21.15.16:0/1258783861 shutdown_connections 2023-05-22T15:39:56.716 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:56.710+0000 7f78cb18c640 1 --2- 172.21.15.16:0/1258783861 >> v2:172.21.15.62:3300/0 conn(0x7f78c40b5a50 0x7f78c40d2900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:56.716 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:56.710+0000 7f78cb18c640 1 --2- 172.21.15.16:0/1258783861 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f78c415bca0 0x7f78c40cda80 unknown :-1 s=CLOSED pgs=872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:56.716 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:56.710+0000 7f78cb18c640 1 -- 172.21.15.16:0/1258783861 >> 172.21.15.16:0/1258783861 conn(0x7f78c40655c0 msgr2=0x7f78c4070e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:39:56.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:56.710+0000 7f78cb18c640 1 -- 172.21.15.16:0/1258783861 shutdown_connections 2023-05-22T15:39:56.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:56.710+0000 7f78cb18c640 1 -- 172.21.15.16:0/1258783861 wait complete. 2023-05-22T15:39:56.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:56.710+0000 7f78cb18c640 1 Processor -- start 2023-05-22T15:39:56.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:56.710+0000 7f78cb18c640 1 -- start start 2023-05-22T15:39:56.719 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:56.710+0000 7f78cb18c640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f78c40b5a50 0x7f78c4152f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:56.719 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:56.710+0000 7f78cb18c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f78c40d2480 0x7f78c4154520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:56.719 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:56.710+0000 7f78cb18c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f78c415bca0 0x7f78c415cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:56.719 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:56.710+0000 7f78cb18c640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78c415e380 con 0x7f78c40d2480 2023-05-22T15:39:56.719 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:56.710+0000 7f78cb18c640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78c415e660 con 0x7f78c415bca0 2023-05-22T15:39:56.719 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:56.710+0000 7f78cb18c640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78c415e940 con 0x7f78c40b5a50 2023-05-22T15:39:56.719 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:56.710+0000 7f78c3fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f78c40d2480 0x7f78c4154520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:56.720 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:56.710+0000 7f78c3fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f78c40d2480 0x7f78c4154520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57570/0 (socket says 172.21.15.16:57570) 2023-05-22T15:39:56.720 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:56.710+0000 7f78c3fff640 1 -- 172.21.15.16:0/614907066 learned_addr learned my addr 172.21.15.16:0/614907066 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:39:56.720 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:56.710+0000 7f78c9729640 1 --2- 172.21.15.16:0/614907066 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f78c415bca0 0x7f78c415cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:56.720 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:56.710+0000 7f78c8f28640 1 --2- 172.21.15.16:0/614907066 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f78c40b5a50 0x7f78c4152f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:56.720 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:56.710+0000 7f78c3fff640 1 -- 172.21.15.16:0/614907066 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f78c40b5a50 msgr2=0x7f78c4152f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:56.720 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:56.710+0000 7f78c3fff640 1 --2- 172.21.15.16:0/614907066 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f78c40b5a50 0x7f78c4152f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:56.720 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:56.710+0000 7f78c3fff640 1 -- 172.21.15.16:0/614907066 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f78c415bca0 msgr2=0x7f78c415cbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:56.721 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:56.710+0000 7f78c3fff640 1 --2- 172.21.15.16:0/614907066 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f78c415bca0 0x7f78c415cbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:56.721 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:56.710+0000 7f78c3fff640 1 -- 172.21.15.16:0/614907066 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78b4009420 con 0x7f78c40d2480 2023-05-22T15:39:56.721 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:56.710+0000 7f78c8f28640 1 --2- 172.21.15.16:0/614907066 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f78c40b5a50 0x7f78c4152f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:39:56.721 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:56.714+0000 7f78c3fff640 1 --2- 172.21.15.16:0/614907066 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f78c40d2480 0x7f78c4154520 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f78b800c500 tx=0x7f78b800c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:39:56.721 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:56.714+0000 7f78c17fa640 1 -- 172.21.15.16:0/614907066 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78b8005cf0 con 0x7f78c40d2480 2023-05-22T15:39:56.721 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:56.714+0000 7f78c17fa640 1 -- 172.21.15.16:0/614907066 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78b8005e50 con 0x7f78c40d2480 2023-05-22T15:39:56.721 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:56.714+0000 7f78cb18c640 1 -- 172.21.15.16:0/614907066 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78c4160360 con 0x7f78c40d2480 2023-05-22T15:39:56.721 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:56.714+0000 7f78c17fa640 1 -- 172.21.15.16:0/614907066 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78b80072f0 con 0x7f78c40d2480 2023-05-22T15:39:56.722 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:56.714+0000 7f78cb18c640 1 -- 172.21.15.16:0/614907066 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78c40cdfc0 con 0x7f78c40d2480 2023-05-22T15:39:56.722 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:56.714+0000 7f78cb18c640 1 -- 172.21.15.16:0/614907066 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78c41604f0 con 0x7f78c40d2480 2023-05-22T15:39:56.723 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:56.714+0000 7f78c17fa640 1 -- 172.21.15.16:0/614907066 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f78b8007450 con 0x7f78c40d2480 2023-05-22T15:39:56.723 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:56.714+0000 7f78c17fa640 1 --2- 172.21.15.16:0/614907066 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f78b00654a0 0x7f78b0067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:56.723 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:56.714+0000 7f78c17fa640 1 -- 172.21.15.16:0/614907066 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 6175+0+0 (secure 0 0 0) 0x7f78b8081f60 con 0x7f78c40d2480 2023-05-22T15:39:56.723 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:56.714+0000 7f78c8f28640 1 --2- 172.21.15.16:0/614907066 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f78b00654a0 0x7f78b0067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:56.723 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:56.714+0000 7f78c8f28640 1 --2- 172.21.15.16:0/614907066 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f78b00654a0 0x7f78b0067950 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f78b40af000 tx=0x7f78b4008040 comp rx=0 tx=0).ready entity=mgr.6240 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:39:56.724 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:56.718+0000 7f78c17fa640 1 -- 172.21.15.16:0/614907066 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f78b804a620 con 0x7f78c40d2480 2023-05-22T15:39:56.878 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:56.874+0000 7f78cb18c640 1 -- 172.21.15.16:0/614907066 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f78c415f140 con 0x7f78c40d2480 2023-05-22T15:39:56.886 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:56.878+0000 7f78c17fa640 1 -- 172.21.15.16:0/614907066 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v16)=0 v16) v1 ==== 122+0+0 (secure 0 0 0) 0x7f78c415f140 con 0x7f78c40d2480 2023-05-22T15:39:56.888 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:56.882+0000 7f78cb18c640 1 -- 172.21.15.16:0/614907066 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f78b00654a0 msgr2=0x7f78b0067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:56.888 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:56.882+0000 7f78cb18c640 1 --2- 172.21.15.16:0/614907066 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f78b00654a0 0x7f78b0067950 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f78b40af000 tx=0x7f78b4008040 comp rx=0 tx=0).stop 2023-05-22T15:39:56.889 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:56.882+0000 7f78cb18c640 1 -- 172.21.15.16:0/614907066 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f78c40d2480 msgr2=0x7f78c4154520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:56.889 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:56.882+0000 7f78cb18c640 1 --2- 172.21.15.16:0/614907066 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f78c40d2480 0x7f78c4154520 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f78b800c500 tx=0x7f78b800c9c0 comp rx=0 tx=0).stop 2023-05-22T15:39:56.889 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:56.882+0000 7f78cb18c640 1 -- 172.21.15.16:0/614907066 shutdown_connections 2023-05-22T15:39:56.889 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:56.882+0000 7f78cb18c640 1 --2- 172.21.15.16:0/614907066 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f78c415bca0 0x7f78c415cbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:56.889 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:56.882+0000 7f78cb18c640 1 --2- 172.21.15.16:0/614907066 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f78c40d2480 0x7f78c4154520 unknown :-1 s=CLOSED pgs=873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:56.889 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:56.882+0000 7f78cb18c640 1 --2- 172.21.15.16:0/614907066 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f78b00654a0 0x7f78b0067950 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:56.890 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:56.882+0000 7f78cb18c640 1 --2- 172.21.15.16:0/614907066 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f78c40b5a50 0x7f78c4152f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:56.890 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:56.882+0000 7f78cb18c640 1 -- 172.21.15.16:0/614907066 >> 172.21.15.16:0/614907066 conn(0x7f78c40655c0 msgr2=0x7f78c40b5320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:39:56.890 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:56.882+0000 7f78cb18c640 1 -- 172.21.15.16:0/614907066 shutdown_connections 2023-05-22T15:39:56.890 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:56.882+0000 7f78cb18c640 1 -- 172.21.15.16:0/614907066 wait complete. 2023-05-22T15:39:56.906 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-05-22T15:39:56.906 DEBUG:teuthology.orchestra.run.smithi016:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-05-22T15:39:56.948 DEBUG:teuthology.orchestra.run.smithi062:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-05-22T15:39:57.007 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-05-22T15:39:57.007 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-05-22T15:39:57.008 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2023-05-22T15:39:57.014 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16345192k,nr_inodes=4086298,mode=755,inode64 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3276544k,mode=755,inode64 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev,inode64 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=15732 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3276540k,nr_inodes=819135,mode=700,uid=1000,gid=1271,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3276544k,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-05-22T15:39:57.014 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2023-05-22T15:39:57.015 DEBUG:teuthology.orchestra.run.smithi062:> dd if=/proc/self/mounts of=/dev/stdout 2023-05-22T15:39:57.020 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16342116k,nr_inodes=4085529,mode=755,inode64 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275932k,mode=755,inode64 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev,inode64 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=13250 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275928k,nr_inodes=818982,mode=700,uid=1000,gid=1271,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275932k,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-05-22T15:39:57.021 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-05-22T15:39:57.021 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2023-05-22T15:39:57.063 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16345192k,nr_inodes=4086298,mode=755,inode64 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3276544k,mode=755,inode64 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev,inode64 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=15732 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3276540k,nr_inodes=819135,mode=700,uid=1000,gid=1271,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3276544k,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-05-22T15:39:57.063 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2023-05-22T15:39:57.063 DEBUG:teuthology.orchestra.run.smithi062:> dd if=/proc/self/mounts of=/dev/stdout 2023-05-22T15:39:57.069 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16342116k,nr_inodes=4085529,mode=755,inode64 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275932k,mode=755,inode64 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev,inode64 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=13250 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275928k,nr_inodes=818982,mode=700,uid=1000,gid=1271,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275932k,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-05-22T15:39:57.070 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2023-05-22T15:39:57.070 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-05-22T15:39:57.242 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:57.238+0000 7f9cb210d640 1 Processor -- start 2023-05-22T15:39:57.242 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:57.238+0000 7f9cb210d640 1 -- start start 2023-05-22T15:39:57.243 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:57.238+0000 7f9cb210d640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f9cac1567c0 0x7f9cac0afb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:57.243 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:57.238+0000 7f9cb210d640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f9cac0cb6a0 0x7f9cac0b0080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:57.244 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:57.238+0000 7f9cb210d640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9cac156b90 con 0x7f9cac159340 2023-05-22T15:39:57.244 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:57.238+0000 7f9cb210d640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9cac15a0b0 con 0x7f9cac1567c0 2023-05-22T15:39:57.244 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:57.238+0000 7f9cb210d640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9cac0ca8e0 con 0x7f9cac0cb6a0 2023-05-22T15:39:57.244 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:57.238+0000 7f9cab7fe640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f9cac1567c0 0x7f9cac0afb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:57.244 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:57.238+0000 7f9cabfff640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f9cac159340 0x7f9cac1597e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:51328/0 (socket says 172.21.15.16:51328) 2023-05-22T15:39:57.244 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:57.238+0000 7f9cabfff640 1 -- 172.21.15.16:0/858182573 learned_addr learned my addr 172.21.15.16:0/858182573 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:39:57.245 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:57.238+0000 7f9caaffd640 1 --2- 172.21.15.16:0/858182573 >> v2:172.21.15.62:3300/0 conn(0x7f9cac0cb6a0 0x7f9cac0b0080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:57.245 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:57.238+0000 7f9cab7fe640 1 -- 172.21.15.16:0/858182573 >> v2:172.21.15.62:3300/0 conn(0x7f9cac0cb6a0 msgr2=0x7f9cac0b0080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:39:57.245 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:57.238+0000 7f9cab7fe640 1 --2- 172.21.15.16:0/858182573 >> v2:172.21.15.62:3300/0 conn(0x7f9cac0cb6a0 0x7f9cac0b0080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:57.245 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:57.238+0000 7f9cab7fe640 1 -- 172.21.15.16:0/858182573 >> v1:172.21.15.16:6789/0 conn(0x7f9cac159340 legacy=0x7f9cac1597e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:57.246 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:57.238+0000 7f9cab7fe640 1 -- 172.21.15.16:0/858182573 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cac0cabc0 con 0x7f9cac1567c0 2023-05-22T15:39:57.246 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:57.238+0000 7f9caa7fc640 1 -- 172.21.15.16:0/858182573 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2424766363 0 0) 0x7f9ca40014a0 con 0x7f9cac159340 2023-05-22T15:39:57.247 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:57.242+0000 7f9cab7fe640 1 --2- 172.21.15.16:0/858182573 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9cac1567c0 0x7f9cac0afb40 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7f9cac15d780 tx=0x7f9c9c01c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ff895c13281273 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:39:57.247 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:57.242+0000 7f9caa7fc640 1 -- 172.21.15.16:0/858182573 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c9c0ab600 con 0x7f9cac1567c0 2023-05-22T15:39:57.247 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:57.242+0000 7f9caa7fc640 1 -- 172.21.15.16:0/858182573 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c9c0a7050 con 0x7f9cac1567c0 2023-05-22T15:39:57.247 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:57.242+0000 7f9caa7fc640 1 -- 172.21.15.16:0/858182573 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c9c0ab980 con 0x7f9cac1567c0 2023-05-22T15:39:57.247 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:57.242+0000 7f9cb210d640 1 -- 172.21.15.16:0/858182573 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9cac1567c0 msgr2=0x7f9cac0afb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:57.247 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:57.242+0000 7f9cb210d640 1 --2- 172.21.15.16:0/858182573 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9cac1567c0 0x7f9cac0afb40 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7f9cac15d780 tx=0x7f9c9c01c460 comp rx=0 tx=0).stop 2023-05-22T15:39:57.248 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:57.242+0000 7f9cb210d640 1 -- 172.21.15.16:0/858182573 shutdown_connections 2023-05-22T15:39:57.248 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:57.242+0000 7f9cb210d640 1 --2- 172.21.15.16:0/858182573 >> v2:172.21.15.62:3300/0 conn(0x7f9cac0cb6a0 0x7f9cac0b0080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:57.248 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:57.242+0000 7f9cb210d640 1 --2- 172.21.15.16:0/858182573 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9cac1567c0 0x7f9cac0afb40 unknown :-1 s=CLOSED pgs=875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:57.248 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:57.242+0000 7f9cb210d640 1 -- 172.21.15.16:0/858182573 >> 172.21.15.16:0/858182573 conn(0x7f9cac069720 msgr2=0x7f9cac069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:39:57.248 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:57.242+0000 7f9cb210d640 1 -- 172.21.15.16:0/858182573 shutdown_connections 2023-05-22T15:39:57.248 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:57.242+0000 7f9cb210d640 1 -- 172.21.15.16:0/858182573 wait complete. 2023-05-22T15:39:57.250 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:57.242+0000 7f9cb210d640 1 Processor -- start 2023-05-22T15:39:57.250 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:57.242+0000 7f9cb210d640 1 -- start start 2023-05-22T15:39:57.250 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:57.242+0000 7f9cb210d640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9cac0cb6a0 0x7f9cac0b30e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:57.250 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:57.242+0000 7f9cb210d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9cac1567c0 0x7f9cac0aef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:57.250 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:57.242+0000 7f9cb210d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9cac159340 0x7f9cac0c73d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:57.251 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:57.242+0000 7f9cb210d640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cac0c70a0 con 0x7f9cac159340 2023-05-22T15:39:57.251 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:57.242+0000 7f9cb210d640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cac0b20d0 con 0x7f9cac1567c0 2023-05-22T15:39:57.251 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:57.242+0000 7f9cb210d640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9cac0b2c50 con 0x7f9cac0cb6a0 2023-05-22T15:39:57.251 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:57.242+0000 7f9cabfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9cac159340 0x7f9cac0c73d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:57.251 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:57.242+0000 7f9cabfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9cac159340 0x7f9cac0c73d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57580/0 (socket says 172.21.15.16:57580) 2023-05-22T15:39:57.251 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:57.242+0000 7f9cabfff640 1 -- 172.21.15.16:0/2233803239 learned_addr learned my addr 172.21.15.16:0/2233803239 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:39:57.251 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:57.242+0000 7f9caaffd640 1 --2- 172.21.15.16:0/2233803239 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9cac1567c0 0x7f9cac0aef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:57.251 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:57.242+0000 7f9cab7fe640 1 --2- 172.21.15.16:0/2233803239 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9cac0cb6a0 0x7f9cac0b30e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:57.252 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:57.242+0000 7f9cabfff640 1 -- 172.21.15.16:0/2233803239 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9cac0cb6a0 msgr2=0x7f9cac0b30e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:57.252 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:57.242+0000 7f9cabfff640 1 --2- 172.21.15.16:0/2233803239 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9cac0cb6a0 0x7f9cac0b30e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:57.252 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:57.242+0000 7f9cabfff640 1 -- 172.21.15.16:0/2233803239 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9cac1567c0 msgr2=0x7f9cac0aef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:57.252 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:57.242+0000 7f9cabfff640 1 --2- 172.21.15.16:0/2233803239 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9cac1567c0 0x7f9cac0aef00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:57.252 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:57.242+0000 7f9cabfff640 1 -- 172.21.15.16:0/2233803239 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c9c009420 con 0x7f9cac159340 2023-05-22T15:39:57.252 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:57.242+0000 7f9cab7fe640 1 --2- 172.21.15.16:0/2233803239 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9cac0cb6a0 0x7f9cac0b30e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:39:57.252 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:57.242+0000 7f9cabfff640 1 --2- 172.21.15.16:0/2233803239 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9cac159340 0x7f9cac0c73d0 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f9ca400c7b0 tx=0x7f9ca400cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:39:57.253 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:57.242+0000 7f9c93fff640 1 -- 172.21.15.16:0/2233803239 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ca400a640 con 0x7f9cac159340 2023-05-22T15:39:57.253 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:57.242+0000 7f9c93fff640 1 -- 172.21.15.16:0/2233803239 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ca400a7c0 con 0x7f9cac159340 2023-05-22T15:39:57.253 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:57.242+0000 7f9cb210d640 1 -- 172.21.15.16:0/2233803239 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9cac0c9010 con 0x7f9cac159340 2023-05-22T15:39:57.253 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:57.242+0000 7f9c93fff640 1 -- 172.21.15.16:0/2233803239 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ca400ab70 con 0x7f9cac159340 2023-05-22T15:39:57.253 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:57.242+0000 7f9cb210d640 1 -- 172.21.15.16:0/2233803239 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9cac0c8e80 con 0x7f9cac159340 2023-05-22T15:39:57.254 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:57.246+0000 7f9cb210d640 1 -- 172.21.15.16:0/2233803239 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9cac0c91a0 con 0x7f9cac159340 2023-05-22T15:39:57.254 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:57.246+0000 7f9c93fff640 1 -- 172.21.15.16:0/2233803239 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f9ca400acd0 con 0x7f9cac159340 2023-05-22T15:39:57.254 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:57.246+0000 7f9c93fff640 1 --2- 172.21.15.16:0/2233803239 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f9c980654d0 0x7f9c98067980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:57.254 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:57.246+0000 7f9cab7fe640 1 --2- 172.21.15.16:0/2233803239 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f9c980654d0 0x7f9c98067980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:57.254 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:57.246+0000 7f9c93fff640 1 -- 172.21.15.16:0/2233803239 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 6175+0+0 (secure 0 0 0) 0x7f9ca40820c0 con 0x7f9cac159340 2023-05-22T15:39:57.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:57.246+0000 7f9cab7fe640 1 --2- 172.21.15.16:0/2233803239 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f9c980654d0 0x7f9c98067980 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f9c9c0ad000 tx=0x7f9c9c008040 comp rx=0 tx=0).ready entity=mgr.6240 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:39:57.256 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:57.250+0000 7f9c93fff640 1 -- 172.21.15.16:0/2233803239 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f9ca404fac0 con 0x7f9cac159340 2023-05-22T15:39:57.442 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:57.438+0000 7f9cb210d640 1 -- 172.21.15.16:0/2233803239 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9cac156b90 con 0x7f9cac159340 2023-05-22T15:39:57.443 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:57.438+0000 7f9c93fff640 1 -- 172.21.15.16:0/2233803239 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) v1 ==== 94+0+3529 (secure 0 0 0) 0x7f9cac156b90 con 0x7f9cac159340 2023-05-22T15:39:57.443 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:39:57.444 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":16,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4904,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6833/3153072235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":3153072235},{"type":"v1","addr":"172.21.15.62:6833","nonce":3153072235}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":5164,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/2783822861","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":2783822861},{"type":"v1","addr":"172.21.15.62:6835","nonce":2783822861}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":5571,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/1361634240","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1361634240},{"type":"v1","addr":"172.21.15.16:6835","nonce":1361634240}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":6393,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/2939586856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":2939586856},{"type":"v1","addr":"172.21.15.16:6837","nonce":2939586856}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":16}],"filesystems":[]} 2023-05-22T15:39:57.444 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 16 2023-05-22T15:39:57.445 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:57.438+0000 7f9cb210d640 1 -- 172.21.15.16:0/2233803239 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f9c980654d0 msgr2=0x7f9c98067980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:57.446 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:57.438+0000 7f9cb210d640 1 --2- 172.21.15.16:0/2233803239 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f9c980654d0 0x7f9c98067980 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f9c9c0ad000 tx=0x7f9c9c008040 comp rx=0 tx=0).stop 2023-05-22T15:39:57.446 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:57.438+0000 7f9cb210d640 1 -- 172.21.15.16:0/2233803239 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9cac159340 msgr2=0x7f9cac0c73d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:57.446 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:57.438+0000 7f9cb210d640 1 --2- 172.21.15.16:0/2233803239 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9cac159340 0x7f9cac0c73d0 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f9ca400c7b0 tx=0x7f9ca400cc70 comp rx=0 tx=0).stop 2023-05-22T15:39:57.446 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:57.438+0000 7f9cb210d640 1 -- 172.21.15.16:0/2233803239 shutdown_connections 2023-05-22T15:39:57.446 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:57.438+0000 7f9cb210d640 1 --2- 172.21.15.16:0/2233803239 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9cac159340 0x7f9cac0c73d0 unknown :-1 s=CLOSED pgs=876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:57.446 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:57.438+0000 7f9cb210d640 1 --2- 172.21.15.16:0/2233803239 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9cac1567c0 0x7f9cac0aef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:57.446 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:57.438+0000 7f9cb210d640 1 --2- 172.21.15.16:0/2233803239 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f9c980654d0 0x7f9c98067980 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:57.447 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:57.438+0000 7f9cb210d640 1 --2- 172.21.15.16:0/2233803239 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9cac0cb6a0 0x7f9cac0b30e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:57.447 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:57.438+0000 7f9cb210d640 1 -- 172.21.15.16:0/2233803239 >> 172.21.15.16:0/2233803239 conn(0x7f9cac069720 msgr2=0x7f9cac0ce020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:39:57.447 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:57.438+0000 7f9cb210d640 1 -- 172.21.15.16:0/2233803239 shutdown_connections 2023-05-22T15:39:57.447 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:57.438+0000 7f9cb210d640 1 -- 172.21.15.16:0/2233803239 wait complete. 2023-05-22T15:39:57.462 INFO:tasks.ceph.mds.a:Restarting daemon 2023-05-22T15:39:57.463 INFO:tasks.ceph.mds.a:Stopping old one... 2023-05-22T15:39:57.463 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-05-22T15:39:57.463 INFO:teuthology.orchestra.run:waiting for 300 2023-05-22T15:39:57.464 INFO:tasks.ceph.mds.c:Restarting daemon 2023-05-22T15:39:57.464 INFO:tasks.ceph.mds.c:Stopping old one... 2023-05-22T15:39:57.464 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-05-22T15:39:57.464 INFO:teuthology.orchestra.run:waiting for 300 2023-05-22T15:39:57.464 INFO:tasks.ceph.mds.b:Restarting daemon 2023-05-22T15:39:57.465 INFO:tasks.ceph.mds.b:Stopping old one... 2023-05-22T15:39:57.465 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-05-22T15:39:57.465 INFO:teuthology.orchestra.run:waiting for 300 2023-05-22T15:39:57.465 INFO:tasks.ceph.mds.d:Restarting daemon 2023-05-22T15:39:57.466 INFO:tasks.ceph.mds.d:Stopping old one... 2023-05-22T15:39:57.466 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-05-22T15:39:57.466 INFO:teuthology.orchestra.run:waiting for 300 2023-05-22T15:39:57.473 INFO:tasks.ceph.mds.a:Stopped 2023-05-22T15:39:57.473 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-05-22T15:39:57.475 INFO:tasks.ceph.mds.c:Stopped 2023-05-22T15:39:57.475 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-05-22T15:39:57.480 INFO:tasks.ceph.mds.d:Stopped 2023-05-22T15:39:57.480 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-05-22T15:39:57.485 INFO:tasks.ceph.mds.b:Stopped 2023-05-22T15:39:57.485 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-05-22T15:39:57.519 INFO:tasks.ceph.mds.a:Started 2023-05-22T15:39:57.519 DEBUG:teuthology.parallel:result is None 2023-05-22T15:39:57.520 INFO:tasks.ceph.mds.c:Started 2023-05-22T15:39:57.520 DEBUG:teuthology.parallel:result is None 2023-05-22T15:39:57.526 INFO:tasks.ceph.mds.d:Started 2023-05-22T15:39:57.527 DEBUG:teuthology.parallel:result is None 2023-05-22T15:39:57.528 INFO:tasks.ceph.mds.b:Started 2023-05-22T15:39:57.528 DEBUG:teuthology.parallel:result is None 2023-05-22T15:39:57.529 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2023-05-22T15:39:57.581 INFO:tasks.ceph.mds.c.smithi016.stdout:starting mds.c at 2023-05-22T15:39:57.588 INFO:tasks.ceph.mds.a.smithi016.stdout:starting mds.a at 2023-05-22T15:39:57.606 INFO:tasks.ceph.mds.d.smithi062.stdout:starting mds.d at 2023-05-22T15:39:57.607 INFO:tasks.ceph.mds.b.smithi062.stdout:starting mds.b at 2023-05-22T15:39:57.654 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:57.650+0000 7f7674357640 1 Processor -- start 2023-05-22T15:39:57.654 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:57.650+0000 7f7674357640 1 -- start start 2023-05-22T15:39:57.655 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:57.650+0000 7f7674357640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f766c0cb6a0 0x7f766c159340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:57.655 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:57.650+0000 7f7674357640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f766c0b0250 0x7f766c159880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:57.655 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:57.650+0000 7f7674357640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f766c156b90 con 0x7f766c1567c0 2023-05-22T15:39:57.656 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:57.650+0000 7f7674357640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f766c15a0b0 con 0x7f766c0cb6a0 2023-05-22T15:39:57.656 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:57.650+0000 7f7674357640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f766c0ca8e0 con 0x7f766c0b0250 2023-05-22T15:39:57.656 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:57.650+0000 7f76720f3640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f766c1567c0 0x7f766c0afb40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:51340/0 (socket says 172.21.15.16:51340) 2023-05-22T15:39:57.656 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:57.650+0000 7f76720f3640 1 -- 172.21.15.16:0/948090443 learned_addr learned my addr 172.21.15.16:0/948090443 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:39:57.656 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:57.650+0000 7f76718f2640 1 --2- 172.21.15.16:0/948090443 >> v2:172.21.15.62:3300/0 conn(0x7f766c0cb6a0 0x7f766c159340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:57.656 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:57.650+0000 7f76728f4640 1 --2- 172.21.15.16:0/948090443 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f766c0b0250 0x7f766c159880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:57.657 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:57.650+0000 7f76718f2640 1 -- 172.21.15.16:0/948090443 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f766c0b0250 msgr2=0x7f766c159880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:39:57.657 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:57.650+0000 7f76718f2640 1 --2- 172.21.15.16:0/948090443 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f766c0b0250 0x7f766c159880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:57.657 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:57.650+0000 7f76718f2640 1 -- 172.21.15.16:0/948090443 >> v1:172.21.15.16:6789/0 conn(0x7f766c1567c0 legacy=0x7f766c0afb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:57.657 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:57.650+0000 7f76718f2640 1 -- 172.21.15.16:0/948090443 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f766c0cabc0 con 0x7f766c0cb6a0 2023-05-22T15:39:57.658 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:57.650+0000 7f76718f2640 1 --2- 172.21.15.16:0/948090443 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f766c0cb6a0 0x7f766c159340 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f765c0097c0 tx=0x7f765c01c480 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f3c07082ef375b08 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:39:57.658 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:57.650+0000 7f76710f1640 1 -- 172.21.15.16:0/948090443 <== mon.1 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f765c0ab600 con 0x7f766c0cb6a0 2023-05-22T15:39:57.658 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:57.650+0000 7f76710f1640 1 -- 172.21.15.16:0/948090443 <== mon.1 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f765c0a7050 con 0x7f766c0cb6a0 2023-05-22T15:39:57.658 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:57.650+0000 7f76710f1640 1 -- 172.21.15.16:0/948090443 <== mon.1 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f765c0ab980 con 0x7f766c0cb6a0 2023-05-22T15:39:57.659 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:57.654+0000 7f7674357640 1 -- 172.21.15.16:0/948090443 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f766c0cb6a0 msgr2=0x7f766c159340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:57.659 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:57.654+0000 7f7674357640 1 --2- 172.21.15.16:0/948090443 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f766c0cb6a0 0x7f766c159340 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f765c0097c0 tx=0x7f765c01c480 comp rx=0 tx=0).stop 2023-05-22T15:39:57.659 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:57.654+0000 7f7674357640 1 -- 172.21.15.16:0/948090443 shutdown_connections 2023-05-22T15:39:57.659 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:57.654+0000 7f7674357640 1 --2- 172.21.15.16:0/948090443 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f766c0b0250 0x7f766c159880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:57.659 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:57.654+0000 7f7674357640 1 --2- 172.21.15.16:0/948090443 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f766c0cb6a0 0x7f766c159340 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:57.660 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:57.654+0000 7f7674357640 1 -- 172.21.15.16:0/948090443 >> 172.21.15.16:0/948090443 conn(0x7f766c069720 msgr2=0x7f766c069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:39:57.660 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:57.654+0000 7f7674357640 1 -- 172.21.15.16:0/948090443 shutdown_connections 2023-05-22T15:39:57.660 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:57.654+0000 7f7674357640 1 -- 172.21.15.16:0/948090443 wait complete. 2023-05-22T15:39:57.661 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:57.654+0000 7f7674357640 1 Processor -- start 2023-05-22T15:39:57.662 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:57.654+0000 7f7674357640 1 -- start start 2023-05-22T15:39:57.662 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:57.654+0000 7f7674357640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f766c0b0250 0x7f766c14d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:57.662 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:57.654+0000 7f7674357640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f766c0cb6a0 0x7f766c0c3800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:57.662 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:57.654+0000 7f7674357640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f766c1567c0 0x7f766c0c3d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:57.662 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:57.654+0000 7f7674357640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f766c0c33c0 con 0x7f766c0b0250 2023-05-22T15:39:57.662 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:57.654+0000 7f7674357640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f766c0c2840 con 0x7f766c1567c0 2023-05-22T15:39:57.662 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:57.654+0000 7f7674357640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f766c0c2b20 con 0x7f766c0cb6a0 2023-05-22T15:39:57.663 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:57.654+0000 7f76720f3640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f766c0b0250 0x7f766c14d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:57.663 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:57.654+0000 7f76720f3640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f766c0b0250 0x7f766c14d530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57610/0 (socket says 172.21.15.16:57610) 2023-05-22T15:39:57.663 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:57.654+0000 7f76720f3640 1 -- 172.21.15.16:0/2403925090 learned_addr learned my addr 172.21.15.16:0/2403925090 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:39:57.663 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:57.654+0000 7f76718f2640 1 --2- 172.21.15.16:0/2403925090 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f766c0cb6a0 0x7f766c0c3800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:57.663 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:57.654+0000 7f76728f4640 1 --2- 172.21.15.16:0/2403925090 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f766c1567c0 0x7f766c0c3d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:57.663 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:57.654+0000 7f76720f3640 1 -- 172.21.15.16:0/2403925090 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f766c0cb6a0 msgr2=0x7f766c0c3800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:57.663 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:57.654+0000 7f76720f3640 1 --2- 172.21.15.16:0/2403925090 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f766c0cb6a0 0x7f766c0c3800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:57.663 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:57.654+0000 7f76720f3640 1 -- 172.21.15.16:0/2403925090 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f766c1567c0 msgr2=0x7f766c0c3d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:57.664 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:57.654+0000 7f76720f3640 1 --2- 172.21.15.16:0/2403925090 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f766c1567c0 0x7f766c0c3d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:57.664 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:57.654+0000 7f76720f3640 1 -- 172.21.15.16:0/2403925090 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f765c009420 con 0x7f766c0b0250 2023-05-22T15:39:57.664 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:57.654+0000 7f76720f3640 1 --2- 172.21.15.16:0/2403925090 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f766c0b0250 0x7f766c14d530 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7f766800c400 tx=0x7f766800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:39:57.664 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:57.654+0000 7f7662ffd640 1 -- 172.21.15.16:0/2403925090 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7668005cf0 con 0x7f766c0b0250 2023-05-22T15:39:57.664 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:57.654+0000 7f7662ffd640 1 -- 172.21.15.16:0/2403925090 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7668005e50 con 0x7f766c0b0250 2023-05-22T15:39:57.664 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:57.654+0000 7f7674357640 1 -- 172.21.15.16:0/2403925090 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f766c0c8e80 con 0x7f766c0b0250 2023-05-22T15:39:57.664 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:57.654+0000 7f7662ffd640 1 -- 172.21.15.16:0/2403925090 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76680072d0 con 0x7f766c0b0250 2023-05-22T15:39:57.665 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:57.654+0000 7f7674357640 1 -- 172.21.15.16:0/2403925090 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f766c159dc0 con 0x7f766c0b0250 2023-05-22T15:39:57.665 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:57.658+0000 7f7674357640 1 -- 172.21.15.16:0/2403925090 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f766c0c9010 con 0x7f766c0b0250 2023-05-22T15:39:57.665 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:57.658+0000 7f7662ffd640 1 -- 172.21.15.16:0/2403925090 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f7668007430 con 0x7f766c0b0250 2023-05-22T15:39:57.666 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:57.658+0000 7f7662ffd640 1 --2- 172.21.15.16:0/2403925090 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f76500654a0 0x7f7650067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:57.666 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:57.658+0000 7f7662ffd640 1 -- 172.21.15.16:0/2403925090 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6175+0+0 (secure 0 0 0) 0x7f7668081de0 con 0x7f766c0b0250 2023-05-22T15:39:57.666 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:57.658+0000 7f76718f2640 1 --2- 172.21.15.16:0/2403925090 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f76500654a0 0x7f7650067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:57.666 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:57.658+0000 7f76718f2640 1 --2- 172.21.15.16:0/2403925090 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f76500654a0 0x7f7650067950 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f765c0ad000 tx=0x7f765c008040 comp rx=0 tx=0).ready entity=mgr.6240 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:39:57.667 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:57.662+0000 7f7662ffd640 1 -- 172.21.15.16:0/2403925090 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f766804a620 con 0x7f766c0b0250 2023-05-22T15:39:57.834 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:57.826+0000 7f7674357640 1 -- 172.21.15.16:0/2403925090 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f766c0c6140 con 0x7f766c0b0250 2023-05-22T15:39:58.663 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:58.658+0000 7f7662ffd640 1 -- 172.21.15.16:0/2403925090 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v69) v1 ==== 97+0+0 (secure 0 0 0) 0x7f766c0c6140 con 0x7f766c0b0250 2023-05-22T15:39:58.664 INFO:teuthology.orchestra.run.smithi016.stderr: removed all blocklist entries 2023-05-22T15:39:58.667 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:58.662+0000 7f7674357640 1 -- 172.21.15.16:0/2403925090 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f76500654a0 msgr2=0x7f7650067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:58.667 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:58.662+0000 7f7674357640 1 --2- 172.21.15.16:0/2403925090 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f76500654a0 0x7f7650067950 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f765c0ad000 tx=0x7f765c008040 comp rx=0 tx=0).stop 2023-05-22T15:39:58.667 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:58.662+0000 7f7674357640 1 -- 172.21.15.16:0/2403925090 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f766c0b0250 msgr2=0x7f766c14d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:58.667 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:58.662+0000 7f7674357640 1 --2- 172.21.15.16:0/2403925090 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f766c0b0250 0x7f766c14d530 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7f766800c400 tx=0x7f766800c8c0 comp rx=0 tx=0).stop 2023-05-22T15:39:58.667 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:58.662+0000 7f7674357640 1 -- 172.21.15.16:0/2403925090 shutdown_connections 2023-05-22T15:39:58.667 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:58.662+0000 7f7674357640 1 --2- 172.21.15.16:0/2403925090 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f766c1567c0 0x7f766c0c3d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:58.668 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:58.662+0000 7f7674357640 1 --2- 172.21.15.16:0/2403925090 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f76500654a0 0x7f7650067950 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:58.668 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:58.662+0000 7f7674357640 1 --2- 172.21.15.16:0/2403925090 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f766c0cb6a0 0x7f766c0c3800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:58.668 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:58.662+0000 7f7674357640 1 --2- 172.21.15.16:0/2403925090 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f766c0b0250 0x7f766c14d530 unknown :-1 s=CLOSED pgs=884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:58.668 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:58.662+0000 7f7674357640 1 -- 172.21.15.16:0/2403925090 >> 172.21.15.16:0/2403925090 conn(0x7f766c069720 msgr2=0x7f766c0afb30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:39:58.668 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:58.662+0000 7f7674357640 1 -- 172.21.15.16:0/2403925090 shutdown_connections 2023-05-22T15:39:58.668 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:58.662+0000 7f7674357640 1 -- 172.21.15.16:0/2403925090 wait complete. 2023-05-22T15:39:58.688 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2023-05-22T15:39:58.814 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:58.810+0000 7fd18f4b1640 1 Processor -- start 2023-05-22T15:39:58.815 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:58.810+0000 7fd18f4b1640 1 -- start start 2023-05-22T15:39:58.815 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:58.810+0000 7fd18f4b1640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fd1881531b0 0x7fd1880afb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:58.816 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:58.810+0000 7fd18f4b1640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd1880b0080 0x7fd188159330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:58.816 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:58.810+0000 7fd18f4b1640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd18815a0a0 con 0x7fd188159870 2023-05-22T15:39:58.816 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:58.810+0000 7fd18f4b1640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1880b0520 con 0x7fd1880b0080 2023-05-22T15:39:58.816 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:58.810+0000 7fd18f4b1640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd188153580 con 0x7fd1881531b0 2023-05-22T15:39:58.816 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:58.810+0000 7fd18ca4c640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd1880b0080 0x7fd188159330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:58.816 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:58.810+0000 7fd18ca4c640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd1880b0080 0x7fd188159330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57614/0 (socket says 172.21.15.16:57614) 2023-05-22T15:39:58.817 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:58.810+0000 7fd18ca4c640 1 -- 172.21.15.16:0/447637464 learned_addr learned my addr 172.21.15.16:0/447637464 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:39:58.817 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:58.810+0000 7fd18d24d640 1 --2- 172.21.15.16:0/447637464 >> v2:172.21.15.62:3300/0 conn(0x7fd1881531b0 0x7fd1880afb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:58.817 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:58.810+0000 7fd18ca4c640 1 -- 172.21.15.16:0/447637464 >> v2:172.21.15.62:3300/0 conn(0x7fd1881531b0 msgr2=0x7fd1880afb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:39:58.817 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:58.810+0000 7fd18ca4c640 1 --2- 172.21.15.16:0/447637464 >> v2:172.21.15.62:3300/0 conn(0x7fd1881531b0 0x7fd1880afb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:58.818 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:58.810+0000 7fd18ca4c640 1 -- 172.21.15.16:0/447637464 >> v1:172.21.15.62:6789/0 conn(0x7fd188159870 legacy=0x7fd1880c8ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:39:58.818 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:58.810+0000 7fd18ca4c640 1 -- 172.21.15.16:0/447637464 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd18815d5e0 con 0x7fd1880b0080 2023-05-22T15:39:58.818 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:58.814+0000 7fd18ca4c640 1 --2- 172.21.15.16:0/447637464 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd1880b0080 0x7fd188159330 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7fd1880b0790 tx=0x7fd17801c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf3dd6128c6ebdea server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:39:58.818 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:58.814+0000 7fd17ffff640 1 -- 172.21.15.16:0/447637464 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1780aa5f0 con 0x7fd1880b0080 2023-05-22T15:39:58.819 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:58.814+0000 7fd17ffff640 1 -- 172.21.15.16:0/447637464 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1780aa750 con 0x7fd1880b0080 2023-05-22T15:39:58.819 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:58.814+0000 7fd17ffff640 1 -- 172.21.15.16:0/447637464 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1780aaae0 con 0x7fd1880b0080 2023-05-22T15:39:58.819 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:58.814+0000 7fd18f4b1640 1 -- 172.21.15.16:0/447637464 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd1880b0080 msgr2=0x7fd188159330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:58.820 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:58.814+0000 7fd18f4b1640 1 --2- 172.21.15.16:0/447637464 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd1880b0080 0x7fd188159330 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7fd1880b0790 tx=0x7fd17801c490 comp rx=0 tx=0).stop 2023-05-22T15:39:58.820 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:58.814+0000 7fd18f4b1640 1 -- 172.21.15.16:0/447637464 shutdown_connections 2023-05-22T15:39:58.820 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:58.814+0000 7fd18f4b1640 1 --2- 172.21.15.16:0/447637464 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd1880b0080 0x7fd188159330 unknown :-1 s=CLOSED pgs=885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:58.820 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:58.814+0000 7fd18f4b1640 1 --2- 172.21.15.16:0/447637464 >> v2:172.21.15.62:3300/0 conn(0x7fd1881531b0 0x7fd1880afb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:58.820 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:58.814+0000 7fd18f4b1640 1 -- 172.21.15.16:0/447637464 >> 172.21.15.16:0/447637464 conn(0x7fd188069720 msgr2=0x7fd188069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:39:58.820 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:58.814+0000 7fd18f4b1640 1 -- 172.21.15.16:0/447637464 shutdown_connections 2023-05-22T15:39:58.820 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:58.814+0000 7fd18f4b1640 1 -- 172.21.15.16:0/447637464 wait complete. 2023-05-22T15:39:58.821 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:58.814+0000 7fd18f4b1640 1 Processor -- start 2023-05-22T15:39:58.821 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:58.814+0000 7fd18f4b1640 1 -- start start 2023-05-22T15:39:58.822 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:58.814+0000 7fd18f4b1640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd1880b0080 0x7fd18814d7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:58.822 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:58.814+0000 7fd18f4b1640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd1881531b0 0x7fd18813f350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:58.822 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:58.814+0000 7fd18f4b1640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd188159870 0x7fd1880ae500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:58.822 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:58.814+0000 7fd18f4b1640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1880aea40 con 0x7fd1880b0080 2023-05-22T15:39:58.822 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:58.814+0000 7fd18f4b1640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1880aec30 con 0x7fd1881531b0 2023-05-22T15:39:58.822 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:58.814+0000 7fd18f4b1640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd18814ca30 con 0x7fd188159870 2023-05-22T15:39:58.823 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:58.818+0000 7fd18d24d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd1880b0080 0x7fd18814d7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:58.823 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:58.818+0000 7fd18d24d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd1880b0080 0x7fd18814d7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57618/0 (socket says 172.21.15.16:57618) 2023-05-22T15:39:58.823 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:58.818+0000 7fd18d24d640 1 -- 172.21.15.16:0/1457401933 learned_addr learned my addr 172.21.15.16:0/1457401933 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:39:58.824 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:58.818+0000 7fd18da4e640 1 --2- 172.21.15.16:0/1457401933 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd188159870 0x7fd1880ae500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:58.824 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:58.818+0000 7fd18ca4c640 1 --2- 172.21.15.16:0/1457401933 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd1881531b0 0x7fd18813f350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:58.824 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:58.818+0000 7fd18d24d640 1 -- 172.21.15.16:0/1457401933 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd188159870 msgr2=0x7fd1880ae500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:58.824 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:58.818+0000 7fd18d24d640 1 --2- 172.21.15.16:0/1457401933 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd188159870 0x7fd1880ae500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:58.825 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:58.818+0000 7fd18d24d640 1 -- 172.21.15.16:0/1457401933 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd1881531b0 msgr2=0x7fd18813f350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:58.825 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:58.818+0000 7fd18d24d640 1 --2- 172.21.15.16:0/1457401933 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd1881531b0 0x7fd18813f350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:58.825 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:58.818+0000 7fd18d24d640 1 -- 172.21.15.16:0/1457401933 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1780094d0 con 0x7fd1880b0080 2023-05-22T15:39:58.825 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:58.818+0000 7fd18d24d640 1 --2- 172.21.15.16:0/1457401933 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd1880b0080 0x7fd18814d7f0 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7fd18400c7f0 tx=0x7fd18400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:39:58.825 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:58.818+0000 7fd17dffb640 1 -- 172.21.15.16:0/1457401933 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd184007820 con 0x7fd1880b0080 2023-05-22T15:39:58.825 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:58.818+0000 7fd17dffb640 1 -- 172.21.15.16:0/1457401933 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd184007980 con 0x7fd1880b0080 2023-05-22T15:39:58.825 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:58.818+0000 7fd18f4b1640 1 -- 172.21.15.16:0/1457401933 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd18815d0c0 con 0x7fd1880b0080 2023-05-22T15:39:58.825 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:58.818+0000 7fd17dffb640 1 -- 172.21.15.16:0/1457401933 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd184007d30 con 0x7fd1880b0080 2023-05-22T15:39:58.826 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:58.818+0000 7fd18f4b1640 1 -- 172.21.15.16:0/1457401933 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd18815cde0 con 0x7fd1880b0080 2023-05-22T15:39:58.826 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:58.818+0000 7fd18f4b1640 1 -- 172.21.15.16:0/1457401933 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd188159c80 con 0x7fd1880b0080 2023-05-22T15:39:58.826 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:58.818+0000 7fd17dffb640 1 -- 172.21.15.16:0/1457401933 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) v1 ==== 84332+0+0 (secure 0 0 0) 0x7fd1840113f0 con 0x7fd1880b0080 2023-05-22T15:39:58.827 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:58.818+0000 7fd17dffb640 1 --2- 172.21.15.16:0/1457401933 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd16c0654d0 0x7fd16c067980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:58.827 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:58.818+0000 7fd17dffb640 1 -- 172.21.15.16:0/1457401933 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5530+0+0 (secure 0 0 0) 0x7fd1840828b0 con 0x7fd1880b0080 2023-05-22T15:39:58.827 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:58.818+0000 7fd18ca4c640 1 --2- 172.21.15.16:0/1457401933 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd16c0654d0 0x7fd16c067980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:58.827 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:58.818+0000 7fd18ca4c640 1 --2- 172.21.15.16:0/1457401933 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd16c0654d0 0x7fd16c067980 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fd1780ae000 tx=0x7fd178008040 comp rx=0 tx=0).ready entity=mgr.6240 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:39:58.829 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:58.822+0000 7fd17dffb640 1 -- 172.21.15.16:0/1457401933 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fd184050550 con 0x7fd1880b0080 2023-05-22T15:39:59.040 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.034+0000 7fd18f4b1640 1 -- 172.21.15.16:0/1457401933 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fd18804dbe0 con 0x7fd1880b0080 2023-05-22T15:39:59.041 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.034+0000 7fd17dffb640 1 -- 172.21.15.16:0/1457401933 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v17) v1 ==== 80+0+5304 (secure 0 0 0) 0x7fd184053b70 con 0x7fd1880b0080 2023-05-22T15:39:59.041 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:39:59.042 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-05-22T15:39:59.042 INFO:teuthology.orchestra.run.smithi016.stdout: "auth_dump": [ 2023-05-22T15:39:59.042 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T15:39:59.042 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "mds.a", 2023-05-22T15:39:59.042 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQCAiWtkw/K0GRAA9hvTMKMmcbgKV68GeoC4ag==", 2023-05-22T15:39:59.042 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-05-22T15:39:59.043 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2023-05-22T15:39:59.043 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow *", 2023-05-22T15:39:59.043 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow *", 2023-05-22T15:39:59.043 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-05-22T15:39:59.043 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T15:39:59.043 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:39:59.043 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T15:39:59.043 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "mds.b", 2023-05-22T15:39:59.044 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQCAiWtkoFzfJRAAxCiB0MGM5apMVA81WDq+dA==", 2023-05-22T15:39:59.044 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-05-22T15:39:59.044 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2023-05-22T15:39:59.044 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow *", 2023-05-22T15:39:59.045 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow *", 2023-05-22T15:39:59.045 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-05-22T15:39:59.045 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T15:39:59.045 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:39:59.045 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T15:39:59.045 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "mds.c", 2023-05-22T15:39:59.045 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQCAiWtkvgOKHxAAMuASPtMqwyoqpZ5I+MRS7A==", 2023-05-22T15:39:59.046 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-05-22T15:39:59.046 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2023-05-22T15:39:59.046 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow *", 2023-05-22T15:39:59.046 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow *", 2023-05-22T15:39:59.046 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-05-22T15:39:59.046 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T15:39:59.046 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:39:59.046 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T15:39:59.047 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "mds.d", 2023-05-22T15:39:59.047 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQCAiWtk6hPQKxAAlCtfC36vja5ftz6uOqUhHg==", 2023-05-22T15:39:59.047 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-05-22T15:39:59.047 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2023-05-22T15:39:59.047 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow *", 2023-05-22T15:39:59.047 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow *", 2023-05-22T15:39:59.047 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-05-22T15:39:59.048 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T15:39:59.048 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:39:59.048 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T15:39:59.048 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.0", 2023-05-22T15:39:59.048 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQCCiWtk6ZLoCRAAKdCQitZuPnWcIpkiXhJeww==", 2023-05-22T15:39:59.048 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-05-22T15:39:59.048 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2023-05-22T15:39:59.049 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2023-05-22T15:39:59.049 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-05-22T15:39:59.049 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T15:39:59.049 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:39:59.049 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T15:39:59.049 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.1", 2023-05-22T15:39:59.049 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQCFiWtkXtVeERAAkgPeDKuFmWXrDiCyqWOYwQ==", 2023-05-22T15:39:59.049 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-05-22T15:39:59.050 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2023-05-22T15:39:59.050 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2023-05-22T15:39:59.050 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-05-22T15:39:59.050 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T15:39:59.050 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:39:59.050 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T15:39:59.050 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.2", 2023-05-22T15:39:59.050 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQCIiWtkdGaRHRAAx3POIp3mnz7p7XcCVMXm5Q==", 2023-05-22T15:39:59.051 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-05-22T15:39:59.051 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2023-05-22T15:39:59.051 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2023-05-22T15:39:59.051 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-05-22T15:39:59.051 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T15:39:59.051 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:39:59.051 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T15:39:59.051 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.3", 2023-05-22T15:39:59.052 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQCLiWtkSBydJhAAsgc19Xaf/GOyy9nm9MLdJQ==", 2023-05-22T15:39:59.052 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-05-22T15:39:59.052 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2023-05-22T15:39:59.052 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2023-05-22T15:39:59.052 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-05-22T15:39:59.052 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T15:39:59.052 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:39:59.053 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T15:39:59.053 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.4", 2023-05-22T15:39:59.053 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQCRiWtkueYvJRAARXbHlKHrTbgTqBArRJfvBA==", 2023-05-22T15:39:59.053 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-05-22T15:39:59.054 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2023-05-22T15:39:59.054 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2023-05-22T15:39:59.054 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-05-22T15:39:59.054 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T15:39:59.054 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:39:59.054 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T15:39:59.054 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.5", 2023-05-22T15:39:59.055 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQCUiWtkH3hwLRAAUekP6y+wkJVPh+OPDiZbzQ==", 2023-05-22T15:39:59.055 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-05-22T15:39:59.055 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2023-05-22T15:39:59.055 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2023-05-22T15:39:59.055 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-05-22T15:39:59.055 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T15:39:59.055 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:39:59.055 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T15:39:59.056 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.6", 2023-05-22T15:39:59.056 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQCXiWtkLsnBNhAAQ8ilX8ZU4ncJ2JJU0I+U4w==", 2023-05-22T15:39:59.056 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-05-22T15:39:59.056 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2023-05-22T15:39:59.056 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2023-05-22T15:39:59.056 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-05-22T15:39:59.056 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T15:39:59.057 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:39:59.057 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T15:39:59.057 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.7", 2023-05-22T15:39:59.057 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQCbiWtkUvXjBBAApMWw7wdfxWq0WEQDL2qprA==", 2023-05-22T15:39:59.057 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-05-22T15:39:59.057 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2023-05-22T15:39:59.058 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2023-05-22T15:39:59.058 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-05-22T15:39:59.058 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T15:39:59.058 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:39:59.058 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T15:39:59.058 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "client.0", 2023-05-22T15:39:59.058 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQCAiWtkZwuUMRAAnYqSFiaT/QRsajdjNgTetA==", 2023-05-22T15:39:59.058 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-05-22T15:39:59.059 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2023-05-22T15:39:59.059 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow r", 2023-05-22T15:39:59.059 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow rw tag cephfs data=cephfs" 2023-05-22T15:39:59.059 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T15:39:59.059 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:39:59.059 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T15:39:59.059 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "client.1", 2023-05-22T15:39:59.060 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQCAiWtkPimmORAABzNL2rANz5TgCl3p++GVng==", 2023-05-22T15:39:59.060 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-05-22T15:39:59.060 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2023-05-22T15:39:59.060 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow r", 2023-05-22T15:39:59.060 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow rw", 2023-05-22T15:39:59.060 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow rwx" 2023-05-22T15:39:59.060 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T15:39:59.060 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:39:59.061 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T15:39:59.061 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "client.2", 2023-05-22T15:39:59.061 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQCAiWtkqkPMNhAAqugrnD4Mh/4At8QAFmzqcg==", 2023-05-22T15:39:59.061 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-05-22T15:39:59.061 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2023-05-22T15:39:59.061 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow r", 2023-05-22T15:39:59.061 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow rw", 2023-05-22T15:39:59.061 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow rwx" 2023-05-22T15:39:59.062 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T15:39:59.062 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:39:59.062 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T15:39:59.062 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "client.3", 2023-05-22T15:39:59.062 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQCBiWtkpvlWAxAAPuedivXBGZGHIO6hSnz8PQ==", 2023-05-22T15:39:59.062 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-05-22T15:39:59.062 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2023-05-22T15:39:59.062 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow r", 2023-05-22T15:39:59.063 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow rw", 2023-05-22T15:39:59.063 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow rwx" 2023-05-22T15:39:59.063 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T15:39:59.063 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:39:59.063 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T15:39:59.063 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "client.admin", 2023-05-22T15:39:59.063 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQCAiWtkxG9hAhAAfnUnRrBRUFoCEtsb2riNOQ==", 2023-05-22T15:39:59.064 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-05-22T15:39:59.064 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow *", 2023-05-22T15:39:59.064 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow *", 2023-05-22T15:39:59.064 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow *", 2023-05-22T15:39:59.064 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-05-22T15:39:59.064 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T15:39:59.064 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:39:59.065 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T15:39:59.065 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "mgr.x", 2023-05-22T15:39:59.065 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQCAiWtkkaIqFhAAb/4BI6FYWOy3qOSaylXLSw==", 2023-05-22T15:39:59.065 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-05-22T15:39:59.065 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow *", 2023-05-22T15:39:59.065 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile mgr", 2023-05-22T15:39:59.065 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-05-22T15:39:59.065 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T15:39:59.066 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:39:59.066 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T15:39:59.066 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "mgr.y", 2023-05-22T15:39:59.066 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQCAiWtkhgVWExAAjDoh7cexFVfsigsYfGkmVg==", 2023-05-22T15:39:59.066 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-05-22T15:39:59.066 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow *", 2023-05-22T15:39:59.066 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile mgr", 2023-05-22T15:39:59.067 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-05-22T15:39:59.067 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T15:39:59.067 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T15:39:59.067 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-05-22T15:39:59.067 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-05-22T15:39:59.068 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.038+0000 7fd18f4b1640 1 -- 172.21.15.16:0/1457401933 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd16c0654d0 msgr2=0x7fd16c067980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:59.069 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.038+0000 7fd18f4b1640 1 --2- 172.21.15.16:0/1457401933 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd16c0654d0 0x7fd16c067980 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fd1780ae000 tx=0x7fd178008040 comp rx=0 tx=0).stop 2023-05-22T15:39:59.069 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.038+0000 7fd18f4b1640 1 -- 172.21.15.16:0/1457401933 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd1880b0080 msgr2=0x7fd18814d7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:59.069 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.038+0000 7fd18f4b1640 1 --2- 172.21.15.16:0/1457401933 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd1880b0080 0x7fd18814d7f0 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7fd18400c7f0 tx=0x7fd18400ccb0 comp rx=0 tx=0).stop 2023-05-22T15:39:59.069 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.038+0000 7fd18f4b1640 1 -- 172.21.15.16:0/1457401933 shutdown_connections 2023-05-22T15:39:59.069 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.038+0000 7fd18f4b1640 1 --2- 172.21.15.16:0/1457401933 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd16c0654d0 0x7fd16c067980 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:59.069 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.038+0000 7fd18f4b1640 1 --2- 172.21.15.16:0/1457401933 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd188159870 0x7fd1880ae500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:59.069 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.038+0000 7fd18f4b1640 1 --2- 172.21.15.16:0/1457401933 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd1881531b0 0x7fd18813f350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:59.070 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.038+0000 7fd18f4b1640 1 --2- 172.21.15.16:0/1457401933 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd1880b0080 0x7fd18814d7f0 unknown :-1 s=CLOSED pgs=886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:59.070 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.038+0000 7fd18f4b1640 1 -- 172.21.15.16:0/1457401933 >> 172.21.15.16:0/1457401933 conn(0x7fd188069720 msgr2=0x7fd1880ce020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:39:59.070 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.038+0000 7fd18f4b1640 1 -- 172.21.15.16:0/1457401933 shutdown_connections 2023-05-22T15:39:59.070 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.038+0000 7fd18f4b1640 1 -- 172.21.15.16:0/1457401933 wait complete. 2023-05-22T15:39:59.070 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-05-22T15:39:59.200 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.194+0000 7f4f59497640 1 Processor -- start 2023-05-22T15:39:59.200 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.194+0000 7f4f59497640 1 -- start start 2023-05-22T15:39:59.201 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.194+0000 7f4f59497640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4f54071500 0x7f4f5415e460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:59.202 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.194+0000 7f4f59497640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f540b0650 con 0x7f4f540c9b40 2023-05-22T15:39:59.202 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.194+0000 7f4f59497640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f540b0370 con 0x7f4f540ca4e0 2023-05-22T15:39:59.202 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.194+0000 7f4f59497640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f5415d980 con 0x7f4f54071500 2023-05-22T15:39:59.202 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.194+0000 7f4f4bfff640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f4f540c9b40 0x7f4f54070df0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:41288/0 (socket says 172.21.15.16:41288) 2023-05-22T15:39:59.202 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.194+0000 7f4f4bfff640 1 -- 172.21.15.16:0/3901635503 learned_addr learned my addr 172.21.15.16:0/3901635503 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:39:59.202 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.194+0000 7f4f537fe640 1 --2- 172.21.15.16:0/3901635503 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4f54071500 0x7f4f5415e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:59.204 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.198+0000 7f4f527fc640 1 -- 172.21.15.16:0/3901635503 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3776557555 0 0) 0x7f4f540b0650 con 0x7f4f540c9b40 2023-05-22T15:39:59.204 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.198+0000 7f4f527fc640 1 -- 172.21.15.16:0/3901635503 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f40003530 con 0x7f4f540c9b40 2023-05-22T15:39:59.204 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.198+0000 7f4f527fc640 1 -- 172.21.15.16:0/3901635503 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 278756635 0 0) 0x7f4f40003530 con 0x7f4f540c9b40 2023-05-22T15:39:59.204 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.198+0000 7f4f527fc640 1 -- 172.21.15.16:0/3901635503 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4f54071500 msgr2=0x7f4f5415e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:39:59.205 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.198+0000 7f4f527fc640 1 --2- 172.21.15.16:0/3901635503 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4f54071500 0x7f4f5415e460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:59.205 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.198+0000 7f4f527fc640 1 -- 172.21.15.16:0/3901635503 >> v1:172.21.15.62:6789/0 conn(0x7f4f540ca4e0 legacy=0x7f4f540afb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:59.205 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.198+0000 7f4f527fc640 1 -- 172.21.15.16:0/3901635503 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f5415dc60 con 0x7f4f540c9b40 2023-05-22T15:39:59.205 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.198+0000 7f4f527fc640 1 -- 172.21.15.16:0/3901635503 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7f4f3c0016a0 con 0x7f4f540c9b40 2023-05-22T15:39:59.205 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.198+0000 7f4f527fc640 1 -- 172.21.15.16:0/3901635503 >> v1:172.21.15.16:6789/0 conn(0x7f4f540c9b40 legacy=0x7f4f54070df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:59.205 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.198+0000 7f4f527fc640 1 --2- 172.21.15.16:0/3901635503 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4f400039b0 0x7f4f40003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:59.206 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.198+0000 7f4f527fc640 1 --2- 172.21.15.16:0/3901635503 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4f54071500 0x7f4f400047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:59.206 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.198+0000 7f4f527fc640 1 --2- 172.21.15.16:0/3901635503 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4f40004d10 0x7f4f40005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:59.206 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.198+0000 7f4f527fc640 1 -- 172.21.15.16:0/3901635503 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f5415d980 con 0x7f4f40004d10 2023-05-22T15:39:59.206 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.198+0000 7f4f527fc640 1 -- 172.21.15.16:0/3901635503 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f4000ba40 con 0x7f4f400039b0 2023-05-22T15:39:59.206 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.198+0000 7f4f527fc640 1 -- 172.21.15.16:0/3901635503 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f40010010 con 0x7f4f54071500 2023-05-22T15:39:59.206 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.198+0000 7f4f537fe640 1 --2- 172.21.15.16:0/3901635503 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4f40004d10 0x7f4f40005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:59.206 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.198+0000 7f4f52ffd640 1 --2- 172.21.15.16:0/3901635503 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4f400039b0 0x7f4f40003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:59.207 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.198+0000 7f4f4bfff640 1 --2- 172.21.15.16:0/3901635503 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4f54071500 0x7f4f400047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:59.207 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.198+0000 7f4f537fe640 1 -- 172.21.15.16:0/3901635503 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4f54071500 msgr2=0x7f4f400047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:39:59.207 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.198+0000 7f4f537fe640 1 --2- 172.21.15.16:0/3901635503 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4f54071500 0x7f4f400047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:59.207 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.198+0000 7f4f537fe640 1 -- 172.21.15.16:0/3901635503 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4f400039b0 msgr2=0x7f4f40003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:39:59.207 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.198+0000 7f4f537fe640 1 --2- 172.21.15.16:0/3901635503 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4f400039b0 0x7f4f40003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:59.207 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.198+0000 7f4f537fe640 1 -- 172.21.15.16:0/3901635503 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f40003530 con 0x7f4f40004d10 2023-05-22T15:39:59.211 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.198+0000 7f4f537fe640 1 --2- 172.21.15.16:0/3901635503 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4f40004d10 0x7f4f40005180 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f4f4c009fc0 tx=0x7f4f4c01cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=732a06b14b94822f server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:39:59.212 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.198+0000 7f4f527fc640 1 -- 172.21.15.16:0/3901635503 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f4c0a5c70 con 0x7f4f40004d10 2023-05-22T15:39:59.212 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.198+0000 7f4f527fc640 1 -- 172.21.15.16:0/3901635503 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f4c00f040 con 0x7f4f40004d10 2023-05-22T15:39:59.212 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.198+0000 7f4f527fc640 1 -- 172.21.15.16:0/3901635503 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f4c0ad440 con 0x7f4f40004d10 2023-05-22T15:39:59.212 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.198+0000 7f4f59497640 1 -- 172.21.15.16:0/3901635503 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4f40004d10 msgr2=0x7f4f40005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:59.212 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.198+0000 7f4f59497640 1 --2- 172.21.15.16:0/3901635503 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4f40004d10 0x7f4f40005180 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f4f4c009fc0 tx=0x7f4f4c01cc60 comp rx=0 tx=0).stop 2023-05-22T15:39:59.213 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.198+0000 7f4f537fe640 1 -- 172.21.15.16:0/3901635503 reap_dead start 2023-05-22T15:39:59.213 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.198+0000 7f4f59497640 1 -- 172.21.15.16:0/3901635503 shutdown_connections 2023-05-22T15:39:59.213 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.198+0000 7f4f59497640 1 -- 172.21.15.16:0/3901635503 >> 172.21.15.16:0/3901635503 conn(0x7f4f54069720 msgr2=0x7f4f54069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:39:59.213 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.198+0000 7f4f59497640 1 -- 172.21.15.16:0/3901635503 shutdown_connections 2023-05-22T15:39:59.213 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.198+0000 7f4f59497640 1 -- 172.21.15.16:0/3901635503 wait complete. 2023-05-22T15:39:59.214 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.202+0000 7f4f59497640 1 Processor -- start 2023-05-22T15:39:59.214 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.202+0000 7f4f59497640 1 -- start start 2023-05-22T15:39:59.214 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.202+0000 7f4f59497640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4f40004d10 0x7f4f5414f070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:59.214 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.202+0000 7f4f59497640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4f54154d20 0x7f4f54155170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:59.214 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.202+0000 7f4f59497640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4f54155cb0 0x7f4f54156120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:59.214 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.202+0000 7f4f59497640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f5414fac0 con 0x7f4f54155cb0 2023-05-22T15:39:59.214 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.202+0000 7f4f59497640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f541556b0 con 0x7f4f40004d10 2023-05-22T15:39:59.214 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.202+0000 7f4f59497640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f5414f5b0 con 0x7f4f54154d20 2023-05-22T15:39:59.215 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.202+0000 7f4f537fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4f54155cb0 0x7f4f54156120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:59.215 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.202+0000 7f4f537fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4f54155cb0 0x7f4f54156120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34168/0 (socket says 172.21.15.16:34168) 2023-05-22T15:39:59.215 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.202+0000 7f4f537fe640 1 -- 172.21.15.16:0/4290276546 learned_addr learned my addr 172.21.15.16:0/4290276546 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:39:59.215 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.202+0000 7f4f52ffd640 1 --2- 172.21.15.16:0/4290276546 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4f40004d10 0x7f4f5414f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:59.215 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.202+0000 7f4f4bfff640 1 --2- 172.21.15.16:0/4290276546 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4f54154d20 0x7f4f54155170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:59.215 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.202+0000 7f4f537fe640 1 -- 172.21.15.16:0/4290276546 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4f54154d20 msgr2=0x7f4f54155170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:59.215 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.202+0000 7f4f537fe640 1 --2- 172.21.15.16:0/4290276546 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4f54154d20 0x7f4f54155170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:59.216 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.202+0000 7f4f537fe640 1 -- 172.21.15.16:0/4290276546 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4f40004d10 msgr2=0x7f4f5414f070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:59.216 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.202+0000 7f4f537fe640 1 --2- 172.21.15.16:0/4290276546 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4f40004d10 0x7f4f5414f070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:59.216 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.202+0000 7f4f537fe640 1 -- 172.21.15.16:0/4290276546 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f540ca8b0 con 0x7f4f54155cb0 2023-05-22T15:39:59.216 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.202+0000 7f4f4bfff640 1 --2- 172.21.15.16:0/4290276546 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4f54154d20 0x7f4f54155170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T15:39:59.216 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.202+0000 7f4f52ffd640 1 --2- 172.21.15.16:0/4290276546 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4f40004d10 0x7f4f5414f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T15:39:59.216 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.202+0000 7f4f537fe640 1 --2- 172.21.15.16:0/4290276546 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4f54155cb0 0x7f4f54156120 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7f4f4c0b0040 tx=0x7f4f4c009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:39:59.216 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.202+0000 7f4f4b7fe640 1 -- 172.21.15.16:0/4290276546 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f4c0039a0 con 0x7f4f54155cb0 2023-05-22T15:39:59.216 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.202+0000 7f4f4b7fe640 1 -- 172.21.15.16:0/4290276546 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f4c003b20 con 0x7f4f54155cb0 2023-05-22T15:39:59.217 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.202+0000 7f4f59497640 1 -- 172.21.15.16:0/4290276546 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f5415df40 con 0x7f4f54155cb0 2023-05-22T15:39:59.217 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.202+0000 7f4f4b7fe640 1 -- 172.21.15.16:0/4290276546 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f4c0ae890 con 0x7f4f54155cb0 2023-05-22T15:39:59.217 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.202+0000 7f4f59497640 1 -- 172.21.15.16:0/4290276546 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f5415e460 con 0x7f4f54155cb0 2023-05-22T15:39:59.217 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.202+0000 7f4f59497640 1 -- 172.21.15.16:0/4290276546 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f5404dbe0 con 0x7f4f54155cb0 2023-05-22T15:39:59.217 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.206+0000 7f4f4b7fe640 1 -- 172.21.15.16:0/4290276546 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f4f4c0b4040 con 0x7f4f54155cb0 2023-05-22T15:39:59.217 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.206+0000 7f4f4b7fe640 1 --2- 172.21.15.16:0/4290276546 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f4f340654a0 0x7f4f34067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:59.217 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.206+0000 7f4f4b7fe640 1 -- 172.21.15.16:0/4290276546 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5530+0+0 (secure 0 0 0) 0x7f4f4c11c8c0 con 0x7f4f54155cb0 2023-05-22T15:39:59.218 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.206+0000 7f4f52ffd640 1 --2- 172.21.15.16:0/4290276546 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f4f340654a0 0x7f4f34067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:59.218 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.206+0000 7f4f52ffd640 1 --2- 172.21.15.16:0/4290276546 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f4f340654a0 0x7f4f34067950 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f4f44005630 tx=0x7f4f44009040 comp rx=0 tx=0).ready entity=mgr.6240 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:39:59.218 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.210+0000 7f4f4b7fe640 1 -- 172.21.15.16:0/4290276546 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f4f4c0ea640 con 0x7f4f54155cb0 2023-05-22T15:39:59.369 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.362+0000 7f4f59497640 1 -- 172.21.15.16:0/4290276546 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4f540cb010 con 0x7f4f54155cb0 2023-05-22T15:39:59.371 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.366+0000 7f4f4b7fe640 1 -- 172.21.15.16:0/4290276546 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v69) v1 ==== 74+0+14328 (secure 0 0 0) 0x7f4f4c0bc090 con 0x7f4f54155cb0 2023-05-22T15:39:59.372 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:39:59.372 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":69,"fsid":"b2719834-7847-4d98-90a6-f65d57f85c49","created":"2023-05-22T15:26:25.259314+0000","modified":"2023-05-22T15:39:58.653150+0000","last_up_change":"2023-05-22T15:26:36.448524+0000","last_in_change":"2023-05-22T15:26:27.675668+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-05-22T15:26:38.019781+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-05-22T15:26:43.570514+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9e7ff854-b25f-4cbb-a5c2-1799d9c53d2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":37021},{"type":"v1","addr":"172.21.15.16:6825","nonce":37021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":37021},{"type":"v1","addr":"172.21.15.16:6827","nonce":37021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":37021},{"type":"v1","addr":"172.21.15.16:6831","nonce":37021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":37021},{"type":"v1","addr":"172.21.15.16:6829","nonce":37021}]},"public_addr":"172.21.15.16:6825/37021","cluster_addr":"172.21.15.16:6827/37021","heartbeat_back_addr":"172.21.15.16:6831/37021","heartbeat_front_addr":"172.21.15.16:6829/37021","state":["exists","up"]},{"osd":1,"uuid":"4702fbc9-a927-4565-aca5-41dc69cc8264","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":37008},{"type":"v1","addr":"172.21.15.16:6801","nonce":37008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":37008},{"type":"v1","addr":"172.21.15.16:6803","nonce":37008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":37008},{"type":"v1","addr":"172.21.15.16:6807","nonce":37008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":37008},{"type":"v1","addr":"172.21.15.16:6805","nonce":37008}]},"public_addr":"172.21.15.16:6801/37008","cluster_addr":"172.21.15.16:6803/37008","heartbeat_back_addr":"172.21.15.16:6807/37008","heartbeat_front_addr":"172.21.15.16:6805/37008","state":["exists","up"]},{"osd":2,"uuid":"1e7bba46-4eba-47ef-bb81-5e2d371638ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":37009},{"type":"v1","addr":"172.21.15.16:6809","nonce":37009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":37009},{"type":"v1","addr":"172.21.15.16:6811","nonce":37009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":37009},{"type":"v1","addr":"172.21.15.16:6815","nonce":37009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":37009},{"type":"v1","addr":"172.21.15.16:6813","nonce":37009}]},"public_addr":"172.21.15.16:6809/37009","cluster_addr":"172.21.15.16:6811/37009","heartbeat_back_addr":"172.21.15.16:6815/37009","heartbeat_front_addr":"172.21.15.16:6813/37009","state":["exists","up"]},{"osd":3,"uuid":"383cacc0-5a88-4904-990e-498ca59ef48a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":37020},{"type":"v1","addr":"172.21.15.16:6817","nonce":37020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":37020},{"type":"v1","addr":"172.21.15.16:6819","nonce":37020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":37020},{"type":"v1","addr":"172.21.15.16:6823","nonce":37020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":37020},{"type":"v1","addr":"172.21.15.16:6821","nonce":37020}]},"public_addr":"172.21.15.16:6817/37020","cluster_addr":"172.21.15.16:6819/37020","heartbeat_back_addr":"172.21.15.16:6823/37020","heartbeat_front_addr":"172.21.15.16:6821/37020","state":["exists","up"]},{"osd":4,"uuid":"970500ef-95b7-42c1-b080-eed4fae7013d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":37353},{"type":"v1","addr":"172.21.15.62:6817","nonce":37353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6819","nonce":37353},{"type":"v1","addr":"172.21.15.62:6822","nonce":37353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37353},{"type":"v1","addr":"172.21.15.62:6831","nonce":37353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37353},{"type":"v1","addr":"172.21.15.62:6826","nonce":37353}]},"public_addr":"172.21.15.62:6817/37353","cluster_addr":"172.21.15.62:6822/37353","heartbeat_back_addr":"172.21.15.62:6831/37353","heartbeat_front_addr":"172.21.15.62:6826/37353","state":["exists","up"]},{"osd":5,"uuid":"022d67db-d4d8-445b-b1b9-8b446a00f5c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":37352},{"type":"v1","addr":"172.21.15.62:6801","nonce":37352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":37352},{"type":"v1","addr":"172.21.15.62:6803","nonce":37352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":37352},{"type":"v1","addr":"172.21.15.62:6807","nonce":37352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":37352},{"type":"v1","addr":"172.21.15.62:6805","nonce":37352}]},"public_addr":"172.21.15.62:6801/37352","cluster_addr":"172.21.15.62:6803/37352","heartbeat_back_addr":"172.21.15.62:6807/37352","heartbeat_front_addr":"172.21.15.62:6805/37352","state":["exists","up"]},{"osd":6,"uuid":"42bd372b-6688-4a1b-8407-e16e8c266e33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":37354},{"type":"v1","addr":"172.21.15.62:6820","nonce":37354}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":37354},{"type":"v1","addr":"172.21.15.62:6823","nonce":37354}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6829","nonce":37354},{"type":"v1","addr":"172.21.15.62:6830","nonce":37354}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6825","nonce":37354},{"type":"v1","addr":"172.21.15.62:6827","nonce":37354}]},"public_addr":"172.21.15.62:6820/37354","cluster_addr":"172.21.15.62:6823/37354","heartbeat_back_addr":"172.21.15.62:6830/37354","heartbeat_front_addr":"172.21.15.62:6827/37354","state":["exists","up"]},{"osd":7,"uuid":"ac70b2a3-f41b-4bf2-b227-7f2cbb55426d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37355},{"type":"v1","addr":"172.21.15.62:6809","nonce":37355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37355},{"type":"v1","addr":"172.21.15.62:6811","nonce":37355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":37355},{"type":"v1","addr":"172.21.15.62:6815","nonce":37355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":37355},{"type":"v1","addr":"172.21.15.62:6813","nonce":37355}]},"public_addr":"172.21.15.62:6809/37355","cluster_addr":"172.21.15.62:6811/37355","heartbeat_back_addr":"172.21.15.62:6815/37355","heartbeat_front_addr":"172.21.15.62:6813/37355","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.368896+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.107273+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.276888+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.287502+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.124791+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.251937+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.255679+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.232797+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-05-22T15:39:59.373 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.366+0000 7f4f59497640 1 -- 172.21.15.16:0/4290276546 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f4f340654a0 msgr2=0x7f4f34067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:59.373 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.366+0000 7f4f59497640 1 --2- 172.21.15.16:0/4290276546 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f4f340654a0 0x7f4f34067950 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f4f44005630 tx=0x7f4f44009040 comp rx=0 tx=0).stop 2023-05-22T15:39:59.373 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.366+0000 7f4f59497640 1 -- 172.21.15.16:0/4290276546 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4f54155cb0 msgr2=0x7f4f54156120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:59.374 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.366+0000 7f4f59497640 1 --2- 172.21.15.16:0/4290276546 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4f54155cb0 0x7f4f54156120 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7f4f4c0b0040 tx=0x7f4f4c009750 comp rx=0 tx=0).stop 2023-05-22T15:39:59.374 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.366+0000 7f4f59497640 1 -- 172.21.15.16:0/4290276546 shutdown_connections 2023-05-22T15:39:59.374 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.366+0000 7f4f59497640 1 --2- 172.21.15.16:0/4290276546 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4f54155cb0 0x7f4f54156120 unknown :-1 s=CLOSED pgs=889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:59.374 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.366+0000 7f4f59497640 1 --2- 172.21.15.16:0/4290276546 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f4f340654a0 0x7f4f34067950 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:59.374 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.366+0000 7f4f59497640 1 --2- 172.21.15.16:0/4290276546 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4f54154d20 0x7f4f54155170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:59.374 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.366+0000 7f4f59497640 1 --2- 172.21.15.16:0/4290276546 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4f40004d10 0x7f4f5414f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:59.374 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.366+0000 7f4f59497640 1 -- 172.21.15.16:0/4290276546 >> 172.21.15.16:0/4290276546 conn(0x7f4f54069720 msgr2=0x7f4f540af5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:39:59.374 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.366+0000 7f4f59497640 1 -- 172.21.15.16:0/4290276546 shutdown_connections 2023-05-22T15:39:59.375 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.366+0000 7f4f59497640 1 -- 172.21.15.16:0/4290276546 wait complete. 2023-05-22T15:39:59.390 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-05-22T15:26:38.019781+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-05-22T15:26:43.570514+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-05-22T15:39:59.391 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-05-22T15:39:59.519 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.514+0000 7f5960648640 1 Processor -- start 2023-05-22T15:39:59.519 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.514+0000 7f5960648640 1 -- start start 2023-05-22T15:39:59.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.514+0000 7f5960648640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f595815cfa0 0x7f595806db00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:59.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.514+0000 7f5960648640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f59580cf870 0x7f59580b1fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:59.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.514+0000 7f5960648640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f595806e040 con 0x7f595806e740 2023-05-22T15:39:59.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.514+0000 7f5960648640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59580cb9f0 con 0x7f595815cfa0 2023-05-22T15:39:59.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.514+0000 7f5960648640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59580b24f0 con 0x7f59580cf870 2023-05-22T15:39:59.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.514+0000 7f595e3e4640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f595815cfa0 0x7f595806db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:59.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.514+0000 7f595e3e4640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f595815cfa0 0x7f595806db00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34176/0 (socket says 172.21.15.16:34176) 2023-05-22T15:39:59.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.514+0000 7f595e3e4640 1 -- 172.21.15.16:0/3430486087 learned_addr learned my addr 172.21.15.16:0/3430486087 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:39:59.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.514+0000 7f595dbe3640 1 --2- 172.21.15.16:0/3430486087 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f59580cf870 0x7f59580b1fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:59.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.514+0000 7f595e3e4640 1 -- 172.21.15.16:0/3430486087 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f59580cf870 msgr2=0x7f59580b1fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:39:59.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.514+0000 7f595e3e4640 1 --2- 172.21.15.16:0/3430486087 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f59580cf870 0x7f59580b1fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:59.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.514+0000 7f595e3e4640 1 -- 172.21.15.16:0/3430486087 >> v1:172.21.15.62:6789/0 conn(0x7f595806e740 legacy=0x7f5958070de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:59.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.514+0000 7f595e3e4640 1 -- 172.21.15.16:0/3430486087 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f595806e490 con 0x7f595815cfa0 2023-05-22T15:39:59.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.518+0000 7f595e3e4640 1 --2- 172.21.15.16:0/3430486087 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f595815cfa0 0x7f595806db00 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7f5954009870 tx=0x7f595401c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5052348ec8a9b753 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:39:59.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.518+0000 7f595d3e2640 1 -- 172.21.15.16:0/3430486087 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59540ab680 con 0x7f595815cfa0 2023-05-22T15:39:59.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.518+0000 7f595d3e2640 1 -- 172.21.15.16:0/3430486087 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59540ab7e0 con 0x7f595815cfa0 2023-05-22T15:39:59.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.518+0000 7f595d3e2640 1 -- 172.21.15.16:0/3430486087 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59540abb70 con 0x7f595815cfa0 2023-05-22T15:39:59.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.518+0000 7f5960648640 1 -- 172.21.15.16:0/3430486087 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f595815cfa0 msgr2=0x7f595806db00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:59.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.518+0000 7f5960648640 1 --2- 172.21.15.16:0/3430486087 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f595815cfa0 0x7f595806db00 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7f5954009870 tx=0x7f595401c480 comp rx=0 tx=0).stop 2023-05-22T15:39:59.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.518+0000 7f5960648640 1 -- 172.21.15.16:0/3430486087 shutdown_connections 2023-05-22T15:39:59.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.518+0000 7f5960648640 1 --2- 172.21.15.16:0/3430486087 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f59580cf870 0x7f59580b1fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:59.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.518+0000 7f5960648640 1 --2- 172.21.15.16:0/3430486087 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f595815cfa0 0x7f595806db00 unknown :-1 s=CLOSED pgs=890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:59.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.518+0000 7f5960648640 1 -- 172.21.15.16:0/3430486087 >> 172.21.15.16:0/3430486087 conn(0x7f59580cc870 msgr2=0x7f59580cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:39:59.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.518+0000 7f5960648640 1 -- 172.21.15.16:0/3430486087 shutdown_connections 2023-05-22T15:39:59.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.518+0000 7f5960648640 1 -- 172.21.15.16:0/3430486087 wait complete. 2023-05-22T15:39:59.526 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.518+0000 7f5960648640 1 Processor -- start 2023-05-22T15:39:59.526 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.518+0000 7f5960648640 1 -- start start 2023-05-22T15:39:59.526 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.518+0000 7f5960648640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f595806e740 0x7f595815bab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:59.527 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.518+0000 7f5960648640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f59580cf870 0x7f595815bff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:59.527 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.518+0000 7f5960648640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f595815cfa0 0x7f5958151f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:59.527 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.518+0000 7f5960648640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59580cd4b0 con 0x7f595815cfa0 2023-05-22T15:39:59.527 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.518+0000 7f5960648640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59580cd790 con 0x7f59580cf870 2023-05-22T15:39:59.527 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.518+0000 7f5960648640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59580cda70 con 0x7f595806e740 2023-05-22T15:39:59.527 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.518+0000 7f595ebe5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f595815cfa0 0x7f5958151f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:59.527 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.518+0000 7f595ebe5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f595815cfa0 0x7f5958151f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34182/0 (socket says 172.21.15.16:34182) 2023-05-22T15:39:59.527 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.518+0000 7f595ebe5640 1 -- 172.21.15.16:0/3191886395 learned_addr learned my addr 172.21.15.16:0/3191886395 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:39:59.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.518+0000 7f595e3e4640 1 --2- 172.21.15.16:0/3191886395 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f595806e740 0x7f595815bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:59.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.518+0000 7f595dbe3640 1 --2- 172.21.15.16:0/3191886395 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f59580cf870 0x7f595815bff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:59.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.518+0000 7f595ebe5640 1 -- 172.21.15.16:0/3191886395 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f595806e740 msgr2=0x7f595815bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:59.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.518+0000 7f595ebe5640 1 --2- 172.21.15.16:0/3191886395 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f595806e740 0x7f595815bab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:59.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.518+0000 7f595ebe5640 1 -- 172.21.15.16:0/3191886395 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f59580cf870 msgr2=0x7f595815bff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:59.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.518+0000 7f595ebe5640 1 --2- 172.21.15.16:0/3191886395 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f59580cf870 0x7f595815bff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:59.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.518+0000 7f595ebe5640 1 -- 172.21.15.16:0/3191886395 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59540094d0 con 0x7f595815cfa0 2023-05-22T15:39:59.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.518+0000 7f595e3e4640 1 --2- 172.21.15.16:0/3191886395 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f595806e740 0x7f595815bab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:39:59.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.518+0000 7f595dbe3640 1 --2- 172.21.15.16:0/3191886395 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f59580cf870 0x7f595815bff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:39:59.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.518+0000 7f595ebe5640 1 --2- 172.21.15.16:0/3191886395 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f595815cfa0 0x7f5958151f10 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7f595000c770 tx=0x7f595000cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:39:59.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.518+0000 7f594effd640 1 -- 172.21.15.16:0/3191886395 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5950003b80 con 0x7f595815cfa0 2023-05-22T15:39:59.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.518+0000 7f594effd640 1 -- 172.21.15.16:0/3191886395 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5950003ce0 con 0x7f595815cfa0 2023-05-22T15:39:59.530 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.518+0000 7f5960648640 1 -- 172.21.15.16:0/3191886395 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59580cb610 con 0x7f595815cfa0 2023-05-22T15:39:59.530 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.518+0000 7f594effd640 1 -- 172.21.15.16:0/3191886395 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f595000aa90 con 0x7f595815cfa0 2023-05-22T15:39:59.530 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.522+0000 7f5960648640 1 -- 172.21.15.16:0/3191886395 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59580cb330 con 0x7f595815cfa0 2023-05-22T15:39:59.531 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.522+0000 7f5960648640 1 -- 172.21.15.16:0/3191886395 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5958071900 con 0x7f595815cfa0 2023-05-22T15:39:59.531 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.522+0000 7f594effd640 1 -- 172.21.15.16:0/3191886395 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f595000abf0 con 0x7f595815cfa0 2023-05-22T15:39:59.531 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.522+0000 7f594effd640 1 --2- 172.21.15.16:0/3191886395 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f593c0654a0 0x7f593c067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:59.531 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.522+0000 7f594effd640 1 -- 172.21.15.16:0/3191886395 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5530+0+0 (secure 0 0 0) 0x7f59500824a0 con 0x7f595815cfa0 2023-05-22T15:39:59.531 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.522+0000 7f595e3e4640 1 --2- 172.21.15.16:0/3191886395 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f593c0654a0 0x7f593c067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:59.531 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.522+0000 7f595e3e4640 1 --2- 172.21.15.16:0/3191886395 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f593c0654a0 0x7f593c067950 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f59540af000 tx=0x7f5954008040 comp rx=0 tx=0).ready entity=mgr.6240 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:39:59.532 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.526+0000 7f594effd640 1 -- 172.21.15.16:0/3191886395 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f5950050220 con 0x7f595815cfa0 2023-05-22T15:39:59.686 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.678+0000 7f5960648640 1 -- 172.21.15.16:0/3191886395 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f595806e040 con 0x7f595815cfa0 2023-05-22T15:39:59.687 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.682+0000 7f594effd640 1 -- 172.21.15.16:0/3191886395 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v69) v1 ==== 93+0+10 (secure 0 0 0) 0x7f595806e040 con 0x7f595815cfa0 2023-05-22T15:39:59.687 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2023-05-22T15:39:59.689 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.682+0000 7f5960648640 1 -- 172.21.15.16:0/3191886395 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f593c0654a0 msgr2=0x7f593c067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:59.689 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.682+0000 7f5960648640 1 --2- 172.21.15.16:0/3191886395 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f593c0654a0 0x7f593c067950 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f59540af000 tx=0x7f5954008040 comp rx=0 tx=0).stop 2023-05-22T15:39:59.689 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.682+0000 7f5960648640 1 -- 172.21.15.16:0/3191886395 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f595815cfa0 msgr2=0x7f5958151f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:59.689 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.682+0000 7f5960648640 1 --2- 172.21.15.16:0/3191886395 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f595815cfa0 0x7f5958151f10 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7f595000c770 tx=0x7f595000cc30 comp rx=0 tx=0).stop 2023-05-22T15:39:59.689 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.682+0000 7f5960648640 1 -- 172.21.15.16:0/3191886395 shutdown_connections 2023-05-22T15:39:59.689 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.682+0000 7f5960648640 1 --2- 172.21.15.16:0/3191886395 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f595815cfa0 0x7f5958151f10 unknown :-1 s=CLOSED pgs=891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:59.690 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.682+0000 7f5960648640 1 --2- 172.21.15.16:0/3191886395 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f59580cf870 0x7f595815bff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:59.690 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.682+0000 7f5960648640 1 --2- 172.21.15.16:0/3191886395 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f593c0654a0 0x7f593c067950 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:59.690 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.682+0000 7f5960648640 1 --2- 172.21.15.16:0/3191886395 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f595806e740 0x7f595815bab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:59.690 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.682+0000 7f5960648640 1 -- 172.21.15.16:0/3191886395 >> 172.21.15.16:0/3191886395 conn(0x7f59580cc870 msgr2=0x7f59580b2750 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:39:59.690 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.682+0000 7f5960648640 1 -- 172.21.15.16:0/3191886395 shutdown_connections 2023-05-22T15:39:59.690 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.682+0000 7f5960648640 1 -- 172.21.15.16:0/3191886395 wait complete. 2023-05-22T15:39:59.706 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-05-22T15:39:59.827 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.822+0000 7f250c165640 1 Processor -- start 2023-05-22T15:39:59.827 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.822+0000 7f250c165640 1 -- start start 2023-05-22T15:39:59.828 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.822+0000 7f250c165640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f2504160170 0x7f25040c8660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:59.828 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.822+0000 7f250c165640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f25040c1380 0x7f25040c9e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:59.828 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.822+0000 7f250c165640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2504164090 con 0x7f25040ca360 2023-05-22T15:39:59.828 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.822+0000 7f250c165640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2504164370 con 0x7f2504160170 2023-05-22T15:39:59.828 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.822+0000 7f250c165640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2504164650 con 0x7f25040c1380 2023-05-22T15:39:59.829 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.822+0000 7f250a702640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f25040ca360 0x7f25040cbbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:41298/0 (socket says 172.21.15.16:41298) 2023-05-22T15:39:59.829 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.822+0000 7f250a702640 1 -- 172.21.15.16:0/813148100 learned_addr learned my addr 172.21.15.16:0/813148100 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:39:59.830 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.822+0000 7f2509700640 1 --2- 172.21.15.16:0/813148100 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f25040c1380 0x7f25040c9e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:59.830 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.822+0000 7f2509f01640 1 --2- 172.21.15.16:0/813148100 >> v2:172.21.15.62:3300/0 conn(0x7f2504160170 0x7f25040c8660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:59.831 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.822+0000 7f2508eff640 1 -- 172.21.15.16:0/813148100 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 216562215 0 0) 0x7f2504164090 con 0x7f25040ca360 2023-05-22T15:39:59.831 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.822+0000 7f2508eff640 1 -- 172.21.15.16:0/813148100 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f24ec003530 con 0x7f25040ca360 2023-05-22T15:39:59.831 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.822+0000 7f2508eff640 1 -- 172.21.15.16:0/813148100 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3689256386 0 0) 0x7f24ec003530 con 0x7f25040ca360 2023-05-22T15:39:59.831 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.822+0000 7f2508eff640 1 -- 172.21.15.16:0/813148100 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f25040c1380 msgr2=0x7f25040c9e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:39:59.831 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.822+0000 7f2508eff640 1 --2- 172.21.15.16:0/813148100 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f25040c1380 0x7f25040c9e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:59.831 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.822+0000 7f2508eff640 1 -- 172.21.15.16:0/813148100 >> v2:172.21.15.62:3300/0 conn(0x7f2504160170 msgr2=0x7f25040c8660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:39:59.831 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.822+0000 7f2508eff640 1 --2- 172.21.15.16:0/813148100 >> v2:172.21.15.62:3300/0 conn(0x7f2504160170 0x7f25040c8660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:59.832 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.822+0000 7f2508eff640 1 -- 172.21.15.16:0/813148100 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2504164930 con 0x7f25040ca360 2023-05-22T15:39:59.832 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.822+0000 7f2508eff640 1 -- 172.21.15.16:0/813148100 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7f24fc001680 con 0x7f25040ca360 2023-05-22T15:39:59.832 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.822+0000 7f2508eff640 1 -- 172.21.15.16:0/813148100 >> v1:172.21.15.16:6789/0 conn(0x7f25040ca360 legacy=0x7f25040cbbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:59.832 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.822+0000 7f2508eff640 1 --2- 172.21.15.16:0/813148100 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f24ec003980 0x7f24ec003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:59.832 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.822+0000 7f2508eff640 1 --2- 172.21.15.16:0/813148100 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f24ec0043b0 0x7f24ec004870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:59.832 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.822+0000 7f2508eff640 1 --2- 172.21.15.16:0/813148100 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f25040c1380 0x7f24ec004f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:59.832 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.822+0000 7f2508eff640 1 -- 172.21.15.16:0/813148100 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2504164370 con 0x7f24ec003980 2023-05-22T15:39:59.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.822+0000 7f2508eff640 1 -- 172.21.15.16:0/813148100 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2504164650 con 0x7f24ec0043b0 2023-05-22T15:39:59.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.822+0000 7f2508eff640 1 -- 172.21.15.16:0/813148100 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24ec00b990 con 0x7f25040c1380 2023-05-22T15:39:59.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.822+0000 7f2509f01640 1 --2- 172.21.15.16:0/813148100 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f24ec003980 0x7f24ec003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:59.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.822+0000 7f250a702640 1 --2- 172.21.15.16:0/813148100 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f25040c1380 0x7f24ec004f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:59.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.822+0000 7f2509700640 1 --2- 172.21.15.16:0/813148100 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f24ec0043b0 0x7f24ec004870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:59.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.822+0000 7f2509f01640 1 -- 172.21.15.16:0/813148100 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f25040c1380 msgr2=0x7f24ec004f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:39:59.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.822+0000 7f2509f01640 1 --2- 172.21.15.16:0/813148100 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f25040c1380 0x7f24ec004f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:59.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.822+0000 7f2509f01640 1 -- 172.21.15.16:0/813148100 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f24ec0043b0 msgr2=0x7f24ec004870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:39:59.834 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.822+0000 7f2509f01640 1 --2- 172.21.15.16:0/813148100 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f24ec0043b0 0x7f24ec004870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:59.834 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.822+0000 7f2509f01640 1 -- 172.21.15.16:0/813148100 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24ec003530 con 0x7f24ec003980 2023-05-22T15:39:59.836 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.826+0000 7f2509f01640 1 --2- 172.21.15.16:0/813148100 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f24ec003980 0x7f24ec003d90 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7f2500009fc0 tx=0x7f250001cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4da3805b1bf3018a server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:39:59.836 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.826+0000 7f2508eff640 1 -- 172.21.15.16:0/813148100 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25000a5a90 con 0x7f24ec003980 2023-05-22T15:39:59.836 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.826+0000 7f2508eff640 1 -- 172.21.15.16:0/813148100 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f250000f040 con 0x7f24ec003980 2023-05-22T15:39:59.836 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.826+0000 7f2508eff640 1 -- 172.21.15.16:0/813148100 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25000a5e30 con 0x7f24ec003980 2023-05-22T15:39:59.837 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.826+0000 7f250c165640 1 -- 172.21.15.16:0/813148100 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f24ec003980 msgr2=0x7f24ec003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:59.837 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.826+0000 7f250c165640 1 --2- 172.21.15.16:0/813148100 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f24ec003980 0x7f24ec003d90 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7f2500009fc0 tx=0x7f250001cca0 comp rx=0 tx=0).stop 2023-05-22T15:39:59.837 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.826+0000 7f250a702640 1 -- 172.21.15.16:0/813148100 reap_dead start 2023-05-22T15:39:59.837 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.826+0000 7f250c165640 1 -- 172.21.15.16:0/813148100 shutdown_connections 2023-05-22T15:39:59.837 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.826+0000 7f250c165640 1 -- 172.21.15.16:0/813148100 >> 172.21.15.16:0/813148100 conn(0x7f2504069720 msgr2=0x7f2504069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:39:59.837 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.826+0000 7f250c165640 1 -- 172.21.15.16:0/813148100 shutdown_connections 2023-05-22T15:39:59.837 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.826+0000 7f250c165640 1 -- 172.21.15.16:0/813148100 wait complete. 2023-05-22T15:39:59.838 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.826+0000 7f250c165640 1 Processor -- start 2023-05-22T15:39:59.838 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.826+0000 7f250c165640 1 -- start start 2023-05-22T15:39:59.838 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.830+0000 7f250c165640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f24ec003980 0x7f25040b0e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:59.838 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.830+0000 7f250c165640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f25040c7290 0x7f25040c7700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:59.838 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.830+0000 7f250c165640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f25040c7c40 0x7f25040b2a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:59.838 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.830+0000 7f250c165640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25040c80b0 con 0x7f25040c7290 2023-05-22T15:39:59.838 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.830+0000 7f250c165640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f250415bb90 con 0x7f24ec003980 2023-05-22T15:39:59.838 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.830+0000 7f250c165640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f250415be70 con 0x7f25040c7c40 2023-05-22T15:39:59.839 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.830+0000 7f2509700640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f25040c7290 0x7f25040c7700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:59.839 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.830+0000 7f2509700640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f25040c7290 0x7f25040c7700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34198/0 (socket says 172.21.15.16:34198) 2023-05-22T15:39:59.839 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.830+0000 7f2509700640 1 -- 172.21.15.16:0/1725592972 learned_addr learned my addr 172.21.15.16:0/1725592972 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:39:59.839 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.830+0000 7f2509f01640 1 --2- 172.21.15.16:0/1725592972 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f24ec003980 0x7f25040b0e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:59.839 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.830+0000 7f250a702640 1 --2- 172.21.15.16:0/1725592972 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f25040c7c40 0x7f25040b2a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:59.839 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.830+0000 7f2509700640 1 -- 172.21.15.16:0/1725592972 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f25040c7c40 msgr2=0x7f25040b2a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:59.839 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.830+0000 7f2509700640 1 --2- 172.21.15.16:0/1725592972 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f25040c7c40 0x7f25040b2a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:59.839 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.830+0000 7f2509700640 1 -- 172.21.15.16:0/1725592972 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f24ec003980 msgr2=0x7f25040b0e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:59.840 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.830+0000 7f2509700640 1 --2- 172.21.15.16:0/1725592972 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f24ec003980 0x7f25040b0e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:39:59.840 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.830+0000 7f2509700640 1 -- 172.21.15.16:0/1725592972 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2504160540 con 0x7f25040c7290 2023-05-22T15:39:59.840 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.830+0000 7f2509700640 1 --2- 172.21.15.16:0/1725592972 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f25040c7290 0x7f25040c7700 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7f24f4010890 tx=0x7f24f4010d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:39:59.840 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.830+0000 7f24fa7fc640 1 -- 172.21.15.16:0/1725592972 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24f4009630 con 0x7f25040c7290 2023-05-22T15:39:59.840 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.830+0000 7f24fa7fc640 1 -- 172.21.15.16:0/1725592972 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24f400eb60 con 0x7f25040c7290 2023-05-22T15:39:59.840 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.830+0000 7f250c165640 1 -- 172.21.15.16:0/1725592972 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25040ca760 con 0x7f25040c7290 2023-05-22T15:39:59.840 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.830+0000 7f24fa7fc640 1 -- 172.21.15.16:0/1725592972 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24f401c400 con 0x7f25040c7290 2023-05-22T15:39:59.841 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.830+0000 7f250c165640 1 -- 172.21.15.16:0/1725592972 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25041606d0 con 0x7f25040c7290 2023-05-22T15:39:59.841 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.830+0000 7f250c165640 1 -- 172.21.15.16:0/1725592972 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f250404dbe0 con 0x7f25040c7290 2023-05-22T15:39:59.842 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.830+0000 7f24fa7fc640 1 -- 172.21.15.16:0/1725592972 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f24f401c560 con 0x7f25040c7290 2023-05-22T15:39:59.842 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.830+0000 7f24fa7fc640 1 --2- 172.21.15.16:0/1725592972 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f24f00654d0 0x7f24f0067980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:39:59.842 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.830+0000 7f24fa7fc640 1 -- 172.21.15.16:0/1725592972 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5530+0+0 (secure 0 0 0) 0x7f24f4082d30 con 0x7f25040c7290 2023-05-22T15:39:59.842 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.830+0000 7f2509f01640 1 --2- 172.21.15.16:0/1725592972 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f24f00654d0 0x7f24f0067980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:39:59.842 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.834+0000 7f2509f01640 1 --2- 172.21.15.16:0/1725592972 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f24f00654d0 0x7f24f0067980 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f25000af000 tx=0x7f25000a7040 comp rx=0 tx=0).ready entity=mgr.6240 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:39:59.842 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.834+0000 7f24fa7fc640 1 -- 172.21.15.16:0/1725592972 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f24f4054590 con 0x7f25040c7290 2023-05-22T15:39:59.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.990+0000 7f250c165640 1 -- 172.21.15.16:0/1725592972 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f25040cb000 con 0x7f25040c7290 2023-05-22T15:39:59.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.990+0000 7f24fa7fc640 1 -- 172.21.15.16:0/1725592972 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v69) v1 ==== 92+0+10 (secure 0 0 0) 0x7f24f4054120 con 0x7f25040c7290 2023-05-22T15:39:59.997 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 8 2023-05-22T15:39:59.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.994+0000 7f250c165640 1 -- 172.21.15.16:0/1725592972 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f24f00654d0 msgr2=0x7f24f0067980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:59.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.994+0000 7f250c165640 1 --2- 172.21.15.16:0/1725592972 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f24f00654d0 0x7f24f0067980 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f25000af000 tx=0x7f25000a7040 comp rx=0 tx=0).stop 2023-05-22T15:39:59.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.994+0000 7f250c165640 1 -- 172.21.15.16:0/1725592972 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f25040c7290 msgr2=0x7f25040c7700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:39:59.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.994+0000 7f250c165640 1 --2- 172.21.15.16:0/1725592972 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f25040c7290 0x7f25040c7700 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7f24f4010890 tx=0x7f24f4010d50 comp rx=0 tx=0).stop 2023-05-22T15:39:59.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.994+0000 7f250c165640 1 -- 172.21.15.16:0/1725592972 shutdown_connections 2023-05-22T15:40:00.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.994+0000 7f250c165640 1 --2- 172.21.15.16:0/1725592972 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f24f00654d0 0x7f24f0067980 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:00.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.994+0000 7f250c165640 1 --2- 172.21.15.16:0/1725592972 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f25040c7c40 0x7f25040b2a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:00.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.994+0000 7f250c165640 1 --2- 172.21.15.16:0/1725592972 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f25040c7290 0x7f25040c7700 unknown :-1 s=CLOSED pgs=894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:00.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.994+0000 7f250c165640 1 --2- 172.21.15.16:0/1725592972 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f24ec003980 0x7f25040b0e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:00.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.994+0000 7f250c165640 1 -- 172.21.15.16:0/1725592972 >> 172.21.15.16:0/1725592972 conn(0x7f2504069720 msgr2=0x7f2504166420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:40:00.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.994+0000 7f250c165640 1 -- 172.21.15.16:0/1725592972 shutdown_connections 2023-05-22T15:40:00.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:39:59.994+0000 7f250c165640 1 -- 172.21.15.16:0/1725592972 wait complete. 2023-05-22T15:40:00.016 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-05-22T15:40:00.130 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:00.126+0000 7fcbc686f640 1 Processor -- start 2023-05-22T15:40:00.131 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:00.126+0000 7fcbc686f640 1 -- start start 2023-05-22T15:40:00.132 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:00.126+0000 7fcbc686f640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fcbc006bfa0 0x7fcbc015cf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:40:00.132 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:00.126+0000 7fcbc686f640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fcbc00ce160 0x7fcbc015fb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:40:00.132 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:00.126+0000 7fcbc686f640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbc00cf7f0 con 0x7fcbc006dd90 2023-05-22T15:40:00.132 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:00.126+0000 7fcbc686f640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcbc006c440 con 0x7fcbc006bfa0 2023-05-22T15:40:00.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:00.126+0000 7fcbc686f640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcbc006c720 con 0x7fcbc00ce160 2023-05-22T15:40:00.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:00.126+0000 7fcbb77fe640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fcbc006bfa0 0x7fcbc015cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:40:00.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:00.126+0000 7fcbbffff640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fcbc006dd90 0x7fcbc006e160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:41310/0 (socket says 172.21.15.16:41310) 2023-05-22T15:40:00.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:00.126+0000 7fcbbffff640 1 -- 172.21.15.16:0/417127589 learned_addr learned my addr 172.21.15.16:0/417127589 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:40:00.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:00.126+0000 7fcbc4e0c640 1 --2- 172.21.15.16:0/417127589 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fcbc00ce160 0x7fcbc015fb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:40:00.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:00.126+0000 7fcbb77fe640 1 -- 172.21.15.16:0/417127589 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fcbc00ce160 msgr2=0x7fcbc015fb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:40:00.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:00.126+0000 7fcbb77fe640 1 --2- 172.21.15.16:0/417127589 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fcbc00ce160 0x7fcbc015fb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:00.134 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:00.126+0000 7fcbb77fe640 1 -- 172.21.15.16:0/417127589 >> v1:172.21.15.16:6789/0 conn(0x7fcbc006dd90 legacy=0x7fcbc006e160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:40:00.134 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:00.126+0000 7fcbbf7fe640 1 -- 172.21.15.16:0/417127589 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1785552676 0 0) 0x7fcbc00cf7f0 con 0x7fcbc006dd90 2023-05-22T15:40:00.134 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:00.126+0000 7fcbb77fe640 1 -- 172.21.15.16:0/417127589 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbc00cfa90 con 0x7fcbc006bfa0 2023-05-22T15:40:00.135 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:00.126+0000 7fcbb77fe640 1 --2- 172.21.15.16:0/417127589 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcbc006bfa0 0x7fcbc015cf00 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7fcba80097c0 tx=0x7fcba801c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e932d9fc6e42f9d4 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:40:00.135 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:00.126+0000 7fcbbf7fe640 1 -- 172.21.15.16:0/417127589 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcba80a6040 con 0x7fcbc006bfa0 2023-05-22T15:40:00.135 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:00.130+0000 7fcbbf7fe640 1 -- 172.21.15.16:0/417127589 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcba80aa600 con 0x7fcbc006bfa0 2023-05-22T15:40:00.135 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:00.130+0000 7fcbbf7fe640 1 -- 172.21.15.16:0/417127589 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcba80aa990 con 0x7fcbc006bfa0 2023-05-22T15:40:00.136 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:00.130+0000 7fcbc686f640 1 -- 172.21.15.16:0/417127589 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcbc006bfa0 msgr2=0x7fcbc015cf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:40:00.136 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:00.130+0000 7fcbc686f640 1 --2- 172.21.15.16:0/417127589 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcbc006bfa0 0x7fcbc015cf00 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7fcba80097c0 tx=0x7fcba801c480 comp rx=0 tx=0).stop 2023-05-22T15:40:00.136 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:00.130+0000 7fcbc686f640 1 -- 172.21.15.16:0/417127589 shutdown_connections 2023-05-22T15:40:00.136 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:00.130+0000 7fcbc686f640 1 --2- 172.21.15.16:0/417127589 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fcbc00ce160 0x7fcbc015fb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:00.136 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:00.130+0000 7fcbc686f640 1 --2- 172.21.15.16:0/417127589 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcbc006bfa0 0x7fcbc015cf00 unknown :-1 s=CLOSED pgs=896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:00.136 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:00.130+0000 7fcbc686f640 1 -- 172.21.15.16:0/417127589 >> 172.21.15.16:0/417127589 conn(0x7fcbc006f5c0 msgr2=0x7fcbc006f9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:40:00.137 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:00.130+0000 7fcbc686f640 1 -- 172.21.15.16:0/417127589 shutdown_connections 2023-05-22T15:40:00.137 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:00.130+0000 7fcbc686f640 1 -- 172.21.15.16:0/417127589 wait complete. 2023-05-22T15:40:00.138 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:00.130+0000 7fcbc686f640 1 Processor -- start 2023-05-22T15:40:00.139 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:00.130+0000 7fcbc686f640 1 -- start start 2023-05-22T15:40:00.139 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:00.130+0000 7fcbc686f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcbc006bfa0 0x7fcbc00b5be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:40:00.139 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:00.130+0000 7fcbc686f640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fcbc006dd90 0x7fcbc00c3d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:40:00.139 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:00.130+0000 7fcbc686f640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcbc00ce160 0x7fcbc015a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:40:00.139 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:00.130+0000 7fcbc686f640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbc00c5010 con 0x7fcbc006bfa0 2023-05-22T15:40:00.139 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:00.130+0000 7fcbc686f640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbc00c4770 con 0x7fcbc00ce160 2023-05-22T15:40:00.140 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:00.130+0000 7fcbc686f640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcbc00c4a50 con 0x7fcbc006dd90 2023-05-22T15:40:00.140 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:00.130+0000 7fcbbffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcbc006bfa0 0x7fcbc00b5be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:40:00.140 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:00.130+0000 7fcbbffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcbc006bfa0 0x7fcbc00b5be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34206/0 (socket says 172.21.15.16:34206) 2023-05-22T15:40:00.140 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:00.130+0000 7fcbbffff640 1 -- 172.21.15.16:0/3183289097 learned_addr learned my addr 172.21.15.16:0/3183289097 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:40:00.140 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:00.130+0000 7fcbb77fe640 1 --2- 172.21.15.16:0/3183289097 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fcbc006dd90 0x7fcbc00c3d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:40:00.140 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:00.130+0000 7fcbc4e0c640 1 --2- 172.21.15.16:0/3183289097 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcbc00ce160 0x7fcbc015a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:40:00.140 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:00.130+0000 7fcbbffff640 1 -- 172.21.15.16:0/3183289097 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fcbc006dd90 msgr2=0x7fcbc00c3d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:40:00.141 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:00.130+0000 7fcbbffff640 1 --2- 172.21.15.16:0/3183289097 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fcbc006dd90 0x7fcbc00c3d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:00.141 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:00.130+0000 7fcbbffff640 1 -- 172.21.15.16:0/3183289097 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcbc00ce160 msgr2=0x7fcbc015a0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:40:00.141 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:00.130+0000 7fcbbffff640 1 --2- 172.21.15.16:0/3183289097 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcbc00ce160 0x7fcbc015a0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:00.141 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:00.130+0000 7fcbbffff640 1 -- 172.21.15.16:0/3183289097 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcba8009420 con 0x7fcbc006bfa0 2023-05-22T15:40:00.141 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:00.130+0000 7fcbbffff640 1 --2- 172.21.15.16:0/3183289097 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcbc006bfa0 0x7fcbc00b5be0 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7fcbb000c480 tx=0x7fcbb000c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:40:00.141 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:00.130+0000 7fcbbd7fa640 1 -- 172.21.15.16:0/3183289097 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbb00074e0 con 0x7fcbc006bfa0 2023-05-22T15:40:00.141 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:00.130+0000 7fcbbd7fa640 1 -- 172.21.15.16:0/3183289097 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbb0007660 con 0x7fcbc006bfa0 2023-05-22T15:40:00.142 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:00.130+0000 7fcbbd7fa640 1 -- 172.21.15.16:0/3183289097 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbb0004a30 con 0x7fcbc006bfa0 2023-05-22T15:40:00.142 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:00.130+0000 7fcbc686f640 1 -- 172.21.15.16:0/3183289097 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcbc015c2c0 con 0x7fcbc006bfa0 2023-05-22T15:40:00.142 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:00.130+0000 7fcbc686f640 1 -- 172.21.15.16:0/3183289097 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcbc015c130 con 0x7fcbc006bfa0 2023-05-22T15:40:00.142 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:00.134+0000 7fcbc686f640 1 -- 172.21.15.16:0/3183289097 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcbc0168be0 con 0x7fcbc006bfa0 2023-05-22T15:40:00.143 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:00.134+0000 7fcbbd7fa640 1 -- 172.21.15.16:0/3183289097 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) v1 ==== 84332+0+0 (secure 0 0 0) 0x7fcbb0004b90 con 0x7fcbc006bfa0 2023-05-22T15:40:00.143 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:00.134+0000 7fcbbd7fa640 1 --2- 172.21.15.16:0/3183289097 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fcba00654d0 0x7fcba0067980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:40:00.143 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:00.134+0000 7fcbbd7fa640 1 -- 172.21.15.16:0/3183289097 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5530+0+0 (secure 0 0 0) 0x7fcbb0081920 con 0x7fcbc006bfa0 2023-05-22T15:40:00.143 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:00.134+0000 7fcbb77fe640 1 --2- 172.21.15.16:0/3183289097 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fcba00654d0 0x7fcba0067980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:40:00.143 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:00.134+0000 7fcbb77fe640 1 --2- 172.21.15.16:0/3183289097 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fcba00654d0 0x7fcba0067980 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fcba80ae000 tx=0x7fcba80b5040 comp rx=0 tx=0).ready entity=mgr.6240 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:40:00.144 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:00.138+0000 7fcbbd7fa640 1 -- 172.21.15.16:0/3183289097 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fcbb0053160 con 0x7fcbc006bfa0 2023-05-22T15:40:00.301 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:00.294+0000 7fcbc686f640 1 -- 172.21.15.16:0/3183289097 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fcbc00b1cc0 con 0x7fcbc006bfa0 2023-05-22T15:40:00.301 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:00.294+0000 7fcbbd7fa640 1 -- 172.21.15.16:0/3183289097 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v69) v1 ==== 84+0+126 (secure 0 0 0) 0x7fcbb0018020 con 0x7fcbc006bfa0 2023-05-22T15:40:00.302 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:40:00.302 INFO:teuthology.orchestra.run.smithi016.stdout:[ 2023-05-22T15:40:00.302 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T15:40:00.302 INFO:teuthology.orchestra.run.smithi016.stdout: "poolnum": 1, 2023-05-22T15:40:00.302 INFO:teuthology.orchestra.run.smithi016.stdout: "poolname": ".mgr" 2023-05-22T15:40:00.303 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:40:00.303 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T15:40:00.303 INFO:teuthology.orchestra.run.smithi016.stdout: "poolnum": 2, 2023-05-22T15:40:00.303 INFO:teuthology.orchestra.run.smithi016.stdout: "poolname": "rbd" 2023-05-22T15:40:00.303 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T15:40:00.303 INFO:teuthology.orchestra.run.smithi016.stdout:] 2023-05-22T15:40:00.304 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:00.298+0000 7fcbc686f640 1 -- 172.21.15.16:0/3183289097 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fcba00654d0 msgr2=0x7fcba0067980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:40:00.304 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:00.298+0000 7fcbc686f640 1 --2- 172.21.15.16:0/3183289097 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fcba00654d0 0x7fcba0067980 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fcba80ae000 tx=0x7fcba80b5040 comp rx=0 tx=0).stop 2023-05-22T15:40:00.304 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:00.298+0000 7fcbc686f640 1 -- 172.21.15.16:0/3183289097 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcbc006bfa0 msgr2=0x7fcbc00b5be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:40:00.305 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:00.298+0000 7fcbc686f640 1 --2- 172.21.15.16:0/3183289097 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcbc006bfa0 0x7fcbc00b5be0 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7fcbb000c480 tx=0x7fcbb000c940 comp rx=0 tx=0).stop 2023-05-22T15:40:00.305 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:00.298+0000 7fcbc686f640 1 -- 172.21.15.16:0/3183289097 shutdown_connections 2023-05-22T15:40:00.305 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:00.298+0000 7fcbc686f640 1 --2- 172.21.15.16:0/3183289097 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcbc00ce160 0x7fcbc015a0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:00.305 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:00.298+0000 7fcbc686f640 1 --2- 172.21.15.16:0/3183289097 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fcba00654d0 0x7fcba0067980 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:00.305 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:00.298+0000 7fcbc686f640 1 --2- 172.21.15.16:0/3183289097 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fcbc006dd90 0x7fcbc00c3d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:00.305 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:00.298+0000 7fcbc686f640 1 --2- 172.21.15.16:0/3183289097 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcbc006bfa0 0x7fcbc00b5be0 unknown :-1 s=CLOSED pgs=897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:00.305 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:00.298+0000 7fcbc686f640 1 -- 172.21.15.16:0/3183289097 >> 172.21.15.16:0/3183289097 conn(0x7fcbc006f5c0 msgr2=0x7fcbc006f990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:40:00.305 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:00.298+0000 7fcbc686f640 1 -- 172.21.15.16:0/3183289097 shutdown_connections 2023-05-22T15:40:00.306 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:00.298+0000 7fcbc686f640 1 -- 172.21.15.16:0/3183289097 wait complete. 2023-05-22T15:40:00.318 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-05-22T15:40:00.319 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-05-22T15:40:00.448 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:00.442+0000 7fa40ebb0640 1 Processor -- start 2023-05-22T15:40:00.449 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:00.442+0000 7fa40ebb0640 1 -- start start 2023-05-22T15:40:00.450 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:00.442+0000 7fa40ebb0640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa408070c10 0x7fa4080cc4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:40:00.450 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:00.442+0000 7fa40ebb0640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4080ccbb0 con 0x7fa4080c9e40 2023-05-22T15:40:00.450 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:00.442+0000 7fa40ebb0640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4080cd7c0 con 0x7fa40815d650 2023-05-22T15:40:00.450 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:00.442+0000 7fa40ebb0640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4080cdaa0 con 0x7fa408070c10 2023-05-22T15:40:00.450 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:00.446+0000 7fa407fff640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fa4080c9e40 0x7fa40806bfa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:41316/0 (socket says 172.21.15.16:41316) 2023-05-22T15:40:00.450 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:00.446+0000 7fa407fff640 1 -- 172.21.15.16:0/838438936 learned_addr learned my addr 172.21.15.16:0/838438936 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:40:00.452 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:00.446+0000 7fa40c94c640 1 --2- 172.21.15.16:0/838438936 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa408070c10 0x7fa4080cc4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:40:00.452 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:00.446+0000 7fa4077fe640 1 -- 172.21.15.16:0/838438936 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 473750286 0 0) 0x7fa4080ccbb0 con 0x7fa4080c9e40 2023-05-22T15:40:00.452 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:00.446+0000 7fa4077fe640 1 -- 172.21.15.16:0/838438936 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa3f8003530 con 0x7fa4080c9e40 2023-05-22T15:40:00.453 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:00.446+0000 7fa4077fe640 1 -- 172.21.15.16:0/838438936 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2136872666 0 0) 0x7fa3f8003530 con 0x7fa4080c9e40 2023-05-22T15:40:00.453 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:00.446+0000 7fa4077fe640 1 -- 172.21.15.16:0/838438936 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa408070c10 msgr2=0x7fa4080cc4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:40:00.453 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:00.446+0000 7fa4077fe640 1 --2- 172.21.15.16:0/838438936 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa408070c10 0x7fa4080cc4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:00.453 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:00.446+0000 7fa4077fe640 1 -- 172.21.15.16:0/838438936 >> v1:172.21.15.62:6789/0 conn(0x7fa40815d650 legacy=0x7fa4080ce2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:40:00.453 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:00.446+0000 7fa4077fe640 1 -- 172.21.15.16:0/838438936 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4080cdd80 con 0x7fa4080c9e40 2023-05-22T15:40:00.454 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:00.446+0000 7fa4077fe640 1 -- 172.21.15.16:0/838438936 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7fa3f4001680 con 0x7fa4080c9e40 2023-05-22T15:40:00.454 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:00.446+0000 7fa4077fe640 1 -- 172.21.15.16:0/838438936 >> v1:172.21.15.16:6789/0 conn(0x7fa4080c9e40 legacy=0x7fa40806bfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:40:00.454 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:00.446+0000 7fa4077fe640 1 --2- 172.21.15.16:0/838438936 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa408070c10 0x7fa3f80039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:40:00.454 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:00.446+0000 7fa4077fe640 1 --2- 172.21.15.16:0/838438936 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa3f8003ef0 0x7fa3f80047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:40:00.454 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:00.446+0000 7fa4077fe640 1 --2- 172.21.15.16:0/838438936 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa3f8004d10 0x7fa3f8005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:40:00.454 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:00.446+0000 7fa4077fe640 1 -- 172.21.15.16:0/838438936 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4080cdaa0 con 0x7fa3f8004d10 2023-05-22T15:40:00.454 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:00.446+0000 7fa4077fe640 1 -- 172.21.15.16:0/838438936 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3f800b9c0 con 0x7fa3f8003ef0 2023-05-22T15:40:00.455 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:00.446+0000 7fa4077fe640 1 -- 172.21.15.16:0/838438936 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3f800bb80 con 0x7fa408070c10 2023-05-22T15:40:00.455 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:00.446+0000 7fa40d14d640 1 --2- 172.21.15.16:0/838438936 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa3f8004d10 0x7fa3f8005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:40:00.455 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:00.446+0000 7fa40c94c640 1 --2- 172.21.15.16:0/838438936 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa408070c10 0x7fa3f80039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:40:00.455 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:00.446+0000 7fa407fff640 1 --2- 172.21.15.16:0/838438936 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa3f8003ef0 0x7fa3f80047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:40:00.455 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:00.446+0000 7fa40d14d640 1 -- 172.21.15.16:0/838438936 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa408070c10 msgr2=0x7fa3f80039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:40:00.455 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:00.446+0000 7fa40d14d640 1 --2- 172.21.15.16:0/838438936 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa408070c10 0x7fa3f80039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:00.455 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:00.446+0000 7fa40d14d640 1 -- 172.21.15.16:0/838438936 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa3f8003ef0 msgr2=0x7fa3f80047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:40:00.456 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:00.446+0000 7fa40d14d640 1 --2- 172.21.15.16:0/838438936 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa3f8003ef0 0x7fa3f80047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:00.456 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:00.446+0000 7fa40d14d640 1 -- 172.21.15.16:0/838438936 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3f8003530 con 0x7fa3f8004d10 2023-05-22T15:40:00.456 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:00.446+0000 7fa407fff640 1 --2- 172.21.15.16:0/838438936 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa3f8003ef0 0x7fa3f80047d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:40:00.458 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:00.446+0000 7fa40d14d640 1 --2- 172.21.15.16:0/838438936 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa3f8004d10 0x7fa3f8005180 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7fa400001f60 tx=0x7fa40001c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aad402d75ebfd536 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:40:00.458 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:00.446+0000 7fa4077fe640 1 -- 172.21.15.16:0/838438936 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4000a5750 con 0x7fa3f8004d10 2023-05-22T15:40:00.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:00.446+0000 7fa4077fe640 1 -- 172.21.15.16:0/838438936 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3f4001d80 con 0x7fa3f8004d10 2023-05-22T15:40:00.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:00.446+0000 7fa4077fe640 1 -- 172.21.15.16:0/838438936 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4000a5b30 con 0x7fa3f8004d10 2023-05-22T15:40:00.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:00.450+0000 7fa40ebb0640 1 -- 172.21.15.16:0/838438936 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa3f8004d10 msgr2=0x7fa3f8005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:40:00.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:00.450+0000 7fa40ebb0640 1 --2- 172.21.15.16:0/838438936 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa3f8004d10 0x7fa3f8005180 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7fa400001f60 tx=0x7fa40001c860 comp rx=0 tx=0).stop 2023-05-22T15:40:00.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:00.450+0000 7fa40d14d640 1 -- 172.21.15.16:0/838438936 reap_dead start 2023-05-22T15:40:00.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:00.450+0000 7fa40ebb0640 1 -- 172.21.15.16:0/838438936 shutdown_connections 2023-05-22T15:40:00.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:00.450+0000 7fa40ebb0640 1 -- 172.21.15.16:0/838438936 >> 172.21.15.16:0/838438936 conn(0x7fa408069720 msgr2=0x7fa408069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:40:00.460 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:00.450+0000 7fa40ebb0640 1 -- 172.21.15.16:0/838438936 shutdown_connections 2023-05-22T15:40:00.460 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:00.450+0000 7fa40ebb0640 1 -- 172.21.15.16:0/838438936 wait complete. 2023-05-22T15:40:00.460 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:00.450+0000 7fa40ebb0640 1 Processor -- start 2023-05-22T15:40:00.460 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:00.450+0000 7fa40ebb0640 1 -- start start 2023-05-22T15:40:00.460 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:00.450+0000 7fa40ebb0640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa3f8004d10 0x7fa4080c3aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:40:00.460 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:00.450+0000 7fa40ebb0640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa4080b3cd0 0x7fa4080c6ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:40:00.460 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:00.450+0000 7fa40ebb0640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa4080ca800 0x7fa408150300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:40:00.461 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:00.450+0000 7fa40ebb0640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4080c3fe0 con 0x7fa4080ca800 2023-05-22T15:40:00.461 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:00.450+0000 7fa40ebb0640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4080c3610 con 0x7fa3f8004d10 2023-05-22T15:40:00.461 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:00.450+0000 7fa40ebb0640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4080c7730 con 0x7fa4080b3cd0 2023-05-22T15:40:00.461 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:00.450+0000 7fa40d14d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa4080ca800 0x7fa408150300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:40:00.461 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:00.450+0000 7fa40d14d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa4080ca800 0x7fa408150300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34228/0 (socket says 172.21.15.16:34228) 2023-05-22T15:40:00.461 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:00.450+0000 7fa40d14d640 1 -- 172.21.15.16:0/92472822 learned_addr learned my addr 172.21.15.16:0/92472822 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:40:00.461 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:00.450+0000 7fa40c94c640 1 --2- 172.21.15.16:0/92472822 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa3f8004d10 0x7fa4080c3aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:40:00.462 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:00.450+0000 7fa407fff640 1 --2- 172.21.15.16:0/92472822 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa4080b3cd0 0x7fa4080c6ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:40:00.462 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:00.450+0000 7fa40d14d640 1 -- 172.21.15.16:0/92472822 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa4080b3cd0 msgr2=0x7fa4080c6ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:40:00.462 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:00.450+0000 7fa40d14d640 1 --2- 172.21.15.16:0/92472822 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa4080b3cd0 0x7fa4080c6ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:00.462 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:00.450+0000 7fa40d14d640 1 -- 172.21.15.16:0/92472822 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa3f8004d10 msgr2=0x7fa4080c3aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:40:00.462 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:00.450+0000 7fa40d14d640 1 --2- 172.21.15.16:0/92472822 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa3f8004d10 0x7fa4080c3aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:00.462 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:00.450+0000 7fa40d14d640 1 -- 172.21.15.16:0/92472822 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4080cce90 con 0x7fa4080ca800 2023-05-22T15:40:00.462 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:00.450+0000 7fa407fff640 1 --2- 172.21.15.16:0/92472822 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa4080b3cd0 0x7fa4080c6ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:40:00.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:00.450+0000 7fa40d14d640 1 --2- 172.21.15.16:0/92472822 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa4080ca800 0x7fa408150300 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7fa400001f60 tx=0x7fa3f8005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:40:00.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:00.450+0000 7fa4057fa640 1 -- 172.21.15.16:0/92472822 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa400003d90 con 0x7fa4080ca800 2023-05-22T15:40:00.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:00.450+0000 7fa4057fa640 1 -- 172.21.15.16:0/92472822 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa400005070 con 0x7fa4080ca800 2023-05-22T15:40:00.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:00.450+0000 7fa40ebb0640 1 -- 172.21.15.16:0/92472822 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa408070fe0 con 0x7fa4080ca800 2023-05-22T15:40:00.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:00.450+0000 7fa4057fa640 1 -- 172.21.15.16:0/92472822 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4000053e0 con 0x7fa4080ca800 2023-05-22T15:40:00.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:00.450+0000 7fa40ebb0640 1 -- 172.21.15.16:0/92472822 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4080ca290 con 0x7fa4080ca800 2023-05-22T15:40:00.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:00.454+0000 7fa40ebb0640 1 -- 172.21.15.16:0/92472822 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4080c3780 con 0x7fa4080ca800 2023-05-22T15:40:00.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:00.454+0000 7fa4057fa640 1 -- 172.21.15.16:0/92472822 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) v1 ==== 84332+0+0 (secure 0 0 0) 0x7fa4000b4040 con 0x7fa4080ca800 2023-05-22T15:40:00.464 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:00.454+0000 7fa4057fa640 1 --2- 172.21.15.16:0/92472822 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fa3ec0654d0 0x7fa3ec067980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:40:00.464 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:00.454+0000 7fa4057fa640 1 -- 172.21.15.16:0/92472822 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5530+0+0 (secure 0 0 0) 0x7fa40011bb40 con 0x7fa4080ca800 2023-05-22T15:40:00.465 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:00.454+0000 7fa40c94c640 1 --2- 172.21.15.16:0/92472822 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fa3ec0654d0 0x7fa3ec067980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:40:00.465 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:00.454+0000 7fa40c94c640 1 --2- 172.21.15.16:0/92472822 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fa3ec0654d0 0x7fa3ec067980 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fa3fc009a30 tx=0x7fa3fc008040 comp rx=0 tx=0).ready entity=mgr.6240 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:40:00.465 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:00.458+0000 7fa4057fa640 1 -- 172.21.15.16:0/92472822 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fa4000e9a00 con 0x7fa4080ca800 2023-05-22T15:40:00.619 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:00.614+0000 7fa40ebb0640 1 -- 172.21.15.16:0/92472822 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fa40804dbe0 con 0x7fa4080ca800 2023-05-22T15:40:01.372 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:01.366+0000 7fa4057fa640 1 -- 172.21.15.16:0/92472822 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v70) v1 ==== 138+0+0 (secure 0 0 0) 0x7fa4000ed020 con 0x7fa4080ca800 2023-05-22T15:40:01.373 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'cephfs_metadata' created 2023-05-22T15:40:01.376 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:01.370+0000 7fa40ebb0640 1 -- 172.21.15.16:0/92472822 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fa3ec0654d0 msgr2=0x7fa3ec067980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:40:01.376 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:01.370+0000 7fa40ebb0640 1 --2- 172.21.15.16:0/92472822 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fa3ec0654d0 0x7fa3ec067980 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fa3fc009a30 tx=0x7fa3fc008040 comp rx=0 tx=0).stop 2023-05-22T15:40:01.376 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:01.370+0000 7fa40ebb0640 1 -- 172.21.15.16:0/92472822 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa4080ca800 msgr2=0x7fa408150300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:40:01.376 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:01.370+0000 7fa40ebb0640 1 --2- 172.21.15.16:0/92472822 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa4080ca800 0x7fa408150300 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7fa400001f60 tx=0x7fa3f8005800 comp rx=0 tx=0).stop 2023-05-22T15:40:01.377 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:01.370+0000 7fa40ebb0640 1 -- 172.21.15.16:0/92472822 shutdown_connections 2023-05-22T15:40:01.377 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:01.370+0000 7fa40ebb0640 1 --2- 172.21.15.16:0/92472822 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa4080ca800 0x7fa408150300 unknown :-1 s=CLOSED pgs=900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:01.377 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:01.370+0000 7fa40ebb0640 1 --2- 172.21.15.16:0/92472822 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fa3ec0654d0 0x7fa3ec067980 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:01.377 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:01.370+0000 7fa40ebb0640 1 --2- 172.21.15.16:0/92472822 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa4080b3cd0 0x7fa4080c6ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:01.377 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:01.370+0000 7fa40ebb0640 1 --2- 172.21.15.16:0/92472822 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa3f8004d10 0x7fa4080c3aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:01.377 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:01.370+0000 7fa40ebb0640 1 -- 172.21.15.16:0/92472822 >> 172.21.15.16:0/92472822 conn(0x7fa408069720 msgr2=0x7fa408151c40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:40:01.377 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:01.370+0000 7fa40ebb0640 1 -- 172.21.15.16:0/92472822 shutdown_connections 2023-05-22T15:40:01.378 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:01.370+0000 7fa40ebb0640 1 -- 172.21.15.16:0/92472822 wait complete. 2023-05-22T15:40:01.401 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-05-22T15:40:01.537 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:01.530+0000 7faf9c2c0640 1 Processor -- start 2023-05-22T15:40:01.537 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:01.530+0000 7faf9c2c0640 1 -- start start 2023-05-22T15:40:01.540 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:01.530+0000 7faf9c2c0640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7faf9415bca0 0x7faf940cda80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:40:01.540 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:01.530+0000 7faf9c2c0640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7faf940b5a50 0x7faf940d2900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:40:01.540 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:01.530+0000 7faf9c2c0640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf940cccc0 con 0x7faf940d2480 2023-05-22T15:40:01.540 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:01.530+0000 7faf9c2c0640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf940cd660 con 0x7faf9415bca0 2023-05-22T15:40:01.541 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:01.530+0000 7faf9c2c0640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf940d2e40 con 0x7faf940b5a50 2023-05-22T15:40:01.541 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:01.534+0000 7faf9a05c640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7faf9415bca0 0x7faf940cda80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:40:01.541 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:01.534+0000 7faf9a05c640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7faf9415bca0 0x7faf940cda80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34242/0 (socket says 172.21.15.16:34242) 2023-05-22T15:40:01.541 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:01.534+0000 7faf9a05c640 1 -- 172.21.15.16:0/189277166 learned_addr learned my addr 172.21.15.16:0/189277166 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:40:01.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:01.534+0000 7faf9a85d640 1 --2- 172.21.15.16:0/189277166 >> v2:172.21.15.62:3300/0 conn(0x7faf940b5a50 0x7faf940d2900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:40:01.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:01.534+0000 7faf9a05c640 1 -- 172.21.15.16:0/189277166 >> v2:172.21.15.62:3300/0 conn(0x7faf940b5a50 msgr2=0x7faf940d2900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:40:01.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:01.534+0000 7faf9a05c640 1 --2- 172.21.15.16:0/189277166 >> v2:172.21.15.62:3300/0 conn(0x7faf940b5a50 0x7faf940d2900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:01.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:01.534+0000 7faf9a05c640 1 -- 172.21.15.16:0/189277166 >> v1:172.21.15.62:6789/0 conn(0x7faf940d2480 legacy=0x7faf940b5340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:40:01.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:01.534+0000 7faf9a05c640 1 -- 172.21.15.16:0/189277166 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf940ccfa0 con 0x7faf9415bca0 2023-05-22T15:40:01.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:01.534+0000 7faf9a85d640 1 --2- 172.21.15.16:0/189277166 >> v2:172.21.15.62:3300/0 conn(0x7faf940b5a50 0x7faf940d2900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:40:01.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:01.534+0000 7faf9a05c640 1 --2- 172.21.15.16:0/189277166 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faf9415bca0 0x7faf940cda80 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7faf940c5fc0 tx=0x7faf9001c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f35f77b433ddc374 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:40:01.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:01.534+0000 7faf9905a640 1 -- 172.21.15.16:0/189277166 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf900a5870 con 0x7faf9415bca0 2023-05-22T15:40:01.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:01.534+0000 7faf9905a640 1 -- 172.21.15.16:0/189277166 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf900a59d0 con 0x7faf9415bca0 2023-05-22T15:40:01.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:01.534+0000 7faf9905a640 1 -- 172.21.15.16:0/189277166 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf900a5d60 con 0x7faf9415bca0 2023-05-22T15:40:01.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:01.538+0000 7faf9c2c0640 1 -- 172.21.15.16:0/189277166 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faf9415bca0 msgr2=0x7faf940cda80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:40:01.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:01.538+0000 7faf9c2c0640 1 --2- 172.21.15.16:0/189277166 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faf9415bca0 0x7faf940cda80 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7faf940c5fc0 tx=0x7faf9001c990 comp rx=0 tx=0).stop 2023-05-22T15:40:01.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:01.538+0000 7faf9c2c0640 1 -- 172.21.15.16:0/189277166 shutdown_connections 2023-05-22T15:40:01.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:01.538+0000 7faf9c2c0640 1 --2- 172.21.15.16:0/189277166 >> v2:172.21.15.62:3300/0 conn(0x7faf940b5a50 0x7faf940d2900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:01.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:01.538+0000 7faf9c2c0640 1 --2- 172.21.15.16:0/189277166 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faf9415bca0 0x7faf940cda80 unknown :-1 s=CLOSED pgs=901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:01.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:01.538+0000 7faf9c2c0640 1 -- 172.21.15.16:0/189277166 >> 172.21.15.16:0/189277166 conn(0x7faf940655c0 msgr2=0x7faf94070e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:40:01.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:01.538+0000 7faf9c2c0640 1 -- 172.21.15.16:0/189277166 shutdown_connections 2023-05-22T15:40:01.549 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:01.538+0000 7faf9c2c0640 1 -- 172.21.15.16:0/189277166 wait complete. 2023-05-22T15:40:01.549 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:01.538+0000 7faf9c2c0640 1 Processor -- start 2023-05-22T15:40:01.549 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:01.538+0000 7faf9c2c0640 1 -- start start 2023-05-22T15:40:01.549 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:01.538+0000 7faf9c2c0640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faf940b5a50 0x7faf9415e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:40:01.549 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:01.538+0000 7faf9c2c0640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7faf940d2480 0x7faf9415a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:40:01.550 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:01.538+0000 7faf9c2c0640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faf9415bca0 0x7faf940c82e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:40:01.550 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:01.538+0000 7faf9c2c0640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf9415abf0 con 0x7faf9415bca0 2023-05-22T15:40:01.550 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:01.538+0000 7faf9c2c0640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf940ca840 con 0x7faf940b5a50 2023-05-22T15:40:01.550 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:01.538+0000 7faf9c2c0640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf940c8820 con 0x7faf940d2480 2023-05-22T15:40:01.550 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:01.538+0000 7faf9a85d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faf9415bca0 0x7faf940c82e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:40:01.551 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:01.538+0000 7faf9a85d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faf9415bca0 0x7faf940c82e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34246/0 (socket says 172.21.15.16:34246) 2023-05-22T15:40:01.551 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:01.538+0000 7faf9a85d640 1 -- 172.21.15.16:0/1000643331 learned_addr learned my addr 172.21.15.16:0/1000643331 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:40:01.551 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:01.538+0000 7faf9a05c640 1 --2- 172.21.15.16:0/1000643331 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faf940b5a50 0x7faf9415e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:40:01.551 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:01.538+0000 7faf9985b640 1 --2- 172.21.15.16:0/1000643331 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7faf940d2480 0x7faf9415a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:40:01.551 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:01.538+0000 7faf9a85d640 1 -- 172.21.15.16:0/1000643331 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7faf940d2480 msgr2=0x7faf9415a6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:40:01.551 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:01.538+0000 7faf9a85d640 1 --2- 172.21.15.16:0/1000643331 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7faf940d2480 0x7faf9415a6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:01.551 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:01.538+0000 7faf9a85d640 1 -- 172.21.15.16:0/1000643331 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faf940b5a50 msgr2=0x7faf9415e760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:40:01.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:01.538+0000 7faf9a85d640 1 --2- 172.21.15.16:0/1000643331 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faf940b5a50 0x7faf9415e760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:01.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:01.538+0000 7faf9a85d640 1 -- 172.21.15.16:0/1000643331 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf90009b30 con 0x7faf9415bca0 2023-05-22T15:40:01.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:01.538+0000 7faf9985b640 1 --2- 172.21.15.16:0/1000643331 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7faf940d2480 0x7faf9415a6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:40:01.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:01.538+0000 7faf9a85d640 1 --2- 172.21.15.16:0/1000643331 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faf9415bca0 0x7faf940c82e0 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7faf8c00c850 tx=0x7faf8c00cd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:40:01.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:01.538+0000 7faf8affd640 1 -- 172.21.15.16:0/1000643331 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf8c004240 con 0x7faf9415bca0 2023-05-22T15:40:01.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:01.538+0000 7faf8affd640 1 -- 172.21.15.16:0/1000643331 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf8c0043c0 con 0x7faf9415bca0 2023-05-22T15:40:01.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:01.538+0000 7faf9c2c0640 1 -- 172.21.15.16:0/1000643331 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf94160360 con 0x7faf9415bca0 2023-05-22T15:40:01.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:01.538+0000 7faf8affd640 1 -- 172.21.15.16:0/1000643331 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf8c004530 con 0x7faf9415bca0 2023-05-22T15:40:01.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:01.538+0000 7faf9c2c0640 1 -- 172.21.15.16:0/1000643331 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf940cdfc0 con 0x7faf9415bca0 2023-05-22T15:40:01.554 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:01.542+0000 7faf9c2c0640 1 -- 172.21.15.16:0/1000643331 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf941604f0 con 0x7faf9415bca0 2023-05-22T15:40:01.554 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:01.542+0000 7faf8affd640 1 -- 172.21.15.16:0/1000643331 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) v1 ==== 84332+0+0 (secure 0 0 0) 0x7faf8c003860 con 0x7faf9415bca0 2023-05-22T15:40:01.554 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:01.542+0000 7faf8affd640 1 --2- 172.21.15.16:0/1000643331 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7faf780654a0 0x7faf78067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:40:01.555 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:01.542+0000 7faf8affd640 1 -- 172.21.15.16:0/1000643331 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 5900+0+0 (secure 0 0 0) 0x7faf8c082c50 con 0x7faf9415bca0 2023-05-22T15:40:01.555 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:01.542+0000 7faf9a05c640 1 --2- 172.21.15.16:0/1000643331 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7faf780654a0 0x7faf78067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:40:01.555 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:01.542+0000 7faf9a05c640 1 --2- 172.21.15.16:0/1000643331 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7faf780654a0 0x7faf78067950 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7faf900ae000 tx=0x7faf900a7070 comp rx=0 tx=0).ready entity=mgr.6240 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:40:01.555 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:01.546+0000 7faf8affd640 1 -- 172.21.15.16:0/1000643331 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7faf8c088020 con 0x7faf9415bca0 2023-05-22T15:40:01.709 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:01.702+0000 7faf9c2c0640 1 -- 172.21.15.16:0/1000643331 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7faf9404dbe0 con 0x7faf9415bca0 2023-05-22T15:40:02.382 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:02.374+0000 7faf8affd640 1 -- 172.21.15.16:0/1000643331 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v71) v1 ==== 170+0+0 (secure 0 0 0) 0x7faf8c0150d0 con 0x7faf9415bca0 2023-05-22T15:40:02.385 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'cephfs_data' created 2023-05-22T15:40:02.393 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:02.386+0000 7faf9c2c0640 1 -- 172.21.15.16:0/1000643331 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7faf780654a0 msgr2=0x7faf78067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:40:02.393 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:02.386+0000 7faf9c2c0640 1 --2- 172.21.15.16:0/1000643331 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7faf780654a0 0x7faf78067950 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7faf900ae000 tx=0x7faf900a7070 comp rx=0 tx=0).stop 2023-05-22T15:40:02.394 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:02.386+0000 7faf9c2c0640 1 -- 172.21.15.16:0/1000643331 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faf9415bca0 msgr2=0x7faf940c82e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:40:02.394 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:02.386+0000 7faf9c2c0640 1 --2- 172.21.15.16:0/1000643331 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faf9415bca0 0x7faf940c82e0 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7faf8c00c850 tx=0x7faf8c00cd10 comp rx=0 tx=0).stop 2023-05-22T15:40:02.397 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:02.390+0000 7faf9c2c0640 1 -- 172.21.15.16:0/1000643331 shutdown_connections 2023-05-22T15:40:02.397 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:02.390+0000 7faf9c2c0640 1 --2- 172.21.15.16:0/1000643331 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faf9415bca0 0x7faf940c82e0 unknown :-1 s=CLOSED pgs=902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:02.397 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:02.390+0000 7faf9c2c0640 1 --2- 172.21.15.16:0/1000643331 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7faf780654a0 0x7faf78067950 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:02.397 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:02.390+0000 7faf9c2c0640 1 --2- 172.21.15.16:0/1000643331 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7faf940d2480 0x7faf9415a6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:02.397 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:02.390+0000 7faf9c2c0640 1 --2- 172.21.15.16:0/1000643331 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faf940b5a50 0x7faf9415e760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:02.398 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:02.390+0000 7faf9c2c0640 1 -- 172.21.15.16:0/1000643331 >> 172.21.15.16:0/1000643331 conn(0x7faf940655c0 msgr2=0x7faf9415f240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:40:02.398 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:02.394+0000 7faf9c2c0640 1 -- 172.21.15.16:0/1000643331 shutdown_connections 2023-05-22T15:40:02.398 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:02.394+0000 7faf9c2c0640 1 -- 172.21.15.16:0/1000643331 wait complete. 2023-05-22T15:40:02.427 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-05-22T15:40:02.550 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:02.546+0000 7fd05ef51640 1 Processor -- start 2023-05-22T15:40:02.551 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:02.546+0000 7fd05ef51640 1 -- start start 2023-05-22T15:40:02.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:02.546+0000 7fd05ef51640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fd058159d40 0x7fd0580cb260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:40:02.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:02.546+0000 7fd05ef51640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd058070c30 con 0x7fd058159970 2023-05-22T15:40:02.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:02.546+0000 7fd05ef51640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd058070e60 con 0x7fd05806b6c0 2023-05-22T15:40:02.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:02.546+0000 7fd05ef51640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd05815dd90 con 0x7fd058159d40 2023-05-22T15:40:02.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:02.546+0000 7fd05cced640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fd058159970 0x7fd05815d680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:41320/0 (socket says 172.21.15.16:41320) 2023-05-22T15:40:02.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:02.546+0000 7fd05cced640 1 -- 172.21.15.16:0/2145268188 learned_addr learned my addr 172.21.15.16:0/2145268188 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:40:02.554 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:02.546+0000 7fd057fff640 1 --2- 172.21.15.16:0/2145268188 >> v2:172.21.15.62:3300/0 conn(0x7fd058159d40 0x7fd0580cb260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:40:02.555 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:02.546+0000 7fd0577fe640 1 -- 172.21.15.16:0/2145268188 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 397189556 0 0) 0x7fd058070c30 con 0x7fd058159970 2023-05-22T15:40:02.555 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:02.546+0000 7fd0577fe640 1 -- 172.21.15.16:0/2145268188 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd040003530 con 0x7fd058159970 2023-05-22T15:40:02.555 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:02.546+0000 7fd0577fe640 1 -- 172.21.15.16:0/2145268188 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3189913184 0 0) 0x7fd040003530 con 0x7fd058159970 2023-05-22T15:40:02.555 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:02.546+0000 7fd0577fe640 1 -- 172.21.15.16:0/2145268188 >> v2:172.21.15.62:3300/0 conn(0x7fd058159d40 msgr2=0x7fd0580cb260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:40:02.556 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:02.546+0000 7fd0577fe640 1 --2- 172.21.15.16:0/2145268188 >> v2:172.21.15.62:3300/0 conn(0x7fd058159d40 0x7fd0580cb260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:02.556 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:02.546+0000 7fd0577fe640 1 -- 172.21.15.16:0/2145268188 >> v1:172.21.15.62:6789/0 conn(0x7fd05806b6c0 legacy=0x7fd05815a5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:40:02.556 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:02.546+0000 7fd0577fe640 1 -- 172.21.15.16:0/2145268188 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd058071140 con 0x7fd058159970 2023-05-22T15:40:02.557 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:02.546+0000 7fd0577fe640 1 -- 172.21.15.16:0/2145268188 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7fd048001680 con 0x7fd058159970 2023-05-22T15:40:02.557 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:02.546+0000 7fd0577fe640 1 -- 172.21.15.16:0/2145268188 >> v1:172.21.15.16:6789/0 conn(0x7fd058159970 legacy=0x7fd05815d680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:40:02.557 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:02.546+0000 7fd0577fe640 1 --2- 172.21.15.16:0/2145268188 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd040003980 0x7fd040003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:40:02.557 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:02.546+0000 7fd0577fe640 1 --2- 172.21.15.16:0/2145268188 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd0400047d0 0x7fd040004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:40:02.558 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:02.546+0000 7fd0577fe640 1 --2- 172.21.15.16:0/2145268188 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd040005180 0x7fd04000b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:40:02.558 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:02.546+0000 7fd0577fe640 1 -- 172.21.15.16:0/2145268188 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd05815dd90 con 0x7fd040003980 2023-05-22T15:40:02.558 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:02.546+0000 7fd0577fe640 1 -- 172.21.15.16:0/2145268188 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd04000c090 con 0x7fd040005180 2023-05-22T15:40:02.559 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:02.546+0000 7fd0577fe640 1 -- 172.21.15.16:0/2145268188 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd04000c250 con 0x7fd0400047d0 2023-05-22T15:40:02.559 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:02.546+0000 7fd05cced640 1 --2- 172.21.15.16:0/2145268188 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd040003980 0x7fd040003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:40:02.559 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:02.546+0000 7fd05d4ee640 1 --2- 172.21.15.16:0/2145268188 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd040005180 0x7fd04000b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:40:02.560 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:02.546+0000 7fd057fff640 1 --2- 172.21.15.16:0/2145268188 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd0400047d0 0x7fd040004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:40:02.560 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:02.546+0000 7fd05cced640 1 -- 172.21.15.16:0/2145268188 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd0400047d0 msgr2=0x7fd040004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:40:02.560 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:02.546+0000 7fd05cced640 1 --2- 172.21.15.16:0/2145268188 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd0400047d0 0x7fd040004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:02.561 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:02.546+0000 7fd05cced640 1 -- 172.21.15.16:0/2145268188 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd040005180 msgr2=0x7fd04000b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:40:02.561 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:02.546+0000 7fd05cced640 1 --2- 172.21.15.16:0/2145268188 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd040005180 0x7fd04000b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:02.561 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:02.546+0000 7fd05cced640 1 -- 172.21.15.16:0/2145268188 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd040003530 con 0x7fd040003980 2023-05-22T15:40:02.561 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:02.546+0000 7fd05d4ee640 1 -- 172.21.15.16:0/2145268188 reap_dead start 2023-05-22T15:40:02.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:02.546+0000 7fd057fff640 1 --2- 172.21.15.16:0/2145268188 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd0400047d0 0x7fd040004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T15:40:02.566 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:02.550+0000 7fd05cced640 1 --2- 172.21.15.16:0/2145268188 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd040003980 0x7fd040003d90 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7fd04800c240 tx=0x7fd04801cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c76e6e11287946 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:40:02.566 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:02.550+0000 7fd0577fe640 1 -- 172.21.15.16:0/2145268188 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd048005ae0 con 0x7fd040003980 2023-05-22T15:40:02.567 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:02.550+0000 7fd0577fe640 1 -- 172.21.15.16:0/2145268188 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd048005c40 con 0x7fd040003980 2023-05-22T15:40:02.567 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:02.550+0000 7fd0577fe640 1 -- 172.21.15.16:0/2145268188 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0480a5a50 con 0x7fd040003980 2023-05-22T15:40:02.567 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:02.550+0000 7fd05ef51640 1 -- 172.21.15.16:0/2145268188 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd040003980 msgr2=0x7fd040003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:40:02.567 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:02.550+0000 7fd05ef51640 1 --2- 172.21.15.16:0/2145268188 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd040003980 0x7fd040003d90 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7fd04800c240 tx=0x7fd04801cc40 comp rx=0 tx=0).stop 2023-05-22T15:40:02.567 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:02.550+0000 7fd05ef51640 1 -- 172.21.15.16:0/2145268188 shutdown_connections 2023-05-22T15:40:02.567 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:02.550+0000 7fd05ef51640 1 --2- 172.21.15.16:0/2145268188 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd040003980 0x7fd040003d90 unknown :-1 s=CLOSED pgs=904 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:02.567 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:02.550+0000 7fd05ef51640 1 -- 172.21.15.16:0/2145268188 >> 172.21.15.16:0/2145268188 conn(0x7fd058069720 msgr2=0x7fd058069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:40:02.568 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:02.550+0000 7fd05ef51640 1 -- 172.21.15.16:0/2145268188 shutdown_connections 2023-05-22T15:40:02.568 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:02.550+0000 7fd05ef51640 1 -- 172.21.15.16:0/2145268188 wait complete. 2023-05-22T15:40:02.568 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:02.550+0000 7fd05ef51640 1 Processor -- start 2023-05-22T15:40:02.568 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:02.550+0000 7fd05ef51640 1 -- start start 2023-05-22T15:40:02.568 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:02.554+0000 7fd05ef51640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd040003980 0x7fd0580b0e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:40:02.568 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:02.554+0000 7fd05ef51640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd0580b0650 0x7fd058145410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:40:02.569 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:02.554+0000 7fd05ef51640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd05815c5c0 0x7fd05815ca30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:40:02.569 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:02.554+0000 7fd05ef51640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd05815cf70 con 0x7fd040003980 2023-05-22T15:40:02.569 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:02.554+0000 7fd05ef51640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd058145950 con 0x7fd0580b0650 2023-05-22T15:40:02.569 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:02.554+0000 7fd05ef51640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd058152270 con 0x7fd05815c5c0 2023-05-22T15:40:02.569 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:02.554+0000 7fd05cced640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd040003980 0x7fd0580b0e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:40:02.569 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:02.554+0000 7fd05cced640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd040003980 0x7fd0580b0e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34270/0 (socket says 172.21.15.16:34270) 2023-05-22T15:40:02.569 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:02.554+0000 7fd05cced640 1 -- 172.21.15.16:0/4031437707 learned_addr learned my addr 172.21.15.16:0/4031437707 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:40:02.570 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:02.554+0000 7fd057fff640 1 --2- 172.21.15.16:0/4031437707 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd0580b0650 0x7fd058145410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:40:02.570 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:02.554+0000 7fd05d4ee640 1 --2- 172.21.15.16:0/4031437707 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd05815c5c0 0x7fd05815ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:40:02.570 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:02.554+0000 7fd05cced640 1 -- 172.21.15.16:0/4031437707 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd05815c5c0 msgr2=0x7fd05815ca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:40:02.570 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:02.554+0000 7fd05cced640 1 --2- 172.21.15.16:0/4031437707 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd05815c5c0 0x7fd05815ca30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:02.570 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:02.554+0000 7fd05cced640 1 -- 172.21.15.16:0/4031437707 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd0580b0650 msgr2=0x7fd058145410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:40:02.570 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:02.554+0000 7fd05cced640 1 --2- 172.21.15.16:0/4031437707 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd0580b0650 0x7fd058145410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:02.570 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:02.554+0000 7fd05cced640 1 -- 172.21.15.16:0/4031437707 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd05806bea0 con 0x7fd040003980 2023-05-22T15:40:02.571 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:02.554+0000 7fd057fff640 1 --2- 172.21.15.16:0/4031437707 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd0580b0650 0x7fd058145410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:40:02.571 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:02.554+0000 7fd05d4ee640 1 --2- 172.21.15.16:0/4031437707 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd05815c5c0 0x7fd05815ca30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:40:02.571 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:02.554+0000 7fd05cced640 1 --2- 172.21.15.16:0/4031437707 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd040003980 0x7fd0580b0e40 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7fd04800c240 tx=0x7fd040005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:40:02.571 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:02.554+0000 7fd0557fa640 1 -- 172.21.15.16:0/4031437707 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0480b4070 con 0x7fd040003980 2023-05-22T15:40:02.571 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:02.554+0000 7fd0557fa640 1 -- 172.21.15.16:0/4031437707 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd048005db0 con 0x7fd040003980 2023-05-22T15:40:02.571 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:02.554+0000 7fd05ef51640 1 -- 172.21.15.16:0/4031437707 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd05815da90 con 0x7fd040003980 2023-05-22T15:40:02.571 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:02.554+0000 7fd0557fa640 1 -- 172.21.15.16:0/4031437707 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd048001680 con 0x7fd040003980 2023-05-22T15:40:02.572 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:02.554+0000 7fd05ef51640 1 -- 172.21.15.16:0/4031437707 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd05806c030 con 0x7fd040003980 2023-05-22T15:40:02.572 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:02.554+0000 7fd05ef51640 1 -- 172.21.15.16:0/4031437707 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd05804dbe0 con 0x7fd040003980 2023-05-22T15:40:02.572 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:02.554+0000 7fd0557fa640 1 -- 172.21.15.16:0/4031437707 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) v1 ==== 84332+0+0 (secure 0 0 0) 0x7fd0480a5810 con 0x7fd040003980 2023-05-22T15:40:02.572 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:02.558+0000 7fd0557fa640 1 --2- 172.21.15.16:0/4031437707 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd0440654d0 0x7fd044067980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:40:02.572 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:02.558+0000 7fd0557fa640 1 -- 172.21.15.16:0/4031437707 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6282+0+0 (secure 0 0 0) 0x7fd04811c940 con 0x7fd040003980 2023-05-22T15:40:02.573 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:02.558+0000 7fd057fff640 1 --2- 172.21.15.16:0/4031437707 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd0440654d0 0x7fd044067980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:40:02.573 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:02.558+0000 7fd057fff640 1 --2- 172.21.15.16:0/4031437707 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd0440654d0 0x7fd044067980 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fd058153420 tx=0x7fd04c008040 comp rx=0 tx=0).ready entity=mgr.6240 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:40:02.573 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:02.558+0000 7fd0557fa640 1 -- 172.21.15.16:0/4031437707 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fd0480ea4e0 con 0x7fd040003980 2023-05-22T15:40:02.753 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:02.746+0000 7fd05ef51640 1 -- 172.21.15.16:0/4031437707 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fd0580c16a0 con 0x7fd040003980 2023-05-22T15:40:02.771 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:02.766+0000 7fd0557fa640 1 -- 172.21.15.16:0/4031437707 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 8 and data pool 9 v18) v1 ==== 172+0+0 (secure 0 0 0) 0x7fd0480edb00 con 0x7fd040003980 2023-05-22T15:40:02.773 INFO:teuthology.orchestra.run.smithi016.stderr:new fs with metadata pool 8 and data pool 9 2023-05-22T15:40:02.784 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:02.778+0000 7fd05ef51640 1 -- 172.21.15.16:0/4031437707 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd0440654d0 msgr2=0x7fd044067980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:40:02.785 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:02.778+0000 7fd05ef51640 1 --2- 172.21.15.16:0/4031437707 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd0440654d0 0x7fd044067980 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fd058153420 tx=0x7fd04c008040 comp rx=0 tx=0).stop 2023-05-22T15:40:02.785 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:02.778+0000 7fd05ef51640 1 -- 172.21.15.16:0/4031437707 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd040003980 msgr2=0x7fd0580b0e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:40:02.785 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:02.778+0000 7fd05ef51640 1 --2- 172.21.15.16:0/4031437707 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd040003980 0x7fd0580b0e40 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7fd04800c240 tx=0x7fd040005800 comp rx=0 tx=0).stop 2023-05-22T15:40:02.786 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:02.778+0000 7fd05ef51640 1 -- 172.21.15.16:0/4031437707 shutdown_connections 2023-05-22T15:40:02.786 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:02.778+0000 7fd05ef51640 1 --2- 172.21.15.16:0/4031437707 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd0440654d0 0x7fd044067980 secure :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fd058153420 tx=0x7fd04c008040 comp rx=0 tx=0).stop 2023-05-22T15:40:02.787 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:02.778+0000 7fd05ef51640 1 --2- 172.21.15.16:0/4031437707 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd05815c5c0 0x7fd05815ca30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:02.787 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:02.778+0000 7fd05ef51640 1 --2- 172.21.15.16:0/4031437707 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd0580b0650 0x7fd058145410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:02.787 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:02.778+0000 7fd05ef51640 1 --2- 172.21.15.16:0/4031437707 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd040003980 0x7fd0580b0e40 unknown :-1 s=CLOSED pgs=905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:02.787 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:02.778+0000 7fd05ef51640 1 -- 172.21.15.16:0/4031437707 >> 172.21.15.16:0/4031437707 conn(0x7fd058069720 msgr2=0x7fd0580c2c50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:40:02.791 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:02.786+0000 7fd05ef51640 1 -- 172.21.15.16:0/4031437707 shutdown_connections 2023-05-22T15:40:02.792 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:02.786+0000 7fd05ef51640 1 -- 172.21.15.16:0/4031437707 wait complete. 2023-05-22T15:40:02.814 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-05-22T15:40:02.934 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:02.926+0000 7f048f503640 1 Processor -- start 2023-05-22T15:40:02.934 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:02.930+0000 7f048f503640 1 -- start start 2023-05-22T15:40:02.935 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:02.930+0000 7f048f503640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f04880b7dd0 0x7f04880ccab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:40:02.936 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:02.930+0000 7f048f503640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f0488160250 0x7f04880d0410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:40:02.936 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:02.930+0000 7f048f503640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04880cd170 con 0x7f048815f700 2023-05-22T15:40:02.936 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:02.930+0000 7f048f503640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04880d3360 con 0x7f04880b7dd0 2023-05-22T15:40:02.937 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:02.930+0000 7f048f503640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04880d2ac0 con 0x7f0488160250 2023-05-22T15:40:02.937 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:02.930+0000 7f048d29f640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f04880b7dd0 0x7f04880ccab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:40:02.937 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:02.930+0000 7f048ca9e640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f048815f700 0x7f048815fb40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:41322/0 (socket says 172.21.15.16:41322) 2023-05-22T15:40:02.937 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:02.930+0000 7f048ca9e640 1 -- 172.21.15.16:0/2492143497 learned_addr learned my addr 172.21.15.16:0/2492143497 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:40:02.938 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:02.930+0000 7f048daa0640 1 --2- 172.21.15.16:0/2492143497 >> v2:172.21.15.62:3300/0 conn(0x7f0488160250 0x7f04880d0410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:40:02.938 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:02.930+0000 7f048d29f640 1 -- 172.21.15.16:0/2492143497 >> v2:172.21.15.62:3300/0 conn(0x7f0488160250 msgr2=0x7f04880d0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:40:02.938 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:02.930+0000 7f048d29f640 1 --2- 172.21.15.16:0/2492143497 >> v2:172.21.15.62:3300/0 conn(0x7f0488160250 0x7f04880d0410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:02.938 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:02.930+0000 7f048d29f640 1 -- 172.21.15.16:0/2492143497 >> v1:172.21.15.16:6789/0 conn(0x7f048815f700 legacy=0x7f048815fb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:40:02.939 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:02.930+0000 7f047ffff640 1 -- 172.21.15.16:0/2492143497 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1833662920 0 0) 0x7f04880cd170 con 0x7f048815f700 2023-05-22T15:40:02.939 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:02.930+0000 7f048d29f640 1 -- 172.21.15.16:0/2492143497 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04880d2da0 con 0x7f04880b7dd0 2023-05-22T15:40:02.944 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:02.930+0000 7f048d29f640 1 --2- 172.21.15.16:0/2492143497 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f04880b7dd0 0x7f04880ccab0 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7f048815c190 tx=0x7f048401c430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9c6cfa46fbd0565 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:40:02.944 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:02.934+0000 7f047ffff640 1 -- 172.21.15.16:0/2492143497 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04840ab5f0 con 0x7f04880b7dd0 2023-05-22T15:40:02.944 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:02.934+0000 7f047ffff640 1 -- 172.21.15.16:0/2492143497 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04840ab750 con 0x7f04880b7dd0 2023-05-22T15:40:02.944 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:02.934+0000 7f047ffff640 1 -- 172.21.15.16:0/2492143497 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04840abae0 con 0x7f04880b7dd0 2023-05-22T15:40:02.945 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:02.934+0000 7f048f503640 1 -- 172.21.15.16:0/2492143497 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f04880b7dd0 msgr2=0x7f04880ccab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:40:02.945 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:02.934+0000 7f048f503640 1 --2- 172.21.15.16:0/2492143497 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f04880b7dd0 0x7f04880ccab0 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7f048815c190 tx=0x7f048401c430 comp rx=0 tx=0).stop 2023-05-22T15:40:02.945 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:02.934+0000 7f048f503640 1 -- 172.21.15.16:0/2492143497 shutdown_connections 2023-05-22T15:40:02.945 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:02.934+0000 7f048f503640 1 --2- 172.21.15.16:0/2492143497 >> v2:172.21.15.62:3300/0 conn(0x7f0488160250 0x7f04880d0410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:02.946 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:02.934+0000 7f048f503640 1 --2- 172.21.15.16:0/2492143497 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f04880b7dd0 0x7f04880ccab0 unknown :-1 s=CLOSED pgs=907 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:02.946 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:02.934+0000 7f048f503640 1 -- 172.21.15.16:0/2492143497 >> 172.21.15.16:0/2492143497 conn(0x7f04880655c0 msgr2=0x7f0488070e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:40:02.946 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:02.934+0000 7f048f503640 1 -- 172.21.15.16:0/2492143497 shutdown_connections 2023-05-22T15:40:02.947 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:02.934+0000 7f048f503640 1 -- 172.21.15.16:0/2492143497 wait complete. 2023-05-22T15:40:02.947 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:02.934+0000 7f048f503640 1 Processor -- start 2023-05-22T15:40:02.947 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:02.934+0000 7f048f503640 1 -- start start 2023-05-22T15:40:02.947 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:02.934+0000 7f048f503640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f04880b7dd0 0x7f048814a300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:40:02.947 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:02.934+0000 7f048f503640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f048815f700 0x7f0488165b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:40:02.947 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:02.934+0000 7f048f503640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0488160250 0x7f0488166050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:40:02.948 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:02.934+0000 7f048f503640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04881f7cf0 con 0x7f0488160250 2023-05-22T15:40:02.948 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:02.934+0000 7f048f503640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04881f70e0 con 0x7f048815f700 2023-05-22T15:40:02.948 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:02.934+0000 7f048f503640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04881f73c0 con 0x7f04880b7dd0 2023-05-22T15:40:02.948 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:02.934+0000 7f048daa0640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0488160250 0x7f0488166050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:40:02.948 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:02.934+0000 7f048daa0640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0488160250 0x7f0488166050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34284/0 (socket says 172.21.15.16:34284) 2023-05-22T15:40:02.948 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:02.934+0000 7f048daa0640 1 -- 172.21.15.16:0/3163908898 learned_addr learned my addr 172.21.15.16:0/3163908898 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:40:02.948 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:02.934+0000 7f048ca9e640 1 --2- 172.21.15.16:0/3163908898 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f048815f700 0x7f0488165b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:40:02.949 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:02.934+0000 7f048d29f640 1 --2- 172.21.15.16:0/3163908898 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f04880b7dd0 0x7f048814a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:40:02.949 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:02.934+0000 7f048daa0640 1 -- 172.21.15.16:0/3163908898 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f04880b7dd0 msgr2=0x7f048814a300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:40:02.949 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:02.934+0000 7f048daa0640 1 --2- 172.21.15.16:0/3163908898 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f04880b7dd0 0x7f048814a300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:02.949 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:02.934+0000 7f048daa0640 1 -- 172.21.15.16:0/3163908898 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f048815f700 msgr2=0x7f0488165b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:40:02.949 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:02.934+0000 7f048daa0640 1 --2- 172.21.15.16:0/3163908898 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f048815f700 0x7f0488165b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:02.949 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:02.934+0000 7f048daa0640 1 -- 172.21.15.16:0/3163908898 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0484009420 con 0x7f0488160250 2023-05-22T15:40:02.949 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:02.934+0000 7f048ca9e640 1 --2- 172.21.15.16:0/3163908898 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f048815f700 0x7f0488165b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T15:40:02.950 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:02.934+0000 7f048daa0640 1 --2- 172.21.15.16:0/3163908898 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0488160250 0x7f0488166050 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7f0480007e60 tx=0x7f048000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:40:02.950 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:02.938+0000 7f047dffb640 1 -- 172.21.15.16:0/3163908898 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0480014420 con 0x7f0488160250 2023-05-22T15:40:02.950 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:02.938+0000 7f047dffb640 1 -- 172.21.15.16:0/3163908898 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04800145a0 con 0x7f0488160250 2023-05-22T15:40:02.950 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:02.938+0000 7f047dffb640 1 -- 172.21.15.16:0/3163908898 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0480014950 con 0x7f0488160250 2023-05-22T15:40:02.950 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:02.938+0000 7f048f503640 1 -- 172.21.15.16:0/3163908898 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04880d12f0 con 0x7f0488160250 2023-05-22T15:40:02.950 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:02.938+0000 7f048f503640 1 -- 172.21.15.16:0/3163908898 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04880d1160 con 0x7f0488160250 2023-05-22T15:40:02.950 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:02.938+0000 7f048f503640 1 -- 172.21.15.16:0/3163908898 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04880d1480 con 0x7f0488160250 2023-05-22T15:40:02.950 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:02.938+0000 7f047dffb640 1 -- 172.21.15.16:0/3163908898 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f0480014ab0 con 0x7f0488160250 2023-05-22T15:40:02.951 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:02.938+0000 7f047dffb640 1 --2- 172.21.15.16:0/3163908898 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f046c0654a0 0x7f046c067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:40:02.952 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:02.938+0000 7f047dffb640 1 -- 172.21.15.16:0/3163908898 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f04800833e0 con 0x7f0488160250 2023-05-22T15:40:02.952 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:02.938+0000 7f048d29f640 1 --2- 172.21.15.16:0/3163908898 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f046c0654a0 0x7f046c067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:40:02.952 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:02.938+0000 7f048d29f640 1 --2- 172.21.15.16:0/3163908898 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f046c0654a0 0x7f046c067950 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f04840af000 tx=0x7f04840a7000 comp rx=0 tx=0).ready entity=mgr.6240 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:40:02.952 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:02.942+0000 7f047dffb640 1 -- 172.21.15.16:0/3163908898 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f04800548d0 con 0x7f0488160250 2023-05-22T15:40:03.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:03.102+0000 7f048f503640 1 -- 172.21.15.16:0/3163908898 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f048804dbe0 con 0x7f0488160250 2023-05-22T15:40:03.108 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:03.102+0000 7f047dffb640 1 -- 172.21.15.16:0/3163908898 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v72) v1 ==== 74+0+18049 (secure 0 0 0) 0x7f04800544c0 con 0x7f0488160250 2023-05-22T15:40:03.109 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:40:03.109 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":72,"fsid":"b2719834-7847-4d98-90a6-f65d57f85c49","created":"2023-05-22T15:26:25.259314+0000","modified":"2023-05-22T15:40:02.753701+0000","last_up_change":"2023-05-22T15:26:36.448524+0000","last_in_change":"2023-05-22T15:26:27.675668+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-05-22T15:26:38.019781+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-05-22T15:26:43.570514+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2023-05-22T15:40:00.663040+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"72","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2023-05-22T15:40:01.750680+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"72","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9e7ff854-b25f-4cbb-a5c2-1799d9c53d2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":37021},{"type":"v1","addr":"172.21.15.16:6825","nonce":37021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":37021},{"type":"v1","addr":"172.21.15.16:6827","nonce":37021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":37021},{"type":"v1","addr":"172.21.15.16:6831","nonce":37021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":37021},{"type":"v1","addr":"172.21.15.16:6829","nonce":37021}]},"public_addr":"172.21.15.16:6825/37021","cluster_addr":"172.21.15.16:6827/37021","heartbeat_back_addr":"172.21.15.16:6831/37021","heartbeat_front_addr":"172.21.15.16:6829/37021","state":["exists","up"]},{"osd":1,"uuid":"4702fbc9-a927-4565-aca5-41dc69cc8264","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":37008},{"type":"v1","addr":"172.21.15.16:6801","nonce":37008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":37008},{"type":"v1","addr":"172.21.15.16:6803","nonce":37008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":37008},{"type":"v1","addr":"172.21.15.16:6807","nonce":37008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":37008},{"type":"v1","addr":"172.21.15.16:6805","nonce":37008}]},"public_addr":"172.21.15.16:6801/37008","cluster_addr":"172.21.15.16:6803/37008","heartbeat_back_addr":"172.21.15.16:6807/37008","heartbeat_front_addr":"172.21.15.16:6805/37008","state":["exists","up"]},{"osd":2,"uuid":"1e7bba46-4eba-47ef-bb81-5e2d371638ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":37009},{"type":"v1","addr":"172.21.15.16:6809","nonce":37009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":37009},{"type":"v1","addr":"172.21.15.16:6811","nonce":37009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":37009},{"type":"v1","addr":"172.21.15.16:6815","nonce":37009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":37009},{"type":"v1","addr":"172.21.15.16:6813","nonce":37009}]},"public_addr":"172.21.15.16:6809/37009","cluster_addr":"172.21.15.16:6811/37009","heartbeat_back_addr":"172.21.15.16:6815/37009","heartbeat_front_addr":"172.21.15.16:6813/37009","state":["exists","up"]},{"osd":3,"uuid":"383cacc0-5a88-4904-990e-498ca59ef48a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":37020},{"type":"v1","addr":"172.21.15.16:6817","nonce":37020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":37020},{"type":"v1","addr":"172.21.15.16:6819","nonce":37020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":37020},{"type":"v1","addr":"172.21.15.16:6823","nonce":37020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":37020},{"type":"v1","addr":"172.21.15.16:6821","nonce":37020}]},"public_addr":"172.21.15.16:6817/37020","cluster_addr":"172.21.15.16:6819/37020","heartbeat_back_addr":"172.21.15.16:6823/37020","heartbeat_front_addr":"172.21.15.16:6821/37020","state":["exists","up"]},{"osd":4,"uuid":"970500ef-95b7-42c1-b080-eed4fae7013d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":37353},{"type":"v1","addr":"172.21.15.62:6817","nonce":37353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6819","nonce":37353},{"type":"v1","addr":"172.21.15.62:6822","nonce":37353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37353},{"type":"v1","addr":"172.21.15.62:6831","nonce":37353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37353},{"type":"v1","addr":"172.21.15.62:6826","nonce":37353}]},"public_addr":"172.21.15.62:6817/37353","cluster_addr":"172.21.15.62:6822/37353","heartbeat_back_addr":"172.21.15.62:6831/37353","heartbeat_front_addr":"172.21.15.62:6826/37353","state":["exists","up"]},{"osd":5,"uuid":"022d67db-d4d8-445b-b1b9-8b446a00f5c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":37352},{"type":"v1","addr":"172.21.15.62:6801","nonce":37352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":37352},{"type":"v1","addr":"172.21.15.62:6803","nonce":37352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":37352},{"type":"v1","addr":"172.21.15.62:6807","nonce":37352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":37352},{"type":"v1","addr":"172.21.15.62:6805","nonce":37352}]},"public_addr":"172.21.15.62:6801/37352","cluster_addr":"172.21.15.62:6803/37352","heartbeat_back_addr":"172.21.15.62:6807/37352","heartbeat_front_addr":"172.21.15.62:6805/37352","state":["exists","up"]},{"osd":6,"uuid":"42bd372b-6688-4a1b-8407-e16e8c266e33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":37354},{"type":"v1","addr":"172.21.15.62:6820","nonce":37354}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":37354},{"type":"v1","addr":"172.21.15.62:6823","nonce":37354}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6829","nonce":37354},{"type":"v1","addr":"172.21.15.62:6830","nonce":37354}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6825","nonce":37354},{"type":"v1","addr":"172.21.15.62:6827","nonce":37354}]},"public_addr":"172.21.15.62:6820/37354","cluster_addr":"172.21.15.62:6823/37354","heartbeat_back_addr":"172.21.15.62:6830/37354","heartbeat_front_addr":"172.21.15.62:6827/37354","state":["exists","up"]},{"osd":7,"uuid":"ac70b2a3-f41b-4bf2-b227-7f2cbb55426d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37355},{"type":"v1","addr":"172.21.15.62:6809","nonce":37355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37355},{"type":"v1","addr":"172.21.15.62:6811","nonce":37355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":37355},{"type":"v1","addr":"172.21.15.62:6815","nonce":37355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":37355},{"type":"v1","addr":"172.21.15.62:6813","nonce":37355}]},"public_addr":"172.21.15.62:6809/37355","cluster_addr":"172.21.15.62:6811/37355","heartbeat_back_addr":"172.21.15.62:6815/37355","heartbeat_front_addr":"172.21.15.62:6813/37355","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.368896+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.107273+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.276888+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.287502+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.124791+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.251937+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.255679+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.232797+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-05-22T15:40:03.111 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:03.106+0000 7f048f503640 1 -- 172.21.15.16:0/3163908898 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f046c0654a0 msgr2=0x7f046c067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:40:03.111 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:03.106+0000 7f048f503640 1 --2- 172.21.15.16:0/3163908898 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f046c0654a0 0x7f046c067950 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f04840af000 tx=0x7f04840a7000 comp rx=0 tx=0).stop 2023-05-22T15:40:03.111 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:03.106+0000 7f048f503640 1 -- 172.21.15.16:0/3163908898 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0488160250 msgr2=0x7f0488166050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:40:03.111 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:03.106+0000 7f048f503640 1 --2- 172.21.15.16:0/3163908898 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0488160250 0x7f0488166050 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7f0480007e60 tx=0x7f048000e4b0 comp rx=0 tx=0).stop 2023-05-22T15:40:03.111 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:03.106+0000 7f048f503640 1 -- 172.21.15.16:0/3163908898 shutdown_connections 2023-05-22T15:40:03.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:03.106+0000 7f048f503640 1 --2- 172.21.15.16:0/3163908898 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0488160250 0x7f0488166050 unknown :-1 s=CLOSED pgs=908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:03.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:03.106+0000 7f048f503640 1 --2- 172.21.15.16:0/3163908898 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f048815f700 0x7f0488165b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:03.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:03.106+0000 7f048f503640 1 --2- 172.21.15.16:0/3163908898 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f046c0654a0 0x7f046c067950 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:03.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:03.106+0000 7f048f503640 1 --2- 172.21.15.16:0/3163908898 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f04880b7dd0 0x7f048814a300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:03.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:03.106+0000 7f048f503640 1 -- 172.21.15.16:0/3163908898 >> 172.21.15.16:0/3163908898 conn(0x7f04880655c0 msgr2=0x7f048815fb40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:40:03.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:03.106+0000 7f048f503640 1 -- 172.21.15.16:0/3163908898 shutdown_connections 2023-05-22T15:40:03.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:03.106+0000 7f048f503640 1 -- 172.21.15.16:0/3163908898 wait complete. 2023-05-22T15:40:03.129 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-05-22T15:40:03.258 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:03.254+0000 7f585d4ba640 1 Processor -- start 2023-05-22T15:40:03.258 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:03.254+0000 7f585d4ba640 1 -- start start 2023-05-22T15:40:03.259 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:03.254+0000 7f585d4ba640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f58580b22e0 0x7f585806bec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:40:03.259 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:03.254+0000 7f585d4ba640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f585806c400 0x7f58580cf8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:40:03.260 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:03.254+0000 7f585d4ba640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f585806db40 con 0x7f58580cbf80 2023-05-22T15:40:03.260 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:03.254+0000 7f585d4ba640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f585806c880 con 0x7f58580b22e0 2023-05-22T15:40:03.260 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:03.254+0000 7f585d4ba640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58580cf3f0 con 0x7f585806c400 2023-05-22T15:40:03.260 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:03.254+0000 7f584e7fc640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f58580b22e0 0x7f585806bec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:40:03.260 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:03.254+0000 7f584e7fc640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f58580b22e0 0x7f585806bec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34292/0 (socket says 172.21.15.16:34292) 2023-05-22T15:40:03.260 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:03.254+0000 7f584e7fc640 1 -- 172.21.15.16:0/1256065743 learned_addr learned my addr 172.21.15.16:0/1256065743 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:40:03.261 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:03.254+0000 7f58577fe640 1 --2- 172.21.15.16:0/1256065743 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f585806c400 0x7f58580cf8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:40:03.261 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:03.254+0000 7f584e7fc640 1 -- 172.21.15.16:0/1256065743 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f585806c400 msgr2=0x7f58580cf8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:40:03.261 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:03.254+0000 7f584e7fc640 1 --2- 172.21.15.16:0/1256065743 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f585806c400 0x7f58580cf8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:03.261 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:03.254+0000 7f584e7fc640 1 -- 172.21.15.16:0/1256065743 >> v1:172.21.15.62:6789/0 conn(0x7f58580cbf80 legacy=0x7f585806b210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:40:03.261 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:03.254+0000 7f584e7fc640 1 -- 172.21.15.16:0/1256065743 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f585815c080 con 0x7f58580b22e0 2023-05-22T15:40:03.262 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:03.258+0000 7f584e7fc640 1 --2- 172.21.15.16:0/1256065743 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f58580b22e0 0x7f585806bec0 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7f5840009870 tx=0x7f584001c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f52157fcebb7b40 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:40:03.262 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:03.258+0000 7f58567fc640 1 -- 172.21.15.16:0/1256065743 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58400ab680 con 0x7f58580b22e0 2023-05-22T15:40:03.262 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:03.258+0000 7f58567fc640 1 -- 172.21.15.16:0/1256065743 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58400ab7e0 con 0x7f58580b22e0 2023-05-22T15:40:03.263 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:03.258+0000 7f58567fc640 1 -- 172.21.15.16:0/1256065743 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58400abb70 con 0x7f58580b22e0 2023-05-22T15:40:03.263 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:03.258+0000 7f585d4ba640 1 -- 172.21.15.16:0/1256065743 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f58580b22e0 msgr2=0x7f585806bec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:40:03.263 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:03.258+0000 7f585d4ba640 1 --2- 172.21.15.16:0/1256065743 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f58580b22e0 0x7f585806bec0 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7f5840009870 tx=0x7f584001c480 comp rx=0 tx=0).stop 2023-05-22T15:40:03.264 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:03.258+0000 7f585d4ba640 1 -- 172.21.15.16:0/1256065743 shutdown_connections 2023-05-22T15:40:03.264 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:03.258+0000 7f585d4ba640 1 --2- 172.21.15.16:0/1256065743 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f585806c400 0x7f58580cf8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:03.264 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:03.258+0000 7f585d4ba640 1 --2- 172.21.15.16:0/1256065743 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f58580b22e0 0x7f585806bec0 unknown :-1 s=CLOSED pgs=909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:03.264 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:03.258+0000 7f585d4ba640 1 -- 172.21.15.16:0/1256065743 >> 172.21.15.16:0/1256065743 conn(0x7f585806f560 msgr2=0x7f585806f960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:40:03.264 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:03.258+0000 7f585d4ba640 1 -- 172.21.15.16:0/1256065743 shutdown_connections 2023-05-22T15:40:03.264 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:03.258+0000 7f585d4ba640 1 -- 172.21.15.16:0/1256065743 wait complete. 2023-05-22T15:40:03.266 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:03.258+0000 7f585d4ba640 1 Processor -- start 2023-05-22T15:40:03.266 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:03.258+0000 7f585d4ba640 1 -- start start 2023-05-22T15:40:03.266 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:03.258+0000 7f585d4ba640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f585806c400 0x7f5858159150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:40:03.266 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:03.258+0000 7f585d4ba640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f58580b22e0 0x7f5858156be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:40:03.266 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:03.258+0000 7f585d4ba640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f58580cbf80 0x7f58580b5530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:40:03.266 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:03.258+0000 7f585d4ba640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5858159690 con 0x7f585806c400 2023-05-22T15:40:03.267 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:03.258+0000 7f585d4ba640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58580b4dc0 con 0x7f58580cbf80 2023-05-22T15:40:03.267 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:03.258+0000 7f585d4ba640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58581598c0 con 0x7f58580b22e0 2023-05-22T15:40:03.267 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:03.258+0000 7f5856ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f585806c400 0x7f5858159150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:40:03.267 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:03.258+0000 7f5856ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f585806c400 0x7f5858159150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34308/0 (socket says 172.21.15.16:34308) 2023-05-22T15:40:03.267 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:03.258+0000 7f5856ffd640 1 -- 172.21.15.16:0/4040717215 learned_addr learned my addr 172.21.15.16:0/4040717215 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:40:03.267 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:03.258+0000 7f58577fe640 1 --2- 172.21.15.16:0/4040717215 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f58580cbf80 0x7f58580b5530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:40:03.268 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:03.258+0000 7f584e7fc640 1 --2- 172.21.15.16:0/4040717215 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f58580b22e0 0x7f5858156be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:40:03.268 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:03.258+0000 7f5856ffd640 1 -- 172.21.15.16:0/4040717215 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f58580b22e0 msgr2=0x7f5858156be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:40:03.268 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:03.258+0000 7f5856ffd640 1 --2- 172.21.15.16:0/4040717215 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f58580b22e0 0x7f5858156be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:03.268 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:03.258+0000 7f5856ffd640 1 -- 172.21.15.16:0/4040717215 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f58580cbf80 msgr2=0x7f58580b5530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:40:03.268 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:03.258+0000 7f5856ffd640 1 --2- 172.21.15.16:0/4040717215 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f58580cbf80 0x7f58580b5530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:03.268 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:03.258+0000 7f5856ffd640 1 -- 172.21.15.16:0/4040717215 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58400094d0 con 0x7f585806c400 2023-05-22T15:40:03.268 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:03.258+0000 7f5856ffd640 1 --2- 172.21.15.16:0/4040717215 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f585806c400 0x7f5858159150 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7f584800c7a0 tx=0x7f584800cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:40:03.269 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:03.258+0000 7f584ffff640 1 -- 172.21.15.16:0/4040717215 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5848003b80 con 0x7f585806c400 2023-05-22T15:40:03.269 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:03.258+0000 7f584ffff640 1 -- 172.21.15.16:0/4040717215 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5848003ce0 con 0x7f585806c400 2023-05-22T15:40:03.269 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:03.258+0000 7f585d4ba640 1 -- 172.21.15.16:0/4040717215 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58580cc350 con 0x7f585806c400 2023-05-22T15:40:03.269 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:03.258+0000 7f584ffff640 1 -- 172.21.15.16:0/4040717215 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f584800aa90 con 0x7f585806c400 2023-05-22T15:40:03.269 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:03.258+0000 7f585d4ba640 1 -- 172.21.15.16:0/4040717215 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f585806e380 con 0x7f585806c400 2023-05-22T15:40:03.270 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:03.262+0000 7f585d4ba640 1 -- 172.21.15.16:0/4040717215 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58580d0620 con 0x7f585806c400 2023-05-22T15:40:03.270 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:03.262+0000 7f584ffff640 1 -- 172.21.15.16:0/4040717215 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f584800abf0 con 0x7f585806c400 2023-05-22T15:40:03.270 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:03.262+0000 7f584ffff640 1 --2- 172.21.15.16:0/4040717215 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f58380654a0 0x7f5838067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:40:03.270 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:03.262+0000 7f584ffff640 1 -- 172.21.15.16:0/4040717215 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f5848082950 con 0x7f585806c400 2023-05-22T15:40:03.271 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:03.262+0000 7f584e7fc640 1 --2- 172.21.15.16:0/4040717215 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f58380654a0 0x7f5838067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:40:03.271 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:03.262+0000 7f584e7fc640 1 --2- 172.21.15.16:0/4040717215 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f58380654a0 0x7f5838067950 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f58400af000 tx=0x7f58400a7000 comp rx=0 tx=0).ready entity=mgr.6240 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:40:03.271 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:03.266+0000 7f584ffff640 1 -- 172.21.15.16:0/4040717215 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f5848050380 con 0x7f585806c400 2023-05-22T15:40:03.426 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:03.418+0000 7f585d4ba640 1 -- 172.21.15.16:0/4040717215 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f585804dbe0 con 0x7f585806c400 2023-05-22T15:40:03.428 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:03.422+0000 7f584ffff640 1 -- 172.21.15.16:0/4040717215 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v72) v1 ==== 74+0+18049 (secure 0 0 0) 0x7f58480539a0 con 0x7f585806c400 2023-05-22T15:40:03.429 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:40:03.430 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":72,"fsid":"b2719834-7847-4d98-90a6-f65d57f85c49","created":"2023-05-22T15:26:25.259314+0000","modified":"2023-05-22T15:40:02.753701+0000","last_up_change":"2023-05-22T15:26:36.448524+0000","last_in_change":"2023-05-22T15:26:27.675668+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-05-22T15:26:38.019781+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-05-22T15:26:43.570514+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2023-05-22T15:40:00.663040+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"72","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2023-05-22T15:40:01.750680+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"72","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9e7ff854-b25f-4cbb-a5c2-1799d9c53d2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":37021},{"type":"v1","addr":"172.21.15.16:6825","nonce":37021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":37021},{"type":"v1","addr":"172.21.15.16:6827","nonce":37021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":37021},{"type":"v1","addr":"172.21.15.16:6831","nonce":37021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":37021},{"type":"v1","addr":"172.21.15.16:6829","nonce":37021}]},"public_addr":"172.21.15.16:6825/37021","cluster_addr":"172.21.15.16:6827/37021","heartbeat_back_addr":"172.21.15.16:6831/37021","heartbeat_front_addr":"172.21.15.16:6829/37021","state":["exists","up"]},{"osd":1,"uuid":"4702fbc9-a927-4565-aca5-41dc69cc8264","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":37008},{"type":"v1","addr":"172.21.15.16:6801","nonce":37008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":37008},{"type":"v1","addr":"172.21.15.16:6803","nonce":37008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":37008},{"type":"v1","addr":"172.21.15.16:6807","nonce":37008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":37008},{"type":"v1","addr":"172.21.15.16:6805","nonce":37008}]},"public_addr":"172.21.15.16:6801/37008","cluster_addr":"172.21.15.16:6803/37008","heartbeat_back_addr":"172.21.15.16:6807/37008","heartbeat_front_addr":"172.21.15.16:6805/37008","state":["exists","up"]},{"osd":2,"uuid":"1e7bba46-4eba-47ef-bb81-5e2d371638ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":37009},{"type":"v1","addr":"172.21.15.16:6809","nonce":37009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":37009},{"type":"v1","addr":"172.21.15.16:6811","nonce":37009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":37009},{"type":"v1","addr":"172.21.15.16:6815","nonce":37009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":37009},{"type":"v1","addr":"172.21.15.16:6813","nonce":37009}]},"public_addr":"172.21.15.16:6809/37009","cluster_addr":"172.21.15.16:6811/37009","heartbeat_back_addr":"172.21.15.16:6815/37009","heartbeat_front_addr":"172.21.15.16:6813/37009","state":["exists","up"]},{"osd":3,"uuid":"383cacc0-5a88-4904-990e-498ca59ef48a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":37020},{"type":"v1","addr":"172.21.15.16:6817","nonce":37020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":37020},{"type":"v1","addr":"172.21.15.16:6819","nonce":37020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":37020},{"type":"v1","addr":"172.21.15.16:6823","nonce":37020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":37020},{"type":"v1","addr":"172.21.15.16:6821","nonce":37020}]},"public_addr":"172.21.15.16:6817/37020","cluster_addr":"172.21.15.16:6819/37020","heartbeat_back_addr":"172.21.15.16:6823/37020","heartbeat_front_addr":"172.21.15.16:6821/37020","state":["exists","up"]},{"osd":4,"uuid":"970500ef-95b7-42c1-b080-eed4fae7013d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":37353},{"type":"v1","addr":"172.21.15.62:6817","nonce":37353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6819","nonce":37353},{"type":"v1","addr":"172.21.15.62:6822","nonce":37353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37353},{"type":"v1","addr":"172.21.15.62:6831","nonce":37353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37353},{"type":"v1","addr":"172.21.15.62:6826","nonce":37353}]},"public_addr":"172.21.15.62:6817/37353","cluster_addr":"172.21.15.62:6822/37353","heartbeat_back_addr":"172.21.15.62:6831/37353","heartbeat_front_addr":"172.21.15.62:6826/37353","state":["exists","up"]},{"osd":5,"uuid":"022d67db-d4d8-445b-b1b9-8b446a00f5c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":37352},{"type":"v1","addr":"172.21.15.62:6801","nonce":37352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":37352},{"type":"v1","addr":"172.21.15.62:6803","nonce":37352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":37352},{"type":"v1","addr":"172.21.15.62:6807","nonce":37352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":37352},{"type":"v1","addr":"172.21.15.62:6805","nonce":37352}]},"public_addr":"172.21.15.62:6801/37352","cluster_addr":"172.21.15.62:6803/37352","heartbeat_back_addr":"172.21.15.62:6807/37352","heartbeat_front_addr":"172.21.15.62:6805/37352","state":["exists","up"]},{"osd":6,"uuid":"42bd372b-6688-4a1b-8407-e16e8c266e33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":37354},{"type":"v1","addr":"172.21.15.62:6820","nonce":37354}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":37354},{"type":"v1","addr":"172.21.15.62:6823","nonce":37354}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6829","nonce":37354},{"type":"v1","addr":"172.21.15.62:6830","nonce":37354}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6825","nonce":37354},{"type":"v1","addr":"172.21.15.62:6827","nonce":37354}]},"public_addr":"172.21.15.62:6820/37354","cluster_addr":"172.21.15.62:6823/37354","heartbeat_back_addr":"172.21.15.62:6830/37354","heartbeat_front_addr":"172.21.15.62:6827/37354","state":["exists","up"]},{"osd":7,"uuid":"ac70b2a3-f41b-4bf2-b227-7f2cbb55426d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37355},{"type":"v1","addr":"172.21.15.62:6809","nonce":37355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37355},{"type":"v1","addr":"172.21.15.62:6811","nonce":37355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":37355},{"type":"v1","addr":"172.21.15.62:6815","nonce":37355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":37355},{"type":"v1","addr":"172.21.15.62:6813","nonce":37355}]},"public_addr":"172.21.15.62:6809/37355","cluster_addr":"172.21.15.62:6811/37355","heartbeat_back_addr":"172.21.15.62:6815/37355","heartbeat_front_addr":"172.21.15.62:6813/37355","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.368896+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.107273+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.276888+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.287502+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.124791+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.251937+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.255679+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.232797+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-05-22T15:40:03.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:03.422+0000 7f585d4ba640 1 -- 172.21.15.16:0/4040717215 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f58380654a0 msgr2=0x7f5838067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:40:03.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:03.422+0000 7f585d4ba640 1 --2- 172.21.15.16:0/4040717215 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f58380654a0 0x7f5838067950 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f58400af000 tx=0x7f58400a7000 comp rx=0 tx=0).stop 2023-05-22T15:40:03.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:03.422+0000 7f585d4ba640 1 -- 172.21.15.16:0/4040717215 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f585806c400 msgr2=0x7f5858159150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:40:03.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:03.422+0000 7f585d4ba640 1 --2- 172.21.15.16:0/4040717215 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f585806c400 0x7f5858159150 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7f584800c7a0 tx=0x7f584800cc60 comp rx=0 tx=0).stop 2023-05-22T15:40:03.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:03.426+0000 7f585d4ba640 1 -- 172.21.15.16:0/4040717215 shutdown_connections 2023-05-22T15:40:03.432 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:03.426+0000 7f585d4ba640 1 --2- 172.21.15.16:0/4040717215 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f58580cbf80 0x7f58580b5530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:03.432 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:03.426+0000 7f585d4ba640 1 --2- 172.21.15.16:0/4040717215 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f58380654a0 0x7f5838067950 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:03.432 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:03.426+0000 7f585d4ba640 1 --2- 172.21.15.16:0/4040717215 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f58580b22e0 0x7f5858156be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:03.432 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:03.426+0000 7f585d4ba640 1 --2- 172.21.15.16:0/4040717215 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f585806c400 0x7f5858159150 unknown :-1 s=CLOSED pgs=910 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:03.432 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:03.426+0000 7f585d4ba640 1 -- 172.21.15.16:0/4040717215 >> 172.21.15.16:0/4040717215 conn(0x7f585806f560 msgr2=0x7f585806f930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:40:03.432 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:03.426+0000 7f585d4ba640 1 -- 172.21.15.16:0/4040717215 shutdown_connections 2023-05-22T15:40:03.432 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:03.426+0000 7f585d4ba640 1 -- 172.21.15.16:0/4040717215 wait complete. 2023-05-22T15:40:03.447 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-05-22T15:40:03.571 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:03.566+0000 7f21f1ea4640 1 Processor -- start 2023-05-22T15:40:03.571 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:03.566+0000 7f21f1ea4640 1 -- start start 2023-05-22T15:40:03.572 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:03.566+0000 7f21f1ea4640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f21ec0d2480 0x7f21ec0d2900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:40:03.572 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:03.566+0000 7f21f1ea4640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f21ec0b5340 0x7f21ec0b5790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:40:03.572 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:03.566+0000 7f21f1ea4640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21ec0cccc0 con 0x7f21ec15bca0 2023-05-22T15:40:03.572 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:03.566+0000 7f21f1ea4640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21ec0cd660 con 0x7f21ec0d2480 2023-05-22T15:40:03.572 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:03.566+0000 7f21f1ea4640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21ec0ce190 con 0x7f21ec0b5340 2023-05-22T15:40:03.573 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:03.566+0000 7f21eb7fe640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f21ec15bca0 0x7f21ec0cda80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:41338/0 (socket says 172.21.15.16:41338) 2023-05-22T15:40:03.573 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:03.566+0000 7f21eb7fe640 1 -- 172.21.15.16:0/1071231190 learned_addr learned my addr 172.21.15.16:0/1071231190 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:40:03.573 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:03.566+0000 7f21eaffd640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f21ec0d2480 0x7f21ec0d2900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:40:03.573 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:03.566+0000 7f21ebfff640 1 --2- 172.21.15.16:0/1071231190 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f21ec0b5340 0x7f21ec0b5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:40:03.574 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:03.566+0000 7f21eaffd640 1 -- 172.21.15.16:0/1071231190 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f21ec0b5340 msgr2=0x7f21ec0b5790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:40:03.574 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:03.566+0000 7f21eaffd640 1 --2- 172.21.15.16:0/1071231190 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f21ec0b5340 0x7f21ec0b5790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:03.574 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:03.566+0000 7f21eaffd640 1 -- 172.21.15.16:0/1071231190 >> v1:172.21.15.16:6789/0 conn(0x7f21ec15bca0 legacy=0x7f21ec0cda80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:40:03.574 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:03.566+0000 7f21ea7fc640 1 -- 172.21.15.16:0/1071231190 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1957872256 0 0) 0x7f21ec0cccc0 con 0x7f21ec15bca0 2023-05-22T15:40:03.574 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:03.566+0000 7f21eaffd640 1 -- 172.21.15.16:0/1071231190 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21ec0ccfa0 con 0x7f21ec0d2480 2023-05-22T15:40:03.575 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:03.570+0000 7f21eaffd640 1 --2- 172.21.15.16:0/1071231190 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f21ec0d2480 0x7f21ec0d2900 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7f21dc0097f0 tx=0x7f21dc01c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=585d3dfc41ad81ba server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:40:03.575 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:03.570+0000 7f21ea7fc640 1 -- 172.21.15.16:0/1071231190 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc0ad640 con 0x7f21ec0d2480 2023-05-22T15:40:03.575 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:03.570+0000 7f21ea7fc640 1 -- 172.21.15.16:0/1071231190 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc0ad7a0 con 0x7f21ec0d2480 2023-05-22T15:40:03.575 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:03.570+0000 7f21ea7fc640 1 -- 172.21.15.16:0/1071231190 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc0adb30 con 0x7f21ec0d2480 2023-05-22T15:40:03.576 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:03.570+0000 7f21f1ea4640 1 -- 172.21.15.16:0/1071231190 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f21ec0d2480 msgr2=0x7f21ec0d2900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:40:03.576 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:03.570+0000 7f21f1ea4640 1 --2- 172.21.15.16:0/1071231190 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f21ec0d2480 0x7f21ec0d2900 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7f21dc0097f0 tx=0x7f21dc01c480 comp rx=0 tx=0).stop 2023-05-22T15:40:03.576 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:03.570+0000 7f21f1ea4640 1 -- 172.21.15.16:0/1071231190 shutdown_connections 2023-05-22T15:40:03.577 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:03.570+0000 7f21f1ea4640 1 --2- 172.21.15.16:0/1071231190 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f21ec0b5340 0x7f21ec0b5790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:03.577 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:03.570+0000 7f21f1ea4640 1 --2- 172.21.15.16:0/1071231190 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f21ec0d2480 0x7f21ec0d2900 unknown :-1 s=CLOSED pgs=912 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:03.577 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:03.570+0000 7f21f1ea4640 1 -- 172.21.15.16:0/1071231190 >> 172.21.15.16:0/1071231190 conn(0x7f21ec0655c0 msgr2=0x7f21ec070e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:40:03.577 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:03.570+0000 7f21f1ea4640 1 -- 172.21.15.16:0/1071231190 shutdown_connections 2023-05-22T15:40:03.577 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:03.570+0000 7f21f1ea4640 1 -- 172.21.15.16:0/1071231190 wait complete. 2023-05-22T15:40:03.579 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:03.570+0000 7f21f1ea4640 1 Processor -- start 2023-05-22T15:40:03.579 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:03.570+0000 7f21f1ea4640 1 -- start start 2023-05-22T15:40:03.579 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:03.570+0000 7f21f1ea4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f21ec0b5340 0x7f21ec15cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:40:03.579 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:03.570+0000 7f21f1ea4640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f21ec0d2480 0x7f21ec148ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:40:03.579 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:03.570+0000 7f21f1ea4640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f21ec15bca0 0x7f21ec148620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:40:03.579 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:03.570+0000 7f21f1ea4640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21ec15df10 con 0x7f21ec0b5340 2023-05-22T15:40:03.579 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:03.570+0000 7f21f1ea4640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21ec15d420 con 0x7f21ec15bca0 2023-05-22T15:40:03.580 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:03.570+0000 7f21f1ea4640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21ec15d700 con 0x7f21ec0d2480 2023-05-22T15:40:03.580 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:03.570+0000 7f21eb7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f21ec0b5340 0x7f21ec15cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:40:03.580 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:03.574+0000 7f21eb7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f21ec0b5340 0x7f21ec15cbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34326/0 (socket says 172.21.15.16:34326) 2023-05-22T15:40:03.580 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:03.574+0000 7f21eb7fe640 1 -- 172.21.15.16:0/4100116735 learned_addr learned my addr 172.21.15.16:0/4100116735 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:40:03.580 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:03.574+0000 7f21ebfff640 1 --2- 172.21.15.16:0/4100116735 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f21ec15bca0 0x7f21ec148620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:40:03.580 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:03.574+0000 7f21eaffd640 1 --2- 172.21.15.16:0/4100116735 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f21ec0d2480 0x7f21ec148ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:40:03.580 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:03.574+0000 7f21eb7fe640 1 -- 172.21.15.16:0/4100116735 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f21ec0d2480 msgr2=0x7f21ec148ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:40:03.581 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:03.574+0000 7f21eb7fe640 1 --2- 172.21.15.16:0/4100116735 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f21ec0d2480 0x7f21ec148ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:03.581 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:03.574+0000 7f21eb7fe640 1 -- 172.21.15.16:0/4100116735 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f21ec15bca0 msgr2=0x7f21ec148620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:40:03.581 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:03.574+0000 7f21eb7fe640 1 --2- 172.21.15.16:0/4100116735 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f21ec15bca0 0x7f21ec148620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:03.581 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:03.574+0000 7f21eb7fe640 1 -- 172.21.15.16:0/4100116735 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21dc009420 con 0x7f21ec0b5340 2023-05-22T15:40:03.581 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:03.574+0000 7f21ebfff640 1 --2- 172.21.15.16:0/4100116735 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f21ec15bca0 0x7f21ec148620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:40:03.581 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:03.574+0000 7f21eaffd640 1 --2- 172.21.15.16:0/4100116735 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f21ec0d2480 0x7f21ec148ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:40:03.581 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:03.574+0000 7f21eb7fe640 1 --2- 172.21.15.16:0/4100116735 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f21ec0b5340 0x7f21ec15cbd0 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7f21e0007fd0 tx=0x7f21e000c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:40:03.582 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:03.574+0000 7f21d3fff640 1 -- 172.21.15.16:0/4100116735 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e0005cf0 con 0x7f21ec0b5340 2023-05-22T15:40:03.582 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:03.574+0000 7f21d3fff640 1 -- 172.21.15.16:0/4100116735 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e0005e70 con 0x7f21ec0b5340 2023-05-22T15:40:03.582 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:03.574+0000 7f21d3fff640 1 -- 172.21.15.16:0/4100116735 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e00072f0 con 0x7f21ec0b5340 2023-05-22T15:40:03.582 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:03.574+0000 7f21f1ea4640 1 -- 172.21.15.16:0/4100116735 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21ec160360 con 0x7f21ec0b5340 2023-05-22T15:40:03.582 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:03.574+0000 7f21f1ea4640 1 -- 172.21.15.16:0/4100116735 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21ec0b5cd0 con 0x7f21ec0b5340 2023-05-22T15:40:03.583 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:03.574+0000 7f21f1ea4640 1 -- 172.21.15.16:0/4100116735 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21ec1604f0 con 0x7f21ec0b5340 2023-05-22T15:40:03.583 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:03.574+0000 7f21d3fff640 1 -- 172.21.15.16:0/4100116735 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f21e0007450 con 0x7f21ec0b5340 2023-05-22T15:40:03.583 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:03.574+0000 7f21d3fff640 1 --2- 172.21.15.16:0/4100116735 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f21d80654a0 0x7f21d8067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:40:03.583 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:03.574+0000 7f21eaffd640 1 --2- 172.21.15.16:0/4100116735 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f21d80654a0 0x7f21d8067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:40:03.583 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:03.574+0000 7f21d3fff640 1 -- 172.21.15.16:0/4100116735 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f21e0082510 con 0x7f21ec0b5340 2023-05-22T15:40:03.584 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:03.574+0000 7f21eaffd640 1 --2- 172.21.15.16:0/4100116735 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f21d80654a0 0x7f21d8067950 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f21dc0af000 tx=0x7f21dc0a7070 comp rx=0 tx=0).ready entity=mgr.6240 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:40:03.584 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:03.578+0000 7f21d3fff640 1 -- 172.21.15.16:0/4100116735 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f21e0050020 con 0x7f21ec0b5340 2023-05-22T15:40:03.773 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:03.766+0000 7f21f1ea4640 1 -- 172.21.15.16:0/4100116735 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f21ec04dbe0 con 0x7f21ec0b5340 2023-05-22T15:40:04.778 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:04.770+0000 7f21d3fff640 1 -- 172.21.15.16:0/4100116735 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v21) v1 ==== 118+0+0 (secure 0 0 0) 0x7f21e0053640 con 0x7f21ec0b5340 2023-05-22T15:40:04.781 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:04.774+0000 7f21f1ea4640 1 -- 172.21.15.16:0/4100116735 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f21d80654a0 msgr2=0x7f21d8067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:40:04.781 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:04.774+0000 7f21f1ea4640 1 --2- 172.21.15.16:0/4100116735 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f21d80654a0 0x7f21d8067950 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f21dc0af000 tx=0x7f21dc0a7070 comp rx=0 tx=0).stop 2023-05-22T15:40:04.781 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:04.774+0000 7f21f1ea4640 1 -- 172.21.15.16:0/4100116735 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f21ec0b5340 msgr2=0x7f21ec15cbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:40:04.781 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:04.774+0000 7f21f1ea4640 1 --2- 172.21.15.16:0/4100116735 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f21ec0b5340 0x7f21ec15cbd0 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7f21e0007fd0 tx=0x7f21e000c8e0 comp rx=0 tx=0).stop 2023-05-22T15:40:04.781 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:04.774+0000 7f21f1ea4640 1 -- 172.21.15.16:0/4100116735 shutdown_connections 2023-05-22T15:40:04.781 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:04.774+0000 7f21f1ea4640 1 --2- 172.21.15.16:0/4100116735 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f21ec15bca0 0x7f21ec148620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:04.782 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:04.774+0000 7f21f1ea4640 1 --2- 172.21.15.16:0/4100116735 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f21d80654a0 0x7f21d8067950 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:04.782 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:04.774+0000 7f21f1ea4640 1 --2- 172.21.15.16:0/4100116735 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f21ec0d2480 0x7f21ec148ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:04.782 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:04.774+0000 7f21f1ea4640 1 --2- 172.21.15.16:0/4100116735 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f21ec0b5340 0x7f21ec15cbd0 unknown :-1 s=CLOSED pgs=913 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:04.782 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:04.774+0000 7f21f1ea4640 1 -- 172.21.15.16:0/4100116735 >> 172.21.15.16:0/4100116735 conn(0x7f21ec0655c0 msgr2=0x7f21ec154210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:40:04.782 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:04.774+0000 7f21f1ea4640 1 -- 172.21.15.16:0/4100116735 shutdown_connections 2023-05-22T15:40:04.782 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:04.774+0000 7f21f1ea4640 1 -- 172.21.15.16:0/4100116735 wait complete. 2023-05-22T15:40:04.804 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2023-05-22T15:40:04.805 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-05-22T15:40:04.935 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:04.930+0000 7f3e44005640 1 Processor -- start 2023-05-22T15:40:04.935 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:04.930+0000 7f3e44005640 1 -- start start 2023-05-22T15:40:04.937 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:04.930+0000 7f3e44005640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3e3c158a10 0x7f3e3c06e480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:40:04.937 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:04.930+0000 7f3e44005640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e3c158de0 con 0x7f3e3c070760 2023-05-22T15:40:04.938 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:04.930+0000 7f3e44005640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e3c0711e0 con 0x7f3e3c06cdf0 2023-05-22T15:40:04.938 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:04.930+0000 7f3e44005640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e3c06e060 con 0x7f3e3c158a10 2023-05-22T15:40:04.938 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:04.930+0000 7f3e415a0640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f3e3c070760 0x7f3e3c06c6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:41342/0 (socket says 172.21.15.16:41342) 2023-05-22T15:40:04.938 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:04.930+0000 7f3e415a0640 1 -- 172.21.15.16:0/237852194 learned_addr learned my addr 172.21.15.16:0/237852194 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:40:04.938 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:04.930+0000 7f3e41da1640 1 --2- 172.21.15.16:0/237852194 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3e3c158a10 0x7f3e3c06e480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:40:04.938 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:04.930+0000 7f3e40d9f640 1 -- 172.21.15.16:0/237852194 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4198249731 0 0) 0x7f3e3c158de0 con 0x7f3e3c070760 2023-05-22T15:40:04.938 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:04.930+0000 7f3e40d9f640 1 -- 172.21.15.16:0/237852194 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e2c003530 con 0x7f3e3c070760 2023-05-22T15:40:04.938 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:04.930+0000 7f3e40d9f640 1 -- 172.21.15.16:0/237852194 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 570890823 0 0) 0x7f3e2c003530 con 0x7f3e3c070760 2023-05-22T15:40:04.939 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:04.930+0000 7f3e40d9f640 1 -- 172.21.15.16:0/237852194 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3e3c158a10 msgr2=0x7f3e3c06e480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:40:04.939 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:04.930+0000 7f3e40d9f640 1 --2- 172.21.15.16:0/237852194 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3e3c158a10 0x7f3e3c06e480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:04.939 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:04.930+0000 7f3e40d9f640 1 -- 172.21.15.16:0/237852194 >> v1:172.21.15.62:6789/0 conn(0x7f3e3c06cdf0 legacy=0x7f3e3c0c9a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:40:04.939 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:04.930+0000 7f3e40d9f640 1 -- 172.21.15.16:0/237852194 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e3c0714c0 con 0x7f3e3c070760 2023-05-22T15:40:04.939 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:04.930+0000 7f3e40d9f640 1 -- 172.21.15.16:0/237852194 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7f3e30001a00 con 0x7f3e3c070760 2023-05-22T15:40:04.939 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:04.930+0000 7f3e40d9f640 1 -- 172.21.15.16:0/237852194 >> v1:172.21.15.16:6789/0 conn(0x7f3e3c070760 legacy=0x7f3e3c06c6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:40:04.939 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:04.930+0000 7f3e40d9f640 1 --2- 172.21.15.16:0/237852194 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3e2c0039b0 0x7f3e2c003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:40:04.940 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:04.930+0000 7f3e40d9f640 1 --2- 172.21.15.16:0/237852194 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3e3c158a10 0x7f3e2c0047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:40:04.940 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:04.930+0000 7f3e40d9f640 1 --2- 172.21.15.16:0/237852194 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3e2c004d10 0x7f3e2c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:40:04.940 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:04.930+0000 7f3e40d9f640 1 -- 172.21.15.16:0/237852194 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e3c06e060 con 0x7f3e2c004d10 2023-05-22T15:40:04.940 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:04.930+0000 7f3e40d9f640 1 -- 172.21.15.16:0/237852194 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e2c00baa0 con 0x7f3e2c0039b0 2023-05-22T15:40:04.940 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:04.930+0000 7f3e40d9f640 1 -- 172.21.15.16:0/237852194 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e2c00bc60 con 0x7f3e3c158a10 2023-05-22T15:40:04.940 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:04.930+0000 7f3e425a2640 1 --2- 172.21.15.16:0/237852194 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3e2c004d10 0x7f3e2c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:40:04.940 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:04.930+0000 7f3e41da1640 1 --2- 172.21.15.16:0/237852194 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3e2c0039b0 0x7f3e2c003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:40:04.940 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:04.930+0000 7f3e415a0640 1 --2- 172.21.15.16:0/237852194 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3e3c158a10 0x7f3e2c0047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:40:04.941 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:04.930+0000 7f3e425a2640 1 -- 172.21.15.16:0/237852194 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3e3c158a10 msgr2=0x7f3e2c0047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:40:04.941 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:04.930+0000 7f3e425a2640 1 --2- 172.21.15.16:0/237852194 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3e3c158a10 0x7f3e2c0047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:04.941 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:04.930+0000 7f3e425a2640 1 -- 172.21.15.16:0/237852194 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3e2c0039b0 msgr2=0x7f3e2c003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:40:04.941 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:04.930+0000 7f3e425a2640 1 --2- 172.21.15.16:0/237852194 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3e2c0039b0 0x7f3e2c003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:04.941 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:04.930+0000 7f3e425a2640 1 -- 172.21.15.16:0/237852194 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e2c003530 con 0x7f3e2c004d10 2023-05-22T15:40:04.942 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:04.934+0000 7f3e425a2640 1 --2- 172.21.15.16:0/237852194 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3e2c004d10 0x7f3e2c005180 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7f3e34001750 tx=0x7f3e3401c3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6af9806c98d85ad6 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:40:04.942 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:04.934+0000 7f3e40d9f640 1 -- 172.21.15.16:0/237852194 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e340052d0 con 0x7f3e2c004d10 2023-05-22T15:40:04.942 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:04.934+0000 7f3e40d9f640 1 -- 172.21.15.16:0/237852194 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e340a7050 con 0x7f3e2c004d10 2023-05-22T15:40:04.943 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:04.934+0000 7f3e40d9f640 1 -- 172.21.15.16:0/237852194 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e34003600 con 0x7f3e2c004d10 2023-05-22T15:40:04.943 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:04.934+0000 7f3e44005640 1 -- 172.21.15.16:0/237852194 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3e2c004d10 msgr2=0x7f3e2c005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:40:04.943 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:04.934+0000 7f3e44005640 1 --2- 172.21.15.16:0/237852194 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3e2c004d10 0x7f3e2c005180 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7f3e34001750 tx=0x7f3e3401c3c0 comp rx=0 tx=0).stop 2023-05-22T15:40:04.943 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:04.934+0000 7f3e425a2640 1 -- 172.21.15.16:0/237852194 reap_dead start 2023-05-22T15:40:04.943 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:04.934+0000 7f3e44005640 1 -- 172.21.15.16:0/237852194 shutdown_connections 2023-05-22T15:40:04.943 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:04.934+0000 7f3e44005640 1 -- 172.21.15.16:0/237852194 >> 172.21.15.16:0/237852194 conn(0x7f3e3c0cc870 msgr2=0x7f3e3c0cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:40:04.943 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:04.934+0000 7f3e44005640 1 -- 172.21.15.16:0/237852194 shutdown_connections 2023-05-22T15:40:04.944 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:04.934+0000 7f3e44005640 1 -- 172.21.15.16:0/237852194 wait complete. 2023-05-22T15:40:04.944 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:04.938+0000 7f3e44005640 1 Processor -- start 2023-05-22T15:40:04.944 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:04.938+0000 7f3e44005640 1 -- start start 2023-05-22T15:40:04.945 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:04.938+0000 7f3e44005640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3e3c1eeba0 0x7f3e3c155f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:40:04.945 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:04.938+0000 7f3e44005640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3e3c0b18b0 0x7f3e3c0affe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:40:04.945 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:04.938+0000 7f3e44005640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3e3c153de0 0x7f3e3c154280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:40:04.946 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:04.938+0000 7f3e44005640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e3c1547c0 con 0x7f3e3c1eeba0 2023-05-22T15:40:04.946 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:04.938+0000 7f3e44005640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e3c0b0810 con 0x7f3e3c153de0 2023-05-22T15:40:04.946 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:04.938+0000 7f3e44005640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e3c0b0af0 con 0x7f3e3c0b18b0 2023-05-22T15:40:04.946 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:04.938+0000 7f3e41da1640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3e3c1eeba0 0x7f3e3c155f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:40:04.946 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:04.938+0000 7f3e41da1640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3e3c1eeba0 0x7f3e3c155f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34342/0 (socket says 172.21.15.16:34342) 2023-05-22T15:40:04.946 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:04.938+0000 7f3e41da1640 1 -- 172.21.15.16:0/1941623587 learned_addr learned my addr 172.21.15.16:0/1941623587 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:40:04.946 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:04.938+0000 7f3e415a0640 1 --2- 172.21.15.16:0/1941623587 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3e3c0b18b0 0x7f3e3c0affe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:40:04.947 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:04.938+0000 7f3e425a2640 1 --2- 172.21.15.16:0/1941623587 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3e3c153de0 0x7f3e3c154280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:40:04.947 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:04.938+0000 7f3e41da1640 1 -- 172.21.15.16:0/1941623587 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3e3c0b18b0 msgr2=0x7f3e3c0affe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:40:04.947 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:04.938+0000 7f3e41da1640 1 --2- 172.21.15.16:0/1941623587 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3e3c0b18b0 0x7f3e3c0affe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:04.947 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:04.938+0000 7f3e41da1640 1 -- 172.21.15.16:0/1941623587 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3e3c153de0 msgr2=0x7f3e3c154280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:40:04.947 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:04.938+0000 7f3e41da1640 1 --2- 172.21.15.16:0/1941623587 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3e3c153de0 0x7f3e3c154280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:04.947 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:04.938+0000 7f3e41da1640 1 -- 172.21.15.16:0/1941623587 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e3c0cfe40 con 0x7f3e3c1eeba0 2023-05-22T15:40:04.947 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:04.938+0000 7f3e41da1640 1 --2- 172.21.15.16:0/1941623587 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3e3c1eeba0 0x7f3e3c155f20 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7f3e280115e0 tx=0x7f3e28011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:40:04.947 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:04.938+0000 7f3e3a7fc640 1 -- 172.21.15.16:0/1941623587 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e2800f8f0 con 0x7f3e3c1eeba0 2023-05-22T15:40:04.948 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:04.938+0000 7f3e3a7fc640 1 -- 172.21.15.16:0/1941623587 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e2800fa50 con 0x7f3e3c1eeba0 2023-05-22T15:40:04.948 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:04.938+0000 7f3e3a7fc640 1 -- 172.21.15.16:0/1941623587 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e2800fdc0 con 0x7f3e3c1eeba0 2023-05-22T15:40:04.948 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:04.938+0000 7f3e44005640 1 -- 172.21.15.16:0/1941623587 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e3c06dc80 con 0x7f3e3c1eeba0 2023-05-22T15:40:04.948 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:04.938+0000 7f3e44005640 1 -- 172.21.15.16:0/1941623587 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e3c070bb0 con 0x7f3e3c1eeba0 2023-05-22T15:40:04.948 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:04.938+0000 7f3e44005640 1 -- 172.21.15.16:0/1941623587 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e3c04dbe0 con 0x7f3e3c1eeba0 2023-05-22T15:40:04.949 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:04.938+0000 7f3e3a7fc640 1 -- 172.21.15.16:0/1941623587 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f3e28015460 con 0x7f3e3c1eeba0 2023-05-22T15:40:04.949 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:04.942+0000 7f3e3a7fc640 1 --2- 172.21.15.16:0/1941623587 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f3e200654d0 0x7f3e20067980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:40:04.949 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:04.942+0000 7f3e3a7fc640 1 -- 172.21.15.16:0/1941623587 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f3e28083010 con 0x7f3e3c1eeba0 2023-05-22T15:40:04.949 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:04.942+0000 7f3e415a0640 1 --2- 172.21.15.16:0/1941623587 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f3e200654d0 0x7f3e20067980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:40:04.949 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:04.942+0000 7f3e415a0640 1 --2- 172.21.15.16:0/1941623587 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f3e200654d0 0x7f3e20067980 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f3e300013e0 tx=0x7f3e30009040 comp rx=0 tx=0).ready entity=mgr.6240 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:40:04.950 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:04.946+0000 7f3e3a7fc640 1 -- 172.21.15.16:0/1941623587 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f3e280509e0 con 0x7f3e3c1eeba0 2023-05-22T15:40:05.138 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:05.134+0000 7f3e44005640 1 -- 172.21.15.16:0/1941623587 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f3e3c0b50e0 con 0x7f3e3c1eeba0 2023-05-22T15:40:05.789 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:05.782+0000 7f3e3a7fc640 1 -- 172.21.15.16:0/1941623587 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v22) v1 ==== 122+0+0 (secure 0 0 0) 0x7f3e28054000 con 0x7f3e3c1eeba0 2023-05-22T15:40:05.792 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:05.786+0000 7f3e44005640 1 -- 172.21.15.16:0/1941623587 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f3e200654d0 msgr2=0x7f3e20067980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:40:05.793 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:05.786+0000 7f3e44005640 1 --2- 172.21.15.16:0/1941623587 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f3e200654d0 0x7f3e20067980 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f3e300013e0 tx=0x7f3e30009040 comp rx=0 tx=0).stop 2023-05-22T15:40:05.793 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:05.786+0000 7f3e44005640 1 -- 172.21.15.16:0/1941623587 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3e3c1eeba0 msgr2=0x7f3e3c155f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:40:05.793 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:05.786+0000 7f3e44005640 1 --2- 172.21.15.16:0/1941623587 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3e3c1eeba0 0x7f3e3c155f20 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7f3e280115e0 tx=0x7f3e28011aa0 comp rx=0 tx=0).stop 2023-05-22T15:40:05.794 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:05.786+0000 7f3e44005640 1 -- 172.21.15.16:0/1941623587 shutdown_connections 2023-05-22T15:40:05.794 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:05.786+0000 7f3e44005640 1 --2- 172.21.15.16:0/1941623587 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3e3c153de0 0x7f3e3c154280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:05.794 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:05.786+0000 7f3e44005640 1 --2- 172.21.15.16:0/1941623587 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f3e200654d0 0x7f3e20067980 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:05.794 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:05.786+0000 7f3e44005640 1 --2- 172.21.15.16:0/1941623587 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3e3c0b18b0 0x7f3e3c0affe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:05.794 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:05.786+0000 7f3e44005640 1 --2- 172.21.15.16:0/1941623587 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3e3c1eeba0 0x7f3e3c155f20 unknown :-1 s=CLOSED pgs=916 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:05.795 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:05.786+0000 7f3e44005640 1 -- 172.21.15.16:0/1941623587 >> 172.21.15.16:0/1941623587 conn(0x7f3e3c0cc870 msgr2=0x7f3e3c06e2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:40:05.795 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:05.786+0000 7f3e44005640 1 -- 172.21.15.16:0/1941623587 shutdown_connections 2023-05-22T15:40:05.795 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:05.786+0000 7f3e44005640 1 -- 172.21.15.16:0/1941623587 wait complete. 2023-05-22T15:40:05.814 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2023-05-22T15:40:05.815 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-05-22T15:40:05.946 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:05.942+0000 7ff33a342640 1 Processor -- start 2023-05-22T15:40:05.946 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:05.942+0000 7ff33a342640 1 -- start start 2023-05-22T15:40:05.947 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:05.942+0000 7ff33a342640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff3340ccab0 0x7ff33415fe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:40:05.947 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:05.942+0000 7ff33a342640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff3340d0410 0x7ff3340d0890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:40:05.948 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:05.942+0000 7ff33a342640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3340ccef0 con 0x7ff3340b7dd0 2023-05-22T15:40:05.948 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:05.942+0000 7ff33a342640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3340d3360 con 0x7ff3340d0410 2023-05-22T15:40:05.948 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:05.942+0000 7ff33a342640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3340cd170 con 0x7ff3340ccab0 2023-05-22T15:40:05.948 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:05.942+0000 7ff333fff640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7ff3340b7dd0 0x7ff33415f700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:41352/0 (socket says 172.21.15.16:41352) 2023-05-22T15:40:05.948 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:05.942+0000 7ff333fff640 1 -- 172.21.15.16:0/584582837 learned_addr learned my addr 172.21.15.16:0/584582837 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:40:05.948 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:05.942+0000 7ff3388df640 1 --2- 172.21.15.16:0/584582837 >> v2:172.21.15.16:3300/0 conn(0x7ff3340d0410 0x7ff3340d0890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:40:05.949 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:05.942+0000 7ff3337fe640 1 --2- 172.21.15.16:0/584582837 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff3340ccab0 0x7ff33415fe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:40:05.949 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:05.942+0000 7ff332ffd640 1 -- 172.21.15.16:0/584582837 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4197246734 0 0) 0x7ff3340ccef0 con 0x7ff3340b7dd0 2023-05-22T15:40:05.949 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:05.942+0000 7ff332ffd640 1 -- 172.21.15.16:0/584582837 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff31c003530 con 0x7ff3340b7dd0 2023-05-22T15:40:05.949 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:05.942+0000 7ff3388df640 1 -- 172.21.15.16:0/584582837 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff3340ccab0 msgr2=0x7ff33415fe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:40:05.949 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:05.942+0000 7ff3388df640 1 --2- 172.21.15.16:0/584582837 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff3340ccab0 0x7ff33415fe10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:05.950 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:05.942+0000 7ff3388df640 1 -- 172.21.15.16:0/584582837 >> v1:172.21.15.16:6789/0 conn(0x7ff3340b7dd0 legacy=0x7ff33415f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:40:05.950 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:05.942+0000 7ff3388df640 1 -- 172.21.15.16:0/584582837 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3340d2ac0 con 0x7ff3340d0410 2023-05-22T15:40:05.951 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:05.946+0000 7ff3388df640 1 --2- 172.21.15.16:0/584582837 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff3340d0410 0x7ff3340d0890 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7ff32c009d90 tx=0x7ff32c01c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af8eb0c8c3b8b65f server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:40:05.951 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:05.946+0000 7ff332ffd640 1 -- 172.21.15.16:0/584582837 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff32c0a59e0 con 0x7ff3340d0410 2023-05-22T15:40:05.951 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:05.946+0000 7ff332ffd640 1 -- 172.21.15.16:0/584582837 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff32c0a5b40 con 0x7ff3340d0410 2023-05-22T15:40:05.951 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:05.946+0000 7ff332ffd640 1 -- 172.21.15.16:0/584582837 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff32c0ad440 con 0x7ff3340d0410 2023-05-22T15:40:05.951 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:05.946+0000 7ff33a342640 1 -- 172.21.15.16:0/584582837 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff3340d0410 msgr2=0x7ff3340d0890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:40:05.951 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:05.946+0000 7ff33a342640 1 --2- 172.21.15.16:0/584582837 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff3340d0410 0x7ff3340d0890 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7ff32c009d90 tx=0x7ff32c01c920 comp rx=0 tx=0).stop 2023-05-22T15:40:05.952 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:05.946+0000 7ff33a342640 1 -- 172.21.15.16:0/584582837 shutdown_connections 2023-05-22T15:40:05.952 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:05.946+0000 7ff33a342640 1 --2- 172.21.15.16:0/584582837 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff3340d0410 0x7ff3340d0890 unknown :-1 s=CLOSED pgs=918 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:05.952 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:05.946+0000 7ff33a342640 1 --2- 172.21.15.16:0/584582837 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff3340ccab0 0x7ff33415fe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:05.952 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:05.946+0000 7ff33a342640 1 -- 172.21.15.16:0/584582837 >> 172.21.15.16:0/584582837 conn(0x7ff3340655c0 msgr2=0x7ff334070e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:40:05.952 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:05.946+0000 7ff33a342640 1 -- 172.21.15.16:0/584582837 shutdown_connections 2023-05-22T15:40:05.952 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:05.946+0000 7ff33a342640 1 -- 172.21.15.16:0/584582837 wait complete. 2023-05-22T15:40:05.953 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:05.946+0000 7ff33a342640 1 Processor -- start 2023-05-22T15:40:05.953 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:05.946+0000 7ff33a342640 1 -- start start 2023-05-22T15:40:05.954 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:05.946+0000 7ff33a342640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff3340b7dd0 0x7ff334157e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:40:05.954 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:05.946+0000 7ff33a342640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff3340ccab0 0x7ff3340c8de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:40:05.954 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:05.946+0000 7ff33a342640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff3340d0410 0x7ff3340b2890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:40:05.955 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:05.946+0000 7ff33a342640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3340c5ec0 con 0x7ff3340d0410 2023-05-22T15:40:05.955 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:05.946+0000 7ff33a342640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3340cbd30 con 0x7ff3340ccab0 2023-05-22T15:40:05.955 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:05.946+0000 7ff33a342640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3340cc010 con 0x7ff3340b7dd0 2023-05-22T15:40:05.955 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:05.946+0000 7ff3388df640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff3340d0410 0x7ff3340b2890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:40:05.955 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:05.946+0000 7ff3388df640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff3340d0410 0x7ff3340b2890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34356/0 (socket says 172.21.15.16:34356) 2023-05-22T15:40:05.955 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:05.946+0000 7ff3388df640 1 -- 172.21.15.16:0/1233280937 learned_addr learned my addr 172.21.15.16:0/1233280937 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:40:05.955 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:05.950+0000 7ff3337fe640 1 --2- 172.21.15.16:0/1233280937 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff3340ccab0 0x7ff3340c8de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:40:05.956 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:05.950+0000 7ff333fff640 1 --2- 172.21.15.16:0/1233280937 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff3340b7dd0 0x7ff334157e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:40:05.956 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:05.950+0000 7ff3388df640 1 -- 172.21.15.16:0/1233280937 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff3340b7dd0 msgr2=0x7ff334157e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:40:05.956 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:05.950+0000 7ff3388df640 1 --2- 172.21.15.16:0/1233280937 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff3340b7dd0 0x7ff334157e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:05.956 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:05.950+0000 7ff3388df640 1 -- 172.21.15.16:0/1233280937 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff3340ccab0 msgr2=0x7ff3340c8de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:40:05.956 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:05.950+0000 7ff3388df640 1 --2- 172.21.15.16:0/1233280937 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff3340ccab0 0x7ff3340c8de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:05.956 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:05.950+0000 7ff3388df640 1 -- 172.21.15.16:0/1233280937 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff32c009a70 con 0x7ff3340d0410 2023-05-22T15:40:05.956 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:05.950+0000 7ff3388df640 1 --2- 172.21.15.16:0/1233280937 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff3340d0410 0x7ff3340b2890 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7ff32c01a550 tx=0x7ff32c0196c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:40:05.957 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:05.950+0000 7ff330ff9640 1 -- 172.21.15.16:0/1233280937 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff32c0bd070 con 0x7ff3340d0410 2023-05-22T15:40:05.957 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:05.950+0000 7ff330ff9640 1 -- 172.21.15.16:0/1233280937 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff32c004040 con 0x7ff3340d0410 2023-05-22T15:40:05.957 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:05.950+0000 7ff330ff9640 1 -- 172.21.15.16:0/1233280937 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff32c004430 con 0x7ff3340d0410 2023-05-22T15:40:05.958 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:05.950+0000 7ff33a342640 1 -- 172.21.15.16:0/1233280937 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff334160350 con 0x7ff3340d0410 2023-05-22T15:40:05.958 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:05.950+0000 7ff33a342640 1 -- 172.21.15.16:0/1233280937 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3340cf730 con 0x7ff3340d0410 2023-05-22T15:40:05.958 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:05.950+0000 7ff33a342640 1 -- 172.21.15.16:0/1233280937 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3341604e0 con 0x7ff3340d0410 2023-05-22T15:40:05.958 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:05.950+0000 7ff330ff9640 1 -- 172.21.15.16:0/1233280937 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) v1 ==== 84332+0+0 (secure 0 0 0) 0x7ff32c0b4030 con 0x7ff3340d0410 2023-05-22T15:40:05.958 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:05.950+0000 7ff330ff9640 1 --2- 172.21.15.16:0/1233280937 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ff3200654a0 0x7ff320067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:40:05.958 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:05.950+0000 7ff330ff9640 1 -- 172.21.15.16:0/1233280937 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 6382+0+0 (secure 0 0 0) 0x7ff32c11d5d0 con 0x7ff3340d0410 2023-05-22T15:40:05.959 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:05.950+0000 7ff333fff640 1 --2- 172.21.15.16:0/1233280937 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ff3200654a0 0x7ff320067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:40:05.959 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:05.954+0000 7ff333fff640 1 --2- 172.21.15.16:0/1233280937 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ff3200654a0 0x7ff320067950 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7ff324005890 tx=0x7ff324009040 comp rx=0 tx=0).ready entity=mgr.6240 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:40:05.961 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:05.954+0000 7ff330ff9640 1 -- 172.21.15.16:0/1233280937 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7ff32c0eaf60 con 0x7ff3340d0410 2023-05-22T15:40:06.146 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.138+0000 7ff33a342640 1 -- 172.21.15.16:0/1233280937 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff33404dbe0 con 0x7ff3340d0410 2023-05-22T15:40:06.146 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.142+0000 7ff330ff9640 1 -- 172.21.15.16:0/1233280937 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) v1 ==== 94+0+4645 (secure 0 0 0) 0x7ff32c0ee580 con 0x7ff3340d0410 2023-05-22T15:40:06.147 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:40:06.147 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":22,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5399,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6833/1859135164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":1859135164},{"type":"v1","addr":"172.21.15.62:6833","nonce":1859135164}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":17},{"gid":5836,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/3067106582","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":3067106582},{"type":"v1","addr":"172.21.15.62:6835","nonce":3067106582}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":17},{"gid":6462,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/717991089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":717991089},{"type":"v1","addr":"172.21.15.16:6835","nonce":717991089}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":17}],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-05-22T15:40:02.754483+0000","modified":"2023-05-22T15:40:05.773388+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6468},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6468":{"gid":6468,"name":"a","rank":0,"incarnation":18,"state":"up:active","state_seq":3,"addr":"172.21.15.16:6837/4263396864","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":4263396864},{"type":"v1","addr":"172.21.15.16:6837","nonce":4263396864}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2023-05-22T15:40:06.147 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 22 2023-05-22T15:40:06.149 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.142+0000 7ff33a342640 1 -- 172.21.15.16:0/1233280937 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ff3200654a0 msgr2=0x7ff320067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:40:06.149 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.142+0000 7ff33a342640 1 --2- 172.21.15.16:0/1233280937 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ff3200654a0 0x7ff320067950 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7ff324005890 tx=0x7ff324009040 comp rx=0 tx=0).stop 2023-05-22T15:40:06.149 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.142+0000 7ff33a342640 1 -- 172.21.15.16:0/1233280937 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff3340d0410 msgr2=0x7ff3340b2890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:40:06.149 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.142+0000 7ff33a342640 1 --2- 172.21.15.16:0/1233280937 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff3340d0410 0x7ff3340b2890 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7ff32c01a550 tx=0x7ff32c0196c0 comp rx=0 tx=0).stop 2023-05-22T15:40:06.149 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.142+0000 7ff33a342640 1 -- 172.21.15.16:0/1233280937 shutdown_connections 2023-05-22T15:40:06.150 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.142+0000 7ff33a342640 1 --2- 172.21.15.16:0/1233280937 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff3340d0410 0x7ff3340b2890 unknown :-1 s=CLOSED pgs=919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:06.150 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.142+0000 7ff33a342640 1 --2- 172.21.15.16:0/1233280937 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff3340ccab0 0x7ff3340c8de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:06.150 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.142+0000 7ff33a342640 1 --2- 172.21.15.16:0/1233280937 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ff3200654a0 0x7ff320067950 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:06.150 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.142+0000 7ff33a342640 1 --2- 172.21.15.16:0/1233280937 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff3340b7dd0 0x7ff334157e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:06.150 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.142+0000 7ff33a342640 1 -- 172.21.15.16:0/1233280937 >> 172.21.15.16:0/1233280937 conn(0x7ff3340655c0 msgr2=0x7ff334158dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:40:06.150 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.142+0000 7ff33a342640 1 -- 172.21.15.16:0/1233280937 shutdown_connections 2023-05-22T15:40:06.150 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.142+0000 7ff33a342640 1 -- 172.21.15.16:0/1233280937 wait complete. 2023-05-22T15:40:06.166 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-05-22T15:40:06.290 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.286+0000 7f383fabf640 1 Processor -- start 2023-05-22T15:40:06.290 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.286+0000 7f383fabf640 1 -- start start 2023-05-22T15:40:06.291 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.286+0000 7f383fabf640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f38380c9b40 0x7f383815e460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:40:06.291 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.286+0000 7f383fabf640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3838070df0 0x7f3838071210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:40:06.292 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.286+0000 7f383fabf640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38380b0650 con 0x7f38380ca4e0 2023-05-22T15:40:06.292 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.286+0000 7f383fabf640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38380b0370 con 0x7f38380c9b40 2023-05-22T15:40:06.292 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.286+0000 7f383fabf640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f383815d980 con 0x7f3838070df0 2023-05-22T15:40:06.292 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.286+0000 7f383d05a640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f38380c9b40 0x7f383815e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:40:06.292 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.286+0000 7f383d85b640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f38380ca4e0 0x7f38380afb40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:41358/0 (socket says 172.21.15.16:41358) 2023-05-22T15:40:06.292 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.286+0000 7f383d85b640 1 -- 172.21.15.16:0/4269851859 learned_addr learned my addr 172.21.15.16:0/4269851859 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:40:06.293 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.286+0000 7f383d05a640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f38380c9b40 0x7f383815e460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34366/0 (socket says 172.21.15.16:34366) 2023-05-22T15:40:06.293 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.286+0000 7f383e05c640 1 --2- 172.21.15.16:0/4269851859 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3838070df0 0x7f3838071210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:40:06.293 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.286+0000 7f383d05a640 1 -- 172.21.15.16:0/4269851859 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3838070df0 msgr2=0x7f3838071210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:40:06.293 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.286+0000 7f383d05a640 1 --2- 172.21.15.16:0/4269851859 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3838070df0 0x7f3838071210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:06.293 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.286+0000 7f383d05a640 1 -- 172.21.15.16:0/4269851859 >> v1:172.21.15.16:6789/0 conn(0x7f38380ca4e0 legacy=0x7f38380afb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:40:06.293 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.286+0000 7f383d05a640 1 -- 172.21.15.16:0/4269851859 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f383815dc60 con 0x7f38380c9b40 2023-05-22T15:40:06.294 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.286+0000 7f383d05a640 1 --2- 172.21.15.16:0/4269851859 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38380c9b40 0x7f383815e460 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7f38280097c0 tx=0x7f382801c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c36fe814c2441b7 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:40:06.294 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.286+0000 7f383c859640 1 -- 172.21.15.16:0/4269851859 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38280ac410 con 0x7f38380c9b40 2023-05-22T15:40:06.295 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.286+0000 7f383c859640 1 -- 172.21.15.16:0/4269851859 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38280ac570 con 0x7f38380c9b40 2023-05-22T15:40:06.295 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.286+0000 7f383c859640 1 -- 172.21.15.16:0/4269851859 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38280ac900 con 0x7f38380c9b40 2023-05-22T15:40:06.295 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.290+0000 7f383fabf640 1 -- 172.21.15.16:0/4269851859 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38380c9b40 msgr2=0x7f383815e460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:40:06.295 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.290+0000 7f383fabf640 1 --2- 172.21.15.16:0/4269851859 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38380c9b40 0x7f383815e460 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7f38280097c0 tx=0x7f382801c450 comp rx=0 tx=0).stop 2023-05-22T15:40:06.295 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.290+0000 7f383fabf640 1 -- 172.21.15.16:0/4269851859 shutdown_connections 2023-05-22T15:40:06.295 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.290+0000 7f383fabf640 1 --2- 172.21.15.16:0/4269851859 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3838070df0 0x7f3838071210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:06.295 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.290+0000 7f383fabf640 1 --2- 172.21.15.16:0/4269851859 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38380c9b40 0x7f383815e460 unknown :-1 s=CLOSED pgs=921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:06.295 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.290+0000 7f383fabf640 1 -- 172.21.15.16:0/4269851859 >> 172.21.15.16:0/4269851859 conn(0x7f3838069720 msgr2=0x7f3838069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:40:06.296 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.290+0000 7f383fabf640 1 -- 172.21.15.16:0/4269851859 shutdown_connections 2023-05-22T15:40:06.296 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.290+0000 7f383fabf640 1 -- 172.21.15.16:0/4269851859 wait complete. 2023-05-22T15:40:06.296 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.290+0000 7f383fabf640 1 Processor -- start 2023-05-22T15:40:06.296 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.290+0000 7f383fabf640 1 -- start start 2023-05-22T15:40:06.298 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.290+0000 7f383fabf640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3838070df0 0x7f38380adb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:40:06.298 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.290+0000 7f383fabf640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f38380c9b40 0x7f38380ae040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:40:06.299 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.290+0000 7f383fabf640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38380ca4e0 0x7f38380ab7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:40:06.299 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.290+0000 7f383fabf640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38380ccd10 con 0x7f38380ca4e0 2023-05-22T15:40:06.299 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.290+0000 7f383fabf640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38381598e0 con 0x7f3838070df0 2023-05-22T15:40:06.299 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.290+0000 7f383fabf640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38380cc100 con 0x7f38380c9b40 2023-05-22T15:40:06.299 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.290+0000 7f383e05c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38380ca4e0 0x7f38380ab7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:40:06.299 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.290+0000 7f383e05c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38380ca4e0 0x7f38380ab7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34378/0 (socket says 172.21.15.16:34378) 2023-05-22T15:40:06.299 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.290+0000 7f383e05c640 1 -- 172.21.15.16:0/1336454876 learned_addr learned my addr 172.21.15.16:0/1336454876 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:40:06.299 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.290+0000 7f383d05a640 1 --2- 172.21.15.16:0/1336454876 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f38380c9b40 0x7f38380ae040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:40:06.300 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.290+0000 7f383d85b640 1 --2- 172.21.15.16:0/1336454876 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3838070df0 0x7f38380adb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:40:06.300 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.290+0000 7f383e05c640 1 -- 172.21.15.16:0/1336454876 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f38380c9b40 msgr2=0x7f38380ae040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:40:06.300 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.290+0000 7f383e05c640 1 --2- 172.21.15.16:0/1336454876 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f38380c9b40 0x7f38380ae040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:06.300 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.290+0000 7f383e05c640 1 -- 172.21.15.16:0/1336454876 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3838070df0 msgr2=0x7f38380adb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:40:06.300 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.290+0000 7f383e05c640 1 --2- 172.21.15.16:0/1336454876 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3838070df0 0x7f38380adb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:06.300 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.290+0000 7f383e05c640 1 -- 172.21.15.16:0/1336454876 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3828009420 con 0x7f38380ca4e0 2023-05-22T15:40:06.300 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.294+0000 7f383e05c640 1 --2- 172.21.15.16:0/1336454876 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38380ca4e0 0x7f38380ab7d0 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7f3830007e40 tx=0x7f383000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:40:06.301 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.294+0000 7f382e7fc640 1 -- 172.21.15.16:0/1336454876 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3830014620 con 0x7f38380ca4e0 2023-05-22T15:40:06.301 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.294+0000 7f382e7fc640 1 -- 172.21.15.16:0/1336454876 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f383000ee90 con 0x7f38380ca4e0 2023-05-22T15:40:06.301 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.294+0000 7f382e7fc640 1 -- 172.21.15.16:0/1336454876 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38300149e0 con 0x7f38380ca4e0 2023-05-22T15:40:06.301 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.294+0000 7f383fabf640 1 -- 172.21.15.16:0/1336454876 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38380c8e80 con 0x7f38380ca4e0 2023-05-22T15:40:06.301 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.294+0000 7f383fabf640 1 -- 172.21.15.16:0/1336454876 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38380ca8b0 con 0x7f38380ca4e0 2023-05-22T15:40:06.302 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.294+0000 7f383fabf640 1 -- 172.21.15.16:0/1336454876 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38380c9010 con 0x7f38380ca4e0 2023-05-22T15:40:06.302 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.294+0000 7f382e7fc640 1 -- 172.21.15.16:0/1336454876 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f3830014b40 con 0x7f38380ca4e0 2023-05-22T15:40:06.302 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.294+0000 7f382e7fc640 1 --2- 172.21.15.16:0/1336454876 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f381c0654d0 0x7f381c067980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:40:06.303 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.294+0000 7f382e7fc640 1 -- 172.21.15.16:0/1336454876 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f38300832d0 con 0x7f38380ca4e0 2023-05-22T15:40:06.303 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.294+0000 7f383d85b640 1 --2- 172.21.15.16:0/1336454876 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f381c0654d0 0x7f381c067980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:40:06.303 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.294+0000 7f383d85b640 1 --2- 172.21.15.16:0/1336454876 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f381c0654d0 0x7f381c067980 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f3834000f70 tx=0x7f3834009040 comp rx=0 tx=0).ready entity=mgr.6240 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:40:06.304 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.298+0000 7f382e7fc640 1 -- 172.21.15.16:0/1336454876 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f3830050e40 con 0x7f38380ca4e0 2023-05-22T15:40:06.458 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.450+0000 7f383fabf640 1 -- 172.21.15.16:0/1336454876 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f38380ccec0 con 0x7f38380ca4e0 2023-05-22T15:40:06.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.454+0000 7f382e7fc640 1 -- 172.21.15.16:0/1336454876 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v73) v1 ==== 74+0+18036 (secure 0 0 0) 0x7f38380ccec0 con 0x7f38380ca4e0 2023-05-22T15:40:06.460 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:40:06.460 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":73,"fsid":"b2719834-7847-4d98-90a6-f65d57f85c49","created":"2023-05-22T15:26:25.259314+0000","modified":"2023-05-22T15:40:03.758503+0000","last_up_change":"2023-05-22T15:26:36.448524+0000","last_in_change":"2023-05-22T15:26:27.675668+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-05-22T15:26:38.019781+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-05-22T15:26:43.570514+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2023-05-22T15:40:00.663040+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"72","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2023-05-22T15:40:01.750680+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"73","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9e7ff854-b25f-4cbb-a5c2-1799d9c53d2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":37021},{"type":"v1","addr":"172.21.15.16:6825","nonce":37021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":37021},{"type":"v1","addr":"172.21.15.16:6827","nonce":37021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":37021},{"type":"v1","addr":"172.21.15.16:6831","nonce":37021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":37021},{"type":"v1","addr":"172.21.15.16:6829","nonce":37021}]},"public_addr":"172.21.15.16:6825/37021","cluster_addr":"172.21.15.16:6827/37021","heartbeat_back_addr":"172.21.15.16:6831/37021","heartbeat_front_addr":"172.21.15.16:6829/37021","state":["exists","up"]},{"osd":1,"uuid":"4702fbc9-a927-4565-aca5-41dc69cc8264","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":37008},{"type":"v1","addr":"172.21.15.16:6801","nonce":37008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":37008},{"type":"v1","addr":"172.21.15.16:6803","nonce":37008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":37008},{"type":"v1","addr":"172.21.15.16:6807","nonce":37008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":37008},{"type":"v1","addr":"172.21.15.16:6805","nonce":37008}]},"public_addr":"172.21.15.16:6801/37008","cluster_addr":"172.21.15.16:6803/37008","heartbeat_back_addr":"172.21.15.16:6807/37008","heartbeat_front_addr":"172.21.15.16:6805/37008","state":["exists","up"]},{"osd":2,"uuid":"1e7bba46-4eba-47ef-bb81-5e2d371638ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":37009},{"type":"v1","addr":"172.21.15.16:6809","nonce":37009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":37009},{"type":"v1","addr":"172.21.15.16:6811","nonce":37009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":37009},{"type":"v1","addr":"172.21.15.16:6815","nonce":37009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":37009},{"type":"v1","addr":"172.21.15.16:6813","nonce":37009}]},"public_addr":"172.21.15.16:6809/37009","cluster_addr":"172.21.15.16:6811/37009","heartbeat_back_addr":"172.21.15.16:6815/37009","heartbeat_front_addr":"172.21.15.16:6813/37009","state":["exists","up"]},{"osd":3,"uuid":"383cacc0-5a88-4904-990e-498ca59ef48a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":37020},{"type":"v1","addr":"172.21.15.16:6817","nonce":37020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":37020},{"type":"v1","addr":"172.21.15.16:6819","nonce":37020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":37020},{"type":"v1","addr":"172.21.15.16:6823","nonce":37020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":37020},{"type":"v1","addr":"172.21.15.16:6821","nonce":37020}]},"public_addr":"172.21.15.16:6817/37020","cluster_addr":"172.21.15.16:6819/37020","heartbeat_back_addr":"172.21.15.16:6823/37020","heartbeat_front_addr":"172.21.15.16:6821/37020","state":["exists","up"]},{"osd":4,"uuid":"970500ef-95b7-42c1-b080-eed4fae7013d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":37353},{"type":"v1","addr":"172.21.15.62:6817","nonce":37353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6819","nonce":37353},{"type":"v1","addr":"172.21.15.62:6822","nonce":37353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37353},{"type":"v1","addr":"172.21.15.62:6831","nonce":37353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37353},{"type":"v1","addr":"172.21.15.62:6826","nonce":37353}]},"public_addr":"172.21.15.62:6817/37353","cluster_addr":"172.21.15.62:6822/37353","heartbeat_back_addr":"172.21.15.62:6831/37353","heartbeat_front_addr":"172.21.15.62:6826/37353","state":["exists","up"]},{"osd":5,"uuid":"022d67db-d4d8-445b-b1b9-8b446a00f5c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":37352},{"type":"v1","addr":"172.21.15.62:6801","nonce":37352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":37352},{"type":"v1","addr":"172.21.15.62:6803","nonce":37352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":37352},{"type":"v1","addr":"172.21.15.62:6807","nonce":37352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":37352},{"type":"v1","addr":"172.21.15.62:6805","nonce":37352}]},"public_addr":"172.21.15.62:6801/37352","cluster_addr":"172.21.15.62:6803/37352","heartbeat_back_addr":"172.21.15.62:6807/37352","heartbeat_front_addr":"172.21.15.62:6805/37352","state":["exists","up"]},{"osd":6,"uuid":"42bd372b-6688-4a1b-8407-e16e8c266e33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":37354},{"type":"v1","addr":"172.21.15.62:6820","nonce":37354}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":37354},{"type":"v1","addr":"172.21.15.62:6823","nonce":37354}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6829","nonce":37354},{"type":"v1","addr":"172.21.15.62:6830","nonce":37354}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6825","nonce":37354},{"type":"v1","addr":"172.21.15.62:6827","nonce":37354}]},"public_addr":"172.21.15.62:6820/37354","cluster_addr":"172.21.15.62:6823/37354","heartbeat_back_addr":"172.21.15.62:6830/37354","heartbeat_front_addr":"172.21.15.62:6827/37354","state":["exists","up"]},{"osd":7,"uuid":"ac70b2a3-f41b-4bf2-b227-7f2cbb55426d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37355},{"type":"v1","addr":"172.21.15.62:6809","nonce":37355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37355},{"type":"v1","addr":"172.21.15.62:6811","nonce":37355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":37355},{"type":"v1","addr":"172.21.15.62:6815","nonce":37355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":37355},{"type":"v1","addr":"172.21.15.62:6813","nonce":37355}]},"public_addr":"172.21.15.62:6809/37355","cluster_addr":"172.21.15.62:6811/37355","heartbeat_back_addr":"172.21.15.62:6815/37355","heartbeat_front_addr":"172.21.15.62:6813/37355","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.368896+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.107273+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.276888+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.287502+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.124791+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.251937+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.255679+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.232797+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-05-22T15:40:06.462 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.454+0000 7f383fabf640 1 -- 172.21.15.16:0/1336454876 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f381c0654d0 msgr2=0x7f381c067980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:40:06.462 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.454+0000 7f383fabf640 1 --2- 172.21.15.16:0/1336454876 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f381c0654d0 0x7f381c067980 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f3834000f70 tx=0x7f3834009040 comp rx=0 tx=0).stop 2023-05-22T15:40:06.462 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.454+0000 7f383fabf640 1 -- 172.21.15.16:0/1336454876 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38380ca4e0 msgr2=0x7f38380ab7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:40:06.462 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.454+0000 7f383fabf640 1 --2- 172.21.15.16:0/1336454876 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38380ca4e0 0x7f38380ab7d0 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7f3830007e40 tx=0x7f383000e4b0 comp rx=0 tx=0).stop 2023-05-22T15:40:06.462 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.454+0000 7f383fabf640 1 -- 172.21.15.16:0/1336454876 shutdown_connections 2023-05-22T15:40:06.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.454+0000 7f383fabf640 1 --2- 172.21.15.16:0/1336454876 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38380ca4e0 0x7f38380ab7d0 unknown :-1 s=CLOSED pgs=922 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:06.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.454+0000 7f383fabf640 1 --2- 172.21.15.16:0/1336454876 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f381c0654d0 0x7f381c067980 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:06.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.454+0000 7f383fabf640 1 --2- 172.21.15.16:0/1336454876 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f38380c9b40 0x7f38380ae040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:06.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.454+0000 7f383fabf640 1 --2- 172.21.15.16:0/1336454876 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3838070df0 0x7f38380adb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:06.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.454+0000 7f383fabf640 1 -- 172.21.15.16:0/1336454876 >> 172.21.15.16:0/1336454876 conn(0x7f3838069720 msgr2=0x7f38381565a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:40:06.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.454+0000 7f383fabf640 1 -- 172.21.15.16:0/1336454876 shutdown_connections 2023-05-22T15:40:06.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.458+0000 7f383fabf640 1 -- 172.21.15.16:0/1336454876 wait complete. 2023-05-22T15:40:06.480 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-05-22T15:40:06.481 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-05-22T15:40:06.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.594+0000 7fa2fc8e4640 1 Processor -- start 2023-05-22T15:40:06.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.594+0000 7fa2fc8e4640 1 -- start start 2023-05-22T15:40:06.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.594+0000 7fa2fc8e4640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fa2f8071640 0x7fa2f8160090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:40:06.600 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.594+0000 7fa2fc8e4640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fa2f81605d0 0x7fa2f815c270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:40:06.600 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.594+0000 7fa2fc8e4640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2f8069ae0 con 0x7fa2f80ce480 2023-05-22T15:40:06.600 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.594+0000 7fa2fc8e4640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2f815f990 con 0x7fa2f81605d0 2023-05-22T15:40:06.600 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.594+0000 7fa2fc8e4640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2f8071ac0 con 0x7fa2f8071640 2023-05-22T15:40:06.600 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.594+0000 7fa2f659c640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fa2f80ce480 0x7fa2f80ce850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:41366/0 (socket says 172.21.15.16:41366) 2023-05-22T15:40:06.600 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.594+0000 7fa2f659c640 1 -- 172.21.15.16:0/162565231 learned_addr learned my addr 172.21.15.16:0/162565231 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:40:06.601 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.594+0000 7fa2f6d9d640 1 --2- 172.21.15.16:0/162565231 >> v2:172.21.15.16:3300/0 conn(0x7fa2f81605d0 0x7fa2f815c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:40:06.601 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.594+0000 7fa2f5d9b640 1 --2- 172.21.15.16:0/162565231 >> v2:172.21.15.62:3300/0 conn(0x7fa2f8071640 0x7fa2f8160090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:40:06.601 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.594+0000 7fa2f559a640 1 -- 172.21.15.16:0/162565231 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4260572482 0 0) 0x7fa2f8069ae0 con 0x7fa2f80ce480 2023-05-22T15:40:06.601 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.594+0000 7fa2f559a640 1 -- 172.21.15.16:0/162565231 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa2e0003530 con 0x7fa2f80ce480 2023-05-22T15:40:06.601 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.594+0000 7fa2f6d9d640 1 -- 172.21.15.16:0/162565231 >> v2:172.21.15.62:3300/0 conn(0x7fa2f8071640 msgr2=0x7fa2f8160090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:40:06.601 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.594+0000 7fa2f6d9d640 1 --2- 172.21.15.16:0/162565231 >> v2:172.21.15.62:3300/0 conn(0x7fa2f8071640 0x7fa2f8160090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:06.601 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.594+0000 7fa2f6d9d640 1 -- 172.21.15.16:0/162565231 >> v1:172.21.15.16:6789/0 conn(0x7fa2f80ce480 legacy=0x7fa2f80ce850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:40:06.602 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.594+0000 7fa2f6d9d640 1 -- 172.21.15.16:0/162565231 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2f8069dc0 con 0x7fa2f81605d0 2023-05-22T15:40:06.603 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.594+0000 7fa2f6d9d640 1 --2- 172.21.15.16:0/162565231 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa2f81605d0 0x7fa2f815c270 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7fa2f80b1d50 tx=0x7fa2f001c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ede77ceab9d9e173 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:40:06.603 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.598+0000 7fa2f559a640 1 -- 172.21.15.16:0/162565231 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2f00a59a0 con 0x7fa2f81605d0 2023-05-22T15:40:06.603 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.598+0000 7fa2f559a640 1 -- 172.21.15.16:0/162565231 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2f00a5b00 con 0x7fa2f81605d0 2023-05-22T15:40:06.603 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.598+0000 7fa2f559a640 1 -- 172.21.15.16:0/162565231 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2f00a5e90 con 0x7fa2f81605d0 2023-05-22T15:40:06.603 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.598+0000 7fa2fc8e4640 1 -- 172.21.15.16:0/162565231 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa2f81605d0 msgr2=0x7fa2f815c270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:40:06.603 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.598+0000 7fa2fc8e4640 1 --2- 172.21.15.16:0/162565231 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa2f81605d0 0x7fa2f815c270 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7fa2f80b1d50 tx=0x7fa2f001c8e0 comp rx=0 tx=0).stop 2023-05-22T15:40:06.604 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.598+0000 7fa2fc8e4640 1 -- 172.21.15.16:0/162565231 shutdown_connections 2023-05-22T15:40:06.604 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.598+0000 7fa2fc8e4640 1 --2- 172.21.15.16:0/162565231 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa2f81605d0 0x7fa2f815c270 unknown :-1 s=CLOSED pgs=924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:06.604 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.598+0000 7fa2fc8e4640 1 --2- 172.21.15.16:0/162565231 >> v2:172.21.15.62:3300/0 conn(0x7fa2f8071640 0x7fa2f8160090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:06.604 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.598+0000 7fa2fc8e4640 1 -- 172.21.15.16:0/162565231 >> 172.21.15.16:0/162565231 conn(0x7fa2f80cc870 msgr2=0x7fa2f80cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:40:06.604 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.598+0000 7fa2fc8e4640 1 -- 172.21.15.16:0/162565231 shutdown_connections 2023-05-22T15:40:06.604 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.598+0000 7fa2fc8e4640 1 -- 172.21.15.16:0/162565231 wait complete. 2023-05-22T15:40:06.606 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.598+0000 7fa2fc8e4640 1 Processor -- start 2023-05-22T15:40:06.606 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.598+0000 7fa2fc8e4640 1 -- start start 2023-05-22T15:40:06.606 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.598+0000 7fa2fc8e4640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa2f8071640 0x7fa2f8152d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:40:06.606 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.598+0000 7fa2fc8e4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa2f80ce480 0x7fa2f81552c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:40:06.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.598+0000 7fa2fc8e4640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa2f81605d0 0x7fa2f80afe70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:40:06.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.598+0000 7fa2fc8e4640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2f8154e30 con 0x7fa2f80ce480 2023-05-22T15:40:06.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.598+0000 7fa2fc8e4640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2f8153290 con 0x7fa2f81605d0 2023-05-22T15:40:06.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.598+0000 7fa2fc8e4640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2f8155800 con 0x7fa2f8071640 2023-05-22T15:40:06.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.598+0000 7fa2f5d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa2f80ce480 0x7fa2f81552c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:40:06.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.598+0000 7fa2f5d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa2f80ce480 0x7fa2f81552c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34394/0 (socket says 172.21.15.16:34394) 2023-05-22T15:40:06.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.598+0000 7fa2f5d9b640 1 -- 172.21.15.16:0/2606686672 learned_addr learned my addr 172.21.15.16:0/2606686672 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:40:06.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.598+0000 7fa2f6d9d640 1 --2- 172.21.15.16:0/2606686672 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa2f81605d0 0x7fa2f80afe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:40:06.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.598+0000 7fa2f659c640 1 --2- 172.21.15.16:0/2606686672 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa2f8071640 0x7fa2f8152d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:40:06.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.598+0000 7fa2f5d9b640 1 -- 172.21.15.16:0/2606686672 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa2f8071640 msgr2=0x7fa2f8152d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:40:06.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.598+0000 7fa2f5d9b640 1 --2- 172.21.15.16:0/2606686672 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa2f8071640 0x7fa2f8152d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:06.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.598+0000 7fa2f5d9b640 1 -- 172.21.15.16:0/2606686672 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa2f81605d0 msgr2=0x7fa2f80afe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:40:06.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.598+0000 7fa2f5d9b640 1 --2- 172.21.15.16:0/2606686672 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa2f81605d0 0x7fa2f80afe70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:06.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.598+0000 7fa2f5d9b640 1 -- 172.21.15.16:0/2606686672 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2f0009a70 con 0x7fa2f80ce480 2023-05-22T15:40:06.609 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.598+0000 7fa2f5d9b640 1 --2- 172.21.15.16:0/2606686672 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa2f80ce480 0x7fa2f81552c0 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7fa2ec00d5f0 tx=0x7fa2ec00dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:40:06.609 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.598+0000 7fa2e6ffd640 1 -- 172.21.15.16:0/2606686672 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2ec0125f0 con 0x7fa2f80ce480 2023-05-22T15:40:06.609 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.598+0000 7fa2e6ffd640 1 -- 172.21.15.16:0/2606686672 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2ec012750 con 0x7fa2f80ce480 2023-05-22T15:40:06.609 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.598+0000 7fa2e6ffd640 1 -- 172.21.15.16:0/2606686672 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2ec012ae0 con 0x7fa2f80ce480 2023-05-22T15:40:06.609 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.598+0000 7fa2fc8e4640 1 -- 172.21.15.16:0/2606686672 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2f80698b0 con 0x7fa2f80ce480 2023-05-22T15:40:06.609 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.598+0000 7fa2fc8e4640 1 -- 172.21.15.16:0/2606686672 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2f8069720 con 0x7fa2f80ce480 2023-05-22T15:40:06.610 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.602+0000 7fa2fc8e4640 1 -- 172.21.15.16:0/2606686672 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2f80c5e70 con 0x7fa2f80ce480 2023-05-22T15:40:06.610 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.602+0000 7fa2e6ffd640 1 -- 172.21.15.16:0/2606686672 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) v1 ==== 84332+0+0 (secure 0 0 0) 0x7fa2ec012c40 con 0x7fa2f80ce480 2023-05-22T15:40:06.610 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.602+0000 7fa2e6ffd640 1 --2- 172.21.15.16:0/2606686672 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fa2d80654d0 0x7fa2d8067980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:40:06.611 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.602+0000 7fa2e6ffd640 1 -- 172.21.15.16:0/2606686672 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 6382+0+0 (secure 0 0 0) 0x7fa2ec082fd0 con 0x7fa2f80ce480 2023-05-22T15:40:06.611 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.602+0000 7fa2f659c640 1 --2- 172.21.15.16:0/2606686672 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fa2d80654d0 0x7fa2d8067980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:40:06.611 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.602+0000 7fa2f659c640 1 --2- 172.21.15.16:0/2606686672 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fa2d80654d0 0x7fa2d8067980 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fa2e80011f0 tx=0x7fa2e8009040 comp rx=0 tx=0).ready entity=mgr.6240 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:40:06.612 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.606+0000 7fa2e6ffd640 1 -- 172.21.15.16:0/2606686672 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fa2ec0509a0 con 0x7fa2f80ce480 2023-05-22T15:40:06.762 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.758+0000 7fa2fc8e4640 1 -- 172.21.15.16:0/2606686672 --> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fa2f804dbe0 con 0x7fa2d80654d0 2023-05-22T15:40:06.788 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.782+0000 7fa2e6ffd640 1 -- 172.21.15.16:0/2606686672 <== mgr.6240 v2:172.21.15.16:6832/36921 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+310089 (secure 0 0 0) 0x7fa2f804dbe0 con 0x7fa2d80654d0 2023-05-22T15:40:06.791 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:40:06.792 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2023-05-22T15:40:06.794 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.786+0000 7fa2fc8e4640 1 -- 172.21.15.16:0/2606686672 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fa2d80654d0 msgr2=0x7fa2d8067980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:40:06.794 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.786+0000 7fa2fc8e4640 1 --2- 172.21.15.16:0/2606686672 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fa2d80654d0 0x7fa2d8067980 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fa2e80011f0 tx=0x7fa2e8009040 comp rx=0 tx=0).stop 2023-05-22T15:40:06.794 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.786+0000 7fa2fc8e4640 1 -- 172.21.15.16:0/2606686672 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa2f80ce480 msgr2=0x7fa2f81552c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:40:06.794 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.786+0000 7fa2fc8e4640 1 --2- 172.21.15.16:0/2606686672 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa2f80ce480 0x7fa2f81552c0 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7fa2ec00d5f0 tx=0x7fa2ec00dab0 comp rx=0 tx=0).stop 2023-05-22T15:40:06.794 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.786+0000 7fa2fc8e4640 1 -- 172.21.15.16:0/2606686672 shutdown_connections 2023-05-22T15:40:06.795 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.786+0000 7fa2fc8e4640 1 --2- 172.21.15.16:0/2606686672 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa2f81605d0 0x7fa2f80afe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:06.795 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.786+0000 7fa2fc8e4640 1 --2- 172.21.15.16:0/2606686672 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa2f80ce480 0x7fa2f81552c0 unknown :-1 s=CLOSED pgs=925 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:06.795 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.786+0000 7fa2fc8e4640 1 --2- 172.21.15.16:0/2606686672 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fa2d80654d0 0x7fa2d8067980 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:06.795 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.786+0000 7fa2fc8e4640 1 --2- 172.21.15.16:0/2606686672 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa2f8071640 0x7fa2f8152d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:06.795 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.786+0000 7fa2fc8e4640 1 -- 172.21.15.16:0/2606686672 >> 172.21.15.16:0/2606686672 conn(0x7fa2f80cc870 msgr2=0x7fa2f8158af0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:40:06.795 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.786+0000 7fa2fc8e4640 1 -- 172.21.15.16:0/2606686672 shutdown_connections 2023-05-22T15:40:06.796 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.786+0000 7fa2fc8e4640 1 -- 172.21.15.16:0/2606686672 wait complete. 2023-05-22T15:40:06.811 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":25,"stamp":"2023-05-22T15:40:05.090362+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592698,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":308,"num_read_kb":512,"num_write":243,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":177,"ondisk_log_size":177,"up":210,"acting":210,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":79,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":195880,"kb_used_data":7784,"kb_used_omap":0,"kb_used_meta":188032,"kb_avail":754778840,"statfs":{"total":773094113280,"available":772893532160,"internally_reserved":0,"allocated":7970816,"data_stored":4667042,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":192544768},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":72,"apply_latency_ms":72,"commit_latency_ns":72000000,"apply_latency_ns":72000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2330,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001218"},"pg_stats":[{"pgid":"9.3f","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.781736+0000","last_change":"2023-05-22T15:40:02.779921+0000","last_active":"2023-05-22T15:40:02.781736+0000","last_peered":"2023-05-22T15:40:02.781736+0000","last_clean":"2023-05-22T15:40:02.781736+0000","last_became_active":"2023-05-22T15:40:02.779630+0000","last_became_peered":"2023-05-22T15:40:02.779630+0000","last_unstale":"2023-05-22T15:40:02.781736+0000","last_undegraded":"2023-05-22T15:40:02.781736+0000","last_fullsized":"2023-05-22T15:40:02.781736+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_clean_scrub_stamp":"2023-05-22T15:40:02.364399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T18:28:48.622897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3e","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.793198+0000","last_change":"2023-05-22T15:40:02.791660+0000","last_active":"2023-05-22T15:40:02.793198+0000","last_peered":"2023-05-22T15:40:02.793198+0000","last_clean":"2023-05-22T15:40:02.793198+0000","last_became_active":"2023-05-22T15:40:02.791427+0000","last_became_peered":"2023-05-22T15:40:02.791427+0000","last_unstale":"2023-05-22T15:40:02.793198+0000","last_undegraded":"2023-05-22T15:40:02.793198+0000","last_fullsized":"2023-05-22T15:40:02.793198+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_clean_scrub_stamp":"2023-05-22T15:40:02.364399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T03:24:02.342328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.3d","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.840900+0000","last_change":"2023-05-22T15:40:02.839056+0000","last_active":"2023-05-22T15:40:02.840900+0000","last_peered":"2023-05-22T15:40:02.840900+0000","last_clean":"2023-05-22T15:40:02.840900+0000","last_became_active":"2023-05-22T15:40:02.838286+0000","last_became_peered":"2023-05-22T15:40:02.838286+0000","last_unstale":"2023-05-22T15:40:02.840900+0000","last_undegraded":"2023-05-22T15:40:02.840900+0000","last_fullsized":"2023-05-22T15:40:02.840900+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_clean_scrub_stamp":"2023-05-22T15:40:02.364399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:15:23.770715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.3c","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.813018+0000","last_change":"2023-05-22T15:40:02.784140+0000","last_active":"2023-05-22T15:40:02.813018+0000","last_peered":"2023-05-22T15:40:02.813018+0000","last_clean":"2023-05-22T15:40:02.813018+0000","last_became_active":"2023-05-22T15:40:02.782943+0000","last_became_peered":"2023-05-22T15:40:02.782943+0000","last_unstale":"2023-05-22T15:40:02.813018+0000","last_undegraded":"2023-05-22T15:40:02.813018+0000","last_fullsized":"2023-05-22T15:40:02.813018+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_clean_scrub_stamp":"2023-05-22T15:40:02.364399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T16:42:44.882436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3b","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.809202+0000","last_change":"2023-05-22T15:40:02.808254+0000","last_active":"2023-05-22T15:40:02.809202+0000","last_peered":"2023-05-22T15:40:02.809202+0000","last_clean":"2023-05-22T15:40:02.809202+0000","last_became_active":"2023-05-22T15:40:02.807935+0000","last_became_peered":"2023-05-22T15:40:02.807935+0000","last_unstale":"2023-05-22T15:40:02.809202+0000","last_undegraded":"2023-05-22T15:40:02.809202+0000","last_fullsized":"2023-05-22T15:40:02.809202+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_clean_scrub_stamp":"2023-05-22T15:40:02.364399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:13:12.338903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3a","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.794462+0000","last_change":"2023-05-22T15:40:02.780909+0000","last_active":"2023-05-22T15:40:02.794462+0000","last_peered":"2023-05-22T15:40:02.794462+0000","last_clean":"2023-05-22T15:40:02.794462+0000","last_became_active":"2023-05-22T15:40:02.779811+0000","last_became_peered":"2023-05-22T15:40:02.779811+0000","last_unstale":"2023-05-22T15:40:02.794462+0000","last_undegraded":"2023-05-22T15:40:02.794462+0000","last_fullsized":"2023-05-22T15:40:02.794462+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_clean_scrub_stamp":"2023-05-22T15:40:02.364399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:39:11.668037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.39","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.840421+0000","last_change":"2023-05-22T15:40:02.821670+0000","last_active":"2023-05-22T15:40:02.840421+0000","last_peered":"2023-05-22T15:40:02.840421+0000","last_clean":"2023-05-22T15:40:02.840421+0000","last_became_active":"2023-05-22T15:40:02.819555+0000","last_became_peered":"2023-05-22T15:40:02.819555+0000","last_unstale":"2023-05-22T15:40:02.840421+0000","last_undegraded":"2023-05-22T15:40:02.840421+0000","last_fullsized":"2023-05-22T15:40:02.840421+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_clean_scrub_stamp":"2023-05-22T15:40:02.364399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:49:45.980298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.38","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.820792+0000","last_change":"2023-05-22T15:40:02.819968+0000","last_active":"2023-05-22T15:40:02.820792+0000","last_peered":"2023-05-22T15:40:02.820792+0000","last_clean":"2023-05-22T15:40:02.820792+0000","last_became_active":"2023-05-22T15:40:02.819721+0000","last_became_peered":"2023-05-22T15:40:02.819721+0000","last_unstale":"2023-05-22T15:40:02.820792+0000","last_undegraded":"2023-05-22T15:40:02.820792+0000","last_fullsized":"2023-05-22T15:40:02.820792+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_clean_scrub_stamp":"2023-05-22T15:40:02.364399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:53:20.742871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.37","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.838729+0000","last_change":"2023-05-22T15:40:02.818930+0000","last_active":"2023-05-22T15:40:02.838729+0000","last_peered":"2023-05-22T15:40:02.838729+0000","last_clean":"2023-05-22T15:40:02.838729+0000","last_became_active":"2023-05-22T15:40:02.818309+0000","last_became_peered":"2023-05-22T15:40:02.818309+0000","last_unstale":"2023-05-22T15:40:02.838729+0000","last_undegraded":"2023-05-22T15:40:02.838729+0000","last_fullsized":"2023-05-22T15:40:02.838729+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_clean_scrub_stamp":"2023-05-22T15:40:02.364399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T18:55:07.687839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.36","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.811337+0000","last_change":"2023-05-22T15:40:02.810378+0000","last_active":"2023-05-22T15:40:02.811337+0000","last_peered":"2023-05-22T15:40:02.811337+0000","last_clean":"2023-05-22T15:40:02.811337+0000","last_became_active":"2023-05-22T15:40:02.810099+0000","last_became_peered":"2023-05-22T15:40:02.810099+0000","last_unstale":"2023-05-22T15:40:02.811337+0000","last_undegraded":"2023-05-22T15:40:02.811337+0000","last_fullsized":"2023-05-22T15:40:02.811337+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_clean_scrub_stamp":"2023-05-22T15:40:02.364399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:24:41.408664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.35","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.793322+0000","last_change":"2023-05-22T15:40:02.787237+0000","last_active":"2023-05-22T15:40:02.793322+0000","last_peered":"2023-05-22T15:40:02.793322+0000","last_clean":"2023-05-22T15:40:02.793322+0000","last_became_active":"2023-05-22T15:40:02.786973+0000","last_became_peered":"2023-05-22T15:40:02.786973+0000","last_unstale":"2023-05-22T15:40:02.793322+0000","last_undegraded":"2023-05-22T15:40:02.793322+0000","last_fullsized":"2023-05-22T15:40:02.793322+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_clean_scrub_stamp":"2023-05-22T15:40:02.364399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:16:00.201080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.34","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.813761+0000","last_change":"2023-05-22T15:40:02.786906+0000","last_active":"2023-05-22T15:40:02.813761+0000","last_peered":"2023-05-22T15:40:02.813761+0000","last_clean":"2023-05-22T15:40:02.813761+0000","last_became_active":"2023-05-22T15:40:02.786094+0000","last_became_peered":"2023-05-22T15:40:02.786094+0000","last_unstale":"2023-05-22T15:40:02.813761+0000","last_undegraded":"2023-05-22T15:40:02.813761+0000","last_fullsized":"2023-05-22T15:40:02.813761+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_clean_scrub_stamp":"2023-05-22T15:40:02.364399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:49:45.980298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.33","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.817249+0000","last_change":"2023-05-22T15:40:02.801624+0000","last_active":"2023-05-22T15:40:02.817249+0000","last_peered":"2023-05-22T15:40:02.817249+0000","last_clean":"2023-05-22T15:40:02.817249+0000","last_became_active":"2023-05-22T15:40:02.795684+0000","last_became_peered":"2023-05-22T15:40:02.795684+0000","last_unstale":"2023-05-22T15:40:02.817249+0000","last_undegraded":"2023-05-22T15:40:02.817249+0000","last_fullsized":"2023-05-22T15:40:02.817249+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_clean_scrub_stamp":"2023-05-22T15:40:02.364399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:08:19.275487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.32","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.837582+0000","last_change":"2023-05-22T15:40:02.818926+0000","last_active":"2023-05-22T15:40:02.837582+0000","last_peered":"2023-05-22T15:40:02.837582+0000","last_clean":"2023-05-22T15:40:02.837582+0000","last_became_active":"2023-05-22T15:40:02.818292+0000","last_became_peered":"2023-05-22T15:40:02.818292+0000","last_unstale":"2023-05-22T15:40:02.837582+0000","last_undegraded":"2023-05-22T15:40:02.837582+0000","last_fullsized":"2023-05-22T15:40:02.837582+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_clean_scrub_stamp":"2023-05-22T15:40:02.364399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:32:29.815721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.31","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.839069+0000","last_change":"2023-05-22T15:40:02.820283+0000","last_active":"2023-05-22T15:40:02.839069+0000","last_peered":"2023-05-22T15:40:02.839069+0000","last_clean":"2023-05-22T15:40:02.839069+0000","last_became_active":"2023-05-22T15:40:02.818679+0000","last_became_peered":"2023-05-22T15:40:02.818679+0000","last_unstale":"2023-05-22T15:40:02.839069+0000","last_undegraded":"2023-05-22T15:40:02.839069+0000","last_fullsized":"2023-05-22T15:40:02.839069+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_clean_scrub_stamp":"2023-05-22T15:40:02.364399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:59:22.265916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.30","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.817210+0000","last_change":"2023-05-22T15:40:02.816563+0000","last_active":"2023-05-22T15:40:02.817210+0000","last_peered":"2023-05-22T15:40:02.817210+0000","last_clean":"2023-05-22T15:40:02.817210+0000","last_became_active":"2023-05-22T15:40:02.816313+0000","last_became_peered":"2023-05-22T15:40:02.816313+0000","last_unstale":"2023-05-22T15:40:02.817210+0000","last_undegraded":"2023-05-22T15:40:02.817210+0000","last_fullsized":"2023-05-22T15:40:02.817210+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_clean_scrub_stamp":"2023-05-22T15:40:02.364399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:26:16.725005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.2f","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.811365+0000","last_change":"2023-05-22T15:40:02.810702+0000","last_active":"2023-05-22T15:40:02.811365+0000","last_peered":"2023-05-22T15:40:02.811365+0000","last_clean":"2023-05-22T15:40:02.811365+0000","last_became_active":"2023-05-22T15:40:02.810432+0000","last_became_peered":"2023-05-22T15:40:02.810432+0000","last_unstale":"2023-05-22T15:40:02.811365+0000","last_undegraded":"2023-05-22T15:40:02.811365+0000","last_fullsized":"2023-05-22T15:40:02.811365+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_clean_scrub_stamp":"2023-05-22T15:40:02.364399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:28:29.223431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2e","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.837775+0000","last_change":"2023-05-22T15:40:02.831752+0000","last_active":"2023-05-22T15:40:02.837775+0000","last_peered":"2023-05-22T15:40:02.837775+0000","last_clean":"2023-05-22T15:40:02.837775+0000","last_became_active":"2023-05-22T15:40:02.830649+0000","last_became_peered":"2023-05-22T15:40:02.830649+0000","last_unstale":"2023-05-22T15:40:02.837775+0000","last_undegraded":"2023-05-22T15:40:02.837775+0000","last_fullsized":"2023-05-22T15:40:02.837775+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_clean_scrub_stamp":"2023-05-22T15:40:02.364399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:34:11.612573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2d","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.780081+0000","last_change":"2023-05-22T15:40:02.778871+0000","last_active":"2023-05-22T15:40:02.780081+0000","last_peered":"2023-05-22T15:40:02.780081+0000","last_clean":"2023-05-22T15:40:02.780081+0000","last_became_active":"2023-05-22T15:40:02.778557+0000","last_became_peered":"2023-05-22T15:40:02.778557+0000","last_unstale":"2023-05-22T15:40:02.780081+0000","last_undegraded":"2023-05-22T15:40:02.780081+0000","last_fullsized":"2023-05-22T15:40:02.780081+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_clean_scrub_stamp":"2023-05-22T15:40:02.364399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:31:50.868086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.2c","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.784146+0000","last_change":"2023-05-22T15:40:02.782901+0000","last_active":"2023-05-22T15:40:02.784146+0000","last_peered":"2023-05-22T15:40:02.784146+0000","last_clean":"2023-05-22T15:40:02.784146+0000","last_became_active":"2023-05-22T15:40:02.781896+0000","last_became_peered":"2023-05-22T15:40:02.781896+0000","last_unstale":"2023-05-22T15:40:02.784146+0000","last_undegraded":"2023-05-22T15:40:02.784146+0000","last_fullsized":"2023-05-22T15:40:02.784146+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_clean_scrub_stamp":"2023-05-22T15:40:02.364399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:56:01.718556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2b","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.836924+0000","last_change":"2023-05-22T15:40:02.829751+0000","last_active":"2023-05-22T15:40:02.836924+0000","last_peered":"2023-05-22T15:40:02.836924+0000","last_clean":"2023-05-22T15:40:02.836924+0000","last_became_active":"2023-05-22T15:40:02.829265+0000","last_became_peered":"2023-05-22T15:40:02.829265+0000","last_unstale":"2023-05-22T15:40:02.836924+0000","last_undegraded":"2023-05-22T15:40:02.836924+0000","last_fullsized":"2023-05-22T15:40:02.836924+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_clean_scrub_stamp":"2023-05-22T15:40:02.364399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:03:39.548392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2a","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.785380+0000","last_change":"2023-05-22T15:40:02.782405+0000","last_active":"2023-05-22T15:40:02.785380+0000","last_peered":"2023-05-22T15:40:02.785380+0000","last_clean":"2023-05-22T15:40:02.785380+0000","last_became_active":"2023-05-22T15:40:02.782132+0000","last_became_peered":"2023-05-22T15:40:02.782132+0000","last_unstale":"2023-05-22T15:40:02.785380+0000","last_undegraded":"2023-05-22T15:40:02.785380+0000","last_fullsized":"2023-05-22T15:40:02.785380+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_clean_scrub_stamp":"2023-05-22T15:40:02.364399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:18:45.607512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.29","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.830432+0000","last_change":"2023-05-22T15:40:02.818884+0000","last_active":"2023-05-22T15:40:02.830432+0000","last_peered":"2023-05-22T15:40:02.830432+0000","last_clean":"2023-05-22T15:40:02.830432+0000","last_became_active":"2023-05-22T15:40:02.818270+0000","last_became_peered":"2023-05-22T15:40:02.818270+0000","last_unstale":"2023-05-22T15:40:02.830432+0000","last_undegraded":"2023-05-22T15:40:02.830432+0000","last_fullsized":"2023-05-22T15:40:02.830432+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_clean_scrub_stamp":"2023-05-22T15:40:02.364399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T18:55:07.687839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.28","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.824976+0000","last_change":"2023-05-22T15:40:02.815134+0000","last_active":"2023-05-22T15:40:02.824976+0000","last_peered":"2023-05-22T15:40:02.824976+0000","last_clean":"2023-05-22T15:40:02.824976+0000","last_became_active":"2023-05-22T15:40:02.809339+0000","last_became_peered":"2023-05-22T15:40:02.809339+0000","last_unstale":"2023-05-22T15:40:02.824976+0000","last_undegraded":"2023-05-22T15:40:02.824976+0000","last_fullsized":"2023-05-22T15:40:02.824976+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_clean_scrub_stamp":"2023-05-22T15:40:02.364399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:35:22.712578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.27","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.794070+0000","last_change":"2023-05-22T15:40:02.791708+0000","last_active":"2023-05-22T15:40:02.794070+0000","last_peered":"2023-05-22T15:40:02.794070+0000","last_clean":"2023-05-22T15:40:02.794070+0000","last_became_active":"2023-05-22T15:40:02.791426+0000","last_became_peered":"2023-05-22T15:40:02.791426+0000","last_unstale":"2023-05-22T15:40:02.794070+0000","last_undegraded":"2023-05-22T15:40:02.794070+0000","last_fullsized":"2023-05-22T15:40:02.794070+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_clean_scrub_stamp":"2023-05-22T15:40:02.364399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:56:03.869179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.26","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.827059+0000","last_change":"2023-05-22T15:40:02.820844+0000","last_active":"2023-05-22T15:40:02.827059+0000","last_peered":"2023-05-22T15:40:02.827059+0000","last_clean":"2023-05-22T15:40:02.827059+0000","last_became_active":"2023-05-22T15:40:02.820069+0000","last_became_peered":"2023-05-22T15:40:02.820069+0000","last_unstale":"2023-05-22T15:40:02.827059+0000","last_undegraded":"2023-05-22T15:40:02.827059+0000","last_fullsized":"2023-05-22T15:40:02.827059+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_clean_scrub_stamp":"2023-05-22T15:40:02.364399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:32:29.815721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.25","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.811522+0000","last_change":"2023-05-22T15:40:02.809736+0000","last_active":"2023-05-22T15:40:02.811522+0000","last_peered":"2023-05-22T15:40:02.811522+0000","last_clean":"2023-05-22T15:40:02.811522+0000","last_became_active":"2023-05-22T15:40:02.809472+0000","last_became_peered":"2023-05-22T15:40:02.809472+0000","last_unstale":"2023-05-22T15:40:02.811522+0000","last_undegraded":"2023-05-22T15:40:02.811522+0000","last_fullsized":"2023-05-22T15:40:02.811522+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_clean_scrub_stamp":"2023-05-22T15:40:02.364399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T18:16:05.626894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.24","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.809191+0000","last_change":"2023-05-22T15:40:02.808246+0000","last_active":"2023-05-22T15:40:02.809191+0000","last_peered":"2023-05-22T15:40:02.809191+0000","last_clean":"2023-05-22T15:40:02.809191+0000","last_became_active":"2023-05-22T15:40:02.807971+0000","last_became_peered":"2023-05-22T15:40:02.807971+0000","last_unstale":"2023-05-22T15:40:02.809191+0000","last_undegraded":"2023-05-22T15:40:02.809191+0000","last_fullsized":"2023-05-22T15:40:02.809191+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_clean_scrub_stamp":"2023-05-22T15:40:02.364399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:05:21.391829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.23","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.778713+0000","last_change":"2023-05-22T15:40:02.776218+0000","last_active":"2023-05-22T15:40:02.778713+0000","last_peered":"2023-05-22T15:40:02.778713+0000","last_clean":"2023-05-22T15:40:02.778713+0000","last_became_active":"2023-05-22T15:40:02.775924+0000","last_became_peered":"2023-05-22T15:40:02.775924+0000","last_unstale":"2023-05-22T15:40:02.778713+0000","last_undegraded":"2023-05-22T15:40:02.778713+0000","last_fullsized":"2023-05-22T15:40:02.778713+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_clean_scrub_stamp":"2023-05-22T15:40:02.364399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T18:16:05.626894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.22","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.841758+0000","last_change":"2023-05-22T15:40:02.840087+0000","last_active":"2023-05-22T15:40:02.841758+0000","last_peered":"2023-05-22T15:40:02.841758+0000","last_clean":"2023-05-22T15:40:02.841758+0000","last_became_active":"2023-05-22T15:40:02.838145+0000","last_became_peered":"2023-05-22T15:40:02.838145+0000","last_unstale":"2023-05-22T15:40:02.841758+0000","last_undegraded":"2023-05-22T15:40:02.841758+0000","last_fullsized":"2023-05-22T15:40:02.841758+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_clean_scrub_stamp":"2023-05-22T15:40:02.364399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T16:42:44.882436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.21","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.790952+0000","last_change":"2023-05-22T15:40:02.788115+0000","last_active":"2023-05-22T15:40:02.790952+0000","last_peered":"2023-05-22T15:40:02.790952+0000","last_clean":"2023-05-22T15:40:02.790952+0000","last_became_active":"2023-05-22T15:40:02.787838+0000","last_became_peered":"2023-05-22T15:40:02.787838+0000","last_unstale":"2023-05-22T15:40:02.790952+0000","last_undegraded":"2023-05-22T15:40:02.790952+0000","last_fullsized":"2023-05-22T15:40:02.790952+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_clean_scrub_stamp":"2023-05-22T15:40:02.364399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:19:59.892493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.20","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.792780+0000","last_change":"2023-05-22T15:40:02.774767+0000","last_active":"2023-05-22T15:40:02.792780+0000","last_peered":"2023-05-22T15:40:02.792780+0000","last_clean":"2023-05-22T15:40:02.792780+0000","last_became_active":"2023-05-22T15:40:02.774127+0000","last_became_peered":"2023-05-22T15:40:02.774127+0000","last_unstale":"2023-05-22T15:40:02.792780+0000","last_undegraded":"2023-05-22T15:40:02.792780+0000","last_fullsized":"2023-05-22T15:40:02.792780+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_clean_scrub_stamp":"2023-05-22T15:40:02.364399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T03:11:31.485165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.795265+0000","last_change":"2023-05-22T15:40:02.786593+0000","last_active":"2023-05-22T15:40:02.795265+0000","last_peered":"2023-05-22T15:40:02.795265+0000","last_clean":"2023-05-22T15:40:02.795265+0000","last_became_active":"2023-05-22T15:40:02.786334+0000","last_became_peered":"2023-05-22T15:40:02.786334+0000","last_unstale":"2023-05-22T15:40:02.795265+0000","last_undegraded":"2023-05-22T15:40:02.795265+0000","last_fullsized":"2023-05-22T15:40:02.795265+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_clean_scrub_stamp":"2023-05-22T15:40:02.364399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:03:44.838990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1f","version":"72'4","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.814167+0000","last_change":"2023-05-22T15:40:02.408457+0000","last_active":"2023-05-22T15:40:02.814167+0000","last_peered":"2023-05-22T15:40:02.814167+0000","last_clean":"2023-05-22T15:40:02.814167+0000","last_became_active":"2023-05-22T15:40:02.408157+0000","last_became_peered":"2023-05-22T15:40:02.408157+0000","last_unstale":"2023-05-22T15:40:02.814167+0000","last_undegraded":"2023-05-22T15:40:02.814167+0000","last_fullsized":"2023-05-22T15:40:02.814167+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:40:01.360691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:40:01.360691+0000","last_clean_scrub_stamp":"2023-05-22T15:40:01.360691+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:49:44.976589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":566,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.798668+0000","last_change":"2023-05-22T15:40:02.797775+0000","last_active":"2023-05-22T15:40:02.798668+0000","last_peered":"2023-05-22T15:40:02.798668+0000","last_clean":"2023-05-22T15:40:02.798668+0000","last_became_active":"2023-05-22T15:40:02.797573+0000","last_became_peered":"2023-05-22T15:40:02.797573+0000","last_unstale":"2023-05-22T15:40:02.798668+0000","last_undegraded":"2023-05-22T15:40:02.798668+0000","last_fullsized":"2023-05-22T15:40:02.798668+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_clean_scrub_stamp":"2023-05-22T15:40:02.364399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:05:21.391829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1e","version":"72'1","reported_seq":11,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.819727+0000","last_change":"2023-05-22T15:40:02.417453+0000","last_active":"2023-05-22T15:40:02.819727+0000","last_peered":"2023-05-22T15:40:02.819727+0000","last_clean":"2023-05-22T15:40:02.819727+0000","last_became_active":"2023-05-22T15:40:02.417144+0000","last_became_peered":"2023-05-22T15:40:02.417144+0000","last_unstale":"2023-05-22T15:40:02.819727+0000","last_undegraded":"2023-05-22T15:40:02.819727+0000","last_fullsized":"2023-05-22T15:40:02.819727+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:40:01.360691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:40:01.360691+0000","last_clean_scrub_stamp":"2023-05-22T15:40:01.360691+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:59:21.262207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.795371+0000","last_change":"2023-05-22T15:40:02.789568+0000","last_active":"2023-05-22T15:40:02.795371+0000","last_peered":"2023-05-22T15:40:02.795371+0000","last_clean":"2023-05-22T15:40:02.795371+0000","last_became_active":"2023-05-22T15:40:02.789272+0000","last_became_peered":"2023-05-22T15:40:02.789272+0000","last_unstale":"2023-05-22T15:40:02.795371+0000","last_undegraded":"2023-05-22T15:40:02.795371+0000","last_fullsized":"2023-05-22T15:40:02.795371+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_clean_scrub_stamp":"2023-05-22T15:40:02.364399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T15:55:35.316310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"72'2","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.795399+0000","last_change":"2023-05-22T15:40:02.405212+0000","last_active":"2023-05-22T15:40:02.795399+0000","last_peered":"2023-05-22T15:40:02.795399+0000","last_clean":"2023-05-22T15:40:02.795399+0000","last_became_active":"2023-05-22T15:40:02.404665+0000","last_became_peered":"2023-05-22T15:40:02.404665+0000","last_unstale":"2023-05-22T15:40:02.795399+0000","last_undegraded":"2023-05-22T15:40:02.795399+0000","last_fullsized":"2023-05-22T15:40:02.795399+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:40:01.360691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:40:01.360691+0000","last_clean_scrub_stamp":"2023-05-22T15:40:01.360691+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:37:20.385027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.794406+0000","last_change":"2023-05-22T15:40:02.778286+0000","last_active":"2023-05-22T15:40:02.794406+0000","last_peered":"2023-05-22T15:40:02.794406+0000","last_clean":"2023-05-22T15:40:02.794406+0000","last_became_active":"2023-05-22T15:40:02.777257+0000","last_became_peered":"2023-05-22T15:40:02.777257+0000","last_unstale":"2023-05-22T15:40:02.794406+0000","last_undegraded":"2023-05-22T15:40:02.794406+0000","last_fullsized":"2023-05-22T15:40:02.794406+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_clean_scrub_stamp":"2023-05-22T15:40:02.364399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:42:34.280673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1c","version":"72'2","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.829702+0000","last_change":"2023-05-22T15:40:02.415257+0000","last_active":"2023-05-22T15:40:02.829702+0000","last_peered":"2023-05-22T15:40:02.829702+0000","last_clean":"2023-05-22T15:40:02.829702+0000","last_became_active":"2023-05-22T15:40:02.414276+0000","last_became_peered":"2023-05-22T15:40:02.414276+0000","last_unstale":"2023-05-22T15:40:02.829702+0000","last_undegraded":"2023-05-22T15:40:02.829702+0000","last_fullsized":"2023-05-22T15:40:02.829702+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:40:01.360691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:40:01.360691+0000","last_clean_scrub_stamp":"2023-05-22T15:40:01.360691+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:42:55.348036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.782496+0000","last_change":"2023-05-22T15:40:02.780209+0000","last_active":"2023-05-22T15:40:02.782496+0000","last_peered":"2023-05-22T15:40:02.782496+0000","last_clean":"2023-05-22T15:40:02.782496+0000","last_became_active":"2023-05-22T15:40:02.779941+0000","last_became_peered":"2023-05-22T15:40:02.779941+0000","last_unstale":"2023-05-22T15:40:02.782496+0000","last_undegraded":"2023-05-22T15:40:02.782496+0000","last_fullsized":"2023-05-22T15:40:02.782496+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_clean_scrub_stamp":"2023-05-22T15:40:02.364399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:18:56.678275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1b","version":"72'1","reported_seq":11,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.837207+0000","last_change":"2023-05-22T15:40:02.414064+0000","last_active":"2023-05-22T15:40:02.837207+0000","last_peered":"2023-05-22T15:40:02.837207+0000","last_clean":"2023-05-22T15:40:02.837207+0000","last_became_active":"2023-05-22T15:40:02.412878+0000","last_became_peered":"2023-05-22T15:40:02.412878+0000","last_unstale":"2023-05-22T15:40:02.837207+0000","last_undegraded":"2023-05-22T15:40:02.837207+0000","last_fullsized":"2023-05-22T15:40:02.837207+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:40:01.360691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:40:01.360691+0000","last_clean_scrub_stamp":"2023-05-22T15:40:01.360691+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:53:28.798573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.783317+0000","last_change":"2023-05-22T15:40:02.782024+0000","last_active":"2023-05-22T15:40:02.783317+0000","last_peered":"2023-05-22T15:40:02.783317+0000","last_clean":"2023-05-22T15:40:02.783317+0000","last_became_active":"2023-05-22T15:40:02.781702+0000","last_became_peered":"2023-05-22T15:40:02.781702+0000","last_unstale":"2023-05-22T15:40:02.783317+0000","last_undegraded":"2023-05-22T15:40:02.783317+0000","last_fullsized":"2023-05-22T15:40:02.783317+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_clean_scrub_stamp":"2023-05-22T15:40:02.364399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:03:44.838990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1a","version":"72'1","reported_seq":11,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.809662+0000","last_change":"2023-05-22T15:40:02.421209+0000","last_active":"2023-05-22T15:40:02.809662+0000","last_peered":"2023-05-22T15:40:02.809662+0000","last_clean":"2023-05-22T15:40:02.809662+0000","last_became_active":"2023-05-22T15:40:02.420869+0000","last_became_peered":"2023-05-22T15:40:02.420869+0000","last_unstale":"2023-05-22T15:40:02.809662+0000","last_undegraded":"2023-05-22T15:40:02.809662+0000","last_fullsized":"2023-05-22T15:40:02.809662+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:40:01.360691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:40:01.360691+0000","last_clean_scrub_stamp":"2023-05-22T15:40:01.360691+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:32:28.812012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.824935+0000","last_change":"2023-05-22T15:40:02.808669+0000","last_active":"2023-05-22T15:40:02.824935+0000","last_peered":"2023-05-22T15:40:02.824935+0000","last_clean":"2023-05-22T15:40:02.824935+0000","last_became_active":"2023-05-22T15:40:02.807906+0000","last_became_peered":"2023-05-22T15:40:02.807906+0000","last_unstale":"2023-05-22T15:40:02.824935+0000","last_undegraded":"2023-05-22T15:40:02.824935+0000","last_fullsized":"2023-05-22T15:40:02.824935+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_clean_scrub_stamp":"2023-05-22T15:40:02.364399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:37:37.439227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":10,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.770029+0000","last_change":"2023-05-22T15:40:02.411052+0000","last_active":"2023-05-22T15:40:02.770029+0000","last_peered":"2023-05-22T15:40:02.770029+0000","last_clean":"2023-05-22T15:40:02.770029+0000","last_became_active":"2023-05-22T15:40:02.410320+0000","last_became_peered":"2023-05-22T15:40:02.410320+0000","last_unstale":"2023-05-22T15:40:02.770029+0000","last_undegraded":"2023-05-22T15:40:02.770029+0000","last_fullsized":"2023-05-22T15:40:02.770029+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:40:01.360691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:40:01.360691+0000","last_clean_scrub_stamp":"2023-05-22T15:40:01.360691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T18:12:12.676405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.796404+0000","last_change":"2023-05-22T15:40:02.795831+0000","last_active":"2023-05-22T15:40:02.796404+0000","last_peered":"2023-05-22T15:40:02.796404+0000","last_clean":"2023-05-22T15:40:02.796404+0000","last_became_active":"2023-05-22T15:40:02.795583+0000","last_became_peered":"2023-05-22T15:40:02.795583+0000","last_unstale":"2023-05-22T15:40:02.796404+0000","last_undegraded":"2023-05-22T15:40:02.796404+0000","last_fullsized":"2023-05-22T15:40:02.796404+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_clean_scrub_stamp":"2023-05-22T15:40:02.364399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:47:39.526320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":10,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.781756+0000","last_change":"2023-05-22T15:40:02.410980+0000","last_active":"2023-05-22T15:40:02.781756+0000","last_peered":"2023-05-22T15:40:02.781756+0000","last_clean":"2023-05-22T15:40:02.781756+0000","last_became_active":"2023-05-22T15:40:02.410248+0000","last_became_peered":"2023-05-22T15:40:02.410248+0000","last_unstale":"2023-05-22T15:40:02.781756+0000","last_undegraded":"2023-05-22T15:40:02.781756+0000","last_fullsized":"2023-05-22T15:40:02.781756+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:40:01.360691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:40:01.360691+0000","last_clean_scrub_stamp":"2023-05-22T15:40:01.360691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:27:40.572807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.837793+0000","last_change":"2023-05-22T15:40:02.818942+0000","last_active":"2023-05-22T15:40:02.837793+0000","last_peered":"2023-05-22T15:40:02.837793+0000","last_clean":"2023-05-22T15:40:02.837793+0000","last_became_active":"2023-05-22T15:40:02.818386+0000","last_became_peered":"2023-05-22T15:40:02.818386+0000","last_unstale":"2023-05-22T15:40:02.837793+0000","last_undegraded":"2023-05-22T15:40:02.837793+0000","last_fullsized":"2023-05-22T15:40:02.837793+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_clean_scrub_stamp":"2023-05-22T15:40:02.364399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:02:47.354363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":10,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.806642+0000","last_change":"2023-05-22T15:40:02.429499+0000","last_active":"2023-05-22T15:40:02.806642+0000","last_peered":"2023-05-22T15:40:02.806642+0000","last_clean":"2023-05-22T15:40:02.806642+0000","last_became_active":"2023-05-22T15:40:02.428907+0000","last_became_peered":"2023-05-22T15:40:02.428907+0000","last_unstale":"2023-05-22T15:40:02.806642+0000","last_undegraded":"2023-05-22T15:40:02.806642+0000","last_fullsized":"2023-05-22T15:40:02.806642+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:40:01.360691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:40:01.360691+0000","last_clean_scrub_stamp":"2023-05-22T15:40:01.360691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:06:57.809773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.837079+0000","last_change":"2023-05-22T15:40:02.831128+0000","last_active":"2023-05-22T15:40:02.837079+0000","last_peered":"2023-05-22T15:40:02.837079+0000","last_clean":"2023-05-22T15:40:02.837079+0000","last_became_active":"2023-05-22T15:40:02.830566+0000","last_became_peered":"2023-05-22T15:40:02.830566+0000","last_unstale":"2023-05-22T15:40:02.837079+0000","last_undegraded":"2023-05-22T15:40:02.837079+0000","last_fullsized":"2023-05-22T15:40:02.837079+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_clean_scrub_stamp":"2023-05-22T15:40:02.364399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T16:05:38.299803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":10,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.766216+0000","last_change":"2023-05-22T15:40:02.402559+0000","last_active":"2023-05-22T15:40:02.766216+0000","last_peered":"2023-05-22T15:40:02.766216+0000","last_clean":"2023-05-22T15:40:02.766216+0000","last_became_active":"2023-05-22T15:40:02.402062+0000","last_became_peered":"2023-05-22T15:40:02.402062+0000","last_unstale":"2023-05-22T15:40:02.766216+0000","last_undegraded":"2023-05-22T15:40:02.766216+0000","last_fullsized":"2023-05-22T15:40:02.766216+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:40:01.360691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:40:01.360691+0000","last_clean_scrub_stamp":"2023-05-22T15:40:01.360691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:49:44.976589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.824196+0000","last_change":"2023-05-22T15:40:02.822750+0000","last_active":"2023-05-22T15:40:02.824196+0000","last_peered":"2023-05-22T15:40:02.824196+0000","last_clean":"2023-05-22T15:40:02.824196+0000","last_became_active":"2023-05-22T15:40:02.822446+0000","last_became_peered":"2023-05-22T15:40:02.822446+0000","last_unstale":"2023-05-22T15:40:02.824196+0000","last_undegraded":"2023-05-22T15:40:02.824196+0000","last_fullsized":"2023-05-22T15:40:02.824196+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_clean_scrub_stamp":"2023-05-22T15:40:02.364399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:06:58.813481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.15","version":"72'3","reported_seq":13,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.831542+0000","last_change":"2023-05-22T15:40:02.428653+0000","last_active":"2023-05-22T15:40:02.831542+0000","last_peered":"2023-05-22T15:40:02.831542+0000","last_clean":"2023-05-22T15:40:02.831542+0000","last_became_active":"2023-05-22T15:40:02.427985+0000","last_became_peered":"2023-05-22T15:40:02.427985+0000","last_unstale":"2023-05-22T15:40:02.831542+0000","last_undegraded":"2023-05-22T15:40:02.831542+0000","last_fullsized":"2023-05-22T15:40:02.831542+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:40:01.360691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:40:01.360691+0000","last_clean_scrub_stamp":"2023-05-22T15:40:01.360691+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:18:20.056393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.836963+0000","last_change":"2023-05-22T15:40:02.830104+0000","last_active":"2023-05-22T15:40:02.836963+0000","last_peered":"2023-05-22T15:40:02.836963+0000","last_clean":"2023-05-22T15:40:02.836963+0000","last_became_active":"2023-05-22T15:40:02.829439+0000","last_became_peered":"2023-05-22T15:40:02.829439+0000","last_unstale":"2023-05-22T15:40:02.836963+0000","last_undegraded":"2023-05-22T15:40:02.836963+0000","last_fullsized":"2023-05-22T15:40:02.836963+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_clean_scrub_stamp":"2023-05-22T15:40:02.364399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:59:22.265916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.14","version":"72'6","reported_seq":16,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.842400+0000","last_change":"2023-05-22T15:40:02.415057+0000","last_active":"2023-05-22T15:40:02.842400+0000","last_peered":"2023-05-22T15:40:02.842400+0000","last_clean":"2023-05-22T15:40:02.842400+0000","last_became_active":"2023-05-22T15:40:02.412040+0000","last_became_peered":"2023-05-22T15:40:02.412040+0000","last_unstale":"2023-05-22T15:40:02.842400+0000","last_undegraded":"2023-05-22T15:40:02.842400+0000","last_fullsized":"2023-05-22T15:40:02.842400+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:40:01.360691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:40:01.360691+0000","last_clean_scrub_stamp":"2023-05-22T15:40:01.360691+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:41:13.995283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1548,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.813582+0000","last_change":"2023-05-22T15:40:02.792527+0000","last_active":"2023-05-22T15:40:02.813582+0000","last_peered":"2023-05-22T15:40:02.813582+0000","last_clean":"2023-05-22T15:40:02.813582+0000","last_became_active":"2023-05-22T15:40:02.791805+0000","last_became_peered":"2023-05-22T15:40:02.791805+0000","last_unstale":"2023-05-22T15:40:02.813582+0000","last_undegraded":"2023-05-22T15:40:02.813582+0000","last_fullsized":"2023-05-22T15:40:02.813582+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_clean_scrub_stamp":"2023-05-22T15:40:02.364399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:59:22.265916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.13","version":"72'2","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.790538+0000","last_change":"2023-05-22T15:40:02.407045+0000","last_active":"2023-05-22T15:40:02.790538+0000","last_peered":"2023-05-22T15:40:02.790538+0000","last_clean":"2023-05-22T15:40:02.790538+0000","last_became_active":"2023-05-22T15:40:02.406699+0000","last_became_peered":"2023-05-22T15:40:02.406699+0000","last_unstale":"2023-05-22T15:40:02.790538+0000","last_undegraded":"2023-05-22T15:40:02.790538+0000","last_fullsized":"2023-05-22T15:40:02.790538+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:40:01.360691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:40:01.360691+0000","last_clean_scrub_stamp":"2023-05-22T15:40:01.360691+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:34:10.608865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.833683+0000","last_change":"2023-05-22T15:40:02.823616+0000","last_active":"2023-05-22T15:40:02.833683+0000","last_peered":"2023-05-22T15:40:02.833683+0000","last_clean":"2023-05-22T15:40:02.833683+0000","last_became_active":"2023-05-22T15:40:02.821749+0000","last_became_peered":"2023-05-22T15:40:02.821749+0000","last_unstale":"2023-05-22T15:40:02.833683+0000","last_undegraded":"2023-05-22T15:40:02.833683+0000","last_fullsized":"2023-05-22T15:40:02.833683+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_clean_scrub_stamp":"2023-05-22T15:40:02.364399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T15:46:56.191505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":10,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.790265+0000","last_change":"2023-05-22T15:40:02.430134+0000","last_active":"2023-05-22T15:40:02.790265+0000","last_peered":"2023-05-22T15:40:02.790265+0000","last_clean":"2023-05-22T15:40:02.790265+0000","last_became_active":"2023-05-22T15:40:02.429312+0000","last_became_peered":"2023-05-22T15:40:02.429312+0000","last_unstale":"2023-05-22T15:40:02.790265+0000","last_undegraded":"2023-05-22T15:40:02.790265+0000","last_fullsized":"2023-05-22T15:40:02.790265+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:40:01.360691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:40:01.360691+0000","last_clean_scrub_stamp":"2023-05-22T15:40:01.360691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:54:19.121093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.839140+0000","last_change":"2023-05-22T15:40:02.819212+0000","last_active":"2023-05-22T15:40:02.839140+0000","last_peered":"2023-05-22T15:40:02.839140+0000","last_clean":"2023-05-22T15:40:02.839140+0000","last_became_active":"2023-05-22T15:40:02.818034+0000","last_became_peered":"2023-05-22T15:40:02.818034+0000","last_unstale":"2023-05-22T15:40:02.839140+0000","last_undegraded":"2023-05-22T15:40:02.839140+0000","last_fullsized":"2023-05-22T15:40:02.839140+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_clean_scrub_stamp":"2023-05-22T15:40:02.364399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T16:42:44.882436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":10,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.770058+0000","last_change":"2023-05-22T15:40:02.415313+0000","last_active":"2023-05-22T15:40:02.770058+0000","last_peered":"2023-05-22T15:40:02.770058+0000","last_clean":"2023-05-22T15:40:02.770058+0000","last_became_active":"2023-05-22T15:40:02.414454+0000","last_became_peered":"2023-05-22T15:40:02.414454+0000","last_unstale":"2023-05-22T15:40:02.770058+0000","last_undegraded":"2023-05-22T15:40:02.770058+0000","last_fullsized":"2023-05-22T15:40:02.770058+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:40:01.360691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:40:01.360691+0000","last_clean_scrub_stamp":"2023-05-22T15:40:01.360691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:56:25.008963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.836988+0000","last_change":"2023-05-22T15:40:02.830208+0000","last_active":"2023-05-22T15:40:02.836988+0000","last_peered":"2023-05-22T15:40:02.836988+0000","last_clean":"2023-05-22T15:40:02.836988+0000","last_became_active":"2023-05-22T15:40:02.829380+0000","last_became_peered":"2023-05-22T15:40:02.829380+0000","last_unstale":"2023-05-22T15:40:02.836988+0000","last_undegraded":"2023-05-22T15:40:02.836988+0000","last_fullsized":"2023-05-22T15:40:02.836988+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_clean_scrub_stamp":"2023-05-22T15:40:02.364399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:32:29.815721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.10","version":"72'8","reported_seq":18,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.838032+0000","last_change":"2023-05-22T15:40:02.429012+0000","last_active":"2023-05-22T15:40:02.838032+0000","last_peered":"2023-05-22T15:40:02.838032+0000","last_clean":"2023-05-22T15:40:02.838032+0000","last_became_active":"2023-05-22T15:40:02.428136+0000","last_became_peered":"2023-05-22T15:40:02.428136+0000","last_unstale":"2023-05-22T15:40:02.838032+0000","last_undegraded":"2023-05-22T15:40:02.838032+0000","last_fullsized":"2023-05-22T15:40:02.838032+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:40:01.360691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:40:01.360691+0000","last_clean_scrub_stamp":"2023-05-22T15:40:01.360691+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:13:11.335194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.813159+0000","last_change":"2023-05-22T15:40:02.785890+0000","last_active":"2023-05-22T15:40:02.813159+0000","last_peered":"2023-05-22T15:40:02.813159+0000","last_clean":"2023-05-22T15:40:02.813159+0000","last_became_active":"2023-05-22T15:40:02.784750+0000","last_became_peered":"2023-05-22T15:40:02.784750+0000","last_unstale":"2023-05-22T15:40:02.813159+0000","last_undegraded":"2023-05-22T15:40:02.813159+0000","last_fullsized":"2023-05-22T15:40:02.813159+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_clean_scrub_stamp":"2023-05-22T15:40:02.364399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:03:39.548392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.7","version":"72'2","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.793410+0000","last_change":"2023-05-22T15:40:02.398139+0000","last_active":"2023-05-22T15:40:02.793410+0000","last_peered":"2023-05-22T15:40:02.793410+0000","last_clean":"2023-05-22T15:40:02.793410+0000","last_became_active":"2023-05-22T15:40:02.397785+0000","last_became_peered":"2023-05-22T15:40:02.397785+0000","last_unstale":"2023-05-22T15:40:02.793410+0000","last_undegraded":"2023-05-22T15:40:02.793410+0000","last_fullsized":"2023-05-22T15:40:02.793410+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:40:01.360691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:40:01.360691+0000","last_clean_scrub_stamp":"2023-05-22T15:40:01.360691+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:10:33.424873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.813654+0000","last_change":"2023-05-22T15:40:02.798752+0000","last_active":"2023-05-22T15:40:02.813654+0000","last_peered":"2023-05-22T15:40:02.813654+0000","last_clean":"2023-05-22T15:40:02.813654+0000","last_became_active":"2023-05-22T15:40:02.797984+0000","last_became_peered":"2023-05-22T15:40:02.797984+0000","last_unstale":"2023-05-22T15:40:02.813654+0000","last_undegraded":"2023-05-22T15:40:02.813654+0000","last_fullsized":"2023-05-22T15:40:02.813654+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_clean_scrub_stamp":"2023-05-22T15:40:02.364399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:34:11.612573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":10,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.765566+0000","last_change":"2023-05-22T15:40:02.422170+0000","last_active":"2023-05-22T15:40:02.765566+0000","last_peered":"2023-05-22T15:40:02.765566+0000","last_clean":"2023-05-22T15:40:02.765566+0000","last_became_active":"2023-05-22T15:40:02.421687+0000","last_became_peered":"2023-05-22T15:40:02.421687+0000","last_unstale":"2023-05-22T15:40:02.765566+0000","last_undegraded":"2023-05-22T15:40:02.765566+0000","last_fullsized":"2023-05-22T15:40:02.765566+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:40:01.360691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:40:01.360691+0000","last_clean_scrub_stamp":"2023-05-22T15:40:01.360691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:34:10.608865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.824184+0000","last_change":"2023-05-22T15:40:02.822612+0000","last_active":"2023-05-22T15:40:02.824184+0000","last_peered":"2023-05-22T15:40:02.824184+0000","last_clean":"2023-05-22T15:40:02.824184+0000","last_became_active":"2023-05-22T15:40:02.822322+0000","last_became_peered":"2023-05-22T15:40:02.822322+0000","last_unstale":"2023-05-22T15:40:02.824184+0000","last_undegraded":"2023-05-22T15:40:02.824184+0000","last_fullsized":"2023-05-22T15:40:02.824184+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_clean_scrub_stamp":"2023-05-22T15:40:02.364399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:18:21.060101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":10,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.790240+0000","last_change":"2023-05-22T15:40:02.428303+0000","last_active":"2023-05-22T15:40:02.790240+0000","last_peered":"2023-05-22T15:40:02.790240+0000","last_clean":"2023-05-22T15:40:02.790240+0000","last_became_active":"2023-05-22T15:40:02.427706+0000","last_became_peered":"2023-05-22T15:40:02.427706+0000","last_unstale":"2023-05-22T15:40:02.790240+0000","last_undegraded":"2023-05-22T15:40:02.790240+0000","last_fullsized":"2023-05-22T15:40:02.790240+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:40:01.360691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:40:01.360691+0000","last_clean_scrub_stamp":"2023-05-22T15:40:01.360691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T15:50:34.811208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.794466+0000","last_change":"2023-05-22T15:40:02.778514+0000","last_active":"2023-05-22T15:40:02.794466+0000","last_peered":"2023-05-22T15:40:02.794466+0000","last_clean":"2023-05-22T15:40:02.794466+0000","last_became_active":"2023-05-22T15:40:02.777931+0000","last_became_peered":"2023-05-22T15:40:02.777931+0000","last_unstale":"2023-05-22T15:40:02.794466+0000","last_undegraded":"2023-05-22T15:40:02.794466+0000","last_fullsized":"2023-05-22T15:40:02.794466+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_clean_scrub_stamp":"2023-05-22T15:40:02.364399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:39:40.293308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":10,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.770545+0000","last_change":"2023-05-22T15:40:02.423063+0000","last_active":"2023-05-22T15:40:02.770545+0000","last_peered":"2023-05-22T15:40:02.770545+0000","last_clean":"2023-05-22T15:40:02.770545+0000","last_became_active":"2023-05-22T15:40:02.416443+0000","last_became_peered":"2023-05-22T15:40:02.416443+0000","last_unstale":"2023-05-22T15:40:02.770545+0000","last_undegraded":"2023-05-22T15:40:02.770545+0000","last_fullsized":"2023-05-22T15:40:02.770545+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:40:01.360691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:40:01.360691+0000","last_clean_scrub_stamp":"2023-05-22T15:40:01.360691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:24:40.404955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.793210+0000","last_change":"2023-05-22T15:40:02.791761+0000","last_active":"2023-05-22T15:40:02.793210+0000","last_peered":"2023-05-22T15:40:02.793210+0000","last_clean":"2023-05-22T15:40:02.793210+0000","last_became_active":"2023-05-22T15:40:02.791487+0000","last_became_peered":"2023-05-22T15:40:02.791487+0000","last_unstale":"2023-05-22T15:40:02.793210+0000","last_undegraded":"2023-05-22T15:40:02.793210+0000","last_fullsized":"2023-05-22T15:40:02.793210+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_clean_scrub_stamp":"2023-05-22T15:40:02.364399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:12:27.685858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":10,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.773828+0000","last_change":"2023-05-22T15:40:02.408599+0000","last_active":"2023-05-22T15:40:02.773828+0000","last_peered":"2023-05-22T15:40:02.773828+0000","last_clean":"2023-05-22T15:40:02.773828+0000","last_became_active":"2023-05-22T15:40:02.408247+0000","last_became_peered":"2023-05-22T15:40:02.408247+0000","last_unstale":"2023-05-22T15:40:02.773828+0000","last_undegraded":"2023-05-22T15:40:02.773828+0000","last_fullsized":"2023-05-22T15:40:02.773828+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:40:01.360691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:40:01.360691+0000","last_clean_scrub_stamp":"2023-05-22T15:40:01.360691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T16:53:02.734735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.833704+0000","last_change":"2023-05-22T15:40:02.824182+0000","last_active":"2023-05-22T15:40:02.833704+0000","last_peered":"2023-05-22T15:40:02.833704+0000","last_clean":"2023-05-22T15:40:02.833704+0000","last_became_active":"2023-05-22T15:40:02.822848+0000","last_became_peered":"2023-05-22T15:40:02.822848+0000","last_unstale":"2023-05-22T15:40:02.833704+0000","last_undegraded":"2023-05-22T15:40:02.833704+0000","last_fullsized":"2023-05-22T15:40:02.833704+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_clean_scrub_stamp":"2023-05-22T15:40:02.364399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:02:47.354363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":10,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.783230+0000","last_change":"2023-05-22T15:40:02.415326+0000","last_active":"2023-05-22T15:40:02.783230+0000","last_peered":"2023-05-22T15:40:02.783230+0000","last_clean":"2023-05-22T15:40:02.783230+0000","last_became_active":"2023-05-22T15:40:02.414400+0000","last_became_peered":"2023-05-22T15:40:02.414400+0000","last_unstale":"2023-05-22T15:40:02.783230+0000","last_undegraded":"2023-05-22T15:40:02.783230+0000","last_fullsized":"2023-05-22T15:40:02.783230+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:40:01.360691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:40:01.360691+0000","last_clean_scrub_stamp":"2023-05-22T15:40:01.360691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:37:20.385027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.798622+0000","last_change":"2023-05-22T15:40:02.797293+0000","last_active":"2023-05-22T15:40:02.798622+0000","last_peered":"2023-05-22T15:40:02.798622+0000","last_clean":"2023-05-22T15:40:02.798622+0000","last_became_active":"2023-05-22T15:40:02.796428+0000","last_became_peered":"2023-05-22T15:40:02.796428+0000","last_unstale":"2023-05-22T15:40:02.798622+0000","last_undegraded":"2023-05-22T15:40:02.798622+0000","last_fullsized":"2023-05-22T15:40:02.798622+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_clean_scrub_stamp":"2023-05-22T15:40:02.364399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T18:12:13.680113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":10,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.774828+0000","last_change":"2023-05-22T15:40:02.405292+0000","last_active":"2023-05-22T15:40:02.774828+0000","last_peered":"2023-05-22T15:40:02.774828+0000","last_clean":"2023-05-22T15:40:02.774828+0000","last_became_active":"2023-05-22T15:40:02.404828+0000","last_became_peered":"2023-05-22T15:40:02.404828+0000","last_unstale":"2023-05-22T15:40:02.774828+0000","last_undegraded":"2023-05-22T15:40:02.774828+0000","last_fullsized":"2023-05-22T15:40:02.774828+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:40:01.360691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:40:01.360691+0000","last_clean_scrub_stamp":"2023-05-22T15:40:01.360691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T16:27:28.680714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.810848+0000","last_change":"2023-05-22T15:40:02.809663+0000","last_active":"2023-05-22T15:40:02.810848+0000","last_peered":"2023-05-22T15:40:02.810848+0000","last_clean":"2023-05-22T15:40:02.810848+0000","last_became_active":"2023-05-22T15:40:02.809256+0000","last_became_peered":"2023-05-22T15:40:02.809256+0000","last_unstale":"2023-05-22T15:40:02.810848+0000","last_undegraded":"2023-05-22T15:40:02.810848+0000","last_fullsized":"2023-05-22T15:40:02.810848+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_clean_scrub_stamp":"2023-05-22T15:40:02.364399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T18:23:56.809553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":10,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.765035+0000","last_change":"2023-05-22T15:40:02.408908+0000","last_active":"2023-05-22T15:40:02.765035+0000","last_peered":"2023-05-22T15:40:02.765035+0000","last_clean":"2023-05-22T15:40:02.765035+0000","last_became_active":"2023-05-22T15:40:02.408259+0000","last_became_peered":"2023-05-22T15:40:02.408259+0000","last_unstale":"2023-05-22T15:40:02.765035+0000","last_undegraded":"2023-05-22T15:40:02.765035+0000","last_fullsized":"2023-05-22T15:40:02.765035+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:40:01.360691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:40:01.360691+0000","last_clean_scrub_stamp":"2023-05-22T15:40:01.360691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:02:46.350654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.786075+0000","last_change":"2023-05-22T15:40:02.784711+0000","last_active":"2023-05-22T15:40:02.786075+0000","last_peered":"2023-05-22T15:40:02.786075+0000","last_clean":"2023-05-22T15:40:02.786075+0000","last_became_active":"2023-05-22T15:40:02.784244+0000","last_became_peered":"2023-05-22T15:40:02.784244+0000","last_unstale":"2023-05-22T15:40:02.786075+0000","last_undegraded":"2023-05-22T15:40:02.786075+0000","last_fullsized":"2023-05-22T15:40:02.786075+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_clean_scrub_stamp":"2023-05-22T15:40:02.364399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:26:16.725005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":10,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.790989+0000","last_change":"2023-05-22T15:40:02.420259+0000","last_active":"2023-05-22T15:40:02.790989+0000","last_peered":"2023-05-22T15:40:02.790989+0000","last_clean":"2023-05-22T15:40:02.790989+0000","last_became_active":"2023-05-22T15:40:02.419445+0000","last_became_peered":"2023-05-22T15:40:02.419445+0000","last_unstale":"2023-05-22T15:40:02.790989+0000","last_undegraded":"2023-05-22T15:40:02.790989+0000","last_fullsized":"2023-05-22T15:40:02.790989+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:40:01.360691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:40:01.360691+0000","last_clean_scrub_stamp":"2023-05-22T15:40:01.360691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:59:56.458175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"60'139","reported_seq":295,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.406615+0000","last_change":"2023-05-22T15:26:39.318204+0000","last_active":"2023-05-22T15:40:02.406615+0000","last_peered":"2023-05-22T15:40:02.406615+0000","last_clean":"2023-05-22T15:40:02.406615+0000","last_became_active":"2023-05-22T15:26:39.315370+0000","last_became_peered":"2023-05-22T15:26:39.315370+0000","last_unstale":"2023-05-22T15:40:02.406615+0000","last_undegraded":"2023-05-22T15:40:02.406615+0000","last_fullsized":"2023-05-22T15:40:02.406615+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:38.020076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:38.020076+0000","last_clean_scrub_stamp":"2023-05-22T15:26:38.020076+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:06:39.207543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":308,"num_read_kb":512,"num_write":199,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":66,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.382130+0000","last_change":"2023-05-22T15:26:44.971951+0000","last_active":"2023-05-22T15:40:02.382130+0000","last_peered":"2023-05-22T15:40:02.382130+0000","last_clean":"2023-05-22T15:40:02.382130+0000","last_became_active":"2023-05-22T15:26:44.970872+0000","last_became_peered":"2023-05-22T15:26:44.970872+0000","last_unstale":"2023-05-22T15:40:02.382130+0000","last_undegraded":"2023-05-22T15:40:02.382130+0000","last_fullsized":"2023-05-22T15:40:02.382130+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:55:34.752949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.797718+0000","last_change":"2023-05-22T15:40:02.796011+0000","last_active":"2023-05-22T15:40:02.797718+0000","last_peered":"2023-05-22T15:40:02.797718+0000","last_clean":"2023-05-22T15:40:02.797718+0000","last_became_active":"2023-05-22T15:40:02.795707+0000","last_became_peered":"2023-05-22T15:40:02.795707+0000","last_unstale":"2023-05-22T15:40:02.797718+0000","last_undegraded":"2023-05-22T15:40:02.797718+0000","last_fullsized":"2023-05-22T15:40:02.797718+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_clean_scrub_stamp":"2023-05-22T15:40:02.364399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T18:27:29.359577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":10,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.781593+0000","last_change":"2023-05-22T15:40:02.401638+0000","last_active":"2023-05-22T15:40:02.781593+0000","last_peered":"2023-05-22T15:40:02.781593+0000","last_clean":"2023-05-22T15:40:02.781593+0000","last_became_active":"2023-05-22T15:40:02.401306+0000","last_became_peered":"2023-05-22T15:40:02.401306+0000","last_unstale":"2023-05-22T15:40:02.781593+0000","last_undegraded":"2023-05-22T15:40:02.781593+0000","last_fullsized":"2023-05-22T15:40:02.781593+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:40:01.360691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:40:01.360691+0000","last_clean_scrub_stamp":"2023-05-22T15:40:01.360691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:43:27.054935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":64,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.406393+0000","last_change":"2023-05-22T15:26:44.977198+0000","last_active":"2023-05-22T15:40:02.406393+0000","last_peered":"2023-05-22T15:40:02.406393+0000","last_clean":"2023-05-22T15:40:02.406393+0000","last_became_active":"2023-05-22T15:26:44.976445+0000","last_became_peered":"2023-05-22T15:26:44.976445+0000","last_unstale":"2023-05-22T15:40:02.406393+0000","last_undegraded":"2023-05-22T15:40:02.406393+0000","last_fullsized":"2023-05-22T15:40:02.406393+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:12:41.464054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.813500+0000","last_change":"2023-05-22T15:40:02.792455+0000","last_active":"2023-05-22T15:40:02.813500+0000","last_peered":"2023-05-22T15:40:02.813500+0000","last_clean":"2023-05-22T15:40:02.813500+0000","last_became_active":"2023-05-22T15:40:02.791720+0000","last_became_peered":"2023-05-22T15:40:02.791720+0000","last_unstale":"2023-05-22T15:40:02.813500+0000","last_undegraded":"2023-05-22T15:40:02.813500+0000","last_fullsized":"2023-05-22T15:40:02.813500+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_clean_scrub_stamp":"2023-05-22T15:40:02.364399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:02:47.354363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.b","version":"72'2","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.837665+0000","last_change":"2023-05-22T15:40:02.411102+0000","last_active":"2023-05-22T15:40:02.837665+0000","last_peered":"2023-05-22T15:40:02.837665+0000","last_clean":"2023-05-22T15:40:02.837665+0000","last_became_active":"2023-05-22T15:40:02.410649+0000","last_became_peered":"2023-05-22T15:40:02.410649+0000","last_unstale":"2023-05-22T15:40:02.837665+0000","last_undegraded":"2023-05-22T15:40:02.837665+0000","last_fullsized":"2023-05-22T15:40:02.837665+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:40:01.360691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:40:01.360691+0000","last_clean_scrub_stamp":"2023-05-22T15:40:01.360691+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:45:01.491989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":65,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.374837+0000","last_change":"2023-05-22T15:26:44.969469+0000","last_active":"2023-05-22T15:40:02.374837+0000","last_peered":"2023-05-22T15:40:02.374837+0000","last_clean":"2023-05-22T15:40:02.374837+0000","last_became_active":"2023-05-22T15:26:44.969083+0000","last_became_peered":"2023-05-22T15:26:44.969083+0000","last_unstale":"2023-05-22T15:40:02.374837+0000","last_undegraded":"2023-05-22T15:40:02.374837+0000","last_fullsized":"2023-05-22T15:40:02.374837+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:29:23.670067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.795213+0000","last_change":"2023-05-22T15:40:02.786028+0000","last_active":"2023-05-22T15:40:02.795213+0000","last_peered":"2023-05-22T15:40:02.795213+0000","last_clean":"2023-05-22T15:40:02.795213+0000","last_became_active":"2023-05-22T15:40:02.785742+0000","last_became_peered":"2023-05-22T15:40:02.785742+0000","last_unstale":"2023-05-22T15:40:02.795213+0000","last_undegraded":"2023-05-22T15:40:02.795213+0000","last_fullsized":"2023-05-22T15:40:02.795213+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_clean_scrub_stamp":"2023-05-22T15:40:02.364399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:56:01.718556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":10,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.769763+0000","last_change":"2023-05-22T15:40:02.397757+0000","last_active":"2023-05-22T15:40:02.769763+0000","last_peered":"2023-05-22T15:40:02.769763+0000","last_clean":"2023-05-22T15:40:02.769763+0000","last_became_active":"2023-05-22T15:40:02.395637+0000","last_became_peered":"2023-05-22T15:40:02.395637+0000","last_unstale":"2023-05-22T15:40:02.769763+0000","last_undegraded":"2023-05-22T15:40:02.769763+0000","last_fullsized":"2023-05-22T15:40:02.769763+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:40:01.360691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:40:01.360691+0000","last_clean_scrub_stamp":"2023-05-22T15:40:01.360691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:15:59.197371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":66,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.381931+0000","last_change":"2023-05-22T15:26:44.976632+0000","last_active":"2023-05-22T15:40:02.381931+0000","last_peered":"2023-05-22T15:40:02.381931+0000","last_clean":"2023-05-22T15:40:02.381931+0000","last_became_active":"2023-05-22T15:26:44.975546+0000","last_became_peered":"2023-05-22T15:26:44.975546+0000","last_unstale":"2023-05-22T15:40:02.381931+0000","last_undegraded":"2023-05-22T15:40:02.381931+0000","last_fullsized":"2023-05-22T15:40:02.381931+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:42:57.973030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.813543+0000","last_change":"2023-05-22T15:40:02.794726+0000","last_active":"2023-05-22T15:40:02.813543+0000","last_peered":"2023-05-22T15:40:02.813543+0000","last_clean":"2023-05-22T15:40:02.813543+0000","last_became_active":"2023-05-22T15:40:02.794009+0000","last_became_peered":"2023-05-22T15:40:02.794009+0000","last_unstale":"2023-05-22T15:40:02.813543+0000","last_undegraded":"2023-05-22T15:40:02.813543+0000","last_fullsized":"2023-05-22T15:40:02.813543+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_clean_scrub_stamp":"2023-05-22T15:40:02.364399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T18:55:07.687839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.e","version":"72'2","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.795400+0000","last_change":"2023-05-22T15:40:02.401649+0000","last_active":"2023-05-22T15:40:02.795400+0000","last_peered":"2023-05-22T15:40:02.795400+0000","last_clean":"2023-05-22T15:40:02.795400+0000","last_became_active":"2023-05-22T15:40:02.401266+0000","last_became_peered":"2023-05-22T15:40:02.401266+0000","last_unstale":"2023-05-22T15:40:02.795400+0000","last_undegraded":"2023-05-22T15:40:02.795400+0000","last_fullsized":"2023-05-22T15:40:02.795400+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:40:01.360691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:40:01.360691+0000","last_clean_scrub_stamp":"2023-05-22T15:40:01.360691+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:03:38.544683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":60,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.395423+0000","last_change":"2023-05-22T15:26:45.091475+0000","last_active":"2023-05-22T15:40:02.395423+0000","last_peered":"2023-05-22T15:40:02.395423+0000","last_clean":"2023-05-22T15:40:02.395423+0000","last_became_active":"2023-05-22T15:26:45.091143+0000","last_became_peered":"2023-05-22T15:26:45.091143+0000","last_unstale":"2023-05-22T15:40:02.395423+0000","last_undegraded":"2023-05-22T15:40:02.395423+0000","last_fullsized":"2023-05-22T15:40:02.395423+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:12:41.464054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.841762+0000","last_change":"2023-05-22T15:40:02.840221+0000","last_active":"2023-05-22T15:40:02.841762+0000","last_peered":"2023-05-22T15:40:02.841762+0000","last_clean":"2023-05-22T15:40:02.841762+0000","last_became_active":"2023-05-22T15:40:02.839649+0000","last_became_peered":"2023-05-22T15:40:02.839649+0000","last_unstale":"2023-05-22T15:40:02.841762+0000","last_undegraded":"2023-05-22T15:40:02.841762+0000","last_fullsized":"2023-05-22T15:40:02.841762+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_clean_scrub_stamp":"2023-05-22T15:40:02.364399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:49:45.980298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.f","version":"72'1","reported_seq":11,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.790522+0000","last_change":"2023-05-22T15:40:02.401451+0000","last_active":"2023-05-22T15:40:02.790522+0000","last_peered":"2023-05-22T15:40:02.790522+0000","last_clean":"2023-05-22T15:40:02.790522+0000","last_became_active":"2023-05-22T15:40:02.400950+0000","last_became_peered":"2023-05-22T15:40:02.400950+0000","last_unstale":"2023-05-22T15:40:02.790522+0000","last_undegraded":"2023-05-22T15:40:02.790522+0000","last_fullsized":"2023-05-22T15:40:02.790522+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:40:01.360691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:40:01.360691+0000","last_clean_scrub_stamp":"2023-05-22T15:40:01.360691+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:32:28.812012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":64,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.411885+0000","last_change":"2023-05-22T15:26:45.092367+0000","last_active":"2023-05-22T15:40:02.411885+0000","last_peered":"2023-05-22T15:40:02.411885+0000","last_clean":"2023-05-22T15:40:02.411885+0000","last_became_active":"2023-05-22T15:26:45.091951+0000","last_became_peered":"2023-05-22T15:26:45.091951+0000","last_unstale":"2023-05-22T15:40:02.411885+0000","last_undegraded":"2023-05-22T15:40:02.411885+0000","last_fullsized":"2023-05-22T15:40:02.411885+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:42:57.973030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.837196+0000","last_change":"2023-05-22T15:40:02.831422+0000","last_active":"2023-05-22T15:40:02.837196+0000","last_peered":"2023-05-22T15:40:02.837196+0000","last_clean":"2023-05-22T15:40:02.837196+0000","last_became_active":"2023-05-22T15:40:02.830698+0000","last_became_peered":"2023-05-22T15:40:02.830698+0000","last_unstale":"2023-05-22T15:40:02.837196+0000","last_undegraded":"2023-05-22T15:40:02.837196+0000","last_fullsized":"2023-05-22T15:40:02.837196+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_clean_scrub_stamp":"2023-05-22T15:40:02.364399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:45:02.495698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":10,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.763194+0000","last_change":"2023-05-22T15:40:02.396758+0000","last_active":"2023-05-22T15:40:02.763194+0000","last_peered":"2023-05-22T15:40:02.763194+0000","last_clean":"2023-05-22T15:40:02.763194+0000","last_became_active":"2023-05-22T15:40:02.396137+0000","last_became_peered":"2023-05-22T15:40:02.396137+0000","last_unstale":"2023-05-22T15:40:02.763194+0000","last_undegraded":"2023-05-22T15:40:02.763194+0000","last_fullsized":"2023-05-22T15:40:02.763194+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:40:01.360691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:40:01.360691+0000","last_clean_scrub_stamp":"2023-05-22T15:40:01.360691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:45:01.491989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":60,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.395737+0000","last_change":"2023-05-22T15:26:44.978858+0000","last_active":"2023-05-22T15:40:02.395737+0000","last_peered":"2023-05-22T15:40:02.395737+0000","last_clean":"2023-05-22T15:40:02.395737+0000","last_became_active":"2023-05-22T15:26:44.978443+0000","last_became_peered":"2023-05-22T15:26:44.978443+0000","last_unstale":"2023-05-22T15:40:02.395737+0000","last_undegraded":"2023-05-22T15:40:02.395737+0000","last_fullsized":"2023-05-22T15:40:02.395737+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:46:29.290243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.823471+0000","last_change":"2023-05-22T15:40:02.821765+0000","last_active":"2023-05-22T15:40:02.823471+0000","last_peered":"2023-05-22T15:40:02.823471+0000","last_clean":"2023-05-22T15:40:02.823471+0000","last_became_active":"2023-05-22T15:40:02.821558+0000","last_became_peered":"2023-05-22T15:40:02.821558+0000","last_unstale":"2023-05-22T15:40:02.823471+0000","last_undegraded":"2023-05-22T15:40:02.823471+0000","last_fullsized":"2023-05-22T15:40:02.823471+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_clean_scrub_stamp":"2023-05-22T15:40:02.364399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T18:27:29.359577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"72'1","reported_seq":11,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.818846+0000","last_change":"2023-05-22T15:40:02.422583+0000","last_active":"2023-05-22T15:40:02.818846+0000","last_peered":"2023-05-22T15:40:02.818846+0000","last_clean":"2023-05-22T15:40:02.818846+0000","last_became_active":"2023-05-22T15:40:02.418357+0000","last_became_peered":"2023-05-22T15:40:02.418357+0000","last_unstale":"2023-05-22T15:40:02.818846+0000","last_undegraded":"2023-05-22T15:40:02.818846+0000","last_fullsized":"2023-05-22T15:40:02.818846+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:40:01.360691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:40:01.360691+0000","last_clean_scrub_stamp":"2023-05-22T15:40:01.360691+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T18:44:35.502420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":61,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.403978+0000","last_change":"2023-05-22T15:26:44.982421+0000","last_active":"2023-05-22T15:40:02.403978+0000","last_peered":"2023-05-22T15:40:02.403978+0000","last_clean":"2023-05-22T15:40:02.403978+0000","last_became_active":"2023-05-22T15:26:44.981586+0000","last_became_peered":"2023-05-22T15:26:44.981586+0000","last_unstale":"2023-05-22T15:40:02.403978+0000","last_undegraded":"2023-05-22T15:40:02.403978+0000","last_fullsized":"2023-05-22T15:40:02.403978+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:14:11.156544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":8,"num_pg":32,"stat_sum":{"num_bytes":2330,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":308,"num_read_kb":512,"num_write":199,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575012,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28736,"kb_used_data":1400,"kb_used_omap":0,"kb_used_meta":27328,"kb_avail":94343104,"statfs":{"total":96636764160,"available":96607338496,"internally_reserved":0,"allocated":1433600,"data_stored":1022295,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27983872},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 15:39:57 2023","interfaces":[{"interface":"back","average":{"1min":0.84399999999999997,"5min":0.86799999999999999,"15min":0.86499999999999999},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.40899999999999997},"max":{"1min":1.2150000000000001,"5min":1.359,"15min":1.7709999999999999},"last":0.88400000000000001},{"interface":"front","average":{"1min":0.77700000000000002,"5min":0.88800000000000001,"15min":0.86599999999999999},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":1.2090000000000001,"5min":1.3169999999999999,"15min":1.4239999999999999},"last":1.1539999999999999}]},{"osd":1,"last update":"Mon May 22 15:39:57 2023","interfaces":[{"interface":"back","average":{"1min":0.92300000000000004,"5min":0.97999999999999998,"15min":0.98299999999999998},"min":{"1min":0.5,"5min":0.5,"15min":0.5},"max":{"1min":1.3240000000000001,"5min":1.389,"15min":1.8300000000000001},"last":0.92800000000000005},{"interface":"front","average":{"1min":0.95099999999999996,"5min":1.036,"15min":1.0189999999999999},"min":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":1.3169999999999999,"5min":1.337,"15min":2.161},"last":1.081}]},{"osd":2,"last update":"Mon May 22 15:39:57 2023","interfaces":[{"interface":"back","average":{"1min":0.81599999999999995,"5min":0.879,"15min":0.90200000000000002},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.46800000000000003},"max":{"1min":1.1160000000000001,"5min":1.244,"15min":6.7279999999999998},"last":0.82599999999999996},{"interface":"front","average":{"1min":0.88800000000000001,"5min":0.96699999999999997,"15min":0.95999999999999996},"min":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.51000000000000001},"max":{"1min":1.177,"5min":1.331,"15min":6.8810000000000002},"last":0.70199999999999996}]},{"osd":3,"last update":"Mon May 22 15:39:57 2023","interfaces":[{"interface":"back","average":{"1min":0.92100000000000004,"5min":0.95199999999999996,"15min":0.96499999999999997},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.53800000000000003},"max":{"1min":1.3360000000000001,"5min":1.5580000000000001,"15min":3.0569999999999999},"last":1.0149999999999999},{"interface":"front","average":{"1min":0.95099999999999996,"5min":0.96999999999999997,"15min":0.995},"min":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.54900000000000004},"max":{"1min":1.264,"5min":1.268,"15min":1.6879999999999999},"last":1.0369999999999999}]},{"osd":4,"last update":"Mon May 22 15:39:57 2023","interfaces":[{"interface":"back","average":{"1min":0.87,"5min":0.879,"15min":0.89000000000000001},"min":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"max":{"1min":1.621,"5min":1.621,"15min":1.621},"last":0.95599999999999996},{"interface":"front","average":{"1min":0.98899999999999999,"5min":0.98599999999999999,"15min":1},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.4319999999999999,"5min":1.7689999999999999,"15min":1.958},"last":1.131}]},{"osd":5,"last update":"Mon May 22 15:39:57 2023","interfaces":[{"interface":"back","average":{"1min":1.0229999999999999,"5min":0.999,"15min":1.0189999999999999},"min":{"1min":0.68300000000000005,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":1.409,"5min":1.5029999999999999,"15min":1.5029999999999999},"last":0.99199999999999999},{"interface":"front","average":{"1min":1.071,"5min":1.111,"15min":1.1379999999999999},"min":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":1.5229999999999999,"5min":1.5229999999999999,"15min":5.0839999999999996},"last":1.1080000000000001}]},{"osd":6,"last update":"Mon May 22 15:39:57 2023","interfaces":[{"interface":"back","average":{"1min":1.004,"5min":1.048,"15min":1.0800000000000001},"min":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"max":{"1min":1.484,"5min":1.484,"15min":5.1719999999999997},"last":1.0600000000000001},{"interface":"front","average":{"1min":1.1259999999999999,"5min":1.133,"15min":1.145},"min":{"1min":0.73699999999999999,"5min":0.72199999999999998,"15min":0.72199999999999998},"max":{"1min":1.702,"5min":1.702,"15min":2.04},"last":1.171}]}]},{"osd":6,"up_from":13,"seq":55834575013,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29076,"kb_used_data":844,"kb_used_omap":0,"kb_used_meta":28224,"kb_avail":94342764,"statfs":{"total":96636764160,"available":96606990336,"internally_reserved":0,"allocated":864256,"data_stored":446315,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":28901376},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 15:39:02 2023","interfaces":[{"interface":"back","average":{"1min":0.92300000000000004,"5min":0.94399999999999995,"15min":0.89500000000000002},"min":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.53900000000000003},"max":{"1min":1.327,"5min":1.3380000000000001,"15min":1.355},"last":1.0329999999999999},{"interface":"front","average":{"1min":0.93999999999999995,"5min":0.88,"15min":0.92300000000000004},"min":{"1min":0.626,"5min":0.626,"15min":0.57399999999999995},"max":{"1min":1.4159999999999999,"5min":1.4159999999999999,"15min":1.4159999999999999},"last":1.5680000000000001}]},{"osd":1,"last update":"Mon May 22 15:39:02 2023","interfaces":[{"interface":"back","average":{"1min":0.95299999999999996,"5min":0.88800000000000001,"15min":0.90100000000000002},"min":{"1min":0.66200000000000003,"5min":0.621,"15min":0.53500000000000003},"max":{"1min":1.3169999999999999,"5min":1.3169999999999999,"15min":1.375},"last":0.92300000000000004},{"interface":"front","average":{"1min":0.96799999999999997,"5min":0.96599999999999997,"15min":0.94499999999999995},"min":{"1min":0.67800000000000005,"5min":0.63200000000000001,"15min":0.53600000000000003},"max":{"1min":1.2789999999999999,"5min":1.2789999999999999,"15min":1.325},"last":1.51}]},{"osd":2,"last update":"Mon May 22 15:39:02 2023","interfaces":[{"interface":"back","average":{"1min":0.99299999999999999,"5min":0.997,"15min":0.96799999999999997},"min":{"1min":0.83699999999999997,"5min":0.64700000000000002,"15min":0.64500000000000002},"max":{"1min":1.2030000000000001,"5min":1.2210000000000001,"15min":1.3220000000000001},"last":1.425},{"interface":"front","average":{"1min":1.0449999999999999,"5min":1.048,"15min":1.052},"min":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.57699999999999996},"max":{"1min":1.2390000000000001,"5min":1.3129999999999999,"15min":1.4930000000000001},"last":1.3120000000000001}]},{"osd":3,"last update":"Mon May 22 15:39:02 2023","interfaces":[{"interface":"back","average":{"1min":0.995,"5min":0.97799999999999998,"15min":0.98799999999999999},"min":{"1min":0.71799999999999997,"5min":0.68700000000000006,"15min":0.626},"max":{"1min":1.2330000000000001,"5min":1.2330000000000001,"15min":1.4430000000000001},"last":1.385},{"interface":"front","average":{"1min":0.98999999999999999,"5min":0.95799999999999996,"15min":0.96099999999999997},"min":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.56499999999999995},"max":{"1min":1.2549999999999999,"5min":1.2549999999999999,"15min":1.486},"last":1.2629999999999999}]},{"osd":4,"last update":"Mon May 22 15:39:02 2023","interfaces":[{"interface":"back","average":{"1min":0.87,"5min":0.872,"15min":0.94899999999999995},"min":{"1min":0.61599999999999999,"5min":0.59399999999999997,"15min":0.57599999999999996},"max":{"1min":1.3959999999999999,"5min":1.3959999999999999,"15min":1.6459999999999999},"last":1.171},{"interface":"front","average":{"1min":1.056,"5min":1.02,"15min":1.0600000000000001},"min":{"1min":0.71999999999999997,"5min":0.61299999999999999,"15min":0.61299999999999999},"max":{"1min":1.347,"5min":1.347,"15min":1.5469999999999999},"last":1.6120000000000001}]},{"osd":5,"last update":"Mon May 22 15:39:02 2023","interfaces":[{"interface":"back","average":{"1min":1.0129999999999999,"5min":0.96999999999999997,"15min":0.996},"min":{"1min":0.71599999999999997,"5min":0.64400000000000002,"15min":0.64400000000000002},"max":{"1min":1.401,"5min":1.401,"15min":1.621},"last":1.3460000000000001},{"interface":"front","average":{"1min":1.0680000000000001,"5min":1.0720000000000001,"15min":1.105},"min":{"1min":0.74399999999999999,"5min":0.68700000000000006,"15min":0.68700000000000006},"max":{"1min":1.4610000000000001,"5min":1.4610000000000001,"15min":1.7509999999999999},"last":1.4670000000000001}]},{"osd":7,"last update":"Mon May 22 15:39:02 2023","interfaces":[{"interface":"back","average":{"1min":1.099,"5min":1.1120000000000001,"15min":1.165},"min":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.58399999999999996},"max":{"1min":1.47,"5min":1.492,"15min":1.8120000000000001},"last":1.6890000000000001},{"interface":"front","average":{"1min":1.139,"5min":1.1240000000000001,"15min":1.141},"min":{"1min":0.80200000000000005,"5min":0.75,"15min":0.68999999999999995},"max":{"1min":1.2809999999999999,"5min":1.4259999999999999,"15min":1.6819999999999999},"last":1.716}]}]},{"osd":5,"up_from":13,"seq":55834575013,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":22588,"kb_used_data":820,"kb_used_omap":0,"kb_used_meta":21760,"kb_avail":94349252,"statfs":{"total":96636764160,"available":96613634048,"internally_reserved":0,"allocated":839680,"data_stored":431927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":22282240},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 15:40:02 2023","interfaces":[{"interface":"back","average":{"1min":0.76700000000000002,"5min":0.85899999999999999,"15min":0.85899999999999999},"min":{"1min":0.54900000000000004,"5min":0.54000000000000004,"15min":0.48399999999999999},"max":{"1min":1.3440000000000001,"5min":1.3440000000000001,"15min":1.3440000000000001},"last":1.3440000000000001},{"interface":"front","average":{"1min":0.91600000000000004,"5min":0.89900000000000002,"15min":0.86699999999999999},"min":{"1min":0.65400000000000003,"5min":0.497,"15min":0.497},"max":{"1min":2.1699999999999999,"5min":2.1699999999999999,"15min":2.1699999999999999},"last":0.83299999999999996}]},{"osd":1,"last update":"Mon May 22 15:40:02 2023","interfaces":[{"interface":"back","average":{"1min":0.91100000000000003,"5min":0.99099999999999999,"15min":0.97699999999999998},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":1.9370000000000001,"5min":1.9370000000000001,"15min":1.9370000000000001},"last":1.169},{"interface":"front","average":{"1min":0.97099999999999997,"5min":1.0269999999999999,"15min":1.0149999999999999},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.2350000000000001,"5min":1.284,"15min":1.399},"last":1.1359999999999999}]},{"osd":2,"last update":"Mon May 22 15:40:02 2023","interfaces":[{"interface":"back","average":{"1min":0.98999999999999999,"5min":0.95799999999999996,"15min":0.92500000000000004},"min":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":1.9830000000000001,"5min":1.9830000000000001,"15min":1.9830000000000001},"last":1.0129999999999999},{"interface":"front","average":{"1min":0.95299999999999996,"5min":0.97399999999999998,"15min":0.92100000000000004},"min":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.53800000000000003},"max":{"1min":2.1120000000000001,"5min":2.1120000000000001,"15min":2.1120000000000001},"last":0.72399999999999998}]},{"osd":3,"last update":"Mon May 22 15:40:02 2023","interfaces":[{"interface":"back","average":{"1min":1.032,"5min":1.002,"15min":0.95499999999999996},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.4419999999999999,"5min":1.4419999999999999,"15min":1.6839999999999999},"last":1.252},{"interface":"front","average":{"1min":1.175,"5min":1.069,"15min":1.04},"min":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.58299999999999996},"max":{"1min":2.52,"5min":2.52,"15min":2.52},"last":1.2130000000000001}]},{"osd":4,"last update":"Mon May 22 15:40:02 2023","interfaces":[{"interface":"back","average":{"1min":1.113,"5min":0.96399999999999997,"15min":0.93999999999999995},"min":{"1min":0.73399999999999999,"5min":0.60599999999999998,"15min":0.57599999999999996},"max":{"1min":2.226,"5min":2.226,"15min":2.226},"last":0.93999999999999995},{"interface":"front","average":{"1min":1.1659999999999999,"5min":1.054,"15min":1.0389999999999999},"min":{"1min":0.67800000000000005,"5min":0.629,"15min":0.622},"max":{"1min":1.6859999999999999,"5min":1.6859999999999999,"15min":1.6859999999999999},"last":1.4079999999999999}]},{"osd":6,"last update":"Mon May 22 15:40:02 2023","interfaces":[{"interface":"back","average":{"1min":1.194,"5min":1.079,"15min":1.0529999999999999},"min":{"1min":0.67900000000000005,"5min":0.65500000000000003,"15min":0.61299999999999999},"max":{"1min":2.831,"5min":2.831,"15min":2.831},"last":1.101},{"interface":"front","average":{"1min":1.1919999999999999,"5min":1.1220000000000001,"15min":1.081},"min":{"1min":0.85199999999999998,"5min":0.76000000000000001,"15min":0.69299999999999995},"max":{"1min":2.3690000000000002,"5min":2.3690000000000002,"15min":2.3690000000000002},"last":1.0669999999999999}]},{"osd":7,"last update":"Mon May 22 15:40:02 2023","interfaces":[{"interface":"back","average":{"1min":1.2729999999999999,"5min":1.175,"15min":1.1439999999999999},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":1.849,"5min":1.849,"15min":2.8260000000000001},"last":1.4550000000000001},{"interface":"front","average":{"1min":1.323,"5min":1.2070000000000001,"15min":1.169},"min":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.71699999999999997},"max":{"1min":2.7429999999999999,"5min":2.7429999999999999,"15min":2.7429999999999999},"last":1.3120000000000001}]}]},{"osd":4,"up_from":13,"seq":55834575011,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":24700,"kb_used_data":820,"kb_used_omap":0,"kb_used_meta":23872,"kb_avail":94347140,"statfs":{"total":96636764160,"available":96611471360,"internally_reserved":0,"allocated":839680,"data_stored":431927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":24444928},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 15:39:01 2023","interfaces":[{"interface":"back","average":{"1min":0.99199999999999999,"5min":0.93300000000000005,"15min":0.89700000000000002},"min":{"1min":0.64800000000000002,"5min":0.56699999999999995,"15min":0.51400000000000001},"max":{"1min":1.2370000000000001,"5min":1.3049999999999999,"15min":1.657},"last":1.016},{"interface":"front","average":{"1min":0.86199999999999999,"5min":0.93799999999999994,"15min":0.91900000000000004},"min":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.51800000000000002},"max":{"1min":1.1259999999999999,"5min":1.2430000000000001,"15min":1.633},"last":0.66600000000000004}]},{"osd":1,"last update":"Mon May 22 15:39:01 2023","interfaces":[{"interface":"back","average":{"1min":1.004,"5min":0.98799999999999999,"15min":0.95299999999999996},"min":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.54000000000000004},"max":{"1min":1.383,"5min":1.383,"15min":1.3859999999999999},"last":1.2},{"interface":"front","average":{"1min":0.93300000000000005,"5min":0.93899999999999995,"15min":0.89500000000000002},"min":{"1min":0.68999999999999995,"5min":0.66300000000000003,"15min":0.51600000000000001},"max":{"1min":1.3120000000000001,"5min":1.3640000000000001,"15min":1.3640000000000001},"last":1.298}]},{"osd":2,"last update":"Mon May 22 15:39:01 2023","interfaces":[{"interface":"back","average":{"1min":1.0309999999999999,"5min":0.96099999999999997,"15min":0.95299999999999996},"min":{"1min":0.77400000000000002,"5min":0.67100000000000004,"15min":0.59299999999999997},"max":{"1min":1.4199999999999999,"5min":1.4199999999999999,"15min":1.6950000000000001},"last":1.2370000000000001},{"interface":"front","average":{"1min":1.016,"5min":1.0029999999999999,"15min":1.0249999999999999},"min":{"1min":0.80300000000000005,"5min":0.63800000000000001,"15min":0.57199999999999995},"max":{"1min":1.325,"5min":1.361,"15min":1.9770000000000001},"last":0.79100000000000004}]},{"osd":3,"last update":"Mon May 22 15:39:01 2023","interfaces":[{"interface":"back","average":{"1min":0.97099999999999997,"5min":0.94899999999999995,"15min":0.94999999999999996},"min":{"1min":0.78600000000000003,"5min":0.67900000000000005,"15min":0.54600000000000004},"max":{"1min":1.222,"5min":1.3169999999999999,"15min":1.665},"last":1.1499999999999999},{"interface":"front","average":{"1min":1.0860000000000001,"5min":1.05,"15min":1.05},"min":{"1min":0.77400000000000002,"5min":0.747,"15min":0.64200000000000002},"max":{"1min":1.3440000000000001,"5min":1.3480000000000001,"15min":1.538},"last":0.874}]},{"osd":5,"last update":"Mon May 22 15:39:01 2023","interfaces":[{"interface":"back","average":{"1min":1.0369999999999999,"5min":0.96499999999999997,"15min":1.018},"min":{"1min":0.65200000000000002,"5min":0.52700000000000002,"15min":0.51200000000000001},"max":{"1min":1.3979999999999999,"5min":1.3979999999999999,"15min":1.7270000000000001},"last":1.1060000000000001},{"interface":"front","average":{"1min":0.98699999999999999,"5min":0.99099999999999999,"15min":0.98899999999999999},"min":{"1min":0.71199999999999997,"5min":0.66400000000000003,"15min":0.626},"max":{"1min":1.266,"5min":1.4139999999999999,"15min":1.6719999999999999},"last":0.97899999999999998}]},{"osd":6,"last update":"Mon May 22 15:39:01 2023","interfaces":[{"interface":"back","average":{"1min":1.0680000000000001,"5min":0.98299999999999998,"15min":1.034},"min":{"1min":0.67500000000000004,"5min":0.623,"15min":0.58099999999999996},"max":{"1min":1.288,"5min":1.3280000000000001,"15min":1.6020000000000001},"last":0.748},{"interface":"front","average":{"1min":1.1279999999999999,"5min":1.117,"15min":1.1259999999999999},"min":{"1min":0.81200000000000006,"5min":0.70899999999999996,"15min":0.69799999999999995},"max":{"1min":1.4430000000000001,"5min":1.4430000000000001,"15min":2.0699999999999998},"last":0.93200000000000005}]},{"osd":7,"last update":"Mon May 22 15:39:01 2023","interfaces":[{"interface":"back","average":{"1min":1.0760000000000001,"5min":1.042,"15min":1.093},"min":{"1min":0.76200000000000001,"5min":0.71899999999999997,"15min":0.67700000000000005},"max":{"1min":1.4870000000000001,"5min":1.4870000000000001,"15min":1.6639999999999999},"last":0.82999999999999996},{"interface":"front","average":{"1min":1.1579999999999999,"5min":1.143,"15min":1.157},"min":{"1min":0.86099999999999999,"5min":0.71499999999999997,"15min":0.71499999999999997},"max":{"1min":1.5720000000000001,"5min":1.5720000000000001,"15min":1.7410000000000001},"last":1.329}]}]},{"osd":3,"up_from":13,"seq":55834575011,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29756,"kb_used_data":820,"kb_used_omap":0,"kb_used_meta":28928,"kb_avail":94342084,"statfs":{"total":96636764160,"available":96606294016,"internally_reserved":0,"allocated":839680,"data_stored":431927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":29622272},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 15:39:48 2023","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.64000000000000001,"15min":0.61399999999999999},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":0.91700000000000004,"5min":1.1739999999999999,"15min":1.1739999999999999},"last":0.26300000000000001},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.72199999999999998,"15min":0.67300000000000004},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.23699999999999999},"max":{"1min":1.1799999999999999,"5min":1.1799999999999999,"15min":1.296},"last":0.374}]},{"osd":1,"last update":"Mon May 22 15:39:48 2023","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.65900000000000003,"15min":0.65100000000000002},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.92900000000000005,"5min":1.0509999999999999,"15min":1.0600000000000001},"last":0.32900000000000001},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.79600000000000004,"15min":0.75},"min":{"1min":0.313,"5min":0.313,"15min":0.216},"max":{"1min":0.875,"5min":1.155,"15min":1.6579999999999999},"last":0.42199999999999999}]},{"osd":2,"last update":"Mon May 22 15:39:48 2023","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.82799999999999996,"15min":0.79400000000000004},"min":{"1min":0.374,"5min":0.374,"15min":0.32200000000000001},"max":{"1min":1.111,"5min":1.2989999999999999,"15min":1.2989999999999999},"last":0.36599999999999999},{"interface":"front","average":{"1min":0.70299999999999996,"5min":0.82799999999999996,"15min":0.80000000000000004},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.29699999999999999},"max":{"1min":0.98099999999999998,"5min":1.359,"15min":1.855},"last":0.40000000000000002}]},{"osd":4,"last update":"Mon May 22 15:39:48 2023","interfaces":[{"interface":"back","average":{"1min":0.85699999999999998,"5min":0.94999999999999996,"15min":0.94699999999999995},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":1.069,"5min":1.2310000000000001,"15min":1.5820000000000001},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.91300000000000003,"5min":1.0329999999999999,"15min":1.008},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.3740000000000001,"5min":1.3740000000000001,"15min":1.8069999999999999},"last":0.85099999999999998}]},{"osd":5,"last update":"Mon May 22 15:39:48 2023","interfaces":[{"interface":"back","average":{"1min":0.88900000000000001,"5min":1.024,"15min":0.98399999999999999},"min":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":1.3149999999999999,"5min":1.4339999999999999,"15min":1.4339999999999999},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.93700000000000006,"5min":1.0509999999999999,"15min":1.052},"min":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":1.1459999999999999,"5min":1.4359999999999999,"15min":2.4820000000000002},"last":0.55900000000000005}]},{"osd":6,"last update":"Mon May 22 15:39:48 2023","interfaces":[{"interface":"back","average":{"1min":0.89100000000000001,"5min":1.048,"15min":1.022},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.4139999999999999,"5min":1.4139999999999999,"15min":1.863},"last":0.60499999999999998},{"interface":"front","average":{"1min":1.0289999999999999,"5min":1.095,"15min":1.0880000000000001},"min":{"1min":0.749,"5min":0.749,"15min":0.53500000000000003},"max":{"1min":1.28,"5min":1.5009999999999999,"15min":2.6589999999999998},"last":0.67300000000000004}]},{"osd":7,"last update":"Mon May 22 15:39:48 2023","interfaces":[{"interface":"back","average":{"1min":1.0169999999999999,"5min":1.113,"15min":1.121},"min":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":1.369,"5min":1.5429999999999999,"15min":2.762},"last":0.69699999999999995},{"interface":"front","average":{"1min":0.99199999999999999,"5min":1.0880000000000001,"15min":1.0760000000000001},"min":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":1.24,"5min":1.4079999999999999,"15min":1.714},"last":0.61899999999999999}]}]},{"osd":2,"up_from":13,"seq":55834575012,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":16060,"kb_used_data":820,"kb_used_omap":0,"kb_used_meta":15232,"kb_avail":94355780,"statfs":{"total":96636764160,"available":96620318720,"internally_reserved":0,"allocated":839680,"data_stored":431927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":15597568},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 15:39:53 2023","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.66400000000000003,"15min":0.64100000000000001},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.317},"max":{"1min":1.8240000000000001,"5min":1.8240000000000001,"15min":1.8240000000000001},"last":0.623},{"interface":"front","average":{"1min":0.80900000000000005,"5min":0.78200000000000003,"15min":0.73099999999999998},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.253},"max":{"1min":1.8160000000000001,"5min":1.8160000000000001,"15min":1.8160000000000001},"last":0.76100000000000001}]},{"osd":1,"last update":"Mon May 22 15:39:53 2023","interfaces":[{"interface":"back","average":{"1min":0.72199999999999998,"5min":0.751,"15min":0.72999999999999998},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.26900000000000002},"max":{"1min":0.91600000000000004,"5min":1.1950000000000001,"15min":1.5309999999999999},"last":0.72499999999999998},{"interface":"front","average":{"1min":0.88300000000000001,"5min":0.84699999999999998,"15min":0.78400000000000003},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.29099999999999998},"max":{"1min":2.5089999999999999,"5min":2.5089999999999999,"15min":2.5089999999999999},"last":1.141}]},{"osd":3,"last update":"Mon May 22 15:39:53 2023","interfaces":[{"interface":"back","average":{"1min":0.84299999999999997,"5min":0.745,"15min":0.71399999999999997},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.26700000000000002},"max":{"1min":2.7629999999999999,"5min":2.7629999999999999,"15min":2.7629999999999999},"last":0.878},{"interface":"front","average":{"1min":0.96799999999999997,"5min":0.91600000000000004,"15min":0.85899999999999999},"min":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.33000000000000002},"max":{"1min":2.6840000000000002,"5min":2.6840000000000002,"15min":2.6840000000000002},"last":0.93799999999999994}]},{"osd":4,"last update":"Mon May 22 15:39:53 2023","interfaces":[{"interface":"back","average":{"1min":1.0900000000000001,"5min":1.038,"15min":0.999},"min":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.55500000000000005},"max":{"1min":2.8039999999999998,"5min":2.8039999999999998,"15min":2.8039999999999998},"last":1.054},{"interface":"front","average":{"1min":1.089,"5min":1.0529999999999999,"15min":1.02},"min":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":2.0819999999999999,"5min":2.0819999999999999,"15min":2.0819999999999999},"last":1.2789999999999999}]},{"osd":5,"last update":"Mon May 22 15:39:53 2023","interfaces":[{"interface":"back","average":{"1min":1.125,"5min":1.0649999999999999,"15min":1.034},"min":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.55200000000000005},"max":{"1min":2.3860000000000001,"5min":2.3860000000000001,"15min":2.4569999999999999},"last":0.99199999999999999},{"interface":"front","average":{"1min":1.109,"5min":1.032,"15min":1},"min":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.56200000000000006},"max":{"1min":3.0299999999999998,"5min":3.0299999999999998,"15min":3.0299999999999998},"last":1.0900000000000001}]},{"osd":6,"last update":"Mon May 22 15:39:53 2023","interfaces":[{"interface":"back","average":{"1min":1.1659999999999999,"5min":1.0640000000000001,"15min":1.0529999999999999},"min":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.56399999999999995},"max":{"1min":2.883,"5min":2.883,"15min":3.488},"last":1.2390000000000001},{"interface":"front","average":{"1min":1.2190000000000001,"5min":1.1259999999999999,"15min":1.1100000000000001},"min":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.53400000000000003},"max":{"1min":3.1459999999999999,"5min":3.1459999999999999,"15min":3.7330000000000001},"last":1.1910000000000001}]},{"osd":7,"last update":"Mon May 22 15:39:53 2023","interfaces":[{"interface":"back","average":{"1min":1.2370000000000001,"5min":1.1599999999999999,"15min":1.119},"min":{"1min":0.84399999999999997,"5min":0.77000000000000002,"15min":0.58299999999999996},"max":{"1min":2.5990000000000002,"5min":2.5990000000000002,"15min":2.5990000000000002},"last":1.361},{"interface":"front","average":{"1min":1.246,"5min":1.1559999999999999,"15min":1.0940000000000001},"min":{"1min":0.84599999999999997,"5min":0.77200000000000002,"15min":0.54000000000000004},"max":{"1min":3.8799999999999999,"5min":3.8799999999999999,"15min":3.8799999999999999},"last":1.339}]}]},{"osd":1,"up_from":13,"seq":55834575012,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":22908,"kb_used_data":820,"kb_used_omap":0,"kb_used_meta":22080,"kb_avail":94348932,"statfs":{"total":96636764160,"available":96613306368,"internally_reserved":0,"allocated":839680,"data_stored":431927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":22609920},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 15:39:05 2023","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.63900000000000001,"15min":0.63400000000000001},"min":{"1min":0.52000000000000002,"5min":0.439,"15min":0.28799999999999998},"max":{"1min":0.91800000000000004,"5min":1.0569999999999999,"15min":1.7689999999999999},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.80200000000000005,"5min":0.80500000000000005,"15min":0.75700000000000001},"min":{"1min":0.56299999999999994,"5min":0.49099999999999999,"15min":0.36399999999999999},"max":{"1min":1.046,"5min":1.1439999999999999,"15min":1.3799999999999999},"last":0.64600000000000002}]},{"osd":2,"last update":"Mon May 22 15:39:05 2023","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":0.74399999999999999,"15min":0.70499999999999996},"min":{"1min":0.60599999999999998,"5min":0.38600000000000001,"15min":0.34100000000000003},"max":{"1min":1.1060000000000001,"5min":1.1060000000000001,"15min":1.7290000000000001},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.79100000000000004,"15min":0.72299999999999998},"min":{"1min":0.53900000000000003,"5min":0.50800000000000001,"15min":0.27800000000000002},"max":{"1min":1.2170000000000001,"5min":1.2629999999999999,"15min":2.4049999999999998},"last":0.40600000000000003}]},{"osd":3,"last update":"Mon May 22 15:39:05 2023","interfaces":[{"interface":"back","average":{"1min":0.86099999999999999,"5min":0.85699999999999998,"15min":0.81200000000000006},"min":{"1min":0.63400000000000001,"5min":0.59599999999999997,"15min":0.317},"max":{"1min":1.1519999999999999,"5min":1.2450000000000001,"15min":1.5069999999999999},"last":0.435},{"interface":"front","average":{"1min":0.84299999999999997,"5min":0.82199999999999995,"15min":0.79800000000000004},"min":{"1min":0.65600000000000003,"5min":0.58399999999999996,"15min":0.315},"max":{"1min":1.1859999999999999,"5min":1.1970000000000001,"15min":7.5620000000000003},"last":0.55600000000000005}]},{"osd":4,"last update":"Mon May 22 15:39:05 2023","interfaces":[{"interface":"back","average":{"1min":1.0089999999999999,"5min":0.96399999999999997,"15min":0.93000000000000005},"min":{"1min":0.749,"5min":0.63700000000000001,"15min":0.54200000000000004},"max":{"1min":1.383,"5min":1.486,"15min":1.8169999999999999},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.98899999999999999,"5min":1.014,"15min":0.97599999999999998},"min":{"1min":0.80500000000000005,"5min":0.79800000000000004,"15min":0.54300000000000004},"max":{"1min":1.206,"5min":1.3520000000000001,"15min":1.3520000000000001},"last":0.58999999999999997}]},{"osd":5,"last update":"Mon May 22 15:39:05 2023","interfaces":[{"interface":"back","average":{"1min":1.071,"5min":1.0669999999999999,"15min":1.0189999999999999},"min":{"1min":0.76800000000000002,"5min":0.73099999999999998,"15min":0.59499999999999997},"max":{"1min":1.244,"5min":1.395,"15min":1.7170000000000001},"last":0.83799999999999997},{"interface":"front","average":{"1min":1.0640000000000001,"5min":1.0589999999999999,"15min":1.0329999999999999},"min":{"1min":0.78500000000000003,"5min":0.70999999999999996,"15min":0.59199999999999997},"max":{"1min":1.6850000000000001,"5min":1.6850000000000001,"15min":4.4710000000000001},"last":0.60799999999999998}]},{"osd":6,"last update":"Mon May 22 15:39:05 2023","interfaces":[{"interface":"back","average":{"1min":1.04,"5min":1.0589999999999999,"15min":1.056},"min":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.63800000000000001},"max":{"1min":1.262,"5min":1.5469999999999999,"15min":14.968999999999999},"last":0.627},{"interface":"front","average":{"1min":1.0800000000000001,"5min":1.117,"15min":1.1419999999999999},"min":{"1min":0.90000000000000002,"5min":0.85199999999999998,"15min":0.56699999999999995},"max":{"1min":1.3859999999999999,"5min":1.5269999999999999,"15min":13.242000000000001},"last":0.73599999999999999}]},{"osd":7,"last update":"Mon May 22 15:39:05 2023","interfaces":[{"interface":"back","average":{"1min":1.127,"5min":1.125,"15min":1.111},"min":{"1min":0.79700000000000004,"5min":0.747,"15min":0.63},"max":{"1min":1.306,"5min":1.4730000000000001,"15min":1.9259999999999999},"last":0.69599999999999995},{"interface":"front","average":{"1min":1.1779999999999999,"5min":1.139,"15min":1.111},"min":{"1min":0.86899999999999999,"5min":0.82399999999999995,"15min":0.46400000000000002},"max":{"1min":1.8600000000000001,"5min":1.8600000000000001,"15min":5.681},"last":0.79900000000000004}]}]},{"osd":0,"up_from":13,"seq":55834575013,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":22056,"kb_used_data":1440,"kb_used_omap":0,"kb_used_meta":20608,"kb_avail":94349784,"statfs":{"total":96636764160,"available":96614178816,"internally_reserved":0,"allocated":1474560,"data_stored":1038797,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":21102592},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon May 22 15:39:01 2023","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.69599999999999995,"15min":0.65200000000000002},"min":{"1min":0.51400000000000001,"5min":0.51100000000000001,"15min":0.24399999999999999},"max":{"1min":1,"5min":1.125,"15min":1.3049999999999999},"last":0.79100000000000004},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.747,"15min":0.72399999999999998},"min":{"1min":0.54700000000000004,"5min":0.52000000000000002,"15min":0.26200000000000001},"max":{"1min":1.0189999999999999,"5min":1.2390000000000001,"15min":1.2390000000000001},"last":0.745}]},{"osd":2,"last update":"Mon May 22 15:39:01 2023","interfaces":[{"interface":"back","average":{"1min":0.74099999999999999,"5min":0.76200000000000001,"15min":0.72699999999999998},"min":{"1min":0.57599999999999996,"5min":0.54700000000000004,"15min":0.28399999999999997},"max":{"1min":1.145,"5min":1.145,"15min":1.4059999999999999},"last":0.91600000000000004},{"interface":"front","average":{"1min":0.80900000000000005,"5min":0.82499999999999996,"15min":0.79300000000000004},"min":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.29699999999999999},"max":{"1min":1.1850000000000001,"5min":1.2030000000000001,"15min":1.3999999999999999},"last":0.94699999999999995}]},{"osd":3,"last update":"Mon May 22 15:39:01 2023","interfaces":[{"interface":"back","average":{"1min":0.75,"5min":0.75800000000000001,"15min":0.73999999999999999},"min":{"1min":0.55100000000000005,"5min":0.43099999999999999,"15min":0.36599999999999999},"max":{"1min":1.0660000000000001,"5min":1.1579999999999999,"15min":1.28},"last":0.872},{"interface":"front","average":{"1min":0.90300000000000002,"5min":0.89300000000000002,"15min":0.84399999999999997},"min":{"1min":0.73299999999999998,"5min":0.61899999999999999,"15min":0.33000000000000002},"max":{"1min":1.3500000000000001,"5min":1.3660000000000001,"15min":1.4570000000000001},"last":1.0920000000000001}]},{"osd":4,"last update":"Mon May 22 15:39:01 2023","interfaces":[{"interface":"back","average":{"1min":0.95699999999999996,"5min":0.97099999999999997,"15min":0.93200000000000005},"min":{"1min":0.76800000000000002,"5min":0.75900000000000001,"15min":0.53600000000000003},"max":{"1min":1.2529999999999999,"5min":1.4790000000000001,"15min":1.696},"last":1.0629999999999999},{"interface":"front","average":{"1min":1.0329999999999999,"5min":1.0449999999999999,"15min":0.999},"min":{"1min":0.85299999999999998,"5min":0.79100000000000004,"15min":0.54900000000000004},"max":{"1min":1.2370000000000001,"5min":1.5089999999999999,"15min":1.6020000000000001},"last":1.1519999999999999}]},{"osd":5,"last update":"Mon May 22 15:39:01 2023","interfaces":[{"interface":"back","average":{"1min":1.0880000000000001,"5min":1.0740000000000001,"15min":1.0429999999999999},"min":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.56699999999999995},"max":{"1min":1.458,"5min":1.5229999999999999,"15min":1.6850000000000001},"last":1.038},{"interface":"front","average":{"1min":1.0209999999999999,"5min":1.0209999999999999,"15min":0.98999999999999999},"min":{"1min":0.80000000000000004,"5min":0.64700000000000002,"15min":0.54800000000000004},"max":{"1min":1.3400000000000001,"5min":1.4019999999999999,"15min":1.4510000000000001},"last":1.1200000000000001}]},{"osd":6,"last update":"Mon May 22 15:39:01 2023","interfaces":[{"interface":"back","average":{"1min":1.135,"5min":1.1140000000000001,"15min":1.0529999999999999},"min":{"1min":0.89500000000000002,"5min":0.82199999999999995,"15min":0.61099999999999999},"max":{"1min":1.651,"5min":1.669,"15min":1.7450000000000001},"last":1.357},{"interface":"front","average":{"1min":1.137,"5min":1.119,"15min":1.0680000000000001},"min":{"1min":0.88300000000000001,"5min":0.875,"15min":0.54200000000000004},"max":{"1min":1.514,"5min":1.7070000000000001,"15min":1.7070000000000001},"last":1.3080000000000001}]},{"osd":7,"last update":"Mon May 22 15:39:01 2023","interfaces":[{"interface":"back","average":{"1min":1.1659999999999999,"5min":1.1479999999999999,"15min":1.107},"min":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.56499999999999995},"max":{"1min":1.5580000000000001,"5min":1.5620000000000001,"15min":1.629},"last":1.1919999999999999},{"interface":"front","average":{"1min":1.1040000000000001,"5min":1.107,"15min":1.0620000000000001},"min":{"1min":0.91300000000000003,"5min":0.70399999999999996,"15min":0.58399999999999996},"max":{"1min":1.377,"5min":1.478,"15min":1.5469999999999999},"last":1.3799999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2114,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-05-22T15:40:06.820 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-05-22T15:40:06.982 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.978+0000 7f0d64573640 1 Processor -- start 2023-05-22T15:40:06.982 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.978+0000 7f0d64573640 1 -- start start 2023-05-22T15:40:06.983 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.978+0000 7f0d64573640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f0d5c0ca4e0 0x7f0d5c15e460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:40:06.983 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.978+0000 7f0d64573640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0d5c0c9b40 0x7f0d5c0afb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:40:06.983 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.978+0000 7f0d64573640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d5c0b0650 con 0x7f0d5c070df0 2023-05-22T15:40:06.984 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.978+0000 7f0d64573640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d5c0b0370 con 0x7f0d5c0ca4e0 2023-05-22T15:40:06.984 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.978+0000 7f0d64573640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d5c15d980 con 0x7f0d5c0c9b40 2023-05-22T15:40:06.984 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.978+0000 7f0d61b0e640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0d5c0c9b40 0x7f0d5c0afb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:40:06.985 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.978+0000 7f0d6230f640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f0d5c0ca4e0 0x7f0d5c15e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:40:06.985 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.978+0000 7f0d62b10640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f0d5c070df0 0x7f0d5c071210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.16:37348/0 (socket says 172.21.15.16:37348) 2023-05-22T15:40:06.985 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.978+0000 7f0d61b0e640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0d5c0c9b40 0x7f0d5c0afb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3301/0 says I am v2:172.21.15.16:49060/0 (socket says 172.21.15.16:49060) 2023-05-22T15:40:06.985 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.978+0000 7f0d62b10640 1 -- 172.21.15.16:0/3755798211 learned_addr learned my addr 172.21.15.16:0/3755798211 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:40:06.985 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.978+0000 7f0d61b0e640 1 -- 172.21.15.16:0/3755798211 >> v2:172.21.15.62:3300/0 conn(0x7f0d5c0ca4e0 msgr2=0x7f0d5c15e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:40:06.985 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.978+0000 7f0d61b0e640 1 --2- 172.21.15.16:0/3755798211 >> v2:172.21.15.62:3300/0 conn(0x7f0d5c0ca4e0 0x7f0d5c15e460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:06.985 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.978+0000 7f0d61b0e640 1 -- 172.21.15.16:0/3755798211 >> v1:172.21.15.62:6789/0 conn(0x7f0d5c070df0 legacy=0x7f0d5c071210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:40:06.986 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.978+0000 7f0d61b0e640 1 -- 172.21.15.16:0/3755798211 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d5c15dc60 con 0x7f0d5c0c9b40 2023-05-22T15:40:06.987 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.982+0000 7f0d61b0e640 1 --2- 172.21.15.16:0/3755798211 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0d5c0c9b40 0x7f0d5c0afb40 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f0d4c0097c0 tx=0x7f0d4c01c3c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7074a592b2738f71 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:40:06.987 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.982+0000 7f0d6130d640 1 -- 172.21.15.16:0/3755798211 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d4c0ac600 con 0x7f0d5c0c9b40 2023-05-22T15:40:06.987 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.982+0000 7f0d6130d640 1 -- 172.21.15.16:0/3755798211 <== mon.2 v2:172.21.15.62:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d4c0a8050 con 0x7f0d5c0c9b40 2023-05-22T15:40:06.987 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.982+0000 7f0d6130d640 1 -- 172.21.15.16:0/3755798211 <== mon.2 v2:172.21.15.62:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d4c0ac980 con 0x7f0d5c0c9b40 2023-05-22T15:40:06.988 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.982+0000 7f0d64573640 1 -- 172.21.15.16:0/3755798211 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0d5c0c9b40 msgr2=0x7f0d5c0afb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:40:06.988 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.982+0000 7f0d64573640 1 --2- 172.21.15.16:0/3755798211 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0d5c0c9b40 0x7f0d5c0afb40 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f0d4c0097c0 tx=0x7f0d4c01c3c0 comp rx=0 tx=0).stop 2023-05-22T15:40:06.988 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.982+0000 7f0d64573640 1 -- 172.21.15.16:0/3755798211 shutdown_connections 2023-05-22T15:40:06.989 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.982+0000 7f0d64573640 1 --2- 172.21.15.16:0/3755798211 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0d5c0c9b40 0x7f0d5c0afb40 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:06.989 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.982+0000 7f0d64573640 1 --2- 172.21.15.16:0/3755798211 >> v2:172.21.15.62:3300/0 conn(0x7f0d5c0ca4e0 0x7f0d5c15e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:06.989 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.982+0000 7f0d64573640 1 -- 172.21.15.16:0/3755798211 >> 172.21.15.16:0/3755798211 conn(0x7f0d5c069720 msgr2=0x7f0d5c069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:40:06.989 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.982+0000 7f0d64573640 1 -- 172.21.15.16:0/3755798211 shutdown_connections 2023-05-22T15:40:06.989 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.982+0000 7f0d64573640 1 -- 172.21.15.16:0/3755798211 wait complete. 2023-05-22T15:40:06.991 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.982+0000 7f0d64573640 1 Processor -- start 2023-05-22T15:40:06.992 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.982+0000 7f0d64573640 1 -- start start 2023-05-22T15:40:06.992 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.986+0000 7f0d64573640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d5c070df0 0x7f0d5c0ae800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:40:06.992 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.986+0000 7f0d64573640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0d5c0c9b40 0x7f0d5c151a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:40:06.992 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.986+0000 7f0d64573640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0d5c0ca4e0 0x7f0d5c151fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:40:06.992 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.986+0000 7f0d64573640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d5c0c5510 con 0x7f0d5c070df0 2023-05-22T15:40:06.992 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.986+0000 7f0d64573640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d5c1f8920 con 0x7f0d5c0c9b40 2023-05-22T15:40:06.992 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.986+0000 7f0d64573640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d5c15b480 con 0x7f0d5c0ca4e0 2023-05-22T15:40:06.993 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.986+0000 7f0d6230f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d5c070df0 0x7f0d5c0ae800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:40:06.993 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.986+0000 7f0d6230f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d5c070df0 0x7f0d5c0ae800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34398/0 (socket says 172.21.15.16:34398) 2023-05-22T15:40:06.993 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.986+0000 7f0d6230f640 1 -- 172.21.15.16:0/643736363 learned_addr learned my addr 172.21.15.16:0/643736363 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:40:06.993 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.986+0000 7f0d61b0e640 1 --2- 172.21.15.16:0/643736363 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0d5c0c9b40 0x7f0d5c151a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:40:06.993 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.986+0000 7f0d62b10640 1 --2- 172.21.15.16:0/643736363 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0d5c0ca4e0 0x7f0d5c151fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:40:06.993 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.986+0000 7f0d6230f640 1 -- 172.21.15.16:0/643736363 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0d5c0ca4e0 msgr2=0x7f0d5c151fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:40:06.993 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.986+0000 7f0d6230f640 1 --2- 172.21.15.16:0/643736363 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0d5c0ca4e0 0x7f0d5c151fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:06.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.986+0000 7f0d6230f640 1 -- 172.21.15.16:0/643736363 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0d5c0c9b40 msgr2=0x7f0d5c151a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:40:06.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.986+0000 7f0d6230f640 1 --2- 172.21.15.16:0/643736363 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0d5c0c9b40 0x7f0d5c151a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:06.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.986+0000 7f0d6230f640 1 -- 172.21.15.16:0/643736363 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d4c009420 con 0x7f0d5c070df0 2023-05-22T15:40:06.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.986+0000 7f0d6230f640 1 --2- 172.21.15.16:0/643736363 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d5c070df0 0x7f0d5c0ae800 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7f0d5800e870 tx=0x7f0d5800ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:40:06.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.986+0000 7f0d52ffd640 1 -- 172.21.15.16:0/643736363 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d58009510 con 0x7f0d5c070df0 2023-05-22T15:40:06.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.986+0000 7f0d52ffd640 1 -- 172.21.15.16:0/643736363 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d5800eec0 con 0x7f0d5c070df0 2023-05-22T15:40:06.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.986+0000 7f0d64573640 1 -- 172.21.15.16:0/643736363 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d5c0c8e80 con 0x7f0d5c070df0 2023-05-22T15:40:06.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.986+0000 7f0d52ffd640 1 -- 172.21.15.16:0/643736363 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d5800cb40 con 0x7f0d5c070df0 2023-05-22T15:40:06.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.986+0000 7f0d64573640 1 -- 172.21.15.16:0/643736363 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d5c0ca8b0 con 0x7f0d5c070df0 2023-05-22T15:40:06.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.986+0000 7f0d64573640 1 -- 172.21.15.16:0/643736363 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d5c0c9010 con 0x7f0d5c070df0 2023-05-22T15:40:06.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.986+0000 7f0d52ffd640 1 -- 172.21.15.16:0/643736363 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f0d5800cca0 con 0x7f0d5c070df0 2023-05-22T15:40:06.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.990+0000 7f0d52ffd640 1 --2- 172.21.15.16:0/643736363 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f0d400654d0 0x7f0d40067980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:40:06.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.990+0000 7f0d52ffd640 1 -- 172.21.15.16:0/643736363 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f0d58083370 con 0x7f0d5c070df0 2023-05-22T15:40:06.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.990+0000 7f0d61b0e640 1 --2- 172.21.15.16:0/643736363 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f0d400654d0 0x7f0d40067980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:40:06.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.990+0000 7f0d61b0e640 1 --2- 172.21.15.16:0/643736363 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f0d400654d0 0x7f0d40067980 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f0d4c0ae000 tx=0x7f0d4c008040 comp rx=0 tx=0).ready entity=mgr.6240 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:40:06.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:06.994+0000 7f0d52ffd640 1 -- 172.21.15.16:0/643736363 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f0d58050e30 con 0x7f0d5c070df0 2023-05-22T15:40:07.149 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:07.142+0000 7f0d64573640 1 -- 172.21.15.16:0/643736363 --> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f0d5c04dbe0 con 0x7f0d400654d0 2023-05-22T15:40:07.159 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:07.154+0000 7f0d52ffd640 1 -- 172.21.15.16:0/643736363 <== mgr.6240 v2:172.21.15.16:6832/36921 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+310089 (secure 0 0 0) 0x7f0d5c04dbe0 con 0x7f0d400654d0 2023-05-22T15:40:07.161 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:40:07.164 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2023-05-22T15:40:07.165 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:07.158+0000 7f0d64573640 1 -- 172.21.15.16:0/643736363 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f0d400654d0 msgr2=0x7f0d40067980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:40:07.165 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:07.158+0000 7f0d64573640 1 --2- 172.21.15.16:0/643736363 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f0d400654d0 0x7f0d40067980 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f0d4c0ae000 tx=0x7f0d4c008040 comp rx=0 tx=0).stop 2023-05-22T15:40:07.165 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:07.158+0000 7f0d64573640 1 -- 172.21.15.16:0/643736363 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d5c070df0 msgr2=0x7f0d5c0ae800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:40:07.165 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:07.158+0000 7f0d64573640 1 --2- 172.21.15.16:0/643736363 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d5c070df0 0x7f0d5c0ae800 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7f0d5800e870 tx=0x7f0d5800ec70 comp rx=0 tx=0).stop 2023-05-22T15:40:07.166 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:07.158+0000 7f0d64573640 1 -- 172.21.15.16:0/643736363 shutdown_connections 2023-05-22T15:40:07.166 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:07.158+0000 7f0d64573640 1 --2- 172.21.15.16:0/643736363 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f0d400654d0 0x7f0d40067980 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:07.166 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:07.158+0000 7f0d64573640 1 --2- 172.21.15.16:0/643736363 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0d5c0ca4e0 0x7f0d5c151fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:07.166 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:07.158+0000 7f0d64573640 1 --2- 172.21.15.16:0/643736363 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0d5c0c9b40 0x7f0d5c151a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:07.166 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:07.158+0000 7f0d64573640 1 --2- 172.21.15.16:0/643736363 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d5c070df0 0x7f0d5c0ae800 unknown :-1 s=CLOSED pgs=926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:07.166 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:07.158+0000 7f0d64573640 1 -- 172.21.15.16:0/643736363 >> 172.21.15.16:0/643736363 conn(0x7f0d5c069720 msgr2=0x7f0d5c0b1080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:40:07.166 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:07.158+0000 7f0d64573640 1 -- 172.21.15.16:0/643736363 shutdown_connections 2023-05-22T15:40:07.167 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:07.158+0000 7f0d64573640 1 -- 172.21.15.16:0/643736363 wait complete. 2023-05-22T15:40:07.182 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":26,"stamp":"2023-05-22T15:40:07.090935+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592698,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":308,"num_read_kb":512,"num_write":243,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":177,"ondisk_log_size":177,"up":210,"acting":210,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":79,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":195880,"kb_used_data":7784,"kb_used_omap":0,"kb_used_meta":188032,"kb_avail":754778840,"statfs":{"total":773094113280,"available":772893532160,"internally_reserved":0,"allocated":7970816,"data_stored":4667042,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":192544768},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":72,"apply_latency_ms":72,"commit_latency_ns":72000000,"apply_latency_ns":72000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2330,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.724812"},"pg_stats":[{"pgid":"9.3f","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.781736+0000","last_change":"2023-05-22T15:40:02.779921+0000","last_active":"2023-05-22T15:40:02.781736+0000","last_peered":"2023-05-22T15:40:02.781736+0000","last_clean":"2023-05-22T15:40:02.781736+0000","last_became_active":"2023-05-22T15:40:02.779630+0000","last_became_peered":"2023-05-22T15:40:02.779630+0000","last_unstale":"2023-05-22T15:40:02.781736+0000","last_undegraded":"2023-05-22T15:40:02.781736+0000","last_fullsized":"2023-05-22T15:40:02.781736+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_clean_scrub_stamp":"2023-05-22T15:40:02.364399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T18:28:48.622897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3e","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.793198+0000","last_change":"2023-05-22T15:40:02.791660+0000","last_active":"2023-05-22T15:40:02.793198+0000","last_peered":"2023-05-22T15:40:02.793198+0000","last_clean":"2023-05-22T15:40:02.793198+0000","last_became_active":"2023-05-22T15:40:02.791427+0000","last_became_peered":"2023-05-22T15:40:02.791427+0000","last_unstale":"2023-05-22T15:40:02.793198+0000","last_undegraded":"2023-05-22T15:40:02.793198+0000","last_fullsized":"2023-05-22T15:40:02.793198+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_clean_scrub_stamp":"2023-05-22T15:40:02.364399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T03:24:02.342328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.3d","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.840900+0000","last_change":"2023-05-22T15:40:02.839056+0000","last_active":"2023-05-22T15:40:02.840900+0000","last_peered":"2023-05-22T15:40:02.840900+0000","last_clean":"2023-05-22T15:40:02.840900+0000","last_became_active":"2023-05-22T15:40:02.838286+0000","last_became_peered":"2023-05-22T15:40:02.838286+0000","last_unstale":"2023-05-22T15:40:02.840900+0000","last_undegraded":"2023-05-22T15:40:02.840900+0000","last_fullsized":"2023-05-22T15:40:02.840900+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_clean_scrub_stamp":"2023-05-22T15:40:02.364399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:15:23.770715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.3c","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.813018+0000","last_change":"2023-05-22T15:40:02.784140+0000","last_active":"2023-05-22T15:40:02.813018+0000","last_peered":"2023-05-22T15:40:02.813018+0000","last_clean":"2023-05-22T15:40:02.813018+0000","last_became_active":"2023-05-22T15:40:02.782943+0000","last_became_peered":"2023-05-22T15:40:02.782943+0000","last_unstale":"2023-05-22T15:40:02.813018+0000","last_undegraded":"2023-05-22T15:40:02.813018+0000","last_fullsized":"2023-05-22T15:40:02.813018+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_clean_scrub_stamp":"2023-05-22T15:40:02.364399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T16:42:44.882436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3b","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.809202+0000","last_change":"2023-05-22T15:40:02.808254+0000","last_active":"2023-05-22T15:40:02.809202+0000","last_peered":"2023-05-22T15:40:02.809202+0000","last_clean":"2023-05-22T15:40:02.809202+0000","last_became_active":"2023-05-22T15:40:02.807935+0000","last_became_peered":"2023-05-22T15:40:02.807935+0000","last_unstale":"2023-05-22T15:40:02.809202+0000","last_undegraded":"2023-05-22T15:40:02.809202+0000","last_fullsized":"2023-05-22T15:40:02.809202+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_clean_scrub_stamp":"2023-05-22T15:40:02.364399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:13:12.338903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3a","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.794462+0000","last_change":"2023-05-22T15:40:02.780909+0000","last_active":"2023-05-22T15:40:02.794462+0000","last_peered":"2023-05-22T15:40:02.794462+0000","last_clean":"2023-05-22T15:40:02.794462+0000","last_became_active":"2023-05-22T15:40:02.779811+0000","last_became_peered":"2023-05-22T15:40:02.779811+0000","last_unstale":"2023-05-22T15:40:02.794462+0000","last_undegraded":"2023-05-22T15:40:02.794462+0000","last_fullsized":"2023-05-22T15:40:02.794462+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_clean_scrub_stamp":"2023-05-22T15:40:02.364399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:39:11.668037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.39","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.840421+0000","last_change":"2023-05-22T15:40:02.821670+0000","last_active":"2023-05-22T15:40:02.840421+0000","last_peered":"2023-05-22T15:40:02.840421+0000","last_clean":"2023-05-22T15:40:02.840421+0000","last_became_active":"2023-05-22T15:40:02.819555+0000","last_became_peered":"2023-05-22T15:40:02.819555+0000","last_unstale":"2023-05-22T15:40:02.840421+0000","last_undegraded":"2023-05-22T15:40:02.840421+0000","last_fullsized":"2023-05-22T15:40:02.840421+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_clean_scrub_stamp":"2023-05-22T15:40:02.364399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:49:45.980298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.38","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.820792+0000","last_change":"2023-05-22T15:40:02.819968+0000","last_active":"2023-05-22T15:40:02.820792+0000","last_peered":"2023-05-22T15:40:02.820792+0000","last_clean":"2023-05-22T15:40:02.820792+0000","last_became_active":"2023-05-22T15:40:02.819721+0000","last_became_peered":"2023-05-22T15:40:02.819721+0000","last_unstale":"2023-05-22T15:40:02.820792+0000","last_undegraded":"2023-05-22T15:40:02.820792+0000","last_fullsized":"2023-05-22T15:40:02.820792+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_clean_scrub_stamp":"2023-05-22T15:40:02.364399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:53:20.742871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.37","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.838729+0000","last_change":"2023-05-22T15:40:02.818930+0000","last_active":"2023-05-22T15:40:02.838729+0000","last_peered":"2023-05-22T15:40:02.838729+0000","last_clean":"2023-05-22T15:40:02.838729+0000","last_became_active":"2023-05-22T15:40:02.818309+0000","last_became_peered":"2023-05-22T15:40:02.818309+0000","last_unstale":"2023-05-22T15:40:02.838729+0000","last_undegraded":"2023-05-22T15:40:02.838729+0000","last_fullsized":"2023-05-22T15:40:02.838729+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_clean_scrub_stamp":"2023-05-22T15:40:02.364399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T18:55:07.687839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.36","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.811337+0000","last_change":"2023-05-22T15:40:02.810378+0000","last_active":"2023-05-22T15:40:02.811337+0000","last_peered":"2023-05-22T15:40:02.811337+0000","last_clean":"2023-05-22T15:40:02.811337+0000","last_became_active":"2023-05-22T15:40:02.810099+0000","last_became_peered":"2023-05-22T15:40:02.810099+0000","last_unstale":"2023-05-22T15:40:02.811337+0000","last_undegraded":"2023-05-22T15:40:02.811337+0000","last_fullsized":"2023-05-22T15:40:02.811337+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_clean_scrub_stamp":"2023-05-22T15:40:02.364399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:24:41.408664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.35","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.793322+0000","last_change":"2023-05-22T15:40:02.787237+0000","last_active":"2023-05-22T15:40:02.793322+0000","last_peered":"2023-05-22T15:40:02.793322+0000","last_clean":"2023-05-22T15:40:02.793322+0000","last_became_active":"2023-05-22T15:40:02.786973+0000","last_became_peered":"2023-05-22T15:40:02.786973+0000","last_unstale":"2023-05-22T15:40:02.793322+0000","last_undegraded":"2023-05-22T15:40:02.793322+0000","last_fullsized":"2023-05-22T15:40:02.793322+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_clean_scrub_stamp":"2023-05-22T15:40:02.364399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:16:00.201080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.34","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.813761+0000","last_change":"2023-05-22T15:40:02.786906+0000","last_active":"2023-05-22T15:40:02.813761+0000","last_peered":"2023-05-22T15:40:02.813761+0000","last_clean":"2023-05-22T15:40:02.813761+0000","last_became_active":"2023-05-22T15:40:02.786094+0000","last_became_peered":"2023-05-22T15:40:02.786094+0000","last_unstale":"2023-05-22T15:40:02.813761+0000","last_undegraded":"2023-05-22T15:40:02.813761+0000","last_fullsized":"2023-05-22T15:40:02.813761+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_clean_scrub_stamp":"2023-05-22T15:40:02.364399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:49:45.980298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.33","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.817249+0000","last_change":"2023-05-22T15:40:02.801624+0000","last_active":"2023-05-22T15:40:02.817249+0000","last_peered":"2023-05-22T15:40:02.817249+0000","last_clean":"2023-05-22T15:40:02.817249+0000","last_became_active":"2023-05-22T15:40:02.795684+0000","last_became_peered":"2023-05-22T15:40:02.795684+0000","last_unstale":"2023-05-22T15:40:02.817249+0000","last_undegraded":"2023-05-22T15:40:02.817249+0000","last_fullsized":"2023-05-22T15:40:02.817249+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_clean_scrub_stamp":"2023-05-22T15:40:02.364399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:08:19.275487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.32","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.837582+0000","last_change":"2023-05-22T15:40:02.818926+0000","last_active":"2023-05-22T15:40:02.837582+0000","last_peered":"2023-05-22T15:40:02.837582+0000","last_clean":"2023-05-22T15:40:02.837582+0000","last_became_active":"2023-05-22T15:40:02.818292+0000","last_became_peered":"2023-05-22T15:40:02.818292+0000","last_unstale":"2023-05-22T15:40:02.837582+0000","last_undegraded":"2023-05-22T15:40:02.837582+0000","last_fullsized":"2023-05-22T15:40:02.837582+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_clean_scrub_stamp":"2023-05-22T15:40:02.364399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:32:29.815721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.31","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.839069+0000","last_change":"2023-05-22T15:40:02.820283+0000","last_active":"2023-05-22T15:40:02.839069+0000","last_peered":"2023-05-22T15:40:02.839069+0000","last_clean":"2023-05-22T15:40:02.839069+0000","last_became_active":"2023-05-22T15:40:02.818679+0000","last_became_peered":"2023-05-22T15:40:02.818679+0000","last_unstale":"2023-05-22T15:40:02.839069+0000","last_undegraded":"2023-05-22T15:40:02.839069+0000","last_fullsized":"2023-05-22T15:40:02.839069+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_clean_scrub_stamp":"2023-05-22T15:40:02.364399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:59:22.265916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.30","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.817210+0000","last_change":"2023-05-22T15:40:02.816563+0000","last_active":"2023-05-22T15:40:02.817210+0000","last_peered":"2023-05-22T15:40:02.817210+0000","last_clean":"2023-05-22T15:40:02.817210+0000","last_became_active":"2023-05-22T15:40:02.816313+0000","last_became_peered":"2023-05-22T15:40:02.816313+0000","last_unstale":"2023-05-22T15:40:02.817210+0000","last_undegraded":"2023-05-22T15:40:02.817210+0000","last_fullsized":"2023-05-22T15:40:02.817210+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_clean_scrub_stamp":"2023-05-22T15:40:02.364399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:26:16.725005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.2f","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.811365+0000","last_change":"2023-05-22T15:40:02.810702+0000","last_active":"2023-05-22T15:40:02.811365+0000","last_peered":"2023-05-22T15:40:02.811365+0000","last_clean":"2023-05-22T15:40:02.811365+0000","last_became_active":"2023-05-22T15:40:02.810432+0000","last_became_peered":"2023-05-22T15:40:02.810432+0000","last_unstale":"2023-05-22T15:40:02.811365+0000","last_undegraded":"2023-05-22T15:40:02.811365+0000","last_fullsized":"2023-05-22T15:40:02.811365+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_clean_scrub_stamp":"2023-05-22T15:40:02.364399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:28:29.223431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2e","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.837775+0000","last_change":"2023-05-22T15:40:02.831752+0000","last_active":"2023-05-22T15:40:02.837775+0000","last_peered":"2023-05-22T15:40:02.837775+0000","last_clean":"2023-05-22T15:40:02.837775+0000","last_became_active":"2023-05-22T15:40:02.830649+0000","last_became_peered":"2023-05-22T15:40:02.830649+0000","last_unstale":"2023-05-22T15:40:02.837775+0000","last_undegraded":"2023-05-22T15:40:02.837775+0000","last_fullsized":"2023-05-22T15:40:02.837775+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_clean_scrub_stamp":"2023-05-22T15:40:02.364399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:34:11.612573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2d","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.780081+0000","last_change":"2023-05-22T15:40:02.778871+0000","last_active":"2023-05-22T15:40:02.780081+0000","last_peered":"2023-05-22T15:40:02.780081+0000","last_clean":"2023-05-22T15:40:02.780081+0000","last_became_active":"2023-05-22T15:40:02.778557+0000","last_became_peered":"2023-05-22T15:40:02.778557+0000","last_unstale":"2023-05-22T15:40:02.780081+0000","last_undegraded":"2023-05-22T15:40:02.780081+0000","last_fullsized":"2023-05-22T15:40:02.780081+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_clean_scrub_stamp":"2023-05-22T15:40:02.364399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:31:50.868086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.2c","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.784146+0000","last_change":"2023-05-22T15:40:02.782901+0000","last_active":"2023-05-22T15:40:02.784146+0000","last_peered":"2023-05-22T15:40:02.784146+0000","last_clean":"2023-05-22T15:40:02.784146+0000","last_became_active":"2023-05-22T15:40:02.781896+0000","last_became_peered":"2023-05-22T15:40:02.781896+0000","last_unstale":"2023-05-22T15:40:02.784146+0000","last_undegraded":"2023-05-22T15:40:02.784146+0000","last_fullsized":"2023-05-22T15:40:02.784146+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_clean_scrub_stamp":"2023-05-22T15:40:02.364399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:56:01.718556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2b","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.836924+0000","last_change":"2023-05-22T15:40:02.829751+0000","last_active":"2023-05-22T15:40:02.836924+0000","last_peered":"2023-05-22T15:40:02.836924+0000","last_clean":"2023-05-22T15:40:02.836924+0000","last_became_active":"2023-05-22T15:40:02.829265+0000","last_became_peered":"2023-05-22T15:40:02.829265+0000","last_unstale":"2023-05-22T15:40:02.836924+0000","last_undegraded":"2023-05-22T15:40:02.836924+0000","last_fullsized":"2023-05-22T15:40:02.836924+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_clean_scrub_stamp":"2023-05-22T15:40:02.364399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:03:39.548392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2a","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.785380+0000","last_change":"2023-05-22T15:40:02.782405+0000","last_active":"2023-05-22T15:40:02.785380+0000","last_peered":"2023-05-22T15:40:02.785380+0000","last_clean":"2023-05-22T15:40:02.785380+0000","last_became_active":"2023-05-22T15:40:02.782132+0000","last_became_peered":"2023-05-22T15:40:02.782132+0000","last_unstale":"2023-05-22T15:40:02.785380+0000","last_undegraded":"2023-05-22T15:40:02.785380+0000","last_fullsized":"2023-05-22T15:40:02.785380+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_clean_scrub_stamp":"2023-05-22T15:40:02.364399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:18:45.607512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.29","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.830432+0000","last_change":"2023-05-22T15:40:02.818884+0000","last_active":"2023-05-22T15:40:02.830432+0000","last_peered":"2023-05-22T15:40:02.830432+0000","last_clean":"2023-05-22T15:40:02.830432+0000","last_became_active":"2023-05-22T15:40:02.818270+0000","last_became_peered":"2023-05-22T15:40:02.818270+0000","last_unstale":"2023-05-22T15:40:02.830432+0000","last_undegraded":"2023-05-22T15:40:02.830432+0000","last_fullsized":"2023-05-22T15:40:02.830432+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_clean_scrub_stamp":"2023-05-22T15:40:02.364399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T18:55:07.687839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.28","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.824976+0000","last_change":"2023-05-22T15:40:02.815134+0000","last_active":"2023-05-22T15:40:02.824976+0000","last_peered":"2023-05-22T15:40:02.824976+0000","last_clean":"2023-05-22T15:40:02.824976+0000","last_became_active":"2023-05-22T15:40:02.809339+0000","last_became_peered":"2023-05-22T15:40:02.809339+0000","last_unstale":"2023-05-22T15:40:02.824976+0000","last_undegraded":"2023-05-22T15:40:02.824976+0000","last_fullsized":"2023-05-22T15:40:02.824976+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_clean_scrub_stamp":"2023-05-22T15:40:02.364399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:35:22.712578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.27","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.794070+0000","last_change":"2023-05-22T15:40:02.791708+0000","last_active":"2023-05-22T15:40:02.794070+0000","last_peered":"2023-05-22T15:40:02.794070+0000","last_clean":"2023-05-22T15:40:02.794070+0000","last_became_active":"2023-05-22T15:40:02.791426+0000","last_became_peered":"2023-05-22T15:40:02.791426+0000","last_unstale":"2023-05-22T15:40:02.794070+0000","last_undegraded":"2023-05-22T15:40:02.794070+0000","last_fullsized":"2023-05-22T15:40:02.794070+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_clean_scrub_stamp":"2023-05-22T15:40:02.364399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:56:03.869179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.26","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.827059+0000","last_change":"2023-05-22T15:40:02.820844+0000","last_active":"2023-05-22T15:40:02.827059+0000","last_peered":"2023-05-22T15:40:02.827059+0000","last_clean":"2023-05-22T15:40:02.827059+0000","last_became_active":"2023-05-22T15:40:02.820069+0000","last_became_peered":"2023-05-22T15:40:02.820069+0000","last_unstale":"2023-05-22T15:40:02.827059+0000","last_undegraded":"2023-05-22T15:40:02.827059+0000","last_fullsized":"2023-05-22T15:40:02.827059+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_clean_scrub_stamp":"2023-05-22T15:40:02.364399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:32:29.815721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.25","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.811522+0000","last_change":"2023-05-22T15:40:02.809736+0000","last_active":"2023-05-22T15:40:02.811522+0000","last_peered":"2023-05-22T15:40:02.811522+0000","last_clean":"2023-05-22T15:40:02.811522+0000","last_became_active":"2023-05-22T15:40:02.809472+0000","last_became_peered":"2023-05-22T15:40:02.809472+0000","last_unstale":"2023-05-22T15:40:02.811522+0000","last_undegraded":"2023-05-22T15:40:02.811522+0000","last_fullsized":"2023-05-22T15:40:02.811522+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_clean_scrub_stamp":"2023-05-22T15:40:02.364399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T18:16:05.626894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.24","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.809191+0000","last_change":"2023-05-22T15:40:02.808246+0000","last_active":"2023-05-22T15:40:02.809191+0000","last_peered":"2023-05-22T15:40:02.809191+0000","last_clean":"2023-05-22T15:40:02.809191+0000","last_became_active":"2023-05-22T15:40:02.807971+0000","last_became_peered":"2023-05-22T15:40:02.807971+0000","last_unstale":"2023-05-22T15:40:02.809191+0000","last_undegraded":"2023-05-22T15:40:02.809191+0000","last_fullsized":"2023-05-22T15:40:02.809191+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_clean_scrub_stamp":"2023-05-22T15:40:02.364399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:05:21.391829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.23","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.778713+0000","last_change":"2023-05-22T15:40:02.776218+0000","last_active":"2023-05-22T15:40:02.778713+0000","last_peered":"2023-05-22T15:40:02.778713+0000","last_clean":"2023-05-22T15:40:02.778713+0000","last_became_active":"2023-05-22T15:40:02.775924+0000","last_became_peered":"2023-05-22T15:40:02.775924+0000","last_unstale":"2023-05-22T15:40:02.778713+0000","last_undegraded":"2023-05-22T15:40:02.778713+0000","last_fullsized":"2023-05-22T15:40:02.778713+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_clean_scrub_stamp":"2023-05-22T15:40:02.364399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T18:16:05.626894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.22","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.841758+0000","last_change":"2023-05-22T15:40:02.840087+0000","last_active":"2023-05-22T15:40:02.841758+0000","last_peered":"2023-05-22T15:40:02.841758+0000","last_clean":"2023-05-22T15:40:02.841758+0000","last_became_active":"2023-05-22T15:40:02.838145+0000","last_became_peered":"2023-05-22T15:40:02.838145+0000","last_unstale":"2023-05-22T15:40:02.841758+0000","last_undegraded":"2023-05-22T15:40:02.841758+0000","last_fullsized":"2023-05-22T15:40:02.841758+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_clean_scrub_stamp":"2023-05-22T15:40:02.364399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T16:42:44.882436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.21","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.790952+0000","last_change":"2023-05-22T15:40:02.788115+0000","last_active":"2023-05-22T15:40:02.790952+0000","last_peered":"2023-05-22T15:40:02.790952+0000","last_clean":"2023-05-22T15:40:02.790952+0000","last_became_active":"2023-05-22T15:40:02.787838+0000","last_became_peered":"2023-05-22T15:40:02.787838+0000","last_unstale":"2023-05-22T15:40:02.790952+0000","last_undegraded":"2023-05-22T15:40:02.790952+0000","last_fullsized":"2023-05-22T15:40:02.790952+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_clean_scrub_stamp":"2023-05-22T15:40:02.364399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:19:59.892493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.20","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.792780+0000","last_change":"2023-05-22T15:40:02.774767+0000","last_active":"2023-05-22T15:40:02.792780+0000","last_peered":"2023-05-22T15:40:02.792780+0000","last_clean":"2023-05-22T15:40:02.792780+0000","last_became_active":"2023-05-22T15:40:02.774127+0000","last_became_peered":"2023-05-22T15:40:02.774127+0000","last_unstale":"2023-05-22T15:40:02.792780+0000","last_undegraded":"2023-05-22T15:40:02.792780+0000","last_fullsized":"2023-05-22T15:40:02.792780+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_clean_scrub_stamp":"2023-05-22T15:40:02.364399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T03:11:31.485165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.795265+0000","last_change":"2023-05-22T15:40:02.786593+0000","last_active":"2023-05-22T15:40:02.795265+0000","last_peered":"2023-05-22T15:40:02.795265+0000","last_clean":"2023-05-22T15:40:02.795265+0000","last_became_active":"2023-05-22T15:40:02.786334+0000","last_became_peered":"2023-05-22T15:40:02.786334+0000","last_unstale":"2023-05-22T15:40:02.795265+0000","last_undegraded":"2023-05-22T15:40:02.795265+0000","last_fullsized":"2023-05-22T15:40:02.795265+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_clean_scrub_stamp":"2023-05-22T15:40:02.364399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:03:44.838990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1f","version":"72'4","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.814167+0000","last_change":"2023-05-22T15:40:02.408457+0000","last_active":"2023-05-22T15:40:02.814167+0000","last_peered":"2023-05-22T15:40:02.814167+0000","last_clean":"2023-05-22T15:40:02.814167+0000","last_became_active":"2023-05-22T15:40:02.408157+0000","last_became_peered":"2023-05-22T15:40:02.408157+0000","last_unstale":"2023-05-22T15:40:02.814167+0000","last_undegraded":"2023-05-22T15:40:02.814167+0000","last_fullsized":"2023-05-22T15:40:02.814167+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:40:01.360691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:40:01.360691+0000","last_clean_scrub_stamp":"2023-05-22T15:40:01.360691+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:49:44.976589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":566,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.798668+0000","last_change":"2023-05-22T15:40:02.797775+0000","last_active":"2023-05-22T15:40:02.798668+0000","last_peered":"2023-05-22T15:40:02.798668+0000","last_clean":"2023-05-22T15:40:02.798668+0000","last_became_active":"2023-05-22T15:40:02.797573+0000","last_became_peered":"2023-05-22T15:40:02.797573+0000","last_unstale":"2023-05-22T15:40:02.798668+0000","last_undegraded":"2023-05-22T15:40:02.798668+0000","last_fullsized":"2023-05-22T15:40:02.798668+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_clean_scrub_stamp":"2023-05-22T15:40:02.364399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:05:21.391829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1e","version":"72'1","reported_seq":11,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.819727+0000","last_change":"2023-05-22T15:40:02.417453+0000","last_active":"2023-05-22T15:40:02.819727+0000","last_peered":"2023-05-22T15:40:02.819727+0000","last_clean":"2023-05-22T15:40:02.819727+0000","last_became_active":"2023-05-22T15:40:02.417144+0000","last_became_peered":"2023-05-22T15:40:02.417144+0000","last_unstale":"2023-05-22T15:40:02.819727+0000","last_undegraded":"2023-05-22T15:40:02.819727+0000","last_fullsized":"2023-05-22T15:40:02.819727+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:40:01.360691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:40:01.360691+0000","last_clean_scrub_stamp":"2023-05-22T15:40:01.360691+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:59:21.262207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.795371+0000","last_change":"2023-05-22T15:40:02.789568+0000","last_active":"2023-05-22T15:40:02.795371+0000","last_peered":"2023-05-22T15:40:02.795371+0000","last_clean":"2023-05-22T15:40:02.795371+0000","last_became_active":"2023-05-22T15:40:02.789272+0000","last_became_peered":"2023-05-22T15:40:02.789272+0000","last_unstale":"2023-05-22T15:40:02.795371+0000","last_undegraded":"2023-05-22T15:40:02.795371+0000","last_fullsized":"2023-05-22T15:40:02.795371+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_clean_scrub_stamp":"2023-05-22T15:40:02.364399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T15:55:35.316310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"72'2","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.795399+0000","last_change":"2023-05-22T15:40:02.405212+0000","last_active":"2023-05-22T15:40:02.795399+0000","last_peered":"2023-05-22T15:40:02.795399+0000","last_clean":"2023-05-22T15:40:02.795399+0000","last_became_active":"2023-05-22T15:40:02.404665+0000","last_became_peered":"2023-05-22T15:40:02.404665+0000","last_unstale":"2023-05-22T15:40:02.795399+0000","last_undegraded":"2023-05-22T15:40:02.795399+0000","last_fullsized":"2023-05-22T15:40:02.795399+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:40:01.360691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:40:01.360691+0000","last_clean_scrub_stamp":"2023-05-22T15:40:01.360691+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:37:20.385027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.794406+0000","last_change":"2023-05-22T15:40:02.778286+0000","last_active":"2023-05-22T15:40:02.794406+0000","last_peered":"2023-05-22T15:40:02.794406+0000","last_clean":"2023-05-22T15:40:02.794406+0000","last_became_active":"2023-05-22T15:40:02.777257+0000","last_became_peered":"2023-05-22T15:40:02.777257+0000","last_unstale":"2023-05-22T15:40:02.794406+0000","last_undegraded":"2023-05-22T15:40:02.794406+0000","last_fullsized":"2023-05-22T15:40:02.794406+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_clean_scrub_stamp":"2023-05-22T15:40:02.364399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:42:34.280673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1c","version":"72'2","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.829702+0000","last_change":"2023-05-22T15:40:02.415257+0000","last_active":"2023-05-22T15:40:02.829702+0000","last_peered":"2023-05-22T15:40:02.829702+0000","last_clean":"2023-05-22T15:40:02.829702+0000","last_became_active":"2023-05-22T15:40:02.414276+0000","last_became_peered":"2023-05-22T15:40:02.414276+0000","last_unstale":"2023-05-22T15:40:02.829702+0000","last_undegraded":"2023-05-22T15:40:02.829702+0000","last_fullsized":"2023-05-22T15:40:02.829702+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:40:01.360691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:40:01.360691+0000","last_clean_scrub_stamp":"2023-05-22T15:40:01.360691+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:42:55.348036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.782496+0000","last_change":"2023-05-22T15:40:02.780209+0000","last_active":"2023-05-22T15:40:02.782496+0000","last_peered":"2023-05-22T15:40:02.782496+0000","last_clean":"2023-05-22T15:40:02.782496+0000","last_became_active":"2023-05-22T15:40:02.779941+0000","last_became_peered":"2023-05-22T15:40:02.779941+0000","last_unstale":"2023-05-22T15:40:02.782496+0000","last_undegraded":"2023-05-22T15:40:02.782496+0000","last_fullsized":"2023-05-22T15:40:02.782496+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_clean_scrub_stamp":"2023-05-22T15:40:02.364399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:18:56.678275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1b","version":"72'1","reported_seq":11,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.837207+0000","last_change":"2023-05-22T15:40:02.414064+0000","last_active":"2023-05-22T15:40:02.837207+0000","last_peered":"2023-05-22T15:40:02.837207+0000","last_clean":"2023-05-22T15:40:02.837207+0000","last_became_active":"2023-05-22T15:40:02.412878+0000","last_became_peered":"2023-05-22T15:40:02.412878+0000","last_unstale":"2023-05-22T15:40:02.837207+0000","last_undegraded":"2023-05-22T15:40:02.837207+0000","last_fullsized":"2023-05-22T15:40:02.837207+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:40:01.360691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:40:01.360691+0000","last_clean_scrub_stamp":"2023-05-22T15:40:01.360691+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:53:28.798573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.783317+0000","last_change":"2023-05-22T15:40:02.782024+0000","last_active":"2023-05-22T15:40:02.783317+0000","last_peered":"2023-05-22T15:40:02.783317+0000","last_clean":"2023-05-22T15:40:02.783317+0000","last_became_active":"2023-05-22T15:40:02.781702+0000","last_became_peered":"2023-05-22T15:40:02.781702+0000","last_unstale":"2023-05-22T15:40:02.783317+0000","last_undegraded":"2023-05-22T15:40:02.783317+0000","last_fullsized":"2023-05-22T15:40:02.783317+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_clean_scrub_stamp":"2023-05-22T15:40:02.364399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:03:44.838990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1a","version":"72'1","reported_seq":11,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.809662+0000","last_change":"2023-05-22T15:40:02.421209+0000","last_active":"2023-05-22T15:40:02.809662+0000","last_peered":"2023-05-22T15:40:02.809662+0000","last_clean":"2023-05-22T15:40:02.809662+0000","last_became_active":"2023-05-22T15:40:02.420869+0000","last_became_peered":"2023-05-22T15:40:02.420869+0000","last_unstale":"2023-05-22T15:40:02.809662+0000","last_undegraded":"2023-05-22T15:40:02.809662+0000","last_fullsized":"2023-05-22T15:40:02.809662+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:40:01.360691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:40:01.360691+0000","last_clean_scrub_stamp":"2023-05-22T15:40:01.360691+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:32:28.812012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.824935+0000","last_change":"2023-05-22T15:40:02.808669+0000","last_active":"2023-05-22T15:40:02.824935+0000","last_peered":"2023-05-22T15:40:02.824935+0000","last_clean":"2023-05-22T15:40:02.824935+0000","last_became_active":"2023-05-22T15:40:02.807906+0000","last_became_peered":"2023-05-22T15:40:02.807906+0000","last_unstale":"2023-05-22T15:40:02.824935+0000","last_undegraded":"2023-05-22T15:40:02.824935+0000","last_fullsized":"2023-05-22T15:40:02.824935+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_clean_scrub_stamp":"2023-05-22T15:40:02.364399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:37:37.439227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":10,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.770029+0000","last_change":"2023-05-22T15:40:02.411052+0000","last_active":"2023-05-22T15:40:02.770029+0000","last_peered":"2023-05-22T15:40:02.770029+0000","last_clean":"2023-05-22T15:40:02.770029+0000","last_became_active":"2023-05-22T15:40:02.410320+0000","last_became_peered":"2023-05-22T15:40:02.410320+0000","last_unstale":"2023-05-22T15:40:02.770029+0000","last_undegraded":"2023-05-22T15:40:02.770029+0000","last_fullsized":"2023-05-22T15:40:02.770029+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:40:01.360691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:40:01.360691+0000","last_clean_scrub_stamp":"2023-05-22T15:40:01.360691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T18:12:12.676405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.796404+0000","last_change":"2023-05-22T15:40:02.795831+0000","last_active":"2023-05-22T15:40:02.796404+0000","last_peered":"2023-05-22T15:40:02.796404+0000","last_clean":"2023-05-22T15:40:02.796404+0000","last_became_active":"2023-05-22T15:40:02.795583+0000","last_became_peered":"2023-05-22T15:40:02.795583+0000","last_unstale":"2023-05-22T15:40:02.796404+0000","last_undegraded":"2023-05-22T15:40:02.796404+0000","last_fullsized":"2023-05-22T15:40:02.796404+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_clean_scrub_stamp":"2023-05-22T15:40:02.364399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:47:39.526320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":10,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.781756+0000","last_change":"2023-05-22T15:40:02.410980+0000","last_active":"2023-05-22T15:40:02.781756+0000","last_peered":"2023-05-22T15:40:02.781756+0000","last_clean":"2023-05-22T15:40:02.781756+0000","last_became_active":"2023-05-22T15:40:02.410248+0000","last_became_peered":"2023-05-22T15:40:02.410248+0000","last_unstale":"2023-05-22T15:40:02.781756+0000","last_undegraded":"2023-05-22T15:40:02.781756+0000","last_fullsized":"2023-05-22T15:40:02.781756+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:40:01.360691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:40:01.360691+0000","last_clean_scrub_stamp":"2023-05-22T15:40:01.360691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:27:40.572807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.837793+0000","last_change":"2023-05-22T15:40:02.818942+0000","last_active":"2023-05-22T15:40:02.837793+0000","last_peered":"2023-05-22T15:40:02.837793+0000","last_clean":"2023-05-22T15:40:02.837793+0000","last_became_active":"2023-05-22T15:40:02.818386+0000","last_became_peered":"2023-05-22T15:40:02.818386+0000","last_unstale":"2023-05-22T15:40:02.837793+0000","last_undegraded":"2023-05-22T15:40:02.837793+0000","last_fullsized":"2023-05-22T15:40:02.837793+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_clean_scrub_stamp":"2023-05-22T15:40:02.364399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:02:47.354363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":10,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.806642+0000","last_change":"2023-05-22T15:40:02.429499+0000","last_active":"2023-05-22T15:40:02.806642+0000","last_peered":"2023-05-22T15:40:02.806642+0000","last_clean":"2023-05-22T15:40:02.806642+0000","last_became_active":"2023-05-22T15:40:02.428907+0000","last_became_peered":"2023-05-22T15:40:02.428907+0000","last_unstale":"2023-05-22T15:40:02.806642+0000","last_undegraded":"2023-05-22T15:40:02.806642+0000","last_fullsized":"2023-05-22T15:40:02.806642+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:40:01.360691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:40:01.360691+0000","last_clean_scrub_stamp":"2023-05-22T15:40:01.360691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:06:57.809773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.837079+0000","last_change":"2023-05-22T15:40:02.831128+0000","last_active":"2023-05-22T15:40:02.837079+0000","last_peered":"2023-05-22T15:40:02.837079+0000","last_clean":"2023-05-22T15:40:02.837079+0000","last_became_active":"2023-05-22T15:40:02.830566+0000","last_became_peered":"2023-05-22T15:40:02.830566+0000","last_unstale":"2023-05-22T15:40:02.837079+0000","last_undegraded":"2023-05-22T15:40:02.837079+0000","last_fullsized":"2023-05-22T15:40:02.837079+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_clean_scrub_stamp":"2023-05-22T15:40:02.364399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T16:05:38.299803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":10,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.766216+0000","last_change":"2023-05-22T15:40:02.402559+0000","last_active":"2023-05-22T15:40:02.766216+0000","last_peered":"2023-05-22T15:40:02.766216+0000","last_clean":"2023-05-22T15:40:02.766216+0000","last_became_active":"2023-05-22T15:40:02.402062+0000","last_became_peered":"2023-05-22T15:40:02.402062+0000","last_unstale":"2023-05-22T15:40:02.766216+0000","last_undegraded":"2023-05-22T15:40:02.766216+0000","last_fullsized":"2023-05-22T15:40:02.766216+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:40:01.360691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:40:01.360691+0000","last_clean_scrub_stamp":"2023-05-22T15:40:01.360691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:49:44.976589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.824196+0000","last_change":"2023-05-22T15:40:02.822750+0000","last_active":"2023-05-22T15:40:02.824196+0000","last_peered":"2023-05-22T15:40:02.824196+0000","last_clean":"2023-05-22T15:40:02.824196+0000","last_became_active":"2023-05-22T15:40:02.822446+0000","last_became_peered":"2023-05-22T15:40:02.822446+0000","last_unstale":"2023-05-22T15:40:02.824196+0000","last_undegraded":"2023-05-22T15:40:02.824196+0000","last_fullsized":"2023-05-22T15:40:02.824196+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_clean_scrub_stamp":"2023-05-22T15:40:02.364399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:06:58.813481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.15","version":"72'3","reported_seq":13,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.831542+0000","last_change":"2023-05-22T15:40:02.428653+0000","last_active":"2023-05-22T15:40:02.831542+0000","last_peered":"2023-05-22T15:40:02.831542+0000","last_clean":"2023-05-22T15:40:02.831542+0000","last_became_active":"2023-05-22T15:40:02.427985+0000","last_became_peered":"2023-05-22T15:40:02.427985+0000","last_unstale":"2023-05-22T15:40:02.831542+0000","last_undegraded":"2023-05-22T15:40:02.831542+0000","last_fullsized":"2023-05-22T15:40:02.831542+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:40:01.360691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:40:01.360691+0000","last_clean_scrub_stamp":"2023-05-22T15:40:01.360691+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:18:20.056393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.836963+0000","last_change":"2023-05-22T15:40:02.830104+0000","last_active":"2023-05-22T15:40:02.836963+0000","last_peered":"2023-05-22T15:40:02.836963+0000","last_clean":"2023-05-22T15:40:02.836963+0000","last_became_active":"2023-05-22T15:40:02.829439+0000","last_became_peered":"2023-05-22T15:40:02.829439+0000","last_unstale":"2023-05-22T15:40:02.836963+0000","last_undegraded":"2023-05-22T15:40:02.836963+0000","last_fullsized":"2023-05-22T15:40:02.836963+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_clean_scrub_stamp":"2023-05-22T15:40:02.364399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:59:22.265916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.14","version":"72'6","reported_seq":16,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.842400+0000","last_change":"2023-05-22T15:40:02.415057+0000","last_active":"2023-05-22T15:40:02.842400+0000","last_peered":"2023-05-22T15:40:02.842400+0000","last_clean":"2023-05-22T15:40:02.842400+0000","last_became_active":"2023-05-22T15:40:02.412040+0000","last_became_peered":"2023-05-22T15:40:02.412040+0000","last_unstale":"2023-05-22T15:40:02.842400+0000","last_undegraded":"2023-05-22T15:40:02.842400+0000","last_fullsized":"2023-05-22T15:40:02.842400+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:40:01.360691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:40:01.360691+0000","last_clean_scrub_stamp":"2023-05-22T15:40:01.360691+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:41:13.995283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1548,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.813582+0000","last_change":"2023-05-22T15:40:02.792527+0000","last_active":"2023-05-22T15:40:02.813582+0000","last_peered":"2023-05-22T15:40:02.813582+0000","last_clean":"2023-05-22T15:40:02.813582+0000","last_became_active":"2023-05-22T15:40:02.791805+0000","last_became_peered":"2023-05-22T15:40:02.791805+0000","last_unstale":"2023-05-22T15:40:02.813582+0000","last_undegraded":"2023-05-22T15:40:02.813582+0000","last_fullsized":"2023-05-22T15:40:02.813582+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_clean_scrub_stamp":"2023-05-22T15:40:02.364399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:59:22.265916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.13","version":"72'2","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.790538+0000","last_change":"2023-05-22T15:40:02.407045+0000","last_active":"2023-05-22T15:40:02.790538+0000","last_peered":"2023-05-22T15:40:02.790538+0000","last_clean":"2023-05-22T15:40:02.790538+0000","last_became_active":"2023-05-22T15:40:02.406699+0000","last_became_peered":"2023-05-22T15:40:02.406699+0000","last_unstale":"2023-05-22T15:40:02.790538+0000","last_undegraded":"2023-05-22T15:40:02.790538+0000","last_fullsized":"2023-05-22T15:40:02.790538+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:40:01.360691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:40:01.360691+0000","last_clean_scrub_stamp":"2023-05-22T15:40:01.360691+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:34:10.608865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.833683+0000","last_change":"2023-05-22T15:40:02.823616+0000","last_active":"2023-05-22T15:40:02.833683+0000","last_peered":"2023-05-22T15:40:02.833683+0000","last_clean":"2023-05-22T15:40:02.833683+0000","last_became_active":"2023-05-22T15:40:02.821749+0000","last_became_peered":"2023-05-22T15:40:02.821749+0000","last_unstale":"2023-05-22T15:40:02.833683+0000","last_undegraded":"2023-05-22T15:40:02.833683+0000","last_fullsized":"2023-05-22T15:40:02.833683+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_clean_scrub_stamp":"2023-05-22T15:40:02.364399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T15:46:56.191505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":10,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.790265+0000","last_change":"2023-05-22T15:40:02.430134+0000","last_active":"2023-05-22T15:40:02.790265+0000","last_peered":"2023-05-22T15:40:02.790265+0000","last_clean":"2023-05-22T15:40:02.790265+0000","last_became_active":"2023-05-22T15:40:02.429312+0000","last_became_peered":"2023-05-22T15:40:02.429312+0000","last_unstale":"2023-05-22T15:40:02.790265+0000","last_undegraded":"2023-05-22T15:40:02.790265+0000","last_fullsized":"2023-05-22T15:40:02.790265+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:40:01.360691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:40:01.360691+0000","last_clean_scrub_stamp":"2023-05-22T15:40:01.360691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:54:19.121093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.839140+0000","last_change":"2023-05-22T15:40:02.819212+0000","last_active":"2023-05-22T15:40:02.839140+0000","last_peered":"2023-05-22T15:40:02.839140+0000","last_clean":"2023-05-22T15:40:02.839140+0000","last_became_active":"2023-05-22T15:40:02.818034+0000","last_became_peered":"2023-05-22T15:40:02.818034+0000","last_unstale":"2023-05-22T15:40:02.839140+0000","last_undegraded":"2023-05-22T15:40:02.839140+0000","last_fullsized":"2023-05-22T15:40:02.839140+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_clean_scrub_stamp":"2023-05-22T15:40:02.364399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T16:42:44.882436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":10,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.770058+0000","last_change":"2023-05-22T15:40:02.415313+0000","last_active":"2023-05-22T15:40:02.770058+0000","last_peered":"2023-05-22T15:40:02.770058+0000","last_clean":"2023-05-22T15:40:02.770058+0000","last_became_active":"2023-05-22T15:40:02.414454+0000","last_became_peered":"2023-05-22T15:40:02.414454+0000","last_unstale":"2023-05-22T15:40:02.770058+0000","last_undegraded":"2023-05-22T15:40:02.770058+0000","last_fullsized":"2023-05-22T15:40:02.770058+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:40:01.360691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:40:01.360691+0000","last_clean_scrub_stamp":"2023-05-22T15:40:01.360691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:56:25.008963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.836988+0000","last_change":"2023-05-22T15:40:02.830208+0000","last_active":"2023-05-22T15:40:02.836988+0000","last_peered":"2023-05-22T15:40:02.836988+0000","last_clean":"2023-05-22T15:40:02.836988+0000","last_became_active":"2023-05-22T15:40:02.829380+0000","last_became_peered":"2023-05-22T15:40:02.829380+0000","last_unstale":"2023-05-22T15:40:02.836988+0000","last_undegraded":"2023-05-22T15:40:02.836988+0000","last_fullsized":"2023-05-22T15:40:02.836988+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_clean_scrub_stamp":"2023-05-22T15:40:02.364399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:32:29.815721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.10","version":"72'8","reported_seq":18,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.838032+0000","last_change":"2023-05-22T15:40:02.429012+0000","last_active":"2023-05-22T15:40:02.838032+0000","last_peered":"2023-05-22T15:40:02.838032+0000","last_clean":"2023-05-22T15:40:02.838032+0000","last_became_active":"2023-05-22T15:40:02.428136+0000","last_became_peered":"2023-05-22T15:40:02.428136+0000","last_unstale":"2023-05-22T15:40:02.838032+0000","last_undegraded":"2023-05-22T15:40:02.838032+0000","last_fullsized":"2023-05-22T15:40:02.838032+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:40:01.360691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:40:01.360691+0000","last_clean_scrub_stamp":"2023-05-22T15:40:01.360691+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:13:11.335194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.813159+0000","last_change":"2023-05-22T15:40:02.785890+0000","last_active":"2023-05-22T15:40:02.813159+0000","last_peered":"2023-05-22T15:40:02.813159+0000","last_clean":"2023-05-22T15:40:02.813159+0000","last_became_active":"2023-05-22T15:40:02.784750+0000","last_became_peered":"2023-05-22T15:40:02.784750+0000","last_unstale":"2023-05-22T15:40:02.813159+0000","last_undegraded":"2023-05-22T15:40:02.813159+0000","last_fullsized":"2023-05-22T15:40:02.813159+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_clean_scrub_stamp":"2023-05-22T15:40:02.364399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:03:39.548392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.7","version":"72'2","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.793410+0000","last_change":"2023-05-22T15:40:02.398139+0000","last_active":"2023-05-22T15:40:02.793410+0000","last_peered":"2023-05-22T15:40:02.793410+0000","last_clean":"2023-05-22T15:40:02.793410+0000","last_became_active":"2023-05-22T15:40:02.397785+0000","last_became_peered":"2023-05-22T15:40:02.397785+0000","last_unstale":"2023-05-22T15:40:02.793410+0000","last_undegraded":"2023-05-22T15:40:02.793410+0000","last_fullsized":"2023-05-22T15:40:02.793410+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:40:01.360691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:40:01.360691+0000","last_clean_scrub_stamp":"2023-05-22T15:40:01.360691+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:10:33.424873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.813654+0000","last_change":"2023-05-22T15:40:02.798752+0000","last_active":"2023-05-22T15:40:02.813654+0000","last_peered":"2023-05-22T15:40:02.813654+0000","last_clean":"2023-05-22T15:40:02.813654+0000","last_became_active":"2023-05-22T15:40:02.797984+0000","last_became_peered":"2023-05-22T15:40:02.797984+0000","last_unstale":"2023-05-22T15:40:02.813654+0000","last_undegraded":"2023-05-22T15:40:02.813654+0000","last_fullsized":"2023-05-22T15:40:02.813654+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_clean_scrub_stamp":"2023-05-22T15:40:02.364399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:34:11.612573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":10,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.765566+0000","last_change":"2023-05-22T15:40:02.422170+0000","last_active":"2023-05-22T15:40:02.765566+0000","last_peered":"2023-05-22T15:40:02.765566+0000","last_clean":"2023-05-22T15:40:02.765566+0000","last_became_active":"2023-05-22T15:40:02.421687+0000","last_became_peered":"2023-05-22T15:40:02.421687+0000","last_unstale":"2023-05-22T15:40:02.765566+0000","last_undegraded":"2023-05-22T15:40:02.765566+0000","last_fullsized":"2023-05-22T15:40:02.765566+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:40:01.360691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:40:01.360691+0000","last_clean_scrub_stamp":"2023-05-22T15:40:01.360691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:34:10.608865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.824184+0000","last_change":"2023-05-22T15:40:02.822612+0000","last_active":"2023-05-22T15:40:02.824184+0000","last_peered":"2023-05-22T15:40:02.824184+0000","last_clean":"2023-05-22T15:40:02.824184+0000","last_became_active":"2023-05-22T15:40:02.822322+0000","last_became_peered":"2023-05-22T15:40:02.822322+0000","last_unstale":"2023-05-22T15:40:02.824184+0000","last_undegraded":"2023-05-22T15:40:02.824184+0000","last_fullsized":"2023-05-22T15:40:02.824184+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_clean_scrub_stamp":"2023-05-22T15:40:02.364399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:18:21.060101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":10,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.790240+0000","last_change":"2023-05-22T15:40:02.428303+0000","last_active":"2023-05-22T15:40:02.790240+0000","last_peered":"2023-05-22T15:40:02.790240+0000","last_clean":"2023-05-22T15:40:02.790240+0000","last_became_active":"2023-05-22T15:40:02.427706+0000","last_became_peered":"2023-05-22T15:40:02.427706+0000","last_unstale":"2023-05-22T15:40:02.790240+0000","last_undegraded":"2023-05-22T15:40:02.790240+0000","last_fullsized":"2023-05-22T15:40:02.790240+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:40:01.360691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:40:01.360691+0000","last_clean_scrub_stamp":"2023-05-22T15:40:01.360691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T15:50:34.811208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.794466+0000","last_change":"2023-05-22T15:40:02.778514+0000","last_active":"2023-05-22T15:40:02.794466+0000","last_peered":"2023-05-22T15:40:02.794466+0000","last_clean":"2023-05-22T15:40:02.794466+0000","last_became_active":"2023-05-22T15:40:02.777931+0000","last_became_peered":"2023-05-22T15:40:02.777931+0000","last_unstale":"2023-05-22T15:40:02.794466+0000","last_undegraded":"2023-05-22T15:40:02.794466+0000","last_fullsized":"2023-05-22T15:40:02.794466+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_clean_scrub_stamp":"2023-05-22T15:40:02.364399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:39:40.293308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":10,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.770545+0000","last_change":"2023-05-22T15:40:02.423063+0000","last_active":"2023-05-22T15:40:02.770545+0000","last_peered":"2023-05-22T15:40:02.770545+0000","last_clean":"2023-05-22T15:40:02.770545+0000","last_became_active":"2023-05-22T15:40:02.416443+0000","last_became_peered":"2023-05-22T15:40:02.416443+0000","last_unstale":"2023-05-22T15:40:02.770545+0000","last_undegraded":"2023-05-22T15:40:02.770545+0000","last_fullsized":"2023-05-22T15:40:02.770545+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:40:01.360691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:40:01.360691+0000","last_clean_scrub_stamp":"2023-05-22T15:40:01.360691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:24:40.404955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.793210+0000","last_change":"2023-05-22T15:40:02.791761+0000","last_active":"2023-05-22T15:40:02.793210+0000","last_peered":"2023-05-22T15:40:02.793210+0000","last_clean":"2023-05-22T15:40:02.793210+0000","last_became_active":"2023-05-22T15:40:02.791487+0000","last_became_peered":"2023-05-22T15:40:02.791487+0000","last_unstale":"2023-05-22T15:40:02.793210+0000","last_undegraded":"2023-05-22T15:40:02.793210+0000","last_fullsized":"2023-05-22T15:40:02.793210+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_clean_scrub_stamp":"2023-05-22T15:40:02.364399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:12:27.685858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":10,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.773828+0000","last_change":"2023-05-22T15:40:02.408599+0000","last_active":"2023-05-22T15:40:02.773828+0000","last_peered":"2023-05-22T15:40:02.773828+0000","last_clean":"2023-05-22T15:40:02.773828+0000","last_became_active":"2023-05-22T15:40:02.408247+0000","last_became_peered":"2023-05-22T15:40:02.408247+0000","last_unstale":"2023-05-22T15:40:02.773828+0000","last_undegraded":"2023-05-22T15:40:02.773828+0000","last_fullsized":"2023-05-22T15:40:02.773828+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:40:01.360691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:40:01.360691+0000","last_clean_scrub_stamp":"2023-05-22T15:40:01.360691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T16:53:02.734735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.833704+0000","last_change":"2023-05-22T15:40:02.824182+0000","last_active":"2023-05-22T15:40:02.833704+0000","last_peered":"2023-05-22T15:40:02.833704+0000","last_clean":"2023-05-22T15:40:02.833704+0000","last_became_active":"2023-05-22T15:40:02.822848+0000","last_became_peered":"2023-05-22T15:40:02.822848+0000","last_unstale":"2023-05-22T15:40:02.833704+0000","last_undegraded":"2023-05-22T15:40:02.833704+0000","last_fullsized":"2023-05-22T15:40:02.833704+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_clean_scrub_stamp":"2023-05-22T15:40:02.364399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:02:47.354363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":10,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.783230+0000","last_change":"2023-05-22T15:40:02.415326+0000","last_active":"2023-05-22T15:40:02.783230+0000","last_peered":"2023-05-22T15:40:02.783230+0000","last_clean":"2023-05-22T15:40:02.783230+0000","last_became_active":"2023-05-22T15:40:02.414400+0000","last_became_peered":"2023-05-22T15:40:02.414400+0000","last_unstale":"2023-05-22T15:40:02.783230+0000","last_undegraded":"2023-05-22T15:40:02.783230+0000","last_fullsized":"2023-05-22T15:40:02.783230+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:40:01.360691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:40:01.360691+0000","last_clean_scrub_stamp":"2023-05-22T15:40:01.360691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:37:20.385027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.798622+0000","last_change":"2023-05-22T15:40:02.797293+0000","last_active":"2023-05-22T15:40:02.798622+0000","last_peered":"2023-05-22T15:40:02.798622+0000","last_clean":"2023-05-22T15:40:02.798622+0000","last_became_active":"2023-05-22T15:40:02.796428+0000","last_became_peered":"2023-05-22T15:40:02.796428+0000","last_unstale":"2023-05-22T15:40:02.798622+0000","last_undegraded":"2023-05-22T15:40:02.798622+0000","last_fullsized":"2023-05-22T15:40:02.798622+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_clean_scrub_stamp":"2023-05-22T15:40:02.364399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T18:12:13.680113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":10,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.774828+0000","last_change":"2023-05-22T15:40:02.405292+0000","last_active":"2023-05-22T15:40:02.774828+0000","last_peered":"2023-05-22T15:40:02.774828+0000","last_clean":"2023-05-22T15:40:02.774828+0000","last_became_active":"2023-05-22T15:40:02.404828+0000","last_became_peered":"2023-05-22T15:40:02.404828+0000","last_unstale":"2023-05-22T15:40:02.774828+0000","last_undegraded":"2023-05-22T15:40:02.774828+0000","last_fullsized":"2023-05-22T15:40:02.774828+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:40:01.360691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:40:01.360691+0000","last_clean_scrub_stamp":"2023-05-22T15:40:01.360691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T16:27:28.680714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.810848+0000","last_change":"2023-05-22T15:40:02.809663+0000","last_active":"2023-05-22T15:40:02.810848+0000","last_peered":"2023-05-22T15:40:02.810848+0000","last_clean":"2023-05-22T15:40:02.810848+0000","last_became_active":"2023-05-22T15:40:02.809256+0000","last_became_peered":"2023-05-22T15:40:02.809256+0000","last_unstale":"2023-05-22T15:40:02.810848+0000","last_undegraded":"2023-05-22T15:40:02.810848+0000","last_fullsized":"2023-05-22T15:40:02.810848+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_clean_scrub_stamp":"2023-05-22T15:40:02.364399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T18:23:56.809553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":10,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.765035+0000","last_change":"2023-05-22T15:40:02.408908+0000","last_active":"2023-05-22T15:40:02.765035+0000","last_peered":"2023-05-22T15:40:02.765035+0000","last_clean":"2023-05-22T15:40:02.765035+0000","last_became_active":"2023-05-22T15:40:02.408259+0000","last_became_peered":"2023-05-22T15:40:02.408259+0000","last_unstale":"2023-05-22T15:40:02.765035+0000","last_undegraded":"2023-05-22T15:40:02.765035+0000","last_fullsized":"2023-05-22T15:40:02.765035+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:40:01.360691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:40:01.360691+0000","last_clean_scrub_stamp":"2023-05-22T15:40:01.360691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:02:46.350654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.786075+0000","last_change":"2023-05-22T15:40:02.784711+0000","last_active":"2023-05-22T15:40:02.786075+0000","last_peered":"2023-05-22T15:40:02.786075+0000","last_clean":"2023-05-22T15:40:02.786075+0000","last_became_active":"2023-05-22T15:40:02.784244+0000","last_became_peered":"2023-05-22T15:40:02.784244+0000","last_unstale":"2023-05-22T15:40:02.786075+0000","last_undegraded":"2023-05-22T15:40:02.786075+0000","last_fullsized":"2023-05-22T15:40:02.786075+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_clean_scrub_stamp":"2023-05-22T15:40:02.364399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:26:16.725005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":10,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.790989+0000","last_change":"2023-05-22T15:40:02.420259+0000","last_active":"2023-05-22T15:40:02.790989+0000","last_peered":"2023-05-22T15:40:02.790989+0000","last_clean":"2023-05-22T15:40:02.790989+0000","last_became_active":"2023-05-22T15:40:02.419445+0000","last_became_peered":"2023-05-22T15:40:02.419445+0000","last_unstale":"2023-05-22T15:40:02.790989+0000","last_undegraded":"2023-05-22T15:40:02.790989+0000","last_fullsized":"2023-05-22T15:40:02.790989+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:40:01.360691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:40:01.360691+0000","last_clean_scrub_stamp":"2023-05-22T15:40:01.360691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:59:56.458175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"60'139","reported_seq":295,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.406615+0000","last_change":"2023-05-22T15:26:39.318204+0000","last_active":"2023-05-22T15:40:02.406615+0000","last_peered":"2023-05-22T15:40:02.406615+0000","last_clean":"2023-05-22T15:40:02.406615+0000","last_became_active":"2023-05-22T15:26:39.315370+0000","last_became_peered":"2023-05-22T15:26:39.315370+0000","last_unstale":"2023-05-22T15:40:02.406615+0000","last_undegraded":"2023-05-22T15:40:02.406615+0000","last_fullsized":"2023-05-22T15:40:02.406615+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:38.020076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:38.020076+0000","last_clean_scrub_stamp":"2023-05-22T15:26:38.020076+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:06:39.207543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":308,"num_read_kb":512,"num_write":199,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":66,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.382130+0000","last_change":"2023-05-22T15:26:44.971951+0000","last_active":"2023-05-22T15:40:02.382130+0000","last_peered":"2023-05-22T15:40:02.382130+0000","last_clean":"2023-05-22T15:40:02.382130+0000","last_became_active":"2023-05-22T15:26:44.970872+0000","last_became_peered":"2023-05-22T15:26:44.970872+0000","last_unstale":"2023-05-22T15:40:02.382130+0000","last_undegraded":"2023-05-22T15:40:02.382130+0000","last_fullsized":"2023-05-22T15:40:02.382130+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:55:34.752949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.797718+0000","last_change":"2023-05-22T15:40:02.796011+0000","last_active":"2023-05-22T15:40:02.797718+0000","last_peered":"2023-05-22T15:40:02.797718+0000","last_clean":"2023-05-22T15:40:02.797718+0000","last_became_active":"2023-05-22T15:40:02.795707+0000","last_became_peered":"2023-05-22T15:40:02.795707+0000","last_unstale":"2023-05-22T15:40:02.797718+0000","last_undegraded":"2023-05-22T15:40:02.797718+0000","last_fullsized":"2023-05-22T15:40:02.797718+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_clean_scrub_stamp":"2023-05-22T15:40:02.364399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T18:27:29.359577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":10,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.781593+0000","last_change":"2023-05-22T15:40:02.401638+0000","last_active":"2023-05-22T15:40:02.781593+0000","last_peered":"2023-05-22T15:40:02.781593+0000","last_clean":"2023-05-22T15:40:02.781593+0000","last_became_active":"2023-05-22T15:40:02.401306+0000","last_became_peered":"2023-05-22T15:40:02.401306+0000","last_unstale":"2023-05-22T15:40:02.781593+0000","last_undegraded":"2023-05-22T15:40:02.781593+0000","last_fullsized":"2023-05-22T15:40:02.781593+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:40:01.360691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:40:01.360691+0000","last_clean_scrub_stamp":"2023-05-22T15:40:01.360691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:43:27.054935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":64,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.406393+0000","last_change":"2023-05-22T15:26:44.977198+0000","last_active":"2023-05-22T15:40:02.406393+0000","last_peered":"2023-05-22T15:40:02.406393+0000","last_clean":"2023-05-22T15:40:02.406393+0000","last_became_active":"2023-05-22T15:26:44.976445+0000","last_became_peered":"2023-05-22T15:26:44.976445+0000","last_unstale":"2023-05-22T15:40:02.406393+0000","last_undegraded":"2023-05-22T15:40:02.406393+0000","last_fullsized":"2023-05-22T15:40:02.406393+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:12:41.464054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.813500+0000","last_change":"2023-05-22T15:40:02.792455+0000","last_active":"2023-05-22T15:40:02.813500+0000","last_peered":"2023-05-22T15:40:02.813500+0000","last_clean":"2023-05-22T15:40:02.813500+0000","last_became_active":"2023-05-22T15:40:02.791720+0000","last_became_peered":"2023-05-22T15:40:02.791720+0000","last_unstale":"2023-05-22T15:40:02.813500+0000","last_undegraded":"2023-05-22T15:40:02.813500+0000","last_fullsized":"2023-05-22T15:40:02.813500+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_clean_scrub_stamp":"2023-05-22T15:40:02.364399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:02:47.354363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.b","version":"72'2","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.837665+0000","last_change":"2023-05-22T15:40:02.411102+0000","last_active":"2023-05-22T15:40:02.837665+0000","last_peered":"2023-05-22T15:40:02.837665+0000","last_clean":"2023-05-22T15:40:02.837665+0000","last_became_active":"2023-05-22T15:40:02.410649+0000","last_became_peered":"2023-05-22T15:40:02.410649+0000","last_unstale":"2023-05-22T15:40:02.837665+0000","last_undegraded":"2023-05-22T15:40:02.837665+0000","last_fullsized":"2023-05-22T15:40:02.837665+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:40:01.360691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:40:01.360691+0000","last_clean_scrub_stamp":"2023-05-22T15:40:01.360691+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:45:01.491989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":65,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.374837+0000","last_change":"2023-05-22T15:26:44.969469+0000","last_active":"2023-05-22T15:40:02.374837+0000","last_peered":"2023-05-22T15:40:02.374837+0000","last_clean":"2023-05-22T15:40:02.374837+0000","last_became_active":"2023-05-22T15:26:44.969083+0000","last_became_peered":"2023-05-22T15:26:44.969083+0000","last_unstale":"2023-05-22T15:40:02.374837+0000","last_undegraded":"2023-05-22T15:40:02.374837+0000","last_fullsized":"2023-05-22T15:40:02.374837+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:29:23.670067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.795213+0000","last_change":"2023-05-22T15:40:02.786028+0000","last_active":"2023-05-22T15:40:02.795213+0000","last_peered":"2023-05-22T15:40:02.795213+0000","last_clean":"2023-05-22T15:40:02.795213+0000","last_became_active":"2023-05-22T15:40:02.785742+0000","last_became_peered":"2023-05-22T15:40:02.785742+0000","last_unstale":"2023-05-22T15:40:02.795213+0000","last_undegraded":"2023-05-22T15:40:02.795213+0000","last_fullsized":"2023-05-22T15:40:02.795213+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_clean_scrub_stamp":"2023-05-22T15:40:02.364399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:56:01.718556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":10,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.769763+0000","last_change":"2023-05-22T15:40:02.397757+0000","last_active":"2023-05-22T15:40:02.769763+0000","last_peered":"2023-05-22T15:40:02.769763+0000","last_clean":"2023-05-22T15:40:02.769763+0000","last_became_active":"2023-05-22T15:40:02.395637+0000","last_became_peered":"2023-05-22T15:40:02.395637+0000","last_unstale":"2023-05-22T15:40:02.769763+0000","last_undegraded":"2023-05-22T15:40:02.769763+0000","last_fullsized":"2023-05-22T15:40:02.769763+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:40:01.360691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:40:01.360691+0000","last_clean_scrub_stamp":"2023-05-22T15:40:01.360691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:15:59.197371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":66,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.381931+0000","last_change":"2023-05-22T15:26:44.976632+0000","last_active":"2023-05-22T15:40:02.381931+0000","last_peered":"2023-05-22T15:40:02.381931+0000","last_clean":"2023-05-22T15:40:02.381931+0000","last_became_active":"2023-05-22T15:26:44.975546+0000","last_became_peered":"2023-05-22T15:26:44.975546+0000","last_unstale":"2023-05-22T15:40:02.381931+0000","last_undegraded":"2023-05-22T15:40:02.381931+0000","last_fullsized":"2023-05-22T15:40:02.381931+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:42:57.973030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.813543+0000","last_change":"2023-05-22T15:40:02.794726+0000","last_active":"2023-05-22T15:40:02.813543+0000","last_peered":"2023-05-22T15:40:02.813543+0000","last_clean":"2023-05-22T15:40:02.813543+0000","last_became_active":"2023-05-22T15:40:02.794009+0000","last_became_peered":"2023-05-22T15:40:02.794009+0000","last_unstale":"2023-05-22T15:40:02.813543+0000","last_undegraded":"2023-05-22T15:40:02.813543+0000","last_fullsized":"2023-05-22T15:40:02.813543+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_clean_scrub_stamp":"2023-05-22T15:40:02.364399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T18:55:07.687839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.e","version":"72'2","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.795400+0000","last_change":"2023-05-22T15:40:02.401649+0000","last_active":"2023-05-22T15:40:02.795400+0000","last_peered":"2023-05-22T15:40:02.795400+0000","last_clean":"2023-05-22T15:40:02.795400+0000","last_became_active":"2023-05-22T15:40:02.401266+0000","last_became_peered":"2023-05-22T15:40:02.401266+0000","last_unstale":"2023-05-22T15:40:02.795400+0000","last_undegraded":"2023-05-22T15:40:02.795400+0000","last_fullsized":"2023-05-22T15:40:02.795400+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:40:01.360691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:40:01.360691+0000","last_clean_scrub_stamp":"2023-05-22T15:40:01.360691+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:03:38.544683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":60,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.395423+0000","last_change":"2023-05-22T15:26:45.091475+0000","last_active":"2023-05-22T15:40:02.395423+0000","last_peered":"2023-05-22T15:40:02.395423+0000","last_clean":"2023-05-22T15:40:02.395423+0000","last_became_active":"2023-05-22T15:26:45.091143+0000","last_became_peered":"2023-05-22T15:26:45.091143+0000","last_unstale":"2023-05-22T15:40:02.395423+0000","last_undegraded":"2023-05-22T15:40:02.395423+0000","last_fullsized":"2023-05-22T15:40:02.395423+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:12:41.464054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.841762+0000","last_change":"2023-05-22T15:40:02.840221+0000","last_active":"2023-05-22T15:40:02.841762+0000","last_peered":"2023-05-22T15:40:02.841762+0000","last_clean":"2023-05-22T15:40:02.841762+0000","last_became_active":"2023-05-22T15:40:02.839649+0000","last_became_peered":"2023-05-22T15:40:02.839649+0000","last_unstale":"2023-05-22T15:40:02.841762+0000","last_undegraded":"2023-05-22T15:40:02.841762+0000","last_fullsized":"2023-05-22T15:40:02.841762+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_clean_scrub_stamp":"2023-05-22T15:40:02.364399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:49:45.980298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.f","version":"72'1","reported_seq":11,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.790522+0000","last_change":"2023-05-22T15:40:02.401451+0000","last_active":"2023-05-22T15:40:02.790522+0000","last_peered":"2023-05-22T15:40:02.790522+0000","last_clean":"2023-05-22T15:40:02.790522+0000","last_became_active":"2023-05-22T15:40:02.400950+0000","last_became_peered":"2023-05-22T15:40:02.400950+0000","last_unstale":"2023-05-22T15:40:02.790522+0000","last_undegraded":"2023-05-22T15:40:02.790522+0000","last_fullsized":"2023-05-22T15:40:02.790522+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:40:01.360691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:40:01.360691+0000","last_clean_scrub_stamp":"2023-05-22T15:40:01.360691+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:32:28.812012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":64,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.411885+0000","last_change":"2023-05-22T15:26:45.092367+0000","last_active":"2023-05-22T15:40:02.411885+0000","last_peered":"2023-05-22T15:40:02.411885+0000","last_clean":"2023-05-22T15:40:02.411885+0000","last_became_active":"2023-05-22T15:26:45.091951+0000","last_became_peered":"2023-05-22T15:26:45.091951+0000","last_unstale":"2023-05-22T15:40:02.411885+0000","last_undegraded":"2023-05-22T15:40:02.411885+0000","last_fullsized":"2023-05-22T15:40:02.411885+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:42:57.973030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.837196+0000","last_change":"2023-05-22T15:40:02.831422+0000","last_active":"2023-05-22T15:40:02.837196+0000","last_peered":"2023-05-22T15:40:02.837196+0000","last_clean":"2023-05-22T15:40:02.837196+0000","last_became_active":"2023-05-22T15:40:02.830698+0000","last_became_peered":"2023-05-22T15:40:02.830698+0000","last_unstale":"2023-05-22T15:40:02.837196+0000","last_undegraded":"2023-05-22T15:40:02.837196+0000","last_fullsized":"2023-05-22T15:40:02.837196+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_clean_scrub_stamp":"2023-05-22T15:40:02.364399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:45:02.495698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":10,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.763194+0000","last_change":"2023-05-22T15:40:02.396758+0000","last_active":"2023-05-22T15:40:02.763194+0000","last_peered":"2023-05-22T15:40:02.763194+0000","last_clean":"2023-05-22T15:40:02.763194+0000","last_became_active":"2023-05-22T15:40:02.396137+0000","last_became_peered":"2023-05-22T15:40:02.396137+0000","last_unstale":"2023-05-22T15:40:02.763194+0000","last_undegraded":"2023-05-22T15:40:02.763194+0000","last_fullsized":"2023-05-22T15:40:02.763194+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:40:01.360691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:40:01.360691+0000","last_clean_scrub_stamp":"2023-05-22T15:40:01.360691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:45:01.491989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":60,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.395737+0000","last_change":"2023-05-22T15:26:44.978858+0000","last_active":"2023-05-22T15:40:02.395737+0000","last_peered":"2023-05-22T15:40:02.395737+0000","last_clean":"2023-05-22T15:40:02.395737+0000","last_became_active":"2023-05-22T15:26:44.978443+0000","last_became_peered":"2023-05-22T15:26:44.978443+0000","last_unstale":"2023-05-22T15:40:02.395737+0000","last_undegraded":"2023-05-22T15:40:02.395737+0000","last_fullsized":"2023-05-22T15:40:02.395737+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:46:29.290243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.823471+0000","last_change":"2023-05-22T15:40:02.821765+0000","last_active":"2023-05-22T15:40:02.823471+0000","last_peered":"2023-05-22T15:40:02.823471+0000","last_clean":"2023-05-22T15:40:02.823471+0000","last_became_active":"2023-05-22T15:40:02.821558+0000","last_became_peered":"2023-05-22T15:40:02.821558+0000","last_unstale":"2023-05-22T15:40:02.823471+0000","last_undegraded":"2023-05-22T15:40:02.823471+0000","last_fullsized":"2023-05-22T15:40:02.823471+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:40:02.364399+0000","last_clean_scrub_stamp":"2023-05-22T15:40:02.364399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T18:27:29.359577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"72'1","reported_seq":11,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.818846+0000","last_change":"2023-05-22T15:40:02.422583+0000","last_active":"2023-05-22T15:40:02.818846+0000","last_peered":"2023-05-22T15:40:02.818846+0000","last_clean":"2023-05-22T15:40:02.818846+0000","last_became_active":"2023-05-22T15:40:02.418357+0000","last_became_peered":"2023-05-22T15:40:02.418357+0000","last_unstale":"2023-05-22T15:40:02.818846+0000","last_undegraded":"2023-05-22T15:40:02.818846+0000","last_fullsized":"2023-05-22T15:40:02.818846+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:40:01.360691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:40:01.360691+0000","last_clean_scrub_stamp":"2023-05-22T15:40:01.360691+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T18:44:35.502420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":61,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-05-22T15:40:02.403978+0000","last_change":"2023-05-22T15:26:44.982421+0000","last_active":"2023-05-22T15:40:02.403978+0000","last_peered":"2023-05-22T15:40:02.403978+0000","last_clean":"2023-05-22T15:40:02.403978+0000","last_became_active":"2023-05-22T15:26:44.981586+0000","last_became_peered":"2023-05-22T15:26:44.981586+0000","last_unstale":"2023-05-22T15:40:02.403978+0000","last_undegraded":"2023-05-22T15:40:02.403978+0000","last_fullsized":"2023-05-22T15:40:02.403978+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:14:11.156544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":8,"num_pg":32,"stat_sum":{"num_bytes":2330,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":308,"num_read_kb":512,"num_write":199,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575012,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28736,"kb_used_data":1400,"kb_used_omap":0,"kb_used_meta":27328,"kb_avail":94343104,"statfs":{"total":96636764160,"available":96607338496,"internally_reserved":0,"allocated":1433600,"data_stored":1022295,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27983872},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 15:39:57 2023","interfaces":[{"interface":"back","average":{"1min":0.84399999999999997,"5min":0.86799999999999999,"15min":0.86499999999999999},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.40899999999999997},"max":{"1min":1.2150000000000001,"5min":1.359,"15min":1.7709999999999999},"last":0.88400000000000001},{"interface":"front","average":{"1min":0.77700000000000002,"5min":0.88800000000000001,"15min":0.86599999999999999},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":1.2090000000000001,"5min":1.3169999999999999,"15min":1.4239999999999999},"last":1.1539999999999999}]},{"osd":1,"last update":"Mon May 22 15:39:57 2023","interfaces":[{"interface":"back","average":{"1min":0.92300000000000004,"5min":0.97999999999999998,"15min":0.98299999999999998},"min":{"1min":0.5,"5min":0.5,"15min":0.5},"max":{"1min":1.3240000000000001,"5min":1.389,"15min":1.8300000000000001},"last":0.92800000000000005},{"interface":"front","average":{"1min":0.95099999999999996,"5min":1.036,"15min":1.0189999999999999},"min":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":1.3169999999999999,"5min":1.337,"15min":2.161},"last":1.081}]},{"osd":2,"last update":"Mon May 22 15:39:57 2023","interfaces":[{"interface":"back","average":{"1min":0.81599999999999995,"5min":0.879,"15min":0.90200000000000002},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.46800000000000003},"max":{"1min":1.1160000000000001,"5min":1.244,"15min":6.7279999999999998},"last":0.82599999999999996},{"interface":"front","average":{"1min":0.88800000000000001,"5min":0.96699999999999997,"15min":0.95999999999999996},"min":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.51000000000000001},"max":{"1min":1.177,"5min":1.331,"15min":6.8810000000000002},"last":0.70199999999999996}]},{"osd":3,"last update":"Mon May 22 15:39:57 2023","interfaces":[{"interface":"back","average":{"1min":0.92100000000000004,"5min":0.95199999999999996,"15min":0.96499999999999997},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.53800000000000003},"max":{"1min":1.3360000000000001,"5min":1.5580000000000001,"15min":3.0569999999999999},"last":1.0149999999999999},{"interface":"front","average":{"1min":0.95099999999999996,"5min":0.96999999999999997,"15min":0.995},"min":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.54900000000000004},"max":{"1min":1.264,"5min":1.268,"15min":1.6879999999999999},"last":1.0369999999999999}]},{"osd":4,"last update":"Mon May 22 15:39:57 2023","interfaces":[{"interface":"back","average":{"1min":0.87,"5min":0.879,"15min":0.89000000000000001},"min":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"max":{"1min":1.621,"5min":1.621,"15min":1.621},"last":0.95599999999999996},{"interface":"front","average":{"1min":0.98899999999999999,"5min":0.98599999999999999,"15min":1},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.4319999999999999,"5min":1.7689999999999999,"15min":1.958},"last":1.131}]},{"osd":5,"last update":"Mon May 22 15:39:57 2023","interfaces":[{"interface":"back","average":{"1min":1.0229999999999999,"5min":0.999,"15min":1.0189999999999999},"min":{"1min":0.68300000000000005,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":1.409,"5min":1.5029999999999999,"15min":1.5029999999999999},"last":0.99199999999999999},{"interface":"front","average":{"1min":1.071,"5min":1.111,"15min":1.1379999999999999},"min":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":1.5229999999999999,"5min":1.5229999999999999,"15min":5.0839999999999996},"last":1.1080000000000001}]},{"osd":6,"last update":"Mon May 22 15:39:57 2023","interfaces":[{"interface":"back","average":{"1min":1.004,"5min":1.048,"15min":1.0800000000000001},"min":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"max":{"1min":1.484,"5min":1.484,"15min":5.1719999999999997},"last":1.0600000000000001},{"interface":"front","average":{"1min":1.1259999999999999,"5min":1.133,"15min":1.145},"min":{"1min":0.73699999999999999,"5min":0.72199999999999998,"15min":0.72199999999999998},"max":{"1min":1.702,"5min":1.702,"15min":2.04},"last":1.171}]}]},{"osd":6,"up_from":13,"seq":55834575013,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29076,"kb_used_data":844,"kb_used_omap":0,"kb_used_meta":28224,"kb_avail":94342764,"statfs":{"total":96636764160,"available":96606990336,"internally_reserved":0,"allocated":864256,"data_stored":446315,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":28901376},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 15:39:02 2023","interfaces":[{"interface":"back","average":{"1min":0.92300000000000004,"5min":0.94399999999999995,"15min":0.89500000000000002},"min":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.53900000000000003},"max":{"1min":1.327,"5min":1.3380000000000001,"15min":1.355},"last":1.0329999999999999},{"interface":"front","average":{"1min":0.93999999999999995,"5min":0.88,"15min":0.92300000000000004},"min":{"1min":0.626,"5min":0.626,"15min":0.57399999999999995},"max":{"1min":1.4159999999999999,"5min":1.4159999999999999,"15min":1.4159999999999999},"last":1.5680000000000001}]},{"osd":1,"last update":"Mon May 22 15:39:02 2023","interfaces":[{"interface":"back","average":{"1min":0.95299999999999996,"5min":0.88800000000000001,"15min":0.90100000000000002},"min":{"1min":0.66200000000000003,"5min":0.621,"15min":0.53500000000000003},"max":{"1min":1.3169999999999999,"5min":1.3169999999999999,"15min":1.375},"last":0.92300000000000004},{"interface":"front","average":{"1min":0.96799999999999997,"5min":0.96599999999999997,"15min":0.94499999999999995},"min":{"1min":0.67800000000000005,"5min":0.63200000000000001,"15min":0.53600000000000003},"max":{"1min":1.2789999999999999,"5min":1.2789999999999999,"15min":1.325},"last":1.51}]},{"osd":2,"last update":"Mon May 22 15:39:02 2023","interfaces":[{"interface":"back","average":{"1min":0.99299999999999999,"5min":0.997,"15min":0.96799999999999997},"min":{"1min":0.83699999999999997,"5min":0.64700000000000002,"15min":0.64500000000000002},"max":{"1min":1.2030000000000001,"5min":1.2210000000000001,"15min":1.3220000000000001},"last":1.425},{"interface":"front","average":{"1min":1.0449999999999999,"5min":1.048,"15min":1.052},"min":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.57699999999999996},"max":{"1min":1.2390000000000001,"5min":1.3129999999999999,"15min":1.4930000000000001},"last":1.3120000000000001}]},{"osd":3,"last update":"Mon May 22 15:39:02 2023","interfaces":[{"interface":"back","average":{"1min":0.995,"5min":0.97799999999999998,"15min":0.98799999999999999},"min":{"1min":0.71799999999999997,"5min":0.68700000000000006,"15min":0.626},"max":{"1min":1.2330000000000001,"5min":1.2330000000000001,"15min":1.4430000000000001},"last":1.385},{"interface":"front","average":{"1min":0.98999999999999999,"5min":0.95799999999999996,"15min":0.96099999999999997},"min":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.56499999999999995},"max":{"1min":1.2549999999999999,"5min":1.2549999999999999,"15min":1.486},"last":1.2629999999999999}]},{"osd":4,"last update":"Mon May 22 15:39:02 2023","interfaces":[{"interface":"back","average":{"1min":0.87,"5min":0.872,"15min":0.94899999999999995},"min":{"1min":0.61599999999999999,"5min":0.59399999999999997,"15min":0.57599999999999996},"max":{"1min":1.3959999999999999,"5min":1.3959999999999999,"15min":1.6459999999999999},"last":1.171},{"interface":"front","average":{"1min":1.056,"5min":1.02,"15min":1.0600000000000001},"min":{"1min":0.71999999999999997,"5min":0.61299999999999999,"15min":0.61299999999999999},"max":{"1min":1.347,"5min":1.347,"15min":1.5469999999999999},"last":1.6120000000000001}]},{"osd":5,"last update":"Mon May 22 15:39:02 2023","interfaces":[{"interface":"back","average":{"1min":1.0129999999999999,"5min":0.96999999999999997,"15min":0.996},"min":{"1min":0.71599999999999997,"5min":0.64400000000000002,"15min":0.64400000000000002},"max":{"1min":1.401,"5min":1.401,"15min":1.621},"last":1.3460000000000001},{"interface":"front","average":{"1min":1.0680000000000001,"5min":1.0720000000000001,"15min":1.105},"min":{"1min":0.74399999999999999,"5min":0.68700000000000006,"15min":0.68700000000000006},"max":{"1min":1.4610000000000001,"5min":1.4610000000000001,"15min":1.7509999999999999},"last":1.4670000000000001}]},{"osd":7,"last update":"Mon May 22 15:39:02 2023","interfaces":[{"interface":"back","average":{"1min":1.099,"5min":1.1120000000000001,"15min":1.165},"min":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.58399999999999996},"max":{"1min":1.47,"5min":1.492,"15min":1.8120000000000001},"last":1.6890000000000001},{"interface":"front","average":{"1min":1.139,"5min":1.1240000000000001,"15min":1.141},"min":{"1min":0.80200000000000005,"5min":0.75,"15min":0.68999999999999995},"max":{"1min":1.2809999999999999,"5min":1.4259999999999999,"15min":1.6819999999999999},"last":1.716}]}]},{"osd":5,"up_from":13,"seq":55834575013,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":22588,"kb_used_data":820,"kb_used_omap":0,"kb_used_meta":21760,"kb_avail":94349252,"statfs":{"total":96636764160,"available":96613634048,"internally_reserved":0,"allocated":839680,"data_stored":431927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":22282240},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 15:40:02 2023","interfaces":[{"interface":"back","average":{"1min":0.76700000000000002,"5min":0.85899999999999999,"15min":0.85899999999999999},"min":{"1min":0.54900000000000004,"5min":0.54000000000000004,"15min":0.48399999999999999},"max":{"1min":1.3440000000000001,"5min":1.3440000000000001,"15min":1.3440000000000001},"last":1.3440000000000001},{"interface":"front","average":{"1min":0.91600000000000004,"5min":0.89900000000000002,"15min":0.86699999999999999},"min":{"1min":0.65400000000000003,"5min":0.497,"15min":0.497},"max":{"1min":2.1699999999999999,"5min":2.1699999999999999,"15min":2.1699999999999999},"last":0.83299999999999996}]},{"osd":1,"last update":"Mon May 22 15:40:02 2023","interfaces":[{"interface":"back","average":{"1min":0.91100000000000003,"5min":0.99099999999999999,"15min":0.97699999999999998},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":1.9370000000000001,"5min":1.9370000000000001,"15min":1.9370000000000001},"last":1.169},{"interface":"front","average":{"1min":0.97099999999999997,"5min":1.0269999999999999,"15min":1.0149999999999999},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.2350000000000001,"5min":1.284,"15min":1.399},"last":1.1359999999999999}]},{"osd":2,"last update":"Mon May 22 15:40:02 2023","interfaces":[{"interface":"back","average":{"1min":0.98999999999999999,"5min":0.95799999999999996,"15min":0.92500000000000004},"min":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":1.9830000000000001,"5min":1.9830000000000001,"15min":1.9830000000000001},"last":1.0129999999999999},{"interface":"front","average":{"1min":0.95299999999999996,"5min":0.97399999999999998,"15min":0.92100000000000004},"min":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.53800000000000003},"max":{"1min":2.1120000000000001,"5min":2.1120000000000001,"15min":2.1120000000000001},"last":0.72399999999999998}]},{"osd":3,"last update":"Mon May 22 15:40:02 2023","interfaces":[{"interface":"back","average":{"1min":1.032,"5min":1.002,"15min":0.95499999999999996},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.4419999999999999,"5min":1.4419999999999999,"15min":1.6839999999999999},"last":1.252},{"interface":"front","average":{"1min":1.175,"5min":1.069,"15min":1.04},"min":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.58299999999999996},"max":{"1min":2.52,"5min":2.52,"15min":2.52},"last":1.2130000000000001}]},{"osd":4,"last update":"Mon May 22 15:40:02 2023","interfaces":[{"interface":"back","average":{"1min":1.113,"5min":0.96399999999999997,"15min":0.93999999999999995},"min":{"1min":0.73399999999999999,"5min":0.60599999999999998,"15min":0.57599999999999996},"max":{"1min":2.226,"5min":2.226,"15min":2.226},"last":0.93999999999999995},{"interface":"front","average":{"1min":1.1659999999999999,"5min":1.054,"15min":1.0389999999999999},"min":{"1min":0.67800000000000005,"5min":0.629,"15min":0.622},"max":{"1min":1.6859999999999999,"5min":1.6859999999999999,"15min":1.6859999999999999},"last":1.4079999999999999}]},{"osd":6,"last update":"Mon May 22 15:40:02 2023","interfaces":[{"interface":"back","average":{"1min":1.194,"5min":1.079,"15min":1.0529999999999999},"min":{"1min":0.67900000000000005,"5min":0.65500000000000003,"15min":0.61299999999999999},"max":{"1min":2.831,"5min":2.831,"15min":2.831},"last":1.101},{"interface":"front","average":{"1min":1.1919999999999999,"5min":1.1220000000000001,"15min":1.081},"min":{"1min":0.85199999999999998,"5min":0.76000000000000001,"15min":0.69299999999999995},"max":{"1min":2.3690000000000002,"5min":2.3690000000000002,"15min":2.3690000000000002},"last":1.0669999999999999}]},{"osd":7,"last update":"Mon May 22 15:40:02 2023","interfaces":[{"interface":"back","average":{"1min":1.2729999999999999,"5min":1.175,"15min":1.1439999999999999},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":1.849,"5min":1.849,"15min":2.8260000000000001},"last":1.4550000000000001},{"interface":"front","average":{"1min":1.323,"5min":1.2070000000000001,"15min":1.169},"min":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.71699999999999997},"max":{"1min":2.7429999999999999,"5min":2.7429999999999999,"15min":2.7429999999999999},"last":1.3120000000000001}]}]},{"osd":4,"up_from":13,"seq":55834575011,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":24700,"kb_used_data":820,"kb_used_omap":0,"kb_used_meta":23872,"kb_avail":94347140,"statfs":{"total":96636764160,"available":96611471360,"internally_reserved":0,"allocated":839680,"data_stored":431927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":24444928},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 15:39:01 2023","interfaces":[{"interface":"back","average":{"1min":0.99199999999999999,"5min":0.93300000000000005,"15min":0.89700000000000002},"min":{"1min":0.64800000000000002,"5min":0.56699999999999995,"15min":0.51400000000000001},"max":{"1min":1.2370000000000001,"5min":1.3049999999999999,"15min":1.657},"last":1.016},{"interface":"front","average":{"1min":0.86199999999999999,"5min":0.93799999999999994,"15min":0.91900000000000004},"min":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.51800000000000002},"max":{"1min":1.1259999999999999,"5min":1.2430000000000001,"15min":1.633},"last":0.66600000000000004}]},{"osd":1,"last update":"Mon May 22 15:39:01 2023","interfaces":[{"interface":"back","average":{"1min":1.004,"5min":0.98799999999999999,"15min":0.95299999999999996},"min":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.54000000000000004},"max":{"1min":1.383,"5min":1.383,"15min":1.3859999999999999},"last":1.2},{"interface":"front","average":{"1min":0.93300000000000005,"5min":0.93899999999999995,"15min":0.89500000000000002},"min":{"1min":0.68999999999999995,"5min":0.66300000000000003,"15min":0.51600000000000001},"max":{"1min":1.3120000000000001,"5min":1.3640000000000001,"15min":1.3640000000000001},"last":1.298}]},{"osd":2,"last update":"Mon May 22 15:39:01 2023","interfaces":[{"interface":"back","average":{"1min":1.0309999999999999,"5min":0.96099999999999997,"15min":0.95299999999999996},"min":{"1min":0.77400000000000002,"5min":0.67100000000000004,"15min":0.59299999999999997},"max":{"1min":1.4199999999999999,"5min":1.4199999999999999,"15min":1.6950000000000001},"last":1.2370000000000001},{"interface":"front","average":{"1min":1.016,"5min":1.0029999999999999,"15min":1.0249999999999999},"min":{"1min":0.80300000000000005,"5min":0.63800000000000001,"15min":0.57199999999999995},"max":{"1min":1.325,"5min":1.361,"15min":1.9770000000000001},"last":0.79100000000000004}]},{"osd":3,"last update":"Mon May 22 15:39:01 2023","interfaces":[{"interface":"back","average":{"1min":0.97099999999999997,"5min":0.94899999999999995,"15min":0.94999999999999996},"min":{"1min":0.78600000000000003,"5min":0.67900000000000005,"15min":0.54600000000000004},"max":{"1min":1.222,"5min":1.3169999999999999,"15min":1.665},"last":1.1499999999999999},{"interface":"front","average":{"1min":1.0860000000000001,"5min":1.05,"15min":1.05},"min":{"1min":0.77400000000000002,"5min":0.747,"15min":0.64200000000000002},"max":{"1min":1.3440000000000001,"5min":1.3480000000000001,"15min":1.538},"last":0.874}]},{"osd":5,"last update":"Mon May 22 15:39:01 2023","interfaces":[{"interface":"back","average":{"1min":1.0369999999999999,"5min":0.96499999999999997,"15min":1.018},"min":{"1min":0.65200000000000002,"5min":0.52700000000000002,"15min":0.51200000000000001},"max":{"1min":1.3979999999999999,"5min":1.3979999999999999,"15min":1.7270000000000001},"last":1.1060000000000001},{"interface":"front","average":{"1min":0.98699999999999999,"5min":0.99099999999999999,"15min":0.98899999999999999},"min":{"1min":0.71199999999999997,"5min":0.66400000000000003,"15min":0.626},"max":{"1min":1.266,"5min":1.4139999999999999,"15min":1.6719999999999999},"last":0.97899999999999998}]},{"osd":6,"last update":"Mon May 22 15:39:01 2023","interfaces":[{"interface":"back","average":{"1min":1.0680000000000001,"5min":0.98299999999999998,"15min":1.034},"min":{"1min":0.67500000000000004,"5min":0.623,"15min":0.58099999999999996},"max":{"1min":1.288,"5min":1.3280000000000001,"15min":1.6020000000000001},"last":0.748},{"interface":"front","average":{"1min":1.1279999999999999,"5min":1.117,"15min":1.1259999999999999},"min":{"1min":0.81200000000000006,"5min":0.70899999999999996,"15min":0.69799999999999995},"max":{"1min":1.4430000000000001,"5min":1.4430000000000001,"15min":2.0699999999999998},"last":0.93200000000000005}]},{"osd":7,"last update":"Mon May 22 15:39:01 2023","interfaces":[{"interface":"back","average":{"1min":1.0760000000000001,"5min":1.042,"15min":1.093},"min":{"1min":0.76200000000000001,"5min":0.71899999999999997,"15min":0.67700000000000005},"max":{"1min":1.4870000000000001,"5min":1.4870000000000001,"15min":1.6639999999999999},"last":0.82999999999999996},{"interface":"front","average":{"1min":1.1579999999999999,"5min":1.143,"15min":1.157},"min":{"1min":0.86099999999999999,"5min":0.71499999999999997,"15min":0.71499999999999997},"max":{"1min":1.5720000000000001,"5min":1.5720000000000001,"15min":1.7410000000000001},"last":1.329}]}]},{"osd":3,"up_from":13,"seq":55834575011,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29756,"kb_used_data":820,"kb_used_omap":0,"kb_used_meta":28928,"kb_avail":94342084,"statfs":{"total":96636764160,"available":96606294016,"internally_reserved":0,"allocated":839680,"data_stored":431927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":29622272},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 15:39:48 2023","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.64000000000000001,"15min":0.61399999999999999},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":0.91700000000000004,"5min":1.1739999999999999,"15min":1.1739999999999999},"last":0.26300000000000001},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.72199999999999998,"15min":0.67300000000000004},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.23699999999999999},"max":{"1min":1.1799999999999999,"5min":1.1799999999999999,"15min":1.296},"last":0.374}]},{"osd":1,"last update":"Mon May 22 15:39:48 2023","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.65900000000000003,"15min":0.65100000000000002},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.92900000000000005,"5min":1.0509999999999999,"15min":1.0600000000000001},"last":0.32900000000000001},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.79600000000000004,"15min":0.75},"min":{"1min":0.313,"5min":0.313,"15min":0.216},"max":{"1min":0.875,"5min":1.155,"15min":1.6579999999999999},"last":0.42199999999999999}]},{"osd":2,"last update":"Mon May 22 15:39:48 2023","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.82799999999999996,"15min":0.79400000000000004},"min":{"1min":0.374,"5min":0.374,"15min":0.32200000000000001},"max":{"1min":1.111,"5min":1.2989999999999999,"15min":1.2989999999999999},"last":0.36599999999999999},{"interface":"front","average":{"1min":0.70299999999999996,"5min":0.82799999999999996,"15min":0.80000000000000004},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.29699999999999999},"max":{"1min":0.98099999999999998,"5min":1.359,"15min":1.855},"last":0.40000000000000002}]},{"osd":4,"last update":"Mon May 22 15:39:48 2023","interfaces":[{"interface":"back","average":{"1min":0.85699999999999998,"5min":0.94999999999999996,"15min":0.94699999999999995},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":1.069,"5min":1.2310000000000001,"15min":1.5820000000000001},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.91300000000000003,"5min":1.0329999999999999,"15min":1.008},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.3740000000000001,"5min":1.3740000000000001,"15min":1.8069999999999999},"last":0.85099999999999998}]},{"osd":5,"last update":"Mon May 22 15:39:48 2023","interfaces":[{"interface":"back","average":{"1min":0.88900000000000001,"5min":1.024,"15min":0.98399999999999999},"min":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":1.3149999999999999,"5min":1.4339999999999999,"15min":1.4339999999999999},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.93700000000000006,"5min":1.0509999999999999,"15min":1.052},"min":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":1.1459999999999999,"5min":1.4359999999999999,"15min":2.4820000000000002},"last":0.55900000000000005}]},{"osd":6,"last update":"Mon May 22 15:39:48 2023","interfaces":[{"interface":"back","average":{"1min":0.89100000000000001,"5min":1.048,"15min":1.022},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.4139999999999999,"5min":1.4139999999999999,"15min":1.863},"last":0.60499999999999998},{"interface":"front","average":{"1min":1.0289999999999999,"5min":1.095,"15min":1.0880000000000001},"min":{"1min":0.749,"5min":0.749,"15min":0.53500000000000003},"max":{"1min":1.28,"5min":1.5009999999999999,"15min":2.6589999999999998},"last":0.67300000000000004}]},{"osd":7,"last update":"Mon May 22 15:39:48 2023","interfaces":[{"interface":"back","average":{"1min":1.0169999999999999,"5min":1.113,"15min":1.121},"min":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":1.369,"5min":1.5429999999999999,"15min":2.762},"last":0.69699999999999995},{"interface":"front","average":{"1min":0.99199999999999999,"5min":1.0880000000000001,"15min":1.0760000000000001},"min":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":1.24,"5min":1.4079999999999999,"15min":1.714},"last":0.61899999999999999}]}]},{"osd":2,"up_from":13,"seq":55834575012,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":16060,"kb_used_data":820,"kb_used_omap":0,"kb_used_meta":15232,"kb_avail":94355780,"statfs":{"total":96636764160,"available":96620318720,"internally_reserved":0,"allocated":839680,"data_stored":431927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":15597568},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 15:39:53 2023","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.66400000000000003,"15min":0.64100000000000001},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.317},"max":{"1min":1.8240000000000001,"5min":1.8240000000000001,"15min":1.8240000000000001},"last":0.623},{"interface":"front","average":{"1min":0.80900000000000005,"5min":0.78200000000000003,"15min":0.73099999999999998},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.253},"max":{"1min":1.8160000000000001,"5min":1.8160000000000001,"15min":1.8160000000000001},"last":0.76100000000000001}]},{"osd":1,"last update":"Mon May 22 15:39:53 2023","interfaces":[{"interface":"back","average":{"1min":0.72199999999999998,"5min":0.751,"15min":0.72999999999999998},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.26900000000000002},"max":{"1min":0.91600000000000004,"5min":1.1950000000000001,"15min":1.5309999999999999},"last":0.72499999999999998},{"interface":"front","average":{"1min":0.88300000000000001,"5min":0.84699999999999998,"15min":0.78400000000000003},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.29099999999999998},"max":{"1min":2.5089999999999999,"5min":2.5089999999999999,"15min":2.5089999999999999},"last":1.141}]},{"osd":3,"last update":"Mon May 22 15:39:53 2023","interfaces":[{"interface":"back","average":{"1min":0.84299999999999997,"5min":0.745,"15min":0.71399999999999997},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.26700000000000002},"max":{"1min":2.7629999999999999,"5min":2.7629999999999999,"15min":2.7629999999999999},"last":0.878},{"interface":"front","average":{"1min":0.96799999999999997,"5min":0.91600000000000004,"15min":0.85899999999999999},"min":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.33000000000000002},"max":{"1min":2.6840000000000002,"5min":2.6840000000000002,"15min":2.6840000000000002},"last":0.93799999999999994}]},{"osd":4,"last update":"Mon May 22 15:39:53 2023","interfaces":[{"interface":"back","average":{"1min":1.0900000000000001,"5min":1.038,"15min":0.999},"min":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.55500000000000005},"max":{"1min":2.8039999999999998,"5min":2.8039999999999998,"15min":2.8039999999999998},"last":1.054},{"interface":"front","average":{"1min":1.089,"5min":1.0529999999999999,"15min":1.02},"min":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":2.0819999999999999,"5min":2.0819999999999999,"15min":2.0819999999999999},"last":1.2789999999999999}]},{"osd":5,"last update":"Mon May 22 15:39:53 2023","interfaces":[{"interface":"back","average":{"1min":1.125,"5min":1.0649999999999999,"15min":1.034},"min":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.55200000000000005},"max":{"1min":2.3860000000000001,"5min":2.3860000000000001,"15min":2.4569999999999999},"last":0.99199999999999999},{"interface":"front","average":{"1min":1.109,"5min":1.032,"15min":1},"min":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.56200000000000006},"max":{"1min":3.0299999999999998,"5min":3.0299999999999998,"15min":3.0299999999999998},"last":1.0900000000000001}]},{"osd":6,"last update":"Mon May 22 15:39:53 2023","interfaces":[{"interface":"back","average":{"1min":1.1659999999999999,"5min":1.0640000000000001,"15min":1.0529999999999999},"min":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.56399999999999995},"max":{"1min":2.883,"5min":2.883,"15min":3.488},"last":1.2390000000000001},{"interface":"front","average":{"1min":1.2190000000000001,"5min":1.1259999999999999,"15min":1.1100000000000001},"min":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.53400000000000003},"max":{"1min":3.1459999999999999,"5min":3.1459999999999999,"15min":3.7330000000000001},"last":1.1910000000000001}]},{"osd":7,"last update":"Mon May 22 15:39:53 2023","interfaces":[{"interface":"back","average":{"1min":1.2370000000000001,"5min":1.1599999999999999,"15min":1.119},"min":{"1min":0.84399999999999997,"5min":0.77000000000000002,"15min":0.58299999999999996},"max":{"1min":2.5990000000000002,"5min":2.5990000000000002,"15min":2.5990000000000002},"last":1.361},{"interface":"front","average":{"1min":1.246,"5min":1.1559999999999999,"15min":1.0940000000000001},"min":{"1min":0.84599999999999997,"5min":0.77200000000000002,"15min":0.54000000000000004},"max":{"1min":3.8799999999999999,"5min":3.8799999999999999,"15min":3.8799999999999999},"last":1.339}]}]},{"osd":1,"up_from":13,"seq":55834575012,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":22908,"kb_used_data":820,"kb_used_omap":0,"kb_used_meta":22080,"kb_avail":94348932,"statfs":{"total":96636764160,"available":96613306368,"internally_reserved":0,"allocated":839680,"data_stored":431927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":22609920},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 15:39:05 2023","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.63900000000000001,"15min":0.63400000000000001},"min":{"1min":0.52000000000000002,"5min":0.439,"15min":0.28799999999999998},"max":{"1min":0.91800000000000004,"5min":1.0569999999999999,"15min":1.7689999999999999},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.80200000000000005,"5min":0.80500000000000005,"15min":0.75700000000000001},"min":{"1min":0.56299999999999994,"5min":0.49099999999999999,"15min":0.36399999999999999},"max":{"1min":1.046,"5min":1.1439999999999999,"15min":1.3799999999999999},"last":0.64600000000000002}]},{"osd":2,"last update":"Mon May 22 15:39:05 2023","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":0.74399999999999999,"15min":0.70499999999999996},"min":{"1min":0.60599999999999998,"5min":0.38600000000000001,"15min":0.34100000000000003},"max":{"1min":1.1060000000000001,"5min":1.1060000000000001,"15min":1.7290000000000001},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.79100000000000004,"15min":0.72299999999999998},"min":{"1min":0.53900000000000003,"5min":0.50800000000000001,"15min":0.27800000000000002},"max":{"1min":1.2170000000000001,"5min":1.2629999999999999,"15min":2.4049999999999998},"last":0.40600000000000003}]},{"osd":3,"last update":"Mon May 22 15:39:05 2023","interfaces":[{"interface":"back","average":{"1min":0.86099999999999999,"5min":0.85699999999999998,"15min":0.81200000000000006},"min":{"1min":0.63400000000000001,"5min":0.59599999999999997,"15min":0.317},"max":{"1min":1.1519999999999999,"5min":1.2450000000000001,"15min":1.5069999999999999},"last":0.435},{"interface":"front","average":{"1min":0.84299999999999997,"5min":0.82199999999999995,"15min":0.79800000000000004},"min":{"1min":0.65600000000000003,"5min":0.58399999999999996,"15min":0.315},"max":{"1min":1.1859999999999999,"5min":1.1970000000000001,"15min":7.5620000000000003},"last":0.55600000000000005}]},{"osd":4,"last update":"Mon May 22 15:39:05 2023","interfaces":[{"interface":"back","average":{"1min":1.0089999999999999,"5min":0.96399999999999997,"15min":0.93000000000000005},"min":{"1min":0.749,"5min":0.63700000000000001,"15min":0.54200000000000004},"max":{"1min":1.383,"5min":1.486,"15min":1.8169999999999999},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.98899999999999999,"5min":1.014,"15min":0.97599999999999998},"min":{"1min":0.80500000000000005,"5min":0.79800000000000004,"15min":0.54300000000000004},"max":{"1min":1.206,"5min":1.3520000000000001,"15min":1.3520000000000001},"last":0.58999999999999997}]},{"osd":5,"last update":"Mon May 22 15:39:05 2023","interfaces":[{"interface":"back","average":{"1min":1.071,"5min":1.0669999999999999,"15min":1.0189999999999999},"min":{"1min":0.76800000000000002,"5min":0.73099999999999998,"15min":0.59499999999999997},"max":{"1min":1.244,"5min":1.395,"15min":1.7170000000000001},"last":0.83799999999999997},{"interface":"front","average":{"1min":1.0640000000000001,"5min":1.0589999999999999,"15min":1.0329999999999999},"min":{"1min":0.78500000000000003,"5min":0.70999999999999996,"15min":0.59199999999999997},"max":{"1min":1.6850000000000001,"5min":1.6850000000000001,"15min":4.4710000000000001},"last":0.60799999999999998}]},{"osd":6,"last update":"Mon May 22 15:39:05 2023","interfaces":[{"interface":"back","average":{"1min":1.04,"5min":1.0589999999999999,"15min":1.056},"min":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.63800000000000001},"max":{"1min":1.262,"5min":1.5469999999999999,"15min":14.968999999999999},"last":0.627},{"interface":"front","average":{"1min":1.0800000000000001,"5min":1.117,"15min":1.1419999999999999},"min":{"1min":0.90000000000000002,"5min":0.85199999999999998,"15min":0.56699999999999995},"max":{"1min":1.3859999999999999,"5min":1.5269999999999999,"15min":13.242000000000001},"last":0.73599999999999999}]},{"osd":7,"last update":"Mon May 22 15:39:05 2023","interfaces":[{"interface":"back","average":{"1min":1.127,"5min":1.125,"15min":1.111},"min":{"1min":0.79700000000000004,"5min":0.747,"15min":0.63},"max":{"1min":1.306,"5min":1.4730000000000001,"15min":1.9259999999999999},"last":0.69599999999999995},{"interface":"front","average":{"1min":1.1779999999999999,"5min":1.139,"15min":1.111},"min":{"1min":0.86899999999999999,"5min":0.82399999999999995,"15min":0.46400000000000002},"max":{"1min":1.8600000000000001,"5min":1.8600000000000001,"15min":5.681},"last":0.79900000000000004}]}]},{"osd":0,"up_from":13,"seq":55834575013,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":22056,"kb_used_data":1440,"kb_used_omap":0,"kb_used_meta":20608,"kb_avail":94349784,"statfs":{"total":96636764160,"available":96614178816,"internally_reserved":0,"allocated":1474560,"data_stored":1038797,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":21102592},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon May 22 15:39:01 2023","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.69599999999999995,"15min":0.65200000000000002},"min":{"1min":0.51400000000000001,"5min":0.51100000000000001,"15min":0.24399999999999999},"max":{"1min":1,"5min":1.125,"15min":1.3049999999999999},"last":0.79100000000000004},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.747,"15min":0.72399999999999998},"min":{"1min":0.54700000000000004,"5min":0.52000000000000002,"15min":0.26200000000000001},"max":{"1min":1.0189999999999999,"5min":1.2390000000000001,"15min":1.2390000000000001},"last":0.745}]},{"osd":2,"last update":"Mon May 22 15:39:01 2023","interfaces":[{"interface":"back","average":{"1min":0.74099999999999999,"5min":0.76200000000000001,"15min":0.72699999999999998},"min":{"1min":0.57599999999999996,"5min":0.54700000000000004,"15min":0.28399999999999997},"max":{"1min":1.145,"5min":1.145,"15min":1.4059999999999999},"last":0.91600000000000004},{"interface":"front","average":{"1min":0.80900000000000005,"5min":0.82499999999999996,"15min":0.79300000000000004},"min":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.29699999999999999},"max":{"1min":1.1850000000000001,"5min":1.2030000000000001,"15min":1.3999999999999999},"last":0.94699999999999995}]},{"osd":3,"last update":"Mon May 22 15:39:01 2023","interfaces":[{"interface":"back","average":{"1min":0.75,"5min":0.75800000000000001,"15min":0.73999999999999999},"min":{"1min":0.55100000000000005,"5min":0.43099999999999999,"15min":0.36599999999999999},"max":{"1min":1.0660000000000001,"5min":1.1579999999999999,"15min":1.28},"last":0.872},{"interface":"front","average":{"1min":0.90300000000000002,"5min":0.89300000000000002,"15min":0.84399999999999997},"min":{"1min":0.73299999999999998,"5min":0.61899999999999999,"15min":0.33000000000000002},"max":{"1min":1.3500000000000001,"5min":1.3660000000000001,"15min":1.4570000000000001},"last":1.0920000000000001}]},{"osd":4,"last update":"Mon May 22 15:39:01 2023","interfaces":[{"interface":"back","average":{"1min":0.95699999999999996,"5min":0.97099999999999997,"15min":0.93200000000000005},"min":{"1min":0.76800000000000002,"5min":0.75900000000000001,"15min":0.53600000000000003},"max":{"1min":1.2529999999999999,"5min":1.4790000000000001,"15min":1.696},"last":1.0629999999999999},{"interface":"front","average":{"1min":1.0329999999999999,"5min":1.0449999999999999,"15min":0.999},"min":{"1min":0.85299999999999998,"5min":0.79100000000000004,"15min":0.54900000000000004},"max":{"1min":1.2370000000000001,"5min":1.5089999999999999,"15min":1.6020000000000001},"last":1.1519999999999999}]},{"osd":5,"last update":"Mon May 22 15:39:01 2023","interfaces":[{"interface":"back","average":{"1min":1.0880000000000001,"5min":1.0740000000000001,"15min":1.0429999999999999},"min":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.56699999999999995},"max":{"1min":1.458,"5min":1.5229999999999999,"15min":1.6850000000000001},"last":1.038},{"interface":"front","average":{"1min":1.0209999999999999,"5min":1.0209999999999999,"15min":0.98999999999999999},"min":{"1min":0.80000000000000004,"5min":0.64700000000000002,"15min":0.54800000000000004},"max":{"1min":1.3400000000000001,"5min":1.4019999999999999,"15min":1.4510000000000001},"last":1.1200000000000001}]},{"osd":6,"last update":"Mon May 22 15:39:01 2023","interfaces":[{"interface":"back","average":{"1min":1.135,"5min":1.1140000000000001,"15min":1.0529999999999999},"min":{"1min":0.89500000000000002,"5min":0.82199999999999995,"15min":0.61099999999999999},"max":{"1min":1.651,"5min":1.669,"15min":1.7450000000000001},"last":1.357},{"interface":"front","average":{"1min":1.137,"5min":1.119,"15min":1.0680000000000001},"min":{"1min":0.88300000000000001,"5min":0.875,"15min":0.54200000000000004},"max":{"1min":1.514,"5min":1.7070000000000001,"15min":1.7070000000000001},"last":1.3080000000000001}]},{"osd":7,"last update":"Mon May 22 15:39:01 2023","interfaces":[{"interface":"back","average":{"1min":1.1659999999999999,"5min":1.1479999999999999,"15min":1.107},"min":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.56499999999999995},"max":{"1min":1.5580000000000001,"5min":1.5620000000000001,"15min":1.629},"last":1.1919999999999999},{"interface":"front","average":{"1min":1.1040000000000001,"5min":1.107,"15min":1.0620000000000001},"min":{"1min":0.91300000000000003,"5min":0.70399999999999996,"15min":0.58399999999999996},"max":{"1min":1.377,"5min":1.478,"15min":1.5469999999999999},"last":1.3799999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2114,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-05-22T15:40:07.189 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-05-22T15:40:07.190 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2023-05-22T15:40:07.368 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:07.362+0000 7f069561d640 1 Processor -- start 2023-05-22T15:40:07.368 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:07.362+0000 7f069561d640 1 -- start start 2023-05-22T15:40:07.368 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:07.362+0000 7f069561d640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0690159970 0x7f0690159d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:40:07.368 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:07.362+0000 7f069561d640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0690070c30 con 0x7f069015d680 2023-05-22T15:40:07.368 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:07.362+0000 7f069561d640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0690070e60 con 0x7f069015dad0 2023-05-22T15:40:07.369 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:07.362+0000 7f069561d640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06900cb970 con 0x7f0690159970 2023-05-22T15:40:07.369 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:07.362+0000 7f068e7fc640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f069015d680 0x7f06900cb260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:41372/0 (socket says 172.21.15.16:41372) 2023-05-22T15:40:07.370 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:07.362+0000 7f068e7fc640 1 -- 172.21.15.16:0/1289025645 learned_addr learned my addr 172.21.15.16:0/1289025645 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:40:07.370 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:07.362+0000 7f068effd640 1 --2- 172.21.15.16:0/1289025645 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0690159970 0x7f0690159d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:40:07.370 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:07.362+0000 7f068dffb640 1 -- 172.21.15.16:0/1289025645 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2518251646 0 0) 0x7f0690070c30 con 0x7f069015d680 2023-05-22T15:40:07.370 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:07.362+0000 7f068dffb640 1 -- 172.21.15.16:0/1289025645 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0678003530 con 0x7f069015d680 2023-05-22T15:40:07.370 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:07.362+0000 7f068effd640 1 -- 172.21.15.16:0/1289025645 >> v1:172.21.15.62:6789/0 conn(0x7f069015dad0 legacy=0x7f069006b6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:40:07.370 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:07.362+0000 7f068dffb640 1 -- 172.21.15.16:0/1289025645 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 257310946 0 0) 0x7f0678003530 con 0x7f069015d680 2023-05-22T15:40:07.370 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:07.362+0000 7f068effd640 1 -- 172.21.15.16:0/1289025645 >> v1:172.21.15.16:6789/0 conn(0x7f069015d680 legacy=0x7f06900cb260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:40:07.371 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:07.362+0000 7f068effd640 1 -- 172.21.15.16:0/1289025645 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0690071140 con 0x7f0690159970 2023-05-22T15:40:07.371 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:07.366+0000 7f068effd640 1 --2- 172.21.15.16:0/1289025645 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0690159970 0x7f0690159d40 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f06800097f0 tx=0x7f068001c320 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6af97a2795a13be4 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:40:07.371 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:07.366+0000 7f068dffb640 1 -- 172.21.15.16:0/1289025645 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0684001680 con 0x7f0690159970 2023-05-22T15:40:07.371 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:07.366+0000 7f068dffb640 1 -- 172.21.15.16:0/1289025645 <== mon.2 v2:172.21.15.62:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06800a7050 con 0x7f0690159970 2023-05-22T15:40:07.372 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:07.366+0000 7f068dffb640 1 -- 172.21.15.16:0/1289025645 <== mon.2 v2:172.21.15.62:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06800ab600 con 0x7f0690159970 2023-05-22T15:40:07.372 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:07.366+0000 7f069561d640 1 -- 172.21.15.16:0/1289025645 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0690159970 msgr2=0x7f0690159d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:40:07.373 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:07.366+0000 7f069561d640 1 --2- 172.21.15.16:0/1289025645 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0690159970 0x7f0690159d40 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f06800097f0 tx=0x7f068001c320 comp rx=0 tx=0).stop 2023-05-22T15:40:07.373 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:07.366+0000 7f069561d640 1 -- 172.21.15.16:0/1289025645 shutdown_connections 2023-05-22T15:40:07.373 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:07.366+0000 7f069561d640 1 --2- 172.21.15.16:0/1289025645 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0690159970 0x7f0690159d40 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:07.373 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:07.366+0000 7f069561d640 1 -- 172.21.15.16:0/1289025645 >> 172.21.15.16:0/1289025645 conn(0x7f0690069720 msgr2=0x7f0690069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:40:07.373 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:07.366+0000 7f069561d640 1 -- 172.21.15.16:0/1289025645 shutdown_connections 2023-05-22T15:40:07.373 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:07.366+0000 7f069561d640 1 -- 172.21.15.16:0/1289025645 wait complete. 2023-05-22T15:40:07.374 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:07.370+0000 7f069561d640 1 Processor -- start 2023-05-22T15:40:07.374 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:07.370+0000 7f069561d640 1 -- start start 2023-05-22T15:40:07.375 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:07.370+0000 7f069561d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0690159970 0x7f0690157da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:40:07.375 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:07.370+0000 7f069561d640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f069015d680 0x7f06900c1b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:40:07.376 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:07.370+0000 7f069561d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f069015dad0 0x7f06900c2080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:40:07.376 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:07.370+0000 7f069561d640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0690153af0 con 0x7f069015dad0 2023-05-22T15:40:07.376 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:07.370+0000 7f069561d640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06900c25c0 con 0x7f0690159970 2023-05-22T15:40:07.376 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:07.370+0000 7f069561d640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06900b1fd0 con 0x7f069015d680 2023-05-22T15:40:07.376 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:07.370+0000 7f068f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f069015dad0 0x7f06900c2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:40:07.376 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:07.370+0000 7f068f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f069015dad0 0x7f06900c2080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34402/0 (socket says 172.21.15.16:34402) 2023-05-22T15:40:07.377 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:07.370+0000 7f068f7fe640 1 -- 172.21.15.16:0/317989570 learned_addr learned my addr 172.21.15.16:0/317989570 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:40:07.377 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:07.370+0000 7f068e7fc640 1 --2- 172.21.15.16:0/317989570 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f069015d680 0x7f06900c1b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:40:07.377 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:07.370+0000 7f068effd640 1 --2- 172.21.15.16:0/317989570 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0690159970 0x7f0690157da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:40:07.377 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:07.370+0000 7f068f7fe640 1 -- 172.21.15.16:0/317989570 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f069015d680 msgr2=0x7f06900c1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:40:07.377 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:07.370+0000 7f068f7fe640 1 --2- 172.21.15.16:0/317989570 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f069015d680 0x7f06900c1b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:07.377 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:07.370+0000 7f068f7fe640 1 -- 172.21.15.16:0/317989570 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0690159970 msgr2=0x7f0690157da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:40:07.377 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:07.370+0000 7f068f7fe640 1 --2- 172.21.15.16:0/317989570 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0690159970 0x7f0690157da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:07.377 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:07.370+0000 7f068f7fe640 1 -- 172.21.15.16:0/317989570 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0680009420 con 0x7f069015dad0 2023-05-22T15:40:07.378 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:07.370+0000 7f068f7fe640 1 --2- 172.21.15.16:0/317989570 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f069015dad0 0x7f06900c2080 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7f068800c760 tx=0x7f068800cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:40:07.379 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:07.370+0000 7f06777fe640 1 -- 172.21.15.16:0/317989570 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0688003b80 con 0x7f069015dad0 2023-05-22T15:40:07.379 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:07.370+0000 7f06777fe640 1 -- 172.21.15.16:0/317989570 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0688003ce0 con 0x7f069015dad0 2023-05-22T15:40:07.379 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:07.370+0000 7f069561d640 1 -- 172.21.15.16:0/317989570 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06900cae20 con 0x7f069015dad0 2023-05-22T15:40:07.379 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:07.370+0000 7f06777fe640 1 -- 172.21.15.16:0/317989570 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f068800aa30 con 0x7f069015dad0 2023-05-22T15:40:07.379 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:07.370+0000 7f069561d640 1 -- 172.21.15.16:0/317989570 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06900cac90 con 0x7f069015dad0 2023-05-22T15:40:07.379 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:07.370+0000 7f06777fe640 1 -- 172.21.15.16:0/317989570 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f068800ab90 con 0x7f069015dad0 2023-05-22T15:40:07.379 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:07.370+0000 7f069561d640 1 -- 172.21.15.16:0/317989570 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f069006bdd0 con 0x7f069015dad0 2023-05-22T15:40:07.380 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:07.374+0000 7f06777fe640 1 --2- 172.21.15.16:0/317989570 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f067c0654a0 0x7f067c067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:40:07.380 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:07.374+0000 7f068effd640 1 --2- 172.21.15.16:0/317989570 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f067c0654a0 0x7f067c067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:40:07.380 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:07.374+0000 7f06777fe640 1 -- 172.21.15.16:0/317989570 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f06880827f0 con 0x7f069015dad0 2023-05-22T15:40:07.380 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:07.374+0000 7f068effd640 1 --2- 172.21.15.16:0/317989570 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f067c0654a0 0x7f067c067950 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f06800ad000 tx=0x7f06800a6000 comp rx=0 tx=0).ready entity=mgr.6240 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:40:07.382 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:07.378+0000 7f06777fe640 1 -- 172.21.15.16:0/317989570 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f0688050210 con 0x7f069015dad0 2023-05-22T15:40:07.588 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:07.582+0000 7f069561d640 1 -- 172.21.15.16:0/317989570 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f069004dbe0 con 0x7f069015dad0 2023-05-22T15:40:07.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:07.590+0000 7f06777fe640 1 -- 172.21.15.16:0/317989570 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v18) v1 ==== 190+0+0 (secure 0 0 0) 0x7f0688053830 con 0x7f069015dad0 2023-05-22T15:40:07.597 INFO:teuthology.orchestra.run.smithi016.stderr:updated caps for client.0 2023-05-22T15:40:07.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:07.594+0000 7f069561d640 1 -- 172.21.15.16:0/317989570 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f067c0654a0 msgr2=0x7f067c067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:40:07.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:07.594+0000 7f069561d640 1 --2- 172.21.15.16:0/317989570 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f067c0654a0 0x7f067c067950 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f06800ad000 tx=0x7f06800a6000 comp rx=0 tx=0).stop 2023-05-22T15:40:07.600 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:07.594+0000 7f069561d640 1 -- 172.21.15.16:0/317989570 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f069015dad0 msgr2=0x7f06900c2080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:40:07.600 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:07.594+0000 7f069561d640 1 --2- 172.21.15.16:0/317989570 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f069015dad0 0x7f06900c2080 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7f068800c760 tx=0x7f068800cc20 comp rx=0 tx=0).stop 2023-05-22T15:40:07.600 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:07.594+0000 7f069561d640 1 -- 172.21.15.16:0/317989570 shutdown_connections 2023-05-22T15:40:07.600 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:07.594+0000 7f069561d640 1 --2- 172.21.15.16:0/317989570 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f069015dad0 0x7f06900c2080 unknown :-1 s=CLOSED pgs=928 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:07.600 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:07.594+0000 7f069561d640 1 --2- 172.21.15.16:0/317989570 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f067c0654a0 0x7f067c067950 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:07.600 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:07.594+0000 7f069561d640 1 --2- 172.21.15.16:0/317989570 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f069015d680 0x7f06900c1b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:07.600 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:07.594+0000 7f069561d640 1 --2- 172.21.15.16:0/317989570 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0690159970 0x7f0690157da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:07.601 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:07.594+0000 7f069561d640 1 -- 172.21.15.16:0/317989570 >> 172.21.15.16:0/317989570 conn(0x7f0690069720 msgr2=0x7f0690164d60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:40:07.601 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:07.594+0000 7f069561d640 1 -- 172.21.15.16:0/317989570 shutdown_connections 2023-05-22T15:40:07.601 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:07.594+0000 7f069561d640 1 -- 172.21.15.16:0/317989570 wait complete. 2023-05-22T15:40:07.614 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-05-22T15:40:07.737 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:07.730+0000 7fd181640640 1 Processor -- start 2023-05-22T15:40:07.737 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:07.730+0000 7fd181640640 1 -- start start 2023-05-22T15:40:07.738 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:07.734+0000 7fd181640640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd17c0cd730 0x7fd17c0afb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:40:07.739 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:07.734+0000 7fd181640640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fd17c0b0080 0x7fd17c0acba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:40:07.739 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:07.734+0000 7fd181640640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd17c0cc300 con 0x7fd17c1531d0 2023-05-22T15:40:07.739 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:07.734+0000 7fd181640640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd17c15d330 con 0x7fd17c0cd730 2023-05-22T15:40:07.739 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:07.734+0000 7fd181640640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd17c15c4a0 con 0x7fd17c0b0080 2023-05-22T15:40:07.739 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:07.734+0000 7fd17a7fc640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd17c0cd730 0x7fd17c0afb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:40:07.739 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:07.734+0000 7fd17a7fc640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd17c0cd730 0x7fd17c0afb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34412/0 (socket says 172.21.15.16:34412) 2023-05-22T15:40:07.740 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:07.734+0000 7fd17a7fc640 1 -- 172.21.15.16:0/3646456489 learned_addr learned my addr 172.21.15.16:0/3646456489 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:40:07.740 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:07.734+0000 7fd17b7fe640 1 --2- 172.21.15.16:0/3646456489 >> v2:172.21.15.62:3300/0 conn(0x7fd17c0b0080 0x7fd17c0acba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:40:07.740 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:07.734+0000 7fd17a7fc640 1 -- 172.21.15.16:0/3646456489 >> v2:172.21.15.62:3300/0 conn(0x7fd17c0b0080 msgr2=0x7fd17c0acba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:40:07.740 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:07.734+0000 7fd17a7fc640 1 --2- 172.21.15.16:0/3646456489 >> v2:172.21.15.62:3300/0 conn(0x7fd17c0b0080 0x7fd17c0acba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:07.740 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:07.734+0000 7fd17a7fc640 1 -- 172.21.15.16:0/3646456489 >> v1:172.21.15.62:6789/0 conn(0x7fd17c1531d0 legacy=0x7fd17c0cbbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:40:07.740 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:07.734+0000 7fd17a7fc640 1 -- 172.21.15.16:0/3646456489 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd17c0b04c0 con 0x7fd17c0cd730 2023-05-22T15:40:07.740 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:07.734+0000 7fd17b7fe640 1 --2- 172.21.15.16:0/3646456489 >> v2:172.21.15.62:3300/0 conn(0x7fd17c0b0080 0x7fd17c0acba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T15:40:07.741 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:07.734+0000 7fd17a7fc640 1 --2- 172.21.15.16:0/3646456489 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd17c0cd730 0x7fd17c0afb40 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7fd17c06f3e0 tx=0x7fd17001c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5bdb54634996b415 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:40:07.742 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:07.734+0000 7fd179ffb640 1 -- 172.21.15.16:0/3646456489 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1700a6040 con 0x7fd17c0cd730 2023-05-22T15:40:07.742 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:07.734+0000 7fd179ffb640 1 -- 172.21.15.16:0/3646456489 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1700aa400 con 0x7fd17c0cd730 2023-05-22T15:40:07.742 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:07.734+0000 7fd179ffb640 1 -- 172.21.15.16:0/3646456489 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1700aa790 con 0x7fd17c0cd730 2023-05-22T15:40:07.742 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:07.734+0000 7fd181640640 1 -- 172.21.15.16:0/3646456489 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd17c0cd730 msgr2=0x7fd17c0afb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:40:07.742 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:07.734+0000 7fd181640640 1 --2- 172.21.15.16:0/3646456489 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd17c0cd730 0x7fd17c0afb40 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7fd17c06f3e0 tx=0x7fd17001c4a0 comp rx=0 tx=0).stop 2023-05-22T15:40:07.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:07.734+0000 7fd181640640 1 -- 172.21.15.16:0/3646456489 shutdown_connections 2023-05-22T15:40:07.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:07.734+0000 7fd181640640 1 --2- 172.21.15.16:0/3646456489 >> v2:172.21.15.62:3300/0 conn(0x7fd17c0b0080 0x7fd17c0acba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:07.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:07.734+0000 7fd181640640 1 --2- 172.21.15.16:0/3646456489 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd17c0cd730 0x7fd17c0afb40 unknown :-1 s=CLOSED pgs=929 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:07.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:07.734+0000 7fd181640640 1 -- 172.21.15.16:0/3646456489 >> 172.21.15.16:0/3646456489 conn(0x7fd17c069720 msgr2=0x7fd17c069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:40:07.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:07.734+0000 7fd181640640 1 -- 172.21.15.16:0/3646456489 shutdown_connections 2023-05-22T15:40:07.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:07.734+0000 7fd181640640 1 -- 172.21.15.16:0/3646456489 wait complete. 2023-05-22T15:40:07.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:07.738+0000 7fd181640640 1 Processor -- start 2023-05-22T15:40:07.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:07.738+0000 7fd181640640 1 -- start start 2023-05-22T15:40:07.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:07.738+0000 7fd181640640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd17c0b0080 0x7fd17c0ca440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:40:07.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:07.738+0000 7fd181640640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd17c0cd730 0x7fd17c1493d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:40:07.746 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:07.738+0000 7fd181640640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd17c1531d0 0x7fd17c15ea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:40:07.746 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:07.738+0000 7fd181640640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd17c149b50 con 0x7fd17c0cd730 2023-05-22T15:40:07.746 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:07.738+0000 7fd181640640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd17c149910 con 0x7fd17c1531d0 2023-05-22T15:40:07.746 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:07.738+0000 7fd181640640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd17c149df0 con 0x7fd17c0b0080 2023-05-22T15:40:07.746 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:07.738+0000 7fd17a7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd17c0cd730 0x7fd17c1493d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:40:07.746 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:07.738+0000 7fd17a7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd17c0cd730 0x7fd17c1493d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34428/0 (socket says 172.21.15.16:34428) 2023-05-22T15:40:07.746 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:07.738+0000 7fd17a7fc640 1 -- 172.21.15.16:0/4191410016 learned_addr learned my addr 172.21.15.16:0/4191410016 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:40:07.747 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:07.738+0000 7fd17affd640 1 --2- 172.21.15.16:0/4191410016 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd17c0b0080 0x7fd17c0ca440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:40:07.747 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:07.738+0000 7fd17b7fe640 1 --2- 172.21.15.16:0/4191410016 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd17c1531d0 0x7fd17c15ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:40:07.747 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:07.738+0000 7fd17a7fc640 1 -- 172.21.15.16:0/4191410016 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd17c0b0080 msgr2=0x7fd17c0ca440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:40:07.747 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:07.738+0000 7fd17a7fc640 1 --2- 172.21.15.16:0/4191410016 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd17c0b0080 0x7fd17c0ca440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:07.747 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:07.738+0000 7fd17a7fc640 1 -- 172.21.15.16:0/4191410016 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd17c1531d0 msgr2=0x7fd17c15ea80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:40:07.747 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:07.738+0000 7fd17a7fc640 1 --2- 172.21.15.16:0/4191410016 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd17c1531d0 0x7fd17c15ea80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:07.747 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:07.738+0000 7fd17a7fc640 1 -- 172.21.15.16:0/4191410016 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1700094d0 con 0x7fd17c0cd730 2023-05-22T15:40:07.748 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:07.738+0000 7fd17a7fc640 1 --2- 172.21.15.16:0/4191410016 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd17c0cd730 0x7fd17c1493d0 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7fd170000c10 tx=0x7fd17001ace0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:40:07.748 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:07.738+0000 7fd1637fe640 1 -- 172.21.15.16:0/4191410016 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1700b3800 con 0x7fd17c0cd730 2023-05-22T15:40:07.748 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:07.738+0000 7fd1637fe640 1 -- 172.21.15.16:0/4191410016 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1700b3960 con 0x7fd17c0cd730 2023-05-22T15:40:07.748 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:07.738+0000 7fd181640640 1 -- 172.21.15.16:0/4191410016 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd17c15cc70 con 0x7fd17c0cd730 2023-05-22T15:40:07.748 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:07.738+0000 7fd1637fe640 1 -- 172.21.15.16:0/4191410016 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1700b3d10 con 0x7fd17c0cd730 2023-05-22T15:40:07.748 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:07.738+0000 7fd181640640 1 -- 172.21.15.16:0/4191410016 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd17c15c990 con 0x7fd17c0cd730 2023-05-22T15:40:07.749 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:07.738+0000 7fd181640640 1 -- 172.21.15.16:0/4191410016 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd17c15cf50 con 0x7fd17c0cd730 2023-05-22T15:40:07.749 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:07.742+0000 7fd1637fe640 1 -- 172.21.15.16:0/4191410016 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) v1 ==== 84332+0+0 (secure 0 0 0) 0x7fd1700c6430 con 0x7fd17c0cd730 2023-05-22T15:40:07.750 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:07.742+0000 7fd1637fe640 1 --2- 172.21.15.16:0/4191410016 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd1680654d0 0x7fd168067980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:40:07.750 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:07.742+0000 7fd1637fe640 1 -- 172.21.15.16:0/4191410016 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 6382+0+0 (secure 0 0 0) 0x7fd17011d8d0 con 0x7fd17c0cd730 2023-05-22T15:40:07.750 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:07.742+0000 7fd17affd640 1 --2- 172.21.15.16:0/4191410016 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd1680654d0 0x7fd168067980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:40:07.750 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:07.742+0000 7fd17affd640 1 --2- 172.21.15.16:0/4191410016 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd1680654d0 0x7fd168067980 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fd16c00b810 tx=0x7fd16c008040 comp rx=0 tx=0).ready entity=mgr.6240 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:40:07.750 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:07.746+0000 7fd1637fe640 1 -- 172.21.15.16:0/4191410016 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fd1700eb220 con 0x7fd17c0cd730 2023-05-22T15:40:07.935 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:07.930+0000 7fd181640640 1 -- 172.21.15.16:0/4191410016 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd17c04dbe0 con 0x7fd17c0cd730 2023-05-22T15:40:07.935 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:07.930+0000 7fd1637fe640 1 -- 172.21.15.16:0/4191410016 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) v1 ==== 94+0+4645 (secure 0 0 0) 0x7fd1700aae60 con 0x7fd17c0cd730 2023-05-22T15:40:07.936 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:40:07.936 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":22,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5399,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6833/1859135164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":1859135164},{"type":"v1","addr":"172.21.15.62:6833","nonce":1859135164}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":17},{"gid":5836,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/3067106582","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":3067106582},{"type":"v1","addr":"172.21.15.62:6835","nonce":3067106582}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":17},{"gid":6462,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/717991089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":717991089},{"type":"v1","addr":"172.21.15.16:6835","nonce":717991089}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":17}],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-05-22T15:40:02.754483+0000","modified":"2023-05-22T15:40:05.773388+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6468},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6468":{"gid":6468,"name":"a","rank":0,"incarnation":18,"state":"up:active","state_seq":3,"addr":"172.21.15.16:6837/4263396864","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":4263396864},{"type":"v1","addr":"172.21.15.16:6837","nonce":4263396864}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2023-05-22T15:40:07.937 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 22 2023-05-22T15:40:07.938 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:07.930+0000 7fd181640640 1 -- 172.21.15.16:0/4191410016 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd1680654d0 msgr2=0x7fd168067980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:40:07.938 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:07.930+0000 7fd181640640 1 --2- 172.21.15.16:0/4191410016 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd1680654d0 0x7fd168067980 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fd16c00b810 tx=0x7fd16c008040 comp rx=0 tx=0).stop 2023-05-22T15:40:07.938 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:07.930+0000 7fd181640640 1 -- 172.21.15.16:0/4191410016 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd17c0cd730 msgr2=0x7fd17c1493d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:40:07.938 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:07.930+0000 7fd181640640 1 --2- 172.21.15.16:0/4191410016 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd17c0cd730 0x7fd17c1493d0 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7fd170000c10 tx=0x7fd17001ace0 comp rx=0 tx=0).stop 2023-05-22T15:40:07.938 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:07.930+0000 7fd181640640 1 -- 172.21.15.16:0/4191410016 shutdown_connections 2023-05-22T15:40:07.939 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:07.930+0000 7fd181640640 1 --2- 172.21.15.16:0/4191410016 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd17c1531d0 0x7fd17c15ea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:07.939 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:07.930+0000 7fd181640640 1 --2- 172.21.15.16:0/4191410016 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd17c0cd730 0x7fd17c1493d0 unknown :-1 s=CLOSED pgs=930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:07.939 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:07.930+0000 7fd181640640 1 --2- 172.21.15.16:0/4191410016 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd1680654d0 0x7fd168067980 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:07.939 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:07.930+0000 7fd181640640 1 --2- 172.21.15.16:0/4191410016 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd17c0b0080 0x7fd17c0ca440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:07.939 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:07.934+0000 7fd181640640 1 -- 172.21.15.16:0/4191410016 >> 172.21.15.16:0/4191410016 conn(0x7fd17c069720 msgr2=0x7fd17c0ce650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:40:07.939 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:07.934+0000 7fd181640640 1 -- 172.21.15.16:0/4191410016 shutdown_connections 2023-05-22T15:40:07.939 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:07.934+0000 7fd181640640 1 -- 172.21.15.16:0/4191410016 wait complete. 2023-05-22T15:40:08.355 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-05-22T15:40:02.754483+0000', 'modified': '2023-05-22T15:40:05.773388+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6468}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6468': {'gid': 6468, 'name': 'a', 'rank': 0, 'incarnation': 18, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.16:6837/4263396864', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6836', 'nonce': 4263396864}, {'type': 'v1', 'addr': '172.21.15.16:6837', 'nonce': 4263396864}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [9], 'metadata_pool': 8, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-05-22T15:40:08.356 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-05-22T15:40:08.356 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2023-05-22T15:40:08.494 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.486+0000 7f5947489640 1 Processor -- start 2023-05-22T15:40:08.495 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.486+0000 7f5947489640 1 -- start start 2023-05-22T15:40:08.495 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.486+0000 7f5947489640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f59400ca4e0 0x7f594015e460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:40:08.495 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.486+0000 7f5947489640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f59400c9b40 0x7f59400afb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:40:08.495 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.486+0000 7f5947489640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5940070df0 0x7f5940071210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:40:08.495 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.486+0000 7f5947489640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59400b0650 con 0x7f59400c9b40 2023-05-22T15:40:08.496 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.486+0000 7f5947489640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59400b0370 con 0x7f59400ca4e0 2023-05-22T15:40:08.496 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.486+0000 7f5947489640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f594015d980 con 0x7f5940070df0 2023-05-22T15:40:08.496 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.486+0000 7f5944a24640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f59400c9b40 0x7f59400afb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:40:08.496 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.486+0000 7f5944a24640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f59400c9b40 0x7f59400afb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34442/0 (socket says 172.21.15.16:34442) 2023-05-22T15:40:08.496 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.486+0000 7f5944a24640 1 -- 172.21.15.16:0/3180628374 learned_addr learned my addr 172.21.15.16:0/3180628374 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:40:08.496 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.486+0000 7f5945a26640 1 --2- 172.21.15.16:0/3180628374 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5940070df0 0x7f5940071210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:40:08.496 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.486+0000 7f5945225640 1 --2- 172.21.15.16:0/3180628374 >> v2:172.21.15.62:3300/0 conn(0x7f59400ca4e0 0x7f594015e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:40:08.497 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.486+0000 7f5944a24640 1 -- 172.21.15.16:0/3180628374 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5940070df0 msgr2=0x7f5940071210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:40:08.497 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.486+0000 7f5944a24640 1 --2- 172.21.15.16:0/3180628374 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5940070df0 0x7f5940071210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:08.497 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.486+0000 7f5944a24640 1 -- 172.21.15.16:0/3180628374 >> v2:172.21.15.62:3300/0 conn(0x7f59400ca4e0 msgr2=0x7f594015e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:40:08.497 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.486+0000 7f5944a24640 1 --2- 172.21.15.16:0/3180628374 >> v2:172.21.15.62:3300/0 conn(0x7f59400ca4e0 0x7f594015e460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:08.497 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.486+0000 7f5944a24640 1 -- 172.21.15.16:0/3180628374 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f594015dc60 con 0x7f59400c9b40 2023-05-22T15:40:08.499 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.490+0000 7f5944a24640 1 --2- 172.21.15.16:0/3180628374 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f59400c9b40 0x7f59400afb40 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7f59400cad40 tx=0x7f593001c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b0af785add5de99 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:40:08.499 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.490+0000 7f5937fff640 1 -- 172.21.15.16:0/3180628374 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59300ab670 con 0x7f59400c9b40 2023-05-22T15:40:08.499 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.490+0000 7f5937fff640 1 -- 172.21.15.16:0/3180628374 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59300ab7d0 con 0x7f59400c9b40 2023-05-22T15:40:08.499 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.490+0000 7f5937fff640 1 -- 172.21.15.16:0/3180628374 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59300abb60 con 0x7f59400c9b40 2023-05-22T15:40:08.500 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.490+0000 7f5947489640 1 -- 172.21.15.16:0/3180628374 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f59400c9b40 msgr2=0x7f59400afb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:40:08.500 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.490+0000 7f5947489640 1 --2- 172.21.15.16:0/3180628374 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f59400c9b40 0x7f59400afb40 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7f59400cad40 tx=0x7f593001c4d0 comp rx=0 tx=0).stop 2023-05-22T15:40:08.500 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.490+0000 7f5947489640 1 -- 172.21.15.16:0/3180628374 shutdown_connections 2023-05-22T15:40:08.500 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.490+0000 7f5947489640 1 --2- 172.21.15.16:0/3180628374 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f59400c9b40 0x7f59400afb40 unknown :-1 s=CLOSED pgs=931 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:08.500 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.490+0000 7f5947489640 1 --2- 172.21.15.16:0/3180628374 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5940070df0 0x7f5940071210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:08.500 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.490+0000 7f5947489640 1 --2- 172.21.15.16:0/3180628374 >> v2:172.21.15.62:3300/0 conn(0x7f59400ca4e0 0x7f594015e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:08.501 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.490+0000 7f5947489640 1 -- 172.21.15.16:0/3180628374 >> 172.21.15.16:0/3180628374 conn(0x7f5940069720 msgr2=0x7f5940069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:40:08.501 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.490+0000 7f5947489640 1 -- 172.21.15.16:0/3180628374 shutdown_connections 2023-05-22T15:40:08.501 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.490+0000 7f5947489640 1 -- 172.21.15.16:0/3180628374 wait complete. 2023-05-22T15:40:08.501 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.494+0000 7f5947489640 1 Processor -- start 2023-05-22T15:40:08.501 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.494+0000 7f5947489640 1 -- start start 2023-05-22T15:40:08.501 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.494+0000 7f5947489640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5940070df0 0x7f5940153c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:40:08.501 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.494+0000 7f5947489640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f59400c9b40 0x7f5940152470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:40:08.502 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.494+0000 7f5947489640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f59400ca4e0 0x7f5940158450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:40:08.502 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.494+0000 7f5947489640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5940153800 con 0x7f5940070df0 2023-05-22T15:40:08.502 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.494+0000 7f5947489640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5940152b80 con 0x7f59400ca4e0 2023-05-22T15:40:08.502 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.494+0000 7f5947489640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5940152e60 con 0x7f59400c9b40 2023-05-22T15:40:08.502 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.494+0000 7f5945225640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5940070df0 0x7f5940153c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:40:08.502 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.494+0000 7f5945225640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5940070df0 0x7f5940153c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34458/0 (socket says 172.21.15.16:34458) 2023-05-22T15:40:08.502 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.494+0000 7f5945225640 1 -- 172.21.15.16:0/111924021 learned_addr learned my addr 172.21.15.16:0/111924021 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:40:08.502 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.494+0000 7f5944a24640 1 --2- 172.21.15.16:0/111924021 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f59400c9b40 0x7f5940152470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:40:08.503 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.494+0000 7f5945a26640 1 --2- 172.21.15.16:0/111924021 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f59400ca4e0 0x7f5940158450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:40:08.503 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.494+0000 7f5945225640 1 -- 172.21.15.16:0/111924021 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f59400c9b40 msgr2=0x7f5940152470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:40:08.503 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.494+0000 7f5945225640 1 --2- 172.21.15.16:0/111924021 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f59400c9b40 0x7f5940152470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:08.503 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.494+0000 7f5945225640 1 -- 172.21.15.16:0/111924021 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f59400ca4e0 msgr2=0x7f5940158450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:40:08.503 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.494+0000 7f5945225640 1 --2- 172.21.15.16:0/111924021 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f59400ca4e0 0x7f5940158450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:08.503 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.494+0000 7f5945225640 1 -- 172.21.15.16:0/111924021 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59300094d0 con 0x7f5940070df0 2023-05-22T15:40:08.506 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.494+0000 7f5945225640 1 --2- 172.21.15.16:0/111924021 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5940070df0 0x7f5940153c40 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7f593c00d5f0 tx=0x7f593c00dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:40:08.506 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.494+0000 7f5935ffb640 1 -- 172.21.15.16:0/111924021 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f593c0125f0 con 0x7f5940070df0 2023-05-22T15:40:08.506 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.494+0000 7f5935ffb640 1 -- 172.21.15.16:0/111924021 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f593c012750 con 0x7f5940070df0 2023-05-22T15:40:08.506 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.494+0000 7f5947489640 1 -- 172.21.15.16:0/111924021 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59400ca8b0 con 0x7f5940070df0 2023-05-22T15:40:08.507 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.494+0000 7f5935ffb640 1 -- 172.21.15.16:0/111924021 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f593c012ae0 con 0x7f5940070df0 2023-05-22T15:40:08.507 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.494+0000 7f5947489640 1 -- 172.21.15.16:0/111924021 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5940071750 con 0x7f5940070df0 2023-05-22T15:40:08.507 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.494+0000 7f5935ffb640 1 -- 172.21.15.16:0/111924021 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f593c012c40 con 0x7f5940070df0 2023-05-22T15:40:08.507 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.494+0000 7f5935ffb640 1 --2- 172.21.15.16:0/111924021 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f59240654a0 0x7f5924067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:40:08.507 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.498+0000 7f5935ffb640 1 -- 172.21.15.16:0/111924021 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f593c082ff0 con 0x7f5940070df0 2023-05-22T15:40:08.507 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.498+0000 7f5944a24640 1 --2- 172.21.15.16:0/111924021 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f59240654a0 0x7f5924067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:40:08.508 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.498+0000 7f5944a24640 1 --2- 172.21.15.16:0/111924021 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f59240654a0 0x7f5924067950 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f59300af000 tx=0x7f59300a7000 comp rx=0 tx=0).ready entity=mgr.6240 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:40:08.508 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.498+0000 7f594a5ab000 1 Processor -- start 2023-05-22T15:40:08.508 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.498+0000 7f594a5ab000 1 -- start start 2023-05-22T15:40:08.508 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.498+0000 7f594a5ab000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558a4fcb8fd0 0x558a4ff2bc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:40:08.508 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.498+0000 7f594a5ab000 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558a4ff2c170 0x558a4ff2c540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:40:08.508 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.498+0000 7f594a5ab000 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x558a4fedfa10 0x558a4fedfde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:40:08.508 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.498+0000 7f594a5ab000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x558a4fda9bd0 con 0x558a4fcb8fd0 2023-05-22T15:40:08.509 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.498+0000 7f594a5ab000 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x558a4fcc6dd0 con 0x558a4ff2c170 2023-05-22T15:40:08.509 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.498+0000 7f594a5ab000 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x558a4fd37640 con 0x558a4fedfa10 2023-05-22T15:40:08.509 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.498+0000 7f5945225640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558a4fcb8fd0 0x558a4ff2bc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:40:08.509 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.498+0000 7f5945225640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558a4fcb8fd0 0x558a4ff2bc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34468/0 (socket says 172.21.15.16:34468) 2023-05-22T15:40:08.509 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.498+0000 7f5945225640 1 -- 172.21.15.16:0/609647629 learned_addr learned my addr 172.21.15.16:0/609647629 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:40:08.509 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.498+0000 7f5944a24640 1 --2- 172.21.15.16:0/609647629 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558a4ff2c170 0x558a4ff2c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:40:08.509 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.498+0000 7f5945a26640 1 --2- 172.21.15.16:0/609647629 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x558a4fedfa10 0x558a4fedfde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:40:08.509 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.498+0000 7f5945225640 1 -- 172.21.15.16:0/609647629 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x558a4fedfa10 msgr2=0x558a4fedfde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:40:08.510 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.498+0000 7f5945225640 1 --2- 172.21.15.16:0/609647629 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x558a4fedfa10 0x558a4fedfde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:08.510 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.498+0000 7f5945225640 1 -- 172.21.15.16:0/609647629 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558a4ff2c170 msgr2=0x558a4ff2c540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:40:08.510 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.498+0000 7f5945225640 1 --2- 172.21.15.16:0/609647629 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558a4ff2c170 0x558a4ff2c540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:08.510 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.498+0000 7f5945225640 1 -- 172.21.15.16:0/609647629 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558a4fcf60b0 con 0x558a4fcb8fd0 2023-05-22T15:40:08.510 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.498+0000 7f5945a26640 1 --2- 172.21.15.16:0/609647629 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x558a4fedfa10 0x558a4fedfde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:40:08.510 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.498+0000 7f5944a24640 1 --2- 172.21.15.16:0/609647629 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558a4ff2c170 0x558a4ff2c540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:40:08.510 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.498+0000 7f5945225640 1 --2- 172.21.15.16:0/609647629 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558a4fcb8fd0 0x558a4ff2bc30 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7f593c00dfb0 tx=0x7f593c0540f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4ddcbc8a7683e5a server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:40:08.511 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.498+0000 7f592affd640 1 -- 172.21.15.16:0/609647629 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f593c06b020 con 0x558a4fcb8fd0 2023-05-22T15:40:08.511 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.498+0000 7f592affd640 1 -- 172.21.15.16:0/609647629 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f593c052af0 con 0x558a4fcb8fd0 2023-05-22T15:40:08.511 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.498+0000 7f592affd640 1 -- 172.21.15.16:0/609647629 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f593c0581b0 con 0x558a4fcb8fd0 2023-05-22T15:40:08.511 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.498+0000 7f594a5ab000 1 -- 172.21.15.16:0/609647629 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558a4fcb8fd0 msgr2=0x558a4ff2bc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:40:08.511 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.498+0000 7f594a5ab000 1 --2- 172.21.15.16:0/609647629 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558a4fcb8fd0 0x558a4ff2bc30 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7f593c00dfb0 tx=0x7f593c0540f0 comp rx=0 tx=0).stop 2023-05-22T15:40:08.511 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.498+0000 7f594a5ab000 1 -- 172.21.15.16:0/609647629 shutdown_connections 2023-05-22T15:40:08.511 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.498+0000 7f594a5ab000 1 --2- 172.21.15.16:0/609647629 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x558a4fedfa10 0x558a4fedfde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:08.511 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.498+0000 7f594a5ab000 1 --2- 172.21.15.16:0/609647629 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558a4ff2c170 0x558a4ff2c540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:08.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.498+0000 7f594a5ab000 1 --2- 172.21.15.16:0/609647629 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558a4fcb8fd0 0x558a4ff2bc30 unknown :-1 s=CLOSED pgs=933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:08.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.498+0000 7f594a5ab000 1 -- 172.21.15.16:0/609647629 >> 172.21.15.16:0/609647629 conn(0x558a4fd88270 msgr2=0x558a4ff0f220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:40:08.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.498+0000 7f594a5ab000 1 -- 172.21.15.16:0/609647629 shutdown_connections 2023-05-22T15:40:08.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.498+0000 7f594a5ab000 1 -- 172.21.15.16:0/609647629 wait complete. 2023-05-22T15:40:08.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.498+0000 7f594a5ab000 1 -- start start 2023-05-22T15:40:08.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.498+0000 7f594a5ab000 1 Processor -- start 2023-05-22T15:40:08.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.502+0000 7f594a5ab000 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x558a4fcb8fd0 0x558a4fee7a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:40:08.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.502+0000 7f594a5ab000 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558a4fedfa10 0x558a4fee7fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:40:08.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.502+0000 7f594a5ab000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558a4ff2c170 0x558a4ff114a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:40:08.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.502+0000 7f594a5ab000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x558a4fd30780 con 0x558a4ff2c170 2023-05-22T15:40:08.516 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.502+0000 7f594a5ab000 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x558a4fcee250 con 0x558a4fedfa10 2023-05-22T15:40:08.516 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.502+0000 7f594a5ab000 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x558a4fcedef0 con 0x558a4fcb8fd0 2023-05-22T15:40:08.516 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.502+0000 7f594a5ab000 10 client.0 _ll_register_callbacks cb 0x558a4fecdb50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-05-22T15:40:08.516 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.502+0000 7f5945a26640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558a4ff2c170 0x558a4ff114a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:40:08.516 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.502+0000 7f5945a26640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558a4ff2c170 0x558a4ff114a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34484/0 (socket says 172.21.15.16:34484) 2023-05-22T15:40:08.516 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.502+0000 7f5945a26640 1 -- 172.21.15.16:0/3965856476 learned_addr learned my addr 172.21.15.16:0/3965856476 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:40:08.516 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.502+0000 7f5945225640 1 --2- 172.21.15.16:0/3965856476 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x558a4fcb8fd0 0x558a4fee7a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:40:08.517 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.502+0000 7f5944a24640 1 --2- 172.21.15.16:0/3965856476 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558a4fedfa10 0x558a4fee7fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:40:08.517 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.502+0000 7f5945a26640 1 -- 172.21.15.16:0/3965856476 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x558a4fcb8fd0 msgr2=0x558a4fee7a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:40:08.517 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.502+0000 7f5945a26640 1 --2- 172.21.15.16:0/3965856476 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x558a4fcb8fd0 0x558a4fee7a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:08.517 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.502+0000 7f5945a26640 1 -- 172.21.15.16:0/3965856476 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558a4fedfa10 msgr2=0x558a4fee7fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:40:08.517 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.502+0000 7f5945a26640 1 --2- 172.21.15.16:0/3965856476 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558a4fedfa10 0x558a4fee7fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:08.517 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.502+0000 7f5945a26640 1 -- 172.21.15.16:0/3965856476 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5940071750 con 0x558a4ff2c170 2023-05-22T15:40:08.517 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.502+0000 7f5944a24640 1 --2- 172.21.15.16:0/3965856476 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558a4fedfa10 0x558a4fee7fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:40:08.517 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.502+0000 7f5945225640 1 --2- 172.21.15.16:0/3965856476 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x558a4fcb8fd0 0x558a4fee7a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:40:08.518 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.502+0000 7f5945a26640 1 --2- 172.21.15.16:0/3965856476 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558a4ff2c170 0x558a4ff114a0 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7f59380079a0 tx=0x7f5938007e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e5ca23ccdcdd615 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:40:08.518 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.502+0000 7f591ffff640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2023-05-22T15:40:08.518 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.502+0000 7f591ffff640 1 -- 172.21.15.16:0/3965856476 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5938010a10 con 0x558a4ff2c170 2023-05-22T15:40:08.518 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.502+0000 7f591ffff640 1 -- 172.21.15.16:0/3965856476 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(73..73 src has 1..73) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f5938010b70 con 0x558a4ff2c170 2023-05-22T15:40:08.518 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.502+0000 7f594a5ab000 1 -- 172.21.15.16:0/3965856476 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x558a4fcf4c60 con 0x558a4ff2c170 2023-05-22T15:40:08.518 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.502+0000 7f591ffff640 1 -- 172.21.15.16:0/3965856476 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5938010d30 con 0x558a4ff2c170 2023-05-22T15:40:08.518 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.502+0000 7f594a5ab000 10 client.6621 fetch_fsmap learned FSMap version 22 2023-05-22T15:40:08.518 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.502+0000 7f594a5ab000 1 -- 172.21.15.16:0/3965856476 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=2+}) v3 -- 0x558a4fa2d890 con 0x558a4ff2c170 2023-05-22T15:40:08.519 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.502+0000 7f591ffff640 1 -- 172.21.15.16:0/3965856476 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x558a4fcf4c60 con 0x558a4ff2c170 2023-05-22T15:40:08.519 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.502+0000 7f591ffff640 1 -- 172.21.15.16:0/3965856476 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 22) v1 ==== 3106+0+0 (secure 0 0 0) 0x7f593801f350 con 0x558a4ff2c170 2023-05-22T15:40:08.519 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.502+0000 7f594a5ab000 10 client.6621 fetch_fsmap finished waiting for FSMap version 22 2023-05-22T15:40:08.519 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.502+0000 7f594a5ab000 10 client.6621 resolve_mds: resolved name 'a' to daemon mds.a 2023-05-22T15:40:08.519 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.502+0000 7f594a5ab000 20 client.6621 populate_metadata read hostname 'smithi016' 2023-05-22T15:40:08.519 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.502+0000 7f594a5ab000 1 --2- 172.21.15.16:0/3965856476 >> [v2:172.21.15.16:6836/4263396864,v1:172.21.15.16:6837/4263396864] conn(0x558a4ff3b8e0 0x558a4ff3dcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:40:08.519 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.502+0000 7f594a5ab000 4 client.6621 mds_command: new command op to 6468 tid=0[{"prefix": "get_command_descriptions"}] 2023-05-22T15:40:08.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.502+0000 7f594a5ab000 1 -- 172.21.15.16:0/3965856476 --> [v2:172.21.15.16:6836/4263396864,v1:172.21.15.16:6837/4263396864] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x558a4fd2f0d0 con 0x558a4ff3b8e0 2023-05-22T15:40:08.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.502+0000 7f5945225640 1 --2- 172.21.15.16:0/3965856476 >> [v2:172.21.15.16:6836/4263396864,v1:172.21.15.16:6837/4263396864] conn(0x558a4ff3b8e0 0x558a4ff3dcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:40:08.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.502+0000 7f591ffff640 10 client.6621 ms_handle_connect on v2:172.21.15.16:6836/4263396864 2023-05-22T15:40:08.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.502+0000 7f5945225640 1 --2- 172.21.15.16:0/3965856476 >> [v2:172.21.15.16:6836/4263396864,v1:172.21.15.16:6837/4263396864] conn(0x558a4ff3b8e0 0x558a4ff3dcc0 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6220ec9e369aec92 server_cookie=596c787b5b036886 in_seq=0 out_seq=0 2023-05-22T15:40:08.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.502+0000 7f591ffff640 1 -- 172.21.15.16:0/3965856476 <== mds.0 v2:172.21.15.16:6836/4263396864 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x558a4fd2f0d0 con 0x558a4ff3b8e0 2023-05-22T15:40:08.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.502+0000 7f591ffff640 10 client.6621 handle_command_reply: tid=0 2023-05-22T15:40:08.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.502+0000 7f594a5ab000 1 client.6621 shutdown 2023-05-22T15:40:08.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.502+0000 7f594a5ab000 1 -- 172.21.15.16:0/3965856476 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558a4ff2c170 msgr2=0x558a4ff114a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:40:08.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.502+0000 7f594a5ab000 1 --2- 172.21.15.16:0/3965856476 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558a4ff2c170 0x558a4ff114a0 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7f59380079a0 tx=0x7f5938007e60 comp rx=0 tx=0).stop 2023-05-22T15:40:08.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.502+0000 7f594a5ab000 1 -- 172.21.15.16:0/3965856476 shutdown_connections 2023-05-22T15:40:08.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.502+0000 7f594a5ab000 1 --2- 172.21.15.16:0/3965856476 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558a4ff2c170 0x558a4ff114a0 unknown :-1 s=CLOSED pgs=934 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:08.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.502+0000 7f594a5ab000 1 --2- 172.21.15.16:0/3965856476 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558a4fedfa10 0x558a4fee7fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:08.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.502+0000 7f594a5ab000 1 --2- 172.21.15.16:0/3965856476 >> [v2:172.21.15.16:6836/4263396864,v1:172.21.15.16:6837/4263396864] conn(0x558a4ff3b8e0 0x558a4ff3dcc0 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:08.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.502+0000 7f594a5ab000 1 --2- 172.21.15.16:0/3965856476 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x558a4fcb8fd0 0x558a4fee7a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:08.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.502+0000 7f591ffff640 0 client.6621 ms_handle_reset on v2:172.21.15.16:6836/4263396864 2023-05-22T15:40:08.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.502+0000 7f594a5ab000 1 -- 172.21.15.16:0/3965856476 >> 172.21.15.16:0/3965856476 conn(0x558a4fd88270 msgr2=0x558a4fedf100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:40:08.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.502+0000 7f594a5ab000 1 -- 172.21.15.16:0/3965856476 shutdown_connections 2023-05-22T15:40:08.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.502+0000 7f594a5ab000 1 -- 172.21.15.16:0/3965856476 wait complete. 2023-05-22T15:40:08.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.502+0000 7f594a5ab000 20 client.6621 trim_cache size 0 max 16384 2023-05-22T15:40:08.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.518+0000 7f594a5ab000 1 Processor -- start 2023-05-22T15:40:08.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.518+0000 7f594a5ab000 1 -- start start 2023-05-22T15:40:08.526 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.522+0000 7f594a5ab000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558a4fcb8fd0 0x558a4fedfde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:40:08.527 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.522+0000 7f594a5ab000 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x558a4fedfa10 0x558a4fee0320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:40:08.527 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.522+0000 7f594a5ab000 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558a4ff2c170 0x558a4fee0860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:40:08.527 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.522+0000 7f594a5ab000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x558a4f9ee940 con 0x558a4fcb8fd0 2023-05-22T15:40:08.527 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.522+0000 7f594a5ab000 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x558a4fd332e0 con 0x558a4ff2c170 2023-05-22T15:40:08.527 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.522+0000 7f594a5ab000 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x558a4fd2f610 con 0x558a4fedfa10 2023-05-22T15:40:08.527 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.522+0000 7f5945225640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558a4fcb8fd0 0x558a4fedfde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:40:08.527 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.522+0000 7f5945225640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558a4fcb8fd0 0x558a4fedfde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34500/0 (socket says 172.21.15.16:34500) 2023-05-22T15:40:08.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.522+0000 7f5945225640 1 -- 172.21.15.16:0/667049256 learned_addr learned my addr 172.21.15.16:0/667049256 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:40:08.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.522+0000 7f5945225640 1 -- 172.21.15.16:0/667049256 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x558a4fedfa10 msgr2=0x558a4fee0320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:40:08.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.522+0000 7f5945225640 1 --2- 172.21.15.16:0/667049256 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x558a4fedfa10 0x558a4fee0320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:08.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.522+0000 7f5945225640 1 -- 172.21.15.16:0/667049256 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558a4ff2c170 msgr2=0x558a4fee0860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:40:08.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.522+0000 7f5945225640 1 --2- 172.21.15.16:0/667049256 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558a4ff2c170 0x558a4fee0860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:08.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.522+0000 7f5945225640 1 -- 172.21.15.16:0/667049256 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558a4fcf6360 con 0x558a4fcb8fd0 2023-05-22T15:40:08.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.522+0000 7f5945225640 1 --2- 172.21.15.16:0/667049256 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558a4fcb8fd0 0x558a4fedfde0 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7f593c00d5c0 tx=0x7f593c01f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15a407beb1615210 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:40:08.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.522+0000 7f592b7fe640 1 -- 172.21.15.16:0/667049256 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f593c06b070 con 0x558a4fcb8fd0 2023-05-22T15:40:08.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.522+0000 7f592b7fe640 1 -- 172.21.15.16:0/667049256 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f593c012ef0 con 0x558a4fcb8fd0 2023-05-22T15:40:08.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.522+0000 7f592b7fe640 1 -- 172.21.15.16:0/667049256 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f593c05a2b0 con 0x558a4fcb8fd0 2023-05-22T15:40:08.531 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.522+0000 7f594a5ab000 1 -- 172.21.15.16:0/667049256 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558a4fcb8fd0 msgr2=0x558a4fedfde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:40:08.532 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.522+0000 7f594a5ab000 1 --2- 172.21.15.16:0/667049256 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558a4fcb8fd0 0x558a4fedfde0 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7f593c00d5c0 tx=0x7f593c01f000 comp rx=0 tx=0).stop 2023-05-22T15:40:08.532 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.522+0000 7f594a5ab000 1 -- 172.21.15.16:0/667049256 shutdown_connections 2023-05-22T15:40:08.532 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.522+0000 7f594a5ab000 1 --2- 172.21.15.16:0/667049256 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558a4ff2c170 0x558a4fee0860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:08.532 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.522+0000 7f594a5ab000 1 --2- 172.21.15.16:0/667049256 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x558a4fedfa10 0x558a4fee0320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:08.532 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.522+0000 7f594a5ab000 1 --2- 172.21.15.16:0/667049256 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558a4fcb8fd0 0x558a4fedfde0 unknown :-1 s=CLOSED pgs=935 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:08.532 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.522+0000 7f594a5ab000 1 -- 172.21.15.16:0/667049256 >> 172.21.15.16:0/667049256 conn(0x558a4fd88270 msgr2=0x558a4ff0ef10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:40:08.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.522+0000 7f594a5ab000 1 -- 172.21.15.16:0/667049256 shutdown_connections 2023-05-22T15:40:08.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.522+0000 7f594a5ab000 1 -- 172.21.15.16:0/667049256 wait complete. 2023-05-22T15:40:08.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.522+0000 7f594a5ab000 1 -- start start 2023-05-22T15:40:08.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.522+0000 7f594a5ab000 1 Processor -- start 2023-05-22T15:40:08.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.522+0000 7f594a5ab000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558a4fcb8fd0 0x558a4ff87cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:40:08.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.522+0000 7f594a5ab000 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558a4fedfa10 0x558a4fee20a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:40:08.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.522+0000 7f594a5ab000 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x558a4ff2c170 0x558a4fee25e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:40:08.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.522+0000 7f594a5ab000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x558a4fd2db80 con 0x558a4fcb8fd0 2023-05-22T15:40:08.534 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.522+0000 7f594a5ab000 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x558a4fd388d0 con 0x558a4fedfa10 2023-05-22T15:40:08.534 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.522+0000 7f594a5ab000 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x558a4fcff8e0 con 0x558a4ff2c170 2023-05-22T15:40:08.534 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.522+0000 7f594a5ab000 10 client.0 _ll_register_callbacks cb 0x558a4fd80eb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-05-22T15:40:08.534 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.522+0000 7f5945225640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558a4fcb8fd0 0x558a4ff87cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:40:08.534 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.522+0000 7f5945225640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558a4fcb8fd0 0x558a4ff87cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34516/0 (socket says 172.21.15.16:34516) 2023-05-22T15:40:08.534 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.522+0000 7f5945225640 1 -- 172.21.15.16:0/476334617 learned_addr learned my addr 172.21.15.16:0/476334617 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:40:08.534 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.522+0000 7f5945225640 1 -- 172.21.15.16:0/476334617 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x558a4ff2c170 msgr2=0x558a4fee25e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:40:08.535 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.522+0000 7f5945a26640 1 --2- 172.21.15.16:0/476334617 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x558a4ff2c170 0x558a4fee25e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:40:08.535 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.522+0000 7f5944a24640 1 --2- 172.21.15.16:0/476334617 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558a4fedfa10 0x558a4fee20a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:40:08.535 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.522+0000 7f5945225640 1 --2- 172.21.15.16:0/476334617 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x558a4ff2c170 0x558a4fee25e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:08.535 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.522+0000 7f5945225640 1 -- 172.21.15.16:0/476334617 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558a4fedfa10 msgr2=0x558a4fee20a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:40:08.535 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.522+0000 7f5945225640 1 --2- 172.21.15.16:0/476334617 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558a4fedfa10 0x558a4fee20a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:08.535 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.522+0000 7f5945225640 1 -- 172.21.15.16:0/476334617 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x558a4fcf60b0 con 0x558a4fcb8fd0 2023-05-22T15:40:08.535 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.522+0000 7f5945225640 1 --2- 172.21.15.16:0/476334617 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558a4fcb8fd0 0x558a4ff87cd0 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7f593c081170 tx=0x7f593c05a040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f03d9d74f989d09 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:40:08.535 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.522+0000 7f591ffff640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2023-05-22T15:40:08.536 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.522+0000 7f591ffff640 1 -- 172.21.15.16:0/476334617 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f593c012ef0 con 0x558a4fcb8fd0 2023-05-22T15:40:08.536 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.522+0000 7f591ffff640 1 -- 172.21.15.16:0/476334617 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(73..73 src has 1..73) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f593c05a0b0 con 0x558a4fcb8fd0 2023-05-22T15:40:08.536 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.522+0000 7f594a5ab000 1 -- 172.21.15.16:0/476334617 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x558a4fcd74d0 con 0x558a4fcb8fd0 2023-05-22T15:40:08.536 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.522+0000 7f591ffff640 1 -- 172.21.15.16:0/476334617 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f593c053ff0 con 0x558a4fcb8fd0 2023-05-22T15:40:08.536 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.522+0000 7f594a5ab000 10 client.6627 fetch_fsmap learned FSMap version 22 2023-05-22T15:40:08.536 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.522+0000 7f594a5ab000 1 -- 172.21.15.16:0/476334617 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=2+}) v3 -- 0x558a4fcdbbe0 con 0x558a4fcb8fd0 2023-05-22T15:40:08.536 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.526+0000 7f591ffff640 1 -- 172.21.15.16:0/476334617 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x558a4fcd74d0 con 0x558a4fcb8fd0 2023-05-22T15:40:08.536 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.526+0000 7f591ffff640 1 -- 172.21.15.16:0/476334617 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 22) v1 ==== 3106+0+0 (secure 0 0 0) 0x7f593c0581b0 con 0x558a4fcb8fd0 2023-05-22T15:40:08.537 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.526+0000 7f594a5ab000 10 client.6627 fetch_fsmap finished waiting for FSMap version 22 2023-05-22T15:40:08.537 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.526+0000 7f594a5ab000 10 client.6627 resolve_mds: resolved name 'a' to daemon mds.a 2023-05-22T15:40:08.537 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.526+0000 7f594a5ab000 20 client.6627 populate_metadata read hostname 'smithi016' 2023-05-22T15:40:08.537 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.526+0000 7f594a5ab000 1 --2- 172.21.15.16:0/476334617 >> [v2:172.21.15.16:6836/4263396864,v1:172.21.15.16:6837/4263396864] conn(0x558a4fee2b20 0x558a4fee2ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:40:08.537 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.526+0000 7f594a5ab000 4 client.6627 mds_command: new command op to 6468 tid=0[{"prefix": "status"}] 2023-05-22T15:40:08.537 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.526+0000 7f594a5ab000 1 -- 172.21.15.16:0/476334617 --> [v2:172.21.15.16:6836/4263396864,v1:172.21.15.16:6837/4263396864] -- command(tid 0: {"prefix": "status"}) v1 -- 0x558a4fd29800 con 0x558a4fee2b20 2023-05-22T15:40:08.537 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.526+0000 7f5944a24640 1 --2- 172.21.15.16:0/476334617 >> [v2:172.21.15.16:6836/4263396864,v1:172.21.15.16:6837/4263396864] conn(0x558a4fee2b20 0x558a4fee2ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:40:08.538 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.526+0000 7f5944a24640 1 --2- 172.21.15.16:0/476334617 >> [v2:172.21.15.16:6836/4263396864,v1:172.21.15.16:6837/4263396864] conn(0x558a4fee2b20 0x558a4fee2ef0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=eecef5deefa7f68e server_cookie=a32ea09087a70fd8 in_seq=0 out_seq=0 2023-05-22T15:40:08.538 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.526+0000 7f591ffff640 10 client.6627 ms_handle_connect on v2:172.21.15.16:6836/4263396864 2023-05-22T15:40:08.538 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.526+0000 7f591ffff640 1 -- 172.21.15.16:0/476334617 <== mds.0 v2:172.21.15.16:6836/4263396864 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x558a4fd29800 con 0x558a4fee2b20 2023-05-22T15:40:08.538 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.526+0000 7f591ffff640 10 client.6627 handle_command_reply: tid=0 2023-05-22T15:40:08.538 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.526+0000 7f594a5ab000 1 client.6627 shutdown 2023-05-22T15:40:08.538 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.526+0000 7f594a5ab000 1 -- 172.21.15.16:0/476334617 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558a4fcb8fd0 msgr2=0x558a4ff87cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:40:08.538 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.526+0000 7f594a5ab000 1 --2- 172.21.15.16:0/476334617 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558a4fcb8fd0 0x558a4ff87cd0 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7f593c081170 tx=0x7f593c05a040 comp rx=0 tx=0).stop 2023-05-22T15:40:08.538 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.526+0000 7f594a5ab000 1 -- 172.21.15.16:0/476334617 shutdown_connections 2023-05-22T15:40:08.539 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.526+0000 7f594a5ab000 1 --2- 172.21.15.16:0/476334617 >> [v2:172.21.15.16:6836/4263396864,v1:172.21.15.16:6837/4263396864] conn(0x558a4fee2b20 0x558a4fee2ef0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:08.539 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.526+0000 7f594a5ab000 1 --2- 172.21.15.16:0/476334617 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x558a4ff2c170 0x558a4fee25e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:08.539 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.526+0000 7f594a5ab000 1 --2- 172.21.15.16:0/476334617 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558a4fedfa10 0x558a4fee20a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:08.539 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.526+0000 7f594a5ab000 1 --2- 172.21.15.16:0/476334617 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558a4fcb8fd0 0x558a4ff87cd0 unknown :-1 s=CLOSED pgs=936 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:08.539 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.526+0000 7f591ffff640 0 client.6627 ms_handle_reset on v2:172.21.15.16:6836/4263396864 2023-05-22T15:40:08.539 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.526+0000 7f594a5ab000 1 -- 172.21.15.16:0/476334617 >> 172.21.15.16:0/476334617 conn(0x558a4fd88270 msgr2=0x558a4ff111d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:40:08.539 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.526+0000 7f594a5ab000 1 -- 172.21.15.16:0/476334617 shutdown_connections 2023-05-22T15:40:08.539 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.526+0000 7f594a5ab000 1 -- 172.21.15.16:0/476334617 wait complete. 2023-05-22T15:40:08.540 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.526+0000 7f594a5ab000 20 client.6627 trim_cache size 0 max 16384 2023-05-22T15:40:08.540 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-05-22T15:40:08.540 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_fsid": "b2719834-7847-4d98-90a6-f65d57f85c49", 2023-05-22T15:40:08.540 INFO:teuthology.orchestra.run.smithi016.stdout: "whoami": 0, 2023-05-22T15:40:08.540 INFO:teuthology.orchestra.run.smithi016.stdout: "id": 6468, 2023-05-22T15:40:08.541 INFO:teuthology.orchestra.run.smithi016.stdout: "want_state": "up:active", 2023-05-22T15:40:08.541 INFO:teuthology.orchestra.run.smithi016.stdout: "state": "up:active", 2023-05-22T15:40:08.541 INFO:teuthology.orchestra.run.smithi016.stdout: "fs_name": "cephfs", 2023-05-22T15:40:08.541 INFO:teuthology.orchestra.run.smithi016.stdout: "rank_uptime": 5.7719878009999999, 2023-05-22T15:40:08.541 INFO:teuthology.orchestra.run.smithi016.stdout: "mdsmap_epoch": 22, 2023-05-22T15:40:08.541 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch": 73, 2023-05-22T15:40:08.541 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch_barrier": 73, 2023-05-22T15:40:08.542 INFO:teuthology.orchestra.run.smithi016.stdout: "uptime": 10.941380667000001 2023-05-22T15:40:08.542 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-05-22T15:40:08.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.526+0000 7f5947489640 1 -- 172.21.15.16:0/111924021 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f59240654a0 msgr2=0x7f5924067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:40:08.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.526+0000 7f5947489640 1 --2- 172.21.15.16:0/111924021 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f59240654a0 0x7f5924067950 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f59300af000 tx=0x7f59300a7000 comp rx=0 tx=0).stop 2023-05-22T15:40:08.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.526+0000 7f5947489640 1 -- 172.21.15.16:0/111924021 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5940070df0 msgr2=0x7f5940153c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:40:08.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.526+0000 7f5947489640 1 --2- 172.21.15.16:0/111924021 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5940070df0 0x7f5940153c40 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7f593c00d5f0 tx=0x7f593c00dab0 comp rx=0 tx=0).stop 2023-05-22T15:40:08.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.526+0000 7f5947489640 1 -- 172.21.15.16:0/111924021 shutdown_connections 2023-05-22T15:40:08.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.526+0000 7f5947489640 1 --2- 172.21.15.16:0/111924021 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f59400ca4e0 0x7f5940158450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:08.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.526+0000 7f5947489640 1 --2- 172.21.15.16:0/111924021 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f59240654a0 0x7f5924067950 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:08.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.526+0000 7f5947489640 1 --2- 172.21.15.16:0/111924021 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f59400c9b40 0x7f5940152470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:08.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.526+0000 7f5947489640 1 --2- 172.21.15.16:0/111924021 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5940070df0 0x7f5940153c40 unknown :-1 s=CLOSED pgs=932 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:08.544 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.526+0000 7f5947489640 1 -- 172.21.15.16:0/111924021 >> 172.21.15.16:0/111924021 conn(0x7f5940069720 msgr2=0x7f59400cbf90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:40:08.544 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.526+0000 7f5947489640 1 -- 172.21.15.16:0/111924021 shutdown_connections 2023-05-22T15:40:08.544 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.526+0000 7f5947489640 1 -- 172.21.15.16:0/111924021 wait complete. 2023-05-22T15:40:08.548 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-05-22T15:40:08.570 INFO:tasks.ceph.mds.a.smithi016.stderr:2023-05-22T15:40:08.566+0000 7f96ea98b640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-05-22T15:40:08.661 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.654+0000 7ff75731c640 1 Processor -- start 2023-05-22T15:40:08.661 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.654+0000 7ff75731c640 1 -- start start 2023-05-22T15:40:08.663 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.658+0000 7ff75731c640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff750158290 0x7ff7500c8880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:40:08.663 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.658+0000 7ff75731c640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7ff750158660 0x7ff7500cdfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:40:08.663 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.658+0000 7ff75731c640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff7500cce20 0x7ff7500ce4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:40:08.664 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.658+0000 7ff75731c640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7500c92b0 con 0x7ff750158290 2023-05-22T15:40:08.664 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.658+0000 7ff75731c640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7500cc060 con 0x7ff750158660 2023-05-22T15:40:08.664 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.658+0000 7ff75731c640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7500cc340 con 0x7ff7500cce20 2023-05-22T15:40:08.664 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.658+0000 7ff7550b8640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff750158290 0x7ff7500c8880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:40:08.664 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.658+0000 7ff7550b8640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff750158290 0x7ff7500c8880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34518/0 (socket says 172.21.15.16:34518) 2023-05-22T15:40:08.665 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.658+0000 7ff7550b8640 1 -- 172.21.15.16:0/4118200234 learned_addr learned my addr 172.21.15.16:0/4118200234 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:40:08.665 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.658+0000 7ff7548b7640 1 --2- 172.21.15.16:0/4118200234 >> v2:172.21.15.62:3300/0 conn(0x7ff750158660 0x7ff7500cdfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:40:08.665 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.658+0000 7ff7558b9640 1 --2- 172.21.15.16:0/4118200234 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff7500cce20 0x7ff7500ce4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:40:08.665 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.658+0000 7ff7550b8640 1 -- 172.21.15.16:0/4118200234 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff7500cce20 msgr2=0x7ff7500ce4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:40:08.665 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.658+0000 7ff7550b8640 1 --2- 172.21.15.16:0/4118200234 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff7500cce20 0x7ff7500ce4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:08.666 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.658+0000 7ff7550b8640 1 -- 172.21.15.16:0/4118200234 >> v2:172.21.15.62:3300/0 conn(0x7ff750158660 msgr2=0x7ff7500cdfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:40:08.666 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.658+0000 7ff7550b8640 1 --2- 172.21.15.16:0/4118200234 >> v2:172.21.15.62:3300/0 conn(0x7ff750158660 0x7ff7500cdfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:08.666 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.658+0000 7ff7550b8640 1 -- 172.21.15.16:0/4118200234 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7500cc620 con 0x7ff750158290 2023-05-22T15:40:08.666 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.658+0000 7ff7558b9640 1 --2- 172.21.15.16:0/4118200234 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff7500cce20 0x7ff7500ce4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:40:08.668 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.658+0000 7ff7550b8640 1 --2- 172.21.15.16:0/4118200234 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff750158290 0x7ff7500c8880 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7ff75006d810 tx=0x7ff74c01e9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b78111dd17760a4d server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:40:08.668 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.658+0000 7ff747fff640 1 -- 172.21.15.16:0/4118200234 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff74c0a79e0 con 0x7ff750158290 2023-05-22T15:40:08.668 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.658+0000 7ff747fff640 1 -- 172.21.15.16:0/4118200234 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff74c0a7b40 con 0x7ff750158290 2023-05-22T15:40:08.668 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.658+0000 7ff747fff640 1 -- 172.21.15.16:0/4118200234 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff74c0af440 con 0x7ff750158290 2023-05-22T15:40:08.668 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.662+0000 7ff75731c640 1 -- 172.21.15.16:0/4118200234 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff750158290 msgr2=0x7ff7500c8880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:40:08.668 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.662+0000 7ff75731c640 1 --2- 172.21.15.16:0/4118200234 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff750158290 0x7ff7500c8880 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7ff75006d810 tx=0x7ff74c01e9b0 comp rx=0 tx=0).stop 2023-05-22T15:40:08.669 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.662+0000 7ff75731c640 1 -- 172.21.15.16:0/4118200234 shutdown_connections 2023-05-22T15:40:08.669 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.662+0000 7ff75731c640 1 --2- 172.21.15.16:0/4118200234 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff7500cce20 0x7ff7500ce4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:08.669 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.662+0000 7ff75731c640 1 --2- 172.21.15.16:0/4118200234 >> v2:172.21.15.62:3300/0 conn(0x7ff750158660 0x7ff7500cdfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:08.669 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.662+0000 7ff75731c640 1 --2- 172.21.15.16:0/4118200234 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff750158290 0x7ff7500c8880 unknown :-1 s=CLOSED pgs=937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:08.669 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.662+0000 7ff75731c640 1 -- 172.21.15.16:0/4118200234 >> 172.21.15.16:0/4118200234 conn(0x7ff750069720 msgr2=0x7ff750069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:40:08.669 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.662+0000 7ff75731c640 1 -- 172.21.15.16:0/4118200234 shutdown_connections 2023-05-22T15:40:08.669 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.662+0000 7ff75731c640 1 -- 172.21.15.16:0/4118200234 wait complete. 2023-05-22T15:40:08.669 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.662+0000 7ff75731c640 1 Processor -- start 2023-05-22T15:40:08.670 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.662+0000 7ff75731c640 1 -- start start 2023-05-22T15:40:08.671 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.662+0000 7ff75731c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff7500cce20 0x7ff750145b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:40:08.671 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.662+0000 7ff75731c640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff750158290 0x7ff750156af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:40:08.671 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.662+0000 7ff75731c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff750158660 0x7ff750150f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:40:08.672 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.662+0000 7ff75731c640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff750146090 con 0x7ff750158660 2023-05-22T15:40:08.672 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.662+0000 7ff75731c640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff750157aa0 con 0x7ff7500cce20 2023-05-22T15:40:08.672 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.662+0000 7ff75731c640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7501574e0 con 0x7ff750158290 2023-05-22T15:40:08.672 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.662+0000 7ff7558b9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff750158660 0x7ff750150f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:40:08.672 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.662+0000 7ff7558b9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff750158660 0x7ff750150f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34532/0 (socket says 172.21.15.16:34532) 2023-05-22T15:40:08.672 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.662+0000 7ff7558b9640 1 -- 172.21.15.16:0/1353749323 learned_addr learned my addr 172.21.15.16:0/1353749323 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:40:08.673 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.662+0000 7ff7548b7640 1 --2- 172.21.15.16:0/1353749323 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff750158290 0x7ff750156af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:40:08.673 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.662+0000 7ff7550b8640 1 --2- 172.21.15.16:0/1353749323 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff7500cce20 0x7ff750145b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:40:08.673 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.662+0000 7ff7558b9640 1 -- 172.21.15.16:0/1353749323 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff750158290 msgr2=0x7ff750156af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:40:08.673 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.662+0000 7ff7558b9640 1 --2- 172.21.15.16:0/1353749323 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff750158290 0x7ff750156af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:08.673 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.662+0000 7ff7558b9640 1 -- 172.21.15.16:0/1353749323 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff7500cce20 msgr2=0x7ff750145b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:40:08.673 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.662+0000 7ff7558b9640 1 --2- 172.21.15.16:0/1353749323 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff7500cce20 0x7ff750145b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:08.673 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.662+0000 7ff7558b9640 1 -- 172.21.15.16:0/1353749323 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff74c00a670 con 0x7ff750158660 2023-05-22T15:40:08.673 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.662+0000 7ff7558b9640 1 --2- 172.21.15.16:0/1353749323 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff750158660 0x7ff750150f00 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7ff74800cea0 tx=0x7ff74800cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:40:08.674 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.662+0000 7ff745ffb640 1 -- 172.21.15.16:0/1353749323 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff748003ba0 con 0x7ff750158660 2023-05-22T15:40:08.674 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.662+0000 7ff745ffb640 1 -- 172.21.15.16:0/1353749323 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff748003d20 con 0x7ff750158660 2023-05-22T15:40:08.674 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.662+0000 7ff745ffb640 1 -- 172.21.15.16:0/1353749323 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff74800d880 con 0x7ff750158660 2023-05-22T15:40:08.674 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.662+0000 7ff75731c640 1 -- 172.21.15.16:0/1353749323 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7500cabc0 con 0x7ff750158660 2023-05-22T15:40:08.674 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.662+0000 7ff75731c640 1 -- 172.21.15.16:0/1353749323 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7500caa30 con 0x7ff750158660 2023-05-22T15:40:08.674 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.666+0000 7ff75731c640 1 -- 172.21.15.16:0/1353749323 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7500cad50 con 0x7ff750158660 2023-05-22T15:40:08.674 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.666+0000 7ff745ffb640 1 -- 172.21.15.16:0/1353749323 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) v1 ==== 84332+0+0 (secure 0 0 0) 0x7ff74800d9e0 con 0x7ff750158660 2023-05-22T15:40:08.675 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.666+0000 7ff745ffb640 1 --2- 172.21.15.16:0/1353749323 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ff7340654a0 0x7ff734067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:40:08.675 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.666+0000 7ff745ffb640 1 -- 172.21.15.16:0/1353749323 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 6382+0+0 (secure 0 0 0) 0x7ff748083440 con 0x7ff750158660 2023-05-22T15:40:08.675 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.666+0000 7ff7550b8640 1 --2- 172.21.15.16:0/1353749323 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ff7340654a0 0x7ff734067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:40:08.676 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.666+0000 7ff7550b8640 1 --2- 172.21.15.16:0/1353749323 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ff7340654a0 0x7ff734067950 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7ff74c0b1000 tx=0x7ff74c0a9070 comp rx=0 tx=0).ready entity=mgr.6240 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:40:08.676 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.670+0000 7ff745ffb640 1 -- 172.21.15.16:0/1353749323 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7ff74804c620 con 0x7ff750158660 2023-05-22T15:40:08.868 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.862+0000 7ff75731c640 1 -- 172.21.15.16:0/1353749323 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7ff7500721a0 con 0x7ff750158660 2023-05-22T15:40:08.869 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.862+0000 7ff745ffb640 1 -- 172.21.15.16:0/1353749323 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v22) v1 ==== 53+0+73 (secure 0 0 0) 0x7ff7500721a0 con 0x7ff750158660 2023-05-22T15:40:08.870 INFO:teuthology.orchestra.run.smithi016.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-05-22T15:40:08.870 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.866+0000 7ff75731c640 1 -- 172.21.15.16:0/1353749323 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ff7340654a0 msgr2=0x7ff734067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:40:08.871 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.866+0000 7ff75731c640 1 --2- 172.21.15.16:0/1353749323 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ff7340654a0 0x7ff734067950 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7ff74c0b1000 tx=0x7ff74c0a9070 comp rx=0 tx=0).stop 2023-05-22T15:40:08.871 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.866+0000 7ff75731c640 1 -- 172.21.15.16:0/1353749323 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff750158660 msgr2=0x7ff750150f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:40:08.871 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.866+0000 7ff75731c640 1 --2- 172.21.15.16:0/1353749323 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff750158660 0x7ff750150f00 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7ff74800cea0 tx=0x7ff74800cf90 comp rx=0 tx=0).stop 2023-05-22T15:40:08.872 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.866+0000 7ff75731c640 1 -- 172.21.15.16:0/1353749323 shutdown_connections 2023-05-22T15:40:08.872 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.866+0000 7ff75731c640 1 --2- 172.21.15.16:0/1353749323 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff750158660 0x7ff750150f00 unknown :-1 s=CLOSED pgs=938 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:08.872 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.866+0000 7ff75731c640 1 --2- 172.21.15.16:0/1353749323 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ff7340654a0 0x7ff734067950 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:08.872 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.866+0000 7ff75731c640 1 --2- 172.21.15.16:0/1353749323 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff750158290 0x7ff750156af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:08.872 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.866+0000 7ff75731c640 1 --2- 172.21.15.16:0/1353749323 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff7500cce20 0x7ff750145b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:08.872 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.866+0000 7ff75731c640 1 -- 172.21.15.16:0/1353749323 >> 172.21.15.16:0/1353749323 conn(0x7ff750069720 msgr2=0x7ff75006f8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:40:08.872 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.866+0000 7ff75731c640 1 -- 172.21.15.16:0/1353749323 shutdown_connections 2023-05-22T15:40:08.873 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:08.866+0000 7ff75731c640 1 -- 172.21.15.16:0/1353749323 wait complete. 2023-05-22T15:40:08.887 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-05-22T15:40:08.887 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-05-22T15:40:08.887 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi016.front.sepia.ceph.com 2023-05-22T15:40:08.888 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-05-22T15:40:08.888 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-05-22T15:40:08.888 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-05-22T15:40:08.889 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-05-22T15:40:08.889 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-05-22T15:40:08.889 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-05-22T15:40:08.890 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:40:08.890 DEBUG:teuthology.orchestra.run.smithi016:> ip addr 2023-05-22T15:40:08.936 INFO:teuthology.orchestra.run.smithi016.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-05-22T15:40:08.936 INFO:teuthology.orchestra.run.smithi016.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-05-22T15:40:08.936 INFO:teuthology.orchestra.run.smithi016.stdout: inet 127.0.0.1/8 scope host lo 2023-05-22T15:40:08.937 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-05-22T15:40:08.937 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 ::1/128 scope host 2023-05-22T15:40:08.937 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-05-22T15:40:08.937 INFO:teuthology.orchestra.run.smithi016.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-05-22T15:40:08.937 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:6b:fd:e0 brd ff:ff:ff:ff:ff:ff 2023-05-22T15:40:08.937 INFO:teuthology.orchestra.run.smithi016.stdout: altname enp7s0f0 2023-05-22T15:40:08.938 INFO:teuthology.orchestra.run.smithi016.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-05-22T15:40:08.938 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:6b:fd:e1 brd ff:ff:ff:ff:ff:ff 2023-05-22T15:40:08.938 INFO:teuthology.orchestra.run.smithi016.stdout: altname enp7s0f1 2023-05-22T15:40:08.938 INFO:teuthology.orchestra.run.smithi016.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2023-05-22T15:40:08.938 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:bd:15:64 brd ff:ff:ff:ff:ff:ff 2023-05-22T15:40:08.938 INFO:teuthology.orchestra.run.smithi016.stdout: altname enp1s0f0 2023-05-22T15:40:08.939 INFO:teuthology.orchestra.run.smithi016.stdout: inet 172.21.15.16/20 brd 172.21.15.255 scope global dynamic ens1f0 2023-05-22T15:40:08.939 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft 41707sec preferred_lft 41707sec 2023-05-22T15:40:08.939 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::ec4:7aff:febd:1564/64 scope link 2023-05-22T15:40:08.939 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-05-22T15:40:08.940 INFO:teuthology.orchestra.run.smithi016.stdout:5: ens1f1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-05-22T15:40:08.940 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:bd:15:65 brd ff:ff:ff:ff:ff:ff 2023-05-22T15:40:08.940 INFO:teuthology.orchestra.run.smithi016.stdout: altname enp1s0f1 2023-05-22T15:40:08.940 INFO:teuthology.orchestra.run.smithi016.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2023-05-22T15:40:08.940 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 02:42:1f:b8:98:0a brd ff:ff:ff:ff:ff:ff 2023-05-22T15:40:08.940 INFO:teuthology.orchestra.run.smithi016.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2023-05-22T15:40:08.940 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-05-22T15:40:08.941 INFO:teuthology.orchestra.run.smithi016.stdout:9: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-05-22T15:40:08.941 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 5e:1f:84:6d:79:ae brd ff:ff:ff:ff:ff:ff 2023-05-22T15:40:08.941 INFO:teuthology.orchestra.run.smithi016.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-05-22T15:40:08.941 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-05-22T15:40:08.941 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::5c1f:84ff:fe6d:79ae/64 scope link 2023-05-22T15:40:08.941 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-05-22T15:40:08.941 INFO:teuthology.orchestra.run.smithi016.stdout:10: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-05-22T15:40:08.942 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0e:5c:f6:1f:82:2f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-05-22T15:40:08.942 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::c5c:f6ff:fe1f:822f/64 scope link 2023-05-22T15:40:08.942 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-05-22T15:40:08.942 INFO:teuthology.orchestra.run.smithi016.stdout:11: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-05-22T15:40:08.942 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether c6:a6:3d:69:d0:db brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-05-22T15:40:08.942 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::c4a6:3dff:fe69:d0db/64 scope link 2023-05-22T15:40:08.942 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-05-22T15:40:08.943 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:40:08.943 DEBUG:teuthology.orchestra.run.smithi016:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-05-22T15:40:08.990 INFO:teuthology.orchestra.run.smithi016.stdout:1 2023-05-22T15:40:08.991 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:40:08.991 DEBUG:teuthology.orchestra.run.smithi016:> route 2023-05-22T15:40:09.040 INFO:teuthology.orchestra.run.smithi016.stdout:Kernel IP routing table 2023-05-22T15:40:09.040 INFO:teuthology.orchestra.run.smithi016.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-05-22T15:40:09.040 INFO:teuthology.orchestra.run.smithi016.stdout:default _gateway 0.0.0.0 UG 0 0 0 ens1f0 2023-05-22T15:40:09.041 INFO:teuthology.orchestra.run.smithi016.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2023-05-22T15:40:09.041 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 0 0 0 ens1f0 2023-05-22T15:40:09.041 INFO:teuthology.orchestra.run.smithi016.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-05-22T15:40:09.041 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:40:09.042 DEBUG:teuthology.orchestra.run.smithi016:> (cd / && exec bash -c ' 2023-05-22T15:40:09.042 DEBUG:teuthology.orchestra.run.smithi016:> set -e 2023-05-22T15:40:09.042 DEBUG:teuthology.orchestra.run.smithi016:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2023-05-22T15:40:09.043 DEBUG:teuthology.orchestra.run.smithi016:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2023-05-22T15:40:09.043 DEBUG:teuthology.orchestra.run.smithi016:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2023-05-22T15:40:09.043 DEBUG:teuthology.orchestra.run.smithi016:> ') 2023-05-22T15:40:09.118 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:40:09.118 DEBUG:teuthology.orchestra.run.smithi016:> ip netns list 2023-05-22T15:40:09.167 INFO:teuthology.orchestra.run.smithi016.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-05-22T15:40:09.167 INFO:teuthology.orchestra.run.smithi016.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-05-22T15:40:09.168 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-05-22T15:40:09.168 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:40:09.168 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip link set brx.0 up 2023-05-22T15:40:09.223 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2023-05-22T15:40:09.223 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-05-22T15:40:09.223 DEBUG:teuthology.orchestra.run.smithi016:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-05-22T15:40:09.271 INFO:teuthology.orchestra.run.smithi016.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-05-22T15:40:09.271 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-05-22T15:40:09.271 DEBUG:teuthology.orchestra.run.smithi016:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-05-22T15:40:09.318 DEBUG:teuthology.orchestra.run.smithi016:> sudo modprobe fuse 2023-05-22T15:40:09.376 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-05-22T15:40:09.377 DEBUG:teuthology.orchestra.run.smithi016:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-05-22T15:40:09.434 INFO:teuthology.orchestra.run.smithi016.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-05-22T15:40:09.435 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-05-22T15:40:09.436 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:40:09.436 DEBUG:teuthology.orchestra.run.smithi016:> ls /sys/fs/fuse/connections 2023-05-22T15:40:09.482 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2023-05-22T15:40:09.483 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2023-05-22T15:40:09.527 DEBUG:teuthology.orchestra.run.smithi016:> sudo modprobe fuse 2023-05-22T15:40:09.542 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-05-22T15:40:09.543 DEBUG:teuthology.orchestra.run.smithi016:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-05-22T15:40:09.599 INFO:teuthology.orchestra.run.smithi016.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-05-22T15:40:09.600 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-05-22T15:40:09.601 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:40:09.601 DEBUG:teuthology.orchestra.run.smithi016:> ls /sys/fs/fuse/connections 2023-05-22T15:40:09.604 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi016.stderr:2023-05-22T15:40:09.598+0000 7f120ddeb580 -1 init, newargv = 0x5632a337ff90 newargc=15 2023-05-22T15:40:09.605 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi016.stderr:ceph-fuse[52930]: starting ceph client 2023-05-22T15:40:09.611 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi016.stderr:ceph-fuse[52930]: starting fuse 2023-05-22T15:40:10.710 DEBUG:teuthology.orchestra.run.smithi016:> sudo modprobe fuse 2023-05-22T15:40:10.727 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-05-22T15:40:10.727 DEBUG:teuthology.orchestra.run.smithi016:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-05-22T15:40:10.783 INFO:teuthology.orchestra.run.smithi016.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-05-22T15:40:10.784 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-05-22T15:40:10.785 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:40:10.785 DEBUG:teuthology.orchestra.run.smithi016:> ls /sys/fs/fuse/connections 2023-05-22T15:40:10.831 INFO:teuthology.orchestra.run.smithi016.stdout:45 2023-05-22T15:40:10.831 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [45] 2023-05-22T15:40:10.832 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2023-05-22T15:40:10.832 DEBUG:teuthology.orchestra.run.smithi016:> import glob 2023-05-22T15:40:10.832 DEBUG:teuthology.orchestra.run.smithi016:> import re 2023-05-22T15:40:10.833 DEBUG:teuthology.orchestra.run.smithi016:> import os 2023-05-22T15:40:10.833 DEBUG:teuthology.orchestra.run.smithi016:> import subprocess 2023-05-22T15:40:10.833 DEBUG:teuthology.orchestra.run.smithi016:> 2023-05-22T15:40:10.834 DEBUG:teuthology.orchestra.run.smithi016:> def _find_admin_socket(client_name): 2023-05-22T15:40:10.834 DEBUG:teuthology.orchestra.run.smithi016:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2023-05-22T15:40:10.834 DEBUG:teuthology.orchestra.run.smithi016:> files = glob.glob(asok_path) 2023-05-22T15:40:10.834 DEBUG:teuthology.orchestra.run.smithi016:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2023-05-22T15:40:10.835 DEBUG:teuthology.orchestra.run.smithi016:> 2023-05-22T15:40:10.835 DEBUG:teuthology.orchestra.run.smithi016:> # Given a non-glob path, it better be there 2023-05-22T15:40:10.835 DEBUG:teuthology.orchestra.run.smithi016:> if "*" not in asok_path: 2023-05-22T15:40:10.836 DEBUG:teuthology.orchestra.run.smithi016:> assert(len(files) == 1) 2023-05-22T15:40:10.836 DEBUG:teuthology.orchestra.run.smithi016:> return files[0] 2023-05-22T15:40:10.836 DEBUG:teuthology.orchestra.run.smithi016:> 2023-05-22T15:40:10.837 DEBUG:teuthology.orchestra.run.smithi016:> for f in files: 2023-05-22T15:40:10.837 DEBUG:teuthology.orchestra.run.smithi016:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-05-22T15:40:10.837 DEBUG:teuthology.orchestra.run.smithi016:> if os.path.exists("/proc/{0}".format(pid)): 2023-05-22T15:40:10.837 DEBUG:teuthology.orchestra.run.smithi016:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-05-22T15:40:10.838 DEBUG:teuthology.orchestra.run.smithi016:> contents = proc_f.read() 2023-05-22T15:40:10.838 DEBUG:teuthology.orchestra.run.smithi016:> if mountpoint in contents: 2023-05-22T15:40:10.838 DEBUG:teuthology.orchestra.run.smithi016:> return f 2023-05-22T15:40:10.839 DEBUG:teuthology.orchestra.run.smithi016:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-05-22T15:40:10.839 DEBUG:teuthology.orchestra.run.smithi016:> 2023-05-22T15:40:10.839 DEBUG:teuthology.orchestra.run.smithi016:> print(_find_admin_socket("client.0")) 2023-05-22T15:40:10.839 DEBUG:teuthology.orchestra.run.smithi016:> ' 2023-05-22T15:40:10.955 INFO:teuthology.orchestra.run.smithi016.stdout:/var/run/ceph/ceph-client.0.52930.asok 2023-05-22T15:40:11.137 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.52930.asok 2023-05-22T15:40:11.137 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:40:11.137 DEBUG:teuthology.orchestra.run.smithi016:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.52930.asok status 2023-05-22T15:40:11.297 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-05-22T15:40:11.297 INFO:teuthology.orchestra.run.smithi016.stdout: "metadata": { 2023-05-22T15:40:11.298 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph_sha1": "9c9ba4df22024d9d8d6cf9f98a4b96751fc78d0c", 2023-05-22T15:40:11.298 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph_version": "ceph version 18.0.0-3192-g9c9ba4df (9c9ba4df22024d9d8d6cf9f98a4b96751fc78d0c) reef (dev)", 2023-05-22T15:40:11.298 INFO:teuthology.orchestra.run.smithi016.stdout: "entity_id": "0", 2023-05-22T15:40:11.298 INFO:teuthology.orchestra.run.smithi016.stdout: "hostname": "smithi016", 2023-05-22T15:40:11.298 INFO:teuthology.orchestra.run.smithi016.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2023-05-22T15:40:11.299 INFO:teuthology.orchestra.run.smithi016.stdout: "pid": "52930", 2023-05-22T15:40:11.299 INFO:teuthology.orchestra.run.smithi016.stdout: "root": "/" 2023-05-22T15:40:11.299 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:40:11.299 INFO:teuthology.orchestra.run.smithi016.stdout: "dentry_count": 0, 2023-05-22T15:40:11.299 INFO:teuthology.orchestra.run.smithi016.stdout: "dentry_pinned_count": 0, 2023-05-22T15:40:11.299 INFO:teuthology.orchestra.run.smithi016.stdout: "id": 6639, 2023-05-22T15:40:11.299 INFO:teuthology.orchestra.run.smithi016.stdout: "inst": { 2023-05-22T15:40:11.299 INFO:teuthology.orchestra.run.smithi016.stdout: "name": { 2023-05-22T15:40:11.300 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "client", 2023-05-22T15:40:11.300 INFO:teuthology.orchestra.run.smithi016.stdout: "num": 6639 2023-05-22T15:40:11.300 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:40:11.300 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": { 2023-05-22T15:40:11.300 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-05-22T15:40:11.300 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "192.168.0.1:0", 2023-05-22T15:40:11.300 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 1440890816 2023-05-22T15:40:11.301 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T15:40:11.301 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:40:11.301 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": { 2023-05-22T15:40:11.301 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-05-22T15:40:11.301 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "192.168.0.1:0", 2023-05-22T15:40:11.301 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 1440890816 2023-05-22T15:40:11.301 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:40:11.302 INFO:teuthology.orchestra.run.smithi016.stdout: "inst_str": "client.6639 192.168.0.1:0/1440890816", 2023-05-22T15:40:11.302 INFO:teuthology.orchestra.run.smithi016.stdout: "addr_str": "192.168.0.1:0/1440890816", 2023-05-22T15:40:11.302 INFO:teuthology.orchestra.run.smithi016.stdout: "inode_count": 1, 2023-05-22T15:40:11.302 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_epoch": 22, 2023-05-22T15:40:11.302 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_epoch": 73, 2023-05-22T15:40:11.302 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_epoch_barrier": 0, 2023-05-22T15:40:11.302 INFO:teuthology.orchestra.run.smithi016.stdout: "blocklisted": false, 2023-05-22T15:40:11.302 INFO:teuthology.orchestra.run.smithi016.stdout: "fs_name": "cephfs" 2023-05-22T15:40:11.303 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-05-22T15:40:11.311 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:40:11.311 DEBUG:teuthology.orchestra.run.smithi016:> stat --file-system '--printf=%T 2023-05-22T15:40:11.311 DEBUG:teuthology.orchestra.run.smithi016:> ' -- /home/ubuntu/cephtest/mnt.0 2023-05-22T15:40:11.317 INFO:teuthology.orchestra.run.smithi016.stdout:fuseblk 2023-05-22T15:40:11.317 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2023-05-22T15:40:11.317 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:40:11.318 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-05-22T15:40:11.368 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2023-05-22T15:40:11.532 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:11.526+0000 7f53bd60b640 1 Processor -- start 2023-05-22T15:40:11.532 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:11.526+0000 7f53bd60b640 1 -- start start 2023-05-22T15:40:11.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:11.526+0000 7f53bd60b640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f53b80c9b40 0x7f53b815e460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:40:11.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:11.526+0000 7f53bd60b640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f53b8070df0 0x7f53b8071210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:40:11.534 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:11.526+0000 7f53bd60b640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53b80b0650 con 0x7f53b80ca4e0 2023-05-22T15:40:11.534 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:11.526+0000 7f53bd60b640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53b80b0370 con 0x7f53b8070df0 2023-05-22T15:40:11.534 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:11.526+0000 7f53bd60b640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53b815d980 con 0x7f53b80c9b40 2023-05-22T15:40:11.534 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:11.526+0000 7f53b6ffd640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f53b80ca4e0 0x7f53b80afb40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:59064/0 (socket says 172.21.15.16:59064) 2023-05-22T15:40:11.534 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:11.526+0000 7f53b6ffd640 1 -- 172.21.15.16:0/1539174525 learned_addr learned my addr 172.21.15.16:0/1539174525 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:40:11.534 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:11.526+0000 7f53b67fc640 1 --2- 172.21.15.16:0/1539174525 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f53b80c9b40 0x7f53b815e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:40:11.534 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:11.526+0000 7f53b77fe640 1 --2- 172.21.15.16:0/1539174525 >> v2:172.21.15.62:3300/0 conn(0x7f53b8070df0 0x7f53b8071210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:40:11.535 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:11.526+0000 7f53b5ffb640 1 -- 172.21.15.16:0/1539174525 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 703076806 0 0) 0x7f53b80b0650 con 0x7f53b80ca4e0 2023-05-22T15:40:11.535 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:11.526+0000 7f53b5ffb640 1 -- 172.21.15.16:0/1539174525 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53a0003530 con 0x7f53b80ca4e0 2023-05-22T15:40:11.536 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:11.530+0000 7f53b5ffb640 1 -- 172.21.15.16:0/1539174525 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4006890314 0 0) 0x7f53a0003530 con 0x7f53b80ca4e0 2023-05-22T15:40:11.536 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:11.530+0000 7f53b5ffb640 1 -- 172.21.15.16:0/1539174525 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f53b80c9b40 msgr2=0x7f53b815e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:40:11.536 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:11.530+0000 7f53b5ffb640 1 --2- 172.21.15.16:0/1539174525 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f53b80c9b40 0x7f53b815e460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:11.536 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:11.530+0000 7f53b5ffb640 1 -- 172.21.15.16:0/1539174525 >> v2:172.21.15.62:3300/0 conn(0x7f53b8070df0 msgr2=0x7f53b8071210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:40:11.537 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:11.530+0000 7f53b5ffb640 1 --2- 172.21.15.16:0/1539174525 >> v2:172.21.15.62:3300/0 conn(0x7f53b8070df0 0x7f53b8071210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:11.537 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:11.530+0000 7f53b5ffb640 1 -- 172.21.15.16:0/1539174525 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53b815dc60 con 0x7f53b80ca4e0 2023-05-22T15:40:11.537 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:11.530+0000 7f53b5ffb640 1 -- 172.21.15.16:0/1539174525 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7f53a8001680 con 0x7f53b80ca4e0 2023-05-22T15:40:11.537 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:11.530+0000 7f53b5ffb640 1 -- 172.21.15.16:0/1539174525 >> v1:172.21.15.16:6789/0 conn(0x7f53b80ca4e0 legacy=0x7f53b80afb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:40:11.537 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:11.530+0000 7f53b5ffb640 1 --2- 172.21.15.16:0/1539174525 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f53b80c9b40 0x7f53a0003980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:40:11.537 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:11.530+0000 7f53b5ffb640 1 --2- 172.21.15.16:0/1539174525 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f53a0003f00 0x7f53a00047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:40:11.537 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:11.530+0000 7f53b5ffb640 1 --2- 172.21.15.16:0/1539174525 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f53a0004d10 0x7f53a0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:40:11.537 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:11.530+0000 7f53b5ffb640 1 -- 172.21.15.16:0/1539174525 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53b80b0370 con 0x7f53a0003f00 2023-05-22T15:40:11.538 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:11.530+0000 7f53b5ffb640 1 -- 172.21.15.16:0/1539174525 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53b815d980 con 0x7f53a0004d10 2023-05-22T15:40:11.538 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:11.530+0000 7f53b5ffb640 1 -- 172.21.15.16:0/1539174525 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53a000b9c0 con 0x7f53b80c9b40 2023-05-22T15:40:11.538 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:11.530+0000 7f53b67fc640 1 --2- 172.21.15.16:0/1539174525 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f53a0003f00 0x7f53a00047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:40:11.538 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:11.530+0000 7f53b77fe640 1 --2- 172.21.15.16:0/1539174525 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f53a0004d10 0x7f53a0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:40:11.538 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:11.530+0000 7f53b6ffd640 1 --2- 172.21.15.16:0/1539174525 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f53b80c9b40 0x7f53a0003980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:40:11.538 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:11.530+0000 7f53b67fc640 1 -- 172.21.15.16:0/1539174525 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f53b80c9b40 msgr2=0x7f53a0003980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:40:11.538 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:11.530+0000 7f53b67fc640 1 --2- 172.21.15.16:0/1539174525 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f53b80c9b40 0x7f53a0003980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:11.538 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:11.530+0000 7f53b67fc640 1 -- 172.21.15.16:0/1539174525 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f53a0004d10 msgr2=0x7f53a0005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:40:11.539 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:11.530+0000 7f53b67fc640 1 --2- 172.21.15.16:0/1539174525 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f53a0004d10 0x7f53a0005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:11.539 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:11.530+0000 7f53b67fc640 1 -- 172.21.15.16:0/1539174525 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53a0003530 con 0x7f53a0003f00 2023-05-22T15:40:11.539 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:11.530+0000 7f53b6ffd640 1 --2- 172.21.15.16:0/1539174525 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f53b80c9b40 0x7f53a0003980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:40:11.541 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:11.530+0000 7f53b67fc640 1 --2- 172.21.15.16:0/1539174525 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f53a0003f00 0x7f53a00047d0 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7f53ac009ae0 tx=0x7f53ac01c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e118b2ec34b6353f server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:40:11.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:11.530+0000 7f53b5ffb640 1 -- 172.21.15.16:0/1539174525 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53ac0a6510 con 0x7f53a0003f00 2023-05-22T15:40:11.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:11.530+0000 7f53b5ffb640 1 -- 172.21.15.16:0/1539174525 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53ac0a6670 con 0x7f53a0003f00 2023-05-22T15:40:11.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:11.530+0000 7f53b5ffb640 1 -- 172.21.15.16:0/1539174525 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53ac0a6a20 con 0x7f53a0003f00 2023-05-22T15:40:11.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:11.530+0000 7f53bd60b640 1 -- 172.21.15.16:0/1539174525 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f53a0003f00 msgr2=0x7f53a00047d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:40:11.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:11.530+0000 7f53bd60b640 1 --2- 172.21.15.16:0/1539174525 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f53a0003f00 0x7f53a00047d0 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7f53ac009ae0 tx=0x7f53ac01c9c0 comp rx=0 tx=0).stop 2023-05-22T15:40:11.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:11.530+0000 7f53b77fe640 1 -- 172.21.15.16:0/1539174525 reap_dead start 2023-05-22T15:40:11.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:11.530+0000 7f53bd60b640 1 -- 172.21.15.16:0/1539174525 shutdown_connections 2023-05-22T15:40:11.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:11.530+0000 7f53bd60b640 1 -- 172.21.15.16:0/1539174525 >> 172.21.15.16:0/1539174525 conn(0x7f53b8069720 msgr2=0x7f53b8069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:40:11.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:11.530+0000 7f53bd60b640 1 -- 172.21.15.16:0/1539174525 shutdown_connections 2023-05-22T15:40:11.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:11.530+0000 7f53bd60b640 1 -- 172.21.15.16:0/1539174525 wait complete. 2023-05-22T15:40:11.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:11.534+0000 7f53bd60b640 1 Processor -- start 2023-05-22T15:40:11.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:11.534+0000 7f53bd60b640 1 -- start start 2023-05-22T15:40:11.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:11.534+0000 7f53bd60b640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f53a0003f00 0x7f53b8154620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:40:11.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:11.534+0000 7f53bd60b640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f53b80c5ee0 0x7f53b80cc900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:40:11.544 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:11.534+0000 7f53bd60b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f53b81539e0 0x7f53b80aec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:40:11.544 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:11.534+0000 7f53bd60b640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53b80cce40 con 0x7f53b81539e0 2023-05-22T15:40:11.544 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:11.534+0000 7f53bd60b640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53b80cd110 con 0x7f53a0003f00 2023-05-22T15:40:11.544 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:11.534+0000 7f53bd60b640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53b80cd3f0 con 0x7f53b80c5ee0 2023-05-22T15:40:11.544 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:11.534+0000 7f53b77fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f53b81539e0 0x7f53b80aec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:40:11.544 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:11.534+0000 7f53b77fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f53b81539e0 0x7f53b80aec80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42222/0 (socket says 172.21.15.16:42222) 2023-05-22T15:40:11.544 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:11.534+0000 7f53b77fe640 1 -- 172.21.15.16:0/2263754936 learned_addr learned my addr 172.21.15.16:0/2263754936 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:40:11.544 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:11.534+0000 7f53b6ffd640 1 --2- 172.21.15.16:0/2263754936 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f53a0003f00 0x7f53b8154620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:40:11.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:11.534+0000 7f53b67fc640 1 --2- 172.21.15.16:0/2263754936 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f53b80c5ee0 0x7f53b80cc900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:40:11.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:11.534+0000 7f53b77fe640 1 -- 172.21.15.16:0/2263754936 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f53b80c5ee0 msgr2=0x7f53b80cc900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:40:11.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:11.534+0000 7f53b77fe640 1 --2- 172.21.15.16:0/2263754936 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f53b80c5ee0 0x7f53b80cc900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:11.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:11.534+0000 7f53b77fe640 1 -- 172.21.15.16:0/2263754936 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f53a0003f00 msgr2=0x7f53b8154620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:40:11.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:11.534+0000 7f53b77fe640 1 --2- 172.21.15.16:0/2263754936 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f53a0003f00 0x7f53b8154620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:11.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:11.534+0000 7f53b77fe640 1 -- 172.21.15.16:0/2263754936 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53b80c8e80 con 0x7f53b81539e0 2023-05-22T15:40:11.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:11.534+0000 7f53b67fc640 1 --2- 172.21.15.16:0/2263754936 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f53b80c5ee0 0x7f53b80cc900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:40:11.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:11.534+0000 7f53b77fe640 1 --2- 172.21.15.16:0/2263754936 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f53b81539e0 0x7f53b80aec80 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7f53b80cdef0 tx=0x7f53b0007d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:40:11.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:11.534+0000 7f539f7fe640 1 -- 172.21.15.16:0/2263754936 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53b000e070 con 0x7f53b81539e0 2023-05-22T15:40:11.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:11.534+0000 7f539f7fe640 1 -- 172.21.15.16:0/2263754936 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53b0003c70 con 0x7f53b81539e0 2023-05-22T15:40:11.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:11.534+0000 7f53bd60b640 1 -- 172.21.15.16:0/2263754936 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53b80ca8b0 con 0x7f53b81539e0 2023-05-22T15:40:11.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:11.534+0000 7f539f7fe640 1 -- 172.21.15.16:0/2263754936 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53b0003e60 con 0x7f53b81539e0 2023-05-22T15:40:11.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:11.534+0000 7f53bd60b640 1 -- 172.21.15.16:0/2263754936 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53b80c9010 con 0x7f53b81539e0 2023-05-22T15:40:11.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:11.534+0000 7f53bd60b640 1 -- 172.21.15.16:0/2263754936 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53b804dbe0 con 0x7f53b81539e0 2023-05-22T15:40:11.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:11.538+0000 7f539f7fe640 1 -- 172.21.15.16:0/2263754936 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f53b0004030 con 0x7f53b81539e0 2023-05-22T15:40:11.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:11.538+0000 7f539f7fe640 1 --2- 172.21.15.16:0/2263754936 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f53a40654d0 0x7f53a4067980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:40:11.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:11.538+0000 7f539f7fe640 1 -- 172.21.15.16:0/2263754936 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f53b0083620 con 0x7f53b81539e0 2023-05-22T15:40:11.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:11.538+0000 7f53b6ffd640 1 --2- 172.21.15.16:0/2263754936 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f53a40654d0 0x7f53a4067980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:40:11.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:11.538+0000 7f53b6ffd640 1 --2- 172.21.15.16:0/2263754936 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f53a40654d0 0x7f53a4067980 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f53a8000c10 tx=0x7f53a8009040 comp rx=0 tx=0).ready entity=mgr.6240 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:40:11.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:11.542+0000 7f539f7fe640 1 -- 172.21.15.16:0/2263754936 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f53b0051050 con 0x7f53b81539e0 2023-05-22T15:40:11.723 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:11.718+0000 7f53bd60b640 1 -- 172.21.15.16:0/2263754936 --> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f53b80cb010 con 0x7f53a40654d0 2023-05-22T15:40:11.725 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:11.718+0000 7f539f7fe640 1 -- 172.21.15.16:0/2263754936 <== mgr.6240 v2:172.21.15.16:6832/36921 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f53b80cb010 con 0x7f53a40654d0 2023-05-22T15:40:11.725 INFO:teuthology.orchestra.run.smithi016.stdout:[ 2023-05-22T15:40:11.726 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T15:40:11.726 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "cephfs" 2023-05-22T15:40:11.726 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T15:40:11.726 INFO:teuthology.orchestra.run.smithi016.stdout:] 2023-05-22T15:40:11.727 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:11.722+0000 7f53bd60b640 1 -- 172.21.15.16:0/2263754936 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f53a40654d0 msgr2=0x7f53a4067980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:40:11.727 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:11.722+0000 7f53bd60b640 1 --2- 172.21.15.16:0/2263754936 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f53a40654d0 0x7f53a4067980 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f53a8000c10 tx=0x7f53a8009040 comp rx=0 tx=0).stop 2023-05-22T15:40:11.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:11.722+0000 7f53bd60b640 1 -- 172.21.15.16:0/2263754936 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f53b81539e0 msgr2=0x7f53b80aec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:40:11.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:11.722+0000 7f53bd60b640 1 --2- 172.21.15.16:0/2263754936 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f53b81539e0 0x7f53b80aec80 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7f53b80cdef0 tx=0x7f53b0007d40 comp rx=0 tx=0).stop 2023-05-22T15:40:11.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:11.722+0000 7f53bd60b640 1 -- 172.21.15.16:0/2263754936 shutdown_connections 2023-05-22T15:40:11.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:11.722+0000 7f53bd60b640 1 --2- 172.21.15.16:0/2263754936 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f53b81539e0 0x7f53b80aec80 unknown :-1 s=CLOSED pgs=943 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:11.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:11.722+0000 7f53bd60b640 1 --2- 172.21.15.16:0/2263754936 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f53a40654d0 0x7f53a4067980 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:11.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:11.722+0000 7f53bd60b640 1 --2- 172.21.15.16:0/2263754936 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f53b80c5ee0 0x7f53b80cc900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:11.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:11.722+0000 7f53bd60b640 1 --2- 172.21.15.16:0/2263754936 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f53a0003f00 0x7f53b8154620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:11.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:11.722+0000 7f53bd60b640 1 -- 172.21.15.16:0/2263754936 >> 172.21.15.16:0/2263754936 conn(0x7f53b8069720 msgr2=0x7f53b81603e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:40:11.729 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:11.722+0000 7f53bd60b640 1 -- 172.21.15.16:0/2263754936 shutdown_connections 2023-05-22T15:40:11.729 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:11.722+0000 7f53bd60b640 1 -- 172.21.15.16:0/2263754936 wait complete. 2023-05-22T15:40:11.744 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2023-05-22T15:40:11.873 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:11.866+0000 7f2c47876640 1 Processor -- start 2023-05-22T15:40:11.873 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:11.866+0000 7f2c47876640 1 -- start start 2023-05-22T15:40:11.874 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:11.866+0000 7f2c47876640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2c4015bca0 0x7f2c40160160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:40:11.874 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:11.866+0000 7f2c47876640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f2c400b4a70 0x7f2c400d0890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:40:11.874 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:11.866+0000 7f2c47876640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c400d0400 con 0x7f2c400b3ee0 2023-05-22T15:40:11.874 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:11.866+0000 7f2c47876640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c4015c070 con 0x7f2c4015bca0 2023-05-22T15:40:11.875 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:11.866+0000 7f2c47876640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c401606a0 con 0x7f2c400b4a70 2023-05-22T15:40:11.875 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:11.866+0000 7f2c45612640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2c4015bca0 0x7f2c40160160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:40:11.875 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:11.866+0000 7f2c45612640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2c4015bca0 0x7f2c40160160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42236/0 (socket says 172.21.15.16:42236) 2023-05-22T15:40:11.875 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:11.866+0000 7f2c44e11640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f2c400b3ee0 0x7f2c400b4360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:59066/0 (socket says 172.21.15.16:59066) 2023-05-22T15:40:11.875 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:11.866+0000 7f2c45612640 1 -- 172.21.15.16:0/2252788928 learned_addr learned my addr 172.21.15.16:0/2252788928 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:40:11.875 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:11.870+0000 7f2c45e13640 1 --2- 172.21.15.16:0/2252788928 >> v2:172.21.15.62:3300/0 conn(0x7f2c400b4a70 0x7f2c400d0890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:40:11.876 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:11.870+0000 7f2c45612640 1 -- 172.21.15.16:0/2252788928 >> v2:172.21.15.62:3300/0 conn(0x7f2c400b4a70 msgr2=0x7f2c400d0890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:40:11.876 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:11.870+0000 7f2c45612640 1 --2- 172.21.15.16:0/2252788928 >> v2:172.21.15.62:3300/0 conn(0x7f2c400b4a70 0x7f2c400d0890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:11.876 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:11.870+0000 7f2c45612640 1 -- 172.21.15.16:0/2252788928 >> v1:172.21.15.16:6789/0 conn(0x7f2c400b3ee0 legacy=0x7f2c400b4360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:40:11.876 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:11.870+0000 7f2c45612640 1 -- 172.21.15.16:0/2252788928 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c400d00d0 con 0x7f2c4015bca0 2023-05-22T15:40:11.877 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:11.870+0000 7f2c45612640 1 --2- 172.21.15.16:0/2252788928 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2c4015bca0 0x7f2c40160160 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7f2c4008b6e0 tx=0x7f2c3001c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ed8833115ba5d3b server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:40:11.877 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:11.870+0000 7f2c3ffff640 1 -- 172.21.15.16:0/2252788928 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c300a5740 con 0x7f2c4015bca0 2023-05-22T15:40:11.877 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:11.870+0000 7f2c3ffff640 1 -- 172.21.15.16:0/2252788928 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c300a58a0 con 0x7f2c4015bca0 2023-05-22T15:40:11.878 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:11.870+0000 7f2c3ffff640 1 -- 172.21.15.16:0/2252788928 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c300a5c30 con 0x7f2c4015bca0 2023-05-22T15:40:11.878 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:11.870+0000 7f2c47876640 1 -- 172.21.15.16:0/2252788928 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2c4015bca0 msgr2=0x7f2c40160160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:40:11.878 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:11.870+0000 7f2c47876640 1 --2- 172.21.15.16:0/2252788928 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2c4015bca0 0x7f2c40160160 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7f2c4008b6e0 tx=0x7f2c3001c880 comp rx=0 tx=0).stop 2023-05-22T15:40:11.878 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:11.870+0000 7f2c47876640 1 -- 172.21.15.16:0/2252788928 shutdown_connections 2023-05-22T15:40:11.878 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:11.870+0000 7f2c47876640 1 --2- 172.21.15.16:0/2252788928 >> v2:172.21.15.62:3300/0 conn(0x7f2c400b4a70 0x7f2c400d0890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:11.878 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:11.870+0000 7f2c47876640 1 --2- 172.21.15.16:0/2252788928 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2c4015bca0 0x7f2c40160160 unknown :-1 s=CLOSED pgs=945 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:11.878 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:11.870+0000 7f2c47876640 1 -- 172.21.15.16:0/2252788928 >> 172.21.15.16:0/2252788928 conn(0x7f2c400657f0 msgr2=0x7f2c40070e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:40:11.879 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:11.870+0000 7f2c47876640 1 -- 172.21.15.16:0/2252788928 shutdown_connections 2023-05-22T15:40:11.879 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:11.870+0000 7f2c47876640 1 -- 172.21.15.16:0/2252788928 wait complete. 2023-05-22T15:40:11.880 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:11.874+0000 7f2c47876640 1 Processor -- start 2023-05-22T15:40:11.880 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:11.874+0000 7f2c47876640 1 -- start start 2023-05-22T15:40:11.881 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:11.874+0000 7f2c47876640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2c400b3ee0 0x7f2c400b2300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:40:11.881 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:11.874+0000 7f2c47876640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2c400b4a70 0x7f2c4014d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:40:11.881 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:11.874+0000 7f2c47876640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2c4015bca0 0x7f2c4014d8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:40:11.881 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:11.874+0000 7f2c47876640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c4014de30 con 0x7f2c400b4a70 2023-05-22T15:40:11.881 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:11.874+0000 7f2c47876640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c4014e2b0 con 0x7f2c400b3ee0 2023-05-22T15:40:11.881 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:11.874+0000 7f2c47876640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c4014cf20 con 0x7f2c4015bca0 2023-05-22T15:40:11.881 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:11.874+0000 7f2c44e11640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2c400b4a70 0x7f2c4014d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:40:11.882 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:11.874+0000 7f2c44e11640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2c400b4a70 0x7f2c4014d3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42248/0 (socket says 172.21.15.16:42248) 2023-05-22T15:40:11.882 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:11.874+0000 7f2c44e11640 1 -- 172.21.15.16:0/3198905088 learned_addr learned my addr 172.21.15.16:0/3198905088 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:40:11.882 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:11.874+0000 7f2c45612640 1 --2- 172.21.15.16:0/3198905088 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2c400b3ee0 0x7f2c400b2300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:40:11.882 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:11.874+0000 7f2c45e13640 1 --2- 172.21.15.16:0/3198905088 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2c4015bca0 0x7f2c4014d8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:40:11.882 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:11.874+0000 7f2c44e11640 1 -- 172.21.15.16:0/3198905088 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2c4015bca0 msgr2=0x7f2c4014d8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:40:11.882 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:11.874+0000 7f2c44e11640 1 --2- 172.21.15.16:0/3198905088 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2c4015bca0 0x7f2c4014d8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:11.882 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:11.874+0000 7f2c44e11640 1 -- 172.21.15.16:0/3198905088 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2c400b3ee0 msgr2=0x7f2c400b2300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:40:11.882 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:11.874+0000 7f2c44e11640 1 --2- 172.21.15.16:0/3198905088 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2c400b3ee0 0x7f2c400b2300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:11.883 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:11.874+0000 7f2c44e11640 1 -- 172.21.15.16:0/3198905088 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c30009b30 con 0x7f2c400b4a70 2023-05-22T15:40:11.883 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:11.874+0000 7f2c44e11640 1 --2- 172.21.15.16:0/3198905088 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2c400b4a70 0x7f2c4014d3b0 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7f2c3400c450 tx=0x7f2c3400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:40:11.883 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:11.874+0000 7f2c3dffb640 1 -- 172.21.15.16:0/3198905088 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c34007480 con 0x7f2c400b4a70 2023-05-22T15:40:11.883 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:11.874+0000 7f2c3dffb640 1 -- 172.21.15.16:0/3198905088 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c340075e0 con 0x7f2c400b4a70 2023-05-22T15:40:11.883 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:11.874+0000 7f2c47876640 1 -- 172.21.15.16:0/3198905088 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c400d15e0 con 0x7f2c400b4a70 2023-05-22T15:40:11.883 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:11.874+0000 7f2c3dffb640 1 -- 172.21.15.16:0/3198905088 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c34004a30 con 0x7f2c400b4a70 2023-05-22T15:40:11.883 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:11.874+0000 7f2c47876640 1 -- 172.21.15.16:0/3198905088 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c400cd1a0 con 0x7f2c400b4a70 2023-05-22T15:40:11.884 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:11.874+0000 7f2c47876640 1 -- 172.21.15.16:0/3198905088 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c400d1770 con 0x7f2c400b4a70 2023-05-22T15:40:11.884 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:11.878+0000 7f2c3dffb640 1 -- 172.21.15.16:0/3198905088 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f2c34004b90 con 0x7f2c400b4a70 2023-05-22T15:40:11.884 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:11.878+0000 7f2c3dffb640 1 --2- 172.21.15.16:0/3198905088 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f2c2c0654a0 0x7f2c2c067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:40:11.885 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:11.878+0000 7f2c3dffb640 1 -- 172.21.15.16:0/3198905088 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f2c34081b70 con 0x7f2c400b4a70 2023-05-22T15:40:11.885 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:11.878+0000 7f2c45612640 1 --2- 172.21.15.16:0/3198905088 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f2c2c0654a0 0x7f2c2c067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:40:11.885 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:11.878+0000 7f2c45612640 1 --2- 172.21.15.16:0/3198905088 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f2c2c0654a0 0x7f2c2c067950 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f2c300ae000 tx=0x7f2c300a7070 comp rx=0 tx=0).ready entity=mgr.6240 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:40:11.886 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:11.882+0000 7f2c3dffb640 1 -- 172.21.15.16:0/3198905088 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f2c3404f5a0 con 0x7f2c400b4a70 2023-05-22T15:40:12.075 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:12.070+0000 7f2c47876640 1 -- 172.21.15.16:0/3198905088 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7f2c4004dbe0 con 0x7f2c400b4a70 2023-05-22T15:40:12.384 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:12.378+0000 7f2c3dffb640 1 -- 172.21.15.16:0/3198905088 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mgrmap(e 17) v1 ==== 84349+0+0 (secure 0 0 0) 0x7f2c3407fdf0 con 0x7f2c400b4a70 2023-05-22T15:40:12.385 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:12.378+0000 7f2c3dffb640 1 -- 172.21.15.16:0/3198905088 <== mon.0 v2:172.21.15.16:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v17) v1 ==== 92+0+0 (secure 0 0 0) 0x7f2c3404f5a0 con 0x7f2c400b4a70 2023-05-22T15:40:12.388 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:12.382+0000 7f2c47876640 1 -- 172.21.15.16:0/3198905088 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f2c2c0654a0 msgr2=0x7f2c2c067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:40:12.389 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:12.382+0000 7f2c47876640 1 --2- 172.21.15.16:0/3198905088 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f2c2c0654a0 0x7f2c2c067950 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f2c300ae000 tx=0x7f2c300a7070 comp rx=0 tx=0).stop 2023-05-22T15:40:12.389 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:12.382+0000 7f2c47876640 1 -- 172.21.15.16:0/3198905088 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2c400b4a70 msgr2=0x7f2c4014d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:40:12.389 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:12.382+0000 7f2c47876640 1 --2- 172.21.15.16:0/3198905088 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2c400b4a70 0x7f2c4014d3b0 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7f2c3400c450 tx=0x7f2c3400c910 comp rx=0 tx=0).stop 2023-05-22T15:40:12.389 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:12.382+0000 7f2c47876640 1 -- 172.21.15.16:0/3198905088 shutdown_connections 2023-05-22T15:40:12.389 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:12.382+0000 7f2c47876640 1 --2- 172.21.15.16:0/3198905088 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f2c2c0654a0 0x7f2c2c067950 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:12.389 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:12.382+0000 7f2c47876640 1 --2- 172.21.15.16:0/3198905088 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2c4015bca0 0x7f2c4014d8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:12.389 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:12.382+0000 7f2c47876640 1 --2- 172.21.15.16:0/3198905088 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2c400b4a70 0x7f2c4014d3b0 unknown :-1 s=CLOSED pgs=946 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:12.390 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:12.382+0000 7f2c47876640 1 --2- 172.21.15.16:0/3198905088 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2c400b3ee0 0x7f2c400b2300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:12.390 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:12.382+0000 7f2c47876640 1 -- 172.21.15.16:0/3198905088 >> 172.21.15.16:0/3198905088 conn(0x7f2c400657f0 msgr2=0x7f2c400b6be0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:40:12.390 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:12.382+0000 7f2c47876640 1 -- 172.21.15.16:0/3198905088 shutdown_connections 2023-05-22T15:40:12.390 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:12.382+0000 7f2c47876640 1 -- 172.21.15.16:0/3198905088 wait complete. 2023-05-22T15:40:12.409 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2023-05-22T15:40:12.508 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:40:12.505+0000 7f5831205180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-05-22T15:40:12.510 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:40:12.506+0000 7feb5e26d180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-05-22T15:40:12.530 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:12.526+0000 7f0aac707640 1 Processor -- start 2023-05-22T15:40:12.530 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:12.526+0000 7f0aac707640 1 -- start start 2023-05-22T15:40:12.532 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:12.526+0000 7f0aac707640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0aa415c350 0x7f0aa415c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:40:12.532 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:12.526+0000 7f0aac707640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f0aa4069720 0x7f0aa4069b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:40:12.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:12.526+0000 7f0aac707640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0aa40c95f0 con 0x7f0aa40ade80 2023-05-22T15:40:12.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:12.526+0000 7f0aac707640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0aa40ca270 con 0x7f0aa4069720 2023-05-22T15:40:12.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:12.526+0000 7f0aac707640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0aa40ca690 con 0x7f0aa415c350 2023-05-22T15:40:12.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:12.526+0000 7f0aaaca4640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f0aa4069720 0x7f0aa4069b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:40:12.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:12.526+0000 7f0aaa4a3640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f0aa40ade80 0x7f0aa40cc400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:59086/0 (socket says 172.21.15.16:59086) 2023-05-22T15:40:12.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:12.526+0000 7f0aaa4a3640 1 -- 172.21.15.16:0/540830344 learned_addr learned my addr 172.21.15.16:0/540830344 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:40:12.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:12.526+0000 7f0aa9ca2640 1 --2- 172.21.15.16:0/540830344 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0aa415c350 0x7f0aa415c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:40:12.534 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:12.526+0000 7f0aa94a1640 1 -- 172.21.15.16:0/540830344 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2912609530 0 0) 0x7f0aa40c95f0 con 0x7f0aa40ade80 2023-05-22T15:40:12.534 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:12.526+0000 7f0aa94a1640 1 -- 172.21.15.16:0/540830344 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a90003530 con 0x7f0aa40ade80 2023-05-22T15:40:12.534 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:12.526+0000 7f0aaaca4640 1 -- 172.21.15.16:0/540830344 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0aa415c350 msgr2=0x7f0aa415c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:40:12.534 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:12.526+0000 7f0aaaca4640 1 --2- 172.21.15.16:0/540830344 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0aa415c350 0x7f0aa415c7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:12.534 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:12.526+0000 7f0aaaca4640 1 -- 172.21.15.16:0/540830344 >> v1:172.21.15.16:6789/0 conn(0x7f0aa40ade80 legacy=0x7f0aa40cc400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:40:12.534 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:12.526+0000 7f0aaaca4640 1 -- 172.21.15.16:0/540830344 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0aa40c98d0 con 0x7f0aa4069720 2023-05-22T15:40:12.536 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:12.526+0000 7f0aaaca4640 1 --2- 172.21.15.16:0/540830344 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0aa4069720 0x7f0aa4069b60 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7f0a9c0097f0 tx=0x7f0a9c01c3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d1e481b09a860fa server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:40:12.536 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:12.530+0000 7f0aa94a1640 1 -- 172.21.15.16:0/540830344 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a9c0ab600 con 0x7f0aa4069720 2023-05-22T15:40:12.536 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:12.530+0000 7f0aa94a1640 1 -- 172.21.15.16:0/540830344 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a9c0a7050 con 0x7f0aa4069720 2023-05-22T15:40:12.536 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:12.530+0000 7f0aa94a1640 1 -- 172.21.15.16:0/540830344 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a9c0ab980 con 0x7f0aa4069720 2023-05-22T15:40:12.536 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:12.530+0000 7f0aac707640 1 -- 172.21.15.16:0/540830344 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0aa4069720 msgr2=0x7f0aa4069b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:40:12.536 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:12.530+0000 7f0aac707640 1 --2- 172.21.15.16:0/540830344 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0aa4069720 0x7f0aa4069b60 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7f0a9c0097f0 tx=0x7f0a9c01c3f0 comp rx=0 tx=0).stop 2023-05-22T15:40:12.537 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:12.530+0000 7f0aac707640 1 -- 172.21.15.16:0/540830344 shutdown_connections 2023-05-22T15:40:12.537 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:12.530+0000 7f0aac707640 1 --2- 172.21.15.16:0/540830344 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0aa4069720 0x7f0aa4069b60 unknown :-1 s=CLOSED pgs=950 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:12.537 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:12.530+0000 7f0aac707640 1 --2- 172.21.15.16:0/540830344 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0aa415c350 0x7f0aa415c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:12.537 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:12.530+0000 7f0aac707640 1 -- 172.21.15.16:0/540830344 >> 172.21.15.16:0/540830344 conn(0x7f0aa40b1110 msgr2=0x7f0aa40ce830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:40:12.537 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:12.530+0000 7f0aac707640 1 -- 172.21.15.16:0/540830344 shutdown_connections 2023-05-22T15:40:12.537 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:12.530+0000 7f0aac707640 1 -- 172.21.15.16:0/540830344 wait complete. 2023-05-22T15:40:12.538 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:12.530+0000 7f0aac707640 1 Processor -- start 2023-05-22T15:40:12.538 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:12.530+0000 7f0aac707640 1 -- start start 2023-05-22T15:40:12.540 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:12.530+0000 7f0aac707640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0aa4069720 0x7f0aa4148a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:40:12.540 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:12.530+0000 7f0aac707640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0aa40ade80 0x7f0aa41483e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:40:12.541 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:12.530+0000 7f0aac707640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0aa415c350 0x7f0aa40ce830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:40:12.541 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:12.530+0000 7f0aac707640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0aa4152800 con 0x7f0aa415c350 2023-05-22T15:40:12.541 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:12.530+0000 7f0aac707640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0aa4148920 con 0x7f0aa4069720 2023-05-22T15:40:12.541 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:12.530+0000 7f0aac707640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0aa40ced70 con 0x7f0aa40ade80 2023-05-22T15:40:12.541 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:12.530+0000 7f0aaaca4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0aa415c350 0x7f0aa40ce830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:40:12.541 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:12.530+0000 7f0aaaca4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0aa415c350 0x7f0aa40ce830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42280/0 (socket says 172.21.15.16:42280) 2023-05-22T15:40:12.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:12.530+0000 7f0aaaca4640 1 -- 172.21.15.16:0/1664122525 learned_addr learned my addr 172.21.15.16:0/1664122525 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:40:12.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:12.530+0000 7f0aa9ca2640 1 --2- 172.21.15.16:0/1664122525 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0aa40ade80 0x7f0aa41483e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:40:12.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:12.530+0000 7f0aaa4a3640 1 --2- 172.21.15.16:0/1664122525 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0aa4069720 0x7f0aa4148a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:40:12.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:12.530+0000 7f0aaaca4640 1 -- 172.21.15.16:0/1664122525 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0aa40ade80 msgr2=0x7f0aa41483e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:40:12.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:12.530+0000 7f0aaaca4640 1 --2- 172.21.15.16:0/1664122525 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0aa40ade80 0x7f0aa41483e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:12.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:12.530+0000 7f0aaaca4640 1 -- 172.21.15.16:0/1664122525 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0aa4069720 msgr2=0x7f0aa4148a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:40:12.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:12.530+0000 7f0aaaca4640 1 --2- 172.21.15.16:0/1664122525 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0aa4069720 0x7f0aa4148a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:12.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:12.530+0000 7f0aaaca4640 1 -- 172.21.15.16:0/1664122525 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a9c009420 con 0x7f0aa415c350 2023-05-22T15:40:12.544 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:12.530+0000 7f0aaaca4640 1 --2- 172.21.15.16:0/1664122525 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0aa415c350 0x7f0aa40ce830 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7f0a9c0ae040 tx=0x7f0a9c019190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:40:12.544 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:12.530+0000 7f0a9affd640 1 -- 172.21.15.16:0/1664122525 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a9c0b4410 con 0x7f0aa415c350 2023-05-22T15:40:12.544 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:12.530+0000 7f0a9affd640 1 -- 172.21.15.16:0/1664122525 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a9c0b4570 con 0x7f0aa415c350 2023-05-22T15:40:12.544 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:12.530+0000 7f0aac707640 1 -- 172.21.15.16:0/1664122525 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0aa40c9210 con 0x7f0aa415c350 2023-05-22T15:40:12.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:12.530+0000 7f0a9affd640 1 -- 172.21.15.16:0/1664122525 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a9c0b4900 con 0x7f0aa415c350 2023-05-22T15:40:12.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:12.530+0000 7f0aac707640 1 -- 172.21.15.16:0/1664122525 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0aa415d090 con 0x7f0aa415c350 2023-05-22T15:40:12.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:12.534+0000 7f0aac707640 1 -- 172.21.15.16:0/1664122525 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0aa40c93a0 con 0x7f0aa415c350 2023-05-22T15:40:12.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:12.534+0000 7f0a9affd640 1 -- 172.21.15.16:0/1664122525 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) v1 ==== 84349+0+0 (secure 0 0 0) 0x7f0a9c0b2040 con 0x7f0aa415c350 2023-05-22T15:40:12.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:12.534+0000 7f0a9affd640 1 --2- 172.21.15.16:0/1664122525 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f0a880654f0 0x7f0a880679a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:40:12.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:12.534+0000 7f0aaa4a3640 1 -- 172.21.15.16:0/1664122525 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f0a880654f0 msgr2=0x7f0a880679a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/36921 2023-05-22T15:40:12.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:12.534+0000 7f0a9affd640 1 -- 172.21.15.16:0/1664122525 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f0aa406a8b0 con 0x7f0aa415c350 2023-05-22T15:40:12.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:12.534+0000 7f0aaa4a3640 1 --2- 172.21.15.16:0/1664122525 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f0a880654f0 0x7f0a880679a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-05-22T15:40:12.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:12.538+0000 7f0a9affd640 1 -- 172.21.15.16:0/1664122525 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f0a9c0ea5c0 con 0x7f0aa415c350 2023-05-22T15:40:12.696 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:12.690+0000 7f0aac707640 1 -- 172.21.15.16:0/1664122525 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7f0aa4149300 con 0x7f0aa415c350 2023-05-22T15:40:12.703 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:12.698+0000 7f0a9affd640 1 -- 172.21.15.16:0/1664122525 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v17)=0 v17) v1 ==== 138+0+0 (secure 0 0 0) 0x7f0a9c0c20d0 con 0x7f0aa415c350 2023-05-22T15:40:12.706 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:12.698+0000 7f0aac707640 1 -- 172.21.15.16:0/1664122525 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f0a880654f0 msgr2=0x7f0a880679a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-05-22T15:40:12.706 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:12.698+0000 7f0aac707640 1 --2- 172.21.15.16:0/1664122525 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f0a880654f0 0x7f0a880679a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:12.706 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:12.698+0000 7f0aac707640 1 -- 172.21.15.16:0/1664122525 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0aa415c350 msgr2=0x7f0aa40ce830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:40:12.706 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:12.698+0000 7f0aac707640 1 --2- 172.21.15.16:0/1664122525 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0aa415c350 0x7f0aa40ce830 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7f0a9c0ae040 tx=0x7f0a9c019190 comp rx=0 tx=0).stop 2023-05-22T15:40:12.706 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:12.698+0000 7f0aac707640 1 -- 172.21.15.16:0/1664122525 shutdown_connections 2023-05-22T15:40:12.706 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:12.698+0000 7f0aac707640 1 --2- 172.21.15.16:0/1664122525 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0aa415c350 0x7f0aa40ce830 unknown :-1 s=CLOSED pgs=951 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:12.706 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:12.698+0000 7f0aac707640 1 --2- 172.21.15.16:0/1664122525 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f0a880654f0 0x7f0a880679a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:12.707 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:12.698+0000 7f0aac707640 1 --2- 172.21.15.16:0/1664122525 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0aa40ade80 0x7f0aa41483e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:12.707 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:12.698+0000 7f0aac707640 1 --2- 172.21.15.16:0/1664122525 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0aa4069720 0x7f0aa4148a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:12.707 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:12.698+0000 7f0aac707640 1 -- 172.21.15.16:0/1664122525 >> 172.21.15.16:0/1664122525 conn(0x7f0aa40b1110 msgr2=0x7f0aa40ccba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:40:12.707 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:12.698+0000 7f0aac707640 1 -- 172.21.15.16:0/1664122525 shutdown_connections 2023-05-22T15:40:12.707 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:12.698+0000 7f0aac707640 1 -- 172.21.15.16:0/1664122525 wait complete. 2023-05-22T15:40:12.722 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2023-05-22T15:40:12.838 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:12.830+0000 7f9bdd8ac640 1 Processor -- start 2023-05-22T15:40:12.838 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:12.830+0000 7f9bdd8ac640 1 -- start start 2023-05-22T15:40:12.839 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:12.834+0000 7f9bdd8ac640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f9bd815bca0 0x7f9bd80b5220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:40:12.839 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:12.834+0000 7f9bdd8ac640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9bd80cd230 con 0x7f9bd80b4560 2023-05-22T15:40:12.839 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:12.834+0000 7f9bdd8ac640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9bd80ceb10 con 0x7f9bd80ce650 2023-05-22T15:40:12.839 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:12.834+0000 7f9bdd8ac640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9bd80cc990 con 0x7f9bd815bca0 2023-05-22T15:40:12.839 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:12.834+0000 7f9bd77fe640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f9bd80b4560 0x7f9bd80d0d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:59088/0 (socket says 172.21.15.16:59088) 2023-05-22T15:40:12.839 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:12.834+0000 7f9bd77fe640 1 -- 172.21.15.16:0/2187019581 learned_addr learned my addr 172.21.15.16:0/2187019581 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:40:12.841 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:12.834+0000 7f9bd6ffd640 1 --2- 172.21.15.16:0/2187019581 >> v2:172.21.15.62:3300/0 conn(0x7f9bd815bca0 0x7f9bd80b5220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:40:12.841 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:12.834+0000 7f9bd67fc640 1 -- 172.21.15.16:0/2187019581 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2387996315 0 0) 0x7f9bd00014a0 con 0x7f9bd80b4560 2023-05-22T15:40:12.841 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:12.834+0000 7f9bd67fc640 1 -- 172.21.15.16:0/2187019581 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9bc4003530 con 0x7f9bd80b4560 2023-05-22T15:40:12.842 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:12.834+0000 7f9bd67fc640 1 -- 172.21.15.16:0/2187019581 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2518575319 0 0) 0x7f9bc4003530 con 0x7f9bd80b4560 2023-05-22T15:40:12.842 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:12.834+0000 7f9bd67fc640 1 -- 172.21.15.16:0/2187019581 >> v2:172.21.15.62:3300/0 conn(0x7f9bd815bca0 msgr2=0x7f9bd80b5220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:40:12.842 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:12.834+0000 7f9bd67fc640 1 --2- 172.21.15.16:0/2187019581 >> v2:172.21.15.62:3300/0 conn(0x7f9bd815bca0 0x7f9bd80b5220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:12.842 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:12.834+0000 7f9bd67fc640 1 -- 172.21.15.16:0/2187019581 >> v1:172.21.15.62:6789/0 conn(0x7f9bd80ce650 legacy=0x7f9bd80b3e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:40:12.842 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:12.834+0000 7f9bd67fc640 1 -- 172.21.15.16:0/2187019581 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bd80ccc70 con 0x7f9bd80b4560 2023-05-22T15:40:12.842 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:12.834+0000 7f9bd67fc640 1 -- 172.21.15.16:0/2187019581 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7f9bd0001b90 con 0x7f9bd80b4560 2023-05-22T15:40:12.843 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:12.834+0000 7f9bd67fc640 1 -- 172.21.15.16:0/2187019581 >> v1:172.21.15.16:6789/0 conn(0x7f9bd80b4560 legacy=0x7f9bd80d0d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:40:12.843 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:12.834+0000 7f9bd67fc640 1 --2- 172.21.15.16:0/2187019581 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9bc4003980 0x7f9bc4003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:40:12.843 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:12.834+0000 7f9bd67fc640 1 --2- 172.21.15.16:0/2187019581 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9bc40047d0 0x7f9bc4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:40:12.843 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:12.834+0000 7f9bd67fc640 1 --2- 172.21.15.16:0/2187019581 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9bc4005180 0x7f9bc400b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:40:12.843 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:12.834+0000 7f9bd67fc640 1 -- 172.21.15.16:0/2187019581 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bd80cc990 con 0x7f9bc4005180 2023-05-22T15:40:12.844 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:12.834+0000 7f9bd67fc640 1 -- 172.21.15.16:0/2187019581 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bc400bfe0 con 0x7f9bc4003980 2023-05-22T15:40:12.844 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:12.834+0000 7f9bd67fc640 1 -- 172.21.15.16:0/2187019581 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9bc400c1a0 con 0x7f9bc40047d0 2023-05-22T15:40:12.844 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:12.834+0000 7f9bd77fe640 1 --2- 172.21.15.16:0/2187019581 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9bc4005180 0x7f9bc400b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:40:12.844 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:12.834+0000 7f9bd6ffd640 1 --2- 172.21.15.16:0/2187019581 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9bc4003980 0x7f9bc4003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:40:12.844 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:12.834+0000 7f9bcffff640 1 --2- 172.21.15.16:0/2187019581 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9bc40047d0 0x7f9bc4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:40:12.844 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:12.834+0000 7f9bd77fe640 1 -- 172.21.15.16:0/2187019581 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9bc40047d0 msgr2=0x7f9bc4004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:40:12.845 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:12.834+0000 7f9bd77fe640 1 --2- 172.21.15.16:0/2187019581 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9bc40047d0 0x7f9bc4004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:12.845 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:12.834+0000 7f9bd77fe640 1 -- 172.21.15.16:0/2187019581 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9bc4003980 msgr2=0x7f9bc4003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:40:12.845 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:12.834+0000 7f9bd77fe640 1 --2- 172.21.15.16:0/2187019581 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9bc4003980 0x7f9bc4003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:12.845 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:12.834+0000 7f9bd77fe640 1 -- 172.21.15.16:0/2187019581 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bc4003530 con 0x7f9bc4005180 2023-05-22T15:40:12.845 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:12.834+0000 7f9bcffff640 1 --2- 172.21.15.16:0/2187019581 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9bc40047d0 0x7f9bc4004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:40:12.851 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:12.838+0000 7f9bd77fe640 1 -- 172.21.15.16:0/2187019581 reap_dead start 2023-05-22T15:40:12.851 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:12.838+0000 7f9bd77fe640 1 --2- 172.21.15.16:0/2187019581 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9bc4005180 0x7f9bc400b990 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7f9bd000a810 tx=0x7f9bd001d320 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e95ffd779499898 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:40:12.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:12.838+0000 7f9bd67fc640 1 -- 172.21.15.16:0/2187019581 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bd00ad630 con 0x7f9bc4005180 2023-05-22T15:40:12.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:12.838+0000 7f9bd67fc640 1 -- 172.21.15.16:0/2187019581 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9bd00a8070 con 0x7f9bc4005180 2023-05-22T15:40:12.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:12.838+0000 7f9bd67fc640 1 -- 172.21.15.16:0/2187019581 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bd00ad9f0 con 0x7f9bc4005180 2023-05-22T15:40:12.853 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:12.838+0000 7f9bdd8ac640 1 -- 172.21.15.16:0/2187019581 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9bc4005180 msgr2=0x7f9bc400b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:40:12.853 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:12.838+0000 7f9bdd8ac640 1 --2- 172.21.15.16:0/2187019581 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9bc4005180 0x7f9bc400b990 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7f9bd000a810 tx=0x7f9bd001d320 comp rx=0 tx=0).stop 2023-05-22T15:40:12.853 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:12.838+0000 7f9bdd8ac640 1 -- 172.21.15.16:0/2187019581 shutdown_connections 2023-05-22T15:40:12.853 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:12.838+0000 7f9bdd8ac640 1 --2- 172.21.15.16:0/2187019581 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9bc4005180 0x7f9bc400b990 unknown :-1 s=CLOSED pgs=953 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:12.854 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:12.838+0000 7f9bdd8ac640 1 -- 172.21.15.16:0/2187019581 >> 172.21.15.16:0/2187019581 conn(0x7f9bd80d23d0 msgr2=0x7f9bd80d27d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:40:12.854 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:12.838+0000 7f9bdd8ac640 1 -- 172.21.15.16:0/2187019581 shutdown_connections 2023-05-22T15:40:12.854 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:12.838+0000 7f9bdd8ac640 1 -- 172.21.15.16:0/2187019581 wait complete. 2023-05-22T15:40:12.854 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:12.838+0000 7f9bdd8ac640 1 Processor -- start 2023-05-22T15:40:12.854 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:12.838+0000 7f9bdd8ac640 1 -- start start 2023-05-22T15:40:12.854 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:12.842+0000 7f9bdd8ac640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9bc4005180 0x7f9bd814e460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:40:12.854 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:12.842+0000 7f9bdd8ac640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9bd815ede0 0x7f9bd80c7e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:40:12.855 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:12.842+0000 7f9bdd8ac640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9bd815c3f0 0x7f9bd815a0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:40:12.855 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:12.842+0000 7f9bdd8ac640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bd814e9a0 con 0x7f9bc4005180 2023-05-22T15:40:12.855 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:12.842+0000 7f9bdd8ac640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bd80c7090 con 0x7f9bd815ede0 2023-05-22T15:40:12.855 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:12.842+0000 7f9bdd8ac640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9bd80c7370 con 0x7f9bd815c3f0 2023-05-22T15:40:12.855 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:12.842+0000 7f9bd6ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9bc4005180 0x7f9bd814e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:40:12.855 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:12.842+0000 7f9bd6ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9bc4005180 0x7f9bd814e460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42312/0 (socket says 172.21.15.16:42312) 2023-05-22T15:40:12.856 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:12.842+0000 7f9bd6ffd640 1 -- 172.21.15.16:0/3467584023 learned_addr learned my addr 172.21.15.16:0/3467584023 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:40:12.856 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:12.842+0000 7f9bcffff640 1 --2- 172.21.15.16:0/3467584023 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9bd815ede0 0x7f9bd80c7e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:40:12.856 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:12.842+0000 7f9bd77fe640 1 --2- 172.21.15.16:0/3467584023 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9bd815c3f0 0x7f9bd815a0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:40:12.856 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:12.842+0000 7f9bd6ffd640 1 -- 172.21.15.16:0/3467584023 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9bd815c3f0 msgr2=0x7f9bd815a0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:40:12.857 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:12.842+0000 7f9bd6ffd640 1 --2- 172.21.15.16:0/3467584023 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9bd815c3f0 0x7f9bd815a0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:12.857 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:12.842+0000 7f9bd6ffd640 1 -- 172.21.15.16:0/3467584023 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9bd815ede0 msgr2=0x7f9bd80c7e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:40:12.857 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:12.842+0000 7f9bd6ffd640 1 --2- 172.21.15.16:0/3467584023 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9bd815ede0 0x7f9bd80c7e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:12.857 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:12.842+0000 7f9bd6ffd640 1 -- 172.21.15.16:0/3467584023 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bd815c070 con 0x7f9bc4005180 2023-05-22T15:40:12.858 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:12.842+0000 7f9bd6ffd640 1 --2- 172.21.15.16:0/3467584023 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9bc4005180 0x7f9bd814e460 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7f9bc8011560 tx=0x7f9bc8011a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:40:12.858 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:12.842+0000 7f9bd4ff9640 1 -- 172.21.15.16:0/3467584023 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bc8015930 con 0x7f9bc4005180 2023-05-22T15:40:12.858 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:12.842+0000 7f9bd4ff9640 1 -- 172.21.15.16:0/3467584023 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9bc8015a90 con 0x7f9bc4005180 2023-05-22T15:40:12.859 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:12.842+0000 7f9bdd8ac640 1 -- 172.21.15.16:0/3467584023 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9bd80ccf50 con 0x7f9bc4005180 2023-05-22T15:40:12.859 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:12.842+0000 7f9bd4ff9640 1 -- 172.21.15.16:0/3467584023 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bc8015e60 con 0x7f9bc4005180 2023-05-22T15:40:12.859 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:12.842+0000 7f9bdd8ac640 1 -- 172.21.15.16:0/3467584023 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9bd815c200 con 0x7f9bc4005180 2023-05-22T15:40:12.859 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:12.842+0000 7f9bdd8ac640 1 -- 172.21.15.16:0/3467584023 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9bd80738f0 con 0x7f9bc4005180 2023-05-22T15:40:12.860 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:12.842+0000 7f9bd4ff9640 1 -- 172.21.15.16:0/3467584023 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) v1 ==== 84349+0+0 (secure 0 0 0) 0x7f9bc800f460 con 0x7f9bc4005180 2023-05-22T15:40:12.860 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:12.842+0000 7f9bd4ff9640 1 --2- 172.21.15.16:0/3467584023 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f9bb8065520 0x7f9bb80679d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:40:12.860 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:12.842+0000 7f9bcffff640 1 -- 172.21.15.16:0/3467584023 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f9bb8065520 msgr2=0x7f9bb80679d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/36921 2023-05-22T15:40:12.861 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:12.842+0000 7f9bd4ff9640 1 -- 172.21.15.16:0/3467584023 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f9bc8083680 con 0x7f9bc4005180 2023-05-22T15:40:12.861 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:12.842+0000 7f9bcffff640 1 --2- 172.21.15.16:0/3467584023 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f9bb8065520 0x7f9bb80679d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-05-22T15:40:12.862 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:12.846+0000 7f9bd4ff9640 1 -- 172.21.15.16:0/3467584023 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f9bc8051140 con 0x7f9bc4005180 2023-05-22T15:40:12.906 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:40:12.902+0000 7feb5e26d180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-05-22T15:40:12.914 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:40:12.913+0000 7f5831205180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-05-22T15:40:13.002 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:12.998+0000 7f9bdd8ac640 1 -- 172.21.15.16:0/3467584023 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7f9bd80b3f30 con 0x7f9bc4005180 2023-05-22T15:40:13.010 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:13.002+0000 7f9bd4ff9640 1 -- 172.21.15.16:0/3467584023 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v18)=0 v18) v1 ==== 133+0+0 (secure 0 0 0) 0x7f9bc8054760 con 0x7f9bc4005180 2023-05-22T15:40:13.014 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:13.010+0000 7f9bdd8ac640 1 -- 172.21.15.16:0/3467584023 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f9bb8065520 msgr2=0x7f9bb80679d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-05-22T15:40:13.015 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:13.010+0000 7f9bdd8ac640 1 --2- 172.21.15.16:0/3467584023 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f9bb8065520 0x7f9bb80679d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:13.015 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:13.010+0000 7f9bdd8ac640 1 -- 172.21.15.16:0/3467584023 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9bc4005180 msgr2=0x7f9bd814e460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:40:13.015 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:13.010+0000 7f9bdd8ac640 1 --2- 172.21.15.16:0/3467584023 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9bc4005180 0x7f9bd814e460 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7f9bc8011560 tx=0x7f9bc8011a20 comp rx=0 tx=0).stop 2023-05-22T15:40:13.015 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:13.010+0000 7f9bdd8ac640 1 -- 172.21.15.16:0/3467584023 shutdown_connections 2023-05-22T15:40:13.015 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:13.010+0000 7f9bdd8ac640 1 --2- 172.21.15.16:0/3467584023 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f9bb8065520 0x7f9bb80679d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:13.015 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:13.010+0000 7f9bdd8ac640 1 --2- 172.21.15.16:0/3467584023 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9bd815c3f0 0x7f9bd815a0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:13.016 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:13.010+0000 7f9bdd8ac640 1 --2- 172.21.15.16:0/3467584023 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9bd815ede0 0x7f9bd80c7e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:13.016 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:13.010+0000 7f9bdd8ac640 1 --2- 172.21.15.16:0/3467584023 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9bc4005180 0x7f9bd814e460 unknown :-1 s=CLOSED pgs=954 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:13.016 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:13.010+0000 7f9bdd8ac640 1 -- 172.21.15.16:0/3467584023 >> 172.21.15.16:0/3467584023 conn(0x7f9bd80d23d0 msgr2=0x7f9bd80b22b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:40:13.016 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:13.010+0000 7f9bdd8ac640 1 -- 172.21.15.16:0/3467584023 shutdown_connections 2023-05-22T15:40:13.016 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:13.010+0000 7f9bdd8ac640 1 -- 172.21.15.16:0/3467584023 wait complete. 2023-05-22T15:40:13.030 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:40:13.031 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1) 2023-05-22T15:40:13.042 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add --fs cephfs snap_test_dir1 1M 2023-05-22T15:40:13.047 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:40:13.042+0000 7feb5e26d180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-05-22T15:40:13.053 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:40:13.049+0000 7f5831205180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-05-22T15:40:13.165 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:13.158+0000 7fd09f301640 1 Processor -- start 2023-05-22T15:40:13.165 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:13.158+0000 7fd09f301640 1 -- start start 2023-05-22T15:40:13.167 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:13.162+0000 7fd09f301640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd0980b1700 0x7fd09806fef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:40:13.168 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:13.162+0000 7fd09f301640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fd098070430 0x7fd09806ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:40:13.168 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:13.162+0000 7fd09f301640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd09806b9c0 con 0x7fd098159ac0 2023-05-22T15:40:13.168 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:13.162+0000 7fd09f301640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd09806c820 con 0x7fd0980b1700 2023-05-22T15:40:13.168 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:13.162+0000 7fd09f301640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd098070870 con 0x7fd098070430 2023-05-22T15:40:13.168 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:13.162+0000 7fd09c89c640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd0980b1700 0x7fd09806fef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:40:13.168 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:13.162+0000 7fd09c89c640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd0980b1700 0x7fd09806fef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42328/0 (socket says 172.21.15.16:42328) 2023-05-22T15:40:13.169 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:13.162+0000 7fd09c89c640 1 -- 172.21.15.16:0/1761336346 learned_addr learned my addr 172.21.15.16:0/1761336346 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:40:13.169 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:13.162+0000 7fd09d89e640 1 --2- 172.21.15.16:0/1761336346 >> v2:172.21.15.62:3300/0 conn(0x7fd098070430 0x7fd09806ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:40:13.169 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:13.162+0000 7fd09c89c640 1 -- 172.21.15.16:0/1761336346 >> v2:172.21.15.62:3300/0 conn(0x7fd098070430 msgr2=0x7fd09806ccb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:40:13.169 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:13.162+0000 7fd09c89c640 1 --2- 172.21.15.16:0/1761336346 >> v2:172.21.15.62:3300/0 conn(0x7fd098070430 0x7fd09806ccb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:13.169 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:13.162+0000 7fd09c89c640 1 -- 172.21.15.16:0/1761336346 >> v1:172.21.15.62:6789/0 conn(0x7fd098159ac0 legacy=0x7fd0980b40b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:40:13.169 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:13.162+0000 7fd09c89c640 1 -- 172.21.15.16:0/1761336346 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0980b1b80 con 0x7fd0980b1700 2023-05-22T15:40:13.170 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:13.162+0000 7fd09c89c640 1 --2- 172.21.15.16:0/1761336346 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd0980b1700 0x7fd09806fef0 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7fd0980c8ed0 tx=0x7fd08801c4c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7b258133ca2acb server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:40:13.170 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:13.166+0000 7fd08ffff640 1 -- 172.21.15.16:0/1761336346 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0880ac410 con 0x7fd0980b1700 2023-05-22T15:40:13.170 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:13.166+0000 7fd08ffff640 1 -- 172.21.15.16:0/1761336346 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0880ac570 con 0x7fd0980b1700 2023-05-22T15:40:13.170 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:13.166+0000 7fd08ffff640 1 -- 172.21.15.16:0/1761336346 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0880ac900 con 0x7fd0980b1700 2023-05-22T15:40:13.170 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:13.166+0000 7fd09f301640 1 -- 172.21.15.16:0/1761336346 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd0980b1700 msgr2=0x7fd09806fef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:40:13.171 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:13.166+0000 7fd09f301640 1 --2- 172.21.15.16:0/1761336346 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd0980b1700 0x7fd09806fef0 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7fd0980c8ed0 tx=0x7fd08801c4c0 comp rx=0 tx=0).stop 2023-05-22T15:40:13.171 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:13.166+0000 7fd09f301640 1 -- 172.21.15.16:0/1761336346 shutdown_connections 2023-05-22T15:40:13.171 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:13.166+0000 7fd09f301640 1 --2- 172.21.15.16:0/1761336346 >> v2:172.21.15.62:3300/0 conn(0x7fd098070430 0x7fd09806ccb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:13.172 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:13.166+0000 7fd09f301640 1 --2- 172.21.15.16:0/1761336346 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd0980b1700 0x7fd09806fef0 unknown :-1 s=CLOSED pgs=955 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:13.172 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:13.166+0000 7fd09f301640 1 -- 172.21.15.16:0/1761336346 >> 172.21.15.16:0/1761336346 conn(0x7fd0980ce8d0 msgr2=0x7fd0980cecd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:40:13.172 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:13.166+0000 7fd09f301640 1 -- 172.21.15.16:0/1761336346 shutdown_connections 2023-05-22T15:40:13.172 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:13.166+0000 7fd09f301640 1 -- 172.21.15.16:0/1761336346 wait complete. 2023-05-22T15:40:13.173 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:13.166+0000 7fd09f301640 1 Processor -- start 2023-05-22T15:40:13.173 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:13.166+0000 7fd09f301640 1 -- start start 2023-05-22T15:40:13.175 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:13.166+0000 7fd09f301640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd098070430 0x7fd0980c4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:40:13.175 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:13.166+0000 7fd09f301640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd0980b1700 0x7fd098153e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:40:13.175 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:13.166+0000 7fd09f301640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd098159ac0 0x7fd0980c63c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:40:13.175 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:13.166+0000 7fd09f301640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd098154360 con 0x7fd0980b1700 2023-05-22T15:40:13.175 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:13.166+0000 7fd09f301640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0980c6900 con 0x7fd098070430 2023-05-22T15:40:13.175 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:13.166+0000 7fd09f301640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0980c6ba0 con 0x7fd098159ac0 2023-05-22T15:40:13.176 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:13.170+0000 7fd09c89c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd0980b1700 0x7fd098153e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:40:13.176 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:13.170+0000 7fd09c89c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd0980b1700 0x7fd098153e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42332/0 (socket says 172.21.15.16:42332) 2023-05-22T15:40:13.176 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:13.170+0000 7fd09c89c640 1 -- 172.21.15.16:0/3606743018 learned_addr learned my addr 172.21.15.16:0/3606743018 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:40:13.176 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:13.170+0000 7fd09d09d640 1 --2- 172.21.15.16:0/3606743018 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd098070430 0x7fd0980c4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:40:13.176 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:13.170+0000 7fd09d89e640 1 --2- 172.21.15.16:0/3606743018 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd098159ac0 0x7fd0980c63c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:40:13.176 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:13.170+0000 7fd09c89c640 1 -- 172.21.15.16:0/3606743018 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd098159ac0 msgr2=0x7fd0980c63c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:40:13.176 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:13.170+0000 7fd09c89c640 1 --2- 172.21.15.16:0/3606743018 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd098159ac0 0x7fd0980c63c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:13.177 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:13.170+0000 7fd09c89c640 1 -- 172.21.15.16:0/3606743018 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd098070430 msgr2=0x7fd0980c4e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:40:13.177 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:13.170+0000 7fd09c89c640 1 --2- 172.21.15.16:0/3606743018 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd098070430 0x7fd0980c4e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:13.177 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:13.170+0000 7fd09c89c640 1 -- 172.21.15.16:0/3606743018 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0880094d0 con 0x7fd0980b1700 2023-05-22T15:40:13.177 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:13.170+0000 7fd09d89e640 1 --2- 172.21.15.16:0/3606743018 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd098159ac0 0x7fd0980c63c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T15:40:13.177 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:13.170+0000 7fd09c89c640 1 --2- 172.21.15.16:0/3606743018 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd0980b1700 0x7fd098153e20 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7fd0880af040 tx=0x7fd088019260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:40:13.177 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:13.170+0000 7fd08dffb640 1 -- 172.21.15.16:0/3606743018 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0880ac410 con 0x7fd0980b1700 2023-05-22T15:40:13.177 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:13.170+0000 7fd08dffb640 1 -- 172.21.15.16:0/3606743018 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0880b7470 con 0x7fd0980b1700 2023-05-22T15:40:13.177 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:13.170+0000 7fd09f301640 1 -- 172.21.15.16:0/3606743018 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd09806c4b0 con 0x7fd0980b1700 2023-05-22T15:40:13.178 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:13.170+0000 7fd08dffb640 1 -- 172.21.15.16:0/3606743018 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0880b7820 con 0x7fd0980b1700 2023-05-22T15:40:13.178 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:13.170+0000 7fd09f301640 1 -- 172.21.15.16:0/3606743018 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd09806c1d0 con 0x7fd0980b1700 2023-05-22T15:40:13.178 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:13.170+0000 7fd09f301640 1 -- 172.21.15.16:0/3606743018 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd09804dbe0 con 0x7fd0980b1700 2023-05-22T15:40:13.179 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:13.170+0000 7fd08dffb640 1 -- 172.21.15.16:0/3606743018 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) v1 ==== 84349+0+0 (secure 0 0 0) 0x7fd0880b3070 con 0x7fd0980b1700 2023-05-22T15:40:13.179 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:13.170+0000 7fd08dffb640 1 --2- 172.21.15.16:0/3606743018 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd07c0654f0 0x7fd07c0679a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:40:13.179 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:13.170+0000 7fd09d09d640 1 -- 172.21.15.16:0/3606743018 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd07c0654f0 msgr2=0x7fd07c0679a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/36921 2023-05-22T15:40:13.179 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:13.170+0000 7fd08dffb640 1 -- 172.21.15.16:0/3606743018 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 6382+0+0 (secure 0 0 0) 0x7fd08811d360 con 0x7fd0980b1700 2023-05-22T15:40:13.179 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:13.170+0000 7fd09d09d640 1 --2- 172.21.15.16:0/3606743018 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd07c0654f0 0x7fd07c0679a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-05-22T15:40:13.180 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:13.174+0000 7fd08dffb640 1 -- 172.21.15.16:0/3606743018 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fd0880eac70 con 0x7fd0980b1700 2023-05-22T15:40:13.351 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:13.346+0000 7fd09f301640 1 -- 172.21.15.16:0/3606743018 --> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "snap_test_dir1", "snap_schedule": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7fd098158930 con 0x7fd07c0654f0 2023-05-22T15:40:13.378 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:13.374+0000 7fd09d09d640 1 -- 172.21.15.16:0/3606743018 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd07c0654f0 msgr2=0x7fd07c0679a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/36921 2023-05-22T15:40:13.378 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:13.374+0000 7fd09d09d640 1 --2- 172.21.15.16:0/3606743018 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd07c0654f0 0x7fd07c0679a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-05-22T15:40:13.498 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:40:13.497+0000 7f5831205180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-05-22T15:40:13.499 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:40:13.494+0000 7feb5e26d180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-05-22T15:40:13.552 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:40:13.549+0000 7f5831205180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-05-22T15:40:13.552 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:40:13.546+0000 7feb5e26d180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-05-22T15:40:13.602 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:40:13.601+0000 7f5831205180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-05-22T15:40:13.603 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:40:13.598+0000 7feb5e26d180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-05-22T15:40:13.680 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:40:13.677+0000 7f5831205180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-05-22T15:40:13.681 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:40:13.674+0000 7feb5e26d180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-05-22T15:40:13.731 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:40:13.729+0000 7f5831205180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-05-22T15:40:13.732 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:40:13.726+0000 7feb5e26d180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-05-22T15:40:13.782 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:13.778+0000 7fd09d09d640 1 -- 172.21.15.16:0/3606743018 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd07c0654f0 msgr2=0x7fd07c0679a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/36921 2023-05-22T15:40:13.782 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:13.778+0000 7fd09d09d640 1 --2- 172.21.15.16:0/3606743018 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd07c0654f0 0x7fd07c0679a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-05-22T15:40:13.808 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:40:13.805+0000 7f5831205180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-05-22T15:40:13.808 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:40:13.802+0000 7feb5e26d180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-05-22T15:40:13.860 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:40:13.857+0000 7f5831205180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-05-22T15:40:13.862 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:40:13.854+0000 7feb5e26d180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-05-22T15:40:13.993 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:40:13.986+0000 7feb5e26d180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-05-22T15:40:13.993 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:40:13.989+0000 7f5831205180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-05-22T15:40:14.045 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:40:14.038+0000 7feb5e26d180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-05-22T15:40:14.046 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:40:14.045+0000 7f5831205180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-05-22T15:40:14.269 INFO:tasks.ceph.mgr.y.smithi016.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2023-05-22T15:40:14.269 INFO:tasks.ceph.mgr.y.smithi016.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2023-05-22T15:40:14.269 INFO:tasks.ceph.mgr.y.smithi016.stderr: from numpy import show_config as show_numpy_config 2023-05-22T15:40:14.270 INFO:tasks.ceph.mgr.x.smithi062.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2023-05-22T15:40:14.270 INFO:tasks.ceph.mgr.x.smithi062.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2023-05-22T15:40:14.270 INFO:tasks.ceph.mgr.x.smithi062.stderr: from numpy import show_config as show_numpy_config 2023-05-22T15:40:14.277 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:40:14.270+0000 7feb5e26d180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-05-22T15:40:14.278 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:40:14.277+0000 7f5831205180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-05-22T15:40:14.401 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:40:14.394+0000 7feb5e26d180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-05-22T15:40:14.402 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:40:14.401+0000 7f5831205180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-05-22T15:40:14.515 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:40:14.510+0000 7feb5e26d180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-05-22T15:40:14.516 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:40:14.513+0000 7f5831205180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-05-22T15:40:14.579 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:40:14.574+0000 7feb5e26d180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-05-22T15:40:14.580 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:40:14.577+0000 7f5831205180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-05-22T15:40:14.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:14.582+0000 7fd09d09d640 1 -- 172.21.15.16:0/3606743018 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd07c0654f0 msgr2=0x7fd07c0679a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/36921 2023-05-22T15:40:14.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:14.582+0000 7fd09d09d640 1 --2- 172.21.15.16:0/3606743018 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd07c0654f0 0x7fd07c0679a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-05-22T15:40:14.705 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:40:14.701+0000 7f5831205180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-05-22T15:40:14.705 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:40:14.698+0000 7feb5e26d180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-05-22T15:40:15.199 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:40:15.194+0000 7feb5e26d180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-05-22T15:40:15.200 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:40:15.197+0000 7f5831205180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-05-22T15:40:15.275 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:40:15.270+0000 7feb5e26d180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-05-22T15:40:15.276 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:40:15.273+0000 7f5831205180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-05-22T15:40:15.414 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:40:15.413+0000 7f5831205180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-05-22T15:40:15.415 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:40:15.410+0000 7feb5e26d180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-05-22T15:40:15.491 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:40:15.489+0000 7f5831205180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-05-22T15:40:15.491 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:40:15.486+0000 7feb5e26d180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-05-22T15:40:15.739 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:40:15.737+0000 7f5831205180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-05-22T15:40:15.740 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:40:15.734+0000 7feb5e26d180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-05-22T15:40:15.793 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:40:15.789+0000 7f5831205180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-05-22T15:40:15.794 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:40:15.790+0000 7feb5e26d180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-05-22T15:40:15.831 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:15.826+0000 7fd08dffb640 1 -- 172.21.15.16:0/3606743018 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mgrmap(e 18) v1 ==== 83958+0+0 (secure 0 0 0) 0x7fd0880eb040 con 0x7fd0980b1700 2023-05-22T15:40:15.831 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:15.826+0000 7fd08dffb640 1 -- 172.21.15.16:0/3606743018 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd07c0654f0 msgr2=0x7fd07c0679a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-05-22T15:40:15.831 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:15.826+0000 7fd08dffb640 1 --2- 172.21.15.16:0/3606743018 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd07c0654f0 0x7fd07c0679a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:15.970 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:40:15.966+0000 7feadf4e7640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T15:40:15.971 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:40:15.966+0000 7feadf4e7640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T15:40:15.971 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:40:15.966+0000 7feadf4e7640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T15:40:15.971 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:40:15.966+0000 7feadf4e7640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T15:40:15.971 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:40:15.966+0000 7feadf4e7640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T15:40:15.971 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:40:15.966+0000 7feadc4e1640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T15:40:15.971 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:40:15.966+0000 7feadc4e1640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T15:40:15.972 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:40:15.966+0000 7feadc4e1640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T15:40:15.972 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:40:15.966+0000 7feadc4e1640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T15:40:15.972 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:40:15.966+0000 7feadc4e1640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T15:40:16.841 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:16.834+0000 7fd08dffb640 1 -- 172.21.15.16:0/3606743018 <== mon.0 v2:172.21.15.16:3300/0 8 ==== mgrmap(e 19) v1 ==== 84085+0+0 (secure 0 0 0) 0x7fd0880dcb70 con 0x7fd0980b1700 2023-05-22T15:40:16.841 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:16.834+0000 7fd08dffb640 1 --2- 172.21.15.16:0/3606743018 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd07c0654f0 0x7fd07c067920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:40:16.841 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:16.834+0000 7fd08dffb640 1 -- 172.21.15.16:0/3606743018 --> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "snap_test_dir1", "snap_schedule": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7fd098158930 con 0x7fd07c0654f0 2023-05-22T15:40:16.850 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:16.842+0000 7fd09d09d640 1 --2- 172.21.15.16:0/3606743018 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd07c0654f0 0x7fd07c067920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:40:16.854 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:16.850+0000 7fd09d09d640 1 --2- 172.21.15.16:0/3606743018 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd07c0654f0 0x7fd07c067920 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fd094009340 tx=0x7fd094009060 comp rx=0 tx=0).ready entity=mgr.6663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:40:16.992 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:16.986+0000 7fd08dffb640 1 -- 172.21.15.16:0/3606743018 <== mgr.6663 v2:172.21.15.16:6832/36921 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fd098158930 con 0x7fd07c0654f0 2023-05-22T15:40:16.992 INFO:teuthology.orchestra.run.smithi016.stdout:Schedule set for path snap_test_dir1 2023-05-22T15:40:16.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:16.990+0000 7fd09f301640 1 -- 172.21.15.16:0/3606743018 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd07c0654f0 msgr2=0x7fd07c067920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:40:16.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:16.990+0000 7fd09f301640 1 --2- 172.21.15.16:0/3606743018 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd07c0654f0 0x7fd07c067920 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fd094009340 tx=0x7fd094009060 comp rx=0 tx=0).stop 2023-05-22T15:40:16.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:16.990+0000 7fd09f301640 1 -- 172.21.15.16:0/3606743018 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd0980b1700 msgr2=0x7fd098153e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:40:16.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:16.990+0000 7fd09f301640 1 --2- 172.21.15.16:0/3606743018 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd0980b1700 0x7fd098153e20 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7fd0880af040 tx=0x7fd088019260 comp rx=0 tx=0).stop 2023-05-22T15:40:16.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:16.990+0000 7fd09f301640 1 -- 172.21.15.16:0/3606743018 shutdown_connections 2023-05-22T15:40:16.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:16.990+0000 7fd09f301640 1 --2- 172.21.15.16:0/3606743018 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd07c0654f0 0x7fd07c067920 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:16.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:16.990+0000 7fd09f301640 1 --2- 172.21.15.16:0/3606743018 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd098159ac0 0x7fd0980c63c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:16.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:16.990+0000 7fd09f301640 1 --2- 172.21.15.16:0/3606743018 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd0980b1700 0x7fd098153e20 unknown :-1 s=CLOSED pgs=956 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:16.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:16.990+0000 7fd09f301640 1 --2- 172.21.15.16:0/3606743018 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd098070430 0x7fd0980c4e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:16.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:16.990+0000 7fd09f301640 1 -- 172.21.15.16:0/3606743018 >> 172.21.15.16:0/3606743018 conn(0x7fd0980ce8d0 msgr2=0x7fd098158e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:40:16.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:16.990+0000 7fd09f301640 1 -- 172.21.15.16:0/3606743018 shutdown_connections 2023-05-22T15:40:16.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:16.990+0000 7fd09f301640 1 -- 172.21.15.16:0/3606743018 wait complete. 2023-05-22T15:40:17.016 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1 2023-05-22T15:40:17.016 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add --fs cephfs snap_test_dir1 2M 2023-05-22T15:40:17.149 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:17.142+0000 7f0a8281a640 1 Processor -- start 2023-05-22T15:40:17.150 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:17.142+0000 7f0a8281a640 1 -- start start 2023-05-22T15:40:17.150 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:17.146+0000 7f0a8281a640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0a7c15c8e0 0x7f0a7c0ccc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:40:17.150 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:17.146+0000 7f0a8281a640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f0a7c0cef10 0x7f0a7c0cf370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:40:17.151 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:17.146+0000 7f0a8281a640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a7c0d2a20 con 0x7f0a7c0d2eb0 2023-05-22T15:40:17.151 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:17.146+0000 7f0a8281a640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a7c0cd230 con 0x7f0a7c0cef10 2023-05-22T15:40:17.151 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:17.146+0000 7f0a8281a640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a7c0d23d0 con 0x7f0a7c15c8e0 2023-05-22T15:40:17.152 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:17.146+0000 7f0a7bfff640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0a7c15c8e0 0x7f0a7c0ccc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:40:17.152 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:17.146+0000 7f0a80db7640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f0a7c0cef10 0x7f0a7c0cf370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:40:17.152 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:17.146+0000 7f0a7bfff640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0a7c15c8e0 0x7f0a7c0ccc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3301/0 says I am v2:172.21.15.16:53810/0 (socket says 172.21.15.16:53810) 2023-05-22T15:40:17.152 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:17.146+0000 7f0a7bfff640 1 -- 172.21.15.16:0/2674445025 learned_addr learned my addr 172.21.15.16:0/2674445025 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:40:17.153 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:17.146+0000 7f0a7bfff640 1 -- 172.21.15.16:0/2674445025 >> v2:172.21.15.62:3300/0 conn(0x7f0a7c0cef10 msgr2=0x7f0a7c0cf370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:40:17.153 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:17.146+0000 7f0a7bfff640 1 --2- 172.21.15.16:0/2674445025 >> v2:172.21.15.62:3300/0 conn(0x7f0a7c0cef10 0x7f0a7c0cf370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:17.153 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:17.146+0000 7f0a7bfff640 1 -- 172.21.15.16:0/2674445025 >> v1:172.21.15.62:6789/0 conn(0x7f0a7c0d2eb0 legacy=0x7f0a7c15ccb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:40:17.153 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:17.146+0000 7f0a7bfff640 1 -- 172.21.15.16:0/2674445025 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a7c0d26b0 con 0x7f0a7c15c8e0 2023-05-22T15:40:17.153 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:17.146+0000 7f0a80db7640 1 --2- 172.21.15.16:0/2674445025 >> v2:172.21.15.62:3300/0 conn(0x7f0a7c0cef10 0x7f0a7c0cf370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-05-22T15:40:17.155 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:17.150+0000 7f0a7bfff640 1 --2- 172.21.15.16:0/2674445025 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0a7c15c8e0 0x7f0a7c0ccc50 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f0a6c009e50 tx=0x7f0a6c01c940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=493dbb40c2ab73de server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:40:17.156 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:17.150+0000 7f0a7affd640 1 -- 172.21.15.16:0/2674445025 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a6c0a57e0 con 0x7f0a7c15c8e0 2023-05-22T15:40:17.156 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:17.150+0000 7f0a7affd640 1 -- 172.21.15.16:0/2674445025 <== mon.2 v2:172.21.15.62:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a6c0a5940 con 0x7f0a7c15c8e0 2023-05-22T15:40:17.156 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:17.150+0000 7f0a7affd640 1 -- 172.21.15.16:0/2674445025 <== mon.2 v2:172.21.15.62:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a6c0a5cd0 con 0x7f0a7c15c8e0 2023-05-22T15:40:17.157 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:17.150+0000 7f0a8281a640 1 -- 172.21.15.16:0/2674445025 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0a7c15c8e0 msgr2=0x7f0a7c0ccc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:40:17.157 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:17.150+0000 7f0a8281a640 1 --2- 172.21.15.16:0/2674445025 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0a7c15c8e0 0x7f0a7c0ccc50 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f0a6c009e50 tx=0x7f0a6c01c940 comp rx=0 tx=0).stop 2023-05-22T15:40:17.157 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:17.150+0000 7f0a8281a640 1 -- 172.21.15.16:0/2674445025 shutdown_connections 2023-05-22T15:40:17.157 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:17.150+0000 7f0a8281a640 1 --2- 172.21.15.16:0/2674445025 >> v2:172.21.15.62:3300/0 conn(0x7f0a7c0cef10 0x7f0a7c0cf370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:17.157 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:17.150+0000 7f0a8281a640 1 --2- 172.21.15.16:0/2674445025 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0a7c15c8e0 0x7f0a7c0ccc50 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:17.158 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:17.150+0000 7f0a8281a640 1 -- 172.21.15.16:0/2674445025 >> 172.21.15.16:0/2674445025 conn(0x7f0a7c070e50 msgr2=0x7f0a7c071250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:40:17.158 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:17.150+0000 7f0a8281a640 1 -- 172.21.15.16:0/2674445025 shutdown_connections 2023-05-22T15:40:17.158 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:17.150+0000 7f0a8281a640 1 -- 172.21.15.16:0/2674445025 wait complete. 2023-05-22T15:40:17.160 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:17.150+0000 7f0a8281a640 1 Processor -- start 2023-05-22T15:40:17.160 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:17.150+0000 7f0a8281a640 1 -- start start 2023-05-22T15:40:17.160 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:17.154+0000 7f0a8281a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0a7c0cef10 0x7f0a7c152460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:40:17.160 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:17.154+0000 7f0a8281a640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0a7c0d2eb0 0x7f0a7c1529a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:40:17.161 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:17.154+0000 7f0a8281a640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0a7c15c8e0 0x7f0a7c144cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:40:17.161 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:17.154+0000 7f0a8281a640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a7c156190 con 0x7f0a7c0cef10 2023-05-22T15:40:17.161 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:17.154+0000 7f0a8281a640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a7c155510 con 0x7f0a7c0d2eb0 2023-05-22T15:40:17.161 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:17.154+0000 7f0a8281a640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a7c1557f0 con 0x7f0a7c15c8e0 2023-05-22T15:40:17.162 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:17.154+0000 7f0a7bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0a7c0cef10 0x7f0a7c152460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:40:17.162 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:17.154+0000 7f0a7bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0a7c0cef10 0x7f0a7c152460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42492/0 (socket says 172.21.15.16:42492) 2023-05-22T15:40:17.162 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:17.154+0000 7f0a7bfff640 1 -- 172.21.15.16:0/3213430433 learned_addr learned my addr 172.21.15.16:0/3213430433 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:40:17.162 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:17.154+0000 7f0a7b7fe640 1 --2- 172.21.15.16:0/3213430433 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0a7c0d2eb0 0x7f0a7c1529a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:40:17.163 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:17.154+0000 7f0a80db7640 1 --2- 172.21.15.16:0/3213430433 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0a7c15c8e0 0x7f0a7c144cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:40:17.163 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:17.154+0000 7f0a7bfff640 1 -- 172.21.15.16:0/3213430433 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0a7c15c8e0 msgr2=0x7f0a7c144cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:40:17.163 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:17.154+0000 7f0a7bfff640 1 --2- 172.21.15.16:0/3213430433 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0a7c15c8e0 0x7f0a7c144cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:17.164 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:17.154+0000 7f0a7bfff640 1 -- 172.21.15.16:0/3213430433 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0a7c0d2eb0 msgr2=0x7f0a7c1529a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:40:17.164 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:17.154+0000 7f0a7bfff640 1 --2- 172.21.15.16:0/3213430433 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0a7c0d2eb0 0x7f0a7c1529a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:17.164 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:17.154+0000 7f0a7bfff640 1 -- 172.21.15.16:0/3213430433 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a74009420 con 0x7f0a7c0cef10 2023-05-22T15:40:17.165 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:17.154+0000 7f0a80db7640 1 --2- 172.21.15.16:0/3213430433 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0a7c15c8e0 0x7f0a7c144cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T15:40:17.165 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:17.154+0000 7f0a7b7fe640 1 --2- 172.21.15.16:0/3213430433 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0a7c0d2eb0 0x7f0a7c1529a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T15:40:17.165 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:17.154+0000 7f0a7bfff640 1 --2- 172.21.15.16:0/3213430433 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0a7c0cef10 0x7f0a7c152460 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7f0a6c01a570 tx=0x7f0a6c0196e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:40:17.165 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:17.154+0000 7f0a78ff9640 1 -- 172.21.15.16:0/3213430433 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a6c0b8670 con 0x7f0a7c0cef10 2023-05-22T15:40:17.166 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:17.154+0000 7f0a78ff9640 1 -- 172.21.15.16:0/3213430433 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a6c0b87d0 con 0x7f0a7c0cef10 2023-05-22T15:40:17.166 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:17.154+0000 7f0a8281a640 1 -- 172.21.15.16:0/3213430433 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a6c009b30 con 0x7f0a7c0cef10 2023-05-22T15:40:17.166 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:17.154+0000 7f0a78ff9640 1 -- 172.21.15.16:0/3213430433 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a6c0b8b80 con 0x7f0a7c0cef10 2023-05-22T15:40:17.167 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:17.154+0000 7f0a8281a640 1 -- 172.21.15.16:0/3213430433 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a7c0d3410 con 0x7f0a7c0cef10 2023-05-22T15:40:17.168 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:17.154+0000 7f0a8281a640 1 -- 172.21.15.16:0/3213430433 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a7c0d3280 con 0x7f0a7c0cef10 2023-05-22T15:40:17.168 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:17.154+0000 7f0a78ff9640 1 -- 172.21.15.16:0/3213430433 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) v1 ==== 84085+0+0 (secure 0 0 0) 0x7f0a6c0b8ce0 con 0x7f0a7c0cef10 2023-05-22T15:40:17.169 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:17.154+0000 7f0a78ff9640 1 --2- 172.21.15.16:0/3213430433 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f0a5c065220 0x7f0a5c0676d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:40:17.169 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:17.154+0000 7f0a78ff9640 1 -- 172.21.15.16:0/3213430433 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 6726+0+0 (secure 0 0 0) 0x7f0a6c11dc80 con 0x7f0a7c0cef10 2023-05-22T15:40:17.169 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:17.154+0000 7f0a7b7fe640 1 --2- 172.21.15.16:0/3213430433 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f0a5c065220 0x7f0a5c0676d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:40:17.169 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:17.158+0000 7f0a7b7fe640 1 --2- 172.21.15.16:0/3213430433 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f0a5c065220 0x7f0a5c0676d0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f0a7c0b2010 tx=0x7f0a64009040 comp rx=0 tx=0).ready entity=mgr.6663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:40:17.170 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:17.158+0000 7f0a78ff9640 1 -- 172.21.15.16:0/3213430433 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f0a6c0eb9a0 con 0x7f0a7c0cef10 2023-05-22T15:40:17.340 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:17.334+0000 7f0a8281a640 1 -- 172.21.15.16:0/3213430433 --> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "snap_test_dir1", "snap_schedule": "2M", "target": ["mon-mgr", ""]}) v1 -- 0x7f0a7c0d0bf0 con 0x7f0a5c065220 2023-05-22T15:40:17.451 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:17.446+0000 7f0a78ff9640 1 -- 172.21.15.16:0/3213430433 <== mgr.6663 v2:172.21.15.16:6832/36921 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f0a7c0d0bf0 con 0x7f0a5c065220 2023-05-22T15:40:17.452 INFO:teuthology.orchestra.run.smithi016.stdout:Schedule set for path snap_test_dir1 2023-05-22T15:40:17.454 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:17.450+0000 7f0a8281a640 1 -- 172.21.15.16:0/3213430433 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f0a5c065220 msgr2=0x7f0a5c0676d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:40:17.455 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:17.450+0000 7f0a8281a640 1 --2- 172.21.15.16:0/3213430433 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f0a5c065220 0x7f0a5c0676d0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f0a7c0b2010 tx=0x7f0a64009040 comp rx=0 tx=0).stop 2023-05-22T15:40:17.455 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:17.450+0000 7f0a8281a640 1 -- 172.21.15.16:0/3213430433 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0a7c0cef10 msgr2=0x7f0a7c152460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:40:17.455 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:17.450+0000 7f0a8281a640 1 --2- 172.21.15.16:0/3213430433 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0a7c0cef10 0x7f0a7c152460 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7f0a6c01a570 tx=0x7f0a6c0196e0 comp rx=0 tx=0).stop 2023-05-22T15:40:17.455 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:17.450+0000 7f0a8281a640 1 -- 172.21.15.16:0/3213430433 shutdown_connections 2023-05-22T15:40:17.455 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:17.450+0000 7f0a8281a640 1 --2- 172.21.15.16:0/3213430433 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f0a5c065220 0x7f0a5c0676d0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:17.455 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:17.450+0000 7f0a8281a640 1 --2- 172.21.15.16:0/3213430433 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0a7c15c8e0 0x7f0a7c144cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:17.455 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:17.450+0000 7f0a8281a640 1 --2- 172.21.15.16:0/3213430433 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0a7c0d2eb0 0x7f0a7c1529a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:17.456 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:17.450+0000 7f0a8281a640 1 --2- 172.21.15.16:0/3213430433 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0a7c0cef10 0x7f0a7c152460 unknown :-1 s=CLOSED pgs=972 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:17.456 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:17.450+0000 7f0a8281a640 1 -- 172.21.15.16:0/3213430433 >> 172.21.15.16:0/3213430433 conn(0x7f0a7c070e50 msgr2=0x7f0a7c14f160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:40:17.456 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:17.450+0000 7f0a8281a640 1 -- 172.21.15.16:0/3213430433 shutdown_connections 2023-05-22T15:40:17.456 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:17.450+0000 7f0a8281a640 1 -- 172.21.15.16:0/3213430433 wait complete. 2023-05-22T15:40:17.474 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1 2023-05-22T15:40:17.474 DEBUG:tasks.cephfs.test_snap_schedules:expecting snap snap_test_dir1/.snap/scheduled-2023-05-22-15_41_00 in ~43s... 2023-05-22T15:40:17.475 DEBUG:tasks.cephfs.test_snap_schedules:expecting snap snap_test_dir1/.snap/scheduled-2023-05-22-15_42_00 in ~103s... 2023-05-22T15:40:17.475 DEBUG:tasks.cephfs.test_snap_schedules:expected_schedule: ['1M', '2M'], expected_retention: [] 2023-05-22T15:40:17.475 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule list --fs cephfs --format json snap_test_dir1 2023-05-22T15:40:17.611 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:17.606+0000 7f0c7653f640 1 Processor -- start 2023-05-22T15:40:17.611 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:17.606+0000 7f0c7653f640 1 -- start start 2023-05-22T15:40:17.612 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:17.606+0000 7f0c7653f640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0c700b7c90 0x7f0c700d2440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:40:17.613 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:17.606+0000 7f0c7653f640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f0c700b46e0 0x7f0c700d0460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:40:17.613 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:17.606+0000 7f0c7653f640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c700cd2d0 con 0x7f0c700cf210 2023-05-22T15:40:17.613 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:17.606+0000 7f0c7653f640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c700b8060 con 0x7f0c700b46e0 2023-05-22T15:40:17.613 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:17.606+0000 7f0c7653f640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c700cf680 con 0x7f0c700b7c90 2023-05-22T15:40:17.613 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:17.606+0000 7f0c6f7fe640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f0c700cf210 0x7f0c700d2980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:59098/0 (socket says 172.21.15.16:59098) 2023-05-22T15:40:17.613 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:17.606+0000 7f0c6f7fe640 1 -- 172.21.15.16:0/2086926395 learned_addr learned my addr 172.21.15.16:0/2086926395 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:40:17.614 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:17.606+0000 7f0c6ffff640 1 --2- 172.21.15.16:0/2086926395 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0c700b7c90 0x7f0c700d2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:40:17.614 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:17.606+0000 7f0c74adc640 1 --2- 172.21.15.16:0/2086926395 >> v2:172.21.15.62:3300/0 conn(0x7f0c700b46e0 0x7f0c700d0460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:40:17.614 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:17.606+0000 7f0c6effd640 1 -- 172.21.15.16:0/2086926395 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3027543250 0 0) 0x7f0c700cd2d0 con 0x7f0c700cf210 2023-05-22T15:40:17.614 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:17.606+0000 7f0c6effd640 1 -- 172.21.15.16:0/2086926395 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c5c003530 con 0x7f0c700cf210 2023-05-22T15:40:17.615 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:17.606+0000 7f0c6effd640 1 -- 172.21.15.16:0/2086926395 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1013811012 0 0) 0x7f0c5c003530 con 0x7f0c700cf210 2023-05-22T15:40:17.615 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:17.606+0000 7f0c6effd640 1 -- 172.21.15.16:0/2086926395 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0c700b7c90 msgr2=0x7f0c700d2440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:40:17.616 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:17.606+0000 7f0c6effd640 1 --2- 172.21.15.16:0/2086926395 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0c700b7c90 0x7f0c700d2440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:17.616 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:17.606+0000 7f0c6effd640 1 -- 172.21.15.16:0/2086926395 >> v2:172.21.15.62:3300/0 conn(0x7f0c700b46e0 msgr2=0x7f0c700d0460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:40:17.616 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:17.606+0000 7f0c6effd640 1 --2- 172.21.15.16:0/2086926395 >> v2:172.21.15.62:3300/0 conn(0x7f0c700b46e0 0x7f0c700d0460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:17.616 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:17.606+0000 7f0c6effd640 1 -- 172.21.15.16:0/2086926395 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c700ccc80 con 0x7f0c700cf210 2023-05-22T15:40:17.616 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:17.606+0000 7f0c6effd640 1 -- 172.21.15.16:0/2086926395 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7f0c58001680 con 0x7f0c700cf210 2023-05-22T15:40:17.616 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:17.606+0000 7f0c6effd640 1 -- 172.21.15.16:0/2086926395 >> v1:172.21.15.16:6789/0 conn(0x7f0c700cf210 legacy=0x7f0c700d2980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:40:17.617 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:17.606+0000 7f0c6effd640 1 --2- 172.21.15.16:0/2086926395 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0c5c003980 0x7f0c5c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:40:17.617 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:17.606+0000 7f0c6effd640 1 --2- 172.21.15.16:0/2086926395 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0c700b7c90 0x7f0c5c0047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:40:17.617 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:17.606+0000 7f0c6effd640 1 --2- 172.21.15.16:0/2086926395 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0c5c004d10 0x7f0c5c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:40:17.617 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:17.606+0000 7f0c6effd640 1 -- 172.21.15.16:0/2086926395 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c700b8060 con 0x7f0c5c004d10 2023-05-22T15:40:17.617 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:17.606+0000 7f0c6effd640 1 -- 172.21.15.16:0/2086926395 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c700cf680 con 0x7f0c5c003980 2023-05-22T15:40:17.617 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:17.606+0000 7f0c6effd640 1 -- 172.21.15.16:0/2086926395 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c5c00b990 con 0x7f0c700b7c90 2023-05-22T15:40:17.617 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:17.606+0000 7f0c74adc640 1 --2- 172.21.15.16:0/2086926395 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0c5c004d10 0x7f0c5c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:40:17.617 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:17.606+0000 7f0c6f7fe640 1 --2- 172.21.15.16:0/2086926395 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0c700b7c90 0x7f0c5c0047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:40:17.618 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:17.606+0000 7f0c6ffff640 1 --2- 172.21.15.16:0/2086926395 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0c5c003980 0x7f0c5c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:40:17.618 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:17.606+0000 7f0c74adc640 1 -- 172.21.15.16:0/2086926395 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0c700b7c90 msgr2=0x7f0c5c0047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:40:17.618 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:17.606+0000 7f0c74adc640 1 --2- 172.21.15.16:0/2086926395 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0c700b7c90 0x7f0c5c0047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:17.618 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:17.606+0000 7f0c74adc640 1 -- 172.21.15.16:0/2086926395 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0c5c003980 msgr2=0x7f0c5c003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:40:17.618 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:17.606+0000 7f0c74adc640 1 --2- 172.21.15.16:0/2086926395 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0c5c003980 0x7f0c5c003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:17.618 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:17.606+0000 7f0c74adc640 1 -- 172.21.15.16:0/2086926395 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c5c003530 con 0x7f0c5c004d10 2023-05-22T15:40:17.619 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:17.606+0000 7f0c6f7fe640 1 --2- 172.21.15.16:0/2086926395 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0c700b7c90 0x7f0c5c0047d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:40:17.619 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:17.606+0000 7f0c6ffff640 1 --2- 172.21.15.16:0/2086926395 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0c5c003980 0x7f0c5c003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:40:17.619 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:17.610+0000 7f0c74adc640 1 --2- 172.21.15.16:0/2086926395 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0c5c004d10 0x7f0c5c005180 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7f0c68009fc0 tx=0x7f0c6801cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9002d23512644195 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:40:17.621 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:17.610+0000 7f0c6effd640 1 -- 172.21.15.16:0/2086926395 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c680a5cb0 con 0x7f0c5c004d10 2023-05-22T15:40:17.621 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:17.610+0000 7f0c6effd640 1 -- 172.21.15.16:0/2086926395 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c6800f040 con 0x7f0c5c004d10 2023-05-22T15:40:17.621 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:17.610+0000 7f0c6effd640 1 -- 172.21.15.16:0/2086926395 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c680ad680 con 0x7f0c5c004d10 2023-05-22T15:40:17.621 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:17.610+0000 7f0c7653f640 1 -- 172.21.15.16:0/2086926395 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0c5c004d10 msgr2=0x7f0c5c005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:40:17.622 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:17.610+0000 7f0c7653f640 1 --2- 172.21.15.16:0/2086926395 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0c5c004d10 0x7f0c5c005180 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7f0c68009fc0 tx=0x7f0c6801cc40 comp rx=0 tx=0).stop 2023-05-22T15:40:17.622 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:17.610+0000 7f0c74adc640 1 -- 172.21.15.16:0/2086926395 reap_dead start 2023-05-22T15:40:17.622 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:17.610+0000 7f0c7653f640 1 -- 172.21.15.16:0/2086926395 shutdown_connections 2023-05-22T15:40:17.622 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:17.610+0000 7f0c7653f640 1 -- 172.21.15.16:0/2086926395 >> 172.21.15.16:0/2086926395 conn(0x7f0c700655c0 msgr2=0x7f0c70070e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:40:17.622 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:17.610+0000 7f0c7653f640 1 -- 172.21.15.16:0/2086926395 shutdown_connections 2023-05-22T15:40:17.622 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:17.610+0000 7f0c7653f640 1 -- 172.21.15.16:0/2086926395 wait complete. 2023-05-22T15:40:17.622 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:17.614+0000 7f0c7653f640 1 Processor -- start 2023-05-22T15:40:17.623 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:17.614+0000 7f0c7653f640 1 -- start start 2023-05-22T15:40:17.623 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:17.614+0000 7f0c7653f640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0c5c004d10 0x7f0c700a8e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:40:17.623 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:17.614+0000 7f0c7653f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0c7015ce70 0x7f0c701480c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:40:17.623 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:17.614+0000 7f0c7653f640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0c70148600 0x7f0c701f4440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:40:17.623 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:17.614+0000 7f0c7653f640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c70148ba0 con 0x7f0c7015ce70 2023-05-22T15:40:17.623 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:17.614+0000 7f0c7653f640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c7015d3e0 con 0x7f0c5c004d10 2023-05-22T15:40:17.623 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:17.614+0000 7f0c7653f640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c7015d6c0 con 0x7f0c70148600 2023-05-22T15:40:17.623 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:17.614+0000 7f0c6f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0c7015ce70 0x7f0c701480c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:40:17.624 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:17.614+0000 7f0c6f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0c7015ce70 0x7f0c701480c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42504/0 (socket says 172.21.15.16:42504) 2023-05-22T15:40:17.624 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:17.614+0000 7f0c6f7fe640 1 -- 172.21.15.16:0/3821738432 learned_addr learned my addr 172.21.15.16:0/3821738432 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:40:17.624 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:17.614+0000 7f0c6ffff640 1 --2- 172.21.15.16:0/3821738432 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0c5c004d10 0x7f0c700a8e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:40:17.624 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:17.614+0000 7f0c74adc640 1 --2- 172.21.15.16:0/3821738432 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0c70148600 0x7f0c701f4440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:40:17.624 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:17.614+0000 7f0c6f7fe640 1 -- 172.21.15.16:0/3821738432 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0c70148600 msgr2=0x7f0c701f4440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:40:17.624 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:17.614+0000 7f0c6f7fe640 1 --2- 172.21.15.16:0/3821738432 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0c70148600 0x7f0c701f4440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:17.624 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:17.614+0000 7f0c6f7fe640 1 -- 172.21.15.16:0/3821738432 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0c5c004d10 msgr2=0x7f0c700a8e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:40:17.625 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:17.614+0000 7f0c6f7fe640 1 --2- 172.21.15.16:0/3821738432 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0c5c004d10 0x7f0c700a8e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:17.625 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:17.614+0000 7f0c6f7fe640 1 -- 172.21.15.16:0/3821738432 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c700d3090 con 0x7f0c7015ce70 2023-05-22T15:40:17.625 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:17.614+0000 7f0c6f7fe640 1 --2- 172.21.15.16:0/3821738432 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0c7015ce70 0x7f0c701480c0 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7f0c5800d8d0 tx=0x7f0c5800dd90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:40:17.625 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:17.614+0000 7f0c6cff9640 1 -- 172.21.15.16:0/3821738432 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c580074e0 con 0x7f0c7015ce70 2023-05-22T15:40:17.625 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:17.614+0000 7f0c6cff9640 1 -- 172.21.15.16:0/3821738432 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c58007640 con 0x7f0c7015ce70 2023-05-22T15:40:17.625 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:17.614+0000 7f0c7653f640 1 -- 172.21.15.16:0/3821738432 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c700ccf60 con 0x7f0c7015ce70 2023-05-22T15:40:17.625 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:17.614+0000 7f0c6cff9640 1 -- 172.21.15.16:0/3821738432 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c58004a10 con 0x7f0c7015ce70 2023-05-22T15:40:17.625 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:17.614+0000 7f0c7653f640 1 -- 172.21.15.16:0/3821738432 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c700d3220 con 0x7f0c7015ce70 2023-05-22T15:40:17.626 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:17.614+0000 7f0c7653f640 1 -- 172.21.15.16:0/3821738432 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c7004dbe0 con 0x7f0c7015ce70 2023-05-22T15:40:17.626 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:17.614+0000 7f0c6cff9640 1 -- 172.21.15.16:0/3821738432 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) v1 ==== 84085+0+0 (secure 0 0 0) 0x7f0c58004b70 con 0x7f0c7015ce70 2023-05-22T15:40:17.626 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:17.618+0000 7f0c6cff9640 1 --2- 172.21.15.16:0/3821738432 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f0c50065250 0x7f0c50067700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:40:17.627 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:17.618+0000 7f0c6cff9640 1 -- 172.21.15.16:0/3821738432 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 6726+0+0 (secure 0 0 0) 0x7f0c58082920 con 0x7f0c7015ce70 2023-05-22T15:40:17.627 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:17.618+0000 7f0c6ffff640 1 --2- 172.21.15.16:0/3821738432 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f0c50065250 0x7f0c50067700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:40:17.627 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:17.618+0000 7f0c6ffff640 1 --2- 172.21.15.16:0/3821738432 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f0c50065250 0x7f0c50067700 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f0c70160740 tx=0x7f0c60008040 comp rx=0 tx=0).ready entity=mgr.6663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:40:17.627 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:17.618+0000 7f0c6cff9640 1 -- 172.21.15.16:0/3821738432 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f0c58050700 con 0x7f0c7015ce70 2023-05-22T15:40:17.797 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:17.790+0000 7f0c7653f640 1 -- 172.21.15.16:0/3821738432 --> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "snap_test_dir1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f0c700b16d0 con 0x7f0c50065250 2023-05-22T15:40:17.810 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:17.806+0000 7f0c6cff9640 1 -- 172.21.15.16:0/3821738432 <== mgr.6663 v2:172.21.15.16:6832/36921 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+76 (secure 0 0 0) 0x7f0c700b16d0 con 0x7f0c50065250 2023-05-22T15:40:17.810 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:40:17.810 INFO:teuthology.orchestra.run.smithi016.stdout:{"path": "snap_test_dir1", "schedule": ["1M", "2M"], "retention": [{}, {}]} 2023-05-22T15:40:17.813 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:17.806+0000 7f0c7653f640 1 -- 172.21.15.16:0/3821738432 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f0c50065250 msgr2=0x7f0c50067700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:40:17.813 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:17.806+0000 7f0c7653f640 1 --2- 172.21.15.16:0/3821738432 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f0c50065250 0x7f0c50067700 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f0c70160740 tx=0x7f0c60008040 comp rx=0 tx=0).stop 2023-05-22T15:40:17.813 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:17.806+0000 7f0c7653f640 1 -- 172.21.15.16:0/3821738432 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0c7015ce70 msgr2=0x7f0c701480c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:40:17.813 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:17.806+0000 7f0c7653f640 1 --2- 172.21.15.16:0/3821738432 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0c7015ce70 0x7f0c701480c0 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7f0c5800d8d0 tx=0x7f0c5800dd90 comp rx=0 tx=0).stop 2023-05-22T15:40:17.813 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:17.806+0000 7f0c7653f640 1 -- 172.21.15.16:0/3821738432 shutdown_connections 2023-05-22T15:40:17.813 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:17.806+0000 7f0c7653f640 1 --2- 172.21.15.16:0/3821738432 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f0c50065250 0x7f0c50067700 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:17.813 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:17.806+0000 7f0c7653f640 1 --2- 172.21.15.16:0/3821738432 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0c70148600 0x7f0c701f4440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:17.814 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:17.806+0000 7f0c7653f640 1 --2- 172.21.15.16:0/3821738432 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0c7015ce70 0x7f0c701480c0 unknown :-1 s=CLOSED pgs=975 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:17.814 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:17.806+0000 7f0c7653f640 1 --2- 172.21.15.16:0/3821738432 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0c5c004d10 0x7f0c700a8e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:40:17.814 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:17.806+0000 7f0c7653f640 1 -- 172.21.15.16:0/3821738432 >> 172.21.15.16:0/3821738432 conn(0x7f0c700655c0 msgr2=0x7f0c700b72f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:40:17.814 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:17.806+0000 7f0c7653f640 1 -- 172.21.15.16:0/3821738432 shutdown_connections 2023-05-22T15:40:17.814 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:40:17.806+0000 7f0c7653f640 1 -- 172.21.15.16:0/3821738432 wait complete. 2023-05-22T15:40:17.830 DEBUG:tasks.cephfs.test_snap_schedules:res= {"path": "snap_test_dir1", "schedule": ["1M", "2M"], "retention": [{}, {}]} 2023-05-22T15:40:17.830 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'path': 'snap_test_dir1', 'schedule': ['1M', '2M'], 'retention': [{}, {}]} 2023-05-22T15:40:17.831 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:40:17.831 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T15:40:17.842 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-05-22T15:40:17.843 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T15:40:17.843 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T15:40:18.844 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:40:18.845 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T15:40:18.863 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-05-22T15:40:18.863 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T15:40:18.864 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T15:40:19.865 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:40:19.865 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T15:40:19.880 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-05-22T15:40:19.880 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T15:40:19.880 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T15:40:20.881 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:40:20.882 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T15:40:20.896 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-05-22T15:40:20.896 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T15:40:20.897 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T15:40:21.830 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-05-22T15:40:21.833 DEBUG:teuthology.orchestra.run.smithi062:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-05-22T15:40:21.898 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:40:21.898 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T15:40:21.930 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-05-22T15:40:21.931 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T15:40:21.931 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T15:40:22.932 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:40:22.933 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T15:40:22.949 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-05-22T15:40:22.949 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T15:40:22.950 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T15:40:23.950 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:40:23.951 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T15:40:23.967 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-05-22T15:40:23.967 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T15:40:23.967 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T15:40:24.968 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:40:24.969 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T15:40:24.985 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-05-22T15:40:24.986 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T15:40:24.986 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T15:40:25.988 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:40:25.988 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T15:40:26.004 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-05-22T15:40:26.005 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T15:40:26.005 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T15:40:27.006 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:40:27.006 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T15:40:27.022 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-05-22T15:40:27.022 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T15:40:27.022 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T15:40:28.023 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:40:28.024 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T15:40:28.039 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-05-22T15:40:28.039 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T15:40:28.040 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T15:40:29.041 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:40:29.041 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T15:40:29.056 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-05-22T15:40:29.056 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T15:40:29.057 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T15:40:30.057 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:40:30.058 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T15:40:30.073 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-05-22T15:40:30.074 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T15:40:30.074 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T15:40:31.075 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:40:31.076 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T15:40:31.089 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-05-22T15:40:31.090 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T15:40:31.090 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T15:40:32.091 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:40:32.092 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T15:40:32.109 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-05-22T15:40:32.109 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T15:40:32.110 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T15:40:33.111 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:40:33.111 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T15:40:33.127 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-05-22T15:40:33.128 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T15:40:33.129 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T15:40:34.130 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:40:34.131 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T15:40:34.148 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-05-22T15:40:34.148 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T15:40:34.148 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T15:40:35.149 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:40:35.150 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T15:40:35.165 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-05-22T15:40:35.165 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T15:40:35.165 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T15:40:36.167 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:40:36.167 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T15:40:36.183 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-05-22T15:40:36.183 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T15:40:36.184 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T15:40:37.185 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:40:37.186 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T15:40:37.203 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-05-22T15:40:37.203 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T15:40:37.204 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T15:40:38.205 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:40:38.206 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T15:40:38.221 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-05-22T15:40:38.221 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T15:40:38.221 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T15:40:39.222 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:40:39.222 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T15:40:39.237 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-05-22T15:40:39.238 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T15:40:39.238 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T15:40:40.239 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:40:40.240 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T15:40:40.255 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-05-22T15:40:40.255 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T15:40:40.256 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T15:40:41.256 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:40:41.257 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T15:40:41.274 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-05-22T15:40:41.274 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T15:40:41.274 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T15:40:42.275 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:40:42.276 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T15:40:42.291 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-05-22T15:40:42.291 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T15:40:42.291 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T15:40:43.292 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:40:43.293 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T15:40:43.309 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-05-22T15:40:43.310 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T15:40:43.310 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T15:40:44.311 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:40:44.312 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T15:40:44.328 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-05-22T15:40:44.329 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T15:40:44.329 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T15:40:45.330 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:40:45.330 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T15:40:45.346 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-05-22T15:40:45.347 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T15:40:45.347 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T15:40:46.348 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:40:46.348 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T15:40:46.364 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-05-22T15:40:46.364 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T15:40:46.365 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T15:40:47.365 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:40:47.366 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T15:40:47.381 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-05-22T15:40:47.381 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T15:40:47.382 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T15:40:48.383 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:40:48.384 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T15:40:48.400 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-05-22T15:40:48.400 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T15:40:48.400 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T15:40:49.402 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:40:49.402 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T15:40:49.418 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-05-22T15:40:49.418 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T15:40:49.419 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T15:40:50.420 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:40:50.420 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T15:40:50.435 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-05-22T15:40:50.435 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T15:40:50.436 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T15:40:51.437 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:40:51.437 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T15:40:51.451 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-05-22T15:40:51.452 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T15:40:51.452 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T15:40:51.878 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-05-22T15:40:51.880 DEBUG:teuthology.orchestra.run.smithi062:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-05-22T15:40:52.454 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:40:52.454 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T15:40:52.470 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-05-22T15:40:52.471 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T15:40:52.471 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T15:40:53.472 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:40:53.473 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T15:40:53.487 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-05-22T15:40:53.488 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T15:40:53.488 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T15:40:54.489 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:40:54.489 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T15:40:54.504 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-05-22T15:40:54.504 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T15:40:54.505 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T15:40:55.506 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:40:55.506 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T15:40:55.521 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-05-22T15:40:55.522 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T15:40:55.522 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T15:40:56.523 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:40:56.523 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T15:40:56.538 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-05-22T15:40:56.538 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T15:40:56.539 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T15:40:57.539 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:40:57.540 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T15:40:57.555 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-05-22T15:40:57.555 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T15:40:57.556 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T15:40:58.557 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:40:58.557 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T15:40:58.572 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-05-22T15:40:58.572 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T15:40:58.572 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T15:40:59.574 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:40:59.574 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T15:40:59.589 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-05-22T15:40:59.589 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T15:40:59.589 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T15:41:00.470 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:41:00.469+0000 7feacf4c7640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T15:41:00.471 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:41:00.469+0000 7feacf4c7640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T15:41:00.471 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:41:00.469+0000 7feacf4c7640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T15:41:00.471 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:41:00.469+0000 7feacf4c7640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T15:41:00.471 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:41:00.469+0000 7feacf4c7640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T15:41:00.590 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:41:00.591 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T15:41:00.605 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_41_00_UTC 2023-05-22T15:41:00.607 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-05-22-15_41_00_UTC'} 2023-05-22T15:41:00.608 INFO:tasks.cephfs.test_snap_schedules:added: {'scheduled-2023-05-22-15_41_00_UTC'} 2023-05-22T15:41:00.608 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T15:41:00.608 DEBUG:tasks.cephfs.test_snap_schedules:snapshots added=['scheduled-2023-05-22-15_41_00_UTC'] 2023-05-22T15:41:00.609 DEBUG:tasks.cephfs.test_snap_schedules:exec=1684770017.474784, now = 1684770060.60933, timo = 43 2023-05-22T15:41:01.610 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:41:01.611 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T15:41:01.625 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_41_00_UTC 2023-05-22T15:41:01.626 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-05-22-15_41_00_UTC'} 2023-05-22T15:41:01.627 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T15:41:01.627 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T15:41:02.627 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:41:02.628 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T15:41:02.642 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_41_00_UTC 2023-05-22T15:41:02.643 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-05-22-15_41_00_UTC'} 2023-05-22T15:41:02.644 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T15:41:02.644 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T15:41:03.645 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:41:03.646 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T15:41:03.662 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_41_00_UTC 2023-05-22T15:41:03.663 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-05-22-15_41_00_UTC'} 2023-05-22T15:41:03.663 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T15:41:03.663 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T15:41:04.664 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:41:04.665 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T15:41:04.679 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_41_00_UTC 2023-05-22T15:41:04.680 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-05-22-15_41_00_UTC'} 2023-05-22T15:41:04.680 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T15:41:04.681 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T15:41:05.682 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:41:05.683 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T15:41:05.700 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_41_00_UTC 2023-05-22T15:41:05.701 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-05-22-15_41_00_UTC'} 2023-05-22T15:41:05.701 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T15:41:05.701 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T15:41:06.702 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:41:06.703 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T15:41:06.718 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_41_00_UTC 2023-05-22T15:41:06.719 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-05-22-15_41_00_UTC'} 2023-05-22T15:41:06.719 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T15:41:06.720 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T15:41:07.720 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:41:07.721 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T15:41:07.739 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_41_00_UTC 2023-05-22T15:41:07.740 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-05-22-15_41_00_UTC'} 2023-05-22T15:41:07.740 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T15:41:07.740 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T15:41:08.741 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:41:08.742 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T15:41:08.758 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_41_00_UTC 2023-05-22T15:41:08.759 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-05-22-15_41_00_UTC'} 2023-05-22T15:41:08.759 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T15:41:08.759 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T15:41:09.760 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:41:09.761 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T15:41:09.775 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_41_00_UTC 2023-05-22T15:41:09.776 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-05-22-15_41_00_UTC'} 2023-05-22T15:41:09.777 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T15:41:09.777 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T15:41:10.778 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:41:10.779 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T15:41:10.793 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_41_00_UTC 2023-05-22T15:41:10.794 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-05-22-15_41_00_UTC'} 2023-05-22T15:41:10.795 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T15:41:10.795 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T15:41:11.796 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:41:11.797 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T15:41:11.813 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_41_00_UTC 2023-05-22T15:41:11.814 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-05-22-15_41_00_UTC'} 2023-05-22T15:41:11.814 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T15:41:11.814 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T15:41:12.815 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:41:12.816 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T15:41:12.830 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_41_00_UTC 2023-05-22T15:41:12.831 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-05-22-15_41_00_UTC'} 2023-05-22T15:41:12.831 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T15:41:12.832 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T15:41:13.832 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:41:13.833 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T15:41:13.848 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_41_00_UTC 2023-05-22T15:41:13.850 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-05-22-15_41_00_UTC'} 2023-05-22T15:41:13.851 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T15:41:13.851 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T15:41:14.852 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:41:14.852 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T15:41:14.868 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_41_00_UTC 2023-05-22T15:41:14.869 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-05-22-15_41_00_UTC'} 2023-05-22T15:41:14.869 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T15:41:14.869 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T15:41:15.871 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:41:15.871 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T15:41:15.884 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_41_00_UTC 2023-05-22T15:41:15.885 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-05-22-15_41_00_UTC'} 2023-05-22T15:41:15.885 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T15:41:15.885 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T15:41:16.886 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:41:16.887 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T15:41:16.903 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_41_00_UTC 2023-05-22T15:41:16.904 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-05-22-15_41_00_UTC'} 2023-05-22T15:41:16.904 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T15:41:16.904 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T15:41:17.905 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:41:17.906 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T15:41:17.922 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_41_00_UTC 2023-05-22T15:41:17.923 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-05-22-15_41_00_UTC'} 2023-05-22T15:41:17.923 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T15:41:17.923 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T15:41:18.924 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:41:18.925 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T15:41:18.941 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_41_00_UTC 2023-05-22T15:41:18.942 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-05-22-15_41_00_UTC'} 2023-05-22T15:41:18.942 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T15:41:18.943 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T15:41:19.944 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:41:19.944 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T15:41:19.958 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_41_00_UTC 2023-05-22T15:41:19.959 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-05-22-15_41_00_UTC'} 2023-05-22T15:41:19.960 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T15:41:19.960 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T15:41:20.961 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:41:20.961 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T15:41:20.977 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_41_00_UTC 2023-05-22T15:41:20.978 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-05-22-15_41_00_UTC'} 2023-05-22T15:41:20.979 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T15:41:20.979 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T15:41:21.951 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-05-22T15:41:21.954 DEBUG:teuthology.orchestra.run.smithi062:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-05-22T15:41:21.980 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:41:21.981 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T15:41:22.050 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_41_00_UTC 2023-05-22T15:41:22.051 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-05-22-15_41_00_UTC'} 2023-05-22T15:41:22.051 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T15:41:22.052 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T15:41:23.053 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:41:23.053 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T15:41:23.068 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_41_00_UTC 2023-05-22T15:41:23.069 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-05-22-15_41_00_UTC'} 2023-05-22T15:41:23.070 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T15:41:23.070 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T15:41:24.071 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:41:24.072 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T15:41:24.088 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_41_00_UTC 2023-05-22T15:41:24.089 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-05-22-15_41_00_UTC'} 2023-05-22T15:41:24.089 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T15:41:24.089 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T15:41:25.091 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:41:25.091 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T15:41:25.107 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_41_00_UTC 2023-05-22T15:41:25.108 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-05-22-15_41_00_UTC'} 2023-05-22T15:41:25.108 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T15:41:25.108 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T15:41:26.110 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:41:26.110 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T15:41:26.126 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_41_00_UTC 2023-05-22T15:41:26.127 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-05-22-15_41_00_UTC'} 2023-05-22T15:41:26.127 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T15:41:26.127 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T15:41:27.129 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:41:27.130 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T15:41:27.145 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_41_00_UTC 2023-05-22T15:41:27.146 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-05-22-15_41_00_UTC'} 2023-05-22T15:41:27.146 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T15:41:27.146 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T15:41:28.147 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:41:28.148 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T15:41:28.161 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_41_00_UTC 2023-05-22T15:41:28.162 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-05-22-15_41_00_UTC'} 2023-05-22T15:41:28.163 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T15:41:28.163 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T15:41:29.164 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:41:29.164 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T15:41:29.179 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_41_00_UTC 2023-05-22T15:41:29.180 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-05-22-15_41_00_UTC'} 2023-05-22T15:41:29.180 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T15:41:29.181 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T15:41:30.181 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:41:30.182 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T15:41:30.197 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_41_00_UTC 2023-05-22T15:41:30.198 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-05-22-15_41_00_UTC'} 2023-05-22T15:41:30.199 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T15:41:30.199 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T15:41:31.200 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:41:31.200 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T15:41:31.215 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_41_00_UTC 2023-05-22T15:41:31.216 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-05-22-15_41_00_UTC'} 2023-05-22T15:41:31.217 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T15:41:31.217 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T15:41:32.218 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:41:32.218 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T15:41:32.234 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_41_00_UTC 2023-05-22T15:41:32.236 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-05-22-15_41_00_UTC'} 2023-05-22T15:41:32.236 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T15:41:32.237 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T15:41:33.238 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:41:33.239 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T15:41:33.253 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_41_00_UTC 2023-05-22T15:41:33.254 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-05-22-15_41_00_UTC'} 2023-05-22T15:41:33.255 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T15:41:33.255 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T15:41:34.256 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:41:34.257 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T15:41:34.271 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_41_00_UTC 2023-05-22T15:41:34.272 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-05-22-15_41_00_UTC'} 2023-05-22T15:41:34.273 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T15:41:34.273 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T15:41:35.274 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:41:35.275 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T15:41:35.291 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_41_00_UTC 2023-05-22T15:41:35.292 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-05-22-15_41_00_UTC'} 2023-05-22T15:41:35.292 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T15:41:35.292 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T15:41:36.294 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:41:36.294 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T15:41:36.308 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_41_00_UTC 2023-05-22T15:41:36.309 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-05-22-15_41_00_UTC'} 2023-05-22T15:41:36.309 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T15:41:36.309 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T15:41:37.310 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:41:37.311 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T15:41:37.324 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_41_00_UTC 2023-05-22T15:41:37.325 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-05-22-15_41_00_UTC'} 2023-05-22T15:41:37.326 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T15:41:37.326 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T15:41:38.327 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:41:38.327 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T15:41:38.342 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_41_00_UTC 2023-05-22T15:41:38.343 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-05-22-15_41_00_UTC'} 2023-05-22T15:41:38.343 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T15:41:38.343 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T15:41:39.344 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:41:39.345 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T15:41:39.361 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_41_00_UTC 2023-05-22T15:41:39.362 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-05-22-15_41_00_UTC'} 2023-05-22T15:41:39.362 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T15:41:39.363 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T15:41:40.364 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:41:40.364 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T15:41:40.378 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_41_00_UTC 2023-05-22T15:41:40.378 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-05-22-15_41_00_UTC'} 2023-05-22T15:41:40.379 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T15:41:40.379 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T15:41:41.380 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:41:41.381 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T15:41:41.395 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_41_00_UTC 2023-05-22T15:41:41.396 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-05-22-15_41_00_UTC'} 2023-05-22T15:41:41.397 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T15:41:41.397 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T15:41:42.398 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:41:42.399 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T15:41:42.415 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_41_00_UTC 2023-05-22T15:41:42.416 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-05-22-15_41_00_UTC'} 2023-05-22T15:41:42.417 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T15:41:42.417 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T15:41:43.419 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:41:43.420 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T15:41:43.434 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_41_00_UTC 2023-05-22T15:41:43.435 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-05-22-15_41_00_UTC'} 2023-05-22T15:41:43.436 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T15:41:43.437 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T15:41:44.438 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:41:44.439 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T15:41:44.452 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_41_00_UTC 2023-05-22T15:41:44.453 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-05-22-15_41_00_UTC'} 2023-05-22T15:41:44.454 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T15:41:44.454 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T15:41:45.454 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:41:45.455 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T15:41:45.470 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_41_00_UTC 2023-05-22T15:41:45.471 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-05-22-15_41_00_UTC'} 2023-05-22T15:41:45.471 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T15:41:45.472 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T15:41:46.472 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:41:46.473 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T15:41:46.487 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_41_00_UTC 2023-05-22T15:41:46.488 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-05-22-15_41_00_UTC'} 2023-05-22T15:41:46.489 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T15:41:46.489 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T15:41:47.490 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:41:47.491 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T15:41:47.506 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_41_00_UTC 2023-05-22T15:41:47.507 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-05-22-15_41_00_UTC'} 2023-05-22T15:41:47.507 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T15:41:47.507 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T15:41:48.509 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:41:48.510 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T15:41:48.525 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_41_00_UTC 2023-05-22T15:41:48.526 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-05-22-15_41_00_UTC'} 2023-05-22T15:41:48.527 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T15:41:48.528 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T15:41:49.529 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:41:49.530 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T15:41:49.545 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_41_00_UTC 2023-05-22T15:41:49.545 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-05-22-15_41_00_UTC'} 2023-05-22T15:41:49.545 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T15:41:49.546 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T15:41:50.546 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:41:50.547 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T15:41:50.561 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_41_00_UTC 2023-05-22T15:41:50.562 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-05-22-15_41_00_UTC'} 2023-05-22T15:41:50.563 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T15:41:50.563 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T15:41:51.564 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:41:51.564 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T15:41:51.578 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_41_00_UTC 2023-05-22T15:41:51.579 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-05-22-15_41_00_UTC'} 2023-05-22T15:41:51.579 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T15:41:51.580 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T15:41:51.996 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-05-22T15:41:51.999 DEBUG:teuthology.orchestra.run.smithi062:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-05-22T15:41:52.580 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:41:52.581 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T15:41:52.595 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_41_00_UTC 2023-05-22T15:41:52.596 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-05-22-15_41_00_UTC'} 2023-05-22T15:41:52.597 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T15:41:52.597 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T15:41:53.599 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:41:53.599 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T15:41:53.613 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_41_00_UTC 2023-05-22T15:41:53.614 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-05-22-15_41_00_UTC'} 2023-05-22T15:41:53.615 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T15:41:53.615 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T15:41:54.615 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:41:54.616 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T15:41:54.630 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_41_00_UTC 2023-05-22T15:41:54.631 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-05-22-15_41_00_UTC'} 2023-05-22T15:41:54.631 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T15:41:54.631 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T15:41:55.632 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:41:55.633 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T15:41:55.647 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_41_00_UTC 2023-05-22T15:41:55.648 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-05-22-15_41_00_UTC'} 2023-05-22T15:41:55.648 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T15:41:55.648 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T15:41:56.649 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:41:56.650 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T15:41:56.663 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_41_00_UTC 2023-05-22T15:41:56.664 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-05-22-15_41_00_UTC'} 2023-05-22T15:41:56.665 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T15:41:56.665 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T15:41:57.666 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:41:57.666 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T15:41:57.680 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_41_00_UTC 2023-05-22T15:41:57.681 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-05-22-15_41_00_UTC'} 2023-05-22T15:41:57.681 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T15:41:57.682 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T15:41:58.682 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:41:58.683 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T15:41:58.696 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_41_00_UTC 2023-05-22T15:41:58.697 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-05-22-15_41_00_UTC'} 2023-05-22T15:41:58.697 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T15:41:58.697 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T15:41:59.698 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:41:59.699 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T15:41:59.714 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_41_00_UTC 2023-05-22T15:41:59.714 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-05-22-15_41_00_UTC'} 2023-05-22T15:41:59.714 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T15:41:59.715 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T15:42:00.716 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:42:00.717 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T15:42:00.731 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_41_00_UTC 2023-05-22T15:42:00.731 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_42_00_UTC 2023-05-22T15:42:00.732 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-05-22-15_41_00_UTC', 'scheduled-2023-05-22-15_42_00_UTC'} 2023-05-22T15:42:00.732 INFO:tasks.cephfs.test_snap_schedules:added: {'scheduled-2023-05-22-15_42_00_UTC'} 2023-05-22T15:42:00.732 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T15:42:00.733 DEBUG:tasks.cephfs.test_snap_schedules:snapshots added=['scheduled-2023-05-22-15_42_00_UTC'] 2023-05-22T15:42:00.733 DEBUG:tasks.cephfs.test_snap_schedules:exec=1684770017.474784, now = 1684770120.7333882, timo = 103 2023-05-22T15:42:01.734 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove --fs cephfs snap_test_dir1 2023-05-22T15:42:01.875 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:01.872+0000 7fc43f2f0640 1 Processor -- start 2023-05-22T15:42:01.875 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:01.872+0000 7fc43f2f0640 1 -- start start 2023-05-22T15:42:01.876 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:01.872+0000 7fc43f2f0640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc438159a30 0x7fc43806e380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:01.876 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:01.872+0000 7fc43f2f0640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc4380cf570 0x7fc4380cf9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:01.876 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:01.872+0000 7fc43f2f0640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4380cde70 con 0x7fc43815d610 2023-05-22T15:42:01.876 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:01.872+0000 7fc43f2f0640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4380cc520 con 0x7fc4380cf570 2023-05-22T15:42:01.877 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:01.872+0000 7fc43f2f0640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4380cff30 con 0x7fc438159a30 2023-05-22T15:42:01.877 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:01.872+0000 7fc43d88d640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc4380cf570 0x7fc4380cf9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:01.877 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:01.872+0000 7fc43d88d640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc4380cf570 0x7fc4380cf9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50680/0 (socket says 172.21.15.16:50680) 2023-05-22T15:42:01.877 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:01.872+0000 7fc43d88d640 1 -- 172.21.15.16:0/2922005977 learned_addr learned my addr 172.21.15.16:0/2922005977 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:42:01.877 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:01.872+0000 7fc43d08c640 1 --2- 172.21.15.16:0/2922005977 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc438159a30 0x7fc43806e380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:01.878 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:01.872+0000 7fc43d88d640 1 -- 172.21.15.16:0/2922005977 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc438159a30 msgr2=0x7fc43806e380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:42:01.878 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:01.872+0000 7fc43d88d640 1 --2- 172.21.15.16:0/2922005977 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc438159a30 0x7fc43806e380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:01.878 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:01.872+0000 7fc43d88d640 1 -- 172.21.15.16:0/2922005977 >> v1:172.21.15.62:6789/0 conn(0x7fc43815d610 legacy=0x7fc4380cbe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:01.878 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:01.872+0000 7fc43d88d640 1 -- 172.21.15.16:0/2922005977 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc43815db20 con 0x7fc4380cf570 2023-05-22T15:42:01.878 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:01.872+0000 7fc43d08c640 1 --2- 172.21.15.16:0/2922005977 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc438159a30 0x7fc43806e380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T15:42:01.879 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:01.876+0000 7fc43d88d640 1 --2- 172.21.15.16:0/2922005977 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4380cf570 0x7fc4380cf9f0 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7fc430009870 tx=0x7fc43001c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc408f78341794bf server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:01.879 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:01.876+0000 7fc437fff640 1 -- 172.21.15.16:0/2922005977 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4300aa680 con 0x7fc4380cf570 2023-05-22T15:42:01.879 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:01.876+0000 7fc437fff640 1 -- 172.21.15.16:0/2922005977 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4300aa7e0 con 0x7fc4380cf570 2023-05-22T15:42:01.880 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:01.876+0000 7fc437fff640 1 -- 172.21.15.16:0/2922005977 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4300aab70 con 0x7fc4380cf570 2023-05-22T15:42:01.880 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:01.876+0000 7fc43f2f0640 1 -- 172.21.15.16:0/2922005977 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4380cf570 msgr2=0x7fc4380cf9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:01.881 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:01.876+0000 7fc43f2f0640 1 --2- 172.21.15.16:0/2922005977 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4380cf570 0x7fc4380cf9f0 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7fc430009870 tx=0x7fc43001c520 comp rx=0 tx=0).stop 2023-05-22T15:42:01.881 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:01.876+0000 7fc43f2f0640 1 -- 172.21.15.16:0/2922005977 shutdown_connections 2023-05-22T15:42:01.881 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:01.876+0000 7fc43f2f0640 1 --2- 172.21.15.16:0/2922005977 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4380cf570 0x7fc4380cf9f0 unknown :-1 s=CLOSED pgs=977 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:01.881 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:01.876+0000 7fc43f2f0640 1 --2- 172.21.15.16:0/2922005977 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc438159a30 0x7fc43806e380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:01.881 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:01.876+0000 7fc43f2f0640 1 -- 172.21.15.16:0/2922005977 >> 172.21.15.16:0/2922005977 conn(0x7fc4380700b0 msgr2=0x7fc43806d9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:01.881 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:01.876+0000 7fc43f2f0640 1 -- 172.21.15.16:0/2922005977 shutdown_connections 2023-05-22T15:42:01.881 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:01.876+0000 7fc43f2f0640 1 -- 172.21.15.16:0/2922005977 wait complete. 2023-05-22T15:42:01.883 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:01.876+0000 7fc43f2f0640 1 Processor -- start 2023-05-22T15:42:01.883 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:01.876+0000 7fc43f2f0640 1 -- start start 2023-05-22T15:42:01.883 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:01.880+0000 7fc43f2f0640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc4380cf570 0x7fc43814aa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:01.883 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:01.880+0000 7fc43f2f0640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc438159a30 0x7fc43814c870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:01.884 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:01.880+0000 7fc43f2f0640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc43815d610 0x7fc43815ba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:01.884 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:01.880+0000 7fc43f2f0640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4381389b0 con 0x7fc438159a30 2023-05-22T15:42:01.884 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:01.880+0000 7fc43f2f0640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc438150380 con 0x7fc43815d610 2023-05-22T15:42:01.884 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:01.880+0000 7fc43f2f0640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc438137d70 con 0x7fc4380cf570 2023-05-22T15:42:01.884 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:01.880+0000 7fc43c88b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc438159a30 0x7fc43814c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:01.884 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:01.880+0000 7fc43c88b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc438159a30 0x7fc43814c870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50692/0 (socket says 172.21.15.16:50692) 2023-05-22T15:42:01.884 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:01.880+0000 7fc43c88b640 1 -- 172.21.15.16:0/2567718732 learned_addr learned my addr 172.21.15.16:0/2567718732 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:42:01.885 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:01.880+0000 7fc43d88d640 1 --2- 172.21.15.16:0/2567718732 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc43815d610 0x7fc43815ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:01.885 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:01.880+0000 7fc43d08c640 1 --2- 172.21.15.16:0/2567718732 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc4380cf570 0x7fc43814aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:01.885 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:01.880+0000 7fc43c88b640 1 -- 172.21.15.16:0/2567718732 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc4380cf570 msgr2=0x7fc43814aa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:01.885 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:01.880+0000 7fc43c88b640 1 --2- 172.21.15.16:0/2567718732 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc4380cf570 0x7fc43814aa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:01.885 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:01.880+0000 7fc43c88b640 1 -- 172.21.15.16:0/2567718732 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc43815d610 msgr2=0x7fc43815ba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:01.885 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:01.880+0000 7fc43c88b640 1 --2- 172.21.15.16:0/2567718732 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc43815d610 0x7fc43815ba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:01.885 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:01.880+0000 7fc43c88b640 1 -- 172.21.15.16:0/2567718732 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4300094d0 con 0x7fc438159a30 2023-05-22T15:42:01.886 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:01.880+0000 7fc43c88b640 1 --2- 172.21.15.16:0/2567718732 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc438159a30 0x7fc43814c870 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7fc42c00c500 tx=0x7fc42c00c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:01.886 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:01.880+0000 7fc435ffb640 1 -- 172.21.15.16:0/2567718732 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc42c005220 con 0x7fc438159a30 2023-05-22T15:42:01.886 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:01.880+0000 7fc435ffb640 1 -- 172.21.15.16:0/2567718732 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc42c005380 con 0x7fc438159a30 2023-05-22T15:42:01.886 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:01.880+0000 7fc43f2f0640 1 -- 172.21.15.16:0/2567718732 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc43815e3a0 con 0x7fc438159a30 2023-05-22T15:42:01.886 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:01.880+0000 7fc435ffb640 1 -- 172.21.15.16:0/2567718732 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc42c007270 con 0x7fc438159a30 2023-05-22T15:42:01.886 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:01.880+0000 7fc43f2f0640 1 -- 172.21.15.16:0/2567718732 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc43815e0c0 con 0x7fc438159a30 2023-05-22T15:42:01.887 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:01.880+0000 7fc43f2f0640 1 -- 172.21.15.16:0/2567718732 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc438159e00 con 0x7fc438159a30 2023-05-22T15:42:01.887 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:01.880+0000 7fc435ffb640 1 -- 172.21.15.16:0/2567718732 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 22) v1 ==== 84361+0+0 (secure 0 0 0) 0x7fc42c0073d0 con 0x7fc438159a30 2023-05-22T15:42:01.887 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:01.880+0000 7fc435ffb640 1 --2- 172.21.15.16:0/2567718732 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fc4240654f0 0x7fc4240679a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:01.887 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:01.880+0000 7fc435ffb640 1 -- 172.21.15.16:0/2567718732 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 6726+0+0 (secure 0 0 0) 0x7fc42c082600 con 0x7fc438159a30 2023-05-22T15:42:01.888 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:01.880+0000 7fc43d08c640 1 --2- 172.21.15.16:0/2567718732 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fc4240654f0 0x7fc4240679a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:01.888 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:01.880+0000 7fc43d08c640 1 --2- 172.21.15.16:0/2567718732 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fc4240654f0 0x7fc4240679a0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fc43814fd20 tx=0x7fc428003a20 comp rx=0 tx=0).ready entity=mgr.6663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:01.889 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:01.884+0000 7fc435ffb640 1 -- 172.21.15.16:0/2567718732 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fc42c04fee0 con 0x7fc438159a30 2023-05-22T15:42:02.062 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:02.060+0000 7fc43f2f0640 1 -- 172.21.15.16:0/2567718732 --> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "fs": "cephfs", "path": "snap_test_dir1", "target": ["mon-mgr", ""]}) v1 -- 0x7fc438066910 con 0x7fc4240654f0 2023-05-22T15:42:02.210 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:02.204+0000 7fc435ffb640 1 -- 172.21.15.16:0/2567718732 <== mgr.6663 v2:172.21.15.16:6832/36921 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7fc438066910 con 0x7fc4240654f0 2023-05-22T15:42:02.210 INFO:teuthology.orchestra.run.smithi016.stdout:Schedule removed for path snap_test_dir1 2023-05-22T15:42:02.212 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:02.208+0000 7fc43f2f0640 1 -- 172.21.15.16:0/2567718732 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fc4240654f0 msgr2=0x7fc4240679a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:02.212 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:02.208+0000 7fc43f2f0640 1 --2- 172.21.15.16:0/2567718732 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fc4240654f0 0x7fc4240679a0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fc43814fd20 tx=0x7fc428003a20 comp rx=0 tx=0).stop 2023-05-22T15:42:02.213 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:02.208+0000 7fc43f2f0640 1 -- 172.21.15.16:0/2567718732 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc438159a30 msgr2=0x7fc43814c870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:02.213 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:02.208+0000 7fc43f2f0640 1 --2- 172.21.15.16:0/2567718732 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc438159a30 0x7fc43814c870 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7fc42c00c500 tx=0x7fc42c00c9c0 comp rx=0 tx=0).stop 2023-05-22T15:42:02.213 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:02.208+0000 7fc43f2f0640 1 -- 172.21.15.16:0/2567718732 shutdown_connections 2023-05-22T15:42:02.213 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:02.208+0000 7fc43f2f0640 1 --2- 172.21.15.16:0/2567718732 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc43815d610 0x7fc43815ba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:02.213 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:02.208+0000 7fc43f2f0640 1 --2- 172.21.15.16:0/2567718732 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc438159a30 0x7fc43814c870 unknown :-1 s=CLOSED pgs=978 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:02.213 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:02.208+0000 7fc43f2f0640 1 --2- 172.21.15.16:0/2567718732 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fc4240654f0 0x7fc4240679a0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:02.213 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:02.208+0000 7fc43f2f0640 1 --2- 172.21.15.16:0/2567718732 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc4380cf570 0x7fc43814aa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:02.214 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:02.208+0000 7fc43f2f0640 1 -- 172.21.15.16:0/2567718732 >> 172.21.15.16:0/2567718732 conn(0x7fc4380700b0 msgr2=0x7fc4380d00a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:02.214 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:02.208+0000 7fc43f2f0640 1 -- 172.21.15.16:0/2567718732 shutdown_connections 2023-05-22T15:42:02.214 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:02.208+0000 7fc43f2f0640 1 -- 172.21.15.16:0/2567718732 wait complete. 2023-05-22T15:42:02.234 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path snap_test_dir1 2023-05-22T15:42:02.234 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:42:02.235 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T15:42:02.245 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_41_00_UTC 2023-05-22T15:42:02.245 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_42_00_UTC 2023-05-22T15:42:02.245 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-05-22-15_41_00_UTC 2023-05-22T15:42:02.246 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:42:02.246 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-05-22-15_41_00_UTC) 2023-05-22T15:42:02.302 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-05-22-15_42_00_UTC 2023-05-22T15:42:02.303 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:42:02.303 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-05-22-15_42_00_UTC) 2023-05-22T15:42:02.363 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:42:02.363 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1) 2023-05-22T15:42:02.418 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2023-05-22T15:42:02.601 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:02.596+0000 7fb1e877b640 1 Processor -- start 2023-05-22T15:42:02.601 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:02.596+0000 7fb1e877b640 1 -- start start 2023-05-22T15:42:02.602 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:02.600+0000 7fb1e877b640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb1e00ce900 0x7fb1e00ced80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:02.602 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:02.600+0000 7fb1e877b640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb1e00cf2c0 0x7fb1e00d0890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:02.603 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:02.600+0000 7fb1e877b640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1e00d0400 con 0x7fb1e015bca0 2023-05-22T15:42:02.603 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:02.600+0000 7fb1e877b640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1e015c070 con 0x7fb1e00ce900 2023-05-22T15:42:02.603 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:02.600+0000 7fb1e877b640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1e00d00d0 con 0x7fb1e00cf2c0 2023-05-22T15:42:02.603 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:02.600+0000 7fb1e5d16640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb1e00ce900 0x7fb1e00ced80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:02.603 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:02.600+0000 7fb1e5d16640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb1e00ce900 0x7fb1e00ced80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50698/0 (socket says 172.21.15.16:50698) 2023-05-22T15:42:02.604 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:02.600+0000 7fb1e5d16640 1 -- 172.21.15.16:0/102168906 learned_addr learned my addr 172.21.15.16:0/102168906 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:42:02.604 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:02.600+0000 7fb1e6d18640 1 --2- 172.21.15.16:0/102168906 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb1e00cf2c0 0x7fb1e00d0890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:02.604 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:02.600+0000 7fb1e5d16640 1 -- 172.21.15.16:0/102168906 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb1e00cf2c0 msgr2=0x7fb1e00d0890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:42:02.604 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:02.600+0000 7fb1e5d16640 1 --2- 172.21.15.16:0/102168906 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb1e00cf2c0 0x7fb1e00d0890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:02.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:02.600+0000 7fb1e5d16640 1 -- 172.21.15.16:0/102168906 >> v1:172.21.15.62:6789/0 conn(0x7fb1e015bca0 legacy=0x7fb1e0160160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:02.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:02.600+0000 7fb1e5d16640 1 -- 172.21.15.16:0/102168906 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1e00ccbe0 con 0x7fb1e00ce900 2023-05-22T15:42:02.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:02.600+0000 7fb1e5d16640 1 --2- 172.21.15.16:0/102168906 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb1e00ce900 0x7fb1e00ced80 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7fb1d0009870 tx=0x7fb1d001c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef67e2a9dd084ad9 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:02.606 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:02.600+0000 7fb1e5515640 1 -- 172.21.15.16:0/102168906 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1d00a6040 con 0x7fb1e00ce900 2023-05-22T15:42:02.606 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:02.600+0000 7fb1e5515640 1 -- 172.21.15.16:0/102168906 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1d00aa600 con 0x7fb1e00ce900 2023-05-22T15:42:02.606 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:02.600+0000 7fb1e5515640 1 -- 172.21.15.16:0/102168906 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1d00aa990 con 0x7fb1e00ce900 2023-05-22T15:42:02.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:02.604+0000 7fb1e877b640 1 -- 172.21.15.16:0/102168906 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb1e00ce900 msgr2=0x7fb1e00ced80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:02.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:02.604+0000 7fb1e877b640 1 --2- 172.21.15.16:0/102168906 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb1e00ce900 0x7fb1e00ced80 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7fb1d0009870 tx=0x7fb1d001c480 comp rx=0 tx=0).stop 2023-05-22T15:42:02.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:02.604+0000 7fb1e877b640 1 -- 172.21.15.16:0/102168906 shutdown_connections 2023-05-22T15:42:02.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:02.604+0000 7fb1e877b640 1 --2- 172.21.15.16:0/102168906 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb1e00cf2c0 0x7fb1e00d0890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:02.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:02.604+0000 7fb1e877b640 1 --2- 172.21.15.16:0/102168906 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb1e00ce900 0x7fb1e00ced80 unknown :-1 s=CLOSED pgs=979 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:02.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:02.604+0000 7fb1e877b640 1 -- 172.21.15.16:0/102168906 >> 172.21.15.16:0/102168906 conn(0x7fb1e00657f0 msgr2=0x7fb1e0070e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:02.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:02.604+0000 7fb1e877b640 1 -- 172.21.15.16:0/102168906 shutdown_connections 2023-05-22T15:42:02.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:02.604+0000 7fb1e877b640 1 -- 172.21.15.16:0/102168906 wait complete. 2023-05-22T15:42:02.610 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:02.604+0000 7fb1e877b640 1 Processor -- start 2023-05-22T15:42:02.610 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:02.604+0000 7fb1e877b640 1 -- start start 2023-05-22T15:42:02.610 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:02.604+0000 7fb1e877b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb1e00ce900 0x7fb1e00cb110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:02.610 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:02.604+0000 7fb1e877b640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb1e00cf2c0 0x7fb1e00c6020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:02.611 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:02.604+0000 7fb1e877b640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb1e015bca0 0x7fb1e00c9b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:02.611 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:02.604+0000 7fb1e877b640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1e00cb650 con 0x7fb1e00ce900 2023-05-22T15:42:02.611 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:02.604+0000 7fb1e877b640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1e00c92b0 con 0x7fb1e015bca0 2023-05-22T15:42:02.611 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:02.604+0000 7fb1e877b640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1e00b6100 con 0x7fb1e00cf2c0 2023-05-22T15:42:02.611 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:02.604+0000 7fb1e6517640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb1e00ce900 0x7fb1e00cb110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:02.611 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:02.604+0000 7fb1e6517640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb1e00ce900 0x7fb1e00cb110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50700/0 (socket says 172.21.15.16:50700) 2023-05-22T15:42:02.611 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:02.604+0000 7fb1e6517640 1 -- 172.21.15.16:0/721546280 learned_addr learned my addr 172.21.15.16:0/721546280 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:42:02.612 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:02.604+0000 7fb1e5d16640 1 --2- 172.21.15.16:0/721546280 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb1e00cf2c0 0x7fb1e00c6020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:02.612 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:02.604+0000 7fb1e6d18640 1 --2- 172.21.15.16:0/721546280 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb1e015bca0 0x7fb1e00c9b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:02.612 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:02.604+0000 7fb1e6517640 1 -- 172.21.15.16:0/721546280 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb1e00cf2c0 msgr2=0x7fb1e00c6020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:02.612 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:02.604+0000 7fb1e6517640 1 --2- 172.21.15.16:0/721546280 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb1e00cf2c0 0x7fb1e00c6020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:02.612 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:02.604+0000 7fb1e6517640 1 -- 172.21.15.16:0/721546280 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb1e015bca0 msgr2=0x7fb1e00c9b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:02.612 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:02.604+0000 7fb1e6517640 1 --2- 172.21.15.16:0/721546280 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb1e015bca0 0x7fb1e00c9b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:02.612 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:02.604+0000 7fb1e6517640 1 -- 172.21.15.16:0/721546280 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1d00094d0 con 0x7fb1e00ce900 2023-05-22T15:42:02.613 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:02.604+0000 7fb1e6517640 1 --2- 172.21.15.16:0/721546280 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb1e00ce900 0x7fb1e00cb110 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7fb1dc00c3c0 tx=0x7fb1dc00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:02.613 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:02.604+0000 7fb1d6ffd640 1 -- 172.21.15.16:0/721546280 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1dc0052d0 con 0x7fb1e00ce900 2023-05-22T15:42:02.613 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:02.604+0000 7fb1d6ffd640 1 -- 172.21.15.16:0/721546280 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1dc005430 con 0x7fb1e00ce900 2023-05-22T15:42:02.613 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:02.604+0000 7fb1e877b640 1 -- 172.21.15.16:0/721546280 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1e00d15e0 con 0x7fb1e00ce900 2023-05-22T15:42:02.613 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:02.604+0000 7fb1d6ffd640 1 -- 172.21.15.16:0/721546280 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1dc003840 con 0x7fb1e00ce900 2023-05-22T15:42:02.613 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:02.604+0000 7fb1e877b640 1 -- 172.21.15.16:0/721546280 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1e00cd1a0 con 0x7fb1e00ce900 2023-05-22T15:42:02.614 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:02.608+0000 7fb1e877b640 1 -- 172.21.15.16:0/721546280 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1e00d1770 con 0x7fb1e00ce900 2023-05-22T15:42:02.614 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:02.608+0000 7fb1d6ffd640 1 -- 172.21.15.16:0/721546280 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 22) v1 ==== 84361+0+0 (secure 0 0 0) 0x7fb1dc0039a0 con 0x7fb1e00ce900 2023-05-22T15:42:02.614 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:02.608+0000 7fb1d6ffd640 1 --2- 172.21.15.16:0/721546280 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fb1c40654f0 0x7fb1c40679a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:02.614 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:02.608+0000 7fb1d6ffd640 1 -- 172.21.15.16:0/721546280 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 6726+0+0 (secure 0 0 0) 0x7fb1dc082230 con 0x7fb1e00ce900 2023-05-22T15:42:02.615 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:02.608+0000 7fb1e5d16640 1 --2- 172.21.15.16:0/721546280 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fb1c40654f0 0x7fb1c40679a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:02.615 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:02.608+0000 7fb1e5d16640 1 --2- 172.21.15.16:0/721546280 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fb1c40654f0 0x7fb1c40679a0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fb1d00ae000 tx=0x7fb1d00a7000 comp rx=0 tx=0).ready entity=mgr.6663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:02.616 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:02.612+0000 7fb1d6ffd640 1 -- 172.21.15.16:0/721546280 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fb1dc04fa90 con 0x7fb1e00ce900 2023-05-22T15:42:02.815 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:02.812+0000 7fb1e877b640 1 -- 172.21.15.16:0/721546280 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7fb1e004dbe0 con 0x7fb1e00ce900 2023-05-22T15:42:03.473 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:03.468+0000 7fb1d6ffd640 1 -- 172.21.15.16:0/721546280 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mgrmap(e 23) v1 ==== 84344+0+0 (secure 0 0 0) 0x7fb1dc04fe60 con 0x7fb1e00ce900 2023-05-22T15:42:03.475 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:03.472+0000 7fb1d6ffd640 1 -- 172.21.15.16:0/721546280 <== mon.0 v2:172.21.15.16:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v23) v1 ==== 93+0+0 (secure 0 0 0) 0x7fb1dc053520 con 0x7fb1e00ce900 2023-05-22T15:42:03.478 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:03.476+0000 7fb1e877b640 1 -- 172.21.15.16:0/721546280 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fb1c40654f0 msgr2=0x7fb1c40679a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:03.478 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:03.476+0000 7fb1e877b640 1 --2- 172.21.15.16:0/721546280 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fb1c40654f0 0x7fb1c40679a0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fb1d00ae000 tx=0x7fb1d00a7000 comp rx=0 tx=0).stop 2023-05-22T15:42:03.479 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:03.476+0000 7fb1e877b640 1 -- 172.21.15.16:0/721546280 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb1e00ce900 msgr2=0x7fb1e00cb110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:03.479 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:03.476+0000 7fb1e877b640 1 --2- 172.21.15.16:0/721546280 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb1e00ce900 0x7fb1e00cb110 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7fb1dc00c3c0 tx=0x7fb1dc00c880 comp rx=0 tx=0).stop 2023-05-22T15:42:03.479 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:03.476+0000 7fb1e877b640 1 -- 172.21.15.16:0/721546280 shutdown_connections 2023-05-22T15:42:03.479 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:03.476+0000 7fb1e877b640 1 --2- 172.21.15.16:0/721546280 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb1e015bca0 0x7fb1e00c9b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:03.479 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:03.476+0000 7fb1e877b640 1 --2- 172.21.15.16:0/721546280 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fb1c40654f0 0x7fb1c40679a0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:03.480 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:03.476+0000 7fb1e877b640 1 --2- 172.21.15.16:0/721546280 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb1e00cf2c0 0x7fb1e00c6020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:03.480 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:03.476+0000 7fb1e877b640 1 --2- 172.21.15.16:0/721546280 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb1e00ce900 0x7fb1e00cb110 unknown :-1 s=CLOSED pgs=980 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:03.480 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:03.476+0000 7fb1e877b640 1 -- 172.21.15.16:0/721546280 >> 172.21.15.16:0/721546280 conn(0x7fb1e00657f0 msgr2=0x7fb1e0154cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:03.480 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:03.476+0000 7fb1e877b640 1 -- 172.21.15.16:0/721546280 shutdown_connections 2023-05-22T15:42:03.480 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:03.476+0000 7fb1e877b640 1 -- 172.21.15.16:0/721546280 wait complete. 2023-05-22T15:42:03.499 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-05-22T15:42:03.500 DEBUG:teuthology.orchestra.run.smithi016:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-05-22T15:42:03.535 DEBUG:teuthology.orchestra.run.smithi062:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-05-22T15:42:03.583 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-05-22T15:42:03.584 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-05-22T15:42:03.584 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2023-05-22T15:42:03.591 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-05-22T15:42:03.601 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:42:03.596+0000 7fe4172e7180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-05-22T15:42:03.604 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:42:03.599+0000 7f1b042b5180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-05-22T15:42:03.715 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:03.712+0000 7f66f0d51640 1 Processor -- start 2023-05-22T15:42:03.715 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:03.712+0000 7f66f0d51640 1 -- start start 2023-05-22T15:42:03.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:03.712+0000 7f66f0d51640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f66ec0c8250 0x7f66ec06c220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:03.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:03.712+0000 7f66f0d51640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f66ec0cc6c0 0x7f66ec06c760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:03.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:03.712+0000 7f66f0d51640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66ec15c210 con 0x7f66ec1563c0 2023-05-22T15:42:03.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:03.712+0000 7f66f0d51640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66ec160700 con 0x7f66ec0c8250 2023-05-22T15:42:03.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:03.712+0000 7f66f0d51640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66ec15c480 con 0x7f66ec0cc6c0 2023-05-22T15:42:03.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:03.712+0000 7f66e3fff640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f66ec0c8250 0x7f66ec06c220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:03.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:03.712+0000 7f66e3fff640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f66ec0c8250 0x7f66ec06c220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50720/0 (socket says 172.21.15.16:50720) 2023-05-22T15:42:03.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:03.712+0000 7f66e3fff640 1 -- 172.21.15.16:0/291972324 learned_addr learned my addr 172.21.15.16:0/291972324 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:42:03.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:03.712+0000 7f66ead9d640 1 --2- 172.21.15.16:0/291972324 >> v2:172.21.15.62:3300/0 conn(0x7f66ec0cc6c0 0x7f66ec06c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:03.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:03.712+0000 7f66e3fff640 1 -- 172.21.15.16:0/291972324 >> v2:172.21.15.62:3300/0 conn(0x7f66ec0cc6c0 msgr2=0x7f66ec06c760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:42:03.719 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:03.712+0000 7f66e3fff640 1 --2- 172.21.15.16:0/291972324 >> v2:172.21.15.62:3300/0 conn(0x7f66ec0cc6c0 0x7f66ec06c760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:03.719 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:03.712+0000 7f66e3fff640 1 -- 172.21.15.16:0/291972324 >> v1:172.21.15.62:6789/0 conn(0x7f66ec1563c0 legacy=0x7f66ec0c7b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:03.719 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:03.712+0000 7f66e3fff640 1 -- 172.21.15.16:0/291972324 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66ec15c760 con 0x7f66ec0c8250 2023-05-22T15:42:03.719 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:03.716+0000 7f66e3fff640 1 --2- 172.21.15.16:0/291972324 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66ec0c8250 0x7f66ec06c220 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7f66ec159cd0 tx=0x7f66d401c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0b05fc53401368e server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:03.720 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:03.716+0000 7f66e9d9b640 1 -- 172.21.15.16:0/291972324 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66d40aa430 con 0x7f66ec0c8250 2023-05-22T15:42:03.720 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:03.716+0000 7f66e9d9b640 1 -- 172.21.15.16:0/291972324 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66d40aa590 con 0x7f66ec0c8250 2023-05-22T15:42:03.720 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:03.716+0000 7f66e9d9b640 1 -- 172.21.15.16:0/291972324 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66d40aa920 con 0x7f66ec0c8250 2023-05-22T15:42:03.721 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:03.716+0000 7f66f0d51640 1 -- 172.21.15.16:0/291972324 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66ec0c8250 msgr2=0x7f66ec06c220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:03.721 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:03.716+0000 7f66f0d51640 1 --2- 172.21.15.16:0/291972324 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66ec0c8250 0x7f66ec06c220 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7f66ec159cd0 tx=0x7f66d401c4d0 comp rx=0 tx=0).stop 2023-05-22T15:42:03.721 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:03.716+0000 7f66f0d51640 1 -- 172.21.15.16:0/291972324 shutdown_connections 2023-05-22T15:42:03.721 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:03.716+0000 7f66f0d51640 1 --2- 172.21.15.16:0/291972324 >> v2:172.21.15.62:3300/0 conn(0x7f66ec0cc6c0 0x7f66ec06c760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:03.721 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:03.716+0000 7f66f0d51640 1 --2- 172.21.15.16:0/291972324 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66ec0c8250 0x7f66ec06c220 unknown :-1 s=CLOSED pgs=984 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:03.721 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:03.716+0000 7f66f0d51640 1 -- 172.21.15.16:0/291972324 >> 172.21.15.16:0/291972324 conn(0x7f66ec0b1110 msgr2=0x7f66ec0ce830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:03.721 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:03.716+0000 7f66f0d51640 1 -- 172.21.15.16:0/291972324 shutdown_connections 2023-05-22T15:42:03.722 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:03.716+0000 7f66f0d51640 1 -- 172.21.15.16:0/291972324 wait complete. 2023-05-22T15:42:03.724 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:03.716+0000 7f66f0d51640 1 Processor -- start 2023-05-22T15:42:03.725 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:03.716+0000 7f66f0d51640 1 -- start start 2023-05-22T15:42:03.725 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:03.720+0000 7f66f0d51640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f66ec0c8250 0x7f66ec0b17c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:03.725 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:03.720+0000 7f66f0d51640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f66ec0cc6c0 0x7f66ec1556f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:03.726 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:03.720+0000 7f66f0d51640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66ec14e3b0 0x7f66ec14c640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:03.726 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:03.720+0000 7f66f0d51640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66ec15f1e0 con 0x7f66ec14e3b0 2023-05-22T15:42:03.726 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:03.720+0000 7f66f0d51640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66ec155c30 con 0x7f66ec0c8250 2023-05-22T15:42:03.726 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:03.720+0000 7f66f0d51640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66ec15ef00 con 0x7f66ec0cc6c0 2023-05-22T15:42:03.727 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:03.720+0000 7f66ead9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66ec14e3b0 0x7f66ec14c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:03.727 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:03.720+0000 7f66ead9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66ec14e3b0 0x7f66ec14c640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50724/0 (socket says 172.21.15.16:50724) 2023-05-22T15:42:03.727 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:03.720+0000 7f66ead9d640 1 -- 172.21.15.16:0/1143141434 learned_addr learned my addr 172.21.15.16:0/1143141434 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:42:03.727 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:03.720+0000 7f66e3fff640 1 --2- 172.21.15.16:0/1143141434 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f66ec0cc6c0 0x7f66ec1556f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:03.727 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:03.720+0000 7f66ea59c640 1 --2- 172.21.15.16:0/1143141434 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f66ec0c8250 0x7f66ec0b17c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:03.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:03.720+0000 7f66ead9d640 1 -- 172.21.15.16:0/1143141434 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f66ec0cc6c0 msgr2=0x7f66ec1556f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:03.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:03.720+0000 7f66ead9d640 1 --2- 172.21.15.16:0/1143141434 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f66ec0cc6c0 0x7f66ec1556f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:03.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:03.720+0000 7f66ead9d640 1 -- 172.21.15.16:0/1143141434 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f66ec0c8250 msgr2=0x7f66ec0b17c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:03.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:03.720+0000 7f66ead9d640 1 --2- 172.21.15.16:0/1143141434 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f66ec0c8250 0x7f66ec0b17c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:03.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:03.720+0000 7f66ead9d640 1 -- 172.21.15.16:0/1143141434 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66d40094d0 con 0x7f66ec14e3b0 2023-05-22T15:42:03.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:03.720+0000 7f66ead9d640 1 --2- 172.21.15.16:0/1143141434 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66ec14e3b0 0x7f66ec14c640 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7f66e4007e70 tx=0x7f66e400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:03.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:03.720+0000 7f66e2ffd640 1 -- 172.21.15.16:0/1143141434 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66e4014620 con 0x7f66ec14e3b0 2023-05-22T15:42:03.729 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:03.720+0000 7f66e2ffd640 1 -- 172.21.15.16:0/1143141434 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66e40147a0 con 0x7f66ec14e3b0 2023-05-22T15:42:03.729 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:03.720+0000 7f66f0d51640 1 -- 172.21.15.16:0/1143141434 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66ec06ce30 con 0x7f66ec14e3b0 2023-05-22T15:42:03.729 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:03.720+0000 7f66e2ffd640 1 -- 172.21.15.16:0/1143141434 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66e4014b50 con 0x7f66ec14e3b0 2023-05-22T15:42:03.729 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:03.720+0000 7f66f0d51640 1 -- 172.21.15.16:0/1143141434 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66ec06cca0 con 0x7f66ec14e3b0 2023-05-22T15:42:03.730 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:03.720+0000 7f66f0d51640 1 -- 172.21.15.16:0/1143141434 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66ec15ff00 con 0x7f66ec14e3b0 2023-05-22T15:42:03.730 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:03.720+0000 7f66e2ffd640 1 -- 172.21.15.16:0/1143141434 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 23) v1 ==== 84344+0+0 (secure 0 0 0) 0x7f66e4014cb0 con 0x7f66ec14e3b0 2023-05-22T15:42:03.730 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:03.720+0000 7f66e2ffd640 1 --2- 172.21.15.16:0/1143141434 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f66cc0654a0 0x7f66cc067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:03.730 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:03.720+0000 7f66ea59c640 1 -- 172.21.15.16:0/1143141434 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f66cc0654a0 msgr2=0x7f66cc067950 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/36921 2023-05-22T15:42:03.730 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:03.720+0000 7f66e2ffd640 1 -- 172.21.15.16:0/1143141434 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 6814+0+0 (secure 0 0 0) 0x7f66e4083560 con 0x7f66ec14e3b0 2023-05-22T15:42:03.731 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:03.720+0000 7f66ea59c640 1 --2- 172.21.15.16:0/1143141434 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f66cc0654a0 0x7f66cc067950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-05-22T15:42:03.731 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:03.724+0000 7f66e2ffd640 1 -- 172.21.15.16:0/1143141434 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f66e40548b0 con 0x7f66ec14e3b0 2023-05-22T15:42:03.883 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:03.880+0000 7f66f0d51640 1 -- 172.21.15.16:0/1143141434 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f66ec155410 con 0x7f66ec14e3b0 2023-05-22T15:42:03.885 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:03.880+0000 7f66e2ffd640 1 -- 172.21.15.16:0/1143141434 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v75) v1 ==== 74+0+18680 (secure 0 0 0) 0x7f66e40544a0 con 0x7f66ec14e3b0 2023-05-22T15:42:03.885 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:42:03.886 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":75,"fsid":"b2719834-7847-4d98-90a6-f65d57f85c49","created":"2023-05-22T15:26:25.259314+0000","modified":"2023-05-22T15:42:03.439253+0000","last_up_change":"2023-05-22T15:26:36.448524+0000","last_in_change":"2023-05-22T15:26:27.675668+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-05-22T15:26:38.019781+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-05-22T15:26:43.570514+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2023-05-22T15:40:00.663040+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"72","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2023-05-22T15:40:01.750680+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":4,"snap_epoch":75,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9e7ff854-b25f-4cbb-a5c2-1799d9c53d2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":37021},{"type":"v1","addr":"172.21.15.16:6825","nonce":37021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":37021},{"type":"v1","addr":"172.21.15.16:6827","nonce":37021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":37021},{"type":"v1","addr":"172.21.15.16:6831","nonce":37021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":37021},{"type":"v1","addr":"172.21.15.16:6829","nonce":37021}]},"public_addr":"172.21.15.16:6825/37021","cluster_addr":"172.21.15.16:6827/37021","heartbeat_back_addr":"172.21.15.16:6831/37021","heartbeat_front_addr":"172.21.15.16:6829/37021","state":["exists","up"]},{"osd":1,"uuid":"4702fbc9-a927-4565-aca5-41dc69cc8264","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":37008},{"type":"v1","addr":"172.21.15.16:6801","nonce":37008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":37008},{"type":"v1","addr":"172.21.15.16:6803","nonce":37008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":37008},{"type":"v1","addr":"172.21.15.16:6807","nonce":37008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":37008},{"type":"v1","addr":"172.21.15.16:6805","nonce":37008}]},"public_addr":"172.21.15.16:6801/37008","cluster_addr":"172.21.15.16:6803/37008","heartbeat_back_addr":"172.21.15.16:6807/37008","heartbeat_front_addr":"172.21.15.16:6805/37008","state":["exists","up"]},{"osd":2,"uuid":"1e7bba46-4eba-47ef-bb81-5e2d371638ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":37009},{"type":"v1","addr":"172.21.15.16:6809","nonce":37009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":37009},{"type":"v1","addr":"172.21.15.16:6811","nonce":37009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":37009},{"type":"v1","addr":"172.21.15.16:6815","nonce":37009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":37009},{"type":"v1","addr":"172.21.15.16:6813","nonce":37009}]},"public_addr":"172.21.15.16:6809/37009","cluster_addr":"172.21.15.16:6811/37009","heartbeat_back_addr":"172.21.15.16:6815/37009","heartbeat_front_addr":"172.21.15.16:6813/37009","state":["exists","up"]},{"osd":3,"uuid":"383cacc0-5a88-4904-990e-498ca59ef48a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":37020},{"type":"v1","addr":"172.21.15.16:6817","nonce":37020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":37020},{"type":"v1","addr":"172.21.15.16:6819","nonce":37020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":37020},{"type":"v1","addr":"172.21.15.16:6823","nonce":37020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":37020},{"type":"v1","addr":"172.21.15.16:6821","nonce":37020}]},"public_addr":"172.21.15.16:6817/37020","cluster_addr":"172.21.15.16:6819/37020","heartbeat_back_addr":"172.21.15.16:6823/37020","heartbeat_front_addr":"172.21.15.16:6821/37020","state":["exists","up"]},{"osd":4,"uuid":"970500ef-95b7-42c1-b080-eed4fae7013d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":37353},{"type":"v1","addr":"172.21.15.62:6817","nonce":37353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6819","nonce":37353},{"type":"v1","addr":"172.21.15.62:6822","nonce":37353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37353},{"type":"v1","addr":"172.21.15.62:6831","nonce":37353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37353},{"type":"v1","addr":"172.21.15.62:6826","nonce":37353}]},"public_addr":"172.21.15.62:6817/37353","cluster_addr":"172.21.15.62:6822/37353","heartbeat_back_addr":"172.21.15.62:6831/37353","heartbeat_front_addr":"172.21.15.62:6826/37353","state":["exists","up"]},{"osd":5,"uuid":"022d67db-d4d8-445b-b1b9-8b446a00f5c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":37352},{"type":"v1","addr":"172.21.15.62:6801","nonce":37352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":37352},{"type":"v1","addr":"172.21.15.62:6803","nonce":37352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":37352},{"type":"v1","addr":"172.21.15.62:6807","nonce":37352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":37352},{"type":"v1","addr":"172.21.15.62:6805","nonce":37352}]},"public_addr":"172.21.15.62:6801/37352","cluster_addr":"172.21.15.62:6803/37352","heartbeat_back_addr":"172.21.15.62:6807/37352","heartbeat_front_addr":"172.21.15.62:6805/37352","state":["exists","up"]},{"osd":6,"uuid":"42bd372b-6688-4a1b-8407-e16e8c266e33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":37354},{"type":"v1","addr":"172.21.15.62:6820","nonce":37354}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":37354},{"type":"v1","addr":"172.21.15.62:6823","nonce":37354}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6829","nonce":37354},{"type":"v1","addr":"172.21.15.62:6830","nonce":37354}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6825","nonce":37354},{"type":"v1","addr":"172.21.15.62:6827","nonce":37354}]},"public_addr":"172.21.15.62:6820/37354","cluster_addr":"172.21.15.62:6823/37354","heartbeat_back_addr":"172.21.15.62:6830/37354","heartbeat_front_addr":"172.21.15.62:6827/37354","state":["exists","up"]},{"osd":7,"uuid":"ac70b2a3-f41b-4bf2-b227-7f2cbb55426d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37355},{"type":"v1","addr":"172.21.15.62:6809","nonce":37355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37355},{"type":"v1","addr":"172.21.15.62:6811","nonce":37355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":37355},{"type":"v1","addr":"172.21.15.62:6815","nonce":37355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":37355},{"type":"v1","addr":"172.21.15.62:6813","nonce":37355}]},"public_addr":"172.21.15.62:6809/37355","cluster_addr":"172.21.15.62:6811/37355","heartbeat_back_addr":"172.21.15.62:6815/37355","heartbeat_front_addr":"172.21.15.62:6813/37355","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.368896+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.107273+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.276888+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.287502+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.124791+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.251937+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.255679+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.232797+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:0/2468042232":"2023-05-23T15:40:15.816078+0000","172.21.15.16:0/1582569975":"2023-05-23T15:40:15.816078+0000","172.21.15.16:0/2252481002":"2023-05-23T15:40:15.816078+0000","172.21.15.16:0/174222799":"2023-05-23T15:40:15.816078+0000","172.21.15.16:0/4017778835":"2023-05-23T15:40:15.816078+0000","172.21.15.16:0/3438310522":"2023-05-23T15:40:15.816078+0000","172.21.15.16:6833/36921":"2023-05-23T15:40:15.816078+0000","172.21.15.16:6832/36921":"2023-05-23T15:40:15.816078+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":9,"snaps":[{"begin":2,"length":1},{"begin":4,"length":1}]}],"new_removed_snaps":[{"pool":9,"snaps":[{"begin":2,"length":1},{"begin":4,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-05-22T15:42:03.888 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:03.884+0000 7f66f0d51640 1 -- 172.21.15.16:0/1143141434 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f66cc0654a0 msgr2=0x7f66cc067950 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-05-22T15:42:03.888 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:03.884+0000 7f66f0d51640 1 --2- 172.21.15.16:0/1143141434 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f66cc0654a0 0x7f66cc067950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:03.888 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:03.884+0000 7f66f0d51640 1 -- 172.21.15.16:0/1143141434 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66ec14e3b0 msgr2=0x7f66ec14c640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:03.888 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:03.884+0000 7f66f0d51640 1 --2- 172.21.15.16:0/1143141434 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66ec14e3b0 0x7f66ec14c640 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7f66e4007e70 tx=0x7f66e400e4b0 comp rx=0 tx=0).stop 2023-05-22T15:42:03.888 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:03.884+0000 7f66f0d51640 1 -- 172.21.15.16:0/1143141434 shutdown_connections 2023-05-22T15:42:03.889 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:03.884+0000 7f66f0d51640 1 --2- 172.21.15.16:0/1143141434 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66ec14e3b0 0x7f66ec14c640 unknown :-1 s=CLOSED pgs=985 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:03.889 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:03.884+0000 7f66f0d51640 1 --2- 172.21.15.16:0/1143141434 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f66cc0654a0 0x7f66cc067950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:03.889 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:03.884+0000 7f66f0d51640 1 --2- 172.21.15.16:0/1143141434 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f66ec0cc6c0 0x7f66ec1556f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:03.889 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:03.884+0000 7f66f0d51640 1 --2- 172.21.15.16:0/1143141434 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f66ec0c8250 0x7f66ec0b17c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:03.890 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:03.884+0000 7f66f0d51640 1 -- 172.21.15.16:0/1143141434 >> 172.21.15.16:0/1143141434 conn(0x7f66ec0b1110 msgr2=0x7f66ec0cb240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:03.890 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:03.884+0000 7f66f0d51640 1 -- 172.21.15.16:0/1143141434 shutdown_connections 2023-05-22T15:42:03.890 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:03.884+0000 7f66f0d51640 1 -- 172.21.15.16:0/1143141434 wait complete. 2023-05-22T15:42:03.903 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-05-22T15:26:38.019781+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-05-22T15:26:43.570514+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_metadata', 'create_time': '2023-05-22T15:40:00.663040+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '72', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 9, 'pool_name': 'cephfs_data', 'create_time': '2023-05-22T15:40:01.750680+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '75', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 4, 'snap_epoch': 75, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-05-22T15:42:03.905 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-05-22T15:42:04.009 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:42:04.004+0000 7fe4172e7180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-05-22T15:42:04.013 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:42:04.011+0000 7f1b042b5180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-05-22T15:42:04.031 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.020+0000 7fb9bfad0640 1 Processor -- start 2023-05-22T15:42:04.031 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.020+0000 7fb9bfad0640 1 -- start start 2023-05-22T15:42:04.031 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.024+0000 7fb9bfad0640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb9b8158a10 0x7fb9b806e480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:04.031 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.024+0000 7fb9bfad0640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb9b806cdf0 0x7fb9b8070bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:04.031 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.024+0000 7fb9bfad0640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9b8158de0 con 0x7fb9b8070760 2023-05-22T15:42:04.032 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.024+0000 7fb9bfad0640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9b806e060 con 0x7fb9b806cdf0 2023-05-22T15:42:04.032 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.024+0000 7fb9bfad0640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9b80711e0 con 0x7fb9b8158a10 2023-05-22T15:42:04.032 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.024+0000 7fb9be06d640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb9b806cdf0 0x7fb9b8070bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:04.032 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.024+0000 7fb9be06d640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb9b806cdf0 0x7fb9b8070bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50732/0 (socket says 172.21.15.16:50732) 2023-05-22T15:42:04.032 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.024+0000 7fb9be06d640 1 -- 172.21.15.16:0/3672716202 learned_addr learned my addr 172.21.15.16:0/3672716202 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:42:04.032 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.024+0000 7fb9bd86c640 1 --2- 172.21.15.16:0/3672716202 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb9b8158a10 0x7fb9b806e480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:04.032 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.024+0000 7fb9be06d640 1 -- 172.21.15.16:0/3672716202 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb9b8158a10 msgr2=0x7fb9b806e480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:42:04.032 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.024+0000 7fb9be06d640 1 --2- 172.21.15.16:0/3672716202 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb9b8158a10 0x7fb9b806e480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:04.033 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.024+0000 7fb9be06d640 1 -- 172.21.15.16:0/3672716202 >> v1:172.21.15.62:6789/0 conn(0x7fb9b8070760 legacy=0x7fb9b806c6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:04.033 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.024+0000 7fb9be06d640 1 -- 172.21.15.16:0/3672716202 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9b80714c0 con 0x7fb9b806cdf0 2023-05-22T15:42:04.033 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.024+0000 7fb9be06d640 1 --2- 172.21.15.16:0/3672716202 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb9b806cdf0 0x7fb9b8070bb0 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7fb9b000d150 tx=0x7fb9b001eb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b17a2516ce4fd80 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:04.033 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.024+0000 7fb9bc86a640 1 -- 172.21.15.16:0/3672716202 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9b000b2d0 con 0x7fb9b806cdf0 2023-05-22T15:42:04.033 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.024+0000 7fb9bc86a640 1 -- 172.21.15.16:0/3672716202 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9b000b430 con 0x7fb9b806cdf0 2023-05-22T15:42:04.033 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.024+0000 7fb9bc86a640 1 -- 172.21.15.16:0/3672716202 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9b00a7c90 con 0x7fb9b806cdf0 2023-05-22T15:42:04.033 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.028+0000 7fb9bfad0640 1 -- 172.21.15.16:0/3672716202 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb9b806cdf0 msgr2=0x7fb9b8070bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:04.034 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.028+0000 7fb9bfad0640 1 --2- 172.21.15.16:0/3672716202 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb9b806cdf0 0x7fb9b8070bb0 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7fb9b000d150 tx=0x7fb9b001eb80 comp rx=0 tx=0).stop 2023-05-22T15:42:04.034 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.028+0000 7fb9bfad0640 1 -- 172.21.15.16:0/3672716202 shutdown_connections 2023-05-22T15:42:04.034 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.028+0000 7fb9bfad0640 1 --2- 172.21.15.16:0/3672716202 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb9b806cdf0 0x7fb9b8070bb0 unknown :-1 s=CLOSED pgs=986 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:04.034 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.028+0000 7fb9bfad0640 1 --2- 172.21.15.16:0/3672716202 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb9b8158a10 0x7fb9b806e480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:04.034 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.028+0000 7fb9bfad0640 1 -- 172.21.15.16:0/3672716202 >> 172.21.15.16:0/3672716202 conn(0x7fb9b80cc870 msgr2=0x7fb9b80cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:04.034 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.028+0000 7fb9bfad0640 1 -- 172.21.15.16:0/3672716202 shutdown_connections 2023-05-22T15:42:04.034 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.028+0000 7fb9bfad0640 1 -- 172.21.15.16:0/3672716202 wait complete. 2023-05-22T15:42:04.036 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.028+0000 7fb9bfad0640 1 Processor -- start 2023-05-22T15:42:04.036 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.028+0000 7fb9bfad0640 1 -- start start 2023-05-22T15:42:04.037 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.028+0000 7fb9bfad0640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb9b806cdf0 0x7fb9b815b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:04.037 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.028+0000 7fb9bfad0640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb9b8070760 0x7fb9b814cac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:04.037 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.028+0000 7fb9bfad0640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb9b8158a10 0x7fb9b814d000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:04.037 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.028+0000 7fb9bfad0640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9b814d540 con 0x7fb9b8070760 2023-05-22T15:42:04.037 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.028+0000 7fb9bfad0640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9b815be50 con 0x7fb9b806cdf0 2023-05-22T15:42:04.037 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.028+0000 7fb9bfad0640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9b80b4e00 con 0x7fb9b8158a10 2023-05-22T15:42:04.037 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.028+0000 7fb9bd06b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb9b8070760 0x7fb9b814cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:04.037 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.028+0000 7fb9bd06b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb9b8070760 0x7fb9b814cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50738/0 (socket says 172.21.15.16:50738) 2023-05-22T15:42:04.038 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.028+0000 7fb9bd06b640 1 -- 172.21.15.16:0/3981201653 learned_addr learned my addr 172.21.15.16:0/3981201653 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:42:04.038 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.028+0000 7fb9bd86c640 1 --2- 172.21.15.16:0/3981201653 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb9b806cdf0 0x7fb9b815b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:04.038 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.028+0000 7fb9be06d640 1 --2- 172.21.15.16:0/3981201653 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb9b8158a10 0x7fb9b814d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:04.038 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.028+0000 7fb9bd06b640 1 -- 172.21.15.16:0/3981201653 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb9b8158a10 msgr2=0x7fb9b814d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:04.038 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.028+0000 7fb9bd06b640 1 --2- 172.21.15.16:0/3981201653 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb9b8158a10 0x7fb9b814d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:04.038 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.028+0000 7fb9bd06b640 1 -- 172.21.15.16:0/3981201653 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb9b806cdf0 msgr2=0x7fb9b815b750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:04.038 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.028+0000 7fb9bd06b640 1 --2- 172.21.15.16:0/3981201653 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb9b806cdf0 0x7fb9b815b750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:04.039 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.028+0000 7fb9bd06b640 1 -- 172.21.15.16:0/3981201653 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9b000bd40 con 0x7fb9b8070760 2023-05-22T15:42:04.039 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.028+0000 7fb9be06d640 1 --2- 172.21.15.16:0/3981201653 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb9b8158a10 0x7fb9b814d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T15:42:04.039 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.028+0000 7fb9bd86c640 1 --2- 172.21.15.16:0/3981201653 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb9b806cdf0 0x7fb9b815b750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T15:42:04.039 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.032+0000 7fb9bd06b640 1 --2- 172.21.15.16:0/3981201653 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb9b8070760 0x7fb9b814cac0 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7fb9a800c500 tx=0x7fb9a800c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:04.039 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.032+0000 7fb9ae7fc640 1 -- 172.21.15.16:0/3981201653 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9a8005220 con 0x7fb9b8070760 2023-05-22T15:42:04.039 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.032+0000 7fb9ae7fc640 1 -- 172.21.15.16:0/3981201653 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9a8005380 con 0x7fb9b8070760 2023-05-22T15:42:04.039 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.032+0000 7fb9ae7fc640 1 -- 172.21.15.16:0/3981201653 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9a8007270 con 0x7fb9b8070760 2023-05-22T15:42:04.039 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.032+0000 7fb9bfad0640 1 -- 172.21.15.16:0/3981201653 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9b806eb50 con 0x7fb9b8070760 2023-05-22T15:42:04.040 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.032+0000 7fb9bfad0640 1 -- 172.21.15.16:0/3981201653 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9b806e9c0 con 0x7fb9b8070760 2023-05-22T15:42:04.040 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.032+0000 7fb9bfad0640 1 -- 172.21.15.16:0/3981201653 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9b80ca280 con 0x7fb9b8070760 2023-05-22T15:42:04.040 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.032+0000 7fb9ae7fc640 1 -- 172.21.15.16:0/3981201653 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 23) v1 ==== 84344+0+0 (secure 0 0 0) 0x7fb9a80073d0 con 0x7fb9b8070760 2023-05-22T15:42:04.041 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.032+0000 7fb9ae7fc640 1 --2- 172.21.15.16:0/3981201653 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fb99c0654a0 0x7fb99c067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:04.041 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.032+0000 7fb9bd86c640 1 -- 172.21.15.16:0/3981201653 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fb99c0654a0 msgr2=0x7fb99c067950 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/36921 2023-05-22T15:42:04.041 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.032+0000 7fb9ae7fc640 1 -- 172.21.15.16:0/3981201653 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 6814+0+0 (secure 0 0 0) 0x7fb9a8082660 con 0x7fb9b8070760 2023-05-22T15:42:04.041 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.032+0000 7fb9bd86c640 1 --2- 172.21.15.16:0/3981201653 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fb99c0654a0 0x7fb99c067950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-05-22T15:42:04.041 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.036+0000 7fb9ae7fc640 1 -- 172.21.15.16:0/3981201653 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fb9a804fe50 con 0x7fb9b8070760 2023-05-22T15:42:04.148 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:42:04.143+0000 7f1b042b5180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-05-22T15:42:04.149 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:42:04.144+0000 7fe4172e7180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-05-22T15:42:04.191 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.188+0000 7fb9bfad0640 1 -- 172.21.15.16:0/3981201653 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fb9b804dbe0 con 0x7fb9b8070760 2023-05-22T15:42:04.192 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.188+0000 7fb9ae7fc640 1 -- 172.21.15.16:0/3981201653 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v75) v1 ==== 93+0+10 (secure 0 0 0) 0x7fb9a8053470 con 0x7fb9b8070760 2023-05-22T15:42:04.193 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2023-05-22T15:42:04.194 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.192+0000 7fb9bfad0640 1 -- 172.21.15.16:0/3981201653 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fb99c0654a0 msgr2=0x7fb99c067950 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-05-22T15:42:04.194 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.192+0000 7fb9bfad0640 1 --2- 172.21.15.16:0/3981201653 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fb99c0654a0 0x7fb99c067950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:04.195 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.192+0000 7fb9bfad0640 1 -- 172.21.15.16:0/3981201653 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb9b8070760 msgr2=0x7fb9b814cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:04.195 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.192+0000 7fb9bfad0640 1 --2- 172.21.15.16:0/3981201653 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb9b8070760 0x7fb9b814cac0 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7fb9a800c500 tx=0x7fb9a800c9c0 comp rx=0 tx=0).stop 2023-05-22T15:42:04.195 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.192+0000 7fb9bfad0640 1 -- 172.21.15.16:0/3981201653 shutdown_connections 2023-05-22T15:42:04.195 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.192+0000 7fb9bfad0640 1 --2- 172.21.15.16:0/3981201653 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fb99c0654a0 0x7fb99c067950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:04.195 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.192+0000 7fb9bfad0640 1 --2- 172.21.15.16:0/3981201653 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb9b8158a10 0x7fb9b814d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:04.195 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.192+0000 7fb9bfad0640 1 --2- 172.21.15.16:0/3981201653 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb9b8070760 0x7fb9b814cac0 unknown :-1 s=CLOSED pgs=987 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:04.196 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.192+0000 7fb9bfad0640 1 --2- 172.21.15.16:0/3981201653 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb9b806cdf0 0x7fb9b815b750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:04.196 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.192+0000 7fb9bfad0640 1 -- 172.21.15.16:0/3981201653 >> 172.21.15.16:0/3981201653 conn(0x7fb9b80cc870 msgr2=0x7fb9b815c4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:04.196 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.192+0000 7fb9bfad0640 1 -- 172.21.15.16:0/3981201653 shutdown_connections 2023-05-22T15:42:04.196 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.192+0000 7fb9bfad0640 1 -- 172.21.15.16:0/3981201653 wait complete. 2023-05-22T15:42:04.210 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-05-22T15:42:04.334 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.332+0000 7f52f522f640 1 Processor -- start 2023-05-22T15:42:04.335 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.332+0000 7f52f522f640 1 -- start start 2023-05-22T15:42:04.335 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.332+0000 7f52f522f640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f52f0070760 0x7f52f0070be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:04.336 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.332+0000 7f52f522f640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f52f006c6e0 0x7f52f006cb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:04.336 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.332+0000 7f52f522f640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52f006eb90 con 0x7f52f0158a10 2023-05-22T15:42:04.336 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.332+0000 7f52f522f640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52f006e060 con 0x7f52f0070760 2023-05-22T15:42:04.336 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.332+0000 7f52f522f640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52f006d0c0 con 0x7f52f006c6e0 2023-05-22T15:42:04.336 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.332+0000 7f52ee59c640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f52f0070760 0x7f52f0070be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:04.336 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.332+0000 7f52ee59c640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f52f0070760 0x7f52f0070be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50748/0 (socket says 172.21.15.16:50748) 2023-05-22T15:42:04.337 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.332+0000 7f52ee59c640 1 -- 172.21.15.16:0/1394317724 learned_addr learned my addr 172.21.15.16:0/1394317724 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:42:04.337 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.332+0000 7f52ef59e640 1 --2- 172.21.15.16:0/1394317724 >> v2:172.21.15.62:3300/0 conn(0x7f52f006c6e0 0x7f52f006cb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:04.337 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.332+0000 7f52ee59c640 1 -- 172.21.15.16:0/1394317724 >> v2:172.21.15.62:3300/0 conn(0x7f52f006c6e0 msgr2=0x7f52f006cb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:42:04.337 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.332+0000 7f52ee59c640 1 --2- 172.21.15.16:0/1394317724 >> v2:172.21.15.62:3300/0 conn(0x7f52f006c6e0 0x7f52f006cb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:04.337 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.332+0000 7f52ee59c640 1 -- 172.21.15.16:0/1394317724 >> v1:172.21.15.62:6789/0 conn(0x7f52f0158a10 legacy=0x7f52f006e480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:04.337 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.332+0000 7f52ee59c640 1 -- 172.21.15.16:0/1394317724 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52f0158de0 con 0x7f52f0070760 2023-05-22T15:42:04.338 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.332+0000 7f52ee59c640 1 --2- 172.21.15.16:0/1394317724 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f52f0070760 0x7f52f0070be0 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7f52f0071be0 tx=0x7f52d801c470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85b7ee0b45fcc4e2 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:04.338 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.336+0000 7f52edd9b640 1 -- 172.21.15.16:0/1394317724 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52d80ab670 con 0x7f52f0070760 2023-05-22T15:42:04.338 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.336+0000 7f52edd9b640 1 -- 172.21.15.16:0/1394317724 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52d80ab7d0 con 0x7f52f0070760 2023-05-22T15:42:04.338 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.336+0000 7f52edd9b640 1 -- 172.21.15.16:0/1394317724 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52d80abb60 con 0x7f52f0070760 2023-05-22T15:42:04.339 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.336+0000 7f52f522f640 1 -- 172.21.15.16:0/1394317724 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f52f0070760 msgr2=0x7f52f0070be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:04.339 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.336+0000 7f52f522f640 1 --2- 172.21.15.16:0/1394317724 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f52f0070760 0x7f52f0070be0 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7f52f0071be0 tx=0x7f52d801c470 comp rx=0 tx=0).stop 2023-05-22T15:42:04.339 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.336+0000 7f52f522f640 1 -- 172.21.15.16:0/1394317724 shutdown_connections 2023-05-22T15:42:04.340 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.336+0000 7f52f522f640 1 --2- 172.21.15.16:0/1394317724 >> v2:172.21.15.62:3300/0 conn(0x7f52f006c6e0 0x7f52f006cb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:04.340 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.336+0000 7f52f522f640 1 --2- 172.21.15.16:0/1394317724 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f52f0070760 0x7f52f0070be0 unknown :-1 s=CLOSED pgs=988 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:04.340 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.336+0000 7f52f522f640 1 -- 172.21.15.16:0/1394317724 >> 172.21.15.16:0/1394317724 conn(0x7f52f00cc870 msgr2=0x7f52f00cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:04.340 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.336+0000 7f52f522f640 1 -- 172.21.15.16:0/1394317724 shutdown_connections 2023-05-22T15:42:04.340 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.336+0000 7f52f522f640 1 -- 172.21.15.16:0/1394317724 wait complete. 2023-05-22T15:42:04.342 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.336+0000 7f52f522f640 1 Processor -- start 2023-05-22T15:42:04.342 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.336+0000 7f52f522f640 1 -- start start 2023-05-22T15:42:04.342 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.336+0000 7f52f522f640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f52f006c6e0 0x7f52f014f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:04.342 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.336+0000 7f52f522f640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f52f0070760 0x7f52f00b42d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:04.342 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.336+0000 7f52f522f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f52f0158a10 0x7f52f0157900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:04.342 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.336+0000 7f52f522f640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52f014f6e0 con 0x7f52f0158a10 2023-05-22T15:42:04.343 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.336+0000 7f52f522f640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52f0153250 con 0x7f52f0070760 2023-05-22T15:42:04.343 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.336+0000 7f52f522f640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52f0153530 con 0x7f52f006c6e0 2023-05-22T15:42:04.343 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.336+0000 7f52ef59e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f52f0158a10 0x7f52f0157900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:04.343 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.336+0000 7f52ef59e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f52f0158a10 0x7f52f0157900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50760/0 (socket says 172.21.15.16:50760) 2023-05-22T15:42:04.343 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.336+0000 7f52ef59e640 1 -- 172.21.15.16:0/3808791755 learned_addr learned my addr 172.21.15.16:0/3808791755 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:42:04.343 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.336+0000 7f52ee59c640 1 --2- 172.21.15.16:0/3808791755 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f52f0070760 0x7f52f00b42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:04.343 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.336+0000 7f52eed9d640 1 --2- 172.21.15.16:0/3808791755 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f52f006c6e0 0x7f52f014f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:04.343 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.336+0000 7f52ef59e640 1 -- 172.21.15.16:0/3808791755 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f52f006c6e0 msgr2=0x7f52f014f1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:04.344 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.336+0000 7f52ef59e640 1 --2- 172.21.15.16:0/3808791755 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f52f006c6e0 0x7f52f014f1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:04.344 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.336+0000 7f52ef59e640 1 -- 172.21.15.16:0/3808791755 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f52f0070760 msgr2=0x7f52f00b42d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:04.344 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.336+0000 7f52ef59e640 1 --2- 172.21.15.16:0/3808791755 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f52f0070760 0x7f52f00b42d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:04.344 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.336+0000 7f52ef59e640 1 -- 172.21.15.16:0/3808791755 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52d80094d0 con 0x7f52f0158a10 2023-05-22T15:42:04.344 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.336+0000 7f52ef59e640 1 --2- 172.21.15.16:0/3808791755 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f52f0158a10 0x7f52f0157900 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7f52e000c7f0 tx=0x7f52e000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:04.344 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.340+0000 7f52e77fe640 1 -- 172.21.15.16:0/3808791755 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52e0004220 con 0x7f52f0158a10 2023-05-22T15:42:04.344 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.340+0000 7f52e77fe640 1 -- 172.21.15.16:0/3808791755 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52e0004380 con 0x7f52f0158a10 2023-05-22T15:42:04.345 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.340+0000 7f52e77fe640 1 -- 172.21.15.16:0/3808791755 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52e0003840 con 0x7f52f0158a10 2023-05-22T15:42:04.345 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.340+0000 7f52f522f640 1 -- 172.21.15.16:0/3808791755 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52f00cfe40 con 0x7f52f0158a10 2023-05-22T15:42:04.345 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.340+0000 7f52f522f640 1 -- 172.21.15.16:0/3808791755 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52f006dc80 con 0x7f52f0158a10 2023-05-22T15:42:04.346 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.340+0000 7f52f522f640 1 -- 172.21.15.16:0/3808791755 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52f00ca280 con 0x7f52f0158a10 2023-05-22T15:42:04.346 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.340+0000 7f52e77fe640 1 -- 172.21.15.16:0/3808791755 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 23) v1 ==== 84344+0+0 (secure 0 0 0) 0x7f52e0003bb0 con 0x7f52f0158a10 2023-05-22T15:42:04.346 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.340+0000 7f52e77fe640 1 --2- 172.21.15.16:0/3808791755 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f52d00654a0 0x7f52d0067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:04.346 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.340+0000 7f52eed9d640 1 -- 172.21.15.16:0/3808791755 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f52d00654a0 msgr2=0x7f52d0067950 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/36921 2023-05-22T15:42:04.346 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.340+0000 7f52e77fe640 1 -- 172.21.15.16:0/3808791755 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 6814+0+0 (secure 0 0 0) 0x7f52e0082c40 con 0x7f52f0158a10 2023-05-22T15:42:04.346 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.340+0000 7f52eed9d640 1 --2- 172.21.15.16:0/3808791755 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f52d00654a0 0x7f52d0067950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-05-22T15:42:04.348 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.344+0000 7f52e77fe640 1 -- 172.21.15.16:0/3808791755 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f52e00505b0 con 0x7f52f0158a10 2023-05-22T15:42:04.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.520+0000 7f52f522f640 1 -- 172.21.15.16:0/3808791755 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f52f004dbe0 con 0x7f52f0158a10 2023-05-22T15:42:04.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.520+0000 7f52e77fe640 1 -- 172.21.15.16:0/3808791755 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v76) v1 ==== 92+0+10 (secure 0 0 0) 0x7f52e001e020 con 0x7f52f0158a10 2023-05-22T15:42:04.525 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 8 2023-05-22T15:42:04.527 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.524+0000 7f52f522f640 1 -- 172.21.15.16:0/3808791755 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f52d00654a0 msgr2=0x7f52d0067950 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-05-22T15:42:04.527 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.524+0000 7f52f522f640 1 --2- 172.21.15.16:0/3808791755 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f52d00654a0 0x7f52d0067950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:04.527 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.524+0000 7f52f522f640 1 -- 172.21.15.16:0/3808791755 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f52f0158a10 msgr2=0x7f52f0157900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:04.527 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.524+0000 7f52f522f640 1 --2- 172.21.15.16:0/3808791755 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f52f0158a10 0x7f52f0157900 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7f52e000c7f0 tx=0x7f52e000ccb0 comp rx=0 tx=0).stop 2023-05-22T15:42:04.527 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.524+0000 7f52f522f640 1 -- 172.21.15.16:0/3808791755 shutdown_connections 2023-05-22T15:42:04.527 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.524+0000 7f52f522f640 1 --2- 172.21.15.16:0/3808791755 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f52f0158a10 0x7f52f0157900 unknown :-1 s=CLOSED pgs=989 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:04.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.524+0000 7f52f522f640 1 --2- 172.21.15.16:0/3808791755 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f52f0070760 0x7f52f00b42d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:04.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.524+0000 7f52f522f640 1 --2- 172.21.15.16:0/3808791755 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f52d00654a0 0x7f52d0067950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:04.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.524+0000 7f52f522f640 1 --2- 172.21.15.16:0/3808791755 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f52f006c6e0 0x7f52f014f1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:04.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.524+0000 7f52f522f640 1 -- 172.21.15.16:0/3808791755 >> 172.21.15.16:0/3808791755 conn(0x7f52f00cc870 msgr2=0x7f52f00cad10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:04.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.524+0000 7f52f522f640 1 -- 172.21.15.16:0/3808791755 shutdown_connections 2023-05-22T15:42:04.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.524+0000 7f52f522f640 1 -- 172.21.15.16:0/3808791755 wait complete. 2023-05-22T15:42:04.543 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-05-22T15:42:04.596 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:42:04.591+0000 7f1b042b5180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-05-22T15:42:04.632 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:42:04.628+0000 7fe4172e7180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-05-22T15:42:04.649 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:42:04.647+0000 7f1b042b5180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-05-22T15:42:04.663 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.660+0000 7feb0f446640 1 Processor -- start 2023-05-22T15:42:04.663 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.660+0000 7feb0f446640 1 -- start start 2023-05-22T15:42:04.664 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.660+0000 7feb0f446640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7feb080ade80 0x7feb080cc400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:04.665 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.660+0000 7feb0f446640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7feb0815c7c0 0x7feb081593b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:04.665 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.660+0000 7feb0f446640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb080c95f0 con 0x7feb0815c350 2023-05-22T15:42:04.665 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.660+0000 7feb0f446640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb080ca270 con 0x7feb0815c7c0 2023-05-22T15:42:04.665 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.660+0000 7feb0f446640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb080ca690 con 0x7feb080ade80 2023-05-22T15:42:04.665 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.660+0000 7feb0d9e3640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7feb0815c7c0 0x7feb081593b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:04.665 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.660+0000 7feb0d9e3640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7feb0815c7c0 0x7feb081593b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50766/0 (socket says 172.21.15.16:50766) 2023-05-22T15:42:04.665 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.660+0000 7feb0d9e3640 1 -- 172.21.15.16:0/246302563 learned_addr learned my addr 172.21.15.16:0/246302563 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:42:04.666 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.660+0000 7feb0d1e2640 1 --2- 172.21.15.16:0/246302563 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7feb080ade80 0x7feb080cc400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:04.666 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.660+0000 7feb0d9e3640 1 -- 172.21.15.16:0/246302563 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7feb080ade80 msgr2=0x7feb080cc400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:42:04.666 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.660+0000 7feb0d9e3640 1 --2- 172.21.15.16:0/246302563 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7feb080ade80 0x7feb080cc400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:04.666 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.660+0000 7feb0d9e3640 1 -- 172.21.15.16:0/246302563 >> v1:172.21.15.62:6789/0 conn(0x7feb0815c350 legacy=0x7feb08158ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:42:04.666 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.660+0000 7feb0d9e3640 1 -- 172.21.15.16:0/246302563 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb080c98d0 con 0x7feb0815c7c0 2023-05-22T15:42:04.668 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.664+0000 7feb0d9e3640 1 --2- 172.21.15.16:0/246302563 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feb0815c7c0 0x7feb081593b0 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7feb00009e80 tx=0x7feb0001c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e525bdcf8d0dc9aa server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:04.668 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.664+0000 7feafffff640 1 -- 172.21.15.16:0/246302563 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb000a5770 con 0x7feb0815c7c0 2023-05-22T15:42:04.668 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.664+0000 7feafffff640 1 -- 172.21.15.16:0/246302563 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb000a58d0 con 0x7feb0815c7c0 2023-05-22T15:42:04.669 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.664+0000 7feafffff640 1 -- 172.21.15.16:0/246302563 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb000a5c60 con 0x7feb0815c7c0 2023-05-22T15:42:04.669 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.664+0000 7feb0f446640 1 -- 172.21.15.16:0/246302563 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feb0815c7c0 msgr2=0x7feb081593b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:04.669 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.664+0000 7feb0f446640 1 --2- 172.21.15.16:0/246302563 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feb0815c7c0 0x7feb081593b0 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7feb00009e80 tx=0x7feb0001c8f0 comp rx=0 tx=0).stop 2023-05-22T15:42:04.670 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.664+0000 7feb0f446640 1 -- 172.21.15.16:0/246302563 shutdown_connections 2023-05-22T15:42:04.670 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.664+0000 7feb0f446640 1 --2- 172.21.15.16:0/246302563 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feb0815c7c0 0x7feb081593b0 unknown :-1 s=CLOSED pgs=990 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:04.670 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.664+0000 7feb0f446640 1 --2- 172.21.15.16:0/246302563 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7feb080ade80 0x7feb080cc400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:04.670 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.664+0000 7feb0f446640 1 -- 172.21.15.16:0/246302563 >> 172.21.15.16:0/246302563 conn(0x7feb080b1110 msgr2=0x7feb080ce830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:04.670 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.664+0000 7feb0f446640 1 -- 172.21.15.16:0/246302563 shutdown_connections 2023-05-22T15:42:04.670 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.664+0000 7feb0f446640 1 -- 172.21.15.16:0/246302563 wait complete. 2023-05-22T15:42:04.671 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.668+0000 7feb0f446640 1 Processor -- start 2023-05-22T15:42:04.671 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.668+0000 7feb0f446640 1 -- start start 2023-05-22T15:42:04.672 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.668+0000 7feb0f446640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7feb080ade80 0x7feb08148050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:04.673 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.668+0000 7feb0f446640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feb0815c350 0x7feb0814f9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:04.673 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.668+0000 7feb0f446640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7feb0815c7c0 0x7feb08151100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:04.673 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.668+0000 7feb0f446640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb08150080 con 0x7feb0815c350 2023-05-22T15:42:04.673 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.668+0000 7feb0f446640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb08150360 con 0x7feb080ade80 2023-05-22T15:42:04.673 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.668+0000 7feb0f446640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb08150640 con 0x7feb0815c7c0 2023-05-22T15:42:04.673 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.668+0000 7feb0c9e1640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feb0815c350 0x7feb0814f9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:04.673 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.668+0000 7feb0c9e1640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feb0815c350 0x7feb0814f9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50782/0 (socket says 172.21.15.16:50782) 2023-05-22T15:42:04.673 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.668+0000 7feb0c9e1640 1 -- 172.21.15.16:0/3543795305 learned_addr learned my addr 172.21.15.16:0/3543795305 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:42:04.674 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.668+0000 7feb0d1e2640 1 --2- 172.21.15.16:0/3543795305 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7feb080ade80 0x7feb08148050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:04.674 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.668+0000 7feb0d9e3640 1 --2- 172.21.15.16:0/3543795305 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7feb0815c7c0 0x7feb08151100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:04.674 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.668+0000 7feb0c9e1640 1 -- 172.21.15.16:0/3543795305 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7feb0815c7c0 msgr2=0x7feb08151100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:04.674 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.668+0000 7feb0c9e1640 1 --2- 172.21.15.16:0/3543795305 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7feb0815c7c0 0x7feb08151100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:04.674 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.668+0000 7feb0c9e1640 1 -- 172.21.15.16:0/3543795305 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7feb080ade80 msgr2=0x7feb08148050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:04.674 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.668+0000 7feb0c9e1640 1 --2- 172.21.15.16:0/3543795305 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7feb080ade80 0x7feb08148050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:04.674 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.668+0000 7feb0c9e1640 1 -- 172.21.15.16:0/3543795305 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb00009b30 con 0x7feb0815c350 2023-05-22T15:42:04.675 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.668+0000 7feb0d9e3640 1 --2- 172.21.15.16:0/3543795305 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7feb0815c7c0 0x7feb08151100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T15:42:04.675 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.668+0000 7feb0c9e1640 1 --2- 172.21.15.16:0/3543795305 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feb0815c350 0x7feb0814f9a0 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7feb0400b410 tx=0x7feb0400b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:04.675 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.668+0000 7feafdffb640 1 -- 172.21.15.16:0/3543795305 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb04004020 con 0x7feb0815c350 2023-05-22T15:42:04.675 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.668+0000 7feafdffb640 1 -- 172.21.15.16:0/3543795305 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb04004180 con 0x7feb0815c350 2023-05-22T15:42:04.675 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.668+0000 7feafdffb640 1 -- 172.21.15.16:0/3543795305 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb04011630 con 0x7feb0815c350 2023-05-22T15:42:04.675 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.668+0000 7feb0f446640 1 -- 172.21.15.16:0/3543795305 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb080cc940 con 0x7feb0815c350 2023-05-22T15:42:04.675 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.668+0000 7feb0f446640 1 -- 172.21.15.16:0/3543795305 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb0815cc00 con 0x7feb0815c350 2023-05-22T15:42:04.676 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.668+0000 7feb0f446640 1 -- 172.21.15.16:0/3543795305 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb0815d090 con 0x7feb0815c350 2023-05-22T15:42:04.676 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.668+0000 7feafdffb640 1 -- 172.21.15.16:0/3543795305 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 23) v1 ==== 84344+0+0 (secure 0 0 0) 0x7feb04011790 con 0x7feb0815c350 2023-05-22T15:42:04.676 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.672+0000 7feafdffb640 1 --2- 172.21.15.16:0/3543795305 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7feaec0654a0 0x7feaec067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:04.676 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.672+0000 7feb0d1e2640 1 -- 172.21.15.16:0/3543795305 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7feaec0654a0 msgr2=0x7feaec067950 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/36921 2023-05-22T15:42:04.677 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.672+0000 7feafdffb640 1 -- 172.21.15.16:0/3543795305 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 6798+0+0 (secure 0 0 0) 0x7feb04082960 con 0x7feb0815c350 2023-05-22T15:42:04.677 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.672+0000 7feb0d1e2640 1 --2- 172.21.15.16:0/3543795305 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7feaec0654a0 0x7feaec067950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-05-22T15:42:04.678 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.676+0000 7feafdffb640 1 -- 172.21.15.16:0/3543795305 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7feb040501f0 con 0x7feb0815c350 2023-05-22T15:42:04.691 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:42:04.688+0000 7fe4172e7180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-05-22T15:42:04.699 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:42:04.695+0000 7f1b042b5180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-05-22T15:42:04.743 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:42:04.740+0000 7fe4172e7180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-05-22T15:42:04.777 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:42:04.775+0000 7f1b042b5180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-05-22T15:42:04.822 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:42:04.816+0000 7fe4172e7180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-05-22T15:42:04.826 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.824+0000 7feb0f446640 1 -- 172.21.15.16:0/3543795305 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7feb0804dbe0 con 0x7feb0815c350 2023-05-22T15:42:04.827 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.824+0000 7feafdffb640 1 -- 172.21.15.16:0/3543795305 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v76) v1 ==== 104+0+11 (secure 0 0 0) 0x7feb04053810 con 0x7feb0815c350 2023-05-22T15:42:04.827 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2023-05-22T15:42:04.827 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:42:04.823+0000 7f1b042b5180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-05-22T15:42:04.829 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.824+0000 7feb0f446640 1 -- 172.21.15.16:0/3543795305 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7feaec0654a0 msgr2=0x7feaec067950 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-05-22T15:42:04.829 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.824+0000 7feb0f446640 1 --2- 172.21.15.16:0/3543795305 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7feaec0654a0 0x7feaec067950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:04.829 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.824+0000 7feb0f446640 1 -- 172.21.15.16:0/3543795305 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feb0815c350 msgr2=0x7feb0814f9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:04.829 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.824+0000 7feb0f446640 1 --2- 172.21.15.16:0/3543795305 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feb0815c350 0x7feb0814f9a0 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7feb0400b410 tx=0x7feb0400b8d0 comp rx=0 tx=0).stop 2023-05-22T15:42:04.830 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.824+0000 7feb0f446640 1 -- 172.21.15.16:0/3543795305 shutdown_connections 2023-05-22T15:42:04.830 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.824+0000 7feb0f446640 1 --2- 172.21.15.16:0/3543795305 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7feaec0654a0 0x7feaec067950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:04.830 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.824+0000 7feb0f446640 1 --2- 172.21.15.16:0/3543795305 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7feb0815c7c0 0x7feb08151100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:04.830 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.824+0000 7feb0f446640 1 --2- 172.21.15.16:0/3543795305 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feb0815c350 0x7feb0814f9a0 unknown :-1 s=CLOSED pgs=991 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:04.830 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.824+0000 7feb0f446640 1 --2- 172.21.15.16:0/3543795305 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7feb080ade80 0x7feb08148050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:04.830 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.824+0000 7feb0f446640 1 -- 172.21.15.16:0/3543795305 >> 172.21.15.16:0/3543795305 conn(0x7feb080b1110 msgr2=0x7feb08158ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:04.831 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.824+0000 7feb0f446640 1 -- 172.21.15.16:0/3543795305 shutdown_connections 2023-05-22T15:42:04.831 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.824+0000 7feb0f446640 1 -- 172.21.15.16:0/3543795305 wait complete. 2023-05-22T15:42:04.845 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-05-22T15:42:04.875 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:42:04.872+0000 7fe4172e7180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-05-22T15:42:04.903 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:42:04.899+0000 7f1b042b5180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-05-22T15:42:04.955 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:42:04.952+0000 7fe4172e7180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-05-22T15:42:04.955 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:42:04.951+0000 7f1b042b5180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-05-22T15:42:04.965 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.960+0000 7f3e8d15d640 1 Processor -- start 2023-05-22T15:42:04.965 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.960+0000 7f3e8d15d640 1 -- start start 2023-05-22T15:42:04.966 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.964+0000 7f3e8d15d640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f3e8815c8b0 0x7f3e880cee30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:04.966 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.964+0000 7f3e8d15d640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e8806c3e0 con 0x7f3e8806bf60 2023-05-22T15:42:04.966 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.964+0000 7f3e8d15d640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e8806c640 con 0x7f3e8806db80 2023-05-22T15:42:04.966 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.964+0000 7f3e8d15d640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e8815a0c0 con 0x7f3e8815c8b0 2023-05-22T15:42:04.966 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.964+0000 7f3e8659c640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f3e8806bf60 0x7f3e8815cc80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:54280/0 (socket says 172.21.15.16:54280) 2023-05-22T15:42:04.967 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.964+0000 7f3e8659c640 1 -- 172.21.15.16:0/958863088 learned_addr learned my addr 172.21.15.16:0/958863088 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:42:04.968 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.964+0000 7f3e86d9d640 1 --2- 172.21.15.16:0/958863088 >> v2:172.21.15.62:3300/0 conn(0x7f3e8815c8b0 0x7f3e880cee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:04.968 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.964+0000 7f3e85d9b640 1 -- 172.21.15.16:0/958863088 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 672166810 0 0) 0x7f3e8806c3e0 con 0x7f3e8806bf60 2023-05-22T15:42:04.968 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.964+0000 7f3e85d9b640 1 -- 172.21.15.16:0/958863088 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e70003530 con 0x7f3e8806bf60 2023-05-22T15:42:04.968 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.964+0000 7f3e85d9b640 1 -- 172.21.15.16:0/958863088 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1224717814 0 0) 0x7f3e70003530 con 0x7f3e8806bf60 2023-05-22T15:42:04.969 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.964+0000 7f3e85d9b640 1 -- 172.21.15.16:0/958863088 >> v2:172.21.15.62:3300/0 conn(0x7f3e8815c8b0 msgr2=0x7f3e880cee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:42:04.969 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.964+0000 7f3e85d9b640 1 --2- 172.21.15.16:0/958863088 >> v2:172.21.15.62:3300/0 conn(0x7f3e8815c8b0 0x7f3e880cee30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:04.969 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.964+0000 7f3e85d9b640 1 -- 172.21.15.16:0/958863088 >> v1:172.21.15.62:6789/0 conn(0x7f3e8806db80 legacy=0x7f3e880cf8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:04.969 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.964+0000 7f3e85d9b640 1 -- 172.21.15.16:0/958863088 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e880ce370 con 0x7f3e8806bf60 2023-05-22T15:42:04.969 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.964+0000 7f3e85d9b640 1 -- 172.21.15.16:0/958863088 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7f3e7c001680 con 0x7f3e8806bf60 2023-05-22T15:42:04.969 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.964+0000 7f3e85d9b640 1 -- 172.21.15.16:0/958863088 >> v1:172.21.15.16:6789/0 conn(0x7f3e8806bf60 legacy=0x7f3e8815cc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:04.969 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.964+0000 7f3e85d9b640 1 --2- 172.21.15.16:0/958863088 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3e70003980 0x7f3e70003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:04.970 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.964+0000 7f3e85d9b640 1 --2- 172.21.15.16:0/958863088 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3e700047d0 0x7f3e70004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:04.970 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.964+0000 7f3e85d9b640 1 --2- 172.21.15.16:0/958863088 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3e70005180 0x7f3e7000b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:04.970 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.964+0000 7f3e85d9b640 1 -- 172.21.15.16:0/958863088 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e8815a0c0 con 0x7f3e70003980 2023-05-22T15:42:04.970 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.964+0000 7f3e85d9b640 1 -- 172.21.15.16:0/958863088 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e7000c000 con 0x7f3e700047d0 2023-05-22T15:42:04.970 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.964+0000 7f3e85d9b640 1 -- 172.21.15.16:0/958863088 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e7000c1c0 con 0x7f3e70005180 2023-05-22T15:42:04.970 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.964+0000 7f3e86d9d640 1 --2- 172.21.15.16:0/958863088 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3e70003980 0x7f3e70003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:04.970 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.964+0000 7f3e8759e640 1 --2- 172.21.15.16:0/958863088 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3e70005180 0x7f3e7000b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:04.970 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.964+0000 7f3e8659c640 1 --2- 172.21.15.16:0/958863088 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3e700047d0 0x7f3e70004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:04.971 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.964+0000 7f3e86d9d640 1 -- 172.21.15.16:0/958863088 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3e70005180 msgr2=0x7f3e7000b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:42:04.971 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.964+0000 7f3e86d9d640 1 --2- 172.21.15.16:0/958863088 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3e70005180 0x7f3e7000b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:04.971 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.964+0000 7f3e86d9d640 1 -- 172.21.15.16:0/958863088 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3e700047d0 msgr2=0x7f3e70004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:42:04.971 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.964+0000 7f3e86d9d640 1 --2- 172.21.15.16:0/958863088 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3e700047d0 0x7f3e70004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:04.971 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.964+0000 7f3e86d9d640 1 -- 172.21.15.16:0/958863088 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e70003530 con 0x7f3e70003980 2023-05-22T15:42:04.971 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.964+0000 7f3e8759e640 1 -- 172.21.15.16:0/958863088 reap_dead start 2023-05-22T15:42:04.971 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.964+0000 7f3e8659c640 1 --2- 172.21.15.16:0/958863088 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3e700047d0 0x7f3e70004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:42:04.973 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.964+0000 7f3e86d9d640 1 --2- 172.21.15.16:0/958863088 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3e70003980 0x7f3e70003d90 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7f3e78009ae0 tx=0x7f3e7801ca70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=727f233b27c91dbe server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:04.974 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.964+0000 7f3e85d9b640 1 -- 172.21.15.16:0/958863088 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e780a9470 con 0x7f3e70003980 2023-05-22T15:42:04.974 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.964+0000 7f3e85d9b640 1 -- 172.21.15.16:0/958863088 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e780a95d0 con 0x7f3e70003980 2023-05-22T15:42:04.974 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.964+0000 7f3e85d9b640 1 -- 172.21.15.16:0/958863088 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e780a9980 con 0x7f3e70003980 2023-05-22T15:42:04.974 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.968+0000 7f3e8d15d640 1 -- 172.21.15.16:0/958863088 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3e70003980 msgr2=0x7f3e70003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:04.974 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.968+0000 7f3e8d15d640 1 --2- 172.21.15.16:0/958863088 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3e70003980 0x7f3e70003d90 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7f3e78009ae0 tx=0x7f3e7801ca70 comp rx=0 tx=0).stop 2023-05-22T15:42:04.974 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.968+0000 7f3e8d15d640 1 -- 172.21.15.16:0/958863088 shutdown_connections 2023-05-22T15:42:04.974 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.968+0000 7f3e8d15d640 1 --2- 172.21.15.16:0/958863088 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3e70003980 0x7f3e70003d90 unknown :-1 s=CLOSED pgs=993 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:04.974 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.968+0000 7f3e8d15d640 1 -- 172.21.15.16:0/958863088 >> 172.21.15.16:0/958863088 conn(0x7f3e8806f5a0 msgr2=0x7f3e8806f9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:04.975 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.968+0000 7f3e8d15d640 1 -- 172.21.15.16:0/958863088 shutdown_connections 2023-05-22T15:42:04.975 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.968+0000 7f3e8d15d640 1 -- 172.21.15.16:0/958863088 wait complete. 2023-05-22T15:42:04.975 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.968+0000 7f3e8d15d640 1 Processor -- start 2023-05-22T15:42:04.975 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.968+0000 7f3e8d15d640 1 -- start start 2023-05-22T15:42:04.975 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.968+0000 7f3e8d15d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3e70003980 0x7f3e880c9fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:04.975 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.968+0000 7f3e8d15d640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3e881f9120 0x7f3e880c8d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:04.975 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.968+0000 7f3e8d15d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3e881f8510 0x7f3e881f8980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:04.976 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.968+0000 7f3e8d15d640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e881f8ec0 con 0x7f3e881f8510 2023-05-22T15:42:04.976 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.968+0000 7f3e8d15d640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e880c6e60 con 0x7f3e70003980 2023-05-22T15:42:04.976 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.968+0000 7f3e8d15d640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e880c6250 con 0x7f3e881f9120 2023-05-22T15:42:04.976 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.968+0000 7f3e8759e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3e881f8510 0x7f3e881f8980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:04.976 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.968+0000 7f3e8759e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3e881f8510 0x7f3e881f8980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50804/0 (socket says 172.21.15.16:50804) 2023-05-22T15:42:04.976 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.968+0000 7f3e8759e640 1 -- 172.21.15.16:0/4280173884 learned_addr learned my addr 172.21.15.16:0/4280173884 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:42:04.976 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.968+0000 7f3e86d9d640 1 --2- 172.21.15.16:0/4280173884 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3e70003980 0x7f3e880c9fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:04.976 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.968+0000 7f3e8659c640 1 --2- 172.21.15.16:0/4280173884 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3e881f9120 0x7f3e880c8d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:04.977 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.968+0000 7f3e8759e640 1 -- 172.21.15.16:0/4280173884 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3e881f9120 msgr2=0x7f3e880c8d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:04.977 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.968+0000 7f3e8759e640 1 --2- 172.21.15.16:0/4280173884 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3e881f9120 0x7f3e880c8d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:04.977 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.968+0000 7f3e8759e640 1 -- 172.21.15.16:0/4280173884 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3e70003980 msgr2=0x7f3e880c9fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:04.977 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.968+0000 7f3e8759e640 1 --2- 172.21.15.16:0/4280173884 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3e70003980 0x7f3e880c9fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:04.977 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.968+0000 7f3e8759e640 1 -- 172.21.15.16:0/4280173884 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e8806e360 con 0x7f3e881f8510 2023-05-22T15:42:04.977 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.968+0000 7f3e8759e640 1 --2- 172.21.15.16:0/4280173884 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3e881f8510 0x7f3e881f8980 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7f3e8806ac70 tx=0x7f3e8000e840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:04.977 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.968+0000 7f3e6f7fe640 1 -- 172.21.15.16:0/4280173884 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e8000ed00 con 0x7f3e881f8510 2023-05-22T15:42:04.978 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.968+0000 7f3e6f7fe640 1 -- 172.21.15.16:0/4280173884 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e8000eea0 con 0x7f3e881f8510 2023-05-22T15:42:04.978 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.968+0000 7f3e6f7fe640 1 -- 172.21.15.16:0/4280173884 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e80003bc0 con 0x7f3e881f8510 2023-05-22T15:42:04.978 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.972+0000 7f3e8d15d640 1 -- 172.21.15.16:0/4280173884 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e8815d490 con 0x7f3e881f8510 2023-05-22T15:42:04.979 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.972+0000 7f3e8d15d640 1 -- 172.21.15.16:0/4280173884 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e8806e4f0 con 0x7f3e881f8510 2023-05-22T15:42:04.979 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.972+0000 7f3e8d15d640 1 -- 172.21.15.16:0/4280173884 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e8804dbe0 con 0x7f3e881f8510 2023-05-22T15:42:04.979 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.972+0000 7f3e6f7fe640 1 -- 172.21.15.16:0/4280173884 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 23) v1 ==== 84344+0+0 (secure 0 0 0) 0x7f3e80003d20 con 0x7f3e881f8510 2023-05-22T15:42:04.979 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.972+0000 7f3e6f7fe640 1 --2- 172.21.15.16:0/4280173884 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f3e740654d0 0x7f3e74067980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:04.979 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.972+0000 7f3e86d9d640 1 -- 172.21.15.16:0/4280173884 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f3e740654d0 msgr2=0x7f3e74067980 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/36921 2023-05-22T15:42:04.979 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.972+0000 7f3e6f7fe640 1 -- 172.21.15.16:0/4280173884 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 6798+0+0 (secure 0 0 0) 0x7f3e80081d10 con 0x7f3e881f8510 2023-05-22T15:42:04.979 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.972+0000 7f3e86d9d640 1 --2- 172.21.15.16:0/4280173884 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f3e740654d0 0x7f3e74067980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-05-22T15:42:04.980 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:04.976+0000 7f3e6f7fe640 1 -- 172.21.15.16:0/4280173884 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f3e8004c620 con 0x7f3e881f8510 2023-05-22T15:42:05.009 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:42:05.004+0000 7fe4172e7180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-05-22T15:42:05.092 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:42:05.087+0000 7f1b042b5180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-05-22T15:42:05.136 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:05.132+0000 7f3e8d15d640 1 -- 172.21.15.16:0/4280173884 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f3e880c6fa0 con 0x7f3e881f8510 2023-05-22T15:42:05.137 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:05.132+0000 7f3e6f7fe640 1 -- 172.21.15.16:0/4280173884 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v76) v1 ==== 100+0+11 (secure 0 0 0) 0x7f3e80019020 con 0x7f3e881f8510 2023-05-22T15:42:05.137 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 64 2023-05-22T15:42:05.139 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:05.136+0000 7f3e8d15d640 1 -- 172.21.15.16:0/4280173884 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f3e740654d0 msgr2=0x7f3e74067980 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-05-22T15:42:05.140 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:05.136+0000 7f3e8d15d640 1 --2- 172.21.15.16:0/4280173884 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f3e740654d0 0x7f3e74067980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:05.140 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:05.136+0000 7f3e8d15d640 1 -- 172.21.15.16:0/4280173884 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3e881f8510 msgr2=0x7f3e881f8980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:05.140 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:05.136+0000 7f3e8d15d640 1 --2- 172.21.15.16:0/4280173884 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3e881f8510 0x7f3e881f8980 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7f3e8806ac70 tx=0x7f3e8000e840 comp rx=0 tx=0).stop 2023-05-22T15:42:05.141 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:05.136+0000 7f3e8d15d640 1 -- 172.21.15.16:0/4280173884 shutdown_connections 2023-05-22T15:42:05.141 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:05.136+0000 7f3e8d15d640 1 --2- 172.21.15.16:0/4280173884 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3e881f8510 0x7f3e881f8980 unknown :-1 s=CLOSED pgs=994 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:05.141 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:05.136+0000 7f3e8d15d640 1 --2- 172.21.15.16:0/4280173884 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f3e740654d0 0x7f3e74067980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:05.142 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:05.136+0000 7f3e8d15d640 1 --2- 172.21.15.16:0/4280173884 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3e881f9120 0x7f3e880c8d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:05.142 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:05.136+0000 7f3e8d15d640 1 --2- 172.21.15.16:0/4280173884 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3e70003980 0x7f3e880c9fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:05.142 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:05.136+0000 7f3e8d15d640 1 -- 172.21.15.16:0/4280173884 >> 172.21.15.16:0/4280173884 conn(0x7f3e8806f5a0 msgr2=0x7f3e880b1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:05.143 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:05.136+0000 7f3e8d15d640 1 -- 172.21.15.16:0/4280173884 shutdown_connections 2023-05-22T15:42:05.143 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:05.136+0000 7f3e8d15d640 1 -- 172.21.15.16:0/4280173884 wait complete. 2023-05-22T15:42:05.152 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:42:05.148+0000 7fe4172e7180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-05-22T15:42:05.156 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-05-22T15:42:05.157 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:42:05.151+0000 7f1b042b5180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-05-22T15:42:05.207 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:42:05.204+0000 7fe4172e7180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-05-22T15:42:05.311 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:05.308+0000 7f0aeb6f2640 1 Processor -- start 2023-05-22T15:42:05.311 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:05.308+0000 7f0aeb6f2640 1 -- start start 2023-05-22T15:42:05.312 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:05.308+0000 7f0aeb6f2640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0ae41563c0 0x7f0ae40c7b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:05.312 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:05.308+0000 7f0aeb6f2640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f0ae40cc6c0 0x7f0ae40c8080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:05.314 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:05.308+0000 7f0aeb6f2640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ae415c210 con 0x7f0ae406c220 2023-05-22T15:42:05.314 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:05.308+0000 7f0aeb6f2640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ae4160700 con 0x7f0ae40cc6c0 2023-05-22T15:42:05.314 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:05.308+0000 7f0aeb6f2640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ae406cdd0 con 0x7f0ae41563c0 2023-05-22T15:42:05.314 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:05.308+0000 7f0ae948e640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0ae41563c0 0x7f0ae40c7b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:05.314 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:05.308+0000 7f0ae8c8d640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f0ae40cc6c0 0x7f0ae40c8080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:05.314 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:05.308+0000 7f0ae9c8f640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f0ae406c220 0x7f0ae406c6c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.16:57128/0 (socket says 172.21.15.16:57128) 2023-05-22T15:42:05.315 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:05.308+0000 7f0ae9c8f640 1 -- 172.21.15.16:0/754519428 learned_addr learned my addr 172.21.15.16:0/754519428 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:42:05.316 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:05.308+0000 7f0ae8c8d640 1 -- 172.21.15.16:0/754519428 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0ae41563c0 msgr2=0x7f0ae40c7b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:42:05.316 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:05.308+0000 7f0ae8c8d640 1 --2- 172.21.15.16:0/754519428 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0ae41563c0 0x7f0ae40c7b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:05.316 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:05.308+0000 7f0ae8c8d640 1 -- 172.21.15.16:0/754519428 >> v1:172.21.15.62:6789/0 conn(0x7f0ae406c220 legacy=0x7f0ae406c6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:05.317 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:05.308+0000 7f0ae8c8d640 1 -- 172.21.15.16:0/754519428 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ae415c480 con 0x7f0ae40cc6c0 2023-05-22T15:42:05.317 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:05.308+0000 7f0ae948e640 1 --2- 172.21.15.16:0/754519428 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0ae41563c0 0x7f0ae40c7b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-05-22T15:42:05.317 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:05.312+0000 7f0ae8c8d640 1 --2- 172.21.15.16:0/754519428 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0ae40cc6c0 0x7f0ae40c8080 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f0ad80097c0 tx=0x7f0ad801c450 comp rx=0 tx=0).ready entity=mon.1 client_cookie=512e7ae04051c044 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:05.317 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:05.312+0000 7f0ad7fff640 1 -- 172.21.15.16:0/754519428 <== mon.1 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ad80ab650 con 0x7f0ae40cc6c0 2023-05-22T15:42:05.317 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:05.312+0000 7f0ad7fff640 1 -- 172.21.15.16:0/754519428 <== mon.1 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ad80ab7b0 con 0x7f0ae40cc6c0 2023-05-22T15:42:05.317 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:05.312+0000 7f0ad7fff640 1 -- 172.21.15.16:0/754519428 <== mon.1 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ad80abb40 con 0x7f0ae40cc6c0 2023-05-22T15:42:05.317 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:05.312+0000 7f0aeb6f2640 1 -- 172.21.15.16:0/754519428 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0ae40cc6c0 msgr2=0x7f0ae40c8080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:05.317 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:05.312+0000 7f0aeb6f2640 1 --2- 172.21.15.16:0/754519428 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0ae40cc6c0 0x7f0ae40c8080 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f0ad80097c0 tx=0x7f0ad801c450 comp rx=0 tx=0).stop 2023-05-22T15:42:05.318 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:05.312+0000 7f0aeb6f2640 1 -- 172.21.15.16:0/754519428 shutdown_connections 2023-05-22T15:42:05.318 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:05.312+0000 7f0aeb6f2640 1 --2- 172.21.15.16:0/754519428 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0ae40cc6c0 0x7f0ae40c8080 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:05.318 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:05.312+0000 7f0aeb6f2640 1 --2- 172.21.15.16:0/754519428 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0ae41563c0 0x7f0ae40c7b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:05.318 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:05.312+0000 7f0aeb6f2640 1 -- 172.21.15.16:0/754519428 >> 172.21.15.16:0/754519428 conn(0x7f0ae40b1110 msgr2=0x7f0ae40ce830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:05.318 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:05.312+0000 7f0aeb6f2640 1 -- 172.21.15.16:0/754519428 shutdown_connections 2023-05-22T15:42:05.318 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:05.312+0000 7f0aeb6f2640 1 -- 172.21.15.16:0/754519428 wait complete. 2023-05-22T15:42:05.321 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:05.312+0000 7f0aeb6f2640 1 Processor -- start 2023-05-22T15:42:05.321 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:05.312+0000 7f0aeb6f2640 1 -- start start 2023-05-22T15:42:05.322 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:05.312+0000 7f0aeb6f2640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0ae406c220 0x7f0ae4151a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:05.322 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:05.312+0000 7f0aeb6f2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ae40cc6c0 0x7f0ae41447f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:05.322 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:05.312+0000 7f0aeb6f2640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0ae41507e0 0x7f0ae40c8e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:05.322 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:05.312+0000 7f0aeb6f2640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ae40b02f0 con 0x7f0ae40cc6c0 2023-05-22T15:42:05.322 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:05.312+0000 7f0aeb6f2640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ae40b0010 con 0x7f0ae406c220 2023-05-22T15:42:05.322 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:05.312+0000 7f0aeb6f2640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ae414fd00 con 0x7f0ae41507e0 2023-05-22T15:42:05.323 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:05.312+0000 7f0ae8c8d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ae40cc6c0 0x7f0ae41447f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:05.323 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:05.316+0000 7f0ae8c8d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ae40cc6c0 0x7f0ae41447f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50808/0 (socket says 172.21.15.16:50808) 2023-05-22T15:42:05.323 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:05.316+0000 7f0ae8c8d640 1 -- 172.21.15.16:0/4263058985 learned_addr learned my addr 172.21.15.16:0/4263058985 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:42:05.323 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:05.316+0000 7f0ae948e640 1 --2- 172.21.15.16:0/4263058985 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0ae406c220 0x7f0ae4151a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:05.323 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:05.316+0000 7f0ae9c8f640 1 --2- 172.21.15.16:0/4263058985 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0ae41507e0 0x7f0ae40c8e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:05.324 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:05.316+0000 7f0ae8c8d640 1 -- 172.21.15.16:0/4263058985 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0ae41507e0 msgr2=0x7f0ae40c8e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:05.324 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:05.316+0000 7f0ae8c8d640 1 --2- 172.21.15.16:0/4263058985 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0ae41507e0 0x7f0ae40c8e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:05.324 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:05.316+0000 7f0ae8c8d640 1 -- 172.21.15.16:0/4263058985 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0ae406c220 msgr2=0x7f0ae4151a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:05.324 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:05.316+0000 7f0ae8c8d640 1 --2- 172.21.15.16:0/4263058985 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0ae406c220 0x7f0ae4151a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:05.324 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:05.316+0000 7f0ae8c8d640 1 -- 172.21.15.16:0/4263058985 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ae0009420 con 0x7f0ae40cc6c0 2023-05-22T15:42:05.324 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:05.316+0000 7f0ae8c8d640 1 --2- 172.21.15.16:0/4263058985 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ae40cc6c0 0x7f0ae41447f0 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7f0ad80ad040 tx=0x7f0ad801ac90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:05.325 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:05.316+0000 7f0ad5ffb640 1 -- 172.21.15.16:0/4263058985 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ad80bf610 con 0x7f0ae40cc6c0 2023-05-22T15:42:05.325 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:05.316+0000 7f0ad5ffb640 1 -- 172.21.15.16:0/4263058985 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ad80bf790 con 0x7f0ae40cc6c0 2023-05-22T15:42:05.325 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:05.316+0000 7f0aeb6f2640 1 -- 172.21.15.16:0/4263058985 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ad8009420 con 0x7f0ae40cc6c0 2023-05-22T15:42:05.325 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:05.316+0000 7f0ad5ffb640 1 -- 172.21.15.16:0/4263058985 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ad80bfb40 con 0x7f0ae40cc6c0 2023-05-22T15:42:05.325 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:05.316+0000 7f0aeb6f2640 1 -- 172.21.15.16:0/4263058985 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ae415ff00 con 0x7f0ae40cc6c0 2023-05-22T15:42:05.325 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:05.316+0000 7f0aeb6f2640 1 -- 172.21.15.16:0/4263058985 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ae415cd20 con 0x7f0ae40cc6c0 2023-05-22T15:42:05.325 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:05.316+0000 7f0ad5ffb640 1 -- 172.21.15.16:0/4263058985 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 23) v1 ==== 84344+0+0 (secure 0 0 0) 0x7f0ad80b5400 con 0x7f0ae40cc6c0 2023-05-22T15:42:05.326 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:05.316+0000 7f0ad5ffb640 1 --2- 172.21.15.16:0/4263058985 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f0ac80654d0 0x7f0ac8067980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:05.326 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:05.316+0000 7f0ae948e640 1 -- 172.21.15.16:0/4263058985 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f0ac80654d0 msgr2=0x7f0ac8067980 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/36921 2023-05-22T15:42:05.326 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:05.316+0000 7f0ad5ffb640 1 -- 172.21.15.16:0/4263058985 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 6798+0+0 (secure 0 0 0) 0x7f0ad811dc90 con 0x7f0ae40cc6c0 2023-05-22T15:42:05.326 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:05.316+0000 7f0ae948e640 1 --2- 172.21.15.16:0/4263058985 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f0ac80654d0 0x7f0ac8067980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-05-22T15:42:05.327 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:05.320+0000 7f0ad5ffb640 1 -- 172.21.15.16:0/4263058985 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f0ad80eb4d0 con 0x7f0ae40cc6c0 2023-05-22T15:42:05.381 INFO:tasks.ceph.mgr.x.smithi062.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2023-05-22T15:42:05.381 INFO:tasks.ceph.mgr.x.smithi062.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2023-05-22T15:42:05.382 INFO:tasks.ceph.mgr.x.smithi062.stderr: from numpy import show_config as show_numpy_config 2023-05-22T15:42:05.389 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:42:05.387+0000 7f1b042b5180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-05-22T15:42:05.433 INFO:tasks.ceph.mgr.y.smithi016.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2023-05-22T15:42:05.434 INFO:tasks.ceph.mgr.y.smithi016.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2023-05-22T15:42:05.434 INFO:tasks.ceph.mgr.y.smithi016.stderr: from numpy import show_config as show_numpy_config 2023-05-22T15:42:05.442 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:42:05.436+0000 7fe4172e7180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-05-22T15:42:05.515 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:42:05.511+0000 7f1b042b5180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-05-22T15:42:05.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:05.520+0000 7f0ae948e640 1 -- 172.21.15.16:0/4263058985 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f0ac80654d0 msgr2=0x7f0ac8067980 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/36921 2023-05-22T15:42:05.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:05.520+0000 7f0ae948e640 1 --2- 172.21.15.16:0/4263058985 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f0ac80654d0 0x7f0ac8067980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-05-22T15:42:05.539 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:05.536+0000 7f0aeb6f2640 1 -- 172.21.15.16:0/4263058985 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f0ae4160090 con 0x7f0ae40cc6c0 2023-05-22T15:42:05.540 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:05.536+0000 7f0ad5ffb640 1 -- 172.21.15.16:0/4263058985 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 8 entries v76) v1 ==== 80+0+459 (secure 0 0 0) 0x7f0ad80eeaf0 con 0x7f0ae40cc6c0 2023-05-22T15:42:05.540 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/2468042232 2023-05-23T15:40:15.816078+0000 2023-05-22T15:42:05.540 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/1582569975 2023-05-23T15:40:15.816078+0000 2023-05-22T15:42:05.541 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/2252481002 2023-05-23T15:40:15.816078+0000 2023-05-22T15:42:05.541 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/174222799 2023-05-23T15:40:15.816078+0000 2023-05-22T15:42:05.541 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/4017778835 2023-05-23T15:40:15.816078+0000 2023-05-22T15:42:05.541 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/3438310522 2023-05-23T15:40:15.816078+0000 2023-05-22T15:42:05.542 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:6833/36921 2023-05-23T15:40:15.816078+0000 2023-05-22T15:42:05.542 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:6832/36921 2023-05-23T15:40:15.816078+0000 2023-05-22T15:42:05.544 INFO:teuthology.orchestra.run.smithi016.stderr:listed 8 entries 2023-05-22T15:42:05.544 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:05.536+0000 7f0aeb6f2640 1 -- 172.21.15.16:0/4263058985 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f0ac80654d0 msgr2=0x7f0ac8067980 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-05-22T15:42:05.544 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:05.536+0000 7f0aeb6f2640 1 --2- 172.21.15.16:0/4263058985 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f0ac80654d0 0x7f0ac8067980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:05.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:05.536+0000 7f0aeb6f2640 1 -- 172.21.15.16:0/4263058985 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ae40cc6c0 msgr2=0x7f0ae41447f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:05.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:05.536+0000 7f0aeb6f2640 1 --2- 172.21.15.16:0/4263058985 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ae40cc6c0 0x7f0ae41447f0 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7f0ad80ad040 tx=0x7f0ad801ac90 comp rx=0 tx=0).stop 2023-05-22T15:42:05.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:05.540+0000 7f0aeb6f2640 1 -- 172.21.15.16:0/4263058985 shutdown_connections 2023-05-22T15:42:05.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:05.540+0000 7f0aeb6f2640 1 --2- 172.21.15.16:0/4263058985 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f0ac80654d0 0x7f0ac8067980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:05.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:05.540+0000 7f0aeb6f2640 1 --2- 172.21.15.16:0/4263058985 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0ae41507e0 0x7f0ae40c8e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:05.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:05.540+0000 7f0aeb6f2640 1 --2- 172.21.15.16:0/4263058985 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ae40cc6c0 0x7f0ae41447f0 unknown :-1 s=CLOSED pgs=995 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:05.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:05.540+0000 7f0aeb6f2640 1 --2- 172.21.15.16:0/4263058985 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0ae406c220 0x7f0ae4151a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:05.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:05.540+0000 7f0aeb6f2640 1 -- 172.21.15.16:0/4263058985 >> 172.21.15.16:0/4263058985 conn(0x7f0ae40b1110 msgr2=0x7f0ae40705f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:05.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:05.540+0000 7f0aeb6f2640 1 -- 172.21.15.16:0/4263058985 shutdown_connections 2023-05-22T15:42:05.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:05.540+0000 7f0aeb6f2640 1 -- 172.21.15.16:0/4263058985 wait complete. 2023-05-22T15:42:05.558 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi016.front.sepia.ceph.com... 2023-05-22T15:42:05.558 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:42:05.558 DEBUG:teuthology.orchestra.run.smithi016:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2023-05-22T15:42:05.569 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:42:05.564+0000 7fe4172e7180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-05-22T15:42:05.578 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi016.stderr:ceph-fuse[52930]: fuse finished with error 0 and tester_r 0 2023-05-22T15:42:05.579 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi016.front.sepia.ceph.com 2023-05-22T15:42:05.579 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:42:05.580 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-05-22T15:42:05.629 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:42:05.627+0000 7f1b042b5180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-05-22T15:42:05.685 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:42:05.680+0000 7fe4172e7180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-05-22T15:42:05.692 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:42:05.687+0000 7f1b042b5180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-05-22T15:42:05.746 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2023-05-22T15:42:05.746 DEBUG:teuthology.orchestra.run.smithi062:> dd if=/proc/self/mounts of=/dev/stdout 2023-05-22T15:42:05.748 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:42:05.744+0000 7fe4172e7180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-05-22T15:42:05.754 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16342116k,nr_inodes=4085529,mode=755,inode64 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275932k,mode=755,inode64 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev,inode64 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=13250 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275928k,nr_inodes=818982,mode=700,uid=1000,gid=1271,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275932k,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-05-22T15:42:05.754 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi062.front.sepia.ceph.com 2023-05-22T15:42:05.754 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:42:05.755 DEBUG:teuthology.orchestra.run.smithi062:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2023-05-22T15:42:05.801 INFO:teuthology.orchestra.run.smithi062.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2023-05-22T15:42:05.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-05-22T15:42:05.803 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-05-22T15:42:05.803 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2023-05-22T15:42:05.809 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16345192k,nr_inodes=4086298,mode=755,inode64 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3276544k,mode=755,inode64 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev,inode64 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=15732 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3276540k,nr_inodes=819135,mode=700,uid=1000,gid=1271,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3276544k,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-05-22T15:42:05.809 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi016.front.sepia.ceph.com 2023-05-22T15:42:05.810 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:42:05.810 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2023-05-22T15:42:05.816 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:42:05.811+0000 7f1b042b5180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-05-22T15:42:05.856 INFO:teuthology.orchestra.run.smithi016.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2023-05-22T15:42:05.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-05-22T15:42:05.857 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2023-05-22T15:42:05.857 DEBUG:teuthology.orchestra.run.smithi062:> dd if=/proc/self/mounts of=/dev/stdout 2023-05-22T15:42:05.863 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16342116k,nr_inodes=4085529,mode=755,inode64 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275932k,mode=755,inode64 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev,inode64 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=13250 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275928k,nr_inodes=818982,mode=700,uid=1000,gid=1271,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275932k,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-05-22T15:42:05.863 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi062.front.sepia.ceph.com 2023-05-22T15:42:05.864 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:42:05.865 DEBUG:teuthology.orchestra.run.smithi062:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2023-05-22T15:42:05.874 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:42:05.872+0000 7fe4172e7180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-05-22T15:42:05.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-05-22T15:42:05.911 INFO:teuthology.orchestra.run.smithi062.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2023-05-22T15:42:05.912 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-05-22T15:42:06.034 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.028+0000 7ff6f4a5f640 1 Processor -- start 2023-05-22T15:42:06.034 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.028+0000 7ff6f4a5f640 1 -- start start 2023-05-22T15:42:06.035 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.032+0000 7ff6f4a5f640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff6f00cf870 0x7ff6f006db00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:06.036 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.032+0000 7ff6f4a5f640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff6f006e740 0x7ff6f006ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:06.036 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.032+0000 7ff6f4a5f640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6f006e040 con 0x7ff6f015cfa0 2023-05-22T15:42:06.036 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.032+0000 7ff6f4a5f640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6f00cb9f0 con 0x7ff6f006e740 2023-05-22T15:42:06.036 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.032+0000 7ff6f4a5f640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6f006f100 con 0x7ff6f00cf870 2023-05-22T15:42:06.036 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.032+0000 7ff6eed9d640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff6f006e740 0x7ff6f006ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:06.037 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.032+0000 7ff6ee59c640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7ff6f015cfa0 0x7ff6f00b1fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:54282/0 (socket says 172.21.15.16:54282) 2023-05-22T15:42:06.037 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.032+0000 7ff6ee59c640 1 -- 172.21.15.16:0/3945424958 learned_addr learned my addr 172.21.15.16:0/3945424958 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:42:06.037 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.032+0000 7ff6eed9d640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff6f006e740 0x7ff6f006ebc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50812/0 (socket says 172.21.15.16:50812) 2023-05-22T15:42:06.038 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.032+0000 7ff6edd9b640 1 --2- 172.21.15.16:0/3945424958 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff6f00cf870 0x7ff6f006db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:06.038 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.032+0000 7ff6eed9d640 1 -- 172.21.15.16:0/3945424958 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff6f00cf870 msgr2=0x7ff6f006db00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:42:06.038 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.032+0000 7ff6eed9d640 1 --2- 172.21.15.16:0/3945424958 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff6f00cf870 0x7ff6f006db00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:06.038 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.032+0000 7ff6eed9d640 1 -- 172.21.15.16:0/3945424958 >> v1:172.21.15.16:6789/0 conn(0x7ff6f015cfa0 legacy=0x7ff6f00b1fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:06.038 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.032+0000 7ff6ed59a640 1 -- 172.21.15.16:0/3945424958 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 504586501 0 0) 0x7ff6d8001120 con 0x7ff6f015cfa0 2023-05-22T15:42:06.038 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.032+0000 7ff6eed9d640 1 -- 172.21.15.16:0/3945424958 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6f006e490 con 0x7ff6f006e740 2023-05-22T15:42:06.044 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.032+0000 7ff6eed9d640 1 --2- 172.21.15.16:0/3945424958 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff6f006e740 0x7ff6f006ebc0 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7ff6e80097c0 tx=0x7ff6e801c3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5f7278692af0ed0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:06.044 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.032+0000 7ff6ed59a640 1 -- 172.21.15.16:0/3945424958 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6e80ac620 con 0x7ff6f006e740 2023-05-22T15:42:06.045 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.032+0000 7ff6ed59a640 1 -- 172.21.15.16:0/3945424958 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6e80ac780 con 0x7ff6f006e740 2023-05-22T15:42:06.045 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.032+0000 7ff6ed59a640 1 -- 172.21.15.16:0/3945424958 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6e80acb10 con 0x7ff6f006e740 2023-05-22T15:42:06.045 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.036+0000 7ff6f4a5f640 1 -- 172.21.15.16:0/3945424958 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff6f006e740 msgr2=0x7ff6f006ebc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:06.046 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.036+0000 7ff6f4a5f640 1 --2- 172.21.15.16:0/3945424958 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff6f006e740 0x7ff6f006ebc0 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7ff6e80097c0 tx=0x7ff6e801c3c0 comp rx=0 tx=0).stop 2023-05-22T15:42:06.046 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.036+0000 7ff6f4a5f640 1 -- 172.21.15.16:0/3945424958 shutdown_connections 2023-05-22T15:42:06.046 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.036+0000 7ff6f4a5f640 1 --2- 172.21.15.16:0/3945424958 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff6f006e740 0x7ff6f006ebc0 unknown :-1 s=CLOSED pgs=997 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:06.046 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.036+0000 7ff6f4a5f640 1 --2- 172.21.15.16:0/3945424958 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff6f00cf870 0x7ff6f006db00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:06.047 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.036+0000 7ff6f4a5f640 1 -- 172.21.15.16:0/3945424958 >> 172.21.15.16:0/3945424958 conn(0x7ff6f00cc870 msgr2=0x7ff6f00cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:06.047 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.036+0000 7ff6f4a5f640 1 -- 172.21.15.16:0/3945424958 shutdown_connections 2023-05-22T15:42:06.047 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.036+0000 7ff6f4a5f640 1 -- 172.21.15.16:0/3945424958 wait complete. 2023-05-22T15:42:06.047 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.036+0000 7ff6f4a5f640 1 Processor -- start 2023-05-22T15:42:06.047 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.036+0000 7ff6f4a5f640 1 -- start start 2023-05-22T15:42:06.048 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.036+0000 7ff6f4a5f640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff6f006e740 0x7ff6f00b2fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:06.048 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.036+0000 7ff6f4a5f640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff6f00cf870 0x7ff6f01f7310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:06.048 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.036+0000 7ff6f4a5f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff6f015cfa0 0x7ff6f015a4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:06.048 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.036+0000 7ff6f4a5f640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6f00b3830 con 0x7ff6f015cfa0 2023-05-22T15:42:06.049 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.036+0000 7ff6f4a5f640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6f00b3b10 con 0x7ff6f006e740 2023-05-22T15:42:06.049 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.036+0000 7ff6f4a5f640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6f00b3df0 con 0x7ff6f00cf870 2023-05-22T15:42:06.049 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.036+0000 7ff6eed9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff6f015cfa0 0x7ff6f015a4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:06.050 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.036+0000 7ff6eed9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff6f015cfa0 0x7ff6f015a4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50826/0 (socket says 172.21.15.16:50826) 2023-05-22T15:42:06.050 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.036+0000 7ff6eed9d640 1 -- 172.21.15.16:0/2736445844 learned_addr learned my addr 172.21.15.16:0/2736445844 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:42:06.050 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.036+0000 7ff6ee59c640 1 --2- 172.21.15.16:0/2736445844 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff6f006e740 0x7ff6f00b2fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:06.050 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.036+0000 7ff6edd9b640 1 --2- 172.21.15.16:0/2736445844 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff6f00cf870 0x7ff6f01f7310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:06.050 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.036+0000 7ff6eed9d640 1 -- 172.21.15.16:0/2736445844 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff6f00cf870 msgr2=0x7ff6f01f7310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:06.050 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.036+0000 7ff6eed9d640 1 --2- 172.21.15.16:0/2736445844 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff6f00cf870 0x7ff6f01f7310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:06.051 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.036+0000 7ff6eed9d640 1 -- 172.21.15.16:0/2736445844 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff6f006e740 msgr2=0x7ff6f00b2fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:06.051 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.036+0000 7ff6eed9d640 1 --2- 172.21.15.16:0/2736445844 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff6f006e740 0x7ff6f00b2fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:06.051 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.036+0000 7ff6eed9d640 1 -- 172.21.15.16:0/2736445844 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6e8009420 con 0x7ff6f015cfa0 2023-05-22T15:42:06.051 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.036+0000 7ff6eed9d640 1 --2- 172.21.15.16:0/2736445844 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff6f015cfa0 0x7ff6f015a4d0 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7ff6e801aa70 tx=0x7ff6e8019160 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:06.051 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.036+0000 7ff6e6ffd640 1 -- 172.21.15.16:0/2736445844 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6e80ac620 con 0x7ff6f015cfa0 2023-05-22T15:42:06.051 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.036+0000 7ff6e6ffd640 1 -- 172.21.15.16:0/2736445844 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6e8004030 con 0x7ff6f015cfa0 2023-05-22T15:42:06.051 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.036+0000 7ff6f4a5f640 1 -- 172.21.15.16:0/2736445844 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6f00cb610 con 0x7ff6f015cfa0 2023-05-22T15:42:06.052 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.036+0000 7ff6e6ffd640 1 -- 172.21.15.16:0/2736445844 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6e80043a0 con 0x7ff6f015cfa0 2023-05-22T15:42:06.052 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.036+0000 7ff6f4a5f640 1 -- 172.21.15.16:0/2736445844 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6f00cb330 con 0x7ff6f015cfa0 2023-05-22T15:42:06.052 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.040+0000 7ff6f4a5f640 1 -- 172.21.15.16:0/2736445844 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6f015d370 con 0x7ff6f015cfa0 2023-05-22T15:42:06.052 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.040+0000 7ff6e6ffd640 1 -- 172.21.15.16:0/2736445844 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 23) v1 ==== 84344+0+0 (secure 0 0 0) 0x7ff6e80b3040 con 0x7ff6f015cfa0 2023-05-22T15:42:06.052 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.040+0000 7ff6e6ffd640 1 --2- 172.21.15.16:0/2736445844 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ff6d00654a0 0x7ff6d0067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:06.052 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.040+0000 7ff6ee59c640 1 -- 172.21.15.16:0/2736445844 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ff6d00654a0 msgr2=0x7ff6d0067950 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/36921 2023-05-22T15:42:06.052 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.040+0000 7ff6e6ffd640 1 -- 172.21.15.16:0/2736445844 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 6798+0+0 (secure 0 0 0) 0x7ff6e811cc20 con 0x7ff6f015cfa0 2023-05-22T15:42:06.053 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.040+0000 7ff6ee59c640 1 --2- 172.21.15.16:0/2736445844 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ff6d00654a0 0x7ff6d0067950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-05-22T15:42:06.053 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.044+0000 7ff6e6ffd640 1 -- 172.21.15.16:0/2736445844 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7ff6e80ea450 con 0x7ff6f015cfa0 2023-05-22T15:42:06.246 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.244+0000 7ff6ee59c640 1 -- 172.21.15.16:0/2736445844 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ff6d00654a0 msgr2=0x7ff6d0067950 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/36921 2023-05-22T15:42:06.246 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.244+0000 7ff6ee59c640 1 --2- 172.21.15.16:0/2736445844 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ff6d00654a0 0x7ff6d0067950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-05-22T15:42:06.247 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.244+0000 7ff6f4a5f640 1 -- 172.21.15.16:0/2736445844 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff6f00b3f30 con 0x7ff6f015cfa0 2023-05-22T15:42:06.248 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.244+0000 7ff6e6ffd640 1 -- 172.21.15.16:0/2736445844 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) v1 ==== 94+0+4645 (secure 0 0 0) 0x7ff6f00b3f30 con 0x7ff6f015cfa0 2023-05-22T15:42:06.248 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:42:06.249 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":22,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5399,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6833/1859135164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":1859135164},{"type":"v1","addr":"172.21.15.62:6833","nonce":1859135164}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":17},{"gid":5836,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/3067106582","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":3067106582},{"type":"v1","addr":"172.21.15.62:6835","nonce":3067106582}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":17},{"gid":6462,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/717991089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":717991089},{"type":"v1","addr":"172.21.15.16:6835","nonce":717991089}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":17}],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-05-22T15:40:02.754483+0000","modified":"2023-05-22T15:40:05.773388+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6468},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6468":{"gid":6468,"name":"a","rank":0,"incarnation":18,"state":"up:active","state_seq":3,"addr":"172.21.15.16:6837/4263396864","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":4263396864},{"type":"v1","addr":"172.21.15.16:6837","nonce":4263396864}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2023-05-22T15:42:06.249 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 22 2023-05-22T15:42:06.250 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.248+0000 7ff6f4a5f640 1 -- 172.21.15.16:0/2736445844 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ff6d00654a0 msgr2=0x7ff6d0067950 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-05-22T15:42:06.251 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.248+0000 7ff6f4a5f640 1 --2- 172.21.15.16:0/2736445844 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ff6d00654a0 0x7ff6d0067950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:06.251 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.248+0000 7ff6f4a5f640 1 -- 172.21.15.16:0/2736445844 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff6f015cfa0 msgr2=0x7ff6f015a4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:06.251 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.248+0000 7ff6f4a5f640 1 --2- 172.21.15.16:0/2736445844 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff6f015cfa0 0x7ff6f015a4d0 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7ff6e801aa70 tx=0x7ff6e8019160 comp rx=0 tx=0).stop 2023-05-22T15:42:06.251 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.248+0000 7ff6f4a5f640 1 -- 172.21.15.16:0/2736445844 shutdown_connections 2023-05-22T15:42:06.251 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.248+0000 7ff6f4a5f640 1 --2- 172.21.15.16:0/2736445844 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff6f015cfa0 0x7ff6f015a4d0 unknown :-1 s=CLOSED pgs=998 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:06.251 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.248+0000 7ff6f4a5f640 1 --2- 172.21.15.16:0/2736445844 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ff6d00654a0 0x7ff6d0067950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:06.251 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.248+0000 7ff6f4a5f640 1 --2- 172.21.15.16:0/2736445844 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff6f00cf870 0x7ff6f01f7310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:06.252 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.248+0000 7ff6f4a5f640 1 --2- 172.21.15.16:0/2736445844 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff6f006e740 0x7ff6f00b2fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:06.252 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.248+0000 7ff6f4a5f640 1 -- 172.21.15.16:0/2736445844 >> 172.21.15.16:0/2736445844 conn(0x7ff6f00cc870 msgr2=0x7ff6f00aebc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:06.252 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.248+0000 7ff6f4a5f640 1 -- 172.21.15.16:0/2736445844 shutdown_connections 2023-05-22T15:42:06.252 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.248+0000 7ff6f4a5f640 1 -- 172.21.15.16:0/2736445844 wait complete. 2023-05-22T15:42:06.267 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-05-22T15:42:06.309 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:42:06.307+0000 7f1b042b5180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-05-22T15:42:06.378 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.372+0000 7ff6ebf71640 1 Processor -- start 2023-05-22T15:42:06.378 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.372+0000 7ff6ebf71640 1 -- start start 2023-05-22T15:42:06.379 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.376+0000 7ff6ebf71640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7ff6e4071640 0x7ff6e4160090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:06.379 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.376+0000 7ff6ebf71640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff6e41605d0 0x7ff6e415c270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:06.379 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.376+0000 7ff6ebf71640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6e4069ae0 con 0x7ff6e40ce480 2023-05-22T15:42:06.379 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.376+0000 7ff6ebf71640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6e415f990 con 0x7ff6e4071640 2023-05-22T15:42:06.379 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.376+0000 7ff6ebf71640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6e4071ac0 con 0x7ff6e41605d0 2023-05-22T15:42:06.380 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.376+0000 7ff6e950c640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7ff6e4071640 0x7ff6e4160090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:06.380 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.376+0000 7ff6ea50e640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff6e41605d0 0x7ff6e415c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:06.380 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.376+0000 7ff6e950c640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7ff6e4071640 0x7ff6e4160090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.16:39170/0 (socket says 172.21.15.16:39170) 2023-05-22T15:42:06.380 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.376+0000 7ff6e9d0d640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7ff6e40ce480 0x7ff6e40ce850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.16:57132/0 (socket says 172.21.15.16:57132) 2023-05-22T15:42:06.381 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.376+0000 7ff6e950c640 1 -- 172.21.15.16:0/273725990 learned_addr learned my addr 172.21.15.16:0/273725990 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:42:06.381 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.376+0000 7ff6e950c640 1 -- 172.21.15.16:0/273725990 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff6e41605d0 msgr2=0x7ff6e415c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:42:06.381 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.376+0000 7ff6e950c640 1 --2- 172.21.15.16:0/273725990 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff6e41605d0 0x7ff6e415c270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:06.381 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.376+0000 7ff6e950c640 1 -- 172.21.15.16:0/273725990 >> v1:172.21.15.62:6789/0 conn(0x7ff6e40ce480 legacy=0x7ff6e40ce850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:06.381 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.376+0000 7ff6e950c640 1 -- 172.21.15.16:0/273725990 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6e4069dc0 con 0x7ff6e4071640 2023-05-22T15:42:06.382 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.376+0000 7ff6e950c640 1 --2- 172.21.15.16:0/273725990 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff6e4071640 0x7ff6e4160090 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7ff6e0009870 tx=0x7ff6e001c510 comp rx=0 tx=0).ready entity=mon.1 client_cookie=eeb8270f3d3bbe56 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:06.382 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.376+0000 7ff6e8d0b640 1 -- 172.21.15.16:0/273725990 <== mon.1 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6e00aa670 con 0x7ff6e4071640 2023-05-22T15:42:06.382 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.380+0000 7ff6e8d0b640 1 -- 172.21.15.16:0/273725990 <== mon.1 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6e00aa7d0 con 0x7ff6e4071640 2023-05-22T15:42:06.382 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.380+0000 7ff6e8d0b640 1 -- 172.21.15.16:0/273725990 <== mon.1 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6e00aab60 con 0x7ff6e4071640 2023-05-22T15:42:06.383 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.380+0000 7ff6ebf71640 1 -- 172.21.15.16:0/273725990 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff6e4071640 msgr2=0x7ff6e4160090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:06.383 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.380+0000 7ff6ebf71640 1 --2- 172.21.15.16:0/273725990 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff6e4071640 0x7ff6e4160090 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7ff6e0009870 tx=0x7ff6e001c510 comp rx=0 tx=0).stop 2023-05-22T15:42:06.383 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.380+0000 7ff6ebf71640 1 -- 172.21.15.16:0/273725990 shutdown_connections 2023-05-22T15:42:06.383 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.380+0000 7ff6ebf71640 1 --2- 172.21.15.16:0/273725990 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff6e41605d0 0x7ff6e415c270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:06.384 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.380+0000 7ff6ebf71640 1 --2- 172.21.15.16:0/273725990 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff6e4071640 0x7ff6e4160090 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:06.384 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.380+0000 7ff6ebf71640 1 -- 172.21.15.16:0/273725990 >> 172.21.15.16:0/273725990 conn(0x7ff6e40cc870 msgr2=0x7ff6e40cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:06.384 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.380+0000 7ff6ebf71640 1 -- 172.21.15.16:0/273725990 shutdown_connections 2023-05-22T15:42:06.384 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.380+0000 7ff6ebf71640 1 -- 172.21.15.16:0/273725990 wait complete. 2023-05-22T15:42:06.385 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.380+0000 7ff6ebf71640 1 Processor -- start 2023-05-22T15:42:06.386 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.380+0000 7ff6ebf71640 1 -- start start 2023-05-22T15:42:06.386 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.380+0000 7ff6ebf71640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff6e4071640 0x7ff6e40ab9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:06.386 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.380+0000 7ff6ebf71640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff6e40ce480 0x7ff6e41f8110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:06.386 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.380+0000 7ff6ebf71640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff6e41605d0 0x7ff6e414c450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:06.386 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.380+0000 7ff6ebf71640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6e40ae680 con 0x7ff6e41605d0 2023-05-22T15:42:06.386 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.380+0000 7ff6ebf71640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6e41f9040 con 0x7ff6e40ce480 2023-05-22T15:42:06.387 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.380+0000 7ff6ebf71640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6e41f87a0 con 0x7ff6e4071640 2023-05-22T15:42:06.387 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.380+0000 7ff6ea50e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff6e41605d0 0x7ff6e414c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:06.387 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.380+0000 7ff6ea50e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff6e41605d0 0x7ff6e414c450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50834/0 (socket says 172.21.15.16:50834) 2023-05-22T15:42:06.387 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.380+0000 7ff6ea50e640 1 -- 172.21.15.16:0/2386926188 learned_addr learned my addr 172.21.15.16:0/2386926188 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:42:06.387 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.380+0000 7ff6e950c640 1 --2- 172.21.15.16:0/2386926188 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff6e40ce480 0x7ff6e41f8110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:06.387 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.380+0000 7ff6e9d0d640 1 --2- 172.21.15.16:0/2386926188 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff6e4071640 0x7ff6e40ab9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:06.387 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.380+0000 7ff6ea50e640 1 -- 172.21.15.16:0/2386926188 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff6e4071640 msgr2=0x7ff6e40ab9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:06.387 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.380+0000 7ff6ea50e640 1 --2- 172.21.15.16:0/2386926188 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff6e4071640 0x7ff6e40ab9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:06.388 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.380+0000 7ff6ea50e640 1 -- 172.21.15.16:0/2386926188 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff6e40ce480 msgr2=0x7ff6e41f8110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:06.388 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.380+0000 7ff6ea50e640 1 --2- 172.21.15.16:0/2386926188 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff6e40ce480 0x7ff6e41f8110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:06.388 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.380+0000 7ff6ea50e640 1 -- 172.21.15.16:0/2386926188 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6e00094d0 con 0x7ff6e41605d0 2023-05-22T15:42:06.388 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.380+0000 7ff6ea50e640 1 --2- 172.21.15.16:0/2386926188 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff6e41605d0 0x7ff6e414c450 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7ff6dc00efc0 tx=0x7ff6dc00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:06.388 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.380+0000 7ff6da7fc640 1 -- 172.21.15.16:0/2386926188 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6dc009b20 con 0x7ff6e41605d0 2023-05-22T15:42:06.388 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.380+0000 7ff6da7fc640 1 -- 172.21.15.16:0/2386926188 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6dc00f040 con 0x7ff6e41605d0 2023-05-22T15:42:06.388 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.380+0000 7ff6da7fc640 1 -- 172.21.15.16:0/2386926188 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6dc0146c0 con 0x7ff6e41605d0 2023-05-22T15:42:06.389 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.380+0000 7ff6ebf71640 1 -- 172.21.15.16:0/2386926188 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6e40698b0 con 0x7ff6e41605d0 2023-05-22T15:42:06.389 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.380+0000 7ff6ebf71640 1 -- 172.21.15.16:0/2386926188 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6e4069720 con 0x7ff6e41605d0 2023-05-22T15:42:06.389 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:42:06.383+0000 7f1b042b5180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-05-22T15:42:06.390 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.384+0000 7ff6ebf71640 1 -- 172.21.15.16:0/2386926188 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6e40c5e70 con 0x7ff6e41605d0 2023-05-22T15:42:06.390 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.384+0000 7ff6da7fc640 1 -- 172.21.15.16:0/2386926188 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 23) v1 ==== 84344+0+0 (secure 0 0 0) 0x7ff6dc014820 con 0x7ff6e41605d0 2023-05-22T15:42:06.390 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.384+0000 7ff6da7fc640 1 --2- 172.21.15.16:0/2386926188 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ff6c80654a0 0x7ff6c8067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:06.390 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.384+0000 7ff6e9d0d640 1 -- 172.21.15.16:0/2386926188 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ff6c80654a0 msgr2=0x7ff6c8067950 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/36921 2023-05-22T15:42:06.390 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.384+0000 7ff6da7fc640 1 -- 172.21.15.16:0/2386926188 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 6798+0+0 (secure 0 0 0) 0x7ff6dc083b60 con 0x7ff6e41605d0 2023-05-22T15:42:06.391 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.384+0000 7ff6e9d0d640 1 --2- 172.21.15.16:0/2386926188 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ff6c80654a0 0x7ff6c8067950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-05-22T15:42:06.391 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.388+0000 7ff6da7fc640 1 -- 172.21.15.16:0/2386926188 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7ff6dc054d10 con 0x7ff6e41605d0 2023-05-22T15:42:06.402 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:42:06.400+0000 7fe4172e7180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-05-22T15:42:06.481 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:42:06.476+0000 7fe4172e7180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-05-22T15:42:06.523 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:42:06.519+0000 7f1b042b5180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-05-22T15:42:06.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.540+0000 7ff6ebf71640 1 -- 172.21.15.16:0/2386926188 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff6e404dbe0 con 0x7ff6e41605d0 2023-05-22T15:42:06.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.540+0000 7ff6da7fc640 1 -- 172.21.15.16:0/2386926188 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v76) v1 ==== 74+0+18657 (secure 0 0 0) 0x7ff6dc009e60 con 0x7ff6e41605d0 2023-05-22T15:42:06.545 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:42:06.545 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":76,"fsid":"b2719834-7847-4d98-90a6-f65d57f85c49","created":"2023-05-22T15:26:25.259314+0000","modified":"2023-05-22T15:42:04.442363+0000","last_up_change":"2023-05-22T15:26:36.448524+0000","last_in_change":"2023-05-22T15:26:27.675668+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-05-22T15:26:38.019781+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-05-22T15:26:43.570514+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2023-05-22T15:40:00.663040+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"72","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2023-05-22T15:40:01.750680+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":5,"snap_epoch":76,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9e7ff854-b25f-4cbb-a5c2-1799d9c53d2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":37021},{"type":"v1","addr":"172.21.15.16:6825","nonce":37021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":37021},{"type":"v1","addr":"172.21.15.16:6827","nonce":37021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":37021},{"type":"v1","addr":"172.21.15.16:6831","nonce":37021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":37021},{"type":"v1","addr":"172.21.15.16:6829","nonce":37021}]},"public_addr":"172.21.15.16:6825/37021","cluster_addr":"172.21.15.16:6827/37021","heartbeat_back_addr":"172.21.15.16:6831/37021","heartbeat_front_addr":"172.21.15.16:6829/37021","state":["exists","up"]},{"osd":1,"uuid":"4702fbc9-a927-4565-aca5-41dc69cc8264","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":37008},{"type":"v1","addr":"172.21.15.16:6801","nonce":37008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":37008},{"type":"v1","addr":"172.21.15.16:6803","nonce":37008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":37008},{"type":"v1","addr":"172.21.15.16:6807","nonce":37008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":37008},{"type":"v1","addr":"172.21.15.16:6805","nonce":37008}]},"public_addr":"172.21.15.16:6801/37008","cluster_addr":"172.21.15.16:6803/37008","heartbeat_back_addr":"172.21.15.16:6807/37008","heartbeat_front_addr":"172.21.15.16:6805/37008","state":["exists","up"]},{"osd":2,"uuid":"1e7bba46-4eba-47ef-bb81-5e2d371638ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":37009},{"type":"v1","addr":"172.21.15.16:6809","nonce":37009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":37009},{"type":"v1","addr":"172.21.15.16:6811","nonce":37009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":37009},{"type":"v1","addr":"172.21.15.16:6815","nonce":37009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":37009},{"type":"v1","addr":"172.21.15.16:6813","nonce":37009}]},"public_addr":"172.21.15.16:6809/37009","cluster_addr":"172.21.15.16:6811/37009","heartbeat_back_addr":"172.21.15.16:6815/37009","heartbeat_front_addr":"172.21.15.16:6813/37009","state":["exists","up"]},{"osd":3,"uuid":"383cacc0-5a88-4904-990e-498ca59ef48a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":37020},{"type":"v1","addr":"172.21.15.16:6817","nonce":37020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":37020},{"type":"v1","addr":"172.21.15.16:6819","nonce":37020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":37020},{"type":"v1","addr":"172.21.15.16:6823","nonce":37020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":37020},{"type":"v1","addr":"172.21.15.16:6821","nonce":37020}]},"public_addr":"172.21.15.16:6817/37020","cluster_addr":"172.21.15.16:6819/37020","heartbeat_back_addr":"172.21.15.16:6823/37020","heartbeat_front_addr":"172.21.15.16:6821/37020","state":["exists","up"]},{"osd":4,"uuid":"970500ef-95b7-42c1-b080-eed4fae7013d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":37353},{"type":"v1","addr":"172.21.15.62:6817","nonce":37353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6819","nonce":37353},{"type":"v1","addr":"172.21.15.62:6822","nonce":37353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37353},{"type":"v1","addr":"172.21.15.62:6831","nonce":37353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37353},{"type":"v1","addr":"172.21.15.62:6826","nonce":37353}]},"public_addr":"172.21.15.62:6817/37353","cluster_addr":"172.21.15.62:6822/37353","heartbeat_back_addr":"172.21.15.62:6831/37353","heartbeat_front_addr":"172.21.15.62:6826/37353","state":["exists","up"]},{"osd":5,"uuid":"022d67db-d4d8-445b-b1b9-8b446a00f5c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":37352},{"type":"v1","addr":"172.21.15.62:6801","nonce":37352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":37352},{"type":"v1","addr":"172.21.15.62:6803","nonce":37352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":37352},{"type":"v1","addr":"172.21.15.62:6807","nonce":37352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":37352},{"type":"v1","addr":"172.21.15.62:6805","nonce":37352}]},"public_addr":"172.21.15.62:6801/37352","cluster_addr":"172.21.15.62:6803/37352","heartbeat_back_addr":"172.21.15.62:6807/37352","heartbeat_front_addr":"172.21.15.62:6805/37352","state":["exists","up"]},{"osd":6,"uuid":"42bd372b-6688-4a1b-8407-e16e8c266e33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":37354},{"type":"v1","addr":"172.21.15.62:6820","nonce":37354}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":37354},{"type":"v1","addr":"172.21.15.62:6823","nonce":37354}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6829","nonce":37354},{"type":"v1","addr":"172.21.15.62:6830","nonce":37354}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6825","nonce":37354},{"type":"v1","addr":"172.21.15.62:6827","nonce":37354}]},"public_addr":"172.21.15.62:6820/37354","cluster_addr":"172.21.15.62:6823/37354","heartbeat_back_addr":"172.21.15.62:6830/37354","heartbeat_front_addr":"172.21.15.62:6827/37354","state":["exists","up"]},{"osd":7,"uuid":"ac70b2a3-f41b-4bf2-b227-7f2cbb55426d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37355},{"type":"v1","addr":"172.21.15.62:6809","nonce":37355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37355},{"type":"v1","addr":"172.21.15.62:6811","nonce":37355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":37355},{"type":"v1","addr":"172.21.15.62:6815","nonce":37355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":37355},{"type":"v1","addr":"172.21.15.62:6813","nonce":37355}]},"public_addr":"172.21.15.62:6809/37355","cluster_addr":"172.21.15.62:6811/37355","heartbeat_back_addr":"172.21.15.62:6815/37355","heartbeat_front_addr":"172.21.15.62:6813/37355","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.368896+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.107273+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.276888+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.287502+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.124791+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.251937+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.255679+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.232797+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:0/2468042232":"2023-05-23T15:40:15.816078+0000","172.21.15.16:0/1582569975":"2023-05-23T15:40:15.816078+0000","172.21.15.16:0/2252481002":"2023-05-23T15:40:15.816078+0000","172.21.15.16:0/174222799":"2023-05-23T15:40:15.816078+0000","172.21.15.16:0/4017778835":"2023-05-23T15:40:15.816078+0000","172.21.15.16:0/3438310522":"2023-05-23T15:40:15.816078+0000","172.21.15.16:6833/36921":"2023-05-23T15:40:15.816078+0000","172.21.15.16:6832/36921":"2023-05-23T15:40:15.816078+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":9,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[{"pool":9,"snaps":[{"begin":3,"length":1},{"begin":5,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-05-22T15:42:06.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.544+0000 7ff6ebf71640 1 -- 172.21.15.16:0/2386926188 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ff6c80654a0 msgr2=0x7ff6c8067950 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-05-22T15:42:06.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.544+0000 7ff6ebf71640 1 --2- 172.21.15.16:0/2386926188 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ff6c80654a0 0x7ff6c8067950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:06.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.544+0000 7ff6ebf71640 1 -- 172.21.15.16:0/2386926188 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff6e41605d0 msgr2=0x7ff6e414c450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:06.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.544+0000 7ff6ebf71640 1 --2- 172.21.15.16:0/2386926188 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff6e41605d0 0x7ff6e414c450 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7ff6dc00efc0 tx=0x7ff6dc00c490 comp rx=0 tx=0).stop 2023-05-22T15:42:06.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.544+0000 7ff6ebf71640 1 -- 172.21.15.16:0/2386926188 shutdown_connections 2023-05-22T15:42:06.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.544+0000 7ff6ebf71640 1 --2- 172.21.15.16:0/2386926188 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff6e41605d0 0x7ff6e414c450 unknown :-1 s=CLOSED pgs=999 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:06.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.544+0000 7ff6ebf71640 1 --2- 172.21.15.16:0/2386926188 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff6e40ce480 0x7ff6e41f8110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:06.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.544+0000 7ff6ebf71640 1 --2- 172.21.15.16:0/2386926188 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ff6c80654a0 0x7ff6c8067950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:06.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.544+0000 7ff6ebf71640 1 --2- 172.21.15.16:0/2386926188 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff6e4071640 0x7ff6e40ab9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:06.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.544+0000 7ff6ebf71640 1 -- 172.21.15.16:0/2386926188 >> 172.21.15.16:0/2386926188 conn(0x7ff6e40cc870 msgr2=0x7ff6e415d910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:06.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.544+0000 7ff6ebf71640 1 -- 172.21.15.16:0/2386926188 shutdown_connections 2023-05-22T15:42:06.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.544+0000 7ff6ebf71640 1 -- 172.21.15.16:0/2386926188 wait complete. 2023-05-22T15:42:06.563 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-05-22T15:26:38.019781+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-05-22T15:26:43.570514+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_metadata', 'create_time': '2023-05-22T15:40:00.663040+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '72', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 9, 'pool_name': 'cephfs_data', 'create_time': '2023-05-22T15:40:01.750680+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '76', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 5, 'snap_epoch': 76, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-05-22T15:42:06.564 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-05-22T15:42:06.599 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:42:06.595+0000 7f1b042b5180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-05-22T15:42:06.623 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:42:06.620+0000 7fe4172e7180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-05-22T15:42:06.679 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.676+0000 7f8c8ae93640 1 Processor -- start 2023-05-22T15:42:06.679 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.676+0000 7f8c8ae93640 1 -- start start 2023-05-22T15:42:06.680 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.676+0000 7f8c8ae93640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f8c840c67d0 0x7f8c840cf730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:06.681 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.676+0000 7f8c8ae93640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f8c840b2880 0x7f8c840cbfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:06.681 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.676+0000 7f8c8ae93640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c840cfc70 con 0x7f8c840cc520 2023-05-22T15:42:06.681 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.676+0000 7f8c8ae93640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c8415a300 con 0x7f8c840c67d0 2023-05-22T15:42:06.681 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.676+0000 7f8c8ae93640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c840c4dc0 con 0x7f8c840b2880 2023-05-22T15:42:06.681 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.676+0000 7f8c88c2f640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f8c840c67d0 0x7f8c840cf730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:06.681 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.676+0000 7f8c88c2f640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f8c840c67d0 0x7f8c840cf730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50838/0 (socket says 172.21.15.16:50838) 2023-05-22T15:42:06.681 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.676+0000 7f8c88c2f640 1 -- 172.21.15.16:0/373309596 learned_addr learned my addr 172.21.15.16:0/373309596 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:42:06.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.676+0000 7f8c83fff640 1 --2- 172.21.15.16:0/373309596 >> v2:172.21.15.62:3300/0 conn(0x7f8c840b2880 0x7f8c840cbfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:06.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.676+0000 7f8c88c2f640 1 -- 172.21.15.16:0/373309596 >> v2:172.21.15.62:3300/0 conn(0x7f8c840b2880 msgr2=0x7f8c840cbfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:42:06.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.676+0000 7f8c88c2f640 1 --2- 172.21.15.16:0/373309596 >> v2:172.21.15.62:3300/0 conn(0x7f8c840b2880 0x7f8c840cbfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:06.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.676+0000 7f8c88c2f640 1 -- 172.21.15.16:0/373309596 >> v1:172.21.15.62:6789/0 conn(0x7f8c840cc520 legacy=0x7f8c840ce160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:42:06.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.676+0000 7f8c88c2f640 1 -- 172.21.15.16:0/373309596 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c840c6ba0 con 0x7f8c840c67d0 2023-05-22T15:42:06.683 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.680+0000 7f8c88c2f640 1 --2- 172.21.15.16:0/373309596 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8c840c67d0 0x7f8c840cf730 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7f8c840ccc40 tx=0x7f8c7801c400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37b80b5cc9049c56 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:06.683 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.680+0000 7f8c837fe640 1 -- 172.21.15.16:0/373309596 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c780ab600 con 0x7f8c840c67d0 2023-05-22T15:42:06.683 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.680+0000 7f8c837fe640 1 -- 172.21.15.16:0/373309596 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c780a7050 con 0x7f8c840c67d0 2023-05-22T15:42:06.683 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.680+0000 7f8c837fe640 1 -- 172.21.15.16:0/373309596 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c780ab980 con 0x7f8c840c67d0 2023-05-22T15:42:06.684 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.680+0000 7f8c8ae93640 1 -- 172.21.15.16:0/373309596 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8c840c67d0 msgr2=0x7f8c840cf730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:06.684 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.680+0000 7f8c8ae93640 1 --2- 172.21.15.16:0/373309596 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8c840c67d0 0x7f8c840cf730 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7f8c840ccc40 tx=0x7f8c7801c400 comp rx=0 tx=0).stop 2023-05-22T15:42:06.684 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.680+0000 7f8c8ae93640 1 -- 172.21.15.16:0/373309596 shutdown_connections 2023-05-22T15:42:06.685 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.680+0000 7f8c8ae93640 1 --2- 172.21.15.16:0/373309596 >> v2:172.21.15.62:3300/0 conn(0x7f8c840b2880 0x7f8c840cbfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:06.685 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.680+0000 7f8c8ae93640 1 --2- 172.21.15.16:0/373309596 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8c840c67d0 0x7f8c840cf730 unknown :-1 s=CLOSED pgs=1000 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:06.685 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.680+0000 7f8c8ae93640 1 -- 172.21.15.16:0/373309596 >> 172.21.15.16:0/373309596 conn(0x7f8c8406f5a0 msgr2=0x7f8c8406f9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:06.685 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.680+0000 7f8c8ae93640 1 -- 172.21.15.16:0/373309596 shutdown_connections 2023-05-22T15:42:06.685 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.680+0000 7f8c8ae93640 1 -- 172.21.15.16:0/373309596 wait complete. 2023-05-22T15:42:06.686 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.680+0000 7f8c8ae93640 1 Processor -- start 2023-05-22T15:42:06.686 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.680+0000 7f8c8ae93640 1 -- start start 2023-05-22T15:42:06.688 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.684+0000 7f8c8ae93640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8c840b2880 0x7f8c840b50a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:06.688 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.684+0000 7f8c8ae93640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8c840c67d0 0x7f8c84148cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:06.688 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.684+0000 7f8c8ae93640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8c840cc520 0x7f8c8415b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:06.688 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.684+0000 7f8c8ae93640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c84149200 con 0x7f8c840cc520 2023-05-22T15:42:06.689 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.684+0000 7f8c8ae93640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c840b3e30 con 0x7f8c840c67d0 2023-05-22T15:42:06.689 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.684+0000 7f8c8ae93640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c840b45c0 con 0x7f8c840b2880 2023-05-22T15:42:06.689 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.684+0000 7f8c89430640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8c840cc520 0x7f8c8415b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:06.689 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.684+0000 7f8c89430640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8c840cc520 0x7f8c8415b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50850/0 (socket says 172.21.15.16:50850) 2023-05-22T15:42:06.689 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.684+0000 7f8c89430640 1 -- 172.21.15.16:0/113531119 learned_addr learned my addr 172.21.15.16:0/113531119 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:42:06.689 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.684+0000 7f8c83fff640 1 --2- 172.21.15.16:0/113531119 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8c840c67d0 0x7f8c84148cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:06.690 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.684+0000 7f8c88c2f640 1 --2- 172.21.15.16:0/113531119 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8c840b2880 0x7f8c840b50a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:06.690 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.684+0000 7f8c89430640 1 -- 172.21.15.16:0/113531119 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8c840b2880 msgr2=0x7f8c840b50a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:06.690 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.684+0000 7f8c89430640 1 --2- 172.21.15.16:0/113531119 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8c840b2880 0x7f8c840b50a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:06.690 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.684+0000 7f8c89430640 1 -- 172.21.15.16:0/113531119 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8c840c67d0 msgr2=0x7f8c84148cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:06.690 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.684+0000 7f8c89430640 1 --2- 172.21.15.16:0/113531119 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8c840c67d0 0x7f8c84148cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:06.691 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.684+0000 7f8c89430640 1 -- 172.21.15.16:0/113531119 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c780094d0 con 0x7f8c840cc520 2023-05-22T15:42:06.691 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.684+0000 7f8c83fff640 1 --2- 172.21.15.16:0/113531119 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8c840c67d0 0x7f8c84148cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T15:42:06.691 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.684+0000 7f8c88c2f640 1 --2- 172.21.15.16:0/113531119 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8c840b2880 0x7f8c840b50a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T15:42:06.691 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.684+0000 7f8c89430640 1 --2- 172.21.15.16:0/113531119 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8c840cc520 0x7f8c8415b9a0 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7f8c7c00c780 tx=0x7f8c7c00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:06.691 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.684+0000 7f8c817fa640 1 -- 172.21.15.16:0/113531119 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c7c004b70 con 0x7f8c840cc520 2023-05-22T15:42:06.691 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.684+0000 7f8c817fa640 1 -- 172.21.15.16:0/113531119 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c7c00ce90 con 0x7f8c840cc520 2023-05-22T15:42:06.692 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.684+0000 7f8c8ae93640 1 -- 172.21.15.16:0/113531119 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c840d0790 con 0x7f8c840cc520 2023-05-22T15:42:06.692 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.684+0000 7f8c817fa640 1 -- 172.21.15.16:0/113531119 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c7c00a850 con 0x7f8c840cc520 2023-05-22T15:42:06.692 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.684+0000 7f8c8ae93640 1 -- 172.21.15.16:0/113531119 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c840d04b0 con 0x7f8c840cc520 2023-05-22T15:42:06.693 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.684+0000 7f8c8ae93640 1 -- 172.21.15.16:0/113531119 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c840d0a70 con 0x7f8c840cc520 2023-05-22T15:42:06.693 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.684+0000 7f8c817fa640 1 -- 172.21.15.16:0/113531119 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 23) v1 ==== 84344+0+0 (secure 0 0 0) 0x7f8c7c00a9b0 con 0x7f8c840cc520 2023-05-22T15:42:06.693 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.688+0000 7f8c817fa640 1 --2- 172.21.15.16:0/113531119 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f8c680654a0 0x7f8c68067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:06.694 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.688+0000 7f8c88c2f640 1 -- 172.21.15.16:0/113531119 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f8c680654a0 msgr2=0x7f8c68067950 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/36921 2023-05-22T15:42:06.694 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.688+0000 7f8c817fa640 1 -- 172.21.15.16:0/113531119 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 6798+0+0 (secure 0 0 0) 0x7f8c7c082e10 con 0x7f8c840cc520 2023-05-22T15:42:06.694 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.688+0000 7f8c88c2f640 1 --2- 172.21.15.16:0/113531119 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f8c680654a0 0x7f8c68067950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-05-22T15:42:06.694 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.688+0000 7f8c817fa640 1 -- 172.21.15.16:0/113531119 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f8c7c086030 con 0x7f8c840cc520 2023-05-22T15:42:06.702 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:42:06.700+0000 7fe4172e7180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-05-22T15:42:06.845 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:42:06.843+0000 7f1b042b5180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-05-22T15:42:06.846 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.844+0000 7f8c8ae93640 1 -- 172.21.15.16:0/113531119 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f8c8404dbe0 con 0x7f8c840cc520 2023-05-22T15:42:06.847 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.844+0000 7f8c817fa640 1 -- 172.21.15.16:0/113531119 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v76) v1 ==== 93+0+10 (secure 0 0 0) 0x7f8c7c0540c0 con 0x7f8c840cc520 2023-05-22T15:42:06.847 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2023-05-22T15:42:06.849 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.844+0000 7f8c8ae93640 1 -- 172.21.15.16:0/113531119 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f8c680654a0 msgr2=0x7f8c68067950 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-05-22T15:42:06.849 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.844+0000 7f8c8ae93640 1 --2- 172.21.15.16:0/113531119 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f8c680654a0 0x7f8c68067950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:06.849 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.844+0000 7f8c8ae93640 1 -- 172.21.15.16:0/113531119 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8c840cc520 msgr2=0x7f8c8415b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:06.849 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.844+0000 7f8c8ae93640 1 --2- 172.21.15.16:0/113531119 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8c840cc520 0x7f8c8415b9a0 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7f8c7c00c780 tx=0x7f8c7c00cc40 comp rx=0 tx=0).stop 2023-05-22T15:42:06.850 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.844+0000 7f8c8ae93640 1 -- 172.21.15.16:0/113531119 shutdown_connections 2023-05-22T15:42:06.850 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.844+0000 7f8c8ae93640 1 --2- 172.21.15.16:0/113531119 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8c840cc520 0x7f8c8415b9a0 unknown :-1 s=CLOSED pgs=1001 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:06.850 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.844+0000 7f8c8ae93640 1 --2- 172.21.15.16:0/113531119 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8c840c67d0 0x7f8c84148cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:06.850 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.844+0000 7f8c8ae93640 1 --2- 172.21.15.16:0/113531119 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f8c680654a0 0x7f8c68067950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:06.850 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.844+0000 7f8c8ae93640 1 --2- 172.21.15.16:0/113531119 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8c840b2880 0x7f8c840b50a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:06.850 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.844+0000 7f8c8ae93640 1 -- 172.21.15.16:0/113531119 >> 172.21.15.16:0/113531119 conn(0x7f8c8406f5a0 msgr2=0x7f8c840c4f00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:06.850 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.844+0000 7f8c8ae93640 1 -- 172.21.15.16:0/113531119 shutdown_connections 2023-05-22T15:42:06.851 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.844+0000 7f8c8ae93640 1 -- 172.21.15.16:0/113531119 wait complete. 2023-05-22T15:42:06.864 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-05-22T15:42:06.899 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:42:06.895+0000 7f1b042b5180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-05-22T15:42:06.954 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:42:06.952+0000 7fe4172e7180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-05-22T15:42:06.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.992+0000 7f4e39dc6640 1 Processor -- start 2023-05-22T15:42:06.998 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.992+0000 7f4e39dc6640 1 -- start start 2023-05-22T15:42:06.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.996+0000 7f4e39dc6640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4e341563c0 0x7f4e340c7b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:07.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.996+0000 7f4e39dc6640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f4e340cc6c0 0x7f4e340c8080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:07.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.996+0000 7f4e39dc6640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e3415c210 con 0x7f4e3406c220 2023-05-22T15:42:07.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.996+0000 7f4e39dc6640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e34160700 con 0x7f4e340cc6c0 2023-05-22T15:42:07.001 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.996+0000 7f4e39dc6640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e3406cdd0 con 0x7f4e341563c0 2023-05-22T15:42:07.001 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.996+0000 7f4e32ffd640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f4e340cc6c0 0x7f4e340c8080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:07.001 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.996+0000 7f4e32ffd640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f4e340cc6c0 0x7f4e340c8080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50862/0 (socket says 172.21.15.16:50862) 2023-05-22T15:42:07.001 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.996+0000 7f4e32ffd640 1 -- 172.21.15.16:0/3795273005 learned_addr learned my addr 172.21.15.16:0/3795273005 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:42:07.002 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.996+0000 7f4e337fe640 1 --2- 172.21.15.16:0/3795273005 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4e341563c0 0x7f4e340c7b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:07.002 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.996+0000 7f4e32ffd640 1 -- 172.21.15.16:0/3795273005 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4e341563c0 msgr2=0x7f4e340c7b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:42:07.002 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.996+0000 7f4e32ffd640 1 --2- 172.21.15.16:0/3795273005 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4e341563c0 0x7f4e340c7b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:07.003 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.996+0000 7f4e32ffd640 1 -- 172.21.15.16:0/3795273005 >> v1:172.21.15.16:6789/0 conn(0x7f4e3406c220 legacy=0x7f4e3406c6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:07.003 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.996+0000 7f4e32ffd640 1 -- 172.21.15.16:0/3795273005 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e3415c480 con 0x7f4e340cc6c0 2023-05-22T15:42:07.008 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.996+0000 7f4e32ffd640 1 --2- 172.21.15.16:0/3795273005 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4e340cc6c0 0x7f4e340c8080 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7f4e28009870 tx=0x7f4e2801c510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a26015461acf92d server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:07.008 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.996+0000 7f4e327fc640 1 -- 172.21.15.16:0/3795273005 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e280aa670 con 0x7f4e340cc6c0 2023-05-22T15:42:07.008 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.996+0000 7f4e327fc640 1 -- 172.21.15.16:0/3795273005 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e280aa7d0 con 0x7f4e340cc6c0 2023-05-22T15:42:07.009 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.996+0000 7f4e327fc640 1 -- 172.21.15.16:0/3795273005 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e280aab60 con 0x7f4e340cc6c0 2023-05-22T15:42:07.009 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.996+0000 7f4e39dc6640 1 -- 172.21.15.16:0/3795273005 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4e340cc6c0 msgr2=0x7f4e340c8080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:07.009 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.996+0000 7f4e39dc6640 1 --2- 172.21.15.16:0/3795273005 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4e340cc6c0 0x7f4e340c8080 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7f4e28009870 tx=0x7f4e2801c510 comp rx=0 tx=0).stop 2023-05-22T15:42:07.009 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.996+0000 7f4e39dc6640 1 -- 172.21.15.16:0/3795273005 shutdown_connections 2023-05-22T15:42:07.009 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:06.996+0000 7f4e39dc6640 1 --2- 172.21.15.16:0/3795273005 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4e340cc6c0 0x7f4e340c8080 unknown :-1 s=CLOSED pgs=1003 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:07.009 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.000+0000 7f4e39dc6640 1 --2- 172.21.15.16:0/3795273005 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4e341563c0 0x7f4e340c7b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:07.009 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.000+0000 7f4e39dc6640 1 -- 172.21.15.16:0/3795273005 >> 172.21.15.16:0/3795273005 conn(0x7f4e340b1110 msgr2=0x7f4e340ce830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:07.010 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.000+0000 7f4e39dc6640 1 -- 172.21.15.16:0/3795273005 shutdown_connections 2023-05-22T15:42:07.010 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.000+0000 7f4e39dc6640 1 -- 172.21.15.16:0/3795273005 wait complete. 2023-05-22T15:42:07.010 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.000+0000 7f4e39dc6640 1 Processor -- start 2023-05-22T15:42:07.010 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.000+0000 7f4e39dc6640 1 -- start start 2023-05-22T15:42:07.010 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.000+0000 7f4e39dc6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4e3406c220 0x7f4e34143640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:07.010 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.000+0000 7f4e39dc6640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4e340cc6c0 0x7f4e340afd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:07.010 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.000+0000 7f4e39dc6640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4e341451d0 0x7f4e340699b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:07.011 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.000+0000 7f4e39dc6640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e3414d4b0 con 0x7f4e3406c220 2023-05-22T15:42:07.011 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.000+0000 7f4e39dc6640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e341f4a20 con 0x7f4e341451d0 2023-05-22T15:42:07.011 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.000+0000 7f4e39dc6640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e341f5a50 con 0x7f4e340cc6c0 2023-05-22T15:42:07.011 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.000+0000 7f4e337fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4e3406c220 0x7f4e34143640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:07.011 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.000+0000 7f4e337fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4e3406c220 0x7f4e34143640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50870/0 (socket says 172.21.15.16:50870) 2023-05-22T15:42:07.011 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.000+0000 7f4e337fe640 1 -- 172.21.15.16:0/2659427194 learned_addr learned my addr 172.21.15.16:0/2659427194 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:42:07.011 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.000+0000 7f4e32ffd640 1 --2- 172.21.15.16:0/2659427194 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4e340cc6c0 0x7f4e340afd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:07.011 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.000+0000 7f4e33fff640 1 --2- 172.21.15.16:0/2659427194 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4e341451d0 0x7f4e340699b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:07.012 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.000+0000 7f4e337fe640 1 -- 172.21.15.16:0/2659427194 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4e340cc6c0 msgr2=0x7f4e340afd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:07.012 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.000+0000 7f4e337fe640 1 --2- 172.21.15.16:0/2659427194 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4e340cc6c0 0x7f4e340afd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:07.012 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.000+0000 7f4e337fe640 1 -- 172.21.15.16:0/2659427194 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4e341451d0 msgr2=0x7f4e340699b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:07.012 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.000+0000 7f4e337fe640 1 --2- 172.21.15.16:0/2659427194 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4e341451d0 0x7f4e340699b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:07.012 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.000+0000 7f4e337fe640 1 -- 172.21.15.16:0/2659427194 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e280094d0 con 0x7f4e3406c220 2023-05-22T15:42:07.012 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.000+0000 7f4e337fe640 1 --2- 172.21.15.16:0/2659427194 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4e3406c220 0x7f4e34143640 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7f4e2400d620 tx=0x7f4e2400dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:07.012 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.004+0000 7f4e1bfff640 1 -- 172.21.15.16:0/2659427194 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e2400b620 con 0x7f4e3406c220 2023-05-22T15:42:07.013 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.004+0000 7f4e1bfff640 1 -- 172.21.15.16:0/2659427194 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e2400b780 con 0x7f4e3406c220 2023-05-22T15:42:07.013 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.004+0000 7f4e39dc6640 1 -- 172.21.15.16:0/2659427194 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e3415ff00 con 0x7f4e3406c220 2023-05-22T15:42:07.013 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.004+0000 7f4e1bfff640 1 -- 172.21.15.16:0/2659427194 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e2400bb10 con 0x7f4e3406c220 2023-05-22T15:42:07.013 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.004+0000 7f4e39dc6640 1 -- 172.21.15.16:0/2659427194 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e3415cd20 con 0x7f4e3406c220 2023-05-22T15:42:07.013 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.004+0000 7f4e1bfff640 1 -- 172.21.15.16:0/2659427194 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 23) v1 ==== 84344+0+0 (secure 0 0 0) 0x7f4e2400bc70 con 0x7f4e3406c220 2023-05-22T15:42:07.013 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.004+0000 7f4e19ffb640 1 -- 172.21.15.16:0/2659427194 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e0c005260 con 0x7f4e3406c220 2023-05-22T15:42:07.013 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.004+0000 7f4e1bfff640 1 --2- 172.21.15.16:0/2659427194 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f4e200654a0 0x7f4e20067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:07.014 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.004+0000 7f4e32ffd640 1 -- 172.21.15.16:0/2659427194 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f4e200654a0 msgr2=0x7f4e20067950 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/36921 2023-05-22T15:42:07.014 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.004+0000 7f4e1bfff640 1 -- 172.21.15.16:0/2659427194 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 6798+0+0 (secure 0 0 0) 0x7f4e240831a0 con 0x7f4e3406c220 2023-05-22T15:42:07.014 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.004+0000 7f4e32ffd640 1 --2- 172.21.15.16:0/2659427194 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f4e200654a0 0x7f4e20067950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-05-22T15:42:07.015 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.008+0000 7f4e1bfff640 1 -- 172.21.15.16:0/2659427194 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f4e24050a30 con 0x7f4e3406c220 2023-05-22T15:42:07.015 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:42:07.008+0000 7fe4172e7180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-05-22T15:42:07.042 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.040+0000 7f4e1bfff640 1 -- 172.21.15.16:0/2659427194 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mgrmap(e 24) v1 ==== 83941+0+0 (secure 0 0 0) 0x7f4e24081280 con 0x7f4e3406c220 2023-05-22T15:42:07.043 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.040+0000 7f4e1bfff640 1 -- 172.21.15.16:0/2659427194 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f4e200654a0 msgr2=0x7f4e20067950 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-05-22T15:42:07.043 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.040+0000 7f4e1bfff640 1 --2- 172.21.15.16:0/2659427194 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f4e200654a0 0x7f4e20067950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:07.112 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:42:07.108+0000 7fe3a1c72640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T15:42:07.113 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:42:07.108+0000 7fe3a1c72640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T15:42:07.113 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:42:07.108+0000 7fe3a1c72640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T15:42:07.113 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:42:07.108+0000 7fe3a1c72640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T15:42:07.114 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:42:07.108+0000 7fe3a1c72640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T15:42:07.114 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:42:07.108+0000 7fe39fc6e640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T15:42:07.114 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:42:07.108+0000 7fe39fc6e640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T15:42:07.114 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:42:07.108+0000 7fe39fc6e640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T15:42:07.115 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:42:07.108+0000 7fe39fc6e640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T15:42:07.115 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:42:07.108+0000 7fe39fc6e640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T15:42:07.189 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.184+0000 7f4e19ffb640 1 -- 172.21.15.16:0/2659427194 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f4e0c005a90 con 0x7f4e3406c220 2023-05-22T15:42:07.190 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.188+0000 7f4e1bfff640 1 -- 172.21.15.16:0/2659427194 <== mon.0 v2:172.21.15.16:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v77) v1 ==== 92+0+10 (secure 0 0 0) 0x7f4e2407d6a0 con 0x7f4e3406c220 2023-05-22T15:42:07.190 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 8 2023-05-22T15:42:07.192 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.188+0000 7f4e19ffb640 1 -- 172.21.15.16:0/2659427194 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4e3406c220 msgr2=0x7f4e34143640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:07.193 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.188+0000 7f4e19ffb640 1 --2- 172.21.15.16:0/2659427194 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4e3406c220 0x7f4e34143640 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7f4e2400d620 tx=0x7f4e2400dae0 comp rx=0 tx=0).stop 2023-05-22T15:42:07.193 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.188+0000 7f4e19ffb640 1 -- 172.21.15.16:0/2659427194 shutdown_connections 2023-05-22T15:42:07.193 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.188+0000 7f4e19ffb640 1 --2- 172.21.15.16:0/2659427194 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4e341451d0 0x7f4e340699b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:07.194 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.188+0000 7f4e19ffb640 1 --2- 172.21.15.16:0/2659427194 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f4e200654a0 0x7f4e20067950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:07.194 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.188+0000 7f4e19ffb640 1 --2- 172.21.15.16:0/2659427194 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4e340cc6c0 0x7f4e340afd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:07.194 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.188+0000 7f4e19ffb640 1 --2- 172.21.15.16:0/2659427194 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4e3406c220 0x7f4e34143640 unknown :-1 s=CLOSED pgs=1004 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:07.194 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.188+0000 7f4e19ffb640 1 -- 172.21.15.16:0/2659427194 >> 172.21.15.16:0/2659427194 conn(0x7f4e340b1110 msgr2=0x7f4e3406cf10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:07.195 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.188+0000 7f4e19ffb640 1 -- 172.21.15.16:0/2659427194 shutdown_connections 2023-05-22T15:42:07.195 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.188+0000 7f4e19ffb640 1 -- 172.21.15.16:0/2659427194 wait complete. 2023-05-22T15:42:07.207 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-05-22T15:42:07.316 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.312+0000 7f9aa2329640 1 Processor -- start 2023-05-22T15:42:07.317 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.312+0000 7f9aa2329640 1 -- start start 2023-05-22T15:42:07.319 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.312+0000 7f9aa2329640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f9a9c06b6c0 0x7f9a9c0cb4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:07.319 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.312+0000 7f9aa2329640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9a9c06ba90 0x7f9a9c158fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:07.319 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.312+0000 7f9aa2329640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a9c159f70 con 0x7f9a9c159510 2023-05-22T15:42:07.319 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.312+0000 7f9aa2329640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a9c071760 con 0x7f9a9c06b6c0 2023-05-22T15:42:07.319 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.312+0000 7f9aa2329640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a9c0ca7d0 con 0x7f9a9c06ba90 2023-05-22T15:42:07.319 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.312+0000 7f9a9bfff640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f9a9c06b6c0 0x7f9a9c0cb4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:07.320 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.312+0000 7f9a9bfff640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f9a9c06b6c0 0x7f9a9c0cb4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50990/0 (socket says 172.21.15.16:50990) 2023-05-22T15:42:07.320 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.312+0000 7f9a9bfff640 1 -- 172.21.15.16:0/2981903828 learned_addr learned my addr 172.21.15.16:0/2981903828 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:42:07.320 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.312+0000 7f9a9b7fe640 1 --2- 172.21.15.16:0/2981903828 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9a9c06ba90 0x7f9a9c158fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:07.320 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.312+0000 7f9a9bfff640 1 -- 172.21.15.16:0/2981903828 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9a9c06ba90 msgr2=0x7f9a9c158fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:42:07.320 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.312+0000 7f9a9bfff640 1 --2- 172.21.15.16:0/2981903828 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9a9c06ba90 0x7f9a9c158fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:07.321 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.316+0000 7f9a9bfff640 1 -- 172.21.15.16:0/2981903828 >> v1:172.21.15.62:6789/0 conn(0x7f9a9c159510 legacy=0x7f9a9c0cc740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:42:07.321 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.316+0000 7f9a9bfff640 1 -- 172.21.15.16:0/2981903828 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a9c0cb030 con 0x7f9a9c06b6c0 2023-05-22T15:42:07.322 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.316+0000 7f9a9bfff640 1 --2- 172.21.15.16:0/2981903828 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9a9c06b6c0 0x7f9a9c0cb4c0 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7f9a94009e50 tx=0x7f9a9401c9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3a6a2ea938cd2ef server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:07.322 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.316+0000 7f9a9affd640 1 -- 172.21.15.16:0/2981903828 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a940a5740 con 0x7f9a9c06b6c0 2023-05-22T15:42:07.322 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.316+0000 7f9a9affd640 1 -- 172.21.15.16:0/2981903828 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a940a58a0 con 0x7f9a9c06b6c0 2023-05-22T15:42:07.322 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.316+0000 7f9a9affd640 1 -- 172.21.15.16:0/2981903828 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a940a5c30 con 0x7f9a9c06b6c0 2023-05-22T15:42:07.322 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.316+0000 7f9aa2329640 1 -- 172.21.15.16:0/2981903828 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9a9c06b6c0 msgr2=0x7f9a9c0cb4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:07.323 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.316+0000 7f9aa2329640 1 --2- 172.21.15.16:0/2981903828 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9a9c06b6c0 0x7f9a9c0cb4c0 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7f9a94009e50 tx=0x7f9a9401c9a0 comp rx=0 tx=0).stop 2023-05-22T15:42:07.323 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.316+0000 7f9aa2329640 1 -- 172.21.15.16:0/2981903828 shutdown_connections 2023-05-22T15:42:07.323 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.316+0000 7f9aa2329640 1 --2- 172.21.15.16:0/2981903828 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9a9c06ba90 0x7f9a9c158fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:07.323 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.316+0000 7f9aa2329640 1 --2- 172.21.15.16:0/2981903828 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9a9c06b6c0 0x7f9a9c0cb4c0 unknown :-1 s=CLOSED pgs=1018 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:07.323 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.316+0000 7f9aa2329640 1 -- 172.21.15.16:0/2981903828 >> 172.21.15.16:0/2981903828 conn(0x7f9a9c069720 msgr2=0x7f9a9c069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:07.323 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.316+0000 7f9aa2329640 1 -- 172.21.15.16:0/2981903828 shutdown_connections 2023-05-22T15:42:07.323 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.316+0000 7f9aa2329640 1 -- 172.21.15.16:0/2981903828 wait complete. 2023-05-22T15:42:07.324 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.320+0000 7f9aa2329640 1 Processor -- start 2023-05-22T15:42:07.324 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.320+0000 7f9aa2329640 1 -- start start 2023-05-22T15:42:07.325 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.320+0000 7f9aa2329640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9a9c06b6c0 0x7f9a9c1574e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:07.325 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.320+0000 7f9aa2329640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9a9c06ba90 0x7f9a9c14bdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:07.326 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.320+0000 7f9aa2329640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9a9c159510 0x7f9a9c14c300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:07.326 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.320+0000 7f9aa2329640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a9c0c2100 con 0x7f9a9c159510 2023-05-22T15:42:07.326 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.320+0000 7f9aa2329640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a9c0c23e0 con 0x7f9a9c06b6c0 2023-05-22T15:42:07.326 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.320+0000 7f9aa2329640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a9c0c26c0 con 0x7f9a9c06ba90 2023-05-22T15:42:07.326 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.320+0000 7f9aa08c6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9a9c159510 0x7f9a9c14c300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:07.326 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.320+0000 7f9aa08c6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9a9c159510 0x7f9a9c14c300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50998/0 (socket says 172.21.15.16:50998) 2023-05-22T15:42:07.326 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.320+0000 7f9aa08c6640 1 -- 172.21.15.16:0/428062861 learned_addr learned my addr 172.21.15.16:0/428062861 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:42:07.327 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.320+0000 7f9a9bfff640 1 --2- 172.21.15.16:0/428062861 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9a9c06b6c0 0x7f9a9c1574e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:07.327 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.320+0000 7f9a9b7fe640 1 --2- 172.21.15.16:0/428062861 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9a9c06ba90 0x7f9a9c14bdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:07.327 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.320+0000 7f9aa08c6640 1 -- 172.21.15.16:0/428062861 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9a9c06ba90 msgr2=0x7f9a9c14bdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:07.327 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.320+0000 7f9aa08c6640 1 --2- 172.21.15.16:0/428062861 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9a9c06ba90 0x7f9a9c14bdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:07.327 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.320+0000 7f9aa08c6640 1 -- 172.21.15.16:0/428062861 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9a9c06b6c0 msgr2=0x7f9a9c1574e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:07.327 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.320+0000 7f9aa08c6640 1 --2- 172.21.15.16:0/428062861 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9a9c06b6c0 0x7f9a9c1574e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:07.327 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.320+0000 7f9aa08c6640 1 -- 172.21.15.16:0/428062861 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a94009b30 con 0x7f9a9c159510 2023-05-22T15:42:07.328 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.320+0000 7f9a9b7fe640 1 --2- 172.21.15.16:0/428062861 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9a9c06ba90 0x7f9a9c14bdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:42:07.328 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.320+0000 7f9aa08c6640 1 --2- 172.21.15.16:0/428062861 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9a9c159510 0x7f9a9c14c300 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7f9a8c00b410 tx=0x7f9a8c00b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:07.328 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.320+0000 7f9a98ff9640 1 -- 172.21.15.16:0/428062861 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a8c011830 con 0x7f9a9c159510 2023-05-22T15:42:07.328 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.320+0000 7f9a98ff9640 1 -- 172.21.15.16:0/428062861 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a8c0119d0 con 0x7f9a9c159510 2023-05-22T15:42:07.328 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.320+0000 7f9aa2329640 1 -- 172.21.15.16:0/428062861 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a9c159990 con 0x7f9a9c159510 2023-05-22T15:42:07.328 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.320+0000 7f9a98ff9640 1 -- 172.21.15.16:0/428062861 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a8c011d80 con 0x7f9a9c159510 2023-05-22T15:42:07.328 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.320+0000 7f9aa2329640 1 -- 172.21.15.16:0/428062861 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a9c0cacc0 con 0x7f9a9c159510 2023-05-22T15:42:07.328 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.320+0000 7f9aa2329640 1 -- 172.21.15.16:0/428062861 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a9c06be60 con 0x7f9a9c159510 2023-05-22T15:42:07.329 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.320+0000 7f9a98ff9640 1 -- 172.21.15.16:0/428062861 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) v1 ==== 83941+0+0 (secure 0 0 0) 0x7f9a8c0193b0 con 0x7f9a9c159510 2023-05-22T15:42:07.329 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.320+0000 7f9a98ff9640 1 -- 172.21.15.16:0/428062861 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7012+0+0 (secure 0 0 0) 0x7f9a8c0825a0 con 0x7f9a9c159510 2023-05-22T15:42:07.329 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.324+0000 7f9a98ff9640 1 -- 172.21.15.16:0/428062861 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f9a8c050160 con 0x7f9a9c159510 2023-05-22T15:42:07.475 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.472+0000 7f9aa2329640 1 -- 172.21.15.16:0/428062861 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f9a9c0720f0 con 0x7f9a9c159510 2023-05-22T15:42:07.478 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.476+0000 7f9a98ff9640 1 -- 172.21.15.16:0/428062861 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v77) v1 ==== 104+0+11 (secure 0 0 0) 0x7f9a8c053780 con 0x7f9a9c159510 2023-05-22T15:42:07.479 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2023-05-22T15:42:07.481 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.476+0000 7f9aa2329640 1 -- 172.21.15.16:0/428062861 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9a9c159510 msgr2=0x7f9a9c14c300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:07.481 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.476+0000 7f9aa2329640 1 --2- 172.21.15.16:0/428062861 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9a9c159510 0x7f9a9c14c300 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7f9a8c00b410 tx=0x7f9a8c00b8d0 comp rx=0 tx=0).stop 2023-05-22T15:42:07.481 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.476+0000 7f9aa2329640 1 -- 172.21.15.16:0/428062861 shutdown_connections 2023-05-22T15:42:07.481 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.476+0000 7f9aa2329640 1 --2- 172.21.15.16:0/428062861 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9a9c159510 0x7f9a9c14c300 unknown :-1 s=CLOSED pgs=1019 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:07.481 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.476+0000 7f9aa2329640 1 --2- 172.21.15.16:0/428062861 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9a9c06ba90 0x7f9a9c14bdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:07.482 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.476+0000 7f9aa2329640 1 --2- 172.21.15.16:0/428062861 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9a9c06b6c0 0x7f9a9c1574e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:07.482 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.476+0000 7f9aa2329640 1 -- 172.21.15.16:0/428062861 >> 172.21.15.16:0/428062861 conn(0x7f9a9c069720 msgr2=0x7f9a9c06fb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:07.482 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.476+0000 7f9aa2329640 1 -- 172.21.15.16:0/428062861 shutdown_connections 2023-05-22T15:42:07.482 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.476+0000 7f9aa2329640 1 -- 172.21.15.16:0/428062861 wait complete. 2023-05-22T15:42:07.496 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-05-22T15:42:07.611 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.608+0000 7f630e48f640 1 Processor -- start 2023-05-22T15:42:07.612 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.608+0000 7f630e48f640 1 -- start start 2023-05-22T15:42:07.613 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.608+0000 7f630e48f640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6308070c70 0x7f6308071530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:07.613 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.608+0000 7f630e48f640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f630806b6c0 0x7f630806bb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:07.613 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.608+0000 7f630e48f640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f630815d520 con 0x7f630815c580 2023-05-22T15:42:07.613 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.608+0000 7f630e48f640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63080c8b10 con 0x7f630806b6c0 2023-05-22T15:42:07.613 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.608+0000 7f630e48f640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f630815c9d0 con 0x7f6308070c70 2023-05-22T15:42:07.614 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.608+0000 7f6307fff640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6308070c70 0x7f6308071530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:07.615 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.608+0000 7f630ca2c640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f630806b6c0 0x7f630806bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:07.615 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.608+0000 7f6307fff640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6308070c70 0x7f6308071530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3301/0 says I am v2:172.21.15.16:34410/0 (socket says 172.21.15.16:34410) 2023-05-22T15:42:07.616 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.608+0000 7f6307fff640 1 -- 172.21.15.16:0/3405949094 learned_addr learned my addr 172.21.15.16:0/3405949094 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:42:07.616 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.608+0000 7f6307fff640 1 -- 172.21.15.16:0/3405949094 >> v2:172.21.15.62:3300/0 conn(0x7f630806b6c0 msgr2=0x7f630806bb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:42:07.616 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.608+0000 7f6307fff640 1 --2- 172.21.15.16:0/3405949094 >> v2:172.21.15.62:3300/0 conn(0x7f630806b6c0 0x7f630806bb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:07.616 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.608+0000 7f6307fff640 1 -- 172.21.15.16:0/3405949094 >> v1:172.21.15.62:6789/0 conn(0x7f630815c580 legacy=0x7f630806fba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:07.616 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.608+0000 7f6307fff640 1 -- 172.21.15.16:0/3405949094 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f630806c060 con 0x7f6308070c70 2023-05-22T15:42:07.618 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.612+0000 7f6307fff640 1 --2- 172.21.15.16:0/3405949094 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6308070c70 0x7f6308071530 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f62f800bb10 tx=0x7f62f801eb50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2ad5e9ead8898e2b server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:07.618 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.612+0000 7f6306ffd640 1 -- 172.21.15.16:0/3405949094 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62f80a7c20 con 0x7f6308070c70 2023-05-22T15:42:07.618 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.612+0000 7f6306ffd640 1 -- 172.21.15.16:0/3405949094 <== mon.2 v2:172.21.15.62:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62f80a7d80 con 0x7f6308070c70 2023-05-22T15:42:07.618 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.612+0000 7f6306ffd640 1 -- 172.21.15.16:0/3405949094 <== mon.2 v2:172.21.15.62:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62f80af660 con 0x7f6308070c70 2023-05-22T15:42:07.618 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.612+0000 7f630e48f640 1 -- 172.21.15.16:0/3405949094 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6308070c70 msgr2=0x7f6308071530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:07.619 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.612+0000 7f630e48f640 1 --2- 172.21.15.16:0/3405949094 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6308070c70 0x7f6308071530 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f62f800bb10 tx=0x7f62f801eb50 comp rx=0 tx=0).stop 2023-05-22T15:42:07.619 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.612+0000 7f630e48f640 1 -- 172.21.15.16:0/3405949094 shutdown_connections 2023-05-22T15:42:07.619 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.612+0000 7f630e48f640 1 --2- 172.21.15.16:0/3405949094 >> v2:172.21.15.62:3300/0 conn(0x7f630806b6c0 0x7f630806bb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:07.619 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.612+0000 7f630e48f640 1 --2- 172.21.15.16:0/3405949094 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6308070c70 0x7f6308071530 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:07.619 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.612+0000 7f630e48f640 1 -- 172.21.15.16:0/3405949094 >> 172.21.15.16:0/3405949094 conn(0x7f6308069720 msgr2=0x7f6308069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:07.620 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.612+0000 7f630e48f640 1 -- 172.21.15.16:0/3405949094 shutdown_connections 2023-05-22T15:42:07.620 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.612+0000 7f630e48f640 1 -- 172.21.15.16:0/3405949094 wait complete. 2023-05-22T15:42:07.620 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.612+0000 7f630e48f640 1 Processor -- start 2023-05-22T15:42:07.620 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.616+0000 7f630e48f640 1 -- start start 2023-05-22T15:42:07.621 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.616+0000 7f630e48f640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f630806b6c0 0x7f6308150bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:07.622 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.616+0000 7f630e48f640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6308070c70 0x7f63080c7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:07.622 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.616+0000 7f630e48f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f630815c580 0x7f6308156040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:07.622 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.616+0000 7f630e48f640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63080b12d0 con 0x7f630815c580 2023-05-22T15:42:07.622 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.616+0000 7f630e48f640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63080b15b0 con 0x7f6308070c70 2023-05-22T15:42:07.622 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.616+0000 7f630e48f640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63080b1890 con 0x7f630806b6c0 2023-05-22T15:42:07.622 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.616+0000 7f630ca2c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f630815c580 0x7f6308156040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:07.623 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.616+0000 7f630ca2c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f630815c580 0x7f6308156040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51008/0 (socket says 172.21.15.16:51008) 2023-05-22T15:42:07.623 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.616+0000 7f630ca2c640 1 -- 172.21.15.16:0/816596200 learned_addr learned my addr 172.21.15.16:0/816596200 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:42:07.623 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.616+0000 7f6307fff640 1 --2- 172.21.15.16:0/816596200 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f630806b6c0 0x7f6308150bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:07.624 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.616+0000 7f63077fe640 1 --2- 172.21.15.16:0/816596200 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6308070c70 0x7f63080c7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:07.624 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.616+0000 7f630ca2c640 1 -- 172.21.15.16:0/816596200 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f630806b6c0 msgr2=0x7f6308150bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:07.624 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.616+0000 7f630ca2c640 1 --2- 172.21.15.16:0/816596200 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f630806b6c0 0x7f6308150bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:07.625 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.616+0000 7f630ca2c640 1 -- 172.21.15.16:0/816596200 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6308070c70 msgr2=0x7f63080c7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:07.625 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.616+0000 7f630ca2c640 1 --2- 172.21.15.16:0/816596200 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6308070c70 0x7f63080c7ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:07.625 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.616+0000 7f630ca2c640 1 -- 172.21.15.16:0/816596200 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62f800b770 con 0x7f630815c580 2023-05-22T15:42:07.625 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.616+0000 7f630ca2c640 1 --2- 172.21.15.16:0/816596200 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f630815c580 0x7f6308156040 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7f630000efc0 tx=0x7f630000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:07.626 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.616+0000 7f6304ff9640 1 -- 172.21.15.16:0/816596200 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6300014480 con 0x7f630815c580 2023-05-22T15:42:07.626 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.616+0000 7f6304ff9640 1 -- 172.21.15.16:0/816596200 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f630000f040 con 0x7f630815c580 2023-05-22T15:42:07.626 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.616+0000 7f630e48f640 1 -- 172.21.15.16:0/816596200 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63080cc510 con 0x7f630815c580 2023-05-22T15:42:07.627 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.616+0000 7f6304ff9640 1 -- 172.21.15.16:0/816596200 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6300014820 con 0x7f630815c580 2023-05-22T15:42:07.627 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.616+0000 7f630e48f640 1 -- 172.21.15.16:0/816596200 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f630815d240 con 0x7f630815c580 2023-05-22T15:42:07.627 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.616+0000 7f630e48f640 1 -- 172.21.15.16:0/816596200 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6308071040 con 0x7f630815c580 2023-05-22T15:42:07.628 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.616+0000 7f6304ff9640 1 -- 172.21.15.16:0/816596200 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) v1 ==== 83941+0+0 (secure 0 0 0) 0x7f6300014980 con 0x7f630815c580 2023-05-22T15:42:07.629 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.616+0000 7f6304ff9640 1 -- 172.21.15.16:0/816596200 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7012+0+0 (secure 0 0 0) 0x7f63000834b0 con 0x7f630815c580 2023-05-22T15:42:07.629 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.620+0000 7f6304ff9640 1 -- 172.21.15.16:0/816596200 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f63000511c0 con 0x7f630815c580 2023-05-22T15:42:07.772 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.768+0000 7f630e48f640 1 -- 172.21.15.16:0/816596200 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f630804dbe0 con 0x7f630815c580 2023-05-22T15:42:07.773 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.768+0000 7f6304ff9640 1 -- 172.21.15.16:0/816596200 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v77) v1 ==== 100+0+11 (secure 0 0 0) 0x7f63000547e0 con 0x7f630815c580 2023-05-22T15:42:07.774 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 64 2023-05-22T15:42:07.775 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.772+0000 7f630e48f640 1 -- 172.21.15.16:0/816596200 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f630815c580 msgr2=0x7f6308156040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:07.776 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.772+0000 7f630e48f640 1 --2- 172.21.15.16:0/816596200 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f630815c580 0x7f6308156040 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7f630000efc0 tx=0x7f630000c490 comp rx=0 tx=0).stop 2023-05-22T15:42:07.776 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.772+0000 7f630e48f640 1 -- 172.21.15.16:0/816596200 shutdown_connections 2023-05-22T15:42:07.776 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.772+0000 7f630e48f640 1 --2- 172.21.15.16:0/816596200 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f630815c580 0x7f6308156040 unknown :-1 s=CLOSED pgs=1020 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:07.776 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.772+0000 7f630e48f640 1 --2- 172.21.15.16:0/816596200 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6308070c70 0x7f63080c7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:07.776 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.772+0000 7f630e48f640 1 --2- 172.21.15.16:0/816596200 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f630806b6c0 0x7f6308150bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:07.776 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.772+0000 7f630e48f640 1 -- 172.21.15.16:0/816596200 >> 172.21.15.16:0/816596200 conn(0x7f6308069720 msgr2=0x7f6308069af0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:07.777 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.772+0000 7f630e48f640 1 -- 172.21.15.16:0/816596200 shutdown_connections 2023-05-22T15:42:07.777 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.772+0000 7f630e48f640 1 -- 172.21.15.16:0/816596200 wait complete. 2023-05-22T15:42:07.791 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-05-22T15:42:07.901 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.896+0000 7fb585817640 1 Processor -- start 2023-05-22T15:42:07.901 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.896+0000 7fb585817640 1 -- start start 2023-05-22T15:42:07.903 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.896+0000 7fb585817640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb5800ca4e0 0x7fb58015e460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:07.904 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.896+0000 7fb585817640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb5800c9b40 0x7fb5800afb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:07.904 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.896+0000 7fb585817640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5800b0650 con 0x7fb580070df0 2023-05-22T15:42:07.904 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.896+0000 7fb585817640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5800b0370 con 0x7fb5800c9b40 2023-05-22T15:42:07.905 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.896+0000 7fb585817640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb58015d980 con 0x7fb5800ca4e0 2023-05-22T15:42:07.905 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.896+0000 7fb57e7fc640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb5800c9b40 0x7fb5800afb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:07.905 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.900+0000 7fb57e7fc640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb5800c9b40 0x7fb5800afb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51024/0 (socket says 172.21.15.16:51024) 2023-05-22T15:42:07.906 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.900+0000 7fb57e7fc640 1 -- 172.21.15.16:0/2432348045 learned_addr learned my addr 172.21.15.16:0/2432348045 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:42:07.907 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.900+0000 7fb57effd640 1 --2- 172.21.15.16:0/2432348045 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb5800ca4e0 0x7fb58015e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:07.907 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.900+0000 7fb57e7fc640 1 -- 172.21.15.16:0/2432348045 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb5800ca4e0 msgr2=0x7fb58015e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:42:07.907 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.900+0000 7fb57e7fc640 1 --2- 172.21.15.16:0/2432348045 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb5800ca4e0 0x7fb58015e460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:07.907 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.900+0000 7fb57e7fc640 1 -- 172.21.15.16:0/2432348045 >> v1:172.21.15.62:6789/0 conn(0x7fb580070df0 legacy=0x7fb580071210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:07.908 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.900+0000 7fb57e7fc640 1 -- 172.21.15.16:0/2432348045 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb58015dc60 con 0x7fb5800c9b40 2023-05-22T15:42:07.912 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.900+0000 7fb57e7fc640 1 --2- 172.21.15.16:0/2432348045 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb5800c9b40 0x7fb5800afb40 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7fb568009870 tx=0x7fb56801c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6991cfb0a16f0f0d server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:07.913 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.900+0000 7fb57dffb640 1 -- 172.21.15.16:0/2432348045 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5680ab660 con 0x7fb5800c9b40 2023-05-22T15:42:07.913 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.900+0000 7fb57dffb640 1 -- 172.21.15.16:0/2432348045 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5680ab7c0 con 0x7fb5800c9b40 2023-05-22T15:42:07.913 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.900+0000 7fb57dffb640 1 -- 172.21.15.16:0/2432348045 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5680abb50 con 0x7fb5800c9b40 2023-05-22T15:42:07.913 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.900+0000 7fb585817640 1 -- 172.21.15.16:0/2432348045 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb5800c9b40 msgr2=0x7fb5800afb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:07.913 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.900+0000 7fb585817640 1 --2- 172.21.15.16:0/2432348045 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb5800c9b40 0x7fb5800afb40 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7fb568009870 tx=0x7fb56801c480 comp rx=0 tx=0).stop 2023-05-22T15:42:07.913 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.900+0000 7fb585817640 1 -- 172.21.15.16:0/2432348045 shutdown_connections 2023-05-22T15:42:07.913 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.900+0000 7fb585817640 1 --2- 172.21.15.16:0/2432348045 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb5800c9b40 0x7fb5800afb40 unknown :-1 s=CLOSED pgs=1021 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:07.914 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.900+0000 7fb585817640 1 --2- 172.21.15.16:0/2432348045 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb5800ca4e0 0x7fb58015e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:07.914 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.900+0000 7fb585817640 1 -- 172.21.15.16:0/2432348045 >> 172.21.15.16:0/2432348045 conn(0x7fb580069720 msgr2=0x7fb580069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:07.914 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.900+0000 7fb585817640 1 -- 172.21.15.16:0/2432348045 shutdown_connections 2023-05-22T15:42:07.914 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.900+0000 7fb585817640 1 -- 172.21.15.16:0/2432348045 wait complete. 2023-05-22T15:42:07.914 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.904+0000 7fb585817640 1 Processor -- start 2023-05-22T15:42:07.914 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.904+0000 7fb585817640 1 -- start start 2023-05-22T15:42:07.915 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.904+0000 7fb585817640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb580070df0 0x7fb5801f59c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:07.915 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.904+0000 7fb585817640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb5800c9b40 0x7fb5801f3f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:07.915 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.904+0000 7fb585817640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb5800ca4e0 0x7fb5801f4fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:07.915 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.904+0000 7fb585817640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb580150b90 con 0x7fb580070df0 2023-05-22T15:42:07.915 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.904+0000 7fb585817640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5801f4470 con 0x7fb5800c9b40 2023-05-22T15:42:07.916 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.904+0000 7fb585817640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5800ad7c0 con 0x7fb5800ca4e0 2023-05-22T15:42:07.916 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.904+0000 7fb57effd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb580070df0 0x7fb5801f59c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:07.916 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.904+0000 7fb57effd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb580070df0 0x7fb5801f59c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51030/0 (socket says 172.21.15.16:51030) 2023-05-22T15:42:07.916 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.904+0000 7fb57effd640 1 -- 172.21.15.16:0/4173520494 learned_addr learned my addr 172.21.15.16:0/4173520494 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:42:07.916 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.904+0000 7fb57e7fc640 1 --2- 172.21.15.16:0/4173520494 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb5800c9b40 0x7fb5801f3f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:07.916 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.904+0000 7fb57f7fe640 1 --2- 172.21.15.16:0/4173520494 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb5800ca4e0 0x7fb5801f4fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:07.916 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.904+0000 7fb57effd640 1 -- 172.21.15.16:0/4173520494 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb5800ca4e0 msgr2=0x7fb5801f4fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:07.917 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.904+0000 7fb57effd640 1 --2- 172.21.15.16:0/4173520494 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb5800ca4e0 0x7fb5801f4fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:07.917 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.904+0000 7fb57effd640 1 -- 172.21.15.16:0/4173520494 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb5800c9b40 msgr2=0x7fb5801f3f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:07.917 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.904+0000 7fb57effd640 1 --2- 172.21.15.16:0/4173520494 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb5800c9b40 0x7fb5801f3f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:07.917 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.904+0000 7fb57effd640 1 -- 172.21.15.16:0/4173520494 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5680094d0 con 0x7fb580070df0 2023-05-22T15:42:07.917 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.904+0000 7fb57e7fc640 1 --2- 172.21.15.16:0/4173520494 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb5800c9b40 0x7fb5801f3f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:42:07.917 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.904+0000 7fb57f7fe640 1 --2- 172.21.15.16:0/4173520494 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb5800ca4e0 0x7fb5801f4fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:42:07.918 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.904+0000 7fb57effd640 1 --2- 172.21.15.16:0/4173520494 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb580070df0 0x7fb5801f59c0 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7fb578009d10 tx=0x7fb57800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:07.918 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.904+0000 7fb577fff640 1 -- 172.21.15.16:0/4173520494 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb578009b80 con 0x7fb580070df0 2023-05-22T15:42:07.918 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.904+0000 7fb577fff640 1 -- 172.21.15.16:0/4173520494 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb57800f040 con 0x7fb580070df0 2023-05-22T15:42:07.918 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.904+0000 7fb577fff640 1 -- 172.21.15.16:0/4173520494 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb578014680 con 0x7fb580070df0 2023-05-22T15:42:07.918 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.904+0000 7fb585817640 1 -- 172.21.15.16:0/4173520494 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5800c8e80 con 0x7fb580070df0 2023-05-22T15:42:07.919 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.904+0000 7fb585817640 1 -- 172.21.15.16:0/4173520494 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5800ca8b0 con 0x7fb580070df0 2023-05-22T15:42:07.919 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.904+0000 7fb585817640 1 -- 172.21.15.16:0/4173520494 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5800c9010 con 0x7fb580070df0 2023-05-22T15:42:07.919 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.904+0000 7fb577fff640 1 -- 172.21.15.16:0/4173520494 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) v1 ==== 83941+0+0 (secure 0 0 0) 0x7fb5780147e0 con 0x7fb580070df0 2023-05-22T15:42:07.919 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.908+0000 7fb577fff640 1 -- 172.21.15.16:0/4173520494 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7012+0+0 (secure 0 0 0) 0x7fb578083320 con 0x7fb580070df0 2023-05-22T15:42:07.920 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:07.912+0000 7fb577fff640 1 -- 172.21.15.16:0/4173520494 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fb578050ce0 con 0x7fb580070df0 2023-05-22T15:42:08.052 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.048+0000 7fb577fff640 1 -- 172.21.15.16:0/4173520494 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mgrmap(e 25) v1 ==== 84068+0+0 (secure 0 0 0) 0x7fb578050960 con 0x7fb580070df0 2023-05-22T15:42:08.053 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.048+0000 7fb577fff640 1 --2- 172.21.15.16:0/4173520494 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fb56006b710 0x7fb56006daf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:08.056 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.052+0000 7fb57e7fc640 1 --2- 172.21.15.16:0/4173520494 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fb56006b710 0x7fb56006daf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:08.058 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.056+0000 7fb57e7fc640 1 --2- 172.21.15.16:0/4173520494 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fb56006b710 0x7fb56006daf0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fb5680af000 tx=0x7fb568008040 comp rx=0 tx=0).ready entity=mgr.6765 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:08.108 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.104+0000 7fb585817640 1 -- 172.21.15.16:0/4173520494 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb5800cb010 con 0x7fb580070df0 2023-05-22T15:42:08.109 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.104+0000 7fb577fff640 1 -- 172.21.15.16:0/4173520494 <== mon.0 v2:172.21.15.16:3300/0 8 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) v1 ==== 94+0+4645 (secure 0 0 0) 0x7fb57807d4e0 con 0x7fb580070df0 2023-05-22T15:42:08.109 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:42:08.110 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":22,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5399,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6833/1859135164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":1859135164},{"type":"v1","addr":"172.21.15.62:6833","nonce":1859135164}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":17},{"gid":5836,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/3067106582","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":3067106582},{"type":"v1","addr":"172.21.15.62:6835","nonce":3067106582}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":17},{"gid":6462,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/717991089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":717991089},{"type":"v1","addr":"172.21.15.16:6835","nonce":717991089}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":17}],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-05-22T15:40:02.754483+0000","modified":"2023-05-22T15:40:05.773388+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6468},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6468":{"gid":6468,"name":"a","rank":0,"incarnation":18,"state":"up:active","state_seq":3,"addr":"172.21.15.16:6837/4263396864","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":4263396864},{"type":"v1","addr":"172.21.15.16:6837","nonce":4263396864}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2023-05-22T15:42:08.110 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 22 2023-05-22T15:42:08.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.108+0000 7fb585817640 1 -- 172.21.15.16:0/4173520494 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fb56006b710 msgr2=0x7fb56006daf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:08.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.108+0000 7fb585817640 1 --2- 172.21.15.16:0/4173520494 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fb56006b710 0x7fb56006daf0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fb5680af000 tx=0x7fb568008040 comp rx=0 tx=0).stop 2023-05-22T15:42:08.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.108+0000 7fb585817640 1 -- 172.21.15.16:0/4173520494 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb580070df0 msgr2=0x7fb5801f59c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:08.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.108+0000 7fb585817640 1 --2- 172.21.15.16:0/4173520494 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb580070df0 0x7fb5801f59c0 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7fb578009d10 tx=0x7fb57800c490 comp rx=0 tx=0).stop 2023-05-22T15:42:08.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.108+0000 7fb585817640 1 -- 172.21.15.16:0/4173520494 shutdown_connections 2023-05-22T15:42:08.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.108+0000 7fb585817640 1 --2- 172.21.15.16:0/4173520494 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fb56006b710 0x7fb56006daf0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:08.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.108+0000 7fb585817640 1 --2- 172.21.15.16:0/4173520494 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb5800ca4e0 0x7fb5801f4fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:08.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.108+0000 7fb585817640 1 --2- 172.21.15.16:0/4173520494 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb5800c9b40 0x7fb5801f3f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:08.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.108+0000 7fb585817640 1 --2- 172.21.15.16:0/4173520494 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb580070df0 0x7fb5801f59c0 unknown :-1 s=CLOSED pgs=1022 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:08.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.108+0000 7fb585817640 1 -- 172.21.15.16:0/4173520494 >> 172.21.15.16:0/4173520494 conn(0x7fb580069720 msgr2=0x7fb580159580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:08.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.108+0000 7fb585817640 1 -- 172.21.15.16:0/4173520494 shutdown_connections 2023-05-22T15:42:08.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.108+0000 7fb585817640 1 -- 172.21.15.16:0/4173520494 wait complete. 2023-05-22T15:42:08.127 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-05-22T15:42:08.250 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.244+0000 7f4cd94f7640 1 Processor -- start 2023-05-22T15:42:08.250 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.244+0000 7f4cd94f7640 1 -- start start 2023-05-22T15:42:08.251 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.248+0000 7f4cd94f7640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f4cd40ca4e0 0x7f4cd415e460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:08.251 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.248+0000 7f4cd94f7640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f4cd4070df0 0x7f4cd4071210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:08.252 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.248+0000 7f4cd94f7640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4cd40b0650 con 0x7f4cd40c9b40 2023-05-22T15:42:08.252 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.248+0000 7f4cd94f7640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4cd40b0370 con 0x7f4cd4070df0 2023-05-22T15:42:08.252 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.248+0000 7f4cd94f7640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4cd415d980 con 0x7f4cd40ca4e0 2023-05-22T15:42:08.252 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.248+0000 7f4cd37fe640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f4cd4070df0 0x7f4cd4071210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:08.252 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.248+0000 7f4cd37fe640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f4cd4070df0 0x7f4cd4071210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51034/0 (socket says 172.21.15.16:51034) 2023-05-22T15:42:08.252 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.248+0000 7f4cd37fe640 1 -- 172.21.15.16:0/3869196115 learned_addr learned my addr 172.21.15.16:0/3869196115 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:42:08.252 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.248+0000 7f4cd2ffd640 1 --2- 172.21.15.16:0/3869196115 >> v2:172.21.15.62:3300/0 conn(0x7f4cd40ca4e0 0x7f4cd415e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:08.253 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.248+0000 7f4cd37fe640 1 -- 172.21.15.16:0/3869196115 >> v2:172.21.15.62:3300/0 conn(0x7f4cd40ca4e0 msgr2=0x7f4cd415e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:42:08.253 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.248+0000 7f4cd37fe640 1 --2- 172.21.15.16:0/3869196115 >> v2:172.21.15.62:3300/0 conn(0x7f4cd40ca4e0 0x7f4cd415e460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:08.253 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.248+0000 7f4cd37fe640 1 -- 172.21.15.16:0/3869196115 >> v1:172.21.15.62:6789/0 conn(0x7f4cd40c9b40 legacy=0x7f4cd40afb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:08.253 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.248+0000 7f4cd37fe640 1 -- 172.21.15.16:0/3869196115 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cd415dc60 con 0x7f4cd4070df0 2023-05-22T15:42:08.254 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.248+0000 7f4cd37fe640 1 --2- 172.21.15.16:0/3869196115 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4cd4070df0 0x7f4cd4071210 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7f4cd40cad40 tx=0x7f4ccc01c9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7f413c492af14c6 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:08.254 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.248+0000 7f4cd1ffb640 1 -- 172.21.15.16:0/3869196115 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ccc0a57c0 con 0x7f4cd4070df0 2023-05-22T15:42:08.254 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.252+0000 7f4cd1ffb640 1 -- 172.21.15.16:0/3869196115 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ccc0a5920 con 0x7f4cd4070df0 2023-05-22T15:42:08.254 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.252+0000 7f4cd1ffb640 1 -- 172.21.15.16:0/3869196115 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ccc0a5cb0 con 0x7f4cd4070df0 2023-05-22T15:42:08.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.252+0000 7f4cd94f7640 1 -- 172.21.15.16:0/3869196115 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4cd4070df0 msgr2=0x7f4cd4071210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:08.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.252+0000 7f4cd94f7640 1 --2- 172.21.15.16:0/3869196115 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4cd4070df0 0x7f4cd4071210 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7f4cd40cad40 tx=0x7f4ccc01c9a0 comp rx=0 tx=0).stop 2023-05-22T15:42:08.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.252+0000 7f4cd94f7640 1 -- 172.21.15.16:0/3869196115 shutdown_connections 2023-05-22T15:42:08.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.252+0000 7f4cd94f7640 1 --2- 172.21.15.16:0/3869196115 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4cd4070df0 0x7f4cd4071210 unknown :-1 s=CLOSED pgs=1023 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:08.256 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.252+0000 7f4cd94f7640 1 --2- 172.21.15.16:0/3869196115 >> v2:172.21.15.62:3300/0 conn(0x7f4cd40ca4e0 0x7f4cd415e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:08.256 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.252+0000 7f4cd94f7640 1 -- 172.21.15.16:0/3869196115 >> 172.21.15.16:0/3869196115 conn(0x7f4cd4069720 msgr2=0x7f4cd4069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:08.256 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.252+0000 7f4cd94f7640 1 -- 172.21.15.16:0/3869196115 shutdown_connections 2023-05-22T15:42:08.256 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.252+0000 7f4cd94f7640 1 -- 172.21.15.16:0/3869196115 wait complete. 2023-05-22T15:42:08.258 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.252+0000 7f4cd94f7640 1 Processor -- start 2023-05-22T15:42:08.258 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.252+0000 7f4cd94f7640 1 -- start start 2023-05-22T15:42:08.258 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.252+0000 7f4cd94f7640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4cd4070df0 0x7f4cd40b2030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:08.258 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.252+0000 7f4cd94f7640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4cd40c9b40 0x7f4cd40b2570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:08.258 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.252+0000 7f4cd94f7640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4cd40ca4e0 0x7f4cd41f7ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:08.258 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.252+0000 7f4cd94f7640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cd41f77a0 con 0x7f4cd4070df0 2023-05-22T15:42:08.259 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.252+0000 7f4cd94f7640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cd4142b30 con 0x7f4cd40ca4e0 2023-05-22T15:42:08.259 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.252+0000 7f4cd94f7640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4cd4142200 con 0x7f4cd40c9b40 2023-05-22T15:42:08.259 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.252+0000 7f4cd2ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4cd4070df0 0x7f4cd40b2030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:08.259 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.252+0000 7f4cd2ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4cd4070df0 0x7f4cd40b2030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51044/0 (socket says 172.21.15.16:51044) 2023-05-22T15:42:08.259 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.252+0000 7f4cd2ffd640 1 -- 172.21.15.16:0/188816539 learned_addr learned my addr 172.21.15.16:0/188816539 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:42:08.259 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.252+0000 7f4cd27fc640 1 --2- 172.21.15.16:0/188816539 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4cd40c9b40 0x7f4cd40b2570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:08.259 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.252+0000 7f4cd37fe640 1 --2- 172.21.15.16:0/188816539 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4cd40ca4e0 0x7f4cd41f7ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:08.260 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.252+0000 7f4cd2ffd640 1 -- 172.21.15.16:0/188816539 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4cd40c9b40 msgr2=0x7f4cd40b2570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:08.260 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.252+0000 7f4cd2ffd640 1 --2- 172.21.15.16:0/188816539 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4cd40c9b40 0x7f4cd40b2570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:08.260 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.252+0000 7f4cd2ffd640 1 -- 172.21.15.16:0/188816539 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4cd40ca4e0 msgr2=0x7f4cd41f7ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:08.260 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.252+0000 7f4cd2ffd640 1 --2- 172.21.15.16:0/188816539 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4cd40ca4e0 0x7f4cd41f7ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:08.260 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.252+0000 7f4cd2ffd640 1 -- 172.21.15.16:0/188816539 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ccc009b30 con 0x7f4cd4070df0 2023-05-22T15:42:08.260 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.252+0000 7f4cd37fe640 1 --2- 172.21.15.16:0/188816539 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4cd40ca4e0 0x7f4cd41f7ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:42:08.260 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.252+0000 7f4cd2ffd640 1 --2- 172.21.15.16:0/188816539 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4cd4070df0 0x7f4cd40b2030 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7f4cc400d5f0 tx=0x7f4cc400dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:08.261 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.252+0000 7f4cbb7fe640 1 -- 172.21.15.16:0/188816539 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cc40125f0 con 0x7f4cd4070df0 2023-05-22T15:42:08.261 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.252+0000 7f4cbb7fe640 1 -- 172.21.15.16:0/188816539 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4cc4012750 con 0x7f4cd4070df0 2023-05-22T15:42:08.261 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.252+0000 7f4cd94f7640 1 -- 172.21.15.16:0/188816539 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4cd40c8e80 con 0x7f4cd4070df0 2023-05-22T15:42:08.261 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.252+0000 7f4cbb7fe640 1 -- 172.21.15.16:0/188816539 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cc4012b00 con 0x7f4cd4070df0 2023-05-22T15:42:08.261 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.252+0000 7f4cd94f7640 1 -- 172.21.15.16:0/188816539 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4cd40ca8b0 con 0x7f4cd4070df0 2023-05-22T15:42:08.262 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.256+0000 7f4cd94f7640 1 -- 172.21.15.16:0/188816539 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4cd40c9010 con 0x7f4cd4070df0 2023-05-22T15:42:08.262 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.256+0000 7f4cbb7fe640 1 -- 172.21.15.16:0/188816539 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 25) v1 ==== 84068+0+0 (secure 0 0 0) 0x7f4cc4012c60 con 0x7f4cd4070df0 2023-05-22T15:42:08.262 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.256+0000 7f4cbb7fe640 1 --2- 172.21.15.16:0/188816539 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f4cc0065200 0x7f4cc00676b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:08.262 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.256+0000 7f4cbb7fe640 1 -- 172.21.15.16:0/188816539 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7012+0+0 (secure 0 0 0) 0x7f4cc4082d60 con 0x7f4cd4070df0 2023-05-22T15:42:08.262 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.256+0000 7f4cd27fc640 1 --2- 172.21.15.16:0/188816539 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f4cc0065200 0x7f4cc00676b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:08.263 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.256+0000 7f4cd27fc640 1 --2- 172.21.15.16:0/188816539 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f4cc0065200 0x7f4cc00676b0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f4cd40c5300 tx=0x7f4cc8009040 comp rx=0 tx=0).ready entity=mgr.6765 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:08.264 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.260+0000 7f4cbb7fe640 1 -- 172.21.15.16:0/188816539 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f4cc4050960 con 0x7f4cd4070df0 2023-05-22T15:42:08.418 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.412+0000 7f4cd94f7640 1 -- 172.21.15.16:0/188816539 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4cd404dbe0 con 0x7f4cd4070df0 2023-05-22T15:42:08.421 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.416+0000 7f4cbb7fe640 1 -- 172.21.15.16:0/188816539 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v77) v1 ==== 74+0+18963 (secure 0 0 0) 0x7f4cc4053f80 con 0x7f4cd4070df0 2023-05-22T15:42:08.421 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:42:08.422 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":77,"fsid":"b2719834-7847-4d98-90a6-f65d57f85c49","created":"2023-05-22T15:26:25.259314+0000","modified":"2023-05-22T15:42:07.031163+0000","last_up_change":"2023-05-22T15:26:36.448524+0000","last_in_change":"2023-05-22T15:26:27.675668+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-05-22T15:26:38.019781+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-05-22T15:26:43.570514+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2023-05-22T15:40:00.663040+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"72","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2023-05-22T15:40:01.750680+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":5,"snap_epoch":76,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9e7ff854-b25f-4cbb-a5c2-1799d9c53d2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":37021},{"type":"v1","addr":"172.21.15.16:6825","nonce":37021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":37021},{"type":"v1","addr":"172.21.15.16:6827","nonce":37021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":37021},{"type":"v1","addr":"172.21.15.16:6831","nonce":37021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":37021},{"type":"v1","addr":"172.21.15.16:6829","nonce":37021}]},"public_addr":"172.21.15.16:6825/37021","cluster_addr":"172.21.15.16:6827/37021","heartbeat_back_addr":"172.21.15.16:6831/37021","heartbeat_front_addr":"172.21.15.16:6829/37021","state":["exists","up"]},{"osd":1,"uuid":"4702fbc9-a927-4565-aca5-41dc69cc8264","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":37008},{"type":"v1","addr":"172.21.15.16:6801","nonce":37008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":37008},{"type":"v1","addr":"172.21.15.16:6803","nonce":37008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":37008},{"type":"v1","addr":"172.21.15.16:6807","nonce":37008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":37008},{"type":"v1","addr":"172.21.15.16:6805","nonce":37008}]},"public_addr":"172.21.15.16:6801/37008","cluster_addr":"172.21.15.16:6803/37008","heartbeat_back_addr":"172.21.15.16:6807/37008","heartbeat_front_addr":"172.21.15.16:6805/37008","state":["exists","up"]},{"osd":2,"uuid":"1e7bba46-4eba-47ef-bb81-5e2d371638ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":37009},{"type":"v1","addr":"172.21.15.16:6809","nonce":37009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":37009},{"type":"v1","addr":"172.21.15.16:6811","nonce":37009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":37009},{"type":"v1","addr":"172.21.15.16:6815","nonce":37009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":37009},{"type":"v1","addr":"172.21.15.16:6813","nonce":37009}]},"public_addr":"172.21.15.16:6809/37009","cluster_addr":"172.21.15.16:6811/37009","heartbeat_back_addr":"172.21.15.16:6815/37009","heartbeat_front_addr":"172.21.15.16:6813/37009","state":["exists","up"]},{"osd":3,"uuid":"383cacc0-5a88-4904-990e-498ca59ef48a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":37020},{"type":"v1","addr":"172.21.15.16:6817","nonce":37020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":37020},{"type":"v1","addr":"172.21.15.16:6819","nonce":37020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":37020},{"type":"v1","addr":"172.21.15.16:6823","nonce":37020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":37020},{"type":"v1","addr":"172.21.15.16:6821","nonce":37020}]},"public_addr":"172.21.15.16:6817/37020","cluster_addr":"172.21.15.16:6819/37020","heartbeat_back_addr":"172.21.15.16:6823/37020","heartbeat_front_addr":"172.21.15.16:6821/37020","state":["exists","up"]},{"osd":4,"uuid":"970500ef-95b7-42c1-b080-eed4fae7013d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":37353},{"type":"v1","addr":"172.21.15.62:6817","nonce":37353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6819","nonce":37353},{"type":"v1","addr":"172.21.15.62:6822","nonce":37353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37353},{"type":"v1","addr":"172.21.15.62:6831","nonce":37353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37353},{"type":"v1","addr":"172.21.15.62:6826","nonce":37353}]},"public_addr":"172.21.15.62:6817/37353","cluster_addr":"172.21.15.62:6822/37353","heartbeat_back_addr":"172.21.15.62:6831/37353","heartbeat_front_addr":"172.21.15.62:6826/37353","state":["exists","up"]},{"osd":5,"uuid":"022d67db-d4d8-445b-b1b9-8b446a00f5c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":37352},{"type":"v1","addr":"172.21.15.62:6801","nonce":37352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":37352},{"type":"v1","addr":"172.21.15.62:6803","nonce":37352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":37352},{"type":"v1","addr":"172.21.15.62:6807","nonce":37352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":37352},{"type":"v1","addr":"172.21.15.62:6805","nonce":37352}]},"public_addr":"172.21.15.62:6801/37352","cluster_addr":"172.21.15.62:6803/37352","heartbeat_back_addr":"172.21.15.62:6807/37352","heartbeat_front_addr":"172.21.15.62:6805/37352","state":["exists","up"]},{"osd":6,"uuid":"42bd372b-6688-4a1b-8407-e16e8c266e33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":37354},{"type":"v1","addr":"172.21.15.62:6820","nonce":37354}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":37354},{"type":"v1","addr":"172.21.15.62:6823","nonce":37354}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6829","nonce":37354},{"type":"v1","addr":"172.21.15.62:6830","nonce":37354}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6825","nonce":37354},{"type":"v1","addr":"172.21.15.62:6827","nonce":37354}]},"public_addr":"172.21.15.62:6820/37354","cluster_addr":"172.21.15.62:6823/37354","heartbeat_back_addr":"172.21.15.62:6830/37354","heartbeat_front_addr":"172.21.15.62:6827/37354","state":["exists","up"]},{"osd":7,"uuid":"ac70b2a3-f41b-4bf2-b227-7f2cbb55426d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37355},{"type":"v1","addr":"172.21.15.62:6809","nonce":37355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37355},{"type":"v1","addr":"172.21.15.62:6811","nonce":37355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":37355},{"type":"v1","addr":"172.21.15.62:6815","nonce":37355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":37355},{"type":"v1","addr":"172.21.15.62:6813","nonce":37355}]},"public_addr":"172.21.15.62:6809/37355","cluster_addr":"172.21.15.62:6811/37355","heartbeat_back_addr":"172.21.15.62:6815/37355","heartbeat_front_addr":"172.21.15.62:6813/37355","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.368896+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.107273+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.276888+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.287502+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.124791+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.251937+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.255679+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.232797+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:0/1092976379":"2023-05-23T15:42:07.031130+0000","172.21.15.16:0/1384025485":"2023-05-23T15:42:07.031130+0000","172.21.15.16:0/1795855983":"2023-05-23T15:42:07.031130+0000","172.21.15.16:0/2809310279":"2023-05-23T15:42:07.031130+0000","172.21.15.16:0/1661455459":"2023-05-23T15:42:07.031130+0000","172.21.15.16:0/3341986833":"2023-05-23T15:42:07.031130+0000","172.21.15.16:0/2468042232":"2023-05-23T15:40:15.816078+0000","172.21.15.16:0/1582569975":"2023-05-23T15:40:15.816078+0000","172.21.15.16:0/2252481002":"2023-05-23T15:40:15.816078+0000","172.21.15.16:0/174222799":"2023-05-23T15:40:15.816078+0000","172.21.15.16:0/4017778835":"2023-05-23T15:40:15.816078+0000","172.21.15.16:0/3438310522":"2023-05-23T15:40:15.816078+0000","172.21.15.16:6833/36921":"2023-05-23T15:40:15.816078+0000","172.21.15.16:6832/36921":"2023-05-23T15:40:15.816078+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":9,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-05-22T15:42:08.423 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.420+0000 7f4cd94f7640 1 -- 172.21.15.16:0/188816539 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f4cc0065200 msgr2=0x7f4cc00676b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:08.423 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.420+0000 7f4cd94f7640 1 --2- 172.21.15.16:0/188816539 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f4cc0065200 0x7f4cc00676b0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f4cd40c5300 tx=0x7f4cc8009040 comp rx=0 tx=0).stop 2023-05-22T15:42:08.423 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.420+0000 7f4cd94f7640 1 -- 172.21.15.16:0/188816539 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4cd4070df0 msgr2=0x7f4cd40b2030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:08.423 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.420+0000 7f4cd94f7640 1 --2- 172.21.15.16:0/188816539 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4cd4070df0 0x7f4cd40b2030 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7f4cc400d5f0 tx=0x7f4cc400dab0 comp rx=0 tx=0).stop 2023-05-22T15:42:08.423 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.420+0000 7f4cd94f7640 1 -- 172.21.15.16:0/188816539 shutdown_connections 2023-05-22T15:42:08.424 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.420+0000 7f4cd94f7640 1 --2- 172.21.15.16:0/188816539 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4cd40ca4e0 0x7f4cd41f7ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:08.424 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.420+0000 7f4cd94f7640 1 --2- 172.21.15.16:0/188816539 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f4cc0065200 0x7f4cc00676b0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:08.424 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.420+0000 7f4cd94f7640 1 --2- 172.21.15.16:0/188816539 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4cd40c9b40 0x7f4cd40b2570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:08.424 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.420+0000 7f4cd94f7640 1 --2- 172.21.15.16:0/188816539 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4cd4070df0 0x7f4cd40b2030 unknown :-1 s=CLOSED pgs=1024 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:08.424 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.420+0000 7f4cd94f7640 1 -- 172.21.15.16:0/188816539 >> 172.21.15.16:0/188816539 conn(0x7f4cd4069720 msgr2=0x7f4cd40acc00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:08.424 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.420+0000 7f4cd94f7640 1 -- 172.21.15.16:0/188816539 shutdown_connections 2023-05-22T15:42:08.424 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.420+0000 7f4cd94f7640 1 -- 172.21.15.16:0/188816539 wait complete. 2023-05-22T15:42:08.441 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-05-22T15:42:08.441 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-05-22T15:42:08.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.556+0000 7f4af28ef640 1 Processor -- start 2023-05-22T15:42:08.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.560+0000 7f4af28ef640 1 -- start start 2023-05-22T15:42:08.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.560+0000 7f4af28ef640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f4aec0ca4e0 0x7f4aec15e460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:08.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.560+0000 7f4af28ef640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f4aec0c9b40 0x7f4aec0afb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:08.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.560+0000 7f4af28ef640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4aec070df0 0x7f4aec071210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:08.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.560+0000 7f4af28ef640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4aec0b0650 con 0x7f4aec0c9b40 2023-05-22T15:42:08.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.560+0000 7f4af28ef640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4aec0b0370 con 0x7f4aec0ca4e0 2023-05-22T15:42:08.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.560+0000 7f4af28ef640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4aec15d980 con 0x7f4aec070df0 2023-05-22T15:42:08.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.560+0000 7f4ae3fff640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f4aec0c9b40 0x7f4aec0afb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:08.565 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.560+0000 7f4ae3fff640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f4aec0c9b40 0x7f4aec0afb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51054/0 (socket says 172.21.15.16:51054) 2023-05-22T15:42:08.565 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.560+0000 7f4ae3fff640 1 -- 172.21.15.16:0/3289445832 learned_addr learned my addr 172.21.15.16:0/3289445832 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:42:08.565 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.560+0000 7f4aebfff640 1 --2- 172.21.15.16:0/3289445832 >> v2:172.21.15.62:3300/0 conn(0x7f4aec0ca4e0 0x7f4aec15e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:08.565 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.560+0000 7f4af0e8c640 1 --2- 172.21.15.16:0/3289445832 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4aec070df0 0x7f4aec071210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:08.565 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.560+0000 7f4ae3fff640 1 -- 172.21.15.16:0/3289445832 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4aec070df0 msgr2=0x7f4aec071210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:42:08.565 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.560+0000 7f4ae3fff640 1 --2- 172.21.15.16:0/3289445832 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4aec070df0 0x7f4aec071210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:08.565 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.560+0000 7f4ae3fff640 1 -- 172.21.15.16:0/3289445832 >> v2:172.21.15.62:3300/0 conn(0x7f4aec0ca4e0 msgr2=0x7f4aec15e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:42:08.566 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.560+0000 7f4ae3fff640 1 --2- 172.21.15.16:0/3289445832 >> v2:172.21.15.62:3300/0 conn(0x7f4aec0ca4e0 0x7f4aec15e460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:08.566 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.560+0000 7f4ae3fff640 1 -- 172.21.15.16:0/3289445832 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4aec15dc60 con 0x7f4aec0c9b40 2023-05-22T15:42:08.566 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.560+0000 7f4aebfff640 1 --2- 172.21.15.16:0/3289445832 >> v2:172.21.15.62:3300/0 conn(0x7f4aec0ca4e0 0x7f4aec15e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:42:08.568 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.560+0000 7f4ae3fff640 1 --2- 172.21.15.16:0/3289445832 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4aec0c9b40 0x7f4aec0afb40 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7f4aec0cad40 tx=0x7f4ad401c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19866f902bf38d1b server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:08.568 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.560+0000 7f4aeb7fe640 1 -- 172.21.15.16:0/3289445832 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ad40ab670 con 0x7f4aec0c9b40 2023-05-22T15:42:08.568 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.564+0000 7f4aeb7fe640 1 -- 172.21.15.16:0/3289445832 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ad40ab7d0 con 0x7f4aec0c9b40 2023-05-22T15:42:08.568 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.564+0000 7f4aeb7fe640 1 -- 172.21.15.16:0/3289445832 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ad40abb60 con 0x7f4aec0c9b40 2023-05-22T15:42:08.568 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.564+0000 7f4af28ef640 1 -- 172.21.15.16:0/3289445832 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4aec0c9b40 msgr2=0x7f4aec0afb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:08.568 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.564+0000 7f4af28ef640 1 --2- 172.21.15.16:0/3289445832 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4aec0c9b40 0x7f4aec0afb40 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7f4aec0cad40 tx=0x7f4ad401c4d0 comp rx=0 tx=0).stop 2023-05-22T15:42:08.569 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.564+0000 7f4af28ef640 1 -- 172.21.15.16:0/3289445832 shutdown_connections 2023-05-22T15:42:08.569 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.564+0000 7f4af28ef640 1 --2- 172.21.15.16:0/3289445832 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4aec0c9b40 0x7f4aec0afb40 unknown :-1 s=CLOSED pgs=1025 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:08.569 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.564+0000 7f4af28ef640 1 --2- 172.21.15.16:0/3289445832 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4aec070df0 0x7f4aec071210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:08.569 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.564+0000 7f4af28ef640 1 --2- 172.21.15.16:0/3289445832 >> v2:172.21.15.62:3300/0 conn(0x7f4aec0ca4e0 0x7f4aec15e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:08.569 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.564+0000 7f4af28ef640 1 -- 172.21.15.16:0/3289445832 >> 172.21.15.16:0/3289445832 conn(0x7f4aec069720 msgr2=0x7f4aec069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:08.569 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.564+0000 7f4af28ef640 1 -- 172.21.15.16:0/3289445832 shutdown_connections 2023-05-22T15:42:08.569 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.564+0000 7f4af28ef640 1 -- 172.21.15.16:0/3289445832 wait complete. 2023-05-22T15:42:08.571 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.564+0000 7f4af28ef640 1 Processor -- start 2023-05-22T15:42:08.571 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.564+0000 7f4af28ef640 1 -- start start 2023-05-22T15:42:08.571 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.564+0000 7f4af28ef640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4aec070df0 0x7f4aec0af840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:08.571 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.564+0000 7f4af28ef640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4aec0c9b40 0x7f4aec1f8390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:08.572 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.564+0000 7f4af28ef640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4aec0ca4e0 0x7f4aec1f4420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:08.572 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.564+0000 7f4af28ef640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4aec0afd80 con 0x7f4aec0c9b40 2023-05-22T15:42:08.572 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.564+0000 7f4af28ef640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4aec1f5430 con 0x7f4aec0ca4e0 2023-05-22T15:42:08.572 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.564+0000 7f4af28ef640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4aec1f4e70 con 0x7f4aec070df0 2023-05-22T15:42:08.572 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.564+0000 7f4ae3fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4aec0c9b40 0x7f4aec1f8390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:08.572 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.564+0000 7f4ae3fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4aec0c9b40 0x7f4aec1f8390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51060/0 (socket says 172.21.15.16:51060) 2023-05-22T15:42:08.573 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.564+0000 7f4ae3fff640 1 -- 172.21.15.16:0/2524186465 learned_addr learned my addr 172.21.15.16:0/2524186465 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:42:08.573 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.564+0000 7f4af0e8c640 1 --2- 172.21.15.16:0/2524186465 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4aec0ca4e0 0x7f4aec1f4420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:08.573 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.564+0000 7f4aebfff640 1 --2- 172.21.15.16:0/2524186465 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4aec070df0 0x7f4aec0af840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:08.573 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.568+0000 7f4ae3fff640 1 -- 172.21.15.16:0/2524186465 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4aec070df0 msgr2=0x7f4aec0af840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:08.573 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.568+0000 7f4ae3fff640 1 --2- 172.21.15.16:0/2524186465 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4aec070df0 0x7f4aec0af840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:08.573 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.568+0000 7f4ae3fff640 1 -- 172.21.15.16:0/2524186465 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4aec0ca4e0 msgr2=0x7f4aec1f4420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:08.573 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.568+0000 7f4ae3fff640 1 --2- 172.21.15.16:0/2524186465 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4aec0ca4e0 0x7f4aec1f4420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:08.573 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.568+0000 7f4ae3fff640 1 -- 172.21.15.16:0/2524186465 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ad40094d0 con 0x7f4aec0c9b40 2023-05-22T15:42:08.574 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.568+0000 7f4ae3fff640 1 --2- 172.21.15.16:0/2524186465 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4aec0c9b40 0x7f4aec1f8390 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7f4ad401ab80 tx=0x7f4ad401ad10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:08.574 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.568+0000 7f4ae97fa640 1 -- 172.21.15.16:0/2524186465 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ad40b4800 con 0x7f4aec0c9b40 2023-05-22T15:42:08.574 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.568+0000 7f4ae97fa640 1 -- 172.21.15.16:0/2524186465 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ad40b4980 con 0x7f4aec0c9b40 2023-05-22T15:42:08.574 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.568+0000 7f4af28ef640 1 -- 172.21.15.16:0/2524186465 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4aec0ca8b0 con 0x7f4aec0c9b40 2023-05-22T15:42:08.574 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.568+0000 7f4ae97fa640 1 -- 172.21.15.16:0/2524186465 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ad40b4d30 con 0x7f4aec0c9b40 2023-05-22T15:42:08.574 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.568+0000 7f4af28ef640 1 -- 172.21.15.16:0/2524186465 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4aec071750 con 0x7f4aec0c9b40 2023-05-22T15:42:08.574 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.568+0000 7f4af28ef640 1 -- 172.21.15.16:0/2524186465 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4aec0c8e80 con 0x7f4aec0c9b40 2023-05-22T15:42:08.575 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.568+0000 7f4ae97fa640 1 -- 172.21.15.16:0/2524186465 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 25) v1 ==== 84068+0+0 (secure 0 0 0) 0x7f4ad40b3380 con 0x7f4aec0c9b40 2023-05-22T15:42:08.575 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.568+0000 7f4ae97fa640 1 --2- 172.21.15.16:0/2524186465 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f4acc0651d0 0x7f4acc067680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:08.576 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.568+0000 7f4ae97fa640 1 -- 172.21.15.16:0/2524186465 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7012+0+0 (secure 0 0 0) 0x7f4ad411c970 con 0x7f4aec0c9b40 2023-05-22T15:42:08.576 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.568+0000 7f4aebfff640 1 --2- 172.21.15.16:0/2524186465 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f4acc0651d0 0x7f4acc067680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:08.576 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.568+0000 7f4aebfff640 1 --2- 172.21.15.16:0/2524186465 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f4acc0651d0 0x7f4acc067680 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f4aec0c2d60 tx=0x7f4adc008040 comp rx=0 tx=0).ready entity=mgr.6765 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:08.576 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.572+0000 7f4ae97fa640 1 -- 172.21.15.16:0/2524186465 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f4ad40ea5c0 con 0x7f4aec0c9b40 2023-05-22T15:42:08.758 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.756+0000 7f4af28ef640 1 -- 172.21.15.16:0/2524186465 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4aec04dbe0 con 0x7f4aec0c9b40 2023-05-22T15:42:08.759 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.756+0000 7f4ae97fa640 1 -- 172.21.15.16:0/2524186465 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) v1 ==== 94+0+4645 (secure 0 0 0) 0x7f4ad40edbe0 con 0x7f4aec0c9b40 2023-05-22T15:42:08.760 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:42:08.760 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":22,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5399,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6833/1859135164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":1859135164},{"type":"v1","addr":"172.21.15.62:6833","nonce":1859135164}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":17},{"gid":5836,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/3067106582","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":3067106582},{"type":"v1","addr":"172.21.15.62:6835","nonce":3067106582}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":17},{"gid":6462,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/717991089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":717991089},{"type":"v1","addr":"172.21.15.16:6835","nonce":717991089}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":17}],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-05-22T15:40:02.754483+0000","modified":"2023-05-22T15:40:05.773388+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6468},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6468":{"gid":6468,"name":"a","rank":0,"incarnation":18,"state":"up:active","state_seq":3,"addr":"172.21.15.16:6837/4263396864","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":4263396864},{"type":"v1","addr":"172.21.15.16:6837","nonce":4263396864}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2023-05-22T15:42:08.760 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 22 2023-05-22T15:42:08.761 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.756+0000 7f4af28ef640 1 -- 172.21.15.16:0/2524186465 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f4acc0651d0 msgr2=0x7f4acc067680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:08.762 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.756+0000 7f4af28ef640 1 --2- 172.21.15.16:0/2524186465 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f4acc0651d0 0x7f4acc067680 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f4aec0c2d60 tx=0x7f4adc008040 comp rx=0 tx=0).stop 2023-05-22T15:42:08.762 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.756+0000 7f4af28ef640 1 -- 172.21.15.16:0/2524186465 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4aec0c9b40 msgr2=0x7f4aec1f8390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:08.762 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.756+0000 7f4af28ef640 1 --2- 172.21.15.16:0/2524186465 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4aec0c9b40 0x7f4aec1f8390 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7f4ad401ab80 tx=0x7f4ad401ad10 comp rx=0 tx=0).stop 2023-05-22T15:42:08.762 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.756+0000 7f4af28ef640 1 -- 172.21.15.16:0/2524186465 shutdown_connections 2023-05-22T15:42:08.762 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.756+0000 7f4af28ef640 1 --2- 172.21.15.16:0/2524186465 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4aec0ca4e0 0x7f4aec1f4420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:08.762 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.756+0000 7f4af28ef640 1 --2- 172.21.15.16:0/2524186465 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4aec0c9b40 0x7f4aec1f8390 unknown :-1 s=CLOSED pgs=1026 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:08.762 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.756+0000 7f4af28ef640 1 --2- 172.21.15.16:0/2524186465 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f4acc0651d0 0x7f4acc067680 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:08.763 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.756+0000 7f4af28ef640 1 --2- 172.21.15.16:0/2524186465 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4aec070df0 0x7f4aec0af840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:08.763 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.756+0000 7f4af28ef640 1 -- 172.21.15.16:0/2524186465 >> 172.21.15.16:0/2524186465 conn(0x7f4aec069720 msgr2=0x7f4aec0b27e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:08.763 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.756+0000 7f4af28ef640 1 -- 172.21.15.16:0/2524186465 shutdown_connections 2023-05-22T15:42:08.763 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.756+0000 7f4af28ef640 1 -- 172.21.15.16:0/2524186465 wait complete. 2023-05-22T15:42:08.777 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-05-22T15:42:08.890 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.888+0000 7f3441859640 1 Processor -- start 2023-05-22T15:42:08.891 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.888+0000 7f3441859640 1 -- start start 2023-05-22T15:42:08.892 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.888+0000 7f3441859640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f343c160010 0x7f343c0d17a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:08.892 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.888+0000 7f3441859640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f343c0d0a40 0x7f343c0ced50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:08.892 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.888+0000 7f3441859640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f343c0cf290 0x7f343c163b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:08.892 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.888+0000 7f3441859640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f343c0ce8c0 con 0x7f343c160010 2023-05-22T15:42:08.893 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.888+0000 7f3441859640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f343c0d1000 con 0x7f343c0d0a40 2023-05-22T15:42:08.893 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.888+0000 7f3441859640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f343c0ce270 con 0x7f343c0cf290 2023-05-22T15:42:08.893 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.888+0000 7f343affd640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f343c160010 0x7f343c0d17a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:08.893 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.888+0000 7f343affd640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f343c160010 0x7f343c0d17a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51068/0 (socket says 172.21.15.16:51068) 2023-05-22T15:42:08.893 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.888+0000 7f343affd640 1 -- 172.21.15.16:0/1485534058 learned_addr learned my addr 172.21.15.16:0/1485534058 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:42:08.893 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.888+0000 7f343a7fc640 1 --2- 172.21.15.16:0/1485534058 >> v2:172.21.15.62:3300/0 conn(0x7f343c0d0a40 0x7f343c0ced50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:08.893 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.888+0000 7f343b7fe640 1 --2- 172.21.15.16:0/1485534058 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f343c0cf290 0x7f343c163b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:08.894 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.888+0000 7f343affd640 1 -- 172.21.15.16:0/1485534058 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f343c0cf290 msgr2=0x7f343c163b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:42:08.894 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.888+0000 7f343affd640 1 --2- 172.21.15.16:0/1485534058 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f343c0cf290 0x7f343c163b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:08.894 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.888+0000 7f343affd640 1 -- 172.21.15.16:0/1485534058 >> v2:172.21.15.62:3300/0 conn(0x7f343c0d0a40 msgr2=0x7f343c0ced50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:42:08.894 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.888+0000 7f343affd640 1 --2- 172.21.15.16:0/1485534058 >> v2:172.21.15.62:3300/0 conn(0x7f343c0d0a40 0x7f343c0ced50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:08.894 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.888+0000 7f343affd640 1 -- 172.21.15.16:0/1485534058 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f343c0ce550 con 0x7f343c160010 2023-05-22T15:42:08.895 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.892+0000 7f343affd640 1 --2- 172.21.15.16:0/1485534058 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f343c160010 0x7f343c0d17a0 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7f343c071a60 tx=0x7f342401c6d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3844698b45fca9f server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:08.896 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.892+0000 7f3439ffb640 1 -- 172.21.15.16:0/1485534058 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34240a64b0 con 0x7f343c160010 2023-05-22T15:42:08.896 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.892+0000 7f3439ffb640 1 -- 172.21.15.16:0/1485534058 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34240a6610 con 0x7f343c160010 2023-05-22T15:42:08.896 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.892+0000 7f3439ffb640 1 -- 172.21.15.16:0/1485534058 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34240a69a0 con 0x7f343c160010 2023-05-22T15:42:08.896 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.892+0000 7f3441859640 1 -- 172.21.15.16:0/1485534058 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f343c160010 msgr2=0x7f343c0d17a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:08.896 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.892+0000 7f3441859640 1 --2- 172.21.15.16:0/1485534058 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f343c160010 0x7f343c0d17a0 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7f343c071a60 tx=0x7f342401c6d0 comp rx=0 tx=0).stop 2023-05-22T15:42:08.896 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.892+0000 7f3441859640 1 -- 172.21.15.16:0/1485534058 shutdown_connections 2023-05-22T15:42:08.897 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.892+0000 7f3441859640 1 --2- 172.21.15.16:0/1485534058 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f343c0cf290 0x7f343c163b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:08.897 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.892+0000 7f3441859640 1 --2- 172.21.15.16:0/1485534058 >> v2:172.21.15.62:3300/0 conn(0x7f343c0d0a40 0x7f343c0ced50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:08.897 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.892+0000 7f3441859640 1 --2- 172.21.15.16:0/1485534058 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f343c160010 0x7f343c0d17a0 unknown :-1 s=CLOSED pgs=1027 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:08.897 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.892+0000 7f3441859640 1 -- 172.21.15.16:0/1485534058 >> 172.21.15.16:0/1485534058 conn(0x7f343c0655c0 msgr2=0x7f343c070e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:08.897 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.892+0000 7f3441859640 1 -- 172.21.15.16:0/1485534058 shutdown_connections 2023-05-22T15:42:08.897 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.892+0000 7f3441859640 1 -- 172.21.15.16:0/1485534058 wait complete. 2023-05-22T15:42:08.900 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.892+0000 7f3441859640 1 Processor -- start 2023-05-22T15:42:08.900 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.892+0000 7f3441859640 1 -- start start 2023-05-22T15:42:08.900 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.896+0000 7f3441859640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f343c0cf290 0x7f343c157760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:08.901 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.896+0000 7f3441859640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f343c0d0a40 0x7f343c151740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:08.901 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.896+0000 7f3441859640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f343c160010 0x7f343c0c80d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:08.901 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.896+0000 7f3441859640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f343c15ab80 con 0x7f343c0d0a40 2023-05-22T15:42:08.901 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.896+0000 7f3441859640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f343c163370 con 0x7f343c160010 2023-05-22T15:42:08.901 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.896+0000 7f3441859640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f343c15a5c0 con 0x7f343c0cf290 2023-05-22T15:42:08.901 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.896+0000 7f343a7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f343c0d0a40 0x7f343c151740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:08.901 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.896+0000 7f343a7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f343c0d0a40 0x7f343c151740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51070/0 (socket says 172.21.15.16:51070) 2023-05-22T15:42:08.902 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.896+0000 7f343a7fc640 1 -- 172.21.15.16:0/558279630 learned_addr learned my addr 172.21.15.16:0/558279630 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:42:08.902 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.896+0000 7f343affd640 1 --2- 172.21.15.16:0/558279630 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f343c0cf290 0x7f343c157760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:08.902 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.896+0000 7f343b7fe640 1 --2- 172.21.15.16:0/558279630 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f343c160010 0x7f343c0c80d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:08.902 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.896+0000 7f343a7fc640 1 -- 172.21.15.16:0/558279630 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f343c0cf290 msgr2=0x7f343c157760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:08.902 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.896+0000 7f343a7fc640 1 --2- 172.21.15.16:0/558279630 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f343c0cf290 0x7f343c157760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:08.902 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.896+0000 7f343a7fc640 1 -- 172.21.15.16:0/558279630 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f343c160010 msgr2=0x7f343c0c80d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:08.902 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.896+0000 7f343a7fc640 1 --2- 172.21.15.16:0/558279630 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f343c160010 0x7f343c0c80d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:08.903 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.896+0000 7f343a7fc640 1 -- 172.21.15.16:0/558279630 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3424009b30 con 0x7f343c0d0a40 2023-05-22T15:42:08.903 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.896+0000 7f343affd640 1 --2- 172.21.15.16:0/558279630 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f343c0cf290 0x7f343c157760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:42:08.903 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.896+0000 7f343a7fc640 1 --2- 172.21.15.16:0/558279630 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f343c0d0a40 0x7f343c151740 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7f342c00d800 tx=0x7f342c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:08.903 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.896+0000 7f3433fff640 1 -- 172.21.15.16:0/558279630 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f342c00b6c0 con 0x7f343c0d0a40 2023-05-22T15:42:08.903 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.896+0000 7f3433fff640 1 -- 172.21.15.16:0/558279630 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f342c00b820 con 0x7f343c0d0a40 2023-05-22T15:42:08.903 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.896+0000 7f3433fff640 1 -- 172.21.15.16:0/558279630 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f342c00bb90 con 0x7f343c0d0a40 2023-05-22T15:42:08.903 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.896+0000 7f3441859640 1 -- 172.21.15.16:0/558279630 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f343c0ce040 con 0x7f343c0d0a40 2023-05-22T15:42:08.904 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.896+0000 7f3441859640 1 -- 172.21.15.16:0/558279630 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f343c0cdeb0 con 0x7f343c0d0a40 2023-05-22T15:42:08.904 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.896+0000 7f3441859640 1 -- 172.21.15.16:0/558279630 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f343c0d1ce0 con 0x7f343c0d0a40 2023-05-22T15:42:08.905 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.896+0000 7f3433fff640 1 -- 172.21.15.16:0/558279630 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 25) v1 ==== 84068+0+0 (secure 0 0 0) 0x7f342c00bcf0 con 0x7f343c0d0a40 2023-05-22T15:42:08.905 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.896+0000 7f3433fff640 1 --2- 172.21.15.16:0/558279630 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f341c0651d0 0x7f341c067680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:08.905 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.896+0000 7f343affd640 1 --2- 172.21.15.16:0/558279630 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f341c0651d0 0x7f341c067680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:08.905 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.896+0000 7f3433fff640 1 -- 172.21.15.16:0/558279630 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7012+0+0 (secure 0 0 0) 0x7f342c082d90 con 0x7f343c0d0a40 2023-05-22T15:42:08.905 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.896+0000 7f343affd640 1 --2- 172.21.15.16:0/558279630 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f341c0651d0 0x7f341c067680 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f34240ae000 tx=0x7f34240b1040 comp rx=0 tx=0).ready entity=mgr.6765 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:08.905 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:08.900+0000 7f3433fff640 1 -- 172.21.15.16:0/558279630 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f342c0543c0 con 0x7f343c0d0a40 2023-05-22T15:42:09.092 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:09.088+0000 7f3441859640 1 -- 172.21.15.16:0/558279630 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f343c04dbe0 con 0x7f343c0d0a40 2023-05-22T15:42:09.092 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:09.088+0000 7f3433fff640 1 -- 172.21.15.16:0/558279630 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) v1 ==== 94+0+4645 (secure 0 0 0) 0x7f342c053fb0 con 0x7f343c0d0a40 2023-05-22T15:42:09.093 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:42:09.093 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":22,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5399,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6833/1859135164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":1859135164},{"type":"v1","addr":"172.21.15.62:6833","nonce":1859135164}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":17},{"gid":5836,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/3067106582","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":3067106582},{"type":"v1","addr":"172.21.15.62:6835","nonce":3067106582}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":17},{"gid":6462,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/717991089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":717991089},{"type":"v1","addr":"172.21.15.16:6835","nonce":717991089}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":17}],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-05-22T15:40:02.754483+0000","modified":"2023-05-22T15:40:05.773388+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6468},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6468":{"gid":6468,"name":"a","rank":0,"incarnation":18,"state":"up:active","state_seq":3,"addr":"172.21.15.16:6837/4263396864","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":4263396864},{"type":"v1","addr":"172.21.15.16:6837","nonce":4263396864}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2023-05-22T15:42:09.093 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 22 2023-05-22T15:42:09.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:09.092+0000 7f3441859640 1 -- 172.21.15.16:0/558279630 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f341c0651d0 msgr2=0x7f341c067680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:09.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:09.092+0000 7f3441859640 1 --2- 172.21.15.16:0/558279630 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f341c0651d0 0x7f341c067680 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f34240ae000 tx=0x7f34240b1040 comp rx=0 tx=0).stop 2023-05-22T15:42:09.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:09.092+0000 7f3441859640 1 -- 172.21.15.16:0/558279630 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f343c0d0a40 msgr2=0x7f343c151740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:09.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:09.092+0000 7f3441859640 1 --2- 172.21.15.16:0/558279630 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f343c0d0a40 0x7f343c151740 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7f342c00d800 tx=0x7f342c00dcc0 comp rx=0 tx=0).stop 2023-05-22T15:42:09.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:09.092+0000 7f3441859640 1 -- 172.21.15.16:0/558279630 shutdown_connections 2023-05-22T15:42:09.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:09.092+0000 7f3441859640 1 --2- 172.21.15.16:0/558279630 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f343c160010 0x7f343c0c80d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:09.096 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:09.092+0000 7f3441859640 1 --2- 172.21.15.16:0/558279630 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f343c0d0a40 0x7f343c151740 unknown :-1 s=CLOSED pgs=1028 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:09.096 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:09.092+0000 7f3441859640 1 --2- 172.21.15.16:0/558279630 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f341c0651d0 0x7f341c067680 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:09.096 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:09.092+0000 7f3441859640 1 --2- 172.21.15.16:0/558279630 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f343c0cf290 0x7f343c157760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:09.096 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:09.092+0000 7f3441859640 1 -- 172.21.15.16:0/558279630 >> 172.21.15.16:0/558279630 conn(0x7f343c0655c0 msgr2=0x7f343c070e30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:09.096 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:09.092+0000 7f3441859640 1 -- 172.21.15.16:0/558279630 shutdown_connections 2023-05-22T15:42:09.096 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:09.092+0000 7f3441859640 1 -- 172.21.15.16:0/558279630 wait complete. 2023-05-22T15:42:09.111 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-05-22T15:42:09.222 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:09.220+0000 7fe7b64dd640 1 Processor -- start 2023-05-22T15:42:09.223 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:09.220+0000 7fe7b64dd640 1 -- start start 2023-05-22T15:42:09.225 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:09.220+0000 7fe7b64dd640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fe7b0070fb0 0x7fe7b00cd1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:09.225 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:09.220+0000 7fe7b64dd640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe7b00affa0 0x7fe7b0160400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:09.226 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:09.220+0000 7fe7b64dd640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7b0160940 con 0x7fe7b00afb40 2023-05-22T15:42:09.226 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:09.220+0000 7fe7b64dd640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7b00cd8d0 con 0x7fe7b0070fb0 2023-05-22T15:42:09.226 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:09.220+0000 7fe7b64dd640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7b0164de0 con 0x7fe7b00affa0 2023-05-22T15:42:09.226 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:09.220+0000 7fe7af7fe640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fe7b00afb40 0x7fe7b015fcf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:41648/0 (socket says 172.21.15.16:41648) 2023-05-22T15:42:09.226 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:09.220+0000 7fe7af7fe640 1 -- 172.21.15.16:0/3152401087 learned_addr learned my addr 172.21.15.16:0/3152401087 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:42:09.226 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:09.220+0000 7fe7b4a7a640 1 --2- 172.21.15.16:0/3152401087 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe7b00affa0 0x7fe7b0160400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:09.227 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:09.220+0000 7fe7affff640 1 --2- 172.21.15.16:0/3152401087 >> v2:172.21.15.62:3300/0 conn(0x7fe7b0070fb0 0x7fe7b00cd1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:09.227 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:09.220+0000 7fe7aeffd640 1 -- 172.21.15.16:0/3152401087 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2359409989 0 0) 0x7fe7b0160940 con 0x7fe7b00afb40 2023-05-22T15:42:09.227 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:09.220+0000 7fe7aeffd640 1 -- 172.21.15.16:0/3152401087 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe79c003530 con 0x7fe7b00afb40 2023-05-22T15:42:09.227 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:09.220+0000 7fe7aeffd640 1 -- 172.21.15.16:0/3152401087 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4067339400 0 0) 0x7fe79c003530 con 0x7fe7b00afb40 2023-05-22T15:42:09.227 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:09.220+0000 7fe7aeffd640 1 -- 172.21.15.16:0/3152401087 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe7b00affa0 msgr2=0x7fe7b0160400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:42:09.227 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:09.220+0000 7fe7aeffd640 1 --2- 172.21.15.16:0/3152401087 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe7b00affa0 0x7fe7b0160400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:09.228 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:09.220+0000 7fe7aeffd640 1 -- 172.21.15.16:0/3152401087 >> v2:172.21.15.62:3300/0 conn(0x7fe7b0070fb0 msgr2=0x7fe7b00cd1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:42:09.228 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:09.220+0000 7fe7aeffd640 1 --2- 172.21.15.16:0/3152401087 >> v2:172.21.15.62:3300/0 conn(0x7fe7b0070fb0 0x7fe7b00cd1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:09.228 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:09.220+0000 7fe7aeffd640 1 -- 172.21.15.16:0/3152401087 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7b00ca400 con 0x7fe7b00afb40 2023-05-22T15:42:09.228 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:09.220+0000 7fe7aeffd640 1 -- 172.21.15.16:0/3152401087 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7fe798001680 con 0x7fe7b00afb40 2023-05-22T15:42:09.228 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:09.220+0000 7fe7aeffd640 1 -- 172.21.15.16:0/3152401087 >> v1:172.21.15.16:6789/0 conn(0x7fe7b00afb40 legacy=0x7fe7b015fcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:09.228 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:09.220+0000 7fe7aeffd640 1 --2- 172.21.15.16:0/3152401087 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe79c003980 0x7fe79c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:09.229 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:09.220+0000 7fe7aeffd640 1 --2- 172.21.15.16:0/3152401087 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe79c0043b0 0x7fe79c004870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:09.229 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:09.220+0000 7fe7aeffd640 1 --2- 172.21.15.16:0/3152401087 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe7b00affa0 0x7fe79c004f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:09.229 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:09.220+0000 7fe7aeffd640 1 -- 172.21.15.16:0/3152401087 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7b00cd8d0 con 0x7fe79c003980 2023-05-22T15:42:09.229 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:09.220+0000 7fe7aeffd640 1 -- 172.21.15.16:0/3152401087 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7b0164de0 con 0x7fe79c0043b0 2023-05-22T15:42:09.229 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:09.220+0000 7fe7aeffd640 1 -- 172.21.15.16:0/3152401087 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe79c00b990 con 0x7fe7b00affa0 2023-05-22T15:42:09.229 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:09.220+0000 7fe7affff640 1 --2- 172.21.15.16:0/3152401087 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe79c003980 0x7fe79c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:09.229 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:09.220+0000 7fe7b4a7a640 1 --2- 172.21.15.16:0/3152401087 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe7b00affa0 0x7fe79c004f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:09.229 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:09.220+0000 7fe7af7fe640 1 --2- 172.21.15.16:0/3152401087 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe79c0043b0 0x7fe79c004870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:09.230 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:09.220+0000 7fe7affff640 1 -- 172.21.15.16:0/3152401087 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe7b00affa0 msgr2=0x7fe79c004f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:42:09.230 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:09.220+0000 7fe7affff640 1 --2- 172.21.15.16:0/3152401087 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe7b00affa0 0x7fe79c004f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:09.230 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:09.220+0000 7fe7affff640 1 -- 172.21.15.16:0/3152401087 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe79c0043b0 msgr2=0x7fe79c004870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:42:09.230 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:09.220+0000 7fe7affff640 1 --2- 172.21.15.16:0/3152401087 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe79c0043b0 0x7fe79c004870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:09.230 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:09.220+0000 7fe7affff640 1 -- 172.21.15.16:0/3152401087 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe79c003530 con 0x7fe79c003980 2023-05-22T15:42:09.232 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:09.224+0000 7fe7affff640 1 --2- 172.21.15.16:0/3152401087 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe79c003980 0x7fe79c003d90 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7fe7a8009fc0 tx=0x7fe7a801cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf1edb30a9189276 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:09.233 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:09.224+0000 7fe7aeffd640 1 -- 172.21.15.16:0/3152401087 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7a80a5a90 con 0x7fe79c003980 2023-05-22T15:42:09.233 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:09.224+0000 7fe7aeffd640 1 -- 172.21.15.16:0/3152401087 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7a800f040 con 0x7fe79c003980 2023-05-22T15:42:09.233 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:09.224+0000 7fe7aeffd640 1 -- 172.21.15.16:0/3152401087 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7a80a5e30 con 0x7fe79c003980 2023-05-22T15:42:09.233 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:09.224+0000 7fe7b64dd640 1 -- 172.21.15.16:0/3152401087 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe79c003980 msgr2=0x7fe79c003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:09.233 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:09.224+0000 7fe7b64dd640 1 --2- 172.21.15.16:0/3152401087 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe79c003980 0x7fe79c003d90 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7fe7a8009fc0 tx=0x7fe7a801cca0 comp rx=0 tx=0).stop 2023-05-22T15:42:09.233 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:09.224+0000 7fe7b4a7a640 1 -- 172.21.15.16:0/3152401087 reap_dead start 2023-05-22T15:42:09.233 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:09.224+0000 7fe7b64dd640 1 -- 172.21.15.16:0/3152401087 shutdown_connections 2023-05-22T15:42:09.233 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:09.224+0000 7fe7b64dd640 1 -- 172.21.15.16:0/3152401087 >> 172.21.15.16:0/3152401087 conn(0x7fe7b0069720 msgr2=0x7fe7b0069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:09.234 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:09.224+0000 7fe7b64dd640 1 -- 172.21.15.16:0/3152401087 shutdown_connections 2023-05-22T15:42:09.234 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:09.224+0000 7fe7b64dd640 1 -- 172.21.15.16:0/3152401087 wait complete. 2023-05-22T15:42:09.234 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:09.228+0000 7fe7b64dd640 1 Processor -- start 2023-05-22T15:42:09.234 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:09.228+0000 7fe7b64dd640 1 -- start start 2023-05-22T15:42:09.234 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:09.228+0000 7fe7b64dd640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe79c003980 0x7fe7b00c7320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:09.234 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:09.228+0000 7fe7b64dd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe7b0154140 0x7fe7b00c1460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:09.234 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:09.228+0000 7fe7b64dd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe7b00c3540 0x7fe7b015c5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:09.235 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:09.228+0000 7fe7b64dd640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7b00c19a0 con 0x7fe7b0154140 2023-05-22T15:42:09.235 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:09.228+0000 7fe7b64dd640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7b00c7ce0 con 0x7fe7b00c3540 2023-05-22T15:42:09.235 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:09.228+0000 7fe7b64dd640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7b015c130 con 0x7fe79c003980 2023-05-22T15:42:09.235 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:09.228+0000 7fe7af7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe7b0154140 0x7fe7b00c1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:09.235 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:09.228+0000 7fe7af7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe7b0154140 0x7fe7b00c1460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41222/0 (socket says 172.21.15.16:41222) 2023-05-22T15:42:09.235 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:09.228+0000 7fe7af7fe640 1 -- 172.21.15.16:0/3691415795 learned_addr learned my addr 172.21.15.16:0/3691415795 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:42:09.236 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:09.228+0000 7fe7b4a7a640 1 --2- 172.21.15.16:0/3691415795 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe7b00c3540 0x7fe7b015c5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:09.236 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:09.228+0000 7fe7affff640 1 --2- 172.21.15.16:0/3691415795 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe79c003980 0x7fe7b00c7320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:09.236 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:09.228+0000 7fe7af7fe640 1 -- 172.21.15.16:0/3691415795 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe79c003980 msgr2=0x7fe7b00c7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:09.236 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:09.228+0000 7fe7af7fe640 1 --2- 172.21.15.16:0/3691415795 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe79c003980 0x7fe7b00c7320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:09.236 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:09.228+0000 7fe7af7fe640 1 -- 172.21.15.16:0/3691415795 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe7b00c3540 msgr2=0x7fe7b015c5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:09.236 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:09.228+0000 7fe7af7fe640 1 --2- 172.21.15.16:0/3691415795 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe7b00c3540 0x7fe7b015c5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:09.236 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:09.228+0000 7fe7af7fe640 1 -- 172.21.15.16:0/3691415795 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7b00ca040 con 0x7fe7b0154140 2023-05-22T15:42:09.237 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:09.228+0000 7fe7b4a7a640 1 --2- 172.21.15.16:0/3691415795 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe7b00c3540 0x7fe7b015c5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T15:42:09.237 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:09.228+0000 7fe7af7fe640 1 --2- 172.21.15.16:0/3691415795 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe7b0154140 0x7fe7b00c1460 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7fe79800d7a0 tx=0x7fe79800dba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:09.237 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:09.228+0000 7fe7acff9640 1 -- 172.21.15.16:0/3691415795 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7980072d0 con 0x7fe7b0154140 2023-05-22T15:42:09.237 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:09.228+0000 7fe7acff9640 1 -- 172.21.15.16:0/3691415795 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe798007430 con 0x7fe7b0154140 2023-05-22T15:42:09.237 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:09.228+0000 7fe7b64dd640 1 -- 172.21.15.16:0/3691415795 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7b00b03a0 con 0x7fe7b0154140 2023-05-22T15:42:09.237 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:09.228+0000 7fe7acff9640 1 -- 172.21.15.16:0/3691415795 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7980047f0 con 0x7fe7b0154140 2023-05-22T15:42:09.237 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:09.228+0000 7fe7b64dd640 1 -- 172.21.15.16:0/3691415795 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7b00ca1d0 con 0x7fe7b0154140 2023-05-22T15:42:09.238 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:09.228+0000 7fe7b64dd640 1 -- 172.21.15.16:0/3691415795 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7b004dbe0 con 0x7fe7b0154140 2023-05-22T15:42:09.238 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:09.228+0000 7fe7acff9640 1 -- 172.21.15.16:0/3691415795 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 25) v1 ==== 84068+0+0 (secure 0 0 0) 0x7fe798004950 con 0x7fe7b0154140 2023-05-22T15:42:09.239 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:09.228+0000 7fe7acff9640 1 --2- 172.21.15.16:0/3691415795 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fe790065200 0x7fe7900676b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:09.239 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:09.228+0000 7fe7acff9640 1 -- 172.21.15.16:0/3691415795 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7012+0+0 (secure 0 0 0) 0x7fe798082060 con 0x7fe7b0154140 2023-05-22T15:42:09.239 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:09.228+0000 7fe7affff640 1 --2- 172.21.15.16:0/3691415795 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fe790065200 0x7fe7900676b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:09.239 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:09.232+0000 7fe7affff640 1 --2- 172.21.15.16:0/3691415795 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fe790065200 0x7fe7900676b0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fe7a80af000 tx=0x7fe7a80a7040 comp rx=0 tx=0).ready entity=mgr.6765 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:09.239 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:09.232+0000 7fe7acff9640 1 -- 172.21.15.16:0/3691415795 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fe79804fc60 con 0x7fe7b0154140 2023-05-22T15:42:09.387 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:09.384+0000 7fe7b64dd640 1 -- 172.21.15.16:0/3691415795 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe7b015c270 con 0x7fe7b0154140 2023-05-22T15:42:09.390 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:09.388+0000 7fe7acff9640 1 -- 172.21.15.16:0/3691415795 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v77) v1 ==== 74+0+18963 (secure 0 0 0) 0x7fe798019090 con 0x7fe7b0154140 2023-05-22T15:42:09.391 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:42:09.391 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":77,"fsid":"b2719834-7847-4d98-90a6-f65d57f85c49","created":"2023-05-22T15:26:25.259314+0000","modified":"2023-05-22T15:42:07.031163+0000","last_up_change":"2023-05-22T15:26:36.448524+0000","last_in_change":"2023-05-22T15:26:27.675668+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-05-22T15:26:38.019781+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-05-22T15:26:43.570514+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2023-05-22T15:40:00.663040+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"72","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2023-05-22T15:40:01.750680+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":5,"snap_epoch":76,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9e7ff854-b25f-4cbb-a5c2-1799d9c53d2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":37021},{"type":"v1","addr":"172.21.15.16:6825","nonce":37021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":37021},{"type":"v1","addr":"172.21.15.16:6827","nonce":37021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":37021},{"type":"v1","addr":"172.21.15.16:6831","nonce":37021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":37021},{"type":"v1","addr":"172.21.15.16:6829","nonce":37021}]},"public_addr":"172.21.15.16:6825/37021","cluster_addr":"172.21.15.16:6827/37021","heartbeat_back_addr":"172.21.15.16:6831/37021","heartbeat_front_addr":"172.21.15.16:6829/37021","state":["exists","up"]},{"osd":1,"uuid":"4702fbc9-a927-4565-aca5-41dc69cc8264","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":37008},{"type":"v1","addr":"172.21.15.16:6801","nonce":37008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":37008},{"type":"v1","addr":"172.21.15.16:6803","nonce":37008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":37008},{"type":"v1","addr":"172.21.15.16:6807","nonce":37008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":37008},{"type":"v1","addr":"172.21.15.16:6805","nonce":37008}]},"public_addr":"172.21.15.16:6801/37008","cluster_addr":"172.21.15.16:6803/37008","heartbeat_back_addr":"172.21.15.16:6807/37008","heartbeat_front_addr":"172.21.15.16:6805/37008","state":["exists","up"]},{"osd":2,"uuid":"1e7bba46-4eba-47ef-bb81-5e2d371638ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":37009},{"type":"v1","addr":"172.21.15.16:6809","nonce":37009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":37009},{"type":"v1","addr":"172.21.15.16:6811","nonce":37009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":37009},{"type":"v1","addr":"172.21.15.16:6815","nonce":37009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":37009},{"type":"v1","addr":"172.21.15.16:6813","nonce":37009}]},"public_addr":"172.21.15.16:6809/37009","cluster_addr":"172.21.15.16:6811/37009","heartbeat_back_addr":"172.21.15.16:6815/37009","heartbeat_front_addr":"172.21.15.16:6813/37009","state":["exists","up"]},{"osd":3,"uuid":"383cacc0-5a88-4904-990e-498ca59ef48a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":37020},{"type":"v1","addr":"172.21.15.16:6817","nonce":37020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":37020},{"type":"v1","addr":"172.21.15.16:6819","nonce":37020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":37020},{"type":"v1","addr":"172.21.15.16:6823","nonce":37020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":37020},{"type":"v1","addr":"172.21.15.16:6821","nonce":37020}]},"public_addr":"172.21.15.16:6817/37020","cluster_addr":"172.21.15.16:6819/37020","heartbeat_back_addr":"172.21.15.16:6823/37020","heartbeat_front_addr":"172.21.15.16:6821/37020","state":["exists","up"]},{"osd":4,"uuid":"970500ef-95b7-42c1-b080-eed4fae7013d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":37353},{"type":"v1","addr":"172.21.15.62:6817","nonce":37353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6819","nonce":37353},{"type":"v1","addr":"172.21.15.62:6822","nonce":37353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37353},{"type":"v1","addr":"172.21.15.62:6831","nonce":37353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37353},{"type":"v1","addr":"172.21.15.62:6826","nonce":37353}]},"public_addr":"172.21.15.62:6817/37353","cluster_addr":"172.21.15.62:6822/37353","heartbeat_back_addr":"172.21.15.62:6831/37353","heartbeat_front_addr":"172.21.15.62:6826/37353","state":["exists","up"]},{"osd":5,"uuid":"022d67db-d4d8-445b-b1b9-8b446a00f5c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":37352},{"type":"v1","addr":"172.21.15.62:6801","nonce":37352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":37352},{"type":"v1","addr":"172.21.15.62:6803","nonce":37352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":37352},{"type":"v1","addr":"172.21.15.62:6807","nonce":37352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":37352},{"type":"v1","addr":"172.21.15.62:6805","nonce":37352}]},"public_addr":"172.21.15.62:6801/37352","cluster_addr":"172.21.15.62:6803/37352","heartbeat_back_addr":"172.21.15.62:6807/37352","heartbeat_front_addr":"172.21.15.62:6805/37352","state":["exists","up"]},{"osd":6,"uuid":"42bd372b-6688-4a1b-8407-e16e8c266e33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":37354},{"type":"v1","addr":"172.21.15.62:6820","nonce":37354}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":37354},{"type":"v1","addr":"172.21.15.62:6823","nonce":37354}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6829","nonce":37354},{"type":"v1","addr":"172.21.15.62:6830","nonce":37354}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6825","nonce":37354},{"type":"v1","addr":"172.21.15.62:6827","nonce":37354}]},"public_addr":"172.21.15.62:6820/37354","cluster_addr":"172.21.15.62:6823/37354","heartbeat_back_addr":"172.21.15.62:6830/37354","heartbeat_front_addr":"172.21.15.62:6827/37354","state":["exists","up"]},{"osd":7,"uuid":"ac70b2a3-f41b-4bf2-b227-7f2cbb55426d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37355},{"type":"v1","addr":"172.21.15.62:6809","nonce":37355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37355},{"type":"v1","addr":"172.21.15.62:6811","nonce":37355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":37355},{"type":"v1","addr":"172.21.15.62:6815","nonce":37355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":37355},{"type":"v1","addr":"172.21.15.62:6813","nonce":37355}]},"public_addr":"172.21.15.62:6809/37355","cluster_addr":"172.21.15.62:6811/37355","heartbeat_back_addr":"172.21.15.62:6815/37355","heartbeat_front_addr":"172.21.15.62:6813/37355","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.368896+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.107273+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.276888+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.287502+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.124791+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.251937+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.255679+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.232797+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:0/1092976379":"2023-05-23T15:42:07.031130+0000","172.21.15.16:0/1384025485":"2023-05-23T15:42:07.031130+0000","172.21.15.16:0/1795855983":"2023-05-23T15:42:07.031130+0000","172.21.15.16:0/2809310279":"2023-05-23T15:42:07.031130+0000","172.21.15.16:0/1661455459":"2023-05-23T15:42:07.031130+0000","172.21.15.16:0/3341986833":"2023-05-23T15:42:07.031130+0000","172.21.15.16:0/2468042232":"2023-05-23T15:40:15.816078+0000","172.21.15.16:0/1582569975":"2023-05-23T15:40:15.816078+0000","172.21.15.16:0/2252481002":"2023-05-23T15:40:15.816078+0000","172.21.15.16:0/174222799":"2023-05-23T15:40:15.816078+0000","172.21.15.16:0/4017778835":"2023-05-23T15:40:15.816078+0000","172.21.15.16:0/3438310522":"2023-05-23T15:40:15.816078+0000","172.21.15.16:6833/36921":"2023-05-23T15:40:15.816078+0000","172.21.15.16:6832/36921":"2023-05-23T15:40:15.816078+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":9,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-05-22T15:42:09.393 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:09.388+0000 7fe7b64dd640 1 -- 172.21.15.16:0/3691415795 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fe790065200 msgr2=0x7fe7900676b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:09.393 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:09.388+0000 7fe7b64dd640 1 --2- 172.21.15.16:0/3691415795 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fe790065200 0x7fe7900676b0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fe7a80af000 tx=0x7fe7a80a7040 comp rx=0 tx=0).stop 2023-05-22T15:42:09.393 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:09.388+0000 7fe7b64dd640 1 -- 172.21.15.16:0/3691415795 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe7b0154140 msgr2=0x7fe7b00c1460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:09.393 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:09.388+0000 7fe7b64dd640 1 --2- 172.21.15.16:0/3691415795 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe7b0154140 0x7fe7b00c1460 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7fe79800d7a0 tx=0x7fe79800dba0 comp rx=0 tx=0).stop 2023-05-22T15:42:09.393 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:09.388+0000 7fe7b64dd640 1 -- 172.21.15.16:0/3691415795 shutdown_connections 2023-05-22T15:42:09.394 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:09.388+0000 7fe7b64dd640 1 --2- 172.21.15.16:0/3691415795 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe7b00c3540 0x7fe7b015c5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:09.394 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:09.388+0000 7fe7b64dd640 1 --2- 172.21.15.16:0/3691415795 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe7b0154140 0x7fe7b00c1460 unknown :-1 s=CLOSED pgs=1031 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:09.394 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:09.388+0000 7fe7b64dd640 1 --2- 172.21.15.16:0/3691415795 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fe790065200 0x7fe7900676b0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:09.394 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:09.388+0000 7fe7b64dd640 1 --2- 172.21.15.16:0/3691415795 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe79c003980 0x7fe7b00c7320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:09.394 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:09.388+0000 7fe7b64dd640 1 -- 172.21.15.16:0/3691415795 >> 172.21.15.16:0/3691415795 conn(0x7fe7b0069720 msgr2=0x7fe7b015aa30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:09.394 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:09.388+0000 7fe7b64dd640 1 -- 172.21.15.16:0/3691415795 shutdown_connections 2023-05-22T15:42:09.394 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:09.388+0000 7fe7b64dd640 1 -- 172.21.15.16:0/3691415795 wait complete. 2023-05-22T15:42:09.408 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-05-22T15:42:09.527 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:09.524+0000 7fe64047d640 1 Processor -- start 2023-05-22T15:42:09.527 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:09.524+0000 7fe64047d640 1 -- start start 2023-05-22T15:42:09.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:09.524+0000 7fe64047d640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe6380cf870 0x7fe63806db00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:09.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:09.524+0000 7fe64047d640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fe63806e740 0x7fe63806ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:09.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:09.524+0000 7fe64047d640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe63806e040 con 0x7fe63815cfa0 2023-05-22T15:42:09.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:09.524+0000 7fe64047d640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6380cb9f0 con 0x7fe6380cf870 2023-05-22T15:42:09.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:09.524+0000 7fe64047d640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe63806f100 con 0x7fe63806e740 2023-05-22T15:42:09.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:09.524+0000 7fe63da18640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe6380cf870 0x7fe63806db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:09.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:09.524+0000 7fe63da18640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe6380cf870 0x7fe63806db00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41236/0 (socket says 172.21.15.16:41236) 2023-05-22T15:42:09.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:09.524+0000 7fe63da18640 1 -- 172.21.15.16:0/2557511371 learned_addr learned my addr 172.21.15.16:0/2557511371 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:42:09.530 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:09.524+0000 7fe63ea1a640 1 --2- 172.21.15.16:0/2557511371 >> v2:172.21.15.62:3300/0 conn(0x7fe63806e740 0x7fe63806ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:09.530 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:09.524+0000 7fe63da18640 1 -- 172.21.15.16:0/2557511371 >> v2:172.21.15.62:3300/0 conn(0x7fe63806e740 msgr2=0x7fe63806ebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:42:09.530 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:09.524+0000 7fe63da18640 1 --2- 172.21.15.16:0/2557511371 >> v2:172.21.15.62:3300/0 conn(0x7fe63806e740 0x7fe63806ebc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:09.530 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:09.524+0000 7fe63da18640 1 -- 172.21.15.16:0/2557511371 >> v1:172.21.15.16:6789/0 conn(0x7fe63815cfa0 legacy=0x7fe6380b1fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:09.530 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:09.524+0000 7fe63da18640 1 -- 172.21.15.16:0/2557511371 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe63806e490 con 0x7fe6380cf870 2023-05-22T15:42:09.530 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:09.524+0000 7fe63d217640 1 -- 172.21.15.16:0/2557511371 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4218502274 0 0) 0x7fe6300014a0 con 0x7fe63815cfa0 2023-05-22T15:42:09.530 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:09.524+0000 7fe63ea1a640 1 --2- 172.21.15.16:0/2557511371 >> v2:172.21.15.62:3300/0 conn(0x7fe63806e740 0x7fe63806ebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:42:09.531 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:09.528+0000 7fe63da18640 1 --2- 172.21.15.16:0/2557511371 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe6380cf870 0x7fe63806db00 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7fe6380c9760 tx=0x7fe62c01c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f27a6e083db5f2d1 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:09.531 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:09.528+0000 7fe63d217640 1 -- 172.21.15.16:0/2557511371 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe62c0aa430 con 0x7fe6380cf870 2023-05-22T15:42:09.532 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:09.528+0000 7fe63d217640 1 -- 172.21.15.16:0/2557511371 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe62c0aa590 con 0x7fe6380cf870 2023-05-22T15:42:09.532 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:09.528+0000 7fe63d217640 1 -- 172.21.15.16:0/2557511371 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe62c0aa920 con 0x7fe6380cf870 2023-05-22T15:42:09.532 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:09.528+0000 7fe64047d640 1 -- 172.21.15.16:0/2557511371 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe6380cf870 msgr2=0x7fe63806db00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:09.532 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:09.528+0000 7fe64047d640 1 --2- 172.21.15.16:0/2557511371 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe6380cf870 0x7fe63806db00 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7fe6380c9760 tx=0x7fe62c01c4d0 comp rx=0 tx=0).stop 2023-05-22T15:42:09.532 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:09.528+0000 7fe64047d640 1 -- 172.21.15.16:0/2557511371 shutdown_connections 2023-05-22T15:42:09.532 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:09.528+0000 7fe64047d640 1 --2- 172.21.15.16:0/2557511371 >> v2:172.21.15.62:3300/0 conn(0x7fe63806e740 0x7fe63806ebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:09.532 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:09.528+0000 7fe64047d640 1 --2- 172.21.15.16:0/2557511371 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe6380cf870 0x7fe63806db00 unknown :-1 s=CLOSED pgs=1033 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:09.532 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:09.528+0000 7fe64047d640 1 -- 172.21.15.16:0/2557511371 >> 172.21.15.16:0/2557511371 conn(0x7fe6380cc870 msgr2=0x7fe6380cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:09.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:09.528+0000 7fe64047d640 1 -- 172.21.15.16:0/2557511371 shutdown_connections 2023-05-22T15:42:09.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:09.528+0000 7fe64047d640 1 -- 172.21.15.16:0/2557511371 wait complete. 2023-05-22T15:42:09.534 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:09.528+0000 7fe64047d640 1 Processor -- start 2023-05-22T15:42:09.534 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:09.528+0000 7fe64047d640 1 -- start start 2023-05-22T15:42:09.535 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:09.528+0000 7fe64047d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe63806e740 0x7fe6380c7280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:09.535 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:09.528+0000 7fe64047d640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe6380cf870 0x7fe63814dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:09.535 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:09.528+0000 7fe64047d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe63815cfa0 0x7fe63815be40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:09.535 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:09.528+0000 7fe64047d640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6380c7bf0 con 0x7fe63806e740 2023-05-22T15:42:09.535 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:09.528+0000 7fe64047d640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6380c7ed0 con 0x7fe63815cfa0 2023-05-22T15:42:09.535 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:09.528+0000 7fe64047d640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6380c81b0 con 0x7fe6380cf870 2023-05-22T15:42:09.535 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:09.528+0000 7fe63e219640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe63806e740 0x7fe6380c7280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:09.536 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:09.528+0000 7fe63e219640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe63806e740 0x7fe6380c7280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41252/0 (socket says 172.21.15.16:41252) 2023-05-22T15:42:09.536 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:09.528+0000 7fe63e219640 1 -- 172.21.15.16:0/2556058772 learned_addr learned my addr 172.21.15.16:0/2556058772 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:42:09.536 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:09.528+0000 7fe63e219640 1 -- 172.21.15.16:0/2556058772 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe6380cf870 msgr2=0x7fe63814dd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:09.536 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:09.528+0000 7fe63e219640 1 --2- 172.21.15.16:0/2556058772 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe6380cf870 0x7fe63814dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:09.536 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:09.528+0000 7fe63e219640 1 -- 172.21.15.16:0/2556058772 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe63815cfa0 msgr2=0x7fe63815be40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:09.536 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:09.528+0000 7fe63ea1a640 1 --2- 172.21.15.16:0/2556058772 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe63815cfa0 0x7fe63815be40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:09.536 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:09.528+0000 7fe63e219640 1 --2- 172.21.15.16:0/2556058772 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe63815cfa0 0x7fe63815be40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:09.537 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:09.528+0000 7fe63e219640 1 -- 172.21.15.16:0/2556058772 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe62c0094d0 con 0x7fe63806e740 2023-05-22T15:42:09.537 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:09.532+0000 7fe63e219640 1 --2- 172.21.15.16:0/2556058772 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe63806e740 0x7fe6380c7280 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7fe63000c7a0 tx=0x7fe63000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:09.537 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:09.532+0000 7fe636ffd640 1 -- 172.21.15.16:0/2556058772 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe630003db0 con 0x7fe63806e740 2023-05-22T15:42:09.537 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:09.532+0000 7fe636ffd640 1 -- 172.21.15.16:0/2556058772 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe630005070 con 0x7fe63806e740 2023-05-22T15:42:09.537 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:09.532+0000 7fe64047d640 1 -- 172.21.15.16:0/2556058772 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6380cb610 con 0x7fe63806e740 2023-05-22T15:42:09.537 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:09.532+0000 7fe636ffd640 1 -- 172.21.15.16:0/2556058772 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe630005420 con 0x7fe63806e740 2023-05-22T15:42:09.537 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:09.532+0000 7fe64047d640 1 -- 172.21.15.16:0/2556058772 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6380cb330 con 0x7fe63806e740 2023-05-22T15:42:09.538 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:09.532+0000 7fe64047d640 1 -- 172.21.15.16:0/2556058772 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe63815d370 con 0x7fe63806e740 2023-05-22T15:42:09.538 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:09.532+0000 7fe636ffd640 1 -- 172.21.15.16:0/2556058772 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 25) v1 ==== 84068+0+0 (secure 0 0 0) 0x7fe630005580 con 0x7fe63806e740 2023-05-22T15:42:09.538 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:09.532+0000 7fe636ffd640 1 --2- 172.21.15.16:0/2556058772 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fe6240651d0 0x7fe624067680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:09.539 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:09.532+0000 7fe636ffd640 1 -- 172.21.15.16:0/2556058772 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7012+0+0 (secure 0 0 0) 0x7fe630081e20 con 0x7fe63806e740 2023-05-22T15:42:09.539 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:09.532+0000 7fe63da18640 1 --2- 172.21.15.16:0/2556058772 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fe6240651d0 0x7fe624067680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:09.539 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:09.532+0000 7fe63da18640 1 --2- 172.21.15.16:0/2556058772 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fe6240651d0 0x7fe624067680 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fe62c0ae000 tx=0x7fe62c008040 comp rx=0 tx=0).ready entity=mgr.6765 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:09.540 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:09.536+0000 7fe636ffd640 1 -- 172.21.15.16:0/2556058772 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fe63004a620 con 0x7fe63806e740 2023-05-22T15:42:09.720 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:09.716+0000 7fe64047d640 1 -- 172.21.15.16:0/2556058772 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fe63806e040 con 0x7fe63806e740 2023-05-22T15:42:10.062 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:10.060+0000 7fe636ffd640 1 -- 172.21.15.16:0/2556058772 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mgrmap(e 26) v1 ==== 84332+0+0 (secure 0 0 0) 0x7fe63004fea0 con 0x7fe63806e740 2023-05-22T15:42:10.475 INFO:tasks.ceph.mon.a.smithi016.stderr:2023-05-22T15:42:10.472+0000 7f95dd48a640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-05-22T15:42:10.485 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:10.480+0000 7fe636ffd640 1 -- 172.21.15.16:0/2556058772 <== mon.0 v2:172.21.15.16:3300/0 8 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v23) v1 ==== 161+0+0 (secure 0 0 0) 0x7fe6300530f0 con 0x7fe63806e740 2023-05-22T15:42:10.485 INFO:teuthology.orchestra.run.smithi016.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-05-22T15:42:10.488 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:10.484+0000 7fe64047d640 1 -- 172.21.15.16:0/2556058772 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fe6240651d0 msgr2=0x7fe624067680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:10.488 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:10.484+0000 7fe64047d640 1 --2- 172.21.15.16:0/2556058772 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fe6240651d0 0x7fe624067680 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fe62c0ae000 tx=0x7fe62c008040 comp rx=0 tx=0).stop 2023-05-22T15:42:10.488 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:10.484+0000 7fe64047d640 1 -- 172.21.15.16:0/2556058772 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe63806e740 msgr2=0x7fe6380c7280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:10.488 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:10.484+0000 7fe64047d640 1 --2- 172.21.15.16:0/2556058772 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe63806e740 0x7fe6380c7280 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7fe63000c7a0 tx=0x7fe63000cc60 comp rx=0 tx=0).stop 2023-05-22T15:42:10.489 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:10.484+0000 7fe64047d640 1 -- 172.21.15.16:0/2556058772 shutdown_connections 2023-05-22T15:42:10.489 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:10.484+0000 7fe64047d640 1 --2- 172.21.15.16:0/2556058772 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe63815cfa0 0x7fe63815be40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:10.489 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:10.484+0000 7fe64047d640 1 --2- 172.21.15.16:0/2556058772 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fe6240651d0 0x7fe624067680 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:10.489 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:10.484+0000 7fe64047d640 1 --2- 172.21.15.16:0/2556058772 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe6380cf870 0x7fe63814dd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:10.489 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:10.484+0000 7fe64047d640 1 --2- 172.21.15.16:0/2556058772 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe63806e740 0x7fe6380c7280 unknown :-1 s=CLOSED pgs=1034 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:10.489 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:10.484+0000 7fe64047d640 1 -- 172.21.15.16:0/2556058772 >> 172.21.15.16:0/2556058772 conn(0x7fe6380cc870 msgr2=0x7fe6380aeb80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:10.489 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:10.484+0000 7fe64047d640 1 -- 172.21.15.16:0/2556058772 shutdown_connections 2023-05-22T15:42:10.489 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:10.484+0000 7fe64047d640 1 -- 172.21.15.16:0/2556058772 wait complete. 2023-05-22T15:42:10.509 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-05-22T15:42:10.541 INFO:tasks.ceph.mds.a.smithi016.stdout:starting mds.a at 2023-05-22T15:42:10.644 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:10.640+0000 7f4941089640 1 Processor -- start 2023-05-22T15:42:10.645 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:10.640+0000 7f4941089640 1 -- start start 2023-05-22T15:42:10.646 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:10.640+0000 7f4941089640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f493c15bca0 0x7f493c0cda80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:10.646 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:10.640+0000 7f4941089640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f493c0d2480 0x7f493c0d2900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:10.646 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:10.640+0000 7f4941089640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f493c0cccc0 con 0x7f493c0b5340 2023-05-22T15:42:10.646 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:10.640+0000 7f4941089640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f493c0cd660 con 0x7f493c15bca0 2023-05-22T15:42:10.646 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:10.640+0000 7f4941089640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f493c0d2e40 con 0x7f493c0d2480 2023-05-22T15:42:10.647 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:10.640+0000 7f493ad9d640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f493c15bca0 0x7f493c0cda80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:10.647 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:10.640+0000 7f493ad9d640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f493c15bca0 0x7f493c0cda80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41278/0 (socket says 172.21.15.16:41278) 2023-05-22T15:42:10.647 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:10.640+0000 7f493ad9d640 1 -- 172.21.15.16:0/128518361 learned_addr learned my addr 172.21.15.16:0/128518361 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:42:10.647 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:10.644+0000 7f493a59c640 1 --2- 172.21.15.16:0/128518361 >> v2:172.21.15.62:3300/0 conn(0x7f493c0d2480 0x7f493c0d2900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:10.648 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:10.644+0000 7f493ad9d640 1 -- 172.21.15.16:0/128518361 >> v2:172.21.15.62:3300/0 conn(0x7f493c0d2480 msgr2=0x7f493c0d2900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:42:10.649 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:10.644+0000 7f493ad9d640 1 --2- 172.21.15.16:0/128518361 >> v2:172.21.15.62:3300/0 conn(0x7f493c0d2480 0x7f493c0d2900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:10.649 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:10.644+0000 7f493ad9d640 1 -- 172.21.15.16:0/128518361 >> v1:172.21.15.16:6789/0 conn(0x7f493c0b5340 legacy=0x7f493c0b5790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:10.649 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:10.644+0000 7f493ad9d640 1 -- 172.21.15.16:0/128518361 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f493c0ccfa0 con 0x7f493c15bca0 2023-05-22T15:42:10.649 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:10.644+0000 7f493ad9d640 1 --2- 172.21.15.16:0/128518361 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f493c15bca0 0x7f493c0cda80 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7f493c065e80 tx=0x7f492c01c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8909d075a284ba93 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:10.649 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:10.644+0000 7f4939d9b640 1 -- 172.21.15.16:0/128518361 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f492c0aa680 con 0x7f493c15bca0 2023-05-22T15:42:10.650 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:10.644+0000 7f4939d9b640 1 -- 172.21.15.16:0/128518361 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f492c0aa7e0 con 0x7f493c15bca0 2023-05-22T15:42:10.650 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:10.644+0000 7f4939d9b640 1 -- 172.21.15.16:0/128518361 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f492c0aab70 con 0x7f493c15bca0 2023-05-22T15:42:10.652 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:10.644+0000 7f4941089640 1 -- 172.21.15.16:0/128518361 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f493c15bca0 msgr2=0x7f493c0cda80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:10.652 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:10.644+0000 7f4941089640 1 --2- 172.21.15.16:0/128518361 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f493c15bca0 0x7f493c0cda80 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7f493c065e80 tx=0x7f492c01c500 comp rx=0 tx=0).stop 2023-05-22T15:42:10.652 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:10.644+0000 7f4941089640 1 -- 172.21.15.16:0/128518361 shutdown_connections 2023-05-22T15:42:10.652 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:10.644+0000 7f4941089640 1 --2- 172.21.15.16:0/128518361 >> v2:172.21.15.62:3300/0 conn(0x7f493c0d2480 0x7f493c0d2900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:10.652 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:10.644+0000 7f4941089640 1 --2- 172.21.15.16:0/128518361 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f493c15bca0 0x7f493c0cda80 unknown :-1 s=CLOSED pgs=1039 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:10.652 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:10.644+0000 7f4941089640 1 -- 172.21.15.16:0/128518361 >> 172.21.15.16:0/128518361 conn(0x7f493c0655c0 msgr2=0x7f493c070e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:10.652 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:10.644+0000 7f4941089640 1 -- 172.21.15.16:0/128518361 shutdown_connections 2023-05-22T15:42:10.653 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:10.644+0000 7f4941089640 1 -- 172.21.15.16:0/128518361 wait complete. 2023-05-22T15:42:10.653 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:10.648+0000 7f4941089640 1 Processor -- start 2023-05-22T15:42:10.653 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:10.648+0000 7f4941089640 1 -- start start 2023-05-22T15:42:10.653 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:10.648+0000 7f4941089640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f493c0b5340 0x7f493c15c330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:10.653 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:10.648+0000 7f4941089640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f493c0d2480 0x7f493c0b7830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:10.653 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:10.648+0000 7f4941089640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f493c15bca0 0x7f493c0c66b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:10.653 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:10.648+0000 7f4941089640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f493c15a7d0 con 0x7f493c15bca0 2023-05-22T15:42:10.654 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:10.648+0000 7f4941089640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f493c152ef0 con 0x7f493c0b5340 2023-05-22T15:42:10.654 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:10.648+0000 7f4941089640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f493c1531d0 con 0x7f493c0d2480 2023-05-22T15:42:10.654 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:10.648+0000 7f493b59e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f493c15bca0 0x7f493c0c66b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:10.654 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:10.648+0000 7f493b59e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f493c15bca0 0x7f493c0c66b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41284/0 (socket says 172.21.15.16:41284) 2023-05-22T15:42:10.654 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:10.648+0000 7f493b59e640 1 -- 172.21.15.16:0/1751868359 learned_addr learned my addr 172.21.15.16:0/1751868359 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:42:10.654 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:10.648+0000 7f493a59c640 1 --2- 172.21.15.16:0/1751868359 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f493c0d2480 0x7f493c0b7830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:10.654 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:10.648+0000 7f493b59e640 1 -- 172.21.15.16:0/1751868359 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f493c0d2480 msgr2=0x7f493c0b7830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:10.654 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:10.648+0000 7f493b59e640 1 --2- 172.21.15.16:0/1751868359 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f493c0d2480 0x7f493c0b7830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:10.655 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:10.648+0000 7f493b59e640 1 -- 172.21.15.16:0/1751868359 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f493c0b5340 msgr2=0x7f493c15c330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:10.655 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:10.648+0000 7f493b59e640 1 --2- 172.21.15.16:0/1751868359 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f493c0b5340 0x7f493c15c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:10.655 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:10.648+0000 7f493b59e640 1 -- 172.21.15.16:0/1751868359 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f492c0094d0 con 0x7f493c15bca0 2023-05-22T15:42:10.655 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:10.648+0000 7f493b59e640 1 --2- 172.21.15.16:0/1751868359 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f493c15bca0 0x7f493c0c66b0 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7f493400c730 tx=0x7f493400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:10.655 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:10.648+0000 7f49237fe640 1 -- 172.21.15.16:0/1751868359 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4934003b80 con 0x7f493c15bca0 2023-05-22T15:42:10.655 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:10.648+0000 7f49237fe640 1 -- 172.21.15.16:0/1751868359 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4934003ce0 con 0x7f493c15bca0 2023-05-22T15:42:10.655 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:10.648+0000 7f4941089640 1 -- 172.21.15.16:0/1751868359 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f493c160360 con 0x7f493c15bca0 2023-05-22T15:42:10.656 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:10.648+0000 7f49237fe640 1 -- 172.21.15.16:0/1751868359 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f493400aa30 con 0x7f493c15bca0 2023-05-22T15:42:10.656 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:10.648+0000 7f4941089640 1 -- 172.21.15.16:0/1751868359 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f493c0cdfc0 con 0x7f493c15bca0 2023-05-22T15:42:10.656 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:10.648+0000 7f4941089640 1 -- 172.21.15.16:0/1751868359 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f493c1604f0 con 0x7f493c15bca0 2023-05-22T15:42:10.657 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:10.648+0000 7f49237fe640 1 -- 172.21.15.16:0/1751868359 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f493400ab90 con 0x7f493c15bca0 2023-05-22T15:42:10.657 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:10.648+0000 7f49237fe640 1 --2- 172.21.15.16:0/1751868359 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f49280654a0 0x7f4928067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:10.657 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:10.648+0000 7f493ad9d640 1 --2- 172.21.15.16:0/1751868359 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f49280654a0 0x7f4928067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:10.657 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:10.648+0000 7f49237fe640 1 -- 172.21.15.16:0/1751868359 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7098+0+0 (secure 0 0 0) 0x7f4934082c20 con 0x7f493c15bca0 2023-05-22T15:42:10.657 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:10.652+0000 7f493ad9d640 1 --2- 172.21.15.16:0/1751868359 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f49280654a0 0x7f4928067950 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f492c0ae000 tx=0x7f492c008040 comp rx=0 tx=0).ready entity=mgr.6765 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:10.658 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:10.656+0000 7f49237fe640 1 -- 172.21.15.16:0/1751868359 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f49340502f0 con 0x7f493c15bca0 2023-05-22T15:42:10.848 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:10.844+0000 7f4941089640 1 -- 172.21.15.16:0/1751868359 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f493c04dbe0 con 0x7f493c15bca0 2023-05-22T15:42:11.498 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:11.492+0000 7f49237fe640 1 -- 172.21.15.16:0/1751868359 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v24) v1 ==== 104+0+0 (secure 0 0 0) 0x7f4934053910 con 0x7f493c15bca0 2023-05-22T15:42:11.501 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:11.496+0000 7f4941089640 1 -- 172.21.15.16:0/1751868359 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f49280654a0 msgr2=0x7f4928067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:11.501 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:11.496+0000 7f4941089640 1 --2- 172.21.15.16:0/1751868359 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f49280654a0 0x7f4928067950 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f492c0ae000 tx=0x7f492c008040 comp rx=0 tx=0).stop 2023-05-22T15:42:11.501 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:11.496+0000 7f4941089640 1 -- 172.21.15.16:0/1751868359 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f493c15bca0 msgr2=0x7f493c0c66b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:11.501 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:11.496+0000 7f4941089640 1 --2- 172.21.15.16:0/1751868359 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f493c15bca0 0x7f493c0c66b0 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7f493400c730 tx=0x7f493400cbf0 comp rx=0 tx=0).stop 2023-05-22T15:42:11.502 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:11.496+0000 7f4941089640 1 -- 172.21.15.16:0/1751868359 shutdown_connections 2023-05-22T15:42:11.502 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:11.496+0000 7f4941089640 1 --2- 172.21.15.16:0/1751868359 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f493c15bca0 0x7f493c0c66b0 unknown :-1 s=CLOSED pgs=1040 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:11.502 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:11.496+0000 7f4941089640 1 --2- 172.21.15.16:0/1751868359 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f49280654a0 0x7f4928067950 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:11.502 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:11.496+0000 7f4941089640 1 --2- 172.21.15.16:0/1751868359 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f493c0d2480 0x7f493c0b7830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:11.502 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:11.496+0000 7f4941089640 1 --2- 172.21.15.16:0/1751868359 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f493c0b5340 0x7f493c15c330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:11.502 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:11.496+0000 7f4941089640 1 -- 172.21.15.16:0/1751868359 >> 172.21.15.16:0/1751868359 conn(0x7f493c0655c0 msgr2=0x7f493c1541c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:11.502 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:11.496+0000 7f4941089640 1 -- 172.21.15.16:0/1751868359 shutdown_connections 2023-05-22T15:42:11.502 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:11.496+0000 7f4941089640 1 -- 172.21.15.16:0/1751868359 wait complete. 2023-05-22T15:42:11.522 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-05-22T15:42:11.701 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:11.696+0000 7f1c9ced7640 1 Processor -- start 2023-05-22T15:42:11.701 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:11.696+0000 7f1c9ced7640 1 -- start start 2023-05-22T15:42:11.702 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:11.696+0000 7f1c9ced7640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f1c980cef10 0x7f1c980cf310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:11.702 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:11.696+0000 7f1c9ced7640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c980d2a20 con 0x7f1c980ccc50 2023-05-22T15:42:11.702 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:11.696+0000 7f1c9ced7640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c980cd230 con 0x7f1c9815c8e0 2023-05-22T15:42:11.702 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:11.696+0000 7f1c9ced7640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c980d23d0 con 0x7f1c980cef10 2023-05-22T15:42:11.703 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:11.700+0000 7f1c8ffff640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f1c980ccc50 0x7f1c9815ccb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:41684/0 (socket says 172.21.15.16:41684) 2023-05-22T15:42:11.703 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:11.700+0000 7f1c8ffff640 1 -- 172.21.15.16:0/2649970631 learned_addr learned my addr 172.21.15.16:0/2649970631 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:42:11.704 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:11.700+0000 7f1c96d9d640 1 --2- 172.21.15.16:0/2649970631 >> v2:172.21.15.62:3300/0 conn(0x7f1c980cef10 0x7f1c980cf310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:11.704 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:11.700+0000 7f1c95d9b640 1 -- 172.21.15.16:0/2649970631 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2663858905 0 0) 0x7f1c980d2a20 con 0x7f1c980ccc50 2023-05-22T15:42:11.704 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:11.700+0000 7f1c95d9b640 1 -- 172.21.15.16:0/2649970631 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c84003530 con 0x7f1c980ccc50 2023-05-22T15:42:11.704 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:11.700+0000 7f1c95d9b640 1 -- 172.21.15.16:0/2649970631 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 274129592 0 0) 0x7f1c84003530 con 0x7f1c980ccc50 2023-05-22T15:42:11.704 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:11.700+0000 7f1c95d9b640 1 -- 172.21.15.16:0/2649970631 >> v2:172.21.15.62:3300/0 conn(0x7f1c980cef10 msgr2=0x7f1c980cf310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:42:11.704 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:11.700+0000 7f1c95d9b640 1 --2- 172.21.15.16:0/2649970631 >> v2:172.21.15.62:3300/0 conn(0x7f1c980cef10 0x7f1c980cf310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:11.704 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:11.700+0000 7f1c95d9b640 1 -- 172.21.15.16:0/2649970631 >> v1:172.21.15.62:6789/0 conn(0x7f1c9815c8e0 legacy=0x7f1c980d2eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:11.705 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:11.700+0000 7f1c95d9b640 1 -- 172.21.15.16:0/2649970631 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c980d26b0 con 0x7f1c980ccc50 2023-05-22T15:42:11.705 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:11.700+0000 7f1c95d9b640 1 -- 172.21.15.16:0/2649970631 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7f1c80001680 con 0x7f1c980ccc50 2023-05-22T15:42:11.705 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:11.700+0000 7f1c95d9b640 1 -- 172.21.15.16:0/2649970631 >> v1:172.21.15.16:6789/0 conn(0x7f1c980ccc50 legacy=0x7f1c9815ccb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:11.705 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:11.700+0000 7f1c95d9b640 1 --2- 172.21.15.16:0/2649970631 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1c84003980 0x7f1c84003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:11.705 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:11.700+0000 7f1c95d9b640 1 --2- 172.21.15.16:0/2649970631 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1c840047d0 0x7f1c84004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:11.705 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:11.700+0000 7f1c95d9b640 1 --2- 172.21.15.16:0/2649970631 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c84005180 0x7f1c8400b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:11.705 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:11.700+0000 7f1c95d9b640 1 -- 172.21.15.16:0/2649970631 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c980d23d0 con 0x7f1c84005180 2023-05-22T15:42:11.705 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:11.700+0000 7f1c95d9b640 1 -- 172.21.15.16:0/2649970631 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c8400c090 con 0x7f1c840047d0 2023-05-22T15:42:11.706 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:11.700+0000 7f1c95d9b640 1 -- 172.21.15.16:0/2649970631 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c8400c250 con 0x7f1c84003980 2023-05-22T15:42:11.706 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:11.700+0000 7f1c96d9d640 1 --2- 172.21.15.16:0/2649970631 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c84005180 0x7f1c8400b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:11.706 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:11.700+0000 7f1c8ffff640 1 --2- 172.21.15.16:0/2649970631 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1c840047d0 0x7f1c84004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:11.706 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:11.700+0000 7f1c9659c640 1 --2- 172.21.15.16:0/2649970631 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1c84003980 0x7f1c84003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:11.707 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:11.700+0000 7f1c96d9d640 1 -- 172.21.15.16:0/2649970631 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1c84003980 msgr2=0x7f1c84003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:42:11.707 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:11.700+0000 7f1c96d9d640 1 --2- 172.21.15.16:0/2649970631 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1c84003980 0x7f1c84003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:11.707 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:11.700+0000 7f1c96d9d640 1 -- 172.21.15.16:0/2649970631 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1c840047d0 msgr2=0x7f1c84004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:42:11.707 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:11.700+0000 7f1c96d9d640 1 --2- 172.21.15.16:0/2649970631 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1c840047d0 0x7f1c84004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:11.708 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:11.700+0000 7f1c96d9d640 1 -- 172.21.15.16:0/2649970631 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c84003530 con 0x7f1c84005180 2023-05-22T15:42:11.708 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:11.700+0000 7f1c96d9d640 1 -- 172.21.15.16:0/2649970631 reap_dead start 2023-05-22T15:42:11.708 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:11.700+0000 7f1c96d9d640 1 --2- 172.21.15.16:0/2649970631 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c84005180 0x7f1c8400b990 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7f1c90009fc0 tx=0x7f1c9001cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a353fbeb303f9b server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:11.708 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:11.704+0000 7f1c95d9b640 1 -- 172.21.15.16:0/2649970631 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c900a5bb0 con 0x7f1c84005180 2023-05-22T15:42:11.708 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:11.704+0000 7f1c95d9b640 1 -- 172.21.15.16:0/2649970631 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c9000f040 con 0x7f1c84005180 2023-05-22T15:42:11.708 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:11.704+0000 7f1c95d9b640 1 -- 172.21.15.16:0/2649970631 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c900039a0 con 0x7f1c84005180 2023-05-22T15:42:11.708 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:11.704+0000 7f1c9ced7640 1 -- 172.21.15.16:0/2649970631 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c84005180 msgr2=0x7f1c8400b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:11.709 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:11.704+0000 7f1c9ced7640 1 --2- 172.21.15.16:0/2649970631 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c84005180 0x7f1c8400b990 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7f1c90009fc0 tx=0x7f1c9001cca0 comp rx=0 tx=0).stop 2023-05-22T15:42:11.709 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:11.704+0000 7f1c9ced7640 1 -- 172.21.15.16:0/2649970631 shutdown_connections 2023-05-22T15:42:11.709 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:11.704+0000 7f1c9ced7640 1 --2- 172.21.15.16:0/2649970631 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c84005180 0x7f1c8400b990 unknown :-1 s=CLOSED pgs=1042 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:11.709 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:11.704+0000 7f1c9ced7640 1 -- 172.21.15.16:0/2649970631 >> 172.21.15.16:0/2649970631 conn(0x7f1c98070e50 msgr2=0x7f1c98071250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:11.709 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:11.704+0000 7f1c9ced7640 1 -- 172.21.15.16:0/2649970631 shutdown_connections 2023-05-22T15:42:11.709 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:11.704+0000 7f1c9ced7640 1 -- 172.21.15.16:0/2649970631 wait complete. 2023-05-22T15:42:11.711 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:11.704+0000 7f1c9ced7640 1 Processor -- start 2023-05-22T15:42:11.711 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:11.704+0000 7f1c9ced7640 1 -- start start 2023-05-22T15:42:11.711 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:11.704+0000 7f1c9ced7640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1c84005180 0x7f1c981509e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:11.711 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:11.704+0000 7f1c9ced7640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c980b7950 0x7f1c980b7dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:11.712 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:11.704+0000 7f1c9ced7640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1c980c9900 0x7f1c980c9d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:11.712 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:11.704+0000 7f1c9ced7640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c980b8300 con 0x7f1c980b7950 2023-05-22T15:42:11.712 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:11.704+0000 7f1c9ced7640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c980cc6c0 con 0x7f1c84005180 2023-05-22T15:42:11.712 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:11.704+0000 7f1c9ced7640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c98150f20 con 0x7f1c980c9900 2023-05-22T15:42:11.712 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:11.704+0000 7f1c8ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c980b7950 0x7f1c980b7dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:11.712 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:11.704+0000 7f1c8ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c980b7950 0x7f1c980b7dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41310/0 (socket says 172.21.15.16:41310) 2023-05-22T15:42:11.712 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:11.704+0000 7f1c8ffff640 1 -- 172.21.15.16:0/1907131452 learned_addr learned my addr 172.21.15.16:0/1907131452 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:42:11.713 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:11.704+0000 7f1c9659c640 1 --2- 172.21.15.16:0/1907131452 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1c84005180 0x7f1c981509e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:11.713 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:11.704+0000 7f1c96d9d640 1 --2- 172.21.15.16:0/1907131452 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1c980c9900 0x7f1c980c9d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:11.713 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:11.704+0000 7f1c8ffff640 1 -- 172.21.15.16:0/1907131452 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1c980c9900 msgr2=0x7f1c980c9d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:11.713 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:11.704+0000 7f1c8ffff640 1 --2- 172.21.15.16:0/1907131452 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1c980c9900 0x7f1c980c9d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:11.713 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:11.704+0000 7f1c8ffff640 1 -- 172.21.15.16:0/1907131452 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1c84005180 msgr2=0x7f1c981509e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:11.713 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:11.704+0000 7f1c8ffff640 1 --2- 172.21.15.16:0/1907131452 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1c84005180 0x7f1c981509e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:11.713 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:11.704+0000 7f1c8ffff640 1 -- 172.21.15.16:0/1907131452 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c980cd510 con 0x7f1c980b7950 2023-05-22T15:42:11.714 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:11.704+0000 7f1c9659c640 1 --2- 172.21.15.16:0/1907131452 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1c84005180 0x7f1c981509e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T15:42:11.714 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:11.704+0000 7f1c8ffff640 1 --2- 172.21.15.16:0/1907131452 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c980b7950 0x7f1c980b7dc0 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7f1c800124f0 tx=0x7f1c800129b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:11.714 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:11.704+0000 7f1c8effd640 1 -- 172.21.15.16:0/1907131452 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c80014810 con 0x7f1c980b7950 2023-05-22T15:42:11.714 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:11.704+0000 7f1c8effd640 1 -- 172.21.15.16:0/1907131452 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c80014970 con 0x7f1c980b7950 2023-05-22T15:42:11.714 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:11.704+0000 7f1c9ced7640 1 -- 172.21.15.16:0/1907131452 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c980cd080 con 0x7f1c980b7950 2023-05-22T15:42:11.714 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:11.704+0000 7f1c8effd640 1 -- 172.21.15.16:0/1907131452 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c80014d00 con 0x7f1c980b7950 2023-05-22T15:42:11.714 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:11.704+0000 7f1c9ced7640 1 -- 172.21.15.16:0/1907131452 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c980d2eb0 con 0x7f1c980b7950 2023-05-22T15:42:11.715 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:11.708+0000 7f1c8effd640 1 -- 172.21.15.16:0/1907131452 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f1c800153e0 con 0x7f1c980b7950 2023-05-22T15:42:11.715 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:11.708+0000 7f1c9ced7640 1 -- 172.21.15.16:0/1907131452 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c980cbb10 con 0x7f1c980b7950 2023-05-22T15:42:11.715 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:11.708+0000 7f1c8effd640 1 --2- 172.21.15.16:0/1907131452 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f1c780654d0 0x7f1c78067980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:11.716 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:11.708+0000 7f1c8effd640 1 -- 172.21.15.16:0/1907131452 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7098+0+0 (secure 0 0 0) 0x7f1c80085990 con 0x7f1c980b7950 2023-05-22T15:42:11.716 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:11.708+0000 7f1c9659c640 1 --2- 172.21.15.16:0/1907131452 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f1c780654d0 0x7f1c78067980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:11.716 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:11.708+0000 7f1c9659c640 1 --2- 172.21.15.16:0/1907131452 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f1c780654d0 0x7f1c78067980 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f1c98156020 tx=0x7f1c88009040 comp rx=0 tx=0).ready entity=mgr.6765 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:11.716 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:11.712+0000 7f1c8effd640 1 -- 172.21.15.16:0/1907131452 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f1c80053070 con 0x7f1c980b7950 2023-05-22T15:42:11.872 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:11.868+0000 7f1c9ced7640 1 -- 172.21.15.16:0/1907131452 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f1c9804dbe0 con 0x7f1c980b7950 2023-05-22T15:42:12.580 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:12.576+0000 7f1c8effd640 1 -- 172.21.15.16:0/1907131452 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v80) v1 ==== 181+0+0 (secure 0 0 0) 0x7f1c80056690 con 0x7f1c980b7950 2023-05-22T15:42:12.580 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'cephfs_metadata' removed 2023-05-22T15:42:12.584 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:12.580+0000 7f1c9ced7640 1 -- 172.21.15.16:0/1907131452 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f1c780654d0 msgr2=0x7f1c78067980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:12.584 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:12.580+0000 7f1c9ced7640 1 --2- 172.21.15.16:0/1907131452 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f1c780654d0 0x7f1c78067980 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f1c98156020 tx=0x7f1c88009040 comp rx=0 tx=0).stop 2023-05-22T15:42:12.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:12.580+0000 7f1c9ced7640 1 -- 172.21.15.16:0/1907131452 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c980b7950 msgr2=0x7f1c980b7dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:12.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:12.580+0000 7f1c9ced7640 1 --2- 172.21.15.16:0/1907131452 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c980b7950 0x7f1c980b7dc0 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7f1c800124f0 tx=0x7f1c800129b0 comp rx=0 tx=0).stop 2023-05-22T15:42:12.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:12.580+0000 7f1c9ced7640 1 -- 172.21.15.16:0/1907131452 shutdown_connections 2023-05-22T15:42:12.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:12.580+0000 7f1c9ced7640 1 --2- 172.21.15.16:0/1907131452 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f1c780654d0 0x7f1c78067980 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:12.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:12.580+0000 7f1c9ced7640 1 --2- 172.21.15.16:0/1907131452 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1c980c9900 0x7f1c980c9d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:12.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:12.580+0000 7f1c9ced7640 1 --2- 172.21.15.16:0/1907131452 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c980b7950 0x7f1c980b7dc0 unknown :-1 s=CLOSED pgs=1043 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:12.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:12.580+0000 7f1c9ced7640 1 --2- 172.21.15.16:0/1907131452 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1c84005180 0x7f1c981509e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:12.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:12.580+0000 7f1c9ced7640 1 -- 172.21.15.16:0/1907131452 >> 172.21.15.16:0/1907131452 conn(0x7f1c98070e50 msgr2=0x7f1c98151ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:12.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:12.580+0000 7f1c9ced7640 1 -- 172.21.15.16:0/1907131452 shutdown_connections 2023-05-22T15:42:12.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:12.580+0000 7f1c9ced7640 1 -- 172.21.15.16:0/1907131452 wait complete. 2023-05-22T15:42:12.609 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-05-22T15:42:12.740 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:12.736+0000 7fc5d84b0640 1 Processor -- start 2023-05-22T15:42:12.741 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:12.736+0000 7fc5d84b0640 1 -- start start 2023-05-22T15:42:12.742 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:12.736+0000 7fc5d84b0640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fc5d00cc4e0 0x7fc5d006bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:12.742 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:12.736+0000 7fc5d84b0640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc5d015d650 0x7fc5d015dad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:12.742 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:12.736+0000 7fc5d84b0640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5d00cc960 con 0x7fc5d0070c10 2023-05-22T15:42:12.742 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:12.736+0000 7fc5d84b0640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5d015e010 con 0x7fc5d00cc4e0 2023-05-22T15:42:12.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:12.736+0000 7fc5d84b0640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5d00ccbb0 con 0x7fc5d015d650 2023-05-22T15:42:12.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:12.740+0000 7fc5d624c640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fc5d0070c10 0x7fc5d00c9e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:41698/0 (socket says 172.21.15.16:41698) 2023-05-22T15:42:12.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:12.740+0000 7fc5d624c640 1 -- 172.21.15.16:0/930332490 learned_addr learned my addr 172.21.15.16:0/930332490 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:42:12.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:12.740+0000 7fc5d5a4b640 1 --2- 172.21.15.16:0/930332490 >> v2:172.21.15.62:3300/0 conn(0x7fc5d00cc4e0 0x7fc5d006bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:12.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:12.740+0000 7fc5d6a4d640 1 --2- 172.21.15.16:0/930332490 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc5d015d650 0x7fc5d015dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:12.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:12.740+0000 7fc5d524a640 1 -- 172.21.15.16:0/930332490 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1785540792 0 0) 0x7fc5d00cc960 con 0x7fc5d0070c10 2023-05-22T15:42:12.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:12.740+0000 7fc5d524a640 1 -- 172.21.15.16:0/930332490 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5bc003530 con 0x7fc5d0070c10 2023-05-22T15:42:12.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:12.740+0000 7fc5d524a640 1 -- 172.21.15.16:0/930332490 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3075260642 0 0) 0x7fc5bc003530 con 0x7fc5d0070c10 2023-05-22T15:42:12.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:12.740+0000 7fc5d524a640 1 -- 172.21.15.16:0/930332490 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc5d015d650 msgr2=0x7fc5d015dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:42:12.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:12.740+0000 7fc5d524a640 1 --2- 172.21.15.16:0/930332490 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc5d015d650 0x7fc5d015dad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:12.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:12.740+0000 7fc5d524a640 1 -- 172.21.15.16:0/930332490 >> v2:172.21.15.62:3300/0 conn(0x7fc5d00cc4e0 msgr2=0x7fc5d006bfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:42:12.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:12.740+0000 7fc5d524a640 1 --2- 172.21.15.16:0/930332490 >> v2:172.21.15.62:3300/0 conn(0x7fc5d00cc4e0 0x7fc5d006bfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:12.746 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:12.740+0000 7fc5d524a640 1 -- 172.21.15.16:0/930332490 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5d00cd7c0 con 0x7fc5d0070c10 2023-05-22T15:42:12.746 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:12.740+0000 7fc5d524a640 1 -- 172.21.15.16:0/930332490 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7fc5cc001680 con 0x7fc5d0070c10 2023-05-22T15:42:12.746 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:12.740+0000 7fc5d524a640 1 -- 172.21.15.16:0/930332490 >> v1:172.21.15.16:6789/0 conn(0x7fc5d0070c10 legacy=0x7fc5d00c9e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:12.746 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:12.740+0000 7fc5d524a640 1 --2- 172.21.15.16:0/930332490 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc5d015d650 0x7fc5bc003980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:12.746 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:12.740+0000 7fc5d524a640 1 --2- 172.21.15.16:0/930332490 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc5bc003f00 0x7fc5bc0047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:12.746 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:12.740+0000 7fc5d524a640 1 --2- 172.21.15.16:0/930332490 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc5bc004d10 0x7fc5bc005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:12.746 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:12.740+0000 7fc5d524a640 1 -- 172.21.15.16:0/930332490 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5d015e010 con 0x7fc5bc004d10 2023-05-22T15:42:12.747 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:12.740+0000 7fc5d524a640 1 -- 172.21.15.16:0/930332490 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5d00ccbb0 con 0x7fc5bc003f00 2023-05-22T15:42:12.747 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:12.740+0000 7fc5d524a640 1 -- 172.21.15.16:0/930332490 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5bc00b990 con 0x7fc5d015d650 2023-05-22T15:42:12.747 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:12.740+0000 7fc5d6a4d640 1 --2- 172.21.15.16:0/930332490 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc5bc004d10 0x7fc5bc005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:12.747 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:12.740+0000 7fc5d5a4b640 1 --2- 172.21.15.16:0/930332490 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc5bc003f00 0x7fc5bc0047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:12.747 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:12.740+0000 7fc5d624c640 1 --2- 172.21.15.16:0/930332490 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc5d015d650 0x7fc5bc003980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:12.747 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:12.740+0000 7fc5d6a4d640 1 -- 172.21.15.16:0/930332490 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc5d015d650 msgr2=0x7fc5bc003980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:42:12.747 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:12.740+0000 7fc5d6a4d640 1 --2- 172.21.15.16:0/930332490 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc5d015d650 0x7fc5bc003980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:12.747 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:12.740+0000 7fc5d6a4d640 1 -- 172.21.15.16:0/930332490 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc5bc003f00 msgr2=0x7fc5bc0047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:42:12.748 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:12.740+0000 7fc5d6a4d640 1 --2- 172.21.15.16:0/930332490 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc5bc003f00 0x7fc5bc0047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:12.748 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:12.740+0000 7fc5d6a4d640 1 -- 172.21.15.16:0/930332490 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5bc003530 con 0x7fc5bc004d10 2023-05-22T15:42:12.748 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:12.740+0000 7fc5d5a4b640 1 --2- 172.21.15.16:0/930332490 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc5bc003f00 0x7fc5bc0047d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T15:42:12.750 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:12.740+0000 7fc5d6a4d640 1 --2- 172.21.15.16:0/930332490 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc5bc004d10 0x7fc5bc005180 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7fc5c8009fc0 tx=0x7fc5c801cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4135fe67facbc9e7 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:12.750 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:12.740+0000 7fc5d524a640 1 -- 172.21.15.16:0/930332490 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5c80a5c70 con 0x7fc5bc004d10 2023-05-22T15:42:12.750 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:12.740+0000 7fc5d524a640 1 -- 172.21.15.16:0/930332490 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5c800f040 con 0x7fc5bc004d10 2023-05-22T15:42:12.750 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:12.744+0000 7fc5d524a640 1 -- 172.21.15.16:0/930332490 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5c80ad440 con 0x7fc5bc004d10 2023-05-22T15:42:12.751 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:12.744+0000 7fc5d84b0640 1 -- 172.21.15.16:0/930332490 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc5bc004d10 msgr2=0x7fc5bc005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:12.751 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:12.744+0000 7fc5d84b0640 1 --2- 172.21.15.16:0/930332490 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc5bc004d10 0x7fc5bc005180 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7fc5c8009fc0 tx=0x7fc5c801cc60 comp rx=0 tx=0).stop 2023-05-22T15:42:12.751 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:12.744+0000 7fc5d6a4d640 1 -- 172.21.15.16:0/930332490 reap_dead start 2023-05-22T15:42:12.751 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:12.744+0000 7fc5d84b0640 1 -- 172.21.15.16:0/930332490 shutdown_connections 2023-05-22T15:42:12.751 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:12.744+0000 7fc5d84b0640 1 -- 172.21.15.16:0/930332490 >> 172.21.15.16:0/930332490 conn(0x7fc5d0069720 msgr2=0x7fc5d0069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:12.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:12.744+0000 7fc5d84b0640 1 -- 172.21.15.16:0/930332490 shutdown_connections 2023-05-22T15:42:12.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:12.744+0000 7fc5d84b0640 1 -- 172.21.15.16:0/930332490 wait complete. 2023-05-22T15:42:12.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:12.744+0000 7fc5d84b0640 1 Processor -- start 2023-05-22T15:42:12.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:12.744+0000 7fc5d84b0640 1 -- start start 2023-05-22T15:42:12.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:12.744+0000 7fc5d84b0640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc5bc004d10 0x7fc5d00c6310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:12.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:12.744+0000 7fc5d84b0640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc5d0150f70 0x7fc5d00b1350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:12.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:12.744+0000 7fc5d84b0640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc5d00cad20 0x7fc5d00cb1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:12.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:12.744+0000 7fc5d84b0640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5d00cb700 con 0x7fc5bc004d10 2023-05-22T15:42:12.753 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:12.744+0000 7fc5d84b0640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5d01513e0 con 0x7fc5d0150f70 2023-05-22T15:42:12.753 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:12.744+0000 7fc5d84b0640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5d00b1890 con 0x7fc5d00cad20 2023-05-22T15:42:12.754 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:12.744+0000 7fc5d624c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc5bc004d10 0x7fc5d00c6310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:12.754 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:12.744+0000 7fc5d624c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc5bc004d10 0x7fc5d00c6310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41318/0 (socket says 172.21.15.16:41318) 2023-05-22T15:42:12.754 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:12.744+0000 7fc5d624c640 1 -- 172.21.15.16:0/2312691535 learned_addr learned my addr 172.21.15.16:0/2312691535 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:42:12.754 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:12.744+0000 7fc5d5a4b640 1 --2- 172.21.15.16:0/2312691535 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc5d0150f70 0x7fc5d00b1350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:12.754 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:12.744+0000 7fc5d6a4d640 1 --2- 172.21.15.16:0/2312691535 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc5d00cad20 0x7fc5d00cb1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:12.754 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:12.744+0000 7fc5d624c640 1 -- 172.21.15.16:0/2312691535 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc5d00cad20 msgr2=0x7fc5d00cb1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:12.755 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:12.744+0000 7fc5d624c640 1 --2- 172.21.15.16:0/2312691535 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc5d00cad20 0x7fc5d00cb1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:12.755 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:12.744+0000 7fc5d624c640 1 -- 172.21.15.16:0/2312691535 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc5d0150f70 msgr2=0x7fc5d00b1350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:12.755 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:12.744+0000 7fc5d624c640 1 --2- 172.21.15.16:0/2312691535 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc5d0150f70 0x7fc5d00b1350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:12.755 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:12.744+0000 7fc5d624c640 1 -- 172.21.15.16:0/2312691535 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5d00cdd80 con 0x7fc5bc004d10 2023-05-22T15:42:12.755 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:12.744+0000 7fc5d624c640 1 --2- 172.21.15.16:0/2312691535 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc5bc004d10 0x7fc5d00c6310 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7fc5cc00e850 tx=0x7fc5cc00ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:12.755 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:12.744+0000 7fc5c6ffd640 1 -- 172.21.15.16:0/2312691535 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5cc00c8d0 con 0x7fc5bc004d10 2023-05-22T15:42:12.755 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:12.744+0000 7fc5c6ffd640 1 -- 172.21.15.16:0/2312691535 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5cc00ca30 con 0x7fc5bc004d10 2023-05-22T15:42:12.755 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:12.744+0000 7fc5d84b0640 1 -- 172.21.15.16:0/2312691535 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5d00cdaa0 con 0x7fc5bc004d10 2023-05-22T15:42:12.756 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:12.744+0000 7fc5c6ffd640 1 -- 172.21.15.16:0/2312691535 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5cc00cde0 con 0x7fc5bc004d10 2023-05-22T15:42:12.756 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:12.744+0000 7fc5d84b0640 1 -- 172.21.15.16:0/2312691535 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5d0070fe0 con 0x7fc5bc004d10 2023-05-22T15:42:12.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:12.748+0000 7fc5d84b0640 1 -- 172.21.15.16:0/2312691535 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5d004dbe0 con 0x7fc5bc004d10 2023-05-22T15:42:12.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:12.748+0000 7fc5c6ffd640 1 -- 172.21.15.16:0/2312691535 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) v1 ==== 84332+0+0 (secure 0 0 0) 0x7fc5cc013420 con 0x7fc5bc004d10 2023-05-22T15:42:12.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:12.748+0000 7fc5c6ffd640 1 --2- 172.21.15.16:0/2312691535 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fc5b40654d0 0x7fc5b4067980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:12.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:12.748+0000 7fc5c6ffd640 1 -- 172.21.15.16:0/2312691535 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 6632+0+0 (secure 0 0 0) 0x7fc5cc082780 con 0x7fc5bc004d10 2023-05-22T15:42:12.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:12.748+0000 7fc5d5a4b640 1 --2- 172.21.15.16:0/2312691535 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fc5b40654d0 0x7fc5b4067980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:12.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:12.748+0000 7fc5d5a4b640 1 --2- 172.21.15.16:0/2312691535 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fc5b40654d0 0x7fc5b4067980 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fc5d0070650 tx=0x7fc5c0008040 comp rx=0 tx=0).ready entity=mgr.6765 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:12.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:12.752+0000 7fc5c6ffd640 1 -- 172.21.15.16:0/2312691535 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fc5cc050190 con 0x7fc5bc004d10 2023-05-22T15:42:12.915 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:12.912+0000 7fc5d84b0640 1 -- 172.21.15.16:0/2312691535 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fc5d006bd80 con 0x7fc5bc004d10 2023-05-22T15:42:13.593 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:13.580+0000 7fc5c6ffd640 1 -- 172.21.15.16:0/2312691535 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v81) v1 ==== 169+0+0 (secure 0 0 0) 0x7fc5cc0537b0 con 0x7fc5bc004d10 2023-05-22T15:42:13.596 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'cephfs_data' removed 2023-05-22T15:42:13.606 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:13.604+0000 7fc5d84b0640 1 -- 172.21.15.16:0/2312691535 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fc5b40654d0 msgr2=0x7fc5b4067980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:13.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:13.604+0000 7fc5d84b0640 1 --2- 172.21.15.16:0/2312691535 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fc5b40654d0 0x7fc5b4067980 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fc5d0070650 tx=0x7fc5c0008040 comp rx=0 tx=0).stop 2023-05-22T15:42:13.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:13.604+0000 7fc5d84b0640 1 -- 172.21.15.16:0/2312691535 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc5bc004d10 msgr2=0x7fc5d00c6310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:13.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:13.604+0000 7fc5d84b0640 1 --2- 172.21.15.16:0/2312691535 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc5bc004d10 0x7fc5d00c6310 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7fc5cc00e850 tx=0x7fc5cc00ed10 comp rx=0 tx=0).stop 2023-05-22T15:42:13.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:13.604+0000 7fc5d84b0640 1 -- 172.21.15.16:0/2312691535 shutdown_connections 2023-05-22T15:42:13.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:13.604+0000 7fc5d84b0640 1 --2- 172.21.15.16:0/2312691535 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fc5b40654d0 0x7fc5b4067980 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:13.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:13.604+0000 7fc5d84b0640 1 --2- 172.21.15.16:0/2312691535 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc5d00cad20 0x7fc5d00cb1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:13.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:13.604+0000 7fc5d84b0640 1 --2- 172.21.15.16:0/2312691535 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc5d0150f70 0x7fc5d00b1350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:13.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:13.604+0000 7fc5d84b0640 1 --2- 172.21.15.16:0/2312691535 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc5bc004d10 0x7fc5d00c6310 unknown :-1 s=CLOSED pgs=1046 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:13.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:13.604+0000 7fc5d84b0640 1 -- 172.21.15.16:0/2312691535 >> 172.21.15.16:0/2312691535 conn(0x7fc5d0069720 msgr2=0x7fc5d00ca5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:13.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:13.604+0000 7fc5d84b0640 1 -- 172.21.15.16:0/2312691535 shutdown_connections 2023-05-22T15:42:13.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:13.604+0000 7fc5d84b0640 1 -- 172.21.15.16:0/2312691535 wait complete. 2023-05-22T15:42:13.629 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2023-05-22T15:42:13.755 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:13.752+0000 7f859c80c640 1 Processor -- start 2023-05-22T15:42:13.755 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:13.752+0000 7f859c80c640 1 -- start start 2023-05-22T15:42:13.756 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:13.752+0000 7f859c80c640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f859815bca0 0x7f85980cda80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:13.756 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:13.752+0000 7f859c80c640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f85980b5a50 0x7f85980d2900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:13.756 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:13.752+0000 7f859c80c640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85980cccc0 con 0x7f85980d2480 2023-05-22T15:42:13.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:13.752+0000 7f859c80c640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85980cd660 con 0x7f859815bca0 2023-05-22T15:42:13.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:13.752+0000 7f859c80c640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85980d2e40 con 0x7f85980b5a50 2023-05-22T15:42:13.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:13.752+0000 7f859659c640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f859815bca0 0x7f85980cda80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:13.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:13.752+0000 7f8595d9b640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f85980d2480 0x7f85980b5340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:41712/0 (socket says 172.21.15.16:41712) 2023-05-22T15:42:13.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:13.752+0000 7f8595d9b640 1 -- 172.21.15.16:0/15367154 learned_addr learned my addr 172.21.15.16:0/15367154 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:42:13.758 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:13.752+0000 7f8596d9d640 1 --2- 172.21.15.16:0/15367154 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f85980b5a50 0x7f85980d2900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:13.758 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:13.752+0000 7f859659c640 1 -- 172.21.15.16:0/15367154 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f85980b5a50 msgr2=0x7f85980d2900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:42:13.758 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:13.752+0000 7f859659c640 1 --2- 172.21.15.16:0/15367154 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f85980b5a50 0x7f85980d2900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:13.758 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:13.752+0000 7f859559a640 1 -- 172.21.15.16:0/15367154 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3179907284 0 0) 0x7f85980cccc0 con 0x7f85980d2480 2023-05-22T15:42:13.758 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:13.752+0000 7f859659c640 1 -- 172.21.15.16:0/15367154 >> v1:172.21.15.16:6789/0 conn(0x7f85980d2480 legacy=0x7f85980b5340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:13.759 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:13.752+0000 7f859659c640 1 -- 172.21.15.16:0/15367154 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85980ccfa0 con 0x7f859815bca0 2023-05-22T15:42:13.759 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:13.756+0000 7f859659c640 1 --2- 172.21.15.16:0/15367154 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f859815bca0 0x7f85980cda80 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7f85900097c0 tx=0x7f859001c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52693d822ce6a8b5 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:13.759 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:13.756+0000 7f859559a640 1 -- 172.21.15.16:0/15367154 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85900a6040 con 0x7f859815bca0 2023-05-22T15:42:13.759 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:13.756+0000 7f859559a640 1 -- 172.21.15.16:0/15367154 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85900aa600 con 0x7f859815bca0 2023-05-22T15:42:13.760 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:13.756+0000 7f859559a640 1 -- 172.21.15.16:0/15367154 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85900aa990 con 0x7f859815bca0 2023-05-22T15:42:13.760 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:13.756+0000 7f859c80c640 1 -- 172.21.15.16:0/15367154 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f859815bca0 msgr2=0x7f85980cda80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:13.760 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:13.756+0000 7f859c80c640 1 --2- 172.21.15.16:0/15367154 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f859815bca0 0x7f85980cda80 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7f85900097c0 tx=0x7f859001c480 comp rx=0 tx=0).stop 2023-05-22T15:42:13.761 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:13.756+0000 7f859c80c640 1 -- 172.21.15.16:0/15367154 shutdown_connections 2023-05-22T15:42:13.761 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:13.756+0000 7f859c80c640 1 --2- 172.21.15.16:0/15367154 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f85980b5a50 0x7f85980d2900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:13.761 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:13.756+0000 7f859c80c640 1 --2- 172.21.15.16:0/15367154 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f859815bca0 0x7f85980cda80 unknown :-1 s=CLOSED pgs=1048 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:13.761 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:13.756+0000 7f859c80c640 1 -- 172.21.15.16:0/15367154 >> 172.21.15.16:0/15367154 conn(0x7f85980655c0 msgr2=0x7f8598070e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:13.761 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:13.756+0000 7f859c80c640 1 -- 172.21.15.16:0/15367154 shutdown_connections 2023-05-22T15:42:13.761 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:13.756+0000 7f859c80c640 1 -- 172.21.15.16:0/15367154 wait complete. 2023-05-22T15:42:13.763 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:13.756+0000 7f859c80c640 1 Processor -- start 2023-05-22T15:42:13.763 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:13.756+0000 7f859c80c640 1 -- start start 2023-05-22T15:42:13.763 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:13.756+0000 7f859c80c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f85980b5a50 0x7f85981661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:13.764 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:13.756+0000 7f859c80c640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f85980d2480 0x7f85980ca900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:13.764 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:13.756+0000 7f859c80c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f859815bca0 0x7f85980cae40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:13.764 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:13.760+0000 7f859c80c640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8598154770 con 0x7f859815bca0 2023-05-22T15:42:13.764 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:13.760+0000 7f859c80c640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85980ca470 con 0x7f85980b5a50 2023-05-22T15:42:13.764 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:13.760+0000 7f859c80c640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85980c98f0 con 0x7f85980d2480 2023-05-22T15:42:13.764 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:13.760+0000 7f8596d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f859815bca0 0x7f85980cae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:13.764 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:13.760+0000 7f8596d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f859815bca0 0x7f85980cae40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41334/0 (socket says 172.21.15.16:41334) 2023-05-22T15:42:13.764 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:13.760+0000 7f8596d9d640 1 -- 172.21.15.16:0/1226995037 learned_addr learned my addr 172.21.15.16:0/1226995037 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:42:13.765 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:13.760+0000 7f859659c640 1 --2- 172.21.15.16:0/1226995037 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f85980b5a50 0x7f85981661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:13.765 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:13.760+0000 7f8596d9d640 1 -- 172.21.15.16:0/1226995037 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f85980d2480 msgr2=0x7f85980ca900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:13.765 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:13.760+0000 7f8595d9b640 1 --2- 172.21.15.16:0/1226995037 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f85980d2480 0x7f85980ca900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:13.765 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:13.760+0000 7f8596d9d640 1 --2- 172.21.15.16:0/1226995037 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f85980d2480 0x7f85980ca900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:13.765 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:13.760+0000 7f8596d9d640 1 -- 172.21.15.16:0/1226995037 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f85980b5a50 msgr2=0x7f85981661b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:13.765 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:13.760+0000 7f8596d9d640 1 --2- 172.21.15.16:0/1226995037 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f85980b5a50 0x7f85981661b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:13.765 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:13.760+0000 7f8596d9d640 1 -- 172.21.15.16:0/1226995037 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8590009420 con 0x7f859815bca0 2023-05-22T15:42:13.766 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:13.760+0000 7f859659c640 1 --2- 172.21.15.16:0/1226995037 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f85980b5a50 0x7f85981661b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:42:13.766 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:13.760+0000 7f8595d9b640 1 --2- 172.21.15.16:0/1226995037 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f85980d2480 0x7f85980ca900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:42:13.766 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:13.760+0000 7f8596d9d640 1 --2- 172.21.15.16:0/1226995037 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f859815bca0 0x7f85980cae40 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7f858800d590 tx=0x7f858800da50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:13.766 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:13.760+0000 7f858effd640 1 -- 172.21.15.16:0/1226995037 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85880125f0 con 0x7f859815bca0 2023-05-22T15:42:13.766 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:13.760+0000 7f858effd640 1 -- 172.21.15.16:0/1226995037 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8588012770 con 0x7f859815bca0 2023-05-22T15:42:13.766 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:13.760+0000 7f859c80c640 1 -- 172.21.15.16:0/1226995037 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8598160360 con 0x7f859815bca0 2023-05-22T15:42:13.766 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:13.760+0000 7f858effd640 1 -- 172.21.15.16:0/1226995037 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8588012b20 con 0x7f859815bca0 2023-05-22T15:42:13.766 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:13.760+0000 7f859c80c640 1 -- 172.21.15.16:0/1226995037 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85980cdfc0 con 0x7f859815bca0 2023-05-22T15:42:13.767 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:13.760+0000 7f859c80c640 1 -- 172.21.15.16:0/1226995037 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85981604f0 con 0x7f859815bca0 2023-05-22T15:42:13.767 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:13.760+0000 7f858effd640 1 -- 172.21.15.16:0/1226995037 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f8588012c80 con 0x7f859815bca0 2023-05-22T15:42:13.768 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:13.760+0000 7f858effd640 1 --2- 172.21.15.16:0/1226995037 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f85780654a0 0x7f8578067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:13.768 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:13.760+0000 7f858effd640 1 -- 172.21.15.16:0/1226995037 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6218+0+0 (secure 0 0 0) 0x7f8588082e70 con 0x7f859815bca0 2023-05-22T15:42:13.768 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:13.760+0000 7f859659c640 1 --2- 172.21.15.16:0/1226995037 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f85780654a0 0x7f8578067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:13.768 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:13.760+0000 7f859659c640 1 --2- 172.21.15.16:0/1226995037 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f85780654a0 0x7f8578067950 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f85900ae000 tx=0x7f8590008040 comp rx=0 tx=0).ready entity=mgr.6765 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:13.769 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:13.764+0000 7f858effd640 1 -- 172.21.15.16:0/1226995037 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f85880509d0 con 0x7f859815bca0 2023-05-22T15:42:13.932 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:13.928+0000 7f859c80c640 1 -- 172.21.15.16:0/1226995037 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7f85981f42f0 con 0x7f859815bca0 2023-05-22T15:42:13.947 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:13.944+0000 7f858effd640 1 -- 172.21.15.16:0/1226995037 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v19) v1 ==== 120+0+0 (secure 0 0 0) 0x7f8588053ff0 con 0x7f859815bca0 2023-05-22T15:42:13.949 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:13.944+0000 7f859c80c640 1 -- 172.21.15.16:0/1226995037 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f85780654a0 msgr2=0x7f8578067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:13.950 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:13.944+0000 7f859c80c640 1 --2- 172.21.15.16:0/1226995037 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f85780654a0 0x7f8578067950 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f85900ae000 tx=0x7f8590008040 comp rx=0 tx=0).stop 2023-05-22T15:42:13.950 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:13.944+0000 7f859c80c640 1 -- 172.21.15.16:0/1226995037 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f859815bca0 msgr2=0x7f85980cae40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:13.950 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:13.944+0000 7f859c80c640 1 --2- 172.21.15.16:0/1226995037 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f859815bca0 0x7f85980cae40 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7f858800d590 tx=0x7f858800da50 comp rx=0 tx=0).stop 2023-05-22T15:42:13.950 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:13.944+0000 7f859c80c640 1 -- 172.21.15.16:0/1226995037 shutdown_connections 2023-05-22T15:42:13.950 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:13.944+0000 7f859c80c640 1 --2- 172.21.15.16:0/1226995037 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f859815bca0 0x7f85980cae40 unknown :-1 s=CLOSED pgs=1049 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:13.950 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:13.944+0000 7f859c80c640 1 --2- 172.21.15.16:0/1226995037 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f85780654a0 0x7f8578067950 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:13.951 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:13.944+0000 7f859c80c640 1 --2- 172.21.15.16:0/1226995037 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f85980d2480 0x7f85980ca900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:13.951 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:13.944+0000 7f859c80c640 1 --2- 172.21.15.16:0/1226995037 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f85980b5a50 0x7f85981661b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:13.951 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:13.944+0000 7f859c80c640 1 -- 172.21.15.16:0/1226995037 >> 172.21.15.16:0/1226995037 conn(0x7f85980655c0 msgr2=0x7f85980d0030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:13.951 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:13.944+0000 7f859c80c640 1 -- 172.21.15.16:0/1226995037 shutdown_connections 2023-05-22T15:42:13.951 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:13.944+0000 7f859c80c640 1 -- 172.21.15.16:0/1226995037 wait complete. 2023-05-22T15:42:13.968 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2023-05-22T15:42:14.089 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.084+0000 7f9db580d640 1 Processor -- start 2023-05-22T15:42:14.089 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.084+0000 7f9db580d640 1 -- start start 2023-05-22T15:42:14.090 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.084+0000 7f9db580d640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f9db015bca0 0x7f9db0160160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:14.090 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.084+0000 7f9db580d640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9db00b4a70 0x7f9db00d0890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:14.090 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.084+0000 7f9db580d640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9db00d0400 con 0x7f9db00b3ee0 2023-05-22T15:42:14.090 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.084+0000 7f9db580d640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9db015c070 con 0x7f9db015bca0 2023-05-22T15:42:14.091 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.084+0000 7f9db580d640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9db01606a0 con 0x7f9db00b4a70 2023-05-22T15:42:14.091 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.088+0000 7f9dae7fc640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f9db00b3ee0 0x7f9db00b4360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:41714/0 (socket says 172.21.15.16:41714) 2023-05-22T15:42:14.091 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.088+0000 7f9dae7fc640 1 -- 172.21.15.16:0/53968062 learned_addr learned my addr 172.21.15.16:0/53968062 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:42:14.091 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.088+0000 7f9daeffd640 1 --2- 172.21.15.16:0/53968062 >> v2:172.21.15.62:3300/0 conn(0x7f9db015bca0 0x7f9db0160160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:14.091 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.088+0000 7f9daf7fe640 1 --2- 172.21.15.16:0/53968062 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9db00b4a70 0x7f9db00d0890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:14.092 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.088+0000 7f9dadffb640 1 -- 172.21.15.16:0/53968062 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3405082150 0 0) 0x7f9db00d0400 con 0x7f9db00b3ee0 2023-05-22T15:42:14.092 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.088+0000 7f9dadffb640 1 -- 172.21.15.16:0/53968062 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d98003530 con 0x7f9db00b3ee0 2023-05-22T15:42:14.092 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.088+0000 7f9daeffd640 1 -- 172.21.15.16:0/53968062 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9db00b4a70 msgr2=0x7f9db00d0890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:42:14.092 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.088+0000 7f9daeffd640 1 --2- 172.21.15.16:0/53968062 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9db00b4a70 0x7f9db00d0890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:14.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.088+0000 7f9dadffb640 1 -- 172.21.15.16:0/53968062 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 454958786 0 0) 0x7f9d98003530 con 0x7f9db00b3ee0 2023-05-22T15:42:14.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.088+0000 7f9daeffd640 1 -- 172.21.15.16:0/53968062 >> v1:172.21.15.16:6789/0 conn(0x7f9db00b3ee0 legacy=0x7f9db00b4360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:14.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.088+0000 7f9daeffd640 1 -- 172.21.15.16:0/53968062 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9db00d00d0 con 0x7f9db015bca0 2023-05-22T15:42:14.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.088+0000 7f9daeffd640 1 --2- 172.21.15.16:0/53968062 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9db015bca0 0x7f9db0160160 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f9da0009d90 tx=0x7f9da001c720 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a398651545c9ac11 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:14.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.088+0000 7f9dadffb640 1 -- 172.21.15.16:0/53968062 <== mon.1 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9da4001680 con 0x7f9db015bca0 2023-05-22T15:42:14.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.088+0000 7f9dadffb640 1 -- 172.21.15.16:0/53968062 <== mon.1 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9da00a5520 con 0x7f9db015bca0 2023-05-22T15:42:14.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.088+0000 7f9dadffb640 1 -- 172.21.15.16:0/53968062 <== mon.1 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9da00a5890 con 0x7f9db015bca0 2023-05-22T15:42:14.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.088+0000 7f9db580d640 1 -- 172.21.15.16:0/53968062 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9db015bca0 msgr2=0x7f9db0160160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:14.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.088+0000 7f9db580d640 1 --2- 172.21.15.16:0/53968062 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9db015bca0 0x7f9db0160160 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f9da0009d90 tx=0x7f9da001c720 comp rx=0 tx=0).stop 2023-05-22T15:42:14.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.092+0000 7f9db580d640 1 -- 172.21.15.16:0/53968062 shutdown_connections 2023-05-22T15:42:14.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.092+0000 7f9db580d640 1 --2- 172.21.15.16:0/53968062 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9db00b4a70 0x7f9db00d0890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:14.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.092+0000 7f9db580d640 1 --2- 172.21.15.16:0/53968062 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9db015bca0 0x7f9db0160160 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:14.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.092+0000 7f9db580d640 1 -- 172.21.15.16:0/53968062 >> 172.21.15.16:0/53968062 conn(0x7f9db00657f0 msgr2=0x7f9db0070e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:14.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.092+0000 7f9db580d640 1 -- 172.21.15.16:0/53968062 shutdown_connections 2023-05-22T15:42:14.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.092+0000 7f9db580d640 1 -- 172.21.15.16:0/53968062 wait complete. 2023-05-22T15:42:14.096 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.092+0000 7f9db580d640 1 Processor -- start 2023-05-22T15:42:14.096 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.092+0000 7f9db580d640 1 -- start start 2023-05-22T15:42:14.097 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.092+0000 7f9db580d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9db00b3ee0 0x7f9db0148400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:14.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.092+0000 7f9db580d640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9db00b4a70 0x7f9db00ca1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:14.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.092+0000 7f9db580d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9db015bca0 0x7f9db00b68d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:14.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.092+0000 7f9db580d640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9db00cbda0 con 0x7f9db00b3ee0 2023-05-22T15:42:14.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.092+0000 7f9db580d640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9db0148940 con 0x7f9db015bca0 2023-05-22T15:42:14.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.092+0000 7f9db580d640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9db00cb190 con 0x7f9db00b4a70 2023-05-22T15:42:14.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.092+0000 7f9daeffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9db00b3ee0 0x7f9db0148400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:14.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.092+0000 7f9daeffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9db00b3ee0 0x7f9db0148400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41346/0 (socket says 172.21.15.16:41346) 2023-05-22T15:42:14.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.092+0000 7f9daeffd640 1 -- 172.21.15.16:0/2405495677 learned_addr learned my addr 172.21.15.16:0/2405495677 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:42:14.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.092+0000 7f9daf7fe640 1 --2- 172.21.15.16:0/2405495677 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9db015bca0 0x7f9db00b68d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:14.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.092+0000 7f9dae7fc640 1 --2- 172.21.15.16:0/2405495677 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9db00b4a70 0x7f9db00ca1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:14.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.092+0000 7f9daeffd640 1 -- 172.21.15.16:0/2405495677 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9db00b4a70 msgr2=0x7f9db00ca1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:14.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.092+0000 7f9daeffd640 1 --2- 172.21.15.16:0/2405495677 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9db00b4a70 0x7f9db00ca1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:14.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.092+0000 7f9daeffd640 1 -- 172.21.15.16:0/2405495677 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9db015bca0 msgr2=0x7f9db00b68d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:14.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.092+0000 7f9daeffd640 1 --2- 172.21.15.16:0/2405495677 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9db015bca0 0x7f9db00b68d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:14.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.092+0000 7f9daeffd640 1 -- 172.21.15.16:0/2405495677 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9da0009a70 con 0x7f9db00b3ee0 2023-05-22T15:42:14.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.092+0000 7f9daeffd640 1 --2- 172.21.15.16:0/2405495677 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9db00b3ee0 0x7f9db0148400 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7f9da0009d90 tx=0x7f9da00194c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:14.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.092+0000 7f9d97fff640 1 -- 172.21.15.16:0/2405495677 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9da00ad640 con 0x7f9db00b3ee0 2023-05-22T15:42:14.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.092+0000 7f9d97fff640 1 -- 172.21.15.16:0/2405495677 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9da00ad7a0 con 0x7f9db00b3ee0 2023-05-22T15:42:14.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.092+0000 7f9db580d640 1 -- 172.21.15.16:0/2405495677 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9db00d15e0 con 0x7f9db00b3ee0 2023-05-22T15:42:14.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.092+0000 7f9d97fff640 1 -- 172.21.15.16:0/2405495677 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9da00adb10 con 0x7f9db00b3ee0 2023-05-22T15:42:14.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.092+0000 7f9db580d640 1 -- 172.21.15.16:0/2405495677 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9db00cd1a0 con 0x7f9db00b3ee0 2023-05-22T15:42:14.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.096+0000 7f9db580d640 1 -- 172.21.15.16:0/2405495677 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9db00d1770 con 0x7f9db00b3ee0 2023-05-22T15:42:14.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.096+0000 7f9d97fff640 1 -- 172.21.15.16:0/2405495677 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f9da00b4070 con 0x7f9db00b3ee0 2023-05-22T15:42:14.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.096+0000 7f9d97fff640 1 --2- 172.21.15.16:0/2405495677 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f9d9c0654a0 0x7f9d9c067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:14.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.096+0000 7f9d97fff640 1 -- 172.21.15.16:0/2405495677 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6218+0+0 (secure 0 0 0) 0x7f9da011d6a0 con 0x7f9db00b3ee0 2023-05-22T15:42:14.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.096+0000 7f9dae7fc640 1 --2- 172.21.15.16:0/2405495677 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f9d9c0654a0 0x7f9d9c067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:14.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.096+0000 7f9dae7fc640 1 --2- 172.21.15.16:0/2405495677 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f9d9c0654a0 0x7f9d9c067950 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f9db00b6310 tx=0x7f9da4009040 comp rx=0 tx=0).ready entity=mgr.6765 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:14.104 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.100+0000 7f9d97fff640 1 -- 172.21.15.16:0/2405495677 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f9da00eec50 con 0x7f9db00b3ee0 2023-05-22T15:42:14.259 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.256+0000 7f9db580d640 1 -- 172.21.15.16:0/2405495677 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f9db004dbe0 con 0x7f9db00b3ee0 2023-05-22T15:42:14.266 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.264+0000 7f9d97fff640 1 -- 172.21.15.16:0/2405495677 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v20) v1 ==== 104+0+0 (secure 0 0 0) 0x7f9da0009450 con 0x7f9db00b3ee0 2023-05-22T15:42:14.269 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.264+0000 7f9db580d640 1 -- 172.21.15.16:0/2405495677 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f9d9c0654a0 msgr2=0x7f9d9c067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:14.269 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.264+0000 7f9db580d640 1 --2- 172.21.15.16:0/2405495677 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f9d9c0654a0 0x7f9d9c067950 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f9db00b6310 tx=0x7f9da4009040 comp rx=0 tx=0).stop 2023-05-22T15:42:14.269 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.264+0000 7f9db580d640 1 -- 172.21.15.16:0/2405495677 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9db00b3ee0 msgr2=0x7f9db0148400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:14.269 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.264+0000 7f9db580d640 1 --2- 172.21.15.16:0/2405495677 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9db00b3ee0 0x7f9db0148400 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7f9da0009d90 tx=0x7f9da00194c0 comp rx=0 tx=0).stop 2023-05-22T15:42:14.270 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.264+0000 7f9db580d640 1 -- 172.21.15.16:0/2405495677 shutdown_connections 2023-05-22T15:42:14.270 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.264+0000 7f9db580d640 1 --2- 172.21.15.16:0/2405495677 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9db015bca0 0x7f9db00b68d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:14.270 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.264+0000 7f9db580d640 1 --2- 172.21.15.16:0/2405495677 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f9d9c0654a0 0x7f9d9c067950 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:14.270 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.264+0000 7f9db580d640 1 --2- 172.21.15.16:0/2405495677 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9db00b4a70 0x7f9db00ca1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:14.270 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.264+0000 7f9db580d640 1 --2- 172.21.15.16:0/2405495677 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9db00b3ee0 0x7f9db0148400 unknown :-1 s=CLOSED pgs=1051 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:14.270 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.264+0000 7f9db580d640 1 -- 172.21.15.16:0/2405495677 >> 172.21.15.16:0/2405495677 conn(0x7f9db00657f0 msgr2=0x7f9db0150db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:14.270 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.264+0000 7f9db580d640 1 -- 172.21.15.16:0/2405495677 shutdown_connections 2023-05-22T15:42:14.270 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.264+0000 7f9db580d640 1 -- 172.21.15.16:0/2405495677 wait complete. 2023-05-22T15:42:14.286 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2023-05-22T15:42:14.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.412+0000 7f5bea471640 1 Processor -- start 2023-05-22T15:42:14.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.412+0000 7f5bea471640 1 -- start start 2023-05-22T15:42:14.417 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.412+0000 7f5bea471640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f5be4159970 0x7f5be4159d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:14.417 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.412+0000 7f5bea471640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5be4070c30 con 0x7f5be415d680 2023-05-22T15:42:14.418 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.412+0000 7f5bea471640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5be4070e60 con 0x7f5be415db00 2023-05-22T15:42:14.418 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.412+0000 7f5bea471640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5be40cb970 con 0x7f5be4159970 2023-05-22T15:42:14.418 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.412+0000 7f5be37fe640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f5be415d680 0x7f5be40cb260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:41730/0 (socket says 172.21.15.16:41730) 2023-05-22T15:42:14.418 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.412+0000 7f5be37fe640 1 -- 172.21.15.16:0/2976772721 learned_addr learned my addr 172.21.15.16:0/2976772721 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:42:14.418 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.412+0000 7f5be3fff640 1 --2- 172.21.15.16:0/2976772721 >> v2:172.21.15.62:3300/0 conn(0x7f5be4159970 0x7f5be4159d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:14.419 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.412+0000 7f5be2ffd640 1 -- 172.21.15.16:0/2976772721 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2631532662 0 0) 0x7f5be4070c30 con 0x7f5be415d680 2023-05-22T15:42:14.420 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.412+0000 7f5be2ffd640 1 -- 172.21.15.16:0/2976772721 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5bd0003530 con 0x7f5be415d680 2023-05-22T15:42:14.420 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.416+0000 7f5be2ffd640 1 -- 172.21.15.16:0/2976772721 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1904126038 0 0) 0x7f5bd0003530 con 0x7f5be415d680 2023-05-22T15:42:14.420 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.416+0000 7f5be2ffd640 1 -- 172.21.15.16:0/2976772721 >> v2:172.21.15.62:3300/0 conn(0x7f5be4159970 msgr2=0x7f5be4159d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:42:14.420 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.416+0000 7f5be2ffd640 1 --2- 172.21.15.16:0/2976772721 >> v2:172.21.15.62:3300/0 conn(0x7f5be4159970 0x7f5be4159d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:14.420 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.416+0000 7f5be2ffd640 1 -- 172.21.15.16:0/2976772721 >> v1:172.21.15.62:6789/0 conn(0x7f5be415db00 legacy=0x7f5be406b6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:14.420 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.416+0000 7f5be2ffd640 1 -- 172.21.15.16:0/2976772721 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5be4071140 con 0x7f5be415d680 2023-05-22T15:42:14.421 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.416+0000 7f5be2ffd640 1 -- 172.21.15.16:0/2976772721 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7f5bcc001a00 con 0x7f5be415d680 2023-05-22T15:42:14.421 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.416+0000 7f5be2ffd640 1 -- 172.21.15.16:0/2976772721 >> v1:172.21.15.16:6789/0 conn(0x7f5be415d680 legacy=0x7f5be40cb260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:14.421 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.416+0000 7f5be2ffd640 1 --2- 172.21.15.16:0/2976772721 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5bd0003980 0x7f5bd0003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:14.421 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.416+0000 7f5be2ffd640 1 --2- 172.21.15.16:0/2976772721 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5bd00047d0 0x7f5bd0004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:14.421 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.416+0000 7f5be2ffd640 1 --2- 172.21.15.16:0/2976772721 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5bd0005180 0x7f5bd000b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:14.421 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.416+0000 7f5be2ffd640 1 -- 172.21.15.16:0/2976772721 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5be40cb970 con 0x7f5bd0005180 2023-05-22T15:42:14.421 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.416+0000 7f5be2ffd640 1 -- 172.21.15.16:0/2976772721 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bd000c090 con 0x7f5bd0003980 2023-05-22T15:42:14.421 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.416+0000 7f5be2ffd640 1 -- 172.21.15.16:0/2976772721 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5bd000c250 con 0x7f5bd00047d0 2023-05-22T15:42:14.422 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.416+0000 7f5be8a0e640 1 --2- 172.21.15.16:0/2976772721 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5bd0005180 0x7f5bd000b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:14.422 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.416+0000 7f5be3fff640 1 --2- 172.21.15.16:0/2976772721 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5bd0003980 0x7f5bd0003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:14.422 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.416+0000 7f5be37fe640 1 --2- 172.21.15.16:0/2976772721 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5bd00047d0 0x7f5bd0004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:14.422 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.416+0000 7f5be8a0e640 1 -- 172.21.15.16:0/2976772721 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5bd00047d0 msgr2=0x7f5bd0004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:42:14.422 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.416+0000 7f5be8a0e640 1 --2- 172.21.15.16:0/2976772721 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5bd00047d0 0x7f5bd0004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:14.422 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.416+0000 7f5be8a0e640 1 -- 172.21.15.16:0/2976772721 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5bd0003980 msgr2=0x7f5bd0003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:42:14.422 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.416+0000 7f5be8a0e640 1 --2- 172.21.15.16:0/2976772721 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5bd0003980 0x7f5bd0003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:14.423 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.416+0000 7f5be8a0e640 1 -- 172.21.15.16:0/2976772721 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bd0003530 con 0x7f5bd0005180 2023-05-22T15:42:14.423 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.416+0000 7f5be3fff640 1 --2- 172.21.15.16:0/2976772721 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5bd0003980 0x7f5bd0003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T15:42:14.425 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.416+0000 7f5be8a0e640 1 -- 172.21.15.16:0/2976772721 reap_dead start 2023-05-22T15:42:14.425 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.416+0000 7f5be8a0e640 1 --2- 172.21.15.16:0/2976772721 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5bd0005180 0x7f5bd000b990 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7f5bdc001780 tx=0x7f5bdc01c400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9069ce7fe3771c1 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:14.425 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.416+0000 7f5be2ffd640 1 -- 172.21.15.16:0/2976772721 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bdc0052d0 con 0x7f5bd0005180 2023-05-22T15:42:14.425 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.416+0000 7f5be2ffd640 1 -- 172.21.15.16:0/2976772721 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5bdc0a7050 con 0x7f5bd0005180 2023-05-22T15:42:14.425 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.416+0000 7f5be2ffd640 1 -- 172.21.15.16:0/2976772721 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bdc003600 con 0x7f5bd0005180 2023-05-22T15:42:14.425 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.416+0000 7f5bea471640 1 -- 172.21.15.16:0/2976772721 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5bd0005180 msgr2=0x7f5bd000b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:14.426 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.416+0000 7f5bea471640 1 --2- 172.21.15.16:0/2976772721 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5bd0005180 0x7f5bd000b990 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7f5bdc001780 tx=0x7f5bdc01c400 comp rx=0 tx=0).stop 2023-05-22T15:42:14.426 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.416+0000 7f5bea471640 1 -- 172.21.15.16:0/2976772721 shutdown_connections 2023-05-22T15:42:14.426 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.416+0000 7f5bea471640 1 --2- 172.21.15.16:0/2976772721 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5bd0005180 0x7f5bd000b990 unknown :-1 s=CLOSED pgs=1053 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:14.426 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.416+0000 7f5bea471640 1 -- 172.21.15.16:0/2976772721 >> 172.21.15.16:0/2976772721 conn(0x7f5be4069720 msgr2=0x7f5be4069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:14.426 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.416+0000 7f5bea471640 1 -- 172.21.15.16:0/2976772721 shutdown_connections 2023-05-22T15:42:14.426 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.416+0000 7f5bea471640 1 -- 172.21.15.16:0/2976772721 wait complete. 2023-05-22T15:42:14.426 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.420+0000 7f5bea471640 1 Processor -- start 2023-05-22T15:42:14.427 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.420+0000 7f5bea471640 1 -- start start 2023-05-22T15:42:14.427 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.420+0000 7f5bea471640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5bd0005180 0x7f5be4150650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:14.427 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.420+0000 7f5bea471640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5be4150df0 0x7f5be41532a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:14.427 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.420+0000 7f5bea471640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5be41537e0 0x7f5be41559f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:14.427 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.420+0000 7f5bea471640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5be41568c0 con 0x7f5be41537e0 2023-05-22T15:42:14.427 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.420+0000 7f5bea471640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5be4152e10 con 0x7f5be4150df0 2023-05-22T15:42:14.427 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.420+0000 7f5bea471640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5be4151f20 con 0x7f5bd0005180 2023-05-22T15:42:14.427 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.420+0000 7f5be8a0e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5be41537e0 0x7f5be41559f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:14.428 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.420+0000 7f5be8a0e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5be41537e0 0x7f5be41559f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41364/0 (socket says 172.21.15.16:41364) 2023-05-22T15:42:14.428 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.420+0000 7f5be8a0e640 1 -- 172.21.15.16:0/4145007374 learned_addr learned my addr 172.21.15.16:0/4145007374 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:42:14.428 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.420+0000 7f5be37fe640 1 --2- 172.21.15.16:0/4145007374 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5be4150df0 0x7f5be41532a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:14.428 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.420+0000 7f5be3fff640 1 --2- 172.21.15.16:0/4145007374 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5bd0005180 0x7f5be4150650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:14.428 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.420+0000 7f5be8a0e640 1 -- 172.21.15.16:0/4145007374 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5bd0005180 msgr2=0x7f5be4150650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:14.428 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.420+0000 7f5be8a0e640 1 --2- 172.21.15.16:0/4145007374 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5bd0005180 0x7f5be4150650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:14.428 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.420+0000 7f5be8a0e640 1 -- 172.21.15.16:0/4145007374 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5be4150df0 msgr2=0x7f5be41532a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:14.428 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.420+0000 7f5be8a0e640 1 --2- 172.21.15.16:0/4145007374 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5be4150df0 0x7f5be41532a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:14.429 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.420+0000 7f5be8a0e640 1 -- 172.21.15.16:0/4145007374 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5be4071700 con 0x7f5be41537e0 2023-05-22T15:42:14.429 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.420+0000 7f5be37fe640 1 --2- 172.21.15.16:0/4145007374 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5be4150df0 0x7f5be41532a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T15:42:14.429 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.420+0000 7f5be8a0e640 1 --2- 172.21.15.16:0/4145007374 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5be41537e0 0x7f5be41559f0 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7f5bdc01c230 tx=0x7f5bd0005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:14.429 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.420+0000 7f5be0ff9640 1 -- 172.21.15.16:0/4145007374 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bdc0b0070 con 0x7f5be41537e0 2023-05-22T15:42:14.429 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.420+0000 7f5be0ff9640 1 -- 172.21.15.16:0/4145007374 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5bdc0054c0 con 0x7f5be41537e0 2023-05-22T15:42:14.429 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.420+0000 7f5bea471640 1 -- 172.21.15.16:0/4145007374 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5be4071420 con 0x7f5be41537e0 2023-05-22T15:42:14.429 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.420+0000 7f5be0ff9640 1 -- 172.21.15.16:0/4145007374 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bdc007460 con 0x7f5be41537e0 2023-05-22T15:42:14.430 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.420+0000 7f5bea471640 1 -- 172.21.15.16:0/4145007374 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5be40cac90 con 0x7f5be41537e0 2023-05-22T15:42:14.430 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.420+0000 7f5bea471640 1 -- 172.21.15.16:0/4145007374 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5be4152fc0 con 0x7f5be41537e0 2023-05-22T15:42:14.430 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.424+0000 7f5be0ff9640 1 -- 172.21.15.16:0/4145007374 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f5bdc0067e0 con 0x7f5be41537e0 2023-05-22T15:42:14.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.424+0000 7f5be0ff9640 1 --2- 172.21.15.16:0/4145007374 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f5bc40654d0 0x7f5bc4067980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:14.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.424+0000 7f5be0ff9640 1 -- 172.21.15.16:0/4145007374 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6218+0+0 (secure 0 0 0) 0x7f5bdc11b700 con 0x7f5be41537e0 2023-05-22T15:42:14.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.424+0000 7f5be3fff640 1 --2- 172.21.15.16:0/4145007374 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f5bc40654d0 0x7f5bc4067980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:14.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.424+0000 7f5be3fff640 1 --2- 172.21.15.16:0/4145007374 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f5bc40654d0 0x7f5bc4067980 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f5bd4009910 tx=0x7f5bd4008040 comp rx=0 tx=0).ready entity=mgr.6765 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:14.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.428+0000 7f5be0ff9640 1 -- 172.21.15.16:0/4145007374 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f5be4152fc0 con 0x7f5be41537e0 2023-05-22T15:42:14.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.584+0000 7f5bea471640 1 -- 172.21.15.16:0/4145007374 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7f5be4152f80 con 0x7f5be41537e0 2023-05-22T15:42:14.596 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.592+0000 7f5be0ff9640 1 -- 172.21.15.16:0/4145007374 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v21) v1 ==== 115+0+0 (secure 0 0 0) 0x7f5bdc0e90c0 con 0x7f5be41537e0 2023-05-22T15:42:14.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.596+0000 7f5bea471640 1 -- 172.21.15.16:0/4145007374 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f5bc40654d0 msgr2=0x7f5bc4067980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:14.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.596+0000 7f5bea471640 1 --2- 172.21.15.16:0/4145007374 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f5bc40654d0 0x7f5bc4067980 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f5bd4009910 tx=0x7f5bd4008040 comp rx=0 tx=0).stop 2023-05-22T15:42:14.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.596+0000 7f5bea471640 1 -- 172.21.15.16:0/4145007374 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5be41537e0 msgr2=0x7f5be41559f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:14.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.596+0000 7f5bea471640 1 --2- 172.21.15.16:0/4145007374 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5be41537e0 0x7f5be41559f0 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7f5bdc01c230 tx=0x7f5bd0005800 comp rx=0 tx=0).stop 2023-05-22T15:42:14.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.596+0000 7f5bea471640 1 -- 172.21.15.16:0/4145007374 shutdown_connections 2023-05-22T15:42:14.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.596+0000 7f5bea471640 1 --2- 172.21.15.16:0/4145007374 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5be41537e0 0x7f5be41559f0 unknown :-1 s=CLOSED pgs=1054 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:14.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.596+0000 7f5bea471640 1 --2- 172.21.15.16:0/4145007374 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5be4150df0 0x7f5be41532a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:14.600 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.596+0000 7f5bea471640 1 --2- 172.21.15.16:0/4145007374 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f5bc40654d0 0x7f5bc4067980 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:14.600 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.596+0000 7f5bea471640 1 --2- 172.21.15.16:0/4145007374 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5bd0005180 0x7f5be4150650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:14.600 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.596+0000 7f5bea471640 1 -- 172.21.15.16:0/4145007374 >> 172.21.15.16:0/4145007374 conn(0x7f5be4069720 msgr2=0x7f5be406b6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:14.600 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.596+0000 7f5bea471640 1 -- 172.21.15.16:0/4145007374 shutdown_connections 2023-05-22T15:42:14.600 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.596+0000 7f5bea471640 1 -- 172.21.15.16:0/4145007374 wait complete. 2023-05-22T15:42:14.618 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_multi_snap_schedule' 2023-05-22T15:42:14.741 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.736+0000 7fe4f46d7640 1 Processor -- start 2023-05-22T15:42:14.741 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.736+0000 7fe4f46d7640 1 -- start start 2023-05-22T15:42:14.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.740+0000 7fe4f46d7640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe4ec0cb6a0 0x7fe4ec159340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:14.744 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.740+0000 7fe4f46d7640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe4ec0b0250 0x7fe4ec159880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:14.744 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.740+0000 7fe4f46d7640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4ec156b90 con 0x7fe4ec1567c0 2023-05-22T15:42:14.744 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.740+0000 7fe4f46d7640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4ec15a0b0 con 0x7fe4ec0cb6a0 2023-05-22T15:42:14.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.740+0000 7fe4f46d7640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4ec0ca8e0 con 0x7fe4ec0b0250 2023-05-22T15:42:14.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.740+0000 7fe4f1c72640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe4ec0cb6a0 0x7fe4ec159340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:14.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.740+0000 7fe4f1c72640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe4ec0cb6a0 0x7fe4ec159340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41374/0 (socket says 172.21.15.16:41374) 2023-05-22T15:42:14.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.740+0000 7fe4f1c72640 1 -- 172.21.15.16:0/147913860 learned_addr learned my addr 172.21.15.16:0/147913860 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:42:14.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.740+0000 7fe4f2473640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fe4ec1567c0 0x7fe4ec0afb40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:41732/0 (socket says 172.21.15.16:41732) 2023-05-22T15:42:14.746 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.740+0000 7fe4f2c74640 1 --2- 172.21.15.16:0/147913860 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe4ec0b0250 0x7fe4ec159880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:14.746 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.740+0000 7fe4f1c72640 1 -- 172.21.15.16:0/147913860 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe4ec0b0250 msgr2=0x7fe4ec159880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:42:14.746 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.740+0000 7fe4f1c72640 1 --2- 172.21.15.16:0/147913860 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe4ec0b0250 0x7fe4ec159880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:14.747 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.740+0000 7fe4f1c72640 1 -- 172.21.15.16:0/147913860 >> v1:172.21.15.16:6789/0 conn(0x7fe4ec1567c0 legacy=0x7fe4ec0afb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:14.747 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.740+0000 7fe4f1c72640 1 -- 172.21.15.16:0/147913860 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4ec0cabc0 con 0x7fe4ec0cb6a0 2023-05-22T15:42:14.747 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.740+0000 7fe4f2c74640 1 --2- 172.21.15.16:0/147913860 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe4ec0b0250 0x7fe4ec159880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:42:14.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.740+0000 7fe4f1c72640 1 --2- 172.21.15.16:0/147913860 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe4ec0cb6a0 0x7fe4ec159340 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7fe4dc009870 tx=0x7fe4dc01c540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f7ab27382ca6a7b server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:14.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.740+0000 7fe4f1471640 1 -- 172.21.15.16:0/147913860 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4dc0ab600 con 0x7fe4ec0cb6a0 2023-05-22T15:42:14.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.740+0000 7fe4f1471640 1 -- 172.21.15.16:0/147913860 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4dc0a7050 con 0x7fe4ec0cb6a0 2023-05-22T15:42:14.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.740+0000 7fe4f1471640 1 -- 172.21.15.16:0/147913860 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4dc0ab980 con 0x7fe4ec0cb6a0 2023-05-22T15:42:14.753 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.740+0000 7fe4f46d7640 1 -- 172.21.15.16:0/147913860 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe4ec0cb6a0 msgr2=0x7fe4ec159340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:14.753 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.740+0000 7fe4f46d7640 1 --2- 172.21.15.16:0/147913860 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe4ec0cb6a0 0x7fe4ec159340 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7fe4dc009870 tx=0x7fe4dc01c540 comp rx=0 tx=0).stop 2023-05-22T15:42:14.753 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.740+0000 7fe4f46d7640 1 -- 172.21.15.16:0/147913860 shutdown_connections 2023-05-22T15:42:14.753 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.740+0000 7fe4f46d7640 1 --2- 172.21.15.16:0/147913860 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe4ec0b0250 0x7fe4ec159880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:14.754 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.740+0000 7fe4f46d7640 1 --2- 172.21.15.16:0/147913860 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe4ec0cb6a0 0x7fe4ec159340 unknown :-1 s=CLOSED pgs=1056 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:14.754 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.740+0000 7fe4f46d7640 1 -- 172.21.15.16:0/147913860 >> 172.21.15.16:0/147913860 conn(0x7fe4ec069720 msgr2=0x7fe4ec069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:14.754 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.740+0000 7fe4f46d7640 1 -- 172.21.15.16:0/147913860 shutdown_connections 2023-05-22T15:42:14.754 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.740+0000 7fe4f46d7640 1 -- 172.21.15.16:0/147913860 wait complete. 2023-05-22T15:42:14.754 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.744+0000 7fe4f46d7640 1 Processor -- start 2023-05-22T15:42:14.755 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.744+0000 7fe4f46d7640 1 -- start start 2023-05-22T15:42:14.755 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.744+0000 7fe4f46d7640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe4ec0b0250 0x7fe4ec14b960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:14.755 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.744+0000 7fe4f46d7640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe4ec0cb6a0 0x7fe4ec0ac1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:14.755 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.744+0000 7fe4f46d7640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe4ec1567c0 0x7fe4ec0ae440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:14.755 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.744+0000 7fe4f46d7640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4ec15a3c0 con 0x7fe4ec0cb6a0 2023-05-22T15:42:14.755 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.744+0000 7fe4f46d7640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4ec0acc30 con 0x7fe4ec1567c0 2023-05-22T15:42:14.755 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.744+0000 7fe4f46d7640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4ec0acf10 con 0x7fe4ec0b0250 2023-05-22T15:42:14.755 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.744+0000 7fe4f1c72640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe4ec0cb6a0 0x7fe4ec0ac1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:14.756 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.744+0000 7fe4f1c72640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe4ec0cb6a0 0x7fe4ec0ac1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41382/0 (socket says 172.21.15.16:41382) 2023-05-22T15:42:14.756 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.744+0000 7fe4f1c72640 1 -- 172.21.15.16:0/232849584 learned_addr learned my addr 172.21.15.16:0/232849584 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:42:14.756 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.744+0000 7fe4f2c74640 1 --2- 172.21.15.16:0/232849584 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe4ec1567c0 0x7fe4ec0ae440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:14.756 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.744+0000 7fe4f1c72640 1 -- 172.21.15.16:0/232849584 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe4ec0b0250 msgr2=0x7fe4ec14b960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:14.756 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.744+0000 7fe4f1c72640 1 --2- 172.21.15.16:0/232849584 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe4ec0b0250 0x7fe4ec14b960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:14.756 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.744+0000 7fe4f1c72640 1 -- 172.21.15.16:0/232849584 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe4ec1567c0 msgr2=0x7fe4ec0ae440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:14.756 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.744+0000 7fe4f1c72640 1 --2- 172.21.15.16:0/232849584 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe4ec1567c0 0x7fe4ec0ae440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:14.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.744+0000 7fe4f1c72640 1 -- 172.21.15.16:0/232849584 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4dc0094d0 con 0x7fe4ec0cb6a0 2023-05-22T15:42:14.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.744+0000 7fe4f2c74640 1 --2- 172.21.15.16:0/232849584 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe4ec1567c0 0x7fe4ec0ae440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:42:14.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.744+0000 7fe4f1c72640 1 --2- 172.21.15.16:0/232849584 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe4ec0cb6a0 0x7fe4ec0ac1e0 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7fe4dc000c10 tx=0x7fe4dc0192e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:14.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.744+0000 7fe4e2ffd640 1 -- 172.21.15.16:0/232849584 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4dc0b6620 con 0x7fe4ec0cb6a0 2023-05-22T15:42:14.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.744+0000 7fe4e2ffd640 1 -- 172.21.15.16:0/232849584 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4dc0b67c0 con 0x7fe4ec0cb6a0 2023-05-22T15:42:14.758 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.744+0000 7fe4e2ffd640 1 -- 172.21.15.16:0/232849584 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4dc0b6b70 con 0x7fe4ec0cb6a0 2023-05-22T15:42:14.758 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.744+0000 7fe4f46d7640 1 -- 172.21.15.16:0/232849584 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4ec0c8e80 con 0x7fe4ec0cb6a0 2023-05-22T15:42:14.758 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.744+0000 7fe4f46d7640 1 -- 172.21.15.16:0/232849584 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4ec159dc0 con 0x7fe4ec0cb6a0 2023-05-22T15:42:14.758 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.744+0000 7fe4f46d7640 1 -- 172.21.15.16:0/232849584 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4ec0c9010 con 0x7fe4ec0cb6a0 2023-05-22T15:42:14.758 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.744+0000 7fe4e2ffd640 1 -- 172.21.15.16:0/232849584 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) v1 ==== 84332+0+0 (secure 0 0 0) 0x7fe4dc0b2040 con 0x7fe4ec0cb6a0 2023-05-22T15:42:14.758 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.748+0000 7fe4e2ffd640 1 --2- 172.21.15.16:0/232849584 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fe4d00654a0 0x7fe4d0067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:14.759 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.748+0000 7fe4e2ffd640 1 -- 172.21.15.16:0/232849584 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6218+0+0 (secure 0 0 0) 0x7fe4dc11c620 con 0x7fe4ec0cb6a0 2023-05-22T15:42:14.759 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.748+0000 7fe4f2473640 1 --2- 172.21.15.16:0/232849584 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fe4d00654a0 0x7fe4d0067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:14.759 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.748+0000 7fe4f2473640 1 --2- 172.21.15.16:0/232849584 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fe4d00654a0 0x7fe4d0067950 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fe4e80012f0 tx=0x7fe4e8007d90 comp rx=0 tx=0).ready entity=mgr.6765 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:14.760 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.752+0000 7fe4e2ffd640 1 -- 172.21.15.16:0/232849584 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fe4dc0ea0f0 con 0x7fe4ec0cb6a0 2023-05-22T15:42:14.966 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:14.964+0000 7fe4f46d7640 1 -- 172.21.15.16:0/232849584 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_multi_snap_schedule"]} v 0) v1 -- 0x7fe4ec04dbe0 con 0x7fe4ec0cb6a0 2023-05-22T15:42:15.609 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:15.604+0000 7fe4e2ffd640 1 -- 172.21.15.16:0/232849584 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_multi_snap_schedule"]}]=0 v563) v1 ==== 154+0+0 (secure 0 0 0) 0x7fe4dc0ed710 con 0x7fe4ec0cb6a0 2023-05-22T15:42:15.612 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:15.608+0000 7fe4f46d7640 1 -- 172.21.15.16:0/232849584 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fe4d00654a0 msgr2=0x7fe4d0067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:15.612 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:15.608+0000 7fe4f46d7640 1 --2- 172.21.15.16:0/232849584 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fe4d00654a0 0x7fe4d0067950 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fe4e80012f0 tx=0x7fe4e8007d90 comp rx=0 tx=0).stop 2023-05-22T15:42:15.613 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:15.608+0000 7fe4f46d7640 1 -- 172.21.15.16:0/232849584 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe4ec0cb6a0 msgr2=0x7fe4ec0ac1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:15.613 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:15.608+0000 7fe4f46d7640 1 --2- 172.21.15.16:0/232849584 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe4ec0cb6a0 0x7fe4ec0ac1e0 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7fe4dc000c10 tx=0x7fe4dc0192e0 comp rx=0 tx=0).stop 2023-05-22T15:42:15.613 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:15.608+0000 7fe4f46d7640 1 -- 172.21.15.16:0/232849584 shutdown_connections 2023-05-22T15:42:15.613 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:15.608+0000 7fe4f46d7640 1 --2- 172.21.15.16:0/232849584 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe4ec1567c0 0x7fe4ec0ae440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:15.613 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:15.608+0000 7fe4f46d7640 1 --2- 172.21.15.16:0/232849584 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe4ec0cb6a0 0x7fe4ec0ac1e0 unknown :-1 s=CLOSED pgs=1057 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:15.614 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:15.608+0000 7fe4f46d7640 1 --2- 172.21.15.16:0/232849584 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fe4d00654a0 0x7fe4d0067950 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:15.614 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:15.608+0000 7fe4f46d7640 1 --2- 172.21.15.16:0/232849584 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe4ec0b0250 0x7fe4ec14b960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:15.614 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:15.608+0000 7fe4f46d7640 1 -- 172.21.15.16:0/232849584 >> 172.21.15.16:0/232849584 conn(0x7fe4ec069720 msgr2=0x7fe4ec1585b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:15.614 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:15.608+0000 7fe4f46d7640 1 -- 172.21.15.16:0/232849584 shutdown_connections 2023-05-22T15:42:15.614 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:15.608+0000 7fe4f46d7640 1 -- 172.21.15.16:0/232849584 wait complete. 2023-05-22T15:42:15.636 INFO:tasks.cephfs_test_runner:Test exisitence of multiple scheduled snapshots ... ok 2023-05-22T15:42:15.637 INFO:tasks.cephfs_test_runner:Starting test: test_non_existent_schedule (tasks.cephfs.test_snap_schedules.TestSnapSchedules) Test listing non-existing snap schedules failure 2023-05-22T15:42:15.637 INFO:tasks.cephfs_test_runner:test_non_existent_schedule (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2023-05-22T15:42:15.638 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_schedule' 2023-05-22T15:42:15.768 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:15.764+0000 7ff9efa61640 1 Processor -- start 2023-05-22T15:42:15.769 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:15.764+0000 7ff9efa61640 1 -- start start 2023-05-22T15:42:15.770 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:15.764+0000 7ff9efa61640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff9e80ca4e0 0x7ff9e815e460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:15.770 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:15.764+0000 7ff9efa61640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff9e8070df0 0x7ff9e8071210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:15.770 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:15.764+0000 7ff9efa61640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9e80b0650 con 0x7ff9e80c9b40 2023-05-22T15:42:15.770 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:15.764+0000 7ff9efa61640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9e80b0370 con 0x7ff9e80ca4e0 2023-05-22T15:42:15.770 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:15.764+0000 7ff9efa61640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9e815d980 con 0x7ff9e8070df0 2023-05-22T15:42:15.770 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:15.764+0000 7ff9ed7fd640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff9e80ca4e0 0x7ff9e815e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:15.771 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:15.764+0000 7ff9ed7fd640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff9e80ca4e0 0x7ff9e815e460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41396/0 (socket says 172.21.15.16:41396) 2023-05-22T15:42:15.771 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:15.764+0000 7ff9ed7fd640 1 -- 172.21.15.16:0/3995603996 learned_addr learned my addr 172.21.15.16:0/3995603996 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:42:15.771 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:15.764+0000 7ff9edffe640 1 --2- 172.21.15.16:0/3995603996 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff9e8070df0 0x7ff9e8071210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:15.771 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:15.764+0000 7ff9ed7fd640 1 -- 172.21.15.16:0/3995603996 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff9e8070df0 msgr2=0x7ff9e8071210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:42:15.771 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:15.764+0000 7ff9ed7fd640 1 --2- 172.21.15.16:0/3995603996 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff9e8070df0 0x7ff9e8071210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:15.771 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:15.764+0000 7ff9ed7fd640 1 -- 172.21.15.16:0/3995603996 >> v1:172.21.15.16:6789/0 conn(0x7ff9e80c9b40 legacy=0x7ff9e80afb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:15.771 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:15.768+0000 7ff9dffff640 1 -- 172.21.15.16:0/3995603996 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 203423661 0 0) 0x7ff9e80b0650 con 0x7ff9e80c9b40 2023-05-22T15:42:15.772 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:15.768+0000 7ff9ed7fd640 1 -- 172.21.15.16:0/3995603996 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9e815dc60 con 0x7ff9e80ca4e0 2023-05-22T15:42:15.772 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:15.768+0000 7ff9ed7fd640 1 --2- 172.21.15.16:0/3995603996 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff9e80ca4e0 0x7ff9e815e460 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7ff9e4009870 tx=0x7ff9e401c510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46ef16cbd9c808ae server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:15.772 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:15.768+0000 7ff9dffff640 1 -- 172.21.15.16:0/3995603996 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9e40ab670 con 0x7ff9e80ca4e0 2023-05-22T15:42:15.772 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:15.768+0000 7ff9dffff640 1 -- 172.21.15.16:0/3995603996 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9e40ab7d0 con 0x7ff9e80ca4e0 2023-05-22T15:42:15.773 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:15.768+0000 7ff9dffff640 1 -- 172.21.15.16:0/3995603996 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9e40abb60 con 0x7ff9e80ca4e0 2023-05-22T15:42:15.773 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:15.768+0000 7ff9efa61640 1 -- 172.21.15.16:0/3995603996 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff9e80ca4e0 msgr2=0x7ff9e815e460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:15.774 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:15.768+0000 7ff9efa61640 1 --2- 172.21.15.16:0/3995603996 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff9e80ca4e0 0x7ff9e815e460 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7ff9e4009870 tx=0x7ff9e401c510 comp rx=0 tx=0).stop 2023-05-22T15:42:15.774 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:15.768+0000 7ff9efa61640 1 -- 172.21.15.16:0/3995603996 shutdown_connections 2023-05-22T15:42:15.774 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:15.768+0000 7ff9efa61640 1 --2- 172.21.15.16:0/3995603996 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff9e8070df0 0x7ff9e8071210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:15.774 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:15.768+0000 7ff9efa61640 1 --2- 172.21.15.16:0/3995603996 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff9e80ca4e0 0x7ff9e815e460 unknown :-1 s=CLOSED pgs=1059 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:15.774 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:15.768+0000 7ff9efa61640 1 -- 172.21.15.16:0/3995603996 >> 172.21.15.16:0/3995603996 conn(0x7ff9e8069720 msgr2=0x7ff9e8069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:15.774 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:15.768+0000 7ff9efa61640 1 -- 172.21.15.16:0/3995603996 shutdown_connections 2023-05-22T15:42:15.774 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:15.768+0000 7ff9efa61640 1 -- 172.21.15.16:0/3995603996 wait complete. 2023-05-22T15:42:15.776 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:15.772+0000 7ff9efa61640 1 Processor -- start 2023-05-22T15:42:15.776 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:15.772+0000 7ff9efa61640 1 -- start start 2023-05-22T15:42:15.777 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:15.772+0000 7ff9efa61640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff9e8070df0 0x7ff9e80c3810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:15.777 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:15.772+0000 7ff9efa61640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff9e80c9b40 0x7ff9e8157030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:15.777 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:15.772+0000 7ff9efa61640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff9e80ca4e0 0x7ff9e80c1790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:15.777 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:15.772+0000 7ff9efa61640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9e8150590 con 0x7ff9e80c9b40 2023-05-22T15:42:15.777 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:15.772+0000 7ff9efa61640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9e8150870 con 0x7ff9e8070df0 2023-05-22T15:42:15.777 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:15.772+0000 7ff9efa61640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9e8150b50 con 0x7ff9e80ca4e0 2023-05-22T15:42:15.777 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:15.772+0000 7ff9ecffc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff9e80c9b40 0x7ff9e8157030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:15.777 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:15.772+0000 7ff9ecffc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff9e80c9b40 0x7ff9e8157030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41404/0 (socket says 172.21.15.16:41404) 2023-05-22T15:42:15.778 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:15.772+0000 7ff9ecffc640 1 -- 172.21.15.16:0/2492219702 learned_addr learned my addr 172.21.15.16:0/2492219702 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:42:15.778 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:15.772+0000 7ff9edffe640 1 --2- 172.21.15.16:0/2492219702 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff9e80ca4e0 0x7ff9e80c1790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:15.778 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:15.772+0000 7ff9ed7fd640 1 --2- 172.21.15.16:0/2492219702 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff9e8070df0 0x7ff9e80c3810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:15.778 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:15.772+0000 7ff9ecffc640 1 -- 172.21.15.16:0/2492219702 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff9e80ca4e0 msgr2=0x7ff9e80c1790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:15.779 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:15.772+0000 7ff9ecffc640 1 --2- 172.21.15.16:0/2492219702 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff9e80ca4e0 0x7ff9e80c1790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:15.779 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:15.772+0000 7ff9ecffc640 1 -- 172.21.15.16:0/2492219702 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff9e8070df0 msgr2=0x7ff9e80c3810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:15.779 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:15.772+0000 7ff9ecffc640 1 --2- 172.21.15.16:0/2492219702 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff9e8070df0 0x7ff9e80c3810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:15.779 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:15.772+0000 7ff9ecffc640 1 -- 172.21.15.16:0/2492219702 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9e40094d0 con 0x7ff9e80c9b40 2023-05-22T15:42:15.779 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:15.772+0000 7ff9ecffc640 1 --2- 172.21.15.16:0/2492219702 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff9e80c9b40 0x7ff9e8157030 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7ff9d800c480 tx=0x7ff9d800c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:15.779 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:15.772+0000 7ff9ddffb640 1 -- 172.21.15.16:0/2492219702 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9d80052d0 con 0x7ff9e80c9b40 2023-05-22T15:42:15.779 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:15.772+0000 7ff9ddffb640 1 -- 172.21.15.16:0/2492219702 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9d8005430 con 0x7ff9e80c9b40 2023-05-22T15:42:15.780 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:15.772+0000 7ff9ddffb640 1 -- 172.21.15.16:0/2492219702 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9d8012870 con 0x7ff9e80c9b40 2023-05-22T15:42:15.780 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:15.772+0000 7ff9efa61640 1 -- 172.21.15.16:0/2492219702 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9e80c8e80 con 0x7ff9e80c9b40 2023-05-22T15:42:15.780 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:15.772+0000 7ff9efa61640 1 -- 172.21.15.16:0/2492219702 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9e80ca8b0 con 0x7ff9e80c9b40 2023-05-22T15:42:15.781 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:15.772+0000 7ff9efa61640 1 -- 172.21.15.16:0/2492219702 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9e80c9010 con 0x7ff9e80c9b40 2023-05-22T15:42:15.781 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:15.772+0000 7ff9ddffb640 1 -- 172.21.15.16:0/2492219702 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) v1 ==== 84332+0+0 (secure 0 0 0) 0x7ff9d80129d0 con 0x7ff9e80c9b40 2023-05-22T15:42:15.781 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:15.776+0000 7ff9ddffb640 1 --2- 172.21.15.16:0/2492219702 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ff9cc0654a0 0x7ff9cc067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:15.781 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:15.776+0000 7ff9ddffb640 1 -- 172.21.15.16:0/2492219702 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6218+0+0 (secure 0 0 0) 0x7ff9d8082440 con 0x7ff9e80c9b40 2023-05-22T15:42:15.781 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:15.776+0000 7ff9ed7fd640 1 --2- 172.21.15.16:0/2492219702 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ff9cc0654a0 0x7ff9cc067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:15.781 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:15.776+0000 7ff9ed7fd640 1 --2- 172.21.15.16:0/2492219702 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ff9cc0654a0 0x7ff9cc067950 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7ff9e40af000 tx=0x7ff9e40a7000 comp rx=0 tx=0).ready entity=mgr.6765 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:15.782 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:15.780+0000 7ff9ddffb640 1 -- 172.21.15.16:0/2492219702 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7ff9d8050050 con 0x7ff9e80c9b40 2023-05-22T15:42:15.992 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:15.988+0000 7ff9efa61640 1 -- 172.21.15.16:0/2492219702 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_schedule"]} v 0) v1 -- 0x7ff9e804dbe0 con 0x7ff9e80c9b40 2023-05-22T15:42:16.612 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:16.608+0000 7ff9ddffb640 1 -- 172.21.15.16:0/2492219702 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_schedule"]}]=0 v564) v1 ==== 159+0+0 (secure 0 0 0) 0x7ff9d8053670 con 0x7ff9e80c9b40 2023-05-22T15:42:16.615 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:16.612+0000 7ff9efa61640 1 -- 172.21.15.16:0/2492219702 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ff9cc0654a0 msgr2=0x7ff9cc067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:16.615 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:16.612+0000 7ff9efa61640 1 --2- 172.21.15.16:0/2492219702 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ff9cc0654a0 0x7ff9cc067950 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7ff9e40af000 tx=0x7ff9e40a7000 comp rx=0 tx=0).stop 2023-05-22T15:42:16.615 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:16.612+0000 7ff9efa61640 1 -- 172.21.15.16:0/2492219702 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff9e80c9b40 msgr2=0x7ff9e8157030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:16.616 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:16.612+0000 7ff9efa61640 1 --2- 172.21.15.16:0/2492219702 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff9e80c9b40 0x7ff9e8157030 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7ff9d800c480 tx=0x7ff9d800c940 comp rx=0 tx=0).stop 2023-05-22T15:42:16.616 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:16.612+0000 7ff9efa61640 1 -- 172.21.15.16:0/2492219702 shutdown_connections 2023-05-22T15:42:16.616 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:16.612+0000 7ff9efa61640 1 --2- 172.21.15.16:0/2492219702 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ff9cc0654a0 0x7ff9cc067950 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:16.616 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:16.612+0000 7ff9efa61640 1 --2- 172.21.15.16:0/2492219702 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff9e80ca4e0 0x7ff9e80c1790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:16.616 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:16.612+0000 7ff9efa61640 1 --2- 172.21.15.16:0/2492219702 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff9e80c9b40 0x7ff9e8157030 unknown :-1 s=CLOSED pgs=1060 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:16.616 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:16.612+0000 7ff9efa61640 1 --2- 172.21.15.16:0/2492219702 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff9e8070df0 0x7ff9e80c3810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:16.616 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:16.612+0000 7ff9efa61640 1 -- 172.21.15.16:0/2492219702 >> 172.21.15.16:0/2492219702 conn(0x7ff9e8069720 msgr2=0x7ff9e80b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:16.617 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:16.612+0000 7ff9efa61640 1 -- 172.21.15.16:0/2492219702 shutdown_connections 2023-05-22T15:42:16.617 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:16.612+0000 7ff9efa61640 1 -- 172.21.15.16:0/2492219702 wait complete. 2023-05-22T15:42:16.635 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-05-22T15:42:16.767 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:16.764+0000 7f23af1ef640 1 Processor -- start 2023-05-22T15:42:16.767 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:16.764+0000 7f23af1ef640 1 -- start start 2023-05-22T15:42:16.768 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:16.764+0000 7f23af1ef640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f23a80cee30 0x7f23a806bf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:16.768 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:16.764+0000 7f23af1ef640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f23a806db80 0x7f23a806dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:16.768 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:16.764+0000 7f23af1ef640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23a806e520 con 0x7f23a815c8b0 2023-05-22T15:42:16.768 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:16.764+0000 7f23af1ef640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23a815a0c0 con 0x7f23a80cee30 2023-05-22T15:42:16.768 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:16.764+0000 7f23af1ef640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23a80ce160 con 0x7f23a806db80 2023-05-22T15:42:16.769 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:16.764+0000 7f23a7fff640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f23a80cee30 0x7f23a806bf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:16.770 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:16.764+0000 7f23acf8b640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f23a815c8b0 0x7f23a815cc80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:41760/0 (socket says 172.21.15.16:41760) 2023-05-22T15:42:16.770 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:16.764+0000 7f23acf8b640 1 -- 172.21.15.16:0/864969037 learned_addr learned my addr 172.21.15.16:0/864969037 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:42:16.770 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:16.764+0000 7f23ad78c640 1 --2- 172.21.15.16:0/864969037 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f23a806db80 0x7f23a806dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:16.770 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:16.764+0000 7f23a7fff640 1 -- 172.21.15.16:0/864969037 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f23a806db80 msgr2=0x7f23a806dfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:42:16.770 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:16.764+0000 7f23a7fff640 1 --2- 172.21.15.16:0/864969037 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f23a806db80 0x7f23a806dfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:16.770 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:16.764+0000 7f23a7fff640 1 -- 172.21.15.16:0/864969037 >> v1:172.21.15.16:6789/0 conn(0x7f23a815c8b0 legacy=0x7f23a815cc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:16.770 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:16.764+0000 7f23a7fff640 1 -- 172.21.15.16:0/864969037 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23a806c640 con 0x7f23a80cee30 2023-05-22T15:42:16.771 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:16.764+0000 7f23a77fe640 1 -- 172.21.15.16:0/864969037 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4028829128 0 0) 0x7f23a806e520 con 0x7f23a815c8b0 2023-05-22T15:42:16.772 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:16.768+0000 7f23a7fff640 1 --2- 172.21.15.16:0/864969037 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f23a80cee30 0x7f23a806bf60 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7f2390009d90 tx=0x7f239001c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c31898e07b5a6b01 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:16.772 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:16.768+0000 7f23a77fe640 1 -- 172.21.15.16:0/864969037 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23900a59e0 con 0x7f23a80cee30 2023-05-22T15:42:16.772 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:16.768+0000 7f23a77fe640 1 -- 172.21.15.16:0/864969037 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23900a5b40 con 0x7f23a80cee30 2023-05-22T15:42:16.772 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:16.768+0000 7f23a77fe640 1 -- 172.21.15.16:0/864969037 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23900ad440 con 0x7f23a80cee30 2023-05-22T15:42:16.772 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:16.768+0000 7f23af1ef640 1 -- 172.21.15.16:0/864969037 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f23a80cee30 msgr2=0x7f23a806bf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:16.773 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:16.768+0000 7f23af1ef640 1 --2- 172.21.15.16:0/864969037 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f23a80cee30 0x7f23a806bf60 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7f2390009d90 tx=0x7f239001c920 comp rx=0 tx=0).stop 2023-05-22T15:42:16.773 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:16.768+0000 7f23af1ef640 1 -- 172.21.15.16:0/864969037 shutdown_connections 2023-05-22T15:42:16.773 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:16.768+0000 7f23af1ef640 1 --2- 172.21.15.16:0/864969037 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f23a806db80 0x7f23a806dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:16.773 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:16.768+0000 7f23af1ef640 1 --2- 172.21.15.16:0/864969037 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f23a80cee30 0x7f23a806bf60 unknown :-1 s=CLOSED pgs=1062 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:16.773 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:16.768+0000 7f23af1ef640 1 -- 172.21.15.16:0/864969037 >> 172.21.15.16:0/864969037 conn(0x7f23a806f5a0 msgr2=0x7f23a806f9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:16.773 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:16.768+0000 7f23af1ef640 1 -- 172.21.15.16:0/864969037 shutdown_connections 2023-05-22T15:42:16.773 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:16.768+0000 7f23af1ef640 1 -- 172.21.15.16:0/864969037 wait complete. 2023-05-22T15:42:16.775 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:16.768+0000 7f23af1ef640 1 Processor -- start 2023-05-22T15:42:16.775 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:16.768+0000 7f23af1ef640 1 -- start start 2023-05-22T15:42:16.775 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:16.768+0000 7f23af1ef640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f23a806db80 0x7f23a8152330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:16.776 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:16.768+0000 7f23af1ef640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f23a80cee30 0x7f23a815ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:16.776 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:16.768+0000 7f23af1ef640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f23a815c8b0 0x7f23a8149680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:16.776 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:16.768+0000 7f23af1ef640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23a81578c0 con 0x7f23a80cee30 2023-05-22T15:42:16.776 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:16.768+0000 7f23af1ef640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23a815b150 con 0x7f23a815c8b0 2023-05-22T15:42:16.776 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:16.768+0000 7f23af1ef640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23a815b3f0 con 0x7f23a806db80 2023-05-22T15:42:16.776 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:16.772+0000 7f23a7fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f23a80cee30 0x7f23a815ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:16.776 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:16.772+0000 7f23a7fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f23a80cee30 0x7f23a815ac10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41418/0 (socket says 172.21.15.16:41418) 2023-05-22T15:42:16.777 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:16.772+0000 7f23a7fff640 1 -- 172.21.15.16:0/290999709 learned_addr learned my addr 172.21.15.16:0/290999709 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:42:16.777 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:16.772+0000 7f23acf8b640 1 --2- 172.21.15.16:0/290999709 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f23a806db80 0x7f23a8152330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:16.777 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:16.772+0000 7f23ad78c640 1 --2- 172.21.15.16:0/290999709 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f23a815c8b0 0x7f23a8149680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:16.777 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:16.772+0000 7f23a7fff640 1 -- 172.21.15.16:0/290999709 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f23a806db80 msgr2=0x7f23a8152330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:16.777 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:16.772+0000 7f23a7fff640 1 --2- 172.21.15.16:0/290999709 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f23a806db80 0x7f23a8152330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:16.777 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:16.772+0000 7f23a7fff640 1 -- 172.21.15.16:0/290999709 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f23a815c8b0 msgr2=0x7f23a8149680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:16.777 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:16.772+0000 7f23a7fff640 1 --2- 172.21.15.16:0/290999709 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f23a815c8b0 0x7f23a8149680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:16.778 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:16.772+0000 7f23a7fff640 1 -- 172.21.15.16:0/290999709 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2390009a70 con 0x7f23a80cee30 2023-05-22T15:42:16.778 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:16.772+0000 7f23a7fff640 1 --2- 172.21.15.16:0/290999709 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f23a80cee30 0x7f23a815ac10 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7f239001afd0 tx=0x7f23900196c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:16.778 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:16.772+0000 7f23a57fa640 1 -- 172.21.15.16:0/290999709 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23900a59e0 con 0x7f23a80cee30 2023-05-22T15:42:16.778 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:16.772+0000 7f23a57fa640 1 -- 172.21.15.16:0/290999709 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2390009450 con 0x7f23a80cee30 2023-05-22T15:42:16.778 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:16.772+0000 7f23af1ef640 1 -- 172.21.15.16:0/290999709 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23a815d390 con 0x7f23a80cee30 2023-05-22T15:42:16.778 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:16.772+0000 7f23a57fa640 1 -- 172.21.15.16:0/290999709 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23900b9670 con 0x7f23a80cee30 2023-05-22T15:42:16.778 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:16.772+0000 7f23af1ef640 1 -- 172.21.15.16:0/290999709 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23a80ce930 con 0x7f23a80cee30 2023-05-22T15:42:16.779 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:16.772+0000 7f23af1ef640 1 -- 172.21.15.16:0/290999709 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23a815d520 con 0x7f23a80cee30 2023-05-22T15:42:16.779 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:16.772+0000 7f23a57fa640 1 -- 172.21.15.16:0/290999709 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f23900b97d0 con 0x7f23a80cee30 2023-05-22T15:42:16.779 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:16.772+0000 7f23a57fa640 1 --2- 172.21.15.16:0/290999709 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f23880654a0 0x7f2388067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:16.780 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:16.772+0000 7f23a57fa640 1 -- 172.21.15.16:0/290999709 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6218+0+0 (secure 0 0 0) 0x7f239011e680 con 0x7f23a80cee30 2023-05-22T15:42:16.780 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:16.772+0000 7f23acf8b640 1 --2- 172.21.15.16:0/290999709 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f23880654a0 0x7f2388067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:16.780 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:16.772+0000 7f23acf8b640 1 --2- 172.21.15.16:0/290999709 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f23880654a0 0x7f2388067950 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f2398000f70 tx=0x7f2398007c10 comp rx=0 tx=0).ready entity=mgr.6765 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:16.780 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:16.776+0000 7f23a57fa640 1 -- 172.21.15.16:0/290999709 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f23900ec2c0 con 0x7f23a80cee30 2023-05-22T15:42:16.934 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:16.932+0000 7f23af1ef640 1 -- 172.21.15.16:0/290999709 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f23a8157a00 con 0x7f23a80cee30 2023-05-22T15:42:16.940 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:16.936+0000 7f23a57fa640 1 -- 172.21.15.16:0/290999709 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v22)=0 v22) v1 ==== 122+0+0 (secure 0 0 0) 0x7f23a8157a00 con 0x7f23a80cee30 2023-05-22T15:42:16.942 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:16.936+0000 7f23af1ef640 1 -- 172.21.15.16:0/290999709 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f23880654a0 msgr2=0x7f2388067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:16.942 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:16.936+0000 7f23af1ef640 1 --2- 172.21.15.16:0/290999709 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f23880654a0 0x7f2388067950 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f2398000f70 tx=0x7f2398007c10 comp rx=0 tx=0).stop 2023-05-22T15:42:16.942 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:16.940+0000 7f23af1ef640 1 -- 172.21.15.16:0/290999709 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f23a80cee30 msgr2=0x7f23a815ac10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:16.943 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:16.940+0000 7f23af1ef640 1 --2- 172.21.15.16:0/290999709 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f23a80cee30 0x7f23a815ac10 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7f239001afd0 tx=0x7f23900196c0 comp rx=0 tx=0).stop 2023-05-22T15:42:16.943 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:16.940+0000 7f23af1ef640 1 -- 172.21.15.16:0/290999709 shutdown_connections 2023-05-22T15:42:16.943 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:16.940+0000 7f23af1ef640 1 --2- 172.21.15.16:0/290999709 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f23a815c8b0 0x7f23a8149680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:16.943 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:16.940+0000 7f23af1ef640 1 --2- 172.21.15.16:0/290999709 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f23a80cee30 0x7f23a815ac10 unknown :-1 s=CLOSED pgs=1063 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:16.943 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:16.940+0000 7f23af1ef640 1 --2- 172.21.15.16:0/290999709 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f23880654a0 0x7f2388067950 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:16.943 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:16.940+0000 7f23af1ef640 1 --2- 172.21.15.16:0/290999709 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f23a806db80 0x7f23a8152330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:16.943 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:16.940+0000 7f23af1ef640 1 -- 172.21.15.16:0/290999709 >> 172.21.15.16:0/290999709 conn(0x7f23a806f5a0 msgr2=0x7f23a80afca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:16.943 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:16.940+0000 7f23af1ef640 1 -- 172.21.15.16:0/290999709 shutdown_connections 2023-05-22T15:42:16.944 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:16.940+0000 7f23af1ef640 1 -- 172.21.15.16:0/290999709 wait complete. 2023-05-22T15:42:16.959 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-05-22T15:42:16.959 DEBUG:teuthology.orchestra.run.smithi016:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-05-22T15:42:17.011 DEBUG:teuthology.orchestra.run.smithi062:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-05-22T15:42:17.062 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-05-22T15:42:17.063 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-05-22T15:42:17.063 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2023-05-22T15:42:17.068 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16345192k,nr_inodes=4086298,mode=755,inode64 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3276544k,mode=755,inode64 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev,inode64 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=15732 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3276540k,nr_inodes=819135,mode=700,uid=1000,gid=1271,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3276544k,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-05-22T15:42:17.070 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2023-05-22T15:42:17.070 DEBUG:teuthology.orchestra.run.smithi062:> dd if=/proc/self/mounts of=/dev/stdout 2023-05-22T15:42:17.076 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16342116k,nr_inodes=4085529,mode=755,inode64 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275932k,mode=755,inode64 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev,inode64 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=13250 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275928k,nr_inodes=818982,mode=700,uid=1000,gid=1271,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275932k,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-05-22T15:42:17.077 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-05-22T15:42:17.077 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2023-05-22T15:42:17.115 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16345192k,nr_inodes=4086298,mode=755,inode64 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3276544k,mode=755,inode64 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev,inode64 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=15732 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3276540k,nr_inodes=819135,mode=700,uid=1000,gid=1271,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3276544k,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-05-22T15:42:17.116 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2023-05-22T15:42:17.116 DEBUG:teuthology.orchestra.run.smithi062:> dd if=/proc/self/mounts of=/dev/stdout 2023-05-22T15:42:17.130 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16342116k,nr_inodes=4085529,mode=755,inode64 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275932k,mode=755,inode64 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev,inode64 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=13250 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275928k,nr_inodes=818982,mode=700,uid=1000,gid=1271,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275932k,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-05-22T15:42:17.130 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2023-05-22T15:42:17.131 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-05-22T15:42:17.293 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:17.288+0000 7f9c3236c640 1 Processor -- start 2023-05-22T15:42:17.293 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:17.288+0000 7f9c3236c640 1 -- start start 2023-05-22T15:42:17.294 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:17.288+0000 7f9c3236c640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f9c2c0cb6a0 0x7f9c2c159340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:17.295 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:17.288+0000 7f9c3236c640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f9c2c0b0250 0x7f9c2c159880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:17.295 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:17.288+0000 7f9c3236c640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c2c156b90 con 0x7f9c2c1567c0 2023-05-22T15:42:17.295 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:17.288+0000 7f9c3236c640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c2c15a0b0 con 0x7f9c2c0b0250 2023-05-22T15:42:17.295 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:17.288+0000 7f9c3236c640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c2c0ca8e0 con 0x7f9c2c0cb6a0 2023-05-22T15:42:17.295 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:17.292+0000 7f9c30909640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f9c2c0b0250 0x7f9c2c159880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:17.295 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:17.292+0000 7f9c30909640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f9c2c0b0250 0x7f9c2c159880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41430/0 (socket says 172.21.15.16:41430) 2023-05-22T15:42:17.296 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:17.292+0000 7f9c30909640 1 -- 172.21.15.16:0/2216379291 learned_addr learned my addr 172.21.15.16:0/2216379291 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:42:17.296 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:17.292+0000 7f9c237fe640 1 --2- 172.21.15.16:0/2216379291 >> v2:172.21.15.62:3300/0 conn(0x7f9c2c0cb6a0 0x7f9c2c159340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:17.296 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:17.292+0000 7f9c30909640 1 -- 172.21.15.16:0/2216379291 >> v2:172.21.15.62:3300/0 conn(0x7f9c2c0cb6a0 msgr2=0x7f9c2c159340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:42:17.296 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:17.292+0000 7f9c30909640 1 --2- 172.21.15.16:0/2216379291 >> v2:172.21.15.62:3300/0 conn(0x7f9c2c0cb6a0 0x7f9c2c159340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:17.296 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:17.292+0000 7f9c30909640 1 -- 172.21.15.16:0/2216379291 >> v1:172.21.15.62:6789/0 conn(0x7f9c2c1567c0 legacy=0x7f9c2c0afb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:42:17.297 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:17.292+0000 7f9c30909640 1 -- 172.21.15.16:0/2216379291 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c2c0cabc0 con 0x7f9c2c0b0250 2023-05-22T15:42:17.297 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:17.292+0000 7f9c30909640 1 --2- 172.21.15.16:0/2216379291 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9c2c0b0250 0x7f9c2c159880 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7f9c2c15d780 tx=0x7f9c2401eb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=604c65ee6e3febd1 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:17.298 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:17.292+0000 7f9c2b7fe640 1 -- 172.21.15.16:0/2216379291 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c2400b2d0 con 0x7f9c2c0b0250 2023-05-22T15:42:17.298 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:17.292+0000 7f9c2b7fe640 1 -- 172.21.15.16:0/2216379291 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c2400b430 con 0x7f9c2c0b0250 2023-05-22T15:42:17.298 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:17.292+0000 7f9c2b7fe640 1 -- 172.21.15.16:0/2216379291 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c240a7a60 con 0x7f9c2c0b0250 2023-05-22T15:42:17.298 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:17.292+0000 7f9c3236c640 1 -- 172.21.15.16:0/2216379291 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9c2c0b0250 msgr2=0x7f9c2c159880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:17.298 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:17.292+0000 7f9c3236c640 1 --2- 172.21.15.16:0/2216379291 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9c2c0b0250 0x7f9c2c159880 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7f9c2c15d780 tx=0x7f9c2401eb90 comp rx=0 tx=0).stop 2023-05-22T15:42:17.298 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:17.292+0000 7f9c3236c640 1 -- 172.21.15.16:0/2216379291 shutdown_connections 2023-05-22T15:42:17.298 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:17.292+0000 7f9c3236c640 1 --2- 172.21.15.16:0/2216379291 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9c2c0b0250 0x7f9c2c159880 unknown :-1 s=CLOSED pgs=1064 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:17.299 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:17.292+0000 7f9c3236c640 1 --2- 172.21.15.16:0/2216379291 >> v2:172.21.15.62:3300/0 conn(0x7f9c2c0cb6a0 0x7f9c2c159340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:17.299 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:17.292+0000 7f9c3236c640 1 -- 172.21.15.16:0/2216379291 >> 172.21.15.16:0/2216379291 conn(0x7f9c2c069720 msgr2=0x7f9c2c069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:17.299 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:17.292+0000 7f9c3236c640 1 -- 172.21.15.16:0/2216379291 shutdown_connections 2023-05-22T15:42:17.299 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:17.292+0000 7f9c3236c640 1 -- 172.21.15.16:0/2216379291 wait complete. 2023-05-22T15:42:17.301 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:17.296+0000 7f9c3236c640 1 Processor -- start 2023-05-22T15:42:17.301 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:17.296+0000 7f9c3236c640 1 -- start start 2023-05-22T15:42:17.302 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:17.296+0000 7f9c3236c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9c2c0b0250 0x7f9c2c14b350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:17.302 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:17.296+0000 7f9c3236c640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9c2c0cb6a0 0x7f9c2c1553b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:17.302 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:17.296+0000 7f9c3236c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9c2c1567c0 0x7f9c2c1558f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:17.302 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:17.296+0000 7f9c3236c640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c2c0c7f70 con 0x7f9c2c1567c0 2023-05-22T15:42:17.302 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:17.296+0000 7f9c3236c640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c2c0ade40 con 0x7f9c2c0b0250 2023-05-22T15:42:17.302 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:17.296+0000 7f9c3236c640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c2c0ae0e0 con 0x7f9c2c0cb6a0 2023-05-22T15:42:17.302 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:17.296+0000 7f9c30909640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9c2c1567c0 0x7f9c2c1558f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:17.303 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:17.296+0000 7f9c30909640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9c2c1567c0 0x7f9c2c1558f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41438/0 (socket says 172.21.15.16:41438) 2023-05-22T15:42:17.303 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:17.296+0000 7f9c30909640 1 -- 172.21.15.16:0/4184102335 learned_addr learned my addr 172.21.15.16:0/4184102335 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:42:17.303 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:17.296+0000 7f9c2bfff640 1 --2- 172.21.15.16:0/4184102335 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9c2c0b0250 0x7f9c2c14b350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:17.303 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:17.296+0000 7f9c237fe640 1 --2- 172.21.15.16:0/4184102335 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9c2c0cb6a0 0x7f9c2c1553b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:17.303 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:17.296+0000 7f9c30909640 1 -- 172.21.15.16:0/4184102335 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9c2c0cb6a0 msgr2=0x7f9c2c1553b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:17.303 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:17.296+0000 7f9c30909640 1 --2- 172.21.15.16:0/4184102335 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9c2c0cb6a0 0x7f9c2c1553b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:17.303 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:17.296+0000 7f9c30909640 1 -- 172.21.15.16:0/4184102335 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9c2c0b0250 msgr2=0x7f9c2c14b350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:17.304 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:17.296+0000 7f9c30909640 1 --2- 172.21.15.16:0/4184102335 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9c2c0b0250 0x7f9c2c14b350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:17.304 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:17.296+0000 7f9c30909640 1 -- 172.21.15.16:0/4184102335 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c2400bd40 con 0x7f9c2c1567c0 2023-05-22T15:42:17.304 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:17.296+0000 7f9c30909640 1 --2- 172.21.15.16:0/4184102335 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9c2c1567c0 0x7f9c2c1558f0 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 crypto rx=0x7f9c240b1040 tx=0x7f9c2401b930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:17.304 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:17.296+0000 7f9c297fa640 1 -- 172.21.15.16:0/4184102335 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c24009800 con 0x7f9c2c1567c0 2023-05-22T15:42:17.304 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:17.296+0000 7f9c297fa640 1 -- 172.21.15.16:0/4184102335 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c240099a0 con 0x7f9c2c1567c0 2023-05-22T15:42:17.305 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:17.296+0000 7f9c3236c640 1 -- 172.21.15.16:0/4184102335 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c2c0c8e80 con 0x7f9c2c1567c0 2023-05-22T15:42:17.305 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:17.296+0000 7f9c297fa640 1 -- 172.21.15.16:0/4184102335 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c24009d50 con 0x7f9c2c1567c0 2023-05-22T15:42:17.305 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:17.296+0000 7f9c3236c640 1 -- 172.21.15.16:0/4184102335 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c2c159dc0 con 0x7f9c2c1567c0 2023-05-22T15:42:17.305 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:17.296+0000 7f9c3236c640 1 -- 172.21.15.16:0/4184102335 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c2c0c9010 con 0x7f9c2c1567c0 2023-05-22T15:42:17.306 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:17.300+0000 7f9c297fa640 1 -- 172.21.15.16:0/4184102335 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f9c240b9430 con 0x7f9c2c1567c0 2023-05-22T15:42:17.306 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:17.300+0000 7f9c297fa640 1 --2- 172.21.15.16:0/4184102335 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f9c0c0654d0 0x7f9c0c067980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:17.306 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:17.300+0000 7f9c297fa640 1 -- 172.21.15.16:0/4184102335 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6218+0+0 (secure 0 0 0) 0x7f9c2411e210 con 0x7f9c2c1567c0 2023-05-22T15:42:17.306 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:17.300+0000 7f9c2bfff640 1 --2- 172.21.15.16:0/4184102335 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f9c0c0654d0 0x7f9c0c067980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:17.306 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:17.300+0000 7f9c2bfff640 1 --2- 172.21.15.16:0/4184102335 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f9c0c0654d0 0x7f9c0c067980 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f9c1c00b810 tx=0x7f9c1c008040 comp rx=0 tx=0).ready entity=mgr.6765 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:17.307 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:17.304+0000 7f9c297fa640 1 -- 172.21.15.16:0/4184102335 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f9c240ebeb0 con 0x7f9c2c1567c0 2023-05-22T15:42:17.495 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:17.492+0000 7f9c3236c640 1 -- 172.21.15.16:0/4184102335 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9c2c072160 con 0x7f9c2c1567c0 2023-05-22T15:42:17.495 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:17.492+0000 7f9c297fa640 1 -- 172.21.15.16:0/4184102335 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+3526 (secure 0 0 0) 0x7f9c240ef4d0 con 0x7f9c2c1567c0 2023-05-22T15:42:17.496 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:42:17.496 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":24,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5399,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6833/1859135164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":1859135164},{"type":"v1","addr":"172.21.15.62:6833","nonce":1859135164}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":17},{"gid":5836,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/3067106582","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":3067106582},{"type":"v1","addr":"172.21.15.62:6835","nonce":3067106582}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":17},{"gid":6462,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/717991089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":717991089},{"type":"v1","addr":"172.21.15.16:6835","nonce":717991089}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":17},{"gid":6921,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/3719730139","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":3719730139},{"type":"v1","addr":"172.21.15.16:6837","nonce":3719730139}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":24}],"filesystems":[]} 2023-05-22T15:42:17.496 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 24 2023-05-22T15:42:17.498 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:17.492+0000 7f9c3236c640 1 -- 172.21.15.16:0/4184102335 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f9c0c0654d0 msgr2=0x7f9c0c067980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:17.498 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:17.492+0000 7f9c3236c640 1 --2- 172.21.15.16:0/4184102335 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f9c0c0654d0 0x7f9c0c067980 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f9c1c00b810 tx=0x7f9c1c008040 comp rx=0 tx=0).stop 2023-05-22T15:42:17.498 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:17.492+0000 7f9c3236c640 1 -- 172.21.15.16:0/4184102335 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9c2c1567c0 msgr2=0x7f9c2c1558f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:17.498 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:17.492+0000 7f9c3236c640 1 --2- 172.21.15.16:0/4184102335 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9c2c1567c0 0x7f9c2c1558f0 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 crypto rx=0x7f9c240b1040 tx=0x7f9c2401b930 comp rx=0 tx=0).stop 2023-05-22T15:42:17.498 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:17.492+0000 7f9c3236c640 1 -- 172.21.15.16:0/4184102335 shutdown_connections 2023-05-22T15:42:17.499 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:17.492+0000 7f9c3236c640 1 --2- 172.21.15.16:0/4184102335 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9c2c1567c0 0x7f9c2c1558f0 unknown :-1 s=CLOSED pgs=1065 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:17.499 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:17.492+0000 7f9c3236c640 1 --2- 172.21.15.16:0/4184102335 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f9c0c0654d0 0x7f9c0c067980 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:17.499 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:17.492+0000 7f9c3236c640 1 --2- 172.21.15.16:0/4184102335 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9c2c0cb6a0 0x7f9c2c1553b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:17.499 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:17.492+0000 7f9c3236c640 1 --2- 172.21.15.16:0/4184102335 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9c2c0b0250 0x7f9c2c14b350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:17.499 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:17.492+0000 7f9c3236c640 1 -- 172.21.15.16:0/4184102335 >> 172.21.15.16:0/4184102335 conn(0x7f9c2c069720 msgr2=0x7f9c2c0ce9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:17.499 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:17.496+0000 7f9c3236c640 1 -- 172.21.15.16:0/4184102335 shutdown_connections 2023-05-22T15:42:17.499 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:17.496+0000 7f9c3236c640 1 -- 172.21.15.16:0/4184102335 wait complete. 2023-05-22T15:42:17.515 INFO:tasks.ceph.mds.a:Restarting daemon 2023-05-22T15:42:17.516 INFO:tasks.ceph.mds.a:Stopping old one... 2023-05-22T15:42:17.516 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-05-22T15:42:17.516 INFO:teuthology.orchestra.run:waiting for 300 2023-05-22T15:42:17.517 INFO:tasks.ceph.mds.c:Restarting daemon 2023-05-22T15:42:17.517 INFO:tasks.ceph.mds.c:Stopping old one... 2023-05-22T15:42:17.517 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-05-22T15:42:17.518 INFO:teuthology.orchestra.run:waiting for 300 2023-05-22T15:42:17.518 INFO:tasks.ceph.mds.b:Restarting daemon 2023-05-22T15:42:17.518 INFO:tasks.ceph.mds.b:Stopping old one... 2023-05-22T15:42:17.519 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-05-22T15:42:17.519 INFO:teuthology.orchestra.run:waiting for 300 2023-05-22T15:42:17.520 INFO:tasks.ceph.mds.d:Restarting daemon 2023-05-22T15:42:17.520 INFO:tasks.ceph.mds.d:Stopping old one... 2023-05-22T15:42:17.520 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-05-22T15:42:17.521 INFO:teuthology.orchestra.run:waiting for 300 2023-05-22T15:42:17.528 INFO:tasks.ceph.mds.a:Stopped 2023-05-22T15:42:17.528 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-05-22T15:42:17.529 INFO:tasks.ceph.mds.c:Stopped 2023-05-22T15:42:17.529 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-05-22T15:42:17.540 INFO:tasks.ceph.mds.b:Stopped 2023-05-22T15:42:17.540 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-05-22T15:42:17.541 INFO:tasks.ceph.mds.d:Stopped 2023-05-22T15:42:17.541 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-05-22T15:42:17.571 INFO:tasks.ceph.mds.a:Started 2023-05-22T15:42:17.571 DEBUG:teuthology.parallel:result is None 2023-05-22T15:42:17.572 INFO:tasks.ceph.mds.c:Started 2023-05-22T15:42:17.572 DEBUG:teuthology.parallel:result is None 2023-05-22T15:42:17.582 INFO:tasks.ceph.mds.b:Started 2023-05-22T15:42:17.583 DEBUG:teuthology.parallel:result is None 2023-05-22T15:42:17.583 INFO:tasks.ceph.mds.d:Started 2023-05-22T15:42:17.583 DEBUG:teuthology.parallel:result is None 2023-05-22T15:42:17.584 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2023-05-22T15:42:17.635 INFO:tasks.ceph.mds.a.smithi016.stdout:starting mds.a at 2023-05-22T15:42:17.636 INFO:tasks.ceph.mds.c.smithi016.stdout:starting mds.c at 2023-05-22T15:42:17.657 INFO:tasks.ceph.mds.b.smithi062.stdout:starting mds.b at 2023-05-22T15:42:17.660 INFO:tasks.ceph.mds.d.smithi062.stdout:starting mds.d at 2023-05-22T15:42:17.708 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:17.704+0000 7f0ee6fa1640 1 Processor -- start 2023-05-22T15:42:17.709 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:17.704+0000 7f0ee6fa1640 1 -- start start 2023-05-22T15:42:17.710 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:17.704+0000 7f0ee6fa1640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f0ee00ade80 0x7f0ee00cc400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:17.710 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:17.704+0000 7f0ee6fa1640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f0ee015c7f0 0x7f0ee01593b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:17.710 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:17.704+0000 7f0ee6fa1640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ee00c95f0 con 0x7f0ee015c350 2023-05-22T15:42:17.710 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:17.704+0000 7f0ee6fa1640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ee00ca270 con 0x7f0ee015c7f0 2023-05-22T15:42:17.710 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:17.704+0000 7f0ee6fa1640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ee00ca690 con 0x7f0ee00ade80 2023-05-22T15:42:17.710 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:17.704+0000 7f0ee553e640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f0ee015c7f0 0x7f0ee01593b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:17.711 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:17.704+0000 7f0ee553e640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f0ee015c7f0 0x7f0ee01593b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41486/0 (socket says 172.21.15.16:41486) 2023-05-22T15:42:17.711 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:17.704+0000 7f0ee4d3d640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f0ee00ade80 0x7f0ee00cc400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:17.711 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:17.704+0000 7f0ee553e640 1 -- 172.21.15.16:0/840186128 learned_addr learned my addr 172.21.15.16:0/840186128 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:42:17.711 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:17.704+0000 7f0ee553e640 1 -- 172.21.15.16:0/840186128 >> v2:172.21.15.62:3300/0 conn(0x7f0ee00ade80 msgr2=0x7f0ee00cc400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:42:17.711 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:17.704+0000 7f0ee553e640 1 --2- 172.21.15.16:0/840186128 >> v2:172.21.15.62:3300/0 conn(0x7f0ee00ade80 0x7f0ee00cc400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:17.711 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:17.704+0000 7f0ee553e640 1 -- 172.21.15.16:0/840186128 >> v1:172.21.15.62:6789/0 conn(0x7f0ee015c350 legacy=0x7f0ee0158ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:17.711 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:17.704+0000 7f0ee553e640 1 -- 172.21.15.16:0/840186128 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ee00c98d0 con 0x7f0ee015c7f0 2023-05-22T15:42:17.712 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:17.704+0000 7f0edf7fe640 1 -- 172.21.15.16:0/840186128 <== mon.1 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4070973959 0 0) 0x7f0ee00c95f0 con 0x7f0ee015c350 2023-05-22T15:42:17.712 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:17.708+0000 7f0ee553e640 1 --2- 172.21.15.16:0/840186128 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ee015c7f0 0x7f0ee01593b0 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 crypto rx=0x7f0ee00cf660 tx=0x7f0ed801c440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d02a1b4bb83c353 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:17.712 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:17.708+0000 7f0edf7fe640 1 -- 172.21.15.16:0/840186128 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ed80ab640 con 0x7f0ee015c7f0 2023-05-22T15:42:17.712 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:17.708+0000 7f0edf7fe640 1 -- 172.21.15.16:0/840186128 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ed80a7070 con 0x7f0ee015c7f0 2023-05-22T15:42:17.713 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:17.708+0000 7f0edf7fe640 1 -- 172.21.15.16:0/840186128 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ed80ab9c0 con 0x7f0ee015c7f0 2023-05-22T15:42:17.713 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:17.708+0000 7f0ee6fa1640 1 -- 172.21.15.16:0/840186128 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ee015c7f0 msgr2=0x7f0ee01593b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:17.713 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:17.708+0000 7f0ee6fa1640 1 --2- 172.21.15.16:0/840186128 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ee015c7f0 0x7f0ee01593b0 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 crypto rx=0x7f0ee00cf660 tx=0x7f0ed801c440 comp rx=0 tx=0).stop 2023-05-22T15:42:17.714 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:17.708+0000 7f0ee6fa1640 1 -- 172.21.15.16:0/840186128 shutdown_connections 2023-05-22T15:42:17.714 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:17.708+0000 7f0ee6fa1640 1 --2- 172.21.15.16:0/840186128 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ee015c7f0 0x7f0ee01593b0 unknown :-1 s=CLOSED pgs=1073 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:17.714 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:17.708+0000 7f0ee6fa1640 1 --2- 172.21.15.16:0/840186128 >> v2:172.21.15.62:3300/0 conn(0x7f0ee00ade80 0x7f0ee00cc400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:17.714 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:17.708+0000 7f0ee6fa1640 1 -- 172.21.15.16:0/840186128 >> 172.21.15.16:0/840186128 conn(0x7f0ee00b1110 msgr2=0x7f0ee00ce830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:17.714 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:17.708+0000 7f0ee6fa1640 1 -- 172.21.15.16:0/840186128 shutdown_connections 2023-05-22T15:42:17.714 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:17.708+0000 7f0ee6fa1640 1 -- 172.21.15.16:0/840186128 wait complete. 2023-05-22T15:42:17.716 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:17.712+0000 7f0ee6fa1640 1 Processor -- start 2023-05-22T15:42:17.716 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:17.712+0000 7f0ee6fa1640 1 -- start start 2023-05-22T15:42:17.716 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:17.712+0000 7f0ee6fa1640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0ee00ade80 0x7f0ee014e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:17.716 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:17.712+0000 7f0ee6fa1640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0ee015c350 0x7f0ee01492f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:17.716 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:17.712+0000 7f0ee6fa1640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ee015c7f0 0x7f0ee015b200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:17.716 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:17.712+0000 7f0ee6fa1640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ee014f920 con 0x7f0ee015c7f0 2023-05-22T15:42:17.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:17.712+0000 7f0ee6fa1640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ee014eca0 con 0x7f0ee00ade80 2023-05-22T15:42:17.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:17.712+0000 7f0ee6fa1640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ee014ef80 con 0x7f0ee015c350 2023-05-22T15:42:17.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:17.712+0000 7f0ee553e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ee015c7f0 0x7f0ee015b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:17.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:17.712+0000 7f0ee553e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ee015c7f0 0x7f0ee015b200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41488/0 (socket says 172.21.15.16:41488) 2023-05-22T15:42:17.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:17.712+0000 7f0ee553e640 1 -- 172.21.15.16:0/323818061 learned_addr learned my addr 172.21.15.16:0/323818061 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:42:17.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:17.712+0000 7f0ee4d3d640 1 --2- 172.21.15.16:0/323818061 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0ee00ade80 0x7f0ee014e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:17.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:17.712+0000 7f0edffff640 1 --2- 172.21.15.16:0/323818061 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0ee015c350 0x7f0ee01492f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:17.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:17.712+0000 7f0ee553e640 1 -- 172.21.15.16:0/323818061 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0ee015c350 msgr2=0x7f0ee01492f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:17.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:17.712+0000 7f0ee553e640 1 --2- 172.21.15.16:0/323818061 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0ee015c350 0x7f0ee01492f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:17.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:17.712+0000 7f0ee553e640 1 -- 172.21.15.16:0/323818061 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0ee00ade80 msgr2=0x7f0ee014e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:17.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:17.712+0000 7f0ee553e640 1 --2- 172.21.15.16:0/323818061 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0ee00ade80 0x7f0ee014e5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:17.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:17.712+0000 7f0ee553e640 1 -- 172.21.15.16:0/323818061 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ed80094d0 con 0x7f0ee015c7f0 2023-05-22T15:42:17.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:17.712+0000 7f0ee553e640 1 --2- 172.21.15.16:0/323818061 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ee015c7f0 0x7f0ee015b200 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7f0ed8019050 tx=0x7f0ed801ac80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:17.820 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:17.712+0000 7f0edd7fa640 1 -- 172.21.15.16:0/323818061 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ed8004230 con 0x7f0ee015c7f0 2023-05-22T15:42:17.820 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:17.712+0000 7f0edd7fa640 1 -- 172.21.15.16:0/323818061 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ed80b8070 con 0x7f0ee015c7f0 2023-05-22T15:42:17.820 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:17.712+0000 7f0ee6fa1640 1 -- 172.21.15.16:0/323818061 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ee00cc940 con 0x7f0ee015c7f0 2023-05-22T15:42:17.821 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:17.712+0000 7f0edd7fa640 1 -- 172.21.15.16:0/323818061 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ed80ab700 con 0x7f0ee015c7f0 2023-05-22T15:42:17.821 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:17.712+0000 7f0ee6fa1640 1 -- 172.21.15.16:0/323818061 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ee015cc10 con 0x7f0ee015c7f0 2023-05-22T15:42:17.822 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:17.712+0000 7f0ee6fa1640 1 -- 172.21.15.16:0/323818061 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ee015d090 con 0x7f0ee015c7f0 2023-05-22T15:42:17.822 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:17.712+0000 7f0edd7fa640 1 -- 172.21.15.16:0/323818061 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f0ed80ab860 con 0x7f0ee015c7f0 2023-05-22T15:42:17.822 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:17.712+0000 7f0edd7fa640 1 --2- 172.21.15.16:0/323818061 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f0ecc0654a0 0x7f0ecc067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:17.822 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:17.712+0000 7f0edd7fa640 1 -- 172.21.15.16:0/323818061 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6218+0+0 (secure 0 0 0) 0x7f0ed811cd80 con 0x7f0ee015c7f0 2023-05-22T15:42:17.822 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:17.712+0000 7f0ee4d3d640 1 --2- 172.21.15.16:0/323818061 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f0ecc0654a0 0x7f0ecc067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:17.822 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:17.716+0000 7f0ee4d3d640 1 --2- 172.21.15.16:0/323818061 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f0ecc0654a0 0x7f0ecc067950 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f0ed0011340 tx=0x7f0ed000b040 comp rx=0 tx=0).ready entity=mgr.6765 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:17.823 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:17.716+0000 7f0edd7fa640 1 -- 172.21.15.16:0/323818061 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f0ed8120030 con 0x7f0ee015c7f0 2023-05-22T15:42:17.887 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:17.884+0000 7f0ee6fa1640 1 -- 172.21.15.16:0/323818061 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f0ee014fa60 con 0x7f0ee015c7f0 2023-05-22T15:42:18.713 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:18.708+0000 7f0edd7fa640 1 -- 172.21.15.16:0/323818061 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v86) v1 ==== 97+0+0 (secure 0 0 0) 0x7f0ee014fa60 con 0x7f0ee015c7f0 2023-05-22T15:42:18.714 INFO:teuthology.orchestra.run.smithi016.stderr: removed all blocklist entries 2023-05-22T15:42:18.716 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:18.712+0000 7f0ee6fa1640 1 -- 172.21.15.16:0/323818061 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f0ecc0654a0 msgr2=0x7f0ecc067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:18.716 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:18.712+0000 7f0ee6fa1640 1 --2- 172.21.15.16:0/323818061 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f0ecc0654a0 0x7f0ecc067950 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f0ed0011340 tx=0x7f0ed000b040 comp rx=0 tx=0).stop 2023-05-22T15:42:18.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:18.712+0000 7f0ee6fa1640 1 -- 172.21.15.16:0/323818061 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ee015c7f0 msgr2=0x7f0ee015b200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:18.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:18.712+0000 7f0ee6fa1640 1 --2- 172.21.15.16:0/323818061 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ee015c7f0 0x7f0ee015b200 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7f0ed8019050 tx=0x7f0ed801ac80 comp rx=0 tx=0).stop 2023-05-22T15:42:18.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:18.712+0000 7f0ee6fa1640 1 -- 172.21.15.16:0/323818061 shutdown_connections 2023-05-22T15:42:18.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:18.712+0000 7f0ee6fa1640 1 --2- 172.21.15.16:0/323818061 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ee015c7f0 0x7f0ee015b200 unknown :-1 s=CLOSED pgs=1074 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:18.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:18.712+0000 7f0ee6fa1640 1 --2- 172.21.15.16:0/323818061 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f0ecc0654a0 0x7f0ecc067950 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:18.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:18.712+0000 7f0ee6fa1640 1 --2- 172.21.15.16:0/323818061 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0ee015c350 0x7f0ee01492f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:18.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:18.712+0000 7f0ee6fa1640 1 --2- 172.21.15.16:0/323818061 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0ee00ade80 0x7f0ee014e5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:18.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:18.712+0000 7f0ee6fa1640 1 -- 172.21.15.16:0/323818061 >> 172.21.15.16:0/323818061 conn(0x7f0ee00b1110 msgr2=0x7f0ee006cbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:18.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:18.712+0000 7f0ee6fa1640 1 -- 172.21.15.16:0/323818061 shutdown_connections 2023-05-22T15:42:18.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:18.712+0000 7f0ee6fa1640 1 -- 172.21.15.16:0/323818061 wait complete. 2023-05-22T15:42:18.735 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2023-05-22T15:42:18.862 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:18.856+0000 7ff30baee640 1 Processor -- start 2023-05-22T15:42:18.862 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:18.856+0000 7ff30baee640 1 -- start start 2023-05-22T15:42:18.863 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:18.860+0000 7ff30baee640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff304070f10 0x7ff3040afb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:18.863 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:18.860+0000 7ff30baee640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7ff30415ad40 0x7ff3040c8670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:18.863 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:18.860+0000 7ff30baee640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff30415a8b0 con 0x7ff30415e2f0 2023-05-22T15:42:18.863 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:18.860+0000 7ff30baee640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3040b0200 con 0x7ff304070f10 2023-05-22T15:42:18.863 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:18.860+0000 7ff30baee640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3040b04e0 con 0x7ff30415ad40 2023-05-22T15:42:18.864 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:18.860+0000 7ff30988a640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff304070f10 0x7ff3040afb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:18.864 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:18.860+0000 7ff309089640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7ff30415e2f0 0x7ff3040712e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:41770/0 (socket says 172.21.15.16:41770) 2023-05-22T15:42:18.864 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:18.860+0000 7ff309089640 1 -- 172.21.15.16:0/3529140447 learned_addr learned my addr 172.21.15.16:0/3529140447 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:42:18.865 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:18.860+0000 7ff30988a640 1 -- 172.21.15.16:0/3529140447 >> v2:172.21.15.62:3300/0 conn(0x7ff30415ad40 msgr2=0x7ff3040c8670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:42:18.865 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:18.860+0000 7ff30a08b640 1 --2- 172.21.15.16:0/3529140447 >> v2:172.21.15.62:3300/0 conn(0x7ff30415ad40 0x7ff3040c8670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:18.865 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:18.860+0000 7ff308888640 1 -- 172.21.15.16:0/3529140447 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1466658667 0 0) 0x7ff30415a8b0 con 0x7ff30415e2f0 2023-05-22T15:42:18.865 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:18.860+0000 7ff30988a640 1 --2- 172.21.15.16:0/3529140447 >> v2:172.21.15.62:3300/0 conn(0x7ff30415ad40 0x7ff3040c8670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:18.865 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:18.860+0000 7ff30988a640 1 -- 172.21.15.16:0/3529140447 >> v1:172.21.15.16:6789/0 conn(0x7ff30415e2f0 legacy=0x7ff3040712e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:18.865 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:18.860+0000 7ff30988a640 1 -- 172.21.15.16:0/3529140447 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff30415a540 con 0x7ff304070f10 2023-05-22T15:42:18.865 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:18.860+0000 7ff30a08b640 1 --2- 172.21.15.16:0/3529140447 >> v2:172.21.15.62:3300/0 conn(0x7ff30415ad40 0x7ff3040c8670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:42:18.866 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:18.860+0000 7ff30988a640 1 --2- 172.21.15.16:0/3529140447 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff304070f10 0x7ff3040afb40 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 crypto rx=0x7ff3040719f0 tx=0x7ff30001c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47007e0846a3d5bd server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:18.867 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:18.860+0000 7ff308888640 1 -- 172.21.15.16:0/3529140447 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3000ad660 con 0x7ff304070f10 2023-05-22T15:42:18.867 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:18.864+0000 7ff308888640 1 -- 172.21.15.16:0/3529140447 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3000ad7c0 con 0x7ff304070f10 2023-05-22T15:42:18.867 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:18.864+0000 7ff308888640 1 -- 172.21.15.16:0/3529140447 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3000adb50 con 0x7ff304070f10 2023-05-22T15:42:18.867 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:18.864+0000 7ff30baee640 1 -- 172.21.15.16:0/3529140447 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff304070f10 msgr2=0x7ff3040afb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:18.867 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:18.864+0000 7ff30baee640 1 --2- 172.21.15.16:0/3529140447 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff304070f10 0x7ff3040afb40 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 crypto rx=0x7ff3040719f0 tx=0x7ff30001c3a0 comp rx=0 tx=0).stop 2023-05-22T15:42:18.867 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:18.864+0000 7ff30baee640 1 -- 172.21.15.16:0/3529140447 shutdown_connections 2023-05-22T15:42:18.867 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:18.864+0000 7ff30baee640 1 --2- 172.21.15.16:0/3529140447 >> v2:172.21.15.62:3300/0 conn(0x7ff30415ad40 0x7ff3040c8670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:18.868 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:18.864+0000 7ff30baee640 1 --2- 172.21.15.16:0/3529140447 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff304070f10 0x7ff3040afb40 unknown :-1 s=CLOSED pgs=1076 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:18.868 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:18.864+0000 7ff30baee640 1 -- 172.21.15.16:0/3529140447 >> 172.21.15.16:0/3529140447 conn(0x7ff304069720 msgr2=0x7ff304069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:18.868 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:18.864+0000 7ff30baee640 1 -- 172.21.15.16:0/3529140447 shutdown_connections 2023-05-22T15:42:18.868 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:18.864+0000 7ff30baee640 1 -- 172.21.15.16:0/3529140447 wait complete. 2023-05-22T15:42:18.869 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:18.864+0000 7ff30baee640 1 Processor -- start 2023-05-22T15:42:18.870 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:18.864+0000 7ff30baee640 1 -- start start 2023-05-22T15:42:18.870 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:18.864+0000 7ff30baee640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff304070f10 0x7ff3040c5d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:18.870 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:18.864+0000 7ff30baee640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff30415ad40 0x7ff304148b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:18.870 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:18.864+0000 7ff30baee640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff30415e2f0 0x7ff3040b16a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:18.870 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:18.864+0000 7ff30baee640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3040c6270 con 0x7ff30415ad40 2023-05-22T15:42:18.870 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:18.864+0000 7ff30baee640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3040c4ea0 con 0x7ff30415e2f0 2023-05-22T15:42:18.871 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:18.864+0000 7ff30baee640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3040c58a0 con 0x7ff304070f10 2023-05-22T15:42:18.871 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:18.864+0000 7ff309089640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff30415ad40 0x7ff304148b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:18.871 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:18.864+0000 7ff309089640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff30415ad40 0x7ff304148b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41510/0 (socket says 172.21.15.16:41510) 2023-05-22T15:42:18.871 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:18.864+0000 7ff309089640 1 -- 172.21.15.16:0/3134670552 learned_addr learned my addr 172.21.15.16:0/3134670552 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:42:18.871 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:18.864+0000 7ff30988a640 1 --2- 172.21.15.16:0/3134670552 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff304070f10 0x7ff3040c5d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:18.871 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:18.864+0000 7ff30a08b640 1 --2- 172.21.15.16:0/3134670552 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff30415e2f0 0x7ff3040b16a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:18.871 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:18.864+0000 7ff309089640 1 -- 172.21.15.16:0/3134670552 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff304070f10 msgr2=0x7ff3040c5d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:18.872 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:18.864+0000 7ff309089640 1 --2- 172.21.15.16:0/3134670552 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff304070f10 0x7ff3040c5d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:18.872 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:18.864+0000 7ff309089640 1 -- 172.21.15.16:0/3134670552 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff30415e2f0 msgr2=0x7ff3040b16a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:18.872 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:18.864+0000 7ff309089640 1 --2- 172.21.15.16:0/3134670552 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff30415e2f0 0x7ff3040b16a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:18.872 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:18.864+0000 7ff309089640 1 -- 172.21.15.16:0/3134670552 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff300009420 con 0x7ff30415ad40 2023-05-22T15:42:18.872 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:18.864+0000 7ff30a08b640 1 --2- 172.21.15.16:0/3134670552 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff30415e2f0 0x7ff3040b16a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:42:18.872 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:18.864+0000 7ff309089640 1 --2- 172.21.15.16:0/3134670552 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff30415ad40 0x7ff304148b80 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7ff2f400c4b0 tx=0x7ff2f400c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:18.872 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:18.864+0000 7ff2fa7fc640 1 -- 172.21.15.16:0/3134670552 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2f40074e0 con 0x7ff30415ad40 2023-05-22T15:42:18.872 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:18.864+0000 7ff30baee640 1 -- 172.21.15.16:0/3134670552 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff304157bb0 con 0x7ff30415ad40 2023-05-22T15:42:18.873 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:18.864+0000 7ff2fa7fc640 1 -- 172.21.15.16:0/3134670552 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2f4007660 con 0x7ff30415ad40 2023-05-22T15:42:18.873 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:18.864+0000 7ff2fa7fc640 1 -- 172.21.15.16:0/3134670552 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2f4004a30 con 0x7ff30415ad40 2023-05-22T15:42:18.873 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:18.864+0000 7ff30baee640 1 -- 172.21.15.16:0/3134670552 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff30415b140 con 0x7ff30415ad40 2023-05-22T15:42:18.874 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:18.868+0000 7ff30baee640 1 -- 172.21.15.16:0/3134670552 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff304166c10 con 0x7ff30415ad40 2023-05-22T15:42:18.874 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:18.868+0000 7ff2fa7fc640 1 -- 172.21.15.16:0/3134670552 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) v1 ==== 84332+0+0 (secure 0 0 0) 0x7ff2f4004b90 con 0x7ff30415ad40 2023-05-22T15:42:18.874 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:18.868+0000 7ff2fa7fc640 1 --2- 172.21.15.16:0/3134670552 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ff2e80654a0 0x7ff2e8067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:18.874 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:18.868+0000 7ff2fa7fc640 1 -- 172.21.15.16:0/3134670552 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 5530+0+0 (secure 0 0 0) 0x7ff2f4081cf0 con 0x7ff30415ad40 2023-05-22T15:42:18.874 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:18.868+0000 7ff30988a640 1 --2- 172.21.15.16:0/3134670552 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ff2e80654a0 0x7ff2e8067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:18.874 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:18.868+0000 7ff30988a640 1 --2- 172.21.15.16:0/3134670552 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ff2e80654a0 0x7ff2e8067950 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7ff3000af000 tx=0x7ff3000a7000 comp rx=0 tx=0).ready entity=mgr.6765 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:18.875 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:18.872+0000 7ff2fa7fc640 1 -- 172.21.15.16:0/3134670552 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7ff2f404fa10 con 0x7ff30415ad40 2023-05-22T15:42:19.083 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.080+0000 7ff30baee640 1 -- 172.21.15.16:0/3134670552 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7ff30404dbe0 con 0x7ff30415ad40 2023-05-22T15:42:19.084 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.080+0000 7ff2fa7fc640 1 -- 172.21.15.16:0/3134670552 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v18) v1 ==== 80+0+5304 (secure 0 0 0) 0x7ff2f4018090 con 0x7ff30415ad40 2023-05-22T15:42:19.084 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:42:19.085 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-05-22T15:42:19.085 INFO:teuthology.orchestra.run.smithi016.stdout: "auth_dump": [ 2023-05-22T15:42:19.085 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T15:42:19.085 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "mds.a", 2023-05-22T15:42:19.085 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQCAiWtkw/K0GRAA9hvTMKMmcbgKV68GeoC4ag==", 2023-05-22T15:42:19.085 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-05-22T15:42:19.085 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2023-05-22T15:42:19.086 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow *", 2023-05-22T15:42:19.086 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow *", 2023-05-22T15:42:19.086 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-05-22T15:42:19.087 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T15:42:19.087 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:42:19.087 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T15:42:19.087 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "mds.b", 2023-05-22T15:42:19.087 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQCAiWtkoFzfJRAAxCiB0MGM5apMVA81WDq+dA==", 2023-05-22T15:42:19.088 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-05-22T15:42:19.088 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2023-05-22T15:42:19.088 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow *", 2023-05-22T15:42:19.088 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow *", 2023-05-22T15:42:19.088 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-05-22T15:42:19.088 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T15:42:19.089 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:42:19.089 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T15:42:19.089 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "mds.c", 2023-05-22T15:42:19.089 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQCAiWtkvgOKHxAAMuASPtMqwyoqpZ5I+MRS7A==", 2023-05-22T15:42:19.090 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-05-22T15:42:19.090 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2023-05-22T15:42:19.090 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow *", 2023-05-22T15:42:19.090 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow *", 2023-05-22T15:42:19.090 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-05-22T15:42:19.091 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T15:42:19.091 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:42:19.091 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T15:42:19.091 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "mds.d", 2023-05-22T15:42:19.091 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQCAiWtk6hPQKxAAlCtfC36vja5ftz6uOqUhHg==", 2023-05-22T15:42:19.091 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-05-22T15:42:19.091 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2023-05-22T15:42:19.092 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow *", 2023-05-22T15:42:19.092 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow *", 2023-05-22T15:42:19.092 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-05-22T15:42:19.092 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T15:42:19.092 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:42:19.092 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T15:42:19.093 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.0", 2023-05-22T15:42:19.093 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQCCiWtk6ZLoCRAAKdCQitZuPnWcIpkiXhJeww==", 2023-05-22T15:42:19.093 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-05-22T15:42:19.093 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2023-05-22T15:42:19.094 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2023-05-22T15:42:19.094 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-05-22T15:42:19.094 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T15:42:19.094 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:42:19.094 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T15:42:19.094 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.1", 2023-05-22T15:42:19.095 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQCFiWtkXtVeERAAkgPeDKuFmWXrDiCyqWOYwQ==", 2023-05-22T15:42:19.095 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-05-22T15:42:19.095 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2023-05-22T15:42:19.095 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2023-05-22T15:42:19.095 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-05-22T15:42:19.095 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T15:42:19.095 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:42:19.096 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T15:42:19.096 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.2", 2023-05-22T15:42:19.096 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQCIiWtkdGaRHRAAx3POIp3mnz7p7XcCVMXm5Q==", 2023-05-22T15:42:19.096 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-05-22T15:42:19.096 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2023-05-22T15:42:19.096 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2023-05-22T15:42:19.096 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-05-22T15:42:19.097 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T15:42:19.097 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:42:19.097 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T15:42:19.097 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.3", 2023-05-22T15:42:19.097 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQCLiWtkSBydJhAAsgc19Xaf/GOyy9nm9MLdJQ==", 2023-05-22T15:42:19.097 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-05-22T15:42:19.098 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2023-05-22T15:42:19.098 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2023-05-22T15:42:19.098 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-05-22T15:42:19.098 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T15:42:19.098 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:42:19.099 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T15:42:19.099 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.4", 2023-05-22T15:42:19.099 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQCRiWtkueYvJRAARXbHlKHrTbgTqBArRJfvBA==", 2023-05-22T15:42:19.099 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-05-22T15:42:19.099 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2023-05-22T15:42:19.099 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2023-05-22T15:42:19.099 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-05-22T15:42:19.100 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T15:42:19.100 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:42:19.100 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T15:42:19.100 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.5", 2023-05-22T15:42:19.100 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQCUiWtkH3hwLRAAUekP6y+wkJVPh+OPDiZbzQ==", 2023-05-22T15:42:19.101 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-05-22T15:42:19.101 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2023-05-22T15:42:19.101 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2023-05-22T15:42:19.101 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-05-22T15:42:19.102 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T15:42:19.102 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:42:19.102 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T15:42:19.102 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.6", 2023-05-22T15:42:19.102 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQCXiWtkLsnBNhAAQ8ilX8ZU4ncJ2JJU0I+U4w==", 2023-05-22T15:42:19.102 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-05-22T15:42:19.102 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2023-05-22T15:42:19.103 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2023-05-22T15:42:19.103 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-05-22T15:42:19.103 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T15:42:19.103 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:42:19.103 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T15:42:19.104 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.7", 2023-05-22T15:42:19.104 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQCbiWtkUvXjBBAApMWw7wdfxWq0WEQDL2qprA==", 2023-05-22T15:42:19.104 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-05-22T15:42:19.104 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2023-05-22T15:42:19.104 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2023-05-22T15:42:19.104 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-05-22T15:42:19.104 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T15:42:19.105 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:42:19.105 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T15:42:19.105 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "client.0", 2023-05-22T15:42:19.105 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQCAiWtkZwuUMRAAnYqSFiaT/QRsajdjNgTetA==", 2023-05-22T15:42:19.105 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-05-22T15:42:19.105 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2023-05-22T15:42:19.105 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow r", 2023-05-22T15:42:19.106 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow rw tag cephfs data=cephfs" 2023-05-22T15:42:19.106 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T15:42:19.106 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:42:19.106 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T15:42:19.106 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "client.1", 2023-05-22T15:42:19.106 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQCAiWtkPimmORAABzNL2rANz5TgCl3p++GVng==", 2023-05-22T15:42:19.107 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-05-22T15:42:19.107 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2023-05-22T15:42:19.107 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow r", 2023-05-22T15:42:19.107 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow rw", 2023-05-22T15:42:19.107 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow rwx" 2023-05-22T15:42:19.107 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T15:42:19.107 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:42:19.108 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T15:42:19.108 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "client.2", 2023-05-22T15:42:19.108 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQCAiWtkqkPMNhAAqugrnD4Mh/4At8QAFmzqcg==", 2023-05-22T15:42:19.108 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-05-22T15:42:19.108 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2023-05-22T15:42:19.108 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow r", 2023-05-22T15:42:19.108 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow rw", 2023-05-22T15:42:19.109 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow rwx" 2023-05-22T15:42:19.109 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T15:42:19.109 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:42:19.109 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T15:42:19.109 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "client.3", 2023-05-22T15:42:19.109 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQCBiWtkpvlWAxAAPuedivXBGZGHIO6hSnz8PQ==", 2023-05-22T15:42:19.109 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-05-22T15:42:19.110 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2023-05-22T15:42:19.110 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow r", 2023-05-22T15:42:19.110 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow rw", 2023-05-22T15:42:19.110 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow rwx" 2023-05-22T15:42:19.110 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T15:42:19.110 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:42:19.111 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T15:42:19.111 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "client.admin", 2023-05-22T15:42:19.111 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQCAiWtkxG9hAhAAfnUnRrBRUFoCEtsb2riNOQ==", 2023-05-22T15:42:19.111 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-05-22T15:42:19.111 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow *", 2023-05-22T15:42:19.112 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow *", 2023-05-22T15:42:19.112 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow *", 2023-05-22T15:42:19.112 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-05-22T15:42:19.112 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T15:42:19.112 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:42:19.112 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T15:42:19.113 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "mgr.x", 2023-05-22T15:42:19.113 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQCAiWtkkaIqFhAAb/4BI6FYWOy3qOSaylXLSw==", 2023-05-22T15:42:19.113 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-05-22T15:42:19.113 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow *", 2023-05-22T15:42:19.113 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile mgr", 2023-05-22T15:42:19.113 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-05-22T15:42:19.113 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T15:42:19.114 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:42:19.114 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T15:42:19.114 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "mgr.y", 2023-05-22T15:42:19.114 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQCAiWtkhgVWExAAjDoh7cexFVfsigsYfGkmVg==", 2023-05-22T15:42:19.114 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-05-22T15:42:19.114 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow *", 2023-05-22T15:42:19.115 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile mgr", 2023-05-22T15:42:19.115 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-05-22T15:42:19.115 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T15:42:19.115 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T15:42:19.115 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-05-22T15:42:19.115 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-05-22T15:42:19.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.084+0000 7ff30baee640 1 -- 172.21.15.16:0/3134670552 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ff2e80654a0 msgr2=0x7ff2e8067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:19.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.084+0000 7ff30baee640 1 --2- 172.21.15.16:0/3134670552 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ff2e80654a0 0x7ff2e8067950 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7ff3000af000 tx=0x7ff3000a7000 comp rx=0 tx=0).stop 2023-05-22T15:42:19.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.084+0000 7ff30baee640 1 -- 172.21.15.16:0/3134670552 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff30415ad40 msgr2=0x7ff304148b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:19.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.084+0000 7ff30baee640 1 --2- 172.21.15.16:0/3134670552 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff30415ad40 0x7ff304148b80 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7ff2f400c4b0 tx=0x7ff2f400c970 comp rx=0 tx=0).stop 2023-05-22T15:42:19.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.084+0000 7ff30baee640 1 -- 172.21.15.16:0/3134670552 shutdown_connections 2023-05-22T15:42:19.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.084+0000 7ff30baee640 1 --2- 172.21.15.16:0/3134670552 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff30415e2f0 0x7ff3040b16a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:19.118 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.084+0000 7ff30baee640 1 --2- 172.21.15.16:0/3134670552 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff30415ad40 0x7ff304148b80 unknown :-1 s=CLOSED pgs=1077 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:19.118 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.084+0000 7ff30baee640 1 --2- 172.21.15.16:0/3134670552 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ff2e80654a0 0x7ff2e8067950 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:19.118 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.084+0000 7ff30baee640 1 --2- 172.21.15.16:0/3134670552 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff304070f10 0x7ff3040c5d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:19.118 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.084+0000 7ff30baee640 1 -- 172.21.15.16:0/3134670552 >> 172.21.15.16:0/3134670552 conn(0x7ff304069720 msgr2=0x7ff3040c99b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:19.118 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.084+0000 7ff30baee640 1 -- 172.21.15.16:0/3134670552 shutdown_connections 2023-05-22T15:42:19.118 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.084+0000 7ff30baee640 1 -- 172.21.15.16:0/3134670552 wait complete. 2023-05-22T15:42:19.119 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-05-22T15:42:19.248 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.244+0000 7f82957d9640 1 Processor -- start 2023-05-22T15:42:19.248 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.244+0000 7f82957d9640 1 -- start start 2023-05-22T15:42:19.249 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.244+0000 7f82957d9640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f829015cfa0 0x7f829006db00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:19.249 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.244+0000 7f82957d9640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f829006e740 0x7f829006ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:19.249 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.244+0000 7f82957d9640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f829006e040 con 0x7f82900cf870 2023-05-22T15:42:19.249 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.244+0000 7f82957d9640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82900cb9f0 con 0x7f829015cfa0 2023-05-22T15:42:19.249 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.244+0000 7f82957d9640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f829006f100 con 0x7f829006e740 2023-05-22T15:42:19.250 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.244+0000 7f828e7fc640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f82900cf870 0x7f82900b1fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:33408/0 (socket says 172.21.15.16:33408) 2023-05-22T15:42:19.250 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.244+0000 7f828effd640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f829015cfa0 0x7f829006db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:19.250 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.244+0000 7f828e7fc640 1 -- 172.21.15.16:0/1809756320 learned_addr learned my addr 172.21.15.16:0/1809756320 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:42:19.251 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.244+0000 7f828f7fe640 1 --2- 172.21.15.16:0/1809756320 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f829006e740 0x7f829006ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:19.251 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.244+0000 7f828effd640 1 -- 172.21.15.16:0/1809756320 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f829006e740 msgr2=0x7f829006ebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:42:19.251 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.244+0000 7f828effd640 1 --2- 172.21.15.16:0/1809756320 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f829006e740 0x7f829006ebc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:19.251 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.244+0000 7f828effd640 1 -- 172.21.15.16:0/1809756320 >> v1:172.21.15.16:6789/0 conn(0x7f82900cf870 legacy=0x7f82900b1fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:19.251 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.244+0000 7f828effd640 1 -- 172.21.15.16:0/1809756320 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f829006e490 con 0x7f829015cfa0 2023-05-22T15:42:19.252 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.248+0000 7f828effd640 1 --2- 172.21.15.16:0/1809756320 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f829015cfa0 0x7f829006db00 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7f8280009d90 tx=0x7f828001c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3227aba24c4c7b38 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:19.252 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.248+0000 7f828dffb640 1 -- 172.21.15.16:0/1809756320 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82800a59e0 con 0x7f829015cfa0 2023-05-22T15:42:19.252 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.248+0000 7f828dffb640 1 -- 172.21.15.16:0/1809756320 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82800a5b40 con 0x7f829015cfa0 2023-05-22T15:42:19.252 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.248+0000 7f828dffb640 1 -- 172.21.15.16:0/1809756320 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82800ad440 con 0x7f829015cfa0 2023-05-22T15:42:19.253 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.248+0000 7f82957d9640 1 -- 172.21.15.16:0/1809756320 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f829015cfa0 msgr2=0x7f829006db00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:19.253 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.248+0000 7f82957d9640 1 --2- 172.21.15.16:0/1809756320 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f829015cfa0 0x7f829006db00 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7f8280009d90 tx=0x7f828001c920 comp rx=0 tx=0).stop 2023-05-22T15:42:19.253 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.248+0000 7f82957d9640 1 -- 172.21.15.16:0/1809756320 shutdown_connections 2023-05-22T15:42:19.253 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.248+0000 7f82957d9640 1 --2- 172.21.15.16:0/1809756320 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f829006e740 0x7f829006ebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:19.254 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.248+0000 7f82957d9640 1 --2- 172.21.15.16:0/1809756320 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f829015cfa0 0x7f829006db00 unknown :-1 s=CLOSED pgs=1079 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:19.254 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.248+0000 7f82957d9640 1 -- 172.21.15.16:0/1809756320 >> 172.21.15.16:0/1809756320 conn(0x7f82900cc870 msgr2=0x7f82900cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:19.254 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.248+0000 7f82957d9640 1 -- 172.21.15.16:0/1809756320 shutdown_connections 2023-05-22T15:42:19.254 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.248+0000 7f82957d9640 1 -- 172.21.15.16:0/1809756320 wait complete. 2023-05-22T15:42:19.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.248+0000 7f82957d9640 1 Processor -- start 2023-05-22T15:42:19.256 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.248+0000 7f82957d9640 1 -- start start 2023-05-22T15:42:19.256 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.252+0000 7f82957d9640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f829006e740 0x7f8290149820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:19.256 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.252+0000 7f82957d9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82900cf870 0x7f82900ce140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:19.256 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.252+0000 7f82957d9640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f829015cfa0 0x7f8290151e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:19.256 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.252+0000 7f82957d9640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82900cd0a0 con 0x7f82900cf870 2023-05-22T15:42:19.256 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.252+0000 7f82957d9640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82900cd380 con 0x7f829006e740 2023-05-22T15:42:19.257 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.252+0000 7f82957d9640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82900cd660 con 0x7f829015cfa0 2023-05-22T15:42:19.257 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.252+0000 7f828e7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82900cf870 0x7f82900ce140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:19.257 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.252+0000 7f828e7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82900cf870 0x7f82900ce140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34624/0 (socket says 172.21.15.16:34624) 2023-05-22T15:42:19.257 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.252+0000 7f828e7fc640 1 -- 172.21.15.16:0/2053937358 learned_addr learned my addr 172.21.15.16:0/2053937358 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:42:19.257 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.252+0000 7f828f7fe640 1 --2- 172.21.15.16:0/2053937358 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f829015cfa0 0x7f8290151e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:19.257 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.252+0000 7f828effd640 1 --2- 172.21.15.16:0/2053937358 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f829006e740 0x7f8290149820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:19.258 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.252+0000 7f828e7fc640 1 -- 172.21.15.16:0/2053937358 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f829015cfa0 msgr2=0x7f8290151e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:19.258 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.252+0000 7f828e7fc640 1 --2- 172.21.15.16:0/2053937358 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f829015cfa0 0x7f8290151e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:19.259 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.252+0000 7f828e7fc640 1 -- 172.21.15.16:0/2053937358 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f829006e740 msgr2=0x7f8290149820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:19.259 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.252+0000 7f828e7fc640 1 --2- 172.21.15.16:0/2053937358 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f829006e740 0x7f8290149820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:19.259 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.252+0000 7f828e7fc640 1 -- 172.21.15.16:0/2053937358 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8280009a70 con 0x7f82900cf870 2023-05-22T15:42:19.259 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.252+0000 7f828e7fc640 1 --2- 172.21.15.16:0/2053937358 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82900cf870 0x7f82900ce140 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 crypto rx=0x7f8284005890 tx=0x7f828400c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:19.259 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.252+0000 7f82777fe640 1 -- 172.21.15.16:0/2053937358 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8284005cf0 con 0x7f82900cf870 2023-05-22T15:42:19.259 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.252+0000 7f82777fe640 1 -- 172.21.15.16:0/2053937358 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8284005e50 con 0x7f82900cf870 2023-05-22T15:42:19.259 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.252+0000 7f82777fe640 1 -- 172.21.15.16:0/2053937358 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82840072b0 con 0x7f82900cf870 2023-05-22T15:42:19.260 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.252+0000 7f82957d9640 1 -- 172.21.15.16:0/2053937358 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82900cb610 con 0x7f82900cf870 2023-05-22T15:42:19.260 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.252+0000 7f82957d9640 1 -- 172.21.15.16:0/2053937358 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82900cb330 con 0x7f82900cf870 2023-05-22T15:42:19.260 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.252+0000 7f82957d9640 1 -- 172.21.15.16:0/2053937358 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f829015d370 con 0x7f82900cf870 2023-05-22T15:42:19.261 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.252+0000 7f82777fe640 1 -- 172.21.15.16:0/2053937358 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f8284007410 con 0x7f82900cf870 2023-05-22T15:42:19.261 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.252+0000 7f82777fe640 1 --2- 172.21.15.16:0/2053937358 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f827c0654a0 0x7f827c067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:19.261 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.252+0000 7f82777fe640 1 -- 172.21.15.16:0/2053937358 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 5530+0+0 (secure 0 0 0) 0x7f8284081b20 con 0x7f82900cf870 2023-05-22T15:42:19.261 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.252+0000 7f828effd640 1 --2- 172.21.15.16:0/2053937358 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f827c0654a0 0x7f827c067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:19.261 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.256+0000 7f828effd640 1 --2- 172.21.15.16:0/2053937358 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f827c0654a0 0x7f827c067950 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f82800af000 tx=0x7f82800a7070 comp rx=0 tx=0).ready entity=mgr.6765 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:19.262 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.256+0000 7f82777fe640 1 -- 172.21.15.16:0/2053937358 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f828404f8a0 con 0x7f82900cf870 2023-05-22T15:42:19.417 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.412+0000 7f82957d9640 1 -- 172.21.15.16:0/2053937358 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f829006e040 con 0x7f82900cf870 2023-05-22T15:42:19.419 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.416+0000 7f82777fe640 1 -- 172.21.15.16:0/2053937358 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v86) v1 ==== 74+0+14328 (secure 0 0 0) 0x7f829006e040 con 0x7f82900cf870 2023-05-22T15:42:19.419 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:42:19.419 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":86,"fsid":"b2719834-7847-4d98-90a6-f65d57f85c49","created":"2023-05-22T15:26:25.259314+0000","modified":"2023-05-22T15:42:18.703817+0000","last_up_change":"2023-05-22T15:26:36.448524+0000","last_in_change":"2023-05-22T15:26:27.675668+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-05-22T15:26:38.019781+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-05-22T15:26:43.570514+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9e7ff854-b25f-4cbb-a5c2-1799d9c53d2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":37021},{"type":"v1","addr":"172.21.15.16:6825","nonce":37021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":37021},{"type":"v1","addr":"172.21.15.16:6827","nonce":37021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":37021},{"type":"v1","addr":"172.21.15.16:6831","nonce":37021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":37021},{"type":"v1","addr":"172.21.15.16:6829","nonce":37021}]},"public_addr":"172.21.15.16:6825/37021","cluster_addr":"172.21.15.16:6827/37021","heartbeat_back_addr":"172.21.15.16:6831/37021","heartbeat_front_addr":"172.21.15.16:6829/37021","state":["exists","up"]},{"osd":1,"uuid":"4702fbc9-a927-4565-aca5-41dc69cc8264","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":37008},{"type":"v1","addr":"172.21.15.16:6801","nonce":37008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":37008},{"type":"v1","addr":"172.21.15.16:6803","nonce":37008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":37008},{"type":"v1","addr":"172.21.15.16:6807","nonce":37008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":37008},{"type":"v1","addr":"172.21.15.16:6805","nonce":37008}]},"public_addr":"172.21.15.16:6801/37008","cluster_addr":"172.21.15.16:6803/37008","heartbeat_back_addr":"172.21.15.16:6807/37008","heartbeat_front_addr":"172.21.15.16:6805/37008","state":["exists","up"]},{"osd":2,"uuid":"1e7bba46-4eba-47ef-bb81-5e2d371638ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":37009},{"type":"v1","addr":"172.21.15.16:6809","nonce":37009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":37009},{"type":"v1","addr":"172.21.15.16:6811","nonce":37009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":37009},{"type":"v1","addr":"172.21.15.16:6815","nonce":37009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":37009},{"type":"v1","addr":"172.21.15.16:6813","nonce":37009}]},"public_addr":"172.21.15.16:6809/37009","cluster_addr":"172.21.15.16:6811/37009","heartbeat_back_addr":"172.21.15.16:6815/37009","heartbeat_front_addr":"172.21.15.16:6813/37009","state":["exists","up"]},{"osd":3,"uuid":"383cacc0-5a88-4904-990e-498ca59ef48a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":37020},{"type":"v1","addr":"172.21.15.16:6817","nonce":37020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":37020},{"type":"v1","addr":"172.21.15.16:6819","nonce":37020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":37020},{"type":"v1","addr":"172.21.15.16:6823","nonce":37020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":37020},{"type":"v1","addr":"172.21.15.16:6821","nonce":37020}]},"public_addr":"172.21.15.16:6817/37020","cluster_addr":"172.21.15.16:6819/37020","heartbeat_back_addr":"172.21.15.16:6823/37020","heartbeat_front_addr":"172.21.15.16:6821/37020","state":["exists","up"]},{"osd":4,"uuid":"970500ef-95b7-42c1-b080-eed4fae7013d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":37353},{"type":"v1","addr":"172.21.15.62:6817","nonce":37353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6819","nonce":37353},{"type":"v1","addr":"172.21.15.62:6822","nonce":37353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37353},{"type":"v1","addr":"172.21.15.62:6831","nonce":37353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37353},{"type":"v1","addr":"172.21.15.62:6826","nonce":37353}]},"public_addr":"172.21.15.62:6817/37353","cluster_addr":"172.21.15.62:6822/37353","heartbeat_back_addr":"172.21.15.62:6831/37353","heartbeat_front_addr":"172.21.15.62:6826/37353","state":["exists","up"]},{"osd":5,"uuid":"022d67db-d4d8-445b-b1b9-8b446a00f5c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":37352},{"type":"v1","addr":"172.21.15.62:6801","nonce":37352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":37352},{"type":"v1","addr":"172.21.15.62:6803","nonce":37352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":37352},{"type":"v1","addr":"172.21.15.62:6807","nonce":37352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":37352},{"type":"v1","addr":"172.21.15.62:6805","nonce":37352}]},"public_addr":"172.21.15.62:6801/37352","cluster_addr":"172.21.15.62:6803/37352","heartbeat_back_addr":"172.21.15.62:6807/37352","heartbeat_front_addr":"172.21.15.62:6805/37352","state":["exists","up"]},{"osd":6,"uuid":"42bd372b-6688-4a1b-8407-e16e8c266e33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":37354},{"type":"v1","addr":"172.21.15.62:6820","nonce":37354}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":37354},{"type":"v1","addr":"172.21.15.62:6823","nonce":37354}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6829","nonce":37354},{"type":"v1","addr":"172.21.15.62:6830","nonce":37354}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6825","nonce":37354},{"type":"v1","addr":"172.21.15.62:6827","nonce":37354}]},"public_addr":"172.21.15.62:6820/37354","cluster_addr":"172.21.15.62:6823/37354","heartbeat_back_addr":"172.21.15.62:6830/37354","heartbeat_front_addr":"172.21.15.62:6827/37354","state":["exists","up"]},{"osd":7,"uuid":"ac70b2a3-f41b-4bf2-b227-7f2cbb55426d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37355},{"type":"v1","addr":"172.21.15.62:6809","nonce":37355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37355},{"type":"v1","addr":"172.21.15.62:6811","nonce":37355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":37355},{"type":"v1","addr":"172.21.15.62:6815","nonce":37355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":37355},{"type":"v1","addr":"172.21.15.62:6813","nonce":37355}]},"public_addr":"172.21.15.62:6809/37355","cluster_addr":"172.21.15.62:6811/37355","heartbeat_back_addr":"172.21.15.62:6815/37355","heartbeat_front_addr":"172.21.15.62:6813/37355","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.368896+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.107273+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.276888+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.287502+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.124791+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.251937+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.255679+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.232797+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-05-22T15:42:19.421 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.416+0000 7f82957d9640 1 -- 172.21.15.16:0/2053937358 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f827c0654a0 msgr2=0x7f827c067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:19.421 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.416+0000 7f82957d9640 1 --2- 172.21.15.16:0/2053937358 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f827c0654a0 0x7f827c067950 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f82800af000 tx=0x7f82800a7070 comp rx=0 tx=0).stop 2023-05-22T15:42:19.421 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.416+0000 7f82957d9640 1 -- 172.21.15.16:0/2053937358 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82900cf870 msgr2=0x7f82900ce140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:19.422 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.416+0000 7f82957d9640 1 --2- 172.21.15.16:0/2053937358 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82900cf870 0x7f82900ce140 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 crypto rx=0x7f8284005890 tx=0x7f828400c960 comp rx=0 tx=0).stop 2023-05-22T15:42:19.422 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.416+0000 7f82957d9640 1 -- 172.21.15.16:0/2053937358 shutdown_connections 2023-05-22T15:42:19.422 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.416+0000 7f82957d9640 1 --2- 172.21.15.16:0/2053937358 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f827c0654a0 0x7f827c067950 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:19.422 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.416+0000 7f82957d9640 1 --2- 172.21.15.16:0/2053937358 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f829015cfa0 0x7f8290151e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:19.422 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.416+0000 7f82957d9640 1 --2- 172.21.15.16:0/2053937358 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82900cf870 0x7f82900ce140 unknown :-1 s=CLOSED pgs=1080 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:19.422 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.416+0000 7f82957d9640 1 --2- 172.21.15.16:0/2053937358 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f829006e740 0x7f8290149820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:19.422 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.416+0000 7f82957d9640 1 -- 172.21.15.16:0/2053937358 >> 172.21.15.16:0/2053937358 conn(0x7f82900cc870 msgr2=0x7f829015acd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:19.422 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.416+0000 7f82957d9640 1 -- 172.21.15.16:0/2053937358 shutdown_connections 2023-05-22T15:42:19.423 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.416+0000 7f82957d9640 1 -- 172.21.15.16:0/2053937358 wait complete. 2023-05-22T15:42:19.439 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-05-22T15:26:38.019781+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-05-22T15:26:43.570514+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-05-22T15:42:19.440 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-05-22T15:42:19.571 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.568+0000 7fca2d7cc640 1 Processor -- start 2023-05-22T15:42:19.571 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.568+0000 7fca2d7cc640 1 -- start start 2023-05-22T15:42:19.572 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.568+0000 7fca2d7cc640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fca280ca530 0x7fca280c8660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:19.573 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.568+0000 7fca2d7cc640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fca280c1380 0x7fca2806f8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:19.573 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.568+0000 7fca2d7cc640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca28164090 con 0x7fca28160170 2023-05-22T15:42:19.573 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.568+0000 7fca2d7cc640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca28164370 con 0x7fca280ca530 2023-05-22T15:42:19.574 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.568+0000 7fca2d7cc640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca28164650 con 0x7fca280c1380 2023-05-22T15:42:19.574 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.568+0000 7fca26ffd640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fca28160170 0x7fca280c9e20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:33416/0 (socket says 172.21.15.16:33416) 2023-05-22T15:42:19.574 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.568+0000 7fca26ffd640 1 -- 172.21.15.16:0/3013774944 learned_addr learned my addr 172.21.15.16:0/3013774944 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:42:19.575 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.568+0000 7fca267fc640 1 --2- 172.21.15.16:0/3013774944 >> v2:172.21.15.62:3300/0 conn(0x7fca280ca530 0x7fca280c8660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:19.575 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.568+0000 7fca277fe640 1 --2- 172.21.15.16:0/3013774944 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fca280c1380 0x7fca2806f8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:19.575 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.568+0000 7fca25ffb640 1 -- 172.21.15.16:0/3013774944 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 999280831 0 0) 0x7fca28164090 con 0x7fca28160170 2023-05-22T15:42:19.575 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.568+0000 7fca25ffb640 1 -- 172.21.15.16:0/3013774944 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca14003530 con 0x7fca28160170 2023-05-22T15:42:19.581 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.568+0000 7fca25ffb640 1 -- 172.21.15.16:0/3013774944 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2169267104 0 0) 0x7fca14003530 con 0x7fca28160170 2023-05-22T15:42:19.581 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.568+0000 7fca25ffb640 1 -- 172.21.15.16:0/3013774944 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fca280c1380 msgr2=0x7fca2806f8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:42:19.582 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.568+0000 7fca25ffb640 1 --2- 172.21.15.16:0/3013774944 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fca280c1380 0x7fca2806f8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:19.582 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.568+0000 7fca25ffb640 1 -- 172.21.15.16:0/3013774944 >> v2:172.21.15.62:3300/0 conn(0x7fca280ca530 msgr2=0x7fca280c8660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:42:19.582 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.568+0000 7fca25ffb640 1 --2- 172.21.15.16:0/3013774944 >> v2:172.21.15.62:3300/0 conn(0x7fca280ca530 0x7fca280c8660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:19.582 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.568+0000 7fca25ffb640 1 -- 172.21.15.16:0/3013774944 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca28164930 con 0x7fca28160170 2023-05-22T15:42:19.582 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.568+0000 7fca25ffb640 1 -- 172.21.15.16:0/3013774944 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7fca20001a00 con 0x7fca28160170 2023-05-22T15:42:19.582 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.568+0000 7fca25ffb640 1 -- 172.21.15.16:0/3013774944 >> v1:172.21.15.16:6789/0 conn(0x7fca28160170 legacy=0x7fca280c9e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:19.583 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.568+0000 7fca25ffb640 1 --2- 172.21.15.16:0/3013774944 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fca14003980 0x7fca14003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:19.583 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.568+0000 7fca25ffb640 1 --2- 172.21.15.16:0/3013774944 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fca140043b0 0x7fca14004870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:19.583 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.568+0000 7fca25ffb640 1 --2- 172.21.15.16:0/3013774944 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fca280c1380 0x7fca14004f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:19.583 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.568+0000 7fca25ffb640 1 -- 172.21.15.16:0/3013774944 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca28164370 con 0x7fca14003980 2023-05-22T15:42:19.583 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.568+0000 7fca25ffb640 1 -- 172.21.15.16:0/3013774944 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca28164650 con 0x7fca140043b0 2023-05-22T15:42:19.584 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.568+0000 7fca25ffb640 1 -- 172.21.15.16:0/3013774944 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca1400b990 con 0x7fca280c1380 2023-05-22T15:42:19.584 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.568+0000 7fca26ffd640 1 --2- 172.21.15.16:0/3013774944 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fca14003980 0x7fca14003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:19.584 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.568+0000 7fca267fc640 1 --2- 172.21.15.16:0/3013774944 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fca140043b0 0x7fca14004870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:19.584 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.568+0000 7fca277fe640 1 --2- 172.21.15.16:0/3013774944 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fca280c1380 0x7fca14004f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:19.584 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.568+0000 7fca26ffd640 1 -- 172.21.15.16:0/3013774944 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fca280c1380 msgr2=0x7fca14004f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:42:19.584 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.568+0000 7fca26ffd640 1 --2- 172.21.15.16:0/3013774944 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fca280c1380 0x7fca14004f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:19.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.568+0000 7fca26ffd640 1 -- 172.21.15.16:0/3013774944 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fca140043b0 msgr2=0x7fca14004870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:42:19.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.568+0000 7fca26ffd640 1 --2- 172.21.15.16:0/3013774944 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fca140043b0 0x7fca14004870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:19.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.568+0000 7fca26ffd640 1 -- 172.21.15.16:0/3013774944 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca14003530 con 0x7fca14003980 2023-05-22T15:42:19.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.572+0000 7fca26ffd640 1 --2- 172.21.15.16:0/3013774944 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fca14003980 0x7fca14003d90 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7fca2000a860 tx=0x7fca2001d360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea02c5f7101e6092 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:19.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.572+0000 7fca25ffb640 1 -- 172.21.15.16:0/3013774944 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca200ad630 con 0x7fca14003980 2023-05-22T15:42:19.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.572+0000 7fca25ffb640 1 -- 172.21.15.16:0/3013774944 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca200a8070 con 0x7fca14003980 2023-05-22T15:42:19.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.572+0000 7fca25ffb640 1 -- 172.21.15.16:0/3013774944 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca200ada10 con 0x7fca14003980 2023-05-22T15:42:19.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.572+0000 7fca2d7cc640 1 -- 172.21.15.16:0/3013774944 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fca14003980 msgr2=0x7fca14003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:19.587 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.572+0000 7fca2d7cc640 1 --2- 172.21.15.16:0/3013774944 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fca14003980 0x7fca14003d90 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7fca2000a860 tx=0x7fca2001d360 comp rx=0 tx=0).stop 2023-05-22T15:42:19.587 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.572+0000 7fca277fe640 1 -- 172.21.15.16:0/3013774944 reap_dead start 2023-05-22T15:42:19.587 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.572+0000 7fca2d7cc640 1 -- 172.21.15.16:0/3013774944 shutdown_connections 2023-05-22T15:42:19.587 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.572+0000 7fca2d7cc640 1 -- 172.21.15.16:0/3013774944 >> 172.21.15.16:0/3013774944 conn(0x7fca28069720 msgr2=0x7fca28069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:19.588 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.572+0000 7fca2d7cc640 1 -- 172.21.15.16:0/3013774944 shutdown_connections 2023-05-22T15:42:19.588 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.572+0000 7fca2d7cc640 1 -- 172.21.15.16:0/3013774944 wait complete. 2023-05-22T15:42:19.588 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.572+0000 7fca2d7cc640 1 Processor -- start 2023-05-22T15:42:19.589 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.572+0000 7fca2d7cc640 1 -- start start 2023-05-22T15:42:19.589 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.576+0000 7fca2d7cc640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fca14003980 0x7fca280c7b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:19.589 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.576+0000 7fca2d7cc640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fca281638b0 0x7fca28162690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:19.589 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.576+0000 7fca2d7cc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fca2815f090 0x7fca2815f530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:19.590 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.576+0000 7fca2d7cc640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca2815fa70 con 0x7fca2815f090 2023-05-22T15:42:19.590 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.576+0000 7fca2d7cc640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca28154de0 con 0x7fca14003980 2023-05-22T15:42:19.590 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.576+0000 7fca2d7cc640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca280c5260 con 0x7fca281638b0 2023-05-22T15:42:19.590 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.576+0000 7fca277fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fca2815f090 0x7fca2815f530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:19.590 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.576+0000 7fca277fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fca2815f090 0x7fca2815f530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34632/0 (socket says 172.21.15.16:34632) 2023-05-22T15:42:19.591 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.576+0000 7fca277fe640 1 -- 172.21.15.16:0/2160290633 learned_addr learned my addr 172.21.15.16:0/2160290633 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:42:19.591 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.576+0000 7fca26ffd640 1 --2- 172.21.15.16:0/2160290633 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fca14003980 0x7fca280c7b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:19.591 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.576+0000 7fca267fc640 1 --2- 172.21.15.16:0/2160290633 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fca281638b0 0x7fca28162690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:19.591 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.576+0000 7fca277fe640 1 -- 172.21.15.16:0/2160290633 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fca281638b0 msgr2=0x7fca28162690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:19.591 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.576+0000 7fca277fe640 1 --2- 172.21.15.16:0/2160290633 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fca281638b0 0x7fca28162690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:19.591 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.576+0000 7fca277fe640 1 -- 172.21.15.16:0/2160290633 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fca14003980 msgr2=0x7fca280c7b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:19.591 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.576+0000 7fca277fe640 1 --2- 172.21.15.16:0/2160290633 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fca14003980 0x7fca280c7b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:19.592 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.576+0000 7fca277fe640 1 -- 172.21.15.16:0/2160290633 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca281606d0 con 0x7fca2815f090 2023-05-22T15:42:19.592 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.576+0000 7fca277fe640 1 --2- 172.21.15.16:0/2160290633 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fca2815f090 0x7fca2815f530 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7fca28071cf0 tx=0x7fca18011550 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:19.592 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.576+0000 7fca1f7fe640 1 -- 172.21.15.16:0/2160290633 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca18004500 con 0x7fca2815f090 2023-05-22T15:42:19.592 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.576+0000 7fca1f7fe640 1 -- 172.21.15.16:0/2160290633 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca180118a0 con 0x7fca2815f090 2023-05-22T15:42:19.592 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.576+0000 7fca2d7cc640 1 -- 172.21.15.16:0/2160290633 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca28160540 con 0x7fca2815f090 2023-05-22T15:42:19.592 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.576+0000 7fca1f7fe640 1 -- 172.21.15.16:0/2160290633 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca18011c90 con 0x7fca2815f090 2023-05-22T15:42:19.592 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.576+0000 7fca2d7cc640 1 -- 172.21.15.16:0/2160290633 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca280700d0 con 0x7fca2815f090 2023-05-22T15:42:19.593 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.576+0000 7fca2d7cc640 1 -- 172.21.15.16:0/2160290633 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca2804dbe0 con 0x7fca2815f090 2023-05-22T15:42:19.593 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.576+0000 7fca1f7fe640 1 -- 172.21.15.16:0/2160290633 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) v1 ==== 84332+0+0 (secure 0 0 0) 0x7fca18003620 con 0x7fca2815f090 2023-05-22T15:42:19.593 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.576+0000 7fca1f7fe640 1 --2- 172.21.15.16:0/2160290633 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fca080654a0 0x7fca08067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:19.593 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.576+0000 7fca1f7fe640 1 -- 172.21.15.16:0/2160290633 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 5530+0+0 (secure 0 0 0) 0x7fca180820e0 con 0x7fca2815f090 2023-05-22T15:42:19.593 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.576+0000 7fca26ffd640 1 --2- 172.21.15.16:0/2160290633 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fca080654a0 0x7fca08067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:19.594 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.580+0000 7fca26ffd640 1 --2- 172.21.15.16:0/2160290633 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fca080654a0 0x7fca08067950 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fca200ad000 tx=0x7fca20001e00 comp rx=0 tx=0).ready entity=mgr.6765 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:19.594 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.580+0000 7fca1f7fe640 1 -- 172.21.15.16:0/2160290633 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fca18053920 con 0x7fca2815f090 2023-05-22T15:42:19.744 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.740+0000 7fca2d7cc640 1 -- 172.21.15.16:0/2160290633 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fca280cb000 con 0x7fca2815f090 2023-05-22T15:42:19.744 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.740+0000 7fca1f7fe640 1 -- 172.21.15.16:0/2160290633 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v86) v1 ==== 93+0+10 (secure 0 0 0) 0x7fca18011df0 con 0x7fca2815f090 2023-05-22T15:42:19.745 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2023-05-22T15:42:19.746 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.744+0000 7fca2d7cc640 1 -- 172.21.15.16:0/2160290633 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fca080654a0 msgr2=0x7fca08067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:19.746 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.744+0000 7fca2d7cc640 1 --2- 172.21.15.16:0/2160290633 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fca080654a0 0x7fca08067950 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fca200ad000 tx=0x7fca20001e00 comp rx=0 tx=0).stop 2023-05-22T15:42:19.747 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.744+0000 7fca2d7cc640 1 -- 172.21.15.16:0/2160290633 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fca2815f090 msgr2=0x7fca2815f530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:19.747 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.744+0000 7fca2d7cc640 1 --2- 172.21.15.16:0/2160290633 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fca2815f090 0x7fca2815f530 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7fca28071cf0 tx=0x7fca18011550 comp rx=0 tx=0).stop 2023-05-22T15:42:19.747 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.744+0000 7fca2d7cc640 1 -- 172.21.15.16:0/2160290633 shutdown_connections 2023-05-22T15:42:19.747 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.744+0000 7fca2d7cc640 1 --2- 172.21.15.16:0/2160290633 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fca2815f090 0x7fca2815f530 unknown :-1 s=CLOSED pgs=1083 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:19.747 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.744+0000 7fca2d7cc640 1 --2- 172.21.15.16:0/2160290633 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fca080654a0 0x7fca08067950 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:19.747 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.744+0000 7fca2d7cc640 1 --2- 172.21.15.16:0/2160290633 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fca281638b0 0x7fca28162690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:19.747 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.744+0000 7fca2d7cc640 1 --2- 172.21.15.16:0/2160290633 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fca14003980 0x7fca280c7b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:19.748 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.744+0000 7fca2d7cc640 1 -- 172.21.15.16:0/2160290633 >> 172.21.15.16:0/2160290633 conn(0x7fca28069720 msgr2=0x7fca2815b2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:19.748 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.744+0000 7fca2d7cc640 1 -- 172.21.15.16:0/2160290633 shutdown_connections 2023-05-22T15:42:19.748 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.744+0000 7fca2d7cc640 1 -- 172.21.15.16:0/2160290633 wait complete. 2023-05-22T15:42:19.764 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-05-22T15:42:19.895 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.892+0000 7f9565cc4640 1 Processor -- start 2023-05-22T15:42:19.895 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.892+0000 7f9565cc4640 1 -- start start 2023-05-22T15:42:19.896 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.892+0000 7f9565cc4640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f95600cee30 0x7f956006bf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:19.896 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.892+0000 7f9565cc4640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f956006db80 0x7f956006dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:19.896 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.892+0000 7f9565cc4640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f956006e4f0 con 0x7f956015c8b0 2023-05-22T15:42:19.896 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.892+0000 7f9565cc4640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f956015a0c0 con 0x7f956006db80 2023-05-22T15:42:19.896 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.892+0000 7f9565cc4640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95600ce160 con 0x7f95600cee30 2023-05-22T15:42:19.897 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.892+0000 7f955ffff640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f956006db80 0x7f956006dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:19.897 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.892+0000 7f955ffff640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f956006db80 0x7f956006dfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34638/0 (socket says 172.21.15.16:34638) 2023-05-22T15:42:19.897 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.892+0000 7f955ffff640 1 -- 172.21.15.16:0/1237809995 learned_addr learned my addr 172.21.15.16:0/1237809995 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:42:19.898 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.892+0000 7f955effd640 1 --2- 172.21.15.16:0/1237809995 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f95600cee30 0x7f956006bf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:19.898 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.892+0000 7f955ffff640 1 -- 172.21.15.16:0/1237809995 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f95600cee30 msgr2=0x7f956006bf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:42:19.898 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.892+0000 7f955ffff640 1 --2- 172.21.15.16:0/1237809995 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f95600cee30 0x7f956006bf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:19.898 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.892+0000 7f955ffff640 1 -- 172.21.15.16:0/1237809995 >> v1:172.21.15.62:6789/0 conn(0x7f956015c8b0 legacy=0x7f956015cc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:19.898 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.892+0000 7f955ffff640 1 -- 172.21.15.16:0/1237809995 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f956006c640 con 0x7f956006db80 2023-05-22T15:42:19.898 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.896+0000 7f955ffff640 1 --2- 172.21.15.16:0/1237809995 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f956006db80 0x7f956006dfb0 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7f9558009e50 tx=0x7f955801c9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cfea0dfbded3649a server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:19.899 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.896+0000 7f955e7fc640 1 -- 172.21.15.16:0/1237809995 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95580a57c0 con 0x7f956006db80 2023-05-22T15:42:19.900 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.896+0000 7f955e7fc640 1 -- 172.21.15.16:0/1237809995 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95580a5920 con 0x7f956006db80 2023-05-22T15:42:19.900 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.896+0000 7f955e7fc640 1 -- 172.21.15.16:0/1237809995 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95580a5cb0 con 0x7f956006db80 2023-05-22T15:42:19.900 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.896+0000 7f9565cc4640 1 -- 172.21.15.16:0/1237809995 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f956006db80 msgr2=0x7f956006dfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:19.900 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.896+0000 7f9565cc4640 1 --2- 172.21.15.16:0/1237809995 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f956006db80 0x7f956006dfb0 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7f9558009e50 tx=0x7f955801c9e0 comp rx=0 tx=0).stop 2023-05-22T15:42:19.900 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.896+0000 7f9565cc4640 1 -- 172.21.15.16:0/1237809995 shutdown_connections 2023-05-22T15:42:19.900 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.896+0000 7f9565cc4640 1 --2- 172.21.15.16:0/1237809995 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f956006db80 0x7f956006dfb0 unknown :-1 s=CLOSED pgs=1084 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:19.900 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.896+0000 7f9565cc4640 1 --2- 172.21.15.16:0/1237809995 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f95600cee30 0x7f956006bf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:19.901 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.896+0000 7f9565cc4640 1 -- 172.21.15.16:0/1237809995 >> 172.21.15.16:0/1237809995 conn(0x7f956006f5a0 msgr2=0x7f956006f9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:19.901 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.896+0000 7f9565cc4640 1 -- 172.21.15.16:0/1237809995 shutdown_connections 2023-05-22T15:42:19.901 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.896+0000 7f9565cc4640 1 -- 172.21.15.16:0/1237809995 wait complete. 2023-05-22T15:42:19.902 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.896+0000 7f9565cc4640 1 Processor -- start 2023-05-22T15:42:19.902 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.896+0000 7f9565cc4640 1 -- start start 2023-05-22T15:42:19.902 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.896+0000 7f9565cc4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f956006db80 0x7f9560151d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:19.903 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.896+0000 7f9565cc4640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f95600cee30 0x7f956015b4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:19.903 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.896+0000 7f9565cc4640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f956015c8b0 0x7f95601509d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:19.903 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.896+0000 7f9565cc4640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95600c8bf0 con 0x7f956006db80 2023-05-22T15:42:19.903 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.896+0000 7f9565cc4640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95600c8e00 con 0x7f956015c8b0 2023-05-22T15:42:19.903 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.896+0000 7f9565cc4640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95600c90e0 con 0x7f95600cee30 2023-05-22T15:42:19.903 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.896+0000 7f955f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f956006db80 0x7f9560151d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:19.903 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.896+0000 7f955f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f956006db80 0x7f9560151d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34650/0 (socket says 172.21.15.16:34650) 2023-05-22T15:42:19.904 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.896+0000 7f955f7fe640 1 -- 172.21.15.16:0/1094643199 learned_addr learned my addr 172.21.15.16:0/1094643199 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:42:19.904 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.900+0000 7f955ffff640 1 --2- 172.21.15.16:0/1094643199 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f956015c8b0 0x7f95601509d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:19.904 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.900+0000 7f955effd640 1 --2- 172.21.15.16:0/1094643199 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f95600cee30 0x7f956015b4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:19.904 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.900+0000 7f955f7fe640 1 -- 172.21.15.16:0/1094643199 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f95600cee30 msgr2=0x7f956015b4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:19.904 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.900+0000 7f955f7fe640 1 --2- 172.21.15.16:0/1094643199 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f95600cee30 0x7f956015b4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:19.904 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.900+0000 7f955f7fe640 1 -- 172.21.15.16:0/1094643199 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f956015c8b0 msgr2=0x7f95601509d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:19.904 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.900+0000 7f955f7fe640 1 --2- 172.21.15.16:0/1094643199 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f956015c8b0 0x7f95601509d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:19.905 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.900+0000 7f955f7fe640 1 -- 172.21.15.16:0/1094643199 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9558009b30 con 0x7f956006db80 2023-05-22T15:42:19.905 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.900+0000 7f955f7fe640 1 --2- 172.21.15.16:0/1094643199 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f956006db80 0x7f9560151d80 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7f955000c500 tx=0x7f955000c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:19.906 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.900+0000 7f9547fff640 1 -- 172.21.15.16:0/1094643199 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9550005220 con 0x7f956006db80 2023-05-22T15:42:19.906 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.900+0000 7f9547fff640 1 -- 172.21.15.16:0/1094643199 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9550005380 con 0x7f956006db80 2023-05-22T15:42:19.906 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.900+0000 7f9565cc4640 1 -- 172.21.15.16:0/1094643199 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f956015d390 con 0x7f956006db80 2023-05-22T15:42:19.906 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.900+0000 7f9547fff640 1 -- 172.21.15.16:0/1094643199 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9550007270 con 0x7f956006db80 2023-05-22T15:42:19.906 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.900+0000 7f9565cc4640 1 -- 172.21.15.16:0/1094643199 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95600ce930 con 0x7f956006db80 2023-05-22T15:42:19.906 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.900+0000 7f9565cc4640 1 -- 172.21.15.16:0/1094643199 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f956015d520 con 0x7f956006db80 2023-05-22T15:42:19.906 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.900+0000 7f9547fff640 1 -- 172.21.15.16:0/1094643199 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f95500073d0 con 0x7f956006db80 2023-05-22T15:42:19.907 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.900+0000 7f9547fff640 1 --2- 172.21.15.16:0/1094643199 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f954c0654d0 0x7f954c067980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:19.907 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.900+0000 7f9547fff640 1 -- 172.21.15.16:0/1094643199 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 5530+0+0 (secure 0 0 0) 0x7f9550082060 con 0x7f956006db80 2023-05-22T15:42:19.907 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.900+0000 7f955effd640 1 --2- 172.21.15.16:0/1094643199 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f954c0654d0 0x7f954c067980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:19.907 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.900+0000 7f955effd640 1 --2- 172.21.15.16:0/1094643199 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f954c0654d0 0x7f954c067980 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f95600afc30 tx=0x7f9554008040 comp rx=0 tx=0).ready entity=mgr.6765 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:19.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:19.904+0000 7f9547fff640 1 -- 172.21.15.16:0/1094643199 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f955004fdf0 con 0x7f956006db80 2023-05-22T15:42:20.062 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:20.060+0000 7f9565cc4640 1 -- 172.21.15.16:0/1094643199 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f956004dbe0 con 0x7f956006db80 2023-05-22T15:42:20.063 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:20.060+0000 7f9547fff640 1 -- 172.21.15.16:0/1094643199 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v86) v1 ==== 92+0+10 (secure 0 0 0) 0x7f9550053410 con 0x7f956006db80 2023-05-22T15:42:20.064 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 8 2023-05-22T15:42:20.066 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:20.060+0000 7f9565cc4640 1 -- 172.21.15.16:0/1094643199 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f954c0654d0 msgr2=0x7f954c067980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:20.067 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:20.060+0000 7f9565cc4640 1 --2- 172.21.15.16:0/1094643199 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f954c0654d0 0x7f954c067980 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f95600afc30 tx=0x7f9554008040 comp rx=0 tx=0).stop 2023-05-22T15:42:20.067 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:20.060+0000 7f9565cc4640 1 -- 172.21.15.16:0/1094643199 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f956006db80 msgr2=0x7f9560151d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:20.067 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:20.060+0000 7f9565cc4640 1 --2- 172.21.15.16:0/1094643199 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f956006db80 0x7f9560151d80 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7f955000c500 tx=0x7f955000c9c0 comp rx=0 tx=0).stop 2023-05-22T15:42:20.067 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:20.060+0000 7f9565cc4640 1 -- 172.21.15.16:0/1094643199 shutdown_connections 2023-05-22T15:42:20.068 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:20.060+0000 7f9565cc4640 1 --2- 172.21.15.16:0/1094643199 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f956015c8b0 0x7f95601509d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:20.068 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:20.060+0000 7f9565cc4640 1 --2- 172.21.15.16:0/1094643199 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f954c0654d0 0x7f954c067980 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:20.068 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:20.060+0000 7f9565cc4640 1 --2- 172.21.15.16:0/1094643199 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f95600cee30 0x7f956015b4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:20.069 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:20.060+0000 7f9565cc4640 1 --2- 172.21.15.16:0/1094643199 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f956006db80 0x7f9560151d80 unknown :-1 s=CLOSED pgs=1085 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:20.069 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:20.060+0000 7f9565cc4640 1 -- 172.21.15.16:0/1094643199 >> 172.21.15.16:0/1094643199 conn(0x7f956006f5a0 msgr2=0x7f95600ccdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:20.069 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:20.060+0000 7f9565cc4640 1 -- 172.21.15.16:0/1094643199 shutdown_connections 2023-05-22T15:42:20.070 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:20.060+0000 7f9565cc4640 1 -- 172.21.15.16:0/1094643199 wait complete. 2023-05-22T15:42:20.083 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-05-22T15:42:20.210 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:20.204+0000 7fd3e6de3640 1 Processor -- start 2023-05-22T15:42:20.210 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:20.204+0000 7fd3e6de3640 1 -- start start 2023-05-22T15:42:20.211 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:20.208+0000 7fd3e6de3640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fd3e00c9010 0x7fd3e015d260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:20.212 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:20.208+0000 7fd3e6de3640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd3e006c8c0 0x7fd3e006ccf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:20.212 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:20.208+0000 7fd3e6de3640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3e015d7a0 con 0x7fd3e0158c40 2023-05-22T15:42:20.212 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:20.208+0000 7fd3e6de3640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3e006d230 con 0x7fd3e006c8c0 2023-05-22T15:42:20.212 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:20.208+0000 7fd3e6de3640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3e00c9640 con 0x7fd3e00c9010 2023-05-22T15:42:20.212 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:20.208+0000 7fd3e4b7f640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fd3e0158c40 0x7fd3e006fa00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:33428/0 (socket says 172.21.15.16:33428) 2023-05-22T15:42:20.213 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:20.208+0000 7fd3e4b7f640 1 -- 172.21.15.16:0/2393330685 learned_addr learned my addr 172.21.15.16:0/2393330685 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:42:20.213 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:20.208+0000 7fd3e5380640 1 --2- 172.21.15.16:0/2393330685 >> v2:172.21.15.16:3300/0 conn(0x7fd3e006c8c0 0x7fd3e006ccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:20.213 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:20.208+0000 7fd3dffff640 1 --2- 172.21.15.16:0/2393330685 >> v2:172.21.15.62:3300/0 conn(0x7fd3e00c9010 0x7fd3e015d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:20.213 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:20.208+0000 7fd3df7fe640 1 -- 172.21.15.16:0/2393330685 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2260770872 0 0) 0x7fd3d0001120 con 0x7fd3e0158c40 2023-05-22T15:42:20.213 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:20.208+0000 7fd3df7fe640 1 -- 172.21.15.16:0/2393330685 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd3c8003530 con 0x7fd3e0158c40 2023-05-22T15:42:20.213 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:20.208+0000 7fd3e5380640 1 -- 172.21.15.16:0/2393330685 >> v2:172.21.15.62:3300/0 conn(0x7fd3e00c9010 msgr2=0x7fd3e015d260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:42:20.213 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:20.208+0000 7fd3e5380640 1 --2- 172.21.15.16:0/2393330685 >> v2:172.21.15.62:3300/0 conn(0x7fd3e00c9010 0x7fd3e015d260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:20.213 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:20.208+0000 7fd3e5380640 1 -- 172.21.15.16:0/2393330685 >> v1:172.21.15.16:6789/0 conn(0x7fd3e0158c40 legacy=0x7fd3e006fa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:20.214 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:20.208+0000 7fd3e5380640 1 -- 172.21.15.16:0/2393330685 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3e00cc420 con 0x7fd3e006c8c0 2023-05-22T15:42:20.214 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:20.208+0000 7fd3e5380640 1 --2- 172.21.15.16:0/2393330685 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd3e006c8c0 0x7fd3e006ccf0 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7fd3e006d480 tx=0x7fd3d801f0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50a535d2da4de150 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:20.215 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:20.208+0000 7fd3df7fe640 1 -- 172.21.15.16:0/2393330685 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3d80b0440 con 0x7fd3e006c8c0 2023-05-22T15:42:20.215 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:20.208+0000 7fd3df7fe640 1 -- 172.21.15.16:0/2393330685 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3d80b05a0 con 0x7fd3e006c8c0 2023-05-22T15:42:20.215 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:20.212+0000 7fd3df7fe640 1 -- 172.21.15.16:0/2393330685 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3d80b0930 con 0x7fd3e006c8c0 2023-05-22T15:42:20.215 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:20.212+0000 7fd3e6de3640 1 -- 172.21.15.16:0/2393330685 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd3e006c8c0 msgr2=0x7fd3e006ccf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:20.215 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:20.212+0000 7fd3e6de3640 1 --2- 172.21.15.16:0/2393330685 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd3e006c8c0 0x7fd3e006ccf0 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7fd3e006d480 tx=0x7fd3d801f0f0 comp rx=0 tx=0).stop 2023-05-22T15:42:20.215 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:20.212+0000 7fd3e6de3640 1 -- 172.21.15.16:0/2393330685 shutdown_connections 2023-05-22T15:42:20.216 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:20.212+0000 7fd3e6de3640 1 --2- 172.21.15.16:0/2393330685 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd3e006c8c0 0x7fd3e006ccf0 unknown :-1 s=CLOSED pgs=1087 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:20.216 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:20.212+0000 7fd3e6de3640 1 --2- 172.21.15.16:0/2393330685 >> v2:172.21.15.62:3300/0 conn(0x7fd3e00c9010 0x7fd3e015d260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:20.216 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:20.212+0000 7fd3e6de3640 1 -- 172.21.15.16:0/2393330685 >> 172.21.15.16:0/2393330685 conn(0x7fd3e00ce040 msgr2=0x7fd3e00ce440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:20.216 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:20.212+0000 7fd3e6de3640 1 -- 172.21.15.16:0/2393330685 shutdown_connections 2023-05-22T15:42:20.216 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:20.212+0000 7fd3e6de3640 1 -- 172.21.15.16:0/2393330685 wait complete. 2023-05-22T15:42:20.218 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:20.212+0000 7fd3e6de3640 1 Processor -- start 2023-05-22T15:42:20.218 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:20.212+0000 7fd3e6de3640 1 -- start start 2023-05-22T15:42:20.218 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:20.212+0000 7fd3e6de3640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd3e006c8c0 0x7fd3e014e150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:20.219 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:20.212+0000 7fd3e6de3640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd3e00c9010 0x7fd3e00b0e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:20.219 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:20.212+0000 7fd3e6de3640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd3e0158c40 0x7fd3e015a060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:20.219 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:20.212+0000 7fd3e6de3640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3e015b070 con 0x7fd3e0158c40 2023-05-22T15:42:20.219 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:20.212+0000 7fd3e6de3640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3e015aab0 con 0x7fd3e00c9010 2023-05-22T15:42:20.219 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:20.212+0000 7fd3e6de3640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3e015ad90 con 0x7fd3e006c8c0 2023-05-22T15:42:20.219 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:20.212+0000 7fd3e5380640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd3e0158c40 0x7fd3e015a060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:20.220 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:20.212+0000 7fd3e5380640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd3e0158c40 0x7fd3e015a060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34664/0 (socket says 172.21.15.16:34664) 2023-05-22T15:42:20.220 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:20.212+0000 7fd3e5380640 1 -- 172.21.15.16:0/2344770001 learned_addr learned my addr 172.21.15.16:0/2344770001 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:42:20.220 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:20.212+0000 7fd3e4b7f640 1 --2- 172.21.15.16:0/2344770001 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd3e006c8c0 0x7fd3e014e150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:20.220 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:20.212+0000 7fd3dffff640 1 --2- 172.21.15.16:0/2344770001 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd3e00c9010 0x7fd3e00b0e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:20.220 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:20.212+0000 7fd3e5380640 1 -- 172.21.15.16:0/2344770001 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd3e006c8c0 msgr2=0x7fd3e014e150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:20.220 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:20.212+0000 7fd3e5380640 1 --2- 172.21.15.16:0/2344770001 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd3e006c8c0 0x7fd3e014e150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:20.220 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:20.212+0000 7fd3e5380640 1 -- 172.21.15.16:0/2344770001 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd3e00c9010 msgr2=0x7fd3e00b0e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:20.220 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:20.212+0000 7fd3e5380640 1 --2- 172.21.15.16:0/2344770001 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd3e00c9010 0x7fd3e00b0e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:20.221 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:20.212+0000 7fd3e5380640 1 -- 172.21.15.16:0/2344770001 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3d8009a70 con 0x7fd3e0158c40 2023-05-22T15:42:20.221 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:20.212+0000 7fd3dffff640 1 --2- 172.21.15.16:0/2344770001 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd3e00c9010 0x7fd3e00b0e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:42:20.221 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:20.212+0000 7fd3e4b7f640 1 --2- 172.21.15.16:0/2344770001 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd3e006c8c0 0x7fd3e014e150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:42:20.221 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:20.212+0000 7fd3e5380640 1 --2- 172.21.15.16:0/2344770001 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd3e0158c40 0x7fd3e015a060 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7fd3d801bd00 tx=0x7fd3d801be90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:20.221 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:20.212+0000 7fd3dd7fa640 1 -- 172.21.15.16:0/2344770001 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3d80b0440 con 0x7fd3e0158c40 2023-05-22T15:42:20.221 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:20.212+0000 7fd3dd7fa640 1 -- 172.21.15.16:0/2344770001 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3d80bc860 con 0x7fd3e0158c40 2023-05-22T15:42:20.221 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:20.212+0000 7fd3e6de3640 1 -- 172.21.15.16:0/2344770001 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3e00cc9e0 con 0x7fd3e0158c40 2023-05-22T15:42:20.222 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:20.212+0000 7fd3dd7fa640 1 -- 172.21.15.16:0/2344770001 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3d80bcc10 con 0x7fd3e0158c40 2023-05-22T15:42:20.222 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:20.212+0000 7fd3e6de3640 1 -- 172.21.15.16:0/2344770001 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3e00cc700 con 0x7fd3e0158c40 2023-05-22T15:42:20.222 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:20.216+0000 7fd3e6de3640 1 -- 172.21.15.16:0/2344770001 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3e00c9920 con 0x7fd3e0158c40 2023-05-22T15:42:20.222 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:20.216+0000 7fd3dd7fa640 1 -- 172.21.15.16:0/2344770001 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) v1 ==== 84332+0+0 (secure 0 0 0) 0x7fd3d80ba400 con 0x7fd3e0158c40 2023-05-22T15:42:20.223 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:20.216+0000 7fd3dd7fa640 1 --2- 172.21.15.16:0/2344770001 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd3cc0654a0 0x7fd3cc067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:20.223 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:20.216+0000 7fd3dd7fa640 1 -- 172.21.15.16:0/2344770001 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 5530+0+0 (secure 0 0 0) 0x7fd3d8121500 con 0x7fd3e0158c40 2023-05-22T15:42:20.223 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:20.216+0000 7fd3e4b7f640 1 --2- 172.21.15.16:0/2344770001 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd3cc0654a0 0x7fd3cc067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:20.223 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:20.216+0000 7fd3e4b7f640 1 --2- 172.21.15.16:0/2344770001 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd3cc0654a0 0x7fd3cc067950 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fd3d0007fd0 tx=0x7fd3d0009040 comp rx=0 tx=0).ready entity=mgr.6765 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:20.223 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:20.220+0000 7fd3dd7fa640 1 -- 172.21.15.16:0/2344770001 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fd3d80ef210 con 0x7fd3e0158c40 2023-05-22T15:42:20.375 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:20.372+0000 7fd3e6de3640 1 -- 172.21.15.16:0/2344770001 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fd3e00738f0 con 0x7fd3e0158c40 2023-05-22T15:42:20.375 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:20.372+0000 7fd3dd7fa640 1 -- 172.21.15.16:0/2344770001 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v86) v1 ==== 84+0+126 (secure 0 0 0) 0x7fd3d811fe60 con 0x7fd3e0158c40 2023-05-22T15:42:20.376 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:42:20.376 INFO:teuthology.orchestra.run.smithi016.stdout:[ 2023-05-22T15:42:20.376 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T15:42:20.376 INFO:teuthology.orchestra.run.smithi016.stdout: "poolnum": 1, 2023-05-22T15:42:20.376 INFO:teuthology.orchestra.run.smithi016.stdout: "poolname": ".mgr" 2023-05-22T15:42:20.377 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:42:20.377 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T15:42:20.377 INFO:teuthology.orchestra.run.smithi016.stdout: "poolnum": 2, 2023-05-22T15:42:20.377 INFO:teuthology.orchestra.run.smithi016.stdout: "poolname": "rbd" 2023-05-22T15:42:20.377 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T15:42:20.377 INFO:teuthology.orchestra.run.smithi016.stdout:] 2023-05-22T15:42:20.378 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:20.372+0000 7fd3e6de3640 1 -- 172.21.15.16:0/2344770001 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd3cc0654a0 msgr2=0x7fd3cc067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:20.378 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:20.372+0000 7fd3e6de3640 1 --2- 172.21.15.16:0/2344770001 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd3cc0654a0 0x7fd3cc067950 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fd3d0007fd0 tx=0x7fd3d0009040 comp rx=0 tx=0).stop 2023-05-22T15:42:20.379 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:20.372+0000 7fd3e6de3640 1 -- 172.21.15.16:0/2344770001 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd3e0158c40 msgr2=0x7fd3e015a060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:20.379 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:20.372+0000 7fd3e6de3640 1 --2- 172.21.15.16:0/2344770001 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd3e0158c40 0x7fd3e015a060 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7fd3d801bd00 tx=0x7fd3d801be90 comp rx=0 tx=0).stop 2023-05-22T15:42:20.379 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:20.372+0000 7fd3e6de3640 1 -- 172.21.15.16:0/2344770001 shutdown_connections 2023-05-22T15:42:20.379 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:20.372+0000 7fd3e6de3640 1 --2- 172.21.15.16:0/2344770001 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd3e0158c40 0x7fd3e015a060 unknown :-1 s=CLOSED pgs=1088 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:20.379 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:20.372+0000 7fd3e6de3640 1 --2- 172.21.15.16:0/2344770001 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd3e00c9010 0x7fd3e00b0e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:20.379 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:20.372+0000 7fd3e6de3640 1 --2- 172.21.15.16:0/2344770001 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd3cc0654a0 0x7fd3cc067950 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:20.379 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:20.372+0000 7fd3e6de3640 1 --2- 172.21.15.16:0/2344770001 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd3e006c8c0 0x7fd3e014e150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:20.380 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:20.372+0000 7fd3e6de3640 1 -- 172.21.15.16:0/2344770001 >> 172.21.15.16:0/2344770001 conn(0x7fd3e00ce040 msgr2=0x7fd3e0157bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:20.380 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:20.372+0000 7fd3e6de3640 1 -- 172.21.15.16:0/2344770001 shutdown_connections 2023-05-22T15:42:20.380 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:20.372+0000 7fd3e6de3640 1 -- 172.21.15.16:0/2344770001 wait complete. 2023-05-22T15:42:20.393 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-05-22T15:42:20.394 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-05-22T15:42:20.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:20.520+0000 7f2f25244640 1 Processor -- start 2023-05-22T15:42:20.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:20.520+0000 7f2f25244640 1 -- start start 2023-05-22T15:42:20.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:20.520+0000 7f2f25244640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2f200ce530 0x7f2f2015c540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:20.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:20.520+0000 7f2f25244640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f200c3930 con 0x7f2f200cbe80 2023-05-22T15:42:20.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:20.520+0000 7f2f25244640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f20158f00 con 0x7f2f200ce900 2023-05-22T15:42:20.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:20.520+0000 7f2f25244640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f20159d90 con 0x7f2f200ce530 2023-05-22T15:42:20.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:20.520+0000 7f2f1f59e640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f2f200cbe80 0x7f2f200cc680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:33442/0 (socket says 172.21.15.16:33442) 2023-05-22T15:42:20.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:20.520+0000 7f2f1f59e640 1 -- 172.21.15.16:0/1235491000 learned_addr learned my addr 172.21.15.16:0/1235491000 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:42:20.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:20.520+0000 7f2f1ed9d640 1 --2- 172.21.15.16:0/1235491000 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2f200ce530 0x7f2f2015c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:20.526 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:20.520+0000 7f2f1e59c640 1 -- 172.21.15.16:0/1235491000 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1273373309 0 0) 0x7f2f180014a0 con 0x7f2f200cbe80 2023-05-22T15:42:20.526 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:20.520+0000 7f2f1e59c640 1 -- 172.21.15.16:0/1235491000 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f0c003530 con 0x7f2f200cbe80 2023-05-22T15:42:20.526 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:20.520+0000 7f2f1e59c640 1 -- 172.21.15.16:0/1235491000 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 928702812 0 0) 0x7f2f0c003530 con 0x7f2f200cbe80 2023-05-22T15:42:20.526 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:20.520+0000 7f2f1e59c640 1 -- 172.21.15.16:0/1235491000 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2f200ce530 msgr2=0x7f2f2015c540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:42:20.526 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:20.520+0000 7f2f1e59c640 1 --2- 172.21.15.16:0/1235491000 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2f200ce530 0x7f2f2015c540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:20.526 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:20.520+0000 7f2f1e59c640 1 -- 172.21.15.16:0/1235491000 >> v1:172.21.15.62:6789/0 conn(0x7f2f200ce900 legacy=0x7f2f20159680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:20.526 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:20.520+0000 7f2f1e59c640 1 -- 172.21.15.16:0/1235491000 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f2015cc40 con 0x7f2f200cbe80 2023-05-22T15:42:20.526 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:20.520+0000 7f2f1e59c640 1 -- 172.21.15.16:0/1235491000 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7f2f18001b90 con 0x7f2f200cbe80 2023-05-22T15:42:20.527 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:20.520+0000 7f2f1e59c640 1 -- 172.21.15.16:0/1235491000 >> v1:172.21.15.16:6789/0 conn(0x7f2f200cbe80 legacy=0x7f2f200cc680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:20.527 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:20.520+0000 7f2f1e59c640 1 --2- 172.21.15.16:0/1235491000 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2f200ce530 0x7f2f0c0039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:20.527 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:20.520+0000 7f2f1e59c640 1 --2- 172.21.15.16:0/1235491000 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2f0c003fe0 0x7f2f0c0047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:20.527 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:20.520+0000 7f2f1e59c640 1 --2- 172.21.15.16:0/1235491000 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f0c004d10 0x7f2f0c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:20.527 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:20.520+0000 7f2f1e59c640 1 -- 172.21.15.16:0/1235491000 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f20159d90 con 0x7f2f0c004d10 2023-05-22T15:42:20.527 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:20.520+0000 7f2f1e59c640 1 -- 172.21.15.16:0/1235491000 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f0c00bb20 con 0x7f2f0c003fe0 2023-05-22T15:42:20.527 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:20.520+0000 7f2f1e59c640 1 -- 172.21.15.16:0/1235491000 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f0c00bce0 con 0x7f2f200ce530 2023-05-22T15:42:20.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:20.520+0000 7f2f1f59e640 1 --2- 172.21.15.16:0/1235491000 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f0c004d10 0x7f2f0c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:20.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:20.520+0000 7f2f1659c640 1 --2- 172.21.15.16:0/1235491000 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2f0c003fe0 0x7f2f0c0047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:20.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:20.520+0000 7f2f1ed9d640 1 --2- 172.21.15.16:0/1235491000 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2f200ce530 0x7f2f0c0039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:20.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:20.520+0000 7f2f1f59e640 1 -- 172.21.15.16:0/1235491000 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2f200ce530 msgr2=0x7f2f0c0039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:42:20.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:20.520+0000 7f2f1f59e640 1 --2- 172.21.15.16:0/1235491000 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2f200ce530 0x7f2f0c0039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:20.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:20.520+0000 7f2f1f59e640 1 -- 172.21.15.16:0/1235491000 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2f0c003fe0 msgr2=0x7f2f0c0047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:42:20.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:20.520+0000 7f2f1f59e640 1 --2- 172.21.15.16:0/1235491000 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2f0c003fe0 0x7f2f0c0047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:20.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:20.520+0000 7f2f1f59e640 1 -- 172.21.15.16:0/1235491000 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f0c003530 con 0x7f2f0c004d10 2023-05-22T15:42:20.531 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:20.524+0000 7f2f1f59e640 1 --2- 172.21.15.16:0/1235491000 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f0c004d10 0x7f2f0c005180 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7f2f1800a830 tx=0x7f2f1801d340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4508d5bb7a26b47 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:20.531 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:20.524+0000 7f2f1e59c640 1 -- 172.21.15.16:0/1235491000 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f180ad630 con 0x7f2f0c004d10 2023-05-22T15:42:20.531 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:20.524+0000 7f2f1e59c640 1 -- 172.21.15.16:0/1235491000 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f180a8070 con 0x7f2f0c004d10 2023-05-22T15:42:20.531 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:20.524+0000 7f2f1e59c640 1 -- 172.21.15.16:0/1235491000 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f180ada30 con 0x7f2f0c004d10 2023-05-22T15:42:20.532 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:20.524+0000 7f2f25244640 1 -- 172.21.15.16:0/1235491000 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f0c004d10 msgr2=0x7f2f0c005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:20.532 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:20.524+0000 7f2f25244640 1 --2- 172.21.15.16:0/1235491000 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f0c004d10 0x7f2f0c005180 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7f2f1800a830 tx=0x7f2f1801d340 comp rx=0 tx=0).stop 2023-05-22T15:42:20.532 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:20.524+0000 7f2f1f59e640 1 -- 172.21.15.16:0/1235491000 reap_dead start 2023-05-22T15:42:20.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:20.524+0000 7f2f25244640 1 -- 172.21.15.16:0/1235491000 shutdown_connections 2023-05-22T15:42:20.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:20.524+0000 7f2f25244640 1 -- 172.21.15.16:0/1235491000 >> 172.21.15.16:0/1235491000 conn(0x7f2f20069720 msgr2=0x7f2f20069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:20.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:20.524+0000 7f2f25244640 1 -- 172.21.15.16:0/1235491000 shutdown_connections 2023-05-22T15:42:20.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:20.524+0000 7f2f25244640 1 -- 172.21.15.16:0/1235491000 wait complete. 2023-05-22T15:42:20.534 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:20.524+0000 7f2f25244640 1 Processor -- start 2023-05-22T15:42:20.534 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:20.524+0000 7f2f25244640 1 -- start start 2023-05-22T15:42:20.534 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:20.524+0000 7f2f25244640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2f0c004d10 0x7f2f200b0fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:20.534 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:20.524+0000 7f2f25244640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2f200c53d0 0x7f2f2014d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:20.535 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:20.524+0000 7f2f25244640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f200c5840 0x7f2f2014dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:20.535 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:20.524+0000 7f2f25244640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f2014e210 con 0x7f2f200c5840 2023-05-22T15:42:20.535 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:20.524+0000 7f2f25244640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f200b23a0 con 0x7f2f200c53d0 2023-05-22T15:42:20.535 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:20.524+0000 7f2f25244640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f200b14f0 con 0x7f2f0c004d10 2023-05-22T15:42:20.536 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:20.524+0000 7f2f1f59e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f200c5840 0x7f2f2014dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:20.536 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:20.524+0000 7f2f1f59e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f200c5840 0x7f2f2014dcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34682/0 (socket says 172.21.15.16:34682) 2023-05-22T15:42:20.536 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:20.524+0000 7f2f1f59e640 1 -- 172.21.15.16:0/3940259999 learned_addr learned my addr 172.21.15.16:0/3940259999 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:42:20.537 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:20.524+0000 7f2f1ed9d640 1 --2- 172.21.15.16:0/3940259999 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2f0c004d10 0x7f2f200b0fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:20.537 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:20.524+0000 7f2f1659c640 1 --2- 172.21.15.16:0/3940259999 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2f200c53d0 0x7f2f2014d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:20.537 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:20.524+0000 7f2f1f59e640 1 -- 172.21.15.16:0/3940259999 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2f0c004d10 msgr2=0x7f2f200b0fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:20.537 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:20.524+0000 7f2f1f59e640 1 --2- 172.21.15.16:0/3940259999 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2f0c004d10 0x7f2f200b0fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:20.537 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:20.524+0000 7f2f1f59e640 1 -- 172.21.15.16:0/3940259999 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2f200c53d0 msgr2=0x7f2f2014d790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:20.538 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:20.524+0000 7f2f1f59e640 1 --2- 172.21.15.16:0/3940259999 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2f200c53d0 0x7f2f2014d790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:20.538 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:20.524+0000 7f2f1f59e640 1 -- 172.21.15.16:0/3940259999 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f2015d200 con 0x7f2f200c5840 2023-05-22T15:42:20.538 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:20.524+0000 7f2f1659c640 1 --2- 172.21.15.16:0/3940259999 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2f200c53d0 0x7f2f2014d790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:42:20.538 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:20.528+0000 7f2f1f59e640 1 --2- 172.21.15.16:0/3940259999 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f200c5840 0x7f2f2014dcd0 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 crypto rx=0x7f2f1801cdb0 tx=0x7f2f0c005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:20.538 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:20.528+0000 7f2f17fff640 1 -- 172.21.15.16:0/3940259999 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f18001b90 con 0x7f2f200c5840 2023-05-22T15:42:20.538 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:20.528+0000 7f2f17fff640 1 -- 172.21.15.16:0/3940259999 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f180014a0 con 0x7f2f200c5840 2023-05-22T15:42:20.538 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:20.528+0000 7f2f25244640 1 -- 172.21.15.16:0/3940259999 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f2015cf20 con 0x7f2f200c5840 2023-05-22T15:42:20.538 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:20.528+0000 7f2f17fff640 1 -- 172.21.15.16:0/3940259999 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f18004d70 con 0x7f2f200c5840 2023-05-22T15:42:20.539 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:20.528+0000 7f2f25244640 1 -- 172.21.15.16:0/3940259999 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f20159160 con 0x7f2f200c5840 2023-05-22T15:42:20.539 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:20.528+0000 7f2f25244640 1 -- 172.21.15.16:0/3940259999 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f200ada70 con 0x7f2f200c5840 2023-05-22T15:42:20.540 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:20.528+0000 7f2f17fff640 1 -- 172.21.15.16:0/3940259999 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f2f180aa030 con 0x7f2f200c5840 2023-05-22T15:42:20.540 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:20.528+0000 7f2f17fff640 1 --2- 172.21.15.16:0/3940259999 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f2f000654d0 0x7f2f00067980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:20.540 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:20.528+0000 7f2f17fff640 1 -- 172.21.15.16:0/3940259999 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 5530+0+0 (secure 0 0 0) 0x7f2f1811cc30 con 0x7f2f200c5840 2023-05-22T15:42:20.540 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:20.528+0000 7f2f1ed9d640 1 --2- 172.21.15.16:0/3940259999 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f2f000654d0 0x7f2f00067980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:20.540 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:20.528+0000 7f2f1ed9d640 1 --2- 172.21.15.16:0/3940259999 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f2f000654d0 0x7f2f00067980 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f2f200c4ad0 tx=0x7f2f10008040 comp rx=0 tx=0).ready entity=mgr.6765 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:20.540 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:20.532+0000 7f2f17fff640 1 -- 172.21.15.16:0/3940259999 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f2f180ea810 con 0x7f2f200c5840 2023-05-22T15:42:20.689 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:20.684+0000 7f2f25244640 1 -- 172.21.15.16:0/3940259999 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f2f200c3930 con 0x7f2f200c5840 2023-05-22T15:42:21.432 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:21.428+0000 7f2f17fff640 1 -- 172.21.15.16:0/3940259999 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v87) v1 ==== 138+0+0 (secure 0 0 0) 0x7f2f200c3930 con 0x7f2f200c5840 2023-05-22T15:42:21.432 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'cephfs_metadata' created 2023-05-22T15:42:21.437 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:21.432+0000 7f2f25244640 1 -- 172.21.15.16:0/3940259999 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f2f000654d0 msgr2=0x7f2f00067980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:21.437 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:21.432+0000 7f2f25244640 1 --2- 172.21.15.16:0/3940259999 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f2f000654d0 0x7f2f00067980 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f2f200c4ad0 tx=0x7f2f10008040 comp rx=0 tx=0).stop 2023-05-22T15:42:21.437 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:21.432+0000 7f2f25244640 1 -- 172.21.15.16:0/3940259999 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f200c5840 msgr2=0x7f2f2014dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:21.437 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:21.432+0000 7f2f25244640 1 --2- 172.21.15.16:0/3940259999 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f200c5840 0x7f2f2014dcd0 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 crypto rx=0x7f2f1801cdb0 tx=0x7f2f0c005800 comp rx=0 tx=0).stop 2023-05-22T15:42:21.438 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:21.436+0000 7f2f25244640 1 -- 172.21.15.16:0/3940259999 shutdown_connections 2023-05-22T15:42:21.438 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:21.436+0000 7f2f25244640 1 --2- 172.21.15.16:0/3940259999 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f200c5840 0x7f2f2014dcd0 unknown :-1 s=CLOSED pgs=1091 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:21.438 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:21.436+0000 7f2f25244640 1 --2- 172.21.15.16:0/3940259999 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2f200c53d0 0x7f2f2014d790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:21.438 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:21.436+0000 7f2f25244640 1 --2- 172.21.15.16:0/3940259999 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f2f000654d0 0x7f2f00067980 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:21.438 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:21.436+0000 7f2f25244640 1 --2- 172.21.15.16:0/3940259999 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2f0c004d10 0x7f2f200b0fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:21.439 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:21.436+0000 7f2f25244640 1 -- 172.21.15.16:0/3940259999 >> 172.21.15.16:0/3940259999 conn(0x7f2f20069720 msgr2=0x7f2f200c3b50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:21.439 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:21.436+0000 7f2f25244640 1 -- 172.21.15.16:0/3940259999 shutdown_connections 2023-05-22T15:42:21.439 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:21.436+0000 7f2f25244640 1 -- 172.21.15.16:0/3940259999 wait complete. 2023-05-22T15:42:21.459 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-05-22T15:42:21.595 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:21.592+0000 7f54703e6640 1 Processor -- start 2023-05-22T15:42:21.595 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:21.592+0000 7f54703e6640 1 -- start start 2023-05-22T15:42:21.596 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:21.592+0000 7f54703e6640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f54680ce480 0x7f5468071640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:21.596 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:21.592+0000 7f54703e6640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f546815cde0 con 0x7f546815c270 2023-05-22T15:42:21.596 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:21.592+0000 7f54703e6640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5468069ae0 con 0x7f54680ce850 2023-05-22T15:42:21.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:21.592+0000 7f54703e6640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f546815f990 con 0x7f54680ce480 2023-05-22T15:42:21.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:21.592+0000 7f546e983640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f546815c270 0x7f546815c6d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:33444/0 (socket says 172.21.15.16:33444) 2023-05-22T15:42:21.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:21.592+0000 7f546e983640 1 -- 172.21.15.16:0/781597744 learned_addr learned my addr 172.21.15.16:0/781597744 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:42:21.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:21.592+0000 7f546e182640 1 --2- 172.21.15.16:0/781597744 >> v2:172.21.15.62:3300/0 conn(0x7f54680ce480 0x7f5468071640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:21.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:21.592+0000 7f546d180640 1 -- 172.21.15.16:0/781597744 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2594716929 0 0) 0x7f546815cde0 con 0x7f546815c270 2023-05-22T15:42:21.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:21.592+0000 7f546d180640 1 -- 172.21.15.16:0/781597744 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5454003530 con 0x7f546815c270 2023-05-22T15:42:21.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:21.592+0000 7f546d180640 1 -- 172.21.15.16:0/781597744 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 370784714 0 0) 0x7f5454003530 con 0x7f546815c270 2023-05-22T15:42:21.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:21.592+0000 7f546d180640 1 -- 172.21.15.16:0/781597744 >> v2:172.21.15.62:3300/0 conn(0x7f54680ce480 msgr2=0x7f5468071640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:42:21.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:21.592+0000 7f546d180640 1 --2- 172.21.15.16:0/781597744 >> v2:172.21.15.62:3300/0 conn(0x7f54680ce480 0x7f5468071640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:21.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:21.592+0000 7f546d180640 1 -- 172.21.15.16:0/781597744 >> v1:172.21.15.62:6789/0 conn(0x7f54680ce850 legacy=0x7f5468160090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:21.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:21.592+0000 7f546d180640 1 -- 172.21.15.16:0/781597744 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5468069dc0 con 0x7f546815c270 2023-05-22T15:42:21.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:21.592+0000 7f546d180640 1 -- 172.21.15.16:0/781597744 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7f5460001a00 con 0x7f546815c270 2023-05-22T15:42:21.600 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:21.592+0000 7f546d180640 1 -- 172.21.15.16:0/781597744 >> v1:172.21.15.16:6789/0 conn(0x7f546815c270 legacy=0x7f546815c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:21.600 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:21.592+0000 7f546d180640 1 --2- 172.21.15.16:0/781597744 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5454003980 0x7f5454003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:21.600 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:21.592+0000 7f546d180640 1 --2- 172.21.15.16:0/781597744 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f54540047d0 0x7f5454004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:21.600 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:21.592+0000 7f546d180640 1 --2- 172.21.15.16:0/781597744 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5454005180 0x7f545400b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:21.600 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:21.592+0000 7f546d180640 1 -- 172.21.15.16:0/781597744 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f546815f990 con 0x7f5454005180 2023-05-22T15:42:21.600 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:21.592+0000 7f546d180640 1 -- 172.21.15.16:0/781597744 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f545400c000 con 0x7f54540047d0 2023-05-22T15:42:21.600 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:21.592+0000 7f546d180640 1 -- 172.21.15.16:0/781597744 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f545400c1c0 con 0x7f5454003980 2023-05-22T15:42:21.601 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:21.592+0000 7f546e983640 1 --2- 172.21.15.16:0/781597744 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5454005180 0x7f545400b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:21.601 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:21.592+0000 7f546e182640 1 --2- 172.21.15.16:0/781597744 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5454003980 0x7f5454003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:21.601 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:21.592+0000 7f546d981640 1 --2- 172.21.15.16:0/781597744 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f54540047d0 0x7f5454004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:21.601 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:21.596+0000 7f546e983640 1 -- 172.21.15.16:0/781597744 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5454003980 msgr2=0x7f5454003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:42:21.601 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:21.596+0000 7f546e983640 1 --2- 172.21.15.16:0/781597744 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5454003980 0x7f5454003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:21.601 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:21.596+0000 7f546e983640 1 -- 172.21.15.16:0/781597744 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f54540047d0 msgr2=0x7f5454004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:42:21.601 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:21.596+0000 7f546e983640 1 --2- 172.21.15.16:0/781597744 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f54540047d0 0x7f5454004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:21.601 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:21.596+0000 7f546e983640 1 -- 172.21.15.16:0/781597744 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5454003530 con 0x7f5454005180 2023-05-22T15:42:21.602 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:21.596+0000 7f546d981640 1 --2- 172.21.15.16:0/781597744 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f54540047d0 0x7f5454004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T15:42:21.604 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:21.596+0000 7f546e983640 1 -- 172.21.15.16:0/781597744 reap_dead start 2023-05-22T15:42:21.604 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:21.596+0000 7f546e983640 1 --2- 172.21.15.16:0/781597744 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5454005180 0x7f545400b990 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7f546000a830 tx=0x7f546001d340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4700f68004a78ee server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:21.604 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:21.596+0000 7f546d180640 1 -- 172.21.15.16:0/781597744 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54600072d0 con 0x7f5454005180 2023-05-22T15:42:21.604 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:21.596+0000 7f546d180640 1 -- 172.21.15.16:0/781597744 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54600a8070 con 0x7f5454005180 2023-05-22T15:42:21.604 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:21.596+0000 7f546d180640 1 -- 172.21.15.16:0/781597744 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5460007690 con 0x7f5454005180 2023-05-22T15:42:21.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:21.596+0000 7f54703e6640 1 -- 172.21.15.16:0/781597744 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5454005180 msgr2=0x7f545400b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:21.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:21.596+0000 7f54703e6640 1 --2- 172.21.15.16:0/781597744 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5454005180 0x7f545400b990 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7f546000a830 tx=0x7f546001d340 comp rx=0 tx=0).stop 2023-05-22T15:42:21.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:21.596+0000 7f54703e6640 1 -- 172.21.15.16:0/781597744 shutdown_connections 2023-05-22T15:42:21.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:21.596+0000 7f54703e6640 1 --2- 172.21.15.16:0/781597744 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5454005180 0x7f545400b990 unknown :-1 s=CLOSED pgs=1093 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:21.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:21.596+0000 7f54703e6640 1 -- 172.21.15.16:0/781597744 >> 172.21.15.16:0/781597744 conn(0x7f54680cc870 msgr2=0x7f54680cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:21.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:21.596+0000 7f54703e6640 1 -- 172.21.15.16:0/781597744 shutdown_connections 2023-05-22T15:42:21.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:21.596+0000 7f54703e6640 1 -- 172.21.15.16:0/781597744 wait complete. 2023-05-22T15:42:21.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:21.600+0000 7f54703e6640 1 Processor -- start 2023-05-22T15:42:21.606 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:21.600+0000 7f54703e6640 1 -- start start 2023-05-22T15:42:21.606 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:21.600+0000 7f54703e6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5454005180 0x7f546814c170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:21.606 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:21.600+0000 7f54703e6640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f54680c7240 0x7f54680ac940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:21.606 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:21.600+0000 7f54703e6640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f54680ace80 0x7f54680c79c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:21.606 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:21.600+0000 7f54703e6640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f546814cbb0 con 0x7f5454005180 2023-05-22T15:42:21.606 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:21.600+0000 7f54703e6640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54680ad2f0 con 0x7f54680c7240 2023-05-22T15:42:21.606 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:21.600+0000 7f54703e6640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f546814c6b0 con 0x7f54680ace80 2023-05-22T15:42:21.606 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:21.600+0000 7f546e182640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5454005180 0x7f546814c170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:21.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:21.600+0000 7f546e182640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5454005180 0x7f546814c170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34710/0 (socket says 172.21.15.16:34710) 2023-05-22T15:42:21.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:21.600+0000 7f546e182640 1 -- 172.21.15.16:0/3491760903 learned_addr learned my addr 172.21.15.16:0/3491760903 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:42:21.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:21.600+0000 7f546d981640 1 --2- 172.21.15.16:0/3491760903 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f54680c7240 0x7f54680ac940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:21.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:21.600+0000 7f546e983640 1 --2- 172.21.15.16:0/3491760903 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f54680ace80 0x7f54680c79c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:21.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:21.600+0000 7f546e182640 1 -- 172.21.15.16:0/3491760903 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f54680ace80 msgr2=0x7f54680c79c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:21.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:21.600+0000 7f546e182640 1 --2- 172.21.15.16:0/3491760903 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f54680ace80 0x7f54680c79c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:21.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:21.600+0000 7f546e182640 1 -- 172.21.15.16:0/3491760903 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f54680c7240 msgr2=0x7f54680ac940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:21.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:21.600+0000 7f546e182640 1 --2- 172.21.15.16:0/3491760903 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f54680c7240 0x7f54680ac940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:21.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:21.600+0000 7f546e182640 1 -- 172.21.15.16:0/3491760903 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f546815fb70 con 0x7f5454005180 2023-05-22T15:42:21.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:21.600+0000 7f546e983640 1 --2- 172.21.15.16:0/3491760903 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f54680ace80 0x7f54680c79c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:42:21.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:21.600+0000 7f546e182640 1 --2- 172.21.15.16:0/3491760903 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5454005180 0x7f546814c170 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 crypto rx=0x7f5464011960 tx=0x7f5464011e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:21.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:21.600+0000 7f545affd640 1 -- 172.21.15.16:0/3491760903 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f546400fa90 con 0x7f5454005180 2023-05-22T15:42:21.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:21.600+0000 7f545affd640 1 -- 172.21.15.16:0/3491760903 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f546400fbf0 con 0x7f5454005180 2023-05-22T15:42:21.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:21.600+0000 7f54703e6640 1 -- 172.21.15.16:0/3491760903 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f546806a0a0 con 0x7f5454005180 2023-05-22T15:42:21.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:21.600+0000 7f545affd640 1 -- 172.21.15.16:0/3491760903 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5464014440 con 0x7f5454005180 2023-05-22T15:42:21.609 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:21.600+0000 7f54703e6640 1 -- 172.21.15.16:0/3491760903 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54680cecd0 con 0x7f5454005180 2023-05-22T15:42:21.609 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:21.600+0000 7f54703e6640 1 -- 172.21.15.16:0/3491760903 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f546804dbe0 con 0x7f5454005180 2023-05-22T15:42:21.610 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:21.600+0000 7f545affd640 1 -- 172.21.15.16:0/3491760903 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f54640145a0 con 0x7f5454005180 2023-05-22T15:42:21.610 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:21.600+0000 7f545affd640 1 --2- 172.21.15.16:0/3491760903 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f544c0654a0 0x7f544c067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:21.610 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:21.600+0000 7f545affd640 1 -- 172.21.15.16:0/3491760903 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 5900+0+0 (secure 0 0 0) 0x7f5464083140 con 0x7f5454005180 2023-05-22T15:42:21.610 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:21.600+0000 7f546d981640 1 --2- 172.21.15.16:0/3491760903 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f544c0654a0 0x7f544c067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:21.610 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:21.604+0000 7f546d981640 1 --2- 172.21.15.16:0/3491760903 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f544c0654a0 0x7f544c067950 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f545c000c10 tx=0x7f545c009040 comp rx=0 tx=0).ready entity=mgr.6765 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:21.610 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:21.608+0000 7f545affd640 1 -- 172.21.15.16:0/3491760903 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f5464050d50 con 0x7f5454005180 2023-05-22T15:42:21.764 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:21.760+0000 7f54703e6640 1 -- 172.21.15.16:0/3491760903 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f5468070a10 con 0x7f5454005180 2023-05-22T15:42:22.046 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-05-22T15:42:22.048 DEBUG:teuthology.orchestra.run.smithi062:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-05-22T15:42:22.442 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:22.436+0000 7f545affd640 1 -- 172.21.15.16:0/3491760903 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v88) v1 ==== 170+0+0 (secure 0 0 0) 0x7f5464054370 con 0x7f5454005180 2023-05-22T15:42:22.452 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'cephfs_data' created 2023-05-22T15:42:22.457 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:22.452+0000 7f54703e6640 1 -- 172.21.15.16:0/3491760903 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f544c0654a0 msgr2=0x7f544c067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:22.458 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:22.452+0000 7f54703e6640 1 --2- 172.21.15.16:0/3491760903 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f544c0654a0 0x7f544c067950 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f545c000c10 tx=0x7f545c009040 comp rx=0 tx=0).stop 2023-05-22T15:42:22.458 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:22.452+0000 7f54703e6640 1 -- 172.21.15.16:0/3491760903 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5454005180 msgr2=0x7f546814c170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:22.458 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:22.452+0000 7f54703e6640 1 --2- 172.21.15.16:0/3491760903 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5454005180 0x7f546814c170 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 crypto rx=0x7f5464011960 tx=0x7f5464011e20 comp rx=0 tx=0).stop 2023-05-22T15:42:22.458 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:22.456+0000 7f54703e6640 1 -- 172.21.15.16:0/3491760903 shutdown_connections 2023-05-22T15:42:22.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:22.456+0000 7f54703e6640 1 --2- 172.21.15.16:0/3491760903 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f544c0654a0 0x7f544c067950 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:22.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:22.456+0000 7f54703e6640 1 --2- 172.21.15.16:0/3491760903 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f54680ace80 0x7f54680c79c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:22.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:22.456+0000 7f54703e6640 1 --2- 172.21.15.16:0/3491760903 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f54680c7240 0x7f54680ac940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:22.460 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:22.456+0000 7f54703e6640 1 --2- 172.21.15.16:0/3491760903 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5454005180 0x7f546814c170 unknown :-1 s=CLOSED pgs=1094 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:22.460 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:22.456+0000 7f54703e6640 1 -- 172.21.15.16:0/3491760903 >> 172.21.15.16:0/3491760903 conn(0x7f54680cc870 msgr2=0x7f546815a4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:22.461 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:22.456+0000 7f54703e6640 1 -- 172.21.15.16:0/3491760903 shutdown_connections 2023-05-22T15:42:22.461 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:22.456+0000 7f54703e6640 1 -- 172.21.15.16:0/3491760903 wait complete. 2023-05-22T15:42:22.485 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-05-22T15:42:22.606 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:22.604+0000 7f0e0d1a6640 1 Processor -- start 2023-05-22T15:42:22.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:22.604+0000 7f0e0d1a6640 1 -- start start 2023-05-22T15:42:22.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:22.604+0000 7f0e0d1a6640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f0e0815bca0 0x7f0e08160160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:22.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:22.604+0000 7f0e0d1a6640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0e080b3ee0 0x7f0e080b4330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:22.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:22.604+0000 7f0e0d1a6640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e080d0400 con 0x7f0e080b4870 2023-05-22T15:42:22.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:22.604+0000 7f0e0d1a6640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e0815c070 con 0x7f0e0815bca0 2023-05-22T15:42:22.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:22.604+0000 7f0e0d1a6640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e081606a0 con 0x7f0e080b3ee0 2023-05-22T15:42:22.609 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:22.604+0000 7f0e0659c640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0e080b3ee0 0x7f0e080b4330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:22.609 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:22.604+0000 7f0e0659c640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0e080b3ee0 0x7f0e080b4330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3301/0 says I am v2:172.21.15.16:40682/0 (socket says 172.21.15.16:40682) 2023-05-22T15:42:22.609 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:22.604+0000 7f0e0659c640 1 -- 172.21.15.16:0/236076330 learned_addr learned my addr 172.21.15.16:0/236076330 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:42:22.609 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:22.604+0000 7f0e06d9d640 1 --2- 172.21.15.16:0/236076330 >> v2:172.21.15.62:3300/0 conn(0x7f0e0815bca0 0x7f0e08160160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:22.610 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:22.604+0000 7f0e0659c640 1 -- 172.21.15.16:0/236076330 >> v2:172.21.15.62:3300/0 conn(0x7f0e0815bca0 msgr2=0x7f0e08160160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:42:22.610 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:22.604+0000 7f0e0659c640 1 --2- 172.21.15.16:0/236076330 >> v2:172.21.15.62:3300/0 conn(0x7f0e0815bca0 0x7f0e08160160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:22.610 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:22.604+0000 7f0e0659c640 1 -- 172.21.15.16:0/236076330 >> v1:172.21.15.62:6789/0 conn(0x7f0e080b4870 legacy=0x7f0e080ce900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:22.610 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:22.604+0000 7f0e0659c640 1 -- 172.21.15.16:0/236076330 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e080d00d0 con 0x7f0e080b3ee0 2023-05-22T15:42:22.610 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:22.604+0000 7f0e06d9d640 1 --2- 172.21.15.16:0/236076330 >> v2:172.21.15.62:3300/0 conn(0x7f0e0815bca0 0x7f0e08160160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-05-22T15:42:22.611 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:22.608+0000 7f0e0659c640 1 --2- 172.21.15.16:0/236076330 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0e080b3ee0 0x7f0e080b4330 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f0dfc009870 tx=0x7f0dfc01c480 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d522ada85ae6d752 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:22.611 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:22.608+0000 7f0e05d9b640 1 -- 172.21.15.16:0/236076330 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dfc0ab680 con 0x7f0e080b3ee0 2023-05-22T15:42:22.611 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:22.608+0000 7f0e05d9b640 1 -- 172.21.15.16:0/236076330 <== mon.2 v2:172.21.15.62:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0dfc0ab7e0 con 0x7f0e080b3ee0 2023-05-22T15:42:22.612 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:22.608+0000 7f0e05d9b640 1 -- 172.21.15.16:0/236076330 <== mon.2 v2:172.21.15.62:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dfc0abb70 con 0x7f0e080b3ee0 2023-05-22T15:42:22.612 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:22.608+0000 7f0e0d1a6640 1 -- 172.21.15.16:0/236076330 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0e080b3ee0 msgr2=0x7f0e080b4330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:22.613 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:22.608+0000 7f0e0d1a6640 1 --2- 172.21.15.16:0/236076330 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0e080b3ee0 0x7f0e080b4330 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f0dfc009870 tx=0x7f0dfc01c480 comp rx=0 tx=0).stop 2023-05-22T15:42:22.613 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:22.608+0000 7f0e0d1a6640 1 -- 172.21.15.16:0/236076330 shutdown_connections 2023-05-22T15:42:22.613 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:22.608+0000 7f0e0d1a6640 1 --2- 172.21.15.16:0/236076330 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0e080b3ee0 0x7f0e080b4330 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:22.613 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:22.608+0000 7f0e0d1a6640 1 --2- 172.21.15.16:0/236076330 >> v2:172.21.15.62:3300/0 conn(0x7f0e0815bca0 0x7f0e08160160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:22.613 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:22.608+0000 7f0e0d1a6640 1 -- 172.21.15.16:0/236076330 >> 172.21.15.16:0/236076330 conn(0x7f0e080657f0 msgr2=0x7f0e08070e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:22.613 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:22.608+0000 7f0e0d1a6640 1 -- 172.21.15.16:0/236076330 shutdown_connections 2023-05-22T15:42:22.613 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:22.608+0000 7f0e0d1a6640 1 -- 172.21.15.16:0/236076330 wait complete. 2023-05-22T15:42:22.614 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:22.608+0000 7f0e0d1a6640 1 Processor -- start 2023-05-22T15:42:22.614 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:22.608+0000 7f0e0d1a6640 1 -- start start 2023-05-22T15:42:22.615 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:22.612+0000 7f0e0d1a6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e080b3ee0 0x7f0e081515e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:22.615 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:22.612+0000 7f0e0d1a6640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0e080b4870 0x7f0e080c9aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:22.615 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:22.612+0000 7f0e0d1a6640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0e0815bca0 0x7f0e080c9fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:22.615 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:22.612+0000 7f0e0d1a6640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e080c8e00 con 0x7f0e080b3ee0 2023-05-22T15:42:22.615 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:22.612+0000 7f0e0d1a6640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e080c90e0 con 0x7f0e080b4870 2023-05-22T15:42:22.615 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:22.612+0000 7f0e0d1a6640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e080c7790 con 0x7f0e0815bca0 2023-05-22T15:42:22.615 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:22.612+0000 7f0e06d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e080b3ee0 0x7f0e081515e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:22.616 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:22.612+0000 7f0e06d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e080b3ee0 0x7f0e081515e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34716/0 (socket says 172.21.15.16:34716) 2023-05-22T15:42:22.616 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:22.612+0000 7f0e06d9d640 1 -- 172.21.15.16:0/1872022098 learned_addr learned my addr 172.21.15.16:0/1872022098 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:42:22.617 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:22.612+0000 7f0e0759e640 1 --2- 172.21.15.16:0/1872022098 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0e0815bca0 0x7f0e080c9fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:22.617 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:22.612+0000 7f0e0659c640 1 --2- 172.21.15.16:0/1872022098 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0e080b4870 0x7f0e080c9aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:22.617 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:22.612+0000 7f0e06d9d640 1 -- 172.21.15.16:0/1872022098 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0e0815bca0 msgr2=0x7f0e080c9fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:22.617 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:22.612+0000 7f0e06d9d640 1 --2- 172.21.15.16:0/1872022098 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0e0815bca0 0x7f0e080c9fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:22.617 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:22.612+0000 7f0e06d9d640 1 -- 172.21.15.16:0/1872022098 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0e080b4870 msgr2=0x7f0e080c9aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:22.617 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:22.612+0000 7f0e06d9d640 1 --2- 172.21.15.16:0/1872022098 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0e080b4870 0x7f0e080c9aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:22.617 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:22.612+0000 7f0e06d9d640 1 -- 172.21.15.16:0/1872022098 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0df8009420 con 0x7f0e080b3ee0 2023-05-22T15:42:22.618 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:22.612+0000 7f0e06d9d640 1 --2- 172.21.15.16:0/1872022098 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e080b3ee0 0x7f0e081515e0 secure :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 crypto rx=0x7f0df800ebe0 tx=0x7f0df800c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:22.618 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:22.612+0000 7f0def7fe640 1 -- 172.21.15.16:0/1872022098 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0df800eee0 con 0x7f0e080b3ee0 2023-05-22T15:42:22.618 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:22.612+0000 7f0def7fe640 1 -- 172.21.15.16:0/1872022098 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0df800cc30 con 0x7f0e080b3ee0 2023-05-22T15:42:22.618 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:22.612+0000 7f0e0d1a6640 1 -- 172.21.15.16:0/1872022098 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0dfc0094d0 con 0x7f0e080b3ee0 2023-05-22T15:42:22.618 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:22.612+0000 7f0def7fe640 1 -- 172.21.15.16:0/1872022098 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0df8013470 con 0x7f0e080b3ee0 2023-05-22T15:42:22.618 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:22.612+0000 7f0e0d1a6640 1 -- 172.21.15.16:0/1872022098 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e080cd1a0 con 0x7f0e080b3ee0 2023-05-22T15:42:22.618 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:22.612+0000 7f0e0d1a6640 1 -- 172.21.15.16:0/1872022098 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e080b4c70 con 0x7f0e080b3ee0 2023-05-22T15:42:22.619 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:22.612+0000 7f0def7fe640 1 -- 172.21.15.16:0/1872022098 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f0df80135d0 con 0x7f0e080b3ee0 2023-05-22T15:42:22.619 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:22.616+0000 7f0def7fe640 1 --2- 172.21.15.16:0/1872022098 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f0df40654a0 0x7f0df4067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:22.619 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:22.616+0000 7f0def7fe640 1 -- 172.21.15.16:0/1872022098 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 6282+0+0 (secure 0 0 0) 0x7f0df8083580 con 0x7f0e080b3ee0 2023-05-22T15:42:22.619 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:22.616+0000 7f0e0659c640 1 --2- 172.21.15.16:0/1872022098 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f0df40654a0 0x7f0df4067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:22.620 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:22.616+0000 7f0e0659c640 1 --2- 172.21.15.16:0/1872022098 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f0df40654a0 0x7f0df4067950 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f0dfc0af000 tx=0x7f0dfc0a7000 comp rx=0 tx=0).ready entity=mgr.6765 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:22.622 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:22.620+0000 7f0def7fe640 1 -- 172.21.15.16:0/1872022098 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f0df8054ab0 con 0x7f0e080b3ee0 2023-05-22T15:42:22.810 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:22.808+0000 7f0e0d1a6640 1 -- 172.21.15.16:0/1872022098 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f0e08168af0 con 0x7f0e080b3ee0 2023-05-22T15:42:22.821 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:22.816+0000 7f0def7fe640 1 -- 172.21.15.16:0/1872022098 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 10 and data pool 11 v26) v1 ==== 174+0+0 (secure 0 0 0) 0x7f0df8054640 con 0x7f0e080b3ee0 2023-05-22T15:42:22.823 INFO:teuthology.orchestra.run.smithi016.stderr:new fs with metadata pool 10 and data pool 11 2023-05-22T15:42:22.837 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:22.832+0000 7f0e0d1a6640 1 -- 172.21.15.16:0/1872022098 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f0df40654a0 msgr2=0x7f0df4067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:22.837 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:22.832+0000 7f0e0d1a6640 1 --2- 172.21.15.16:0/1872022098 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f0df40654a0 0x7f0df4067950 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f0dfc0af000 tx=0x7f0dfc0a7000 comp rx=0 tx=0).stop 2023-05-22T15:42:22.837 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:22.832+0000 7f0e0d1a6640 1 -- 172.21.15.16:0/1872022098 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e080b3ee0 msgr2=0x7f0e081515e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:22.837 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:22.832+0000 7f0e0d1a6640 1 --2- 172.21.15.16:0/1872022098 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e080b3ee0 0x7f0e081515e0 secure :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 crypto rx=0x7f0df800ebe0 tx=0x7f0df800c520 comp rx=0 tx=0).stop 2023-05-22T15:42:22.837 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:22.832+0000 7f0e0d1a6640 1 -- 172.21.15.16:0/1872022098 shutdown_connections 2023-05-22T15:42:22.837 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:22.832+0000 7f0e0d1a6640 1 --2- 172.21.15.16:0/1872022098 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f0df40654a0 0x7f0df4067950 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:22.838 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:22.832+0000 7f0e0d1a6640 1 --2- 172.21.15.16:0/1872022098 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0e0815bca0 0x7f0e080c9fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:22.838 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:22.832+0000 7f0e0d1a6640 1 --2- 172.21.15.16:0/1872022098 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0e080b4870 0x7f0e080c9aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:22.838 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:22.832+0000 7f0e0d1a6640 1 --2- 172.21.15.16:0/1872022098 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e080b3ee0 0x7f0e081515e0 unknown :-1 s=CLOSED pgs=1095 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:22.838 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:22.832+0000 7f0e0d1a6640 1 -- 172.21.15.16:0/1872022098 >> 172.21.15.16:0/1872022098 conn(0x7f0e080657f0 msgr2=0x7f0e080b5b70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:22.838 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:22.832+0000 7f0e0d1a6640 1 -- 172.21.15.16:0/1872022098 shutdown_connections 2023-05-22T15:42:22.839 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:22.832+0000 7f0e0d1a6640 1 -- 172.21.15.16:0/1872022098 wait complete. 2023-05-22T15:42:22.859 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-05-22T15:42:22.986 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:22.984+0000 7fcabd944640 1 Processor -- start 2023-05-22T15:42:22.986 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:22.984+0000 7fcabd944640 1 -- start start 2023-05-22T15:42:22.987 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:22.984+0000 7fcabd944640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fcab815c350 0x7fcab815c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:22.987 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:22.984+0000 7fcabd944640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fcab8069720 0x7fcab8069b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:22.987 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:22.984+0000 7fcabd944640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcab80c95f0 con 0x7fcab80ade80 2023-05-22T15:42:22.988 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:22.984+0000 7fcabd944640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcab80ca270 con 0x7fcab815c350 2023-05-22T15:42:22.988 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:22.984+0000 7fcabd944640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcab815cd30 con 0x7fcab8069720 2023-05-22T15:42:22.988 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:22.984+0000 7fcaaffff640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fcab815c350 0x7fcab815c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:22.988 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:22.984+0000 7fcab6ffd640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fcab80ade80 0x7fcab80cc400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:33450/0 (socket says 172.21.15.16:33450) 2023-05-22T15:42:22.988 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:22.984+0000 7fcaaffff640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fcab815c350 0x7fcab815c7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34726/0 (socket says 172.21.15.16:34726) 2023-05-22T15:42:22.988 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:22.984+0000 7fcab6ffd640 1 -- 172.21.15.16:0/1860294650 learned_addr learned my addr 172.21.15.16:0/1860294650 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:42:22.988 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:22.984+0000 7fcab77fe640 1 --2- 172.21.15.16:0/1860294650 >> v2:172.21.15.62:3300/0 conn(0x7fcab8069720 0x7fcab8069b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:22.989 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:22.984+0000 7fcaaffff640 1 -- 172.21.15.16:0/1860294650 >> v2:172.21.15.62:3300/0 conn(0x7fcab8069720 msgr2=0x7fcab8069b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:42:22.989 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:22.984+0000 7fcaaffff640 1 --2- 172.21.15.16:0/1860294650 >> v2:172.21.15.62:3300/0 conn(0x7fcab8069720 0x7fcab8069b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:22.989 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:22.984+0000 7fcaaffff640 1 -- 172.21.15.16:0/1860294650 >> v1:172.21.15.16:6789/0 conn(0x7fcab80ade80 legacy=0x7fcab80cc400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:22.989 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:22.984+0000 7fcab67fc640 1 -- 172.21.15.16:0/1860294650 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2286156002 0 0) 0x7fcab80c95f0 con 0x7fcab80ade80 2023-05-22T15:42:22.990 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:22.984+0000 7fcaaffff640 1 -- 172.21.15.16:0/1860294650 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcab80c98d0 con 0x7fcab815c350 2023-05-22T15:42:22.990 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:22.984+0000 7fcaaffff640 1 --2- 172.21.15.16:0/1860294650 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcab815c350 0x7fcab815c7f0 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 crypto rx=0x7fcab8065c50 tx=0x7fcaa001e8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=487acf8c4bb5f369 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:22.991 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:22.988+0000 7fcab67fc640 1 -- 172.21.15.16:0/1860294650 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcaa00a7790 con 0x7fcab815c350 2023-05-22T15:42:22.991 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:22.988+0000 7fcab67fc640 1 -- 172.21.15.16:0/1860294650 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcaa00a78f0 con 0x7fcab815c350 2023-05-22T15:42:22.991 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:22.988+0000 7fcab67fc640 1 -- 172.21.15.16:0/1860294650 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcaa00a7c80 con 0x7fcab815c350 2023-05-22T15:42:22.991 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:22.988+0000 7fcabd944640 1 -- 172.21.15.16:0/1860294650 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcab815c350 msgr2=0x7fcab815c7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:22.991 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:22.988+0000 7fcabd944640 1 --2- 172.21.15.16:0/1860294650 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcab815c350 0x7fcab815c7f0 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 crypto rx=0x7fcab8065c50 tx=0x7fcaa001e8c0 comp rx=0 tx=0).stop 2023-05-22T15:42:22.991 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:22.988+0000 7fcabd944640 1 -- 172.21.15.16:0/1860294650 shutdown_connections 2023-05-22T15:42:22.991 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:22.988+0000 7fcabd944640 1 --2- 172.21.15.16:0/1860294650 >> v2:172.21.15.62:3300/0 conn(0x7fcab8069720 0x7fcab8069b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:22.992 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:22.988+0000 7fcabd944640 1 --2- 172.21.15.16:0/1860294650 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcab815c350 0x7fcab815c7f0 unknown :-1 s=CLOSED pgs=1097 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:22.992 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:22.988+0000 7fcabd944640 1 -- 172.21.15.16:0/1860294650 >> 172.21.15.16:0/1860294650 conn(0x7fcab80b1110 msgr2=0x7fcab80ce830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:22.992 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:22.988+0000 7fcabd944640 1 -- 172.21.15.16:0/1860294650 shutdown_connections 2023-05-22T15:42:22.992 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:22.988+0000 7fcabd944640 1 -- 172.21.15.16:0/1860294650 wait complete. 2023-05-22T15:42:22.993 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:22.988+0000 7fcabd944640 1 Processor -- start 2023-05-22T15:42:22.993 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:22.988+0000 7fcabd944640 1 -- start start 2023-05-22T15:42:22.993 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:22.988+0000 7fcabd944640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcab8069720 0x7fcab814a520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:22.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:22.988+0000 7fcabd944640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcab80ade80 0x7fcab80c7b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:22.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:22.988+0000 7fcabd944640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fcab815c350 0x7fcab80c8080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:22.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:22.988+0000 7fcabd944640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcab80af1a0 con 0x7fcab80ade80 2023-05-22T15:42:22.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:22.988+0000 7fcabd944640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcab80af480 con 0x7fcab8069720 2023-05-22T15:42:22.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:22.988+0000 7fcabd944640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcab80af760 con 0x7fcab815c350 2023-05-22T15:42:22.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:22.988+0000 7fcaaffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcab80ade80 0x7fcab80c7b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:22.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:22.988+0000 7fcaaffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcab80ade80 0x7fcab80c7b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34740/0 (socket says 172.21.15.16:34740) 2023-05-22T15:42:22.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:22.988+0000 7fcaaffff640 1 -- 172.21.15.16:0/2698387983 learned_addr learned my addr 172.21.15.16:0/2698387983 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:42:22.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:22.988+0000 7fcab77fe640 1 --2- 172.21.15.16:0/2698387983 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fcab815c350 0x7fcab80c8080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:22.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:22.988+0000 7fcab6ffd640 1 --2- 172.21.15.16:0/2698387983 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcab8069720 0x7fcab814a520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:22.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:22.988+0000 7fcaaffff640 1 -- 172.21.15.16:0/2698387983 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fcab815c350 msgr2=0x7fcab80c8080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:22.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:22.988+0000 7fcaaffff640 1 --2- 172.21.15.16:0/2698387983 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fcab815c350 0x7fcab80c8080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:22.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:22.988+0000 7fcaaffff640 1 -- 172.21.15.16:0/2698387983 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcab8069720 msgr2=0x7fcab814a520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:22.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:22.988+0000 7fcaaffff640 1 --2- 172.21.15.16:0/2698387983 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcab8069720 0x7fcab814a520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:22.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:22.988+0000 7fcaaffff640 1 -- 172.21.15.16:0/2698387983 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcaa000b6c0 con 0x7fcab80ade80 2023-05-22T15:42:22.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:22.988+0000 7fcaaffff640 1 --2- 172.21.15.16:0/2698387983 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcab80ade80 0x7fcab80c7b40 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 crypto rx=0x7fcaa001b4d0 tx=0x7fcaa001d100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:22.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:22.988+0000 7fcab4ff9640 1 -- 172.21.15.16:0/2698387983 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcaa0004230 con 0x7fcab80ade80 2023-05-22T15:42:22.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:22.988+0000 7fcab4ff9640 1 -- 172.21.15.16:0/2698387983 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcaa0004390 con 0x7fcab80ade80 2023-05-22T15:42:22.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:22.988+0000 7fcab4ff9640 1 -- 172.21.15.16:0/2698387983 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcaa0003820 con 0x7fcab80ade80 2023-05-22T15:42:22.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:22.988+0000 7fcabd944640 1 -- 172.21.15.16:0/2698387983 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcab80c9210 con 0x7fcab80ade80 2023-05-22T15:42:22.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:22.988+0000 7fcabd944640 1 -- 172.21.15.16:0/2698387983 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcab815d090 con 0x7fcab80ade80 2023-05-22T15:42:22.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:22.992+0000 7fcabd944640 1 -- 172.21.15.16:0/2698387983 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcab80c93a0 con 0x7fcab80ade80 2023-05-22T15:42:22.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:22.992+0000 7fcab4ff9640 1 -- 172.21.15.16:0/2698387983 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) v1 ==== 84332+0+0 (secure 0 0 0) 0x7fcaa00b6040 con 0x7fcab80ade80 2023-05-22T15:42:22.998 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:22.992+0000 7fcab4ff9640 1 --2- 172.21.15.16:0/2698387983 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fca980654a0 0x7fca98067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:22.998 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:22.992+0000 7fcab4ff9640 1 -- 172.21.15.16:0/2698387983 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 6382+0+0 (secure 0 0 0) 0x7fcaa011de60 con 0x7fcab80ade80 2023-05-22T15:42:22.998 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:22.992+0000 7fcab6ffd640 1 --2- 172.21.15.16:0/2698387983 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fca980654a0 0x7fca98067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:22.998 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:22.992+0000 7fcab6ffd640 1 --2- 172.21.15.16:0/2698387983 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fca980654a0 0x7fca98067950 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fcaa8000f70 tx=0x7fcaa8007b60 comp rx=0 tx=0).ready entity=mgr.6765 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:22.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:22.996+0000 7fcab4ff9640 1 -- 172.21.15.16:0/2698387983 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fcaa0148620 con 0x7fcab80ade80 2023-05-22T15:42:23.157 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:23.152+0000 7fcabd944640 1 -- 172.21.15.16:0/2698387983 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcab80b1d50 con 0x7fcab80ade80 2023-05-22T15:42:23.160 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:23.156+0000 7fcab4ff9640 1 -- 172.21.15.16:0/2698387983 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+18052 (secure 0 0 0) 0x7fcab80b1d50 con 0x7fcab80ade80 2023-05-22T15:42:23.161 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:42:23.161 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":89,"fsid":"b2719834-7847-4d98-90a6-f65d57f85c49","created":"2023-05-22T15:26:25.259314+0000","modified":"2023-05-22T15:42:22.811131+0000","last_up_change":"2023-05-22T15:26:36.448524+0000","last_in_change":"2023-05-22T15:26:27.675668+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-05-22T15:26:38.019781+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-05-22T15:26:43.570514+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2023-05-22T15:42:20.730757+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2023-05-22T15:42:21.805016+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9e7ff854-b25f-4cbb-a5c2-1799d9c53d2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":37021},{"type":"v1","addr":"172.21.15.16:6825","nonce":37021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":37021},{"type":"v1","addr":"172.21.15.16:6827","nonce":37021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":37021},{"type":"v1","addr":"172.21.15.16:6831","nonce":37021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":37021},{"type":"v1","addr":"172.21.15.16:6829","nonce":37021}]},"public_addr":"172.21.15.16:6825/37021","cluster_addr":"172.21.15.16:6827/37021","heartbeat_back_addr":"172.21.15.16:6831/37021","heartbeat_front_addr":"172.21.15.16:6829/37021","state":["exists","up"]},{"osd":1,"uuid":"4702fbc9-a927-4565-aca5-41dc69cc8264","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":37008},{"type":"v1","addr":"172.21.15.16:6801","nonce":37008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":37008},{"type":"v1","addr":"172.21.15.16:6803","nonce":37008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":37008},{"type":"v1","addr":"172.21.15.16:6807","nonce":37008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":37008},{"type":"v1","addr":"172.21.15.16:6805","nonce":37008}]},"public_addr":"172.21.15.16:6801/37008","cluster_addr":"172.21.15.16:6803/37008","heartbeat_back_addr":"172.21.15.16:6807/37008","heartbeat_front_addr":"172.21.15.16:6805/37008","state":["exists","up"]},{"osd":2,"uuid":"1e7bba46-4eba-47ef-bb81-5e2d371638ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":37009},{"type":"v1","addr":"172.21.15.16:6809","nonce":37009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":37009},{"type":"v1","addr":"172.21.15.16:6811","nonce":37009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":37009},{"type":"v1","addr":"172.21.15.16:6815","nonce":37009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":37009},{"type":"v1","addr":"172.21.15.16:6813","nonce":37009}]},"public_addr":"172.21.15.16:6809/37009","cluster_addr":"172.21.15.16:6811/37009","heartbeat_back_addr":"172.21.15.16:6815/37009","heartbeat_front_addr":"172.21.15.16:6813/37009","state":["exists","up"]},{"osd":3,"uuid":"383cacc0-5a88-4904-990e-498ca59ef48a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":37020},{"type":"v1","addr":"172.21.15.16:6817","nonce":37020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":37020},{"type":"v1","addr":"172.21.15.16:6819","nonce":37020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":37020},{"type":"v1","addr":"172.21.15.16:6823","nonce":37020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":37020},{"type":"v1","addr":"172.21.15.16:6821","nonce":37020}]},"public_addr":"172.21.15.16:6817/37020","cluster_addr":"172.21.15.16:6819/37020","heartbeat_back_addr":"172.21.15.16:6823/37020","heartbeat_front_addr":"172.21.15.16:6821/37020","state":["exists","up"]},{"osd":4,"uuid":"970500ef-95b7-42c1-b080-eed4fae7013d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":37353},{"type":"v1","addr":"172.21.15.62:6817","nonce":37353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6819","nonce":37353},{"type":"v1","addr":"172.21.15.62:6822","nonce":37353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37353},{"type":"v1","addr":"172.21.15.62:6831","nonce":37353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37353},{"type":"v1","addr":"172.21.15.62:6826","nonce":37353}]},"public_addr":"172.21.15.62:6817/37353","cluster_addr":"172.21.15.62:6822/37353","heartbeat_back_addr":"172.21.15.62:6831/37353","heartbeat_front_addr":"172.21.15.62:6826/37353","state":["exists","up"]},{"osd":5,"uuid":"022d67db-d4d8-445b-b1b9-8b446a00f5c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":37352},{"type":"v1","addr":"172.21.15.62:6801","nonce":37352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":37352},{"type":"v1","addr":"172.21.15.62:6803","nonce":37352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":37352},{"type":"v1","addr":"172.21.15.62:6807","nonce":37352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":37352},{"type":"v1","addr":"172.21.15.62:6805","nonce":37352}]},"public_addr":"172.21.15.62:6801/37352","cluster_addr":"172.21.15.62:6803/37352","heartbeat_back_addr":"172.21.15.62:6807/37352","heartbeat_front_addr":"172.21.15.62:6805/37352","state":["exists","up"]},{"osd":6,"uuid":"42bd372b-6688-4a1b-8407-e16e8c266e33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":37354},{"type":"v1","addr":"172.21.15.62:6820","nonce":37354}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":37354},{"type":"v1","addr":"172.21.15.62:6823","nonce":37354}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6829","nonce":37354},{"type":"v1","addr":"172.21.15.62:6830","nonce":37354}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6825","nonce":37354},{"type":"v1","addr":"172.21.15.62:6827","nonce":37354}]},"public_addr":"172.21.15.62:6820/37354","cluster_addr":"172.21.15.62:6823/37354","heartbeat_back_addr":"172.21.15.62:6830/37354","heartbeat_front_addr":"172.21.15.62:6827/37354","state":["exists","up"]},{"osd":7,"uuid":"ac70b2a3-f41b-4bf2-b227-7f2cbb55426d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37355},{"type":"v1","addr":"172.21.15.62:6809","nonce":37355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37355},{"type":"v1","addr":"172.21.15.62:6811","nonce":37355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":37355},{"type":"v1","addr":"172.21.15.62:6815","nonce":37355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":37355},{"type":"v1","addr":"172.21.15.62:6813","nonce":37355}]},"public_addr":"172.21.15.62:6809/37355","cluster_addr":"172.21.15.62:6811/37355","heartbeat_back_addr":"172.21.15.62:6815/37355","heartbeat_front_addr":"172.21.15.62:6813/37355","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.368896+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.107273+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.276888+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.287502+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.124791+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.251937+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.255679+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.232797+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-05-22T15:42:23.162 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:23.160+0000 7fcabd944640 1 -- 172.21.15.16:0/2698387983 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fca980654a0 msgr2=0x7fca98067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:23.163 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:23.160+0000 7fcabd944640 1 --2- 172.21.15.16:0/2698387983 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fca980654a0 0x7fca98067950 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fcaa8000f70 tx=0x7fcaa8007b60 comp rx=0 tx=0).stop 2023-05-22T15:42:23.163 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:23.160+0000 7fcabd944640 1 -- 172.21.15.16:0/2698387983 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcab80ade80 msgr2=0x7fcab80c7b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:23.163 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:23.160+0000 7fcabd944640 1 --2- 172.21.15.16:0/2698387983 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcab80ade80 0x7fcab80c7b40 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 crypto rx=0x7fcaa001b4d0 tx=0x7fcaa001d100 comp rx=0 tx=0).stop 2023-05-22T15:42:23.163 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:23.160+0000 7fcabd944640 1 -- 172.21.15.16:0/2698387983 shutdown_connections 2023-05-22T15:42:23.163 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:23.160+0000 7fcabd944640 1 --2- 172.21.15.16:0/2698387983 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fca980654a0 0x7fca98067950 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:23.163 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:23.160+0000 7fcabd944640 1 --2- 172.21.15.16:0/2698387983 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fcab815c350 0x7fcab80c8080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:23.163 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:23.160+0000 7fcabd944640 1 --2- 172.21.15.16:0/2698387983 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcab80ade80 0x7fcab80c7b40 unknown :-1 s=CLOSED pgs=1098 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:23.164 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:23.160+0000 7fcabd944640 1 --2- 172.21.15.16:0/2698387983 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcab8069720 0x7fcab814a520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:23.164 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:23.160+0000 7fcabd944640 1 -- 172.21.15.16:0/2698387983 >> 172.21.15.16:0/2698387983 conn(0x7fcab80b1110 msgr2=0x7fcab806fed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:23.164 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:23.160+0000 7fcabd944640 1 -- 172.21.15.16:0/2698387983 shutdown_connections 2023-05-22T15:42:23.164 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:23.160+0000 7fcabd944640 1 -- 172.21.15.16:0/2698387983 wait complete. 2023-05-22T15:42:23.181 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-05-22T15:42:23.302 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:23.296+0000 7f9dd7b26640 1 Processor -- start 2023-05-22T15:42:23.302 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:23.296+0000 7f9dd7b26640 1 -- start start 2023-05-22T15:42:23.303 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:23.300+0000 7f9dd7b26640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f9dd00d0e40 0x7f9dd01648d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:23.303 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:23.300+0000 7f9dd7b26640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9dd00cf9a0 con 0x7f9dd0159cd0 2023-05-22T15:42:23.303 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:23.300+0000 7f9dd7b26640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9dd015aba0 con 0x7f9dd00b2880 2023-05-22T15:42:23.304 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:23.300+0000 7f9dd7b26640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9dd015ae80 con 0x7f9dd00d0e40 2023-05-22T15:42:23.304 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:23.300+0000 7f9dd58c2640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f9dd0159cd0 0x7f9dd015a0a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:33452/0 (socket says 172.21.15.16:33452) 2023-05-22T15:42:23.304 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:23.300+0000 7f9dd58c2640 1 -- 172.21.15.16:0/1238816291 learned_addr learned my addr 172.21.15.16:0/1238816291 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:42:23.304 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:23.300+0000 7f9dd60c3640 1 --2- 172.21.15.16:0/1238816291 >> v2:172.21.15.62:3300/0 conn(0x7f9dd00d0e40 0x7f9dd01648d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:23.305 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:23.300+0000 7f9dd48c0640 1 -- 172.21.15.16:0/1238816291 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3829658590 0 0) 0x7f9dd00cf9a0 con 0x7f9dd0159cd0 2023-05-22T15:42:23.305 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:23.300+0000 7f9dd48c0640 1 -- 172.21.15.16:0/1238816291 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9dbc003530 con 0x7f9dd0159cd0 2023-05-22T15:42:23.305 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:23.300+0000 7f9dd48c0640 1 -- 172.21.15.16:0/1238816291 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2870455093 0 0) 0x7f9dbc003530 con 0x7f9dd0159cd0 2023-05-22T15:42:23.306 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:23.300+0000 7f9dd48c0640 1 -- 172.21.15.16:0/1238816291 >> v2:172.21.15.62:3300/0 conn(0x7f9dd00d0e40 msgr2=0x7f9dd01648d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:42:23.306 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:23.300+0000 7f9dd48c0640 1 --2- 172.21.15.16:0/1238816291 >> v2:172.21.15.62:3300/0 conn(0x7f9dd00d0e40 0x7f9dd01648d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:23.306 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:23.300+0000 7f9dd48c0640 1 -- 172.21.15.16:0/1238816291 >> v1:172.21.15.62:6789/0 conn(0x7f9dd00b2880 legacy=0x7f9dd00cbfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:23.306 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:23.300+0000 7f9dd48c0640 1 -- 172.21.15.16:0/1238816291 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9dd015b160 con 0x7f9dd0159cd0 2023-05-22T15:42:23.306 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:23.300+0000 7f9dd48c0640 1 -- 172.21.15.16:0/1238816291 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7f9dc4001680 con 0x7f9dd0159cd0 2023-05-22T15:42:23.306 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:23.300+0000 7f9dd48c0640 1 -- 172.21.15.16:0/1238816291 >> v1:172.21.15.16:6789/0 conn(0x7f9dd0159cd0 legacy=0x7f9dd015a0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:23.306 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:23.300+0000 7f9dd48c0640 1 --2- 172.21.15.16:0/1238816291 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9dbc003980 0x7f9dbc003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:23.307 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:23.300+0000 7f9dd48c0640 1 --2- 172.21.15.16:0/1238816291 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9dbc0047d0 0x7f9dbc004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:23.307 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:23.300+0000 7f9dd48c0640 1 --2- 172.21.15.16:0/1238816291 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9dbc005180 0x7f9dbc00b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:23.307 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:23.300+0000 7f9dd48c0640 1 -- 172.21.15.16:0/1238816291 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dd015ae80 con 0x7f9dbc0047d0 2023-05-22T15:42:23.307 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:23.300+0000 7f9dd48c0640 1 -- 172.21.15.16:0/1238816291 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dbc00c000 con 0x7f9dbc005180 2023-05-22T15:42:23.307 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:23.300+0000 7f9dd48c0640 1 -- 172.21.15.16:0/1238816291 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9dbc00c1c0 con 0x7f9dbc003980 2023-05-22T15:42:23.307 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:23.300+0000 7f9dd50c1640 1 --2- 172.21.15.16:0/1238816291 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9dbc0047d0 0x7f9dbc004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:23.307 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:23.300+0000 7f9dd60c3640 1 --2- 172.21.15.16:0/1238816291 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9dbc005180 0x7f9dbc00b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:23.307 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:23.300+0000 7f9dd58c2640 1 --2- 172.21.15.16:0/1238816291 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9dbc003980 0x7f9dbc003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:23.308 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:23.300+0000 7f9dd50c1640 1 -- 172.21.15.16:0/1238816291 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9dbc003980 msgr2=0x7f9dbc003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:42:23.308 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:23.300+0000 7f9dd50c1640 1 --2- 172.21.15.16:0/1238816291 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9dbc003980 0x7f9dbc003d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:23.308 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:23.300+0000 7f9dd50c1640 1 -- 172.21.15.16:0/1238816291 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9dbc005180 msgr2=0x7f9dbc00b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:42:23.308 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:23.300+0000 7f9dd50c1640 1 --2- 172.21.15.16:0/1238816291 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9dbc005180 0x7f9dbc00b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:23.308 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:23.300+0000 7f9dd60c3640 1 -- 172.21.15.16:0/1238816291 reap_dead start 2023-05-22T15:42:23.308 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:23.300+0000 7f9dd50c1640 1 -- 172.21.15.16:0/1238816291 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9dbc003530 con 0x7f9dbc0047d0 2023-05-22T15:42:23.310 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:23.304+0000 7f9dd50c1640 1 --2- 172.21.15.16:0/1238816291 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9dbc0047d0 0x7f9dbc004c40 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7f9dcc001e70 tx=0x7f9dcc01c740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ddc6f89cecd325b server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:23.310 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:23.304+0000 7f9dd48c0640 1 -- 172.21.15.16:0/1238816291 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dcc003b80 con 0x7f9dbc0047d0 2023-05-22T15:42:23.310 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:23.304+0000 7f9dd48c0640 1 -- 172.21.15.16:0/1238816291 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9dcc003ce0 con 0x7f9dbc0047d0 2023-05-22T15:42:23.311 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:23.304+0000 7f9dd48c0640 1 -- 172.21.15.16:0/1238816291 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dcc0052c0 con 0x7f9dbc0047d0 2023-05-22T15:42:23.311 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:23.304+0000 7f9dd7b26640 1 -- 172.21.15.16:0/1238816291 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9dbc0047d0 msgr2=0x7f9dbc004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:23.311 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:23.304+0000 7f9dd7b26640 1 --2- 172.21.15.16:0/1238816291 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9dbc0047d0 0x7f9dbc004c40 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7f9dcc001e70 tx=0x7f9dcc01c740 comp rx=0 tx=0).stop 2023-05-22T15:42:23.311 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:23.304+0000 7f9dd7b26640 1 -- 172.21.15.16:0/1238816291 shutdown_connections 2023-05-22T15:42:23.311 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:23.304+0000 7f9dd7b26640 1 --2- 172.21.15.16:0/1238816291 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9dbc0047d0 0x7f9dbc004c40 unknown :-1 s=CLOSED pgs=1100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:23.311 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:23.304+0000 7f9dd7b26640 1 -- 172.21.15.16:0/1238816291 >> 172.21.15.16:0/1238816291 conn(0x7f9dd006f5c0 msgr2=0x7f9dd006f9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:23.311 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:23.304+0000 7f9dd7b26640 1 -- 172.21.15.16:0/1238816291 shutdown_connections 2023-05-22T15:42:23.312 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:23.304+0000 7f9dd7b26640 1 -- 172.21.15.16:0/1238816291 wait complete. 2023-05-22T15:42:23.312 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:23.304+0000 7f9dd7b26640 1 Processor -- start 2023-05-22T15:42:23.312 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:23.304+0000 7f9dd7b26640 1 -- start start 2023-05-22T15:42:23.312 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:23.304+0000 7f9dd7b26640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9dbc0047d0 0x7f9dd01ef390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:23.312 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:23.304+0000 7f9dd7b26640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9dd00c5580 0x7f9dd014ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:23.312 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:23.304+0000 7f9dd7b26640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9dd00c59f0 0x7f9dd014f340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:23.312 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:23.304+0000 7f9dd7b26640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dd00b6230 con 0x7f9dd00c59f0 2023-05-22T15:42:23.312 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:23.304+0000 7f9dd7b26640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dd014f880 con 0x7f9dd00c5580 2023-05-22T15:42:23.313 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:23.304+0000 7f9dd7b26640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9dd01f0550 con 0x7f9dbc0047d0 2023-05-22T15:42:23.313 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:23.308+0000 7f9dd60c3640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9dd00c59f0 0x7f9dd014f340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:23.313 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:23.308+0000 7f9dd60c3640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9dd00c59f0 0x7f9dd014f340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34762/0 (socket says 172.21.15.16:34762) 2023-05-22T15:42:23.313 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:23.308+0000 7f9dd60c3640 1 -- 172.21.15.16:0/789052522 learned_addr learned my addr 172.21.15.16:0/789052522 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:42:23.313 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:23.308+0000 7f9dd58c2640 1 --2- 172.21.15.16:0/789052522 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9dbc0047d0 0x7f9dd01ef390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:23.313 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:23.308+0000 7f9dd50c1640 1 --2- 172.21.15.16:0/789052522 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9dd00c5580 0x7f9dd014ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:23.313 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:23.308+0000 7f9dd60c3640 1 -- 172.21.15.16:0/789052522 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9dbc0047d0 msgr2=0x7f9dd01ef390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:23.313 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:23.308+0000 7f9dd60c3640 1 --2- 172.21.15.16:0/789052522 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9dbc0047d0 0x7f9dd01ef390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:23.314 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:23.308+0000 7f9dd60c3640 1 -- 172.21.15.16:0/789052522 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9dd00c5580 msgr2=0x7f9dd014ee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:23.314 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:23.308+0000 7f9dd60c3640 1 --2- 172.21.15.16:0/789052522 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9dd00c5580 0x7f9dd014ee00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:23.314 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:23.308+0000 7f9dd60c3640 1 -- 172.21.15.16:0/789052522 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9dd015a0a0 con 0x7f9dd00c59f0 2023-05-22T15:42:23.314 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:23.308+0000 7f9dd60c3640 1 --2- 172.21.15.16:0/789052522 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9dd00c59f0 0x7f9dd014f340 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 crypto rx=0x7f9dd00cd930 tx=0x7f9dc8011550 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:23.315 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:23.308+0000 7f9dc27fc640 1 -- 172.21.15.16:0/789052522 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dc8004500 con 0x7f9dd00c59f0 2023-05-22T15:42:23.315 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:23.308+0000 7f9dc27fc640 1 -- 172.21.15.16:0/789052522 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9dc80118a0 con 0x7f9dd00c59f0 2023-05-22T15:42:23.315 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:23.308+0000 7f9dc27fc640 1 -- 172.21.15.16:0/789052522 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dc8011c70 con 0x7f9dd00c59f0 2023-05-22T15:42:23.315 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:23.308+0000 7f9dd7b26640 1 -- 172.21.15.16:0/789052522 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9dd00cf630 con 0x7f9dd00c59f0 2023-05-22T15:42:23.316 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:23.308+0000 7f9dd7b26640 1 -- 172.21.15.16:0/789052522 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9dd015a230 con 0x7f9dd00c59f0 2023-05-22T15:42:23.316 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:23.308+0000 7f9dd7b26640 1 -- 172.21.15.16:0/789052522 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9dd004dbe0 con 0x7f9dd00c59f0 2023-05-22T15:42:23.316 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:23.308+0000 7f9dc27fc640 1 -- 172.21.15.16:0/789052522 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f9dc8003620 con 0x7f9dd00c59f0 2023-05-22T15:42:23.316 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:23.308+0000 7f9dc27fc640 1 --2- 172.21.15.16:0/789052522 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f9db40654a0 0x7f9db4067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:23.316 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:23.308+0000 7f9dc27fc640 1 -- 172.21.15.16:0/789052522 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f9dc80825c0 con 0x7f9dd00c59f0 2023-05-22T15:42:23.316 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:23.308+0000 7f9dd58c2640 1 --2- 172.21.15.16:0/789052522 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f9db40654a0 0x7f9db4067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:23.316 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:23.308+0000 7f9dd58c2640 1 --2- 172.21.15.16:0/789052522 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f9db40654a0 0x7f9db4067950 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f9dc40017b0 tx=0x7f9dc40015a0 comp rx=0 tx=0).ready entity=mgr.6765 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:23.317 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:23.312+0000 7f9dc27fc640 1 -- 172.21.15.16:0/789052522 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f9dc8050020 con 0x7f9dd00c59f0 2023-05-22T15:42:23.474 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:23.472+0000 7f9dd7b26640 1 -- 172.21.15.16:0/789052522 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9dd00c9730 con 0x7f9dd00c59f0 2023-05-22T15:42:23.477 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:23.472+0000 7f9dc27fc640 1 -- 172.21.15.16:0/789052522 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+18052 (secure 0 0 0) 0x7f9dc8053640 con 0x7f9dd00c59f0 2023-05-22T15:42:23.478 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:42:23.478 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":89,"fsid":"b2719834-7847-4d98-90a6-f65d57f85c49","created":"2023-05-22T15:26:25.259314+0000","modified":"2023-05-22T15:42:22.811131+0000","last_up_change":"2023-05-22T15:26:36.448524+0000","last_in_change":"2023-05-22T15:26:27.675668+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-05-22T15:26:38.019781+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-05-22T15:26:43.570514+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2023-05-22T15:42:20.730757+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2023-05-22T15:42:21.805016+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9e7ff854-b25f-4cbb-a5c2-1799d9c53d2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":37021},{"type":"v1","addr":"172.21.15.16:6825","nonce":37021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":37021},{"type":"v1","addr":"172.21.15.16:6827","nonce":37021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":37021},{"type":"v1","addr":"172.21.15.16:6831","nonce":37021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":37021},{"type":"v1","addr":"172.21.15.16:6829","nonce":37021}]},"public_addr":"172.21.15.16:6825/37021","cluster_addr":"172.21.15.16:6827/37021","heartbeat_back_addr":"172.21.15.16:6831/37021","heartbeat_front_addr":"172.21.15.16:6829/37021","state":["exists","up"]},{"osd":1,"uuid":"4702fbc9-a927-4565-aca5-41dc69cc8264","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":37008},{"type":"v1","addr":"172.21.15.16:6801","nonce":37008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":37008},{"type":"v1","addr":"172.21.15.16:6803","nonce":37008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":37008},{"type":"v1","addr":"172.21.15.16:6807","nonce":37008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":37008},{"type":"v1","addr":"172.21.15.16:6805","nonce":37008}]},"public_addr":"172.21.15.16:6801/37008","cluster_addr":"172.21.15.16:6803/37008","heartbeat_back_addr":"172.21.15.16:6807/37008","heartbeat_front_addr":"172.21.15.16:6805/37008","state":["exists","up"]},{"osd":2,"uuid":"1e7bba46-4eba-47ef-bb81-5e2d371638ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":37009},{"type":"v1","addr":"172.21.15.16:6809","nonce":37009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":37009},{"type":"v1","addr":"172.21.15.16:6811","nonce":37009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":37009},{"type":"v1","addr":"172.21.15.16:6815","nonce":37009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":37009},{"type":"v1","addr":"172.21.15.16:6813","nonce":37009}]},"public_addr":"172.21.15.16:6809/37009","cluster_addr":"172.21.15.16:6811/37009","heartbeat_back_addr":"172.21.15.16:6815/37009","heartbeat_front_addr":"172.21.15.16:6813/37009","state":["exists","up"]},{"osd":3,"uuid":"383cacc0-5a88-4904-990e-498ca59ef48a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":37020},{"type":"v1","addr":"172.21.15.16:6817","nonce":37020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":37020},{"type":"v1","addr":"172.21.15.16:6819","nonce":37020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":37020},{"type":"v1","addr":"172.21.15.16:6823","nonce":37020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":37020},{"type":"v1","addr":"172.21.15.16:6821","nonce":37020}]},"public_addr":"172.21.15.16:6817/37020","cluster_addr":"172.21.15.16:6819/37020","heartbeat_back_addr":"172.21.15.16:6823/37020","heartbeat_front_addr":"172.21.15.16:6821/37020","state":["exists","up"]},{"osd":4,"uuid":"970500ef-95b7-42c1-b080-eed4fae7013d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":37353},{"type":"v1","addr":"172.21.15.62:6817","nonce":37353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6819","nonce":37353},{"type":"v1","addr":"172.21.15.62:6822","nonce":37353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37353},{"type":"v1","addr":"172.21.15.62:6831","nonce":37353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37353},{"type":"v1","addr":"172.21.15.62:6826","nonce":37353}]},"public_addr":"172.21.15.62:6817/37353","cluster_addr":"172.21.15.62:6822/37353","heartbeat_back_addr":"172.21.15.62:6831/37353","heartbeat_front_addr":"172.21.15.62:6826/37353","state":["exists","up"]},{"osd":5,"uuid":"022d67db-d4d8-445b-b1b9-8b446a00f5c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":37352},{"type":"v1","addr":"172.21.15.62:6801","nonce":37352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":37352},{"type":"v1","addr":"172.21.15.62:6803","nonce":37352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":37352},{"type":"v1","addr":"172.21.15.62:6807","nonce":37352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":37352},{"type":"v1","addr":"172.21.15.62:6805","nonce":37352}]},"public_addr":"172.21.15.62:6801/37352","cluster_addr":"172.21.15.62:6803/37352","heartbeat_back_addr":"172.21.15.62:6807/37352","heartbeat_front_addr":"172.21.15.62:6805/37352","state":["exists","up"]},{"osd":6,"uuid":"42bd372b-6688-4a1b-8407-e16e8c266e33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":37354},{"type":"v1","addr":"172.21.15.62:6820","nonce":37354}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":37354},{"type":"v1","addr":"172.21.15.62:6823","nonce":37354}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6829","nonce":37354},{"type":"v1","addr":"172.21.15.62:6830","nonce":37354}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6825","nonce":37354},{"type":"v1","addr":"172.21.15.62:6827","nonce":37354}]},"public_addr":"172.21.15.62:6820/37354","cluster_addr":"172.21.15.62:6823/37354","heartbeat_back_addr":"172.21.15.62:6830/37354","heartbeat_front_addr":"172.21.15.62:6827/37354","state":["exists","up"]},{"osd":7,"uuid":"ac70b2a3-f41b-4bf2-b227-7f2cbb55426d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37355},{"type":"v1","addr":"172.21.15.62:6809","nonce":37355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37355},{"type":"v1","addr":"172.21.15.62:6811","nonce":37355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":37355},{"type":"v1","addr":"172.21.15.62:6815","nonce":37355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":37355},{"type":"v1","addr":"172.21.15.62:6813","nonce":37355}]},"public_addr":"172.21.15.62:6809/37355","cluster_addr":"172.21.15.62:6811/37355","heartbeat_back_addr":"172.21.15.62:6815/37355","heartbeat_front_addr":"172.21.15.62:6813/37355","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.368896+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.107273+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.276888+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.287502+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.124791+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.251937+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.255679+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.232797+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-05-22T15:42:23.480 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:23.476+0000 7f9dd7b26640 1 -- 172.21.15.16:0/789052522 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f9db40654a0 msgr2=0x7f9db4067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:23.480 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:23.476+0000 7f9dd7b26640 1 --2- 172.21.15.16:0/789052522 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f9db40654a0 0x7f9db4067950 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f9dc40017b0 tx=0x7f9dc40015a0 comp rx=0 tx=0).stop 2023-05-22T15:42:23.480 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:23.476+0000 7f9dd7b26640 1 -- 172.21.15.16:0/789052522 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9dd00c59f0 msgr2=0x7f9dd014f340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:23.480 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:23.476+0000 7f9dd7b26640 1 --2- 172.21.15.16:0/789052522 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9dd00c59f0 0x7f9dd014f340 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 crypto rx=0x7f9dd00cd930 tx=0x7f9dc8011550 comp rx=0 tx=0).stop 2023-05-22T15:42:23.481 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:23.476+0000 7f9dd7b26640 1 -- 172.21.15.16:0/789052522 shutdown_connections 2023-05-22T15:42:23.481 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:23.476+0000 7f9dd7b26640 1 --2- 172.21.15.16:0/789052522 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9dd00c59f0 0x7f9dd014f340 unknown :-1 s=CLOSED pgs=1101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:23.481 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:23.476+0000 7f9dd7b26640 1 --2- 172.21.15.16:0/789052522 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9dd00c5580 0x7f9dd014ee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:23.481 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:23.476+0000 7f9dd7b26640 1 --2- 172.21.15.16:0/789052522 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f9db40654a0 0x7f9db4067950 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:23.481 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:23.476+0000 7f9dd7b26640 1 --2- 172.21.15.16:0/789052522 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9dbc0047d0 0x7f9dd01ef390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:23.481 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:23.476+0000 7f9dd7b26640 1 -- 172.21.15.16:0/789052522 >> 172.21.15.16:0/789052522 conn(0x7f9dd006f5c0 msgr2=0x7f9dd00cfe30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:23.482 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:23.476+0000 7f9dd7b26640 1 -- 172.21.15.16:0/789052522 shutdown_connections 2023-05-22T15:42:23.482 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:23.476+0000 7f9dd7b26640 1 -- 172.21.15.16:0/789052522 wait complete. 2023-05-22T15:42:23.497 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-05-22T15:42:23.629 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:23.624+0000 7f7287b59640 1 Processor -- start 2023-05-22T15:42:23.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:23.624+0000 7f7287b59640 1 -- start start 2023-05-22T15:42:23.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:23.628+0000 7f7287b59640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f72801563c0 0x7f72800c7b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:23.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:23.628+0000 7f7287b59640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f728015c210 con 0x7f72800cc6c0 2023-05-22T15:42:23.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:23.628+0000 7f7287b59640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f728015c480 con 0x7f72800c8080 2023-05-22T15:42:23.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:23.628+0000 7f7287b59640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72800c8520 con 0x7f72801563c0 2023-05-22T15:42:23.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:23.628+0000 7f72850f4640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f72800cc6c0 0x7f728006c220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:33460/0 (socket says 172.21.15.16:33460) 2023-05-22T15:42:23.632 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:23.628+0000 7f72850f4640 1 -- 172.21.15.16:0/3532579555 learned_addr learned my addr 172.21.15.16:0/3532579555 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:42:23.633 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:23.628+0000 7f72858f5640 1 --2- 172.21.15.16:0/3532579555 >> v2:172.21.15.62:3300/0 conn(0x7f72801563c0 0x7f72800c7b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:23.633 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:23.628+0000 7f72848f3640 1 -- 172.21.15.16:0/3532579555 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1898233036 0 0) 0x7f728015c210 con 0x7f72800cc6c0 2023-05-22T15:42:23.633 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:23.628+0000 7f72848f3640 1 -- 172.21.15.16:0/3532579555 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7268003530 con 0x7f72800cc6c0 2023-05-22T15:42:23.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:23.628+0000 7f72848f3640 1 -- 172.21.15.16:0/3532579555 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1310142772 0 0) 0x7f7268003530 con 0x7f72800cc6c0 2023-05-22T15:42:23.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:23.628+0000 7f72848f3640 1 -- 172.21.15.16:0/3532579555 >> v2:172.21.15.62:3300/0 conn(0x7f72801563c0 msgr2=0x7f72800c7b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:42:23.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:23.628+0000 7f72848f3640 1 --2- 172.21.15.16:0/3532579555 >> v2:172.21.15.62:3300/0 conn(0x7f72801563c0 0x7f72800c7b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:23.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:23.628+0000 7f72848f3640 1 -- 172.21.15.16:0/3532579555 >> v1:172.21.15.62:6789/0 conn(0x7f72800c8080 legacy=0x7f728015f6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:23.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:23.628+0000 7f72848f3640 1 -- 172.21.15.16:0/3532579555 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f728015c760 con 0x7f72800cc6c0 2023-05-22T15:42:23.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:23.628+0000 7f72848f3640 1 -- 172.21.15.16:0/3532579555 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7f7270001a00 con 0x7f72800cc6c0 2023-05-22T15:42:23.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:23.628+0000 7f72848f3640 1 -- 172.21.15.16:0/3532579555 >> v1:172.21.15.16:6789/0 conn(0x7f72800cc6c0 legacy=0x7f728006c220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:23.635 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:23.628+0000 7f72848f3640 1 --2- 172.21.15.16:0/3532579555 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7268003980 0x7f7268003d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:23.635 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:23.628+0000 7f72848f3640 1 --2- 172.21.15.16:0/3532579555 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f72680047d0 0x7f7268004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:23.635 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:23.628+0000 7f72848f3640 1 --2- 172.21.15.16:0/3532579555 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7268005180 0x7f726800b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:23.635 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:23.628+0000 7f72848f3640 1 -- 172.21.15.16:0/3532579555 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f726800bfb0 con 0x7f7268003980 2023-05-22T15:42:23.635 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:23.628+0000 7f72848f3640 1 -- 172.21.15.16:0/3532579555 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f726800c120 con 0x7f7268005180 2023-05-22T15:42:23.636 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:23.628+0000 7f72848f3640 1 -- 172.21.15.16:0/3532579555 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f726800c2e0 con 0x7f72680047d0 2023-05-22T15:42:23.636 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:23.628+0000 7f72858f5640 1 --2- 172.21.15.16:0/3532579555 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7268003980 0x7f7268003d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:23.636 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:23.628+0000 7f72850f4640 1 --2- 172.21.15.16:0/3532579555 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f72680047d0 0x7f7268004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:23.636 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:23.628+0000 7f72860f6640 1 --2- 172.21.15.16:0/3532579555 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7268005180 0x7f726800b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:23.636 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:23.628+0000 7f72858f5640 1 -- 172.21.15.16:0/3532579555 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f72680047d0 msgr2=0x7f7268004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:42:23.636 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:23.628+0000 7f72858f5640 1 --2- 172.21.15.16:0/3532579555 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f72680047d0 0x7f7268004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:23.636 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:23.628+0000 7f72858f5640 1 -- 172.21.15.16:0/3532579555 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7268005180 msgr2=0x7f726800b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:42:23.637 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:23.628+0000 7f72858f5640 1 --2- 172.21.15.16:0/3532579555 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7268005180 0x7f726800b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:23.637 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:23.628+0000 7f72858f5640 1 -- 172.21.15.16:0/3532579555 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7268003530 con 0x7f7268003980 2023-05-22T15:42:23.637 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:23.628+0000 7f72860f6640 1 --2- 172.21.15.16:0/3532579555 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7268005180 0x7f726800b990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T15:42:23.637 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:23.628+0000 7f72860f6640 1 -- 172.21.15.16:0/3532579555 reap_dead start 2023-05-22T15:42:23.639 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:23.632+0000 7f72858f5640 1 --2- 172.21.15.16:0/3532579555 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7268003980 0x7f7268003d50 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7f7278009ae0 tx=0x7f727801ca50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8dfa19c2857c56e6 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:23.639 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:23.632+0000 7f72848f3640 1 -- 172.21.15.16:0/3532579555 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72780a5640 con 0x7f7268003980 2023-05-22T15:42:23.640 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:23.632+0000 7f72848f3640 1 -- 172.21.15.16:0/3532579555 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72780a57a0 con 0x7f7268003980 2023-05-22T15:42:23.640 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:23.632+0000 7f72848f3640 1 -- 172.21.15.16:0/3532579555 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72780a5b50 con 0x7f7268003980 2023-05-22T15:42:23.640 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:23.632+0000 7f7287b59640 1 -- 172.21.15.16:0/3532579555 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7268003980 msgr2=0x7f7268003d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:23.640 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:23.632+0000 7f7287b59640 1 --2- 172.21.15.16:0/3532579555 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7268003980 0x7f7268003d50 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7f7278009ae0 tx=0x7f727801ca50 comp rx=0 tx=0).stop 2023-05-22T15:42:23.640 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:23.632+0000 7f7287b59640 1 -- 172.21.15.16:0/3532579555 shutdown_connections 2023-05-22T15:42:23.641 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:23.632+0000 7f7287b59640 1 --2- 172.21.15.16:0/3532579555 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7268003980 0x7f7268003d50 unknown :-1 s=CLOSED pgs=1103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:23.641 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:23.632+0000 7f7287b59640 1 -- 172.21.15.16:0/3532579555 >> 172.21.15.16:0/3532579555 conn(0x7f72800b1110 msgr2=0x7f72800ce830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:23.641 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:23.632+0000 7f7287b59640 1 -- 172.21.15.16:0/3532579555 shutdown_connections 2023-05-22T15:42:23.641 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:23.632+0000 7f7287b59640 1 -- 172.21.15.16:0/3532579555 wait complete. 2023-05-22T15:42:23.641 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:23.632+0000 7f7287b59640 1 Processor -- start 2023-05-22T15:42:23.641 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:23.632+0000 7f7287b59640 1 -- start start 2023-05-22T15:42:23.642 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:23.632+0000 7f7287b59640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7268003980 0x7f72800ca5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:23.642 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:23.632+0000 7f7287b59640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f72801567c0 0x7f72801f0b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:23.642 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:23.632+0000 7f7287b59640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f728015ad70 0x7f728015b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:23.642 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:23.632+0000 7f7287b59640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72800cab10 con 0x7f728015ad70 2023-05-22T15:42:23.642 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:23.632+0000 7f7287b59640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f728015b720 con 0x7f72801567c0 2023-05-22T15:42:23.642 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:23.632+0000 7f7287b59640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f728015e100 con 0x7f7268003980 2023-05-22T15:42:23.642 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:23.632+0000 7f72860f6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f728015ad70 0x7f728015b1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:23.643 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:23.632+0000 7f72860f6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f728015ad70 0x7f728015b1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34774/0 (socket says 172.21.15.16:34774) 2023-05-22T15:42:23.643 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:23.632+0000 7f72860f6640 1 -- 172.21.15.16:0/2843593605 learned_addr learned my addr 172.21.15.16:0/2843593605 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:42:23.643 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:23.632+0000 7f72858f5640 1 --2- 172.21.15.16:0/2843593605 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7268003980 0x7f72800ca5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:23.643 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:23.632+0000 7f72850f4640 1 --2- 172.21.15.16:0/2843593605 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f72801567c0 0x7f72801f0b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:23.643 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:23.636+0000 7f72860f6640 1 -- 172.21.15.16:0/2843593605 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7268003980 msgr2=0x7f72800ca5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:23.643 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:23.636+0000 7f72860f6640 1 --2- 172.21.15.16:0/2843593605 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7268003980 0x7f72800ca5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:23.643 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:23.636+0000 7f72860f6640 1 -- 172.21.15.16:0/2843593605 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f72801567c0 msgr2=0x7f72801f0b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:23.644 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:23.636+0000 7f72860f6640 1 --2- 172.21.15.16:0/2843593605 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f72801567c0 0x7f72801f0b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:23.644 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:23.636+0000 7f72860f6640 1 -- 172.21.15.16:0/2843593605 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f728015cd20 con 0x7f728015ad70 2023-05-22T15:42:23.644 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:23.636+0000 7f72860f6640 1 --2- 172.21.15.16:0/2843593605 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f728015ad70 0x7f728015b1e0 secure :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 crypto rx=0x7f7274007ad0 tx=0x7f7274011ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:23.644 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:23.636+0000 7f727e7fc640 1 -- 172.21.15.16:0/2843593605 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72740137a0 con 0x7f728015ad70 2023-05-22T15:42:23.644 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:23.636+0000 7f727e7fc640 1 -- 172.21.15.16:0/2843593605 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7274013900 con 0x7f728015ad70 2023-05-22T15:42:23.644 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:23.636+0000 7f7287b59640 1 -- 172.21.15.16:0/2843593605 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f728015ca40 con 0x7f728015ad70 2023-05-22T15:42:23.645 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:23.636+0000 7f727e7fc640 1 -- 172.21.15.16:0/2843593605 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7274013c90 con 0x7f728015ad70 2023-05-22T15:42:23.645 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:23.636+0000 7f7287b59640 1 -- 172.21.15.16:0/2843593605 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72800c7b40 con 0x7f728015ad70 2023-05-22T15:42:23.645 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:23.636+0000 7f7287b59640 1 -- 172.21.15.16:0/2843593605 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f728004dbe0 con 0x7f728015ad70 2023-05-22T15:42:23.646 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:23.636+0000 7f727e7fc640 1 -- 172.21.15.16:0/2843593605 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f72740203b0 con 0x7f728015ad70 2023-05-22T15:42:23.646 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:23.636+0000 7f727e7fc640 1 --2- 172.21.15.16:0/2843593605 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f726c0654a0 0x7f726c067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:23.646 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:23.636+0000 7f727e7fc640 1 -- 172.21.15.16:0/2843593605 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f7274085310 con 0x7f728015ad70 2023-05-22T15:42:23.646 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:23.636+0000 7f72858f5640 1 --2- 172.21.15.16:0/2843593605 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f726c0654a0 0x7f726c067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:23.646 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:23.636+0000 7f72858f5640 1 --2- 172.21.15.16:0/2843593605 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f726c0654a0 0x7f726c067950 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f72780af000 tx=0x7f72780046d0 comp rx=0 tx=0).ready entity=mgr.6765 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:23.646 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:23.640+0000 7f727e7fc640 1 -- 172.21.15.16:0/2843593605 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f7274052d90 con 0x7f728015ad70 2023-05-22T15:42:23.841 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:23.836+0000 7f7287b59640 1 -- 172.21.15.16:0/2843593605 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f72800cf080 con 0x7f728015ad70 2023-05-22T15:42:24.835 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:24.832+0000 7f727e7fc640 1 -- 172.21.15.16:0/2843593605 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v29) v1 ==== 118+0+0 (secure 0 0 0) 0x7f72740563b0 con 0x7f728015ad70 2023-05-22T15:42:24.839 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:24.836+0000 7f7287b59640 1 -- 172.21.15.16:0/2843593605 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f726c0654a0 msgr2=0x7f726c067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:24.839 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:24.836+0000 7f7287b59640 1 --2- 172.21.15.16:0/2843593605 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f726c0654a0 0x7f726c067950 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f72780af000 tx=0x7f72780046d0 comp rx=0 tx=0).stop 2023-05-22T15:42:24.840 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:24.836+0000 7f7287b59640 1 -- 172.21.15.16:0/2843593605 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f728015ad70 msgr2=0x7f728015b1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:24.840 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:24.836+0000 7f7287b59640 1 --2- 172.21.15.16:0/2843593605 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f728015ad70 0x7f728015b1e0 secure :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 crypto rx=0x7f7274007ad0 tx=0x7f7274011ce0 comp rx=0 tx=0).stop 2023-05-22T15:42:24.840 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:24.836+0000 7f7287b59640 1 -- 172.21.15.16:0/2843593605 shutdown_connections 2023-05-22T15:42:24.840 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:24.836+0000 7f7287b59640 1 --2- 172.21.15.16:0/2843593605 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f728015ad70 0x7f728015b1e0 unknown :-1 s=CLOSED pgs=1104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:24.840 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:24.836+0000 7f7287b59640 1 --2- 172.21.15.16:0/2843593605 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f72801567c0 0x7f72801f0b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:24.840 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:24.836+0000 7f7287b59640 1 --2- 172.21.15.16:0/2843593605 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f726c0654a0 0x7f726c067950 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:24.840 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:24.836+0000 7f7287b59640 1 --2- 172.21.15.16:0/2843593605 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7268003980 0x7f72800ca5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:24.841 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:24.836+0000 7f7287b59640 1 -- 172.21.15.16:0/2843593605 >> 172.21.15.16:0/2843593605 conn(0x7f72800b1110 msgr2=0x7f728006c1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:24.841 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:24.836+0000 7f7287b59640 1 -- 172.21.15.16:0/2843593605 shutdown_connections 2023-05-22T15:42:24.841 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:24.836+0000 7f7287b59640 1 -- 172.21.15.16:0/2843593605 wait complete. 2023-05-22T15:42:24.862 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2023-05-22T15:42:24.863 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-05-22T15:42:24.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:24.992+0000 7fa1f87b1640 1 Processor -- start 2023-05-22T15:42:24.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:24.992+0000 7fa1f87b1640 1 -- start start 2023-05-22T15:42:24.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:24.992+0000 7fa1f87b1640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa1f0070c70 0x7fa1f0071530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:24.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:24.992+0000 7fa1f87b1640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fa1f015c580 0x7fa1f006fba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:24.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:24.992+0000 7fa1f87b1640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fa1f006b6c0 0x7fa1f006bb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:24.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:24.992+0000 7fa1f87b1640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1f015d520 con 0x7fa1f015c580 2023-05-22T15:42:24.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:24.992+0000 7fa1f87b1640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1f006c040 con 0x7fa1f006b6c0 2023-05-22T15:42:24.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:24.992+0000 7fa1f87b1640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1f00c8b10 con 0x7fa1f0070c70 2023-05-22T15:42:24.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:24.992+0000 7fa1f5d4c640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fa1f015c580 0x7fa1f006fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:24.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:24.992+0000 7fa1f5d4c640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fa1f015c580 0x7fa1f006fba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34778/0 (socket says 172.21.15.16:34778) 2023-05-22T15:42:24.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:24.992+0000 7fa1f5d4c640 1 -- 172.21.15.16:0/1856689231 learned_addr learned my addr 172.21.15.16:0/1856689231 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:42:24.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:24.992+0000 7fa1f654d640 1 --2- 172.21.15.16:0/1856689231 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa1f0070c70 0x7fa1f0071530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:24.998 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:24.992+0000 7fa1f6d4e640 1 --2- 172.21.15.16:0/1856689231 >> v2:172.21.15.62:3300/0 conn(0x7fa1f006b6c0 0x7fa1f006bb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:24.998 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:24.992+0000 7fa1f5d4c640 1 -- 172.21.15.16:0/1856689231 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa1f0070c70 msgr2=0x7fa1f0071530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:42:24.998 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:24.992+0000 7fa1f5d4c640 1 --2- 172.21.15.16:0/1856689231 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa1f0070c70 0x7fa1f0071530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:24.998 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:24.992+0000 7fa1f5d4c640 1 -- 172.21.15.16:0/1856689231 >> v2:172.21.15.62:3300/0 conn(0x7fa1f006b6c0 msgr2=0x7fa1f006bb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:42:24.998 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:24.992+0000 7fa1f5d4c640 1 --2- 172.21.15.16:0/1856689231 >> v2:172.21.15.62:3300/0 conn(0x7fa1f006b6c0 0x7fa1f006bb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:24.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:24.992+0000 7fa1f5d4c640 1 -- 172.21.15.16:0/1856689231 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1f015c9d0 con 0x7fa1f015c580 2023-05-22T15:42:24.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:24.996+0000 7fa1f5d4c640 1 --2- 172.21.15.16:0/1856689231 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa1f015c580 0x7fa1f006fba0 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 crypto rx=0x7fa1f006f570 tx=0x7fa1e001c470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6723e0caac684c13 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:25.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:24.996+0000 7fa1f554b640 1 -- 172.21.15.16:0/1856689231 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1e00a7070 con 0x7fa1f015c580 2023-05-22T15:42:25.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:24.996+0000 7fa1f554b640 1 -- 172.21.15.16:0/1856689231 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1e00ab430 con 0x7fa1f015c580 2023-05-22T15:42:25.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:24.996+0000 7fa1f554b640 1 -- 172.21.15.16:0/1856689231 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1e00ab7c0 con 0x7fa1f015c580 2023-05-22T15:42:25.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:24.996+0000 7fa1f87b1640 1 -- 172.21.15.16:0/1856689231 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa1f015c580 msgr2=0x7fa1f006fba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:25.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:24.996+0000 7fa1f87b1640 1 --2- 172.21.15.16:0/1856689231 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa1f015c580 0x7fa1f006fba0 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 crypto rx=0x7fa1f006f570 tx=0x7fa1e001c470 comp rx=0 tx=0).stop 2023-05-22T15:42:25.001 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:24.996+0000 7fa1f87b1640 1 -- 172.21.15.16:0/1856689231 shutdown_connections 2023-05-22T15:42:25.001 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:24.996+0000 7fa1f87b1640 1 --2- 172.21.15.16:0/1856689231 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa1f015c580 0x7fa1f006fba0 unknown :-1 s=CLOSED pgs=1105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:25.001 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:24.996+0000 7fa1f87b1640 1 --2- 172.21.15.16:0/1856689231 >> v2:172.21.15.62:3300/0 conn(0x7fa1f006b6c0 0x7fa1f006bb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:25.001 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:24.996+0000 7fa1f87b1640 1 --2- 172.21.15.16:0/1856689231 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa1f0070c70 0x7fa1f0071530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:25.001 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:24.996+0000 7fa1f87b1640 1 -- 172.21.15.16:0/1856689231 >> 172.21.15.16:0/1856689231 conn(0x7fa1f0069720 msgr2=0x7fa1f0069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:25.002 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:24.996+0000 7fa1f87b1640 1 -- 172.21.15.16:0/1856689231 shutdown_connections 2023-05-22T15:42:25.002 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:24.996+0000 7fa1f87b1640 1 -- 172.21.15.16:0/1856689231 wait complete. 2023-05-22T15:42:25.004 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:24.996+0000 7fa1f87b1640 1 Processor -- start 2023-05-22T15:42:25.004 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:24.996+0000 7fa1f87b1640 1 -- start start 2023-05-22T15:42:25.004 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:24.996+0000 7fa1f87b1640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa1f006b6c0 0x7fa1f0157510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:25.004 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:24.996+0000 7fa1f87b1640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa1f0070c70 0x7fa1f00c30b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:25.004 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:24.996+0000 7fa1f87b1640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa1f015c580 0x7fa1f00caae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:25.005 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:24.996+0000 7fa1f87b1640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1f00c46f0 con 0x7fa1f0070c70 2023-05-22T15:42:25.005 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:24.996+0000 7fa1f87b1640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1f0157a50 con 0x7fa1f006b6c0 2023-05-22T15:42:25.005 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:24.996+0000 7fa1f87b1640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1f00c3ae0 con 0x7fa1f015c580 2023-05-22T15:42:25.005 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:24.996+0000 7fa1f5d4c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa1f0070c70 0x7fa1f00c30b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:25.005 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:24.996+0000 7fa1f5d4c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa1f0070c70 0x7fa1f00c30b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34792/0 (socket says 172.21.15.16:34792) 2023-05-22T15:42:25.005 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:24.996+0000 7fa1f5d4c640 1 -- 172.21.15.16:0/3865529985 learned_addr learned my addr 172.21.15.16:0/3865529985 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:42:25.005 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:24.996+0000 7fa1f654d640 1 --2- 172.21.15.16:0/3865529985 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa1f006b6c0 0x7fa1f0157510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:25.006 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:24.996+0000 7fa1f6d4e640 1 --2- 172.21.15.16:0/3865529985 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa1f015c580 0x7fa1f00caae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:25.006 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:24.996+0000 7fa1f5d4c640 1 -- 172.21.15.16:0/3865529985 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa1f015c580 msgr2=0x7fa1f00caae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:25.006 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:24.996+0000 7fa1f5d4c640 1 --2- 172.21.15.16:0/3865529985 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa1f015c580 0x7fa1f00caae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:25.006 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:24.996+0000 7fa1f5d4c640 1 -- 172.21.15.16:0/3865529985 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa1f006b6c0 msgr2=0x7fa1f0157510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:25.006 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:24.996+0000 7fa1f5d4c640 1 --2- 172.21.15.16:0/3865529985 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa1f006b6c0 0x7fa1f0157510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:25.006 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:24.996+0000 7fa1f5d4c640 1 -- 172.21.15.16:0/3865529985 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1e00094d0 con 0x7fa1f0070c70 2023-05-22T15:42:25.006 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:25.000+0000 7fa1f5d4c640 1 --2- 172.21.15.16:0/3865529985 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa1f0070c70 0x7fa1f00c30b0 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 crypto rx=0x7fa1e001ab20 tx=0x7fa1e0019210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:25.007 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:25.000+0000 7fa1e6ffd640 1 -- 172.21.15.16:0/3865529985 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1e00b4830 con 0x7fa1f0070c70 2023-05-22T15:42:25.007 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:25.000+0000 7fa1e6ffd640 1 -- 172.21.15.16:0/3865529985 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1e00b4990 con 0x7fa1f0070c70 2023-05-22T15:42:25.007 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:25.000+0000 7fa1f87b1640 1 -- 172.21.15.16:0/3865529985 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1f015d240 con 0x7fa1f0070c70 2023-05-22T15:42:25.007 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:25.000+0000 7fa1e6ffd640 1 -- 172.21.15.16:0/3865529985 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1e00b4d00 con 0x7fa1f0070c70 2023-05-22T15:42:25.007 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:25.000+0000 7fa1f87b1640 1 -- 172.21.15.16:0/3865529985 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1f015cf60 con 0x7fa1f0070c70 2023-05-22T15:42:25.007 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:25.000+0000 7fa1f87b1640 1 -- 172.21.15.16:0/3865529985 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1f00cc510 con 0x7fa1f0070c70 2023-05-22T15:42:25.007 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:25.000+0000 7fa1e6ffd640 1 -- 172.21.15.16:0/3865529985 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) v1 ==== 84332+0+0 (secure 0 0 0) 0x7fa1e00c0460 con 0x7fa1f0070c70 2023-05-22T15:42:25.008 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:25.000+0000 7fa1e6ffd640 1 --2- 172.21.15.16:0/3865529985 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fa1d40654a0 0x7fa1d4067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:25.008 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:25.000+0000 7fa1e6ffd640 1 -- 172.21.15.16:0/3865529985 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 6382+0+0 (secure 0 0 0) 0x7fa1e011d3e0 con 0x7fa1f0070c70 2023-05-22T15:42:25.009 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:25.000+0000 7fa1f654d640 1 --2- 172.21.15.16:0/3865529985 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fa1d40654a0 0x7fa1d4067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:25.009 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:25.000+0000 7fa1f654d640 1 --2- 172.21.15.16:0/3865529985 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fa1d40654a0 0x7fa1d4067950 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fa1f00c1ae0 tx=0x7fa1ec008040 comp rx=0 tx=0).ready entity=mgr.6765 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:25.009 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:25.004+0000 7fa1e6ffd640 1 -- 172.21.15.16:0/3865529985 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fa1e00eaca0 con 0x7fa1f0070c70 2023-05-22T15:42:25.199 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:25.196+0000 7fa1f87b1640 1 -- 172.21.15.16:0/3865529985 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fa1f004dbe0 con 0x7fa1f0070c70 2023-05-22T15:42:25.848 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:25.844+0000 7fa1e6ffd640 1 -- 172.21.15.16:0/3865529985 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v30) v1 ==== 122+0+0 (secure 0 0 0) 0x7fa1e00b4e60 con 0x7fa1f0070c70 2023-05-22T15:42:25.851 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:25.848+0000 7fa1f87b1640 1 -- 172.21.15.16:0/3865529985 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fa1d40654a0 msgr2=0x7fa1d4067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:25.851 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:25.848+0000 7fa1f87b1640 1 --2- 172.21.15.16:0/3865529985 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fa1d40654a0 0x7fa1d4067950 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fa1f00c1ae0 tx=0x7fa1ec008040 comp rx=0 tx=0).stop 2023-05-22T15:42:25.851 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:25.848+0000 7fa1f87b1640 1 -- 172.21.15.16:0/3865529985 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa1f0070c70 msgr2=0x7fa1f00c30b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:25.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:25.848+0000 7fa1f87b1640 1 --2- 172.21.15.16:0/3865529985 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa1f0070c70 0x7fa1f00c30b0 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 crypto rx=0x7fa1e001ab20 tx=0x7fa1e0019210 comp rx=0 tx=0).stop 2023-05-22T15:42:25.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:25.848+0000 7fa1f87b1640 1 -- 172.21.15.16:0/3865529985 shutdown_connections 2023-05-22T15:42:25.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:25.848+0000 7fa1f87b1640 1 --2- 172.21.15.16:0/3865529985 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fa1d40654a0 0x7fa1d4067950 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:25.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:25.848+0000 7fa1f87b1640 1 --2- 172.21.15.16:0/3865529985 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa1f015c580 0x7fa1f00caae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:25.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:25.848+0000 7fa1f87b1640 1 --2- 172.21.15.16:0/3865529985 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa1f0070c70 0x7fa1f00c30b0 unknown :-1 s=CLOSED pgs=1106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:25.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:25.848+0000 7fa1f87b1640 1 --2- 172.21.15.16:0/3865529985 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa1f006b6c0 0x7fa1f0157510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:25.853 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:25.848+0000 7fa1f87b1640 1 -- 172.21.15.16:0/3865529985 >> 172.21.15.16:0/3865529985 conn(0x7fa1f0069720 msgr2=0x7fa1f0069af0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:25.853 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:25.848+0000 7fa1f87b1640 1 -- 172.21.15.16:0/3865529985 shutdown_connections 2023-05-22T15:42:25.853 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:25.848+0000 7fa1f87b1640 1 -- 172.21.15.16:0/3865529985 wait complete. 2023-05-22T15:42:25.871 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2023-05-22T15:42:25.872 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-05-22T15:42:26.001 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:25.996+0000 7f48a656c640 1 Processor -- start 2023-05-22T15:42:26.002 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:25.996+0000 7f48a656c640 1 -- start start 2023-05-22T15:42:26.002 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.000+0000 7f48a656c640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f48a015fc50 0x7f48a0160020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:26.003 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.000+0000 7f48a656c640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f48a015c920 0x7f48a00d0030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:26.003 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.000+0000 7f48a656c640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48a015c490 con 0x7f48a015cd70 2023-05-22T15:42:26.003 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.000+0000 7f48a656c640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48a00cf7a0 con 0x7f48a015fc50 2023-05-22T15:42:26.003 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.000+0000 7f48a656c640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48a015c140 con 0x7f48a015c920 2023-05-22T15:42:26.003 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.000+0000 7f489ffff640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f48a015fc50 0x7f48a0160020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:26.004 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.000+0000 7f489ffff640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f48a015fc50 0x7f48a0160020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34802/0 (socket says 172.21.15.16:34802) 2023-05-22T15:42:26.004 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.000+0000 7f489ffff640 1 -- 172.21.15.16:0/2786142742 learned_addr learned my addr 172.21.15.16:0/2786142742 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:42:26.004 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.000+0000 7f489f7fe640 1 --2- 172.21.15.16:0/2786142742 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f48a015c920 0x7f48a00d0030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:26.004 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.000+0000 7f489ffff640 1 -- 172.21.15.16:0/2786142742 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f48a015c920 msgr2=0x7f48a00d0030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:42:26.005 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.000+0000 7f489ffff640 1 --2- 172.21.15.16:0/2786142742 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f48a015c920 0x7f48a00d0030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:26.005 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.000+0000 7f489ffff640 1 -- 172.21.15.16:0/2786142742 >> v1:172.21.15.62:6789/0 conn(0x7f48a015cd70 legacy=0x7f48a00b3ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:42:26.005 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.000+0000 7f489ffff640 1 -- 172.21.15.16:0/2786142742 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48a00ccb50 con 0x7f48a015fc50 2023-05-22T15:42:26.006 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.000+0000 7f489ffff640 1 --2- 172.21.15.16:0/2786142742 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f48a015fc50 0x7f48a0160020 secure :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 crypto rx=0x7f4888009870 tx=0x7f488801c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff136258a6f1d082 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:26.006 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.000+0000 7f489effd640 1 -- 172.21.15.16:0/2786142742 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48880aa5f0 con 0x7f48a015fc50 2023-05-22T15:42:26.006 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.004+0000 7f489effd640 1 -- 172.21.15.16:0/2786142742 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48880aa750 con 0x7f48a015fc50 2023-05-22T15:42:26.006 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.004+0000 7f489effd640 1 -- 172.21.15.16:0/2786142742 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48880aaae0 con 0x7f48a015fc50 2023-05-22T15:42:26.007 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.004+0000 7f48a656c640 1 -- 172.21.15.16:0/2786142742 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f48a015fc50 msgr2=0x7f48a0160020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:26.007 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.004+0000 7f48a656c640 1 --2- 172.21.15.16:0/2786142742 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f48a015fc50 0x7f48a0160020 secure :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 crypto rx=0x7f4888009870 tx=0x7f488801c4d0 comp rx=0 tx=0).stop 2023-05-22T15:42:26.007 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.004+0000 7f48a656c640 1 -- 172.21.15.16:0/2786142742 shutdown_connections 2023-05-22T15:42:26.008 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.004+0000 7f48a656c640 1 --2- 172.21.15.16:0/2786142742 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f48a015c920 0x7f48a00d0030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:26.008 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.004+0000 7f48a656c640 1 --2- 172.21.15.16:0/2786142742 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f48a015fc50 0x7f48a0160020 unknown :-1 s=CLOSED pgs=1107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:26.008 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.004+0000 7f48a656c640 1 -- 172.21.15.16:0/2786142742 >> 172.21.15.16:0/2786142742 conn(0x7f48a00657f0 msgr2=0x7f48a0070e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:26.008 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.004+0000 7f48a656c640 1 -- 172.21.15.16:0/2786142742 shutdown_connections 2023-05-22T15:42:26.008 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.004+0000 7f48a656c640 1 -- 172.21.15.16:0/2786142742 wait complete. 2023-05-22T15:42:26.010 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.004+0000 7f48a656c640 1 Processor -- start 2023-05-22T15:42:26.010 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.004+0000 7f48a656c640 1 -- start start 2023-05-22T15:42:26.010 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.004+0000 7f48a656c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f48a015c920 0x7f48a00b3200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:26.010 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.004+0000 7f48a656c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f48a015cd70 0x7f48a014aca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:26.011 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.004+0000 7f48a656c640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f48a015fc50 0x7f48a014c5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:26.011 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.004+0000 7f48a656c640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48a014b540 con 0x7f48a015c920 2023-05-22T15:42:26.011 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.004+0000 7f48a656c640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48a014b820 con 0x7f48a015cd70 2023-05-22T15:42:26.011 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.004+0000 7f48a656c640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48a014bb00 con 0x7f48a015fc50 2023-05-22T15:42:26.011 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.004+0000 7f489ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f48a015c920 0x7f48a00b3200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:26.011 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.004+0000 7f489ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f48a015c920 0x7f48a00b3200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34818/0 (socket says 172.21.15.16:34818) 2023-05-22T15:42:26.012 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.004+0000 7f489ffff640 1 -- 172.21.15.16:0/2015638448 learned_addr learned my addr 172.21.15.16:0/2015638448 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:42:26.012 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.004+0000 7f48a4b09640 1 --2- 172.21.15.16:0/2015638448 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f48a015fc50 0x7f48a014c5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:26.012 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.004+0000 7f489f7fe640 1 --2- 172.21.15.16:0/2015638448 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f48a015cd70 0x7f48a014aca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:26.012 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.004+0000 7f489ffff640 1 -- 172.21.15.16:0/2015638448 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f48a015fc50 msgr2=0x7f48a014c5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:26.012 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.004+0000 7f489ffff640 1 --2- 172.21.15.16:0/2015638448 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f48a015fc50 0x7f48a014c5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:26.012 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.004+0000 7f489ffff640 1 -- 172.21.15.16:0/2015638448 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f48a015cd70 msgr2=0x7f48a014aca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:26.013 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.004+0000 7f489ffff640 1 --2- 172.21.15.16:0/2015638448 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f48a015cd70 0x7f48a014aca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:26.013 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.004+0000 7f489ffff640 1 -- 172.21.15.16:0/2015638448 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48880094d0 con 0x7f48a015c920 2023-05-22T15:42:26.013 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.004+0000 7f48a4b09640 1 --2- 172.21.15.16:0/2015638448 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f48a015fc50 0x7f48a014c5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:42:26.013 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.004+0000 7f489ffff640 1 --2- 172.21.15.16:0/2015638448 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f48a015c920 0x7f48a00b3200 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7f48880ac040 tx=0x7f488801ad10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:26.013 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.004+0000 7f489cff9640 1 -- 172.21.15.16:0/2015638448 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48880b3800 con 0x7f48a015c920 2023-05-22T15:42:26.013 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.004+0000 7f489cff9640 1 -- 172.21.15.16:0/2015638448 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48880b3960 con 0x7f48a015c920 2023-05-22T15:42:26.014 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.004+0000 7f48a656c640 1 -- 172.21.15.16:0/2015638448 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48a0160560 con 0x7f48a015c920 2023-05-22T15:42:26.014 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.004+0000 7f489cff9640 1 -- 172.21.15.16:0/2015638448 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48880b3d10 con 0x7f48a015c920 2023-05-22T15:42:26.014 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.004+0000 7f48a656c640 1 -- 172.21.15.16:0/2015638448 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48a00d0980 con 0x7f48a015c920 2023-05-22T15:42:26.015 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.008+0000 7f48a656c640 1 -- 172.21.15.16:0/2015638448 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48a0168ad0 con 0x7f48a015c920 2023-05-22T15:42:26.015 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.008+0000 7f489cff9640 1 -- 172.21.15.16:0/2015638448 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f48880c9380 con 0x7f48a015c920 2023-05-22T15:42:26.015 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.008+0000 7f489cff9640 1 --2- 172.21.15.16:0/2015638448 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f48800654a0 0x7f4880067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:26.016 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.008+0000 7f489cff9640 1 -- 172.21.15.16:0/2015638448 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f48880a7070 con 0x7f48a015c920 2023-05-22T15:42:26.016 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.008+0000 7f489f7fe640 1 --2- 172.21.15.16:0/2015638448 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f48800654a0 0x7f4880067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:26.016 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.008+0000 7f489f7fe640 1 --2- 172.21.15.16:0/2015638448 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f48800654a0 0x7f4880067950 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f4890003fd0 tx=0x7f48900039a0 comp rx=0 tx=0).ready entity=mgr.6765 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:26.017 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.012+0000 7f489cff9640 1 -- 172.21.15.16:0/2015638448 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f48880eb070 con 0x7f48a015c920 2023-05-22T15:42:26.203 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.200+0000 7f48a656c640 1 -- 172.21.15.16:0/2015638448 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f48a004dbe0 con 0x7f48a015c920 2023-05-22T15:42:26.203 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.200+0000 7f489cff9640 1 -- 172.21.15.16:0/2015638448 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+4647 (secure 0 0 0) 0x7f48880ee690 con 0x7f48a015c920 2023-05-22T15:42:26.204 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:42:26.204 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":30,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6833/428835470","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":428835470},{"type":"v1","addr":"172.21.15.62:6833","nonce":428835470}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":25},{"gid":6238,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/4010598958","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":4010598958},{"type":"v1","addr":"172.21.15.62:6835","nonce":4010598958}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":25},{"gid":7008,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/2240458030","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6835","nonce":2240458030},{"type":"v1","addr":"172.21.15.16:6837","nonce":2240458030}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":25}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-05-22T15:42:22.811659+0000","modified":"2023-05-22T15:42:25.830716+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7011},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7011":{"gid":7011,"name":"a","rank":0,"incarnation":26,"state":"up:active","state_seq":3,"addr":"172.21.15.16:6836/1772031282","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1772031282},{"type":"v1","addr":"172.21.15.16:6836","nonce":1772031282}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2023-05-22T15:42:26.204 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 30 2023-05-22T15:42:26.205 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.200+0000 7f48a656c640 1 -- 172.21.15.16:0/2015638448 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f48800654a0 msgr2=0x7f4880067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:26.206 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.200+0000 7f48a656c640 1 --2- 172.21.15.16:0/2015638448 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f48800654a0 0x7f4880067950 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f4890003fd0 tx=0x7f48900039a0 comp rx=0 tx=0).stop 2023-05-22T15:42:26.206 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.200+0000 7f48a656c640 1 -- 172.21.15.16:0/2015638448 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f48a015c920 msgr2=0x7f48a00b3200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:26.206 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.200+0000 7f48a656c640 1 --2- 172.21.15.16:0/2015638448 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f48a015c920 0x7f48a00b3200 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7f48880ac040 tx=0x7f488801ad10 comp rx=0 tx=0).stop 2023-05-22T15:42:26.206 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.200+0000 7f48a656c640 1 -- 172.21.15.16:0/2015638448 shutdown_connections 2023-05-22T15:42:26.206 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.200+0000 7f48a656c640 1 --2- 172.21.15.16:0/2015638448 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f48800654a0 0x7f4880067950 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:26.206 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.200+0000 7f48a656c640 1 --2- 172.21.15.16:0/2015638448 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f48a015fc50 0x7f48a014c5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:26.207 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.200+0000 7f48a656c640 1 --2- 172.21.15.16:0/2015638448 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f48a015cd70 0x7f48a014aca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:26.207 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.200+0000 7f48a656c640 1 --2- 172.21.15.16:0/2015638448 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f48a015c920 0x7f48a00b3200 unknown :-1 s=CLOSED pgs=1108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:26.207 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.200+0000 7f48a656c640 1 -- 172.21.15.16:0/2015638448 >> 172.21.15.16:0/2015638448 conn(0x7f48a00657f0 msgr2=0x7f48a00b3be0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:26.207 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.200+0000 7f48a656c640 1 -- 172.21.15.16:0/2015638448 shutdown_connections 2023-05-22T15:42:26.207 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.200+0000 7f48a656c640 1 -- 172.21.15.16:0/2015638448 wait complete. 2023-05-22T15:42:26.223 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-05-22T15:42:26.338 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.332+0000 7f5ad5618640 1 Processor -- start 2023-05-22T15:42:26.338 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.336+0000 7f5ad5618640 1 -- start start 2023-05-22T15:42:26.339 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.336+0000 7f5ad5618640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f5ad00ca4e0 0x7f5ad015e460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:26.339 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.336+0000 7f5ad5618640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5ad0070df0 0x7f5ad0071210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:26.339 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.336+0000 7f5ad5618640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ad00b0650 con 0x7f5ad00c9b40 2023-05-22T15:42:26.340 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.336+0000 7f5ad5618640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ad00b0370 con 0x7f5ad00ca4e0 2023-05-22T15:42:26.340 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.336+0000 7f5ad5618640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ad015d980 con 0x7f5ad0070df0 2023-05-22T15:42:26.340 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.336+0000 7f5aceffd640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f5ad00ca4e0 0x7f5ad015e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:26.340 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.336+0000 7f5aceffd640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f5ad00ca4e0 0x7f5ad015e460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34834/0 (socket says 172.21.15.16:34834) 2023-05-22T15:42:26.340 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.336+0000 7f5aceffd640 1 -- 172.21.15.16:0/168631128 learned_addr learned my addr 172.21.15.16:0/168631128 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:42:26.341 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.336+0000 7f5acf7fe640 1 --2- 172.21.15.16:0/168631128 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5ad0070df0 0x7f5ad0071210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:26.341 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.336+0000 7f5aceffd640 1 -- 172.21.15.16:0/168631128 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5ad0070df0 msgr2=0x7f5ad0071210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:42:26.341 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.336+0000 7f5aceffd640 1 --2- 172.21.15.16:0/168631128 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5ad0070df0 0x7f5ad0071210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:26.341 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.336+0000 7f5aceffd640 1 -- 172.21.15.16:0/168631128 >> v1:172.21.15.16:6789/0 conn(0x7f5ad00c9b40 legacy=0x7f5ad00afb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:26.342 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.336+0000 7f5acdffb640 1 -- 172.21.15.16:0/168631128 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2346705860 0 0) 0x7f5ad00b0650 con 0x7f5ad00c9b40 2023-05-22T15:42:26.342 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.336+0000 7f5aceffd640 1 -- 172.21.15.16:0/168631128 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ad015dc60 con 0x7f5ad00ca4e0 2023-05-22T15:42:26.342 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.336+0000 7f5acf7fe640 1 --2- 172.21.15.16:0/168631128 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5ad0070df0 0x7f5ad0071210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:42:26.343 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.336+0000 7f5aceffd640 1 --2- 172.21.15.16:0/168631128 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5ad00ca4e0 0x7f5ad015e460 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 crypto rx=0x7f5ac0009e50 tx=0x7f5ac001c9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0927dc8e0b8a75a server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:26.343 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.340+0000 7f5acdffb640 1 -- 172.21.15.16:0/168631128 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ac00a5810 con 0x7f5ad00ca4e0 2023-05-22T15:42:26.343 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.340+0000 7f5acdffb640 1 -- 172.21.15.16:0/168631128 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ac00a5970 con 0x7f5ad00ca4e0 2023-05-22T15:42:26.343 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.340+0000 7f5acdffb640 1 -- 172.21.15.16:0/168631128 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ac00a5d00 con 0x7f5ad00ca4e0 2023-05-22T15:42:26.344 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.340+0000 7f5ad5618640 1 -- 172.21.15.16:0/168631128 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5ad00ca4e0 msgr2=0x7f5ad015e460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:26.344 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.340+0000 7f5ad5618640 1 --2- 172.21.15.16:0/168631128 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5ad00ca4e0 0x7f5ad015e460 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 crypto rx=0x7f5ac0009e50 tx=0x7f5ac001c9e0 comp rx=0 tx=0).stop 2023-05-22T15:42:26.344 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.340+0000 7f5ad5618640 1 -- 172.21.15.16:0/168631128 shutdown_connections 2023-05-22T15:42:26.344 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.340+0000 7f5ad5618640 1 --2- 172.21.15.16:0/168631128 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5ad0070df0 0x7f5ad0071210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:26.344 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.340+0000 7f5ad5618640 1 --2- 172.21.15.16:0/168631128 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5ad00ca4e0 0x7f5ad015e460 unknown :-1 s=CLOSED pgs=1110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:26.344 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.340+0000 7f5ad5618640 1 -- 172.21.15.16:0/168631128 >> 172.21.15.16:0/168631128 conn(0x7f5ad0069720 msgr2=0x7f5ad0069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:26.344 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.340+0000 7f5ad5618640 1 -- 172.21.15.16:0/168631128 shutdown_connections 2023-05-22T15:42:26.344 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.340+0000 7f5ad5618640 1 -- 172.21.15.16:0/168631128 wait complete. 2023-05-22T15:42:26.346 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.340+0000 7f5ad5618640 1 Processor -- start 2023-05-22T15:42:26.346 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.340+0000 7f5ad5618640 1 -- start start 2023-05-22T15:42:26.346 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.340+0000 7f5ad5618640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5ad0070df0 0x7f5ad0159750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:26.347 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.340+0000 7f5ad5618640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5ad00c9b40 0x7f5ad01f6220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:26.347 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.340+0000 7f5ad5618640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5ad00ca4e0 0x7f5ad01f2270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:26.347 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.340+0000 7f5ad5618640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ad01f32c0 con 0x7f5ad00c9b40 2023-05-22T15:42:26.347 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.340+0000 7f5ad5618640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ad01f5e00 con 0x7f5ad0070df0 2023-05-22T15:42:26.347 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.340+0000 7f5ad5618640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ad01f2d00 con 0x7f5ad00ca4e0 2023-05-22T15:42:26.347 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.340+0000 7f5ace7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5ad00c9b40 0x7f5ad01f6220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:26.347 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.340+0000 7f5ace7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5ad00c9b40 0x7f5ad01f6220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34844/0 (socket says 172.21.15.16:34844) 2023-05-22T15:42:26.348 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.340+0000 7f5ace7fc640 1 -- 172.21.15.16:0/105111958 learned_addr learned my addr 172.21.15.16:0/105111958 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:42:26.348 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.340+0000 7f5acf7fe640 1 --2- 172.21.15.16:0/105111958 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5ad00ca4e0 0x7f5ad01f2270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:26.348 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.340+0000 7f5aceffd640 1 --2- 172.21.15.16:0/105111958 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5ad0070df0 0x7f5ad0159750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:26.348 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.340+0000 7f5ace7fc640 1 -- 172.21.15.16:0/105111958 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5ad00ca4e0 msgr2=0x7f5ad01f2270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:26.348 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.340+0000 7f5ace7fc640 1 --2- 172.21.15.16:0/105111958 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5ad00ca4e0 0x7f5ad01f2270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:26.348 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.340+0000 7f5ace7fc640 1 -- 172.21.15.16:0/105111958 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5ad0070df0 msgr2=0x7f5ad0159750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:26.348 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.340+0000 7f5ace7fc640 1 --2- 172.21.15.16:0/105111958 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5ad0070df0 0x7f5ad0159750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:26.348 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.340+0000 7f5ace7fc640 1 -- 172.21.15.16:0/105111958 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ac0009b30 con 0x7f5ad00c9b40 2023-05-22T15:42:26.349 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.340+0000 7f5ace7fc640 1 --2- 172.21.15.16:0/105111958 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5ad00c9b40 0x7f5ad01f6220 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x7f5ac400c550 tx=0x7f5ac400ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:26.349 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.340+0000 7f5ab77fe640 1 -- 172.21.15.16:0/105111958 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ac40052d0 con 0x7f5ad00c9b40 2023-05-22T15:42:26.349 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.340+0000 7f5ab77fe640 1 -- 172.21.15.16:0/105111958 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ac4005430 con 0x7f5ad00c9b40 2023-05-22T15:42:26.349 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.340+0000 7f5ad5618640 1 -- 172.21.15.16:0/105111958 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ad00c8e80 con 0x7f5ad00c9b40 2023-05-22T15:42:26.349 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.340+0000 7f5ab77fe640 1 -- 172.21.15.16:0/105111958 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ac4012870 con 0x7f5ad00c9b40 2023-05-22T15:42:26.349 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.340+0000 7f5ad5618640 1 -- 172.21.15.16:0/105111958 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ad00ca8b0 con 0x7f5ad00c9b40 2023-05-22T15:42:26.349 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.344+0000 7f5ad5618640 1 -- 172.21.15.16:0/105111958 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ad00c9010 con 0x7f5ad00c9b40 2023-05-22T15:42:26.350 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.344+0000 7f5ab77fe640 1 -- 172.21.15.16:0/105111958 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f5ac40129d0 con 0x7f5ad00c9b40 2023-05-22T15:42:26.350 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.344+0000 7f5ab77fe640 1 --2- 172.21.15.16:0/105111958 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f5abc0654a0 0x7f5abc067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:26.350 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.344+0000 7f5ab77fe640 1 -- 172.21.15.16:0/105111958 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f5ac40826a0 con 0x7f5ad00c9b40 2023-05-22T15:42:26.351 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.344+0000 7f5aceffd640 1 --2- 172.21.15.16:0/105111958 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f5abc0654a0 0x7f5abc067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:26.351 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.344+0000 7f5aceffd640 1 --2- 172.21.15.16:0/105111958 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f5abc0654a0 0x7f5abc067950 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f5ac00ae000 tx=0x7f5ac00a7070 comp rx=0 tx=0).ready entity=mgr.6765 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:26.351 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.348+0000 7f5ab77fe640 1 -- 172.21.15.16:0/105111958 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f5ac40500e0 con 0x7f5ad00c9b40 2023-05-22T15:42:26.503 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.500+0000 7f5ad5618640 1 -- 172.21.15.16:0/105111958 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5ad01f5f40 con 0x7f5ad00c9b40 2023-05-22T15:42:26.505 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.500+0000 7f5ab77fe640 1 -- 172.21.15.16:0/105111958 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) v1 ==== 74+0+18039 (secure 0 0 0) 0x7f5ac4053700 con 0x7f5ad00c9b40 2023-05-22T15:42:26.506 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:42:26.506 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":90,"fsid":"b2719834-7847-4d98-90a6-f65d57f85c49","created":"2023-05-22T15:26:25.259314+0000","modified":"2023-05-22T15:42:23.816424+0000","last_up_change":"2023-05-22T15:26:36.448524+0000","last_in_change":"2023-05-22T15:26:27.675668+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-05-22T15:26:38.019781+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-05-22T15:26:43.570514+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2023-05-22T15:42:20.730757+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2023-05-22T15:42:21.805016+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9e7ff854-b25f-4cbb-a5c2-1799d9c53d2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":37021},{"type":"v1","addr":"172.21.15.16:6825","nonce":37021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":37021},{"type":"v1","addr":"172.21.15.16:6827","nonce":37021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":37021},{"type":"v1","addr":"172.21.15.16:6831","nonce":37021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":37021},{"type":"v1","addr":"172.21.15.16:6829","nonce":37021}]},"public_addr":"172.21.15.16:6825/37021","cluster_addr":"172.21.15.16:6827/37021","heartbeat_back_addr":"172.21.15.16:6831/37021","heartbeat_front_addr":"172.21.15.16:6829/37021","state":["exists","up"]},{"osd":1,"uuid":"4702fbc9-a927-4565-aca5-41dc69cc8264","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":37008},{"type":"v1","addr":"172.21.15.16:6801","nonce":37008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":37008},{"type":"v1","addr":"172.21.15.16:6803","nonce":37008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":37008},{"type":"v1","addr":"172.21.15.16:6807","nonce":37008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":37008},{"type":"v1","addr":"172.21.15.16:6805","nonce":37008}]},"public_addr":"172.21.15.16:6801/37008","cluster_addr":"172.21.15.16:6803/37008","heartbeat_back_addr":"172.21.15.16:6807/37008","heartbeat_front_addr":"172.21.15.16:6805/37008","state":["exists","up"]},{"osd":2,"uuid":"1e7bba46-4eba-47ef-bb81-5e2d371638ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":37009},{"type":"v1","addr":"172.21.15.16:6809","nonce":37009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":37009},{"type":"v1","addr":"172.21.15.16:6811","nonce":37009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":37009},{"type":"v1","addr":"172.21.15.16:6815","nonce":37009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":37009},{"type":"v1","addr":"172.21.15.16:6813","nonce":37009}]},"public_addr":"172.21.15.16:6809/37009","cluster_addr":"172.21.15.16:6811/37009","heartbeat_back_addr":"172.21.15.16:6815/37009","heartbeat_front_addr":"172.21.15.16:6813/37009","state":["exists","up"]},{"osd":3,"uuid":"383cacc0-5a88-4904-990e-498ca59ef48a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":37020},{"type":"v1","addr":"172.21.15.16:6817","nonce":37020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":37020},{"type":"v1","addr":"172.21.15.16:6819","nonce":37020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":37020},{"type":"v1","addr":"172.21.15.16:6823","nonce":37020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":37020},{"type":"v1","addr":"172.21.15.16:6821","nonce":37020}]},"public_addr":"172.21.15.16:6817/37020","cluster_addr":"172.21.15.16:6819/37020","heartbeat_back_addr":"172.21.15.16:6823/37020","heartbeat_front_addr":"172.21.15.16:6821/37020","state":["exists","up"]},{"osd":4,"uuid":"970500ef-95b7-42c1-b080-eed4fae7013d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":37353},{"type":"v1","addr":"172.21.15.62:6817","nonce":37353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6819","nonce":37353},{"type":"v1","addr":"172.21.15.62:6822","nonce":37353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37353},{"type":"v1","addr":"172.21.15.62:6831","nonce":37353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37353},{"type":"v1","addr":"172.21.15.62:6826","nonce":37353}]},"public_addr":"172.21.15.62:6817/37353","cluster_addr":"172.21.15.62:6822/37353","heartbeat_back_addr":"172.21.15.62:6831/37353","heartbeat_front_addr":"172.21.15.62:6826/37353","state":["exists","up"]},{"osd":5,"uuid":"022d67db-d4d8-445b-b1b9-8b446a00f5c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":37352},{"type":"v1","addr":"172.21.15.62:6801","nonce":37352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":37352},{"type":"v1","addr":"172.21.15.62:6803","nonce":37352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":37352},{"type":"v1","addr":"172.21.15.62:6807","nonce":37352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":37352},{"type":"v1","addr":"172.21.15.62:6805","nonce":37352}]},"public_addr":"172.21.15.62:6801/37352","cluster_addr":"172.21.15.62:6803/37352","heartbeat_back_addr":"172.21.15.62:6807/37352","heartbeat_front_addr":"172.21.15.62:6805/37352","state":["exists","up"]},{"osd":6,"uuid":"42bd372b-6688-4a1b-8407-e16e8c266e33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":37354},{"type":"v1","addr":"172.21.15.62:6820","nonce":37354}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":37354},{"type":"v1","addr":"172.21.15.62:6823","nonce":37354}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6829","nonce":37354},{"type":"v1","addr":"172.21.15.62:6830","nonce":37354}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6825","nonce":37354},{"type":"v1","addr":"172.21.15.62:6827","nonce":37354}]},"public_addr":"172.21.15.62:6820/37354","cluster_addr":"172.21.15.62:6823/37354","heartbeat_back_addr":"172.21.15.62:6830/37354","heartbeat_front_addr":"172.21.15.62:6827/37354","state":["exists","up"]},{"osd":7,"uuid":"ac70b2a3-f41b-4bf2-b227-7f2cbb55426d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37355},{"type":"v1","addr":"172.21.15.62:6809","nonce":37355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37355},{"type":"v1","addr":"172.21.15.62:6811","nonce":37355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":37355},{"type":"v1","addr":"172.21.15.62:6815","nonce":37355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":37355},{"type":"v1","addr":"172.21.15.62:6813","nonce":37355}]},"public_addr":"172.21.15.62:6809/37355","cluster_addr":"172.21.15.62:6811/37355","heartbeat_back_addr":"172.21.15.62:6815/37355","heartbeat_front_addr":"172.21.15.62:6813/37355","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.368896+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.107273+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.276888+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.287502+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.124791+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.251937+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.255679+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.232797+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-05-22T15:42:26.508 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.504+0000 7f5ad5618640 1 -- 172.21.15.16:0/105111958 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f5abc0654a0 msgr2=0x7f5abc067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:26.508 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.504+0000 7f5ad5618640 1 --2- 172.21.15.16:0/105111958 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f5abc0654a0 0x7f5abc067950 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f5ac00ae000 tx=0x7f5ac00a7070 comp rx=0 tx=0).stop 2023-05-22T15:42:26.508 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.504+0000 7f5ad5618640 1 -- 172.21.15.16:0/105111958 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5ad00c9b40 msgr2=0x7f5ad01f6220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:26.508 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.504+0000 7f5ad5618640 1 --2- 172.21.15.16:0/105111958 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5ad00c9b40 0x7f5ad01f6220 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x7f5ac400c550 tx=0x7f5ac400ca10 comp rx=0 tx=0).stop 2023-05-22T15:42:26.508 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.504+0000 7f5ad5618640 1 -- 172.21.15.16:0/105111958 shutdown_connections 2023-05-22T15:42:26.508 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.504+0000 7f5ad5618640 1 --2- 172.21.15.16:0/105111958 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f5abc0654a0 0x7f5abc067950 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:26.509 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.504+0000 7f5ad5618640 1 --2- 172.21.15.16:0/105111958 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5ad00ca4e0 0x7f5ad01f2270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:26.509 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.504+0000 7f5ad5618640 1 --2- 172.21.15.16:0/105111958 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5ad00c9b40 0x7f5ad01f6220 unknown :-1 s=CLOSED pgs=1111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:26.509 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.504+0000 7f5ad5618640 1 --2- 172.21.15.16:0/105111958 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5ad0070df0 0x7f5ad0159750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:26.509 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.504+0000 7f5ad5618640 1 -- 172.21.15.16:0/105111958 >> 172.21.15.16:0/105111958 conn(0x7f5ad0069720 msgr2=0x7f5ad00aea70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:26.509 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.504+0000 7f5ad5618640 1 -- 172.21.15.16:0/105111958 shutdown_connections 2023-05-22T15:42:26.509 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.504+0000 7f5ad5618640 1 -- 172.21.15.16:0/105111958 wait complete. 2023-05-22T15:42:26.524 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-05-22T15:42:26.524 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-05-22T15:42:26.646 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.644+0000 7f0fe76ed640 1 Processor -- start 2023-05-22T15:42:26.647 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.644+0000 7f0fe76ed640 1 -- start start 2023-05-22T15:42:26.647 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.644+0000 7f0fe76ed640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0fe00c9b40 0x7f0fe015e460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:26.648 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.644+0000 7f0fe76ed640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f0fe0070df0 0x7f0fe0071210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:26.648 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.644+0000 7f0fe76ed640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0fe00b0650 con 0x7f0fe00ca4e0 2023-05-22T15:42:26.648 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.644+0000 7f0fe76ed640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0fe00b0370 con 0x7f0fe0070df0 2023-05-22T15:42:26.648 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.644+0000 7f0fe76ed640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0fe015d980 con 0x7f0fe00c9b40 2023-05-22T15:42:26.649 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.644+0000 7f0fe5c8a640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f0fe0070df0 0x7f0fe0071210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:26.649 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.644+0000 7f0fe5c8a640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f0fe0070df0 0x7f0fe0071210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34856/0 (socket says 172.21.15.16:34856) 2023-05-22T15:42:26.649 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.644+0000 7f0fe5c8a640 1 -- 172.21.15.16:0/3000584596 learned_addr learned my addr 172.21.15.16:0/3000584596 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:42:26.649 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.644+0000 7f0fe4c88640 1 --2- 172.21.15.16:0/3000584596 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0fe00c9b40 0x7f0fe015e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:26.650 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.644+0000 7f0fe5c8a640 1 -- 172.21.15.16:0/3000584596 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0fe00c9b40 msgr2=0x7f0fe015e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:42:26.650 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.644+0000 7f0fe5c8a640 1 --2- 172.21.15.16:0/3000584596 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0fe00c9b40 0x7f0fe015e460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:26.650 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.644+0000 7f0fe5c8a640 1 -- 172.21.15.16:0/3000584596 >> v1:172.21.15.62:6789/0 conn(0x7f0fe00ca4e0 legacy=0x7f0fe00afb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:26.650 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.644+0000 7f0fe5c8a640 1 -- 172.21.15.16:0/3000584596 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fe015dc60 con 0x7f0fe0070df0 2023-05-22T15:42:26.651 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.648+0000 7f0fe5c8a640 1 --2- 172.21.15.16:0/3000584596 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0fe0070df0 0x7f0fe0071210 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 crypto rx=0x7f0fd8009e50 tx=0x7f0fd801c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=913668f6d1731ce5 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:26.651 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.648+0000 7f0fd7fff640 1 -- 172.21.15.16:0/3000584596 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fd80a5770 con 0x7f0fe0070df0 2023-05-22T15:42:26.652 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.648+0000 7f0fd7fff640 1 -- 172.21.15.16:0/3000584596 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fd80a58d0 con 0x7f0fe0070df0 2023-05-22T15:42:26.652 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.648+0000 7f0fd7fff640 1 -- 172.21.15.16:0/3000584596 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fd80a5c60 con 0x7f0fe0070df0 2023-05-22T15:42:26.652 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.648+0000 7f0fe76ed640 1 -- 172.21.15.16:0/3000584596 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0fe0070df0 msgr2=0x7f0fe0071210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:26.653 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.648+0000 7f0fe76ed640 1 --2- 172.21.15.16:0/3000584596 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0fe0070df0 0x7f0fe0071210 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 crypto rx=0x7f0fd8009e50 tx=0x7f0fd801c950 comp rx=0 tx=0).stop 2023-05-22T15:42:26.653 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.648+0000 7f0fe76ed640 1 -- 172.21.15.16:0/3000584596 shutdown_connections 2023-05-22T15:42:26.653 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.648+0000 7f0fe76ed640 1 --2- 172.21.15.16:0/3000584596 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0fe0070df0 0x7f0fe0071210 unknown :-1 s=CLOSED pgs=1112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:26.653 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.648+0000 7f0fe76ed640 1 --2- 172.21.15.16:0/3000584596 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0fe00c9b40 0x7f0fe015e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:26.653 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.648+0000 7f0fe76ed640 1 -- 172.21.15.16:0/3000584596 >> 172.21.15.16:0/3000584596 conn(0x7f0fe0069720 msgr2=0x7f0fe0069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:26.653 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.648+0000 7f0fe76ed640 1 -- 172.21.15.16:0/3000584596 shutdown_connections 2023-05-22T15:42:26.654 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.648+0000 7f0fe76ed640 1 -- 172.21.15.16:0/3000584596 wait complete. 2023-05-22T15:42:26.655 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.648+0000 7f0fe76ed640 1 Processor -- start 2023-05-22T15:42:26.655 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.648+0000 7f0fe76ed640 1 -- start start 2023-05-22T15:42:26.655 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.652+0000 7f0fe76ed640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0fe0070df0 0x7f0fe00aef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:26.655 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.652+0000 7f0fe76ed640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0fe00c9b40 0x7f0fe015a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:26.656 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.652+0000 7f0fe76ed640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0fe00ca4e0 0x7f0fe006b990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:26.656 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.652+0000 7f0fe76ed640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fe015a7f0 con 0x7f0fe00c9b40 2023-05-22T15:42:26.656 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.652+0000 7f0fe76ed640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fe015c500 con 0x7f0fe0070df0 2023-05-22T15:42:26.656 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.652+0000 7f0fe76ed640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0fe015bc10 con 0x7f0fe00ca4e0 2023-05-22T15:42:26.656 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.652+0000 7f0fe4c88640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0fe00c9b40 0x7f0fe015a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:26.656 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.652+0000 7f0fe4c88640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0fe00c9b40 0x7f0fe015a2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34866/0 (socket says 172.21.15.16:34866) 2023-05-22T15:42:26.656 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.652+0000 7f0fe4c88640 1 -- 172.21.15.16:0/1083958462 learned_addr learned my addr 172.21.15.16:0/1083958462 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:42:26.657 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.652+0000 7f0fe5c8a640 1 --2- 172.21.15.16:0/1083958462 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0fe00ca4e0 0x7f0fe006b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:26.657 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.652+0000 7f0fe4c88640 1 -- 172.21.15.16:0/1083958462 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0fe00ca4e0 msgr2=0x7f0fe006b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:26.657 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.652+0000 7f0fe4c88640 1 --2- 172.21.15.16:0/1083958462 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0fe00ca4e0 0x7f0fe006b990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:26.657 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.652+0000 7f0fe5489640 1 --2- 172.21.15.16:0/1083958462 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0fe0070df0 0x7f0fe00aef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:26.657 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.652+0000 7f0fe4c88640 1 -- 172.21.15.16:0/1083958462 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0fe0070df0 msgr2=0x7f0fe00aef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:26.657 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.652+0000 7f0fe4c88640 1 --2- 172.21.15.16:0/1083958462 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0fe0070df0 0x7f0fe00aef60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:26.657 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.652+0000 7f0fe4c88640 1 -- 172.21.15.16:0/1083958462 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fd8009b30 con 0x7f0fe00c9b40 2023-05-22T15:42:26.658 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.652+0000 7f0fe4c88640 1 --2- 172.21.15.16:0/1083958462 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0fe00c9b40 0x7f0fe015a2b0 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 crypto rx=0x7f0fd000d800 tx=0x7f0fd000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:26.658 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.652+0000 7f0fd5ffb640 1 -- 172.21.15.16:0/1083958462 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fd000b6c0 con 0x7f0fe00c9b40 2023-05-22T15:42:26.658 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.652+0000 7f0fd5ffb640 1 -- 172.21.15.16:0/1083958462 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fd000b820 con 0x7f0fe00c9b40 2023-05-22T15:42:26.658 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.652+0000 7f0fd5ffb640 1 -- 172.21.15.16:0/1083958462 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fd000bb90 con 0x7f0fe00c9b40 2023-05-22T15:42:26.658 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.652+0000 7f0fe76ed640 1 -- 172.21.15.16:0/1083958462 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0fe00c8e80 con 0x7f0fe00c9b40 2023-05-22T15:42:26.658 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.652+0000 7f0fe76ed640 1 -- 172.21.15.16:0/1083958462 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0fe00ca8b0 con 0x7f0fe00c9b40 2023-05-22T15:42:26.659 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.652+0000 7f0fe76ed640 1 -- 172.21.15.16:0/1083958462 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0fe00c9010 con 0x7f0fe00c9b40 2023-05-22T15:42:26.659 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.652+0000 7f0fd5ffb640 1 -- 172.21.15.16:0/1083958462 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f0fd000bcf0 con 0x7f0fe00c9b40 2023-05-22T15:42:26.659 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.652+0000 7f0fd5ffb640 1 --2- 172.21.15.16:0/1083958462 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f0fc40654a0 0x7f0fc4067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:26.659 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.652+0000 7f0fe5489640 1 --2- 172.21.15.16:0/1083958462 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f0fc40654a0 0x7f0fc4067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:26.660 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.652+0000 7f0fd5ffb640 1 -- 172.21.15.16:0/1083958462 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f0fd0083110 con 0x7f0fe00c9b40 2023-05-22T15:42:26.660 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.656+0000 7f0fe5489640 1 --2- 172.21.15.16:0/1083958462 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f0fc40654a0 0x7f0fc4067950 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f0fdc000f70 tx=0x7f0fdc009040 comp rx=0 tx=0).ready entity=mgr.6765 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:26.662 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.656+0000 7f0fd5ffb640 1 -- 172.21.15.16:0/1083958462 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f0fd00544d0 con 0x7f0fe00c9b40 2023-05-22T15:42:26.808 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.804+0000 7f0fe76ed640 1 -- 172.21.15.16:0/1083958462 --> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f0fe004dbe0 con 0x7f0fc40654a0 2023-05-22T15:42:26.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.828+0000 7f0fd5ffb640 1 -- 172.21.15.16:0/1083958462 <== mgr.6765 v2:172.21.15.16:6832/36921 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+310044 (secure 0 0 0) 0x7f0fe004dbe0 con 0x7f0fc40654a0 2023-05-22T15:42:26.836 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:42:26.838 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2023-05-22T15:42:26.839 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.832+0000 7f0fe76ed640 1 -- 172.21.15.16:0/1083958462 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f0fc40654a0 msgr2=0x7f0fc4067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:26.839 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.832+0000 7f0fe76ed640 1 --2- 172.21.15.16:0/1083958462 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f0fc40654a0 0x7f0fc4067950 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f0fdc000f70 tx=0x7f0fdc009040 comp rx=0 tx=0).stop 2023-05-22T15:42:26.840 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.832+0000 7f0fe76ed640 1 -- 172.21.15.16:0/1083958462 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0fe00c9b40 msgr2=0x7f0fe015a2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:26.840 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.832+0000 7f0fe76ed640 1 --2- 172.21.15.16:0/1083958462 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0fe00c9b40 0x7f0fe015a2b0 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 crypto rx=0x7f0fd000d800 tx=0x7f0fd000dcc0 comp rx=0 tx=0).stop 2023-05-22T15:42:26.840 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.832+0000 7f0fe76ed640 1 -- 172.21.15.16:0/1083958462 shutdown_connections 2023-05-22T15:42:26.840 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.832+0000 7f0fe76ed640 1 --2- 172.21.15.16:0/1083958462 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f0fc40654a0 0x7f0fc4067950 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:26.840 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.832+0000 7f0fe76ed640 1 --2- 172.21.15.16:0/1083958462 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0fe00ca4e0 0x7f0fe006b990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:26.841 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.832+0000 7f0fe76ed640 1 --2- 172.21.15.16:0/1083958462 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0fe00c9b40 0x7f0fe015a2b0 unknown :-1 s=CLOSED pgs=1113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:26.841 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.832+0000 7f0fe76ed640 1 --2- 172.21.15.16:0/1083958462 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0fe0070df0 0x7f0fe00aef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:26.841 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.832+0000 7f0fe76ed640 1 -- 172.21.15.16:0/1083958462 >> 172.21.15.16:0/1083958462 conn(0x7f0fe0069720 msgr2=0x7f0fe00b0eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:26.841 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.832+0000 7f0fe76ed640 1 -- 172.21.15.16:0/1083958462 shutdown_connections 2023-05-22T15:42:26.841 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:26.832+0000 7f0fe76ed640 1 -- 172.21.15.16:0/1083958462 wait complete. 2023-05-22T15:42:26.855 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":25,"stamp":"2023-05-22T15:42:25.054550+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592698,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":500,"num_read_kb":924,"num_write":283,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":199,"ondisk_log_size":199,"up":210,"acting":210,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":35,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":206756,"kb_used_data":9508,"kb_used_omap":0,"kb_used_meta":197184,"kb_avail":754767964,"statfs":{"total":773094113280,"available":772882395136,"internally_reserved":0,"allocated":9736192,"data_stored":5627780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":201916416},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":71,"apply_latency_ms":71,"commit_latency_ns":71000000,"apply_latency_ns":71000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2330,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001138"},"pg_stats":[{"pgid":"11.3f","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.904859+0000","last_change":"2023-05-22T15:42:22.894158+0000","last_active":"2023-05-22T15:42:22.904859+0000","last_peered":"2023-05-22T15:42:22.904859+0000","last_clean":"2023-05-22T15:42:22.904859+0000","last_became_active":"2023-05-22T15:42:22.893003+0000","last_became_peered":"2023-05-22T15:42:22.893003+0000","last_unstale":"2023-05-22T15:42:22.904859+0000","last_undegraded":"2023-05-22T15:42:22.904859+0000","last_fullsized":"2023-05-22T15:42:22.904859+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_clean_scrub_stamp":"2023-05-22T15:42:22.424085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T16:20:50.290808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.3e","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.897274+0000","last_change":"2023-05-22T15:42:22.885226+0000","last_active":"2023-05-22T15:42:22.897274+0000","last_peered":"2023-05-22T15:42:22.897274+0000","last_clean":"2023-05-22T15:42:22.897274+0000","last_became_active":"2023-05-22T15:42:22.882088+0000","last_became_peered":"2023-05-22T15:42:22.882088+0000","last_unstale":"2023-05-22T15:42:22.897274+0000","last_undegraded":"2023-05-22T15:42:22.897274+0000","last_fullsized":"2023-05-22T15:42:22.897274+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_clean_scrub_stamp":"2023-05-22T15:42:22.424085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T15:49:20.912521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.3d","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.840258+0000","last_change":"2023-05-22T15:42:22.838847+0000","last_active":"2023-05-22T15:42:22.840258+0000","last_peered":"2023-05-22T15:42:22.840258+0000","last_clean":"2023-05-22T15:42:22.840258+0000","last_became_active":"2023-05-22T15:42:22.838548+0000","last_became_peered":"2023-05-22T15:42:22.838548+0000","last_unstale":"2023-05-22T15:42:22.840258+0000","last_undegraded":"2023-05-22T15:42:22.840258+0000","last_fullsized":"2023-05-22T15:42:22.840258+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_clean_scrub_stamp":"2023-05-22T15:42:22.424085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:26:08.824903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.3c","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.879554+0000","last_change":"2023-05-22T15:42:22.878514+0000","last_active":"2023-05-22T15:42:22.879554+0000","last_peered":"2023-05-22T15:42:22.879554+0000","last_clean":"2023-05-22T15:42:22.879554+0000","last_became_active":"2023-05-22T15:42:22.877705+0000","last_became_peered":"2023-05-22T15:42:22.877705+0000","last_unstale":"2023-05-22T15:42:22.879554+0000","last_undegraded":"2023-05-22T15:42:22.879554+0000","last_fullsized":"2023-05-22T15:42:22.879554+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_clean_scrub_stamp":"2023-05-22T15:42:22.424085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:51:29.460959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.3b","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.891490+0000","last_change":"2023-05-22T15:42:22.890668+0000","last_active":"2023-05-22T15:42:22.891490+0000","last_peered":"2023-05-22T15:42:22.891490+0000","last_clean":"2023-05-22T15:42:22.891490+0000","last_became_active":"2023-05-22T15:42:22.890359+0000","last_became_peered":"2023-05-22T15:42:22.890359+0000","last_unstale":"2023-05-22T15:42:22.891490+0000","last_undegraded":"2023-05-22T15:42:22.891490+0000","last_fullsized":"2023-05-22T15:42:22.891490+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_clean_scrub_stamp":"2023-05-22T15:42:22.424085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:24:27.275366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3a","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.897816+0000","last_change":"2023-05-22T15:42:22.882144+0000","last_active":"2023-05-22T15:42:22.897816+0000","last_peered":"2023-05-22T15:42:22.897816+0000","last_clean":"2023-05-22T15:42:22.897816+0000","last_became_active":"2023-05-22T15:42:22.880797+0000","last_became_peered":"2023-05-22T15:42:22.880797+0000","last_unstale":"2023-05-22T15:42:22.897816+0000","last_undegraded":"2023-05-22T15:42:22.897816+0000","last_fullsized":"2023-05-22T15:42:22.897816+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_clean_scrub_stamp":"2023-05-22T15:42:22.424085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:51:29.460959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.39","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.870224+0000","last_change":"2023-05-22T15:42:22.848473+0000","last_active":"2023-05-22T15:42:22.870224+0000","last_peered":"2023-05-22T15:42:22.870224+0000","last_clean":"2023-05-22T15:42:22.870224+0000","last_became_active":"2023-05-22T15:42:22.847775+0000","last_became_peered":"2023-05-22T15:42:22.847775+0000","last_unstale":"2023-05-22T15:42:22.870224+0000","last_undegraded":"2023-05-22T15:42:22.870224+0000","last_fullsized":"2023-05-22T15:42:22.870224+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_clean_scrub_stamp":"2023-05-22T15:42:22.424085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:45:22.277730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.38","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.896379+0000","last_change":"2023-05-22T15:42:22.882560+0000","last_active":"2023-05-22T15:42:22.896379+0000","last_peered":"2023-05-22T15:42:22.896379+0000","last_clean":"2023-05-22T15:42:22.896379+0000","last_became_active":"2023-05-22T15:42:22.877494+0000","last_became_peered":"2023-05-22T15:42:22.877494+0000","last_unstale":"2023-05-22T15:42:22.896379+0000","last_undegraded":"2023-05-22T15:42:22.896379+0000","last_fullsized":"2023-05-22T15:42:22.896379+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_clean_scrub_stamp":"2023-05-22T15:42:22.424085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T03:03:57.842619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.37","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.840713+0000","last_change":"2023-05-22T15:42:22.837811+0000","last_active":"2023-05-22T15:42:22.840713+0000","last_peered":"2023-05-22T15:42:22.840713+0000","last_clean":"2023-05-22T15:42:22.840713+0000","last_became_active":"2023-05-22T15:42:22.837518+0000","last_became_peered":"2023-05-22T15:42:22.837518+0000","last_unstale":"2023-05-22T15:42:22.840713+0000","last_undegraded":"2023-05-22T15:42:22.840713+0000","last_fullsized":"2023-05-22T15:42:22.840713+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_clean_scrub_stamp":"2023-05-22T15:42:22.424085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:32:58.984742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.36","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.886652+0000","last_change":"2023-05-22T15:42:22.871344+0000","last_active":"2023-05-22T15:42:22.886652+0000","last_peered":"2023-05-22T15:42:22.886652+0000","last_clean":"2023-05-22T15:42:22.886652+0000","last_became_active":"2023-05-22T15:42:22.870582+0000","last_became_peered":"2023-05-22T15:42:22.870582+0000","last_unstale":"2023-05-22T15:42:22.886652+0000","last_undegraded":"2023-05-22T15:42:22.886652+0000","last_fullsized":"2023-05-22T15:42:22.886652+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_clean_scrub_stamp":"2023-05-22T15:42:22.424085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:12:13.799667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.35","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.904737+0000","last_change":"2023-05-22T15:42:22.892644+0000","last_active":"2023-05-22T15:42:22.904737+0000","last_peered":"2023-05-22T15:42:22.904737+0000","last_clean":"2023-05-22T15:42:22.904737+0000","last_became_active":"2023-05-22T15:42:22.892222+0000","last_became_peered":"2023-05-22T15:42:22.892222+0000","last_unstale":"2023-05-22T15:42:22.904737+0000","last_undegraded":"2023-05-22T15:42:22.904737+0000","last_fullsized":"2023-05-22T15:42:22.904737+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_clean_scrub_stamp":"2023-05-22T15:42:22.424085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:17:04.192121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.34","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.833620+0000","last_change":"2023-05-22T15:42:22.829981+0000","last_active":"2023-05-22T15:42:22.833620+0000","last_peered":"2023-05-22T15:42:22.833620+0000","last_clean":"2023-05-22T15:42:22.833620+0000","last_became_active":"2023-05-22T15:42:22.829718+0000","last_became_peered":"2023-05-22T15:42:22.829718+0000","last_unstale":"2023-05-22T15:42:22.833620+0000","last_undegraded":"2023-05-22T15:42:22.833620+0000","last_fullsized":"2023-05-22T15:42:22.833620+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_clean_scrub_stamp":"2023-05-22T15:42:22.424085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T15:49:20.912521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.33","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.882882+0000","last_change":"2023-05-22T15:42:22.882224+0000","last_active":"2023-05-22T15:42:22.882882+0000","last_peered":"2023-05-22T15:42:22.882882+0000","last_clean":"2023-05-22T15:42:22.882882+0000","last_became_active":"2023-05-22T15:42:22.882024+0000","last_became_peered":"2023-05-22T15:42:22.882024+0000","last_unstale":"2023-05-22T15:42:22.882882+0000","last_undegraded":"2023-05-22T15:42:22.882882+0000","last_fullsized":"2023-05-22T15:42:22.882882+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_clean_scrub_stamp":"2023-05-22T15:42:22.424085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:11:47.018486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.32","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.866705+0000","last_change":"2023-05-22T15:42:22.865607+0000","last_active":"2023-05-22T15:42:22.866705+0000","last_peered":"2023-05-22T15:42:22.866705+0000","last_clean":"2023-05-22T15:42:22.866705+0000","last_became_active":"2023-05-22T15:42:22.865328+0000","last_became_peered":"2023-05-22T15:42:22.865328+0000","last_unstale":"2023-05-22T15:42:22.866705+0000","last_undegraded":"2023-05-22T15:42:22.866705+0000","last_fullsized":"2023-05-22T15:42:22.866705+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_clean_scrub_stamp":"2023-05-22T15:42:22.424085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:41:51.956588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.31","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.838194+0000","last_change":"2023-05-22T15:42:22.829796+0000","last_active":"2023-05-22T15:42:22.838194+0000","last_peered":"2023-05-22T15:42:22.838194+0000","last_clean":"2023-05-22T15:42:22.838194+0000","last_became_active":"2023-05-22T15:42:22.829501+0000","last_became_peered":"2023-05-22T15:42:22.829501+0000","last_unstale":"2023-05-22T15:42:22.838194+0000","last_undegraded":"2023-05-22T15:42:22.838194+0000","last_fullsized":"2023-05-22T15:42:22.838194+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_clean_scrub_stamp":"2023-05-22T15:42:22.424085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:43:03.995839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.30","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.888377+0000","last_change":"2023-05-22T15:42:22.880152+0000","last_active":"2023-05-22T15:42:22.888377+0000","last_peered":"2023-05-22T15:42:22.888377+0000","last_clean":"2023-05-22T15:42:22.888377+0000","last_became_active":"2023-05-22T15:42:22.879339+0000","last_became_peered":"2023-05-22T15:42:22.879339+0000","last_unstale":"2023-05-22T15:42:22.888377+0000","last_undegraded":"2023-05-22T15:42:22.888377+0000","last_fullsized":"2023-05-22T15:42:22.888377+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_clean_scrub_stamp":"2023-05-22T15:42:22.424085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:05:27.121829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2f","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.840728+0000","last_change":"2023-05-22T15:42:22.838081+0000","last_active":"2023-05-22T15:42:22.840728+0000","last_peered":"2023-05-22T15:42:22.840728+0000","last_clean":"2023-05-22T15:42:22.840728+0000","last_became_active":"2023-05-22T15:42:22.837802+0000","last_became_peered":"2023-05-22T15:42:22.837802+0000","last_unstale":"2023-05-22T15:42:22.840728+0000","last_undegraded":"2023-05-22T15:42:22.840728+0000","last_fullsized":"2023-05-22T15:42:22.840728+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_clean_scrub_stamp":"2023-05-22T15:42:22.424085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:56:40.685104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.2e","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.879546+0000","last_change":"2023-05-22T15:42:22.878055+0000","last_active":"2023-05-22T15:42:22.879546+0000","last_peered":"2023-05-22T15:42:22.879546+0000","last_clean":"2023-05-22T15:42:22.879546+0000","last_became_active":"2023-05-22T15:42:22.877710+0000","last_became_peered":"2023-05-22T15:42:22.877710+0000","last_unstale":"2023-05-22T15:42:22.879546+0000","last_undegraded":"2023-05-22T15:42:22.879546+0000","last_fullsized":"2023-05-22T15:42:22.879546+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_clean_scrub_stamp":"2023-05-22T15:42:22.424085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:32:58.984742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.2d","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.888178+0000","last_change":"2023-05-22T15:42:22.881271+0000","last_active":"2023-05-22T15:42:22.888178+0000","last_peered":"2023-05-22T15:42:22.888178+0000","last_clean":"2023-05-22T15:42:22.888178+0000","last_became_active":"2023-05-22T15:42:22.878300+0000","last_became_peered":"2023-05-22T15:42:22.878300+0000","last_unstale":"2023-05-22T15:42:22.888178+0000","last_undegraded":"2023-05-22T15:42:22.888178+0000","last_fullsized":"2023-05-22T15:42:22.888178+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_clean_scrub_stamp":"2023-05-22T15:42:22.424085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:02:52.959763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2c","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.853781+0000","last_change":"2023-05-22T15:42:22.852710+0000","last_active":"2023-05-22T15:42:22.853781+0000","last_peered":"2023-05-22T15:42:22.853781+0000","last_clean":"2023-05-22T15:42:22.853781+0000","last_became_active":"2023-05-22T15:42:22.852289+0000","last_became_peered":"2023-05-22T15:42:22.852289+0000","last_unstale":"2023-05-22T15:42:22.853781+0000","last_undegraded":"2023-05-22T15:42:22.853781+0000","last_fullsized":"2023-05-22T15:42:22.853781+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_clean_scrub_stamp":"2023-05-22T15:42:22.424085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:25:08.886473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2b","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.886266+0000","last_change":"2023-05-22T15:42:22.869779+0000","last_active":"2023-05-22T15:42:22.886266+0000","last_peered":"2023-05-22T15:42:22.886266+0000","last_clean":"2023-05-22T15:42:22.886266+0000","last_became_active":"2023-05-22T15:42:22.869063+0000","last_became_peered":"2023-05-22T15:42:22.869063+0000","last_unstale":"2023-05-22T15:42:22.886266+0000","last_undegraded":"2023-05-22T15:42:22.886266+0000","last_fullsized":"2023-05-22T15:42:22.886266+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_clean_scrub_stamp":"2023-05-22T15:42:22.424085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:02:51.734869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2a","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.877954+0000","last_change":"2023-05-22T15:42:22.851172+0000","last_active":"2023-05-22T15:42:22.877954+0000","last_peered":"2023-05-22T15:42:22.877954+0000","last_clean":"2023-05-22T15:42:22.877954+0000","last_became_active":"2023-05-22T15:42:22.850434+0000","last_became_peered":"2023-05-22T15:42:22.850434+0000","last_unstale":"2023-05-22T15:42:22.877954+0000","last_undegraded":"2023-05-22T15:42:22.877954+0000","last_fullsized":"2023-05-22T15:42:22.877954+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_clean_scrub_stamp":"2023-05-22T15:42:22.424085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:57:49.944485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.29","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.904730+0000","last_change":"2023-05-22T15:42:22.892552+0000","last_active":"2023-05-22T15:42:22.904730+0000","last_peered":"2023-05-22T15:42:22.904730+0000","last_clean":"2023-05-22T15:42:22.904730+0000","last_became_active":"2023-05-22T15:42:22.891506+0000","last_became_peered":"2023-05-22T15:42:22.891506+0000","last_unstale":"2023-05-22T15:42:22.904730+0000","last_undegraded":"2023-05-22T15:42:22.904730+0000","last_fullsized":"2023-05-22T15:42:22.904730+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_clean_scrub_stamp":"2023-05-22T15:42:22.424085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:45:22.277730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.28","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.904602+0000","last_change":"2023-05-22T15:42:22.890427+0000","last_active":"2023-05-22T15:42:22.904602+0000","last_peered":"2023-05-22T15:42:22.904602+0000","last_clean":"2023-05-22T15:42:22.904602+0000","last_became_active":"2023-05-22T15:42:22.889859+0000","last_became_peered":"2023-05-22T15:42:22.889859+0000","last_unstale":"2023-05-22T15:42:22.904602+0000","last_undegraded":"2023-05-22T15:42:22.904602+0000","last_fullsized":"2023-05-22T15:42:22.904602+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_clean_scrub_stamp":"2023-05-22T15:42:22.424085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:56:03.682465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.27","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.898864+0000","last_change":"2023-05-22T15:42:22.895817+0000","last_active":"2023-05-22T15:42:22.898864+0000","last_peered":"2023-05-22T15:42:22.898864+0000","last_clean":"2023-05-22T15:42:22.898864+0000","last_became_active":"2023-05-22T15:42:22.895387+0000","last_became_peered":"2023-05-22T15:42:22.895387+0000","last_unstale":"2023-05-22T15:42:22.898864+0000","last_undegraded":"2023-05-22T15:42:22.898864+0000","last_fullsized":"2023-05-22T15:42:22.898864+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_clean_scrub_stamp":"2023-05-22T15:42:22.424085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:58:20.411983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.26","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.884315+0000","last_change":"2023-05-22T15:42:22.881297+0000","last_active":"2023-05-22T15:42:22.884315+0000","last_peered":"2023-05-22T15:42:22.884315+0000","last_clean":"2023-05-22T15:42:22.884315+0000","last_became_active":"2023-05-22T15:42:22.880447+0000","last_became_peered":"2023-05-22T15:42:22.880447+0000","last_unstale":"2023-05-22T15:42:22.884315+0000","last_undegraded":"2023-05-22T15:42:22.884315+0000","last_fullsized":"2023-05-22T15:42:22.884315+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_clean_scrub_stamp":"2023-05-22T15:42:22.424085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T18:23:22.270527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.25","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.895279+0000","last_change":"2023-05-22T15:42:22.894745+0000","last_active":"2023-05-22T15:42:22.895279+0000","last_peered":"2023-05-22T15:42:22.895279+0000","last_clean":"2023-05-22T15:42:22.895279+0000","last_became_active":"2023-05-22T15:42:22.894459+0000","last_became_peered":"2023-05-22T15:42:22.894459+0000","last_unstale":"2023-05-22T15:42:22.895279+0000","last_undegraded":"2023-05-22T15:42:22.895279+0000","last_fullsized":"2023-05-22T15:42:22.895279+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_clean_scrub_stamp":"2023-05-22T15:42:22.424085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:26:20.201566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.24","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.878026+0000","last_change":"2023-05-22T15:42:22.851280+0000","last_active":"2023-05-22T15:42:22.878026+0000","last_peered":"2023-05-22T15:42:22.878026+0000","last_clean":"2023-05-22T15:42:22.878026+0000","last_became_active":"2023-05-22T15:42:22.850543+0000","last_became_peered":"2023-05-22T15:42:22.850543+0000","last_unstale":"2023-05-22T15:42:22.878026+0000","last_undegraded":"2023-05-22T15:42:22.878026+0000","last_fullsized":"2023-05-22T15:42:22.878026+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_clean_scrub_stamp":"2023-05-22T15:42:22.424085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:04:44.765497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.23","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.864167+0000","last_change":"2023-05-22T15:42:22.863431+0000","last_active":"2023-05-22T15:42:22.864167+0000","last_peered":"2023-05-22T15:42:22.864167+0000","last_clean":"2023-05-22T15:42:22.864167+0000","last_became_active":"2023-05-22T15:42:22.863095+0000","last_became_peered":"2023-05-22T15:42:22.863095+0000","last_unstale":"2023-05-22T15:42:22.864167+0000","last_undegraded":"2023-05-22T15:42:22.864167+0000","last_fullsized":"2023-05-22T15:42:22.864167+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_clean_scrub_stamp":"2023-05-22T15:42:22.424085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:02:39.112412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.22","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.884132+0000","last_change":"2023-05-22T15:42:22.880416+0000","last_active":"2023-05-22T15:42:22.884132+0000","last_peered":"2023-05-22T15:42:22.884132+0000","last_clean":"2023-05-22T15:42:22.884132+0000","last_became_active":"2023-05-22T15:42:22.879633+0000","last_became_peered":"2023-05-22T15:42:22.879633+0000","last_unstale":"2023-05-22T15:42:22.884132+0000","last_undegraded":"2023-05-22T15:42:22.884132+0000","last_fullsized":"2023-05-22T15:42:22.884132+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_clean_scrub_stamp":"2023-05-22T15:42:22.424085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:58:20.411983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.21","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.865455+0000","last_change":"2023-05-22T15:42:22.847185+0000","last_active":"2023-05-22T15:42:22.865455+0000","last_peered":"2023-05-22T15:42:22.865455+0000","last_clean":"2023-05-22T15:42:22.865455+0000","last_became_active":"2023-05-22T15:42:22.846459+0000","last_became_peered":"2023-05-22T15:42:22.846459+0000","last_unstale":"2023-05-22T15:42:22.865455+0000","last_undegraded":"2023-05-22T15:42:22.865455+0000","last_fullsized":"2023-05-22T15:42:22.865455+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_clean_scrub_stamp":"2023-05-22T15:42:22.424085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:17:04.192121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.20","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.887948+0000","last_change":"2023-05-22T15:42:22.877407+0000","last_active":"2023-05-22T15:42:22.887948+0000","last_peered":"2023-05-22T15:42:22.887948+0000","last_clean":"2023-05-22T15:42:22.887948+0000","last_became_active":"2023-05-22T15:42:22.876698+0000","last_became_peered":"2023-05-22T15:42:22.876698+0000","last_unstale":"2023-05-22T15:42:22.887948+0000","last_undegraded":"2023-05-22T15:42:22.887948+0000","last_fullsized":"2023-05-22T15:42:22.887948+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_clean_scrub_stamp":"2023-05-22T15:42:22.424085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:41:51.956588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.904795+0000","last_change":"2023-05-22T15:42:22.893202+0000","last_active":"2023-05-22T15:42:22.904795+0000","last_peered":"2023-05-22T15:42:22.904795+0000","last_clean":"2023-05-22T15:42:22.904795+0000","last_became_active":"2023-05-22T15:42:22.891232+0000","last_became_peered":"2023-05-22T15:42:22.891232+0000","last_unstale":"2023-05-22T15:42:22.904795+0000","last_undegraded":"2023-05-22T15:42:22.904795+0000","last_fullsized":"2023-05-22T15:42:22.904795+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_clean_scrub_stamp":"2023-05-22T15:42:22.424085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:05:27.121829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1f","version":"89'4","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.892726+0000","last_change":"2023-05-22T15:42:22.481619+0000","last_active":"2023-05-22T15:42:22.892726+0000","last_peered":"2023-05-22T15:42:22.892726+0000","last_clean":"2023-05-22T15:42:22.892726+0000","last_became_active":"2023-05-22T15:42:22.481095+0000","last_became_peered":"2023-05-22T15:42:22.481095+0000","last_unstale":"2023-05-22T15:42:22.892726+0000","last_undegraded":"2023-05-22T15:42:22.892726+0000","last_fullsized":"2023-05-22T15:42:22.892726+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:21.420311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:21.420311+0000","last_clean_scrub_stamp":"2023-05-22T15:42:21.420311+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:11:08.656000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":566,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.1f","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.840422+0000","last_change":"2023-05-22T15:42:22.835146+0000","last_active":"2023-05-22T15:42:22.840422+0000","last_peered":"2023-05-22T15:42:22.840422+0000","last_clean":"2023-05-22T15:42:22.840422+0000","last_became_active":"2023-05-22T15:42:22.834823+0000","last_became_peered":"2023-05-22T15:42:22.834823+0000","last_unstale":"2023-05-22T15:42:22.840422+0000","last_undegraded":"2023-05-22T15:42:22.840422+0000","last_fullsized":"2023-05-22T15:42:22.840422+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_clean_scrub_stamp":"2023-05-22T15:42:22.424085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T15:49:20.912521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.1e","version":"89'1","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.837175+0000","last_change":"2023-05-22T15:42:22.460364+0000","last_active":"2023-05-22T15:42:22.837175+0000","last_peered":"2023-05-22T15:42:22.837175+0000","last_clean":"2023-05-22T15:42:22.837175+0000","last_became_active":"2023-05-22T15:42:22.460043+0000","last_became_peered":"2023-05-22T15:42:22.460043+0000","last_unstale":"2023-05-22T15:42:22.837175+0000","last_undegraded":"2023-05-22T15:42:22.837175+0000","last_fullsized":"2023-05-22T15:42:22.837175+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:21.420311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:21.420311+0000","last_clean_scrub_stamp":"2023-05-22T15:42:21.420311+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:58:19.408209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.902206+0000","last_change":"2023-05-22T15:42:22.898485+0000","last_active":"2023-05-22T15:42:22.902206+0000","last_peered":"2023-05-22T15:42:22.902206+0000","last_clean":"2023-05-22T15:42:22.902206+0000","last_became_active":"2023-05-22T15:42:22.897703+0000","last_became_peered":"2023-05-22T15:42:22.897703+0000","last_unstale":"2023-05-22T15:42:22.902206+0000","last_undegraded":"2023-05-22T15:42:22.902206+0000","last_fullsized":"2023-05-22T15:42:22.902206+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_clean_scrub_stamp":"2023-05-22T15:42:22.424085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:56:40.685104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1d","version":"89'2","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.876656+0000","last_change":"2023-05-22T15:42:22.487361+0000","last_active":"2023-05-22T15:42:22.876656+0000","last_peered":"2023-05-22T15:42:22.876656+0000","last_clean":"2023-05-22T15:42:22.876656+0000","last_became_active":"2023-05-22T15:42:22.486708+0000","last_became_peered":"2023-05-22T15:42:22.486708+0000","last_unstale":"2023-05-22T15:42:22.876656+0000","last_undegraded":"2023-05-22T15:42:22.876656+0000","last_fullsized":"2023-05-22T15:42:22.876656+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:21.420311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:21.420311+0000","last_clean_scrub_stamp":"2023-05-22T15:42:21.420311+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:25:37.626039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.905012+0000","last_change":"2023-05-22T15:42:22.894898+0000","last_active":"2023-05-22T15:42:22.905012+0000","last_peered":"2023-05-22T15:42:22.905012+0000","last_clean":"2023-05-22T15:42:22.905012+0000","last_became_active":"2023-05-22T15:42:22.894265+0000","last_became_peered":"2023-05-22T15:42:22.894265+0000","last_unstale":"2023-05-22T15:42:22.905012+0000","last_undegraded":"2023-05-22T15:42:22.905012+0000","last_fullsized":"2023-05-22T15:42:22.905012+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_clean_scrub_stamp":"2023-05-22T15:42:22.424085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:02:51.734869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1c","version":"89'2","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.878480+0000","last_change":"2023-05-22T15:42:22.470932+0000","last_active":"2023-05-22T15:42:22.878480+0000","last_peered":"2023-05-22T15:42:22.878480+0000","last_clean":"2023-05-22T15:42:22.878480+0000","last_became_active":"2023-05-22T15:42:22.470204+0000","last_became_peered":"2023-05-22T15:42:22.470204+0000","last_unstale":"2023-05-22T15:42:22.878480+0000","last_undegraded":"2023-05-22T15:42:22.878480+0000","last_fullsized":"2023-05-22T15:42:22.878480+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:21.420311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:21.420311+0000","last_clean_scrub_stamp":"2023-05-22T15:42:21.420311+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T03:02:20.624682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.875161+0000","last_change":"2023-05-22T15:42:22.865169+0000","last_active":"2023-05-22T15:42:22.875161+0000","last_peered":"2023-05-22T15:42:22.875161+0000","last_clean":"2023-05-22T15:42:22.875161+0000","last_became_active":"2023-05-22T15:42:22.864225+0000","last_became_peered":"2023-05-22T15:42:22.864225+0000","last_unstale":"2023-05-22T15:42:22.875161+0000","last_undegraded":"2023-05-22T15:42:22.875161+0000","last_fullsized":"2023-05-22T15:42:22.875161+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_clean_scrub_stamp":"2023-05-22T15:42:22.424085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T03:21:46.783251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.1b","version":"89'1","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.869943+0000","last_change":"2023-05-22T15:42:22.487840+0000","last_active":"2023-05-22T15:42:22.869943+0000","last_peered":"2023-05-22T15:42:22.869943+0000","last_clean":"2023-05-22T15:42:22.869943+0000","last_became_active":"2023-05-22T15:42:22.487502+0000","last_became_peered":"2023-05-22T15:42:22.487502+0000","last_unstale":"2023-05-22T15:42:22.869943+0000","last_undegraded":"2023-05-22T15:42:22.869943+0000","last_fullsized":"2023-05-22T15:42:22.869943+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:21.420311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:21.420311+0000","last_clean_scrub_stamp":"2023-05-22T15:42:21.420311+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:56:02.678692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.896968+0000","last_change":"2023-05-22T15:42:22.893248+0000","last_active":"2023-05-22T15:42:22.896968+0000","last_peered":"2023-05-22T15:42:22.896968+0000","last_clean":"2023-05-22T15:42:22.896968+0000","last_became_active":"2023-05-22T15:42:22.892636+0000","last_became_peered":"2023-05-22T15:42:22.892636+0000","last_unstale":"2023-05-22T15:42:22.896968+0000","last_undegraded":"2023-05-22T15:42:22.896968+0000","last_fullsized":"2023-05-22T15:42:22.896968+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_clean_scrub_stamp":"2023-05-22T15:42:22.424085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T18:23:22.270527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1a","version":"89'1","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.878187+0000","last_change":"2023-05-22T15:42:22.475557+0000","last_active":"2023-05-22T15:42:22.878187+0000","last_peered":"2023-05-22T15:42:22.878187+0000","last_clean":"2023-05-22T15:42:22.878187+0000","last_became_active":"2023-05-22T15:42:22.471570+0000","last_became_peered":"2023-05-22T15:42:22.471570+0000","last_unstale":"2023-05-22T15:42:22.878187+0000","last_undegraded":"2023-05-22T15:42:22.878187+0000","last_fullsized":"2023-05-22T15:42:22.878187+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:21.420311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:21.420311+0000","last_clean_scrub_stamp":"2023-05-22T15:42:21.420311+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:40:42.401972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.869934+0000","last_change":"2023-05-22T15:42:22.868913+0000","last_active":"2023-05-22T15:42:22.869934+0000","last_peered":"2023-05-22T15:42:22.869934+0000","last_clean":"2023-05-22T15:42:22.869934+0000","last_became_active":"2023-05-22T15:42:22.868590+0000","last_became_peered":"2023-05-22T15:42:22.868590+0000","last_unstale":"2023-05-22T15:42:22.869934+0000","last_undegraded":"2023-05-22T15:42:22.869934+0000","last_fullsized":"2023-05-22T15:42:22.869934+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_clean_scrub_stamp":"2023-05-22T15:42:22.424085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:51:29.460959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.822355+0000","last_change":"2023-05-22T15:42:22.488555+0000","last_active":"2023-05-22T15:42:22.822355+0000","last_peered":"2023-05-22T15:42:22.822355+0000","last_clean":"2023-05-22T15:42:22.822355+0000","last_became_active":"2023-05-22T15:42:22.488238+0000","last_became_peered":"2023-05-22T15:42:22.488238+0000","last_unstale":"2023-05-22T15:42:22.822355+0000","last_undegraded":"2023-05-22T15:42:22.822355+0000","last_fullsized":"2023-05-22T15:42:22.822355+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:21.420311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:21.420311+0000","last_clean_scrub_stamp":"2023-05-22T15:42:21.420311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:37:31.027858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.890682+0000","last_change":"2023-05-22T15:42:22.889892+0000","last_active":"2023-05-22T15:42:22.890682+0000","last_peered":"2023-05-22T15:42:22.890682+0000","last_clean":"2023-05-22T15:42:22.890682+0000","last_became_active":"2023-05-22T15:42:22.889596+0000","last_became_peered":"2023-05-22T15:42:22.889596+0000","last_unstale":"2023-05-22T15:42:22.890682+0000","last_undegraded":"2023-05-22T15:42:22.890682+0000","last_fullsized":"2023-05-22T15:42:22.890682+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_clean_scrub_stamp":"2023-05-22T15:42:22.424085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:58:08.771792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.820055+0000","last_change":"2023-05-22T15:42:22.457062+0000","last_active":"2023-05-22T15:42:22.820055+0000","last_peered":"2023-05-22T15:42:22.820055+0000","last_clean":"2023-05-22T15:42:22.820055+0000","last_became_active":"2023-05-22T15:42:22.456726+0000","last_became_peered":"2023-05-22T15:42:22.456726+0000","last_unstale":"2023-05-22T15:42:22.820055+0000","last_undegraded":"2023-05-22T15:42:22.820055+0000","last_fullsized":"2023-05-22T15:42:22.820055+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:21.420311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:21.420311+0000","last_clean_scrub_stamp":"2023-05-22T15:42:21.420311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:40:42.401972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.896786+0000","last_change":"2023-05-22T15:42:22.891623+0000","last_active":"2023-05-22T15:42:22.896786+0000","last_peered":"2023-05-22T15:42:22.896786+0000","last_clean":"2023-05-22T15:42:22.896786+0000","last_became_active":"2023-05-22T15:42:22.891076+0000","last_became_peered":"2023-05-22T15:42:22.891076+0000","last_unstale":"2023-05-22T15:42:22.896786+0000","last_undegraded":"2023-05-22T15:42:22.896786+0000","last_fullsized":"2023-05-22T15:42:22.896786+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_clean_scrub_stamp":"2023-05-22T15:42:22.424085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:02:51.734869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.832515+0000","last_change":"2023-05-22T15:42:22.489229+0000","last_active":"2023-05-22T15:42:22.832515+0000","last_peered":"2023-05-22T15:42:22.832515+0000","last_clean":"2023-05-22T15:42:22.832515+0000","last_became_active":"2023-05-22T15:42:22.488128+0000","last_became_peered":"2023-05-22T15:42:22.488128+0000","last_unstale":"2023-05-22T15:42:22.832515+0000","last_undegraded":"2023-05-22T15:42:22.832515+0000","last_fullsized":"2023-05-22T15:42:22.832515+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:21.420311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:21.420311+0000","last_clean_scrub_stamp":"2023-05-22T15:42:21.420311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:11:08.656000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.904880+0000","last_change":"2023-05-22T15:42:22.894454+0000","last_active":"2023-05-22T15:42:22.904880+0000","last_peered":"2023-05-22T15:42:22.904880+0000","last_clean":"2023-05-22T15:42:22.904880+0000","last_became_active":"2023-05-22T15:42:22.893741+0000","last_became_peered":"2023-05-22T15:42:22.893741+0000","last_unstale":"2023-05-22T15:42:22.904880+0000","last_undegraded":"2023-05-22T15:42:22.904880+0000","last_fullsized":"2023-05-22T15:42:22.904880+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_clean_scrub_stamp":"2023-05-22T15:42:22.424085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:11:43.253954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.855433+0000","last_change":"2023-05-22T15:42:22.483656+0000","last_active":"2023-05-22T15:42:22.855433+0000","last_peered":"2023-05-22T15:42:22.855433+0000","last_clean":"2023-05-22T15:42:22.855433+0000","last_became_active":"2023-05-22T15:42:22.482887+0000","last_became_peered":"2023-05-22T15:42:22.482887+0000","last_unstale":"2023-05-22T15:42:22.855433+0000","last_undegraded":"2023-05-22T15:42:22.855433+0000","last_fullsized":"2023-05-22T15:42:22.855433+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:21.420311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:21.420311+0000","last_clean_scrub_stamp":"2023-05-22T15:42:21.420311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T15:54:16.759039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.14","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.900116+0000","last_change":"2023-05-22T15:42:22.885448+0000","last_active":"2023-05-22T15:42:22.900116+0000","last_peered":"2023-05-22T15:42:22.900116+0000","last_clean":"2023-05-22T15:42:22.900116+0000","last_became_active":"2023-05-22T15:42:22.880308+0000","last_became_peered":"2023-05-22T15:42:22.880308+0000","last_unstale":"2023-05-22T15:42:22.900116+0000","last_undegraded":"2023-05-22T15:42:22.900116+0000","last_fullsized":"2023-05-22T15:42:22.900116+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_clean_scrub_stamp":"2023-05-22T15:42:22.424085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:32:58.984742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.15","version":"89'3","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.846264+0000","last_change":"2023-05-22T15:42:22.480544+0000","last_active":"2023-05-22T15:42:22.846264+0000","last_peered":"2023-05-22T15:42:22.846264+0000","last_clean":"2023-05-22T15:42:22.846264+0000","last_became_active":"2023-05-22T15:42:22.480208+0000","last_became_peered":"2023-05-22T15:42:22.480208+0000","last_unstale":"2023-05-22T15:42:22.846264+0000","last_undegraded":"2023-05-22T15:42:22.846264+0000","last_fullsized":"2023-05-22T15:42:22.846264+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:21.420311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:21.420311+0000","last_clean_scrub_stamp":"2023-05-22T15:42:21.420311+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T16:20:49.287034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.833337+0000","last_change":"2023-05-22T15:42:22.828778+0000","last_active":"2023-05-22T15:42:22.833337+0000","last_peered":"2023-05-22T15:42:22.833337+0000","last_clean":"2023-05-22T15:42:22.833337+0000","last_became_active":"2023-05-22T15:42:22.828441+0000","last_became_peered":"2023-05-22T15:42:22.828441+0000","last_unstale":"2023-05-22T15:42:22.833337+0000","last_undegraded":"2023-05-22T15:42:22.833337+0000","last_fullsized":"2023-05-22T15:42:22.833337+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_clean_scrub_stamp":"2023-05-22T15:42:22.424085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:28:29.036420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.14","version":"89'6","reported_seq":16,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.872307+0000","last_change":"2023-05-22T15:42:22.460189+0000","last_active":"2023-05-22T15:42:22.872307+0000","last_peered":"2023-05-22T15:42:22.872307+0000","last_clean":"2023-05-22T15:42:22.872307+0000","last_became_active":"2023-05-22T15:42:22.459872+0000","last_became_peered":"2023-05-22T15:42:22.459872+0000","last_unstale":"2023-05-22T15:42:22.872307+0000","last_undegraded":"2023-05-22T15:42:22.872307+0000","last_fullsized":"2023-05-22T15:42:22.872307+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:21.420311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:21.420311+0000","last_clean_scrub_stamp":"2023-05-22T15:42:21.420311+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:44:00.313804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1548,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.904617+0000","last_change":"2023-05-22T15:42:22.890528+0000","last_active":"2023-05-22T15:42:22.904617+0000","last_peered":"2023-05-22T15:42:22.904617+0000","last_clean":"2023-05-22T15:42:22.904617+0000","last_became_active":"2023-05-22T15:42:22.889799+0000","last_became_peered":"2023-05-22T15:42:22.889799+0000","last_unstale":"2023-05-22T15:42:22.904617+0000","last_undegraded":"2023-05-22T15:42:22.904617+0000","last_fullsized":"2023-05-22T15:42:22.904617+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_clean_scrub_stamp":"2023-05-22T15:42:22.424085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:58:20.411983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.13","version":"89'2","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.872731+0000","last_change":"2023-05-22T15:42:22.464127+0000","last_active":"2023-05-22T15:42:22.872731+0000","last_peered":"2023-05-22T15:42:22.872731+0000","last_clean":"2023-05-22T15:42:22.872731+0000","last_became_active":"2023-05-22T15:42:22.462383+0000","last_became_peered":"2023-05-22T15:42:22.462383+0000","last_unstale":"2023-05-22T15:42:22.872731+0000","last_undegraded":"2023-05-22T15:42:22.872731+0000","last_fullsized":"2023-05-22T15:42:22.872731+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:21.420311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:21.420311+0000","last_clean_scrub_stamp":"2023-05-22T15:42:21.420311+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T18:55:05.165963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.878408+0000","last_change":"2023-05-22T15:42:22.872839+0000","last_active":"2023-05-22T15:42:22.878408+0000","last_peered":"2023-05-22T15:42:22.878408+0000","last_clean":"2023-05-22T15:42:22.878408+0000","last_became_active":"2023-05-22T15:42:22.871798+0000","last_became_peered":"2023-05-22T15:42:22.871798+0000","last_unstale":"2023-05-22T15:42:22.878408+0000","last_undegraded":"2023-05-22T15:42:22.878408+0000","last_fullsized":"2023-05-22T15:42:22.878408+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_clean_scrub_stamp":"2023-05-22T15:42:22.424085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:43:09.346983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.855709+0000","last_change":"2023-05-22T15:42:22.472923+0000","last_active":"2023-05-22T15:42:22.855709+0000","last_peered":"2023-05-22T15:42:22.855709+0000","last_clean":"2023-05-22T15:42:22.855709+0000","last_became_active":"2023-05-22T15:42:22.470796+0000","last_became_peered":"2023-05-22T15:42:22.470796+0000","last_unstale":"2023-05-22T15:42:22.855709+0000","last_undegraded":"2023-05-22T15:42:22.855709+0000","last_fullsized":"2023-05-22T15:42:22.855709+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:21.420311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:21.420311+0000","last_clean_scrub_stamp":"2023-05-22T15:42:21.420311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:20:10.951568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.879255+0000","last_change":"2023-05-22T15:42:22.878412+0000","last_active":"2023-05-22T15:42:22.879255+0000","last_peered":"2023-05-22T15:42:22.879255+0000","last_clean":"2023-05-22T15:42:22.879255+0000","last_became_active":"2023-05-22T15:42:22.878012+0000","last_became_peered":"2023-05-22T15:42:22.878012+0000","last_unstale":"2023-05-22T15:42:22.879255+0000","last_undegraded":"2023-05-22T15:42:22.879255+0000","last_fullsized":"2023-05-22T15:42:22.879255+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_clean_scrub_stamp":"2023-05-22T15:42:22.424085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:40:34.254174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.838716+0000","last_change":"2023-05-22T15:42:22.475545+0000","last_active":"2023-05-22T15:42:22.838716+0000","last_peered":"2023-05-22T15:42:22.838716+0000","last_clean":"2023-05-22T15:42:22.838716+0000","last_became_active":"2023-05-22T15:42:22.474862+0000","last_became_peered":"2023-05-22T15:42:22.474862+0000","last_unstale":"2023-05-22T15:42:22.838716+0000","last_undegraded":"2023-05-22T15:42:22.838716+0000","last_fullsized":"2023-05-22T15:42:22.838716+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:21.420311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:21.420311+0000","last_clean_scrub_stamp":"2023-05-22T15:42:21.420311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:10:29.610631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.840675+0000","last_change":"2023-05-22T15:42:22.835298+0000","last_active":"2023-05-22T15:42:22.840675+0000","last_peered":"2023-05-22T15:42:22.840675+0000","last_clean":"2023-05-22T15:42:22.840675+0000","last_became_active":"2023-05-22T15:42:22.834994+0000","last_became_peered":"2023-05-22T15:42:22.834994+0000","last_unstale":"2023-05-22T15:42:22.840675+0000","last_undegraded":"2023-05-22T15:42:22.840675+0000","last_fullsized":"2023-05-22T15:42:22.840675+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_clean_scrub_stamp":"2023-05-22T15:42:22.424085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:40:09.493359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.10","version":"89'8","reported_seq":18,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.898704+0000","last_change":"2023-05-22T15:42:22.485986+0000","last_active":"2023-05-22T15:42:22.898704+0000","last_peered":"2023-05-22T15:42:22.898704+0000","last_clean":"2023-05-22T15:42:22.898704+0000","last_became_active":"2023-05-22T15:42:22.485166+0000","last_became_peered":"2023-05-22T15:42:22.485166+0000","last_unstale":"2023-05-22T15:42:22.898704+0000","last_undegraded":"2023-05-22T15:42:22.898704+0000","last_fullsized":"2023-05-22T15:42:22.898704+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:21.420311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:21.420311+0000","last_clean_scrub_stamp":"2023-05-22T15:42:21.420311+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T16:27:16.644358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.838259+0000","last_change":"2023-05-22T15:42:22.833492+0000","last_active":"2023-05-22T15:42:22.838259+0000","last_peered":"2023-05-22T15:42:22.838259+0000","last_clean":"2023-05-22T15:42:22.838259+0000","last_became_active":"2023-05-22T15:42:22.833168+0000","last_became_peered":"2023-05-22T15:42:22.833168+0000","last_unstale":"2023-05-22T15:42:22.838259+0000","last_undegraded":"2023-05-22T15:42:22.838259+0000","last_fullsized":"2023-05-22T15:42:22.838259+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_clean_scrub_stamp":"2023-05-22T15:42:22.424085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T16:45:37.677128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"89'2","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.883114+0000","last_change":"2023-05-22T15:42:22.476545+0000","last_active":"2023-05-22T15:42:22.883114+0000","last_peered":"2023-05-22T15:42:22.883114+0000","last_clean":"2023-05-22T15:42:22.883114+0000","last_became_active":"2023-05-22T15:42:22.475817+0000","last_became_peered":"2023-05-22T15:42:22.475817+0000","last_unstale":"2023-05-22T15:42:22.883114+0000","last_undegraded":"2023-05-22T15:42:22.883114+0000","last_fullsized":"2023-05-22T15:42:22.883114+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:21.420311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:21.420311+0000","last_clean_scrub_stamp":"2023-05-22T15:42:21.420311+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:48:49.697519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.863757+0000","last_change":"2023-05-22T15:42:22.847558+0000","last_active":"2023-05-22T15:42:22.863757+0000","last_peered":"2023-05-22T15:42:22.863757+0000","last_clean":"2023-05-22T15:42:22.863757+0000","last_became_active":"2023-05-22T15:42:22.846836+0000","last_became_peered":"2023-05-22T15:42:22.846836+0000","last_unstale":"2023-05-22T15:42:22.863757+0000","last_undegraded":"2023-05-22T15:42:22.863757+0000","last_fullsized":"2023-05-22T15:42:22.863757+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_clean_scrub_stamp":"2023-05-22T15:42:22.424085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:12:13.799667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.840291+0000","last_change":"2023-05-22T15:42:22.463938+0000","last_active":"2023-05-22T15:42:22.840291+0000","last_peered":"2023-05-22T15:42:22.840291+0000","last_clean":"2023-05-22T15:42:22.840291+0000","last_became_active":"2023-05-22T15:42:22.463146+0000","last_became_peered":"2023-05-22T15:42:22.463146+0000","last_unstale":"2023-05-22T15:42:22.840291+0000","last_undegraded":"2023-05-22T15:42:22.840291+0000","last_fullsized":"2023-05-22T15:42:22.840291+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:21.420311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:21.420311+0000","last_clean_scrub_stamp":"2023-05-22T15:42:21.420311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:33:19.124743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.904620+0000","last_change":"2023-05-22T15:42:22.890342+0000","last_active":"2023-05-22T15:42:22.904620+0000","last_peered":"2023-05-22T15:42:22.904620+0000","last_clean":"2023-05-22T15:42:22.904620+0000","last_became_active":"2023-05-22T15:42:22.889863+0000","last_became_peered":"2023-05-22T15:42:22.889863+0000","last_unstale":"2023-05-22T15:42:22.904620+0000","last_undegraded":"2023-05-22T15:42:22.904620+0000","last_fullsized":"2023-05-22T15:42:22.904620+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_clean_scrub_stamp":"2023-05-22T15:42:22.424085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T18:12:08.205505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.820217+0000","last_change":"2023-05-22T15:42:22.458720+0000","last_active":"2023-05-22T15:42:22.820217+0000","last_peered":"2023-05-22T15:42:22.820217+0000","last_clean":"2023-05-22T15:42:22.820217+0000","last_became_active":"2023-05-22T15:42:22.458403+0000","last_became_peered":"2023-05-22T15:42:22.458403+0000","last_unstale":"2023-05-22T15:42:22.820217+0000","last_undegraded":"2023-05-22T15:42:22.820217+0000","last_fullsized":"2023-05-22T15:42:22.820217+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:21.420311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:21.420311+0000","last_clean_scrub_stamp":"2023-05-22T15:42:21.420311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T16:31:14.665256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.904802+0000","last_change":"2023-05-22T15:42:22.893748+0000","last_active":"2023-05-22T15:42:22.904802+0000","last_peered":"2023-05-22T15:42:22.904802+0000","last_clean":"2023-05-22T15:42:22.904802+0000","last_became_active":"2023-05-22T15:42:22.893073+0000","last_became_peered":"2023-05-22T15:42:22.893073+0000","last_unstale":"2023-05-22T15:42:22.904802+0000","last_undegraded":"2023-05-22T15:42:22.904802+0000","last_fullsized":"2023-05-22T15:42:22.904802+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_clean_scrub_stamp":"2023-05-22T15:42:22.424085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:12:13.799667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.826337+0000","last_change":"2023-05-22T15:42:22.488108+0000","last_active":"2023-05-22T15:42:22.826337+0000","last_peered":"2023-05-22T15:42:22.826337+0000","last_clean":"2023-05-22T15:42:22.826337+0000","last_became_active":"2023-05-22T15:42:22.487683+0000","last_became_peered":"2023-05-22T15:42:22.487683+0000","last_unstale":"2023-05-22T15:42:22.826337+0000","last_undegraded":"2023-05-22T15:42:22.826337+0000","last_fullsized":"2023-05-22T15:42:22.826337+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:21.420311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:21.420311+0000","last_clean_scrub_stamp":"2023-05-22T15:42:21.420311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:04:43.761724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.840649+0000","last_change":"2023-05-22T15:42:22.837171+0000","last_active":"2023-05-22T15:42:22.840649+0000","last_peered":"2023-05-22T15:42:22.840649+0000","last_clean":"2023-05-22T15:42:22.840649+0000","last_became_active":"2023-05-22T15:42:22.834859+0000","last_became_peered":"2023-05-22T15:42:22.834859+0000","last_unstale":"2023-05-22T15:42:22.840649+0000","last_undegraded":"2023-05-22T15:42:22.840649+0000","last_fullsized":"2023-05-22T15:42:22.840649+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_clean_scrub_stamp":"2023-05-22T15:42:22.424085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T03:03:57.842619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.819810+0000","last_change":"2023-05-22T15:42:22.449651+0000","last_active":"2023-05-22T15:42:22.819810+0000","last_peered":"2023-05-22T15:42:22.819810+0000","last_clean":"2023-05-22T15:42:22.819810+0000","last_became_active":"2023-05-22T15:42:22.449039+0000","last_became_peered":"2023-05-22T15:42:22.449039+0000","last_unstale":"2023-05-22T15:42:22.819810+0000","last_undegraded":"2023-05-22T15:42:22.819810+0000","last_fullsized":"2023-05-22T15:42:22.819810+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:21.420311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:21.420311+0000","last_clean_scrub_stamp":"2023-05-22T15:42:21.420311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:17:21.439320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.902230+0000","last_change":"2023-05-22T15:42:22.898314+0000","last_active":"2023-05-22T15:42:22.902230+0000","last_peered":"2023-05-22T15:42:22.902230+0000","last_clean":"2023-05-22T15:42:22.902230+0000","last_became_active":"2023-05-22T15:42:22.897870+0000","last_became_peered":"2023-05-22T15:42:22.897870+0000","last_unstale":"2023-05-22T15:42:22.902230+0000","last_undegraded":"2023-05-22T15:42:22.902230+0000","last_fullsized":"2023-05-22T15:42:22.902230+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_clean_scrub_stamp":"2023-05-22T15:42:22.424085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:24:27.275366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.822162+0000","last_change":"2023-05-22T15:42:22.488090+0000","last_active":"2023-05-22T15:42:22.822162+0000","last_peered":"2023-05-22T15:42:22.822162+0000","last_clean":"2023-05-22T15:42:22.822162+0000","last_became_active":"2023-05-22T15:42:22.487780+0000","last_became_peered":"2023-05-22T15:42:22.487780+0000","last_unstale":"2023-05-22T15:42:22.822162+0000","last_undegraded":"2023-05-22T15:42:22.822162+0000","last_fullsized":"2023-05-22T15:42:22.822162+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:21.420311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:21.420311+0000","last_clean_scrub_stamp":"2023-05-22T15:42:21.420311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T18:57:06.138974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.837240+0000","last_change":"2023-05-22T15:42:22.834742+0000","last_active":"2023-05-22T15:42:22.837240+0000","last_peered":"2023-05-22T15:42:22.837240+0000","last_clean":"2023-05-22T15:42:22.837240+0000","last_became_active":"2023-05-22T15:42:22.834440+0000","last_became_peered":"2023-05-22T15:42:22.834440+0000","last_unstale":"2023-05-22T15:42:22.837240+0000","last_undegraded":"2023-05-22T15:42:22.837240+0000","last_fullsized":"2023-05-22T15:42:22.837240+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_clean_scrub_stamp":"2023-05-22T15:42:22.424085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:56:40.685104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.831673+0000","last_change":"2023-05-22T15:42:22.485738+0000","last_active":"2023-05-22T15:42:22.831673+0000","last_peered":"2023-05-22T15:42:22.831673+0000","last_clean":"2023-05-22T15:42:22.831673+0000","last_became_active":"2023-05-22T15:42:22.485068+0000","last_became_peered":"2023-05-22T15:42:22.485068+0000","last_unstale":"2023-05-22T15:42:22.831673+0000","last_undegraded":"2023-05-22T15:42:22.831673+0000","last_fullsized":"2023-05-22T15:42:22.831673+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:21.420311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:21.420311+0000","last_clean_scrub_stamp":"2023-05-22T15:42:21.420311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:10:29.610631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.888080+0000","last_change":"2023-05-22T15:42:22.880082+0000","last_active":"2023-05-22T15:42:22.888080+0000","last_peered":"2023-05-22T15:42:22.888080+0000","last_clean":"2023-05-22T15:42:22.888080+0000","last_became_active":"2023-05-22T15:42:22.876593+0000","last_became_peered":"2023-05-22T15:42:22.876593+0000","last_unstale":"2023-05-22T15:42:22.888080+0000","last_undegraded":"2023-05-22T15:42:22.888080+0000","last_fullsized":"2023-05-22T15:42:22.888080+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_clean_scrub_stamp":"2023-05-22T15:42:22.424085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:28:29.036420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.853844+0000","last_change":"2023-05-22T15:42:22.476592+0000","last_active":"2023-05-22T15:42:22.853844+0000","last_peered":"2023-05-22T15:42:22.853844+0000","last_clean":"2023-05-22T15:42:22.853844+0000","last_became_active":"2023-05-22T15:42:22.475833+0000","last_became_peered":"2023-05-22T15:42:22.475833+0000","last_unstale":"2023-05-22T15:42:22.853844+0000","last_undegraded":"2023-05-22T15:42:22.853844+0000","last_fullsized":"2023-05-22T15:42:22.853844+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:21.420311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:21.420311+0000","last_clean_scrub_stamp":"2023-05-22T15:42:21.420311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:29:53.579459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.878707+0000","last_change":"2023-05-22T15:42:22.877564+0000","last_active":"2023-05-22T15:42:22.878707+0000","last_peered":"2023-05-22T15:42:22.878707+0000","last_clean":"2023-05-22T15:42:22.878707+0000","last_became_active":"2023-05-22T15:42:22.877185+0000","last_became_peered":"2023-05-22T15:42:22.877185+0000","last_unstale":"2023-05-22T15:42:22.878707+0000","last_undegraded":"2023-05-22T15:42:22.878707+0000","last_fullsized":"2023-05-22T15:42:22.878707+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_clean_scrub_stamp":"2023-05-22T15:42:22.424085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:04:18.558225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.b","version":"89'2","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.881660+0000","last_change":"2023-05-22T15:42:22.470815+0000","last_active":"2023-05-22T15:42:22.881660+0000","last_peered":"2023-05-22T15:42:22.881660+0000","last_clean":"2023-05-22T15:42:22.881660+0000","last_became_active":"2023-05-22T15:42:22.470017+0000","last_became_peered":"2023-05-22T15:42:22.470017+0000","last_unstale":"2023-05-22T15:42:22.881660+0000","last_undegraded":"2023-05-22T15:42:22.881660+0000","last_fullsized":"2023-05-22T15:42:22.881660+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:21.420311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:21.420311+0000","last_clean_scrub_stamp":"2023-05-22T15:42:21.420311+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T16:27:44.490234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"77'161","reported_seq":390,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.454789+0000","last_change":"2023-05-22T15:26:39.318204+0000","last_active":"2023-05-22T15:42:22.454789+0000","last_peered":"2023-05-22T15:42:22.454789+0000","last_clean":"2023-05-22T15:42:22.454789+0000","last_became_active":"2023-05-22T15:26:39.315370+0000","last_became_peered":"2023-05-22T15:26:39.315370+0000","last_unstale":"2023-05-22T15:42:22.454789+0000","last_undegraded":"2023-05-22T15:42:22.454789+0000","last_fullsized":"2023-05-22T15:42:22.454789+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:38.020076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:38.020076+0000","last_clean_scrub_stamp":"2023-05-22T15:26:38.020076+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:06:39.207543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":500,"num_read_kb":924,"num_write":239,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":84,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.444823+0000","last_change":"2023-05-22T15:26:44.971951+0000","last_active":"2023-05-22T15:42:22.444823+0000","last_peered":"2023-05-22T15:42:22.444823+0000","last_clean":"2023-05-22T15:42:22.444823+0000","last_became_active":"2023-05-22T15:26:44.970872+0000","last_became_peered":"2023-05-22T15:26:44.970872+0000","last_unstale":"2023-05-22T15:42:22.444823+0000","last_undegraded":"2023-05-22T15:42:22.444823+0000","last_fullsized":"2023-05-22T15:42:22.444823+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:55:34.752949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.881848+0000","last_change":"2023-05-22T15:42:22.881328+0000","last_active":"2023-05-22T15:42:22.881848+0000","last_peered":"2023-05-22T15:42:22.881848+0000","last_clean":"2023-05-22T15:42:22.881848+0000","last_became_active":"2023-05-22T15:42:22.881046+0000","last_became_peered":"2023-05-22T15:42:22.881046+0000","last_unstale":"2023-05-22T15:42:22.881848+0000","last_undegraded":"2023-05-22T15:42:22.881848+0000","last_fullsized":"2023-05-22T15:42:22.881848+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_clean_scrub_stamp":"2023-05-22T15:42:22.424085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T16:42:25.904179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.826275+0000","last_change":"2023-05-22T15:42:22.460425+0000","last_active":"2023-05-22T15:42:22.826275+0000","last_peered":"2023-05-22T15:42:22.826275+0000","last_clean":"2023-05-22T15:42:22.826275+0000","last_became_active":"2023-05-22T15:42:22.460116+0000","last_became_peered":"2023-05-22T15:42:22.460116+0000","last_unstale":"2023-05-22T15:42:22.826275+0000","last_undegraded":"2023-05-22T15:42:22.826275+0000","last_fullsized":"2023-05-22T15:42:22.826275+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:21.420311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:21.420311+0000","last_clean_scrub_stamp":"2023-05-22T15:42:21.420311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T18:23:21.266754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":81,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.454608+0000","last_change":"2023-05-22T15:26:44.977198+0000","last_active":"2023-05-22T15:42:22.454608+0000","last_peered":"2023-05-22T15:42:22.454608+0000","last_clean":"2023-05-22T15:42:22.454608+0000","last_became_active":"2023-05-22T15:26:44.976445+0000","last_became_peered":"2023-05-22T15:26:44.976445+0000","last_unstale":"2023-05-22T15:42:22.454608+0000","last_undegraded":"2023-05-22T15:42:22.454608+0000","last_fullsized":"2023-05-22T15:42:22.454608+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:12:41.464054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.881138+0000","last_change":"2023-05-22T15:42:22.880493+0000","last_active":"2023-05-22T15:42:22.881138+0000","last_peered":"2023-05-22T15:42:22.881138+0000","last_clean":"2023-05-22T15:42:22.881138+0000","last_became_active":"2023-05-22T15:42:22.880204+0000","last_became_peered":"2023-05-22T15:42:22.880204+0000","last_unstale":"2023-05-22T15:42:22.881138+0000","last_undegraded":"2023-05-22T15:42:22.881138+0000","last_fullsized":"2023-05-22T15:42:22.881138+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_clean_scrub_stamp":"2023-05-22T15:42:22.424085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T03:03:54.290576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.839012+0000","last_change":"2023-05-22T15:42:22.476917+0000","last_active":"2023-05-22T15:42:22.839012+0000","last_peered":"2023-05-22T15:42:22.839012+0000","last_clean":"2023-05-22T15:42:22.839012+0000","last_became_active":"2023-05-22T15:42:22.476209+0000","last_became_peered":"2023-05-22T15:42:22.476209+0000","last_unstale":"2023-05-22T15:42:22.839012+0000","last_undegraded":"2023-05-22T15:42:22.839012+0000","last_fullsized":"2023-05-22T15:42:22.839012+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:21.420311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:21.420311+0000","last_clean_scrub_stamp":"2023-05-22T15:42:21.420311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T16:27:16.644358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":80,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.434673+0000","last_change":"2023-05-22T15:26:44.969469+0000","last_active":"2023-05-22T15:42:22.434673+0000","last_peered":"2023-05-22T15:42:22.434673+0000","last_clean":"2023-05-22T15:42:22.434673+0000","last_became_active":"2023-05-22T15:26:44.969083+0000","last_became_peered":"2023-05-22T15:26:44.969083+0000","last_unstale":"2023-05-22T15:42:22.434673+0000","last_undegraded":"2023-05-22T15:42:22.434673+0000","last_fullsized":"2023-05-22T15:42:22.434673+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:29:23.670067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.839489+0000","last_change":"2023-05-22T15:42:22.837977+0000","last_active":"2023-05-22T15:42:22.839489+0000","last_peered":"2023-05-22T15:42:22.839489+0000","last_clean":"2023-05-22T15:42:22.839489+0000","last_became_active":"2023-05-22T15:42:22.837699+0000","last_became_peered":"2023-05-22T15:42:22.837699+0000","last_unstale":"2023-05-22T15:42:22.839489+0000","last_undegraded":"2023-05-22T15:42:22.839489+0000","last_fullsized":"2023-05-22T15:42:22.839489+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_clean_scrub_stamp":"2023-05-22T15:42:22.424085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:44:21.508851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.857077+0000","last_change":"2023-05-22T15:42:22.472394+0000","last_active":"2023-05-22T15:42:22.857077+0000","last_peered":"2023-05-22T15:42:22.857077+0000","last_clean":"2023-05-22T15:42:22.857077+0000","last_became_active":"2023-05-22T15:42:22.469928+0000","last_became_peered":"2023-05-22T15:42:22.469928+0000","last_unstale":"2023-05-22T15:42:22.857077+0000","last_undegraded":"2023-05-22T15:42:22.857077+0000","last_fullsized":"2023-05-22T15:42:22.857077+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:21.420311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:21.420311+0000","last_clean_scrub_stamp":"2023-05-22T15:42:21.420311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T18:08:33.493154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":84,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.444459+0000","last_change":"2023-05-22T15:26:44.976632+0000","last_active":"2023-05-22T15:42:22.444459+0000","last_peered":"2023-05-22T15:42:22.444459+0000","last_clean":"2023-05-22T15:42:22.444459+0000","last_became_active":"2023-05-22T15:26:44.975546+0000","last_became_peered":"2023-05-22T15:26:44.975546+0000","last_unstale":"2023-05-22T15:42:22.444459+0000","last_undegraded":"2023-05-22T15:42:22.444459+0000","last_fullsized":"2023-05-22T15:42:22.444459+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:42:57.973030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.854023+0000","last_change":"2023-05-22T15:42:22.852710+0000","last_active":"2023-05-22T15:42:22.854023+0000","last_peered":"2023-05-22T15:42:22.854023+0000","last_clean":"2023-05-22T15:42:22.854023+0000","last_became_active":"2023-05-22T15:42:22.852348+0000","last_became_peered":"2023-05-22T15:42:22.852348+0000","last_unstale":"2023-05-22T15:42:22.854023+0000","last_undegraded":"2023-05-22T15:42:22.854023+0000","last_fullsized":"2023-05-22T15:42:22.854023+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_clean_scrub_stamp":"2023-05-22T15:42:22.424085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T16:55:23.105053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.848334+0000","last_change":"2023-05-22T15:42:22.491297+0000","last_active":"2023-05-22T15:42:22.848334+0000","last_peered":"2023-05-22T15:42:22.848334+0000","last_clean":"2023-05-22T15:42:22.848334+0000","last_became_active":"2023-05-22T15:42:22.490770+0000","last_became_peered":"2023-05-22T15:42:22.490770+0000","last_unstale":"2023-05-22T15:42:22.848334+0000","last_undegraded":"2023-05-22T15:42:22.848334+0000","last_fullsized":"2023-05-22T15:42:22.848334+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:21.420311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:21.420311+0000","last_clean_scrub_stamp":"2023-05-22T15:42:21.420311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T16:27:16.644358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":74,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.448308+0000","last_change":"2023-05-22T15:26:45.091475+0000","last_active":"2023-05-22T15:42:22.448308+0000","last_peered":"2023-05-22T15:42:22.448308+0000","last_clean":"2023-05-22T15:42:22.448308+0000","last_became_active":"2023-05-22T15:26:45.091143+0000","last_became_peered":"2023-05-22T15:26:45.091143+0000","last_unstale":"2023-05-22T15:42:22.448308+0000","last_undegraded":"2023-05-22T15:42:22.448308+0000","last_fullsized":"2023-05-22T15:42:22.448308+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:12:41.464054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.834241+0000","last_change":"2023-05-22T15:42:22.830100+0000","last_active":"2023-05-22T15:42:22.834241+0000","last_peered":"2023-05-22T15:42:22.834241+0000","last_clean":"2023-05-22T15:42:22.834241+0000","last_became_active":"2023-05-22T15:42:22.827908+0000","last_became_peered":"2023-05-22T15:42:22.827908+0000","last_unstale":"2023-05-22T15:42:22.834241+0000","last_undegraded":"2023-05-22T15:42:22.834241+0000","last_fullsized":"2023-05-22T15:42:22.834241+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_clean_scrub_stamp":"2023-05-22T15:42:22.424085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:24:27.275366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.d","version":"89'1","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.883095+0000","last_change":"2023-05-22T15:42:22.490470+0000","last_active":"2023-05-22T15:42:22.883095+0000","last_peered":"2023-05-22T15:42:22.883095+0000","last_clean":"2023-05-22T15:42:22.883095+0000","last_became_active":"2023-05-22T15:42:22.489987+0000","last_became_peered":"2023-05-22T15:42:22.489987+0000","last_unstale":"2023-05-22T15:42:22.883095+0000","last_undegraded":"2023-05-22T15:42:22.883095+0000","last_fullsized":"2023-05-22T15:42:22.883095+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:21.420311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:21.420311+0000","last_clean_scrub_stamp":"2023-05-22T15:42:21.420311+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T18:13:50.414629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":81,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.462533+0000","last_change":"2023-05-22T15:26:45.092367+0000","last_active":"2023-05-22T15:42:22.462533+0000","last_peered":"2023-05-22T15:42:22.462533+0000","last_clean":"2023-05-22T15:42:22.462533+0000","last_became_active":"2023-05-22T15:26:45.091951+0000","last_became_peered":"2023-05-22T15:26:45.091951+0000","last_unstale":"2023-05-22T15:42:22.462533+0000","last_undegraded":"2023-05-22T15:42:22.462533+0000","last_fullsized":"2023-05-22T15:42:22.462533+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:42:57.973030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.904997+0000","last_change":"2023-05-22T15:42:22.895829+0000","last_active":"2023-05-22T15:42:22.904997+0000","last_peered":"2023-05-22T15:42:22.904997+0000","last_clean":"2023-05-22T15:42:22.904997+0000","last_became_active":"2023-05-22T15:42:22.895226+0000","last_became_peered":"2023-05-22T15:42:22.895226+0000","last_unstale":"2023-05-22T15:42:22.904997+0000","last_undegraded":"2023-05-22T15:42:22.904997+0000","last_fullsized":"2023-05-22T15:42:22.904997+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_clean_scrub_stamp":"2023-05-22T15:42:22.424085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:57:49.944485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.e","version":"89'2","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.876669+0000","last_change":"2023-05-22T15:42:22.487669+0000","last_active":"2023-05-22T15:42:22.876669+0000","last_peered":"2023-05-22T15:42:22.876669+0000","last_clean":"2023-05-22T15:42:22.876669+0000","last_became_active":"2023-05-22T15:42:22.486890+0000","last_became_peered":"2023-05-22T15:42:22.486890+0000","last_unstale":"2023-05-22T15:42:22.876669+0000","last_undegraded":"2023-05-22T15:42:22.876669+0000","last_fullsized":"2023-05-22T15:42:22.876669+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:21.420311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:21.420311+0000","last_clean_scrub_stamp":"2023-05-22T15:42:21.420311+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:59:22.855595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":74,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.449144+0000","last_change":"2023-05-22T15:26:44.978858+0000","last_active":"2023-05-22T15:42:22.449144+0000","last_peered":"2023-05-22T15:42:22.449144+0000","last_clean":"2023-05-22T15:42:22.449144+0000","last_became_active":"2023-05-22T15:26:44.978443+0000","last_became_peered":"2023-05-22T15:26:44.978443+0000","last_unstale":"2023-05-22T15:42:22.449144+0000","last_undegraded":"2023-05-22T15:42:22.449144+0000","last_fullsized":"2023-05-22T15:42:22.449144+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:46:29.290243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.886338+0000","last_change":"2023-05-22T15:42:22.869686+0000","last_active":"2023-05-22T15:42:22.886338+0000","last_peered":"2023-05-22T15:42:22.886338+0000","last_clean":"2023-05-22T15:42:22.886338+0000","last_became_active":"2023-05-22T15:42:22.868726+0000","last_became_peered":"2023-05-22T15:42:22.868726+0000","last_unstale":"2023-05-22T15:42:22.886338+0000","last_undegraded":"2023-05-22T15:42:22.886338+0000","last_fullsized":"2023-05-22T15:42:22.886338+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_clean_scrub_stamp":"2023-05-22T15:42:22.424085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:43:09.346983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.f","version":"89'1","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.879761+0000","last_change":"2023-05-22T15:42:22.473483+0000","last_active":"2023-05-22T15:42:22.879761+0000","last_peered":"2023-05-22T15:42:22.879761+0000","last_clean":"2023-05-22T15:42:22.879761+0000","last_became_active":"2023-05-22T15:42:22.472160+0000","last_became_peered":"2023-05-22T15:42:22.472160+0000","last_unstale":"2023-05-22T15:42:22.879761+0000","last_undegraded":"2023-05-22T15:42:22.879761+0000","last_fullsized":"2023-05-22T15:42:22.879761+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:21.420311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:21.420311+0000","last_clean_scrub_stamp":"2023-05-22T15:42:21.420311+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:17:21.439320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":76,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.456529+0000","last_change":"2023-05-22T15:26:44.982421+0000","last_active":"2023-05-22T15:42:22.456529+0000","last_peered":"2023-05-22T15:42:22.456529+0000","last_clean":"2023-05-22T15:42:22.456529+0000","last_became_active":"2023-05-22T15:26:44.981586+0000","last_became_peered":"2023-05-22T15:26:44.981586+0000","last_unstale":"2023-05-22T15:42:22.456529+0000","last_undegraded":"2023-05-22T15:42:22.456529+0000","last_fullsized":"2023-05-22T15:42:22.456529+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:14:11.156544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":10,"num_pg":32,"stat_sum":{"num_bytes":2330,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":500,"num_read_kb":924,"num_write":239,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161,"ondisk_log_size":161,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575039,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30068,"kb_used_data":1644,"kb_used_omap":0,"kb_used_meta":28416,"kb_avail":94341772,"statfs":{"total":96636764160,"available":96605974528,"internally_reserved":0,"allocated":1683456,"data_stored":1155640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":29097984},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 15:41:58 2023","interfaces":[{"interface":"back","average":{"1min":0.88900000000000001,"5min":0.86099999999999999,"15min":0.86699999999999999},"min":{"1min":0.624,"5min":0.44800000000000001,"15min":0.40899999999999997},"max":{"1min":1.1830000000000001,"5min":1.4199999999999999,"15min":1.7709999999999999},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.86899999999999999,"5min":0.871,"15min":0.876},"min":{"1min":0.63600000000000001,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":1.2050000000000001,"5min":1.3169999999999999,"15min":1.4239999999999999},"last":0.60599999999999998}]},{"osd":1,"last update":"Mon May 22 15:41:58 2023","interfaces":[{"interface":"back","average":{"1min":0.94799999999999995,"5min":0.98199999999999998,"15min":0.98199999999999998},"min":{"1min":0.77300000000000002,"5min":0.5,"15min":0.5},"max":{"1min":1.385,"5min":1.389,"15min":1.8300000000000001},"last":0.92100000000000004},{"interface":"front","average":{"1min":1.0229999999999999,"5min":1.018,"15min":1.0229999999999999},"min":{"1min":0.60799999999999998,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":1.2130000000000001,"5min":1.3169999999999999,"15min":2.161},"last":0.70699999999999996}]},{"osd":2,"last update":"Mon May 22 15:41:58 2023","interfaces":[{"interface":"back","average":{"1min":0.84899999999999998,"5min":0.87,"15min":0.90400000000000003},"min":{"1min":0.67300000000000004,"5min":0.49399999999999999,"15min":0.46800000000000003},"max":{"1min":1.181,"5min":1.2070000000000001,"15min":6.7279999999999998},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.95799999999999996,"5min":0.94799999999999995,"15min":0.96299999999999997},"min":{"1min":0.59199999999999997,"5min":0.52300000000000002,"15min":0.51000000000000001},"max":{"1min":1.151,"5min":1.2569999999999999,"15min":6.8810000000000002},"last":0.79500000000000004}]},{"osd":3,"last update":"Mon May 22 15:41:58 2023","interfaces":[{"interface":"back","average":{"1min":1.0109999999999999,"5min":0.95299999999999996,"15min":0.96099999999999997},"min":{"1min":0.70199999999999996,"5min":0.55900000000000005,"15min":0.53800000000000003},"max":{"1min":1.254,"5min":1.3360000000000001,"15min":3.0569999999999999},"last":1.091},{"interface":"front","average":{"1min":0.91500000000000004,"5min":0.96599999999999997,"15min":0.98399999999999999},"min":{"1min":0.68200000000000005,"5min":0.61799999999999999,"15min":0.54900000000000004},"max":{"1min":1.2989999999999999,"5min":1.3029999999999999,"15min":1.6879999999999999},"last":0.84699999999999998}]},{"osd":4,"last update":"Mon May 22 15:41:58 2023","interfaces":[{"interface":"back","average":{"1min":0.83599999999999997,"5min":0.89600000000000002,"15min":0.88400000000000001},"min":{"1min":0.62,"5min":0.57399999999999995,"15min":0.57399999999999995},"max":{"1min":1.093,"5min":1.621,"15min":1.621},"last":1.266},{"interface":"front","average":{"1min":0.98299999999999998,"5min":0.99199999999999999,"15min":0.98299999999999998},"min":{"1min":0.749,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.359,"5min":1.4319999999999999,"15min":1.958},"last":1.2130000000000001}]},{"osd":5,"last update":"Mon May 22 15:41:58 2023","interfaces":[{"interface":"back","average":{"1min":0.995,"5min":1.022,"15min":1.002},"min":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.56699999999999995},"max":{"1min":1.3939999999999999,"5min":1.409,"15min":1.5029999999999999},"last":1.177},{"interface":"front","average":{"1min":1.1220000000000001,"5min":1.121,"15min":1.1359999999999999},"min":{"1min":0.88800000000000001,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":1.325,"5min":1.5229999999999999,"15min":5.0839999999999996},"last":1.141}]},{"osd":6,"last update":"Mon May 22 15:41:58 2023","interfaces":[{"interface":"back","average":{"1min":1.0169999999999999,"5min":1.0449999999999999,"15min":1.071},"min":{"1min":0.73399999999999999,"5min":0.65800000000000003,"15min":0.65800000000000003},"max":{"1min":1.256,"5min":1.484,"15min":5.1719999999999997},"last":0.96799999999999997},{"interface":"front","average":{"1min":1.109,"5min":1.133,"15min":1.1339999999999999},"min":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"max":{"1min":1.462,"5min":1.702,"15min":2.04},"last":1.3220000000000001}]}]},{"osd":6,"up_from":13,"seq":55834575040,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29712,"kb_used_data":1032,"kb_used_omap":0,"kb_used_meta":28672,"kb_avail":94342128,"statfs":{"total":96636764160,"available":96606339072,"internally_reserved":0,"allocated":1056768,"data_stored":550276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":29360128},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 15:42:06 2023","interfaces":[{"interface":"back","average":{"1min":0.86599999999999999,"5min":0.89000000000000001,"15min":0.89800000000000002},"min":{"1min":0.59299999999999997,"5min":0.496,"15min":0.496},"max":{"1min":1.323,"5min":1.327,"15min":1.355},"last":0.89900000000000002},{"interface":"front","average":{"1min":0.99199999999999999,"5min":0.90300000000000002,"15min":0.90900000000000003},"min":{"1min":0.66200000000000003,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":1.264,"5min":1.5680000000000001,"15min":1.5680000000000001},"last":0.76000000000000001}]},{"osd":1,"last update":"Mon May 22 15:42:06 2023","interfaces":[{"interface":"back","average":{"1min":0.92800000000000005,"5min":0.92800000000000005,"15min":0.91300000000000003},"min":{"1min":0.623,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":1.2709999999999999,"5min":1.3169999999999999,"15min":1.375},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.93400000000000005,"5min":0.94499999999999995,"15min":0.94899999999999995},"min":{"1min":0.64400000000000002,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.2709999999999999,"5min":1.554,"15min":1.554},"last":0.98799999999999999}]},{"osd":2,"last update":"Mon May 22 15:42:06 2023","interfaces":[{"interface":"back","average":{"1min":1.0029999999999999,"5min":0.97099999999999997,"15min":0.97999999999999998},"min":{"1min":0.72599999999999998,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":1.3660000000000001,"5min":1.425,"15min":1.425},"last":1.0700000000000001},{"interface":"front","average":{"1min":1.073,"5min":1.048,"15min":1.0509999999999999},"min":{"1min":0.84299999999999997,"5min":0.442,"15min":0.442},"max":{"1min":1.2909999999999999,"5min":1.5369999999999999,"15min":1.5369999999999999},"last":0.81899999999999995}]},{"osd":3,"last update":"Mon May 22 15:42:06 2023","interfaces":[{"interface":"back","average":{"1min":0.95799999999999996,"5min":0.99099999999999999,"15min":0.99399999999999999},"min":{"1min":0.72599999999999998,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":1.1830000000000001,"5min":1.679,"15min":1.679},"last":1.042},{"interface":"front","average":{"1min":0.96799999999999997,"5min":0.95699999999999996,"15min":0.96099999999999997},"min":{"1min":0.62,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.159,"5min":1.4570000000000001,"15min":1.486},"last":1.1559999999999999}]},{"osd":4,"last update":"Mon May 22 15:42:06 2023","interfaces":[{"interface":"back","average":{"1min":0.85699999999999998,"5min":0.91700000000000004,"15min":0.93100000000000005},"min":{"1min":0.64900000000000002,"5min":0.60299999999999998,"15min":0.57599999999999996},"max":{"1min":1.1479999999999999,"5min":1.6859999999999999,"15min":1.6859999999999999},"last":0.62},{"interface":"front","average":{"1min":0.96199999999999997,"5min":1.016,"15min":1.032},"min":{"1min":0.67700000000000005,"5min":0.622,"15min":0.61299999999999999},"max":{"1min":1.321,"5min":1.8300000000000001,"15min":1.8300000000000001},"last":0.85499999999999998}]},{"osd":5,"last update":"Mon May 22 15:42:06 2023","interfaces":[{"interface":"back","average":{"1min":1.01,"5min":0.98099999999999998,"15min":0.97999999999999998},"min":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.64400000000000002},"max":{"1min":1.4530000000000001,"5min":1.6499999999999999,"15min":1.6499999999999999},"last":0.66900000000000004},{"interface":"front","average":{"1min":1.0760000000000001,"5min":1.103,"15min":1.1000000000000001},"min":{"1min":0.68799999999999994,"5min":0.68000000000000005,"15min":0.68000000000000005},"max":{"1min":1.3919999999999999,"5min":1.607,"15min":1.7509999999999999},"last":1.0169999999999999}]},{"osd":7,"last update":"Mon May 22 15:42:06 2023","interfaces":[{"interface":"back","average":{"1min":1.1180000000000001,"5min":1.1160000000000001,"15min":1.1359999999999999},"min":{"1min":0.78700000000000003,"5min":0.68400000000000005,"15min":0.58399999999999996},"max":{"1min":1.3740000000000001,"5min":1.9179999999999999,"15min":1.9179999999999999},"last":1.109},{"interface":"front","average":{"1min":1.1040000000000001,"5min":1.139,"15min":1.133},"min":{"1min":0.81899999999999995,"5min":0.74099999999999999,"15min":0.68999999999999995},"max":{"1min":1.3580000000000001,"5min":1.8520000000000001,"15min":1.8520000000000001},"last":0.94299999999999995}]}]},{"osd":5,"up_from":13,"seq":55834575040,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":23696,"kb_used_data":1032,"kb_used_omap":0,"kb_used_meta":22656,"kb_avail":94348144,"statfs":{"total":96636764160,"available":96612499456,"internally_reserved":0,"allocated":1056768,"data_stored":550276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":23199744},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 15:42:04 2023","interfaces":[{"interface":"back","average":{"1min":0.92100000000000004,"5min":0.86799999999999999,"15min":0.88},"min":{"1min":0.68899999999999995,"5min":0.54900000000000004,"15min":0.48399999999999999},"max":{"1min":1.2849999999999999,"5min":1.3440000000000001,"15min":1.3440000000000001},"last":1.016},{"interface":"front","average":{"1min":0.93600000000000005,"5min":0.89800000000000002,"15min":0.88},"min":{"1min":0.58499999999999996,"5min":0.57499999999999996,"15min":0.497},"max":{"1min":1.2549999999999999,"5min":2.1699999999999999,"15min":2.1699999999999999},"last":1.0449999999999999}]},{"osd":1,"last update":"Mon May 22 15:42:04 2023","interfaces":[{"interface":"back","average":{"1min":1.099,"5min":1.004,"15min":0.996},"min":{"1min":0.87,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":1.337,"5min":1.9370000000000001,"15min":1.9370000000000001},"last":1.163},{"interface":"front","average":{"1min":1.028,"5min":0.999,"15min":1.02},"min":{"1min":0.67000000000000004,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.3680000000000001,"5min":1.3680000000000001,"15min":1.399},"last":1.1120000000000001}]},{"osd":2,"last update":"Mon May 22 15:42:04 2023","interfaces":[{"interface":"back","average":{"1min":0.95699999999999996,"5min":0.96299999999999997,"15min":0.93200000000000005},"min":{"1min":0.72699999999999998,"5min":0.504,"15min":0.504},"max":{"1min":1.2549999999999999,"5min":1.9830000000000001,"15min":1.9830000000000001},"last":1.1379999999999999},{"interface":"front","average":{"1min":1.0189999999999999,"5min":0.96099999999999997,"15min":0.93700000000000006},"min":{"1min":0.75900000000000001,"5min":0.60799999999999998,"15min":0.53800000000000003},"max":{"1min":1.321,"5min":2.1120000000000001,"15min":2.1120000000000001},"last":0.95199999999999996}]},{"osd":3,"last update":"Mon May 22 15:42:04 2023","interfaces":[{"interface":"back","average":{"1min":1.0029999999999999,"5min":0.98899999999999999,"15min":0.95699999999999996},"min":{"1min":0.67000000000000004,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.2809999999999999,"5min":1.4419999999999999,"15min":1.6839999999999999},"last":0.79000000000000004},{"interface":"front","average":{"1min":0.97399999999999998,"5min":1.056,"15min":1.038},"min":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":1.1970000000000001,"5min":2.52,"15min":2.52},"last":0.88400000000000001}]},{"osd":4,"last update":"Mon May 22 15:42:04 2023","interfaces":[{"interface":"back","average":{"1min":0.90900000000000003,"5min":0.97799999999999998,"15min":0.93799999999999994},"min":{"1min":0.68200000000000005,"5min":0.60599999999999998,"15min":0.57599999999999996},"max":{"1min":1.2609999999999999,"5min":2.226,"15min":2.226},"last":1.073},{"interface":"front","average":{"1min":1.075,"5min":1.0649999999999999,"15min":1.038},"min":{"1min":0.63200000000000001,"5min":0.60199999999999998,"15min":0.60199999999999998},"max":{"1min":1.4239999999999999,"5min":1.6859999999999999,"15min":1.6859999999999999},"last":0.69099999999999995}]},{"osd":6,"last update":"Mon May 22 15:42:04 2023","interfaces":[{"interface":"back","average":{"1min":1.016,"5min":1.0840000000000001,"15min":1.0449999999999999},"min":{"1min":0.748,"5min":0.67900000000000005,"15min":0.61299999999999999},"max":{"1min":1.345,"5min":2.831,"15min":2.831},"last":0.751},{"interface":"front","average":{"1min":1.085,"5min":1.1080000000000001,"15min":1.0780000000000001},"min":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.69299999999999995},"max":{"1min":1.3380000000000001,"5min":2.3690000000000002,"15min":2.3690000000000002},"last":0.81899999999999995}]},{"osd":7,"last update":"Mon May 22 15:42:04 2023","interfaces":[{"interface":"back","average":{"1min":1.077,"5min":1.1639999999999999,"15min":1.125},"min":{"1min":0.70999999999999996,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":1.47,"5min":1.849,"15min":2.8260000000000001},"last":0.92000000000000004},{"interface":"front","average":{"1min":1.1579999999999999,"5min":1.22,"15min":1.1699999999999999},"min":{"1min":0.80400000000000005,"5min":0.75700000000000001,"15min":0.71699999999999997},"max":{"1min":1.3220000000000001,"5min":2.7429999999999999,"15min":2.7429999999999999},"last":0.99199999999999999}]}]},{"osd":4,"up_from":13,"seq":55834575038,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":25424,"kb_used_data":1032,"kb_used_omap":0,"kb_used_meta":24384,"kb_avail":94346416,"statfs":{"total":96636764160,"available":96610729984,"internally_reserved":0,"allocated":1056768,"data_stored":550276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":24969216},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 15:42:07 2023","interfaces":[{"interface":"back","average":{"1min":0.88,"5min":0.91300000000000003,"15min":0.90600000000000003},"min":{"1min":0.58299999999999996,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":1.3360000000000001,"5min":1.9910000000000001,"15min":1.9910000000000001},"last":0.80200000000000005},{"interface":"front","average":{"1min":0.96899999999999997,"5min":0.91900000000000004,"15min":0.93200000000000005},"min":{"1min":0.63500000000000001,"5min":0.373,"15min":0.373},"max":{"1min":1.306,"5min":1.3779999999999999,"15min":1.633},"last":0.65100000000000002}]},{"osd":1,"last update":"Mon May 22 15:42:07 2023","interfaces":[{"interface":"back","average":{"1min":1.0029999999999999,"5min":0.98199999999999998,"15min":0.97699999999999998},"min":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.54000000000000004},"max":{"1min":1.2689999999999999,"5min":1.383,"15min":1.3859999999999999},"last":0.71399999999999997},{"interface":"front","average":{"1min":0.88900000000000001,"5min":0.93500000000000005,"15min":0.91200000000000003},"min":{"1min":0.60299999999999998,"5min":0.55100000000000005,"15min":0.51600000000000001},"max":{"1min":1.149,"5min":1.3120000000000001,"15min":1.3640000000000001},"last":0.86699999999999999}]},{"osd":2,"last update":"Mon May 22 15:42:07 2023","interfaces":[{"interface":"back","average":{"1min":0.92800000000000005,"5min":0.94699999999999995,"15min":0.94699999999999995},"min":{"1min":0.747,"5min":0.495,"15min":0.495},"max":{"1min":1.212,"5min":2.1019999999999999,"15min":2.1019999999999999},"last":1.0780000000000001},{"interface":"front","average":{"1min":1.0489999999999999,"5min":1.01,"15min":1.0109999999999999},"min":{"1min":0.77600000000000002,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":1.4390000000000001,"5min":1.4390000000000001,"15min":1.9770000000000001},"last":0.91900000000000004}]},{"osd":3,"last update":"Mon May 22 15:42:07 2023","interfaces":[{"interface":"back","average":{"1min":0.89800000000000002,"5min":0.91800000000000004,"15min":0.94299999999999995},"min":{"1min":0.72399999999999998,"5min":0.63100000000000001,"15min":0.54600000000000004},"max":{"1min":1.2490000000000001,"5min":1.29,"15min":1.665},"last":1.153},{"interface":"front","average":{"1min":1.075,"5min":1.0529999999999999,"15min":1.038},"min":{"1min":0.85399999999999998,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":1.238,"5min":2.1720000000000002,"15min":2.1720000000000002},"last":1.218}]},{"osd":5,"last update":"Mon May 22 15:42:07 2023","interfaces":[{"interface":"back","average":{"1min":1.0569999999999999,"5min":1.0409999999999999,"15min":1},"min":{"1min":0.83399999999999996,"5min":0.621,"15min":0.51200000000000001},"max":{"1min":1.373,"5min":1.472,"15min":1.7270000000000001},"last":1.0349999999999999},{"interface":"front","average":{"1min":1.012,"5min":0.98599999999999999,"15min":0.97999999999999998},"min":{"1min":0.70799999999999996,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":1.325,"5min":1.401,"15min":1.6719999999999999},"last":1.2649999999999999}]},{"osd":6,"last update":"Mon May 22 15:42:07 2023","interfaces":[{"interface":"back","average":{"1min":1.0089999999999999,"5min":1.028,"15min":1.0029999999999999},"min":{"1min":0.64500000000000002,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":1.351,"5min":2.2349999999999999,"15min":2.2349999999999999},"last":1.3620000000000001},{"interface":"front","average":{"1min":1.1379999999999999,"5min":1.1419999999999999,"15min":1.1160000000000001},"min":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.69799999999999995},"max":{"1min":1.486,"5min":1.8220000000000001,"15min":2.0699999999999998},"last":1.1160000000000001}]},{"osd":7,"last update":"Mon May 22 15:42:07 2023","interfaces":[{"interface":"back","average":{"1min":1.087,"5min":1.0580000000000001,"15min":1.0680000000000001},"min":{"1min":0.78200000000000003,"5min":0.71899999999999997,"15min":0.67700000000000005},"max":{"1min":1.3140000000000001,"5min":1.5800000000000001,"15min":1.6639999999999999},"last":1.446},{"interface":"front","average":{"1min":1.1930000000000001,"5min":1.1639999999999999,"15min":1.1459999999999999},"min":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.71499999999999997},"max":{"1min":1.4370000000000001,"5min":2.306,"15min":2.306},"last":1.413}]}]},{"osd":3,"up_from":13,"seq":55834575038,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34588,"kb_used_data":1044,"kb_used_omap":0,"kb_used_meta":33536,"kb_avail":94337252,"statfs":{"total":96636764160,"available":96601346048,"internally_reserved":0,"allocated":1069056,"data_stored":557224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":34340864},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 15:41:51 2023","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.63100000000000001,"15min":0.625},"min":{"1min":0.50700000000000001,"5min":0.19,"15min":0.19},"max":{"1min":0.79100000000000004,"5min":1.194,"15min":1.194},"last":0.69599999999999995},{"interface":"front","average":{"1min":0.79700000000000004,"5min":0.71999999999999997,"15min":0.69899999999999995},"min":{"1min":0.54700000000000004,"5min":0.32600000000000001,"15min":0.23699999999999999},"max":{"1min":0.97399999999999998,"5min":1.1799999999999999,"15min":1.296},"last":0.55700000000000005}]},{"osd":1,"last update":"Mon May 22 15:41:51 2023","interfaces":[{"interface":"back","average":{"1min":0.69499999999999995,"5min":0.64300000000000002,"15min":0.66300000000000003},"min":{"1min":0.44900000000000001,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":1.004,"5min":1.008,"15min":1.0600000000000001},"last":0.80000000000000004},{"interface":"front","average":{"1min":0.82599999999999996,"5min":0.77500000000000002,"15min":0.76700000000000002},"min":{"1min":0.57799999999999996,"5min":0.313,"15min":0.216},"max":{"1min":1.1359999999999999,"5min":1.222,"15min":1.6579999999999999},"last":0.45300000000000001}]},{"osd":2,"last update":"Mon May 22 15:41:51 2023","interfaces":[{"interface":"back","average":{"1min":0.85899999999999999,"5min":0.79800000000000004,"15min":0.81100000000000005},"min":{"1min":0.60899999999999999,"5min":0.36599999999999999,"15min":0.32200000000000001},"max":{"1min":1.157,"5min":1.157,"15min":1.2989999999999999},"last":1.506},{"interface":"front","average":{"1min":0.82099999999999995,"5min":0.80200000000000005,"15min":0.80800000000000005},"min":{"1min":0.60199999999999998,"5min":0.40000000000000002,"15min":0.29699999999999999},"max":{"1min":1.0900000000000001,"5min":1.181,"15min":1.855},"last":1.0740000000000001}]},{"osd":4,"last update":"Mon May 22 15:41:51 2023","interfaces":[{"interface":"back","average":{"1min":0.96999999999999997,"5min":0.94799999999999995,"15min":0.95599999999999996},"min":{"1min":0.80400000000000005,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":1.1970000000000001,"5min":1.2370000000000001,"15min":1.5820000000000001},"last":1.413},{"interface":"front","average":{"1min":1.0529999999999999,"5min":1.0009999999999999,"15min":1.0109999999999999},"min":{"1min":0.82099999999999995,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.238,"5min":1.3740000000000001,"15min":1.8069999999999999},"last":0.92800000000000005}]},{"osd":5,"last update":"Mon May 22 15:41:51 2023","interfaces":[{"interface":"back","average":{"1min":1.0549999999999999,"5min":1.0109999999999999,"15min":1.01},"min":{"1min":0.873,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":1.24,"5min":1.4059999999999999,"15min":1.4339999999999999},"last":1.2070000000000001},{"interface":"front","average":{"1min":1.0189999999999999,"5min":1.016,"15min":1.0489999999999999},"min":{"1min":0.80500000000000005,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":1.2410000000000001,"5min":1.381,"15min":2.4820000000000002},"last":1.3}]},{"osd":6,"last update":"Mon May 22 15:41:51 2023","interfaces":[{"interface":"back","average":{"1min":1.0680000000000001,"5min":1.0249999999999999,"15min":1.0409999999999999},"min":{"1min":0.873,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.254,"5min":1.4139999999999999,"15min":1.863},"last":1.0249999999999999},{"interface":"front","average":{"1min":1.0620000000000001,"5min":1.0640000000000001,"15min":1.0820000000000001},"min":{"1min":0.76600000000000001,"5min":0.66600000000000004,"15min":0.53500000000000003},"max":{"1min":1.25,"5min":1.4470000000000001,"15min":2.6589999999999998},"last":1.159}]},{"osd":7,"last update":"Mon May 22 15:41:51 2023","interfaces":[{"interface":"back","average":{"1min":1.105,"5min":1.085,"15min":1.111},"min":{"1min":0.84799999999999998,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":1.3089999999999999,"5min":1.4990000000000001,"15min":2.762},"last":1.2270000000000001},{"interface":"front","average":{"1min":1.0740000000000001,"5min":1.0549999999999999,"15min":1.077},"min":{"1min":0.71399999999999997,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":1.3799999999999999,"5min":1.3799999999999999,"15min":1.714},"last":1.8700000000000001}]}]},{"osd":2,"up_from":13,"seq":55834575039,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":16820,"kb_used_data":1068,"kb_used_omap":0,"kb_used_meta":15744,"kb_avail":94355020,"statfs":{"total":96636764160,"available":96619540480,"internally_reserved":0,"allocated":1093632,"data_stored":566220,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":16121856},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 15:41:55 2023","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.66000000000000003,"15min":0.65000000000000002},"min":{"1min":0.499,"5min":0.35699999999999998,"15min":0.317},"max":{"1min":0.96999999999999997,"5min":1.8240000000000001,"15min":1.8240000000000001},"last":0.83899999999999997},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.77600000000000002,"15min":0.74299999999999999},"min":{"1min":0.52500000000000002,"5min":0.39600000000000002,"15min":0.253},"max":{"1min":1.0740000000000001,"5min":1.8160000000000001,"15min":1.8160000000000001},"last":0.63100000000000001}]},{"osd":1,"last update":"Mon May 22 15:41:55 2023","interfaces":[{"interface":"back","average":{"1min":0.80600000000000005,"5min":0.74399999999999999,"15min":0.73599999999999999},"min":{"1min":0.46700000000000003,"5min":0.27000000000000002,"15min":0.26900000000000002},"max":{"1min":1.0329999999999999,"5min":1.2,"15min":1.5309999999999999},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.877,"5min":0.83899999999999997,"15min":0.80700000000000005},"min":{"1min":0.60099999999999998,"5min":0.39600000000000002,"15min":0.29099999999999998},"max":{"1min":1.246,"5min":2.5089999999999999,"15min":2.5089999999999999},"last":0.91700000000000004}]},{"osd":3,"last update":"Mon May 22 15:41:55 2023","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.72799999999999998,"15min":0.72099999999999997},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.26700000000000002},"max":{"1min":0.96199999999999997,"5min":2.7629999999999999,"15min":2.7629999999999999},"last":0.79200000000000004},{"interface":"front","average":{"1min":0.85699999999999998,"5min":0.89700000000000002,"15min":0.873},"min":{"1min":0.56100000000000005,"5min":0.44400000000000001,"15min":0.33000000000000002},"max":{"1min":1.042,"5min":2.6840000000000002,"15min":2.6840000000000002},"last":0.67800000000000005}]},{"osd":4,"last update":"Mon May 22 15:41:55 2023","interfaces":[{"interface":"back","average":{"1min":1.012,"5min":1.0329999999999999,"15min":1.0109999999999999},"min":{"1min":0.67700000000000005,"5min":0.65800000000000003,"15min":0.55500000000000005},"max":{"1min":1.244,"5min":2.8039999999999998,"15min":2.8039999999999998},"last":0.999},{"interface":"front","average":{"1min":1.0009999999999999,"5min":1.05,"15min":1.024},"min":{"1min":0.77000000000000002,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":1.3,"5min":2.0819999999999999,"15min":2.0819999999999999},"last":1.2210000000000001}]},{"osd":5,"last update":"Mon May 22 15:41:55 2023","interfaces":[{"interface":"back","average":{"1min":1.1080000000000001,"5min":1.0720000000000001,"15min":1.034},"min":{"1min":0.74099999999999999,"5min":0.629,"15min":0.55200000000000005},"max":{"1min":1.4099999999999999,"5min":2.3860000000000001,"15min":2.4569999999999999},"last":1.274},{"interface":"front","average":{"1min":0.98099999999999998,"5min":1.006,"15min":0.998},"min":{"1min":0.77800000000000002,"5min":0.64100000000000001,"15min":0.56200000000000006},"max":{"1min":1.327,"5min":3.0299999999999998,"15min":3.0299999999999998},"last":1.0269999999999999}]},{"osd":6,"last update":"Mon May 22 15:41:55 2023","interfaces":[{"interface":"back","average":{"1min":1.008,"5min":1.0349999999999999,"15min":1.0389999999999999},"min":{"1min":0.72899999999999998,"5min":0.67400000000000004,"15min":0.56399999999999995},"max":{"1min":1.2250000000000001,"5min":2.883,"15min":3.488},"last":0.88300000000000001},{"interface":"front","average":{"1min":1.0820000000000001,"5min":1.1160000000000001,"15min":1.1000000000000001},"min":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.53400000000000003},"max":{"1min":1.365,"5min":3.1459999999999999,"15min":3.7330000000000001},"last":1.1080000000000001}]},{"osd":7,"last update":"Mon May 22 15:41:55 2023","interfaces":[{"interface":"back","average":{"1min":1.149,"5min":1.1499999999999999,"15min":1.123},"min":{"1min":0.77600000000000002,"5min":0.77000000000000002,"15min":0.58299999999999996},"max":{"1min":1.4970000000000001,"5min":2.5990000000000002,"15min":2.5990000000000002},"last":1.1479999999999999},{"interface":"front","average":{"1min":1.0860000000000001,"5min":1.1200000000000001,"15min":1.0980000000000001},"min":{"1min":0.65500000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":1.3839999999999999,"5min":3.8799999999999999,"15min":3.8799999999999999},"last":0.96999999999999997}]}]},{"osd":1,"up_from":13,"seq":55834575039,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":23504,"kb_used_data":1032,"kb_used_omap":0,"kb_used_meta":22464,"kb_avail":94348336,"statfs":{"total":96636764160,"available":96612696064,"internally_reserved":0,"allocated":1056768,"data_stored":550276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":23003136},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 15:42:12 2023","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.63200000000000001,"15min":0.63800000000000001},"min":{"1min":0.37,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":1.2270000000000001,"5min":1.2270000000000001,"15min":1.7689999999999999},"last":0.26300000000000001},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.78200000000000003,"15min":0.77100000000000002},"min":{"1min":0.38800000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":1.0649999999999999,"5min":1.2290000000000001,"15min":1.3799999999999999},"last":0.29399999999999998}]},{"osd":2,"last update":"Mon May 22 15:42:12 2023","interfaces":[{"interface":"back","average":{"1min":0.78700000000000003,"5min":0.76700000000000002,"15min":0.73299999999999998},"min":{"1min":0.51300000000000001,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":1.391,"5min":1.391,"15min":1.7290000000000001},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.79100000000000004,"5min":0.77200000000000002,"15min":0.75600000000000001},"min":{"1min":0.47799999999999998,"5min":0.35899999999999999,"15min":0.27800000000000002},"max":{"1min":1.2270000000000001,"5min":1.276,"15min":2.4049999999999998},"last":0.36299999999999999}]},{"osd":3,"last update":"Mon May 22 15:42:12 2023","interfaces":[{"interface":"back","average":{"1min":0.88600000000000001,"5min":0.82199999999999995,"15min":0.82899999999999996},"min":{"1min":0.41499999999999998,"5min":0.41399999999999998,"15min":0.317},"max":{"1min":1.657,"5min":1.657,"15min":1.657},"last":0.33500000000000002},{"interface":"front","average":{"1min":0.89000000000000001,"5min":0.82399999999999995,"15min":0.82699999999999996},"min":{"1min":0.46000000000000002,"5min":0.315,"15min":0.315},"max":{"1min":2.0569999999999999,"5min":2.0569999999999999,"15min":7.5620000000000003},"last":0.441}]},{"osd":4,"last update":"Mon May 22 15:42:12 2023","interfaces":[{"interface":"back","average":{"1min":1.002,"5min":0.96999999999999997,"15min":0.95099999999999996},"min":{"1min":0.58499999999999996,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":2.0169999999999999,"5min":2.0169999999999999,"15min":2.0169999999999999},"last":0.65400000000000003},{"interface":"front","average":{"1min":1.03,"5min":0.98599999999999999,"15min":0.98899999999999999},"min":{"1min":0.64100000000000001,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":1.4650000000000001,"5min":1.5369999999999999,"15min":1.5369999999999999},"last":0.504}]},{"osd":5,"last update":"Mon May 22 15:42:12 2023","interfaces":[{"interface":"back","average":{"1min":1.069,"5min":1.0349999999999999,"15min":1.0329999999999999},"min":{"1min":0.59899999999999998,"5min":0.443,"15min":0.443},"max":{"1min":1.9570000000000001,"5min":1.9570000000000001,"15min":1.9570000000000001},"last":0.71699999999999997},{"interface":"front","average":{"1min":1.0720000000000001,"5min":1.0509999999999999,"15min":1.048},"min":{"1min":0.68000000000000005,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":2.1200000000000001,"5min":2.1200000000000001,"15min":4.4710000000000001},"last":0.53400000000000003}]},{"osd":6,"last update":"Mon May 22 15:42:12 2023","interfaces":[{"interface":"back","average":{"1min":1.0449999999999999,"5min":1.012,"15min":1.0680000000000001},"min":{"1min":0.54000000000000004,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.5720000000000001,"5min":1.5720000000000001,"15min":14.968999999999999},"last":0.69099999999999995},{"interface":"front","average":{"1min":1.1319999999999999,"5min":1.083,"15min":1.1359999999999999},"min":{"1min":0.69799999999999995,"5min":0.61399999999999999,"15min":0.56699999999999995},"max":{"1min":1.7629999999999999,"5min":1.7629999999999999,"15min":13.242000000000001},"last":0.54600000000000004}]},{"osd":7,"last update":"Mon May 22 15:42:12 2023","interfaces":[{"interface":"back","average":{"1min":1.1559999999999999,"5min":1.1140000000000001,"15min":1.109},"min":{"1min":0.66200000000000003,"5min":0.56799999999999995,"15min":0.56799999999999995},"max":{"1min":1.865,"5min":1.865,"15min":1.9259999999999999},"last":0.66900000000000004},{"interface":"front","average":{"1min":1.1739999999999999,"5min":1.145,"15min":1.131},"min":{"1min":0.78000000000000003,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":2.2200000000000002,"5min":2.2200000000000002,"15min":5.681},"last":0.57399999999999995}]}]},{"osd":0,"up_from":13,"seq":55834575040,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":22944,"kb_used_data":1624,"kb_used_omap":0,"kb_used_meta":21312,"kb_avail":94348896,"statfs":{"total":96636764160,"available":96613269504,"internally_reserved":0,"allocated":1662976,"data_stored":1147592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":21823488},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon May 22 15:42:07 2023","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.66100000000000003,"15min":0.66800000000000004},"min":{"1min":0.248,"5min":0.223,"15min":0.223},"max":{"1min":0.86899999999999999,"5min":1.077,"15min":1.3049999999999999},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.76300000000000001,"5min":0.72999999999999998,"15min":0.73299999999999998},"min":{"1min":0.53600000000000003,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":1.0369999999999999,"5min":1.2390000000000001,"15min":1.2390000000000001},"last":0.70799999999999996}]},{"osd":2,"last update":"Mon May 22 15:42:07 2023","interfaces":[{"interface":"back","average":{"1min":0.73599999999999999,"5min":0.74199999999999999,"15min":0.74399999999999999},"min":{"1min":0.318,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":1.155,"5min":1.155,"15min":1.4059999999999999},"last":1.004},{"interface":"front","average":{"1min":0.79500000000000004,"5min":0.78000000000000003,"15min":0.79900000000000004},"min":{"1min":0.55400000000000005,"5min":0.32900000000000001,"15min":0.29699999999999999},"max":{"1min":1.05,"5min":1.1850000000000001,"15min":1.3999999999999999},"last":0.81999999999999995}]},{"osd":3,"last update":"Mon May 22 15:42:07 2023","interfaces":[{"interface":"back","average":{"1min":0.86699999999999999,"5min":0.755,"15min":0.76000000000000001},"min":{"1min":0.55600000000000005,"5min":0.40899999999999997,"15min":0.36599999999999999},"max":{"1min":1.9299999999999999,"5min":1.9299999999999999,"15min":1.9299999999999999},"last":0.76300000000000001},{"interface":"front","average":{"1min":0.85799999999999998,"5min":0.86399999999999999,"15min":0.86799999999999999},"min":{"1min":0.35199999999999998,"5min":0.34399999999999997,"15min":0.33000000000000002},"max":{"1min":1.0669999999999999,"5min":1.3500000000000001,"15min":1.4570000000000001},"last":0.84399999999999997}]},{"osd":4,"last update":"Mon May 22 15:42:07 2023","interfaces":[{"interface":"back","average":{"1min":1.0429999999999999,"5min":0.94899999999999995,"15min":0.94799999999999995},"min":{"1min":0.751,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":2.923,"5min":2.923,"15min":2.923},"last":0.94699999999999995},{"interface":"front","average":{"1min":1.073,"5min":1.0269999999999999,"15min":1.022},"min":{"1min":0.86399999999999999,"5min":0.64600000000000002,"15min":0.54900000000000004},"max":{"1min":1.8160000000000001,"5min":1.8160000000000001,"15min":1.8160000000000001},"last":1.1830000000000001}]},{"osd":5,"last update":"Mon May 22 15:42:07 2023","interfaces":[{"interface":"back","average":{"1min":0.94699999999999995,"5min":1.014,"15min":1.038},"min":{"1min":0.66500000000000004,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.1040000000000001,"5min":1.4750000000000001,"15min":1.6850000000000001},"last":0.90100000000000002},{"interface":"front","average":{"1min":1.2170000000000001,"5min":1.032,"15min":1.01},"min":{"1min":0.76800000000000002,"5min":0.55300000000000005,"15min":0.54800000000000004},"max":{"1min":5.0519999999999996,"5min":5.0519999999999996,"15min":5.0519999999999996},"last":1.0329999999999999}]},{"osd":6,"last update":"Mon May 22 15:42:07 2023","interfaces":[{"interface":"back","average":{"1min":1.038,"5min":1.0740000000000001,"15min":1.075},"min":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":1.53,"5min":1.651,"15min":1.7450000000000001},"last":1.097},{"interface":"front","average":{"1min":1.2749999999999999,"5min":1.139,"15min":1.103},"min":{"1min":0.91500000000000004,"5min":0.499,"15min":0.499},"max":{"1min":5.2640000000000002,"5min":5.2640000000000002,"15min":5.2640000000000002},"last":1.071}]},{"osd":7,"last update":"Mon May 22 15:42:07 2023","interfaces":[{"interface":"back","average":{"1min":1.0700000000000001,"5min":1.1120000000000001,"15min":1.1180000000000001},"min":{"1min":0.70699999999999996,"5min":0.65700000000000003,"15min":0.56499999999999995},"max":{"1min":1.2789999999999999,"5min":1.6419999999999999,"15min":1.6419999999999999},"last":0.96999999999999997},{"interface":"front","average":{"1min":1.228,"5min":1.1000000000000001,"15min":1.087},"min":{"1min":0.85999999999999999,"5min":0.60299999999999998,"15min":0.58399999999999996},"max":{"1min":3.9060000000000001,"5min":3.9060000000000001,"15min":3.9060000000000001},"last":1.1419999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2148,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-05-22T15:42:26.863 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-05-22T15:42:27.022 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.020+0000 7f85cf050640 1 Processor -- start 2023-05-22T15:42:27.023 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.020+0000 7f85cf050640 1 -- start start 2023-05-22T15:42:27.023 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.020+0000 7f85cf050640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f85c8071640 0x7f85c8160090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:27.023 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.020+0000 7f85cf050640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f85c81605d0 0x7f85c815c270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:27.024 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.020+0000 7f85cf050640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85c8069ae0 con 0x7f85c80ce480 2023-05-22T15:42:27.024 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.020+0000 7f85cf050640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85c815f990 con 0x7f85c8071640 2023-05-22T15:42:27.024 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.020+0000 7f85cf050640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85c8071ac0 con 0x7f85c81605d0 2023-05-22T15:42:27.025 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.020+0000 7f85c7fff640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f85c8071640 0x7f85c8160090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:27.025 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.020+0000 7f85c7fff640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f85c8071640 0x7f85c8160090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34880/0 (socket says 172.21.15.16:34880) 2023-05-22T15:42:27.025 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.020+0000 7f85c7fff640 1 -- 172.21.15.16:0/2566991392 learned_addr learned my addr 172.21.15.16:0/2566991392 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:42:27.026 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.020+0000 7f85cd5ed640 1 --2- 172.21.15.16:0/2566991392 >> v2:172.21.15.62:3300/0 conn(0x7f85c81605d0 0x7f85c815c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:27.026 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.020+0000 7f85c7fff640 1 -- 172.21.15.16:0/2566991392 >> v2:172.21.15.62:3300/0 conn(0x7f85c81605d0 msgr2=0x7f85c815c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:42:27.026 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.020+0000 7f85c7fff640 1 --2- 172.21.15.16:0/2566991392 >> v2:172.21.15.62:3300/0 conn(0x7f85c81605d0 0x7f85c815c270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:27.026 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.020+0000 7f85c7fff640 1 -- 172.21.15.16:0/2566991392 >> v1:172.21.15.62:6789/0 conn(0x7f85c80ce480 legacy=0x7f85c80ce850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:42:27.026 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.020+0000 7f85c7fff640 1 -- 172.21.15.16:0/2566991392 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85c8069dc0 con 0x7f85c8071640 2023-05-22T15:42:27.027 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.024+0000 7f85c7fff640 1 --2- 172.21.15.16:0/2566991392 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f85c8071640 0x7f85c8160090 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 crypto rx=0x7f85c80aeb30 tx=0x7f85b401c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee81d09e5f4b301a server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:27.028 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.024+0000 7f85c77fe640 1 -- 172.21.15.16:0/2566991392 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85b40aa430 con 0x7f85c8071640 2023-05-22T15:42:27.028 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.024+0000 7f85c77fe640 1 -- 172.21.15.16:0/2566991392 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85b40aa590 con 0x7f85c8071640 2023-05-22T15:42:27.028 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.024+0000 7f85c77fe640 1 -- 172.21.15.16:0/2566991392 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85b40aa920 con 0x7f85c8071640 2023-05-22T15:42:27.028 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.024+0000 7f85cf050640 1 -- 172.21.15.16:0/2566991392 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f85c8071640 msgr2=0x7f85c8160090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:27.028 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.024+0000 7f85cf050640 1 --2- 172.21.15.16:0/2566991392 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f85c8071640 0x7f85c8160090 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 crypto rx=0x7f85c80aeb30 tx=0x7f85b401c4d0 comp rx=0 tx=0).stop 2023-05-22T15:42:27.028 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.024+0000 7f85cf050640 1 -- 172.21.15.16:0/2566991392 shutdown_connections 2023-05-22T15:42:27.029 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.024+0000 7f85cf050640 1 --2- 172.21.15.16:0/2566991392 >> v2:172.21.15.62:3300/0 conn(0x7f85c81605d0 0x7f85c815c270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:27.029 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.024+0000 7f85cf050640 1 --2- 172.21.15.16:0/2566991392 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f85c8071640 0x7f85c8160090 unknown :-1 s=CLOSED pgs=1114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:27.029 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.024+0000 7f85cf050640 1 -- 172.21.15.16:0/2566991392 >> 172.21.15.16:0/2566991392 conn(0x7f85c80cc870 msgr2=0x7f85c80cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:27.029 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.024+0000 7f85cf050640 1 -- 172.21.15.16:0/2566991392 shutdown_connections 2023-05-22T15:42:27.029 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.024+0000 7f85cf050640 1 -- 172.21.15.16:0/2566991392 wait complete. 2023-05-22T15:42:27.031 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.024+0000 7f85cf050640 1 Processor -- start 2023-05-22T15:42:27.031 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.024+0000 7f85cf050640 1 -- start start 2023-05-22T15:42:27.031 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.024+0000 7f85cf050640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f85c8071640 0x7f85c80b0b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:27.031 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.024+0000 7f85cf050640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f85c80ce480 0x7f85c815ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:27.032 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.024+0000 7f85cf050640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f85c81605d0 0x7f85c814af00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:27.032 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.024+0000 7f85cf050640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85c814a110 con 0x7f85c80ce480 2023-05-22T15:42:27.032 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.024+0000 7f85cf050640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85c814aa70 con 0x7f85c81605d0 2023-05-22T15:42:27.032 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.024+0000 7f85cf050640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85c814bb30 con 0x7f85c8071640 2023-05-22T15:42:27.032 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.024+0000 7f85c7fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f85c80ce480 0x7f85c815ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:27.032 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.024+0000 7f85c7fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f85c80ce480 0x7f85c815ae10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34882/0 (socket says 172.21.15.16:34882) 2023-05-22T15:42:27.033 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.024+0000 7f85c7fff640 1 -- 172.21.15.16:0/2585366061 learned_addr learned my addr 172.21.15.16:0/2585366061 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:42:27.033 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.024+0000 7f85ccdec640 1 --2- 172.21.15.16:0/2585366061 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f85c8071640 0x7f85c80b0b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:27.033 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.024+0000 7f85cd5ed640 1 --2- 172.21.15.16:0/2585366061 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f85c81605d0 0x7f85c814af00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:27.033 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.028+0000 7f85c7fff640 1 -- 172.21.15.16:0/2585366061 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f85c8071640 msgr2=0x7f85c80b0b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:27.033 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.028+0000 7f85c7fff640 1 --2- 172.21.15.16:0/2585366061 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f85c8071640 0x7f85c80b0b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:27.033 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.028+0000 7f85c7fff640 1 -- 172.21.15.16:0/2585366061 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f85c81605d0 msgr2=0x7f85c814af00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:27.033 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.028+0000 7f85c7fff640 1 --2- 172.21.15.16:0/2585366061 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f85c81605d0 0x7f85c814af00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:27.034 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.028+0000 7f85c7fff640 1 -- 172.21.15.16:0/2585366061 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85b40094d0 con 0x7f85c80ce480 2023-05-22T15:42:27.034 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.028+0000 7f85cd5ed640 1 --2- 172.21.15.16:0/2585366061 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f85c81605d0 0x7f85c814af00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T15:42:27.034 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.028+0000 7f85c7fff640 1 --2- 172.21.15.16:0/2585366061 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f85c80ce480 0x7f85c815ae10 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 crypto rx=0x7f85b40190e0 tx=0x7f85b401ad10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:27.034 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.028+0000 7f85c57fa640 1 -- 172.21.15.16:0/2585366061 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85b40aa430 con 0x7f85c80ce480 2023-05-22T15:42:27.034 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.028+0000 7f85c57fa640 1 -- 172.21.15.16:0/2585366061 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85b40b3800 con 0x7f85c80ce480 2023-05-22T15:42:27.034 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.028+0000 7f85c57fa640 1 -- 172.21.15.16:0/2585366061 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85b40b3bb0 con 0x7f85c80ce480 2023-05-22T15:42:27.034 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.028+0000 7f85cf050640 1 -- 172.21.15.16:0/2585366061 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85c80698b0 con 0x7f85c80ce480 2023-05-22T15:42:27.035 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.028+0000 7f85cf050640 1 -- 172.21.15.16:0/2585366061 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85c8069720 con 0x7f85c80ce480 2023-05-22T15:42:27.035 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.028+0000 7f85cf050640 1 -- 172.21.15.16:0/2585366061 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85c80c5e70 con 0x7f85c80ce480 2023-05-22T15:42:27.036 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.028+0000 7f85c57fa640 1 -- 172.21.15.16:0/2585366061 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f85b40b3d10 con 0x7f85c80ce480 2023-05-22T15:42:27.036 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.028+0000 7f85c57fa640 1 --2- 172.21.15.16:0/2585366061 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f85ac0654a0 0x7f85ac067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:27.036 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.028+0000 7f85c57fa640 1 -- 172.21.15.16:0/2585366061 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f85b411d0b0 con 0x7f85c80ce480 2023-05-22T15:42:27.036 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.028+0000 7f85ccdec640 1 --2- 172.21.15.16:0/2585366061 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f85ac0654a0 0x7f85ac067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:27.036 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.028+0000 7f85ccdec640 1 --2- 172.21.15.16:0/2585366061 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f85ac0654a0 0x7f85ac067950 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f85bc000f70 tx=0x7f85bc009040 comp rx=0 tx=0).ready entity=mgr.6765 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:27.037 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.032+0000 7f85c57fa640 1 -- 172.21.15.16:0/2585366061 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f85b40eaa80 con 0x7f85c80ce480 2023-05-22T15:42:27.185 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.180+0000 7f85cf050640 1 -- 172.21.15.16:0/2585366061 --> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f85c804dbe0 con 0x7f85ac0654a0 2023-05-22T15:42:27.198 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.192+0000 7f85c57fa640 1 -- 172.21.15.16:0/2585366061 <== mgr.6765 v2:172.21.15.16:6832/36921 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+310044 (secure 0 0 0) 0x7f85c804dbe0 con 0x7f85ac0654a0 2023-05-22T15:42:27.199 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:42:27.203 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2023-05-22T15:42:27.204 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.200+0000 7f85cf050640 1 -- 172.21.15.16:0/2585366061 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f85ac0654a0 msgr2=0x7f85ac067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:27.204 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.200+0000 7f85cf050640 1 --2- 172.21.15.16:0/2585366061 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f85ac0654a0 0x7f85ac067950 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f85bc000f70 tx=0x7f85bc009040 comp rx=0 tx=0).stop 2023-05-22T15:42:27.204 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.200+0000 7f85cf050640 1 -- 172.21.15.16:0/2585366061 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f85c80ce480 msgr2=0x7f85c815ae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:27.205 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.200+0000 7f85cf050640 1 --2- 172.21.15.16:0/2585366061 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f85c80ce480 0x7f85c815ae10 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 crypto rx=0x7f85b40190e0 tx=0x7f85b401ad10 comp rx=0 tx=0).stop 2023-05-22T15:42:27.205 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.200+0000 7f85cf050640 1 -- 172.21.15.16:0/2585366061 shutdown_connections 2023-05-22T15:42:27.205 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.200+0000 7f85cf050640 1 --2- 172.21.15.16:0/2585366061 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f85c81605d0 0x7f85c814af00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:27.205 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.200+0000 7f85cf050640 1 --2- 172.21.15.16:0/2585366061 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f85c80ce480 0x7f85c815ae10 unknown :-1 s=CLOSED pgs=1115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:27.205 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.200+0000 7f85cf050640 1 --2- 172.21.15.16:0/2585366061 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f85ac0654a0 0x7f85ac067950 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:27.206 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.200+0000 7f85cf050640 1 --2- 172.21.15.16:0/2585366061 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f85c8071640 0x7f85c80b0b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:27.206 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.200+0000 7f85cf050640 1 -- 172.21.15.16:0/2585366061 >> 172.21.15.16:0/2585366061 conn(0x7f85c80cc870 msgr2=0x7f85c815d5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:27.206 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.200+0000 7f85cf050640 1 -- 172.21.15.16:0/2585366061 shutdown_connections 2023-05-22T15:42:27.206 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.200+0000 7f85cf050640 1 -- 172.21.15.16:0/2585366061 wait complete. 2023-05-22T15:42:27.218 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":26,"stamp":"2023-05-22T15:42:27.054967+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592698,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":500,"num_read_kb":924,"num_write":283,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":199,"ondisk_log_size":199,"up":210,"acting":210,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":35,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":206756,"kb_used_data":9508,"kb_used_omap":0,"kb_used_meta":197184,"kb_avail":754767964,"statfs":{"total":773094113280,"available":772882395136,"internally_reserved":0,"allocated":9736192,"data_stored":5627780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":201916416},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":71,"apply_latency_ms":71,"commit_latency_ns":71000000,"apply_latency_ns":71000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2330,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.628886"},"pg_stats":[{"pgid":"11.3f","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.904859+0000","last_change":"2023-05-22T15:42:22.894158+0000","last_active":"2023-05-22T15:42:22.904859+0000","last_peered":"2023-05-22T15:42:22.904859+0000","last_clean":"2023-05-22T15:42:22.904859+0000","last_became_active":"2023-05-22T15:42:22.893003+0000","last_became_peered":"2023-05-22T15:42:22.893003+0000","last_unstale":"2023-05-22T15:42:22.904859+0000","last_undegraded":"2023-05-22T15:42:22.904859+0000","last_fullsized":"2023-05-22T15:42:22.904859+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_clean_scrub_stamp":"2023-05-22T15:42:22.424085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T16:20:50.290808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.3e","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.897274+0000","last_change":"2023-05-22T15:42:22.885226+0000","last_active":"2023-05-22T15:42:22.897274+0000","last_peered":"2023-05-22T15:42:22.897274+0000","last_clean":"2023-05-22T15:42:22.897274+0000","last_became_active":"2023-05-22T15:42:22.882088+0000","last_became_peered":"2023-05-22T15:42:22.882088+0000","last_unstale":"2023-05-22T15:42:22.897274+0000","last_undegraded":"2023-05-22T15:42:22.897274+0000","last_fullsized":"2023-05-22T15:42:22.897274+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_clean_scrub_stamp":"2023-05-22T15:42:22.424085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T15:49:20.912521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.3d","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.840258+0000","last_change":"2023-05-22T15:42:22.838847+0000","last_active":"2023-05-22T15:42:22.840258+0000","last_peered":"2023-05-22T15:42:22.840258+0000","last_clean":"2023-05-22T15:42:22.840258+0000","last_became_active":"2023-05-22T15:42:22.838548+0000","last_became_peered":"2023-05-22T15:42:22.838548+0000","last_unstale":"2023-05-22T15:42:22.840258+0000","last_undegraded":"2023-05-22T15:42:22.840258+0000","last_fullsized":"2023-05-22T15:42:22.840258+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_clean_scrub_stamp":"2023-05-22T15:42:22.424085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:26:08.824903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.3c","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.879554+0000","last_change":"2023-05-22T15:42:22.878514+0000","last_active":"2023-05-22T15:42:22.879554+0000","last_peered":"2023-05-22T15:42:22.879554+0000","last_clean":"2023-05-22T15:42:22.879554+0000","last_became_active":"2023-05-22T15:42:22.877705+0000","last_became_peered":"2023-05-22T15:42:22.877705+0000","last_unstale":"2023-05-22T15:42:22.879554+0000","last_undegraded":"2023-05-22T15:42:22.879554+0000","last_fullsized":"2023-05-22T15:42:22.879554+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_clean_scrub_stamp":"2023-05-22T15:42:22.424085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:51:29.460959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.3b","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.891490+0000","last_change":"2023-05-22T15:42:22.890668+0000","last_active":"2023-05-22T15:42:22.891490+0000","last_peered":"2023-05-22T15:42:22.891490+0000","last_clean":"2023-05-22T15:42:22.891490+0000","last_became_active":"2023-05-22T15:42:22.890359+0000","last_became_peered":"2023-05-22T15:42:22.890359+0000","last_unstale":"2023-05-22T15:42:22.891490+0000","last_undegraded":"2023-05-22T15:42:22.891490+0000","last_fullsized":"2023-05-22T15:42:22.891490+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_clean_scrub_stamp":"2023-05-22T15:42:22.424085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:24:27.275366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3a","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.897816+0000","last_change":"2023-05-22T15:42:22.882144+0000","last_active":"2023-05-22T15:42:22.897816+0000","last_peered":"2023-05-22T15:42:22.897816+0000","last_clean":"2023-05-22T15:42:22.897816+0000","last_became_active":"2023-05-22T15:42:22.880797+0000","last_became_peered":"2023-05-22T15:42:22.880797+0000","last_unstale":"2023-05-22T15:42:22.897816+0000","last_undegraded":"2023-05-22T15:42:22.897816+0000","last_fullsized":"2023-05-22T15:42:22.897816+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_clean_scrub_stamp":"2023-05-22T15:42:22.424085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:51:29.460959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.39","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.870224+0000","last_change":"2023-05-22T15:42:22.848473+0000","last_active":"2023-05-22T15:42:22.870224+0000","last_peered":"2023-05-22T15:42:22.870224+0000","last_clean":"2023-05-22T15:42:22.870224+0000","last_became_active":"2023-05-22T15:42:22.847775+0000","last_became_peered":"2023-05-22T15:42:22.847775+0000","last_unstale":"2023-05-22T15:42:22.870224+0000","last_undegraded":"2023-05-22T15:42:22.870224+0000","last_fullsized":"2023-05-22T15:42:22.870224+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_clean_scrub_stamp":"2023-05-22T15:42:22.424085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:45:22.277730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.38","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.896379+0000","last_change":"2023-05-22T15:42:22.882560+0000","last_active":"2023-05-22T15:42:22.896379+0000","last_peered":"2023-05-22T15:42:22.896379+0000","last_clean":"2023-05-22T15:42:22.896379+0000","last_became_active":"2023-05-22T15:42:22.877494+0000","last_became_peered":"2023-05-22T15:42:22.877494+0000","last_unstale":"2023-05-22T15:42:22.896379+0000","last_undegraded":"2023-05-22T15:42:22.896379+0000","last_fullsized":"2023-05-22T15:42:22.896379+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_clean_scrub_stamp":"2023-05-22T15:42:22.424085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T03:03:57.842619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.37","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.840713+0000","last_change":"2023-05-22T15:42:22.837811+0000","last_active":"2023-05-22T15:42:22.840713+0000","last_peered":"2023-05-22T15:42:22.840713+0000","last_clean":"2023-05-22T15:42:22.840713+0000","last_became_active":"2023-05-22T15:42:22.837518+0000","last_became_peered":"2023-05-22T15:42:22.837518+0000","last_unstale":"2023-05-22T15:42:22.840713+0000","last_undegraded":"2023-05-22T15:42:22.840713+0000","last_fullsized":"2023-05-22T15:42:22.840713+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_clean_scrub_stamp":"2023-05-22T15:42:22.424085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:32:58.984742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.36","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.886652+0000","last_change":"2023-05-22T15:42:22.871344+0000","last_active":"2023-05-22T15:42:22.886652+0000","last_peered":"2023-05-22T15:42:22.886652+0000","last_clean":"2023-05-22T15:42:22.886652+0000","last_became_active":"2023-05-22T15:42:22.870582+0000","last_became_peered":"2023-05-22T15:42:22.870582+0000","last_unstale":"2023-05-22T15:42:22.886652+0000","last_undegraded":"2023-05-22T15:42:22.886652+0000","last_fullsized":"2023-05-22T15:42:22.886652+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_clean_scrub_stamp":"2023-05-22T15:42:22.424085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:12:13.799667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.35","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.904737+0000","last_change":"2023-05-22T15:42:22.892644+0000","last_active":"2023-05-22T15:42:22.904737+0000","last_peered":"2023-05-22T15:42:22.904737+0000","last_clean":"2023-05-22T15:42:22.904737+0000","last_became_active":"2023-05-22T15:42:22.892222+0000","last_became_peered":"2023-05-22T15:42:22.892222+0000","last_unstale":"2023-05-22T15:42:22.904737+0000","last_undegraded":"2023-05-22T15:42:22.904737+0000","last_fullsized":"2023-05-22T15:42:22.904737+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_clean_scrub_stamp":"2023-05-22T15:42:22.424085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:17:04.192121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.34","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.833620+0000","last_change":"2023-05-22T15:42:22.829981+0000","last_active":"2023-05-22T15:42:22.833620+0000","last_peered":"2023-05-22T15:42:22.833620+0000","last_clean":"2023-05-22T15:42:22.833620+0000","last_became_active":"2023-05-22T15:42:22.829718+0000","last_became_peered":"2023-05-22T15:42:22.829718+0000","last_unstale":"2023-05-22T15:42:22.833620+0000","last_undegraded":"2023-05-22T15:42:22.833620+0000","last_fullsized":"2023-05-22T15:42:22.833620+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_clean_scrub_stamp":"2023-05-22T15:42:22.424085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T15:49:20.912521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.33","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.882882+0000","last_change":"2023-05-22T15:42:22.882224+0000","last_active":"2023-05-22T15:42:22.882882+0000","last_peered":"2023-05-22T15:42:22.882882+0000","last_clean":"2023-05-22T15:42:22.882882+0000","last_became_active":"2023-05-22T15:42:22.882024+0000","last_became_peered":"2023-05-22T15:42:22.882024+0000","last_unstale":"2023-05-22T15:42:22.882882+0000","last_undegraded":"2023-05-22T15:42:22.882882+0000","last_fullsized":"2023-05-22T15:42:22.882882+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_clean_scrub_stamp":"2023-05-22T15:42:22.424085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:11:47.018486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.32","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.866705+0000","last_change":"2023-05-22T15:42:22.865607+0000","last_active":"2023-05-22T15:42:22.866705+0000","last_peered":"2023-05-22T15:42:22.866705+0000","last_clean":"2023-05-22T15:42:22.866705+0000","last_became_active":"2023-05-22T15:42:22.865328+0000","last_became_peered":"2023-05-22T15:42:22.865328+0000","last_unstale":"2023-05-22T15:42:22.866705+0000","last_undegraded":"2023-05-22T15:42:22.866705+0000","last_fullsized":"2023-05-22T15:42:22.866705+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_clean_scrub_stamp":"2023-05-22T15:42:22.424085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:41:51.956588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.31","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.838194+0000","last_change":"2023-05-22T15:42:22.829796+0000","last_active":"2023-05-22T15:42:22.838194+0000","last_peered":"2023-05-22T15:42:22.838194+0000","last_clean":"2023-05-22T15:42:22.838194+0000","last_became_active":"2023-05-22T15:42:22.829501+0000","last_became_peered":"2023-05-22T15:42:22.829501+0000","last_unstale":"2023-05-22T15:42:22.838194+0000","last_undegraded":"2023-05-22T15:42:22.838194+0000","last_fullsized":"2023-05-22T15:42:22.838194+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_clean_scrub_stamp":"2023-05-22T15:42:22.424085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:43:03.995839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.30","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.888377+0000","last_change":"2023-05-22T15:42:22.880152+0000","last_active":"2023-05-22T15:42:22.888377+0000","last_peered":"2023-05-22T15:42:22.888377+0000","last_clean":"2023-05-22T15:42:22.888377+0000","last_became_active":"2023-05-22T15:42:22.879339+0000","last_became_peered":"2023-05-22T15:42:22.879339+0000","last_unstale":"2023-05-22T15:42:22.888377+0000","last_undegraded":"2023-05-22T15:42:22.888377+0000","last_fullsized":"2023-05-22T15:42:22.888377+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_clean_scrub_stamp":"2023-05-22T15:42:22.424085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:05:27.121829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2f","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.840728+0000","last_change":"2023-05-22T15:42:22.838081+0000","last_active":"2023-05-22T15:42:22.840728+0000","last_peered":"2023-05-22T15:42:22.840728+0000","last_clean":"2023-05-22T15:42:22.840728+0000","last_became_active":"2023-05-22T15:42:22.837802+0000","last_became_peered":"2023-05-22T15:42:22.837802+0000","last_unstale":"2023-05-22T15:42:22.840728+0000","last_undegraded":"2023-05-22T15:42:22.840728+0000","last_fullsized":"2023-05-22T15:42:22.840728+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_clean_scrub_stamp":"2023-05-22T15:42:22.424085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:56:40.685104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.2e","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.879546+0000","last_change":"2023-05-22T15:42:22.878055+0000","last_active":"2023-05-22T15:42:22.879546+0000","last_peered":"2023-05-22T15:42:22.879546+0000","last_clean":"2023-05-22T15:42:22.879546+0000","last_became_active":"2023-05-22T15:42:22.877710+0000","last_became_peered":"2023-05-22T15:42:22.877710+0000","last_unstale":"2023-05-22T15:42:22.879546+0000","last_undegraded":"2023-05-22T15:42:22.879546+0000","last_fullsized":"2023-05-22T15:42:22.879546+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_clean_scrub_stamp":"2023-05-22T15:42:22.424085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:32:58.984742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.2d","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.888178+0000","last_change":"2023-05-22T15:42:22.881271+0000","last_active":"2023-05-22T15:42:22.888178+0000","last_peered":"2023-05-22T15:42:22.888178+0000","last_clean":"2023-05-22T15:42:22.888178+0000","last_became_active":"2023-05-22T15:42:22.878300+0000","last_became_peered":"2023-05-22T15:42:22.878300+0000","last_unstale":"2023-05-22T15:42:22.888178+0000","last_undegraded":"2023-05-22T15:42:22.888178+0000","last_fullsized":"2023-05-22T15:42:22.888178+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_clean_scrub_stamp":"2023-05-22T15:42:22.424085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:02:52.959763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2c","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.853781+0000","last_change":"2023-05-22T15:42:22.852710+0000","last_active":"2023-05-22T15:42:22.853781+0000","last_peered":"2023-05-22T15:42:22.853781+0000","last_clean":"2023-05-22T15:42:22.853781+0000","last_became_active":"2023-05-22T15:42:22.852289+0000","last_became_peered":"2023-05-22T15:42:22.852289+0000","last_unstale":"2023-05-22T15:42:22.853781+0000","last_undegraded":"2023-05-22T15:42:22.853781+0000","last_fullsized":"2023-05-22T15:42:22.853781+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_clean_scrub_stamp":"2023-05-22T15:42:22.424085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:25:08.886473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2b","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.886266+0000","last_change":"2023-05-22T15:42:22.869779+0000","last_active":"2023-05-22T15:42:22.886266+0000","last_peered":"2023-05-22T15:42:22.886266+0000","last_clean":"2023-05-22T15:42:22.886266+0000","last_became_active":"2023-05-22T15:42:22.869063+0000","last_became_peered":"2023-05-22T15:42:22.869063+0000","last_unstale":"2023-05-22T15:42:22.886266+0000","last_undegraded":"2023-05-22T15:42:22.886266+0000","last_fullsized":"2023-05-22T15:42:22.886266+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_clean_scrub_stamp":"2023-05-22T15:42:22.424085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:02:51.734869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2a","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.877954+0000","last_change":"2023-05-22T15:42:22.851172+0000","last_active":"2023-05-22T15:42:22.877954+0000","last_peered":"2023-05-22T15:42:22.877954+0000","last_clean":"2023-05-22T15:42:22.877954+0000","last_became_active":"2023-05-22T15:42:22.850434+0000","last_became_peered":"2023-05-22T15:42:22.850434+0000","last_unstale":"2023-05-22T15:42:22.877954+0000","last_undegraded":"2023-05-22T15:42:22.877954+0000","last_fullsized":"2023-05-22T15:42:22.877954+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_clean_scrub_stamp":"2023-05-22T15:42:22.424085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:57:49.944485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.29","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.904730+0000","last_change":"2023-05-22T15:42:22.892552+0000","last_active":"2023-05-22T15:42:22.904730+0000","last_peered":"2023-05-22T15:42:22.904730+0000","last_clean":"2023-05-22T15:42:22.904730+0000","last_became_active":"2023-05-22T15:42:22.891506+0000","last_became_peered":"2023-05-22T15:42:22.891506+0000","last_unstale":"2023-05-22T15:42:22.904730+0000","last_undegraded":"2023-05-22T15:42:22.904730+0000","last_fullsized":"2023-05-22T15:42:22.904730+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_clean_scrub_stamp":"2023-05-22T15:42:22.424085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:45:22.277730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.28","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.904602+0000","last_change":"2023-05-22T15:42:22.890427+0000","last_active":"2023-05-22T15:42:22.904602+0000","last_peered":"2023-05-22T15:42:22.904602+0000","last_clean":"2023-05-22T15:42:22.904602+0000","last_became_active":"2023-05-22T15:42:22.889859+0000","last_became_peered":"2023-05-22T15:42:22.889859+0000","last_unstale":"2023-05-22T15:42:22.904602+0000","last_undegraded":"2023-05-22T15:42:22.904602+0000","last_fullsized":"2023-05-22T15:42:22.904602+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_clean_scrub_stamp":"2023-05-22T15:42:22.424085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:56:03.682465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.27","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.898864+0000","last_change":"2023-05-22T15:42:22.895817+0000","last_active":"2023-05-22T15:42:22.898864+0000","last_peered":"2023-05-22T15:42:22.898864+0000","last_clean":"2023-05-22T15:42:22.898864+0000","last_became_active":"2023-05-22T15:42:22.895387+0000","last_became_peered":"2023-05-22T15:42:22.895387+0000","last_unstale":"2023-05-22T15:42:22.898864+0000","last_undegraded":"2023-05-22T15:42:22.898864+0000","last_fullsized":"2023-05-22T15:42:22.898864+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_clean_scrub_stamp":"2023-05-22T15:42:22.424085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:58:20.411983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.26","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.884315+0000","last_change":"2023-05-22T15:42:22.881297+0000","last_active":"2023-05-22T15:42:22.884315+0000","last_peered":"2023-05-22T15:42:22.884315+0000","last_clean":"2023-05-22T15:42:22.884315+0000","last_became_active":"2023-05-22T15:42:22.880447+0000","last_became_peered":"2023-05-22T15:42:22.880447+0000","last_unstale":"2023-05-22T15:42:22.884315+0000","last_undegraded":"2023-05-22T15:42:22.884315+0000","last_fullsized":"2023-05-22T15:42:22.884315+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_clean_scrub_stamp":"2023-05-22T15:42:22.424085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T18:23:22.270527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.25","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.895279+0000","last_change":"2023-05-22T15:42:22.894745+0000","last_active":"2023-05-22T15:42:22.895279+0000","last_peered":"2023-05-22T15:42:22.895279+0000","last_clean":"2023-05-22T15:42:22.895279+0000","last_became_active":"2023-05-22T15:42:22.894459+0000","last_became_peered":"2023-05-22T15:42:22.894459+0000","last_unstale":"2023-05-22T15:42:22.895279+0000","last_undegraded":"2023-05-22T15:42:22.895279+0000","last_fullsized":"2023-05-22T15:42:22.895279+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_clean_scrub_stamp":"2023-05-22T15:42:22.424085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:26:20.201566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.24","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.878026+0000","last_change":"2023-05-22T15:42:22.851280+0000","last_active":"2023-05-22T15:42:22.878026+0000","last_peered":"2023-05-22T15:42:22.878026+0000","last_clean":"2023-05-22T15:42:22.878026+0000","last_became_active":"2023-05-22T15:42:22.850543+0000","last_became_peered":"2023-05-22T15:42:22.850543+0000","last_unstale":"2023-05-22T15:42:22.878026+0000","last_undegraded":"2023-05-22T15:42:22.878026+0000","last_fullsized":"2023-05-22T15:42:22.878026+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_clean_scrub_stamp":"2023-05-22T15:42:22.424085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:04:44.765497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.23","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.864167+0000","last_change":"2023-05-22T15:42:22.863431+0000","last_active":"2023-05-22T15:42:22.864167+0000","last_peered":"2023-05-22T15:42:22.864167+0000","last_clean":"2023-05-22T15:42:22.864167+0000","last_became_active":"2023-05-22T15:42:22.863095+0000","last_became_peered":"2023-05-22T15:42:22.863095+0000","last_unstale":"2023-05-22T15:42:22.864167+0000","last_undegraded":"2023-05-22T15:42:22.864167+0000","last_fullsized":"2023-05-22T15:42:22.864167+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_clean_scrub_stamp":"2023-05-22T15:42:22.424085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:02:39.112412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.22","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.884132+0000","last_change":"2023-05-22T15:42:22.880416+0000","last_active":"2023-05-22T15:42:22.884132+0000","last_peered":"2023-05-22T15:42:22.884132+0000","last_clean":"2023-05-22T15:42:22.884132+0000","last_became_active":"2023-05-22T15:42:22.879633+0000","last_became_peered":"2023-05-22T15:42:22.879633+0000","last_unstale":"2023-05-22T15:42:22.884132+0000","last_undegraded":"2023-05-22T15:42:22.884132+0000","last_fullsized":"2023-05-22T15:42:22.884132+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_clean_scrub_stamp":"2023-05-22T15:42:22.424085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:58:20.411983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.21","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.865455+0000","last_change":"2023-05-22T15:42:22.847185+0000","last_active":"2023-05-22T15:42:22.865455+0000","last_peered":"2023-05-22T15:42:22.865455+0000","last_clean":"2023-05-22T15:42:22.865455+0000","last_became_active":"2023-05-22T15:42:22.846459+0000","last_became_peered":"2023-05-22T15:42:22.846459+0000","last_unstale":"2023-05-22T15:42:22.865455+0000","last_undegraded":"2023-05-22T15:42:22.865455+0000","last_fullsized":"2023-05-22T15:42:22.865455+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_clean_scrub_stamp":"2023-05-22T15:42:22.424085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:17:04.192121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.20","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.887948+0000","last_change":"2023-05-22T15:42:22.877407+0000","last_active":"2023-05-22T15:42:22.887948+0000","last_peered":"2023-05-22T15:42:22.887948+0000","last_clean":"2023-05-22T15:42:22.887948+0000","last_became_active":"2023-05-22T15:42:22.876698+0000","last_became_peered":"2023-05-22T15:42:22.876698+0000","last_unstale":"2023-05-22T15:42:22.887948+0000","last_undegraded":"2023-05-22T15:42:22.887948+0000","last_fullsized":"2023-05-22T15:42:22.887948+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_clean_scrub_stamp":"2023-05-22T15:42:22.424085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:41:51.956588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.904795+0000","last_change":"2023-05-22T15:42:22.893202+0000","last_active":"2023-05-22T15:42:22.904795+0000","last_peered":"2023-05-22T15:42:22.904795+0000","last_clean":"2023-05-22T15:42:22.904795+0000","last_became_active":"2023-05-22T15:42:22.891232+0000","last_became_peered":"2023-05-22T15:42:22.891232+0000","last_unstale":"2023-05-22T15:42:22.904795+0000","last_undegraded":"2023-05-22T15:42:22.904795+0000","last_fullsized":"2023-05-22T15:42:22.904795+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_clean_scrub_stamp":"2023-05-22T15:42:22.424085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:05:27.121829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1f","version":"89'4","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.892726+0000","last_change":"2023-05-22T15:42:22.481619+0000","last_active":"2023-05-22T15:42:22.892726+0000","last_peered":"2023-05-22T15:42:22.892726+0000","last_clean":"2023-05-22T15:42:22.892726+0000","last_became_active":"2023-05-22T15:42:22.481095+0000","last_became_peered":"2023-05-22T15:42:22.481095+0000","last_unstale":"2023-05-22T15:42:22.892726+0000","last_undegraded":"2023-05-22T15:42:22.892726+0000","last_fullsized":"2023-05-22T15:42:22.892726+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:21.420311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:21.420311+0000","last_clean_scrub_stamp":"2023-05-22T15:42:21.420311+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:11:08.656000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":566,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.1f","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.840422+0000","last_change":"2023-05-22T15:42:22.835146+0000","last_active":"2023-05-22T15:42:22.840422+0000","last_peered":"2023-05-22T15:42:22.840422+0000","last_clean":"2023-05-22T15:42:22.840422+0000","last_became_active":"2023-05-22T15:42:22.834823+0000","last_became_peered":"2023-05-22T15:42:22.834823+0000","last_unstale":"2023-05-22T15:42:22.840422+0000","last_undegraded":"2023-05-22T15:42:22.840422+0000","last_fullsized":"2023-05-22T15:42:22.840422+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_clean_scrub_stamp":"2023-05-22T15:42:22.424085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T15:49:20.912521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.1e","version":"89'1","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.837175+0000","last_change":"2023-05-22T15:42:22.460364+0000","last_active":"2023-05-22T15:42:22.837175+0000","last_peered":"2023-05-22T15:42:22.837175+0000","last_clean":"2023-05-22T15:42:22.837175+0000","last_became_active":"2023-05-22T15:42:22.460043+0000","last_became_peered":"2023-05-22T15:42:22.460043+0000","last_unstale":"2023-05-22T15:42:22.837175+0000","last_undegraded":"2023-05-22T15:42:22.837175+0000","last_fullsized":"2023-05-22T15:42:22.837175+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:21.420311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:21.420311+0000","last_clean_scrub_stamp":"2023-05-22T15:42:21.420311+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:58:19.408209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.902206+0000","last_change":"2023-05-22T15:42:22.898485+0000","last_active":"2023-05-22T15:42:22.902206+0000","last_peered":"2023-05-22T15:42:22.902206+0000","last_clean":"2023-05-22T15:42:22.902206+0000","last_became_active":"2023-05-22T15:42:22.897703+0000","last_became_peered":"2023-05-22T15:42:22.897703+0000","last_unstale":"2023-05-22T15:42:22.902206+0000","last_undegraded":"2023-05-22T15:42:22.902206+0000","last_fullsized":"2023-05-22T15:42:22.902206+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_clean_scrub_stamp":"2023-05-22T15:42:22.424085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:56:40.685104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1d","version":"89'2","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.876656+0000","last_change":"2023-05-22T15:42:22.487361+0000","last_active":"2023-05-22T15:42:22.876656+0000","last_peered":"2023-05-22T15:42:22.876656+0000","last_clean":"2023-05-22T15:42:22.876656+0000","last_became_active":"2023-05-22T15:42:22.486708+0000","last_became_peered":"2023-05-22T15:42:22.486708+0000","last_unstale":"2023-05-22T15:42:22.876656+0000","last_undegraded":"2023-05-22T15:42:22.876656+0000","last_fullsized":"2023-05-22T15:42:22.876656+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:21.420311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:21.420311+0000","last_clean_scrub_stamp":"2023-05-22T15:42:21.420311+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:25:37.626039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.905012+0000","last_change":"2023-05-22T15:42:22.894898+0000","last_active":"2023-05-22T15:42:22.905012+0000","last_peered":"2023-05-22T15:42:22.905012+0000","last_clean":"2023-05-22T15:42:22.905012+0000","last_became_active":"2023-05-22T15:42:22.894265+0000","last_became_peered":"2023-05-22T15:42:22.894265+0000","last_unstale":"2023-05-22T15:42:22.905012+0000","last_undegraded":"2023-05-22T15:42:22.905012+0000","last_fullsized":"2023-05-22T15:42:22.905012+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_clean_scrub_stamp":"2023-05-22T15:42:22.424085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:02:51.734869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1c","version":"89'2","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.878480+0000","last_change":"2023-05-22T15:42:22.470932+0000","last_active":"2023-05-22T15:42:22.878480+0000","last_peered":"2023-05-22T15:42:22.878480+0000","last_clean":"2023-05-22T15:42:22.878480+0000","last_became_active":"2023-05-22T15:42:22.470204+0000","last_became_peered":"2023-05-22T15:42:22.470204+0000","last_unstale":"2023-05-22T15:42:22.878480+0000","last_undegraded":"2023-05-22T15:42:22.878480+0000","last_fullsized":"2023-05-22T15:42:22.878480+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:21.420311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:21.420311+0000","last_clean_scrub_stamp":"2023-05-22T15:42:21.420311+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T03:02:20.624682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.875161+0000","last_change":"2023-05-22T15:42:22.865169+0000","last_active":"2023-05-22T15:42:22.875161+0000","last_peered":"2023-05-22T15:42:22.875161+0000","last_clean":"2023-05-22T15:42:22.875161+0000","last_became_active":"2023-05-22T15:42:22.864225+0000","last_became_peered":"2023-05-22T15:42:22.864225+0000","last_unstale":"2023-05-22T15:42:22.875161+0000","last_undegraded":"2023-05-22T15:42:22.875161+0000","last_fullsized":"2023-05-22T15:42:22.875161+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_clean_scrub_stamp":"2023-05-22T15:42:22.424085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T03:21:46.783251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.1b","version":"89'1","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.869943+0000","last_change":"2023-05-22T15:42:22.487840+0000","last_active":"2023-05-22T15:42:22.869943+0000","last_peered":"2023-05-22T15:42:22.869943+0000","last_clean":"2023-05-22T15:42:22.869943+0000","last_became_active":"2023-05-22T15:42:22.487502+0000","last_became_peered":"2023-05-22T15:42:22.487502+0000","last_unstale":"2023-05-22T15:42:22.869943+0000","last_undegraded":"2023-05-22T15:42:22.869943+0000","last_fullsized":"2023-05-22T15:42:22.869943+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:21.420311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:21.420311+0000","last_clean_scrub_stamp":"2023-05-22T15:42:21.420311+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:56:02.678692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.896968+0000","last_change":"2023-05-22T15:42:22.893248+0000","last_active":"2023-05-22T15:42:22.896968+0000","last_peered":"2023-05-22T15:42:22.896968+0000","last_clean":"2023-05-22T15:42:22.896968+0000","last_became_active":"2023-05-22T15:42:22.892636+0000","last_became_peered":"2023-05-22T15:42:22.892636+0000","last_unstale":"2023-05-22T15:42:22.896968+0000","last_undegraded":"2023-05-22T15:42:22.896968+0000","last_fullsized":"2023-05-22T15:42:22.896968+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_clean_scrub_stamp":"2023-05-22T15:42:22.424085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T18:23:22.270527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1a","version":"89'1","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.878187+0000","last_change":"2023-05-22T15:42:22.475557+0000","last_active":"2023-05-22T15:42:22.878187+0000","last_peered":"2023-05-22T15:42:22.878187+0000","last_clean":"2023-05-22T15:42:22.878187+0000","last_became_active":"2023-05-22T15:42:22.471570+0000","last_became_peered":"2023-05-22T15:42:22.471570+0000","last_unstale":"2023-05-22T15:42:22.878187+0000","last_undegraded":"2023-05-22T15:42:22.878187+0000","last_fullsized":"2023-05-22T15:42:22.878187+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:21.420311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:21.420311+0000","last_clean_scrub_stamp":"2023-05-22T15:42:21.420311+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:40:42.401972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.869934+0000","last_change":"2023-05-22T15:42:22.868913+0000","last_active":"2023-05-22T15:42:22.869934+0000","last_peered":"2023-05-22T15:42:22.869934+0000","last_clean":"2023-05-22T15:42:22.869934+0000","last_became_active":"2023-05-22T15:42:22.868590+0000","last_became_peered":"2023-05-22T15:42:22.868590+0000","last_unstale":"2023-05-22T15:42:22.869934+0000","last_undegraded":"2023-05-22T15:42:22.869934+0000","last_fullsized":"2023-05-22T15:42:22.869934+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_clean_scrub_stamp":"2023-05-22T15:42:22.424085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:51:29.460959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.822355+0000","last_change":"2023-05-22T15:42:22.488555+0000","last_active":"2023-05-22T15:42:22.822355+0000","last_peered":"2023-05-22T15:42:22.822355+0000","last_clean":"2023-05-22T15:42:22.822355+0000","last_became_active":"2023-05-22T15:42:22.488238+0000","last_became_peered":"2023-05-22T15:42:22.488238+0000","last_unstale":"2023-05-22T15:42:22.822355+0000","last_undegraded":"2023-05-22T15:42:22.822355+0000","last_fullsized":"2023-05-22T15:42:22.822355+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:21.420311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:21.420311+0000","last_clean_scrub_stamp":"2023-05-22T15:42:21.420311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:37:31.027858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.890682+0000","last_change":"2023-05-22T15:42:22.889892+0000","last_active":"2023-05-22T15:42:22.890682+0000","last_peered":"2023-05-22T15:42:22.890682+0000","last_clean":"2023-05-22T15:42:22.890682+0000","last_became_active":"2023-05-22T15:42:22.889596+0000","last_became_peered":"2023-05-22T15:42:22.889596+0000","last_unstale":"2023-05-22T15:42:22.890682+0000","last_undegraded":"2023-05-22T15:42:22.890682+0000","last_fullsized":"2023-05-22T15:42:22.890682+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_clean_scrub_stamp":"2023-05-22T15:42:22.424085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:58:08.771792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.820055+0000","last_change":"2023-05-22T15:42:22.457062+0000","last_active":"2023-05-22T15:42:22.820055+0000","last_peered":"2023-05-22T15:42:22.820055+0000","last_clean":"2023-05-22T15:42:22.820055+0000","last_became_active":"2023-05-22T15:42:22.456726+0000","last_became_peered":"2023-05-22T15:42:22.456726+0000","last_unstale":"2023-05-22T15:42:22.820055+0000","last_undegraded":"2023-05-22T15:42:22.820055+0000","last_fullsized":"2023-05-22T15:42:22.820055+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:21.420311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:21.420311+0000","last_clean_scrub_stamp":"2023-05-22T15:42:21.420311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:40:42.401972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.896786+0000","last_change":"2023-05-22T15:42:22.891623+0000","last_active":"2023-05-22T15:42:22.896786+0000","last_peered":"2023-05-22T15:42:22.896786+0000","last_clean":"2023-05-22T15:42:22.896786+0000","last_became_active":"2023-05-22T15:42:22.891076+0000","last_became_peered":"2023-05-22T15:42:22.891076+0000","last_unstale":"2023-05-22T15:42:22.896786+0000","last_undegraded":"2023-05-22T15:42:22.896786+0000","last_fullsized":"2023-05-22T15:42:22.896786+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_clean_scrub_stamp":"2023-05-22T15:42:22.424085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:02:51.734869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.832515+0000","last_change":"2023-05-22T15:42:22.489229+0000","last_active":"2023-05-22T15:42:22.832515+0000","last_peered":"2023-05-22T15:42:22.832515+0000","last_clean":"2023-05-22T15:42:22.832515+0000","last_became_active":"2023-05-22T15:42:22.488128+0000","last_became_peered":"2023-05-22T15:42:22.488128+0000","last_unstale":"2023-05-22T15:42:22.832515+0000","last_undegraded":"2023-05-22T15:42:22.832515+0000","last_fullsized":"2023-05-22T15:42:22.832515+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:21.420311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:21.420311+0000","last_clean_scrub_stamp":"2023-05-22T15:42:21.420311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:11:08.656000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.904880+0000","last_change":"2023-05-22T15:42:22.894454+0000","last_active":"2023-05-22T15:42:22.904880+0000","last_peered":"2023-05-22T15:42:22.904880+0000","last_clean":"2023-05-22T15:42:22.904880+0000","last_became_active":"2023-05-22T15:42:22.893741+0000","last_became_peered":"2023-05-22T15:42:22.893741+0000","last_unstale":"2023-05-22T15:42:22.904880+0000","last_undegraded":"2023-05-22T15:42:22.904880+0000","last_fullsized":"2023-05-22T15:42:22.904880+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_clean_scrub_stamp":"2023-05-22T15:42:22.424085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:11:43.253954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.855433+0000","last_change":"2023-05-22T15:42:22.483656+0000","last_active":"2023-05-22T15:42:22.855433+0000","last_peered":"2023-05-22T15:42:22.855433+0000","last_clean":"2023-05-22T15:42:22.855433+0000","last_became_active":"2023-05-22T15:42:22.482887+0000","last_became_peered":"2023-05-22T15:42:22.482887+0000","last_unstale":"2023-05-22T15:42:22.855433+0000","last_undegraded":"2023-05-22T15:42:22.855433+0000","last_fullsized":"2023-05-22T15:42:22.855433+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:21.420311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:21.420311+0000","last_clean_scrub_stamp":"2023-05-22T15:42:21.420311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T15:54:16.759039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.14","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.900116+0000","last_change":"2023-05-22T15:42:22.885448+0000","last_active":"2023-05-22T15:42:22.900116+0000","last_peered":"2023-05-22T15:42:22.900116+0000","last_clean":"2023-05-22T15:42:22.900116+0000","last_became_active":"2023-05-22T15:42:22.880308+0000","last_became_peered":"2023-05-22T15:42:22.880308+0000","last_unstale":"2023-05-22T15:42:22.900116+0000","last_undegraded":"2023-05-22T15:42:22.900116+0000","last_fullsized":"2023-05-22T15:42:22.900116+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_clean_scrub_stamp":"2023-05-22T15:42:22.424085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:32:58.984742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.15","version":"89'3","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.846264+0000","last_change":"2023-05-22T15:42:22.480544+0000","last_active":"2023-05-22T15:42:22.846264+0000","last_peered":"2023-05-22T15:42:22.846264+0000","last_clean":"2023-05-22T15:42:22.846264+0000","last_became_active":"2023-05-22T15:42:22.480208+0000","last_became_peered":"2023-05-22T15:42:22.480208+0000","last_unstale":"2023-05-22T15:42:22.846264+0000","last_undegraded":"2023-05-22T15:42:22.846264+0000","last_fullsized":"2023-05-22T15:42:22.846264+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:21.420311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:21.420311+0000","last_clean_scrub_stamp":"2023-05-22T15:42:21.420311+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T16:20:49.287034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.833337+0000","last_change":"2023-05-22T15:42:22.828778+0000","last_active":"2023-05-22T15:42:22.833337+0000","last_peered":"2023-05-22T15:42:22.833337+0000","last_clean":"2023-05-22T15:42:22.833337+0000","last_became_active":"2023-05-22T15:42:22.828441+0000","last_became_peered":"2023-05-22T15:42:22.828441+0000","last_unstale":"2023-05-22T15:42:22.833337+0000","last_undegraded":"2023-05-22T15:42:22.833337+0000","last_fullsized":"2023-05-22T15:42:22.833337+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_clean_scrub_stamp":"2023-05-22T15:42:22.424085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:28:29.036420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.14","version":"89'6","reported_seq":16,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.872307+0000","last_change":"2023-05-22T15:42:22.460189+0000","last_active":"2023-05-22T15:42:22.872307+0000","last_peered":"2023-05-22T15:42:22.872307+0000","last_clean":"2023-05-22T15:42:22.872307+0000","last_became_active":"2023-05-22T15:42:22.459872+0000","last_became_peered":"2023-05-22T15:42:22.459872+0000","last_unstale":"2023-05-22T15:42:22.872307+0000","last_undegraded":"2023-05-22T15:42:22.872307+0000","last_fullsized":"2023-05-22T15:42:22.872307+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:21.420311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:21.420311+0000","last_clean_scrub_stamp":"2023-05-22T15:42:21.420311+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:44:00.313804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1548,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.904617+0000","last_change":"2023-05-22T15:42:22.890528+0000","last_active":"2023-05-22T15:42:22.904617+0000","last_peered":"2023-05-22T15:42:22.904617+0000","last_clean":"2023-05-22T15:42:22.904617+0000","last_became_active":"2023-05-22T15:42:22.889799+0000","last_became_peered":"2023-05-22T15:42:22.889799+0000","last_unstale":"2023-05-22T15:42:22.904617+0000","last_undegraded":"2023-05-22T15:42:22.904617+0000","last_fullsized":"2023-05-22T15:42:22.904617+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_clean_scrub_stamp":"2023-05-22T15:42:22.424085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:58:20.411983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.13","version":"89'2","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.872731+0000","last_change":"2023-05-22T15:42:22.464127+0000","last_active":"2023-05-22T15:42:22.872731+0000","last_peered":"2023-05-22T15:42:22.872731+0000","last_clean":"2023-05-22T15:42:22.872731+0000","last_became_active":"2023-05-22T15:42:22.462383+0000","last_became_peered":"2023-05-22T15:42:22.462383+0000","last_unstale":"2023-05-22T15:42:22.872731+0000","last_undegraded":"2023-05-22T15:42:22.872731+0000","last_fullsized":"2023-05-22T15:42:22.872731+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:21.420311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:21.420311+0000","last_clean_scrub_stamp":"2023-05-22T15:42:21.420311+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T18:55:05.165963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.878408+0000","last_change":"2023-05-22T15:42:22.872839+0000","last_active":"2023-05-22T15:42:22.878408+0000","last_peered":"2023-05-22T15:42:22.878408+0000","last_clean":"2023-05-22T15:42:22.878408+0000","last_became_active":"2023-05-22T15:42:22.871798+0000","last_became_peered":"2023-05-22T15:42:22.871798+0000","last_unstale":"2023-05-22T15:42:22.878408+0000","last_undegraded":"2023-05-22T15:42:22.878408+0000","last_fullsized":"2023-05-22T15:42:22.878408+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_clean_scrub_stamp":"2023-05-22T15:42:22.424085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:43:09.346983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.855709+0000","last_change":"2023-05-22T15:42:22.472923+0000","last_active":"2023-05-22T15:42:22.855709+0000","last_peered":"2023-05-22T15:42:22.855709+0000","last_clean":"2023-05-22T15:42:22.855709+0000","last_became_active":"2023-05-22T15:42:22.470796+0000","last_became_peered":"2023-05-22T15:42:22.470796+0000","last_unstale":"2023-05-22T15:42:22.855709+0000","last_undegraded":"2023-05-22T15:42:22.855709+0000","last_fullsized":"2023-05-22T15:42:22.855709+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:21.420311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:21.420311+0000","last_clean_scrub_stamp":"2023-05-22T15:42:21.420311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:20:10.951568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.879255+0000","last_change":"2023-05-22T15:42:22.878412+0000","last_active":"2023-05-22T15:42:22.879255+0000","last_peered":"2023-05-22T15:42:22.879255+0000","last_clean":"2023-05-22T15:42:22.879255+0000","last_became_active":"2023-05-22T15:42:22.878012+0000","last_became_peered":"2023-05-22T15:42:22.878012+0000","last_unstale":"2023-05-22T15:42:22.879255+0000","last_undegraded":"2023-05-22T15:42:22.879255+0000","last_fullsized":"2023-05-22T15:42:22.879255+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_clean_scrub_stamp":"2023-05-22T15:42:22.424085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:40:34.254174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.838716+0000","last_change":"2023-05-22T15:42:22.475545+0000","last_active":"2023-05-22T15:42:22.838716+0000","last_peered":"2023-05-22T15:42:22.838716+0000","last_clean":"2023-05-22T15:42:22.838716+0000","last_became_active":"2023-05-22T15:42:22.474862+0000","last_became_peered":"2023-05-22T15:42:22.474862+0000","last_unstale":"2023-05-22T15:42:22.838716+0000","last_undegraded":"2023-05-22T15:42:22.838716+0000","last_fullsized":"2023-05-22T15:42:22.838716+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:21.420311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:21.420311+0000","last_clean_scrub_stamp":"2023-05-22T15:42:21.420311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:10:29.610631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.840675+0000","last_change":"2023-05-22T15:42:22.835298+0000","last_active":"2023-05-22T15:42:22.840675+0000","last_peered":"2023-05-22T15:42:22.840675+0000","last_clean":"2023-05-22T15:42:22.840675+0000","last_became_active":"2023-05-22T15:42:22.834994+0000","last_became_peered":"2023-05-22T15:42:22.834994+0000","last_unstale":"2023-05-22T15:42:22.840675+0000","last_undegraded":"2023-05-22T15:42:22.840675+0000","last_fullsized":"2023-05-22T15:42:22.840675+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_clean_scrub_stamp":"2023-05-22T15:42:22.424085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:40:09.493359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.10","version":"89'8","reported_seq":18,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.898704+0000","last_change":"2023-05-22T15:42:22.485986+0000","last_active":"2023-05-22T15:42:22.898704+0000","last_peered":"2023-05-22T15:42:22.898704+0000","last_clean":"2023-05-22T15:42:22.898704+0000","last_became_active":"2023-05-22T15:42:22.485166+0000","last_became_peered":"2023-05-22T15:42:22.485166+0000","last_unstale":"2023-05-22T15:42:22.898704+0000","last_undegraded":"2023-05-22T15:42:22.898704+0000","last_fullsized":"2023-05-22T15:42:22.898704+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:21.420311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:21.420311+0000","last_clean_scrub_stamp":"2023-05-22T15:42:21.420311+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T16:27:16.644358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.838259+0000","last_change":"2023-05-22T15:42:22.833492+0000","last_active":"2023-05-22T15:42:22.838259+0000","last_peered":"2023-05-22T15:42:22.838259+0000","last_clean":"2023-05-22T15:42:22.838259+0000","last_became_active":"2023-05-22T15:42:22.833168+0000","last_became_peered":"2023-05-22T15:42:22.833168+0000","last_unstale":"2023-05-22T15:42:22.838259+0000","last_undegraded":"2023-05-22T15:42:22.838259+0000","last_fullsized":"2023-05-22T15:42:22.838259+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_clean_scrub_stamp":"2023-05-22T15:42:22.424085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T16:45:37.677128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"89'2","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.883114+0000","last_change":"2023-05-22T15:42:22.476545+0000","last_active":"2023-05-22T15:42:22.883114+0000","last_peered":"2023-05-22T15:42:22.883114+0000","last_clean":"2023-05-22T15:42:22.883114+0000","last_became_active":"2023-05-22T15:42:22.475817+0000","last_became_peered":"2023-05-22T15:42:22.475817+0000","last_unstale":"2023-05-22T15:42:22.883114+0000","last_undegraded":"2023-05-22T15:42:22.883114+0000","last_fullsized":"2023-05-22T15:42:22.883114+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:21.420311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:21.420311+0000","last_clean_scrub_stamp":"2023-05-22T15:42:21.420311+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:48:49.697519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.863757+0000","last_change":"2023-05-22T15:42:22.847558+0000","last_active":"2023-05-22T15:42:22.863757+0000","last_peered":"2023-05-22T15:42:22.863757+0000","last_clean":"2023-05-22T15:42:22.863757+0000","last_became_active":"2023-05-22T15:42:22.846836+0000","last_became_peered":"2023-05-22T15:42:22.846836+0000","last_unstale":"2023-05-22T15:42:22.863757+0000","last_undegraded":"2023-05-22T15:42:22.863757+0000","last_fullsized":"2023-05-22T15:42:22.863757+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_clean_scrub_stamp":"2023-05-22T15:42:22.424085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:12:13.799667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.840291+0000","last_change":"2023-05-22T15:42:22.463938+0000","last_active":"2023-05-22T15:42:22.840291+0000","last_peered":"2023-05-22T15:42:22.840291+0000","last_clean":"2023-05-22T15:42:22.840291+0000","last_became_active":"2023-05-22T15:42:22.463146+0000","last_became_peered":"2023-05-22T15:42:22.463146+0000","last_unstale":"2023-05-22T15:42:22.840291+0000","last_undegraded":"2023-05-22T15:42:22.840291+0000","last_fullsized":"2023-05-22T15:42:22.840291+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:21.420311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:21.420311+0000","last_clean_scrub_stamp":"2023-05-22T15:42:21.420311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:33:19.124743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.904620+0000","last_change":"2023-05-22T15:42:22.890342+0000","last_active":"2023-05-22T15:42:22.904620+0000","last_peered":"2023-05-22T15:42:22.904620+0000","last_clean":"2023-05-22T15:42:22.904620+0000","last_became_active":"2023-05-22T15:42:22.889863+0000","last_became_peered":"2023-05-22T15:42:22.889863+0000","last_unstale":"2023-05-22T15:42:22.904620+0000","last_undegraded":"2023-05-22T15:42:22.904620+0000","last_fullsized":"2023-05-22T15:42:22.904620+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_clean_scrub_stamp":"2023-05-22T15:42:22.424085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T18:12:08.205505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.820217+0000","last_change":"2023-05-22T15:42:22.458720+0000","last_active":"2023-05-22T15:42:22.820217+0000","last_peered":"2023-05-22T15:42:22.820217+0000","last_clean":"2023-05-22T15:42:22.820217+0000","last_became_active":"2023-05-22T15:42:22.458403+0000","last_became_peered":"2023-05-22T15:42:22.458403+0000","last_unstale":"2023-05-22T15:42:22.820217+0000","last_undegraded":"2023-05-22T15:42:22.820217+0000","last_fullsized":"2023-05-22T15:42:22.820217+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:21.420311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:21.420311+0000","last_clean_scrub_stamp":"2023-05-22T15:42:21.420311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T16:31:14.665256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.904802+0000","last_change":"2023-05-22T15:42:22.893748+0000","last_active":"2023-05-22T15:42:22.904802+0000","last_peered":"2023-05-22T15:42:22.904802+0000","last_clean":"2023-05-22T15:42:22.904802+0000","last_became_active":"2023-05-22T15:42:22.893073+0000","last_became_peered":"2023-05-22T15:42:22.893073+0000","last_unstale":"2023-05-22T15:42:22.904802+0000","last_undegraded":"2023-05-22T15:42:22.904802+0000","last_fullsized":"2023-05-22T15:42:22.904802+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_clean_scrub_stamp":"2023-05-22T15:42:22.424085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:12:13.799667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.826337+0000","last_change":"2023-05-22T15:42:22.488108+0000","last_active":"2023-05-22T15:42:22.826337+0000","last_peered":"2023-05-22T15:42:22.826337+0000","last_clean":"2023-05-22T15:42:22.826337+0000","last_became_active":"2023-05-22T15:42:22.487683+0000","last_became_peered":"2023-05-22T15:42:22.487683+0000","last_unstale":"2023-05-22T15:42:22.826337+0000","last_undegraded":"2023-05-22T15:42:22.826337+0000","last_fullsized":"2023-05-22T15:42:22.826337+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:21.420311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:21.420311+0000","last_clean_scrub_stamp":"2023-05-22T15:42:21.420311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:04:43.761724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.840649+0000","last_change":"2023-05-22T15:42:22.837171+0000","last_active":"2023-05-22T15:42:22.840649+0000","last_peered":"2023-05-22T15:42:22.840649+0000","last_clean":"2023-05-22T15:42:22.840649+0000","last_became_active":"2023-05-22T15:42:22.834859+0000","last_became_peered":"2023-05-22T15:42:22.834859+0000","last_unstale":"2023-05-22T15:42:22.840649+0000","last_undegraded":"2023-05-22T15:42:22.840649+0000","last_fullsized":"2023-05-22T15:42:22.840649+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_clean_scrub_stamp":"2023-05-22T15:42:22.424085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T03:03:57.842619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.819810+0000","last_change":"2023-05-22T15:42:22.449651+0000","last_active":"2023-05-22T15:42:22.819810+0000","last_peered":"2023-05-22T15:42:22.819810+0000","last_clean":"2023-05-22T15:42:22.819810+0000","last_became_active":"2023-05-22T15:42:22.449039+0000","last_became_peered":"2023-05-22T15:42:22.449039+0000","last_unstale":"2023-05-22T15:42:22.819810+0000","last_undegraded":"2023-05-22T15:42:22.819810+0000","last_fullsized":"2023-05-22T15:42:22.819810+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:21.420311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:21.420311+0000","last_clean_scrub_stamp":"2023-05-22T15:42:21.420311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:17:21.439320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.902230+0000","last_change":"2023-05-22T15:42:22.898314+0000","last_active":"2023-05-22T15:42:22.902230+0000","last_peered":"2023-05-22T15:42:22.902230+0000","last_clean":"2023-05-22T15:42:22.902230+0000","last_became_active":"2023-05-22T15:42:22.897870+0000","last_became_peered":"2023-05-22T15:42:22.897870+0000","last_unstale":"2023-05-22T15:42:22.902230+0000","last_undegraded":"2023-05-22T15:42:22.902230+0000","last_fullsized":"2023-05-22T15:42:22.902230+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_clean_scrub_stamp":"2023-05-22T15:42:22.424085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:24:27.275366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.822162+0000","last_change":"2023-05-22T15:42:22.488090+0000","last_active":"2023-05-22T15:42:22.822162+0000","last_peered":"2023-05-22T15:42:22.822162+0000","last_clean":"2023-05-22T15:42:22.822162+0000","last_became_active":"2023-05-22T15:42:22.487780+0000","last_became_peered":"2023-05-22T15:42:22.487780+0000","last_unstale":"2023-05-22T15:42:22.822162+0000","last_undegraded":"2023-05-22T15:42:22.822162+0000","last_fullsized":"2023-05-22T15:42:22.822162+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:21.420311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:21.420311+0000","last_clean_scrub_stamp":"2023-05-22T15:42:21.420311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T18:57:06.138974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.837240+0000","last_change":"2023-05-22T15:42:22.834742+0000","last_active":"2023-05-22T15:42:22.837240+0000","last_peered":"2023-05-22T15:42:22.837240+0000","last_clean":"2023-05-22T15:42:22.837240+0000","last_became_active":"2023-05-22T15:42:22.834440+0000","last_became_peered":"2023-05-22T15:42:22.834440+0000","last_unstale":"2023-05-22T15:42:22.837240+0000","last_undegraded":"2023-05-22T15:42:22.837240+0000","last_fullsized":"2023-05-22T15:42:22.837240+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_clean_scrub_stamp":"2023-05-22T15:42:22.424085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:56:40.685104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.831673+0000","last_change":"2023-05-22T15:42:22.485738+0000","last_active":"2023-05-22T15:42:22.831673+0000","last_peered":"2023-05-22T15:42:22.831673+0000","last_clean":"2023-05-22T15:42:22.831673+0000","last_became_active":"2023-05-22T15:42:22.485068+0000","last_became_peered":"2023-05-22T15:42:22.485068+0000","last_unstale":"2023-05-22T15:42:22.831673+0000","last_undegraded":"2023-05-22T15:42:22.831673+0000","last_fullsized":"2023-05-22T15:42:22.831673+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:21.420311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:21.420311+0000","last_clean_scrub_stamp":"2023-05-22T15:42:21.420311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:10:29.610631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.888080+0000","last_change":"2023-05-22T15:42:22.880082+0000","last_active":"2023-05-22T15:42:22.888080+0000","last_peered":"2023-05-22T15:42:22.888080+0000","last_clean":"2023-05-22T15:42:22.888080+0000","last_became_active":"2023-05-22T15:42:22.876593+0000","last_became_peered":"2023-05-22T15:42:22.876593+0000","last_unstale":"2023-05-22T15:42:22.888080+0000","last_undegraded":"2023-05-22T15:42:22.888080+0000","last_fullsized":"2023-05-22T15:42:22.888080+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_clean_scrub_stamp":"2023-05-22T15:42:22.424085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:28:29.036420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.853844+0000","last_change":"2023-05-22T15:42:22.476592+0000","last_active":"2023-05-22T15:42:22.853844+0000","last_peered":"2023-05-22T15:42:22.853844+0000","last_clean":"2023-05-22T15:42:22.853844+0000","last_became_active":"2023-05-22T15:42:22.475833+0000","last_became_peered":"2023-05-22T15:42:22.475833+0000","last_unstale":"2023-05-22T15:42:22.853844+0000","last_undegraded":"2023-05-22T15:42:22.853844+0000","last_fullsized":"2023-05-22T15:42:22.853844+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:21.420311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:21.420311+0000","last_clean_scrub_stamp":"2023-05-22T15:42:21.420311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:29:53.579459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.878707+0000","last_change":"2023-05-22T15:42:22.877564+0000","last_active":"2023-05-22T15:42:22.878707+0000","last_peered":"2023-05-22T15:42:22.878707+0000","last_clean":"2023-05-22T15:42:22.878707+0000","last_became_active":"2023-05-22T15:42:22.877185+0000","last_became_peered":"2023-05-22T15:42:22.877185+0000","last_unstale":"2023-05-22T15:42:22.878707+0000","last_undegraded":"2023-05-22T15:42:22.878707+0000","last_fullsized":"2023-05-22T15:42:22.878707+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_clean_scrub_stamp":"2023-05-22T15:42:22.424085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:04:18.558225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.b","version":"89'2","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.881660+0000","last_change":"2023-05-22T15:42:22.470815+0000","last_active":"2023-05-22T15:42:22.881660+0000","last_peered":"2023-05-22T15:42:22.881660+0000","last_clean":"2023-05-22T15:42:22.881660+0000","last_became_active":"2023-05-22T15:42:22.470017+0000","last_became_peered":"2023-05-22T15:42:22.470017+0000","last_unstale":"2023-05-22T15:42:22.881660+0000","last_undegraded":"2023-05-22T15:42:22.881660+0000","last_fullsized":"2023-05-22T15:42:22.881660+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:21.420311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:21.420311+0000","last_clean_scrub_stamp":"2023-05-22T15:42:21.420311+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T16:27:44.490234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"77'161","reported_seq":390,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.454789+0000","last_change":"2023-05-22T15:26:39.318204+0000","last_active":"2023-05-22T15:42:22.454789+0000","last_peered":"2023-05-22T15:42:22.454789+0000","last_clean":"2023-05-22T15:42:22.454789+0000","last_became_active":"2023-05-22T15:26:39.315370+0000","last_became_peered":"2023-05-22T15:26:39.315370+0000","last_unstale":"2023-05-22T15:42:22.454789+0000","last_undegraded":"2023-05-22T15:42:22.454789+0000","last_fullsized":"2023-05-22T15:42:22.454789+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:38.020076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:38.020076+0000","last_clean_scrub_stamp":"2023-05-22T15:26:38.020076+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:06:39.207543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":500,"num_read_kb":924,"num_write":239,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":84,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.444823+0000","last_change":"2023-05-22T15:26:44.971951+0000","last_active":"2023-05-22T15:42:22.444823+0000","last_peered":"2023-05-22T15:42:22.444823+0000","last_clean":"2023-05-22T15:42:22.444823+0000","last_became_active":"2023-05-22T15:26:44.970872+0000","last_became_peered":"2023-05-22T15:26:44.970872+0000","last_unstale":"2023-05-22T15:42:22.444823+0000","last_undegraded":"2023-05-22T15:42:22.444823+0000","last_fullsized":"2023-05-22T15:42:22.444823+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:55:34.752949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.881848+0000","last_change":"2023-05-22T15:42:22.881328+0000","last_active":"2023-05-22T15:42:22.881848+0000","last_peered":"2023-05-22T15:42:22.881848+0000","last_clean":"2023-05-22T15:42:22.881848+0000","last_became_active":"2023-05-22T15:42:22.881046+0000","last_became_peered":"2023-05-22T15:42:22.881046+0000","last_unstale":"2023-05-22T15:42:22.881848+0000","last_undegraded":"2023-05-22T15:42:22.881848+0000","last_fullsized":"2023-05-22T15:42:22.881848+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_clean_scrub_stamp":"2023-05-22T15:42:22.424085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T16:42:25.904179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.826275+0000","last_change":"2023-05-22T15:42:22.460425+0000","last_active":"2023-05-22T15:42:22.826275+0000","last_peered":"2023-05-22T15:42:22.826275+0000","last_clean":"2023-05-22T15:42:22.826275+0000","last_became_active":"2023-05-22T15:42:22.460116+0000","last_became_peered":"2023-05-22T15:42:22.460116+0000","last_unstale":"2023-05-22T15:42:22.826275+0000","last_undegraded":"2023-05-22T15:42:22.826275+0000","last_fullsized":"2023-05-22T15:42:22.826275+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:21.420311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:21.420311+0000","last_clean_scrub_stamp":"2023-05-22T15:42:21.420311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T18:23:21.266754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":81,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.454608+0000","last_change":"2023-05-22T15:26:44.977198+0000","last_active":"2023-05-22T15:42:22.454608+0000","last_peered":"2023-05-22T15:42:22.454608+0000","last_clean":"2023-05-22T15:42:22.454608+0000","last_became_active":"2023-05-22T15:26:44.976445+0000","last_became_peered":"2023-05-22T15:26:44.976445+0000","last_unstale":"2023-05-22T15:42:22.454608+0000","last_undegraded":"2023-05-22T15:42:22.454608+0000","last_fullsized":"2023-05-22T15:42:22.454608+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:12:41.464054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.881138+0000","last_change":"2023-05-22T15:42:22.880493+0000","last_active":"2023-05-22T15:42:22.881138+0000","last_peered":"2023-05-22T15:42:22.881138+0000","last_clean":"2023-05-22T15:42:22.881138+0000","last_became_active":"2023-05-22T15:42:22.880204+0000","last_became_peered":"2023-05-22T15:42:22.880204+0000","last_unstale":"2023-05-22T15:42:22.881138+0000","last_undegraded":"2023-05-22T15:42:22.881138+0000","last_fullsized":"2023-05-22T15:42:22.881138+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_clean_scrub_stamp":"2023-05-22T15:42:22.424085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T03:03:54.290576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.839012+0000","last_change":"2023-05-22T15:42:22.476917+0000","last_active":"2023-05-22T15:42:22.839012+0000","last_peered":"2023-05-22T15:42:22.839012+0000","last_clean":"2023-05-22T15:42:22.839012+0000","last_became_active":"2023-05-22T15:42:22.476209+0000","last_became_peered":"2023-05-22T15:42:22.476209+0000","last_unstale":"2023-05-22T15:42:22.839012+0000","last_undegraded":"2023-05-22T15:42:22.839012+0000","last_fullsized":"2023-05-22T15:42:22.839012+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:21.420311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:21.420311+0000","last_clean_scrub_stamp":"2023-05-22T15:42:21.420311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T16:27:16.644358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":80,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.434673+0000","last_change":"2023-05-22T15:26:44.969469+0000","last_active":"2023-05-22T15:42:22.434673+0000","last_peered":"2023-05-22T15:42:22.434673+0000","last_clean":"2023-05-22T15:42:22.434673+0000","last_became_active":"2023-05-22T15:26:44.969083+0000","last_became_peered":"2023-05-22T15:26:44.969083+0000","last_unstale":"2023-05-22T15:42:22.434673+0000","last_undegraded":"2023-05-22T15:42:22.434673+0000","last_fullsized":"2023-05-22T15:42:22.434673+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:29:23.670067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.839489+0000","last_change":"2023-05-22T15:42:22.837977+0000","last_active":"2023-05-22T15:42:22.839489+0000","last_peered":"2023-05-22T15:42:22.839489+0000","last_clean":"2023-05-22T15:42:22.839489+0000","last_became_active":"2023-05-22T15:42:22.837699+0000","last_became_peered":"2023-05-22T15:42:22.837699+0000","last_unstale":"2023-05-22T15:42:22.839489+0000","last_undegraded":"2023-05-22T15:42:22.839489+0000","last_fullsized":"2023-05-22T15:42:22.839489+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_clean_scrub_stamp":"2023-05-22T15:42:22.424085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:44:21.508851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.857077+0000","last_change":"2023-05-22T15:42:22.472394+0000","last_active":"2023-05-22T15:42:22.857077+0000","last_peered":"2023-05-22T15:42:22.857077+0000","last_clean":"2023-05-22T15:42:22.857077+0000","last_became_active":"2023-05-22T15:42:22.469928+0000","last_became_peered":"2023-05-22T15:42:22.469928+0000","last_unstale":"2023-05-22T15:42:22.857077+0000","last_undegraded":"2023-05-22T15:42:22.857077+0000","last_fullsized":"2023-05-22T15:42:22.857077+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:21.420311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:21.420311+0000","last_clean_scrub_stamp":"2023-05-22T15:42:21.420311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T18:08:33.493154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":84,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.444459+0000","last_change":"2023-05-22T15:26:44.976632+0000","last_active":"2023-05-22T15:42:22.444459+0000","last_peered":"2023-05-22T15:42:22.444459+0000","last_clean":"2023-05-22T15:42:22.444459+0000","last_became_active":"2023-05-22T15:26:44.975546+0000","last_became_peered":"2023-05-22T15:26:44.975546+0000","last_unstale":"2023-05-22T15:42:22.444459+0000","last_undegraded":"2023-05-22T15:42:22.444459+0000","last_fullsized":"2023-05-22T15:42:22.444459+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:42:57.973030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.854023+0000","last_change":"2023-05-22T15:42:22.852710+0000","last_active":"2023-05-22T15:42:22.854023+0000","last_peered":"2023-05-22T15:42:22.854023+0000","last_clean":"2023-05-22T15:42:22.854023+0000","last_became_active":"2023-05-22T15:42:22.852348+0000","last_became_peered":"2023-05-22T15:42:22.852348+0000","last_unstale":"2023-05-22T15:42:22.854023+0000","last_undegraded":"2023-05-22T15:42:22.854023+0000","last_fullsized":"2023-05-22T15:42:22.854023+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_clean_scrub_stamp":"2023-05-22T15:42:22.424085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T16:55:23.105053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.848334+0000","last_change":"2023-05-22T15:42:22.491297+0000","last_active":"2023-05-22T15:42:22.848334+0000","last_peered":"2023-05-22T15:42:22.848334+0000","last_clean":"2023-05-22T15:42:22.848334+0000","last_became_active":"2023-05-22T15:42:22.490770+0000","last_became_peered":"2023-05-22T15:42:22.490770+0000","last_unstale":"2023-05-22T15:42:22.848334+0000","last_undegraded":"2023-05-22T15:42:22.848334+0000","last_fullsized":"2023-05-22T15:42:22.848334+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:21.420311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:21.420311+0000","last_clean_scrub_stamp":"2023-05-22T15:42:21.420311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T16:27:16.644358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":74,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.448308+0000","last_change":"2023-05-22T15:26:45.091475+0000","last_active":"2023-05-22T15:42:22.448308+0000","last_peered":"2023-05-22T15:42:22.448308+0000","last_clean":"2023-05-22T15:42:22.448308+0000","last_became_active":"2023-05-22T15:26:45.091143+0000","last_became_peered":"2023-05-22T15:26:45.091143+0000","last_unstale":"2023-05-22T15:42:22.448308+0000","last_undegraded":"2023-05-22T15:42:22.448308+0000","last_fullsized":"2023-05-22T15:42:22.448308+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:12:41.464054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.834241+0000","last_change":"2023-05-22T15:42:22.830100+0000","last_active":"2023-05-22T15:42:22.834241+0000","last_peered":"2023-05-22T15:42:22.834241+0000","last_clean":"2023-05-22T15:42:22.834241+0000","last_became_active":"2023-05-22T15:42:22.827908+0000","last_became_peered":"2023-05-22T15:42:22.827908+0000","last_unstale":"2023-05-22T15:42:22.834241+0000","last_undegraded":"2023-05-22T15:42:22.834241+0000","last_fullsized":"2023-05-22T15:42:22.834241+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_clean_scrub_stamp":"2023-05-22T15:42:22.424085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:24:27.275366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.d","version":"89'1","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.883095+0000","last_change":"2023-05-22T15:42:22.490470+0000","last_active":"2023-05-22T15:42:22.883095+0000","last_peered":"2023-05-22T15:42:22.883095+0000","last_clean":"2023-05-22T15:42:22.883095+0000","last_became_active":"2023-05-22T15:42:22.489987+0000","last_became_peered":"2023-05-22T15:42:22.489987+0000","last_unstale":"2023-05-22T15:42:22.883095+0000","last_undegraded":"2023-05-22T15:42:22.883095+0000","last_fullsized":"2023-05-22T15:42:22.883095+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:21.420311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:21.420311+0000","last_clean_scrub_stamp":"2023-05-22T15:42:21.420311+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T18:13:50.414629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":81,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.462533+0000","last_change":"2023-05-22T15:26:45.092367+0000","last_active":"2023-05-22T15:42:22.462533+0000","last_peered":"2023-05-22T15:42:22.462533+0000","last_clean":"2023-05-22T15:42:22.462533+0000","last_became_active":"2023-05-22T15:26:45.091951+0000","last_became_peered":"2023-05-22T15:26:45.091951+0000","last_unstale":"2023-05-22T15:42:22.462533+0000","last_undegraded":"2023-05-22T15:42:22.462533+0000","last_fullsized":"2023-05-22T15:42:22.462533+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:42:57.973030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.904997+0000","last_change":"2023-05-22T15:42:22.895829+0000","last_active":"2023-05-22T15:42:22.904997+0000","last_peered":"2023-05-22T15:42:22.904997+0000","last_clean":"2023-05-22T15:42:22.904997+0000","last_became_active":"2023-05-22T15:42:22.895226+0000","last_became_peered":"2023-05-22T15:42:22.895226+0000","last_unstale":"2023-05-22T15:42:22.904997+0000","last_undegraded":"2023-05-22T15:42:22.904997+0000","last_fullsized":"2023-05-22T15:42:22.904997+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_clean_scrub_stamp":"2023-05-22T15:42:22.424085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:57:49.944485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.e","version":"89'2","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.876669+0000","last_change":"2023-05-22T15:42:22.487669+0000","last_active":"2023-05-22T15:42:22.876669+0000","last_peered":"2023-05-22T15:42:22.876669+0000","last_clean":"2023-05-22T15:42:22.876669+0000","last_became_active":"2023-05-22T15:42:22.486890+0000","last_became_peered":"2023-05-22T15:42:22.486890+0000","last_unstale":"2023-05-22T15:42:22.876669+0000","last_undegraded":"2023-05-22T15:42:22.876669+0000","last_fullsized":"2023-05-22T15:42:22.876669+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:21.420311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:21.420311+0000","last_clean_scrub_stamp":"2023-05-22T15:42:21.420311+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:59:22.855595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":74,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.449144+0000","last_change":"2023-05-22T15:26:44.978858+0000","last_active":"2023-05-22T15:42:22.449144+0000","last_peered":"2023-05-22T15:42:22.449144+0000","last_clean":"2023-05-22T15:42:22.449144+0000","last_became_active":"2023-05-22T15:26:44.978443+0000","last_became_peered":"2023-05-22T15:26:44.978443+0000","last_unstale":"2023-05-22T15:42:22.449144+0000","last_undegraded":"2023-05-22T15:42:22.449144+0000","last_fullsized":"2023-05-22T15:42:22.449144+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:46:29.290243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":9,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.886338+0000","last_change":"2023-05-22T15:42:22.869686+0000","last_active":"2023-05-22T15:42:22.886338+0000","last_peered":"2023-05-22T15:42:22.886338+0000","last_clean":"2023-05-22T15:42:22.886338+0000","last_became_active":"2023-05-22T15:42:22.868726+0000","last_became_peered":"2023-05-22T15:42:22.868726+0000","last_unstale":"2023-05-22T15:42:22.886338+0000","last_undegraded":"2023-05-22T15:42:22.886338+0000","last_fullsized":"2023-05-22T15:42:22.886338+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:22.424085+0000","last_clean_scrub_stamp":"2023-05-22T15:42:22.424085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:43:09.346983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.f","version":"89'1","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.879761+0000","last_change":"2023-05-22T15:42:22.473483+0000","last_active":"2023-05-22T15:42:22.879761+0000","last_peered":"2023-05-22T15:42:22.879761+0000","last_clean":"2023-05-22T15:42:22.879761+0000","last_became_active":"2023-05-22T15:42:22.472160+0000","last_became_peered":"2023-05-22T15:42:22.472160+0000","last_unstale":"2023-05-22T15:42:22.879761+0000","last_undegraded":"2023-05-22T15:42:22.879761+0000","last_fullsized":"2023-05-22T15:42:22.879761+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:21.420311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:21.420311+0000","last_clean_scrub_stamp":"2023-05-22T15:42:21.420311+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:17:21.439320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":76,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-05-22T15:42:22.456529+0000","last_change":"2023-05-22T15:26:44.982421+0000","last_active":"2023-05-22T15:42:22.456529+0000","last_peered":"2023-05-22T15:42:22.456529+0000","last_clean":"2023-05-22T15:42:22.456529+0000","last_became_active":"2023-05-22T15:26:44.981586+0000","last_became_peered":"2023-05-22T15:26:44.981586+0000","last_unstale":"2023-05-22T15:42:22.456529+0000","last_undegraded":"2023-05-22T15:42:22.456529+0000","last_fullsized":"2023-05-22T15:42:22.456529+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:14:11.156544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":10,"num_pg":32,"stat_sum":{"num_bytes":2330,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":500,"num_read_kb":924,"num_write":239,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161,"ondisk_log_size":161,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575039,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30068,"kb_used_data":1644,"kb_used_omap":0,"kb_used_meta":28416,"kb_avail":94341772,"statfs":{"total":96636764160,"available":96605974528,"internally_reserved":0,"allocated":1683456,"data_stored":1155640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":29097984},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 15:41:58 2023","interfaces":[{"interface":"back","average":{"1min":0.88900000000000001,"5min":0.86099999999999999,"15min":0.86699999999999999},"min":{"1min":0.624,"5min":0.44800000000000001,"15min":0.40899999999999997},"max":{"1min":1.1830000000000001,"5min":1.4199999999999999,"15min":1.7709999999999999},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.86899999999999999,"5min":0.871,"15min":0.876},"min":{"1min":0.63600000000000001,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":1.2050000000000001,"5min":1.3169999999999999,"15min":1.4239999999999999},"last":0.60599999999999998}]},{"osd":1,"last update":"Mon May 22 15:41:58 2023","interfaces":[{"interface":"back","average":{"1min":0.94799999999999995,"5min":0.98199999999999998,"15min":0.98199999999999998},"min":{"1min":0.77300000000000002,"5min":0.5,"15min":0.5},"max":{"1min":1.385,"5min":1.389,"15min":1.8300000000000001},"last":0.92100000000000004},{"interface":"front","average":{"1min":1.0229999999999999,"5min":1.018,"15min":1.0229999999999999},"min":{"1min":0.60799999999999998,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":1.2130000000000001,"5min":1.3169999999999999,"15min":2.161},"last":0.70699999999999996}]},{"osd":2,"last update":"Mon May 22 15:41:58 2023","interfaces":[{"interface":"back","average":{"1min":0.84899999999999998,"5min":0.87,"15min":0.90400000000000003},"min":{"1min":0.67300000000000004,"5min":0.49399999999999999,"15min":0.46800000000000003},"max":{"1min":1.181,"5min":1.2070000000000001,"15min":6.7279999999999998},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.95799999999999996,"5min":0.94799999999999995,"15min":0.96299999999999997},"min":{"1min":0.59199999999999997,"5min":0.52300000000000002,"15min":0.51000000000000001},"max":{"1min":1.151,"5min":1.2569999999999999,"15min":6.8810000000000002},"last":0.79500000000000004}]},{"osd":3,"last update":"Mon May 22 15:41:58 2023","interfaces":[{"interface":"back","average":{"1min":1.0109999999999999,"5min":0.95299999999999996,"15min":0.96099999999999997},"min":{"1min":0.70199999999999996,"5min":0.55900000000000005,"15min":0.53800000000000003},"max":{"1min":1.254,"5min":1.3360000000000001,"15min":3.0569999999999999},"last":1.091},{"interface":"front","average":{"1min":0.91500000000000004,"5min":0.96599999999999997,"15min":0.98399999999999999},"min":{"1min":0.68200000000000005,"5min":0.61799999999999999,"15min":0.54900000000000004},"max":{"1min":1.2989999999999999,"5min":1.3029999999999999,"15min":1.6879999999999999},"last":0.84699999999999998}]},{"osd":4,"last update":"Mon May 22 15:41:58 2023","interfaces":[{"interface":"back","average":{"1min":0.83599999999999997,"5min":0.89600000000000002,"15min":0.88400000000000001},"min":{"1min":0.62,"5min":0.57399999999999995,"15min":0.57399999999999995},"max":{"1min":1.093,"5min":1.621,"15min":1.621},"last":1.266},{"interface":"front","average":{"1min":0.98299999999999998,"5min":0.99199999999999999,"15min":0.98299999999999998},"min":{"1min":0.749,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.359,"5min":1.4319999999999999,"15min":1.958},"last":1.2130000000000001}]},{"osd":5,"last update":"Mon May 22 15:41:58 2023","interfaces":[{"interface":"back","average":{"1min":0.995,"5min":1.022,"15min":1.002},"min":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.56699999999999995},"max":{"1min":1.3939999999999999,"5min":1.409,"15min":1.5029999999999999},"last":1.177},{"interface":"front","average":{"1min":1.1220000000000001,"5min":1.121,"15min":1.1359999999999999},"min":{"1min":0.88800000000000001,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":1.325,"5min":1.5229999999999999,"15min":5.0839999999999996},"last":1.141}]},{"osd":6,"last update":"Mon May 22 15:41:58 2023","interfaces":[{"interface":"back","average":{"1min":1.0169999999999999,"5min":1.0449999999999999,"15min":1.071},"min":{"1min":0.73399999999999999,"5min":0.65800000000000003,"15min":0.65800000000000003},"max":{"1min":1.256,"5min":1.484,"15min":5.1719999999999997},"last":0.96799999999999997},{"interface":"front","average":{"1min":1.109,"5min":1.133,"15min":1.1339999999999999},"min":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"max":{"1min":1.462,"5min":1.702,"15min":2.04},"last":1.3220000000000001}]}]},{"osd":6,"up_from":13,"seq":55834575040,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29712,"kb_used_data":1032,"kb_used_omap":0,"kb_used_meta":28672,"kb_avail":94342128,"statfs":{"total":96636764160,"available":96606339072,"internally_reserved":0,"allocated":1056768,"data_stored":550276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":29360128},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 15:42:06 2023","interfaces":[{"interface":"back","average":{"1min":0.86599999999999999,"5min":0.89000000000000001,"15min":0.89800000000000002},"min":{"1min":0.59299999999999997,"5min":0.496,"15min":0.496},"max":{"1min":1.323,"5min":1.327,"15min":1.355},"last":0.89900000000000002},{"interface":"front","average":{"1min":0.99199999999999999,"5min":0.90300000000000002,"15min":0.90900000000000003},"min":{"1min":0.66200000000000003,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":1.264,"5min":1.5680000000000001,"15min":1.5680000000000001},"last":0.76000000000000001}]},{"osd":1,"last update":"Mon May 22 15:42:06 2023","interfaces":[{"interface":"back","average":{"1min":0.92800000000000005,"5min":0.92800000000000005,"15min":0.91300000000000003},"min":{"1min":0.623,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":1.2709999999999999,"5min":1.3169999999999999,"15min":1.375},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.93400000000000005,"5min":0.94499999999999995,"15min":0.94899999999999995},"min":{"1min":0.64400000000000002,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.2709999999999999,"5min":1.554,"15min":1.554},"last":0.98799999999999999}]},{"osd":2,"last update":"Mon May 22 15:42:06 2023","interfaces":[{"interface":"back","average":{"1min":1.0029999999999999,"5min":0.97099999999999997,"15min":0.97999999999999998},"min":{"1min":0.72599999999999998,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":1.3660000000000001,"5min":1.425,"15min":1.425},"last":1.0700000000000001},{"interface":"front","average":{"1min":1.073,"5min":1.048,"15min":1.0509999999999999},"min":{"1min":0.84299999999999997,"5min":0.442,"15min":0.442},"max":{"1min":1.2909999999999999,"5min":1.5369999999999999,"15min":1.5369999999999999},"last":0.81899999999999995}]},{"osd":3,"last update":"Mon May 22 15:42:06 2023","interfaces":[{"interface":"back","average":{"1min":0.95799999999999996,"5min":0.99099999999999999,"15min":0.99399999999999999},"min":{"1min":0.72599999999999998,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":1.1830000000000001,"5min":1.679,"15min":1.679},"last":1.042},{"interface":"front","average":{"1min":0.96799999999999997,"5min":0.95699999999999996,"15min":0.96099999999999997},"min":{"1min":0.62,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.159,"5min":1.4570000000000001,"15min":1.486},"last":1.1559999999999999}]},{"osd":4,"last update":"Mon May 22 15:42:06 2023","interfaces":[{"interface":"back","average":{"1min":0.85699999999999998,"5min":0.91700000000000004,"15min":0.93100000000000005},"min":{"1min":0.64900000000000002,"5min":0.60299999999999998,"15min":0.57599999999999996},"max":{"1min":1.1479999999999999,"5min":1.6859999999999999,"15min":1.6859999999999999},"last":0.62},{"interface":"front","average":{"1min":0.96199999999999997,"5min":1.016,"15min":1.032},"min":{"1min":0.67700000000000005,"5min":0.622,"15min":0.61299999999999999},"max":{"1min":1.321,"5min":1.8300000000000001,"15min":1.8300000000000001},"last":0.85499999999999998}]},{"osd":5,"last update":"Mon May 22 15:42:06 2023","interfaces":[{"interface":"back","average":{"1min":1.01,"5min":0.98099999999999998,"15min":0.97999999999999998},"min":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.64400000000000002},"max":{"1min":1.4530000000000001,"5min":1.6499999999999999,"15min":1.6499999999999999},"last":0.66900000000000004},{"interface":"front","average":{"1min":1.0760000000000001,"5min":1.103,"15min":1.1000000000000001},"min":{"1min":0.68799999999999994,"5min":0.68000000000000005,"15min":0.68000000000000005},"max":{"1min":1.3919999999999999,"5min":1.607,"15min":1.7509999999999999},"last":1.0169999999999999}]},{"osd":7,"last update":"Mon May 22 15:42:06 2023","interfaces":[{"interface":"back","average":{"1min":1.1180000000000001,"5min":1.1160000000000001,"15min":1.1359999999999999},"min":{"1min":0.78700000000000003,"5min":0.68400000000000005,"15min":0.58399999999999996},"max":{"1min":1.3740000000000001,"5min":1.9179999999999999,"15min":1.9179999999999999},"last":1.109},{"interface":"front","average":{"1min":1.1040000000000001,"5min":1.139,"15min":1.133},"min":{"1min":0.81899999999999995,"5min":0.74099999999999999,"15min":0.68999999999999995},"max":{"1min":1.3580000000000001,"5min":1.8520000000000001,"15min":1.8520000000000001},"last":0.94299999999999995}]}]},{"osd":5,"up_from":13,"seq":55834575040,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":23696,"kb_used_data":1032,"kb_used_omap":0,"kb_used_meta":22656,"kb_avail":94348144,"statfs":{"total":96636764160,"available":96612499456,"internally_reserved":0,"allocated":1056768,"data_stored":550276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":23199744},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 15:42:04 2023","interfaces":[{"interface":"back","average":{"1min":0.92100000000000004,"5min":0.86799999999999999,"15min":0.88},"min":{"1min":0.68899999999999995,"5min":0.54900000000000004,"15min":0.48399999999999999},"max":{"1min":1.2849999999999999,"5min":1.3440000000000001,"15min":1.3440000000000001},"last":1.016},{"interface":"front","average":{"1min":0.93600000000000005,"5min":0.89800000000000002,"15min":0.88},"min":{"1min":0.58499999999999996,"5min":0.57499999999999996,"15min":0.497},"max":{"1min":1.2549999999999999,"5min":2.1699999999999999,"15min":2.1699999999999999},"last":1.0449999999999999}]},{"osd":1,"last update":"Mon May 22 15:42:04 2023","interfaces":[{"interface":"back","average":{"1min":1.099,"5min":1.004,"15min":0.996},"min":{"1min":0.87,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":1.337,"5min":1.9370000000000001,"15min":1.9370000000000001},"last":1.163},{"interface":"front","average":{"1min":1.028,"5min":0.999,"15min":1.02},"min":{"1min":0.67000000000000004,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.3680000000000001,"5min":1.3680000000000001,"15min":1.399},"last":1.1120000000000001}]},{"osd":2,"last update":"Mon May 22 15:42:04 2023","interfaces":[{"interface":"back","average":{"1min":0.95699999999999996,"5min":0.96299999999999997,"15min":0.93200000000000005},"min":{"1min":0.72699999999999998,"5min":0.504,"15min":0.504},"max":{"1min":1.2549999999999999,"5min":1.9830000000000001,"15min":1.9830000000000001},"last":1.1379999999999999},{"interface":"front","average":{"1min":1.0189999999999999,"5min":0.96099999999999997,"15min":0.93700000000000006},"min":{"1min":0.75900000000000001,"5min":0.60799999999999998,"15min":0.53800000000000003},"max":{"1min":1.321,"5min":2.1120000000000001,"15min":2.1120000000000001},"last":0.95199999999999996}]},{"osd":3,"last update":"Mon May 22 15:42:04 2023","interfaces":[{"interface":"back","average":{"1min":1.0029999999999999,"5min":0.98899999999999999,"15min":0.95699999999999996},"min":{"1min":0.67000000000000004,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.2809999999999999,"5min":1.4419999999999999,"15min":1.6839999999999999},"last":0.79000000000000004},{"interface":"front","average":{"1min":0.97399999999999998,"5min":1.056,"15min":1.038},"min":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":1.1970000000000001,"5min":2.52,"15min":2.52},"last":0.88400000000000001}]},{"osd":4,"last update":"Mon May 22 15:42:04 2023","interfaces":[{"interface":"back","average":{"1min":0.90900000000000003,"5min":0.97799999999999998,"15min":0.93799999999999994},"min":{"1min":0.68200000000000005,"5min":0.60599999999999998,"15min":0.57599999999999996},"max":{"1min":1.2609999999999999,"5min":2.226,"15min":2.226},"last":1.073},{"interface":"front","average":{"1min":1.075,"5min":1.0649999999999999,"15min":1.038},"min":{"1min":0.63200000000000001,"5min":0.60199999999999998,"15min":0.60199999999999998},"max":{"1min":1.4239999999999999,"5min":1.6859999999999999,"15min":1.6859999999999999},"last":0.69099999999999995}]},{"osd":6,"last update":"Mon May 22 15:42:04 2023","interfaces":[{"interface":"back","average":{"1min":1.016,"5min":1.0840000000000001,"15min":1.0449999999999999},"min":{"1min":0.748,"5min":0.67900000000000005,"15min":0.61299999999999999},"max":{"1min":1.345,"5min":2.831,"15min":2.831},"last":0.751},{"interface":"front","average":{"1min":1.085,"5min":1.1080000000000001,"15min":1.0780000000000001},"min":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.69299999999999995},"max":{"1min":1.3380000000000001,"5min":2.3690000000000002,"15min":2.3690000000000002},"last":0.81899999999999995}]},{"osd":7,"last update":"Mon May 22 15:42:04 2023","interfaces":[{"interface":"back","average":{"1min":1.077,"5min":1.1639999999999999,"15min":1.125},"min":{"1min":0.70999999999999996,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":1.47,"5min":1.849,"15min":2.8260000000000001},"last":0.92000000000000004},{"interface":"front","average":{"1min":1.1579999999999999,"5min":1.22,"15min":1.1699999999999999},"min":{"1min":0.80400000000000005,"5min":0.75700000000000001,"15min":0.71699999999999997},"max":{"1min":1.3220000000000001,"5min":2.7429999999999999,"15min":2.7429999999999999},"last":0.99199999999999999}]}]},{"osd":4,"up_from":13,"seq":55834575038,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":25424,"kb_used_data":1032,"kb_used_omap":0,"kb_used_meta":24384,"kb_avail":94346416,"statfs":{"total":96636764160,"available":96610729984,"internally_reserved":0,"allocated":1056768,"data_stored":550276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":24969216},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 15:42:07 2023","interfaces":[{"interface":"back","average":{"1min":0.88,"5min":0.91300000000000003,"15min":0.90600000000000003},"min":{"1min":0.58299999999999996,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":1.3360000000000001,"5min":1.9910000000000001,"15min":1.9910000000000001},"last":0.80200000000000005},{"interface":"front","average":{"1min":0.96899999999999997,"5min":0.91900000000000004,"15min":0.93200000000000005},"min":{"1min":0.63500000000000001,"5min":0.373,"15min":0.373},"max":{"1min":1.306,"5min":1.3779999999999999,"15min":1.633},"last":0.65100000000000002}]},{"osd":1,"last update":"Mon May 22 15:42:07 2023","interfaces":[{"interface":"back","average":{"1min":1.0029999999999999,"5min":0.98199999999999998,"15min":0.97699999999999998},"min":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.54000000000000004},"max":{"1min":1.2689999999999999,"5min":1.383,"15min":1.3859999999999999},"last":0.71399999999999997},{"interface":"front","average":{"1min":0.88900000000000001,"5min":0.93500000000000005,"15min":0.91200000000000003},"min":{"1min":0.60299999999999998,"5min":0.55100000000000005,"15min":0.51600000000000001},"max":{"1min":1.149,"5min":1.3120000000000001,"15min":1.3640000000000001},"last":0.86699999999999999}]},{"osd":2,"last update":"Mon May 22 15:42:07 2023","interfaces":[{"interface":"back","average":{"1min":0.92800000000000005,"5min":0.94699999999999995,"15min":0.94699999999999995},"min":{"1min":0.747,"5min":0.495,"15min":0.495},"max":{"1min":1.212,"5min":2.1019999999999999,"15min":2.1019999999999999},"last":1.0780000000000001},{"interface":"front","average":{"1min":1.0489999999999999,"5min":1.01,"15min":1.0109999999999999},"min":{"1min":0.77600000000000002,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":1.4390000000000001,"5min":1.4390000000000001,"15min":1.9770000000000001},"last":0.91900000000000004}]},{"osd":3,"last update":"Mon May 22 15:42:07 2023","interfaces":[{"interface":"back","average":{"1min":0.89800000000000002,"5min":0.91800000000000004,"15min":0.94299999999999995},"min":{"1min":0.72399999999999998,"5min":0.63100000000000001,"15min":0.54600000000000004},"max":{"1min":1.2490000000000001,"5min":1.29,"15min":1.665},"last":1.153},{"interface":"front","average":{"1min":1.075,"5min":1.0529999999999999,"15min":1.038},"min":{"1min":0.85399999999999998,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":1.238,"5min":2.1720000000000002,"15min":2.1720000000000002},"last":1.218}]},{"osd":5,"last update":"Mon May 22 15:42:07 2023","interfaces":[{"interface":"back","average":{"1min":1.0569999999999999,"5min":1.0409999999999999,"15min":1},"min":{"1min":0.83399999999999996,"5min":0.621,"15min":0.51200000000000001},"max":{"1min":1.373,"5min":1.472,"15min":1.7270000000000001},"last":1.0349999999999999},{"interface":"front","average":{"1min":1.012,"5min":0.98599999999999999,"15min":0.97999999999999998},"min":{"1min":0.70799999999999996,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":1.325,"5min":1.401,"15min":1.6719999999999999},"last":1.2649999999999999}]},{"osd":6,"last update":"Mon May 22 15:42:07 2023","interfaces":[{"interface":"back","average":{"1min":1.0089999999999999,"5min":1.028,"15min":1.0029999999999999},"min":{"1min":0.64500000000000002,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":1.351,"5min":2.2349999999999999,"15min":2.2349999999999999},"last":1.3620000000000001},{"interface":"front","average":{"1min":1.1379999999999999,"5min":1.1419999999999999,"15min":1.1160000000000001},"min":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.69799999999999995},"max":{"1min":1.486,"5min":1.8220000000000001,"15min":2.0699999999999998},"last":1.1160000000000001}]},{"osd":7,"last update":"Mon May 22 15:42:07 2023","interfaces":[{"interface":"back","average":{"1min":1.087,"5min":1.0580000000000001,"15min":1.0680000000000001},"min":{"1min":0.78200000000000003,"5min":0.71899999999999997,"15min":0.67700000000000005},"max":{"1min":1.3140000000000001,"5min":1.5800000000000001,"15min":1.6639999999999999},"last":1.446},{"interface":"front","average":{"1min":1.1930000000000001,"5min":1.1639999999999999,"15min":1.1459999999999999},"min":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.71499999999999997},"max":{"1min":1.4370000000000001,"5min":2.306,"15min":2.306},"last":1.413}]}]},{"osd":3,"up_from":13,"seq":55834575038,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34588,"kb_used_data":1044,"kb_used_omap":0,"kb_used_meta":33536,"kb_avail":94337252,"statfs":{"total":96636764160,"available":96601346048,"internally_reserved":0,"allocated":1069056,"data_stored":557224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":34340864},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 15:41:51 2023","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.63100000000000001,"15min":0.625},"min":{"1min":0.50700000000000001,"5min":0.19,"15min":0.19},"max":{"1min":0.79100000000000004,"5min":1.194,"15min":1.194},"last":0.69599999999999995},{"interface":"front","average":{"1min":0.79700000000000004,"5min":0.71999999999999997,"15min":0.69899999999999995},"min":{"1min":0.54700000000000004,"5min":0.32600000000000001,"15min":0.23699999999999999},"max":{"1min":0.97399999999999998,"5min":1.1799999999999999,"15min":1.296},"last":0.55700000000000005}]},{"osd":1,"last update":"Mon May 22 15:41:51 2023","interfaces":[{"interface":"back","average":{"1min":0.69499999999999995,"5min":0.64300000000000002,"15min":0.66300000000000003},"min":{"1min":0.44900000000000001,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":1.004,"5min":1.008,"15min":1.0600000000000001},"last":0.80000000000000004},{"interface":"front","average":{"1min":0.82599999999999996,"5min":0.77500000000000002,"15min":0.76700000000000002},"min":{"1min":0.57799999999999996,"5min":0.313,"15min":0.216},"max":{"1min":1.1359999999999999,"5min":1.222,"15min":1.6579999999999999},"last":0.45300000000000001}]},{"osd":2,"last update":"Mon May 22 15:41:51 2023","interfaces":[{"interface":"back","average":{"1min":0.85899999999999999,"5min":0.79800000000000004,"15min":0.81100000000000005},"min":{"1min":0.60899999999999999,"5min":0.36599999999999999,"15min":0.32200000000000001},"max":{"1min":1.157,"5min":1.157,"15min":1.2989999999999999},"last":1.506},{"interface":"front","average":{"1min":0.82099999999999995,"5min":0.80200000000000005,"15min":0.80800000000000005},"min":{"1min":0.60199999999999998,"5min":0.40000000000000002,"15min":0.29699999999999999},"max":{"1min":1.0900000000000001,"5min":1.181,"15min":1.855},"last":1.0740000000000001}]},{"osd":4,"last update":"Mon May 22 15:41:51 2023","interfaces":[{"interface":"back","average":{"1min":0.96999999999999997,"5min":0.94799999999999995,"15min":0.95599999999999996},"min":{"1min":0.80400000000000005,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":1.1970000000000001,"5min":1.2370000000000001,"15min":1.5820000000000001},"last":1.413},{"interface":"front","average":{"1min":1.0529999999999999,"5min":1.0009999999999999,"15min":1.0109999999999999},"min":{"1min":0.82099999999999995,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.238,"5min":1.3740000000000001,"15min":1.8069999999999999},"last":0.92800000000000005}]},{"osd":5,"last update":"Mon May 22 15:41:51 2023","interfaces":[{"interface":"back","average":{"1min":1.0549999999999999,"5min":1.0109999999999999,"15min":1.01},"min":{"1min":0.873,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":1.24,"5min":1.4059999999999999,"15min":1.4339999999999999},"last":1.2070000000000001},{"interface":"front","average":{"1min":1.0189999999999999,"5min":1.016,"15min":1.0489999999999999},"min":{"1min":0.80500000000000005,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":1.2410000000000001,"5min":1.381,"15min":2.4820000000000002},"last":1.3}]},{"osd":6,"last update":"Mon May 22 15:41:51 2023","interfaces":[{"interface":"back","average":{"1min":1.0680000000000001,"5min":1.0249999999999999,"15min":1.0409999999999999},"min":{"1min":0.873,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.254,"5min":1.4139999999999999,"15min":1.863},"last":1.0249999999999999},{"interface":"front","average":{"1min":1.0620000000000001,"5min":1.0640000000000001,"15min":1.0820000000000001},"min":{"1min":0.76600000000000001,"5min":0.66600000000000004,"15min":0.53500000000000003},"max":{"1min":1.25,"5min":1.4470000000000001,"15min":2.6589999999999998},"last":1.159}]},{"osd":7,"last update":"Mon May 22 15:41:51 2023","interfaces":[{"interface":"back","average":{"1min":1.105,"5min":1.085,"15min":1.111},"min":{"1min":0.84799999999999998,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":1.3089999999999999,"5min":1.4990000000000001,"15min":2.762},"last":1.2270000000000001},{"interface":"front","average":{"1min":1.0740000000000001,"5min":1.0549999999999999,"15min":1.077},"min":{"1min":0.71399999999999997,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":1.3799999999999999,"5min":1.3799999999999999,"15min":1.714},"last":1.8700000000000001}]}]},{"osd":2,"up_from":13,"seq":55834575039,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":16820,"kb_used_data":1068,"kb_used_omap":0,"kb_used_meta":15744,"kb_avail":94355020,"statfs":{"total":96636764160,"available":96619540480,"internally_reserved":0,"allocated":1093632,"data_stored":566220,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":16121856},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 15:41:55 2023","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.66000000000000003,"15min":0.65000000000000002},"min":{"1min":0.499,"5min":0.35699999999999998,"15min":0.317},"max":{"1min":0.96999999999999997,"5min":1.8240000000000001,"15min":1.8240000000000001},"last":0.83899999999999997},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.77600000000000002,"15min":0.74299999999999999},"min":{"1min":0.52500000000000002,"5min":0.39600000000000002,"15min":0.253},"max":{"1min":1.0740000000000001,"5min":1.8160000000000001,"15min":1.8160000000000001},"last":0.63100000000000001}]},{"osd":1,"last update":"Mon May 22 15:41:55 2023","interfaces":[{"interface":"back","average":{"1min":0.80600000000000005,"5min":0.74399999999999999,"15min":0.73599999999999999},"min":{"1min":0.46700000000000003,"5min":0.27000000000000002,"15min":0.26900000000000002},"max":{"1min":1.0329999999999999,"5min":1.2,"15min":1.5309999999999999},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.877,"5min":0.83899999999999997,"15min":0.80700000000000005},"min":{"1min":0.60099999999999998,"5min":0.39600000000000002,"15min":0.29099999999999998},"max":{"1min":1.246,"5min":2.5089999999999999,"15min":2.5089999999999999},"last":0.91700000000000004}]},{"osd":3,"last update":"Mon May 22 15:41:55 2023","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.72799999999999998,"15min":0.72099999999999997},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.26700000000000002},"max":{"1min":0.96199999999999997,"5min":2.7629999999999999,"15min":2.7629999999999999},"last":0.79200000000000004},{"interface":"front","average":{"1min":0.85699999999999998,"5min":0.89700000000000002,"15min":0.873},"min":{"1min":0.56100000000000005,"5min":0.44400000000000001,"15min":0.33000000000000002},"max":{"1min":1.042,"5min":2.6840000000000002,"15min":2.6840000000000002},"last":0.67800000000000005}]},{"osd":4,"last update":"Mon May 22 15:41:55 2023","interfaces":[{"interface":"back","average":{"1min":1.012,"5min":1.0329999999999999,"15min":1.0109999999999999},"min":{"1min":0.67700000000000005,"5min":0.65800000000000003,"15min":0.55500000000000005},"max":{"1min":1.244,"5min":2.8039999999999998,"15min":2.8039999999999998},"last":0.999},{"interface":"front","average":{"1min":1.0009999999999999,"5min":1.05,"15min":1.024},"min":{"1min":0.77000000000000002,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":1.3,"5min":2.0819999999999999,"15min":2.0819999999999999},"last":1.2210000000000001}]},{"osd":5,"last update":"Mon May 22 15:41:55 2023","interfaces":[{"interface":"back","average":{"1min":1.1080000000000001,"5min":1.0720000000000001,"15min":1.034},"min":{"1min":0.74099999999999999,"5min":0.629,"15min":0.55200000000000005},"max":{"1min":1.4099999999999999,"5min":2.3860000000000001,"15min":2.4569999999999999},"last":1.274},{"interface":"front","average":{"1min":0.98099999999999998,"5min":1.006,"15min":0.998},"min":{"1min":0.77800000000000002,"5min":0.64100000000000001,"15min":0.56200000000000006},"max":{"1min":1.327,"5min":3.0299999999999998,"15min":3.0299999999999998},"last":1.0269999999999999}]},{"osd":6,"last update":"Mon May 22 15:41:55 2023","interfaces":[{"interface":"back","average":{"1min":1.008,"5min":1.0349999999999999,"15min":1.0389999999999999},"min":{"1min":0.72899999999999998,"5min":0.67400000000000004,"15min":0.56399999999999995},"max":{"1min":1.2250000000000001,"5min":2.883,"15min":3.488},"last":0.88300000000000001},{"interface":"front","average":{"1min":1.0820000000000001,"5min":1.1160000000000001,"15min":1.1000000000000001},"min":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.53400000000000003},"max":{"1min":1.365,"5min":3.1459999999999999,"15min":3.7330000000000001},"last":1.1080000000000001}]},{"osd":7,"last update":"Mon May 22 15:41:55 2023","interfaces":[{"interface":"back","average":{"1min":1.149,"5min":1.1499999999999999,"15min":1.123},"min":{"1min":0.77600000000000002,"5min":0.77000000000000002,"15min":0.58299999999999996},"max":{"1min":1.4970000000000001,"5min":2.5990000000000002,"15min":2.5990000000000002},"last":1.1479999999999999},{"interface":"front","average":{"1min":1.0860000000000001,"5min":1.1200000000000001,"15min":1.0980000000000001},"min":{"1min":0.65500000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":1.3839999999999999,"5min":3.8799999999999999,"15min":3.8799999999999999},"last":0.96999999999999997}]}]},{"osd":1,"up_from":13,"seq":55834575039,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":23504,"kb_used_data":1032,"kb_used_omap":0,"kb_used_meta":22464,"kb_avail":94348336,"statfs":{"total":96636764160,"available":96612696064,"internally_reserved":0,"allocated":1056768,"data_stored":550276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":23003136},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 15:42:12 2023","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.63200000000000001,"15min":0.63800000000000001},"min":{"1min":0.37,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":1.2270000000000001,"5min":1.2270000000000001,"15min":1.7689999999999999},"last":0.26300000000000001},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.78200000000000003,"15min":0.77100000000000002},"min":{"1min":0.38800000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":1.0649999999999999,"5min":1.2290000000000001,"15min":1.3799999999999999},"last":0.29399999999999998}]},{"osd":2,"last update":"Mon May 22 15:42:12 2023","interfaces":[{"interface":"back","average":{"1min":0.78700000000000003,"5min":0.76700000000000002,"15min":0.73299999999999998},"min":{"1min":0.51300000000000001,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":1.391,"5min":1.391,"15min":1.7290000000000001},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.79100000000000004,"5min":0.77200000000000002,"15min":0.75600000000000001},"min":{"1min":0.47799999999999998,"5min":0.35899999999999999,"15min":0.27800000000000002},"max":{"1min":1.2270000000000001,"5min":1.276,"15min":2.4049999999999998},"last":0.36299999999999999}]},{"osd":3,"last update":"Mon May 22 15:42:12 2023","interfaces":[{"interface":"back","average":{"1min":0.88600000000000001,"5min":0.82199999999999995,"15min":0.82899999999999996},"min":{"1min":0.41499999999999998,"5min":0.41399999999999998,"15min":0.317},"max":{"1min":1.657,"5min":1.657,"15min":1.657},"last":0.33500000000000002},{"interface":"front","average":{"1min":0.89000000000000001,"5min":0.82399999999999995,"15min":0.82699999999999996},"min":{"1min":0.46000000000000002,"5min":0.315,"15min":0.315},"max":{"1min":2.0569999999999999,"5min":2.0569999999999999,"15min":7.5620000000000003},"last":0.441}]},{"osd":4,"last update":"Mon May 22 15:42:12 2023","interfaces":[{"interface":"back","average":{"1min":1.002,"5min":0.96999999999999997,"15min":0.95099999999999996},"min":{"1min":0.58499999999999996,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":2.0169999999999999,"5min":2.0169999999999999,"15min":2.0169999999999999},"last":0.65400000000000003},{"interface":"front","average":{"1min":1.03,"5min":0.98599999999999999,"15min":0.98899999999999999},"min":{"1min":0.64100000000000001,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":1.4650000000000001,"5min":1.5369999999999999,"15min":1.5369999999999999},"last":0.504}]},{"osd":5,"last update":"Mon May 22 15:42:12 2023","interfaces":[{"interface":"back","average":{"1min":1.069,"5min":1.0349999999999999,"15min":1.0329999999999999},"min":{"1min":0.59899999999999998,"5min":0.443,"15min":0.443},"max":{"1min":1.9570000000000001,"5min":1.9570000000000001,"15min":1.9570000000000001},"last":0.71699999999999997},{"interface":"front","average":{"1min":1.0720000000000001,"5min":1.0509999999999999,"15min":1.048},"min":{"1min":0.68000000000000005,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":2.1200000000000001,"5min":2.1200000000000001,"15min":4.4710000000000001},"last":0.53400000000000003}]},{"osd":6,"last update":"Mon May 22 15:42:12 2023","interfaces":[{"interface":"back","average":{"1min":1.0449999999999999,"5min":1.012,"15min":1.0680000000000001},"min":{"1min":0.54000000000000004,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.5720000000000001,"5min":1.5720000000000001,"15min":14.968999999999999},"last":0.69099999999999995},{"interface":"front","average":{"1min":1.1319999999999999,"5min":1.083,"15min":1.1359999999999999},"min":{"1min":0.69799999999999995,"5min":0.61399999999999999,"15min":0.56699999999999995},"max":{"1min":1.7629999999999999,"5min":1.7629999999999999,"15min":13.242000000000001},"last":0.54600000000000004}]},{"osd":7,"last update":"Mon May 22 15:42:12 2023","interfaces":[{"interface":"back","average":{"1min":1.1559999999999999,"5min":1.1140000000000001,"15min":1.109},"min":{"1min":0.66200000000000003,"5min":0.56799999999999995,"15min":0.56799999999999995},"max":{"1min":1.865,"5min":1.865,"15min":1.9259999999999999},"last":0.66900000000000004},{"interface":"front","average":{"1min":1.1739999999999999,"5min":1.145,"15min":1.131},"min":{"1min":0.78000000000000003,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":2.2200000000000002,"5min":2.2200000000000002,"15min":5.681},"last":0.57399999999999995}]}]},{"osd":0,"up_from":13,"seq":55834575040,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":22944,"kb_used_data":1624,"kb_used_omap":0,"kb_used_meta":21312,"kb_avail":94348896,"statfs":{"total":96636764160,"available":96613269504,"internally_reserved":0,"allocated":1662976,"data_stored":1147592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":21823488},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon May 22 15:42:07 2023","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.66100000000000003,"15min":0.66800000000000004},"min":{"1min":0.248,"5min":0.223,"15min":0.223},"max":{"1min":0.86899999999999999,"5min":1.077,"15min":1.3049999999999999},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.76300000000000001,"5min":0.72999999999999998,"15min":0.73299999999999998},"min":{"1min":0.53600000000000003,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":1.0369999999999999,"5min":1.2390000000000001,"15min":1.2390000000000001},"last":0.70799999999999996}]},{"osd":2,"last update":"Mon May 22 15:42:07 2023","interfaces":[{"interface":"back","average":{"1min":0.73599999999999999,"5min":0.74199999999999999,"15min":0.74399999999999999},"min":{"1min":0.318,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":1.155,"5min":1.155,"15min":1.4059999999999999},"last":1.004},{"interface":"front","average":{"1min":0.79500000000000004,"5min":0.78000000000000003,"15min":0.79900000000000004},"min":{"1min":0.55400000000000005,"5min":0.32900000000000001,"15min":0.29699999999999999},"max":{"1min":1.05,"5min":1.1850000000000001,"15min":1.3999999999999999},"last":0.81999999999999995}]},{"osd":3,"last update":"Mon May 22 15:42:07 2023","interfaces":[{"interface":"back","average":{"1min":0.86699999999999999,"5min":0.755,"15min":0.76000000000000001},"min":{"1min":0.55600000000000005,"5min":0.40899999999999997,"15min":0.36599999999999999},"max":{"1min":1.9299999999999999,"5min":1.9299999999999999,"15min":1.9299999999999999},"last":0.76300000000000001},{"interface":"front","average":{"1min":0.85799999999999998,"5min":0.86399999999999999,"15min":0.86799999999999999},"min":{"1min":0.35199999999999998,"5min":0.34399999999999997,"15min":0.33000000000000002},"max":{"1min":1.0669999999999999,"5min":1.3500000000000001,"15min":1.4570000000000001},"last":0.84399999999999997}]},{"osd":4,"last update":"Mon May 22 15:42:07 2023","interfaces":[{"interface":"back","average":{"1min":1.0429999999999999,"5min":0.94899999999999995,"15min":0.94799999999999995},"min":{"1min":0.751,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":2.923,"5min":2.923,"15min":2.923},"last":0.94699999999999995},{"interface":"front","average":{"1min":1.073,"5min":1.0269999999999999,"15min":1.022},"min":{"1min":0.86399999999999999,"5min":0.64600000000000002,"15min":0.54900000000000004},"max":{"1min":1.8160000000000001,"5min":1.8160000000000001,"15min":1.8160000000000001},"last":1.1830000000000001}]},{"osd":5,"last update":"Mon May 22 15:42:07 2023","interfaces":[{"interface":"back","average":{"1min":0.94699999999999995,"5min":1.014,"15min":1.038},"min":{"1min":0.66500000000000004,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.1040000000000001,"5min":1.4750000000000001,"15min":1.6850000000000001},"last":0.90100000000000002},{"interface":"front","average":{"1min":1.2170000000000001,"5min":1.032,"15min":1.01},"min":{"1min":0.76800000000000002,"5min":0.55300000000000005,"15min":0.54800000000000004},"max":{"1min":5.0519999999999996,"5min":5.0519999999999996,"15min":5.0519999999999996},"last":1.0329999999999999}]},{"osd":6,"last update":"Mon May 22 15:42:07 2023","interfaces":[{"interface":"back","average":{"1min":1.038,"5min":1.0740000000000001,"15min":1.075},"min":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":1.53,"5min":1.651,"15min":1.7450000000000001},"last":1.097},{"interface":"front","average":{"1min":1.2749999999999999,"5min":1.139,"15min":1.103},"min":{"1min":0.91500000000000004,"5min":0.499,"15min":0.499},"max":{"1min":5.2640000000000002,"5min":5.2640000000000002,"15min":5.2640000000000002},"last":1.071}]},{"osd":7,"last update":"Mon May 22 15:42:07 2023","interfaces":[{"interface":"back","average":{"1min":1.0700000000000001,"5min":1.1120000000000001,"15min":1.1180000000000001},"min":{"1min":0.70699999999999996,"5min":0.65700000000000003,"15min":0.56499999999999995},"max":{"1min":1.2789999999999999,"5min":1.6419999999999999,"15min":1.6419999999999999},"last":0.96999999999999997},{"interface":"front","average":{"1min":1.228,"5min":1.1000000000000001,"15min":1.087},"min":{"1min":0.85999999999999999,"5min":0.60299999999999998,"15min":0.58399999999999996},"max":{"1min":3.9060000000000001,"5min":3.9060000000000001,"15min":3.9060000000000001},"last":1.1419999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2148,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-05-22T15:42:27.226 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-05-22T15:42:27.226 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2023-05-22T15:42:27.399 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.396+0000 7f77bdad8640 1 Processor -- start 2023-05-22T15:42:27.400 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.396+0000 7f77bdad8640 1 -- start start 2023-05-22T15:42:27.400 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.396+0000 7f77bdad8640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f77b80c8d70 0x7f77b815c120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:27.401 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.396+0000 7f77bdad8640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77b8158cd0 con 0x7f77b815bcd0 2023-05-22T15:42:27.401 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.396+0000 7f77bdad8640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77b815c790 con 0x7f77b8158210 2023-05-22T15:42:27.401 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.396+0000 7f77bdad8640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77b815ca70 con 0x7f77b80c8d70 2023-05-22T15:42:27.401 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.396+0000 7f77b6ffd640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f77b815bcd0 0x7f77b80c8660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:33480/0 (socket says 172.21.15.16:33480) 2023-05-22T15:42:27.401 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.396+0000 7f77b6ffd640 1 -- 172.21.15.16:0/2506606009 learned_addr learned my addr 172.21.15.16:0/2506606009 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:42:27.403 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.396+0000 7f77b7fff640 1 --2- 172.21.15.16:0/2506606009 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f77b80c8d70 0x7f77b815c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:27.403 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.396+0000 7f77b67fc640 1 -- 172.21.15.16:0/2506606009 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 406047168 0 0) 0x7f77b8158cd0 con 0x7f77b815bcd0 2023-05-22T15:42:27.403 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.396+0000 7f77b67fc640 1 -- 172.21.15.16:0/2506606009 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f77a0003530 con 0x7f77b815bcd0 2023-05-22T15:42:27.403 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.396+0000 7f77b67fc640 1 -- 172.21.15.16:0/2506606009 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2759524668 0 0) 0x7f77a0003530 con 0x7f77b815bcd0 2023-05-22T15:42:27.403 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.396+0000 7f77b67fc640 1 -- 172.21.15.16:0/2506606009 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f77b80c8d70 msgr2=0x7f77b815c120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:42:27.403 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.396+0000 7f77b67fc640 1 --2- 172.21.15.16:0/2506606009 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f77b80c8d70 0x7f77b815c120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:27.404 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.396+0000 7f77b67fc640 1 -- 172.21.15.16:0/2506606009 >> v1:172.21.15.62:6789/0 conn(0x7f77b8158210 legacy=0x7f77b8159160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:27.404 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.396+0000 7f77b67fc640 1 -- 172.21.15.16:0/2506606009 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77b8158960 con 0x7f77b815bcd0 2023-05-22T15:42:27.404 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.396+0000 7f77b67fc640 1 -- 172.21.15.16:0/2506606009 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7f77ac001680 con 0x7f77b815bcd0 2023-05-22T15:42:27.404 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.396+0000 7f77b67fc640 1 -- 172.21.15.16:0/2506606009 >> v1:172.21.15.16:6789/0 conn(0x7f77b815bcd0 legacy=0x7f77b80c8660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:27.404 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.396+0000 7f77b67fc640 1 --2- 172.21.15.16:0/2506606009 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f77a00039b0 0x7f77a0003d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:27.404 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.396+0000 7f77b67fc640 1 --2- 172.21.15.16:0/2506606009 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f77b80c8d70 0x7f77a00047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:27.404 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.396+0000 7f77b67fc640 1 --2- 172.21.15.16:0/2506606009 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f77a0004d10 0x7f77a0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:27.404 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.396+0000 7f77b67fc640 1 -- 172.21.15.16:0/2506606009 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77b815ca70 con 0x7f77a00039b0 2023-05-22T15:42:27.405 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.396+0000 7f77b67fc640 1 -- 172.21.15.16:0/2506606009 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77a000dc50 con 0x7f77a0004d10 2023-05-22T15:42:27.405 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.396+0000 7f77b67fc640 1 -- 172.21.15.16:0/2506606009 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77a000de10 con 0x7f77b80c8d70 2023-05-22T15:42:27.405 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.396+0000 7f77b77fe640 1 --2- 172.21.15.16:0/2506606009 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f77a00039b0 0x7f77a0003d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:27.405 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.400+0000 7f77b6ffd640 1 --2- 172.21.15.16:0/2506606009 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f77b80c8d70 0x7f77a00047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:27.405 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.400+0000 7f77b7fff640 1 --2- 172.21.15.16:0/2506606009 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f77a0004d10 0x7f77a0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:27.405 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.400+0000 7f77b77fe640 1 -- 172.21.15.16:0/2506606009 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f77b80c8d70 msgr2=0x7f77a00047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:42:27.405 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.400+0000 7f77b77fe640 1 --2- 172.21.15.16:0/2506606009 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f77b80c8d70 0x7f77a00047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:27.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.400+0000 7f77b77fe640 1 -- 172.21.15.16:0/2506606009 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f77a0004d10 msgr2=0x7f77a0005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:42:27.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.400+0000 7f77b77fe640 1 --2- 172.21.15.16:0/2506606009 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f77a0004d10 0x7f77a0005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:27.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.400+0000 7f77b77fe640 1 -- 172.21.15.16:0/2506606009 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77a0003530 con 0x7f77a00039b0 2023-05-22T15:42:27.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.400+0000 7f77b7fff640 1 --2- 172.21.15.16:0/2506606009 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f77a0004d10 0x7f77a0005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:42:27.408 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.400+0000 7f77b77fe640 1 --2- 172.21.15.16:0/2506606009 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f77a00039b0 0x7f77a0003d80 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 crypto rx=0x7f77a8001860 tx=0x7f77a801c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4812db7b7bc4f4ba server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:27.408 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.400+0000 7f77b67fc640 1 -- 172.21.15.16:0/2506606009 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77a80ab600 con 0x7f77a00039b0 2023-05-22T15:42:27.408 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.400+0000 7f77b67fc640 1 -- 172.21.15.16:0/2506606009 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77a80a7050 con 0x7f77a00039b0 2023-05-22T15:42:27.408 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.400+0000 7f77b67fc640 1 -- 172.21.15.16:0/2506606009 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77a80ab9a0 con 0x7f77a00039b0 2023-05-22T15:42:27.409 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.400+0000 7f77bdad8640 1 -- 172.21.15.16:0/2506606009 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f77a00039b0 msgr2=0x7f77a0003d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:27.409 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.400+0000 7f77bdad8640 1 --2- 172.21.15.16:0/2506606009 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f77a00039b0 0x7f77a0003d80 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 crypto rx=0x7f77a8001860 tx=0x7f77a801c500 comp rx=0 tx=0).stop 2023-05-22T15:42:27.409 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.400+0000 7f77b7fff640 1 -- 172.21.15.16:0/2506606009 reap_dead start 2023-05-22T15:42:27.409 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.400+0000 7f77bdad8640 1 -- 172.21.15.16:0/2506606009 shutdown_connections 2023-05-22T15:42:27.409 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.400+0000 7f77bdad8640 1 -- 172.21.15.16:0/2506606009 >> 172.21.15.16:0/2506606009 conn(0x7f77b8069720 msgr2=0x7f77b8069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:27.409 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.400+0000 7f77bdad8640 1 -- 172.21.15.16:0/2506606009 shutdown_connections 2023-05-22T15:42:27.409 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.400+0000 7f77bdad8640 1 -- 172.21.15.16:0/2506606009 wait complete. 2023-05-22T15:42:27.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.404+0000 7f77bdad8640 1 Processor -- start 2023-05-22T15:42:27.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.404+0000 7f77bdad8640 1 -- start start 2023-05-22T15:42:27.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.404+0000 7f77bdad8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f77a00039b0 0x7f77b80c7f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:27.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.404+0000 7f77bdad8640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f77b8156a40 0x7f77b8153bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:27.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.404+0000 7f77bdad8640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f77b80c3da0 0x7f77b80c4210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:27.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.404+0000 7f77bdad8640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77b80c4750 con 0x7f77a00039b0 2023-05-22T15:42:27.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.404+0000 7f77bdad8640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77b80c8840 con 0x7f77b80c3da0 2023-05-22T15:42:27.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.404+0000 7f77bdad8640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77b81555b0 con 0x7f77b8156a40 2023-05-22T15:42:27.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.404+0000 7f77b77fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f77a00039b0 0x7f77b80c7f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:27.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.404+0000 7f77b77fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f77a00039b0 0x7f77b80c7f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34902/0 (socket says 172.21.15.16:34902) 2023-05-22T15:42:27.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.404+0000 7f77b77fe640 1 -- 172.21.15.16:0/1004972635 learned_addr learned my addr 172.21.15.16:0/1004972635 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:42:27.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.404+0000 7f77b6ffd640 1 --2- 172.21.15.16:0/1004972635 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f77b8156a40 0x7f77b8153bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:27.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.404+0000 7f77b7fff640 1 --2- 172.21.15.16:0/1004972635 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f77b80c3da0 0x7f77b80c4210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:27.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.404+0000 7f77b77fe640 1 -- 172.21.15.16:0/1004972635 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f77b8156a40 msgr2=0x7f77b8153bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:27.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.404+0000 7f77b77fe640 1 --2- 172.21.15.16:0/1004972635 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f77b8156a40 0x7f77b8153bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:27.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.404+0000 7f77b77fe640 1 -- 172.21.15.16:0/1004972635 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f77b80c3da0 msgr2=0x7f77b80c4210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:27.412 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.404+0000 7f77b77fe640 1 --2- 172.21.15.16:0/1004972635 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f77b80c3da0 0x7f77b80c4210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:27.412 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.404+0000 7f77b77fe640 1 -- 172.21.15.16:0/1004972635 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77b80ca820 con 0x7f77a00039b0 2023-05-22T15:42:27.412 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.404+0000 7f77b7fff640 1 --2- 172.21.15.16:0/1004972635 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f77b80c3da0 0x7f77b80c4210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:42:27.412 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.404+0000 7f77b77fe640 1 --2- 172.21.15.16:0/1004972635 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f77a00039b0 0x7f77b80c7f80 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 crypto rx=0x7f77a801c330 tx=0x7f77a0005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:27.412 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.404+0000 7f779ffff640 1 -- 172.21.15.16:0/1004972635 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77a8005390 con 0x7f77a00039b0 2023-05-22T15:42:27.412 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.404+0000 7f779ffff640 1 -- 172.21.15.16:0/1004972635 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77a80aba80 con 0x7f77a00039b0 2023-05-22T15:42:27.412 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.404+0000 7f779ffff640 1 -- 172.21.15.16:0/1004972635 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77a80abdf0 con 0x7f77a00039b0 2023-05-22T15:42:27.413 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.404+0000 7f77bdad8640 1 -- 172.21.15.16:0/1004972635 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77b80ca690 con 0x7f77a00039b0 2023-05-22T15:42:27.413 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.404+0000 7f77bdad8640 1 -- 172.21.15.16:0/1004972635 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77b80ca9b0 con 0x7f77a00039b0 2023-05-22T15:42:27.413 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.404+0000 7f77bdad8640 1 -- 172.21.15.16:0/1004972635 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77b804dbe0 con 0x7f77a00039b0 2023-05-22T15:42:27.414 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.404+0000 7f779ffff640 1 -- 172.21.15.16:0/1004972635 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f77a80b2050 con 0x7f77a00039b0 2023-05-22T15:42:27.414 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.404+0000 7f779ffff640 1 --2- 172.21.15.16:0/1004972635 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f77a40654a0 0x7f77a4067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:27.414 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.408+0000 7f77b6ffd640 1 --2- 172.21.15.16:0/1004972635 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f77a40654a0 0x7f77a4067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:27.414 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.408+0000 7f779ffff640 1 -- 172.21.15.16:0/1004972635 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f77a811c790 con 0x7f77a00039b0 2023-05-22T15:42:27.414 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.408+0000 7f77b6ffd640 1 --2- 172.21.15.16:0/1004972635 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f77a40654a0 0x7f77a4067950 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f77ac001210 tx=0x7f77ac009040 comp rx=0 tx=0).ready entity=mgr.6765 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:27.414 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.408+0000 7f779ffff640 1 -- 172.21.15.16:0/1004972635 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f77a80ea1f0 con 0x7f77a00039b0 2023-05-22T15:42:27.623 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.620+0000 7f77bdad8640 1 -- 172.21.15.16:0/1004972635 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f77b806c480 con 0x7f77a00039b0 2023-05-22T15:42:27.632 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.628+0000 7f779ffff640 1 -- 172.21.15.16:0/1004972635 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v19) v1 ==== 190+0+0 (secure 0 0 0) 0x7f77a80ed810 con 0x7f77a00039b0 2023-05-22T15:42:27.633 INFO:teuthology.orchestra.run.smithi016.stderr:updated caps for client.0 2023-05-22T15:42:27.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.632+0000 7f77bdad8640 1 -- 172.21.15.16:0/1004972635 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f77a40654a0 msgr2=0x7f77a4067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:27.635 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.632+0000 7f77bdad8640 1 --2- 172.21.15.16:0/1004972635 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f77a40654a0 0x7f77a4067950 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f77ac001210 tx=0x7f77ac009040 comp rx=0 tx=0).stop 2023-05-22T15:42:27.635 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.632+0000 7f77bdad8640 1 -- 172.21.15.16:0/1004972635 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f77a00039b0 msgr2=0x7f77b80c7f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:27.635 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.632+0000 7f77bdad8640 1 --2- 172.21.15.16:0/1004972635 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f77a00039b0 0x7f77b80c7f80 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 crypto rx=0x7f77a801c330 tx=0x7f77a0005800 comp rx=0 tx=0).stop 2023-05-22T15:42:27.635 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.632+0000 7f77bdad8640 1 -- 172.21.15.16:0/1004972635 shutdown_connections 2023-05-22T15:42:27.635 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.632+0000 7f77bdad8640 1 --2- 172.21.15.16:0/1004972635 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f77b80c3da0 0x7f77b80c4210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:27.635 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.632+0000 7f77bdad8640 1 --2- 172.21.15.16:0/1004972635 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f77a40654a0 0x7f77a4067950 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:27.636 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.632+0000 7f77bdad8640 1 --2- 172.21.15.16:0/1004972635 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f77b8156a40 0x7f77b8153bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:27.636 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.632+0000 7f77bdad8640 1 --2- 172.21.15.16:0/1004972635 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f77a00039b0 0x7f77b80c7f80 unknown :-1 s=CLOSED pgs=1118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:27.636 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.632+0000 7f77bdad8640 1 -- 172.21.15.16:0/1004972635 >> 172.21.15.16:0/1004972635 conn(0x7f77b8069720 msgr2=0x7f77b8157110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:27.636 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.632+0000 7f77bdad8640 1 -- 172.21.15.16:0/1004972635 shutdown_connections 2023-05-22T15:42:27.636 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.632+0000 7f77bdad8640 1 -- 172.21.15.16:0/1004972635 wait complete. 2023-05-22T15:42:27.652 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-05-22T15:42:27.771 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.768+0000 7f361517f640 1 Processor -- start 2023-05-22T15:42:27.771 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.768+0000 7f361517f640 1 -- start start 2023-05-22T15:42:27.772 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.768+0000 7f361517f640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f3610071500 0x7f361015e460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:27.773 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.768+0000 7f361517f640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36100b0650 con 0x7f36100c9b40 2023-05-22T15:42:27.773 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.768+0000 7f361517f640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36100b0370 con 0x7f36100ca4e0 2023-05-22T15:42:27.773 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.768+0000 7f361517f640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f361015d980 con 0x7f3610071500 2023-05-22T15:42:27.773 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.768+0000 7f360f59e640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f3610071500 0x7f361015e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:27.774 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.768+0000 7f360e59c640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f36100c9b40 0x7f3610070df0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:33496/0 (socket says 172.21.15.16:33496) 2023-05-22T15:42:27.774 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.768+0000 7f360e59c640 1 -- 172.21.15.16:0/2421169762 learned_addr learned my addr 172.21.15.16:0/2421169762 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:42:27.774 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.768+0000 7f360f59e640 1 -- 172.21.15.16:0/2421169762 >> v1:172.21.15.62:6789/0 conn(0x7f36100ca4e0 legacy=0x7f36100afb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:27.774 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.768+0000 7f360f59e640 1 -- 172.21.15.16:0/2421169762 >> v1:172.21.15.16:6789/0 conn(0x7f36100c9b40 legacy=0x7f3610070df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:27.774 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.768+0000 7f360f59e640 1 -- 172.21.15.16:0/2421169762 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f361015dc60 con 0x7f3610071500 2023-05-22T15:42:27.774 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.768+0000 7f360dd9b640 1 -- 172.21.15.16:0/2421169762 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1443900312 0 0) 0x7f36100b0650 con 0x7f36100c9b40 2023-05-22T15:42:27.775 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.772+0000 7f360f59e640 1 --2- 172.21.15.16:0/2421169762 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3610071500 0x7f361015e460 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 crypto rx=0x7f3608009d90 tx=0x7f360801c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d91033c40944b90 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:27.775 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.772+0000 7f360dd9b640 1 -- 172.21.15.16:0/2421169762 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36080a5900 con 0x7f3610071500 2023-05-22T15:42:27.775 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.772+0000 7f360dd9b640 1 -- 172.21.15.16:0/2421169762 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36080a5a60 con 0x7f3610071500 2023-05-22T15:42:27.775 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.772+0000 7f360dd9b640 1 -- 172.21.15.16:0/2421169762 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36080a5df0 con 0x7f3610071500 2023-05-22T15:42:27.776 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.772+0000 7f361517f640 1 -- 172.21.15.16:0/2421169762 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3610071500 msgr2=0x7f361015e460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:27.777 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.772+0000 7f361517f640 1 --2- 172.21.15.16:0/2421169762 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3610071500 0x7f361015e460 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 crypto rx=0x7f3608009d90 tx=0x7f360801c920 comp rx=0 tx=0).stop 2023-05-22T15:42:27.777 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.772+0000 7f361517f640 1 -- 172.21.15.16:0/2421169762 shutdown_connections 2023-05-22T15:42:27.777 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.772+0000 7f361517f640 1 --2- 172.21.15.16:0/2421169762 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3610071500 0x7f361015e460 unknown :-1 s=CLOSED pgs=1120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:27.777 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.772+0000 7f361517f640 1 -- 172.21.15.16:0/2421169762 >> 172.21.15.16:0/2421169762 conn(0x7f3610069720 msgr2=0x7f3610069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:27.777 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.772+0000 7f361517f640 1 -- 172.21.15.16:0/2421169762 shutdown_connections 2023-05-22T15:42:27.777 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.772+0000 7f361517f640 1 -- 172.21.15.16:0/2421169762 wait complete. 2023-05-22T15:42:27.779 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.772+0000 7f361517f640 1 Processor -- start 2023-05-22T15:42:27.779 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.772+0000 7f361517f640 1 -- start start 2023-05-22T15:42:27.779 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.772+0000 7f361517f640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3610071500 0x7f361015a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:27.780 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.772+0000 7f361517f640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f36100c9b40 0x7f36100c67b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:27.780 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.772+0000 7f361517f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f36100ca4e0 0x7f3610155840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:27.780 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.772+0000 7f361517f640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36100c5710 con 0x7f36100ca4e0 2023-05-22T15:42:27.780 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.772+0000 7f361517f640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36100c59f0 con 0x7f36100c9b40 2023-05-22T15:42:27.780 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.772+0000 7f361517f640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36100c5cd0 con 0x7f3610071500 2023-05-22T15:42:27.780 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.776+0000 7f360f59e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f36100ca4e0 0x7f3610155840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:27.781 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.776+0000 7f360f59e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f36100ca4e0 0x7f3610155840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34916/0 (socket says 172.21.15.16:34916) 2023-05-22T15:42:27.781 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.776+0000 7f360f59e640 1 -- 172.21.15.16:0/1226750012 learned_addr learned my addr 172.21.15.16:0/1226750012 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:42:27.781 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.776+0000 7f360ed9d640 1 --2- 172.21.15.16:0/1226750012 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3610071500 0x7f361015a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:27.781 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.776+0000 7f360e59c640 1 --2- 172.21.15.16:0/1226750012 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f36100c9b40 0x7f36100c67b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:27.781 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.776+0000 7f360f59e640 1 -- 172.21.15.16:0/1226750012 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3610071500 msgr2=0x7f361015a2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:27.781 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.776+0000 7f360f59e640 1 --2- 172.21.15.16:0/1226750012 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3610071500 0x7f361015a2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:27.781 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.776+0000 7f360f59e640 1 -- 172.21.15.16:0/1226750012 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f36100c9b40 msgr2=0x7f36100c67b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:27.782 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.776+0000 7f360f59e640 1 --2- 172.21.15.16:0/1226750012 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f36100c9b40 0x7f36100c67b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:27.782 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.776+0000 7f360f59e640 1 -- 172.21.15.16:0/1226750012 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3608009a70 con 0x7f36100ca4e0 2023-05-22T15:42:27.782 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.776+0000 7f360ed9d640 1 --2- 172.21.15.16:0/1226750012 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3610071500 0x7f361015a2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T15:42:27.782 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.776+0000 7f360f59e640 1 --2- 172.21.15.16:0/1226750012 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f36100ca4e0 0x7f3610155840 secure :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 crypto rx=0x7f3608019530 tx=0x7f36080196c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:27.782 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.776+0000 7f35f77fe640 1 -- 172.21.15.16:0/1226750012 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3608003ba0 con 0x7f36100ca4e0 2023-05-22T15:42:27.783 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.776+0000 7f35f77fe640 1 -- 172.21.15.16:0/1226750012 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3608003d00 con 0x7f36100ca4e0 2023-05-22T15:42:27.783 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.776+0000 7f361517f640 1 -- 172.21.15.16:0/1226750012 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36100c9010 con 0x7f36100ca4e0 2023-05-22T15:42:27.783 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.776+0000 7f35f77fe640 1 -- 172.21.15.16:0/1226750012 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36080ad910 con 0x7f36100ca4e0 2023-05-22T15:42:27.783 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.776+0000 7f361517f640 1 -- 172.21.15.16:0/1226750012 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36100c8e80 con 0x7f36100ca4e0 2023-05-22T15:42:27.784 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.776+0000 7f361517f640 1 -- 172.21.15.16:0/1226750012 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36100c91a0 con 0x7f36100ca4e0 2023-05-22T15:42:27.784 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.776+0000 7f35f77fe640 1 -- 172.21.15.16:0/1226750012 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f36080b4070 con 0x7f36100ca4e0 2023-05-22T15:42:27.784 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.776+0000 7f35f77fe640 1 --2- 172.21.15.16:0/1226750012 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f35fc0654a0 0x7f35fc067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:27.784 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.776+0000 7f35f77fe640 1 -- 172.21.15.16:0/1226750012 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f360811dd70 con 0x7f36100ca4e0 2023-05-22T15:42:27.785 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.776+0000 7f360ed9d640 1 --2- 172.21.15.16:0/1226750012 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f35fc0654a0 0x7f35fc067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:27.785 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.776+0000 7f360ed9d640 1 --2- 172.21.15.16:0/1226750012 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f35fc0654a0 0x7f35fc067950 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f3600000f70 tx=0x7f3600009040 comp rx=0 tx=0).ready entity=mgr.6765 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:27.785 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.780+0000 7f35f77fe640 1 -- 172.21.15.16:0/1226750012 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f36080ef340 con 0x7f36100ca4e0 2023-05-22T15:42:27.963 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.960+0000 7f361517f640 1 -- 172.21.15.16:0/1226750012 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f361004dbe0 con 0x7f36100ca4e0 2023-05-22T15:42:27.964 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.960+0000 7f35f77fe640 1 -- 172.21.15.16:0/1226750012 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+4647 (secure 0 0 0) 0x7f36080eef30 con 0x7f36100ca4e0 2023-05-22T15:42:27.965 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:42:27.965 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":30,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6833/428835470","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":428835470},{"type":"v1","addr":"172.21.15.62:6833","nonce":428835470}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":25},{"gid":6238,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/4010598958","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":4010598958},{"type":"v1","addr":"172.21.15.62:6835","nonce":4010598958}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":25},{"gid":7008,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/2240458030","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6835","nonce":2240458030},{"type":"v1","addr":"172.21.15.16:6837","nonce":2240458030}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":25}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-05-22T15:42:22.811659+0000","modified":"2023-05-22T15:42:25.830716+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7011},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7011":{"gid":7011,"name":"a","rank":0,"incarnation":26,"state":"up:active","state_seq":3,"addr":"172.21.15.16:6836/1772031282","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1772031282},{"type":"v1","addr":"172.21.15.16:6836","nonce":1772031282}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2023-05-22T15:42:27.965 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 30 2023-05-22T15:42:27.967 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.964+0000 7f361517f640 1 -- 172.21.15.16:0/1226750012 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f35fc0654a0 msgr2=0x7f35fc067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:27.967 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.964+0000 7f361517f640 1 --2- 172.21.15.16:0/1226750012 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f35fc0654a0 0x7f35fc067950 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f3600000f70 tx=0x7f3600009040 comp rx=0 tx=0).stop 2023-05-22T15:42:27.967 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.964+0000 7f361517f640 1 -- 172.21.15.16:0/1226750012 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f36100ca4e0 msgr2=0x7f3610155840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:27.967 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.964+0000 7f361517f640 1 --2- 172.21.15.16:0/1226750012 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f36100ca4e0 0x7f3610155840 secure :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 crypto rx=0x7f3608019530 tx=0x7f36080196c0 comp rx=0 tx=0).stop 2023-05-22T15:42:27.967 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.964+0000 7f361517f640 1 -- 172.21.15.16:0/1226750012 shutdown_connections 2023-05-22T15:42:27.968 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.964+0000 7f361517f640 1 --2- 172.21.15.16:0/1226750012 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f36100ca4e0 0x7f3610155840 unknown :-1 s=CLOSED pgs=1121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:27.968 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.964+0000 7f361517f640 1 --2- 172.21.15.16:0/1226750012 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f36100c9b40 0x7f36100c67b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:27.968 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.964+0000 7f361517f640 1 --2- 172.21.15.16:0/1226750012 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f35fc0654a0 0x7f35fc067950 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:27.968 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.964+0000 7f361517f640 1 --2- 172.21.15.16:0/1226750012 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3610071500 0x7f361015a2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:27.968 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.964+0000 7f361517f640 1 -- 172.21.15.16:0/1226750012 >> 172.21.15.16:0/1226750012 conn(0x7f3610069720 msgr2=0x7f36100accd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:27.968 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.964+0000 7f361517f640 1 -- 172.21.15.16:0/1226750012 shutdown_connections 2023-05-22T15:42:27.968 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:27.964+0000 7f361517f640 1 -- 172.21.15.16:0/1226750012 wait complete. 2023-05-22T15:42:28.384 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 30, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-05-22T15:42:22.811659+0000', 'modified': '2023-05-22T15:42:25.830716+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7011}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7011': {'gid': 7011, 'name': 'a', 'rank': 0, 'incarnation': 26, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.16:6836/1772031282', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6834', 'nonce': 1772031282}, {'type': 'v1', 'addr': '172.21.15.16:6836', 'nonce': 1772031282}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [11], 'metadata_pool': 10, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-05-22T15:42:28.385 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-05-22T15:42:28.385 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2023-05-22T15:42:28.518 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.512+0000 7fd2a356f640 1 Processor -- start 2023-05-22T15:42:28.518 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.512+0000 7fd2a356f640 1 -- start start 2023-05-22T15:42:28.519 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.516+0000 7fd2a356f640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fd29c0b0250 0x7fd29c159330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:28.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.516+0000 7fd2a356f640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd29c159870 0x7fd29c0c8ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:28.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.516+0000 7fd2a356f640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd29c15a0a0 con 0x7fd29c1531b0 2023-05-22T15:42:28.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.516+0000 7fd2a356f640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd29c153580 con 0x7fd29c159870 2023-05-22T15:42:28.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.516+0000 7fd2a356f640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd29c15d5e0 con 0x7fd29c0b0250 2023-05-22T15:42:28.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.516+0000 7fd2a130b640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fd29c1531b0 0x7fd29c0afb40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:33498/0 (socket says 172.21.15.16:33498) 2023-05-22T15:42:28.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.516+0000 7fd2a130b640 1 -- 172.21.15.16:0/1606890356 learned_addr learned my addr 172.21.15.16:0/1606890356 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:42:28.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.516+0000 7fd2a1b0c640 1 --2- 172.21.15.16:0/1606890356 >> v2:172.21.15.16:3300/0 conn(0x7fd29c159870 0x7fd29c0c8ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:28.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.516+0000 7fd2a0b0a640 1 --2- 172.21.15.16:0/1606890356 >> v2:172.21.15.62:3300/0 conn(0x7fd29c0b0250 0x7fd29c159330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:28.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.516+0000 7fd293fff640 1 -- 172.21.15.16:0/1606890356 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2844628578 0 0) 0x7fd29c15a0a0 con 0x7fd29c1531b0 2023-05-22T15:42:28.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.516+0000 7fd293fff640 1 -- 172.21.15.16:0/1606890356 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd288003530 con 0x7fd29c1531b0 2023-05-22T15:42:28.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.516+0000 7fd2a1b0c640 1 -- 172.21.15.16:0/1606890356 >> v2:172.21.15.62:3300/0 conn(0x7fd29c0b0250 msgr2=0x7fd29c159330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:42:28.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.516+0000 7fd2a1b0c640 1 --2- 172.21.15.16:0/1606890356 >> v2:172.21.15.62:3300/0 conn(0x7fd29c0b0250 0x7fd29c159330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:28.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.516+0000 7fd2a1b0c640 1 -- 172.21.15.16:0/1606890356 >> v1:172.21.15.16:6789/0 conn(0x7fd29c1531b0 legacy=0x7fd29c0afb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:28.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.516+0000 7fd2a1b0c640 1 -- 172.21.15.16:0/1606890356 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd29c15c820 con 0x7fd29c159870 2023-05-22T15:42:28.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.516+0000 7fd2a1b0c640 1 --2- 172.21.15.16:0/1606890356 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd29c159870 0x7fd29c0c8ba0 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 crypto rx=0x7fd29c06d880 tx=0x7fd29401c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=183069a1440c87a6 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:28.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.516+0000 7fd293fff640 1 -- 172.21.15.16:0/1606890356 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2940a59a0 con 0x7fd29c159870 2023-05-22T15:42:28.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.516+0000 7fd293fff640 1 -- 172.21.15.16:0/1606890356 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2940a5b00 con 0x7fd29c159870 2023-05-22T15:42:28.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.516+0000 7fd293fff640 1 -- 172.21.15.16:0/1606890356 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2940a5e90 con 0x7fd29c159870 2023-05-22T15:42:28.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.520+0000 7fd2a356f640 1 -- 172.21.15.16:0/1606890356 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd29c159870 msgr2=0x7fd29c0c8ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:28.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.520+0000 7fd2a356f640 1 --2- 172.21.15.16:0/1606890356 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd29c159870 0x7fd29c0c8ba0 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 crypto rx=0x7fd29c06d880 tx=0x7fd29401c8e0 comp rx=0 tx=0).stop 2023-05-22T15:42:28.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.520+0000 7fd2a356f640 1 -- 172.21.15.16:0/1606890356 shutdown_connections 2023-05-22T15:42:28.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.520+0000 7fd2a356f640 1 --2- 172.21.15.16:0/1606890356 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd29c159870 0x7fd29c0c8ba0 unknown :-1 s=CLOSED pgs=1123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:28.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.520+0000 7fd2a356f640 1 --2- 172.21.15.16:0/1606890356 >> v2:172.21.15.62:3300/0 conn(0x7fd29c0b0250 0x7fd29c159330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:28.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.520+0000 7fd2a356f640 1 -- 172.21.15.16:0/1606890356 >> 172.21.15.16:0/1606890356 conn(0x7fd29c069720 msgr2=0x7fd29c069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:28.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.520+0000 7fd2a356f640 1 -- 172.21.15.16:0/1606890356 shutdown_connections 2023-05-22T15:42:28.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.520+0000 7fd2a356f640 1 -- 172.21.15.16:0/1606890356 wait complete. 2023-05-22T15:42:28.527 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.520+0000 7fd2a356f640 1 Processor -- start 2023-05-22T15:42:28.527 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.520+0000 7fd2a356f640 1 -- start start 2023-05-22T15:42:28.527 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.520+0000 7fd2a356f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd29c0b0250 0x7fd29c0c0d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:28.527 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.520+0000 7fd2a356f640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd29c1531b0 0x7fd29c140d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:28.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.520+0000 7fd2a356f640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd29c159870 0x7fd29c0ce530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:28.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.520+0000 7fd2a356f640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd29c142140 con 0x7fd29c0b0250 2023-05-22T15:42:28.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.520+0000 7fd2a356f640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd29c0c12b0 con 0x7fd29c159870 2023-05-22T15:42:28.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.520+0000 7fd2a356f640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd29c0b22d0 con 0x7fd29c1531b0 2023-05-22T15:42:28.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.520+0000 7fd2a130b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd29c0b0250 0x7fd29c0c0d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:28.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.520+0000 7fd2a130b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd29c0b0250 0x7fd29c0c0d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34930/0 (socket says 172.21.15.16:34930) 2023-05-22T15:42:28.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.520+0000 7fd2a130b640 1 -- 172.21.15.16:0/1334485901 learned_addr learned my addr 172.21.15.16:0/1334485901 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:42:28.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.520+0000 7fd2a0b0a640 1 --2- 172.21.15.16:0/1334485901 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd29c1531b0 0x7fd29c140d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:28.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.520+0000 7fd2a1b0c640 1 --2- 172.21.15.16:0/1334485901 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd29c159870 0x7fd29c0ce530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:28.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.520+0000 7fd2a130b640 1 -- 172.21.15.16:0/1334485901 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd29c1531b0 msgr2=0x7fd29c140d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:28.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.520+0000 7fd2a130b640 1 --2- 172.21.15.16:0/1334485901 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd29c1531b0 0x7fd29c140d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:28.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.520+0000 7fd2a130b640 1 -- 172.21.15.16:0/1334485901 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd29c159870 msgr2=0x7fd29c0ce530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:28.530 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.520+0000 7fd2a130b640 1 --2- 172.21.15.16:0/1334485901 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd29c159870 0x7fd29c0ce530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:28.530 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.520+0000 7fd2a130b640 1 -- 172.21.15.16:0/1334485901 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd294009a70 con 0x7fd29c0b0250 2023-05-22T15:42:28.530 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.520+0000 7fd2a1b0c640 1 --2- 172.21.15.16:0/1334485901 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd29c159870 0x7fd29c0ce530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:42:28.530 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.520+0000 7fd2a130b640 1 --2- 172.21.15.16:0/1334485901 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd29c0b0250 0x7fd29c0c0d70 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 crypto rx=0x7fd29800c4d0 tx=0x7fd29800c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:28.530 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.520+0000 7fd291ffb640 1 -- 172.21.15.16:0/1334485901 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd298005cf0 con 0x7fd29c0b0250 2023-05-22T15:42:28.530 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.520+0000 7fd291ffb640 1 -- 172.21.15.16:0/1334485901 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd298005e50 con 0x7fd29c0b0250 2023-05-22T15:42:28.530 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.520+0000 7fd291ffb640 1 -- 172.21.15.16:0/1334485901 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd298007270 con 0x7fd29c0b0250 2023-05-22T15:42:28.531 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.524+0000 7fd2a356f640 1 -- 172.21.15.16:0/1334485901 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd29c159cc0 con 0x7fd29c0b0250 2023-05-22T15:42:28.531 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.524+0000 7fd2a356f640 1 -- 172.21.15.16:0/1334485901 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd29c15d0c0 con 0x7fd29c0b0250 2023-05-22T15:42:28.537 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.524+0000 7fd291ffb640 1 -- 172.21.15.16:0/1334485901 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) v1 ==== 84332+0+0 (secure 0 0 0) 0x7fd2980073d0 con 0x7fd29c0b0250 2023-05-22T15:42:28.537 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.524+0000 7fd291ffb640 1 --2- 172.21.15.16:0/1334485901 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd2800654a0 0x7fd280067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:28.537 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.524+0000 7fd291ffb640 1 -- 172.21.15.16:0/1334485901 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 6382+0+0 (secure 0 0 0) 0x7fd2980820a0 con 0x7fd29c0b0250 2023-05-22T15:42:28.537 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.524+0000 7fd2a0b0a640 1 --2- 172.21.15.16:0/1334485901 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd2800654a0 0x7fd280067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:28.537 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.524+0000 7fd2a0b0a640 1 --2- 172.21.15.16:0/1334485901 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd2800654a0 0x7fd280067950 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fd29c0aee40 tx=0x7fd28c008040 comp rx=0 tx=0).ready entity=mgr.6765 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:28.537 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.524+0000 7fd2a6691000 1 Processor -- start 2023-05-22T15:42:28.538 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.524+0000 7fd2a6691000 1 -- start start 2023-05-22T15:42:28.538 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.524+0000 7fd2a6691000 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x55d041999fd0 0x55d041c0cc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:28.538 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.524+0000 7fd2a6691000 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d041c0d170 0x55d041c0d540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:28.538 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.524+0000 7fd2a6691000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d041bc0a10 0x55d041bc0de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:28.538 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.524+0000 7fd2a6691000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d041a8abd0 con 0x55d041bc0a10 2023-05-22T15:42:28.539 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.524+0000 7fd2a6691000 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d0419a7dd0 con 0x55d041c0d170 2023-05-22T15:42:28.539 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.524+0000 7fd2a6691000 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x55d041a18640 con 0x55d041999fd0 2023-05-22T15:42:28.539 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.524+0000 7fd2a1b0c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d041bc0a10 0x55d041bc0de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:28.539 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.524+0000 7fd2a1b0c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d041bc0a10 0x55d041bc0de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34932/0 (socket says 172.21.15.16:34932) 2023-05-22T15:42:28.539 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.524+0000 7fd2a1b0c640 1 -- 172.21.15.16:0/1545524529 learned_addr learned my addr 172.21.15.16:0/1545524529 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:42:28.539 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.524+0000 7fd2a130b640 1 --2- 172.21.15.16:0/1545524529 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x55d041999fd0 0x55d041c0cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:28.539 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.524+0000 7fd2a0b0a640 1 --2- 172.21.15.16:0/1545524529 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d041c0d170 0x55d041c0d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:28.540 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.524+0000 7fd2a1b0c640 1 -- 172.21.15.16:0/1545524529 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x55d041999fd0 msgr2=0x55d041c0cc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:42:28.540 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.524+0000 7fd2a1b0c640 1 --2- 172.21.15.16:0/1545524529 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x55d041999fd0 0x55d041c0cc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:28.540 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.524+0000 7fd2a1b0c640 1 -- 172.21.15.16:0/1545524529 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d041c0d170 msgr2=0x55d041c0d540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:42:28.540 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.524+0000 7fd2a1b0c640 1 --2- 172.21.15.16:0/1545524529 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d041c0d170 0x55d041c0d540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:28.540 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.524+0000 7fd2a1b0c640 1 -- 172.21.15.16:0/1545524529 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d0419d70b0 con 0x55d041bc0a10 2023-05-22T15:42:28.540 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.528+0000 7fd2a1b0c640 1 --2- 172.21.15.16:0/1545524529 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d041bc0a10 0x55d041bc0de0 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7fd2940b0040 tx=0x7fd294004ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6de5d41e60327f8 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:28.541 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.528+0000 7fd27effd640 1 -- 172.21.15.16:0/1545524529 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2940ad640 con 0x55d041bc0a10 2023-05-22T15:42:28.541 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.528+0000 7fd27effd640 1 -- 172.21.15.16:0/1545524529 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2940ad7a0 con 0x55d041bc0a10 2023-05-22T15:42:28.541 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.528+0000 7fd27effd640 1 -- 172.21.15.16:0/1545524529 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2940adb50 con 0x55d041bc0a10 2023-05-22T15:42:28.541 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.528+0000 7fd2a6691000 1 -- 172.21.15.16:0/1545524529 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d041bc0a10 msgr2=0x55d041bc0de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:28.541 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.528+0000 7fd2a6691000 1 --2- 172.21.15.16:0/1545524529 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d041bc0a10 0x55d041bc0de0 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7fd2940b0040 tx=0x7fd294004ac0 comp rx=0 tx=0).stop 2023-05-22T15:42:28.541 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.528+0000 7fd2a6691000 1 -- 172.21.15.16:0/1545524529 shutdown_connections 2023-05-22T15:42:28.541 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.528+0000 7fd2a6691000 1 --2- 172.21.15.16:0/1545524529 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d041bc0a10 0x55d041bc0de0 unknown :-1 s=CLOSED pgs=1125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:28.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.528+0000 7fd2a6691000 1 --2- 172.21.15.16:0/1545524529 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d041c0d170 0x55d041c0d540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:28.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.528+0000 7fd2a6691000 1 --2- 172.21.15.16:0/1545524529 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x55d041999fd0 0x55d041c0cc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:28.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.528+0000 7fd2a6691000 1 -- 172.21.15.16:0/1545524529 >> 172.21.15.16:0/1545524529 conn(0x55d041a69270 msgr2=0x55d041bf0220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:28.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.528+0000 7fd2a6691000 1 -- 172.21.15.16:0/1545524529 shutdown_connections 2023-05-22T15:42:28.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.528+0000 7fd2a6691000 1 -- 172.21.15.16:0/1545524529 wait complete. 2023-05-22T15:42:28.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.528+0000 7fd2a6691000 1 -- start start 2023-05-22T15:42:28.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.528+0000 7fd2a6691000 1 Processor -- start 2023-05-22T15:42:28.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.528+0000 7fd2a6691000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d041999fd0 0x55d041bc8a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:28.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.528+0000 7fd2a6691000 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x55d041bc0a10 0x55d041bc8f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:28.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.528+0000 7fd2a6691000 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d041c0d170 0x55d041bf24a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:28.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.528+0000 7fd2a6691000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d041a11780 con 0x55d041999fd0 2023-05-22T15:42:28.544 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.528+0000 7fd2a6691000 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d0419cf250 con 0x55d041c0d170 2023-05-22T15:42:28.544 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.528+0000 7fd2a6691000 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x55d0419ceef0 con 0x55d041bc0a10 2023-05-22T15:42:28.544 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.528+0000 7fd2a6691000 10 client.0 _ll_register_callbacks cb 0x55d041baeb50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-05-22T15:42:28.544 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.528+0000 7fd2a130b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d041999fd0 0x55d041bc8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:28.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.528+0000 7fd2a130b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d041999fd0 0x55d041bc8a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34944/0 (socket says 172.21.15.16:34944) 2023-05-22T15:42:28.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.528+0000 7fd2a130b640 1 -- 172.21.15.16:0/2971501978 learned_addr learned my addr 172.21.15.16:0/2971501978 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:42:28.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.528+0000 7fd2a0b0a640 1 --2- 172.21.15.16:0/2971501978 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x55d041bc0a10 0x55d041bc8f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:28.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.528+0000 7fd2a1b0c640 1 --2- 172.21.15.16:0/2971501978 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d041c0d170 0x55d041bf24a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:28.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.528+0000 7fd2a130b640 1 -- 172.21.15.16:0/2971501978 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x55d041bc0a10 msgr2=0x55d041bc8f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:42:28.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.528+0000 7fd2a130b640 1 --2- 172.21.15.16:0/2971501978 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x55d041bc0a10 0x55d041bc8f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:28.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.528+0000 7fd2a130b640 1 -- 172.21.15.16:0/2971501978 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d041c0d170 msgr2=0x55d041bf24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:42:28.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.528+0000 7fd2a130b640 1 --2- 172.21.15.16:0/2971501978 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d041c0d170 0x55d041bf24a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:28.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.528+0000 7fd2a130b640 1 -- 172.21.15.16:0/2971501978 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd29c15c820 con 0x55d041999fd0 2023-05-22T15:42:28.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.528+0000 7fd2a130b640 1 --2- 172.21.15.16:0/2971501978 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d041999fd0 0x55d041bc8a20 secure :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 crypto rx=0x7fd2980534a0 tx=0x7fd29804fff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e4cbabe65becd7b server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:28.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.528+0000 7fd277fff640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2023-05-22T15:42:28.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.528+0000 7fd277fff640 1 -- 172.21.15.16:0/2971501978 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd29806a020 con 0x55d041999fd0 2023-05-22T15:42:28.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.528+0000 7fd277fff640 1 -- 172.21.15.16:0/2971501978 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 6382+0+0 (secure 0 0 0) 0x7fd298059270 con 0x55d041999fd0 2023-05-22T15:42:28.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.528+0000 7fd2a6691000 1 -- 172.21.15.16:0/2971501978 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55d0419d5c60 con 0x55d041999fd0 2023-05-22T15:42:28.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.528+0000 7fd277fff640 1 -- 172.21.15.16:0/2971501978 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd29804f590 con 0x55d041999fd0 2023-05-22T15:42:28.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.528+0000 7fd2a6691000 10 client.7170 fetch_fsmap learned FSMap version 30 2023-05-22T15:42:28.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.528+0000 7fd2a6691000 1 -- 172.21.15.16:0/2971501978 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=2+}) v3 -- 0x55d04170e890 con 0x55d041999fd0 2023-05-22T15:42:28.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.528+0000 7fd277fff640 1 -- 172.21.15.16:0/2971501978 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd29805f070 con 0x55d041999fd0 2023-05-22T15:42:28.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.528+0000 7fd277fff640 1 -- 172.21.15.16:0/2971501978 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 30) v1 ==== 3106+0+0 (secure 0 0 0) 0x7fd29804f6f0 con 0x55d041999fd0 2023-05-22T15:42:28.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.528+0000 7fd2a6691000 10 client.7170 fetch_fsmap finished waiting for FSMap version 30 2023-05-22T15:42:28.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.528+0000 7fd2a6691000 10 client.7170 resolve_mds: resolved name 'a' to daemon mds.a 2023-05-22T15:42:28.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.528+0000 7fd2a6691000 20 client.7170 populate_metadata read hostname 'smithi016' 2023-05-22T15:42:28.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.528+0000 7fd2a6691000 1 --2- 172.21.15.16:0/2971501978 >> [v2:172.21.15.16:6834/1772031282,v1:172.21.15.16:6836/1772031282] conn(0x55d041c1c8e0 0x55d041c1ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:28.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.528+0000 7fd2a6691000 4 client.7170 mds_command: new command op to 7011 tid=0[{"prefix": "get_command_descriptions"}] 2023-05-22T15:42:28.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.528+0000 7fd2a6691000 1 -- 172.21.15.16:0/2971501978 --> [v2:172.21.15.16:6834/1772031282,v1:172.21.15.16:6836/1772031282] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55d041a100d0 con 0x55d041c1c8e0 2023-05-22T15:42:28.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.532+0000 7fd2a0b0a640 1 --2- 172.21.15.16:0/2971501978 >> [v2:172.21.15.16:6834/1772031282,v1:172.21.15.16:6836/1772031282] conn(0x55d041c1c8e0 0x55d041c1ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:28.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.532+0000 7fd2a0b0a640 1 --2- 172.21.15.16:0/2971501978 >> [v2:172.21.15.16:6834/1772031282,v1:172.21.15.16:6836/1772031282] conn(0x55d041c1c8e0 0x55d041c1ecc0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=27bc30bcc957380b server_cookie=aa0b19e5557866ce in_seq=0 out_seq=0 2023-05-22T15:42:28.549 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.532+0000 7fd277fff640 10 client.7170 ms_handle_connect on v2:172.21.15.16:6834/1772031282 2023-05-22T15:42:28.549 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.532+0000 7fd277fff640 1 -- 172.21.15.16:0/2971501978 <== mds.0 v2:172.21.15.16:6834/1772031282 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x55d041a100d0 con 0x55d041c1c8e0 2023-05-22T15:42:28.549 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.532+0000 7fd277fff640 10 client.7170 handle_command_reply: tid=0 2023-05-22T15:42:28.549 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.532+0000 7fd2a6691000 1 client.7170 shutdown 2023-05-22T15:42:28.549 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.532+0000 7fd2a6691000 1 -- 172.21.15.16:0/2971501978 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d041999fd0 msgr2=0x55d041bc8a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:28.549 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.532+0000 7fd2a6691000 1 --2- 172.21.15.16:0/2971501978 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d041999fd0 0x55d041bc8a20 secure :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 crypto rx=0x7fd2980534a0 tx=0x7fd29804fff0 comp rx=0 tx=0).stop 2023-05-22T15:42:28.550 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.532+0000 7fd2a6691000 1 -- 172.21.15.16:0/2971501978 shutdown_connections 2023-05-22T15:42:28.550 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.532+0000 7fd2a6691000 1 --2- 172.21.15.16:0/2971501978 >> [v2:172.21.15.16:6834/1772031282,v1:172.21.15.16:6836/1772031282] conn(0x55d041c1c8e0 0x55d041c1ecc0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:28.550 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.532+0000 7fd2a6691000 1 --2- 172.21.15.16:0/2971501978 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d041c0d170 0x55d041bf24a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:28.551 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.532+0000 7fd2a6691000 1 --2- 172.21.15.16:0/2971501978 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x55d041bc0a10 0x55d041bc8f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:28.551 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.532+0000 7fd2a6691000 1 --2- 172.21.15.16:0/2971501978 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d041999fd0 0x55d041bc8a20 unknown :-1 s=CLOSED pgs=1126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:28.551 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.532+0000 7fd2a6691000 1 -- 172.21.15.16:0/2971501978 >> 172.21.15.16:0/2971501978 conn(0x55d041a69270 msgr2=0x55d041bc0100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:28.551 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.532+0000 7fd277fff640 0 client.7170 ms_handle_reset on v2:172.21.15.16:6834/1772031282 2023-05-22T15:42:28.551 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.532+0000 7fd2a6691000 1 -- 172.21.15.16:0/2971501978 shutdown_connections 2023-05-22T15:42:28.551 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.532+0000 7fd2a6691000 1 -- 172.21.15.16:0/2971501978 wait complete. 2023-05-22T15:42:28.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.532+0000 7fd2a6691000 20 client.7170 trim_cache size 0 max 16384 2023-05-22T15:42:28.554 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.548+0000 7fd2a6691000 1 Processor -- start 2023-05-22T15:42:28.554 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.548+0000 7fd2a6691000 1 -- start start 2023-05-22T15:42:28.554 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.548+0000 7fd2a6691000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d041999fd0 0x55d041bf1450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:28.554 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.548+0000 7fd2a6691000 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x55d041bc0a10 0x55d041bf1990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:28.554 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.548+0000 7fd2a6691000 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d041c0d170 0x55d041c0c2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:28.554 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.548+0000 7fd2a6691000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d0416cf940 con 0x55d041999fd0 2023-05-22T15:42:28.555 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.548+0000 7fd2a6691000 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d041a198d0 con 0x55d041c0d170 2023-05-22T15:42:28.555 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.548+0000 7fd2a6691000 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x55d041a0bb20 con 0x55d041bc0a10 2023-05-22T15:42:28.555 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.548+0000 7fd2a130b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d041999fd0 0x55d041bf1450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:28.555 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.548+0000 7fd2a130b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d041999fd0 0x55d041bf1450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34950/0 (socket says 172.21.15.16:34950) 2023-05-22T15:42:28.555 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.548+0000 7fd2a130b640 1 -- 172.21.15.16:0/4104687428 learned_addr learned my addr 172.21.15.16:0/4104687428 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:42:28.555 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.548+0000 7fd2a130b640 1 -- 172.21.15.16:0/4104687428 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x55d041bc0a10 msgr2=0x55d041bf1990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:42:28.555 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.548+0000 7fd2a130b640 1 --2- 172.21.15.16:0/4104687428 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x55d041bc0a10 0x55d041bf1990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:28.556 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.548+0000 7fd2a130b640 1 -- 172.21.15.16:0/4104687428 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d041c0d170 msgr2=0x55d041c0c2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:42:28.556 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.548+0000 7fd2a130b640 1 --2- 172.21.15.16:0/4104687428 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d041c0d170 0x55d041c0c2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:28.556 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.548+0000 7fd2a130b640 1 -- 172.21.15.16:0/4104687428 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d0419d7360 con 0x55d041999fd0 2023-05-22T15:42:28.556 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.548+0000 7fd2a130b640 1 --2- 172.21.15.16:0/4104687428 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d041999fd0 0x55d041bf1450 secure :-1 s=READY pgs=1127 cs=0 l=1 rev1=1 crypto rx=0x7fd298057280 tx=0x7fd29801d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2a1cda58f047a79 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:28.556 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.548+0000 7fd27f7fe640 1 -- 172.21.15.16:0/4104687428 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd29806a040 con 0x55d041999fd0 2023-05-22T15:42:28.557 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.548+0000 7fd27f7fe640 1 -- 172.21.15.16:0/4104687428 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd29805f040 con 0x55d041999fd0 2023-05-22T15:42:28.557 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.548+0000 7fd27f7fe640 1 -- 172.21.15.16:0/4104687428 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd29800e040 con 0x55d041999fd0 2023-05-22T15:42:28.557 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.548+0000 7fd2a6691000 1 -- 172.21.15.16:0/4104687428 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d041999fd0 msgr2=0x55d041bf1450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:28.557 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.548+0000 7fd2a6691000 1 --2- 172.21.15.16:0/4104687428 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d041999fd0 0x55d041bf1450 secure :-1 s=READY pgs=1127 cs=0 l=1 rev1=1 crypto rx=0x7fd298057280 tx=0x7fd29801d000 comp rx=0 tx=0).stop 2023-05-22T15:42:28.557 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.548+0000 7fd2a6691000 1 -- 172.21.15.16:0/4104687428 shutdown_connections 2023-05-22T15:42:28.557 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.548+0000 7fd2a6691000 1 --2- 172.21.15.16:0/4104687428 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d041c0d170 0x55d041c0c2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:28.557 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.548+0000 7fd2a6691000 1 --2- 172.21.15.16:0/4104687428 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x55d041bc0a10 0x55d041bf1990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:28.558 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.548+0000 7fd2a6691000 1 --2- 172.21.15.16:0/4104687428 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d041999fd0 0x55d041bf1450 unknown :-1 s=CLOSED pgs=1127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:28.558 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.548+0000 7fd2a6691000 1 -- 172.21.15.16:0/4104687428 >> 172.21.15.16:0/4104687428 conn(0x55d041a69270 msgr2=0x55d041c19e30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:28.558 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.548+0000 7fd2a6691000 1 -- 172.21.15.16:0/4104687428 shutdown_connections 2023-05-22T15:42:28.558 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.548+0000 7fd2a6691000 1 -- 172.21.15.16:0/4104687428 wait complete. 2023-05-22T15:42:28.559 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.548+0000 7fd2a6691000 1 -- start start 2023-05-22T15:42:28.559 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.548+0000 7fd2a6691000 1 Processor -- start 2023-05-22T15:42:28.565 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.552+0000 7fd2a6691000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d041999fd0 0x55d041c0c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:28.566 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.552+0000 7fd2a6691000 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x55d041bc0a10 0x55d041c64d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:28.566 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.552+0000 7fd2a6691000 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d041c0d170 0x55d041c652b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:28.566 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.552+0000 7fd2a6691000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d041a142e0 con 0x55d041999fd0 2023-05-22T15:42:28.566 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.552+0000 7fd2a6691000 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d041a9b250 con 0x55d041c0d170 2023-05-22T15:42:28.566 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.552+0000 7fd2a6691000 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x55d0419e1620 con 0x55d041bc0a10 2023-05-22T15:42:28.567 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.552+0000 7fd2a6691000 10 client.0 _ll_register_callbacks cb 0x55d041a61eb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-05-22T15:42:28.567 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.552+0000 7fd2a130b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d041999fd0 0x55d041c0c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:28.567 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.552+0000 7fd2a130b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d041999fd0 0x55d041c0c5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34966/0 (socket says 172.21.15.16:34966) 2023-05-22T15:42:28.568 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.552+0000 7fd2a130b640 1 -- 172.21.15.16:0/3145019212 learned_addr learned my addr 172.21.15.16:0/3145019212 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:42:28.568 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.552+0000 7fd2a130b640 1 -- 172.21.15.16:0/3145019212 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x55d041bc0a10 msgr2=0x55d041c64d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:42:28.568 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.552+0000 7fd2a0b0a640 1 --2- 172.21.15.16:0/3145019212 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x55d041bc0a10 0x55d041c64d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:28.568 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.552+0000 7fd2a1b0c640 1 --2- 172.21.15.16:0/3145019212 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d041c0d170 0x55d041c652b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:28.569 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.552+0000 7fd2a130b640 1 --2- 172.21.15.16:0/3145019212 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x55d041bc0a10 0x55d041c64d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:28.569 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.552+0000 7fd2a130b640 1 -- 172.21.15.16:0/3145019212 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d041c0d170 msgr2=0x55d041c652b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:42:28.569 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.552+0000 7fd2a130b640 1 --2- 172.21.15.16:0/3145019212 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d041c0d170 0x55d041c652b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:28.569 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.552+0000 7fd2a130b640 1 -- 172.21.15.16:0/3145019212 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d04170e890 con 0x55d041999fd0 2023-05-22T15:42:28.569 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.552+0000 7fd2a0b0a640 1 --2- 172.21.15.16:0/3145019212 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x55d041bc0a10 0x55d041c64d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:42:28.569 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.552+0000 7fd2a130b640 1 --2- 172.21.15.16:0/3145019212 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d041999fd0 0x55d041c0c5d0 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x7fd29805b250 tx=0x7fd298001320 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a28af6f218a8a48 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:28.569 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.552+0000 7fd277fff640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2023-05-22T15:42:28.570 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.552+0000 7fd277fff640 1 -- 172.21.15.16:0/3145019212 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd29806a040 con 0x55d041999fd0 2023-05-22T15:42:28.570 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.552+0000 7fd277fff640 1 -- 172.21.15.16:0/3145019212 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 6382+0+0 (secure 0 0 0) 0x7fd298059270 con 0x55d041999fd0 2023-05-22T15:42:28.570 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.552+0000 7fd2a6691000 1 -- 172.21.15.16:0/3145019212 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55d041c23680 con 0x55d041999fd0 2023-05-22T15:42:28.570 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.552+0000 7fd277fff640 1 -- 172.21.15.16:0/3145019212 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd29800e040 con 0x55d041999fd0 2023-05-22T15:42:28.570 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.552+0000 7fd2a6691000 10 client.7176 fetch_fsmap learned FSMap version 30 2023-05-22T15:42:28.570 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.552+0000 7fd2a6691000 1 -- 172.21.15.16:0/3145019212 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=2+}) v3 -- 0x55d0419bcbe0 con 0x55d041999fd0 2023-05-22T15:42:28.571 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.552+0000 7fd277fff640 1 -- 172.21.15.16:0/3145019212 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55d041c23680 con 0x55d041999fd0 2023-05-22T15:42:28.571 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.552+0000 7fd277fff640 1 -- 172.21.15.16:0/3145019212 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 30) v1 ==== 3106+0+0 (secure 0 0 0) 0x7fd298057300 con 0x55d041999fd0 2023-05-22T15:42:28.571 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.552+0000 7fd2a6691000 10 client.7176 fetch_fsmap finished waiting for FSMap version 30 2023-05-22T15:42:28.571 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.552+0000 7fd2a6691000 10 client.7176 resolve_mds: resolved name 'a' to daemon mds.a 2023-05-22T15:42:28.571 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.552+0000 7fd2a6691000 20 client.7176 populate_metadata read hostname 'smithi016' 2023-05-22T15:42:28.571 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.552+0000 7fd2a6691000 1 --2- 172.21.15.16:0/3145019212 >> [v2:172.21.15.16:6834/1772031282,v1:172.21.15.16:6836/1772031282] conn(0x55d041c65c40 0x55d041c66010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:28.572 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.552+0000 7fd2a6691000 4 client.7176 mds_command: new command op to 7011 tid=0[{"prefix": "status"}] 2023-05-22T15:42:28.572 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.552+0000 7fd2a6691000 1 -- 172.21.15.16:0/3145019212 --> [v2:172.21.15.16:6834/1772031282,v1:172.21.15.16:6836/1772031282] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55d0419b84d0 con 0x55d041c65c40 2023-05-22T15:42:28.572 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.552+0000 7fd2a0b0a640 1 --2- 172.21.15.16:0/3145019212 >> [v2:172.21.15.16:6834/1772031282,v1:172.21.15.16:6836/1772031282] conn(0x55d041c65c40 0x55d041c66010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:28.572 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.552+0000 7fd2a0b0a640 1 --2- 172.21.15.16:0/3145019212 >> [v2:172.21.15.16:6834/1772031282,v1:172.21.15.16:6836/1772031282] conn(0x55d041c65c40 0x55d041c66010 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e4c14bde89b68699 server_cookie=fedcd7672e1411fb in_seq=0 out_seq=0 2023-05-22T15:42:28.572 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.552+0000 7fd277fff640 10 client.7176 ms_handle_connect on v2:172.21.15.16:6834/1772031282 2023-05-22T15:42:28.572 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.552+0000 7fd277fff640 1 -- 172.21.15.16:0/3145019212 <== mds.0 v2:172.21.15.16:6834/1772031282 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+325 (crc 0 0 0) 0x55d0419b84d0 con 0x55d041c65c40 2023-05-22T15:42:28.573 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.552+0000 7fd277fff640 10 client.7176 handle_command_reply: tid=0 2023-05-22T15:42:28.573 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.552+0000 7fd2a6691000 1 client.7176 shutdown 2023-05-22T15:42:28.573 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.552+0000 7fd2a6691000 1 -- 172.21.15.16:0/3145019212 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d041999fd0 msgr2=0x55d041c0c5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:28.573 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.552+0000 7fd2a6691000 1 --2- 172.21.15.16:0/3145019212 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d041999fd0 0x55d041c0c5d0 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x7fd29805b250 tx=0x7fd298001320 comp rx=0 tx=0).stop 2023-05-22T15:42:28.573 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.552+0000 7fd2a6691000 1 -- 172.21.15.16:0/3145019212 shutdown_connections 2023-05-22T15:42:28.573 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.552+0000 7fd2a6691000 1 --2- 172.21.15.16:0/3145019212 >> [v2:172.21.15.16:6834/1772031282,v1:172.21.15.16:6836/1772031282] conn(0x55d041c65c40 0x55d041c66010 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:28.574 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.552+0000 7fd2a6691000 1 --2- 172.21.15.16:0/3145019212 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d041c0d170 0x55d041c652b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:28.574 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.552+0000 7fd2a6691000 1 --2- 172.21.15.16:0/3145019212 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x55d041bc0a10 0x55d041c64d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:28.574 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.552+0000 7fd277fff640 0 client.7176 ms_handle_reset on v2:172.21.15.16:6834/1772031282 2023-05-22T15:42:28.574 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.552+0000 7fd2a6691000 1 --2- 172.21.15.16:0/3145019212 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d041999fd0 0x55d041c0c5d0 unknown :-1 s=CLOSED pgs=1128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:28.574 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.552+0000 7fd2a6691000 1 -- 172.21.15.16:0/3145019212 >> 172.21.15.16:0/3145019212 conn(0x55d041a69270 msgr2=0x55d041c195a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:28.575 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.552+0000 7fd2a6691000 1 -- 172.21.15.16:0/3145019212 shutdown_connections 2023-05-22T15:42:28.575 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.552+0000 7fd2a6691000 1 -- 172.21.15.16:0/3145019212 wait complete. 2023-05-22T15:42:28.575 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.552+0000 7fd2a6691000 20 client.7176 trim_cache size 0 max 16384 2023-05-22T15:42:28.575 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-05-22T15:42:28.576 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_fsid": "b2719834-7847-4d98-90a6-f65d57f85c49", 2023-05-22T15:42:28.576 INFO:teuthology.orchestra.run.smithi016.stdout: "whoami": 0, 2023-05-22T15:42:28.576 INFO:teuthology.orchestra.run.smithi016.stdout: "id": 7011, 2023-05-22T15:42:28.576 INFO:teuthology.orchestra.run.smithi016.stdout: "want_state": "up:active", 2023-05-22T15:42:28.576 INFO:teuthology.orchestra.run.smithi016.stdout: "state": "up:active", 2023-05-22T15:42:28.576 INFO:teuthology.orchestra.run.smithi016.stdout: "fs_name": "cephfs", 2023-05-22T15:42:28.577 INFO:teuthology.orchestra.run.smithi016.stdout: "rank_uptime": 5.7397161089999997, 2023-05-22T15:42:28.577 INFO:teuthology.orchestra.run.smithi016.stdout: "mdsmap_epoch": 30, 2023-05-22T15:42:28.577 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch": 90, 2023-05-22T15:42:28.577 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch_barrier": 90, 2023-05-22T15:42:28.577 INFO:teuthology.orchestra.run.smithi016.stdout: "uptime": 10.919146011 2023-05-22T15:42:28.577 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-05-22T15:42:28.578 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.552+0000 7fd2a356f640 1 -- 172.21.15.16:0/1334485901 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd2800654a0 msgr2=0x7fd280067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:28.579 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.552+0000 7fd2a356f640 1 --2- 172.21.15.16:0/1334485901 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd2800654a0 0x7fd280067950 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fd29c0aee40 tx=0x7fd28c008040 comp rx=0 tx=0).stop 2023-05-22T15:42:28.579 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.552+0000 7fd2a356f640 1 -- 172.21.15.16:0/1334485901 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd29c0b0250 msgr2=0x7fd29c0c0d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:28.579 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.552+0000 7fd2a356f640 1 --2- 172.21.15.16:0/1334485901 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd29c0b0250 0x7fd29c0c0d70 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 crypto rx=0x7fd29800c4d0 tx=0x7fd29800c990 comp rx=0 tx=0).stop 2023-05-22T15:42:28.579 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.552+0000 7fd2a356f640 1 -- 172.21.15.16:0/1334485901 shutdown_connections 2023-05-22T15:42:28.579 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.552+0000 7fd2a356f640 1 --2- 172.21.15.16:0/1334485901 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd29c159870 0x7fd29c0ce530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:28.579 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.552+0000 7fd2a356f640 1 --2- 172.21.15.16:0/1334485901 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd2800654a0 0x7fd280067950 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:28.579 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.552+0000 7fd2a356f640 1 --2- 172.21.15.16:0/1334485901 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd29c1531b0 0x7fd29c140d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:28.580 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.552+0000 7fd2a356f640 1 --2- 172.21.15.16:0/1334485901 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd29c0b0250 0x7fd29c0c0d70 unknown :-1 s=CLOSED pgs=1124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:28.580 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.552+0000 7fd2a356f640 1 -- 172.21.15.16:0/1334485901 >> 172.21.15.16:0/1334485901 conn(0x7fd29c069720 msgr2=0x7fd29c14f440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:28.580 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.552+0000 7fd2a356f640 1 -- 172.21.15.16:0/1334485901 shutdown_connections 2023-05-22T15:42:28.580 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.552+0000 7fd2a356f640 1 -- 172.21.15.16:0/1334485901 wait complete. 2023-05-22T15:42:28.581 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-05-22T15:42:28.635 INFO:tasks.ceph.mds.a.smithi016.stderr:2023-05-22T15:42:28.632+0000 7faa91fcf640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-05-22T15:42:28.695 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.692+0000 7f52c97a8640 1 Processor -- start 2023-05-22T15:42:28.695 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.692+0000 7f52c97a8640 1 -- start start 2023-05-22T15:42:28.696 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.692+0000 7f52c97a8640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f52c40b7dd0 0x7f52c40ccab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:28.696 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.692+0000 7f52c97a8640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f52c415f700 0x7f52c415fb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:28.697 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.692+0000 7f52c97a8640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52c40cd170 con 0x7f52c4160080 2023-05-22T15:42:28.697 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.692+0000 7f52c97a8640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52c40d3360 con 0x7f52c40b7dd0 2023-05-22T15:42:28.697 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.692+0000 7f52c97a8640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52c40d2ac0 con 0x7f52c415f700 2023-05-22T15:42:28.697 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.692+0000 7f52c2ffd640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f52c40b7dd0 0x7f52c40ccab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:28.697 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.692+0000 7f52c2ffd640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f52c40b7dd0 0x7f52c40ccab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34972/0 (socket says 172.21.15.16:34972) 2023-05-22T15:42:28.697 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.692+0000 7f52c2ffd640 1 -- 172.21.15.16:0/156880263 learned_addr learned my addr 172.21.15.16:0/156880263 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:42:28.697 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.692+0000 7f52c27fc640 1 --2- 172.21.15.16:0/156880263 >> v2:172.21.15.62:3300/0 conn(0x7f52c415f700 0x7f52c415fb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:28.698 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.692+0000 7f52c2ffd640 1 -- 172.21.15.16:0/156880263 >> v2:172.21.15.62:3300/0 conn(0x7f52c415f700 msgr2=0x7f52c415fb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:42:28.698 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.692+0000 7f52c2ffd640 1 --2- 172.21.15.16:0/156880263 >> v2:172.21.15.62:3300/0 conn(0x7f52c415f700 0x7f52c415fb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:28.698 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.692+0000 7f52c2ffd640 1 -- 172.21.15.16:0/156880263 >> v1:172.21.15.16:6789/0 conn(0x7f52c4160080 legacy=0x7f52c40ce720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:28.698 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.692+0000 7f52c2ffd640 1 -- 172.21.15.16:0/156880263 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52c40d2da0 con 0x7f52c40b7dd0 2023-05-22T15:42:28.698 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.692+0000 7f52c1ffb640 1 -- 172.21.15.16:0/156880263 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3126639067 0 0) 0x7f52c40cd170 con 0x7f52c4160080 2023-05-22T15:42:28.698 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.692+0000 7f52c27fc640 1 --2- 172.21.15.16:0/156880263 >> v2:172.21.15.62:3300/0 conn(0x7f52c415f700 0x7f52c415fb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:42:28.699 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.696+0000 7f52c2ffd640 1 --2- 172.21.15.16:0/156880263 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f52c40b7dd0 0x7f52c40ccab0 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x7f52c40b1320 tx=0x7f52b401c9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10f959b880fdabe5 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:28.699 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.696+0000 7f52c1ffb640 1 -- 172.21.15.16:0/156880263 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52b40a5800 con 0x7f52c40b7dd0 2023-05-22T15:42:28.699 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.696+0000 7f52c1ffb640 1 -- 172.21.15.16:0/156880263 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52b40a5960 con 0x7f52c40b7dd0 2023-05-22T15:42:28.699 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.696+0000 7f52c1ffb640 1 -- 172.21.15.16:0/156880263 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52b40a5cf0 con 0x7f52c40b7dd0 2023-05-22T15:42:28.699 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.696+0000 7f52c97a8640 1 -- 172.21.15.16:0/156880263 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f52c40b7dd0 msgr2=0x7f52c40ccab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:28.700 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.696+0000 7f52c97a8640 1 --2- 172.21.15.16:0/156880263 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f52c40b7dd0 0x7f52c40ccab0 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x7f52c40b1320 tx=0x7f52b401c9a0 comp rx=0 tx=0).stop 2023-05-22T15:42:28.700 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.696+0000 7f52c97a8640 1 -- 172.21.15.16:0/156880263 shutdown_connections 2023-05-22T15:42:28.701 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.696+0000 7f52c97a8640 1 --2- 172.21.15.16:0/156880263 >> v2:172.21.15.62:3300/0 conn(0x7f52c415f700 0x7f52c415fb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:28.701 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.696+0000 7f52c97a8640 1 --2- 172.21.15.16:0/156880263 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f52c40b7dd0 0x7f52c40ccab0 unknown :-1 s=CLOSED pgs=1130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:28.701 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.696+0000 7f52c97a8640 1 -- 172.21.15.16:0/156880263 >> 172.21.15.16:0/156880263 conn(0x7f52c40655c0 msgr2=0x7f52c4070e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:28.701 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.696+0000 7f52c97a8640 1 -- 172.21.15.16:0/156880263 shutdown_connections 2023-05-22T15:42:28.701 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.696+0000 7f52c97a8640 1 -- 172.21.15.16:0/156880263 wait complete. 2023-05-22T15:42:28.701 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.696+0000 7f52c97a8640 1 Processor -- start 2023-05-22T15:42:28.701 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.696+0000 7f52c97a8640 1 -- start start 2023-05-22T15:42:28.703 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.696+0000 7f52c97a8640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f52c40b7dd0 0x7f52c40c8d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:28.703 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.696+0000 7f52c97a8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f52c415f700 0x7f52c4153870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:28.703 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.696+0000 7f52c97a8640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f52c4160080 0x7f52c40b2f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:28.703 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.696+0000 7f52c97a8640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52c40c9250 con 0x7f52c415f700 2023-05-22T15:42:28.703 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.696+0000 7f52c97a8640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52c4155950 con 0x7f52c40b7dd0 2023-05-22T15:42:28.703 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.696+0000 7f52c97a8640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52c4154610 con 0x7f52c4160080 2023-05-22T15:42:28.704 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.696+0000 7f52c27fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f52c415f700 0x7f52c4153870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:28.704 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.696+0000 7f52c27fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f52c415f700 0x7f52c4153870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34974/0 (socket says 172.21.15.16:34974) 2023-05-22T15:42:28.704 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.696+0000 7f52c27fc640 1 -- 172.21.15.16:0/2488817383 learned_addr learned my addr 172.21.15.16:0/2488817383 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:42:28.704 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.696+0000 7f52c37fe640 1 --2- 172.21.15.16:0/2488817383 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f52c4160080 0x7f52c40b2f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:28.704 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.696+0000 7f52c2ffd640 1 --2- 172.21.15.16:0/2488817383 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f52c40b7dd0 0x7f52c40c8d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:28.704 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.700+0000 7f52c27fc640 1 -- 172.21.15.16:0/2488817383 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f52c4160080 msgr2=0x7f52c40b2f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:28.704 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.700+0000 7f52c27fc640 1 --2- 172.21.15.16:0/2488817383 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f52c4160080 0x7f52c40b2f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:28.704 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.700+0000 7f52c27fc640 1 -- 172.21.15.16:0/2488817383 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f52c40b7dd0 msgr2=0x7f52c40c8d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:28.705 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.700+0000 7f52c27fc640 1 --2- 172.21.15.16:0/2488817383 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f52c40b7dd0 0x7f52c40c8d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:28.705 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.700+0000 7f52c27fc640 1 -- 172.21.15.16:0/2488817383 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52b4009b30 con 0x7f52c415f700 2023-05-22T15:42:28.705 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.700+0000 7f52c27fc640 1 --2- 172.21.15.16:0/2488817383 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f52c415f700 0x7f52c4153870 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 crypto rx=0x7f52ac00c7c0 tx=0x7f52ac00cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:28.705 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.700+0000 7f52bb7fe640 1 -- 172.21.15.16:0/2488817383 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52ac007a30 con 0x7f52c415f700 2023-05-22T15:42:28.705 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.700+0000 7f52bb7fe640 1 -- 172.21.15.16:0/2488817383 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52ac007b90 con 0x7f52c415f700 2023-05-22T15:42:28.705 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.700+0000 7f52c97a8640 1 -- 172.21.15.16:0/2488817383 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52c4168a20 con 0x7f52c415f700 2023-05-22T15:42:28.705 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.700+0000 7f52bb7fe640 1 -- 172.21.15.16:0/2488817383 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52ac00ce70 con 0x7f52c415f700 2023-05-22T15:42:28.706 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.700+0000 7f52c97a8640 1 -- 172.21.15.16:0/2488817383 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52c4168890 con 0x7f52c415f700 2023-05-22T15:42:28.706 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.700+0000 7f52c97a8640 1 -- 172.21.15.16:0/2488817383 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52c4168bb0 con 0x7f52c415f700 2023-05-22T15:42:28.706 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.700+0000 7f52bb7fe640 1 -- 172.21.15.16:0/2488817383 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f52ac0113f0 con 0x7f52c415f700 2023-05-22T15:42:28.707 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.700+0000 7f52bb7fe640 1 --2- 172.21.15.16:0/2488817383 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f52a40654a0 0x7f52a4067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:28.707 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.700+0000 7f52c2ffd640 1 --2- 172.21.15.16:0/2488817383 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f52a40654a0 0x7f52a4067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:28.707 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.700+0000 7f52bb7fe640 1 -- 172.21.15.16:0/2488817383 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f52ac0830d0 con 0x7f52c415f700 2023-05-22T15:42:28.707 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.700+0000 7f52c2ffd640 1 --2- 172.21.15.16:0/2488817383 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f52a40654a0 0x7f52a4067950 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f52b40ae000 tx=0x7f52b40a7070 comp rx=0 tx=0).ready entity=mgr.6765 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:28.708 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.704+0000 7f52bb7fe640 1 -- 172.21.15.16:0/2488817383 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f52ac050990 con 0x7f52c415f700 2023-05-22T15:42:28.898 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.896+0000 7f52c97a8640 1 -- 172.21.15.16:0/2488817383 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f52c404dbe0 con 0x7f52c415f700 2023-05-22T15:42:28.899 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.896+0000 7f52bb7fe640 1 -- 172.21.15.16:0/2488817383 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v30) v1 ==== 53+0+73 (secure 0 0 0) 0x7f52ac053fb0 con 0x7f52c415f700 2023-05-22T15:42:28.899 INFO:teuthology.orchestra.run.smithi016.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-05-22T15:42:28.900 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.896+0000 7f52c97a8640 1 -- 172.21.15.16:0/2488817383 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f52a40654a0 msgr2=0x7f52a4067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:28.901 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.896+0000 7f52c97a8640 1 --2- 172.21.15.16:0/2488817383 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f52a40654a0 0x7f52a4067950 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f52b40ae000 tx=0x7f52b40a7070 comp rx=0 tx=0).stop 2023-05-22T15:42:28.901 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.896+0000 7f52c97a8640 1 -- 172.21.15.16:0/2488817383 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f52c415f700 msgr2=0x7f52c4153870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:28.901 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.896+0000 7f52c97a8640 1 --2- 172.21.15.16:0/2488817383 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f52c415f700 0x7f52c4153870 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 crypto rx=0x7f52ac00c7c0 tx=0x7f52ac00cc80 comp rx=0 tx=0).stop 2023-05-22T15:42:28.901 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.896+0000 7f52c97a8640 1 -- 172.21.15.16:0/2488817383 shutdown_connections 2023-05-22T15:42:28.902 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.896+0000 7f52c97a8640 1 --2- 172.21.15.16:0/2488817383 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f52a40654a0 0x7f52a4067950 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:28.902 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.896+0000 7f52c97a8640 1 --2- 172.21.15.16:0/2488817383 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f52c4160080 0x7f52c40b2f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:28.902 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.896+0000 7f52c97a8640 1 --2- 172.21.15.16:0/2488817383 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f52c415f700 0x7f52c4153870 unknown :-1 s=CLOSED pgs=1131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:28.902 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.896+0000 7f52c97a8640 1 --2- 172.21.15.16:0/2488817383 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f52c40b7dd0 0x7f52c40c8d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:28.902 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.896+0000 7f52c97a8640 1 -- 172.21.15.16:0/2488817383 >> 172.21.15.16:0/2488817383 conn(0x7f52c40655c0 msgr2=0x7f52c40ce6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:28.903 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.896+0000 7f52c97a8640 1 -- 172.21.15.16:0/2488817383 shutdown_connections 2023-05-22T15:42:28.903 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:28.896+0000 7f52c97a8640 1 -- 172.21.15.16:0/2488817383 wait complete. 2023-05-22T15:42:28.918 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-05-22T15:42:28.918 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-05-22T15:42:28.919 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi016.front.sepia.ceph.com 2023-05-22T15:42:28.919 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-05-22T15:42:28.919 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-05-22T15:42:28.920 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-05-22T15:42:28.920 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-05-22T15:42:28.920 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-05-22T15:42:28.920 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-05-22T15:42:28.921 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:42:28.921 DEBUG:teuthology.orchestra.run.smithi016:> ip addr 2023-05-22T15:42:28.927 INFO:teuthology.orchestra.run.smithi016.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-05-22T15:42:28.927 INFO:teuthology.orchestra.run.smithi016.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-05-22T15:42:28.928 INFO:teuthology.orchestra.run.smithi016.stdout: inet 127.0.0.1/8 scope host lo 2023-05-22T15:42:28.928 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-05-22T15:42:28.928 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 ::1/128 scope host 2023-05-22T15:42:28.928 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-05-22T15:42:28.928 INFO:teuthology.orchestra.run.smithi016.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-05-22T15:42:28.928 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:6b:fd:e0 brd ff:ff:ff:ff:ff:ff 2023-05-22T15:42:28.929 INFO:teuthology.orchestra.run.smithi016.stdout: altname enp7s0f0 2023-05-22T15:42:28.929 INFO:teuthology.orchestra.run.smithi016.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-05-22T15:42:28.929 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:6b:fd:e1 brd ff:ff:ff:ff:ff:ff 2023-05-22T15:42:28.929 INFO:teuthology.orchestra.run.smithi016.stdout: altname enp7s0f1 2023-05-22T15:42:28.929 INFO:teuthology.orchestra.run.smithi016.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2023-05-22T15:42:28.929 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:bd:15:64 brd ff:ff:ff:ff:ff:ff 2023-05-22T15:42:28.929 INFO:teuthology.orchestra.run.smithi016.stdout: altname enp1s0f0 2023-05-22T15:42:28.929 INFO:teuthology.orchestra.run.smithi016.stdout: inet 172.21.15.16/20 brd 172.21.15.255 scope global dynamic ens1f0 2023-05-22T15:42:28.930 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft 41567sec preferred_lft 41567sec 2023-05-22T15:42:28.930 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::ec4:7aff:febd:1564/64 scope link 2023-05-22T15:42:28.930 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-05-22T15:42:28.930 INFO:teuthology.orchestra.run.smithi016.stdout:5: ens1f1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-05-22T15:42:28.930 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:bd:15:65 brd ff:ff:ff:ff:ff:ff 2023-05-22T15:42:28.930 INFO:teuthology.orchestra.run.smithi016.stdout: altname enp1s0f1 2023-05-22T15:42:28.930 INFO:teuthology.orchestra.run.smithi016.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2023-05-22T15:42:28.930 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 02:42:1f:b8:98:0a brd ff:ff:ff:ff:ff:ff 2023-05-22T15:42:28.931 INFO:teuthology.orchestra.run.smithi016.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2023-05-22T15:42:28.931 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-05-22T15:42:28.931 INFO:teuthology.orchestra.run.smithi016.stdout:9: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-05-22T15:42:28.931 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 5e:1f:84:6d:79:ae brd ff:ff:ff:ff:ff:ff 2023-05-22T15:42:28.931 INFO:teuthology.orchestra.run.smithi016.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-05-22T15:42:28.931 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-05-22T15:42:28.931 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::5c1f:84ff:fe6d:79ae/64 scope link 2023-05-22T15:42:28.931 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-05-22T15:42:28.932 INFO:teuthology.orchestra.run.smithi016.stdout:10: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-05-22T15:42:28.932 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0e:5c:f6:1f:82:2f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-05-22T15:42:28.932 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::c5c:f6ff:fe1f:822f/64 scope link 2023-05-22T15:42:28.932 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-05-22T15:42:28.932 INFO:teuthology.orchestra.run.smithi016.stdout:11: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-05-22T15:42:28.932 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether c6:a6:3d:69:d0:db brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-05-22T15:42:28.932 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::c4a6:3dff:fe69:d0db/64 scope link 2023-05-22T15:42:28.933 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-05-22T15:42:28.933 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:42:28.933 DEBUG:teuthology.orchestra.run.smithi016:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-05-22T15:42:28.981 INFO:teuthology.orchestra.run.smithi016.stdout:1 2023-05-22T15:42:28.983 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:42:28.983 DEBUG:teuthology.orchestra.run.smithi016:> route 2023-05-22T15:42:29.032 INFO:teuthology.orchestra.run.smithi016.stdout:Kernel IP routing table 2023-05-22T15:42:29.032 INFO:teuthology.orchestra.run.smithi016.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-05-22T15:42:29.032 INFO:teuthology.orchestra.run.smithi016.stdout:default _gateway 0.0.0.0 UG 0 0 0 ens1f0 2023-05-22T15:42:29.033 INFO:teuthology.orchestra.run.smithi016.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2023-05-22T15:42:29.033 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 0 0 0 ens1f0 2023-05-22T15:42:29.033 INFO:teuthology.orchestra.run.smithi016.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-05-22T15:42:29.033 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:42:29.034 DEBUG:teuthology.orchestra.run.smithi016:> (cd / && exec bash -c ' 2023-05-22T15:42:29.034 DEBUG:teuthology.orchestra.run.smithi016:> set -e 2023-05-22T15:42:29.034 DEBUG:teuthology.orchestra.run.smithi016:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2023-05-22T15:42:29.035 DEBUG:teuthology.orchestra.run.smithi016:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2023-05-22T15:42:29.035 DEBUG:teuthology.orchestra.run.smithi016:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2023-05-22T15:42:29.035 DEBUG:teuthology.orchestra.run.smithi016:> ') 2023-05-22T15:42:29.111 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:42:29.111 DEBUG:teuthology.orchestra.run.smithi016:> ip netns list 2023-05-22T15:42:29.159 INFO:teuthology.orchestra.run.smithi016.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-05-22T15:42:29.159 INFO:teuthology.orchestra.run.smithi016.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-05-22T15:42:29.160 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-05-22T15:42:29.160 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:42:29.161 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip link set brx.0 up 2023-05-22T15:42:29.216 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2023-05-22T15:42:29.216 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-05-22T15:42:29.216 DEBUG:teuthology.orchestra.run.smithi016:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-05-22T15:42:29.263 INFO:teuthology.orchestra.run.smithi016.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-05-22T15:42:29.263 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-05-22T15:42:29.264 DEBUG:teuthology.orchestra.run.smithi016:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-05-22T15:42:29.310 DEBUG:teuthology.orchestra.run.smithi016:> sudo modprobe fuse 2023-05-22T15:42:29.368 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-05-22T15:42:29.369 DEBUG:teuthology.orchestra.run.smithi016:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-05-22T15:42:29.422 INFO:teuthology.orchestra.run.smithi016.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-05-22T15:42:29.423 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-05-22T15:42:29.424 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:42:29.424 DEBUG:teuthology.orchestra.run.smithi016:> ls /sys/fs/fuse/connections 2023-05-22T15:42:29.471 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2023-05-22T15:42:29.471 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2023-05-22T15:42:29.515 DEBUG:teuthology.orchestra.run.smithi016:> sudo modprobe fuse 2023-05-22T15:42:29.530 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-05-22T15:42:29.530 DEBUG:teuthology.orchestra.run.smithi016:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-05-22T15:42:29.586 INFO:teuthology.orchestra.run.smithi016.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-05-22T15:42:29.587 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-05-22T15:42:29.587 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:42:29.588 DEBUG:teuthology.orchestra.run.smithi016:> ls /sys/fs/fuse/connections 2023-05-22T15:42:29.596 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi016.stderr:2023-05-22T15:42:29.592+0000 7fc7d0bc0580 -1 init, newargv = 0x55c7bd0dfdd0 newargc=15 2023-05-22T15:42:29.596 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi016.stderr:ceph-fuse[55199]: starting ceph client 2023-05-22T15:42:29.601 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi016.stderr:ceph-fuse[55199]: starting fuse 2023-05-22T15:42:30.701 DEBUG:teuthology.orchestra.run.smithi016:> sudo modprobe fuse 2023-05-22T15:42:30.720 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-05-22T15:42:30.720 DEBUG:teuthology.orchestra.run.smithi016:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-05-22T15:42:30.774 INFO:teuthology.orchestra.run.smithi016.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-05-22T15:42:30.775 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-05-22T15:42:30.776 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:42:30.777 DEBUG:teuthology.orchestra.run.smithi016:> ls /sys/fs/fuse/connections 2023-05-22T15:42:30.823 INFO:teuthology.orchestra.run.smithi016.stdout:45 2023-05-22T15:42:30.824 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [45] 2023-05-22T15:42:30.824 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2023-05-22T15:42:30.824 DEBUG:teuthology.orchestra.run.smithi016:> import glob 2023-05-22T15:42:30.825 DEBUG:teuthology.orchestra.run.smithi016:> import re 2023-05-22T15:42:30.825 DEBUG:teuthology.orchestra.run.smithi016:> import os 2023-05-22T15:42:30.825 DEBUG:teuthology.orchestra.run.smithi016:> import subprocess 2023-05-22T15:42:30.826 DEBUG:teuthology.orchestra.run.smithi016:> 2023-05-22T15:42:30.826 DEBUG:teuthology.orchestra.run.smithi016:> def _find_admin_socket(client_name): 2023-05-22T15:42:30.826 DEBUG:teuthology.orchestra.run.smithi016:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2023-05-22T15:42:30.827 DEBUG:teuthology.orchestra.run.smithi016:> files = glob.glob(asok_path) 2023-05-22T15:42:30.827 DEBUG:teuthology.orchestra.run.smithi016:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2023-05-22T15:42:30.827 DEBUG:teuthology.orchestra.run.smithi016:> 2023-05-22T15:42:30.828 DEBUG:teuthology.orchestra.run.smithi016:> # Given a non-glob path, it better be there 2023-05-22T15:42:30.828 DEBUG:teuthology.orchestra.run.smithi016:> if "*" not in asok_path: 2023-05-22T15:42:30.828 DEBUG:teuthology.orchestra.run.smithi016:> assert(len(files) == 1) 2023-05-22T15:42:30.829 DEBUG:teuthology.orchestra.run.smithi016:> return files[0] 2023-05-22T15:42:30.829 DEBUG:teuthology.orchestra.run.smithi016:> 2023-05-22T15:42:30.829 DEBUG:teuthology.orchestra.run.smithi016:> for f in files: 2023-05-22T15:42:30.830 DEBUG:teuthology.orchestra.run.smithi016:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-05-22T15:42:30.830 DEBUG:teuthology.orchestra.run.smithi016:> if os.path.exists("/proc/{0}".format(pid)): 2023-05-22T15:42:30.830 DEBUG:teuthology.orchestra.run.smithi016:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-05-22T15:42:30.831 DEBUG:teuthology.orchestra.run.smithi016:> contents = proc_f.read() 2023-05-22T15:42:30.831 DEBUG:teuthology.orchestra.run.smithi016:> if mountpoint in contents: 2023-05-22T15:42:30.831 DEBUG:teuthology.orchestra.run.smithi016:> return f 2023-05-22T15:42:30.831 DEBUG:teuthology.orchestra.run.smithi016:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-05-22T15:42:30.832 DEBUG:teuthology.orchestra.run.smithi016:> 2023-05-22T15:42:30.832 DEBUG:teuthology.orchestra.run.smithi016:> print(_find_admin_socket("client.0")) 2023-05-22T15:42:30.832 DEBUG:teuthology.orchestra.run.smithi016:> ' 2023-05-22T15:42:30.946 INFO:teuthology.orchestra.run.smithi016.stdout:/var/run/ceph/ceph-client.0.55199.asok 2023-05-22T15:42:31.129 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.55199.asok 2023-05-22T15:42:31.129 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:42:31.130 DEBUG:teuthology.orchestra.run.smithi016:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.55199.asok status 2023-05-22T15:42:31.289 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-05-22T15:42:31.289 INFO:teuthology.orchestra.run.smithi016.stdout: "metadata": { 2023-05-22T15:42:31.289 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph_sha1": "9c9ba4df22024d9d8d6cf9f98a4b96751fc78d0c", 2023-05-22T15:42:31.290 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph_version": "ceph version 18.0.0-3192-g9c9ba4df (9c9ba4df22024d9d8d6cf9f98a4b96751fc78d0c) reef (dev)", 2023-05-22T15:42:31.290 INFO:teuthology.orchestra.run.smithi016.stdout: "entity_id": "0", 2023-05-22T15:42:31.290 INFO:teuthology.orchestra.run.smithi016.stdout: "hostname": "smithi016", 2023-05-22T15:42:31.290 INFO:teuthology.orchestra.run.smithi016.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2023-05-22T15:42:31.290 INFO:teuthology.orchestra.run.smithi016.stdout: "pid": "55199", 2023-05-22T15:42:31.290 INFO:teuthology.orchestra.run.smithi016.stdout: "root": "/" 2023-05-22T15:42:31.291 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:42:31.291 INFO:teuthology.orchestra.run.smithi016.stdout: "dentry_count": 0, 2023-05-22T15:42:31.291 INFO:teuthology.orchestra.run.smithi016.stdout: "dentry_pinned_count": 0, 2023-05-22T15:42:31.291 INFO:teuthology.orchestra.run.smithi016.stdout: "id": 7191, 2023-05-22T15:42:31.291 INFO:teuthology.orchestra.run.smithi016.stdout: "inst": { 2023-05-22T15:42:31.291 INFO:teuthology.orchestra.run.smithi016.stdout: "name": { 2023-05-22T15:42:31.291 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "client", 2023-05-22T15:42:31.291 INFO:teuthology.orchestra.run.smithi016.stdout: "num": 7191 2023-05-22T15:42:31.292 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:42:31.292 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": { 2023-05-22T15:42:31.292 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-05-22T15:42:31.292 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "192.168.0.1:0", 2023-05-22T15:42:31.292 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 1851973161 2023-05-22T15:42:31.292 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T15:42:31.292 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:42:31.293 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": { 2023-05-22T15:42:31.293 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-05-22T15:42:31.293 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "192.168.0.1:0", 2023-05-22T15:42:31.293 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 1851973161 2023-05-22T15:42:31.293 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:42:31.293 INFO:teuthology.orchestra.run.smithi016.stdout: "inst_str": "client.7191 192.168.0.1:0/1851973161", 2023-05-22T15:42:31.293 INFO:teuthology.orchestra.run.smithi016.stdout: "addr_str": "192.168.0.1:0/1851973161", 2023-05-22T15:42:31.293 INFO:teuthology.orchestra.run.smithi016.stdout: "inode_count": 1, 2023-05-22T15:42:31.294 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_epoch": 30, 2023-05-22T15:42:31.294 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_epoch": 90, 2023-05-22T15:42:31.294 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_epoch_barrier": 0, 2023-05-22T15:42:31.294 INFO:teuthology.orchestra.run.smithi016.stdout: "blocklisted": false, 2023-05-22T15:42:31.294 INFO:teuthology.orchestra.run.smithi016.stdout: "fs_name": "cephfs" 2023-05-22T15:42:31.294 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-05-22T15:42:31.302 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:42:31.303 DEBUG:teuthology.orchestra.run.smithi016:> stat --file-system '--printf=%T 2023-05-22T15:42:31.303 DEBUG:teuthology.orchestra.run.smithi016:> ' -- /home/ubuntu/cephtest/mnt.0 2023-05-22T15:42:31.310 INFO:teuthology.orchestra.run.smithi016.stdout:fuseblk 2023-05-22T15:42:31.310 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2023-05-22T15:42:31.310 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:42:31.311 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-05-22T15:42:31.366 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2023-05-22T15:42:31.539 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:31.536+0000 7f023f286640 1 Processor -- start 2023-05-22T15:42:31.540 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:31.536+0000 7f023f286640 1 -- start start 2023-05-22T15:42:31.540 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:31.536+0000 7f023f286640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f02380c9b40 0x7f023815e460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:31.541 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:31.536+0000 7f023f286640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f0238070df0 0x7f0238071210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:31.541 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:31.536+0000 7f023f286640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02380b0650 con 0x7f02380ca4e0 2023-05-22T15:42:31.541 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:31.536+0000 7f023f286640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02380b0370 con 0x7f0238070df0 2023-05-22T15:42:31.541 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:31.536+0000 7f023f286640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f023815d980 con 0x7f02380c9b40 2023-05-22T15:42:31.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:31.536+0000 7f023c821640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f02380c9b40 0x7f023815e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:31.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:31.536+0000 7f023d823640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f0238070df0 0x7f0238071210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:31.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:31.536+0000 7f023c821640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f02380c9b40 0x7f023815e460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3301/0 says I am v2:172.21.15.16:51988/0 (socket says 172.21.15.16:51988) 2023-05-22T15:42:31.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:31.536+0000 7f023d823640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f0238070df0 0x7f0238071210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.16:49126/0 (socket says 172.21.15.16:49126) 2023-05-22T15:42:31.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:31.536+0000 7f023c821640 1 -- 172.21.15.16:0/616233917 learned_addr learned my addr 172.21.15.16:0/616233917 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:42:31.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:31.536+0000 7f023d823640 1 -- 172.21.15.16:0/616233917 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f02380c9b40 msgr2=0x7f023815e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:42:31.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:31.536+0000 7f023d823640 1 --2- 172.21.15.16:0/616233917 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f02380c9b40 0x7f023815e460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:31.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:31.536+0000 7f023d823640 1 -- 172.21.15.16:0/616233917 >> v1:172.21.15.62:6789/0 conn(0x7f02380ca4e0 legacy=0x7f02380afb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:31.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:31.536+0000 7f023d823640 1 -- 172.21.15.16:0/616233917 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f023815dc60 con 0x7f0238070df0 2023-05-22T15:42:31.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:31.540+0000 7f023d823640 1 --2- 172.21.15.16:0/616233917 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0238070df0 0x7f0238071210 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f02300097c0 tx=0x7f023001c3c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8f6cb38f155072ff server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:31.544 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:31.540+0000 7f022ffff640 1 -- 172.21.15.16:0/616233917 <== mon.1 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02300ab600 con 0x7f0238070df0 2023-05-22T15:42:31.544 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:31.540+0000 7f022ffff640 1 -- 172.21.15.16:0/616233917 <== mon.1 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02300a7050 con 0x7f0238070df0 2023-05-22T15:42:31.544 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:31.540+0000 7f022ffff640 1 -- 172.21.15.16:0/616233917 <== mon.1 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02300ab980 con 0x7f0238070df0 2023-05-22T15:42:31.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:31.540+0000 7f023f286640 1 -- 172.21.15.16:0/616233917 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0238070df0 msgr2=0x7f0238071210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:31.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:31.540+0000 7f023f286640 1 --2- 172.21.15.16:0/616233917 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0238070df0 0x7f0238071210 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f02300097c0 tx=0x7f023001c3c0 comp rx=0 tx=0).stop 2023-05-22T15:42:31.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:31.540+0000 7f023f286640 1 -- 172.21.15.16:0/616233917 shutdown_connections 2023-05-22T15:42:31.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:31.540+0000 7f023f286640 1 --2- 172.21.15.16:0/616233917 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0238070df0 0x7f0238071210 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:31.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:31.540+0000 7f023f286640 1 --2- 172.21.15.16:0/616233917 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f02380c9b40 0x7f023815e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:31.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:31.540+0000 7f023f286640 1 -- 172.21.15.16:0/616233917 >> 172.21.15.16:0/616233917 conn(0x7f0238069720 msgr2=0x7f0238069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:31.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:31.540+0000 7f023f286640 1 -- 172.21.15.16:0/616233917 shutdown_connections 2023-05-22T15:42:31.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:31.540+0000 7f023f286640 1 -- 172.21.15.16:0/616233917 wait complete. 2023-05-22T15:42:31.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:31.544+0000 7f023f286640 1 Processor -- start 2023-05-22T15:42:31.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:31.544+0000 7f023f286640 1 -- start start 2023-05-22T15:42:31.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:31.544+0000 7f023f286640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0238070df0 0x7f02380cd000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:31.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:31.544+0000 7f023f286640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f02380c9b40 0x7f02381525f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:31.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:31.544+0000 7f023f286640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f02380ca4e0 0x7f023815c480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:31.549 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:31.544+0000 7f023f286640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0238152b30 con 0x7f02380ca4e0 2023-05-22T15:42:31.549 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:31.544+0000 7f023f286640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0238162c60 con 0x7f0238070df0 2023-05-22T15:42:31.549 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:31.544+0000 7f023f286640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f023815c9c0 con 0x7f02380c9b40 2023-05-22T15:42:31.549 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:31.544+0000 7f023d823640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f02380ca4e0 0x7f023815c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:31.549 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:31.544+0000 7f023d823640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f02380ca4e0 0x7f023815c480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56870/0 (socket says 172.21.15.16:56870) 2023-05-22T15:42:31.549 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:31.544+0000 7f023d823640 1 -- 172.21.15.16:0/1220332124 learned_addr learned my addr 172.21.15.16:0/1220332124 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:42:31.549 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:31.544+0000 7f023c821640 1 --2- 172.21.15.16:0/1220332124 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f02380c9b40 0x7f02381525f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:31.550 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:31.544+0000 7f023d022640 1 --2- 172.21.15.16:0/1220332124 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0238070df0 0x7f02380cd000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:31.550 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:31.544+0000 7f023d823640 1 -- 172.21.15.16:0/1220332124 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f02380c9b40 msgr2=0x7f02381525f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:31.550 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:31.544+0000 7f023d823640 1 --2- 172.21.15.16:0/1220332124 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f02380c9b40 0x7f02381525f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:31.550 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:31.544+0000 7f023d823640 1 -- 172.21.15.16:0/1220332124 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0238070df0 msgr2=0x7f02380cd000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:31.550 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:31.544+0000 7f023d823640 1 --2- 172.21.15.16:0/1220332124 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0238070df0 0x7f02380cd000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:31.550 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:31.544+0000 7f023d823640 1 -- 172.21.15.16:0/1220332124 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0230009420 con 0x7f02380ca4e0 2023-05-22T15:42:31.550 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:31.544+0000 7f023c821640 1 --2- 172.21.15.16:0/1220332124 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f02380c9b40 0x7f02381525f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T15:42:31.551 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:31.544+0000 7f023d823640 1 --2- 172.21.15.16:0/1220332124 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f02380ca4e0 0x7f023815c480 secure :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 crypto rx=0x7f02300097c0 tx=0x7f0230019160 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:31.551 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:31.544+0000 7f022dffb640 1 -- 172.21.15.16:0/1220332124 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02300b4430 con 0x7f02380ca4e0 2023-05-22T15:42:31.551 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:31.544+0000 7f022dffb640 1 -- 172.21.15.16:0/1220332124 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02300b4590 con 0x7f02380ca4e0 2023-05-22T15:42:31.551 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:31.544+0000 7f022dffb640 1 -- 172.21.15.16:0/1220332124 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02300b4920 con 0x7f02380ca4e0 2023-05-22T15:42:31.551 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:31.544+0000 7f023f286640 1 -- 172.21.15.16:0/1220332124 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02380c8e80 con 0x7f02380ca4e0 2023-05-22T15:42:31.551 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:31.544+0000 7f023f286640 1 -- 172.21.15.16:0/1220332124 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02380ca8b0 con 0x7f02380ca4e0 2023-05-22T15:42:31.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:31.544+0000 7f023f286640 1 -- 172.21.15.16:0/1220332124 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02380c9010 con 0x7f02380ca4e0 2023-05-22T15:42:31.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:31.544+0000 7f022dffb640 1 -- 172.21.15.16:0/1220332124 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f02300b2040 con 0x7f02380ca4e0 2023-05-22T15:42:31.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:31.544+0000 7f022dffb640 1 --2- 172.21.15.16:0/1220332124 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f021c0654a0 0x7f021c067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:31.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:31.548+0000 7f022dffb640 1 -- 172.21.15.16:0/1220332124 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f023011ca20 con 0x7f02380ca4e0 2023-05-22T15:42:31.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:31.548+0000 7f023d022640 1 --2- 172.21.15.16:0/1220332124 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f021c0654a0 0x7f021c067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:31.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:31.548+0000 7f023d022640 1 --2- 172.21.15.16:0/1220332124 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f021c0654a0 0x7f021c067950 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f02340012f0 tx=0x7f0234009040 comp rx=0 tx=0).ready entity=mgr.6765 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:31.554 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:31.552+0000 7f022dffb640 1 -- 172.21.15.16:0/1220332124 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f02300ea3d0 con 0x7f02380ca4e0 2023-05-22T15:42:31.730 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:31.728+0000 7f023f286640 1 -- 172.21.15.16:0/1220332124 --> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f02380660a0 con 0x7f021c0654a0 2023-05-22T15:42:31.731 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:31.728+0000 7f022dffb640 1 -- 172.21.15.16:0/1220332124 <== mgr.6765 v2:172.21.15.16:6832/36921 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f02380660a0 con 0x7f021c0654a0 2023-05-22T15:42:31.732 INFO:teuthology.orchestra.run.smithi016.stdout:[ 2023-05-22T15:42:31.732 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T15:42:31.732 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "cephfs" 2023-05-22T15:42:31.732 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T15:42:31.732 INFO:teuthology.orchestra.run.smithi016.stdout:] 2023-05-22T15:42:31.734 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:31.732+0000 7f023f286640 1 -- 172.21.15.16:0/1220332124 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f021c0654a0 msgr2=0x7f021c067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:31.735 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:31.732+0000 7f023f286640 1 --2- 172.21.15.16:0/1220332124 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f021c0654a0 0x7f021c067950 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f02340012f0 tx=0x7f0234009040 comp rx=0 tx=0).stop 2023-05-22T15:42:31.735 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:31.732+0000 7f023f286640 1 -- 172.21.15.16:0/1220332124 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f02380ca4e0 msgr2=0x7f023815c480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:31.735 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:31.732+0000 7f023f286640 1 --2- 172.21.15.16:0/1220332124 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f02380ca4e0 0x7f023815c480 secure :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 crypto rx=0x7f02300097c0 tx=0x7f0230019160 comp rx=0 tx=0).stop 2023-05-22T15:42:31.735 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:31.732+0000 7f023f286640 1 -- 172.21.15.16:0/1220332124 shutdown_connections 2023-05-22T15:42:31.735 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:31.732+0000 7f023f286640 1 --2- 172.21.15.16:0/1220332124 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f02380ca4e0 0x7f023815c480 unknown :-1 s=CLOSED pgs=1134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:31.735 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:31.732+0000 7f023f286640 1 --2- 172.21.15.16:0/1220332124 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f021c0654a0 0x7f021c067950 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:31.736 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:31.732+0000 7f023f286640 1 --2- 172.21.15.16:0/1220332124 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f02380c9b40 0x7f02381525f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:31.736 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:31.732+0000 7f023f286640 1 --2- 172.21.15.16:0/1220332124 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0238070df0 0x7f02380cd000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:31.736 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:31.732+0000 7f023f286640 1 -- 172.21.15.16:0/1220332124 >> 172.21.15.16:0/1220332124 conn(0x7f0238069720 msgr2=0x7f02380adda0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:31.736 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:31.732+0000 7f023f286640 1 -- 172.21.15.16:0/1220332124 shutdown_connections 2023-05-22T15:42:31.736 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:31.732+0000 7f023f286640 1 -- 172.21.15.16:0/1220332124 wait complete. 2023-05-22T15:42:31.751 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2023-05-22T15:42:31.877 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:31.872+0000 7fb23aa80640 1 Processor -- start 2023-05-22T15:42:31.878 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:31.872+0000 7fb23aa80640 1 -- start start 2023-05-22T15:42:31.878 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:31.876+0000 7fb23aa80640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb234159d40 0x7fb2340cb260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:31.879 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:31.876+0000 7fb23aa80640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb23406b6c0 0x7fb23406bb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:31.879 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:31.876+0000 7fb23aa80640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb234070c30 con 0x7fb234159970 2023-05-22T15:42:31.879 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:31.876+0000 7fb23aa80640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb23415dd90 con 0x7fb23406b6c0 2023-05-22T15:42:31.879 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:31.876+0000 7fb23aa80640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb23415a190 con 0x7fb234159d40 2023-05-22T15:42:31.879 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:31.876+0000 7fb23901d640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb23406b6c0 0x7fb23406bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:31.879 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:31.876+0000 7fb23881c640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fb234159970 0x7fb23415d680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:43638/0 (socket says 172.21.15.16:43638) 2023-05-22T15:42:31.879 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:31.876+0000 7fb23881c640 1 -- 172.21.15.16:0/1308173576 learned_addr learned my addr 172.21.15.16:0/1308173576 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:42:31.880 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:31.876+0000 7fb23901d640 1 --2- 172.21.15.16:0/1308173576 >> v2:172.21.15.16:3300/0 conn(0x7fb23406b6c0 0x7fb23406bb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56878/0 (socket says 172.21.15.16:56878) 2023-05-22T15:42:31.880 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:31.876+0000 7fb233fff640 1 --2- 172.21.15.16:0/1308173576 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb234159d40 0x7fb2340cb260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:31.880 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:31.876+0000 7fb23901d640 1 -- 172.21.15.16:0/1308173576 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb234159d40 msgr2=0x7fb2340cb260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:42:31.881 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:31.876+0000 7fb23901d640 1 --2- 172.21.15.16:0/1308173576 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb234159d40 0x7fb2340cb260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:31.881 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:31.876+0000 7fb23901d640 1 -- 172.21.15.16:0/1308173576 >> v1:172.21.15.16:6789/0 conn(0x7fb234159970 legacy=0x7fb23415d680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:31.881 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:31.876+0000 7fb23901d640 1 -- 172.21.15.16:0/1308173576 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb23406c060 con 0x7fb23406b6c0 2023-05-22T15:42:31.882 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:31.876+0000 7fb23901d640 1 --2- 172.21.15.16:0/1308173576 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb23406b6c0 0x7fb23406bb20 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 crypto rx=0x7fb22c009d90 tx=0x7fb22c01c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12c3e00da846014d server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:31.882 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:31.876+0000 7fb2337fe640 1 -- 172.21.15.16:0/1308173576 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb22c0a5950 con 0x7fb23406b6c0 2023-05-22T15:42:31.882 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:31.876+0000 7fb2337fe640 1 -- 172.21.15.16:0/1308173576 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb22c0a5ab0 con 0x7fb23406b6c0 2023-05-22T15:42:31.882 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:31.876+0000 7fb2337fe640 1 -- 172.21.15.16:0/1308173576 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb22c0a5e40 con 0x7fb23406b6c0 2023-05-22T15:42:31.882 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:31.876+0000 7fb23aa80640 1 -- 172.21.15.16:0/1308173576 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb23406b6c0 msgr2=0x7fb23406bb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:31.883 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:31.876+0000 7fb23aa80640 1 --2- 172.21.15.16:0/1308173576 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb23406b6c0 0x7fb23406bb20 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 crypto rx=0x7fb22c009d90 tx=0x7fb22c01c890 comp rx=0 tx=0).stop 2023-05-22T15:42:31.883 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:31.880+0000 7fb23aa80640 1 -- 172.21.15.16:0/1308173576 shutdown_connections 2023-05-22T15:42:31.883 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:31.880+0000 7fb23aa80640 1 --2- 172.21.15.16:0/1308173576 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb23406b6c0 0x7fb23406bb20 unknown :-1 s=CLOSED pgs=1136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:31.883 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:31.880+0000 7fb23aa80640 1 --2- 172.21.15.16:0/1308173576 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb234159d40 0x7fb2340cb260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:31.883 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:31.880+0000 7fb23aa80640 1 -- 172.21.15.16:0/1308173576 >> 172.21.15.16:0/1308173576 conn(0x7fb234069720 msgr2=0x7fb234069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:31.883 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:31.880+0000 7fb23aa80640 1 -- 172.21.15.16:0/1308173576 shutdown_connections 2023-05-22T15:42:31.883 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:31.880+0000 7fb23aa80640 1 -- 172.21.15.16:0/1308173576 wait complete. 2023-05-22T15:42:31.885 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:31.880+0000 7fb23aa80640 1 Processor -- start 2023-05-22T15:42:31.885 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:31.880+0000 7fb23aa80640 1 -- start start 2023-05-22T15:42:31.886 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:31.880+0000 7fb23aa80640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb23406b6c0 0x7fb23414c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:31.886 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:31.880+0000 7fb23aa80640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb234159970 0x7fb2340b2ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:31.886 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:31.880+0000 7fb23aa80640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb234159d40 0x7fb2340b3010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:31.886 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:31.880+0000 7fb23aa80640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb234143cb0 con 0x7fb23406b6c0 2023-05-22T15:42:31.887 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:31.880+0000 7fb23aa80640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2340c2d20 con 0x7fb234159d40 2023-05-22T15:42:31.887 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:31.880+0000 7fb23aa80640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2340c3000 con 0x7fb234159970 2023-05-22T15:42:31.887 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:31.880+0000 7fb23881c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb23406b6c0 0x7fb23414c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:31.887 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:31.880+0000 7fb23881c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb23406b6c0 0x7fb23414c7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56890/0 (socket says 172.21.15.16:56890) 2023-05-22T15:42:31.888 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:31.880+0000 7fb23881c640 1 -- 172.21.15.16:0/1722355498 learned_addr learned my addr 172.21.15.16:0/1722355498 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:42:31.888 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:31.880+0000 7fb233fff640 1 --2- 172.21.15.16:0/1722355498 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb234159970 0x7fb2340b2ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:31.888 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:31.880+0000 7fb23881c640 1 -- 172.21.15.16:0/1722355498 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb234159970 msgr2=0x7fb2340b2ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:31.888 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:31.880+0000 7fb23881c640 1 --2- 172.21.15.16:0/1722355498 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb234159970 0x7fb2340b2ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:31.888 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:31.880+0000 7fb23881c640 1 -- 172.21.15.16:0/1722355498 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb234159d40 msgr2=0x7fb2340b3010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:31.889 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:31.880+0000 7fb23881c640 1 --2- 172.21.15.16:0/1722355498 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb234159d40 0x7fb2340b3010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:31.889 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:31.880+0000 7fb23881c640 1 -- 172.21.15.16:0/1722355498 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb22c009a70 con 0x7fb23406b6c0 2023-05-22T15:42:31.889 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:31.880+0000 7fb233fff640 1 --2- 172.21.15.16:0/1722355498 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb234159970 0x7fb2340b2ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:42:31.889 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:31.880+0000 7fb23881c640 1 --2- 172.21.15.16:0/1722355498 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb23406b6c0 0x7fb23414c7a0 secure :-1 s=READY pgs=1137 cs=0 l=1 rev1=1 crypto rx=0x7fb22400c500 tx=0x7fb22400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:31.890 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:31.880+0000 7fb2317fa640 1 -- 172.21.15.16:0/1722355498 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb224005cf0 con 0x7fb23406b6c0 2023-05-22T15:42:31.890 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:31.880+0000 7fb2317fa640 1 -- 172.21.15.16:0/1722355498 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb224005e50 con 0x7fb23406b6c0 2023-05-22T15:42:31.890 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:31.880+0000 7fb2317fa640 1 -- 172.21.15.16:0/1722355498 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2240072f0 con 0x7fb23406b6c0 2023-05-22T15:42:31.890 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:31.880+0000 7fb23aa80640 1 -- 172.21.15.16:0/1722355498 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb234071700 con 0x7fb23406b6c0 2023-05-22T15:42:31.891 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:31.880+0000 7fb23aa80640 1 -- 172.21.15.16:0/1722355498 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb234071420 con 0x7fb23406b6c0 2023-05-22T15:42:31.892 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:31.884+0000 7fb23aa80640 1 -- 172.21.15.16:0/1722355498 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2340cb7a0 con 0x7fb23406b6c0 2023-05-22T15:42:31.893 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:31.884+0000 7fb2317fa640 1 -- 172.21.15.16:0/1722355498 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) v1 ==== 84332+0+0 (secure 0 0 0) 0x7fb224007450 con 0x7fb23406b6c0 2023-05-22T15:42:31.893 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:31.884+0000 7fb2317fa640 1 --2- 172.21.15.16:0/1722355498 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fb2200654a0 0x7fb220067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:31.893 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:31.884+0000 7fb2317fa640 1 -- 172.21.15.16:0/1722355498 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 6382+0+0 (secure 0 0 0) 0x7fb224081f90 con 0x7fb23406b6c0 2023-05-22T15:42:31.893 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:31.884+0000 7fb233fff640 1 --2- 172.21.15.16:0/1722355498 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fb2200654a0 0x7fb220067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:31.894 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:31.884+0000 7fb233fff640 1 --2- 172.21.15.16:0/1722355498 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fb2200654a0 0x7fb220067950 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fb23406be20 tx=0x7fb228008040 comp rx=0 tx=0).ready entity=mgr.6765 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:31.894 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:31.888+0000 7fb2317fa640 1 -- 172.21.15.16:0/1722355498 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fb22404a620 con 0x7fb23406b6c0 2023-05-22T15:42:32.079 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:32.076+0000 7fb23aa80640 1 -- 172.21.15.16:0/1722355498 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7fb23404dbe0 con 0x7fb23406b6c0 2023-05-22T15:42:32.456 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:32.452+0000 7fb2317fa640 1 -- 172.21.15.16:0/1722355498 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mgrmap(e 27) v1 ==== 84349+0+0 (secure 0 0 0) 0x7fb22404fed0 con 0x7fb23406b6c0 2023-05-22T15:42:32.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:32.456+0000 7fb2317fa640 1 -- 172.21.15.16:0/1722355498 <== mon.0 v2:172.21.15.16:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v27) v1 ==== 92+0+0 (secure 0 0 0) 0x7fb22405b9f0 con 0x7fb23406b6c0 2023-05-22T15:42:32.462 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:32.456+0000 7fb23aa80640 1 -- 172.21.15.16:0/1722355498 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fb2200654a0 msgr2=0x7fb220067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:32.462 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:32.456+0000 7fb23aa80640 1 --2- 172.21.15.16:0/1722355498 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fb2200654a0 0x7fb220067950 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fb23406be20 tx=0x7fb228008040 comp rx=0 tx=0).stop 2023-05-22T15:42:32.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:32.456+0000 7fb23aa80640 1 -- 172.21.15.16:0/1722355498 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb23406b6c0 msgr2=0x7fb23414c7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:32.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:32.456+0000 7fb23aa80640 1 --2- 172.21.15.16:0/1722355498 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb23406b6c0 0x7fb23414c7a0 secure :-1 s=READY pgs=1137 cs=0 l=1 rev1=1 crypto rx=0x7fb22400c500 tx=0x7fb22400c9c0 comp rx=0 tx=0).stop 2023-05-22T15:42:32.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:32.456+0000 7fb23aa80640 1 -- 172.21.15.16:0/1722355498 shutdown_connections 2023-05-22T15:42:32.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:32.456+0000 7fb23aa80640 1 --2- 172.21.15.16:0/1722355498 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb234159d40 0x7fb2340b3010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:32.464 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:32.456+0000 7fb23aa80640 1 --2- 172.21.15.16:0/1722355498 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fb2200654a0 0x7fb220067950 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:32.464 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:32.456+0000 7fb23aa80640 1 --2- 172.21.15.16:0/1722355498 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb234159970 0x7fb2340b2ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:32.464 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:32.456+0000 7fb23aa80640 1 --2- 172.21.15.16:0/1722355498 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb23406b6c0 0x7fb23414c7a0 unknown :-1 s=CLOSED pgs=1137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:32.465 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:32.456+0000 7fb23aa80640 1 -- 172.21.15.16:0/1722355498 >> 172.21.15.16:0/1722355498 conn(0x7fb234069720 msgr2=0x7fb2340c94f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:32.465 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:32.456+0000 7fb23aa80640 1 -- 172.21.15.16:0/1722355498 shutdown_connections 2023-05-22T15:42:32.465 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:32.456+0000 7fb23aa80640 1 -- 172.21.15.16:0/1722355498 wait complete. 2023-05-22T15:42:32.483 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2023-05-22T15:42:32.584 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:42:32.580+0000 7ff4cebf1180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-05-22T15:42:32.602 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:32.596+0000 7f2ab2a40640 1 Processor -- start 2023-05-22T15:42:32.602 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:32.596+0000 7f2ab2a40640 1 -- start start 2023-05-22T15:42:32.604 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:32.600+0000 7f2ab2a40640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2aac15bca0 0x7f2aac0cda80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:32.604 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:32.600+0000 7f2ab2a40640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2aac0cccc0 con 0x7f2aac0b5a50 2023-05-22T15:42:32.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:32.600+0000 7f2ab2a40640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2aac0ccfa0 con 0x7f2aac0d2480 2023-05-22T15:42:32.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:32.600+0000 7f2ab2a40640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2aac0cd660 con 0x7f2aac15bca0 2023-05-22T15:42:32.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:32.600+0000 7f2ab0fdd640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f2aac0b5a50 0x7f2aac15fb50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:43646/0 (socket says 172.21.15.16:43646) 2023-05-22T15:42:32.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:32.600+0000 7f2ab0fdd640 1 -- 172.21.15.16:0/932755021 learned_addr learned my addr 172.21.15.16:0/932755021 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:42:32.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:32.600+0000 7f2aabfff640 1 --2- 172.21.15.16:0/932755021 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2aac15bca0 0x7f2aac0cda80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:32.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:32.600+0000 7f2aaaffd640 1 -- 172.21.15.16:0/932755021 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 559632205 0 0) 0x7f2aac0cccc0 con 0x7f2aac0b5a50 2023-05-22T15:42:32.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:32.600+0000 7f2aaaffd640 1 -- 172.21.15.16:0/932755021 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a98003530 con 0x7f2aac0b5a50 2023-05-22T15:42:32.606 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:32.600+0000 7f2aaaffd640 1 -- 172.21.15.16:0/932755021 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2695701735 0 0) 0x7f2a98003530 con 0x7f2aac0b5a50 2023-05-22T15:42:32.606 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:32.600+0000 7f2aaaffd640 1 -- 172.21.15.16:0/932755021 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2aac15bca0 msgr2=0x7f2aac0cda80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:42:32.606 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:32.600+0000 7f2aaaffd640 1 --2- 172.21.15.16:0/932755021 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2aac15bca0 0x7f2aac0cda80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:32.606 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:32.600+0000 7f2aaaffd640 1 -- 172.21.15.16:0/932755021 >> v1:172.21.15.62:6789/0 conn(0x7f2aac0d2480 legacy=0x7f2aac0b5340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:32.606 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:32.600+0000 7f2aaaffd640 1 -- 172.21.15.16:0/932755021 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2aac0cd280 con 0x7f2aac0b5a50 2023-05-22T15:42:32.606 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:32.600+0000 7f2aaaffd640 1 -- 172.21.15.16:0/932755021 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7f2aa4001680 con 0x7f2aac0b5a50 2023-05-22T15:42:32.606 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:32.600+0000 7f2aaaffd640 1 -- 172.21.15.16:0/932755021 >> v1:172.21.15.16:6789/0 conn(0x7f2aac0b5a50 legacy=0x7f2aac15fb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:32.606 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:32.600+0000 7f2aaaffd640 1 --2- 172.21.15.16:0/932755021 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2a980039b0 0x7f2a98003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:32.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:32.600+0000 7f2aaaffd640 1 --2- 172.21.15.16:0/932755021 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2aac15bca0 0x7f2a980047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:32.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:32.600+0000 7f2aaaffd640 1 --2- 172.21.15.16:0/932755021 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2a98004d10 0x7f2a98005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:32.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:32.600+0000 7f2aaaffd640 1 -- 172.21.15.16:0/932755021 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2aac0cd660 con 0x7f2a98004d10 2023-05-22T15:42:32.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:32.600+0000 7f2aaaffd640 1 -- 172.21.15.16:0/932755021 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a9800ba40 con 0x7f2a980039b0 2023-05-22T15:42:32.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:32.600+0000 7f2aaaffd640 1 -- 172.21.15.16:0/932755021 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a9800bbd0 con 0x7f2aac15bca0 2023-05-22T15:42:32.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:32.600+0000 7f2ab0fdd640 1 --2- 172.21.15.16:0/932755021 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2a98004d10 0x7f2a98005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:32.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:32.600+0000 7f2aabfff640 1 --2- 172.21.15.16:0/932755021 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2a980039b0 0x7f2a98003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:32.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:32.600+0000 7f2aab7fe640 1 --2- 172.21.15.16:0/932755021 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2aac15bca0 0x7f2a980047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:32.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:32.600+0000 7f2ab0fdd640 1 -- 172.21.15.16:0/932755021 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2aac15bca0 msgr2=0x7f2a980047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:42:32.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:32.600+0000 7f2ab0fdd640 1 --2- 172.21.15.16:0/932755021 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2aac15bca0 0x7f2a980047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:32.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:32.600+0000 7f2ab0fdd640 1 -- 172.21.15.16:0/932755021 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2a980039b0 msgr2=0x7f2a98003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:42:32.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:32.600+0000 7f2ab0fdd640 1 --2- 172.21.15.16:0/932755021 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2a980039b0 0x7f2a98003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:32.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:32.600+0000 7f2ab0fdd640 1 -- 172.21.15.16:0/932755021 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a98003530 con 0x7f2a98004d10 2023-05-22T15:42:32.610 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:32.604+0000 7f2ab0fdd640 1 --2- 172.21.15.16:0/932755021 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2a98004d10 0x7f2a98005180 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 crypto rx=0x7f2aa4001060 tx=0x7f2aa401cc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed5e04fcb9be6c1a server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:32.610 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:32.604+0000 7f2aaaffd640 1 -- 172.21.15.16:0/932755021 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2aa40a5900 con 0x7f2a98004d10 2023-05-22T15:42:32.611 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:32.604+0000 7f2aaaffd640 1 -- 172.21.15.16:0/932755021 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2aa40a5a80 con 0x7f2a98004d10 2023-05-22T15:42:32.611 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:32.604+0000 7f2aaaffd640 1 -- 172.21.15.16:0/932755021 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2aa40a5e30 con 0x7f2a98004d10 2023-05-22T15:42:32.611 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:32.604+0000 7f2ab2a40640 1 -- 172.21.15.16:0/932755021 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2a98004d10 msgr2=0x7f2a98005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:32.611 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:32.604+0000 7f2ab2a40640 1 --2- 172.21.15.16:0/932755021 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2a98004d10 0x7f2a98005180 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 crypto rx=0x7f2aa4001060 tx=0x7f2aa401cc00 comp rx=0 tx=0).stop 2023-05-22T15:42:32.611 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:32.604+0000 7f2ab0fdd640 1 -- 172.21.15.16:0/932755021 reap_dead start 2023-05-22T15:42:32.611 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:32.604+0000 7f2ab2a40640 1 -- 172.21.15.16:0/932755021 shutdown_connections 2023-05-22T15:42:32.611 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:32.604+0000 7f2ab2a40640 1 -- 172.21.15.16:0/932755021 >> 172.21.15.16:0/932755021 conn(0x7f2aac0655c0 msgr2=0x7f2aac070e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:32.612 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:32.604+0000 7f2ab2a40640 1 -- 172.21.15.16:0/932755021 shutdown_connections 2023-05-22T15:42:32.612 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:32.604+0000 7f2ab2a40640 1 -- 172.21.15.16:0/932755021 wait complete. 2023-05-22T15:42:32.612 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:32.604+0000 7f2ab2a40640 1 Processor -- start 2023-05-22T15:42:32.612 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:32.604+0000 7f2ab2a40640 1 -- start start 2023-05-22T15:42:32.612 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:32.604+0000 7f2ab2a40640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2a98004d10 0x7f2aac14ad80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:32.612 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:32.604+0000 7f2ab2a40640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2aac14d770 0x7f2aac0c88c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:32.612 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:32.604+0000 7f2ab2a40640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2aac14dbc0 0x7f2aac159950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:32.612 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:32.604+0000 7f2ab2a40640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2aac14b2c0 con 0x7f2aac14d770 2023-05-22T15:42:32.613 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:32.604+0000 7f2ab2a40640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2aac147a30 con 0x7f2aac14dbc0 2023-05-22T15:42:32.613 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:32.604+0000 7f2ab2a40640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2aac147cd0 con 0x7f2a98004d10 2023-05-22T15:42:32.613 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:32.604+0000 7f2aab7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2aac14d770 0x7f2aac0c88c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:32.613 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:32.604+0000 7f2aab7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2aac14d770 0x7f2aac0c88c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56918/0 (socket says 172.21.15.16:56918) 2023-05-22T15:42:32.613 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:32.604+0000 7f2aab7fe640 1 -- 172.21.15.16:0/2896677989 learned_addr learned my addr 172.21.15.16:0/2896677989 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:42:32.613 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:32.604+0000 7f2aabfff640 1 --2- 172.21.15.16:0/2896677989 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2a98004d10 0x7f2aac14ad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:32.613 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:32.604+0000 7f2ab0fdd640 1 --2- 172.21.15.16:0/2896677989 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2aac14dbc0 0x7f2aac159950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:32.614 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:32.604+0000 7f2aab7fe640 1 -- 172.21.15.16:0/2896677989 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2a98004d10 msgr2=0x7f2aac14ad80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:32.614 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:32.604+0000 7f2aab7fe640 1 --2- 172.21.15.16:0/2896677989 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2a98004d10 0x7f2aac14ad80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:32.614 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:32.604+0000 7f2aab7fe640 1 -- 172.21.15.16:0/2896677989 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2aac14dbc0 msgr2=0x7f2aac159950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:32.614 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:32.604+0000 7f2aab7fe640 1 --2- 172.21.15.16:0/2896677989 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2aac14dbc0 0x7f2aac159950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:32.614 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:32.604+0000 7f2aab7fe640 1 -- 172.21.15.16:0/2896677989 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2aac160260 con 0x7f2aac14d770 2023-05-22T15:42:32.614 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:32.608+0000 7f2aab7fe640 1 --2- 172.21.15.16:0/2896677989 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2aac14d770 0x7f2aac0c88c0 secure :-1 s=READY pgs=1142 cs=0 l=1 rev1=1 crypto rx=0x7f2a9400e4b0 tx=0x7f2a9400e970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:32.614 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:32.608+0000 7f2aa8ff9640 1 -- 172.21.15.16:0/2896677989 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a940052e0 con 0x7f2aac14d770 2023-05-22T15:42:32.614 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:32.608+0000 7f2aa8ff9640 1 -- 172.21.15.16:0/2896677989 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a94005440 con 0x7f2aac14d770 2023-05-22T15:42:32.615 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:32.608+0000 7f2aa8ff9640 1 -- 172.21.15.16:0/2896677989 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a94003840 con 0x7f2aac14d770 2023-05-22T15:42:32.615 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:32.608+0000 7f2ab2a40640 1 -- 172.21.15.16:0/2896677989 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2aac15c070 con 0x7f2aac14d770 2023-05-22T15:42:32.615 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:32.608+0000 7f2ab2a40640 1 -- 172.21.15.16:0/2896677989 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2aac1603f0 con 0x7f2aac14d770 2023-05-22T15:42:32.616 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:32.608+0000 7f2ab2a40640 1 -- 172.21.15.16:0/2896677989 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2aac04dbe0 con 0x7f2aac14d770 2023-05-22T15:42:32.616 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:32.608+0000 7f2aa8ff9640 1 -- 172.21.15.16:0/2896677989 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 27) v1 ==== 84349+0+0 (secure 0 0 0) 0x7f2a940039a0 con 0x7f2aac14d770 2023-05-22T15:42:32.616 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:32.608+0000 7f2aa8ff9640 1 --2- 172.21.15.16:0/2896677989 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f2a8c0654f0 0x7f2a8c0679a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:32.616 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:32.608+0000 7f2aabfff640 1 -- 172.21.15.16:0/2896677989 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f2a8c0654f0 msgr2=0x7f2a8c0679a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/36921 2023-05-22T15:42:32.616 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:32.608+0000 7f2aa8ff9640 1 -- 172.21.15.16:0/2896677989 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f2a94082910 con 0x7f2aac14d770 2023-05-22T15:42:32.616 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:32.608+0000 7f2aabfff640 1 --2- 172.21.15.16:0/2896677989 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f2a8c0654f0 0x7f2a8c0679a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-05-22T15:42:32.617 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:32.612+0000 7f2aa8ff9640 1 -- 172.21.15.16:0/2896677989 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f2a940502e0 con 0x7f2aac14d770 2023-05-22T15:42:32.624 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:42:32.619+0000 7fea464da180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-05-22T15:42:32.772 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:32.768+0000 7f2ab2a40640 1 -- 172.21.15.16:0/2896677989 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7f2aac0b3e50 con 0x7f2aac14d770 2023-05-22T15:42:32.778 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:32.776+0000 7f2aa8ff9640 1 -- 172.21.15.16:0/2896677989 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v23)=0 v23) v1 ==== 138+0+0 (secure 0 0 0) 0x7f2a94053900 con 0x7f2aac14d770 2023-05-22T15:42:32.780 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:32.776+0000 7f2ab2a40640 1 -- 172.21.15.16:0/2896677989 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f2a8c0654f0 msgr2=0x7f2a8c0679a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-05-22T15:42:32.781 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:32.776+0000 7f2ab2a40640 1 --2- 172.21.15.16:0/2896677989 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f2a8c0654f0 0x7f2a8c0679a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:32.781 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:32.776+0000 7f2ab2a40640 1 -- 172.21.15.16:0/2896677989 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2aac14d770 msgr2=0x7f2aac0c88c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:32.781 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:32.776+0000 7f2ab2a40640 1 --2- 172.21.15.16:0/2896677989 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2aac14d770 0x7f2aac0c88c0 secure :-1 s=READY pgs=1142 cs=0 l=1 rev1=1 crypto rx=0x7f2a9400e4b0 tx=0x7f2a9400e970 comp rx=0 tx=0).stop 2023-05-22T15:42:32.781 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:32.776+0000 7f2ab2a40640 1 -- 172.21.15.16:0/2896677989 shutdown_connections 2023-05-22T15:42:32.781 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:32.776+0000 7f2ab2a40640 1 --2- 172.21.15.16:0/2896677989 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2aac14dbc0 0x7f2aac159950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:32.781 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:32.776+0000 7f2ab2a40640 1 --2- 172.21.15.16:0/2896677989 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2aac14d770 0x7f2aac0c88c0 unknown :-1 s=CLOSED pgs=1142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:32.782 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:32.776+0000 7f2ab2a40640 1 --2- 172.21.15.16:0/2896677989 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f2a8c0654f0 0x7f2a8c0679a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:32.782 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:32.776+0000 7f2ab2a40640 1 --2- 172.21.15.16:0/2896677989 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2a98004d10 0x7f2aac14ad80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:32.782 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:32.776+0000 7f2ab2a40640 1 -- 172.21.15.16:0/2896677989 >> 172.21.15.16:0/2896677989 conn(0x7f2aac0655c0 msgr2=0x7f2aac158cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:32.782 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:32.776+0000 7f2ab2a40640 1 -- 172.21.15.16:0/2896677989 shutdown_connections 2023-05-22T15:42:32.782 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:32.776+0000 7f2ab2a40640 1 -- 172.21.15.16:0/2896677989 wait complete. 2023-05-22T15:42:32.797 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2023-05-22T15:42:32.930 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:32.928+0000 7fcf09c31640 1 Processor -- start 2023-05-22T15:42:32.931 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:32.928+0000 7fcf09c31640 1 -- start start 2023-05-22T15:42:32.931 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:32.928+0000 7fcf09c31640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fcf040c67d0 0x7fcf040cf730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:32.931 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:32.928+0000 7fcf09c31640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fcf040cc6f0 0x7fcf040c3db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:32.932 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:32.928+0000 7fcf09c31640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf040cfc70 con 0x7fcf040b2880 2023-05-22T15:42:32.932 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:32.928+0000 7fcf09c31640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf0415a300 con 0x7fcf040cc6f0 2023-05-22T15:42:32.932 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:32.928+0000 7fcf09c31640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf040c6ba0 con 0x7fcf040c67d0 2023-05-22T15:42:32.933 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:32.928+0000 7fcf03fff640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fcf040cc6f0 0x7fcf040c3db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:32.933 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:32.928+0000 7fcf03fff640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fcf040cc6f0 0x7fcf040c3db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56922/0 (socket says 172.21.15.16:56922) 2023-05-22T15:42:32.933 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:32.928+0000 7fcf037fe640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fcf040c67d0 0x7fcf040cf730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:32.933 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:32.928+0000 7fcefaffd640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fcf040b2880 0x7fcf040cbfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:43648/0 (socket says 172.21.15.16:43648) 2023-05-22T15:42:32.933 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:32.928+0000 7fcf03fff640 1 -- 172.21.15.16:0/4157807696 learned_addr learned my addr 172.21.15.16:0/4157807696 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:42:32.933 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:32.928+0000 7fcf03fff640 1 -- 172.21.15.16:0/4157807696 >> v2:172.21.15.62:3300/0 conn(0x7fcf040c67d0 msgr2=0x7fcf040cf730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:42:32.934 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:32.928+0000 7fcf03fff640 1 --2- 172.21.15.16:0/4157807696 >> v2:172.21.15.62:3300/0 conn(0x7fcf040c67d0 0x7fcf040cf730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:32.934 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:32.928+0000 7fcf03fff640 1 -- 172.21.15.16:0/4157807696 >> v1:172.21.15.16:6789/0 conn(0x7fcf040b2880 legacy=0x7fcf040cbfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:32.934 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:32.928+0000 7fcf03fff640 1 -- 172.21.15.16:0/4157807696 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf040cfef0 con 0x7fcf040cc6f0 2023-05-22T15:42:32.934 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:32.928+0000 7fcf02ffd640 1 -- 172.21.15.16:0/4157807696 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3489999016 0 0) 0x7fcf040cfc70 con 0x7fcf040b2880 2023-05-22T15:42:32.935 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:32.932+0000 7fcf03fff640 1 --2- 172.21.15.16:0/4157807696 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcf040cc6f0 0x7fcf040c3db0 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x7fcf0406eac0 tx=0x7fcefc01c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22f426540a30b76f server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:32.935 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:32.932+0000 7fcf02ffd640 1 -- 172.21.15.16:0/4157807696 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcefc0a57d0 con 0x7fcf040cc6f0 2023-05-22T15:42:32.935 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:32.932+0000 7fcf02ffd640 1 -- 172.21.15.16:0/4157807696 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcefc0a5930 con 0x7fcf040cc6f0 2023-05-22T15:42:32.935 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:32.932+0000 7fcf02ffd640 1 -- 172.21.15.16:0/4157807696 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcefc0a5cc0 con 0x7fcf040cc6f0 2023-05-22T15:42:32.936 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:32.932+0000 7fcf09c31640 1 -- 172.21.15.16:0/4157807696 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcf040cc6f0 msgr2=0x7fcf040c3db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:32.936 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:32.932+0000 7fcf09c31640 1 --2- 172.21.15.16:0/4157807696 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcf040cc6f0 0x7fcf040c3db0 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x7fcf0406eac0 tx=0x7fcefc01c910 comp rx=0 tx=0).stop 2023-05-22T15:42:32.936 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:32.932+0000 7fcf09c31640 1 -- 172.21.15.16:0/4157807696 shutdown_connections 2023-05-22T15:42:32.936 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:32.932+0000 7fcf09c31640 1 --2- 172.21.15.16:0/4157807696 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcf040cc6f0 0x7fcf040c3db0 unknown :-1 s=CLOSED pgs=1144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:32.936 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:32.932+0000 7fcf09c31640 1 --2- 172.21.15.16:0/4157807696 >> v2:172.21.15.62:3300/0 conn(0x7fcf040c67d0 0x7fcf040cf730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:32.936 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:32.932+0000 7fcf09c31640 1 -- 172.21.15.16:0/4157807696 >> 172.21.15.16:0/4157807696 conn(0x7fcf0406f5a0 msgr2=0x7fcf0406f9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:32.936 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:32.932+0000 7fcf09c31640 1 -- 172.21.15.16:0/4157807696 shutdown_connections 2023-05-22T15:42:32.937 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:32.932+0000 7fcf09c31640 1 -- 172.21.15.16:0/4157807696 wait complete. 2023-05-22T15:42:32.938 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:32.932+0000 7fcf09c31640 1 Processor -- start 2023-05-22T15:42:32.938 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:32.932+0000 7fcf09c31640 1 -- start start 2023-05-22T15:42:32.938 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:32.932+0000 7fcf09c31640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcf040b2880 0x7fcf040c7590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:32.938 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:32.932+0000 7fcf09c31640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fcf040c67d0 0x7fcf041637b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:32.939 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:32.932+0000 7fcf09c31640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcf040cc6f0 0x7fcf04163cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:32.939 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:32.932+0000 7fcf09c31640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf04159b60 con 0x7fcf040cc6f0 2023-05-22T15:42:32.939 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:32.932+0000 7fcf09c31640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf04153d30 con 0x7fcf040b2880 2023-05-22T15:42:32.939 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:32.932+0000 7fcf09c31640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf04154010 con 0x7fcf040c67d0 2023-05-22T15:42:32.939 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:32.932+0000 7fcf03fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcf040cc6f0 0x7fcf04163cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:32.939 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:32.932+0000 7fcf03fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcf040cc6f0 0x7fcf04163cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56936/0 (socket says 172.21.15.16:56936) 2023-05-22T15:42:32.940 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:32.932+0000 7fcf03fff640 1 -- 172.21.15.16:0/3685567675 learned_addr learned my addr 172.21.15.16:0/3685567675 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:42:32.940 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:32.932+0000 7fcf037fe640 1 --2- 172.21.15.16:0/3685567675 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcf040b2880 0x7fcf040c7590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:32.940 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:32.932+0000 7fcefaffd640 1 --2- 172.21.15.16:0/3685567675 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fcf040c67d0 0x7fcf041637b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:32.940 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:32.932+0000 7fcf03fff640 1 -- 172.21.15.16:0/3685567675 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fcf040c67d0 msgr2=0x7fcf041637b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:32.940 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:32.932+0000 7fcf03fff640 1 --2- 172.21.15.16:0/3685567675 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fcf040c67d0 0x7fcf041637b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:32.940 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:32.932+0000 7fcf03fff640 1 -- 172.21.15.16:0/3685567675 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcf040b2880 msgr2=0x7fcf040c7590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:32.941 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:32.932+0000 7fcf03fff640 1 --2- 172.21.15.16:0/3685567675 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcf040b2880 0x7fcf040c7590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:32.941 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:32.932+0000 7fcf03fff640 1 -- 172.21.15.16:0/3685567675 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcefc009b30 con 0x7fcf040cc6f0 2023-05-22T15:42:32.941 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:32.932+0000 7fcf03fff640 1 --2- 172.21.15.16:0/3685567675 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcf040cc6f0 0x7fcf04163cf0 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 crypto rx=0x7fcefc0af040 tx=0x7fcefc0196b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:32.941 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:32.932+0000 7fcf00ff9640 1 -- 172.21.15.16:0/3685567675 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcefc0b8670 con 0x7fcf040cc6f0 2023-05-22T15:42:32.941 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:32.932+0000 7fcf00ff9640 1 -- 172.21.15.16:0/3685567675 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcefc0b87d0 con 0x7fcf040cc6f0 2023-05-22T15:42:32.941 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:32.932+0000 7fcf09c31640 1 -- 172.21.15.16:0/3685567675 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf040d0a70 con 0x7fcf040cc6f0 2023-05-22T15:42:32.941 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:32.932+0000 7fcf00ff9640 1 -- 172.21.15.16:0/3685567675 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcefc0b8b60 con 0x7fcf040cc6f0 2023-05-22T15:42:32.941 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:32.932+0000 7fcf09c31640 1 -- 172.21.15.16:0/3685567675 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf040d0790 con 0x7fcf040cc6f0 2023-05-22T15:42:32.942 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:32.936+0000 7fcf09c31640 1 -- 172.21.15.16:0/3685567675 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf040d0f90 con 0x7fcf040cc6f0 2023-05-22T15:42:32.942 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:32.936+0000 7fcf00ff9640 1 -- 172.21.15.16:0/3685567675 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 27) v1 ==== 84349+0+0 (secure 0 0 0) 0x7fcefc0b8cc0 con 0x7fcf040cc6f0 2023-05-22T15:42:32.943 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:32.936+0000 7fcf00ff9640 1 --2- 172.21.15.16:0/3685567675 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fcee4065520 0x7fcee40679d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:32.943 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:32.936+0000 7fcf037fe640 1 -- 172.21.15.16:0/3685567675 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fcee4065520 msgr2=0x7fcee40679d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/36921 2023-05-22T15:42:32.943 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:32.936+0000 7fcf00ff9640 1 -- 172.21.15.16:0/3685567675 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 6382+0+0 (secure 0 0 0) 0x7fcefc11e3b0 con 0x7fcf040cc6f0 2023-05-22T15:42:32.943 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:32.936+0000 7fcf037fe640 1 --2- 172.21.15.16:0/3685567675 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fcee4065520 0x7fcee40679d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-05-22T15:42:32.943 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:32.940+0000 7fcf00ff9640 1 -- 172.21.15.16:0/3685567675 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fcefc0ebf50 con 0x7fcf040cc6f0 2023-05-22T15:42:32.993 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:42:32.988+0000 7ff4cebf1180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-05-22T15:42:33.011 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:42:33.007+0000 7fea464da180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-05-22T15:42:33.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:33.112+0000 7fcf09c31640 1 -- 172.21.15.16:0/3685567675 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7fcf0404dbe0 con 0x7fcf040cc6f0 2023-05-22T15:42:33.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:33.120+0000 7fcf00ff9640 1 -- 172.21.15.16:0/3685567675 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v24)=0 v24) v1 ==== 133+0+0 (secure 0 0 0) 0x7fcefc0ef570 con 0x7fcf040cc6f0 2023-05-22T15:42:33.126 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:33.120+0000 7fcf09c31640 1 -- 172.21.15.16:0/3685567675 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fcee4065520 msgr2=0x7fcee40679d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-05-22T15:42:33.126 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:33.120+0000 7fcf09c31640 1 --2- 172.21.15.16:0/3685567675 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fcee4065520 0x7fcee40679d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:33.126 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:33.120+0000 7fcf09c31640 1 -- 172.21.15.16:0/3685567675 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcf040cc6f0 msgr2=0x7fcf04163cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:33.126 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:33.120+0000 7fcf09c31640 1 --2- 172.21.15.16:0/3685567675 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcf040cc6f0 0x7fcf04163cf0 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 crypto rx=0x7fcefc0af040 tx=0x7fcefc0196b0 comp rx=0 tx=0).stop 2023-05-22T15:42:33.127 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:33.124+0000 7fcf09c31640 1 -- 172.21.15.16:0/3685567675 shutdown_connections 2023-05-22T15:42:33.127 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:33.124+0000 7fcf09c31640 1 --2- 172.21.15.16:0/3685567675 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcf040cc6f0 0x7fcf04163cf0 unknown :-1 s=CLOSED pgs=1145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:33.127 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:33.124+0000 7fcf09c31640 1 --2- 172.21.15.16:0/3685567675 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fcee4065520 0x7fcee40679d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:33.127 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:33.124+0000 7fcf09c31640 1 --2- 172.21.15.16:0/3685567675 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fcf040c67d0 0x7fcf041637b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:33.127 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:33.124+0000 7fcf09c31640 1 --2- 172.21.15.16:0/3685567675 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcf040b2880 0x7fcf040c7590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:33.127 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:33.124+0000 7fcf09c31640 1 -- 172.21.15.16:0/3685567675 >> 172.21.15.16:0/3685567675 conn(0x7fcf0406f5a0 msgr2=0x7fcf0415cbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:33.127 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:33.124+0000 7fcf09c31640 1 -- 172.21.15.16:0/3685567675 shutdown_connections 2023-05-22T15:42:33.128 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:33.124+0000 7fcf09c31640 1 -- 172.21.15.16:0/3685567675 wait complete. 2023-05-22T15:42:33.133 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:42:33.128+0000 7ff4cebf1180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-05-22T15:42:33.143 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:42:33.143 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1) 2023-05-22T15:42:33.149 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:42:33.147+0000 7fea464da180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-05-22T15:42:33.153 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule list --fs cephfs snap_test_dir1 2023-05-22T15:42:33.311 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:33.308+0000 7ff4f50ec640 1 Processor -- start 2023-05-22T15:42:33.311 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:33.308+0000 7ff4f50ec640 1 -- start start 2023-05-22T15:42:33.312 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:33.308+0000 7ff4f50ec640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff4f00cc010 0x7ff4f00ccd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:33.312 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:33.308+0000 7ff4f50ec640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff4f00cb440 0x7ff4f00cd240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:33.312 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:33.308+0000 7ff4f50ec640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4f0070bb0 con 0x7ff4f0158950 2023-05-22T15:42:33.313 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:33.308+0000 7ff4f50ec640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4f0070e90 con 0x7ff4f00cb440 2023-05-22T15:42:33.313 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:33.308+0000 7ff4f50ec640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4f0071170 con 0x7ff4f00cc010 2023-05-22T15:42:33.313 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:33.308+0000 7ff4ef59e640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff4f00cb440 0x7ff4f00cd240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:33.313 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:33.308+0000 7ff4ef59e640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff4f00cb440 0x7ff4f00cd240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56948/0 (socket says 172.21.15.16:56948) 2023-05-22T15:42:33.313 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:33.308+0000 7ff4ef59e640 1 -- 172.21.15.16:0/4143957071 learned_addr learned my addr 172.21.15.16:0/4143957071 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:42:33.314 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:33.308+0000 7ff4ee59c640 1 --2- 172.21.15.16:0/4143957071 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff4f00cc010 0x7ff4f00ccd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:33.314 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:33.308+0000 7ff4ef59e640 1 -- 172.21.15.16:0/4143957071 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff4f00cc010 msgr2=0x7ff4f00ccd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:42:33.314 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:33.308+0000 7ff4ef59e640 1 --2- 172.21.15.16:0/4143957071 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff4f00cc010 0x7ff4f00ccd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:33.314 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:33.308+0000 7ff4ef59e640 1 -- 172.21.15.16:0/4143957071 >> v1:172.21.15.62:6789/0 conn(0x7ff4f0158950 legacy=0x7ff4f00cb900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:33.314 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:33.308+0000 7ff4ef59e640 1 -- 172.21.15.16:0/4143957071 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4f0071450 con 0x7ff4f00cb440 2023-05-22T15:42:33.315 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:33.312+0000 7ff4ef59e640 1 --2- 172.21.15.16:0/4143957071 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff4f00cb440 0x7ff4f00cd240 secure :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 crypto rx=0x7ff4e800d150 tx=0x7ff4e801ebe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dda662aeb69b876f server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:33.315 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:33.312+0000 7ff4edd9b640 1 -- 172.21.15.16:0/4143957071 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4e800b2d0 con 0x7ff4f00cb440 2023-05-22T15:42:33.315 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:33.312+0000 7ff4edd9b640 1 -- 172.21.15.16:0/4143957071 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4e800b430 con 0x7ff4f00cb440 2023-05-22T15:42:33.316 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:33.312+0000 7ff4edd9b640 1 -- 172.21.15.16:0/4143957071 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4e80a7be0 con 0x7ff4f00cb440 2023-05-22T15:42:33.316 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:33.312+0000 7ff4f50ec640 1 -- 172.21.15.16:0/4143957071 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff4f00cb440 msgr2=0x7ff4f00cd240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:33.316 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:33.312+0000 7ff4f50ec640 1 --2- 172.21.15.16:0/4143957071 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff4f00cb440 0x7ff4f00cd240 secure :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 crypto rx=0x7ff4e800d150 tx=0x7ff4e801ebe0 comp rx=0 tx=0).stop 2023-05-22T15:42:33.316 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:33.312+0000 7ff4f50ec640 1 -- 172.21.15.16:0/4143957071 shutdown_connections 2023-05-22T15:42:33.316 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:33.312+0000 7ff4f50ec640 1 --2- 172.21.15.16:0/4143957071 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff4f00cb440 0x7ff4f00cd240 unknown :-1 s=CLOSED pgs=1146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:33.316 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:33.312+0000 7ff4f50ec640 1 --2- 172.21.15.16:0/4143957071 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff4f00cc010 0x7ff4f00ccd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:33.317 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:33.312+0000 7ff4f50ec640 1 -- 172.21.15.16:0/4143957071 >> 172.21.15.16:0/4143957071 conn(0x7ff4f00b0cc0 msgr2=0x7ff4f00b10c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:33.317 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:33.312+0000 7ff4f50ec640 1 -- 172.21.15.16:0/4143957071 shutdown_connections 2023-05-22T15:42:33.317 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:33.312+0000 7ff4f50ec640 1 -- 172.21.15.16:0/4143957071 wait complete. 2023-05-22T15:42:33.319 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:33.312+0000 7ff4f50ec640 1 Processor -- start 2023-05-22T15:42:33.319 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:33.312+0000 7ff4f50ec640 1 -- start start 2023-05-22T15:42:33.319 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:33.312+0000 7ff4f50ec640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff4f00cb440 0x7ff4f00c5070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:33.319 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:33.312+0000 7ff4f50ec640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff4f00cc010 0x7ff4f0149490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:33.319 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:33.312+0000 7ff4f50ec640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff4f0158950 0x7ff4f01499d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:33.319 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:33.312+0000 7ff4f50ec640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4f0149f10 con 0x7ff4f00cb440 2023-05-22T15:42:33.320 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:33.312+0000 7ff4f50ec640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4f014a070 con 0x7ff4f00cc010 2023-05-22T15:42:33.320 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:33.312+0000 7ff4f50ec640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4f014abb0 con 0x7ff4f0158950 2023-05-22T15:42:33.320 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:33.312+0000 7ff4eed9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff4f00cb440 0x7ff4f00c5070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:33.320 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:33.312+0000 7ff4eed9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff4f00cb440 0x7ff4f00c5070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56964/0 (socket says 172.21.15.16:56964) 2023-05-22T15:42:33.320 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:33.312+0000 7ff4eed9d640 1 -- 172.21.15.16:0/1086849910 learned_addr learned my addr 172.21.15.16:0/1086849910 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:42:33.320 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:33.312+0000 7ff4ee59c640 1 --2- 172.21.15.16:0/1086849910 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff4f00cc010 0x7ff4f0149490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:33.320 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:33.312+0000 7ff4ef59e640 1 --2- 172.21.15.16:0/1086849910 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff4f0158950 0x7ff4f01499d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:33.321 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:33.316+0000 7ff4eed9d640 1 -- 172.21.15.16:0/1086849910 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff4f0158950 msgr2=0x7ff4f01499d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:33.321 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:33.316+0000 7ff4eed9d640 1 --2- 172.21.15.16:0/1086849910 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff4f0158950 0x7ff4f01499d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:33.321 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:33.316+0000 7ff4eed9d640 1 -- 172.21.15.16:0/1086849910 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff4f00cc010 msgr2=0x7ff4f0149490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:33.321 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:33.316+0000 7ff4eed9d640 1 --2- 172.21.15.16:0/1086849910 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff4f00cc010 0x7ff4f0149490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:33.321 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:33.316+0000 7ff4eed9d640 1 -- 172.21.15.16:0/1086849910 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4e800bd40 con 0x7ff4f00cb440 2023-05-22T15:42:33.321 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:33.316+0000 7ff4eed9d640 1 --2- 172.21.15.16:0/1086849910 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff4f00cb440 0x7ff4f00c5070 secure :-1 s=READY pgs=1147 cs=0 l=1 rev1=1 crypto rx=0x7ff4e000c3c0 tx=0x7ff4e000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:33.321 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:33.316+0000 7ff4d77fe640 1 -- 172.21.15.16:0/1086849910 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4e00052d0 con 0x7ff4f00cb440 2023-05-22T15:42:33.322 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:33.316+0000 7ff4d77fe640 1 -- 172.21.15.16:0/1086849910 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4e0005430 con 0x7ff4f00cb440 2023-05-22T15:42:33.322 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:33.316+0000 7ff4f50ec640 1 -- 172.21.15.16:0/1086849910 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4f0070980 con 0x7ff4f00cb440 2023-05-22T15:42:33.322 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:33.316+0000 7ff4d77fe640 1 -- 172.21.15.16:0/1086849910 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4e0003840 con 0x7ff4f00cb440 2023-05-22T15:42:33.322 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:33.316+0000 7ff4f50ec640 1 -- 172.21.15.16:0/1086849910 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4f00707f0 con 0x7ff4f00cb440 2023-05-22T15:42:33.323 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:33.316+0000 7ff4f50ec640 1 -- 172.21.15.16:0/1086849910 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4f0069720 con 0x7ff4f00cb440 2023-05-22T15:42:33.323 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:33.316+0000 7ff4d77fe640 1 -- 172.21.15.16:0/1086849910 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 27) v1 ==== 84349+0+0 (secure 0 0 0) 0x7ff4e00039a0 con 0x7ff4f00cb440 2023-05-22T15:42:33.323 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:33.316+0000 7ff4d77fe640 1 --2- 172.21.15.16:0/1086849910 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ff4dc065520 0x7ff4dc0679d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:33.323 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:33.316+0000 7ff4ee59c640 1 -- 172.21.15.16:0/1086849910 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ff4dc065520 msgr2=0x7ff4dc0679d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/36921 2023-05-22T15:42:33.323 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:33.316+0000 7ff4d77fe640 1 -- 172.21.15.16:0/1086849910 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 6382+0+0 (secure 0 0 0) 0x7ff4e0082130 con 0x7ff4f00cb440 2023-05-22T15:42:33.323 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:33.316+0000 7ff4ee59c640 1 --2- 172.21.15.16:0/1086849910 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ff4dc065520 0x7ff4dc0679d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-05-22T15:42:33.324 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:33.320+0000 7ff4d77fe640 1 -- 172.21.15.16:0/1086849910 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7ff4e004fa60 con 0x7ff4f00cb440 2023-05-22T15:42:33.493 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:33.488+0000 7ff4f50ec640 1 -- 172.21.15.16:0/1086849910 --> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "snap_test_dir1", "target": ["mon-mgr", ""]}) v1 -- 0x7ff4f014acf0 con 0x7ff4dc065520 2023-05-22T15:42:33.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:33.520+0000 7ff4ee59c640 1 -- 172.21.15.16:0/1086849910 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ff4dc065520 msgr2=0x7ff4dc0679d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/36921 2023-05-22T15:42:33.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:33.520+0000 7ff4ee59c640 1 --2- 172.21.15.16:0/1086849910 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ff4dc065520 0x7ff4dc0679d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-05-22T15:42:33.583 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:42:33.580+0000 7ff4cebf1180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-05-22T15:42:33.596 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:42:33.591+0000 7fea464da180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-05-22T15:42:33.636 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:42:33.632+0000 7ff4cebf1180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-05-22T15:42:33.649 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:42:33.647+0000 7fea464da180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-05-22T15:42:33.687 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:42:33.684+0000 7ff4cebf1180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-05-22T15:42:33.700 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:42:33.695+0000 7fea464da180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-05-22T15:42:33.764 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:42:33.760+0000 7ff4cebf1180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-05-22T15:42:33.778 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:42:33.775+0000 7fea464da180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-05-22T15:42:33.815 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:42:33.812+0000 7ff4cebf1180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-05-22T15:42:33.829 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:42:33.827+0000 7fea464da180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-05-22T15:42:33.893 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:42:33.888+0000 7ff4cebf1180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-05-22T15:42:33.907 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:42:33.903+0000 7fea464da180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-05-22T15:42:33.926 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:33.924+0000 7ff4ee59c640 1 -- 172.21.15.16:0/1086849910 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ff4dc065520 msgr2=0x7ff4dc0679d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/36921 2023-05-22T15:42:33.926 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:33.924+0000 7ff4ee59c640 1 --2- 172.21.15.16:0/1086849910 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ff4dc065520 0x7ff4dc0679d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-05-22T15:42:33.944 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:42:33.940+0000 7ff4cebf1180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-05-22T15:42:33.958 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:42:33.955+0000 7fea464da180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-05-22T15:42:34.076 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:42:34.072+0000 7ff4cebf1180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-05-22T15:42:34.091 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:42:34.087+0000 7fea464da180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-05-22T15:42:34.129 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:42:34.124+0000 7ff4cebf1180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-05-22T15:42:34.144 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:42:34.139+0000 7fea464da180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-05-22T15:42:34.353 INFO:tasks.ceph.mgr.y.smithi016.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2023-05-22T15:42:34.354 INFO:tasks.ceph.mgr.y.smithi016.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2023-05-22T15:42:34.354 INFO:tasks.ceph.mgr.y.smithi016.stderr: from numpy import show_config as show_numpy_config 2023-05-22T15:42:34.361 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:42:34.356+0000 7ff4cebf1180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-05-22T15:42:34.370 INFO:tasks.ceph.mgr.x.smithi062.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2023-05-22T15:42:34.370 INFO:tasks.ceph.mgr.x.smithi062.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2023-05-22T15:42:34.370 INFO:tasks.ceph.mgr.x.smithi062.stderr: from numpy import show_config as show_numpy_config 2023-05-22T15:42:34.378 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:42:34.375+0000 7fea464da180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-05-22T15:42:34.486 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:42:34.480+0000 7ff4cebf1180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-05-22T15:42:34.504 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:42:34.499+0000 7fea464da180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-05-22T15:42:34.599 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:42:34.596+0000 7ff4cebf1180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-05-22T15:42:34.618 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:42:34.615+0000 7fea464da180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-05-22T15:42:34.662 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:42:34.660+0000 7ff4cebf1180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-05-22T15:42:34.682 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:42:34.679+0000 7fea464da180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-05-22T15:42:34.730 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:34.728+0000 7ff4ee59c640 1 -- 172.21.15.16:0/1086849910 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ff4dc065520 msgr2=0x7ff4dc0679d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/36921 2023-05-22T15:42:34.730 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:34.728+0000 7ff4ee59c640 1 --2- 172.21.15.16:0/1086849910 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ff4dc065520 0x7ff4dc0679d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-05-22T15:42:34.787 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:42:34.784+0000 7ff4cebf1180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-05-22T15:42:34.807 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:42:34.803+0000 7fea464da180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-05-22T15:42:35.283 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:42:35.280+0000 7ff4cebf1180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-05-22T15:42:35.307 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:42:35.303+0000 7fea464da180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-05-22T15:42:35.359 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:42:35.356+0000 7ff4cebf1180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-05-22T15:42:35.385 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:42:35.383+0000 7fea464da180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-05-22T15:42:35.498 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:42:35.492+0000 7ff4cebf1180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-05-22T15:42:35.526 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:42:35.523+0000 7fea464da180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-05-22T15:42:35.573 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:42:35.568+0000 7ff4cebf1180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-05-22T15:42:35.602 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:42:35.599+0000 7fea464da180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-05-22T15:42:35.821 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:42:35.816+0000 7ff4cebf1180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-05-22T15:42:35.851 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:42:35.847+0000 7fea464da180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-05-22T15:42:35.875 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:42:35.872+0000 7ff4cebf1180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-05-22T15:42:35.907 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:42:35.903+0000 7fea464da180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-05-22T15:42:35.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:35.908+0000 7ff4d77fe640 1 -- 172.21.15.16:0/1086849910 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mgrmap(e 28) v1 ==== 83958+0+0 (secure 0 0 0) 0x7ff4e004fe30 con 0x7ff4f00cb440 2023-05-22T15:42:35.911 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:35.908+0000 7ff4d77fe640 1 -- 172.21.15.16:0/1086849910 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ff4dc065520 msgr2=0x7ff4dc0679d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-05-22T15:42:35.911 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:35.908+0000 7ff4d77fe640 1 --2- 172.21.15.16:0/1086849910 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ff4dc065520 0x7ff4dc0679d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:36.107 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:42:36.104+0000 7ff44fe6c640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T15:42:36.107 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:42:36.104+0000 7ff44fe6c640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T15:42:36.108 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:42:36.104+0000 7ff44fe6c640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T15:42:36.108 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:42:36.104+0000 7ff44fe6c640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T15:42:36.108 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:42:36.104+0000 7ff44fe6c640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T15:42:36.108 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:42:36.104+0000 7ff44de68640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T15:42:36.108 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:42:36.104+0000 7ff44de68640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T15:42:36.108 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:42:36.104+0000 7ff44de68640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T15:42:36.109 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:42:36.104+0000 7ff44de68640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T15:42:36.109 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:42:36.104+0000 7ff44de68640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T15:42:36.918 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:36.912+0000 7ff4d77fe640 1 -- 172.21.15.16:0/1086849910 <== mon.0 v2:172.21.15.16:3300/0 8 ==== mgrmap(e 29) v1 ==== 84085+0+0 (secure 0 0 0) 0x7ff4e005b380 con 0x7ff4f00cb440 2023-05-22T15:42:36.918 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:36.912+0000 7ff4d77fe640 1 --2- 172.21.15.16:0/1086849910 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ff4dc065520 0x7ff4dc0679d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:36.918 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:36.912+0000 7ff4d77fe640 1 -- 172.21.15.16:0/1086849910 --> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "snap_test_dir1", "target": ["mon-mgr", ""]}) v1 -- 0x7ff4dc00fcb0 con 0x7ff4dc065520 2023-05-22T15:42:36.924 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:36.920+0000 7ff4ee59c640 1 --2- 172.21.15.16:0/1086849910 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ff4dc065520 0x7ff4dc0679d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:36.932 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:36.928+0000 7ff4ee59c640 1 --2- 172.21.15.16:0/1086849910 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ff4dc065520 0x7ff4dc0679d0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7ff4f00c59a0 tx=0x7ff4e4008040 comp rx=0 tx=0).ready entity=mgr.7209 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:36.949 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:42:36.944+0000 7ff457e7c640 -1 mgr.server reply reply (2) No such file or directory SnapSchedule for snap_test_dir1 not found 2023-05-22T15:42:36.949 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:36.944+0000 7ff4d77fe640 1 -- 172.21.15.16:0/1086849910 <== mgr.7209 v2:172.21.15.16:6832/36921 1 ==== mgr_command_reply(tid 0: -2 SnapSchedule for snap_test_dir1 not found) v1 ==== 49+0+0 (secure 0 0 0) 0x7ff4e4004020 con 0x7ff4dc065520 2023-05-22T15:42:36.949 INFO:teuthology.orchestra.run.smithi016.stderr:Error ENOENT: SnapSchedule for snap_test_dir1 not found 2023-05-22T15:42:36.956 DEBUG:teuthology.orchestra.run:got remote process result: 2 2023-05-22T15:42:36.957 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:42:36.957 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1) 2023-05-22T15:42:36.968 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2023-05-22T15:42:37.146 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:37.140+0000 7f068567a640 1 Processor -- start 2023-05-22T15:42:37.146 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:37.140+0000 7f068567a640 1 -- start start 2023-05-22T15:42:37.147 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:37.144+0000 7f068567a640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f06800cc6f0 0x7f0680164a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:37.147 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:37.144+0000 7f068567a640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06800c4dc0 con 0x7f06800c67d0 2023-05-22T15:42:37.147 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:37.144+0000 7f068567a640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06800c6ba0 con 0x7f06800b2880 2023-05-22T15:42:37.148 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:37.144+0000 7f068567a640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f068015a300 con 0x7f06800cc6f0 2023-05-22T15:42:37.148 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:37.144+0000 7f067effd640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f06800c67d0 0x7f06800cf730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:43654/0 (socket says 172.21.15.16:43654) 2023-05-22T15:42:37.148 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:37.144+0000 7f067effd640 1 -- 172.21.15.16:0/934926094 learned_addr learned my addr 172.21.15.16:0/934926094 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:42:37.148 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:37.144+0000 7f067f7fe640 1 --2- 172.21.15.16:0/934926094 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f06800cc6f0 0x7f0680164a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:37.148 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:37.144+0000 7f067dffb640 1 -- 172.21.15.16:0/934926094 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2414991255 0 0) 0x7f06800c4dc0 con 0x7f06800c67d0 2023-05-22T15:42:37.149 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:37.144+0000 7f067dffb640 1 -- 172.21.15.16:0/934926094 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0668003530 con 0x7f06800c67d0 2023-05-22T15:42:37.150 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:37.144+0000 7f067dffb640 1 -- 172.21.15.16:0/934926094 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2181370712 0 0) 0x7f0668003530 con 0x7f06800c67d0 2023-05-22T15:42:37.150 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:37.144+0000 7f067dffb640 1 -- 172.21.15.16:0/934926094 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f06800cc6f0 msgr2=0x7f0680164a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:42:37.150 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:37.144+0000 7f067dffb640 1 --2- 172.21.15.16:0/934926094 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f06800cc6f0 0x7f0680164a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:37.150 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:37.144+0000 7f067dffb640 1 -- 172.21.15.16:0/934926094 >> v1:172.21.15.62:6789/0 conn(0x7f06800b2880 legacy=0x7f06800cbfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:37.151 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:37.144+0000 7f067dffb640 1 -- 172.21.15.16:0/934926094 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06800cfef0 con 0x7f06800c67d0 2023-05-22T15:42:37.151 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:37.144+0000 7f067dffb640 1 -- 172.21.15.16:0/934926094 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7f06700016a0 con 0x7f06800c67d0 2023-05-22T15:42:37.151 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:37.144+0000 7f067dffb640 1 -- 172.21.15.16:0/934926094 >> v1:172.21.15.16:6789/0 conn(0x7f06800c67d0 legacy=0x7f06800cf730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:37.151 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:37.144+0000 7f067dffb640 1 --2- 172.21.15.16:0/934926094 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f06800cc6f0 0x7f06680039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:37.151 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:37.144+0000 7f067dffb640 1 --2- 172.21.15.16:0/934926094 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0668003f30 0x7f06680047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:37.151 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:37.144+0000 7f067dffb640 1 --2- 172.21.15.16:0/934926094 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0668004d10 0x7f0668005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:37.151 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:37.144+0000 7f067dffb640 1 -- 172.21.15.16:0/934926094 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f068015a300 con 0x7f0668003f30 2023-05-22T15:42:37.152 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:37.144+0000 7f067dffb640 1 -- 172.21.15.16:0/934926094 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f066800bb20 con 0x7f0668004d10 2023-05-22T15:42:37.152 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:37.144+0000 7f067dffb640 1 -- 172.21.15.16:0/934926094 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f066800bce0 con 0x7f06800cc6f0 2023-05-22T15:42:37.152 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:37.144+0000 7f067e7fc640 1 --2- 172.21.15.16:0/934926094 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0668003f30 0x7f06680047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:37.152 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:37.144+0000 7f067effd640 1 --2- 172.21.15.16:0/934926094 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f06800cc6f0 0x7f06680039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:37.152 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:37.144+0000 7f067f7fe640 1 --2- 172.21.15.16:0/934926094 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0668004d10 0x7f0668005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:37.152 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:37.144+0000 7f067e7fc640 1 -- 172.21.15.16:0/934926094 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f06800cc6f0 msgr2=0x7f06680039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:42:37.152 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:37.144+0000 7f067e7fc640 1 --2- 172.21.15.16:0/934926094 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f06800cc6f0 0x7f06680039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:37.153 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:37.144+0000 7f067e7fc640 1 -- 172.21.15.16:0/934926094 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0668004d10 msgr2=0x7f0668005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:42:37.153 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:37.144+0000 7f067e7fc640 1 --2- 172.21.15.16:0/934926094 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0668004d10 0x7f0668005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:37.153 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:37.144+0000 7f067e7fc640 1 -- 172.21.15.16:0/934926094 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0668003530 con 0x7f0668003f30 2023-05-22T15:42:37.153 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:37.148+0000 7f067e7fc640 1 --2- 172.21.15.16:0/934926094 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0668003f30 0x7f06680047d0 secure :-1 s=READY pgs=1163 cs=0 l=1 rev1=1 crypto rx=0x7f0674001750 tx=0x7f067401c390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af92d57a63b22632 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:37.156 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:37.148+0000 7f067dffb640 1 -- 172.21.15.16:0/934926094 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06740052d0 con 0x7f0668003f30 2023-05-22T15:42:37.156 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:37.148+0000 7f067dffb640 1 -- 172.21.15.16:0/934926094 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06740a7050 con 0x7f0668003f30 2023-05-22T15:42:37.156 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:37.148+0000 7f067dffb640 1 -- 172.21.15.16:0/934926094 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0674003600 con 0x7f0668003f30 2023-05-22T15:42:37.156 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:37.148+0000 7f068567a640 1 -- 172.21.15.16:0/934926094 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0668003f30 msgr2=0x7f06680047d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:37.157 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:37.148+0000 7f068567a640 1 --2- 172.21.15.16:0/934926094 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0668003f30 0x7f06680047d0 secure :-1 s=READY pgs=1163 cs=0 l=1 rev1=1 crypto rx=0x7f0674001750 tx=0x7f067401c390 comp rx=0 tx=0).stop 2023-05-22T15:42:37.157 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:37.148+0000 7f067f7fe640 1 -- 172.21.15.16:0/934926094 reap_dead start 2023-05-22T15:42:37.157 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:37.148+0000 7f068567a640 1 -- 172.21.15.16:0/934926094 shutdown_connections 2023-05-22T15:42:37.157 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:37.148+0000 7f068567a640 1 -- 172.21.15.16:0/934926094 >> 172.21.15.16:0/934926094 conn(0x7f068006f5a0 msgr2=0x7f068006f9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:37.157 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:37.148+0000 7f068567a640 1 -- 172.21.15.16:0/934926094 shutdown_connections 2023-05-22T15:42:37.158 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:37.148+0000 7f068567a640 1 -- 172.21.15.16:0/934926094 wait complete. 2023-05-22T15:42:37.158 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:37.148+0000 7f068567a640 1 Processor -- start 2023-05-22T15:42:37.158 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:37.148+0000 7f068567a640 1 -- start start 2023-05-22T15:42:37.158 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:37.148+0000 7f068567a640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0668003f30 0x7f0680153a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:37.158 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:37.148+0000 7f068567a640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f06800c4950 0x7f068015cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:37.158 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:37.148+0000 7f068567a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0680161b60 0x7f0680162000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:37.159 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:37.148+0000 7f068567a640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0680162540 con 0x7f0680161b60 2023-05-22T15:42:37.159 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:37.148+0000 7f068567a640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f068015a460 con 0x7f06800c4950 2023-05-22T15:42:37.159 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:37.148+0000 7f068567a640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f068015ac70 con 0x7f0668003f30 2023-05-22T15:42:37.159 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:37.148+0000 7f067f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0680161b60 0x7f0680162000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:37.159 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:37.148+0000 7f067f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0680161b60 0x7f0680162000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57078/0 (socket says 172.21.15.16:57078) 2023-05-22T15:42:37.159 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:37.148+0000 7f067f7fe640 1 -- 172.21.15.16:0/2472471137 learned_addr learned my addr 172.21.15.16:0/2472471137 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:42:37.159 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:37.148+0000 7f067effd640 1 --2- 172.21.15.16:0/2472471137 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0668003f30 0x7f0680153a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:37.160 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:37.148+0000 7f067e7fc640 1 --2- 172.21.15.16:0/2472471137 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f06800c4950 0x7f068015cbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:37.160 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:37.148+0000 7f067f7fe640 1 -- 172.21.15.16:0/2472471137 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0668003f30 msgr2=0x7f0680153a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:37.160 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:37.148+0000 7f067f7fe640 1 --2- 172.21.15.16:0/2472471137 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0668003f30 0x7f0680153a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:37.160 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:37.148+0000 7f067f7fe640 1 -- 172.21.15.16:0/2472471137 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f06800c4950 msgr2=0x7f068015cbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:37.160 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:37.148+0000 7f067f7fe640 1 --2- 172.21.15.16:0/2472471137 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f06800c4950 0x7f068015cbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:37.160 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:37.148+0000 7f067f7fe640 1 -- 172.21.15.16:0/2472471137 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06800d0a70 con 0x7f0680161b60 2023-05-22T15:42:37.160 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:37.148+0000 7f067effd640 1 --2- 172.21.15.16:0/2472471137 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0668003f30 0x7f0680153a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T15:42:37.161 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:37.152+0000 7f067f7fe640 1 --2- 172.21.15.16:0/2472471137 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0680161b60 0x7f0680162000 secure :-1 s=READY pgs=1164 cs=0 l=1 rev1=1 crypto rx=0x7f068006e770 tx=0x7f067800f0d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:37.161 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:37.152+0000 7f06677fe640 1 -- 172.21.15.16:0/2472471137 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f067800f810 con 0x7f0680161b60 2023-05-22T15:42:37.161 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:37.152+0000 7f06677fe640 1 -- 172.21.15.16:0/2472471137 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0678011040 con 0x7f0680161b60 2023-05-22T15:42:37.161 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:37.152+0000 7f068567a640 1 -- 172.21.15.16:0/2472471137 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06800d0790 con 0x7f0680161b60 2023-05-22T15:42:37.161 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:37.152+0000 7f06677fe640 1 -- 172.21.15.16:0/2472471137 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f067800fbd0 con 0x7f0680161b60 2023-05-22T15:42:37.161 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:37.152+0000 7f068567a640 1 -- 172.21.15.16:0/2472471137 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06800d0f90 con 0x7f0680161b60 2023-05-22T15:42:37.161 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:37.152+0000 7f068567a640 1 -- 172.21.15.16:0/2472471137 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06800c4d20 con 0x7f0680161b60 2023-05-22T15:42:37.162 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:37.152+0000 7f06677fe640 1 -- 172.21.15.16:0/2472471137 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 29) v1 ==== 84085+0+0 (secure 0 0 0) 0x7f067800fd30 con 0x7f0680161b60 2023-05-22T15:42:37.162 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:37.152+0000 7f06677fe640 1 --2- 172.21.15.16:0/2472471137 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f066c065250 0x7f066c067700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:37.163 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:37.152+0000 7f06677fe640 1 -- 172.21.15.16:0/2472471137 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 6726+0+0 (secure 0 0 0) 0x7f0678082cc0 con 0x7f0680161b60 2023-05-22T15:42:37.163 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:37.152+0000 7f067effd640 1 --2- 172.21.15.16:0/2472471137 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f066c065250 0x7f066c067700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:37.163 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:37.152+0000 7f067effd640 1 --2- 172.21.15.16:0/2472471137 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f066c065250 0x7f066c067700 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f0670001060 tx=0x7f0670009040 comp rx=0 tx=0).ready entity=mgr.7209 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:37.163 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:37.156+0000 7f06677fe640 1 -- 172.21.15.16:0/2472471137 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f0678054450 con 0x7f0680161b60 2023-05-22T15:42:37.352 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:37.348+0000 7f068567a640 1 -- 172.21.15.16:0/2472471137 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7f068004dbe0 con 0x7f0680161b60 2023-05-22T15:42:37.938 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:37.936+0000 7f06677fe640 1 -- 172.21.15.16:0/2472471137 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mgrmap(e 30) v1 ==== 84068+0+0 (secure 0 0 0) 0x7f0678050df0 con 0x7f0680161b60 2023-05-22T15:42:37.939 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:37.936+0000 7f06677fe640 1 -- 172.21.15.16:0/2472471137 <== mon.0 v2:172.21.15.16:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v30) v1 ==== 93+0+0 (secure 0 0 0) 0x7f0678054220 con 0x7f0680161b60 2023-05-22T15:42:37.943 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:37.940+0000 7f068567a640 1 -- 172.21.15.16:0/2472471137 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f066c065250 msgr2=0x7f066c067700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:37.944 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:37.940+0000 7f068567a640 1 --2- 172.21.15.16:0/2472471137 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f066c065250 0x7f066c067700 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f0670001060 tx=0x7f0670009040 comp rx=0 tx=0).stop 2023-05-22T15:42:37.944 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:37.940+0000 7f068567a640 1 -- 172.21.15.16:0/2472471137 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0680161b60 msgr2=0x7f0680162000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:37.944 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:37.940+0000 7f068567a640 1 --2- 172.21.15.16:0/2472471137 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0680161b60 0x7f0680162000 secure :-1 s=READY pgs=1164 cs=0 l=1 rev1=1 crypto rx=0x7f068006e770 tx=0x7f067800f0d0 comp rx=0 tx=0).stop 2023-05-22T15:42:37.944 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:37.940+0000 7f068567a640 1 -- 172.21.15.16:0/2472471137 shutdown_connections 2023-05-22T15:42:37.944 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:37.940+0000 7f068567a640 1 --2- 172.21.15.16:0/2472471137 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0680161b60 0x7f0680162000 unknown :-1 s=CLOSED pgs=1164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:37.944 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:37.940+0000 7f068567a640 1 --2- 172.21.15.16:0/2472471137 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f06800c4950 0x7f068015cbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:37.945 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:37.940+0000 7f068567a640 1 --2- 172.21.15.16:0/2472471137 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f066c065250 0x7f066c067700 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:37.945 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:37.940+0000 7f068567a640 1 --2- 172.21.15.16:0/2472471137 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0668003f30 0x7f0680153a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:37.945 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:37.940+0000 7f068567a640 1 -- 172.21.15.16:0/2472471137 >> 172.21.15.16:0/2472471137 conn(0x7f068006f5a0 msgr2=0x7f0680154380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:37.945 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:37.940+0000 7f068567a640 1 -- 172.21.15.16:0/2472471137 shutdown_connections 2023-05-22T15:42:37.945 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:37.940+0000 7f068567a640 1 -- 172.21.15.16:0/2472471137 wait complete. 2023-05-22T15:42:37.964 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-05-22T15:42:37.965 DEBUG:teuthology.orchestra.run.smithi016:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-05-22T15:42:37.999 DEBUG:teuthology.orchestra.run.smithi062:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-05-22T15:42:38.046 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-05-22T15:42:38.046 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-05-22T15:42:38.047 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2023-05-22T15:42:38.052 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-05-22T15:42:38.068 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:42:38.063+0000 7ff8c72cb180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-05-22T15:42:38.070 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:42:38.068+0000 7fa9cb8de180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-05-22T15:42:38.193 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.188+0000 7f55b2156640 1 Processor -- start 2023-05-22T15:42:38.194 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.188+0000 7f55b2156640 1 -- start start 2023-05-22T15:42:38.195 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.192+0000 7f55b2156640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f55ac159cd0 0x7f55ac15a0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:38.195 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.192+0000 7f55b2156640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f55ac0cc6f0 0x7f55ac0cfe30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:38.195 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.192+0000 7f55b2156640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55ac0cf9a0 con 0x7f55ac0b2880 2023-05-22T15:42:38.195 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.192+0000 7f55b2156640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55ac15aba0 con 0x7f55ac0cc6f0 2023-05-22T15:42:38.195 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.192+0000 7f55b2156640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55ac15ae80 con 0x7f55ac159cd0 2023-05-22T15:42:38.196 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.192+0000 7f55abfff640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f55ac0cc6f0 0x7f55ac0cfe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:38.196 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.192+0000 7f55aaffd640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f55ac0b2880 0x7f55ac0cbfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:43658/0 (socket says 172.21.15.16:43658) 2023-05-22T15:42:38.196 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.192+0000 7f55aaffd640 1 -- 172.21.15.16:0/4095477756 learned_addr learned my addr 172.21.15.16:0/4095477756 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:42:38.196 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.192+0000 7f55ab7fe640 1 --2- 172.21.15.16:0/4095477756 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f55ac159cd0 0x7f55ac15a0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:38.196 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.192+0000 7f55abfff640 1 -- 172.21.15.16:0/4095477756 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f55ac159cd0 msgr2=0x7f55ac15a0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:42:38.196 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.192+0000 7f55abfff640 1 --2- 172.21.15.16:0/4095477756 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f55ac159cd0 0x7f55ac15a0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:38.197 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.192+0000 7f55abfff640 1 -- 172.21.15.16:0/4095477756 >> v1:172.21.15.16:6789/0 conn(0x7f55ac0b2880 legacy=0x7f55ac0cbfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:38.197 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.192+0000 7f55aa7fc640 1 -- 172.21.15.16:0/4095477756 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1084742423 0 0) 0x7f55ac0cf9a0 con 0x7f55ac0b2880 2023-05-22T15:42:38.197 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.192+0000 7f55abfff640 1 -- 172.21.15.16:0/4095477756 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55ac15b160 con 0x7f55ac0cc6f0 2023-05-22T15:42:38.197 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.192+0000 7f55abfff640 1 --2- 172.21.15.16:0/4095477756 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f55ac0cc6f0 0x7f55ac0cfe30 secure :-1 s=READY pgs=1168 cs=0 l=1 rev1=1 crypto rx=0x7f559c0097c0 tx=0x7f559c01c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=edf56c61df6b408d server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:38.198 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.192+0000 7f55aa7fc640 1 -- 172.21.15.16:0/4095477756 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f559c0ad660 con 0x7f55ac0cc6f0 2023-05-22T15:42:38.198 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.192+0000 7f55aa7fc640 1 -- 172.21.15.16:0/4095477756 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f559c0ad7c0 con 0x7f55ac0cc6f0 2023-05-22T15:42:38.198 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.192+0000 7f55aa7fc640 1 -- 172.21.15.16:0/4095477756 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f559c0adb50 con 0x7f55ac0cc6f0 2023-05-22T15:42:38.199 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.192+0000 7f55b2156640 1 -- 172.21.15.16:0/4095477756 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f55ac0cc6f0 msgr2=0x7f55ac0cfe30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:38.199 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.192+0000 7f55b2156640 1 --2- 172.21.15.16:0/4095477756 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f55ac0cc6f0 0x7f55ac0cfe30 secure :-1 s=READY pgs=1168 cs=0 l=1 rev1=1 crypto rx=0x7f559c0097c0 tx=0x7f559c01c450 comp rx=0 tx=0).stop 2023-05-22T15:42:38.199 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.196+0000 7f55b2156640 1 -- 172.21.15.16:0/4095477756 shutdown_connections 2023-05-22T15:42:38.199 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.196+0000 7f55b2156640 1 --2- 172.21.15.16:0/4095477756 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f55ac0cc6f0 0x7f55ac0cfe30 unknown :-1 s=CLOSED pgs=1168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:38.199 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.196+0000 7f55b2156640 1 --2- 172.21.15.16:0/4095477756 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f55ac159cd0 0x7f55ac15a0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:38.200 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.196+0000 7f55b2156640 1 -- 172.21.15.16:0/4095477756 >> 172.21.15.16:0/4095477756 conn(0x7f55ac06f5c0 msgr2=0x7f55ac06f9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:38.200 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.196+0000 7f55b2156640 1 -- 172.21.15.16:0/4095477756 shutdown_connections 2023-05-22T15:42:38.200 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.196+0000 7f55b2156640 1 -- 172.21.15.16:0/4095477756 wait complete. 2023-05-22T15:42:38.201 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.196+0000 7f55b2156640 1 Processor -- start 2023-05-22T15:42:38.202 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.196+0000 7f55b2156640 1 -- start start 2023-05-22T15:42:38.202 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.196+0000 7f55b2156640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f55ac0b2880 0x7f55ac14ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:38.202 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.196+0000 7f55b2156640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f55ac0cc6f0 0x7f55ac0c3d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:38.202 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.196+0000 7f55b2156640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f55ac159cd0 0x7f55ac0c42d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:38.202 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.196+0000 7f55b2156640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55ac0c5150 con 0x7f55ac0b2880 2023-05-22T15:42:38.202 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.196+0000 7f55b2156640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55ac0b55d0 con 0x7f55ac0cc6f0 2023-05-22T15:42:38.203 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.196+0000 7f55b2156640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55ac14f260 con 0x7f55ac159cd0 2023-05-22T15:42:38.203 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.196+0000 7f55ab7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f55ac0b2880 0x7f55ac14ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:38.203 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.196+0000 7f55ab7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f55ac0b2880 0x7f55ac14ed20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57110/0 (socket says 172.21.15.16:57110) 2023-05-22T15:42:38.203 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.196+0000 7f55ab7fe640 1 -- 172.21.15.16:0/735621368 learned_addr learned my addr 172.21.15.16:0/735621368 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:42:38.203 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.196+0000 7f55abfff640 1 --2- 172.21.15.16:0/735621368 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f55ac159cd0 0x7f55ac0c42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:38.203 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.196+0000 7f55aaffd640 1 --2- 172.21.15.16:0/735621368 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f55ac0cc6f0 0x7f55ac0c3d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:38.203 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.196+0000 7f55ab7fe640 1 -- 172.21.15.16:0/735621368 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f55ac159cd0 msgr2=0x7f55ac0c42d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:38.204 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.196+0000 7f55ab7fe640 1 --2- 172.21.15.16:0/735621368 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f55ac159cd0 0x7f55ac0c42d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:38.204 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.196+0000 7f55ab7fe640 1 -- 172.21.15.16:0/735621368 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f55ac0cc6f0 msgr2=0x7f55ac0c3d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:38.204 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.196+0000 7f55ab7fe640 1 --2- 172.21.15.16:0/735621368 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f55ac0cc6f0 0x7f55ac0c3d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:38.204 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.196+0000 7f55ab7fe640 1 -- 172.21.15.16:0/735621368 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f559c009420 con 0x7f55ac0b2880 2023-05-22T15:42:38.205 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.196+0000 7f55ab7fe640 1 --2- 172.21.15.16:0/735621368 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f55ac0b2880 0x7f55ac14ed20 secure :-1 s=READY pgs=1169 cs=0 l=1 rev1=1 crypto rx=0x7f55a4009db0 tx=0x7f55a400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:38.205 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.196+0000 7f55a3fff640 1 -- 172.21.15.16:0/735621368 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55a4014480 con 0x7f55ac0b2880 2023-05-22T15:42:38.205 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.196+0000 7f55a3fff640 1 -- 172.21.15.16:0/735621368 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55a400f040 con 0x7f55ac0b2880 2023-05-22T15:42:38.205 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.196+0000 7f55b2156640 1 -- 172.21.15.16:0/735621368 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55ac0d0d10 con 0x7f55ac0b2880 2023-05-22T15:42:38.205 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.196+0000 7f55a3fff640 1 -- 172.21.15.16:0/735621368 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55a4014800 con 0x7f55ac0b2880 2023-05-22T15:42:38.205 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.196+0000 7f55b2156640 1 -- 172.21.15.16:0/735621368 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55ac0d0b80 con 0x7f55ac0b2880 2023-05-22T15:42:38.206 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.196+0000 7f55b2156640 1 -- 172.21.15.16:0/735621368 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55ac0d0ea0 con 0x7f55ac0b2880 2023-05-22T15:42:38.206 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.200+0000 7f55a3fff640 1 -- 172.21.15.16:0/735621368 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 30) v1 ==== 84068+0+0 (secure 0 0 0) 0x7f55a4014960 con 0x7f55ac0b2880 2023-05-22T15:42:38.206 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.200+0000 7f55a3fff640 1 --2- 172.21.15.16:0/735621368 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f558c065200 0x7f558c0676b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:38.206 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.200+0000 7f55aaffd640 1 -- 172.21.15.16:0/735621368 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f558c065200 msgr2=0x7f558c0676b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/36921 2023-05-22T15:42:38.207 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.200+0000 7f55a3fff640 1 -- 172.21.15.16:0/735621368 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 6726+0+0 (secure 0 0 0) 0x7f55a4083300 con 0x7f55ac0b2880 2023-05-22T15:42:38.207 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.200+0000 7f55aaffd640 1 --2- 172.21.15.16:0/735621368 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f558c065200 0x7f558c0676b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-05-22T15:42:38.207 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.204+0000 7f55a3fff640 1 -- 172.21.15.16:0/735621368 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f55a4050fa0 con 0x7f55ac0b2880 2023-05-22T15:42:38.360 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.356+0000 7f55b2156640 1 -- 172.21.15.16:0/735621368 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f55ac04dbe0 con 0x7f55ac0b2880 2023-05-22T15:42:38.361 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.356+0000 7f55a3fff640 1 -- 172.21.15.16:0/735621368 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) v1 ==== 74+0+18530 (secure 0 0 0) 0x7f55a40545c0 con 0x7f55ac0b2880 2023-05-22T15:42:38.361 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:42:38.361 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":91,"fsid":"b2719834-7847-4d98-90a6-f65d57f85c49","created":"2023-05-22T15:26:25.259314+0000","modified":"2023-05-22T15:42:35.895812+0000","last_up_change":"2023-05-22T15:26:36.448524+0000","last_in_change":"2023-05-22T15:26:27.675668+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-05-22T15:26:38.019781+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-05-22T15:26:43.570514+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2023-05-22T15:42:20.730757+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2023-05-22T15:42:21.805016+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9e7ff854-b25f-4cbb-a5c2-1799d9c53d2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":37021},{"type":"v1","addr":"172.21.15.16:6825","nonce":37021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":37021},{"type":"v1","addr":"172.21.15.16:6827","nonce":37021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":37021},{"type":"v1","addr":"172.21.15.16:6831","nonce":37021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":37021},{"type":"v1","addr":"172.21.15.16:6829","nonce":37021}]},"public_addr":"172.21.15.16:6825/37021","cluster_addr":"172.21.15.16:6827/37021","heartbeat_back_addr":"172.21.15.16:6831/37021","heartbeat_front_addr":"172.21.15.16:6829/37021","state":["exists","up"]},{"osd":1,"uuid":"4702fbc9-a927-4565-aca5-41dc69cc8264","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":37008},{"type":"v1","addr":"172.21.15.16:6801","nonce":37008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":37008},{"type":"v1","addr":"172.21.15.16:6803","nonce":37008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":37008},{"type":"v1","addr":"172.21.15.16:6807","nonce":37008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":37008},{"type":"v1","addr":"172.21.15.16:6805","nonce":37008}]},"public_addr":"172.21.15.16:6801/37008","cluster_addr":"172.21.15.16:6803/37008","heartbeat_back_addr":"172.21.15.16:6807/37008","heartbeat_front_addr":"172.21.15.16:6805/37008","state":["exists","up"]},{"osd":2,"uuid":"1e7bba46-4eba-47ef-bb81-5e2d371638ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":37009},{"type":"v1","addr":"172.21.15.16:6809","nonce":37009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":37009},{"type":"v1","addr":"172.21.15.16:6811","nonce":37009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":37009},{"type":"v1","addr":"172.21.15.16:6815","nonce":37009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":37009},{"type":"v1","addr":"172.21.15.16:6813","nonce":37009}]},"public_addr":"172.21.15.16:6809/37009","cluster_addr":"172.21.15.16:6811/37009","heartbeat_back_addr":"172.21.15.16:6815/37009","heartbeat_front_addr":"172.21.15.16:6813/37009","state":["exists","up"]},{"osd":3,"uuid":"383cacc0-5a88-4904-990e-498ca59ef48a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":37020},{"type":"v1","addr":"172.21.15.16:6817","nonce":37020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":37020},{"type":"v1","addr":"172.21.15.16:6819","nonce":37020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":37020},{"type":"v1","addr":"172.21.15.16:6823","nonce":37020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":37020},{"type":"v1","addr":"172.21.15.16:6821","nonce":37020}]},"public_addr":"172.21.15.16:6817/37020","cluster_addr":"172.21.15.16:6819/37020","heartbeat_back_addr":"172.21.15.16:6823/37020","heartbeat_front_addr":"172.21.15.16:6821/37020","state":["exists","up"]},{"osd":4,"uuid":"970500ef-95b7-42c1-b080-eed4fae7013d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":37353},{"type":"v1","addr":"172.21.15.62:6817","nonce":37353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6819","nonce":37353},{"type":"v1","addr":"172.21.15.62:6822","nonce":37353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37353},{"type":"v1","addr":"172.21.15.62:6831","nonce":37353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37353},{"type":"v1","addr":"172.21.15.62:6826","nonce":37353}]},"public_addr":"172.21.15.62:6817/37353","cluster_addr":"172.21.15.62:6822/37353","heartbeat_back_addr":"172.21.15.62:6831/37353","heartbeat_front_addr":"172.21.15.62:6826/37353","state":["exists","up"]},{"osd":5,"uuid":"022d67db-d4d8-445b-b1b9-8b446a00f5c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":37352},{"type":"v1","addr":"172.21.15.62:6801","nonce":37352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":37352},{"type":"v1","addr":"172.21.15.62:6803","nonce":37352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":37352},{"type":"v1","addr":"172.21.15.62:6807","nonce":37352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":37352},{"type":"v1","addr":"172.21.15.62:6805","nonce":37352}]},"public_addr":"172.21.15.62:6801/37352","cluster_addr":"172.21.15.62:6803/37352","heartbeat_back_addr":"172.21.15.62:6807/37352","heartbeat_front_addr":"172.21.15.62:6805/37352","state":["exists","up"]},{"osd":6,"uuid":"42bd372b-6688-4a1b-8407-e16e8c266e33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":37354},{"type":"v1","addr":"172.21.15.62:6820","nonce":37354}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":37354},{"type":"v1","addr":"172.21.15.62:6823","nonce":37354}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6829","nonce":37354},{"type":"v1","addr":"172.21.15.62:6830","nonce":37354}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6825","nonce":37354},{"type":"v1","addr":"172.21.15.62:6827","nonce":37354}]},"public_addr":"172.21.15.62:6820/37354","cluster_addr":"172.21.15.62:6823/37354","heartbeat_back_addr":"172.21.15.62:6830/37354","heartbeat_front_addr":"172.21.15.62:6827/37354","state":["exists","up"]},{"osd":7,"uuid":"ac70b2a3-f41b-4bf2-b227-7f2cbb55426d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37355},{"type":"v1","addr":"172.21.15.62:6809","nonce":37355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37355},{"type":"v1","addr":"172.21.15.62:6811","nonce":37355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":37355},{"type":"v1","addr":"172.21.15.62:6815","nonce":37355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":37355},{"type":"v1","addr":"172.21.15.62:6813","nonce":37355}]},"public_addr":"172.21.15.62:6809/37355","cluster_addr":"172.21.15.62:6811/37355","heartbeat_back_addr":"172.21.15.62:6815/37355","heartbeat_front_addr":"172.21.15.62:6813/37355","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.368896+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.107273+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.276888+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.287502+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.124791+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.251937+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.255679+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.232797+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:0/3510757085":"2023-05-23T15:42:35.895737+0000","172.21.15.16:0/4119485618":"2023-05-23T15:42:35.895737+0000","172.21.15.16:0/2050049957":"2023-05-23T15:42:35.895737+0000","172.21.15.16:0/1396723034":"2023-05-23T15:42:35.895737+0000","172.21.15.16:0/3754867799":"2023-05-23T15:42:35.895737+0000","172.21.15.16:6833/36921":"2023-05-23T15:42:35.895737+0000","172.21.15.16:6832/36921":"2023-05-23T15:42:35.895737+0000","172.21.15.16:0/3710518307":"2023-05-23T15:42:35.895737+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-05-22T15:42:38.363 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.360+0000 7f55b2156640 1 -- 172.21.15.16:0/735621368 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f558c065200 msgr2=0x7f558c0676b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-05-22T15:42:38.363 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.360+0000 7f55b2156640 1 --2- 172.21.15.16:0/735621368 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f558c065200 0x7f558c0676b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:38.363 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.360+0000 7f55b2156640 1 -- 172.21.15.16:0/735621368 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f55ac0b2880 msgr2=0x7f55ac14ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:38.364 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.360+0000 7f55b2156640 1 --2- 172.21.15.16:0/735621368 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f55ac0b2880 0x7f55ac14ed20 secure :-1 s=READY pgs=1169 cs=0 l=1 rev1=1 crypto rx=0x7f55a4009db0 tx=0x7f55a400c490 comp rx=0 tx=0).stop 2023-05-22T15:42:38.364 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.360+0000 7f55b2156640 1 -- 172.21.15.16:0/735621368 shutdown_connections 2023-05-22T15:42:38.364 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.360+0000 7f55b2156640 1 --2- 172.21.15.16:0/735621368 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f558c065200 0x7f558c0676b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:38.364 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.360+0000 7f55b2156640 1 --2- 172.21.15.16:0/735621368 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f55ac159cd0 0x7f55ac0c42d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:38.364 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.360+0000 7f55b2156640 1 --2- 172.21.15.16:0/735621368 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f55ac0cc6f0 0x7f55ac0c3d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:38.364 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.360+0000 7f55b2156640 1 --2- 172.21.15.16:0/735621368 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f55ac0b2880 0x7f55ac14ed20 unknown :-1 s=CLOSED pgs=1169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:38.364 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.360+0000 7f55b2156640 1 -- 172.21.15.16:0/735621368 >> 172.21.15.16:0/735621368 conn(0x7f55ac06f5c0 msgr2=0x7f55ac1505b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:38.365 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.360+0000 7f55b2156640 1 -- 172.21.15.16:0/735621368 shutdown_connections 2023-05-22T15:42:38.365 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.360+0000 7f55b2156640 1 -- 172.21.15.16:0/735621368 wait complete. 2023-05-22T15:42:38.380 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-05-22T15:26:38.019781+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-05-22T15:26:43.570514+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 10, 'pool_name': 'cephfs_metadata', 'create_time': '2023-05-22T15:42:20.730757+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 11, 'pool_name': 'cephfs_data', 'create_time': '2023-05-22T15:42:21.805016+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '90', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-05-22T15:42:38.381 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-05-22T15:42:38.454 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:42:38.451+0000 7ff8c72cb180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-05-22T15:42:38.468 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:42:38.464+0000 7fa9cb8de180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-05-22T15:42:38.507 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.504+0000 7febd5416640 1 Processor -- start 2023-05-22T15:42:38.507 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.504+0000 7febd5416640 1 -- start start 2023-05-22T15:42:38.509 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.504+0000 7febd5416640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7febd00cae20 0x7febd006fef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:38.509 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.504+0000 7febd5416640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febd0069720 con 0x7febd0070430 2023-05-22T15:42:38.509 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.504+0000 7febd5416640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febd01543b0 con 0x7febd0153fe0 2023-05-22T15:42:38.509 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.504+0000 7febd5416640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7febd00708b0 con 0x7febd00cae20 2023-05-22T15:42:38.510 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.504+0000 7febce7fc640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7febd00cae20 0x7febd006fef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:38.510 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.504+0000 7febce7fc640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7febd00cae20 0x7febd006fef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57122/0 (socket says 172.21.15.16:57122) 2023-05-22T15:42:38.510 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.504+0000 7febce7fc640 1 -- 172.21.15.16:0/2050763098 learned_addr learned my addr 172.21.15.16:0/2050763098 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:42:38.510 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.504+0000 7febcf7fe640 1 --1- 172.21.15.16:0/2050763098 >> v1:172.21.15.16:6789/0 conn(0x7febd0070430 0x7febd00ad410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:43672/0 (socket says 172.21.15.16:43672) 2023-05-22T15:42:38.510 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.504+0000 7febce7fc640 1 -- 172.21.15.16:0/2050763098 >> v1:172.21.15.62:6789/0 conn(0x7febd0153fe0 legacy=0x7febd006d3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:42:38.510 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.504+0000 7febcdffb640 1 -- 172.21.15.16:0/2050763098 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3074907121 0 0) 0x7febd0069720 con 0x7febd0070430 2023-05-22T15:42:38.510 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.504+0000 7febce7fc640 1 -- 172.21.15.16:0/2050763098 >> v1:172.21.15.16:6789/0 conn(0x7febd0070430 legacy=0x7febd00ad410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:38.511 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.504+0000 7febce7fc640 1 -- 172.21.15.16:0/2050763098 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febd015e360 con 0x7febd00cae20 2023-05-22T15:42:38.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.504+0000 7febce7fc640 1 --2- 172.21.15.16:0/2050763098 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7febd00cae20 0x7febd006fef0 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 crypto rx=0x7febc4009870 tx=0x7febc401c540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a340d982e00f5bde server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:38.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.508+0000 7febcdffb640 1 -- 172.21.15.16:0/2050763098 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febc40ab650 con 0x7febd00cae20 2023-05-22T15:42:38.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.508+0000 7febcdffb640 1 -- 172.21.15.16:0/2050763098 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febc40ab7b0 con 0x7febd00cae20 2023-05-22T15:42:38.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.508+0000 7febcdffb640 1 -- 172.21.15.16:0/2050763098 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febc40abb40 con 0x7febd00cae20 2023-05-22T15:42:38.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.508+0000 7febd5416640 1 -- 172.21.15.16:0/2050763098 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7febd00cae20 msgr2=0x7febd006fef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:38.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.508+0000 7febd5416640 1 --2- 172.21.15.16:0/2050763098 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7febd00cae20 0x7febd006fef0 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 crypto rx=0x7febc4009870 tx=0x7febc401c540 comp rx=0 tx=0).stop 2023-05-22T15:42:38.513 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.508+0000 7febd5416640 1 -- 172.21.15.16:0/2050763098 shutdown_connections 2023-05-22T15:42:38.513 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.508+0000 7febd5416640 1 --2- 172.21.15.16:0/2050763098 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7febd00cae20 0x7febd006fef0 unknown :-1 s=CLOSED pgs=1171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:38.513 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.508+0000 7febd5416640 1 -- 172.21.15.16:0/2050763098 >> 172.21.15.16:0/2050763098 conn(0x7febd00b1110 msgr2=0x7febd00ce830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:38.513 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.508+0000 7febd5416640 1 -- 172.21.15.16:0/2050763098 shutdown_connections 2023-05-22T15:42:38.513 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.508+0000 7febd5416640 1 -- 172.21.15.16:0/2050763098 wait complete. 2023-05-22T15:42:38.513 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.508+0000 7febd5416640 1 Processor -- start 2023-05-22T15:42:38.513 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.508+0000 7febd5416640 1 -- start start 2023-05-22T15:42:38.516 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.508+0000 7febd5416640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7febd0070430 0x7febd014acf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:38.516 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.508+0000 7febd5416640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7febd00cae20 0x7febd015b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:38.517 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.508+0000 7febd5416640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7febd0153fe0 0x7febd0158df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:38.517 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.508+0000 7febd5416640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febd0149f30 con 0x7febd00cae20 2023-05-22T15:42:38.517 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.508+0000 7febd5416640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febd014a210 con 0x7febd0070430 2023-05-22T15:42:38.517 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.508+0000 7febd5416640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febd014a4f0 con 0x7febd0153fe0 2023-05-22T15:42:38.518 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.508+0000 7febce7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7febd00cae20 0x7febd015b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:38.518 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.508+0000 7febce7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7febd00cae20 0x7febd015b2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57128/0 (socket says 172.21.15.16:57128) 2023-05-22T15:42:38.518 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.508+0000 7febce7fc640 1 -- 172.21.15.16:0/2310466216 learned_addr learned my addr 172.21.15.16:0/2310466216 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:42:38.518 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.508+0000 7febcf7fe640 1 --2- 172.21.15.16:0/2310466216 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7febd0153fe0 0x7febd0158df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:38.518 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.508+0000 7febceffd640 1 --2- 172.21.15.16:0/2310466216 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7febd0070430 0x7febd014acf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:38.519 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.508+0000 7febce7fc640 1 -- 172.21.15.16:0/2310466216 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7febd0153fe0 msgr2=0x7febd0158df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:38.519 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.508+0000 7febce7fc640 1 --2- 172.21.15.16:0/2310466216 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7febd0153fe0 0x7febd0158df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:38.519 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.508+0000 7febce7fc640 1 -- 172.21.15.16:0/2310466216 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7febd0070430 msgr2=0x7febd014acf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:38.519 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.508+0000 7febce7fc640 1 --2- 172.21.15.16:0/2310466216 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7febd0070430 0x7febd014acf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:38.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.508+0000 7febce7fc640 1 -- 172.21.15.16:0/2310466216 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febc40094d0 con 0x7febd00cae20 2023-05-22T15:42:38.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.508+0000 7febce7fc640 1 --2- 172.21.15.16:0/2310466216 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7febd00cae20 0x7febd015b2c0 secure :-1 s=READY pgs=1172 cs=0 l=1 rev1=1 crypto rx=0x7febc40ad040 tx=0x7febc401ad80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:38.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.508+0000 7febb77fe640 1 -- 172.21.15.16:0/2310466216 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febc40b4800 con 0x7febd00cae20 2023-05-22T15:42:38.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.508+0000 7febb77fe640 1 -- 172.21.15.16:0/2310466216 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febc40b4980 con 0x7febd00cae20 2023-05-22T15:42:38.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.508+0000 7febd5416640 1 -- 172.21.15.16:0/2310466216 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7febd015de40 con 0x7febd00cae20 2023-05-22T15:42:38.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.508+0000 7febb77fe640 1 -- 172.21.15.16:0/2310466216 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febc40b4d30 con 0x7febd00cae20 2023-05-22T15:42:38.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.508+0000 7febd5416640 1 -- 172.21.15.16:0/2310466216 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7febd015db60 con 0x7febd00cae20 2023-05-22T15:42:38.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.512+0000 7febd5416640 1 -- 172.21.15.16:0/2310466216 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7febd006dd20 con 0x7febd00cae20 2023-05-22T15:42:38.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.512+0000 7febb77fe640 1 -- 172.21.15.16:0/2310466216 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 30) v1 ==== 84068+0+0 (secure 0 0 0) 0x7febc40c7390 con 0x7febd00cae20 2023-05-22T15:42:38.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.512+0000 7febb77fe640 1 --2- 172.21.15.16:0/2310466216 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7febbc0651d0 0x7febbc067680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:38.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.512+0000 7febceffd640 1 -- 172.21.15.16:0/2310466216 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7febbc0651d0 msgr2=0x7febbc067680 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/36921 2023-05-22T15:42:38.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.512+0000 7febb77fe640 1 -- 172.21.15.16:0/2310466216 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 6726+0+0 (secure 0 0 0) 0x7febc411d3f0 con 0x7febd00cae20 2023-05-22T15:42:38.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.512+0000 7febceffd640 1 --2- 172.21.15.16:0/2310466216 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7febbc0651d0 0x7febbc067680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-05-22T15:42:38.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.516+0000 7febb77fe640 1 -- 172.21.15.16:0/2310466216 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7febc40eb160 con 0x7febd00cae20 2023-05-22T15:42:38.592 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:42:38.587+0000 7ff8c72cb180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-05-22T15:42:38.609 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:42:38.604+0000 7fa9cb8de180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-05-22T15:42:38.675 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.672+0000 7febd5416640 1 -- 172.21.15.16:0/2310466216 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7febd006d260 con 0x7febd00cae20 2023-05-22T15:42:38.675 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.672+0000 7febb77fe640 1 -- 172.21.15.16:0/2310466216 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v91) v1 ==== 93+0+10 (secure 0 0 0) 0x7febc40ee780 con 0x7febd00cae20 2023-05-22T15:42:38.676 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2023-05-22T15:42:38.678 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.672+0000 7febd5416640 1 -- 172.21.15.16:0/2310466216 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7febbc0651d0 msgr2=0x7febbc067680 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-05-22T15:42:38.678 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.672+0000 7febd5416640 1 --2- 172.21.15.16:0/2310466216 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7febbc0651d0 0x7febbc067680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:38.678 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.672+0000 7febd5416640 1 -- 172.21.15.16:0/2310466216 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7febd00cae20 msgr2=0x7febd015b2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:38.678 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.672+0000 7febd5416640 1 --2- 172.21.15.16:0/2310466216 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7febd00cae20 0x7febd015b2c0 secure :-1 s=READY pgs=1172 cs=0 l=1 rev1=1 crypto rx=0x7febc40ad040 tx=0x7febc401ad80 comp rx=0 tx=0).stop 2023-05-22T15:42:38.678 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.672+0000 7febd5416640 1 -- 172.21.15.16:0/2310466216 shutdown_connections 2023-05-22T15:42:38.678 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.672+0000 7febd5416640 1 --2- 172.21.15.16:0/2310466216 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7febbc0651d0 0x7febbc067680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:38.679 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.672+0000 7febd5416640 1 --2- 172.21.15.16:0/2310466216 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7febd0153fe0 0x7febd0158df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:38.679 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.672+0000 7febd5416640 1 --2- 172.21.15.16:0/2310466216 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7febd00cae20 0x7febd015b2c0 unknown :-1 s=CLOSED pgs=1172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:38.679 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.672+0000 7febd5416640 1 --2- 172.21.15.16:0/2310466216 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7febd0070430 0x7febd014acf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:38.679 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.672+0000 7febd5416640 1 -- 172.21.15.16:0/2310466216 >> 172.21.15.16:0/2310466216 conn(0x7febd00b1110 msgr2=0x7febd006c9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:38.679 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.672+0000 7febd5416640 1 -- 172.21.15.16:0/2310466216 shutdown_connections 2023-05-22T15:42:38.679 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.672+0000 7febd5416640 1 -- 172.21.15.16:0/2310466216 wait complete. 2023-05-22T15:42:38.694 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-05-22T15:42:38.805 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.800+0000 7f3cd279a640 1 Processor -- start 2023-05-22T15:42:38.805 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.800+0000 7f3cd279a640 1 -- start start 2023-05-22T15:42:38.806 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.800+0000 7f3cd279a640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f3ccc06cdf0 0x7f3ccc070be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:38.806 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.800+0000 7f3cd279a640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ccc06eb90 con 0x7f3ccc070760 2023-05-22T15:42:38.807 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.804+0000 7f3cd279a640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ccc158de0 con 0x7f3ccc158a10 2023-05-22T15:42:38.807 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.804+0000 7f3cd279a640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ccc06e060 con 0x7f3ccc06cdf0 2023-05-22T15:42:38.807 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.804+0000 7f3cd0d37640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f3ccc06cdf0 0x7f3ccc070be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:38.807 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.804+0000 7f3cd0d37640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f3ccc06cdf0 0x7f3ccc070be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57136/0 (socket says 172.21.15.16:57136) 2023-05-22T15:42:38.807 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.804+0000 7f3cd0d37640 1 -- 172.21.15.16:0/1587890830 learned_addr learned my addr 172.21.15.16:0/1587890830 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:42:38.807 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.804+0000 7f3ccb7fe640 1 --1- 172.21.15.16:0/1587890830 >> v1:172.21.15.16:6789/0 conn(0x7f3ccc070760 0x7f3ccc06c6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:43688/0 (socket says 172.21.15.16:43688) 2023-05-22T15:42:38.807 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.804+0000 7f3cd0d37640 1 -- 172.21.15.16:0/1587890830 >> v1:172.21.15.62:6789/0 conn(0x7f3ccc158a10 legacy=0x7f3ccc06e480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:42:38.808 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.804+0000 7f3cd0d37640 1 -- 172.21.15.16:0/1587890830 >> v1:172.21.15.16:6789/0 conn(0x7f3ccc070760 legacy=0x7f3ccc06c6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:38.808 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.804+0000 7f3ccaffd640 1 -- 172.21.15.16:0/1587890830 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 206230417 0 0) 0x7f3ccc06eb90 con 0x7f3ccc070760 2023-05-22T15:42:38.808 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.804+0000 7f3cd0d37640 1 -- 172.21.15.16:0/1587890830 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ccc0711e0 con 0x7f3ccc06cdf0 2023-05-22T15:42:38.808 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.804+0000 7f3cd0d37640 1 --2- 172.21.15.16:0/1587890830 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3ccc06cdf0 0x7f3ccc070be0 secure :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 crypto rx=0x7f3cc4009e50 tx=0x7f3cc401c980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1be6afca9ca40ff4 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:38.809 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.804+0000 7f3ccaffd640 1 -- 172.21.15.16:0/1587890830 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cc40a5840 con 0x7f3ccc06cdf0 2023-05-22T15:42:38.809 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.804+0000 7f3ccaffd640 1 -- 172.21.15.16:0/1587890830 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3cc40a59a0 con 0x7f3ccc06cdf0 2023-05-22T15:42:38.809 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.804+0000 7f3ccaffd640 1 -- 172.21.15.16:0/1587890830 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cc40a5d30 con 0x7f3ccc06cdf0 2023-05-22T15:42:38.809 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.804+0000 7f3cd279a640 1 -- 172.21.15.16:0/1587890830 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3ccc06cdf0 msgr2=0x7f3ccc070be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:38.810 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.804+0000 7f3cd279a640 1 --2- 172.21.15.16:0/1587890830 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3ccc06cdf0 0x7f3ccc070be0 secure :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 crypto rx=0x7f3cc4009e50 tx=0x7f3cc401c980 comp rx=0 tx=0).stop 2023-05-22T15:42:38.810 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.804+0000 7f3cd279a640 1 -- 172.21.15.16:0/1587890830 shutdown_connections 2023-05-22T15:42:38.810 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.804+0000 7f3cd279a640 1 --2- 172.21.15.16:0/1587890830 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3ccc06cdf0 0x7f3ccc070be0 unknown :-1 s=CLOSED pgs=1174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:38.810 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.804+0000 7f3cd279a640 1 -- 172.21.15.16:0/1587890830 >> 172.21.15.16:0/1587890830 conn(0x7f3ccc0cc870 msgr2=0x7f3ccc0cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:38.810 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.804+0000 7f3cd279a640 1 -- 172.21.15.16:0/1587890830 shutdown_connections 2023-05-22T15:42:38.810 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.804+0000 7f3cd279a640 1 -- 172.21.15.16:0/1587890830 wait complete. 2023-05-22T15:42:38.811 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.808+0000 7f3cd279a640 1 Processor -- start 2023-05-22T15:42:38.811 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.808+0000 7f3cd279a640 1 -- start start 2023-05-22T15:42:38.812 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.808+0000 7f3cd279a640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3ccc06cdf0 0x7f3ccc1f7da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:38.813 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.808+0000 7f3cd279a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3ccc070760 0x7f3ccc1f9060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:38.813 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.808+0000 7f3cd279a640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3ccc158a10 0x7f3ccc152af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:38.813 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.808+0000 7f3cd279a640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ccc156fe0 con 0x7f3ccc070760 2023-05-22T15:42:38.813 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.808+0000 7f3cd279a640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ccc1563d0 con 0x7f3ccc06cdf0 2023-05-22T15:42:38.813 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.808+0000 7f3cd279a640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ccc1566b0 con 0x7f3ccc158a10 2023-05-22T15:42:38.813 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.808+0000 7f3ccb7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3ccc070760 0x7f3ccc1f9060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:38.813 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.808+0000 7f3ccb7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3ccc070760 0x7f3ccc1f9060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57138/0 (socket says 172.21.15.16:57138) 2023-05-22T15:42:38.814 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.808+0000 7f3ccb7fe640 1 -- 172.21.15.16:0/2323246124 learned_addr learned my addr 172.21.15.16:0/2323246124 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:42:38.814 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.808+0000 7f3cd0d37640 1 --2- 172.21.15.16:0/2323246124 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3ccc158a10 0x7f3ccc152af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:38.814 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.808+0000 7f3ccbfff640 1 --2- 172.21.15.16:0/2323246124 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3ccc06cdf0 0x7f3ccc1f7da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:38.814 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.808+0000 7f3ccb7fe640 1 -- 172.21.15.16:0/2323246124 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3ccc158a10 msgr2=0x7f3ccc152af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:38.814 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.808+0000 7f3ccb7fe640 1 --2- 172.21.15.16:0/2323246124 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3ccc158a10 0x7f3ccc152af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:38.814 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.808+0000 7f3ccb7fe640 1 -- 172.21.15.16:0/2323246124 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3ccc06cdf0 msgr2=0x7f3ccc1f7da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:38.814 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.808+0000 7f3ccb7fe640 1 --2- 172.21.15.16:0/2323246124 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3ccc06cdf0 0x7f3ccc1f7da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:38.815 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.808+0000 7f3ccb7fe640 1 -- 172.21.15.16:0/2323246124 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cc4009b30 con 0x7f3ccc070760 2023-05-22T15:42:38.815 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.808+0000 7f3ccbfff640 1 --2- 172.21.15.16:0/2323246124 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3ccc06cdf0 0x7f3ccc1f7da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T15:42:38.815 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.808+0000 7f3ccb7fe640 1 --2- 172.21.15.16:0/2323246124 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3ccc070760 0x7f3ccc1f9060 secure :-1 s=READY pgs=1175 cs=0 l=1 rev1=1 crypto rx=0x7f3cc000c410 tx=0x7f3cc000c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:38.815 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.808+0000 7f3cc8ff9640 1 -- 172.21.15.16:0/2323246124 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cc00074e0 con 0x7f3ccc070760 2023-05-22T15:42:38.815 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.808+0000 7f3cc8ff9640 1 -- 172.21.15.16:0/2323246124 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3cc0007640 con 0x7f3ccc070760 2023-05-22T15:42:38.815 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.808+0000 7f3cd279a640 1 -- 172.21.15.16:0/2323246124 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ccc0ca280 con 0x7f3ccc070760 2023-05-22T15:42:38.815 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.808+0000 7f3cc8ff9640 1 -- 172.21.15.16:0/2323246124 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cc0004a30 con 0x7f3ccc070760 2023-05-22T15:42:38.815 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.808+0000 7f3cd279a640 1 -- 172.21.15.16:0/2323246124 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ccc0cfe40 con 0x7f3ccc070760 2023-05-22T15:42:38.816 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.808+0000 7f3cd279a640 1 -- 172.21.15.16:0/2323246124 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ccc0ca410 con 0x7f3ccc070760 2023-05-22T15:42:38.817 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.812+0000 7f3cc8ff9640 1 -- 172.21.15.16:0/2323246124 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 30) v1 ==== 84068+0+0 (secure 0 0 0) 0x7f3cc0004b90 con 0x7f3ccc070760 2023-05-22T15:42:38.817 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.812+0000 7f3cc8ff9640 1 --2- 172.21.15.16:0/2323246124 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f3cb80651d0 0x7f3cb8067680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:38.817 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.812+0000 7f3ccbfff640 1 -- 172.21.15.16:0/2323246124 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f3cb80651d0 msgr2=0x7f3cb8067680 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/36921 2023-05-22T15:42:38.817 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.812+0000 7f3cc8ff9640 1 -- 172.21.15.16:0/2323246124 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 6726+0+0 (secure 0 0 0) 0x7f3cc0082a70 con 0x7f3ccc070760 2023-05-22T15:42:38.817 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.812+0000 7f3ccbfff640 1 --2- 172.21.15.16:0/2323246124 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f3cb80651d0 0x7f3cb8067680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-05-22T15:42:38.818 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.816+0000 7f3cc8ff9640 1 -- 172.21.15.16:0/2323246124 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f3cc0050850 con 0x7f3ccc070760 2023-05-22T15:42:38.944 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.940+0000 7f3cc8ff9640 1 -- 172.21.15.16:0/2323246124 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mgrmap(e 31) v1 ==== 84389+0+0 (secure 0 0 0) 0x7f3cc0050c20 con 0x7f3ccc070760 2023-05-22T15:42:38.971 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.968+0000 7f3cd279a640 1 -- 172.21.15.16:0/2323246124 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f3ccc04dbe0 con 0x7f3ccc070760 2023-05-22T15:42:38.972 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.968+0000 7f3cc8ff9640 1 -- 172.21.15.16:0/2323246124 <== mon.0 v2:172.21.15.16:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v91) v1 ==== 92+0+10 (secure 0 0 0) 0x7f3cc0054050 con 0x7f3ccc070760 2023-05-22T15:42:38.972 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 8 2023-05-22T15:42:38.974 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.968+0000 7f3cd279a640 1 -- 172.21.15.16:0/2323246124 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f3cb80651d0 msgr2=0x7f3cb8067680 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-05-22T15:42:38.974 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.968+0000 7f3cd279a640 1 --2- 172.21.15.16:0/2323246124 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f3cb80651d0 0x7f3cb8067680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:38.975 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.968+0000 7f3cd279a640 1 -- 172.21.15.16:0/2323246124 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3ccc070760 msgr2=0x7f3ccc1f9060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:38.975 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.968+0000 7f3cd279a640 1 --2- 172.21.15.16:0/2323246124 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3ccc070760 0x7f3ccc1f9060 secure :-1 s=READY pgs=1175 cs=0 l=1 rev1=1 crypto rx=0x7f3cc000c410 tx=0x7f3cc000c8d0 comp rx=0 tx=0).stop 2023-05-22T15:42:38.975 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.968+0000 7f3cd279a640 1 -- 172.21.15.16:0/2323246124 shutdown_connections 2023-05-22T15:42:38.975 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.968+0000 7f3cd279a640 1 --2- 172.21.15.16:0/2323246124 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f3cb80651d0 0x7f3cb8067680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:38.975 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.968+0000 7f3cd279a640 1 --2- 172.21.15.16:0/2323246124 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3ccc158a10 0x7f3ccc152af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:38.975 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.968+0000 7f3cd279a640 1 --2- 172.21.15.16:0/2323246124 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3ccc070760 0x7f3ccc1f9060 unknown :-1 s=CLOSED pgs=1175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:38.976 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.968+0000 7f3cd279a640 1 --2- 172.21.15.16:0/2323246124 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3ccc06cdf0 0x7f3ccc1f7da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:38.976 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.968+0000 7f3cd279a640 1 -- 172.21.15.16:0/2323246124 >> 172.21.15.16:0/2323246124 conn(0x7f3ccc0cc870 msgr2=0x7f3ccc0b0c30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:38.976 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.972+0000 7f3cd279a640 1 -- 172.21.15.16:0/2323246124 shutdown_connections 2023-05-22T15:42:38.976 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:38.972+0000 7f3cd279a640 1 -- 172.21.15.16:0/2323246124 wait complete. 2023-05-22T15:42:38.990 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-05-22T15:42:39.041 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:42:39.039+0000 7ff8c72cb180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-05-22T15:42:39.068 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:42:39.064+0000 7fa9cb8de180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-05-22T15:42:39.094 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:42:39.091+0000 7ff8c72cb180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-05-22T15:42:39.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.108+0000 7fe9dbb12640 1 Processor -- start 2023-05-22T15:42:39.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.108+0000 7fe9dbb12640 1 -- start start 2023-05-22T15:42:39.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.112+0000 7fe9dbb12640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe9d4070760 0x7fe9d4070be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:39.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.112+0000 7fe9dbb12640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe9d406c6e0 0x7fe9d406cb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:39.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.112+0000 7fe9dbb12640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9d406eb90 con 0x7fe9d4158a10 2023-05-22T15:42:39.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.112+0000 7fe9dbb12640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9d406e060 con 0x7fe9d4070760 2023-05-22T15:42:39.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.112+0000 7fe9dbb12640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9d406d0c0 con 0x7fe9d406c6e0 2023-05-22T15:42:39.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.112+0000 7fe9d90ad640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe9d4070760 0x7fe9d4070be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:39.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.112+0000 7fe9d90ad640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe9d4070760 0x7fe9d4070be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57142/0 (socket says 172.21.15.16:57142) 2023-05-22T15:42:39.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.112+0000 7fe9d90ad640 1 -- 172.21.15.16:0/2045812702 learned_addr learned my addr 172.21.15.16:0/2045812702 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:42:39.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.112+0000 7fe9da0af640 1 --2- 172.21.15.16:0/2045812702 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe9d406c6e0 0x7fe9d406cb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:39.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.112+0000 7fe9d90ad640 1 -- 172.21.15.16:0/2045812702 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe9d406c6e0 msgr2=0x7fe9d406cb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:42:39.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.112+0000 7fe9d90ad640 1 --2- 172.21.15.16:0/2045812702 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe9d406c6e0 0x7fe9d406cb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:39.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.112+0000 7fe9d90ad640 1 -- 172.21.15.16:0/2045812702 >> v1:172.21.15.62:6789/0 conn(0x7fe9d4158a10 legacy=0x7fe9d406e480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:42:39.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.112+0000 7fe9d90ad640 1 -- 172.21.15.16:0/2045812702 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9d4158de0 con 0x7fe9d4070760 2023-05-22T15:42:39.118 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.112+0000 7fe9d90ad640 1 --2- 172.21.15.16:0/2045812702 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe9d4070760 0x7fe9d4070be0 secure :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 crypto rx=0x7fe9c4009870 tx=0x7fe9c401c470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=590fdadbdb95398c server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:39.118 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.112+0000 7fe9d88ac640 1 -- 172.21.15.16:0/2045812702 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9c40a7070 con 0x7fe9d4070760 2023-05-22T15:42:39.118 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.112+0000 7fe9d88ac640 1 -- 172.21.15.16:0/2045812702 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9c40ab430 con 0x7fe9d4070760 2023-05-22T15:42:39.119 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.112+0000 7fe9d88ac640 1 -- 172.21.15.16:0/2045812702 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9c40ab7c0 con 0x7fe9d4070760 2023-05-22T15:42:39.119 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.116+0000 7fe9dbb12640 1 -- 172.21.15.16:0/2045812702 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe9d4070760 msgr2=0x7fe9d4070be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:39.119 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.116+0000 7fe9dbb12640 1 --2- 172.21.15.16:0/2045812702 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe9d4070760 0x7fe9d4070be0 secure :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 crypto rx=0x7fe9c4009870 tx=0x7fe9c401c470 comp rx=0 tx=0).stop 2023-05-22T15:42:39.119 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.116+0000 7fe9dbb12640 1 -- 172.21.15.16:0/2045812702 shutdown_connections 2023-05-22T15:42:39.119 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.116+0000 7fe9dbb12640 1 --2- 172.21.15.16:0/2045812702 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe9d406c6e0 0x7fe9d406cb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:39.119 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.116+0000 7fe9dbb12640 1 --2- 172.21.15.16:0/2045812702 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe9d4070760 0x7fe9d4070be0 unknown :-1 s=CLOSED pgs=1176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:39.119 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.116+0000 7fe9dbb12640 1 -- 172.21.15.16:0/2045812702 >> 172.21.15.16:0/2045812702 conn(0x7fe9d40cc870 msgr2=0x7fe9d40cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:39.120 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.116+0000 7fe9dbb12640 1 -- 172.21.15.16:0/2045812702 shutdown_connections 2023-05-22T15:42:39.120 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.116+0000 7fe9dbb12640 1 -- 172.21.15.16:0/2045812702 wait complete. 2023-05-22T15:42:39.122 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.116+0000 7fe9dbb12640 1 Processor -- start 2023-05-22T15:42:39.122 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.116+0000 7fe9dbb12640 1 -- start start 2023-05-22T15:42:39.122 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.116+0000 7fe9dbb12640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe9d406c6e0 0x7fe9d40b1f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:39.122 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.116+0000 7fe9dbb12640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe9d4070760 0x7fe9d414f560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:39.122 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.116+0000 7fe9dbb12640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe9d4158a10 0x7fe9d414ffc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:39.122 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.116+0000 7fe9dbb12640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9d415bde0 con 0x7fe9d406c6e0 2023-05-22T15:42:39.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.116+0000 7fe9dbb12640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9d40b2600 con 0x7fe9d4158a10 2023-05-22T15:42:39.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.116+0000 7fe9dbb12640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9d40b28e0 con 0x7fe9d4070760 2023-05-22T15:42:39.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.116+0000 7fe9d98ae640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe9d406c6e0 0x7fe9d40b1f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:39.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.116+0000 7fe9d98ae640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe9d406c6e0 0x7fe9d40b1f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57150/0 (socket says 172.21.15.16:57150) 2023-05-22T15:42:39.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.116+0000 7fe9d98ae640 1 -- 172.21.15.16:0/3245538564 learned_addr learned my addr 172.21.15.16:0/3245538564 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:42:39.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.116+0000 7fe9d90ad640 1 --2- 172.21.15.16:0/3245538564 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe9d4070760 0x7fe9d414f560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:39.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.116+0000 7fe9da0af640 1 --2- 172.21.15.16:0/3245538564 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe9d4158a10 0x7fe9d414ffc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:39.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.116+0000 7fe9d98ae640 1 -- 172.21.15.16:0/3245538564 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe9d4070760 msgr2=0x7fe9d414f560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:39.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.116+0000 7fe9d98ae640 1 --2- 172.21.15.16:0/3245538564 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe9d4070760 0x7fe9d414f560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:39.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.116+0000 7fe9d98ae640 1 -- 172.21.15.16:0/3245538564 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe9d4158a10 msgr2=0x7fe9d414ffc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:39.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.116+0000 7fe9d98ae640 1 --2- 172.21.15.16:0/3245538564 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe9d4158a10 0x7fe9d414ffc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:39.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.116+0000 7fe9d98ae640 1 -- 172.21.15.16:0/3245538564 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9c40094d0 con 0x7fe9d406c6e0 2023-05-22T15:42:39.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.116+0000 7fe9d98ae640 1 --2- 172.21.15.16:0/3245538564 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe9d406c6e0 0x7fe9d40b1f90 secure :-1 s=READY pgs=1177 cs=0 l=1 rev1=1 crypto rx=0x7fe9d000b410 tx=0x7fe9d000b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:39.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.116+0000 7fe9ca7fc640 1 -- 172.21.15.16:0/3245538564 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9d0004020 con 0x7fe9d406c6e0 2023-05-22T15:42:39.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.116+0000 7fe9ca7fc640 1 -- 172.21.15.16:0/3245538564 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9d0004180 con 0x7fe9d406c6e0 2023-05-22T15:42:39.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.116+0000 7fe9dbb12640 1 -- 172.21.15.16:0/3245538564 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9d40cfe40 con 0x7fe9d406c6e0 2023-05-22T15:42:39.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.116+0000 7fe9ca7fc640 1 -- 172.21.15.16:0/3245538564 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9d0011630 con 0x7fe9d406c6e0 2023-05-22T15:42:39.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.116+0000 7fe9dbb12640 1 -- 172.21.15.16:0/3245538564 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9d406dc80 con 0x7fe9d406c6e0 2023-05-22T15:42:39.126 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.120+0000 7fe9dbb12640 1 -- 172.21.15.16:0/3245538564 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9d40ca280 con 0x7fe9d406c6e0 2023-05-22T15:42:39.126 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.120+0000 7fe9ca7fc640 1 -- 172.21.15.16:0/3245538564 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 31) v1 ==== 84389+0+0 (secure 0 0 0) 0x7fe9d0011790 con 0x7fe9d406c6e0 2023-05-22T15:42:39.126 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.120+0000 7fe9ca7fc640 1 --2- 172.21.15.16:0/3245538564 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fe9b8065530 0x7fe9b80679e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:39.126 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.120+0000 7fe9d90ad640 1 -- 172.21.15.16:0/3245538564 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fe9b8065530 msgr2=0x7fe9b80679e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/36921 2023-05-22T15:42:39.126 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.120+0000 7fe9ca7fc640 1 -- 172.21.15.16:0/3245538564 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 6726+0+0 (secure 0 0 0) 0x7fe9d0082ad0 con 0x7fe9d406c6e0 2023-05-22T15:42:39.127 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.120+0000 7fe9d90ad640 1 --2- 172.21.15.16:0/3245538564 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fe9b8065530 0x7fe9b80679e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-05-22T15:42:39.127 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:42:39.120+0000 7fa9cb8de180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-05-22T15:42:39.127 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.124+0000 7fe9ca7fc640 1 -- 172.21.15.16:0/3245538564 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fe9d00502c0 con 0x7fe9d406c6e0 2023-05-22T15:42:39.144 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:42:39.143+0000 7ff8c72cb180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-05-22T15:42:39.175 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:42:39.172+0000 7fa9cb8de180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-05-22T15:42:39.222 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:42:39.219+0000 7ff8c72cb180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-05-22T15:42:39.253 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:42:39.248+0000 7fa9cb8de180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-05-22T15:42:39.274 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:42:39.271+0000 7ff8c72cb180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-05-22T15:42:39.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.276+0000 7fe9dbb12640 1 -- 172.21.15.16:0/3245538564 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fe9d404dbe0 con 0x7fe9d406c6e0 2023-05-22T15:42:39.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.276+0000 7fe9ca7fc640 1 -- 172.21.15.16:0/3245538564 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v91) v1 ==== 104+0+11 (secure 0 0 0) 0x7fe9d0015020 con 0x7fe9d406c6e0 2023-05-22T15:42:39.281 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2023-05-22T15:42:39.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.280+0000 7fe9dbb12640 1 -- 172.21.15.16:0/3245538564 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fe9b8065530 msgr2=0x7fe9b80679e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-05-22T15:42:39.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.280+0000 7fe9dbb12640 1 --2- 172.21.15.16:0/3245538564 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fe9b8065530 0x7fe9b80679e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:39.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.280+0000 7fe9dbb12640 1 -- 172.21.15.16:0/3245538564 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe9d406c6e0 msgr2=0x7fe9d40b1f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:39.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.280+0000 7fe9dbb12640 1 --2- 172.21.15.16:0/3245538564 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe9d406c6e0 0x7fe9d40b1f90 secure :-1 s=READY pgs=1177 cs=0 l=1 rev1=1 crypto rx=0x7fe9d000b410 tx=0x7fe9d000b8d0 comp rx=0 tx=0).stop 2023-05-22T15:42:39.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.280+0000 7fe9dbb12640 1 -- 172.21.15.16:0/3245538564 shutdown_connections 2023-05-22T15:42:39.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.280+0000 7fe9dbb12640 1 --2- 172.21.15.16:0/3245538564 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe9d4158a10 0x7fe9d414ffc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:39.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.280+0000 7fe9dbb12640 1 --2- 172.21.15.16:0/3245538564 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fe9b8065530 0x7fe9b80679e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:39.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.280+0000 7fe9dbb12640 1 --2- 172.21.15.16:0/3245538564 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe9d4070760 0x7fe9d414f560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:39.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.280+0000 7fe9dbb12640 1 --2- 172.21.15.16:0/3245538564 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe9d406c6e0 0x7fe9d40b1f90 unknown :-1 s=CLOSED pgs=1177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:39.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.280+0000 7fe9dbb12640 1 -- 172.21.15.16:0/3245538564 >> 172.21.15.16:0/3245538564 conn(0x7fe9d40cc870 msgr2=0x7fe9d40cae70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:39.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.280+0000 7fe9dbb12640 1 -- 172.21.15.16:0/3245538564 shutdown_connections 2023-05-22T15:42:39.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.280+0000 7fe9dbb12640 1 -- 172.21.15.16:0/3245538564 wait complete. 2023-05-22T15:42:39.299 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-05-22T15:42:39.307 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:42:39.304+0000 7fa9cb8de180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-05-22T15:42:39.350 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:42:39.347+0000 7ff8c72cb180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-05-22T15:42:39.387 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:42:39.384+0000 7fa9cb8de180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-05-22T15:42:39.402 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:42:39.399+0000 7ff8c72cb180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-05-22T15:42:39.427 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.424+0000 7f564ff56640 1 Processor -- start 2023-05-22T15:42:39.427 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.424+0000 7f564ff56640 1 -- start start 2023-05-22T15:42:39.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.424+0000 7f564ff56640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5648153fe0 0x7f564806d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:39.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.424+0000 7f564ff56640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f5648070600 0x7f56480ad410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:39.432 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.424+0000 7f564ff56640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5648069720 con 0x7f56480cae20 2023-05-22T15:42:39.432 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.424+0000 7f564ff56640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56481543b0 con 0x7f5648070600 2023-05-22T15:42:39.432 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.424+0000 7f564ff56640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f564815e360 con 0x7f5648153fe0 2023-05-22T15:42:39.432 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.424+0000 7f564d4f1640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f56480cae20 0x7f564806fef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:41038/0 (socket says 172.21.15.16:41038) 2023-05-22T15:42:39.432 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.424+0000 7f564d4f1640 1 -- 172.21.15.16:0/1514242681 learned_addr learned my addr 172.21.15.16:0/1514242681 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:42:39.432 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.424+0000 7f564dcf2640 1 --2- 172.21.15.16:0/1514242681 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5648153fe0 0x7f564806d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:39.432 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.424+0000 7f564e4f3640 1 --2- 172.21.15.16:0/1514242681 >> v2:172.21.15.62:3300/0 conn(0x7f5648070600 0x7f56480ad410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:39.433 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.424+0000 7f564ccf0640 1 -- 172.21.15.16:0/1514242681 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2313116045 0 0) 0x7f5648069720 con 0x7f56480cae20 2023-05-22T15:42:39.433 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.424+0000 7f564ccf0640 1 -- 172.21.15.16:0/1514242681 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5634003530 con 0x7f56480cae20 2023-05-22T15:42:39.433 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.424+0000 7f564ccf0640 1 -- 172.21.15.16:0/1514242681 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1306131173 0 0) 0x7f5634003530 con 0x7f56480cae20 2023-05-22T15:42:39.433 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.424+0000 7f564ccf0640 1 -- 172.21.15.16:0/1514242681 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5648153fe0 msgr2=0x7f564806d3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:42:39.433 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.424+0000 7f564ccf0640 1 --2- 172.21.15.16:0/1514242681 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5648153fe0 0x7f564806d3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:39.433 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.424+0000 7f564ccf0640 1 -- 172.21.15.16:0/1514242681 >> v2:172.21.15.62:3300/0 conn(0x7f5648070600 msgr2=0x7f56480ad410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:42:39.433 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.424+0000 7f564ccf0640 1 --2- 172.21.15.16:0/1514242681 >> v2:172.21.15.62:3300/0 conn(0x7f5648070600 0x7f56480ad410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:39.433 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.424+0000 7f564ccf0640 1 -- 172.21.15.16:0/1514242681 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f564815d5a0 con 0x7f56480cae20 2023-05-22T15:42:39.434 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.424+0000 7f564ccf0640 1 -- 172.21.15.16:0/1514242681 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7f5638001680 con 0x7f56480cae20 2023-05-22T15:42:39.434 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.424+0000 7f564ccf0640 1 -- 172.21.15.16:0/1514242681 >> v1:172.21.15.16:6789/0 conn(0x7f56480cae20 legacy=0x7f564806fef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:39.434 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.424+0000 7f564ccf0640 1 --2- 172.21.15.16:0/1514242681 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5648153fe0 0x7f5634003980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:39.434 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.424+0000 7f564ccf0640 1 --2- 172.21.15.16:0/1514242681 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5634003f00 0x7f56340047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:39.434 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.424+0000 7f564ccf0640 1 --2- 172.21.15.16:0/1514242681 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5634004d10 0x7f5634005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:39.434 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.424+0000 7f564ccf0640 1 -- 172.21.15.16:0/1514242681 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56481543b0 con 0x7f5634003f00 2023-05-22T15:42:39.434 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.424+0000 7f564ccf0640 1 -- 172.21.15.16:0/1514242681 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f564815e360 con 0x7f5634004d10 2023-05-22T15:42:39.434 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.424+0000 7f564ccf0640 1 -- 172.21.15.16:0/1514242681 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f563400b9c0 con 0x7f5648153fe0 2023-05-22T15:42:39.435 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.424+0000 7f564e4f3640 1 --2- 172.21.15.16:0/1514242681 >> v2:172.21.15.62:3300/0 conn(0x7f5648070600 0x7f56480ad410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T15:42:39.435 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.424+0000 7f564d4f1640 1 --2- 172.21.15.16:0/1514242681 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5634003f00 0x7f56340047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:39.435 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.424+0000 7f564e4f3640 1 --2- 172.21.15.16:0/1514242681 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5634004d10 0x7f5634005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:39.435 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.424+0000 7f564dcf2640 1 --2- 172.21.15.16:0/1514242681 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5648153fe0 0x7f5634003980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:39.435 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.424+0000 7f564d4f1640 1 -- 172.21.15.16:0/1514242681 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5648153fe0 msgr2=0x7f5634003980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:42:39.435 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.424+0000 7f564d4f1640 1 --2- 172.21.15.16:0/1514242681 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5648153fe0 0x7f5634003980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:39.435 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.424+0000 7f564d4f1640 1 -- 172.21.15.16:0/1514242681 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5634004d10 msgr2=0x7f5634005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:42:39.436 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.424+0000 7f564d4f1640 1 --2- 172.21.15.16:0/1514242681 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5634004d10 0x7f5634005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:39.436 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.424+0000 7f564d4f1640 1 -- 172.21.15.16:0/1514242681 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5634003530 con 0x7f5634003f00 2023-05-22T15:42:39.436 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.424+0000 7f564e4f3640 1 --2- 172.21.15.16:0/1514242681 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5634004d10 0x7f5634005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:42:39.436 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.428+0000 7f564d4f1640 1 --2- 172.21.15.16:0/1514242681 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5634003f00 0x7f56340047d0 secure :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 crypto rx=0x7f563800c240 tx=0x7f563801f3d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b14cf577a1ddde19 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:39.436 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.428+0000 7f564ccf0640 1 -- 172.21.15.16:0/1514242681 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5638005ae0 con 0x7f5634003f00 2023-05-22T15:42:39.436 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.428+0000 7f564ccf0640 1 -- 172.21.15.16:0/1514242681 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56380ab070 con 0x7f5634003f00 2023-05-22T15:42:39.436 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.428+0000 7f564ccf0640 1 -- 172.21.15.16:0/1514242681 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5638005e80 con 0x7f5634003f00 2023-05-22T15:42:39.439 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.428+0000 7f564ff56640 1 -- 172.21.15.16:0/1514242681 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5634003f00 msgr2=0x7f56340047d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:39.439 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.428+0000 7f564ff56640 1 --2- 172.21.15.16:0/1514242681 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5634003f00 0x7f56340047d0 secure :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 crypto rx=0x7f563800c240 tx=0x7f563801f3d0 comp rx=0 tx=0).stop 2023-05-22T15:42:39.439 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.428+0000 7f564e4f3640 1 -- 172.21.15.16:0/1514242681 reap_dead start 2023-05-22T15:42:39.439 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.428+0000 7f564ff56640 1 -- 172.21.15.16:0/1514242681 shutdown_connections 2023-05-22T15:42:39.439 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.428+0000 7f564ff56640 1 -- 172.21.15.16:0/1514242681 >> 172.21.15.16:0/1514242681 conn(0x7f56480b1110 msgr2=0x7f56480ce830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:39.439 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.428+0000 7f564ff56640 1 -- 172.21.15.16:0/1514242681 shutdown_connections 2023-05-22T15:42:39.440 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.428+0000 7f564ff56640 1 -- 172.21.15.16:0/1514242681 wait complete. 2023-05-22T15:42:39.440 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.428+0000 7f564ff56640 1 Processor -- start 2023-05-22T15:42:39.440 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.428+0000 7f564ff56640 1 -- start start 2023-05-22T15:42:39.440 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.428+0000 7f564ff56640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5634003f00 0x7f56481f2dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:39.440 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.428+0000 7f564ff56640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5648156ef0 0x7f5648157360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:39.441 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.428+0000 7f564ff56640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f56480aea20 0x7f56480aee90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:39.441 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.428+0000 7f564ff56640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f564814ec00 con 0x7f5634003f00 2023-05-22T15:42:39.441 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.428+0000 7f564ff56640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56480af3d0 con 0x7f56480aea20 2023-05-22T15:42:39.441 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.428+0000 7f564ff56640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56481efb70 con 0x7f5648156ef0 2023-05-22T15:42:39.441 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.428+0000 7f564dcf2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5634003f00 0x7f56481f2dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:39.441 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.428+0000 7f564dcf2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5634003f00 0x7f56481f2dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48976/0 (socket says 172.21.15.16:48976) 2023-05-22T15:42:39.441 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.428+0000 7f564dcf2640 1 -- 172.21.15.16:0/1852058945 learned_addr learned my addr 172.21.15.16:0/1852058945 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:42:39.442 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.432+0000 7f564d4f1640 1 --2- 172.21.15.16:0/1852058945 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5648156ef0 0x7f5648157360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:39.442 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.432+0000 7f564e4f3640 1 --2- 172.21.15.16:0/1852058945 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f56480aea20 0x7f56480aee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:39.442 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.432+0000 7f564dcf2640 1 -- 172.21.15.16:0/1852058945 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5648156ef0 msgr2=0x7f5648157360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:39.442 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.432+0000 7f564dcf2640 1 --2- 172.21.15.16:0/1852058945 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5648156ef0 0x7f5648157360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:39.442 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.432+0000 7f564dcf2640 1 -- 172.21.15.16:0/1852058945 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f56480aea20 msgr2=0x7f56480aee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:39.442 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.432+0000 7f564dcf2640 1 --2- 172.21.15.16:0/1852058945 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f56480aea20 0x7f56480aee90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:39.442 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.432+0000 7f564dcf2640 1 -- 172.21.15.16:0/1852058945 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f564815db60 con 0x7f5634003f00 2023-05-22T15:42:39.442 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.432+0000 7f564d4f1640 1 --2- 172.21.15.16:0/1852058945 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5648156ef0 0x7f5648157360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T15:42:39.443 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.432+0000 7f564dcf2640 1 --2- 172.21.15.16:0/1852058945 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5634003f00 0x7f56481f2dc0 secure :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 crypto rx=0x7f5644010890 tx=0x7f5644010d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:39.443 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.432+0000 7f563e7fc640 1 -- 172.21.15.16:0/1852058945 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5644009630 con 0x7f5634003f00 2023-05-22T15:42:39.443 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.432+0000 7f563e7fc640 1 -- 172.21.15.16:0/1852058945 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f564400eb60 con 0x7f5634003f00 2023-05-22T15:42:39.443 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.432+0000 7f564ff56640 1 -- 172.21.15.16:0/1852058945 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f564815d880 con 0x7f5634003f00 2023-05-22T15:42:39.443 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.432+0000 7f563e7fc640 1 -- 172.21.15.16:0/1852058945 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f564401c400 con 0x7f5634003f00 2023-05-22T15:42:39.443 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.432+0000 7f564ff56640 1 -- 172.21.15.16:0/1852058945 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f564815de40 con 0x7f5634003f00 2023-05-22T15:42:39.443 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.432+0000 7f564ff56640 1 -- 172.21.15.16:0/1852058945 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f564804dbe0 con 0x7f5634003f00 2023-05-22T15:42:39.443 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.432+0000 7f563e7fc640 1 -- 172.21.15.16:0/1852058945 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 31) v1 ==== 84389+0+0 (secure 0 0 0) 0x7f564401c560 con 0x7f5634003f00 2023-05-22T15:42:39.444 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.432+0000 7f563e7fc640 1 --2- 172.21.15.16:0/1852058945 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f562c065530 0x7f562c0679e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:39.444 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.432+0000 7f564d4f1640 1 -- 172.21.15.16:0/1852058945 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f562c065530 msgr2=0x7f562c0679e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/36921 2023-05-22T15:42:39.444 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.432+0000 7f563e7fc640 1 -- 172.21.15.16:0/1852058945 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 6726+0+0 (secure 0 0 0) 0x7f5644083380 con 0x7f5634003f00 2023-05-22T15:42:39.444 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.432+0000 7f564d4f1640 1 --2- 172.21.15.16:0/1852058945 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f562c065530 0x7f562c0679e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-05-22T15:42:39.444 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:42:39.436+0000 7fa9cb8de180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-05-22T15:42:39.445 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.436+0000 7f563e7fc640 1 -- 172.21.15.16:0/1852058945 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f5644050b30 con 0x7f5634003f00 2023-05-22T15:42:39.534 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:42:39.531+0000 7ff8c72cb180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-05-22T15:42:39.575 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:42:39.572+0000 7fa9cb8de180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-05-22T15:42:39.587 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:42:39.583+0000 7ff8c72cb180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-05-22T15:42:39.594 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.588+0000 7f564ff56640 1 -- 172.21.15.16:0/1852058945 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f56480afed0 con 0x7f5634003f00 2023-05-22T15:42:39.595 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.592+0000 7f563e7fc640 1 -- 172.21.15.16:0/1852058945 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v91) v1 ==== 100+0+11 (secure 0 0 0) 0x7f5644054150 con 0x7f5634003f00 2023-05-22T15:42:39.595 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 64 2023-05-22T15:42:39.596 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.592+0000 7f564ff56640 1 -- 172.21.15.16:0/1852058945 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f562c065530 msgr2=0x7f562c0679e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-05-22T15:42:39.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.592+0000 7f564ff56640 1 --2- 172.21.15.16:0/1852058945 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f562c065530 0x7f562c0679e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:39.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.592+0000 7f564ff56640 1 -- 172.21.15.16:0/1852058945 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5634003f00 msgr2=0x7f56481f2dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:39.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.592+0000 7f564ff56640 1 --2- 172.21.15.16:0/1852058945 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5634003f00 0x7f56481f2dc0 secure :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 crypto rx=0x7f5644010890 tx=0x7f5644010d50 comp rx=0 tx=0).stop 2023-05-22T15:42:39.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.592+0000 7f564ff56640 1 -- 172.21.15.16:0/1852058945 shutdown_connections 2023-05-22T15:42:39.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.592+0000 7f564ff56640 1 --2- 172.21.15.16:0/1852058945 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f56480aea20 0x7f56480aee90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:39.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.592+0000 7f564ff56640 1 --2- 172.21.15.16:0/1852058945 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f562c065530 0x7f562c0679e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:39.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.592+0000 7f564ff56640 1 --2- 172.21.15.16:0/1852058945 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5648156ef0 0x7f5648157360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:39.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.592+0000 7f564ff56640 1 --2- 172.21.15.16:0/1852058945 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5634003f00 0x7f56481f2dc0 unknown :-1 s=CLOSED pgs=1180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:39.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.592+0000 7f564ff56640 1 -- 172.21.15.16:0/1852058945 >> 172.21.15.16:0/1852058945 conn(0x7f56480b1110 msgr2=0x7f56480ca260 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:39.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.592+0000 7f564ff56640 1 -- 172.21.15.16:0/1852058945 shutdown_connections 2023-05-22T15:42:39.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.592+0000 7f564ff56640 1 -- 172.21.15.16:0/1852058945 wait complete. 2023-05-22T15:42:39.613 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-05-22T15:42:39.630 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:42:39.624+0000 7fa9cb8de180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-05-22T15:42:39.739 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.736+0000 7fcadec25640 1 Processor -- start 2023-05-22T15:42:39.739 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.736+0000 7fcadec25640 1 -- start start 2023-05-22T15:42:39.740 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.736+0000 7fcadec25640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fcad815c130 0x7fcad80b2330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:39.740 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.736+0000 7fcadec25640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fcad80b2870 0x7fcad80cf510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:39.740 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.736+0000 7fcadec25640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fcad80cfa50 0x7fcad80d01b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:39.741 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.736+0000 7fcadec25640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcad806c4a0 con 0x7fcad815c130 2023-05-22T15:42:39.741 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.736+0000 7fcadec25640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcad80b2cc0 con 0x7fcad80b2870 2023-05-22T15:42:39.741 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.736+0000 7fcadec25640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcad80ce160 con 0x7fcad80cfa50 2023-05-22T15:42:39.741 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.736+0000 7fcadc9c1640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fcad815c130 0x7fcad80b2330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:39.741 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.736+0000 7fcadc9c1640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fcad815c130 0x7fcad80b2330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48990/0 (socket says 172.21.15.16:48990) 2023-05-22T15:42:39.741 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.736+0000 7fcadc9c1640 1 -- 172.21.15.16:0/1457870006 learned_addr learned my addr 172.21.15.16:0/1457870006 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:42:39.741 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.736+0000 7fcadd1c2640 1 --2- 172.21.15.16:0/1457870006 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fcad80cfa50 0x7fcad80d01b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:39.742 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.736+0000 7fcad7fff640 1 --2- 172.21.15.16:0/1457870006 >> v2:172.21.15.62:3300/0 conn(0x7fcad80b2870 0x7fcad80cf510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:39.742 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.736+0000 7fcadc9c1640 1 -- 172.21.15.16:0/1457870006 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fcad80cfa50 msgr2=0x7fcad80d01b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:42:39.742 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.736+0000 7fcadc9c1640 1 --2- 172.21.15.16:0/1457870006 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fcad80cfa50 0x7fcad80d01b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:39.742 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.736+0000 7fcadc9c1640 1 -- 172.21.15.16:0/1457870006 >> v2:172.21.15.62:3300/0 conn(0x7fcad80b2870 msgr2=0x7fcad80cf510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:42:39.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.736+0000 7fcadc9c1640 1 --2- 172.21.15.16:0/1457870006 >> v2:172.21.15.62:3300/0 conn(0x7fcad80b2870 0x7fcad80cf510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:39.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.736+0000 7fcadc9c1640 1 -- 172.21.15.16:0/1457870006 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcad80cece0 con 0x7fcad815c130 2023-05-22T15:42:39.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.736+0000 7fcadd1c2640 1 --2- 172.21.15.16:0/1457870006 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fcad80cfa50 0x7fcad80d01b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T15:42:39.744 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.740+0000 7fcadc9c1640 1 --2- 172.21.15.16:0/1457870006 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcad815c130 0x7fcad80b2330 secure :-1 s=READY pgs=1181 cs=0 l=1 rev1=1 crypto rx=0x7fcad806e960 tx=0x7fcac001c240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1fb9f56cd4581ab4 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:39.744 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.740+0000 7fcad77fe640 1 -- 172.21.15.16:0/1457870006 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcac00ab600 con 0x7fcad815c130 2023-05-22T15:42:39.744 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.740+0000 7fcad77fe640 1 -- 172.21.15.16:0/1457870006 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcac00a7050 con 0x7fcad815c130 2023-05-22T15:42:39.744 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.740+0000 7fcad77fe640 1 -- 172.21.15.16:0/1457870006 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcac00ab980 con 0x7fcad815c130 2023-05-22T15:42:39.744 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.740+0000 7fcadec25640 1 -- 172.21.15.16:0/1457870006 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcad815c130 msgr2=0x7fcad80b2330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:39.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.740+0000 7fcadec25640 1 --2- 172.21.15.16:0/1457870006 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcad815c130 0x7fcad80b2330 secure :-1 s=READY pgs=1181 cs=0 l=1 rev1=1 crypto rx=0x7fcad806e960 tx=0x7fcac001c240 comp rx=0 tx=0).stop 2023-05-22T15:42:39.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.740+0000 7fcadec25640 1 -- 172.21.15.16:0/1457870006 shutdown_connections 2023-05-22T15:42:39.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.740+0000 7fcadec25640 1 --2- 172.21.15.16:0/1457870006 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fcad80cfa50 0x7fcad80d01b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:39.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.740+0000 7fcadec25640 1 --2- 172.21.15.16:0/1457870006 >> v2:172.21.15.62:3300/0 conn(0x7fcad80b2870 0x7fcad80cf510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:39.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.740+0000 7fcadec25640 1 --2- 172.21.15.16:0/1457870006 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcad815c130 0x7fcad80b2330 unknown :-1 s=CLOSED pgs=1181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:39.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.740+0000 7fcadec25640 1 -- 172.21.15.16:0/1457870006 >> 172.21.15.16:0/1457870006 conn(0x7fcad806f5c0 msgr2=0x7fcad806f9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:39.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.740+0000 7fcadec25640 1 -- 172.21.15.16:0/1457870006 shutdown_connections 2023-05-22T15:42:39.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.740+0000 7fcadec25640 1 -- 172.21.15.16:0/1457870006 wait complete. 2023-05-22T15:42:39.747 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.740+0000 7fcadec25640 1 Processor -- start 2023-05-22T15:42:39.747 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.740+0000 7fcadec25640 1 -- start start 2023-05-22T15:42:39.747 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.740+0000 7fcadec25640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fcad80b2870 0x7fcad80c6670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:39.747 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.740+0000 7fcadec25640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcad80cfa50 0x7fcad80c46e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:39.748 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.740+0000 7fcadec25640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcad815c130 0x7fcad8150210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:39.748 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.740+0000 7fcadec25640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcad81538e0 con 0x7fcad815c130 2023-05-22T15:42:39.748 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.740+0000 7fcadec25640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcad8153bc0 con 0x7fcad80cfa50 2023-05-22T15:42:39.748 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.740+0000 7fcadec25640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcad8153ea0 con 0x7fcad80b2870 2023-05-22T15:42:39.748 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.740+0000 7fcadd1c2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcad815c130 0x7fcad8150210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:39.748 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.740+0000 7fcadd1c2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcad815c130 0x7fcad8150210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48992/0 (socket says 172.21.15.16:48992) 2023-05-22T15:42:39.748 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.740+0000 7fcadd1c2640 1 -- 172.21.15.16:0/575579729 learned_addr learned my addr 172.21.15.16:0/575579729 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:42:39.749 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.740+0000 7fcadc9c1640 1 --2- 172.21.15.16:0/575579729 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fcad80b2870 0x7fcad80c6670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:39.749 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.740+0000 7fcad7fff640 1 --2- 172.21.15.16:0/575579729 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcad80cfa50 0x7fcad80c46e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:39.749 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.744+0000 7fcadd1c2640 1 -- 172.21.15.16:0/575579729 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fcad80b2870 msgr2=0x7fcad80c6670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:39.749 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.744+0000 7fcadd1c2640 1 --2- 172.21.15.16:0/575579729 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fcad80b2870 0x7fcad80c6670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:39.749 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.744+0000 7fcadd1c2640 1 -- 172.21.15.16:0/575579729 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcad80cfa50 msgr2=0x7fcad80c46e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:39.749 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.744+0000 7fcadd1c2640 1 --2- 172.21.15.16:0/575579729 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcad80cfa50 0x7fcad80c46e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:39.749 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.744+0000 7fcadd1c2640 1 -- 172.21.15.16:0/575579729 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcac00094d0 con 0x7fcad815c130 2023-05-22T15:42:39.750 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.744+0000 7fcadd1c2640 1 --2- 172.21.15.16:0/575579729 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcad815c130 0x7fcad8150210 secure :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 crypto rx=0x7fcad0007ef0 tx=0x7fcad000d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:39.750 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.744+0000 7fcad57fa640 1 -- 172.21.15.16:0/575579729 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcad0003ba0 con 0x7fcad815c130 2023-05-22T15:42:39.750 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.744+0000 7fcad57fa640 1 -- 172.21.15.16:0/575579729 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcad000f040 con 0x7fcad815c130 2023-05-22T15:42:39.750 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.744+0000 7fcadec25640 1 -- 172.21.15.16:0/575579729 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcad80ce970 con 0x7fcad815c130 2023-05-22T15:42:39.750 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.744+0000 7fcad57fa640 1 -- 172.21.15.16:0/575579729 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcad000de80 con 0x7fcad815c130 2023-05-22T15:42:39.750 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.744+0000 7fcadec25640 1 -- 172.21.15.16:0/575579729 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcad80ce690 con 0x7fcad815c130 2023-05-22T15:42:39.750 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.744+0000 7fcadec25640 1 -- 172.21.15.16:0/575579729 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcad80ca8b0 con 0x7fcad815c130 2023-05-22T15:42:39.751 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.744+0000 7fcad57fa640 1 -- 172.21.15.16:0/575579729 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 31) v1 ==== 84389+0+0 (secure 0 0 0) 0x7fcad0004030 con 0x7fcad815c130 2023-05-22T15:42:39.751 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.744+0000 7fcad57fa640 1 --2- 172.21.15.16:0/575579729 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fcab8065530 0x7fcab80679e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:39.751 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.744+0000 7fcadc9c1640 1 -- 172.21.15.16:0/575579729 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fcab8065530 msgr2=0x7fcab80679e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/36921 2023-05-22T15:42:39.751 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.744+0000 7fcad57fa640 1 -- 172.21.15.16:0/575579729 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 6726+0+0 (secure 0 0 0) 0x7fcad0083070 con 0x7fcad815c130 2023-05-22T15:42:39.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.744+0000 7fcadc9c1640 1 --2- 172.21.15.16:0/575579729 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fcab8065530 0x7fcab80679e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-05-22T15:42:39.753 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.748+0000 7fcad57fa640 1 -- 172.21.15.16:0/575579729 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fcad004d620 con 0x7fcad815c130 2023-05-22T15:42:39.812 INFO:tasks.ceph.mgr.x.smithi062.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2023-05-22T15:42:39.812 INFO:tasks.ceph.mgr.x.smithi062.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2023-05-22T15:42:39.812 INFO:tasks.ceph.mgr.x.smithi062.stderr: from numpy import show_config as show_numpy_config 2023-05-22T15:42:39.819 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:42:39.815+0000 7ff8c72cb180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-05-22T15:42:39.858 INFO:tasks.ceph.mgr.y.smithi016.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2023-05-22T15:42:39.858 INFO:tasks.ceph.mgr.y.smithi016.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2023-05-22T15:42:39.858 INFO:tasks.ceph.mgr.y.smithi016.stderr: from numpy import show_config as show_numpy_config 2023-05-22T15:42:39.866 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:42:39.860+0000 7fa9cb8de180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-05-22T15:42:39.948 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:42:39.943+0000 7ff8c72cb180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-05-22T15:42:39.950 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.948+0000 7fcadc9c1640 1 -- 172.21.15.16:0/575579729 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fcab8065530 msgr2=0x7fcab80679e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/36921 2023-05-22T15:42:39.950 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.948+0000 7fcadc9c1640 1 --2- 172.21.15.16:0/575579729 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fcab8065530 0x7fcab80679e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-05-22T15:42:39.964 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.960+0000 7fcadec25640 1 -- 172.21.15.16:0/575579729 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fcad804dbe0 con 0x7fcad815c130 2023-05-22T15:42:39.965 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.960+0000 7fcad57fa640 1 -- 172.21.15.16:0/575579729 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 8 entries v91) v1 ==== 80+0+460 (secure 0 0 0) 0x7fcad001a1e0 con 0x7fcad815c130 2023-05-22T15:42:39.965 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/3510757085 2023-05-23T15:42:35.895737+0000 2023-05-22T15:42:39.965 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/4119485618 2023-05-23T15:42:35.895737+0000 2023-05-22T15:42:39.965 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/2050049957 2023-05-23T15:42:35.895737+0000 2023-05-22T15:42:39.966 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/1396723034 2023-05-23T15:42:35.895737+0000 2023-05-22T15:42:39.966 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/3754867799 2023-05-23T15:42:35.895737+0000 2023-05-22T15:42:39.966 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:6833/36921 2023-05-23T15:42:35.895737+0000 2023-05-22T15:42:39.966 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:6832/36921 2023-05-23T15:42:35.895737+0000 2023-05-22T15:42:39.966 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/3710518307 2023-05-23T15:42:35.895737+0000 2023-05-22T15:42:39.966 INFO:teuthology.orchestra.run.smithi016.stderr:listed 8 entries 2023-05-22T15:42:39.967 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.964+0000 7fcadec25640 1 -- 172.21.15.16:0/575579729 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fcab8065530 msgr2=0x7fcab80679e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-05-22T15:42:39.967 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.964+0000 7fcadec25640 1 --2- 172.21.15.16:0/575579729 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fcab8065530 0x7fcab80679e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:39.967 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.964+0000 7fcadec25640 1 -- 172.21.15.16:0/575579729 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcad815c130 msgr2=0x7fcad8150210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:39.967 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.964+0000 7fcadec25640 1 --2- 172.21.15.16:0/575579729 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcad815c130 0x7fcad8150210 secure :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 crypto rx=0x7fcad0007ef0 tx=0x7fcad000d4a0 comp rx=0 tx=0).stop 2023-05-22T15:42:39.968 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.964+0000 7fcadec25640 1 -- 172.21.15.16:0/575579729 shutdown_connections 2023-05-22T15:42:39.968 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.964+0000 7fcadec25640 1 --2- 172.21.15.16:0/575579729 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcad815c130 0x7fcad8150210 unknown :-1 s=CLOSED pgs=1182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:39.968 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.964+0000 7fcadec25640 1 --2- 172.21.15.16:0/575579729 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcad80cfa50 0x7fcad80c46e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:39.968 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.964+0000 7fcadec25640 1 --2- 172.21.15.16:0/575579729 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fcab8065530 0x7fcab80679e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:39.968 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.964+0000 7fcadec25640 1 --2- 172.21.15.16:0/575579729 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fcad80b2870 0x7fcad80c6670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:39.968 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.964+0000 7fcadec25640 1 -- 172.21.15.16:0/575579729 >> 172.21.15.16:0/575579729 conn(0x7fcad806f5c0 msgr2=0x7fcad80b17a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:39.968 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.964+0000 7fcadec25640 1 -- 172.21.15.16:0/575579729 shutdown_connections 2023-05-22T15:42:39.969 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:39.964+0000 7fcadec25640 1 -- 172.21.15.16:0/575579729 wait complete. 2023-05-22T15:42:39.983 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi016.front.sepia.ceph.com... 2023-05-22T15:42:39.983 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:42:39.984 DEBUG:teuthology.orchestra.run.smithi016:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2023-05-22T15:42:39.993 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:42:39.988+0000 7fa9cb8de180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-05-22T15:42:39.994 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi016.stderr:ceph-fuse[55199]: fuse finished with error 0 and tester_r 0 2023-05-22T15:42:39.995 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi016.front.sepia.ceph.com 2023-05-22T15:42:39.995 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:42:39.996 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-05-22T15:42:40.062 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:42:40.059+0000 7ff8c72cb180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-05-22T15:42:40.108 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:42:40.104+0000 7fa9cb8de180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-05-22T15:42:40.126 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:42:40.123+0000 7ff8c72cb180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-05-22T15:42:40.153 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2023-05-22T15:42:40.154 DEBUG:teuthology.orchestra.run.smithi062:> dd if=/proc/self/mounts of=/dev/stdout 2023-05-22T15:42:40.159 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16342116k,nr_inodes=4085529,mode=755,inode64 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275932k,mode=755,inode64 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev,inode64 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=13250 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275928k,nr_inodes=818982,mode=700,uid=1000,gid=1271,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275932k,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-05-22T15:42:40.160 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi062.front.sepia.ceph.com 2023-05-22T15:42:40.160 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:42:40.160 DEBUG:teuthology.orchestra.run.smithi062:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2023-05-22T15:42:40.176 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:42:40.172+0000 7fa9cb8de180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-05-22T15:42:40.205 INFO:teuthology.orchestra.run.smithi062.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2023-05-22T15:42:40.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-05-22T15:42:40.206 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-05-22T15:42:40.207 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2023-05-22T15:42:40.212 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16345192k,nr_inodes=4086298,mode=755,inode64 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3276544k,mode=755,inode64 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev,inode64 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=15732 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3276540k,nr_inodes=819135,mode=700,uid=1000,gid=1271,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3276544k,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-05-22T15:42:40.213 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi016.front.sepia.ceph.com 2023-05-22T15:42:40.213 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:42:40.213 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2023-05-22T15:42:40.253 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:42:40.251+0000 7ff8c72cb180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-05-22T15:42:40.258 INFO:teuthology.orchestra.run.smithi016.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2023-05-22T15:42:40.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-05-22T15:42:40.260 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2023-05-22T15:42:40.260 DEBUG:teuthology.orchestra.run.smithi062:> dd if=/proc/self/mounts of=/dev/stdout 2023-05-22T15:42:40.265 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16342116k,nr_inodes=4085529,mode=755,inode64 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275932k,mode=755,inode64 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev,inode64 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=13250 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275928k,nr_inodes=818982,mode=700,uid=1000,gid=1271,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275932k,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-05-22T15:42:40.266 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi062.front.sepia.ceph.com 2023-05-22T15:42:40.266 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:42:40.266 DEBUG:teuthology.orchestra.run.smithi062:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2023-05-22T15:42:40.302 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:42:40.296+0000 7fa9cb8de180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-05-22T15:42:40.313 INFO:teuthology.orchestra.run.smithi062.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2023-05-22T15:42:40.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-05-22T15:42:40.314 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-05-22T15:42:40.426 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:40.424+0000 7f9466050640 1 Processor -- start 2023-05-22T15:42:40.427 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:40.424+0000 7f9466050640 1 -- start start 2023-05-22T15:42:40.427 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:40.424+0000 7f9466050640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f9460153fe0 0x7f946006d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:40.428 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:40.424+0000 7f9466050640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f94600cae20 0x7f946006fef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:40.428 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:40.424+0000 7f9466050640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9460069720 con 0x7f9460070430 2023-05-22T15:42:40.428 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:40.424+0000 7f9466050640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94600708b0 con 0x7f94600cae20 2023-05-22T15:42:40.428 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:40.424+0000 7f9466050640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94601543b0 con 0x7f9460153fe0 2023-05-22T15:42:40.429 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:40.424+0000 7f9456ffd640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f94600cae20 0x7f946006fef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:40.429 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:40.424+0000 7f9456ffd640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f94600cae20 0x7f946006fef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49008/0 (socket says 172.21.15.16:49008) 2023-05-22T15:42:40.429 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:40.424+0000 7f9456ffd640 1 -- 172.21.15.16:0/542546098 learned_addr learned my addr 172.21.15.16:0/542546098 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:42:40.429 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:40.424+0000 7f945f7fe640 1 --2- 172.21.15.16:0/542546098 >> v2:172.21.15.62:3300/0 conn(0x7f9460153fe0 0x7f946006d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:40.430 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:40.424+0000 7f9456ffd640 1 -- 172.21.15.16:0/542546098 >> v2:172.21.15.62:3300/0 conn(0x7f9460153fe0 msgr2=0x7f946006d3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:42:40.430 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:40.424+0000 7f9456ffd640 1 --2- 172.21.15.16:0/542546098 >> v2:172.21.15.62:3300/0 conn(0x7f9460153fe0 0x7f946006d3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:40.430 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:40.424+0000 7f9456ffd640 1 -- 172.21.15.16:0/542546098 >> v1:172.21.15.16:6789/0 conn(0x7f9460070430 legacy=0x7f94600ad410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:40.430 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:40.424+0000 7f9456ffd640 1 -- 172.21.15.16:0/542546098 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f946015e360 con 0x7f94600cae20 2023-05-22T15:42:40.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:40.428+0000 7f9456ffd640 1 --2- 172.21.15.16:0/542546098 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f94600cae20 0x7f946006fef0 secure :-1 s=READY pgs=1184 cs=0 l=1 rev1=1 crypto rx=0x7f946006fcf0 tx=0x7f944801c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5787acf7baf6bff server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:40.433 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:40.428+0000 7f945effd640 1 -- 172.21.15.16:0/542546098 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94480aa430 con 0x7f94600cae20 2023-05-22T15:42:40.433 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:40.428+0000 7f945effd640 1 -- 172.21.15.16:0/542546098 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94480aa590 con 0x7f94600cae20 2023-05-22T15:42:40.434 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:40.428+0000 7f945effd640 1 -- 172.21.15.16:0/542546098 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94480aa920 con 0x7f94600cae20 2023-05-22T15:42:40.434 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:40.432+0000 7f9466050640 1 -- 172.21.15.16:0/542546098 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f94600cae20 msgr2=0x7f946006fef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:40.434 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:40.432+0000 7f9466050640 1 --2- 172.21.15.16:0/542546098 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f94600cae20 0x7f946006fef0 secure :-1 s=READY pgs=1184 cs=0 l=1 rev1=1 crypto rx=0x7f946006fcf0 tx=0x7f944801c4d0 comp rx=0 tx=0).stop 2023-05-22T15:42:40.435 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:40.432+0000 7f9466050640 1 -- 172.21.15.16:0/542546098 shutdown_connections 2023-05-22T15:42:40.435 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:40.432+0000 7f9466050640 1 --2- 172.21.15.16:0/542546098 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f94600cae20 0x7f946006fef0 unknown :-1 s=CLOSED pgs=1184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:40.435 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:40.432+0000 7f9466050640 1 --2- 172.21.15.16:0/542546098 >> v2:172.21.15.62:3300/0 conn(0x7f9460153fe0 0x7f946006d3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:40.435 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:40.432+0000 7f9466050640 1 -- 172.21.15.16:0/542546098 >> 172.21.15.16:0/542546098 conn(0x7f94600b1110 msgr2=0x7f94600ce830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:40.435 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:40.432+0000 7f9466050640 1 -- 172.21.15.16:0/542546098 shutdown_connections 2023-05-22T15:42:40.435 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:40.432+0000 7f9466050640 1 -- 172.21.15.16:0/542546098 wait complete. 2023-05-22T15:42:40.437 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:40.432+0000 7f9466050640 1 Processor -- start 2023-05-22T15:42:40.437 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:40.432+0000 7f9466050640 1 -- start start 2023-05-22T15:42:40.437 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:40.432+0000 7f9466050640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9460070430 0x7f9460159530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:40.437 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:40.432+0000 7f9466050640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f94600cae20 0x7f946013f350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:40.437 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:40.432+0000 7f9466050640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9460153fe0 0x7f9460150e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:40.437 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:40.432+0000 7f9466050640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9460157e80 con 0x7f9460070430 2023-05-22T15:42:40.437 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:40.432+0000 7f9466050640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9460158160 con 0x7f9460153fe0 2023-05-22T15:42:40.438 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:40.432+0000 7f9466050640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9460158440 con 0x7f94600cae20 2023-05-22T15:42:40.438 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:40.432+0000 7f945f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9460070430 0x7f9460159530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:40.438 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:40.432+0000 7f945f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9460070430 0x7f9460159530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49020/0 (socket says 172.21.15.16:49020) 2023-05-22T15:42:40.438 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:40.432+0000 7f945f7fe640 1 -- 172.21.15.16:0/733302755 learned_addr learned my addr 172.21.15.16:0/733302755 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:42:40.438 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:40.432+0000 7f9456ffd640 1 --2- 172.21.15.16:0/733302755 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f94600cae20 0x7f946013f350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:40.438 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:40.432+0000 7f945ffff640 1 --2- 172.21.15.16:0/733302755 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9460153fe0 0x7f9460150e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:40.438 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:40.432+0000 7f945f7fe640 1 -- 172.21.15.16:0/733302755 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f94600cae20 msgr2=0x7f946013f350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:40.438 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:40.432+0000 7f945f7fe640 1 --2- 172.21.15.16:0/733302755 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f94600cae20 0x7f946013f350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:40.439 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:40.432+0000 7f945f7fe640 1 -- 172.21.15.16:0/733302755 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9460153fe0 msgr2=0x7f9460150e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:40.439 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:40.432+0000 7f945f7fe640 1 --2- 172.21.15.16:0/733302755 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9460153fe0 0x7f9460150e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:40.439 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:40.432+0000 7f945f7fe640 1 -- 172.21.15.16:0/733302755 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94480094d0 con 0x7f9460070430 2023-05-22T15:42:40.439 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:40.432+0000 7f945f7fe640 1 --2- 172.21.15.16:0/733302755 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9460070430 0x7f9460159530 secure :-1 s=READY pgs=1185 cs=0 l=1 rev1=1 crypto rx=0x7f945000d5f0 tx=0x7f945000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:40.440 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:40.432+0000 7f945cff9640 1 -- 172.21.15.16:0/733302755 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94500125f0 con 0x7f9460070430 2023-05-22T15:42:40.441 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:40.432+0000 7f945cff9640 1 -- 172.21.15.16:0/733302755 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9450012750 con 0x7f9460070430 2023-05-22T15:42:40.441 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:40.432+0000 7f9466050640 1 -- 172.21.15.16:0/733302755 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f946015de40 con 0x7f9460070430 2023-05-22T15:42:40.441 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:40.432+0000 7f945cff9640 1 -- 172.21.15.16:0/733302755 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9450012ae0 con 0x7f9460070430 2023-05-22T15:42:40.441 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:40.432+0000 7f9466050640 1 -- 172.21.15.16:0/733302755 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f946015db60 con 0x7f9460070430 2023-05-22T15:42:40.441 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:40.436+0000 7f9466050640 1 -- 172.21.15.16:0/733302755 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f946006dd20 con 0x7f9460070430 2023-05-22T15:42:40.441 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:40.436+0000 7f945cff9640 1 -- 172.21.15.16:0/733302755 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 31) v1 ==== 84389+0+0 (secure 0 0 0) 0x7f9450012c40 con 0x7f9460070430 2023-05-22T15:42:40.441 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:40.436+0000 7f945cff9640 1 --2- 172.21.15.16:0/733302755 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f9440065530 0x7f94400679e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:40.442 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:40.436+0000 7f9456ffd640 1 -- 172.21.15.16:0/733302755 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f9440065530 msgr2=0x7f94400679e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/36921 2023-05-22T15:42:40.442 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:40.436+0000 7f945cff9640 1 -- 172.21.15.16:0/733302755 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 6726+0+0 (secure 0 0 0) 0x7f94500832e0 con 0x7f9460070430 2023-05-22T15:42:40.442 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:40.436+0000 7f9456ffd640 1 --2- 172.21.15.16:0/733302755 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f9440065530 0x7f94400679e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-05-22T15:42:40.443 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:40.440+0000 7f945cff9640 1 -- 172.21.15.16:0/733302755 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f9450050a70 con 0x7f9460070430 2023-05-22T15:42:40.623 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:40.620+0000 7f9466050640 1 -- 172.21.15.16:0/733302755 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f946004dbe0 con 0x7f9460070430 2023-05-22T15:42:40.624 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:40.620+0000 7f945cff9640 1 -- 172.21.15.16:0/733302755 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+4647 (secure 0 0 0) 0x7f9450054090 con 0x7f9460070430 2023-05-22T15:42:40.625 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:42:40.625 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":30,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6833/428835470","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":428835470},{"type":"v1","addr":"172.21.15.62:6833","nonce":428835470}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":25},{"gid":6238,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/4010598958","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":4010598958},{"type":"v1","addr":"172.21.15.62:6835","nonce":4010598958}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":25},{"gid":7008,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/2240458030","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6835","nonce":2240458030},{"type":"v1","addr":"172.21.15.16:6837","nonce":2240458030}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":25}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-05-22T15:42:22.811659+0000","modified":"2023-05-22T15:42:25.830716+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7011},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7011":{"gid":7011,"name":"a","rank":0,"incarnation":26,"state":"up:active","state_seq":3,"addr":"172.21.15.16:6836/1772031282","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1772031282},{"type":"v1","addr":"172.21.15.16:6836","nonce":1772031282}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2023-05-22T15:42:40.625 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 30 2023-05-22T15:42:40.626 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:40.624+0000 7f9466050640 1 -- 172.21.15.16:0/733302755 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f9440065530 msgr2=0x7f94400679e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-05-22T15:42:40.626 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:40.624+0000 7f9466050640 1 --2- 172.21.15.16:0/733302755 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f9440065530 0x7f94400679e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:40.627 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:40.624+0000 7f9466050640 1 -- 172.21.15.16:0/733302755 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9460070430 msgr2=0x7f9460159530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:40.627 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:40.624+0000 7f9466050640 1 --2- 172.21.15.16:0/733302755 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9460070430 0x7f9460159530 secure :-1 s=READY pgs=1185 cs=0 l=1 rev1=1 crypto rx=0x7f945000d5f0 tx=0x7f945000dab0 comp rx=0 tx=0).stop 2023-05-22T15:42:40.627 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:40.624+0000 7f9466050640 1 -- 172.21.15.16:0/733302755 shutdown_connections 2023-05-22T15:42:40.627 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:40.624+0000 7f9466050640 1 --2- 172.21.15.16:0/733302755 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9460153fe0 0x7f9460150e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:40.627 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:40.624+0000 7f9466050640 1 --2- 172.21.15.16:0/733302755 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f9440065530 0x7f94400679e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:40.627 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:40.624+0000 7f9466050640 1 --2- 172.21.15.16:0/733302755 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f94600cae20 0x7f946013f350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:40.627 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:40.624+0000 7f9466050640 1 --2- 172.21.15.16:0/733302755 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9460070430 0x7f9460159530 unknown :-1 s=CLOSED pgs=1185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:40.628 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:40.624+0000 7f9466050640 1 -- 172.21.15.16:0/733302755 >> 172.21.15.16:0/733302755 conn(0x7f94600b1110 msgr2=0x7f946015b2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:40.628 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:40.624+0000 7f9466050640 1 -- 172.21.15.16:0/733302755 shutdown_connections 2023-05-22T15:42:40.628 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:40.624+0000 7f9466050640 1 -- 172.21.15.16:0/733302755 wait complete. 2023-05-22T15:42:40.642 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-05-22T15:42:40.752 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:42:40.747+0000 7ff8c72cb180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-05-22T15:42:40.762 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:40.760+0000 7fe18d13d640 1 Processor -- start 2023-05-22T15:42:40.762 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:40.760+0000 7fe18d13d640 1 -- start start 2023-05-22T15:42:40.763 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:40.760+0000 7fe18d13d640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe1881563c0 0x7fe1880c7b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:40.764 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:40.760+0000 7fe18d13d640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe18815c210 con 0x7fe1880cc6c0 2023-05-22T15:42:40.764 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:40.760+0000 7fe18d13d640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe18815c480 con 0x7fe1880c8080 2023-05-22T15:42:40.764 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:40.760+0000 7fe18d13d640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1880c8520 con 0x7fe1881563c0 2023-05-22T15:42:40.764 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:40.760+0000 7fe18659c640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fe1880cc6c0 0x7fe18806c220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:41052/0 (socket says 172.21.15.16:41052) 2023-05-22T15:42:40.764 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:40.760+0000 7fe18659c640 1 -- 172.21.15.16:0/2045518530 learned_addr learned my addr 172.21.15.16:0/2045518530 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:42:40.766 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:40.760+0000 7fe186d9d640 1 --2- 172.21.15.16:0/2045518530 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe1881563c0 0x7fe1880c7b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:40.767 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:40.760+0000 7fe185d9b640 1 -- 172.21.15.16:0/2045518530 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3078857158 0 0) 0x7fe18815c210 con 0x7fe1880cc6c0 2023-05-22T15:42:40.767 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:40.760+0000 7fe185d9b640 1 -- 172.21.15.16:0/2045518530 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe170003530 con 0x7fe1880cc6c0 2023-05-22T15:42:40.767 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:40.760+0000 7fe185d9b640 1 -- 172.21.15.16:0/2045518530 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2627886963 0 0) 0x7fe170003530 con 0x7fe1880cc6c0 2023-05-22T15:42:40.767 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:40.760+0000 7fe185d9b640 1 -- 172.21.15.16:0/2045518530 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe1881563c0 msgr2=0x7fe1880c7b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:42:40.767 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:40.760+0000 7fe185d9b640 1 --2- 172.21.15.16:0/2045518530 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe1881563c0 0x7fe1880c7b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:40.767 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:40.760+0000 7fe185d9b640 1 -- 172.21.15.16:0/2045518530 >> v1:172.21.15.62:6789/0 conn(0x7fe1880c8080 legacy=0x7fe18815f6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:40.768 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:40.760+0000 7fe185d9b640 1 -- 172.21.15.16:0/2045518530 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe18815c760 con 0x7fe1880cc6c0 2023-05-22T15:42:40.768 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:40.760+0000 7fe185d9b640 1 -- 172.21.15.16:0/2045518530 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7fe17c001680 con 0x7fe1880cc6c0 2023-05-22T15:42:40.768 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:40.760+0000 7fe185d9b640 1 -- 172.21.15.16:0/2045518530 >> v1:172.21.15.16:6789/0 conn(0x7fe1880cc6c0 legacy=0x7fe18806c220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:40.768 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:40.760+0000 7fe185d9b640 1 --2- 172.21.15.16:0/2045518530 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe1700039b0 0x7fe170003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:40.768 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:40.760+0000 7fe185d9b640 1 --2- 172.21.15.16:0/2045518530 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe1700047d0 0x7fe170004c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:40.768 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:40.760+0000 7fe185d9b640 1 --2- 172.21.15.16:0/2045518530 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe170005140 0x7fe17000b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:40.768 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:40.760+0000 7fe185d9b640 1 -- 172.21.15.16:0/2045518530 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe17000bed0 con 0x7fe1700039b0 2023-05-22T15:42:40.769 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:40.760+0000 7fe185d9b640 1 -- 172.21.15.16:0/2045518530 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe17000c010 con 0x7fe170005140 2023-05-22T15:42:40.769 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:40.760+0000 7fe185d9b640 1 -- 172.21.15.16:0/2045518530 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe17000c150 con 0x7fe1700047d0 2023-05-22T15:42:40.769 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:40.760+0000 7fe186d9d640 1 --2- 172.21.15.16:0/2045518530 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe1700039b0 0x7fe170003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:40.769 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:40.760+0000 7fe18759e640 1 --2- 172.21.15.16:0/2045518530 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe170005140 0x7fe17000b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:40.769 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:40.760+0000 7fe18659c640 1 --2- 172.21.15.16:0/2045518530 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe1700047d0 0x7fe170004c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:40.769 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:40.760+0000 7fe186d9d640 1 -- 172.21.15.16:0/2045518530 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe1700047d0 msgr2=0x7fe170004c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:42:40.769 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:40.760+0000 7fe186d9d640 1 --2- 172.21.15.16:0/2045518530 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe1700047d0 0x7fe170004c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:40.770 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:40.760+0000 7fe186d9d640 1 -- 172.21.15.16:0/2045518530 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe170005140 msgr2=0x7fe17000b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:42:40.770 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:40.760+0000 7fe186d9d640 1 --2- 172.21.15.16:0/2045518530 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe170005140 0x7fe17000b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:40.770 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:40.760+0000 7fe186d9d640 1 -- 172.21.15.16:0/2045518530 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe170003530 con 0x7fe1700039b0 2023-05-22T15:42:40.770 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:40.760+0000 7fe18659c640 1 --2- 172.21.15.16:0/2045518530 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe1700047d0 0x7fe170004c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:42:40.772 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:40.764+0000 7fe186d9d640 1 --2- 172.21.15.16:0/2045518530 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe1700039b0 0x7fe170003dc0 secure :-1 s=READY pgs=1187 cs=0 l=1 rev1=1 crypto rx=0x7fe178009b10 tx=0x7fe17801ca20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f1e46febd15c6d server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:40.772 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:40.764+0000 7fe185d9b640 1 -- 172.21.15.16:0/2045518530 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1780a6510 con 0x7fe1700039b0 2023-05-22T15:42:40.772 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:40.764+0000 7fe185d9b640 1 -- 172.21.15.16:0/2045518530 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1780a6670 con 0x7fe1700039b0 2023-05-22T15:42:40.773 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:40.764+0000 7fe185d9b640 1 -- 172.21.15.16:0/2045518530 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1780a6a20 con 0x7fe1700039b0 2023-05-22T15:42:40.773 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:40.764+0000 7fe18d13d640 1 -- 172.21.15.16:0/2045518530 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe1700039b0 msgr2=0x7fe170003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:40.773 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:40.764+0000 7fe18d13d640 1 --2- 172.21.15.16:0/2045518530 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe1700039b0 0x7fe170003dc0 secure :-1 s=READY pgs=1187 cs=0 l=1 rev1=1 crypto rx=0x7fe178009b10 tx=0x7fe17801ca20 comp rx=0 tx=0).stop 2023-05-22T15:42:40.773 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:40.764+0000 7fe18759e640 1 -- 172.21.15.16:0/2045518530 reap_dead start 2023-05-22T15:42:40.773 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:40.764+0000 7fe18d13d640 1 -- 172.21.15.16:0/2045518530 shutdown_connections 2023-05-22T15:42:40.773 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:40.764+0000 7fe18d13d640 1 -- 172.21.15.16:0/2045518530 >> 172.21.15.16:0/2045518530 conn(0x7fe1880b1110 msgr2=0x7fe1880ce830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:40.773 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:40.764+0000 7fe18d13d640 1 -- 172.21.15.16:0/2045518530 shutdown_connections 2023-05-22T15:42:40.773 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:40.764+0000 7fe18d13d640 1 -- 172.21.15.16:0/2045518530 wait complete. 2023-05-22T15:42:40.774 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:40.764+0000 7fe18d13d640 1 Processor -- start 2023-05-22T15:42:40.774 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:40.764+0000 7fe18d13d640 1 -- start start 2023-05-22T15:42:40.774 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:40.768+0000 7fe18d13d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe18815ea20 0x7fe188154600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:40.774 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:40.768+0000 7fe18d13d640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe18814b6f0 0x7fe18813cad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:40.774 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:40.768+0000 7fe18d13d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe18815edf0 0x7fe188139fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:40.774 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:40.768+0000 7fe18d13d640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe188153150 con 0x7fe18815ea20 2023-05-22T15:42:40.774 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:40.768+0000 7fe18d13d640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe188154b40 con 0x7fe18815edf0 2023-05-22T15:42:40.774 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:40.768+0000 7fe18d13d640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe18813c050 con 0x7fe18814b6f0 2023-05-22T15:42:40.775 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:40.768+0000 7fe186d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe18815ea20 0x7fe188154600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:40.775 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:40.768+0000 7fe186d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe18815ea20 0x7fe188154600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49028/0 (socket says 172.21.15.16:49028) 2023-05-22T15:42:40.775 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:40.768+0000 7fe186d9d640 1 -- 172.21.15.16:0/726034656 learned_addr learned my addr 172.21.15.16:0/726034656 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:42:40.775 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:40.768+0000 7fe18659c640 1 --2- 172.21.15.16:0/726034656 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe18814b6f0 0x7fe18813cad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:40.775 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:40.768+0000 7fe18759e640 1 --2- 172.21.15.16:0/726034656 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe18815edf0 0x7fe188139fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:40.775 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:40.768+0000 7fe186d9d640 1 -- 172.21.15.16:0/726034656 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe18814b6f0 msgr2=0x7fe18813cad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:40.775 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:40.768+0000 7fe186d9d640 1 --2- 172.21.15.16:0/726034656 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe18814b6f0 0x7fe18813cad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:40.776 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:40.768+0000 7fe186d9d640 1 -- 172.21.15.16:0/726034656 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe18815edf0 msgr2=0x7fe188139fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:40.776 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:40.768+0000 7fe186d9d640 1 --2- 172.21.15.16:0/726034656 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe18815edf0 0x7fe188139fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:40.776 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:40.768+0000 7fe186d9d640 1 -- 172.21.15.16:0/726034656 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe18806cd40 con 0x7fe18815ea20 2023-05-22T15:42:40.776 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:40.768+0000 7fe186d9d640 1 --2- 172.21.15.16:0/726034656 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe18815ea20 0x7fe188154600 secure :-1 s=READY pgs=1188 cs=0 l=1 rev1=1 crypto rx=0x7fe17801c850 tx=0x7fe170005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:40.776 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:40.768+0000 7fe16f7fe640 1 -- 172.21.15.16:0/726034656 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1780a69e0 con 0x7fe18815ea20 2023-05-22T15:42:40.776 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:40.768+0000 7fe16f7fe640 1 -- 172.21.15.16:0/726034656 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1780a6b40 con 0x7fe18815ea20 2023-05-22T15:42:40.776 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:40.768+0000 7fe18d13d640 1 -- 172.21.15.16:0/726034656 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe188160610 con 0x7fe18815ea20 2023-05-22T15:42:40.776 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:40.768+0000 7fe16f7fe640 1 -- 172.21.15.16:0/726034656 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe17800f0c0 con 0x7fe18815ea20 2023-05-22T15:42:40.777 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:40.768+0000 7fe18d13d640 1 -- 172.21.15.16:0/726034656 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe188167a90 con 0x7fe18815ea20 2023-05-22T15:42:40.777 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:40.768+0000 7fe18d13d640 1 -- 172.21.15.16:0/726034656 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe18804dbe0 con 0x7fe18815ea20 2023-05-22T15:42:40.777 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:40.768+0000 7fe16f7fe640 1 -- 172.21.15.16:0/726034656 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 31) v1 ==== 84389+0+0 (secure 0 0 0) 0x7fe178006040 con 0x7fe18815ea20 2023-05-22T15:42:40.778 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:40.768+0000 7fe16f7fe640 1 --2- 172.21.15.16:0/726034656 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fe174065530 0x7fe1740679e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:40.778 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:40.768+0000 7fe18659c640 1 -- 172.21.15.16:0/726034656 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fe174065530 msgr2=0x7fe1740679e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/36921 2023-05-22T15:42:40.778 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:40.768+0000 7fe16f7fe640 1 -- 172.21.15.16:0/726034656 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 6726+0+0 (secure 0 0 0) 0x7fe1780aa070 con 0x7fe18815ea20 2023-05-22T15:42:40.778 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:40.768+0000 7fe18659c640 1 --2- 172.21.15.16:0/726034656 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fe174065530 0x7fe1740679e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-05-22T15:42:40.778 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:40.772+0000 7fe16f7fe640 1 -- 172.21.15.16:0/726034656 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fe1780eb970 con 0x7fe18815ea20 2023-05-22T15:42:40.812 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:42:40.808+0000 7fa9cb8de180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-05-22T15:42:40.829 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:42:40.827+0000 7ff8c72cb180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-05-22T15:42:40.889 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:42:40.884+0000 7fa9cb8de180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-05-22T15:42:40.927 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:40.924+0000 7fe18d13d640 1 -- 172.21.15.16:0/726034656 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe1880cd550 con 0x7fe18815ea20 2023-05-22T15:42:40.930 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:40.924+0000 7fe16f7fe640 1 -- 172.21.15.16:0/726034656 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) v1 ==== 74+0+18530 (secure 0 0 0) 0x7fe1780eef90 con 0x7fe18815ea20 2023-05-22T15:42:40.930 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:42:40.930 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":91,"fsid":"b2719834-7847-4d98-90a6-f65d57f85c49","created":"2023-05-22T15:26:25.259314+0000","modified":"2023-05-22T15:42:35.895812+0000","last_up_change":"2023-05-22T15:26:36.448524+0000","last_in_change":"2023-05-22T15:26:27.675668+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-05-22T15:26:38.019781+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-05-22T15:26:43.570514+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2023-05-22T15:42:20.730757+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2023-05-22T15:42:21.805016+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9e7ff854-b25f-4cbb-a5c2-1799d9c53d2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":37021},{"type":"v1","addr":"172.21.15.16:6825","nonce":37021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":37021},{"type":"v1","addr":"172.21.15.16:6827","nonce":37021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":37021},{"type":"v1","addr":"172.21.15.16:6831","nonce":37021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":37021},{"type":"v1","addr":"172.21.15.16:6829","nonce":37021}]},"public_addr":"172.21.15.16:6825/37021","cluster_addr":"172.21.15.16:6827/37021","heartbeat_back_addr":"172.21.15.16:6831/37021","heartbeat_front_addr":"172.21.15.16:6829/37021","state":["exists","up"]},{"osd":1,"uuid":"4702fbc9-a927-4565-aca5-41dc69cc8264","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":37008},{"type":"v1","addr":"172.21.15.16:6801","nonce":37008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":37008},{"type":"v1","addr":"172.21.15.16:6803","nonce":37008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":37008},{"type":"v1","addr":"172.21.15.16:6807","nonce":37008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":37008},{"type":"v1","addr":"172.21.15.16:6805","nonce":37008}]},"public_addr":"172.21.15.16:6801/37008","cluster_addr":"172.21.15.16:6803/37008","heartbeat_back_addr":"172.21.15.16:6807/37008","heartbeat_front_addr":"172.21.15.16:6805/37008","state":["exists","up"]},{"osd":2,"uuid":"1e7bba46-4eba-47ef-bb81-5e2d371638ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":37009},{"type":"v1","addr":"172.21.15.16:6809","nonce":37009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":37009},{"type":"v1","addr":"172.21.15.16:6811","nonce":37009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":37009},{"type":"v1","addr":"172.21.15.16:6815","nonce":37009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":37009},{"type":"v1","addr":"172.21.15.16:6813","nonce":37009}]},"public_addr":"172.21.15.16:6809/37009","cluster_addr":"172.21.15.16:6811/37009","heartbeat_back_addr":"172.21.15.16:6815/37009","heartbeat_front_addr":"172.21.15.16:6813/37009","state":["exists","up"]},{"osd":3,"uuid":"383cacc0-5a88-4904-990e-498ca59ef48a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":37020},{"type":"v1","addr":"172.21.15.16:6817","nonce":37020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":37020},{"type":"v1","addr":"172.21.15.16:6819","nonce":37020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":37020},{"type":"v1","addr":"172.21.15.16:6823","nonce":37020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":37020},{"type":"v1","addr":"172.21.15.16:6821","nonce":37020}]},"public_addr":"172.21.15.16:6817/37020","cluster_addr":"172.21.15.16:6819/37020","heartbeat_back_addr":"172.21.15.16:6823/37020","heartbeat_front_addr":"172.21.15.16:6821/37020","state":["exists","up"]},{"osd":4,"uuid":"970500ef-95b7-42c1-b080-eed4fae7013d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":37353},{"type":"v1","addr":"172.21.15.62:6817","nonce":37353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6819","nonce":37353},{"type":"v1","addr":"172.21.15.62:6822","nonce":37353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37353},{"type":"v1","addr":"172.21.15.62:6831","nonce":37353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37353},{"type":"v1","addr":"172.21.15.62:6826","nonce":37353}]},"public_addr":"172.21.15.62:6817/37353","cluster_addr":"172.21.15.62:6822/37353","heartbeat_back_addr":"172.21.15.62:6831/37353","heartbeat_front_addr":"172.21.15.62:6826/37353","state":["exists","up"]},{"osd":5,"uuid":"022d67db-d4d8-445b-b1b9-8b446a00f5c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":37352},{"type":"v1","addr":"172.21.15.62:6801","nonce":37352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":37352},{"type":"v1","addr":"172.21.15.62:6803","nonce":37352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":37352},{"type":"v1","addr":"172.21.15.62:6807","nonce":37352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":37352},{"type":"v1","addr":"172.21.15.62:6805","nonce":37352}]},"public_addr":"172.21.15.62:6801/37352","cluster_addr":"172.21.15.62:6803/37352","heartbeat_back_addr":"172.21.15.62:6807/37352","heartbeat_front_addr":"172.21.15.62:6805/37352","state":["exists","up"]},{"osd":6,"uuid":"42bd372b-6688-4a1b-8407-e16e8c266e33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":37354},{"type":"v1","addr":"172.21.15.62:6820","nonce":37354}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":37354},{"type":"v1","addr":"172.21.15.62:6823","nonce":37354}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6829","nonce":37354},{"type":"v1","addr":"172.21.15.62:6830","nonce":37354}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6825","nonce":37354},{"type":"v1","addr":"172.21.15.62:6827","nonce":37354}]},"public_addr":"172.21.15.62:6820/37354","cluster_addr":"172.21.15.62:6823/37354","heartbeat_back_addr":"172.21.15.62:6830/37354","heartbeat_front_addr":"172.21.15.62:6827/37354","state":["exists","up"]},{"osd":7,"uuid":"ac70b2a3-f41b-4bf2-b227-7f2cbb55426d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37355},{"type":"v1","addr":"172.21.15.62:6809","nonce":37355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37355},{"type":"v1","addr":"172.21.15.62:6811","nonce":37355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":37355},{"type":"v1","addr":"172.21.15.62:6815","nonce":37355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":37355},{"type":"v1","addr":"172.21.15.62:6813","nonce":37355}]},"public_addr":"172.21.15.62:6809/37355","cluster_addr":"172.21.15.62:6811/37355","heartbeat_back_addr":"172.21.15.62:6815/37355","heartbeat_front_addr":"172.21.15.62:6813/37355","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.368896+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.107273+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.276888+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.287502+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.124791+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.251937+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.255679+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.232797+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:0/3510757085":"2023-05-23T15:42:35.895737+0000","172.21.15.16:0/4119485618":"2023-05-23T15:42:35.895737+0000","172.21.15.16:0/2050049957":"2023-05-23T15:42:35.895737+0000","172.21.15.16:0/1396723034":"2023-05-23T15:42:35.895737+0000","172.21.15.16:0/3754867799":"2023-05-23T15:42:35.895737+0000","172.21.15.16:6833/36921":"2023-05-23T15:42:35.895737+0000","172.21.15.16:6832/36921":"2023-05-23T15:42:35.895737+0000","172.21.15.16:0/3710518307":"2023-05-23T15:42:35.895737+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-05-22T15:42:40.932 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:40.928+0000 7fe18d13d640 1 -- 172.21.15.16:0/726034656 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fe174065530 msgr2=0x7fe1740679e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-05-22T15:42:40.932 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:40.928+0000 7fe18d13d640 1 --2- 172.21.15.16:0/726034656 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fe174065530 0x7fe1740679e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:40.932 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:40.928+0000 7fe18d13d640 1 -- 172.21.15.16:0/726034656 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe18815ea20 msgr2=0x7fe188154600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:40.933 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:40.928+0000 7fe18d13d640 1 --2- 172.21.15.16:0/726034656 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe18815ea20 0x7fe188154600 secure :-1 s=READY pgs=1188 cs=0 l=1 rev1=1 crypto rx=0x7fe17801c850 tx=0x7fe170005800 comp rx=0 tx=0).stop 2023-05-22T15:42:40.933 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:40.928+0000 7fe18d13d640 1 -- 172.21.15.16:0/726034656 shutdown_connections 2023-05-22T15:42:40.933 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:40.928+0000 7fe18d13d640 1 --2- 172.21.15.16:0/726034656 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe18815edf0 0x7fe188139fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:40.933 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:40.928+0000 7fe18d13d640 1 --2- 172.21.15.16:0/726034656 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fe174065530 0x7fe1740679e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:40.933 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:40.928+0000 7fe18d13d640 1 --2- 172.21.15.16:0/726034656 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe18814b6f0 0x7fe18813cad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:40.933 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:40.928+0000 7fe18d13d640 1 --2- 172.21.15.16:0/726034656 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe18815ea20 0x7fe188154600 unknown :-1 s=CLOSED pgs=1188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:40.933 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:40.928+0000 7fe18d13d640 1 -- 172.21.15.16:0/726034656 >> 172.21.15.16:0/726034656 conn(0x7fe1880b1110 msgr2=0x7fe1880ce830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:40.934 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:40.928+0000 7fe18d13d640 1 -- 172.21.15.16:0/726034656 shutdown_connections 2023-05-22T15:42:40.934 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:40.928+0000 7fe18d13d640 1 -- 172.21.15.16:0/726034656 wait complete. 2023-05-22T15:42:40.948 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-05-22T15:26:38.019781+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-05-22T15:26:43.570514+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 10, 'pool_name': 'cephfs_metadata', 'create_time': '2023-05-22T15:42:20.730757+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 11, 'pool_name': 'cephfs_data', 'create_time': '2023-05-22T15:42:21.805016+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '90', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-05-22T15:42:40.949 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-05-22T15:42:40.967 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:42:40.963+0000 7ff8c72cb180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-05-22T15:42:41.034 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:42:41.032+0000 7fa9cb8de180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-05-22T15:42:41.045 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:42:41.043+0000 7ff8c72cb180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-05-22T15:42:41.066 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.060+0000 7f4b205be640 1 Processor -- start 2023-05-22T15:42:41.066 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.064+0000 7f4b205be640 1 -- start start 2023-05-22T15:42:41.067 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.064+0000 7f4b205be640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f4b181563c0 0x7f4b180c7b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:41.068 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.064+0000 7f4b205be640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f4b180cc6c0 0x7f4b180c8080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:41.068 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.064+0000 7f4b205be640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4b1806c220 0x7f4b1806c6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:41.068 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.064+0000 7f4b205be640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b1815c210 con 0x7f4b180cc6c0 2023-05-22T15:42:41.068 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.064+0000 7f4b205be640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b1815c480 con 0x7f4b181563c0 2023-05-22T15:42:41.068 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.064+0000 7f4b205be640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b1815c760 con 0x7f4b1806c220 2023-05-22T15:42:41.068 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.064+0000 7f4b1db59640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f4b180cc6c0 0x7f4b180c8080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:41.069 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.064+0000 7f4b1db59640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f4b180cc6c0 0x7f4b180c8080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49034/0 (socket says 172.21.15.16:49034) 2023-05-22T15:42:41.069 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.064+0000 7f4b1db59640 1 -- 172.21.15.16:0/1242080927 learned_addr learned my addr 172.21.15.16:0/1242080927 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:42:41.069 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.064+0000 7f4b1e35a640 1 --2- 172.21.15.16:0/1242080927 >> v2:172.21.15.62:3300/0 conn(0x7f4b181563c0 0x7f4b180c7b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:41.069 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.064+0000 7f4b1eb5b640 1 --2- 172.21.15.16:0/1242080927 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4b1806c220 0x7f4b1806c6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:41.069 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.064+0000 7f4b1db59640 1 -- 172.21.15.16:0/1242080927 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4b1806c220 msgr2=0x7f4b1806c6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:42:41.069 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.064+0000 7f4b1db59640 1 --2- 172.21.15.16:0/1242080927 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4b1806c220 0x7f4b1806c6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:41.069 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.064+0000 7f4b1db59640 1 -- 172.21.15.16:0/1242080927 >> v2:172.21.15.62:3300/0 conn(0x7f4b181563c0 msgr2=0x7f4b180c7b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:42:41.069 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.064+0000 7f4b1db59640 1 --2- 172.21.15.16:0/1242080927 >> v2:172.21.15.62:3300/0 conn(0x7f4b181563c0 0x7f4b180c7b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:41.070 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.064+0000 7f4b1db59640 1 -- 172.21.15.16:0/1242080927 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b1815ca40 con 0x7f4b180cc6c0 2023-05-22T15:42:41.070 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.068+0000 7f4b1db59640 1 --2- 172.21.15.16:0/1242080927 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b180cc6c0 0x7f4b180c8080 secure :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 crypto rx=0x7f4b18065ac0 tx=0x7f4b0801c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad72963645ffccb6 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:41.071 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.068+0000 7f4b1d358640 1 -- 172.21.15.16:0/1242080927 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b080aa610 con 0x7f4b180cc6c0 2023-05-22T15:42:41.071 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.068+0000 7f4b1d358640 1 -- 172.21.15.16:0/1242080927 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b080a6040 con 0x7f4b180cc6c0 2023-05-22T15:42:41.071 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.068+0000 7f4b1d358640 1 -- 172.21.15.16:0/1242080927 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b080aa990 con 0x7f4b180cc6c0 2023-05-22T15:42:41.071 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.068+0000 7f4b205be640 1 -- 172.21.15.16:0/1242080927 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b180cc6c0 msgr2=0x7f4b180c8080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:41.072 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.068+0000 7f4b205be640 1 --2- 172.21.15.16:0/1242080927 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b180cc6c0 0x7f4b180c8080 secure :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 crypto rx=0x7f4b18065ac0 tx=0x7f4b0801c4d0 comp rx=0 tx=0).stop 2023-05-22T15:42:41.072 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.068+0000 7f4b205be640 1 -- 172.21.15.16:0/1242080927 shutdown_connections 2023-05-22T15:42:41.072 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.068+0000 7f4b205be640 1 --2- 172.21.15.16:0/1242080927 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b180cc6c0 0x7f4b180c8080 unknown :-1 s=CLOSED pgs=1189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:41.072 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.068+0000 7f4b205be640 1 --2- 172.21.15.16:0/1242080927 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4b1806c220 0x7f4b1806c6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:41.072 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.068+0000 7f4b205be640 1 --2- 172.21.15.16:0/1242080927 >> v2:172.21.15.62:3300/0 conn(0x7f4b181563c0 0x7f4b180c7b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:41.072 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.068+0000 7f4b205be640 1 -- 172.21.15.16:0/1242080927 >> 172.21.15.16:0/1242080927 conn(0x7f4b180b1110 msgr2=0x7f4b180ce830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:41.073 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.068+0000 7f4b205be640 1 -- 172.21.15.16:0/1242080927 shutdown_connections 2023-05-22T15:42:41.073 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.068+0000 7f4b205be640 1 -- 172.21.15.16:0/1242080927 wait complete. 2023-05-22T15:42:41.074 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.068+0000 7f4b205be640 1 Processor -- start 2023-05-22T15:42:41.074 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.068+0000 7f4b205be640 1 -- start start 2023-05-22T15:42:41.074 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.068+0000 7f4b205be640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b1806c220 0x7f4b1815e2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:41.075 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.068+0000 7f4b205be640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4b180cc6c0 0x7f4b180c9770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:41.075 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.068+0000 7f4b205be640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4b1814d000 0x7f4b1814ee10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:41.075 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.068+0000 7f4b205be640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b180cab80 con 0x7f4b1806c220 2023-05-22T15:42:41.075 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.068+0000 7f4b205be640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b1815e7e0 con 0x7f4b180cc6c0 2023-05-22T15:42:41.075 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.068+0000 7f4b205be640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b180c9cb0 con 0x7f4b1814d000 2023-05-22T15:42:41.075 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.068+0000 7f4b1e35a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b1806c220 0x7f4b1815e2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:41.075 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.068+0000 7f4b1e35a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b1806c220 0x7f4b1815e2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49050/0 (socket says 172.21.15.16:49050) 2023-05-22T15:42:41.076 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.068+0000 7f4b1e35a640 1 -- 172.21.15.16:0/1083424001 learned_addr learned my addr 172.21.15.16:0/1083424001 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:42:41.076 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.068+0000 7f4b1db59640 1 --2- 172.21.15.16:0/1083424001 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4b180cc6c0 0x7f4b180c9770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:41.076 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.068+0000 7f4b1eb5b640 1 --2- 172.21.15.16:0/1083424001 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4b1814d000 0x7f4b1814ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:41.076 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.068+0000 7f4b1e35a640 1 -- 172.21.15.16:0/1083424001 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4b1814d000 msgr2=0x7f4b1814ee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:41.076 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.068+0000 7f4b1e35a640 1 --2- 172.21.15.16:0/1083424001 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4b1814d000 0x7f4b1814ee10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:41.076 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.068+0000 7f4b1e35a640 1 -- 172.21.15.16:0/1083424001 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4b180cc6c0 msgr2=0x7f4b180c9770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:41.076 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.068+0000 7f4b1e35a640 1 --2- 172.21.15.16:0/1083424001 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4b180cc6c0 0x7f4b180c9770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:41.076 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.068+0000 7f4b1e35a640 1 -- 172.21.15.16:0/1083424001 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b080094d0 con 0x7f4b1806c220 2023-05-22T15:42:41.077 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.068+0000 7f4b1eb5b640 1 --2- 172.21.15.16:0/1083424001 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4b1814d000 0x7f4b1814ee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:42:41.077 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.072+0000 7f4b1e35a640 1 --2- 172.21.15.16:0/1083424001 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b1806c220 0x7f4b1815e2a0 secure :-1 s=READY pgs=1190 cs=0 l=1 rev1=1 crypto rx=0x7f4b1400d620 tx=0x7f4b1400dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:41.077 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.072+0000 7f4b0effd640 1 -- 172.21.15.16:0/1083424001 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b1400b620 con 0x7f4b1806c220 2023-05-22T15:42:41.077 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.072+0000 7f4b0effd640 1 -- 172.21.15.16:0/1083424001 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b1400b780 con 0x7f4b1806c220 2023-05-22T15:42:41.077 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.072+0000 7f4b205be640 1 -- 172.21.15.16:0/1083424001 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b18070890 con 0x7f4b1806c220 2023-05-22T15:42:41.077 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.072+0000 7f4b0effd640 1 -- 172.21.15.16:0/1083424001 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b1400bb30 con 0x7f4b1806c220 2023-05-22T15:42:41.077 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.072+0000 7f4b205be640 1 -- 172.21.15.16:0/1083424001 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b18070700 con 0x7f4b1806c220 2023-05-22T15:42:41.078 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.072+0000 7f4b205be640 1 -- 172.21.15.16:0/1083424001 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b18070a20 con 0x7f4b1806c220 2023-05-22T15:42:41.078 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.072+0000 7f4b0effd640 1 -- 172.21.15.16:0/1083424001 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 31) v1 ==== 84389+0+0 (secure 0 0 0) 0x7f4b1400bc90 con 0x7f4b1806c220 2023-05-22T15:42:41.078 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.072+0000 7f4b0effd640 1 --2- 172.21.15.16:0/1083424001 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f4afc065530 0x7f4afc0679e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:41.079 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.072+0000 7f4b1db59640 1 -- 172.21.15.16:0/1083424001 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f4afc065530 msgr2=0x7f4afc0679e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/36921 2023-05-22T15:42:41.079 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.072+0000 7f4b0effd640 1 -- 172.21.15.16:0/1083424001 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 6726+0+0 (secure 0 0 0) 0x7f4b140832a0 con 0x7f4b1806c220 2023-05-22T15:42:41.079 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.072+0000 7f4b1db59640 1 --2- 172.21.15.16:0/1083424001 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f4afc065530 0x7f4afc0679e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-05-22T15:42:41.080 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.076+0000 7f4b0effd640 1 -- 172.21.15.16:0/1083424001 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f4b140509f0 con 0x7f4b1806c220 2023-05-22T15:42:41.117 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:42:41.112+0000 7fa9cb8de180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-05-22T15:42:41.234 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.232+0000 7f4b205be640 1 -- 172.21.15.16:0/1083424001 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f4b1804dbe0 con 0x7f4b1806c220 2023-05-22T15:42:41.235 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.232+0000 7f4b0effd640 1 -- 172.21.15.16:0/1083424001 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v91) v1 ==== 93+0+10 (secure 0 0 0) 0x7f4b14007380 con 0x7f4b1806c220 2023-05-22T15:42:41.235 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2023-05-22T15:42:41.237 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.232+0000 7f4b205be640 1 -- 172.21.15.16:0/1083424001 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f4afc065530 msgr2=0x7f4afc0679e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-05-22T15:42:41.237 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.232+0000 7f4b205be640 1 --2- 172.21.15.16:0/1083424001 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f4afc065530 0x7f4afc0679e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:41.237 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.232+0000 7f4b205be640 1 -- 172.21.15.16:0/1083424001 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b1806c220 msgr2=0x7f4b1815e2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:41.237 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.232+0000 7f4b205be640 1 --2- 172.21.15.16:0/1083424001 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b1806c220 0x7f4b1815e2a0 secure :-1 s=READY pgs=1190 cs=0 l=1 rev1=1 crypto rx=0x7f4b1400d620 tx=0x7f4b1400dae0 comp rx=0 tx=0).stop 2023-05-22T15:42:41.237 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.232+0000 7f4b205be640 1 -- 172.21.15.16:0/1083424001 shutdown_connections 2023-05-22T15:42:41.238 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.232+0000 7f4b205be640 1 --2- 172.21.15.16:0/1083424001 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f4afc065530 0x7f4afc0679e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:41.238 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.232+0000 7f4b205be640 1 --2- 172.21.15.16:0/1083424001 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4b1814d000 0x7f4b1814ee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:41.238 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.232+0000 7f4b205be640 1 --2- 172.21.15.16:0/1083424001 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4b180cc6c0 0x7f4b180c9770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:41.238 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.232+0000 7f4b205be640 1 --2- 172.21.15.16:0/1083424001 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b1806c220 0x7f4b1815e2a0 unknown :-1 s=CLOSED pgs=1190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:41.238 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.232+0000 7f4b205be640 1 -- 172.21.15.16:0/1083424001 >> 172.21.15.16:0/1083424001 conn(0x7f4b180b1110 msgr2=0x7f4b180af720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:41.238 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.232+0000 7f4b205be640 1 -- 172.21.15.16:0/1083424001 shutdown_connections 2023-05-22T15:42:41.238 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.232+0000 7f4b205be640 1 -- 172.21.15.16:0/1083424001 wait complete. 2023-05-22T15:42:41.252 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-05-22T15:42:41.294 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:42:41.291+0000 7ff8c72cb180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-05-22T15:42:41.349 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:42:41.347+0000 7ff8c72cb180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-05-22T15:42:41.362 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.360+0000 7ff084820640 1 Processor -- start 2023-05-22T15:42:41.363 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.360+0000 7ff084820640 1 -- start start 2023-05-22T15:42:41.363 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.360+0000 7ff084820640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff0800cf870 0x7ff08006db00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:41.363 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.360+0000 7ff084820640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff08006e740 0x7ff08006ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:41.363 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.360+0000 7ff084820640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff08006e040 con 0x7ff08015cfa0 2023-05-22T15:42:41.364 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.360+0000 7ff084820640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0800cb9f0 con 0x7ff08006e740 2023-05-22T15:42:41.364 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.360+0000 7ff084820640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff08006f100 con 0x7ff0800cf870 2023-05-22T15:42:41.364 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.360+0000 7ff07e59c640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7ff08015cfa0 0x7ff0800b1fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:41054/0 (socket says 172.21.15.16:41054) 2023-05-22T15:42:41.364 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.360+0000 7ff07e59c640 1 -- 172.21.15.16:0/741946069 learned_addr learned my addr 172.21.15.16:0/741946069 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:42:41.364 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.360+0000 7ff07ed9d640 1 --2- 172.21.15.16:0/741946069 >> v2:172.21.15.16:3300/0 conn(0x7ff08006e740 0x7ff08006ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:41.365 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.360+0000 7ff07dd9b640 1 --2- 172.21.15.16:0/741946069 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff0800cf870 0x7ff08006db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:41.365 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.360+0000 7ff07ed9d640 1 -- 172.21.15.16:0/741946069 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff0800cf870 msgr2=0x7ff08006db00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:42:41.365 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.360+0000 7ff07ed9d640 1 --2- 172.21.15.16:0/741946069 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff0800cf870 0x7ff08006db00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:41.365 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.360+0000 7ff07ed9d640 1 -- 172.21.15.16:0/741946069 >> v1:172.21.15.16:6789/0 conn(0x7ff08015cfa0 legacy=0x7ff0800b1fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:41.365 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.360+0000 7ff07ed9d640 1 -- 172.21.15.16:0/741946069 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff08006e490 con 0x7ff08006e740 2023-05-22T15:42:41.367 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.364+0000 7ff07ed9d640 1 --2- 172.21.15.16:0/741946069 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff08006e740 0x7ff08006ebc0 secure :-1 s=READY pgs=1192 cs=0 l=1 rev1=1 crypto rx=0x7ff078009d90 tx=0x7ff07801c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=866b4238bde66341 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:41.367 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.364+0000 7ff07d59a640 1 -- 172.21.15.16:0/741946069 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0780a5950 con 0x7ff08006e740 2023-05-22T15:42:41.367 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.364+0000 7ff07d59a640 1 -- 172.21.15.16:0/741946069 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0780a5ab0 con 0x7ff08006e740 2023-05-22T15:42:41.367 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.364+0000 7ff07d59a640 1 -- 172.21.15.16:0/741946069 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0780a5e40 con 0x7ff08006e740 2023-05-22T15:42:41.368 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.364+0000 7ff084820640 1 -- 172.21.15.16:0/741946069 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff08006e740 msgr2=0x7ff08006ebc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:41.368 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.364+0000 7ff084820640 1 --2- 172.21.15.16:0/741946069 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff08006e740 0x7ff08006ebc0 secure :-1 s=READY pgs=1192 cs=0 l=1 rev1=1 crypto rx=0x7ff078009d90 tx=0x7ff07801c890 comp rx=0 tx=0).stop 2023-05-22T15:42:41.368 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.364+0000 7ff084820640 1 -- 172.21.15.16:0/741946069 shutdown_connections 2023-05-22T15:42:41.369 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.364+0000 7ff084820640 1 --2- 172.21.15.16:0/741946069 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff08006e740 0x7ff08006ebc0 unknown :-1 s=CLOSED pgs=1192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:41.369 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.364+0000 7ff084820640 1 --2- 172.21.15.16:0/741946069 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff0800cf870 0x7ff08006db00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:41.369 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.364+0000 7ff084820640 1 -- 172.21.15.16:0/741946069 >> 172.21.15.16:0/741946069 conn(0x7ff0800cc870 msgr2=0x7ff0800cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:41.369 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.364+0000 7ff084820640 1 -- 172.21.15.16:0/741946069 shutdown_connections 2023-05-22T15:42:41.369 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.364+0000 7ff084820640 1 -- 172.21.15.16:0/741946069 wait complete. 2023-05-22T15:42:41.370 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.364+0000 7ff084820640 1 Processor -- start 2023-05-22T15:42:41.371 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.364+0000 7ff084820640 1 -- start start 2023-05-22T15:42:41.371 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.364+0000 7ff084820640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff08006e740 0x7ff08014f740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:41.371 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.364+0000 7ff084820640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff0800cf870 0x7ff080148450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:41.371 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.364+0000 7ff084820640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff08015cfa0 0x7ff080148b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:41.371 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.364+0000 7ff084820640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff080148990 con 0x7ff08006e740 2023-05-22T15:42:41.371 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.364+0000 7ff084820640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0801552d0 con 0x7ff08015cfa0 2023-05-22T15:42:41.371 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.364+0000 7ff084820640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff08014fd50 con 0x7ff0800cf870 2023-05-22T15:42:41.372 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.368+0000 7ff07e59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff08006e740 0x7ff08014f740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:41.372 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.368+0000 7ff07e59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff08006e740 0x7ff08014f740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49072/0 (socket says 172.21.15.16:49072) 2023-05-22T15:42:41.372 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.368+0000 7ff07e59c640 1 -- 172.21.15.16:0/2638926917 learned_addr learned my addr 172.21.15.16:0/2638926917 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:42:41.372 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.368+0000 7ff07ed9d640 1 --2- 172.21.15.16:0/2638926917 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff08015cfa0 0x7ff080148b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:41.372 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.368+0000 7ff07e59c640 1 -- 172.21.15.16:0/2638926917 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff0800cf870 msgr2=0x7ff080148450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:41.372 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.368+0000 7ff07e59c640 1 --2- 172.21.15.16:0/2638926917 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff0800cf870 0x7ff080148450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:41.372 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.368+0000 7ff07e59c640 1 -- 172.21.15.16:0/2638926917 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff08015cfa0 msgr2=0x7ff080148b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:41.372 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.368+0000 7ff07e59c640 1 --2- 172.21.15.16:0/2638926917 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff08015cfa0 0x7ff080148b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:41.373 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.368+0000 7ff07e59c640 1 -- 172.21.15.16:0/2638926917 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff078009a70 con 0x7ff08006e740 2023-05-22T15:42:41.373 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.368+0000 7ff07e59c640 1 --2- 172.21.15.16:0/2638926917 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff08006e740 0x7ff08014f740 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 crypto rx=0x7ff07400c400 tx=0x7ff07400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:41.373 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.368+0000 7ff06effd640 1 -- 172.21.15.16:0/2638926917 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff074005cf0 con 0x7ff08006e740 2023-05-22T15:42:41.373 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.368+0000 7ff06effd640 1 -- 172.21.15.16:0/2638926917 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff074005e50 con 0x7ff08006e740 2023-05-22T15:42:41.373 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.368+0000 7ff084820640 1 -- 172.21.15.16:0/2638926917 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0800cb610 con 0x7ff08006e740 2023-05-22T15:42:41.373 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.368+0000 7ff06effd640 1 -- 172.21.15.16:0/2638926917 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0740072d0 con 0x7ff08006e740 2023-05-22T15:42:41.374 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.368+0000 7ff084820640 1 -- 172.21.15.16:0/2638926917 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0800cb330 con 0x7ff08006e740 2023-05-22T15:42:41.374 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:42:41.368+0000 7fa9cb8de180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-05-22T15:42:41.375 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.368+0000 7ff06effd640 1 -- 172.21.15.16:0/2638926917 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 31) v1 ==== 84389+0+0 (secure 0 0 0) 0x7ff074007430 con 0x7ff08006e740 2023-05-22T15:42:41.375 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.368+0000 7ff084820640 1 -- 172.21.15.16:0/2638926917 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff08015d370 con 0x7ff08006e740 2023-05-22T15:42:41.375 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.368+0000 7ff06effd640 1 --2- 172.21.15.16:0/2638926917 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ff060065530 0x7ff0600679e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:41.375 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.368+0000 7ff07dd9b640 1 -- 172.21.15.16:0/2638926917 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ff060065530 msgr2=0x7ff0600679e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/36921 2023-05-22T15:42:41.375 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.368+0000 7ff06effd640 1 -- 172.21.15.16:0/2638926917 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 6726+0+0 (secure 0 0 0) 0x7ff074082040 con 0x7ff08006e740 2023-05-22T15:42:41.375 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.368+0000 7ff07dd9b640 1 --2- 172.21.15.16:0/2638926917 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ff060065530 0x7ff0600679e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-05-22T15:42:41.376 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.372+0000 7ff06effd640 1 -- 172.21.15.16:0/2638926917 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7ff07404a620 con 0x7ff08006e740 2023-05-22T15:42:41.428 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:42:41.424+0000 7fa9cb8de180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-05-22T15:42:41.460 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.456+0000 7ff06effd640 1 -- 172.21.15.16:0/2638926917 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mgrmap(e 32) v1 ==== 83941+0+0 (secure 0 0 0) 0x7ff07404fc90 con 0x7ff08006e740 2023-05-22T15:42:41.460 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.456+0000 7ff06effd640 1 -- 172.21.15.16:0/2638926917 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ff060065530 msgr2=0x7ff0600679e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-05-22T15:42:41.460 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.456+0000 7ff06effd640 1 --2- 172.21.15.16:0/2638926917 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ff060065530 0x7ff0600679e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:41.534 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:42:41.532+0000 7fa958a6f640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T15:42:41.534 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:42:41.532+0000 7fa958a6f640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T15:42:41.535 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:42:41.532+0000 7fa958a6f640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T15:42:41.535 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:42:41.532+0000 7fa958a6f640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T15:42:41.535 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:42:41.532+0000 7fa958a6f640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T15:42:41.535 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:42:41.532+0000 7fa955a69640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T15:42:41.536 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:42:41.532+0000 7fa955a69640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T15:42:41.536 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:42:41.532+0000 7fa955a69640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T15:42:41.536 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:42:41.532+0000 7fa955a69640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T15:42:41.536 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:42:41.532+0000 7fa955a69640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T15:42:41.549 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.544+0000 7ff084820640 1 -- 172.21.15.16:0/2638926917 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7ff08006e040 con 0x7ff08006e740 2023-05-22T15:42:41.550 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.544+0000 7ff06effd640 1 -- 172.21.15.16:0/2638926917 <== mon.0 v2:172.21.15.16:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v92) v1 ==== 92+0+10 (secure 0 0 0) 0x7ff08006e040 con 0x7ff08006e740 2023-05-22T15:42:41.550 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 8 2023-05-22T15:42:41.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.548+0000 7ff084820640 1 -- 172.21.15.16:0/2638926917 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff08006e740 msgr2=0x7ff08014f740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:41.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.548+0000 7ff084820640 1 --2- 172.21.15.16:0/2638926917 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff08006e740 0x7ff08014f740 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 crypto rx=0x7ff07400c400 tx=0x7ff07400c8c0 comp rx=0 tx=0).stop 2023-05-22T15:42:41.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.548+0000 7ff084820640 1 -- 172.21.15.16:0/2638926917 shutdown_connections 2023-05-22T15:42:41.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.548+0000 7ff084820640 1 --2- 172.21.15.16:0/2638926917 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff08015cfa0 0x7ff080148b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:41.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.548+0000 7ff084820640 1 --2- 172.21.15.16:0/2638926917 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ff060065530 0x7ff0600679e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:41.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.548+0000 7ff084820640 1 --2- 172.21.15.16:0/2638926917 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff0800cf870 0x7ff080148450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:41.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.548+0000 7ff084820640 1 --2- 172.21.15.16:0/2638926917 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff08006e740 0x7ff08014f740 unknown :-1 s=CLOSED pgs=1193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:41.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.548+0000 7ff084820640 1 -- 172.21.15.16:0/2638926917 >> 172.21.15.16:0/2638926917 conn(0x7ff0800cc870 msgr2=0x7ff08015b5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:41.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.548+0000 7ff084820640 1 -- 172.21.15.16:0/2638926917 shutdown_connections 2023-05-22T15:42:41.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.548+0000 7ff084820640 1 -- 172.21.15.16:0/2638926917 wait complete. 2023-05-22T15:42:41.569 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-05-22T15:42:41.676 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.672+0000 7fa7325d3640 1 Processor -- start 2023-05-22T15:42:41.677 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.672+0000 7fa7325d3640 1 -- start start 2023-05-22T15:42:41.677 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.672+0000 7fa7325d3640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa72c1596e0 0x7fa72c163950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:41.678 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.672+0000 7fa7325d3640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa72c159f70 con 0x7fa72c0cb4c0 2023-05-22T15:42:41.678 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.672+0000 7fa7325d3640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa72c0cb030 con 0x7fa72c06b6c0 2023-05-22T15:42:41.678 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.672+0000 7fa7325d3640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa72c071760 con 0x7fa72c1596e0 2023-05-22T15:42:41.678 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.672+0000 7fa72b7fe640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fa72c0cb4c0 0x7fa72c158fd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:41066/0 (socket says 172.21.15.16:41066) 2023-05-22T15:42:41.678 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.672+0000 7fa72b7fe640 1 -- 172.21.15.16:0/1164875870 learned_addr learned my addr 172.21.15.16:0/1164875870 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:42:41.680 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.676+0000 7fa730b70640 1 --2- 172.21.15.16:0/1164875870 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa72c1596e0 0x7fa72c163950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:41.680 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.676+0000 7fa72affd640 1 -- 172.21.15.16:0/1164875870 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2050297562 0 0) 0x7fa72c159f70 con 0x7fa72c0cb4c0 2023-05-22T15:42:41.680 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.676+0000 7fa72affd640 1 -- 172.21.15.16:0/1164875870 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa714005740 con 0x7fa72c0cb4c0 2023-05-22T15:42:41.680 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.676+0000 7fa72affd640 1 -- 172.21.15.16:0/1164875870 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4209900380 0 0) 0x7fa714005740 con 0x7fa72c0cb4c0 2023-05-22T15:42:41.680 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.676+0000 7fa72affd640 1 -- 172.21.15.16:0/1164875870 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa72c1596e0 msgr2=0x7fa72c163950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:42:41.680 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.676+0000 7fa72affd640 1 --2- 172.21.15.16:0/1164875870 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa72c1596e0 0x7fa72c163950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:41.681 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.676+0000 7fa72affd640 1 -- 172.21.15.16:0/1164875870 >> v1:172.21.15.62:6789/0 conn(0x7fa72c06b6c0 legacy=0x7fa72c06ba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:41.681 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.676+0000 7fa72affd640 1 -- 172.21.15.16:0/1164875870 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa72c159c90 con 0x7fa72c0cb4c0 2023-05-22T15:42:41.681 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.676+0000 7fa72affd640 1 -- 172.21.15.16:0/1164875870 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7fa720001680 con 0x7fa72c0cb4c0 2023-05-22T15:42:41.681 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.676+0000 7fa72affd640 1 -- 172.21.15.16:0/1164875870 >> v1:172.21.15.16:6789/0 conn(0x7fa72c0cb4c0 legacy=0x7fa72c158fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:41.681 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.676+0000 7fa72affd640 1 --2- 172.21.15.16:0/1164875870 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa714005bc0 0x7fa714005fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:41.681 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.676+0000 7fa72affd640 1 --2- 172.21.15.16:0/1164875870 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa72c1596e0 0x7fa7140069e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:41.681 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.676+0000 7fa72affd640 1 --2- 172.21.15.16:0/1164875870 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa714006f20 0x7fa714007390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:41.681 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.676+0000 7fa72affd640 1 -- 172.21.15.16:0/1164875870 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa72c071760 con 0x7fa714005bc0 2023-05-22T15:42:41.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.676+0000 7fa72affd640 1 -- 172.21.15.16:0/1164875870 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa71400dd90 con 0x7fa714006f20 2023-05-22T15:42:41.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.676+0000 7fa72affd640 1 -- 172.21.15.16:0/1164875870 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa71400df50 con 0x7fa72c1596e0 2023-05-22T15:42:41.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.676+0000 7fa72bfff640 1 --2- 172.21.15.16:0/1164875870 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa714005bc0 0x7fa714005fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:41.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.676+0000 7fa72b7fe640 1 --2- 172.21.15.16:0/1164875870 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa72c1596e0 0x7fa7140069e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:41.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.676+0000 7fa730b70640 1 --2- 172.21.15.16:0/1164875870 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa714006f20 0x7fa714007390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:41.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.676+0000 7fa72bfff640 1 -- 172.21.15.16:0/1164875870 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa72c1596e0 msgr2=0x7fa7140069e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:42:41.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.676+0000 7fa72bfff640 1 --2- 172.21.15.16:0/1164875870 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa72c1596e0 0x7fa7140069e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:41.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.676+0000 7fa72bfff640 1 -- 172.21.15.16:0/1164875870 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa714006f20 msgr2=0x7fa714007390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:42:41.683 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.676+0000 7fa72bfff640 1 --2- 172.21.15.16:0/1164875870 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa714006f20 0x7fa714007390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:41.683 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.676+0000 7fa72bfff640 1 -- 172.21.15.16:0/1164875870 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa714005740 con 0x7fa714005bc0 2023-05-22T15:42:41.683 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.676+0000 7fa730b70640 1 --2- 172.21.15.16:0/1164875870 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa714006f20 0x7fa714007390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:42:41.683 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.676+0000 7fa72b7fe640 1 --2- 172.21.15.16:0/1164875870 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa72c1596e0 0x7fa7140069e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:42:41.685 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.676+0000 7fa72bfff640 1 --2- 172.21.15.16:0/1164875870 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa714005bc0 0x7fa714005fd0 secure :-1 s=READY pgs=1208 cs=0 l=1 rev1=1 crypto rx=0x7fa71c001780 tx=0x7fa71c01c400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d43d02de9233e7 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:41.685 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.676+0000 7fa72affd640 1 -- 172.21.15.16:0/1164875870 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa71c0052d0 con 0x7fa714005bc0 2023-05-22T15:42:41.685 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.676+0000 7fa72affd640 1 -- 172.21.15.16:0/1164875870 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa71c0a7050 con 0x7fa714005bc0 2023-05-22T15:42:41.686 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.676+0000 7fa72affd640 1 -- 172.21.15.16:0/1164875870 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa71c003600 con 0x7fa714005bc0 2023-05-22T15:42:41.686 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.676+0000 7fa7325d3640 1 -- 172.21.15.16:0/1164875870 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa714005bc0 msgr2=0x7fa714005fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:41.686 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.676+0000 7fa7325d3640 1 --2- 172.21.15.16:0/1164875870 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa714005bc0 0x7fa714005fd0 secure :-1 s=READY pgs=1208 cs=0 l=1 rev1=1 crypto rx=0x7fa71c001780 tx=0x7fa71c01c400 comp rx=0 tx=0).stop 2023-05-22T15:42:41.686 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.676+0000 7fa730b70640 1 -- 172.21.15.16:0/1164875870 reap_dead start 2023-05-22T15:42:41.686 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.676+0000 7fa7325d3640 1 -- 172.21.15.16:0/1164875870 shutdown_connections 2023-05-22T15:42:41.686 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.676+0000 7fa7325d3640 1 -- 172.21.15.16:0/1164875870 >> 172.21.15.16:0/1164875870 conn(0x7fa72c069720 msgr2=0x7fa72c069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:41.686 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.676+0000 7fa7325d3640 1 -- 172.21.15.16:0/1164875870 shutdown_connections 2023-05-22T15:42:41.687 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.676+0000 7fa7325d3640 1 -- 172.21.15.16:0/1164875870 wait complete. 2023-05-22T15:42:41.687 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.680+0000 7fa7325d3640 1 Processor -- start 2023-05-22T15:42:41.687 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.680+0000 7fa7325d3640 1 -- start start 2023-05-22T15:42:41.687 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.680+0000 7fa7325d3640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa714005bc0 0x7fa72c14ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:41.687 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.680+0000 7fa7325d3640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa72c14f570 0x7fa72c15e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:41.687 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.680+0000 7fa7325d3640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa72c1600e0 0x7fa72c160580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:41.687 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.680+0000 7fa7325d3640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa72c15f040 con 0x7fa72c14f570 2023-05-22T15:42:41.687 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.680+0000 7fa7325d3640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa72c15f320 con 0x7fa72c1600e0 2023-05-22T15:42:41.688 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.680+0000 7fa7325d3640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa72c15f600 con 0x7fa714005bc0 2023-05-22T15:42:41.688 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.680+0000 7fa72b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa72c14f570 0x7fa72c15e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:41.688 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.680+0000 7fa72b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa72c14f570 0x7fa72c15e6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49206/0 (socket says 172.21.15.16:49206) 2023-05-22T15:42:41.688 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.680+0000 7fa72b7fe640 1 -- 172.21.15.16:0/2723270707 learned_addr learned my addr 172.21.15.16:0/2723270707 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:42:41.688 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.680+0000 7fa730b70640 1 --2- 172.21.15.16:0/2723270707 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa72c1600e0 0x7fa72c160580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:41.688 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.680+0000 7fa72bfff640 1 --2- 172.21.15.16:0/2723270707 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa714005bc0 0x7fa72c14ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:41.688 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.680+0000 7fa72b7fe640 1 -- 172.21.15.16:0/2723270707 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa714005bc0 msgr2=0x7fa72c14ec80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:41.689 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.680+0000 7fa72b7fe640 1 --2- 172.21.15.16:0/2723270707 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa714005bc0 0x7fa72c14ec80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:41.689 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.680+0000 7fa72b7fe640 1 -- 172.21.15.16:0/2723270707 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa72c1600e0 msgr2=0x7fa72c160580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:41.689 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.680+0000 7fa72b7fe640 1 --2- 172.21.15.16:0/2723270707 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa72c1600e0 0x7fa72c160580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:41.689 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.680+0000 7fa72b7fe640 1 -- 172.21.15.16:0/2723270707 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa72c1569d0 con 0x7fa72c14f570 2023-05-22T15:42:41.689 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.680+0000 7fa730b70640 1 --2- 172.21.15.16:0/2723270707 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa72c1600e0 0x7fa72c160580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:42:41.689 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.680+0000 7fa72b7fe640 1 --2- 172.21.15.16:0/2723270707 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa72c14f570 0x7fa72c15e6f0 secure :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 crypto rx=0x7fa72000d800 tx=0x7fa72000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:41.689 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.680+0000 7fa728ff9640 1 -- 172.21.15.16:0/2723270707 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7200074e0 con 0x7fa72c14f570 2023-05-22T15:42:41.689 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.680+0000 7fa728ff9640 1 -- 172.21.15.16:0/2723270707 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa720007640 con 0x7fa72c14f570 2023-05-22T15:42:41.690 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.680+0000 7fa7325d3640 1 -- 172.21.15.16:0/2723270707 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa72c156840 con 0x7fa72c14f570 2023-05-22T15:42:41.690 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.680+0000 7fa728ff9640 1 -- 172.21.15.16:0/2723270707 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7200049f0 con 0x7fa72c14f570 2023-05-22T15:42:41.690 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.680+0000 7fa7325d3640 1 -- 172.21.15.16:0/2723270707 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa72c0ce820 con 0x7fa72c14f570 2023-05-22T15:42:41.690 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.680+0000 7fa7325d3640 1 -- 172.21.15.16:0/2723270707 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa72c04dbe0 con 0x7fa72c14f570 2023-05-22T15:42:41.690 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.684+0000 7fa728ff9640 1 -- 172.21.15.16:0/2723270707 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 32) v1 ==== 83941+0+0 (secure 0 0 0) 0x7fa720004b50 con 0x7fa72c14f570 2023-05-22T15:42:41.691 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.684+0000 7fa728ff9640 1 -- 172.21.15.16:0/2723270707 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 7027+0+0 (secure 0 0 0) 0x7fa720082520 con 0x7fa72c14f570 2023-05-22T15:42:41.691 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.684+0000 7fa728ff9640 1 -- 172.21.15.16:0/2723270707 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fa7200501d0 con 0x7fa72c14f570 2023-05-22T15:42:41.835 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.832+0000 7fa7325d3640 1 -- 172.21.15.16:0/2723270707 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fa72c0c1850 con 0x7fa72c14f570 2023-05-22T15:42:41.835 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.832+0000 7fa728ff9640 1 -- 172.21.15.16:0/2723270707 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v92) v1 ==== 104+0+11 (secure 0 0 0) 0x7fa7200190a0 con 0x7fa72c14f570 2023-05-22T15:42:41.836 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2023-05-22T15:42:41.837 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.832+0000 7fa7325d3640 1 -- 172.21.15.16:0/2723270707 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa72c14f570 msgr2=0x7fa72c15e6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:41.838 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.832+0000 7fa7325d3640 1 --2- 172.21.15.16:0/2723270707 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa72c14f570 0x7fa72c15e6f0 secure :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 crypto rx=0x7fa72000d800 tx=0x7fa72000dcc0 comp rx=0 tx=0).stop 2023-05-22T15:42:41.838 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.832+0000 7fa7325d3640 1 -- 172.21.15.16:0/2723270707 shutdown_connections 2023-05-22T15:42:41.838 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.832+0000 7fa7325d3640 1 --2- 172.21.15.16:0/2723270707 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa72c1600e0 0x7fa72c160580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:41.838 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.832+0000 7fa7325d3640 1 --2- 172.21.15.16:0/2723270707 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa72c14f570 0x7fa72c15e6f0 unknown :-1 s=CLOSED pgs=1209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:41.838 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.832+0000 7fa7325d3640 1 --2- 172.21.15.16:0/2723270707 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa714005bc0 0x7fa72c14ec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:41.838 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.832+0000 7fa7325d3640 1 -- 172.21.15.16:0/2723270707 >> 172.21.15.16:0/2723270707 conn(0x7fa72c069720 msgr2=0x7fa72c0b2e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:41.838 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.832+0000 7fa7325d3640 1 -- 172.21.15.16:0/2723270707 shutdown_connections 2023-05-22T15:42:41.839 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.832+0000 7fa7325d3640 1 -- 172.21.15.16:0/2723270707 wait complete. 2023-05-22T15:42:41.853 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-05-22T15:42:41.982 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.976+0000 7fd31dda6640 1 Processor -- start 2023-05-22T15:42:41.983 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.980+0000 7fd31dda6640 1 -- start start 2023-05-22T15:42:41.985 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.980+0000 7fd31dda6640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd3180c67d0 0x7fd3180cf730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:41.985 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.980+0000 7fd31dda6640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd3180b2880 0x7fd3180cbfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:41.985 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.980+0000 7fd31dda6640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3180cfc70 con 0x7fd3180cc520 2023-05-22T15:42:41.985 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.980+0000 7fd31dda6640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd31815a300 con 0x7fd3180c67d0 2023-05-22T15:42:41.986 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.980+0000 7fd31dda6640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3180c4dc0 con 0x7fd3180b2880 2023-05-22T15:42:41.986 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.980+0000 7fd3177fe640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd3180c67d0 0x7fd3180cf730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:41.986 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.980+0000 7fd3177fe640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd3180c67d0 0x7fd3180cf730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49216/0 (socket says 172.21.15.16:49216) 2023-05-22T15:42:41.987 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.980+0000 7fd3177fe640 1 -- 172.21.15.16:0/1460493599 learned_addr learned my addr 172.21.15.16:0/1460493599 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:42:41.987 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.980+0000 7fd316ffd640 1 --2- 172.21.15.16:0/1460493599 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd3180b2880 0x7fd3180cbfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:41.987 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.980+0000 7fd3177fe640 1 -- 172.21.15.16:0/1460493599 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd3180b2880 msgr2=0x7fd3180cbfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:42:41.987 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.980+0000 7fd3177fe640 1 --2- 172.21.15.16:0/1460493599 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd3180b2880 0x7fd3180cbfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:41.988 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.980+0000 7fd3177fe640 1 -- 172.21.15.16:0/1460493599 >> v1:172.21.15.62:6789/0 conn(0x7fd3180cc520 legacy=0x7fd3180ce160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:41.988 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.980+0000 7fd3177fe640 1 -- 172.21.15.16:0/1460493599 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3180c6ba0 con 0x7fd3180c67d0 2023-05-22T15:42:41.988 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.980+0000 7fd316ffd640 1 --2- 172.21.15.16:0/1460493599 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd3180b2880 0x7fd3180cbfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:42:41.993 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.984+0000 7fd3177fe640 1 --2- 172.21.15.16:0/1460493599 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd3180c67d0 0x7fd3180cf730 secure :-1 s=READY pgs=1210 cs=0 l=1 rev1=1 crypto rx=0x7fd308009870 tx=0x7fd30801c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3922dd8adb72e916 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:41.993 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.984+0000 7fd3167fc640 1 -- 172.21.15.16:0/1460493599 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3080ac600 con 0x7fd3180c67d0 2023-05-22T15:42:41.993 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.984+0000 7fd3167fc640 1 -- 172.21.15.16:0/1460493599 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3080a8050 con 0x7fd3180c67d0 2023-05-22T15:42:41.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.984+0000 7fd3167fc640 1 -- 172.21.15.16:0/1460493599 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3080ac980 con 0x7fd3180c67d0 2023-05-22T15:42:41.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.984+0000 7fd31dda6640 1 -- 172.21.15.16:0/1460493599 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd3180c67d0 msgr2=0x7fd3180cf730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:41.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.984+0000 7fd31dda6640 1 --2- 172.21.15.16:0/1460493599 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd3180c67d0 0x7fd3180cf730 secure :-1 s=READY pgs=1210 cs=0 l=1 rev1=1 crypto rx=0x7fd308009870 tx=0x7fd30801c480 comp rx=0 tx=0).stop 2023-05-22T15:42:41.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.984+0000 7fd31dda6640 1 -- 172.21.15.16:0/1460493599 shutdown_connections 2023-05-22T15:42:41.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.984+0000 7fd31dda6640 1 --2- 172.21.15.16:0/1460493599 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd3180b2880 0x7fd3180cbfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:41.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.984+0000 7fd31dda6640 1 --2- 172.21.15.16:0/1460493599 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd3180c67d0 0x7fd3180cf730 unknown :-1 s=CLOSED pgs=1210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:41.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.984+0000 7fd31dda6640 1 -- 172.21.15.16:0/1460493599 >> 172.21.15.16:0/1460493599 conn(0x7fd31806f5a0 msgr2=0x7fd31806f9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:41.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.984+0000 7fd31dda6640 1 -- 172.21.15.16:0/1460493599 shutdown_connections 2023-05-22T15:42:41.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.984+0000 7fd31dda6640 1 -- 172.21.15.16:0/1460493599 wait complete. 2023-05-22T15:42:41.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.984+0000 7fd31dda6640 1 Processor -- start 2023-05-22T15:42:41.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.984+0000 7fd31dda6640 1 -- start start 2023-05-22T15:42:41.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.984+0000 7fd31dda6640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd3180b2880 0x7fd31814d730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:41.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.984+0000 7fd31dda6640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd3180c67d0 0x7fd3180c4fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:41.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.984+0000 7fd31dda6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd3180cc520 0x7fd3180c54e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:41.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.984+0000 7fd31dda6640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd318157a20 con 0x7fd3180cc520 2023-05-22T15:42:41.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.984+0000 7fd31dda6640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd318157cc0 con 0x7fd3180b2880 2023-05-22T15:42:41.998 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.984+0000 7fd31dda6640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd318157fa0 con 0x7fd3180c67d0 2023-05-22T15:42:41.998 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.988+0000 7fd317fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd3180cc520 0x7fd3180c54e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:41.998 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.988+0000 7fd317fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd3180cc520 0x7fd3180c54e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49228/0 (socket says 172.21.15.16:49228) 2023-05-22T15:42:41.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.988+0000 7fd317fff640 1 -- 172.21.15.16:0/4085687987 learned_addr learned my addr 172.21.15.16:0/4085687987 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:42:41.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.988+0000 7fd3177fe640 1 --2- 172.21.15.16:0/4085687987 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd3180b2880 0x7fd31814d730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:41.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.988+0000 7fd316ffd640 1 --2- 172.21.15.16:0/4085687987 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd3180c67d0 0x7fd3180c4fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:41.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.988+0000 7fd317fff640 1 -- 172.21.15.16:0/4085687987 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd3180c67d0 msgr2=0x7fd3180c4fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:42.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.988+0000 7fd317fff640 1 --2- 172.21.15.16:0/4085687987 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd3180c67d0 0x7fd3180c4fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:42.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.988+0000 7fd317fff640 1 -- 172.21.15.16:0/4085687987 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd3180b2880 msgr2=0x7fd31814d730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:42.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.988+0000 7fd317fff640 1 --2- 172.21.15.16:0/4085687987 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd3180b2880 0x7fd31814d730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:42.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.988+0000 7fd317fff640 1 -- 172.21.15.16:0/4085687987 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3080094d0 con 0x7fd3180cc520 2023-05-22T15:42:42.001 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.988+0000 7fd316ffd640 1 --2- 172.21.15.16:0/4085687987 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd3180c67d0 0x7fd3180c4fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:42:42.001 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.988+0000 7fd317fff640 1 --2- 172.21.15.16:0/4085687987 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd3180cc520 0x7fd3180c54e0 secure :-1 s=READY pgs=1211 cs=0 l=1 rev1=1 crypto rx=0x7fd31000c430 tx=0x7fd31000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:42.001 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.988+0000 7fd30ffff640 1 -- 172.21.15.16:0/4085687987 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd310005220 con 0x7fd3180cc520 2023-05-22T15:42:42.001 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.988+0000 7fd30ffff640 1 -- 172.21.15.16:0/4085687987 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd310005380 con 0x7fd3180cc520 2023-05-22T15:42:42.002 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.988+0000 7fd31dda6640 1 -- 172.21.15.16:0/4085687987 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3180d0790 con 0x7fd3180cc520 2023-05-22T15:42:42.002 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.988+0000 7fd30ffff640 1 -- 172.21.15.16:0/4085687987 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd310007270 con 0x7fd3180cc520 2023-05-22T15:42:42.002 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.988+0000 7fd31dda6640 1 -- 172.21.15.16:0/4085687987 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3180d04b0 con 0x7fd3180cc520 2023-05-22T15:42:42.002 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.988+0000 7fd31dda6640 1 -- 172.21.15.16:0/4085687987 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3180d0a70 con 0x7fd3180cc520 2023-05-22T15:42:42.003 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.988+0000 7fd30ffff640 1 -- 172.21.15.16:0/4085687987 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 32) v1 ==== 83941+0+0 (secure 0 0 0) 0x7fd3100073d0 con 0x7fd3180cc520 2023-05-22T15:42:42.003 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.988+0000 7fd30ffff640 1 -- 172.21.15.16:0/4085687987 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 7027+0+0 (secure 0 0 0) 0x7fd310081c50 con 0x7fd3180cc520 2023-05-22T15:42:42.003 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:41.992+0000 7fd30ffff640 1 -- 172.21.15.16:0/4085687987 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fd31004f950 con 0x7fd3180cc520 2023-05-22T15:42:42.145 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.140+0000 7fd31dda6640 1 -- 172.21.15.16:0/4085687987 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fd31804dbe0 con 0x7fd3180cc520 2023-05-22T15:42:42.145 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.140+0000 7fd30ffff640 1 -- 172.21.15.16:0/4085687987 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v92) v1 ==== 100+0+11 (secure 0 0 0) 0x7fd310052f70 con 0x7fd3180cc520 2023-05-22T15:42:42.146 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 64 2023-05-22T15:42:42.148 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.144+0000 7fd31dda6640 1 -- 172.21.15.16:0/4085687987 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd3180cc520 msgr2=0x7fd3180c54e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:42.148 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.144+0000 7fd31dda6640 1 --2- 172.21.15.16:0/4085687987 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd3180cc520 0x7fd3180c54e0 secure :-1 s=READY pgs=1211 cs=0 l=1 rev1=1 crypto rx=0x7fd31000c430 tx=0x7fd31000c8f0 comp rx=0 tx=0).stop 2023-05-22T15:42:42.148 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.144+0000 7fd31dda6640 1 -- 172.21.15.16:0/4085687987 shutdown_connections 2023-05-22T15:42:42.148 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.144+0000 7fd31dda6640 1 --2- 172.21.15.16:0/4085687987 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd3180cc520 0x7fd3180c54e0 unknown :-1 s=CLOSED pgs=1211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:42.149 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.144+0000 7fd31dda6640 1 --2- 172.21.15.16:0/4085687987 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd3180c67d0 0x7fd3180c4fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:42.149 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.144+0000 7fd31dda6640 1 --2- 172.21.15.16:0/4085687987 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd3180b2880 0x7fd31814d730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:42.149 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.144+0000 7fd31dda6640 1 -- 172.21.15.16:0/4085687987 >> 172.21.15.16:0/4085687987 conn(0x7fd31806f5a0 msgr2=0x7fd31806f970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:42.149 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.144+0000 7fd31dda6640 1 -- 172.21.15.16:0/4085687987 shutdown_connections 2023-05-22T15:42:42.150 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.144+0000 7fd31dda6640 1 -- 172.21.15.16:0/4085687987 wait complete. 2023-05-22T15:42:42.163 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-05-22T15:42:42.286 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.280+0000 7f6d31b9b640 1 Processor -- start 2023-05-22T15:42:42.286 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.280+0000 7f6d31b9b640 1 -- start start 2023-05-22T15:42:42.288 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.284+0000 7f6d31b9b640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6d2c0cbf80 0x7f6d2c06b210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:42.288 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.284+0000 7f6d31b9b640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f6d2c0b22e0 0x7f6d2c06bec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:42.288 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.284+0000 7f6d31b9b640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6d2c06c400 0x7f6d2c15f810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:42.289 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.284+0000 7f6d31b9b640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d2c06c880 con 0x7f6d2c0cbf80 2023-05-22T15:42:42.289 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.284+0000 7f6d31b9b640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d2c06db40 con 0x7f6d2c0b22e0 2023-05-22T15:42:42.289 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.284+0000 7f6d31b9b640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d2c0cf3f0 con 0x7f6d2c06c400 2023-05-22T15:42:42.289 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.284+0000 7f6d2b7fe640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6d2c0cbf80 0x7f6d2c06b210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:42.290 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.284+0000 7f6d2b7fe640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6d2c0cbf80 0x7f6d2c06b210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49238/0 (socket says 172.21.15.16:49238) 2023-05-22T15:42:42.290 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.284+0000 7f6d2b7fe640 1 -- 172.21.15.16:0/3499172124 learned_addr learned my addr 172.21.15.16:0/3499172124 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:42:42.290 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.284+0000 7f6d2affd640 1 --2- 172.21.15.16:0/3499172124 >> v2:172.21.15.62:3300/0 conn(0x7f6d2c0b22e0 0x7f6d2c06bec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:42.290 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.284+0000 7f6d2bfff640 1 --2- 172.21.15.16:0/3499172124 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6d2c06c400 0x7f6d2c15f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:42.291 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.284+0000 7f6d2b7fe640 1 -- 172.21.15.16:0/3499172124 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6d2c06c400 msgr2=0x7f6d2c15f810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:42:42.291 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.284+0000 7f6d2b7fe640 1 --2- 172.21.15.16:0/3499172124 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6d2c06c400 0x7f6d2c15f810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:42.291 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.284+0000 7f6d2b7fe640 1 -- 172.21.15.16:0/3499172124 >> v2:172.21.15.62:3300/0 conn(0x7f6d2c0b22e0 msgr2=0x7f6d2c06bec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:42:42.291 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.284+0000 7f6d2b7fe640 1 --2- 172.21.15.16:0/3499172124 >> v2:172.21.15.62:3300/0 conn(0x7f6d2c0b22e0 0x7f6d2c06bec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:42.292 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.284+0000 7f6d2b7fe640 1 -- 172.21.15.16:0/3499172124 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d2c15c080 con 0x7f6d2c0cbf80 2023-05-22T15:42:42.296 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.284+0000 7f6d2b7fe640 1 --2- 172.21.15.16:0/3499172124 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6d2c0cbf80 0x7f6d2c06b210 secure :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 crypto rx=0x7f6d2c0cec50 tx=0x7f6d1c01c180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4318e48ca2274189 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:42.296 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.284+0000 7f6d2a7fc640 1 -- 172.21.15.16:0/3499172124 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d1c0ab600 con 0x7f6d2c0cbf80 2023-05-22T15:42:42.296 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.284+0000 7f6d2a7fc640 1 -- 172.21.15.16:0/3499172124 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d1c0a7050 con 0x7f6d2c0cbf80 2023-05-22T15:42:42.297 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.284+0000 7f6d2a7fc640 1 -- 172.21.15.16:0/3499172124 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d1c0ab980 con 0x7f6d2c0cbf80 2023-05-22T15:42:42.297 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.288+0000 7f6d31b9b640 1 -- 172.21.15.16:0/3499172124 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6d2c0cbf80 msgr2=0x7f6d2c06b210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:42.297 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.288+0000 7f6d31b9b640 1 --2- 172.21.15.16:0/3499172124 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6d2c0cbf80 0x7f6d2c06b210 secure :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 crypto rx=0x7f6d2c0cec50 tx=0x7f6d1c01c180 comp rx=0 tx=0).stop 2023-05-22T15:42:42.297 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.288+0000 7f6d31b9b640 1 -- 172.21.15.16:0/3499172124 shutdown_connections 2023-05-22T15:42:42.298 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.288+0000 7f6d31b9b640 1 --2- 172.21.15.16:0/3499172124 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6d2c06c400 0x7f6d2c15f810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:42.298 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.288+0000 7f6d31b9b640 1 --2- 172.21.15.16:0/3499172124 >> v2:172.21.15.62:3300/0 conn(0x7f6d2c0b22e0 0x7f6d2c06bec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:42.298 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.288+0000 7f6d31b9b640 1 --2- 172.21.15.16:0/3499172124 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6d2c0cbf80 0x7f6d2c06b210 unknown :-1 s=CLOSED pgs=1212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:42.298 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.288+0000 7f6d31b9b640 1 -- 172.21.15.16:0/3499172124 >> 172.21.15.16:0/3499172124 conn(0x7f6d2c06f560 msgr2=0x7f6d2c06f960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:42.299 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.288+0000 7f6d31b9b640 1 -- 172.21.15.16:0/3499172124 shutdown_connections 2023-05-22T15:42:42.299 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.288+0000 7f6d31b9b640 1 -- 172.21.15.16:0/3499172124 wait complete. 2023-05-22T15:42:42.299 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.288+0000 7f6d31b9b640 1 Processor -- start 2023-05-22T15:42:42.299 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.288+0000 7f6d31b9b640 1 -- start start 2023-05-22T15:42:42.300 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.288+0000 7f6d31b9b640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6d2c06c400 0x7f6d2c0c8b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:42.300 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.288+0000 7f6d31b9b640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6d2c0b22e0 0x7f6d2c0c7330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:42.300 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.288+0000 7f6d31b9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6d2c0cbf80 0x7f6d2c15c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:42.300 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.288+0000 7f6d31b9b640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d2c15d0f0 con 0x7f6d2c0cbf80 2023-05-22T15:42:42.301 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.288+0000 7f6d31b9b640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d2c0c7870 con 0x7f6d2c0b22e0 2023-05-22T15:42:42.301 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.288+0000 7f6d31b9b640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d2c0c7aa0 con 0x7f6d2c06c400 2023-05-22T15:42:42.301 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.288+0000 7f6d2bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6d2c0cbf80 0x7f6d2c15c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:42.302 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.288+0000 7f6d2bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6d2c0cbf80 0x7f6d2c15c4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49242/0 (socket says 172.21.15.16:49242) 2023-05-22T15:42:42.302 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.288+0000 7f6d2bfff640 1 -- 172.21.15.16:0/3440448100 learned_addr learned my addr 172.21.15.16:0/3440448100 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:42:42.302 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.288+0000 7f6d2b7fe640 1 --2- 172.21.15.16:0/3440448100 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6d2c06c400 0x7f6d2c0c8b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:42.302 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.288+0000 7f6d2affd640 1 --2- 172.21.15.16:0/3440448100 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6d2c0b22e0 0x7f6d2c0c7330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:42.303 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.288+0000 7f6d2bfff640 1 -- 172.21.15.16:0/3440448100 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6d2c06c400 msgr2=0x7f6d2c0c8b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:42.303 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.288+0000 7f6d2bfff640 1 --2- 172.21.15.16:0/3440448100 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6d2c06c400 0x7f6d2c0c8b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:42.303 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.288+0000 7f6d2bfff640 1 -- 172.21.15.16:0/3440448100 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6d2c0b22e0 msgr2=0x7f6d2c0c7330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:42.303 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.288+0000 7f6d2bfff640 1 --2- 172.21.15.16:0/3440448100 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6d2c0b22e0 0x7f6d2c0c7330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:42.304 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.288+0000 7f6d2bfff640 1 -- 172.21.15.16:0/3440448100 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d1c0094d0 con 0x7f6d2c0cbf80 2023-05-22T15:42:42.304 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.288+0000 7f6d2bfff640 1 --2- 172.21.15.16:0/3440448100 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6d2c0cbf80 0x7f6d2c15c4f0 secure :-1 s=READY pgs=1213 cs=0 l=1 rev1=1 crypto rx=0x7f6d2400d830 tx=0x7f6d2400dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:42.304 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.288+0000 7f6d13fff640 1 -- 172.21.15.16:0/3440448100 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d2400b6c0 con 0x7f6d2c0cbf80 2023-05-22T15:42:42.304 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.288+0000 7f6d13fff640 1 -- 172.21.15.16:0/3440448100 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d2400b820 con 0x7f6d2c0cbf80 2023-05-22T15:42:42.305 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.288+0000 7f6d13fff640 1 -- 172.21.15.16:0/3440448100 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d2400bbd0 con 0x7f6d2c0cbf80 2023-05-22T15:42:42.305 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.288+0000 7f6d31b9b640 1 -- 172.21.15.16:0/3440448100 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d2c06b750 con 0x7f6d2c0cbf80 2023-05-22T15:42:42.305 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.288+0000 7f6d31b9b640 1 -- 172.21.15.16:0/3440448100 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d2c06e380 con 0x7f6d2c0cbf80 2023-05-22T15:42:42.305 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.288+0000 7f6d31b9b640 1 -- 172.21.15.16:0/3440448100 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d2c0cc350 con 0x7f6d2c0cbf80 2023-05-22T15:42:42.306 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.292+0000 7f6d13fff640 1 -- 172.21.15.16:0/3440448100 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 32) v1 ==== 83941+0+0 (secure 0 0 0) 0x7f6d2401e460 con 0x7f6d2c0cbf80 2023-05-22T15:42:42.306 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.292+0000 7f6d13fff640 1 -- 172.21.15.16:0/3440448100 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 7027+0+0 (secure 0 0 0) 0x7f6d24082e80 con 0x7f6d2c0cbf80 2023-05-22T15:42:42.306 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.296+0000 7f6d13fff640 1 -- 172.21.15.16:0/3440448100 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f6d24050b20 con 0x7f6d2c0cbf80 2023-05-22T15:42:42.473 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.468+0000 7f6d13fff640 1 -- 172.21.15.16:0/3440448100 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mgrmap(e 33) v1 ==== 84068+0+0 (secure 0 0 0) 0x7f6d240505d0 con 0x7f6d2c0cbf80 2023-05-22T15:42:42.474 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.472+0000 7f6d13fff640 1 --2- 172.21.15.16:0/3440448100 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f6d1806b680 0x7f6d1806da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:42.477 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.472+0000 7f6d2b7fe640 1 --2- 172.21.15.16:0/3440448100 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f6d1806b680 0x7f6d1806da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:42.480 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.476+0000 7f6d2b7fe640 1 --2- 172.21.15.16:0/3440448100 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f6d1806b680 0x7f6d1806da60 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f6d1c0ad000 tx=0x7f6d1c008040 comp rx=0 tx=0).ready entity=mgr.6436 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:42.492 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.488+0000 7f6d31b9b640 1 -- 172.21.15.16:0/3440448100 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6d2c04dbe0 con 0x7f6d2c0cbf80 2023-05-22T15:42:42.492 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.488+0000 7f6d13fff640 1 -- 172.21.15.16:0/3440448100 <== mon.0 v2:172.21.15.16:3300/0 8 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+4647 (secure 0 0 0) 0x7f6d2407d2c0 con 0x7f6d2c0cbf80 2023-05-22T15:42:42.493 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:42:42.493 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":30,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6833/428835470","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":428835470},{"type":"v1","addr":"172.21.15.62:6833","nonce":428835470}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":25},{"gid":6238,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/4010598958","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":4010598958},{"type":"v1","addr":"172.21.15.62:6835","nonce":4010598958}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":25},{"gid":7008,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/2240458030","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6835","nonce":2240458030},{"type":"v1","addr":"172.21.15.16:6837","nonce":2240458030}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":25}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-05-22T15:42:22.811659+0000","modified":"2023-05-22T15:42:25.830716+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7011},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7011":{"gid":7011,"name":"a","rank":0,"incarnation":26,"state":"up:active","state_seq":3,"addr":"172.21.15.16:6836/1772031282","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1772031282},{"type":"v1","addr":"172.21.15.16:6836","nonce":1772031282}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2023-05-22T15:42:42.493 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 30 2023-05-22T15:42:42.495 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.492+0000 7f6d31b9b640 1 -- 172.21.15.16:0/3440448100 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f6d1806b680 msgr2=0x7f6d1806da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:42.495 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.492+0000 7f6d31b9b640 1 --2- 172.21.15.16:0/3440448100 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f6d1806b680 0x7f6d1806da60 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f6d1c0ad000 tx=0x7f6d1c008040 comp rx=0 tx=0).stop 2023-05-22T15:42:42.495 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.492+0000 7f6d31b9b640 1 -- 172.21.15.16:0/3440448100 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6d2c0cbf80 msgr2=0x7f6d2c15c4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:42.495 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.492+0000 7f6d31b9b640 1 --2- 172.21.15.16:0/3440448100 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6d2c0cbf80 0x7f6d2c15c4f0 secure :-1 s=READY pgs=1213 cs=0 l=1 rev1=1 crypto rx=0x7f6d2400d830 tx=0x7f6d2400dcf0 comp rx=0 tx=0).stop 2023-05-22T15:42:42.495 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.492+0000 7f6d31b9b640 1 -- 172.21.15.16:0/3440448100 shutdown_connections 2023-05-22T15:42:42.495 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.492+0000 7f6d31b9b640 1 --2- 172.21.15.16:0/3440448100 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6d2c0cbf80 0x7f6d2c15c4f0 unknown :-1 s=CLOSED pgs=1213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:42.496 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.492+0000 7f6d31b9b640 1 --2- 172.21.15.16:0/3440448100 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6d2c0b22e0 0x7f6d2c0c7330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:42.496 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.492+0000 7f6d31b9b640 1 --2- 172.21.15.16:0/3440448100 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f6d1806b680 0x7f6d1806da60 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:42.496 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.492+0000 7f6d31b9b640 1 --2- 172.21.15.16:0/3440448100 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6d2c06c400 0x7f6d2c0c8b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:42.496 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.492+0000 7f6d31b9b640 1 -- 172.21.15.16:0/3440448100 >> 172.21.15.16:0/3440448100 conn(0x7f6d2c06f560 msgr2=0x7f6d2c0b07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:42.496 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.492+0000 7f6d31b9b640 1 -- 172.21.15.16:0/3440448100 shutdown_connections 2023-05-22T15:42:42.496 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.492+0000 7f6d31b9b640 1 -- 172.21.15.16:0/3440448100 wait complete. 2023-05-22T15:42:42.511 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-05-22T15:42:42.629 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.624+0000 7fa7cc5b9640 1 Processor -- start 2023-05-22T15:42:42.629 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.624+0000 7fa7cc5b9640 1 -- start start 2023-05-22T15:42:42.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.628+0000 7fa7cc5b9640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fa7c4159cd0 0x7fa7c415a0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:42.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.628+0000 7fa7cc5b9640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa7c40cc6f0 0x7fa7c40cfe30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:42.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.628+0000 7fa7cc5b9640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7c40cf9a0 con 0x7fa7c40b2880 2023-05-22T15:42:42.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.628+0000 7fa7cc5b9640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7c415aba0 con 0x7fa7c4159cd0 2023-05-22T15:42:42.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.628+0000 7fa7cc5b9640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7c415ae80 con 0x7fa7c40cc6f0 2023-05-22T15:42:42.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.628+0000 7fa7ca355640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fa7c4159cd0 0x7fa7c415a0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:42.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.628+0000 7fa7ca355640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fa7c4159cd0 0x7fa7c415a0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49250/0 (socket says 172.21.15.16:49250) 2023-05-22T15:42:42.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.628+0000 7fa7ca355640 1 -- 172.21.15.16:0/2056149432 learned_addr learned my addr 172.21.15.16:0/2056149432 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:42:42.632 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.628+0000 7fa7cab56640 1 --2- 172.21.15.16:0/2056149432 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa7c40cc6f0 0x7fa7c40cfe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:42.632 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.628+0000 7fa7ca355640 1 -- 172.21.15.16:0/2056149432 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa7c40cc6f0 msgr2=0x7fa7c40cfe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:42:42.632 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.628+0000 7fa7ca355640 1 --2- 172.21.15.16:0/2056149432 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa7c40cc6f0 0x7fa7c40cfe30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:42.633 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.628+0000 7fa7ca355640 1 -- 172.21.15.16:0/2056149432 >> v1:172.21.15.62:6789/0 conn(0x7fa7c40b2880 legacy=0x7fa7c40cbfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:42:42.633 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.628+0000 7fa7ca355640 1 -- 172.21.15.16:0/2056149432 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7c415b160 con 0x7fa7c4159cd0 2023-05-22T15:42:42.633 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.628+0000 7fa7ca355640 1 --2- 172.21.15.16:0/2056149432 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa7c4159cd0 0x7fa7c415a0a0 secure :-1 s=READY pgs=1214 cs=0 l=1 rev1=1 crypto rx=0x7fa7c0009870 tx=0x7fa7c001c510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f3b7150b22bf893 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:42.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.628+0000 7fa7c9353640 1 -- 172.21.15.16:0/2056149432 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7c00ab620 con 0x7fa7c4159cd0 2023-05-22T15:42:42.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.628+0000 7fa7c9353640 1 -- 172.21.15.16:0/2056149432 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7c00ab780 con 0x7fa7c4159cd0 2023-05-22T15:42:42.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.628+0000 7fa7c9353640 1 -- 172.21.15.16:0/2056149432 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7c00abb10 con 0x7fa7c4159cd0 2023-05-22T15:42:42.635 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.632+0000 7fa7cc5b9640 1 -- 172.21.15.16:0/2056149432 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa7c4159cd0 msgr2=0x7fa7c415a0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:42.635 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.632+0000 7fa7cc5b9640 1 --2- 172.21.15.16:0/2056149432 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa7c4159cd0 0x7fa7c415a0a0 secure :-1 s=READY pgs=1214 cs=0 l=1 rev1=1 crypto rx=0x7fa7c0009870 tx=0x7fa7c001c510 comp rx=0 tx=0).stop 2023-05-22T15:42:42.635 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.632+0000 7fa7cc5b9640 1 -- 172.21.15.16:0/2056149432 shutdown_connections 2023-05-22T15:42:42.636 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.632+0000 7fa7cc5b9640 1 --2- 172.21.15.16:0/2056149432 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa7c40cc6f0 0x7fa7c40cfe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:42.636 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.632+0000 7fa7cc5b9640 1 --2- 172.21.15.16:0/2056149432 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa7c4159cd0 0x7fa7c415a0a0 unknown :-1 s=CLOSED pgs=1214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:42.636 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.632+0000 7fa7cc5b9640 1 -- 172.21.15.16:0/2056149432 >> 172.21.15.16:0/2056149432 conn(0x7fa7c406f5c0 msgr2=0x7fa7c406f9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:42.636 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.632+0000 7fa7cc5b9640 1 -- 172.21.15.16:0/2056149432 shutdown_connections 2023-05-22T15:42:42.636 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.632+0000 7fa7cc5b9640 1 -- 172.21.15.16:0/2056149432 wait complete. 2023-05-22T15:42:42.638 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.632+0000 7fa7cc5b9640 1 Processor -- start 2023-05-22T15:42:42.638 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.632+0000 7fa7cc5b9640 1 -- start start 2023-05-22T15:42:42.638 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.632+0000 7fa7cc5b9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa7c40b2880 0x7fa7c41539f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:42.638 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.632+0000 7fa7cc5b9640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa7c40cc6f0 0x7fa7c41476f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:42.638 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.632+0000 7fa7cc5b9640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa7c4159cd0 0x7fa7c40b3cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:42.638 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.632+0000 7fa7cc5b9640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7c414a3c0 con 0x7fa7c40b2880 2023-05-22T15:42:42.639 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.632+0000 7fa7cc5b9640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7c40b3a30 con 0x7fa7c4159cd0 2023-05-22T15:42:42.640 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.632+0000 7fa7cc5b9640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7c4149e00 con 0x7fa7c40cc6f0 2023-05-22T15:42:42.640 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.632+0000 7fa7ca355640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa7c40b2880 0x7fa7c41539f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:42.640 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.632+0000 7fa7ca355640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa7c40b2880 0x7fa7c41539f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49262/0 (socket says 172.21.15.16:49262) 2023-05-22T15:42:42.640 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.632+0000 7fa7ca355640 1 -- 172.21.15.16:0/3775828827 learned_addr learned my addr 172.21.15.16:0/3775828827 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:42:42.640 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.632+0000 7fa7cab56640 1 --2- 172.21.15.16:0/3775828827 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa7c4159cd0 0x7fa7c40b3cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:42.641 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.632+0000 7fa7c9b54640 1 --2- 172.21.15.16:0/3775828827 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa7c40cc6f0 0x7fa7c41476f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:42.641 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.632+0000 7fa7ca355640 1 -- 172.21.15.16:0/3775828827 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa7c40cc6f0 msgr2=0x7fa7c41476f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:42.641 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.632+0000 7fa7ca355640 1 --2- 172.21.15.16:0/3775828827 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa7c40cc6f0 0x7fa7c41476f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:42.641 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.632+0000 7fa7ca355640 1 -- 172.21.15.16:0/3775828827 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa7c4159cd0 msgr2=0x7fa7c40b3cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:42.641 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.632+0000 7fa7ca355640 1 --2- 172.21.15.16:0/3775828827 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa7c4159cd0 0x7fa7c40b3cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:42.641 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.632+0000 7fa7ca355640 1 -- 172.21.15.16:0/3775828827 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7c00094d0 con 0x7fa7c40b2880 2023-05-22T15:42:42.641 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.632+0000 7fa7ca355640 1 --2- 172.21.15.16:0/3775828827 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa7c40b2880 0x7fa7c41539f0 secure :-1 s=READY pgs=1215 cs=0 l=1 rev1=1 crypto rx=0x7fa7c00ad040 tx=0x7fa7c001ad50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:42.642 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.632+0000 7fa7baffd640 1 -- 172.21.15.16:0/3775828827 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7c00be5f0 con 0x7fa7c40b2880 2023-05-22T15:42:42.642 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.632+0000 7fa7baffd640 1 -- 172.21.15.16:0/3775828827 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7c00be770 con 0x7fa7c40b2880 2023-05-22T15:42:42.642 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.632+0000 7fa7cc5b9640 1 -- 172.21.15.16:0/3775828827 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7c40d0d10 con 0x7fa7c40b2880 2023-05-22T15:42:42.642 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.632+0000 7fa7baffd640 1 -- 172.21.15.16:0/3775828827 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7c00beb20 con 0x7fa7c40b2880 2023-05-22T15:42:42.642 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.632+0000 7fa7cc5b9640 1 -- 172.21.15.16:0/3775828827 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7c40d0b80 con 0x7fa7c40b2880 2023-05-22T15:42:42.643 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.636+0000 7fa7cc5b9640 1 -- 172.21.15.16:0/3775828827 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7c40d0ea0 con 0x7fa7c40b2880 2023-05-22T15:42:42.643 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.636+0000 7fa7baffd640 1 -- 172.21.15.16:0/3775828827 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 33) v1 ==== 84068+0+0 (secure 0 0 0) 0x7fa7c00b43f0 con 0x7fa7c40b2880 2023-05-22T15:42:42.643 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.636+0000 7fa7baffd640 1 --2- 172.21.15.16:0/3775828827 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fa7a8065200 0x7fa7a80676b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:42.643 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.636+0000 7fa7baffd640 1 -- 172.21.15.16:0/3775828827 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 7027+0+0 (secure 0 0 0) 0x7fa7c011d460 con 0x7fa7c40b2880 2023-05-22T15:42:42.644 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.636+0000 7fa7c9b54640 1 --2- 172.21.15.16:0/3775828827 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fa7a8065200 0x7fa7a80676b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:42.644 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.636+0000 7fa7c9b54640 1 --2- 172.21.15.16:0/3775828827 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fa7a8065200 0x7fa7a80676b0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fa7c4153f30 tx=0x7fa7b4009040 comp rx=0 tx=0).ready entity=mgr.6436 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:42.644 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.640+0000 7fa7baffd640 1 -- 172.21.15.16:0/3775828827 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fa7c00eb0a0 con 0x7fa7c40b2880 2023-05-22T15:42:42.797 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.792+0000 7fa7cc5b9640 1 -- 172.21.15.16:0/3775828827 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa7c404dbe0 con 0x7fa7c40b2880 2023-05-22T15:42:42.800 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.796+0000 7fa7baffd640 1 -- 172.21.15.16:0/3775828827 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v92) v1 ==== 74+0+18961 (secure 0 0 0) 0x7fa7c00ee6c0 con 0x7fa7c40b2880 2023-05-22T15:42:42.800 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:42:42.801 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":92,"fsid":"b2719834-7847-4d98-90a6-f65d57f85c49","created":"2023-05-22T15:26:25.259314+0000","modified":"2023-05-22T15:42:41.449592+0000","last_up_change":"2023-05-22T15:26:36.448524+0000","last_in_change":"2023-05-22T15:26:27.675668+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-05-22T15:26:38.019781+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-05-22T15:26:43.570514+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2023-05-22T15:42:20.730757+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2023-05-22T15:42:21.805016+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9e7ff854-b25f-4cbb-a5c2-1799d9c53d2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":37021},{"type":"v1","addr":"172.21.15.16:6825","nonce":37021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":37021},{"type":"v1","addr":"172.21.15.16:6827","nonce":37021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":37021},{"type":"v1","addr":"172.21.15.16:6831","nonce":37021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":37021},{"type":"v1","addr":"172.21.15.16:6829","nonce":37021}]},"public_addr":"172.21.15.16:6825/37021","cluster_addr":"172.21.15.16:6827/37021","heartbeat_back_addr":"172.21.15.16:6831/37021","heartbeat_front_addr":"172.21.15.16:6829/37021","state":["exists","up"]},{"osd":1,"uuid":"4702fbc9-a927-4565-aca5-41dc69cc8264","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":37008},{"type":"v1","addr":"172.21.15.16:6801","nonce":37008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":37008},{"type":"v1","addr":"172.21.15.16:6803","nonce":37008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":37008},{"type":"v1","addr":"172.21.15.16:6807","nonce":37008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":37008},{"type":"v1","addr":"172.21.15.16:6805","nonce":37008}]},"public_addr":"172.21.15.16:6801/37008","cluster_addr":"172.21.15.16:6803/37008","heartbeat_back_addr":"172.21.15.16:6807/37008","heartbeat_front_addr":"172.21.15.16:6805/37008","state":["exists","up"]},{"osd":2,"uuid":"1e7bba46-4eba-47ef-bb81-5e2d371638ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":37009},{"type":"v1","addr":"172.21.15.16:6809","nonce":37009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":37009},{"type":"v1","addr":"172.21.15.16:6811","nonce":37009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":37009},{"type":"v1","addr":"172.21.15.16:6815","nonce":37009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":37009},{"type":"v1","addr":"172.21.15.16:6813","nonce":37009}]},"public_addr":"172.21.15.16:6809/37009","cluster_addr":"172.21.15.16:6811/37009","heartbeat_back_addr":"172.21.15.16:6815/37009","heartbeat_front_addr":"172.21.15.16:6813/37009","state":["exists","up"]},{"osd":3,"uuid":"383cacc0-5a88-4904-990e-498ca59ef48a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":37020},{"type":"v1","addr":"172.21.15.16:6817","nonce":37020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":37020},{"type":"v1","addr":"172.21.15.16:6819","nonce":37020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":37020},{"type":"v1","addr":"172.21.15.16:6823","nonce":37020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":37020},{"type":"v1","addr":"172.21.15.16:6821","nonce":37020}]},"public_addr":"172.21.15.16:6817/37020","cluster_addr":"172.21.15.16:6819/37020","heartbeat_back_addr":"172.21.15.16:6823/37020","heartbeat_front_addr":"172.21.15.16:6821/37020","state":["exists","up"]},{"osd":4,"uuid":"970500ef-95b7-42c1-b080-eed4fae7013d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":37353},{"type":"v1","addr":"172.21.15.62:6817","nonce":37353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6819","nonce":37353},{"type":"v1","addr":"172.21.15.62:6822","nonce":37353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37353},{"type":"v1","addr":"172.21.15.62:6831","nonce":37353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37353},{"type":"v1","addr":"172.21.15.62:6826","nonce":37353}]},"public_addr":"172.21.15.62:6817/37353","cluster_addr":"172.21.15.62:6822/37353","heartbeat_back_addr":"172.21.15.62:6831/37353","heartbeat_front_addr":"172.21.15.62:6826/37353","state":["exists","up"]},{"osd":5,"uuid":"022d67db-d4d8-445b-b1b9-8b446a00f5c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":37352},{"type":"v1","addr":"172.21.15.62:6801","nonce":37352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":37352},{"type":"v1","addr":"172.21.15.62:6803","nonce":37352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":37352},{"type":"v1","addr":"172.21.15.62:6807","nonce":37352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":37352},{"type":"v1","addr":"172.21.15.62:6805","nonce":37352}]},"public_addr":"172.21.15.62:6801/37352","cluster_addr":"172.21.15.62:6803/37352","heartbeat_back_addr":"172.21.15.62:6807/37352","heartbeat_front_addr":"172.21.15.62:6805/37352","state":["exists","up"]},{"osd":6,"uuid":"42bd372b-6688-4a1b-8407-e16e8c266e33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":37354},{"type":"v1","addr":"172.21.15.62:6820","nonce":37354}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":37354},{"type":"v1","addr":"172.21.15.62:6823","nonce":37354}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6829","nonce":37354},{"type":"v1","addr":"172.21.15.62:6830","nonce":37354}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6825","nonce":37354},{"type":"v1","addr":"172.21.15.62:6827","nonce":37354}]},"public_addr":"172.21.15.62:6820/37354","cluster_addr":"172.21.15.62:6823/37354","heartbeat_back_addr":"172.21.15.62:6830/37354","heartbeat_front_addr":"172.21.15.62:6827/37354","state":["exists","up"]},{"osd":7,"uuid":"ac70b2a3-f41b-4bf2-b227-7f2cbb55426d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37355},{"type":"v1","addr":"172.21.15.62:6809","nonce":37355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37355},{"type":"v1","addr":"172.21.15.62:6811","nonce":37355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":37355},{"type":"v1","addr":"172.21.15.62:6815","nonce":37355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":37355},{"type":"v1","addr":"172.21.15.62:6813","nonce":37355}]},"public_addr":"172.21.15.62:6809/37355","cluster_addr":"172.21.15.62:6811/37355","heartbeat_back_addr":"172.21.15.62:6815/37355","heartbeat_front_addr":"172.21.15.62:6813/37355","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.368896+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.107273+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.276888+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.287502+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.124791+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.251937+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.255679+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.232797+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:0/319876331":"2023-05-23T15:42:41.449418+0000","172.21.15.16:0/2036981971":"2023-05-23T15:42:41.449418+0000","172.21.15.16:0/2576009949":"2023-05-23T15:42:41.449418+0000","172.21.15.16:0/79429311":"2023-05-23T15:42:41.449418+0000","172.21.15.16:0/1473562727":"2023-05-23T15:42:41.449418+0000","172.21.15.16:0/3510757085":"2023-05-23T15:42:35.895737+0000","172.21.15.16:0/4119485618":"2023-05-23T15:42:35.895737+0000","172.21.15.16:0/2522450675":"2023-05-23T15:42:41.449418+0000","172.21.15.16:0/4196662600":"2023-05-23T15:42:41.449418+0000","172.21.15.16:0/2050049957":"2023-05-23T15:42:35.895737+0000","172.21.15.16:0/1396723034":"2023-05-23T15:42:35.895737+0000","172.21.15.16:0/3754867799":"2023-05-23T15:42:35.895737+0000","172.21.15.16:6833/36921":"2023-05-23T15:42:35.895737+0000","172.21.15.16:6832/36921":"2023-05-23T15:42:35.895737+0000","172.21.15.16:0/3710518307":"2023-05-23T15:42:35.895737+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-05-22T15:42:42.802 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.796+0000 7fa7cc5b9640 1 -- 172.21.15.16:0/3775828827 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fa7a8065200 msgr2=0x7fa7a80676b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:42.802 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.796+0000 7fa7cc5b9640 1 --2- 172.21.15.16:0/3775828827 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fa7a8065200 0x7fa7a80676b0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fa7c4153f30 tx=0x7fa7b4009040 comp rx=0 tx=0).stop 2023-05-22T15:42:42.802 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.796+0000 7fa7cc5b9640 1 -- 172.21.15.16:0/3775828827 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa7c40b2880 msgr2=0x7fa7c41539f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:42.802 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.796+0000 7fa7cc5b9640 1 --2- 172.21.15.16:0/3775828827 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa7c40b2880 0x7fa7c41539f0 secure :-1 s=READY pgs=1215 cs=0 l=1 rev1=1 crypto rx=0x7fa7c00ad040 tx=0x7fa7c001ad50 comp rx=0 tx=0).stop 2023-05-22T15:42:42.803 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.796+0000 7fa7cc5b9640 1 -- 172.21.15.16:0/3775828827 shutdown_connections 2023-05-22T15:42:42.803 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.796+0000 7fa7cc5b9640 1 --2- 172.21.15.16:0/3775828827 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa7c4159cd0 0x7fa7c40b3cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:42.803 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.796+0000 7fa7cc5b9640 1 --2- 172.21.15.16:0/3775828827 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fa7a8065200 0x7fa7a80676b0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:42.803 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.796+0000 7fa7cc5b9640 1 --2- 172.21.15.16:0/3775828827 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa7c40cc6f0 0x7fa7c41476f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:42.803 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.796+0000 7fa7cc5b9640 1 --2- 172.21.15.16:0/3775828827 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa7c40b2880 0x7fa7c41539f0 unknown :-1 s=CLOSED pgs=1215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:42.803 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.796+0000 7fa7cc5b9640 1 -- 172.21.15.16:0/3775828827 >> 172.21.15.16:0/3775828827 conn(0x7fa7c406f5c0 msgr2=0x7fa7c4158e80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:42.803 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.796+0000 7fa7cc5b9640 1 -- 172.21.15.16:0/3775828827 shutdown_connections 2023-05-22T15:42:42.804 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.800+0000 7fa7cc5b9640 1 -- 172.21.15.16:0/3775828827 wait complete. 2023-05-22T15:42:42.820 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-05-22T15:42:42.820 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-05-22T15:42:42.940 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.936+0000 7f7e913c4640 1 Processor -- start 2023-05-22T15:42:42.940 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.936+0000 7f7e913c4640 1 -- start start 2023-05-22T15:42:42.941 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.936+0000 7f7e913c4640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f7e8c0ca4e0 0x7f7e8c15e460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:42.941 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.936+0000 7f7e913c4640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7e8c070df0 0x7f7e8c071210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:42.942 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.936+0000 7f7e913c4640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e8c0b0650 con 0x7f7e8c0c9b40 2023-05-22T15:42:42.942 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.936+0000 7f7e913c4640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e8c0b0370 con 0x7f7e8c0ca4e0 2023-05-22T15:42:42.942 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.936+0000 7f7e913c4640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e8c15d980 con 0x7f7e8c070df0 2023-05-22T15:42:42.942 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.936+0000 7f7e8a7fc640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f7e8c0c9b40 0x7f7e8c0afb40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:41078/0 (socket says 172.21.15.16:41078) 2023-05-22T15:42:42.942 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.936+0000 7f7e8a7fc640 1 -- 172.21.15.16:0/1244185548 learned_addr learned my addr 172.21.15.16:0/1244185548 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:42:42.944 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.936+0000 7f7e8b7fe640 1 --2- 172.21.15.16:0/1244185548 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7e8c070df0 0x7f7e8c071210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:42.944 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.936+0000 7f7e8affd640 1 --2- 172.21.15.16:0/1244185548 >> v2:172.21.15.62:3300/0 conn(0x7f7e8c0ca4e0 0x7f7e8c15e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:42.944 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.936+0000 7f7e89ffb640 1 -- 172.21.15.16:0/1244185548 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 362864909 0 0) 0x7f7e8c0b0650 con 0x7f7e8c0c9b40 2023-05-22T15:42:42.944 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.936+0000 7f7e89ffb640 1 -- 172.21.15.16:0/1244185548 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e78003530 con 0x7f7e8c0c9b40 2023-05-22T15:42:42.944 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.940+0000 7f7e89ffb640 1 -- 172.21.15.16:0/1244185548 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3930867413 0 0) 0x7f7e78003530 con 0x7f7e8c0c9b40 2023-05-22T15:42:42.945 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.940+0000 7f7e89ffb640 1 -- 172.21.15.16:0/1244185548 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7e8c070df0 msgr2=0x7f7e8c071210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:42:42.945 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.940+0000 7f7e89ffb640 1 --2- 172.21.15.16:0/1244185548 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7e8c070df0 0x7f7e8c071210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:42.945 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.940+0000 7f7e89ffb640 1 -- 172.21.15.16:0/1244185548 >> v2:172.21.15.62:3300/0 conn(0x7f7e8c0ca4e0 msgr2=0x7f7e8c15e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:42:42.945 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.940+0000 7f7e89ffb640 1 --2- 172.21.15.16:0/1244185548 >> v2:172.21.15.62:3300/0 conn(0x7f7e8c0ca4e0 0x7f7e8c15e460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:42.945 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.940+0000 7f7e89ffb640 1 -- 172.21.15.16:0/1244185548 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e8c15dc60 con 0x7f7e8c0c9b40 2023-05-22T15:42:42.945 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.940+0000 7f7e89ffb640 1 -- 172.21.15.16:0/1244185548 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7f7e7c001a20 con 0x7f7e8c0c9b40 2023-05-22T15:42:42.945 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.940+0000 7f7e89ffb640 1 -- 172.21.15.16:0/1244185548 >> v1:172.21.15.16:6789/0 conn(0x7f7e8c0c9b40 legacy=0x7f7e8c0afb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:42.946 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.940+0000 7f7e89ffb640 1 --2- 172.21.15.16:0/1244185548 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7e78003980 0x7f7e78003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:42.946 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.940+0000 7f7e89ffb640 1 --2- 172.21.15.16:0/1244185548 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e780043b0 0x7f7e78004870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:42.946 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.940+0000 7f7e89ffb640 1 --2- 172.21.15.16:0/1244185548 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7e8c070df0 0x7f7e78004f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:42.946 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.940+0000 7f7e89ffb640 1 -- 172.21.15.16:0/1244185548 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e8c0b0370 con 0x7f7e780043b0 2023-05-22T15:42:42.946 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.940+0000 7f7e89ffb640 1 -- 172.21.15.16:0/1244185548 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e8c15d980 con 0x7f7e78003980 2023-05-22T15:42:42.946 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.940+0000 7f7e89ffb640 1 -- 172.21.15.16:0/1244185548 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e7800b990 con 0x7f7e8c070df0 2023-05-22T15:42:42.946 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.940+0000 7f7e8a7fc640 1 --2- 172.21.15.16:0/1244185548 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e780043b0 0x7f7e78004870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:42.946 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.940+0000 7f7e8b7fe640 1 --2- 172.21.15.16:0/1244185548 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7e8c070df0 0x7f7e78004f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:42.947 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.940+0000 7f7e8affd640 1 --2- 172.21.15.16:0/1244185548 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7e78003980 0x7f7e78003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:42.947 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.940+0000 7f7e8a7fc640 1 -- 172.21.15.16:0/1244185548 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7e8c070df0 msgr2=0x7f7e78004f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:42:42.947 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.940+0000 7f7e8a7fc640 1 --2- 172.21.15.16:0/1244185548 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7e8c070df0 0x7f7e78004f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:42.947 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.940+0000 7f7e8a7fc640 1 -- 172.21.15.16:0/1244185548 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7e78003980 msgr2=0x7f7e78003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:42:42.947 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.940+0000 7f7e8a7fc640 1 --2- 172.21.15.16:0/1244185548 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7e78003980 0x7f7e78003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:42.947 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.940+0000 7f7e8a7fc640 1 -- 172.21.15.16:0/1244185548 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e78003530 con 0x7f7e780043b0 2023-05-22T15:42:42.947 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.940+0000 7f7e8affd640 1 --2- 172.21.15.16:0/1244185548 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7e78003980 0x7f7e78003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:42:42.948 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.940+0000 7f7e8a7fc640 1 --2- 172.21.15.16:0/1244185548 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e780043b0 0x7f7e78004870 secure :-1 s=READY pgs=1217 cs=0 l=1 rev1=1 crypto rx=0x7f7e7c00a850 tx=0x7f7e7c01faa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e8523ceb3d00a44 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:42.949 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.944+0000 7f7e89ffb640 1 -- 172.21.15.16:0/1244185548 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e7c0aa6d0 con 0x7f7e780043b0 2023-05-22T15:42:42.949 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.944+0000 7f7e89ffb640 1 -- 172.21.15.16:0/1244185548 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e7c0aa830 con 0x7f7e780043b0 2023-05-22T15:42:42.949 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.944+0000 7f7e89ffb640 1 -- 172.21.15.16:0/1244185548 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e7c0aabe0 con 0x7f7e780043b0 2023-05-22T15:42:42.949 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.944+0000 7f7e913c4640 1 -- 172.21.15.16:0/1244185548 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e780043b0 msgr2=0x7f7e78004870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:42.949 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.944+0000 7f7e913c4640 1 --2- 172.21.15.16:0/1244185548 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e780043b0 0x7f7e78004870 secure :-1 s=READY pgs=1217 cs=0 l=1 rev1=1 crypto rx=0x7f7e7c00a850 tx=0x7f7e7c01faa0 comp rx=0 tx=0).stop 2023-05-22T15:42:42.949 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.944+0000 7f7e8b7fe640 1 -- 172.21.15.16:0/1244185548 reap_dead start 2023-05-22T15:42:42.949 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.944+0000 7f7e913c4640 1 -- 172.21.15.16:0/1244185548 shutdown_connections 2023-05-22T15:42:42.950 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.944+0000 7f7e913c4640 1 -- 172.21.15.16:0/1244185548 >> 172.21.15.16:0/1244185548 conn(0x7f7e8c069720 msgr2=0x7f7e8c069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:42.950 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.944+0000 7f7e913c4640 1 -- 172.21.15.16:0/1244185548 shutdown_connections 2023-05-22T15:42:42.950 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.944+0000 7f7e913c4640 1 -- 172.21.15.16:0/1244185548 wait complete. 2023-05-22T15:42:42.951 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.944+0000 7f7e913c4640 1 Processor -- start 2023-05-22T15:42:42.951 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.944+0000 7f7e913c4640 1 -- start start 2023-05-22T15:42:42.952 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.948+0000 7f7e913c4640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7e780043b0 0x7f7e8c1e4d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:42.952 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.948+0000 7f7e913c4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e8c154ed0 0x7f7e8c0c3c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:42.952 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.948+0000 7f7e913c4640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7e8c154380 0x7f7e8c1547f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:42.952 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.948+0000 7f7e913c4640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e8c0c1c30 con 0x7f7e8c154ed0 2023-05-22T15:42:42.953 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.948+0000 7f7e913c4640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e8c154d30 con 0x7f7e780043b0 2023-05-22T15:42:42.953 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.948+0000 7f7e913c4640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e8c1e5300 con 0x7f7e8c154380 2023-05-22T15:42:42.953 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.948+0000 7f7e8a7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e8c154ed0 0x7f7e8c0c3c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:42.953 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.948+0000 7f7e8a7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e8c154ed0 0x7f7e8c0c3c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49286/0 (socket says 172.21.15.16:49286) 2023-05-22T15:42:42.953 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.948+0000 7f7e8a7fc640 1 -- 172.21.15.16:0/2815116469 learned_addr learned my addr 172.21.15.16:0/2815116469 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:42:42.953 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.948+0000 7f7e8b7fe640 1 --2- 172.21.15.16:0/2815116469 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7e8c154380 0x7f7e8c1547f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:42.954 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.948+0000 7f7e8a7fc640 1 -- 172.21.15.16:0/2815116469 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7e8c154380 msgr2=0x7f7e8c1547f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:42.954 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.948+0000 7f7e8a7fc640 1 --2- 172.21.15.16:0/2815116469 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7e8c154380 0x7f7e8c1547f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:42.954 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.948+0000 7f7e8a7fc640 1 -- 172.21.15.16:0/2815116469 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7e780043b0 msgr2=0x7f7e8c1e4d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:42.954 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.948+0000 7f7e8a7fc640 1 --2- 172.21.15.16:0/2815116469 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7e780043b0 0x7f7e8c1e4d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:42.954 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.948+0000 7f7e8a7fc640 1 -- 172.21.15.16:0/2815116469 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e8c0ca8b0 con 0x7f7e8c154ed0 2023-05-22T15:42:42.954 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.948+0000 7f7e8a7fc640 1 --2- 172.21.15.16:0/2815116469 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e8c154ed0 0x7f7e8c0c3c70 secure :-1 s=READY pgs=1218 cs=0 l=1 rev1=1 crypto rx=0x7f7e7c01f8d0 tx=0x7f7e78005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:42.954 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.948+0000 7f7e837fe640 1 -- 172.21.15.16:0/2815116469 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e7c0b7070 con 0x7f7e8c154ed0 2023-05-22T15:42:42.954 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.948+0000 7f7e837fe640 1 -- 172.21.15.16:0/2815116469 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e7c0aa4d0 con 0x7f7e8c154ed0 2023-05-22T15:42:42.955 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.948+0000 7f7e913c4640 1 -- 172.21.15.16:0/2815116469 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e8c15df40 con 0x7f7e8c154ed0 2023-05-22T15:42:42.955 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.948+0000 7f7e837fe640 1 -- 172.21.15.16:0/2815116469 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e7c0aabc0 con 0x7f7e8c154ed0 2023-05-22T15:42:42.955 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.948+0000 7f7e913c4640 1 -- 172.21.15.16:0/2815116469 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e8c071210 con 0x7f7e8c154ed0 2023-05-22T15:42:42.956 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.948+0000 7f7e913c4640 1 -- 172.21.15.16:0/2815116469 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e8c0c4fc0 con 0x7f7e8c154ed0 2023-05-22T15:42:42.956 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.948+0000 7f7e837fe640 1 -- 172.21.15.16:0/2815116469 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 33) v1 ==== 84068+0+0 (secure 0 0 0) 0x7f7e7c0aad20 con 0x7f7e8c154ed0 2023-05-22T15:42:42.956 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.948+0000 7f7e837fe640 1 --2- 172.21.15.16:0/2815116469 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f7e6c0651d0 0x7f7e6c067680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:42.956 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.948+0000 7f7e837fe640 1 -- 172.21.15.16:0/2815116469 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 7027+0+0 (secure 0 0 0) 0x7f7e7c120140 con 0x7f7e8c154ed0 2023-05-22T15:42:42.956 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.948+0000 7f7e8affd640 1 --2- 172.21.15.16:0/2815116469 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f7e6c0651d0 0x7f7e6c067680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:42.956 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.952+0000 7f7e8affd640 1 --2- 172.21.15.16:0/2815116469 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f7e6c0651d0 0x7f7e6c067680 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f7e74009a80 tx=0x7f7e74008040 comp rx=0 tx=0).ready entity=mgr.6436 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:42.957 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:42.952+0000 7f7e837fe640 1 -- 172.21.15.16:0/2815116469 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f7e7c0f1760 con 0x7f7e8c154ed0 2023-05-22T15:42:43.138 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.132+0000 7f7e913c4640 1 -- 172.21.15.16:0/2815116469 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7e8c04dbe0 con 0x7f7e8c154ed0 2023-05-22T15:42:43.138 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.136+0000 7f7e837fe640 1 -- 172.21.15.16:0/2815116469 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+4647 (secure 0 0 0) 0x7f7e7c0c0090 con 0x7f7e8c154ed0 2023-05-22T15:42:43.139 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:42:43.139 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":30,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6833/428835470","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":428835470},{"type":"v1","addr":"172.21.15.62:6833","nonce":428835470}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":25},{"gid":6238,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/4010598958","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":4010598958},{"type":"v1","addr":"172.21.15.62:6835","nonce":4010598958}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":25},{"gid":7008,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/2240458030","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6835","nonce":2240458030},{"type":"v1","addr":"172.21.15.16:6837","nonce":2240458030}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":25}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-05-22T15:42:22.811659+0000","modified":"2023-05-22T15:42:25.830716+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7011},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7011":{"gid":7011,"name":"a","rank":0,"incarnation":26,"state":"up:active","state_seq":3,"addr":"172.21.15.16:6836/1772031282","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1772031282},{"type":"v1","addr":"172.21.15.16:6836","nonce":1772031282}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2023-05-22T15:42:43.139 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 30 2023-05-22T15:42:43.141 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.136+0000 7f7e913c4640 1 -- 172.21.15.16:0/2815116469 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f7e6c0651d0 msgr2=0x7f7e6c067680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:43.141 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.136+0000 7f7e913c4640 1 --2- 172.21.15.16:0/2815116469 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f7e6c0651d0 0x7f7e6c067680 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f7e74009a80 tx=0x7f7e74008040 comp rx=0 tx=0).stop 2023-05-22T15:42:43.141 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.136+0000 7f7e913c4640 1 -- 172.21.15.16:0/2815116469 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e8c154ed0 msgr2=0x7f7e8c0c3c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:43.141 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.136+0000 7f7e913c4640 1 --2- 172.21.15.16:0/2815116469 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e8c154ed0 0x7f7e8c0c3c70 secure :-1 s=READY pgs=1218 cs=0 l=1 rev1=1 crypto rx=0x7f7e7c01f8d0 tx=0x7f7e78005800 comp rx=0 tx=0).stop 2023-05-22T15:42:43.141 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.136+0000 7f7e913c4640 1 -- 172.21.15.16:0/2815116469 shutdown_connections 2023-05-22T15:42:43.141 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.136+0000 7f7e913c4640 1 --2- 172.21.15.16:0/2815116469 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f7e6c0651d0 0x7f7e6c067680 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:43.141 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.136+0000 7f7e913c4640 1 --2- 172.21.15.16:0/2815116469 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7e8c154380 0x7f7e8c1547f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:43.142 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.136+0000 7f7e913c4640 1 --2- 172.21.15.16:0/2815116469 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e8c154ed0 0x7f7e8c0c3c70 unknown :-1 s=CLOSED pgs=1218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:43.142 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.136+0000 7f7e913c4640 1 --2- 172.21.15.16:0/2815116469 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7e780043b0 0x7f7e8c1e4d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:43.142 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.136+0000 7f7e913c4640 1 -- 172.21.15.16:0/2815116469 >> 172.21.15.16:0/2815116469 conn(0x7f7e8c069720 msgr2=0x7f7e8c15a870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:43.142 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.136+0000 7f7e913c4640 1 -- 172.21.15.16:0/2815116469 shutdown_connections 2023-05-22T15:42:43.142 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.136+0000 7f7e913c4640 1 -- 172.21.15.16:0/2815116469 wait complete. 2023-05-22T15:42:43.156 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-05-22T15:42:43.266 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.264+0000 7eff6aa07640 1 Processor -- start 2023-05-22T15:42:43.267 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.264+0000 7eff6aa07640 1 -- start start 2023-05-22T15:42:43.267 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.264+0000 7eff6aa07640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7eff64070fb0 0x7eff640cd1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:43.267 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.264+0000 7eff6aa07640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7eff640affa0 0x7eff64160400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:43.268 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.264+0000 7eff6aa07640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff64160940 con 0x7eff640afb40 2023-05-22T15:42:43.268 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.264+0000 7eff6aa07640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff640cd8d0 con 0x7eff64070fb0 2023-05-22T15:42:43.268 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.264+0000 7eff6aa07640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff64164de0 con 0x7eff640affa0 2023-05-22T15:42:43.269 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.264+0000 7eff63fff640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7eff64070fb0 0x7eff640cd1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:43.269 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.264+0000 7eff68fa4640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7eff640affa0 0x7eff64160400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:43.269 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.264+0000 7eff637fe640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7eff640afb40 0x7eff6415fcf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.16:52226/0 (socket says 172.21.15.16:52226) 2023-05-22T15:42:43.269 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.264+0000 7eff637fe640 1 -- 172.21.15.16:0/2504273288 learned_addr learned my addr 172.21.15.16:0/2504273288 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:42:43.269 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.264+0000 7eff63fff640 1 --2- 172.21.15.16:0/2504273288 >> v2:172.21.15.62:3300/0 conn(0x7eff64070fb0 0x7eff640cd1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.16:43030/0 (socket says 172.21.15.16:43030) 2023-05-22T15:42:43.269 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.264+0000 7eff63fff640 1 -- 172.21.15.16:0/2504273288 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7eff640affa0 msgr2=0x7eff64160400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:42:43.270 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.264+0000 7eff63fff640 1 --2- 172.21.15.16:0/2504273288 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7eff640affa0 0x7eff64160400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:43.270 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.264+0000 7eff63fff640 1 -- 172.21.15.16:0/2504273288 >> v1:172.21.15.62:6789/0 conn(0x7eff640afb40 legacy=0x7eff6415fcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:43.270 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.264+0000 7eff63fff640 1 -- 172.21.15.16:0/2504273288 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff640ca400 con 0x7eff64070fb0 2023-05-22T15:42:43.272 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.268+0000 7eff63fff640 1 --2- 172.21.15.16:0/2504273288 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7eff64070fb0 0x7eff640cd1f0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7eff580097f0 tx=0x7eff5801c480 comp rx=0 tx=0).ready entity=mon.1 client_cookie=84b27e97b26271f3 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:43.272 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.268+0000 7eff62ffd640 1 -- 172.21.15.16:0/2504273288 <== mon.1 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff58004230 con 0x7eff64070fb0 2023-05-22T15:42:43.272 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.268+0000 7eff62ffd640 1 -- 172.21.15.16:0/2504273288 <== mon.1 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff580a7050 con 0x7eff64070fb0 2023-05-22T15:42:43.272 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.268+0000 7eff62ffd640 1 -- 172.21.15.16:0/2504273288 <== mon.1 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff58003620 con 0x7eff64070fb0 2023-05-22T15:42:43.273 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.268+0000 7eff6aa07640 1 -- 172.21.15.16:0/2504273288 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7eff64070fb0 msgr2=0x7eff640cd1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:43.274 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.268+0000 7eff6aa07640 1 --2- 172.21.15.16:0/2504273288 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7eff64070fb0 0x7eff640cd1f0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7eff580097f0 tx=0x7eff5801c480 comp rx=0 tx=0).stop 2023-05-22T15:42:43.274 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.268+0000 7eff6aa07640 1 -- 172.21.15.16:0/2504273288 shutdown_connections 2023-05-22T15:42:43.274 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.268+0000 7eff6aa07640 1 --2- 172.21.15.16:0/2504273288 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7eff640affa0 0x7eff64160400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:43.274 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.268+0000 7eff6aa07640 1 --2- 172.21.15.16:0/2504273288 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7eff64070fb0 0x7eff640cd1f0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:43.274 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.268+0000 7eff6aa07640 1 -- 172.21.15.16:0/2504273288 >> 172.21.15.16:0/2504273288 conn(0x7eff64069720 msgr2=0x7eff64069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:43.274 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.268+0000 7eff6aa07640 1 -- 172.21.15.16:0/2504273288 shutdown_connections 2023-05-22T15:42:43.274 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.268+0000 7eff6aa07640 1 -- 172.21.15.16:0/2504273288 wait complete. 2023-05-22T15:42:43.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.272+0000 7eff6aa07640 1 Processor -- start 2023-05-22T15:42:43.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.272+0000 7eff6aa07640 1 -- start start 2023-05-22T15:42:43.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.272+0000 7eff6aa07640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7eff64070fb0 0x7eff64154b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:43.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.272+0000 7eff6aa07640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7eff640afb40 0x7eff641602d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:43.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.272+0000 7eff6aa07640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7eff640affa0 0x7eff640c1b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:43.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.272+0000 7eff6aa07640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff640c22a0 con 0x7eff64070fb0 2023-05-22T15:42:43.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.272+0000 7eff6aa07640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff64159420 con 0x7eff640affa0 2023-05-22T15:42:43.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.272+0000 7eff6aa07640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff640c20b0 con 0x7eff640afb40 2023-05-22T15:42:43.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.272+0000 7eff63fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7eff64070fb0 0x7eff64154b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:43.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.272+0000 7eff63fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7eff64070fb0 0x7eff64154b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49292/0 (socket says 172.21.15.16:49292) 2023-05-22T15:42:43.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.272+0000 7eff63fff640 1 -- 172.21.15.16:0/712957471 learned_addr learned my addr 172.21.15.16:0/712957471 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:42:43.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.272+0000 7eff637fe640 1 --2- 172.21.15.16:0/712957471 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7eff640afb40 0x7eff641602d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:43.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.272+0000 7eff68fa4640 1 --2- 172.21.15.16:0/712957471 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7eff640affa0 0x7eff640c1b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:43.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.272+0000 7eff63fff640 1 -- 172.21.15.16:0/712957471 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7eff640afb40 msgr2=0x7eff641602d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:43.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.272+0000 7eff63fff640 1 --2- 172.21.15.16:0/712957471 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7eff640afb40 0x7eff641602d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:43.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.272+0000 7eff63fff640 1 -- 172.21.15.16:0/712957471 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7eff640affa0 msgr2=0x7eff640c1b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:43.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.272+0000 7eff63fff640 1 --2- 172.21.15.16:0/712957471 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7eff640affa0 0x7eff640c1b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:43.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.272+0000 7eff63fff640 1 -- 172.21.15.16:0/712957471 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff58009420 con 0x7eff64070fb0 2023-05-22T15:42:43.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.272+0000 7eff68fa4640 1 --2- 172.21.15.16:0/712957471 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7eff640affa0 0x7eff640c1b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:42:43.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.272+0000 7eff63fff640 1 --2- 172.21.15.16:0/712957471 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7eff64070fb0 0x7eff64154b60 secure :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 crypto rx=0x7eff5801a0b0 tx=0x7eff58019220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:43.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.272+0000 7eff60ff9640 1 -- 172.21.15.16:0/712957471 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff58003b30 con 0x7eff64070fb0 2023-05-22T15:42:43.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.272+0000 7eff60ff9640 1 -- 172.21.15.16:0/712957471 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff58003c90 con 0x7eff64070fb0 2023-05-22T15:42:43.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.272+0000 7eff6aa07640 1 -- 172.21.15.16:0/712957471 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff640ca040 con 0x7eff64070fb0 2023-05-22T15:42:43.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.272+0000 7eff60ff9640 1 -- 172.21.15.16:0/712957471 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff580ab440 con 0x7eff64070fb0 2023-05-22T15:42:43.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.272+0000 7eff6aa07640 1 -- 172.21.15.16:0/712957471 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff640b03a0 con 0x7eff64070fb0 2023-05-22T15:42:43.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.272+0000 7eff6aa07640 1 -- 172.21.15.16:0/712957471 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff640ca1d0 con 0x7eff64070fb0 2023-05-22T15:42:43.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.272+0000 7eff60ff9640 1 -- 172.21.15.16:0/712957471 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 33) v1 ==== 84068+0+0 (secure 0 0 0) 0x7eff580b2050 con 0x7eff64070fb0 2023-05-22T15:42:43.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.272+0000 7eff60ff9640 1 --2- 172.21.15.16:0/712957471 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7eff500651d0 0x7eff50067680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:43.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.272+0000 7eff60ff9640 1 -- 172.21.15.16:0/712957471 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 7027+0+0 (secure 0 0 0) 0x7eff5811ca70 con 0x7eff64070fb0 2023-05-22T15:42:43.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.272+0000 7eff637fe640 1 --2- 172.21.15.16:0/712957471 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7eff500651d0 0x7eff50067680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:43.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.276+0000 7eff637fe640 1 --2- 172.21.15.16:0/712957471 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7eff500651d0 0x7eff50067680 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7eff54000f70 tx=0x7eff54009040 comp rx=0 tx=0).ready entity=mgr.6436 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:43.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.276+0000 7eff60ff9640 1 -- 172.21.15.16:0/712957471 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7eff580ea660 con 0x7eff64070fb0 2023-05-22T15:42:43.465 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.460+0000 7eff6aa07640 1 -- 172.21.15.16:0/712957471 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7eff640b2910 con 0x7eff64070fb0 2023-05-22T15:42:43.470 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.464+0000 7eff60ff9640 1 -- 172.21.15.16:0/712957471 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+4647 (secure 0 0 0) 0x7eff580edc80 con 0x7eff64070fb0 2023-05-22T15:42:43.470 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:42:43.470 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":30,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6833/428835470","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":428835470},{"type":"v1","addr":"172.21.15.62:6833","nonce":428835470}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":25},{"gid":6238,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/4010598958","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":4010598958},{"type":"v1","addr":"172.21.15.62:6835","nonce":4010598958}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":25},{"gid":7008,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/2240458030","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6835","nonce":2240458030},{"type":"v1","addr":"172.21.15.16:6837","nonce":2240458030}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":25}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-05-22T15:42:22.811659+0000","modified":"2023-05-22T15:42:25.830716+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7011},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7011":{"gid":7011,"name":"a","rank":0,"incarnation":26,"state":"up:active","state_seq":3,"addr":"172.21.15.16:6836/1772031282","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1772031282},{"type":"v1","addr":"172.21.15.16:6836","nonce":1772031282}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2023-05-22T15:42:43.471 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 30 2023-05-22T15:42:43.472 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.468+0000 7eff6aa07640 1 -- 172.21.15.16:0/712957471 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7eff500651d0 msgr2=0x7eff50067680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:43.473 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.468+0000 7eff6aa07640 1 --2- 172.21.15.16:0/712957471 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7eff500651d0 0x7eff50067680 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7eff54000f70 tx=0x7eff54009040 comp rx=0 tx=0).stop 2023-05-22T15:42:43.473 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.468+0000 7eff6aa07640 1 -- 172.21.15.16:0/712957471 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7eff64070fb0 msgr2=0x7eff64154b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:43.473 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.468+0000 7eff6aa07640 1 --2- 172.21.15.16:0/712957471 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7eff64070fb0 0x7eff64154b60 secure :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 crypto rx=0x7eff5801a0b0 tx=0x7eff58019220 comp rx=0 tx=0).stop 2023-05-22T15:42:43.473 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.468+0000 7eff6aa07640 1 -- 172.21.15.16:0/712957471 shutdown_connections 2023-05-22T15:42:43.473 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.468+0000 7eff6aa07640 1 --2- 172.21.15.16:0/712957471 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7eff640affa0 0x7eff640c1b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:43.474 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.468+0000 7eff6aa07640 1 --2- 172.21.15.16:0/712957471 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7eff500651d0 0x7eff50067680 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:43.474 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.468+0000 7eff6aa07640 1 --2- 172.21.15.16:0/712957471 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7eff640afb40 0x7eff641602d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:43.474 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.468+0000 7eff6aa07640 1 --2- 172.21.15.16:0/712957471 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7eff64070fb0 0x7eff64154b60 unknown :-1 s=CLOSED pgs=1219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:43.474 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.468+0000 7eff6aa07640 1 -- 172.21.15.16:0/712957471 >> 172.21.15.16:0/712957471 conn(0x7eff64069720 msgr2=0x7eff6415c6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:43.474 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.468+0000 7eff6aa07640 1 -- 172.21.15.16:0/712957471 shutdown_connections 2023-05-22T15:42:43.474 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.468+0000 7eff6aa07640 1 -- 172.21.15.16:0/712957471 wait complete. 2023-05-22T15:42:43.488 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-05-22T15:42:43.601 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.596+0000 7facd1065640 1 Processor -- start 2023-05-22T15:42:43.602 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.596+0000 7facd1065640 1 -- start start 2023-05-22T15:42:43.603 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.600+0000 7facd1065640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7faccc159a40 0x7faccc0c8ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:43.603 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.600+0000 7facd1065640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faccc15a0a0 con 0x7faccc1531b0 2023-05-22T15:42:43.603 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.600+0000 7facd1065640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faccc153580 con 0x7faccc0b0250 2023-05-22T15:42:43.603 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.600+0000 7facd1065640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faccc159e90 con 0x7faccc159a40 2023-05-22T15:42:43.603 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.600+0000 7faccad9d640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7faccc1531b0 0x7faccc0afb40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:41092/0 (socket says 172.21.15.16:41092) 2023-05-22T15:42:43.604 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.600+0000 7faccad9d640 1 -- 172.21.15.16:0/1890606988 learned_addr learned my addr 172.21.15.16:0/1890606988 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:42:43.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.600+0000 7faccb59e640 1 --2- 172.21.15.16:0/1890606988 >> v2:172.21.15.62:3300/0 conn(0x7faccc159a40 0x7faccc0c8ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:43.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.600+0000 7facca59c640 1 -- 172.21.15.16:0/1890606988 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1421367223 0 0) 0x7faccc15a0a0 con 0x7faccc1531b0 2023-05-22T15:42:43.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.600+0000 7facca59c640 1 -- 172.21.15.16:0/1890606988 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7facb8005740 con 0x7faccc1531b0 2023-05-22T15:42:43.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.600+0000 7facca59c640 1 -- 172.21.15.16:0/1890606988 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 59999941 0 0) 0x7facb8005740 con 0x7faccc1531b0 2023-05-22T15:42:43.606 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.600+0000 7facca59c640 1 -- 172.21.15.16:0/1890606988 >> v2:172.21.15.62:3300/0 conn(0x7faccc159a40 msgr2=0x7faccc0c8ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:42:43.606 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.600+0000 7facca59c640 1 --2- 172.21.15.16:0/1890606988 >> v2:172.21.15.62:3300/0 conn(0x7faccc159a40 0x7faccc0c8ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:43.606 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.600+0000 7facca59c640 1 -- 172.21.15.16:0/1890606988 >> v1:172.21.15.62:6789/0 conn(0x7faccc0b0250 legacy=0x7faccc159330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:43.606 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.600+0000 7facca59c640 1 -- 172.21.15.16:0/1890606988 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faccc15d5e0 con 0x7faccc1531b0 2023-05-22T15:42:43.606 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.600+0000 7facca59c640 1 -- 172.21.15.16:0/1890606988 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7facbc001680 con 0x7faccc1531b0 2023-05-22T15:42:43.606 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.600+0000 7facca59c640 1 -- 172.21.15.16:0/1890606988 >> v1:172.21.15.16:6789/0 conn(0x7faccc1531b0 legacy=0x7faccc0afb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:43.606 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.600+0000 7facca59c640 1 --2- 172.21.15.16:0/1890606988 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7facb8005b90 0x7facb8005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:43.606 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.600+0000 7facca59c640 1 --2- 172.21.15.16:0/1890606988 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7facb80069e0 0x7facb8006e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:43.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.600+0000 7facca59c640 1 --2- 172.21.15.16:0/1890606988 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7facb8007390 0x7facb800dba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:43.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.600+0000 7facca59c640 1 -- 172.21.15.16:0/1890606988 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faccc159e90 con 0x7facb80069e0 2023-05-22T15:42:43.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.600+0000 7facca59c640 1 -- 172.21.15.16:0/1890606988 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facb800e210 con 0x7facb8005b90 2023-05-22T15:42:43.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.600+0000 7facca59c640 1 -- 172.21.15.16:0/1890606988 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7facb800e3d0 con 0x7facb8007390 2023-05-22T15:42:43.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.600+0000 7facc3fff640 1 --2- 172.21.15.16:0/1890606988 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7facb80069e0 0x7facb8006e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:43.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.600+0000 7faccad9d640 1 --2- 172.21.15.16:0/1890606988 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7facb8005b90 0x7facb8005fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:43.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.600+0000 7facc3fff640 1 -- 172.21.15.16:0/1890606988 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7facb8007390 msgr2=0x7facb800dba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:42:43.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.600+0000 7facc3fff640 1 --2- 172.21.15.16:0/1890606988 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7facb8007390 0x7facb800dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:43.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.600+0000 7facc3fff640 1 -- 172.21.15.16:0/1890606988 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7facb8005b90 msgr2=0x7facb8005fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:42:43.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.600+0000 7facc3fff640 1 --2- 172.21.15.16:0/1890606988 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7facb8005b90 0x7facb8005fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:43.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.600+0000 7faccb59e640 1 -- 172.21.15.16:0/1890606988 reap_dead start 2023-05-22T15:42:43.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.600+0000 7facc3fff640 1 -- 172.21.15.16:0/1890606988 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facb8005740 con 0x7facb80069e0 2023-05-22T15:42:43.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.600+0000 7faccad9d640 1 --2- 172.21.15.16:0/1890606988 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7facb8005b90 0x7facb8005fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:42:43.610 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.604+0000 7facc3fff640 1 --2- 172.21.15.16:0/1890606988 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7facb80069e0 0x7facb8006e50 secure :-1 s=READY pgs=1221 cs=0 l=1 rev1=1 crypto rx=0x7faccc06f400 tx=0x7facb401c380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9335547d57a90752 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:43.610 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.604+0000 7facca59c640 1 -- 172.21.15.16:0/1890606988 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facb40052d0 con 0x7facb80069e0 2023-05-22T15:42:43.611 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.604+0000 7facca59c640 1 -- 172.21.15.16:0/1890606988 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facb40a7050 con 0x7facb80069e0 2023-05-22T15:42:43.611 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.604+0000 7facca59c640 1 -- 172.21.15.16:0/1890606988 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facb4003600 con 0x7facb80069e0 2023-05-22T15:42:43.611 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.604+0000 7facd1065640 1 -- 172.21.15.16:0/1890606988 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7facb80069e0 msgr2=0x7facb8006e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:43.611 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.604+0000 7facd1065640 1 --2- 172.21.15.16:0/1890606988 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7facb80069e0 0x7facb8006e50 secure :-1 s=READY pgs=1221 cs=0 l=1 rev1=1 crypto rx=0x7faccc06f400 tx=0x7facb401c380 comp rx=0 tx=0).stop 2023-05-22T15:42:43.611 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.604+0000 7facd1065640 1 -- 172.21.15.16:0/1890606988 shutdown_connections 2023-05-22T15:42:43.611 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.604+0000 7facd1065640 1 --2- 172.21.15.16:0/1890606988 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7facb80069e0 0x7facb8006e50 unknown :-1 s=CLOSED pgs=1221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:43.611 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.604+0000 7facd1065640 1 -- 172.21.15.16:0/1890606988 >> 172.21.15.16:0/1890606988 conn(0x7faccc069720 msgr2=0x7faccc069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:43.612 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.604+0000 7facd1065640 1 -- 172.21.15.16:0/1890606988 shutdown_connections 2023-05-22T15:42:43.612 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.604+0000 7facd1065640 1 -- 172.21.15.16:0/1890606988 wait complete. 2023-05-22T15:42:43.612 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.604+0000 7facd1065640 1 Processor -- start 2023-05-22T15:42:43.612 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.604+0000 7facd1065640 1 -- start start 2023-05-22T15:42:43.612 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.604+0000 7facd1065640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7facb80069e0 0x7faccc1ecd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:43.612 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.604+0000 7facd1065640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faccc0c3470 0x7faccc140740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:43.612 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.604+0000 7facd1065640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faccc1504c0 0x7faccc150930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:43.612 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.604+0000 7facd1065640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faccc150e70 con 0x7faccc1504c0 2023-05-22T15:42:43.613 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.604+0000 7facd1065640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faccc0c1e80 con 0x7faccc0c3470 2023-05-22T15:42:43.613 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.604+0000 7facd1065640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faccc1ed280 con 0x7facb80069e0 2023-05-22T15:42:43.613 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.604+0000 7faccb59e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faccc1504c0 0x7faccc150930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:43.613 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.604+0000 7faccb59e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faccc1504c0 0x7faccc150930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49306/0 (socket says 172.21.15.16:49306) 2023-05-22T15:42:43.613 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.604+0000 7faccb59e640 1 -- 172.21.15.16:0/2148047957 learned_addr learned my addr 172.21.15.16:0/2148047957 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:42:43.613 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.604+0000 7faccad9d640 1 --2- 172.21.15.16:0/2148047957 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7facb80069e0 0x7faccc1ecd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:43.613 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.604+0000 7facc3fff640 1 --2- 172.21.15.16:0/2148047957 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faccc0c3470 0x7faccc140740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:43.614 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.608+0000 7faccb59e640 1 -- 172.21.15.16:0/2148047957 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7facb80069e0 msgr2=0x7faccc1ecd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:43.614 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.608+0000 7faccb59e640 1 --2- 172.21.15.16:0/2148047957 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7facb80069e0 0x7faccc1ecd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:43.614 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.608+0000 7faccb59e640 1 -- 172.21.15.16:0/2148047957 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faccc0c3470 msgr2=0x7faccc140740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:43.614 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.608+0000 7faccb59e640 1 --2- 172.21.15.16:0/2148047957 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faccc0c3470 0x7faccc140740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:43.614 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.608+0000 7faccb59e640 1 -- 172.21.15.16:0/2148047957 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faccc15cb00 con 0x7faccc1504c0 2023-05-22T15:42:43.614 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.608+0000 7faccad9d640 1 --2- 172.21.15.16:0/2148047957 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7facb80069e0 0x7faccc1ecd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:42:43.615 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.608+0000 7facc3fff640 1 --2- 172.21.15.16:0/2148047957 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faccc0c3470 0x7faccc140740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:42:43.615 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.608+0000 7faccb59e640 1 --2- 172.21.15.16:0/2148047957 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faccc1504c0 0x7faccc150930 secure :-1 s=READY pgs=1222 cs=0 l=1 rev1=1 crypto rx=0x7facc400ee50 tx=0x7facc400ef40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:43.615 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.608+0000 7facc37fe640 1 -- 172.21.15.16:0/2148047957 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facc4010040 con 0x7faccc1504c0 2023-05-22T15:42:43.615 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.608+0000 7facc37fe640 1 -- 172.21.15.16:0/2148047957 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facc4014830 con 0x7faccc1504c0 2023-05-22T15:42:43.615 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.608+0000 7facd1065640 1 -- 172.21.15.16:0/2148047957 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faccc15c820 con 0x7faccc1504c0 2023-05-22T15:42:43.615 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.608+0000 7facc37fe640 1 -- 172.21.15.16:0/2148047957 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facc4014be0 con 0x7faccc1504c0 2023-05-22T15:42:43.615 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.608+0000 7facd1065640 1 -- 172.21.15.16:0/2148047957 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faccc15cde0 con 0x7faccc1504c0 2023-05-22T15:42:43.616 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.608+0000 7facd1065640 1 -- 172.21.15.16:0/2148047957 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faccc04dbe0 con 0x7faccc1504c0 2023-05-22T15:42:43.616 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.608+0000 7facc37fe640 1 -- 172.21.15.16:0/2148047957 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 33) v1 ==== 84068+0+0 (secure 0 0 0) 0x7facc4014d40 con 0x7faccc1504c0 2023-05-22T15:42:43.617 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.608+0000 7facc37fe640 1 --2- 172.21.15.16:0/2148047957 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7facac065200 0x7facac0676b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:43.617 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.608+0000 7facc37fe640 1 -- 172.21.15.16:0/2148047957 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 7027+0+0 (secure 0 0 0) 0x7facc4082dd0 con 0x7faccc1504c0 2023-05-22T15:42:43.617 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.608+0000 7faccad9d640 1 --2- 172.21.15.16:0/2148047957 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7facac065200 0x7facac0676b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:43.617 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.608+0000 7faccad9d640 1 --2- 172.21.15.16:0/2148047957 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7facac065200 0x7facac0676b0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7facbc001210 tx=0x7facbc001180 comp rx=0 tx=0).ready entity=mgr.6436 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:43.617 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.612+0000 7facc37fe640 1 -- 172.21.15.16:0/2148047957 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7facc40544b0 con 0x7faccc1504c0 2023-05-22T15:42:43.768 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.764+0000 7facd1065640 1 -- 172.21.15.16:0/2148047957 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7faccc0cc2d0 con 0x7faccc1504c0 2023-05-22T15:42:43.771 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.768+0000 7facc37fe640 1 -- 172.21.15.16:0/2148047957 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v92) v1 ==== 74+0+18961 (secure 0 0 0) 0x7facc40540a0 con 0x7faccc1504c0 2023-05-22T15:42:43.771 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:42:43.772 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":92,"fsid":"b2719834-7847-4d98-90a6-f65d57f85c49","created":"2023-05-22T15:26:25.259314+0000","modified":"2023-05-22T15:42:41.449592+0000","last_up_change":"2023-05-22T15:26:36.448524+0000","last_in_change":"2023-05-22T15:26:27.675668+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-05-22T15:26:38.019781+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-05-22T15:26:43.570514+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2023-05-22T15:42:20.730757+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2023-05-22T15:42:21.805016+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9e7ff854-b25f-4cbb-a5c2-1799d9c53d2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":37021},{"type":"v1","addr":"172.21.15.16:6825","nonce":37021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":37021},{"type":"v1","addr":"172.21.15.16:6827","nonce":37021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":37021},{"type":"v1","addr":"172.21.15.16:6831","nonce":37021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":37021},{"type":"v1","addr":"172.21.15.16:6829","nonce":37021}]},"public_addr":"172.21.15.16:6825/37021","cluster_addr":"172.21.15.16:6827/37021","heartbeat_back_addr":"172.21.15.16:6831/37021","heartbeat_front_addr":"172.21.15.16:6829/37021","state":["exists","up"]},{"osd":1,"uuid":"4702fbc9-a927-4565-aca5-41dc69cc8264","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":37008},{"type":"v1","addr":"172.21.15.16:6801","nonce":37008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":37008},{"type":"v1","addr":"172.21.15.16:6803","nonce":37008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":37008},{"type":"v1","addr":"172.21.15.16:6807","nonce":37008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":37008},{"type":"v1","addr":"172.21.15.16:6805","nonce":37008}]},"public_addr":"172.21.15.16:6801/37008","cluster_addr":"172.21.15.16:6803/37008","heartbeat_back_addr":"172.21.15.16:6807/37008","heartbeat_front_addr":"172.21.15.16:6805/37008","state":["exists","up"]},{"osd":2,"uuid":"1e7bba46-4eba-47ef-bb81-5e2d371638ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":37009},{"type":"v1","addr":"172.21.15.16:6809","nonce":37009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":37009},{"type":"v1","addr":"172.21.15.16:6811","nonce":37009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":37009},{"type":"v1","addr":"172.21.15.16:6815","nonce":37009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":37009},{"type":"v1","addr":"172.21.15.16:6813","nonce":37009}]},"public_addr":"172.21.15.16:6809/37009","cluster_addr":"172.21.15.16:6811/37009","heartbeat_back_addr":"172.21.15.16:6815/37009","heartbeat_front_addr":"172.21.15.16:6813/37009","state":["exists","up"]},{"osd":3,"uuid":"383cacc0-5a88-4904-990e-498ca59ef48a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":37020},{"type":"v1","addr":"172.21.15.16:6817","nonce":37020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":37020},{"type":"v1","addr":"172.21.15.16:6819","nonce":37020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":37020},{"type":"v1","addr":"172.21.15.16:6823","nonce":37020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":37020},{"type":"v1","addr":"172.21.15.16:6821","nonce":37020}]},"public_addr":"172.21.15.16:6817/37020","cluster_addr":"172.21.15.16:6819/37020","heartbeat_back_addr":"172.21.15.16:6823/37020","heartbeat_front_addr":"172.21.15.16:6821/37020","state":["exists","up"]},{"osd":4,"uuid":"970500ef-95b7-42c1-b080-eed4fae7013d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":37353},{"type":"v1","addr":"172.21.15.62:6817","nonce":37353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6819","nonce":37353},{"type":"v1","addr":"172.21.15.62:6822","nonce":37353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37353},{"type":"v1","addr":"172.21.15.62:6831","nonce":37353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37353},{"type":"v1","addr":"172.21.15.62:6826","nonce":37353}]},"public_addr":"172.21.15.62:6817/37353","cluster_addr":"172.21.15.62:6822/37353","heartbeat_back_addr":"172.21.15.62:6831/37353","heartbeat_front_addr":"172.21.15.62:6826/37353","state":["exists","up"]},{"osd":5,"uuid":"022d67db-d4d8-445b-b1b9-8b446a00f5c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":37352},{"type":"v1","addr":"172.21.15.62:6801","nonce":37352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":37352},{"type":"v1","addr":"172.21.15.62:6803","nonce":37352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":37352},{"type":"v1","addr":"172.21.15.62:6807","nonce":37352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":37352},{"type":"v1","addr":"172.21.15.62:6805","nonce":37352}]},"public_addr":"172.21.15.62:6801/37352","cluster_addr":"172.21.15.62:6803/37352","heartbeat_back_addr":"172.21.15.62:6807/37352","heartbeat_front_addr":"172.21.15.62:6805/37352","state":["exists","up"]},{"osd":6,"uuid":"42bd372b-6688-4a1b-8407-e16e8c266e33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":37354},{"type":"v1","addr":"172.21.15.62:6820","nonce":37354}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":37354},{"type":"v1","addr":"172.21.15.62:6823","nonce":37354}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6829","nonce":37354},{"type":"v1","addr":"172.21.15.62:6830","nonce":37354}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6825","nonce":37354},{"type":"v1","addr":"172.21.15.62:6827","nonce":37354}]},"public_addr":"172.21.15.62:6820/37354","cluster_addr":"172.21.15.62:6823/37354","heartbeat_back_addr":"172.21.15.62:6830/37354","heartbeat_front_addr":"172.21.15.62:6827/37354","state":["exists","up"]},{"osd":7,"uuid":"ac70b2a3-f41b-4bf2-b227-7f2cbb55426d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37355},{"type":"v1","addr":"172.21.15.62:6809","nonce":37355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37355},{"type":"v1","addr":"172.21.15.62:6811","nonce":37355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":37355},{"type":"v1","addr":"172.21.15.62:6815","nonce":37355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":37355},{"type":"v1","addr":"172.21.15.62:6813","nonce":37355}]},"public_addr":"172.21.15.62:6809/37355","cluster_addr":"172.21.15.62:6811/37355","heartbeat_back_addr":"172.21.15.62:6815/37355","heartbeat_front_addr":"172.21.15.62:6813/37355","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.368896+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.107273+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.276888+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.287502+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.124791+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.251937+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.255679+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.232797+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:0/319876331":"2023-05-23T15:42:41.449418+0000","172.21.15.16:0/2036981971":"2023-05-23T15:42:41.449418+0000","172.21.15.16:0/2576009949":"2023-05-23T15:42:41.449418+0000","172.21.15.16:0/79429311":"2023-05-23T15:42:41.449418+0000","172.21.15.16:0/1473562727":"2023-05-23T15:42:41.449418+0000","172.21.15.16:0/3510757085":"2023-05-23T15:42:35.895737+0000","172.21.15.16:0/4119485618":"2023-05-23T15:42:35.895737+0000","172.21.15.16:0/2522450675":"2023-05-23T15:42:41.449418+0000","172.21.15.16:0/4196662600":"2023-05-23T15:42:41.449418+0000","172.21.15.16:0/2050049957":"2023-05-23T15:42:35.895737+0000","172.21.15.16:0/1396723034":"2023-05-23T15:42:35.895737+0000","172.21.15.16:0/3754867799":"2023-05-23T15:42:35.895737+0000","172.21.15.16:6833/36921":"2023-05-23T15:42:35.895737+0000","172.21.15.16:6832/36921":"2023-05-23T15:42:35.895737+0000","172.21.15.16:0/3710518307":"2023-05-23T15:42:35.895737+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-05-22T15:42:43.773 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.768+0000 7facd1065640 1 -- 172.21.15.16:0/2148047957 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7facac065200 msgr2=0x7facac0676b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:43.773 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.768+0000 7facd1065640 1 --2- 172.21.15.16:0/2148047957 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7facac065200 0x7facac0676b0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7facbc001210 tx=0x7facbc001180 comp rx=0 tx=0).stop 2023-05-22T15:42:43.774 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.768+0000 7facd1065640 1 -- 172.21.15.16:0/2148047957 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faccc1504c0 msgr2=0x7faccc150930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:43.774 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.768+0000 7facd1065640 1 --2- 172.21.15.16:0/2148047957 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faccc1504c0 0x7faccc150930 secure :-1 s=READY pgs=1222 cs=0 l=1 rev1=1 crypto rx=0x7facc400ee50 tx=0x7facc400ef40 comp rx=0 tx=0).stop 2023-05-22T15:42:43.774 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.768+0000 7facd1065640 1 -- 172.21.15.16:0/2148047957 shutdown_connections 2023-05-22T15:42:43.774 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.768+0000 7facd1065640 1 --2- 172.21.15.16:0/2148047957 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faccc1504c0 0x7faccc150930 unknown :-1 s=CLOSED pgs=1222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:43.775 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.768+0000 7facd1065640 1 --2- 172.21.15.16:0/2148047957 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faccc0c3470 0x7faccc140740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:43.775 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.768+0000 7facd1065640 1 --2- 172.21.15.16:0/2148047957 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7facac065200 0x7facac0676b0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:43.775 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.768+0000 7facd1065640 1 --2- 172.21.15.16:0/2148047957 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7facb80069e0 0x7faccc1ecd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:43.775 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.768+0000 7facd1065640 1 -- 172.21.15.16:0/2148047957 >> 172.21.15.16:0/2148047957 conn(0x7faccc069720 msgr2=0x7faccc0afb30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:43.775 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.768+0000 7facd1065640 1 -- 172.21.15.16:0/2148047957 shutdown_connections 2023-05-22T15:42:43.775 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.768+0000 7facd1065640 1 -- 172.21.15.16:0/2148047957 wait complete. 2023-05-22T15:42:43.789 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-05-22T15:42:43.912 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.907+0000 7f511e954640 1 Processor -- start 2023-05-22T15:42:43.912 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.907+0000 7f511e954640 1 -- start start 2023-05-22T15:42:43.913 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.907+0000 7f511e954640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f51180cd730 0x7f51180afb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:43.913 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.907+0000 7f511e954640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f51180b0080 0x7f51180acba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:43.913 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.907+0000 7f511e954640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51180cc300 con 0x7f51181531d0 2023-05-22T15:42:43.913 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.907+0000 7f511e954640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f511815d330 con 0x7f51180b0080 2023-05-22T15:42:43.913 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.907+0000 7f511e954640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f511815c4a0 con 0x7f51180cd730 2023-05-22T15:42:43.913 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.907+0000 7f5117fff640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f51181531d0 0x7f51180cbbf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:41096/0 (socket says 172.21.15.16:41096) 2023-05-22T15:42:43.914 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.907+0000 7f5117fff640 1 -- 172.21.15.16:0/3656821037 learned_addr learned my addr 172.21.15.16:0/3656821037 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:42:43.915 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.907+0000 7f511cef1640 1 --2- 172.21.15.16:0/3656821037 >> v2:172.21.15.62:3300/0 conn(0x7f51180b0080 0x7f51180acba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:43.915 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.907+0000 7f510ffff640 1 --2- 172.21.15.16:0/3656821037 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f51180cd730 0x7f51180afb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:43.915 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.907+0000 7f51177fe640 1 -- 172.21.15.16:0/3656821037 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4041779743 0 0) 0x7f51180cc300 con 0x7f51181531d0 2023-05-22T15:42:43.915 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.907+0000 7f51177fe640 1 -- 172.21.15.16:0/3656821037 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5104003530 con 0x7f51181531d0 2023-05-22T15:42:43.915 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.907+0000 7f51177fe640 1 -- 172.21.15.16:0/3656821037 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2125808940 0 0) 0x7f5104003530 con 0x7f51181531d0 2023-05-22T15:42:43.916 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.907+0000 7f51177fe640 1 -- 172.21.15.16:0/3656821037 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f51180cd730 msgr2=0x7f51180afb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:42:43.916 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.907+0000 7f51177fe640 1 --2- 172.21.15.16:0/3656821037 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f51180cd730 0x7f51180afb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:43.916 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.907+0000 7f51177fe640 1 -- 172.21.15.16:0/3656821037 >> v2:172.21.15.62:3300/0 conn(0x7f51180b0080 msgr2=0x7f51180acba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:42:43.916 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.907+0000 7f51177fe640 1 --2- 172.21.15.16:0/3656821037 >> v2:172.21.15.62:3300/0 conn(0x7f51180b0080 0x7f51180acba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:43.916 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.907+0000 7f51177fe640 1 -- 172.21.15.16:0/3656821037 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51180cdb60 con 0x7f51181531d0 2023-05-22T15:42:43.916 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.907+0000 7f51177fe640 1 -- 172.21.15.16:0/3656821037 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7f5108001680 con 0x7f51181531d0 2023-05-22T15:42:43.916 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.907+0000 7f51177fe640 1 -- 172.21.15.16:0/3656821037 >> v1:172.21.15.16:6789/0 conn(0x7f51181531d0 legacy=0x7f51180cbbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:43.917 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.907+0000 7f51177fe640 1 --2- 172.21.15.16:0/3656821037 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5104003980 0x7f5104003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:43.917 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.907+0000 7f51177fe640 1 --2- 172.21.15.16:0/3656821037 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f51180cd730 0x7f51040047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:43.917 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.907+0000 7f51177fe640 1 --2- 172.21.15.16:0/3656821037 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5104004d10 0x7f5104005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:43.917 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.907+0000 7f51177fe640 1 -- 172.21.15.16:0/3656821037 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f511815d330 con 0x7f5104003980 2023-05-22T15:42:43.917 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.907+0000 7f51177fe640 1 -- 172.21.15.16:0/3656821037 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f511815c4a0 con 0x7f5104004d10 2023-05-22T15:42:43.917 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.907+0000 7f51177fe640 1 -- 172.21.15.16:0/3656821037 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f510400b990 con 0x7f51180cd730 2023-05-22T15:42:43.917 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.907+0000 7f511cef1640 1 --2- 172.21.15.16:0/3656821037 >> v2:172.21.15.62:3300/0 conn(0x7f51180b0080 0x7f51180acba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-05-22T15:42:43.917 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.911+0000 7f5117fff640 1 --2- 172.21.15.16:0/3656821037 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5104003980 0x7f5104003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:43.918 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.911+0000 7f510ffff640 1 --2- 172.21.15.16:0/3656821037 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f51180cd730 0x7f51040047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:43.918 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.911+0000 7f511cef1640 1 --2- 172.21.15.16:0/3656821037 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5104004d10 0x7f5104005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:43.918 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.911+0000 7f5117fff640 1 -- 172.21.15.16:0/3656821037 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f51180cd730 msgr2=0x7f51040047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:42:43.918 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.911+0000 7f5117fff640 1 --2- 172.21.15.16:0/3656821037 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f51180cd730 0x7f51040047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:43.918 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.911+0000 7f5117fff640 1 -- 172.21.15.16:0/3656821037 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5104004d10 msgr2=0x7f5104005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:42:43.918 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.911+0000 7f5117fff640 1 --2- 172.21.15.16:0/3656821037 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5104004d10 0x7f5104005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:43.918 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.911+0000 7f5117fff640 1 -- 172.21.15.16:0/3656821037 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5104003530 con 0x7f5104003980 2023-05-22T15:42:43.921 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.911+0000 7f5117fff640 1 --2- 172.21.15.16:0/3656821037 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5104003980 0x7f5104003d90 secure :-1 s=READY pgs=1224 cs=0 l=1 rev1=1 crypto rx=0x7f510800c210 tx=0x7f510801cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4851c4d6bf8ef53 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:43.921 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.911+0000 7f51177fe640 1 -- 172.21.15.16:0/3656821037 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51080058e0 con 0x7f5104003980 2023-05-22T15:42:43.921 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.911+0000 7f51177fe640 1 -- 172.21.15.16:0/3656821037 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5108005a60 con 0x7f5104003980 2023-05-22T15:42:43.921 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.911+0000 7f51177fe640 1 -- 172.21.15.16:0/3656821037 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5108005e10 con 0x7f5104003980 2023-05-22T15:42:43.921 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.911+0000 7f511e954640 1 -- 172.21.15.16:0/3656821037 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5104003980 msgr2=0x7f5104003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:43.922 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.911+0000 7f511e954640 1 --2- 172.21.15.16:0/3656821037 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5104003980 0x7f5104003d90 secure :-1 s=READY pgs=1224 cs=0 l=1 rev1=1 crypto rx=0x7f510800c210 tx=0x7f510801cc20 comp rx=0 tx=0).stop 2023-05-22T15:42:43.922 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.911+0000 7f511cef1640 1 -- 172.21.15.16:0/3656821037 reap_dead start 2023-05-22T15:42:43.922 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.911+0000 7f511e954640 1 -- 172.21.15.16:0/3656821037 shutdown_connections 2023-05-22T15:42:43.922 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.911+0000 7f511e954640 1 -- 172.21.15.16:0/3656821037 >> 172.21.15.16:0/3656821037 conn(0x7f5118069720 msgr2=0x7f5118069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:43.922 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.911+0000 7f511e954640 1 -- 172.21.15.16:0/3656821037 shutdown_connections 2023-05-22T15:42:43.922 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.911+0000 7f511e954640 1 -- 172.21.15.16:0/3656821037 wait complete. 2023-05-22T15:42:43.923 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.915+0000 7f511e954640 1 Processor -- start 2023-05-22T15:42:43.923 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.915+0000 7f511e954640 1 -- start start 2023-05-22T15:42:43.923 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.915+0000 7f511e954640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5104003980 0x7f51180b2ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:43.923 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.915+0000 7f511e954640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f51180c92d0 0x7f51180c9970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:43.923 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.915+0000 7f511e954640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f51180c9eb0 0x7f511815e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:43.923 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.915+0000 7f511e954640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51180ca710 con 0x7f5104003980 2023-05-22T15:42:43.923 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.915+0000 7f511e954640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51180ca430 con 0x7f51180c92d0 2023-05-22T15:42:43.924 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.915+0000 7f511e954640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51180c87f0 con 0x7f51180c9eb0 2023-05-22T15:42:43.924 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.915+0000 7f5117fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5104003980 0x7f51180b2ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:43.924 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.915+0000 7f5117fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5104003980 0x7f51180b2ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49320/0 (socket says 172.21.15.16:49320) 2023-05-22T15:42:43.924 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.915+0000 7f5117fff640 1 -- 172.21.15.16:0/3827407217 learned_addr learned my addr 172.21.15.16:0/3827407217 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:42:43.924 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.915+0000 7f510ffff640 1 --2- 172.21.15.16:0/3827407217 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f51180c92d0 0x7f51180c9970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:43.924 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.915+0000 7f511cef1640 1 --2- 172.21.15.16:0/3827407217 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f51180c9eb0 0x7f511815e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:43.924 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.915+0000 7f5117fff640 1 -- 172.21.15.16:0/3827407217 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f51180c9eb0 msgr2=0x7f511815e620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:43.925 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.915+0000 7f5117fff640 1 --2- 172.21.15.16:0/3827407217 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f51180c9eb0 0x7f511815e620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:43.925 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.915+0000 7f5117fff640 1 -- 172.21.15.16:0/3827407217 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f51180c92d0 msgr2=0x7f51180c9970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:43.925 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.915+0000 7f5117fff640 1 --2- 172.21.15.16:0/3827407217 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f51180c92d0 0x7f51180c9970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:43.925 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.915+0000 7f5117fff640 1 -- 172.21.15.16:0/3827407217 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f511815cf50 con 0x7f5104003980 2023-05-22T15:42:43.925 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.915+0000 7f511cef1640 1 --2- 172.21.15.16:0/3827407217 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f51180c9eb0 0x7f511815e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:42:43.925 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.915+0000 7f5117fff640 1 --2- 172.21.15.16:0/3827407217 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5104003980 0x7f51180b2ce0 secure :-1 s=READY pgs=1225 cs=0 l=1 rev1=1 crypto rx=0x7f510800c210 tx=0x7f5104005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:43.925 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.915+0000 7f51157fa640 1 -- 172.21.15.16:0/3827407217 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51080b4070 con 0x7f5104003980 2023-05-22T15:42:43.926 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.915+0000 7f51157fa640 1 -- 172.21.15.16:0/3827407217 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5108001440 con 0x7f5104003980 2023-05-22T15:42:43.926 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.915+0000 7f511e954640 1 -- 172.21.15.16:0/3827407217 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f511815cc70 con 0x7f5104003980 2023-05-22T15:42:43.926 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.915+0000 7f51157fa640 1 -- 172.21.15.16:0/3827407217 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51080017d0 con 0x7f5104003980 2023-05-22T15:42:43.926 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.915+0000 7f511e954640 1 -- 172.21.15.16:0/3827407217 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5118160740 con 0x7f5104003980 2023-05-22T15:42:43.926 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.915+0000 7f511e954640 1 -- 172.21.15.16:0/3827407217 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51180c8650 con 0x7f5104003980 2023-05-22T15:42:43.927 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.915+0000 7f51157fa640 1 -- 172.21.15.16:0/3827407217 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 33) v1 ==== 84068+0+0 (secure 0 0 0) 0x7f5108007090 con 0x7f5104003980 2023-05-22T15:42:43.927 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.915+0000 7f51157fa640 1 --2- 172.21.15.16:0/3827407217 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f50f8065200 0x7f50f80676b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:43.927 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.919+0000 7f51157fa640 1 -- 172.21.15.16:0/3827407217 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 7027+0+0 (secure 0 0 0) 0x7f510811c280 con 0x7f5104003980 2023-05-22T15:42:43.927 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.919+0000 7f510ffff640 1 --2- 172.21.15.16:0/3827407217 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f50f8065200 0x7f50f80676b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:43.927 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.919+0000 7f510ffff640 1 --2- 172.21.15.16:0/3827407217 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f50f8065200 0x7f50f80676b0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f511806a400 tx=0x7f5100008040 comp rx=0 tx=0).ready entity=mgr.6436 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:43.927 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:43.919+0000 7f51157fa640 1 -- 172.21.15.16:0/3827407217 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f51180c8650 con 0x7f5104003980 2023-05-22T15:42:44.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:44.107+0000 7f511e954640 1 -- 172.21.15.16:0/3827407217 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f51180c8650 con 0x7f5104003980 2023-05-22T15:42:44.488 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:44.483+0000 7f51157fa640 1 -- 172.21.15.16:0/3827407217 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mgrmap(e 34) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f51080ea240 con 0x7f5104003980 2023-05-22T15:42:44.947 INFO:tasks.ceph.mon.a.smithi016.stderr:2023-05-22T15:42:44.943+0000 7f95dd48a640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-05-22T15:42:44.957 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:44.951+0000 7f51157fa640 1 -- 172.21.15.16:0/3827407217 <== mon.0 v2:172.21.15.16:3300/0 8 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v31) v1 ==== 161+0+0 (secure 0 0 0) 0x7f51080ed8a0 con 0x7f5104003980 2023-05-22T15:42:44.957 INFO:teuthology.orchestra.run.smithi016.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-05-22T15:42:44.960 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:44.955+0000 7f511e954640 1 -- 172.21.15.16:0/3827407217 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f50f8065200 msgr2=0x7f50f80676b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:44.961 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:44.955+0000 7f511e954640 1 --2- 172.21.15.16:0/3827407217 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f50f8065200 0x7f50f80676b0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f511806a400 tx=0x7f5100008040 comp rx=0 tx=0).stop 2023-05-22T15:42:44.961 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:44.955+0000 7f511e954640 1 -- 172.21.15.16:0/3827407217 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5104003980 msgr2=0x7f51180b2ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:44.961 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:44.955+0000 7f511e954640 1 --2- 172.21.15.16:0/3827407217 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5104003980 0x7f51180b2ce0 secure :-1 s=READY pgs=1225 cs=0 l=1 rev1=1 crypto rx=0x7f510800c210 tx=0x7f5104005800 comp rx=0 tx=0).stop 2023-05-22T15:42:44.961 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:44.955+0000 7f511e954640 1 -- 172.21.15.16:0/3827407217 shutdown_connections 2023-05-22T15:42:44.961 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:44.955+0000 7f511e954640 1 --2- 172.21.15.16:0/3827407217 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f50f8065200 0x7f50f80676b0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:44.961 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:44.955+0000 7f511e954640 1 --2- 172.21.15.16:0/3827407217 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f51180c9eb0 0x7f511815e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:44.961 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:44.955+0000 7f511e954640 1 --2- 172.21.15.16:0/3827407217 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f51180c92d0 0x7f51180c9970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:44.962 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:44.955+0000 7f511e954640 1 --2- 172.21.15.16:0/3827407217 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5104003980 0x7f51180b2ce0 unknown :-1 s=CLOSED pgs=1225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:44.962 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:44.955+0000 7f511e954640 1 -- 172.21.15.16:0/3827407217 >> 172.21.15.16:0/3827407217 conn(0x7f5118069720 msgr2=0x7f511815b070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:44.962 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:44.955+0000 7f511e954640 1 -- 172.21.15.16:0/3827407217 shutdown_connections 2023-05-22T15:42:44.962 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:44.955+0000 7f511e954640 1 -- 172.21.15.16:0/3827407217 wait complete. 2023-05-22T15:42:44.962 INFO:tasks.ceph.mds.a.smithi016.stderr: -2696> 2023-05-22T15:42:28.632+0000 7faa91fcf640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-05-22T15:42:44.985 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-05-22T15:42:44.995 INFO:tasks.ceph.mds.a.smithi016.stdout:starting mds.a at 2023-05-22T15:42:45.109 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:45.103+0000 7f14ff5d6640 1 Processor -- start 2023-05-22T15:42:45.109 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:45.103+0000 7f14ff5d6640 1 -- start start 2023-05-22T15:42:45.110 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:45.103+0000 7f14ff5d6640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f14f8158210 0x7f14f8159160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:45.111 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:45.103+0000 7f14ff5d6640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f14f80c8d70 0x7f14f815c120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:45.111 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:45.103+0000 7f14ff5d6640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14f8158cd0 con 0x7f14f815bcd0 2023-05-22T15:42:45.111 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:45.103+0000 7f14ff5d6640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14f81596a0 con 0x7f14f8158210 2023-05-22T15:42:45.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:45.103+0000 7f14ff5d6640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14f815c790 con 0x7f14f80c8d70 2023-05-22T15:42:45.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:45.107+0000 7f14fd372640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f14f8158210 0x7f14f8159160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:45.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:45.107+0000 7f14fd372640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f14f8158210 0x7f14f8159160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49340/0 (socket says 172.21.15.16:49340) 2023-05-22T15:42:45.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:45.107+0000 7f14fd372640 1 -- 172.21.15.16:0/2566638031 learned_addr learned my addr 172.21.15.16:0/2566638031 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:42:45.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:45.107+0000 7f14fdb73640 1 --2- 172.21.15.16:0/2566638031 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f14f80c8d70 0x7f14f815c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:45.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:45.107+0000 7f14fd372640 1 -- 172.21.15.16:0/2566638031 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f14f80c8d70 msgr2=0x7f14f815c120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:42:45.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:45.107+0000 7f14fd372640 1 --2- 172.21.15.16:0/2566638031 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f14f80c8d70 0x7f14f815c120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:45.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:45.107+0000 7f14fd372640 1 -- 172.21.15.16:0/2566638031 >> v1:172.21.15.62:6789/0 conn(0x7f14f815bcd0 legacy=0x7f14f80c8660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:42:45.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:45.107+0000 7f14fd372640 1 -- 172.21.15.16:0/2566638031 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14f815ca70 con 0x7f14f8158210 2023-05-22T15:42:45.118 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:45.107+0000 7f14fd372640 1 --2- 172.21.15.16:0/2566638031 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f14f8158210 0x7f14f8159160 secure :-1 s=READY pgs=1229 cs=0 l=1 rev1=1 crypto rx=0x7f14e8009870 tx=0x7f14e801c440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47875d5771e611c3 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:45.118 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:45.107+0000 7f14effff640 1 -- 172.21.15.16:0/2566638031 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14e80ab600 con 0x7f14f8158210 2023-05-22T15:42:45.119 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:45.107+0000 7f14effff640 1 -- 172.21.15.16:0/2566638031 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14e80a7050 con 0x7f14f8158210 2023-05-22T15:42:45.119 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:45.107+0000 7f14effff640 1 -- 172.21.15.16:0/2566638031 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14e80ab980 con 0x7f14f8158210 2023-05-22T15:42:45.119 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:45.107+0000 7f14ff5d6640 1 -- 172.21.15.16:0/2566638031 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f14f8158210 msgr2=0x7f14f8159160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:45.119 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:45.107+0000 7f14ff5d6640 1 --2- 172.21.15.16:0/2566638031 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f14f8158210 0x7f14f8159160 secure :-1 s=READY pgs=1229 cs=0 l=1 rev1=1 crypto rx=0x7f14e8009870 tx=0x7f14e801c440 comp rx=0 tx=0).stop 2023-05-22T15:42:45.120 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:45.107+0000 7f14ff5d6640 1 -- 172.21.15.16:0/2566638031 shutdown_connections 2023-05-22T15:42:45.120 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:45.107+0000 7f14ff5d6640 1 --2- 172.21.15.16:0/2566638031 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f14f80c8d70 0x7f14f815c120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:45.120 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:45.107+0000 7f14ff5d6640 1 --2- 172.21.15.16:0/2566638031 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f14f8158210 0x7f14f8159160 unknown :-1 s=CLOSED pgs=1229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:45.121 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:45.107+0000 7f14ff5d6640 1 -- 172.21.15.16:0/2566638031 >> 172.21.15.16:0/2566638031 conn(0x7f14f8069720 msgr2=0x7f14f8069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:45.121 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:45.107+0000 7f14ff5d6640 1 -- 172.21.15.16:0/2566638031 shutdown_connections 2023-05-22T15:42:45.121 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:45.107+0000 7f14ff5d6640 1 -- 172.21.15.16:0/2566638031 wait complete. 2023-05-22T15:42:45.121 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:45.111+0000 7f14ff5d6640 1 Processor -- start 2023-05-22T15:42:45.122 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:45.111+0000 7f14ff5d6640 1 -- start start 2023-05-22T15:42:45.122 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:45.111+0000 7f14ff5d6640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f14f80c8d70 0x7f14f814e960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:45.122 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:45.111+0000 7f14ff5d6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f14f8158210 0x7f14f81626f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:45.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:45.111+0000 7f14ff5d6640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f14f815bcd0 0x7f14f81614b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:45.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:45.111+0000 7f14ff5d6640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14f8070f80 con 0x7f14f8158210 2023-05-22T15:42:45.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:45.111+0000 7f14ff5d6640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14f8162c30 con 0x7f14f815bcd0 2023-05-22T15:42:45.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:45.111+0000 7f14ff5d6640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14f80703b0 con 0x7f14f80c8d70 2023-05-22T15:42:45.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:45.111+0000 7f14fcb71640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f14f8158210 0x7f14f81626f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:45.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:45.111+0000 7f14fcb71640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f14f8158210 0x7f14f81626f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49342/0 (socket says 172.21.15.16:49342) 2023-05-22T15:42:45.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:45.111+0000 7f14fcb71640 1 -- 172.21.15.16:0/2463904787 learned_addr learned my addr 172.21.15.16:0/2463904787 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:42:45.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:45.111+0000 7f14fd372640 1 --2- 172.21.15.16:0/2463904787 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f14f80c8d70 0x7f14f814e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:45.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:45.111+0000 7f14fdb73640 1 --2- 172.21.15.16:0/2463904787 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f14f815bcd0 0x7f14f81614b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:45.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:45.111+0000 7f14fcb71640 1 -- 172.21.15.16:0/2463904787 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f14f80c8d70 msgr2=0x7f14f814e960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:45.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:45.111+0000 7f14fcb71640 1 --2- 172.21.15.16:0/2463904787 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f14f80c8d70 0x7f14f814e960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:45.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:45.111+0000 7f14fcb71640 1 -- 172.21.15.16:0/2463904787 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f14f815bcd0 msgr2=0x7f14f81614b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:45.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:45.111+0000 7f14fcb71640 1 --2- 172.21.15.16:0/2463904787 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f14f815bcd0 0x7f14f81614b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:45.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:45.111+0000 7f14fcb71640 1 -- 172.21.15.16:0/2463904787 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14e80094d0 con 0x7f14f8158210 2023-05-22T15:42:45.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:45.111+0000 7f14fdb73640 1 --2- 172.21.15.16:0/2463904787 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f14f815bcd0 0x7f14f81614b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T15:42:45.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:45.111+0000 7f14fcb71640 1 --2- 172.21.15.16:0/2463904787 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f14f8158210 0x7f14f81626f0 secure :-1 s=READY pgs=1230 cs=0 l=1 rev1=1 crypto rx=0x7f14f400b410 tx=0x7f14f400b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:45.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:45.111+0000 7f14edffb640 1 -- 172.21.15.16:0/2463904787 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14f4004020 con 0x7f14f8158210 2023-05-22T15:42:45.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:45.111+0000 7f14edffb640 1 -- 172.21.15.16:0/2463904787 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14f4004180 con 0x7f14f8158210 2023-05-22T15:42:45.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:45.111+0000 7f14ff5d6640 1 -- 172.21.15.16:0/2463904787 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14f80ca690 con 0x7f14f8158210 2023-05-22T15:42:45.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:45.111+0000 7f14edffb640 1 -- 172.21.15.16:0/2463904787 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14f4011630 con 0x7f14f8158210 2023-05-22T15:42:45.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:45.111+0000 7f14ff5d6640 1 -- 172.21.15.16:0/2463904787 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14f81585e0 con 0x7f14f8158210 2023-05-22T15:42:45.126 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:45.115+0000 7f14ff5d6640 1 -- 172.21.15.16:0/2463904787 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14f80ca820 con 0x7f14f8158210 2023-05-22T15:42:45.126 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:45.115+0000 7f14edffb640 1 -- 172.21.15.16:0/2463904787 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 34) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f14f4011790 con 0x7f14f8158210 2023-05-22T15:42:45.127 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:45.115+0000 7f14edffb640 1 --2- 172.21.15.16:0/2463904787 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f14dc0654a0 0x7f14dc067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:45.127 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:45.115+0000 7f14edffb640 1 -- 172.21.15.16:0/2463904787 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 7113+0+0 (secure 0 0 0) 0x7f14f4082b30 con 0x7f14f8158210 2023-05-22T15:42:45.127 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:45.115+0000 7f14fd372640 1 --2- 172.21.15.16:0/2463904787 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f14dc0654a0 0x7f14dc067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:45.127 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:45.115+0000 7f14fd372640 1 --2- 172.21.15.16:0/2463904787 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f14dc0654a0 0x7f14dc067950 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f14e80ad000 tx=0x7f14e80a6000 comp rx=0 tx=0).ready entity=mgr.6436 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:45.127 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:45.119+0000 7f14edffb640 1 -- 172.21.15.16:0/2463904787 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f14f4050280 con 0x7f14f8158210 2023-05-22T15:42:45.314 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:45.311+0000 7f14ff5d6640 1 -- 172.21.15.16:0/2463904787 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f14f8153270 con 0x7f14f8158210 2023-05-22T15:42:45.970 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:45.963+0000 7f14edffb640 1 -- 172.21.15.16:0/2463904787 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v32) v1 ==== 104+0+0 (secure 0 0 0) 0x7f14f40538a0 con 0x7f14f8158210 2023-05-22T15:42:45.973 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:45.967+0000 7f14ff5d6640 1 -- 172.21.15.16:0/2463904787 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f14dc0654a0 msgr2=0x7f14dc067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:45.973 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:45.967+0000 7f14ff5d6640 1 --2- 172.21.15.16:0/2463904787 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f14dc0654a0 0x7f14dc067950 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f14e80ad000 tx=0x7f14e80a6000 comp rx=0 tx=0).stop 2023-05-22T15:42:45.973 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:45.967+0000 7f14ff5d6640 1 -- 172.21.15.16:0/2463904787 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f14f8158210 msgr2=0x7f14f81626f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:45.973 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:45.967+0000 7f14ff5d6640 1 --2- 172.21.15.16:0/2463904787 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f14f8158210 0x7f14f81626f0 secure :-1 s=READY pgs=1230 cs=0 l=1 rev1=1 crypto rx=0x7f14f400b410 tx=0x7f14f400b8d0 comp rx=0 tx=0).stop 2023-05-22T15:42:45.973 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:45.967+0000 7f14ff5d6640 1 -- 172.21.15.16:0/2463904787 shutdown_connections 2023-05-22T15:42:45.973 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:45.967+0000 7f14ff5d6640 1 --2- 172.21.15.16:0/2463904787 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f14f815bcd0 0x7f14f81614b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:45.974 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:45.967+0000 7f14ff5d6640 1 --2- 172.21.15.16:0/2463904787 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f14f8158210 0x7f14f81626f0 unknown :-1 s=CLOSED pgs=1230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:45.974 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:45.967+0000 7f14ff5d6640 1 --2- 172.21.15.16:0/2463904787 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f14dc0654a0 0x7f14dc067950 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:45.974 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:45.967+0000 7f14ff5d6640 1 --2- 172.21.15.16:0/2463904787 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f14f80c8d70 0x7f14f814e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:45.974 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:45.967+0000 7f14ff5d6640 1 -- 172.21.15.16:0/2463904787 >> 172.21.15.16:0/2463904787 conn(0x7f14f8069720 msgr2=0x7f14f80afb20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:45.974 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:45.967+0000 7f14ff5d6640 1 -- 172.21.15.16:0/2463904787 shutdown_connections 2023-05-22T15:42:45.975 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:45.967+0000 7f14ff5d6640 1 -- 172.21.15.16:0/2463904787 wait complete. 2023-05-22T15:42:45.996 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-05-22T15:42:46.173 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:46.167+0000 7f234dad4640 1 Processor -- start 2023-05-22T15:42:46.173 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:46.167+0000 7f234dad4640 1 -- start start 2023-05-22T15:42:46.174 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:46.167+0000 7f234dad4640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f23480cc4e0 0x7f234806bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:46.174 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:46.167+0000 7f234dad4640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f234815d650 0x7f234815dad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:46.174 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:46.167+0000 7f234dad4640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23480cc930 con 0x7f2348070c10 2023-05-22T15:42:46.174 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:46.167+0000 7f234dad4640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f234815e010 con 0x7f234815d650 2023-05-22T15:42:46.174 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:46.167+0000 7f234dad4640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23480ccbb0 con 0x7f23480cc4e0 2023-05-22T15:42:46.175 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:46.171+0000 7f2347fff640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f234815d650 0x7f234815dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:46.175 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:46.171+0000 7f2347fff640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f234815d650 0x7f234815dad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.16:43152/0 (socket says 172.21.15.16:43152) 2023-05-22T15:42:46.176 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:46.171+0000 7f23477fe640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f2348070c10 0x7f23480c9e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.16:52258/0 (socket says 172.21.15.16:52258) 2023-05-22T15:42:46.176 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:46.171+0000 7f2347fff640 1 -- 172.21.15.16:0/3951591240 learned_addr learned my addr 172.21.15.16:0/3951591240 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:42:46.176 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:46.171+0000 7f2346ffd640 1 --2- 172.21.15.16:0/3951591240 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f23480cc4e0 0x7f234806bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:46.176 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:46.171+0000 7f2347fff640 1 -- 172.21.15.16:0/3951591240 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f23480cc4e0 msgr2=0x7f234806bfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:42:46.176 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:46.171+0000 7f2347fff640 1 --2- 172.21.15.16:0/3951591240 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f23480cc4e0 0x7f234806bfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:46.176 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:46.171+0000 7f2347fff640 1 -- 172.21.15.16:0/3951591240 >> v1:172.21.15.62:6789/0 conn(0x7f2348070c10 legacy=0x7f23480c9e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:46.177 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:46.171+0000 7f2347fff640 1 -- 172.21.15.16:0/3951591240 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23480cd7c0 con 0x7f234815d650 2023-05-22T15:42:46.177 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:46.171+0000 7f2347fff640 1 --2- 172.21.15.16:0/3951591240 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f234815d650 0x7f234815dad0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f2340009e50 tx=0x7f234001c9e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1ec605ae3a265129 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:46.178 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:46.171+0000 7f23467fc640 1 -- 172.21.15.16:0/3951591240 <== mon.1 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23400a58a0 con 0x7f234815d650 2023-05-22T15:42:46.178 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:46.171+0000 7f23467fc640 1 -- 172.21.15.16:0/3951591240 <== mon.1 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23400a5a00 con 0x7f234815d650 2023-05-22T15:42:46.178 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:46.171+0000 7f23467fc640 1 -- 172.21.15.16:0/3951591240 <== mon.1 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23400a5d90 con 0x7f234815d650 2023-05-22T15:42:46.179 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:46.175+0000 7f234dad4640 1 -- 172.21.15.16:0/3951591240 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f234815d650 msgr2=0x7f234815dad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:46.179 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:46.175+0000 7f234dad4640 1 --2- 172.21.15.16:0/3951591240 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f234815d650 0x7f234815dad0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f2340009e50 tx=0x7f234001c9e0 comp rx=0 tx=0).stop 2023-05-22T15:42:46.179 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:46.175+0000 7f234dad4640 1 -- 172.21.15.16:0/3951591240 shutdown_connections 2023-05-22T15:42:46.179 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:46.175+0000 7f234dad4640 1 --2- 172.21.15.16:0/3951591240 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f234815d650 0x7f234815dad0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:46.179 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:46.175+0000 7f234dad4640 1 --2- 172.21.15.16:0/3951591240 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f23480cc4e0 0x7f234806bfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:46.180 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:46.175+0000 7f234dad4640 1 -- 172.21.15.16:0/3951591240 >> 172.21.15.16:0/3951591240 conn(0x7f2348069720 msgr2=0x7f2348069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:46.180 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:46.175+0000 7f234dad4640 1 -- 172.21.15.16:0/3951591240 shutdown_connections 2023-05-22T15:42:46.180 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:46.175+0000 7f234dad4640 1 -- 172.21.15.16:0/3951591240 wait complete. 2023-05-22T15:42:46.181 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:46.175+0000 7f234dad4640 1 Processor -- start 2023-05-22T15:42:46.181 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:46.175+0000 7f234dad4640 1 -- start start 2023-05-22T15:42:46.182 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:46.175+0000 7f234dad4640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2348070c10 0x7f23480c45d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:46.182 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:46.175+0000 7f234dad4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f23480cc4e0 0x7f234814e230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:46.182 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:46.175+0000 7f234dad4640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f234815d650 0x7f2348150790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:46.183 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:46.175+0000 7f234dad4640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f234814f550 con 0x7f23480cc4e0 2023-05-22T15:42:46.183 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:46.175+0000 7f234dad4640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23480c4b10 con 0x7f2348070c10 2023-05-22T15:42:46.183 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:46.175+0000 7f234dad4640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f234814e770 con 0x7f234815d650 2023-05-22T15:42:46.183 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:46.175+0000 7f2346ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f23480cc4e0 0x7f234814e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:46.184 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:46.175+0000 7f2346ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f23480cc4e0 0x7f234814e230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49350/0 (socket says 172.21.15.16:49350) 2023-05-22T15:42:46.184 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:46.175+0000 7f2346ffd640 1 -- 172.21.15.16:0/3756359160 learned_addr learned my addr 172.21.15.16:0/3756359160 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:42:46.184 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:46.175+0000 7f2347fff640 1 --2- 172.21.15.16:0/3756359160 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f234815d650 0x7f2348150790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:46.184 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:46.175+0000 7f23477fe640 1 --2- 172.21.15.16:0/3756359160 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2348070c10 0x7f23480c45d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:46.185 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:46.175+0000 7f2346ffd640 1 -- 172.21.15.16:0/3756359160 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f234815d650 msgr2=0x7f2348150790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:46.185 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:46.175+0000 7f2346ffd640 1 --2- 172.21.15.16:0/3756359160 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f234815d650 0x7f2348150790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:46.185 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:46.175+0000 7f2346ffd640 1 -- 172.21.15.16:0/3756359160 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2348070c10 msgr2=0x7f23480c45d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:46.185 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:46.175+0000 7f2346ffd640 1 --2- 172.21.15.16:0/3756359160 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2348070c10 0x7f23480c45d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:46.185 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:46.175+0000 7f2346ffd640 1 -- 172.21.15.16:0/3756359160 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2340009b30 con 0x7f23480cc4e0 2023-05-22T15:42:46.185 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:46.175+0000 7f23477fe640 1 --2- 172.21.15.16:0/3756359160 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2348070c10 0x7f23480c45d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:42:46.185 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:46.175+0000 7f2347fff640 1 --2- 172.21.15.16:0/3756359160 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f234815d650 0x7f2348150790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:42:46.186 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:46.175+0000 7f2346ffd640 1 --2- 172.21.15.16:0/3756359160 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f23480cc4e0 0x7f234814e230 secure :-1 s=READY pgs=1231 cs=0 l=1 rev1=1 crypto rx=0x7f233c00e870 tx=0x7f233c00ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:46.187 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:46.175+0000 7f232ffff640 1 -- 172.21.15.16:0/3756359160 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f233c009510 con 0x7f23480cc4e0 2023-05-22T15:42:46.187 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:46.175+0000 7f232ffff640 1 -- 172.21.15.16:0/3756359160 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f233c00eec0 con 0x7f23480cc4e0 2023-05-22T15:42:46.187 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:46.175+0000 7f234dad4640 1 -- 172.21.15.16:0/3756359160 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f234806c4e0 con 0x7f23480cc4e0 2023-05-22T15:42:46.187 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:46.175+0000 7f232ffff640 1 -- 172.21.15.16:0/3756359160 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f233c00cb40 con 0x7f23480cc4e0 2023-05-22T15:42:46.188 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:46.175+0000 7f234dad4640 1 -- 172.21.15.16:0/3756359160 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2348070fe0 con 0x7f23480cc4e0 2023-05-22T15:42:46.188 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:46.179+0000 7f234dad4640 1 -- 172.21.15.16:0/3756359160 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23480cce90 con 0x7f23480cc4e0 2023-05-22T15:42:46.188 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:46.179+0000 7f232ffff640 1 -- 172.21.15.16:0/3756359160 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 34) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f233c00cca0 con 0x7f23480cc4e0 2023-05-22T15:42:46.188 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:46.179+0000 7f232ffff640 1 --2- 172.21.15.16:0/3756359160 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f23340654a0 0x7f2334067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:46.188 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:46.179+0000 7f232ffff640 1 -- 172.21.15.16:0/3756359160 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 7113+0+0 (secure 0 0 0) 0x7f233c082e80 con 0x7f23480cc4e0 2023-05-22T15:42:46.188 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:46.179+0000 7f23477fe640 1 --2- 172.21.15.16:0/3756359160 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f23340654a0 0x7f2334067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:46.188 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:46.179+0000 7f23477fe640 1 --2- 172.21.15.16:0/3756359160 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f23340654a0 0x7f2334067950 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f2338000f70 tx=0x7f2338009040 comp rx=0 tx=0).ready entity=mgr.6436 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:46.189 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:46.183+0000 7f232ffff640 1 -- 172.21.15.16:0/3756359160 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f233c050660 con 0x7f23480cc4e0 2023-05-22T15:42:46.345 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:46.339+0000 7f234dad4640 1 -- 172.21.15.16:0/3756359160 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f234804dbe0 con 0x7f23480cc4e0 2023-05-22T15:42:46.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:46.991+0000 7f232ffff640 1 -- 172.21.15.16:0/3756359160 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v94) v1 ==== 181+0+0 (secure 0 0 0) 0x7f233c053c80 con 0x7f23480cc4e0 2023-05-22T15:42:46.996 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'cephfs_metadata' removed 2023-05-22T15:42:47.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:46.995+0000 7f234dad4640 1 -- 172.21.15.16:0/3756359160 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f23340654a0 msgr2=0x7f2334067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:47.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:46.995+0000 7f234dad4640 1 --2- 172.21.15.16:0/3756359160 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f23340654a0 0x7f2334067950 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f2338000f70 tx=0x7f2338009040 comp rx=0 tx=0).stop 2023-05-22T15:42:47.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:46.995+0000 7f234dad4640 1 -- 172.21.15.16:0/3756359160 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f23480cc4e0 msgr2=0x7f234814e230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:47.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:46.995+0000 7f234dad4640 1 --2- 172.21.15.16:0/3756359160 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f23480cc4e0 0x7f234814e230 secure :-1 s=READY pgs=1231 cs=0 l=1 rev1=1 crypto rx=0x7f233c00e870 tx=0x7f233c00ec70 comp rx=0 tx=0).stop 2023-05-22T15:42:47.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:46.995+0000 7f234dad4640 1 -- 172.21.15.16:0/3756359160 shutdown_connections 2023-05-22T15:42:47.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:46.995+0000 7f234dad4640 1 --2- 172.21.15.16:0/3756359160 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f23340654a0 0x7f2334067950 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:47.001 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:46.995+0000 7f234dad4640 1 --2- 172.21.15.16:0/3756359160 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f234815d650 0x7f2348150790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:47.001 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:46.995+0000 7f234dad4640 1 --2- 172.21.15.16:0/3756359160 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f23480cc4e0 0x7f234814e230 unknown :-1 s=CLOSED pgs=1231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:47.001 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:46.995+0000 7f234dad4640 1 --2- 172.21.15.16:0/3756359160 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2348070c10 0x7f23480c45d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:47.001 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:46.995+0000 7f234dad4640 1 -- 172.21.15.16:0/3756359160 >> 172.21.15.16:0/3756359160 conn(0x7f2348069720 msgr2=0x7f234806fff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:47.001 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:46.995+0000 7f234dad4640 1 -- 172.21.15.16:0/3756359160 shutdown_connections 2023-05-22T15:42:47.002 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:46.995+0000 7f234dad4640 1 -- 172.21.15.16:0/3756359160 wait complete. 2023-05-22T15:42:47.023 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-05-22T15:42:47.152 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:47.147+0000 7febfb2a7640 1 Processor -- start 2023-05-22T15:42:47.152 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:47.147+0000 7febfb2a7640 1 -- start start 2023-05-22T15:42:47.153 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:47.147+0000 7febfb2a7640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7febf4159760 0x7febf40d2050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:47.153 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:47.147+0000 7febfb2a7640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7febf40cf550 0x7febf40b7320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:47.154 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:47.147+0000 7febfb2a7640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febf40ccca0 con 0x7febf415cc60 2023-05-22T15:42:47.154 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:47.147+0000 7febfb2a7640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7febf415d210 con 0x7febf4159760 2023-05-22T15:42:47.154 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:47.147+0000 7febfb2a7640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febf40ce730 con 0x7febf40cf550 2023-05-22T15:42:47.156 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:47.147+0000 7febf8842640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7febf415cc60 0x7febf40cee40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:41104/0 (socket says 172.21.15.16:41104) 2023-05-22T15:42:47.156 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:47.147+0000 7febf8842640 1 -- 172.21.15.16:0/1795029973 learned_addr learned my addr 172.21.15.16:0/1795029973 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:42:47.156 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:47.147+0000 7febf9844640 1 --2- 172.21.15.16:0/1795029973 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7febf40cf550 0x7febf40b7320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:47.157 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:47.147+0000 7febf9043640 1 --2- 172.21.15.16:0/1795029973 >> v2:172.21.15.62:3300/0 conn(0x7febf4159760 0x7febf40d2050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:47.157 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:47.147+0000 7febebfff640 1 -- 172.21.15.16:0/1795029973 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 193592257 0 0) 0x7febf40ccca0 con 0x7febf415cc60 2023-05-22T15:42:47.157 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:47.147+0000 7febebfff640 1 -- 172.21.15.16:0/1795029973 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7febe0003530 con 0x7febf415cc60 2023-05-22T15:42:47.157 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:47.151+0000 7febebfff640 1 -- 172.21.15.16:0/1795029973 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 504653457 0 0) 0x7febe0003530 con 0x7febf415cc60 2023-05-22T15:42:47.157 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:47.151+0000 7febebfff640 1 -- 172.21.15.16:0/1795029973 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7febf40cf550 msgr2=0x7febf40b7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:42:47.158 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:47.151+0000 7febebfff640 1 --2- 172.21.15.16:0/1795029973 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7febf40cf550 0x7febf40b7320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:47.158 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:47.151+0000 7febebfff640 1 -- 172.21.15.16:0/1795029973 >> v2:172.21.15.62:3300/0 conn(0x7febf4159760 msgr2=0x7febf40d2050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:42:47.158 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:47.151+0000 7febebfff640 1 --2- 172.21.15.16:0/1795029973 >> v2:172.21.15.62:3300/0 conn(0x7febf4159760 0x7febf40d2050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:47.158 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:47.151+0000 7febebfff640 1 -- 172.21.15.16:0/1795029973 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febf40ccf80 con 0x7febf415cc60 2023-05-22T15:42:47.158 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:47.151+0000 7febebfff640 1 -- 172.21.15.16:0/1795029973 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7febe4001680 con 0x7febf415cc60 2023-05-22T15:42:47.158 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:47.151+0000 7febf9043640 1 --2- 172.21.15.16:0/1795029973 >> v2:172.21.15.62:3300/0 conn(0x7febf4159760 0x7febf40d2050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-05-22T15:42:47.158 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:47.151+0000 7febebfff640 1 -- 172.21.15.16:0/1795029973 >> v1:172.21.15.16:6789/0 conn(0x7febf415cc60 legacy=0x7febf40cee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:47.159 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:47.151+0000 7febebfff640 1 --2- 172.21.15.16:0/1795029973 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7febe0003980 0x7febe0003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:47.159 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:47.151+0000 7febebfff640 1 --2- 172.21.15.16:0/1795029973 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7febe0004360 0x7febe0004830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:47.159 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:47.151+0000 7febebfff640 1 --2- 172.21.15.16:0/1795029973 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7febe0004ea0 0x7febe000b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:47.159 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:47.151+0000 7febebfff640 1 -- 172.21.15.16:0/1795029973 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febf415d210 con 0x7febe0004360 2023-05-22T15:42:47.159 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:47.151+0000 7febebfff640 1 -- 172.21.15.16:0/1795029973 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febf40ce730 con 0x7febe0004ea0 2023-05-22T15:42:47.159 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:47.151+0000 7febebfff640 1 -- 172.21.15.16:0/1795029973 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febe000bed0 con 0x7febe0003980 2023-05-22T15:42:47.159 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:47.151+0000 7febf9844640 1 --2- 172.21.15.16:0/1795029973 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7febf40cf550 0x7febf40b7320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-05-22T15:42:47.160 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:47.151+0000 7febf8842640 1 --2- 172.21.15.16:0/1795029973 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7febe0004360 0x7febe0004830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:47.160 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:47.151+0000 7febf9043640 1 --2- 172.21.15.16:0/1795029973 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7febe0003980 0x7febe0003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:47.160 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:47.151+0000 7febf9844640 1 --2- 172.21.15.16:0/1795029973 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7febe0004ea0 0x7febe000b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:47.160 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:47.151+0000 7febf8842640 1 -- 172.21.15.16:0/1795029973 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7febe0003980 msgr2=0x7febe0003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:42:47.160 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:47.151+0000 7febf8842640 1 --2- 172.21.15.16:0/1795029973 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7febe0003980 0x7febe0003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:47.160 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:47.151+0000 7febf8842640 1 -- 172.21.15.16:0/1795029973 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7febe0004ea0 msgr2=0x7febe000b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:42:47.160 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:47.151+0000 7febf8842640 1 --2- 172.21.15.16:0/1795029973 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7febe0004ea0 0x7febe000b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:47.161 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:47.151+0000 7febf8842640 1 -- 172.21.15.16:0/1795029973 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febe0003530 con 0x7febe0004360 2023-05-22T15:42:47.161 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:47.151+0000 7febf9844640 1 --2- 172.21.15.16:0/1795029973 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7febe0004ea0 0x7febe000b990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:42:47.163 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:47.151+0000 7febf8842640 1 --2- 172.21.15.16:0/1795029973 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7febe0004360 0x7febe0004830 secure :-1 s=READY pgs=1233 cs=0 l=1 rev1=1 crypto rx=0x7febe400c240 tx=0x7febe401f3d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc7d809335167a44 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:47.163 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:47.151+0000 7febebfff640 1 -- 172.21.15.16:0/1795029973 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febe4005ae0 con 0x7febe0004360 2023-05-22T15:42:47.163 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:47.151+0000 7febebfff640 1 -- 172.21.15.16:0/1795029973 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febe40ab070 con 0x7febe0004360 2023-05-22T15:42:47.163 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:47.151+0000 7febebfff640 1 -- 172.21.15.16:0/1795029973 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febe4007070 con 0x7febe0004360 2023-05-22T15:42:47.164 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:47.151+0000 7febfb2a7640 1 -- 172.21.15.16:0/1795029973 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7febe0004360 msgr2=0x7febe0004830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:47.164 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:47.151+0000 7febfb2a7640 1 --2- 172.21.15.16:0/1795029973 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7febe0004360 0x7febe0004830 secure :-1 s=READY pgs=1233 cs=0 l=1 rev1=1 crypto rx=0x7febe400c240 tx=0x7febe401f3d0 comp rx=0 tx=0).stop 2023-05-22T15:42:47.164 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:47.155+0000 7febf9844640 1 -- 172.21.15.16:0/1795029973 reap_dead start 2023-05-22T15:42:47.164 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:47.155+0000 7febfb2a7640 1 -- 172.21.15.16:0/1795029973 shutdown_connections 2023-05-22T15:42:47.164 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:47.155+0000 7febfb2a7640 1 -- 172.21.15.16:0/1795029973 >> 172.21.15.16:0/1795029973 conn(0x7febf4070e50 msgr2=0x7febf4071250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:47.164 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:47.155+0000 7febfb2a7640 1 -- 172.21.15.16:0/1795029973 shutdown_connections 2023-05-22T15:42:47.164 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:47.155+0000 7febfb2a7640 1 -- 172.21.15.16:0/1795029973 wait complete. 2023-05-22T15:42:47.165 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:47.155+0000 7febfb2a7640 1 Processor -- start 2023-05-22T15:42:47.165 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:47.155+0000 7febfb2a7640 1 -- start start 2023-05-22T15:42:47.165 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:47.155+0000 7febfb2a7640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7febe0004360 0x7febf415f3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:47.165 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:47.155+0000 7febfb2a7640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7febf415ba20 0x7febf40b6ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:47.165 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:47.155+0000 7febfb2a7640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7febf40c4e00 0x7febf40c5270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:47.165 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:47.155+0000 7febfb2a7640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febf41565b0 con 0x7febf40c4e00 2023-05-22T15:42:47.165 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:47.155+0000 7febfb2a7640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febf40c57b0 con 0x7febf415ba20 2023-05-22T15:42:47.166 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:47.155+0000 7febfb2a7640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febf40c9620 con 0x7febe0004360 2023-05-22T15:42:47.166 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:47.155+0000 7febf9844640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7febf40c4e00 0x7febf40c5270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:47.166 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:47.155+0000 7febf9844640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7febf40c4e00 0x7febf40c5270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49378/0 (socket says 172.21.15.16:49378) 2023-05-22T15:42:47.166 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:47.155+0000 7febf9844640 1 -- 172.21.15.16:0/1212844784 learned_addr learned my addr 172.21.15.16:0/1212844784 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:42:47.166 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:47.155+0000 7febf9043640 1 --2- 172.21.15.16:0/1212844784 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7febe0004360 0x7febf415f3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:47.166 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:47.155+0000 7febf8842640 1 --2- 172.21.15.16:0/1212844784 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7febf415ba20 0x7febf40b6ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:47.166 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:47.155+0000 7febf9844640 1 -- 172.21.15.16:0/1212844784 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7febe0004360 msgr2=0x7febf415f3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:47.166 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:47.155+0000 7febf9844640 1 --2- 172.21.15.16:0/1212844784 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7febe0004360 0x7febf415f3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:47.167 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:47.155+0000 7febf9844640 1 -- 172.21.15.16:0/1212844784 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7febf415ba20 msgr2=0x7febf40b6ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:47.167 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:47.155+0000 7febf9844640 1 --2- 172.21.15.16:0/1212844784 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7febf415ba20 0x7febf40b6ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:47.167 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:47.155+0000 7febf9844640 1 -- 172.21.15.16:0/1212844784 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febf40cd260 con 0x7febf40c4e00 2023-05-22T15:42:47.167 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:47.155+0000 7febf8842640 1 --2- 172.21.15.16:0/1212844784 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7febf415ba20 0x7febf40b6ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T15:42:47.167 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:47.155+0000 7febf9844640 1 --2- 172.21.15.16:0/1212844784 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7febf40c4e00 0x7febf40c5270 secure :-1 s=READY pgs=1234 cs=0 l=1 rev1=1 crypto rx=0x7febf40b4950 tx=0x7febec00f3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:47.167 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:47.155+0000 7febe9ffb640 1 -- 172.21.15.16:0/1212844784 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febec003db0 con 0x7febf40c4e00 2023-05-22T15:42:47.168 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:47.155+0000 7febe9ffb640 1 -- 172.21.15.16:0/1212844784 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febec011040 con 0x7febf40c4e00 2023-05-22T15:42:47.168 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:47.155+0000 7febfb2a7640 1 -- 172.21.15.16:0/1212844784 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7febf40ce940 con 0x7febf40c4e00 2023-05-22T15:42:47.168 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:47.155+0000 7febe9ffb640 1 -- 172.21.15.16:0/1212844784 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febec004250 con 0x7febf40c4e00 2023-05-22T15:42:47.168 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:47.155+0000 7febfb2a7640 1 -- 172.21.15.16:0/1212844784 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7febf4159b30 con 0x7febf40c4e00 2023-05-22T15:42:47.168 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:47.159+0000 7febfb2a7640 1 -- 172.21.15.16:0/1212844784 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7febf40c97e0 con 0x7febf40c4e00 2023-05-22T15:42:47.168 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:47.159+0000 7febe9ffb640 1 -- 172.21.15.16:0/1212844784 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 34) v1 ==== 84332+0+0 (secure 0 0 0) 0x7febec0043b0 con 0x7febf40c4e00 2023-05-22T15:42:47.169 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:47.159+0000 7febe9ffb640 1 --2- 172.21.15.16:0/1212844784 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7febd80654a0 0x7febd8067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:47.169 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:47.159+0000 7febe9ffb640 1 -- 172.21.15.16:0/1212844784 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6675+0+0 (secure 0 0 0) 0x7febec082780 con 0x7febf40c4e00 2023-05-22T15:42:47.169 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:47.159+0000 7febf9043640 1 --2- 172.21.15.16:0/1212844784 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7febd80654a0 0x7febd8067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:47.170 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:47.159+0000 7febf9043640 1 --2- 172.21.15.16:0/1212844784 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7febd80654a0 0x7febd8067950 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7febf00093f0 tx=0x7febf0008040 comp rx=0 tx=0).ready entity=mgr.6436 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:47.170 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:47.163+0000 7febe9ffb640 1 -- 172.21.15.16:0/1212844784 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7febec050020 con 0x7febf40c4e00 2023-05-22T15:42:47.321 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:47.315+0000 7febfb2a7640 1 -- 172.21.15.16:0/1212844784 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7febf404dbe0 con 0x7febf40c4e00 2023-05-22T15:42:48.017 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.011+0000 7febe9ffb640 1 -- 172.21.15.16:0/1212844784 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v95) v1 ==== 169+0+0 (secure 0 0 0) 0x7febec053640 con 0x7febf40c4e00 2023-05-22T15:42:48.017 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'cephfs_data' removed 2023-05-22T15:42:48.023 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.019+0000 7febfb2a7640 1 -- 172.21.15.16:0/1212844784 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7febd80654a0 msgr2=0x7febd8067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:48.023 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.019+0000 7febfb2a7640 1 --2- 172.21.15.16:0/1212844784 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7febd80654a0 0x7febd8067950 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7febf00093f0 tx=0x7febf0008040 comp rx=0 tx=0).stop 2023-05-22T15:42:48.023 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.019+0000 7febfb2a7640 1 -- 172.21.15.16:0/1212844784 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7febf40c4e00 msgr2=0x7febf40c5270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:48.023 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.019+0000 7febfb2a7640 1 --2- 172.21.15.16:0/1212844784 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7febf40c4e00 0x7febf40c5270 secure :-1 s=READY pgs=1234 cs=0 l=1 rev1=1 crypto rx=0x7febf40b4950 tx=0x7febec00f3c0 comp rx=0 tx=0).stop 2023-05-22T15:42:48.024 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.019+0000 7febfb2a7640 1 -- 172.21.15.16:0/1212844784 shutdown_connections 2023-05-22T15:42:48.024 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.019+0000 7febfb2a7640 1 --2- 172.21.15.16:0/1212844784 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7febf40c4e00 0x7febf40c5270 unknown :-1 s=CLOSED pgs=1234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:48.025 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.019+0000 7febfb2a7640 1 --2- 172.21.15.16:0/1212844784 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7febf415ba20 0x7febf40b6ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:48.025 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.019+0000 7febfb2a7640 1 --2- 172.21.15.16:0/1212844784 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7febd80654a0 0x7febd8067950 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:48.025 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.019+0000 7febfb2a7640 1 --2- 172.21.15.16:0/1212844784 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7febe0004360 0x7febf415f3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:48.025 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.019+0000 7febfb2a7640 1 -- 172.21.15.16:0/1212844784 >> 172.21.15.16:0/1212844784 conn(0x7febf4070e50 msgr2=0x7febf4159dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:48.026 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.023+0000 7febfb2a7640 1 -- 172.21.15.16:0/1212844784 shutdown_connections 2023-05-22T15:42:48.026 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.023+0000 7febfb2a7640 1 -- 172.21.15.16:0/1212844784 wait complete. 2023-05-22T15:42:48.051 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2023-05-22T15:42:48.181 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.175+0000 7f53ab916640 1 Processor -- start 2023-05-22T15:42:48.181 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.175+0000 7f53ab916640 1 -- start start 2023-05-22T15:42:48.182 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.175+0000 7f53ab916640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f53a406b6c0 0x7f53a40cb4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:48.182 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.175+0000 7f53ab916640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53a4159f70 con 0x7f53a406ba90 2023-05-22T15:42:48.182 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.175+0000 7f53ab916640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53a40cb030 con 0x7f53a41596e0 2023-05-22T15:42:48.183 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.175+0000 7f53ab916640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53a4071760 con 0x7f53a406b6c0 2023-05-22T15:42:48.183 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.175+0000 7f53a8eb1640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f53a406ba90 0x7f53a4158fd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:41108/0 (socket says 172.21.15.16:41108) 2023-05-22T15:42:48.183 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.175+0000 7f53a8eb1640 1 -- 172.21.15.16:0/2948694752 learned_addr learned my addr 172.21.15.16:0/2948694752 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:42:48.183 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.179+0000 7f53a96b2640 1 --2- 172.21.15.16:0/2948694752 >> v2:172.21.15.62:3300/0 conn(0x7f53a406b6c0 0x7f53a40cb4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:48.183 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.179+0000 7f539bfff640 1 -- 172.21.15.16:0/2948694752 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3707193304 0 0) 0x7f53a4159f70 con 0x7f53a406ba90 2023-05-22T15:42:48.183 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.179+0000 7f539bfff640 1 -- 172.21.15.16:0/2948694752 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5390003530 con 0x7f53a406ba90 2023-05-22T15:42:48.185 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.179+0000 7f539bfff640 1 -- 172.21.15.16:0/2948694752 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2494052063 0 0) 0x7f5390003530 con 0x7f53a406ba90 2023-05-22T15:42:48.186 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.179+0000 7f539bfff640 1 -- 172.21.15.16:0/2948694752 >> v2:172.21.15.62:3300/0 conn(0x7f53a406b6c0 msgr2=0x7f53a40cb4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:42:48.186 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.179+0000 7f539bfff640 1 --2- 172.21.15.16:0/2948694752 >> v2:172.21.15.62:3300/0 conn(0x7f53a406b6c0 0x7f53a40cb4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:48.186 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.179+0000 7f539bfff640 1 -- 172.21.15.16:0/2948694752 >> v1:172.21.15.62:6789/0 conn(0x7f53a41596e0 legacy=0x7f53a406fba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:48.186 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.179+0000 7f539bfff640 1 -- 172.21.15.16:0/2948694752 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53a4159c90 con 0x7f53a406ba90 2023-05-22T15:42:48.186 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.179+0000 7f539bfff640 1 -- 172.21.15.16:0/2948694752 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7f5394001680 con 0x7f53a406ba90 2023-05-22T15:42:48.186 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.179+0000 7f539bfff640 1 -- 172.21.15.16:0/2948694752 >> v1:172.21.15.16:6789/0 conn(0x7f53a406ba90 legacy=0x7f53a4158fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:48.187 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.179+0000 7f539bfff640 1 --2- 172.21.15.16:0/2948694752 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5390003980 0x7f5390003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:48.187 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.179+0000 7f539bfff640 1 --2- 172.21.15.16:0/2948694752 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f53900047d0 0x7f5390004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:48.187 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.179+0000 7f539bfff640 1 --2- 172.21.15.16:0/2948694752 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5390005180 0x7f539000b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:48.187 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.179+0000 7f539bfff640 1 -- 172.21.15.16:0/2948694752 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53a4071760 con 0x7f53900047d0 2023-05-22T15:42:48.187 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.179+0000 7f539bfff640 1 -- 172.21.15.16:0/2948694752 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f539000c090 con 0x7f5390005180 2023-05-22T15:42:48.187 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.179+0000 7f539bfff640 1 -- 172.21.15.16:0/2948694752 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f539000c250 con 0x7f5390003980 2023-05-22T15:42:48.187 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.179+0000 7f53a8eb1640 1 --2- 172.21.15.16:0/2948694752 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f53900047d0 0x7f5390004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:48.188 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.179+0000 7f53a9eb3640 1 --2- 172.21.15.16:0/2948694752 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5390005180 0x7f539000b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:48.188 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.179+0000 7f53a96b2640 1 --2- 172.21.15.16:0/2948694752 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5390003980 0x7f5390003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:48.188 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.179+0000 7f53a8eb1640 1 -- 172.21.15.16:0/2948694752 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5390003980 msgr2=0x7f5390003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:42:48.189 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.179+0000 7f53a8eb1640 1 --2- 172.21.15.16:0/2948694752 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5390003980 0x7f5390003d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:48.189 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.179+0000 7f53a8eb1640 1 -- 172.21.15.16:0/2948694752 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5390005180 msgr2=0x7f539000b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:42:48.189 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.179+0000 7f53a8eb1640 1 --2- 172.21.15.16:0/2948694752 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5390005180 0x7f539000b990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:48.189 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.179+0000 7f53a9eb3640 1 -- 172.21.15.16:0/2948694752 reap_dead start 2023-05-22T15:42:48.189 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.179+0000 7f53a8eb1640 1 -- 172.21.15.16:0/2948694752 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5390003530 con 0x7f53900047d0 2023-05-22T15:42:48.189 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.179+0000 7f53a8eb1640 1 --2- 172.21.15.16:0/2948694752 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f53900047d0 0x7f5390004c40 secure :-1 s=READY pgs=1236 cs=0 l=1 rev1=1 crypto rx=0x7f539400c240 tx=0x7f539401cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5eba535d3a86594b server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:48.189 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.179+0000 7f539bfff640 1 -- 172.21.15.16:0/2948694752 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5394005ae0 con 0x7f53900047d0 2023-05-22T15:42:48.190 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.179+0000 7f539bfff640 1 -- 172.21.15.16:0/2948694752 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5394005c40 con 0x7f53900047d0 2023-05-22T15:42:48.190 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.179+0000 7f539bfff640 1 -- 172.21.15.16:0/2948694752 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53940072b0 con 0x7f53900047d0 2023-05-22T15:42:48.193 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.179+0000 7f53ab916640 1 -- 172.21.15.16:0/2948694752 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f53900047d0 msgr2=0x7f5390004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:48.193 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.179+0000 7f53ab916640 1 --2- 172.21.15.16:0/2948694752 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f53900047d0 0x7f5390004c40 secure :-1 s=READY pgs=1236 cs=0 l=1 rev1=1 crypto rx=0x7f539400c240 tx=0x7f539401cc40 comp rx=0 tx=0).stop 2023-05-22T15:42:48.193 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.183+0000 7f53ab916640 1 -- 172.21.15.16:0/2948694752 shutdown_connections 2023-05-22T15:42:48.193 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.183+0000 7f53ab916640 1 --2- 172.21.15.16:0/2948694752 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f53900047d0 0x7f5390004c40 unknown :-1 s=CLOSED pgs=1236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:48.194 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.183+0000 7f53ab916640 1 -- 172.21.15.16:0/2948694752 >> 172.21.15.16:0/2948694752 conn(0x7f53a4069720 msgr2=0x7f53a4069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:48.194 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.183+0000 7f53ab916640 1 -- 172.21.15.16:0/2948694752 shutdown_connections 2023-05-22T15:42:48.194 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.183+0000 7f53ab916640 1 -- 172.21.15.16:0/2948694752 wait complete. 2023-05-22T15:42:48.194 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.183+0000 7f53ab916640 1 Processor -- start 2023-05-22T15:42:48.194 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.183+0000 7f53ab916640 1 -- start start 2023-05-22T15:42:48.194 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.183+0000 7f53ab916640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f53900047d0 0x7f53a4152af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:48.194 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.183+0000 7f53ab916640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f53a41505e0 0x7f53a40b36b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:48.195 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.183+0000 7f53ab916640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f53a40b3bf0 0x7f53a4157870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:48.195 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.183+0000 7f53ab916640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53a40b2650 con 0x7f53a41505e0 2023-05-22T15:42:48.195 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.183+0000 7f53ab916640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53a40b28f0 con 0x7f53900047d0 2023-05-22T15:42:48.195 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.183+0000 7f53ab916640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53a40b2bd0 con 0x7f53a40b3bf0 2023-05-22T15:42:48.195 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.183+0000 7f53a8eb1640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f53a41505e0 0x7f53a40b36b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:48.195 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.183+0000 7f53a8eb1640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f53a41505e0 0x7f53a40b36b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49398/0 (socket says 172.21.15.16:49398) 2023-05-22T15:42:48.195 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.183+0000 7f53a8eb1640 1 -- 172.21.15.16:0/2998960338 learned_addr learned my addr 172.21.15.16:0/2998960338 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:42:48.196 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.183+0000 7f53a96b2640 1 --2- 172.21.15.16:0/2998960338 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f53900047d0 0x7f53a4152af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:48.196 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.183+0000 7f53a9eb3640 1 --2- 172.21.15.16:0/2998960338 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f53a40b3bf0 0x7f53a4157870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:48.196 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.183+0000 7f53a8eb1640 1 -- 172.21.15.16:0/2998960338 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f53a40b3bf0 msgr2=0x7f53a4157870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:48.196 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.183+0000 7f53a8eb1640 1 --2- 172.21.15.16:0/2998960338 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f53a40b3bf0 0x7f53a4157870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:48.196 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.183+0000 7f53a8eb1640 1 -- 172.21.15.16:0/2998960338 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f53900047d0 msgr2=0x7f53a4152af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:48.196 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.183+0000 7f53a8eb1640 1 --2- 172.21.15.16:0/2998960338 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f53900047d0 0x7f53a4152af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:48.197 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.183+0000 7f53a8eb1640 1 -- 172.21.15.16:0/2998960338 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53a406bea0 con 0x7f53a41505e0 2023-05-22T15:42:48.197 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.183+0000 7f53a8eb1640 1 --2- 172.21.15.16:0/2998960338 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f53a41505e0 0x7f53a40b36b0 secure :-1 s=READY pgs=1237 cs=0 l=1 rev1=1 crypto rx=0x7f539401ca70 tx=0x7f5390005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:48.197 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.183+0000 7f5399ffb640 1 -- 172.21.15.16:0/2998960338 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53940b4070 con 0x7f53a41505e0 2023-05-22T15:42:48.197 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.183+0000 7f5399ffb640 1 -- 172.21.15.16:0/2998960338 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53940015c0 con 0x7f53a41505e0 2023-05-22T15:42:48.197 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.183+0000 7f53ab916640 1 -- 172.21.15.16:0/2998960338 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53a41593e0 con 0x7f53a41505e0 2023-05-22T15:42:48.198 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.183+0000 7f5399ffb640 1 -- 172.21.15.16:0/2998960338 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5394007290 con 0x7f53a41505e0 2023-05-22T15:42:48.198 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.183+0000 7f53ab916640 1 -- 172.21.15.16:0/2998960338 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53a406c030 con 0x7f53a41505e0 2023-05-22T15:42:48.198 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.187+0000 7f53ab916640 1 -- 172.21.15.16:0/2998960338 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53a404dbe0 con 0x7f53a41505e0 2023-05-22T15:42:48.198 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.187+0000 7f5399ffb640 1 -- 172.21.15.16:0/2998960338 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 34) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f53940073f0 con 0x7f53a41505e0 2023-05-22T15:42:48.198 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.187+0000 7f5399ffb640 1 --2- 172.21.15.16:0/2998960338 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f53880654a0 0x7f5388067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:48.198 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.187+0000 7f5399ffb640 1 -- 172.21.15.16:0/2998960338 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 6261+0+0 (secure 0 0 0) 0x7f539411c790 con 0x7f53a41505e0 2023-05-22T15:42:48.199 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.187+0000 7f53a96b2640 1 --2- 172.21.15.16:0/2998960338 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f53880654a0 0x7f5388067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:48.199 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.187+0000 7f53a96b2640 1 --2- 172.21.15.16:0/2998960338 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f53880654a0 0x7f5388067950 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f53a0009ab0 tx=0x7f53a0008040 comp rx=0 tx=0).ready entity=mgr.6436 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:48.199 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.191+0000 7f5399ffb640 1 -- 172.21.15.16:0/2998960338 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f53940ea2c0 con 0x7f53a41505e0 2023-05-22T15:42:48.353 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.347+0000 7f53ab916640 1 -- 172.21.15.16:0/2998960338 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7f53a40cc0e0 con 0x7f53a41505e0 2023-05-22T15:42:48.359 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.355+0000 7f5399ffb640 1 -- 172.21.15.16:0/2998960338 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v25) v1 ==== 120+0+0 (secure 0 0 0) 0x7f53940ed8e0 con 0x7f53a41505e0 2023-05-22T15:42:48.362 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.355+0000 7f53ab916640 1 -- 172.21.15.16:0/2998960338 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f53880654a0 msgr2=0x7f5388067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:48.362 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.355+0000 7f53ab916640 1 --2- 172.21.15.16:0/2998960338 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f53880654a0 0x7f5388067950 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f53a0009ab0 tx=0x7f53a0008040 comp rx=0 tx=0).stop 2023-05-22T15:42:48.362 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.355+0000 7f53ab916640 1 -- 172.21.15.16:0/2998960338 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f53a41505e0 msgr2=0x7f53a40b36b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:48.362 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.355+0000 7f53ab916640 1 --2- 172.21.15.16:0/2998960338 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f53a41505e0 0x7f53a40b36b0 secure :-1 s=READY pgs=1237 cs=0 l=1 rev1=1 crypto rx=0x7f539401ca70 tx=0x7f5390005800 comp rx=0 tx=0).stop 2023-05-22T15:42:48.362 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.355+0000 7f53ab916640 1 -- 172.21.15.16:0/2998960338 shutdown_connections 2023-05-22T15:42:48.362 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.355+0000 7f53ab916640 1 --2- 172.21.15.16:0/2998960338 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f53880654a0 0x7f5388067950 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:48.363 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.355+0000 7f53ab916640 1 --2- 172.21.15.16:0/2998960338 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f53a40b3bf0 0x7f53a4157870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:48.363 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.355+0000 7f53ab916640 1 --2- 172.21.15.16:0/2998960338 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f53a41505e0 0x7f53a40b36b0 unknown :-1 s=CLOSED pgs=1237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:48.363 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.355+0000 7f53ab916640 1 --2- 172.21.15.16:0/2998960338 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f53900047d0 0x7f53a4152af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:48.363 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.355+0000 7f53ab916640 1 -- 172.21.15.16:0/2998960338 >> 172.21.15.16:0/2998960338 conn(0x7f53a4069720 msgr2=0x7f53a40cdff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:48.363 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.355+0000 7f53ab916640 1 -- 172.21.15.16:0/2998960338 shutdown_connections 2023-05-22T15:42:48.363 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.355+0000 7f53ab916640 1 -- 172.21.15.16:0/2998960338 wait complete. 2023-05-22T15:42:48.380 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2023-05-22T15:42:48.506 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.499+0000 7fb396543640 1 Processor -- start 2023-05-22T15:42:48.506 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.503+0000 7fb396543640 1 -- start start 2023-05-22T15:42:48.507 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.503+0000 7fb396543640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fb39006db10 0x7fb39006dee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:48.507 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.503+0000 7fb396543640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb39006c5f0 0x7fb3900cc7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:48.507 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.503+0000 7fb396543640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb39015bfc0 con 0x7fb39006a830 2023-05-22T15:42:48.508 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.503+0000 7fb396543640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb39015f790 con 0x7fb39006db10 2023-05-22T15:42:48.508 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.503+0000 7fb396543640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb39006e420 con 0x7fb39006c5f0 2023-05-22T15:42:48.509 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.503+0000 7fb38f7fe640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb39006c5f0 0x7fb3900cc7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:48.509 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.503+0000 7fb38ffff640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fb39006db10 0x7fb39006dee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:48.509 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.503+0000 7fb38ffff640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fb39006db10 0x7fb39006dee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.16:43206/0 (socket says 172.21.15.16:43206) 2023-05-22T15:42:48.509 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.503+0000 7fb38f7fe640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb39006c5f0 0x7fb3900cc7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3301/0 says I am v2:172.21.15.16:60814/0 (socket says 172.21.15.16:60814) 2023-05-22T15:42:48.510 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.503+0000 7fb38ffff640 1 -- 172.21.15.16:0/1291024977 learned_addr learned my addr 172.21.15.16:0/1291024977 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:42:48.510 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.503+0000 7fb394ae0640 1 --1- 172.21.15.16:0/1291024977 >> v1:172.21.15.62:6789/0 conn(0x7fb39006a830 0x7fb39015fbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.16:52278/0 (socket says 172.21.15.16:52278) 2023-05-22T15:42:48.510 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.503+0000 7fb38ffff640 1 -- 172.21.15.16:0/1291024977 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb39006c5f0 msgr2=0x7fb3900cc7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:42:48.510 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.503+0000 7fb38ffff640 1 --2- 172.21.15.16:0/1291024977 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb39006c5f0 0x7fb3900cc7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:48.510 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.503+0000 7fb38ffff640 1 -- 172.21.15.16:0/1291024977 >> v1:172.21.15.62:6789/0 conn(0x7fb39006a830 legacy=0x7fb39015fbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:48.510 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.503+0000 7fb38ffff640 1 -- 172.21.15.16:0/1291024977 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3900ccce0 con 0x7fb39006db10 2023-05-22T15:42:48.511 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.503+0000 7fb38f7fe640 1 --2- 172.21.15.16:0/1291024977 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb39006c5f0 0x7fb3900cc7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-05-22T15:42:48.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.507+0000 7fb38ffff640 1 --2- 172.21.15.16:0/1291024977 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb39006db10 0x7fb39006dee0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fb380009e50 tx=0x7fb38001c9e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a769ae4e78c1471b server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:48.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.507+0000 7fb38effd640 1 -- 172.21.15.16:0/1291024977 <== mon.1 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3800a58a0 con 0x7fb39006db10 2023-05-22T15:42:48.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.507+0000 7fb38effd640 1 -- 172.21.15.16:0/1291024977 <== mon.1 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3800a5a00 con 0x7fb39006db10 2023-05-22T15:42:48.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.507+0000 7fb38effd640 1 -- 172.21.15.16:0/1291024977 <== mon.1 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3800a5d90 con 0x7fb39006db10 2023-05-22T15:42:48.513 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.507+0000 7fb396543640 1 -- 172.21.15.16:0/1291024977 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb39006db10 msgr2=0x7fb39006dee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:48.513 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.507+0000 7fb396543640 1 --2- 172.21.15.16:0/1291024977 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb39006db10 0x7fb39006dee0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fb380009e50 tx=0x7fb38001c9e0 comp rx=0 tx=0).stop 2023-05-22T15:42:48.513 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.507+0000 7fb396543640 1 -- 172.21.15.16:0/1291024977 shutdown_connections 2023-05-22T15:42:48.513 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.507+0000 7fb396543640 1 --2- 172.21.15.16:0/1291024977 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb39006c5f0 0x7fb3900cc7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:48.513 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.507+0000 7fb396543640 1 --2- 172.21.15.16:0/1291024977 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb39006db10 0x7fb39006dee0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:48.513 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.507+0000 7fb396543640 1 -- 172.21.15.16:0/1291024977 >> 172.21.15.16:0/1291024977 conn(0x7fb39006f4f0 msgr2=0x7fb39006f8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:48.513 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.507+0000 7fb396543640 1 -- 172.21.15.16:0/1291024977 shutdown_connections 2023-05-22T15:42:48.514 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.507+0000 7fb396543640 1 -- 172.21.15.16:0/1291024977 wait complete. 2023-05-22T15:42:48.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.507+0000 7fb396543640 1 Processor -- start 2023-05-22T15:42:48.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.507+0000 7fb396543640 1 -- start start 2023-05-22T15:42:48.516 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.507+0000 7fb396543640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb39006a830 0x7fb39015d5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:48.516 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.507+0000 7fb396543640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb39006c5f0 0x7fb3901571d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:48.516 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.507+0000 7fb396543640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb39006db10 0x7fb3900c8600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:48.516 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.507+0000 7fb396543640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb39015db20 con 0x7fb39006c5f0 2023-05-22T15:42:48.516 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.507+0000 7fb396543640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb39015c540 con 0x7fb39006db10 2023-05-22T15:42:48.516 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.507+0000 7fb396543640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb39015c820 con 0x7fb39006a830 2023-05-22T15:42:48.516 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.511+0000 7fb38f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb39006c5f0 0x7fb3901571d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:48.517 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.511+0000 7fb38f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb39006c5f0 0x7fb3901571d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49400/0 (socket says 172.21.15.16:49400) 2023-05-22T15:42:48.517 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.511+0000 7fb38f7fe640 1 -- 172.21.15.16:0/2569914555 learned_addr learned my addr 172.21.15.16:0/2569914555 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:42:48.517 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.511+0000 7fb38ffff640 1 --2- 172.21.15.16:0/2569914555 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb39006a830 0x7fb39015d5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:48.517 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.511+0000 7fb394ae0640 1 --2- 172.21.15.16:0/2569914555 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb39006db10 0x7fb3900c8600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:48.517 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.511+0000 7fb38f7fe640 1 -- 172.21.15.16:0/2569914555 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb39006a830 msgr2=0x7fb39015d5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:48.517 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.511+0000 7fb38f7fe640 1 --2- 172.21.15.16:0/2569914555 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb39006a830 0x7fb39015d5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:48.517 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.511+0000 7fb38f7fe640 1 -- 172.21.15.16:0/2569914555 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb39006db10 msgr2=0x7fb3900c8600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:48.518 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.511+0000 7fb38f7fe640 1 --2- 172.21.15.16:0/2569914555 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb39006db10 0x7fb3900c8600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:48.518 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.511+0000 7fb38f7fe640 1 -- 172.21.15.16:0/2569914555 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb384009420 con 0x7fb39006c5f0 2023-05-22T15:42:48.518 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.511+0000 7fb38f7fe640 1 --2- 172.21.15.16:0/2569914555 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb39006c5f0 0x7fb3901571d0 secure :-1 s=READY pgs=1238 cs=0 l=1 rev1=1 crypto rx=0x7fb38400ebe0 tx=0x7fb38400c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:48.518 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.511+0000 7fb38cff9640 1 -- 172.21.15.16:0/2569914555 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb38400eee0 con 0x7fb39006c5f0 2023-05-22T15:42:48.518 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.511+0000 7fb38cff9640 1 -- 172.21.15.16:0/2569914555 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb38400cc30 con 0x7fb39006c5f0 2023-05-22T15:42:48.518 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.511+0000 7fb396543640 1 -- 172.21.15.16:0/2569914555 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb380009b30 con 0x7fb39006c5f0 2023-05-22T15:42:48.518 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.511+0000 7fb38cff9640 1 -- 172.21.15.16:0/2569914555 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb384013470 con 0x7fb39006c5f0 2023-05-22T15:42:48.519 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.511+0000 7fb396543640 1 -- 172.21.15.16:0/2569914555 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3900cfec0 con 0x7fb39006c5f0 2023-05-22T15:42:48.519 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.511+0000 7fb396543640 1 -- 172.21.15.16:0/2569914555 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3900cfbe0 con 0x7fb39006c5f0 2023-05-22T15:42:48.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.511+0000 7fb38cff9640 1 -- 172.21.15.16:0/2569914555 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 34) v1 ==== 84332+0+0 (secure 0 0 0) 0x7fb3840135d0 con 0x7fb39006c5f0 2023-05-22T15:42:48.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.511+0000 7fb38cff9640 1 --2- 172.21.15.16:0/2569914555 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fb37c0654a0 0x7fb37c067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:48.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.511+0000 7fb38ffff640 1 --2- 172.21.15.16:0/2569914555 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fb37c0654a0 0x7fb37c067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:48.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.511+0000 7fb38cff9640 1 -- 172.21.15.16:0/2569914555 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 6261+0+0 (secure 0 0 0) 0x7fb384082d30 con 0x7fb39006c5f0 2023-05-22T15:42:48.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.511+0000 7fb38ffff640 1 --2- 172.21.15.16:0/2569914555 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fb37c0654a0 0x7fb37c067950 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fb3800ae000 tx=0x7fb3800a7070 comp rx=0 tx=0).ready entity=mgr.6436 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:48.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.515+0000 7fb38cff9640 1 -- 172.21.15.16:0/2569914555 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fb384054210 con 0x7fb39006c5f0 2023-05-22T15:42:48.674 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.671+0000 7fb396543640 1 -- 172.21.15.16:0/2569914555 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fb3900cf260 con 0x7fb39006c5f0 2023-05-22T15:42:48.680 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.675+0000 7fb38cff9640 1 -- 172.21.15.16:0/2569914555 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v26) v1 ==== 104+0+0 (secure 0 0 0) 0x7fb384053e00 con 0x7fb39006c5f0 2023-05-22T15:42:48.683 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.679+0000 7fb396543640 1 -- 172.21.15.16:0/2569914555 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fb37c0654a0 msgr2=0x7fb37c067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:48.683 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.679+0000 7fb396543640 1 --2- 172.21.15.16:0/2569914555 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fb37c0654a0 0x7fb37c067950 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fb3800ae000 tx=0x7fb3800a7070 comp rx=0 tx=0).stop 2023-05-22T15:42:48.683 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.679+0000 7fb396543640 1 -- 172.21.15.16:0/2569914555 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb39006c5f0 msgr2=0x7fb3901571d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:48.683 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.679+0000 7fb396543640 1 --2- 172.21.15.16:0/2569914555 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb39006c5f0 0x7fb3901571d0 secure :-1 s=READY pgs=1238 cs=0 l=1 rev1=1 crypto rx=0x7fb38400ebe0 tx=0x7fb38400c520 comp rx=0 tx=0).stop 2023-05-22T15:42:48.684 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.679+0000 7fb396543640 1 -- 172.21.15.16:0/2569914555 shutdown_connections 2023-05-22T15:42:48.684 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.679+0000 7fb396543640 1 --2- 172.21.15.16:0/2569914555 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb39006db10 0x7fb3900c8600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:48.684 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.679+0000 7fb396543640 1 --2- 172.21.15.16:0/2569914555 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb39006c5f0 0x7fb3901571d0 unknown :-1 s=CLOSED pgs=1238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:48.684 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.679+0000 7fb396543640 1 --2- 172.21.15.16:0/2569914555 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fb37c0654a0 0x7fb37c067950 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:48.685 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.679+0000 7fb396543640 1 --2- 172.21.15.16:0/2569914555 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb39006a830 0x7fb39015d5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:48.685 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.679+0000 7fb396543640 1 -- 172.21.15.16:0/2569914555 >> 172.21.15.16:0/2569914555 conn(0x7fb39006f4f0 msgr2=0x7fb3900afc40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:48.685 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.679+0000 7fb396543640 1 -- 172.21.15.16:0/2569914555 shutdown_connections 2023-05-22T15:42:48.685 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.679+0000 7fb396543640 1 -- 172.21.15.16:0/2569914555 wait complete. 2023-05-22T15:42:48.700 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2023-05-22T15:42:48.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.827+0000 7fcd30be5640 1 Processor -- start 2023-05-22T15:42:48.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.827+0000 7fcd30be5640 1 -- start start 2023-05-22T15:42:48.834 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.827+0000 7fcd30be5640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fcd2c15bee0 0x7fcd2c06b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:48.834 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.831+0000 7fcd30be5640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fcd2c0c9b00 0x7fcd2c06bc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:48.834 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.831+0000 7fcd30be5640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd2c0c8660 con 0x7fcd2c0cb730 2023-05-22T15:42:48.835 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.831+0000 7fcd30be5640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd2c0ce4a0 con 0x7fcd2c0c9b00 2023-05-22T15:42:48.835 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.831+0000 7fcd30be5640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd2c0ce780 con 0x7fcd2c15bee0 2023-05-22T15:42:48.835 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.831+0000 7fcd2ad9d640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fcd2c0c9b00 0x7fcd2c06bc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:48.835 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.831+0000 7fcd2ad9d640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fcd2c0c9b00 0x7fcd2c06bc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49404/0 (socket says 172.21.15.16:49404) 2023-05-22T15:42:48.835 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.831+0000 7fcd2ad9d640 1 -- 172.21.15.16:0/2070591336 learned_addr learned my addr 172.21.15.16:0/2070591336 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:42:48.836 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.831+0000 7fcd2a59c640 1 --2- 172.21.15.16:0/2070591336 >> v2:172.21.15.62:3300/0 conn(0x7fcd2c15bee0 0x7fcd2c06b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:48.836 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.831+0000 7fcd2ad9d640 1 -- 172.21.15.16:0/2070591336 >> v2:172.21.15.62:3300/0 conn(0x7fcd2c15bee0 msgr2=0x7fcd2c06b6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:42:48.836 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.831+0000 7fcd2ad9d640 1 --2- 172.21.15.16:0/2070591336 >> v2:172.21.15.62:3300/0 conn(0x7fcd2c15bee0 0x7fcd2c06b6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:48.836 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.831+0000 7fcd2ad9d640 1 -- 172.21.15.16:0/2070591336 >> v1:172.21.15.16:6789/0 conn(0x7fcd2c0cb730 legacy=0x7fcd2c15f850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:48.836 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.831+0000 7fcd2ad9d640 1 -- 172.21.15.16:0/2070591336 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd2c0cea60 con 0x7fcd2c0c9b00 2023-05-22T15:42:48.837 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.831+0000 7fcd2ad9d640 1 --2- 172.21.15.16:0/2070591336 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcd2c0c9b00 0x7fcd2c06bc00 secure :-1 s=READY pgs=1240 cs=0 l=1 rev1=1 crypto rx=0x7fcd2c0ad980 tx=0x7fcd2401c9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9ac4284d1d81517 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:48.837 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.831+0000 7fcd2959a640 1 -- 172.21.15.16:0/2070591336 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd240a5880 con 0x7fcd2c0c9b00 2023-05-22T15:42:48.837 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.831+0000 7fcd2959a640 1 -- 172.21.15.16:0/2070591336 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd240a59e0 con 0x7fcd2c0c9b00 2023-05-22T15:42:48.838 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.831+0000 7fcd2959a640 1 -- 172.21.15.16:0/2070591336 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd240a5d70 con 0x7fcd2c0c9b00 2023-05-22T15:42:48.838 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.831+0000 7fcd30be5640 1 -- 172.21.15.16:0/2070591336 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcd2c0c9b00 msgr2=0x7fcd2c06bc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:48.838 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.831+0000 7fcd30be5640 1 --2- 172.21.15.16:0/2070591336 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcd2c0c9b00 0x7fcd2c06bc00 secure :-1 s=READY pgs=1240 cs=0 l=1 rev1=1 crypto rx=0x7fcd2c0ad980 tx=0x7fcd2401c9a0 comp rx=0 tx=0).stop 2023-05-22T15:42:48.839 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.831+0000 7fcd30be5640 1 -- 172.21.15.16:0/2070591336 shutdown_connections 2023-05-22T15:42:48.839 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.831+0000 7fcd30be5640 1 --2- 172.21.15.16:0/2070591336 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcd2c0c9b00 0x7fcd2c06bc00 unknown :-1 s=CLOSED pgs=1240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:48.839 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.831+0000 7fcd30be5640 1 --2- 172.21.15.16:0/2070591336 >> v2:172.21.15.62:3300/0 conn(0x7fcd2c15bee0 0x7fcd2c06b6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:48.839 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.831+0000 7fcd30be5640 1 -- 172.21.15.16:0/2070591336 >> 172.21.15.16:0/2070591336 conn(0x7fcd2c069720 msgr2=0x7fcd2c069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:48.839 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.831+0000 7fcd30be5640 1 -- 172.21.15.16:0/2070591336 shutdown_connections 2023-05-22T15:42:48.839 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.831+0000 7fcd30be5640 1 -- 172.21.15.16:0/2070591336 wait complete. 2023-05-22T15:42:48.841 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.835+0000 7fcd30be5640 1 Processor -- start 2023-05-22T15:42:48.842 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.835+0000 7fcd30be5640 1 -- start start 2023-05-22T15:42:48.842 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.835+0000 7fcd30be5640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcd2c0c9b00 0x7fcd2c14db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:48.842 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.835+0000 7fcd30be5640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fcd2c0cb730 0x7fcd2c150070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:48.842 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.835+0000 7fcd30be5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcd2c15bee0 0x7fcd2c1505b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:48.842 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.835+0000 7fcd30be5640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd2c0c7c00 con 0x7fcd2c15bee0 2023-05-22T15:42:48.842 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.835+0000 7fcd30be5640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd2c0c50e0 con 0x7fcd2c0c9b00 2023-05-22T15:42:48.843 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.835+0000 7fcd30be5640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd2c0c5980 con 0x7fcd2c0cb730 2023-05-22T15:42:48.843 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.835+0000 7fcd2ad9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcd2c15bee0 0x7fcd2c1505b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:48.843 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.835+0000 7fcd2ad9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcd2c15bee0 0x7fcd2c1505b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49420/0 (socket says 172.21.15.16:49420) 2023-05-22T15:42:48.843 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.835+0000 7fcd2ad9d640 1 -- 172.21.15.16:0/35371390 learned_addr learned my addr 172.21.15.16:0/35371390 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:42:48.843 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.835+0000 7fcd29d9b640 1 --2- 172.21.15.16:0/35371390 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fcd2c0cb730 0x7fcd2c150070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:48.843 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.835+0000 7fcd2a59c640 1 --2- 172.21.15.16:0/35371390 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcd2c0c9b00 0x7fcd2c14db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:48.843 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.835+0000 7fcd2ad9d640 1 -- 172.21.15.16:0/35371390 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fcd2c0cb730 msgr2=0x7fcd2c150070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:48.844 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.835+0000 7fcd2ad9d640 1 --2- 172.21.15.16:0/35371390 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fcd2c0cb730 0x7fcd2c150070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:48.844 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.835+0000 7fcd2ad9d640 1 -- 172.21.15.16:0/35371390 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcd2c0c9b00 msgr2=0x7fcd2c14db70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:48.844 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.835+0000 7fcd2ad9d640 1 --2- 172.21.15.16:0/35371390 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcd2c0c9b00 0x7fcd2c14db70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:48.844 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.835+0000 7fcd2ad9d640 1 -- 172.21.15.16:0/35371390 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd24009b30 con 0x7fcd2c15bee0 2023-05-22T15:42:48.844 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.835+0000 7fcd2ad9d640 1 --2- 172.21.15.16:0/35371390 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcd2c15bee0 0x7fcd2c1505b0 secure :-1 s=READY pgs=1241 cs=0 l=1 rev1=1 crypto rx=0x7fcd240af040 tx=0x7fcd24019740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:48.844 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.835+0000 7fcd1affd640 1 -- 172.21.15.16:0/35371390 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd240b8880 con 0x7fcd2c15bee0 2023-05-22T15:42:48.844 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.835+0000 7fcd1affd640 1 -- 172.21.15.16:0/35371390 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd240b8a00 con 0x7fcd2c15bee0 2023-05-22T15:42:48.845 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.835+0000 7fcd30be5640 1 -- 172.21.15.16:0/35371390 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd2c168d40 con 0x7fcd2c15bee0 2023-05-22T15:42:48.845 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.835+0000 7fcd1affd640 1 -- 172.21.15.16:0/35371390 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd240b8db0 con 0x7fcd2c15bee0 2023-05-22T15:42:48.845 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.835+0000 7fcd30be5640 1 -- 172.21.15.16:0/35371390 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd2c168bb0 con 0x7fcd2c15bee0 2023-05-22T15:42:48.845 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.835+0000 7fcd30be5640 1 -- 172.21.15.16:0/35371390 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd2c168ed0 con 0x7fcd2c15bee0 2023-05-22T15:42:48.846 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.839+0000 7fcd1affd640 1 -- 172.21.15.16:0/35371390 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 34) v1 ==== 84332+0+0 (secure 0 0 0) 0x7fcd240c3460 con 0x7fcd2c15bee0 2023-05-22T15:42:48.846 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.839+0000 7fcd1affd640 1 --2- 172.21.15.16:0/35371390 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fcd0c0654a0 0x7fcd0c067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:48.846 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.839+0000 7fcd1affd640 1 -- 172.21.15.16:0/35371390 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 6261+0+0 (secure 0 0 0) 0x7fcd2411e1a0 con 0x7fcd2c15bee0 2023-05-22T15:42:48.846 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.839+0000 7fcd2a59c640 1 --2- 172.21.15.16:0/35371390 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fcd0c0654a0 0x7fcd0c067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:48.846 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.839+0000 7fcd2a59c640 1 --2- 172.21.15.16:0/35371390 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fcd0c0654a0 0x7fcd0c067950 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fcd1c00a970 tx=0x7fcd1c008040 comp rx=0 tx=0).ready entity=mgr.6436 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:48.847 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.843+0000 7fcd1affd640 1 -- 172.21.15.16:0/35371390 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fcd240ebbe0 con 0x7fcd2c15bee0 2023-05-22T15:42:49.004 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:48.999+0000 7fcd30be5640 1 -- 172.21.15.16:0/35371390 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7fcd2c04dbe0 con 0x7fcd2c15bee0 2023-05-22T15:42:49.016 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:49.011+0000 7fcd1affd640 1 -- 172.21.15.16:0/35371390 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v27) v1 ==== 115+0+0 (secure 0 0 0) 0x7fcd240ef200 con 0x7fcd2c15bee0 2023-05-22T15:42:49.019 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:49.015+0000 7fcd30be5640 1 -- 172.21.15.16:0/35371390 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fcd0c0654a0 msgr2=0x7fcd0c067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:49.019 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:49.015+0000 7fcd30be5640 1 --2- 172.21.15.16:0/35371390 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fcd0c0654a0 0x7fcd0c067950 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fcd1c00a970 tx=0x7fcd1c008040 comp rx=0 tx=0).stop 2023-05-22T15:42:49.019 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:49.015+0000 7fcd30be5640 1 -- 172.21.15.16:0/35371390 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcd2c15bee0 msgr2=0x7fcd2c1505b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:49.019 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:49.015+0000 7fcd30be5640 1 --2- 172.21.15.16:0/35371390 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcd2c15bee0 0x7fcd2c1505b0 secure :-1 s=READY pgs=1241 cs=0 l=1 rev1=1 crypto rx=0x7fcd240af040 tx=0x7fcd24019740 comp rx=0 tx=0).stop 2023-05-22T15:42:49.019 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:49.015+0000 7fcd30be5640 1 -- 172.21.15.16:0/35371390 shutdown_connections 2023-05-22T15:42:49.020 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:49.015+0000 7fcd30be5640 1 --2- 172.21.15.16:0/35371390 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcd2c15bee0 0x7fcd2c1505b0 unknown :-1 s=CLOSED pgs=1241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:49.020 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:49.015+0000 7fcd30be5640 1 --2- 172.21.15.16:0/35371390 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fcd0c0654a0 0x7fcd0c067950 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:49.020 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:49.015+0000 7fcd30be5640 1 --2- 172.21.15.16:0/35371390 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fcd2c0cb730 0x7fcd2c150070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:49.020 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:49.015+0000 7fcd30be5640 1 --2- 172.21.15.16:0/35371390 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcd2c0c9b00 0x7fcd2c14db70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:49.020 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:49.015+0000 7fcd30be5640 1 -- 172.21.15.16:0/35371390 >> 172.21.15.16:0/35371390 conn(0x7fcd2c069720 msgr2=0x7fcd2c0b1220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:49.020 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:49.015+0000 7fcd30be5640 1 -- 172.21.15.16:0/35371390 shutdown_connections 2023-05-22T15:42:49.021 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:49.015+0000 7fcd30be5640 1 -- 172.21.15.16:0/35371390 wait complete. 2023-05-22T15:42:49.037 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_schedule' 2023-05-22T15:42:49.154 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:49.147+0000 7f996f6ee640 1 Processor -- start 2023-05-22T15:42:49.154 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:49.151+0000 7f996f6ee640 1 -- start start 2023-05-22T15:42:49.155 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:49.151+0000 7f996f6ee640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f99680b7dd0 0x7f99680ccab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:49.155 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:49.151+0000 7f996f6ee640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f9968160220 0x7f99680d0410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:49.155 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:49.151+0000 7f996f6ee640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99680cd170 con 0x7f996815f700 2023-05-22T15:42:49.156 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:49.151+0000 7f996f6ee640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99680d3360 con 0x7f9968160220 2023-05-22T15:42:49.156 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:49.151+0000 7f996f6ee640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99680d2ac0 con 0x7f99680b7dd0 2023-05-22T15:42:49.156 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:49.151+0000 7f996cc89640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f996815f700 0x7f996815fb10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:41116/0 (socket says 172.21.15.16:41116) 2023-05-22T15:42:49.156 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:49.151+0000 7f996cc89640 1 -- 172.21.15.16:0/745799831 learned_addr learned my addr 172.21.15.16:0/745799831 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:42:49.156 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:49.151+0000 7f996dc8b640 1 --2- 172.21.15.16:0/745799831 >> v2:172.21.15.16:3300/0 conn(0x7f9968160220 0x7f99680d0410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:49.157 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:49.151+0000 7f996d48a640 1 --2- 172.21.15.16:0/745799831 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f99680b7dd0 0x7f99680ccab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:49.157 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:49.151+0000 7f995ffff640 1 -- 172.21.15.16:0/745799831 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3732813859 0 0) 0x7f99680cd170 con 0x7f996815f700 2023-05-22T15:42:49.157 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:49.151+0000 7f995ffff640 1 -- 172.21.15.16:0/745799831 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9954003530 con 0x7f996815f700 2023-05-22T15:42:49.157 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:49.151+0000 7f996dc8b640 1 -- 172.21.15.16:0/745799831 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f99680b7dd0 msgr2=0x7f99680ccab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:42:49.158 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:49.151+0000 7f996dc8b640 1 --2- 172.21.15.16:0/745799831 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f99680b7dd0 0x7f99680ccab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:49.158 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:49.151+0000 7f996dc8b640 1 -- 172.21.15.16:0/745799831 >> v1:172.21.15.16:6789/0 conn(0x7f996815f700 legacy=0x7f996815fb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:49.158 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:49.151+0000 7f996dc8b640 1 -- 172.21.15.16:0/745799831 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99680d2da0 con 0x7f9968160220 2023-05-22T15:42:49.158 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:49.155+0000 7f996dc8b640 1 --2- 172.21.15.16:0/745799831 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9968160220 0x7f99680d0410 secure :-1 s=READY pgs=1243 cs=0 l=1 rev1=1 crypto rx=0x7f9960009d90 tx=0x7f996001c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ff54918c14f4ea9 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:49.159 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:49.155+0000 7f995ffff640 1 -- 172.21.15.16:0/745799831 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99600a5910 con 0x7f9968160220 2023-05-22T15:42:49.159 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:49.155+0000 7f995ffff640 1 -- 172.21.15.16:0/745799831 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99600a5a70 con 0x7f9968160220 2023-05-22T15:42:49.159 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:49.155+0000 7f995ffff640 1 -- 172.21.15.16:0/745799831 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99600a5e00 con 0x7f9968160220 2023-05-22T15:42:49.160 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:49.155+0000 7f996f6ee640 1 -- 172.21.15.16:0/745799831 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9968160220 msgr2=0x7f99680d0410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:49.160 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:49.155+0000 7f996f6ee640 1 --2- 172.21.15.16:0/745799831 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9968160220 0x7f99680d0410 secure :-1 s=READY pgs=1243 cs=0 l=1 rev1=1 crypto rx=0x7f9960009d90 tx=0x7f996001c890 comp rx=0 tx=0).stop 2023-05-22T15:42:49.161 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:49.155+0000 7f996f6ee640 1 -- 172.21.15.16:0/745799831 shutdown_connections 2023-05-22T15:42:49.161 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:49.155+0000 7f996f6ee640 1 --2- 172.21.15.16:0/745799831 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9968160220 0x7f99680d0410 unknown :-1 s=CLOSED pgs=1243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:49.161 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:49.155+0000 7f996f6ee640 1 --2- 172.21.15.16:0/745799831 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f99680b7dd0 0x7f99680ccab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:49.161 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:49.155+0000 7f996f6ee640 1 -- 172.21.15.16:0/745799831 >> 172.21.15.16:0/745799831 conn(0x7f99680655c0 msgr2=0x7f9968070e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:49.161 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:49.155+0000 7f996f6ee640 1 -- 172.21.15.16:0/745799831 shutdown_connections 2023-05-22T15:42:49.161 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:49.155+0000 7f996f6ee640 1 -- 172.21.15.16:0/745799831 wait complete. 2023-05-22T15:42:49.163 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:49.155+0000 7f996f6ee640 1 Processor -- start 2023-05-22T15:42:49.163 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:49.155+0000 7f996f6ee640 1 -- start start 2023-05-22T15:42:49.163 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:49.155+0000 7f996f6ee640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f99680b7dd0 0x7f9968155980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:49.163 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:49.155+0000 7f996f6ee640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f996815f700 0x7f99681455c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:49.164 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:49.155+0000 7f996f6ee640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9968160220 0x7f9968156ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:49.164 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:49.155+0000 7f996f6ee640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9968157b00 con 0x7f9968160220 2023-05-22T15:42:49.164 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:49.155+0000 7f996f6ee640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99680caa00 con 0x7f99680b7dd0 2023-05-22T15:42:49.164 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:49.155+0000 7f996f6ee640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99680caca0 con 0x7f996815f700 2023-05-22T15:42:49.164 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:49.155+0000 7f996dc8b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9968160220 0x7f9968156ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:49.164 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:49.155+0000 7f996dc8b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9968160220 0x7f9968156ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49452/0 (socket says 172.21.15.16:49452) 2023-05-22T15:42:49.164 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:49.155+0000 7f996dc8b640 1 -- 172.21.15.16:0/181560478 learned_addr learned my addr 172.21.15.16:0/181560478 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:42:49.165 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:49.159+0000 7f996cc89640 1 --2- 172.21.15.16:0/181560478 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f996815f700 0x7f99681455c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:49.165 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:49.159+0000 7f996d48a640 1 --2- 172.21.15.16:0/181560478 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f99680b7dd0 0x7f9968155980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:49.165 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:49.159+0000 7f996dc8b640 1 -- 172.21.15.16:0/181560478 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f996815f700 msgr2=0x7f99681455c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:49.165 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:49.159+0000 7f996dc8b640 1 --2- 172.21.15.16:0/181560478 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f996815f700 0x7f99681455c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:49.165 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:49.159+0000 7f996dc8b640 1 -- 172.21.15.16:0/181560478 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f99680b7dd0 msgr2=0x7f9968155980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:49.165 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:49.159+0000 7f996dc8b640 1 --2- 172.21.15.16:0/181560478 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f99680b7dd0 0x7f9968155980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:49.165 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:49.159+0000 7f996dc8b640 1 -- 172.21.15.16:0/181560478 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9960009a70 con 0x7f9968160220 2023-05-22T15:42:49.166 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:49.159+0000 7f996d48a640 1 --2- 172.21.15.16:0/181560478 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f99680b7dd0 0x7f9968155980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:42:49.166 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:49.159+0000 7f996dc8b640 1 --2- 172.21.15.16:0/181560478 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9968160220 0x7f9968156ba0 secure :-1 s=READY pgs=1244 cs=0 l=1 rev1=1 crypto rx=0x7f99600b0040 tx=0x7f9960019630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:49.166 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:49.159+0000 7f995dffb640 1 -- 172.21.15.16:0/181560478 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99600b9860 con 0x7f9968160220 2023-05-22T15:42:49.166 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:49.159+0000 7f995dffb640 1 -- 172.21.15.16:0/181560478 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99600b99c0 con 0x7f9968160220 2023-05-22T15:42:49.167 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:49.159+0000 7f995dffb640 1 -- 172.21.15.16:0/181560478 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99600b9d70 con 0x7f9968160220 2023-05-22T15:42:49.167 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:49.159+0000 7f996f6ee640 1 -- 172.21.15.16:0/181560478 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99680d12f0 con 0x7f9968160220 2023-05-22T15:42:49.167 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:49.159+0000 7f996f6ee640 1 -- 172.21.15.16:0/181560478 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99680d1160 con 0x7f9968160220 2023-05-22T15:42:49.168 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:49.159+0000 7f996f6ee640 1 -- 172.21.15.16:0/181560478 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99680d1480 con 0x7f9968160220 2023-05-22T15:42:49.168 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:49.159+0000 7f995dffb640 1 -- 172.21.15.16:0/181560478 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 34) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f99600ad440 con 0x7f9968160220 2023-05-22T15:42:49.168 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:49.159+0000 7f995dffb640 1 --2- 172.21.15.16:0/181560478 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f994c0654d0 0x7f994c067980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:49.168 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:49.159+0000 7f996d48a640 1 --2- 172.21.15.16:0/181560478 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f994c0654d0 0x7f994c067980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:49.169 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:49.159+0000 7f995dffb640 1 -- 172.21.15.16:0/181560478 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 6261+0+0 (secure 0 0 0) 0x7f996011d820 con 0x7f9968160220 2023-05-22T15:42:49.169 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:49.159+0000 7f996d48a640 1 --2- 172.21.15.16:0/181560478 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f994c0654d0 0x7f994c067980 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f99640097a0 tx=0x7f9964008040 comp rx=0 tx=0).ready entity=mgr.6436 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:49.169 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:49.163+0000 7f995dffb640 1 -- 172.21.15.16:0/181560478 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f99600eb260 con 0x7f9968160220 2023-05-22T15:42:49.382 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:49.379+0000 7f996f6ee640 1 -- 172.21.15.16:0/181560478 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_schedule"]} v 0) v1 -- 0x7f996804dbe0 con 0x7f9968160220 2023-05-22T15:42:50.008 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:50.003+0000 7f995dffb640 1 -- 172.21.15.16:0/181560478 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_schedule"]}]=0 v593) v1 ==== 156+0+0 (secure 0 0 0) 0x7f99600ee880 con 0x7f9968160220 2023-05-22T15:42:50.012 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:50.007+0000 7f996f6ee640 1 -- 172.21.15.16:0/181560478 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f994c0654d0 msgr2=0x7f994c067980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:50.012 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:50.007+0000 7f996f6ee640 1 --2- 172.21.15.16:0/181560478 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f994c0654d0 0x7f994c067980 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f99640097a0 tx=0x7f9964008040 comp rx=0 tx=0).stop 2023-05-22T15:42:50.012 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:50.007+0000 7f996f6ee640 1 -- 172.21.15.16:0/181560478 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9968160220 msgr2=0x7f9968156ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:50.012 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:50.007+0000 7f996f6ee640 1 --2- 172.21.15.16:0/181560478 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9968160220 0x7f9968156ba0 secure :-1 s=READY pgs=1244 cs=0 l=1 rev1=1 crypto rx=0x7f99600b0040 tx=0x7f9960019630 comp rx=0 tx=0).stop 2023-05-22T15:42:50.013 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:50.007+0000 7f996f6ee640 1 -- 172.21.15.16:0/181560478 shutdown_connections 2023-05-22T15:42:50.013 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:50.007+0000 7f996f6ee640 1 --2- 172.21.15.16:0/181560478 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9968160220 0x7f9968156ba0 unknown :-1 s=CLOSED pgs=1244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:50.013 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:50.007+0000 7f996f6ee640 1 --2- 172.21.15.16:0/181560478 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f994c0654d0 0x7f994c067980 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:50.013 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:50.007+0000 7f996f6ee640 1 --2- 172.21.15.16:0/181560478 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f996815f700 0x7f99681455c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:50.013 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:50.007+0000 7f996f6ee640 1 --2- 172.21.15.16:0/181560478 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f99680b7dd0 0x7f9968155980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:50.013 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:50.007+0000 7f996f6ee640 1 -- 172.21.15.16:0/181560478 >> 172.21.15.16:0/181560478 conn(0x7f99680655c0 msgr2=0x7f99681584a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:50.013 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:50.007+0000 7f996f6ee640 1 -- 172.21.15.16:0/181560478 shutdown_connections 2023-05-22T15:42:50.014 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:50.007+0000 7f996f6ee640 1 -- 172.21.15.16:0/181560478 wait complete. 2023-05-22T15:42:50.034 INFO:tasks.cephfs_test_runner:Test listing non-existing snap schedules failure ... ok 2023-05-22T15:42:50.034 INFO:tasks.cephfs_test_runner:Starting test: test_non_existent_snap_schedule_list (tasks.cephfs.test_snap_schedules.TestSnapSchedules) Test listing snap schedules on a non-existing filesystem path failure 2023-05-22T15:42:50.034 INFO:tasks.cephfs_test_runner:test_non_existent_snap_schedule_list (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2023-05-22T15:42:50.035 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_snap_schedule_list' 2023-05-22T15:42:50.156 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:50.151+0000 7f99bd1fd640 1 Processor -- start 2023-05-22T15:42:50.157 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:50.151+0000 7f99bd1fd640 1 -- start start 2023-05-22T15:42:50.158 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:50.151+0000 7f99bd1fd640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f99b80cae20 0x7f99b806fef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:50.158 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:50.151+0000 7f99bd1fd640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f99b8070430 0x7f99b80ad410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:50.158 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:50.151+0000 7f99bd1fd640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99b8069720 con 0x7f99b8153fe0 2023-05-22T15:42:50.158 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:50.151+0000 7f99bd1fd640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99b80708d0 con 0x7f99b8070430 2023-05-22T15:42:50.159 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:50.151+0000 7f99bd1fd640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99b806db00 con 0x7f99b80cae20 2023-05-22T15:42:50.159 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:50.151+0000 7f99b759e640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f99b8070430 0x7f99b80ad410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:50.159 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:50.151+0000 7f99b759e640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f99b8070430 0x7f99b80ad410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60446/0 (socket says 172.21.15.16:60446) 2023-05-22T15:42:50.159 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:50.151+0000 7f99b759e640 1 -- 172.21.15.16:0/1790534872 learned_addr learned my addr 172.21.15.16:0/1790534872 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:42:50.159 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:50.151+0000 7f99affff640 1 --2- 172.21.15.16:0/1790534872 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f99b80cae20 0x7f99b806fef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:50.159 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:50.151+0000 7f99b759e640 1 -- 172.21.15.16:0/1790534872 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f99b80cae20 msgr2=0x7f99b806fef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:42:50.159 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:50.151+0000 7f99b759e640 1 --2- 172.21.15.16:0/1790534872 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f99b80cae20 0x7f99b806fef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:50.159 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:50.151+0000 7f99b759e640 1 -- 172.21.15.16:0/1790534872 >> v1:172.21.15.62:6789/0 conn(0x7f99b8153fe0 legacy=0x7f99b806d3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:42:50.160 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:50.151+0000 7f99b759e640 1 -- 172.21.15.16:0/1790534872 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99b81543b0 con 0x7f99b8070430 2023-05-22T15:42:50.160 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:50.155+0000 7f99b759e640 1 --2- 172.21.15.16:0/1790534872 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f99b8070430 0x7f99b80ad410 secure :-1 s=READY pgs=1245 cs=0 l=1 rev1=1 crypto rx=0x7f99b0009e80 tx=0x7f99b001c980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19e50da0b22fafc1 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:50.160 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:50.155+0000 7f99b659c640 1 -- 172.21.15.16:0/1790534872 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99b00a5800 con 0x7f99b8070430 2023-05-22T15:42:50.161 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:50.155+0000 7f99b659c640 1 -- 172.21.15.16:0/1790534872 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99b00a5960 con 0x7f99b8070430 2023-05-22T15:42:50.161 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:50.155+0000 7f99b659c640 1 -- 172.21.15.16:0/1790534872 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99b00a5cf0 con 0x7f99b8070430 2023-05-22T15:42:50.161 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:50.155+0000 7f99bd1fd640 1 -- 172.21.15.16:0/1790534872 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f99b8070430 msgr2=0x7f99b80ad410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:50.162 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:50.155+0000 7f99bd1fd640 1 --2- 172.21.15.16:0/1790534872 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f99b8070430 0x7f99b80ad410 secure :-1 s=READY pgs=1245 cs=0 l=1 rev1=1 crypto rx=0x7f99b0009e80 tx=0x7f99b001c980 comp rx=0 tx=0).stop 2023-05-22T15:42:50.162 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:50.155+0000 7f99bd1fd640 1 -- 172.21.15.16:0/1790534872 shutdown_connections 2023-05-22T15:42:50.162 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:50.155+0000 7f99bd1fd640 1 --2- 172.21.15.16:0/1790534872 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f99b8070430 0x7f99b80ad410 unknown :-1 s=CLOSED pgs=1245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:50.162 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:50.155+0000 7f99bd1fd640 1 --2- 172.21.15.16:0/1790534872 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f99b80cae20 0x7f99b806fef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:50.162 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:50.155+0000 7f99bd1fd640 1 -- 172.21.15.16:0/1790534872 >> 172.21.15.16:0/1790534872 conn(0x7f99b80b1110 msgr2=0x7f99b80ce830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:50.162 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:50.155+0000 7f99bd1fd640 1 -- 172.21.15.16:0/1790534872 shutdown_connections 2023-05-22T15:42:50.162 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:50.155+0000 7f99bd1fd640 1 -- 172.21.15.16:0/1790534872 wait complete. 2023-05-22T15:42:50.164 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:50.159+0000 7f99bd1fd640 1 Processor -- start 2023-05-22T15:42:50.165 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:50.159+0000 7f99bd1fd640 1 -- start start 2023-05-22T15:42:50.165 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:50.159+0000 7f99bd1fd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f99b8070430 0x7f99b8150c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:50.165 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:50.159+0000 7f99bd1fd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f99b80cae20 0x7f99b814b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:50.165 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:50.159+0000 7f99bd1fd640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f99b8153fe0 0x7f99b80ae9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:50.165 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:50.159+0000 7f99bd1fd640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99b8156ec0 con 0x7f99b80cae20 2023-05-22T15:42:50.165 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:50.159+0000 7f99bd1fd640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99b81573f0 con 0x7f99b8070430 2023-05-22T15:42:50.165 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:50.159+0000 7f99bd1fd640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99b81507e0 con 0x7f99b8153fe0 2023-05-22T15:42:50.165 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:50.159+0000 7f99affff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f99b80cae20 0x7f99b814b1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:50.166 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:50.159+0000 7f99affff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f99b80cae20 0x7f99b814b1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60450/0 (socket says 172.21.15.16:60450) 2023-05-22T15:42:50.167 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:50.159+0000 7f99affff640 1 -- 172.21.15.16:0/4118701097 learned_addr learned my addr 172.21.15.16:0/4118701097 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:42:50.167 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:50.159+0000 7f99b6d9d640 1 --2- 172.21.15.16:0/4118701097 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f99b8070430 0x7f99b8150c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:50.167 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:50.159+0000 7f99b759e640 1 --2- 172.21.15.16:0/4118701097 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f99b8153fe0 0x7f99b80ae9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:50.167 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:50.159+0000 7f99affff640 1 -- 172.21.15.16:0/4118701097 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f99b8153fe0 msgr2=0x7f99b80ae9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:50.167 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:50.159+0000 7f99affff640 1 --2- 172.21.15.16:0/4118701097 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f99b8153fe0 0x7f99b80ae9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:50.167 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:50.159+0000 7f99affff640 1 -- 172.21.15.16:0/4118701097 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f99b8070430 msgr2=0x7f99b8150c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:50.167 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:50.159+0000 7f99affff640 1 --2- 172.21.15.16:0/4118701097 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f99b8070430 0x7f99b8150c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:50.168 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:50.159+0000 7f99affff640 1 -- 172.21.15.16:0/4118701097 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99b0009b30 con 0x7f99b80cae20 2023-05-22T15:42:50.168 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:50.159+0000 7f99b6d9d640 1 --2- 172.21.15.16:0/4118701097 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f99b8070430 0x7f99b8150c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:42:50.168 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:50.159+0000 7f99affff640 1 --2- 172.21.15.16:0/4118701097 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f99b80cae20 0x7f99b814b1e0 secure :-1 s=READY pgs=1246 cs=0 l=1 rev1=1 crypto rx=0x7f99a800d830 tx=0x7f99a800dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:50.168 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:50.159+0000 7f99af7fe640 1 -- 172.21.15.16:0/4118701097 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99a800b6c0 con 0x7f99b80cae20 2023-05-22T15:42:50.169 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:50.159+0000 7f99af7fe640 1 -- 172.21.15.16:0/4118701097 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99a800b820 con 0x7f99b80cae20 2023-05-22T15:42:50.169 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:50.159+0000 7f99bd1fd640 1 -- 172.21.15.16:0/4118701097 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99b815db60 con 0x7f99b80cae20 2023-05-22T15:42:50.169 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:50.159+0000 7f99af7fe640 1 -- 172.21.15.16:0/4118701097 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99a800bb90 con 0x7f99b80cae20 2023-05-22T15:42:50.169 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:50.159+0000 7f99bd1fd640 1 -- 172.21.15.16:0/4118701097 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99b815d880 con 0x7f99b80cae20 2023-05-22T15:42:50.170 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:50.159+0000 7f99bd1fd640 1 -- 172.21.15.16:0/4118701097 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99b815de40 con 0x7f99b80cae20 2023-05-22T15:42:50.170 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:50.159+0000 7f99af7fe640 1 -- 172.21.15.16:0/4118701097 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 34) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f99a801e460 con 0x7f99b80cae20 2023-05-22T15:42:50.170 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:50.163+0000 7f99af7fe640 1 --2- 172.21.15.16:0/4118701097 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f99980654a0 0x7f9998067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:50.170 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:50.163+0000 7f99af7fe640 1 -- 172.21.15.16:0/4118701097 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 6261+0+0 (secure 0 0 0) 0x7f99a8083240 con 0x7f99b80cae20 2023-05-22T15:42:50.170 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:50.163+0000 7f99b6d9d640 1 --2- 172.21.15.16:0/4118701097 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f99980654a0 0x7f9998067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:50.171 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:50.163+0000 7f99b6d9d640 1 --2- 172.21.15.16:0/4118701097 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f99980654a0 0x7f9998067950 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f99b806d2c0 tx=0x7f99a0009040 comp rx=0 tx=0).ready entity=mgr.6436 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:50.171 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:50.167+0000 7f99af7fe640 1 -- 172.21.15.16:0/4118701097 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f99a8050d10 con 0x7f99b80cae20 2023-05-22T15:42:50.381 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:50.375+0000 7f99bd1fd640 1 -- 172.21.15.16:0/4118701097 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_snap_schedule_list"]} v 0) v1 -- 0x7f99b804dbe0 con 0x7f99b80cae20 2023-05-22T15:42:51.012 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:51.007+0000 7f99af7fe640 1 -- 172.21.15.16:0/4118701097 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_snap_schedule_list"]}]=0 v594) v1 ==== 169+0+0 (secure 0 0 0) 0x7f99a8054330 con 0x7f99b80cae20 2023-05-22T15:42:51.016 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:51.011+0000 7f99bd1fd640 1 -- 172.21.15.16:0/4118701097 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f99980654a0 msgr2=0x7f9998067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:51.016 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:51.011+0000 7f99bd1fd640 1 --2- 172.21.15.16:0/4118701097 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f99980654a0 0x7f9998067950 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f99b806d2c0 tx=0x7f99a0009040 comp rx=0 tx=0).stop 2023-05-22T15:42:51.016 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:51.011+0000 7f99bd1fd640 1 -- 172.21.15.16:0/4118701097 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f99b80cae20 msgr2=0x7f99b814b1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:51.016 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:51.011+0000 7f99bd1fd640 1 --2- 172.21.15.16:0/4118701097 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f99b80cae20 0x7f99b814b1e0 secure :-1 s=READY pgs=1246 cs=0 l=1 rev1=1 crypto rx=0x7f99a800d830 tx=0x7f99a800dcf0 comp rx=0 tx=0).stop 2023-05-22T15:42:51.017 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:51.011+0000 7f99bd1fd640 1 -- 172.21.15.16:0/4118701097 shutdown_connections 2023-05-22T15:42:51.017 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:51.011+0000 7f99bd1fd640 1 --2- 172.21.15.16:0/4118701097 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f99980654a0 0x7f9998067950 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:51.017 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:51.011+0000 7f99bd1fd640 1 --2- 172.21.15.16:0/4118701097 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f99b8153fe0 0x7f99b80ae9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:51.017 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:51.011+0000 7f99bd1fd640 1 --2- 172.21.15.16:0/4118701097 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f99b80cae20 0x7f99b814b1e0 unknown :-1 s=CLOSED pgs=1246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:51.017 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:51.011+0000 7f99bd1fd640 1 --2- 172.21.15.16:0/4118701097 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f99b8070430 0x7f99b8150c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:51.017 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:51.011+0000 7f99bd1fd640 1 -- 172.21.15.16:0/4118701097 >> 172.21.15.16:0/4118701097 conn(0x7f99b80b1110 msgr2=0x7f99b80698b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:51.017 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:51.011+0000 7f99bd1fd640 1 -- 172.21.15.16:0/4118701097 shutdown_connections 2023-05-22T15:42:51.017 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:51.011+0000 7f99bd1fd640 1 -- 172.21.15.16:0/4118701097 wait complete. 2023-05-22T15:42:51.040 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-05-22T15:42:51.175 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:51.171+0000 7f04b10d7640 1 Processor -- start 2023-05-22T15:42:51.175 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:51.171+0000 7f04b10d7640 1 -- start start 2023-05-22T15:42:51.176 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:51.171+0000 7f04b10d7640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f04ac159970 0x7f04ac159d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:51.176 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:51.171+0000 7f04b10d7640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04ac070c30 con 0x7f04ac15dad0 2023-05-22T15:42:51.177 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:51.171+0000 7f04b10d7640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04ac070e60 con 0x7f04ac15d680 2023-05-22T15:42:51.177 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:51.171+0000 7f04b10d7640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04ac0cb970 con 0x7f04ac159970 2023-05-22T15:42:51.177 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:51.171+0000 7f04ab59e640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f04ac15dad0 0x7f04ac06b6c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:44840/0 (socket says 172.21.15.16:44840) 2023-05-22T15:42:51.177 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:51.171+0000 7f04ab59e640 1 -- 172.21.15.16:0/1063533648 learned_addr learned my addr 172.21.15.16:0/1063533648 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:42:51.177 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:51.171+0000 7f04aad9d640 1 --2- 172.21.15.16:0/1063533648 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f04ac159970 0x7f04ac159d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:51.177 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:51.171+0000 7f04a9d9b640 1 -- 172.21.15.16:0/1063533648 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3201119392 0 0) 0x7f04ac070c30 con 0x7f04ac15dad0 2023-05-22T15:42:51.177 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:51.171+0000 7f04a9d9b640 1 -- 172.21.15.16:0/1063533648 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f048c003530 con 0x7f04ac15dad0 2023-05-22T15:42:51.179 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:51.171+0000 7f04a9d9b640 1 -- 172.21.15.16:0/1063533648 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1383732237 0 0) 0x7f048c003530 con 0x7f04ac15dad0 2023-05-22T15:42:51.179 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:51.171+0000 7f04a9d9b640 1 -- 172.21.15.16:0/1063533648 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f04ac159970 msgr2=0x7f04ac159d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:42:51.179 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:51.171+0000 7f04a9d9b640 1 --2- 172.21.15.16:0/1063533648 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f04ac159970 0x7f04ac159d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:51.180 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:51.171+0000 7f04a9d9b640 1 -- 172.21.15.16:0/1063533648 >> v1:172.21.15.62:6789/0 conn(0x7f04ac15d680 legacy=0x7f04ac0cb260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:51.180 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:51.171+0000 7f04a9d9b640 1 -- 172.21.15.16:0/1063533648 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04ac071140 con 0x7f04ac15dad0 2023-05-22T15:42:51.180 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:51.171+0000 7f04a9d9b640 1 -- 172.21.15.16:0/1063533648 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7f04a4001a00 con 0x7f04ac15dad0 2023-05-22T15:42:51.180 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:51.171+0000 7f04a9d9b640 1 -- 172.21.15.16:0/1063533648 >> v1:172.21.15.16:6789/0 conn(0x7f04ac15dad0 legacy=0x7f04ac06b6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:51.180 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:51.171+0000 7f04a9d9b640 1 --2- 172.21.15.16:0/1063533648 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f04ac159970 0x7f048c0039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:51.180 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:51.171+0000 7f04a9d9b640 1 --2- 172.21.15.16:0/1063533648 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f048c003f30 0x7f048c0047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:51.181 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:51.171+0000 7f04a9d9b640 1 --2- 172.21.15.16:0/1063533648 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f048c004d10 0x7f048c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:51.181 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:51.171+0000 7f04a9d9b640 1 -- 172.21.15.16:0/1063533648 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04ac0cb970 con 0x7f048c004d10 2023-05-22T15:42:51.181 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:51.171+0000 7f04a9d9b640 1 -- 172.21.15.16:0/1063533648 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f048c00bb20 con 0x7f048c003f30 2023-05-22T15:42:51.181 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:51.171+0000 7f04a9d9b640 1 -- 172.21.15.16:0/1063533648 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f048c00bce0 con 0x7f04ac159970 2023-05-22T15:42:51.181 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:51.171+0000 7f04ab59e640 1 --2- 172.21.15.16:0/1063533648 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f048c004d10 0x7f048c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:51.181 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:51.171+0000 7f04aa59c640 1 --2- 172.21.15.16:0/1063533648 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f048c003f30 0x7f048c0047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:51.181 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:51.171+0000 7f04aad9d640 1 --2- 172.21.15.16:0/1063533648 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f04ac159970 0x7f048c0039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:51.181 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:51.171+0000 7f04ab59e640 1 -- 172.21.15.16:0/1063533648 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f04ac159970 msgr2=0x7f048c0039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:42:51.182 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:51.171+0000 7f04ab59e640 1 --2- 172.21.15.16:0/1063533648 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f04ac159970 0x7f048c0039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:51.182 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:51.171+0000 7f04ab59e640 1 -- 172.21.15.16:0/1063533648 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f048c003f30 msgr2=0x7f048c0047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:42:51.182 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:51.171+0000 7f04ab59e640 1 --2- 172.21.15.16:0/1063533648 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f048c003f30 0x7f048c0047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:51.182 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:51.171+0000 7f04ab59e640 1 -- 172.21.15.16:0/1063533648 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f048c003530 con 0x7f048c004d10 2023-05-22T15:42:51.182 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:51.175+0000 7f04ab59e640 1 --2- 172.21.15.16:0/1063533648 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f048c004d10 0x7f048c005180 secure :-1 s=READY pgs=1248 cs=0 l=1 rev1=1 crypto rx=0x7f04ac15e780 tx=0x7f04a401fa00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9bf2c86a50bed9cb server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:51.184 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:51.175+0000 7f04a9d9b640 1 -- 172.21.15.16:0/1063533648 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04a40072d0 con 0x7f048c004d10 2023-05-22T15:42:51.184 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:51.175+0000 7f04a9d9b640 1 -- 172.21.15.16:0/1063533648 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04a4007430 con 0x7f048c004d10 2023-05-22T15:42:51.185 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:51.175+0000 7f04a9d9b640 1 -- 172.21.15.16:0/1063533648 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04a4007860 con 0x7f048c004d10 2023-05-22T15:42:51.185 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:51.175+0000 7f04b10d7640 1 -- 172.21.15.16:0/1063533648 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f048c004d10 msgr2=0x7f048c005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:51.185 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:51.175+0000 7f04b10d7640 1 --2- 172.21.15.16:0/1063533648 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f048c004d10 0x7f048c005180 secure :-1 s=READY pgs=1248 cs=0 l=1 rev1=1 crypto rx=0x7f04ac15e780 tx=0x7f04a401fa00 comp rx=0 tx=0).stop 2023-05-22T15:42:51.185 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:51.175+0000 7f04ab59e640 1 -- 172.21.15.16:0/1063533648 reap_dead start 2023-05-22T15:42:51.185 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:51.175+0000 7f04b10d7640 1 -- 172.21.15.16:0/1063533648 shutdown_connections 2023-05-22T15:42:51.186 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:51.175+0000 7f04b10d7640 1 -- 172.21.15.16:0/1063533648 >> 172.21.15.16:0/1063533648 conn(0x7f04ac069720 msgr2=0x7f04ac069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:51.186 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:51.175+0000 7f04b10d7640 1 -- 172.21.15.16:0/1063533648 shutdown_connections 2023-05-22T15:42:51.186 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:51.175+0000 7f04b10d7640 1 -- 172.21.15.16:0/1063533648 wait complete. 2023-05-22T15:42:51.186 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:51.175+0000 7f04b10d7640 1 Processor -- start 2023-05-22T15:42:51.187 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:51.175+0000 7f04b10d7640 1 -- start start 2023-05-22T15:42:51.187 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:51.179+0000 7f04b10d7640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f04ac0c2560 0x7f04ac156f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:51.187 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:51.179+0000 7f04b10d7640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f04ac0b1e90 0x7f04ac151250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:51.187 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:51.179+0000 7f04b10d7640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f04ac0c2930 0x7f04ac151790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:51.188 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:51.179+0000 7f04b10d7640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04ac158280 con 0x7f04ac0c2930 2023-05-22T15:42:51.188 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:51.179+0000 7f04b10d7640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04ac0b3030 con 0x7f04ac0c2560 2023-05-22T15:42:51.188 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:51.179+0000 7f04b10d7640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04ac157670 con 0x7f04ac0b1e90 2023-05-22T15:42:51.188 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:51.179+0000 7f04ab59e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f04ac0c2930 0x7f04ac151790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:51.189 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:51.179+0000 7f04ab59e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f04ac0c2930 0x7f04ac151790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60470/0 (socket says 172.21.15.16:60470) 2023-05-22T15:42:51.189 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:51.179+0000 7f04ab59e640 1 -- 172.21.15.16:0/442909560 learned_addr learned my addr 172.21.15.16:0/442909560 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:42:51.189 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:51.179+0000 7f04aa59c640 1 --2- 172.21.15.16:0/442909560 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f04ac0b1e90 0x7f04ac151250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:51.190 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:51.179+0000 7f04aad9d640 1 --2- 172.21.15.16:0/442909560 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f04ac0c2560 0x7f04ac156f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:51.190 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:51.179+0000 7f04ab59e640 1 -- 172.21.15.16:0/442909560 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f04ac0b1e90 msgr2=0x7f04ac151250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:51.190 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:51.179+0000 7f04ab59e640 1 --2- 172.21.15.16:0/442909560 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f04ac0b1e90 0x7f04ac151250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:51.190 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:51.179+0000 7f04ab59e640 1 -- 172.21.15.16:0/442909560 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f04ac0c2560 msgr2=0x7f04ac156f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:51.191 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:51.179+0000 7f04ab59e640 1 --2- 172.21.15.16:0/442909560 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f04ac0c2560 0x7f04ac156f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:51.191 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:51.179+0000 7f04ab59e640 1 -- 172.21.15.16:0/442909560 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04ac071700 con 0x7f04ac0c2930 2023-05-22T15:42:51.191 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:51.179+0000 7f04ab59e640 1 --2- 172.21.15.16:0/442909560 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f04ac0c2930 0x7f04ac151790 secure :-1 s=READY pgs=1249 cs=0 l=1 rev1=1 crypto rx=0x7f04a401f830 tx=0x7f048c005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:51.191 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:51.179+0000 7f04a37fe640 1 -- 172.21.15.16:0/442909560 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04a40070d0 con 0x7f04ac0c2930 2023-05-22T15:42:51.192 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:51.179+0000 7f04a37fe640 1 -- 172.21.15.16:0/442909560 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04a4007250 con 0x7f04ac0c2930 2023-05-22T15:42:51.192 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:51.179+0000 7f04b10d7640 1 -- 172.21.15.16:0/442909560 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04ac071420 con 0x7f04ac0c2930 2023-05-22T15:42:51.192 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:51.179+0000 7f04a37fe640 1 -- 172.21.15.16:0/442909560 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04a40077e0 con 0x7f04ac0c2930 2023-05-22T15:42:51.192 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:51.179+0000 7f04b10d7640 1 -- 172.21.15.16:0/442909560 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04ac0cac90 con 0x7f04ac0c2930 2023-05-22T15:42:51.192 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:51.179+0000 7f04b10d7640 1 -- 172.21.15.16:0/442909560 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04ac1583f0 con 0x7f04ac0c2930 2023-05-22T15:42:51.193 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:51.179+0000 7f04a37fe640 1 -- 172.21.15.16:0/442909560 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 34) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f04a4007940 con 0x7f04ac0c2930 2023-05-22T15:42:51.193 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:51.179+0000 7f04a37fe640 1 --2- 172.21.15.16:0/442909560 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f04900654a0 0x7f0490067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:51.193 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:51.179+0000 7f04a37fe640 1 -- 172.21.15.16:0/442909560 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 6261+0+0 (secure 0 0 0) 0x7f04a4120000 con 0x7f04ac0c2930 2023-05-22T15:42:51.193 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:51.179+0000 7f04aad9d640 1 --2- 172.21.15.16:0/442909560 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f04900654a0 0x7f0490067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:51.193 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:51.179+0000 7f04aad9d640 1 --2- 172.21.15.16:0/442909560 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f04900654a0 0x7f0490067950 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f0494009a30 tx=0x7f0494008040 comp rx=0 tx=0).ready entity=mgr.6436 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:51.194 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:51.183+0000 7f04a37fe640 1 -- 172.21.15.16:0/442909560 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f04a40eda20 con 0x7f04ac0c2930 2023-05-22T15:42:51.346 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:51.343+0000 7f04b10d7640 1 -- 172.21.15.16:0/442909560 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f04ac04dbe0 con 0x7f04ac0c2930 2023-05-22T15:42:51.356 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:51.351+0000 7f04a37fe640 1 -- 172.21.15.16:0/442909560 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v28)=0 v28) v1 ==== 122+0+0 (secure 0 0 0) 0x7f04a40f1040 con 0x7f04ac0c2930 2023-05-22T15:42:51.359 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:51.355+0000 7f04b10d7640 1 -- 172.21.15.16:0/442909560 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f04900654a0 msgr2=0x7f0490067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:51.359 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:51.355+0000 7f04b10d7640 1 --2- 172.21.15.16:0/442909560 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f04900654a0 0x7f0490067950 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f0494009a30 tx=0x7f0494008040 comp rx=0 tx=0).stop 2023-05-22T15:42:51.359 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:51.355+0000 7f04b10d7640 1 -- 172.21.15.16:0/442909560 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f04ac0c2930 msgr2=0x7f04ac151790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:51.359 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:51.355+0000 7f04b10d7640 1 --2- 172.21.15.16:0/442909560 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f04ac0c2930 0x7f04ac151790 secure :-1 s=READY pgs=1249 cs=0 l=1 rev1=1 crypto rx=0x7f04a401f830 tx=0x7f048c005800 comp rx=0 tx=0).stop 2023-05-22T15:42:51.359 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:51.355+0000 7f04b10d7640 1 -- 172.21.15.16:0/442909560 shutdown_connections 2023-05-22T15:42:51.360 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:51.355+0000 7f04b10d7640 1 --2- 172.21.15.16:0/442909560 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f04ac0c2930 0x7f04ac151790 unknown :-1 s=CLOSED pgs=1249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:51.360 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:51.355+0000 7f04b10d7640 1 --2- 172.21.15.16:0/442909560 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f04900654a0 0x7f0490067950 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:51.360 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:51.355+0000 7f04b10d7640 1 --2- 172.21.15.16:0/442909560 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f04ac0b1e90 0x7f04ac151250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:51.360 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:51.355+0000 7f04b10d7640 1 --2- 172.21.15.16:0/442909560 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f04ac0c2560 0x7f04ac156f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:51.360 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:51.355+0000 7f04b10d7640 1 -- 172.21.15.16:0/442909560 >> 172.21.15.16:0/442909560 conn(0x7f04ac069720 msgr2=0x7f04ac06b5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:51.360 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:51.355+0000 7f04b10d7640 1 -- 172.21.15.16:0/442909560 shutdown_connections 2023-05-22T15:42:51.361 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:51.355+0000 7f04b10d7640 1 -- 172.21.15.16:0/442909560 wait complete. 2023-05-22T15:42:51.377 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-05-22T15:42:51.377 DEBUG:teuthology.orchestra.run.smithi016:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-05-22T15:42:51.415 DEBUG:teuthology.orchestra.run.smithi062:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-05-22T15:42:51.479 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-05-22T15:42:51.479 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-05-22T15:42:51.480 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2023-05-22T15:42:51.486 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16345192k,nr_inodes=4086298,mode=755,inode64 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3276544k,mode=755,inode64 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev,inode64 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=15732 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3276540k,nr_inodes=819135,mode=700,uid=1000,gid=1271,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3276544k,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-05-22T15:42:51.486 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2023-05-22T15:42:51.486 DEBUG:teuthology.orchestra.run.smithi062:> dd if=/proc/self/mounts of=/dev/stdout 2023-05-22T15:42:51.492 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16342116k,nr_inodes=4085529,mode=755,inode64 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275932k,mode=755,inode64 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev,inode64 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=13250 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275928k,nr_inodes=818982,mode=700,uid=1000,gid=1271,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275932k,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-05-22T15:42:51.492 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-05-22T15:42:51.493 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2023-05-22T15:42:51.539 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16345192k,nr_inodes=4086298,mode=755,inode64 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3276544k,mode=755,inode64 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev,inode64 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=15732 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3276540k,nr_inodes=819135,mode=700,uid=1000,gid=1271,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3276544k,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-05-22T15:42:51.539 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2023-05-22T15:42:51.540 DEBUG:teuthology.orchestra.run.smithi062:> dd if=/proc/self/mounts of=/dev/stdout 2023-05-22T15:42:51.545 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16342116k,nr_inodes=4085529,mode=755,inode64 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275932k,mode=755,inode64 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev,inode64 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=13250 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275928k,nr_inodes=818982,mode=700,uid=1000,gid=1271,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275932k,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-05-22T15:42:51.546 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2023-05-22T15:42:51.546 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-05-22T15:42:51.715 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:51.711+0000 7fcafe0e9640 1 Processor -- start 2023-05-22T15:42:51.715 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:51.711+0000 7fcafe0e9640 1 -- start start 2023-05-22T15:42:51.716 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:51.711+0000 7fcafe0e9640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fcaf80b7dd0 0x7fcaf80ccab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:51.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:51.711+0000 7fcafe0e9640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fcaf815f700 0x7fcaf815fb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:51.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:51.711+0000 7fcafe0e9640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcaf80cd170 con 0x7fcaf8160050 2023-05-22T15:42:51.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:51.711+0000 7fcafe0e9640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcaf80d3360 con 0x7fcaf815f700 2023-05-22T15:42:51.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:51.711+0000 7fcafe0e9640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcaf80d2ac0 con 0x7fcaf80b7dd0 2023-05-22T15:42:51.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:51.711+0000 7fcaf6ffd640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fcaf815f700 0x7fcaf815fb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:51.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:51.711+0000 7fcaf6ffd640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fcaf815f700 0x7fcaf815fb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60476/0 (socket says 172.21.15.16:60476) 2023-05-22T15:42:51.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:51.711+0000 7fcaf6ffd640 1 -- 172.21.15.16:0/4235645665 learned_addr learned my addr 172.21.15.16:0/4235645665 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:42:51.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:51.711+0000 7fcaf77fe640 1 --2- 172.21.15.16:0/4235645665 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fcaf80b7dd0 0x7fcaf80ccab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:51.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:51.711+0000 7fcaf6ffd640 1 -- 172.21.15.16:0/4235645665 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fcaf80b7dd0 msgr2=0x7fcaf80ccab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:42:51.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:51.711+0000 7fcaf6ffd640 1 --2- 172.21.15.16:0/4235645665 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fcaf80b7dd0 0x7fcaf80ccab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:51.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:51.711+0000 7fcaf6ffd640 1 -- 172.21.15.16:0/4235645665 >> v1:172.21.15.62:6789/0 conn(0x7fcaf8160050 legacy=0x7fcaf80ce720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:51.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:51.711+0000 7fcaf6ffd640 1 -- 172.21.15.16:0/4235645665 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcaf80d2da0 con 0x7fcaf815f700 2023-05-22T15:42:51.719 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:51.715+0000 7fcaf6ffd640 1 --2- 172.21.15.16:0/4235645665 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcaf815f700 0x7fcaf815fb10 secure :-1 s=READY pgs=1250 cs=0 l=1 rev1=1 crypto rx=0x7fcae0009870 tx=0x7fcae001c510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=106cee2d7d5e7427 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:51.719 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:51.715+0000 7fcaf67fc640 1 -- 172.21.15.16:0/4235645665 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcae00a95f0 con 0x7fcaf815f700 2023-05-22T15:42:51.719 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:51.715+0000 7fcaf67fc640 1 -- 172.21.15.16:0/4235645665 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcae00a9750 con 0x7fcaf815f700 2023-05-22T15:42:51.720 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:51.715+0000 7fcaf67fc640 1 -- 172.21.15.16:0/4235645665 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcae00a9ae0 con 0x7fcaf815f700 2023-05-22T15:42:51.720 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:51.715+0000 7fcafe0e9640 1 -- 172.21.15.16:0/4235645665 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcaf815f700 msgr2=0x7fcaf815fb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:51.720 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:51.715+0000 7fcafe0e9640 1 --2- 172.21.15.16:0/4235645665 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcaf815f700 0x7fcaf815fb10 secure :-1 s=READY pgs=1250 cs=0 l=1 rev1=1 crypto rx=0x7fcae0009870 tx=0x7fcae001c510 comp rx=0 tx=0).stop 2023-05-22T15:42:51.721 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:51.715+0000 7fcafe0e9640 1 -- 172.21.15.16:0/4235645665 shutdown_connections 2023-05-22T15:42:51.721 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:51.715+0000 7fcafe0e9640 1 --2- 172.21.15.16:0/4235645665 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcaf815f700 0x7fcaf815fb10 unknown :-1 s=CLOSED pgs=1250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:51.721 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:51.715+0000 7fcafe0e9640 1 --2- 172.21.15.16:0/4235645665 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fcaf80b7dd0 0x7fcaf80ccab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:51.721 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:51.715+0000 7fcafe0e9640 1 -- 172.21.15.16:0/4235645665 >> 172.21.15.16:0/4235645665 conn(0x7fcaf80655c0 msgr2=0x7fcaf8070e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:51.721 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:51.715+0000 7fcafe0e9640 1 -- 172.21.15.16:0/4235645665 shutdown_connections 2023-05-22T15:42:51.721 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:51.715+0000 7fcafe0e9640 1 -- 172.21.15.16:0/4235645665 wait complete. 2023-05-22T15:42:51.723 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:51.715+0000 7fcafe0e9640 1 Processor -- start 2023-05-22T15:42:51.723 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:51.715+0000 7fcafe0e9640 1 -- start start 2023-05-22T15:42:51.723 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:51.715+0000 7fcafe0e9640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcaf80b7dd0 0x7fcaf815cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:51.724 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:51.715+0000 7fcafe0e9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcaf815f700 0x7fcaf80b21b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:51.724 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:51.715+0000 7fcafe0e9640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fcaf8160050 0x7fcaf8153730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:51.724 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:51.719+0000 7fcafe0e9640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcaf815d420 con 0x7fcaf815f700 2023-05-22T15:42:51.724 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:51.719+0000 7fcafe0e9640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcaf815d700 con 0x7fcaf80b7dd0 2023-05-22T15:42:51.724 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:51.719+0000 7fcafe0e9640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcaf815d9e0 con 0x7fcaf8160050 2023-05-22T15:42:51.724 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:51.719+0000 7fcaf6ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcaf815f700 0x7fcaf80b21b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:51.725 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:51.719+0000 7fcaf6ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcaf815f700 0x7fcaf80b21b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60492/0 (socket says 172.21.15.16:60492) 2023-05-22T15:42:51.725 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:51.719+0000 7fcaf6ffd640 1 -- 172.21.15.16:0/2302619887 learned_addr learned my addr 172.21.15.16:0/2302619887 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:42:51.725 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:51.719+0000 7fcaf77fe640 1 --2- 172.21.15.16:0/2302619887 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcaf80b7dd0 0x7fcaf815cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:51.725 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:51.719+0000 7fcaf7fff640 1 --2- 172.21.15.16:0/2302619887 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fcaf8160050 0x7fcaf8153730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:51.725 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:51.719+0000 7fcaf6ffd640 1 -- 172.21.15.16:0/2302619887 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fcaf8160050 msgr2=0x7fcaf8153730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:51.725 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:51.719+0000 7fcaf6ffd640 1 --2- 172.21.15.16:0/2302619887 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fcaf8160050 0x7fcaf8153730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:51.725 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:51.719+0000 7fcaf6ffd640 1 -- 172.21.15.16:0/2302619887 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcaf80b7dd0 msgr2=0x7fcaf815cbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:51.725 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:51.719+0000 7fcaf6ffd640 1 --2- 172.21.15.16:0/2302619887 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcaf80b7dd0 0x7fcaf815cbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:51.726 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:51.719+0000 7fcaf6ffd640 1 -- 172.21.15.16:0/2302619887 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcae00094d0 con 0x7fcaf815f700 2023-05-22T15:42:51.726 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:51.719+0000 7fcaf6ffd640 1 --2- 172.21.15.16:0/2302619887 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcaf815f700 0x7fcaf80b21b0 secure :-1 s=READY pgs=1251 cs=0 l=1 rev1=1 crypto rx=0x7fcae00ab040 tx=0x7fcae001ad50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:51.726 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:51.719+0000 7fcaeffff640 1 -- 172.21.15.16:0/2302619887 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcae00bc650 con 0x7fcaf815f700 2023-05-22T15:42:51.726 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:51.719+0000 7fcaeffff640 1 -- 172.21.15.16:0/2302619887 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcae00bc7b0 con 0x7fcaf815f700 2023-05-22T15:42:51.726 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:51.719+0000 7fcafe0e9640 1 -- 172.21.15.16:0/2302619887 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcaf8168a20 con 0x7fcaf815f700 2023-05-22T15:42:51.726 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:51.719+0000 7fcaeffff640 1 -- 172.21.15.16:0/2302619887 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcae00bcb60 con 0x7fcaf815f700 2023-05-22T15:42:51.726 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:51.719+0000 7fcafe0e9640 1 -- 172.21.15.16:0/2302619887 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcaf8168890 con 0x7fcaf815f700 2023-05-22T15:42:51.727 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:51.719+0000 7fcafe0e9640 1 -- 172.21.15.16:0/2302619887 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcaf8168bb0 con 0x7fcaf815f700 2023-05-22T15:42:51.727 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:51.719+0000 7fcaeffff640 1 -- 172.21.15.16:0/2302619887 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 34) v1 ==== 84332+0+0 (secure 0 0 0) 0x7fcae00bccc0 con 0x7fcaf815f700 2023-05-22T15:42:51.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:51.719+0000 7fcaeffff640 1 --2- 172.21.15.16:0/2302619887 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fcad80654a0 0x7fcad8067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:51.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:51.719+0000 7fcaeffff640 1 -- 172.21.15.16:0/2302619887 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 6261+0+0 (secure 0 0 0) 0x7fcae011d5c0 con 0x7fcaf815f700 2023-05-22T15:42:51.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:51.719+0000 7fcaf77fe640 1 --2- 172.21.15.16:0/2302619887 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fcad80654a0 0x7fcad8067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:51.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:51.719+0000 7fcaf77fe640 1 --2- 172.21.15.16:0/2302619887 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fcad80654a0 0x7fcad8067950 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fcae8003fd0 tx=0x7fcae80039a0 comp rx=0 tx=0).ready entity=mgr.6436 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:51.729 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:51.723+0000 7fcaeffff640 1 -- 172.21.15.16:0/2302619887 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fcae00eaf50 con 0x7fcaf815f700 2023-05-22T15:42:51.914 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:51.907+0000 7fcafe0e9640 1 -- 172.21.15.16:0/2302619887 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcaf815df00 con 0x7fcaf815f700 2023-05-22T15:42:51.914 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:51.911+0000 7fcaeffff640 1 -- 172.21.15.16:0/2302619887 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) v1 ==== 94+0+3526 (secure 0 0 0) 0x7fcae00ee570 con 0x7fcaf815f700 2023-05-22T15:42:51.915 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:42:51.915 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":32,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6833/428835470","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":428835470},{"type":"v1","addr":"172.21.15.62:6833","nonce":428835470}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":25},{"gid":6238,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/4010598958","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":4010598958},{"type":"v1","addr":"172.21.15.62:6835","nonce":4010598958}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":25},{"gid":7008,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/2240458030","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6835","nonce":2240458030},{"type":"v1","addr":"172.21.15.16:6837","nonce":2240458030}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":25},{"gid":7455,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6836/2013184796","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2013184796},{"type":"v1","addr":"172.21.15.16:6836","nonce":2013184796}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":32}],"filesystems":[]} 2023-05-22T15:42:51.915 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 32 2023-05-22T15:42:51.917 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:51.911+0000 7fcafe0e9640 1 -- 172.21.15.16:0/2302619887 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fcad80654a0 msgr2=0x7fcad8067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:51.917 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:51.911+0000 7fcafe0e9640 1 --2- 172.21.15.16:0/2302619887 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fcad80654a0 0x7fcad8067950 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fcae8003fd0 tx=0x7fcae80039a0 comp rx=0 tx=0).stop 2023-05-22T15:42:51.917 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:51.911+0000 7fcafe0e9640 1 -- 172.21.15.16:0/2302619887 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcaf815f700 msgr2=0x7fcaf80b21b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:51.917 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:51.911+0000 7fcafe0e9640 1 --2- 172.21.15.16:0/2302619887 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcaf815f700 0x7fcaf80b21b0 secure :-1 s=READY pgs=1251 cs=0 l=1 rev1=1 crypto rx=0x7fcae00ab040 tx=0x7fcae001ad50 comp rx=0 tx=0).stop 2023-05-22T15:42:51.917 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:51.911+0000 7fcafe0e9640 1 -- 172.21.15.16:0/2302619887 shutdown_connections 2023-05-22T15:42:51.918 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:51.911+0000 7fcafe0e9640 1 --2- 172.21.15.16:0/2302619887 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fcad80654a0 0x7fcad8067950 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:51.918 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:51.911+0000 7fcafe0e9640 1 --2- 172.21.15.16:0/2302619887 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fcaf8160050 0x7fcaf8153730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:51.918 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:51.911+0000 7fcafe0e9640 1 --2- 172.21.15.16:0/2302619887 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcaf815f700 0x7fcaf80b21b0 unknown :-1 s=CLOSED pgs=1251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:51.918 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:51.911+0000 7fcafe0e9640 1 --2- 172.21.15.16:0/2302619887 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcaf80b7dd0 0x7fcaf815cbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:51.918 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:51.911+0000 7fcafe0e9640 1 -- 172.21.15.16:0/2302619887 >> 172.21.15.16:0/2302619887 conn(0x7fcaf80655c0 msgr2=0x7fcaf80b3920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:51.918 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:51.911+0000 7fcafe0e9640 1 -- 172.21.15.16:0/2302619887 shutdown_connections 2023-05-22T15:42:51.918 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:51.911+0000 7fcafe0e9640 1 -- 172.21.15.16:0/2302619887 wait complete. 2023-05-22T15:42:51.934 INFO:tasks.ceph.mds.a:Restarting daemon 2023-05-22T15:42:51.934 INFO:tasks.ceph.mds.a:Stopping old one... 2023-05-22T15:42:51.935 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-05-22T15:42:51.935 INFO:teuthology.orchestra.run:waiting for 300 2023-05-22T15:42:51.935 INFO:tasks.ceph.mds.c:Restarting daemon 2023-05-22T15:42:51.935 INFO:tasks.ceph.mds.c:Stopping old one... 2023-05-22T15:42:51.935 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-05-22T15:42:51.936 INFO:teuthology.orchestra.run:waiting for 300 2023-05-22T15:42:51.936 INFO:tasks.ceph.mds.b:Restarting daemon 2023-05-22T15:42:51.936 INFO:tasks.ceph.mds.b:Stopping old one... 2023-05-22T15:42:51.936 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-05-22T15:42:51.937 INFO:teuthology.orchestra.run:waiting for 300 2023-05-22T15:42:51.937 INFO:tasks.ceph.mds.d:Restarting daemon 2023-05-22T15:42:51.937 INFO:tasks.ceph.mds.d:Stopping old one... 2023-05-22T15:42:51.937 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-05-22T15:42:51.938 INFO:teuthology.orchestra.run:waiting for 300 2023-05-22T15:42:51.948 INFO:tasks.ceph.mds.c:Stopped 2023-05-22T15:42:51.948 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-05-22T15:42:51.949 INFO:tasks.ceph.mds.a:Stopped 2023-05-22T15:42:51.949 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-05-22T15:42:51.950 INFO:tasks.ceph.mds.d:Stopped 2023-05-22T15:42:51.951 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-05-22T15:42:51.951 INFO:tasks.ceph.mds.b:Stopped 2023-05-22T15:42:51.952 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-05-22T15:42:51.994 INFO:tasks.ceph.mds.d:Started 2023-05-22T15:42:51.995 DEBUG:teuthology.parallel:result is None 2023-05-22T15:42:51.996 INFO:tasks.ceph.mds.c:Started 2023-05-22T15:42:51.996 INFO:tasks.ceph.mds.b:Started 2023-05-22T15:42:51.996 INFO:tasks.ceph.mds.a:Started 2023-05-22T15:42:51.997 DEBUG:teuthology.parallel:result is None 2023-05-22T15:42:51.997 DEBUG:teuthology.parallel:result is None 2023-05-22T15:42:51.997 DEBUG:teuthology.parallel:result is None 2023-05-22T15:42:51.998 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2023-05-22T15:42:52.060 INFO:tasks.ceph.mds.c.smithi016.stdout:starting mds.c at 2023-05-22T15:42:52.060 INFO:tasks.ceph.mds.a.smithi016.stdout:starting mds.a at 2023-05-22T15:42:52.074 INFO:tasks.ceph.mds.b.smithi062.stdout:starting mds.b at 2023-05-22T15:42:52.078 INFO:tasks.ceph.mds.d.smithi062.stdout:starting mds.d at 2023-05-22T15:42:52.111 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-05-22T15:42:52.113 DEBUG:teuthology.orchestra.run.smithi062:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-05-22T15:42:52.119 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:52.115+0000 7f45279a6640 1 Processor -- start 2023-05-22T15:42:52.120 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:52.115+0000 7f45279a6640 1 -- start start 2023-05-22T15:42:52.120 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:52.115+0000 7f45279a6640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f45200ce480 0x7f4520071640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:52.121 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:52.115+0000 7f45279a6640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f452015c270 0x7f452015c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:52.121 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:52.115+0000 7f45279a6640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4520069ae0 con 0x7f45200ce850 2023-05-22T15:42:52.121 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:52.115+0000 7f45279a6640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f452015f990 con 0x7f45200ce480 2023-05-22T15:42:52.121 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:52.115+0000 7f45279a6640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4520069dc0 con 0x7f452015c270 2023-05-22T15:42:52.122 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:52.115+0000 7f4525742640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f45200ce480 0x7f4520071640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:52.122 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:52.115+0000 7f4525742640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f45200ce480 0x7f4520071640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60544/0 (socket says 172.21.15.16:60544) 2023-05-22T15:42:52.122 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:52.115+0000 7f4525742640 1 -- 172.21.15.16:0/1672630632 learned_addr learned my addr 172.21.15.16:0/1672630632 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:42:52.122 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:52.115+0000 7f4525f43640 1 --2- 172.21.15.16:0/1672630632 >> v2:172.21.15.62:3300/0 conn(0x7f452015c270 0x7f452015c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:52.122 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:52.115+0000 7f4525742640 1 -- 172.21.15.16:0/1672630632 >> v2:172.21.15.62:3300/0 conn(0x7f452015c270 msgr2=0x7f452015c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:42:52.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:52.115+0000 7f4525742640 1 --2- 172.21.15.16:0/1672630632 >> v2:172.21.15.62:3300/0 conn(0x7f452015c270 0x7f452015c6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:52.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:52.115+0000 7f4525742640 1 -- 172.21.15.16:0/1672630632 >> v1:172.21.15.62:6789/0 conn(0x7f45200ce850 legacy=0x7f4520160090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:52.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:52.115+0000 7f4525742640 1 -- 172.21.15.16:0/1672630632 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f452006a0a0 con 0x7f45200ce480 2023-05-22T15:42:52.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:52.119+0000 7f4525742640 1 --2- 172.21.15.16:0/1672630632 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f45200ce480 0x7f4520071640 secure :-1 s=READY pgs=1256 cs=0 l=1 rev1=1 crypto rx=0x7f45200aeb30 tx=0x7f451c01c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b83426a3eedbe18b server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:52.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:52.119+0000 7f4517fff640 1 -- 172.21.15.16:0/1672630632 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f451c0aa650 con 0x7f45200ce480 2023-05-22T15:42:52.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:52.119+0000 7f4517fff640 1 -- 172.21.15.16:0/1672630632 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f451c0aa7b0 con 0x7f45200ce480 2023-05-22T15:42:52.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:52.119+0000 7f4517fff640 1 -- 172.21.15.16:0/1672630632 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f451c0aab40 con 0x7f45200ce480 2023-05-22T15:42:52.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:52.119+0000 7f45279a6640 1 -- 172.21.15.16:0/1672630632 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f45200ce480 msgr2=0x7f4520071640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:52.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:52.119+0000 7f45279a6640 1 --2- 172.21.15.16:0/1672630632 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f45200ce480 0x7f4520071640 secure :-1 s=READY pgs=1256 cs=0 l=1 rev1=1 crypto rx=0x7f45200aeb30 tx=0x7f451c01c4d0 comp rx=0 tx=0).stop 2023-05-22T15:42:52.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:52.119+0000 7f45279a6640 1 -- 172.21.15.16:0/1672630632 shutdown_connections 2023-05-22T15:42:52.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:52.119+0000 7f45279a6640 1 --2- 172.21.15.16:0/1672630632 >> v2:172.21.15.62:3300/0 conn(0x7f452015c270 0x7f452015c6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:52.126 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:52.119+0000 7f45279a6640 1 --2- 172.21.15.16:0/1672630632 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f45200ce480 0x7f4520071640 unknown :-1 s=CLOSED pgs=1256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:52.126 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:52.119+0000 7f45279a6640 1 -- 172.21.15.16:0/1672630632 >> 172.21.15.16:0/1672630632 conn(0x7f45200cc870 msgr2=0x7f45200cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:52.126 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:52.119+0000 7f45279a6640 1 -- 172.21.15.16:0/1672630632 shutdown_connections 2023-05-22T15:42:52.126 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:52.119+0000 7f45279a6640 1 -- 172.21.15.16:0/1672630632 wait complete. 2023-05-22T15:42:52.128 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:52.119+0000 7f45279a6640 1 Processor -- start 2023-05-22T15:42:52.128 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:52.119+0000 7f45279a6640 1 -- start start 2023-05-22T15:42:52.128 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:52.123+0000 7f45279a6640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f45200ce480 0x7f45200ae110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:52.128 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:52.123+0000 7f45279a6640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f45200ce850 0x7f452014e440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:52.128 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:52.123+0000 7f45279a6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f452015c270 0x7f45201565e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:52.128 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:52.123+0000 7f45279a6640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4520148dd0 con 0x7f452015c270 2023-05-22T15:42:52.129 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:52.123+0000 7f45279a6640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45200ad400 con 0x7f45200ce850 2023-05-22T15:42:52.129 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:52.123+0000 7f45279a6640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4520157db0 con 0x7f45200ce480 2023-05-22T15:42:52.129 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:52.123+0000 7f4525f43640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f452015c270 0x7f45201565e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:52.129 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:52.123+0000 7f4525f43640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f452015c270 0x7f45201565e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60546/0 (socket says 172.21.15.16:60546) 2023-05-22T15:42:52.129 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:52.123+0000 7f4525f43640 1 -- 172.21.15.16:0/627580602 learned_addr learned my addr 172.21.15.16:0/627580602 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:42:52.129 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:52.123+0000 7f4525742640 1 --2- 172.21.15.16:0/627580602 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f45200ce480 0x7f45200ae110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:52.129 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:52.123+0000 7f4524f41640 1 --2- 172.21.15.16:0/627580602 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f45200ce850 0x7f452014e440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:52.129 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:52.123+0000 7f4525f43640 1 -- 172.21.15.16:0/627580602 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f45200ce480 msgr2=0x7f45200ae110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:52.130 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:52.123+0000 7f4525f43640 1 --2- 172.21.15.16:0/627580602 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f45200ce480 0x7f45200ae110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:52.130 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:52.123+0000 7f4525f43640 1 -- 172.21.15.16:0/627580602 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f45200ce850 msgr2=0x7f452014e440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:52.130 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:52.123+0000 7f4525f43640 1 --2- 172.21.15.16:0/627580602 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f45200ce850 0x7f452014e440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:52.130 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:52.123+0000 7f4525f43640 1 -- 172.21.15.16:0/627580602 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f451c0094d0 con 0x7f452015c270 2023-05-22T15:42:52.130 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:52.123+0000 7f4525f43640 1 --2- 172.21.15.16:0/627580602 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f452015c270 0x7f45201565e0 secure :-1 s=READY pgs=1257 cs=0 l=1 rev1=1 crypto rx=0x7f451800efc0 tx=0x7f451800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:52.130 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:52.123+0000 7f4515ffb640 1 -- 172.21.15.16:0/627580602 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4518014480 con 0x7f452015c270 2023-05-22T15:42:52.131 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:52.123+0000 7f4515ffb640 1 -- 172.21.15.16:0/627580602 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f451800f040 con 0x7f452015c270 2023-05-22T15:42:52.131 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:52.123+0000 7f45279a6640 1 -- 172.21.15.16:0/627580602 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45200698b0 con 0x7f452015c270 2023-05-22T15:42:52.131 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:52.123+0000 7f4515ffb640 1 -- 172.21.15.16:0/627580602 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4518014820 con 0x7f452015c270 2023-05-22T15:42:52.131 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:52.123+0000 7f45279a6640 1 -- 172.21.15.16:0/627580602 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4520069720 con 0x7f452015c270 2023-05-22T15:42:52.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:52.123+0000 7f45279a6640 1 -- 172.21.15.16:0/627580602 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45200c5e70 con 0x7f452015c270 2023-05-22T15:42:52.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:52.123+0000 7f4515ffb640 1 -- 172.21.15.16:0/627580602 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 34) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f4518014980 con 0x7f452015c270 2023-05-22T15:42:52.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:52.123+0000 7f4515ffb640 1 --2- 172.21.15.16:0/627580602 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f45040654a0 0x7f4504067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:52.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:52.123+0000 7f4515ffb640 1 -- 172.21.15.16:0/627580602 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 6261+0+0 (secure 0 0 0) 0x7f45180835d0 con 0x7f452015c270 2023-05-22T15:42:52.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:52.123+0000 7f4525742640 1 --2- 172.21.15.16:0/627580602 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f45040654a0 0x7f4504067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:52.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:52.127+0000 7f4525742640 1 --2- 172.21.15.16:0/627580602 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f45040654a0 0x7f4504067950 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f451c0ae000 tx=0x7f451c0b5040 comp rx=0 tx=0).ready entity=mgr.6436 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:52.134 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:52.127+0000 7f4515ffb640 1 -- 172.21.15.16:0/627580602 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f4518051010 con 0x7f452015c270 2023-05-22T15:42:52.297 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:52.291+0000 7f45279a6640 1 -- 172.21.15.16:0/627580602 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f452004dbe0 con 0x7f452015c270 2023-05-22T15:42:53.120 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.115+0000 7f4515ffb640 1 -- 172.21.15.16:0/627580602 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v100) v1 ==== 97+0+0 (secure 0 0 0) 0x7f4518054630 con 0x7f452015c270 2023-05-22T15:42:53.121 INFO:teuthology.orchestra.run.smithi016.stderr: removed all blocklist entries 2023-05-22T15:42:53.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.119+0000 7f45279a6640 1 -- 172.21.15.16:0/627580602 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f45040654a0 msgr2=0x7f4504067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:53.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.119+0000 7f45279a6640 1 --2- 172.21.15.16:0/627580602 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f45040654a0 0x7f4504067950 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f451c0ae000 tx=0x7f451c0b5040 comp rx=0 tx=0).stop 2023-05-22T15:42:53.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.119+0000 7f45279a6640 1 -- 172.21.15.16:0/627580602 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f452015c270 msgr2=0x7f45201565e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:53.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.119+0000 7f45279a6640 1 --2- 172.21.15.16:0/627580602 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f452015c270 0x7f45201565e0 secure :-1 s=READY pgs=1257 cs=0 l=1 rev1=1 crypto rx=0x7f451800efc0 tx=0x7f451800c490 comp rx=0 tx=0).stop 2023-05-22T15:42:53.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.119+0000 7f45279a6640 1 -- 172.21.15.16:0/627580602 shutdown_connections 2023-05-22T15:42:53.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.119+0000 7f45279a6640 1 --2- 172.21.15.16:0/627580602 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f452015c270 0x7f45201565e0 unknown :-1 s=CLOSED pgs=1257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:53.126 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.119+0000 7f45279a6640 1 --2- 172.21.15.16:0/627580602 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f45200ce850 0x7f452014e440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:53.126 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.119+0000 7f45279a6640 1 --2- 172.21.15.16:0/627580602 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f45040654a0 0x7f4504067950 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:53.126 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.119+0000 7f45279a6640 1 --2- 172.21.15.16:0/627580602 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f45200ce480 0x7f45200ae110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:53.126 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.119+0000 7f45279a6640 1 -- 172.21.15.16:0/627580602 >> 172.21.15.16:0/627580602 conn(0x7f45200cc870 msgr2=0x7f4520155340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:53.127 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.119+0000 7f45279a6640 1 -- 172.21.15.16:0/627580602 shutdown_connections 2023-05-22T15:42:53.127 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.119+0000 7f45279a6640 1 -- 172.21.15.16:0/627580602 wait complete. 2023-05-22T15:42:53.143 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2023-05-22T15:42:53.270 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.263+0000 7fbc703ca640 1 Processor -- start 2023-05-22T15:42:53.270 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.263+0000 7fbc703ca640 1 -- start start 2023-05-22T15:42:53.272 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.267+0000 7fbc703ca640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fbc68145690 0x7fbc680ac1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:53.273 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.267+0000 7fbc703ca640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fbc680ac6e0 0x7fbc6806fed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:53.273 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.267+0000 7fbc703ca640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc6815c1f0 con 0x7fbc681452c0 2023-05-22T15:42:53.273 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.267+0000 7fbc703ca640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc6815d0e0 con 0x7fbc680ac6e0 2023-05-22T15:42:53.273 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.267+0000 7fbc703ca640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc680ab860 con 0x7fbc68145690 2023-05-22T15:42:53.273 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.267+0000 7fbc6e967640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fbc680ac6e0 0x7fbc6806fed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:53.273 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.267+0000 7fbc6e967640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fbc680ac6e0 0x7fbc6806fed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60558/0 (socket says 172.21.15.16:60558) 2023-05-22T15:42:53.274 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.267+0000 7fbc6e967640 1 -- 172.21.15.16:0/3873202330 learned_addr learned my addr 172.21.15.16:0/3873202330 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:42:53.274 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.267+0000 7fbc6d965640 1 --2- 172.21.15.16:0/3873202330 >> v2:172.21.15.62:3300/0 conn(0x7fbc68145690 0x7fbc680ac1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:53.274 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.267+0000 7fbc6e967640 1 -- 172.21.15.16:0/3873202330 >> v2:172.21.15.62:3300/0 conn(0x7fbc68145690 msgr2=0x7fbc680ac1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:42:53.274 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.267+0000 7fbc6e967640 1 --2- 172.21.15.16:0/3873202330 >> v2:172.21.15.62:3300/0 conn(0x7fbc68145690 0x7fbc680ac1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:53.274 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.267+0000 7fbc6e967640 1 -- 172.21.15.16:0/3873202330 >> v1:172.21.15.62:6789/0 conn(0x7fbc681452c0 legacy=0x7fbc6815c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:53.274 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.267+0000 7fbc6e967640 1 -- 172.21.15.16:0/3873202330 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc680abb40 con 0x7fbc680ac6e0 2023-05-22T15:42:53.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.267+0000 7fbc6e967640 1 --2- 172.21.15.16:0/3873202330 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc680ac6e0 0x7fbc6806fed0 secure :-1 s=READY pgs=1258 cs=0 l=1 rev1=1 crypto rx=0x7fbc68070dc0 tx=0x7fbc6001c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86f8eb76e4744b64 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:53.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.267+0000 7fbc6d164640 1 -- 172.21.15.16:0/3873202330 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc600a56e0 con 0x7fbc680ac6e0 2023-05-22T15:42:53.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.267+0000 7fbc6d164640 1 -- 172.21.15.16:0/3873202330 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc600a5840 con 0x7fbc680ac6e0 2023-05-22T15:42:53.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.267+0000 7fbc6d164640 1 -- 172.21.15.16:0/3873202330 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc600a5bd0 con 0x7fbc680ac6e0 2023-05-22T15:42:53.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.271+0000 7fbc703ca640 1 -- 172.21.15.16:0/3873202330 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc680ac6e0 msgr2=0x7fbc6806fed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:53.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.271+0000 7fbc703ca640 1 --2- 172.21.15.16:0/3873202330 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc680ac6e0 0x7fbc6806fed0 secure :-1 s=READY pgs=1258 cs=0 l=1 rev1=1 crypto rx=0x7fbc68070dc0 tx=0x7fbc6001c910 comp rx=0 tx=0).stop 2023-05-22T15:42:53.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.271+0000 7fbc703ca640 1 -- 172.21.15.16:0/3873202330 shutdown_connections 2023-05-22T15:42:53.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.271+0000 7fbc703ca640 1 --2- 172.21.15.16:0/3873202330 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc680ac6e0 0x7fbc6806fed0 unknown :-1 s=CLOSED pgs=1258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:53.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.271+0000 7fbc703ca640 1 --2- 172.21.15.16:0/3873202330 >> v2:172.21.15.62:3300/0 conn(0x7fbc68145690 0x7fbc680ac1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:53.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.271+0000 7fbc703ca640 1 -- 172.21.15.16:0/3873202330 >> 172.21.15.16:0/3873202330 conn(0x7fbc6806e3c0 msgr2=0x7fbc680b1850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:53.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.271+0000 7fbc703ca640 1 -- 172.21.15.16:0/3873202330 shutdown_connections 2023-05-22T15:42:53.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.271+0000 7fbc703ca640 1 -- 172.21.15.16:0/3873202330 wait complete. 2023-05-22T15:42:53.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.271+0000 7fbc703ca640 1 Processor -- start 2023-05-22T15:42:53.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.271+0000 7fbc703ca640 1 -- start start 2023-05-22T15:42:53.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.271+0000 7fbc703ca640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbc680ac6e0 0x7fbc68159740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:53.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.271+0000 7fbc703ca640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc681452c0 0x7fbc68158710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:53.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.271+0000 7fbc703ca640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbc68145690 0x7fbc6813a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:53.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.271+0000 7fbc703ca640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc68158c50 con 0x7fbc681452c0 2023-05-22T15:42:53.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.271+0000 7fbc703ca640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc680cdf30 con 0x7fbc68145690 2023-05-22T15:42:53.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.271+0000 7fbc703ca640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc681408f0 con 0x7fbc680ac6e0 2023-05-22T15:42:53.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.271+0000 7fbc6d965640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc681452c0 0x7fbc68158710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:53.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.271+0000 7fbc6d965640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc681452c0 0x7fbc68158710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60574/0 (socket says 172.21.15.16:60574) 2023-05-22T15:42:53.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.271+0000 7fbc6d965640 1 -- 172.21.15.16:0/947815185 learned_addr learned my addr 172.21.15.16:0/947815185 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:42:53.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.271+0000 7fbc6d965640 1 -- 172.21.15.16:0/947815185 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbc680ac6e0 msgr2=0x7fbc68159740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:53.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.271+0000 7fbc6e967640 1 --2- 172.21.15.16:0/947815185 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbc68145690 0x7fbc6813a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:53.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.271+0000 7fbc6e166640 1 --2- 172.21.15.16:0/947815185 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbc680ac6e0 0x7fbc68159740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:53.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.271+0000 7fbc6d965640 1 --2- 172.21.15.16:0/947815185 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbc680ac6e0 0x7fbc68159740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:53.282 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.271+0000 7fbc6d965640 1 -- 172.21.15.16:0/947815185 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbc68145690 msgr2=0x7fbc6813a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:53.282 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.271+0000 7fbc6d965640 1 --2- 172.21.15.16:0/947815185 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbc68145690 0x7fbc6813a9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:53.282 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.271+0000 7fbc6d965640 1 -- 172.21.15.16:0/947815185 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc60009b30 con 0x7fbc681452c0 2023-05-22T15:42:53.282 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.271+0000 7fbc6d965640 1 --2- 172.21.15.16:0/947815185 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc681452c0 0x7fbc68158710 secure :-1 s=READY pgs=1259 cs=0 l=1 rev1=1 crypto rx=0x7fbc5800c7f0 tx=0x7fbc5800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:53.282 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.271+0000 7fbc5effd640 1 -- 172.21.15.16:0/947815185 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc58007820 con 0x7fbc681452c0 2023-05-22T15:42:53.282 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.271+0000 7fbc5effd640 1 -- 172.21.15.16:0/947815185 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc58007980 con 0x7fbc681452c0 2023-05-22T15:42:53.282 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.271+0000 7fbc703ca640 1 -- 172.21.15.16:0/947815185 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc68160740 con 0x7fbc681452c0 2023-05-22T15:42:53.282 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.271+0000 7fbc5effd640 1 -- 172.21.15.16:0/947815185 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc58007d30 con 0x7fbc681452c0 2023-05-22T15:42:53.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.271+0000 7fbc703ca640 1 -- 172.21.15.16:0/947815185 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc681605b0 con 0x7fbc681452c0 2023-05-22T15:42:53.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.271+0000 7fbc703ca640 1 -- 172.21.15.16:0/947815185 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc68168b80 con 0x7fbc681452c0 2023-05-22T15:42:53.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.275+0000 7fbc5effd640 1 -- 172.21.15.16:0/947815185 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 34) v1 ==== 84332+0+0 (secure 0 0 0) 0x7fbc580113f0 con 0x7fbc681452c0 2023-05-22T15:42:53.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.275+0000 7fbc5effd640 1 --2- 172.21.15.16:0/947815185 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fbc4c0654a0 0x7fbc4c067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:53.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.275+0000 7fbc5effd640 1 -- 172.21.15.16:0/947815185 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5530+0+0 (secure 0 0 0) 0x7fbc58082bd0 con 0x7fbc681452c0 2023-05-22T15:42:53.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.275+0000 7fbc6e166640 1 --2- 172.21.15.16:0/947815185 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fbc4c0654a0 0x7fbc4c067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:53.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.275+0000 7fbc6e166640 1 --2- 172.21.15.16:0/947815185 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fbc4c0654a0 0x7fbc4c067950 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fbc64000f70 tx=0x7fbc64009040 comp rx=0 tx=0).ready entity=mgr.6436 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:53.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.279+0000 7fbc5effd640 1 -- 172.21.15.16:0/947815185 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fbc580507e0 con 0x7fbc681452c0 2023-05-22T15:42:53.492 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.487+0000 7fbc703ca640 1 -- 172.21.15.16:0/947815185 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fbc6804dbe0 con 0x7fbc681452c0 2023-05-22T15:42:53.494 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.487+0000 7fbc5effd640 1 -- 172.21.15.16:0/947815185 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v20) v1 ==== 80+0+5304 (secure 0 0 0) 0x7fbc5801f090 con 0x7fbc681452c0 2023-05-22T15:42:53.495 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:42:53.495 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-05-22T15:42:53.495 INFO:teuthology.orchestra.run.smithi016.stdout: "auth_dump": [ 2023-05-22T15:42:53.495 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T15:42:53.495 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "mds.a", 2023-05-22T15:42:53.496 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQCAiWtkw/K0GRAA9hvTMKMmcbgKV68GeoC4ag==", 2023-05-22T15:42:53.496 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-05-22T15:42:53.496 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2023-05-22T15:42:53.496 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow *", 2023-05-22T15:42:53.496 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow *", 2023-05-22T15:42:53.497 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-05-22T15:42:53.497 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T15:42:53.497 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:42:53.497 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T15:42:53.497 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "mds.b", 2023-05-22T15:42:53.497 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQCAiWtkoFzfJRAAxCiB0MGM5apMVA81WDq+dA==", 2023-05-22T15:42:53.497 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-05-22T15:42:53.498 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2023-05-22T15:42:53.498 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow *", 2023-05-22T15:42:53.498 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow *", 2023-05-22T15:42:53.498 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-05-22T15:42:53.498 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T15:42:53.498 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:42:53.498 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T15:42:53.499 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "mds.c", 2023-05-22T15:42:53.499 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQCAiWtkvgOKHxAAMuASPtMqwyoqpZ5I+MRS7A==", 2023-05-22T15:42:53.499 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-05-22T15:42:53.499 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2023-05-22T15:42:53.499 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow *", 2023-05-22T15:42:53.499 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow *", 2023-05-22T15:42:53.499 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-05-22T15:42:53.500 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T15:42:53.500 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:42:53.500 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T15:42:53.500 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "mds.d", 2023-05-22T15:42:53.500 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQCAiWtk6hPQKxAAlCtfC36vja5ftz6uOqUhHg==", 2023-05-22T15:42:53.501 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-05-22T15:42:53.501 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2023-05-22T15:42:53.501 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow *", 2023-05-22T15:42:53.501 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow *", 2023-05-22T15:42:53.501 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-05-22T15:42:53.501 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T15:42:53.501 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:42:53.501 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T15:42:53.502 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.0", 2023-05-22T15:42:53.502 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQCCiWtk6ZLoCRAAKdCQitZuPnWcIpkiXhJeww==", 2023-05-22T15:42:53.502 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-05-22T15:42:53.502 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2023-05-22T15:42:53.502 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2023-05-22T15:42:53.502 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-05-22T15:42:53.502 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T15:42:53.503 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:42:53.503 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T15:42:53.503 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.1", 2023-05-22T15:42:53.503 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQCFiWtkXtVeERAAkgPeDKuFmWXrDiCyqWOYwQ==", 2023-05-22T15:42:53.503 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-05-22T15:42:53.503 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2023-05-22T15:42:53.504 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2023-05-22T15:42:53.504 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-05-22T15:42:53.504 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T15:42:53.504 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:42:53.504 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T15:42:53.504 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.2", 2023-05-22T15:42:53.504 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQCIiWtkdGaRHRAAx3POIp3mnz7p7XcCVMXm5Q==", 2023-05-22T15:42:53.505 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-05-22T15:42:53.505 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2023-05-22T15:42:53.505 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2023-05-22T15:42:53.505 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-05-22T15:42:53.505 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T15:42:53.505 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:42:53.505 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T15:42:53.506 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.3", 2023-05-22T15:42:53.506 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQCLiWtkSBydJhAAsgc19Xaf/GOyy9nm9MLdJQ==", 2023-05-22T15:42:53.506 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-05-22T15:42:53.506 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2023-05-22T15:42:53.506 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2023-05-22T15:42:53.506 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-05-22T15:42:53.506 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T15:42:53.507 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:42:53.507 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T15:42:53.507 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.4", 2023-05-22T15:42:53.507 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQCRiWtkueYvJRAARXbHlKHrTbgTqBArRJfvBA==", 2023-05-22T15:42:53.507 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-05-22T15:42:53.508 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2023-05-22T15:42:53.508 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2023-05-22T15:42:53.508 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-05-22T15:42:53.508 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T15:42:53.508 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:42:53.508 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T15:42:53.508 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.5", 2023-05-22T15:42:53.509 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQCUiWtkH3hwLRAAUekP6y+wkJVPh+OPDiZbzQ==", 2023-05-22T15:42:53.509 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-05-22T15:42:53.509 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2023-05-22T15:42:53.509 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2023-05-22T15:42:53.510 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-05-22T15:42:53.510 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T15:42:53.510 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:42:53.510 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T15:42:53.510 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.6", 2023-05-22T15:42:53.510 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQCXiWtkLsnBNhAAQ8ilX8ZU4ncJ2JJU0I+U4w==", 2023-05-22T15:42:53.511 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-05-22T15:42:53.511 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2023-05-22T15:42:53.511 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2023-05-22T15:42:53.511 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-05-22T15:42:53.511 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T15:42:53.511 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:42:53.511 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T15:42:53.511 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.7", 2023-05-22T15:42:53.512 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQCbiWtkUvXjBBAApMWw7wdfxWq0WEQDL2qprA==", 2023-05-22T15:42:53.512 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-05-22T15:42:53.512 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2023-05-22T15:42:53.512 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2023-05-22T15:42:53.512 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-05-22T15:42:53.512 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T15:42:53.512 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:42:53.513 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T15:42:53.513 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "client.0", 2023-05-22T15:42:53.513 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQCAiWtkZwuUMRAAnYqSFiaT/QRsajdjNgTetA==", 2023-05-22T15:42:53.513 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-05-22T15:42:53.513 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2023-05-22T15:42:53.513 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow r", 2023-05-22T15:42:53.513 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow rw tag cephfs data=cephfs" 2023-05-22T15:42:53.514 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T15:42:53.514 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:42:53.514 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T15:42:53.514 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "client.1", 2023-05-22T15:42:53.514 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQCAiWtkPimmORAABzNL2rANz5TgCl3p++GVng==", 2023-05-22T15:42:53.515 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-05-22T15:42:53.515 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2023-05-22T15:42:53.515 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow r", 2023-05-22T15:42:53.515 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow rw", 2023-05-22T15:42:53.515 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow rwx" 2023-05-22T15:42:53.515 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T15:42:53.515 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:42:53.515 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T15:42:53.516 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "client.2", 2023-05-22T15:42:53.516 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQCAiWtkqkPMNhAAqugrnD4Mh/4At8QAFmzqcg==", 2023-05-22T15:42:53.516 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-05-22T15:42:53.516 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2023-05-22T15:42:53.516 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow r", 2023-05-22T15:42:53.516 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow rw", 2023-05-22T15:42:53.516 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow rwx" 2023-05-22T15:42:53.517 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T15:42:53.517 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:42:53.517 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T15:42:53.517 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "client.3", 2023-05-22T15:42:53.517 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQCBiWtkpvlWAxAAPuedivXBGZGHIO6hSnz8PQ==", 2023-05-22T15:42:53.517 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-05-22T15:42:53.518 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2023-05-22T15:42:53.518 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow r", 2023-05-22T15:42:53.518 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow rw", 2023-05-22T15:42:53.518 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow rwx" 2023-05-22T15:42:53.518 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T15:42:53.518 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:42:53.519 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T15:42:53.519 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "client.admin", 2023-05-22T15:42:53.519 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQCAiWtkxG9hAhAAfnUnRrBRUFoCEtsb2riNOQ==", 2023-05-22T15:42:53.519 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-05-22T15:42:53.519 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow *", 2023-05-22T15:42:53.519 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow *", 2023-05-22T15:42:53.520 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow *", 2023-05-22T15:42:53.520 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-05-22T15:42:53.520 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T15:42:53.520 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:42:53.521 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T15:42:53.521 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "mgr.x", 2023-05-22T15:42:53.521 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQCAiWtkkaIqFhAAb/4BI6FYWOy3qOSaylXLSw==", 2023-05-22T15:42:53.521 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-05-22T15:42:53.522 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow *", 2023-05-22T15:42:53.522 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile mgr", 2023-05-22T15:42:53.522 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-05-22T15:42:53.522 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T15:42:53.522 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:42:53.523 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T15:42:53.523 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "mgr.y", 2023-05-22T15:42:53.523 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQCAiWtkhgVWExAAjDoh7cexFVfsigsYfGkmVg==", 2023-05-22T15:42:53.523 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-05-22T15:42:53.524 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow *", 2023-05-22T15:42:53.524 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile mgr", 2023-05-22T15:42:53.524 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-05-22T15:42:53.524 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T15:42:53.524 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T15:42:53.525 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-05-22T15:42:53.525 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-05-22T15:42:53.526 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.491+0000 7fbc703ca640 1 -- 172.21.15.16:0/947815185 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fbc4c0654a0 msgr2=0x7fbc4c067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:53.527 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.491+0000 7fbc703ca640 1 --2- 172.21.15.16:0/947815185 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fbc4c0654a0 0x7fbc4c067950 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fbc64000f70 tx=0x7fbc64009040 comp rx=0 tx=0).stop 2023-05-22T15:42:53.527 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.491+0000 7fbc703ca640 1 -- 172.21.15.16:0/947815185 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc681452c0 msgr2=0x7fbc68158710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:53.527 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.491+0000 7fbc703ca640 1 --2- 172.21.15.16:0/947815185 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc681452c0 0x7fbc68158710 secure :-1 s=READY pgs=1259 cs=0 l=1 rev1=1 crypto rx=0x7fbc5800c7f0 tx=0x7fbc5800ccb0 comp rx=0 tx=0).stop 2023-05-22T15:42:53.527 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.491+0000 7fbc703ca640 1 -- 172.21.15.16:0/947815185 shutdown_connections 2023-05-22T15:42:53.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.491+0000 7fbc703ca640 1 --2- 172.21.15.16:0/947815185 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbc68145690 0x7fbc6813a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:53.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.491+0000 7fbc703ca640 1 --2- 172.21.15.16:0/947815185 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc681452c0 0x7fbc68158710 unknown :-1 s=CLOSED pgs=1259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:53.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.491+0000 7fbc703ca640 1 --2- 172.21.15.16:0/947815185 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fbc4c0654a0 0x7fbc4c067950 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:53.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.491+0000 7fbc703ca640 1 --2- 172.21.15.16:0/947815185 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbc680ac6e0 0x7fbc68159740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:53.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.491+0000 7fbc703ca640 1 -- 172.21.15.16:0/947815185 >> 172.21.15.16:0/947815185 conn(0x7fbc6806e3c0 msgr2=0x7fbc68141370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:53.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.491+0000 7fbc703ca640 1 -- 172.21.15.16:0/947815185 shutdown_connections 2023-05-22T15:42:53.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.491+0000 7fbc703ca640 1 -- 172.21.15.16:0/947815185 wait complete. 2023-05-22T15:42:53.530 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-05-22T15:42:53.654 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.651+0000 7fdfd47fc640 1 Processor -- start 2023-05-22T15:42:53.655 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.651+0000 7fdfd47fc640 1 -- start start 2023-05-22T15:42:53.656 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.651+0000 7fdfd47fc640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fdfcc1567c0 0x7fdfcc0afb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:53.656 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.651+0000 7fdfd47fc640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fdfcc159a50 0x7fdfcc0b0080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:53.656 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.651+0000 7fdfd47fc640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdfcc156b90 con 0x7fdfcc0cb6a0 2023-05-22T15:42:53.656 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.651+0000 7fdfd47fc640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdfcc159ec0 con 0x7fdfcc1567c0 2023-05-22T15:42:53.656 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.651+0000 7fdfd47fc640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdfcc15a0b0 con 0x7fdfcc159a50 2023-05-22T15:42:53.657 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.651+0000 7fdfd2598640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fdfcc1567c0 0x7fdfcc0afb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:53.657 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.651+0000 7fdfd1d97640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fdfcc0cb6a0 0x7fdfcc159340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:44846/0 (socket says 172.21.15.16:44846) 2023-05-22T15:42:53.657 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.651+0000 7fdfd1d97640 1 -- 172.21.15.16:0/1630880573 learned_addr learned my addr 172.21.15.16:0/1630880573 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:42:53.657 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.651+0000 7fdfd2d99640 1 --2- 172.21.15.16:0/1630880573 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fdfcc159a50 0x7fdfcc0b0080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:53.657 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.651+0000 7fdfd2598640 1 -- 172.21.15.16:0/1630880573 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fdfcc159a50 msgr2=0x7fdfcc0b0080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:42:53.657 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.651+0000 7fdfd2598640 1 --2- 172.21.15.16:0/1630880573 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fdfcc159a50 0x7fdfcc0b0080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:53.657 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.651+0000 7fdfd2598640 1 -- 172.21.15.16:0/1630880573 >> v1:172.21.15.16:6789/0 conn(0x7fdfcc0cb6a0 legacy=0x7fdfcc159340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:53.658 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.651+0000 7fdfd2598640 1 -- 172.21.15.16:0/1630880573 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfcc0ca8e0 con 0x7fdfcc1567c0 2023-05-22T15:42:53.658 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.651+0000 7fdfd1596640 1 -- 172.21.15.16:0/1630880573 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2226348715 0 0) 0x7fdfc80014a0 con 0x7fdfcc0cb6a0 2023-05-22T15:42:53.659 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.651+0000 7fdfd2598640 1 --2- 172.21.15.16:0/1630880573 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdfcc1567c0 0x7fdfcc0afb40 secure :-1 s=READY pgs=1261 cs=0 l=1 rev1=1 crypto rx=0x7fdfc00097c0 tx=0x7fdfc001c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=caa8471031051afd server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:53.659 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.655+0000 7fdfd1596640 1 -- 172.21.15.16:0/1630880573 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfc00ab600 con 0x7fdfcc1567c0 2023-05-22T15:42:53.659 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.655+0000 7fdfd1596640 1 -- 172.21.15.16:0/1630880573 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdfc00a7050 con 0x7fdfcc1567c0 2023-05-22T15:42:53.659 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.655+0000 7fdfd1596640 1 -- 172.21.15.16:0/1630880573 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfc00ab980 con 0x7fdfcc1567c0 2023-05-22T15:42:53.659 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.655+0000 7fdfd47fc640 1 -- 172.21.15.16:0/1630880573 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdfcc1567c0 msgr2=0x7fdfcc0afb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:53.660 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.655+0000 7fdfd47fc640 1 --2- 172.21.15.16:0/1630880573 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdfcc1567c0 0x7fdfcc0afb40 secure :-1 s=READY pgs=1261 cs=0 l=1 rev1=1 crypto rx=0x7fdfc00097c0 tx=0x7fdfc001c480 comp rx=0 tx=0).stop 2023-05-22T15:42:53.660 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.655+0000 7fdfd47fc640 1 -- 172.21.15.16:0/1630880573 shutdown_connections 2023-05-22T15:42:53.660 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.655+0000 7fdfd47fc640 1 --2- 172.21.15.16:0/1630880573 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fdfcc159a50 0x7fdfcc0b0080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:53.660 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.655+0000 7fdfd47fc640 1 --2- 172.21.15.16:0/1630880573 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdfcc1567c0 0x7fdfcc0afb40 unknown :-1 s=CLOSED pgs=1261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:53.661 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.655+0000 7fdfd47fc640 1 -- 172.21.15.16:0/1630880573 >> 172.21.15.16:0/1630880573 conn(0x7fdfcc069720 msgr2=0x7fdfcc069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:53.661 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.655+0000 7fdfd47fc640 1 -- 172.21.15.16:0/1630880573 shutdown_connections 2023-05-22T15:42:53.661 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.655+0000 7fdfd47fc640 1 -- 172.21.15.16:0/1630880573 wait complete. 2023-05-22T15:42:53.662 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.655+0000 7fdfd47fc640 1 Processor -- start 2023-05-22T15:42:53.662 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.655+0000 7fdfd47fc640 1 -- start start 2023-05-22T15:42:53.663 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.655+0000 7fdfd47fc640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdfcc0cb6a0 0x7fdfcc14c0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:53.663 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.655+0000 7fdfd47fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdfcc1567c0 0x7fdfcc0b3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:53.663 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.655+0000 7fdfd47fc640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fdfcc159a50 0x7fdfcc0c67f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:53.663 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.655+0000 7fdfd47fc640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfcc14d750 con 0x7fdfcc1567c0 2023-05-22T15:42:53.663 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.655+0000 7fdfd47fc640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfcc14cad0 con 0x7fdfcc0cb6a0 2023-05-22T15:42:53.663 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.655+0000 7fdfd47fc640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdfcc14cdb0 con 0x7fdfcc159a50 2023-05-22T15:42:53.663 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.655+0000 7fdfd1d97640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdfcc1567c0 0x7fdfcc0b3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:53.664 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.655+0000 7fdfd1d97640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdfcc1567c0 0x7fdfcc0b3660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60582/0 (socket says 172.21.15.16:60582) 2023-05-22T15:42:53.664 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.655+0000 7fdfd1d97640 1 -- 172.21.15.16:0/2244676183 learned_addr learned my addr 172.21.15.16:0/2244676183 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:42:53.664 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.655+0000 7fdfd2598640 1 --2- 172.21.15.16:0/2244676183 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdfcc0cb6a0 0x7fdfcc14c0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:53.664 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.655+0000 7fdfd2d99640 1 --2- 172.21.15.16:0/2244676183 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fdfcc159a50 0x7fdfcc0c67f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:53.664 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.655+0000 7fdfd1d97640 1 -- 172.21.15.16:0/2244676183 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fdfcc159a50 msgr2=0x7fdfcc0c67f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:53.664 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.655+0000 7fdfd1d97640 1 --2- 172.21.15.16:0/2244676183 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fdfcc159a50 0x7fdfcc0c67f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:53.665 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.655+0000 7fdfd1d97640 1 -- 172.21.15.16:0/2244676183 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdfcc0cb6a0 msgr2=0x7fdfcc14c0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:53.665 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.655+0000 7fdfd1d97640 1 --2- 172.21.15.16:0/2244676183 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdfcc0cb6a0 0x7fdfcc14c0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:53.665 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.655+0000 7fdfd1d97640 1 -- 172.21.15.16:0/2244676183 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfc0009420 con 0x7fdfcc1567c0 2023-05-22T15:42:53.665 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.655+0000 7fdfd2d99640 1 --2- 172.21.15.16:0/2244676183 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fdfcc159a50 0x7fdfcc0c67f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T15:42:53.665 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.655+0000 7fdfd2598640 1 --2- 172.21.15.16:0/2244676183 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdfcc0cb6a0 0x7fdfcc14c0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T15:42:53.665 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.655+0000 7fdfd1d97640 1 --2- 172.21.15.16:0/2244676183 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdfcc1567c0 0x7fdfcc0b3660 secure :-1 s=READY pgs=1262 cs=0 l=1 rev1=1 crypto rx=0x7fdfc800ed00 tx=0x7fdfc800c640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:53.665 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.655+0000 7fdfbeffd640 1 -- 172.21.15.16:0/2244676183 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfc800cd50 con 0x7fdfcc1567c0 2023-05-22T15:42:53.665 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.655+0000 7fdfbeffd640 1 -- 172.21.15.16:0/2244676183 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdfc8004b70 con 0x7fdfcc1567c0 2023-05-22T15:42:53.666 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.655+0000 7fdfd47fc640 1 -- 172.21.15.16:0/2244676183 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdfcc0c8e80 con 0x7fdfcc1567c0 2023-05-22T15:42:53.666 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.655+0000 7fdfd47fc640 1 -- 172.21.15.16:0/2244676183 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdfcc0cb180 con 0x7fdfcc1567c0 2023-05-22T15:42:53.666 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.659+0000 7fdfbeffd640 1 -- 172.21.15.16:0/2244676183 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfc80070d0 con 0x7fdfcc1567c0 2023-05-22T15:42:53.667 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.659+0000 7fdfd47fc640 1 -- 172.21.15.16:0/2244676183 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdfb4005260 con 0x7fdfcc1567c0 2023-05-22T15:42:53.667 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.659+0000 7fdfbeffd640 1 -- 172.21.15.16:0/2244676183 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 34) v1 ==== 84332+0+0 (secure 0 0 0) 0x7fdfc8007230 con 0x7fdfcc1567c0 2023-05-22T15:42:53.667 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.659+0000 7fdfbeffd640 1 --2- 172.21.15.16:0/2244676183 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fdfb00654a0 0x7fdfb0067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:53.667 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.659+0000 7fdfbeffd640 1 -- 172.21.15.16:0/2244676183 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5530+0+0 (secure 0 0 0) 0x7fdfc8085150 con 0x7fdfcc1567c0 2023-05-22T15:42:53.667 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.659+0000 7fdfd2598640 1 --2- 172.21.15.16:0/2244676183 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fdfb00654a0 0x7fdfb0067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:53.667 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.659+0000 7fdfd2598640 1 --2- 172.21.15.16:0/2244676183 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fdfb00654a0 0x7fdfb0067950 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fdfc00ad000 tx=0x7fdfc00a6000 comp rx=0 tx=0).ready entity=mgr.6436 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:53.668 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.663+0000 7fdfbeffd640 1 -- 172.21.15.16:0/2244676183 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fdfc8052db0 con 0x7fdfcc1567c0 2023-05-22T15:42:53.824 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.819+0000 7fdfd47fc640 1 -- 172.21.15.16:0/2244676183 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdfb40057e0 con 0x7fdfcc1567c0 2023-05-22T15:42:53.825 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.819+0000 7fdfbeffd640 1 -- 172.21.15.16:0/2244676183 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v100) v1 ==== 74+0+14330 (secure 0 0 0) 0x7fdfc8015090 con 0x7fdfcc1567c0 2023-05-22T15:42:53.825 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:42:53.826 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":100,"fsid":"b2719834-7847-4d98-90a6-f65d57f85c49","created":"2023-05-22T15:26:25.259314+0000","modified":"2023-05-22T15:42:53.108370+0000","last_up_change":"2023-05-22T15:26:36.448524+0000","last_in_change":"2023-05-22T15:26:27.675668+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-05-22T15:26:38.019781+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-05-22T15:26:43.570514+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9e7ff854-b25f-4cbb-a5c2-1799d9c53d2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":37021},{"type":"v1","addr":"172.21.15.16:6825","nonce":37021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":37021},{"type":"v1","addr":"172.21.15.16:6827","nonce":37021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":37021},{"type":"v1","addr":"172.21.15.16:6831","nonce":37021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":37021},{"type":"v1","addr":"172.21.15.16:6829","nonce":37021}]},"public_addr":"172.21.15.16:6825/37021","cluster_addr":"172.21.15.16:6827/37021","heartbeat_back_addr":"172.21.15.16:6831/37021","heartbeat_front_addr":"172.21.15.16:6829/37021","state":["exists","up"]},{"osd":1,"uuid":"4702fbc9-a927-4565-aca5-41dc69cc8264","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":37008},{"type":"v1","addr":"172.21.15.16:6801","nonce":37008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":37008},{"type":"v1","addr":"172.21.15.16:6803","nonce":37008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":37008},{"type":"v1","addr":"172.21.15.16:6807","nonce":37008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":37008},{"type":"v1","addr":"172.21.15.16:6805","nonce":37008}]},"public_addr":"172.21.15.16:6801/37008","cluster_addr":"172.21.15.16:6803/37008","heartbeat_back_addr":"172.21.15.16:6807/37008","heartbeat_front_addr":"172.21.15.16:6805/37008","state":["exists","up"]},{"osd":2,"uuid":"1e7bba46-4eba-47ef-bb81-5e2d371638ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":37009},{"type":"v1","addr":"172.21.15.16:6809","nonce":37009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":37009},{"type":"v1","addr":"172.21.15.16:6811","nonce":37009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":37009},{"type":"v1","addr":"172.21.15.16:6815","nonce":37009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":37009},{"type":"v1","addr":"172.21.15.16:6813","nonce":37009}]},"public_addr":"172.21.15.16:6809/37009","cluster_addr":"172.21.15.16:6811/37009","heartbeat_back_addr":"172.21.15.16:6815/37009","heartbeat_front_addr":"172.21.15.16:6813/37009","state":["exists","up"]},{"osd":3,"uuid":"383cacc0-5a88-4904-990e-498ca59ef48a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":37020},{"type":"v1","addr":"172.21.15.16:6817","nonce":37020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":37020},{"type":"v1","addr":"172.21.15.16:6819","nonce":37020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":37020},{"type":"v1","addr":"172.21.15.16:6823","nonce":37020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":37020},{"type":"v1","addr":"172.21.15.16:6821","nonce":37020}]},"public_addr":"172.21.15.16:6817/37020","cluster_addr":"172.21.15.16:6819/37020","heartbeat_back_addr":"172.21.15.16:6823/37020","heartbeat_front_addr":"172.21.15.16:6821/37020","state":["exists","up"]},{"osd":4,"uuid":"970500ef-95b7-42c1-b080-eed4fae7013d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":37353},{"type":"v1","addr":"172.21.15.62:6817","nonce":37353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6819","nonce":37353},{"type":"v1","addr":"172.21.15.62:6822","nonce":37353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37353},{"type":"v1","addr":"172.21.15.62:6831","nonce":37353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37353},{"type":"v1","addr":"172.21.15.62:6826","nonce":37353}]},"public_addr":"172.21.15.62:6817/37353","cluster_addr":"172.21.15.62:6822/37353","heartbeat_back_addr":"172.21.15.62:6831/37353","heartbeat_front_addr":"172.21.15.62:6826/37353","state":["exists","up"]},{"osd":5,"uuid":"022d67db-d4d8-445b-b1b9-8b446a00f5c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":37352},{"type":"v1","addr":"172.21.15.62:6801","nonce":37352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":37352},{"type":"v1","addr":"172.21.15.62:6803","nonce":37352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":37352},{"type":"v1","addr":"172.21.15.62:6807","nonce":37352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":37352},{"type":"v1","addr":"172.21.15.62:6805","nonce":37352}]},"public_addr":"172.21.15.62:6801/37352","cluster_addr":"172.21.15.62:6803/37352","heartbeat_back_addr":"172.21.15.62:6807/37352","heartbeat_front_addr":"172.21.15.62:6805/37352","state":["exists","up"]},{"osd":6,"uuid":"42bd372b-6688-4a1b-8407-e16e8c266e33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":37354},{"type":"v1","addr":"172.21.15.62:6820","nonce":37354}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":37354},{"type":"v1","addr":"172.21.15.62:6823","nonce":37354}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6829","nonce":37354},{"type":"v1","addr":"172.21.15.62:6830","nonce":37354}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6825","nonce":37354},{"type":"v1","addr":"172.21.15.62:6827","nonce":37354}]},"public_addr":"172.21.15.62:6820/37354","cluster_addr":"172.21.15.62:6823/37354","heartbeat_back_addr":"172.21.15.62:6830/37354","heartbeat_front_addr":"172.21.15.62:6827/37354","state":["exists","up"]},{"osd":7,"uuid":"ac70b2a3-f41b-4bf2-b227-7f2cbb55426d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37355},{"type":"v1","addr":"172.21.15.62:6809","nonce":37355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37355},{"type":"v1","addr":"172.21.15.62:6811","nonce":37355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":37355},{"type":"v1","addr":"172.21.15.62:6815","nonce":37355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":37355},{"type":"v1","addr":"172.21.15.62:6813","nonce":37355}]},"public_addr":"172.21.15.62:6809/37355","cluster_addr":"172.21.15.62:6811/37355","heartbeat_back_addr":"172.21.15.62:6815/37355","heartbeat_front_addr":"172.21.15.62:6813/37355","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.368896+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.107273+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.276888+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.287502+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.124791+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.251937+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.255679+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.232797+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-05-22T15:42:53.827 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.823+0000 7fdfd47fc640 1 -- 172.21.15.16:0/2244676183 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fdfb00654a0 msgr2=0x7fdfb0067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:53.827 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.823+0000 7fdfd47fc640 1 --2- 172.21.15.16:0/2244676183 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fdfb00654a0 0x7fdfb0067950 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fdfc00ad000 tx=0x7fdfc00a6000 comp rx=0 tx=0).stop 2023-05-22T15:42:53.828 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.823+0000 7fdfd47fc640 1 -- 172.21.15.16:0/2244676183 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdfcc1567c0 msgr2=0x7fdfcc0b3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:53.828 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.823+0000 7fdfd47fc640 1 --2- 172.21.15.16:0/2244676183 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdfcc1567c0 0x7fdfcc0b3660 secure :-1 s=READY pgs=1262 cs=0 l=1 rev1=1 crypto rx=0x7fdfc800ed00 tx=0x7fdfc800c640 comp rx=0 tx=0).stop 2023-05-22T15:42:53.828 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.823+0000 7fdfd47fc640 1 -- 172.21.15.16:0/2244676183 shutdown_connections 2023-05-22T15:42:53.828 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.823+0000 7fdfd47fc640 1 --2- 172.21.15.16:0/2244676183 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fdfb00654a0 0x7fdfb0067950 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:53.828 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.823+0000 7fdfd47fc640 1 --2- 172.21.15.16:0/2244676183 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fdfcc159a50 0x7fdfcc0c67f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:53.828 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.823+0000 7fdfd47fc640 1 --2- 172.21.15.16:0/2244676183 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdfcc1567c0 0x7fdfcc0b3660 unknown :-1 s=CLOSED pgs=1262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:53.829 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.823+0000 7fdfd47fc640 1 --2- 172.21.15.16:0/2244676183 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdfcc0cb6a0 0x7fdfcc14c0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:53.829 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.823+0000 7fdfd47fc640 1 -- 172.21.15.16:0/2244676183 >> 172.21.15.16:0/2244676183 conn(0x7fdfcc069720 msgr2=0x7fdfcc156fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:53.829 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.823+0000 7fdfd47fc640 1 -- 172.21.15.16:0/2244676183 shutdown_connections 2023-05-22T15:42:53.829 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.823+0000 7fdfd47fc640 1 -- 172.21.15.16:0/2244676183 wait complete. 2023-05-22T15:42:53.845 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-05-22T15:26:38.019781+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-05-22T15:26:43.570514+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-05-22T15:42:53.846 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-05-22T15:42:53.970 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.963+0000 7f41bbcc0640 1 Processor -- start 2023-05-22T15:42:53.971 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.963+0000 7f41bbcc0640 1 -- start start 2023-05-22T15:42:53.972 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.967+0000 7f41bbcc0640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f41b4159d40 0x7f41b40cb260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:53.973 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.967+0000 7f41bbcc0640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f41b406b6c0 0x7f41b406bb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:53.973 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.967+0000 7f41bbcc0640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41b4070c30 con 0x7f41b4159970 2023-05-22T15:42:53.973 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.967+0000 7f41bbcc0640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41b415dd90 con 0x7f41b406b6c0 2023-05-22T15:42:53.973 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.967+0000 7f41bbcc0640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41b415a190 con 0x7f41b4159d40 2023-05-22T15:42:53.974 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.967+0000 7f41b925b640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f41b4159d40 0x7f41b40cb260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:53.974 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.967+0000 7f41ba25d640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f41b406b6c0 0x7f41b406bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:53.974 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.967+0000 7f41b925b640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f41b4159d40 0x7f41b40cb260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3301/0 says I am v2:172.21.15.16:40084/0 (socket says 172.21.15.16:40084) 2023-05-22T15:42:53.974 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.967+0000 7f41b925b640 1 -- 172.21.15.16:0/3975103660 learned_addr learned my addr 172.21.15.16:0/3975103660 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:42:53.975 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.967+0000 7f41b925b640 1 -- 172.21.15.16:0/3975103660 >> v2:172.21.15.62:3300/0 conn(0x7f41b406b6c0 msgr2=0x7f41b406bb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:42:53.975 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.967+0000 7f41b925b640 1 --2- 172.21.15.16:0/3975103660 >> v2:172.21.15.62:3300/0 conn(0x7f41b406b6c0 0x7f41b406bb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:53.975 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.967+0000 7f41b925b640 1 -- 172.21.15.16:0/3975103660 >> v1:172.21.15.62:6789/0 conn(0x7f41b4159970 legacy=0x7f41b415d680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:53.976 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.967+0000 7f41b925b640 1 -- 172.21.15.16:0/3975103660 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41b406c060 con 0x7f41b4159d40 2023-05-22T15:42:53.976 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.967+0000 7f41ba25d640 1 --2- 172.21.15.16:0/3975103660 >> v2:172.21.15.62:3300/0 conn(0x7f41b406b6c0 0x7f41b406bb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-05-22T15:42:53.980 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.971+0000 7f41b925b640 1 --2- 172.21.15.16:0/3975103660 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f41b4159d40 0x7f41b40cb260 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f41a4009870 tx=0x7f41a401c480 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e5ee57c6edd3f283 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:53.980 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.971+0000 7f41b8a5a640 1 -- 172.21.15.16:0/3975103660 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41a40ab680 con 0x7f41b4159d40 2023-05-22T15:42:53.980 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.971+0000 7f41b8a5a640 1 -- 172.21.15.16:0/3975103660 <== mon.2 v2:172.21.15.62:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41a40ab7e0 con 0x7f41b4159d40 2023-05-22T15:42:53.981 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.971+0000 7f41b8a5a640 1 -- 172.21.15.16:0/3975103660 <== mon.2 v2:172.21.15.62:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41a40abb70 con 0x7f41b4159d40 2023-05-22T15:42:53.981 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.971+0000 7f41bbcc0640 1 -- 172.21.15.16:0/3975103660 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f41b4159d40 msgr2=0x7f41b40cb260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:53.981 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.971+0000 7f41bbcc0640 1 --2- 172.21.15.16:0/3975103660 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f41b4159d40 0x7f41b40cb260 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f41a4009870 tx=0x7f41a401c480 comp rx=0 tx=0).stop 2023-05-22T15:42:53.982 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.971+0000 7f41bbcc0640 1 -- 172.21.15.16:0/3975103660 shutdown_connections 2023-05-22T15:42:53.982 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.971+0000 7f41bbcc0640 1 --2- 172.21.15.16:0/3975103660 >> v2:172.21.15.62:3300/0 conn(0x7f41b406b6c0 0x7f41b406bb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:53.982 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.971+0000 7f41bbcc0640 1 --2- 172.21.15.16:0/3975103660 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f41b4159d40 0x7f41b40cb260 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:53.982 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.971+0000 7f41bbcc0640 1 -- 172.21.15.16:0/3975103660 >> 172.21.15.16:0/3975103660 conn(0x7f41b4069720 msgr2=0x7f41b4069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:53.983 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.971+0000 7f41bbcc0640 1 -- 172.21.15.16:0/3975103660 shutdown_connections 2023-05-22T15:42:53.983 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.971+0000 7f41bbcc0640 1 -- 172.21.15.16:0/3975103660 wait complete. 2023-05-22T15:42:53.983 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.971+0000 7f41bbcc0640 1 Processor -- start 2023-05-22T15:42:53.983 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.971+0000 7f41bbcc0640 1 -- start start 2023-05-22T15:42:53.984 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.971+0000 7f41bbcc0640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f41b406b6c0 0x7f41b41578c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:53.984 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.971+0000 7f41bbcc0640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f41b4159970 0x7f41b414f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:53.984 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.971+0000 7f41bbcc0640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f41b4159d40 0x7f41b40c9120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:53.985 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.971+0000 7f41bbcc0640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41b40b1c50 con 0x7f41b4159970 2023-05-22T15:42:53.985 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.971+0000 7f41bbcc0640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41b414f630 con 0x7f41b4159d40 2023-05-22T15:42:53.985 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.971+0000 7f41bbcc0640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41b4157f40 con 0x7f41b406b6c0 2023-05-22T15:42:53.985 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.971+0000 7f41b925b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f41b4159970 0x7f41b414f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:53.986 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.971+0000 7f41b925b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f41b4159970 0x7f41b414f0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60594/0 (socket says 172.21.15.16:60594) 2023-05-22T15:42:53.986 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.971+0000 7f41b925b640 1 -- 172.21.15.16:0/3234410710 learned_addr learned my addr 172.21.15.16:0/3234410710 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:42:53.986 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.975+0000 7f41ba25d640 1 --2- 172.21.15.16:0/3234410710 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f41b4159d40 0x7f41b40c9120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:53.986 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.975+0000 7f41b9a5c640 1 --2- 172.21.15.16:0/3234410710 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f41b406b6c0 0x7f41b41578c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:53.987 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.975+0000 7f41b925b640 1 -- 172.21.15.16:0/3234410710 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f41b406b6c0 msgr2=0x7f41b41578c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:53.987 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.975+0000 7f41b925b640 1 --2- 172.21.15.16:0/3234410710 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f41b406b6c0 0x7f41b41578c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:53.987 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.975+0000 7f41b925b640 1 -- 172.21.15.16:0/3234410710 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f41b4159d40 msgr2=0x7f41b40c9120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:53.987 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.975+0000 7f41b925b640 1 --2- 172.21.15.16:0/3234410710 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f41b4159d40 0x7f41b40c9120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:53.988 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.975+0000 7f41b925b640 1 -- 172.21.15.16:0/3234410710 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41ac009a70 con 0x7f41b4159970 2023-05-22T15:42:53.988 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.975+0000 7f41b925b640 1 --2- 172.21.15.16:0/3234410710 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f41b4159970 0x7f41b414f0f0 secure :-1 s=READY pgs=1263 cs=0 l=1 rev1=1 crypto rx=0x7f41a401ab30 tx=0x7f41a4019220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:53.988 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.975+0000 7f41aa7fc640 1 -- 172.21.15.16:0/3234410710 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41a40b5810 con 0x7f41b4159970 2023-05-22T15:42:53.989 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.975+0000 7f41aa7fc640 1 -- 172.21.15.16:0/3234410710 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41a40b59b0 con 0x7f41b4159970 2023-05-22T15:42:53.989 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.975+0000 7f41bbcc0640 1 -- 172.21.15.16:0/3234410710 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41a40094d0 con 0x7f41b4159970 2023-05-22T15:42:53.989 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.975+0000 7f41aa7fc640 1 -- 172.21.15.16:0/3234410710 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41a40b5d60 con 0x7f41b4159970 2023-05-22T15:42:53.990 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.975+0000 7f41bbcc0640 1 -- 172.21.15.16:0/3234410710 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41b4071700 con 0x7f41b4159970 2023-05-22T15:42:53.990 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.975+0000 7f41bbcc0640 1 -- 172.21.15.16:0/3234410710 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41b4071420 con 0x7f41b4159970 2023-05-22T15:42:53.992 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.975+0000 7f41aa7fc640 1 -- 172.21.15.16:0/3234410710 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 34) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f41a40bf410 con 0x7f41b4159970 2023-05-22T15:42:53.992 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.975+0000 7f41aa7fc640 1 --2- 172.21.15.16:0/3234410710 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f41980654a0 0x7f4198067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:53.992 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.975+0000 7f41aa7fc640 1 -- 172.21.15.16:0/3234410710 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5530+0+0 (secure 0 0 0) 0x7f41a411d9e0 con 0x7f41b4159970 2023-05-22T15:42:53.993 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.975+0000 7f41b9a5c640 1 --2- 172.21.15.16:0/3234410710 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f41980654a0 0x7f4198067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:53.993 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.975+0000 7f41b9a5c640 1 --2- 172.21.15.16:0/3234410710 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f41980654a0 0x7f4198067950 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f41b0000f70 tx=0x7f41b0009040 comp rx=0 tx=0).ready entity=mgr.6436 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:53.993 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:53.979+0000 7f41aa7fc640 1 -- 172.21.15.16:0/3234410710 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f41a40eb5f0 con 0x7f41b4159970 2023-05-22T15:42:54.141 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.135+0000 7f41bbcc0640 1 -- 172.21.15.16:0/3234410710 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f41b40cb7a0 con 0x7f41b4159970 2023-05-22T15:42:54.141 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.135+0000 7f41aa7fc640 1 -- 172.21.15.16:0/3234410710 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v100) v1 ==== 93+0+10 (secure 0 0 0) 0x7f41a40eec10 con 0x7f41b4159970 2023-05-22T15:42:54.141 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2023-05-22T15:42:54.143 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.139+0000 7f41bbcc0640 1 -- 172.21.15.16:0/3234410710 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f41980654a0 msgr2=0x7f4198067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:54.143 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.139+0000 7f41bbcc0640 1 --2- 172.21.15.16:0/3234410710 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f41980654a0 0x7f4198067950 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f41b0000f70 tx=0x7f41b0009040 comp rx=0 tx=0).stop 2023-05-22T15:42:54.143 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.139+0000 7f41bbcc0640 1 -- 172.21.15.16:0/3234410710 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f41b4159970 msgr2=0x7f41b414f0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:54.144 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.139+0000 7f41bbcc0640 1 --2- 172.21.15.16:0/3234410710 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f41b4159970 0x7f41b414f0f0 secure :-1 s=READY pgs=1263 cs=0 l=1 rev1=1 crypto rx=0x7f41a401ab30 tx=0x7f41a4019220 comp rx=0 tx=0).stop 2023-05-22T15:42:54.144 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.139+0000 7f41bbcc0640 1 -- 172.21.15.16:0/3234410710 shutdown_connections 2023-05-22T15:42:54.144 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.139+0000 7f41bbcc0640 1 --2- 172.21.15.16:0/3234410710 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f41b4159d40 0x7f41b40c9120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:54.144 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.139+0000 7f41bbcc0640 1 --2- 172.21.15.16:0/3234410710 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f41b4159970 0x7f41b414f0f0 unknown :-1 s=CLOSED pgs=1263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:54.144 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.139+0000 7f41bbcc0640 1 --2- 172.21.15.16:0/3234410710 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f41980654a0 0x7f4198067950 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:54.144 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.139+0000 7f41bbcc0640 1 --2- 172.21.15.16:0/3234410710 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f41b406b6c0 0x7f41b41578c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:54.144 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.139+0000 7f41bbcc0640 1 -- 172.21.15.16:0/3234410710 >> 172.21.15.16:0/3234410710 conn(0x7f41b4069720 msgr2=0x7f41b415adc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:54.145 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.139+0000 7f41bbcc0640 1 -- 172.21.15.16:0/3234410710 shutdown_connections 2023-05-22T15:42:54.145 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.139+0000 7f41bbcc0640 1 -- 172.21.15.16:0/3234410710 wait complete. 2023-05-22T15:42:54.160 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-05-22T15:42:54.289 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.283+0000 7f4eb1b22640 1 Processor -- start 2023-05-22T15:42:54.290 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.283+0000 7f4eb1b22640 1 -- start start 2023-05-22T15:42:54.291 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.287+0000 7f4eb1b22640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f4eac15bcd0 0x7f4eac15c120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:54.291 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.287+0000 7f4eb1b22640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4eac0c8660 0x7f4eac0c8ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:54.291 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.287+0000 7f4eb1b22640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4eac158cd0 con 0x7f4eac158210 2023-05-22T15:42:54.291 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.287+0000 7f4eb1b22640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4eac0c9020 con 0x7f4eac15bcd0 2023-05-22T15:42:54.291 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.287+0000 7f4eb1b22640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4eac15c790 con 0x7f4eac0c8660 2023-05-22T15:42:54.291 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.287+0000 7f4eaaffd640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f4eac15bcd0 0x7f4eac15c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:54.292 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.287+0000 7f4eab7fe640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f4eac158210 0x7f4eac159160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:44852/0 (socket says 172.21.15.16:44852) 2023-05-22T15:42:54.293 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.287+0000 7f4eab7fe640 1 -- 172.21.15.16:0/3512274017 learned_addr learned my addr 172.21.15.16:0/3512274017 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:42:54.293 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.287+0000 7f4eabfff640 1 --2- 172.21.15.16:0/3512274017 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4eac0c8660 0x7f4eac0c8ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:54.294 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.287+0000 7f4eaaffd640 1 -- 172.21.15.16:0/3512274017 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4eac0c8660 msgr2=0x7f4eac0c8ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:42:54.294 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.287+0000 7f4eaaffd640 1 --2- 172.21.15.16:0/3512274017 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4eac0c8660 0x7f4eac0c8ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:54.295 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.287+0000 7f4eaaffd640 1 -- 172.21.15.16:0/3512274017 >> v1:172.21.15.16:6789/0 conn(0x7f4eac158210 legacy=0x7f4eac159160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:54.295 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.287+0000 7f4eaaffd640 1 -- 172.21.15.16:0/3512274017 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4eac15ca70 con 0x7f4eac15bcd0 2023-05-22T15:42:54.295 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.287+0000 7f4eaaffd640 1 --2- 172.21.15.16:0/3512274017 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4eac15bcd0 0x7f4eac15c120 secure :-1 s=READY pgs=1265 cs=0 l=1 rev1=1 crypto rx=0x7f4ea000ba50 tx=0x7f4ea001e870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd748ad226bd4bff server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:54.295 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.287+0000 7f4eaa7fc640 1 -- 172.21.15.16:0/3512274017 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ea00a78e0 con 0x7f4eac15bcd0 2023-05-22T15:42:54.295 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.287+0000 7f4eaa7fc640 1 -- 172.21.15.16:0/3512274017 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ea00ab070 con 0x7f4eac15bcd0 2023-05-22T15:42:54.295 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.287+0000 7f4eaa7fc640 1 -- 172.21.15.16:0/3512274017 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ea00a7c60 con 0x7f4eac15bcd0 2023-05-22T15:42:54.295 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.291+0000 7f4eb1b22640 1 -- 172.21.15.16:0/3512274017 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4eac15bcd0 msgr2=0x7f4eac15c120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:54.296 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.291+0000 7f4eb1b22640 1 --2- 172.21.15.16:0/3512274017 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4eac15bcd0 0x7f4eac15c120 secure :-1 s=READY pgs=1265 cs=0 l=1 rev1=1 crypto rx=0x7f4ea000ba50 tx=0x7f4ea001e870 comp rx=0 tx=0).stop 2023-05-22T15:42:54.296 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.291+0000 7f4eb1b22640 1 -- 172.21.15.16:0/3512274017 shutdown_connections 2023-05-22T15:42:54.296 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.291+0000 7f4eb1b22640 1 --2- 172.21.15.16:0/3512274017 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4eac0c8660 0x7f4eac0c8ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:54.296 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.291+0000 7f4eb1b22640 1 --2- 172.21.15.16:0/3512274017 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4eac15bcd0 0x7f4eac15c120 unknown :-1 s=CLOSED pgs=1265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:54.296 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.291+0000 7f4eb1b22640 1 -- 172.21.15.16:0/3512274017 >> 172.21.15.16:0/3512274017 conn(0x7f4eac069720 msgr2=0x7f4eac069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:54.296 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.291+0000 7f4eb1b22640 1 -- 172.21.15.16:0/3512274017 shutdown_connections 2023-05-22T15:42:54.296 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.291+0000 7f4eb1b22640 1 -- 172.21.15.16:0/3512274017 wait complete. 2023-05-22T15:42:54.298 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.291+0000 7f4eb1b22640 1 Processor -- start 2023-05-22T15:42:54.298 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.291+0000 7f4eb1b22640 1 -- start start 2023-05-22T15:42:54.298 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.291+0000 7f4eb1b22640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4eac0c8660 0x7f4eac154720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:54.299 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.291+0000 7f4eb1b22640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4eac158210 0x7f4eac14e410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:54.299 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.291+0000 7f4eb1b22640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4eac15bcd0 0x7f4eac0c7810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:54.299 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.291+0000 7f4eb1b22640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4eac0b1d20 con 0x7f4eac158210 2023-05-22T15:42:54.299 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.291+0000 7f4eb1b22640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4eac157650 con 0x7f4eac15bcd0 2023-05-22T15:42:54.300 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.291+0000 7f4eb1b22640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4eac0b1f70 con 0x7f4eac0c8660 2023-05-22T15:42:54.300 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.291+0000 7f4eaaffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4eac158210 0x7f4eac14e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:54.300 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.291+0000 7f4eaaffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4eac158210 0x7f4eac14e410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60624/0 (socket says 172.21.15.16:60624) 2023-05-22T15:42:54.300 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.291+0000 7f4eaaffd640 1 -- 172.21.15.16:0/3691302507 learned_addr learned my addr 172.21.15.16:0/3691302507 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:42:54.300 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.291+0000 7f4eabfff640 1 --2- 172.21.15.16:0/3691302507 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4eac15bcd0 0x7f4eac0c7810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:54.300 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.291+0000 7f4eaaffd640 1 -- 172.21.15.16:0/3691302507 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4eac0c8660 msgr2=0x7f4eac154720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:54.300 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.291+0000 7f4eaaffd640 1 --2- 172.21.15.16:0/3691302507 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4eac0c8660 0x7f4eac154720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:54.301 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.291+0000 7f4eaaffd640 1 -- 172.21.15.16:0/3691302507 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4eac15bcd0 msgr2=0x7f4eac0c7810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:54.301 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.291+0000 7f4eaaffd640 1 --2- 172.21.15.16:0/3691302507 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4eac15bcd0 0x7f4eac0c7810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:54.301 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.291+0000 7f4eaaffd640 1 -- 172.21.15.16:0/3691302507 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ea000b6b0 con 0x7f4eac158210 2023-05-22T15:42:54.301 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.291+0000 7f4eaaffd640 1 --2- 172.21.15.16:0/3691302507 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4eac158210 0x7f4eac14e410 secure :-1 s=READY pgs=1266 cs=0 l=1 rev1=1 crypto rx=0x7f4ea001c4a0 tx=0x7f4ea001b610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:54.301 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.291+0000 7f4e93fff640 1 -- 172.21.15.16:0/3691302507 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ea00b1810 con 0x7f4eac158210 2023-05-22T15:42:54.301 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.291+0000 7f4e93fff640 1 -- 172.21.15.16:0/3691302507 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ea00b1990 con 0x7f4eac158210 2023-05-22T15:42:54.301 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.291+0000 7f4e93fff640 1 -- 172.21.15.16:0/3691302507 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ea00b1d40 con 0x7f4eac158210 2023-05-22T15:42:54.301 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.291+0000 7f4eb1b22640 1 -- 172.21.15.16:0/3691302507 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4eac0ca690 con 0x7f4eac158210 2023-05-22T15:42:54.302 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.291+0000 7f4eb1b22640 1 -- 172.21.15.16:0/3691302507 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4eac1585e0 con 0x7f4eac158210 2023-05-22T15:42:54.302 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.295+0000 7f4eb1b22640 1 -- 172.21.15.16:0/3691302507 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4eac0ca820 con 0x7f4eac158210 2023-05-22T15:42:54.303 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.295+0000 7f4e93fff640 1 -- 172.21.15.16:0/3691302507 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 34) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f4ea00ba470 con 0x7f4eac158210 2023-05-22T15:42:54.303 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.295+0000 7f4e93fff640 1 --2- 172.21.15.16:0/3691302507 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f4e980654a0 0x7f4e98067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:54.303 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.295+0000 7f4e93fff640 1 -- 172.21.15.16:0/3691302507 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5530+0+0 (secure 0 0 0) 0x7f4ea011f8d0 con 0x7f4eac158210 2023-05-22T15:42:54.303 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.295+0000 7f4eab7fe640 1 --2- 172.21.15.16:0/3691302507 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f4e980654a0 0x7f4e98067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:54.303 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.295+0000 7f4eab7fe640 1 --2- 172.21.15.16:0/3691302507 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f4e980654a0 0x7f4e98067950 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f4e9c001600 tx=0x7f4e9c007a50 comp rx=0 tx=0).ready entity=mgr.6436 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:54.304 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.299+0000 7f4e93fff640 1 -- 172.21.15.16:0/3691302507 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f4ea00ed5e0 con 0x7f4eac158210 2023-05-22T15:42:54.458 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.451+0000 7f4eb1b22640 1 -- 172.21.15.16:0/3691302507 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f4eac0b20b0 con 0x7f4eac158210 2023-05-22T15:42:54.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.455+0000 7f4e93fff640 1 -- 172.21.15.16:0/3691302507 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v100) v1 ==== 92+0+10 (secure 0 0 0) 0x7f4eac0b20b0 con 0x7f4eac158210 2023-05-22T15:42:54.459 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 8 2023-05-22T15:42:54.460 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.455+0000 7f4eb1b22640 1 -- 172.21.15.16:0/3691302507 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f4e980654a0 msgr2=0x7f4e98067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:54.461 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.455+0000 7f4eb1b22640 1 --2- 172.21.15.16:0/3691302507 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f4e980654a0 0x7f4e98067950 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f4e9c001600 tx=0x7f4e9c007a50 comp rx=0 tx=0).stop 2023-05-22T15:42:54.461 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.455+0000 7f4eb1b22640 1 -- 172.21.15.16:0/3691302507 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4eac158210 msgr2=0x7f4eac14e410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:54.461 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.455+0000 7f4eb1b22640 1 --2- 172.21.15.16:0/3691302507 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4eac158210 0x7f4eac14e410 secure :-1 s=READY pgs=1266 cs=0 l=1 rev1=1 crypto rx=0x7f4ea001c4a0 tx=0x7f4ea001b610 comp rx=0 tx=0).stop 2023-05-22T15:42:54.462 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.455+0000 7f4eb1b22640 1 -- 172.21.15.16:0/3691302507 shutdown_connections 2023-05-22T15:42:54.462 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.455+0000 7f4eb1b22640 1 --2- 172.21.15.16:0/3691302507 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4eac15bcd0 0x7f4eac0c7810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:54.462 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.455+0000 7f4eb1b22640 1 --2- 172.21.15.16:0/3691302507 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4eac158210 0x7f4eac14e410 unknown :-1 s=CLOSED pgs=1266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:54.462 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.455+0000 7f4eb1b22640 1 --2- 172.21.15.16:0/3691302507 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f4e980654a0 0x7f4e98067950 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:54.462 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.455+0000 7f4eb1b22640 1 --2- 172.21.15.16:0/3691302507 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4eac0c8660 0x7f4eac154720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:54.462 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.455+0000 7f4eb1b22640 1 -- 172.21.15.16:0/3691302507 >> 172.21.15.16:0/3691302507 conn(0x7f4eac069720 msgr2=0x7f4eac15b170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:54.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.455+0000 7f4eb1b22640 1 -- 172.21.15.16:0/3691302507 shutdown_connections 2023-05-22T15:42:54.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.455+0000 7f4eb1b22640 1 -- 172.21.15.16:0/3691302507 wait complete. 2023-05-22T15:42:54.478 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-05-22T15:42:54.606 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.599+0000 7f243b594640 1 Processor -- start 2023-05-22T15:42:54.606 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.599+0000 7f243b594640 1 -- start start 2023-05-22T15:42:54.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.603+0000 7f243b594640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f24340c9b40 0x7f243415e460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:54.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.603+0000 7f243b594640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2434070df0 0x7f2434071210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:54.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.603+0000 7f243b594640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24340b0650 con 0x7f24340ca4e0 2023-05-22T15:42:54.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.603+0000 7f243b594640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24340b0370 con 0x7f2434070df0 2023-05-22T15:42:54.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.603+0000 7f243b594640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f243415d980 con 0x7f24340c9b40 2023-05-22T15:42:54.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.603+0000 7f2439b31640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2434070df0 0x7f2434071210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:54.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.603+0000 7f2439b31640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2434070df0 0x7f2434071210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60640/0 (socket says 172.21.15.16:60640) 2023-05-22T15:42:54.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.603+0000 7f2439b31640 1 -- 172.21.15.16:0/4281716631 learned_addr learned my addr 172.21.15.16:0/4281716631 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:42:54.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.603+0000 7f2438b2f640 1 --2- 172.21.15.16:0/4281716631 >> v2:172.21.15.62:3300/0 conn(0x7f24340c9b40 0x7f243415e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:54.609 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.603+0000 7f2439b31640 1 -- 172.21.15.16:0/4281716631 >> v2:172.21.15.62:3300/0 conn(0x7f24340c9b40 msgr2=0x7f243415e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:42:54.609 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.603+0000 7f2439b31640 1 --2- 172.21.15.16:0/4281716631 >> v2:172.21.15.62:3300/0 conn(0x7f24340c9b40 0x7f243415e460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:54.609 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.603+0000 7f2439b31640 1 -- 172.21.15.16:0/4281716631 >> v1:172.21.15.62:6789/0 conn(0x7f24340ca4e0 legacy=0x7f24340afb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:42:54.609 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.603+0000 7f2439b31640 1 -- 172.21.15.16:0/4281716631 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f243415dc60 con 0x7f2434070df0 2023-05-22T15:42:54.610 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.603+0000 7f2439b31640 1 --2- 172.21.15.16:0/4281716631 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2434070df0 0x7f2434071210 secure :-1 s=READY pgs=1267 cs=0 l=1 rev1=1 crypto rx=0x7f243406f0e0 tx=0x7f242401c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ff28bfcd9c15dbe server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:54.610 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.603+0000 7f2433fff640 1 -- 172.21.15.16:0/4281716631 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24240a5780 con 0x7f2434070df0 2023-05-22T15:42:54.610 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.603+0000 7f2433fff640 1 -- 172.21.15.16:0/4281716631 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24240a58e0 con 0x7f2434070df0 2023-05-22T15:42:54.610 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.603+0000 7f2433fff640 1 -- 172.21.15.16:0/4281716631 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24240a5c70 con 0x7f2434070df0 2023-05-22T15:42:54.611 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.607+0000 7f243b594640 1 -- 172.21.15.16:0/4281716631 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2434070df0 msgr2=0x7f2434071210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:54.611 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.607+0000 7f243b594640 1 --2- 172.21.15.16:0/4281716631 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2434070df0 0x7f2434071210 secure :-1 s=READY pgs=1267 cs=0 l=1 rev1=1 crypto rx=0x7f243406f0e0 tx=0x7f242401c960 comp rx=0 tx=0).stop 2023-05-22T15:42:54.611 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.607+0000 7f243b594640 1 -- 172.21.15.16:0/4281716631 shutdown_connections 2023-05-22T15:42:54.611 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.607+0000 7f243b594640 1 --2- 172.21.15.16:0/4281716631 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2434070df0 0x7f2434071210 unknown :-1 s=CLOSED pgs=1267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:54.612 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.607+0000 7f243b594640 1 --2- 172.21.15.16:0/4281716631 >> v2:172.21.15.62:3300/0 conn(0x7f24340c9b40 0x7f243415e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:54.612 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.607+0000 7f243b594640 1 -- 172.21.15.16:0/4281716631 >> 172.21.15.16:0/4281716631 conn(0x7f2434069720 msgr2=0x7f2434069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:54.612 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.607+0000 7f243b594640 1 -- 172.21.15.16:0/4281716631 shutdown_connections 2023-05-22T15:42:54.612 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.607+0000 7f243b594640 1 -- 172.21.15.16:0/4281716631 wait complete. 2023-05-22T15:42:54.612 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.607+0000 7f243b594640 1 Processor -- start 2023-05-22T15:42:54.613 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.607+0000 7f243b594640 1 -- start start 2023-05-22T15:42:54.614 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.607+0000 7f243b594640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2434070df0 0x7f24340b26e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:54.614 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.607+0000 7f243b594640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f24340c9b40 0x7f2434154bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:54.614 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.607+0000 7f243b594640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f24340ca4e0 0x7f24340c3b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:54.614 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.607+0000 7f243b594640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24340c2d90 con 0x7f2434070df0 2023-05-22T15:42:54.615 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.607+0000 7f243b594640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24340c3070 con 0x7f24340c9b40 2023-05-22T15:42:54.615 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.607+0000 7f243b594640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24340c3350 con 0x7f24340ca4e0 2023-05-22T15:42:54.615 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.607+0000 7f2439330640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2434070df0 0x7f24340b26e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:54.615 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.607+0000 7f2439330640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2434070df0 0x7f24340b26e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60648/0 (socket says 172.21.15.16:60648) 2023-05-22T15:42:54.615 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.607+0000 7f2439330640 1 -- 172.21.15.16:0/604105716 learned_addr learned my addr 172.21.15.16:0/604105716 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:42:54.615 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.607+0000 7f2439b31640 1 --2- 172.21.15.16:0/604105716 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f24340ca4e0 0x7f24340c3b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:54.615 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.607+0000 7f2438b2f640 1 --2- 172.21.15.16:0/604105716 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f24340c9b40 0x7f2434154bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:54.615 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.607+0000 7f2439330640 1 -- 172.21.15.16:0/604105716 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f24340ca4e0 msgr2=0x7f24340c3b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:54.616 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.607+0000 7f2439330640 1 --2- 172.21.15.16:0/604105716 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f24340ca4e0 0x7f24340c3b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:54.616 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.607+0000 7f2439330640 1 -- 172.21.15.16:0/604105716 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f24340c9b40 msgr2=0x7f2434154bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:54.616 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.607+0000 7f2439330640 1 --2- 172.21.15.16:0/604105716 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f24340c9b40 0x7f2434154bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:54.616 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.607+0000 7f2439330640 1 -- 172.21.15.16:0/604105716 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2424009b30 con 0x7f2434070df0 2023-05-22T15:42:54.616 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.607+0000 7f2438b2f640 1 --2- 172.21.15.16:0/604105716 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f24340c9b40 0x7f2434154bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:42:54.616 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.607+0000 7f2439b31640 1 --2- 172.21.15.16:0/604105716 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f24340ca4e0 0x7f24340c3b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:42:54.616 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.607+0000 7f2439330640 1 --2- 172.21.15.16:0/604105716 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2434070df0 0x7f24340b26e0 secure :-1 s=READY pgs=1268 cs=0 l=1 rev1=1 crypto rx=0x7f242c00b410 tx=0x7f242c00b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:54.617 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.611+0000 7f2431ffb640 1 -- 172.21.15.16:0/604105716 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f242c004020 con 0x7f2434070df0 2023-05-22T15:42:54.617 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.611+0000 7f2431ffb640 1 -- 172.21.15.16:0/604105716 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f242c004180 con 0x7f2434070df0 2023-05-22T15:42:54.617 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.611+0000 7f243b594640 1 -- 172.21.15.16:0/604105716 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24340c8e80 con 0x7f2434070df0 2023-05-22T15:42:54.617 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.611+0000 7f2431ffb640 1 -- 172.21.15.16:0/604105716 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f242c011630 con 0x7f2434070df0 2023-05-22T15:42:54.617 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.611+0000 7f243b594640 1 -- 172.21.15.16:0/604105716 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24340ca8b0 con 0x7f2434070df0 2023-05-22T15:42:54.618 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.611+0000 7f243b594640 1 -- 172.21.15.16:0/604105716 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24340c9010 con 0x7f2434070df0 2023-05-22T15:42:54.618 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.611+0000 7f2431ffb640 1 -- 172.21.15.16:0/604105716 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 34) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f242c011790 con 0x7f2434070df0 2023-05-22T15:42:54.618 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.611+0000 7f2431ffb640 1 --2- 172.21.15.16:0/604105716 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f24200654a0 0x7f2420067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:54.618 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.611+0000 7f2431ffb640 1 -- 172.21.15.16:0/604105716 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5530+0+0 (secure 0 0 0) 0x7f242c082590 con 0x7f2434070df0 2023-05-22T15:42:54.618 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.611+0000 7f2438b2f640 1 --2- 172.21.15.16:0/604105716 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f24200654a0 0x7f2420067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:54.619 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.611+0000 7f2438b2f640 1 --2- 172.21.15.16:0/604105716 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f24200654a0 0x7f2420067950 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f24340b1ab0 tx=0x7f2428008040 comp rx=0 tx=0).ready entity=mgr.6436 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:54.620 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.615+0000 7f2431ffb640 1 -- 172.21.15.16:0/604105716 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f242c0502b0 con 0x7f2434070df0 2023-05-22T15:42:54.778 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.775+0000 7f243b594640 1 -- 172.21.15.16:0/604105716 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f24340c2ed0 con 0x7f2434070df0 2023-05-22T15:42:54.780 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.775+0000 7f2431ffb640 1 -- 172.21.15.16:0/604105716 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v100) v1 ==== 84+0+126 (secure 0 0 0) 0x7f24340c2ed0 con 0x7f2434070df0 2023-05-22T15:42:54.780 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:42:54.781 INFO:teuthology.orchestra.run.smithi016.stdout:[ 2023-05-22T15:42:54.781 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T15:42:54.781 INFO:teuthology.orchestra.run.smithi016.stdout: "poolnum": 1, 2023-05-22T15:42:54.781 INFO:teuthology.orchestra.run.smithi016.stdout: "poolname": ".mgr" 2023-05-22T15:42:54.782 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:42:54.782 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T15:42:54.782 INFO:teuthology.orchestra.run.smithi016.stdout: "poolnum": 2, 2023-05-22T15:42:54.783 INFO:teuthology.orchestra.run.smithi016.stdout: "poolname": "rbd" 2023-05-22T15:42:54.783 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T15:42:54.783 INFO:teuthology.orchestra.run.smithi016.stdout:] 2023-05-22T15:42:54.785 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.775+0000 7f243b594640 1 -- 172.21.15.16:0/604105716 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f24200654a0 msgr2=0x7f2420067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:54.785 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.775+0000 7f243b594640 1 --2- 172.21.15.16:0/604105716 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f24200654a0 0x7f2420067950 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f24340b1ab0 tx=0x7f2428008040 comp rx=0 tx=0).stop 2023-05-22T15:42:54.785 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.775+0000 7f243b594640 1 -- 172.21.15.16:0/604105716 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2434070df0 msgr2=0x7f24340b26e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:54.785 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.775+0000 7f243b594640 1 --2- 172.21.15.16:0/604105716 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2434070df0 0x7f24340b26e0 secure :-1 s=READY pgs=1268 cs=0 l=1 rev1=1 crypto rx=0x7f242c00b410 tx=0x7f242c00b8d0 comp rx=0 tx=0).stop 2023-05-22T15:42:54.785 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.775+0000 7f243b594640 1 -- 172.21.15.16:0/604105716 shutdown_connections 2023-05-22T15:42:54.785 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.775+0000 7f243b594640 1 --2- 172.21.15.16:0/604105716 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f24200654a0 0x7f2420067950 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:54.785 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.775+0000 7f243b594640 1 --2- 172.21.15.16:0/604105716 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f24340ca4e0 0x7f24340c3b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:54.786 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.775+0000 7f243b594640 1 --2- 172.21.15.16:0/604105716 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f24340c9b40 0x7f2434154bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:54.786 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.775+0000 7f243b594640 1 --2- 172.21.15.16:0/604105716 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2434070df0 0x7f24340b26e0 unknown :-1 s=CLOSED pgs=1268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:54.786 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.775+0000 7f243b594640 1 -- 172.21.15.16:0/604105716 >> 172.21.15.16:0/604105716 conn(0x7f2434069720 msgr2=0x7f24341567f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:54.786 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.775+0000 7f243b594640 1 -- 172.21.15.16:0/604105716 shutdown_connections 2023-05-22T15:42:54.786 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.775+0000 7f243b594640 1 -- 172.21.15.16:0/604105716 wait complete. 2023-05-22T15:42:54.799 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-05-22T15:42:54.800 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-05-22T15:42:54.921 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.915+0000 7ff6a0893640 1 Processor -- start 2023-05-22T15:42:54.922 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.915+0000 7ff6a0893640 1 -- start start 2023-05-22T15:42:54.923 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.919+0000 7ff6a0893640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff69c0c4a40 0x7ff69c0caab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:54.923 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.919+0000 7ff6a0893640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff69c0caff0 0x7ff69c15f8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:54.923 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.919+0000 7ff6a0893640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff69c0c4f90 con 0x7ff69c15bd90 2023-05-22T15:42:54.923 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.919+0000 7ff6a0893640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff69c06c700 con 0x7ff69c0caff0 2023-05-22T15:42:54.923 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.919+0000 7ff6a0893640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff69c0cb440 con 0x7ff69c0c4a40 2023-05-22T15:42:54.924 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.919+0000 7ff69a59c640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7ff69c15bd90 0x7ff69c0ce9a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:44860/0 (socket says 172.21.15.16:44860) 2023-05-22T15:42:54.924 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.919+0000 7ff69a59c640 1 -- 172.21.15.16:0/3641374907 learned_addr learned my addr 172.21.15.16:0/3641374907 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:42:54.924 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.919+0000 7ff69ad9d640 1 --2- 172.21.15.16:0/3641374907 >> v2:172.21.15.16:3300/0 conn(0x7ff69c0caff0 0x7ff69c15f8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:54.924 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.919+0000 7ff691d9b640 1 --2- 172.21.15.16:0/3641374907 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff69c0c4a40 0x7ff69c0caab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:54.925 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.919+0000 7ff699d9b640 1 -- 172.21.15.16:0/3641374907 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 548895544 0 0) 0x7ff68c001120 con 0x7ff69c15bd90 2023-05-22T15:42:54.925 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.919+0000 7ff69ad9d640 1 -- 172.21.15.16:0/3641374907 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff69c0c4a40 msgr2=0x7ff69c0caab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:42:54.925 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.919+0000 7ff69ad9d640 1 --2- 172.21.15.16:0/3641374907 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff69c0c4a40 0x7ff69c0caab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:54.925 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.919+0000 7ff69ad9d640 1 -- 172.21.15.16:0/3641374907 >> v1:172.21.15.16:6789/0 conn(0x7ff69c15bd90 legacy=0x7ff69c0ce9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:54.925 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.919+0000 7ff69ad9d640 1 -- 172.21.15.16:0/3641374907 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff69c0ad450 con 0x7ff69c0caff0 2023-05-22T15:42:54.925 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.919+0000 7ff691d9b640 1 --2- 172.21.15.16:0/3641374907 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff69c0c4a40 0x7ff69c0caab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T15:42:54.926 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.919+0000 7ff69ad9d640 1 --2- 172.21.15.16:0/3641374907 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff69c0caff0 0x7ff69c15f8d0 secure :-1 s=READY pgs=1270 cs=0 l=1 rev1=1 crypto rx=0x7ff694009d90 tx=0x7ff69401c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3bc0f5610392c98 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:54.927 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.919+0000 7ff699d9b640 1 -- 172.21.15.16:0/3641374907 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6940a5810 con 0x7ff69c0caff0 2023-05-22T15:42:54.927 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.919+0000 7ff699d9b640 1 -- 172.21.15.16:0/3641374907 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6940a5970 con 0x7ff69c0caff0 2023-05-22T15:42:54.927 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.919+0000 7ff699d9b640 1 -- 172.21.15.16:0/3641374907 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6940a5d00 con 0x7ff69c0caff0 2023-05-22T15:42:54.927 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.923+0000 7ff6a0893640 1 -- 172.21.15.16:0/3641374907 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff69c0caff0 msgr2=0x7ff69c15f8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:54.927 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.923+0000 7ff6a0893640 1 --2- 172.21.15.16:0/3641374907 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff69c0caff0 0x7ff69c15f8d0 secure :-1 s=READY pgs=1270 cs=0 l=1 rev1=1 crypto rx=0x7ff694009d90 tx=0x7ff69401c950 comp rx=0 tx=0).stop 2023-05-22T15:42:54.927 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.923+0000 7ff6a0893640 1 -- 172.21.15.16:0/3641374907 shutdown_connections 2023-05-22T15:42:54.927 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.923+0000 7ff6a0893640 1 --2- 172.21.15.16:0/3641374907 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff69c0caff0 0x7ff69c15f8d0 unknown :-1 s=CLOSED pgs=1270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:54.928 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.923+0000 7ff6a0893640 1 --2- 172.21.15.16:0/3641374907 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff69c0c4a40 0x7ff69c0caab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:54.928 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.923+0000 7ff6a0893640 1 -- 172.21.15.16:0/3641374907 >> 172.21.15.16:0/3641374907 conn(0x7ff69c069720 msgr2=0x7ff69c069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:54.928 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.923+0000 7ff6a0893640 1 -- 172.21.15.16:0/3641374907 shutdown_connections 2023-05-22T15:42:54.928 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.923+0000 7ff6a0893640 1 -- 172.21.15.16:0/3641374907 wait complete. 2023-05-22T15:42:54.930 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.923+0000 7ff6a0893640 1 Processor -- start 2023-05-22T15:42:54.930 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.923+0000 7ff6a0893640 1 -- start start 2023-05-22T15:42:54.930 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.923+0000 7ff6a0893640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff69c0c4a40 0x7ff69c15d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:54.930 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.923+0000 7ff6a0893640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff69c0caff0 0x7ff69c15ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:54.930 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.923+0000 7ff6a0893640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff69c15bd90 0x7ff69c0b34a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:54.930 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.923+0000 7ff6a0893640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff69c15da30 con 0x7ff69c15bd90 2023-05-22T15:42:54.930 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.923+0000 7ff6a0893640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff69c0b1370 con 0x7ff69c0c4a40 2023-05-22T15:42:54.931 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.923+0000 7ff6a0893640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff69c0b1650 con 0x7ff69c0caff0 2023-05-22T15:42:54.931 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.923+0000 7ff69ad9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff69c15bd90 0x7ff69c0b34a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:54.931 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.923+0000 7ff69ad9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff69c15bd90 0x7ff69c0b34a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60664/0 (socket says 172.21.15.16:60664) 2023-05-22T15:42:54.931 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.923+0000 7ff69ad9d640 1 -- 172.21.15.16:0/1748530290 learned_addr learned my addr 172.21.15.16:0/1748530290 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:42:54.931 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.923+0000 7ff691d9b640 1 --2- 172.21.15.16:0/1748530290 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff69c0caff0 0x7ff69c15ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:54.931 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.923+0000 7ff69a59c640 1 --2- 172.21.15.16:0/1748530290 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff69c0c4a40 0x7ff69c15d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:54.931 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.923+0000 7ff69ad9d640 1 -- 172.21.15.16:0/1748530290 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff69c0caff0 msgr2=0x7ff69c15ac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:54.932 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.923+0000 7ff69ad9d640 1 --2- 172.21.15.16:0/1748530290 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff69c0caff0 0x7ff69c15ac60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:54.932 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.923+0000 7ff69ad9d640 1 -- 172.21.15.16:0/1748530290 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff69c0c4a40 msgr2=0x7ff69c15d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:54.932 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.923+0000 7ff69ad9d640 1 --2- 172.21.15.16:0/1748530290 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff69c0c4a40 0x7ff69c15d4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:54.932 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.923+0000 7ff69ad9d640 1 -- 172.21.15.16:0/1748530290 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff694009a70 con 0x7ff69c15bd90 2023-05-22T15:42:54.932 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.923+0000 7ff691d9b640 1 --2- 172.21.15.16:0/1748530290 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff69c0caff0 0x7ff69c15ac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:42:54.932 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.923+0000 7ff69ad9d640 1 --2- 172.21.15.16:0/1748530290 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff69c15bd90 0x7ff69c0b34a0 secure :-1 s=READY pgs=1271 cs=0 l=1 rev1=1 crypto rx=0x7ff69401a580 tx=0x7ff6940196f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:54.933 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.923+0000 7ff6937fe640 1 -- 172.21.15.16:0/1748530290 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6940a57d0 con 0x7ff69c15bd90 2023-05-22T15:42:54.933 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.923+0000 7ff6937fe640 1 -- 172.21.15.16:0/1748530290 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6940ad650 con 0x7ff69c15bd90 2023-05-22T15:42:54.933 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.923+0000 7ff6937fe640 1 -- 172.21.15.16:0/1748530290 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6940ada00 con 0x7ff69c15bd90 2023-05-22T15:42:54.933 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.923+0000 7ff6a0893640 1 -- 172.21.15.16:0/1748530290 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff69c0ad8e0 con 0x7ff69c15bd90 2023-05-22T15:42:54.933 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.923+0000 7ff6a0893640 1 -- 172.21.15.16:0/1748530290 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff69c0ad120 con 0x7ff69c15bd90 2023-05-22T15:42:54.934 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.927+0000 7ff6a0893640 1 -- 172.21.15.16:0/1748530290 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff69c168c70 con 0x7ff69c15bd90 2023-05-22T15:42:54.934 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.927+0000 7ff6937fe640 1 -- 172.21.15.16:0/1748530290 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 34) v1 ==== 84332+0+0 (secure 0 0 0) 0x7ff6940adb60 con 0x7ff69c15bd90 2023-05-22T15:42:54.934 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.927+0000 7ff6937fe640 1 --2- 172.21.15.16:0/1748530290 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ff67c0654a0 0x7ff67c067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:54.934 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.927+0000 7ff6937fe640 1 -- 172.21.15.16:0/1748530290 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5530+0+0 (secure 0 0 0) 0x7ff69411d8e0 con 0x7ff69c15bd90 2023-05-22T15:42:54.934 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.927+0000 7ff69a59c640 1 --2- 172.21.15.16:0/1748530290 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ff67c0654a0 0x7ff67c067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:54.935 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.927+0000 7ff69a59c640 1 --2- 172.21.15.16:0/1748530290 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ff67c0654a0 0x7ff67c067950 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7ff68c000f70 tx=0x7ff68c009040 comp rx=0 tx=0).ready entity=mgr.6436 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:54.935 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:54.931+0000 7ff6937fe640 1 -- 172.21.15.16:0/1748530290 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7ff6940ef090 con 0x7ff69c15bd90 2023-05-22T15:42:55.088 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:55.083+0000 7ff6a0893640 1 -- 172.21.15.16:0/1748530290 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7ff69c04dbe0 con 0x7ff69c15bd90 2023-05-22T15:42:55.193 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:55.187+0000 7ff6937fe640 1 -- 172.21.15.16:0/1748530290 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v101) v1 ==== 138+0+0 (secure 0 0 0) 0x7ff6940c60b0 con 0x7ff69c15bd90 2023-05-22T15:42:55.194 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'cephfs_metadata' created 2023-05-22T15:42:55.198 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:55.191+0000 7ff6a0893640 1 -- 172.21.15.16:0/1748530290 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ff67c0654a0 msgr2=0x7ff67c067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:55.198 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:55.191+0000 7ff6a0893640 1 --2- 172.21.15.16:0/1748530290 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ff67c0654a0 0x7ff67c067950 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7ff68c000f70 tx=0x7ff68c009040 comp rx=0 tx=0).stop 2023-05-22T15:42:55.198 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:55.191+0000 7ff6a0893640 1 -- 172.21.15.16:0/1748530290 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff69c15bd90 msgr2=0x7ff69c0b34a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:55.198 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:55.191+0000 7ff6a0893640 1 --2- 172.21.15.16:0/1748530290 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff69c15bd90 0x7ff69c0b34a0 secure :-1 s=READY pgs=1271 cs=0 l=1 rev1=1 crypto rx=0x7ff69401a580 tx=0x7ff6940196f0 comp rx=0 tx=0).stop 2023-05-22T15:42:55.199 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:55.195+0000 7ff6a0893640 1 -- 172.21.15.16:0/1748530290 shutdown_connections 2023-05-22T15:42:55.199 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:55.195+0000 7ff6a0893640 1 --2- 172.21.15.16:0/1748530290 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff69c15bd90 0x7ff69c0b34a0 unknown :-1 s=CLOSED pgs=1271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:55.199 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:55.195+0000 7ff6a0893640 1 --2- 172.21.15.16:0/1748530290 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ff67c0654a0 0x7ff67c067950 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:55.200 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:55.195+0000 7ff6a0893640 1 --2- 172.21.15.16:0/1748530290 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff69c0caff0 0x7ff69c15ac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:55.200 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:55.195+0000 7ff6a0893640 1 --2- 172.21.15.16:0/1748530290 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff69c0c4a40 0x7ff69c15d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:55.200 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:55.195+0000 7ff6a0893640 1 -- 172.21.15.16:0/1748530290 >> 172.21.15.16:0/1748530290 conn(0x7ff69c069720 msgr2=0x7ff69c157d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:55.200 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:55.195+0000 7ff6a0893640 1 -- 172.21.15.16:0/1748530290 shutdown_connections 2023-05-22T15:42:55.201 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:55.195+0000 7ff6a0893640 1 -- 172.21.15.16:0/1748530290 wait complete. 2023-05-22T15:42:55.222 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-05-22T15:42:55.355 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:55.351+0000 7fe411c3d640 1 Processor -- start 2023-05-22T15:42:55.356 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:55.351+0000 7fe411c3d640 1 -- start start 2023-05-22T15:42:55.356 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:55.351+0000 7fe411c3d640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe40c0c2700 0x7fe40c06e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:55.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:55.351+0000 7fe411c3d640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe40c070c30 con 0x7fe40c06ecf0 2023-05-22T15:42:55.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:55.351+0000 7fe411c3d640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe40c06e320 con 0x7fe40c1648d0 2023-05-22T15:42:55.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:55.351+0000 7fe411c3d640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe40c06f190 con 0x7fe40c0c2700 2023-05-22T15:42:55.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:55.351+0000 7fe40affd640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe40c0c2700 0x7fe40c06e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:55.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:55.351+0000 7fe40bfff640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fe40c06ecf0 0x7fe40c0ce380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:44866/0 (socket says 172.21.15.16:44866) 2023-05-22T15:42:55.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:55.351+0000 7fe40bfff640 1 -- 172.21.15.16:0/356791567 learned_addr learned my addr 172.21.15.16:0/356791567 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:42:55.358 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:55.351+0000 7fe40affd640 1 -- 172.21.15.16:0/356791567 >> v1:172.21.15.62:6789/0 conn(0x7fe40c1648d0 legacy=0x7fe40c0c97b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:55.358 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:55.351+0000 7fe40affd640 1 -- 172.21.15.16:0/356791567 >> v1:172.21.15.16:6789/0 conn(0x7fe40c06ecf0 legacy=0x7fe40c0ce380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:55.358 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:55.351+0000 7fe40affd640 1 -- 172.21.15.16:0/356791567 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe40c06a2c0 con 0x7fe40c0c2700 2023-05-22T15:42:55.358 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:55.351+0000 7fe40a7fc640 1 -- 172.21.15.16:0/356791567 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2129617518 0 0) 0x7fe40c070c30 con 0x7fe40c06ecf0 2023-05-22T15:42:55.359 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:55.355+0000 7fe40affd640 1 --2- 172.21.15.16:0/356791567 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe40c0c2700 0x7fe40c06e7b0 secure :-1 s=READY pgs=1273 cs=0 l=1 rev1=1 crypto rx=0x7fe4000097c0 tx=0x7fe40001c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e73593c1401c3816 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:55.359 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:55.355+0000 7fe40a7fc640 1 -- 172.21.15.16:0/356791567 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4000ab600 con 0x7fe40c0c2700 2023-05-22T15:42:55.360 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:55.355+0000 7fe40a7fc640 1 -- 172.21.15.16:0/356791567 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4000a7050 con 0x7fe40c0c2700 2023-05-22T15:42:55.360 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:55.355+0000 7fe40a7fc640 1 -- 172.21.15.16:0/356791567 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4000ab980 con 0x7fe40c0c2700 2023-05-22T15:42:55.361 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:55.355+0000 7fe411c3d640 1 -- 172.21.15.16:0/356791567 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe40c0c2700 msgr2=0x7fe40c06e7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:55.361 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:55.355+0000 7fe411c3d640 1 --2- 172.21.15.16:0/356791567 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe40c0c2700 0x7fe40c06e7b0 secure :-1 s=READY pgs=1273 cs=0 l=1 rev1=1 crypto rx=0x7fe4000097c0 tx=0x7fe40001c480 comp rx=0 tx=0).stop 2023-05-22T15:42:55.361 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:55.355+0000 7fe411c3d640 1 -- 172.21.15.16:0/356791567 shutdown_connections 2023-05-22T15:42:55.361 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:55.355+0000 7fe411c3d640 1 --2- 172.21.15.16:0/356791567 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe40c0c2700 0x7fe40c06e7b0 unknown :-1 s=CLOSED pgs=1273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:55.361 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:55.355+0000 7fe411c3d640 1 -- 172.21.15.16:0/356791567 >> 172.21.15.16:0/356791567 conn(0x7fe40c0cc870 msgr2=0x7fe40c0cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:55.361 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:55.355+0000 7fe411c3d640 1 -- 172.21.15.16:0/356791567 shutdown_connections 2023-05-22T15:42:55.362 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:55.355+0000 7fe411c3d640 1 -- 172.21.15.16:0/356791567 wait complete. 2023-05-22T15:42:55.363 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:55.355+0000 7fe411c3d640 1 Processor -- start 2023-05-22T15:42:55.364 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:55.355+0000 7fe411c3d640 1 -- start start 2023-05-22T15:42:55.364 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:55.359+0000 7fe411c3d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe40c06ecf0 0x7fe40c1555c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:55.364 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:55.359+0000 7fe411c3d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe40c0c2700 0x7fe40c157740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:55.364 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:55.359+0000 7fe411c3d640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe40c1648d0 0x7fe40c15a510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:55.364 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:55.359+0000 7fe411c3d640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe40c155b00 con 0x7fe40c0c2700 2023-05-22T15:42:55.364 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:55.359+0000 7fe411c3d640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe40c157440 con 0x7fe40c06ecf0 2023-05-22T15:42:55.364 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:55.359+0000 7fe411c3d640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe40c1562a0 con 0x7fe40c1648d0 2023-05-22T15:42:55.365 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:55.359+0000 7fe40affd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe40c0c2700 0x7fe40c157740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:55.365 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:55.359+0000 7fe40affd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe40c0c2700 0x7fe40c157740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60682/0 (socket says 172.21.15.16:60682) 2023-05-22T15:42:55.365 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:55.359+0000 7fe40affd640 1 -- 172.21.15.16:0/2480078428 learned_addr learned my addr 172.21.15.16:0/2480078428 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:42:55.365 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:55.359+0000 7fe40b7fe640 1 --2- 172.21.15.16:0/2480078428 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe40c06ecf0 0x7fe40c1555c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:55.365 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:55.359+0000 7fe40bfff640 1 --2- 172.21.15.16:0/2480078428 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe40c1648d0 0x7fe40c15a510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:55.365 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:55.359+0000 7fe40affd640 1 -- 172.21.15.16:0/2480078428 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe40c1648d0 msgr2=0x7fe40c15a510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:55.366 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:55.359+0000 7fe40affd640 1 --2- 172.21.15.16:0/2480078428 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe40c1648d0 0x7fe40c15a510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:55.366 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:55.359+0000 7fe40affd640 1 -- 172.21.15.16:0/2480078428 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe40c06ecf0 msgr2=0x7fe40c1555c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:55.366 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:55.359+0000 7fe40affd640 1 --2- 172.21.15.16:0/2480078428 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe40c06ecf0 0x7fe40c1555c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:55.366 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:55.359+0000 7fe40affd640 1 -- 172.21.15.16:0/2480078428 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe400009420 con 0x7fe40c0c2700 2023-05-22T15:42:55.366 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:55.359+0000 7fe40bfff640 1 --2- 172.21.15.16:0/2480078428 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe40c1648d0 0x7fe40c15a510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:42:55.366 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:55.359+0000 7fe40affd640 1 --2- 172.21.15.16:0/2480078428 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe40c0c2700 0x7fe40c157740 secure :-1 s=READY pgs=1274 cs=0 l=1 rev1=1 crypto rx=0x7fe4000097c0 tx=0x7fe400019220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:55.367 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:55.359+0000 7fe3f3fff640 1 -- 172.21.15.16:0/2480078428 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4000b4840 con 0x7fe40c0c2700 2023-05-22T15:42:55.367 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:55.359+0000 7fe3f3fff640 1 -- 172.21.15.16:0/2480078428 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4000b49c0 con 0x7fe40c0c2700 2023-05-22T15:42:55.367 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:55.359+0000 7fe411c3d640 1 -- 172.21.15.16:0/2480078428 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe40c0c9ec0 con 0x7fe40c0c2700 2023-05-22T15:42:55.367 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:55.359+0000 7fe3f3fff640 1 -- 172.21.15.16:0/2480078428 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4000b4d70 con 0x7fe40c0c2700 2023-05-22T15:42:55.367 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:55.359+0000 7fe411c3d640 1 -- 172.21.15.16:0/2480078428 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe40c06dfb0 con 0x7fe40c0c2700 2023-05-22T15:42:55.368 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:55.359+0000 7fe411c3d640 1 -- 172.21.15.16:0/2480078428 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe40c0711c0 con 0x7fe40c0c2700 2023-05-22T15:42:55.368 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:55.359+0000 7fe3f3fff640 1 -- 172.21.15.16:0/2480078428 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 34) v1 ==== 84332+0+0 (secure 0 0 0) 0x7fe4000b2040 con 0x7fe40c0c2700 2023-05-22T15:42:55.368 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:55.359+0000 7fe3f3fff640 1 --2- 172.21.15.16:0/2480078428 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fe3f80654a0 0x7fe3f8067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:55.368 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:55.359+0000 7fe3f3fff640 1 -- 172.21.15.16:0/2480078428 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 5900+0+0 (secure 0 0 0) 0x7fe40011cdf0 con 0x7fe40c0c2700 2023-05-22T15:42:55.368 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:55.359+0000 7fe40b7fe640 1 --2- 172.21.15.16:0/2480078428 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fe3f80654a0 0x7fe3f8067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:55.369 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:55.359+0000 7fe40b7fe640 1 --2- 172.21.15.16:0/2480078428 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fe3f80654a0 0x7fe3f8067950 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fe4040012f0 tx=0x7fe404009040 comp rx=0 tx=0).ready entity=mgr.6436 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:55.370 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:55.367+0000 7fe3f3fff640 1 -- 172.21.15.16:0/2480078428 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fe4000ea920 con 0x7fe40c0c2700 2023-05-22T15:42:55.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:55.519+0000 7fe411c3d640 1 -- 172.21.15.16:0/2480078428 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fe40c0cb4f0 con 0x7fe40c0c2700 2023-05-22T15:42:56.200 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:56.195+0000 7fe3f3fff640 1 -- 172.21.15.16:0/2480078428 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v102) v1 ==== 170+0+0 (secure 0 0 0) 0x7fe4000b80d0 con 0x7fe40c0c2700 2023-05-22T15:42:56.201 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'cephfs_data' created 2023-05-22T15:42:56.208 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:56.203+0000 7fe411c3d640 1 -- 172.21.15.16:0/2480078428 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fe3f80654a0 msgr2=0x7fe3f8067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:56.209 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:56.203+0000 7fe411c3d640 1 --2- 172.21.15.16:0/2480078428 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fe3f80654a0 0x7fe3f8067950 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fe4040012f0 tx=0x7fe404009040 comp rx=0 tx=0).stop 2023-05-22T15:42:56.209 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:56.203+0000 7fe411c3d640 1 -- 172.21.15.16:0/2480078428 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe40c0c2700 msgr2=0x7fe40c157740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:56.209 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:56.203+0000 7fe411c3d640 1 --2- 172.21.15.16:0/2480078428 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe40c0c2700 0x7fe40c157740 secure :-1 s=READY pgs=1274 cs=0 l=1 rev1=1 crypto rx=0x7fe4000097c0 tx=0x7fe400019220 comp rx=0 tx=0).stop 2023-05-22T15:42:56.209 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:56.203+0000 7fe411c3d640 1 -- 172.21.15.16:0/2480078428 shutdown_connections 2023-05-22T15:42:56.209 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:56.203+0000 7fe411c3d640 1 --2- 172.21.15.16:0/2480078428 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fe3f80654a0 0x7fe3f8067950 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:56.209 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:56.203+0000 7fe411c3d640 1 --2- 172.21.15.16:0/2480078428 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe40c1648d0 0x7fe40c15a510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:56.209 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:56.203+0000 7fe411c3d640 1 --2- 172.21.15.16:0/2480078428 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe40c0c2700 0x7fe40c157740 unknown :-1 s=CLOSED pgs=1274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:56.210 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:56.203+0000 7fe411c3d640 1 --2- 172.21.15.16:0/2480078428 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe40c06ecf0 0x7fe40c1555c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:56.210 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:56.203+0000 7fe411c3d640 1 -- 172.21.15.16:0/2480078428 >> 172.21.15.16:0/2480078428 conn(0x7fe40c0cc870 msgr2=0x7fe40c0cefb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:56.210 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:56.203+0000 7fe411c3d640 1 -- 172.21.15.16:0/2480078428 shutdown_connections 2023-05-22T15:42:56.210 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:56.203+0000 7fe411c3d640 1 -- 172.21.15.16:0/2480078428 wait complete. 2023-05-22T15:42:56.243 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-05-22T15:42:56.362 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:56.355+0000 7f26d8ff6640 1 Processor -- start 2023-05-22T15:42:56.363 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:56.355+0000 7f26d8ff6640 1 -- start start 2023-05-22T15:42:56.364 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:56.359+0000 7f26d8ff6640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f26d415bca0 0x7f26d40cda80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:56.364 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:56.359+0000 7f26d8ff6640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26d40cccc0 con 0x7f26d40b5a50 2023-05-22T15:42:56.365 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:56.359+0000 7f26d8ff6640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26d40ccfa0 con 0x7f26d40d2480 2023-05-22T15:42:56.365 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:56.359+0000 7f26d8ff6640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26d40cd660 con 0x7f26d415bca0 2023-05-22T15:42:56.365 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:56.359+0000 7f26d259c640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f26d415bca0 0x7f26d40cda80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:56.365 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:56.359+0000 7f26d2d9d640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f26d40b5a50 0x7f26d415fb50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:44876/0 (socket says 172.21.15.16:44876) 2023-05-22T15:42:56.366 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:56.359+0000 7f26d2d9d640 1 -- 172.21.15.16:0/3635956165 learned_addr learned my addr 172.21.15.16:0/3635956165 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:42:56.366 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:56.359+0000 7f26d259c640 1 -- 172.21.15.16:0/3635956165 >> v1:172.21.15.62:6789/0 conn(0x7f26d40d2480 legacy=0x7f26d40b5340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:56.366 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:56.359+0000 7f26d259c640 1 -- 172.21.15.16:0/3635956165 >> v1:172.21.15.16:6789/0 conn(0x7f26d40b5a50 legacy=0x7f26d415fb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:56.367 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:56.359+0000 7f26d159a640 1 -- 172.21.15.16:0/3635956165 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1252052229 0 0) 0x7f26d40cccc0 con 0x7f26d40b5a50 2023-05-22T15:42:56.367 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:56.359+0000 7f26d259c640 1 -- 172.21.15.16:0/3635956165 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26d40cd280 con 0x7f26d415bca0 2023-05-22T15:42:56.371 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:56.363+0000 7f26d259c640 1 --2- 172.21.15.16:0/3635956165 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f26d415bca0 0x7f26d40cda80 secure :-1 s=READY pgs=1276 cs=0 l=1 rev1=1 crypto rx=0x7f26c40097c0 tx=0x7f26c401c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86d2ba814e5c4220 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:56.371 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:56.363+0000 7f26d159a640 1 -- 172.21.15.16:0/3635956165 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26c40a8050 con 0x7f26d415bca0 2023-05-22T15:42:56.372 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:56.363+0000 7f26d159a640 1 -- 172.21.15.16:0/3635956165 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26c40ac5f0 con 0x7f26d415bca0 2023-05-22T15:42:56.372 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:56.363+0000 7f26d159a640 1 -- 172.21.15.16:0/3635956165 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26c40ac980 con 0x7f26d415bca0 2023-05-22T15:42:56.372 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:56.363+0000 7f26d8ff6640 1 -- 172.21.15.16:0/3635956165 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f26d415bca0 msgr2=0x7f26d40cda80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:56.372 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:56.363+0000 7f26d8ff6640 1 --2- 172.21.15.16:0/3635956165 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f26d415bca0 0x7f26d40cda80 secure :-1 s=READY pgs=1276 cs=0 l=1 rev1=1 crypto rx=0x7f26c40097c0 tx=0x7f26c401c480 comp rx=0 tx=0).stop 2023-05-22T15:42:56.373 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:56.363+0000 7f26d8ff6640 1 -- 172.21.15.16:0/3635956165 shutdown_connections 2023-05-22T15:42:56.373 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:56.363+0000 7f26d8ff6640 1 --2- 172.21.15.16:0/3635956165 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f26d415bca0 0x7f26d40cda80 unknown :-1 s=CLOSED pgs=1276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:56.373 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:56.363+0000 7f26d8ff6640 1 -- 172.21.15.16:0/3635956165 >> 172.21.15.16:0/3635956165 conn(0x7f26d40655c0 msgr2=0x7f26d4070e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:56.374 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:56.363+0000 7f26d8ff6640 1 -- 172.21.15.16:0/3635956165 shutdown_connections 2023-05-22T15:42:56.374 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:56.363+0000 7f26d8ff6640 1 -- 172.21.15.16:0/3635956165 wait complete. 2023-05-22T15:42:56.374 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:56.363+0000 7f26d8ff6640 1 Processor -- start 2023-05-22T15:42:56.374 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:56.363+0000 7f26d8ff6640 1 -- start start 2023-05-22T15:42:56.374 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:56.363+0000 7f26d8ff6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f26d40b5a50 0x7f26d40ca0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:56.374 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:56.363+0000 7f26d8ff6640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f26d40d2480 0x7f26d40cef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:56.374 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:56.363+0000 7f26d8ff6640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f26d415bca0 0x7f26d40cf440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:56.375 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:56.363+0000 7f26d8ff6640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26d4158d30 con 0x7f26d40b5a50 2023-05-22T15:42:56.375 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:56.363+0000 7f26d8ff6640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26d4159010 con 0x7f26d40d2480 2023-05-22T15:42:56.375 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:56.363+0000 7f26d8ff6640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26d41592f0 con 0x7f26d415bca0 2023-05-22T15:42:56.375 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:56.363+0000 7f26d259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f26d40b5a50 0x7f26d40ca0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:56.375 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:56.363+0000 7f26d259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f26d40b5a50 0x7f26d40ca0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60706/0 (socket says 172.21.15.16:60706) 2023-05-22T15:42:56.375 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:56.363+0000 7f26d259c640 1 -- 172.21.15.16:0/2641799710 learned_addr learned my addr 172.21.15.16:0/2641799710 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:42:56.375 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:56.363+0000 7f26d259c640 1 -- 172.21.15.16:0/2641799710 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f26d415bca0 msgr2=0x7f26d40cf440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:56.376 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:56.363+0000 7f26d2d9d640 1 --2- 172.21.15.16:0/2641799710 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f26d415bca0 0x7f26d40cf440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:56.376 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:56.363+0000 7f26d1d9b640 1 --2- 172.21.15.16:0/2641799710 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f26d40d2480 0x7f26d40cef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:56.376 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:56.363+0000 7f26d259c640 1 --2- 172.21.15.16:0/2641799710 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f26d415bca0 0x7f26d40cf440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:56.376 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:56.363+0000 7f26d259c640 1 -- 172.21.15.16:0/2641799710 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f26d40d2480 msgr2=0x7f26d40cef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:56.376 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:56.363+0000 7f26d259c640 1 --2- 172.21.15.16:0/2641799710 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f26d40d2480 0x7f26d40cef00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:56.376 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:56.363+0000 7f26d259c640 1 -- 172.21.15.16:0/2641799710 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26c4009420 con 0x7f26d40b5a50 2023-05-22T15:42:56.376 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:56.363+0000 7f26d259c640 1 --2- 172.21.15.16:0/2641799710 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f26d40b5a50 0x7f26d40ca0f0 secure :-1 s=READY pgs=1277 cs=0 l=1 rev1=1 crypto rx=0x7f26c40af040 tx=0x7f26c401acc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:56.377 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:56.363+0000 7f26c2ffd640 1 -- 172.21.15.16:0/2641799710 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26c40b8860 con 0x7f26d40b5a50 2023-05-22T15:42:56.377 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:56.367+0000 7f26c2ffd640 1 -- 172.21.15.16:0/2641799710 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26c40b89e0 con 0x7f26d40b5a50 2023-05-22T15:42:56.377 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:56.367+0000 7f26d8ff6640 1 -- 172.21.15.16:0/2641799710 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26d40d2e20 con 0x7f26d40b5a50 2023-05-22T15:42:56.377 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:56.367+0000 7f26c2ffd640 1 -- 172.21.15.16:0/2641799710 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26c40b8d90 con 0x7f26d40b5a50 2023-05-22T15:42:56.377 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:56.367+0000 7f26d8ff6640 1 -- 172.21.15.16:0/2641799710 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26d40d2c90 con 0x7f26d40b5a50 2023-05-22T15:42:56.377 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:56.367+0000 7f26d8ff6640 1 -- 172.21.15.16:0/2641799710 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26d416a470 con 0x7f26d40b5a50 2023-05-22T15:42:56.378 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:56.367+0000 7f26c2ffd640 1 -- 172.21.15.16:0/2641799710 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 34) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f26c40b6400 con 0x7f26d40b5a50 2023-05-22T15:42:56.378 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:56.367+0000 7f26c2ffd640 1 --2- 172.21.15.16:0/2641799710 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f26b40654a0 0x7f26b4067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:56.379 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:56.367+0000 7f26c2ffd640 1 -- 172.21.15.16:0/2641799710 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 6282+0+0 (secure 0 0 0) 0x7f26c411ced0 con 0x7f26d40b5a50 2023-05-22T15:42:56.379 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:56.367+0000 7f26d1d9b640 1 --2- 172.21.15.16:0/2641799710 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f26b40654a0 0x7f26b4067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:56.379 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:56.367+0000 7f26d1d9b640 1 --2- 172.21.15.16:0/2641799710 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f26b40654a0 0x7f26b4067950 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f26d40c8100 tx=0x7f26c8009040 comp rx=0 tx=0).ready entity=mgr.6436 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:56.379 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:56.371+0000 7f26c2ffd640 1 -- 172.21.15.16:0/2641799710 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f26c40ea820 con 0x7f26d40b5a50 2023-05-22T15:42:56.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:56.559+0000 7f26d8ff6640 1 -- 172.21.15.16:0/2641799710 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f26d404dbe0 con 0x7f26d40b5a50 2023-05-22T15:42:56.574 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:56.571+0000 7f26c2ffd640 1 -- 172.21.15.16:0/2641799710 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 12 and data pool 13 v34) v1 ==== 174+0+0 (secure 0 0 0) 0x7f26c40ede40 con 0x7f26d40b5a50 2023-05-22T15:42:56.574 INFO:teuthology.orchestra.run.smithi016.stderr:new fs with metadata pool 12 and data pool 13 2023-05-22T15:42:56.578 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:56.571+0000 7f26d8ff6640 1 -- 172.21.15.16:0/2641799710 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f26b40654a0 msgr2=0x7f26b4067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:56.578 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:56.571+0000 7f26d8ff6640 1 --2- 172.21.15.16:0/2641799710 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f26b40654a0 0x7f26b4067950 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f26d40c8100 tx=0x7f26c8009040 comp rx=0 tx=0).stop 2023-05-22T15:42:56.578 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:56.571+0000 7f26d8ff6640 1 -- 172.21.15.16:0/2641799710 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f26d40b5a50 msgr2=0x7f26d40ca0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:56.579 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:56.571+0000 7f26d8ff6640 1 --2- 172.21.15.16:0/2641799710 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f26d40b5a50 0x7f26d40ca0f0 secure :-1 s=READY pgs=1277 cs=0 l=1 rev1=1 crypto rx=0x7f26c40af040 tx=0x7f26c401acc0 comp rx=0 tx=0).stop 2023-05-22T15:42:56.579 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:56.571+0000 7f26d8ff6640 1 -- 172.21.15.16:0/2641799710 shutdown_connections 2023-05-22T15:42:56.579 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:56.571+0000 7f26d8ff6640 1 --2- 172.21.15.16:0/2641799710 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f26b40654a0 0x7f26b4067950 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:56.579 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:56.571+0000 7f26d8ff6640 1 --2- 172.21.15.16:0/2641799710 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f26d415bca0 0x7f26d40cf440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:56.579 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:56.571+0000 7f26d8ff6640 1 --2- 172.21.15.16:0/2641799710 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f26d40d2480 0x7f26d40cef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:56.580 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:56.571+0000 7f26d8ff6640 1 --2- 172.21.15.16:0/2641799710 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f26d40b5a50 0x7f26d40ca0f0 unknown :-1 s=CLOSED pgs=1277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:56.580 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:56.571+0000 7f26d8ff6640 1 -- 172.21.15.16:0/2641799710 >> 172.21.15.16:0/2641799710 conn(0x7f26d40655c0 msgr2=0x7f26d40b3820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:56.580 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:56.571+0000 7f26d8ff6640 1 -- 172.21.15.16:0/2641799710 shutdown_connections 2023-05-22T15:42:56.580 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:56.571+0000 7f26d8ff6640 1 -- 172.21.15.16:0/2641799710 wait complete. 2023-05-22T15:42:56.608 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-05-22T15:42:56.730 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:56.723+0000 7f2dd6f40640 1 Processor -- start 2023-05-22T15:42:56.730 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:56.723+0000 7f2dd6f40640 1 -- start start 2023-05-22T15:42:56.732 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:56.727+0000 7f2dd6f40640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f2dd00ca4e0 0x7f2dd015e460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:56.732 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:56.727+0000 7f2dd6f40640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2dd00c9b40 0x7f2dd00afb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:56.732 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:56.727+0000 7f2dd6f40640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2dd00b0650 con 0x7f2dd0070df0 2023-05-22T15:42:56.732 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:56.727+0000 7f2dd6f40640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2dd00b0370 con 0x7f2dd00c9b40 2023-05-22T15:42:56.732 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:56.727+0000 7f2dd6f40640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2dd015d980 con 0x7f2dd00ca4e0 2023-05-22T15:42:56.732 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:56.727+0000 7f2dcffff640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2dd00c9b40 0x7f2dd00afb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:56.733 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:56.727+0000 7f2dcffff640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2dd00c9b40 0x7f2dd00afb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60710/0 (socket says 172.21.15.16:60710) 2023-05-22T15:42:56.733 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:56.727+0000 7f2dcffff640 1 -- 172.21.15.16:0/3916213400 learned_addr learned my addr 172.21.15.16:0/3916213400 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:42:56.733 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:56.727+0000 7f2dd4cdc640 1 --2- 172.21.15.16:0/3916213400 >> v2:172.21.15.62:3300/0 conn(0x7f2dd00ca4e0 0x7f2dd015e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:56.733 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:56.727+0000 7f2dcffff640 1 -- 172.21.15.16:0/3916213400 >> v2:172.21.15.62:3300/0 conn(0x7f2dd00ca4e0 msgr2=0x7f2dd015e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:42:56.733 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:56.727+0000 7f2dcffff640 1 --2- 172.21.15.16:0/3916213400 >> v2:172.21.15.62:3300/0 conn(0x7f2dd00ca4e0 0x7f2dd015e460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:56.733 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:56.727+0000 7f2dcffff640 1 -- 172.21.15.16:0/3916213400 >> v1:172.21.15.62:6789/0 conn(0x7f2dd0070df0 legacy=0x7f2dd0071210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:42:56.733 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:56.727+0000 7f2dcffff640 1 -- 172.21.15.16:0/3916213400 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dd015dc60 con 0x7f2dd00c9b40 2023-05-22T15:42:56.733 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:56.727+0000 7f2dd4cdc640 1 --2- 172.21.15.16:0/3916213400 >> v2:172.21.15.62:3300/0 conn(0x7f2dd00ca4e0 0x7f2dd015e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:42:56.734 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:56.727+0000 7f2dcffff640 1 --2- 172.21.15.16:0/3916213400 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2dd00c9b40 0x7f2dd00afb40 secure :-1 s=READY pgs=1278 cs=0 l=1 rev1=1 crypto rx=0x7f2dd00cad40 tx=0x7f2dc401c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a44428da5cb5aa6 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:56.734 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:56.731+0000 7f2dcf7fe640 1 -- 172.21.15.16:0/3916213400 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dc40ab670 con 0x7f2dd00c9b40 2023-05-22T15:42:56.734 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:56.731+0000 7f2dcf7fe640 1 -- 172.21.15.16:0/3916213400 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2dc40ab7d0 con 0x7f2dd00c9b40 2023-05-22T15:42:56.734 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:56.731+0000 7f2dcf7fe640 1 -- 172.21.15.16:0/3916213400 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dc40abb60 con 0x7f2dd00c9b40 2023-05-22T15:42:56.735 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:56.731+0000 7f2dd6f40640 1 -- 172.21.15.16:0/3916213400 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2dd00c9b40 msgr2=0x7f2dd00afb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:56.735 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:56.731+0000 7f2dd6f40640 1 --2- 172.21.15.16:0/3916213400 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2dd00c9b40 0x7f2dd00afb40 secure :-1 s=READY pgs=1278 cs=0 l=1 rev1=1 crypto rx=0x7f2dd00cad40 tx=0x7f2dc401c490 comp rx=0 tx=0).stop 2023-05-22T15:42:56.735 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:56.731+0000 7f2dd6f40640 1 -- 172.21.15.16:0/3916213400 shutdown_connections 2023-05-22T15:42:56.736 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:56.731+0000 7f2dd6f40640 1 --2- 172.21.15.16:0/3916213400 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2dd00c9b40 0x7f2dd00afb40 unknown :-1 s=CLOSED pgs=1278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:56.736 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:56.731+0000 7f2dd6f40640 1 --2- 172.21.15.16:0/3916213400 >> v2:172.21.15.62:3300/0 conn(0x7f2dd00ca4e0 0x7f2dd015e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:56.736 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:56.731+0000 7f2dd6f40640 1 -- 172.21.15.16:0/3916213400 >> 172.21.15.16:0/3916213400 conn(0x7f2dd0069720 msgr2=0x7f2dd0069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:56.736 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:56.731+0000 7f2dd6f40640 1 -- 172.21.15.16:0/3916213400 shutdown_connections 2023-05-22T15:42:56.736 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:56.731+0000 7f2dd6f40640 1 -- 172.21.15.16:0/3916213400 wait complete. 2023-05-22T15:42:56.738 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:56.731+0000 7f2dd6f40640 1 Processor -- start 2023-05-22T15:42:56.738 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:56.731+0000 7f2dd6f40640 1 -- start start 2023-05-22T15:42:56.738 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:56.731+0000 7f2dd6f40640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2dd0070df0 0x7f2dd01f8490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:56.738 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:56.731+0000 7f2dd6f40640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2dd00c9b40 0x7f2dd014f070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:56.738 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:56.731+0000 7f2dd6f40640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2dd00ca4e0 0x7f2dd006b730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:56.738 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:56.731+0000 7f2dd6f40640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dd0155320 con 0x7f2dd0070df0 2023-05-22T15:42:56.739 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:56.731+0000 7f2dd6f40640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dd00cbbf0 con 0x7f2dd00c9b40 2023-05-22T15:42:56.739 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:56.731+0000 7f2dd6f40640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2dd01f89d0 con 0x7f2dd00ca4e0 2023-05-22T15:42:56.739 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:56.731+0000 7f2dd4cdc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2dd0070df0 0x7f2dd01f8490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:56.739 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:56.731+0000 7f2dd4cdc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2dd0070df0 0x7f2dd01f8490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60724/0 (socket says 172.21.15.16:60724) 2023-05-22T15:42:56.740 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:56.731+0000 7f2dd4cdc640 1 -- 172.21.15.16:0/4013830022 learned_addr learned my addr 172.21.15.16:0/4013830022 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:42:56.740 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:56.731+0000 7f2dcffff640 1 --2- 172.21.15.16:0/4013830022 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2dd00c9b40 0x7f2dd014f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:56.740 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:56.731+0000 7f2dd54dd640 1 --2- 172.21.15.16:0/4013830022 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2dd00ca4e0 0x7f2dd006b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:56.740 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:56.731+0000 7f2dd4cdc640 1 -- 172.21.15.16:0/4013830022 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2dd00ca4e0 msgr2=0x7f2dd006b730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:56.740 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:56.731+0000 7f2dd4cdc640 1 --2- 172.21.15.16:0/4013830022 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2dd00ca4e0 0x7f2dd006b730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:56.740 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:56.731+0000 7f2dd4cdc640 1 -- 172.21.15.16:0/4013830022 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2dd00c9b40 msgr2=0x7f2dd014f070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:56.741 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:56.731+0000 7f2dd4cdc640 1 --2- 172.21.15.16:0/4013830022 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2dd00c9b40 0x7f2dd014f070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:56.741 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:56.731+0000 7f2dd4cdc640 1 -- 172.21.15.16:0/4013830022 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dc40094d0 con 0x7f2dd0070df0 2023-05-22T15:42:56.741 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:56.731+0000 7f2dd4cdc640 1 --2- 172.21.15.16:0/4013830022 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2dd0070df0 0x7f2dd01f8490 secure :-1 s=READY pgs=1279 cs=0 l=1 rev1=1 crypto rx=0x7f2dc000c7f0 tx=0x7f2dc000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:56.741 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:56.731+0000 7f2dcd7fa640 1 -- 172.21.15.16:0/4013830022 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dc0007820 con 0x7f2dd0070df0 2023-05-22T15:42:56.741 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:56.731+0000 7f2dcd7fa640 1 -- 172.21.15.16:0/4013830022 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2dc0007980 con 0x7f2dd0070df0 2023-05-22T15:42:56.741 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:56.731+0000 7f2dcd7fa640 1 -- 172.21.15.16:0/4013830022 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dc0007d30 con 0x7f2dd0070df0 2023-05-22T15:42:56.741 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:56.731+0000 7f2dd6f40640 1 -- 172.21.15.16:0/4013830022 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2dd00c8e80 con 0x7f2dd0070df0 2023-05-22T15:42:56.741 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:56.735+0000 7f2dd6f40640 1 -- 172.21.15.16:0/4013830022 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2dd00ca8b0 con 0x7f2dd0070df0 2023-05-22T15:42:56.742 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:56.735+0000 7f2dd6f40640 1 -- 172.21.15.16:0/4013830022 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2dd00c9010 con 0x7f2dd0070df0 2023-05-22T15:42:56.742 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:56.735+0000 7f2dcd7fa640 1 -- 172.21.15.16:0/4013830022 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 34) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f2dc0024480 con 0x7f2dd0070df0 2023-05-22T15:42:56.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:56.735+0000 7f2dcd7fa640 1 --2- 172.21.15.16:0/4013830022 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f2dbc0654a0 0x7f2dbc067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:56.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:56.735+0000 7f2dcd7fa640 1 -- 172.21.15.16:0/4013830022 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f2dc0083340 con 0x7f2dd0070df0 2023-05-22T15:42:56.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:56.735+0000 7f2dcffff640 1 --2- 172.21.15.16:0/4013830022 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f2dbc0654a0 0x7f2dbc067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:56.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:56.735+0000 7f2dcffff640 1 --2- 172.21.15.16:0/4013830022 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f2dbc0654a0 0x7f2dbc067950 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f2dc40af000 tx=0x7f2dc40a7000 comp rx=0 tx=0).ready entity=mgr.6436 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:56.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:56.739+0000 7f2dcd7fa640 1 -- 172.21.15.16:0/4013830022 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f2dc0050cf0 con 0x7f2dd0070df0 2023-05-22T15:42:56.899 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:56.895+0000 7f2dd6f40640 1 -- 172.21.15.16:0/4013830022 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2dd00cb010 con 0x7f2dd0070df0 2023-05-22T15:42:56.902 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:56.895+0000 7f2dcd7fa640 1 -- 172.21.15.16:0/4013830022 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v103) v1 ==== 74+0+18055 (secure 0 0 0) 0x7f2dc0054310 con 0x7f2dd0070df0 2023-05-22T15:42:56.903 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:42:56.903 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":103,"fsid":"b2719834-7847-4d98-90a6-f65d57f85c49","created":"2023-05-22T15:26:25.259314+0000","modified":"2023-05-22T15:42:56.564315+0000","last_up_change":"2023-05-22T15:26:36.448524+0000","last_in_change":"2023-05-22T15:26:27.675668+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-05-22T15:26:38.019781+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-05-22T15:26:43.570514+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2023-05-22T15:42:55.131159+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"103","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2023-05-22T15:42:55.567899+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"103","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9e7ff854-b25f-4cbb-a5c2-1799d9c53d2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":102,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":37021},{"type":"v1","addr":"172.21.15.16:6825","nonce":37021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":37021},{"type":"v1","addr":"172.21.15.16:6827","nonce":37021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":37021},{"type":"v1","addr":"172.21.15.16:6831","nonce":37021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":37021},{"type":"v1","addr":"172.21.15.16:6829","nonce":37021}]},"public_addr":"172.21.15.16:6825/37021","cluster_addr":"172.21.15.16:6827/37021","heartbeat_back_addr":"172.21.15.16:6831/37021","heartbeat_front_addr":"172.21.15.16:6829/37021","state":["exists","up"]},{"osd":1,"uuid":"4702fbc9-a927-4565-aca5-41dc69cc8264","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":102,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":37008},{"type":"v1","addr":"172.21.15.16:6801","nonce":37008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":37008},{"type":"v1","addr":"172.21.15.16:6803","nonce":37008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":37008},{"type":"v1","addr":"172.21.15.16:6807","nonce":37008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":37008},{"type":"v1","addr":"172.21.15.16:6805","nonce":37008}]},"public_addr":"172.21.15.16:6801/37008","cluster_addr":"172.21.15.16:6803/37008","heartbeat_back_addr":"172.21.15.16:6807/37008","heartbeat_front_addr":"172.21.15.16:6805/37008","state":["exists","up"]},{"osd":2,"uuid":"1e7bba46-4eba-47ef-bb81-5e2d371638ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":102,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":37009},{"type":"v1","addr":"172.21.15.16:6809","nonce":37009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":37009},{"type":"v1","addr":"172.21.15.16:6811","nonce":37009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":37009},{"type":"v1","addr":"172.21.15.16:6815","nonce":37009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":37009},{"type":"v1","addr":"172.21.15.16:6813","nonce":37009}]},"public_addr":"172.21.15.16:6809/37009","cluster_addr":"172.21.15.16:6811/37009","heartbeat_back_addr":"172.21.15.16:6815/37009","heartbeat_front_addr":"172.21.15.16:6813/37009","state":["exists","up"]},{"osd":3,"uuid":"383cacc0-5a88-4904-990e-498ca59ef48a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":102,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":37020},{"type":"v1","addr":"172.21.15.16:6817","nonce":37020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":37020},{"type":"v1","addr":"172.21.15.16:6819","nonce":37020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":37020},{"type":"v1","addr":"172.21.15.16:6823","nonce":37020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":37020},{"type":"v1","addr":"172.21.15.16:6821","nonce":37020}]},"public_addr":"172.21.15.16:6817/37020","cluster_addr":"172.21.15.16:6819/37020","heartbeat_back_addr":"172.21.15.16:6823/37020","heartbeat_front_addr":"172.21.15.16:6821/37020","state":["exists","up"]},{"osd":4,"uuid":"970500ef-95b7-42c1-b080-eed4fae7013d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":102,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":37353},{"type":"v1","addr":"172.21.15.62:6817","nonce":37353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6819","nonce":37353},{"type":"v1","addr":"172.21.15.62:6822","nonce":37353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37353},{"type":"v1","addr":"172.21.15.62:6831","nonce":37353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37353},{"type":"v1","addr":"172.21.15.62:6826","nonce":37353}]},"public_addr":"172.21.15.62:6817/37353","cluster_addr":"172.21.15.62:6822/37353","heartbeat_back_addr":"172.21.15.62:6831/37353","heartbeat_front_addr":"172.21.15.62:6826/37353","state":["exists","up"]},{"osd":5,"uuid":"022d67db-d4d8-445b-b1b9-8b446a00f5c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":102,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":37352},{"type":"v1","addr":"172.21.15.62:6801","nonce":37352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":37352},{"type":"v1","addr":"172.21.15.62:6803","nonce":37352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":37352},{"type":"v1","addr":"172.21.15.62:6807","nonce":37352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":37352},{"type":"v1","addr":"172.21.15.62:6805","nonce":37352}]},"public_addr":"172.21.15.62:6801/37352","cluster_addr":"172.21.15.62:6803/37352","heartbeat_back_addr":"172.21.15.62:6807/37352","heartbeat_front_addr":"172.21.15.62:6805/37352","state":["exists","up"]},{"osd":6,"uuid":"42bd372b-6688-4a1b-8407-e16e8c266e33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":102,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":37354},{"type":"v1","addr":"172.21.15.62:6820","nonce":37354}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":37354},{"type":"v1","addr":"172.21.15.62:6823","nonce":37354}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6829","nonce":37354},{"type":"v1","addr":"172.21.15.62:6830","nonce":37354}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6825","nonce":37354},{"type":"v1","addr":"172.21.15.62:6827","nonce":37354}]},"public_addr":"172.21.15.62:6820/37354","cluster_addr":"172.21.15.62:6823/37354","heartbeat_back_addr":"172.21.15.62:6830/37354","heartbeat_front_addr":"172.21.15.62:6827/37354","state":["exists","up"]},{"osd":7,"uuid":"ac70b2a3-f41b-4bf2-b227-7f2cbb55426d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":102,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37355},{"type":"v1","addr":"172.21.15.62:6809","nonce":37355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37355},{"type":"v1","addr":"172.21.15.62:6811","nonce":37355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":37355},{"type":"v1","addr":"172.21.15.62:6815","nonce":37355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":37355},{"type":"v1","addr":"172.21.15.62:6813","nonce":37355}]},"public_addr":"172.21.15.62:6809/37355","cluster_addr":"172.21.15.62:6811/37355","heartbeat_back_addr":"172.21.15.62:6815/37355","heartbeat_front_addr":"172.21.15.62:6813/37355","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.368896+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.107273+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.276888+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.287502+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.124791+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.251937+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.255679+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.232797+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-05-22T15:42:56.905 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:56.899+0000 7f2dd6f40640 1 -- 172.21.15.16:0/4013830022 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f2dbc0654a0 msgr2=0x7f2dbc067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:56.906 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:56.899+0000 7f2dd6f40640 1 --2- 172.21.15.16:0/4013830022 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f2dbc0654a0 0x7f2dbc067950 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f2dc40af000 tx=0x7f2dc40a7000 comp rx=0 tx=0).stop 2023-05-22T15:42:56.906 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:56.899+0000 7f2dd6f40640 1 -- 172.21.15.16:0/4013830022 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2dd0070df0 msgr2=0x7f2dd01f8490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:56.906 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:56.899+0000 7f2dd6f40640 1 --2- 172.21.15.16:0/4013830022 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2dd0070df0 0x7f2dd01f8490 secure :-1 s=READY pgs=1279 cs=0 l=1 rev1=1 crypto rx=0x7f2dc000c7f0 tx=0x7f2dc000ccb0 comp rx=0 tx=0).stop 2023-05-22T15:42:56.906 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:56.899+0000 7f2dd6f40640 1 -- 172.21.15.16:0/4013830022 shutdown_connections 2023-05-22T15:42:56.907 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:56.899+0000 7f2dd6f40640 1 --2- 172.21.15.16:0/4013830022 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f2dbc0654a0 0x7f2dbc067950 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:56.907 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:56.899+0000 7f2dd6f40640 1 --2- 172.21.15.16:0/4013830022 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2dd00ca4e0 0x7f2dd006b730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:56.907 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:56.899+0000 7f2dd6f40640 1 --2- 172.21.15.16:0/4013830022 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2dd00c9b40 0x7f2dd014f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:56.908 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:56.899+0000 7f2dd6f40640 1 --2- 172.21.15.16:0/4013830022 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2dd0070df0 0x7f2dd01f8490 unknown :-1 s=CLOSED pgs=1279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:56.908 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:56.899+0000 7f2dd6f40640 1 -- 172.21.15.16:0/4013830022 >> 172.21.15.16:0/4013830022 conn(0x7f2dd0069720 msgr2=0x7f2dd00b0eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:56.908 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:56.899+0000 7f2dd6f40640 1 -- 172.21.15.16:0/4013830022 shutdown_connections 2023-05-22T15:42:56.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:56.899+0000 7f2dd6f40640 1 -- 172.21.15.16:0/4013830022 wait complete. 2023-05-22T15:42:56.924 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-05-22T15:42:57.037 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.031+0000 7f6b082e3640 1 Processor -- start 2023-05-22T15:42:57.038 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.031+0000 7f6b082e3640 1 -- start start 2023-05-22T15:42:57.039 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.035+0000 7f6b082e3640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6b0015c130 0x7f6b000b2330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:57.039 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.035+0000 7f6b082e3640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f6b000cfc20 0x7f6b000ca8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:57.039 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.035+0000 7f6b082e3640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b000b2cc0 con 0x7f6b000b2870 2023-05-22T15:42:57.040 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.035+0000 7f6b082e3640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b0006c4a0 con 0x7f6b0015c130 2023-05-22T15:42:57.040 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.035+0000 7f6b082e3640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b000ce160 con 0x7f6b000cfc20 2023-05-22T15:42:57.040 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.035+0000 7f6b0607f640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6b0015c130 0x7f6b000b2330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:57.040 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.035+0000 7f6b0607f640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6b0015c130 0x7f6b000b2330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60736/0 (socket says 172.21.15.16:60736) 2023-05-22T15:42:57.040 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.035+0000 7f6b0607f640 1 -- 172.21.15.16:0/1723551098 learned_addr learned my addr 172.21.15.16:0/1723551098 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:42:57.040 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.035+0000 7f6b0607f640 1 -- 172.21.15.16:0/1723551098 >> v2:172.21.15.62:3300/0 conn(0x7f6b000cfc20 msgr2=0x7f6b000ca8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:42:57.041 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.035+0000 7f6b06880640 1 --2- 172.21.15.16:0/1723551098 >> v2:172.21.15.62:3300/0 conn(0x7f6b000cfc20 0x7f6b000ca8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:57.041 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.035+0000 7f6b0607f640 1 --2- 172.21.15.16:0/1723551098 >> v2:172.21.15.62:3300/0 conn(0x7f6b000cfc20 0x7f6b000ca8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:57.041 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.035+0000 7f6b0607f640 1 -- 172.21.15.16:0/1723551098 >> v1:172.21.15.62:6789/0 conn(0x7f6b000b2870 legacy=0x7f6b000cf510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:42:57.041 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.035+0000 7f6b0607f640 1 -- 172.21.15.16:0/1723551098 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b000cece0 con 0x7f6b0015c130 2023-05-22T15:42:57.041 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.035+0000 7f6b06880640 1 --2- 172.21.15.16:0/1723551098 >> v2:172.21.15.62:3300/0 conn(0x7f6b000cfc20 0x7f6b000ca8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:42:57.042 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.035+0000 7f6b0607f640 1 --2- 172.21.15.16:0/1723551098 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6b0015c130 0x7f6b000b2330 secure :-1 s=READY pgs=1280 cs=0 l=1 rev1=1 crypto rx=0x7f6b00070310 tx=0x7f6afc01c4c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c296ebf4fa61ebe9 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:57.042 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.035+0000 7f6b0507d640 1 -- 172.21.15.16:0/1723551098 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6afc0ac410 con 0x7f6b0015c130 2023-05-22T15:42:57.042 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.035+0000 7f6b0507d640 1 -- 172.21.15.16:0/1723551098 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6afc0ac570 con 0x7f6b0015c130 2023-05-22T15:42:57.043 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.035+0000 7f6b0507d640 1 -- 172.21.15.16:0/1723551098 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6afc0ac900 con 0x7f6b0015c130 2023-05-22T15:42:57.043 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.035+0000 7f6b082e3640 1 -- 172.21.15.16:0/1723551098 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6b0015c130 msgr2=0x7f6b000b2330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:57.043 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.035+0000 7f6b082e3640 1 --2- 172.21.15.16:0/1723551098 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6b0015c130 0x7f6b000b2330 secure :-1 s=READY pgs=1280 cs=0 l=1 rev1=1 crypto rx=0x7f6b00070310 tx=0x7f6afc01c4c0 comp rx=0 tx=0).stop 2023-05-22T15:42:57.043 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.035+0000 7f6b082e3640 1 -- 172.21.15.16:0/1723551098 shutdown_connections 2023-05-22T15:42:57.043 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.035+0000 7f6b082e3640 1 --2- 172.21.15.16:0/1723551098 >> v2:172.21.15.62:3300/0 conn(0x7f6b000cfc20 0x7f6b000ca8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:57.043 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.035+0000 7f6b082e3640 1 --2- 172.21.15.16:0/1723551098 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6b0015c130 0x7f6b000b2330 unknown :-1 s=CLOSED pgs=1280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:57.043 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.035+0000 7f6b082e3640 1 -- 172.21.15.16:0/1723551098 >> 172.21.15.16:0/1723551098 conn(0x7f6b0006f5c0 msgr2=0x7f6b0006f9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:57.044 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.035+0000 7f6b082e3640 1 -- 172.21.15.16:0/1723551098 shutdown_connections 2023-05-22T15:42:57.044 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.035+0000 7f6b082e3640 1 -- 172.21.15.16:0/1723551098 wait complete. 2023-05-22T15:42:57.046 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.039+0000 7f6b082e3640 1 Processor -- start 2023-05-22T15:42:57.046 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.039+0000 7f6b082e3640 1 -- start start 2023-05-22T15:42:57.046 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.039+0000 7f6b082e3640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6b000b2870 0x7f6b0015b7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:57.046 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.039+0000 7f6b082e3640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6b000cfc20 0x7f6b00152150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:57.046 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.039+0000 7f6b082e3640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6b0015c130 0x7f6b00150530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:57.047 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.039+0000 7f6b082e3640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b0015bd10 con 0x7f6b0015c130 2023-05-22T15:42:57.047 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.039+0000 7f6b082e3640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b000b31c0 con 0x7f6b000b2870 2023-05-22T15:42:57.047 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.039+0000 7f6b082e3640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b00152690 con 0x7f6b000cfc20 2023-05-22T15:42:57.047 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.039+0000 7f6b06880640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6b0015c130 0x7f6b00150530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:57.047 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.039+0000 7f6b06880640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6b0015c130 0x7f6b00150530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60748/0 (socket says 172.21.15.16:60748) 2023-05-22T15:42:57.047 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.039+0000 7f6b06880640 1 -- 172.21.15.16:0/837118031 learned_addr learned my addr 172.21.15.16:0/837118031 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:42:57.048 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.039+0000 7f6b0607f640 1 --2- 172.21.15.16:0/837118031 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6b000b2870 0x7f6b0015b7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:57.048 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.039+0000 7f6b0587e640 1 --2- 172.21.15.16:0/837118031 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6b000cfc20 0x7f6b00152150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:57.048 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.039+0000 7f6b06880640 1 -- 172.21.15.16:0/837118031 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6b000cfc20 msgr2=0x7f6b00152150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:57.048 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.039+0000 7f6b06880640 1 --2- 172.21.15.16:0/837118031 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6b000cfc20 0x7f6b00152150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:57.048 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.039+0000 7f6b06880640 1 -- 172.21.15.16:0/837118031 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6b000b2870 msgr2=0x7f6b0015b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:57.048 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.039+0000 7f6b06880640 1 --2- 172.21.15.16:0/837118031 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6b000b2870 0x7f6b0015b7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:57.049 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.039+0000 7f6b06880640 1 -- 172.21.15.16:0/837118031 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6afc0094d0 con 0x7f6b0015c130 2023-05-22T15:42:57.049 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.039+0000 7f6b06880640 1 --2- 172.21.15.16:0/837118031 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6b0015c130 0x7f6b00150530 secure :-1 s=READY pgs=1281 cs=0 l=1 rev1=1 crypto rx=0x7f6af800cef0 tx=0x7f6af800a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:57.049 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.039+0000 7f6af6ffd640 1 -- 172.21.15.16:0/837118031 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6af800d840 con 0x7f6b0015c130 2023-05-22T15:42:57.049 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.039+0000 7f6af6ffd640 1 -- 172.21.15.16:0/837118031 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6af800d9c0 con 0x7f6b0015c130 2023-05-22T15:42:57.049 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.039+0000 7f6b082e3640 1 -- 172.21.15.16:0/837118031 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b000cb600 con 0x7f6b0015c130 2023-05-22T15:42:57.050 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.039+0000 7f6af6ffd640 1 -- 172.21.15.16:0/837118031 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6af800dd70 con 0x7f6b0015c130 2023-05-22T15:42:57.050 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.039+0000 7f6b082e3640 1 -- 172.21.15.16:0/837118031 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b000ce970 con 0x7f6b0015c130 2023-05-22T15:42:57.050 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.039+0000 7f6b082e3640 1 -- 172.21.15.16:0/837118031 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b000cb790 con 0x7f6b0015c130 2023-05-22T15:42:57.050 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.039+0000 7f6af6ffd640 1 -- 172.21.15.16:0/837118031 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 34) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f6af801e3b0 con 0x7f6b0015c130 2023-05-22T15:42:57.051 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.043+0000 7f6af6ffd640 1 --2- 172.21.15.16:0/837118031 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f6ae40654d0 0x7f6ae4067980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:57.051 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.043+0000 7f6af6ffd640 1 -- 172.21.15.16:0/837118031 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f6af80838a0 con 0x7f6b0015c130 2023-05-22T15:42:57.051 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.043+0000 7f6b0607f640 1 --2- 172.21.15.16:0/837118031 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f6ae40654d0 0x7f6ae4067980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:57.052 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.043+0000 7f6b0607f640 1 --2- 172.21.15.16:0/837118031 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f6ae40654d0 0x7f6ae4067980 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f6afc0ae000 tx=0x7f6afc0a6040 comp rx=0 tx=0).ready entity=mgr.6436 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:57.052 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.043+0000 7f6af6ffd640 1 -- 172.21.15.16:0/837118031 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f6af80514a0 con 0x7f6b0015c130 2023-05-22T15:42:57.202 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.199+0000 7f6b082e3640 1 -- 172.21.15.16:0/837118031 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6b0004dbe0 con 0x7f6b0015c130 2023-05-22T15:42:57.205 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.199+0000 7f6af6ffd640 1 -- 172.21.15.16:0/837118031 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v103) v1 ==== 74+0+18055 (secure 0 0 0) 0x7f6af8054ac0 con 0x7f6b0015c130 2023-05-22T15:42:57.206 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:42:57.206 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":103,"fsid":"b2719834-7847-4d98-90a6-f65d57f85c49","created":"2023-05-22T15:26:25.259314+0000","modified":"2023-05-22T15:42:56.564315+0000","last_up_change":"2023-05-22T15:26:36.448524+0000","last_in_change":"2023-05-22T15:26:27.675668+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-05-22T15:26:38.019781+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-05-22T15:26:43.570514+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2023-05-22T15:42:55.131159+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"103","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2023-05-22T15:42:55.567899+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"103","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9e7ff854-b25f-4cbb-a5c2-1799d9c53d2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":102,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":37021},{"type":"v1","addr":"172.21.15.16:6825","nonce":37021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":37021},{"type":"v1","addr":"172.21.15.16:6827","nonce":37021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":37021},{"type":"v1","addr":"172.21.15.16:6831","nonce":37021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":37021},{"type":"v1","addr":"172.21.15.16:6829","nonce":37021}]},"public_addr":"172.21.15.16:6825/37021","cluster_addr":"172.21.15.16:6827/37021","heartbeat_back_addr":"172.21.15.16:6831/37021","heartbeat_front_addr":"172.21.15.16:6829/37021","state":["exists","up"]},{"osd":1,"uuid":"4702fbc9-a927-4565-aca5-41dc69cc8264","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":102,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":37008},{"type":"v1","addr":"172.21.15.16:6801","nonce":37008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":37008},{"type":"v1","addr":"172.21.15.16:6803","nonce":37008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":37008},{"type":"v1","addr":"172.21.15.16:6807","nonce":37008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":37008},{"type":"v1","addr":"172.21.15.16:6805","nonce":37008}]},"public_addr":"172.21.15.16:6801/37008","cluster_addr":"172.21.15.16:6803/37008","heartbeat_back_addr":"172.21.15.16:6807/37008","heartbeat_front_addr":"172.21.15.16:6805/37008","state":["exists","up"]},{"osd":2,"uuid":"1e7bba46-4eba-47ef-bb81-5e2d371638ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":102,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":37009},{"type":"v1","addr":"172.21.15.16:6809","nonce":37009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":37009},{"type":"v1","addr":"172.21.15.16:6811","nonce":37009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":37009},{"type":"v1","addr":"172.21.15.16:6815","nonce":37009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":37009},{"type":"v1","addr":"172.21.15.16:6813","nonce":37009}]},"public_addr":"172.21.15.16:6809/37009","cluster_addr":"172.21.15.16:6811/37009","heartbeat_back_addr":"172.21.15.16:6815/37009","heartbeat_front_addr":"172.21.15.16:6813/37009","state":["exists","up"]},{"osd":3,"uuid":"383cacc0-5a88-4904-990e-498ca59ef48a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":102,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":37020},{"type":"v1","addr":"172.21.15.16:6817","nonce":37020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":37020},{"type":"v1","addr":"172.21.15.16:6819","nonce":37020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":37020},{"type":"v1","addr":"172.21.15.16:6823","nonce":37020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":37020},{"type":"v1","addr":"172.21.15.16:6821","nonce":37020}]},"public_addr":"172.21.15.16:6817/37020","cluster_addr":"172.21.15.16:6819/37020","heartbeat_back_addr":"172.21.15.16:6823/37020","heartbeat_front_addr":"172.21.15.16:6821/37020","state":["exists","up"]},{"osd":4,"uuid":"970500ef-95b7-42c1-b080-eed4fae7013d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":102,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":37353},{"type":"v1","addr":"172.21.15.62:6817","nonce":37353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6819","nonce":37353},{"type":"v1","addr":"172.21.15.62:6822","nonce":37353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37353},{"type":"v1","addr":"172.21.15.62:6831","nonce":37353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37353},{"type":"v1","addr":"172.21.15.62:6826","nonce":37353}]},"public_addr":"172.21.15.62:6817/37353","cluster_addr":"172.21.15.62:6822/37353","heartbeat_back_addr":"172.21.15.62:6831/37353","heartbeat_front_addr":"172.21.15.62:6826/37353","state":["exists","up"]},{"osd":5,"uuid":"022d67db-d4d8-445b-b1b9-8b446a00f5c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":102,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":37352},{"type":"v1","addr":"172.21.15.62:6801","nonce":37352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":37352},{"type":"v1","addr":"172.21.15.62:6803","nonce":37352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":37352},{"type":"v1","addr":"172.21.15.62:6807","nonce":37352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":37352},{"type":"v1","addr":"172.21.15.62:6805","nonce":37352}]},"public_addr":"172.21.15.62:6801/37352","cluster_addr":"172.21.15.62:6803/37352","heartbeat_back_addr":"172.21.15.62:6807/37352","heartbeat_front_addr":"172.21.15.62:6805/37352","state":["exists","up"]},{"osd":6,"uuid":"42bd372b-6688-4a1b-8407-e16e8c266e33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":102,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":37354},{"type":"v1","addr":"172.21.15.62:6820","nonce":37354}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":37354},{"type":"v1","addr":"172.21.15.62:6823","nonce":37354}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6829","nonce":37354},{"type":"v1","addr":"172.21.15.62:6830","nonce":37354}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6825","nonce":37354},{"type":"v1","addr":"172.21.15.62:6827","nonce":37354}]},"public_addr":"172.21.15.62:6820/37354","cluster_addr":"172.21.15.62:6823/37354","heartbeat_back_addr":"172.21.15.62:6830/37354","heartbeat_front_addr":"172.21.15.62:6827/37354","state":["exists","up"]},{"osd":7,"uuid":"ac70b2a3-f41b-4bf2-b227-7f2cbb55426d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":102,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37355},{"type":"v1","addr":"172.21.15.62:6809","nonce":37355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37355},{"type":"v1","addr":"172.21.15.62:6811","nonce":37355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":37355},{"type":"v1","addr":"172.21.15.62:6815","nonce":37355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":37355},{"type":"v1","addr":"172.21.15.62:6813","nonce":37355}]},"public_addr":"172.21.15.62:6809/37355","cluster_addr":"172.21.15.62:6811/37355","heartbeat_back_addr":"172.21.15.62:6815/37355","heartbeat_front_addr":"172.21.15.62:6813/37355","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.368896+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.107273+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.276888+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.287502+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.124791+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.251937+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.255679+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.232797+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-05-22T15:42:57.208 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.203+0000 7f6b082e3640 1 -- 172.21.15.16:0/837118031 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f6ae40654d0 msgr2=0x7f6ae4067980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:57.208 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.203+0000 7f6b082e3640 1 --2- 172.21.15.16:0/837118031 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f6ae40654d0 0x7f6ae4067980 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f6afc0ae000 tx=0x7f6afc0a6040 comp rx=0 tx=0).stop 2023-05-22T15:42:57.208 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.203+0000 7f6b082e3640 1 -- 172.21.15.16:0/837118031 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6b0015c130 msgr2=0x7f6b00150530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:57.208 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.203+0000 7f6b082e3640 1 --2- 172.21.15.16:0/837118031 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6b0015c130 0x7f6b00150530 secure :-1 s=READY pgs=1281 cs=0 l=1 rev1=1 crypto rx=0x7f6af800cef0 tx=0x7f6af800a710 comp rx=0 tx=0).stop 2023-05-22T15:42:57.208 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.203+0000 7f6b082e3640 1 -- 172.21.15.16:0/837118031 shutdown_connections 2023-05-22T15:42:57.209 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.203+0000 7f6b082e3640 1 --2- 172.21.15.16:0/837118031 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6b0015c130 0x7f6b00150530 unknown :-1 s=CLOSED pgs=1281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:57.209 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.203+0000 7f6b082e3640 1 --2- 172.21.15.16:0/837118031 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f6ae40654d0 0x7f6ae4067980 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:57.209 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.203+0000 7f6b082e3640 1 --2- 172.21.15.16:0/837118031 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6b000cfc20 0x7f6b00152150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:57.209 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.203+0000 7f6b082e3640 1 --2- 172.21.15.16:0/837118031 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6b000b2870 0x7f6b0015b7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:57.209 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.203+0000 7f6b082e3640 1 -- 172.21.15.16:0/837118031 >> 172.21.15.16:0/837118031 conn(0x7f6b0006f5c0 msgr2=0x7f6b000cf400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:57.209 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.203+0000 7f6b082e3640 1 -- 172.21.15.16:0/837118031 shutdown_connections 2023-05-22T15:42:57.209 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.203+0000 7f6b082e3640 1 -- 172.21.15.16:0/837118031 wait complete. 2023-05-22T15:42:57.226 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-05-22T15:42:57.354 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.351+0000 7f18ea752640 1 Processor -- start 2023-05-22T15:42:57.355 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.351+0000 7f18ea752640 1 -- start start 2023-05-22T15:42:57.356 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.351+0000 7f18ea752640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f18e406b6c0 0x7f18e40cb4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:57.356 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.351+0000 7f18ea752640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f18e406ba90 0x7f18e4158fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:57.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.351+0000 7f18ea752640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f18e4159510 0x7f18e40cc740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:57.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.351+0000 7f18ea752640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18e4071760 con 0x7f18e4159510 2023-05-22T15:42:57.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.351+0000 7f18ea752640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18e40ca7d0 con 0x7f18e406b6c0 2023-05-22T15:42:57.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.351+0000 7f18ea752640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18e4159f70 con 0x7f18e406ba90 2023-05-22T15:42:57.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.351+0000 7f18e8cef640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f18e4159510 0x7f18e40cc740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:57.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.351+0000 7f18e8cef640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f18e4159510 0x7f18e40cc740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60754/0 (socket says 172.21.15.16:60754) 2023-05-22T15:42:57.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.351+0000 7f18e8cef640 1 -- 172.21.15.16:0/2747746409 learned_addr learned my addr 172.21.15.16:0/2747746409 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:42:57.358 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.351+0000 7f18e3fff640 1 --2- 172.21.15.16:0/2747746409 >> v2:172.21.15.62:3300/0 conn(0x7f18e406b6c0 0x7f18e40cb4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:57.358 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.351+0000 7f18e37fe640 1 --2- 172.21.15.16:0/2747746409 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f18e406ba90 0x7f18e4158fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:57.358 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.351+0000 7f18e8cef640 1 -- 172.21.15.16:0/2747746409 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f18e406ba90 msgr2=0x7f18e4158fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:42:57.358 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.351+0000 7f18e8cef640 1 --2- 172.21.15.16:0/2747746409 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f18e406ba90 0x7f18e4158fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:57.358 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.351+0000 7f18e8cef640 1 -- 172.21.15.16:0/2747746409 >> v2:172.21.15.62:3300/0 conn(0x7f18e406b6c0 msgr2=0x7f18e40cb4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:42:57.358 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.351+0000 7f18e8cef640 1 --2- 172.21.15.16:0/2747746409 >> v2:172.21.15.62:3300/0 conn(0x7f18e406b6c0 0x7f18e40cb4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:57.359 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.351+0000 7f18e8cef640 1 -- 172.21.15.16:0/2747746409 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18e40cb030 con 0x7f18e4159510 2023-05-22T15:42:57.359 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.351+0000 7f18e3fff640 1 --2- 172.21.15.16:0/2747746409 >> v2:172.21.15.62:3300/0 conn(0x7f18e406b6c0 0x7f18e40cb4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:42:57.359 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.351+0000 7f18e37fe640 1 --2- 172.21.15.16:0/2747746409 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f18e406ba90 0x7f18e4158fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:42:57.362 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.355+0000 7f18e8cef640 1 --2- 172.21.15.16:0/2747746409 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f18e4159510 0x7f18e40cc740 secure :-1 s=READY pgs=1282 cs=0 l=1 rev1=1 crypto rx=0x7f18e40cbe10 tx=0x7f18dc01c210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6e5f50e0ae2cf70 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:57.362 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.355+0000 7f18e2ffd640 1 -- 172.21.15.16:0/2747746409 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18dc0ab600 con 0x7f18e4159510 2023-05-22T15:42:57.363 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.355+0000 7f18e2ffd640 1 -- 172.21.15.16:0/2747746409 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18dc0a7050 con 0x7f18e4159510 2023-05-22T15:42:57.363 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.355+0000 7f18e2ffd640 1 -- 172.21.15.16:0/2747746409 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18dc0ab980 con 0x7f18e4159510 2023-05-22T15:42:57.363 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.355+0000 7f18ea752640 1 -- 172.21.15.16:0/2747746409 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f18e4159510 msgr2=0x7f18e40cc740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:57.363 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.355+0000 7f18ea752640 1 --2- 172.21.15.16:0/2747746409 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f18e4159510 0x7f18e40cc740 secure :-1 s=READY pgs=1282 cs=0 l=1 rev1=1 crypto rx=0x7f18e40cbe10 tx=0x7f18dc01c210 comp rx=0 tx=0).stop 2023-05-22T15:42:57.363 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.355+0000 7f18ea752640 1 -- 172.21.15.16:0/2747746409 shutdown_connections 2023-05-22T15:42:57.364 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.355+0000 7f18ea752640 1 --2- 172.21.15.16:0/2747746409 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f18e4159510 0x7f18e40cc740 unknown :-1 s=CLOSED pgs=1282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:57.364 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.355+0000 7f18ea752640 1 --2- 172.21.15.16:0/2747746409 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f18e406ba90 0x7f18e4158fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:57.365 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.355+0000 7f18ea752640 1 --2- 172.21.15.16:0/2747746409 >> v2:172.21.15.62:3300/0 conn(0x7f18e406b6c0 0x7f18e40cb4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:57.365 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.355+0000 7f18ea752640 1 -- 172.21.15.16:0/2747746409 >> 172.21.15.16:0/2747746409 conn(0x7f18e4069720 msgr2=0x7f18e4069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:57.365 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.355+0000 7f18ea752640 1 -- 172.21.15.16:0/2747746409 shutdown_connections 2023-05-22T15:42:57.365 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.355+0000 7f18ea752640 1 -- 172.21.15.16:0/2747746409 wait complete. 2023-05-22T15:42:57.365 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.355+0000 7f18ea752640 1 Processor -- start 2023-05-22T15:42:57.366 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.355+0000 7f18ea752640 1 -- start start 2023-05-22T15:42:57.366 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.355+0000 7f18ea752640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f18e406b6c0 0x7f18e40c4400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:57.366 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.355+0000 7f18ea752640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f18e406ba90 0x7f18e40c3970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:57.366 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.355+0000 7f18ea752640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f18e4159510 0x7f18e40b2d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:57.366 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.355+0000 7f18ea752640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18e40c3370 con 0x7f18e4159510 2023-05-22T15:42:57.367 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.355+0000 7f18ea752640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18e40b2590 con 0x7f18e406ba90 2023-05-22T15:42:57.367 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.355+0000 7f18ea752640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18e40c4940 con 0x7f18e406b6c0 2023-05-22T15:42:57.367 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.355+0000 7f18e8cef640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f18e4159510 0x7f18e40b2d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:57.367 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.355+0000 7f18e8cef640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f18e4159510 0x7f18e40b2d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60770/0 (socket says 172.21.15.16:60770) 2023-05-22T15:42:57.367 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.355+0000 7f18e8cef640 1 -- 172.21.15.16:0/1790650723 learned_addr learned my addr 172.21.15.16:0/1790650723 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:42:57.367 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.355+0000 7f18e37fe640 1 --2- 172.21.15.16:0/1790650723 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f18e406ba90 0x7f18e40c3970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:57.367 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.355+0000 7f18e3fff640 1 --2- 172.21.15.16:0/1790650723 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f18e406b6c0 0x7f18e40c4400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:57.368 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.355+0000 7f18e8cef640 1 -- 172.21.15.16:0/1790650723 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f18e406b6c0 msgr2=0x7f18e40c4400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:57.368 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.355+0000 7f18e8cef640 1 --2- 172.21.15.16:0/1790650723 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f18e406b6c0 0x7f18e40c4400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:57.368 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.355+0000 7f18e8cef640 1 -- 172.21.15.16:0/1790650723 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f18e406ba90 msgr2=0x7f18e40c3970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:57.368 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.355+0000 7f18e8cef640 1 --2- 172.21.15.16:0/1790650723 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f18e406ba90 0x7f18e40c3970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:57.368 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.355+0000 7f18e8cef640 1 -- 172.21.15.16:0/1790650723 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18dc0094d0 con 0x7f18e4159510 2023-05-22T15:42:57.368 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.355+0000 7f18e3fff640 1 --2- 172.21.15.16:0/1790650723 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f18e406b6c0 0x7f18e40c4400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T15:42:57.369 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.355+0000 7f18e37fe640 1 --2- 172.21.15.16:0/1790650723 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f18e406ba90 0x7f18e40c3970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T15:42:57.369 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.359+0000 7f18e8cef640 1 --2- 172.21.15.16:0/1790650723 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f18e4159510 0x7f18e40b2d50 secure :-1 s=READY pgs=1283 cs=0 l=1 rev1=1 crypto rx=0x7f18dc0ae040 tx=0x7f18dc018fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:57.369 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.359+0000 7f18e0ff9640 1 -- 172.21.15.16:0/1790650723 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18dc0ac230 con 0x7f18e4159510 2023-05-22T15:42:57.369 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.359+0000 7f18e0ff9640 1 -- 172.21.15.16:0/1790650723 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18dc0b4430 con 0x7f18e4159510 2023-05-22T15:42:57.369 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.359+0000 7f18e0ff9640 1 -- 172.21.15.16:0/1790650723 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18dc0b47e0 con 0x7f18e4159510 2023-05-22T15:42:57.369 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.359+0000 7f18ea752640 1 -- 172.21.15.16:0/1790650723 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18e4159990 con 0x7f18e4159510 2023-05-22T15:42:57.370 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.359+0000 7f18ea752640 1 -- 172.21.15.16:0/1790650723 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18e40cacc0 con 0x7f18e4159510 2023-05-22T15:42:57.371 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.359+0000 7f18ea752640 1 -- 172.21.15.16:0/1790650723 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18e406be60 con 0x7f18e4159510 2023-05-22T15:42:57.371 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.359+0000 7f18e0ff9640 1 -- 172.21.15.16:0/1790650723 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 34) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f18dc0b2040 con 0x7f18e4159510 2023-05-22T15:42:57.371 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.359+0000 7f18e0ff9640 1 --2- 172.21.15.16:0/1790650723 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f18d00654a0 0x7f18d0067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:57.371 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.359+0000 7f18e0ff9640 1 -- 172.21.15.16:0/1790650723 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f18dc11cf60 con 0x7f18e4159510 2023-05-22T15:42:57.371 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.359+0000 7f18e3fff640 1 --2- 172.21.15.16:0/1790650723 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f18d00654a0 0x7f18d0067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:57.372 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.359+0000 7f18e3fff640 1 --2- 172.21.15.16:0/1790650723 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f18d00654a0 0x7f18d0067950 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f18d4007130 tx=0x7f18d4007670 comp rx=0 tx=0).ready entity=mgr.6436 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:57.372 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.363+0000 7f18e0ff9640 1 -- 172.21.15.16:0/1790650723 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f18dc0ea880 con 0x7f18e4159510 2023-05-22T15:42:57.551 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.547+0000 7f18ea752640 1 -- 172.21.15.16:0/1790650723 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f18e404dbe0 con 0x7f18e4159510 2023-05-22T15:42:57.577 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.571+0000 7f18e0ff9640 1 -- 172.21.15.16:0/1790650723 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v35) v1 ==== 118+0+0 (secure 0 0 0) 0x7f18dc0b00b0 con 0x7f18e4159510 2023-05-22T15:42:57.580 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.575+0000 7f18ea752640 1 -- 172.21.15.16:0/1790650723 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f18d00654a0 msgr2=0x7f18d0067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:57.580 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.575+0000 7f18ea752640 1 --2- 172.21.15.16:0/1790650723 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f18d00654a0 0x7f18d0067950 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f18d4007130 tx=0x7f18d4007670 comp rx=0 tx=0).stop 2023-05-22T15:42:57.581 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.575+0000 7f18ea752640 1 -- 172.21.15.16:0/1790650723 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f18e4159510 msgr2=0x7f18e40b2d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:57.581 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.575+0000 7f18ea752640 1 --2- 172.21.15.16:0/1790650723 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f18e4159510 0x7f18e40b2d50 secure :-1 s=READY pgs=1283 cs=0 l=1 rev1=1 crypto rx=0x7f18dc0ae040 tx=0x7f18dc018fb0 comp rx=0 tx=0).stop 2023-05-22T15:42:57.581 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.575+0000 7f18ea752640 1 -- 172.21.15.16:0/1790650723 shutdown_connections 2023-05-22T15:42:57.581 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.575+0000 7f18ea752640 1 --2- 172.21.15.16:0/1790650723 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f18e4159510 0x7f18e40b2d50 unknown :-1 s=CLOSED pgs=1283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:57.581 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.575+0000 7f18ea752640 1 --2- 172.21.15.16:0/1790650723 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f18e406ba90 0x7f18e40c3970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:57.581 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.575+0000 7f18ea752640 1 --2- 172.21.15.16:0/1790650723 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f18d00654a0 0x7f18d0067950 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:57.582 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.575+0000 7f18ea752640 1 --2- 172.21.15.16:0/1790650723 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f18e406b6c0 0x7f18e40c4400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:57.582 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.575+0000 7f18ea752640 1 -- 172.21.15.16:0/1790650723 >> 172.21.15.16:0/1790650723 conn(0x7f18e4069720 msgr2=0x7f18e4155860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:57.582 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.575+0000 7f18ea752640 1 -- 172.21.15.16:0/1790650723 shutdown_connections 2023-05-22T15:42:57.582 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.575+0000 7f18ea752640 1 -- 172.21.15.16:0/1790650723 wait complete. 2023-05-22T15:42:57.596 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2023-05-22T15:42:57.596 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-05-22T15:42:57.722 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.715+0000 7f9118776640 1 Processor -- start 2023-05-22T15:42:57.722 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.719+0000 7f9118776640 1 -- start start 2023-05-22T15:42:57.723 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.719+0000 7f9118776640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f911015bca0 0x7f91100cda80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:57.724 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.719+0000 7f9118776640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f91100b5a50 0x7f91100d2900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:57.724 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.719+0000 7f9118776640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91100cccc0 con 0x7f91100d2480 2023-05-22T15:42:57.724 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.719+0000 7f9118776640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91100cd660 con 0x7f911015bca0 2023-05-22T15:42:57.724 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.719+0000 7f9118776640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91100d2e40 con 0x7f91100b5a50 2023-05-22T15:42:57.724 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.719+0000 7f9116512640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f911015bca0 0x7f91100cda80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:57.724 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.719+0000 7f9116512640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f911015bca0 0x7f91100cda80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60784/0 (socket says 172.21.15.16:60784) 2023-05-22T15:42:57.724 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.719+0000 7f9116512640 1 -- 172.21.15.16:0/554655982 learned_addr learned my addr 172.21.15.16:0/554655982 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:42:57.725 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.719+0000 7f9116d13640 1 --2- 172.21.15.16:0/554655982 >> v2:172.21.15.62:3300/0 conn(0x7f91100b5a50 0x7f91100d2900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:57.725 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.719+0000 7f9116512640 1 -- 172.21.15.16:0/554655982 >> v2:172.21.15.62:3300/0 conn(0x7f91100b5a50 msgr2=0x7f91100d2900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:42:57.725 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.719+0000 7f9116512640 1 --2- 172.21.15.16:0/554655982 >> v2:172.21.15.62:3300/0 conn(0x7f91100b5a50 0x7f91100d2900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:57.726 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.719+0000 7f9116512640 1 -- 172.21.15.16:0/554655982 >> v1:172.21.15.62:6789/0 conn(0x7f91100d2480 legacy=0x7f91100b5340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:42:57.726 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.719+0000 7f9116512640 1 -- 172.21.15.16:0/554655982 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91100ccfa0 con 0x7f911015bca0 2023-05-22T15:42:57.726 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.723+0000 7f9116512640 1 --2- 172.21.15.16:0/554655982 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f911015bca0 0x7f91100cda80 secure :-1 s=READY pgs=1284 cs=0 l=1 rev1=1 crypto rx=0x7f91100c5fc0 tx=0x7f910c01c4c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=425078d5d91978d5 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:57.727 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.723+0000 7f9115510640 1 -- 172.21.15.16:0/554655982 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f910c0aa640 con 0x7f911015bca0 2023-05-22T15:42:57.727 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.723+0000 7f9115510640 1 -- 172.21.15.16:0/554655982 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f910c0aa7a0 con 0x7f911015bca0 2023-05-22T15:42:57.727 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.723+0000 7f9115510640 1 -- 172.21.15.16:0/554655982 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f910c0aab30 con 0x7f911015bca0 2023-05-22T15:42:57.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.723+0000 7f9118776640 1 -- 172.21.15.16:0/554655982 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f911015bca0 msgr2=0x7f91100cda80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:57.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.723+0000 7f9118776640 1 --2- 172.21.15.16:0/554655982 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f911015bca0 0x7f91100cda80 secure :-1 s=READY pgs=1284 cs=0 l=1 rev1=1 crypto rx=0x7f91100c5fc0 tx=0x7f910c01c4c0 comp rx=0 tx=0).stop 2023-05-22T15:42:57.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.723+0000 7f9118776640 1 -- 172.21.15.16:0/554655982 shutdown_connections 2023-05-22T15:42:57.729 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.723+0000 7f9118776640 1 --2- 172.21.15.16:0/554655982 >> v2:172.21.15.62:3300/0 conn(0x7f91100b5a50 0x7f91100d2900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:57.729 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.723+0000 7f9118776640 1 --2- 172.21.15.16:0/554655982 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f911015bca0 0x7f91100cda80 unknown :-1 s=CLOSED pgs=1284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:57.729 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.723+0000 7f9118776640 1 -- 172.21.15.16:0/554655982 >> 172.21.15.16:0/554655982 conn(0x7f91100655c0 msgr2=0x7f9110070e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:57.729 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.723+0000 7f9118776640 1 -- 172.21.15.16:0/554655982 shutdown_connections 2023-05-22T15:42:57.729 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.723+0000 7f9118776640 1 -- 172.21.15.16:0/554655982 wait complete. 2023-05-22T15:42:57.731 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.723+0000 7f9118776640 1 Processor -- start 2023-05-22T15:42:57.731 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.723+0000 7f9118776640 1 -- start start 2023-05-22T15:42:57.731 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.727+0000 7f9118776640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f91100b5a50 0x7f91101545a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:57.731 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.727+0000 7f9118776640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f91100d2480 0x7f9110156f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:57.731 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.727+0000 7f9118776640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f911015bca0 0x7f911015e1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:57.731 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.727+0000 7f9118776640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9110149c70 con 0x7f91100b5a50 2023-05-22T15:42:57.732 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.727+0000 7f9118776640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f911014a650 con 0x7f91100d2480 2023-05-22T15:42:57.732 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.727+0000 7f9118776640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9110148dc0 con 0x7f911015bca0 2023-05-22T15:42:57.732 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.727+0000 7f9116512640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f91100b5a50 0x7f91101545a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:57.732 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.727+0000 7f9116512640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f91100b5a50 0x7f91101545a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60788/0 (socket says 172.21.15.16:60788) 2023-05-22T15:42:57.732 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.727+0000 7f9116512640 1 -- 172.21.15.16:0/3693020297 learned_addr learned my addr 172.21.15.16:0/3693020297 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:42:57.732 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.727+0000 7f9115d11640 1 --2- 172.21.15.16:0/3693020297 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f91100d2480 0x7f9110156f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:57.733 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.727+0000 7f9116d13640 1 --2- 172.21.15.16:0/3693020297 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f911015bca0 0x7f911015e1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:57.733 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.727+0000 7f9116512640 1 -- 172.21.15.16:0/3693020297 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f911015bca0 msgr2=0x7f911015e1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:57.733 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.727+0000 7f9116512640 1 --2- 172.21.15.16:0/3693020297 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f911015bca0 0x7f911015e1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:57.733 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.727+0000 7f9116512640 1 -- 172.21.15.16:0/3693020297 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f91100d2480 msgr2=0x7f9110156f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:57.733 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.727+0000 7f9116512640 1 --2- 172.21.15.16:0/3693020297 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f91100d2480 0x7f9110156f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:57.733 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.727+0000 7f9116512640 1 -- 172.21.15.16:0/3693020297 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f910c0094d0 con 0x7f91100b5a50 2023-05-22T15:42:57.734 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.727+0000 7f9116d13640 1 --2- 172.21.15.16:0/3693020297 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f911015bca0 0x7f911015e1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:42:57.734 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.727+0000 7f9115d11640 1 --2- 172.21.15.16:0/3693020297 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f91100d2480 0x7f9110156f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T15:42:57.734 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.727+0000 7f9116512640 1 --2- 172.21.15.16:0/3693020297 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f91100b5a50 0x7f91101545a0 secure :-1 s=READY pgs=1285 cs=0 l=1 rev1=1 crypto rx=0x7f910c0ac040 tx=0x7f910c01ad00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:57.735 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.727+0000 7f9106ffd640 1 -- 172.21.15.16:0/3693020297 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f910c0bd5f0 con 0x7f91100b5a50 2023-05-22T15:42:57.735 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.727+0000 7f9106ffd640 1 -- 172.21.15.16:0/3693020297 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f910c0bd750 con 0x7f91100b5a50 2023-05-22T15:42:57.735 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.727+0000 7f9118776640 1 -- 172.21.15.16:0/3693020297 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9110160360 con 0x7f91100b5a50 2023-05-22T15:42:57.735 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.727+0000 7f9106ffd640 1 -- 172.21.15.16:0/3693020297 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f910c0bdae0 con 0x7f91100b5a50 2023-05-22T15:42:57.735 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.727+0000 7f9118776640 1 -- 172.21.15.16:0/3693020297 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91100cdfc0 con 0x7f91100b5a50 2023-05-22T15:42:57.736 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.727+0000 7f9118776640 1 -- 172.21.15.16:0/3693020297 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91101604f0 con 0x7f91100b5a50 2023-05-22T15:42:57.736 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.727+0000 7f9106ffd640 1 -- 172.21.15.16:0/3693020297 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 34) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f910c0a8050 con 0x7f91100b5a50 2023-05-22T15:42:57.736 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.727+0000 7f9106ffd640 1 --2- 172.21.15.16:0/3693020297 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f90f40654a0 0x7f90f4067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:57.736 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.727+0000 7f9115d11640 1 --2- 172.21.15.16:0/3693020297 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f90f40654a0 0x7f90f4067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:57.736 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.727+0000 7f9106ffd640 1 -- 172.21.15.16:0/3693020297 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f910c11cdc0 con 0x7f91100b5a50 2023-05-22T15:42:57.737 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.727+0000 7f9115d11640 1 --2- 172.21.15.16:0/3693020297 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f90f40654a0 0x7f90f4067950 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f911014e8d0 tx=0x7f9100009040 comp rx=0 tx=0).ready entity=mgr.6436 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:57.737 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.731+0000 7f9106ffd640 1 -- 172.21.15.16:0/3693020297 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f910c0ea730 con 0x7f91100b5a50 2023-05-22T15:42:57.918 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:57.915+0000 7f9118776640 1 -- 172.21.15.16:0/3693020297 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f911004dbe0 con 0x7f91100b5a50 2023-05-22T15:42:58.039 INFO:tasks.ceph.mds.a.smithi016.stderr:2023-05-22T15:42:58.035+0000 7f2d6f035640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-05-22T15:42:58.589 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:58.583+0000 7f9106ffd640 1 -- 172.21.15.16:0/3693020297 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v36) v1 ==== 122+0+0 (secure 0 0 0) 0x7f910c0edd50 con 0x7f91100b5a50 2023-05-22T15:42:58.592 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:58.587+0000 7f9118776640 1 -- 172.21.15.16:0/3693020297 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f90f40654a0 msgr2=0x7f90f4067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:58.593 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:58.587+0000 7f9118776640 1 --2- 172.21.15.16:0/3693020297 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f90f40654a0 0x7f90f4067950 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f911014e8d0 tx=0x7f9100009040 comp rx=0 tx=0).stop 2023-05-22T15:42:58.593 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:58.587+0000 7f9118776640 1 -- 172.21.15.16:0/3693020297 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f91100b5a50 msgr2=0x7f91101545a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:58.593 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:58.587+0000 7f9118776640 1 --2- 172.21.15.16:0/3693020297 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f91100b5a50 0x7f91101545a0 secure :-1 s=READY pgs=1285 cs=0 l=1 rev1=1 crypto rx=0x7f910c0ac040 tx=0x7f910c01ad00 comp rx=0 tx=0).stop 2023-05-22T15:42:58.593 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:58.587+0000 7f9118776640 1 -- 172.21.15.16:0/3693020297 shutdown_connections 2023-05-22T15:42:58.593 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:58.587+0000 7f9118776640 1 --2- 172.21.15.16:0/3693020297 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f90f40654a0 0x7f90f4067950 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:58.593 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:58.587+0000 7f9118776640 1 --2- 172.21.15.16:0/3693020297 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f911015bca0 0x7f911015e1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:58.593 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:58.587+0000 7f9118776640 1 --2- 172.21.15.16:0/3693020297 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f91100d2480 0x7f9110156f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:58.594 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:58.587+0000 7f9118776640 1 --2- 172.21.15.16:0/3693020297 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f91100b5a50 0x7f91101545a0 unknown :-1 s=CLOSED pgs=1285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:58.594 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:58.587+0000 7f9118776640 1 -- 172.21.15.16:0/3693020297 >> 172.21.15.16:0/3693020297 conn(0x7f91100655c0 msgr2=0x7f91101581f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:58.594 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:58.587+0000 7f9118776640 1 -- 172.21.15.16:0/3693020297 shutdown_connections 2023-05-22T15:42:58.594 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:58.587+0000 7f9118776640 1 -- 172.21.15.16:0/3693020297 wait complete. 2023-05-22T15:42:58.614 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2023-05-22T15:42:58.615 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-05-22T15:42:58.742 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:58.735+0000 7fca6a2b6640 1 Processor -- start 2023-05-22T15:42:58.742 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:58.735+0000 7fca6a2b6640 1 -- start start 2023-05-22T15:42:58.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:58.739+0000 7fca6a2b6640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fca640b0250 0x7fca64159330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:58.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:58.739+0000 7fca6a2b6640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fca64159870 0x7fca640c8ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:58.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:58.739+0000 7fca6a2b6640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca6415a0a0 con 0x7fca641531b0 2023-05-22T15:42:58.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:58.739+0000 7fca6a2b6640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca64153580 con 0x7fca64159870 2023-05-22T15:42:58.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:58.739+0000 7fca6a2b6640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca6415d5e0 con 0x7fca640b0250 2023-05-22T15:42:58.744 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:58.739+0000 7fca637fe640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fca640b0250 0x7fca64159330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:58.744 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:58.739+0000 7fca637fe640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fca640b0250 0x7fca64159330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3301/0 says I am v2:172.21.15.16:40218/0 (socket says 172.21.15.16:40218) 2023-05-22T15:42:58.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:58.739+0000 7fca637fe640 1 -- 172.21.15.16:0/4208471096 learned_addr learned my addr 172.21.15.16:0/4208471096 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:42:58.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:58.739+0000 7fca68853640 1 --2- 172.21.15.16:0/4208471096 >> v2:172.21.15.62:3300/0 conn(0x7fca64159870 0x7fca640c8ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:58.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:58.739+0000 7fca637fe640 1 -- 172.21.15.16:0/4208471096 >> v2:172.21.15.62:3300/0 conn(0x7fca64159870 msgr2=0x7fca640c8ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:42:58.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:58.739+0000 7fca637fe640 1 --2- 172.21.15.16:0/4208471096 >> v2:172.21.15.62:3300/0 conn(0x7fca64159870 0x7fca640c8ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:58.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:58.739+0000 7fca637fe640 1 -- 172.21.15.16:0/4208471096 >> v1:172.21.15.62:6789/0 conn(0x7fca641531b0 legacy=0x7fca640afb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:58.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:58.739+0000 7fca637fe640 1 -- 172.21.15.16:0/4208471096 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca6415c820 con 0x7fca640b0250 2023-05-22T15:42:58.746 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:58.739+0000 7fca68853640 1 --2- 172.21.15.16:0/4208471096 >> v2:172.21.15.62:3300/0 conn(0x7fca64159870 0x7fca640c8ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-05-22T15:42:58.746 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:58.743+0000 7fca637fe640 1 --2- 172.21.15.16:0/4208471096 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fca640b0250 0x7fca64159330 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fca58009870 tx=0x7fca5801c510 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7b52f32698690505 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:58.747 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:58.743+0000 7fca62ffd640 1 -- 172.21.15.16:0/4208471096 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca580aa670 con 0x7fca640b0250 2023-05-22T15:42:58.747 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:58.743+0000 7fca62ffd640 1 -- 172.21.15.16:0/4208471096 <== mon.2 v2:172.21.15.62:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca580aa7d0 con 0x7fca640b0250 2023-05-22T15:42:58.747 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:58.743+0000 7fca62ffd640 1 -- 172.21.15.16:0/4208471096 <== mon.2 v2:172.21.15.62:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca580aab60 con 0x7fca640b0250 2023-05-22T15:42:58.748 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:58.743+0000 7fca6a2b6640 1 -- 172.21.15.16:0/4208471096 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fca640b0250 msgr2=0x7fca64159330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:58.748 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:58.743+0000 7fca6a2b6640 1 --2- 172.21.15.16:0/4208471096 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fca640b0250 0x7fca64159330 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fca58009870 tx=0x7fca5801c510 comp rx=0 tx=0).stop 2023-05-22T15:42:58.748 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:58.743+0000 7fca6a2b6640 1 -- 172.21.15.16:0/4208471096 shutdown_connections 2023-05-22T15:42:58.748 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:58.743+0000 7fca6a2b6640 1 --2- 172.21.15.16:0/4208471096 >> v2:172.21.15.62:3300/0 conn(0x7fca64159870 0x7fca640c8ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:58.748 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:58.743+0000 7fca6a2b6640 1 --2- 172.21.15.16:0/4208471096 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fca640b0250 0x7fca64159330 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:58.749 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:58.743+0000 7fca6a2b6640 1 -- 172.21.15.16:0/4208471096 >> 172.21.15.16:0/4208471096 conn(0x7fca64069720 msgr2=0x7fca64069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:58.749 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:58.743+0000 7fca6a2b6640 1 -- 172.21.15.16:0/4208471096 shutdown_connections 2023-05-22T15:42:58.749 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:58.743+0000 7fca6a2b6640 1 -- 172.21.15.16:0/4208471096 wait complete. 2023-05-22T15:42:58.750 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:58.743+0000 7fca6a2b6640 1 Processor -- start 2023-05-22T15:42:58.751 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:58.743+0000 7fca6a2b6640 1 -- start start 2023-05-22T15:42:58.751 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:58.743+0000 7fca6a2b6640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fca640b0250 0x7fca640b1970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:58.751 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:58.743+0000 7fca6a2b6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fca641531b0 0x7fca64158a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:58.751 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:58.743+0000 7fca6a2b6640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fca64159870 0x7fca640b31b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:58.751 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:58.743+0000 7fca6a2b6640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca640b3e00 con 0x7fca641531b0 2023-05-22T15:42:58.751 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:58.743+0000 7fca6a2b6640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca640b3b20 con 0x7fca64159870 2023-05-22T15:42:58.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:58.743+0000 7fca6a2b6640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca6414d0b0 con 0x7fca640b0250 2023-05-22T15:42:58.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:58.743+0000 7fca637fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fca641531b0 0x7fca64158a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:58.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:58.743+0000 7fca637fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fca641531b0 0x7fca64158a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60794/0 (socket says 172.21.15.16:60794) 2023-05-22T15:42:58.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:58.743+0000 7fca637fe640 1 -- 172.21.15.16:0/900103442 learned_addr learned my addr 172.21.15.16:0/900103442 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:42:58.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:58.743+0000 7fca68853640 1 --2- 172.21.15.16:0/900103442 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fca64159870 0x7fca640b31b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:58.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:58.743+0000 7fca63fff640 1 --2- 172.21.15.16:0/900103442 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fca640b0250 0x7fca640b1970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:58.753 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:58.743+0000 7fca637fe640 1 -- 172.21.15.16:0/900103442 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fca640b0250 msgr2=0x7fca640b1970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:58.753 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:58.743+0000 7fca637fe640 1 --2- 172.21.15.16:0/900103442 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fca640b0250 0x7fca640b1970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:58.753 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:58.743+0000 7fca637fe640 1 -- 172.21.15.16:0/900103442 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fca64159870 msgr2=0x7fca640b31b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:58.753 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:58.743+0000 7fca637fe640 1 --2- 172.21.15.16:0/900103442 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fca64159870 0x7fca640b31b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:58.753 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:58.743+0000 7fca637fe640 1 -- 172.21.15.16:0/900103442 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca5c009a70 con 0x7fca641531b0 2023-05-22T15:42:58.754 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:58.743+0000 7fca637fe640 1 --2- 172.21.15.16:0/900103442 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fca641531b0 0x7fca64158a00 secure :-1 s=READY pgs=1286 cs=0 l=1 rev1=1 crypto rx=0x7fca580ac040 tx=0x7fca5801ad50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:58.754 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:58.747+0000 7fca60ff9640 1 -- 172.21.15.16:0/900103442 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca580b3800 con 0x7fca641531b0 2023-05-22T15:42:58.754 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:58.747+0000 7fca60ff9640 1 -- 172.21.15.16:0/900103442 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca580b3980 con 0x7fca641531b0 2023-05-22T15:42:58.754 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:58.747+0000 7fca6a2b6640 1 -- 172.21.15.16:0/900103442 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca580094d0 con 0x7fca641531b0 2023-05-22T15:42:58.754 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:58.747+0000 7fca60ff9640 1 -- 172.21.15.16:0/900103442 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca580b3d30 con 0x7fca641531b0 2023-05-22T15:42:58.754 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:58.747+0000 7fca6a2b6640 1 -- 172.21.15.16:0/900103442 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca64159c40 con 0x7fca641531b0 2023-05-22T15:42:58.755 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:58.747+0000 7fca6a2b6640 1 -- 172.21.15.16:0/900103442 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca6415d0c0 con 0x7fca641531b0 2023-05-22T15:42:58.755 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:58.747+0000 7fca60ff9640 1 -- 172.21.15.16:0/900103442 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 34) v1 ==== 84332+0+0 (secure 0 0 0) 0x7fca580c7420 con 0x7fca641531b0 2023-05-22T15:42:58.756 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:58.747+0000 7fca60ff9640 1 --2- 172.21.15.16:0/900103442 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fca500654a0 0x7fca50067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:58.756 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:58.747+0000 7fca60ff9640 1 -- 172.21.15.16:0/900103442 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 6382+0+0 (secure 0 0 0) 0x7fca5811d960 con 0x7fca641531b0 2023-05-22T15:42:58.756 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:58.747+0000 7fca63fff640 1 --2- 172.21.15.16:0/900103442 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fca500654a0 0x7fca50067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:58.756 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:58.747+0000 7fca63fff640 1 --2- 172.21.15.16:0/900103442 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fca500654a0 0x7fca50067950 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fca54007fd0 tx=0x7fca54009040 comp rx=0 tx=0).ready entity=mgr.6436 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:58.756 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:58.751+0000 7fca60ff9640 1 -- 172.21.15.16:0/900103442 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fca580eb290 con 0x7fca641531b0 2023-05-22T15:42:58.934 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:58.931+0000 7fca6a2b6640 1 -- 172.21.15.16:0/900103442 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fca64159dd0 con 0x7fca641531b0 2023-05-22T15:42:58.935 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:58.931+0000 7fca60ff9640 1 -- 172.21.15.16:0/900103442 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) v1 ==== 94+0+4650 (secure 0 0 0) 0x7fca580ee8b0 con 0x7fca641531b0 2023-05-22T15:42:58.936 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:42:58.936 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":36,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6059,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6833/3017549713","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":3017549713},{"type":"v1","addr":"172.21.15.62:6833","nonce":3017549713}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":33},{"gid":6607,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/3268651705","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":3268651705},{"type":"v1","addr":"172.21.15.62:6835","nonce":3268651705}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":33},{"gid":7524,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6836/2104991509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2104991509},{"type":"v1","addr":"172.21.15.16:6836","nonce":2104991509}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":33}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-05-22T15:42:56.565074+0000","modified":"2023-05-22T15:42:58.580533+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7527},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7527":{"gid":7527,"name":"a","rank":0,"incarnation":34,"state":"up:active","state_seq":3,"addr":"172.21.15.16:6837/1353635569","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6835","nonce":1353635569},{"type":"v1","addr":"172.21.15.16:6837","nonce":1353635569}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2023-05-22T15:42:58.936 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 36 2023-05-22T15:42:58.937 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:58.931+0000 7fca6a2b6640 1 -- 172.21.15.16:0/900103442 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fca500654a0 msgr2=0x7fca50067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:58.938 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:58.931+0000 7fca6a2b6640 1 --2- 172.21.15.16:0/900103442 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fca500654a0 0x7fca50067950 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fca54007fd0 tx=0x7fca54009040 comp rx=0 tx=0).stop 2023-05-22T15:42:58.938 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:58.931+0000 7fca6a2b6640 1 -- 172.21.15.16:0/900103442 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fca641531b0 msgr2=0x7fca64158a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:58.938 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:58.931+0000 7fca6a2b6640 1 --2- 172.21.15.16:0/900103442 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fca641531b0 0x7fca64158a00 secure :-1 s=READY pgs=1286 cs=0 l=1 rev1=1 crypto rx=0x7fca580ac040 tx=0x7fca5801ad50 comp rx=0 tx=0).stop 2023-05-22T15:42:58.938 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:58.931+0000 7fca6a2b6640 1 -- 172.21.15.16:0/900103442 shutdown_connections 2023-05-22T15:42:58.938 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:58.931+0000 7fca6a2b6640 1 --2- 172.21.15.16:0/900103442 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fca64159870 0x7fca640b31b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:58.938 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:58.931+0000 7fca6a2b6640 1 --2- 172.21.15.16:0/900103442 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fca641531b0 0x7fca64158a00 unknown :-1 s=CLOSED pgs=1286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:58.938 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:58.931+0000 7fca6a2b6640 1 --2- 172.21.15.16:0/900103442 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fca500654a0 0x7fca50067950 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:58.939 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:58.931+0000 7fca6a2b6640 1 --2- 172.21.15.16:0/900103442 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fca640b0250 0x7fca640b1970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:58.939 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:58.931+0000 7fca6a2b6640 1 -- 172.21.15.16:0/900103442 >> 172.21.15.16:0/900103442 conn(0x7fca64069720 msgr2=0x7fca640ce3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:58.939 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:58.931+0000 7fca6a2b6640 1 -- 172.21.15.16:0/900103442 shutdown_connections 2023-05-22T15:42:58.939 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:58.931+0000 7fca6a2b6640 1 -- 172.21.15.16:0/900103442 wait complete. 2023-05-22T15:42:58.954 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-05-22T15:42:59.072 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.067+0000 7f2d149cf640 1 Processor -- start 2023-05-22T15:42:59.072 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.067+0000 7f2d149cf640 1 -- start start 2023-05-22T15:42:59.073 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.067+0000 7f2d149cf640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2d101531b0 0x7f2d100afb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:59.073 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.067+0000 7f2d149cf640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f2d100b0080 0x7f2d10159330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:59.073 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.067+0000 7f2d149cf640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d1015a0a0 con 0x7f2d10159870 2023-05-22T15:42:59.073 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.067+0000 7f2d149cf640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d100b04f0 con 0x7f2d100b0080 2023-05-22T15:42:59.074 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.067+0000 7f2d149cf640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d10153580 con 0x7f2d101531b0 2023-05-22T15:42:59.074 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.067+0000 7f2d0e59c640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2d101531b0 0x7f2d100afb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:59.074 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.067+0000 7f2d07fff640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f2d100b0080 0x7f2d10159330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:59.074 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.067+0000 7f2d0ed9d640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f2d10159870 0x7f2d100c8ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.16:37312/0 (socket says 172.21.15.16:37312) 2023-05-22T15:42:59.074 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.067+0000 7f2d0ed9d640 1 -- 172.21.15.16:0/2667967137 learned_addr learned my addr 172.21.15.16:0/2667967137 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:42:59.074 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.067+0000 7f2d0e59c640 1 -- 172.21.15.16:0/2667967137 >> v2:172.21.15.62:3300/0 conn(0x7f2d100b0080 msgr2=0x7f2d10159330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:42:59.075 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.067+0000 7f2d0e59c640 1 --2- 172.21.15.16:0/2667967137 >> v2:172.21.15.62:3300/0 conn(0x7f2d100b0080 0x7f2d10159330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:59.075 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.067+0000 7f2d0e59c640 1 -- 172.21.15.16:0/2667967137 >> v1:172.21.15.62:6789/0 conn(0x7f2d10159870 legacy=0x7f2d100c8ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:59.075 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.067+0000 7f2d0e59c640 1 -- 172.21.15.16:0/2667967137 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d1015d5e0 con 0x7f2d101531b0 2023-05-22T15:42:59.075 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.067+0000 7f2d07fff640 1 --2- 172.21.15.16:0/2667967137 >> v2:172.21.15.62:3300/0 conn(0x7f2d100b0080 0x7f2d10159330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-05-22T15:42:59.077 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.071+0000 7f2d0e59c640 1 --2- 172.21.15.16:0/2667967137 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2d101531b0 0x7f2d100afb40 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f2d000097c0 tx=0x7f2d0001c450 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aabe3ed73528eb7a server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:59.077 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.071+0000 7f2d0dd9b640 1 -- 172.21.15.16:0/2667967137 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d000ab650 con 0x7f2d101531b0 2023-05-22T15:42:59.077 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.071+0000 7f2d0dd9b640 1 -- 172.21.15.16:0/2667967137 <== mon.2 v2:172.21.15.62:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d000ab7b0 con 0x7f2d101531b0 2023-05-22T15:42:59.078 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.071+0000 7f2d0dd9b640 1 -- 172.21.15.16:0/2667967137 <== mon.2 v2:172.21.15.62:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d000abb40 con 0x7f2d101531b0 2023-05-22T15:42:59.078 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.071+0000 7f2d149cf640 1 -- 172.21.15.16:0/2667967137 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2d101531b0 msgr2=0x7f2d100afb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:59.078 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.071+0000 7f2d149cf640 1 --2- 172.21.15.16:0/2667967137 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2d101531b0 0x7f2d100afb40 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f2d000097c0 tx=0x7f2d0001c450 comp rx=0 tx=0).stop 2023-05-22T15:42:59.078 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.071+0000 7f2d149cf640 1 -- 172.21.15.16:0/2667967137 shutdown_connections 2023-05-22T15:42:59.079 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.071+0000 7f2d149cf640 1 --2- 172.21.15.16:0/2667967137 >> v2:172.21.15.62:3300/0 conn(0x7f2d100b0080 0x7f2d10159330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:59.079 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.071+0000 7f2d149cf640 1 --2- 172.21.15.16:0/2667967137 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2d101531b0 0x7f2d100afb40 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:59.079 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.071+0000 7f2d149cf640 1 -- 172.21.15.16:0/2667967137 >> 172.21.15.16:0/2667967137 conn(0x7f2d10069720 msgr2=0x7f2d10069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:59.080 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.071+0000 7f2d149cf640 1 -- 172.21.15.16:0/2667967137 shutdown_connections 2023-05-22T15:42:59.080 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.071+0000 7f2d149cf640 1 -- 172.21.15.16:0/2667967137 wait complete. 2023-05-22T15:42:59.083 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.071+0000 7f2d149cf640 1 Processor -- start 2023-05-22T15:42:59.083 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.071+0000 7f2d149cf640 1 -- start start 2023-05-22T15:42:59.084 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.075+0000 7f2d149cf640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2d100b0080 0x7f2d1014a470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:59.084 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.075+0000 7f2d149cf640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2d101531b0 0x7f2d1015a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:59.084 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.075+0000 7f2d149cf640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2d10159870 0x7f2d1014f790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:59.084 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.075+0000 7f2d149cf640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d100b1a60 con 0x7f2d10159870 2023-05-22T15:42:59.084 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.075+0000 7f2d149cf640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d100b1d40 con 0x7f2d100b0080 2023-05-22T15:42:59.084 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.075+0000 7f2d149cf640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d100c1b30 con 0x7f2d101531b0 2023-05-22T15:42:59.085 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.075+0000 7f2d0ed9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2d10159870 0x7f2d1014f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:59.085 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.075+0000 7f2d0ed9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2d10159870 0x7f2d1014f790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60796/0 (socket says 172.21.15.16:60796) 2023-05-22T15:42:59.085 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.075+0000 7f2d0ed9d640 1 -- 172.21.15.16:0/1294056754 learned_addr learned my addr 172.21.15.16:0/1294056754 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:42:59.085 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.075+0000 7f2d0e59c640 1 --2- 172.21.15.16:0/1294056754 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2d100b0080 0x7f2d1014a470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:59.085 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.075+0000 7f2d07fff640 1 --2- 172.21.15.16:0/1294056754 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2d101531b0 0x7f2d1015a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:59.085 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.075+0000 7f2d0ed9d640 1 -- 172.21.15.16:0/1294056754 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2d101531b0 msgr2=0x7f2d1015a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:59.086 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.075+0000 7f2d0ed9d640 1 --2- 172.21.15.16:0/1294056754 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2d101531b0 0x7f2d1015a7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:59.086 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.075+0000 7f2d0ed9d640 1 -- 172.21.15.16:0/1294056754 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2d100b0080 msgr2=0x7f2d1014a470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:59.086 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.075+0000 7f2d0ed9d640 1 --2- 172.21.15.16:0/1294056754 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2d100b0080 0x7f2d1014a470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:59.086 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.075+0000 7f2d0ed9d640 1 -- 172.21.15.16:0/1294056754 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cf8009420 con 0x7f2d10159870 2023-05-22T15:42:59.086 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.075+0000 7f2d0ed9d640 1 --2- 172.21.15.16:0/1294056754 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2d10159870 0x7f2d1014f790 secure :-1 s=READY pgs=1287 cs=0 l=1 rev1=1 crypto rx=0x7f2d0800ca80 tx=0x7f2d0800cf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:59.086 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.075+0000 7f2d06ffd640 1 -- 172.21.15.16:0/1294056754 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d0800ac30 con 0x7f2d10159870 2023-05-22T15:42:59.087 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.075+0000 7f2d06ffd640 1 -- 172.21.15.16:0/1294056754 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d0800ad90 con 0x7f2d10159870 2023-05-22T15:42:59.087 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.075+0000 7f2d06ffd640 1 -- 172.21.15.16:0/1294056754 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d08012600 con 0x7f2d10159870 2023-05-22T15:42:59.087 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.075+0000 7f2d149cf640 1 -- 172.21.15.16:0/1294056754 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d00009420 con 0x7f2d10159870 2023-05-22T15:42:59.087 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.075+0000 7f2d149cf640 1 -- 172.21.15.16:0/1294056754 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d1015d0c0 con 0x7f2d10159870 2023-05-22T15:42:59.087 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.075+0000 7f2d149cf640 1 -- 172.21.15.16:0/1294056754 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d1015cde0 con 0x7f2d10159870 2023-05-22T15:42:59.087 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.075+0000 7f2d06ffd640 1 -- 172.21.15.16:0/1294056754 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 34) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f2d08007070 con 0x7f2d10159870 2023-05-22T15:42:59.087 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.075+0000 7f2d06ffd640 1 --2- 172.21.15.16:0/1294056754 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f2cf00654a0 0x7f2cf0067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:59.088 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.075+0000 7f2d06ffd640 1 -- 172.21.15.16:0/1294056754 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f2d08082900 con 0x7f2d10159870 2023-05-22T15:42:59.088 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.075+0000 7f2d0e59c640 1 --2- 172.21.15.16:0/1294056754 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f2cf00654a0 0x7f2cf0067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:59.088 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.079+0000 7f2d0e59c640 1 --2- 172.21.15.16:0/1294056754 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f2cf00654a0 0x7f2cf0067950 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f2d000af000 tx=0x7f2d000a7000 comp rx=0 tx=0).ready entity=mgr.6436 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:59.089 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.079+0000 7f2d06ffd640 1 -- 172.21.15.16:0/1294056754 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f2d0804a620 con 0x7f2d10159870 2023-05-22T15:42:59.238 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.231+0000 7f2d149cf640 1 -- 172.21.15.16:0/1294056754 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2d10159c70 con 0x7f2d10159870 2023-05-22T15:42:59.239 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.235+0000 7f2d06ffd640 1 -- 172.21.15.16:0/1294056754 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v104) v1 ==== 74+0+18042 (secure 0 0 0) 0x7f2d08050380 con 0x7f2d10159870 2023-05-22T15:42:59.240 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:42:59.240 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":104,"fsid":"b2719834-7847-4d98-90a6-f65d57f85c49","created":"2023-05-22T15:26:25.259314+0000","modified":"2023-05-22T15:42:57.569291+0000","last_up_change":"2023-05-22T15:26:36.448524+0000","last_in_change":"2023-05-22T15:26:27.675668+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-05-22T15:26:38.019781+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-05-22T15:26:43.570514+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2023-05-22T15:42:55.131159+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"103","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2023-05-22T15:42:55.567899+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"104","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9e7ff854-b25f-4cbb-a5c2-1799d9c53d2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":102,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":37021},{"type":"v1","addr":"172.21.15.16:6825","nonce":37021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":37021},{"type":"v1","addr":"172.21.15.16:6827","nonce":37021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":37021},{"type":"v1","addr":"172.21.15.16:6831","nonce":37021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":37021},{"type":"v1","addr":"172.21.15.16:6829","nonce":37021}]},"public_addr":"172.21.15.16:6825/37021","cluster_addr":"172.21.15.16:6827/37021","heartbeat_back_addr":"172.21.15.16:6831/37021","heartbeat_front_addr":"172.21.15.16:6829/37021","state":["exists","up"]},{"osd":1,"uuid":"4702fbc9-a927-4565-aca5-41dc69cc8264","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":102,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":37008},{"type":"v1","addr":"172.21.15.16:6801","nonce":37008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":37008},{"type":"v1","addr":"172.21.15.16:6803","nonce":37008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":37008},{"type":"v1","addr":"172.21.15.16:6807","nonce":37008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":37008},{"type":"v1","addr":"172.21.15.16:6805","nonce":37008}]},"public_addr":"172.21.15.16:6801/37008","cluster_addr":"172.21.15.16:6803/37008","heartbeat_back_addr":"172.21.15.16:6807/37008","heartbeat_front_addr":"172.21.15.16:6805/37008","state":["exists","up"]},{"osd":2,"uuid":"1e7bba46-4eba-47ef-bb81-5e2d371638ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":102,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":37009},{"type":"v1","addr":"172.21.15.16:6809","nonce":37009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":37009},{"type":"v1","addr":"172.21.15.16:6811","nonce":37009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":37009},{"type":"v1","addr":"172.21.15.16:6815","nonce":37009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":37009},{"type":"v1","addr":"172.21.15.16:6813","nonce":37009}]},"public_addr":"172.21.15.16:6809/37009","cluster_addr":"172.21.15.16:6811/37009","heartbeat_back_addr":"172.21.15.16:6815/37009","heartbeat_front_addr":"172.21.15.16:6813/37009","state":["exists","up"]},{"osd":3,"uuid":"383cacc0-5a88-4904-990e-498ca59ef48a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":102,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":37020},{"type":"v1","addr":"172.21.15.16:6817","nonce":37020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":37020},{"type":"v1","addr":"172.21.15.16:6819","nonce":37020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":37020},{"type":"v1","addr":"172.21.15.16:6823","nonce":37020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":37020},{"type":"v1","addr":"172.21.15.16:6821","nonce":37020}]},"public_addr":"172.21.15.16:6817/37020","cluster_addr":"172.21.15.16:6819/37020","heartbeat_back_addr":"172.21.15.16:6823/37020","heartbeat_front_addr":"172.21.15.16:6821/37020","state":["exists","up"]},{"osd":4,"uuid":"970500ef-95b7-42c1-b080-eed4fae7013d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":102,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":37353},{"type":"v1","addr":"172.21.15.62:6817","nonce":37353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6819","nonce":37353},{"type":"v1","addr":"172.21.15.62:6822","nonce":37353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37353},{"type":"v1","addr":"172.21.15.62:6831","nonce":37353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37353},{"type":"v1","addr":"172.21.15.62:6826","nonce":37353}]},"public_addr":"172.21.15.62:6817/37353","cluster_addr":"172.21.15.62:6822/37353","heartbeat_back_addr":"172.21.15.62:6831/37353","heartbeat_front_addr":"172.21.15.62:6826/37353","state":["exists","up"]},{"osd":5,"uuid":"022d67db-d4d8-445b-b1b9-8b446a00f5c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":102,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":37352},{"type":"v1","addr":"172.21.15.62:6801","nonce":37352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":37352},{"type":"v1","addr":"172.21.15.62:6803","nonce":37352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":37352},{"type":"v1","addr":"172.21.15.62:6807","nonce":37352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":37352},{"type":"v1","addr":"172.21.15.62:6805","nonce":37352}]},"public_addr":"172.21.15.62:6801/37352","cluster_addr":"172.21.15.62:6803/37352","heartbeat_back_addr":"172.21.15.62:6807/37352","heartbeat_front_addr":"172.21.15.62:6805/37352","state":["exists","up"]},{"osd":6,"uuid":"42bd372b-6688-4a1b-8407-e16e8c266e33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":102,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":37354},{"type":"v1","addr":"172.21.15.62:6820","nonce":37354}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":37354},{"type":"v1","addr":"172.21.15.62:6823","nonce":37354}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6829","nonce":37354},{"type":"v1","addr":"172.21.15.62:6830","nonce":37354}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6825","nonce":37354},{"type":"v1","addr":"172.21.15.62:6827","nonce":37354}]},"public_addr":"172.21.15.62:6820/37354","cluster_addr":"172.21.15.62:6823/37354","heartbeat_back_addr":"172.21.15.62:6830/37354","heartbeat_front_addr":"172.21.15.62:6827/37354","state":["exists","up"]},{"osd":7,"uuid":"ac70b2a3-f41b-4bf2-b227-7f2cbb55426d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":102,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37355},{"type":"v1","addr":"172.21.15.62:6809","nonce":37355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37355},{"type":"v1","addr":"172.21.15.62:6811","nonce":37355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":37355},{"type":"v1","addr":"172.21.15.62:6815","nonce":37355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":37355},{"type":"v1","addr":"172.21.15.62:6813","nonce":37355}]},"public_addr":"172.21.15.62:6809/37355","cluster_addr":"172.21.15.62:6811/37355","heartbeat_back_addr":"172.21.15.62:6815/37355","heartbeat_front_addr":"172.21.15.62:6813/37355","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.368896+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.107273+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.276888+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.287502+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.124791+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.251937+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.255679+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.232797+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-05-22T15:42:59.241 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.235+0000 7f2d149cf640 1 -- 172.21.15.16:0/1294056754 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f2cf00654a0 msgr2=0x7f2cf0067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:59.242 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.235+0000 7f2d149cf640 1 --2- 172.21.15.16:0/1294056754 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f2cf00654a0 0x7f2cf0067950 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f2d000af000 tx=0x7f2d000a7000 comp rx=0 tx=0).stop 2023-05-22T15:42:59.242 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.235+0000 7f2d149cf640 1 -- 172.21.15.16:0/1294056754 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2d10159870 msgr2=0x7f2d1014f790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:59.242 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.235+0000 7f2d149cf640 1 --2- 172.21.15.16:0/1294056754 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2d10159870 0x7f2d1014f790 secure :-1 s=READY pgs=1287 cs=0 l=1 rev1=1 crypto rx=0x7f2d0800ca80 tx=0x7f2d0800cf40 comp rx=0 tx=0).stop 2023-05-22T15:42:59.242 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.235+0000 7f2d149cf640 1 -- 172.21.15.16:0/1294056754 shutdown_connections 2023-05-22T15:42:59.242 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.235+0000 7f2d149cf640 1 --2- 172.21.15.16:0/1294056754 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2d10159870 0x7f2d1014f790 unknown :-1 s=CLOSED pgs=1287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:59.242 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.235+0000 7f2d149cf640 1 --2- 172.21.15.16:0/1294056754 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f2cf00654a0 0x7f2cf0067950 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:59.243 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.235+0000 7f2d149cf640 1 --2- 172.21.15.16:0/1294056754 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2d101531b0 0x7f2d1015a7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:59.243 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.235+0000 7f2d149cf640 1 --2- 172.21.15.16:0/1294056754 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2d100b0080 0x7f2d1014a470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:59.243 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.235+0000 7f2d149cf640 1 -- 172.21.15.16:0/1294056754 >> 172.21.15.16:0/1294056754 conn(0x7f2d10069720 msgr2=0x7f2d100cc4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:59.243 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.235+0000 7f2d149cf640 1 -- 172.21.15.16:0/1294056754 shutdown_connections 2023-05-22T15:42:59.243 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.235+0000 7f2d149cf640 1 -- 172.21.15.16:0/1294056754 wait complete. 2023-05-22T15:42:59.259 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-05-22T15:42:59.260 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-05-22T15:42:59.375 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.371+0000 7ff8937f4640 1 Processor -- start 2023-05-22T15:42:59.375 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.371+0000 7ff8937f4640 1 -- start start 2023-05-22T15:42:59.377 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.371+0000 7ff8937f4640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff88c0ccab0 0x7ff88c15fe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:59.377 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.371+0000 7ff8937f4640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7ff88c0d0410 0x7ff88c0d0810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:59.377 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.371+0000 7ff8937f4640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff88c0ccef0 con 0x7ff88c0b7dd0 2023-05-22T15:42:59.377 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.371+0000 7ff8937f4640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff88c0d3360 con 0x7ff88c0ccab0 2023-05-22T15:42:59.377 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.371+0000 7ff8937f4640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff88c0cd170 con 0x7ff88c0d0410 2023-05-22T15:42:59.377 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.371+0000 7ff890d8f640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff88c0ccab0 0x7ff88c15fe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:59.377 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.371+0000 7ff890d8f640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff88c0ccab0 0x7ff88c15fe10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49762/0 (socket says 172.21.15.16:49762) 2023-05-22T15:42:59.378 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.371+0000 7ff890d8f640 1 -- 172.21.15.16:0/1234753423 learned_addr learned my addr 172.21.15.16:0/1234753423 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:42:59.378 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.371+0000 7ff891d91640 1 --2- 172.21.15.16:0/1234753423 >> v2:172.21.15.62:3300/0 conn(0x7ff88c0d0410 0x7ff88c0d0810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:59.378 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.371+0000 7ff890d8f640 1 -- 172.21.15.16:0/1234753423 >> v2:172.21.15.62:3300/0 conn(0x7ff88c0d0410 msgr2=0x7ff88c0d0810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:42:59.378 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.371+0000 7ff890d8f640 1 --2- 172.21.15.16:0/1234753423 >> v2:172.21.15.62:3300/0 conn(0x7ff88c0d0410 0x7ff88c0d0810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:59.378 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.371+0000 7ff890d8f640 1 -- 172.21.15.16:0/1234753423 >> v1:172.21.15.62:6789/0 conn(0x7ff88c0b7dd0 legacy=0x7ff88c15f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:42:59.378 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.371+0000 7ff890d8f640 1 -- 172.21.15.16:0/1234753423 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff88c0d2ac0 con 0x7ff88c0ccab0 2023-05-22T15:42:59.379 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.375+0000 7ff890d8f640 1 --2- 172.21.15.16:0/1234753423 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff88c0ccab0 0x7ff88c15fe10 secure :-1 s=READY pgs=1288 cs=0 l=1 rev1=1 crypto rx=0x7ff88c15c190 tx=0x7ff87c01c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3095eb94c2e40d35 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:59.379 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.375+0000 7ff883fff640 1 -- 172.21.15.16:0/1234753423 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff87c0aa5f0 con 0x7ff88c0ccab0 2023-05-22T15:42:59.380 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.375+0000 7ff883fff640 1 -- 172.21.15.16:0/1234753423 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff87c0aa750 con 0x7ff88c0ccab0 2023-05-22T15:42:59.380 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.375+0000 7ff883fff640 1 -- 172.21.15.16:0/1234753423 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff87c0aaae0 con 0x7ff88c0ccab0 2023-05-22T15:42:59.380 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.375+0000 7ff8937f4640 1 -- 172.21.15.16:0/1234753423 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff88c0ccab0 msgr2=0x7ff88c15fe10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:59.380 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.375+0000 7ff8937f4640 1 --2- 172.21.15.16:0/1234753423 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff88c0ccab0 0x7ff88c15fe10 secure :-1 s=READY pgs=1288 cs=0 l=1 rev1=1 crypto rx=0x7ff88c15c190 tx=0x7ff87c01c490 comp rx=0 tx=0).stop 2023-05-22T15:42:59.380 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.375+0000 7ff8937f4640 1 -- 172.21.15.16:0/1234753423 shutdown_connections 2023-05-22T15:42:59.380 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.375+0000 7ff8937f4640 1 --2- 172.21.15.16:0/1234753423 >> v2:172.21.15.62:3300/0 conn(0x7ff88c0d0410 0x7ff88c0d0810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:59.381 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.375+0000 7ff8937f4640 1 --2- 172.21.15.16:0/1234753423 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff88c0ccab0 0x7ff88c15fe10 unknown :-1 s=CLOSED pgs=1288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:59.381 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.375+0000 7ff8937f4640 1 -- 172.21.15.16:0/1234753423 >> 172.21.15.16:0/1234753423 conn(0x7ff88c0655c0 msgr2=0x7ff88c070e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:59.381 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.375+0000 7ff8937f4640 1 -- 172.21.15.16:0/1234753423 shutdown_connections 2023-05-22T15:42:59.381 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.375+0000 7ff8937f4640 1 -- 172.21.15.16:0/1234753423 wait complete. 2023-05-22T15:42:59.383 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.375+0000 7ff8937f4640 1 Processor -- start 2023-05-22T15:42:59.384 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.375+0000 7ff8937f4640 1 -- start start 2023-05-22T15:42:59.385 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.375+0000 7ff8937f4640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff88c0b7dd0 0x7ff88c15c200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:59.385 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.375+0000 7ff8937f4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff88c0ccab0 0x7ff88c153640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:59.385 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.375+0000 7ff8937f4640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff88c0d0410 0x7ff88c15e8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:59.385 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.375+0000 7ff8937f4640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff88c15a2c0 con 0x7ff88c0ccab0 2023-05-22T15:42:59.386 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.375+0000 7ff8937f4640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff88c15a5a0 con 0x7ff88c0b7dd0 2023-05-22T15:42:59.386 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.375+0000 7ff8937f4640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff88c15a880 con 0x7ff88c0d0410 2023-05-22T15:42:59.386 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.375+0000 7ff890d8f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff88c0ccab0 0x7ff88c153640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:59.386 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.375+0000 7ff890d8f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff88c0ccab0 0x7ff88c153640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49776/0 (socket says 172.21.15.16:49776) 2023-05-22T15:42:59.387 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.375+0000 7ff890d8f640 1 -- 172.21.15.16:0/1086894102 learned_addr learned my addr 172.21.15.16:0/1086894102 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:42:59.387 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.379+0000 7ff891590640 1 --2- 172.21.15.16:0/1086894102 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff88c0b7dd0 0x7ff88c15c200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:59.387 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.379+0000 7ff891d91640 1 --2- 172.21.15.16:0/1086894102 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff88c0d0410 0x7ff88c15e8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:59.388 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.379+0000 7ff890d8f640 1 -- 172.21.15.16:0/1086894102 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff88c0d0410 msgr2=0x7ff88c15e8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:59.388 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.379+0000 7ff890d8f640 1 --2- 172.21.15.16:0/1086894102 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff88c0d0410 0x7ff88c15e8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:59.388 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.379+0000 7ff890d8f640 1 -- 172.21.15.16:0/1086894102 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff88c0b7dd0 msgr2=0x7ff88c15c200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:59.388 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.379+0000 7ff890d8f640 1 --2- 172.21.15.16:0/1086894102 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff88c0b7dd0 0x7ff88c15c200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:59.389 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.379+0000 7ff890d8f640 1 -- 172.21.15.16:0/1086894102 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff87c0094d0 con 0x7ff88c0ccab0 2023-05-22T15:42:59.389 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.379+0000 7ff890d8f640 1 --2- 172.21.15.16:0/1086894102 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff88c0ccab0 0x7ff88c153640 secure :-1 s=READY pgs=1289 cs=0 l=1 rev1=1 crypto rx=0x7ff87c01a0c0 tx=0x7ff87c01acd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:59.389 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.379+0000 7ff881ffb640 1 -- 172.21.15.16:0/1086894102 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff87c0aa5f0 con 0x7ff88c0ccab0 2023-05-22T15:42:59.390 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.379+0000 7ff881ffb640 1 -- 172.21.15.16:0/1086894102 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff87c0b3800 con 0x7ff88c0ccab0 2023-05-22T15:42:59.390 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.379+0000 7ff881ffb640 1 -- 172.21.15.16:0/1086894102 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff87c0b3bb0 con 0x7ff88c0ccab0 2023-05-22T15:42:59.390 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.379+0000 7ff8937f4640 1 -- 172.21.15.16:0/1086894102 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff88c160350 con 0x7ff88c0ccab0 2023-05-22T15:42:59.390 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.379+0000 7ff8937f4640 1 -- 172.21.15.16:0/1086894102 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff88c0cf730 con 0x7ff88c0ccab0 2023-05-22T15:42:59.392 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.379+0000 7ff8937f4640 1 -- 172.21.15.16:0/1086894102 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff88c1604e0 con 0x7ff88c0ccab0 2023-05-22T15:42:59.392 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.379+0000 7ff881ffb640 1 -- 172.21.15.16:0/1086894102 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 34) v1 ==== 84332+0+0 (secure 0 0 0) 0x7ff87c0b3d10 con 0x7ff88c0ccab0 2023-05-22T15:42:59.392 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.379+0000 7ff881ffb640 1 --2- 172.21.15.16:0/1086894102 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ff8700654a0 0x7ff870067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:59.393 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.379+0000 7ff881ffb640 1 -- 172.21.15.16:0/1086894102 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 6382+0+0 (secure 0 0 0) 0x7ff87c0a7070 con 0x7ff88c0ccab0 2023-05-22T15:42:59.393 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.379+0000 7ff891590640 1 --2- 172.21.15.16:0/1086894102 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ff8700654a0 0x7ff870067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:59.393 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.379+0000 7ff891590640 1 --2- 172.21.15.16:0/1086894102 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ff8700654a0 0x7ff870067950 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7ff888004e30 tx=0x7ff888009040 comp rx=0 tx=0).ready entity=mgr.6436 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:59.393 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.383+0000 7ff881ffb640 1 -- 172.21.15.16:0/1086894102 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7ff87c0eacf0 con 0x7ff88c0ccab0 2023-05-22T15:42:59.534 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.531+0000 7ff8937f4640 1 -- 172.21.15.16:0/1086894102 --> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff88c04dbe0 con 0x7ff8700654a0 2023-05-22T15:42:59.559 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.555+0000 7ff881ffb640 1 -- 172.21.15.16:0/1086894102 <== mgr.6436 v2:172.21.15.16:6832/36921 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+310504 (secure 0 0 0) 0x7ff88c04dbe0 con 0x7ff8700654a0 2023-05-22T15:42:59.563 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:42:59.565 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2023-05-22T15:42:59.566 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.559+0000 7ff8937f4640 1 -- 172.21.15.16:0/1086894102 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ff8700654a0 msgr2=0x7ff870067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:59.566 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.559+0000 7ff8937f4640 1 --2- 172.21.15.16:0/1086894102 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ff8700654a0 0x7ff870067950 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7ff888004e30 tx=0x7ff888009040 comp rx=0 tx=0).stop 2023-05-22T15:42:59.567 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.559+0000 7ff8937f4640 1 -- 172.21.15.16:0/1086894102 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff88c0ccab0 msgr2=0x7ff88c153640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:59.567 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.559+0000 7ff8937f4640 1 --2- 172.21.15.16:0/1086894102 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff88c0ccab0 0x7ff88c153640 secure :-1 s=READY pgs=1289 cs=0 l=1 rev1=1 crypto rx=0x7ff87c01a0c0 tx=0x7ff87c01acd0 comp rx=0 tx=0).stop 2023-05-22T15:42:59.567 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.559+0000 7ff8937f4640 1 -- 172.21.15.16:0/1086894102 shutdown_connections 2023-05-22T15:42:59.567 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.559+0000 7ff8937f4640 1 --2- 172.21.15.16:0/1086894102 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ff8700654a0 0x7ff870067950 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:59.568 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.559+0000 7ff8937f4640 1 --2- 172.21.15.16:0/1086894102 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff88c0d0410 0x7ff88c15e8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:59.568 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.559+0000 7ff8937f4640 1 --2- 172.21.15.16:0/1086894102 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff88c0ccab0 0x7ff88c153640 unknown :-1 s=CLOSED pgs=1289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:59.568 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.559+0000 7ff8937f4640 1 --2- 172.21.15.16:0/1086894102 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff88c0b7dd0 0x7ff88c15c200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:59.568 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.559+0000 7ff8937f4640 1 -- 172.21.15.16:0/1086894102 >> 172.21.15.16:0/1086894102 conn(0x7ff88c0655c0 msgr2=0x7ff88c070e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:59.569 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.559+0000 7ff8937f4640 1 -- 172.21.15.16:0/1086894102 shutdown_connections 2023-05-22T15:42:59.569 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.559+0000 7ff8937f4640 1 -- 172.21.15.16:0/1086894102 wait complete. 2023-05-22T15:42:59.581 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2023-05-22T15:42:59.480035+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592698,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":692,"num_read_kb":1336,"num_write":323,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":221,"ondisk_log_size":221,"up":210,"acting":210,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":67,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":211632,"kb_used_data":11056,"kb_used_omap":0,"kb_used_meta":200512,"kb_avail":754763088,"statfs":{"total":773094113280,"available":772877402112,"internally_reserved":0,"allocated":11321344,"data_stored":6466766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":205324288},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":60,"apply_latency_ms":60,"commit_latency_ns":60000000,"apply_latency_ns":60000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2330,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.293283"},"pg_stats":[{"pgid":"13.3f","version":"0'0","reported_seq":9,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.648813+0000","last_change":"2023-05-22T15:42:56.645184+0000","last_active":"2023-05-22T15:42:56.648813+0000","last_peered":"2023-05-22T15:42:56.648813+0000","last_clean":"2023-05-22T15:42:56.648813+0000","last_became_active":"2023-05-22T15:42:56.644521+0000","last_became_peered":"2023-05-22T15:42:56.644521+0000","last_unstale":"2023-05-22T15:42:56.648813+0000","last_undegraded":"2023-05-22T15:42:56.648813+0000","last_fullsized":"2023-05-22T15:42:56.648813+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_clean_scrub_stamp":"2023-05-22T15:42:56.185172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:11:10.859188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.3e","version":"0'0","reported_seq":9,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.642132+0000","last_change":"2023-05-22T15:42:56.641150+0000","last_active":"2023-05-22T15:42:56.642132+0000","last_peered":"2023-05-22T15:42:56.642132+0000","last_clean":"2023-05-22T15:42:56.642132+0000","last_became_active":"2023-05-22T15:42:56.640893+0000","last_became_peered":"2023-05-22T15:42:56.640893+0000","last_unstale":"2023-05-22T15:42:56.642132+0000","last_undegraded":"2023-05-22T15:42:56.642132+0000","last_fullsized":"2023-05-22T15:42:56.642132+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_clean_scrub_stamp":"2023-05-22T15:42:56.185172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:53:48.911917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.3d","version":"0'0","reported_seq":9,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.646639+0000","last_change":"2023-05-22T15:42:56.642827+0000","last_active":"2023-05-22T15:42:56.646639+0000","last_peered":"2023-05-22T15:42:56.646639+0000","last_clean":"2023-05-22T15:42:56.646639+0000","last_became_active":"2023-05-22T15:42:56.642218+0000","last_became_peered":"2023-05-22T15:42:56.642218+0000","last_unstale":"2023-05-22T15:42:56.646639+0000","last_undegraded":"2023-05-22T15:42:56.646639+0000","last_fullsized":"2023-05-22T15:42:56.646639+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_clean_scrub_stamp":"2023-05-22T15:42:56.185172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:33:42.886208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.3c","version":"0'0","reported_seq":9,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.599780+0000","last_change":"2023-05-22T15:42:56.595191+0000","last_active":"2023-05-22T15:42:56.599780+0000","last_peered":"2023-05-22T15:42:56.599780+0000","last_clean":"2023-05-22T15:42:56.599780+0000","last_became_active":"2023-05-22T15:42:56.594139+0000","last_became_peered":"2023-05-22T15:42:56.594139+0000","last_unstale":"2023-05-22T15:42:56.599780+0000","last_undegraded":"2023-05-22T15:42:56.599780+0000","last_fullsized":"2023-05-22T15:42:56.599780+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_clean_scrub_stamp":"2023-05-22T15:42:56.185172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T18:07:13.587805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.3b","version":"0'0","reported_seq":9,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.620964+0000","last_change":"2023-05-22T15:42:56.607640+0000","last_active":"2023-05-22T15:42:56.620964+0000","last_peered":"2023-05-22T15:42:56.620964+0000","last_clean":"2023-05-22T15:42:56.620964+0000","last_became_active":"2023-05-22T15:42:56.599485+0000","last_became_peered":"2023-05-22T15:42:56.599485+0000","last_unstale":"2023-05-22T15:42:56.620964+0000","last_undegraded":"2023-05-22T15:42:56.620964+0000","last_fullsized":"2023-05-22T15:42:56.620964+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_clean_scrub_stamp":"2023-05-22T15:42:56.185172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:58:17.652601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.3a","version":"0'0","reported_seq":9,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.630118+0000","last_change":"2023-05-22T15:42:56.593856+0000","last_active":"2023-05-22T15:42:56.630118+0000","last_peered":"2023-05-22T15:42:56.630118+0000","last_clean":"2023-05-22T15:42:56.630118+0000","last_became_active":"2023-05-22T15:42:56.592963+0000","last_became_peered":"2023-05-22T15:42:56.592963+0000","last_unstale":"2023-05-22T15:42:56.630118+0000","last_undegraded":"2023-05-22T15:42:56.630118+0000","last_fullsized":"2023-05-22T15:42:56.630118+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_clean_scrub_stamp":"2023-05-22T15:42:56.185172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:35:58.004110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.39","version":"0'0","reported_seq":9,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.624344+0000","last_change":"2023-05-22T15:42:56.623486+0000","last_active":"2023-05-22T15:42:56.624344+0000","last_peered":"2023-05-22T15:42:56.624344+0000","last_clean":"2023-05-22T15:42:56.624344+0000","last_became_active":"2023-05-22T15:42:56.623143+0000","last_became_peered":"2023-05-22T15:42:56.623143+0000","last_unstale":"2023-05-22T15:42:56.624344+0000","last_undegraded":"2023-05-22T15:42:56.624344+0000","last_fullsized":"2023-05-22T15:42:56.624344+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_clean_scrub_stamp":"2023-05-22T15:42:56.185172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:10:36.752607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.38","version":"0'0","reported_seq":9,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.630029+0000","last_change":"2023-05-22T15:42:56.602489+0000","last_active":"2023-05-22T15:42:56.630029+0000","last_peered":"2023-05-22T15:42:56.630029+0000","last_clean":"2023-05-22T15:42:56.630029+0000","last_became_active":"2023-05-22T15:42:56.601238+0000","last_became_peered":"2023-05-22T15:42:56.601238+0000","last_unstale":"2023-05-22T15:42:56.630029+0000","last_undegraded":"2023-05-22T15:42:56.630029+0000","last_fullsized":"2023-05-22T15:42:56.630029+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_clean_scrub_stamp":"2023-05-22T15:42:56.185172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:37:31.421851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.37","version":"0'0","reported_seq":9,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.630289+0000","last_change":"2023-05-22T15:42:56.608853+0000","last_active":"2023-05-22T15:42:56.630289+0000","last_peered":"2023-05-22T15:42:56.630289+0000","last_clean":"2023-05-22T15:42:56.630289+0000","last_became_active":"2023-05-22T15:42:56.608034+0000","last_became_peered":"2023-05-22T15:42:56.608034+0000","last_unstale":"2023-05-22T15:42:56.630289+0000","last_undegraded":"2023-05-22T15:42:56.630289+0000","last_fullsized":"2023-05-22T15:42:56.630289+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_clean_scrub_stamp":"2023-05-22T15:42:56.185172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:33:32.621247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.36","version":"0'0","reported_seq":9,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.640612+0000","last_change":"2023-05-22T15:42:56.622967+0000","last_active":"2023-05-22T15:42:56.640612+0000","last_peered":"2023-05-22T15:42:56.640612+0000","last_clean":"2023-05-22T15:42:56.640612+0000","last_became_active":"2023-05-22T15:42:56.619649+0000","last_became_peered":"2023-05-22T15:42:56.619649+0000","last_unstale":"2023-05-22T15:42:56.640612+0000","last_undegraded":"2023-05-22T15:42:56.640612+0000","last_fullsized":"2023-05-22T15:42:56.640612+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_clean_scrub_stamp":"2023-05-22T15:42:56.185172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:37:31.421851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.35","version":"0'0","reported_seq":9,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.622397+0000","last_change":"2023-05-22T15:42:56.621540+0000","last_active":"2023-05-22T15:42:56.622397+0000","last_peered":"2023-05-22T15:42:56.622397+0000","last_clean":"2023-05-22T15:42:56.622397+0000","last_became_active":"2023-05-22T15:42:56.621208+0000","last_became_peered":"2023-05-22T15:42:56.621208+0000","last_unstale":"2023-05-22T15:42:56.622397+0000","last_undegraded":"2023-05-22T15:42:56.622397+0000","last_fullsized":"2023-05-22T15:42:56.622397+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_clean_scrub_stamp":"2023-05-22T15:42:56.185172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:45:35.903593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.34","version":"0'0","reported_seq":9,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.646248+0000","last_change":"2023-05-22T15:42:56.639118+0000","last_active":"2023-05-22T15:42:56.646248+0000","last_peered":"2023-05-22T15:42:56.646248+0000","last_clean":"2023-05-22T15:42:56.646248+0000","last_became_active":"2023-05-22T15:42:56.636421+0000","last_became_peered":"2023-05-22T15:42:56.636421+0000","last_unstale":"2023-05-22T15:42:56.646248+0000","last_undegraded":"2023-05-22T15:42:56.646248+0000","last_fullsized":"2023-05-22T15:42:56.646248+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_clean_scrub_stamp":"2023-05-22T15:42:56.185172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:15:13.237338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.33","version":"0'0","reported_seq":9,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.637493+0000","last_change":"2023-05-22T15:42:56.625982+0000","last_active":"2023-05-22T15:42:56.637493+0000","last_peered":"2023-05-22T15:42:56.637493+0000","last_clean":"2023-05-22T15:42:56.637493+0000","last_became_active":"2023-05-22T15:42:56.623302+0000","last_became_peered":"2023-05-22T15:42:56.623302+0000","last_unstale":"2023-05-22T15:42:56.637493+0000","last_undegraded":"2023-05-22T15:42:56.637493+0000","last_fullsized":"2023-05-22T15:42:56.637493+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_clean_scrub_stamp":"2023-05-22T15:42:56.185172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:23:15.856405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.32","version":"0'0","reported_seq":9,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.601726+0000","last_change":"2023-05-22T15:42:56.598440+0000","last_active":"2023-05-22T15:42:56.601726+0000","last_peered":"2023-05-22T15:42:56.601726+0000","last_clean":"2023-05-22T15:42:56.601726+0000","last_became_active":"2023-05-22T15:42:56.598136+0000","last_became_peered":"2023-05-22T15:42:56.598136+0000","last_unstale":"2023-05-22T15:42:56.601726+0000","last_undegraded":"2023-05-22T15:42:56.601726+0000","last_fullsized":"2023-05-22T15:42:56.601726+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_clean_scrub_stamp":"2023-05-22T15:42:56.185172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:00:34.414109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.31","version":"0'0","reported_seq":9,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.625366+0000","last_change":"2023-05-22T15:42:56.624583+0000","last_active":"2023-05-22T15:42:56.625366+0000","last_peered":"2023-05-22T15:42:56.625366+0000","last_clean":"2023-05-22T15:42:56.625366+0000","last_became_active":"2023-05-22T15:42:56.624353+0000","last_became_peered":"2023-05-22T15:42:56.624353+0000","last_unstale":"2023-05-22T15:42:56.625366+0000","last_undegraded":"2023-05-22T15:42:56.625366+0000","last_fullsized":"2023-05-22T15:42:56.625366+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_clean_scrub_stamp":"2023-05-22T15:42:56.185172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:23:14.021552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.30","version":"0'0","reported_seq":9,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.632651+0000","last_change":"2023-05-22T15:42:56.627501+0000","last_active":"2023-05-22T15:42:56.632651+0000","last_peered":"2023-05-22T15:42:56.632651+0000","last_clean":"2023-05-22T15:42:56.632651+0000","last_became_active":"2023-05-22T15:42:56.626875+0000","last_became_peered":"2023-05-22T15:42:56.626875+0000","last_unstale":"2023-05-22T15:42:56.632651+0000","last_undegraded":"2023-05-22T15:42:56.632651+0000","last_fullsized":"2023-05-22T15:42:56.632651+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_clean_scrub_stamp":"2023-05-22T15:42:56.185172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:33:56.028414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2f","version":"0'0","reported_seq":9,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.611771+0000","last_change":"2023-05-22T15:42:56.595382+0000","last_active":"2023-05-22T15:42:56.611771+0000","last_peered":"2023-05-22T15:42:56.611771+0000","last_clean":"2023-05-22T15:42:56.611771+0000","last_became_active":"2023-05-22T15:42:56.594590+0000","last_became_peered":"2023-05-22T15:42:56.594590+0000","last_unstale":"2023-05-22T15:42:56.611771+0000","last_undegraded":"2023-05-22T15:42:56.611771+0000","last_fullsized":"2023-05-22T15:42:56.611771+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_clean_scrub_stamp":"2023-05-22T15:42:56.185172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:26:51.212963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2e","version":"0'0","reported_seq":9,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.624723+0000","last_change":"2023-05-22T15:42:56.623690+0000","last_active":"2023-05-22T15:42:56.624723+0000","last_peered":"2023-05-22T15:42:56.624723+0000","last_clean":"2023-05-22T15:42:56.624723+0000","last_became_active":"2023-05-22T15:42:56.623320+0000","last_became_peered":"2023-05-22T15:42:56.623320+0000","last_unstale":"2023-05-22T15:42:56.624723+0000","last_undegraded":"2023-05-22T15:42:56.624723+0000","last_fullsized":"2023-05-22T15:42:56.624723+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_clean_scrub_stamp":"2023-05-22T15:42:56.185172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T16:22:07.524139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2d","version":"0'0","reported_seq":9,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.606120+0000","last_change":"2023-05-22T15:42:56.599343+0000","last_active":"2023-05-22T15:42:56.606120+0000","last_peered":"2023-05-22T15:42:56.606120+0000","last_clean":"2023-05-22T15:42:56.606120+0000","last_became_active":"2023-05-22T15:42:56.598878+0000","last_became_peered":"2023-05-22T15:42:56.598878+0000","last_unstale":"2023-05-22T15:42:56.606120+0000","last_undegraded":"2023-05-22T15:42:56.606120+0000","last_fullsized":"2023-05-22T15:42:56.606120+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_clean_scrub_stamp":"2023-05-22T15:42:56.185172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:30:35.129326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.2c","version":"0'0","reported_seq":9,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.590108+0000","last_change":"2023-05-22T15:42:56.585137+0000","last_active":"2023-05-22T15:42:56.590108+0000","last_peered":"2023-05-22T15:42:56.590108+0000","last_clean":"2023-05-22T15:42:56.590108+0000","last_became_active":"2023-05-22T15:42:56.584436+0000","last_became_peered":"2023-05-22T15:42:56.584436+0000","last_unstale":"2023-05-22T15:42:56.590108+0000","last_undegraded":"2023-05-22T15:42:56.590108+0000","last_fullsized":"2023-05-22T15:42:56.590108+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_clean_scrub_stamp":"2023-05-22T15:42:56.185172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:42:20.650549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2b","version":"0'0","reported_seq":9,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.582952+0000","last_change":"2023-05-22T15:42:56.581991+0000","last_active":"2023-05-22T15:42:56.582952+0000","last_peered":"2023-05-22T15:42:56.582952+0000","last_clean":"2023-05-22T15:42:56.582952+0000","last_became_active":"2023-05-22T15:42:56.581713+0000","last_became_peered":"2023-05-22T15:42:56.581713+0000","last_unstale":"2023-05-22T15:42:56.582952+0000","last_undegraded":"2023-05-22T15:42:56.582952+0000","last_fullsized":"2023-05-22T15:42:56.582952+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_clean_scrub_stamp":"2023-05-22T15:42:56.185172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:09:09.562683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2a","version":"0'0","reported_seq":9,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.605766+0000","last_change":"2023-05-22T15:42:56.603209+0000","last_active":"2023-05-22T15:42:56.605766+0000","last_peered":"2023-05-22T15:42:56.605766+0000","last_clean":"2023-05-22T15:42:56.605766+0000","last_became_active":"2023-05-22T15:42:56.602949+0000","last_became_peered":"2023-05-22T15:42:56.602949+0000","last_unstale":"2023-05-22T15:42:56.605766+0000","last_undegraded":"2023-05-22T15:42:56.605766+0000","last_fullsized":"2023-05-22T15:42:56.605766+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_clean_scrub_stamp":"2023-05-22T15:42:56.185172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:29:59.008510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.29","version":"0'0","reported_seq":9,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.640922+0000","last_change":"2023-05-22T15:42:56.625299+0000","last_active":"2023-05-22T15:42:56.640922+0000","last_peered":"2023-05-22T15:42:56.640922+0000","last_clean":"2023-05-22T15:42:56.640922+0000","last_became_active":"2023-05-22T15:42:56.624521+0000","last_became_peered":"2023-05-22T15:42:56.624521+0000","last_unstale":"2023-05-22T15:42:56.640922+0000","last_undegraded":"2023-05-22T15:42:56.640922+0000","last_fullsized":"2023-05-22T15:42:56.640922+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_clean_scrub_stamp":"2023-05-22T15:42:56.185172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:53:52.392012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.28","version":"0'0","reported_seq":9,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.590010+0000","last_change":"2023-05-22T15:42:56.584601+0000","last_active":"2023-05-22T15:42:56.590010+0000","last_peered":"2023-05-22T15:42:56.590010+0000","last_clean":"2023-05-22T15:42:56.590010+0000","last_became_active":"2023-05-22T15:42:56.584317+0000","last_became_peered":"2023-05-22T15:42:56.584317+0000","last_unstale":"2023-05-22T15:42:56.590010+0000","last_undegraded":"2023-05-22T15:42:56.590010+0000","last_fullsized":"2023-05-22T15:42:56.590010+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_clean_scrub_stamp":"2023-05-22T15:42:56.185172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:59:24.891214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.27","version":"0'0","reported_seq":9,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.635353+0000","last_change":"2023-05-22T15:42:56.634247+0000","last_active":"2023-05-22T15:42:56.635353+0000","last_peered":"2023-05-22T15:42:56.635353+0000","last_clean":"2023-05-22T15:42:56.635353+0000","last_became_active":"2023-05-22T15:42:56.633957+0000","last_became_peered":"2023-05-22T15:42:56.633957+0000","last_unstale":"2023-05-22T15:42:56.635353+0000","last_undegraded":"2023-05-22T15:42:56.635353+0000","last_fullsized":"2023-05-22T15:42:56.635353+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_clean_scrub_stamp":"2023-05-22T15:42:56.185172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T03:30:09.923365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.26","version":"0'0","reported_seq":9,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.637650+0000","last_change":"2023-05-22T15:42:56.629495+0000","last_active":"2023-05-22T15:42:56.637650+0000","last_peered":"2023-05-22T15:42:56.637650+0000","last_clean":"2023-05-22T15:42:56.637650+0000","last_became_active":"2023-05-22T15:42:56.628731+0000","last_became_peered":"2023-05-22T15:42:56.628731+0000","last_unstale":"2023-05-22T15:42:56.637650+0000","last_undegraded":"2023-05-22T15:42:56.637650+0000","last_fullsized":"2023-05-22T15:42:56.637650+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_clean_scrub_stamp":"2023-05-22T15:42:56.185172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:57:40.502973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.25","version":"0'0","reported_seq":9,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.612043+0000","last_change":"2023-05-22T15:42:56.601023+0000","last_active":"2023-05-22T15:42:56.612043+0000","last_peered":"2023-05-22T15:42:56.612043+0000","last_clean":"2023-05-22T15:42:56.612043+0000","last_became_active":"2023-05-22T15:42:56.597820+0000","last_became_peered":"2023-05-22T15:42:56.597820+0000","last_unstale":"2023-05-22T15:42:56.612043+0000","last_undegraded":"2023-05-22T15:42:56.612043+0000","last_fullsized":"2023-05-22T15:42:56.612043+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_clean_scrub_stamp":"2023-05-22T15:42:56.185172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:57:37.896404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.24","version":"0'0","reported_seq":9,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.648267+0000","last_change":"2023-05-22T15:42:56.641209+0000","last_active":"2023-05-22T15:42:56.648267+0000","last_peered":"2023-05-22T15:42:56.648267+0000","last_clean":"2023-05-22T15:42:56.648267+0000","last_became_active":"2023-05-22T15:42:56.640689+0000","last_became_peered":"2023-05-22T15:42:56.640689+0000","last_unstale":"2023-05-22T15:42:56.648267+0000","last_undegraded":"2023-05-22T15:42:56.648267+0000","last_fullsized":"2023-05-22T15:42:56.648267+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_clean_scrub_stamp":"2023-05-22T15:42:56.185172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:05:52.591919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.23","version":"0'0","reported_seq":9,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.602625+0000","last_change":"2023-05-22T15:42:56.595383+0000","last_active":"2023-05-22T15:42:56.602625+0000","last_peered":"2023-05-22T15:42:56.602625+0000","last_clean":"2023-05-22T15:42:56.602625+0000","last_became_active":"2023-05-22T15:42:56.595117+0000","last_became_peered":"2023-05-22T15:42:56.595117+0000","last_unstale":"2023-05-22T15:42:56.602625+0000","last_undegraded":"2023-05-22T15:42:56.602625+0000","last_fullsized":"2023-05-22T15:42:56.602625+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_clean_scrub_stamp":"2023-05-22T15:42:56.185172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:09:09.562683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.22","version":"0'0","reported_seq":9,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.605411+0000","last_change":"2023-05-22T15:42:56.599349+0000","last_active":"2023-05-22T15:42:56.605411+0000","last_peered":"2023-05-22T15:42:56.605411+0000","last_clean":"2023-05-22T15:42:56.605411+0000","last_became_active":"2023-05-22T15:42:56.598970+0000","last_became_peered":"2023-05-22T15:42:56.598970+0000","last_unstale":"2023-05-22T15:42:56.605411+0000","last_undegraded":"2023-05-22T15:42:56.605411+0000","last_fullsized":"2023-05-22T15:42:56.605411+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_clean_scrub_stamp":"2023-05-22T15:42:56.185172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:05:52.591919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.21","version":"0'0","reported_seq":9,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.611848+0000","last_change":"2023-05-22T15:42:56.598275+0000","last_active":"2023-05-22T15:42:56.611848+0000","last_peered":"2023-05-22T15:42:56.611848+0000","last_clean":"2023-05-22T15:42:56.611848+0000","last_became_active":"2023-05-22T15:42:56.597447+0000","last_became_peered":"2023-05-22T15:42:56.597447+0000","last_unstale":"2023-05-22T15:42:56.611848+0000","last_undegraded":"2023-05-22T15:42:56.611848+0000","last_fullsized":"2023-05-22T15:42:56.611848+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_clean_scrub_stamp":"2023-05-22T15:42:56.185172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:40:39.423664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.20","version":"0'0","reported_seq":9,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.640734+0000","last_change":"2023-05-22T15:42:56.620133+0000","last_active":"2023-05-22T15:42:56.640734+0000","last_peered":"2023-05-22T15:42:56.640734+0000","last_clean":"2023-05-22T15:42:56.640734+0000","last_became_active":"2023-05-22T15:42:56.619225+0000","last_became_peered":"2023-05-22T15:42:56.619225+0000","last_unstale":"2023-05-22T15:42:56.640734+0000","last_undegraded":"2023-05-22T15:42:56.640734+0000","last_fullsized":"2023-05-22T15:42:56.640734+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_clean_scrub_stamp":"2023-05-22T15:42:56.185172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:48:43.959640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1e","version":"0'0","reported_seq":9,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.646383+0000","last_change":"2023-05-22T15:42:56.642799+0000","last_active":"2023-05-22T15:42:56.646383+0000","last_peered":"2023-05-22T15:42:56.646383+0000","last_clean":"2023-05-22T15:42:56.646383+0000","last_became_active":"2023-05-22T15:42:56.642102+0000","last_became_peered":"2023-05-22T15:42:56.642102+0000","last_unstale":"2023-05-22T15:42:56.646383+0000","last_undegraded":"2023-05-22T15:42:56.646383+0000","last_fullsized":"2023-05-22T15:42:56.646383+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_clean_scrub_stamp":"2023-05-22T15:42:56.185172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:30:35.129326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1f","version":"103'4","reported_seq":14,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.648907+0000","last_change":"2023-05-22T15:42:56.247802+0000","last_active":"2023-05-22T15:42:56.648907+0000","last_peered":"2023-05-22T15:42:56.648907+0000","last_clean":"2023-05-22T15:42:56.648907+0000","last_became_active":"2023-05-22T15:42:56.247046+0000","last_became_peered":"2023-05-22T15:42:56.247046+0000","last_unstale":"2023-05-22T15:42:56.648907+0000","last_undegraded":"2023-05-22T15:42:56.648907+0000","last_fullsized":"2023-05-22T15:42:56.648907+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:55.181398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:55.181398+0000","last_clean_scrub_stamp":"2023-05-22T15:42:55.181398+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:59:23.887440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":566,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1f","version":"0'0","reported_seq":9,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.646456+0000","last_change":"2023-05-22T15:42:56.632638+0000","last_active":"2023-05-22T15:42:56.646456+0000","last_peered":"2023-05-22T15:42:56.646456+0000","last_clean":"2023-05-22T15:42:56.646456+0000","last_became_active":"2023-05-22T15:42:56.623920+0000","last_became_peered":"2023-05-22T15:42:56.623920+0000","last_unstale":"2023-05-22T15:42:56.646456+0000","last_undegraded":"2023-05-22T15:42:56.646456+0000","last_fullsized":"2023-05-22T15:42:56.646456+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_clean_scrub_stamp":"2023-05-22T15:42:56.185172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:52:06.057485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1e","version":"103'1","reported_seq":11,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.600338+0000","last_change":"2023-05-22T15:42:56.233159+0000","last_active":"2023-05-22T15:42:56.600338+0000","last_peered":"2023-05-22T15:42:56.600338+0000","last_clean":"2023-05-22T15:42:56.600338+0000","last_became_active":"2023-05-22T15:42:56.232184+0000","last_became_peered":"2023-05-22T15:42:56.232184+0000","last_unstale":"2023-05-22T15:42:56.600338+0000","last_undegraded":"2023-05-22T15:42:56.600338+0000","last_fullsized":"2023-05-22T15:42:56.600338+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:55.181398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:55.181398+0000","last_clean_scrub_stamp":"2023-05-22T15:42:55.181398+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:11:09.855414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1c","version":"0'0","reported_seq":9,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.646289+0000","last_change":"2023-05-22T15:42:56.639197+0000","last_active":"2023-05-22T15:42:56.646289+0000","last_peered":"2023-05-22T15:42:56.646289+0000","last_clean":"2023-05-22T15:42:56.646289+0000","last_became_active":"2023-05-22T15:42:56.636246+0000","last_became_peered":"2023-05-22T15:42:56.636246+0000","last_unstale":"2023-05-22T15:42:56.646289+0000","last_undegraded":"2023-05-22T15:42:56.646289+0000","last_fullsized":"2023-05-22T15:42:56.646289+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_clean_scrub_stamp":"2023-05-22T15:42:56.185172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T18:44:11.980099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1d","version":"103'2","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.585681+0000","last_change":"2023-05-22T15:42:56.227802+0000","last_active":"2023-05-22T15:42:56.585681+0000","last_peered":"2023-05-22T15:42:56.585681+0000","last_clean":"2023-05-22T15:42:56.585681+0000","last_became_active":"2023-05-22T15:42:56.227511+0000","last_became_peered":"2023-05-22T15:42:56.227511+0000","last_unstale":"2023-05-22T15:42:56.585681+0000","last_undegraded":"2023-05-22T15:42:56.585681+0000","last_fullsized":"2023-05-22T15:42:56.585681+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:55.181398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:55.181398+0000","last_clean_scrub_stamp":"2023-05-22T15:42:55.181398+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:01:27.080923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1d","version":"0'0","reported_seq":9,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.637791+0000","last_change":"2023-05-22T15:42:56.631432+0000","last_active":"2023-05-22T15:42:56.637791+0000","last_peered":"2023-05-22T15:42:56.637791+0000","last_clean":"2023-05-22T15:42:56.637791+0000","last_became_active":"2023-05-22T15:42:56.629634+0000","last_became_peered":"2023-05-22T15:42:56.629634+0000","last_unstale":"2023-05-22T15:42:56.637791+0000","last_undegraded":"2023-05-22T15:42:56.637791+0000","last_fullsized":"2023-05-22T15:42:56.637791+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_clean_scrub_stamp":"2023-05-22T15:42:56.185172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:40:39.423664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1c","version":"103'2","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.600196+0000","last_change":"2023-05-22T15:42:56.228991+0000","last_active":"2023-05-22T15:42:56.600196+0000","last_peered":"2023-05-22T15:42:56.600196+0000","last_clean":"2023-05-22T15:42:56.600196+0000","last_became_active":"2023-05-22T15:42:56.228645+0000","last_became_peered":"2023-05-22T15:42:56.228645+0000","last_unstale":"2023-05-22T15:42:56.600196+0000","last_undegraded":"2023-05-22T15:42:56.600196+0000","last_fullsized":"2023-05-22T15:42:56.600196+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:55.181398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:55.181398+0000","last_clean_scrub_stamp":"2023-05-22T15:42:55.181398+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T16:55:34.285195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1a","version":"0'0","reported_seq":9,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.605326+0000","last_change":"2023-05-22T15:42:56.598288+0000","last_active":"2023-05-22T15:42:56.605326+0000","last_peered":"2023-05-22T15:42:56.605326+0000","last_clean":"2023-05-22T15:42:56.605326+0000","last_became_active":"2023-05-22T15:42:56.598007+0000","last_became_peered":"2023-05-22T15:42:56.598007+0000","last_unstale":"2023-05-22T15:42:56.605326+0000","last_undegraded":"2023-05-22T15:42:56.605326+0000","last_fullsized":"2023-05-22T15:42:56.605326+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_clean_scrub_stamp":"2023-05-22T15:42:56.185172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:28:10.838599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.1b","version":"103'1","reported_seq":11,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.608736+0000","last_change":"2023-05-22T15:42:56.234058+0000","last_active":"2023-05-22T15:42:56.608736+0000","last_peered":"2023-05-22T15:42:56.608736+0000","last_clean":"2023-05-22T15:42:56.608736+0000","last_became_active":"2023-05-22T15:42:56.232800+0000","last_became_peered":"2023-05-22T15:42:56.232800+0000","last_unstale":"2023-05-22T15:42:56.608736+0000","last_undegraded":"2023-05-22T15:42:56.608736+0000","last_fullsized":"2023-05-22T15:42:56.608736+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:55.181398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:55.181398+0000","last_clean_scrub_stamp":"2023-05-22T15:42:55.181398+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:49:37.988943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1b","version":"0'0","reported_seq":9,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.620910+0000","last_change":"2023-05-22T15:42:56.607031+0000","last_active":"2023-05-22T15:42:56.620910+0000","last_peered":"2023-05-22T15:42:56.620910+0000","last_clean":"2023-05-22T15:42:56.620910+0000","last_became_active":"2023-05-22T15:42:56.599657+0000","last_became_peered":"2023-05-22T15:42:56.599657+0000","last_unstale":"2023-05-22T15:42:56.620910+0000","last_undegraded":"2023-05-22T15:42:56.620910+0000","last_fullsized":"2023-05-22T15:42:56.620910+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_clean_scrub_stamp":"2023-05-22T15:42:56.185172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T18:00:17.614805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1a","version":"103'1","reported_seq":11,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.611692+0000","last_change":"2023-05-22T15:42:56.234276+0000","last_active":"2023-05-22T15:42:56.611692+0000","last_peered":"2023-05-22T15:42:56.611692+0000","last_clean":"2023-05-22T15:42:56.611692+0000","last_became_active":"2023-05-22T15:42:56.233491+0000","last_became_peered":"2023-05-22T15:42:56.233491+0000","last_unstale":"2023-05-22T15:42:56.611692+0000","last_undegraded":"2023-05-22T15:42:56.611692+0000","last_fullsized":"2023-05-22T15:42:56.611692+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:55.181398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:55.181398+0000","last_clean_scrub_stamp":"2023-05-22T15:42:55.181398+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T18:44:10.976325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":9,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.637370+0000","last_change":"2023-05-22T15:42:56.624785+0000","last_active":"2023-05-22T15:42:56.637370+0000","last_peered":"2023-05-22T15:42:56.637370+0000","last_clean":"2023-05-22T15:42:56.637370+0000","last_became_active":"2023-05-22T15:42:56.623512+0000","last_became_peered":"2023-05-22T15:42:56.623512+0000","last_unstale":"2023-05-22T15:42:56.637370+0000","last_undegraded":"2023-05-22T15:42:56.637370+0000","last_fullsized":"2023-05-22T15:42:56.637370+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_clean_scrub_stamp":"2023-05-22T15:42:56.185172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T16:55:35.288969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":10,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.572555+0000","last_change":"2023-05-22T15:42:56.236271+0000","last_active":"2023-05-22T15:42:56.572555+0000","last_peered":"2023-05-22T15:42:56.572555+0000","last_clean":"2023-05-22T15:42:56.572555+0000","last_became_active":"2023-05-22T15:42:56.235956+0000","last_became_peered":"2023-05-22T15:42:56.235956+0000","last_unstale":"2023-05-22T15:42:56.572555+0000","last_undegraded":"2023-05-22T15:42:56.572555+0000","last_fullsized":"2023-05-22T15:42:56.572555+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:55.181398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:55.181398+0000","last_clean_scrub_stamp":"2023-05-22T15:42:55.181398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:40:38.419890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":9,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.620627+0000","last_change":"2023-05-22T15:42:56.620040+0000","last_active":"2023-05-22T15:42:56.620627+0000","last_peered":"2023-05-22T15:42:56.620627+0000","last_clean":"2023-05-22T15:42:56.620627+0000","last_became_active":"2023-05-22T15:42:56.619755+0000","last_became_peered":"2023-05-22T15:42:56.619755+0000","last_unstale":"2023-05-22T15:42:56.620627+0000","last_undegraded":"2023-05-22T15:42:56.620627+0000","last_fullsized":"2023-05-22T15:42:56.620627+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_clean_scrub_stamp":"2023-05-22T15:42:56.185172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:30:35.129326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":10,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.592956+0000","last_change":"2023-05-22T15:42:56.241097+0000","last_active":"2023-05-22T15:42:56.592956+0000","last_peered":"2023-05-22T15:42:56.592956+0000","last_clean":"2023-05-22T15:42:56.592956+0000","last_became_active":"2023-05-22T15:42:56.240759+0000","last_became_peered":"2023-05-22T15:42:56.240759+0000","last_unstale":"2023-05-22T15:42:56.592956+0000","last_undegraded":"2023-05-22T15:42:56.592956+0000","last_fullsized":"2023-05-22T15:42:56.592956+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:55.181398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:55.181398+0000","last_clean_scrub_stamp":"2023-05-22T15:42:55.181398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:36:14.623130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":9,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.597812+0000","last_change":"2023-05-22T15:42:56.596326+0000","last_active":"2023-05-22T15:42:56.597812+0000","last_peered":"2023-05-22T15:42:56.597812+0000","last_clean":"2023-05-22T15:42:56.597812+0000","last_became_active":"2023-05-22T15:42:56.596035+0000","last_became_peered":"2023-05-22T15:42:56.596035+0000","last_unstale":"2023-05-22T15:42:56.597812+0000","last_undegraded":"2023-05-22T15:42:56.597812+0000","last_fullsized":"2023-05-22T15:42:56.597812+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_clean_scrub_stamp":"2023-05-22T15:42:56.185172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:26:51.212963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":10,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.624956+0000","last_change":"2023-05-22T15:42:56.247608+0000","last_active":"2023-05-22T15:42:56.624956+0000","last_peered":"2023-05-22T15:42:56.624956+0000","last_clean":"2023-05-22T15:42:56.624956+0000","last_became_active":"2023-05-22T15:42:56.246408+0000","last_became_peered":"2023-05-22T15:42:56.246408+0000","last_unstale":"2023-05-22T15:42:56.624956+0000","last_undegraded":"2023-05-22T15:42:56.624956+0000","last_fullsized":"2023-05-22T15:42:56.624956+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:55.181398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:55.181398+0000","last_clean_scrub_stamp":"2023-05-22T15:42:55.181398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:40:49.967414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":9,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.629400+0000","last_change":"2023-05-22T15:42:56.594234+0000","last_active":"2023-05-22T15:42:56.629400+0000","last_peered":"2023-05-22T15:42:56.629400+0000","last_clean":"2023-05-22T15:42:56.629400+0000","last_became_active":"2023-05-22T15:42:56.593463+0000","last_became_peered":"2023-05-22T15:42:56.593463+0000","last_unstale":"2023-05-22T15:42:56.629400+0000","last_undegraded":"2023-05-22T15:42:56.629400+0000","last_fullsized":"2023-05-22T15:42:56.629400+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_clean_scrub_stamp":"2023-05-22T15:42:56.185172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:57:40.502973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":10,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.622818+0000","last_change":"2023-05-22T15:42:56.245119+0000","last_active":"2023-05-22T15:42:56.622818+0000","last_peered":"2023-05-22T15:42:56.622818+0000","last_clean":"2023-05-22T15:42:56.622818+0000","last_became_active":"2023-05-22T15:42:56.244477+0000","last_became_peered":"2023-05-22T15:42:56.244477+0000","last_unstale":"2023-05-22T15:42:56.622818+0000","last_undegraded":"2023-05-22T15:42:56.622818+0000","last_fullsized":"2023-05-22T15:42:56.622818+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:55.181398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:55.181398+0000","last_clean_scrub_stamp":"2023-05-22T15:42:55.181398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:08:25.261985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.14","version":"0'0","reported_seq":9,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.638816+0000","last_change":"2023-05-22T15:42:56.638182+0000","last_active":"2023-05-22T15:42:56.638816+0000","last_peered":"2023-05-22T15:42:56.638816+0000","last_clean":"2023-05-22T15:42:56.638816+0000","last_became_active":"2023-05-22T15:42:56.637838+0000","last_became_peered":"2023-05-22T15:42:56.637838+0000","last_unstale":"2023-05-22T15:42:56.638816+0000","last_undegraded":"2023-05-22T15:42:56.638816+0000","last_fullsized":"2023-05-22T15:42:56.638816+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_clean_scrub_stamp":"2023-05-22T15:42:56.185172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:23:14.021552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.15","version":"103'3","reported_seq":13,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.640652+0000","last_change":"2023-05-22T15:42:56.246663+0000","last_active":"2023-05-22T15:42:56.640652+0000","last_peered":"2023-05-22T15:42:56.640652+0000","last_clean":"2023-05-22T15:42:56.640652+0000","last_became_active":"2023-05-22T15:42:56.245925+0000","last_became_peered":"2023-05-22T15:42:56.245925+0000","last_unstale":"2023-05-22T15:42:56.640652+0000","last_undegraded":"2023-05-22T15:42:56.640652+0000","last_fullsized":"2023-05-22T15:42:56.640652+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:55.181398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:55.181398+0000","last_clean_scrub_stamp":"2023-05-22T15:42:55.181398+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:36:14.623130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.15","version":"0'0","reported_seq":9,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.616612+0000","last_change":"2023-05-22T15:42:56.604169+0000","last_active":"2023-05-22T15:42:56.616612+0000","last_peered":"2023-05-22T15:42:56.616612+0000","last_clean":"2023-05-22T15:42:56.616612+0000","last_became_active":"2023-05-22T15:42:56.603406+0000","last_became_peered":"2023-05-22T15:42:56.603406+0000","last_unstale":"2023-05-22T15:42:56.616612+0000","last_undegraded":"2023-05-22T15:42:56.616612+0000","last_fullsized":"2023-05-22T15:42:56.616612+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_clean_scrub_stamp":"2023-05-22T15:42:56.185172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:33:42.886208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.14","version":"103'6","reported_seq":16,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.644751+0000","last_change":"2023-05-22T15:42:56.231238+0000","last_active":"2023-05-22T15:42:56.644751+0000","last_peered":"2023-05-22T15:42:56.644751+0000","last_clean":"2023-05-22T15:42:56.644751+0000","last_became_active":"2023-05-22T15:42:56.230500+0000","last_became_peered":"2023-05-22T15:42:56.230500+0000","last_unstale":"2023-05-22T15:42:56.644751+0000","last_undegraded":"2023-05-22T15:42:56.644751+0000","last_fullsized":"2023-05-22T15:42:56.644751+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:55.181398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:55.181398+0000","last_clean_scrub_stamp":"2023-05-22T15:42:55.181398+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:28:09.834825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1548,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":9,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.606062+0000","last_change":"2023-05-22T15:42:56.598573+0000","last_active":"2023-05-22T15:42:56.606062+0000","last_peered":"2023-05-22T15:42:56.606062+0000","last_clean":"2023-05-22T15:42:56.606062+0000","last_became_active":"2023-05-22T15:42:56.598275+0000","last_became_peered":"2023-05-22T15:42:56.598275+0000","last_unstale":"2023-05-22T15:42:56.606062+0000","last_undegraded":"2023-05-22T15:42:56.606062+0000","last_fullsized":"2023-05-22T15:42:56.606062+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_clean_scrub_stamp":"2023-05-22T15:42:56.185172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T18:00:17.614805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.13","version":"103'2","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.606623+0000","last_change":"2023-05-22T15:42:56.215827+0000","last_active":"2023-05-22T15:42:56.606623+0000","last_peered":"2023-05-22T15:42:56.606623+0000","last_clean":"2023-05-22T15:42:56.606623+0000","last_became_active":"2023-05-22T15:42:56.215525+0000","last_became_peered":"2023-05-22T15:42:56.215525+0000","last_unstale":"2023-05-22T15:42:56.606623+0000","last_undegraded":"2023-05-22T15:42:56.606623+0000","last_fullsized":"2023-05-22T15:42:56.606623+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:55.181398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:55.181398+0000","last_clean_scrub_stamp":"2023-05-22T15:42:55.181398+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:04:20.717819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.13","version":"0'0","reported_seq":9,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.637298+0000","last_change":"2023-05-22T15:42:56.624651+0000","last_active":"2023-05-22T15:42:56.637298+0000","last_peered":"2023-05-22T15:42:56.637298+0000","last_clean":"2023-05-22T15:42:56.637298+0000","last_became_active":"2023-05-22T15:42:56.623992+0000","last_became_peered":"2023-05-22T15:42:56.623992+0000","last_unstale":"2023-05-22T15:42:56.637298+0000","last_undegraded":"2023-05-22T15:42:56.637298+0000","last_fullsized":"2023-05-22T15:42:56.637298+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_clean_scrub_stamp":"2023-05-22T15:42:56.185172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:33:56.028414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":10,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.586197+0000","last_change":"2023-05-22T15:42:56.218515+0000","last_active":"2023-05-22T15:42:56.586197+0000","last_peered":"2023-05-22T15:42:56.586197+0000","last_clean":"2023-05-22T15:42:56.586197+0000","last_became_active":"2023-05-22T15:42:56.218149+0000","last_became_peered":"2023-05-22T15:42:56.218149+0000","last_unstale":"2023-05-22T15:42:56.586197+0000","last_undegraded":"2023-05-22T15:42:56.586197+0000","last_fullsized":"2023-05-22T15:42:56.586197+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:55.181398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:55.181398+0000","last_clean_scrub_stamp":"2023-05-22T15:42:55.181398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:58:16.648827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.10","version":"0'0","reported_seq":9,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.605458+0000","last_change":"2023-05-22T15:42:56.601026+0000","last_active":"2023-05-22T15:42:56.605458+0000","last_peered":"2023-05-22T15:42:56.605458+0000","last_clean":"2023-05-22T15:42:56.605458+0000","last_became_active":"2023-05-22T15:42:56.600758+0000","last_became_peered":"2023-05-22T15:42:56.600758+0000","last_unstale":"2023-05-22T15:42:56.605458+0000","last_undegraded":"2023-05-22T15:42:56.605458+0000","last_fullsized":"2023-05-22T15:42:56.605458+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_clean_scrub_stamp":"2023-05-22T15:42:56.185172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:09:51.134436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":10,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.580765+0000","last_change":"2023-05-22T15:42:56.234364+0000","last_active":"2023-05-22T15:42:56.580765+0000","last_peered":"2023-05-22T15:42:56.580765+0000","last_clean":"2023-05-22T15:42:56.580765+0000","last_became_active":"2023-05-22T15:42:56.233647+0000","last_became_peered":"2023-05-22T15:42:56.233647+0000","last_unstale":"2023-05-22T15:42:56.580765+0000","last_undegraded":"2023-05-22T15:42:56.580765+0000","last_fullsized":"2023-05-22T15:42:56.580765+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:55.181398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:55.181398+0000","last_clean_scrub_stamp":"2023-05-22T15:42:55.181398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:09:50.130662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":9,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.635357+0000","last_change":"2023-05-22T15:42:56.634351+0000","last_active":"2023-05-22T15:42:56.635357+0000","last_peered":"2023-05-22T15:42:56.635357+0000","last_clean":"2023-05-22T15:42:56.635357+0000","last_became_active":"2023-05-22T15:42:56.633939+0000","last_became_peered":"2023-05-22T15:42:56.633939+0000","last_unstale":"2023-05-22T15:42:56.635357+0000","last_undegraded":"2023-05-22T15:42:56.635357+0000","last_fullsized":"2023-05-22T15:42:56.635357+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_clean_scrub_stamp":"2023-05-22T15:42:56.185172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:49:38.992717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.10","version":"103'8","reported_seq":18,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.603145+0000","last_change":"2023-05-22T15:42:56.222730+0000","last_active":"2023-05-22T15:42:56.603145+0000","last_peered":"2023-05-22T15:42:56.603145+0000","last_clean":"2023-05-22T15:42:56.603145+0000","last_became_active":"2023-05-22T15:42:56.222283+0000","last_became_peered":"2023-05-22T15:42:56.222283+0000","last_unstale":"2023-05-22T15:42:56.603145+0000","last_undegraded":"2023-05-22T15:42:56.603145+0000","last_fullsized":"2023-05-22T15:42:56.603145+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:55.181398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:55.181398+0000","last_clean_scrub_stamp":"2023-05-22T15:42:55.181398+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:35:57.000336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.6","version":"0'0","reported_seq":9,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.634802+0000","last_change":"2023-05-22T15:42:56.631350+0000","last_active":"2023-05-22T15:42:56.634802+0000","last_peered":"2023-05-22T15:42:56.634802+0000","last_clean":"2023-05-22T15:42:56.634802+0000","last_became_active":"2023-05-22T15:42:56.630605+0000","last_became_peered":"2023-05-22T15:42:56.630605+0000","last_unstale":"2023-05-22T15:42:56.634802+0000","last_undegraded":"2023-05-22T15:42:56.634802+0000","last_fullsized":"2023-05-22T15:42:56.634802+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_clean_scrub_stamp":"2023-05-22T15:42:56.185172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:45:20.475919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.7","version":"103'2","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.609334+0000","last_change":"2023-05-22T15:42:56.234291+0000","last_active":"2023-05-22T15:42:56.609334+0000","last_peered":"2023-05-22T15:42:56.609334+0000","last_clean":"2023-05-22T15:42:56.609334+0000","last_became_active":"2023-05-22T15:42:56.234013+0000","last_became_peered":"2023-05-22T15:42:56.234013+0000","last_unstale":"2023-05-22T15:42:56.609334+0000","last_undegraded":"2023-05-22T15:42:56.609334+0000","last_fullsized":"2023-05-22T15:42:56.609334+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:55.181398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:55.181398+0000","last_clean_scrub_stamp":"2023-05-22T15:42:55.181398+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:43:51.407055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.7","version":"0'0","reported_seq":9,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.649123+0000","last_change":"2023-05-22T15:42:56.647851+0000","last_active":"2023-05-22T15:42:56.649123+0000","last_peered":"2023-05-22T15:42:56.649123+0000","last_clean":"2023-05-22T15:42:56.649123+0000","last_became_active":"2023-05-22T15:42:56.647280+0000","last_became_peered":"2023-05-22T15:42:56.647280+0000","last_unstale":"2023-05-22T15:42:56.649123+0000","last_undegraded":"2023-05-22T15:42:56.649123+0000","last_fullsized":"2023-05-22T15:42:56.649123+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_clean_scrub_stamp":"2023-05-22T15:42:56.185172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:58:13.431047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":10,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.590994+0000","last_change":"2023-05-22T15:42:56.216354+0000","last_active":"2023-05-22T15:42:56.590994+0000","last_peered":"2023-05-22T15:42:56.590994+0000","last_clean":"2023-05-22T15:42:56.590994+0000","last_became_active":"2023-05-22T15:42:56.215931+0000","last_became_peered":"2023-05-22T15:42:56.215931+0000","last_unstale":"2023-05-22T15:42:56.590994+0000","last_undegraded":"2023-05-22T15:42:56.590994+0000","last_fullsized":"2023-05-22T15:42:56.590994+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:55.181398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:55.181398+0000","last_clean_scrub_stamp":"2023-05-22T15:42:55.181398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T18:09:24.459116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":9,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.646200+0000","last_change":"2023-05-22T15:42:56.638543+0000","last_active":"2023-05-22T15:42:56.646200+0000","last_peered":"2023-05-22T15:42:56.646200+0000","last_clean":"2023-05-22T15:42:56.646200+0000","last_became_active":"2023-05-22T15:42:56.635593+0000","last_became_peered":"2023-05-22T15:42:56.635593+0000","last_unstale":"2023-05-22T15:42:56.646200+0000","last_undegraded":"2023-05-22T15:42:56.646200+0000","last_fullsized":"2023-05-22T15:42:56.646200+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_clean_scrub_stamp":"2023-05-22T15:42:56.185172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T03:30:09.923365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":10,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.596605+0000","last_change":"2023-05-22T15:42:56.245367+0000","last_active":"2023-05-22T15:42:56.596605+0000","last_peered":"2023-05-22T15:42:56.596605+0000","last_clean":"2023-05-22T15:42:56.596605+0000","last_became_active":"2023-05-22T15:42:56.244621+0000","last_became_peered":"2023-05-22T15:42:56.244621+0000","last_unstale":"2023-05-22T15:42:56.596605+0000","last_undegraded":"2023-05-22T15:42:56.596605+0000","last_fullsized":"2023-05-22T15:42:56.596605+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:55.181398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:55.181398+0000","last_clean_scrub_stamp":"2023-05-22T15:42:55.181398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:25:22.447013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":9,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.611938+0000","last_change":"2023-05-22T15:42:56.597061+0000","last_active":"2023-05-22T15:42:56.611938+0000","last_peered":"2023-05-22T15:42:56.611938+0000","last_clean":"2023-05-22T15:42:56.611938+0000","last_became_active":"2023-05-22T15:42:56.596241+0000","last_became_peered":"2023-05-22T15:42:56.596241+0000","last_unstale":"2023-05-22T15:42:56.611938+0000","last_undegraded":"2023-05-22T15:42:56.611938+0000","last_fullsized":"2023-05-22T15:42:56.611938+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_clean_scrub_stamp":"2023-05-22T15:42:56.185172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:58:13.431047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":10,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.617492+0000","last_change":"2023-05-22T15:42:56.245487+0000","last_active":"2023-05-22T15:42:56.617492+0000","last_peered":"2023-05-22T15:42:56.617492+0000","last_clean":"2023-05-22T15:42:56.617492+0000","last_became_active":"2023-05-22T15:42:56.244894+0000","last_became_peered":"2023-05-22T15:42:56.244894+0000","last_unstale":"2023-05-22T15:42:56.617492+0000","last_undegraded":"2023-05-22T15:42:56.617492+0000","last_fullsized":"2023-05-22T15:42:56.617492+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:55.181398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:55.181398+0000","last_clean_scrub_stamp":"2023-05-22T15:42:55.181398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T18:09:24.459116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":9,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.630176+0000","last_change":"2023-05-22T15:42:56.603008+0000","last_active":"2023-05-22T15:42:56.630176+0000","last_peered":"2023-05-22T15:42:56.630176+0000","last_clean":"2023-05-22T15:42:56.630176+0000","last_became_active":"2023-05-22T15:42:56.602175+0000","last_became_peered":"2023-05-22T15:42:56.602175+0000","last_unstale":"2023-05-22T15:42:56.630176+0000","last_undegraded":"2023-05-22T15:42:56.630176+0000","last_fullsized":"2023-05-22T15:42:56.630176+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_clean_scrub_stamp":"2023-05-22T15:42:56.185172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T18:12:07.084825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":10,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.572651+0000","last_change":"2023-05-22T15:42:56.222791+0000","last_active":"2023-05-22T15:42:56.572651+0000","last_peered":"2023-05-22T15:42:56.572651+0000","last_clean":"2023-05-22T15:42:56.572651+0000","last_became_active":"2023-05-22T15:42:56.222498+0000","last_became_peered":"2023-05-22T15:42:56.222498+0000","last_unstale":"2023-05-22T15:42:56.572651+0000","last_undegraded":"2023-05-22T15:42:56.572651+0000","last_fullsized":"2023-05-22T15:42:56.572651+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:55.181398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:55.181398+0000","last_clean_scrub_stamp":"2023-05-22T15:42:55.181398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:33:55.024640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":9,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.606919+0000","last_change":"2023-05-22T15:42:56.602967+0000","last_active":"2023-05-22T15:42:56.606919+0000","last_peered":"2023-05-22T15:42:56.606919+0000","last_clean":"2023-05-22T15:42:56.606919+0000","last_became_active":"2023-05-22T15:42:56.602702+0000","last_became_peered":"2023-05-22T15:42:56.602702+0000","last_unstale":"2023-05-22T15:42:56.606919+0000","last_undegraded":"2023-05-22T15:42:56.606919+0000","last_fullsized":"2023-05-22T15:42:56.606919+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_clean_scrub_stamp":"2023-05-22T15:42:56.185172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:26:51.212963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":10,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.584269+0000","last_change":"2023-05-22T15:42:56.234244+0000","last_active":"2023-05-22T15:42:56.584269+0000","last_peered":"2023-05-22T15:42:56.584269+0000","last_clean":"2023-05-22T15:42:56.584269+0000","last_became_active":"2023-05-22T15:42:56.232845+0000","last_became_peered":"2023-05-22T15:42:56.232845+0000","last_unstale":"2023-05-22T15:42:56.584269+0000","last_undegraded":"2023-05-22T15:42:56.584269+0000","last_fullsized":"2023-05-22T15:42:56.584269+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:55.181398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:55.181398+0000","last_clean_scrub_stamp":"2023-05-22T15:42:55.181398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:52:05.053711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":9,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.637432+0000","last_change":"2023-05-22T15:42:56.625205+0000","last_active":"2023-05-22T15:42:56.637432+0000","last_peered":"2023-05-22T15:42:56.637432+0000","last_clean":"2023-05-22T15:42:56.637432+0000","last_became_active":"2023-05-22T15:42:56.624194+0000","last_became_peered":"2023-05-22T15:42:56.624194+0000","last_unstale":"2023-05-22T15:42:56.637432+0000","last_undegraded":"2023-05-22T15:42:56.637432+0000","last_fullsized":"2023-05-22T15:42:56.637432+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_clean_scrub_stamp":"2023-05-22T15:42:56.185172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:48:06.492503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":10,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.606300+0000","last_change":"2023-05-22T15:42:56.245513+0000","last_active":"2023-05-22T15:42:56.606300+0000","last_peered":"2023-05-22T15:42:56.606300+0000","last_clean":"2023-05-22T15:42:56.606300+0000","last_became_active":"2023-05-22T15:42:56.244769+0000","last_became_peered":"2023-05-22T15:42:56.244769+0000","last_unstale":"2023-05-22T15:42:56.606300+0000","last_undegraded":"2023-05-22T15:42:56.606300+0000","last_fullsized":"2023-05-22T15:42:56.606300+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:55.181398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:55.181398+0000","last_clean_scrub_stamp":"2023-05-22T15:42:55.181398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:04:20.717819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":9,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.640974+0000","last_change":"2023-05-22T15:42:56.626621+0000","last_active":"2023-05-22T15:42:56.640974+0000","last_peered":"2023-05-22T15:42:56.640974+0000","last_clean":"2023-05-22T15:42:56.640974+0000","last_became_active":"2023-05-22T15:42:56.625149+0000","last_became_peered":"2023-05-22T15:42:56.625149+0000","last_unstale":"2023-05-22T15:42:56.640974+0000","last_undegraded":"2023-05-22T15:42:56.640974+0000","last_fullsized":"2023-05-22T15:42:56.640974+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_clean_scrub_stamp":"2023-05-22T15:42:56.185172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:49:38.992717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":10,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.582503+0000","last_change":"2023-05-22T15:42:56.245693+0000","last_active":"2023-05-22T15:42:56.582503+0000","last_peered":"2023-05-22T15:42:56.582503+0000","last_clean":"2023-05-22T15:42:56.582503+0000","last_became_active":"2023-05-22T15:42:56.244970+0000","last_became_peered":"2023-05-22T15:42:56.244970+0000","last_unstale":"2023-05-22T15:42:56.582503+0000","last_undegraded":"2023-05-22T15:42:56.582503+0000","last_fullsized":"2023-05-22T15:42:56.582503+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:55.181398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:55.181398+0000","last_clean_scrub_stamp":"2023-05-22T15:42:55.181398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:53:51.388237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":9,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.631034+0000","last_change":"2023-05-22T15:42:56.629886+0000","last_active":"2023-05-22T15:42:56.631034+0000","last_peered":"2023-05-22T15:42:56.631034+0000","last_clean":"2023-05-22T15:42:56.631034+0000","last_became_active":"2023-05-22T15:42:56.629626+0000","last_became_peered":"2023-05-22T15:42:56.629626+0000","last_unstale":"2023-05-22T15:42:56.631034+0000","last_undegraded":"2023-05-22T15:42:56.631034+0000","last_fullsized":"2023-05-22T15:42:56.631034+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_clean_scrub_stamp":"2023-05-22T15:42:56.185172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T18:07:13.587805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"103'1","reported_seq":11,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.633953+0000","last_change":"2023-05-22T15:42:56.235686+0000","last_active":"2023-05-22T15:42:56.633953+0000","last_peered":"2023-05-22T15:42:56.633953+0000","last_clean":"2023-05-22T15:42:56.633953+0000","last_became_active":"2023-05-22T15:42:56.234953+0000","last_became_peered":"2023-05-22T15:42:56.234953+0000","last_unstale":"2023-05-22T15:42:56.633953+0000","last_undegraded":"2023-05-22T15:42:56.633953+0000","last_fullsized":"2023-05-22T15:42:56.633953+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:55.181398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:55.181398+0000","last_clean_scrub_stamp":"2023-05-22T15:42:55.181398+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:53:51.388237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"92'183","reported_seq":481,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.223463+0000","last_change":"2023-05-22T15:26:39.318204+0000","last_active":"2023-05-22T15:42:56.223463+0000","last_peered":"2023-05-22T15:42:56.223463+0000","last_clean":"2023-05-22T15:42:56.223463+0000","last_became_active":"2023-05-22T15:26:39.315370+0000","last_became_peered":"2023-05-22T15:26:39.315370+0000","last_unstale":"2023-05-22T15:42:56.223463+0000","last_undegraded":"2023-05-22T15:42:56.223463+0000","last_fullsized":"2023-05-22T15:42:56.223463+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:38.020076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:38.020076+0000","last_clean_scrub_stamp":"2023-05-22T15:26:38.020076+0000","objects_scrubbed":0,"log_size":183,"log_dups_size":0,"ondisk_log_size":183,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:06:39.207543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":692,"num_read_kb":1336,"num_write":279,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":97,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.224737+0000","last_change":"2023-05-22T15:26:44.971951+0000","last_active":"2023-05-22T15:42:56.224737+0000","last_peered":"2023-05-22T15:42:56.224737+0000","last_clean":"2023-05-22T15:42:56.224737+0000","last_became_active":"2023-05-22T15:26:44.970872+0000","last_became_peered":"2023-05-22T15:26:44.970872+0000","last_unstale":"2023-05-22T15:42:56.224737+0000","last_undegraded":"2023-05-22T15:42:56.224737+0000","last_fullsized":"2023-05-22T15:42:56.224737+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:55:34.752949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.f","version":"0'0","reported_seq":9,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.646497+0000","last_change":"2023-05-22T15:42:56.633999+0000","last_active":"2023-05-22T15:42:56.646497+0000","last_peered":"2023-05-22T15:42:56.646497+0000","last_clean":"2023-05-22T15:42:56.646497+0000","last_became_active":"2023-05-22T15:42:56.633336+0000","last_became_peered":"2023-05-22T15:42:56.633336+0000","last_unstale":"2023-05-22T15:42:56.646497+0000","last_undegraded":"2023-05-22T15:42:56.646497+0000","last_fullsized":"2023-05-22T15:42:56.646497+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_clean_scrub_stamp":"2023-05-22T15:42:56.185172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T18:12:07.084825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.e","version":"103'2","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.602596+0000","last_change":"2023-05-22T15:42:56.240197+0000","last_active":"2023-05-22T15:42:56.602596+0000","last_peered":"2023-05-22T15:42:56.602596+0000","last_clean":"2023-05-22T15:42:56.602596+0000","last_became_active":"2023-05-22T15:42:56.239901+0000","last_became_peered":"2023-05-22T15:42:56.239901+0000","last_unstale":"2023-05-22T15:42:56.602596+0000","last_undegraded":"2023-05-22T15:42:56.602596+0000","last_fullsized":"2023-05-22T15:42:56.602596+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:55.181398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:55.181398+0000","last_clean_scrub_stamp":"2023-05-22T15:42:55.181398+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:23:14.852630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":94,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.223238+0000","last_change":"2023-05-22T15:26:44.977198+0000","last_active":"2023-05-22T15:42:56.223238+0000","last_peered":"2023-05-22T15:42:56.223238+0000","last_clean":"2023-05-22T15:42:56.223238+0000","last_became_active":"2023-05-22T15:26:44.976445+0000","last_became_peered":"2023-05-22T15:26:44.976445+0000","last_unstale":"2023-05-22T15:42:56.223238+0000","last_undegraded":"2023-05-22T15:42:56.223238+0000","last_fullsized":"2023-05-22T15:42:56.223238+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:12:41.464054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.e","version":"0'0","reported_seq":9,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.643988+0000","last_change":"2023-05-22T15:42:56.639805+0000","last_active":"2023-05-22T15:42:56.643988+0000","last_peered":"2023-05-22T15:42:56.643988+0000","last_clean":"2023-05-22T15:42:56.643988+0000","last_became_active":"2023-05-22T15:42:56.639089+0000","last_became_peered":"2023-05-22T15:42:56.639089+0000","last_unstale":"2023-05-22T15:42:56.643988+0000","last_undegraded":"2023-05-22T15:42:56.643988+0000","last_fullsized":"2023-05-22T15:42:56.643988+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_clean_scrub_stamp":"2023-05-22T15:42:56.185172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:01:28.084697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.f","version":"103'1","reported_seq":11,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.646773+0000","last_change":"2023-05-22T15:42:56.247444+0000","last_active":"2023-05-22T15:42:56.646773+0000","last_peered":"2023-05-22T15:42:56.646773+0000","last_clean":"2023-05-22T15:42:56.646773+0000","last_became_active":"2023-05-22T15:42:56.245737+0000","last_became_peered":"2023-05-22T15:42:56.245737+0000","last_unstale":"2023-05-22T15:42:56.646773+0000","last_undegraded":"2023-05-22T15:42:56.646773+0000","last_fullsized":"2023-05-22T15:42:56.646773+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:55.181398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:55.181398+0000","last_clean_scrub_stamp":"2023-05-22T15:42:55.181398+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:28:44.717288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":92,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.224362+0000","last_change":"2023-05-22T15:26:44.969469+0000","last_active":"2023-05-22T15:42:56.224362+0000","last_peered":"2023-05-22T15:42:56.224362+0000","last_clean":"2023-05-22T15:42:56.224362+0000","last_became_active":"2023-05-22T15:26:44.969083+0000","last_became_peered":"2023-05-22T15:26:44.969083+0000","last_unstale":"2023-05-22T15:42:56.224362+0000","last_undegraded":"2023-05-22T15:42:56.224362+0000","last_fullsized":"2023-05-22T15:42:56.224362+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:29:23.670067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.d","version":"0'0","reported_seq":9,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.611994+0000","last_change":"2023-05-22T15:42:56.597937+0000","last_active":"2023-05-22T15:42:56.611994+0000","last_peered":"2023-05-22T15:42:56.611994+0000","last_clean":"2023-05-22T15:42:56.611994+0000","last_became_active":"2023-05-22T15:42:56.597160+0000","last_became_peered":"2023-05-22T15:42:56.597160+0000","last_unstale":"2023-05-22T15:42:56.611994+0000","last_undegraded":"2023-05-22T15:42:56.611994+0000","last_fullsized":"2023-05-22T15:42:56.611994+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_clean_scrub_stamp":"2023-05-22T15:42:56.185172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:59:56.263719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":10,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.573067+0000","last_change":"2023-05-22T15:42:56.231395+0000","last_active":"2023-05-22T15:42:56.573067+0000","last_peered":"2023-05-22T15:42:56.573067+0000","last_clean":"2023-05-22T15:42:56.573067+0000","last_became_active":"2023-05-22T15:42:56.231099+0000","last_became_peered":"2023-05-22T15:42:56.231099+0000","last_unstale":"2023-05-22T15:42:56.573067+0000","last_undegraded":"2023-05-22T15:42:56.573067+0000","last_fullsized":"2023-05-22T15:42:56.573067+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:55.181398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:55.181398+0000","last_clean_scrub_stamp":"2023-05-22T15:42:55.181398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:58:12.427273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":97,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.224671+0000","last_change":"2023-05-22T15:26:44.976632+0000","last_active":"2023-05-22T15:42:56.224671+0000","last_peered":"2023-05-22T15:42:56.224671+0000","last_clean":"2023-05-22T15:42:56.224671+0000","last_became_active":"2023-05-22T15:26:44.975546+0000","last_became_peered":"2023-05-22T15:26:44.975546+0000","last_unstale":"2023-05-22T15:42:56.224671+0000","last_undegraded":"2023-05-22T15:42:56.224671+0000","last_fullsized":"2023-05-22T15:42:56.224671+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:42:57.973030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.b","version":"0'0","reported_seq":9,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.629940+0000","last_change":"2023-05-22T15:42:56.601620+0000","last_active":"2023-05-22T15:42:56.629940+0000","last_peered":"2023-05-22T15:42:56.629940+0000","last_clean":"2023-05-22T15:42:56.629940+0000","last_became_active":"2023-05-22T15:42:56.600789+0000","last_became_peered":"2023-05-22T15:42:56.600789+0000","last_unstale":"2023-05-22T15:42:56.629940+0000","last_undegraded":"2023-05-22T15:42:56.629940+0000","last_fullsized":"2023-05-22T15:42:56.629940+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_clean_scrub_stamp":"2023-05-22T15:42:56.185172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:48:43.959640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":10,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.577390+0000","last_change":"2023-05-22T15:42:56.226626+0000","last_active":"2023-05-22T15:42:56.577390+0000","last_peered":"2023-05-22T15:42:56.577390+0000","last_clean":"2023-05-22T15:42:56.577390+0000","last_became_active":"2023-05-22T15:42:56.224567+0000","last_became_peered":"2023-05-22T15:42:56.224567+0000","last_unstale":"2023-05-22T15:42:56.577390+0000","last_undegraded":"2023-05-22T15:42:56.577390+0000","last_fullsized":"2023-05-22T15:42:56.577390+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:55.181398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:55.181398+0000","last_clean_scrub_stamp":"2023-05-22T15:42:55.181398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T03:04:27.047889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":86,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.214969+0000","last_change":"2023-05-22T15:26:45.091475+0000","last_active":"2023-05-22T15:42:56.214969+0000","last_peered":"2023-05-22T15:42:56.214969+0000","last_clean":"2023-05-22T15:42:56.214969+0000","last_became_active":"2023-05-22T15:26:45.091143+0000","last_became_peered":"2023-05-22T15:26:45.091143+0000","last_unstale":"2023-05-22T15:42:56.214969+0000","last_undegraded":"2023-05-22T15:42:56.214969+0000","last_fullsized":"2023-05-22T15:42:56.214969+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:12:41.464054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":9,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.600088+0000","last_change":"2023-05-22T15:42:56.598005+0000","last_active":"2023-05-22T15:42:56.600088+0000","last_peered":"2023-05-22T15:42:56.600088+0000","last_clean":"2023-05-22T15:42:56.600088+0000","last_became_active":"2023-05-22T15:42:56.597699+0000","last_became_peered":"2023-05-22T15:42:56.597699+0000","last_unstale":"2023-05-22T15:42:56.600088+0000","last_undegraded":"2023-05-22T15:42:56.600088+0000","last_fullsized":"2023-05-22T15:42:56.600088+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_clean_scrub_stamp":"2023-05-22T15:42:56.185172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:53:23.214996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"103'2","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.591730+0000","last_change":"2023-05-22T15:42:56.228716+0000","last_active":"2023-05-22T15:42:56.591730+0000","last_peered":"2023-05-22T15:42:56.591730+0000","last_clean":"2023-05-22T15:42:56.591730+0000","last_became_active":"2023-05-22T15:42:56.228373+0000","last_became_peered":"2023-05-22T15:42:56.228373+0000","last_unstale":"2023-05-22T15:42:56.591730+0000","last_undegraded":"2023-05-22T15:42:56.591730+0000","last_fullsized":"2023-05-22T15:42:56.591730+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:55.181398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:55.181398+0000","last_clean_scrub_stamp":"2023-05-22T15:42:55.181398+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:33:41.882434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":93,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.223741+0000","last_change":"2023-05-22T15:26:45.092367+0000","last_active":"2023-05-22T15:42:56.223741+0000","last_peered":"2023-05-22T15:42:56.223741+0000","last_clean":"2023-05-22T15:42:56.223741+0000","last_became_active":"2023-05-22T15:26:45.091951+0000","last_became_peered":"2023-05-22T15:26:45.091951+0000","last_unstale":"2023-05-22T15:42:56.223741+0000","last_undegraded":"2023-05-22T15:42:56.223741+0000","last_fullsized":"2023-05-22T15:42:56.223741+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:42:57.973030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":9,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.630003+0000","last_change":"2023-05-22T15:42:56.608910+0000","last_active":"2023-05-22T15:42:56.630003+0000","last_peered":"2023-05-22T15:42:56.630003+0000","last_clean":"2023-05-22T15:42:56.630003+0000","last_became_active":"2023-05-22T15:42:56.608054+0000","last_became_peered":"2023-05-22T15:42:56.608054+0000","last_unstale":"2023-05-22T15:42:56.630003+0000","last_undegraded":"2023-05-22T15:42:56.630003+0000","last_fullsized":"2023-05-22T15:42:56.630003+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_clean_scrub_stamp":"2023-05-22T15:42:56.185172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:31:37.786481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":10,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.586380+0000","last_change":"2023-05-22T15:42:56.228719+0000","last_active":"2023-05-22T15:42:56.586380+0000","last_peered":"2023-05-22T15:42:56.586380+0000","last_clean":"2023-05-22T15:42:56.586380+0000","last_became_active":"2023-05-22T15:42:56.228366+0000","last_became_peered":"2023-05-22T15:42:56.228366+0000","last_unstale":"2023-05-22T15:42:56.586380+0000","last_undegraded":"2023-05-22T15:42:56.586380+0000","last_fullsized":"2023-05-22T15:42:56.586380+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:55.181398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:55.181398+0000","last_clean_scrub_stamp":"2023-05-22T15:42:55.181398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:58:12.427273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":86,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.215002+0000","last_change":"2023-05-22T15:26:44.978858+0000","last_active":"2023-05-22T15:42:56.215002+0000","last_peered":"2023-05-22T15:42:56.215002+0000","last_clean":"2023-05-22T15:42:56.215002+0000","last_became_active":"2023-05-22T15:26:44.978443+0000","last_became_peered":"2023-05-22T15:26:44.978443+0000","last_unstale":"2023-05-22T15:42:56.215002+0000","last_undegraded":"2023-05-22T15:42:56.215002+0000","last_fullsized":"2023-05-22T15:42:56.215002+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:46:29.290243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":9,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.614857+0000","last_change":"2023-05-22T15:42:56.614340+0000","last_active":"2023-05-22T15:42:56.614857+0000","last_peered":"2023-05-22T15:42:56.614857+0000","last_clean":"2023-05-22T15:42:56.614857+0000","last_became_active":"2023-05-22T15:42:56.613999+0000","last_became_peered":"2023-05-22T15:42:56.613999+0000","last_unstale":"2023-05-22T15:42:56.614857+0000","last_undegraded":"2023-05-22T15:42:56.614857+0000","last_fullsized":"2023-05-22T15:42:56.614857+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_clean_scrub_stamp":"2023-05-22T15:42:56.185172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:28:45.721063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":10,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.576955+0000","last_change":"2023-05-22T15:42:56.244970+0000","last_active":"2023-05-22T15:42:56.576955+0000","last_peered":"2023-05-22T15:42:56.576955+0000","last_clean":"2023-05-22T15:42:56.576955+0000","last_became_active":"2023-05-22T15:42:56.244657+0000","last_became_peered":"2023-05-22T15:42:56.244657+0000","last_unstale":"2023-05-22T15:42:56.576955+0000","last_undegraded":"2023-05-22T15:42:56.576955+0000","last_fullsized":"2023-05-22T15:42:56.576955+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:55.181398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:55.181398+0000","last_clean_scrub_stamp":"2023-05-22T15:42:55.181398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:33:14.769505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":88,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.233924+0000","last_change":"2023-05-22T15:26:44.982421+0000","last_active":"2023-05-22T15:42:56.233924+0000","last_peered":"2023-05-22T15:42:56.233924+0000","last_clean":"2023-05-22T15:42:56.233924+0000","last_became_active":"2023-05-22T15:26:44.981586+0000","last_became_peered":"2023-05-22T15:26:44.981586+0000","last_unstale":"2023-05-22T15:42:56.233924+0000","last_undegraded":"2023-05-22T15:42:56.233924+0000","last_fullsized":"2023-05-22T15:42:56.233924+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:14:11.156544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":13,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":12,"num_pg":32,"stat_sum":{"num_bytes":2330,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":692,"num_read_kb":1336,"num_write":279,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":183,"ondisk_log_size":183,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575045,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30560,"kb_used_data":1816,"kb_used_omap":0,"kb_used_meta":28736,"kb_avail":94341280,"statfs":{"total":96636764160,"available":96605470720,"internally_reserved":0,"allocated":1859584,"data_stored":1251625,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":29425664},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 15:41:58 2023","interfaces":[{"interface":"back","average":{"1min":0.88900000000000001,"5min":0.86099999999999999,"15min":0.86699999999999999},"min":{"1min":0.624,"5min":0.44800000000000001,"15min":0.40899999999999997},"max":{"1min":1.1830000000000001,"5min":1.4199999999999999,"15min":1.7709999999999999},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.86899999999999999,"5min":0.871,"15min":0.876},"min":{"1min":0.63600000000000001,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":1.2050000000000001,"5min":1.3169999999999999,"15min":1.4239999999999999},"last":0.95899999999999996}]},{"osd":1,"last update":"Mon May 22 15:41:58 2023","interfaces":[{"interface":"back","average":{"1min":0.94799999999999995,"5min":0.98199999999999998,"15min":0.98199999999999998},"min":{"1min":0.77300000000000002,"5min":0.5,"15min":0.5},"max":{"1min":1.385,"5min":1.389,"15min":1.8300000000000001},"last":1.0980000000000001},{"interface":"front","average":{"1min":1.0229999999999999,"5min":1.018,"15min":1.0229999999999999},"min":{"1min":0.60799999999999998,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":1.2130000000000001,"5min":1.3169999999999999,"15min":2.161},"last":1.0580000000000001}]},{"osd":2,"last update":"Mon May 22 15:41:58 2023","interfaces":[{"interface":"back","average":{"1min":0.84899999999999998,"5min":0.87,"15min":0.90400000000000003},"min":{"1min":0.67300000000000004,"5min":0.49399999999999999,"15min":0.46800000000000003},"max":{"1min":1.181,"5min":1.2070000000000001,"15min":6.7279999999999998},"last":0.90000000000000002},{"interface":"front","average":{"1min":0.95799999999999996,"5min":0.94799999999999995,"15min":0.96299999999999997},"min":{"1min":0.59199999999999997,"5min":0.52300000000000002,"15min":0.51000000000000001},"max":{"1min":1.151,"5min":1.2569999999999999,"15min":6.8810000000000002},"last":1.0349999999999999}]},{"osd":3,"last update":"Mon May 22 15:41:58 2023","interfaces":[{"interface":"back","average":{"1min":1.0109999999999999,"5min":0.95299999999999996,"15min":0.96099999999999997},"min":{"1min":0.70199999999999996,"5min":0.55900000000000005,"15min":0.53800000000000003},"max":{"1min":1.254,"5min":1.3360000000000001,"15min":3.0569999999999999},"last":0.69999999999999996},{"interface":"front","average":{"1min":0.91500000000000004,"5min":0.96599999999999997,"15min":0.98399999999999999},"min":{"1min":0.68200000000000005,"5min":0.61799999999999999,"15min":0.54900000000000004},"max":{"1min":1.2989999999999999,"5min":1.3029999999999999,"15min":1.6879999999999999},"last":1.0029999999999999}]},{"osd":4,"last update":"Mon May 22 15:41:58 2023","interfaces":[{"interface":"back","average":{"1min":0.83599999999999997,"5min":0.89600000000000002,"15min":0.88400000000000001},"min":{"1min":0.62,"5min":0.57399999999999995,"15min":0.57399999999999995},"max":{"1min":1.093,"5min":1.621,"15min":1.621},"last":0.86599999999999999},{"interface":"front","average":{"1min":0.98299999999999998,"5min":0.99199999999999999,"15min":0.98299999999999998},"min":{"1min":0.749,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.359,"5min":1.4319999999999999,"15min":1.958},"last":0.91600000000000004}]},{"osd":5,"last update":"Mon May 22 15:41:58 2023","interfaces":[{"interface":"back","average":{"1min":0.995,"5min":1.022,"15min":1.002},"min":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.56699999999999995},"max":{"1min":1.3939999999999999,"5min":1.409,"15min":1.5029999999999999},"last":0.82599999999999996},{"interface":"front","average":{"1min":1.1220000000000001,"5min":1.121,"15min":1.1359999999999999},"min":{"1min":0.88800000000000001,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":1.325,"5min":1.5229999999999999,"15min":5.0839999999999996},"last":1.119}]},{"osd":6,"last update":"Mon May 22 15:41:58 2023","interfaces":[{"interface":"back","average":{"1min":1.0169999999999999,"5min":1.0449999999999999,"15min":1.071},"min":{"1min":0.73399999999999999,"5min":0.65800000000000003,"15min":0.65800000000000003},"max":{"1min":1.256,"5min":1.484,"15min":5.1719999999999997},"last":1.079},{"interface":"front","average":{"1min":1.109,"5min":1.133,"15min":1.1339999999999999},"min":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"max":{"1min":1.462,"5min":1.702,"15min":2.04},"last":0.80100000000000005}]}]},{"osd":6,"up_from":13,"seq":55834575046,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30348,"kb_used_data":1220,"kb_used_omap":0,"kb_used_meta":29120,"kb_avail":94341492,"statfs":{"total":96636764160,"available":96605687808,"internally_reserved":0,"allocated":1249280,"data_stored":653739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":29818880},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 15:42:06 2023","interfaces":[{"interface":"back","average":{"1min":0.86599999999999999,"5min":0.89000000000000001,"15min":0.89800000000000002},"min":{"1min":0.59299999999999997,"5min":0.496,"15min":0.496},"max":{"1min":1.323,"5min":1.327,"15min":1.355},"last":0.71499999999999997},{"interface":"front","average":{"1min":0.99199999999999999,"5min":0.90300000000000002,"15min":0.90900000000000003},"min":{"1min":0.66200000000000003,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":1.264,"5min":1.5680000000000001,"15min":1.5680000000000001},"last":1.155}]},{"osd":1,"last update":"Mon May 22 15:42:06 2023","interfaces":[{"interface":"back","average":{"1min":0.92800000000000005,"5min":0.92800000000000005,"15min":0.91300000000000003},"min":{"1min":0.623,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":1.2709999999999999,"5min":1.3169999999999999,"15min":1.375},"last":0.82499999999999996},{"interface":"front","average":{"1min":0.93400000000000005,"5min":0.94499999999999995,"15min":0.94899999999999995},"min":{"1min":0.64400000000000002,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.2709999999999999,"5min":1.554,"15min":1.554},"last":1.325}]},{"osd":2,"last update":"Mon May 22 15:42:06 2023","interfaces":[{"interface":"back","average":{"1min":1.0029999999999999,"5min":0.97099999999999997,"15min":0.97999999999999998},"min":{"1min":0.72599999999999998,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":1.3660000000000001,"5min":1.425,"15min":1.425},"last":1.0309999999999999},{"interface":"front","average":{"1min":1.073,"5min":1.048,"15min":1.0509999999999999},"min":{"1min":0.84299999999999997,"5min":0.442,"15min":0.442},"max":{"1min":1.2909999999999999,"5min":1.5369999999999999,"15min":1.5369999999999999},"last":1.0640000000000001}]},{"osd":3,"last update":"Mon May 22 15:42:06 2023","interfaces":[{"interface":"back","average":{"1min":0.95799999999999996,"5min":0.99099999999999999,"15min":0.99399999999999999},"min":{"1min":0.72599999999999998,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":1.1830000000000001,"5min":1.679,"15min":1.679},"last":0.96699999999999997},{"interface":"front","average":{"1min":0.96799999999999997,"5min":0.95699999999999996,"15min":0.96099999999999997},"min":{"1min":0.62,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.159,"5min":1.4570000000000001,"15min":1.486},"last":0.85199999999999998}]},{"osd":4,"last update":"Mon May 22 15:42:06 2023","interfaces":[{"interface":"back","average":{"1min":0.85699999999999998,"5min":0.91700000000000004,"15min":0.93100000000000005},"min":{"1min":0.64900000000000002,"5min":0.60299999999999998,"15min":0.57599999999999996},"max":{"1min":1.1479999999999999,"5min":1.6859999999999999,"15min":1.6859999999999999},"last":0.91800000000000004},{"interface":"front","average":{"1min":0.96199999999999997,"5min":1.016,"15min":1.032},"min":{"1min":0.67700000000000005,"5min":0.622,"15min":0.61299999999999999},"max":{"1min":1.321,"5min":1.8300000000000001,"15min":1.8300000000000001},"last":1.2509999999999999}]},{"osd":5,"last update":"Mon May 22 15:42:06 2023","interfaces":[{"interface":"back","average":{"1min":1.01,"5min":0.98099999999999998,"15min":0.97999999999999998},"min":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.64400000000000002},"max":{"1min":1.4530000000000001,"5min":1.6499999999999999,"15min":1.6499999999999999},"last":1.1160000000000001},{"interface":"front","average":{"1min":1.0760000000000001,"5min":1.103,"15min":1.1000000000000001},"min":{"1min":0.68799999999999994,"5min":0.68000000000000005,"15min":0.68000000000000005},"max":{"1min":1.3919999999999999,"5min":1.607,"15min":1.7509999999999999},"last":1.2170000000000001}]},{"osd":7,"last update":"Mon May 22 15:42:06 2023","interfaces":[{"interface":"back","average":{"1min":1.1180000000000001,"5min":1.1160000000000001,"15min":1.1359999999999999},"min":{"1min":0.78700000000000003,"5min":0.68400000000000005,"15min":0.58399999999999996},"max":{"1min":1.3740000000000001,"5min":1.9179999999999999,"15min":1.9179999999999999},"last":1.3859999999999999},{"interface":"front","average":{"1min":1.1040000000000001,"5min":1.139,"15min":1.133},"min":{"1min":0.81899999999999995,"5min":0.74099999999999999,"15min":0.68999999999999995},"max":{"1min":1.3580000000000001,"5min":1.8520000000000001,"15min":1.8520000000000001},"last":1.282}]}]},{"osd":5,"up_from":13,"seq":55834575046,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":24296,"kb_used_data":1248,"kb_used_omap":0,"kb_used_meta":23040,"kb_avail":94347544,"statfs":{"total":96636764160,"available":96611885056,"internally_reserved":0,"allocated":1277952,"data_stored":662757,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":23592960},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 15:42:04 2023","interfaces":[{"interface":"back","average":{"1min":0.92100000000000004,"5min":0.86799999999999999,"15min":0.88},"min":{"1min":0.68899999999999995,"5min":0.54900000000000004,"15min":0.48399999999999999},"max":{"1min":1.2849999999999999,"5min":1.3440000000000001,"15min":1.3440000000000001},"last":0.93300000000000005},{"interface":"front","average":{"1min":0.93600000000000005,"5min":0.89800000000000002,"15min":0.88},"min":{"1min":0.58499999999999996,"5min":0.57499999999999996,"15min":0.497},"max":{"1min":1.2549999999999999,"5min":2.1699999999999999,"15min":2.1699999999999999},"last":0.58099999999999996}]},{"osd":1,"last update":"Mon May 22 15:42:04 2023","interfaces":[{"interface":"back","average":{"1min":1.099,"5min":1.004,"15min":0.996},"min":{"1min":0.87,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":1.337,"5min":1.9370000000000001,"15min":1.9370000000000001},"last":0.878},{"interface":"front","average":{"1min":1.028,"5min":0.999,"15min":1.02},"min":{"1min":0.67000000000000004,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.3680000000000001,"5min":1.3680000000000001,"15min":1.399},"last":1.1819999999999999}]},{"osd":2,"last update":"Mon May 22 15:42:04 2023","interfaces":[{"interface":"back","average":{"1min":0.95699999999999996,"5min":0.96299999999999997,"15min":0.93200000000000005},"min":{"1min":0.72699999999999998,"5min":0.504,"15min":0.504},"max":{"1min":1.2549999999999999,"5min":1.9830000000000001,"15min":1.9830000000000001},"last":0.82099999999999995},{"interface":"front","average":{"1min":1.0189999999999999,"5min":0.96099999999999997,"15min":0.93700000000000006},"min":{"1min":0.75900000000000001,"5min":0.60799999999999998,"15min":0.53800000000000003},"max":{"1min":1.321,"5min":2.1120000000000001,"15min":2.1120000000000001},"last":1.218}]},{"osd":3,"last update":"Mon May 22 15:42:04 2023","interfaces":[{"interface":"back","average":{"1min":1.0029999999999999,"5min":0.98899999999999999,"15min":0.95699999999999996},"min":{"1min":0.67000000000000004,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.2809999999999999,"5min":1.4419999999999999,"15min":1.6839999999999999},"last":0.79100000000000004},{"interface":"front","average":{"1min":0.97399999999999998,"5min":1.056,"15min":1.038},"min":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":1.1970000000000001,"5min":2.52,"15min":2.52},"last":0.999}]},{"osd":4,"last update":"Mon May 22 15:42:04 2023","interfaces":[{"interface":"back","average":{"1min":0.90900000000000003,"5min":0.97799999999999998,"15min":0.93799999999999994},"min":{"1min":0.68200000000000005,"5min":0.60599999999999998,"15min":0.57599999999999996},"max":{"1min":1.2609999999999999,"5min":2.226,"15min":2.226},"last":0.73799999999999999},{"interface":"front","average":{"1min":1.075,"5min":1.0649999999999999,"15min":1.038},"min":{"1min":0.63200000000000001,"5min":0.60199999999999998,"15min":0.60199999999999998},"max":{"1min":1.4239999999999999,"5min":1.6859999999999999,"15min":1.6859999999999999},"last":1.0940000000000001}]},{"osd":6,"last update":"Mon May 22 15:42:04 2023","interfaces":[{"interface":"back","average":{"1min":1.016,"5min":1.0840000000000001,"15min":1.0449999999999999},"min":{"1min":0.748,"5min":0.67900000000000005,"15min":0.61299999999999999},"max":{"1min":1.345,"5min":2.831,"15min":2.831},"last":1.2929999999999999},{"interface":"front","average":{"1min":1.085,"5min":1.1080000000000001,"15min":1.0780000000000001},"min":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.69299999999999995},"max":{"1min":1.3380000000000001,"5min":2.3690000000000002,"15min":2.3690000000000002},"last":1.046}]},{"osd":7,"last update":"Mon May 22 15:42:04 2023","interfaces":[{"interface":"back","average":{"1min":1.077,"5min":1.1639999999999999,"15min":1.125},"min":{"1min":0.70999999999999996,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":1.47,"5min":1.849,"15min":2.8260000000000001},"last":1.2609999999999999},{"interface":"front","average":{"1min":1.1579999999999999,"5min":1.22,"15min":1.1699999999999999},"min":{"1min":0.80400000000000005,"5min":0.75700000000000001,"15min":0.71699999999999997},"max":{"1min":1.3220000000000001,"5min":2.7429999999999999,"15min":2.7429999999999999},"last":1.135}]}]},{"osd":4,"up_from":13,"seq":55834575044,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":26072,"kb_used_data":1232,"kb_used_omap":0,"kb_used_meta":24832,"kb_avail":94345768,"statfs":{"total":96636764160,"available":96610066432,"internally_reserved":0,"allocated":1261568,"data_stored":661187,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":25427968},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 15:42:07 2023","interfaces":[{"interface":"back","average":{"1min":0.88,"5min":0.91300000000000003,"15min":0.90600000000000003},"min":{"1min":0.58299999999999996,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":1.3360000000000001,"5min":1.9910000000000001,"15min":1.9910000000000001},"last":0.91400000000000003},{"interface":"front","average":{"1min":0.96899999999999997,"5min":0.91900000000000004,"15min":0.93200000000000005},"min":{"1min":0.63500000000000001,"5min":0.373,"15min":0.373},"max":{"1min":1.306,"5min":1.3779999999999999,"15min":1.633},"last":1.0109999999999999}]},{"osd":1,"last update":"Mon May 22 15:42:07 2023","interfaces":[{"interface":"back","average":{"1min":1.0029999999999999,"5min":0.98199999999999998,"15min":0.97699999999999998},"min":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.54000000000000004},"max":{"1min":1.2689999999999999,"5min":1.383,"15min":1.3859999999999999},"last":0.63},{"interface":"front","average":{"1min":0.88900000000000001,"5min":0.93500000000000005,"15min":0.91200000000000003},"min":{"1min":0.60299999999999998,"5min":0.55100000000000005,"15min":0.51600000000000001},"max":{"1min":1.149,"5min":1.3120000000000001,"15min":1.3640000000000001},"last":0.56799999999999995}]},{"osd":2,"last update":"Mon May 22 15:42:07 2023","interfaces":[{"interface":"back","average":{"1min":0.92800000000000005,"5min":0.94699999999999995,"15min":0.94699999999999995},"min":{"1min":0.747,"5min":0.495,"15min":0.495},"max":{"1min":1.212,"5min":2.1019999999999999,"15min":2.1019999999999999},"last":1.2769999999999999},{"interface":"front","average":{"1min":1.0489999999999999,"5min":1.01,"15min":1.0109999999999999},"min":{"1min":0.77600000000000002,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":1.4390000000000001,"5min":1.4390000000000001,"15min":1.9770000000000001},"last":0.76600000000000001}]},{"osd":3,"last update":"Mon May 22 15:42:07 2023","interfaces":[{"interface":"back","average":{"1min":0.89800000000000002,"5min":0.91800000000000004,"15min":0.94299999999999995},"min":{"1min":0.72399999999999998,"5min":0.63100000000000001,"15min":0.54600000000000004},"max":{"1min":1.2490000000000001,"5min":1.29,"15min":1.665},"last":0.83099999999999996},{"interface":"front","average":{"1min":1.075,"5min":1.0529999999999999,"15min":1.038},"min":{"1min":0.85399999999999998,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":1.238,"5min":2.1720000000000002,"15min":2.1720000000000002},"last":1.1319999999999999}]},{"osd":5,"last update":"Mon May 22 15:42:07 2023","interfaces":[{"interface":"back","average":{"1min":1.0569999999999999,"5min":1.0409999999999999,"15min":1},"min":{"1min":0.83399999999999996,"5min":0.621,"15min":0.51200000000000001},"max":{"1min":1.373,"5min":1.472,"15min":1.7270000000000001},"last":0.873},{"interface":"front","average":{"1min":1.012,"5min":0.98599999999999999,"15min":0.97999999999999998},"min":{"1min":0.70799999999999996,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":1.325,"5min":1.401,"15min":1.6719999999999999},"last":0.96399999999999997}]},{"osd":6,"last update":"Mon May 22 15:42:07 2023","interfaces":[{"interface":"back","average":{"1min":1.0089999999999999,"5min":1.028,"15min":1.0029999999999999},"min":{"1min":0.64500000000000002,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":1.351,"5min":2.2349999999999999,"15min":2.2349999999999999},"last":1.2290000000000001},{"interface":"front","average":{"1min":1.1379999999999999,"5min":1.1419999999999999,"15min":1.1160000000000001},"min":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.69799999999999995},"max":{"1min":1.486,"5min":1.8220000000000001,"15min":2.0699999999999998},"last":1.103}]},{"osd":7,"last update":"Mon May 22 15:42:07 2023","interfaces":[{"interface":"back","average":{"1min":1.087,"5min":1.0580000000000001,"15min":1.0680000000000001},"min":{"1min":0.78200000000000003,"5min":0.71899999999999997,"15min":0.67700000000000005},"max":{"1min":1.3140000000000001,"5min":1.5800000000000001,"15min":1.6639999999999999},"last":1.069},{"interface":"front","average":{"1min":1.1930000000000001,"5min":1.1639999999999999,"15min":1.1459999999999999},"min":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.71499999999999997},"max":{"1min":1.4370000000000001,"5min":2.306,"15min":2.306},"last":1.327}]}]},{"osd":3,"up_from":13,"seq":55834575044,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35108,"kb_used_data":1244,"kb_used_omap":0,"kb_used_meta":33856,"kb_avail":94336732,"statfs":{"total":96636764160,"available":96600813568,"internally_reserved":0,"allocated":1273856,"data_stored":662735,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":34668544},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 15:42:54 2023","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.61199999999999999,"15min":0.628},"min":{"1min":0.312,"5min":0.19,"15min":0.19},"max":{"1min":1.1519999999999999,"5min":1.194,"15min":1.194},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.68700000000000006,"15min":0.69999999999999996},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.88300000000000001,"5min":1.1799999999999999,"15min":1.296},"last":0.373}]},{"osd":1,"last update":"Mon May 22 15:42:54 2023","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.61899999999999999,"15min":0.66300000000000003},"min":{"1min":0.315,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.80000000000000004,"5min":1.008,"15min":1.0600000000000001},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.71899999999999997,"15min":0.75800000000000001},"min":{"1min":0.34399999999999997,"5min":0.313,"15min":0.216},"max":{"1min":0.82299999999999995,"5min":1.222,"15min":1.222},"last":0.42599999999999999}]},{"osd":2,"last update":"Mon May 22 15:42:54 2023","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.76700000000000002,"15min":0.81200000000000006},"min":{"1min":0.41599999999999998,"5min":0.36599999999999999,"15min":0.32200000000000001},"max":{"1min":1.506,"5min":1.506,"15min":1.506},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.748,"15min":0.80200000000000005},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.29699999999999999},"max":{"1min":1.097,"5min":1.181,"15min":1.468},"last":0.57399999999999995}]},{"osd":4,"last update":"Mon May 22 15:42:54 2023","interfaces":[{"interface":"back","average":{"1min":0.84399999999999997,"5min":0.92300000000000004,"15min":0.95299999999999996},"min":{"1min":0.54500000000000004,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":1.413,"5min":1.413,"15min":1.4470000000000001},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.81799999999999995,"5min":0.94699999999999995,"15min":1.0009999999999999},"min":{"1min":0.64800000000000002,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.077,"5min":1.3740000000000001,"15min":1.4470000000000001},"last":0.81599999999999995}]},{"osd":5,"last update":"Mon May 22 15:42:54 2023","interfaces":[{"interface":"back","average":{"1min":0.83199999999999996,"5min":0.96099999999999997,"15min":1.01},"min":{"1min":0.53200000000000003,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":1.2070000000000001,"5min":1.3149999999999999,"15min":1.4339999999999999},"last":0.872},{"interface":"front","average":{"1min":0.94899999999999995,"5min":0.98999999999999999,"15min":1.0429999999999999},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.4299999999999999,"5min":1.4299999999999999,"15min":1.643},"last":0.68200000000000005}]},{"osd":6,"last update":"Mon May 22 15:42:54 2023","interfaces":[{"interface":"back","average":{"1min":0.85099999999999998,"5min":0.98099999999999998,"15min":1.0369999999999999},"min":{"1min":0.47699999999999998,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.1539999999999999,"5min":1.4139999999999999,"15min":1.863},"last":0.75600000000000001},{"interface":"front","average":{"1min":0.93100000000000005,"5min":1.0229999999999999,"15min":1.0720000000000001},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":1.2909999999999999,"5min":1.3109999999999999,"15min":1.704},"last":0.84199999999999997}]},{"osd":7,"last update":"Mon May 22 15:42:54 2023","interfaces":[{"interface":"back","average":{"1min":0.96699999999999997,"5min":1.0469999999999999,"15min":1.1000000000000001},"min":{"1min":0.58599999999999997,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":1.407,"5min":1.407,"15min":1.5640000000000001},"last":0.79300000000000004},{"interface":"front","average":{"1min":0.94099999999999995,"5min":1.024,"15min":1.0700000000000001},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":1.8700000000000001,"5min":1.8700000000000001,"15min":1.8700000000000001},"last":0.64900000000000002}]}]},{"osd":2,"up_from":13,"seq":55834575045,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":17372,"kb_used_data":1236,"kb_used_omap":0,"kb_used_meta":16128,"kb_avail":94354468,"statfs":{"total":96636764160,"available":96618975232,"internally_reserved":0,"allocated":1265664,"data_stored":661753,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":16515072},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 15:41:55 2023","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.66000000000000003,"15min":0.65000000000000002},"min":{"1min":0.499,"5min":0.35699999999999998,"15min":0.317},"max":{"1min":0.96999999999999997,"5min":1.8240000000000001,"15min":1.8240000000000001},"last":0.28599999999999998},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.77600000000000002,"15min":0.74299999999999999},"min":{"1min":0.52500000000000002,"5min":0.39600000000000002,"15min":0.253},"max":{"1min":1.0740000000000001,"5min":1.8160000000000001,"15min":1.8160000000000001},"last":0.57499999999999996}]},{"osd":1,"last update":"Mon May 22 15:41:55 2023","interfaces":[{"interface":"back","average":{"1min":0.80600000000000005,"5min":0.74399999999999999,"15min":0.73599999999999999},"min":{"1min":0.46700000000000003,"5min":0.27000000000000002,"15min":0.26900000000000002},"max":{"1min":1.0329999999999999,"5min":1.2,"15min":1.5309999999999999},"last":0.374},{"interface":"front","average":{"1min":0.877,"5min":0.83899999999999997,"15min":0.80700000000000005},"min":{"1min":0.60099999999999998,"5min":0.39600000000000002,"15min":0.29099999999999998},"max":{"1min":1.246,"5min":2.5089999999999999,"15min":2.5089999999999999},"last":0.63300000000000001}]},{"osd":3,"last update":"Mon May 22 15:41:55 2023","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.72799999999999998,"15min":0.72099999999999997},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.26700000000000002},"max":{"1min":0.96199999999999997,"5min":2.7629999999999999,"15min":2.7629999999999999},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.85699999999999998,"5min":0.89700000000000002,"15min":0.873},"min":{"1min":0.56100000000000005,"5min":0.44400000000000001,"15min":0.33000000000000002},"max":{"1min":1.042,"5min":2.6840000000000002,"15min":2.6840000000000002},"last":0.433}]},{"osd":4,"last update":"Mon May 22 15:41:55 2023","interfaces":[{"interface":"back","average":{"1min":1.012,"5min":1.0329999999999999,"15min":1.0109999999999999},"min":{"1min":0.67700000000000005,"5min":0.65800000000000003,"15min":0.55500000000000005},"max":{"1min":1.244,"5min":2.8039999999999998,"15min":2.8039999999999998},"last":0.76500000000000001},{"interface":"front","average":{"1min":1.0009999999999999,"5min":1.05,"15min":1.024},"min":{"1min":0.77000000000000002,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":1.3,"5min":2.0819999999999999,"15min":2.0819999999999999},"last":0.73699999999999999}]},{"osd":5,"last update":"Mon May 22 15:41:55 2023","interfaces":[{"interface":"back","average":{"1min":1.1080000000000001,"5min":1.0720000000000001,"15min":1.034},"min":{"1min":0.74099999999999999,"5min":0.629,"15min":0.55200000000000005},"max":{"1min":1.4099999999999999,"5min":2.3860000000000001,"15min":2.4569999999999999},"last":0.80300000000000005},{"interface":"front","average":{"1min":0.98099999999999998,"5min":1.006,"15min":0.998},"min":{"1min":0.77800000000000002,"5min":0.64100000000000001,"15min":0.56200000000000006},"max":{"1min":1.327,"5min":3.0299999999999998,"15min":3.0299999999999998},"last":0.60299999999999998}]},{"osd":6,"last update":"Mon May 22 15:41:55 2023","interfaces":[{"interface":"back","average":{"1min":1.008,"5min":1.0349999999999999,"15min":1.0389999999999999},"min":{"1min":0.72899999999999998,"5min":0.67400000000000004,"15min":0.56399999999999995},"max":{"1min":1.2250000000000001,"5min":2.883,"15min":3.488},"last":0.82999999999999996},{"interface":"front","average":{"1min":1.0820000000000001,"5min":1.1160000000000001,"15min":1.1000000000000001},"min":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.53400000000000003},"max":{"1min":1.365,"5min":3.1459999999999999,"15min":3.7330000000000001},"last":0.59199999999999997}]},{"osd":7,"last update":"Mon May 22 15:41:55 2023","interfaces":[{"interface":"back","average":{"1min":1.149,"5min":1.1499999999999999,"15min":1.123},"min":{"1min":0.77600000000000002,"5min":0.77000000000000002,"15min":0.58299999999999996},"max":{"1min":1.4970000000000001,"5min":2.5990000000000002,"15min":2.5990000000000002},"last":0.66100000000000003},{"interface":"front","average":{"1min":1.0860000000000001,"5min":1.1200000000000001,"15min":1.0980000000000001},"min":{"1min":0.65500000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":1.3839999999999999,"5min":3.8799999999999999,"15min":3.8799999999999999},"last":0.78200000000000003}]}]},{"osd":1,"up_from":13,"seq":55834575045,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":24292,"kb_used_data":1244,"kb_used_omap":0,"kb_used_meta":23040,"kb_avail":94347548,"statfs":{"total":96636764160,"available":96611889152,"internally_reserved":0,"allocated":1273856,"data_stored":661381,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":23592960},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 15:42:12 2023","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.63200000000000001,"15min":0.63800000000000001},"min":{"1min":0.37,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":1.2270000000000001,"5min":1.2270000000000001,"15min":1.7689999999999999},"last":0.495},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.78200000000000003,"15min":0.77100000000000002},"min":{"1min":0.38800000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":1.0649999999999999,"5min":1.2290000000000001,"15min":1.3799999999999999},"last":0.52900000000000003}]},{"osd":2,"last update":"Mon May 22 15:42:12 2023","interfaces":[{"interface":"back","average":{"1min":0.78700000000000003,"5min":0.76700000000000002,"15min":0.73299999999999998},"min":{"1min":0.51300000000000001,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":1.391,"5min":1.391,"15min":1.7290000000000001},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.79100000000000004,"5min":0.77200000000000002,"15min":0.75600000000000001},"min":{"1min":0.47799999999999998,"5min":0.35899999999999999,"15min":0.27800000000000002},"max":{"1min":1.2270000000000001,"5min":1.276,"15min":2.4049999999999998},"last":0.63500000000000001}]},{"osd":3,"last update":"Mon May 22 15:42:12 2023","interfaces":[{"interface":"back","average":{"1min":0.88600000000000001,"5min":0.82199999999999995,"15min":0.82899999999999996},"min":{"1min":0.41499999999999998,"5min":0.41399999999999998,"15min":0.317},"max":{"1min":1.657,"5min":1.657,"15min":1.657},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.89000000000000001,"5min":0.82399999999999995,"15min":0.82699999999999996},"min":{"1min":0.46000000000000002,"5min":0.315,"15min":0.315},"max":{"1min":2.0569999999999999,"5min":2.0569999999999999,"15min":7.5620000000000003},"last":0.58299999999999996}]},{"osd":4,"last update":"Mon May 22 15:42:12 2023","interfaces":[{"interface":"back","average":{"1min":1.002,"5min":0.96999999999999997,"15min":0.95099999999999996},"min":{"1min":0.58499999999999996,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":2.0169999999999999,"5min":2.0169999999999999,"15min":2.0169999999999999},"last":1.103},{"interface":"front","average":{"1min":1.03,"5min":0.98599999999999999,"15min":0.98899999999999999},"min":{"1min":0.64100000000000001,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":1.4650000000000001,"5min":1.5369999999999999,"15min":1.5369999999999999},"last":0.89600000000000002}]},{"osd":5,"last update":"Mon May 22 15:42:12 2023","interfaces":[{"interface":"back","average":{"1min":1.069,"5min":1.0349999999999999,"15min":1.0329999999999999},"min":{"1min":0.59899999999999998,"5min":0.443,"15min":0.443},"max":{"1min":1.9570000000000001,"5min":1.9570000000000001,"15min":1.9570000000000001},"last":0.93100000000000005},{"interface":"front","average":{"1min":1.0720000000000001,"5min":1.0509999999999999,"15min":1.048},"min":{"1min":0.68000000000000005,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":2.1200000000000001,"5min":2.1200000000000001,"15min":4.4710000000000001},"last":0.85299999999999998}]},{"osd":6,"last update":"Mon May 22 15:42:12 2023","interfaces":[{"interface":"back","average":{"1min":1.0449999999999999,"5min":1.012,"15min":1.0680000000000001},"min":{"1min":0.54000000000000004,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.5720000000000001,"5min":1.5720000000000001,"15min":14.968999999999999},"last":0.98499999999999999},{"interface":"front","average":{"1min":1.1319999999999999,"5min":1.083,"15min":1.1359999999999999},"min":{"1min":0.69799999999999995,"5min":0.61399999999999999,"15min":0.56699999999999995},"max":{"1min":1.7629999999999999,"5min":1.7629999999999999,"15min":13.242000000000001},"last":0.82499999999999996}]},{"osd":7,"last update":"Mon May 22 15:42:12 2023","interfaces":[{"interface":"back","average":{"1min":1.1559999999999999,"5min":1.1140000000000001,"15min":1.109},"min":{"1min":0.66200000000000003,"5min":0.56799999999999995,"15min":0.56799999999999995},"max":{"1min":1.865,"5min":1.865,"15min":1.9259999999999999},"last":0.77800000000000002},{"interface":"front","average":{"1min":1.1739999999999999,"5min":1.145,"15min":1.131},"min":{"1min":0.78000000000000003,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":2.2200000000000002,"5min":2.2200000000000002,"15min":5.681},"last":1.0129999999999999}]}]},{"osd":0,"up_from":13,"seq":55834575046,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":23584,"kb_used_data":1816,"kb_used_omap":0,"kb_used_meta":21760,"kb_avail":94348256,"statfs":{"total":96636764160,"available":96612614144,"internally_reserved":0,"allocated":1859584,"data_stored":1251589,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":22282240},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon May 22 15:42:07 2023","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.66100000000000003,"15min":0.66800000000000004},"min":{"1min":0.248,"5min":0.223,"15min":0.223},"max":{"1min":0.86899999999999999,"5min":1.077,"15min":1.3049999999999999},"last":0.36199999999999999},{"interface":"front","average":{"1min":0.76300000000000001,"5min":0.72999999999999998,"15min":0.73299999999999998},"min":{"1min":0.53600000000000003,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":1.0369999999999999,"5min":1.2390000000000001,"15min":1.2390000000000001},"last":0.34200000000000003}]},{"osd":2,"last update":"Mon May 22 15:42:07 2023","interfaces":[{"interface":"back","average":{"1min":0.73599999999999999,"5min":0.74199999999999999,"15min":0.74399999999999999},"min":{"1min":0.318,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":1.155,"5min":1.155,"15min":1.4059999999999999},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.79500000000000004,"5min":0.78000000000000003,"15min":0.79900000000000004},"min":{"1min":0.55400000000000005,"5min":0.32900000000000001,"15min":0.29699999999999999},"max":{"1min":1.05,"5min":1.1850000000000001,"15min":1.3999999999999999},"last":0.38500000000000001}]},{"osd":3,"last update":"Mon May 22 15:42:07 2023","interfaces":[{"interface":"back","average":{"1min":0.86699999999999999,"5min":0.755,"15min":0.76000000000000001},"min":{"1min":0.55600000000000005,"5min":0.40899999999999997,"15min":0.36599999999999999},"max":{"1min":1.9299999999999999,"5min":1.9299999999999999,"15min":1.9299999999999999},"last":0.31},{"interface":"front","average":{"1min":0.85799999999999998,"5min":0.86399999999999999,"15min":0.86799999999999999},"min":{"1min":0.35199999999999998,"5min":0.34399999999999997,"15min":0.33000000000000002},"max":{"1min":1.0669999999999999,"5min":1.3500000000000001,"15min":1.4570000000000001},"last":0.41399999999999998}]},{"osd":4,"last update":"Mon May 22 15:42:07 2023","interfaces":[{"interface":"back","average":{"1min":1.0429999999999999,"5min":0.94899999999999995,"15min":0.94799999999999995},"min":{"1min":0.751,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":2.923,"5min":2.923,"15min":2.923},"last":0.75700000000000001},{"interface":"front","average":{"1min":1.073,"5min":1.0269999999999999,"15min":1.022},"min":{"1min":0.86399999999999999,"5min":0.64600000000000002,"15min":0.54900000000000004},"max":{"1min":1.8160000000000001,"5min":1.8160000000000001,"15min":1.8160000000000001},"last":0.67400000000000004}]},{"osd":5,"last update":"Mon May 22 15:42:07 2023","interfaces":[{"interface":"back","average":{"1min":0.94699999999999995,"5min":1.014,"15min":1.038},"min":{"1min":0.66500000000000004,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.1040000000000001,"5min":1.4750000000000001,"15min":1.6850000000000001},"last":0.82899999999999996},{"interface":"front","average":{"1min":1.2170000000000001,"5min":1.032,"15min":1.01},"min":{"1min":0.76800000000000002,"5min":0.55300000000000005,"15min":0.54800000000000004},"max":{"1min":5.0519999999999996,"5min":5.0519999999999996,"15min":5.0519999999999996},"last":0.57199999999999995}]},{"osd":6,"last update":"Mon May 22 15:42:07 2023","interfaces":[{"interface":"back","average":{"1min":1.038,"5min":1.0740000000000001,"15min":1.075},"min":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":1.53,"5min":1.651,"15min":1.7450000000000001},"last":0.58699999999999997},{"interface":"front","average":{"1min":1.2749999999999999,"5min":1.139,"15min":1.103},"min":{"1min":0.91500000000000004,"5min":0.499,"15min":0.499},"max":{"1min":5.2640000000000002,"5min":5.2640000000000002,"15min":5.2640000000000002},"last":0.81000000000000005}]},{"osd":7,"last update":"Mon May 22 15:42:07 2023","interfaces":[{"interface":"back","average":{"1min":1.0700000000000001,"5min":1.1120000000000001,"15min":1.1180000000000001},"min":{"1min":0.70699999999999996,"5min":0.65700000000000003,"15min":0.56499999999999995},"max":{"1min":1.2789999999999999,"5min":1.6419999999999999,"15min":1.6419999999999999},"last":0.77500000000000002},{"interface":"front","average":{"1min":1.228,"5min":1.1000000000000001,"15min":1.087},"min":{"1min":0.85999999999999999,"5min":0.60299999999999998,"15min":0.58399999999999996},"max":{"1min":3.9060000000000001,"5min":3.9060000000000001,"15min":3.9060000000000001},"last":0.63300000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1570,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-05-22T15:42:59.592 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-05-22T15:42:59.748 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.743+0000 7ffb0eeb5640 1 Processor -- start 2023-05-22T15:42:59.749 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.743+0000 7ffb0eeb5640 1 -- start start 2023-05-22T15:42:59.749 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.743+0000 7ffb0eeb5640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7ffb080b7dd0 0x7ffb080ccab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:59.750 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.743+0000 7ffb0eeb5640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ffb08160250 0x7ffb080d0410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:59.750 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.743+0000 7ffb0eeb5640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb080cd170 con 0x7ffb0815f700 2023-05-22T15:42:59.750 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.743+0000 7ffb0eeb5640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb080d3360 con 0x7ffb080b7dd0 2023-05-22T15:42:59.750 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.743+0000 7ffb0eeb5640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb080d2ac0 con 0x7ffb08160250 2023-05-22T15:42:59.750 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.743+0000 7ffb07fff640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7ffb0815f700 0x7ffb0815fb40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:48752/0 (socket says 172.21.15.16:48752) 2023-05-22T15:42:59.750 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.743+0000 7ffb07fff640 1 -- 172.21.15.16:0/2890028382 learned_addr learned my addr 172.21.15.16:0/2890028382 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:42:59.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.743+0000 7ffb0cc51640 1 --2- 172.21.15.16:0/2890028382 >> v2:172.21.15.62:3300/0 conn(0x7ffb080b7dd0 0x7ffb080ccab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:59.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.743+0000 7ffb0d452640 1 --2- 172.21.15.16:0/2890028382 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ffb08160250 0x7ffb080d0410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:59.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.743+0000 7ffb077fe640 1 -- 172.21.15.16:0/2890028382 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3775793204 0 0) 0x7ffb080cd170 con 0x7ffb0815f700 2023-05-22T15:42:59.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.743+0000 7ffb077fe640 1 -- 172.21.15.16:0/2890028382 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffaf8003530 con 0x7ffb0815f700 2023-05-22T15:42:59.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.747+0000 7ffb077fe640 1 -- 172.21.15.16:0/2890028382 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1585060066 0 0) 0x7ffaf8003530 con 0x7ffb0815f700 2023-05-22T15:42:59.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.747+0000 7ffb077fe640 1 -- 172.21.15.16:0/2890028382 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ffb08160250 msgr2=0x7ffb080d0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:42:59.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.747+0000 7ffb077fe640 1 --2- 172.21.15.16:0/2890028382 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ffb08160250 0x7ffb080d0410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:59.753 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.747+0000 7ffb077fe640 1 -- 172.21.15.16:0/2890028382 >> v2:172.21.15.62:3300/0 conn(0x7ffb080b7dd0 msgr2=0x7ffb080ccab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:42:59.753 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.747+0000 7ffb077fe640 1 --2- 172.21.15.16:0/2890028382 >> v2:172.21.15.62:3300/0 conn(0x7ffb080b7dd0 0x7ffb080ccab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:59.753 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.747+0000 7ffb077fe640 1 -- 172.21.15.16:0/2890028382 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb080d2da0 con 0x7ffb0815f700 2023-05-22T15:42:59.753 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.747+0000 7ffb077fe640 1 -- 172.21.15.16:0/2890028382 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7ffaf4001680 con 0x7ffb0815f700 2023-05-22T15:42:59.753 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.747+0000 7ffb077fe640 1 -- 172.21.15.16:0/2890028382 >> v1:172.21.15.16:6789/0 conn(0x7ffb0815f700 legacy=0x7ffb0815fb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:59.753 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.747+0000 7ffb077fe640 1 --2- 172.21.15.16:0/2890028382 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ffb08160250 0x7ffaf8003980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:59.754 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.747+0000 7ffb077fe640 1 --2- 172.21.15.16:0/2890028382 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffaf8003f00 0x7ffaf80047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:59.754 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.747+0000 7ffb077fe640 1 --2- 172.21.15.16:0/2890028382 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffaf8004d10 0x7ffaf8005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:59.754 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.747+0000 7ffb077fe640 1 -- 172.21.15.16:0/2890028382 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb080d3360 con 0x7ffaf8003f00 2023-05-22T15:42:59.754 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.747+0000 7ffb077fe640 1 -- 172.21.15.16:0/2890028382 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb080d2ac0 con 0x7ffaf8004d10 2023-05-22T15:42:59.754 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.747+0000 7ffb077fe640 1 -- 172.21.15.16:0/2890028382 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffaf800b990 con 0x7ffb08160250 2023-05-22T15:42:59.754 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.747+0000 7ffb07fff640 1 --2- 172.21.15.16:0/2890028382 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffaf8003f00 0x7ffaf80047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:59.755 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.747+0000 7ffb0d452640 1 --2- 172.21.15.16:0/2890028382 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffaf8004d10 0x7ffaf8005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:59.755 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.747+0000 7ffb0cc51640 1 --2- 172.21.15.16:0/2890028382 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ffb08160250 0x7ffaf8003980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:59.755 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.747+0000 7ffb07fff640 1 -- 172.21.15.16:0/2890028382 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ffb08160250 msgr2=0x7ffaf8003980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:42:59.755 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.747+0000 7ffb07fff640 1 --2- 172.21.15.16:0/2890028382 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ffb08160250 0x7ffaf8003980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:59.755 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.747+0000 7ffb07fff640 1 -- 172.21.15.16:0/2890028382 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffaf8004d10 msgr2=0x7ffaf8005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:42:59.755 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.747+0000 7ffb07fff640 1 --2- 172.21.15.16:0/2890028382 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffaf8004d10 0x7ffaf8005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:59.755 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.747+0000 7ffb07fff640 1 -- 172.21.15.16:0/2890028382 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffaf8003530 con 0x7ffaf8003f00 2023-05-22T15:42:59.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.747+0000 7ffb07fff640 1 --2- 172.21.15.16:0/2890028382 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffaf8003f00 0x7ffaf80047d0 secure :-1 s=READY pgs=1291 cs=0 l=1 rev1=1 crypto rx=0x7ffaf400c290 tx=0x7ffaf401f430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3a649506597d62d server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:59.758 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.747+0000 7ffb077fe640 1 -- 172.21.15.16:0/2890028382 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffaf4005ae0 con 0x7ffaf8003f00 2023-05-22T15:42:59.758 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.747+0000 7ffb077fe640 1 -- 172.21.15.16:0/2890028382 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffaf40ab070 con 0x7ffaf8003f00 2023-05-22T15:42:59.758 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.747+0000 7ffb077fe640 1 -- 172.21.15.16:0/2890028382 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffaf4005e80 con 0x7ffaf8003f00 2023-05-22T15:42:59.758 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.751+0000 7ffb0eeb5640 1 -- 172.21.15.16:0/2890028382 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffaf8003f00 msgr2=0x7ffaf80047d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:59.758 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.751+0000 7ffb0eeb5640 1 --2- 172.21.15.16:0/2890028382 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffaf8003f00 0x7ffaf80047d0 secure :-1 s=READY pgs=1291 cs=0 l=1 rev1=1 crypto rx=0x7ffaf400c290 tx=0x7ffaf401f430 comp rx=0 tx=0).stop 2023-05-22T15:42:59.758 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.751+0000 7ffb0d452640 1 -- 172.21.15.16:0/2890028382 reap_dead start 2023-05-22T15:42:59.758 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.751+0000 7ffb0eeb5640 1 -- 172.21.15.16:0/2890028382 shutdown_connections 2023-05-22T15:42:59.759 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.751+0000 7ffb0eeb5640 1 -- 172.21.15.16:0/2890028382 >> 172.21.15.16:0/2890028382 conn(0x7ffb080655c0 msgr2=0x7ffb08070e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:59.759 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.751+0000 7ffb0eeb5640 1 -- 172.21.15.16:0/2890028382 shutdown_connections 2023-05-22T15:42:59.759 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.751+0000 7ffb0eeb5640 1 -- 172.21.15.16:0/2890028382 wait complete. 2023-05-22T15:42:59.759 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.751+0000 7ffb0eeb5640 1 Processor -- start 2023-05-22T15:42:59.759 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.751+0000 7ffb0eeb5640 1 -- start start 2023-05-22T15:42:59.759 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.751+0000 7ffb0eeb5640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ffaf8003f00 0x7ffb0814c890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:59.759 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.751+0000 7ffb0eeb5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffb080c8f40 0x7ffb080c6dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:59.760 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.751+0000 7ffb0eeb5640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffb0815b000 0x7ffb080c7310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:59.760 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.751+0000 7ffb0eeb5640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb0815d2d0 con 0x7ffb080c8f40 2023-05-22T15:42:59.760 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.751+0000 7ffb0eeb5640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb0815c650 con 0x7ffb0815b000 2023-05-22T15:42:59.760 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.751+0000 7ffb0eeb5640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb0815c930 con 0x7ffaf8003f00 2023-05-22T15:42:59.760 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.751+0000 7ffb07fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffb080c8f40 0x7ffb080c6dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:59.760 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.751+0000 7ffb07fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffb080c8f40 0x7ffb080c6dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49796/0 (socket says 172.21.15.16:49796) 2023-05-22T15:42:59.760 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.751+0000 7ffb07fff640 1 -- 172.21.15.16:0/3530925717 learned_addr learned my addr 172.21.15.16:0/3530925717 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:42:59.760 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.751+0000 7ffb0d452640 1 --2- 172.21.15.16:0/3530925717 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffb0815b000 0x7ffb080c7310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:59.761 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.751+0000 7ffb0cc51640 1 --2- 172.21.15.16:0/3530925717 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ffaf8003f00 0x7ffb0814c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:59.761 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.751+0000 7ffb07fff640 1 -- 172.21.15.16:0/3530925717 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ffaf8003f00 msgr2=0x7ffb0814c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:59.761 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.751+0000 7ffb07fff640 1 --2- 172.21.15.16:0/3530925717 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ffaf8003f00 0x7ffb0814c890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:59.761 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.751+0000 7ffb07fff640 1 -- 172.21.15.16:0/3530925717 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffb0815b000 msgr2=0x7ffb080c7310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:59.761 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.751+0000 7ffb07fff640 1 --2- 172.21.15.16:0/3530925717 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffb0815b000 0x7ffb080c7310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:59.761 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.751+0000 7ffb07fff640 1 -- 172.21.15.16:0/3530925717 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb080cf730 con 0x7ffb080c8f40 2023-05-22T15:42:59.761 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.751+0000 7ffb07fff640 1 --2- 172.21.15.16:0/3530925717 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffb080c8f40 0x7ffb080c6dd0 secure :-1 s=READY pgs=1292 cs=0 l=1 rev1=1 crypto rx=0x7ffaf401f260 tx=0x7ffaf8005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:59.762 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.751+0000 7ffb057fa640 1 -- 172.21.15.16:0/3530925717 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffaf40b7070 con 0x7ffb080c8f40 2023-05-22T15:42:59.762 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.751+0000 7ffb057fa640 1 -- 172.21.15.16:0/3530925717 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffaf40048b0 con 0x7ffb080c8f40 2023-05-22T15:42:59.762 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.751+0000 7ffb0eeb5640 1 -- 172.21.15.16:0/3530925717 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb080d3080 con 0x7ffb080c8f40 2023-05-22T15:42:59.762 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.751+0000 7ffb057fa640 1 -- 172.21.15.16:0/3530925717 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffaf4004c60 con 0x7ffb080c8f40 2023-05-22T15:42:59.762 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.751+0000 7ffb0eeb5640 1 -- 172.21.15.16:0/3530925717 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb080d0410 con 0x7ffb080c8f40 2023-05-22T15:42:59.763 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.755+0000 7ffb0eeb5640 1 -- 172.21.15.16:0/3530925717 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb080d1e80 con 0x7ffb080c8f40 2023-05-22T15:42:59.763 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.755+0000 7ffb057fa640 1 -- 172.21.15.16:0/3530925717 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 34) v1 ==== 84332+0+0 (secure 0 0 0) 0x7ffaf40ad020 con 0x7ffb080c8f40 2023-05-22T15:42:59.763 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.755+0000 7ffb057fa640 1 --2- 172.21.15.16:0/3530925717 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ffaec0654a0 0x7ffaec067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:42:59.763 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.755+0000 7ffb057fa640 1 -- 172.21.15.16:0/3530925717 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 6382+0+0 (secure 0 0 0) 0x7ffaf411f610 con 0x7ffb080c8f40 2023-05-22T15:42:59.763 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.755+0000 7ffb0cc51640 1 --2- 172.21.15.16:0/3530925717 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ffaec0654a0 0x7ffaec067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:42:59.764 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.755+0000 7ffb0cc51640 1 --2- 172.21.15.16:0/3530925717 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ffaec0654a0 0x7ffaec067950 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7ffafc009a30 tx=0x7ffafc008040 comp rx=0 tx=0).ready entity=mgr.6436 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:42:59.764 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.759+0000 7ffb057fa640 1 -- 172.21.15.16:0/3530925717 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7ffaf40f0950 con 0x7ffb080c8f40 2023-05-22T15:42:59.915 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.911+0000 7ffb0eeb5640 1 -- 172.21.15.16:0/3530925717 --> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ffb0804dbe0 con 0x7ffaec0654a0 2023-05-22T15:42:59.927 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.923+0000 7ffb057fa640 1 -- 172.21.15.16:0/3530925717 <== mgr.6436 v2:172.21.15.16:6832/36921 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+310504 (secure 0 0 0) 0x7ffb0804dbe0 con 0x7ffaec0654a0 2023-05-22T15:42:59.930 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:42:59.931 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2023-05-22T15:42:59.932 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.927+0000 7ffb0eeb5640 1 -- 172.21.15.16:0/3530925717 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ffaec0654a0 msgr2=0x7ffaec067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:59.933 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.927+0000 7ffb0eeb5640 1 --2- 172.21.15.16:0/3530925717 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ffaec0654a0 0x7ffaec067950 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7ffafc009a30 tx=0x7ffafc008040 comp rx=0 tx=0).stop 2023-05-22T15:42:59.933 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.927+0000 7ffb0eeb5640 1 -- 172.21.15.16:0/3530925717 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffb080c8f40 msgr2=0x7ffb080c6dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:42:59.933 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.927+0000 7ffb0eeb5640 1 --2- 172.21.15.16:0/3530925717 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffb080c8f40 0x7ffb080c6dd0 secure :-1 s=READY pgs=1292 cs=0 l=1 rev1=1 crypto rx=0x7ffaf401f260 tx=0x7ffaf8005800 comp rx=0 tx=0).stop 2023-05-22T15:42:59.933 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.927+0000 7ffb0eeb5640 1 -- 172.21.15.16:0/3530925717 shutdown_connections 2023-05-22T15:42:59.933 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.927+0000 7ffb0eeb5640 1 --2- 172.21.15.16:0/3530925717 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffb0815b000 0x7ffb080c7310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:59.933 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.927+0000 7ffb0eeb5640 1 --2- 172.21.15.16:0/3530925717 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffb080c8f40 0x7ffb080c6dd0 unknown :-1 s=CLOSED pgs=1292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:59.933 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.927+0000 7ffb0eeb5640 1 --2- 172.21.15.16:0/3530925717 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ffaec0654a0 0x7ffaec067950 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:59.934 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.927+0000 7ffb0eeb5640 1 --2- 172.21.15.16:0/3530925717 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ffaf8003f00 0x7ffb0814c890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:42:59.934 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.927+0000 7ffb0eeb5640 1 -- 172.21.15.16:0/3530925717 >> 172.21.15.16:0/3530925717 conn(0x7ffb080655c0 msgr2=0x7ffb0806a6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:42:59.934 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.927+0000 7ffb0eeb5640 1 -- 172.21.15.16:0/3530925717 shutdown_connections 2023-05-22T15:42:59.934 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:42:59.927+0000 7ffb0eeb5640 1 -- 172.21.15.16:0/3530925717 wait complete. 2023-05-22T15:42:59.948 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2023-05-22T15:42:59.480035+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592698,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":692,"num_read_kb":1336,"num_write":323,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":221,"ondisk_log_size":221,"up":210,"acting":210,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":67,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":211632,"kb_used_data":11056,"kb_used_omap":0,"kb_used_meta":200512,"kb_avail":754763088,"statfs":{"total":773094113280,"available":772877402112,"internally_reserved":0,"allocated":11321344,"data_stored":6466766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":205324288},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":60,"apply_latency_ms":60,"commit_latency_ns":60000000,"apply_latency_ns":60000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2330,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.293283"},"pg_stats":[{"pgid":"13.3f","version":"0'0","reported_seq":9,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.648813+0000","last_change":"2023-05-22T15:42:56.645184+0000","last_active":"2023-05-22T15:42:56.648813+0000","last_peered":"2023-05-22T15:42:56.648813+0000","last_clean":"2023-05-22T15:42:56.648813+0000","last_became_active":"2023-05-22T15:42:56.644521+0000","last_became_peered":"2023-05-22T15:42:56.644521+0000","last_unstale":"2023-05-22T15:42:56.648813+0000","last_undegraded":"2023-05-22T15:42:56.648813+0000","last_fullsized":"2023-05-22T15:42:56.648813+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_clean_scrub_stamp":"2023-05-22T15:42:56.185172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:11:10.859188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.3e","version":"0'0","reported_seq":9,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.642132+0000","last_change":"2023-05-22T15:42:56.641150+0000","last_active":"2023-05-22T15:42:56.642132+0000","last_peered":"2023-05-22T15:42:56.642132+0000","last_clean":"2023-05-22T15:42:56.642132+0000","last_became_active":"2023-05-22T15:42:56.640893+0000","last_became_peered":"2023-05-22T15:42:56.640893+0000","last_unstale":"2023-05-22T15:42:56.642132+0000","last_undegraded":"2023-05-22T15:42:56.642132+0000","last_fullsized":"2023-05-22T15:42:56.642132+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_clean_scrub_stamp":"2023-05-22T15:42:56.185172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:53:48.911917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.3d","version":"0'0","reported_seq":9,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.646639+0000","last_change":"2023-05-22T15:42:56.642827+0000","last_active":"2023-05-22T15:42:56.646639+0000","last_peered":"2023-05-22T15:42:56.646639+0000","last_clean":"2023-05-22T15:42:56.646639+0000","last_became_active":"2023-05-22T15:42:56.642218+0000","last_became_peered":"2023-05-22T15:42:56.642218+0000","last_unstale":"2023-05-22T15:42:56.646639+0000","last_undegraded":"2023-05-22T15:42:56.646639+0000","last_fullsized":"2023-05-22T15:42:56.646639+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_clean_scrub_stamp":"2023-05-22T15:42:56.185172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:33:42.886208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.3c","version":"0'0","reported_seq":9,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.599780+0000","last_change":"2023-05-22T15:42:56.595191+0000","last_active":"2023-05-22T15:42:56.599780+0000","last_peered":"2023-05-22T15:42:56.599780+0000","last_clean":"2023-05-22T15:42:56.599780+0000","last_became_active":"2023-05-22T15:42:56.594139+0000","last_became_peered":"2023-05-22T15:42:56.594139+0000","last_unstale":"2023-05-22T15:42:56.599780+0000","last_undegraded":"2023-05-22T15:42:56.599780+0000","last_fullsized":"2023-05-22T15:42:56.599780+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_clean_scrub_stamp":"2023-05-22T15:42:56.185172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T18:07:13.587805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.3b","version":"0'0","reported_seq":9,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.620964+0000","last_change":"2023-05-22T15:42:56.607640+0000","last_active":"2023-05-22T15:42:56.620964+0000","last_peered":"2023-05-22T15:42:56.620964+0000","last_clean":"2023-05-22T15:42:56.620964+0000","last_became_active":"2023-05-22T15:42:56.599485+0000","last_became_peered":"2023-05-22T15:42:56.599485+0000","last_unstale":"2023-05-22T15:42:56.620964+0000","last_undegraded":"2023-05-22T15:42:56.620964+0000","last_fullsized":"2023-05-22T15:42:56.620964+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_clean_scrub_stamp":"2023-05-22T15:42:56.185172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:58:17.652601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.3a","version":"0'0","reported_seq":9,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.630118+0000","last_change":"2023-05-22T15:42:56.593856+0000","last_active":"2023-05-22T15:42:56.630118+0000","last_peered":"2023-05-22T15:42:56.630118+0000","last_clean":"2023-05-22T15:42:56.630118+0000","last_became_active":"2023-05-22T15:42:56.592963+0000","last_became_peered":"2023-05-22T15:42:56.592963+0000","last_unstale":"2023-05-22T15:42:56.630118+0000","last_undegraded":"2023-05-22T15:42:56.630118+0000","last_fullsized":"2023-05-22T15:42:56.630118+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_clean_scrub_stamp":"2023-05-22T15:42:56.185172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:35:58.004110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.39","version":"0'0","reported_seq":9,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.624344+0000","last_change":"2023-05-22T15:42:56.623486+0000","last_active":"2023-05-22T15:42:56.624344+0000","last_peered":"2023-05-22T15:42:56.624344+0000","last_clean":"2023-05-22T15:42:56.624344+0000","last_became_active":"2023-05-22T15:42:56.623143+0000","last_became_peered":"2023-05-22T15:42:56.623143+0000","last_unstale":"2023-05-22T15:42:56.624344+0000","last_undegraded":"2023-05-22T15:42:56.624344+0000","last_fullsized":"2023-05-22T15:42:56.624344+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_clean_scrub_stamp":"2023-05-22T15:42:56.185172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:10:36.752607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.38","version":"0'0","reported_seq":9,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.630029+0000","last_change":"2023-05-22T15:42:56.602489+0000","last_active":"2023-05-22T15:42:56.630029+0000","last_peered":"2023-05-22T15:42:56.630029+0000","last_clean":"2023-05-22T15:42:56.630029+0000","last_became_active":"2023-05-22T15:42:56.601238+0000","last_became_peered":"2023-05-22T15:42:56.601238+0000","last_unstale":"2023-05-22T15:42:56.630029+0000","last_undegraded":"2023-05-22T15:42:56.630029+0000","last_fullsized":"2023-05-22T15:42:56.630029+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_clean_scrub_stamp":"2023-05-22T15:42:56.185172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:37:31.421851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.37","version":"0'0","reported_seq":9,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.630289+0000","last_change":"2023-05-22T15:42:56.608853+0000","last_active":"2023-05-22T15:42:56.630289+0000","last_peered":"2023-05-22T15:42:56.630289+0000","last_clean":"2023-05-22T15:42:56.630289+0000","last_became_active":"2023-05-22T15:42:56.608034+0000","last_became_peered":"2023-05-22T15:42:56.608034+0000","last_unstale":"2023-05-22T15:42:56.630289+0000","last_undegraded":"2023-05-22T15:42:56.630289+0000","last_fullsized":"2023-05-22T15:42:56.630289+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_clean_scrub_stamp":"2023-05-22T15:42:56.185172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:33:32.621247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.36","version":"0'0","reported_seq":9,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.640612+0000","last_change":"2023-05-22T15:42:56.622967+0000","last_active":"2023-05-22T15:42:56.640612+0000","last_peered":"2023-05-22T15:42:56.640612+0000","last_clean":"2023-05-22T15:42:56.640612+0000","last_became_active":"2023-05-22T15:42:56.619649+0000","last_became_peered":"2023-05-22T15:42:56.619649+0000","last_unstale":"2023-05-22T15:42:56.640612+0000","last_undegraded":"2023-05-22T15:42:56.640612+0000","last_fullsized":"2023-05-22T15:42:56.640612+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_clean_scrub_stamp":"2023-05-22T15:42:56.185172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:37:31.421851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.35","version":"0'0","reported_seq":9,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.622397+0000","last_change":"2023-05-22T15:42:56.621540+0000","last_active":"2023-05-22T15:42:56.622397+0000","last_peered":"2023-05-22T15:42:56.622397+0000","last_clean":"2023-05-22T15:42:56.622397+0000","last_became_active":"2023-05-22T15:42:56.621208+0000","last_became_peered":"2023-05-22T15:42:56.621208+0000","last_unstale":"2023-05-22T15:42:56.622397+0000","last_undegraded":"2023-05-22T15:42:56.622397+0000","last_fullsized":"2023-05-22T15:42:56.622397+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_clean_scrub_stamp":"2023-05-22T15:42:56.185172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:45:35.903593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.34","version":"0'0","reported_seq":9,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.646248+0000","last_change":"2023-05-22T15:42:56.639118+0000","last_active":"2023-05-22T15:42:56.646248+0000","last_peered":"2023-05-22T15:42:56.646248+0000","last_clean":"2023-05-22T15:42:56.646248+0000","last_became_active":"2023-05-22T15:42:56.636421+0000","last_became_peered":"2023-05-22T15:42:56.636421+0000","last_unstale":"2023-05-22T15:42:56.646248+0000","last_undegraded":"2023-05-22T15:42:56.646248+0000","last_fullsized":"2023-05-22T15:42:56.646248+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_clean_scrub_stamp":"2023-05-22T15:42:56.185172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:15:13.237338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.33","version":"0'0","reported_seq":9,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.637493+0000","last_change":"2023-05-22T15:42:56.625982+0000","last_active":"2023-05-22T15:42:56.637493+0000","last_peered":"2023-05-22T15:42:56.637493+0000","last_clean":"2023-05-22T15:42:56.637493+0000","last_became_active":"2023-05-22T15:42:56.623302+0000","last_became_peered":"2023-05-22T15:42:56.623302+0000","last_unstale":"2023-05-22T15:42:56.637493+0000","last_undegraded":"2023-05-22T15:42:56.637493+0000","last_fullsized":"2023-05-22T15:42:56.637493+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_clean_scrub_stamp":"2023-05-22T15:42:56.185172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:23:15.856405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.32","version":"0'0","reported_seq":9,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.601726+0000","last_change":"2023-05-22T15:42:56.598440+0000","last_active":"2023-05-22T15:42:56.601726+0000","last_peered":"2023-05-22T15:42:56.601726+0000","last_clean":"2023-05-22T15:42:56.601726+0000","last_became_active":"2023-05-22T15:42:56.598136+0000","last_became_peered":"2023-05-22T15:42:56.598136+0000","last_unstale":"2023-05-22T15:42:56.601726+0000","last_undegraded":"2023-05-22T15:42:56.601726+0000","last_fullsized":"2023-05-22T15:42:56.601726+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_clean_scrub_stamp":"2023-05-22T15:42:56.185172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:00:34.414109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.31","version":"0'0","reported_seq":9,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.625366+0000","last_change":"2023-05-22T15:42:56.624583+0000","last_active":"2023-05-22T15:42:56.625366+0000","last_peered":"2023-05-22T15:42:56.625366+0000","last_clean":"2023-05-22T15:42:56.625366+0000","last_became_active":"2023-05-22T15:42:56.624353+0000","last_became_peered":"2023-05-22T15:42:56.624353+0000","last_unstale":"2023-05-22T15:42:56.625366+0000","last_undegraded":"2023-05-22T15:42:56.625366+0000","last_fullsized":"2023-05-22T15:42:56.625366+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_clean_scrub_stamp":"2023-05-22T15:42:56.185172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:23:14.021552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.30","version":"0'0","reported_seq":9,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.632651+0000","last_change":"2023-05-22T15:42:56.627501+0000","last_active":"2023-05-22T15:42:56.632651+0000","last_peered":"2023-05-22T15:42:56.632651+0000","last_clean":"2023-05-22T15:42:56.632651+0000","last_became_active":"2023-05-22T15:42:56.626875+0000","last_became_peered":"2023-05-22T15:42:56.626875+0000","last_unstale":"2023-05-22T15:42:56.632651+0000","last_undegraded":"2023-05-22T15:42:56.632651+0000","last_fullsized":"2023-05-22T15:42:56.632651+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_clean_scrub_stamp":"2023-05-22T15:42:56.185172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:33:56.028414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2f","version":"0'0","reported_seq":9,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.611771+0000","last_change":"2023-05-22T15:42:56.595382+0000","last_active":"2023-05-22T15:42:56.611771+0000","last_peered":"2023-05-22T15:42:56.611771+0000","last_clean":"2023-05-22T15:42:56.611771+0000","last_became_active":"2023-05-22T15:42:56.594590+0000","last_became_peered":"2023-05-22T15:42:56.594590+0000","last_unstale":"2023-05-22T15:42:56.611771+0000","last_undegraded":"2023-05-22T15:42:56.611771+0000","last_fullsized":"2023-05-22T15:42:56.611771+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_clean_scrub_stamp":"2023-05-22T15:42:56.185172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:26:51.212963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2e","version":"0'0","reported_seq":9,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.624723+0000","last_change":"2023-05-22T15:42:56.623690+0000","last_active":"2023-05-22T15:42:56.624723+0000","last_peered":"2023-05-22T15:42:56.624723+0000","last_clean":"2023-05-22T15:42:56.624723+0000","last_became_active":"2023-05-22T15:42:56.623320+0000","last_became_peered":"2023-05-22T15:42:56.623320+0000","last_unstale":"2023-05-22T15:42:56.624723+0000","last_undegraded":"2023-05-22T15:42:56.624723+0000","last_fullsized":"2023-05-22T15:42:56.624723+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_clean_scrub_stamp":"2023-05-22T15:42:56.185172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T16:22:07.524139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2d","version":"0'0","reported_seq":9,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.606120+0000","last_change":"2023-05-22T15:42:56.599343+0000","last_active":"2023-05-22T15:42:56.606120+0000","last_peered":"2023-05-22T15:42:56.606120+0000","last_clean":"2023-05-22T15:42:56.606120+0000","last_became_active":"2023-05-22T15:42:56.598878+0000","last_became_peered":"2023-05-22T15:42:56.598878+0000","last_unstale":"2023-05-22T15:42:56.606120+0000","last_undegraded":"2023-05-22T15:42:56.606120+0000","last_fullsized":"2023-05-22T15:42:56.606120+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_clean_scrub_stamp":"2023-05-22T15:42:56.185172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:30:35.129326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.2c","version":"0'0","reported_seq":9,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.590108+0000","last_change":"2023-05-22T15:42:56.585137+0000","last_active":"2023-05-22T15:42:56.590108+0000","last_peered":"2023-05-22T15:42:56.590108+0000","last_clean":"2023-05-22T15:42:56.590108+0000","last_became_active":"2023-05-22T15:42:56.584436+0000","last_became_peered":"2023-05-22T15:42:56.584436+0000","last_unstale":"2023-05-22T15:42:56.590108+0000","last_undegraded":"2023-05-22T15:42:56.590108+0000","last_fullsized":"2023-05-22T15:42:56.590108+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_clean_scrub_stamp":"2023-05-22T15:42:56.185172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:42:20.650549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2b","version":"0'0","reported_seq":9,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.582952+0000","last_change":"2023-05-22T15:42:56.581991+0000","last_active":"2023-05-22T15:42:56.582952+0000","last_peered":"2023-05-22T15:42:56.582952+0000","last_clean":"2023-05-22T15:42:56.582952+0000","last_became_active":"2023-05-22T15:42:56.581713+0000","last_became_peered":"2023-05-22T15:42:56.581713+0000","last_unstale":"2023-05-22T15:42:56.582952+0000","last_undegraded":"2023-05-22T15:42:56.582952+0000","last_fullsized":"2023-05-22T15:42:56.582952+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_clean_scrub_stamp":"2023-05-22T15:42:56.185172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:09:09.562683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2a","version":"0'0","reported_seq":9,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.605766+0000","last_change":"2023-05-22T15:42:56.603209+0000","last_active":"2023-05-22T15:42:56.605766+0000","last_peered":"2023-05-22T15:42:56.605766+0000","last_clean":"2023-05-22T15:42:56.605766+0000","last_became_active":"2023-05-22T15:42:56.602949+0000","last_became_peered":"2023-05-22T15:42:56.602949+0000","last_unstale":"2023-05-22T15:42:56.605766+0000","last_undegraded":"2023-05-22T15:42:56.605766+0000","last_fullsized":"2023-05-22T15:42:56.605766+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_clean_scrub_stamp":"2023-05-22T15:42:56.185172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:29:59.008510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.29","version":"0'0","reported_seq":9,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.640922+0000","last_change":"2023-05-22T15:42:56.625299+0000","last_active":"2023-05-22T15:42:56.640922+0000","last_peered":"2023-05-22T15:42:56.640922+0000","last_clean":"2023-05-22T15:42:56.640922+0000","last_became_active":"2023-05-22T15:42:56.624521+0000","last_became_peered":"2023-05-22T15:42:56.624521+0000","last_unstale":"2023-05-22T15:42:56.640922+0000","last_undegraded":"2023-05-22T15:42:56.640922+0000","last_fullsized":"2023-05-22T15:42:56.640922+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_clean_scrub_stamp":"2023-05-22T15:42:56.185172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:53:52.392012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.28","version":"0'0","reported_seq":9,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.590010+0000","last_change":"2023-05-22T15:42:56.584601+0000","last_active":"2023-05-22T15:42:56.590010+0000","last_peered":"2023-05-22T15:42:56.590010+0000","last_clean":"2023-05-22T15:42:56.590010+0000","last_became_active":"2023-05-22T15:42:56.584317+0000","last_became_peered":"2023-05-22T15:42:56.584317+0000","last_unstale":"2023-05-22T15:42:56.590010+0000","last_undegraded":"2023-05-22T15:42:56.590010+0000","last_fullsized":"2023-05-22T15:42:56.590010+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_clean_scrub_stamp":"2023-05-22T15:42:56.185172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:59:24.891214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.27","version":"0'0","reported_seq":9,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.635353+0000","last_change":"2023-05-22T15:42:56.634247+0000","last_active":"2023-05-22T15:42:56.635353+0000","last_peered":"2023-05-22T15:42:56.635353+0000","last_clean":"2023-05-22T15:42:56.635353+0000","last_became_active":"2023-05-22T15:42:56.633957+0000","last_became_peered":"2023-05-22T15:42:56.633957+0000","last_unstale":"2023-05-22T15:42:56.635353+0000","last_undegraded":"2023-05-22T15:42:56.635353+0000","last_fullsized":"2023-05-22T15:42:56.635353+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_clean_scrub_stamp":"2023-05-22T15:42:56.185172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T03:30:09.923365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.26","version":"0'0","reported_seq":9,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.637650+0000","last_change":"2023-05-22T15:42:56.629495+0000","last_active":"2023-05-22T15:42:56.637650+0000","last_peered":"2023-05-22T15:42:56.637650+0000","last_clean":"2023-05-22T15:42:56.637650+0000","last_became_active":"2023-05-22T15:42:56.628731+0000","last_became_peered":"2023-05-22T15:42:56.628731+0000","last_unstale":"2023-05-22T15:42:56.637650+0000","last_undegraded":"2023-05-22T15:42:56.637650+0000","last_fullsized":"2023-05-22T15:42:56.637650+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_clean_scrub_stamp":"2023-05-22T15:42:56.185172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:57:40.502973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.25","version":"0'0","reported_seq":9,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.612043+0000","last_change":"2023-05-22T15:42:56.601023+0000","last_active":"2023-05-22T15:42:56.612043+0000","last_peered":"2023-05-22T15:42:56.612043+0000","last_clean":"2023-05-22T15:42:56.612043+0000","last_became_active":"2023-05-22T15:42:56.597820+0000","last_became_peered":"2023-05-22T15:42:56.597820+0000","last_unstale":"2023-05-22T15:42:56.612043+0000","last_undegraded":"2023-05-22T15:42:56.612043+0000","last_fullsized":"2023-05-22T15:42:56.612043+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_clean_scrub_stamp":"2023-05-22T15:42:56.185172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:57:37.896404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.24","version":"0'0","reported_seq":9,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.648267+0000","last_change":"2023-05-22T15:42:56.641209+0000","last_active":"2023-05-22T15:42:56.648267+0000","last_peered":"2023-05-22T15:42:56.648267+0000","last_clean":"2023-05-22T15:42:56.648267+0000","last_became_active":"2023-05-22T15:42:56.640689+0000","last_became_peered":"2023-05-22T15:42:56.640689+0000","last_unstale":"2023-05-22T15:42:56.648267+0000","last_undegraded":"2023-05-22T15:42:56.648267+0000","last_fullsized":"2023-05-22T15:42:56.648267+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_clean_scrub_stamp":"2023-05-22T15:42:56.185172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:05:52.591919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.23","version":"0'0","reported_seq":9,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.602625+0000","last_change":"2023-05-22T15:42:56.595383+0000","last_active":"2023-05-22T15:42:56.602625+0000","last_peered":"2023-05-22T15:42:56.602625+0000","last_clean":"2023-05-22T15:42:56.602625+0000","last_became_active":"2023-05-22T15:42:56.595117+0000","last_became_peered":"2023-05-22T15:42:56.595117+0000","last_unstale":"2023-05-22T15:42:56.602625+0000","last_undegraded":"2023-05-22T15:42:56.602625+0000","last_fullsized":"2023-05-22T15:42:56.602625+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_clean_scrub_stamp":"2023-05-22T15:42:56.185172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:09:09.562683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.22","version":"0'0","reported_seq":9,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.605411+0000","last_change":"2023-05-22T15:42:56.599349+0000","last_active":"2023-05-22T15:42:56.605411+0000","last_peered":"2023-05-22T15:42:56.605411+0000","last_clean":"2023-05-22T15:42:56.605411+0000","last_became_active":"2023-05-22T15:42:56.598970+0000","last_became_peered":"2023-05-22T15:42:56.598970+0000","last_unstale":"2023-05-22T15:42:56.605411+0000","last_undegraded":"2023-05-22T15:42:56.605411+0000","last_fullsized":"2023-05-22T15:42:56.605411+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_clean_scrub_stamp":"2023-05-22T15:42:56.185172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:05:52.591919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.21","version":"0'0","reported_seq":9,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.611848+0000","last_change":"2023-05-22T15:42:56.598275+0000","last_active":"2023-05-22T15:42:56.611848+0000","last_peered":"2023-05-22T15:42:56.611848+0000","last_clean":"2023-05-22T15:42:56.611848+0000","last_became_active":"2023-05-22T15:42:56.597447+0000","last_became_peered":"2023-05-22T15:42:56.597447+0000","last_unstale":"2023-05-22T15:42:56.611848+0000","last_undegraded":"2023-05-22T15:42:56.611848+0000","last_fullsized":"2023-05-22T15:42:56.611848+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_clean_scrub_stamp":"2023-05-22T15:42:56.185172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:40:39.423664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.20","version":"0'0","reported_seq":9,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.640734+0000","last_change":"2023-05-22T15:42:56.620133+0000","last_active":"2023-05-22T15:42:56.640734+0000","last_peered":"2023-05-22T15:42:56.640734+0000","last_clean":"2023-05-22T15:42:56.640734+0000","last_became_active":"2023-05-22T15:42:56.619225+0000","last_became_peered":"2023-05-22T15:42:56.619225+0000","last_unstale":"2023-05-22T15:42:56.640734+0000","last_undegraded":"2023-05-22T15:42:56.640734+0000","last_fullsized":"2023-05-22T15:42:56.640734+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_clean_scrub_stamp":"2023-05-22T15:42:56.185172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:48:43.959640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1e","version":"0'0","reported_seq":9,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.646383+0000","last_change":"2023-05-22T15:42:56.642799+0000","last_active":"2023-05-22T15:42:56.646383+0000","last_peered":"2023-05-22T15:42:56.646383+0000","last_clean":"2023-05-22T15:42:56.646383+0000","last_became_active":"2023-05-22T15:42:56.642102+0000","last_became_peered":"2023-05-22T15:42:56.642102+0000","last_unstale":"2023-05-22T15:42:56.646383+0000","last_undegraded":"2023-05-22T15:42:56.646383+0000","last_fullsized":"2023-05-22T15:42:56.646383+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_clean_scrub_stamp":"2023-05-22T15:42:56.185172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:30:35.129326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1f","version":"103'4","reported_seq":14,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.648907+0000","last_change":"2023-05-22T15:42:56.247802+0000","last_active":"2023-05-22T15:42:56.648907+0000","last_peered":"2023-05-22T15:42:56.648907+0000","last_clean":"2023-05-22T15:42:56.648907+0000","last_became_active":"2023-05-22T15:42:56.247046+0000","last_became_peered":"2023-05-22T15:42:56.247046+0000","last_unstale":"2023-05-22T15:42:56.648907+0000","last_undegraded":"2023-05-22T15:42:56.648907+0000","last_fullsized":"2023-05-22T15:42:56.648907+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:55.181398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:55.181398+0000","last_clean_scrub_stamp":"2023-05-22T15:42:55.181398+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:59:23.887440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":566,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1f","version":"0'0","reported_seq":9,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.646456+0000","last_change":"2023-05-22T15:42:56.632638+0000","last_active":"2023-05-22T15:42:56.646456+0000","last_peered":"2023-05-22T15:42:56.646456+0000","last_clean":"2023-05-22T15:42:56.646456+0000","last_became_active":"2023-05-22T15:42:56.623920+0000","last_became_peered":"2023-05-22T15:42:56.623920+0000","last_unstale":"2023-05-22T15:42:56.646456+0000","last_undegraded":"2023-05-22T15:42:56.646456+0000","last_fullsized":"2023-05-22T15:42:56.646456+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_clean_scrub_stamp":"2023-05-22T15:42:56.185172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:52:06.057485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1e","version":"103'1","reported_seq":11,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.600338+0000","last_change":"2023-05-22T15:42:56.233159+0000","last_active":"2023-05-22T15:42:56.600338+0000","last_peered":"2023-05-22T15:42:56.600338+0000","last_clean":"2023-05-22T15:42:56.600338+0000","last_became_active":"2023-05-22T15:42:56.232184+0000","last_became_peered":"2023-05-22T15:42:56.232184+0000","last_unstale":"2023-05-22T15:42:56.600338+0000","last_undegraded":"2023-05-22T15:42:56.600338+0000","last_fullsized":"2023-05-22T15:42:56.600338+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:55.181398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:55.181398+0000","last_clean_scrub_stamp":"2023-05-22T15:42:55.181398+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:11:09.855414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1c","version":"0'0","reported_seq":9,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.646289+0000","last_change":"2023-05-22T15:42:56.639197+0000","last_active":"2023-05-22T15:42:56.646289+0000","last_peered":"2023-05-22T15:42:56.646289+0000","last_clean":"2023-05-22T15:42:56.646289+0000","last_became_active":"2023-05-22T15:42:56.636246+0000","last_became_peered":"2023-05-22T15:42:56.636246+0000","last_unstale":"2023-05-22T15:42:56.646289+0000","last_undegraded":"2023-05-22T15:42:56.646289+0000","last_fullsized":"2023-05-22T15:42:56.646289+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_clean_scrub_stamp":"2023-05-22T15:42:56.185172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T18:44:11.980099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1d","version":"103'2","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.585681+0000","last_change":"2023-05-22T15:42:56.227802+0000","last_active":"2023-05-22T15:42:56.585681+0000","last_peered":"2023-05-22T15:42:56.585681+0000","last_clean":"2023-05-22T15:42:56.585681+0000","last_became_active":"2023-05-22T15:42:56.227511+0000","last_became_peered":"2023-05-22T15:42:56.227511+0000","last_unstale":"2023-05-22T15:42:56.585681+0000","last_undegraded":"2023-05-22T15:42:56.585681+0000","last_fullsized":"2023-05-22T15:42:56.585681+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:55.181398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:55.181398+0000","last_clean_scrub_stamp":"2023-05-22T15:42:55.181398+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:01:27.080923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1d","version":"0'0","reported_seq":9,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.637791+0000","last_change":"2023-05-22T15:42:56.631432+0000","last_active":"2023-05-22T15:42:56.637791+0000","last_peered":"2023-05-22T15:42:56.637791+0000","last_clean":"2023-05-22T15:42:56.637791+0000","last_became_active":"2023-05-22T15:42:56.629634+0000","last_became_peered":"2023-05-22T15:42:56.629634+0000","last_unstale":"2023-05-22T15:42:56.637791+0000","last_undegraded":"2023-05-22T15:42:56.637791+0000","last_fullsized":"2023-05-22T15:42:56.637791+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_clean_scrub_stamp":"2023-05-22T15:42:56.185172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:40:39.423664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1c","version":"103'2","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.600196+0000","last_change":"2023-05-22T15:42:56.228991+0000","last_active":"2023-05-22T15:42:56.600196+0000","last_peered":"2023-05-22T15:42:56.600196+0000","last_clean":"2023-05-22T15:42:56.600196+0000","last_became_active":"2023-05-22T15:42:56.228645+0000","last_became_peered":"2023-05-22T15:42:56.228645+0000","last_unstale":"2023-05-22T15:42:56.600196+0000","last_undegraded":"2023-05-22T15:42:56.600196+0000","last_fullsized":"2023-05-22T15:42:56.600196+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:55.181398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:55.181398+0000","last_clean_scrub_stamp":"2023-05-22T15:42:55.181398+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T16:55:34.285195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1a","version":"0'0","reported_seq":9,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.605326+0000","last_change":"2023-05-22T15:42:56.598288+0000","last_active":"2023-05-22T15:42:56.605326+0000","last_peered":"2023-05-22T15:42:56.605326+0000","last_clean":"2023-05-22T15:42:56.605326+0000","last_became_active":"2023-05-22T15:42:56.598007+0000","last_became_peered":"2023-05-22T15:42:56.598007+0000","last_unstale":"2023-05-22T15:42:56.605326+0000","last_undegraded":"2023-05-22T15:42:56.605326+0000","last_fullsized":"2023-05-22T15:42:56.605326+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_clean_scrub_stamp":"2023-05-22T15:42:56.185172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:28:10.838599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.1b","version":"103'1","reported_seq":11,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.608736+0000","last_change":"2023-05-22T15:42:56.234058+0000","last_active":"2023-05-22T15:42:56.608736+0000","last_peered":"2023-05-22T15:42:56.608736+0000","last_clean":"2023-05-22T15:42:56.608736+0000","last_became_active":"2023-05-22T15:42:56.232800+0000","last_became_peered":"2023-05-22T15:42:56.232800+0000","last_unstale":"2023-05-22T15:42:56.608736+0000","last_undegraded":"2023-05-22T15:42:56.608736+0000","last_fullsized":"2023-05-22T15:42:56.608736+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:55.181398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:55.181398+0000","last_clean_scrub_stamp":"2023-05-22T15:42:55.181398+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:49:37.988943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1b","version":"0'0","reported_seq":9,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.620910+0000","last_change":"2023-05-22T15:42:56.607031+0000","last_active":"2023-05-22T15:42:56.620910+0000","last_peered":"2023-05-22T15:42:56.620910+0000","last_clean":"2023-05-22T15:42:56.620910+0000","last_became_active":"2023-05-22T15:42:56.599657+0000","last_became_peered":"2023-05-22T15:42:56.599657+0000","last_unstale":"2023-05-22T15:42:56.620910+0000","last_undegraded":"2023-05-22T15:42:56.620910+0000","last_fullsized":"2023-05-22T15:42:56.620910+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_clean_scrub_stamp":"2023-05-22T15:42:56.185172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T18:00:17.614805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1a","version":"103'1","reported_seq":11,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.611692+0000","last_change":"2023-05-22T15:42:56.234276+0000","last_active":"2023-05-22T15:42:56.611692+0000","last_peered":"2023-05-22T15:42:56.611692+0000","last_clean":"2023-05-22T15:42:56.611692+0000","last_became_active":"2023-05-22T15:42:56.233491+0000","last_became_peered":"2023-05-22T15:42:56.233491+0000","last_unstale":"2023-05-22T15:42:56.611692+0000","last_undegraded":"2023-05-22T15:42:56.611692+0000","last_fullsized":"2023-05-22T15:42:56.611692+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:55.181398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:55.181398+0000","last_clean_scrub_stamp":"2023-05-22T15:42:55.181398+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T18:44:10.976325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":9,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.637370+0000","last_change":"2023-05-22T15:42:56.624785+0000","last_active":"2023-05-22T15:42:56.637370+0000","last_peered":"2023-05-22T15:42:56.637370+0000","last_clean":"2023-05-22T15:42:56.637370+0000","last_became_active":"2023-05-22T15:42:56.623512+0000","last_became_peered":"2023-05-22T15:42:56.623512+0000","last_unstale":"2023-05-22T15:42:56.637370+0000","last_undegraded":"2023-05-22T15:42:56.637370+0000","last_fullsized":"2023-05-22T15:42:56.637370+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_clean_scrub_stamp":"2023-05-22T15:42:56.185172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T16:55:35.288969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":10,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.572555+0000","last_change":"2023-05-22T15:42:56.236271+0000","last_active":"2023-05-22T15:42:56.572555+0000","last_peered":"2023-05-22T15:42:56.572555+0000","last_clean":"2023-05-22T15:42:56.572555+0000","last_became_active":"2023-05-22T15:42:56.235956+0000","last_became_peered":"2023-05-22T15:42:56.235956+0000","last_unstale":"2023-05-22T15:42:56.572555+0000","last_undegraded":"2023-05-22T15:42:56.572555+0000","last_fullsized":"2023-05-22T15:42:56.572555+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:55.181398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:55.181398+0000","last_clean_scrub_stamp":"2023-05-22T15:42:55.181398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:40:38.419890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":9,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.620627+0000","last_change":"2023-05-22T15:42:56.620040+0000","last_active":"2023-05-22T15:42:56.620627+0000","last_peered":"2023-05-22T15:42:56.620627+0000","last_clean":"2023-05-22T15:42:56.620627+0000","last_became_active":"2023-05-22T15:42:56.619755+0000","last_became_peered":"2023-05-22T15:42:56.619755+0000","last_unstale":"2023-05-22T15:42:56.620627+0000","last_undegraded":"2023-05-22T15:42:56.620627+0000","last_fullsized":"2023-05-22T15:42:56.620627+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_clean_scrub_stamp":"2023-05-22T15:42:56.185172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:30:35.129326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":10,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.592956+0000","last_change":"2023-05-22T15:42:56.241097+0000","last_active":"2023-05-22T15:42:56.592956+0000","last_peered":"2023-05-22T15:42:56.592956+0000","last_clean":"2023-05-22T15:42:56.592956+0000","last_became_active":"2023-05-22T15:42:56.240759+0000","last_became_peered":"2023-05-22T15:42:56.240759+0000","last_unstale":"2023-05-22T15:42:56.592956+0000","last_undegraded":"2023-05-22T15:42:56.592956+0000","last_fullsized":"2023-05-22T15:42:56.592956+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:55.181398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:55.181398+0000","last_clean_scrub_stamp":"2023-05-22T15:42:55.181398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:36:14.623130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":9,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.597812+0000","last_change":"2023-05-22T15:42:56.596326+0000","last_active":"2023-05-22T15:42:56.597812+0000","last_peered":"2023-05-22T15:42:56.597812+0000","last_clean":"2023-05-22T15:42:56.597812+0000","last_became_active":"2023-05-22T15:42:56.596035+0000","last_became_peered":"2023-05-22T15:42:56.596035+0000","last_unstale":"2023-05-22T15:42:56.597812+0000","last_undegraded":"2023-05-22T15:42:56.597812+0000","last_fullsized":"2023-05-22T15:42:56.597812+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_clean_scrub_stamp":"2023-05-22T15:42:56.185172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:26:51.212963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":10,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.624956+0000","last_change":"2023-05-22T15:42:56.247608+0000","last_active":"2023-05-22T15:42:56.624956+0000","last_peered":"2023-05-22T15:42:56.624956+0000","last_clean":"2023-05-22T15:42:56.624956+0000","last_became_active":"2023-05-22T15:42:56.246408+0000","last_became_peered":"2023-05-22T15:42:56.246408+0000","last_unstale":"2023-05-22T15:42:56.624956+0000","last_undegraded":"2023-05-22T15:42:56.624956+0000","last_fullsized":"2023-05-22T15:42:56.624956+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:55.181398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:55.181398+0000","last_clean_scrub_stamp":"2023-05-22T15:42:55.181398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:40:49.967414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":9,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.629400+0000","last_change":"2023-05-22T15:42:56.594234+0000","last_active":"2023-05-22T15:42:56.629400+0000","last_peered":"2023-05-22T15:42:56.629400+0000","last_clean":"2023-05-22T15:42:56.629400+0000","last_became_active":"2023-05-22T15:42:56.593463+0000","last_became_peered":"2023-05-22T15:42:56.593463+0000","last_unstale":"2023-05-22T15:42:56.629400+0000","last_undegraded":"2023-05-22T15:42:56.629400+0000","last_fullsized":"2023-05-22T15:42:56.629400+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_clean_scrub_stamp":"2023-05-22T15:42:56.185172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:57:40.502973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":10,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.622818+0000","last_change":"2023-05-22T15:42:56.245119+0000","last_active":"2023-05-22T15:42:56.622818+0000","last_peered":"2023-05-22T15:42:56.622818+0000","last_clean":"2023-05-22T15:42:56.622818+0000","last_became_active":"2023-05-22T15:42:56.244477+0000","last_became_peered":"2023-05-22T15:42:56.244477+0000","last_unstale":"2023-05-22T15:42:56.622818+0000","last_undegraded":"2023-05-22T15:42:56.622818+0000","last_fullsized":"2023-05-22T15:42:56.622818+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:55.181398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:55.181398+0000","last_clean_scrub_stamp":"2023-05-22T15:42:55.181398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:08:25.261985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.14","version":"0'0","reported_seq":9,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.638816+0000","last_change":"2023-05-22T15:42:56.638182+0000","last_active":"2023-05-22T15:42:56.638816+0000","last_peered":"2023-05-22T15:42:56.638816+0000","last_clean":"2023-05-22T15:42:56.638816+0000","last_became_active":"2023-05-22T15:42:56.637838+0000","last_became_peered":"2023-05-22T15:42:56.637838+0000","last_unstale":"2023-05-22T15:42:56.638816+0000","last_undegraded":"2023-05-22T15:42:56.638816+0000","last_fullsized":"2023-05-22T15:42:56.638816+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_clean_scrub_stamp":"2023-05-22T15:42:56.185172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:23:14.021552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.15","version":"103'3","reported_seq":13,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.640652+0000","last_change":"2023-05-22T15:42:56.246663+0000","last_active":"2023-05-22T15:42:56.640652+0000","last_peered":"2023-05-22T15:42:56.640652+0000","last_clean":"2023-05-22T15:42:56.640652+0000","last_became_active":"2023-05-22T15:42:56.245925+0000","last_became_peered":"2023-05-22T15:42:56.245925+0000","last_unstale":"2023-05-22T15:42:56.640652+0000","last_undegraded":"2023-05-22T15:42:56.640652+0000","last_fullsized":"2023-05-22T15:42:56.640652+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:55.181398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:55.181398+0000","last_clean_scrub_stamp":"2023-05-22T15:42:55.181398+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:36:14.623130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.15","version":"0'0","reported_seq":9,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.616612+0000","last_change":"2023-05-22T15:42:56.604169+0000","last_active":"2023-05-22T15:42:56.616612+0000","last_peered":"2023-05-22T15:42:56.616612+0000","last_clean":"2023-05-22T15:42:56.616612+0000","last_became_active":"2023-05-22T15:42:56.603406+0000","last_became_peered":"2023-05-22T15:42:56.603406+0000","last_unstale":"2023-05-22T15:42:56.616612+0000","last_undegraded":"2023-05-22T15:42:56.616612+0000","last_fullsized":"2023-05-22T15:42:56.616612+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_clean_scrub_stamp":"2023-05-22T15:42:56.185172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:33:42.886208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.14","version":"103'6","reported_seq":16,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.644751+0000","last_change":"2023-05-22T15:42:56.231238+0000","last_active":"2023-05-22T15:42:56.644751+0000","last_peered":"2023-05-22T15:42:56.644751+0000","last_clean":"2023-05-22T15:42:56.644751+0000","last_became_active":"2023-05-22T15:42:56.230500+0000","last_became_peered":"2023-05-22T15:42:56.230500+0000","last_unstale":"2023-05-22T15:42:56.644751+0000","last_undegraded":"2023-05-22T15:42:56.644751+0000","last_fullsized":"2023-05-22T15:42:56.644751+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:55.181398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:55.181398+0000","last_clean_scrub_stamp":"2023-05-22T15:42:55.181398+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:28:09.834825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1548,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":9,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.606062+0000","last_change":"2023-05-22T15:42:56.598573+0000","last_active":"2023-05-22T15:42:56.606062+0000","last_peered":"2023-05-22T15:42:56.606062+0000","last_clean":"2023-05-22T15:42:56.606062+0000","last_became_active":"2023-05-22T15:42:56.598275+0000","last_became_peered":"2023-05-22T15:42:56.598275+0000","last_unstale":"2023-05-22T15:42:56.606062+0000","last_undegraded":"2023-05-22T15:42:56.606062+0000","last_fullsized":"2023-05-22T15:42:56.606062+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_clean_scrub_stamp":"2023-05-22T15:42:56.185172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T18:00:17.614805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.13","version":"103'2","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.606623+0000","last_change":"2023-05-22T15:42:56.215827+0000","last_active":"2023-05-22T15:42:56.606623+0000","last_peered":"2023-05-22T15:42:56.606623+0000","last_clean":"2023-05-22T15:42:56.606623+0000","last_became_active":"2023-05-22T15:42:56.215525+0000","last_became_peered":"2023-05-22T15:42:56.215525+0000","last_unstale":"2023-05-22T15:42:56.606623+0000","last_undegraded":"2023-05-22T15:42:56.606623+0000","last_fullsized":"2023-05-22T15:42:56.606623+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:55.181398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:55.181398+0000","last_clean_scrub_stamp":"2023-05-22T15:42:55.181398+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:04:20.717819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.13","version":"0'0","reported_seq":9,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.637298+0000","last_change":"2023-05-22T15:42:56.624651+0000","last_active":"2023-05-22T15:42:56.637298+0000","last_peered":"2023-05-22T15:42:56.637298+0000","last_clean":"2023-05-22T15:42:56.637298+0000","last_became_active":"2023-05-22T15:42:56.623992+0000","last_became_peered":"2023-05-22T15:42:56.623992+0000","last_unstale":"2023-05-22T15:42:56.637298+0000","last_undegraded":"2023-05-22T15:42:56.637298+0000","last_fullsized":"2023-05-22T15:42:56.637298+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_clean_scrub_stamp":"2023-05-22T15:42:56.185172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:33:56.028414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":10,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.586197+0000","last_change":"2023-05-22T15:42:56.218515+0000","last_active":"2023-05-22T15:42:56.586197+0000","last_peered":"2023-05-22T15:42:56.586197+0000","last_clean":"2023-05-22T15:42:56.586197+0000","last_became_active":"2023-05-22T15:42:56.218149+0000","last_became_peered":"2023-05-22T15:42:56.218149+0000","last_unstale":"2023-05-22T15:42:56.586197+0000","last_undegraded":"2023-05-22T15:42:56.586197+0000","last_fullsized":"2023-05-22T15:42:56.586197+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:55.181398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:55.181398+0000","last_clean_scrub_stamp":"2023-05-22T15:42:55.181398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:58:16.648827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.10","version":"0'0","reported_seq":9,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.605458+0000","last_change":"2023-05-22T15:42:56.601026+0000","last_active":"2023-05-22T15:42:56.605458+0000","last_peered":"2023-05-22T15:42:56.605458+0000","last_clean":"2023-05-22T15:42:56.605458+0000","last_became_active":"2023-05-22T15:42:56.600758+0000","last_became_peered":"2023-05-22T15:42:56.600758+0000","last_unstale":"2023-05-22T15:42:56.605458+0000","last_undegraded":"2023-05-22T15:42:56.605458+0000","last_fullsized":"2023-05-22T15:42:56.605458+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_clean_scrub_stamp":"2023-05-22T15:42:56.185172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:09:51.134436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":10,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.580765+0000","last_change":"2023-05-22T15:42:56.234364+0000","last_active":"2023-05-22T15:42:56.580765+0000","last_peered":"2023-05-22T15:42:56.580765+0000","last_clean":"2023-05-22T15:42:56.580765+0000","last_became_active":"2023-05-22T15:42:56.233647+0000","last_became_peered":"2023-05-22T15:42:56.233647+0000","last_unstale":"2023-05-22T15:42:56.580765+0000","last_undegraded":"2023-05-22T15:42:56.580765+0000","last_fullsized":"2023-05-22T15:42:56.580765+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:55.181398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:55.181398+0000","last_clean_scrub_stamp":"2023-05-22T15:42:55.181398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:09:50.130662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":9,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.635357+0000","last_change":"2023-05-22T15:42:56.634351+0000","last_active":"2023-05-22T15:42:56.635357+0000","last_peered":"2023-05-22T15:42:56.635357+0000","last_clean":"2023-05-22T15:42:56.635357+0000","last_became_active":"2023-05-22T15:42:56.633939+0000","last_became_peered":"2023-05-22T15:42:56.633939+0000","last_unstale":"2023-05-22T15:42:56.635357+0000","last_undegraded":"2023-05-22T15:42:56.635357+0000","last_fullsized":"2023-05-22T15:42:56.635357+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_clean_scrub_stamp":"2023-05-22T15:42:56.185172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:49:38.992717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.10","version":"103'8","reported_seq":18,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.603145+0000","last_change":"2023-05-22T15:42:56.222730+0000","last_active":"2023-05-22T15:42:56.603145+0000","last_peered":"2023-05-22T15:42:56.603145+0000","last_clean":"2023-05-22T15:42:56.603145+0000","last_became_active":"2023-05-22T15:42:56.222283+0000","last_became_peered":"2023-05-22T15:42:56.222283+0000","last_unstale":"2023-05-22T15:42:56.603145+0000","last_undegraded":"2023-05-22T15:42:56.603145+0000","last_fullsized":"2023-05-22T15:42:56.603145+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:55.181398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:55.181398+0000","last_clean_scrub_stamp":"2023-05-22T15:42:55.181398+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:35:57.000336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.6","version":"0'0","reported_seq":9,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.634802+0000","last_change":"2023-05-22T15:42:56.631350+0000","last_active":"2023-05-22T15:42:56.634802+0000","last_peered":"2023-05-22T15:42:56.634802+0000","last_clean":"2023-05-22T15:42:56.634802+0000","last_became_active":"2023-05-22T15:42:56.630605+0000","last_became_peered":"2023-05-22T15:42:56.630605+0000","last_unstale":"2023-05-22T15:42:56.634802+0000","last_undegraded":"2023-05-22T15:42:56.634802+0000","last_fullsized":"2023-05-22T15:42:56.634802+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_clean_scrub_stamp":"2023-05-22T15:42:56.185172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:45:20.475919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.7","version":"103'2","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.609334+0000","last_change":"2023-05-22T15:42:56.234291+0000","last_active":"2023-05-22T15:42:56.609334+0000","last_peered":"2023-05-22T15:42:56.609334+0000","last_clean":"2023-05-22T15:42:56.609334+0000","last_became_active":"2023-05-22T15:42:56.234013+0000","last_became_peered":"2023-05-22T15:42:56.234013+0000","last_unstale":"2023-05-22T15:42:56.609334+0000","last_undegraded":"2023-05-22T15:42:56.609334+0000","last_fullsized":"2023-05-22T15:42:56.609334+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:55.181398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:55.181398+0000","last_clean_scrub_stamp":"2023-05-22T15:42:55.181398+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:43:51.407055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.7","version":"0'0","reported_seq":9,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.649123+0000","last_change":"2023-05-22T15:42:56.647851+0000","last_active":"2023-05-22T15:42:56.649123+0000","last_peered":"2023-05-22T15:42:56.649123+0000","last_clean":"2023-05-22T15:42:56.649123+0000","last_became_active":"2023-05-22T15:42:56.647280+0000","last_became_peered":"2023-05-22T15:42:56.647280+0000","last_unstale":"2023-05-22T15:42:56.649123+0000","last_undegraded":"2023-05-22T15:42:56.649123+0000","last_fullsized":"2023-05-22T15:42:56.649123+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_clean_scrub_stamp":"2023-05-22T15:42:56.185172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:58:13.431047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":10,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.590994+0000","last_change":"2023-05-22T15:42:56.216354+0000","last_active":"2023-05-22T15:42:56.590994+0000","last_peered":"2023-05-22T15:42:56.590994+0000","last_clean":"2023-05-22T15:42:56.590994+0000","last_became_active":"2023-05-22T15:42:56.215931+0000","last_became_peered":"2023-05-22T15:42:56.215931+0000","last_unstale":"2023-05-22T15:42:56.590994+0000","last_undegraded":"2023-05-22T15:42:56.590994+0000","last_fullsized":"2023-05-22T15:42:56.590994+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:55.181398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:55.181398+0000","last_clean_scrub_stamp":"2023-05-22T15:42:55.181398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T18:09:24.459116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":9,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.646200+0000","last_change":"2023-05-22T15:42:56.638543+0000","last_active":"2023-05-22T15:42:56.646200+0000","last_peered":"2023-05-22T15:42:56.646200+0000","last_clean":"2023-05-22T15:42:56.646200+0000","last_became_active":"2023-05-22T15:42:56.635593+0000","last_became_peered":"2023-05-22T15:42:56.635593+0000","last_unstale":"2023-05-22T15:42:56.646200+0000","last_undegraded":"2023-05-22T15:42:56.646200+0000","last_fullsized":"2023-05-22T15:42:56.646200+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_clean_scrub_stamp":"2023-05-22T15:42:56.185172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T03:30:09.923365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":10,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.596605+0000","last_change":"2023-05-22T15:42:56.245367+0000","last_active":"2023-05-22T15:42:56.596605+0000","last_peered":"2023-05-22T15:42:56.596605+0000","last_clean":"2023-05-22T15:42:56.596605+0000","last_became_active":"2023-05-22T15:42:56.244621+0000","last_became_peered":"2023-05-22T15:42:56.244621+0000","last_unstale":"2023-05-22T15:42:56.596605+0000","last_undegraded":"2023-05-22T15:42:56.596605+0000","last_fullsized":"2023-05-22T15:42:56.596605+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:55.181398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:55.181398+0000","last_clean_scrub_stamp":"2023-05-22T15:42:55.181398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:25:22.447013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":9,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.611938+0000","last_change":"2023-05-22T15:42:56.597061+0000","last_active":"2023-05-22T15:42:56.611938+0000","last_peered":"2023-05-22T15:42:56.611938+0000","last_clean":"2023-05-22T15:42:56.611938+0000","last_became_active":"2023-05-22T15:42:56.596241+0000","last_became_peered":"2023-05-22T15:42:56.596241+0000","last_unstale":"2023-05-22T15:42:56.611938+0000","last_undegraded":"2023-05-22T15:42:56.611938+0000","last_fullsized":"2023-05-22T15:42:56.611938+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_clean_scrub_stamp":"2023-05-22T15:42:56.185172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:58:13.431047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":10,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.617492+0000","last_change":"2023-05-22T15:42:56.245487+0000","last_active":"2023-05-22T15:42:56.617492+0000","last_peered":"2023-05-22T15:42:56.617492+0000","last_clean":"2023-05-22T15:42:56.617492+0000","last_became_active":"2023-05-22T15:42:56.244894+0000","last_became_peered":"2023-05-22T15:42:56.244894+0000","last_unstale":"2023-05-22T15:42:56.617492+0000","last_undegraded":"2023-05-22T15:42:56.617492+0000","last_fullsized":"2023-05-22T15:42:56.617492+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:55.181398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:55.181398+0000","last_clean_scrub_stamp":"2023-05-22T15:42:55.181398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T18:09:24.459116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":9,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.630176+0000","last_change":"2023-05-22T15:42:56.603008+0000","last_active":"2023-05-22T15:42:56.630176+0000","last_peered":"2023-05-22T15:42:56.630176+0000","last_clean":"2023-05-22T15:42:56.630176+0000","last_became_active":"2023-05-22T15:42:56.602175+0000","last_became_peered":"2023-05-22T15:42:56.602175+0000","last_unstale":"2023-05-22T15:42:56.630176+0000","last_undegraded":"2023-05-22T15:42:56.630176+0000","last_fullsized":"2023-05-22T15:42:56.630176+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_clean_scrub_stamp":"2023-05-22T15:42:56.185172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T18:12:07.084825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":10,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.572651+0000","last_change":"2023-05-22T15:42:56.222791+0000","last_active":"2023-05-22T15:42:56.572651+0000","last_peered":"2023-05-22T15:42:56.572651+0000","last_clean":"2023-05-22T15:42:56.572651+0000","last_became_active":"2023-05-22T15:42:56.222498+0000","last_became_peered":"2023-05-22T15:42:56.222498+0000","last_unstale":"2023-05-22T15:42:56.572651+0000","last_undegraded":"2023-05-22T15:42:56.572651+0000","last_fullsized":"2023-05-22T15:42:56.572651+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:55.181398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:55.181398+0000","last_clean_scrub_stamp":"2023-05-22T15:42:55.181398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:33:55.024640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":9,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.606919+0000","last_change":"2023-05-22T15:42:56.602967+0000","last_active":"2023-05-22T15:42:56.606919+0000","last_peered":"2023-05-22T15:42:56.606919+0000","last_clean":"2023-05-22T15:42:56.606919+0000","last_became_active":"2023-05-22T15:42:56.602702+0000","last_became_peered":"2023-05-22T15:42:56.602702+0000","last_unstale":"2023-05-22T15:42:56.606919+0000","last_undegraded":"2023-05-22T15:42:56.606919+0000","last_fullsized":"2023-05-22T15:42:56.606919+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_clean_scrub_stamp":"2023-05-22T15:42:56.185172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:26:51.212963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":10,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.584269+0000","last_change":"2023-05-22T15:42:56.234244+0000","last_active":"2023-05-22T15:42:56.584269+0000","last_peered":"2023-05-22T15:42:56.584269+0000","last_clean":"2023-05-22T15:42:56.584269+0000","last_became_active":"2023-05-22T15:42:56.232845+0000","last_became_peered":"2023-05-22T15:42:56.232845+0000","last_unstale":"2023-05-22T15:42:56.584269+0000","last_undegraded":"2023-05-22T15:42:56.584269+0000","last_fullsized":"2023-05-22T15:42:56.584269+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:55.181398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:55.181398+0000","last_clean_scrub_stamp":"2023-05-22T15:42:55.181398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:52:05.053711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":9,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.637432+0000","last_change":"2023-05-22T15:42:56.625205+0000","last_active":"2023-05-22T15:42:56.637432+0000","last_peered":"2023-05-22T15:42:56.637432+0000","last_clean":"2023-05-22T15:42:56.637432+0000","last_became_active":"2023-05-22T15:42:56.624194+0000","last_became_peered":"2023-05-22T15:42:56.624194+0000","last_unstale":"2023-05-22T15:42:56.637432+0000","last_undegraded":"2023-05-22T15:42:56.637432+0000","last_fullsized":"2023-05-22T15:42:56.637432+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_clean_scrub_stamp":"2023-05-22T15:42:56.185172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:48:06.492503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":10,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.606300+0000","last_change":"2023-05-22T15:42:56.245513+0000","last_active":"2023-05-22T15:42:56.606300+0000","last_peered":"2023-05-22T15:42:56.606300+0000","last_clean":"2023-05-22T15:42:56.606300+0000","last_became_active":"2023-05-22T15:42:56.244769+0000","last_became_peered":"2023-05-22T15:42:56.244769+0000","last_unstale":"2023-05-22T15:42:56.606300+0000","last_undegraded":"2023-05-22T15:42:56.606300+0000","last_fullsized":"2023-05-22T15:42:56.606300+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:55.181398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:55.181398+0000","last_clean_scrub_stamp":"2023-05-22T15:42:55.181398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:04:20.717819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":9,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.640974+0000","last_change":"2023-05-22T15:42:56.626621+0000","last_active":"2023-05-22T15:42:56.640974+0000","last_peered":"2023-05-22T15:42:56.640974+0000","last_clean":"2023-05-22T15:42:56.640974+0000","last_became_active":"2023-05-22T15:42:56.625149+0000","last_became_peered":"2023-05-22T15:42:56.625149+0000","last_unstale":"2023-05-22T15:42:56.640974+0000","last_undegraded":"2023-05-22T15:42:56.640974+0000","last_fullsized":"2023-05-22T15:42:56.640974+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_clean_scrub_stamp":"2023-05-22T15:42:56.185172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:49:38.992717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":10,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.582503+0000","last_change":"2023-05-22T15:42:56.245693+0000","last_active":"2023-05-22T15:42:56.582503+0000","last_peered":"2023-05-22T15:42:56.582503+0000","last_clean":"2023-05-22T15:42:56.582503+0000","last_became_active":"2023-05-22T15:42:56.244970+0000","last_became_peered":"2023-05-22T15:42:56.244970+0000","last_unstale":"2023-05-22T15:42:56.582503+0000","last_undegraded":"2023-05-22T15:42:56.582503+0000","last_fullsized":"2023-05-22T15:42:56.582503+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:55.181398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:55.181398+0000","last_clean_scrub_stamp":"2023-05-22T15:42:55.181398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:53:51.388237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":9,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.631034+0000","last_change":"2023-05-22T15:42:56.629886+0000","last_active":"2023-05-22T15:42:56.631034+0000","last_peered":"2023-05-22T15:42:56.631034+0000","last_clean":"2023-05-22T15:42:56.631034+0000","last_became_active":"2023-05-22T15:42:56.629626+0000","last_became_peered":"2023-05-22T15:42:56.629626+0000","last_unstale":"2023-05-22T15:42:56.631034+0000","last_undegraded":"2023-05-22T15:42:56.631034+0000","last_fullsized":"2023-05-22T15:42:56.631034+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_clean_scrub_stamp":"2023-05-22T15:42:56.185172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T18:07:13.587805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"103'1","reported_seq":11,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.633953+0000","last_change":"2023-05-22T15:42:56.235686+0000","last_active":"2023-05-22T15:42:56.633953+0000","last_peered":"2023-05-22T15:42:56.633953+0000","last_clean":"2023-05-22T15:42:56.633953+0000","last_became_active":"2023-05-22T15:42:56.234953+0000","last_became_peered":"2023-05-22T15:42:56.234953+0000","last_unstale":"2023-05-22T15:42:56.633953+0000","last_undegraded":"2023-05-22T15:42:56.633953+0000","last_fullsized":"2023-05-22T15:42:56.633953+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:55.181398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:55.181398+0000","last_clean_scrub_stamp":"2023-05-22T15:42:55.181398+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:53:51.388237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"92'183","reported_seq":481,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.223463+0000","last_change":"2023-05-22T15:26:39.318204+0000","last_active":"2023-05-22T15:42:56.223463+0000","last_peered":"2023-05-22T15:42:56.223463+0000","last_clean":"2023-05-22T15:42:56.223463+0000","last_became_active":"2023-05-22T15:26:39.315370+0000","last_became_peered":"2023-05-22T15:26:39.315370+0000","last_unstale":"2023-05-22T15:42:56.223463+0000","last_undegraded":"2023-05-22T15:42:56.223463+0000","last_fullsized":"2023-05-22T15:42:56.223463+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:38.020076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:38.020076+0000","last_clean_scrub_stamp":"2023-05-22T15:26:38.020076+0000","objects_scrubbed":0,"log_size":183,"log_dups_size":0,"ondisk_log_size":183,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:06:39.207543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":692,"num_read_kb":1336,"num_write":279,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":97,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.224737+0000","last_change":"2023-05-22T15:26:44.971951+0000","last_active":"2023-05-22T15:42:56.224737+0000","last_peered":"2023-05-22T15:42:56.224737+0000","last_clean":"2023-05-22T15:42:56.224737+0000","last_became_active":"2023-05-22T15:26:44.970872+0000","last_became_peered":"2023-05-22T15:26:44.970872+0000","last_unstale":"2023-05-22T15:42:56.224737+0000","last_undegraded":"2023-05-22T15:42:56.224737+0000","last_fullsized":"2023-05-22T15:42:56.224737+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:55:34.752949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.f","version":"0'0","reported_seq":9,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.646497+0000","last_change":"2023-05-22T15:42:56.633999+0000","last_active":"2023-05-22T15:42:56.646497+0000","last_peered":"2023-05-22T15:42:56.646497+0000","last_clean":"2023-05-22T15:42:56.646497+0000","last_became_active":"2023-05-22T15:42:56.633336+0000","last_became_peered":"2023-05-22T15:42:56.633336+0000","last_unstale":"2023-05-22T15:42:56.646497+0000","last_undegraded":"2023-05-22T15:42:56.646497+0000","last_fullsized":"2023-05-22T15:42:56.646497+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_clean_scrub_stamp":"2023-05-22T15:42:56.185172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T18:12:07.084825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.e","version":"103'2","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.602596+0000","last_change":"2023-05-22T15:42:56.240197+0000","last_active":"2023-05-22T15:42:56.602596+0000","last_peered":"2023-05-22T15:42:56.602596+0000","last_clean":"2023-05-22T15:42:56.602596+0000","last_became_active":"2023-05-22T15:42:56.239901+0000","last_became_peered":"2023-05-22T15:42:56.239901+0000","last_unstale":"2023-05-22T15:42:56.602596+0000","last_undegraded":"2023-05-22T15:42:56.602596+0000","last_fullsized":"2023-05-22T15:42:56.602596+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:55.181398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:55.181398+0000","last_clean_scrub_stamp":"2023-05-22T15:42:55.181398+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:23:14.852630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":94,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.223238+0000","last_change":"2023-05-22T15:26:44.977198+0000","last_active":"2023-05-22T15:42:56.223238+0000","last_peered":"2023-05-22T15:42:56.223238+0000","last_clean":"2023-05-22T15:42:56.223238+0000","last_became_active":"2023-05-22T15:26:44.976445+0000","last_became_peered":"2023-05-22T15:26:44.976445+0000","last_unstale":"2023-05-22T15:42:56.223238+0000","last_undegraded":"2023-05-22T15:42:56.223238+0000","last_fullsized":"2023-05-22T15:42:56.223238+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:12:41.464054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.e","version":"0'0","reported_seq":9,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.643988+0000","last_change":"2023-05-22T15:42:56.639805+0000","last_active":"2023-05-22T15:42:56.643988+0000","last_peered":"2023-05-22T15:42:56.643988+0000","last_clean":"2023-05-22T15:42:56.643988+0000","last_became_active":"2023-05-22T15:42:56.639089+0000","last_became_peered":"2023-05-22T15:42:56.639089+0000","last_unstale":"2023-05-22T15:42:56.643988+0000","last_undegraded":"2023-05-22T15:42:56.643988+0000","last_fullsized":"2023-05-22T15:42:56.643988+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_clean_scrub_stamp":"2023-05-22T15:42:56.185172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:01:28.084697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.f","version":"103'1","reported_seq":11,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.646773+0000","last_change":"2023-05-22T15:42:56.247444+0000","last_active":"2023-05-22T15:42:56.646773+0000","last_peered":"2023-05-22T15:42:56.646773+0000","last_clean":"2023-05-22T15:42:56.646773+0000","last_became_active":"2023-05-22T15:42:56.245737+0000","last_became_peered":"2023-05-22T15:42:56.245737+0000","last_unstale":"2023-05-22T15:42:56.646773+0000","last_undegraded":"2023-05-22T15:42:56.646773+0000","last_fullsized":"2023-05-22T15:42:56.646773+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:55.181398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:55.181398+0000","last_clean_scrub_stamp":"2023-05-22T15:42:55.181398+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:28:44.717288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":92,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.224362+0000","last_change":"2023-05-22T15:26:44.969469+0000","last_active":"2023-05-22T15:42:56.224362+0000","last_peered":"2023-05-22T15:42:56.224362+0000","last_clean":"2023-05-22T15:42:56.224362+0000","last_became_active":"2023-05-22T15:26:44.969083+0000","last_became_peered":"2023-05-22T15:26:44.969083+0000","last_unstale":"2023-05-22T15:42:56.224362+0000","last_undegraded":"2023-05-22T15:42:56.224362+0000","last_fullsized":"2023-05-22T15:42:56.224362+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:29:23.670067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.d","version":"0'0","reported_seq":9,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.611994+0000","last_change":"2023-05-22T15:42:56.597937+0000","last_active":"2023-05-22T15:42:56.611994+0000","last_peered":"2023-05-22T15:42:56.611994+0000","last_clean":"2023-05-22T15:42:56.611994+0000","last_became_active":"2023-05-22T15:42:56.597160+0000","last_became_peered":"2023-05-22T15:42:56.597160+0000","last_unstale":"2023-05-22T15:42:56.611994+0000","last_undegraded":"2023-05-22T15:42:56.611994+0000","last_fullsized":"2023-05-22T15:42:56.611994+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_clean_scrub_stamp":"2023-05-22T15:42:56.185172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:59:56.263719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":10,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.573067+0000","last_change":"2023-05-22T15:42:56.231395+0000","last_active":"2023-05-22T15:42:56.573067+0000","last_peered":"2023-05-22T15:42:56.573067+0000","last_clean":"2023-05-22T15:42:56.573067+0000","last_became_active":"2023-05-22T15:42:56.231099+0000","last_became_peered":"2023-05-22T15:42:56.231099+0000","last_unstale":"2023-05-22T15:42:56.573067+0000","last_undegraded":"2023-05-22T15:42:56.573067+0000","last_fullsized":"2023-05-22T15:42:56.573067+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:55.181398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:55.181398+0000","last_clean_scrub_stamp":"2023-05-22T15:42:55.181398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:58:12.427273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":97,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.224671+0000","last_change":"2023-05-22T15:26:44.976632+0000","last_active":"2023-05-22T15:42:56.224671+0000","last_peered":"2023-05-22T15:42:56.224671+0000","last_clean":"2023-05-22T15:42:56.224671+0000","last_became_active":"2023-05-22T15:26:44.975546+0000","last_became_peered":"2023-05-22T15:26:44.975546+0000","last_unstale":"2023-05-22T15:42:56.224671+0000","last_undegraded":"2023-05-22T15:42:56.224671+0000","last_fullsized":"2023-05-22T15:42:56.224671+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:42:57.973030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.b","version":"0'0","reported_seq":9,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.629940+0000","last_change":"2023-05-22T15:42:56.601620+0000","last_active":"2023-05-22T15:42:56.629940+0000","last_peered":"2023-05-22T15:42:56.629940+0000","last_clean":"2023-05-22T15:42:56.629940+0000","last_became_active":"2023-05-22T15:42:56.600789+0000","last_became_peered":"2023-05-22T15:42:56.600789+0000","last_unstale":"2023-05-22T15:42:56.629940+0000","last_undegraded":"2023-05-22T15:42:56.629940+0000","last_fullsized":"2023-05-22T15:42:56.629940+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_clean_scrub_stamp":"2023-05-22T15:42:56.185172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:48:43.959640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":10,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.577390+0000","last_change":"2023-05-22T15:42:56.226626+0000","last_active":"2023-05-22T15:42:56.577390+0000","last_peered":"2023-05-22T15:42:56.577390+0000","last_clean":"2023-05-22T15:42:56.577390+0000","last_became_active":"2023-05-22T15:42:56.224567+0000","last_became_peered":"2023-05-22T15:42:56.224567+0000","last_unstale":"2023-05-22T15:42:56.577390+0000","last_undegraded":"2023-05-22T15:42:56.577390+0000","last_fullsized":"2023-05-22T15:42:56.577390+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:55.181398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:55.181398+0000","last_clean_scrub_stamp":"2023-05-22T15:42:55.181398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T03:04:27.047889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":86,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.214969+0000","last_change":"2023-05-22T15:26:45.091475+0000","last_active":"2023-05-22T15:42:56.214969+0000","last_peered":"2023-05-22T15:42:56.214969+0000","last_clean":"2023-05-22T15:42:56.214969+0000","last_became_active":"2023-05-22T15:26:45.091143+0000","last_became_peered":"2023-05-22T15:26:45.091143+0000","last_unstale":"2023-05-22T15:42:56.214969+0000","last_undegraded":"2023-05-22T15:42:56.214969+0000","last_fullsized":"2023-05-22T15:42:56.214969+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:12:41.464054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":9,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.600088+0000","last_change":"2023-05-22T15:42:56.598005+0000","last_active":"2023-05-22T15:42:56.600088+0000","last_peered":"2023-05-22T15:42:56.600088+0000","last_clean":"2023-05-22T15:42:56.600088+0000","last_became_active":"2023-05-22T15:42:56.597699+0000","last_became_peered":"2023-05-22T15:42:56.597699+0000","last_unstale":"2023-05-22T15:42:56.600088+0000","last_undegraded":"2023-05-22T15:42:56.600088+0000","last_fullsized":"2023-05-22T15:42:56.600088+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_clean_scrub_stamp":"2023-05-22T15:42:56.185172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:53:23.214996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"103'2","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.591730+0000","last_change":"2023-05-22T15:42:56.228716+0000","last_active":"2023-05-22T15:42:56.591730+0000","last_peered":"2023-05-22T15:42:56.591730+0000","last_clean":"2023-05-22T15:42:56.591730+0000","last_became_active":"2023-05-22T15:42:56.228373+0000","last_became_peered":"2023-05-22T15:42:56.228373+0000","last_unstale":"2023-05-22T15:42:56.591730+0000","last_undegraded":"2023-05-22T15:42:56.591730+0000","last_fullsized":"2023-05-22T15:42:56.591730+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:55.181398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:55.181398+0000","last_clean_scrub_stamp":"2023-05-22T15:42:55.181398+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:33:41.882434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":93,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.223741+0000","last_change":"2023-05-22T15:26:45.092367+0000","last_active":"2023-05-22T15:42:56.223741+0000","last_peered":"2023-05-22T15:42:56.223741+0000","last_clean":"2023-05-22T15:42:56.223741+0000","last_became_active":"2023-05-22T15:26:45.091951+0000","last_became_peered":"2023-05-22T15:26:45.091951+0000","last_unstale":"2023-05-22T15:42:56.223741+0000","last_undegraded":"2023-05-22T15:42:56.223741+0000","last_fullsized":"2023-05-22T15:42:56.223741+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:42:57.973030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":9,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.630003+0000","last_change":"2023-05-22T15:42:56.608910+0000","last_active":"2023-05-22T15:42:56.630003+0000","last_peered":"2023-05-22T15:42:56.630003+0000","last_clean":"2023-05-22T15:42:56.630003+0000","last_became_active":"2023-05-22T15:42:56.608054+0000","last_became_peered":"2023-05-22T15:42:56.608054+0000","last_unstale":"2023-05-22T15:42:56.630003+0000","last_undegraded":"2023-05-22T15:42:56.630003+0000","last_fullsized":"2023-05-22T15:42:56.630003+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_clean_scrub_stamp":"2023-05-22T15:42:56.185172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:31:37.786481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":10,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.586380+0000","last_change":"2023-05-22T15:42:56.228719+0000","last_active":"2023-05-22T15:42:56.586380+0000","last_peered":"2023-05-22T15:42:56.586380+0000","last_clean":"2023-05-22T15:42:56.586380+0000","last_became_active":"2023-05-22T15:42:56.228366+0000","last_became_peered":"2023-05-22T15:42:56.228366+0000","last_unstale":"2023-05-22T15:42:56.586380+0000","last_undegraded":"2023-05-22T15:42:56.586380+0000","last_fullsized":"2023-05-22T15:42:56.586380+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:55.181398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:55.181398+0000","last_clean_scrub_stamp":"2023-05-22T15:42:55.181398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:58:12.427273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":86,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.215002+0000","last_change":"2023-05-22T15:26:44.978858+0000","last_active":"2023-05-22T15:42:56.215002+0000","last_peered":"2023-05-22T15:42:56.215002+0000","last_clean":"2023-05-22T15:42:56.215002+0000","last_became_active":"2023-05-22T15:26:44.978443+0000","last_became_peered":"2023-05-22T15:26:44.978443+0000","last_unstale":"2023-05-22T15:42:56.215002+0000","last_undegraded":"2023-05-22T15:42:56.215002+0000","last_fullsized":"2023-05-22T15:42:56.215002+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:46:29.290243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":9,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.614857+0000","last_change":"2023-05-22T15:42:56.614340+0000","last_active":"2023-05-22T15:42:56.614857+0000","last_peered":"2023-05-22T15:42:56.614857+0000","last_clean":"2023-05-22T15:42:56.614857+0000","last_became_active":"2023-05-22T15:42:56.613999+0000","last_became_peered":"2023-05-22T15:42:56.613999+0000","last_unstale":"2023-05-22T15:42:56.614857+0000","last_undegraded":"2023-05-22T15:42:56.614857+0000","last_fullsized":"2023-05-22T15:42:56.614857+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:56.185172+0000","last_clean_scrub_stamp":"2023-05-22T15:42:56.185172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:28:45.721063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":10,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.576955+0000","last_change":"2023-05-22T15:42:56.244970+0000","last_active":"2023-05-22T15:42:56.576955+0000","last_peered":"2023-05-22T15:42:56.576955+0000","last_clean":"2023-05-22T15:42:56.576955+0000","last_became_active":"2023-05-22T15:42:56.244657+0000","last_became_peered":"2023-05-22T15:42:56.244657+0000","last_unstale":"2023-05-22T15:42:56.576955+0000","last_undegraded":"2023-05-22T15:42:56.576955+0000","last_fullsized":"2023-05-22T15:42:56.576955+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:42:55.181398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:42:55.181398+0000","last_clean_scrub_stamp":"2023-05-22T15:42:55.181398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:33:14.769505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":88,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-05-22T15:42:56.233924+0000","last_change":"2023-05-22T15:26:44.982421+0000","last_active":"2023-05-22T15:42:56.233924+0000","last_peered":"2023-05-22T15:42:56.233924+0000","last_clean":"2023-05-22T15:42:56.233924+0000","last_became_active":"2023-05-22T15:26:44.981586+0000","last_became_peered":"2023-05-22T15:26:44.981586+0000","last_unstale":"2023-05-22T15:42:56.233924+0000","last_undegraded":"2023-05-22T15:42:56.233924+0000","last_fullsized":"2023-05-22T15:42:56.233924+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:14:11.156544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":13,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":12,"num_pg":32,"stat_sum":{"num_bytes":2330,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":692,"num_read_kb":1336,"num_write":279,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":183,"ondisk_log_size":183,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575045,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30560,"kb_used_data":1816,"kb_used_omap":0,"kb_used_meta":28736,"kb_avail":94341280,"statfs":{"total":96636764160,"available":96605470720,"internally_reserved":0,"allocated":1859584,"data_stored":1251625,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":29425664},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 15:41:58 2023","interfaces":[{"interface":"back","average":{"1min":0.88900000000000001,"5min":0.86099999999999999,"15min":0.86699999999999999},"min":{"1min":0.624,"5min":0.44800000000000001,"15min":0.40899999999999997},"max":{"1min":1.1830000000000001,"5min":1.4199999999999999,"15min":1.7709999999999999},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.86899999999999999,"5min":0.871,"15min":0.876},"min":{"1min":0.63600000000000001,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":1.2050000000000001,"5min":1.3169999999999999,"15min":1.4239999999999999},"last":0.95899999999999996}]},{"osd":1,"last update":"Mon May 22 15:41:58 2023","interfaces":[{"interface":"back","average":{"1min":0.94799999999999995,"5min":0.98199999999999998,"15min":0.98199999999999998},"min":{"1min":0.77300000000000002,"5min":0.5,"15min":0.5},"max":{"1min":1.385,"5min":1.389,"15min":1.8300000000000001},"last":1.0980000000000001},{"interface":"front","average":{"1min":1.0229999999999999,"5min":1.018,"15min":1.0229999999999999},"min":{"1min":0.60799999999999998,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":1.2130000000000001,"5min":1.3169999999999999,"15min":2.161},"last":1.0580000000000001}]},{"osd":2,"last update":"Mon May 22 15:41:58 2023","interfaces":[{"interface":"back","average":{"1min":0.84899999999999998,"5min":0.87,"15min":0.90400000000000003},"min":{"1min":0.67300000000000004,"5min":0.49399999999999999,"15min":0.46800000000000003},"max":{"1min":1.181,"5min":1.2070000000000001,"15min":6.7279999999999998},"last":0.90000000000000002},{"interface":"front","average":{"1min":0.95799999999999996,"5min":0.94799999999999995,"15min":0.96299999999999997},"min":{"1min":0.59199999999999997,"5min":0.52300000000000002,"15min":0.51000000000000001},"max":{"1min":1.151,"5min":1.2569999999999999,"15min":6.8810000000000002},"last":1.0349999999999999}]},{"osd":3,"last update":"Mon May 22 15:41:58 2023","interfaces":[{"interface":"back","average":{"1min":1.0109999999999999,"5min":0.95299999999999996,"15min":0.96099999999999997},"min":{"1min":0.70199999999999996,"5min":0.55900000000000005,"15min":0.53800000000000003},"max":{"1min":1.254,"5min":1.3360000000000001,"15min":3.0569999999999999},"last":0.69999999999999996},{"interface":"front","average":{"1min":0.91500000000000004,"5min":0.96599999999999997,"15min":0.98399999999999999},"min":{"1min":0.68200000000000005,"5min":0.61799999999999999,"15min":0.54900000000000004},"max":{"1min":1.2989999999999999,"5min":1.3029999999999999,"15min":1.6879999999999999},"last":1.0029999999999999}]},{"osd":4,"last update":"Mon May 22 15:41:58 2023","interfaces":[{"interface":"back","average":{"1min":0.83599999999999997,"5min":0.89600000000000002,"15min":0.88400000000000001},"min":{"1min":0.62,"5min":0.57399999999999995,"15min":0.57399999999999995},"max":{"1min":1.093,"5min":1.621,"15min":1.621},"last":0.86599999999999999},{"interface":"front","average":{"1min":0.98299999999999998,"5min":0.99199999999999999,"15min":0.98299999999999998},"min":{"1min":0.749,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.359,"5min":1.4319999999999999,"15min":1.958},"last":0.91600000000000004}]},{"osd":5,"last update":"Mon May 22 15:41:58 2023","interfaces":[{"interface":"back","average":{"1min":0.995,"5min":1.022,"15min":1.002},"min":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.56699999999999995},"max":{"1min":1.3939999999999999,"5min":1.409,"15min":1.5029999999999999},"last":0.82599999999999996},{"interface":"front","average":{"1min":1.1220000000000001,"5min":1.121,"15min":1.1359999999999999},"min":{"1min":0.88800000000000001,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":1.325,"5min":1.5229999999999999,"15min":5.0839999999999996},"last":1.119}]},{"osd":6,"last update":"Mon May 22 15:41:58 2023","interfaces":[{"interface":"back","average":{"1min":1.0169999999999999,"5min":1.0449999999999999,"15min":1.071},"min":{"1min":0.73399999999999999,"5min":0.65800000000000003,"15min":0.65800000000000003},"max":{"1min":1.256,"5min":1.484,"15min":5.1719999999999997},"last":1.079},{"interface":"front","average":{"1min":1.109,"5min":1.133,"15min":1.1339999999999999},"min":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"max":{"1min":1.462,"5min":1.702,"15min":2.04},"last":0.80100000000000005}]}]},{"osd":6,"up_from":13,"seq":55834575046,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30348,"kb_used_data":1220,"kb_used_omap":0,"kb_used_meta":29120,"kb_avail":94341492,"statfs":{"total":96636764160,"available":96605687808,"internally_reserved":0,"allocated":1249280,"data_stored":653739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":29818880},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 15:42:06 2023","interfaces":[{"interface":"back","average":{"1min":0.86599999999999999,"5min":0.89000000000000001,"15min":0.89800000000000002},"min":{"1min":0.59299999999999997,"5min":0.496,"15min":0.496},"max":{"1min":1.323,"5min":1.327,"15min":1.355},"last":0.71499999999999997},{"interface":"front","average":{"1min":0.99199999999999999,"5min":0.90300000000000002,"15min":0.90900000000000003},"min":{"1min":0.66200000000000003,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":1.264,"5min":1.5680000000000001,"15min":1.5680000000000001},"last":1.155}]},{"osd":1,"last update":"Mon May 22 15:42:06 2023","interfaces":[{"interface":"back","average":{"1min":0.92800000000000005,"5min":0.92800000000000005,"15min":0.91300000000000003},"min":{"1min":0.623,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":1.2709999999999999,"5min":1.3169999999999999,"15min":1.375},"last":0.82499999999999996},{"interface":"front","average":{"1min":0.93400000000000005,"5min":0.94499999999999995,"15min":0.94899999999999995},"min":{"1min":0.64400000000000002,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.2709999999999999,"5min":1.554,"15min":1.554},"last":1.325}]},{"osd":2,"last update":"Mon May 22 15:42:06 2023","interfaces":[{"interface":"back","average":{"1min":1.0029999999999999,"5min":0.97099999999999997,"15min":0.97999999999999998},"min":{"1min":0.72599999999999998,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":1.3660000000000001,"5min":1.425,"15min":1.425},"last":1.0309999999999999},{"interface":"front","average":{"1min":1.073,"5min":1.048,"15min":1.0509999999999999},"min":{"1min":0.84299999999999997,"5min":0.442,"15min":0.442},"max":{"1min":1.2909999999999999,"5min":1.5369999999999999,"15min":1.5369999999999999},"last":1.0640000000000001}]},{"osd":3,"last update":"Mon May 22 15:42:06 2023","interfaces":[{"interface":"back","average":{"1min":0.95799999999999996,"5min":0.99099999999999999,"15min":0.99399999999999999},"min":{"1min":0.72599999999999998,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":1.1830000000000001,"5min":1.679,"15min":1.679},"last":0.96699999999999997},{"interface":"front","average":{"1min":0.96799999999999997,"5min":0.95699999999999996,"15min":0.96099999999999997},"min":{"1min":0.62,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.159,"5min":1.4570000000000001,"15min":1.486},"last":0.85199999999999998}]},{"osd":4,"last update":"Mon May 22 15:42:06 2023","interfaces":[{"interface":"back","average":{"1min":0.85699999999999998,"5min":0.91700000000000004,"15min":0.93100000000000005},"min":{"1min":0.64900000000000002,"5min":0.60299999999999998,"15min":0.57599999999999996},"max":{"1min":1.1479999999999999,"5min":1.6859999999999999,"15min":1.6859999999999999},"last":0.91800000000000004},{"interface":"front","average":{"1min":0.96199999999999997,"5min":1.016,"15min":1.032},"min":{"1min":0.67700000000000005,"5min":0.622,"15min":0.61299999999999999},"max":{"1min":1.321,"5min":1.8300000000000001,"15min":1.8300000000000001},"last":1.2509999999999999}]},{"osd":5,"last update":"Mon May 22 15:42:06 2023","interfaces":[{"interface":"back","average":{"1min":1.01,"5min":0.98099999999999998,"15min":0.97999999999999998},"min":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.64400000000000002},"max":{"1min":1.4530000000000001,"5min":1.6499999999999999,"15min":1.6499999999999999},"last":1.1160000000000001},{"interface":"front","average":{"1min":1.0760000000000001,"5min":1.103,"15min":1.1000000000000001},"min":{"1min":0.68799999999999994,"5min":0.68000000000000005,"15min":0.68000000000000005},"max":{"1min":1.3919999999999999,"5min":1.607,"15min":1.7509999999999999},"last":1.2170000000000001}]},{"osd":7,"last update":"Mon May 22 15:42:06 2023","interfaces":[{"interface":"back","average":{"1min":1.1180000000000001,"5min":1.1160000000000001,"15min":1.1359999999999999},"min":{"1min":0.78700000000000003,"5min":0.68400000000000005,"15min":0.58399999999999996},"max":{"1min":1.3740000000000001,"5min":1.9179999999999999,"15min":1.9179999999999999},"last":1.3859999999999999},{"interface":"front","average":{"1min":1.1040000000000001,"5min":1.139,"15min":1.133},"min":{"1min":0.81899999999999995,"5min":0.74099999999999999,"15min":0.68999999999999995},"max":{"1min":1.3580000000000001,"5min":1.8520000000000001,"15min":1.8520000000000001},"last":1.282}]}]},{"osd":5,"up_from":13,"seq":55834575046,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":24296,"kb_used_data":1248,"kb_used_omap":0,"kb_used_meta":23040,"kb_avail":94347544,"statfs":{"total":96636764160,"available":96611885056,"internally_reserved":0,"allocated":1277952,"data_stored":662757,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":23592960},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 15:42:04 2023","interfaces":[{"interface":"back","average":{"1min":0.92100000000000004,"5min":0.86799999999999999,"15min":0.88},"min":{"1min":0.68899999999999995,"5min":0.54900000000000004,"15min":0.48399999999999999},"max":{"1min":1.2849999999999999,"5min":1.3440000000000001,"15min":1.3440000000000001},"last":0.93300000000000005},{"interface":"front","average":{"1min":0.93600000000000005,"5min":0.89800000000000002,"15min":0.88},"min":{"1min":0.58499999999999996,"5min":0.57499999999999996,"15min":0.497},"max":{"1min":1.2549999999999999,"5min":2.1699999999999999,"15min":2.1699999999999999},"last":0.58099999999999996}]},{"osd":1,"last update":"Mon May 22 15:42:04 2023","interfaces":[{"interface":"back","average":{"1min":1.099,"5min":1.004,"15min":0.996},"min":{"1min":0.87,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":1.337,"5min":1.9370000000000001,"15min":1.9370000000000001},"last":0.878},{"interface":"front","average":{"1min":1.028,"5min":0.999,"15min":1.02},"min":{"1min":0.67000000000000004,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.3680000000000001,"5min":1.3680000000000001,"15min":1.399},"last":1.1819999999999999}]},{"osd":2,"last update":"Mon May 22 15:42:04 2023","interfaces":[{"interface":"back","average":{"1min":0.95699999999999996,"5min":0.96299999999999997,"15min":0.93200000000000005},"min":{"1min":0.72699999999999998,"5min":0.504,"15min":0.504},"max":{"1min":1.2549999999999999,"5min":1.9830000000000001,"15min":1.9830000000000001},"last":0.82099999999999995},{"interface":"front","average":{"1min":1.0189999999999999,"5min":0.96099999999999997,"15min":0.93700000000000006},"min":{"1min":0.75900000000000001,"5min":0.60799999999999998,"15min":0.53800000000000003},"max":{"1min":1.321,"5min":2.1120000000000001,"15min":2.1120000000000001},"last":1.218}]},{"osd":3,"last update":"Mon May 22 15:42:04 2023","interfaces":[{"interface":"back","average":{"1min":1.0029999999999999,"5min":0.98899999999999999,"15min":0.95699999999999996},"min":{"1min":0.67000000000000004,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.2809999999999999,"5min":1.4419999999999999,"15min":1.6839999999999999},"last":0.79100000000000004},{"interface":"front","average":{"1min":0.97399999999999998,"5min":1.056,"15min":1.038},"min":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":1.1970000000000001,"5min":2.52,"15min":2.52},"last":0.999}]},{"osd":4,"last update":"Mon May 22 15:42:04 2023","interfaces":[{"interface":"back","average":{"1min":0.90900000000000003,"5min":0.97799999999999998,"15min":0.93799999999999994},"min":{"1min":0.68200000000000005,"5min":0.60599999999999998,"15min":0.57599999999999996},"max":{"1min":1.2609999999999999,"5min":2.226,"15min":2.226},"last":0.73799999999999999},{"interface":"front","average":{"1min":1.075,"5min":1.0649999999999999,"15min":1.038},"min":{"1min":0.63200000000000001,"5min":0.60199999999999998,"15min":0.60199999999999998},"max":{"1min":1.4239999999999999,"5min":1.6859999999999999,"15min":1.6859999999999999},"last":1.0940000000000001}]},{"osd":6,"last update":"Mon May 22 15:42:04 2023","interfaces":[{"interface":"back","average":{"1min":1.016,"5min":1.0840000000000001,"15min":1.0449999999999999},"min":{"1min":0.748,"5min":0.67900000000000005,"15min":0.61299999999999999},"max":{"1min":1.345,"5min":2.831,"15min":2.831},"last":1.2929999999999999},{"interface":"front","average":{"1min":1.085,"5min":1.1080000000000001,"15min":1.0780000000000001},"min":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.69299999999999995},"max":{"1min":1.3380000000000001,"5min":2.3690000000000002,"15min":2.3690000000000002},"last":1.046}]},{"osd":7,"last update":"Mon May 22 15:42:04 2023","interfaces":[{"interface":"back","average":{"1min":1.077,"5min":1.1639999999999999,"15min":1.125},"min":{"1min":0.70999999999999996,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":1.47,"5min":1.849,"15min":2.8260000000000001},"last":1.2609999999999999},{"interface":"front","average":{"1min":1.1579999999999999,"5min":1.22,"15min":1.1699999999999999},"min":{"1min":0.80400000000000005,"5min":0.75700000000000001,"15min":0.71699999999999997},"max":{"1min":1.3220000000000001,"5min":2.7429999999999999,"15min":2.7429999999999999},"last":1.135}]}]},{"osd":4,"up_from":13,"seq":55834575044,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":26072,"kb_used_data":1232,"kb_used_omap":0,"kb_used_meta":24832,"kb_avail":94345768,"statfs":{"total":96636764160,"available":96610066432,"internally_reserved":0,"allocated":1261568,"data_stored":661187,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":25427968},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 15:42:07 2023","interfaces":[{"interface":"back","average":{"1min":0.88,"5min":0.91300000000000003,"15min":0.90600000000000003},"min":{"1min":0.58299999999999996,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":1.3360000000000001,"5min":1.9910000000000001,"15min":1.9910000000000001},"last":0.91400000000000003},{"interface":"front","average":{"1min":0.96899999999999997,"5min":0.91900000000000004,"15min":0.93200000000000005},"min":{"1min":0.63500000000000001,"5min":0.373,"15min":0.373},"max":{"1min":1.306,"5min":1.3779999999999999,"15min":1.633},"last":1.0109999999999999}]},{"osd":1,"last update":"Mon May 22 15:42:07 2023","interfaces":[{"interface":"back","average":{"1min":1.0029999999999999,"5min":0.98199999999999998,"15min":0.97699999999999998},"min":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.54000000000000004},"max":{"1min":1.2689999999999999,"5min":1.383,"15min":1.3859999999999999},"last":0.63},{"interface":"front","average":{"1min":0.88900000000000001,"5min":0.93500000000000005,"15min":0.91200000000000003},"min":{"1min":0.60299999999999998,"5min":0.55100000000000005,"15min":0.51600000000000001},"max":{"1min":1.149,"5min":1.3120000000000001,"15min":1.3640000000000001},"last":0.56799999999999995}]},{"osd":2,"last update":"Mon May 22 15:42:07 2023","interfaces":[{"interface":"back","average":{"1min":0.92800000000000005,"5min":0.94699999999999995,"15min":0.94699999999999995},"min":{"1min":0.747,"5min":0.495,"15min":0.495},"max":{"1min":1.212,"5min":2.1019999999999999,"15min":2.1019999999999999},"last":1.2769999999999999},{"interface":"front","average":{"1min":1.0489999999999999,"5min":1.01,"15min":1.0109999999999999},"min":{"1min":0.77600000000000002,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":1.4390000000000001,"5min":1.4390000000000001,"15min":1.9770000000000001},"last":0.76600000000000001}]},{"osd":3,"last update":"Mon May 22 15:42:07 2023","interfaces":[{"interface":"back","average":{"1min":0.89800000000000002,"5min":0.91800000000000004,"15min":0.94299999999999995},"min":{"1min":0.72399999999999998,"5min":0.63100000000000001,"15min":0.54600000000000004},"max":{"1min":1.2490000000000001,"5min":1.29,"15min":1.665},"last":0.83099999999999996},{"interface":"front","average":{"1min":1.075,"5min":1.0529999999999999,"15min":1.038},"min":{"1min":0.85399999999999998,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":1.238,"5min":2.1720000000000002,"15min":2.1720000000000002},"last":1.1319999999999999}]},{"osd":5,"last update":"Mon May 22 15:42:07 2023","interfaces":[{"interface":"back","average":{"1min":1.0569999999999999,"5min":1.0409999999999999,"15min":1},"min":{"1min":0.83399999999999996,"5min":0.621,"15min":0.51200000000000001},"max":{"1min":1.373,"5min":1.472,"15min":1.7270000000000001},"last":0.873},{"interface":"front","average":{"1min":1.012,"5min":0.98599999999999999,"15min":0.97999999999999998},"min":{"1min":0.70799999999999996,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":1.325,"5min":1.401,"15min":1.6719999999999999},"last":0.96399999999999997}]},{"osd":6,"last update":"Mon May 22 15:42:07 2023","interfaces":[{"interface":"back","average":{"1min":1.0089999999999999,"5min":1.028,"15min":1.0029999999999999},"min":{"1min":0.64500000000000002,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":1.351,"5min":2.2349999999999999,"15min":2.2349999999999999},"last":1.2290000000000001},{"interface":"front","average":{"1min":1.1379999999999999,"5min":1.1419999999999999,"15min":1.1160000000000001},"min":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.69799999999999995},"max":{"1min":1.486,"5min":1.8220000000000001,"15min":2.0699999999999998},"last":1.103}]},{"osd":7,"last update":"Mon May 22 15:42:07 2023","interfaces":[{"interface":"back","average":{"1min":1.087,"5min":1.0580000000000001,"15min":1.0680000000000001},"min":{"1min":0.78200000000000003,"5min":0.71899999999999997,"15min":0.67700000000000005},"max":{"1min":1.3140000000000001,"5min":1.5800000000000001,"15min":1.6639999999999999},"last":1.069},{"interface":"front","average":{"1min":1.1930000000000001,"5min":1.1639999999999999,"15min":1.1459999999999999},"min":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.71499999999999997},"max":{"1min":1.4370000000000001,"5min":2.306,"15min":2.306},"last":1.327}]}]},{"osd":3,"up_from":13,"seq":55834575044,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35108,"kb_used_data":1244,"kb_used_omap":0,"kb_used_meta":33856,"kb_avail":94336732,"statfs":{"total":96636764160,"available":96600813568,"internally_reserved":0,"allocated":1273856,"data_stored":662735,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":34668544},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 15:42:54 2023","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.61199999999999999,"15min":0.628},"min":{"1min":0.312,"5min":0.19,"15min":0.19},"max":{"1min":1.1519999999999999,"5min":1.194,"15min":1.194},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.68700000000000006,"15min":0.69999999999999996},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.88300000000000001,"5min":1.1799999999999999,"15min":1.296},"last":0.373}]},{"osd":1,"last update":"Mon May 22 15:42:54 2023","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.61899999999999999,"15min":0.66300000000000003},"min":{"1min":0.315,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.80000000000000004,"5min":1.008,"15min":1.0600000000000001},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.71899999999999997,"15min":0.75800000000000001},"min":{"1min":0.34399999999999997,"5min":0.313,"15min":0.216},"max":{"1min":0.82299999999999995,"5min":1.222,"15min":1.222},"last":0.42599999999999999}]},{"osd":2,"last update":"Mon May 22 15:42:54 2023","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.76700000000000002,"15min":0.81200000000000006},"min":{"1min":0.41599999999999998,"5min":0.36599999999999999,"15min":0.32200000000000001},"max":{"1min":1.506,"5min":1.506,"15min":1.506},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.748,"15min":0.80200000000000005},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.29699999999999999},"max":{"1min":1.097,"5min":1.181,"15min":1.468},"last":0.57399999999999995}]},{"osd":4,"last update":"Mon May 22 15:42:54 2023","interfaces":[{"interface":"back","average":{"1min":0.84399999999999997,"5min":0.92300000000000004,"15min":0.95299999999999996},"min":{"1min":0.54500000000000004,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":1.413,"5min":1.413,"15min":1.4470000000000001},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.81799999999999995,"5min":0.94699999999999995,"15min":1.0009999999999999},"min":{"1min":0.64800000000000002,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.077,"5min":1.3740000000000001,"15min":1.4470000000000001},"last":0.81599999999999995}]},{"osd":5,"last update":"Mon May 22 15:42:54 2023","interfaces":[{"interface":"back","average":{"1min":0.83199999999999996,"5min":0.96099999999999997,"15min":1.01},"min":{"1min":0.53200000000000003,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":1.2070000000000001,"5min":1.3149999999999999,"15min":1.4339999999999999},"last":0.872},{"interface":"front","average":{"1min":0.94899999999999995,"5min":0.98999999999999999,"15min":1.0429999999999999},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.4299999999999999,"5min":1.4299999999999999,"15min":1.643},"last":0.68200000000000005}]},{"osd":6,"last update":"Mon May 22 15:42:54 2023","interfaces":[{"interface":"back","average":{"1min":0.85099999999999998,"5min":0.98099999999999998,"15min":1.0369999999999999},"min":{"1min":0.47699999999999998,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.1539999999999999,"5min":1.4139999999999999,"15min":1.863},"last":0.75600000000000001},{"interface":"front","average":{"1min":0.93100000000000005,"5min":1.0229999999999999,"15min":1.0720000000000001},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":1.2909999999999999,"5min":1.3109999999999999,"15min":1.704},"last":0.84199999999999997}]},{"osd":7,"last update":"Mon May 22 15:42:54 2023","interfaces":[{"interface":"back","average":{"1min":0.96699999999999997,"5min":1.0469999999999999,"15min":1.1000000000000001},"min":{"1min":0.58599999999999997,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":1.407,"5min":1.407,"15min":1.5640000000000001},"last":0.79300000000000004},{"interface":"front","average":{"1min":0.94099999999999995,"5min":1.024,"15min":1.0700000000000001},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":1.8700000000000001,"5min":1.8700000000000001,"15min":1.8700000000000001},"last":0.64900000000000002}]}]},{"osd":2,"up_from":13,"seq":55834575045,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":17372,"kb_used_data":1236,"kb_used_omap":0,"kb_used_meta":16128,"kb_avail":94354468,"statfs":{"total":96636764160,"available":96618975232,"internally_reserved":0,"allocated":1265664,"data_stored":661753,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":16515072},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 15:41:55 2023","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.66000000000000003,"15min":0.65000000000000002},"min":{"1min":0.499,"5min":0.35699999999999998,"15min":0.317},"max":{"1min":0.96999999999999997,"5min":1.8240000000000001,"15min":1.8240000000000001},"last":0.28599999999999998},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.77600000000000002,"15min":0.74299999999999999},"min":{"1min":0.52500000000000002,"5min":0.39600000000000002,"15min":0.253},"max":{"1min":1.0740000000000001,"5min":1.8160000000000001,"15min":1.8160000000000001},"last":0.57499999999999996}]},{"osd":1,"last update":"Mon May 22 15:41:55 2023","interfaces":[{"interface":"back","average":{"1min":0.80600000000000005,"5min":0.74399999999999999,"15min":0.73599999999999999},"min":{"1min":0.46700000000000003,"5min":0.27000000000000002,"15min":0.26900000000000002},"max":{"1min":1.0329999999999999,"5min":1.2,"15min":1.5309999999999999},"last":0.374},{"interface":"front","average":{"1min":0.877,"5min":0.83899999999999997,"15min":0.80700000000000005},"min":{"1min":0.60099999999999998,"5min":0.39600000000000002,"15min":0.29099999999999998},"max":{"1min":1.246,"5min":2.5089999999999999,"15min":2.5089999999999999},"last":0.63300000000000001}]},{"osd":3,"last update":"Mon May 22 15:41:55 2023","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.72799999999999998,"15min":0.72099999999999997},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.26700000000000002},"max":{"1min":0.96199999999999997,"5min":2.7629999999999999,"15min":2.7629999999999999},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.85699999999999998,"5min":0.89700000000000002,"15min":0.873},"min":{"1min":0.56100000000000005,"5min":0.44400000000000001,"15min":0.33000000000000002},"max":{"1min":1.042,"5min":2.6840000000000002,"15min":2.6840000000000002},"last":0.433}]},{"osd":4,"last update":"Mon May 22 15:41:55 2023","interfaces":[{"interface":"back","average":{"1min":1.012,"5min":1.0329999999999999,"15min":1.0109999999999999},"min":{"1min":0.67700000000000005,"5min":0.65800000000000003,"15min":0.55500000000000005},"max":{"1min":1.244,"5min":2.8039999999999998,"15min":2.8039999999999998},"last":0.76500000000000001},{"interface":"front","average":{"1min":1.0009999999999999,"5min":1.05,"15min":1.024},"min":{"1min":0.77000000000000002,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":1.3,"5min":2.0819999999999999,"15min":2.0819999999999999},"last":0.73699999999999999}]},{"osd":5,"last update":"Mon May 22 15:41:55 2023","interfaces":[{"interface":"back","average":{"1min":1.1080000000000001,"5min":1.0720000000000001,"15min":1.034},"min":{"1min":0.74099999999999999,"5min":0.629,"15min":0.55200000000000005},"max":{"1min":1.4099999999999999,"5min":2.3860000000000001,"15min":2.4569999999999999},"last":0.80300000000000005},{"interface":"front","average":{"1min":0.98099999999999998,"5min":1.006,"15min":0.998},"min":{"1min":0.77800000000000002,"5min":0.64100000000000001,"15min":0.56200000000000006},"max":{"1min":1.327,"5min":3.0299999999999998,"15min":3.0299999999999998},"last":0.60299999999999998}]},{"osd":6,"last update":"Mon May 22 15:41:55 2023","interfaces":[{"interface":"back","average":{"1min":1.008,"5min":1.0349999999999999,"15min":1.0389999999999999},"min":{"1min":0.72899999999999998,"5min":0.67400000000000004,"15min":0.56399999999999995},"max":{"1min":1.2250000000000001,"5min":2.883,"15min":3.488},"last":0.82999999999999996},{"interface":"front","average":{"1min":1.0820000000000001,"5min":1.1160000000000001,"15min":1.1000000000000001},"min":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.53400000000000003},"max":{"1min":1.365,"5min":3.1459999999999999,"15min":3.7330000000000001},"last":0.59199999999999997}]},{"osd":7,"last update":"Mon May 22 15:41:55 2023","interfaces":[{"interface":"back","average":{"1min":1.149,"5min":1.1499999999999999,"15min":1.123},"min":{"1min":0.77600000000000002,"5min":0.77000000000000002,"15min":0.58299999999999996},"max":{"1min":1.4970000000000001,"5min":2.5990000000000002,"15min":2.5990000000000002},"last":0.66100000000000003},{"interface":"front","average":{"1min":1.0860000000000001,"5min":1.1200000000000001,"15min":1.0980000000000001},"min":{"1min":0.65500000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":1.3839999999999999,"5min":3.8799999999999999,"15min":3.8799999999999999},"last":0.78200000000000003}]}]},{"osd":1,"up_from":13,"seq":55834575045,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":24292,"kb_used_data":1244,"kb_used_omap":0,"kb_used_meta":23040,"kb_avail":94347548,"statfs":{"total":96636764160,"available":96611889152,"internally_reserved":0,"allocated":1273856,"data_stored":661381,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":23592960},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 15:42:12 2023","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.63200000000000001,"15min":0.63800000000000001},"min":{"1min":0.37,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":1.2270000000000001,"5min":1.2270000000000001,"15min":1.7689999999999999},"last":0.495},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.78200000000000003,"15min":0.77100000000000002},"min":{"1min":0.38800000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":1.0649999999999999,"5min":1.2290000000000001,"15min":1.3799999999999999},"last":0.52900000000000003}]},{"osd":2,"last update":"Mon May 22 15:42:12 2023","interfaces":[{"interface":"back","average":{"1min":0.78700000000000003,"5min":0.76700000000000002,"15min":0.73299999999999998},"min":{"1min":0.51300000000000001,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":1.391,"5min":1.391,"15min":1.7290000000000001},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.79100000000000004,"5min":0.77200000000000002,"15min":0.75600000000000001},"min":{"1min":0.47799999999999998,"5min":0.35899999999999999,"15min":0.27800000000000002},"max":{"1min":1.2270000000000001,"5min":1.276,"15min":2.4049999999999998},"last":0.63500000000000001}]},{"osd":3,"last update":"Mon May 22 15:42:12 2023","interfaces":[{"interface":"back","average":{"1min":0.88600000000000001,"5min":0.82199999999999995,"15min":0.82899999999999996},"min":{"1min":0.41499999999999998,"5min":0.41399999999999998,"15min":0.317},"max":{"1min":1.657,"5min":1.657,"15min":1.657},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.89000000000000001,"5min":0.82399999999999995,"15min":0.82699999999999996},"min":{"1min":0.46000000000000002,"5min":0.315,"15min":0.315},"max":{"1min":2.0569999999999999,"5min":2.0569999999999999,"15min":7.5620000000000003},"last":0.58299999999999996}]},{"osd":4,"last update":"Mon May 22 15:42:12 2023","interfaces":[{"interface":"back","average":{"1min":1.002,"5min":0.96999999999999997,"15min":0.95099999999999996},"min":{"1min":0.58499999999999996,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":2.0169999999999999,"5min":2.0169999999999999,"15min":2.0169999999999999},"last":1.103},{"interface":"front","average":{"1min":1.03,"5min":0.98599999999999999,"15min":0.98899999999999999},"min":{"1min":0.64100000000000001,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":1.4650000000000001,"5min":1.5369999999999999,"15min":1.5369999999999999},"last":0.89600000000000002}]},{"osd":5,"last update":"Mon May 22 15:42:12 2023","interfaces":[{"interface":"back","average":{"1min":1.069,"5min":1.0349999999999999,"15min":1.0329999999999999},"min":{"1min":0.59899999999999998,"5min":0.443,"15min":0.443},"max":{"1min":1.9570000000000001,"5min":1.9570000000000001,"15min":1.9570000000000001},"last":0.93100000000000005},{"interface":"front","average":{"1min":1.0720000000000001,"5min":1.0509999999999999,"15min":1.048},"min":{"1min":0.68000000000000005,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":2.1200000000000001,"5min":2.1200000000000001,"15min":4.4710000000000001},"last":0.85299999999999998}]},{"osd":6,"last update":"Mon May 22 15:42:12 2023","interfaces":[{"interface":"back","average":{"1min":1.0449999999999999,"5min":1.012,"15min":1.0680000000000001},"min":{"1min":0.54000000000000004,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.5720000000000001,"5min":1.5720000000000001,"15min":14.968999999999999},"last":0.98499999999999999},{"interface":"front","average":{"1min":1.1319999999999999,"5min":1.083,"15min":1.1359999999999999},"min":{"1min":0.69799999999999995,"5min":0.61399999999999999,"15min":0.56699999999999995},"max":{"1min":1.7629999999999999,"5min":1.7629999999999999,"15min":13.242000000000001},"last":0.82499999999999996}]},{"osd":7,"last update":"Mon May 22 15:42:12 2023","interfaces":[{"interface":"back","average":{"1min":1.1559999999999999,"5min":1.1140000000000001,"15min":1.109},"min":{"1min":0.66200000000000003,"5min":0.56799999999999995,"15min":0.56799999999999995},"max":{"1min":1.865,"5min":1.865,"15min":1.9259999999999999},"last":0.77800000000000002},{"interface":"front","average":{"1min":1.1739999999999999,"5min":1.145,"15min":1.131},"min":{"1min":0.78000000000000003,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":2.2200000000000002,"5min":2.2200000000000002,"15min":5.681},"last":1.0129999999999999}]}]},{"osd":0,"up_from":13,"seq":55834575046,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":23584,"kb_used_data":1816,"kb_used_omap":0,"kb_used_meta":21760,"kb_avail":94348256,"statfs":{"total":96636764160,"available":96612614144,"internally_reserved":0,"allocated":1859584,"data_stored":1251589,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":22282240},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon May 22 15:42:07 2023","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.66100000000000003,"15min":0.66800000000000004},"min":{"1min":0.248,"5min":0.223,"15min":0.223},"max":{"1min":0.86899999999999999,"5min":1.077,"15min":1.3049999999999999},"last":0.36199999999999999},{"interface":"front","average":{"1min":0.76300000000000001,"5min":0.72999999999999998,"15min":0.73299999999999998},"min":{"1min":0.53600000000000003,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":1.0369999999999999,"5min":1.2390000000000001,"15min":1.2390000000000001},"last":0.34200000000000003}]},{"osd":2,"last update":"Mon May 22 15:42:07 2023","interfaces":[{"interface":"back","average":{"1min":0.73599999999999999,"5min":0.74199999999999999,"15min":0.74399999999999999},"min":{"1min":0.318,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":1.155,"5min":1.155,"15min":1.4059999999999999},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.79500000000000004,"5min":0.78000000000000003,"15min":0.79900000000000004},"min":{"1min":0.55400000000000005,"5min":0.32900000000000001,"15min":0.29699999999999999},"max":{"1min":1.05,"5min":1.1850000000000001,"15min":1.3999999999999999},"last":0.38500000000000001}]},{"osd":3,"last update":"Mon May 22 15:42:07 2023","interfaces":[{"interface":"back","average":{"1min":0.86699999999999999,"5min":0.755,"15min":0.76000000000000001},"min":{"1min":0.55600000000000005,"5min":0.40899999999999997,"15min":0.36599999999999999},"max":{"1min":1.9299999999999999,"5min":1.9299999999999999,"15min":1.9299999999999999},"last":0.31},{"interface":"front","average":{"1min":0.85799999999999998,"5min":0.86399999999999999,"15min":0.86799999999999999},"min":{"1min":0.35199999999999998,"5min":0.34399999999999997,"15min":0.33000000000000002},"max":{"1min":1.0669999999999999,"5min":1.3500000000000001,"15min":1.4570000000000001},"last":0.41399999999999998}]},{"osd":4,"last update":"Mon May 22 15:42:07 2023","interfaces":[{"interface":"back","average":{"1min":1.0429999999999999,"5min":0.94899999999999995,"15min":0.94799999999999995},"min":{"1min":0.751,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":2.923,"5min":2.923,"15min":2.923},"last":0.75700000000000001},{"interface":"front","average":{"1min":1.073,"5min":1.0269999999999999,"15min":1.022},"min":{"1min":0.86399999999999999,"5min":0.64600000000000002,"15min":0.54900000000000004},"max":{"1min":1.8160000000000001,"5min":1.8160000000000001,"15min":1.8160000000000001},"last":0.67400000000000004}]},{"osd":5,"last update":"Mon May 22 15:42:07 2023","interfaces":[{"interface":"back","average":{"1min":0.94699999999999995,"5min":1.014,"15min":1.038},"min":{"1min":0.66500000000000004,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.1040000000000001,"5min":1.4750000000000001,"15min":1.6850000000000001},"last":0.82899999999999996},{"interface":"front","average":{"1min":1.2170000000000001,"5min":1.032,"15min":1.01},"min":{"1min":0.76800000000000002,"5min":0.55300000000000005,"15min":0.54800000000000004},"max":{"1min":5.0519999999999996,"5min":5.0519999999999996,"15min":5.0519999999999996},"last":0.57199999999999995}]},{"osd":6,"last update":"Mon May 22 15:42:07 2023","interfaces":[{"interface":"back","average":{"1min":1.038,"5min":1.0740000000000001,"15min":1.075},"min":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":1.53,"5min":1.651,"15min":1.7450000000000001},"last":0.58699999999999997},{"interface":"front","average":{"1min":1.2749999999999999,"5min":1.139,"15min":1.103},"min":{"1min":0.91500000000000004,"5min":0.499,"15min":0.499},"max":{"1min":5.2640000000000002,"5min":5.2640000000000002,"15min":5.2640000000000002},"last":0.81000000000000005}]},{"osd":7,"last update":"Mon May 22 15:42:07 2023","interfaces":[{"interface":"back","average":{"1min":1.0700000000000001,"5min":1.1120000000000001,"15min":1.1180000000000001},"min":{"1min":0.70699999999999996,"5min":0.65700000000000003,"15min":0.56499999999999995},"max":{"1min":1.2789999999999999,"5min":1.6419999999999999,"15min":1.6419999999999999},"last":0.77500000000000002},{"interface":"front","average":{"1min":1.228,"5min":1.1000000000000001,"15min":1.087},"min":{"1min":0.85999999999999999,"5min":0.60299999999999998,"15min":0.58399999999999996},"max":{"1min":3.9060000000000001,"5min":3.9060000000000001,"15min":3.9060000000000001},"last":0.63300000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1570,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-05-22T15:42:59.956 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-05-22T15:42:59.957 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2023-05-22T15:43:00.135 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:00.131+0000 7f00664fb640 1 Processor -- start 2023-05-22T15:43:00.136 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:00.131+0000 7f00664fb640 1 -- start start 2023-05-22T15:43:00.137 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:00.131+0000 7f00664fb640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f006006b6c0 0x7f00600cb4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:00.137 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:00.131+0000 7f00664fb640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f00601596e0 0x7f006006fba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:00.137 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:00.131+0000 7f00664fb640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0060159f70 con 0x7f006006ba90 2023-05-22T15:43:00.137 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:00.131+0000 7f00664fb640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0060071760 con 0x7f006006b6c0 2023-05-22T15:43:00.137 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:00.131+0000 7f00664fb640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00600ca7d0 con 0x7f00601596e0 2023-05-22T15:43:00.139 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:00.131+0000 7f005f7fe640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f006006ba90 0x7f0060158fd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:48758/0 (socket says 172.21.15.16:48758) 2023-05-22T15:43:00.139 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:00.131+0000 7f005f7fe640 1 -- 172.21.15.16:0/3335271624 learned_addr learned my addr 172.21.15.16:0/3335271624 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:43:00.139 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:00.131+0000 7f005ffff640 1 --2- 172.21.15.16:0/3335271624 >> v2:172.21.15.62:3300/0 conn(0x7f006006b6c0 0x7f00600cb4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:00.139 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:00.131+0000 7f0064a98640 1 --2- 172.21.15.16:0/3335271624 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f00601596e0 0x7f006006fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:00.139 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:00.131+0000 7f005effd640 1 -- 172.21.15.16:0/3335271624 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 820075817 0 0) 0x7f0060159f70 con 0x7f006006ba90 2023-05-22T15:43:00.139 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:00.131+0000 7f005effd640 1 -- 172.21.15.16:0/3335271624 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f004c003530 con 0x7f006006ba90 2023-05-22T15:43:00.139 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:00.131+0000 7f005effd640 1 -- 172.21.15.16:0/3335271624 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4275823503 0 0) 0x7f004c003530 con 0x7f006006ba90 2023-05-22T15:43:00.140 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:00.131+0000 7f005effd640 1 -- 172.21.15.16:0/3335271624 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f00601596e0 msgr2=0x7f006006fba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:43:00.141 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:00.131+0000 7f005effd640 1 --2- 172.21.15.16:0/3335271624 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f00601596e0 0x7f006006fba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:00.141 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:00.131+0000 7f005effd640 1 -- 172.21.15.16:0/3335271624 >> v2:172.21.15.62:3300/0 conn(0x7f006006b6c0 msgr2=0x7f00600cb4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:43:00.141 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:00.131+0000 7f005effd640 1 --2- 172.21.15.16:0/3335271624 >> v2:172.21.15.62:3300/0 conn(0x7f006006b6c0 0x7f00600cb4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:00.141 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:00.131+0000 7f005effd640 1 -- 172.21.15.16:0/3335271624 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00600cb030 con 0x7f006006ba90 2023-05-22T15:43:00.141 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:00.131+0000 7f005effd640 1 -- 172.21.15.16:0/3335271624 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7f0050001680 con 0x7f006006ba90 2023-05-22T15:43:00.142 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:00.131+0000 7f005effd640 1 -- 172.21.15.16:0/3335271624 >> v1:172.21.15.16:6789/0 conn(0x7f006006ba90 legacy=0x7f0060158fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:00.142 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:00.131+0000 7f005effd640 1 --2- 172.21.15.16:0/3335271624 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f00601596e0 0x7f004c003980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:00.142 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:00.131+0000 7f005effd640 1 --2- 172.21.15.16:0/3335271624 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f004c003f00 0x7f004c0047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:00.142 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:00.131+0000 7f005effd640 1 --2- 172.21.15.16:0/3335271624 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f004c004d10 0x7f004c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:00.142 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:00.131+0000 7f005effd640 1 -- 172.21.15.16:0/3335271624 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0060071760 con 0x7f004c003f00 2023-05-22T15:43:00.142 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:00.131+0000 7f005effd640 1 -- 172.21.15.16:0/3335271624 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00600ca7d0 con 0x7f004c004d10 2023-05-22T15:43:00.142 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:00.131+0000 7f005effd640 1 -- 172.21.15.16:0/3335271624 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f004c00dbd0 con 0x7f00601596e0 2023-05-22T15:43:00.142 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:00.131+0000 7f005f7fe640 1 --2- 172.21.15.16:0/3335271624 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f004c003f00 0x7f004c0047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:00.143 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:00.135+0000 7f005ffff640 1 --2- 172.21.15.16:0/3335271624 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f00601596e0 0x7f004c003980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:00.143 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:00.135+0000 7f0064a98640 1 --2- 172.21.15.16:0/3335271624 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f004c004d10 0x7f004c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:00.143 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:00.135+0000 7f005f7fe640 1 -- 172.21.15.16:0/3335271624 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f00601596e0 msgr2=0x7f004c003980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:43:00.143 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:00.135+0000 7f005f7fe640 1 --2- 172.21.15.16:0/3335271624 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f00601596e0 0x7f004c003980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:00.143 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:00.135+0000 7f005f7fe640 1 -- 172.21.15.16:0/3335271624 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f004c004d10 msgr2=0x7f004c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:43:00.143 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:00.135+0000 7f005f7fe640 1 --2- 172.21.15.16:0/3335271624 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f004c004d10 0x7f004c005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:00.143 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:00.135+0000 7f005f7fe640 1 -- 172.21.15.16:0/3335271624 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f004c003530 con 0x7f004c003f00 2023-05-22T15:43:00.146 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:00.135+0000 7f005f7fe640 1 --2- 172.21.15.16:0/3335271624 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f004c003f00 0x7f004c0047d0 secure :-1 s=READY pgs=1294 cs=0 l=1 rev1=1 crypto rx=0x7f005000c210 tx=0x7f005001cbd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1a53205025c5b91 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:43:00.146 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:00.135+0000 7f005effd640 1 -- 172.21.15.16:0/3335271624 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00500a5910 con 0x7f004c003f00 2023-05-22T15:43:00.146 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:00.135+0000 7f005effd640 1 -- 172.21.15.16:0/3335271624 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00500a5a70 con 0x7f004c003f00 2023-05-22T15:43:00.146 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:00.135+0000 7f005effd640 1 -- 172.21.15.16:0/3335271624 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00500a5ea0 con 0x7f004c003f00 2023-05-22T15:43:00.146 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:00.135+0000 7f00664fb640 1 -- 172.21.15.16:0/3335271624 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f004c003f00 msgr2=0x7f004c0047d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:00.147 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:00.135+0000 7f00664fb640 1 --2- 172.21.15.16:0/3335271624 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f004c003f00 0x7f004c0047d0 secure :-1 s=READY pgs=1294 cs=0 l=1 rev1=1 crypto rx=0x7f005000c210 tx=0x7f005001cbd0 comp rx=0 tx=0).stop 2023-05-22T15:43:00.147 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:00.135+0000 7f0064a98640 1 -- 172.21.15.16:0/3335271624 reap_dead start 2023-05-22T15:43:00.147 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:00.135+0000 7f00664fb640 1 -- 172.21.15.16:0/3335271624 shutdown_connections 2023-05-22T15:43:00.147 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:00.135+0000 7f00664fb640 1 -- 172.21.15.16:0/3335271624 >> 172.21.15.16:0/3335271624 conn(0x7f0060069720 msgr2=0x7f0060069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:43:00.147 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:00.135+0000 7f00664fb640 1 -- 172.21.15.16:0/3335271624 shutdown_connections 2023-05-22T15:43:00.147 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:00.135+0000 7f00664fb640 1 -- 172.21.15.16:0/3335271624 wait complete. 2023-05-22T15:43:00.147 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:00.139+0000 7f00664fb640 1 Processor -- start 2023-05-22T15:43:00.147 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:00.139+0000 7f00664fb640 1 -- start start 2023-05-22T15:43:00.148 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:00.139+0000 7f00664fb640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f004c003f00 0x7f00600c6000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:00.148 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:00.139+0000 7f00664fb640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f00600b0020 0x7f00601554a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:00.148 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:00.139+0000 7f00664fb640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f00600c3aa0 0x7f00600c66e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:00.148 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:00.139+0000 7f00664fb640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00600c4410 con 0x7f004c003f00 2023-05-22T15:43:00.148 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:00.139+0000 7f00664fb640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00601559e0 con 0x7f00600b0020 2023-05-22T15:43:00.148 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:00.139+0000 7f00664fb640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00600c5280 con 0x7f00600c3aa0 2023-05-22T15:43:00.148 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:00.139+0000 7f005ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f004c003f00 0x7f00600c6000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:00.149 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:00.139+0000 7f005ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f004c003f00 0x7f00600c6000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49800/0 (socket says 172.21.15.16:49800) 2023-05-22T15:43:00.149 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:00.139+0000 7f005ffff640 1 -- 172.21.15.16:0/2181046977 learned_addr learned my addr 172.21.15.16:0/2181046977 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:43:00.149 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:00.139+0000 7f005f7fe640 1 --2- 172.21.15.16:0/2181046977 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f00600b0020 0x7f00601554a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:00.149 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:00.139+0000 7f0064a98640 1 --2- 172.21.15.16:0/2181046977 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f00600c3aa0 0x7f00600c66e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:00.149 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:00.139+0000 7f005ffff640 1 -- 172.21.15.16:0/2181046977 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f00600c3aa0 msgr2=0x7f00600c66e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:00.149 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:00.139+0000 7f005ffff640 1 --2- 172.21.15.16:0/2181046977 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f00600c3aa0 0x7f00600c66e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:00.149 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:00.139+0000 7f005ffff640 1 -- 172.21.15.16:0/2181046977 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f00600b0020 msgr2=0x7f00601554a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:00.150 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:00.139+0000 7f005ffff640 1 --2- 172.21.15.16:0/2181046977 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f00600b0020 0x7f00601554a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:00.150 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:00.139+0000 7f005ffff640 1 -- 172.21.15.16:0/2181046977 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f006006bea0 con 0x7f004c003f00 2023-05-22T15:43:00.150 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:00.139+0000 7f0064a98640 1 --2- 172.21.15.16:0/2181046977 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f00600c3aa0 0x7f00600c66e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T15:43:00.150 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:00.139+0000 7f005f7fe640 1 --2- 172.21.15.16:0/2181046977 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f00600b0020 0x7f00601554a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T15:43:00.150 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:00.139+0000 7f005ffff640 1 --2- 172.21.15.16:0/2181046977 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f004c003f00 0x7f00600c6000 secure :-1 s=READY pgs=1295 cs=0 l=1 rev1=1 crypto rx=0x7f00480115e0 tx=0x7f0048011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:43:00.150 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:00.139+0000 7f005cff9640 1 -- 172.21.15.16:0/2181046977 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f004800f8f0 con 0x7f004c003f00 2023-05-22T15:43:00.150 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:00.139+0000 7f005cff9640 1 -- 172.21.15.16:0/2181046977 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f004800fa50 con 0x7f004c003f00 2023-05-22T15:43:00.150 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:00.139+0000 7f00664fb640 1 -- 172.21.15.16:0/2181046977 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00600cacc0 con 0x7f004c003f00 2023-05-22T15:43:00.151 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:00.139+0000 7f005cff9640 1 -- 172.21.15.16:0/2181046977 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f004800fdc0 con 0x7f004c003f00 2023-05-22T15:43:00.151 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:00.139+0000 7f00664fb640 1 -- 172.21.15.16:0/2181046977 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f006006c030 con 0x7f004c003f00 2023-05-22T15:43:00.151 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:00.139+0000 7f00664fb640 1 -- 172.21.15.16:0/2181046977 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f006004dbe0 con 0x7f004c003f00 2023-05-22T15:43:00.151 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:00.139+0000 7f005cff9640 1 -- 172.21.15.16:0/2181046977 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 34) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f0048015460 con 0x7f004c003f00 2023-05-22T15:43:00.151 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:00.139+0000 7f005cff9640 1 --2- 172.21.15.16:0/2181046977 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f00400654a0 0x7f0040067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:00.151 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:00.143+0000 7f005cff9640 1 -- 172.21.15.16:0/2181046977 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f0048083440 con 0x7f004c003f00 2023-05-22T15:43:00.151 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:00.143+0000 7f005f7fe640 1 --2- 172.21.15.16:0/2181046977 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f00400654a0 0x7f0040067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:00.152 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:00.143+0000 7f005f7fe640 1 --2- 172.21.15.16:0/2181046977 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f00400654a0 0x7f0040067950 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f00500aa000 tx=0x7f0050001a10 comp rx=0 tx=0).ready entity=mgr.6436 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:43:00.152 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:00.147+0000 7f005cff9640 1 -- 172.21.15.16:0/2181046977 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f0048050e00 con 0x7f004c003f00 2023-05-22T15:43:00.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:00.351+0000 7f00664fb640 1 -- 172.21.15.16:0/2181046977 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f0060070660 con 0x7f004c003f00 2023-05-22T15:43:00.364 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:00.359+0000 7f005cff9640 1 -- 172.21.15.16:0/2181046977 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v21) v1 ==== 190+0+0 (secure 0 0 0) 0x7f0048054420 con 0x7f004c003f00 2023-05-22T15:43:00.365 INFO:teuthology.orchestra.run.smithi016.stderr:updated caps for client.0 2023-05-22T15:43:00.366 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:00.363+0000 7f00664fb640 1 -- 172.21.15.16:0/2181046977 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f00400654a0 msgr2=0x7f0040067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:00.367 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:00.363+0000 7f00664fb640 1 --2- 172.21.15.16:0/2181046977 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f00400654a0 0x7f0040067950 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f00500aa000 tx=0x7f0050001a10 comp rx=0 tx=0).stop 2023-05-22T15:43:00.367 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:00.363+0000 7f00664fb640 1 -- 172.21.15.16:0/2181046977 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f004c003f00 msgr2=0x7f00600c6000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:00.367 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:00.363+0000 7f00664fb640 1 --2- 172.21.15.16:0/2181046977 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f004c003f00 0x7f00600c6000 secure :-1 s=READY pgs=1295 cs=0 l=1 rev1=1 crypto rx=0x7f00480115e0 tx=0x7f0048011aa0 comp rx=0 tx=0).stop 2023-05-22T15:43:00.367 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:00.363+0000 7f00664fb640 1 -- 172.21.15.16:0/2181046977 shutdown_connections 2023-05-22T15:43:00.367 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:00.363+0000 7f00664fb640 1 --2- 172.21.15.16:0/2181046977 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f00400654a0 0x7f0040067950 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:00.367 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:00.363+0000 7f00664fb640 1 --2- 172.21.15.16:0/2181046977 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f00600c3aa0 0x7f00600c66e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:00.368 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:00.363+0000 7f00664fb640 1 --2- 172.21.15.16:0/2181046977 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f00600b0020 0x7f00601554a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:00.368 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:00.363+0000 7f00664fb640 1 --2- 172.21.15.16:0/2181046977 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f004c003f00 0x7f00600c6000 unknown :-1 s=CLOSED pgs=1295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:00.368 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:00.363+0000 7f00664fb640 1 -- 172.21.15.16:0/2181046977 >> 172.21.15.16:0/2181046977 conn(0x7f0060069720 msgr2=0x7f006015a850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:43:00.368 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:00.363+0000 7f00664fb640 1 -- 172.21.15.16:0/2181046977 shutdown_connections 2023-05-22T15:43:00.368 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:00.363+0000 7f00664fb640 1 -- 172.21.15.16:0/2181046977 wait complete. 2023-05-22T15:43:00.383 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-05-22T15:43:00.510 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:00.507+0000 7f81aed60640 1 Processor -- start 2023-05-22T15:43:00.510 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:00.507+0000 7f81aed60640 1 -- start start 2023-05-22T15:43:00.511 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:00.507+0000 7f81aed60640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f81a80b0250 0x7f81a8159330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:00.511 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:00.507+0000 7f81aed60640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f81a8159870 0x7f81a80c8ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:00.511 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:00.507+0000 7f81aed60640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81a815a0a0 con 0x7f81a81531b0 2023-05-22T15:43:00.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:00.507+0000 7f81aed60640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81a8153580 con 0x7f81a8159870 2023-05-22T15:43:00.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:00.507+0000 7f81aed60640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81a815d5e0 con 0x7f81a80b0250 2023-05-22T15:43:00.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:00.507+0000 7f81ad2fd640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f81a8159870 0x7f81a80c8ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:00.513 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:00.507+0000 7f81ad2fd640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f81a8159870 0x7f81a80c8ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.16:33130/0 (socket says 172.21.15.16:33130) 2023-05-22T15:43:00.513 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:00.507+0000 7f81ad2fd640 1 -- 172.21.15.16:0/2770542935 learned_addr learned my addr 172.21.15.16:0/2770542935 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:43:00.513 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:00.507+0000 7f81acafc640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f81a81531b0 0x7f81a80afb40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.16:58960/0 (socket says 172.21.15.16:58960) 2023-05-22T15:43:00.513 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:00.507+0000 7f81a7fff640 1 --2- 172.21.15.16:0/2770542935 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f81a80b0250 0x7f81a8159330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:00.513 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:00.507+0000 7f81ad2fd640 1 -- 172.21.15.16:0/2770542935 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f81a80b0250 msgr2=0x7f81a8159330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:43:00.514 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:00.507+0000 7f81ad2fd640 1 --2- 172.21.15.16:0/2770542935 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f81a80b0250 0x7f81a8159330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:00.514 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:00.507+0000 7f81ad2fd640 1 -- 172.21.15.16:0/2770542935 >> v1:172.21.15.62:6789/0 conn(0x7f81a81531b0 legacy=0x7f81a80afb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:00.514 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:00.507+0000 7f81ad2fd640 1 -- 172.21.15.16:0/2770542935 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81a815c820 con 0x7f81a8159870 2023-05-22T15:43:00.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:00.511+0000 7f81ad2fd640 1 --2- 172.21.15.16:0/2770542935 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f81a8159870 0x7f81a80c8ba0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f81a0009e50 tx=0x7f81a001c9e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6f9b6dcdcc1fe246 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:43:00.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:00.511+0000 7f81a77fe640 1 -- 172.21.15.16:0/2770542935 <== mon.1 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81a00a58a0 con 0x7f81a8159870 2023-05-22T15:43:00.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:00.511+0000 7f81a77fe640 1 -- 172.21.15.16:0/2770542935 <== mon.1 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81a00a5a00 con 0x7f81a8159870 2023-05-22T15:43:00.516 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:00.511+0000 7f81a77fe640 1 -- 172.21.15.16:0/2770542935 <== mon.1 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81a00a5d90 con 0x7f81a8159870 2023-05-22T15:43:00.516 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:00.511+0000 7f81aed60640 1 -- 172.21.15.16:0/2770542935 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f81a8159870 msgr2=0x7f81a80c8ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:00.516 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:00.511+0000 7f81aed60640 1 --2- 172.21.15.16:0/2770542935 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f81a8159870 0x7f81a80c8ba0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f81a0009e50 tx=0x7f81a001c9e0 comp rx=0 tx=0).stop 2023-05-22T15:43:00.517 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:00.511+0000 7f81aed60640 1 -- 172.21.15.16:0/2770542935 shutdown_connections 2023-05-22T15:43:00.517 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:00.511+0000 7f81aed60640 1 --2- 172.21.15.16:0/2770542935 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f81a8159870 0x7f81a80c8ba0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:00.517 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:00.511+0000 7f81aed60640 1 --2- 172.21.15.16:0/2770542935 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f81a80b0250 0x7f81a8159330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:00.517 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:00.511+0000 7f81aed60640 1 -- 172.21.15.16:0/2770542935 >> 172.21.15.16:0/2770542935 conn(0x7f81a8069720 msgr2=0x7f81a8069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:43:00.517 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:00.511+0000 7f81aed60640 1 -- 172.21.15.16:0/2770542935 shutdown_connections 2023-05-22T15:43:00.517 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:00.511+0000 7f81aed60640 1 -- 172.21.15.16:0/2770542935 wait complete. 2023-05-22T15:43:00.518 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:00.511+0000 7f81aed60640 1 Processor -- start 2023-05-22T15:43:00.518 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:00.511+0000 7f81aed60640 1 -- start start 2023-05-22T15:43:00.519 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:00.515+0000 7f81aed60640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f81a80b0250 0x7f81a80b38e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:00.519 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:00.515+0000 7f81aed60640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f81a81531b0 0x7f81a80b2370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:00.519 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:00.515+0000 7f81aed60640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f81a8159870 0x7f81a8151af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:00.519 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:00.515+0000 7f81aed60640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81a814a630 con 0x7f81a81531b0 2023-05-22T15:43:00.519 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:00.515+0000 7f81aed60640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81a80b1f80 con 0x7f81a80b0250 2023-05-22T15:43:00.519 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:00.515+0000 7f81aed60640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81a80b3450 con 0x7f81a8159870 2023-05-22T15:43:00.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:00.515+0000 7f81a7fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f81a81531b0 0x7f81a80b2370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:00.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:00.515+0000 7f81a7fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f81a81531b0 0x7f81a80b2370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49814/0 (socket says 172.21.15.16:49814) 2023-05-22T15:43:00.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:00.515+0000 7f81a7fff640 1 -- 172.21.15.16:0/3361542502 learned_addr learned my addr 172.21.15.16:0/3361542502 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:43:00.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:00.515+0000 7f81ad2fd640 1 --2- 172.21.15.16:0/3361542502 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f81a8159870 0x7f81a8151af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:00.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:00.515+0000 7f81acafc640 1 --2- 172.21.15.16:0/3361542502 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f81a80b0250 0x7f81a80b38e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:00.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:00.515+0000 7f81a7fff640 1 -- 172.21.15.16:0/3361542502 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f81a8159870 msgr2=0x7f81a8151af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:00.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:00.515+0000 7f81a7fff640 1 --2- 172.21.15.16:0/3361542502 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f81a8159870 0x7f81a8151af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:00.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:00.515+0000 7f81a7fff640 1 -- 172.21.15.16:0/3361542502 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f81a80b0250 msgr2=0x7f81a80b38e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:00.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:00.515+0000 7f81a7fff640 1 --2- 172.21.15.16:0/3361542502 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f81a80b0250 0x7f81a80b38e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:00.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:00.515+0000 7f81a7fff640 1 -- 172.21.15.16:0/3361542502 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81a0009b30 con 0x7f81a81531b0 2023-05-22T15:43:00.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:00.515+0000 7f81ad2fd640 1 --2- 172.21.15.16:0/3361542502 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f81a8159870 0x7f81a8151af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T15:43:00.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:00.515+0000 7f81a7fff640 1 --2- 172.21.15.16:0/3361542502 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f81a81531b0 0x7f81a80b2370 secure :-1 s=READY pgs=1296 cs=0 l=1 rev1=1 crypto rx=0x7f819400e8a0 tx=0x7f819400ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:43:00.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:00.515+0000 7f81a57fa640 1 -- 172.21.15.16:0/3361542502 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f819400ee90 con 0x7f81a81531b0 2023-05-22T15:43:00.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:00.515+0000 7f81a57fa640 1 -- 172.21.15.16:0/3361542502 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f819400c940 con 0x7f81a81531b0 2023-05-22T15:43:00.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:00.515+0000 7f81aed60640 1 -- 172.21.15.16:0/3361542502 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81a8159c80 con 0x7f81a81531b0 2023-05-22T15:43:00.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:00.515+0000 7f81a57fa640 1 -- 172.21.15.16:0/3361542502 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f819400ccb0 con 0x7f81a81531b0 2023-05-22T15:43:00.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:00.515+0000 7f81aed60640 1 -- 172.21.15.16:0/3361542502 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81a815d0c0 con 0x7f81a81531b0 2023-05-22T15:43:00.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:00.515+0000 7f81aed60640 1 -- 172.21.15.16:0/3361542502 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81a8159e10 con 0x7f81a81531b0 2023-05-22T15:43:00.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:00.515+0000 7f81a57fa640 1 -- 172.21.15.16:0/3361542502 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 34) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f8194013470 con 0x7f81a81531b0 2023-05-22T15:43:00.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:00.515+0000 7f81a57fa640 1 --2- 172.21.15.16:0/3361542502 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f818c0654a0 0x7f818c067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:00.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:00.515+0000 7f81a57fa640 1 -- 172.21.15.16:0/3361542502 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f81940829d0 con 0x7f81a81531b0 2023-05-22T15:43:00.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:00.515+0000 7f81acafc640 1 --2- 172.21.15.16:0/3361542502 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f818c0654a0 0x7f818c067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:00.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:00.519+0000 7f81acafc640 1 --2- 172.21.15.16:0/3361542502 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f818c0654a0 0x7f818c067950 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f819c000f70 tx=0x7f819c007e50 comp rx=0 tx=0).ready entity=mgr.6436 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:43:00.526 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:00.519+0000 7f81a57fa640 1 -- 172.21.15.16:0/3361542502 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f81940504e0 con 0x7f81a81531b0 2023-05-22T15:43:00.714 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:00.707+0000 7f81aed60640 1 -- 172.21.15.16:0/3361542502 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f81a80b3e20 con 0x7f81a81531b0 2023-05-22T15:43:00.715 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:00.711+0000 7f81a57fa640 1 -- 172.21.15.16:0/3361542502 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) v1 ==== 94+0+4650 (secure 0 0 0) 0x7f8194053b00 con 0x7f81a81531b0 2023-05-22T15:43:00.715 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:43:00.715 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":36,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6059,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6833/3017549713","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":3017549713},{"type":"v1","addr":"172.21.15.62:6833","nonce":3017549713}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":33},{"gid":6607,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/3268651705","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":3268651705},{"type":"v1","addr":"172.21.15.62:6835","nonce":3268651705}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":33},{"gid":7524,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6836/2104991509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2104991509},{"type":"v1","addr":"172.21.15.16:6836","nonce":2104991509}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":33}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-05-22T15:42:56.565074+0000","modified":"2023-05-22T15:42:58.580533+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7527},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7527":{"gid":7527,"name":"a","rank":0,"incarnation":34,"state":"up:active","state_seq":3,"addr":"172.21.15.16:6837/1353635569","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6835","nonce":1353635569},{"type":"v1","addr":"172.21.15.16:6837","nonce":1353635569}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2023-05-22T15:43:00.716 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 36 2023-05-22T15:43:00.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:00.711+0000 7f81aed60640 1 -- 172.21.15.16:0/3361542502 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f818c0654a0 msgr2=0x7f818c067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:00.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:00.711+0000 7f81aed60640 1 --2- 172.21.15.16:0/3361542502 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f818c0654a0 0x7f818c067950 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f819c000f70 tx=0x7f819c007e50 comp rx=0 tx=0).stop 2023-05-22T15:43:00.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:00.711+0000 7f81aed60640 1 -- 172.21.15.16:0/3361542502 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f81a81531b0 msgr2=0x7f81a80b2370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:00.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:00.711+0000 7f81aed60640 1 --2- 172.21.15.16:0/3361542502 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f81a81531b0 0x7f81a80b2370 secure :-1 s=READY pgs=1296 cs=0 l=1 rev1=1 crypto rx=0x7f819400e8a0 tx=0x7f819400ed60 comp rx=0 tx=0).stop 2023-05-22T15:43:00.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:00.711+0000 7f81aed60640 1 -- 172.21.15.16:0/3361542502 shutdown_connections 2023-05-22T15:43:00.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:00.711+0000 7f81aed60640 1 --2- 172.21.15.16:0/3361542502 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f818c0654a0 0x7f818c067950 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:00.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:00.711+0000 7f81aed60640 1 --2- 172.21.15.16:0/3361542502 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f81a8159870 0x7f81a8151af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:00.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:00.711+0000 7f81aed60640 1 --2- 172.21.15.16:0/3361542502 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f81a81531b0 0x7f81a80b2370 unknown :-1 s=CLOSED pgs=1296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:00.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:00.711+0000 7f81aed60640 1 --2- 172.21.15.16:0/3361542502 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f81a80b0250 0x7f81a80b38e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:00.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:00.711+0000 7f81aed60640 1 -- 172.21.15.16:0/3361542502 >> 172.21.15.16:0/3361542502 conn(0x7f81a8069720 msgr2=0x7f81a80700a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:43:00.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:00.711+0000 7f81aed60640 1 -- 172.21.15.16:0/3361542502 shutdown_connections 2023-05-22T15:43:00.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:00.711+0000 7f81aed60640 1 -- 172.21.15.16:0/3361542502 wait complete. 2023-05-22T15:43:01.137 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 36, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-05-22T15:42:56.565074+0000', 'modified': '2023-05-22T15:42:58.580533+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7527}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7527': {'gid': 7527, 'name': 'a', 'rank': 0, 'incarnation': 34, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.16:6837/1353635569', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6835', 'nonce': 1353635569}, {'type': 'v1', 'addr': '172.21.15.16:6837', 'nonce': 1353635569}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [13], 'metadata_pool': 12, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-05-22T15:43:01.137 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-05-22T15:43:01.138 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2023-05-22T15:43:01.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.271+0000 7f9fac0f2640 1 Processor -- start 2023-05-22T15:43:01.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.271+0000 7f9fac0f2640 1 -- start start 2023-05-22T15:43:01.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.271+0000 7f9fac0f2640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9fa40c9b40 0x7f9fa415e460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:01.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.271+0000 7f9fac0f2640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f9fa4070df0 0x7f9fa4071210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:01.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.271+0000 7f9fac0f2640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fa40b0650 con 0x7f9fa40ca4e0 2023-05-22T15:43:01.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.271+0000 7f9fac0f2640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9fa40b0370 con 0x7f9fa4070df0 2023-05-22T15:43:01.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.271+0000 7f9fac0f2640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9fa415d980 con 0x7f9fa40c9b40 2023-05-22T15:43:01.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.271+0000 7f9fa9e8e640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f9fa40ca4e0 0x7f9fa40afb40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:48774/0 (socket says 172.21.15.16:48774) 2023-05-22T15:43:01.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.271+0000 7f9fa9e8e640 1 -- 172.21.15.16:0/788743539 learned_addr learned my addr 172.21.15.16:0/788743539 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:43:01.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.271+0000 7f9faa68f640 1 --2- 172.21.15.16:0/788743539 >> v2:172.21.15.16:3300/0 conn(0x7f9fa4070df0 0x7f9fa4071210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:01.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.271+0000 7f9fa968d640 1 --2- 172.21.15.16:0/788743539 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9fa40c9b40 0x7f9fa415e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:01.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.271+0000 7f9fa8e8c640 1 -- 172.21.15.16:0/788743539 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3662024414 0 0) 0x7f9fa40b0650 con 0x7f9fa40ca4e0 2023-05-22T15:43:01.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.271+0000 7f9fa8e8c640 1 -- 172.21.15.16:0/788743539 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f90003530 con 0x7f9fa40ca4e0 2023-05-22T15:43:01.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.271+0000 7f9faa68f640 1 -- 172.21.15.16:0/788743539 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9fa40c9b40 msgr2=0x7f9fa415e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:43:01.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.271+0000 7f9faa68f640 1 --2- 172.21.15.16:0/788743539 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9fa40c9b40 0x7f9fa415e460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:01.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.271+0000 7f9faa68f640 1 -- 172.21.15.16:0/788743539 >> v1:172.21.15.16:6789/0 conn(0x7f9fa40ca4e0 legacy=0x7f9fa40afb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:01.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.271+0000 7f9faa68f640 1 -- 172.21.15.16:0/788743539 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fa415dc60 con 0x7f9fa4070df0 2023-05-22T15:43:01.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.275+0000 7f9faa68f640 1 --2- 172.21.15.16:0/788743539 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9fa4070df0 0x7f9fa4071210 secure :-1 s=READY pgs=1298 cs=0 l=1 rev1=1 crypto rx=0x7f9f9c0097c0 tx=0x7f9f9c01c3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4084bc5092bce98c server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:43:01.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.275+0000 7f9fa8e8c640 1 -- 172.21.15.16:0/788743539 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f9c0ab600 con 0x7f9fa4070df0 2023-05-22T15:43:01.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.275+0000 7f9fa8e8c640 1 -- 172.21.15.16:0/788743539 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f9c0a7050 con 0x7f9fa4070df0 2023-05-22T15:43:01.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.275+0000 7f9fa8e8c640 1 -- 172.21.15.16:0/788743539 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f9c0ab960 con 0x7f9fa4070df0 2023-05-22T15:43:01.282 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.275+0000 7f9fac0f2640 1 -- 172.21.15.16:0/788743539 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9fa4070df0 msgr2=0x7f9fa4071210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:01.282 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.275+0000 7f9fac0f2640 1 --2- 172.21.15.16:0/788743539 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9fa4070df0 0x7f9fa4071210 secure :-1 s=READY pgs=1298 cs=0 l=1 rev1=1 crypto rx=0x7f9f9c0097c0 tx=0x7f9f9c01c3c0 comp rx=0 tx=0).stop 2023-05-22T15:43:01.282 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.275+0000 7f9fac0f2640 1 -- 172.21.15.16:0/788743539 shutdown_connections 2023-05-22T15:43:01.282 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.275+0000 7f9fac0f2640 1 --2- 172.21.15.16:0/788743539 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9fa4070df0 0x7f9fa4071210 unknown :-1 s=CLOSED pgs=1298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:01.282 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.275+0000 7f9fac0f2640 1 --2- 172.21.15.16:0/788743539 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9fa40c9b40 0x7f9fa415e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:01.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.275+0000 7f9fac0f2640 1 -- 172.21.15.16:0/788743539 >> 172.21.15.16:0/788743539 conn(0x7f9fa4069720 msgr2=0x7f9fa4069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:43:01.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.275+0000 7f9fac0f2640 1 -- 172.21.15.16:0/788743539 shutdown_connections 2023-05-22T15:43:01.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.275+0000 7f9fac0f2640 1 -- 172.21.15.16:0/788743539 wait complete. 2023-05-22T15:43:01.285 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.279+0000 7f9fac0f2640 1 Processor -- start 2023-05-22T15:43:01.285 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.279+0000 7f9fac0f2640 1 -- start start 2023-05-22T15:43:01.285 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.279+0000 7f9fac0f2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9fa4070df0 0x7f9fa40c7b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:01.285 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.279+0000 7f9fac0f2640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9fa40c9b40 0x7f9fa415ae50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:01.285 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.279+0000 7f9fac0f2640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9fa40ca4e0 0x7f9fa4150140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:01.285 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.279+0000 7f9fac0f2640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fa40c7690 con 0x7f9fa4070df0 2023-05-22T15:43:01.285 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.279+0000 7f9fac0f2640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fa40c6a80 con 0x7f9fa40c9b40 2023-05-22T15:43:01.286 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.279+0000 7f9fac0f2640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9fa40c6d60 con 0x7f9fa40ca4e0 2023-05-22T15:43:01.286 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.279+0000 7f9fa9e8e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9fa4070df0 0x7f9fa40c7b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:01.286 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.279+0000 7f9fa9e8e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9fa4070df0 0x7f9fa40c7b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49832/0 (socket says 172.21.15.16:49832) 2023-05-22T15:43:01.286 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.279+0000 7f9fa9e8e640 1 -- 172.21.15.16:0/502925819 learned_addr learned my addr 172.21.15.16:0/502925819 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:43:01.286 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.279+0000 7f9fa968d640 1 --2- 172.21.15.16:0/502925819 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9fa40c9b40 0x7f9fa415ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:01.286 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.279+0000 7f9faa68f640 1 --2- 172.21.15.16:0/502925819 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9fa40ca4e0 0x7f9fa4150140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:01.286 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.279+0000 7f9fa9e8e640 1 -- 172.21.15.16:0/502925819 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9fa40ca4e0 msgr2=0x7f9fa4150140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:01.287 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.279+0000 7f9fa9e8e640 1 --2- 172.21.15.16:0/502925819 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9fa40ca4e0 0x7f9fa4150140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:01.287 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.279+0000 7f9fa9e8e640 1 -- 172.21.15.16:0/502925819 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9fa40c9b40 msgr2=0x7f9fa415ae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:01.287 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.279+0000 7f9fa9e8e640 1 --2- 172.21.15.16:0/502925819 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9fa40c9b40 0x7f9fa415ae50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:01.287 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.279+0000 7f9fa9e8e640 1 -- 172.21.15.16:0/502925819 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f9c009420 con 0x7f9fa4070df0 2023-05-22T15:43:01.287 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.279+0000 7f9fa968d640 1 --2- 172.21.15.16:0/502925819 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9fa40c9b40 0x7f9fa415ae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:43:01.288 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.279+0000 7f9fa9e8e640 1 --2- 172.21.15.16:0/502925819 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9fa4070df0 0x7f9fa40c7b20 secure :-1 s=READY pgs=1299 cs=0 l=1 rev1=1 crypto rx=0x7f9fa000c3c0 tx=0x7f9fa000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:43:01.288 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.279+0000 7f9f967fc640 1 -- 172.21.15.16:0/502925819 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fa0005cf0 con 0x7f9fa4070df0 2023-05-22T15:43:01.288 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.279+0000 7f9f967fc640 1 -- 172.21.15.16:0/502925819 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9fa0005e50 con 0x7f9fa4070df0 2023-05-22T15:43:01.288 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.279+0000 7f9fac0f2640 1 -- 172.21.15.16:0/502925819 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9fa40c8e80 con 0x7f9fa4070df0 2023-05-22T15:43:01.288 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.279+0000 7f9f967fc640 1 -- 172.21.15.16:0/502925819 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fa00072f0 con 0x7f9fa4070df0 2023-05-22T15:43:01.289 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.279+0000 7f9fac0f2640 1 -- 172.21.15.16:0/502925819 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9fa40ca8b0 con 0x7f9fa4070df0 2023-05-22T15:43:01.291 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.283+0000 7f9f967fc640 1 -- 172.21.15.16:0/502925819 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 34) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f9fa0007450 con 0x7f9fa4070df0 2023-05-22T15:43:01.291 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.283+0000 7f9f967fc640 1 --2- 172.21.15.16:0/502925819 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f9f880654a0 0x7f9f88067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:01.291 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.283+0000 7f9f967fc640 1 -- 172.21.15.16:0/502925819 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f9fa0082130 con 0x7f9fa4070df0 2023-05-22T15:43:01.291 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.283+0000 7f9fa968d640 1 --2- 172.21.15.16:0/502925819 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f9f880654a0 0x7f9f88067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:01.291 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.283+0000 7f9fa968d640 1 --2- 172.21.15.16:0/502925819 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f9f880654a0 0x7f9f88067950 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f9fa40b0220 tx=0x7f9f9800b040 comp rx=0 tx=0).ready entity=mgr.6436 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:43:01.292 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.283+0000 7f9faf214000 1 Processor -- start 2023-05-22T15:43:01.292 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.283+0000 7f9faf214000 1 -- start start 2023-05-22T15:43:01.292 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.283+0000 7f9faf214000 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b510922fd0 0x55b510b95c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:01.292 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.283+0000 7f9faf214000 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x55b510b96170 0x55b510b96540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:01.292 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.283+0000 7f9faf214000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b510b49a10 0x55b510b49de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:01.292 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.283+0000 7f9faf214000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b510a13bd0 con 0x55b510b49a10 2023-05-22T15:43:01.292 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.283+0000 7f9faf214000 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b510930dd0 con 0x55b510922fd0 2023-05-22T15:43:01.293 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.283+0000 7f9faf214000 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b5109a1640 con 0x55b510b96170 2023-05-22T15:43:01.293 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.283+0000 7f9faa68f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b510b49a10 0x55b510b49de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:01.293 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.283+0000 7f9faa68f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b510b49a10 0x55b510b49de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49838/0 (socket says 172.21.15.16:49838) 2023-05-22T15:43:01.293 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.283+0000 7f9faa68f640 1 -- 172.21.15.16:0/3003742078 learned_addr learned my addr 172.21.15.16:0/3003742078 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:43:01.293 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.283+0000 7f9fa968d640 1 --2- 172.21.15.16:0/3003742078 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x55b510b96170 0x55b510b96540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:01.293 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.283+0000 7f9fa9e8e640 1 --2- 172.21.15.16:0/3003742078 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b510922fd0 0x55b510b95c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:01.293 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.283+0000 7f9faa68f640 1 -- 172.21.15.16:0/3003742078 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x55b510b96170 msgr2=0x55b510b96540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:43:01.293 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.283+0000 7f9faa68f640 1 --2- 172.21.15.16:0/3003742078 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x55b510b96170 0x55b510b96540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:01.294 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.283+0000 7f9faa68f640 1 -- 172.21.15.16:0/3003742078 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b510922fd0 msgr2=0x55b510b95c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:43:01.294 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.283+0000 7f9faa68f640 1 --2- 172.21.15.16:0/3003742078 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b510922fd0 0x55b510b95c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:01.294 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.283+0000 7f9faa68f640 1 -- 172.21.15.16:0/3003742078 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b5109600b0 con 0x55b510b49a10 2023-05-22T15:43:01.294 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.283+0000 7f9faa68f640 1 --2- 172.21.15.16:0/3003742078 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b510b49a10 0x55b510b49de0 secure :-1 s=READY pgs=1300 cs=0 l=1 rev1=1 crypto rx=0x7f9f9c01c400 tx=0x7f9f9c004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6eae15eb13bdd58 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:43:01.294 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.283+0000 7f9f877fe640 1 -- 172.21.15.16:0/3003742078 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f9c004340 con 0x55b510b49a10 2023-05-22T15:43:01.295 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.283+0000 7f9f877fe640 1 -- 172.21.15.16:0/3003742078 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f9c0044a0 con 0x55b510b49a10 2023-05-22T15:43:01.295 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.283+0000 7f9f877fe640 1 -- 172.21.15.16:0/3003742078 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f9c003840 con 0x55b510b49a10 2023-05-22T15:43:01.295 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.287+0000 7f9faf214000 1 -- 172.21.15.16:0/3003742078 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b510b49a10 msgr2=0x55b510b49de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:01.295 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.287+0000 7f9faf214000 1 --2- 172.21.15.16:0/3003742078 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b510b49a10 0x55b510b49de0 secure :-1 s=READY pgs=1300 cs=0 l=1 rev1=1 crypto rx=0x7f9f9c01c400 tx=0x7f9f9c004740 comp rx=0 tx=0).stop 2023-05-22T15:43:01.295 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.287+0000 7f9faf214000 1 -- 172.21.15.16:0/3003742078 shutdown_connections 2023-05-22T15:43:01.295 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.287+0000 7f9faf214000 1 --2- 172.21.15.16:0/3003742078 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b510b49a10 0x55b510b49de0 unknown :-1 s=CLOSED pgs=1300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:01.295 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.287+0000 7f9faf214000 1 --2- 172.21.15.16:0/3003742078 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x55b510b96170 0x55b510b96540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:01.296 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.287+0000 7f9faf214000 1 --2- 172.21.15.16:0/3003742078 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b510922fd0 0x55b510b95c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:01.296 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.287+0000 7f9faf214000 1 -- 172.21.15.16:0/3003742078 >> 172.21.15.16:0/3003742078 conn(0x55b5109f2270 msgr2=0x55b510b79220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:43:01.296 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.287+0000 7f9faf214000 1 -- 172.21.15.16:0/3003742078 shutdown_connections 2023-05-22T15:43:01.296 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.287+0000 7f9faf214000 1 -- 172.21.15.16:0/3003742078 wait complete. 2023-05-22T15:43:01.299 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.287+0000 7f9faf214000 1 -- start start 2023-05-22T15:43:01.300 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.287+0000 7f9faf214000 1 Processor -- start 2023-05-22T15:43:01.300 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.287+0000 7f9faf214000 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b510922fd0 0x55b510b51a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:01.300 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.287+0000 7f9faf214000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b510b49a10 0x55b510b51f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:01.300 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.287+0000 7f9faf214000 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x55b510b96170 0x55b510b7b4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:01.300 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.287+0000 7f9faf214000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b51099a780 con 0x55b510b49a10 2023-05-22T15:43:01.300 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.287+0000 7f9faf214000 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b510958250 con 0x55b510922fd0 2023-05-22T15:43:01.301 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.287+0000 7f9faf214000 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b510957ef0 con 0x55b510b96170 2023-05-22T15:43:01.301 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.287+0000 7f9faf214000 10 client.0 _ll_register_callbacks cb 0x55b510b37b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-05-22T15:43:01.301 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.287+0000 7f9fa968d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b510b49a10 0x55b510b51f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:01.301 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.287+0000 7f9fa968d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b510b49a10 0x55b510b51f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49852/0 (socket says 172.21.15.16:49852) 2023-05-22T15:43:01.301 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.287+0000 7f9fa968d640 1 -- 172.21.15.16:0/2371545006 learned_addr learned my addr 172.21.15.16:0/2371545006 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:43:01.302 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.287+0000 7f9fa9e8e640 1 --2- 172.21.15.16:0/2371545006 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b510922fd0 0x55b510b51a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:01.302 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.287+0000 7f9faa68f640 1 --2- 172.21.15.16:0/2371545006 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x55b510b96170 0x55b510b7b4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:01.302 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.287+0000 7f9fa968d640 1 -- 172.21.15.16:0/2371545006 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x55b510b96170 msgr2=0x55b510b7b4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:43:01.302 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.287+0000 7f9fa968d640 1 --2- 172.21.15.16:0/2371545006 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x55b510b96170 0x55b510b7b4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:01.302 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.287+0000 7f9fa968d640 1 -- 172.21.15.16:0/2371545006 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b510922fd0 msgr2=0x55b510b51a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:43:01.302 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.287+0000 7f9fa968d640 1 --2- 172.21.15.16:0/2371545006 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b510922fd0 0x55b510b51a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:01.303 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.287+0000 7f9fa968d640 1 -- 172.21.15.16:0/2371545006 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9fa415dc60 con 0x55b510b49a10 2023-05-22T15:43:01.303 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.287+0000 7f9faa68f640 1 --2- 172.21.15.16:0/2371545006 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x55b510b96170 0x55b510b7b4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:43:01.303 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.287+0000 7f9fa968d640 1 --2- 172.21.15.16:0/2371545006 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b510b49a10 0x55b510b51f70 secure :-1 s=READY pgs=1301 cs=0 l=1 rev1=1 crypto rx=0x7f9f98016b70 tx=0x7f9f98004ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cef466be633c9357 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:43:01.303 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.287+0000 7f9f84ff9640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2023-05-22T15:43:01.303 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.287+0000 7f9f84ff9640 1 -- 172.21.15.16:0/2371545006 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f98014ce0 con 0x55b510b49a10 2023-05-22T15:43:01.303 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.287+0000 7f9f84ff9640 1 -- 172.21.15.16:0/2371545006 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(104..104 src has 1..104) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f9f98014e40 con 0x55b510b49a10 2023-05-22T15:43:01.304 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.287+0000 7f9faf214000 1 -- 172.21.15.16:0/2371545006 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55b51095ec60 con 0x55b510b49a10 2023-05-22T15:43:01.304 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.287+0000 7f9f84ff9640 1 -- 172.21.15.16:0/2371545006 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9f980039a0 con 0x55b510b49a10 2023-05-22T15:43:01.304 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.287+0000 7f9faf214000 10 client.7671 fetch_fsmap learned FSMap version 36 2023-05-22T15:43:01.304 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.287+0000 7f9faf214000 1 -- 172.21.15.16:0/2371545006 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=2+}) v3 -- 0x55b510697890 con 0x55b510b49a10 2023-05-22T15:43:01.304 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.287+0000 7f9f84ff9640 1 -- 172.21.15.16:0/2371545006 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55b51095ec60 con 0x55b510b49a10 2023-05-22T15:43:01.305 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.287+0000 7f9f84ff9640 1 -- 172.21.15.16:0/2371545006 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 36) v1 ==== 3106+0+0 (secure 0 0 0) 0x7f9f98003b00 con 0x55b510b49a10 2023-05-22T15:43:01.305 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.287+0000 7f9faf214000 10 client.7671 fetch_fsmap finished waiting for FSMap version 36 2023-05-22T15:43:01.305 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.287+0000 7f9faf214000 10 client.7671 resolve_mds: resolved name 'a' to daemon mds.a 2023-05-22T15:43:01.305 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.287+0000 7f9faf214000 20 client.7671 populate_metadata read hostname 'smithi016' 2023-05-22T15:43:01.305 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.287+0000 7f9faf214000 1 --2- 172.21.15.16:0/2371545006 >> [v2:172.21.15.16:6835/1353635569,v1:172.21.15.16:6837/1353635569] conn(0x55b510ba58e0 0x55b510ba7cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:01.305 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.287+0000 7f9faf214000 4 client.7671 mds_command: new command op to 7527 tid=0[{"prefix": "get_command_descriptions"}] 2023-05-22T15:43:01.305 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.287+0000 7f9faf214000 1 -- 172.21.15.16:0/2371545006 --> [v2:172.21.15.16:6835/1353635569,v1:172.21.15.16:6837/1353635569] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55b5109990d0 con 0x55b510ba58e0 2023-05-22T15:43:01.306 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.287+0000 7f9fa9e8e640 1 --2- 172.21.15.16:0/2371545006 >> [v2:172.21.15.16:6835/1353635569,v1:172.21.15.16:6837/1353635569] conn(0x55b510ba58e0 0x55b510ba7cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:01.306 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.291+0000 7f9fa9e8e640 1 --2- 172.21.15.16:0/2371545006 >> [v2:172.21.15.16:6835/1353635569,v1:172.21.15.16:6837/1353635569] conn(0x55b510ba58e0 0x55b510ba7cc0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1b5983b936485dd4 server_cookie=f68bfbb06645e18d in_seq=0 out_seq=0 2023-05-22T15:43:01.306 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.291+0000 7f9f84ff9640 10 client.7671 ms_handle_connect on v2:172.21.15.16:6835/1353635569 2023-05-22T15:43:01.306 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.291+0000 7f9f84ff9640 1 -- 172.21.15.16:0/2371545006 <== mds.0 v2:172.21.15.16:6835/1353635569 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x55b5109990d0 con 0x55b510ba58e0 2023-05-22T15:43:01.306 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.291+0000 7f9f84ff9640 10 client.7671 handle_command_reply: tid=0 2023-05-22T15:43:01.306 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.291+0000 7f9faf214000 1 client.7671 shutdown 2023-05-22T15:43:01.306 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.291+0000 7f9faf214000 1 -- 172.21.15.16:0/2371545006 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b510b49a10 msgr2=0x55b510b51f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:01.307 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.291+0000 7f9faf214000 1 --2- 172.21.15.16:0/2371545006 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b510b49a10 0x55b510b51f70 secure :-1 s=READY pgs=1301 cs=0 l=1 rev1=1 crypto rx=0x7f9f98016b70 tx=0x7f9f98004ac0 comp rx=0 tx=0).stop 2023-05-22T15:43:01.307 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.291+0000 7f9faf214000 1 -- 172.21.15.16:0/2371545006 shutdown_connections 2023-05-22T15:43:01.307 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.291+0000 7f9faf214000 1 --2- 172.21.15.16:0/2371545006 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x55b510b96170 0x55b510b7b4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:01.307 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.291+0000 7f9faf214000 1 --2- 172.21.15.16:0/2371545006 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b510b49a10 0x55b510b51f70 unknown :-1 s=CLOSED pgs=1301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:01.307 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.291+0000 7f9faf214000 1 --2- 172.21.15.16:0/2371545006 >> [v2:172.21.15.16:6835/1353635569,v1:172.21.15.16:6837/1353635569] conn(0x55b510ba58e0 0x55b510ba7cc0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:01.307 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.291+0000 7f9faf214000 1 --2- 172.21.15.16:0/2371545006 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b510922fd0 0x55b510b51a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:01.307 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.291+0000 7f9f84ff9640 0 client.7671 ms_handle_reset on v2:172.21.15.16:6835/1353635569 2023-05-22T15:43:01.307 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.291+0000 7f9faf214000 1 -- 172.21.15.16:0/2371545006 >> 172.21.15.16:0/2371545006 conn(0x55b5109f2270 msgr2=0x55b510b49100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:43:01.308 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.291+0000 7f9faf214000 1 -- 172.21.15.16:0/2371545006 shutdown_connections 2023-05-22T15:43:01.308 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.291+0000 7f9faf214000 1 -- 172.21.15.16:0/2371545006 wait complete. 2023-05-22T15:43:01.308 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.291+0000 7f9faf214000 20 client.7671 trim_cache size 0 max 16384 2023-05-22T15:43:01.311 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.307+0000 7f9faf214000 1 Processor -- start 2023-05-22T15:43:01.311 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.307+0000 7f9faf214000 1 -- start start 2023-05-22T15:43:01.312 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.307+0000 7f9faf214000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b510922fd0 0x55b510b4e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:01.312 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.307+0000 7f9faf214000 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x55b510b49a10 0x55b510b4e730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:01.312 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.307+0000 7f9faf214000 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b510b96170 0x55b510b94300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:01.312 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.307+0000 7f9faf214000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b510658940 con 0x55b510922fd0 2023-05-22T15:43:01.313 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.307+0000 7f9faf214000 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b510b7a0a0 con 0x55b510b96170 2023-05-22T15:43:01.313 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.307+0000 7f9faf214000 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b510997b80 con 0x55b510b49a10 2023-05-22T15:43:01.313 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.307+0000 7f9fa9e8e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b510922fd0 0x55b510b4e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:01.313 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.307+0000 7f9fa9e8e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b510922fd0 0x55b510b4e1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49866/0 (socket says 172.21.15.16:49866) 2023-05-22T15:43:01.313 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.307+0000 7f9fa9e8e640 1 -- 172.21.15.16:0/2692381256 learned_addr learned my addr 172.21.15.16:0/2692381256 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:43:01.313 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.307+0000 7f9fa9e8e640 1 -- 172.21.15.16:0/2692381256 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x55b510b49a10 msgr2=0x55b510b4e730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:43:01.313 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.307+0000 7f9fa968d640 1 --2- 172.21.15.16:0/2692381256 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x55b510b49a10 0x55b510b4e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:01.314 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.307+0000 7f9faa68f640 1 --2- 172.21.15.16:0/2692381256 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b510b96170 0x55b510b94300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:01.314 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.307+0000 7f9fa9e8e640 1 --2- 172.21.15.16:0/2692381256 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x55b510b49a10 0x55b510b4e730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:01.314 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.307+0000 7f9fa9e8e640 1 -- 172.21.15.16:0/2692381256 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b510b96170 msgr2=0x55b510b94300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:43:01.315 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.307+0000 7f9fa9e8e640 1 --2- 172.21.15.16:0/2692381256 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b510b96170 0x55b510b94300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:01.315 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.307+0000 7f9fa9e8e640 1 -- 172.21.15.16:0/2692381256 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b510960360 con 0x55b510922fd0 2023-05-22T15:43:01.315 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.307+0000 7f9faa68f640 1 --2- 172.21.15.16:0/2692381256 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b510b96170 0x55b510b94300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:43:01.315 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.307+0000 7f9fa9e8e640 1 --2- 172.21.15.16:0/2692381256 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b510922fd0 0x55b510b4e1f0 secure :-1 s=READY pgs=1302 cs=0 l=1 rev1=1 crypto rx=0x7f9fa0053750 tx=0x7f9fa0007f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c74590fa3360200 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:43:01.320 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.307+0000 7f9f87fff640 1 -- 172.21.15.16:0/2692381256 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fa0080530 con 0x55b510922fd0 2023-05-22T15:43:01.321 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.307+0000 7f9f87fff640 1 -- 172.21.15.16:0/2692381256 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9fa005b420 con 0x55b510922fd0 2023-05-22T15:43:01.321 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.307+0000 7f9f87fff640 1 -- 172.21.15.16:0/2692381256 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fa0054bb0 con 0x55b510922fd0 2023-05-22T15:43:01.321 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.307+0000 7f9faf214000 1 -- 172.21.15.16:0/2692381256 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b510922fd0 msgr2=0x55b510b4e1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:01.322 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.307+0000 7f9faf214000 1 --2- 172.21.15.16:0/2692381256 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b510922fd0 0x55b510b4e1f0 secure :-1 s=READY pgs=1302 cs=0 l=1 rev1=1 crypto rx=0x7f9fa0053750 tx=0x7f9fa0007f90 comp rx=0 tx=0).stop 2023-05-22T15:43:01.322 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.307+0000 7f9faf214000 1 -- 172.21.15.16:0/2692381256 shutdown_connections 2023-05-22T15:43:01.322 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.307+0000 7f9faf214000 1 --2- 172.21.15.16:0/2692381256 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b510b96170 0x55b510b94300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:01.322 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.307+0000 7f9faf214000 1 --2- 172.21.15.16:0/2692381256 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x55b510b49a10 0x55b510b4e730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:01.323 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.307+0000 7f9faf214000 1 --2- 172.21.15.16:0/2692381256 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b510922fd0 0x55b510b4e1f0 unknown :-1 s=CLOSED pgs=1302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:01.323 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.307+0000 7f9faf214000 1 -- 172.21.15.16:0/2692381256 >> 172.21.15.16:0/2692381256 conn(0x55b5109f2270 msgr2=0x55b510b78f10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:43:01.323 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.307+0000 7f9faf214000 1 -- 172.21.15.16:0/2692381256 shutdown_connections 2023-05-22T15:43:01.323 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.307+0000 7f9faf214000 1 -- 172.21.15.16:0/2692381256 wait complete. 2023-05-22T15:43:01.324 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.307+0000 7f9faf214000 1 -- start start 2023-05-22T15:43:01.324 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.307+0000 7f9faf214000 1 Processor -- start 2023-05-22T15:43:01.324 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.311+0000 7f9faf214000 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b510922fd0 0x55b510bf7b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:01.324 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.311+0000 7f9faf214000 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x55b510b49a10 0x55b510bfa090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:01.325 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.311+0000 7f9faf214000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b510b96170 0x55b510bfc5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:01.325 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.311+0000 7f9faf214000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b5109a28d0 con 0x55b510b96170 2023-05-22T15:43:01.325 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.311+0000 7f9faf214000 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b510994b20 con 0x55b510922fd0 2023-05-22T15:43:01.325 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.311+0000 7f9faf214000 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b510b43e80 con 0x55b510b49a10 2023-05-22T15:43:01.326 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.311+0000 7f9faf214000 10 client.0 _ll_register_callbacks cb 0x55b5109eaeb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-05-22T15:43:01.326 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.311+0000 7f9faa68f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b510b96170 0x55b510bfc5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:01.326 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.311+0000 7f9faa68f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b510b96170 0x55b510bfc5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49880/0 (socket says 172.21.15.16:49880) 2023-05-22T15:43:01.326 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.311+0000 7f9faa68f640 1 -- 172.21.15.16:0/3985074707 learned_addr learned my addr 172.21.15.16:0/3985074707 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:43:01.327 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.311+0000 7f9fa9e8e640 1 --2- 172.21.15.16:0/3985074707 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b510922fd0 0x55b510bf7b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:01.327 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.311+0000 7f9fa968d640 1 --2- 172.21.15.16:0/3985074707 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x55b510b49a10 0x55b510bfa090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:01.327 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.311+0000 7f9faa68f640 1 -- 172.21.15.16:0/3985074707 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x55b510b49a10 msgr2=0x55b510bfa090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:43:01.327 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.311+0000 7f9faa68f640 1 --2- 172.21.15.16:0/3985074707 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x55b510b49a10 0x55b510bfa090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:01.328 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.311+0000 7f9faa68f640 1 -- 172.21.15.16:0/3985074707 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b510922fd0 msgr2=0x55b510bf7b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:43:01.328 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.311+0000 7f9faa68f640 1 --2- 172.21.15.16:0/3985074707 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b510922fd0 0x55b510bf7b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:01.328 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.311+0000 7f9faa68f640 1 -- 172.21.15.16:0/3985074707 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9fa40ca8b0 con 0x55b510b96170 2023-05-22T15:43:01.328 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.311+0000 7f9fa968d640 1 --2- 172.21.15.16:0/3985074707 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x55b510b49a10 0x55b510bfa090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:43:01.329 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.311+0000 7f9faa68f640 1 --2- 172.21.15.16:0/3985074707 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b510b96170 0x55b510bfc5e0 secure :-1 s=READY pgs=1303 cs=0 l=1 rev1=1 crypto rx=0x7f9f9c01c400 tx=0x7f9f9c0046d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1035f61e4cb7b17 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:43:01.329 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.311+0000 7f9f84ff9640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2023-05-22T15:43:01.329 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.311+0000 7f9f84ff9640 1 -- 172.21.15.16:0/3985074707 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f9c003820 con 0x55b510b96170 2023-05-22T15:43:01.330 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.311+0000 7f9f84ff9640 1 -- 172.21.15.16:0/3985074707 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(104..104 src has 1..104) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f9f9c0a4db0 con 0x55b510b96170 2023-05-22T15:43:01.330 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.311+0000 7f9faf214000 1 -- 172.21.15.16:0/3985074707 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55b510b50bb0 con 0x55b510b96170 2023-05-22T15:43:01.330 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.311+0000 7f9f84ff9640 1 -- 172.21.15.16:0/3985074707 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9f9c003980 con 0x55b510b96170 2023-05-22T15:43:01.330 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.311+0000 7f9faf214000 10 client.7677 fetch_fsmap learned FSMap version 36 2023-05-22T15:43:01.331 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.311+0000 7f9faf214000 1 -- 172.21.15.16:0/3985074707 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=2+}) v3 -- 0x55b510945be0 con 0x55b510b96170 2023-05-22T15:43:01.331 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.311+0000 7f9f84ff9640 1 -- 172.21.15.16:0/3985074707 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55b510b50bb0 con 0x55b510b96170 2023-05-22T15:43:01.331 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.311+0000 7f9f84ff9640 1 -- 172.21.15.16:0/3985074707 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 36) v1 ==== 3106+0+0 (secure 0 0 0) 0x7f9f9c003ae0 con 0x55b510b96170 2023-05-22T15:43:01.332 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.311+0000 7f9faf214000 10 client.7677 fetch_fsmap finished waiting for FSMap version 36 2023-05-22T15:43:01.332 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.311+0000 7f9faf214000 10 client.7677 resolve_mds: resolved name 'a' to daemon mds.a 2023-05-22T15:43:01.332 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.311+0000 7f9faf214000 20 client.7677 populate_metadata read hostname 'smithi016' 2023-05-22T15:43:01.332 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.311+0000 7f9faf214000 1 --2- 172.21.15.16:0/3985074707 >> [v2:172.21.15.16:6835/1353635569,v1:172.21.15.16:6837/1353635569] conn(0x55b510bfcd30 0x55b510bff110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:01.333 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.311+0000 7f9faf214000 4 client.7677 mds_command: new command op to 7527 tid=0[{"prefix": "status"}] 2023-05-22T15:43:01.333 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.311+0000 7f9faf214000 1 -- 172.21.15.16:0/3985074707 --> [v2:172.21.15.16:6835/1353635569,v1:172.21.15.16:6837/1353635569] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55b5109414d0 con 0x55b510bfcd30 2023-05-22T15:43:01.333 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.311+0000 7f9fa9e8e640 1 --2- 172.21.15.16:0/3985074707 >> [v2:172.21.15.16:6835/1353635569,v1:172.21.15.16:6837/1353635569] conn(0x55b510bfcd30 0x55b510bff110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:01.333 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.311+0000 7f9fa9e8e640 1 --2- 172.21.15.16:0/3985074707 >> [v2:172.21.15.16:6835/1353635569,v1:172.21.15.16:6837/1353635569] conn(0x55b510bfcd30 0x55b510bff110 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=efbeba30e66cad38 server_cookie=cd350d2bf716b5b3 in_seq=0 out_seq=0 2023-05-22T15:43:01.333 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.311+0000 7f9f84ff9640 10 client.7677 ms_handle_connect on v2:172.21.15.16:6835/1353635569 2023-05-22T15:43:01.333 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.311+0000 7f9f84ff9640 1 -- 172.21.15.16:0/3985074707 <== mds.0 v2:172.21.15.16:6835/1353635569 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+333 (crc 0 0 0) 0x55b5109414d0 con 0x55b510bfcd30 2023-05-22T15:43:01.333 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.311+0000 7f9f84ff9640 10 client.7677 handle_command_reply: tid=0 2023-05-22T15:43:01.334 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.311+0000 7f9faf214000 1 client.7677 shutdown 2023-05-22T15:43:01.334 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.311+0000 7f9faf214000 1 -- 172.21.15.16:0/3985074707 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b510b96170 msgr2=0x55b510bfc5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:01.334 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.311+0000 7f9faf214000 1 --2- 172.21.15.16:0/3985074707 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b510b96170 0x55b510bfc5e0 secure :-1 s=READY pgs=1303 cs=0 l=1 rev1=1 crypto rx=0x7f9f9c01c400 tx=0x7f9f9c0046d0 comp rx=0 tx=0).stop 2023-05-22T15:43:01.334 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.311+0000 7f9faf214000 1 -- 172.21.15.16:0/3985074707 shutdown_connections 2023-05-22T15:43:01.334 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.311+0000 7f9faf214000 1 --2- 172.21.15.16:0/3985074707 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b510b96170 0x55b510bfc5e0 unknown :-1 s=CLOSED pgs=1303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:01.334 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.311+0000 7f9faf214000 1 --2- 172.21.15.16:0/3985074707 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x55b510b49a10 0x55b510bfa090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:01.335 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.311+0000 7f9faf214000 1 --2- 172.21.15.16:0/3985074707 >> [v2:172.21.15.16:6835/1353635569,v1:172.21.15.16:6837/1353635569] conn(0x55b510bfcd30 0x55b510bff110 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:01.335 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.311+0000 7f9faf214000 1 --2- 172.21.15.16:0/3985074707 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b510922fd0 0x55b510bf7b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:01.335 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.311+0000 7f9f84ff9640 0 client.7677 ms_handle_reset on v2:172.21.15.16:6835/1353635569 2023-05-22T15:43:01.335 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.311+0000 7f9faf214000 1 -- 172.21.15.16:0/3985074707 >> 172.21.15.16:0/3985074707 conn(0x55b5109f2270 msgr2=0x55b510b7b1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:43:01.336 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.311+0000 7f9faf214000 1 -- 172.21.15.16:0/3985074707 shutdown_connections 2023-05-22T15:43:01.336 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.311+0000 7f9faf214000 1 -- 172.21.15.16:0/3985074707 wait complete. 2023-05-22T15:43:01.336 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.311+0000 7f9faf214000 20 client.7677 trim_cache size 0 max 16384 2023-05-22T15:43:01.337 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-05-22T15:43:01.337 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_fsid": "b2719834-7847-4d98-90a6-f65d57f85c49", 2023-05-22T15:43:01.337 INFO:teuthology.orchestra.run.smithi016.stdout: "whoami": 0, 2023-05-22T15:43:01.338 INFO:teuthology.orchestra.run.smithi016.stdout: "id": 7527, 2023-05-22T15:43:01.338 INFO:teuthology.orchestra.run.smithi016.stdout: "want_state": "up:active", 2023-05-22T15:43:01.338 INFO:teuthology.orchestra.run.smithi016.stdout: "state": "up:active", 2023-05-22T15:43:01.338 INFO:teuthology.orchestra.run.smithi016.stdout: "fs_name": "cephfs", 2023-05-22T15:43:01.339 INFO:teuthology.orchestra.run.smithi016.stdout: "rank_uptime": 4.7472503279999998, 2023-05-22T15:43:01.339 INFO:teuthology.orchestra.run.smithi016.stdout: "mdsmap_epoch": 36, 2023-05-22T15:43:01.339 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch": 104, 2023-05-22T15:43:01.339 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch_barrier": 104, 2023-05-22T15:43:01.340 INFO:teuthology.orchestra.run.smithi016.stdout: "uptime": 9.2547699399999992 2023-05-22T15:43:01.340 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-05-22T15:43:01.342 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.311+0000 7f9fac0f2640 1 -- 172.21.15.16:0/502925819 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f9f880654a0 msgr2=0x7f9f88067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:01.342 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.311+0000 7f9fac0f2640 1 --2- 172.21.15.16:0/502925819 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f9f880654a0 0x7f9f88067950 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f9fa40b0220 tx=0x7f9f9800b040 comp rx=0 tx=0).stop 2023-05-22T15:43:01.342 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.311+0000 7f9fac0f2640 1 -- 172.21.15.16:0/502925819 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9fa4070df0 msgr2=0x7f9fa40c7b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:01.342 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.311+0000 7f9fac0f2640 1 --2- 172.21.15.16:0/502925819 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9fa4070df0 0x7f9fa40c7b20 secure :-1 s=READY pgs=1299 cs=0 l=1 rev1=1 crypto rx=0x7f9fa000c3c0 tx=0x7f9fa000c880 comp rx=0 tx=0).stop 2023-05-22T15:43:01.343 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.311+0000 7f9fac0f2640 1 -- 172.21.15.16:0/502925819 shutdown_connections 2023-05-22T15:43:01.343 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.311+0000 7f9fac0f2640 1 --2- 172.21.15.16:0/502925819 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f9f880654a0 0x7f9f88067950 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:01.343 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.311+0000 7f9fac0f2640 1 --2- 172.21.15.16:0/502925819 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9fa40ca4e0 0x7f9fa4150140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:01.343 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.311+0000 7f9fac0f2640 1 --2- 172.21.15.16:0/502925819 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9fa40c9b40 0x7f9fa415ae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:01.344 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.311+0000 7f9fac0f2640 1 --2- 172.21.15.16:0/502925819 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9fa4070df0 0x7f9fa40c7b20 unknown :-1 s=CLOSED pgs=1299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:01.344 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.311+0000 7f9fac0f2640 1 -- 172.21.15.16:0/502925819 >> 172.21.15.16:0/502925819 conn(0x7f9fa4069720 msgr2=0x7f9fa40adda0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:43:01.344 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.311+0000 7f9fac0f2640 1 -- 172.21.15.16:0/502925819 shutdown_connections 2023-05-22T15:43:01.344 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.311+0000 7f9fac0f2640 1 -- 172.21.15.16:0/502925819 wait complete. 2023-05-22T15:43:01.345 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-05-22T15:43:01.458 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.455+0000 7f3f98af1640 1 Processor -- start 2023-05-22T15:43:01.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.455+0000 7f3f98af1640 1 -- start start 2023-05-22T15:43:01.460 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.455+0000 7f3f98af1640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f3f940ca4e0 0x7f3f9415e460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:01.460 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.455+0000 7f3f98af1640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f3f940c9b40 0x7f3f940afb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:01.460 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.455+0000 7f3f98af1640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f940b0650 con 0x7f3f94070df0 2023-05-22T15:43:01.460 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.455+0000 7f3f98af1640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f940b0370 con 0x7f3f940c9b40 2023-05-22T15:43:01.460 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.455+0000 7f3f98af1640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f9415d980 con 0x7f3f940ca4e0 2023-05-22T15:43:01.460 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.455+0000 7f3f91d9b640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f3f940c9b40 0x7f3f940afb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:01.461 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.455+0000 7f3f91d9b640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f3f940c9b40 0x7f3f940afb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49894/0 (socket says 172.21.15.16:49894) 2023-05-22T15:43:01.461 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.455+0000 7f3f91d9b640 1 -- 172.21.15.16:0/2556512712 learned_addr learned my addr 172.21.15.16:0/2556512712 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:43:01.461 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.455+0000 7f3f9259c640 1 --2- 172.21.15.16:0/2556512712 >> v2:172.21.15.62:3300/0 conn(0x7f3f940ca4e0 0x7f3f9415e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:01.461 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.455+0000 7f3f91d9b640 1 -- 172.21.15.16:0/2556512712 >> v2:172.21.15.62:3300/0 conn(0x7f3f940ca4e0 msgr2=0x7f3f9415e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:43:01.462 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.455+0000 7f3f91d9b640 1 --2- 172.21.15.16:0/2556512712 >> v2:172.21.15.62:3300/0 conn(0x7f3f940ca4e0 0x7f3f9415e460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:01.462 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.455+0000 7f3f91d9b640 1 -- 172.21.15.16:0/2556512712 >> v1:172.21.15.62:6789/0 conn(0x7f3f94070df0 legacy=0x7f3f94071210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:43:01.462 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.455+0000 7f3f91d9b640 1 -- 172.21.15.16:0/2556512712 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f9415dc60 con 0x7f3f940c9b40 2023-05-22T15:43:01.462 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.455+0000 7f3f9259c640 1 --2- 172.21.15.16:0/2556512712 >> v2:172.21.15.62:3300/0 conn(0x7f3f940ca4e0 0x7f3f9415e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:43:01.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.459+0000 7f3f91d9b640 1 --2- 172.21.15.16:0/2556512712 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3f940c9b40 0x7f3f940afb40 secure :-1 s=READY pgs=1304 cs=0 l=1 rev1=1 crypto rx=0x7f3f940cad40 tx=0x7f3f7c01c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6a6339db87842c0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:43:01.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.459+0000 7f3f9159a640 1 -- 172.21.15.16:0/2556512712 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f7c0ab670 con 0x7f3f940c9b40 2023-05-22T15:43:01.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.459+0000 7f3f9159a640 1 -- 172.21.15.16:0/2556512712 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f7c0ab7d0 con 0x7f3f940c9b40 2023-05-22T15:43:01.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.459+0000 7f3f9159a640 1 -- 172.21.15.16:0/2556512712 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f7c0abb60 con 0x7f3f940c9b40 2023-05-22T15:43:01.464 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.459+0000 7f3f98af1640 1 -- 172.21.15.16:0/2556512712 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3f940c9b40 msgr2=0x7f3f940afb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:01.464 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.459+0000 7f3f98af1640 1 --2- 172.21.15.16:0/2556512712 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3f940c9b40 0x7f3f940afb40 secure :-1 s=READY pgs=1304 cs=0 l=1 rev1=1 crypto rx=0x7f3f940cad40 tx=0x7f3f7c01c490 comp rx=0 tx=0).stop 2023-05-22T15:43:01.464 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.459+0000 7f3f98af1640 1 -- 172.21.15.16:0/2556512712 shutdown_connections 2023-05-22T15:43:01.464 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.459+0000 7f3f98af1640 1 --2- 172.21.15.16:0/2556512712 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3f940c9b40 0x7f3f940afb40 unknown :-1 s=CLOSED pgs=1304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:01.464 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.459+0000 7f3f98af1640 1 --2- 172.21.15.16:0/2556512712 >> v2:172.21.15.62:3300/0 conn(0x7f3f940ca4e0 0x7f3f9415e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:01.465 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.459+0000 7f3f98af1640 1 -- 172.21.15.16:0/2556512712 >> 172.21.15.16:0/2556512712 conn(0x7f3f94069720 msgr2=0x7f3f94069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:43:01.465 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.459+0000 7f3f98af1640 1 -- 172.21.15.16:0/2556512712 shutdown_connections 2023-05-22T15:43:01.465 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.459+0000 7f3f98af1640 1 -- 172.21.15.16:0/2556512712 wait complete. 2023-05-22T15:43:01.466 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.459+0000 7f3f98af1640 1 Processor -- start 2023-05-22T15:43:01.466 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.459+0000 7f3f98af1640 1 -- start start 2023-05-22T15:43:01.466 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.459+0000 7f3f98af1640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3f94070df0 0x7f3f94149b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:01.466 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.459+0000 7f3f98af1640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3f940c9b40 0x7f3f940b2a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:01.467 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.459+0000 7f3f98af1640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3f940ca4e0 0x7f3f9414f200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:01.467 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.459+0000 7f3f98af1640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f9414a360 con 0x7f3f940c9b40 2023-05-22T15:43:01.467 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.459+0000 7f3f98af1640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f940b0eb0 con 0x7f3f940ca4e0 2023-05-22T15:43:01.467 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.459+0000 7f3f98af1640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f940cd150 con 0x7f3f94070df0 2023-05-22T15:43:01.467 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.459+0000 7f3f91d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3f940c9b40 0x7f3f940b2a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:01.467 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.459+0000 7f3f91d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3f940c9b40 0x7f3f940b2a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49902/0 (socket says 172.21.15.16:49902) 2023-05-22T15:43:01.467 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.459+0000 7f3f91d9b640 1 -- 172.21.15.16:0/2436963693 learned_addr learned my addr 172.21.15.16:0/2436963693 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:43:01.468 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.459+0000 7f3f92d9d640 1 --2- 172.21.15.16:0/2436963693 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3f940ca4e0 0x7f3f9414f200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:01.468 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.459+0000 7f3f9259c640 1 --2- 172.21.15.16:0/2436963693 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3f94070df0 0x7f3f94149b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:01.468 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.463+0000 7f3f91d9b640 1 -- 172.21.15.16:0/2436963693 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3f94070df0 msgr2=0x7f3f94149b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:01.468 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.463+0000 7f3f91d9b640 1 --2- 172.21.15.16:0/2436963693 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3f94070df0 0x7f3f94149b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:01.468 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.463+0000 7f3f91d9b640 1 -- 172.21.15.16:0/2436963693 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3f940ca4e0 msgr2=0x7f3f9414f200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:01.468 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.463+0000 7f3f91d9b640 1 --2- 172.21.15.16:0/2436963693 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3f940ca4e0 0x7f3f9414f200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:01.468 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.463+0000 7f3f91d9b640 1 -- 172.21.15.16:0/2436963693 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f7c0094d0 con 0x7f3f940c9b40 2023-05-22T15:43:01.469 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.463+0000 7f3f91d9b640 1 --2- 172.21.15.16:0/2436963693 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3f940c9b40 0x7f3f940b2a70 secure :-1 s=READY pgs=1305 cs=0 l=1 rev1=1 crypto rx=0x7f3f7c01a0c0 tx=0x7f3f7c01acd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:43:01.469 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.463+0000 7f3f8affd640 1 -- 172.21.15.16:0/2436963693 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f7c0b48c0 con 0x7f3f940c9b40 2023-05-22T15:43:01.470 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.463+0000 7f3f8affd640 1 -- 172.21.15.16:0/2436963693 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f7c0b4a40 con 0x7f3f940c9b40 2023-05-22T15:43:01.470 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.463+0000 7f3f98af1640 1 -- 172.21.15.16:0/2436963693 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f940c8e80 con 0x7f3f940c9b40 2023-05-22T15:43:01.470 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.463+0000 7f3f8affd640 1 -- 172.21.15.16:0/2436963693 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f7c0b4df0 con 0x7f3f940c9b40 2023-05-22T15:43:01.470 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.463+0000 7f3f98af1640 1 -- 172.21.15.16:0/2436963693 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f940ca8b0 con 0x7f3f940c9b40 2023-05-22T15:43:01.470 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.463+0000 7f3f98af1640 1 -- 172.21.15.16:0/2436963693 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f940c9010 con 0x7f3f940c9b40 2023-05-22T15:43:01.471 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.463+0000 7f3f8affd640 1 -- 172.21.15.16:0/2436963693 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 34) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f3f7c0b3380 con 0x7f3f940c9b40 2023-05-22T15:43:01.471 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.463+0000 7f3f8affd640 1 --2- 172.21.15.16:0/2436963693 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f3f740654a0 0x7f3f74067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:01.471 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.463+0000 7f3f8affd640 1 -- 172.21.15.16:0/2436963693 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f3f7c11d4f0 con 0x7f3f940c9b40 2023-05-22T15:43:01.471 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.463+0000 7f3f9259c640 1 --2- 172.21.15.16:0/2436963693 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f3f740654a0 0x7f3f74067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:01.471 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.463+0000 7f3f9259c640 1 --2- 172.21.15.16:0/2436963693 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f3f740654a0 0x7f3f74067950 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f3f84006fd0 tx=0x7f3f84008040 comp rx=0 tx=0).ready entity=mgr.6436 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:43:01.472 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.467+0000 7f3f8affd640 1 -- 172.21.15.16:0/2436963693 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f3f7c0eae60 con 0x7f3f940c9b40 2023-05-22T15:43:01.660 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.655+0000 7f3f98af1640 1 -- 172.21.15.16:0/2436963693 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f3f9404dbe0 con 0x7f3f940c9b40 2023-05-22T15:43:01.660 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.655+0000 7f3f8affd640 1 -- 172.21.15.16:0/2436963693 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v36) v1 ==== 53+0+73 (secure 0 0 0) 0x7f3f7c0ee480 con 0x7f3f940c9b40 2023-05-22T15:43:01.661 INFO:teuthology.orchestra.run.smithi016.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-05-22T15:43:01.663 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.659+0000 7f3f98af1640 1 -- 172.21.15.16:0/2436963693 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f3f740654a0 msgr2=0x7f3f74067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:01.664 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.659+0000 7f3f98af1640 1 --2- 172.21.15.16:0/2436963693 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f3f740654a0 0x7f3f74067950 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f3f84006fd0 tx=0x7f3f84008040 comp rx=0 tx=0).stop 2023-05-22T15:43:01.664 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.659+0000 7f3f98af1640 1 -- 172.21.15.16:0/2436963693 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3f940c9b40 msgr2=0x7f3f940b2a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:01.664 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.659+0000 7f3f98af1640 1 --2- 172.21.15.16:0/2436963693 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3f940c9b40 0x7f3f940b2a70 secure :-1 s=READY pgs=1305 cs=0 l=1 rev1=1 crypto rx=0x7f3f7c01a0c0 tx=0x7f3f7c01acd0 comp rx=0 tx=0).stop 2023-05-22T15:43:01.664 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.659+0000 7f3f98af1640 1 -- 172.21.15.16:0/2436963693 shutdown_connections 2023-05-22T15:43:01.664 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.659+0000 7f3f98af1640 1 --2- 172.21.15.16:0/2436963693 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3f940ca4e0 0x7f3f9414f200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:01.664 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.659+0000 7f3f98af1640 1 --2- 172.21.15.16:0/2436963693 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3f940c9b40 0x7f3f940b2a70 unknown :-1 s=CLOSED pgs=1305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:01.664 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.659+0000 7f3f98af1640 1 --2- 172.21.15.16:0/2436963693 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f3f740654a0 0x7f3f74067950 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:01.665 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.659+0000 7f3f98af1640 1 --2- 172.21.15.16:0/2436963693 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3f94070df0 0x7f3f94149b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:01.665 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.659+0000 7f3f98af1640 1 -- 172.21.15.16:0/2436963693 >> 172.21.15.16:0/2436963693 conn(0x7f3f94069720 msgr2=0x7f3f940af2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:43:01.665 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.659+0000 7f3f98af1640 1 -- 172.21.15.16:0/2436963693 shutdown_connections 2023-05-22T15:43:01.665 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:01.659+0000 7f3f98af1640 1 -- 172.21.15.16:0/2436963693 wait complete. 2023-05-22T15:43:01.678 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-05-22T15:43:01.679 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-05-22T15:43:01.679 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi016.front.sepia.ceph.com 2023-05-22T15:43:01.679 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-05-22T15:43:01.680 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-05-22T15:43:01.681 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-05-22T15:43:01.681 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-05-22T15:43:01.681 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-05-22T15:43:01.681 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-05-22T15:43:01.682 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:43:01.682 DEBUG:teuthology.orchestra.run.smithi016:> ip addr 2023-05-22T15:43:01.689 INFO:teuthology.orchestra.run.smithi016.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-05-22T15:43:01.689 INFO:teuthology.orchestra.run.smithi016.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-05-22T15:43:01.690 INFO:teuthology.orchestra.run.smithi016.stdout: inet 127.0.0.1/8 scope host lo 2023-05-22T15:43:01.690 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-05-22T15:43:01.690 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 ::1/128 scope host 2023-05-22T15:43:01.691 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-05-22T15:43:01.691 INFO:teuthology.orchestra.run.smithi016.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-05-22T15:43:01.691 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:6b:fd:e0 brd ff:ff:ff:ff:ff:ff 2023-05-22T15:43:01.691 INFO:teuthology.orchestra.run.smithi016.stdout: altname enp7s0f0 2023-05-22T15:43:01.691 INFO:teuthology.orchestra.run.smithi016.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-05-22T15:43:01.692 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:6b:fd:e1 brd ff:ff:ff:ff:ff:ff 2023-05-22T15:43:01.692 INFO:teuthology.orchestra.run.smithi016.stdout: altname enp7s0f1 2023-05-22T15:43:01.692 INFO:teuthology.orchestra.run.smithi016.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2023-05-22T15:43:01.692 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:bd:15:64 brd ff:ff:ff:ff:ff:ff 2023-05-22T15:43:01.693 INFO:teuthology.orchestra.run.smithi016.stdout: altname enp1s0f0 2023-05-22T15:43:01.693 INFO:teuthology.orchestra.run.smithi016.stdout: inet 172.21.15.16/20 brd 172.21.15.255 scope global dynamic ens1f0 2023-05-22T15:43:01.693 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft 41535sec preferred_lft 41535sec 2023-05-22T15:43:01.693 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::ec4:7aff:febd:1564/64 scope link 2023-05-22T15:43:01.694 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-05-22T15:43:01.694 INFO:teuthology.orchestra.run.smithi016.stdout:5: ens1f1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-05-22T15:43:01.694 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:bd:15:65 brd ff:ff:ff:ff:ff:ff 2023-05-22T15:43:01.694 INFO:teuthology.orchestra.run.smithi016.stdout: altname enp1s0f1 2023-05-22T15:43:01.695 INFO:teuthology.orchestra.run.smithi016.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2023-05-22T15:43:01.695 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 02:42:1f:b8:98:0a brd ff:ff:ff:ff:ff:ff 2023-05-22T15:43:01.695 INFO:teuthology.orchestra.run.smithi016.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2023-05-22T15:43:01.695 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-05-22T15:43:01.696 INFO:teuthology.orchestra.run.smithi016.stdout:9: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-05-22T15:43:01.696 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 5e:1f:84:6d:79:ae brd ff:ff:ff:ff:ff:ff 2023-05-22T15:43:01.696 INFO:teuthology.orchestra.run.smithi016.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-05-22T15:43:01.697 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-05-22T15:43:01.697 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::5c1f:84ff:fe6d:79ae/64 scope link 2023-05-22T15:43:01.698 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-05-22T15:43:01.698 INFO:teuthology.orchestra.run.smithi016.stdout:10: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-05-22T15:43:01.698 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0e:5c:f6:1f:82:2f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-05-22T15:43:01.698 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::c5c:f6ff:fe1f:822f/64 scope link 2023-05-22T15:43:01.699 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-05-22T15:43:01.699 INFO:teuthology.orchestra.run.smithi016.stdout:11: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-05-22T15:43:01.699 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether c6:a6:3d:69:d0:db brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-05-22T15:43:01.699 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::c4a6:3dff:fe69:d0db/64 scope link 2023-05-22T15:43:01.700 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-05-22T15:43:01.700 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:43:01.701 DEBUG:teuthology.orchestra.run.smithi016:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-05-22T15:43:01.740 INFO:teuthology.orchestra.run.smithi016.stdout:1 2023-05-22T15:43:01.741 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:43:01.742 DEBUG:teuthology.orchestra.run.smithi016:> route 2023-05-22T15:43:01.788 INFO:teuthology.orchestra.run.smithi016.stdout:Kernel IP routing table 2023-05-22T15:43:01.789 INFO:teuthology.orchestra.run.smithi016.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-05-22T15:43:01.789 INFO:teuthology.orchestra.run.smithi016.stdout:default _gateway 0.0.0.0 UG 0 0 0 ens1f0 2023-05-22T15:43:01.790 INFO:teuthology.orchestra.run.smithi016.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2023-05-22T15:43:01.790 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 0 0 0 ens1f0 2023-05-22T15:43:01.790 INFO:teuthology.orchestra.run.smithi016.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-05-22T15:43:01.791 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:43:01.792 DEBUG:teuthology.orchestra.run.smithi016:> (cd / && exec bash -c ' 2023-05-22T15:43:01.792 DEBUG:teuthology.orchestra.run.smithi016:> set -e 2023-05-22T15:43:01.793 DEBUG:teuthology.orchestra.run.smithi016:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2023-05-22T15:43:01.793 DEBUG:teuthology.orchestra.run.smithi016:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2023-05-22T15:43:01.794 DEBUG:teuthology.orchestra.run.smithi016:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2023-05-22T15:43:01.795 DEBUG:teuthology.orchestra.run.smithi016:> ') 2023-05-22T15:43:01.863 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:43:01.864 DEBUG:teuthology.orchestra.run.smithi016:> ip netns list 2023-05-22T15:43:01.911 INFO:teuthology.orchestra.run.smithi016.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-05-22T15:43:01.911 INFO:teuthology.orchestra.run.smithi016.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-05-22T15:43:01.912 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-05-22T15:43:01.912 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:43:01.912 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip link set brx.0 up 2023-05-22T15:43:01.967 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2023-05-22T15:43:01.968 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-05-22T15:43:01.968 DEBUG:teuthology.orchestra.run.smithi016:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-05-22T15:43:02.015 INFO:teuthology.orchestra.run.smithi016.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-05-22T15:43:02.015 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-05-22T15:43:02.015 DEBUG:teuthology.orchestra.run.smithi016:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-05-22T15:43:02.062 DEBUG:teuthology.orchestra.run.smithi016:> sudo modprobe fuse 2023-05-22T15:43:02.120 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-05-22T15:43:02.121 DEBUG:teuthology.orchestra.run.smithi016:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-05-22T15:43:02.175 INFO:teuthology.orchestra.run.smithi016.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-05-22T15:43:02.176 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-05-22T15:43:02.176 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:43:02.177 DEBUG:teuthology.orchestra.run.smithi016:> ls /sys/fs/fuse/connections 2023-05-22T15:43:02.223 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2023-05-22T15:43:02.223 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2023-05-22T15:43:02.268 DEBUG:teuthology.orchestra.run.smithi016:> sudo modprobe fuse 2023-05-22T15:43:02.284 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-05-22T15:43:02.285 DEBUG:teuthology.orchestra.run.smithi016:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-05-22T15:43:02.337 INFO:teuthology.orchestra.run.smithi016.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-05-22T15:43:02.338 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-05-22T15:43:02.339 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:43:02.339 DEBUG:teuthology.orchestra.run.smithi016:> ls /sys/fs/fuse/connections 2023-05-22T15:43:02.352 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi016.stderr:2023-05-22T15:43:02.347+0000 7f64b898a580 -1 init, newargv = 0x55730f068ae0 newargc=15 2023-05-22T15:43:02.352 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi016.stderr:ceph-fuse[57151]: starting ceph client 2023-05-22T15:43:02.359 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi016.stderr:ceph-fuse[57151]: starting fuse 2023-05-22T15:43:03.457 DEBUG:teuthology.orchestra.run.smithi016:> sudo modprobe fuse 2023-05-22T15:43:03.473 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-05-22T15:43:03.474 DEBUG:teuthology.orchestra.run.smithi016:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-05-22T15:43:03.527 INFO:teuthology.orchestra.run.smithi016.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-05-22T15:43:03.528 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-05-22T15:43:03.529 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:43:03.529 DEBUG:teuthology.orchestra.run.smithi016:> ls /sys/fs/fuse/connections 2023-05-22T15:43:03.575 INFO:teuthology.orchestra.run.smithi016.stdout:45 2023-05-22T15:43:03.576 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [45] 2023-05-22T15:43:03.576 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2023-05-22T15:43:03.577 DEBUG:teuthology.orchestra.run.smithi016:> import glob 2023-05-22T15:43:03.577 DEBUG:teuthology.orchestra.run.smithi016:> import re 2023-05-22T15:43:03.577 DEBUG:teuthology.orchestra.run.smithi016:> import os 2023-05-22T15:43:03.578 DEBUG:teuthology.orchestra.run.smithi016:> import subprocess 2023-05-22T15:43:03.578 DEBUG:teuthology.orchestra.run.smithi016:> 2023-05-22T15:43:03.579 DEBUG:teuthology.orchestra.run.smithi016:> def _find_admin_socket(client_name): 2023-05-22T15:43:03.579 DEBUG:teuthology.orchestra.run.smithi016:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2023-05-22T15:43:03.579 DEBUG:teuthology.orchestra.run.smithi016:> files = glob.glob(asok_path) 2023-05-22T15:43:03.580 DEBUG:teuthology.orchestra.run.smithi016:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2023-05-22T15:43:03.580 DEBUG:teuthology.orchestra.run.smithi016:> 2023-05-22T15:43:03.580 DEBUG:teuthology.orchestra.run.smithi016:> # Given a non-glob path, it better be there 2023-05-22T15:43:03.581 DEBUG:teuthology.orchestra.run.smithi016:> if "*" not in asok_path: 2023-05-22T15:43:03.581 DEBUG:teuthology.orchestra.run.smithi016:> assert(len(files) == 1) 2023-05-22T15:43:03.581 DEBUG:teuthology.orchestra.run.smithi016:> return files[0] 2023-05-22T15:43:03.581 DEBUG:teuthology.orchestra.run.smithi016:> 2023-05-22T15:43:03.582 DEBUG:teuthology.orchestra.run.smithi016:> for f in files: 2023-05-22T15:43:03.582 DEBUG:teuthology.orchestra.run.smithi016:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-05-22T15:43:03.582 DEBUG:teuthology.orchestra.run.smithi016:> if os.path.exists("/proc/{0}".format(pid)): 2023-05-22T15:43:03.583 DEBUG:teuthology.orchestra.run.smithi016:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-05-22T15:43:03.583 DEBUG:teuthology.orchestra.run.smithi016:> contents = proc_f.read() 2023-05-22T15:43:03.583 DEBUG:teuthology.orchestra.run.smithi016:> if mountpoint in contents: 2023-05-22T15:43:03.583 DEBUG:teuthology.orchestra.run.smithi016:> return f 2023-05-22T15:43:03.584 DEBUG:teuthology.orchestra.run.smithi016:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-05-22T15:43:03.584 DEBUG:teuthology.orchestra.run.smithi016:> 2023-05-22T15:43:03.584 DEBUG:teuthology.orchestra.run.smithi016:> print(_find_admin_socket("client.0")) 2023-05-22T15:43:03.585 DEBUG:teuthology.orchestra.run.smithi016:> ' 2023-05-22T15:43:03.701 INFO:teuthology.orchestra.run.smithi016.stdout:/var/run/ceph/ceph-client.0.57151.asok 2023-05-22T15:43:03.883 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.57151.asok 2023-05-22T15:43:03.883 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:43:03.883 DEBUG:teuthology.orchestra.run.smithi016:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.57151.asok status 2023-05-22T15:43:04.046 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-05-22T15:43:04.046 INFO:teuthology.orchestra.run.smithi016.stdout: "metadata": { 2023-05-22T15:43:04.047 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph_sha1": "9c9ba4df22024d9d8d6cf9f98a4b96751fc78d0c", 2023-05-22T15:43:04.047 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph_version": "ceph version 18.0.0-3192-g9c9ba4df (9c9ba4df22024d9d8d6cf9f98a4b96751fc78d0c) reef (dev)", 2023-05-22T15:43:04.047 INFO:teuthology.orchestra.run.smithi016.stdout: "entity_id": "0", 2023-05-22T15:43:04.047 INFO:teuthology.orchestra.run.smithi016.stdout: "hostname": "smithi016", 2023-05-22T15:43:04.047 INFO:teuthology.orchestra.run.smithi016.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2023-05-22T15:43:04.047 INFO:teuthology.orchestra.run.smithi016.stdout: "pid": "57151", 2023-05-22T15:43:04.047 INFO:teuthology.orchestra.run.smithi016.stdout: "root": "/" 2023-05-22T15:43:04.048 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:43:04.048 INFO:teuthology.orchestra.run.smithi016.stdout: "dentry_count": 0, 2023-05-22T15:43:04.048 INFO:teuthology.orchestra.run.smithi016.stdout: "dentry_pinned_count": 0, 2023-05-22T15:43:04.048 INFO:teuthology.orchestra.run.smithi016.stdout: "id": 7689, 2023-05-22T15:43:04.048 INFO:teuthology.orchestra.run.smithi016.stdout: "inst": { 2023-05-22T15:43:04.048 INFO:teuthology.orchestra.run.smithi016.stdout: "name": { 2023-05-22T15:43:04.048 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "client", 2023-05-22T15:43:04.049 INFO:teuthology.orchestra.run.smithi016.stdout: "num": 7689 2023-05-22T15:43:04.049 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:43:04.049 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": { 2023-05-22T15:43:04.049 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-05-22T15:43:04.049 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "192.168.0.1:0", 2023-05-22T15:43:04.049 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 241459281 2023-05-22T15:43:04.049 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T15:43:04.049 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:43:04.050 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": { 2023-05-22T15:43:04.050 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-05-22T15:43:04.050 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "192.168.0.1:0", 2023-05-22T15:43:04.050 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 241459281 2023-05-22T15:43:04.050 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:43:04.050 INFO:teuthology.orchestra.run.smithi016.stdout: "inst_str": "client.7689 192.168.0.1:0/241459281", 2023-05-22T15:43:04.050 INFO:teuthology.orchestra.run.smithi016.stdout: "addr_str": "192.168.0.1:0/241459281", 2023-05-22T15:43:04.051 INFO:teuthology.orchestra.run.smithi016.stdout: "inode_count": 1, 2023-05-22T15:43:04.051 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_epoch": 36, 2023-05-22T15:43:04.051 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_epoch": 104, 2023-05-22T15:43:04.052 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_epoch_barrier": 0, 2023-05-22T15:43:04.052 INFO:teuthology.orchestra.run.smithi016.stdout: "blocklisted": false, 2023-05-22T15:43:04.052 INFO:teuthology.orchestra.run.smithi016.stdout: "fs_name": "cephfs" 2023-05-22T15:43:04.052 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-05-22T15:43:04.060 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:43:04.060 DEBUG:teuthology.orchestra.run.smithi016:> stat --file-system '--printf=%T 2023-05-22T15:43:04.061 DEBUG:teuthology.orchestra.run.smithi016:> ' -- /home/ubuntu/cephtest/mnt.0 2023-05-22T15:43:04.067 INFO:teuthology.orchestra.run.smithi016.stdout:fuseblk 2023-05-22T15:43:04.067 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2023-05-22T15:43:04.068 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:43:04.068 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-05-22T15:43:04.124 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2023-05-22T15:43:04.299 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:04.295+0000 7f226005b640 1 Processor -- start 2023-05-22T15:43:04.300 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:04.295+0000 7f226005b640 1 -- start start 2023-05-22T15:43:04.300 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:04.295+0000 7f226005b640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f225815cfa0 0x7f225806db00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:04.300 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:04.295+0000 7f226005b640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f225806e740 0x7f225806ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:04.301 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:04.295+0000 7f226005b640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f225806e040 con 0x7f22580cf870 2023-05-22T15:43:04.301 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:04.295+0000 7f226005b640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22580cb9f0 con 0x7f225815cfa0 2023-05-22T15:43:04.301 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:04.295+0000 7f226005b640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f225806f100 con 0x7f225806e740 2023-05-22T15:43:04.302 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:04.295+0000 7f225e5f8640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f225806e740 0x7f225806ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:04.302 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:04.295+0000 7f225ddf7640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f225815cfa0 0x7f225806db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:04.302 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:04.295+0000 7f225e5f8640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f225806e740 0x7f225806ebc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3301/0 says I am v2:172.21.15.16:43764/0 (socket says 172.21.15.16:43764) 2023-05-22T15:43:04.302 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:04.295+0000 7f225d5f6640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f22580cf870 0x7f22580b1fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.16:58982/0 (socket says 172.21.15.16:58982) 2023-05-22T15:43:04.302 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:04.295+0000 7f225ddf7640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f225815cfa0 0x7f225806db00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.16:33226/0 (socket says 172.21.15.16:33226) 2023-05-22T15:43:04.303 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:04.295+0000 7f225d5f6640 1 -- 172.21.15.16:0/122470783 learned_addr learned my addr 172.21.15.16:0/122470783 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:43:04.303 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:04.295+0000 7f225ddf7640 1 -- 172.21.15.16:0/122470783 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f225806e740 msgr2=0x7f225806ebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:43:04.303 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:04.295+0000 7f225ddf7640 1 --2- 172.21.15.16:0/122470783 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f225806e740 0x7f225806ebc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:04.303 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:04.295+0000 7f225ddf7640 1 -- 172.21.15.16:0/122470783 >> v1:172.21.15.62:6789/0 conn(0x7f22580cf870 legacy=0x7f22580b1fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:04.303 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:04.295+0000 7f225ddf7640 1 -- 172.21.15.16:0/122470783 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f225806e490 con 0x7f225815cfa0 2023-05-22T15:43:04.303 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:04.295+0000 7f225e5f8640 1 --2- 172.21.15.16:0/122470783 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f225806e740 0x7f225806ebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-05-22T15:43:04.304 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:04.299+0000 7f225ddf7640 1 --2- 172.21.15.16:0/122470783 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f225815cfa0 0x7f225806db00 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f2250009d90 tx=0x7f225001c920 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9ddb12abbbe3d72f server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:43:04.304 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:04.299+0000 7f225cdf5640 1 -- 172.21.15.16:0/122470783 <== mon.1 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22500a59e0 con 0x7f225815cfa0 2023-05-22T15:43:04.304 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:04.299+0000 7f225cdf5640 1 -- 172.21.15.16:0/122470783 <== mon.1 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22500a5b40 con 0x7f225815cfa0 2023-05-22T15:43:04.304 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:04.299+0000 7f225cdf5640 1 -- 172.21.15.16:0/122470783 <== mon.1 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22500ad440 con 0x7f225815cfa0 2023-05-22T15:43:04.305 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:04.299+0000 7f226005b640 1 -- 172.21.15.16:0/122470783 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f225815cfa0 msgr2=0x7f225806db00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:04.305 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:04.299+0000 7f226005b640 1 --2- 172.21.15.16:0/122470783 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f225815cfa0 0x7f225806db00 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f2250009d90 tx=0x7f225001c920 comp rx=0 tx=0).stop 2023-05-22T15:43:04.305 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:04.299+0000 7f226005b640 1 -- 172.21.15.16:0/122470783 shutdown_connections 2023-05-22T15:43:04.306 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:04.299+0000 7f226005b640 1 --2- 172.21.15.16:0/122470783 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f225806e740 0x7f225806ebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:04.306 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:04.299+0000 7f226005b640 1 --2- 172.21.15.16:0/122470783 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f225815cfa0 0x7f225806db00 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:04.306 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:04.299+0000 7f226005b640 1 -- 172.21.15.16:0/122470783 >> 172.21.15.16:0/122470783 conn(0x7f22580cc870 msgr2=0x7f22580cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:43:04.306 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:04.299+0000 7f226005b640 1 -- 172.21.15.16:0/122470783 shutdown_connections 2023-05-22T15:43:04.306 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:04.299+0000 7f226005b640 1 -- 172.21.15.16:0/122470783 wait complete. 2023-05-22T15:43:04.308 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:04.303+0000 7f226005b640 1 Processor -- start 2023-05-22T15:43:04.308 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:04.303+0000 7f226005b640 1 -- start start 2023-05-22T15:43:04.308 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:04.303+0000 7f226005b640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f225806e740 0x7f22581576a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:04.308 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:04.303+0000 7f226005b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f22580cf870 0x7f2258149940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:04.308 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:04.303+0000 7f226005b640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f225815cfa0 0x7f22580c6c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:04.309 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:04.303+0000 7f226005b640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2258149e80 con 0x7f22580cf870 2023-05-22T15:43:04.309 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:04.303+0000 7f226005b640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22580c89a0 con 0x7f225815cfa0 2023-05-22T15:43:04.309 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:04.303+0000 7f226005b640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22580c8c80 con 0x7f225806e740 2023-05-22T15:43:04.309 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:04.303+0000 7f225d5f6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f22580cf870 0x7f2258149940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:04.309 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:04.303+0000 7f225d5f6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f22580cf870 0x7f2258149940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49908/0 (socket says 172.21.15.16:49908) 2023-05-22T15:43:04.309 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:04.303+0000 7f225d5f6640 1 -- 172.21.15.16:0/3538505657 learned_addr learned my addr 172.21.15.16:0/3538505657 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:43:04.309 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:04.303+0000 7f225ddf7640 1 --2- 172.21.15.16:0/3538505657 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f225806e740 0x7f22581576a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:04.310 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:04.303+0000 7f225e5f8640 1 --2- 172.21.15.16:0/3538505657 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f225815cfa0 0x7f22580c6c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:04.310 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:04.303+0000 7f225d5f6640 1 -- 172.21.15.16:0/3538505657 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f225806e740 msgr2=0x7f22581576a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:04.310 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:04.303+0000 7f225d5f6640 1 --2- 172.21.15.16:0/3538505657 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f225806e740 0x7f22581576a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:04.310 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:04.303+0000 7f225d5f6640 1 -- 172.21.15.16:0/3538505657 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f225815cfa0 msgr2=0x7f22580c6c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:04.310 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:04.303+0000 7f225d5f6640 1 --2- 172.21.15.16:0/3538505657 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f225815cfa0 0x7f22580c6c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:04.310 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:04.303+0000 7f225d5f6640 1 -- 172.21.15.16:0/3538505657 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2248009420 con 0x7f22580cf870 2023-05-22T15:43:04.310 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:04.303+0000 7f225e5f8640 1 --2- 172.21.15.16:0/3538505657 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f225815cfa0 0x7f22580c6c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T15:43:04.310 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:04.303+0000 7f225d5f6640 1 --2- 172.21.15.16:0/3538505657 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f22580cf870 0x7f2258149940 secure :-1 s=READY pgs=1308 cs=0 l=1 rev1=1 crypto rx=0x7f225400c420 tx=0x7f225400c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:43:04.311 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:04.303+0000 7f224e7fc640 1 -- 172.21.15.16:0/3538505657 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2254005cf0 con 0x7f22580cf870 2023-05-22T15:43:04.311 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:04.303+0000 7f224e7fc640 1 -- 172.21.15.16:0/3538505657 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2254005e50 con 0x7f22580cf870 2023-05-22T15:43:04.311 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:04.303+0000 7f226005b640 1 -- 172.21.15.16:0/3538505657 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2250009a70 con 0x7f22580cf870 2023-05-22T15:43:04.311 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:04.303+0000 7f224e7fc640 1 -- 172.21.15.16:0/3538505657 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22540072f0 con 0x7f22580cf870 2023-05-22T15:43:04.311 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:04.303+0000 7f226005b640 1 -- 172.21.15.16:0/3538505657 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22580cb610 con 0x7f22580cf870 2023-05-22T15:43:04.312 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:04.303+0000 7f226005b640 1 -- 172.21.15.16:0/3538505657 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22580cb330 con 0x7f22580cf870 2023-05-22T15:43:04.312 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:04.303+0000 7f224e7fc640 1 -- 172.21.15.16:0/3538505657 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 34) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f2254007450 con 0x7f22580cf870 2023-05-22T15:43:04.312 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:04.303+0000 7f224e7fc640 1 --2- 172.21.15.16:0/3538505657 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f22440654a0 0x7f2244067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:04.313 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:04.307+0000 7f224e7fc640 1 -- 172.21.15.16:0/3538505657 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f2254081f30 con 0x7f22580cf870 2023-05-22T15:43:04.313 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:04.307+0000 7f225ddf7640 1 --2- 172.21.15.16:0/3538505657 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f22440654a0 0x7f2244067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:04.313 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:04.307+0000 7f225ddf7640 1 --2- 172.21.15.16:0/3538505657 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f22440654a0 0x7f2244067950 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f22500af000 tx=0x7f22500a7070 comp rx=0 tx=0).ready entity=mgr.6436 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:43:04.314 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:04.311+0000 7f224e7fc640 1 -- 172.21.15.16:0/3538505657 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f225404f930 con 0x7f22580cf870 2023-05-22T15:43:04.490 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:04.487+0000 7f226005b640 1 -- 172.21.15.16:0/3538505657 --> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f22580c8df0 con 0x7f22440654a0 2023-05-22T15:43:04.492 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:04.487+0000 7f224e7fc640 1 -- 172.21.15.16:0/3538505657 <== mgr.6436 v2:172.21.15.16:6832/36921 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f22580c8df0 con 0x7f22440654a0 2023-05-22T15:43:04.492 INFO:teuthology.orchestra.run.smithi016.stdout:[ 2023-05-22T15:43:04.493 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T15:43:04.493 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "cephfs" 2023-05-22T15:43:04.493 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T15:43:04.493 INFO:teuthology.orchestra.run.smithi016.stdout:] 2023-05-22T15:43:04.494 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:04.491+0000 7f226005b640 1 -- 172.21.15.16:0/3538505657 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f22440654a0 msgr2=0x7f2244067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:04.494 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:04.491+0000 7f226005b640 1 --2- 172.21.15.16:0/3538505657 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f22440654a0 0x7f2244067950 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f22500af000 tx=0x7f22500a7070 comp rx=0 tx=0).stop 2023-05-22T15:43:04.494 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:04.491+0000 7f226005b640 1 -- 172.21.15.16:0/3538505657 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f22580cf870 msgr2=0x7f2258149940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:04.494 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:04.491+0000 7f226005b640 1 --2- 172.21.15.16:0/3538505657 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f22580cf870 0x7f2258149940 secure :-1 s=READY pgs=1308 cs=0 l=1 rev1=1 crypto rx=0x7f225400c420 tx=0x7f225400c8e0 comp rx=0 tx=0).stop 2023-05-22T15:43:04.495 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:04.491+0000 7f226005b640 1 -- 172.21.15.16:0/3538505657 shutdown_connections 2023-05-22T15:43:04.495 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:04.491+0000 7f226005b640 1 --2- 172.21.15.16:0/3538505657 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f225815cfa0 0x7f22580c6c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:04.496 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:04.491+0000 7f226005b640 1 --2- 172.21.15.16:0/3538505657 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f22580cf870 0x7f2258149940 unknown :-1 s=CLOSED pgs=1308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:04.496 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:04.491+0000 7f226005b640 1 --2- 172.21.15.16:0/3538505657 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f22440654a0 0x7f2244067950 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:04.496 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:04.491+0000 7f226005b640 1 --2- 172.21.15.16:0/3538505657 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f225806e740 0x7f22581576a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:04.496 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:04.491+0000 7f226005b640 1 -- 172.21.15.16:0/3538505657 >> 172.21.15.16:0/3538505657 conn(0x7f22580cc870 msgr2=0x7f22580afde0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:43:04.496 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:04.491+0000 7f226005b640 1 -- 172.21.15.16:0/3538505657 shutdown_connections 2023-05-22T15:43:04.496 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:04.491+0000 7f226005b640 1 -- 172.21.15.16:0/3538505657 wait complete. 2023-05-22T15:43:04.511 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2023-05-22T15:43:04.635 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:04.631+0000 7f922794b640 1 Processor -- start 2023-05-22T15:43:04.635 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:04.631+0000 7f922794b640 1 -- start start 2023-05-22T15:43:04.637 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:04.631+0000 7f922794b640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9220153fe0 0x7f922006d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:04.637 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:04.631+0000 7f922794b640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f92200cae20 0x7f922006fef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:04.637 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:04.631+0000 7f922794b640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9220069720 con 0x7f9220070430 2023-05-22T15:43:04.637 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:04.631+0000 7f922794b640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92200708d0 con 0x7f92200cae20 2023-05-22T15:43:04.637 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:04.631+0000 7f922794b640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92201543b0 con 0x7f9220153fe0 2023-05-22T15:43:04.637 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:04.631+0000 7f9225ee8640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f9220070430 0x7f92200ad410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:48788/0 (socket says 172.21.15.16:48788) 2023-05-22T15:43:04.638 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:04.631+0000 7f9225ee8640 1 -- 172.21.15.16:0/3471875777 learned_addr learned my addr 172.21.15.16:0/3471875777 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:43:04.638 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:04.631+0000 7f92256e7640 1 --2- 172.21.15.16:0/3471875777 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9220153fe0 0x7f922006d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:04.638 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:04.631+0000 7f9224ee6640 1 --2- 172.21.15.16:0/3471875777 >> v2:172.21.15.62:3300/0 conn(0x7f92200cae20 0x7f922006fef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:04.638 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:04.631+0000 7f9217fff640 1 -- 172.21.15.16:0/3471875777 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3550610731 0 0) 0x7f9220069720 con 0x7f9220070430 2023-05-22T15:43:04.638 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:04.631+0000 7f9217fff640 1 -- 172.21.15.16:0/3471875777 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f920c003530 con 0x7f9220070430 2023-05-22T15:43:04.640 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:04.631+0000 7f9217fff640 1 -- 172.21.15.16:0/3471875777 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 273579334 0 0) 0x7f920c003530 con 0x7f9220070430 2023-05-22T15:43:04.640 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:04.631+0000 7f9217fff640 1 -- 172.21.15.16:0/3471875777 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9220153fe0 msgr2=0x7f922006d3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:43:04.640 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:04.631+0000 7f9217fff640 1 --2- 172.21.15.16:0/3471875777 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9220153fe0 0x7f922006d3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:04.640 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:04.631+0000 7f9217fff640 1 -- 172.21.15.16:0/3471875777 >> v2:172.21.15.62:3300/0 conn(0x7f92200cae20 msgr2=0x7f922006fef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:43:04.641 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:04.631+0000 7f9217fff640 1 --2- 172.21.15.16:0/3471875777 >> v2:172.21.15.62:3300/0 conn(0x7f92200cae20 0x7f922006fef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:04.641 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:04.631+0000 7f9217fff640 1 -- 172.21.15.16:0/3471875777 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f922015e360 con 0x7f9220070430 2023-05-22T15:43:04.641 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:04.631+0000 7f9217fff640 1 -- 172.21.15.16:0/3471875777 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7f9218001a00 con 0x7f9220070430 2023-05-22T15:43:04.641 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:04.631+0000 7f9217fff640 1 -- 172.21.15.16:0/3471875777 >> v1:172.21.15.16:6789/0 conn(0x7f9220070430 legacy=0x7f92200ad410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:04.641 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:04.631+0000 7f9217fff640 1 --2- 172.21.15.16:0/3471875777 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f920c003980 0x7f920c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:04.641 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:04.631+0000 7f9217fff640 1 --2- 172.21.15.16:0/3471875777 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f920c0043b0 0x7f920c004870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:04.641 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:04.631+0000 7f9217fff640 1 --2- 172.21.15.16:0/3471875777 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9220153fe0 0x7f920c004f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:04.642 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:04.631+0000 7f9217fff640 1 -- 172.21.15.16:0/3471875777 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92201543b0 con 0x7f920c003980 2023-05-22T15:43:04.642 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:04.631+0000 7f9217fff640 1 -- 172.21.15.16:0/3471875777 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f920c005680 con 0x7f920c0043b0 2023-05-22T15:43:04.642 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:04.631+0000 7f9217fff640 1 -- 172.21.15.16:0/3471875777 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f920c00ba10 con 0x7f9220153fe0 2023-05-22T15:43:04.642 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:04.631+0000 7f92256e7640 1 --2- 172.21.15.16:0/3471875777 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f920c003980 0x7f920c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:04.642 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:04.631+0000 7f9224ee6640 1 --2- 172.21.15.16:0/3471875777 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f920c0043b0 0x7f920c004870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:04.642 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:04.631+0000 7f92256e7640 1 -- 172.21.15.16:0/3471875777 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9220153fe0 msgr2=0x7f920c004f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:43:04.642 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:04.631+0000 7f9225ee8640 1 --2- 172.21.15.16:0/3471875777 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9220153fe0 0x7f920c004f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:04.643 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:04.631+0000 7f92256e7640 1 --2- 172.21.15.16:0/3471875777 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9220153fe0 0x7f920c004f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:04.643 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:04.631+0000 7f92256e7640 1 -- 172.21.15.16:0/3471875777 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f920c0043b0 msgr2=0x7f920c004870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:43:04.643 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:04.631+0000 7f92256e7640 1 --2- 172.21.15.16:0/3471875777 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f920c0043b0 0x7f920c004870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:04.643 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:04.631+0000 7f92256e7640 1 -- 172.21.15.16:0/3471875777 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f920c003530 con 0x7f920c003980 2023-05-22T15:43:04.643 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:04.631+0000 7f9224ee6640 1 --2- 172.21.15.16:0/3471875777 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f920c0043b0 0x7f920c004870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:43:04.643 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:04.635+0000 7f92256e7640 1 --2- 172.21.15.16:0/3471875777 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f920c003980 0x7f920c003d90 secure :-1 s=READY pgs=1310 cs=0 l=1 rev1=1 crypto rx=0x7f921c009ae0 tx=0x7f921c01ca20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77b3eaadab1aa354 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:43:04.644 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:04.635+0000 7f9217fff640 1 -- 172.21.15.16:0/3471875777 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f921c0a5570 con 0x7f920c003980 2023-05-22T15:43:04.644 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:04.635+0000 7f9217fff640 1 -- 172.21.15.16:0/3471875777 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f921c0a56d0 con 0x7f920c003980 2023-05-22T15:43:04.644 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:04.635+0000 7f9217fff640 1 -- 172.21.15.16:0/3471875777 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f921c0a5a80 con 0x7f920c003980 2023-05-22T15:43:04.647 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:04.635+0000 7f922794b640 1 -- 172.21.15.16:0/3471875777 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f920c003980 msgr2=0x7f920c003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:04.647 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:04.635+0000 7f922794b640 1 --2- 172.21.15.16:0/3471875777 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f920c003980 0x7f920c003d90 secure :-1 s=READY pgs=1310 cs=0 l=1 rev1=1 crypto rx=0x7f921c009ae0 tx=0x7f921c01ca20 comp rx=0 tx=0).stop 2023-05-22T15:43:04.647 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:04.635+0000 7f9225ee8640 1 -- 172.21.15.16:0/3471875777 reap_dead start 2023-05-22T15:43:04.647 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:04.635+0000 7f922794b640 1 -- 172.21.15.16:0/3471875777 shutdown_connections 2023-05-22T15:43:04.647 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:04.635+0000 7f922794b640 1 -- 172.21.15.16:0/3471875777 >> 172.21.15.16:0/3471875777 conn(0x7f92200b1110 msgr2=0x7f92200ce830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:43:04.648 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:04.635+0000 7f922794b640 1 -- 172.21.15.16:0/3471875777 shutdown_connections 2023-05-22T15:43:04.648 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:04.635+0000 7f922794b640 1 -- 172.21.15.16:0/3471875777 wait complete. 2023-05-22T15:43:04.648 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:04.635+0000 7f922794b640 1 Processor -- start 2023-05-22T15:43:04.648 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:04.639+0000 7f922794b640 1 -- start start 2023-05-22T15:43:04.648 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:04.639+0000 7f922794b640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f920c003980 0x7f9220142210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:04.648 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:04.639+0000 7f922794b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f92200c7b40 0x7f92200c7fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:04.649 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:04.639+0000 7f922794b640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9220156760 0x7f9220156c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:04.649 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:04.639+0000 7f922794b640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92200c84f0 con 0x7f92200c7b40 2023-05-22T15:43:04.649 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:04.639+0000 7f922794b640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9220148b20 con 0x7f920c003980 2023-05-22T15:43:04.649 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:04.639+0000 7f922794b640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9220148fb0 con 0x7f9220156760 2023-05-22T15:43:04.649 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:04.639+0000 7f9224ee6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f92200c7b40 0x7f92200c7fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:04.649 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:04.639+0000 7f9224ee6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f92200c7b40 0x7f92200c7fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49934/0 (socket says 172.21.15.16:49934) 2023-05-22T15:43:04.649 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:04.639+0000 7f9224ee6640 1 -- 172.21.15.16:0/2316694719 learned_addr learned my addr 172.21.15.16:0/2316694719 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:43:04.650 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:04.639+0000 7f92256e7640 1 --2- 172.21.15.16:0/2316694719 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f920c003980 0x7f9220142210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:04.650 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:04.639+0000 7f9225ee8640 1 --2- 172.21.15.16:0/2316694719 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9220156760 0x7f9220156c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:04.650 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:04.639+0000 7f9224ee6640 1 -- 172.21.15.16:0/2316694719 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9220156760 msgr2=0x7f9220156c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:04.650 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:04.639+0000 7f9224ee6640 1 --2- 172.21.15.16:0/2316694719 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9220156760 0x7f9220156c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:04.650 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:04.639+0000 7f9224ee6640 1 -- 172.21.15.16:0/2316694719 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f920c003980 msgr2=0x7f9220142210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:04.650 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:04.639+0000 7f9224ee6640 1 --2- 172.21.15.16:0/2316694719 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f920c003980 0x7f9220142210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:04.650 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:04.639+0000 7f9224ee6640 1 -- 172.21.15.16:0/2316694719 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f922015de40 con 0x7f92200c7b40 2023-05-22T15:43:04.651 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:04.639+0000 7f9225ee8640 1 --2- 172.21.15.16:0/2316694719 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9220156760 0x7f9220156c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:43:04.651 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:04.639+0000 7f9224ee6640 1 --2- 172.21.15.16:0/2316694719 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f92200c7b40 0x7f92200c7fb0 secure :-1 s=READY pgs=1311 cs=0 l=1 rev1=1 crypto rx=0x7f92100109b0 tx=0x7f9210010e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:43:04.651 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:04.639+0000 7f9215ffb640 1 -- 172.21.15.16:0/2316694719 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f921000ec20 con 0x7f92200c7b40 2023-05-22T15:43:04.651 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:04.639+0000 7f9215ffb640 1 -- 172.21.15.16:0/2316694719 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f921000ed80 con 0x7f92200c7b40 2023-05-22T15:43:04.651 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:04.639+0000 7f922794b640 1 -- 172.21.15.16:0/2316694719 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f922015db60 con 0x7f92200c7b40 2023-05-22T15:43:04.651 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:04.639+0000 7f9215ffb640 1 -- 172.21.15.16:0/2316694719 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9210004220 con 0x7f92200c7b40 2023-05-22T15:43:04.651 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:04.639+0000 7f922794b640 1 -- 172.21.15.16:0/2316694719 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f922006dd20 con 0x7f92200c7b40 2023-05-22T15:43:04.652 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:04.639+0000 7f922794b640 1 -- 172.21.15.16:0/2316694719 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f922004dbe0 con 0x7f92200c7b40 2023-05-22T15:43:04.652 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:04.639+0000 7f9215ffb640 1 -- 172.21.15.16:0/2316694719 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 34) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f9210004380 con 0x7f92200c7b40 2023-05-22T15:43:04.652 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:04.639+0000 7f9215ffb640 1 --2- 172.21.15.16:0/2316694719 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f92040654a0 0x7f9204067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:04.652 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:04.639+0000 7f9215ffb640 1 -- 172.21.15.16:0/2316694719 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f92100829d0 con 0x7f92200c7b40 2023-05-22T15:43:04.652 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:04.639+0000 7f92256e7640 1 --2- 172.21.15.16:0/2316694719 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f92040654a0 0x7f9204067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:04.652 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:04.643+0000 7f92256e7640 1 --2- 172.21.15.16:0/2316694719 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f92040654a0 0x7f9204067950 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f921c0af000 tx=0x7f921c0046d0 comp rx=0 tx=0).ready entity=mgr.6436 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:43:04.653 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:04.643+0000 7f9215ffb640 1 -- 172.21.15.16:0/2316694719 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f9210050490 con 0x7f92200c7b40 2023-05-22T15:43:04.845 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:04.839+0000 7f922794b640 1 -- 172.21.15.16:0/2316694719 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7f9220148c60 con 0x7f92200c7b40 2023-05-22T15:43:05.650 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:05.643+0000 7f9215ffb640 1 -- 172.21.15.16:0/2316694719 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mgrmap(e 35) v1 ==== 84349+0+0 (secure 0 0 0) 0x7f9210050860 con 0x7f92200c7b40 2023-05-22T15:43:05.652 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:05.647+0000 7f9215ffb640 1 -- 172.21.15.16:0/2316694719 <== mon.0 v2:172.21.15.16:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v35) v1 ==== 92+0+0 (secure 0 0 0) 0x7f9210050490 con 0x7f92200c7b40 2023-05-22T15:43:05.654 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:05.651+0000 7f922794b640 1 -- 172.21.15.16:0/2316694719 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f92040654a0 msgr2=0x7f9204067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:05.655 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:05.651+0000 7f922794b640 1 --2- 172.21.15.16:0/2316694719 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f92040654a0 0x7f9204067950 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f921c0af000 tx=0x7f921c0046d0 comp rx=0 tx=0).stop 2023-05-22T15:43:05.655 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:05.651+0000 7f922794b640 1 -- 172.21.15.16:0/2316694719 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f92200c7b40 msgr2=0x7f92200c7fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:05.655 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:05.651+0000 7f922794b640 1 --2- 172.21.15.16:0/2316694719 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f92200c7b40 0x7f92200c7fb0 secure :-1 s=READY pgs=1311 cs=0 l=1 rev1=1 crypto rx=0x7f92100109b0 tx=0x7f9210010e70 comp rx=0 tx=0).stop 2023-05-22T15:43:05.655 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:05.651+0000 7f922794b640 1 -- 172.21.15.16:0/2316694719 shutdown_connections 2023-05-22T15:43:05.655 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:05.651+0000 7f922794b640 1 --2- 172.21.15.16:0/2316694719 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f92040654a0 0x7f9204067950 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:05.655 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:05.651+0000 7f922794b640 1 --2- 172.21.15.16:0/2316694719 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9220156760 0x7f9220156c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:05.656 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:05.651+0000 7f922794b640 1 --2- 172.21.15.16:0/2316694719 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f92200c7b40 0x7f92200c7fb0 unknown :-1 s=CLOSED pgs=1311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:05.656 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:05.651+0000 7f922794b640 1 --2- 172.21.15.16:0/2316694719 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f920c003980 0x7f9220142210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:05.656 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:05.651+0000 7f922794b640 1 -- 172.21.15.16:0/2316694719 >> 172.21.15.16:0/2316694719 conn(0x7f92200b1110 msgr2=0x7f9220151c80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:43:05.656 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:05.651+0000 7f922794b640 1 -- 172.21.15.16:0/2316694719 shutdown_connections 2023-05-22T15:43:05.657 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:05.651+0000 7f922794b640 1 -- 172.21.15.16:0/2316694719 wait complete. 2023-05-22T15:43:05.675 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2023-05-22T15:43:05.774 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:43:05.771+0000 7f4053df2180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-05-22T15:43:05.778 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:43:05.775+0000 7f32d4c2d180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-05-22T15:43:05.791 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:05.787+0000 7f2407df8640 1 Processor -- start 2023-05-22T15:43:05.791 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:05.787+0000 7f2407df8640 1 -- start start 2023-05-22T15:43:05.792 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:05.787+0000 7f2407df8640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2400071640 0x7f2400160090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:05.792 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:05.787+0000 7f2407df8640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f24001605d0 0x7f240015c270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:05.792 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:05.787+0000 7f2407df8640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2400069ae0 con 0x7f24000ce480 2023-05-22T15:43:05.793 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:05.787+0000 7f2407df8640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f240015f990 con 0x7f24001605d0 2023-05-22T15:43:05.793 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:05.787+0000 7f2407df8640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2400071a90 con 0x7f2400071640 2023-05-22T15:43:05.793 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:05.787+0000 7f2405393640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2400071640 0x7f2400160090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:05.793 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:05.787+0000 7f2405393640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2400071640 0x7f2400160090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3301/0 says I am v2:172.21.15.16:43832/0 (socket says 172.21.15.16:43832) 2023-05-22T15:43:05.793 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:05.787+0000 7f2405393640 1 -- 172.21.15.16:0/3994243592 learned_addr learned my addr 172.21.15.16:0/3994243592 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:43:05.793 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:05.787+0000 7f2406395640 1 --2- 172.21.15.16:0/3994243592 >> v2:172.21.15.62:3300/0 conn(0x7f24001605d0 0x7f240015c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:05.794 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:05.787+0000 7f2405393640 1 -- 172.21.15.16:0/3994243592 >> v2:172.21.15.62:3300/0 conn(0x7f24001605d0 msgr2=0x7f240015c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:43:05.794 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:05.787+0000 7f2405393640 1 --2- 172.21.15.16:0/3994243592 >> v2:172.21.15.62:3300/0 conn(0x7f24001605d0 0x7f240015c270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:05.794 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:05.787+0000 7f2405393640 1 -- 172.21.15.16:0/3994243592 >> v1:172.21.15.62:6789/0 conn(0x7f24000ce480 legacy=0x7f24000ce850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:05.794 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:05.787+0000 7f2405393640 1 -- 172.21.15.16:0/3994243592 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2400069dc0 con 0x7f2400071640 2023-05-22T15:43:05.795 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:05.791+0000 7f2405393640 1 --2- 172.21.15.16:0/3994243592 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2400071640 0x7f2400160090 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f23f0009870 tx=0x7f23f001c510 comp rx=0 tx=0).ready entity=mon.2 client_cookie=deb92e0f241cb3ee server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:43:05.795 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:05.791+0000 7f2404b92640 1 -- 172.21.15.16:0/3994243592 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23f00aa670 con 0x7f2400071640 2023-05-22T15:43:05.796 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:05.791+0000 7f2404b92640 1 -- 172.21.15.16:0/3994243592 <== mon.2 v2:172.21.15.62:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23f00aa7d0 con 0x7f2400071640 2023-05-22T15:43:05.796 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:05.791+0000 7f2404b92640 1 -- 172.21.15.16:0/3994243592 <== mon.2 v2:172.21.15.62:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23f00aab60 con 0x7f2400071640 2023-05-22T15:43:05.796 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:05.791+0000 7f2407df8640 1 -- 172.21.15.16:0/3994243592 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2400071640 msgr2=0x7f2400160090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:05.797 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:05.791+0000 7f2407df8640 1 --2- 172.21.15.16:0/3994243592 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2400071640 0x7f2400160090 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f23f0009870 tx=0x7f23f001c510 comp rx=0 tx=0).stop 2023-05-22T15:43:05.797 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:05.791+0000 7f2407df8640 1 -- 172.21.15.16:0/3994243592 shutdown_connections 2023-05-22T15:43:05.797 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:05.791+0000 7f2407df8640 1 --2- 172.21.15.16:0/3994243592 >> v2:172.21.15.62:3300/0 conn(0x7f24001605d0 0x7f240015c270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:05.797 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:05.791+0000 7f2407df8640 1 --2- 172.21.15.16:0/3994243592 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2400071640 0x7f2400160090 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:05.797 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:05.791+0000 7f2407df8640 1 -- 172.21.15.16:0/3994243592 >> 172.21.15.16:0/3994243592 conn(0x7f24000cc870 msgr2=0x7f24000cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:43:05.797 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:05.791+0000 7f2407df8640 1 -- 172.21.15.16:0/3994243592 shutdown_connections 2023-05-22T15:43:05.798 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:05.791+0000 7f2407df8640 1 -- 172.21.15.16:0/3994243592 wait complete. 2023-05-22T15:43:05.799 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:05.791+0000 7f2407df8640 1 Processor -- start 2023-05-22T15:43:05.800 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:05.791+0000 7f2407df8640 1 -- start start 2023-05-22T15:43:05.800 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:05.795+0000 7f2407df8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2400071640 0x7f24001f4ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:05.800 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:05.795+0000 7f2407df8640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f24000ce480 0x7f24000aef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:05.800 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:05.795+0000 7f2407df8640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f24001605d0 0x7f24000b0950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:05.800 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:05.795+0000 7f2407df8640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2400156030 con 0x7f2400071640 2023-05-22T15:43:05.800 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:05.795+0000 7f2407df8640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24001f5000 con 0x7f24001605d0 2023-05-22T15:43:05.801 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:05.795+0000 7f2407df8640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24000af4c0 con 0x7f24000ce480 2023-05-22T15:43:05.801 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:05.795+0000 7f2405b94640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2400071640 0x7f24001f4ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:05.801 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:05.795+0000 7f2405b94640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2400071640 0x7f24001f4ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49962/0 (socket says 172.21.15.16:49962) 2023-05-22T15:43:05.801 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:05.795+0000 7f2405b94640 1 -- 172.21.15.16:0/2296929208 learned_addr learned my addr 172.21.15.16:0/2296929208 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:43:05.801 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:05.795+0000 7f2405393640 1 --2- 172.21.15.16:0/2296929208 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f24000ce480 0x7f24000aef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:05.801 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:05.795+0000 7f2406395640 1 --2- 172.21.15.16:0/2296929208 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f24001605d0 0x7f24000b0950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:05.802 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:05.795+0000 7f2405b94640 1 -- 172.21.15.16:0/2296929208 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f24000ce480 msgr2=0x7f24000aef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:05.802 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:05.795+0000 7f2405b94640 1 --2- 172.21.15.16:0/2296929208 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f24000ce480 0x7f24000aef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:05.802 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:05.795+0000 7f2405b94640 1 -- 172.21.15.16:0/2296929208 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f24001605d0 msgr2=0x7f24000b0950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:05.802 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:05.795+0000 7f2405b94640 1 --2- 172.21.15.16:0/2296929208 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f24001605d0 0x7f24000b0950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:05.802 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:05.795+0000 7f2405b94640 1 -- 172.21.15.16:0/2296929208 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23f00094d0 con 0x7f2400071640 2023-05-22T15:43:05.802 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:05.795+0000 7f2406395640 1 --2- 172.21.15.16:0/2296929208 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f24001605d0 0x7f24000b0950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T15:43:05.803 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:05.795+0000 7f2405b94640 1 --2- 172.21.15.16:0/2296929208 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2400071640 0x7f24001f4ac0 secure :-1 s=READY pgs=1314 cs=0 l=1 rev1=1 crypto rx=0x7f23fc00c7a0 tx=0x7f23fc00cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:43:05.803 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:05.795+0000 7f23f67fc640 1 -- 172.21.15.16:0/2296929208 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23fc003b80 con 0x7f2400071640 2023-05-22T15:43:05.803 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:05.795+0000 7f23f67fc640 1 -- 172.21.15.16:0/2296929208 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23fc003ce0 con 0x7f2400071640 2023-05-22T15:43:05.803 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:05.795+0000 7f2407df8640 1 -- 172.21.15.16:0/2296929208 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24000698b0 con 0x7f2400071640 2023-05-22T15:43:05.803 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:05.795+0000 7f23f67fc640 1 -- 172.21.15.16:0/2296929208 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23fc00aa90 con 0x7f2400071640 2023-05-22T15:43:05.803 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:05.795+0000 7f2407df8640 1 -- 172.21.15.16:0/2296929208 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2400069720 con 0x7f2400071640 2023-05-22T15:43:05.804 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:05.795+0000 7f2407df8640 1 -- 172.21.15.16:0/2296929208 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24000c5e70 con 0x7f2400071640 2023-05-22T15:43:05.804 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:05.795+0000 7f23f67fc640 1 -- 172.21.15.16:0/2296929208 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 35) v1 ==== 84349+0+0 (secure 0 0 0) 0x7f23fc00abf0 con 0x7f2400071640 2023-05-22T15:43:05.805 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:05.795+0000 7f23f67fc640 1 --2- 172.21.15.16:0/2296929208 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f23e40654f0 0x7f23e40679a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:05.805 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:05.795+0000 7f2405393640 1 -- 172.21.15.16:0/2296929208 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f23e40654f0 msgr2=0x7f23e40679a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/36921 2023-05-22T15:43:05.805 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:05.795+0000 7f23f67fc640 1 -- 172.21.15.16:0/2296929208 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f23fc083480 con 0x7f2400071640 2023-05-22T15:43:05.805 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:05.795+0000 7f2405393640 1 --2- 172.21.15.16:0/2296929208 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f23e40654f0 0x7f23e40679a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-05-22T15:43:05.805 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:05.799+0000 7f23f67fc640 1 -- 172.21.15.16:0/2296929208 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f23fc050f40 con 0x7f2400071640 2023-05-22T15:43:05.959 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:05.955+0000 7f2407df8640 1 -- 172.21.15.16:0/2296929208 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7f24001f52c0 con 0x7f2400071640 2023-05-22T15:43:05.966 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:05.963+0000 7f23f67fc640 1 -- 172.21.15.16:0/2296929208 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v29)=0 v29) v1 ==== 138+0+0 (secure 0 0 0) 0x7f23fc054560 con 0x7f2400071640 2023-05-22T15:43:05.968 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:05.963+0000 7f2407df8640 1 -- 172.21.15.16:0/2296929208 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f23e40654f0 msgr2=0x7f23e40679a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-05-22T15:43:05.969 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:05.963+0000 7f2407df8640 1 --2- 172.21.15.16:0/2296929208 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f23e40654f0 0x7f23e40679a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:05.969 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:05.963+0000 7f2407df8640 1 -- 172.21.15.16:0/2296929208 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2400071640 msgr2=0x7f24001f4ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:05.969 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:05.963+0000 7f2407df8640 1 --2- 172.21.15.16:0/2296929208 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2400071640 0x7f24001f4ac0 secure :-1 s=READY pgs=1314 cs=0 l=1 rev1=1 crypto rx=0x7f23fc00c7a0 tx=0x7f23fc00cc60 comp rx=0 tx=0).stop 2023-05-22T15:43:05.969 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:05.963+0000 7f2407df8640 1 -- 172.21.15.16:0/2296929208 shutdown_connections 2023-05-22T15:43:05.969 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:05.963+0000 7f2407df8640 1 --2- 172.21.15.16:0/2296929208 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f24001605d0 0x7f24000b0950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:05.970 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:05.963+0000 7f2407df8640 1 --2- 172.21.15.16:0/2296929208 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f23e40654f0 0x7f23e40679a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:05.970 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:05.963+0000 7f2407df8640 1 --2- 172.21.15.16:0/2296929208 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f24000ce480 0x7f24000aef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:05.970 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:05.963+0000 7f2407df8640 1 --2- 172.21.15.16:0/2296929208 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2400071640 0x7f24001f4ac0 unknown :-1 s=CLOSED pgs=1314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:05.970 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:05.963+0000 7f2407df8640 1 -- 172.21.15.16:0/2296929208 >> 172.21.15.16:0/2296929208 conn(0x7f24000cc870 msgr2=0x7f240015ea30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:43:05.970 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:05.963+0000 7f2407df8640 1 -- 172.21.15.16:0/2296929208 shutdown_connections 2023-05-22T15:43:05.970 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:05.963+0000 7f2407df8640 1 -- 172.21.15.16:0/2296929208 wait complete. 2023-05-22T15:43:05.985 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2023-05-22T15:43:06.107 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:06.103+0000 7fc814f0c640 1 Processor -- start 2023-05-22T15:43:06.108 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:06.103+0000 7fc814f0c640 1 -- start start 2023-05-22T15:43:06.108 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:06.103+0000 7fc814f0c640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc8100b1cd0 0x7fc8100cbbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:06.109 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:06.103+0000 7fc814f0c640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8100b4650 con 0x7fc8100cc120 2023-05-22T15:43:06.109 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:06.103+0000 7fc814f0c640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8100b4040 con 0x7fc81015d0a0 2023-05-22T15:43:06.109 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:06.103+0000 7fc814f0c640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8100cc5c0 con 0x7fc8100b1cd0 2023-05-22T15:43:06.109 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:06.103+0000 7fc80dd9b640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc8100b1cd0 0x7fc8100cbbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:06.109 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:06.103+0000 7fc80ed9d640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fc8100cc120 0x7fc81006daf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:48816/0 (socket says 172.21.15.16:48816) 2023-05-22T15:43:06.109 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:06.103+0000 7fc80dd9b640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc8100b1cd0 0x7fc8100cbbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49964/0 (socket says 172.21.15.16:49964) 2023-05-22T15:43:06.109 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:06.103+0000 7fc80ed9d640 1 -- 172.21.15.16:0/329540471 learned_addr learned my addr 172.21.15.16:0/329540471 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:43:06.110 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:06.103+0000 7fc80dd9b640 1 -- 172.21.15.16:0/329540471 >> v1:172.21.15.62:6789/0 conn(0x7fc81015d0a0 legacy=0x7fc8100c72b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:06.110 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:06.103+0000 7fc80dd9b640 1 -- 172.21.15.16:0/329540471 >> v1:172.21.15.16:6789/0 conn(0x7fc8100cc120 legacy=0x7fc81006daf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:06.110 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:06.103+0000 7fc80dd9b640 1 -- 172.21.15.16:0/329540471 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8100cc8a0 con 0x7fc8100b1cd0 2023-05-22T15:43:06.111 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:06.107+0000 7fc80dd9b640 1 --2- 172.21.15.16:0/329540471 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc8100b1cd0 0x7fc8100cbbe0 secure :-1 s=READY pgs=1316 cs=0 l=1 rev1=1 crypto rx=0x7fc8000097c0 tx=0x7fc80001c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=495f1685de046698 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:43:06.111 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:06.107+0000 7fc80d59a640 1 -- 172.21.15.16:0/329540471 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8000a8050 con 0x7fc8100b1cd0 2023-05-22T15:43:06.111 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:06.107+0000 7fc80d59a640 1 -- 172.21.15.16:0/329540471 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8000ac5f0 con 0x7fc8100b1cd0 2023-05-22T15:43:06.111 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:06.107+0000 7fc80d59a640 1 -- 172.21.15.16:0/329540471 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8000ac980 con 0x7fc8100b1cd0 2023-05-22T15:43:06.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:06.107+0000 7fc814f0c640 1 -- 172.21.15.16:0/329540471 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc8100b1cd0 msgr2=0x7fc8100cbbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:06.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:06.107+0000 7fc814f0c640 1 --2- 172.21.15.16:0/329540471 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc8100b1cd0 0x7fc8100cbbe0 secure :-1 s=READY pgs=1316 cs=0 l=1 rev1=1 crypto rx=0x7fc8000097c0 tx=0x7fc80001c480 comp rx=0 tx=0).stop 2023-05-22T15:43:06.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:06.107+0000 7fc814f0c640 1 -- 172.21.15.16:0/329540471 shutdown_connections 2023-05-22T15:43:06.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:06.107+0000 7fc814f0c640 1 --2- 172.21.15.16:0/329540471 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc8100b1cd0 0x7fc8100cbbe0 unknown :-1 s=CLOSED pgs=1316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:06.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:06.107+0000 7fc814f0c640 1 -- 172.21.15.16:0/329540471 >> 172.21.15.16:0/329540471 conn(0x7fc810065af0 msgr2=0x7fc81006f650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:43:06.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:06.107+0000 7fc814f0c640 1 -- 172.21.15.16:0/329540471 shutdown_connections 2023-05-22T15:43:06.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:06.107+0000 7fc814f0c640 1 -- 172.21.15.16:0/329540471 wait complete. 2023-05-22T15:43:06.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:06.107+0000 7fc814f0c640 1 Processor -- start 2023-05-22T15:43:06.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:06.107+0000 7fc814f0c640 1 -- start start 2023-05-22T15:43:06.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:06.107+0000 7fc814f0c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc8100b1cd0 0x7fc8100d0690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:06.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:06.107+0000 7fc814f0c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc8100cc120 0x7fc8100ce5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:06.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:06.107+0000 7fc814f0c640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc81015d0a0 0x7fc8101513c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:06.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:06.107+0000 7fc814f0c640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc81006b100 con 0x7fc8100b1cd0 2023-05-22T15:43:06.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:06.107+0000 7fc814f0c640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc81006a830 con 0x7fc8100cc120 2023-05-22T15:43:06.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:06.107+0000 7fc814f0c640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc81006c690 con 0x7fc81015d0a0 2023-05-22T15:43:06.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:06.107+0000 7fc80e59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc8100b1cd0 0x7fc8100d0690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:06.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:06.107+0000 7fc80e59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc8100b1cd0 0x7fc8100d0690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49976/0 (socket says 172.21.15.16:49976) 2023-05-22T15:43:06.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:06.107+0000 7fc80e59c640 1 -- 172.21.15.16:0/403261990 learned_addr learned my addr 172.21.15.16:0/403261990 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:43:06.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:06.107+0000 7fc80dd9b640 1 --2- 172.21.15.16:0/403261990 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc8100cc120 0x7fc8100ce5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:06.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:06.107+0000 7fc80ed9d640 1 --2- 172.21.15.16:0/403261990 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc81015d0a0 0x7fc8101513c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:06.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:06.111+0000 7fc80e59c640 1 -- 172.21.15.16:0/403261990 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc81015d0a0 msgr2=0x7fc8101513c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:06.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:06.111+0000 7fc80e59c640 1 --2- 172.21.15.16:0/403261990 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc81015d0a0 0x7fc8101513c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:06.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:06.111+0000 7fc80e59c640 1 -- 172.21.15.16:0/403261990 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc8100cc120 msgr2=0x7fc8100ce5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:06.118 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:06.111+0000 7fc80e59c640 1 --2- 172.21.15.16:0/403261990 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc8100cc120 0x7fc8100ce5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:06.118 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:06.111+0000 7fc80e59c640 1 -- 172.21.15.16:0/403261990 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc800009420 con 0x7fc8100b1cd0 2023-05-22T15:43:06.118 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:06.111+0000 7fc80e59c640 1 --2- 172.21.15.16:0/403261990 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc8100b1cd0 0x7fc8100d0690 secure :-1 s=READY pgs=1317 cs=0 l=1 rev1=1 crypto rx=0x7fc80400c3c0 tx=0x7fc80400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:43:06.118 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:06.111+0000 7fc7feffd640 1 -- 172.21.15.16:0/403261990 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8040052d0 con 0x7fc8100b1cd0 2023-05-22T15:43:06.118 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:06.111+0000 7fc7feffd640 1 -- 172.21.15.16:0/403261990 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc804005430 con 0x7fc8100b1cd0 2023-05-22T15:43:06.118 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:06.111+0000 7fc7feffd640 1 -- 172.21.15.16:0/403261990 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc804003840 con 0x7fc8100b1cd0 2023-05-22T15:43:06.118 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:06.111+0000 7fc814f0c640 1 -- 172.21.15.16:0/403261990 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8100b5460 con 0x7fc8100b1cd0 2023-05-22T15:43:06.119 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:06.111+0000 7fc814f0c640 1 -- 172.21.15.16:0/403261990 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8100b52d0 con 0x7fc8100b1cd0 2023-05-22T15:43:06.119 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:06.111+0000 7fc814f0c640 1 -- 172.21.15.16:0/403261990 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc810167570 con 0x7fc8100b1cd0 2023-05-22T15:43:06.119 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:06.111+0000 7fc7feffd640 1 -- 172.21.15.16:0/403261990 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 35) v1 ==== 84349+0+0 (secure 0 0 0) 0x7fc8040039a0 con 0x7fc8100b1cd0 2023-05-22T15:43:06.120 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:06.111+0000 7fc7feffd640 1 --2- 172.21.15.16:0/403261990 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fc7f0065520 0x7fc7f00679d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:06.120 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:06.111+0000 7fc80dd9b640 1 -- 172.21.15.16:0/403261990 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fc7f0065520 msgr2=0x7fc7f00679d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/36921 2023-05-22T15:43:06.120 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:06.111+0000 7fc7feffd640 1 -- 172.21.15.16:0/403261990 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 6382+0+0 (secure 0 0 0) 0x7fc804082000 con 0x7fc8100b1cd0 2023-05-22T15:43:06.120 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:06.111+0000 7fc80dd9b640 1 --2- 172.21.15.16:0/403261990 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fc7f0065520 0x7fc7f00679d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-05-22T15:43:06.121 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:06.115+0000 7fc7feffd640 1 -- 172.21.15.16:0/403261990 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fc80404fa60 con 0x7fc8100b1cd0 2023-05-22T15:43:06.162 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:43:06.159+0000 7f4053df2180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-05-22T15:43:06.170 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:43:06.163+0000 7f32d4c2d180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-05-22T15:43:06.268 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:06.263+0000 7fc814f0c640 1 -- 172.21.15.16:0/403261990 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7fc81004dbe0 con 0x7fc8100b1cd0 2023-05-22T15:43:06.274 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:06.271+0000 7fc7feffd640 1 -- 172.21.15.16:0/403261990 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v30)=0 v30) v1 ==== 133+0+0 (secure 0 0 0) 0x7fc804053080 con 0x7fc8100b1cd0 2023-05-22T15:43:06.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:06.271+0000 7fc814f0c640 1 -- 172.21.15.16:0/403261990 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fc7f0065520 msgr2=0x7fc7f00679d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-05-22T15:43:06.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:06.271+0000 7fc814f0c640 1 --2- 172.21.15.16:0/403261990 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fc7f0065520 0x7fc7f00679d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:06.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:06.271+0000 7fc814f0c640 1 -- 172.21.15.16:0/403261990 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc8100b1cd0 msgr2=0x7fc8100d0690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:06.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:06.271+0000 7fc814f0c640 1 --2- 172.21.15.16:0/403261990 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc8100b1cd0 0x7fc8100d0690 secure :-1 s=READY pgs=1317 cs=0 l=1 rev1=1 crypto rx=0x7fc80400c3c0 tx=0x7fc80400c880 comp rx=0 tx=0).stop 2023-05-22T15:43:06.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:06.271+0000 7fc814f0c640 1 -- 172.21.15.16:0/403261990 shutdown_connections 2023-05-22T15:43:06.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:06.271+0000 7fc814f0c640 1 --2- 172.21.15.16:0/403261990 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fc7f0065520 0x7fc7f00679d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:06.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:06.271+0000 7fc814f0c640 1 --2- 172.21.15.16:0/403261990 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc81015d0a0 0x7fc8101513c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:06.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:06.271+0000 7fc814f0c640 1 --2- 172.21.15.16:0/403261990 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc8100cc120 0x7fc8100ce5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:06.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:06.271+0000 7fc814f0c640 1 --2- 172.21.15.16:0/403261990 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc8100b1cd0 0x7fc8100d0690 unknown :-1 s=CLOSED pgs=1317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:06.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:06.271+0000 7fc814f0c640 1 -- 172.21.15.16:0/403261990 >> 172.21.15.16:0/403261990 conn(0x7fc810065af0 msgr2=0x7fc8100c7b70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:43:06.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:06.271+0000 7fc814f0c640 1 -- 172.21.15.16:0/403261990 shutdown_connections 2023-05-22T15:43:06.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:06.271+0000 7fc814f0c640 1 -- 172.21.15.16:0/403261990 wait complete. 2023-05-22T15:43:06.295 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule list --fs cephfs snap_test_dir1 2023-05-22T15:43:06.299 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:43:06.295+0000 7f4053df2180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-05-22T15:43:06.308 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:43:06.303+0000 7f32d4c2d180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-05-22T15:43:06.413 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:06.407+0000 7f191ec68640 1 Processor -- start 2023-05-22T15:43:06.414 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:06.407+0000 7f191ec68640 1 -- start start 2023-05-22T15:43:06.415 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:06.411+0000 7f191ec68640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f1918069720 0x7f1918069af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:06.415 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:06.411+0000 7f191ec68640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1918070d20 con 0x7f19180cc880 2023-05-22T15:43:06.415 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:06.411+0000 7f191ec68640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f191815e5d0 con 0x7f19180718e0 2023-05-22T15:43:06.415 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:06.411+0000 7f191ec68640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19180b1c70 con 0x7f1918069720 2023-05-22T15:43:06.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:06.411+0000 7f191ca04640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f1918069720 0x7f1918069af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:06.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:06.411+0000 7f191ca04640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f1918069720 0x7f1918069af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49990/0 (socket says 172.21.15.16:49990) 2023-05-22T15:43:06.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:06.411+0000 7f191ca04640 1 -- 172.21.15.16:0/3239084368 learned_addr learned my addr 172.21.15.16:0/3239084368 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:43:06.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:06.411+0000 7f191ca04640 1 -- 172.21.15.16:0/3239084368 >> v1:172.21.15.62:6789/0 conn(0x7f19180718e0 legacy=0x7f191806ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:43:06.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:06.411+0000 7f191ca04640 1 -- 172.21.15.16:0/3239084368 >> v1:172.21.15.16:6789/0 conn(0x7f19180cc880 legacy=0x7f191806c700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:06.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:06.411+0000 7f191ca04640 1 -- 172.21.15.16:0/3239084368 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1918159f40 con 0x7f1918069720 2023-05-22T15:43:06.417 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:06.411+0000 7f191ca04640 1 --2- 172.21.15.16:0/3239084368 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1918069720 0x7f1918069af0 secure :-1 s=READY pgs=1319 cs=0 l=1 rev1=1 crypto rx=0x7f1908009e50 tx=0x7f190801c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4484773d9786dc5 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:43:06.418 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:06.411+0000 7f19177fe640 1 -- 172.21.15.16:0/3239084368 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19080a56f0 con 0x7f1918069720 2023-05-22T15:43:06.418 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:06.411+0000 7f19177fe640 1 -- 172.21.15.16:0/3239084368 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19080a5850 con 0x7f1918069720 2023-05-22T15:43:06.418 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:06.411+0000 7f19177fe640 1 -- 172.21.15.16:0/3239084368 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19080a5be0 con 0x7f1918069720 2023-05-22T15:43:06.418 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:06.411+0000 7f191ec68640 1 -- 172.21.15.16:0/3239084368 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1918069720 msgr2=0x7f1918069af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:06.419 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:06.411+0000 7f191ec68640 1 --2- 172.21.15.16:0/3239084368 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1918069720 0x7f1918069af0 secure :-1 s=READY pgs=1319 cs=0 l=1 rev1=1 crypto rx=0x7f1908009e50 tx=0x7f190801c940 comp rx=0 tx=0).stop 2023-05-22T15:43:06.419 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:06.411+0000 7f191ec68640 1 -- 172.21.15.16:0/3239084368 shutdown_connections 2023-05-22T15:43:06.419 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:06.411+0000 7f191ec68640 1 --2- 172.21.15.16:0/3239084368 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1918069720 0x7f1918069af0 unknown :-1 s=CLOSED pgs=1319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:06.419 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:06.411+0000 7f191ec68640 1 -- 172.21.15.16:0/3239084368 >> 172.21.15.16:0/3239084368 conn(0x7f19180b0cc0 msgr2=0x7f19180b10c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:43:06.420 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:06.411+0000 7f191ec68640 1 -- 172.21.15.16:0/3239084368 shutdown_connections 2023-05-22T15:43:06.420 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:06.411+0000 7f191ec68640 1 -- 172.21.15.16:0/3239084368 wait complete. 2023-05-22T15:43:06.423 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:06.415+0000 7f191ec68640 1 Processor -- start 2023-05-22T15:43:06.423 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:06.415+0000 7f191ec68640 1 -- start start 2023-05-22T15:43:06.423 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:06.415+0000 7f191ec68640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1918069720 0x7f1918144370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:06.423 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:06.415+0000 7f191ec68640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1918157120 0x7f1918157590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:06.424 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:06.415+0000 7f191ec68640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f19180c7f50 0x7f19180c83c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:06.424 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:06.415+0000 7f191ec68640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1918158c80 con 0x7f1918069720 2023-05-22T15:43:06.424 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:06.415+0000 7f191ec68640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19180c8900 con 0x7f19180c7f50 2023-05-22T15:43:06.424 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:06.415+0000 7f191ec68640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1918157be0 con 0x7f1918157120 2023-05-22T15:43:06.424 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:06.415+0000 7f191ca04640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1918069720 0x7f1918144370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:06.425 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:06.415+0000 7f191ca04640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1918069720 0x7f1918144370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50004/0 (socket says 172.21.15.16:50004) 2023-05-22T15:43:06.425 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:06.415+0000 7f191ca04640 1 -- 172.21.15.16:0/1621202914 learned_addr learned my addr 172.21.15.16:0/1621202914 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:43:06.425 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:06.415+0000 7f191d205640 1 --2- 172.21.15.16:0/1621202914 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f19180c7f50 0x7f19180c83c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:06.425 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:06.415+0000 7f1917fff640 1 --2- 172.21.15.16:0/1621202914 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1918157120 0x7f1918157590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:06.425 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:06.415+0000 7f191ca04640 1 -- 172.21.15.16:0/1621202914 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1918157120 msgr2=0x7f1918157590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:06.425 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:06.415+0000 7f191ca04640 1 --2- 172.21.15.16:0/1621202914 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1918157120 0x7f1918157590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:06.425 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:06.415+0000 7f191ca04640 1 -- 172.21.15.16:0/1621202914 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f19180c7f50 msgr2=0x7f19180c83c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:06.425 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:06.415+0000 7f191ca04640 1 --2- 172.21.15.16:0/1621202914 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f19180c7f50 0x7f19180c83c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:06.426 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:06.415+0000 7f191ca04640 1 -- 172.21.15.16:0/1621202914 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1908009b30 con 0x7f1918069720 2023-05-22T15:43:06.426 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:06.415+0000 7f1917fff640 1 --2- 172.21.15.16:0/1621202914 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1918157120 0x7f1918157590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:43:06.426 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:06.415+0000 7f191ca04640 1 --2- 172.21.15.16:0/1621202914 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1918069720 0x7f1918144370 secure :-1 s=READY pgs=1320 cs=0 l=1 rev1=1 crypto rx=0x7f19080af040 tx=0x7f19080196e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:43:06.426 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:06.415+0000 7f19157fa640 1 -- 172.21.15.16:0/1621202914 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19080b8880 con 0x7f1918069720 2023-05-22T15:43:06.426 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:06.415+0000 7f19157fa640 1 -- 172.21.15.16:0/1621202914 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19080b8a00 con 0x7f1918069720 2023-05-22T15:43:06.426 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:06.415+0000 7f191ec68640 1 -- 172.21.15.16:0/1621202914 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f191815ddd0 con 0x7f1918069720 2023-05-22T15:43:06.426 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:06.415+0000 7f19157fa640 1 -- 172.21.15.16:0/1621202914 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19080b8db0 con 0x7f1918069720 2023-05-22T15:43:06.427 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:06.415+0000 7f191ec68640 1 -- 172.21.15.16:0/1621202914 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f191815daf0 con 0x7f1918069720 2023-05-22T15:43:06.427 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:06.415+0000 7f191ec68640 1 -- 172.21.15.16:0/1621202914 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f191815e0b0 con 0x7f1918069720 2023-05-22T15:43:06.427 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:06.419+0000 7f19157fa640 1 -- 172.21.15.16:0/1621202914 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 35) v1 ==== 84349+0+0 (secure 0 0 0) 0x7f19080b6440 con 0x7f1918069720 2023-05-22T15:43:06.427 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:06.419+0000 7f19157fa640 1 --2- 172.21.15.16:0/1621202914 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f19040654f0 0x7f19040679a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:06.427 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:06.419+0000 7f1917fff640 1 -- 172.21.15.16:0/1621202914 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f19040654f0 msgr2=0x7f19040679a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/36921 2023-05-22T15:43:06.427 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:06.419+0000 7f19157fa640 1 -- 172.21.15.16:0/1621202914 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f190811e3b0 con 0x7f1918069720 2023-05-22T15:43:06.427 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:06.419+0000 7f1917fff640 1 --2- 172.21.15.16:0/1621202914 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f19040654f0 0x7f19040679a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-05-22T15:43:06.428 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:06.423+0000 7f19157fa640 1 -- 172.21.15.16:0/1621202914 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f19080ebe10 con 0x7f1918069720 2023-05-22T15:43:06.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:06.595+0000 7f191ec68640 1 -- 172.21.15.16:0/1621202914 --> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "snap_test_dir1", "target": ["mon-mgr", ""]}) v1 -- 0x7f191804dbe0 con 0x7f19040654f0 2023-05-22T15:43:06.626 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:06.623+0000 7f1917fff640 1 -- 172.21.15.16:0/1621202914 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f19040654f0 msgr2=0x7f19040679a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/36921 2023-05-22T15:43:06.626 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:06.623+0000 7f1917fff640 1 --2- 172.21.15.16:0/1621202914 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f19040654f0 0x7f19040679a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-05-22T15:43:06.744 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:43:06.739+0000 7f4053df2180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-05-22T15:43:06.759 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:43:06.755+0000 7f32d4c2d180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-05-22T15:43:06.797 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:43:06.795+0000 7f4053df2180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-05-22T15:43:06.812 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:43:06.807+0000 7f32d4c2d180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-05-22T15:43:06.848 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:43:06.843+0000 7f4053df2180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-05-22T15:43:06.863 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:43:06.859+0000 7f32d4c2d180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-05-22T15:43:06.926 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:43:06.923+0000 7f4053df2180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-05-22T15:43:06.940 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:43:06.935+0000 7f32d4c2d180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-05-22T15:43:06.977 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:43:06.975+0000 7f4053df2180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-05-22T15:43:06.991 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:43:06.987+0000 7f32d4c2d180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-05-22T15:43:07.030 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:07.027+0000 7f1917fff640 1 -- 172.21.15.16:0/1621202914 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f19040654f0 msgr2=0x7f19040679a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/36921 2023-05-22T15:43:07.030 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:07.027+0000 7f1917fff640 1 --2- 172.21.15.16:0/1621202914 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f19040654f0 0x7f19040679a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-05-22T15:43:07.053 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:43:07.051+0000 7f4053df2180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-05-22T15:43:07.067 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:43:07.063+0000 7f32d4c2d180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-05-22T15:43:07.105 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:43:07.103+0000 7f4053df2180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-05-22T15:43:07.119 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:43:07.115+0000 7f32d4c2d180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-05-22T15:43:07.237 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:43:07.235+0000 7f4053df2180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-05-22T15:43:07.250 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:43:07.247+0000 7f32d4c2d180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-05-22T15:43:07.290 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:43:07.287+0000 7f4053df2180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-05-22T15:43:07.303 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:43:07.299+0000 7f32d4c2d180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-05-22T15:43:07.514 INFO:tasks.ceph.mgr.x.smithi062.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2023-05-22T15:43:07.514 INFO:tasks.ceph.mgr.x.smithi062.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2023-05-22T15:43:07.515 INFO:tasks.ceph.mgr.x.smithi062.stderr: from numpy import show_config as show_numpy_config 2023-05-22T15:43:07.522 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:43:07.519+0000 7f4053df2180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-05-22T15:43:07.527 INFO:tasks.ceph.mgr.y.smithi016.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2023-05-22T15:43:07.528 INFO:tasks.ceph.mgr.y.smithi016.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2023-05-22T15:43:07.528 INFO:tasks.ceph.mgr.y.smithi016.stderr: from numpy import show_config as show_numpy_config 2023-05-22T15:43:07.535 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:43:07.531+0000 7f32d4c2d180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-05-22T15:43:07.647 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:43:07.643+0000 7f4053df2180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-05-22T15:43:07.659 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:43:07.655+0000 7f32d4c2d180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-05-22T15:43:07.759 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:43:07.755+0000 7f4053df2180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-05-22T15:43:07.772 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:43:07.767+0000 7f32d4c2d180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-05-22T15:43:07.823 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:43:07.819+0000 7f4053df2180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-05-22T15:43:07.834 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:07.831+0000 7f1917fff640 1 -- 172.21.15.16:0/1621202914 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f19040654f0 msgr2=0x7f19040679a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/36921 2023-05-22T15:43:07.834 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:07.831+0000 7f1917fff640 1 --2- 172.21.15.16:0/1621202914 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f19040654f0 0x7f19040679a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-05-22T15:43:07.835 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:43:07.831+0000 7f32d4c2d180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-05-22T15:43:07.948 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:43:07.943+0000 7f4053df2180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-05-22T15:43:07.960 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:43:07.955+0000 7f32d4c2d180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-05-22T15:43:08.442 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:43:08.439+0000 7f4053df2180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-05-22T15:43:08.452 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:43:08.447+0000 7f32d4c2d180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-05-22T15:43:08.518 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:43:08.515+0000 7f4053df2180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-05-22T15:43:08.528 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:43:08.523+0000 7f32d4c2d180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-05-22T15:43:08.655 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:43:08.651+0000 7f4053df2180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-05-22T15:43:08.665 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:43:08.659+0000 7f32d4c2d180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-05-22T15:43:08.733 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:43:08.731+0000 7f4053df2180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-05-22T15:43:08.741 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:43:08.735+0000 7f32d4c2d180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-05-22T15:43:08.981 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:43:08.979+0000 7f4053df2180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-05-22T15:43:08.987 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:43:08.983+0000 7f32d4c2d180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-05-22T15:43:09.036 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:43:09.031+0000 7f4053df2180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-05-22T15:43:09.042 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:43:09.039+0000 7f32d4c2d180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-05-22T15:43:09.076 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:09.067+0000 7f19157fa640 1 -- 172.21.15.16:0/1621202914 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mgrmap(e 36) v1 ==== 83958+0+0 (secure 0 0 0) 0x7f19080ec1e0 con 0x7f1918069720 2023-05-22T15:43:09.076 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:09.071+0000 7f19157fa640 1 -- 172.21.15.16:0/1621202914 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f19040654f0 msgr2=0x7f19040679a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-05-22T15:43:09.076 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:09.071+0000 7f19157fa640 1 --2- 172.21.15.16:0/1621202914 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f19040654f0 0x7f19040679a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:09.237 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:43:09.231+0000 7f3253ea3640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T15:43:09.237 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:43:09.231+0000 7f3253ea3640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T15:43:09.237 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:43:09.231+0000 7f3253ea3640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T15:43:09.237 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:43:09.231+0000 7f3253ea3640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T15:43:09.237 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:43:09.231+0000 7f3253ea3640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T15:43:09.238 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:43:09.231+0000 7f3256ea9640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T15:43:09.238 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:43:09.231+0000 7f3256ea9640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T15:43:09.238 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:43:09.231+0000 7f3256ea9640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T15:43:09.238 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:43:09.231+0000 7f3256ea9640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T15:43:09.239 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:43:09.231+0000 7f3256ea9640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T15:43:10.137 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:10.131+0000 7f19157fa640 1 -- 172.21.15.16:0/1621202914 <== mon.0 v2:172.21.15.16:3300/0 8 ==== mgrmap(e 37) v1 ==== 84085+0+0 (secure 0 0 0) 0x7f190811c160 con 0x7f1918069720 2023-05-22T15:43:10.137 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:10.131+0000 7f19157fa640 1 --2- 172.21.15.16:0/1621202914 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f19040654f0 0x7f1904067920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:10.137 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:10.131+0000 7f19157fa640 1 -- 172.21.15.16:0/1621202914 --> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "snap_test_dir1", "target": ["mon-mgr", ""]}) v1 -- 0x7f191804dbe0 con 0x7f19040654f0 2023-05-22T15:43:10.139 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:10.135+0000 7f1917fff640 1 --2- 172.21.15.16:0/1621202914 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f19040654f0 0x7f1904067920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:10.149 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:10.143+0000 7f1917fff640 1 --2- 172.21.15.16:0/1621202914 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f19040654f0 0x7f1904067920 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f191806d950 tx=0x7f190c007db0 comp rx=0 tx=0).ready entity=mgr.7707 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:43:10.161 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:43:10.155+0000 7f325deb7640 -1 mgr.server reply reply (2) No such file or directory SnapSchedule for snap_test_dir1 not found 2023-05-22T15:43:10.161 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:10.155+0000 7f19157fa640 1 -- 172.21.15.16:0/1621202914 <== mgr.7707 v2:172.21.15.16:6832/36921 1 ==== mgr_command_reply(tid 0: -2 SnapSchedule for snap_test_dir1 not found) v1 ==== 49+0+0 (secure 0 0 0) 0x7f191804dbe0 con 0x7f19040654f0 2023-05-22T15:43:10.162 INFO:teuthology.orchestra.run.smithi016.stderr:Error ENOENT: SnapSchedule for snap_test_dir1 not found 2023-05-22T15:43:10.168 DEBUG:teuthology.orchestra.run:got remote process result: 2 2023-05-22T15:43:10.169 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2023-05-22T15:43:10.342 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:10.335+0000 7f46d5c95640 1 Processor -- start 2023-05-22T15:43:10.342 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:10.335+0000 7f46d5c95640 1 -- start start 2023-05-22T15:43:10.343 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:10.339+0000 7f46d5c95640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f46d0159d40 0x7f46d00cb260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:10.343 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:10.339+0000 7f46d5c95640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46d0070c30 con 0x7f46d006b6c0 2023-05-22T15:43:10.343 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:10.339+0000 7f46d5c95640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46d0070e60 con 0x7f46d0159970 2023-05-22T15:43:10.343 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:10.339+0000 7f46d5c95640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46d015dd90 con 0x7f46d0159d40 2023-05-22T15:43:10.344 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:10.339+0000 7f46cffff640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f46d006b6c0 0x7f46d015a5b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:57072/0 (socket says 172.21.15.16:57072) 2023-05-22T15:43:10.344 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:10.339+0000 7f46cffff640 1 -- 172.21.15.16:0/1794919899 learned_addr learned my addr 172.21.15.16:0/1794919899 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:43:10.345 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:10.339+0000 7f46c6ffd640 1 --2- 172.21.15.16:0/1794919899 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f46d0159d40 0x7f46d00cb260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:10.345 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:10.339+0000 7f46ceffd640 1 -- 172.21.15.16:0/1794919899 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2482793677 0 0) 0x7f46d0070c30 con 0x7f46d006b6c0 2023-05-22T15:43:10.346 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:10.339+0000 7f46ceffd640 1 -- 172.21.15.16:0/1794919899 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46bc003530 con 0x7f46d006b6c0 2023-05-22T15:43:10.346 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:10.339+0000 7f46ceffd640 1 -- 172.21.15.16:0/1794919899 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1677062611 0 0) 0x7f46bc003530 con 0x7f46d006b6c0 2023-05-22T15:43:10.346 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:10.339+0000 7f46ceffd640 1 -- 172.21.15.16:0/1794919899 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f46d0159d40 msgr2=0x7f46d00cb260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:43:10.346 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:10.339+0000 7f46ceffd640 1 --2- 172.21.15.16:0/1794919899 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f46d0159d40 0x7f46d00cb260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:10.346 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:10.339+0000 7f46ceffd640 1 -- 172.21.15.16:0/1794919899 >> v1:172.21.15.62:6789/0 conn(0x7f46d0159970 legacy=0x7f46d015d680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:10.346 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:10.339+0000 7f46ceffd640 1 -- 172.21.15.16:0/1794919899 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46d0071140 con 0x7f46d006b6c0 2023-05-22T15:43:10.346 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:10.339+0000 7f46ceffd640 1 -- 172.21.15.16:0/1794919899 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7f46c8001a00 con 0x7f46d006b6c0 2023-05-22T15:43:10.347 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:10.339+0000 7f46ceffd640 1 -- 172.21.15.16:0/1794919899 >> v1:172.21.15.16:6789/0 conn(0x7f46d006b6c0 legacy=0x7f46d015a5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:10.347 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:10.339+0000 7f46ceffd640 1 --2- 172.21.15.16:0/1794919899 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f46bc0039b0 0x7f46bc003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:10.347 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:10.339+0000 7f46ceffd640 1 --2- 172.21.15.16:0/1794919899 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f46d0159d40 0x7f46bc0047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:10.347 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:10.339+0000 7f46ceffd640 1 --2- 172.21.15.16:0/1794919899 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f46bc004d10 0x7f46bc005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:10.347 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:10.339+0000 7f46ceffd640 1 -- 172.21.15.16:0/1794919899 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46d015dd90 con 0x7f46bc0039b0 2023-05-22T15:43:10.347 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:10.339+0000 7f46ceffd640 1 -- 172.21.15.16:0/1794919899 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46bc00bb20 con 0x7f46bc004d10 2023-05-22T15:43:10.348 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:10.339+0000 7f46ceffd640 1 -- 172.21.15.16:0/1794919899 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46bc00bce0 con 0x7f46d0159d40 2023-05-22T15:43:10.348 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:10.339+0000 7f46cf7fe640 1 --2- 172.21.15.16:0/1794919899 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f46bc0039b0 0x7f46bc003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:10.348 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:10.339+0000 7f46cf7fe640 1 -- 172.21.15.16:0/1794919899 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f46d0159d40 msgr2=0x7f46bc0047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:43:10.348 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:10.339+0000 7f46cf7fe640 1 --2- 172.21.15.16:0/1794919899 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f46d0159d40 0x7f46bc0047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:10.348 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:10.339+0000 7f46cf7fe640 1 -- 172.21.15.16:0/1794919899 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f46bc004d10 msgr2=0x7f46bc005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:43:10.348 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:10.339+0000 7f46cf7fe640 1 --2- 172.21.15.16:0/1794919899 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f46bc004d10 0x7f46bc005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:10.348 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:10.339+0000 7f46cf7fe640 1 -- 172.21.15.16:0/1794919899 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46bc003530 con 0x7f46bc0039b0 2023-05-22T15:43:10.351 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:10.343+0000 7f46cf7fe640 1 --2- 172.21.15.16:0/1794919899 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f46bc0039b0 0x7f46bc003dc0 secure :-1 s=READY pgs=1335 cs=0 l=1 rev1=1 crypto rx=0x7f46c0001750 tx=0x7f46c001c390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d67e334494ba4bce server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:43:10.351 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:10.343+0000 7f46ceffd640 1 -- 172.21.15.16:0/1794919899 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46c00052d0 con 0x7f46bc0039b0 2023-05-22T15:43:10.351 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:10.343+0000 7f46ceffd640 1 -- 172.21.15.16:0/1794919899 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46c00a7050 con 0x7f46bc0039b0 2023-05-22T15:43:10.352 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:10.343+0000 7f46ceffd640 1 -- 172.21.15.16:0/1794919899 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46c0003600 con 0x7f46bc0039b0 2023-05-22T15:43:10.352 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:10.343+0000 7f46d5c95640 1 -- 172.21.15.16:0/1794919899 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f46bc0039b0 msgr2=0x7f46bc003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:10.352 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:10.343+0000 7f46d5c95640 1 --2- 172.21.15.16:0/1794919899 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f46bc0039b0 0x7f46bc003dc0 secure :-1 s=READY pgs=1335 cs=0 l=1 rev1=1 crypto rx=0x7f46c0001750 tx=0x7f46c001c390 comp rx=0 tx=0).stop 2023-05-22T15:43:10.352 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:10.343+0000 7f46cffff640 1 -- 172.21.15.16:0/1794919899 reap_dead start 2023-05-22T15:43:10.352 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:10.343+0000 7f46d5c95640 1 -- 172.21.15.16:0/1794919899 shutdown_connections 2023-05-22T15:43:10.352 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:10.343+0000 7f46d5c95640 1 -- 172.21.15.16:0/1794919899 >> 172.21.15.16:0/1794919899 conn(0x7f46d0069720 msgr2=0x7f46d0069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:43:10.352 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:10.343+0000 7f46d5c95640 1 -- 172.21.15.16:0/1794919899 shutdown_connections 2023-05-22T15:43:10.353 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:10.343+0000 7f46d5c95640 1 -- 172.21.15.16:0/1794919899 wait complete. 2023-05-22T15:43:10.353 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:10.343+0000 7f46d5c95640 1 Processor -- start 2023-05-22T15:43:10.353 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:10.343+0000 7f46d5c95640 1 -- start start 2023-05-22T15:43:10.353 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:10.343+0000 7f46d5c95640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f46bc0039b0 0x7f46d00b2be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:10.353 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:10.343+0000 7f46d5c95640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f46d00b3120 0x7f46d01545e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:10.353 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:10.343+0000 7f46d5c95640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f46d00b2040 0x7f46d0154b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:10.353 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:10.343+0000 7f46d5c95640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46d015a190 con 0x7f46d00b3120 2023-05-22T15:43:10.354 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:10.343+0000 7f46d5c95640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46d00c32d0 con 0x7f46bc0039b0 2023-05-22T15:43:10.354 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:10.343+0000 7f46d5c95640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46d014fd70 con 0x7f46d00b2040 2023-05-22T15:43:10.354 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:10.343+0000 7f46c6ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f46d00b3120 0x7f46d01545e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:10.354 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:10.343+0000 7f46c6ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f46d00b3120 0x7f46d01545e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45602/0 (socket says 172.21.15.16:45602) 2023-05-22T15:43:10.354 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:10.343+0000 7f46c6ffd640 1 -- 172.21.15.16:0/2762765533 learned_addr learned my addr 172.21.15.16:0/2762765533 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:43:10.354 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:10.343+0000 7f46cffff640 1 --2- 172.21.15.16:0/2762765533 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f46d00b2040 0x7f46d0154b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:10.354 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:10.343+0000 7f46c6ffd640 1 -- 172.21.15.16:0/2762765533 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f46d00b2040 msgr2=0x7f46d0154b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:10.354 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:10.343+0000 7f46c6ffd640 1 --2- 172.21.15.16:0/2762765533 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f46d00b2040 0x7f46d0154b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:10.355 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:10.343+0000 7f46cf7fe640 1 --2- 172.21.15.16:0/2762765533 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f46bc0039b0 0x7f46d00b2be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:10.355 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:10.343+0000 7f46c6ffd640 1 -- 172.21.15.16:0/2762765533 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f46bc0039b0 msgr2=0x7f46d00b2be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:10.355 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:10.343+0000 7f46c6ffd640 1 --2- 172.21.15.16:0/2762765533 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f46bc0039b0 0x7f46d00b2be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:10.355 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:10.343+0000 7f46c6ffd640 1 -- 172.21.15.16:0/2762765533 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46d006c060 con 0x7f46d00b3120 2023-05-22T15:43:10.355 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:10.347+0000 7f46c6ffd640 1 --2- 172.21.15.16:0/2762765533 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f46d00b3120 0x7f46d01545e0 secure :-1 s=READY pgs=1336 cs=0 l=1 rev1=1 crypto rx=0x7f46b800e7f0 tx=0x7f46b800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:43:10.355 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:10.347+0000 7f46ccff9640 1 -- 172.21.15.16:0/2762765533 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46b800c940 con 0x7f46d00b3120 2023-05-22T15:43:10.355 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:10.347+0000 7f46ccff9640 1 -- 172.21.15.16:0/2762765533 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46b800caa0 con 0x7f46d00b3120 2023-05-22T15:43:10.355 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:10.347+0000 7f46d5c95640 1 -- 172.21.15.16:0/2762765533 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46d006bed0 con 0x7f46d00b3120 2023-05-22T15:43:10.356 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:10.347+0000 7f46ccff9640 1 -- 172.21.15.16:0/2762765533 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46b800ce50 con 0x7f46d00b3120 2023-05-22T15:43:10.356 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:10.347+0000 7f46d5c95640 1 -- 172.21.15.16:0/2762765533 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46d00cac90 con 0x7f46d00b3120 2023-05-22T15:43:10.356 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:10.347+0000 7f46d5c95640 1 -- 172.21.15.16:0/2762765533 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46d004dbe0 con 0x7f46d00b3120 2023-05-22T15:43:10.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:10.347+0000 7f46ccff9640 1 -- 172.21.15.16:0/2762765533 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 37) v1 ==== 84085+0+0 (secure 0 0 0) 0x7f46b8013470 con 0x7f46d00b3120 2023-05-22T15:43:10.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:10.347+0000 7f46ccff9640 1 --2- 172.21.15.16:0/2762765533 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f46b0065220 0x7f46b00676d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:10.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:10.347+0000 7f46ccff9640 1 -- 172.21.15.16:0/2762765533 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 6726+0+0 (secure 0 0 0) 0x7f46b8083050 con 0x7f46d00b3120 2023-05-22T15:43:10.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:10.347+0000 7f46cf7fe640 1 --2- 172.21.15.16:0/2762765533 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f46b0065220 0x7f46b00676d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:10.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:10.347+0000 7f46cf7fe640 1 --2- 172.21.15.16:0/2762765533 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f46b0065220 0x7f46b00676d0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f46bc004750 tx=0x7f46c00011a0 comp rx=0 tx=0).ready entity=mgr.7707 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:43:10.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:10.351+0000 7f46ccff9640 1 -- 172.21.15.16:0/2762765533 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f46b8050cc0 con 0x7f46d00b3120 2023-05-22T15:43:10.554 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:10.547+0000 7f46d5c95640 1 -- 172.21.15.16:0/2762765533 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7f46d00c3410 con 0x7f46d00b3120 2023-05-22T15:43:11.462 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:11.459+0000 7f46ccff9640 1 -- 172.21.15.16:0/2762765533 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mgrmap(e 38) v1 ==== 84389+0+0 (secure 0 0 0) 0x7f46b8051090 con 0x7f46d00b3120 2023-05-22T15:43:11.464 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:11.459+0000 7f46ccff9640 1 -- 172.21.15.16:0/2762765533 <== mon.0 v2:172.21.15.16:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v38) v1 ==== 93+0+0 (secure 0 0 0) 0x7f46b8054750 con 0x7f46d00b3120 2023-05-22T15:43:11.467 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:11.463+0000 7f46d5c95640 1 -- 172.21.15.16:0/2762765533 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f46b0065220 msgr2=0x7f46b00676d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:11.467 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:11.463+0000 7f46d5c95640 1 --2- 172.21.15.16:0/2762765533 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f46b0065220 0x7f46b00676d0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f46bc004750 tx=0x7f46c00011a0 comp rx=0 tx=0).stop 2023-05-22T15:43:11.468 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:11.463+0000 7f46d5c95640 1 -- 172.21.15.16:0/2762765533 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f46d00b3120 msgr2=0x7f46d01545e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:11.468 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:11.463+0000 7f46d5c95640 1 --2- 172.21.15.16:0/2762765533 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f46d00b3120 0x7f46d01545e0 secure :-1 s=READY pgs=1336 cs=0 l=1 rev1=1 crypto rx=0x7f46b800e7f0 tx=0x7f46b800ecb0 comp rx=0 tx=0).stop 2023-05-22T15:43:11.468 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:11.463+0000 7f46d5c95640 1 -- 172.21.15.16:0/2762765533 shutdown_connections 2023-05-22T15:43:11.468 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:11.463+0000 7f46d5c95640 1 --2- 172.21.15.16:0/2762765533 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f46b0065220 0x7f46b00676d0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:11.468 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:11.463+0000 7f46d5c95640 1 --2- 172.21.15.16:0/2762765533 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f46d00b2040 0x7f46d0154b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:11.468 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:11.463+0000 7f46d5c95640 1 --2- 172.21.15.16:0/2762765533 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f46d00b3120 0x7f46d01545e0 unknown :-1 s=CLOSED pgs=1336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:11.469 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:11.463+0000 7f46d5c95640 1 --2- 172.21.15.16:0/2762765533 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f46bc0039b0 0x7f46d00b2be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:11.469 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:11.463+0000 7f46d5c95640 1 -- 172.21.15.16:0/2762765533 >> 172.21.15.16:0/2762765533 conn(0x7f46d0069720 msgr2=0x7f46d015c960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:43:11.469 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:11.463+0000 7f46d5c95640 1 -- 172.21.15.16:0/2762765533 shutdown_connections 2023-05-22T15:43:11.469 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:11.463+0000 7f46d5c95640 1 -- 172.21.15.16:0/2762765533 wait complete. 2023-05-22T15:43:11.487 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-05-22T15:43:11.488 DEBUG:teuthology.orchestra.run.smithi016:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-05-22T15:43:11.528 DEBUG:teuthology.orchestra.run.smithi062:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-05-22T15:43:11.566 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-05-22T15:43:11.566 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-05-22T15:43:11.566 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2023-05-22T15:43:11.574 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-05-22T15:43:11.587 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:43:11.583+0000 7ffb72709180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-05-22T15:43:11.593 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:43:11.587+0000 7f9700ac6180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-05-22T15:43:11.719 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:11.711+0000 7fad321ad640 1 Processor -- start 2023-05-22T15:43:11.719 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:11.715+0000 7fad321ad640 1 -- start start 2023-05-22T15:43:11.720 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:11.715+0000 7fad321ad640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fad2c0ce480 0x7fad2c071640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:11.721 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:11.715+0000 7fad321ad640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fad2c0ce850 0x7fad2c160090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:11.721 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:11.715+0000 7fad321ad640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fad2c1605d0 0x7fad2c15c270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:11.721 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:11.715+0000 7fad321ad640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad2c15f990 con 0x7fad2c0ce850 2023-05-22T15:43:11.721 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:11.715+0000 7fad321ad640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad2c069ae0 con 0x7fad2c1605d0 2023-05-22T15:43:11.721 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:11.715+0000 7fad321ad640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad2c069dc0 con 0x7fad2c0ce480 2023-05-22T15:43:11.721 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:11.715+0000 7fad2affd640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fad2c0ce850 0x7fad2c160090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:11.722 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:11.715+0000 7fad2affd640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fad2c0ce850 0x7fad2c160090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45628/0 (socket says 172.21.15.16:45628) 2023-05-22T15:43:11.722 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:11.715+0000 7fad2affd640 1 -- 172.21.15.16:0/123158899 learned_addr learned my addr 172.21.15.16:0/123158899 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:43:11.722 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:11.715+0000 7fad2b7fe640 1 --2- 172.21.15.16:0/123158899 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fad2c0ce480 0x7fad2c071640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:11.722 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:11.715+0000 7fad2bfff640 1 --2- 172.21.15.16:0/123158899 >> v2:172.21.15.62:3300/0 conn(0x7fad2c1605d0 0x7fad2c15c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:11.722 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:11.715+0000 7fad2affd640 1 -- 172.21.15.16:0/123158899 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fad2c0ce480 msgr2=0x7fad2c071640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:43:11.722 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:11.715+0000 7fad2affd640 1 --2- 172.21.15.16:0/123158899 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fad2c0ce480 0x7fad2c071640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:11.722 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:11.715+0000 7fad2affd640 1 -- 172.21.15.16:0/123158899 >> v2:172.21.15.62:3300/0 conn(0x7fad2c1605d0 msgr2=0x7fad2c15c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:43:11.723 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:11.715+0000 7fad2affd640 1 --2- 172.21.15.16:0/123158899 >> v2:172.21.15.62:3300/0 conn(0x7fad2c1605d0 0x7fad2c15c270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:11.723 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:11.715+0000 7fad2affd640 1 -- 172.21.15.16:0/123158899 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad2c06a0a0 con 0x7fad2c0ce850 2023-05-22T15:43:11.723 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:11.715+0000 7fad2bfff640 1 --2- 172.21.15.16:0/123158899 >> v2:172.21.15.62:3300/0 conn(0x7fad2c1605d0 0x7fad2c15c270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T15:43:11.724 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:11.715+0000 7fad2affd640 1 --2- 172.21.15.16:0/123158899 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fad2c0ce850 0x7fad2c160090 secure :-1 s=READY pgs=1340 cs=0 l=1 rev1=1 crypto rx=0x7fad2c070880 tx=0x7fad2001c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e666286ce0442eb server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:43:11.724 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:11.719+0000 7fad2a7fc640 1 -- 172.21.15.16:0/123158899 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad200ac410 con 0x7fad2c0ce850 2023-05-22T15:43:11.724 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:11.719+0000 7fad2a7fc640 1 -- 172.21.15.16:0/123158899 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad200ac570 con 0x7fad2c0ce850 2023-05-22T15:43:11.724 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:11.719+0000 7fad2a7fc640 1 -- 172.21.15.16:0/123158899 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad200ac900 con 0x7fad2c0ce850 2023-05-22T15:43:11.725 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:11.719+0000 7fad321ad640 1 -- 172.21.15.16:0/123158899 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fad2c0ce850 msgr2=0x7fad2c160090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:11.725 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:11.719+0000 7fad321ad640 1 --2- 172.21.15.16:0/123158899 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fad2c0ce850 0x7fad2c160090 secure :-1 s=READY pgs=1340 cs=0 l=1 rev1=1 crypto rx=0x7fad2c070880 tx=0x7fad2001c4d0 comp rx=0 tx=0).stop 2023-05-22T15:43:11.725 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:11.719+0000 7fad321ad640 1 -- 172.21.15.16:0/123158899 shutdown_connections 2023-05-22T15:43:11.725 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:11.719+0000 7fad321ad640 1 --2- 172.21.15.16:0/123158899 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fad2c0ce850 0x7fad2c160090 unknown :-1 s=CLOSED pgs=1340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:11.725 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:11.719+0000 7fad321ad640 1 --2- 172.21.15.16:0/123158899 >> v2:172.21.15.62:3300/0 conn(0x7fad2c1605d0 0x7fad2c15c270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:11.725 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:11.719+0000 7fad321ad640 1 --2- 172.21.15.16:0/123158899 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fad2c0ce480 0x7fad2c071640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:11.725 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:11.719+0000 7fad321ad640 1 -- 172.21.15.16:0/123158899 >> 172.21.15.16:0/123158899 conn(0x7fad2c0cc870 msgr2=0x7fad2c0cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:43:11.726 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:11.719+0000 7fad321ad640 1 -- 172.21.15.16:0/123158899 shutdown_connections 2023-05-22T15:43:11.726 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:11.719+0000 7fad321ad640 1 -- 172.21.15.16:0/123158899 wait complete. 2023-05-22T15:43:11.726 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:11.719+0000 7fad321ad640 1 Processor -- start 2023-05-22T15:43:11.726 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:11.719+0000 7fad321ad640 1 -- start start 2023-05-22T15:43:11.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:11.719+0000 7fad321ad640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fad2c0ce480 0x7fad2c14fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:11.729 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:11.719+0000 7fad321ad640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fad2c0ce850 0x7fad2c0ab840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:11.729 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:11.719+0000 7fad321ad640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fad2c1605d0 0x7fad2c14d0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:11.729 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:11.719+0000 7fad321ad640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad2c14f030 con 0x7fad2c1605d0 2023-05-22T15:43:11.729 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:11.719+0000 7fad321ad640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad2c0abd80 con 0x7fad2c0ce850 2023-05-22T15:43:11.729 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:11.719+0000 7fad321ad640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad2c14eba0 con 0x7fad2c0ce480 2023-05-22T15:43:11.729 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:11.719+0000 7fad2bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fad2c1605d0 0x7fad2c14d0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:11.729 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:11.719+0000 7fad2bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fad2c1605d0 0x7fad2c14d0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45644/0 (socket says 172.21.15.16:45644) 2023-05-22T15:43:11.730 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:11.719+0000 7fad2bfff640 1 -- 172.21.15.16:0/4255466587 learned_addr learned my addr 172.21.15.16:0/4255466587 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:43:11.730 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:11.719+0000 7fad2b7fe640 1 --2- 172.21.15.16:0/4255466587 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fad2c0ce480 0x7fad2c14fbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:11.730 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:11.719+0000 7fad2affd640 1 --2- 172.21.15.16:0/4255466587 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fad2c0ce850 0x7fad2c0ab840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:11.730 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:11.719+0000 7fad2bfff640 1 -- 172.21.15.16:0/4255466587 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fad2c0ce480 msgr2=0x7fad2c14fbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:11.730 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:11.719+0000 7fad2bfff640 1 --2- 172.21.15.16:0/4255466587 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fad2c0ce480 0x7fad2c14fbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:11.730 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:11.719+0000 7fad2bfff640 1 -- 172.21.15.16:0/4255466587 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fad2c0ce850 msgr2=0x7fad2c0ab840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:11.730 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:11.719+0000 7fad2bfff640 1 --2- 172.21.15.16:0/4255466587 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fad2c0ce850 0x7fad2c0ab840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:11.730 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:11.719+0000 7fad2bfff640 1 -- 172.21.15.16:0/4255466587 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad200094d0 con 0x7fad2c1605d0 2023-05-22T15:43:11.731 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:11.719+0000 7fad2b7fe640 1 --2- 172.21.15.16:0/4255466587 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fad2c0ce480 0x7fad2c14fbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T15:43:11.731 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:11.719+0000 7fad2bfff640 1 --2- 172.21.15.16:0/4255466587 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fad2c1605d0 0x7fad2c14d0d0 secure :-1 s=READY pgs=1341 cs=0 l=1 rev1=1 crypto rx=0x7fad1c00e810 tx=0x7fad1c00ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:43:11.731 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:11.719+0000 7fad13fff640 1 -- 172.21.15.16:0/4255466587 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad1c00c670 con 0x7fad2c1605d0 2023-05-22T15:43:11.731 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:11.719+0000 7fad13fff640 1 -- 172.21.15.16:0/4255466587 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad1c00c7f0 con 0x7fad2c1605d0 2023-05-22T15:43:11.731 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:11.719+0000 7fad321ad640 1 -- 172.21.15.16:0/4255466587 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad2c0698b0 con 0x7fad2c1605d0 2023-05-22T15:43:11.731 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:11.719+0000 7fad13fff640 1 -- 172.21.15.16:0/4255466587 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad1c00cba0 con 0x7fad2c1605d0 2023-05-22T15:43:11.731 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:11.723+0000 7fad321ad640 1 -- 172.21.15.16:0/4255466587 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad2c069720 con 0x7fad2c1605d0 2023-05-22T15:43:11.732 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:11.723+0000 7fad321ad640 1 -- 172.21.15.16:0/4255466587 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad2c0c5e70 con 0x7fad2c1605d0 2023-05-22T15:43:11.732 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:11.723+0000 7fad13fff640 1 -- 172.21.15.16:0/4255466587 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 38) v1 ==== 84389+0+0 (secure 0 0 0) 0x7fad1c00cd00 con 0x7fad2c1605d0 2023-05-22T15:43:11.732 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:11.723+0000 7fad13fff640 1 --2- 172.21.15.16:0/4255466587 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fad18065530 0x7fad180679e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:11.732 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:11.723+0000 7fad2b7fe640 1 -- 172.21.15.16:0/4255466587 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fad18065530 msgr2=0x7fad180679e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/36921 2023-05-22T15:43:11.732 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:11.723+0000 7fad13fff640 1 -- 172.21.15.16:0/4255466587 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 6726+0+0 (secure 0 0 0) 0x7fad1c082d70 con 0x7fad2c1605d0 2023-05-22T15:43:11.732 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:11.723+0000 7fad2b7fe640 1 --2- 172.21.15.16:0/4255466587 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fad18065530 0x7fad180679e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-05-22T15:43:11.733 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:11.727+0000 7fad13fff640 1 -- 172.21.15.16:0/4255466587 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fad1c0503f0 con 0x7fad2c1605d0 2023-05-22T15:43:11.886 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:11.879+0000 7fad321ad640 1 -- 172.21.15.16:0/4255466587 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fad2c04dbe0 con 0x7fad2c1605d0 2023-05-22T15:43:11.887 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:11.883+0000 7fad13fff640 1 -- 172.21.15.16:0/4255466587 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v105) v1 ==== 74+0+18531 (secure 0 0 0) 0x7fad1c021020 con 0x7fad2c1605d0 2023-05-22T15:43:11.888 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:43:11.888 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":105,"fsid":"b2719834-7847-4d98-90a6-f65d57f85c49","created":"2023-05-22T15:26:25.259314+0000","modified":"2023-05-22T15:43:09.063772+0000","last_up_change":"2023-05-22T15:26:36.448524+0000","last_in_change":"2023-05-22T15:26:27.675668+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-05-22T15:26:38.019781+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-05-22T15:26:43.570514+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2023-05-22T15:42:55.131159+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"103","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2023-05-22T15:42:55.567899+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"104","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9e7ff854-b25f-4cbb-a5c2-1799d9c53d2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":102,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":37021},{"type":"v1","addr":"172.21.15.16:6825","nonce":37021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":37021},{"type":"v1","addr":"172.21.15.16:6827","nonce":37021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":37021},{"type":"v1","addr":"172.21.15.16:6831","nonce":37021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":37021},{"type":"v1","addr":"172.21.15.16:6829","nonce":37021}]},"public_addr":"172.21.15.16:6825/37021","cluster_addr":"172.21.15.16:6827/37021","heartbeat_back_addr":"172.21.15.16:6831/37021","heartbeat_front_addr":"172.21.15.16:6829/37021","state":["exists","up"]},{"osd":1,"uuid":"4702fbc9-a927-4565-aca5-41dc69cc8264","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":102,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":37008},{"type":"v1","addr":"172.21.15.16:6801","nonce":37008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":37008},{"type":"v1","addr":"172.21.15.16:6803","nonce":37008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":37008},{"type":"v1","addr":"172.21.15.16:6807","nonce":37008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":37008},{"type":"v1","addr":"172.21.15.16:6805","nonce":37008}]},"public_addr":"172.21.15.16:6801/37008","cluster_addr":"172.21.15.16:6803/37008","heartbeat_back_addr":"172.21.15.16:6807/37008","heartbeat_front_addr":"172.21.15.16:6805/37008","state":["exists","up"]},{"osd":2,"uuid":"1e7bba46-4eba-47ef-bb81-5e2d371638ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":102,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":37009},{"type":"v1","addr":"172.21.15.16:6809","nonce":37009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":37009},{"type":"v1","addr":"172.21.15.16:6811","nonce":37009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":37009},{"type":"v1","addr":"172.21.15.16:6815","nonce":37009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":37009},{"type":"v1","addr":"172.21.15.16:6813","nonce":37009}]},"public_addr":"172.21.15.16:6809/37009","cluster_addr":"172.21.15.16:6811/37009","heartbeat_back_addr":"172.21.15.16:6815/37009","heartbeat_front_addr":"172.21.15.16:6813/37009","state":["exists","up"]},{"osd":3,"uuid":"383cacc0-5a88-4904-990e-498ca59ef48a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":102,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":37020},{"type":"v1","addr":"172.21.15.16:6817","nonce":37020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":37020},{"type":"v1","addr":"172.21.15.16:6819","nonce":37020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":37020},{"type":"v1","addr":"172.21.15.16:6823","nonce":37020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":37020},{"type":"v1","addr":"172.21.15.16:6821","nonce":37020}]},"public_addr":"172.21.15.16:6817/37020","cluster_addr":"172.21.15.16:6819/37020","heartbeat_back_addr":"172.21.15.16:6823/37020","heartbeat_front_addr":"172.21.15.16:6821/37020","state":["exists","up"]},{"osd":4,"uuid":"970500ef-95b7-42c1-b080-eed4fae7013d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":102,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":37353},{"type":"v1","addr":"172.21.15.62:6817","nonce":37353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6819","nonce":37353},{"type":"v1","addr":"172.21.15.62:6822","nonce":37353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37353},{"type":"v1","addr":"172.21.15.62:6831","nonce":37353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37353},{"type":"v1","addr":"172.21.15.62:6826","nonce":37353}]},"public_addr":"172.21.15.62:6817/37353","cluster_addr":"172.21.15.62:6822/37353","heartbeat_back_addr":"172.21.15.62:6831/37353","heartbeat_front_addr":"172.21.15.62:6826/37353","state":["exists","up"]},{"osd":5,"uuid":"022d67db-d4d8-445b-b1b9-8b446a00f5c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":102,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":37352},{"type":"v1","addr":"172.21.15.62:6801","nonce":37352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":37352},{"type":"v1","addr":"172.21.15.62:6803","nonce":37352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":37352},{"type":"v1","addr":"172.21.15.62:6807","nonce":37352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":37352},{"type":"v1","addr":"172.21.15.62:6805","nonce":37352}]},"public_addr":"172.21.15.62:6801/37352","cluster_addr":"172.21.15.62:6803/37352","heartbeat_back_addr":"172.21.15.62:6807/37352","heartbeat_front_addr":"172.21.15.62:6805/37352","state":["exists","up"]},{"osd":6,"uuid":"42bd372b-6688-4a1b-8407-e16e8c266e33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":102,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":37354},{"type":"v1","addr":"172.21.15.62:6820","nonce":37354}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":37354},{"type":"v1","addr":"172.21.15.62:6823","nonce":37354}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6829","nonce":37354},{"type":"v1","addr":"172.21.15.62:6830","nonce":37354}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6825","nonce":37354},{"type":"v1","addr":"172.21.15.62:6827","nonce":37354}]},"public_addr":"172.21.15.62:6820/37354","cluster_addr":"172.21.15.62:6823/37354","heartbeat_back_addr":"172.21.15.62:6830/37354","heartbeat_front_addr":"172.21.15.62:6827/37354","state":["exists","up"]},{"osd":7,"uuid":"ac70b2a3-f41b-4bf2-b227-7f2cbb55426d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":102,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37355},{"type":"v1","addr":"172.21.15.62:6809","nonce":37355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37355},{"type":"v1","addr":"172.21.15.62:6811","nonce":37355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":37355},{"type":"v1","addr":"172.21.15.62:6815","nonce":37355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":37355},{"type":"v1","addr":"172.21.15.62:6813","nonce":37355}]},"public_addr":"172.21.15.62:6809/37355","cluster_addr":"172.21.15.62:6811/37355","heartbeat_back_addr":"172.21.15.62:6815/37355","heartbeat_front_addr":"172.21.15.62:6813/37355","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.368896+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.107273+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.276888+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.287502+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.124791+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.251937+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.255679+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.232797+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:0/1315052983":"2023-05-23T15:43:09.063704+0000","172.21.15.16:0/3981695108":"2023-05-23T15:43:09.063704+0000","172.21.15.16:0/92534915":"2023-05-23T15:43:09.063704+0000","172.21.15.16:6833/36921":"2023-05-23T15:43:09.063704+0000","172.21.15.16:6832/36921":"2023-05-23T15:43:09.063704+0000","172.21.15.16:0/2741778632":"2023-05-23T15:43:09.063704+0000","172.21.15.16:0/3940121619":"2023-05-23T15:43:09.063704+0000","172.21.15.16:0/2073223440":"2023-05-23T15:43:09.063704+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-05-22T15:43:11.890 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:11.883+0000 7fad321ad640 1 -- 172.21.15.16:0/4255466587 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fad18065530 msgr2=0x7fad180679e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-05-22T15:43:11.890 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:11.883+0000 7fad321ad640 1 --2- 172.21.15.16:0/4255466587 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fad18065530 0x7fad180679e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:11.891 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:11.883+0000 7fad321ad640 1 -- 172.21.15.16:0/4255466587 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fad2c1605d0 msgr2=0x7fad2c14d0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:11.892 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:11.883+0000 7fad321ad640 1 --2- 172.21.15.16:0/4255466587 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fad2c1605d0 0x7fad2c14d0d0 secure :-1 s=READY pgs=1341 cs=0 l=1 rev1=1 crypto rx=0x7fad1c00e810 tx=0x7fad1c00ecd0 comp rx=0 tx=0).stop 2023-05-22T15:43:11.893 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:11.883+0000 7fad321ad640 1 -- 172.21.15.16:0/4255466587 shutdown_connections 2023-05-22T15:43:11.893 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:11.883+0000 7fad321ad640 1 --2- 172.21.15.16:0/4255466587 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fad2c1605d0 0x7fad2c14d0d0 unknown :-1 s=CLOSED pgs=1341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:11.893 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:11.883+0000 7fad321ad640 1 --2- 172.21.15.16:0/4255466587 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fad2c0ce850 0x7fad2c0ab840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:11.893 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:11.883+0000 7fad321ad640 1 --2- 172.21.15.16:0/4255466587 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fad18065530 0x7fad180679e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:11.893 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:11.883+0000 7fad321ad640 1 --2- 172.21.15.16:0/4255466587 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fad2c0ce480 0x7fad2c14fbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:11.893 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:11.883+0000 7fad321ad640 1 -- 172.21.15.16:0/4255466587 >> 172.21.15.16:0/4255466587 conn(0x7fad2c0cc870 msgr2=0x7fad2c06e990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:43:11.894 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:11.883+0000 7fad321ad640 1 -- 172.21.15.16:0/4255466587 shutdown_connections 2023-05-22T15:43:11.894 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:11.883+0000 7fad321ad640 1 -- 172.21.15.16:0/4255466587 wait complete. 2023-05-22T15:43:11.906 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-05-22T15:26:38.019781+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-05-22T15:26:43.570514+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 12, 'pool_name': 'cephfs_metadata', 'create_time': '2023-05-22T15:42:55.131159+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '103', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 13, 'pool_name': 'cephfs_data', 'create_time': '2023-05-22T15:42:55.567899+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '104', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-05-22T15:43:11.907 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-05-22T15:43:11.982 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:43:11.979+0000 7ffb72709180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-05-22T15:43:11.989 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:43:11.983+0000 7f9700ac6180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-05-22T15:43:12.025 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.019+0000 7f9e66b20640 1 Processor -- start 2023-05-22T15:43:12.026 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.019+0000 7f9e66b20640 1 -- start start 2023-05-22T15:43:12.026 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.023+0000 7f9e66b20640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9e600cae20 0x7f9e6006fef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:12.027 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.023+0000 7f9e66b20640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f9e60070430 0x7f9e600ad410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:12.027 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.023+0000 7f9e66b20640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e60069720 con 0x7f9e60153fe0 2023-05-22T15:43:12.027 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.023+0000 7f9e66b20640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e600708d0 con 0x7f9e60070430 2023-05-22T15:43:12.027 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.023+0000 7f9e66b20640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e6006db00 con 0x7f9e600cae20 2023-05-22T15:43:12.028 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.023+0000 7f9e5ffff640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9e600cae20 0x7f9e6006fef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:12.028 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.023+0000 7f9e650bd640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f9e60070430 0x7f9e600ad410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:12.028 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.023+0000 7f9e648bc640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f9e60153fe0 0x7f9e6006d3f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.16:42084/0 (socket says 172.21.15.16:42084) 2023-05-22T15:43:12.028 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.023+0000 7f9e648bc640 1 -- 172.21.15.16:0/2515606200 learned_addr learned my addr 172.21.15.16:0/2515606200 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:43:12.028 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.023+0000 7f9e650bd640 1 -- 172.21.15.16:0/2515606200 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9e600cae20 msgr2=0x7f9e6006fef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:43:12.028 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.023+0000 7f9e650bd640 1 --2- 172.21.15.16:0/2515606200 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9e600cae20 0x7f9e6006fef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:12.029 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.023+0000 7f9e650bd640 1 -- 172.21.15.16:0/2515606200 >> v1:172.21.15.62:6789/0 conn(0x7f9e60153fe0 legacy=0x7f9e6006d3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:12.029 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.023+0000 7f9e650bd640 1 -- 172.21.15.16:0/2515606200 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e601543b0 con 0x7f9e60070430 2023-05-22T15:43:12.029 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.023+0000 7f9e5f7fe640 1 -- 172.21.15.16:0/2515606200 <== mon.1 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3031504105 0 0) 0x7f9e60069720 con 0x7f9e60153fe0 2023-05-22T15:43:12.030 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.023+0000 7f9e650bd640 1 --2- 172.21.15.16:0/2515606200 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9e60070430 0x7f9e600ad410 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f9e58009dc0 tx=0x7f9e5801c900 comp rx=0 tx=0).ready entity=mon.1 client_cookie=144c28b4d6d8b9cb server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:43:12.030 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.027+0000 7f9e5f7fe640 1 -- 172.21.15.16:0/2515606200 <== mon.1 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e580a59c0 con 0x7f9e60070430 2023-05-22T15:43:12.030 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.027+0000 7f9e5f7fe640 1 -- 172.21.15.16:0/2515606200 <== mon.1 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e580a5b20 con 0x7f9e60070430 2023-05-22T15:43:12.030 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.027+0000 7f9e5f7fe640 1 -- 172.21.15.16:0/2515606200 <== mon.1 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e580ad440 con 0x7f9e60070430 2023-05-22T15:43:12.031 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.027+0000 7f9e66b20640 1 -- 172.21.15.16:0/2515606200 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9e60070430 msgr2=0x7f9e600ad410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:12.031 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.027+0000 7f9e66b20640 1 --2- 172.21.15.16:0/2515606200 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9e60070430 0x7f9e600ad410 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f9e58009dc0 tx=0x7f9e5801c900 comp rx=0 tx=0).stop 2023-05-22T15:43:12.031 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.027+0000 7f9e66b20640 1 -- 172.21.15.16:0/2515606200 shutdown_connections 2023-05-22T15:43:12.032 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.027+0000 7f9e66b20640 1 --2- 172.21.15.16:0/2515606200 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9e60070430 0x7f9e600ad410 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:12.032 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.027+0000 7f9e66b20640 1 --2- 172.21.15.16:0/2515606200 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9e600cae20 0x7f9e6006fef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:12.032 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.027+0000 7f9e66b20640 1 -- 172.21.15.16:0/2515606200 >> 172.21.15.16:0/2515606200 conn(0x7f9e600b1110 msgr2=0x7f9e600ce830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:43:12.032 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.027+0000 7f9e66b20640 1 -- 172.21.15.16:0/2515606200 shutdown_connections 2023-05-22T15:43:12.032 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.027+0000 7f9e66b20640 1 -- 172.21.15.16:0/2515606200 wait complete. 2023-05-22T15:43:12.033 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.027+0000 7f9e66b20640 1 Processor -- start 2023-05-22T15:43:12.033 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.027+0000 7f9e66b20640 1 -- start start 2023-05-22T15:43:12.034 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.027+0000 7f9e66b20640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9e60070430 0x7f9e601f48c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:12.034 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.027+0000 7f9e66b20640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9e600cae20 0x7f9e601f2a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:12.034 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.027+0000 7f9e66b20640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e60153fe0 0x7f9e601f2320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:12.034 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.027+0000 7f9e66b20640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e60150910 con 0x7f9e60153fe0 2023-05-22T15:43:12.034 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.027+0000 7f9e66b20640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e60156320 con 0x7f9e600cae20 2023-05-22T15:43:12.034 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.027+0000 7f9e66b20640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e601f2860 con 0x7f9e60070430 2023-05-22T15:43:12.034 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.027+0000 7f9e650bd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e60153fe0 0x7f9e601f2320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:12.034 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.027+0000 7f9e650bd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e60153fe0 0x7f9e601f2320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45656/0 (socket says 172.21.15.16:45656) 2023-05-22T15:43:12.035 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.027+0000 7f9e650bd640 1 -- 172.21.15.16:0/3721999904 learned_addr learned my addr 172.21.15.16:0/3721999904 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:43:12.035 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.027+0000 7f9e648bc640 1 --2- 172.21.15.16:0/3721999904 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9e60070430 0x7f9e601f48c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:12.035 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.027+0000 7f9e5ffff640 1 --2- 172.21.15.16:0/3721999904 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9e600cae20 0x7f9e601f2a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:12.035 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.027+0000 7f9e650bd640 1 -- 172.21.15.16:0/3721999904 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9e60070430 msgr2=0x7f9e601f48c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:12.035 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.027+0000 7f9e650bd640 1 --2- 172.21.15.16:0/3721999904 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9e60070430 0x7f9e601f48c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:12.035 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.027+0000 7f9e650bd640 1 -- 172.21.15.16:0/3721999904 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9e600cae20 msgr2=0x7f9e601f2a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:12.035 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.027+0000 7f9e650bd640 1 --2- 172.21.15.16:0/3721999904 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9e600cae20 0x7f9e601f2a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:12.036 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.027+0000 7f9e650bd640 1 -- 172.21.15.16:0/3721999904 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e58009a70 con 0x7f9e60153fe0 2023-05-22T15:43:12.037 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.027+0000 7f9e650bd640 1 --2- 172.21.15.16:0/3721999904 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e60153fe0 0x7f9e601f2320 secure :-1 s=READY pgs=1342 cs=0 l=1 rev1=1 crypto rx=0x7f9e5801cde0 tx=0x7f9e580196a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:43:12.037 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.027+0000 7f9e5d7fa640 1 -- 172.21.15.16:0/3721999904 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e58009450 con 0x7f9e60153fe0 2023-05-22T15:43:12.037 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.027+0000 7f9e5d7fa640 1 -- 172.21.15.16:0/3721999904 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e580adbf0 con 0x7f9e60153fe0 2023-05-22T15:43:12.037 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.027+0000 7f9e5d7fa640 1 -- 172.21.15.16:0/3721999904 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e580b8420 con 0x7f9e60153fe0 2023-05-22T15:43:12.037 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.027+0000 7f9e66b20640 1 -- 172.21.15.16:0/3721999904 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e6015db60 con 0x7f9e60153fe0 2023-05-22T15:43:12.037 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.031+0000 7f9e66b20640 1 -- 172.21.15.16:0/3721999904 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e6015d880 con 0x7f9e60153fe0 2023-05-22T15:43:12.037 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.031+0000 7f9e66b20640 1 -- 172.21.15.16:0/3721999904 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e6015de40 con 0x7f9e60153fe0 2023-05-22T15:43:12.038 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.031+0000 7f9e5d7fa640 1 -- 172.21.15.16:0/3721999904 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 38) v1 ==== 84389+0+0 (secure 0 0 0) 0x7f9e580b8580 con 0x7f9e60153fe0 2023-05-22T15:43:12.038 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.031+0000 7f9e5d7fa640 1 --2- 172.21.15.16:0/3721999904 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f9e44065560 0x7f9e44067a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:12.038 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.031+0000 7f9e648bc640 1 -- 172.21.15.16:0/3721999904 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f9e44065560 msgr2=0x7f9e44067a10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/36921 2023-05-22T15:43:12.038 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.031+0000 7f9e5d7fa640 1 -- 172.21.15.16:0/3721999904 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 6726+0+0 (secure 0 0 0) 0x7f9e5811e270 con 0x7f9e60153fe0 2023-05-22T15:43:12.038 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.031+0000 7f9e648bc640 1 --2- 172.21.15.16:0/3721999904 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f9e44065560 0x7f9e44067a10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-05-22T15:43:12.043 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.039+0000 7f9e5d7fa640 1 -- 172.21.15.16:0/3721999904 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f9e580ebab0 con 0x7f9e60153fe0 2023-05-22T15:43:12.119 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:43:12.115+0000 7ffb72709180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-05-22T15:43:12.131 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:43:12.127+0000 7f9700ac6180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-05-22T15:43:12.198 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.191+0000 7f9e66b20640 1 -- 172.21.15.16:0/3721999904 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f9e60150a50 con 0x7f9e60153fe0 2023-05-22T15:43:12.198 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.195+0000 7f9e5d7fa640 1 -- 172.21.15.16:0/3721999904 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v105) v1 ==== 93+0+10 (secure 0 0 0) 0x7f9e60150a50 con 0x7f9e60153fe0 2023-05-22T15:43:12.199 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2023-05-22T15:43:12.201 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.195+0000 7f9e66b20640 1 -- 172.21.15.16:0/3721999904 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f9e44065560 msgr2=0x7f9e44067a10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-05-22T15:43:12.201 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.195+0000 7f9e66b20640 1 --2- 172.21.15.16:0/3721999904 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f9e44065560 0x7f9e44067a10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:12.201 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.195+0000 7f9e66b20640 1 -- 172.21.15.16:0/3721999904 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e60153fe0 msgr2=0x7f9e601f2320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:12.201 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.195+0000 7f9e66b20640 1 --2- 172.21.15.16:0/3721999904 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e60153fe0 0x7f9e601f2320 secure :-1 s=READY pgs=1342 cs=0 l=1 rev1=1 crypto rx=0x7f9e5801cde0 tx=0x7f9e580196a0 comp rx=0 tx=0).stop 2023-05-22T15:43:12.201 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.195+0000 7f9e66b20640 1 -- 172.21.15.16:0/3721999904 shutdown_connections 2023-05-22T15:43:12.202 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.195+0000 7f9e66b20640 1 --2- 172.21.15.16:0/3721999904 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e60153fe0 0x7f9e601f2320 unknown :-1 s=CLOSED pgs=1342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:12.202 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.195+0000 7f9e66b20640 1 --2- 172.21.15.16:0/3721999904 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9e600cae20 0x7f9e601f2a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:12.202 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.195+0000 7f9e66b20640 1 --2- 172.21.15.16:0/3721999904 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f9e44065560 0x7f9e44067a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:12.203 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.195+0000 7f9e66b20640 1 --2- 172.21.15.16:0/3721999904 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9e60070430 0x7f9e601f48c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:12.203 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.195+0000 7f9e66b20640 1 -- 172.21.15.16:0/3721999904 >> 172.21.15.16:0/3721999904 conn(0x7f9e600b1110 msgr2=0x7f9e6006d3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:43:12.203 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.195+0000 7f9e66b20640 1 -- 172.21.15.16:0/3721999904 shutdown_connections 2023-05-22T15:43:12.204 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.195+0000 7f9e66b20640 1 -- 172.21.15.16:0/3721999904 wait complete. 2023-05-22T15:43:12.217 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-05-22T15:43:12.334 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.327+0000 7f5ada247640 1 Processor -- start 2023-05-22T15:43:12.334 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.327+0000 7f5ada247640 1 -- start start 2023-05-22T15:43:12.335 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.331+0000 7f5ada247640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f5ad40b2880 0x7f5ad40cbfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:12.335 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.331+0000 7f5ada247640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f5ad40cc520 0x7f5ad40c3db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:12.335 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.331+0000 7f5ada247640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ad40c6ba0 con 0x7f5ad40c67d0 2023-05-22T15:43:12.335 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.331+0000 7f5ada247640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ad415a300 con 0x7f5ad40b2880 2023-05-22T15:43:12.336 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.331+0000 7f5ada247640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ad40cfef0 con 0x7f5ad40cc520 2023-05-22T15:43:12.336 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.331+0000 7f5ad2ffd640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f5ad40b2880 0x7f5ad40cbfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:12.336 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.331+0000 7f5ad2ffd640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f5ad40b2880 0x7f5ad40cbfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45662/0 (socket says 172.21.15.16:45662) 2023-05-22T15:43:12.336 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.331+0000 7f5ad2ffd640 1 -- 172.21.15.16:0/516402537 learned_addr learned my addr 172.21.15.16:0/516402537 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:43:12.336 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.331+0000 7f5ad3fff640 1 --2- 172.21.15.16:0/516402537 >> v2:172.21.15.62:3300/0 conn(0x7f5ad40cc520 0x7f5ad40c3db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:12.337 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.331+0000 7f5ad2ffd640 1 -- 172.21.15.16:0/516402537 >> v2:172.21.15.62:3300/0 conn(0x7f5ad40cc520 msgr2=0x7f5ad40c3db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:43:12.337 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.331+0000 7f5ad2ffd640 1 --2- 172.21.15.16:0/516402537 >> v2:172.21.15.62:3300/0 conn(0x7f5ad40cc520 0x7f5ad40c3db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:12.337 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.331+0000 7f5ad2ffd640 1 -- 172.21.15.16:0/516402537 >> v1:172.21.15.62:6789/0 conn(0x7f5ad40c67d0 legacy=0x7f5ad40cf730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:12.337 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.331+0000 7f5ad2ffd640 1 -- 172.21.15.16:0/516402537 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ad40d01d0 con 0x7f5ad40b2880 2023-05-22T15:43:12.338 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.331+0000 7f5ad2ffd640 1 --2- 172.21.15.16:0/516402537 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5ad40b2880 0x7f5ad40cbfe0 secure :-1 s=READY pgs=1343 cs=0 l=1 rev1=1 crypto rx=0x7f5ad40b6360 tx=0x7f5abc01c440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49c25575c3e926b8 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:43:12.338 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.331+0000 7f5ad27fc640 1 -- 172.21.15.16:0/516402537 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5abc0ab640 con 0x7f5ad40b2880 2023-05-22T15:43:12.338 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.331+0000 7f5ad27fc640 1 -- 172.21.15.16:0/516402537 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5abc0a7070 con 0x7f5ad40b2880 2023-05-22T15:43:12.338 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.331+0000 7f5ad27fc640 1 -- 172.21.15.16:0/516402537 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5abc0ab9c0 con 0x7f5ad40b2880 2023-05-22T15:43:12.339 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.335+0000 7f5ada247640 1 -- 172.21.15.16:0/516402537 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5ad40b2880 msgr2=0x7f5ad40cbfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:12.340 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.335+0000 7f5ada247640 1 --2- 172.21.15.16:0/516402537 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5ad40b2880 0x7f5ad40cbfe0 secure :-1 s=READY pgs=1343 cs=0 l=1 rev1=1 crypto rx=0x7f5ad40b6360 tx=0x7f5abc01c440 comp rx=0 tx=0).stop 2023-05-22T15:43:12.340 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.335+0000 7f5ada247640 1 -- 172.21.15.16:0/516402537 shutdown_connections 2023-05-22T15:43:12.340 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.335+0000 7f5ada247640 1 --2- 172.21.15.16:0/516402537 >> v2:172.21.15.62:3300/0 conn(0x7f5ad40cc520 0x7f5ad40c3db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:12.340 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.335+0000 7f5ada247640 1 --2- 172.21.15.16:0/516402537 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5ad40b2880 0x7f5ad40cbfe0 unknown :-1 s=CLOSED pgs=1343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:12.340 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.335+0000 7f5ada247640 1 -- 172.21.15.16:0/516402537 >> 172.21.15.16:0/516402537 conn(0x7f5ad406f5a0 msgr2=0x7f5ad406f9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:43:12.340 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.335+0000 7f5ada247640 1 -- 172.21.15.16:0/516402537 shutdown_connections 2023-05-22T15:43:12.341 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.335+0000 7f5ada247640 1 -- 172.21.15.16:0/516402537 wait complete. 2023-05-22T15:43:12.341 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.335+0000 7f5ada247640 1 Processor -- start 2023-05-22T15:43:12.341 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.335+0000 7f5ada247640 1 -- start start 2023-05-22T15:43:12.342 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.335+0000 7f5ada247640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5ad40b2880 0x7f5ad414ef10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:12.342 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.335+0000 7f5ada247640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5ad40c67d0 0x7f5ad414d730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:12.343 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.335+0000 7f5ada247640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5ad40cc520 0x7f5ad40b0600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:12.343 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.335+0000 7f5ada247640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ad40b10a0 con 0x7f5ad40cc520 2023-05-22T15:43:12.343 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.335+0000 7f5ada247640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ad414e190 con 0x7f5ad40b2880 2023-05-22T15:43:12.343 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.335+0000 7f5ada247640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ad414e430 con 0x7f5ad40c67d0 2023-05-22T15:43:12.343 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.335+0000 7f5ad3fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5ad40cc520 0x7f5ad40b0600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:12.343 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.335+0000 7f5ad3fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5ad40cc520 0x7f5ad40b0600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45664/0 (socket says 172.21.15.16:45664) 2023-05-22T15:43:12.343 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.335+0000 7f5ad3fff640 1 -- 172.21.15.16:0/364397673 learned_addr learned my addr 172.21.15.16:0/364397673 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:43:12.343 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.335+0000 7f5ad37fe640 1 --2- 172.21.15.16:0/364397673 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5ad40b2880 0x7f5ad414ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:12.344 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.335+0000 7f5ad2ffd640 1 --2- 172.21.15.16:0/364397673 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5ad40c67d0 0x7f5ad414d730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:12.344 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.335+0000 7f5ad3fff640 1 -- 172.21.15.16:0/364397673 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5ad40c67d0 msgr2=0x7f5ad414d730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:12.344 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.335+0000 7f5ad3fff640 1 --2- 172.21.15.16:0/364397673 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5ad40c67d0 0x7f5ad414d730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:12.344 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.335+0000 7f5ad3fff640 1 -- 172.21.15.16:0/364397673 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5ad40b2880 msgr2=0x7f5ad414ef10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:12.344 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.335+0000 7f5ad3fff640 1 --2- 172.21.15.16:0/364397673 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5ad40b2880 0x7f5ad414ef10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:12.344 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.335+0000 7f5ad3fff640 1 -- 172.21.15.16:0/364397673 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5abc0094d0 con 0x7f5ad40cc520 2023-05-22T15:43:12.344 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.335+0000 7f5ad37fe640 1 --2- 172.21.15.16:0/364397673 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5ad40b2880 0x7f5ad414ef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T15:43:12.345 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.335+0000 7f5ad3fff640 1 --2- 172.21.15.16:0/364397673 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5ad40cc520 0x7f5ad40b0600 secure :-1 s=READY pgs=1344 cs=0 l=1 rev1=1 crypto rx=0x7f5ac400efc0 tx=0x7f5ac400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:43:12.345 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.335+0000 7f5acbfff640 1 -- 172.21.15.16:0/364397673 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ac4014480 con 0x7f5ad40cc520 2023-05-22T15:43:12.345 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.335+0000 7f5acbfff640 1 -- 172.21.15.16:0/364397673 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ac400f040 con 0x7f5ad40cc520 2023-05-22T15:43:12.345 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.335+0000 7f5ada247640 1 -- 172.21.15.16:0/364397673 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ad40d0f90 con 0x7f5ad40cc520 2023-05-22T15:43:12.345 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.335+0000 7f5acbfff640 1 -- 172.21.15.16:0/364397673 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ac4014820 con 0x7f5ad40cc520 2023-05-22T15:43:12.345 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.335+0000 7f5ada247640 1 -- 172.21.15.16:0/364397673 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ad40d0a70 con 0x7f5ad40cc520 2023-05-22T15:43:12.345 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.339+0000 7f5ada247640 1 -- 172.21.15.16:0/364397673 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ad40cc8f0 con 0x7f5ad40cc520 2023-05-22T15:43:12.346 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.339+0000 7f5acbfff640 1 -- 172.21.15.16:0/364397673 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 38) v1 ==== 84389+0+0 (secure 0 0 0) 0x7f5ac4014980 con 0x7f5ad40cc520 2023-05-22T15:43:12.346 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.339+0000 7f5acbfff640 1 --2- 172.21.15.16:0/364397673 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f5ab4065530 0x7f5ab40679e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:12.346 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.339+0000 7f5ad37fe640 1 -- 172.21.15.16:0/364397673 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f5ab4065530 msgr2=0x7f5ab40679e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/36921 2023-05-22T15:43:12.346 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.339+0000 7f5acbfff640 1 -- 172.21.15.16:0/364397673 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 6726+0+0 (secure 0 0 0) 0x7f5ac4083e50 con 0x7f5ad40cc520 2023-05-22T15:43:12.347 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.339+0000 7f5ad37fe640 1 --2- 172.21.15.16:0/364397673 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f5ab4065530 0x7f5ab40679e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-05-22T15:43:12.347 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.343+0000 7f5acbfff640 1 -- 172.21.15.16:0/364397673 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f5ac40514a0 con 0x7f5ad40cc520 2023-05-22T15:43:12.501 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.495+0000 7f5ada247640 1 -- 172.21.15.16:0/364397673 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f5ad404dbe0 con 0x7f5ad40cc520 2023-05-22T15:43:12.501 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.495+0000 7f5acbfff640 1 -- 172.21.15.16:0/364397673 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v105) v1 ==== 92+0+10 (secure 0 0 0) 0x7f5ac40180a0 con 0x7f5ad40cc520 2023-05-22T15:43:12.502 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 8 2023-05-22T15:43:12.504 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.499+0000 7f5ada247640 1 -- 172.21.15.16:0/364397673 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f5ab4065530 msgr2=0x7f5ab40679e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-05-22T15:43:12.504 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.499+0000 7f5ada247640 1 --2- 172.21.15.16:0/364397673 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f5ab4065530 0x7f5ab40679e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:12.504 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.499+0000 7f5ada247640 1 -- 172.21.15.16:0/364397673 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5ad40cc520 msgr2=0x7f5ad40b0600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:12.504 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.499+0000 7f5ada247640 1 --2- 172.21.15.16:0/364397673 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5ad40cc520 0x7f5ad40b0600 secure :-1 s=READY pgs=1344 cs=0 l=1 rev1=1 crypto rx=0x7f5ac400efc0 tx=0x7f5ac400c490 comp rx=0 tx=0).stop 2023-05-22T15:43:12.504 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.499+0000 7f5ada247640 1 -- 172.21.15.16:0/364397673 shutdown_connections 2023-05-22T15:43:12.504 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.499+0000 7f5ada247640 1 --2- 172.21.15.16:0/364397673 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5ad40cc520 0x7f5ad40b0600 unknown :-1 s=CLOSED pgs=1344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:12.505 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.499+0000 7f5ada247640 1 --2- 172.21.15.16:0/364397673 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f5ab4065530 0x7f5ab40679e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:12.505 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.499+0000 7f5ada247640 1 --2- 172.21.15.16:0/364397673 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5ad40c67d0 0x7f5ad414d730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:12.505 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.499+0000 7f5ada247640 1 --2- 172.21.15.16:0/364397673 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5ad40b2880 0x7f5ad414ef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:12.505 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.499+0000 7f5ada247640 1 -- 172.21.15.16:0/364397673 >> 172.21.15.16:0/364397673 conn(0x7f5ad406f5a0 msgr2=0x7f5ad406f970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:43:12.505 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.499+0000 7f5ada247640 1 -- 172.21.15.16:0/364397673 shutdown_connections 2023-05-22T15:43:12.505 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.499+0000 7f5ada247640 1 -- 172.21.15.16:0/364397673 wait complete. 2023-05-22T15:43:12.521 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-05-22T15:43:12.562 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:43:12.559+0000 7ffb72709180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-05-22T15:43:12.586 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:43:12.583+0000 7f9700ac6180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-05-22T15:43:12.617 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:43:12.615+0000 7ffb72709180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-05-22T15:43:12.635 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.631+0000 7f224923f640 1 Processor -- start 2023-05-22T15:43:12.636 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.631+0000 7f224923f640 1 -- start start 2023-05-22T15:43:12.637 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.631+0000 7f224923f640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2244070760 0x7f2244070be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:12.637 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.631+0000 7f224923f640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f224406eb90 con 0x7f2244158a10 2023-05-22T15:43:12.637 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.631+0000 7f224923f640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2244158de0 con 0x7f224406c6e0 2023-05-22T15:43:12.637 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.631+0000 7f224923f640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f224406e060 con 0x7f2244070760 2023-05-22T15:43:12.637 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.631+0000 7f224259c640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2244070760 0x7f2244070be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:12.637 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.631+0000 7f2242d9d640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f2244158a10 0x7f224406e480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:57088/0 (socket says 172.21.15.16:57088) 2023-05-22T15:43:12.638 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.631+0000 7f2242d9d640 1 -- 172.21.15.16:0/1586513099 learned_addr learned my addr 172.21.15.16:0/1586513099 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:43:12.638 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.631+0000 7f224259c640 1 -- 172.21.15.16:0/1586513099 >> v1:172.21.15.62:6789/0 conn(0x7f224406c6e0 legacy=0x7f22440c9a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:12.638 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.631+0000 7f224259c640 1 -- 172.21.15.16:0/1586513099 >> v1:172.21.15.16:6789/0 conn(0x7f2244158a10 legacy=0x7f224406e480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:12.639 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.631+0000 7f224259c640 1 -- 172.21.15.16:0/1586513099 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22440711e0 con 0x7f2244070760 2023-05-22T15:43:12.639 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.631+0000 7f2241d9b640 1 -- 172.21.15.16:0/1586513099 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4239279650 0 0) 0x7f224406eb90 con 0x7f2244158a10 2023-05-22T15:43:12.639 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.635+0000 7f224259c640 1 --2- 172.21.15.16:0/1586513099 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2244070760 0x7f2244070be0 secure :-1 s=READY pgs=1346 cs=0 l=1 rev1=1 crypto rx=0x7f22380097c0 tx=0x7f223801c3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd82f13a694e0db4 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:43:12.639 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.635+0000 7f2241d9b640 1 -- 172.21.15.16:0/1586513099 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22380ab600 con 0x7f2244070760 2023-05-22T15:43:12.640 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.635+0000 7f2241d9b640 1 -- 172.21.15.16:0/1586513099 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22380a7050 con 0x7f2244070760 2023-05-22T15:43:12.640 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.635+0000 7f2241d9b640 1 -- 172.21.15.16:0/1586513099 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22380ab980 con 0x7f2244070760 2023-05-22T15:43:12.640 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.635+0000 7f224923f640 1 -- 172.21.15.16:0/1586513099 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2244070760 msgr2=0x7f2244070be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:12.641 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.635+0000 7f224923f640 1 --2- 172.21.15.16:0/1586513099 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2244070760 0x7f2244070be0 secure :-1 s=READY pgs=1346 cs=0 l=1 rev1=1 crypto rx=0x7f22380097c0 tx=0x7f223801c3f0 comp rx=0 tx=0).stop 2023-05-22T15:43:12.641 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.635+0000 7f224923f640 1 -- 172.21.15.16:0/1586513099 shutdown_connections 2023-05-22T15:43:12.641 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.635+0000 7f224923f640 1 --2- 172.21.15.16:0/1586513099 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2244070760 0x7f2244070be0 unknown :-1 s=CLOSED pgs=1346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:12.641 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.635+0000 7f224923f640 1 -- 172.21.15.16:0/1586513099 >> 172.21.15.16:0/1586513099 conn(0x7f22440cc870 msgr2=0x7f22440cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:43:12.641 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.635+0000 7f224923f640 1 -- 172.21.15.16:0/1586513099 shutdown_connections 2023-05-22T15:43:12.641 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.635+0000 7f224923f640 1 -- 172.21.15.16:0/1586513099 wait complete. 2023-05-22T15:43:12.643 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.635+0000 7f224923f640 1 Processor -- start 2023-05-22T15:43:12.643 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.635+0000 7f224923f640 1 -- start start 2023-05-22T15:43:12.644 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.635+0000 7f224923f640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f224406c6e0 0x7f224414dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:12.644 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.635+0000 7f224923f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2244070760 0x7f22440b4ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:12.644 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.635+0000 7f224923f640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2244158a10 0x7f224415ae30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:12.644 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.635+0000 7f224923f640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f224415b370 con 0x7f2244070760 2023-05-22T15:43:12.644 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.635+0000 7f224923f640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22440b25f0 con 0x7f224406c6e0 2023-05-22T15:43:12.644 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.635+0000 7f224923f640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22440b2030 con 0x7f2244158a10 2023-05-22T15:43:12.645 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.639+0000 7f224259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2244070760 0x7f22440b4ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:12.645 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.639+0000 7f224259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2244070760 0x7f22440b4ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45682/0 (socket says 172.21.15.16:45682) 2023-05-22T15:43:12.645 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.639+0000 7f224259c640 1 -- 172.21.15.16:0/1676073252 learned_addr learned my addr 172.21.15.16:0/1676073252 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:43:12.645 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.639+0000 7f2242d9d640 1 --2- 172.21.15.16:0/1676073252 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f224406c6e0 0x7f224414dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:12.645 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.639+0000 7f224359e640 1 --2- 172.21.15.16:0/1676073252 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2244158a10 0x7f224415ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:12.645 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.639+0000 7f224259c640 1 -- 172.21.15.16:0/1676073252 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2244158a10 msgr2=0x7f224415ae30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:12.645 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.639+0000 7f224259c640 1 --2- 172.21.15.16:0/1676073252 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2244158a10 0x7f224415ae30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:12.645 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.639+0000 7f224259c640 1 -- 172.21.15.16:0/1676073252 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f224406c6e0 msgr2=0x7f224414dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:12.646 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.639+0000 7f224259c640 1 --2- 172.21.15.16:0/1676073252 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f224406c6e0 0x7f224414dcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:12.646 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.639+0000 7f224259c640 1 -- 172.21.15.16:0/1676073252 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2238009420 con 0x7f2244070760 2023-05-22T15:43:12.646 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.639+0000 7f224259c640 1 --2- 172.21.15.16:0/1676073252 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2244070760 0x7f22440b4ab0 secure :-1 s=READY pgs=1347 cs=0 l=1 rev1=1 crypto rx=0x7f2238019000 tx=0x7f223801ac30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:43:12.646 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.639+0000 7f222b7fe640 1 -- 172.21.15.16:0/1676073252 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22380b7650 con 0x7f2244070760 2023-05-22T15:43:12.646 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.639+0000 7f222b7fe640 1 -- 172.21.15.16:0/1676073252 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22380abe90 con 0x7f2244070760 2023-05-22T15:43:12.646 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.639+0000 7f224923f640 1 -- 172.21.15.16:0/1676073252 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f224406cf10 con 0x7f2244070760 2023-05-22T15:43:12.646 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.639+0000 7f222b7fe640 1 -- 172.21.15.16:0/1676073252 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22380b7a10 con 0x7f2244070760 2023-05-22T15:43:12.646 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.639+0000 7f224923f640 1 -- 172.21.15.16:0/1676073252 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22440cfe40 con 0x7f2244070760 2023-05-22T15:43:12.647 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.639+0000 7f224923f640 1 -- 172.21.15.16:0/1676073252 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2244167120 con 0x7f2244070760 2023-05-22T15:43:12.647 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.639+0000 7f222b7fe640 1 -- 172.21.15.16:0/1676073252 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 38) v1 ==== 84389+0+0 (secure 0 0 0) 0x7f22380b7b70 con 0x7f2244070760 2023-05-22T15:43:12.647 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.639+0000 7f222b7fe640 1 --2- 172.21.15.16:0/1676073252 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f2230065560 0x7f2230067a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:12.648 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.639+0000 7f2242d9d640 1 -- 172.21.15.16:0/1676073252 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f2230065560 msgr2=0x7f2230067a10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/36921 2023-05-22T15:43:12.648 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:43:12.639+0000 7f9700ac6180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-05-22T15:43:12.649 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.639+0000 7f222b7fe640 1 -- 172.21.15.16:0/1676073252 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 6726+0+0 (secure 0 0 0) 0x7f223811db50 con 0x7f2244070760 2023-05-22T15:43:12.649 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.639+0000 7f2242d9d640 1 --2- 172.21.15.16:0/1676073252 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f2230065560 0x7f2230067a10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-05-22T15:43:12.650 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.643+0000 7f222b7fe640 1 -- 172.21.15.16:0/1676073252 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f223811de60 con 0x7f2244070760 2023-05-22T15:43:12.670 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:43:12.667+0000 7ffb72709180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-05-22T15:43:12.696 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:43:12.691+0000 7f9700ac6180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-05-22T15:43:12.748 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:43:12.743+0000 7ffb72709180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-05-22T15:43:12.775 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:43:12.771+0000 7f9700ac6180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-05-22T15:43:12.799 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:43:12.795+0000 7ffb72709180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-05-22T15:43:12.801 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.795+0000 7f224923f640 1 -- 172.21.15.16:0/1676073252 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f224404dbe0 con 0x7f2244070760 2023-05-22T15:43:12.802 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.795+0000 7f222b7fe640 1 -- 172.21.15.16:0/1676073252 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v105) v1 ==== 104+0+11 (secure 0 0 0) 0x7f22380eb280 con 0x7f2244070760 2023-05-22T15:43:12.802 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2023-05-22T15:43:12.804 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.799+0000 7f224923f640 1 -- 172.21.15.16:0/1676073252 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f2230065560 msgr2=0x7f2230067a10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-05-22T15:43:12.805 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.799+0000 7f224923f640 1 --2- 172.21.15.16:0/1676073252 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f2230065560 0x7f2230067a10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:12.805 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.799+0000 7f224923f640 1 -- 172.21.15.16:0/1676073252 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2244070760 msgr2=0x7f22440b4ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:12.805 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.799+0000 7f224923f640 1 --2- 172.21.15.16:0/1676073252 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2244070760 0x7f22440b4ab0 secure :-1 s=READY pgs=1347 cs=0 l=1 rev1=1 crypto rx=0x7f2238019000 tx=0x7f223801ac30 comp rx=0 tx=0).stop 2023-05-22T15:43:12.805 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.799+0000 7f224923f640 1 -- 172.21.15.16:0/1676073252 shutdown_connections 2023-05-22T15:43:12.805 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.799+0000 7f224923f640 1 --2- 172.21.15.16:0/1676073252 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f2230065560 0x7f2230067a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:12.805 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.799+0000 7f224923f640 1 --2- 172.21.15.16:0/1676073252 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2244158a10 0x7f224415ae30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:12.805 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.799+0000 7f224923f640 1 --2- 172.21.15.16:0/1676073252 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2244070760 0x7f22440b4ab0 unknown :-1 s=CLOSED pgs=1347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:12.806 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.799+0000 7f224923f640 1 --2- 172.21.15.16:0/1676073252 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f224406c6e0 0x7f224414dcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:12.806 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.799+0000 7f224923f640 1 -- 172.21.15.16:0/1676073252 >> 172.21.15.16:0/1676073252 conn(0x7f22440cc870 msgr2=0x7f22440aebc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:43:12.806 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.799+0000 7f224923f640 1 -- 172.21.15.16:0/1676073252 shutdown_connections 2023-05-22T15:43:12.806 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.799+0000 7f224923f640 1 -- 172.21.15.16:0/1676073252 wait complete. 2023-05-22T15:43:12.821 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-05-22T15:43:12.827 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:43:12.823+0000 7f9700ac6180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-05-22T15:43:12.875 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:43:12.871+0000 7ffb72709180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-05-22T15:43:12.908 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:43:12.903+0000 7f9700ac6180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-05-22T15:43:12.927 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:43:12.923+0000 7ffb72709180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-05-22T15:43:12.954 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.947+0000 7f75bc4c1640 1 Processor -- start 2023-05-22T15:43:12.954 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.947+0000 7f75bc4c1640 1 -- start start 2023-05-22T15:43:12.956 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.951+0000 7f75bc4c1640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f75b40c67d0 0x7f75b40cf730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:12.956 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.951+0000 7f75bc4c1640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f75b40cc6f0 0x7f75b40c3db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:12.957 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.951+0000 7f75bc4c1640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75b40cfc70 con 0x7f75b40b2880 2023-05-22T15:43:12.957 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.951+0000 7f75bc4c1640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75b415a300 con 0x7f75b40cc6f0 2023-05-22T15:43:12.957 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.951+0000 7f75bc4c1640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75b40c6ba0 con 0x7f75b40c67d0 2023-05-22T15:43:12.957 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.951+0000 7f75baa5e640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f75b40cc6f0 0x7f75b40c3db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:12.958 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.951+0000 7f75baa5e640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f75b40cc6f0 0x7f75b40c3db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45684/0 (socket says 172.21.15.16:45684) 2023-05-22T15:43:12.958 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.951+0000 7f75baa5e640 1 -- 172.21.15.16:0/1980813237 learned_addr learned my addr 172.21.15.16:0/1980813237 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:43:12.958 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.951+0000 7f75ba25d640 1 --2- 172.21.15.16:0/1980813237 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f75b40c67d0 0x7f75b40cf730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:12.958 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.951+0000 7f75baa5e640 1 -- 172.21.15.16:0/1980813237 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f75b40c67d0 msgr2=0x7f75b40cf730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:43:12.959 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.951+0000 7f75baa5e640 1 --2- 172.21.15.16:0/1980813237 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f75b40c67d0 0x7f75b40cf730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:12.959 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.951+0000 7f75baa5e640 1 -- 172.21.15.16:0/1980813237 >> v1:172.21.15.62:6789/0 conn(0x7f75b40b2880 legacy=0x7f75b40cbfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:12.959 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.951+0000 7f75baa5e640 1 -- 172.21.15.16:0/1980813237 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75b40cfef0 con 0x7f75b40cc6f0 2023-05-22T15:43:12.963 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.951+0000 7f75baa5e640 1 --2- 172.21.15.16:0/1980813237 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f75b40cc6f0 0x7f75b40c3db0 secure :-1 s=READY pgs=1348 cs=0 l=1 rev1=1 crypto rx=0x7f75ac009e50 tx=0x7f75ac01c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3379a2fc9f910a96 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:43:12.963 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.951+0000 7f75b925b640 1 -- 172.21.15.16:0/1980813237 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75ac0a5810 con 0x7f75b40cc6f0 2023-05-22T15:43:12.963 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.951+0000 7f75b925b640 1 -- 172.21.15.16:0/1980813237 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75ac0a5970 con 0x7f75b40cc6f0 2023-05-22T15:43:12.964 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.951+0000 7f75b925b640 1 -- 172.21.15.16:0/1980813237 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75ac0a5d00 con 0x7f75b40cc6f0 2023-05-22T15:43:12.964 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.951+0000 7f75bc4c1640 1 -- 172.21.15.16:0/1980813237 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f75b40cc6f0 msgr2=0x7f75b40c3db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:12.964 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.951+0000 7f75bc4c1640 1 --2- 172.21.15.16:0/1980813237 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f75b40cc6f0 0x7f75b40c3db0 secure :-1 s=READY pgs=1348 cs=0 l=1 rev1=1 crypto rx=0x7f75ac009e50 tx=0x7f75ac01c950 comp rx=0 tx=0).stop 2023-05-22T15:43:12.964 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.951+0000 7f75bc4c1640 1 -- 172.21.15.16:0/1980813237 shutdown_connections 2023-05-22T15:43:12.965 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.951+0000 7f75bc4c1640 1 --2- 172.21.15.16:0/1980813237 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f75b40cc6f0 0x7f75b40c3db0 unknown :-1 s=CLOSED pgs=1348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:12.965 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.951+0000 7f75bc4c1640 1 --2- 172.21.15.16:0/1980813237 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f75b40c67d0 0x7f75b40cf730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:12.965 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.951+0000 7f75bc4c1640 1 -- 172.21.15.16:0/1980813237 >> 172.21.15.16:0/1980813237 conn(0x7f75b406f5a0 msgr2=0x7f75b406f9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:43:12.965 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.951+0000 7f75bc4c1640 1 -- 172.21.15.16:0/1980813237 shutdown_connections 2023-05-22T15:43:12.966 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.951+0000 7f75bc4c1640 1 -- 172.21.15.16:0/1980813237 wait complete. 2023-05-22T15:43:12.966 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.955+0000 7f75bc4c1640 1 Processor -- start 2023-05-22T15:43:12.966 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.955+0000 7f75bc4c1640 1 -- start start 2023-05-22T15:43:12.967 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.955+0000 7f75bc4c1640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f75b40b2880 0x7f75b40b3090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:12.967 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.955+0000 7f75bc4c1640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f75b40c67d0 0x7f75b4153190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:12.967 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.955+0000 7f75bc4c1640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f75b40cc6f0 0x7f75b40b4a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:12.967 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.955+0000 7f75bc4c1640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75b41539d0 con 0x7f75b40cc6f0 2023-05-22T15:43:12.968 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.955+0000 7f75bc4c1640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75b40b35d0 con 0x7f75b40c67d0 2023-05-22T15:43:12.968 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.955+0000 7f75bc4c1640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75b415ca60 con 0x7f75b40b2880 2023-05-22T15:43:12.968 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.955+0000 7f75baa5e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f75b40cc6f0 0x7f75b40b4a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:12.968 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.955+0000 7f75baa5e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f75b40cc6f0 0x7f75b40b4a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45686/0 (socket says 172.21.15.16:45686) 2023-05-22T15:43:12.968 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.955+0000 7f75baa5e640 1 -- 172.21.15.16:0/705883303 learned_addr learned my addr 172.21.15.16:0/705883303 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:43:12.968 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.955+0000 7f75ba25d640 1 --2- 172.21.15.16:0/705883303 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f75b40b2880 0x7f75b40b3090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:12.968 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.955+0000 7f75b9a5c640 1 --2- 172.21.15.16:0/705883303 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f75b40c67d0 0x7f75b4153190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:12.969 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.955+0000 7f75baa5e640 1 -- 172.21.15.16:0/705883303 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f75b40b2880 msgr2=0x7f75b40b3090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:12.969 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.955+0000 7f75baa5e640 1 --2- 172.21.15.16:0/705883303 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f75b40b2880 0x7f75b40b3090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:12.969 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.955+0000 7f75baa5e640 1 -- 172.21.15.16:0/705883303 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f75b40c67d0 msgr2=0x7f75b4153190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:12.969 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.955+0000 7f75baa5e640 1 --2- 172.21.15.16:0/705883303 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f75b40c67d0 0x7f75b4153190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:12.969 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.955+0000 7f75baa5e640 1 -- 172.21.15.16:0/705883303 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75ac009b30 con 0x7f75b40cc6f0 2023-05-22T15:43:12.969 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.955+0000 7f75b9a5c640 1 --2- 172.21.15.16:0/705883303 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f75b40c67d0 0x7f75b4153190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:43:12.969 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.955+0000 7f75baa5e640 1 --2- 172.21.15.16:0/705883303 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f75b40cc6f0 0x7f75b40b4a60 secure :-1 s=READY pgs=1349 cs=0 l=1 rev1=1 crypto rx=0x7f75ac0af040 tx=0x7f75ac0196f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:43:12.970 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.955+0000 7f75aaffd640 1 -- 172.21.15.16:0/705883303 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75ac0b8670 con 0x7f75b40cc6f0 2023-05-22T15:43:12.970 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.955+0000 7f75aaffd640 1 -- 172.21.15.16:0/705883303 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75ac0b87d0 con 0x7f75b40cc6f0 2023-05-22T15:43:12.970 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.955+0000 7f75bc4c1640 1 -- 172.21.15.16:0/705883303 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75b40d0a70 con 0x7f75b40cc6f0 2023-05-22T15:43:12.970 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.955+0000 7f75aaffd640 1 -- 172.21.15.16:0/705883303 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75ac0b8b60 con 0x7f75b40cc6f0 2023-05-22T15:43:12.970 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.955+0000 7f75bc4c1640 1 -- 172.21.15.16:0/705883303 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75b40d0790 con 0x7f75b40cc6f0 2023-05-22T15:43:12.970 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:43:12.955+0000 7f9700ac6180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-05-22T15:43:12.971 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.955+0000 7f75bc4c1640 1 -- 172.21.15.16:0/705883303 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75b40d0f90 con 0x7f75b40cc6f0 2023-05-22T15:43:12.971 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.959+0000 7f75aaffd640 1 -- 172.21.15.16:0/705883303 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 38) v1 ==== 84389+0+0 (secure 0 0 0) 0x7f75ac0b8cc0 con 0x7f75b40cc6f0 2023-05-22T15:43:12.972 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.959+0000 7f75aaffd640 1 --2- 172.21.15.16:0/705883303 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f7598065530 0x7f75980679e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:12.972 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.959+0000 7f75ba25d640 1 -- 172.21.15.16:0/705883303 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f7598065530 msgr2=0x7f75980679e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/36921 2023-05-22T15:43:12.972 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.959+0000 7f75aaffd640 1 -- 172.21.15.16:0/705883303 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 6726+0+0 (secure 0 0 0) 0x7f75ac11e470 con 0x7f75b40cc6f0 2023-05-22T15:43:12.972 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.959+0000 7f75ba25d640 1 --2- 172.21.15.16:0/705883303 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f7598065530 0x7f75980679e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-05-22T15:43:12.972 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:12.963+0000 7f75aaffd640 1 -- 172.21.15.16:0/705883303 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f75ac0ebbd0 con 0x7f75b40cc6f0 2023-05-22T15:43:13.060 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:43:13.055+0000 7ffb72709180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-05-22T15:43:13.095 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:43:13.091+0000 7f9700ac6180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-05-22T15:43:13.113 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:43:13.111+0000 7ffb72709180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-05-22T15:43:13.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:13.107+0000 7f75bc4c1640 1 -- 172.21.15.16:0/705883303 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f75b404dbe0 con 0x7f75b40cc6f0 2023-05-22T15:43:13.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:13.107+0000 7f75aaffd640 1 -- 172.21.15.16:0/705883303 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v105) v1 ==== 100+0+11 (secure 0 0 0) 0x7f75ac0ef1f0 con 0x7f75b40cc6f0 2023-05-22T15:43:13.114 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 64 2023-05-22T15:43:13.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:13.111+0000 7f75bc4c1640 1 -- 172.21.15.16:0/705883303 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f7598065530 msgr2=0x7f75980679e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-05-22T15:43:13.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:13.111+0000 7f75bc4c1640 1 --2- 172.21.15.16:0/705883303 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f7598065530 0x7f75980679e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:13.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:13.111+0000 7f75bc4c1640 1 -- 172.21.15.16:0/705883303 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f75b40cc6f0 msgr2=0x7f75b40b4a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:13.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:13.111+0000 7f75bc4c1640 1 --2- 172.21.15.16:0/705883303 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f75b40cc6f0 0x7f75b40b4a60 secure :-1 s=READY pgs=1349 cs=0 l=1 rev1=1 crypto rx=0x7f75ac0af040 tx=0x7f75ac0196f0 comp rx=0 tx=0).stop 2023-05-22T15:43:13.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:13.111+0000 7f75bc4c1640 1 -- 172.21.15.16:0/705883303 shutdown_connections 2023-05-22T15:43:13.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:13.111+0000 7f75bc4c1640 1 --2- 172.21.15.16:0/705883303 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f75b40cc6f0 0x7f75b40b4a60 unknown :-1 s=CLOSED pgs=1349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:13.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:13.111+0000 7f75bc4c1640 1 --2- 172.21.15.16:0/705883303 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f75b40c67d0 0x7f75b4153190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:13.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:13.111+0000 7f75bc4c1640 1 --2- 172.21.15.16:0/705883303 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f7598065530 0x7f75980679e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:13.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:13.111+0000 7f75bc4c1640 1 --2- 172.21.15.16:0/705883303 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f75b40b2880 0x7f75b40b3090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:13.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:13.111+0000 7f75bc4c1640 1 -- 172.21.15.16:0/705883303 >> 172.21.15.16:0/705883303 conn(0x7f75b406f5a0 msgr2=0x7f75b41562a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:43:13.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:13.111+0000 7f75bc4c1640 1 -- 172.21.15.16:0/705883303 shutdown_connections 2023-05-22T15:43:13.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:13.111+0000 7f75bc4c1640 1 -- 172.21.15.16:0/705883303 wait complete. 2023-05-22T15:43:13.133 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-05-22T15:43:13.150 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:43:13.143+0000 7f9700ac6180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-05-22T15:43:13.271 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:13.267+0000 7fb7df0cd640 1 Processor -- start 2023-05-22T15:43:13.272 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:13.267+0000 7fb7df0cd640 1 -- start start 2023-05-22T15:43:13.273 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:13.267+0000 7fb7df0cd640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb7d80cf870 0x7fb7d806db00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:13.273 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:13.267+0000 7fb7df0cd640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb7d806e740 0x7fb7d806ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:13.273 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:13.267+0000 7fb7df0cd640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7d806e040 con 0x7fb7d815cfa0 2023-05-22T15:43:13.273 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:13.267+0000 7fb7df0cd640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7d80cb9f0 con 0x7fb7d80cf870 2023-05-22T15:43:13.273 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:13.267+0000 7fb7df0cd640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7d806f100 con 0x7fb7d806e740 2023-05-22T15:43:13.273 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:13.267+0000 7fb7d7fff640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb7d80cf870 0x7fb7d806db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:13.274 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:13.267+0000 7fb7dce69640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fb7d815cfa0 0x7fb7d80b1fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:57096/0 (socket says 172.21.15.16:57096) 2023-05-22T15:43:13.274 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:13.267+0000 7fb7dce69640 1 -- 172.21.15.16:0/2912063425 learned_addr learned my addr 172.21.15.16:0/2912063425 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:43:13.274 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:13.267+0000 7fb7d7fff640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb7d80cf870 0x7fb7d806db00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45688/0 (socket says 172.21.15.16:45688) 2023-05-22T15:43:13.274 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:13.267+0000 7fb7dd66a640 1 --2- 172.21.15.16:0/2912063425 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb7d806e740 0x7fb7d806ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:13.274 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:13.267+0000 7fb7d7fff640 1 -- 172.21.15.16:0/2912063425 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb7d806e740 msgr2=0x7fb7d806ebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:43:13.274 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:13.267+0000 7fb7d7fff640 1 --2- 172.21.15.16:0/2912063425 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb7d806e740 0x7fb7d806ebc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:13.274 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:13.267+0000 7fb7d7fff640 1 -- 172.21.15.16:0/2912063425 >> v1:172.21.15.16:6789/0 conn(0x7fb7d815cfa0 legacy=0x7fb7d80b1fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:13.274 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:13.267+0000 7fb7d7fff640 1 -- 172.21.15.16:0/2912063425 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7d806e490 con 0x7fb7d80cf870 2023-05-22T15:43:13.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:13.267+0000 7fb7d77fe640 1 -- 172.21.15.16:0/2912063425 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 565469911 0 0) 0x7fb7c4001120 con 0x7fb7d815cfa0 2023-05-22T15:43:13.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:13.271+0000 7fb7d7fff640 1 --2- 172.21.15.16:0/2912063425 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb7d80cf870 0x7fb7d806db00 secure :-1 s=READY pgs=1351 cs=0 l=1 rev1=1 crypto rx=0x7fb7cc0097c0 tx=0x7fb7cc01c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81a90bbb7be0a73e server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:43:13.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:13.271+0000 7fb7d77fe640 1 -- 172.21.15.16:0/2912063425 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7cc0ab650 con 0x7fb7d80cf870 2023-05-22T15:43:13.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:13.271+0000 7fb7d77fe640 1 -- 172.21.15.16:0/2912063425 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7cc0ab7b0 con 0x7fb7d80cf870 2023-05-22T15:43:13.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:13.271+0000 7fb7d77fe640 1 -- 172.21.15.16:0/2912063425 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7cc0abb40 con 0x7fb7d80cf870 2023-05-22T15:43:13.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:13.271+0000 7fb7df0cd640 1 -- 172.21.15.16:0/2912063425 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb7d80cf870 msgr2=0x7fb7d806db00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:13.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:13.271+0000 7fb7df0cd640 1 --2- 172.21.15.16:0/2912063425 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb7d80cf870 0x7fb7d806db00 secure :-1 s=READY pgs=1351 cs=0 l=1 rev1=1 crypto rx=0x7fb7cc0097c0 tx=0x7fb7cc01c450 comp rx=0 tx=0).stop 2023-05-22T15:43:13.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:13.271+0000 7fb7df0cd640 1 -- 172.21.15.16:0/2912063425 shutdown_connections 2023-05-22T15:43:13.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:13.271+0000 7fb7df0cd640 1 --2- 172.21.15.16:0/2912063425 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb7d806e740 0x7fb7d806ebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:13.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:13.271+0000 7fb7df0cd640 1 --2- 172.21.15.16:0/2912063425 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb7d80cf870 0x7fb7d806db00 unknown :-1 s=CLOSED pgs=1351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:13.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:13.271+0000 7fb7df0cd640 1 -- 172.21.15.16:0/2912063425 >> 172.21.15.16:0/2912063425 conn(0x7fb7d80cc870 msgr2=0x7fb7d80cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:43:13.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:13.271+0000 7fb7df0cd640 1 -- 172.21.15.16:0/2912063425 shutdown_connections 2023-05-22T15:43:13.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:13.271+0000 7fb7df0cd640 1 -- 172.21.15.16:0/2912063425 wait complete. 2023-05-22T15:43:13.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:13.275+0000 7fb7df0cd640 1 Processor -- start 2023-05-22T15:43:13.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:13.275+0000 7fb7df0cd640 1 -- start start 2023-05-22T15:43:13.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:13.275+0000 7fb7df0cd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb7d806e740 0x7fb7d814cd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:13.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:13.275+0000 7fb7df0cd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb7d80cf870 0x7fb7d8158ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:13.282 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:13.275+0000 7fb7df0cd640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb7d815cfa0 0x7fb7d8159220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:13.282 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:13.275+0000 7fb7df0cd640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7d80c6fb0 con 0x7fb7d80cf870 2023-05-22T15:43:13.282 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:13.275+0000 7fb7df0cd640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7d80c7290 con 0x7fb7d806e740 2023-05-22T15:43:13.282 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:13.275+0000 7fb7df0cd640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7d80c7570 con 0x7fb7d815cfa0 2023-05-22T15:43:13.282 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:13.275+0000 7fb7d7fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb7d80cf870 0x7fb7d8158ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:13.282 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:13.275+0000 7fb7dce69640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb7d806e740 0x7fb7d814cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:13.282 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:13.275+0000 7fb7d7fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb7d80cf870 0x7fb7d8158ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45702/0 (socket says 172.21.15.16:45702) 2023-05-22T15:43:13.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:13.275+0000 7fb7d7fff640 1 -- 172.21.15.16:0/2550863626 learned_addr learned my addr 172.21.15.16:0/2550863626 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:43:13.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:13.275+0000 7fb7dd66a640 1 --2- 172.21.15.16:0/2550863626 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb7d815cfa0 0x7fb7d8159220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:13.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:13.275+0000 7fb7d7fff640 1 -- 172.21.15.16:0/2550863626 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb7d815cfa0 msgr2=0x7fb7d8159220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:13.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:13.275+0000 7fb7d7fff640 1 --2- 172.21.15.16:0/2550863626 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb7d815cfa0 0x7fb7d8159220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:13.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:13.275+0000 7fb7d7fff640 1 -- 172.21.15.16:0/2550863626 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb7d806e740 msgr2=0x7fb7d814cd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:13.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:13.275+0000 7fb7d7fff640 1 --2- 172.21.15.16:0/2550863626 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb7d806e740 0x7fb7d814cd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:13.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:13.275+0000 7fb7d7fff640 1 -- 172.21.15.16:0/2550863626 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7cc009420 con 0x7fb7d80cf870 2023-05-22T15:43:13.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:13.275+0000 7fb7dce69640 1 --2- 172.21.15.16:0/2550863626 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb7d806e740 0x7fb7d814cd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-05-22T15:43:13.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:13.279+0000 7fb7d7fff640 1 --2- 172.21.15.16:0/2550863626 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb7d80cf870 0x7fb7d8158ce0 secure :-1 s=READY pgs=1352 cs=0 l=1 rev1=1 crypto rx=0x7fb7cc01ab00 tx=0x7fb7cc01ac90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:43:13.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:13.279+0000 7fb7d57fa640 1 -- 172.21.15.16:0/2550863626 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7cc0b5810 con 0x7fb7d80cf870 2023-05-22T15:43:13.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:13.279+0000 7fb7d57fa640 1 -- 172.21.15.16:0/2550863626 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7cc0b5990 con 0x7fb7d80cf870 2023-05-22T15:43:13.285 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:13.279+0000 7fb7d57fa640 1 -- 172.21.15.16:0/2550863626 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7cc0b5d40 con 0x7fb7d80cf870 2023-05-22T15:43:13.285 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:13.279+0000 7fb7df0cd640 1 -- 172.21.15.16:0/2550863626 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7d80cb610 con 0x7fb7d80cf870 2023-05-22T15:43:13.285 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:13.279+0000 7fb7df0cd640 1 -- 172.21.15.16:0/2550863626 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7d80cb330 con 0x7fb7d80cf870 2023-05-22T15:43:13.285 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:13.279+0000 7fb7df0cd640 1 -- 172.21.15.16:0/2550863626 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7d815d370 con 0x7fb7d80cf870 2023-05-22T15:43:13.286 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:13.279+0000 7fb7d57fa640 1 -- 172.21.15.16:0/2550863626 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 38) v1 ==== 84389+0+0 (secure 0 0 0) 0x7fb7cc0bf410 con 0x7fb7d80cf870 2023-05-22T15:43:13.286 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:13.283+0000 7fb7d57fa640 1 --2- 172.21.15.16:0/2550863626 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fb7bc065530 0x7fb7bc0679e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:13.287 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:13.283+0000 7fb7dce69640 1 -- 172.21.15.16:0/2550863626 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fb7bc065530 msgr2=0x7fb7bc0679e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/36921 2023-05-22T15:43:13.287 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:13.283+0000 7fb7dce69640 1 --2- 172.21.15.16:0/2550863626 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fb7bc065530 0x7fb7bc0679e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-05-22T15:43:13.294 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:13.291+0000 7fb7d57fa640 1 -- 172.21.15.16:0/2550863626 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 6726+0+0 (secure 0 0 0) 0x7fb7cc0f63a0 con 0x7fb7d80cf870 2023-05-22T15:43:13.298 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:13.291+0000 7fb7d57fa640 1 -- 172.21.15.16:0/2550863626 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fb7cc122060 con 0x7fb7d80cf870 2023-05-22T15:43:13.338 INFO:tasks.ceph.mgr.x.smithi062.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2023-05-22T15:43:13.338 INFO:tasks.ceph.mgr.x.smithi062.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2023-05-22T15:43:13.338 INFO:tasks.ceph.mgr.x.smithi062.stderr: from numpy import show_config as show_numpy_config 2023-05-22T15:43:13.346 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:43:13.343+0000 7ffb72709180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-05-22T15:43:13.379 INFO:tasks.ceph.mgr.y.smithi016.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2023-05-22T15:43:13.379 INFO:tasks.ceph.mgr.y.smithi016.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2023-05-22T15:43:13.380 INFO:tasks.ceph.mgr.y.smithi016.stderr: from numpy import show_config as show_numpy_config 2023-05-22T15:43:13.386 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:43:13.383+0000 7f9700ac6180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-05-22T15:43:13.471 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:43:13.467+0000 7ffb72709180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-05-22T15:43:13.490 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:13.487+0000 7fb7dce69640 1 -- 172.21.15.16:0/2550863626 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fb7bc065530 msgr2=0x7fb7bc0679e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/36921 2023-05-22T15:43:13.490 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:13.487+0000 7fb7dce69640 1 --2- 172.21.15.16:0/2550863626 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fb7bc065530 0x7fb7bc0679e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-05-22T15:43:13.494 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:13.491+0000 7fb7df0cd640 1 -- 172.21.15.16:0/2550863626 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fb7d804dbe0 con 0x7fb7d80cf870 2023-05-22T15:43:13.495 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:13.491+0000 7fb7d57fa640 1 -- 172.21.15.16:0/2550863626 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 8 entries v105) v1 ==== 80+0+458 (secure 0 0 0) 0x7fb7cc122020 con 0x7fb7d80cf870 2023-05-22T15:43:13.495 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/1315052983 2023-05-23T15:43:09.063704+0000 2023-05-22T15:43:13.495 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/3981695108 2023-05-23T15:43:09.063704+0000 2023-05-22T15:43:13.496 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/92534915 2023-05-23T15:43:09.063704+0000 2023-05-22T15:43:13.496 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:6833/36921 2023-05-23T15:43:09.063704+0000 2023-05-22T15:43:13.496 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:6832/36921 2023-05-23T15:43:09.063704+0000 2023-05-22T15:43:13.496 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/2741778632 2023-05-23T15:43:09.063704+0000 2023-05-22T15:43:13.496 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/3940121619 2023-05-23T15:43:09.063704+0000 2023-05-22T15:43:13.496 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/2073223440 2023-05-23T15:43:09.063704+0000 2023-05-22T15:43:13.497 INFO:teuthology.orchestra.run.smithi016.stderr:listed 8 entries 2023-05-22T15:43:13.497 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:13.491+0000 7fb7df0cd640 1 -- 172.21.15.16:0/2550863626 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fb7bc065530 msgr2=0x7fb7bc0679e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-05-22T15:43:13.498 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:13.491+0000 7fb7df0cd640 1 --2- 172.21.15.16:0/2550863626 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fb7bc065530 0x7fb7bc0679e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:13.498 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:13.491+0000 7fb7df0cd640 1 -- 172.21.15.16:0/2550863626 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb7d80cf870 msgr2=0x7fb7d8158ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:13.498 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:13.491+0000 7fb7df0cd640 1 --2- 172.21.15.16:0/2550863626 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb7d80cf870 0x7fb7d8158ce0 secure :-1 s=READY pgs=1352 cs=0 l=1 rev1=1 crypto rx=0x7fb7cc01ab00 tx=0x7fb7cc01ac90 comp rx=0 tx=0).stop 2023-05-22T15:43:13.498 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:13.491+0000 7fb7df0cd640 1 -- 172.21.15.16:0/2550863626 shutdown_connections 2023-05-22T15:43:13.499 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:13.491+0000 7fb7df0cd640 1 --2- 172.21.15.16:0/2550863626 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fb7bc065530 0x7fb7bc0679e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:13.499 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:13.491+0000 7fb7df0cd640 1 --2- 172.21.15.16:0/2550863626 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb7d815cfa0 0x7fb7d8159220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:13.499 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:13.491+0000 7fb7df0cd640 1 --2- 172.21.15.16:0/2550863626 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb7d80cf870 0x7fb7d8158ce0 unknown :-1 s=CLOSED pgs=1352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:13.499 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:13.491+0000 7fb7df0cd640 1 --2- 172.21.15.16:0/2550863626 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb7d806e740 0x7fb7d814cd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:13.499 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:13.491+0000 7fb7df0cd640 1 -- 172.21.15.16:0/2550863626 >> 172.21.15.16:0/2550863626 conn(0x7fb7d80cc870 msgr2=0x7fb7d80affe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:43:13.499 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:13.491+0000 7fb7df0cd640 1 -- 172.21.15.16:0/2550863626 shutdown_connections 2023-05-22T15:43:13.499 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:13.491+0000 7fb7df0cd640 1 -- 172.21.15.16:0/2550863626 wait complete. 2023-05-22T15:43:13.513 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi016.front.sepia.ceph.com... 2023-05-22T15:43:13.514 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:43:13.514 DEBUG:teuthology.orchestra.run.smithi016:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2023-05-22T15:43:13.515 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:43:13.507+0000 7f9700ac6180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-05-22T15:43:13.525 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi016.stderr:ceph-fuse[57151]: fuse finished with error 0 and tester_r 0 2023-05-22T15:43:13.526 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi016.front.sepia.ceph.com 2023-05-22T15:43:13.526 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:43:13.527 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-05-22T15:43:13.585 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:43:13.583+0000 7ffb72709180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-05-22T15:43:13.595 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2023-05-22T15:43:13.595 DEBUG:teuthology.orchestra.run.smithi062:> dd if=/proc/self/mounts of=/dev/stdout 2023-05-22T15:43:13.602 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16342116k,nr_inodes=4085529,mode=755,inode64 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275932k,mode=755,inode64 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev,inode64 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=13250 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275928k,nr_inodes=818982,mode=700,uid=1000,gid=1271,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275932k,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-05-22T15:43:13.602 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi062.front.sepia.ceph.com 2023-05-22T15:43:13.603 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:43:13.604 DEBUG:teuthology.orchestra.run.smithi062:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2023-05-22T15:43:13.630 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:43:13.627+0000 7f9700ac6180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-05-22T15:43:13.649 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:43:13.647+0000 7ffb72709180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-05-22T15:43:13.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-05-22T15:43:13.650 INFO:teuthology.orchestra.run.smithi062.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2023-05-22T15:43:13.650 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-05-22T15:43:13.651 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2023-05-22T15:43:13.656 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16345192k,nr_inodes=4086298,mode=755,inode64 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3276544k,mode=755,inode64 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev,inode64 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=15732 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3276540k,nr_inodes=819135,mode=700,uid=1000,gid=1271,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3276544k,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-05-22T15:43:13.656 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi016.front.sepia.ceph.com 2023-05-22T15:43:13.657 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:43:13.657 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2023-05-22T15:43:13.695 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:43:13.691+0000 7f9700ac6180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-05-22T15:43:13.700 INFO:teuthology.orchestra.run.smithi016.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2023-05-22T15:43:13.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-05-22T15:43:13.701 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2023-05-22T15:43:13.702 DEBUG:teuthology.orchestra.run.smithi062:> dd if=/proc/self/mounts of=/dev/stdout 2023-05-22T15:43:13.708 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16342116k,nr_inodes=4085529,mode=755,inode64 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275932k,mode=755,inode64 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev,inode64 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=13250 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275928k,nr_inodes=818982,mode=700,uid=1000,gid=1271,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275932k,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-05-22T15:43:13.708 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi062.front.sepia.ceph.com 2023-05-22T15:43:13.708 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:43:13.709 DEBUG:teuthology.orchestra.run.smithi062:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2023-05-22T15:43:13.753 INFO:teuthology.orchestra.run.smithi062.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2023-05-22T15:43:13.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-05-22T15:43:13.755 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-05-22T15:43:13.775 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:43:13.771+0000 7ffb72709180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-05-22T15:43:13.823 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:43:13.819+0000 7f9700ac6180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-05-22T15:43:13.871 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:13.867+0000 7fbe5a7ab640 1 Processor -- start 2023-05-22T15:43:13.871 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:13.867+0000 7fbe5a7ab640 1 -- start start 2023-05-22T15:43:13.872 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:13.867+0000 7fbe5a7ab640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fbe540cf870 0x7fbe5406db00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:13.872 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:13.867+0000 7fbe5a7ab640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fbe5406e740 0x7fbe5406ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:13.872 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:13.867+0000 7fbe5a7ab640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe5406e040 con 0x7fbe5415cfa0 2023-05-22T15:43:13.873 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:13.867+0000 7fbe5a7ab640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe540cb9f0 con 0x7fbe540cf870 2023-05-22T15:43:13.873 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:13.867+0000 7fbe5a7ab640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe5406f100 con 0x7fbe5406e740 2023-05-22T15:43:13.873 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:13.867+0000 7fbe4bfff640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fbe540cf870 0x7fbe5406db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:13.873 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:13.867+0000 7fbe4bfff640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fbe540cf870 0x7fbe5406db00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45710/0 (socket says 172.21.15.16:45710) 2023-05-22T15:43:13.873 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:13.867+0000 7fbe4bfff640 1 -- 172.21.15.16:0/2178272268 learned_addr learned my addr 172.21.15.16:0/2178272268 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:43:13.873 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:13.867+0000 7fbe58d48640 1 --2- 172.21.15.16:0/2178272268 >> v2:172.21.15.62:3300/0 conn(0x7fbe5406e740 0x7fbe5406ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:13.873 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:13.867+0000 7fbe4bfff640 1 -- 172.21.15.16:0/2178272268 >> v2:172.21.15.62:3300/0 conn(0x7fbe5406e740 msgr2=0x7fbe5406ebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:43:13.874 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:13.867+0000 7fbe4bfff640 1 --2- 172.21.15.16:0/2178272268 >> v2:172.21.15.62:3300/0 conn(0x7fbe5406e740 0x7fbe5406ebc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:13.874 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:13.867+0000 7fbe4bfff640 1 -- 172.21.15.16:0/2178272268 >> v1:172.21.15.62:6789/0 conn(0x7fbe5415cfa0 legacy=0x7fbe540b1fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:43:13.874 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:13.867+0000 7fbe4bfff640 1 -- 172.21.15.16:0/2178272268 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe5406e490 con 0x7fbe540cf870 2023-05-22T15:43:13.874 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:13.871+0000 7fbe4bfff640 1 --2- 172.21.15.16:0/2178272268 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbe540cf870 0x7fbe5406db00 secure :-1 s=READY pgs=1353 cs=0 l=1 rev1=1 crypto rx=0x7fbe540c9760 tx=0x7fbe3c01c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5bb76d94e01a50bf server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:43:13.875 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:13.871+0000 7fbe537fe640 1 -- 172.21.15.16:0/2178272268 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe3c0aa5f0 con 0x7fbe540cf870 2023-05-22T15:43:13.875 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:13.871+0000 7fbe537fe640 1 -- 172.21.15.16:0/2178272268 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe3c0aa750 con 0x7fbe540cf870 2023-05-22T15:43:13.875 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:13.871+0000 7fbe537fe640 1 -- 172.21.15.16:0/2178272268 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe3c0aaae0 con 0x7fbe540cf870 2023-05-22T15:43:13.876 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:13.871+0000 7fbe5a7ab640 1 -- 172.21.15.16:0/2178272268 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbe540cf870 msgr2=0x7fbe5406db00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:13.876 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:13.871+0000 7fbe5a7ab640 1 --2- 172.21.15.16:0/2178272268 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbe540cf870 0x7fbe5406db00 secure :-1 s=READY pgs=1353 cs=0 l=1 rev1=1 crypto rx=0x7fbe540c9760 tx=0x7fbe3c01c490 comp rx=0 tx=0).stop 2023-05-22T15:43:13.876 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:13.871+0000 7fbe5a7ab640 1 -- 172.21.15.16:0/2178272268 shutdown_connections 2023-05-22T15:43:13.876 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:13.871+0000 7fbe5a7ab640 1 --2- 172.21.15.16:0/2178272268 >> v2:172.21.15.62:3300/0 conn(0x7fbe5406e740 0x7fbe5406ebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:13.877 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:13.871+0000 7fbe5a7ab640 1 --2- 172.21.15.16:0/2178272268 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbe540cf870 0x7fbe5406db00 unknown :-1 s=CLOSED pgs=1353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:13.877 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:13.871+0000 7fbe5a7ab640 1 -- 172.21.15.16:0/2178272268 >> 172.21.15.16:0/2178272268 conn(0x7fbe540cc870 msgr2=0x7fbe540cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:43:13.877 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:13.871+0000 7fbe5a7ab640 1 -- 172.21.15.16:0/2178272268 shutdown_connections 2023-05-22T15:43:13.877 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:13.871+0000 7fbe5a7ab640 1 -- 172.21.15.16:0/2178272268 wait complete. 2023-05-22T15:43:13.879 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:13.871+0000 7fbe5a7ab640 1 Processor -- start 2023-05-22T15:43:13.879 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:13.871+0000 7fbe5a7ab640 1 -- start start 2023-05-22T15:43:13.879 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:13.871+0000 7fbe5a7ab640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbe5406e740 0x7fbe54151a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:13.879 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:13.871+0000 7fbe5a7ab640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbe540cf870 0x7fbe540b39f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:13.879 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:13.871+0000 7fbe5a7ab640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbe5415cfa0 0x7fbe54156310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:13.879 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:13.871+0000 7fbe5a7ab640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe541641a0 con 0x7fbe540cf870 2023-05-22T15:43:13.880 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:13.871+0000 7fbe5a7ab640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe54162c10 con 0x7fbe5406e740 2023-05-22T15:43:13.880 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:13.871+0000 7fbe5a7ab640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe54162ef0 con 0x7fbe5415cfa0 2023-05-22T15:43:13.880 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:13.871+0000 7fbe4bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbe540cf870 0x7fbe540b39f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:13.880 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:13.871+0000 7fbe4bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbe540cf870 0x7fbe540b39f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45724/0 (socket says 172.21.15.16:45724) 2023-05-22T15:43:13.880 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:13.871+0000 7fbe4bfff640 1 -- 172.21.15.16:0/1243365594 learned_addr learned my addr 172.21.15.16:0/1243365594 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:43:13.880 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:13.871+0000 7fbe53fff640 1 --2- 172.21.15.16:0/1243365594 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbe5406e740 0x7fbe54151a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:13.880 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:13.871+0000 7fbe58d48640 1 --2- 172.21.15.16:0/1243365594 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbe5415cfa0 0x7fbe54156310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:13.881 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:13.871+0000 7fbe4bfff640 1 -- 172.21.15.16:0/1243365594 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbe5415cfa0 msgr2=0x7fbe54156310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:13.881 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:13.871+0000 7fbe4bfff640 1 --2- 172.21.15.16:0/1243365594 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbe5415cfa0 0x7fbe54156310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:13.881 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:13.871+0000 7fbe4bfff640 1 -- 172.21.15.16:0/1243365594 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbe5406e740 msgr2=0x7fbe54151a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:13.881 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:13.871+0000 7fbe4bfff640 1 --2- 172.21.15.16:0/1243365594 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbe5406e740 0x7fbe54151a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:13.881 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:13.871+0000 7fbe4bfff640 1 -- 172.21.15.16:0/1243365594 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe3c0094d0 con 0x7fbe540cf870 2023-05-22T15:43:13.881 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:13.871+0000 7fbe58d48640 1 --2- 172.21.15.16:0/1243365594 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbe5415cfa0 0x7fbe54156310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:43:13.881 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:13.871+0000 7fbe53fff640 1 --2- 172.21.15.16:0/1243365594 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbe5406e740 0x7fbe54151a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T15:43:13.882 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:13.871+0000 7fbe4bfff640 1 --2- 172.21.15.16:0/1243365594 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbe540cf870 0x7fbe540b39f0 secure :-1 s=READY pgs=1354 cs=0 l=1 rev1=1 crypto rx=0x7fbe3c0ac040 tx=0x7fbe3c01acd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:43:13.882 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:13.871+0000 7fbe517fa640 1 -- 172.21.15.16:0/1243365594 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe3c0b3800 con 0x7fbe540cf870 2023-05-22T15:43:13.882 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:13.871+0000 7fbe517fa640 1 -- 172.21.15.16:0/1243365594 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe3c0b3960 con 0x7fbe540cf870 2023-05-22T15:43:13.882 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:13.871+0000 7fbe5a7ab640 1 -- 172.21.15.16:0/1243365594 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe540cb610 con 0x7fbe540cf870 2023-05-22T15:43:13.882 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:13.871+0000 7fbe517fa640 1 -- 172.21.15.16:0/1243365594 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe3c0b3cf0 con 0x7fbe540cf870 2023-05-22T15:43:13.882 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:13.875+0000 7fbe5a7ab640 1 -- 172.21.15.16:0/1243365594 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe540cb330 con 0x7fbe540cf870 2023-05-22T15:43:13.882 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:13.875+0000 7fbe5a7ab640 1 -- 172.21.15.16:0/1243365594 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe5415d370 con 0x7fbe540cf870 2023-05-22T15:43:13.883 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:13.875+0000 7fbe517fa640 1 -- 172.21.15.16:0/1243365594 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 38) v1 ==== 84389+0+0 (secure 0 0 0) 0x7fbe3c0c8430 con 0x7fbe540cf870 2023-05-22T15:43:13.883 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:13.875+0000 7fbe517fa640 1 --2- 172.21.15.16:0/1243365594 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fbe34065560 0x7fbe34067a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:13.883 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:13.875+0000 7fbe53fff640 1 -- 172.21.15.16:0/1243365594 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fbe34065560 msgr2=0x7fbe34067a10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/36921 2023-05-22T15:43:13.884 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:13.875+0000 7fbe517fa640 1 -- 172.21.15.16:0/1243365594 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 6726+0+0 (secure 0 0 0) 0x7fbe3c0a7070 con 0x7fbe540cf870 2023-05-22T15:43:13.884 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:13.875+0000 7fbe53fff640 1 --2- 172.21.15.16:0/1243365594 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fbe34065560 0x7fbe34067a10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-05-22T15:43:13.884 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:13.879+0000 7fbe517fa640 1 -- 172.21.15.16:0/1243365594 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fbe3c0eaf80 con 0x7fbe540cf870 2023-05-22T15:43:14.067 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.063+0000 7fbe5a7ab640 1 -- 172.21.15.16:0/1243365594 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbe5406e040 con 0x7fbe540cf870 2023-05-22T15:43:14.068 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.063+0000 7fbe517fa640 1 -- 172.21.15.16:0/1243365594 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) v1 ==== 94+0+4650 (secure 0 0 0) 0x7fbe5406e040 con 0x7fbe540cf870 2023-05-22T15:43:14.068 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:43:14.069 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":36,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6059,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6833/3017549713","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":3017549713},{"type":"v1","addr":"172.21.15.62:6833","nonce":3017549713}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":33},{"gid":6607,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/3268651705","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":3268651705},{"type":"v1","addr":"172.21.15.62:6835","nonce":3268651705}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":33},{"gid":7524,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6836/2104991509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2104991509},{"type":"v1","addr":"172.21.15.16:6836","nonce":2104991509}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":33}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-05-22T15:42:56.565074+0000","modified":"2023-05-22T15:42:58.580533+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7527},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7527":{"gid":7527,"name":"a","rank":0,"incarnation":34,"state":"up:active","state_seq":3,"addr":"172.21.15.16:6837/1353635569","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6835","nonce":1353635569},{"type":"v1","addr":"172.21.15.16:6837","nonce":1353635569}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2023-05-22T15:43:14.069 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 36 2023-05-22T15:43:14.070 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.063+0000 7fbe5a7ab640 1 -- 172.21.15.16:0/1243365594 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fbe34065560 msgr2=0x7fbe34067a10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-05-22T15:43:14.070 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.063+0000 7fbe5a7ab640 1 --2- 172.21.15.16:0/1243365594 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fbe34065560 0x7fbe34067a10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:14.070 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.063+0000 7fbe5a7ab640 1 -- 172.21.15.16:0/1243365594 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbe540cf870 msgr2=0x7fbe540b39f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:14.071 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.063+0000 7fbe5a7ab640 1 --2- 172.21.15.16:0/1243365594 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbe540cf870 0x7fbe540b39f0 secure :-1 s=READY pgs=1354 cs=0 l=1 rev1=1 crypto rx=0x7fbe3c0ac040 tx=0x7fbe3c01acd0 comp rx=0 tx=0).stop 2023-05-22T15:43:14.071 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.063+0000 7fbe5a7ab640 1 -- 172.21.15.16:0/1243365594 shutdown_connections 2023-05-22T15:43:14.071 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.063+0000 7fbe5a7ab640 1 --2- 172.21.15.16:0/1243365594 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fbe34065560 0x7fbe34067a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:14.071 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.063+0000 7fbe5a7ab640 1 --2- 172.21.15.16:0/1243365594 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbe5415cfa0 0x7fbe54156310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:14.071 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.063+0000 7fbe5a7ab640 1 --2- 172.21.15.16:0/1243365594 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbe540cf870 0x7fbe540b39f0 unknown :-1 s=CLOSED pgs=1354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:14.071 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.063+0000 7fbe5a7ab640 1 --2- 172.21.15.16:0/1243365594 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbe5406e740 0x7fbe54151a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:14.071 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.067+0000 7fbe5a7ab640 1 -- 172.21.15.16:0/1243365594 >> 172.21.15.16:0/1243365594 conn(0x7fbe540cc870 msgr2=0x7fbe5415bb90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:43:14.072 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.067+0000 7fbe5a7ab640 1 -- 172.21.15.16:0/1243365594 shutdown_connections 2023-05-22T15:43:14.072 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.067+0000 7fbe5a7ab640 1 -- 172.21.15.16:0/1243365594 wait complete. 2023-05-22T15:43:14.087 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-05-22T15:43:14.202 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.195+0000 7f4c2dd84640 1 Processor -- start 2023-05-22T15:43:14.202 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.195+0000 7f4c2dd84640 1 -- start start 2023-05-22T15:43:14.203 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.199+0000 7f4c2dd84640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4c28159cd0 0x7f4c2815a0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:14.203 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.199+0000 7f4c2dd84640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c280cf9a0 con 0x7f4c280b2880 2023-05-22T15:43:14.203 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.199+0000 7f4c2dd84640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c2815aba0 con 0x7f4c280cc6f0 2023-05-22T15:43:14.203 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.199+0000 7f4c2dd84640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c2815ae80 con 0x7f4c28159cd0 2023-05-22T15:43:14.204 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.199+0000 7f4c1ffff640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f4c280b2880 0x7f4c280cbfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:57106/0 (socket says 172.21.15.16:57106) 2023-05-22T15:43:14.204 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.199+0000 7f4c1ffff640 1 -- 172.21.15.16:0/211737106 learned_addr learned my addr 172.21.15.16:0/211737106 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:43:14.204 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.199+0000 7f4c277fe640 1 --2- 172.21.15.16:0/211737106 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4c28159cd0 0x7f4c2815a0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:14.208 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.199+0000 7f4c26ffd640 1 -- 172.21.15.16:0/211737106 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1623599777 0 0) 0x7f4c280cf9a0 con 0x7f4c280b2880 2023-05-22T15:43:14.209 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.199+0000 7f4c26ffd640 1 -- 172.21.15.16:0/211737106 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c14003530 con 0x7f4c280b2880 2023-05-22T15:43:14.209 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.199+0000 7f4c26ffd640 1 -- 172.21.15.16:0/211737106 <== mon.1 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 740422846 0 0) 0x7f4c2815aba0 con 0x7f4c280cc6f0 2023-05-22T15:43:14.209 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.199+0000 7f4c26ffd640 1 -- 172.21.15.16:0/211737106 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c280cf9a0 con 0x7f4c280cc6f0 2023-05-22T15:43:14.209 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.199+0000 7f4c26ffd640 1 -- 172.21.15.16:0/211737106 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3112698814 0 0) 0x7f4c14003530 con 0x7f4c280b2880 2023-05-22T15:43:14.210 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.199+0000 7f4c26ffd640 1 -- 172.21.15.16:0/211737106 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4c28159cd0 msgr2=0x7f4c2815a0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:43:14.210 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.199+0000 7f4c26ffd640 1 --2- 172.21.15.16:0/211737106 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4c28159cd0 0x7f4c2815a0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:14.210 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.199+0000 7f4c26ffd640 1 -- 172.21.15.16:0/211737106 >> v1:172.21.15.62:6789/0 conn(0x7f4c280cc6f0 legacy=0x7f4c280cfe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:14.210 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.199+0000 7f4c26ffd640 1 -- 172.21.15.16:0/211737106 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c2815b160 con 0x7f4c280b2880 2023-05-22T15:43:14.210 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.199+0000 7f4c26ffd640 1 -- 172.21.15.16:0/211737106 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7f4c10001680 con 0x7f4c280b2880 2023-05-22T15:43:14.210 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.199+0000 7f4c26ffd640 1 -- 172.21.15.16:0/211737106 >> v1:172.21.15.16:6789/0 conn(0x7f4c280b2880 legacy=0x7f4c280cbfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:14.210 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.199+0000 7f4c26ffd640 1 --2- 172.21.15.16:0/211737106 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4c14003b40 0x7f4c14003f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:14.211 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.199+0000 7f4c26ffd640 1 --2- 172.21.15.16:0/211737106 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4c140047d0 0x7f4c14004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:14.211 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.199+0000 7f4c26ffd640 1 --2- 172.21.15.16:0/211737106 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4c28159cd0 0x7f4c14005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:14.211 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.199+0000 7f4c26ffd640 1 -- 172.21.15.16:0/211737106 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c2815ae80 con 0x7f4c140047d0 2023-05-22T15:43:14.211 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.199+0000 7f4c26ffd640 1 -- 172.21.15.16:0/211737106 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c1400ba40 con 0x7f4c14003b40 2023-05-22T15:43:14.211 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.199+0000 7f4c26ffd640 1 -- 172.21.15.16:0/211737106 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c1400bbb0 con 0x7f4c28159cd0 2023-05-22T15:43:14.212 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.199+0000 7f4c1ffff640 1 --2- 172.21.15.16:0/211737106 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4c140047d0 0x7f4c14004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:14.212 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.199+0000 7f4c277fe640 1 --2- 172.21.15.16:0/211737106 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4c14003b40 0x7f4c14003f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:14.212 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.199+0000 7f4c27fff640 1 --2- 172.21.15.16:0/211737106 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4c28159cd0 0x7f4c14005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:14.212 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.199+0000 7f4c1ffff640 1 -- 172.21.15.16:0/211737106 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4c28159cd0 msgr2=0x7f4c14005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:43:14.212 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.199+0000 7f4c1ffff640 1 --2- 172.21.15.16:0/211737106 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4c28159cd0 0x7f4c14005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:14.212 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.199+0000 7f4c1ffff640 1 -- 172.21.15.16:0/211737106 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4c14003b40 msgr2=0x7f4c14003f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:43:14.213 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.199+0000 7f4c1ffff640 1 --2- 172.21.15.16:0/211737106 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4c14003b40 0x7f4c14003f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:14.213 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.199+0000 7f4c1ffff640 1 -- 172.21.15.16:0/211737106 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c14003530 con 0x7f4c140047d0 2023-05-22T15:43:14.213 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.203+0000 7f4c1ffff640 1 --2- 172.21.15.16:0/211737106 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4c140047d0 0x7f4c14004c40 secure :-1 s=READY pgs=1356 cs=0 l=1 rev1=1 crypto rx=0x7f4c1000c160 tx=0x7f4c1001cba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2cfc83373823c457 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:43:14.213 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.203+0000 7f4c26ffd640 1 -- 172.21.15.16:0/211737106 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c100058e0 con 0x7f4c140047d0 2023-05-22T15:43:14.213 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.203+0000 7f4c26ffd640 1 -- 172.21.15.16:0/211737106 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c10005a60 con 0x7f4c140047d0 2023-05-22T15:43:14.213 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.203+0000 7f4c26ffd640 1 -- 172.21.15.16:0/211737106 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c10005e10 con 0x7f4c140047d0 2023-05-22T15:43:14.213 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.203+0000 7f4c2dd84640 1 -- 172.21.15.16:0/211737106 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4c140047d0 msgr2=0x7f4c14004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:14.214 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.203+0000 7f4c2dd84640 1 --2- 172.21.15.16:0/211737106 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4c140047d0 0x7f4c14004c40 secure :-1 s=READY pgs=1356 cs=0 l=1 rev1=1 crypto rx=0x7f4c1000c160 tx=0x7f4c1001cba0 comp rx=0 tx=0).stop 2023-05-22T15:43:14.214 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.203+0000 7f4c27fff640 1 -- 172.21.15.16:0/211737106 reap_dead start 2023-05-22T15:43:14.214 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.203+0000 7f4c2dd84640 1 -- 172.21.15.16:0/211737106 shutdown_connections 2023-05-22T15:43:14.214 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.203+0000 7f4c2dd84640 1 -- 172.21.15.16:0/211737106 >> 172.21.15.16:0/211737106 conn(0x7f4c2806f5c0 msgr2=0x7f4c2806f9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:43:14.214 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.203+0000 7f4c2dd84640 1 -- 172.21.15.16:0/211737106 shutdown_connections 2023-05-22T15:43:14.214 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.203+0000 7f4c2dd84640 1 -- 172.21.15.16:0/211737106 wait complete. 2023-05-22T15:43:14.216 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.203+0000 7f4c2dd84640 1 Processor -- start 2023-05-22T15:43:14.217 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.203+0000 7f4c2dd84640 1 -- start start 2023-05-22T15:43:14.217 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.207+0000 7f4c2dd84640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4c140047d0 0x7f4c280b45b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:14.217 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.207+0000 7f4c2dd84640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4c281509d0 0x7f4c2814f200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:14.217 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.207+0000 7f4c2dd84640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4c280b3c20 0x7f4c28152530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:14.217 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.207+0000 7f4c2dd84640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c28151120 con 0x7f4c140047d0 2023-05-22T15:43:14.217 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.207+0000 7f4c2dd84640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c280c8380 con 0x7f4c281509d0 2023-05-22T15:43:14.218 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.207+0000 7f4c2dd84640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c2814f740 con 0x7f4c280b3c20 2023-05-22T15:43:14.218 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.207+0000 7f4c277fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4c140047d0 0x7f4c280b45b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:14.218 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.207+0000 7f4c277fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4c140047d0 0x7f4c280b45b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45734/0 (socket says 172.21.15.16:45734) 2023-05-22T15:43:14.218 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.207+0000 7f4c277fe640 1 -- 172.21.15.16:0/2003511919 learned_addr learned my addr 172.21.15.16:0/2003511919 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:43:14.218 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.207+0000 7f4c1ffff640 1 --2- 172.21.15.16:0/2003511919 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4c281509d0 0x7f4c2814f200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:14.218 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.207+0000 7f4c27fff640 1 --2- 172.21.15.16:0/2003511919 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4c280b3c20 0x7f4c28152530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:14.218 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.207+0000 7f4c277fe640 1 -- 172.21.15.16:0/2003511919 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4c280b3c20 msgr2=0x7f4c28152530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:14.219 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.207+0000 7f4c277fe640 1 --2- 172.21.15.16:0/2003511919 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4c280b3c20 0x7f4c28152530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:14.219 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.207+0000 7f4c277fe640 1 -- 172.21.15.16:0/2003511919 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4c281509d0 msgr2=0x7f4c2814f200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:14.219 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.207+0000 7f4c277fe640 1 --2- 172.21.15.16:0/2003511919 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4c281509d0 0x7f4c2814f200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:14.219 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.207+0000 7f4c277fe640 1 -- 172.21.15.16:0/2003511919 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c2815a0a0 con 0x7f4c140047d0 2023-05-22T15:43:14.219 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.207+0000 7f4c1ffff640 1 --2- 172.21.15.16:0/2003511919 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4c281509d0 0x7f4c2814f200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T15:43:14.219 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.207+0000 7f4c277fe640 1 --2- 172.21.15.16:0/2003511919 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4c140047d0 0x7f4c280b45b0 secure :-1 s=READY pgs=1357 cs=0 l=1 rev1=1 crypto rx=0x7f4c18011a00 tx=0x7f4c18011ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:43:14.219 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.207+0000 7f4c24ff9640 1 -- 172.21.15.16:0/2003511919 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c1800fb50 con 0x7f4c140047d0 2023-05-22T15:43:14.220 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.207+0000 7f4c24ff9640 1 -- 172.21.15.16:0/2003511919 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c1800fcb0 con 0x7f4c140047d0 2023-05-22T15:43:14.220 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.207+0000 7f4c24ff9640 1 -- 172.21.15.16:0/2003511919 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c18014440 con 0x7f4c140047d0 2023-05-22T15:43:14.220 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.207+0000 7f4c2dd84640 1 -- 172.21.15.16:0/2003511919 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c280cf630 con 0x7f4c140047d0 2023-05-22T15:43:14.220 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.207+0000 7f4c2dd84640 1 -- 172.21.15.16:0/2003511919 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c2815a230 con 0x7f4c140047d0 2023-05-22T15:43:14.220 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.207+0000 7f4c2dd84640 1 -- 172.21.15.16:0/2003511919 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c2804dbe0 con 0x7f4c140047d0 2023-05-22T15:43:14.220 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.207+0000 7f4c24ff9640 1 -- 172.21.15.16:0/2003511919 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 38) v1 ==== 84389+0+0 (secure 0 0 0) 0x7f4c180145a0 con 0x7f4c140047d0 2023-05-22T15:43:14.220 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.207+0000 7f4c24ff9640 1 --2- 172.21.15.16:0/2003511919 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f4c08065530 0x7f4c080679e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:14.221 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.211+0000 7f4c24ff9640 1 -- 172.21.15.16:0/2003511919 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 6726+0+0 (secure 0 0 0) 0x7f4c18083640 con 0x7f4c140047d0 2023-05-22T15:43:14.221 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.211+0000 7f4c1ffff640 1 -- 172.21.15.16:0/2003511919 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f4c08065530 msgr2=0x7f4c080679e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/36921 2023-05-22T15:43:14.221 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.211+0000 7f4c1ffff640 1 --2- 172.21.15.16:0/2003511919 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f4c08065530 0x7f4c080679e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-05-22T15:43:14.221 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.215+0000 7f4c24ff9640 1 -- 172.21.15.16:0/2003511919 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f4c18050df0 con 0x7f4c140047d0 2023-05-22T15:43:14.271 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:43:14.267+0000 7ffb72709180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-05-22T15:43:14.329 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:43:14.323+0000 7f9700ac6180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-05-22T15:43:14.348 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:43:14.343+0000 7ffb72709180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-05-22T15:43:14.372 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.367+0000 7f4c2dd84640 1 -- 172.21.15.16:0/2003511919 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4c28157380 con 0x7f4c140047d0 2023-05-22T15:43:14.374 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.371+0000 7f4c24ff9640 1 -- 172.21.15.16:0/2003511919 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v105) v1 ==== 74+0+18531 (secure 0 0 0) 0x7f4c18054410 con 0x7f4c140047d0 2023-05-22T15:43:14.375 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:43:14.375 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":105,"fsid":"b2719834-7847-4d98-90a6-f65d57f85c49","created":"2023-05-22T15:26:25.259314+0000","modified":"2023-05-22T15:43:09.063772+0000","last_up_change":"2023-05-22T15:26:36.448524+0000","last_in_change":"2023-05-22T15:26:27.675668+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-05-22T15:26:38.019781+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-05-22T15:26:43.570514+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2023-05-22T15:42:55.131159+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"103","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2023-05-22T15:42:55.567899+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"104","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9e7ff854-b25f-4cbb-a5c2-1799d9c53d2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":102,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":37021},{"type":"v1","addr":"172.21.15.16:6825","nonce":37021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":37021},{"type":"v1","addr":"172.21.15.16:6827","nonce":37021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":37021},{"type":"v1","addr":"172.21.15.16:6831","nonce":37021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":37021},{"type":"v1","addr":"172.21.15.16:6829","nonce":37021}]},"public_addr":"172.21.15.16:6825/37021","cluster_addr":"172.21.15.16:6827/37021","heartbeat_back_addr":"172.21.15.16:6831/37021","heartbeat_front_addr":"172.21.15.16:6829/37021","state":["exists","up"]},{"osd":1,"uuid":"4702fbc9-a927-4565-aca5-41dc69cc8264","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":102,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":37008},{"type":"v1","addr":"172.21.15.16:6801","nonce":37008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":37008},{"type":"v1","addr":"172.21.15.16:6803","nonce":37008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":37008},{"type":"v1","addr":"172.21.15.16:6807","nonce":37008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":37008},{"type":"v1","addr":"172.21.15.16:6805","nonce":37008}]},"public_addr":"172.21.15.16:6801/37008","cluster_addr":"172.21.15.16:6803/37008","heartbeat_back_addr":"172.21.15.16:6807/37008","heartbeat_front_addr":"172.21.15.16:6805/37008","state":["exists","up"]},{"osd":2,"uuid":"1e7bba46-4eba-47ef-bb81-5e2d371638ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":102,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":37009},{"type":"v1","addr":"172.21.15.16:6809","nonce":37009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":37009},{"type":"v1","addr":"172.21.15.16:6811","nonce":37009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":37009},{"type":"v1","addr":"172.21.15.16:6815","nonce":37009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":37009},{"type":"v1","addr":"172.21.15.16:6813","nonce":37009}]},"public_addr":"172.21.15.16:6809/37009","cluster_addr":"172.21.15.16:6811/37009","heartbeat_back_addr":"172.21.15.16:6815/37009","heartbeat_front_addr":"172.21.15.16:6813/37009","state":["exists","up"]},{"osd":3,"uuid":"383cacc0-5a88-4904-990e-498ca59ef48a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":102,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":37020},{"type":"v1","addr":"172.21.15.16:6817","nonce":37020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":37020},{"type":"v1","addr":"172.21.15.16:6819","nonce":37020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":37020},{"type":"v1","addr":"172.21.15.16:6823","nonce":37020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":37020},{"type":"v1","addr":"172.21.15.16:6821","nonce":37020}]},"public_addr":"172.21.15.16:6817/37020","cluster_addr":"172.21.15.16:6819/37020","heartbeat_back_addr":"172.21.15.16:6823/37020","heartbeat_front_addr":"172.21.15.16:6821/37020","state":["exists","up"]},{"osd":4,"uuid":"970500ef-95b7-42c1-b080-eed4fae7013d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":102,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":37353},{"type":"v1","addr":"172.21.15.62:6817","nonce":37353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6819","nonce":37353},{"type":"v1","addr":"172.21.15.62:6822","nonce":37353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37353},{"type":"v1","addr":"172.21.15.62:6831","nonce":37353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37353},{"type":"v1","addr":"172.21.15.62:6826","nonce":37353}]},"public_addr":"172.21.15.62:6817/37353","cluster_addr":"172.21.15.62:6822/37353","heartbeat_back_addr":"172.21.15.62:6831/37353","heartbeat_front_addr":"172.21.15.62:6826/37353","state":["exists","up"]},{"osd":5,"uuid":"022d67db-d4d8-445b-b1b9-8b446a00f5c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":102,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":37352},{"type":"v1","addr":"172.21.15.62:6801","nonce":37352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":37352},{"type":"v1","addr":"172.21.15.62:6803","nonce":37352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":37352},{"type":"v1","addr":"172.21.15.62:6807","nonce":37352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":37352},{"type":"v1","addr":"172.21.15.62:6805","nonce":37352}]},"public_addr":"172.21.15.62:6801/37352","cluster_addr":"172.21.15.62:6803/37352","heartbeat_back_addr":"172.21.15.62:6807/37352","heartbeat_front_addr":"172.21.15.62:6805/37352","state":["exists","up"]},{"osd":6,"uuid":"42bd372b-6688-4a1b-8407-e16e8c266e33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":102,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":37354},{"type":"v1","addr":"172.21.15.62:6820","nonce":37354}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":37354},{"type":"v1","addr":"172.21.15.62:6823","nonce":37354}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6829","nonce":37354},{"type":"v1","addr":"172.21.15.62:6830","nonce":37354}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6825","nonce":37354},{"type":"v1","addr":"172.21.15.62:6827","nonce":37354}]},"public_addr":"172.21.15.62:6820/37354","cluster_addr":"172.21.15.62:6823/37354","heartbeat_back_addr":"172.21.15.62:6830/37354","heartbeat_front_addr":"172.21.15.62:6827/37354","state":["exists","up"]},{"osd":7,"uuid":"ac70b2a3-f41b-4bf2-b227-7f2cbb55426d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":102,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37355},{"type":"v1","addr":"172.21.15.62:6809","nonce":37355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37355},{"type":"v1","addr":"172.21.15.62:6811","nonce":37355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":37355},{"type":"v1","addr":"172.21.15.62:6815","nonce":37355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":37355},{"type":"v1","addr":"172.21.15.62:6813","nonce":37355}]},"public_addr":"172.21.15.62:6809/37355","cluster_addr":"172.21.15.62:6811/37355","heartbeat_back_addr":"172.21.15.62:6815/37355","heartbeat_front_addr":"172.21.15.62:6813/37355","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.368896+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.107273+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.276888+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.287502+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.124791+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.251937+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.255679+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.232797+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:0/1315052983":"2023-05-23T15:43:09.063704+0000","172.21.15.16:0/3981695108":"2023-05-23T15:43:09.063704+0000","172.21.15.16:0/92534915":"2023-05-23T15:43:09.063704+0000","172.21.15.16:6833/36921":"2023-05-23T15:43:09.063704+0000","172.21.15.16:6832/36921":"2023-05-23T15:43:09.063704+0000","172.21.15.16:0/2741778632":"2023-05-23T15:43:09.063704+0000","172.21.15.16:0/3940121619":"2023-05-23T15:43:09.063704+0000","172.21.15.16:0/2073223440":"2023-05-23T15:43:09.063704+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-05-22T15:43:14.377 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.371+0000 7f4c2dd84640 1 -- 172.21.15.16:0/2003511919 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f4c08065530 msgr2=0x7f4c080679e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-05-22T15:43:14.377 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.371+0000 7f4c2dd84640 1 --2- 172.21.15.16:0/2003511919 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f4c08065530 0x7f4c080679e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:14.377 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.371+0000 7f4c2dd84640 1 -- 172.21.15.16:0/2003511919 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4c140047d0 msgr2=0x7f4c280b45b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:14.377 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.371+0000 7f4c2dd84640 1 --2- 172.21.15.16:0/2003511919 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4c140047d0 0x7f4c280b45b0 secure :-1 s=READY pgs=1357 cs=0 l=1 rev1=1 crypto rx=0x7f4c18011a00 tx=0x7f4c18011ec0 comp rx=0 tx=0).stop 2023-05-22T15:43:14.378 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.371+0000 7f4c2dd84640 1 -- 172.21.15.16:0/2003511919 shutdown_connections 2023-05-22T15:43:14.378 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.371+0000 7f4c2dd84640 1 --2- 172.21.15.16:0/2003511919 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f4c08065530 0x7f4c080679e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:14.378 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.371+0000 7f4c2dd84640 1 --2- 172.21.15.16:0/2003511919 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4c280b3c20 0x7f4c28152530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:14.378 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.371+0000 7f4c2dd84640 1 --2- 172.21.15.16:0/2003511919 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4c281509d0 0x7f4c2814f200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:14.378 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.371+0000 7f4c2dd84640 1 --2- 172.21.15.16:0/2003511919 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4c140047d0 0x7f4c280b45b0 unknown :-1 s=CLOSED pgs=1357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:14.378 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.371+0000 7f4c2dd84640 1 -- 172.21.15.16:0/2003511919 >> 172.21.15.16:0/2003511919 conn(0x7f4c2806f5c0 msgr2=0x7f4c280cd1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:43:14.378 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.371+0000 7f4c2dd84640 1 -- 172.21.15.16:0/2003511919 shutdown_connections 2023-05-22T15:43:14.379 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.371+0000 7f4c2dd84640 1 -- 172.21.15.16:0/2003511919 wait complete. 2023-05-22T15:43:14.393 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-05-22T15:26:38.019781+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-05-22T15:26:43.570514+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 12, 'pool_name': 'cephfs_metadata', 'create_time': '2023-05-22T15:42:55.131159+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '103', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 13, 'pool_name': 'cephfs_data', 'create_time': '2023-05-22T15:42:55.567899+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '104', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-05-22T15:43:14.395 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-05-22T15:43:14.408 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:43:14.403+0000 7f9700ac6180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-05-22T15:43:14.486 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:43:14.483+0000 7ffb72709180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-05-22T15:43:14.507 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.503+0000 7eff51bba640 1 Processor -- start 2023-05-22T15:43:14.508 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.503+0000 7eff51bba640 1 -- start start 2023-05-22T15:43:14.509 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.503+0000 7eff51bba640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7eff4c0d0580 0x7eff4c0d1e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:14.509 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.503+0000 7eff51bba640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7eff4c0d2350 0x7eff4c0afe40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:14.509 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.503+0000 7eff51bba640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff4c0d0a00 con 0x7eff4c15bda0 2023-05-22T15:43:14.509 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.503+0000 7eff51bba640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff4c0b3ba0 con 0x7eff4c0d2350 2023-05-22T15:43:14.510 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.503+0000 7eff51bba640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff4c0af080 con 0x7eff4c0d0580 2023-05-22T15:43:14.510 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.503+0000 7eff513b9640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7eff4c0d2350 0x7eff4c0afe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:14.510 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.503+0000 7eff513b9640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7eff4c0d2350 0x7eff4c0afe40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45744/0 (socket says 172.21.15.16:45744) 2023-05-22T15:43:14.510 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.503+0000 7eff513b9640 1 -- 172.21.15.16:0/35458657 learned_addr learned my addr 172.21.15.16:0/35458657 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:43:14.510 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.503+0000 7eff4bfff640 1 --2- 172.21.15.16:0/35458657 >> v2:172.21.15.62:3300/0 conn(0x7eff4c0d0580 0x7eff4c0d1e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:14.510 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.503+0000 7eff513b9640 1 -- 172.21.15.16:0/35458657 >> v2:172.21.15.62:3300/0 conn(0x7eff4c0d0580 msgr2=0x7eff4c0d1e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:43:14.511 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.503+0000 7eff513b9640 1 --2- 172.21.15.16:0/35458657 >> v2:172.21.15.62:3300/0 conn(0x7eff4c0d0580 0x7eff4c0d1e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:14.511 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.503+0000 7eff513b9640 1 -- 172.21.15.16:0/35458657 >> v1:172.21.15.62:6789/0 conn(0x7eff4c15bda0 legacy=0x7eff4c0b3350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:14.511 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.503+0000 7eff513b9640 1 -- 172.21.15.16:0/35458657 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff4c0af360 con 0x7eff4c0d2350 2023-05-22T15:43:14.511 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.503+0000 7eff4bfff640 1 --2- 172.21.15.16:0/35458657 >> v2:172.21.15.62:3300/0 conn(0x7eff4c0d0580 0x7eff4c0d1e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T15:43:14.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.507+0000 7eff513b9640 1 --2- 172.21.15.16:0/35458657 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7eff4c0d2350 0x7eff4c0afe40 secure :-1 s=READY pgs=1358 cs=0 l=1 rev1=1 crypto rx=0x7eff4c0cfd70 tx=0x7eff4401c9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35fd1f0c220ed825 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:43:14.513 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.507+0000 7eff4b7fe640 1 -- 172.21.15.16:0/35458657 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff440a5860 con 0x7eff4c0d2350 2023-05-22T15:43:14.513 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.507+0000 7eff4b7fe640 1 -- 172.21.15.16:0/35458657 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff440a59c0 con 0x7eff4c0d2350 2023-05-22T15:43:14.513 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.507+0000 7eff4b7fe640 1 -- 172.21.15.16:0/35458657 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff440a5d50 con 0x7eff4c0d2350 2023-05-22T15:43:14.518 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.507+0000 7eff51bba640 1 -- 172.21.15.16:0/35458657 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7eff4c0d2350 msgr2=0x7eff4c0afe40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:14.519 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.507+0000 7eff51bba640 1 --2- 172.21.15.16:0/35458657 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7eff4c0d2350 0x7eff4c0afe40 secure :-1 s=READY pgs=1358 cs=0 l=1 rev1=1 crypto rx=0x7eff4c0cfd70 tx=0x7eff4401c9a0 comp rx=0 tx=0).stop 2023-05-22T15:43:14.519 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.507+0000 7eff51bba640 1 -- 172.21.15.16:0/35458657 shutdown_connections 2023-05-22T15:43:14.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.507+0000 7eff51bba640 1 --2- 172.21.15.16:0/35458657 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7eff4c0d2350 0x7eff4c0afe40 unknown :-1 s=CLOSED pgs=1358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:14.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.507+0000 7eff51bba640 1 --2- 172.21.15.16:0/35458657 >> v2:172.21.15.62:3300/0 conn(0x7eff4c0d0580 0x7eff4c0d1e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:14.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.507+0000 7eff51bba640 1 -- 172.21.15.16:0/35458657 >> 172.21.15.16:0/35458657 conn(0x7eff4c071770 msgr2=0x7eff4c09b580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:43:14.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.507+0000 7eff51bba640 1 -- 172.21.15.16:0/35458657 shutdown_connections 2023-05-22T15:43:14.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.507+0000 7eff51bba640 1 -- 172.21.15.16:0/35458657 wait complete. 2023-05-22T15:43:14.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.507+0000 7eff51bba640 1 Processor -- start 2023-05-22T15:43:14.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.507+0000 7eff51bba640 1 -- start start 2023-05-22T15:43:14.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.511+0000 7eff51bba640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7eff4c0d0580 0x7eff4c14e4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:14.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.511+0000 7eff51bba640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7eff4c0d2350 0x7eff4c06af10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:14.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.511+0000 7eff51bba640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7eff4c15bda0 0x7eff4c06b450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:14.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.511+0000 7eff51bba640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff4c14ea20 con 0x7eff4c15bda0 2023-05-22T15:43:14.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.511+0000 7eff51bba640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff4c154cb0 con 0x7eff4c0d2350 2023-05-22T15:43:14.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.511+0000 7eff51bba640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff4c15c4a0 con 0x7eff4c0d0580 2023-05-22T15:43:14.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.511+0000 7eff513b9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7eff4c15bda0 0x7eff4c06b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:14.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.511+0000 7eff4bfff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7eff4c0d2350 0x7eff4c06af10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:14.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.511+0000 7eff513b9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7eff4c15bda0 0x7eff4c06b450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45758/0 (socket says 172.21.15.16:45758) 2023-05-22T15:43:14.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.511+0000 7eff513b9640 1 -- 172.21.15.16:0/411206292 learned_addr learned my addr 172.21.15.16:0/411206292 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:43:14.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.511+0000 7eff50bb8640 1 --2- 172.21.15.16:0/411206292 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7eff4c0d0580 0x7eff4c14e4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:14.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.511+0000 7eff513b9640 1 -- 172.21.15.16:0/411206292 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7eff4c0d0580 msgr2=0x7eff4c14e4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:14.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.511+0000 7eff513b9640 1 --2- 172.21.15.16:0/411206292 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7eff4c0d0580 0x7eff4c14e4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:14.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.511+0000 7eff513b9640 1 -- 172.21.15.16:0/411206292 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7eff4c0d2350 msgr2=0x7eff4c06af10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:14.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.511+0000 7eff513b9640 1 --2- 172.21.15.16:0/411206292 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7eff4c0d2350 0x7eff4c06af10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:14.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.511+0000 7eff513b9640 1 -- 172.21.15.16:0/411206292 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff44009b30 con 0x7eff4c15bda0 2023-05-22T15:43:14.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.511+0000 7eff513b9640 1 --2- 172.21.15.16:0/411206292 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7eff4c15bda0 0x7eff4c06b450 secure :-1 s=READY pgs=1359 cs=0 l=1 rev1=1 crypto rx=0x7eff440195b0 tx=0x7eff44019740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:43:14.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.511+0000 7eff497fa640 1 -- 172.21.15.16:0/411206292 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff440b8670 con 0x7eff4c15bda0 2023-05-22T15:43:14.526 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.511+0000 7eff497fa640 1 -- 172.21.15.16:0/411206292 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff440b8810 con 0x7eff4c15bda0 2023-05-22T15:43:14.526 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.511+0000 7eff497fa640 1 -- 172.21.15.16:0/411206292 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff440b8bc0 con 0x7eff4c15bda0 2023-05-22T15:43:14.526 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.511+0000 7eff51bba640 1 -- 172.21.15.16:0/411206292 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff4c15c170 con 0x7eff4c15bda0 2023-05-22T15:43:14.526 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.511+0000 7eff51bba640 1 -- 172.21.15.16:0/411206292 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff4c0d27b0 con 0x7eff4c15bda0 2023-05-22T15:43:14.527 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.511+0000 7eff51bba640 1 -- 172.21.15.16:0/411206292 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff4c15c300 con 0x7eff4c15bda0 2023-05-22T15:43:14.527 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.511+0000 7eff497fa640 1 -- 172.21.15.16:0/411206292 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 38) v1 ==== 84389+0+0 (secure 0 0 0) 0x7eff440b8d20 con 0x7eff4c15bda0 2023-05-22T15:43:14.527 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.511+0000 7eff497fa640 1 --2- 172.21.15.16:0/411206292 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7eff30065530 0x7eff300679e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:14.527 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.511+0000 7eff50bb8640 1 -- 172.21.15.16:0/411206292 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7eff30065530 msgr2=0x7eff300679e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/36921 2023-05-22T15:43:14.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.511+0000 7eff497fa640 1 -- 172.21.15.16:0/411206292 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 6726+0+0 (secure 0 0 0) 0x7eff4411ee70 con 0x7eff4c15bda0 2023-05-22T15:43:14.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.511+0000 7eff50bb8640 1 --2- 172.21.15.16:0/411206292 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7eff30065530 0x7eff300679e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-05-22T15:43:14.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.515+0000 7eff497fa640 1 -- 172.21.15.16:0/411206292 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7eff440ec540 con 0x7eff4c15bda0 2023-05-22T15:43:14.553 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:43:14.547+0000 7f9700ac6180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-05-22T15:43:14.563 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:43:14.559+0000 7ffb72709180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-05-22T15:43:14.631 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:43:14.627+0000 7f9700ac6180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-05-22T15:43:14.670 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.667+0000 7eff51bba640 1 -- 172.21.15.16:0/411206292 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7eff4c15c300 con 0x7eff4c15bda0 2023-05-22T15:43:14.671 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.667+0000 7eff497fa640 1 -- 172.21.15.16:0/411206292 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v105) v1 ==== 93+0+10 (secure 0 0 0) 0x7eff4c15c300 con 0x7eff4c15bda0 2023-05-22T15:43:14.671 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2023-05-22T15:43:14.673 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.667+0000 7eff51bba640 1 -- 172.21.15.16:0/411206292 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7eff30065530 msgr2=0x7eff300679e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-05-22T15:43:14.673 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.667+0000 7eff51bba640 1 --2- 172.21.15.16:0/411206292 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7eff30065530 0x7eff300679e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:14.673 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.667+0000 7eff51bba640 1 -- 172.21.15.16:0/411206292 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7eff4c15bda0 msgr2=0x7eff4c06b450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:14.673 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.667+0000 7eff51bba640 1 --2- 172.21.15.16:0/411206292 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7eff4c15bda0 0x7eff4c06b450 secure :-1 s=READY pgs=1359 cs=0 l=1 rev1=1 crypto rx=0x7eff440195b0 tx=0x7eff44019740 comp rx=0 tx=0).stop 2023-05-22T15:43:14.673 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.667+0000 7eff51bba640 1 -- 172.21.15.16:0/411206292 shutdown_connections 2023-05-22T15:43:14.674 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.667+0000 7eff51bba640 1 --2- 172.21.15.16:0/411206292 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7eff4c15bda0 0x7eff4c06b450 unknown :-1 s=CLOSED pgs=1359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:14.674 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.667+0000 7eff51bba640 1 --2- 172.21.15.16:0/411206292 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7eff4c0d2350 0x7eff4c06af10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:14.674 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.667+0000 7eff51bba640 1 --2- 172.21.15.16:0/411206292 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7eff30065530 0x7eff300679e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:14.674 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.667+0000 7eff51bba640 1 --2- 172.21.15.16:0/411206292 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7eff4c0d0580 0x7eff4c14e4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:14.674 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.667+0000 7eff51bba640 1 -- 172.21.15.16:0/411206292 >> 172.21.15.16:0/411206292 conn(0x7eff4c071770 msgr2=0x7eff4c06a7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:43:14.674 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.667+0000 7eff51bba640 1 -- 172.21.15.16:0/411206292 shutdown_connections 2023-05-22T15:43:14.674 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.667+0000 7eff51bba640 1 -- 172.21.15.16:0/411206292 wait complete. 2023-05-22T15:43:14.688 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-05-22T15:43:14.810 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:43:14.807+0000 7ffb72709180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-05-22T15:43:14.843 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.839+0000 7f509bf43640 1 Processor -- start 2023-05-22T15:43:14.843 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.839+0000 7f509bf43640 1 -- start start 2023-05-22T15:43:14.844 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.839+0000 7f509bf43640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f50940b2880 0x7f50940cbfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:14.844 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.839+0000 7f509bf43640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f50940cc520 0x7f50940c3db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:14.844 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.839+0000 7f509bf43640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50940c6ba0 con 0x7f50940c67d0 2023-05-22T15:43:14.845 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.839+0000 7f509bf43640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f509415a300 con 0x7f50940cc520 2023-05-22T15:43:14.845 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.839+0000 7f509bf43640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50940cfef0 con 0x7f50940b2880 2023-05-22T15:43:14.845 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.839+0000 7f5099cdf640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f50940c67d0 0x7f50940cf730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:57108/0 (socket says 172.21.15.16:57108) 2023-05-22T15:43:14.845 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.839+0000 7f5099cdf640 1 -- 172.21.15.16:0/1050936526 learned_addr learned my addr 172.21.15.16:0/1050936526 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:43:14.845 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.839+0000 7f509a4e0640 1 --2- 172.21.15.16:0/1050936526 >> v2:172.21.15.62:3300/0 conn(0x7f50940cc520 0x7f50940c3db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:14.845 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.839+0000 7f50994de640 1 --2- 172.21.15.16:0/1050936526 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f50940b2880 0x7f50940cbfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:14.845 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.839+0000 7f5098cdd640 1 -- 172.21.15.16:0/1050936526 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3138531479 0 0) 0x7f50940c6ba0 con 0x7f50940c67d0 2023-05-22T15:43:14.846 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.839+0000 7f5098cdd640 1 -- 172.21.15.16:0/1050936526 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5080003530 con 0x7f50940c67d0 2023-05-22T15:43:14.846 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.839+0000 7f5098cdd640 1 -- 172.21.15.16:0/1050936526 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1378770542 0 0) 0x7f5080003530 con 0x7f50940c67d0 2023-05-22T15:43:14.846 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.839+0000 7f509a4e0640 1 -- 172.21.15.16:0/1050936526 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f50940b2880 msgr2=0x7f50940cbfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:43:14.847 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.839+0000 7f509a4e0640 1 --2- 172.21.15.16:0/1050936526 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f50940b2880 0x7f50940cbfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:14.847 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.839+0000 7f509a4e0640 1 -- 172.21.15.16:0/1050936526 >> v1:172.21.15.16:6789/0 conn(0x7f50940c67d0 legacy=0x7f50940cf730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:14.847 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.839+0000 7f509a4e0640 1 -- 172.21.15.16:0/1050936526 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50940d01d0 con 0x7f50940cc520 2023-05-22T15:43:14.848 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.843+0000 7f509a4e0640 1 --2- 172.21.15.16:0/1050936526 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f50940cc520 0x7f50940c3db0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f508c009d90 tx=0x7f508c01c7b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=587eb0b4e90d14b1 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:43:14.848 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.843+0000 7f5098cdd640 1 -- 172.21.15.16:0/1050936526 <== mon.1 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5090001680 con 0x7f50940cc520 2023-05-22T15:43:14.848 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.843+0000 7f5098cdd640 1 -- 172.21.15.16:0/1050936526 <== mon.1 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f508c0a55b0 con 0x7f50940cc520 2023-05-22T15:43:14.848 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.843+0000 7f5098cdd640 1 -- 172.21.15.16:0/1050936526 <== mon.1 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f508c0a5940 con 0x7f50940cc520 2023-05-22T15:43:14.849 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.843+0000 7f509bf43640 1 -- 172.21.15.16:0/1050936526 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f50940cc520 msgr2=0x7f50940c3db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:14.849 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.843+0000 7f509bf43640 1 --2- 172.21.15.16:0/1050936526 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f50940cc520 0x7f50940c3db0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f508c009d90 tx=0x7f508c01c7b0 comp rx=0 tx=0).stop 2023-05-22T15:43:14.849 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.843+0000 7f509bf43640 1 -- 172.21.15.16:0/1050936526 shutdown_connections 2023-05-22T15:43:14.849 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.843+0000 7f509bf43640 1 --2- 172.21.15.16:0/1050936526 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f50940cc520 0x7f50940c3db0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:14.849 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.843+0000 7f509bf43640 1 --2- 172.21.15.16:0/1050936526 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f50940b2880 0x7f50940cbfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:14.849 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.843+0000 7f509bf43640 1 -- 172.21.15.16:0/1050936526 >> 172.21.15.16:0/1050936526 conn(0x7f509406f5a0 msgr2=0x7f509406f9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:43:14.849 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.843+0000 7f509bf43640 1 -- 172.21.15.16:0/1050936526 shutdown_connections 2023-05-22T15:43:14.850 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.843+0000 7f509bf43640 1 -- 172.21.15.16:0/1050936526 wait complete. 2023-05-22T15:43:14.851 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.843+0000 7f509bf43640 1 Processor -- start 2023-05-22T15:43:14.851 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.843+0000 7f509bf43640 1 -- start start 2023-05-22T15:43:14.851 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.843+0000 7f509bf43640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f50940b2880 0x7f509414cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:14.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.843+0000 7f509bf43640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f50940c67d0 0x7f50940b3090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:14.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.843+0000 7f509bf43640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50940cc520 0x7f50941551b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:14.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.843+0000 7f509bf43640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50940b5840 con 0x7f50940cc520 2023-05-22T15:43:14.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.843+0000 7f509bf43640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5094154760 con 0x7f50940c67d0 2023-05-22T15:43:14.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.843+0000 7f509bf43640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5094154d20 con 0x7f50940b2880 2023-05-22T15:43:14.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.843+0000 7f509a4e0640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50940cc520 0x7f50941551b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:14.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.843+0000 7f509a4e0640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50940cc520 0x7f50941551b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45772/0 (socket says 172.21.15.16:45772) 2023-05-22T15:43:14.853 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.843+0000 7f509a4e0640 1 -- 172.21.15.16:0/3819741391 learned_addr learned my addr 172.21.15.16:0/3819741391 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:43:14.853 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.843+0000 7f50994de640 1 --2- 172.21.15.16:0/3819741391 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f50940c67d0 0x7f50940b3090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:14.853 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.847+0000 7f5099cdf640 1 --2- 172.21.15.16:0/3819741391 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f50940b2880 0x7f509414cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:14.853 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.847+0000 7f509a4e0640 1 -- 172.21.15.16:0/3819741391 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f50940b2880 msgr2=0x7f509414cc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:14.853 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.847+0000 7f509a4e0640 1 --2- 172.21.15.16:0/3819741391 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f50940b2880 0x7f509414cc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:14.853 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.847+0000 7f509a4e0640 1 -- 172.21.15.16:0/3819741391 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f50940c67d0 msgr2=0x7f50940b3090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:14.853 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.847+0000 7f509a4e0640 1 --2- 172.21.15.16:0/3819741391 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f50940c67d0 0x7f50940b3090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:14.854 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.847+0000 7f509a4e0640 1 -- 172.21.15.16:0/3819741391 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f508c009a70 con 0x7f50940cc520 2023-05-22T15:43:14.854 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.847+0000 7f509a4e0640 1 --2- 172.21.15.16:0/3819741391 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50940cc520 0x7f50941551b0 secure :-1 s=READY pgs=1361 cs=0 l=1 rev1=1 crypto rx=0x7f508c009d90 tx=0x7f508c019550 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:43:14.854 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.847+0000 7f508a7fc640 1 -- 172.21.15.16:0/3819741391 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f508c009450 con 0x7f50940cc520 2023-05-22T15:43:14.854 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.847+0000 7f508a7fc640 1 -- 172.21.15.16:0/3819741391 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f508c0a5e00 con 0x7f50940cc520 2023-05-22T15:43:14.854 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.847+0000 7f509bf43640 1 -- 172.21.15.16:0/3819741391 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50940cc9a0 con 0x7f50940cc520 2023-05-22T15:43:14.854 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.847+0000 7f508a7fc640 1 -- 172.21.15.16:0/3819741391 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f508c0ad890 con 0x7f50940cc520 2023-05-22T15:43:14.854 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.847+0000 7f509bf43640 1 -- 172.21.15.16:0/3819741391 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50940d0a70 con 0x7f50940cc520 2023-05-22T15:43:14.855 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.847+0000 7f509bf43640 1 -- 172.21.15.16:0/3819741391 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50940d0f90 con 0x7f50940cc520 2023-05-22T15:43:14.855 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.847+0000 7f508a7fc640 1 -- 172.21.15.16:0/3819741391 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 38) v1 ==== 84389+0+0 (secure 0 0 0) 0x7f508c0ad9f0 con 0x7f50940cc520 2023-05-22T15:43:14.856 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.847+0000 7f508a7fc640 1 --2- 172.21.15.16:0/3819741391 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f5078065560 0x7f5078067a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:14.856 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.847+0000 7f5099cdf640 1 -- 172.21.15.16:0/3819741391 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f5078065560 msgr2=0x7f5078067a10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/36921 2023-05-22T15:43:14.856 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.847+0000 7f508a7fc640 1 -- 172.21.15.16:0/3819741391 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 6726+0+0 (secure 0 0 0) 0x7f508c11e080 con 0x7f50940cc520 2023-05-22T15:43:14.856 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.847+0000 7f5099cdf640 1 --2- 172.21.15.16:0/3819741391 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f5078065560 0x7f5078067a10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-05-22T15:43:14.857 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.851+0000 7f508a7fc640 1 -- 172.21.15.16:0/3819741391 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f508c0eb860 con 0x7f50940cc520 2023-05-22T15:43:14.865 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:43:14.863+0000 7ffb72709180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-05-22T15:43:14.884 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:43:14.879+0000 7f9700ac6180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-05-22T15:43:14.939 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:43:14.935+0000 7f9700ac6180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-05-22T15:43:14.974 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.967+0000 7f508a7fc640 1 -- 172.21.15.16:0/3819741391 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mgrmap(e 39) v1 ==== 83941+0+0 (secure 0 0 0) 0x7f508c0ebc30 con 0x7f50940cc520 2023-05-22T15:43:14.974 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.971+0000 7f508a7fc640 1 -- 172.21.15.16:0/3819741391 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f5078065560 msgr2=0x7f5078067a10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-05-22T15:43:14.975 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:14.971+0000 7f508a7fc640 1 --2- 172.21.15.16:0/3819741391 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f5078065560 0x7f5078067a10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:15.019 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.015+0000 7f509bf43640 1 -- 172.21.15.16:0/3819741391 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f5094154e90 con 0x7f50940cc520 2023-05-22T15:43:15.019 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.015+0000 7f508a7fc640 1 -- 172.21.15.16:0/3819741391 <== mon.0 v2:172.21.15.16:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v106) v1 ==== 92+0+10 (secure 0 0 0) 0x7f508c104840 con 0x7f50940cc520 2023-05-22T15:43:15.019 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 8 2023-05-22T15:43:15.021 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.015+0000 7f509bf43640 1 -- 172.21.15.16:0/3819741391 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50940cc520 msgr2=0x7f50941551b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:15.021 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.015+0000 7f509bf43640 1 --2- 172.21.15.16:0/3819741391 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50940cc520 0x7f50941551b0 secure :-1 s=READY pgs=1361 cs=0 l=1 rev1=1 crypto rx=0x7f508c009d90 tx=0x7f508c019550 comp rx=0 tx=0).stop 2023-05-22T15:43:15.021 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.015+0000 7f509bf43640 1 -- 172.21.15.16:0/3819741391 shutdown_connections 2023-05-22T15:43:15.022 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.015+0000 7f509bf43640 1 --2- 172.21.15.16:0/3819741391 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50940cc520 0x7f50941551b0 unknown :-1 s=CLOSED pgs=1361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:15.022 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.015+0000 7f509bf43640 1 --2- 172.21.15.16:0/3819741391 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f50940c67d0 0x7f50940b3090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:15.022 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.015+0000 7f509bf43640 1 --2- 172.21.15.16:0/3819741391 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f5078065560 0x7f5078067a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:15.022 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.015+0000 7f509bf43640 1 --2- 172.21.15.16:0/3819741391 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f50940b2880 0x7f509414cc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:15.022 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.015+0000 7f509bf43640 1 -- 172.21.15.16:0/3819741391 >> 172.21.15.16:0/3819741391 conn(0x7f509406f5a0 msgr2=0x7f50940b0ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:43:15.022 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.015+0000 7f509bf43640 1 -- 172.21.15.16:0/3819741391 shutdown_connections 2023-05-22T15:43:15.022 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.015+0000 7f509bf43640 1 -- 172.21.15.16:0/3819741391 wait complete. 2023-05-22T15:43:15.040 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-05-22T15:43:15.058 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:43:15.051+0000 7f968b452640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T15:43:15.058 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:43:15.051+0000 7f968b452640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T15:43:15.058 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:43:15.051+0000 7f968b452640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T15:43:15.058 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:43:15.051+0000 7f968b452640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T15:43:15.058 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:43:15.051+0000 7f968b452640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T15:43:15.059 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:43:15.055+0000 7f9688c4d640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T15:43:15.059 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:43:15.055+0000 7f9688c4d640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T15:43:15.059 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:43:15.055+0000 7f9688c4d640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T15:43:15.059 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:43:15.055+0000 7f9688c4d640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T15:43:15.060 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:43:15.055+0000 7f9688c4d640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T15:43:15.157 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.151+0000 7f62c0668640 1 Processor -- start 2023-05-22T15:43:15.157 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.151+0000 7f62c0668640 1 -- start start 2023-05-22T15:43:15.158 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.151+0000 7f62c0668640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f62b80cf870 0x7f62b806db00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:15.158 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.151+0000 7f62c0668640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f62b806e740 0x7f62b806ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:15.158 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.151+0000 7f62c0668640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62b806e040 con 0x7f62b815cfa0 2023-05-22T15:43:15.159 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.151+0000 7f62c0668640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62b80cb9f0 con 0x7f62b806e740 2023-05-22T15:43:15.159 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.151+0000 7f62c0668640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62b806f100 con 0x7f62b80cf870 2023-05-22T15:43:15.159 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.151+0000 7f62bec05640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f62b806e740 0x7f62b806ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:15.159 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.151+0000 7f62bec05640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f62b806e740 0x7f62b806ebc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45894/0 (socket says 172.21.15.16:45894) 2023-05-22T15:43:15.159 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.151+0000 7f62bec05640 1 -- 172.21.15.16:0/2824160936 learned_addr learned my addr 172.21.15.16:0/2824160936 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:43:15.159 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.155+0000 7f62bdc03640 1 --2- 172.21.15.16:0/2824160936 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f62b80cf870 0x7f62b806db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:15.159 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.155+0000 7f62bec05640 1 -- 172.21.15.16:0/2824160936 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f62b80cf870 msgr2=0x7f62b806db00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:43:15.160 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.155+0000 7f62bec05640 1 --2- 172.21.15.16:0/2824160936 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f62b80cf870 0x7f62b806db00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:15.160 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.155+0000 7f62bec05640 1 -- 172.21.15.16:0/2824160936 >> v1:172.21.15.62:6789/0 conn(0x7f62b815cfa0 legacy=0x7f62b80b1fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:43:15.160 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.155+0000 7f62bec05640 1 -- 172.21.15.16:0/2824160936 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62b806e490 con 0x7f62b806e740 2023-05-22T15:43:15.161 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.155+0000 7f62bec05640 1 --2- 172.21.15.16:0/2824160936 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f62b806e740 0x7f62b806ebc0 secure :-1 s=READY pgs=1375 cs=0 l=1 rev1=1 crypto rx=0x7f62b0009e50 tx=0x7f62b001c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab31c64ef4e07ca1 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:43:15.161 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.155+0000 7f62bd402640 1 -- 172.21.15.16:0/2824160936 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62b00a5810 con 0x7f62b806e740 2023-05-22T15:43:15.161 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.155+0000 7f62bd402640 1 -- 172.21.15.16:0/2824160936 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62b00a5970 con 0x7f62b806e740 2023-05-22T15:43:15.161 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.155+0000 7f62bd402640 1 -- 172.21.15.16:0/2824160936 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62b00a5d00 con 0x7f62b806e740 2023-05-22T15:43:15.162 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.155+0000 7f62c0668640 1 -- 172.21.15.16:0/2824160936 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f62b806e740 msgr2=0x7f62b806ebc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:15.162 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.155+0000 7f62c0668640 1 --2- 172.21.15.16:0/2824160936 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f62b806e740 0x7f62b806ebc0 secure :-1 s=READY pgs=1375 cs=0 l=1 rev1=1 crypto rx=0x7f62b0009e50 tx=0x7f62b001c950 comp rx=0 tx=0).stop 2023-05-22T15:43:15.162 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.155+0000 7f62c0668640 1 -- 172.21.15.16:0/2824160936 shutdown_connections 2023-05-22T15:43:15.162 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.155+0000 7f62c0668640 1 --2- 172.21.15.16:0/2824160936 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f62b806e740 0x7f62b806ebc0 unknown :-1 s=CLOSED pgs=1375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:15.162 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.155+0000 7f62c0668640 1 --2- 172.21.15.16:0/2824160936 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f62b80cf870 0x7f62b806db00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:15.162 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.155+0000 7f62c0668640 1 -- 172.21.15.16:0/2824160936 >> 172.21.15.16:0/2824160936 conn(0x7f62b80cc870 msgr2=0x7f62b80cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:43:15.163 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.155+0000 7f62c0668640 1 -- 172.21.15.16:0/2824160936 shutdown_connections 2023-05-22T15:43:15.163 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.155+0000 7f62c0668640 1 -- 172.21.15.16:0/2824160936 wait complete. 2023-05-22T15:43:15.166 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.159+0000 7f62c0668640 1 Processor -- start 2023-05-22T15:43:15.167 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.159+0000 7f62c0668640 1 -- start start 2023-05-22T15:43:15.167 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.159+0000 7f62c0668640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f62b806e740 0x7f62b81480a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:15.167 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.159+0000 7f62c0668640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f62b80cf870 0x7f62b81490c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:15.167 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.159+0000 7f62c0668640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f62b815cfa0 0x7f62b8149600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:15.168 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.159+0000 7f62c0668640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62b814a8b0 con 0x7f62b815cfa0 2023-05-22T15:43:15.168 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.159+0000 7f62c0668640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62b80c6340 con 0x7f62b80cf870 2023-05-22T15:43:15.168 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.159+0000 7f62c0668640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62b8149c30 con 0x7f62b806e740 2023-05-22T15:43:15.168 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.159+0000 7f62bec05640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f62b815cfa0 0x7f62b8149600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:15.168 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.159+0000 7f62bec05640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f62b815cfa0 0x7f62b8149600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45900/0 (socket says 172.21.15.16:45900) 2023-05-22T15:43:15.168 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.159+0000 7f62bec05640 1 -- 172.21.15.16:0/1905293584 learned_addr learned my addr 172.21.15.16:0/1905293584 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:43:15.169 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.159+0000 7f62be404640 1 --2- 172.21.15.16:0/1905293584 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f62b806e740 0x7f62b81480a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:15.169 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.159+0000 7f62bdc03640 1 --2- 172.21.15.16:0/1905293584 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f62b80cf870 0x7f62b81490c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:15.169 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.159+0000 7f62bec05640 1 -- 172.21.15.16:0/1905293584 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f62b806e740 msgr2=0x7f62b81480a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:15.169 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.159+0000 7f62bec05640 1 --2- 172.21.15.16:0/1905293584 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f62b806e740 0x7f62b81480a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:15.169 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.159+0000 7f62bec05640 1 -- 172.21.15.16:0/1905293584 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f62b80cf870 msgr2=0x7f62b81490c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:15.169 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.159+0000 7f62bec05640 1 --2- 172.21.15.16:0/1905293584 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f62b80cf870 0x7f62b81490c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:15.169 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.159+0000 7f62bec05640 1 -- 172.21.15.16:0/1905293584 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62b0009b30 con 0x7f62b815cfa0 2023-05-22T15:43:15.170 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.159+0000 7f62bdc03640 1 --2- 172.21.15.16:0/1905293584 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f62b80cf870 0x7f62b81490c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:43:15.170 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.159+0000 7f62bec05640 1 --2- 172.21.15.16:0/1905293584 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f62b815cfa0 0x7f62b8149600 secure :-1 s=READY pgs=1376 cs=0 l=1 rev1=1 crypto rx=0x7f62b0009e50 tx=0x7f62b00196f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:43:15.170 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.159+0000 7f62aeffd640 1 -- 172.21.15.16:0/1905293584 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62b00b8670 con 0x7f62b815cfa0 2023-05-22T15:43:15.170 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.159+0000 7f62aeffd640 1 -- 172.21.15.16:0/1905293584 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62b00b87d0 con 0x7f62b815cfa0 2023-05-22T15:43:15.170 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.159+0000 7f62aeffd640 1 -- 172.21.15.16:0/1905293584 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62b00b8b60 con 0x7f62b815cfa0 2023-05-22T15:43:15.171 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.159+0000 7f62c0668640 1 -- 172.21.15.16:0/1905293584 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62b80cb610 con 0x7f62b815cfa0 2023-05-22T15:43:15.171 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.159+0000 7f62c0668640 1 -- 172.21.15.16:0/1905293584 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62b80cb330 con 0x7f62b815cfa0 2023-05-22T15:43:15.171 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.159+0000 7f62c0668640 1 -- 172.21.15.16:0/1905293584 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62b815d370 con 0x7f62b815cfa0 2023-05-22T15:43:15.171 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.159+0000 7f62aeffd640 1 -- 172.21.15.16:0/1905293584 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 39) v1 ==== 83941+0+0 (secure 0 0 0) 0x7f62b00b8cc0 con 0x7f62b815cfa0 2023-05-22T15:43:15.171 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.159+0000 7f62aeffd640 1 -- 172.21.15.16:0/1905293584 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 7027+0+0 (secure 0 0 0) 0x7f62b011dca0 con 0x7f62b815cfa0 2023-05-22T15:43:15.172 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.163+0000 7f62aeffd640 1 -- 172.21.15.16:0/1905293584 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f62b0122070 con 0x7f62b815cfa0 2023-05-22T15:43:15.318 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.315+0000 7f62c0668640 1 -- 172.21.15.16:0/1905293584 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f62b814a9f0 con 0x7f62b815cfa0 2023-05-22T15:43:15.319 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.315+0000 7f62aeffd640 1 -- 172.21.15.16:0/1905293584 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v106) v1 ==== 104+0+11 (secure 0 0 0) 0x7f62b814a9f0 con 0x7f62b815cfa0 2023-05-22T15:43:15.319 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2023-05-22T15:43:15.321 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.315+0000 7f62c0668640 1 -- 172.21.15.16:0/1905293584 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f62b815cfa0 msgr2=0x7f62b8149600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:15.321 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.315+0000 7f62c0668640 1 --2- 172.21.15.16:0/1905293584 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f62b815cfa0 0x7f62b8149600 secure :-1 s=READY pgs=1376 cs=0 l=1 rev1=1 crypto rx=0x7f62b0009e50 tx=0x7f62b00196f0 comp rx=0 tx=0).stop 2023-05-22T15:43:15.321 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.315+0000 7f62c0668640 1 -- 172.21.15.16:0/1905293584 shutdown_connections 2023-05-22T15:43:15.322 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.315+0000 7f62c0668640 1 --2- 172.21.15.16:0/1905293584 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f62b815cfa0 0x7f62b8149600 unknown :-1 s=CLOSED pgs=1376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:15.322 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.315+0000 7f62c0668640 1 --2- 172.21.15.16:0/1905293584 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f62b80cf870 0x7f62b81490c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:15.322 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.315+0000 7f62c0668640 1 --2- 172.21.15.16:0/1905293584 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f62b806e740 0x7f62b81480a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:15.322 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.315+0000 7f62c0668640 1 -- 172.21.15.16:0/1905293584 >> 172.21.15.16:0/1905293584 conn(0x7f62b80cc870 msgr2=0x7f62b8151360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:43:15.322 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.315+0000 7f62c0668640 1 -- 172.21.15.16:0/1905293584 shutdown_connections 2023-05-22T15:43:15.322 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.315+0000 7f62c0668640 1 -- 172.21.15.16:0/1905293584 wait complete. 2023-05-22T15:43:15.337 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-05-22T15:43:15.503 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.499+0000 7f0886d87640 1 Processor -- start 2023-05-22T15:43:15.503 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.499+0000 7f0886d87640 1 -- start start 2023-05-22T15:43:15.504 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.499+0000 7f0886d87640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f0880159970 0x7f0880159d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:15.504 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.499+0000 7f0886d87640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f088015db00 0x7f088006b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:15.505 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.499+0000 7f0886d87640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0880070c30 con 0x7f088015d680 2023-05-22T15:43:15.505 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.499+0000 7f0886d87640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08800cb970 con 0x7f0880159970 2023-05-22T15:43:15.505 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.499+0000 7f0886d87640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0880070e60 con 0x7f088015db00 2023-05-22T15:43:15.505 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.499+0000 7f087ffff640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f088015d680 0x7f08800cb260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:57112/0 (socket says 172.21.15.16:57112) 2023-05-22T15:43:15.505 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.499+0000 7f087ffff640 1 -- 172.21.15.16:0/646296448 learned_addr learned my addr 172.21.15.16:0/646296448 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:43:15.505 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.499+0000 7f0884b23640 1 --2- 172.21.15.16:0/646296448 >> v2:172.21.15.62:3300/0 conn(0x7f0880159970 0x7f0880159d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:15.506 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.499+0000 7f0885324640 1 --2- 172.21.15.16:0/646296448 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f088015db00 0x7f088006b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:15.507 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.499+0000 7f087f7fe640 1 -- 172.21.15.16:0/646296448 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2438615658 0 0) 0x7f0880070c30 con 0x7f088015d680 2023-05-22T15:43:15.507 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.499+0000 7f087f7fe640 1 -- 172.21.15.16:0/646296448 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0868003530 con 0x7f088015d680 2023-05-22T15:43:15.507 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.499+0000 7f087f7fe640 1 -- 172.21.15.16:0/646296448 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4084563897 0 0) 0x7f0868003530 con 0x7f088015d680 2023-05-22T15:43:15.508 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.499+0000 7f087f7fe640 1 -- 172.21.15.16:0/646296448 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f088015db00 msgr2=0x7f088006b6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:43:15.508 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.499+0000 7f087f7fe640 1 --2- 172.21.15.16:0/646296448 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f088015db00 0x7f088006b6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:15.508 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.499+0000 7f087f7fe640 1 -- 172.21.15.16:0/646296448 >> v2:172.21.15.62:3300/0 conn(0x7f0880159970 msgr2=0x7f0880159d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:43:15.508 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.499+0000 7f087f7fe640 1 --2- 172.21.15.16:0/646296448 >> v2:172.21.15.62:3300/0 conn(0x7f0880159970 0x7f0880159d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:15.508 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.499+0000 7f087f7fe640 1 -- 172.21.15.16:0/646296448 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0880071140 con 0x7f088015d680 2023-05-22T15:43:15.508 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.499+0000 7f087f7fe640 1 -- 172.21.15.16:0/646296448 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7f0874001680 con 0x7f088015d680 2023-05-22T15:43:15.509 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.499+0000 7f087f7fe640 1 -- 172.21.15.16:0/646296448 >> v1:172.21.15.16:6789/0 conn(0x7f088015d680 legacy=0x7f08800cb260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:15.509 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.499+0000 7f087f7fe640 1 --2- 172.21.15.16:0/646296448 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0868003980 0x7f0868003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:15.509 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.499+0000 7f087f7fe640 1 --2- 172.21.15.16:0/646296448 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f08680047d0 0x7f0868004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:15.509 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.499+0000 7f087f7fe640 1 --2- 172.21.15.16:0/646296448 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f088015db00 0x7f0868005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:15.509 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.499+0000 7f087f7fe640 1 -- 172.21.15.16:0/646296448 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08800cb970 con 0x7f0868003980 2023-05-22T15:43:15.510 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.499+0000 7f087f7fe640 1 -- 172.21.15.16:0/646296448 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0880070e60 con 0x7f08680047d0 2023-05-22T15:43:15.510 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.499+0000 7f087f7fe640 1 -- 172.21.15.16:0/646296448 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f086800b9c0 con 0x7f088015db00 2023-05-22T15:43:15.510 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.499+0000 7f0884b23640 1 --2- 172.21.15.16:0/646296448 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0868003980 0x7f0868003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:15.510 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.499+0000 7f0884b23640 1 -- 172.21.15.16:0/646296448 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f088015db00 msgr2=0x7f0868005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:43:15.510 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.499+0000 7f0884b23640 1 --2- 172.21.15.16:0/646296448 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f088015db00 0x7f0868005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:15.510 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.499+0000 7f0884b23640 1 -- 172.21.15.16:0/646296448 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f08680047d0 msgr2=0x7f0868004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:43:15.511 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.499+0000 7f0884b23640 1 --2- 172.21.15.16:0/646296448 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f08680047d0 0x7f0868004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:15.511 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.499+0000 7f0884b23640 1 -- 172.21.15.16:0/646296448 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0868003530 con 0x7f0868003980 2023-05-22T15:43:15.513 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.503+0000 7f0884b23640 1 --2- 172.21.15.16:0/646296448 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0868003980 0x7f0868003d90 secure :-1 s=READY pgs=1378 cs=0 l=1 rev1=1 crypto rx=0x7f0870009fc0 tx=0x7f087001cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4e4c0e338d765f server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:43:15.513 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.503+0000 7f087f7fe640 1 -- 172.21.15.16:0/646296448 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08700a5c60 con 0x7f0868003980 2023-05-22T15:43:15.513 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.503+0000 7f087f7fe640 1 -- 172.21.15.16:0/646296448 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f087000f040 con 0x7f0868003980 2023-05-22T15:43:15.514 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.503+0000 7f087f7fe640 1 -- 172.21.15.16:0/646296448 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08700ad440 con 0x7f0868003980 2023-05-22T15:43:15.514 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.503+0000 7f0886d87640 1 -- 172.21.15.16:0/646296448 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0868003980 msgr2=0x7f0868003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:15.514 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.503+0000 7f0886d87640 1 --2- 172.21.15.16:0/646296448 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0868003980 0x7f0868003d90 secure :-1 s=READY pgs=1378 cs=0 l=1 rev1=1 crypto rx=0x7f0870009fc0 tx=0x7f087001cc70 comp rx=0 tx=0).stop 2023-05-22T15:43:15.514 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.503+0000 7f0885324640 1 -- 172.21.15.16:0/646296448 reap_dead start 2023-05-22T15:43:15.514 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.503+0000 7f0886d87640 1 -- 172.21.15.16:0/646296448 shutdown_connections 2023-05-22T15:43:15.514 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.503+0000 7f0886d87640 1 -- 172.21.15.16:0/646296448 >> 172.21.15.16:0/646296448 conn(0x7f0880069720 msgr2=0x7f0880069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:43:15.514 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.503+0000 7f0886d87640 1 -- 172.21.15.16:0/646296448 shutdown_connections 2023-05-22T15:43:15.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.503+0000 7f0886d87640 1 -- 172.21.15.16:0/646296448 wait complete. 2023-05-22T15:43:15.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.503+0000 7f0886d87640 1 Processor -- start 2023-05-22T15:43:15.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.503+0000 7f0886d87640 1 -- start start 2023-05-22T15:43:15.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.507+0000 7f0886d87640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0868003980 0x7f08800aef10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:15.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.507+0000 7f0886d87640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f08801538a0 0x7f0880152a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:15.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.507+0000 7f0886d87640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08801532e0 0x7f08800b2a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:15.516 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.507+0000 7f0886d87640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08800aeaf0 con 0x7f08801532e0 2023-05-22T15:43:15.516 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.507+0000 7f0886d87640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08800ade70 con 0x7f08801538a0 2023-05-22T15:43:15.516 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.507+0000 7f0886d87640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08800ae150 con 0x7f0868003980 2023-05-22T15:43:15.516 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.507+0000 7f0885324640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08801532e0 0x7f08800b2a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:15.517 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.507+0000 7f0885324640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08801532e0 0x7f08800b2a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45916/0 (socket says 172.21.15.16:45916) 2023-05-22T15:43:15.517 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.507+0000 7f0885324640 1 -- 172.21.15.16:0/2295531635 learned_addr learned my addr 172.21.15.16:0/2295531635 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:43:15.517 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.507+0000 7f0884b23640 1 --2- 172.21.15.16:0/2295531635 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0868003980 0x7f08800aef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:15.517 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.507+0000 7f087ffff640 1 --2- 172.21.15.16:0/2295531635 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f08801538a0 0x7f0880152a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:15.518 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.507+0000 7f0885324640 1 -- 172.21.15.16:0/2295531635 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0868003980 msgr2=0x7f08800aef10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:15.518 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.507+0000 7f0885324640 1 --2- 172.21.15.16:0/2295531635 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0868003980 0x7f08800aef10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:15.518 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.507+0000 7f0885324640 1 -- 172.21.15.16:0/2295531635 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f08801538a0 msgr2=0x7f0880152a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:15.518 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.507+0000 7f0885324640 1 --2- 172.21.15.16:0/2295531635 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f08801538a0 0x7f0880152a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:15.519 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.507+0000 7f0885324640 1 -- 172.21.15.16:0/2295531635 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08800cae20 con 0x7f08801532e0 2023-05-22T15:43:15.519 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.507+0000 7f0884b23640 1 --2- 172.21.15.16:0/2295531635 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0868003980 0x7f08800aef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:43:15.519 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.507+0000 7f0885324640 1 --2- 172.21.15.16:0/2295531635 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08801532e0 0x7f08800b2a70 secure :-1 s=READY pgs=1379 cs=0 l=1 rev1=1 crypto rx=0x7f08800707b0 tx=0x7f087800e830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:43:15.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.507+0000 7f087d7fa640 1 -- 172.21.15.16:0/2295531635 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f087800ecd0 con 0x7f08801532e0 2023-05-22T15:43:15.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.507+0000 7f087d7fa640 1 -- 172.21.15.16:0/2295531635 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f087800ee30 con 0x7f08801532e0 2023-05-22T15:43:15.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.507+0000 7f0886d87640 1 -- 172.21.15.16:0/2295531635 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08800cac90 con 0x7f08801532e0 2023-05-22T15:43:15.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.507+0000 7f087d7fa640 1 -- 172.21.15.16:0/2295531635 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08780136b0 con 0x7f08801532e0 2023-05-22T15:43:15.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.507+0000 7f0886d87640 1 -- 172.21.15.16:0/2295531635 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f088015a5b0 con 0x7f08801532e0 2023-05-22T15:43:15.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.507+0000 7f0886d87640 1 -- 172.21.15.16:0/2295531635 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08800aec30 con 0x7f08801532e0 2023-05-22T15:43:15.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.507+0000 7f087d7fa640 1 -- 172.21.15.16:0/2295531635 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 39) v1 ==== 83941+0+0 (secure 0 0 0) 0x7f0878013810 con 0x7f08801532e0 2023-05-22T15:43:15.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.507+0000 7f087d7fa640 1 -- 172.21.15.16:0/2295531635 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 7027+0+0 (secure 0 0 0) 0x7f0878081ba0 con 0x7f08801532e0 2023-05-22T15:43:15.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.511+0000 7f087d7fa640 1 -- 172.21.15.16:0/2295531635 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f08800aec30 con 0x7f08801532e0 2023-05-22T15:43:15.663 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.659+0000 7f0886d87640 1 -- 172.21.15.16:0/2295531635 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f08800aec30 con 0x7f08801532e0 2023-05-22T15:43:15.663 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.659+0000 7f087d7fa640 1 -- 172.21.15.16:0/2295531635 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v106) v1 ==== 100+0+11 (secure 0 0 0) 0x7f087804f8a0 con 0x7f08801532e0 2023-05-22T15:43:15.664 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 64 2023-05-22T15:43:15.665 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.659+0000 7f0886d87640 1 -- 172.21.15.16:0/2295531635 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08801532e0 msgr2=0x7f08800b2a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:15.666 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.659+0000 7f0886d87640 1 --2- 172.21.15.16:0/2295531635 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08801532e0 0x7f08800b2a70 secure :-1 s=READY pgs=1379 cs=0 l=1 rev1=1 crypto rx=0x7f08800707b0 tx=0x7f087800e830 comp rx=0 tx=0).stop 2023-05-22T15:43:15.666 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.659+0000 7f0886d87640 1 -- 172.21.15.16:0/2295531635 shutdown_connections 2023-05-22T15:43:15.666 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.659+0000 7f0886d87640 1 --2- 172.21.15.16:0/2295531635 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08801532e0 0x7f08800b2a70 unknown :-1 s=CLOSED pgs=1379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:15.666 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.659+0000 7f0886d87640 1 --2- 172.21.15.16:0/2295531635 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f08801538a0 0x7f0880152a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:15.666 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.659+0000 7f0886d87640 1 --2- 172.21.15.16:0/2295531635 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0868003980 0x7f08800aef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:15.666 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.659+0000 7f0886d87640 1 -- 172.21.15.16:0/2295531635 >> 172.21.15.16:0/2295531635 conn(0x7f0880069720 msgr2=0x7f08801553a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:43:15.667 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.659+0000 7f0886d87640 1 -- 172.21.15.16:0/2295531635 shutdown_connections 2023-05-22T15:43:15.667 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.659+0000 7f0886d87640 1 -- 172.21.15.16:0/2295531635 wait complete. 2023-05-22T15:43:15.682 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-05-22T15:43:15.799 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.795+0000 7fe1264a6640 1 Processor -- start 2023-05-22T15:43:15.799 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.795+0000 7fe1264a6640 1 -- start start 2023-05-22T15:43:15.801 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.795+0000 7fe1264a6640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fe12006e250 0x7fe1200cc780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:15.802 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.795+0000 7fe1264a6640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe1200b5e80 0x7fe1200b3e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:15.802 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.795+0000 7fe1264a6640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe120159ee0 con 0x7fe120159b10 2023-05-22T15:43:15.802 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.795+0000 7fe1264a6640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1200b30b0 con 0x7fe12006e250 2023-05-22T15:43:15.802 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.795+0000 7fe1264a6640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1200b2810 con 0x7fe1200b5e80 2023-05-22T15:43:15.802 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.795+0000 7fe11ffff640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fe120159b10 0x7fe12006db40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:57114/0 (socket says 172.21.15.16:57114) 2023-05-22T15:43:15.802 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.795+0000 7fe11ffff640 1 -- 172.21.15.16:0/3171923078 learned_addr learned my addr 172.21.15.16:0/3171923078 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:43:15.803 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.795+0000 7fe124a43640 1 --2- 172.21.15.16:0/3171923078 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe1200b5e80 0x7fe1200b3e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:15.803 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.795+0000 7fe11f7fe640 1 --2- 172.21.15.16:0/3171923078 >> v2:172.21.15.62:3300/0 conn(0x7fe12006e250 0x7fe1200cc780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:15.803 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.795+0000 7fe11effd640 1 -- 172.21.15.16:0/3171923078 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1459919843 0 0) 0x7fe120159ee0 con 0x7fe120159b10 2023-05-22T15:43:15.803 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.795+0000 7fe11effd640 1 -- 172.21.15.16:0/3171923078 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe108003530 con 0x7fe120159b10 2023-05-22T15:43:15.803 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.795+0000 7fe11effd640 1 -- 172.21.15.16:0/3171923078 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3007504534 0 0) 0x7fe108003530 con 0x7fe120159b10 2023-05-22T15:43:15.803 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.795+0000 7fe11effd640 1 -- 172.21.15.16:0/3171923078 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe1200b5e80 msgr2=0x7fe1200b3e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:43:15.804 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.795+0000 7fe11effd640 1 --2- 172.21.15.16:0/3171923078 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe1200b5e80 0x7fe1200b3e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:15.804 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.795+0000 7fe11effd640 1 -- 172.21.15.16:0/3171923078 >> v2:172.21.15.62:3300/0 conn(0x7fe12006e250 msgr2=0x7fe1200cc780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:43:15.804 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.795+0000 7fe11effd640 1 --2- 172.21.15.16:0/3171923078 >> v2:172.21.15.62:3300/0 conn(0x7fe12006e250 0x7fe1200cc780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:15.804 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.795+0000 7fe11effd640 1 -- 172.21.15.16:0/3171923078 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1200b2af0 con 0x7fe120159b10 2023-05-22T15:43:15.804 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.795+0000 7fe11effd640 1 -- 172.21.15.16:0/3171923078 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7fe114001680 con 0x7fe120159b10 2023-05-22T15:43:15.804 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.795+0000 7fe11effd640 1 -- 172.21.15.16:0/3171923078 >> v1:172.21.15.16:6789/0 conn(0x7fe120159b10 legacy=0x7fe12006db40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:15.804 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.795+0000 7fe11effd640 1 --2- 172.21.15.16:0/3171923078 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe108003980 0x7fe108003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:15.805 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.795+0000 7fe11effd640 1 --2- 172.21.15.16:0/3171923078 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe1080043b0 0x7fe108004870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:15.805 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.795+0000 7fe11effd640 1 --2- 172.21.15.16:0/3171923078 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe1200b5e80 0x7fe108004f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:15.805 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.795+0000 7fe11effd640 1 -- 172.21.15.16:0/3171923078 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1200b30b0 con 0x7fe108003980 2023-05-22T15:43:15.805 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.795+0000 7fe11effd640 1 -- 172.21.15.16:0/3171923078 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1200b2810 con 0x7fe1080043b0 2023-05-22T15:43:15.805 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.795+0000 7fe11effd640 1 -- 172.21.15.16:0/3171923078 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe10800b990 con 0x7fe1200b5e80 2023-05-22T15:43:15.805 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.795+0000 7fe11ffff640 1 --2- 172.21.15.16:0/3171923078 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe108003980 0x7fe108003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:15.805 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.795+0000 7fe11f7fe640 1 --2- 172.21.15.16:0/3171923078 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe1080043b0 0x7fe108004870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:15.806 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.795+0000 7fe124a43640 1 --2- 172.21.15.16:0/3171923078 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe1200b5e80 0x7fe108004f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:15.806 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.795+0000 7fe11ffff640 1 -- 172.21.15.16:0/3171923078 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe1200b5e80 msgr2=0x7fe108004f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:43:15.806 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.795+0000 7fe11ffff640 1 --2- 172.21.15.16:0/3171923078 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe1200b5e80 0x7fe108004f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:15.806 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.795+0000 7fe11ffff640 1 -- 172.21.15.16:0/3171923078 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe1080043b0 msgr2=0x7fe108004870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:43:15.806 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.795+0000 7fe11ffff640 1 --2- 172.21.15.16:0/3171923078 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe1080043b0 0x7fe108004870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:15.807 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.795+0000 7fe11ffff640 1 -- 172.21.15.16:0/3171923078 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe108003530 con 0x7fe108003980 2023-05-22T15:43:15.807 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.795+0000 7fe124a43640 1 --2- 172.21.15.16:0/3171923078 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe1200b5e80 0x7fe108004f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:43:15.809 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.799+0000 7fe11ffff640 1 --2- 172.21.15.16:0/3171923078 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe108003980 0x7fe108003d90 secure :-1 s=READY pgs=1381 cs=0 l=1 rev1=1 crypto rx=0x7fe11400c210 tx=0x7fe11401cc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c1abae48d5db480 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:43:15.809 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.799+0000 7fe11effd640 1 -- 172.21.15.16:0/3171923078 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1140a5b30 con 0x7fe108003980 2023-05-22T15:43:15.810 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.799+0000 7fe11effd640 1 -- 172.21.15.16:0/3171923078 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1140a5c90 con 0x7fe108003980 2023-05-22T15:43:15.810 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.799+0000 7fe11effd640 1 -- 172.21.15.16:0/3171923078 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1140058e0 con 0x7fe108003980 2023-05-22T15:43:15.810 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.799+0000 7fe1264a6640 1 -- 172.21.15.16:0/3171923078 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe108003980 msgr2=0x7fe108003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:15.810 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.799+0000 7fe1264a6640 1 --2- 172.21.15.16:0/3171923078 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe108003980 0x7fe108003d90 secure :-1 s=READY pgs=1381 cs=0 l=1 rev1=1 crypto rx=0x7fe11400c210 tx=0x7fe11401cc50 comp rx=0 tx=0).stop 2023-05-22T15:43:15.810 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.799+0000 7fe124a43640 1 -- 172.21.15.16:0/3171923078 reap_dead start 2023-05-22T15:43:15.810 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.799+0000 7fe1264a6640 1 -- 172.21.15.16:0/3171923078 shutdown_connections 2023-05-22T15:43:15.810 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.799+0000 7fe1264a6640 1 -- 172.21.15.16:0/3171923078 >> 172.21.15.16:0/3171923078 conn(0x7fe12006f560 msgr2=0x7fe12006f960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:43:15.811 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.799+0000 7fe1264a6640 1 -- 172.21.15.16:0/3171923078 shutdown_connections 2023-05-22T15:43:15.811 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.799+0000 7fe1264a6640 1 -- 172.21.15.16:0/3171923078 wait complete. 2023-05-22T15:43:15.811 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.799+0000 7fe1264a6640 1 Processor -- start 2023-05-22T15:43:15.811 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.799+0000 7fe1264a6640 1 -- start start 2023-05-22T15:43:15.811 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.803+0000 7fe1264a6640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe108003980 0x7fe120161fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:15.811 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.803+0000 7fe1264a6640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe1200c9340 0x7fe120150330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:15.811 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.803+0000 7fe1264a6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe1200cbc30 0x7fe1200cc0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:15.812 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.803+0000 7fe1264a6640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe120152b50 con 0x7fe1200cbc30 2023-05-22T15:43:15.812 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.803+0000 7fe1264a6640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1200cc5e0 con 0x7fe108003980 2023-05-22T15:43:15.812 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.803+0000 7fe1264a6640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe12015e6c0 con 0x7fe1200c9340 2023-05-22T15:43:15.812 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.803+0000 7fe124a43640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe1200cbc30 0x7fe1200cc0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:15.812 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.803+0000 7fe124a43640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe1200cbc30 0x7fe1200cc0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45932/0 (socket says 172.21.15.16:45932) 2023-05-22T15:43:15.812 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.803+0000 7fe124a43640 1 -- 172.21.15.16:0/3628227184 learned_addr learned my addr 172.21.15.16:0/3628227184 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:43:15.813 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.803+0000 7fe11ffff640 1 --2- 172.21.15.16:0/3628227184 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe108003980 0x7fe120161fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:15.813 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.803+0000 7fe11f7fe640 1 --2- 172.21.15.16:0/3628227184 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe1200c9340 0x7fe120150330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:15.813 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.803+0000 7fe124a43640 1 -- 172.21.15.16:0/3628227184 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe1200c9340 msgr2=0x7fe120150330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:15.813 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.803+0000 7fe124a43640 1 --2- 172.21.15.16:0/3628227184 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe1200c9340 0x7fe120150330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:15.813 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.803+0000 7fe124a43640 1 -- 172.21.15.16:0/3628227184 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe108003980 msgr2=0x7fe120161fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:15.813 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.803+0000 7fe124a43640 1 --2- 172.21.15.16:0/3628227184 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe108003980 0x7fe120161fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:15.813 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.803+0000 7fe124a43640 1 -- 172.21.15.16:0/3628227184 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1200b4640 con 0x7fe1200cbc30 2023-05-22T15:43:15.814 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.803+0000 7fe11f7fe640 1 --2- 172.21.15.16:0/3628227184 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe1200c9340 0x7fe120150330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:43:15.814 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.803+0000 7fe124a43640 1 --2- 172.21.15.16:0/3628227184 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe1200cbc30 0x7fe1200cc0a0 secure :-1 s=READY pgs=1382 cs=0 l=1 rev1=1 crypto rx=0x7fe118009fd0 tx=0x7fe11800f0d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:43:15.814 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.803+0000 7fe11cff9640 1 -- 172.21.15.16:0/3628227184 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe11800f840 con 0x7fe1200cbc30 2023-05-22T15:43:15.814 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.803+0000 7fe11cff9640 1 -- 172.21.15.16:0/3628227184 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe118011040 con 0x7fe1200cbc30 2023-05-22T15:43:15.814 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.803+0000 7fe1264a6640 1 -- 172.21.15.16:0/3628227184 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1200b3910 con 0x7fe1200cbc30 2023-05-22T15:43:15.814 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.803+0000 7fe11cff9640 1 -- 172.21.15.16:0/3628227184 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe11800fbc0 con 0x7fe1200cbc30 2023-05-22T15:43:15.814 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.803+0000 7fe1264a6640 1 -- 172.21.15.16:0/3628227184 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1200b47d0 con 0x7fe1200cbc30 2023-05-22T15:43:15.815 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.803+0000 7fe1264a6640 1 -- 172.21.15.16:0/3628227184 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe12004dbe0 con 0x7fe1200cbc30 2023-05-22T15:43:15.815 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.803+0000 7fe11cff9640 1 -- 172.21.15.16:0/3628227184 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 39) v1 ==== 83941+0+0 (secure 0 0 0) 0x7fe11800fd20 con 0x7fe1200cbc30 2023-05-22T15:43:15.815 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.803+0000 7fe11cff9640 1 -- 172.21.15.16:0/3628227184 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 7027+0+0 (secure 0 0 0) 0x7fe1180829e0 con 0x7fe1200cbc30 2023-05-22T15:43:15.815 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.807+0000 7fe11cff9640 1 -- 172.21.15.16:0/3628227184 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fe118050720 con 0x7fe1200cbc30 2023-05-22T15:43:15.982 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.975+0000 7fe11cff9640 1 -- 172.21.15.16:0/3628227184 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mgrmap(e 40) v1 ==== 84068+0+0 (secure 0 0 0) 0x7fe1180501d0 con 0x7fe1200cbc30 2023-05-22T15:43:15.983 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.979+0000 7fe11cff9640 1 --2- 172.21.15.16:0/3628227184 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fe10c06b6b0 0x7fe10c06da90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:15.983 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.979+0000 7fe11ffff640 1 --2- 172.21.15.16:0/3628227184 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fe10c06b6b0 0x7fe10c06da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:15.989 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:15.983+0000 7fe11ffff640 1 --2- 172.21.15.16:0/3628227184 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fe10c06b6b0 0x7fe10c06da90 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fe1140ab000 tx=0x7fe1140a7070 comp rx=0 tx=0).ready entity=mgr.7788 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:43:16.007 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.003+0000 7fe1264a6640 1 -- 172.21.15.16:0/3628227184 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe1200ca590 con 0x7fe1200cbc30 2023-05-22T15:43:16.007 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.003+0000 7fe11cff9640 1 -- 172.21.15.16:0/3628227184 <== mon.0 v2:172.21.15.16:3300/0 8 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) v1 ==== 94+0+4650 (secure 0 0 0) 0x7fe11805e0a0 con 0x7fe1200cbc30 2023-05-22T15:43:16.008 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:43:16.009 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":36,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6059,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6833/3017549713","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":3017549713},{"type":"v1","addr":"172.21.15.62:6833","nonce":3017549713}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":33},{"gid":6607,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/3268651705","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":3268651705},{"type":"v1","addr":"172.21.15.62:6835","nonce":3268651705}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":33},{"gid":7524,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6836/2104991509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2104991509},{"type":"v1","addr":"172.21.15.16:6836","nonce":2104991509}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":33}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-05-22T15:42:56.565074+0000","modified":"2023-05-22T15:42:58.580533+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7527},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7527":{"gid":7527,"name":"a","rank":0,"incarnation":34,"state":"up:active","state_seq":3,"addr":"172.21.15.16:6837/1353635569","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6835","nonce":1353635569},{"type":"v1","addr":"172.21.15.16:6837","nonce":1353635569}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2023-05-22T15:43:16.009 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 36 2023-05-22T15:43:16.010 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.003+0000 7fe1264a6640 1 -- 172.21.15.16:0/3628227184 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fe10c06b6b0 msgr2=0x7fe10c06da90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:16.010 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.003+0000 7fe1264a6640 1 --2- 172.21.15.16:0/3628227184 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fe10c06b6b0 0x7fe10c06da90 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fe1140ab000 tx=0x7fe1140a7070 comp rx=0 tx=0).stop 2023-05-22T15:43:16.010 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.003+0000 7fe1264a6640 1 -- 172.21.15.16:0/3628227184 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe1200cbc30 msgr2=0x7fe1200cc0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:16.010 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.003+0000 7fe1264a6640 1 --2- 172.21.15.16:0/3628227184 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe1200cbc30 0x7fe1200cc0a0 secure :-1 s=READY pgs=1382 cs=0 l=1 rev1=1 crypto rx=0x7fe118009fd0 tx=0x7fe11800f0d0 comp rx=0 tx=0).stop 2023-05-22T15:43:16.010 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.003+0000 7fe1264a6640 1 -- 172.21.15.16:0/3628227184 shutdown_connections 2023-05-22T15:43:16.011 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.003+0000 7fe1264a6640 1 --2- 172.21.15.16:0/3628227184 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe1200cbc30 0x7fe1200cc0a0 unknown :-1 s=CLOSED pgs=1382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:16.011 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.003+0000 7fe1264a6640 1 --2- 172.21.15.16:0/3628227184 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fe10c06b6b0 0x7fe10c06da90 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:16.011 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.003+0000 7fe1264a6640 1 --2- 172.21.15.16:0/3628227184 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe1200c9340 0x7fe120150330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:16.011 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.003+0000 7fe1264a6640 1 --2- 172.21.15.16:0/3628227184 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe108003980 0x7fe120161fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:16.011 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.003+0000 7fe1264a6640 1 -- 172.21.15.16:0/3628227184 >> 172.21.15.16:0/3628227184 conn(0x7fe12006f560 msgr2=0x7fe120154650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:43:16.011 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.003+0000 7fe1264a6640 1 -- 172.21.15.16:0/3628227184 shutdown_connections 2023-05-22T15:43:16.011 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.003+0000 7fe1264a6640 1 -- 172.21.15.16:0/3628227184 wait complete. 2023-05-22T15:43:16.026 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-05-22T15:43:16.142 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.139+0000 7f95c08ba640 1 Processor -- start 2023-05-22T15:43:16.143 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.139+0000 7f95c08ba640 1 -- start start 2023-05-22T15:43:16.144 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.139+0000 7f95c08ba640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f95bc1531d0 0x7f95bc0cd730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:16.144 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.139+0000 7f95c08ba640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95bc15c6b0 con 0x7f95bc0cbbf0 2023-05-22T15:43:16.144 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.139+0000 7f95c08ba640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95bc15c990 con 0x7f95bc0b0250 2023-05-22T15:43:16.144 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.139+0000 7f95c08ba640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95bc15d330 con 0x7f95bc1531d0 2023-05-22T15:43:16.144 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.139+0000 7f95ba59c640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f95bc1531d0 0x7f95bc0cd730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:16.144 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.139+0000 7f95ba59c640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f95bc1531d0 0x7f95bc0cd730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45936/0 (socket says 172.21.15.16:45936) 2023-05-22T15:43:16.144 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.139+0000 7f95ba59c640 1 -- 172.21.15.16:0/3264592033 learned_addr learned my addr 172.21.15.16:0/3264592033 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:43:16.145 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.139+0000 7f95ba59c640 1 -- 172.21.15.16:0/3264592033 >> v1:172.21.15.62:6789/0 conn(0x7f95bc0b0250 legacy=0x7f95bc0acba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:43:16.145 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.139+0000 7f95ba59c640 1 -- 172.21.15.16:0/3264592033 >> v1:172.21.15.16:6789/0 conn(0x7f95bc0cbbf0 legacy=0x7f95bc0afb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:16.145 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.139+0000 7f95ba59c640 1 -- 172.21.15.16:0/3264592033 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95bc15cc70 con 0x7f95bc1531d0 2023-05-22T15:43:16.146 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.143+0000 7f95ba59c640 1 --2- 172.21.15.16:0/3264592033 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f95bc1531d0 0x7f95bc0cd730 secure :-1 s=READY pgs=1384 cs=0 l=1 rev1=1 crypto rx=0x7f95ac009870 tx=0x7f95ac01c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f95f9fa20473d07 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:43:16.146 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.143+0000 7f95b9d9b640 1 -- 172.21.15.16:0/3264592033 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95ac0ac640 con 0x7f95bc1531d0 2023-05-22T15:43:16.146 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.143+0000 7f95b9d9b640 1 -- 172.21.15.16:0/3264592033 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95ac0ac7a0 con 0x7f95bc1531d0 2023-05-22T15:43:16.147 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.143+0000 7f95b9d9b640 1 -- 172.21.15.16:0/3264592033 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95ac0acb30 con 0x7f95bc1531d0 2023-05-22T15:43:16.147 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.143+0000 7f95c08ba640 1 -- 172.21.15.16:0/3264592033 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f95bc1531d0 msgr2=0x7f95bc0cd730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:16.147 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.143+0000 7f95c08ba640 1 --2- 172.21.15.16:0/3264592033 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f95bc1531d0 0x7f95bc0cd730 secure :-1 s=READY pgs=1384 cs=0 l=1 rev1=1 crypto rx=0x7f95ac009870 tx=0x7f95ac01c500 comp rx=0 tx=0).stop 2023-05-22T15:43:16.148 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.143+0000 7f95c08ba640 1 -- 172.21.15.16:0/3264592033 shutdown_connections 2023-05-22T15:43:16.148 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.143+0000 7f95c08ba640 1 --2- 172.21.15.16:0/3264592033 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f95bc1531d0 0x7f95bc0cd730 unknown :-1 s=CLOSED pgs=1384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:16.148 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.143+0000 7f95c08ba640 1 -- 172.21.15.16:0/3264592033 >> 172.21.15.16:0/3264592033 conn(0x7f95bc069720 msgr2=0x7f95bc069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:43:16.148 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.143+0000 7f95c08ba640 1 -- 172.21.15.16:0/3264592033 shutdown_connections 2023-05-22T15:43:16.148 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.143+0000 7f95c08ba640 1 -- 172.21.15.16:0/3264592033 wait complete. 2023-05-22T15:43:16.151 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.143+0000 7f95c08ba640 1 Processor -- start 2023-05-22T15:43:16.151 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.143+0000 7f95c08ba640 1 -- start start 2023-05-22T15:43:16.151 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.143+0000 7f95c08ba640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f95bc0b0250 0x7f95bc15db90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:16.151 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.143+0000 7f95c08ba640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f95bc0cbbf0 0x7f95bc14a770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:16.151 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.143+0000 7f95c08ba640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f95bc1531d0 0x7f95bc14acb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:16.152 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.143+0000 7f95c08ba640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95bc14dfb0 con 0x7f95bc1531d0 2023-05-22T15:43:16.152 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.143+0000 7f95c08ba640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95bc147310 con 0x7f95bc0b0250 2023-05-22T15:43:16.152 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.143+0000 7f95c08ba640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95bc14b660 con 0x7f95bc0cbbf0 2023-05-22T15:43:16.152 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.143+0000 7f95bad9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f95bc1531d0 0x7f95bc14acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:16.152 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.143+0000 7f95bad9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f95bc1531d0 0x7f95bc14acb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45952/0 (socket says 172.21.15.16:45952) 2023-05-22T15:43:16.152 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.143+0000 7f95bad9d640 1 -- 172.21.15.16:0/1424828737 learned_addr learned my addr 172.21.15.16:0/1424828737 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:43:16.152 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.143+0000 7f95ba59c640 1 --2- 172.21.15.16:0/1424828737 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f95bc0b0250 0x7f95bc15db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:16.153 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.143+0000 7f95b3fff640 1 --2- 172.21.15.16:0/1424828737 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f95bc0cbbf0 0x7f95bc14a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:16.153 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.143+0000 7f95bad9d640 1 -- 172.21.15.16:0/1424828737 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f95bc0cbbf0 msgr2=0x7f95bc14a770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:16.153 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.143+0000 7f95bad9d640 1 --2- 172.21.15.16:0/1424828737 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f95bc0cbbf0 0x7f95bc14a770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:16.153 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.143+0000 7f95bad9d640 1 -- 172.21.15.16:0/1424828737 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f95bc0b0250 msgr2=0x7f95bc15db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:16.153 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.143+0000 7f95bad9d640 1 --2- 172.21.15.16:0/1424828737 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f95bc0b0250 0x7f95bc15db90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:16.153 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.143+0000 7f95bad9d640 1 -- 172.21.15.16:0/1424828737 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95ac0094d0 con 0x7f95bc1531d0 2023-05-22T15:43:16.153 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.143+0000 7f95b3fff640 1 --2- 172.21.15.16:0/1424828737 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f95bc0cbbf0 0x7f95bc14a770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:43:16.154 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.143+0000 7f95bad9d640 1 --2- 172.21.15.16:0/1424828737 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f95bc1531d0 0x7f95bc14acb0 secure :-1 s=READY pgs=1385 cs=0 l=1 rev1=1 crypto rx=0x7f95b400c780 tx=0x7f95b400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:43:16.154 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.147+0000 7f95b2ffd640 1 -- 172.21.15.16:0/1424828737 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95b400a9f0 con 0x7f95bc1531d0 2023-05-22T15:43:16.154 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.147+0000 7f95b2ffd640 1 -- 172.21.15.16:0/1424828737 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95b400ce90 con 0x7f95bc1531d0 2023-05-22T15:43:16.154 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.147+0000 7f95c08ba640 1 -- 172.21.15.16:0/1424828737 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95bc169e40 con 0x7f95bc1531d0 2023-05-22T15:43:16.154 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.147+0000 7f95b2ffd640 1 -- 172.21.15.16:0/1424828737 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95b400add0 con 0x7f95bc1531d0 2023-05-22T15:43:16.154 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.147+0000 7f95c08ba640 1 -- 172.21.15.16:0/1424828737 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95bc160950 con 0x7f95bc1531d0 2023-05-22T15:43:16.154 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.147+0000 7f95c08ba640 1 -- 172.21.15.16:0/1424828737 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95bc169fd0 con 0x7f95bc1531d0 2023-05-22T15:43:16.155 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.147+0000 7f95b2ffd640 1 -- 172.21.15.16:0/1424828737 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 40) v1 ==== 84068+0+0 (secure 0 0 0) 0x7f95b4012420 con 0x7f95bc1531d0 2023-05-22T15:43:16.155 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.147+0000 7f95b2ffd640 1 --2- 172.21.15.16:0/1424828737 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f959c0651d0 0x7f959c067680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:16.155 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.147+0000 7f95ba59c640 1 --2- 172.21.15.16:0/1424828737 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f959c0651d0 0x7f959c067680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:16.155 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.147+0000 7f95b2ffd640 1 -- 172.21.15.16:0/1424828737 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 7027+0+0 (secure 0 0 0) 0x7f95b4082a20 con 0x7f95bc1531d0 2023-05-22T15:43:16.156 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.147+0000 7f95ba59c640 1 --2- 172.21.15.16:0/1424828737 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f959c0651d0 0x7f959c067680 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f95ac0ae000 tx=0x7f95ac0a7000 comp rx=0 tx=0).ready entity=mgr.7788 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:43:16.156 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.151+0000 7f95b2ffd640 1 -- 172.21.15.16:0/1424828737 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f95b40505b0 con 0x7f95bc1531d0 2023-05-22T15:43:16.306 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.303+0000 7f95c08ba640 1 -- 172.21.15.16:0/1424828737 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f95bc0cb770 con 0x7f95bc1531d0 2023-05-22T15:43:16.309 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.303+0000 7f95b2ffd640 1 -- 172.21.15.16:0/1424828737 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v106) v1 ==== 74+0+18962 (secure 0 0 0) 0x7f95b4016090 con 0x7f95bc1531d0 2023-05-22T15:43:16.310 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:43:16.310 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":106,"fsid":"b2719834-7847-4d98-90a6-f65d57f85c49","created":"2023-05-22T15:26:25.259314+0000","modified":"2023-05-22T15:43:14.959874+0000","last_up_change":"2023-05-22T15:26:36.448524+0000","last_in_change":"2023-05-22T15:26:27.675668+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-05-22T15:26:38.019781+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-05-22T15:26:43.570514+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2023-05-22T15:42:55.131159+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"103","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2023-05-22T15:42:55.567899+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"104","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9e7ff854-b25f-4cbb-a5c2-1799d9c53d2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":102,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":37021},{"type":"v1","addr":"172.21.15.16:6825","nonce":37021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":37021},{"type":"v1","addr":"172.21.15.16:6827","nonce":37021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":37021},{"type":"v1","addr":"172.21.15.16:6831","nonce":37021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":37021},{"type":"v1","addr":"172.21.15.16:6829","nonce":37021}]},"public_addr":"172.21.15.16:6825/37021","cluster_addr":"172.21.15.16:6827/37021","heartbeat_back_addr":"172.21.15.16:6831/37021","heartbeat_front_addr":"172.21.15.16:6829/37021","state":["exists","up"]},{"osd":1,"uuid":"4702fbc9-a927-4565-aca5-41dc69cc8264","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":102,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":37008},{"type":"v1","addr":"172.21.15.16:6801","nonce":37008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":37008},{"type":"v1","addr":"172.21.15.16:6803","nonce":37008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":37008},{"type":"v1","addr":"172.21.15.16:6807","nonce":37008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":37008},{"type":"v1","addr":"172.21.15.16:6805","nonce":37008}]},"public_addr":"172.21.15.16:6801/37008","cluster_addr":"172.21.15.16:6803/37008","heartbeat_back_addr":"172.21.15.16:6807/37008","heartbeat_front_addr":"172.21.15.16:6805/37008","state":["exists","up"]},{"osd":2,"uuid":"1e7bba46-4eba-47ef-bb81-5e2d371638ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":102,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":37009},{"type":"v1","addr":"172.21.15.16:6809","nonce":37009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":37009},{"type":"v1","addr":"172.21.15.16:6811","nonce":37009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":37009},{"type":"v1","addr":"172.21.15.16:6815","nonce":37009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":37009},{"type":"v1","addr":"172.21.15.16:6813","nonce":37009}]},"public_addr":"172.21.15.16:6809/37009","cluster_addr":"172.21.15.16:6811/37009","heartbeat_back_addr":"172.21.15.16:6815/37009","heartbeat_front_addr":"172.21.15.16:6813/37009","state":["exists","up"]},{"osd":3,"uuid":"383cacc0-5a88-4904-990e-498ca59ef48a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":102,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":37020},{"type":"v1","addr":"172.21.15.16:6817","nonce":37020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":37020},{"type":"v1","addr":"172.21.15.16:6819","nonce":37020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":37020},{"type":"v1","addr":"172.21.15.16:6823","nonce":37020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":37020},{"type":"v1","addr":"172.21.15.16:6821","nonce":37020}]},"public_addr":"172.21.15.16:6817/37020","cluster_addr":"172.21.15.16:6819/37020","heartbeat_back_addr":"172.21.15.16:6823/37020","heartbeat_front_addr":"172.21.15.16:6821/37020","state":["exists","up"]},{"osd":4,"uuid":"970500ef-95b7-42c1-b080-eed4fae7013d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":102,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":37353},{"type":"v1","addr":"172.21.15.62:6817","nonce":37353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6819","nonce":37353},{"type":"v1","addr":"172.21.15.62:6822","nonce":37353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37353},{"type":"v1","addr":"172.21.15.62:6831","nonce":37353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37353},{"type":"v1","addr":"172.21.15.62:6826","nonce":37353}]},"public_addr":"172.21.15.62:6817/37353","cluster_addr":"172.21.15.62:6822/37353","heartbeat_back_addr":"172.21.15.62:6831/37353","heartbeat_front_addr":"172.21.15.62:6826/37353","state":["exists","up"]},{"osd":5,"uuid":"022d67db-d4d8-445b-b1b9-8b446a00f5c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":102,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":37352},{"type":"v1","addr":"172.21.15.62:6801","nonce":37352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":37352},{"type":"v1","addr":"172.21.15.62:6803","nonce":37352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":37352},{"type":"v1","addr":"172.21.15.62:6807","nonce":37352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":37352},{"type":"v1","addr":"172.21.15.62:6805","nonce":37352}]},"public_addr":"172.21.15.62:6801/37352","cluster_addr":"172.21.15.62:6803/37352","heartbeat_back_addr":"172.21.15.62:6807/37352","heartbeat_front_addr":"172.21.15.62:6805/37352","state":["exists","up"]},{"osd":6,"uuid":"42bd372b-6688-4a1b-8407-e16e8c266e33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":102,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":37354},{"type":"v1","addr":"172.21.15.62:6820","nonce":37354}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":37354},{"type":"v1","addr":"172.21.15.62:6823","nonce":37354}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6829","nonce":37354},{"type":"v1","addr":"172.21.15.62:6830","nonce":37354}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6825","nonce":37354},{"type":"v1","addr":"172.21.15.62:6827","nonce":37354}]},"public_addr":"172.21.15.62:6820/37354","cluster_addr":"172.21.15.62:6823/37354","heartbeat_back_addr":"172.21.15.62:6830/37354","heartbeat_front_addr":"172.21.15.62:6827/37354","state":["exists","up"]},{"osd":7,"uuid":"ac70b2a3-f41b-4bf2-b227-7f2cbb55426d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":102,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37355},{"type":"v1","addr":"172.21.15.62:6809","nonce":37355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37355},{"type":"v1","addr":"172.21.15.62:6811","nonce":37355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":37355},{"type":"v1","addr":"172.21.15.62:6815","nonce":37355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":37355},{"type":"v1","addr":"172.21.15.62:6813","nonce":37355}]},"public_addr":"172.21.15.62:6809/37355","cluster_addr":"172.21.15.62:6811/37355","heartbeat_back_addr":"172.21.15.62:6815/37355","heartbeat_front_addr":"172.21.15.62:6813/37355","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.368896+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.107273+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.276888+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.287502+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.124791+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.251937+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.255679+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.232797+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:0/2165805468":"2023-05-23T15:43:14.959816+0000","172.21.15.16:0/2196574811":"2023-05-23T15:43:14.959816+0000","172.21.15.16:0/747277139":"2023-05-23T15:43:14.959816+0000","172.21.15.16:0/291514430":"2023-05-23T15:43:14.959816+0000","172.21.15.16:0/3869784079":"2023-05-23T15:43:14.959816+0000","172.21.15.16:0/638271246":"2023-05-23T15:43:14.959816+0000","172.21.15.16:0/1315052983":"2023-05-23T15:43:09.063704+0000","172.21.15.16:0/3981695108":"2023-05-23T15:43:09.063704+0000","172.21.15.16:0/92534915":"2023-05-23T15:43:09.063704+0000","172.21.15.16:6833/36921":"2023-05-23T15:43:09.063704+0000","172.21.15.16:6832/36921":"2023-05-23T15:43:09.063704+0000","172.21.15.16:0/2880991909":"2023-05-23T15:43:14.959816+0000","172.21.15.16:0/2741778632":"2023-05-23T15:43:09.063704+0000","172.21.15.16:0/3940121619":"2023-05-23T15:43:09.063704+0000","172.21.15.16:0/2073223440":"2023-05-23T15:43:09.063704+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-05-22T15:43:16.312 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.307+0000 7f95c08ba640 1 -- 172.21.15.16:0/1424828737 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f959c0651d0 msgr2=0x7f959c067680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:16.312 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.307+0000 7f95c08ba640 1 --2- 172.21.15.16:0/1424828737 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f959c0651d0 0x7f959c067680 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f95ac0ae000 tx=0x7f95ac0a7000 comp rx=0 tx=0).stop 2023-05-22T15:43:16.312 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.307+0000 7f95c08ba640 1 -- 172.21.15.16:0/1424828737 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f95bc1531d0 msgr2=0x7f95bc14acb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:16.312 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.307+0000 7f95c08ba640 1 --2- 172.21.15.16:0/1424828737 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f95bc1531d0 0x7f95bc14acb0 secure :-1 s=READY pgs=1385 cs=0 l=1 rev1=1 crypto rx=0x7f95b400c780 tx=0x7f95b400cc40 comp rx=0 tx=0).stop 2023-05-22T15:43:16.312 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.307+0000 7f95c08ba640 1 -- 172.21.15.16:0/1424828737 shutdown_connections 2023-05-22T15:43:16.313 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.307+0000 7f95c08ba640 1 --2- 172.21.15.16:0/1424828737 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f95bc1531d0 0x7f95bc14acb0 unknown :-1 s=CLOSED pgs=1385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:16.313 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.307+0000 7f95c08ba640 1 --2- 172.21.15.16:0/1424828737 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f959c0651d0 0x7f959c067680 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:16.313 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.307+0000 7f95c08ba640 1 --2- 172.21.15.16:0/1424828737 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f95bc0cbbf0 0x7f95bc14a770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:16.313 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.307+0000 7f95c08ba640 1 --2- 172.21.15.16:0/1424828737 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f95bc0b0250 0x7f95bc15db90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:16.313 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.307+0000 7f95c08ba640 1 -- 172.21.15.16:0/1424828737 >> 172.21.15.16:0/1424828737 conn(0x7f95bc069720 msgr2=0x7f95bc069af0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:43:16.313 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.307+0000 7f95c08ba640 1 -- 172.21.15.16:0/1424828737 shutdown_connections 2023-05-22T15:43:16.314 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.307+0000 7f95c08ba640 1 -- 172.21.15.16:0/1424828737 wait complete. 2023-05-22T15:43:16.327 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-05-22T15:43:16.328 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-05-22T15:43:16.443 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.439+0000 7fa385ed6640 1 Processor -- start 2023-05-22T15:43:16.443 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.439+0000 7fa385ed6640 1 -- start start 2023-05-22T15:43:16.444 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.439+0000 7fa385ed6640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fa3801531b0 0x7fa3800afb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:16.444 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.439+0000 7fa385ed6640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa38015a0a0 con 0x7fa380159a40 2023-05-22T15:43:16.444 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.439+0000 7fa385ed6640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa380159e50 con 0x7fa3800b0080 2023-05-22T15:43:16.444 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.439+0000 7fa385ed6640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3800b0520 con 0x7fa3801531b0 2023-05-22T15:43:16.444 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.439+0000 7fa37ffff640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fa380159a40 0x7fa3800c8ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:57140/0 (socket says 172.21.15.16:57140) 2023-05-22T15:43:16.445 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.439+0000 7fa37ffff640 1 -- 172.21.15.16:0/3316648968 learned_addr learned my addr 172.21.15.16:0/3316648968 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:43:16.446 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.439+0000 7fa37f7fe640 1 --2- 172.21.15.16:0/3316648968 >> v2:172.21.15.62:3300/0 conn(0x7fa3801531b0 0x7fa3800afb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:16.446 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.439+0000 7fa37e7fc640 1 -- 172.21.15.16:0/3316648968 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2172706979 0 0) 0x7fa38015a0a0 con 0x7fa380159a40 2023-05-22T15:43:16.447 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.439+0000 7fa37e7fc640 1 -- 172.21.15.16:0/3316648968 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa368003530 con 0x7fa380159a40 2023-05-22T15:43:16.447 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.439+0000 7fa37e7fc640 1 -- 172.21.15.16:0/3316648968 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1497183682 0 0) 0x7fa368003530 con 0x7fa380159a40 2023-05-22T15:43:16.447 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.439+0000 7fa37e7fc640 1 -- 172.21.15.16:0/3316648968 >> v2:172.21.15.62:3300/0 conn(0x7fa3801531b0 msgr2=0x7fa3800afb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:43:16.447 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.439+0000 7fa37e7fc640 1 --2- 172.21.15.16:0/3316648968 >> v2:172.21.15.62:3300/0 conn(0x7fa3801531b0 0x7fa3800afb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:16.447 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.439+0000 7fa37e7fc640 1 -- 172.21.15.16:0/3316648968 >> v1:172.21.15.62:6789/0 conn(0x7fa3800b0080 legacy=0x7fa380159330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:16.447 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.439+0000 7fa37e7fc640 1 -- 172.21.15.16:0/3316648968 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa380153580 con 0x7fa380159a40 2023-05-22T15:43:16.447 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.439+0000 7fa37e7fc640 1 -- 172.21.15.16:0/3316648968 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7fa378001a00 con 0x7fa380159a40 2023-05-22T15:43:16.448 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.439+0000 7fa37e7fc640 1 -- 172.21.15.16:0/3316648968 >> v1:172.21.15.16:6789/0 conn(0x7fa380159a40 legacy=0x7fa3800c8ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:16.448 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.439+0000 7fa37e7fc640 1 --2- 172.21.15.16:0/3316648968 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa3680039d0 0x7fa368003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:16.448 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.439+0000 7fa37e7fc640 1 --2- 172.21.15.16:0/3316648968 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa3680047d0 0x7fa368004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:16.448 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.439+0000 7fa37e7fc640 1 --2- 172.21.15.16:0/3316648968 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa368005180 0x7fa36800b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:16.448 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.439+0000 7fa37e7fc640 1 -- 172.21.15.16:0/3316648968 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa36800c060 con 0x7fa3680039d0 2023-05-22T15:43:16.448 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.439+0000 7fa37e7fc640 1 -- 172.21.15.16:0/3316648968 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa36800c1a0 con 0x7fa3680047d0 2023-05-22T15:43:16.448 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.439+0000 7fa37e7fc640 1 -- 172.21.15.16:0/3316648968 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa36800e570 con 0x7fa368005180 2023-05-22T15:43:16.449 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.439+0000 7fa37f7fe640 1 --2- 172.21.15.16:0/3316648968 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa3680039d0 0x7fa368003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:16.449 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.439+0000 7fa37effd640 1 --2- 172.21.15.16:0/3316648968 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa3680047d0 0x7fa368004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:16.449 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.439+0000 7fa37ffff640 1 --2- 172.21.15.16:0/3316648968 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa368005180 0x7fa36800b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:16.449 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.439+0000 7fa37f7fe640 1 -- 172.21.15.16:0/3316648968 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa368005180 msgr2=0x7fa36800b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:43:16.449 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.439+0000 7fa37f7fe640 1 --2- 172.21.15.16:0/3316648968 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa368005180 0x7fa36800b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:16.449 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.439+0000 7fa37f7fe640 1 -- 172.21.15.16:0/3316648968 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa3680047d0 msgr2=0x7fa368004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:43:16.449 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.439+0000 7fa37f7fe640 1 --2- 172.21.15.16:0/3316648968 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa3680047d0 0x7fa368004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:16.450 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.439+0000 7fa37f7fe640 1 -- 172.21.15.16:0/3316648968 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa368003530 con 0x7fa3680039d0 2023-05-22T15:43:16.450 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.439+0000 7fa37ffff640 1 --2- 172.21.15.16:0/3316648968 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa368005180 0x7fa36800b990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:43:16.450 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.439+0000 7fa37ffff640 1 -- 172.21.15.16:0/3316648968 reap_dead start 2023-05-22T15:43:16.452 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.443+0000 7fa37f7fe640 1 --2- 172.21.15.16:0/3316648968 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa3680039d0 0x7fa368003de0 secure :-1 s=READY pgs=1387 cs=0 l=1 rev1=1 crypto rx=0x7fa374009ab0 tx=0x7fa37401ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e2f894c6d023301 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:43:16.452 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.443+0000 7fa37e7fc640 1 -- 172.21.15.16:0/3316648968 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3740a8450 con 0x7fa3680039d0 2023-05-22T15:43:16.453 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.443+0000 7fa37e7fc640 1 -- 172.21.15.16:0/3316648968 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3740a85b0 con 0x7fa3680039d0 2023-05-22T15:43:16.453 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.443+0000 7fa37e7fc640 1 -- 172.21.15.16:0/3316648968 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3740a8960 con 0x7fa3680039d0 2023-05-22T15:43:16.453 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.443+0000 7fa385ed6640 1 -- 172.21.15.16:0/3316648968 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa3680039d0 msgr2=0x7fa368003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:16.453 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.443+0000 7fa385ed6640 1 --2- 172.21.15.16:0/3316648968 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa3680039d0 0x7fa368003de0 secure :-1 s=READY pgs=1387 cs=0 l=1 rev1=1 crypto rx=0x7fa374009ab0 tx=0x7fa37401ca10 comp rx=0 tx=0).stop 2023-05-22T15:43:16.453 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.443+0000 7fa385ed6640 1 -- 172.21.15.16:0/3316648968 shutdown_connections 2023-05-22T15:43:16.453 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.443+0000 7fa385ed6640 1 --2- 172.21.15.16:0/3316648968 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa3680039d0 0x7fa368003de0 unknown :-1 s=CLOSED pgs=1387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:16.453 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.443+0000 7fa385ed6640 1 -- 172.21.15.16:0/3316648968 >> 172.21.15.16:0/3316648968 conn(0x7fa380069720 msgr2=0x7fa380069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:43:16.454 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.443+0000 7fa385ed6640 1 -- 172.21.15.16:0/3316648968 shutdown_connections 2023-05-22T15:43:16.454 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.443+0000 7fa385ed6640 1 -- 172.21.15.16:0/3316648968 wait complete. 2023-05-22T15:43:16.454 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.443+0000 7fa385ed6640 1 Processor -- start 2023-05-22T15:43:16.454 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.443+0000 7fa385ed6640 1 -- start start 2023-05-22T15:43:16.454 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.447+0000 7fa385ed6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa3680039d0 0x7fa3800c1020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:16.454 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.447+0000 7fa385ed6640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa38014ff60 0x7fa380150870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:16.454 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.447+0000 7fa385ed6640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa380150db0 0x7fa38014e100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:16.455 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.447+0000 7fa385ed6640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa380151230 con 0x7fa3680039d0 2023-05-22T15:43:16.455 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.447+0000 7fa385ed6640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa38014eec0 con 0x7fa38014ff60 2023-05-22T15:43:16.455 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.447+0000 7fa385ed6640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa38014f1a0 con 0x7fa380150db0 2023-05-22T15:43:16.455 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.447+0000 7fa37f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa3680039d0 0x7fa3800c1020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:16.455 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.447+0000 7fa37f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa3680039d0 0x7fa3800c1020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45982/0 (socket says 172.21.15.16:45982) 2023-05-22T15:43:16.455 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.447+0000 7fa37f7fe640 1 -- 172.21.15.16:0/1635965734 learned_addr learned my addr 172.21.15.16:0/1635965734 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:43:16.455 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.447+0000 7fa37effd640 1 --2- 172.21.15.16:0/1635965734 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa38014ff60 0x7fa380150870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:16.455 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.447+0000 7fa37ffff640 1 --2- 172.21.15.16:0/1635965734 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa380150db0 0x7fa38014e100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:16.456 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.447+0000 7fa37f7fe640 1 -- 172.21.15.16:0/1635965734 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa380150db0 msgr2=0x7fa38014e100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:16.456 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.447+0000 7fa37f7fe640 1 --2- 172.21.15.16:0/1635965734 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa380150db0 0x7fa38014e100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:16.456 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.447+0000 7fa37f7fe640 1 -- 172.21.15.16:0/1635965734 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa38014ff60 msgr2=0x7fa380150870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:16.456 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.447+0000 7fa37f7fe640 1 --2- 172.21.15.16:0/1635965734 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa38014ff60 0x7fa380150870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:16.456 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.447+0000 7fa37f7fe640 1 -- 172.21.15.16:0/1635965734 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa38015d0c0 con 0x7fa3680039d0 2023-05-22T15:43:16.456 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.447+0000 7fa37f7fe640 1 --2- 172.21.15.16:0/1635965734 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa3680039d0 0x7fa3800c1020 secure :-1 s=READY pgs=1388 cs=0 l=1 rev1=1 crypto rx=0x7fa37401c840 tx=0x7fa3740197b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:43:16.456 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.447+0000 7fa367fff640 1 -- 172.21.15.16:0/1635965734 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3740a8e20 con 0x7fa3680039d0 2023-05-22T15:43:16.456 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.447+0000 7fa367fff640 1 -- 172.21.15.16:0/1635965734 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa37400f000 con 0x7fa3680039d0 2023-05-22T15:43:16.457 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.447+0000 7fa385ed6640 1 -- 172.21.15.16:0/1635965734 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa38015cde0 con 0x7fa3680039d0 2023-05-22T15:43:16.457 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.447+0000 7fa367fff640 1 -- 172.21.15.16:0/1635965734 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3740b9650 con 0x7fa3680039d0 2023-05-22T15:43:16.457 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.447+0000 7fa385ed6640 1 -- 172.21.15.16:0/1635965734 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa380159740 con 0x7fa3680039d0 2023-05-22T15:43:16.457 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.447+0000 7fa385ed6640 1 -- 172.21.15.16:0/1635965734 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3801511b0 con 0x7fa3680039d0 2023-05-22T15:43:16.458 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.447+0000 7fa367fff640 1 -- 172.21.15.16:0/1635965734 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 40) v1 ==== 84068+0+0 (secure 0 0 0) 0x7fa3740b0400 con 0x7fa3680039d0 2023-05-22T15:43:16.458 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.447+0000 7fa367fff640 1 --2- 172.21.15.16:0/1635965734 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fa36c0651d0 0x7fa36c067680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:16.458 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.447+0000 7fa367fff640 1 -- 172.21.15.16:0/1635965734 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 7027+0+0 (secure 0 0 0) 0x7fa37411c7b0 con 0x7fa3680039d0 2023-05-22T15:43:16.458 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.451+0000 7fa37effd640 1 --2- 172.21.15.16:0/1635965734 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fa36c0651d0 0x7fa36c067680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:16.458 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.451+0000 7fa37effd640 1 --2- 172.21.15.16:0/1635965734 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fa36c0651d0 0x7fa36c067680 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fa3800c1b30 tx=0x7fa370009040 comp rx=0 tx=0).ready entity=mgr.7788 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:43:16.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.451+0000 7fa367fff640 1 -- 172.21.15.16:0/1635965734 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fa3740ea400 con 0x7fa3680039d0 2023-05-22T15:43:16.642 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.639+0000 7fa385ed6640 1 -- 172.21.15.16:0/1635965734 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa38015e820 con 0x7fa3680039d0 2023-05-22T15:43:16.643 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.639+0000 7fa367fff640 1 -- 172.21.15.16:0/1635965734 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) v1 ==== 94+0+4650 (secure 0 0 0) 0x7fa3740eda20 con 0x7fa3680039d0 2023-05-22T15:43:16.644 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:43:16.644 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":36,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6059,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6833/3017549713","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":3017549713},{"type":"v1","addr":"172.21.15.62:6833","nonce":3017549713}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":33},{"gid":6607,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/3268651705","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":3268651705},{"type":"v1","addr":"172.21.15.62:6835","nonce":3268651705}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":33},{"gid":7524,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6836/2104991509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2104991509},{"type":"v1","addr":"172.21.15.16:6836","nonce":2104991509}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":33}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-05-22T15:42:56.565074+0000","modified":"2023-05-22T15:42:58.580533+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7527},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7527":{"gid":7527,"name":"a","rank":0,"incarnation":34,"state":"up:active","state_seq":3,"addr":"172.21.15.16:6837/1353635569","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6835","nonce":1353635569},{"type":"v1","addr":"172.21.15.16:6837","nonce":1353635569}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2023-05-22T15:43:16.645 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 36 2023-05-22T15:43:16.646 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.639+0000 7fa385ed6640 1 -- 172.21.15.16:0/1635965734 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fa36c0651d0 msgr2=0x7fa36c067680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:16.646 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.639+0000 7fa385ed6640 1 --2- 172.21.15.16:0/1635965734 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fa36c0651d0 0x7fa36c067680 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fa3800c1b30 tx=0x7fa370009040 comp rx=0 tx=0).stop 2023-05-22T15:43:16.647 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.639+0000 7fa385ed6640 1 -- 172.21.15.16:0/1635965734 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa3680039d0 msgr2=0x7fa3800c1020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:16.647 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.639+0000 7fa385ed6640 1 --2- 172.21.15.16:0/1635965734 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa3680039d0 0x7fa3800c1020 secure :-1 s=READY pgs=1388 cs=0 l=1 rev1=1 crypto rx=0x7fa37401c840 tx=0x7fa3740197b0 comp rx=0 tx=0).stop 2023-05-22T15:43:16.648 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.639+0000 7fa385ed6640 1 -- 172.21.15.16:0/1635965734 shutdown_connections 2023-05-22T15:43:16.648 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.639+0000 7fa385ed6640 1 --2- 172.21.15.16:0/1635965734 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fa36c0651d0 0x7fa36c067680 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:16.648 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.639+0000 7fa385ed6640 1 --2- 172.21.15.16:0/1635965734 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa380150db0 0x7fa38014e100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:16.648 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.639+0000 7fa385ed6640 1 --2- 172.21.15.16:0/1635965734 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa38014ff60 0x7fa380150870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:16.649 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.639+0000 7fa385ed6640 1 --2- 172.21.15.16:0/1635965734 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa3680039d0 0x7fa3800c1020 unknown :-1 s=CLOSED pgs=1388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:16.649 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.639+0000 7fa385ed6640 1 -- 172.21.15.16:0/1635965734 >> 172.21.15.16:0/1635965734 conn(0x7fa380069720 msgr2=0x7fa38015df80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:43:16.649 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.639+0000 7fa385ed6640 1 -- 172.21.15.16:0/1635965734 shutdown_connections 2023-05-22T15:43:16.650 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.639+0000 7fa385ed6640 1 -- 172.21.15.16:0/1635965734 wait complete. 2023-05-22T15:43:16.663 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-05-22T15:43:16.781 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.775+0000 7f8b44168640 1 Processor -- start 2023-05-22T15:43:16.781 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.775+0000 7f8b44168640 1 -- start start 2023-05-22T15:43:16.782 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.775+0000 7f8b44168640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8b3c15c920 0x7f8b3c0d0030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:16.782 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.775+0000 7f8b44168640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f8b3c15cd70 0x7f8b3c0d0570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:16.782 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.775+0000 7f8b44168640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b3c15c490 con 0x7f8b3c15fc50 2023-05-22T15:43:16.782 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.775+0000 7f8b44168640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b3c0cf7a0 con 0x7f8b3c15cd70 2023-05-22T15:43:16.782 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.775+0000 7f8b44168640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b3c15c140 con 0x7f8b3c15c920 2023-05-22T15:43:16.782 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.775+0000 7f8b41703640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8b3c15c920 0x7f8b3c0d0030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:16.783 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.775+0000 7f8b42705640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f8b3c15cd70 0x7f8b3c0d0570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:16.783 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.775+0000 7f8b41703640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8b3c15c920 0x7f8b3c0d0030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3301/0 says I am v2:172.21.15.16:40920/0 (socket says 172.21.15.16:40920) 2023-05-22T15:43:16.783 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.775+0000 7f8b41703640 1 -- 172.21.15.16:0/2676764379 learned_addr learned my addr 172.21.15.16:0/2676764379 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:43:16.783 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.779+0000 7f8b41703640 1 -- 172.21.15.16:0/2676764379 >> v2:172.21.15.62:3300/0 conn(0x7f8b3c15cd70 msgr2=0x7f8b3c0d0570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:43:16.784 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.779+0000 7f8b41703640 1 --2- 172.21.15.16:0/2676764379 >> v2:172.21.15.62:3300/0 conn(0x7f8b3c15cd70 0x7f8b3c0d0570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:16.784 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.779+0000 7f8b41703640 1 -- 172.21.15.16:0/2676764379 >> v1:172.21.15.62:6789/0 conn(0x7f8b3c15fc50 legacy=0x7f8b3c160020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:16.784 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.779+0000 7f8b41703640 1 -- 172.21.15.16:0/2676764379 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b3c0ccb50 con 0x7f8b3c15c920 2023-05-22T15:43:16.785 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.779+0000 7f8b41703640 1 --2- 172.21.15.16:0/2676764379 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8b3c15c920 0x7f8b3c0d0030 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f8b2c009870 tx=0x7f8b2c01c540 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cb54b5af9f78b449 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:43:16.785 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.779+0000 7f8b40f02640 1 -- 172.21.15.16:0/2676764379 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b2c0ab650 con 0x7f8b3c15c920 2023-05-22T15:43:16.785 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.779+0000 7f8b40f02640 1 -- 172.21.15.16:0/2676764379 <== mon.2 v2:172.21.15.62:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b2c0ab7b0 con 0x7f8b3c15c920 2023-05-22T15:43:16.786 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.779+0000 7f8b40f02640 1 -- 172.21.15.16:0/2676764379 <== mon.2 v2:172.21.15.62:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b2c0abb40 con 0x7f8b3c15c920 2023-05-22T15:43:16.786 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.779+0000 7f8b44168640 1 -- 172.21.15.16:0/2676764379 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8b3c15c920 msgr2=0x7f8b3c0d0030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:16.786 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.779+0000 7f8b44168640 1 --2- 172.21.15.16:0/2676764379 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8b3c15c920 0x7f8b3c0d0030 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f8b2c009870 tx=0x7f8b2c01c540 comp rx=0 tx=0).stop 2023-05-22T15:43:16.786 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.779+0000 7f8b44168640 1 -- 172.21.15.16:0/2676764379 shutdown_connections 2023-05-22T15:43:16.786 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.779+0000 7f8b44168640 1 --2- 172.21.15.16:0/2676764379 >> v2:172.21.15.62:3300/0 conn(0x7f8b3c15cd70 0x7f8b3c0d0570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:16.786 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.779+0000 7f8b44168640 1 --2- 172.21.15.16:0/2676764379 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8b3c15c920 0x7f8b3c0d0030 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:16.786 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.779+0000 7f8b44168640 1 -- 172.21.15.16:0/2676764379 >> 172.21.15.16:0/2676764379 conn(0x7f8b3c0657f0 msgr2=0x7f8b3c070e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:43:16.787 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.779+0000 7f8b44168640 1 -- 172.21.15.16:0/2676764379 shutdown_connections 2023-05-22T15:43:16.787 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.779+0000 7f8b44168640 1 -- 172.21.15.16:0/2676764379 wait complete. 2023-05-22T15:43:16.788 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.783+0000 7f8b44168640 1 Processor -- start 2023-05-22T15:43:16.789 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.783+0000 7f8b44168640 1 -- start start 2023-05-22T15:43:16.789 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.783+0000 7f8b44168640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8b3c15c920 0x7f8b3c148850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:16.789 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.783+0000 7f8b44168640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8b3c15cd70 0x7f8b3c0b45b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:16.789 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.783+0000 7f8b44168640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8b3c15fc50 0x7f8b3c14d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:16.790 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.783+0000 7f8b44168640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b3c14a110 con 0x7f8b3c15cd70 2023-05-22T15:43:16.790 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.783+0000 7f8b44168640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b3c0b2c90 con 0x7f8b3c15c920 2023-05-22T15:43:16.790 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.783+0000 7f8b44168640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b3c0b2f70 con 0x7f8b3c15fc50 2023-05-22T15:43:16.790 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.783+0000 7f8b41703640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8b3c15cd70 0x7f8b3c0b45b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:16.790 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.783+0000 7f8b41703640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8b3c15cd70 0x7f8b3c0b45b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45988/0 (socket says 172.21.15.16:45988) 2023-05-22T15:43:16.790 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.783+0000 7f8b41703640 1 -- 172.21.15.16:0/1165420276 learned_addr learned my addr 172.21.15.16:0/1165420276 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:43:16.790 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.783+0000 7f8b42705640 1 --2- 172.21.15.16:0/1165420276 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8b3c15fc50 0x7f8b3c14d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:16.791 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.783+0000 7f8b41f04640 1 --2- 172.21.15.16:0/1165420276 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8b3c15c920 0x7f8b3c148850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:16.791 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.783+0000 7f8b41703640 1 -- 172.21.15.16:0/1165420276 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8b3c15fc50 msgr2=0x7f8b3c14d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:16.791 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.783+0000 7f8b41703640 1 --2- 172.21.15.16:0/1165420276 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8b3c15fc50 0x7f8b3c14d450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:16.791 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.783+0000 7f8b41703640 1 -- 172.21.15.16:0/1165420276 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8b3c15c920 msgr2=0x7f8b3c148850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:16.791 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.783+0000 7f8b41703640 1 --2- 172.21.15.16:0/1165420276 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8b3c15c920 0x7f8b3c148850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:16.791 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.783+0000 7f8b41703640 1 -- 172.21.15.16:0/1165420276 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b2c0094d0 con 0x7f8b3c15cd70 2023-05-22T15:43:16.791 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.783+0000 7f8b41f04640 1 --2- 172.21.15.16:0/1165420276 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8b3c15c920 0x7f8b3c148850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T15:43:16.792 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.783+0000 7f8b41703640 1 --2- 172.21.15.16:0/1165420276 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8b3c15cd70 0x7f8b3c0b45b0 secure :-1 s=READY pgs=1389 cs=0 l=1 rev1=1 crypto rx=0x7f8b2c0ad040 tx=0x7f8b2c01ad80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:43:16.792 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.783+0000 7f8b327fc640 1 -- 172.21.15.16:0/1165420276 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b2c0b4800 con 0x7f8b3c15cd70 2023-05-22T15:43:16.792 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.783+0000 7f8b327fc640 1 -- 172.21.15.16:0/1165420276 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b2c0b4960 con 0x7f8b3c15cd70 2023-05-22T15:43:16.792 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.783+0000 7f8b44168640 1 -- 172.21.15.16:0/1165420276 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b3c0cf130 con 0x7f8b3c15cd70 2023-05-22T15:43:16.792 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.783+0000 7f8b327fc640 1 -- 172.21.15.16:0/1165420276 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b2c0b4cf0 con 0x7f8b3c15cd70 2023-05-22T15:43:16.792 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.783+0000 7f8b44168640 1 -- 172.21.15.16:0/1165420276 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b3c0cefa0 con 0x7f8b3c15cd70 2023-05-22T15:43:16.793 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.783+0000 7f8b44168640 1 -- 172.21.15.16:0/1165420276 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b3c0cf2c0 con 0x7f8b3c15cd70 2023-05-22T15:43:16.793 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.783+0000 7f8b327fc640 1 -- 172.21.15.16:0/1165420276 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 40) v1 ==== 84068+0+0 (secure 0 0 0) 0x7f8b2c0b3380 con 0x7f8b3c15cd70 2023-05-22T15:43:16.794 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.787+0000 7f8b327fc640 1 --2- 172.21.15.16:0/1165420276 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f8b200651d0 0x7f8b20067680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:16.794 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.787+0000 7f8b327fc640 1 -- 172.21.15.16:0/1165420276 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 7027+0+0 (secure 0 0 0) 0x7f8b2c11d0f0 con 0x7f8b3c15cd70 2023-05-22T15:43:16.794 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.787+0000 7f8b41f04640 1 --2- 172.21.15.16:0/1165420276 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f8b200651d0 0x7f8b20067680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:16.794 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.787+0000 7f8b41f04640 1 --2- 172.21.15.16:0/1165420276 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f8b200651d0 0x7f8b20067680 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f8b38000f70 tx=0x7f8b38009040 comp rx=0 tx=0).ready entity=mgr.7788 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:43:16.794 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.791+0000 7f8b327fc640 1 -- 172.21.15.16:0/1165420276 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f8b2c0eacf0 con 0x7f8b3c15cd70 2023-05-22T15:43:16.983 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.979+0000 7f8b44168640 1 -- 172.21.15.16:0/1165420276 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8b3c0b3140 con 0x7f8b3c15cd70 2023-05-22T15:43:16.984 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.979+0000 7f8b327fc640 1 -- 172.21.15.16:0/1165420276 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) v1 ==== 94+0+4650 (secure 0 0 0) 0x7f8b2c0ee310 con 0x7f8b3c15cd70 2023-05-22T15:43:16.984 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:43:16.984 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":36,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6059,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6833/3017549713","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":3017549713},{"type":"v1","addr":"172.21.15.62:6833","nonce":3017549713}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":33},{"gid":6607,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/3268651705","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":3268651705},{"type":"v1","addr":"172.21.15.62:6835","nonce":3268651705}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":33},{"gid":7524,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6836/2104991509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2104991509},{"type":"v1","addr":"172.21.15.16:6836","nonce":2104991509}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":33}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-05-22T15:42:56.565074+0000","modified":"2023-05-22T15:42:58.580533+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7527},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7527":{"gid":7527,"name":"a","rank":0,"incarnation":34,"state":"up:active","state_seq":3,"addr":"172.21.15.16:6837/1353635569","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6835","nonce":1353635569},{"type":"v1","addr":"172.21.15.16:6837","nonce":1353635569}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2023-05-22T15:43:16.985 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 36 2023-05-22T15:43:16.986 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.983+0000 7f8b44168640 1 -- 172.21.15.16:0/1165420276 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f8b200651d0 msgr2=0x7f8b20067680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:16.987 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.983+0000 7f8b44168640 1 --2- 172.21.15.16:0/1165420276 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f8b200651d0 0x7f8b20067680 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f8b38000f70 tx=0x7f8b38009040 comp rx=0 tx=0).stop 2023-05-22T15:43:16.987 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.983+0000 7f8b44168640 1 -- 172.21.15.16:0/1165420276 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8b3c15cd70 msgr2=0x7f8b3c0b45b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:16.987 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.983+0000 7f8b44168640 1 --2- 172.21.15.16:0/1165420276 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8b3c15cd70 0x7f8b3c0b45b0 secure :-1 s=READY pgs=1389 cs=0 l=1 rev1=1 crypto rx=0x7f8b2c0ad040 tx=0x7f8b2c01ad80 comp rx=0 tx=0).stop 2023-05-22T15:43:16.987 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.983+0000 7f8b44168640 1 -- 172.21.15.16:0/1165420276 shutdown_connections 2023-05-22T15:43:16.987 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.983+0000 7f8b44168640 1 --2- 172.21.15.16:0/1165420276 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f8b200651d0 0x7f8b20067680 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:16.987 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.983+0000 7f8b44168640 1 --2- 172.21.15.16:0/1165420276 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8b3c15fc50 0x7f8b3c14d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:16.987 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.983+0000 7f8b44168640 1 --2- 172.21.15.16:0/1165420276 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8b3c15cd70 0x7f8b3c0b45b0 unknown :-1 s=CLOSED pgs=1389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:16.988 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.983+0000 7f8b44168640 1 --2- 172.21.15.16:0/1165420276 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8b3c15c920 0x7f8b3c148850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:16.988 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.983+0000 7f8b44168640 1 -- 172.21.15.16:0/1165420276 >> 172.21.15.16:0/1165420276 conn(0x7f8b3c0657f0 msgr2=0x7f8b3c0b7be0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:43:16.988 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.983+0000 7f8b44168640 1 -- 172.21.15.16:0/1165420276 shutdown_connections 2023-05-22T15:43:16.988 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:16.983+0000 7f8b44168640 1 -- 172.21.15.16:0/1165420276 wait complete. 2023-05-22T15:43:17.002 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-05-22T15:43:17.121 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:17.115+0000 7fb157c5b640 1 Processor -- start 2023-05-22T15:43:17.121 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:17.115+0000 7fb157c5b640 1 -- start start 2023-05-22T15:43:17.122 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:17.115+0000 7fb157c5b640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fb1500b7dd0 0x7fb1500ccab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:17.122 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:17.115+0000 7fb157c5b640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb150160250 0x7fb1500d0410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:17.122 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:17.115+0000 7fb157c5b640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1500cd170 con 0x7fb15015f700 2023-05-22T15:43:17.122 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:17.115+0000 7fb157c5b640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1500d3360 con 0x7fb1500b7dd0 2023-05-22T15:43:17.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:17.115+0000 7fb157c5b640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1500d2ac0 con 0x7fb150160250 2023-05-22T15:43:17.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:17.115+0000 7fb1551f6640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fb15015f700 0x7fb15015fb40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:57154/0 (socket says 172.21.15.16:57154) 2023-05-22T15:43:17.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:17.115+0000 7fb1551f6640 1 -- 172.21.15.16:0/4152390878 learned_addr learned my addr 172.21.15.16:0/4152390878 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:43:17.127 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:17.119+0000 7fb1559f7640 1 --2- 172.21.15.16:0/4152390878 >> v2:172.21.15.62:3300/0 conn(0x7fb1500b7dd0 0x7fb1500ccab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:17.128 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:17.119+0000 7fb1561f8640 1 --2- 172.21.15.16:0/4152390878 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb150160250 0x7fb1500d0410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:17.128 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:17.119+0000 7fb1549f5640 1 -- 172.21.15.16:0/4152390878 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2741949706 0 0) 0x7fb1500cd170 con 0x7fb15015f700 2023-05-22T15:43:17.128 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:17.119+0000 7fb1549f5640 1 -- 172.21.15.16:0/4152390878 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb13c003530 con 0x7fb15015f700 2023-05-22T15:43:17.129 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:17.119+0000 7fb1549f5640 1 -- 172.21.15.16:0/4152390878 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 400571319 0 0) 0x7fb13c003530 con 0x7fb15015f700 2023-05-22T15:43:17.129 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:17.119+0000 7fb1549f5640 1 -- 172.21.15.16:0/4152390878 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb150160250 msgr2=0x7fb1500d0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:43:17.129 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:17.119+0000 7fb1549f5640 1 --2- 172.21.15.16:0/4152390878 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb150160250 0x7fb1500d0410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:17.129 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:17.119+0000 7fb1549f5640 1 -- 172.21.15.16:0/4152390878 >> v2:172.21.15.62:3300/0 conn(0x7fb1500b7dd0 msgr2=0x7fb1500ccab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:43:17.130 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:17.119+0000 7fb1549f5640 1 --2- 172.21.15.16:0/4152390878 >> v2:172.21.15.62:3300/0 conn(0x7fb1500b7dd0 0x7fb1500ccab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:17.130 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:17.119+0000 7fb1549f5640 1 -- 172.21.15.16:0/4152390878 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1500d2da0 con 0x7fb15015f700 2023-05-22T15:43:17.130 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:17.119+0000 7fb1549f5640 1 -- 172.21.15.16:0/4152390878 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7fb140001680 con 0x7fb15015f700 2023-05-22T15:43:17.130 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:17.119+0000 7fb1549f5640 1 -- 172.21.15.16:0/4152390878 >> v1:172.21.15.16:6789/0 conn(0x7fb15015f700 legacy=0x7fb15015fb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:17.131 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:17.119+0000 7fb1549f5640 1 --2- 172.21.15.16:0/4152390878 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb13c003980 0x7fb13c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:17.131 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:17.119+0000 7fb1549f5640 1 --2- 172.21.15.16:0/4152390878 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb13c0043b0 0x7fb13c004870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:17.131 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:17.119+0000 7fb1549f5640 1 --2- 172.21.15.16:0/4152390878 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb150160250 0x7fb13c004f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:17.131 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:17.119+0000 7fb1549f5640 1 -- 172.21.15.16:0/4152390878 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1500d3360 con 0x7fb13c0043b0 2023-05-22T15:43:17.131 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:17.119+0000 7fb1549f5640 1 -- 172.21.15.16:0/4152390878 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1500d2ac0 con 0x7fb13c003980 2023-05-22T15:43:17.132 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:17.119+0000 7fb1549f5640 1 -- 172.21.15.16:0/4152390878 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb13c00b990 con 0x7fb150160250 2023-05-22T15:43:17.132 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:17.119+0000 7fb1551f6640 1 --2- 172.21.15.16:0/4152390878 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb13c0043b0 0x7fb13c004870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:17.132 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:17.119+0000 7fb1551f6640 1 -- 172.21.15.16:0/4152390878 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb150160250 msgr2=0x7fb13c004f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:43:17.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:17.119+0000 7fb1559f7640 1 --2- 172.21.15.16:0/4152390878 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb13c003980 0x7fb13c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:17.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:17.119+0000 7fb1551f6640 1 --2- 172.21.15.16:0/4152390878 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb150160250 0x7fb13c004f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:17.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:17.119+0000 7fb1551f6640 1 -- 172.21.15.16:0/4152390878 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb13c003980 msgr2=0x7fb13c003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:43:17.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:17.119+0000 7fb1551f6640 1 --2- 172.21.15.16:0/4152390878 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb13c003980 0x7fb13c003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:17.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:17.119+0000 7fb1551f6640 1 -- 172.21.15.16:0/4152390878 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb13c003530 con 0x7fb13c0043b0 2023-05-22T15:43:17.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:17.119+0000 7fb1559f7640 1 --2- 172.21.15.16:0/4152390878 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb13c003980 0x7fb13c003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:43:17.134 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:17.119+0000 7fb1551f6640 1 --2- 172.21.15.16:0/4152390878 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb13c0043b0 0x7fb13c004870 secure :-1 s=READY pgs=1391 cs=0 l=1 rev1=1 crypto rx=0x7fb14000c240 tx=0x7fb14001f3d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7ecdf1f60a96e0d server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:43:17.134 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:17.119+0000 7fb1549f5640 1 -- 172.21.15.16:0/4152390878 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb140005ae0 con 0x7fb13c0043b0 2023-05-22T15:43:17.134 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:17.119+0000 7fb1549f5640 1 -- 172.21.15.16:0/4152390878 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1400ab070 con 0x7fb13c0043b0 2023-05-22T15:43:17.134 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:17.119+0000 7fb1549f5640 1 -- 172.21.15.16:0/4152390878 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb140005ea0 con 0x7fb13c0043b0 2023-05-22T15:43:17.134 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:17.119+0000 7fb157c5b640 1 -- 172.21.15.16:0/4152390878 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb13c0043b0 msgr2=0x7fb13c004870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:17.134 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:17.119+0000 7fb157c5b640 1 --2- 172.21.15.16:0/4152390878 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb13c0043b0 0x7fb13c004870 secure :-1 s=READY pgs=1391 cs=0 l=1 rev1=1 crypto rx=0x7fb14000c240 tx=0x7fb14001f3d0 comp rx=0 tx=0).stop 2023-05-22T15:43:17.134 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:17.119+0000 7fb1561f8640 1 -- 172.21.15.16:0/4152390878 reap_dead start 2023-05-22T15:43:17.134 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:17.123+0000 7fb157c5b640 1 -- 172.21.15.16:0/4152390878 shutdown_connections 2023-05-22T15:43:17.135 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:17.123+0000 7fb157c5b640 1 -- 172.21.15.16:0/4152390878 >> 172.21.15.16:0/4152390878 conn(0x7fb1500655c0 msgr2=0x7fb150070e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:43:17.135 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:17.123+0000 7fb157c5b640 1 -- 172.21.15.16:0/4152390878 shutdown_connections 2023-05-22T15:43:17.135 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:17.123+0000 7fb157c5b640 1 -- 172.21.15.16:0/4152390878 wait complete. 2023-05-22T15:43:17.135 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:17.123+0000 7fb157c5b640 1 Processor -- start 2023-05-22T15:43:17.135 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:17.123+0000 7fb157c5b640 1 -- start start 2023-05-22T15:43:17.137 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:17.123+0000 7fb157c5b640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb15015b6a0 0x7fb1500b0480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:17.137 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:17.123+0000 7fb157c5b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb1500cb100 0x7fb1500b09c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:17.137 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:17.123+0000 7fb157c5b640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb1500cab90 0x7fb1501f1090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:17.137 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:17.123+0000 7fb157c5b640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1500c9af0 con 0x7fb1500cb100 2023-05-22T15:43:17.137 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:17.123+0000 7fb157c5b640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1500c9dd0 con 0x7fb15015b6a0 2023-05-22T15:43:17.138 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:17.123+0000 7fb157c5b640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1500ca0b0 con 0x7fb1500cab90 2023-05-22T15:43:17.138 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:17.123+0000 7fb1551f6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb1500cb100 0x7fb1500b09c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:17.138 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:17.123+0000 7fb1559f7640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb15015b6a0 0x7fb1500b0480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:17.138 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:17.123+0000 7fb1551f6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb1500cb100 0x7fb1500b09c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46008/0 (socket says 172.21.15.16:46008) 2023-05-22T15:43:17.138 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:17.123+0000 7fb1551f6640 1 -- 172.21.15.16:0/2493704734 learned_addr learned my addr 172.21.15.16:0/2493704734 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:43:17.138 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:17.123+0000 7fb1561f8640 1 --2- 172.21.15.16:0/2493704734 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb1500cab90 0x7fb1501f1090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:17.138 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:17.123+0000 7fb1551f6640 1 -- 172.21.15.16:0/2493704734 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb1500cab90 msgr2=0x7fb1501f1090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:17.139 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:17.123+0000 7fb1551f6640 1 --2- 172.21.15.16:0/2493704734 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb1500cab90 0x7fb1501f1090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:17.139 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:17.123+0000 7fb1551f6640 1 -- 172.21.15.16:0/2493704734 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb15015b6a0 msgr2=0x7fb1500b0480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:17.139 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:17.123+0000 7fb1551f6640 1 --2- 172.21.15.16:0/2493704734 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb15015b6a0 0x7fb1500b0480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:17.139 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:17.123+0000 7fb1551f6640 1 -- 172.21.15.16:0/2493704734 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1500cf730 con 0x7fb1500cb100 2023-05-22T15:43:17.139 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:17.123+0000 7fb1551f6640 1 --2- 172.21.15.16:0/2493704734 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb1500cb100 0x7fb1500b09c0 secure :-1 s=READY pgs=1392 cs=0 l=1 rev1=1 crypto rx=0x7fb14000c240 tx=0x7fb13c005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:43:17.139 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:17.123+0000 7fb1467fc640 1 -- 172.21.15.16:0/2493704734 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1400b6070 con 0x7fb1500cb100 2023-05-22T15:43:17.139 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:17.123+0000 7fb1467fc640 1 -- 172.21.15.16:0/2493704734 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb140001c60 con 0x7fb1500cb100 2023-05-22T15:43:17.139 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:17.123+0000 7fb157c5b640 1 -- 172.21.15.16:0/2493704734 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1500d3080 con 0x7fb1500cb100 2023-05-22T15:43:17.140 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:17.123+0000 7fb1467fc640 1 -- 172.21.15.16:0/2493704734 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb140001660 con 0x7fb1500cb100 2023-05-22T15:43:17.140 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:17.123+0000 7fb157c5b640 1 -- 172.21.15.16:0/2493704734 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1500d0410 con 0x7fb1500cb100 2023-05-22T15:43:17.140 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:17.127+0000 7fb157c5b640 1 -- 172.21.15.16:0/2493704734 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb15004dbe0 con 0x7fb1500cb100 2023-05-22T15:43:17.140 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:17.127+0000 7fb1467fc640 1 -- 172.21.15.16:0/2493704734 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 40) v1 ==== 84068+0+0 (secure 0 0 0) 0x7fb1400ad020 con 0x7fb1500cb100 2023-05-22T15:43:17.140 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:17.127+0000 7fb1467fc640 1 --2- 172.21.15.16:0/2493704734 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fb1340651d0 0x7fb134067680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:17.141 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:17.127+0000 7fb1559f7640 1 --2- 172.21.15.16:0/2493704734 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fb1340651d0 0x7fb134067680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:17.141 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:17.127+0000 7fb1467fc640 1 -- 172.21.15.16:0/2493704734 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 7027+0+0 (secure 0 0 0) 0x7fb14011f2d0 con 0x7fb1500cb100 2023-05-22T15:43:17.141 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:17.127+0000 7fb1559f7640 1 --2- 172.21.15.16:0/2493704734 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fb1340651d0 0x7fb134067680 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fb14c009b70 tx=0x7fb14c008040 comp rx=0 tx=0).ready entity=mgr.7788 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:43:17.141 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:17.131+0000 7fb1467fc640 1 -- 172.21.15.16:0/2493704734 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fb1400ecf70 con 0x7fb1500cb100 2023-05-22T15:43:17.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:17.279+0000 7fb157c5b640 1 -- 172.21.15.16:0/2493704734 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb150067a50 con 0x7fb1500cb100 2023-05-22T15:43:17.287 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:17.283+0000 7fb1467fc640 1 -- 172.21.15.16:0/2493704734 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v106) v1 ==== 74+0+18962 (secure 0 0 0) 0x7fb1400f0590 con 0x7fb1500cb100 2023-05-22T15:43:17.287 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:43:17.288 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":106,"fsid":"b2719834-7847-4d98-90a6-f65d57f85c49","created":"2023-05-22T15:26:25.259314+0000","modified":"2023-05-22T15:43:14.959874+0000","last_up_change":"2023-05-22T15:26:36.448524+0000","last_in_change":"2023-05-22T15:26:27.675668+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-05-22T15:26:38.019781+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-05-22T15:26:43.570514+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2023-05-22T15:42:55.131159+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"103","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2023-05-22T15:42:55.567899+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"104","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9e7ff854-b25f-4cbb-a5c2-1799d9c53d2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":102,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":37021},{"type":"v1","addr":"172.21.15.16:6825","nonce":37021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":37021},{"type":"v1","addr":"172.21.15.16:6827","nonce":37021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":37021},{"type":"v1","addr":"172.21.15.16:6831","nonce":37021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":37021},{"type":"v1","addr":"172.21.15.16:6829","nonce":37021}]},"public_addr":"172.21.15.16:6825/37021","cluster_addr":"172.21.15.16:6827/37021","heartbeat_back_addr":"172.21.15.16:6831/37021","heartbeat_front_addr":"172.21.15.16:6829/37021","state":["exists","up"]},{"osd":1,"uuid":"4702fbc9-a927-4565-aca5-41dc69cc8264","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":102,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":37008},{"type":"v1","addr":"172.21.15.16:6801","nonce":37008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":37008},{"type":"v1","addr":"172.21.15.16:6803","nonce":37008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":37008},{"type":"v1","addr":"172.21.15.16:6807","nonce":37008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":37008},{"type":"v1","addr":"172.21.15.16:6805","nonce":37008}]},"public_addr":"172.21.15.16:6801/37008","cluster_addr":"172.21.15.16:6803/37008","heartbeat_back_addr":"172.21.15.16:6807/37008","heartbeat_front_addr":"172.21.15.16:6805/37008","state":["exists","up"]},{"osd":2,"uuid":"1e7bba46-4eba-47ef-bb81-5e2d371638ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":102,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":37009},{"type":"v1","addr":"172.21.15.16:6809","nonce":37009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":37009},{"type":"v1","addr":"172.21.15.16:6811","nonce":37009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":37009},{"type":"v1","addr":"172.21.15.16:6815","nonce":37009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":37009},{"type":"v1","addr":"172.21.15.16:6813","nonce":37009}]},"public_addr":"172.21.15.16:6809/37009","cluster_addr":"172.21.15.16:6811/37009","heartbeat_back_addr":"172.21.15.16:6815/37009","heartbeat_front_addr":"172.21.15.16:6813/37009","state":["exists","up"]},{"osd":3,"uuid":"383cacc0-5a88-4904-990e-498ca59ef48a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":102,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":37020},{"type":"v1","addr":"172.21.15.16:6817","nonce":37020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":37020},{"type":"v1","addr":"172.21.15.16:6819","nonce":37020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":37020},{"type":"v1","addr":"172.21.15.16:6823","nonce":37020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":37020},{"type":"v1","addr":"172.21.15.16:6821","nonce":37020}]},"public_addr":"172.21.15.16:6817/37020","cluster_addr":"172.21.15.16:6819/37020","heartbeat_back_addr":"172.21.15.16:6823/37020","heartbeat_front_addr":"172.21.15.16:6821/37020","state":["exists","up"]},{"osd":4,"uuid":"970500ef-95b7-42c1-b080-eed4fae7013d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":102,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":37353},{"type":"v1","addr":"172.21.15.62:6817","nonce":37353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6819","nonce":37353},{"type":"v1","addr":"172.21.15.62:6822","nonce":37353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37353},{"type":"v1","addr":"172.21.15.62:6831","nonce":37353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37353},{"type":"v1","addr":"172.21.15.62:6826","nonce":37353}]},"public_addr":"172.21.15.62:6817/37353","cluster_addr":"172.21.15.62:6822/37353","heartbeat_back_addr":"172.21.15.62:6831/37353","heartbeat_front_addr":"172.21.15.62:6826/37353","state":["exists","up"]},{"osd":5,"uuid":"022d67db-d4d8-445b-b1b9-8b446a00f5c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":102,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":37352},{"type":"v1","addr":"172.21.15.62:6801","nonce":37352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":37352},{"type":"v1","addr":"172.21.15.62:6803","nonce":37352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":37352},{"type":"v1","addr":"172.21.15.62:6807","nonce":37352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":37352},{"type":"v1","addr":"172.21.15.62:6805","nonce":37352}]},"public_addr":"172.21.15.62:6801/37352","cluster_addr":"172.21.15.62:6803/37352","heartbeat_back_addr":"172.21.15.62:6807/37352","heartbeat_front_addr":"172.21.15.62:6805/37352","state":["exists","up"]},{"osd":6,"uuid":"42bd372b-6688-4a1b-8407-e16e8c266e33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":102,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":37354},{"type":"v1","addr":"172.21.15.62:6820","nonce":37354}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":37354},{"type":"v1","addr":"172.21.15.62:6823","nonce":37354}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6829","nonce":37354},{"type":"v1","addr":"172.21.15.62:6830","nonce":37354}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6825","nonce":37354},{"type":"v1","addr":"172.21.15.62:6827","nonce":37354}]},"public_addr":"172.21.15.62:6820/37354","cluster_addr":"172.21.15.62:6823/37354","heartbeat_back_addr":"172.21.15.62:6830/37354","heartbeat_front_addr":"172.21.15.62:6827/37354","state":["exists","up"]},{"osd":7,"uuid":"ac70b2a3-f41b-4bf2-b227-7f2cbb55426d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":102,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37355},{"type":"v1","addr":"172.21.15.62:6809","nonce":37355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37355},{"type":"v1","addr":"172.21.15.62:6811","nonce":37355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":37355},{"type":"v1","addr":"172.21.15.62:6815","nonce":37355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":37355},{"type":"v1","addr":"172.21.15.62:6813","nonce":37355}]},"public_addr":"172.21.15.62:6809/37355","cluster_addr":"172.21.15.62:6811/37355","heartbeat_back_addr":"172.21.15.62:6815/37355","heartbeat_front_addr":"172.21.15.62:6813/37355","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.368896+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.107273+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.276888+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.287502+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.124791+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.251937+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.255679+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.232797+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:0/2165805468":"2023-05-23T15:43:14.959816+0000","172.21.15.16:0/2196574811":"2023-05-23T15:43:14.959816+0000","172.21.15.16:0/747277139":"2023-05-23T15:43:14.959816+0000","172.21.15.16:0/291514430":"2023-05-23T15:43:14.959816+0000","172.21.15.16:0/3869784079":"2023-05-23T15:43:14.959816+0000","172.21.15.16:0/638271246":"2023-05-23T15:43:14.959816+0000","172.21.15.16:0/1315052983":"2023-05-23T15:43:09.063704+0000","172.21.15.16:0/3981695108":"2023-05-23T15:43:09.063704+0000","172.21.15.16:0/92534915":"2023-05-23T15:43:09.063704+0000","172.21.15.16:6833/36921":"2023-05-23T15:43:09.063704+0000","172.21.15.16:6832/36921":"2023-05-23T15:43:09.063704+0000","172.21.15.16:0/2880991909":"2023-05-23T15:43:14.959816+0000","172.21.15.16:0/2741778632":"2023-05-23T15:43:09.063704+0000","172.21.15.16:0/3940121619":"2023-05-23T15:43:09.063704+0000","172.21.15.16:0/2073223440":"2023-05-23T15:43:09.063704+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-05-22T15:43:17.289 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:17.283+0000 7fb157c5b640 1 -- 172.21.15.16:0/2493704734 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fb1340651d0 msgr2=0x7fb134067680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:17.289 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:17.283+0000 7fb157c5b640 1 --2- 172.21.15.16:0/2493704734 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fb1340651d0 0x7fb134067680 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fb14c009b70 tx=0x7fb14c008040 comp rx=0 tx=0).stop 2023-05-22T15:43:17.289 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:17.283+0000 7fb157c5b640 1 -- 172.21.15.16:0/2493704734 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb1500cb100 msgr2=0x7fb1500b09c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:17.289 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:17.283+0000 7fb157c5b640 1 --2- 172.21.15.16:0/2493704734 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb1500cb100 0x7fb1500b09c0 secure :-1 s=READY pgs=1392 cs=0 l=1 rev1=1 crypto rx=0x7fb14000c240 tx=0x7fb13c005800 comp rx=0 tx=0).stop 2023-05-22T15:43:17.289 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:17.283+0000 7fb157c5b640 1 -- 172.21.15.16:0/2493704734 shutdown_connections 2023-05-22T15:43:17.290 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:17.283+0000 7fb157c5b640 1 --2- 172.21.15.16:0/2493704734 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fb1340651d0 0x7fb134067680 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:17.290 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:17.283+0000 7fb157c5b640 1 --2- 172.21.15.16:0/2493704734 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb1500cab90 0x7fb1501f1090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:17.290 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:17.283+0000 7fb157c5b640 1 --2- 172.21.15.16:0/2493704734 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb1500cb100 0x7fb1500b09c0 unknown :-1 s=CLOSED pgs=1392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:17.290 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:17.283+0000 7fb157c5b640 1 --2- 172.21.15.16:0/2493704734 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb15015b6a0 0x7fb1500b0480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:17.290 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:17.283+0000 7fb157c5b640 1 -- 172.21.15.16:0/2493704734 >> 172.21.15.16:0/2493704734 conn(0x7fb1500655c0 msgr2=0x7fb15006a6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:43:17.290 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:17.283+0000 7fb157c5b640 1 -- 172.21.15.16:0/2493704734 shutdown_connections 2023-05-22T15:43:17.290 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:17.283+0000 7fb157c5b640 1 -- 172.21.15.16:0/2493704734 wait complete. 2023-05-22T15:43:17.305 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-05-22T15:43:17.419 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:17.415+0000 7f116e32d640 1 Processor -- start 2023-05-22T15:43:17.419 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:17.415+0000 7f116e32d640 1 -- start start 2023-05-22T15:43:17.423 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:17.415+0000 7f116e32d640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f11680ca4e0 0x7f116815e460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:17.423 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:17.415+0000 7f116e32d640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1168070df0 0x7f1168071210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:17.423 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:17.415+0000 7f116e32d640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11680b0650 con 0x7f11680c9b40 2023-05-22T15:43:17.424 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:17.415+0000 7f116e32d640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11680b0370 con 0x7f11680ca4e0 2023-05-22T15:43:17.424 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:17.415+0000 7f116e32d640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f116815d980 con 0x7f1168070df0 2023-05-22T15:43:17.424 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:17.415+0000 7f11677fe640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f11680c9b40 0x7f11680afb40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:57166/0 (socket says 172.21.15.16:57166) 2023-05-22T15:43:17.425 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:17.415+0000 7f11677fe640 1 -- 172.21.15.16:0/1244352870 learned_addr learned my addr 172.21.15.16:0/1244352870 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:43:17.425 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:17.415+0000 7f1167fff640 1 --2- 172.21.15.16:0/1244352870 >> v2:172.21.15.62:3300/0 conn(0x7f11680ca4e0 0x7f116815e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:17.425 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:17.415+0000 7f116c8ca640 1 --2- 172.21.15.16:0/1244352870 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1168070df0 0x7f1168071210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:17.425 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:17.415+0000 7f1166ffd640 1 -- 172.21.15.16:0/1244352870 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1756966297 0 0) 0x7f11680b0650 con 0x7f11680c9b40 2023-05-22T15:43:17.426 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:17.415+0000 7f1166ffd640 1 -- 172.21.15.16:0/1244352870 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1150003530 con 0x7f11680c9b40 2023-05-22T15:43:17.426 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:17.415+0000 7f1166ffd640 1 -- 172.21.15.16:0/1244352870 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 508101499 0 0) 0x7f1150003530 con 0x7f11680c9b40 2023-05-22T15:43:17.426 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:17.415+0000 7f1166ffd640 1 -- 172.21.15.16:0/1244352870 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1168070df0 msgr2=0x7f1168071210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:43:17.427 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:17.415+0000 7f1166ffd640 1 --2- 172.21.15.16:0/1244352870 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1168070df0 0x7f1168071210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:17.427 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:17.415+0000 7f1166ffd640 1 -- 172.21.15.16:0/1244352870 >> v2:172.21.15.62:3300/0 conn(0x7f11680ca4e0 msgr2=0x7f116815e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:43:17.427 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:17.415+0000 7f1166ffd640 1 --2- 172.21.15.16:0/1244352870 >> v2:172.21.15.62:3300/0 conn(0x7f11680ca4e0 0x7f116815e460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:17.427 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:17.415+0000 7f1166ffd640 1 -- 172.21.15.16:0/1244352870 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f116815dc60 con 0x7f11680c9b40 2023-05-22T15:43:17.428 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:17.415+0000 7f1166ffd640 1 -- 172.21.15.16:0/1244352870 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7f115c0016a0 con 0x7f11680c9b40 2023-05-22T15:43:17.428 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:17.415+0000 7f1167fff640 1 --2- 172.21.15.16:0/1244352870 >> v2:172.21.15.62:3300/0 conn(0x7f11680ca4e0 0x7f116815e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-05-22T15:43:17.428 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:17.415+0000 7f1166ffd640 1 -- 172.21.15.16:0/1244352870 >> v1:172.21.15.16:6789/0 conn(0x7f11680c9b40 legacy=0x7f11680afb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:17.428 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:17.415+0000 7f1166ffd640 1 --2- 172.21.15.16:0/1244352870 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1150003980 0x7f1150003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:17.429 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:17.415+0000 7f1166ffd640 1 --2- 172.21.15.16:0/1244352870 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f11500043b0 0x7f1150004870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:17.429 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:17.415+0000 7f1166ffd640 1 --2- 172.21.15.16:0/1244352870 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1168070df0 0x7f1150004f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:17.429 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:17.415+0000 7f1166ffd640 1 -- 172.21.15.16:0/1244352870 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11680b0370 con 0x7f11500043b0 2023-05-22T15:43:17.429 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:17.415+0000 7f1166ffd640 1 -- 172.21.15.16:0/1244352870 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f116815d980 con 0x7f1150003980 2023-05-22T15:43:17.430 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:17.415+0000 7f1166ffd640 1 -- 172.21.15.16:0/1244352870 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f115000b990 con 0x7f1168070df0 2023-05-22T15:43:17.430 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:17.415+0000 7f11677fe640 1 --2- 172.21.15.16:0/1244352870 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f11500043b0 0x7f1150004870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:17.430 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:17.415+0000 7f1167fff640 1 --2- 172.21.15.16:0/1244352870 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1150003980 0x7f1150003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:17.430 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:17.415+0000 7f116c8ca640 1 --2- 172.21.15.16:0/1244352870 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1168070df0 0x7f1150004f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:17.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:17.415+0000 7f11677fe640 1 -- 172.21.15.16:0/1244352870 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1168070df0 msgr2=0x7f1150004f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:43:17.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:17.415+0000 7f11677fe640 1 --2- 172.21.15.16:0/1244352870 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1168070df0 0x7f1150004f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:17.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:17.415+0000 7f11677fe640 1 -- 172.21.15.16:0/1244352870 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1150003980 msgr2=0x7f1150003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:43:17.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:17.415+0000 7f11677fe640 1 --2- 172.21.15.16:0/1244352870 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1150003980 0x7f1150003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:17.432 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:17.415+0000 7f11677fe640 1 -- 172.21.15.16:0/1244352870 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1150003530 con 0x7f11500043b0 2023-05-22T15:43:17.432 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:17.415+0000 7f1167fff640 1 --2- 172.21.15.16:0/1244352870 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1150003980 0x7f1150003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T15:43:17.437 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:17.419+0000 7f11677fe640 1 --2- 172.21.15.16:0/1244352870 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f11500043b0 0x7f1150004870 secure :-1 s=READY pgs=1394 cs=0 l=1 rev1=1 crypto rx=0x7f115c00c210 tx=0x7f115c01f390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e92d5ce59dcd6fb server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:43:17.437 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:17.419+0000 7f1166ffd640 1 -- 172.21.15.16:0/1244352870 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f115c005ae0 con 0x7f11500043b0 2023-05-22T15:43:17.437 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:17.419+0000 7f1166ffd640 1 -- 172.21.15.16:0/1244352870 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f115c0ab070 con 0x7f11500043b0 2023-05-22T15:43:17.438 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:17.419+0000 7f1166ffd640 1 -- 172.21.15.16:0/1244352870 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f115c005e80 con 0x7f11500043b0 2023-05-22T15:43:17.438 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:17.419+0000 7f116e32d640 1 -- 172.21.15.16:0/1244352870 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f11500043b0 msgr2=0x7f1150004870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:17.438 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:17.419+0000 7f116e32d640 1 --2- 172.21.15.16:0/1244352870 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f11500043b0 0x7f1150004870 secure :-1 s=READY pgs=1394 cs=0 l=1 rev1=1 crypto rx=0x7f115c00c210 tx=0x7f115c01f390 comp rx=0 tx=0).stop 2023-05-22T15:43:17.438 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:17.419+0000 7f116c8ca640 1 -- 172.21.15.16:0/1244352870 reap_dead start 2023-05-22T15:43:17.439 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:17.419+0000 7f116e32d640 1 -- 172.21.15.16:0/1244352870 shutdown_connections 2023-05-22T15:43:17.439 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:17.419+0000 7f116e32d640 1 -- 172.21.15.16:0/1244352870 >> 172.21.15.16:0/1244352870 conn(0x7f1168069720 msgr2=0x7f1168069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:43:17.439 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:17.419+0000 7f116e32d640 1 -- 172.21.15.16:0/1244352870 shutdown_connections 2023-05-22T15:43:17.439 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:17.419+0000 7f116e32d640 1 -- 172.21.15.16:0/1244352870 wait complete. 2023-05-22T15:43:17.440 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:17.419+0000 7f116e32d640 1 Processor -- start 2023-05-22T15:43:17.440 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:17.419+0000 7f116e32d640 1 -- start start 2023-05-22T15:43:17.440 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:17.423+0000 7f116e32d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f11500043b0 0x7f11680afa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:17.441 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:17.423+0000 7f116e32d640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f116814e3b0 0x7f11680c1720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:17.441 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:17.423+0000 7f116e32d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f11680c30c0 0x7f1168150d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:17.441 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:17.423+0000 7f116e32d640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11680c2020 con 0x7f11500043b0 2023-05-22T15:43:17.441 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:17.423+0000 7f116e32d640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11680c2300 con 0x7f11680c30c0 2023-05-22T15:43:17.442 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:17.423+0000 7f116e32d640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11680c25e0 con 0x7f116814e3b0 2023-05-22T15:43:17.442 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:17.423+0000 7f1167fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f11500043b0 0x7f11680afa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:17.442 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:17.423+0000 7f1167fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f11500043b0 0x7f11680afa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46018/0 (socket says 172.21.15.16:46018) 2023-05-22T15:43:17.442 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:17.423+0000 7f1167fff640 1 -- 172.21.15.16:0/1036456541 learned_addr learned my addr 172.21.15.16:0/1036456541 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:43:17.443 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:17.423+0000 7f116c8ca640 1 --2- 172.21.15.16:0/1036456541 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f11680c30c0 0x7f1168150d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:17.443 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:17.423+0000 7f11677fe640 1 --2- 172.21.15.16:0/1036456541 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f116814e3b0 0x7f11680c1720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:17.443 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:17.423+0000 7f1167fff640 1 -- 172.21.15.16:0/1036456541 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f116814e3b0 msgr2=0x7f11680c1720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:17.443 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:17.423+0000 7f1167fff640 1 --2- 172.21.15.16:0/1036456541 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f116814e3b0 0x7f11680c1720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:17.444 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:17.423+0000 7f1167fff640 1 -- 172.21.15.16:0/1036456541 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f11680c30c0 msgr2=0x7f1168150d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:17.444 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:17.423+0000 7f1167fff640 1 --2- 172.21.15.16:0/1036456541 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f11680c30c0 0x7f1168150d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:17.444 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:17.423+0000 7f1167fff640 1 -- 172.21.15.16:0/1036456541 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11680ca8b0 con 0x7f11500043b0 2023-05-22T15:43:17.444 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:17.423+0000 7f11677fe640 1 --2- 172.21.15.16:0/1036456541 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f116814e3b0 0x7f11680c1720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T15:43:17.445 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:17.423+0000 7f1167fff640 1 --2- 172.21.15.16:0/1036456541 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f11500043b0 0x7f11680afa20 secure :-1 s=READY pgs=1395 cs=0 l=1 rev1=1 crypto rx=0x7f11580118d0 tx=0x7f1158011d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:43:17.445 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:17.423+0000 7f1164ff9640 1 -- 172.21.15.16:0/1036456541 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f115800fbd0 con 0x7f11500043b0 2023-05-22T15:43:17.445 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:17.423+0000 7f1164ff9640 1 -- 172.21.15.16:0/1036456541 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f115800fd30 con 0x7f11500043b0 2023-05-22T15:43:17.445 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:17.423+0000 7f116e32d640 1 -- 172.21.15.16:0/1036456541 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f116815df40 con 0x7f11500043b0 2023-05-22T15:43:17.446 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:17.423+0000 7f1164ff9640 1 -- 172.21.15.16:0/1036456541 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1158015660 con 0x7f11500043b0 2023-05-22T15:43:17.446 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:17.423+0000 7f116e32d640 1 -- 172.21.15.16:0/1036456541 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1168071210 con 0x7f11500043b0 2023-05-22T15:43:17.446 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:17.423+0000 7f116e32d640 1 -- 172.21.15.16:0/1036456541 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f116804dbe0 con 0x7f11500043b0 2023-05-22T15:43:17.446 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:17.423+0000 7f1164ff9640 1 -- 172.21.15.16:0/1036456541 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 40) v1 ==== 84068+0+0 (secure 0 0 0) 0x7f11580157c0 con 0x7f11500043b0 2023-05-22T15:43:17.447 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:17.423+0000 7f1164ff9640 1 --2- 172.21.15.16:0/1036456541 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f1154065200 0x7f11540676b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:17.447 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:17.423+0000 7f1164ff9640 1 -- 172.21.15.16:0/1036456541 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 7027+0+0 (secure 0 0 0) 0x7f11580831f0 con 0x7f11500043b0 2023-05-22T15:43:17.447 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:17.423+0000 7f11677fe640 1 --2- 172.21.15.16:0/1036456541 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f1154065200 0x7f11540676b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:17.447 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:17.423+0000 7f11677fe640 1 --2- 172.21.15.16:0/1036456541 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f1154065200 0x7f11540676b0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f115c0ae000 tx=0x7f115c001aa0 comp rx=0 tx=0).ready entity=mgr.7788 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:43:17.448 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:17.427+0000 7f1164ff9640 1 -- 172.21.15.16:0/1036456541 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f1158050de0 con 0x7f11500043b0 2023-05-22T15:43:17.612 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:17.607+0000 7f116e32d640 1 -- 172.21.15.16:0/1036456541 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f11680c4a80 con 0x7f11500043b0 2023-05-22T15:43:17.998 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:17.991+0000 7f1164ff9640 1 -- 172.21.15.16:0/1036456541 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mgrmap(e 41) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f11580511b0 con 0x7f11500043b0 2023-05-22T15:43:18.462 INFO:tasks.ceph.mon.a.smithi016.stderr:2023-05-22T15:43:18.459+0000 7f95dd48a640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-05-22T15:43:18.473 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:18.467+0000 7f1164ff9640 1 -- 172.21.15.16:0/1036456541 <== mon.0 v2:172.21.15.16:3300/0 8 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v37) v1 ==== 161+0+0 (secure 0 0 0) 0x7f1158054810 con 0x7f11500043b0 2023-05-22T15:43:18.474 INFO:teuthology.orchestra.run.smithi016.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-05-22T15:43:18.477 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:18.471+0000 7f116e32d640 1 -- 172.21.15.16:0/1036456541 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f1154065200 msgr2=0x7f11540676b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:18.478 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:18.471+0000 7f116e32d640 1 --2- 172.21.15.16:0/1036456541 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f1154065200 0x7f11540676b0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f115c0ae000 tx=0x7f115c001aa0 comp rx=0 tx=0).stop 2023-05-22T15:43:18.478 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:18.471+0000 7f116e32d640 1 -- 172.21.15.16:0/1036456541 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f11500043b0 msgr2=0x7f11680afa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:18.478 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:18.471+0000 7f116e32d640 1 --2- 172.21.15.16:0/1036456541 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f11500043b0 0x7f11680afa20 secure :-1 s=READY pgs=1395 cs=0 l=1 rev1=1 crypto rx=0x7f11580118d0 tx=0x7f1158011d90 comp rx=0 tx=0).stop 2023-05-22T15:43:18.478 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:18.471+0000 7f116e32d640 1 -- 172.21.15.16:0/1036456541 shutdown_connections 2023-05-22T15:43:18.478 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:18.471+0000 7f116e32d640 1 --2- 172.21.15.16:0/1036456541 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f11680c30c0 0x7f1168150d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:18.478 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:18.471+0000 7f116e32d640 1 --2- 172.21.15.16:0/1036456541 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f1154065200 0x7f11540676b0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:18.478 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:18.471+0000 7f116e32d640 1 --2- 172.21.15.16:0/1036456541 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f116814e3b0 0x7f11680c1720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:18.479 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:18.471+0000 7f116e32d640 1 --2- 172.21.15.16:0/1036456541 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f11500043b0 0x7f11680afa20 unknown :-1 s=CLOSED pgs=1395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:18.479 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:18.471+0000 7f116e32d640 1 -- 172.21.15.16:0/1036456541 >> 172.21.15.16:0/1036456541 conn(0x7f1168069720 msgr2=0x7f116815cc20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:43:18.479 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:18.471+0000 7f116e32d640 1 -- 172.21.15.16:0/1036456541 shutdown_connections 2023-05-22T15:43:18.479 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:18.471+0000 7f116e32d640 1 -- 172.21.15.16:0/1036456541 wait complete. 2023-05-22T15:43:18.479 INFO:tasks.ceph.mds.a.smithi016.stderr: -1637> 2023-05-22T15:42:58.035+0000 7f2d6f035640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-05-22T15:43:18.500 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-05-22T15:43:18.514 INFO:tasks.ceph.mds.a.smithi016.stdout:starting mds.a at 2023-05-22T15:43:18.621 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:18.615+0000 7f3b6d71a640 1 Processor -- start 2023-05-22T15:43:18.621 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:18.615+0000 7f3b6d71a640 1 -- start start 2023-05-22T15:43:18.622 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:18.615+0000 7f3b6d71a640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f3b6815bca0 0x7f3b68160160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:18.623 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:18.619+0000 7f3b6d71a640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3b680b4a70 0x7f3b680d0890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:18.623 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:18.619+0000 7f3b6d71a640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b680d0400 con 0x7f3b680b3ee0 2023-05-22T15:43:18.623 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:18.619+0000 7f3b6d71a640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b6815c070 con 0x7f3b6815bca0 2023-05-22T15:43:18.623 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:18.619+0000 7f3b6d71a640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b681606a0 con 0x7f3b680b4a70 2023-05-22T15:43:18.623 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:18.619+0000 7f3b66ffd640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f3b6815bca0 0x7f3b68160160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:18.623 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:18.619+0000 7f3b66ffd640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f3b6815bca0 0x7f3b68160160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46050/0 (socket says 172.21.15.16:46050) 2023-05-22T15:43:18.623 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:18.619+0000 7f3b66ffd640 1 -- 172.21.15.16:0/1807899900 learned_addr learned my addr 172.21.15.16:0/1807899900 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:43:18.624 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:18.619+0000 7f3b677fe640 1 --2- 172.21.15.16:0/1807899900 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3b680b4a70 0x7f3b680d0890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:18.624 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:18.619+0000 7f3b66ffd640 1 -- 172.21.15.16:0/1807899900 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3b680b4a70 msgr2=0x7f3b680d0890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:43:18.624 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:18.619+0000 7f3b66ffd640 1 --2- 172.21.15.16:0/1807899900 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3b680b4a70 0x7f3b680d0890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:18.624 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:18.619+0000 7f3b66ffd640 1 -- 172.21.15.16:0/1807899900 >> v1:172.21.15.62:6789/0 conn(0x7f3b680b3ee0 legacy=0x7f3b680b4360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:43:18.624 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:18.619+0000 7f3b66ffd640 1 -- 172.21.15.16:0/1807899900 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b680d00d0 con 0x7f3b6815bca0 2023-05-22T15:43:18.625 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:18.619+0000 7f3b66ffd640 1 --2- 172.21.15.16:0/1807899900 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3b6815bca0 0x7f3b68160160 secure :-1 s=READY pgs=1398 cs=0 l=1 rev1=1 crypto rx=0x7f3b58009870 tx=0x7f3b5801c3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=893385dc06669ddf server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:43:18.625 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:18.619+0000 7f3b65ffb640 1 -- 172.21.15.16:0/1807899900 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b580ab600 con 0x7f3b6815bca0 2023-05-22T15:43:18.625 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:18.619+0000 7f3b65ffb640 1 -- 172.21.15.16:0/1807899900 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b580a7050 con 0x7f3b6815bca0 2023-05-22T15:43:18.625 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:18.619+0000 7f3b65ffb640 1 -- 172.21.15.16:0/1807899900 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b580ab980 con 0x7f3b6815bca0 2023-05-22T15:43:18.626 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:18.619+0000 7f3b6d71a640 1 -- 172.21.15.16:0/1807899900 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3b6815bca0 msgr2=0x7f3b68160160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:18.626 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:18.619+0000 7f3b6d71a640 1 --2- 172.21.15.16:0/1807899900 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3b6815bca0 0x7f3b68160160 secure :-1 s=READY pgs=1398 cs=0 l=1 rev1=1 crypto rx=0x7f3b58009870 tx=0x7f3b5801c3b0 comp rx=0 tx=0).stop 2023-05-22T15:43:18.626 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:18.619+0000 7f3b6d71a640 1 -- 172.21.15.16:0/1807899900 shutdown_connections 2023-05-22T15:43:18.626 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:18.619+0000 7f3b6d71a640 1 --2- 172.21.15.16:0/1807899900 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3b680b4a70 0x7f3b680d0890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:18.627 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:18.619+0000 7f3b6d71a640 1 --2- 172.21.15.16:0/1807899900 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3b6815bca0 0x7f3b68160160 unknown :-1 s=CLOSED pgs=1398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:18.627 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:18.619+0000 7f3b6d71a640 1 -- 172.21.15.16:0/1807899900 >> 172.21.15.16:0/1807899900 conn(0x7f3b680657f0 msgr2=0x7f3b68070e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:43:18.627 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:18.619+0000 7f3b6d71a640 1 -- 172.21.15.16:0/1807899900 shutdown_connections 2023-05-22T15:43:18.627 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:18.619+0000 7f3b6d71a640 1 -- 172.21.15.16:0/1807899900 wait complete. 2023-05-22T15:43:18.627 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:18.623+0000 7f3b6d71a640 1 Processor -- start 2023-05-22T15:43:18.628 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:18.623+0000 7f3b6d71a640 1 -- start start 2023-05-22T15:43:18.628 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:18.623+0000 7f3b6d71a640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3b680b3ee0 0x7f3b6815aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:18.629 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:18.623+0000 7f3b6d71a640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3b680b4a70 0x7f3b680c84b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:18.629 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:18.623+0000 7f3b6d71a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3b6815bca0 0x7f3b680c89f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:18.629 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:18.623+0000 7f3b6d71a640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b680c78e0 con 0x7f3b6815bca0 2023-05-22T15:43:18.629 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:18.623+0000 7f3b6d71a640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b680c6cd0 con 0x7f3b680b4a70 2023-05-22T15:43:18.629 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:18.623+0000 7f3b6d71a640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b680c6fb0 con 0x7f3b680b3ee0 2023-05-22T15:43:18.629 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:18.623+0000 7f3b677fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3b6815bca0 0x7f3b680c89f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:18.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:18.623+0000 7f3b677fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3b6815bca0 0x7f3b680c89f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46060/0 (socket says 172.21.15.16:46060) 2023-05-22T15:43:18.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:18.623+0000 7f3b677fe640 1 -- 172.21.15.16:0/363037248 learned_addr learned my addr 172.21.15.16:0/363037248 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:43:18.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:18.623+0000 7f3b66ffd640 1 --2- 172.21.15.16:0/363037248 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3b680b3ee0 0x7f3b6815aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:18.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:18.623+0000 7f3b667fc640 1 --2- 172.21.15.16:0/363037248 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3b680b4a70 0x7f3b680c84b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:18.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:18.623+0000 7f3b677fe640 1 -- 172.21.15.16:0/363037248 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3b680b3ee0 msgr2=0x7f3b6815aba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:18.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:18.623+0000 7f3b677fe640 1 --2- 172.21.15.16:0/363037248 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3b680b3ee0 0x7f3b6815aba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:18.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:18.623+0000 7f3b677fe640 1 -- 172.21.15.16:0/363037248 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3b680b4a70 msgr2=0x7f3b680c84b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:18.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:18.623+0000 7f3b677fe640 1 --2- 172.21.15.16:0/363037248 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3b680b4a70 0x7f3b680c84b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:18.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:18.623+0000 7f3b677fe640 1 -- 172.21.15.16:0/363037248 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b580094d0 con 0x7f3b6815bca0 2023-05-22T15:43:18.632 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:18.623+0000 7f3b677fe640 1 --2- 172.21.15.16:0/363037248 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3b6815bca0 0x7f3b680c89f0 secure :-1 s=READY pgs=1399 cs=0 l=1 rev1=1 crypto rx=0x7f3b60007e40 tx=0x7f3b6000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:43:18.632 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:18.623+0000 7f3b4f7fe640 1 -- 172.21.15.16:0/363037248 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b60014620 con 0x7f3b6815bca0 2023-05-22T15:43:18.632 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:18.623+0000 7f3b4f7fe640 1 -- 172.21.15.16:0/363037248 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b6000ee90 con 0x7f3b6815bca0 2023-05-22T15:43:18.632 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:18.623+0000 7f3b6d71a640 1 -- 172.21.15.16:0/363037248 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b680d15e0 con 0x7f3b6815bca0 2023-05-22T15:43:18.632 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:18.623+0000 7f3b4f7fe640 1 -- 172.21.15.16:0/363037248 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b600149e0 con 0x7f3b6815bca0 2023-05-22T15:43:18.632 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:18.623+0000 7f3b6d71a640 1 -- 172.21.15.16:0/363037248 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b680cd1a0 con 0x7f3b6815bca0 2023-05-22T15:43:18.633 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:18.627+0000 7f3b6d71a640 1 -- 172.21.15.16:0/363037248 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b680d1770 con 0x7f3b6815bca0 2023-05-22T15:43:18.633 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:18.627+0000 7f3b4f7fe640 1 -- 172.21.15.16:0/363037248 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 41) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f3b60014b40 con 0x7f3b6815bca0 2023-05-22T15:43:18.633 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:18.627+0000 7f3b4f7fe640 1 --2- 172.21.15.16:0/363037248 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f3b540654d0 0x7f3b54067980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:18.633 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:18.627+0000 7f3b4f7fe640 1 -- 172.21.15.16:0/363037248 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 7113+0+0 (secure 0 0 0) 0x7f3b600836c0 con 0x7f3b6815bca0 2023-05-22T15:43:18.633 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:18.627+0000 7f3b66ffd640 1 --2- 172.21.15.16:0/363037248 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f3b540654d0 0x7f3b54067980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:18.633 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:18.627+0000 7f3b66ffd640 1 --2- 172.21.15.16:0/363037248 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f3b540654d0 0x7f3b54067980 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f3b580ad000 tx=0x7f3b58008040 comp rx=0 tx=0).ready entity=mgr.7788 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:43:18.635 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:18.631+0000 7f3b4f7fe640 1 -- 172.21.15.16:0/363037248 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f3b60050ea0 con 0x7f3b6815bca0 2023-05-22T15:43:18.825 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:18.819+0000 7f3b6d71a640 1 -- 172.21.15.16:0/363037248 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f3b68151e50 con 0x7f3b6815bca0 2023-05-22T15:43:19.483 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:19.479+0000 7f3b4f7fe640 1 -- 172.21.15.16:0/363037248 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v38) v1 ==== 104+0+0 (secure 0 0 0) 0x7f3b68151e50 con 0x7f3b6815bca0 2023-05-22T15:43:19.487 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:19.483+0000 7f3b6d71a640 1 -- 172.21.15.16:0/363037248 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f3b540654d0 msgr2=0x7f3b54067980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:19.487 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:19.483+0000 7f3b6d71a640 1 --2- 172.21.15.16:0/363037248 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f3b540654d0 0x7f3b54067980 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f3b580ad000 tx=0x7f3b58008040 comp rx=0 tx=0).stop 2023-05-22T15:43:19.487 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:19.483+0000 7f3b6d71a640 1 -- 172.21.15.16:0/363037248 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3b6815bca0 msgr2=0x7f3b680c89f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:19.488 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:19.483+0000 7f3b6d71a640 1 --2- 172.21.15.16:0/363037248 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3b6815bca0 0x7f3b680c89f0 secure :-1 s=READY pgs=1399 cs=0 l=1 rev1=1 crypto rx=0x7f3b60007e40 tx=0x7f3b6000e4b0 comp rx=0 tx=0).stop 2023-05-22T15:43:19.488 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:19.483+0000 7f3b6d71a640 1 -- 172.21.15.16:0/363037248 shutdown_connections 2023-05-22T15:43:19.489 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:19.483+0000 7f3b6d71a640 1 --2- 172.21.15.16:0/363037248 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3b6815bca0 0x7f3b680c89f0 unknown :-1 s=CLOSED pgs=1399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:19.489 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:19.483+0000 7f3b6d71a640 1 --2- 172.21.15.16:0/363037248 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3b680b4a70 0x7f3b680c84b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:19.489 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:19.483+0000 7f3b6d71a640 1 --2- 172.21.15.16:0/363037248 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f3b540654d0 0x7f3b54067980 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:19.489 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:19.483+0000 7f3b6d71a640 1 --2- 172.21.15.16:0/363037248 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3b680b3ee0 0x7f3b6815aba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:19.489 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:19.483+0000 7f3b6d71a640 1 -- 172.21.15.16:0/363037248 >> 172.21.15.16:0/363037248 conn(0x7f3b680657f0 msgr2=0x7f3b680ce720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:43:19.489 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:19.483+0000 7f3b6d71a640 1 -- 172.21.15.16:0/363037248 shutdown_connections 2023-05-22T15:43:19.490 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:19.483+0000 7f3b6d71a640 1 -- 172.21.15.16:0/363037248 wait complete. 2023-05-22T15:43:19.509 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-05-22T15:43:19.683 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:19.679+0000 7f1902c94640 1 Processor -- start 2023-05-22T15:43:19.684 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:19.679+0000 7f1902c94640 1 -- start start 2023-05-22T15:43:19.686 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:19.679+0000 7f1902c94640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f18fc0ce530 0x7f18fc15c540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:19.686 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:19.679+0000 7f1902c94640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f18fc0ce900 0x7f18fc159680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:19.687 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:19.679+0000 7f1902c94640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18fc0c3930 con 0x7f18fc159bc0 2023-05-22T15:43:19.687 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:19.679+0000 7f1902c94640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18fc158f00 con 0x7f18fc0ce900 2023-05-22T15:43:19.687 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:19.679+0000 7f1902c94640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18fc15cc40 con 0x7f18fc0ce530 2023-05-22T15:43:19.687 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:19.679+0000 7f18fbfff640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f18fc0ce900 0x7f18fc159680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:19.688 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:19.679+0000 7f18fbfff640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f18fc0ce900 0x7f18fc159680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57694/0 (socket says 172.21.15.16:57694) 2023-05-22T15:43:19.688 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:19.679+0000 7f18fbfff640 1 -- 172.21.15.16:0/1300510966 learned_addr learned my addr 172.21.15.16:0/1300510966 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:43:19.688 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:19.679+0000 7f1900a30640 1 --2- 172.21.15.16:0/1300510966 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f18fc0ce530 0x7f18fc15c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:19.688 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:19.679+0000 7f18fbfff640 1 -- 172.21.15.16:0/1300510966 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f18fc0ce530 msgr2=0x7f18fc15c540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:43:19.689 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:19.679+0000 7f18fbfff640 1 --2- 172.21.15.16:0/1300510966 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f18fc0ce530 0x7f18fc15c540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:19.689 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:19.679+0000 7f18fbfff640 1 -- 172.21.15.16:0/1300510966 >> v1:172.21.15.62:6789/0 conn(0x7f18fc159bc0 legacy=0x7f18fc0cc680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:19.689 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:19.679+0000 7f18fbfff640 1 -- 172.21.15.16:0/1300510966 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18fc15cf20 con 0x7f18fc0ce900 2023-05-22T15:43:19.689 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:19.679+0000 7f1900a30640 1 --2- 172.21.15.16:0/1300510966 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f18fc0ce530 0x7f18fc15c540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:43:19.692 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:19.683+0000 7f18fbfff640 1 --2- 172.21.15.16:0/1300510966 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f18fc0ce900 0x7f18fc159680 secure :-1 s=READY pgs=1400 cs=0 l=1 rev1=1 crypto rx=0x7f18f0009870 tx=0x7f18f001c540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7409b51d827e324c server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:43:19.692 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:19.683+0000 7f18fb7fe640 1 -- 172.21.15.16:0/1300510966 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18f00a9400 con 0x7f18fc0ce900 2023-05-22T15:43:19.692 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:19.683+0000 7f18fb7fe640 1 -- 172.21.15.16:0/1300510966 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18f00a9560 con 0x7f18fc0ce900 2023-05-22T15:43:19.692 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:19.683+0000 7f18fb7fe640 1 -- 172.21.15.16:0/1300510966 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18f00a98f0 con 0x7f18fc0ce900 2023-05-22T15:43:19.692 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:19.683+0000 7f1902c94640 1 -- 172.21.15.16:0/1300510966 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f18fc0ce900 msgr2=0x7f18fc159680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:19.692 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:19.683+0000 7f1902c94640 1 --2- 172.21.15.16:0/1300510966 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f18fc0ce900 0x7f18fc159680 secure :-1 s=READY pgs=1400 cs=0 l=1 rev1=1 crypto rx=0x7f18f0009870 tx=0x7f18f001c540 comp rx=0 tx=0).stop 2023-05-22T15:43:19.693 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:19.683+0000 7f1902c94640 1 -- 172.21.15.16:0/1300510966 shutdown_connections 2023-05-22T15:43:19.693 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:19.683+0000 7f1902c94640 1 --2- 172.21.15.16:0/1300510966 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f18fc0ce900 0x7f18fc159680 unknown :-1 s=CLOSED pgs=1400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:19.693 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:19.683+0000 7f1902c94640 1 --2- 172.21.15.16:0/1300510966 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f18fc0ce530 0x7f18fc15c540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:19.693 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:19.683+0000 7f1902c94640 1 -- 172.21.15.16:0/1300510966 >> 172.21.15.16:0/1300510966 conn(0x7f18fc069720 msgr2=0x7f18fc069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:43:19.693 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:19.683+0000 7f1902c94640 1 -- 172.21.15.16:0/1300510966 shutdown_connections 2023-05-22T15:43:19.693 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:19.683+0000 7f1902c94640 1 -- 172.21.15.16:0/1300510966 wait complete. 2023-05-22T15:43:19.693 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:19.683+0000 7f1902c94640 1 Processor -- start 2023-05-22T15:43:19.693 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:19.683+0000 7f1902c94640 1 -- start start 2023-05-22T15:43:19.694 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:19.687+0000 7f1902c94640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f18fc0ce530 0x7f18fc14a740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:19.694 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:19.687+0000 7f1902c94640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f18fc0ce900 0x7f18fc155ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:19.694 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:19.687+0000 7f1902c94640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f18fc159bc0 0x7f18fc0cad60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:19.694 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:19.687+0000 7f1902c94640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18fc14ac80 con 0x7f18fc159bc0 2023-05-22T15:43:19.694 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:19.687+0000 7f1902c94640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18fc1561e0 con 0x7f18fc0ce900 2023-05-22T15:43:19.694 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:19.687+0000 7f1902c94640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18fc156480 con 0x7f18fc0ce530 2023-05-22T15:43:19.694 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:19.687+0000 7f1901231640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f18fc159bc0 0x7f18fc0cad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:19.695 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:19.687+0000 7f1901231640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f18fc159bc0 0x7f18fc0cad60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57700/0 (socket says 172.21.15.16:57700) 2023-05-22T15:43:19.695 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:19.687+0000 7f1901231640 1 -- 172.21.15.16:0/2970932193 learned_addr learned my addr 172.21.15.16:0/2970932193 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:43:19.695 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:19.687+0000 7f1900a30640 1 --2- 172.21.15.16:0/2970932193 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f18fc0ce530 0x7f18fc14a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:19.695 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:19.687+0000 7f18fbfff640 1 --2- 172.21.15.16:0/2970932193 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f18fc0ce900 0x7f18fc155ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:19.695 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:19.687+0000 7f1901231640 1 -- 172.21.15.16:0/2970932193 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f18fc0ce530 msgr2=0x7f18fc14a740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:19.695 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:19.687+0000 7f1901231640 1 --2- 172.21.15.16:0/2970932193 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f18fc0ce530 0x7f18fc14a740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:19.695 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:19.687+0000 7f1901231640 1 -- 172.21.15.16:0/2970932193 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f18fc0ce900 msgr2=0x7f18fc155ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:19.695 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:19.687+0000 7f1901231640 1 --2- 172.21.15.16:0/2970932193 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f18fc0ce900 0x7f18fc155ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:19.696 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:19.687+0000 7f1901231640 1 -- 172.21.15.16:0/2970932193 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18f00094d0 con 0x7f18fc159bc0 2023-05-22T15:43:19.696 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:19.687+0000 7f1901231640 1 --2- 172.21.15.16:0/2970932193 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f18fc159bc0 0x7f18fc0cad60 secure :-1 s=READY pgs=1401 cs=0 l=1 rev1=1 crypto rx=0x7f18f400c730 tx=0x7f18f400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:43:19.696 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:19.687+0000 7f18f97fa640 1 -- 172.21.15.16:0/2970932193 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18f4003b80 con 0x7f18fc159bc0 2023-05-22T15:43:19.696 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:19.687+0000 7f18f97fa640 1 -- 172.21.15.16:0/2970932193 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18f4003ce0 con 0x7f18fc159bc0 2023-05-22T15:43:19.696 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:19.687+0000 7f1902c94640 1 -- 172.21.15.16:0/2970932193 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18fc0c9700 con 0x7f18fc159bc0 2023-05-22T15:43:19.696 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:19.687+0000 7f18f97fa640 1 -- 172.21.15.16:0/2970932193 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18f4005310 con 0x7f18fc159bc0 2023-05-22T15:43:19.696 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:19.687+0000 7f1902c94640 1 -- 172.21.15.16:0/2970932193 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18fc0c9570 con 0x7f18fc159bc0 2023-05-22T15:43:19.697 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:19.687+0000 7f1902c94640 1 -- 172.21.15.16:0/2970932193 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18fc0ccd90 con 0x7f18fc159bc0 2023-05-22T15:43:19.697 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:19.687+0000 7f18f97fa640 1 -- 172.21.15.16:0/2970932193 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 41) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f18f4005680 con 0x7f18fc159bc0 2023-05-22T15:43:19.698 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:19.687+0000 7f18f97fa640 1 --2- 172.21.15.16:0/2970932193 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f18e80654a0 0x7f18e8067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:19.698 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:19.687+0000 7f18f97fa640 1 -- 172.21.15.16:0/2970932193 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 7113+0+0 (secure 0 0 0) 0x7f18f40835e0 con 0x7f18fc159bc0 2023-05-22T15:43:19.698 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:19.687+0000 7f1900a30640 1 --2- 172.21.15.16:0/2970932193 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f18e80654a0 0x7f18e8067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:19.698 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:19.691+0000 7f1900a30640 1 --2- 172.21.15.16:0/2970932193 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f18e80654a0 0x7f18e8067950 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f18fc15c480 tx=0x7f18ec008040 comp rx=0 tx=0).ready entity=mgr.7788 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:43:19.698 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:19.691+0000 7f18f97fa640 1 -- 172.21.15.16:0/2970932193 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f18f4050e10 con 0x7f18fc159bc0 2023-05-22T15:43:19.851 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:19.847+0000 7f1902c94640 1 -- 172.21.15.16:0/2970932193 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f18fc0c4ab0 con 0x7f18fc159bc0 2023-05-22T15:43:20.551 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:20.547+0000 7f18f97fa640 1 -- 172.21.15.16:0/2970932193 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v108) v1 ==== 181+0+0 (secure 0 0 0) 0x7f18fc0c4ab0 con 0x7f18fc159bc0 2023-05-22T15:43:20.551 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'cephfs_metadata' removed 2023-05-22T15:43:20.559 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:20.555+0000 7f1902c94640 1 -- 172.21.15.16:0/2970932193 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f18e80654a0 msgr2=0x7f18e8067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:20.560 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:20.555+0000 7f1902c94640 1 --2- 172.21.15.16:0/2970932193 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f18e80654a0 0x7f18e8067950 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f18fc15c480 tx=0x7f18ec008040 comp rx=0 tx=0).stop 2023-05-22T15:43:20.560 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:20.555+0000 7f1902c94640 1 -- 172.21.15.16:0/2970932193 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f18fc159bc0 msgr2=0x7f18fc0cad60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:20.560 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:20.555+0000 7f1902c94640 1 --2- 172.21.15.16:0/2970932193 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f18fc159bc0 0x7f18fc0cad60 secure :-1 s=READY pgs=1401 cs=0 l=1 rev1=1 crypto rx=0x7f18f400c730 tx=0x7f18f400cbf0 comp rx=0 tx=0).stop 2023-05-22T15:43:20.560 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:20.555+0000 7f1902c94640 1 -- 172.21.15.16:0/2970932193 shutdown_connections 2023-05-22T15:43:20.560 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:20.555+0000 7f1902c94640 1 --2- 172.21.15.16:0/2970932193 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f18fc159bc0 0x7f18fc0cad60 unknown :-1 s=CLOSED pgs=1401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:20.561 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:20.555+0000 7f1902c94640 1 --2- 172.21.15.16:0/2970932193 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f18fc0ce900 0x7f18fc155ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:20.561 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:20.555+0000 7f1902c94640 1 --2- 172.21.15.16:0/2970932193 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f18e80654a0 0x7f18e8067950 secure :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f18fc15c480 tx=0x7f18ec008040 comp rx=0 tx=0).stop 2023-05-22T15:43:20.561 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:20.555+0000 7f1902c94640 1 --2- 172.21.15.16:0/2970932193 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f18fc0ce530 0x7f18fc14a740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:20.561 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:20.555+0000 7f1902c94640 1 -- 172.21.15.16:0/2970932193 >> 172.21.15.16:0/2970932193 conn(0x7f18fc069720 msgr2=0x7f18fc0cc680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:43:20.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:20.555+0000 7f1902c94640 1 -- 172.21.15.16:0/2970932193 shutdown_connections 2023-05-22T15:43:20.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:20.559+0000 7f1902c94640 1 -- 172.21.15.16:0/2970932193 wait complete. 2023-05-22T15:43:20.589 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-05-22T15:43:20.726 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:20.723+0000 7f77dce79640 1 Processor -- start 2023-05-22T15:43:20.727 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:20.723+0000 7f77dce79640 1 -- start start 2023-05-22T15:43:20.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:20.723+0000 7f77dce79640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f77d815d610 0x7f77d80cbe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:20.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:20.723+0000 7f77dce79640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f77d80cf570 0x7f77d80cf990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:20.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:20.723+0000 7f77dce79640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77d80cde70 con 0x7f77d8159a30 2023-05-22T15:43:20.729 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:20.723+0000 7f77dce79640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77d80cfed0 con 0x7f77d815d610 2023-05-22T15:43:20.729 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:20.723+0000 7f77dce79640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77d815db20 con 0x7f77d80cf570 2023-05-22T15:43:20.729 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:20.723+0000 7f77cffff640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f77d815d610 0x7f77d80cbe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:20.729 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:20.723+0000 7f77d659c640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f77d8159a30 0x7f77d806e380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:54012/0 (socket says 172.21.15.16:54012) 2023-05-22T15:43:20.729 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:20.723+0000 7f77d659c640 1 -- 172.21.15.16:0/318925080 learned_addr learned my addr 172.21.15.16:0/318925080 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:43:20.729 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:20.723+0000 7f77d6d9d640 1 --2- 172.21.15.16:0/318925080 >> v2:172.21.15.62:3300/0 conn(0x7f77d80cf570 0x7f77d80cf990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:20.730 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:20.723+0000 7f77cffff640 1 -- 172.21.15.16:0/318925080 >> v2:172.21.15.62:3300/0 conn(0x7f77d80cf570 msgr2=0x7f77d80cf990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:43:20.730 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:20.723+0000 7f77cffff640 1 --2- 172.21.15.16:0/318925080 >> v2:172.21.15.62:3300/0 conn(0x7f77d80cf570 0x7f77d80cf990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:20.730 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:20.723+0000 7f77cffff640 1 -- 172.21.15.16:0/318925080 >> v1:172.21.15.16:6789/0 conn(0x7f77d8159a30 legacy=0x7f77d806e380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:20.730 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:20.723+0000 7f77cffff640 1 -- 172.21.15.16:0/318925080 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77d815dde0 con 0x7f77d815d610 2023-05-22T15:43:20.732 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:20.723+0000 7f77cffff640 1 --2- 172.21.15.16:0/318925080 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f77d815d610 0x7f77d80cbe10 secure :-1 s=READY pgs=1403 cs=0 l=1 rev1=1 crypto rx=0x7f77c00097c0 tx=0x7f77c001c410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94169b63e00f7a5a server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:43:20.732 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:20.727+0000 7f77d5d9b640 1 -- 172.21.15.16:0/318925080 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77c00ab610 con 0x7f77d815d610 2023-05-22T15:43:20.733 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:20.727+0000 7f77d5d9b640 1 -- 172.21.15.16:0/318925080 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77c00ab770 con 0x7f77d815d610 2023-05-22T15:43:20.733 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:20.727+0000 7f77d5d9b640 1 -- 172.21.15.16:0/318925080 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77c00abb00 con 0x7f77d815d610 2023-05-22T15:43:20.733 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:20.727+0000 7f77dce79640 1 -- 172.21.15.16:0/318925080 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f77d815d610 msgr2=0x7f77d80cbe10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:20.733 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:20.727+0000 7f77dce79640 1 --2- 172.21.15.16:0/318925080 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f77d815d610 0x7f77d80cbe10 secure :-1 s=READY pgs=1403 cs=0 l=1 rev1=1 crypto rx=0x7f77c00097c0 tx=0x7f77c001c410 comp rx=0 tx=0).stop 2023-05-22T15:43:20.733 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:20.727+0000 7f77dce79640 1 -- 172.21.15.16:0/318925080 shutdown_connections 2023-05-22T15:43:20.733 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:20.727+0000 7f77dce79640 1 --2- 172.21.15.16:0/318925080 >> v2:172.21.15.62:3300/0 conn(0x7f77d80cf570 0x7f77d80cf990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:20.734 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:20.727+0000 7f77dce79640 1 --2- 172.21.15.16:0/318925080 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f77d815d610 0x7f77d80cbe10 unknown :-1 s=CLOSED pgs=1403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:20.734 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:20.727+0000 7f77dce79640 1 -- 172.21.15.16:0/318925080 >> 172.21.15.16:0/318925080 conn(0x7f77d80700b0 msgr2=0x7f77d806d9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:43:20.734 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:20.727+0000 7f77dce79640 1 -- 172.21.15.16:0/318925080 shutdown_connections 2023-05-22T15:43:20.734 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:20.727+0000 7f77dce79640 1 -- 172.21.15.16:0/318925080 wait complete. 2023-05-22T15:43:20.734 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:20.727+0000 7f77dce79640 1 Processor -- start 2023-05-22T15:43:20.734 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:20.727+0000 7f77dce79640 1 -- start start 2023-05-22T15:43:20.736 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:20.727+0000 7f77dce79640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f77d80cf570 0x7f77d814c760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:20.736 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:20.727+0000 7f77dce79640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f77d8159a30 0x7f77d8137cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:20.736 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:20.727+0000 7f77dce79640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f77d815d610 0x7f77d814d050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:20.737 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:20.727+0000 7f77dce79640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77d80c6b90 con 0x7f77d80cf570 2023-05-22T15:43:20.737 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:20.727+0000 7f77dce79640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77d8139380 con 0x7f77d815d610 2023-05-22T15:43:20.737 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:20.727+0000 7f77dce79640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77d8138770 con 0x7f77d8159a30 2023-05-22T15:43:20.737 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:20.727+0000 7f77d659c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f77d80cf570 0x7f77d814c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:20.737 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:20.727+0000 7f77d659c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f77d80cf570 0x7f77d814c760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57710/0 (socket says 172.21.15.16:57710) 2023-05-22T15:43:20.737 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:20.727+0000 7f77d659c640 1 -- 172.21.15.16:0/2782805562 learned_addr learned my addr 172.21.15.16:0/2782805562 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:43:20.737 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:20.727+0000 7f77d6d9d640 1 --2- 172.21.15.16:0/2782805562 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f77d815d610 0x7f77d814d050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:20.738 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:20.727+0000 7f77cffff640 1 --2- 172.21.15.16:0/2782805562 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f77d8159a30 0x7f77d8137cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:20.738 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:20.727+0000 7f77d659c640 1 -- 172.21.15.16:0/2782805562 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f77d8159a30 msgr2=0x7f77d8137cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:20.738 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:20.727+0000 7f77d659c640 1 --2- 172.21.15.16:0/2782805562 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f77d8159a30 0x7f77d8137cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:20.738 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:20.727+0000 7f77d659c640 1 -- 172.21.15.16:0/2782805562 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f77d815d610 msgr2=0x7f77d814d050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:20.738 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:20.727+0000 7f77d659c640 1 --2- 172.21.15.16:0/2782805562 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f77d815d610 0x7f77d814d050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:20.738 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:20.727+0000 7f77d659c640 1 -- 172.21.15.16:0/2782805562 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77c0009420 con 0x7f77d80cf570 2023-05-22T15:43:20.738 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:20.727+0000 7f77d6d9d640 1 --2- 172.21.15.16:0/2782805562 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f77d815d610 0x7f77d814d050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:43:20.739 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:20.727+0000 7f77cffff640 1 --2- 172.21.15.16:0/2782805562 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f77d8159a30 0x7f77d8137cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:43:20.739 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:20.727+0000 7f77d659c640 1 --2- 172.21.15.16:0/2782805562 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f77d80cf570 0x7f77d814c760 secure :-1 s=READY pgs=1404 cs=0 l=1 rev1=1 crypto rx=0x7f77c800c530 tx=0x7f77c800c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:43:20.739 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:20.727+0000 7f77ceffd640 1 -- 172.21.15.16:0/2782805562 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77c8005cf0 con 0x7f77d80cf570 2023-05-22T15:43:20.739 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:20.727+0000 7f77ceffd640 1 -- 172.21.15.16:0/2782805562 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77c8005e50 con 0x7f77d80cf570 2023-05-22T15:43:20.739 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:20.727+0000 7f77dce79640 1 -- 172.21.15.16:0/2782805562 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77d80cc350 con 0x7f77d80cf570 2023-05-22T15:43:20.739 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:20.727+0000 7f77ceffd640 1 -- 172.21.15.16:0/2782805562 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77c8007270 con 0x7f77d80cf570 2023-05-22T15:43:20.739 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:20.727+0000 7f77dce79640 1 -- 172.21.15.16:0/2782805562 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77d8159e00 con 0x7f77d80cf570 2023-05-22T15:43:20.740 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:20.731+0000 7f77dce79640 1 -- 172.21.15.16:0/2782805562 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77d80cc4e0 con 0x7f77d80cf570 2023-05-22T15:43:20.740 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:20.731+0000 7f77ceffd640 1 -- 172.21.15.16:0/2782805562 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 41) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f77c80073d0 con 0x7f77d80cf570 2023-05-22T15:43:20.740 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:20.731+0000 7f77ceffd640 1 --2- 172.21.15.16:0/2782805562 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f77b80654a0 0x7f77b8067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:20.740 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:20.731+0000 7f77ceffd640 1 -- 172.21.15.16:0/2782805562 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 6675+0+0 (secure 0 0 0) 0x7f77c8082140 con 0x7f77d80cf570 2023-05-22T15:43:20.740 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:20.731+0000 7f77cffff640 1 --2- 172.21.15.16:0/2782805562 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f77b80654a0 0x7f77b8067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:20.741 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:20.731+0000 7f77cffff640 1 --2- 172.21.15.16:0/2782805562 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f77b80654a0 0x7f77b8067950 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f77c00af000 tx=0x7f77c00b6040 comp rx=0 tx=0).ready entity=mgr.7788 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:43:20.741 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:20.735+0000 7f77ceffd640 1 -- 172.21.15.16:0/2782805562 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f77c804a620 con 0x7f77d80cf570 2023-05-22T15:43:20.896 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:20.891+0000 7f77dce79640 1 -- 172.21.15.16:0/2782805562 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f77d8066910 con 0x7f77d80cf570 2023-05-22T15:43:21.557 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:21.551+0000 7f77ceffd640 1 -- 172.21.15.16:0/2782805562 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v109) v1 ==== 169+0+0 (secure 0 0 0) 0x7f77c804f9f0 con 0x7f77d80cf570 2023-05-22T15:43:21.557 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'cephfs_data' removed 2023-05-22T15:43:21.561 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:21.555+0000 7f77dce79640 1 -- 172.21.15.16:0/2782805562 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f77b80654a0 msgr2=0x7f77b8067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:21.561 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:21.555+0000 7f77dce79640 1 --2- 172.21.15.16:0/2782805562 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f77b80654a0 0x7f77b8067950 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f77c00af000 tx=0x7f77c00b6040 comp rx=0 tx=0).stop 2023-05-22T15:43:21.561 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:21.555+0000 7f77dce79640 1 -- 172.21.15.16:0/2782805562 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f77d80cf570 msgr2=0x7f77d814c760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:21.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:21.555+0000 7f77dce79640 1 --2- 172.21.15.16:0/2782805562 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f77d80cf570 0x7f77d814c760 secure :-1 s=READY pgs=1404 cs=0 l=1 rev1=1 crypto rx=0x7f77c800c530 tx=0x7f77c800c9f0 comp rx=0 tx=0).stop 2023-05-22T15:43:21.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:21.555+0000 7f77dce79640 1 -- 172.21.15.16:0/2782805562 shutdown_connections 2023-05-22T15:43:21.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:21.555+0000 7f77dce79640 1 --2- 172.21.15.16:0/2782805562 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f77d815d610 0x7f77d814d050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:21.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:21.555+0000 7f77dce79640 1 --2- 172.21.15.16:0/2782805562 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f77b80654a0 0x7f77b8067950 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:21.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:21.555+0000 7f77dce79640 1 --2- 172.21.15.16:0/2782805562 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f77d8159a30 0x7f77d8137cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:21.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:21.555+0000 7f77dce79640 1 --2- 172.21.15.16:0/2782805562 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f77d80cf570 0x7f77d814c760 unknown :-1 s=CLOSED pgs=1404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:21.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:21.555+0000 7f77dce79640 1 -- 172.21.15.16:0/2782805562 >> 172.21.15.16:0/2782805562 conn(0x7f77d80700b0 msgr2=0x7f77d815c480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:43:21.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:21.555+0000 7f77dce79640 1 -- 172.21.15.16:0/2782805562 shutdown_connections 2023-05-22T15:43:21.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:21.559+0000 7f77dce79640 1 -- 172.21.15.16:0/2782805562 wait complete. 2023-05-22T15:43:21.590 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2023-05-22T15:43:21.724 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:21.719+0000 7fd8297c1640 1 Processor -- start 2023-05-22T15:43:21.725 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:21.719+0000 7fd8297c1640 1 -- start start 2023-05-22T15:43:21.725 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:21.719+0000 7fd8297c1640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd824070c70 0x7fd824071530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:21.726 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:21.719+0000 7fd8297c1640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd82406b6c0 0x7fd82406bb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:21.726 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:21.719+0000 7fd8297c1640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd82415d520 con 0x7fd82415c580 2023-05-22T15:43:21.726 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:21.719+0000 7fd8297c1640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd82415ca00 con 0x7fd824070c70 2023-05-22T15:43:21.726 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:21.719+0000 7fd8297c1640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8240c8b10 con 0x7fd82406b6c0 2023-05-22T15:43:21.727 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:21.719+0000 7fd822ffd640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd824070c70 0x7fd824071530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:21.727 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:21.719+0000 7fd822ffd640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd824070c70 0x7fd824071530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57716/0 (socket says 172.21.15.16:57716) 2023-05-22T15:43:21.727 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:21.719+0000 7fd822ffd640 1 -- 172.21.15.16:0/1301431453 learned_addr learned my addr 172.21.15.16:0/1301431453 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:43:21.727 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:21.723+0000 7fd8237fe640 1 --2- 172.21.15.16:0/1301431453 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd82406b6c0 0x7fd82406bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:21.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:21.723+0000 7fd822ffd640 1 -- 172.21.15.16:0/1301431453 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd82406b6c0 msgr2=0x7fd82406bb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:43:21.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:21.723+0000 7fd822ffd640 1 --2- 172.21.15.16:0/1301431453 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd82406b6c0 0x7fd82406bb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:21.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:21.723+0000 7fd822ffd640 1 -- 172.21.15.16:0/1301431453 >> v1:172.21.15.62:6789/0 conn(0x7fd82415c580 legacy=0x7fd82406fba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:21.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:21.723+0000 7fd822ffd640 1 -- 172.21.15.16:0/1301431453 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd82406c060 con 0x7fd824070c70 2023-05-22T15:43:21.729 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:21.723+0000 7fd822ffd640 1 --2- 172.21.15.16:0/1301431453 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd824070c70 0x7fd824071530 secure :-1 s=READY pgs=1405 cs=0 l=1 rev1=1 crypto rx=0x7fd814009e50 tx=0x7fd81401c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d460c761ea0432aa server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:43:21.729 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:21.723+0000 7fd821ffb640 1 -- 172.21.15.16:0/1301431453 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8140a5810 con 0x7fd824070c70 2023-05-22T15:43:21.729 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:21.723+0000 7fd821ffb640 1 -- 172.21.15.16:0/1301431453 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8140a5970 con 0x7fd824070c70 2023-05-22T15:43:21.729 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:21.723+0000 7fd821ffb640 1 -- 172.21.15.16:0/1301431453 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8140a5d00 con 0x7fd824070c70 2023-05-22T15:43:21.730 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:21.723+0000 7fd8297c1640 1 -- 172.21.15.16:0/1301431453 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd824070c70 msgr2=0x7fd824071530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:21.730 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:21.723+0000 7fd8297c1640 1 --2- 172.21.15.16:0/1301431453 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd824070c70 0x7fd824071530 secure :-1 s=READY pgs=1405 cs=0 l=1 rev1=1 crypto rx=0x7fd814009e50 tx=0x7fd81401c950 comp rx=0 tx=0).stop 2023-05-22T15:43:21.730 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:21.723+0000 7fd8297c1640 1 -- 172.21.15.16:0/1301431453 shutdown_connections 2023-05-22T15:43:21.731 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:21.723+0000 7fd8297c1640 1 --2- 172.21.15.16:0/1301431453 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd82406b6c0 0x7fd82406bb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:21.731 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:21.723+0000 7fd8297c1640 1 --2- 172.21.15.16:0/1301431453 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd824070c70 0x7fd824071530 unknown :-1 s=CLOSED pgs=1405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:21.731 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:21.723+0000 7fd8297c1640 1 -- 172.21.15.16:0/1301431453 >> 172.21.15.16:0/1301431453 conn(0x7fd824069720 msgr2=0x7fd824069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:43:21.731 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:21.723+0000 7fd8297c1640 1 -- 172.21.15.16:0/1301431453 shutdown_connections 2023-05-22T15:43:21.731 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:21.723+0000 7fd8297c1640 1 -- 172.21.15.16:0/1301431453 wait complete. 2023-05-22T15:43:21.733 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:21.727+0000 7fd8297c1640 1 Processor -- start 2023-05-22T15:43:21.733 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:21.727+0000 7fd8297c1640 1 -- start start 2023-05-22T15:43:21.733 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:21.727+0000 7fd8297c1640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd82406b6c0 0x7fd8241592e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:21.733 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:21.727+0000 7fd8297c1640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd824070c70 0x7fd824154a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:21.734 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:21.727+0000 7fd8297c1640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd82415c580 0x7fd824154f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:21.734 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:21.727+0000 7fd8297c1640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8241581f0 con 0x7fd824070c70 2023-05-22T15:43:21.734 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:21.727+0000 7fd8297c1640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8240c7c90 con 0x7fd82406b6c0 2023-05-22T15:43:21.734 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:21.727+0000 7fd8297c1640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8240c7f70 con 0x7fd82415c580 2023-05-22T15:43:21.734 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:21.727+0000 7fd8227fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd824070c70 0x7fd824154a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:21.734 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:21.727+0000 7fd8227fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd824070c70 0x7fd824154a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57730/0 (socket says 172.21.15.16:57730) 2023-05-22T15:43:21.734 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:21.727+0000 7fd8227fc640 1 -- 172.21.15.16:0/1772067673 learned_addr learned my addr 172.21.15.16:0/1772067673 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:43:21.735 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:21.727+0000 7fd822ffd640 1 --2- 172.21.15.16:0/1772067673 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd82406b6c0 0x7fd8241592e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:21.735 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:21.727+0000 7fd8237fe640 1 --2- 172.21.15.16:0/1772067673 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd82415c580 0x7fd824154f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:21.735 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:21.727+0000 7fd8227fc640 1 -- 172.21.15.16:0/1772067673 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd82415c580 msgr2=0x7fd824154f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:21.735 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:21.727+0000 7fd8227fc640 1 --2- 172.21.15.16:0/1772067673 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd82415c580 0x7fd824154f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:21.735 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:21.727+0000 7fd8227fc640 1 -- 172.21.15.16:0/1772067673 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd82406b6c0 msgr2=0x7fd8241592e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:21.735 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:21.727+0000 7fd8227fc640 1 --2- 172.21.15.16:0/1772067673 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd82406b6c0 0x7fd8241592e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:21.735 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:21.727+0000 7fd8227fc640 1 -- 172.21.15.16:0/1772067673 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd814009b30 con 0x7fd824070c70 2023-05-22T15:43:21.736 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:21.727+0000 7fd822ffd640 1 --2- 172.21.15.16:0/1772067673 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd82406b6c0 0x7fd8241592e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T15:43:21.736 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:21.727+0000 7fd8227fc640 1 --2- 172.21.15.16:0/1772067673 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd824070c70 0x7fd824154a10 secure :-1 s=READY pgs=1406 cs=0 l=1 rev1=1 crypto rx=0x7fd81800c500 tx=0x7fd81800c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:43:21.736 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:21.727+0000 7fd80b7fe640 1 -- 172.21.15.16:0/1772067673 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd818005220 con 0x7fd824070c70 2023-05-22T15:43:21.736 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:21.727+0000 7fd80b7fe640 1 -- 172.21.15.16:0/1772067673 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd818005380 con 0x7fd824070c70 2023-05-22T15:43:21.736 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:21.727+0000 7fd8297c1640 1 -- 172.21.15.16:0/1772067673 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8240cc510 con 0x7fd824070c70 2023-05-22T15:43:21.737 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:21.727+0000 7fd80b7fe640 1 -- 172.21.15.16:0/1772067673 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd818007270 con 0x7fd824070c70 2023-05-22T15:43:21.737 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:21.727+0000 7fd8297c1640 1 -- 172.21.15.16:0/1772067673 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd82415d240 con 0x7fd824070c70 2023-05-22T15:43:21.737 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:21.727+0000 7fd8297c1640 1 -- 172.21.15.16:0/1772067673 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd824071040 con 0x7fd824070c70 2023-05-22T15:43:21.737 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:21.727+0000 7fd80b7fe640 1 -- 172.21.15.16:0/1772067673 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 41) v1 ==== 84332+0+0 (secure 0 0 0) 0x7fd8180073d0 con 0x7fd824070c70 2023-05-22T15:43:21.738 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:21.731+0000 7fd80b7fe640 1 --2- 172.21.15.16:0/1772067673 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd8100654a0 0x7fd810067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:21.738 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:21.731+0000 7fd80b7fe640 1 -- 172.21.15.16:0/1772067673 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 6261+0+0 (secure 0 0 0) 0x7fd8180823a0 con 0x7fd824070c70 2023-05-22T15:43:21.738 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:21.731+0000 7fd822ffd640 1 --2- 172.21.15.16:0/1772067673 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd8100654a0 0x7fd810067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:21.738 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:21.731+0000 7fd822ffd640 1 --2- 172.21.15.16:0/1772067673 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd8100654a0 0x7fd810067950 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fd8140ae000 tx=0x7fd8140a7070 comp rx=0 tx=0).ready entity=mgr.7788 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:43:21.739 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:21.735+0000 7fd80b7fe640 1 -- 172.21.15.16:0/1772067673 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fd81804fe50 con 0x7fd824070c70 2023-05-22T15:43:21.895 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:21.891+0000 7fd8297c1640 1 -- 172.21.15.16:0/1772067673 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7fd824070840 con 0x7fd824070c70 2023-05-22T15:43:21.902 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:21.895+0000 7fd80b7fe640 1 -- 172.21.15.16:0/1772067673 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v31) v1 ==== 120+0+0 (secure 0 0 0) 0x7fd818053470 con 0x7fd824070c70 2023-05-22T15:43:21.904 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:21.899+0000 7fd8297c1640 1 -- 172.21.15.16:0/1772067673 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd8100654a0 msgr2=0x7fd810067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:21.905 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:21.899+0000 7fd8297c1640 1 --2- 172.21.15.16:0/1772067673 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd8100654a0 0x7fd810067950 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fd8140ae000 tx=0x7fd8140a7070 comp rx=0 tx=0).stop 2023-05-22T15:43:21.905 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:21.899+0000 7fd8297c1640 1 -- 172.21.15.16:0/1772067673 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd824070c70 msgr2=0x7fd824154a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:21.905 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:21.899+0000 7fd8297c1640 1 --2- 172.21.15.16:0/1772067673 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd824070c70 0x7fd824154a10 secure :-1 s=READY pgs=1406 cs=0 l=1 rev1=1 crypto rx=0x7fd81800c500 tx=0x7fd81800c9c0 comp rx=0 tx=0).stop 2023-05-22T15:43:21.905 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:21.899+0000 7fd8297c1640 1 -- 172.21.15.16:0/1772067673 shutdown_connections 2023-05-22T15:43:21.906 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:21.899+0000 7fd8297c1640 1 --2- 172.21.15.16:0/1772067673 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd8100654a0 0x7fd810067950 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:21.906 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:21.899+0000 7fd8297c1640 1 --2- 172.21.15.16:0/1772067673 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd82415c580 0x7fd824154f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:21.907 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:21.899+0000 7fd8297c1640 1 --2- 172.21.15.16:0/1772067673 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd824070c70 0x7fd824154a10 unknown :-1 s=CLOSED pgs=1406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:21.907 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:21.899+0000 7fd8297c1640 1 --2- 172.21.15.16:0/1772067673 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd82406b6c0 0x7fd8241592e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:21.907 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:21.899+0000 7fd8297c1640 1 -- 172.21.15.16:0/1772067673 >> 172.21.15.16:0/1772067673 conn(0x7fd824069720 msgr2=0x7fd824069af0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:43:21.907 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:21.899+0000 7fd8297c1640 1 -- 172.21.15.16:0/1772067673 shutdown_connections 2023-05-22T15:43:21.908 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:21.899+0000 7fd8297c1640 1 -- 172.21.15.16:0/1772067673 wait complete. 2023-05-22T15:43:21.922 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2023-05-22T15:43:22.045 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.039+0000 7efde6b7f640 1 Processor -- start 2023-05-22T15:43:22.045 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.039+0000 7efde6b7f640 1 -- start start 2023-05-22T15:43:22.046 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.039+0000 7efde6b7f640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7efde00cb4c0 0x7efde0158fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:22.046 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.039+0000 7efde6b7f640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7efde0159510 0x7efde006fba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:22.046 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.039+0000 7efde6b7f640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efde0159f70 con 0x7efde006b6c0 2023-05-22T15:43:22.047 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.039+0000 7efde6b7f640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efde0071760 con 0x7efde0159510 2023-05-22T15:43:22.047 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.039+0000 7efde6b7f640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efde00cb910 con 0x7efde00cb4c0 2023-05-22T15:43:22.048 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.043+0000 7efddffff640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7efde00cb4c0 0x7efde0158fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:22.048 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.043+0000 7efde511c640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7efde0159510 0x7efde006fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:22.048 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.043+0000 7efddffff640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7efde00cb4c0 0x7efde0158fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3301/0 says I am v2:172.21.15.16:33430/0 (socket says 172.21.15.16:33430) 2023-05-22T15:43:22.048 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.043+0000 7efddffff640 1 -- 172.21.15.16:0/629050593 learned_addr learned my addr 172.21.15.16:0/629050593 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:43:22.048 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.043+0000 7efddffff640 1 -- 172.21.15.16:0/629050593 >> v2:172.21.15.62:3300/0 conn(0x7efde0159510 msgr2=0x7efde006fba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:43:22.049 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.043+0000 7efddffff640 1 --2- 172.21.15.16:0/629050593 >> v2:172.21.15.62:3300/0 conn(0x7efde0159510 0x7efde006fba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:22.049 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.043+0000 7efddffff640 1 -- 172.21.15.16:0/629050593 >> v1:172.21.15.62:6789/0 conn(0x7efde006b6c0 legacy=0x7efde006ba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:22.049 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.043+0000 7efddffff640 1 -- 172.21.15.16:0/629050593 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efde00cb030 con 0x7efde00cb4c0 2023-05-22T15:43:22.050 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.043+0000 7efddffff640 1 --2- 172.21.15.16:0/629050593 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7efde00cb4c0 0x7efde0158fd0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7efdd4009870 tx=0x7efdd401c540 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e885e9ffbd772300 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:43:22.050 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.047+0000 7efddf7fe640 1 -- 172.21.15.16:0/629050593 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdd40a7070 con 0x7efde00cb4c0 2023-05-22T15:43:22.050 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.047+0000 7efddf7fe640 1 -- 172.21.15.16:0/629050593 <== mon.2 v2:172.21.15.62:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efdd40ab430 con 0x7efde00cb4c0 2023-05-22T15:43:22.050 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.047+0000 7efddf7fe640 1 -- 172.21.15.16:0/629050593 <== mon.2 v2:172.21.15.62:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdd40ab7c0 con 0x7efde00cb4c0 2023-05-22T15:43:22.051 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.047+0000 7efde6b7f640 1 -- 172.21.15.16:0/629050593 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7efde00cb4c0 msgr2=0x7efde0158fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:22.051 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.047+0000 7efde6b7f640 1 --2- 172.21.15.16:0/629050593 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7efde00cb4c0 0x7efde0158fd0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7efdd4009870 tx=0x7efdd401c540 comp rx=0 tx=0).stop 2023-05-22T15:43:22.052 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.047+0000 7efde6b7f640 1 -- 172.21.15.16:0/629050593 shutdown_connections 2023-05-22T15:43:22.052 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.047+0000 7efde6b7f640 1 --2- 172.21.15.16:0/629050593 >> v2:172.21.15.62:3300/0 conn(0x7efde0159510 0x7efde006fba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:22.052 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.047+0000 7efde6b7f640 1 --2- 172.21.15.16:0/629050593 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7efde00cb4c0 0x7efde0158fd0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:22.052 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.047+0000 7efde6b7f640 1 -- 172.21.15.16:0/629050593 >> 172.21.15.16:0/629050593 conn(0x7efde0069720 msgr2=0x7efde0069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:43:22.052 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.047+0000 7efde6b7f640 1 -- 172.21.15.16:0/629050593 shutdown_connections 2023-05-22T15:43:22.052 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.047+0000 7efde6b7f640 1 -- 172.21.15.16:0/629050593 wait complete. 2023-05-22T15:43:22.053 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.047+0000 7efde6b7f640 1 Processor -- start 2023-05-22T15:43:22.053 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.047+0000 7efde6b7f640 1 -- start start 2023-05-22T15:43:22.054 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.047+0000 7efde6b7f640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efde006b6c0 0x7efde014b120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:22.054 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.047+0000 7efde6b7f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efde00cb4c0 0x7efde014f320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:22.054 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.047+0000 7efde6b7f640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7efde0159510 0x7efde014f860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:22.054 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.047+0000 7efde6b7f640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efde00b1f20 con 0x7efde00cb4c0 2023-05-22T15:43:22.055 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.047+0000 7efde6b7f640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efde00b12a0 con 0x7efde006b6c0 2023-05-22T15:43:22.055 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.047+0000 7efde6b7f640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efde00b1580 con 0x7efde0159510 2023-05-22T15:43:22.055 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.047+0000 7efddffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efde00cb4c0 0x7efde014f320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:22.055 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.047+0000 7efddffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efde00cb4c0 0x7efde014f320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57738/0 (socket says 172.21.15.16:57738) 2023-05-22T15:43:22.055 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.047+0000 7efddffff640 1 -- 172.21.15.16:0/3102444926 learned_addr learned my addr 172.21.15.16:0/3102444926 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:43:22.055 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.051+0000 7efde491b640 1 --2- 172.21.15.16:0/3102444926 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efde006b6c0 0x7efde014b120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:22.055 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.051+0000 7efde511c640 1 --2- 172.21.15.16:0/3102444926 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7efde0159510 0x7efde014f860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:22.056 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.051+0000 7efddffff640 1 -- 172.21.15.16:0/3102444926 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7efde0159510 msgr2=0x7efde014f860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:22.057 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.051+0000 7efddffff640 1 --2- 172.21.15.16:0/3102444926 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7efde0159510 0x7efde014f860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:22.057 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.051+0000 7efddffff640 1 -- 172.21.15.16:0/3102444926 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efde006b6c0 msgr2=0x7efde014b120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:22.057 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.051+0000 7efddffff640 1 --2- 172.21.15.16:0/3102444926 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efde006b6c0 0x7efde014b120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:22.057 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.051+0000 7efddffff640 1 -- 172.21.15.16:0/3102444926 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdd40094d0 con 0x7efde00cb4c0 2023-05-22T15:43:22.057 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.051+0000 7efde491b640 1 --2- 172.21.15.16:0/3102444926 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efde006b6c0 0x7efde014b120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T15:43:22.057 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.051+0000 7efddffff640 1 --2- 172.21.15.16:0/3102444926 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efde00cb4c0 0x7efde014f320 secure :-1 s=READY pgs=1407 cs=0 l=1 rev1=1 crypto rx=0x7efdd40ad040 tx=0x7efdd40192e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:43:22.058 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.051+0000 7efddd7fa640 1 -- 172.21.15.16:0/3102444926 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdd40bc070 con 0x7efde00cb4c0 2023-05-22T15:43:22.058 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.051+0000 7efddd7fa640 1 -- 172.21.15.16:0/3102444926 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efdd40b43f0 con 0x7efde00cb4c0 2023-05-22T15:43:22.058 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.051+0000 7efddd7fa640 1 -- 172.21.15.16:0/3102444926 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdd40b4780 con 0x7efde00cb4c0 2023-05-22T15:43:22.058 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.051+0000 7efde6b7f640 1 -- 172.21.15.16:0/3102444926 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efde0159910 con 0x7efde00cb4c0 2023-05-22T15:43:22.058 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.051+0000 7efde6b7f640 1 -- 172.21.15.16:0/3102444926 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efde00cacc0 con 0x7efde00cb4c0 2023-05-22T15:43:22.058 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.051+0000 7efde6b7f640 1 -- 172.21.15.16:0/3102444926 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efde0156840 con 0x7efde00cb4c0 2023-05-22T15:43:22.058 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.051+0000 7efddd7fa640 1 -- 172.21.15.16:0/3102444926 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 41) v1 ==== 84332+0+0 (secure 0 0 0) 0x7efdd40b48e0 con 0x7efde00cb4c0 2023-05-22T15:43:22.059 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.051+0000 7efddd7fa640 1 --2- 172.21.15.16:0/3102444926 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7efdcc0654a0 0x7efdcc067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:22.059 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.051+0000 7efddd7fa640 1 -- 172.21.15.16:0/3102444926 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 6261+0+0 (secure 0 0 0) 0x7efdd411c7a0 con 0x7efde00cb4c0 2023-05-22T15:43:22.059 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.051+0000 7efde491b640 1 --2- 172.21.15.16:0/3102444926 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7efdcc0654a0 0x7efdcc067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:22.059 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.051+0000 7efde491b640 1 --2- 172.21.15.16:0/3102444926 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7efdcc0654a0 0x7efdcc067950 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7efdd0001710 tx=0x7efdd0009040 comp rx=0 tx=0).ready entity=mgr.7788 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:43:22.061 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.055+0000 7efddd7fa640 1 -- 172.21.15.16:0/3102444926 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7efdd40ea1c0 con 0x7efde00cb4c0 2023-05-22T15:43:22.151 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-05-22T15:43:22.154 DEBUG:teuthology.orchestra.run.smithi062:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-05-22T15:43:22.217 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.211+0000 7efde6b7f640 1 -- 172.21.15.16:0/3102444926 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7efde00b2060 con 0x7efde00cb4c0 2023-05-22T15:43:22.223 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.219+0000 7efddd7fa640 1 -- 172.21.15.16:0/3102444926 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v32) v1 ==== 104+0+0 (secure 0 0 0) 0x7efde00b2060 con 0x7efde00cb4c0 2023-05-22T15:43:22.225 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.219+0000 7efde6b7f640 1 -- 172.21.15.16:0/3102444926 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7efdcc0654a0 msgr2=0x7efdcc067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:22.225 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.219+0000 7efde6b7f640 1 --2- 172.21.15.16:0/3102444926 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7efdcc0654a0 0x7efdcc067950 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7efdd0001710 tx=0x7efdd0009040 comp rx=0 tx=0).stop 2023-05-22T15:43:22.225 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.219+0000 7efde6b7f640 1 -- 172.21.15.16:0/3102444926 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efde00cb4c0 msgr2=0x7efde014f320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:22.225 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.219+0000 7efde6b7f640 1 --2- 172.21.15.16:0/3102444926 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efde00cb4c0 0x7efde014f320 secure :-1 s=READY pgs=1407 cs=0 l=1 rev1=1 crypto rx=0x7efdd40ad040 tx=0x7efdd40192e0 comp rx=0 tx=0).stop 2023-05-22T15:43:22.226 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.219+0000 7efde6b7f640 1 -- 172.21.15.16:0/3102444926 shutdown_connections 2023-05-22T15:43:22.226 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.219+0000 7efde6b7f640 1 --2- 172.21.15.16:0/3102444926 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7efdcc0654a0 0x7efdcc067950 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:22.226 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.219+0000 7efde6b7f640 1 --2- 172.21.15.16:0/3102444926 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7efde0159510 0x7efde014f860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:22.226 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.219+0000 7efde6b7f640 1 --2- 172.21.15.16:0/3102444926 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efde00cb4c0 0x7efde014f320 unknown :-1 s=CLOSED pgs=1407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:22.226 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.219+0000 7efde6b7f640 1 --2- 172.21.15.16:0/3102444926 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efde006b6c0 0x7efde014b120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:22.226 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.219+0000 7efde6b7f640 1 -- 172.21.15.16:0/3102444926 >> 172.21.15.16:0/3102444926 conn(0x7efde0069720 msgr2=0x7efde00cd030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:43:22.226 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.219+0000 7efde6b7f640 1 -- 172.21.15.16:0/3102444926 shutdown_connections 2023-05-22T15:43:22.226 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.219+0000 7efde6b7f640 1 -- 172.21.15.16:0/3102444926 wait complete. 2023-05-22T15:43:22.242 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2023-05-22T15:43:22.414 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.407+0000 7f65f516b640 1 Processor -- start 2023-05-22T15:43:22.414 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.407+0000 7f65f516b640 1 -- start start 2023-05-22T15:43:22.415 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.411+0000 7f65f516b640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f65f015c350 0x7f65f015c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:22.415 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.411+0000 7f65f516b640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65f00c95f0 con 0x7f65f0069720 2023-05-22T15:43:22.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.411+0000 7f65f516b640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65f00c98d0 con 0x7f65f00ade80 2023-05-22T15:43:22.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.411+0000 7f65f516b640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65f00ca270 con 0x7f65f015c350 2023-05-22T15:43:22.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.411+0000 7f65ee59c640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f65f015c350 0x7f65f015c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:22.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.411+0000 7f65ef59e640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f65f0069720 0x7f65f0069b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:54018/0 (socket says 172.21.15.16:54018) 2023-05-22T15:43:22.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.411+0000 7f65ef59e640 1 -- 172.21.15.16:0/599538135 learned_addr learned my addr 172.21.15.16:0/599538135 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:43:22.417 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.411+0000 7f65ee59c640 1 -- 172.21.15.16:0/599538135 >> v1:172.21.15.62:6789/0 conn(0x7f65f00ade80 legacy=0x7f65f00cc400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:22.417 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.411+0000 7f65ee59c640 1 -- 172.21.15.16:0/599538135 >> v1:172.21.15.16:6789/0 conn(0x7f65f0069720 legacy=0x7f65f0069b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:22.417 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.411+0000 7f65edd9b640 1 -- 172.21.15.16:0/599538135 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3186337566 0 0) 0x7f65f00c95f0 con 0x7f65f0069720 2023-05-22T15:43:22.417 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.411+0000 7f65ee59c640 1 -- 172.21.15.16:0/599538135 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65f00c9bb0 con 0x7f65f015c350 2023-05-22T15:43:22.418 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.415+0000 7f65ee59c640 1 --2- 172.21.15.16:0/599538135 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f65f015c350 0x7f65f015c7f0 secure :-1 s=READY pgs=1409 cs=0 l=1 rev1=1 crypto rx=0x7f65d80097c0 tx=0x7f65d801c3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cbeb92957e1e4160 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:43:22.418 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.415+0000 7f65edd9b640 1 -- 172.21.15.16:0/599538135 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65d80ab600 con 0x7f65f015c350 2023-05-22T15:43:22.419 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.415+0000 7f65edd9b640 1 -- 172.21.15.16:0/599538135 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65d80a7050 con 0x7f65f015c350 2023-05-22T15:43:22.419 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.415+0000 7f65edd9b640 1 -- 172.21.15.16:0/599538135 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65d80ab980 con 0x7f65f015c350 2023-05-22T15:43:22.419 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.415+0000 7f65f516b640 1 -- 172.21.15.16:0/599538135 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f65f015c350 msgr2=0x7f65f015c7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:22.420 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.415+0000 7f65f516b640 1 --2- 172.21.15.16:0/599538135 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f65f015c350 0x7f65f015c7f0 secure :-1 s=READY pgs=1409 cs=0 l=1 rev1=1 crypto rx=0x7f65d80097c0 tx=0x7f65d801c3f0 comp rx=0 tx=0).stop 2023-05-22T15:43:22.420 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.415+0000 7f65f516b640 1 -- 172.21.15.16:0/599538135 shutdown_connections 2023-05-22T15:43:22.420 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.415+0000 7f65f516b640 1 --2- 172.21.15.16:0/599538135 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f65f015c350 0x7f65f015c7f0 unknown :-1 s=CLOSED pgs=1409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:22.420 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.415+0000 7f65f516b640 1 -- 172.21.15.16:0/599538135 >> 172.21.15.16:0/599538135 conn(0x7f65f00b1110 msgr2=0x7f65f00ce830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:43:22.420 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.415+0000 7f65f516b640 1 -- 172.21.15.16:0/599538135 shutdown_connections 2023-05-22T15:43:22.420 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.415+0000 7f65f516b640 1 -- 172.21.15.16:0/599538135 wait complete. 2023-05-22T15:43:22.422 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.415+0000 7f65f516b640 1 Processor -- start 2023-05-22T15:43:22.422 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.415+0000 7f65f516b640 1 -- start start 2023-05-22T15:43:22.422 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.415+0000 7f65f516b640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f65f0069720 0x7f65f014b320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:22.422 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.415+0000 7f65f516b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f65f00ade80 0x7f65f014a720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:22.422 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.415+0000 7f65f516b640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f65f015acd0 0x7f65f0157b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:22.422 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.415+0000 7f65f516b640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65f01576a0 con 0x7f65f00ade80 2023-05-22T15:43:22.422 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.415+0000 7f65f516b640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65f015b620 con 0x7f65f0069720 2023-05-22T15:43:22.423 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.415+0000 7f65f516b640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65f0156ad0 con 0x7f65f015acd0 2023-05-22T15:43:22.423 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.415+0000 7f65ee59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f65f00ade80 0x7f65f014a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:22.423 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.415+0000 7f65ee59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f65f00ade80 0x7f65f014a720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57756/0 (socket says 172.21.15.16:57756) 2023-05-22T15:43:22.423 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.415+0000 7f65ee59c640 1 -- 172.21.15.16:0/854845308 learned_addr learned my addr 172.21.15.16:0/854845308 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:43:22.423 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.415+0000 7f65eed9d640 1 --2- 172.21.15.16:0/854845308 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f65f0069720 0x7f65f014b320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:22.423 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.415+0000 7f65ef59e640 1 --2- 172.21.15.16:0/854845308 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f65f015acd0 0x7f65f0157b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:22.423 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.415+0000 7f65ee59c640 1 -- 172.21.15.16:0/854845308 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f65f015acd0 msgr2=0x7f65f0157b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:22.424 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.415+0000 7f65ee59c640 1 --2- 172.21.15.16:0/854845308 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f65f015acd0 0x7f65f0157b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:22.424 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.415+0000 7f65ee59c640 1 -- 172.21.15.16:0/854845308 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f65f0069720 msgr2=0x7f65f014b320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:22.424 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.415+0000 7f65ee59c640 1 --2- 172.21.15.16:0/854845308 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f65f0069720 0x7f65f014b320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:22.424 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.415+0000 7f65ee59c640 1 -- 172.21.15.16:0/854845308 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65d8009420 con 0x7f65f00ade80 2023-05-22T15:43:22.424 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.415+0000 7f65eed9d640 1 --2- 172.21.15.16:0/854845308 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f65f0069720 0x7f65f014b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T15:43:22.424 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.415+0000 7f65ee59c640 1 --2- 172.21.15.16:0/854845308 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f65f00ade80 0x7f65f014a720 secure :-1 s=READY pgs=1410 cs=0 l=1 rev1=1 crypto rx=0x7f65d80ae040 tx=0x7f65d8019190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:43:22.425 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.419+0000 7f65e77fe640 1 -- 172.21.15.16:0/854845308 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65d8004230 con 0x7f65f00ade80 2023-05-22T15:43:22.425 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.419+0000 7f65e77fe640 1 -- 172.21.15.16:0/854845308 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65d80043b0 con 0x7f65f00ade80 2023-05-22T15:43:22.426 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.419+0000 7f65f516b640 1 -- 172.21.15.16:0/854845308 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65f00c93a0 con 0x7f65f00ade80 2023-05-22T15:43:22.426 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.419+0000 7f65e77fe640 1 -- 172.21.15.16:0/854845308 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65d8003860 con 0x7f65f00ade80 2023-05-22T15:43:22.426 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.419+0000 7f65f516b640 1 -- 172.21.15.16:0/854845308 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65f00c9210 con 0x7f65f00ade80 2023-05-22T15:43:22.426 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.419+0000 7f65f516b640 1 -- 172.21.15.16:0/854845308 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65f0166700 con 0x7f65f00ade80 2023-05-22T15:43:22.426 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.419+0000 7f65e77fe640 1 -- 172.21.15.16:0/854845308 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 41) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f65d80b2040 con 0x7f65f00ade80 2023-05-22T15:43:22.426 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.419+0000 7f65e77fe640 1 --2- 172.21.15.16:0/854845308 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f65d00654a0 0x7f65d0067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:22.426 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.419+0000 7f65e77fe640 1 -- 172.21.15.16:0/854845308 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 6261+0+0 (secure 0 0 0) 0x7f65d811bff0 con 0x7f65f00ade80 2023-05-22T15:43:22.427 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.419+0000 7f65eed9d640 1 --2- 172.21.15.16:0/854845308 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f65d00654a0 0x7f65d0067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:22.427 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.419+0000 7f65eed9d640 1 --2- 172.21.15.16:0/854845308 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f65d00654a0 0x7f65d0067950 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f65e0000f70 tx=0x7f65e0009040 comp rx=0 tx=0).ready entity=mgr.7788 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:43:22.428 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.423+0000 7f65e77fe640 1 -- 172.21.15.16:0/854845308 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f65d80e9940 con 0x7f65f00ade80 2023-05-22T15:43:22.587 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.583+0000 7f65f516b640 1 -- 172.21.15.16:0/854845308 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7f65f0157850 con 0x7f65f00ade80 2023-05-22T15:43:22.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.591+0000 7f65e77fe640 1 -- 172.21.15.16:0/854845308 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v33) v1 ==== 115+0+0 (secure 0 0 0) 0x7f65f0157850 con 0x7f65f00ade80 2023-05-22T15:43:22.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.595+0000 7f65f516b640 1 -- 172.21.15.16:0/854845308 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f65d00654a0 msgr2=0x7f65d0067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:22.600 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.595+0000 7f65f516b640 1 --2- 172.21.15.16:0/854845308 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f65d00654a0 0x7f65d0067950 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f65e0000f70 tx=0x7f65e0009040 comp rx=0 tx=0).stop 2023-05-22T15:43:22.600 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.595+0000 7f65f516b640 1 -- 172.21.15.16:0/854845308 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f65f00ade80 msgr2=0x7f65f014a720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:22.600 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.595+0000 7f65f516b640 1 --2- 172.21.15.16:0/854845308 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f65f00ade80 0x7f65f014a720 secure :-1 s=READY pgs=1410 cs=0 l=1 rev1=1 crypto rx=0x7f65d80ae040 tx=0x7f65d8019190 comp rx=0 tx=0).stop 2023-05-22T15:43:22.600 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.595+0000 7f65f516b640 1 -- 172.21.15.16:0/854845308 shutdown_connections 2023-05-22T15:43:22.601 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.595+0000 7f65f516b640 1 --2- 172.21.15.16:0/854845308 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f65d00654a0 0x7f65d0067950 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:22.601 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.595+0000 7f65f516b640 1 --2- 172.21.15.16:0/854845308 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f65f015acd0 0x7f65f0157b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:22.601 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.595+0000 7f65f516b640 1 --2- 172.21.15.16:0/854845308 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f65f00ade80 0x7f65f014a720 unknown :-1 s=CLOSED pgs=1410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:22.601 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.595+0000 7f65f516b640 1 --2- 172.21.15.16:0/854845308 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f65f0069720 0x7f65f014b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:22.601 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.595+0000 7f65f516b640 1 -- 172.21.15.16:0/854845308 >> 172.21.15.16:0/854845308 conn(0x7f65f00b1110 msgr2=0x7f65f00cb1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:43:22.601 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.595+0000 7f65f516b640 1 -- 172.21.15.16:0/854845308 shutdown_connections 2023-05-22T15:43:22.602 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.595+0000 7f65f516b640 1 -- 172.21.15.16:0/854845308 wait complete. 2023-05-22T15:43:22.618 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_snap_schedule_list' 2023-05-22T15:43:22.735 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.731+0000 7f377810b640 1 Processor -- start 2023-05-22T15:43:22.736 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.731+0000 7f377810b640 1 -- start start 2023-05-22T15:43:22.737 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.731+0000 7f377810b640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f377015c130 0x7f37700ceb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:22.737 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.731+0000 7f377810b640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f37700b24a0 0x7f377006dd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:22.737 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.731+0000 7f377810b640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f377006bfd0 con 0x7f377006e270 2023-05-22T15:43:22.737 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.731+0000 7f377810b640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37700d1170 con 0x7f37700b24a0 2023-05-22T15:43:22.737 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.731+0000 7f377810b640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f377006e6f0 con 0x7f377015c130 2023-05-22T15:43:22.738 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.731+0000 7f37756a6640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f37700b24a0 0x7f377006dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:22.738 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.731+0000 7f37756a6640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f37700b24a0 0x7f377006dd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57764/0 (socket says 172.21.15.16:57764) 2023-05-22T15:43:22.738 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.731+0000 7f37756a6640 1 -- 172.21.15.16:0/1558555425 learned_addr learned my addr 172.21.15.16:0/1558555425 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:43:22.738 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.731+0000 7f3775ea7640 1 --2- 172.21.15.16:0/1558555425 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f377015c130 0x7f37700ceb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:22.739 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.731+0000 7f37756a6640 1 -- 172.21.15.16:0/1558555425 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f377015c130 msgr2=0x7f37700ceb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:43:22.740 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.731+0000 7f37756a6640 1 --2- 172.21.15.16:0/1558555425 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f377015c130 0x7f37700ceb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:22.740 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.731+0000 7f37756a6640 1 -- 172.21.15.16:0/1558555425 >> v1:172.21.15.62:6789/0 conn(0x7f377006e270 legacy=0x7f377015fb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:43:22.741 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.731+0000 7f37756a6640 1 -- 172.21.15.16:0/1558555425 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f377006c830 con 0x7f37700b24a0 2023-05-22T15:43:22.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.735+0000 7f37756a6640 1 --2- 172.21.15.16:0/1558555425 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f37700b24a0 0x7f377006dd30 secure :-1 s=READY pgs=1411 cs=0 l=1 rev1=1 crypto rx=0x7f3760009870 tx=0x7f376001c470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d912d36ee1a65a3d server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:43:22.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.735+0000 7f3774ea5640 1 -- 172.21.15.16:0/1558555425 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37600ab600 con 0x7f37700b24a0 2023-05-22T15:43:22.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.735+0000 7f3774ea5640 1 -- 172.21.15.16:0/1558555425 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37600a7050 con 0x7f37700b24a0 2023-05-22T15:43:22.744 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.735+0000 7f3774ea5640 1 -- 172.21.15.16:0/1558555425 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37600ab980 con 0x7f37700b24a0 2023-05-22T15:43:22.744 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.735+0000 7f377810b640 1 -- 172.21.15.16:0/1558555425 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f37700b24a0 msgr2=0x7f377006dd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:22.744 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.735+0000 7f377810b640 1 --2- 172.21.15.16:0/1558555425 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f37700b24a0 0x7f377006dd30 secure :-1 s=READY pgs=1411 cs=0 l=1 rev1=1 crypto rx=0x7f3760009870 tx=0x7f376001c470 comp rx=0 tx=0).stop 2023-05-22T15:43:22.744 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.735+0000 7f377810b640 1 -- 172.21.15.16:0/1558555425 shutdown_connections 2023-05-22T15:43:22.744 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.735+0000 7f377810b640 1 --2- 172.21.15.16:0/1558555425 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f37700b24a0 0x7f377006dd30 unknown :-1 s=CLOSED pgs=1411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:22.744 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.735+0000 7f377810b640 1 --2- 172.21.15.16:0/1558555425 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f377015c130 0x7f37700ceb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:22.744 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.735+0000 7f377810b640 1 -- 172.21.15.16:0/1558555425 >> 172.21.15.16:0/1558555425 conn(0x7f377006f5c0 msgr2=0x7f377006f9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:43:22.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.735+0000 7f377810b640 1 -- 172.21.15.16:0/1558555425 shutdown_connections 2023-05-22T15:43:22.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.735+0000 7f377810b640 1 -- 172.21.15.16:0/1558555425 wait complete. 2023-05-22T15:43:22.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.735+0000 7f377810b640 1 Processor -- start 2023-05-22T15:43:22.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.735+0000 7f377810b640 1 -- start start 2023-05-22T15:43:22.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.735+0000 7f377810b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f377006e270 0x7f37700b3aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:22.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.735+0000 7f377810b640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f37700b24a0 0x7f37701558c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:22.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.735+0000 7f377810b640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f377015c130 0x7f377015dc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:22.746 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.735+0000 7f377810b640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37700b4d40 con 0x7f377006e270 2023-05-22T15:43:22.746 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.735+0000 7f377810b640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37700b4130 con 0x7f377015c130 2023-05-22T15:43:22.746 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.739+0000 7f377810b640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37700b4410 con 0x7f37700b24a0 2023-05-22T15:43:22.746 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.739+0000 7f3775ea7640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f377006e270 0x7f37700b3aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:22.746 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.739+0000 7f3775ea7640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f377006e270 0x7f37700b3aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57772/0 (socket says 172.21.15.16:57772) 2023-05-22T15:43:22.746 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.739+0000 7f3775ea7640 1 -- 172.21.15.16:0/1143127047 learned_addr learned my addr 172.21.15.16:0/1143127047 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:43:22.746 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.739+0000 7f37766a8640 1 --2- 172.21.15.16:0/1143127047 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f377015c130 0x7f377015dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:22.746 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.739+0000 7f37756a6640 1 --2- 172.21.15.16:0/1143127047 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f37700b24a0 0x7f37701558c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:22.747 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.739+0000 7f3775ea7640 1 -- 172.21.15.16:0/1143127047 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f37700b24a0 msgr2=0x7f37701558c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:22.747 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.739+0000 7f3775ea7640 1 --2- 172.21.15.16:0/1143127047 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f37700b24a0 0x7f37701558c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:22.747 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.739+0000 7f3775ea7640 1 -- 172.21.15.16:0/1143127047 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f377015c130 msgr2=0x7f377015dc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:22.747 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.739+0000 7f3775ea7640 1 --2- 172.21.15.16:0/1143127047 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f377015c130 0x7f377015dc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:22.747 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.739+0000 7f3775ea7640 1 -- 172.21.15.16:0/1143127047 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37600094d0 con 0x7f377006e270 2023-05-22T15:43:22.747 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.739+0000 7f37766a8640 1 --2- 172.21.15.16:0/1143127047 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f377015c130 0x7f377015dc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:43:22.747 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.739+0000 7f37756a6640 1 --2- 172.21.15.16:0/1143127047 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f37700b24a0 0x7f37701558c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:43:22.748 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.739+0000 7f3775ea7640 1 --2- 172.21.15.16:0/1143127047 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f377006e270 0x7f37700b3aa0 secure :-1 s=READY pgs=1412 cs=0 l=1 rev1=1 crypto rx=0x7f376c007e40 tx=0x7f376c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:43:22.748 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.739+0000 7f37667fc640 1 -- 172.21.15.16:0/1143127047 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f376c014620 con 0x7f377006e270 2023-05-22T15:43:22.748 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.739+0000 7f37667fc640 1 -- 172.21.15.16:0/1143127047 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f376c00ee90 con 0x7f377006e270 2023-05-22T15:43:22.748 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.739+0000 7f377810b640 1 -- 172.21.15.16:0/1143127047 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37700ca9e0 con 0x7f377006e270 2023-05-22T15:43:22.748 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.739+0000 7f37667fc640 1 -- 172.21.15.16:0/1143127047 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f376c0149c0 con 0x7f377006e270 2023-05-22T15:43:22.748 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.739+0000 7f377810b640 1 -- 172.21.15.16:0/1143127047 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f377006c550 con 0x7f377006e270 2023-05-22T15:43:22.749 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.739+0000 7f377810b640 1 -- 172.21.15.16:0/1143127047 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37700cacc0 con 0x7f377006e270 2023-05-22T15:43:22.749 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.739+0000 7f37667fc640 1 -- 172.21.15.16:0/1143127047 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 41) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f376c014b20 con 0x7f377006e270 2023-05-22T15:43:22.750 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.739+0000 7f37667fc640 1 --2- 172.21.15.16:0/1143127047 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f37540654a0 0x7f3754067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:22.750 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.739+0000 7f37667fc640 1 -- 172.21.15.16:0/1143127047 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 6261+0+0 (secure 0 0 0) 0x7f376c0834e0 con 0x7f377006e270 2023-05-22T15:43:22.750 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.739+0000 7f37756a6640 1 --2- 172.21.15.16:0/1143127047 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f37540654a0 0x7f3754067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:22.750 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.739+0000 7f37756a6640 1 --2- 172.21.15.16:0/1143127047 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f37540654a0 0x7f3754067950 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f37600ad000 tx=0x7f37600a6000 comp rx=0 tx=0).ready entity=mgr.7788 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:43:22.750 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.743+0000 7f37667fc640 1 -- 172.21.15.16:0/1143127047 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f376c050ee0 con 0x7f377006e270 2023-05-22T15:43:22.957 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:22.951+0000 7f377810b640 1 -- 172.21.15.16:0/1143127047 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_snap_schedule_list"]} v 0) v1 -- 0x7f377006ff10 con 0x7f377006e270 2023-05-22T15:43:23.576 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:23.567+0000 7f37667fc640 1 -- 172.21.15.16:0/1143127047 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_snap_schedule_list"]}]=0 v623) v1 ==== 166+0+0 (secure 0 0 0) 0x7f376c054500 con 0x7f377006e270 2023-05-22T15:43:23.578 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:23.571+0000 7f377810b640 1 -- 172.21.15.16:0/1143127047 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f37540654a0 msgr2=0x7f3754067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:23.578 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:23.571+0000 7f377810b640 1 --2- 172.21.15.16:0/1143127047 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f37540654a0 0x7f3754067950 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f37600ad000 tx=0x7f37600a6000 comp rx=0 tx=0).stop 2023-05-22T15:43:23.578 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:23.571+0000 7f377810b640 1 -- 172.21.15.16:0/1143127047 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f377006e270 msgr2=0x7f37700b3aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:23.579 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:23.571+0000 7f377810b640 1 --2- 172.21.15.16:0/1143127047 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f377006e270 0x7f37700b3aa0 secure :-1 s=READY pgs=1412 cs=0 l=1 rev1=1 crypto rx=0x7f376c007e40 tx=0x7f376c00e4b0 comp rx=0 tx=0).stop 2023-05-22T15:43:23.579 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:23.571+0000 7f377810b640 1 -- 172.21.15.16:0/1143127047 shutdown_connections 2023-05-22T15:43:23.579 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:23.571+0000 7f377810b640 1 --2- 172.21.15.16:0/1143127047 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f377015c130 0x7f377015dc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:23.579 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:23.571+0000 7f377810b640 1 --2- 172.21.15.16:0/1143127047 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f37540654a0 0x7f3754067950 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:23.579 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:23.571+0000 7f377810b640 1 --2- 172.21.15.16:0/1143127047 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f37700b24a0 0x7f37701558c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:23.579 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:23.571+0000 7f377810b640 1 --2- 172.21.15.16:0/1143127047 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f377006e270 0x7f37700b3aa0 unknown :-1 s=CLOSED pgs=1412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:23.579 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:23.571+0000 7f377810b640 1 -- 172.21.15.16:0/1143127047 >> 172.21.15.16:0/1143127047 conn(0x7f377006f5c0 msgr2=0x7f37700cf730 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:43:23.579 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:23.575+0000 7f377810b640 1 -- 172.21.15.16:0/1143127047 shutdown_connections 2023-05-22T15:43:23.580 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:23.575+0000 7f377810b640 1 -- 172.21.15.16:0/1143127047 wait complete. 2023-05-22T15:43:23.601 INFO:tasks.cephfs_test_runner:Test listing snap schedules on a non-existing filesystem path failure ... ok 2023-05-22T15:43:23.601 INFO:tasks.cephfs_test_runner:Starting test: test_schedule_auto_deactivation_for_non_existent_path (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2023-05-22T15:43:23.602 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_schedule_auto_deactivation_for_non_existent_path' 2023-05-22T15:43:23.731 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:23.727+0000 7ff1efb77640 1 Processor -- start 2023-05-22T15:43:23.731 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:23.727+0000 7ff1efb77640 1 -- start start 2023-05-22T15:43:23.732 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:23.727+0000 7ff1efb77640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff1e8159b10 0x7ff1e80cc780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:23.732 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:23.727+0000 7ff1efb77640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff1e806e250 0x7ff1e80b3e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:23.732 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:23.727+0000 7ff1efb77640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1e8159ee0 con 0x7ff1e80b5e80 2023-05-22T15:43:23.733 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:23.727+0000 7ff1efb77640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1e80b30b0 con 0x7ff1e806e250 2023-05-22T15:43:23.733 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:23.727+0000 7ff1efb77640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1e80b2810 con 0x7ff1e8159b10 2023-05-22T15:43:23.733 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:23.727+0000 7ff1ee114640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff1e806e250 0x7ff1e80b3e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:23.733 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:23.727+0000 7ff1ee114640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff1e806e250 0x7ff1e80b3e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57782/0 (socket says 172.21.15.16:57782) 2023-05-22T15:43:23.733 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:23.727+0000 7ff1ee114640 1 -- 172.21.15.16:0/695000907 learned_addr learned my addr 172.21.15.16:0/695000907 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:43:23.733 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:23.727+0000 7ff1ed913640 1 --2- 172.21.15.16:0/695000907 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff1e8159b10 0x7ff1e80cc780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:23.733 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:23.727+0000 7ff1ee114640 1 -- 172.21.15.16:0/695000907 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff1e8159b10 msgr2=0x7ff1e80cc780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:43:23.733 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:23.727+0000 7ff1ee114640 1 --2- 172.21.15.16:0/695000907 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff1e8159b10 0x7ff1e80cc780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:23.734 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:23.727+0000 7ff1ee114640 1 -- 172.21.15.16:0/695000907 >> v1:172.21.15.62:6789/0 conn(0x7ff1e80b5e80 legacy=0x7ff1e806db40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:23.734 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:23.727+0000 7ff1ee114640 1 -- 172.21.15.16:0/695000907 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1e80b2af0 con 0x7ff1e806e250 2023-05-22T15:43:23.734 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:23.731+0000 7ff1ee114640 1 --2- 172.21.15.16:0/695000907 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff1e806e250 0x7ff1e80b3e30 secure :-1 s=READY pgs=1413 cs=0 l=1 rev1=1 crypto rx=0x7ff1e0009e50 tx=0x7ff1e001c9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d47be9ded49bc6c server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:43:23.735 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:23.731+0000 7ff1ec911640 1 -- 172.21.15.16:0/695000907 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1e00a58a0 con 0x7ff1e806e250 2023-05-22T15:43:23.735 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:23.731+0000 7ff1ec911640 1 -- 172.21.15.16:0/695000907 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1e00a5a00 con 0x7ff1e806e250 2023-05-22T15:43:23.735 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:23.731+0000 7ff1ec911640 1 -- 172.21.15.16:0/695000907 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1e00a5d90 con 0x7ff1e806e250 2023-05-22T15:43:23.735 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:23.731+0000 7ff1efb77640 1 -- 172.21.15.16:0/695000907 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff1e806e250 msgr2=0x7ff1e80b3e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:23.736 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:23.731+0000 7ff1efb77640 1 --2- 172.21.15.16:0/695000907 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff1e806e250 0x7ff1e80b3e30 secure :-1 s=READY pgs=1413 cs=0 l=1 rev1=1 crypto rx=0x7ff1e0009e50 tx=0x7ff1e001c9e0 comp rx=0 tx=0).stop 2023-05-22T15:43:23.736 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:23.731+0000 7ff1efb77640 1 -- 172.21.15.16:0/695000907 shutdown_connections 2023-05-22T15:43:23.736 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:23.731+0000 7ff1efb77640 1 --2- 172.21.15.16:0/695000907 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff1e806e250 0x7ff1e80b3e30 unknown :-1 s=CLOSED pgs=1413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:23.736 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:23.731+0000 7ff1efb77640 1 --2- 172.21.15.16:0/695000907 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff1e8159b10 0x7ff1e80cc780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:23.736 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:23.731+0000 7ff1efb77640 1 -- 172.21.15.16:0/695000907 >> 172.21.15.16:0/695000907 conn(0x7ff1e806f560 msgr2=0x7ff1e806f960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:43:23.736 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:23.731+0000 7ff1efb77640 1 -- 172.21.15.16:0/695000907 shutdown_connections 2023-05-22T15:43:23.737 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:23.731+0000 7ff1efb77640 1 -- 172.21.15.16:0/695000907 wait complete. 2023-05-22T15:43:23.739 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:23.731+0000 7ff1efb77640 1 Processor -- start 2023-05-22T15:43:23.740 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:23.731+0000 7ff1efb77640 1 -- start start 2023-05-22T15:43:23.740 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:23.731+0000 7ff1efb77640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff1e806e250 0x7ff1e80c6030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:23.740 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:23.731+0000 7ff1efb77640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff1e80b5e80 0x7ff1e81532a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:23.740 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:23.731+0000 7ff1efb77640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff1e8159b10 0x7ff1e81537e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:23.741 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:23.731+0000 7ff1efb77640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1e80c6570 con 0x7ff1e80b5e80 2023-05-22T15:43:23.741 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:23.731+0000 7ff1efb77640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1e815b7d0 con 0x7ff1e806e250 2023-05-22T15:43:23.741 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:23.731+0000 7ff1efb77640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1e815ba90 con 0x7ff1e8159b10 2023-05-22T15:43:23.741 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:23.731+0000 7ff1ed112640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff1e80b5e80 0x7ff1e81532a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:23.741 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:23.731+0000 7ff1ed112640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff1e80b5e80 0x7ff1e81532a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57784/0 (socket says 172.21.15.16:57784) 2023-05-22T15:43:23.741 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:23.731+0000 7ff1ed112640 1 -- 172.21.15.16:0/1510802219 learned_addr learned my addr 172.21.15.16:0/1510802219 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:43:23.741 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:23.731+0000 7ff1ee114640 1 --2- 172.21.15.16:0/1510802219 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff1e8159b10 0x7ff1e81537e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:23.742 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:23.731+0000 7ff1ed913640 1 --2- 172.21.15.16:0/1510802219 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff1e806e250 0x7ff1e80c6030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:23.742 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:23.731+0000 7ff1ed112640 1 -- 172.21.15.16:0/1510802219 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff1e8159b10 msgr2=0x7ff1e81537e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:23.742 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:23.731+0000 7ff1ed112640 1 --2- 172.21.15.16:0/1510802219 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff1e8159b10 0x7ff1e81537e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:23.742 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:23.731+0000 7ff1ed112640 1 -- 172.21.15.16:0/1510802219 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff1e806e250 msgr2=0x7ff1e80c6030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:23.742 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:23.731+0000 7ff1ed112640 1 --2- 172.21.15.16:0/1510802219 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff1e806e250 0x7ff1e80c6030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:23.742 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:23.731+0000 7ff1ed112640 1 -- 172.21.15.16:0/1510802219 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1e0009b30 con 0x7ff1e80b5e80 2023-05-22T15:43:23.742 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:23.731+0000 7ff1ee114640 1 --2- 172.21.15.16:0/1510802219 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff1e8159b10 0x7ff1e81537e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:43:23.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:23.731+0000 7ff1ed112640 1 --2- 172.21.15.16:0/1510802219 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff1e80b5e80 0x7ff1e81532a0 secure :-1 s=READY pgs=1414 cs=0 l=1 rev1=1 crypto rx=0x7ff1d800c500 tx=0x7ff1d800c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:43:23.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:23.735+0000 7ff1de7fc640 1 -- 172.21.15.16:0/1510802219 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1d8005220 con 0x7ff1e80b5e80 2023-05-22T15:43:23.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:23.735+0000 7ff1de7fc640 1 -- 172.21.15.16:0/1510802219 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1d8005380 con 0x7ff1e80b5e80 2023-05-22T15:43:23.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:23.735+0000 7ff1efb77640 1 -- 172.21.15.16:0/1510802219 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1e80b4b80 con 0x7ff1e80b5e80 2023-05-22T15:43:23.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:23.735+0000 7ff1de7fc640 1 -- 172.21.15.16:0/1510802219 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1d8007270 con 0x7ff1e80b5e80 2023-05-22T15:43:23.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:23.735+0000 7ff1efb77640 1 -- 172.21.15.16:0/1510802219 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1e80b3910 con 0x7ff1e80b5e80 2023-05-22T15:43:23.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:23.735+0000 7ff1efb77640 1 -- 172.21.15.16:0/1510802219 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1e80b4d10 con 0x7ff1e80b5e80 2023-05-22T15:43:23.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:23.735+0000 7ff1de7fc640 1 -- 172.21.15.16:0/1510802219 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 41) v1 ==== 84332+0+0 (secure 0 0 0) 0x7ff1d80073d0 con 0x7ff1e80b5e80 2023-05-22T15:43:23.744 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:23.735+0000 7ff1de7fc640 1 --2- 172.21.15.16:0/1510802219 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ff1cc0654a0 0x7ff1cc067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:23.744 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:23.735+0000 7ff1de7fc640 1 -- 172.21.15.16:0/1510802219 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 6261+0+0 (secure 0 0 0) 0x7ff1d80823f0 con 0x7ff1e80b5e80 2023-05-22T15:43:23.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:23.735+0000 7ff1ed913640 1 --2- 172.21.15.16:0/1510802219 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ff1cc0654a0 0x7ff1cc067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:23.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:23.735+0000 7ff1ed913640 1 --2- 172.21.15.16:0/1510802219 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ff1cc0654a0 0x7ff1cc067950 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7ff1e400b930 tx=0x7ff1e4008040 comp rx=0 tx=0).ready entity=mgr.7788 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:43:23.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:23.739+0000 7ff1de7fc640 1 -- 172.21.15.16:0/1510802219 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7ff1d804fea0 con 0x7ff1e80b5e80 2023-05-22T15:43:23.946 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:23.943+0000 7ff1efb77640 1 -- 172.21.15.16:0/1510802219 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_schedule_auto_deactivation_for_non_existent_path"]} v 0) v1 -- 0x7ff1e804dbe0 con 0x7ff1e80b5e80 2023-05-22T15:43:24.578 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:24.571+0000 7ff1de7fc640 1 -- 172.21.15.16:0/1510802219 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_schedule_auto_deactivation_for_non_existent_path"]}]=0 v624) v1 ==== 186+0+0 (secure 0 0 0) 0x7ff1d80534c0 con 0x7ff1e80b5e80 2023-05-22T15:43:24.580 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:24.575+0000 7ff1efb77640 1 -- 172.21.15.16:0/1510802219 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ff1cc0654a0 msgr2=0x7ff1cc067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:24.581 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:24.575+0000 7ff1efb77640 1 --2- 172.21.15.16:0/1510802219 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ff1cc0654a0 0x7ff1cc067950 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7ff1e400b930 tx=0x7ff1e4008040 comp rx=0 tx=0).stop 2023-05-22T15:43:24.581 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:24.575+0000 7ff1efb77640 1 -- 172.21.15.16:0/1510802219 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff1e80b5e80 msgr2=0x7ff1e81532a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:24.581 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:24.575+0000 7ff1efb77640 1 --2- 172.21.15.16:0/1510802219 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff1e80b5e80 0x7ff1e81532a0 secure :-1 s=READY pgs=1414 cs=0 l=1 rev1=1 crypto rx=0x7ff1d800c500 tx=0x7ff1d800c9c0 comp rx=0 tx=0).stop 2023-05-22T15:43:24.582 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:24.575+0000 7ff1efb77640 1 -- 172.21.15.16:0/1510802219 shutdown_connections 2023-05-22T15:43:24.582 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:24.575+0000 7ff1efb77640 1 --2- 172.21.15.16:0/1510802219 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ff1cc0654a0 0x7ff1cc067950 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:24.582 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:24.575+0000 7ff1efb77640 1 --2- 172.21.15.16:0/1510802219 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff1e8159b10 0x7ff1e81537e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:24.582 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:24.575+0000 7ff1efb77640 1 --2- 172.21.15.16:0/1510802219 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff1e80b5e80 0x7ff1e81532a0 unknown :-1 s=CLOSED pgs=1414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:24.583 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:24.575+0000 7ff1efb77640 1 --2- 172.21.15.16:0/1510802219 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff1e806e250 0x7ff1e80c6030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:24.583 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:24.575+0000 7ff1efb77640 1 -- 172.21.15.16:0/1510802219 >> 172.21.15.16:0/1510802219 conn(0x7ff1e806f560 msgr2=0x7ff1e81639a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:43:24.583 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:24.575+0000 7ff1efb77640 1 -- 172.21.15.16:0/1510802219 shutdown_connections 2023-05-22T15:43:24.584 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:24.575+0000 7ff1efb77640 1 -- 172.21.15.16:0/1510802219 wait complete. 2023-05-22T15:43:24.605 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-05-22T15:43:24.740 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:24.735+0000 7f6b7dbf0640 1 Processor -- start 2023-05-22T15:43:24.740 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:24.735+0000 7f6b7dbf0640 1 -- start start 2023-05-22T15:43:24.740 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:24.735+0000 7f6b7dbf0640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6b78159970 0x7f6b78159d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:24.741 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:24.735+0000 7f6b7dbf0640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6b7815dad0 0x7f6b7806b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:24.741 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:24.735+0000 7f6b7dbf0640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b78070c30 con 0x7f6b7815d680 2023-05-22T15:43:24.741 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:24.735+0000 7f6b7dbf0640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b780cb970 con 0x7f6b7815dad0 2023-05-22T15:43:24.741 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:24.735+0000 7f6b7dbf0640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b78070e60 con 0x7f6b78159970 2023-05-22T15:43:24.742 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:24.735+0000 7f6b77fff640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6b7815dad0 0x7f6b7806b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:24.742 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:24.735+0000 7f6b77fff640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6b7815dad0 0x7f6b7806b6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57798/0 (socket says 172.21.15.16:57798) 2023-05-22T15:43:24.742 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:24.735+0000 7f6b77fff640 1 -- 172.21.15.16:0/1066506485 learned_addr learned my addr 172.21.15.16:0/1066506485 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:43:24.742 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:24.735+0000 7f6b777fe640 1 --2- 172.21.15.16:0/1066506485 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6b78159970 0x7f6b78159d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:24.742 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:24.735+0000 7f6b77fff640 1 -- 172.21.15.16:0/1066506485 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6b78159970 msgr2=0x7f6b78159d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:43:24.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:24.735+0000 7f6b77fff640 1 --2- 172.21.15.16:0/1066506485 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6b78159970 0x7f6b78159d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:24.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:24.735+0000 7f6b77fff640 1 -- 172.21.15.16:0/1066506485 >> v1:172.21.15.62:6789/0 conn(0x7f6b7815d680 legacy=0x7f6b780cb260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:24.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:24.735+0000 7f6b77fff640 1 -- 172.21.15.16:0/1066506485 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b78071140 con 0x7f6b7815dad0 2023-05-22T15:43:24.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:24.739+0000 7f6b77fff640 1 --2- 172.21.15.16:0/1066506485 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6b7815dad0 0x7f6b7806b6c0 secure :-1 s=READY pgs=1415 cs=0 l=1 rev1=1 crypto rx=0x7f6b70009e50 tx=0x7f6b7001c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d1ea9769b974e1c server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:43:24.744 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:24.739+0000 7f6b767fc640 1 -- 172.21.15.16:0/1066506485 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b700a5720 con 0x7f6b7815dad0 2023-05-22T15:43:24.744 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:24.739+0000 7f6b767fc640 1 -- 172.21.15.16:0/1066506485 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b700a5880 con 0x7f6b7815dad0 2023-05-22T15:43:24.744 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:24.739+0000 7f6b767fc640 1 -- 172.21.15.16:0/1066506485 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b700a5c10 con 0x7f6b7815dad0 2023-05-22T15:43:24.744 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:24.739+0000 7f6b7dbf0640 1 -- 172.21.15.16:0/1066506485 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6b7815dad0 msgr2=0x7f6b7806b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:24.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:24.739+0000 7f6b7dbf0640 1 --2- 172.21.15.16:0/1066506485 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6b7815dad0 0x7f6b7806b6c0 secure :-1 s=READY pgs=1415 cs=0 l=1 rev1=1 crypto rx=0x7f6b70009e50 tx=0x7f6b7001c950 comp rx=0 tx=0).stop 2023-05-22T15:43:24.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:24.739+0000 7f6b7dbf0640 1 -- 172.21.15.16:0/1066506485 shutdown_connections 2023-05-22T15:43:24.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:24.739+0000 7f6b7dbf0640 1 --2- 172.21.15.16:0/1066506485 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6b7815dad0 0x7f6b7806b6c0 unknown :-1 s=CLOSED pgs=1415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:24.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:24.739+0000 7f6b7dbf0640 1 --2- 172.21.15.16:0/1066506485 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6b78159970 0x7f6b78159d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:24.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:24.739+0000 7f6b7dbf0640 1 -- 172.21.15.16:0/1066506485 >> 172.21.15.16:0/1066506485 conn(0x7f6b78069720 msgr2=0x7f6b78069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:43:24.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:24.739+0000 7f6b7dbf0640 1 -- 172.21.15.16:0/1066506485 shutdown_connections 2023-05-22T15:43:24.746 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:24.739+0000 7f6b7dbf0640 1 -- 172.21.15.16:0/1066506485 wait complete. 2023-05-22T15:43:24.747 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:24.739+0000 7f6b7dbf0640 1 Processor -- start 2023-05-22T15:43:24.747 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:24.739+0000 7f6b7dbf0640 1 -- start start 2023-05-22T15:43:24.747 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:24.739+0000 7f6b7dbf0640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6b78159970 0x7f6b7814d950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:24.747 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:24.739+0000 7f6b7dbf0640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6b7815d680 0x7f6b781578d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:24.747 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:24.743+0000 7f6b7dbf0640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6b7815dad0 0x7f6b78157e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:24.748 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:24.743+0000 7f6b7dbf0640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b780b1e40 con 0x7f6b7815dad0 2023-05-22T15:43:24.748 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:24.743+0000 7f6b7dbf0640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b7814fd80 con 0x7f6b78159970 2023-05-22T15:43:24.748 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:24.743+0000 7f6b7dbf0640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b78150060 con 0x7f6b7815d680 2023-05-22T15:43:24.748 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:24.743+0000 7f6b77fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6b7815dad0 0x7f6b78157e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:24.748 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:24.743+0000 7f6b77fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6b7815dad0 0x7f6b78157e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57814/0 (socket says 172.21.15.16:57814) 2023-05-22T15:43:24.748 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:24.743+0000 7f6b77fff640 1 -- 172.21.15.16:0/3528407691 learned_addr learned my addr 172.21.15.16:0/3528407691 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:43:24.748 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:24.743+0000 7f6b777fe640 1 --2- 172.21.15.16:0/3528407691 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6b78159970 0x7f6b7814d950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:24.748 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:24.743+0000 7f6b76ffd640 1 --2- 172.21.15.16:0/3528407691 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6b7815d680 0x7f6b781578d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:24.749 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:24.743+0000 7f6b77fff640 1 -- 172.21.15.16:0/3528407691 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6b7815d680 msgr2=0x7f6b781578d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:24.749 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:24.743+0000 7f6b77fff640 1 --2- 172.21.15.16:0/3528407691 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6b7815d680 0x7f6b781578d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:24.749 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:24.743+0000 7f6b77fff640 1 -- 172.21.15.16:0/3528407691 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6b78159970 msgr2=0x7f6b7814d950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:24.749 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:24.743+0000 7f6b77fff640 1 --2- 172.21.15.16:0/3528407691 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6b78159970 0x7f6b7814d950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:24.749 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:24.743+0000 7f6b77fff640 1 -- 172.21.15.16:0/3528407691 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b70009b30 con 0x7f6b7815dad0 2023-05-22T15:43:24.749 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:24.743+0000 7f6b77fff640 1 --2- 172.21.15.16:0/3528407691 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6b7815dad0 0x7f6b78157e10 secure :-1 s=READY pgs=1416 cs=0 l=1 rev1=1 crypto rx=0x7f6b7001a580 tx=0x7f6b700196f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:43:24.750 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:24.743+0000 7f6b5ffff640 1 -- 172.21.15.16:0/3528407691 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b700b8640 con 0x7f6b7815dad0 2023-05-22T15:43:24.750 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:24.743+0000 7f6b5ffff640 1 -- 172.21.15.16:0/3528407691 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b700b87c0 con 0x7f6b7815dad0 2023-05-22T15:43:24.751 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:24.743+0000 7f6b5ffff640 1 -- 172.21.15.16:0/3528407691 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b700b8b70 con 0x7f6b7815dad0 2023-05-22T15:43:24.751 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:24.743+0000 7f6b7dbf0640 1 -- 172.21.15.16:0/3528407691 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b780cae20 con 0x7f6b7815dad0 2023-05-22T15:43:24.751 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:24.743+0000 7f6b7dbf0640 1 -- 172.21.15.16:0/3528407691 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b780cac90 con 0x7f6b7815dad0 2023-05-22T15:43:24.751 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:24.743+0000 7f6b7dbf0640 1 -- 172.21.15.16:0/3528407691 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b7815adc0 con 0x7f6b7815dad0 2023-05-22T15:43:24.751 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:24.743+0000 7f6b5ffff640 1 -- 172.21.15.16:0/3528407691 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 41) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f6b700b8cd0 con 0x7f6b7815dad0 2023-05-22T15:43:24.751 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:24.743+0000 7f6b5ffff640 1 --2- 172.21.15.16:0/3528407691 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f6b640654a0 0x7f6b64067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:24.751 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:24.743+0000 7f6b5ffff640 1 -- 172.21.15.16:0/3528407691 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 6261+0+0 (secure 0 0 0) 0x7f6b7011dc90 con 0x7f6b7815dad0 2023-05-22T15:43:24.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:24.743+0000 7f6b777fe640 1 --2- 172.21.15.16:0/3528407691 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f6b640654a0 0x7f6b64067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:24.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:24.743+0000 7f6b777fe640 1 --2- 172.21.15.16:0/3528407691 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f6b640654a0 0x7f6b64067950 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f6b7814eb20 tx=0x7f6b68008040 comp rx=0 tx=0).ready entity=mgr.7788 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:43:24.753 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:24.747+0000 7f6b5ffff640 1 -- 172.21.15.16:0/3528407691 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f6b700eb730 con 0x7f6b7815dad0 2023-05-22T15:43:24.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:24.903+0000 7f6b7dbf0640 1 -- 172.21.15.16:0/3528407691 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f6b78150220 con 0x7f6b7815dad0 2023-05-22T15:43:24.916 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:24.911+0000 7f6b5ffff640 1 -- 172.21.15.16:0/3528407691 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v34)=0 v34) v1 ==== 122+0+0 (secure 0 0 0) 0x7f6b70009510 con 0x7f6b7815dad0 2023-05-22T15:43:24.919 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:24.911+0000 7f6b7dbf0640 1 -- 172.21.15.16:0/3528407691 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f6b640654a0 msgr2=0x7f6b64067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:24.919 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:24.911+0000 7f6b7dbf0640 1 --2- 172.21.15.16:0/3528407691 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f6b640654a0 0x7f6b64067950 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f6b7814eb20 tx=0x7f6b68008040 comp rx=0 tx=0).stop 2023-05-22T15:43:24.919 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:24.911+0000 7f6b7dbf0640 1 -- 172.21.15.16:0/3528407691 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6b7815dad0 msgr2=0x7f6b78157e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:24.919 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:24.911+0000 7f6b7dbf0640 1 --2- 172.21.15.16:0/3528407691 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6b7815dad0 0x7f6b78157e10 secure :-1 s=READY pgs=1416 cs=0 l=1 rev1=1 crypto rx=0x7f6b7001a580 tx=0x7f6b700196f0 comp rx=0 tx=0).stop 2023-05-22T15:43:24.919 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:24.911+0000 7f6b7dbf0640 1 -- 172.21.15.16:0/3528407691 shutdown_connections 2023-05-22T15:43:24.919 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:24.911+0000 7f6b7dbf0640 1 --2- 172.21.15.16:0/3528407691 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6b7815dad0 0x7f6b78157e10 unknown :-1 s=CLOSED pgs=1416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:24.920 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:24.911+0000 7f6b7dbf0640 1 --2- 172.21.15.16:0/3528407691 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f6b640654a0 0x7f6b64067950 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:24.920 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:24.911+0000 7f6b7dbf0640 1 --2- 172.21.15.16:0/3528407691 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6b7815d680 0x7f6b781578d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:24.920 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:24.911+0000 7f6b7dbf0640 1 --2- 172.21.15.16:0/3528407691 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6b78159970 0x7f6b7814d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:24.920 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:24.911+0000 7f6b7dbf0640 1 -- 172.21.15.16:0/3528407691 >> 172.21.15.16:0/3528407691 conn(0x7f6b78069720 msgr2=0x7f6b781522c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:43:24.920 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:24.911+0000 7f6b7dbf0640 1 -- 172.21.15.16:0/3528407691 shutdown_connections 2023-05-22T15:43:24.921 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:24.911+0000 7f6b7dbf0640 1 -- 172.21.15.16:0/3528407691 wait complete. 2023-05-22T15:43:24.935 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-05-22T15:43:24.935 DEBUG:teuthology.orchestra.run.smithi016:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-05-22T15:43:24.976 DEBUG:teuthology.orchestra.run.smithi062:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-05-22T15:43:25.030 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-05-22T15:43:25.031 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-05-22T15:43:25.031 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2023-05-22T15:43:25.039 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16345192k,nr_inodes=4086298,mode=755,inode64 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3276544k,mode=755,inode64 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev,inode64 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=15732 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3276540k,nr_inodes=819135,mode=700,uid=1000,gid=1271,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3276544k,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-05-22T15:43:25.039 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2023-05-22T15:43:25.040 DEBUG:teuthology.orchestra.run.smithi062:> dd if=/proc/self/mounts of=/dev/stdout 2023-05-22T15:43:25.046 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16342116k,nr_inodes=4085529,mode=755,inode64 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275932k,mode=755,inode64 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev,inode64 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=13250 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275928k,nr_inodes=818982,mode=700,uid=1000,gid=1271,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275932k,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-05-22T15:43:25.046 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-05-22T15:43:25.047 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2023-05-22T15:43:25.087 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16345192k,nr_inodes=4086298,mode=755,inode64 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3276544k,mode=755,inode64 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev,inode64 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=15732 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3276540k,nr_inodes=819135,mode=700,uid=1000,gid=1271,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3276544k,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-05-22T15:43:25.088 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2023-05-22T15:43:25.088 DEBUG:teuthology.orchestra.run.smithi062:> dd if=/proc/self/mounts of=/dev/stdout 2023-05-22T15:43:25.094 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16342116k,nr_inodes=4085529,mode=755,inode64 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275932k,mode=755,inode64 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev,inode64 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=13250 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275928k,nr_inodes=818982,mode=700,uid=1000,gid=1271,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275932k,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-05-22T15:43:25.094 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2023-05-22T15:43:25.094 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-05-22T15:43:25.259 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:25.255+0000 7f8c0b29b640 1 Processor -- start 2023-05-22T15:43:25.259 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:25.255+0000 7f8c0b29b640 1 -- start start 2023-05-22T15:43:25.260 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:25.255+0000 7f8c0b29b640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f8c040b2880 0x7f8c040cbfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:25.260 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:25.255+0000 7f8c0b29b640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f8c040cc520 0x7f8c040cfe30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:25.261 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:25.255+0000 7f8c0b29b640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c040cf9a0 con 0x7f8c04159cd0 2023-05-22T15:43:25.261 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:25.255+0000 7f8c0b29b640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c0415aba0 con 0x7f8c040cc520 2023-05-22T15:43:25.261 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:25.255+0000 7f8c0b29b640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c0415ae80 con 0x7f8c040b2880 2023-05-22T15:43:25.261 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:25.255+0000 7f8c09037640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f8c04159cd0 0x7f8c0415a0a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:54026/0 (socket says 172.21.15.16:54026) 2023-05-22T15:43:25.261 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:25.255+0000 7f8c09037640 1 -- 172.21.15.16:0/782081787 learned_addr learned my addr 172.21.15.16:0/782081787 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:43:25.261 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:25.255+0000 7f8c09838640 1 --2- 172.21.15.16:0/782081787 >> v2:172.21.15.16:3300/0 conn(0x7f8c040cc520 0x7f8c040cfe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:25.262 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:25.255+0000 7f8bfbfff640 1 -- 172.21.15.16:0/782081787 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3371931960 0 0) 0x7f8c040cf9a0 con 0x7f8c04159cd0 2023-05-22T15:43:25.262 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:25.255+0000 7f8c08836640 1 --2- 172.21.15.16:0/782081787 >> v2:172.21.15.62:3300/0 conn(0x7f8c040b2880 0x7f8c040cbfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:25.262 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:25.255+0000 7f8bfbfff640 1 -- 172.21.15.16:0/782081787 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8bf0003530 con 0x7f8c04159cd0 2023-05-22T15:43:25.262 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:25.255+0000 7f8c09838640 1 -- 172.21.15.16:0/782081787 >> v2:172.21.15.62:3300/0 conn(0x7f8c040b2880 msgr2=0x7f8c040cbfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:43:25.263 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:25.255+0000 7f8c09838640 1 --2- 172.21.15.16:0/782081787 >> v2:172.21.15.62:3300/0 conn(0x7f8c040b2880 0x7f8c040cbfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:25.263 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:25.255+0000 7f8c09838640 1 -- 172.21.15.16:0/782081787 >> v1:172.21.15.16:6789/0 conn(0x7f8c04159cd0 legacy=0x7f8c0415a0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:25.263 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:25.255+0000 7f8c09838640 1 -- 172.21.15.16:0/782081787 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c0415b160 con 0x7f8c040cc520 2023-05-22T15:43:25.264 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:25.259+0000 7f8c09838640 1 --2- 172.21.15.16:0/782081787 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8c040cc520 0x7f8c040cfe30 secure :-1 s=READY pgs=1418 cs=0 l=1 rev1=1 crypto rx=0x7f8c040ca810 tx=0x7f8bfc01c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a729175858d58017 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:43:25.264 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:25.259+0000 7f8bfbfff640 1 -- 172.21.15.16:0/782081787 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bfc0a58b0 con 0x7f8c040cc520 2023-05-22T15:43:25.264 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:25.259+0000 7f8bfbfff640 1 -- 172.21.15.16:0/782081787 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8bfc0a5a10 con 0x7f8c040cc520 2023-05-22T15:43:25.264 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:25.259+0000 7f8bfbfff640 1 -- 172.21.15.16:0/782081787 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bfc0a5da0 con 0x7f8c040cc520 2023-05-22T15:43:25.264 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:25.259+0000 7f8c0b29b640 1 -- 172.21.15.16:0/782081787 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8c040cc520 msgr2=0x7f8c040cfe30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:25.264 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:25.259+0000 7f8c0b29b640 1 --2- 172.21.15.16:0/782081787 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8c040cc520 0x7f8c040cfe30 secure :-1 s=READY pgs=1418 cs=0 l=1 rev1=1 crypto rx=0x7f8c040ca810 tx=0x7f8bfc01c8e0 comp rx=0 tx=0).stop 2023-05-22T15:43:25.265 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:25.259+0000 7f8c0b29b640 1 -- 172.21.15.16:0/782081787 shutdown_connections 2023-05-22T15:43:25.265 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:25.259+0000 7f8c0b29b640 1 --2- 172.21.15.16:0/782081787 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8c040cc520 0x7f8c040cfe30 unknown :-1 s=CLOSED pgs=1418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:25.265 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:25.259+0000 7f8c0b29b640 1 --2- 172.21.15.16:0/782081787 >> v2:172.21.15.62:3300/0 conn(0x7f8c040b2880 0x7f8c040cbfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:25.265 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:25.259+0000 7f8c0b29b640 1 -- 172.21.15.16:0/782081787 >> 172.21.15.16:0/782081787 conn(0x7f8c0406f5c0 msgr2=0x7f8c0406f9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:43:25.265 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:25.259+0000 7f8c0b29b640 1 -- 172.21.15.16:0/782081787 shutdown_connections 2023-05-22T15:43:25.265 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:25.259+0000 7f8c0b29b640 1 -- 172.21.15.16:0/782081787 wait complete. 2023-05-22T15:43:25.266 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:25.259+0000 7f8c0b29b640 1 Processor -- start 2023-05-22T15:43:25.266 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:25.259+0000 7f8c0b29b640 1 -- start start 2023-05-22T15:43:25.267 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:25.259+0000 7f8c0b29b640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8c040b2880 0x7f8c0406b690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:25.267 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:25.259+0000 7f8c0b29b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8c040cc520 0x7f8c0406a9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:25.267 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:25.259+0000 7f8c0b29b640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8c04159cd0 0x7f8c0414e660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:25.267 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:25.259+0000 7f8c0b29b640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c0406c2e0 con 0x7f8c040cc520 2023-05-22T15:43:25.267 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:25.259+0000 7f8c0b29b640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c0406af30 con 0x7f8c04159cd0 2023-05-22T15:43:25.267 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:25.259+0000 7f8c0b29b640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c0406bd20 con 0x7f8c040b2880 2023-05-22T15:43:25.267 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:25.259+0000 7f8c08836640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8c040cc520 0x7f8c0406a9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:25.268 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:25.263+0000 7f8c08836640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8c040cc520 0x7f8c0406a9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57834/0 (socket says 172.21.15.16:57834) 2023-05-22T15:43:25.268 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:25.263+0000 7f8c08836640 1 -- 172.21.15.16:0/2298941042 learned_addr learned my addr 172.21.15.16:0/2298941042 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:43:25.268 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:25.263+0000 7f8c08836640 1 -- 172.21.15.16:0/2298941042 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8c040b2880 msgr2=0x7f8c0406b690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:25.268 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:25.263+0000 7f8c08836640 1 --2- 172.21.15.16:0/2298941042 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8c040b2880 0x7f8c0406b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:25.268 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:25.263+0000 7f8c09838640 1 --2- 172.21.15.16:0/2298941042 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8c04159cd0 0x7f8c0414e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:25.268 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:25.263+0000 7f8c08836640 1 -- 172.21.15.16:0/2298941042 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8c04159cd0 msgr2=0x7f8c0414e660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:25.268 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:25.263+0000 7f8c08836640 1 --2- 172.21.15.16:0/2298941042 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8c04159cd0 0x7f8c0414e660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:25.268 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:25.263+0000 7f8c08836640 1 -- 172.21.15.16:0/2298941042 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bfc009a70 con 0x7f8c040cc520 2023-05-22T15:43:25.269 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:25.263+0000 7f8c08836640 1 --2- 172.21.15.16:0/2298941042 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8c040cc520 0x7f8c0406a9f0 secure :-1 s=READY pgs=1419 cs=0 l=1 rev1=1 crypto rx=0x7f8bf400d800 tx=0x7f8bf400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:43:25.270 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:25.263+0000 7f8bf9ffb640 1 -- 172.21.15.16:0/2298941042 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bf400b660 con 0x7f8c040cc520 2023-05-22T15:43:25.270 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:25.263+0000 7f8bf9ffb640 1 -- 172.21.15.16:0/2298941042 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8bf400b7c0 con 0x7f8c040cc520 2023-05-22T15:43:25.270 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:25.263+0000 7f8c0b29b640 1 -- 172.21.15.16:0/2298941042 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c0415a940 con 0x7f8c040cc520 2023-05-22T15:43:25.270 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:25.263+0000 7f8bf9ffb640 1 -- 172.21.15.16:0/2298941042 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bf400bb30 con 0x7f8c040cc520 2023-05-22T15:43:25.270 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:25.263+0000 7f8c0b29b640 1 -- 172.21.15.16:0/2298941042 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c0415a7b0 con 0x7f8c040cc520 2023-05-22T15:43:25.270 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:25.263+0000 7f8c0b29b640 1 -- 172.21.15.16:0/2298941042 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c040d0b80 con 0x7f8c040cc520 2023-05-22T15:43:25.270 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:25.263+0000 7f8bf9ffb640 1 -- 172.21.15.16:0/2298941042 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 41) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f8bf400bc90 con 0x7f8c040cc520 2023-05-22T15:43:25.271 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:25.263+0000 7f8bf9ffb640 1 --2- 172.21.15.16:0/2298941042 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f8be80654a0 0x7f8be8067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:25.271 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:25.263+0000 7f8bf9ffb640 1 -- 172.21.15.16:0/2298941042 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 6261+0+0 (secure 0 0 0) 0x7f8bf4083af0 con 0x7f8c040cc520 2023-05-22T15:43:25.271 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:25.263+0000 7f8c09037640 1 --2- 172.21.15.16:0/2298941042 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f8be80654a0 0x7f8be8067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:25.271 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:25.263+0000 7f8c09037640 1 --2- 172.21.15.16:0/2298941042 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f8be80654a0 0x7f8be8067950 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f8c00007fd0 tx=0x7f8c00007940 comp rx=0 tx=0).ready entity=mgr.7788 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:43:25.272 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:25.267+0000 7f8bf9ffb640 1 -- 172.21.15.16:0/2298941042 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f8bf4051570 con 0x7f8c040cc520 2023-05-22T15:43:25.460 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:25.455+0000 7f8c0b29b640 1 -- 172.21.15.16:0/2298941042 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8c0404dbe0 con 0x7f8c040cc520 2023-05-22T15:43:25.461 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:25.455+0000 7f8bf9ffb640 1 -- 172.21.15.16:0/2298941042 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 38 v38) v1 ==== 94+0+3529 (secure 0 0 0) 0x7f8bf4054b90 con 0x7f8c040cc520 2023-05-22T15:43:25.461 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:43:25.461 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":38,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6059,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6833/3017549713","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":3017549713},{"type":"v1","addr":"172.21.15.62:6833","nonce":3017549713}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":33},{"gid":6607,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/3268651705","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":3268651705},{"type":"v1","addr":"172.21.15.62:6835","nonce":3268651705}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":33},{"gid":7524,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6836/2104991509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2104991509},{"type":"v1","addr":"172.21.15.16:6836","nonce":2104991509}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":33},{"gid":7938,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/3713042216","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6835","nonce":3713042216},{"type":"v1","addr":"172.21.15.16:6837","nonce":3713042216}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":38}],"filesystems":[]} 2023-05-22T15:43:25.462 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 38 2023-05-22T15:43:25.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:25.459+0000 7f8c0b29b640 1 -- 172.21.15.16:0/2298941042 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f8be80654a0 msgr2=0x7f8be8067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:25.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:25.459+0000 7f8c0b29b640 1 --2- 172.21.15.16:0/2298941042 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f8be80654a0 0x7f8be8067950 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f8c00007fd0 tx=0x7f8c00007940 comp rx=0 tx=0).stop 2023-05-22T15:43:25.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:25.459+0000 7f8c0b29b640 1 -- 172.21.15.16:0/2298941042 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8c040cc520 msgr2=0x7f8c0406a9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:25.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:25.459+0000 7f8c0b29b640 1 --2- 172.21.15.16:0/2298941042 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8c040cc520 0x7f8c0406a9f0 secure :-1 s=READY pgs=1419 cs=0 l=1 rev1=1 crypto rx=0x7f8bf400d800 tx=0x7f8bf400dcc0 comp rx=0 tx=0).stop 2023-05-22T15:43:25.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:25.459+0000 7f8c0b29b640 1 -- 172.21.15.16:0/2298941042 shutdown_connections 2023-05-22T15:43:25.464 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:25.459+0000 7f8c0b29b640 1 --2- 172.21.15.16:0/2298941042 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8c04159cd0 0x7f8c0414e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:25.464 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:25.459+0000 7f8c0b29b640 1 --2- 172.21.15.16:0/2298941042 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8c040cc520 0x7f8c0406a9f0 secure :-1 s=CLOSED pgs=1419 cs=0 l=1 rev1=1 crypto rx=0x7f8bf400d800 tx=0x7f8bf400dcc0 comp rx=0 tx=0).stop 2023-05-22T15:43:25.464 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:25.459+0000 7f8c0b29b640 1 --2- 172.21.15.16:0/2298941042 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f8be80654a0 0x7f8be8067950 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:25.464 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:25.459+0000 7f8c0b29b640 1 --2- 172.21.15.16:0/2298941042 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8c040b2880 0x7f8c0406b690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:25.464 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:25.459+0000 7f8c0b29b640 1 -- 172.21.15.16:0/2298941042 >> 172.21.15.16:0/2298941042 conn(0x7f8c0406f5c0 msgr2=0x7f8c0406f990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:43:25.464 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:25.459+0000 7f8c0b29b640 1 -- 172.21.15.16:0/2298941042 shutdown_connections 2023-05-22T15:43:25.464 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:25.459+0000 7f8c0b29b640 1 -- 172.21.15.16:0/2298941042 wait complete. 2023-05-22T15:43:25.479 INFO:tasks.ceph.mds.a:Restarting daemon 2023-05-22T15:43:25.479 INFO:tasks.ceph.mds.a:Stopping old one... 2023-05-22T15:43:25.480 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-05-22T15:43:25.480 INFO:teuthology.orchestra.run:waiting for 300 2023-05-22T15:43:25.480 INFO:tasks.ceph.mds.c:Restarting daemon 2023-05-22T15:43:25.480 INFO:tasks.ceph.mds.c:Stopping old one... 2023-05-22T15:43:25.481 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-05-22T15:43:25.481 INFO:teuthology.orchestra.run:waiting for 300 2023-05-22T15:43:25.481 INFO:tasks.ceph.mds.b:Restarting daemon 2023-05-22T15:43:25.481 INFO:tasks.ceph.mds.b:Stopping old one... 2023-05-22T15:43:25.481 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-05-22T15:43:25.482 INFO:teuthology.orchestra.run:waiting for 300 2023-05-22T15:43:25.482 INFO:tasks.ceph.mds.d:Restarting daemon 2023-05-22T15:43:25.482 INFO:tasks.ceph.mds.d:Stopping old one... 2023-05-22T15:43:25.482 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-05-22T15:43:25.482 INFO:teuthology.orchestra.run:waiting for 300 2023-05-22T15:43:25.489 INFO:tasks.ceph.mds.c:Stopped 2023-05-22T15:43:25.490 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-05-22T15:43:25.490 INFO:tasks.ceph.mds.a:Stopped 2023-05-22T15:43:25.491 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-05-22T15:43:25.495 INFO:tasks.ceph.mds.d:Stopped 2023-05-22T15:43:25.495 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-05-22T15:43:25.497 INFO:tasks.ceph.mds.b:Stopped 2023-05-22T15:43:25.497 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-05-22T15:43:25.531 INFO:tasks.ceph.mds.c:Started 2023-05-22T15:43:25.532 DEBUG:teuthology.parallel:result is None 2023-05-22T15:43:25.532 INFO:tasks.ceph.mds.a:Started 2023-05-22T15:43:25.533 DEBUG:teuthology.parallel:result is None 2023-05-22T15:43:25.545 INFO:tasks.ceph.mds.d:Started 2023-05-22T15:43:25.546 DEBUG:teuthology.parallel:result is None 2023-05-22T15:43:25.547 INFO:tasks.ceph.mds.b:Started 2023-05-22T15:43:25.547 DEBUG:teuthology.parallel:result is None 2023-05-22T15:43:25.548 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2023-05-22T15:43:25.590 INFO:tasks.ceph.mds.c.smithi016.stdout:starting mds.c at 2023-05-22T15:43:25.590 INFO:tasks.ceph.mds.a.smithi016.stdout:starting mds.a at 2023-05-22T15:43:25.620 INFO:tasks.ceph.mds.b.smithi062.stdout:starting mds.b at 2023-05-22T15:43:25.624 INFO:tasks.ceph.mds.d.smithi062.stdout:starting mds.d at 2023-05-22T15:43:25.673 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:25.667+0000 7f4220891640 1 Processor -- start 2023-05-22T15:43:25.674 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:25.667+0000 7f4220891640 1 -- start start 2023-05-22T15:43:25.675 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:25.671+0000 7f4220891640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f421c0ccab0 0x7f421c15fe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:25.675 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:25.671+0000 7f4220891640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f421c0d0410 0x7f421c0d0830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:25.675 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:25.671+0000 7f4220891640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f421c0ccef0 con 0x7f421c0b7dd0 2023-05-22T15:43:25.675 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:25.671+0000 7f4220891640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f421c0d3360 con 0x7f421c0ccab0 2023-05-22T15:43:25.675 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:25.671+0000 7f4220891640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f421c0cd170 con 0x7f421c0d0410 2023-05-22T15:43:25.676 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:25.671+0000 7f421ad9d640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f421c0d0410 0x7f421c0d0830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:25.676 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:25.671+0000 7f421ad9d640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f421c0d0410 0x7f421c0d0830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3301/0 says I am v2:172.21.15.16:33526/0 (socket says 172.21.15.16:33526) 2023-05-22T15:43:25.676 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:25.671+0000 7f4219d9b640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f421c0ccab0 0x7f421c15fe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:25.677 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:25.671+0000 7f421ad9d640 1 -- 172.21.15.16:0/3995593503 learned_addr learned my addr 172.21.15.16:0/3995593503 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:43:25.677 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:25.671+0000 7f421ad9d640 1 -- 172.21.15.16:0/3995593503 >> v2:172.21.15.62:3300/0 conn(0x7f421c0ccab0 msgr2=0x7f421c15fe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:43:25.677 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:25.671+0000 7f421ad9d640 1 --2- 172.21.15.16:0/3995593503 >> v2:172.21.15.62:3300/0 conn(0x7f421c0ccab0 0x7f421c15fe10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:25.678 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:25.671+0000 7f421ad9d640 1 -- 172.21.15.16:0/3995593503 >> v1:172.21.15.62:6789/0 conn(0x7f421c0b7dd0 legacy=0x7f421c15f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:25.678 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:25.671+0000 7f421ad9d640 1 -- 172.21.15.16:0/3995593503 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f421c0d2ac0 con 0x7f421c0d0410 2023-05-22T15:43:25.681 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:25.671+0000 7f421ad9d640 1 --2- 172.21.15.16:0/3995593503 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f421c0d0410 0x7f421c0d0830 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f4214009e50 tx=0x7f421401c9e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a880b9e294575d01 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:43:25.681 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:25.671+0000 7f421959a640 1 -- 172.21.15.16:0/3995593503 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42140a58a0 con 0x7f421c0d0410 2023-05-22T15:43:25.681 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:25.671+0000 7f421959a640 1 -- 172.21.15.16:0/3995593503 <== mon.2 v2:172.21.15.62:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42140a5a00 con 0x7f421c0d0410 2023-05-22T15:43:25.681 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:25.671+0000 7f421959a640 1 -- 172.21.15.16:0/3995593503 <== mon.2 v2:172.21.15.62:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42140a5d90 con 0x7f421c0d0410 2023-05-22T15:43:25.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:25.671+0000 7f4220891640 1 -- 172.21.15.16:0/3995593503 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f421c0d0410 msgr2=0x7f421c0d0830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:25.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:25.671+0000 7f4220891640 1 --2- 172.21.15.16:0/3995593503 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f421c0d0410 0x7f421c0d0830 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f4214009e50 tx=0x7f421401c9e0 comp rx=0 tx=0).stop 2023-05-22T15:43:25.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:25.671+0000 7f4220891640 1 -- 172.21.15.16:0/3995593503 shutdown_connections 2023-05-22T15:43:25.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:25.671+0000 7f4220891640 1 --2- 172.21.15.16:0/3995593503 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f421c0d0410 0x7f421c0d0830 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:25.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:25.671+0000 7f4220891640 1 --2- 172.21.15.16:0/3995593503 >> v2:172.21.15.62:3300/0 conn(0x7f421c0ccab0 0x7f421c15fe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:25.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:25.671+0000 7f4220891640 1 -- 172.21.15.16:0/3995593503 >> 172.21.15.16:0/3995593503 conn(0x7f421c0655c0 msgr2=0x7f421c070e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:43:25.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:25.671+0000 7f4220891640 1 -- 172.21.15.16:0/3995593503 shutdown_connections 2023-05-22T15:43:25.683 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:25.671+0000 7f4220891640 1 -- 172.21.15.16:0/3995593503 wait complete. 2023-05-22T15:43:25.683 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:25.675+0000 7f4220891640 1 Processor -- start 2023-05-22T15:43:25.683 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:25.675+0000 7f4220891640 1 -- start start 2023-05-22T15:43:25.683 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:25.675+0000 7f4220891640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f421c0b7dd0 0x7f421c0b2860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:25.683 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:25.675+0000 7f4220891640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f421c0ccab0 0x7f421c0b2da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:25.684 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:25.675+0000 7f4220891640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f421c0d0410 0x7f421c0cb550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:25.684 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:25.675+0000 7f4220891640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f421c152db0 con 0x7f421c0b7dd0 2023-05-22T15:43:25.684 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:25.675+0000 7f4220891640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f421c153090 con 0x7f421c0ccab0 2023-05-22T15:43:25.684 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:25.675+0000 7f4220891640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f421c153370 con 0x7f421c0d0410 2023-05-22T15:43:25.684 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:25.675+0000 7f421a59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f421c0b7dd0 0x7f421c0b2860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:25.684 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:25.675+0000 7f421a59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f421c0b7dd0 0x7f421c0b2860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57870/0 (socket says 172.21.15.16:57870) 2023-05-22T15:43:25.685 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:25.675+0000 7f421a59c640 1 -- 172.21.15.16:0/2757912157 learned_addr learned my addr 172.21.15.16:0/2757912157 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:43:25.685 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:25.675+0000 7f421ad9d640 1 --2- 172.21.15.16:0/2757912157 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f421c0d0410 0x7f421c0cb550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:25.685 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:25.675+0000 7f4219d9b640 1 --2- 172.21.15.16:0/2757912157 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f421c0ccab0 0x7f421c0b2da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:25.685 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:25.675+0000 7f421a59c640 1 -- 172.21.15.16:0/2757912157 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f421c0d0410 msgr2=0x7f421c0cb550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:25.685 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:25.675+0000 7f421a59c640 1 --2- 172.21.15.16:0/2757912157 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f421c0d0410 0x7f421c0cb550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:25.685 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:25.675+0000 7f421a59c640 1 -- 172.21.15.16:0/2757912157 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f421c0ccab0 msgr2=0x7f421c0b2da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:25.685 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:25.675+0000 7f421a59c640 1 --2- 172.21.15.16:0/2757912157 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f421c0ccab0 0x7f421c0b2da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:25.685 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:25.675+0000 7f421a59c640 1 -- 172.21.15.16:0/2757912157 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4214009b30 con 0x7f421c0b7dd0 2023-05-22T15:43:25.686 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:25.675+0000 7f421a59c640 1 --2- 172.21.15.16:0/2757912157 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f421c0b7dd0 0x7f421c0b2860 secure :-1 s=READY pgs=1426 cs=0 l=1 rev1=1 crypto rx=0x7f420c00c370 tx=0x7f420c00c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:43:25.686 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:25.675+0000 7f420affd640 1 -- 172.21.15.16:0/2757912157 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f420c0052d0 con 0x7f421c0b7dd0 2023-05-22T15:43:25.686 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:25.675+0000 7f420affd640 1 -- 172.21.15.16:0/2757912157 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f420c005430 con 0x7f421c0b7dd0 2023-05-22T15:43:25.686 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:25.675+0000 7f4220891640 1 -- 172.21.15.16:0/2757912157 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f421c160350 con 0x7f421c0b7dd0 2023-05-22T15:43:25.686 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:25.675+0000 7f420affd640 1 -- 172.21.15.16:0/2757912157 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f420c007260 con 0x7f421c0b7dd0 2023-05-22T15:43:25.686 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:25.675+0000 7f4220891640 1 -- 172.21.15.16:0/2757912157 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f421c0cf730 con 0x7f421c0b7dd0 2023-05-22T15:43:25.687 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:25.675+0000 7f4220891640 1 -- 172.21.15.16:0/2757912157 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f421c1604e0 con 0x7f421c0b7dd0 2023-05-22T15:43:25.687 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:25.679+0000 7f420affd640 1 -- 172.21.15.16:0/2757912157 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 41) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f420c0073c0 con 0x7f421c0b7dd0 2023-05-22T15:43:25.688 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:25.679+0000 7f420affd640 1 --2- 172.21.15.16:0/2757912157 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f41fc0654d0 0x7f41fc067980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:25.688 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:25.679+0000 7f420affd640 1 -- 172.21.15.16:0/2757912157 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 6261+0+0 (secure 0 0 0) 0x7f420c082380 con 0x7f421c0b7dd0 2023-05-22T15:43:25.688 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:25.679+0000 7f4219d9b640 1 --2- 172.21.15.16:0/2757912157 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f41fc0654d0 0x7f41fc067980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:25.688 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:25.679+0000 7f4219d9b640 1 --2- 172.21.15.16:0/2757912157 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f41fc0654d0 0x7f41fc067980 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f421c0b1460 tx=0x7f4210009190 comp rx=0 tx=0).ready entity=mgr.7788 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:43:25.688 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:25.683+0000 7f420affd640 1 -- 172.21.15.16:0/2757912157 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f420c04fe30 con 0x7f421c0b7dd0 2023-05-22T15:43:25.845 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:25.839+0000 7f4220891640 1 -- 172.21.15.16:0/2757912157 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f421c0cc250 con 0x7f421c0b7dd0 2023-05-22T15:43:26.667 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:26.663+0000 7f420affd640 1 -- 172.21.15.16:0/2757912157 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v114) v1 ==== 97+0+0 (secure 0 0 0) 0x7f420c053450 con 0x7f421c0b7dd0 2023-05-22T15:43:26.667 INFO:teuthology.orchestra.run.smithi016.stderr: removed all blocklist entries 2023-05-22T15:43:26.669 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:26.663+0000 7f4220891640 1 -- 172.21.15.16:0/2757912157 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f41fc0654d0 msgr2=0x7f41fc067980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:26.669 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:26.663+0000 7f4220891640 1 --2- 172.21.15.16:0/2757912157 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f41fc0654d0 0x7f41fc067980 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f421c0b1460 tx=0x7f4210009190 comp rx=0 tx=0).stop 2023-05-22T15:43:26.670 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:26.663+0000 7f4220891640 1 -- 172.21.15.16:0/2757912157 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f421c0b7dd0 msgr2=0x7f421c0b2860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:26.670 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:26.663+0000 7f4220891640 1 --2- 172.21.15.16:0/2757912157 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f421c0b7dd0 0x7f421c0b2860 secure :-1 s=READY pgs=1426 cs=0 l=1 rev1=1 crypto rx=0x7f420c00c370 tx=0x7f420c00c830 comp rx=0 tx=0).stop 2023-05-22T15:43:26.670 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:26.663+0000 7f4220891640 1 -- 172.21.15.16:0/2757912157 shutdown_connections 2023-05-22T15:43:26.671 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:26.663+0000 7f4220891640 1 --2- 172.21.15.16:0/2757912157 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f41fc0654d0 0x7f41fc067980 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:26.671 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:26.663+0000 7f4220891640 1 --2- 172.21.15.16:0/2757912157 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f421c0d0410 0x7f421c0cb550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:26.671 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:26.663+0000 7f4220891640 1 --2- 172.21.15.16:0/2757912157 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f421c0ccab0 0x7f421c0b2da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:26.671 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:26.663+0000 7f4220891640 1 --2- 172.21.15.16:0/2757912157 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f421c0b7dd0 0x7f421c0b2860 unknown :-1 s=CLOSED pgs=1426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:26.671 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:26.663+0000 7f4220891640 1 -- 172.21.15.16:0/2757912157 >> 172.21.15.16:0/2757912157 conn(0x7f421c0655c0 msgr2=0x7f421c159e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:43:26.672 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:26.663+0000 7f4220891640 1 -- 172.21.15.16:0/2757912157 shutdown_connections 2023-05-22T15:43:26.672 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:26.663+0000 7f4220891640 1 -- 172.21.15.16:0/2757912157 wait complete. 2023-05-22T15:43:26.691 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2023-05-22T15:43:26.863 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:26.859+0000 7f8177eb4640 1 Processor -- start 2023-05-22T15:43:26.864 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:26.859+0000 7f8177eb4640 1 -- start start 2023-05-22T15:43:26.864 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:26.859+0000 7f8177eb4640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f817015c130 0x7f81700b2330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:26.865 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:26.859+0000 7f8177eb4640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f81700b2870 0x7f81700cf510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:26.865 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:26.859+0000 7f8177eb4640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81700b2cc0 con 0x7f81700cfa50 2023-05-22T15:43:26.865 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:26.859+0000 7f8177eb4640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f817006c4a0 con 0x7f81700b2870 2023-05-22T15:43:26.865 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:26.859+0000 7f8177eb4640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81700ce160 con 0x7f817015c130 2023-05-22T15:43:26.865 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:26.859+0000 7f817544f640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f81700b2870 0x7f81700cf510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:26.865 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:26.859+0000 7f817544f640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f81700b2870 0x7f81700cf510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57880/0 (socket says 172.21.15.16:57880) 2023-05-22T15:43:26.865 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:26.859+0000 7f817544f640 1 -- 172.21.15.16:0/3164377496 learned_addr learned my addr 172.21.15.16:0/3164377496 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:43:26.866 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:26.859+0000 7f8175c50640 1 --2- 172.21.15.16:0/3164377496 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f817015c130 0x7f81700b2330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:26.866 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:26.859+0000 7f817544f640 1 -- 172.21.15.16:0/3164377496 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f817015c130 msgr2=0x7f81700b2330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:43:26.866 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:26.859+0000 7f817544f640 1 --2- 172.21.15.16:0/3164377496 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f817015c130 0x7f81700b2330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:26.866 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:26.859+0000 7f817544f640 1 -- 172.21.15.16:0/3164377496 >> v1:172.21.15.62:6789/0 conn(0x7f81700cfa50 legacy=0x7f81700d01b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:26.866 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:26.859+0000 7f817544f640 1 -- 172.21.15.16:0/3164377496 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81700cece0 con 0x7f81700b2870 2023-05-22T15:43:26.867 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:26.863+0000 7f817544f640 1 --2- 172.21.15.16:0/3164377496 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f81700b2870 0x7f81700cf510 secure :-1 s=READY pgs=1427 cs=0 l=1 rev1=1 crypto rx=0x7f8164009e50 tx=0x7f816401ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c34e1e55ee1dde09 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:43:26.867 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:26.863+0000 7f8174c4e640 1 -- 172.21.15.16:0/3164377496 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81640a56e0 con 0x7f81700b2870 2023-05-22T15:43:26.868 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:26.863+0000 7f8174c4e640 1 -- 172.21.15.16:0/3164377496 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81640a5840 con 0x7f81700b2870 2023-05-22T15:43:26.868 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:26.863+0000 7f8174c4e640 1 -- 172.21.15.16:0/3164377496 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81640a5bd0 con 0x7f81700b2870 2023-05-22T15:43:26.869 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:26.863+0000 7f8177eb4640 1 -- 172.21.15.16:0/3164377496 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f81700b2870 msgr2=0x7f81700cf510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:26.869 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:26.863+0000 7f8177eb4640 1 --2- 172.21.15.16:0/3164377496 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f81700b2870 0x7f81700cf510 secure :-1 s=READY pgs=1427 cs=0 l=1 rev1=1 crypto rx=0x7f8164009e50 tx=0x7f816401ca10 comp rx=0 tx=0).stop 2023-05-22T15:43:26.870 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:26.863+0000 7f8177eb4640 1 -- 172.21.15.16:0/3164377496 shutdown_connections 2023-05-22T15:43:26.870 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:26.863+0000 7f8177eb4640 1 --2- 172.21.15.16:0/3164377496 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f81700b2870 0x7f81700cf510 unknown :-1 s=CLOSED pgs=1427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:26.870 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:26.863+0000 7f8177eb4640 1 --2- 172.21.15.16:0/3164377496 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f817015c130 0x7f81700b2330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:26.870 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:26.863+0000 7f8177eb4640 1 -- 172.21.15.16:0/3164377496 >> 172.21.15.16:0/3164377496 conn(0x7f817006f5c0 msgr2=0x7f817006f9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:43:26.870 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:26.863+0000 7f8177eb4640 1 -- 172.21.15.16:0/3164377496 shutdown_connections 2023-05-22T15:43:26.871 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:26.863+0000 7f8177eb4640 1 -- 172.21.15.16:0/3164377496 wait complete. 2023-05-22T15:43:26.871 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:26.863+0000 7f8177eb4640 1 Processor -- start 2023-05-22T15:43:26.871 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:26.863+0000 7f8177eb4640 1 -- start start 2023-05-22T15:43:26.873 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:26.863+0000 7f8177eb4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f81700b2870 0x7f817015f190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:26.873 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:26.863+0000 7f8177eb4640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f81700cfa50 0x7f81700b43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:26.873 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:26.863+0000 7f8177eb4640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f817015c130 0x7f81701501d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:26.873 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:26.863+0000 7f8177eb4640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81700b48f0 con 0x7f81700b2870 2023-05-22T15:43:26.873 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:26.863+0000 7f8177eb4640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81700c86b0 con 0x7f81700cfa50 2023-05-22T15:43:26.873 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:26.863+0000 7f8177eb4640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81701526c0 con 0x7f817015c130 2023-05-22T15:43:26.873 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:26.863+0000 7f8175c50640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f81700b2870 0x7f817015f190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:26.874 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:26.863+0000 7f8175c50640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f81700b2870 0x7f817015f190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57884/0 (socket says 172.21.15.16:57884) 2023-05-22T15:43:26.874 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:26.863+0000 7f8175c50640 1 -- 172.21.15.16:0/691720190 learned_addr learned my addr 172.21.15.16:0/691720190 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:43:26.874 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:26.867+0000 7f817544f640 1 --2- 172.21.15.16:0/691720190 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f81700cfa50 0x7f81700b43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:26.874 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:26.867+0000 7f8176451640 1 --2- 172.21.15.16:0/691720190 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f817015c130 0x7f81701501d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:26.874 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:26.867+0000 7f8175c50640 1 -- 172.21.15.16:0/691720190 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f817015c130 msgr2=0x7f81701501d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:26.874 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:26.867+0000 7f8175c50640 1 --2- 172.21.15.16:0/691720190 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f817015c130 0x7f81701501d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:26.874 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:26.867+0000 7f8175c50640 1 -- 172.21.15.16:0/691720190 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f81700cfa50 msgr2=0x7f81700b43b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:26.875 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:26.867+0000 7f8175c50640 1 --2- 172.21.15.16:0/691720190 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f81700cfa50 0x7f81700b43b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:26.875 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:26.867+0000 7f8175c50640 1 -- 172.21.15.16:0/691720190 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8164009b30 con 0x7f81700b2870 2023-05-22T15:43:26.875 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:26.867+0000 7f8175c50640 1 --2- 172.21.15.16:0/691720190 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f81700b2870 0x7f817015f190 secure :-1 s=READY pgs=1428 cs=0 l=1 rev1=1 crypto rx=0x7f816000d5f0 tx=0x7f816000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:43:26.875 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:26.867+0000 7f816e7fc640 1 -- 172.21.15.16:0/691720190 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81600125f0 con 0x7f81700b2870 2023-05-22T15:43:26.875 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:26.867+0000 7f816e7fc640 1 -- 172.21.15.16:0/691720190 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8160012750 con 0x7f81700b2870 2023-05-22T15:43:26.875 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:26.867+0000 7f8177eb4640 1 -- 172.21.15.16:0/691720190 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81700ce970 con 0x7f81700b2870 2023-05-22T15:43:26.875 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:26.867+0000 7f816e7fc640 1 -- 172.21.15.16:0/691720190 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8160012ae0 con 0x7f81700b2870 2023-05-22T15:43:26.875 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:26.867+0000 7f8177eb4640 1 -- 172.21.15.16:0/691720190 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81700ce690 con 0x7f81700b2870 2023-05-22T15:43:26.876 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:26.867+0000 7f8177eb4640 1 -- 172.21.15.16:0/691720190 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81700cb8c0 con 0x7f81700b2870 2023-05-22T15:43:26.876 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:26.867+0000 7f816e7fc640 1 -- 172.21.15.16:0/691720190 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 41) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f8160012c40 con 0x7f81700b2870 2023-05-22T15:43:26.876 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:26.867+0000 7f816e7fc640 1 --2- 172.21.15.16:0/691720190 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f815c0654a0 0x7f815c067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:26.876 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:26.867+0000 7f816e7fc640 1 -- 172.21.15.16:0/691720190 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5530+0+0 (secure 0 0 0) 0x7f8160082c80 con 0x7f81700b2870 2023-05-22T15:43:26.876 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:26.867+0000 7f817544f640 1 --2- 172.21.15.16:0/691720190 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f815c0654a0 0x7f815c067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:26.877 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:26.867+0000 7f817544f640 1 --2- 172.21.15.16:0/691720190 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f815c0654a0 0x7f815c067950 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f81640ae000 tx=0x7f81640a7070 comp rx=0 tx=0).ready entity=mgr.7788 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:43:26.877 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:26.871+0000 7f816e7fc640 1 -- 172.21.15.16:0/691720190 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f8160088020 con 0x7f81700b2870 2023-05-22T15:43:27.078 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.071+0000 7f8177eb4640 1 -- 172.21.15.16:0/691720190 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f817004dbe0 con 0x7f81700b2870 2023-05-22T15:43:27.078 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.075+0000 7f816e7fc640 1 -- 172.21.15.16:0/691720190 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v21) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f8160050980 con 0x7f81700b2870 2023-05-22T15:43:27.079 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:43:27.079 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-05-22T15:43:27.079 INFO:teuthology.orchestra.run.smithi016.stdout: "auth_dump": [ 2023-05-22T15:43:27.079 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T15:43:27.079 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "mds.a", 2023-05-22T15:43:27.080 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQCAiWtkw/K0GRAA9hvTMKMmcbgKV68GeoC4ag==", 2023-05-22T15:43:27.080 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-05-22T15:43:27.080 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2023-05-22T15:43:27.080 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow *", 2023-05-22T15:43:27.080 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow *", 2023-05-22T15:43:27.080 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-05-22T15:43:27.081 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T15:43:27.081 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:43:27.081 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T15:43:27.081 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "mds.b", 2023-05-22T15:43:27.081 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQCAiWtkoFzfJRAAxCiB0MGM5apMVA81WDq+dA==", 2023-05-22T15:43:27.082 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-05-22T15:43:27.082 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2023-05-22T15:43:27.082 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow *", 2023-05-22T15:43:27.082 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow *", 2023-05-22T15:43:27.082 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-05-22T15:43:27.082 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T15:43:27.082 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:43:27.082 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T15:43:27.083 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "mds.c", 2023-05-22T15:43:27.083 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQCAiWtkvgOKHxAAMuASPtMqwyoqpZ5I+MRS7A==", 2023-05-22T15:43:27.083 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-05-22T15:43:27.083 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2023-05-22T15:43:27.083 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow *", 2023-05-22T15:43:27.083 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow *", 2023-05-22T15:43:27.083 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-05-22T15:43:27.083 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T15:43:27.084 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:43:27.084 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T15:43:27.084 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "mds.d", 2023-05-22T15:43:27.084 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQCAiWtk6hPQKxAAlCtfC36vja5ftz6uOqUhHg==", 2023-05-22T15:43:27.084 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-05-22T15:43:27.084 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2023-05-22T15:43:27.084 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow *", 2023-05-22T15:43:27.085 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow *", 2023-05-22T15:43:27.085 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-05-22T15:43:27.085 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T15:43:27.085 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:43:27.085 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T15:43:27.085 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.0", 2023-05-22T15:43:27.085 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQCCiWtk6ZLoCRAAKdCQitZuPnWcIpkiXhJeww==", 2023-05-22T15:43:27.085 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-05-22T15:43:27.086 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2023-05-22T15:43:27.086 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2023-05-22T15:43:27.086 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-05-22T15:43:27.086 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T15:43:27.086 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:43:27.086 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T15:43:27.086 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.1", 2023-05-22T15:43:27.087 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQCFiWtkXtVeERAAkgPeDKuFmWXrDiCyqWOYwQ==", 2023-05-22T15:43:27.087 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-05-22T15:43:27.087 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2023-05-22T15:43:27.087 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2023-05-22T15:43:27.087 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-05-22T15:43:27.087 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T15:43:27.087 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:43:27.087 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T15:43:27.088 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.2", 2023-05-22T15:43:27.088 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQCIiWtkdGaRHRAAx3POIp3mnz7p7XcCVMXm5Q==", 2023-05-22T15:43:27.088 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-05-22T15:43:27.088 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2023-05-22T15:43:27.088 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2023-05-22T15:43:27.088 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-05-22T15:43:27.088 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T15:43:27.089 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:43:27.089 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T15:43:27.089 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.3", 2023-05-22T15:43:27.089 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQCLiWtkSBydJhAAsgc19Xaf/GOyy9nm9MLdJQ==", 2023-05-22T15:43:27.089 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-05-22T15:43:27.089 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2023-05-22T15:43:27.090 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2023-05-22T15:43:27.090 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-05-22T15:43:27.090 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T15:43:27.090 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:43:27.090 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T15:43:27.090 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.4", 2023-05-22T15:43:27.090 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQCRiWtkueYvJRAARXbHlKHrTbgTqBArRJfvBA==", 2023-05-22T15:43:27.091 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-05-22T15:43:27.091 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2023-05-22T15:43:27.091 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2023-05-22T15:43:27.091 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-05-22T15:43:27.091 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T15:43:27.092 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:43:27.092 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T15:43:27.092 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.5", 2023-05-22T15:43:27.092 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQCUiWtkH3hwLRAAUekP6y+wkJVPh+OPDiZbzQ==", 2023-05-22T15:43:27.092 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-05-22T15:43:27.092 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2023-05-22T15:43:27.092 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2023-05-22T15:43:27.093 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-05-22T15:43:27.093 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T15:43:27.093 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:43:27.093 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T15:43:27.093 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.6", 2023-05-22T15:43:27.093 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQCXiWtkLsnBNhAAQ8ilX8ZU4ncJ2JJU0I+U4w==", 2023-05-22T15:43:27.094 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-05-22T15:43:27.094 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2023-05-22T15:43:27.094 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2023-05-22T15:43:27.094 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-05-22T15:43:27.094 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T15:43:27.094 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:43:27.094 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T15:43:27.094 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.7", 2023-05-22T15:43:27.095 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQCbiWtkUvXjBBAApMWw7wdfxWq0WEQDL2qprA==", 2023-05-22T15:43:27.095 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-05-22T15:43:27.095 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2023-05-22T15:43:27.095 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2023-05-22T15:43:27.095 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-05-22T15:43:27.095 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T15:43:27.095 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:43:27.095 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T15:43:27.096 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "client.0", 2023-05-22T15:43:27.096 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQCAiWtkZwuUMRAAnYqSFiaT/QRsajdjNgTetA==", 2023-05-22T15:43:27.096 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-05-22T15:43:27.096 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2023-05-22T15:43:27.096 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow r", 2023-05-22T15:43:27.096 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow rw tag cephfs data=cephfs" 2023-05-22T15:43:27.097 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T15:43:27.097 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:43:27.097 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T15:43:27.097 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "client.1", 2023-05-22T15:43:27.097 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQCAiWtkPimmORAABzNL2rANz5TgCl3p++GVng==", 2023-05-22T15:43:27.097 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-05-22T15:43:27.098 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2023-05-22T15:43:27.098 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow r", 2023-05-22T15:43:27.098 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow rw", 2023-05-22T15:43:27.098 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow rwx" 2023-05-22T15:43:27.098 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T15:43:27.098 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:43:27.098 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T15:43:27.098 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "client.2", 2023-05-22T15:43:27.099 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQCAiWtkqkPMNhAAqugrnD4Mh/4At8QAFmzqcg==", 2023-05-22T15:43:27.099 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-05-22T15:43:27.099 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2023-05-22T15:43:27.099 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow r", 2023-05-22T15:43:27.099 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow rw", 2023-05-22T15:43:27.099 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow rwx" 2023-05-22T15:43:27.100 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T15:43:27.100 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:43:27.100 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T15:43:27.100 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "client.3", 2023-05-22T15:43:27.100 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQCBiWtkpvlWAxAAPuedivXBGZGHIO6hSnz8PQ==", 2023-05-22T15:43:27.100 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-05-22T15:43:27.101 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2023-05-22T15:43:27.101 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow r", 2023-05-22T15:43:27.101 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow rw", 2023-05-22T15:43:27.101 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow rwx" 2023-05-22T15:43:27.101 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T15:43:27.101 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:43:27.101 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T15:43:27.101 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "client.admin", 2023-05-22T15:43:27.102 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQCAiWtkxG9hAhAAfnUnRrBRUFoCEtsb2riNOQ==", 2023-05-22T15:43:27.102 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-05-22T15:43:27.102 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow *", 2023-05-22T15:43:27.102 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow *", 2023-05-22T15:43:27.102 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow *", 2023-05-22T15:43:27.103 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-05-22T15:43:27.103 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T15:43:27.103 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:43:27.103 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T15:43:27.103 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "mgr.x", 2023-05-22T15:43:27.103 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQCAiWtkkaIqFhAAb/4BI6FYWOy3qOSaylXLSw==", 2023-05-22T15:43:27.103 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-05-22T15:43:27.104 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow *", 2023-05-22T15:43:27.104 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile mgr", 2023-05-22T15:43:27.104 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-05-22T15:43:27.104 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T15:43:27.104 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:43:27.104 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T15:43:27.104 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "mgr.y", 2023-05-22T15:43:27.105 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQCAiWtkhgVWExAAjDoh7cexFVfsigsYfGkmVg==", 2023-05-22T15:43:27.105 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-05-22T15:43:27.105 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow *", 2023-05-22T15:43:27.105 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile mgr", 2023-05-22T15:43:27.105 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-05-22T15:43:27.105 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T15:43:27.105 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T15:43:27.105 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-05-22T15:43:27.106 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-05-22T15:43:27.107 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.075+0000 7f8177eb4640 1 -- 172.21.15.16:0/691720190 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f815c0654a0 msgr2=0x7f815c067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:27.107 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.075+0000 7f8177eb4640 1 --2- 172.21.15.16:0/691720190 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f815c0654a0 0x7f815c067950 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f81640ae000 tx=0x7f81640a7070 comp rx=0 tx=0).stop 2023-05-22T15:43:27.107 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.075+0000 7f8177eb4640 1 -- 172.21.15.16:0/691720190 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f81700b2870 msgr2=0x7f817015f190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:27.108 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.075+0000 7f8177eb4640 1 --2- 172.21.15.16:0/691720190 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f81700b2870 0x7f817015f190 secure :-1 s=READY pgs=1428 cs=0 l=1 rev1=1 crypto rx=0x7f816000d5f0 tx=0x7f816000dab0 comp rx=0 tx=0).stop 2023-05-22T15:43:27.108 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.075+0000 7f8177eb4640 1 -- 172.21.15.16:0/691720190 shutdown_connections 2023-05-22T15:43:27.108 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.075+0000 7f8177eb4640 1 --2- 172.21.15.16:0/691720190 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f815c0654a0 0x7f815c067950 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:27.108 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.075+0000 7f8177eb4640 1 --2- 172.21.15.16:0/691720190 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f817015c130 0x7f81701501d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:27.108 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.075+0000 7f8177eb4640 1 --2- 172.21.15.16:0/691720190 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f81700cfa50 0x7f81700b43b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:27.108 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.075+0000 7f8177eb4640 1 --2- 172.21.15.16:0/691720190 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f81700b2870 0x7f817015f190 unknown :-1 s=CLOSED pgs=1428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:27.108 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.075+0000 7f8177eb4640 1 -- 172.21.15.16:0/691720190 >> 172.21.15.16:0/691720190 conn(0x7f817006f5c0 msgr2=0x7f81700b1680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:43:27.109 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.075+0000 7f8177eb4640 1 -- 172.21.15.16:0/691720190 shutdown_connections 2023-05-22T15:43:27.109 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.075+0000 7f8177eb4640 1 -- 172.21.15.16:0/691720190 wait complete. 2023-05-22T15:43:27.110 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-05-22T15:43:27.235 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.231+0000 7f4bb6da4640 1 Processor -- start 2023-05-22T15:43:27.236 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.231+0000 7f4bb6da4640 1 -- start start 2023-05-22T15:43:27.237 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.231+0000 7f4bb6da4640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4bb00ca4e0 0x7f4bb015e460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:27.237 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.231+0000 7f4bb6da4640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f4bb00c9b40 0x7f4bb00afb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:27.237 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.231+0000 7f4bb6da4640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4bb00b0650 con 0x7f4bb0070df0 2023-05-22T15:43:27.237 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.231+0000 7f4bb6da4640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4bb00b0370 con 0x7f4bb00c9b40 2023-05-22T15:43:27.237 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.231+0000 7f4bb6da4640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4bb015d980 con 0x7f4bb00ca4e0 2023-05-22T15:43:27.237 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.231+0000 7f4baffff640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f4bb00c9b40 0x7f4bb00afb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:27.238 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.231+0000 7f4baffff640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f4bb00c9b40 0x7f4bb00afb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57900/0 (socket says 172.21.15.16:57900) 2023-05-22T15:43:27.238 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.231+0000 7f4baffff640 1 -- 172.21.15.16:0/3315203560 learned_addr learned my addr 172.21.15.16:0/3315203560 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:43:27.238 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.231+0000 7f4bb4b40640 1 --2- 172.21.15.16:0/3315203560 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4bb00ca4e0 0x7f4bb015e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:27.239 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.231+0000 7f4baffff640 1 -- 172.21.15.16:0/3315203560 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4bb00ca4e0 msgr2=0x7f4bb015e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:43:27.239 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.231+0000 7f4baffff640 1 --2- 172.21.15.16:0/3315203560 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4bb00ca4e0 0x7f4bb015e460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:27.239 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.231+0000 7f4baffff640 1 -- 172.21.15.16:0/3315203560 >> v1:172.21.15.62:6789/0 conn(0x7f4bb0070df0 legacy=0x7f4bb0071210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:27.239 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.231+0000 7f4baffff640 1 -- 172.21.15.16:0/3315203560 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bb015dc60 con 0x7f4bb00c9b40 2023-05-22T15:43:27.241 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.235+0000 7f4baffff640 1 --2- 172.21.15.16:0/3315203560 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4bb00c9b40 0x7f4bb00afb40 secure :-1 s=READY pgs=1429 cs=0 l=1 rev1=1 crypto rx=0x7f4ba4009870 tx=0x7f4ba401c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb31905f79386de7 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:43:27.241 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.235+0000 7f4baf7fe640 1 -- 172.21.15.16:0/3315203560 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ba40ab660 con 0x7f4bb00c9b40 2023-05-22T15:43:27.241 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.235+0000 7f4baf7fe640 1 -- 172.21.15.16:0/3315203560 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ba40ab7c0 con 0x7f4bb00c9b40 2023-05-22T15:43:27.241 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.235+0000 7f4baf7fe640 1 -- 172.21.15.16:0/3315203560 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ba40abb50 con 0x7f4bb00c9b40 2023-05-22T15:43:27.243 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.235+0000 7f4bb6da4640 1 -- 172.21.15.16:0/3315203560 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4bb00c9b40 msgr2=0x7f4bb00afb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:27.243 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.235+0000 7f4bb6da4640 1 --2- 172.21.15.16:0/3315203560 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4bb00c9b40 0x7f4bb00afb40 secure :-1 s=READY pgs=1429 cs=0 l=1 rev1=1 crypto rx=0x7f4ba4009870 tx=0x7f4ba401c480 comp rx=0 tx=0).stop 2023-05-22T15:43:27.244 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.235+0000 7f4bb6da4640 1 -- 172.21.15.16:0/3315203560 shutdown_connections 2023-05-22T15:43:27.244 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.235+0000 7f4bb6da4640 1 --2- 172.21.15.16:0/3315203560 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4bb00c9b40 0x7f4bb00afb40 unknown :-1 s=CLOSED pgs=1429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:27.244 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.235+0000 7f4bb6da4640 1 --2- 172.21.15.16:0/3315203560 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4bb00ca4e0 0x7f4bb015e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:27.245 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.235+0000 7f4bb6da4640 1 -- 172.21.15.16:0/3315203560 >> 172.21.15.16:0/3315203560 conn(0x7f4bb0069720 msgr2=0x7f4bb0069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:43:27.245 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.235+0000 7f4bb6da4640 1 -- 172.21.15.16:0/3315203560 shutdown_connections 2023-05-22T15:43:27.245 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.235+0000 7f4bb6da4640 1 -- 172.21.15.16:0/3315203560 wait complete. 2023-05-22T15:43:27.245 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.239+0000 7f4bb6da4640 1 Processor -- start 2023-05-22T15:43:27.246 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.239+0000 7f4bb6da4640 1 -- start start 2023-05-22T15:43:27.249 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.239+0000 7f4bb6da4640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4bb0070df0 0x7f4bb014c400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:27.249 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.239+0000 7f4bb6da4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4bb00c9b40 0x7f4bb00aeb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:27.250 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.239+0000 7f4bb6da4640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4bb00ca4e0 0x7f4bb00af090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:27.250 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.239+0000 7f4bb6da4640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bb0150690 con 0x7f4bb00c9b40 2023-05-22T15:43:27.250 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.239+0000 7f4bb6da4640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bb00c6f40 con 0x7f4bb0070df0 2023-05-22T15:43:27.251 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.239+0000 7f4bb6da4640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4bb00c7220 con 0x7f4bb00ca4e0 2023-05-22T15:43:27.251 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.239+0000 7f4baffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4bb00c9b40 0x7f4bb00aeb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:27.251 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.239+0000 7f4baffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4bb00c9b40 0x7f4bb00aeb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57908/0 (socket says 172.21.15.16:57908) 2023-05-22T15:43:27.251 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.239+0000 7f4baffff640 1 -- 172.21.15.16:0/13391752 learned_addr learned my addr 172.21.15.16:0/13391752 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:43:27.251 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.239+0000 7f4bb4b40640 1 --2- 172.21.15.16:0/13391752 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4bb0070df0 0x7f4bb014c400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:27.252 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.239+0000 7f4bb5341640 1 --2- 172.21.15.16:0/13391752 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4bb00ca4e0 0x7f4bb00af090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:27.252 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.239+0000 7f4baffff640 1 -- 172.21.15.16:0/13391752 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4bb00ca4e0 msgr2=0x7f4bb00af090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:27.252 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.239+0000 7f4baffff640 1 --2- 172.21.15.16:0/13391752 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4bb00ca4e0 0x7f4bb00af090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:27.252 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.239+0000 7f4baffff640 1 -- 172.21.15.16:0/13391752 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4bb0070df0 msgr2=0x7f4bb014c400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:27.252 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.239+0000 7f4baffff640 1 --2- 172.21.15.16:0/13391752 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4bb0070df0 0x7f4bb014c400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:27.253 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.239+0000 7f4baffff640 1 -- 172.21.15.16:0/13391752 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ba40094d0 con 0x7f4bb00c9b40 2023-05-22T15:43:27.253 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.239+0000 7f4bb5341640 1 --2- 172.21.15.16:0/13391752 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4bb00ca4e0 0x7f4bb00af090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T15:43:27.253 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.239+0000 7f4baffff640 1 --2- 172.21.15.16:0/13391752 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4bb00c9b40 0x7f4bb00aeb50 secure :-1 s=READY pgs=1430 cs=0 l=1 rev1=1 crypto rx=0x7f4ba40ad040 tx=0x7f4ba401acc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:43:27.253 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.239+0000 7f4bad7fa640 1 -- 172.21.15.16:0/13391752 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ba40ab660 con 0x7f4bb00c9b40 2023-05-22T15:43:27.253 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.239+0000 7f4bad7fa640 1 -- 172.21.15.16:0/13391752 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ba40b4820 con 0x7f4bb00c9b40 2023-05-22T15:43:27.253 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.239+0000 7f4bb6da4640 1 -- 172.21.15.16:0/13391752 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4bb00c8e80 con 0x7f4bb00c9b40 2023-05-22T15:43:27.253 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.239+0000 7f4bad7fa640 1 -- 172.21.15.16:0/13391752 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ba40b4bd0 con 0x7f4bb00c9b40 2023-05-22T15:43:27.254 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.239+0000 7f4bb6da4640 1 -- 172.21.15.16:0/13391752 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4bb00ca8b0 con 0x7f4bb00c9b40 2023-05-22T15:43:27.254 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.243+0000 7f4bb6da4640 1 -- 172.21.15.16:0/13391752 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4bb00c9010 con 0x7f4bb00c9b40 2023-05-22T15:43:27.254 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.243+0000 7f4bad7fa640 1 -- 172.21.15.16:0/13391752 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 41) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f4ba40a9070 con 0x7f4bb00c9b40 2023-05-22T15:43:27.254 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.243+0000 7f4bad7fa640 1 --2- 172.21.15.16:0/13391752 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f4b9c0654a0 0x7f4b9c067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:27.254 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.243+0000 7f4bad7fa640 1 -- 172.21.15.16:0/13391752 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5530+0+0 (secure 0 0 0) 0x7f4ba411d3f0 con 0x7f4bb00c9b40 2023-05-22T15:43:27.254 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.243+0000 7f4bb4b40640 1 --2- 172.21.15.16:0/13391752 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f4b9c0654a0 0x7f4b9c067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:27.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.243+0000 7f4bb4b40640 1 --2- 172.21.15.16:0/13391752 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f4b9c0654a0 0x7f4b9c067950 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f4ba00079c0 tx=0x7f4ba0008040 comp rx=0 tx=0).ready entity=mgr.7788 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:43:27.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.247+0000 7f4bad7fa640 1 -- 172.21.15.16:0/13391752 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f4ba40eb1a0 con 0x7f4bb00c9b40 2023-05-22T15:43:27.409 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.403+0000 7f4bb6da4640 1 -- 172.21.15.16:0/13391752 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4bb01507d0 con 0x7f4bb00c9b40 2023-05-22T15:43:27.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.407+0000 7f4bad7fa640 1 -- 172.21.15.16:0/13391752 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v114) v1 ==== 74+0+14338 (secure 0 0 0) 0x7f4bb01507d0 con 0x7f4bb00c9b40 2023-05-22T15:43:27.411 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:43:27.411 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":114,"fsid":"b2719834-7847-4d98-90a6-f65d57f85c49","created":"2023-05-22T15:26:25.259314+0000","modified":"2023-05-22T15:43:26.655845+0000","last_up_change":"2023-05-22T15:26:36.448524+0000","last_in_change":"2023-05-22T15:26:27.675668+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-05-22T15:26:38.019781+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-05-22T15:26:43.570514+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9e7ff854-b25f-4cbb-a5c2-1799d9c53d2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":102,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":37021},{"type":"v1","addr":"172.21.15.16:6825","nonce":37021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":37021},{"type":"v1","addr":"172.21.15.16:6827","nonce":37021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":37021},{"type":"v1","addr":"172.21.15.16:6831","nonce":37021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":37021},{"type":"v1","addr":"172.21.15.16:6829","nonce":37021}]},"public_addr":"172.21.15.16:6825/37021","cluster_addr":"172.21.15.16:6827/37021","heartbeat_back_addr":"172.21.15.16:6831/37021","heartbeat_front_addr":"172.21.15.16:6829/37021","state":["exists","up"]},{"osd":1,"uuid":"4702fbc9-a927-4565-aca5-41dc69cc8264","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":102,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":37008},{"type":"v1","addr":"172.21.15.16:6801","nonce":37008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":37008},{"type":"v1","addr":"172.21.15.16:6803","nonce":37008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":37008},{"type":"v1","addr":"172.21.15.16:6807","nonce":37008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":37008},{"type":"v1","addr":"172.21.15.16:6805","nonce":37008}]},"public_addr":"172.21.15.16:6801/37008","cluster_addr":"172.21.15.16:6803/37008","heartbeat_back_addr":"172.21.15.16:6807/37008","heartbeat_front_addr":"172.21.15.16:6805/37008","state":["exists","up"]},{"osd":2,"uuid":"1e7bba46-4eba-47ef-bb81-5e2d371638ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":102,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":37009},{"type":"v1","addr":"172.21.15.16:6809","nonce":37009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":37009},{"type":"v1","addr":"172.21.15.16:6811","nonce":37009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":37009},{"type":"v1","addr":"172.21.15.16:6815","nonce":37009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":37009},{"type":"v1","addr":"172.21.15.16:6813","nonce":37009}]},"public_addr":"172.21.15.16:6809/37009","cluster_addr":"172.21.15.16:6811/37009","heartbeat_back_addr":"172.21.15.16:6815/37009","heartbeat_front_addr":"172.21.15.16:6813/37009","state":["exists","up"]},{"osd":3,"uuid":"383cacc0-5a88-4904-990e-498ca59ef48a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":102,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":37020},{"type":"v1","addr":"172.21.15.16:6817","nonce":37020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":37020},{"type":"v1","addr":"172.21.15.16:6819","nonce":37020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":37020},{"type":"v1","addr":"172.21.15.16:6823","nonce":37020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":37020},{"type":"v1","addr":"172.21.15.16:6821","nonce":37020}]},"public_addr":"172.21.15.16:6817/37020","cluster_addr":"172.21.15.16:6819/37020","heartbeat_back_addr":"172.21.15.16:6823/37020","heartbeat_front_addr":"172.21.15.16:6821/37020","state":["exists","up"]},{"osd":4,"uuid":"970500ef-95b7-42c1-b080-eed4fae7013d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":102,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":37353},{"type":"v1","addr":"172.21.15.62:6817","nonce":37353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6819","nonce":37353},{"type":"v1","addr":"172.21.15.62:6822","nonce":37353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37353},{"type":"v1","addr":"172.21.15.62:6831","nonce":37353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37353},{"type":"v1","addr":"172.21.15.62:6826","nonce":37353}]},"public_addr":"172.21.15.62:6817/37353","cluster_addr":"172.21.15.62:6822/37353","heartbeat_back_addr":"172.21.15.62:6831/37353","heartbeat_front_addr":"172.21.15.62:6826/37353","state":["exists","up"]},{"osd":5,"uuid":"022d67db-d4d8-445b-b1b9-8b446a00f5c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":102,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":37352},{"type":"v1","addr":"172.21.15.62:6801","nonce":37352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":37352},{"type":"v1","addr":"172.21.15.62:6803","nonce":37352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":37352},{"type":"v1","addr":"172.21.15.62:6807","nonce":37352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":37352},{"type":"v1","addr":"172.21.15.62:6805","nonce":37352}]},"public_addr":"172.21.15.62:6801/37352","cluster_addr":"172.21.15.62:6803/37352","heartbeat_back_addr":"172.21.15.62:6807/37352","heartbeat_front_addr":"172.21.15.62:6805/37352","state":["exists","up"]},{"osd":6,"uuid":"42bd372b-6688-4a1b-8407-e16e8c266e33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":102,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":37354},{"type":"v1","addr":"172.21.15.62:6820","nonce":37354}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":37354},{"type":"v1","addr":"172.21.15.62:6823","nonce":37354}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6829","nonce":37354},{"type":"v1","addr":"172.21.15.62:6830","nonce":37354}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6825","nonce":37354},{"type":"v1","addr":"172.21.15.62:6827","nonce":37354}]},"public_addr":"172.21.15.62:6820/37354","cluster_addr":"172.21.15.62:6823/37354","heartbeat_back_addr":"172.21.15.62:6830/37354","heartbeat_front_addr":"172.21.15.62:6827/37354","state":["exists","up"]},{"osd":7,"uuid":"ac70b2a3-f41b-4bf2-b227-7f2cbb55426d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":102,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37355},{"type":"v1","addr":"172.21.15.62:6809","nonce":37355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37355},{"type":"v1","addr":"172.21.15.62:6811","nonce":37355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":37355},{"type":"v1","addr":"172.21.15.62:6815","nonce":37355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":37355},{"type":"v1","addr":"172.21.15.62:6813","nonce":37355}]},"public_addr":"172.21.15.62:6809/37355","cluster_addr":"172.21.15.62:6811/37355","heartbeat_back_addr":"172.21.15.62:6815/37355","heartbeat_front_addr":"172.21.15.62:6813/37355","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.368896+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.107273+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.276888+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.287502+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.124791+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.251937+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.255679+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.232797+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-05-22T15:43:27.413 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.407+0000 7f4bb6da4640 1 -- 172.21.15.16:0/13391752 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f4b9c0654a0 msgr2=0x7f4b9c067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:27.413 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.407+0000 7f4bb6da4640 1 --2- 172.21.15.16:0/13391752 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f4b9c0654a0 0x7f4b9c067950 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f4ba00079c0 tx=0x7f4ba0008040 comp rx=0 tx=0).stop 2023-05-22T15:43:27.414 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.407+0000 7f4bb6da4640 1 -- 172.21.15.16:0/13391752 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4bb00c9b40 msgr2=0x7f4bb00aeb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:27.414 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.407+0000 7f4bb6da4640 1 --2- 172.21.15.16:0/13391752 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4bb00c9b40 0x7f4bb00aeb50 secure :-1 s=READY pgs=1430 cs=0 l=1 rev1=1 crypto rx=0x7f4ba40ad040 tx=0x7f4ba401acc0 comp rx=0 tx=0).stop 2023-05-22T15:43:27.414 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.407+0000 7f4bb6da4640 1 -- 172.21.15.16:0/13391752 shutdown_connections 2023-05-22T15:43:27.414 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.407+0000 7f4bb6da4640 1 --2- 172.21.15.16:0/13391752 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f4b9c0654a0 0x7f4b9c067950 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:27.414 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.407+0000 7f4bb6da4640 1 --2- 172.21.15.16:0/13391752 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4bb00ca4e0 0x7f4bb00af090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:27.414 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.407+0000 7f4bb6da4640 1 --2- 172.21.15.16:0/13391752 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4bb00c9b40 0x7f4bb00aeb50 unknown :-1 s=CLOSED pgs=1430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:27.414 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.407+0000 7f4bb6da4640 1 --2- 172.21.15.16:0/13391752 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4bb0070df0 0x7f4bb014c400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:27.414 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.407+0000 7f4bb6da4640 1 -- 172.21.15.16:0/13391752 >> 172.21.15.16:0/13391752 conn(0x7f4bb0069720 msgr2=0x7f4bb0159460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:43:27.415 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.407+0000 7f4bb6da4640 1 -- 172.21.15.16:0/13391752 shutdown_connections 2023-05-22T15:43:27.415 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.407+0000 7f4bb6da4640 1 -- 172.21.15.16:0/13391752 wait complete. 2023-05-22T15:43:27.430 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-05-22T15:26:38.019781+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-05-22T15:26:43.570514+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-05-22T15:43:27.431 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-05-22T15:43:27.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.547+0000 7f8fc080c640 1 Processor -- start 2023-05-22T15:43:27.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.547+0000 7f8fc080c640 1 -- start start 2023-05-22T15:43:27.554 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.547+0000 7f8fc080c640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f8fbc0b3a70 0x7f8fbc15d750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:27.555 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.547+0000 7f8fc080c640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fbc0c9e70 con 0x7f8fbc0ce370 2023-05-22T15:43:27.555 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.547+0000 7f8fc080c640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fbc0ca150 con 0x7f8fbc0cdfa0 2023-05-22T15:43:27.555 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.547+0000 7f8fc080c640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8fbc0cba20 con 0x7f8fbc0b3a70 2023-05-22T15:43:27.555 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.547+0000 7f8fbad9d640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f8fbc0ce370 0x7f8fbc06b7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:54044/0 (socket says 172.21.15.16:54044) 2023-05-22T15:43:27.556 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.547+0000 7f8fbad9d640 1 -- 172.21.15.16:0/749476475 learned_addr learned my addr 172.21.15.16:0/749476475 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:43:27.556 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.547+0000 7f8fb9d9b640 1 --2- 172.21.15.16:0/749476475 >> v2:172.21.15.62:3300/0 conn(0x7f8fbc0b3a70 0x7f8fbc15d750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:27.556 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.547+0000 7f8fb959a640 1 -- 172.21.15.16:0/749476475 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2958166961 0 0) 0x7f8fbc0c9e70 con 0x7f8fbc0ce370 2023-05-22T15:43:27.556 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.547+0000 7f8fb959a640 1 -- 172.21.15.16:0/749476475 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8fa4003530 con 0x7f8fbc0ce370 2023-05-22T15:43:27.556 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.547+0000 7f8fb9d9b640 1 -- 172.21.15.16:0/749476475 >> v1:172.21.15.62:6789/0 conn(0x7f8fbc0cdfa0 legacy=0x7f8fbc0cb260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:27.557 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.547+0000 7f8fb9d9b640 1 -- 172.21.15.16:0/749476475 >> v1:172.21.15.16:6789/0 conn(0x7f8fbc0ce370 legacy=0x7f8fbc06b7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:27.557 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.547+0000 7f8fb9d9b640 1 -- 172.21.15.16:0/749476475 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fbc0ca430 con 0x7f8fbc0b3a70 2023-05-22T15:43:27.561 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.551+0000 7f8fb9d9b640 1 --2- 172.21.15.16:0/749476475 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8fbc0b3a70 0x7f8fbc15d750 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f8fb00097c0 tx=0x7f8fb001c480 comp rx=0 tx=0).ready entity=mon.1 client_cookie=89d98237a684637c server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:43:27.561 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.551+0000 7f8fb959a640 1 -- 172.21.15.16:0/749476475 <== mon.1 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fb00ab600 con 0x7f8fbc0b3a70 2023-05-22T15:43:27.561 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.551+0000 7f8fb959a640 1 -- 172.21.15.16:0/749476475 <== mon.1 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fb00a7050 con 0x7f8fbc0b3a70 2023-05-22T15:43:27.561 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.551+0000 7f8fb959a640 1 -- 172.21.15.16:0/749476475 <== mon.1 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fb00ab960 con 0x7f8fbc0b3a70 2023-05-22T15:43:27.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.551+0000 7f8fc080c640 1 -- 172.21.15.16:0/749476475 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8fbc0b3a70 msgr2=0x7f8fbc15d750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:27.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.551+0000 7f8fc080c640 1 --2- 172.21.15.16:0/749476475 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8fbc0b3a70 0x7f8fbc15d750 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f8fb00097c0 tx=0x7f8fb001c480 comp rx=0 tx=0).stop 2023-05-22T15:43:27.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.551+0000 7f8fc080c640 1 -- 172.21.15.16:0/749476475 shutdown_connections 2023-05-22T15:43:27.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.551+0000 7f8fc080c640 1 --2- 172.21.15.16:0/749476475 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8fbc0b3a70 0x7f8fbc15d750 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:27.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.551+0000 7f8fc080c640 1 -- 172.21.15.16:0/749476475 >> 172.21.15.16:0/749476475 conn(0x7f8fbc069720 msgr2=0x7f8fbc069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:43:27.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.551+0000 7f8fc080c640 1 -- 172.21.15.16:0/749476475 shutdown_connections 2023-05-22T15:43:27.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.551+0000 7f8fc080c640 1 -- 172.21.15.16:0/749476475 wait complete. 2023-05-22T15:43:27.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.551+0000 7f8fc080c640 1 Processor -- start 2023-05-22T15:43:27.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.551+0000 7f8fc080c640 1 -- start start 2023-05-22T15:43:27.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.555+0000 7f8fc080c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8fbc0b3a70 0x7f8fbc15e880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:27.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.555+0000 7f8fc080c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8fbc0cdfa0 0x7f8fbc15cd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:27.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.555+0000 7f8fc080c640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8fbc0ce370 0x7f8fbc145380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:27.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.555+0000 7f8fc080c640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fbc0cb320 con 0x7f8fbc0cdfa0 2023-05-22T15:43:27.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.555+0000 7f8fc080c640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fbc15eed0 con 0x7f8fbc0b3a70 2023-05-22T15:43:27.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.555+0000 7f8fc080c640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fbc15f1b0 con 0x7f8fbc0ce370 2023-05-22T15:43:27.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.555+0000 7f8fb9d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8fbc0cdfa0 0x7f8fbc15cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:27.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.555+0000 7f8fb9d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8fbc0cdfa0 0x7f8fbc15cd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57922/0 (socket says 172.21.15.16:57922) 2023-05-22T15:43:27.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.555+0000 7f8fb9d9b640 1 -- 172.21.15.16:0/1699729573 learned_addr learned my addr 172.21.15.16:0/1699729573 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:43:27.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.555+0000 7f8fba59c640 1 --2- 172.21.15.16:0/1699729573 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8fbc0b3a70 0x7f8fbc15e880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:27.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.555+0000 7f8fbad9d640 1 --2- 172.21.15.16:0/1699729573 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8fbc0ce370 0x7f8fbc145380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:27.565 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.555+0000 7f8fb9d9b640 1 -- 172.21.15.16:0/1699729573 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8fbc0ce370 msgr2=0x7f8fbc145380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:27.565 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.555+0000 7f8fb9d9b640 1 --2- 172.21.15.16:0/1699729573 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8fbc0ce370 0x7f8fbc145380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:27.565 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.555+0000 7f8fb9d9b640 1 -- 172.21.15.16:0/1699729573 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8fbc0b3a70 msgr2=0x7f8fbc15e880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:27.565 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.555+0000 7f8fb9d9b640 1 --2- 172.21.15.16:0/1699729573 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8fbc0b3a70 0x7f8fbc15e880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:27.565 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.555+0000 7f8fb9d9b640 1 -- 172.21.15.16:0/1699729573 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fb0009420 con 0x7f8fbc0cdfa0 2023-05-22T15:43:27.565 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.555+0000 7f8fb9d9b640 1 --2- 172.21.15.16:0/1699729573 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8fbc0cdfa0 0x7f8fbc15cd20 secure :-1 s=READY pgs=1432 cs=0 l=1 rev1=1 crypto rx=0x7f8fb0019090 tx=0x7f8fb0019220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:43:27.565 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.555+0000 7f8faaffd640 1 -- 172.21.15.16:0/1699729573 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fb0003af0 con 0x7f8fbc0cdfa0 2023-05-22T15:43:27.565 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.555+0000 7f8faaffd640 1 -- 172.21.15.16:0/1699729573 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fb0003c50 con 0x7f8fbc0cdfa0 2023-05-22T15:43:27.566 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.555+0000 7f8fc080c640 1 -- 172.21.15.16:0/1699729573 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8fbc0ce900 con 0x7f8fbc0cdfa0 2023-05-22T15:43:27.566 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.555+0000 7f8faaffd640 1 -- 172.21.15.16:0/1699729573 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fb00ab400 con 0x7f8fbc0cdfa0 2023-05-22T15:43:27.566 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.555+0000 7f8fc080c640 1 -- 172.21.15.16:0/1699729573 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8fbc0ce770 con 0x7f8fbc0cdfa0 2023-05-22T15:43:27.566 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.555+0000 7f8fc080c640 1 -- 172.21.15.16:0/1699729573 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8fbc06c2e0 con 0x7f8fbc0cdfa0 2023-05-22T15:43:27.567 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.555+0000 7f8faaffd640 1 -- 172.21.15.16:0/1699729573 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 41) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f8fb00b2040 con 0x7f8fbc0cdfa0 2023-05-22T15:43:27.567 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.559+0000 7f8faaffd640 1 --2- 172.21.15.16:0/1699729573 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f8f9c0654d0 0x7f8f9c067980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:27.567 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.559+0000 7f8faaffd640 1 -- 172.21.15.16:0/1699729573 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5530+0+0 (secure 0 0 0) 0x7f8fb011d2a0 con 0x7f8fbc0cdfa0 2023-05-22T15:43:27.568 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.559+0000 7f8fba59c640 1 --2- 172.21.15.16:0/1699729573 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f8f9c0654d0 0x7f8f9c067980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:27.568 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.559+0000 7f8fba59c640 1 --2- 172.21.15.16:0/1699729573 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f8f9c0654d0 0x7f8f9c067980 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f8fac001650 tx=0x7f8fac009040 comp rx=0 tx=0).ready entity=mgr.7788 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:43:27.568 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.559+0000 7f8faaffd640 1 -- 172.21.15.16:0/1699729573 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f8fb00eaef0 con 0x7f8fbc0cdfa0 2023-05-22T15:43:27.720 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.715+0000 7f8fc080c640 1 -- 172.21.15.16:0/1699729573 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f8fbc04dbe0 con 0x7f8fbc0cdfa0 2023-05-22T15:43:27.720 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.715+0000 7f8faaffd640 1 -- 172.21.15.16:0/1699729573 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v114) v1 ==== 93+0+10 (secure 0 0 0) 0x7f8fb00ee510 con 0x7f8fbc0cdfa0 2023-05-22T15:43:27.721 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2023-05-22T15:43:27.723 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.719+0000 7f8fc080c640 1 -- 172.21.15.16:0/1699729573 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f8f9c0654d0 msgr2=0x7f8f9c067980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:27.723 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.719+0000 7f8fc080c640 1 --2- 172.21.15.16:0/1699729573 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f8f9c0654d0 0x7f8f9c067980 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f8fac001650 tx=0x7f8fac009040 comp rx=0 tx=0).stop 2023-05-22T15:43:27.723 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.719+0000 7f8fc080c640 1 -- 172.21.15.16:0/1699729573 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8fbc0cdfa0 msgr2=0x7f8fbc15cd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:27.723 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.719+0000 7f8fc080c640 1 --2- 172.21.15.16:0/1699729573 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8fbc0cdfa0 0x7f8fbc15cd20 secure :-1 s=READY pgs=1432 cs=0 l=1 rev1=1 crypto rx=0x7f8fb0019090 tx=0x7f8fb0019220 comp rx=0 tx=0).stop 2023-05-22T15:43:27.724 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.719+0000 7f8fc080c640 1 -- 172.21.15.16:0/1699729573 shutdown_connections 2023-05-22T15:43:27.724 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.719+0000 7f8fc080c640 1 --2- 172.21.15.16:0/1699729573 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f8f9c0654d0 0x7f8f9c067980 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:27.724 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.719+0000 7f8fc080c640 1 --2- 172.21.15.16:0/1699729573 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8fbc0ce370 0x7f8fbc145380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:27.724 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.719+0000 7f8fc080c640 1 --2- 172.21.15.16:0/1699729573 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8fbc0cdfa0 0x7f8fbc15cd20 unknown :-1 s=CLOSED pgs=1432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:27.724 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.719+0000 7f8fc080c640 1 --2- 172.21.15.16:0/1699729573 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8fbc0b3a70 0x7f8fbc15e880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:27.725 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.719+0000 7f8fc080c640 1 -- 172.21.15.16:0/1699729573 >> 172.21.15.16:0/1699729573 conn(0x7f8fbc069720 msgr2=0x7f8fbc0cbf80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:43:27.725 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.719+0000 7f8fc080c640 1 -- 172.21.15.16:0/1699729573 shutdown_connections 2023-05-22T15:43:27.725 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.719+0000 7f8fc080c640 1 -- 172.21.15.16:0/1699729573 wait complete. 2023-05-22T15:43:27.740 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-05-22T15:43:27.859 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.855+0000 7f20320f2640 1 Processor -- start 2023-05-22T15:43:27.859 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.855+0000 7f20320f2640 1 -- start start 2023-05-22T15:43:27.860 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.855+0000 7f20320f2640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f202c159970 0x7f202c159d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:27.860 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.855+0000 7f20320f2640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f202c15d680 0x7f202c0cb260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:27.860 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.855+0000 7f20320f2640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f202c15dad0 0x7f202c06b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:27.860 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.855+0000 7f20320f2640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f202c070c30 con 0x7f202c15dad0 2023-05-22T15:43:27.860 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.855+0000 7f20320f2640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f202c070e60 con 0x7f202c15d680 2023-05-22T15:43:27.860 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.855+0000 7f20320f2640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f202c071140 con 0x7f202c159970 2023-05-22T15:43:27.861 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.855+0000 7f202bfff640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f202c15dad0 0x7f202c06b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:27.861 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.855+0000 7f202bfff640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f202c15dad0 0x7f202c06b6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57930/0 (socket says 172.21.15.16:57930) 2023-05-22T15:43:27.861 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.855+0000 7f202bfff640 1 -- 172.21.15.16:0/3881395972 learned_addr learned my addr 172.21.15.16:0/3881395972 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:43:27.862 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.855+0000 7f2022ffd640 1 --2- 172.21.15.16:0/3881395972 >> v2:172.21.15.62:3300/0 conn(0x7f202c15d680 0x7f202c0cb260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:27.862 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.855+0000 7f202b7fe640 1 --2- 172.21.15.16:0/3881395972 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f202c159970 0x7f202c159d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:27.862 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.855+0000 7f202bfff640 1 -- 172.21.15.16:0/3881395972 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f202c159970 msgr2=0x7f202c159d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:43:27.862 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.855+0000 7f202bfff640 1 --2- 172.21.15.16:0/3881395972 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f202c159970 0x7f202c159d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:27.862 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.855+0000 7f202bfff640 1 -- 172.21.15.16:0/3881395972 >> v2:172.21.15.62:3300/0 conn(0x7f202c15d680 msgr2=0x7f202c0cb260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:43:27.862 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.855+0000 7f202bfff640 1 --2- 172.21.15.16:0/3881395972 >> v2:172.21.15.62:3300/0 conn(0x7f202c15d680 0x7f202c0cb260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:27.863 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.855+0000 7f202bfff640 1 -- 172.21.15.16:0/3881395972 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f202c071420 con 0x7f202c15dad0 2023-05-22T15:43:27.864 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.859+0000 7f202bfff640 1 --2- 172.21.15.16:0/3881395972 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f202c15dad0 0x7f202c06b6c0 secure :-1 s=READY pgs=1433 cs=0 l=1 rev1=1 crypto rx=0x7f202c071cf0 tx=0x7f202401c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6256a3fbc2f6c650 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:43:27.864 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.859+0000 7f202affd640 1 -- 172.21.15.16:0/3881395972 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20240a56a0 con 0x7f202c15dad0 2023-05-22T15:43:27.864 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.859+0000 7f202affd640 1 -- 172.21.15.16:0/3881395972 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20240a5800 con 0x7f202c15dad0 2023-05-22T15:43:27.864 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.859+0000 7f202affd640 1 -- 172.21.15.16:0/3881395972 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20240a5b90 con 0x7f202c15dad0 2023-05-22T15:43:27.864 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.859+0000 7f20320f2640 1 -- 172.21.15.16:0/3881395972 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f202c15dad0 msgr2=0x7f202c06b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:27.865 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.859+0000 7f20320f2640 1 --2- 172.21.15.16:0/3881395972 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f202c15dad0 0x7f202c06b6c0 secure :-1 s=READY pgs=1433 cs=0 l=1 rev1=1 crypto rx=0x7f202c071cf0 tx=0x7f202401c910 comp rx=0 tx=0).stop 2023-05-22T15:43:27.865 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.859+0000 7f20320f2640 1 -- 172.21.15.16:0/3881395972 shutdown_connections 2023-05-22T15:43:27.865 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.859+0000 7f20320f2640 1 --2- 172.21.15.16:0/3881395972 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f202c15dad0 0x7f202c06b6c0 unknown :-1 s=CLOSED pgs=1433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:27.865 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.859+0000 7f20320f2640 1 --2- 172.21.15.16:0/3881395972 >> v2:172.21.15.62:3300/0 conn(0x7f202c15d680 0x7f202c0cb260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:27.865 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.859+0000 7f20320f2640 1 --2- 172.21.15.16:0/3881395972 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f202c159970 0x7f202c159d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:27.865 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.859+0000 7f20320f2640 1 -- 172.21.15.16:0/3881395972 >> 172.21.15.16:0/3881395972 conn(0x7f202c069720 msgr2=0x7f202c069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:43:27.866 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.859+0000 7f20320f2640 1 -- 172.21.15.16:0/3881395972 shutdown_connections 2023-05-22T15:43:27.866 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.859+0000 7f20320f2640 1 -- 172.21.15.16:0/3881395972 wait complete. 2023-05-22T15:43:27.867 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.859+0000 7f20320f2640 1 Processor -- start 2023-05-22T15:43:27.867 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.859+0000 7f20320f2640 1 -- start start 2023-05-22T15:43:27.868 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.859+0000 7f20320f2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f202c159970 0x7f202c14fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:27.868 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.859+0000 7f20320f2640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f202c15d680 0x7f202c14e3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:27.868 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.859+0000 7f20320f2640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f202c15dad0 0x7f202c0c27b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:27.868 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.859+0000 7f20320f2640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f202c14f7c0 con 0x7f202c159970 2023-05-22T15:43:27.868 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.859+0000 7f20320f2640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f202c14e8f0 con 0x7f202c15dad0 2023-05-22T15:43:27.868 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.859+0000 7f20320f2640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f202c14ebb0 con 0x7f202c15d680 2023-05-22T15:43:27.868 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.859+0000 7f202b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f202c159970 0x7f202c14fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:27.869 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.859+0000 7f202b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f202c159970 0x7f202c14fd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57938/0 (socket says 172.21.15.16:57938) 2023-05-22T15:43:27.869 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.859+0000 7f202b7fe640 1 -- 172.21.15.16:0/4176257821 learned_addr learned my addr 172.21.15.16:0/4176257821 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:43:27.869 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.859+0000 7f202bfff640 1 --2- 172.21.15.16:0/4176257821 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f202c15dad0 0x7f202c0c27b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:27.869 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.859+0000 7f2022ffd640 1 --2- 172.21.15.16:0/4176257821 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f202c15d680 0x7f202c14e3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:27.869 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.863+0000 7f202b7fe640 1 -- 172.21.15.16:0/4176257821 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f202c15d680 msgr2=0x7f202c14e3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:27.869 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.863+0000 7f202b7fe640 1 --2- 172.21.15.16:0/4176257821 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f202c15d680 0x7f202c14e3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:27.869 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.863+0000 7f202b7fe640 1 -- 172.21.15.16:0/4176257821 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f202c15dad0 msgr2=0x7f202c0c27b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:27.870 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.863+0000 7f202b7fe640 1 --2- 172.21.15.16:0/4176257821 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f202c15dad0 0x7f202c0c27b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:27.870 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.863+0000 7f202b7fe640 1 -- 172.21.15.16:0/4176257821 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2024009b30 con 0x7f202c159970 2023-05-22T15:43:27.870 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.863+0000 7f202b7fe640 1 --2- 172.21.15.16:0/4176257821 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f202c159970 0x7f202c14fd60 secure :-1 s=READY pgs=1434 cs=0 l=1 rev1=1 crypto rx=0x7f201c00d800 tx=0x7f201c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:43:27.870 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.863+0000 7f2028ff9640 1 -- 172.21.15.16:0/4176257821 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f201c0096a0 con 0x7f202c159970 2023-05-22T15:43:27.870 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.863+0000 7f2028ff9640 1 -- 172.21.15.16:0/4176257821 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f201c009800 con 0x7f202c159970 2023-05-22T15:43:27.870 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.863+0000 7f20320f2640 1 -- 172.21.15.16:0/4176257821 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f202c0cae20 con 0x7f202c159970 2023-05-22T15:43:27.870 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.863+0000 7f2028ff9640 1 -- 172.21.15.16:0/4176257821 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f201c009b70 con 0x7f202c159970 2023-05-22T15:43:27.871 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.863+0000 7f20320f2640 1 -- 172.21.15.16:0/4176257821 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f202c0cac90 con 0x7f202c159970 2023-05-22T15:43:27.871 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.863+0000 7f20320f2640 1 -- 172.21.15.16:0/4176257821 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f202c15adc0 con 0x7f202c159970 2023-05-22T15:43:27.872 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.863+0000 7f2028ff9640 1 -- 172.21.15.16:0/4176257821 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 41) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f201c009cd0 con 0x7f202c159970 2023-05-22T15:43:27.872 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.863+0000 7f2028ff9640 1 --2- 172.21.15.16:0/4176257821 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f200c0654a0 0x7f200c067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:27.872 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.863+0000 7f2028ff9640 1 -- 172.21.15.16:0/4176257821 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5530+0+0 (secure 0 0 0) 0x7f201c082f00 con 0x7f202c159970 2023-05-22T15:43:27.872 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.863+0000 7f2022ffd640 1 --2- 172.21.15.16:0/4176257821 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f200c0654a0 0x7f200c067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:27.872 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.863+0000 7f2022ffd640 1 --2- 172.21.15.16:0/4176257821 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f200c0654a0 0x7f200c067950 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f202c158650 tx=0x7f2014008040 comp rx=0 tx=0).ready entity=mgr.7788 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:43:27.873 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:27.867+0000 7f2028ff9640 1 -- 172.21.15.16:0/4176257821 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f201c050c80 con 0x7f202c159970 2023-05-22T15:43:28.025 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.019+0000 7f20320f2640 1 -- 172.21.15.16:0/4176257821 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f202c04dbe0 con 0x7f202c159970 2023-05-22T15:43:28.026 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.023+0000 7f2028ff9640 1 -- 172.21.15.16:0/4176257821 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v114) v1 ==== 92+0+10 (secure 0 0 0) 0x7f201c0542a0 con 0x7f202c159970 2023-05-22T15:43:28.026 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 8 2023-05-22T15:43:28.028 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.023+0000 7f20320f2640 1 -- 172.21.15.16:0/4176257821 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f200c0654a0 msgr2=0x7f200c067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:28.029 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.023+0000 7f20320f2640 1 --2- 172.21.15.16:0/4176257821 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f200c0654a0 0x7f200c067950 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f202c158650 tx=0x7f2014008040 comp rx=0 tx=0).stop 2023-05-22T15:43:28.029 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.023+0000 7f20320f2640 1 -- 172.21.15.16:0/4176257821 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f202c159970 msgr2=0x7f202c14fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:28.029 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.023+0000 7f20320f2640 1 --2- 172.21.15.16:0/4176257821 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f202c159970 0x7f202c14fd60 secure :-1 s=READY pgs=1434 cs=0 l=1 rev1=1 crypto rx=0x7f201c00d800 tx=0x7f201c00dcc0 comp rx=0 tx=0).stop 2023-05-22T15:43:28.029 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.023+0000 7f20320f2640 1 -- 172.21.15.16:0/4176257821 shutdown_connections 2023-05-22T15:43:28.029 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.023+0000 7f20320f2640 1 --2- 172.21.15.16:0/4176257821 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f202c15dad0 0x7f202c0c27b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:28.029 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.023+0000 7f20320f2640 1 --2- 172.21.15.16:0/4176257821 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f200c0654a0 0x7f200c067950 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:28.030 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.023+0000 7f20320f2640 1 --2- 172.21.15.16:0/4176257821 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f202c15d680 0x7f202c14e3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:28.030 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.023+0000 7f20320f2640 1 --2- 172.21.15.16:0/4176257821 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f202c159970 0x7f202c14fd60 unknown :-1 s=CLOSED pgs=1434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:28.030 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.023+0000 7f20320f2640 1 -- 172.21.15.16:0/4176257821 >> 172.21.15.16:0/4176257821 conn(0x7f202c069720 msgr2=0x7f202c0c9790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:43:28.030 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.023+0000 7f20320f2640 1 -- 172.21.15.16:0/4176257821 shutdown_connections 2023-05-22T15:43:28.030 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.023+0000 7f20320f2640 1 -- 172.21.15.16:0/4176257821 wait complete. 2023-05-22T15:43:28.046 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-05-22T15:43:28.171 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.167+0000 7f2e159da640 1 Processor -- start 2023-05-22T15:43:28.171 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.167+0000 7f2e159da640 1 -- start start 2023-05-22T15:43:28.172 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.167+0000 7f2e159da640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2e100b7dd0 0x7f2e100ccab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:28.172 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.167+0000 7f2e159da640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f2e1015f700 0x7f2e1015fb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:28.173 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.167+0000 7f2e159da640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2e10160050 0x7f2e100ce720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:28.173 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.167+0000 7f2e159da640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e100d3360 con 0x7f2e10160050 2023-05-22T15:43:28.173 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.167+0000 7f2e159da640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e100cd170 con 0x7f2e1015f700 2023-05-22T15:43:28.173 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.167+0000 7f2e159da640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e100d2ac0 con 0x7f2e100b7dd0 2023-05-22T15:43:28.173 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.167+0000 7f2e0f7fe640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2e10160050 0x7f2e100ce720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:28.173 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.167+0000 7f2e0f7fe640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2e10160050 0x7f2e100ce720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57940/0 (socket says 172.21.15.16:57940) 2023-05-22T15:43:28.174 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.167+0000 7f2e0f7fe640 1 -- 172.21.15.16:0/3354184196 learned_addr learned my addr 172.21.15.16:0/3354184196 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:43:28.174 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.167+0000 7f2e0e7fc640 1 --2- 172.21.15.16:0/3354184196 >> v2:172.21.15.62:3300/0 conn(0x7f2e1015f700 0x7f2e1015fb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:28.174 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.167+0000 7f2e0effd640 1 --2- 172.21.15.16:0/3354184196 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2e100b7dd0 0x7f2e100ccab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:28.175 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.167+0000 7f2e0f7fe640 1 -- 172.21.15.16:0/3354184196 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2e100b7dd0 msgr2=0x7f2e100ccab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:43:28.175 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.167+0000 7f2e0f7fe640 1 --2- 172.21.15.16:0/3354184196 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2e100b7dd0 0x7f2e100ccab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:28.175 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.167+0000 7f2e0f7fe640 1 -- 172.21.15.16:0/3354184196 >> v2:172.21.15.62:3300/0 conn(0x7f2e1015f700 msgr2=0x7f2e1015fb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:43:28.175 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.167+0000 7f2e0f7fe640 1 --2- 172.21.15.16:0/3354184196 >> v2:172.21.15.62:3300/0 conn(0x7f2e1015f700 0x7f2e1015fb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:28.176 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.167+0000 7f2e0f7fe640 1 -- 172.21.15.16:0/3354184196 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e100d2da0 con 0x7f2e10160050 2023-05-22T15:43:28.176 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.167+0000 7f2e0e7fc640 1 --2- 172.21.15.16:0/3354184196 >> v2:172.21.15.62:3300/0 conn(0x7f2e1015f700 0x7f2e1015fb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T15:43:28.176 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.171+0000 7f2e0f7fe640 1 --2- 172.21.15.16:0/3354184196 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e10160050 0x7f2e100ce720 secure :-1 s=READY pgs=1435 cs=0 l=1 rev1=1 crypto rx=0x7f2e100d34c0 tx=0x7f2e0801c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63c1bd5079e29957 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:43:28.176 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.171+0000 7f2e0dffb640 1 -- 172.21.15.16:0/3354184196 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e080aa610 con 0x7f2e10160050 2023-05-22T15:43:28.176 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.171+0000 7f2e0dffb640 1 -- 172.21.15.16:0/3354184196 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e080a6040 con 0x7f2e10160050 2023-05-22T15:43:28.176 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.171+0000 7f2e0dffb640 1 -- 172.21.15.16:0/3354184196 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e080aa990 con 0x7f2e10160050 2023-05-22T15:43:28.177 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.171+0000 7f2e159da640 1 -- 172.21.15.16:0/3354184196 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e10160050 msgr2=0x7f2e100ce720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:28.177 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.171+0000 7f2e159da640 1 --2- 172.21.15.16:0/3354184196 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e10160050 0x7f2e100ce720 secure :-1 s=READY pgs=1435 cs=0 l=1 rev1=1 crypto rx=0x7f2e100d34c0 tx=0x7f2e0801c4d0 comp rx=0 tx=0).stop 2023-05-22T15:43:28.178 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.171+0000 7f2e159da640 1 -- 172.21.15.16:0/3354184196 shutdown_connections 2023-05-22T15:43:28.178 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.171+0000 7f2e159da640 1 --2- 172.21.15.16:0/3354184196 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e10160050 0x7f2e100ce720 unknown :-1 s=CLOSED pgs=1435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:28.178 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.171+0000 7f2e159da640 1 --2- 172.21.15.16:0/3354184196 >> v2:172.21.15.62:3300/0 conn(0x7f2e1015f700 0x7f2e1015fb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:28.178 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.171+0000 7f2e159da640 1 --2- 172.21.15.16:0/3354184196 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2e100b7dd0 0x7f2e100ccab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:28.178 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.171+0000 7f2e159da640 1 -- 172.21.15.16:0/3354184196 >> 172.21.15.16:0/3354184196 conn(0x7f2e100655c0 msgr2=0x7f2e10070e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:43:28.178 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.171+0000 7f2e159da640 1 -- 172.21.15.16:0/3354184196 shutdown_connections 2023-05-22T15:43:28.178 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.171+0000 7f2e159da640 1 -- 172.21.15.16:0/3354184196 wait complete. 2023-05-22T15:43:28.179 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.171+0000 7f2e159da640 1 Processor -- start 2023-05-22T15:43:28.179 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.171+0000 7f2e159da640 1 -- start start 2023-05-22T15:43:28.180 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.171+0000 7f2e159da640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2e100b7dd0 0x7f2e100b3440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:28.180 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.171+0000 7f2e159da640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e1015f700 0x7f2e100c5b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:28.181 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.171+0000 7f2e159da640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2e10160050 0x7f2e10157310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:28.181 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.171+0000 7f2e159da640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e100c7140 con 0x7f2e1015f700 2023-05-22T15:43:28.181 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.171+0000 7f2e159da640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e100c6530 con 0x7f2e100b7dd0 2023-05-22T15:43:28.181 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.171+0000 7f2e159da640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e100c6810 con 0x7f2e10160050 2023-05-22T15:43:28.181 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.171+0000 7f2e0e7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e1015f700 0x7f2e100c5b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:28.181 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.171+0000 7f2e0e7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e1015f700 0x7f2e100c5b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57944/0 (socket says 172.21.15.16:57944) 2023-05-22T15:43:28.182 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.171+0000 7f2e0e7fc640 1 -- 172.21.15.16:0/593083943 learned_addr learned my addr 172.21.15.16:0/593083943 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:43:28.182 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.175+0000 7f2e0effd640 1 --2- 172.21.15.16:0/593083943 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2e100b7dd0 0x7f2e100b3440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:28.182 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.175+0000 7f2e0f7fe640 1 --2- 172.21.15.16:0/593083943 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2e10160050 0x7f2e10157310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:28.182 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.175+0000 7f2e0e7fc640 1 -- 172.21.15.16:0/593083943 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2e10160050 msgr2=0x7f2e10157310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:28.182 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.175+0000 7f2e0e7fc640 1 --2- 172.21.15.16:0/593083943 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2e10160050 0x7f2e10157310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:28.182 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.175+0000 7f2e0e7fc640 1 -- 172.21.15.16:0/593083943 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2e100b7dd0 msgr2=0x7f2e100b3440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:28.182 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.175+0000 7f2e0e7fc640 1 --2- 172.21.15.16:0/593083943 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2e100b7dd0 0x7f2e100b3440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:28.183 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.175+0000 7f2e0e7fc640 1 -- 172.21.15.16:0/593083943 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e080094d0 con 0x7f2e1015f700 2023-05-22T15:43:28.183 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.175+0000 7f2e0e7fc640 1 --2- 172.21.15.16:0/593083943 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e1015f700 0x7f2e100c5b70 secure :-1 s=READY pgs=1436 cs=0 l=1 rev1=1 crypto rx=0x7f2e00009fc0 tx=0x7f2e0000d560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:43:28.183 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.175+0000 7f2df7fff640 1 -- 172.21.15.16:0/593083943 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e00003ba0 con 0x7f2e1015f700 2023-05-22T15:43:28.183 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.175+0000 7f2df7fff640 1 -- 172.21.15.16:0/593083943 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e0000f040 con 0x7f2e1015f700 2023-05-22T15:43:28.183 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.175+0000 7f2e159da640 1 -- 172.21.15.16:0/593083943 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e100cf600 con 0x7f2e1015f700 2023-05-22T15:43:28.183 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.175+0000 7f2df7fff640 1 -- 172.21.15.16:0/593083943 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e00004030 con 0x7f2e1015f700 2023-05-22T15:43:28.183 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.175+0000 7f2e159da640 1 -- 172.21.15.16:0/593083943 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e100cf470 con 0x7f2e1015f700 2023-05-22T15:43:28.183 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.175+0000 7f2e159da640 1 -- 172.21.15.16:0/593083943 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e100cf790 con 0x7f2e1015f700 2023-05-22T15:43:28.184 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.175+0000 7f2df7fff640 1 -- 172.21.15.16:0/593083943 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 41) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f2e00004190 con 0x7f2e1015f700 2023-05-22T15:43:28.184 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.175+0000 7f2df7fff640 1 --2- 172.21.15.16:0/593083943 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f2dfc0654a0 0x7f2dfc067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:28.184 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.175+0000 7f2df7fff640 1 -- 172.21.15.16:0/593083943 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5530+0+0 (secure 0 0 0) 0x7f2e00082c20 con 0x7f2e1015f700 2023-05-22T15:43:28.185 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.175+0000 7f2e0effd640 1 --2- 172.21.15.16:0/593083943 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f2dfc0654a0 0x7f2dfc067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:28.185 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.175+0000 7f2e0effd640 1 --2- 172.21.15.16:0/593083943 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f2dfc0654a0 0x7f2dfc067950 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f2e0400ba50 tx=0x7f2e04008040 comp rx=0 tx=0).ready entity=mgr.7788 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:43:28.185 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.179+0000 7f2df7fff640 1 -- 172.21.15.16:0/593083943 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f2e0004d620 con 0x7f2e1015f700 2023-05-22T15:43:28.337 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.331+0000 7f2e159da640 1 -- 172.21.15.16:0/593083943 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f2e1004dbe0 con 0x7f2e1015f700 2023-05-22T15:43:28.337 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.331+0000 7f2df7fff640 1 -- 172.21.15.16:0/593083943 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v114) v1 ==== 84+0+126 (secure 0 0 0) 0x7f2e0001a020 con 0x7f2e1015f700 2023-05-22T15:43:28.338 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:43:28.338 INFO:teuthology.orchestra.run.smithi016.stdout:[ 2023-05-22T15:43:28.338 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T15:43:28.338 INFO:teuthology.orchestra.run.smithi016.stdout: "poolnum": 1, 2023-05-22T15:43:28.338 INFO:teuthology.orchestra.run.smithi016.stdout: "poolname": ".mgr" 2023-05-22T15:43:28.338 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:43:28.339 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T15:43:28.339 INFO:teuthology.orchestra.run.smithi016.stdout: "poolnum": 2, 2023-05-22T15:43:28.339 INFO:teuthology.orchestra.run.smithi016.stdout: "poolname": "rbd" 2023-05-22T15:43:28.339 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T15:43:28.339 INFO:teuthology.orchestra.run.smithi016.stdout:] 2023-05-22T15:43:28.340 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.335+0000 7f2e159da640 1 -- 172.21.15.16:0/593083943 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f2dfc0654a0 msgr2=0x7f2dfc067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:28.340 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.335+0000 7f2e159da640 1 --2- 172.21.15.16:0/593083943 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f2dfc0654a0 0x7f2dfc067950 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f2e0400ba50 tx=0x7f2e04008040 comp rx=0 tx=0).stop 2023-05-22T15:43:28.340 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.335+0000 7f2e159da640 1 -- 172.21.15.16:0/593083943 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e1015f700 msgr2=0x7f2e100c5b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:28.341 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.335+0000 7f2e159da640 1 --2- 172.21.15.16:0/593083943 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e1015f700 0x7f2e100c5b70 secure :-1 s=READY pgs=1436 cs=0 l=1 rev1=1 crypto rx=0x7f2e00009fc0 tx=0x7f2e0000d560 comp rx=0 tx=0).stop 2023-05-22T15:43:28.341 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.335+0000 7f2e159da640 1 -- 172.21.15.16:0/593083943 shutdown_connections 2023-05-22T15:43:28.341 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.335+0000 7f2e159da640 1 --2- 172.21.15.16:0/593083943 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f2dfc0654a0 0x7f2dfc067950 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:28.341 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.335+0000 7f2e159da640 1 --2- 172.21.15.16:0/593083943 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2e10160050 0x7f2e10157310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:28.341 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.335+0000 7f2e159da640 1 --2- 172.21.15.16:0/593083943 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e1015f700 0x7f2e100c5b70 unknown :-1 s=CLOSED pgs=1436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:28.341 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.335+0000 7f2e159da640 1 --2- 172.21.15.16:0/593083943 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2e100b7dd0 0x7f2e100b3440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:28.341 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.335+0000 7f2e159da640 1 -- 172.21.15.16:0/593083943 >> 172.21.15.16:0/593083943 conn(0x7f2e100655c0 msgr2=0x7f2e100b6b60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:43:28.341 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.335+0000 7f2e159da640 1 -- 172.21.15.16:0/593083943 shutdown_connections 2023-05-22T15:43:28.342 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.335+0000 7f2e159da640 1 -- 172.21.15.16:0/593083943 wait complete. 2023-05-22T15:43:28.355 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-05-22T15:43:28.355 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-05-22T15:43:28.470 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.463+0000 7fb355b0b640 1 Processor -- start 2023-05-22T15:43:28.470 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.467+0000 7fb355b0b640 1 -- start start 2023-05-22T15:43:28.471 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.467+0000 7fb355b0b640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fb350070c10 0x7fb3500cc4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:28.471 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.467+0000 7fb355b0b640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb3500c9e40 0x7fb35006bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:28.472 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.467+0000 7fb355b0b640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb35015d650 0x7fb35015dab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:28.472 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.467+0000 7fb355b0b640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb35015dff0 con 0x7fb3500c9e40 2023-05-22T15:43:28.472 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.467+0000 7fb355b0b640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3500ccbb0 con 0x7fb350070c10 2023-05-22T15:43:28.472 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.467+0000 7fb355b0b640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3500cd7c0 con 0x7fb35015d650 2023-05-22T15:43:28.472 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.467+0000 7fb34effd640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb3500c9e40 0x7fb35006bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:28.472 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.467+0000 7fb34effd640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb3500c9e40 0x7fb35006bfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57960/0 (socket says 172.21.15.16:57960) 2023-05-22T15:43:28.473 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.467+0000 7fb34effd640 1 -- 172.21.15.16:0/686602029 learned_addr learned my addr 172.21.15.16:0/686602029 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:43:28.473 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.467+0000 7fb34f7fe640 1 --2- 172.21.15.16:0/686602029 >> v2:172.21.15.62:3300/0 conn(0x7fb350070c10 0x7fb3500cc4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:28.473 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.467+0000 7fb34ffff640 1 --2- 172.21.15.16:0/686602029 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb35015d650 0x7fb35015dab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:28.474 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.467+0000 7fb34effd640 1 -- 172.21.15.16:0/686602029 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb35015d650 msgr2=0x7fb35015dab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:43:28.474 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.467+0000 7fb34effd640 1 --2- 172.21.15.16:0/686602029 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb35015d650 0x7fb35015dab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:28.474 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.467+0000 7fb34effd640 1 -- 172.21.15.16:0/686602029 >> v2:172.21.15.62:3300/0 conn(0x7fb350070c10 msgr2=0x7fb3500cc4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:43:28.474 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.467+0000 7fb34effd640 1 --2- 172.21.15.16:0/686602029 >> v2:172.21.15.62:3300/0 conn(0x7fb350070c10 0x7fb3500cc4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:28.474 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.467+0000 7fb34effd640 1 -- 172.21.15.16:0/686602029 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3500cdaa0 con 0x7fb3500c9e40 2023-05-22T15:43:28.477 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.471+0000 7fb34effd640 1 --2- 172.21.15.16:0/686602029 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb3500c9e40 0x7fb35006bfa0 secure :-1 s=READY pgs=1437 cs=0 l=1 rev1=1 crypto rx=0x7fb3500cf110 tx=0x7fb34401c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d42e1ff1efb17c16 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:43:28.477 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.471+0000 7fb34e7fc640 1 -- 172.21.15.16:0/686602029 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3440aa610 con 0x7fb3500c9e40 2023-05-22T15:43:28.478 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.471+0000 7fb34e7fc640 1 -- 172.21.15.16:0/686602029 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3440a6040 con 0x7fb3500c9e40 2023-05-22T15:43:28.478 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.471+0000 7fb34e7fc640 1 -- 172.21.15.16:0/686602029 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3440aa990 con 0x7fb3500c9e40 2023-05-22T15:43:28.478 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.471+0000 7fb355b0b640 1 -- 172.21.15.16:0/686602029 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb3500c9e40 msgr2=0x7fb35006bfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:28.479 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.471+0000 7fb355b0b640 1 --2- 172.21.15.16:0/686602029 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb3500c9e40 0x7fb35006bfa0 secure :-1 s=READY pgs=1437 cs=0 l=1 rev1=1 crypto rx=0x7fb3500cf110 tx=0x7fb34401c4d0 comp rx=0 tx=0).stop 2023-05-22T15:43:28.479 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.475+0000 7fb355b0b640 1 -- 172.21.15.16:0/686602029 shutdown_connections 2023-05-22T15:43:28.479 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.475+0000 7fb355b0b640 1 --2- 172.21.15.16:0/686602029 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb3500c9e40 0x7fb35006bfa0 unknown :-1 s=CLOSED pgs=1437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:28.479 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.475+0000 7fb355b0b640 1 --2- 172.21.15.16:0/686602029 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb35015d650 0x7fb35015dab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:28.479 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.475+0000 7fb355b0b640 1 --2- 172.21.15.16:0/686602029 >> v2:172.21.15.62:3300/0 conn(0x7fb350070c10 0x7fb3500cc4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:28.479 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.475+0000 7fb355b0b640 1 -- 172.21.15.16:0/686602029 >> 172.21.15.16:0/686602029 conn(0x7fb350069720 msgr2=0x7fb350069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:43:28.479 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.475+0000 7fb355b0b640 1 -- 172.21.15.16:0/686602029 shutdown_connections 2023-05-22T15:43:28.480 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.475+0000 7fb355b0b640 1 -- 172.21.15.16:0/686602029 wait complete. 2023-05-22T15:43:28.480 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.475+0000 7fb355b0b640 1 Processor -- start 2023-05-22T15:43:28.480 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.475+0000 7fb355b0b640 1 -- start start 2023-05-22T15:43:28.482 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.475+0000 7fb355b0b640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb350070c10 0x7fb35006b680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:28.482 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.475+0000 7fb355b0b640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb3500c9e40 0x7fb35014fc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:28.482 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.475+0000 7fb355b0b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb35015d650 0x7fb3501371e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:28.482 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.475+0000 7fb355b0b640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb350153d40 con 0x7fb35015d650 2023-05-22T15:43:28.482 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.475+0000 7fb355b0b640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb35006bbc0 con 0x7fb3500c9e40 2023-05-22T15:43:28.482 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.475+0000 7fb355b0b640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3500c7ea0 con 0x7fb350070c10 2023-05-22T15:43:28.482 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.475+0000 7fb34ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb35015d650 0x7fb3501371e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:28.483 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.475+0000 7fb34ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb35015d650 0x7fb3501371e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57976/0 (socket says 172.21.15.16:57976) 2023-05-22T15:43:28.483 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.475+0000 7fb34ffff640 1 -- 172.21.15.16:0/4284909931 learned_addr learned my addr 172.21.15.16:0/4284909931 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:43:28.483 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.475+0000 7fb34f7fe640 1 --2- 172.21.15.16:0/4284909931 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb350070c10 0x7fb35006b680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:28.483 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.475+0000 7fb34effd640 1 --2- 172.21.15.16:0/4284909931 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb3500c9e40 0x7fb35014fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:28.483 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.475+0000 7fb34ffff640 1 -- 172.21.15.16:0/4284909931 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb350070c10 msgr2=0x7fb35006b680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:28.483 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.475+0000 7fb34ffff640 1 --2- 172.21.15.16:0/4284909931 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb350070c10 0x7fb35006b680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:28.483 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.475+0000 7fb34ffff640 1 -- 172.21.15.16:0/4284909931 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb3500c9e40 msgr2=0x7fb35014fc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:28.484 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.475+0000 7fb34ffff640 1 --2- 172.21.15.16:0/4284909931 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb3500c9e40 0x7fb35014fc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:28.484 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.475+0000 7fb34ffff640 1 -- 172.21.15.16:0/4284909931 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3440094d0 con 0x7fb35015d650 2023-05-22T15:43:28.484 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.475+0000 7fb34ffff640 1 --2- 172.21.15.16:0/4284909931 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb35015d650 0x7fb3501371e0 secure :-1 s=READY pgs=1438 cs=0 l=1 rev1=1 crypto rx=0x7fb348007e60 tx=0x7fb34800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:43:28.484 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.475+0000 7fb337fff640 1 -- 172.21.15.16:0/4284909931 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb348014420 con 0x7fb35015d650 2023-05-22T15:43:28.484 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.475+0000 7fb337fff640 1 -- 172.21.15.16:0/4284909931 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3480145a0 con 0x7fb35015d650 2023-05-22T15:43:28.484 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.475+0000 7fb337fff640 1 -- 172.21.15.16:0/4284909931 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb348014950 con 0x7fb35015d650 2023-05-22T15:43:28.485 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.475+0000 7fb355b0b640 1 -- 172.21.15.16:0/4284909931 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb35006c4e0 con 0x7fb35015d650 2023-05-22T15:43:28.485 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.475+0000 7fb355b0b640 1 -- 172.21.15.16:0/4284909931 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3500ca2c0 con 0x7fb35015d650 2023-05-22T15:43:28.485 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.479+0000 7fb355b0b640 1 -- 172.21.15.16:0/4284909931 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3500cce90 con 0x7fb35015d650 2023-05-22T15:43:28.486 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.479+0000 7fb337fff640 1 -- 172.21.15.16:0/4284909931 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 41) v1 ==== 84332+0+0 (secure 0 0 0) 0x7fb348014ab0 con 0x7fb35015d650 2023-05-22T15:43:28.486 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.479+0000 7fb337fff640 1 --2- 172.21.15.16:0/4284909931 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fb33c0654d0 0x7fb33c067980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:28.486 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.479+0000 7fb337fff640 1 -- 172.21.15.16:0/4284909931 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5530+0+0 (secure 0 0 0) 0x7fb3480830d0 con 0x7fb35015d650 2023-05-22T15:43:28.486 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.479+0000 7fb34f7fe640 1 --2- 172.21.15.16:0/4284909931 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fb33c0654d0 0x7fb33c067980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:28.486 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.479+0000 7fb34f7fe640 1 --2- 172.21.15.16:0/4284909931 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fb33c0654d0 0x7fb33c067980 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fb3501546d0 tx=0x7fb340008040 comp rx=0 tx=0).ready entity=mgr.7788 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:43:28.488 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.483+0000 7fb337fff640 1 -- 172.21.15.16:0/4284909931 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fb348050db0 con 0x7fb35015d650 2023-05-22T15:43:28.641 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.635+0000 7fb355b0b640 1 -- 172.21.15.16:0/4284909931 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fb35004dbe0 con 0x7fb35015d650 2023-05-22T15:43:28.746 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.739+0000 7fb337fff640 1 -- 172.21.15.16:0/4284909931 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v115) v1 ==== 138+0+0 (secure 0 0 0) 0x7fb3480543d0 con 0x7fb35015d650 2023-05-22T15:43:28.746 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'cephfs_metadata' created 2023-05-22T15:43:28.753 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.747+0000 7fb355b0b640 1 -- 172.21.15.16:0/4284909931 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fb33c0654d0 msgr2=0x7fb33c067980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:28.753 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.747+0000 7fb355b0b640 1 --2- 172.21.15.16:0/4284909931 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fb33c0654d0 0x7fb33c067980 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fb3501546d0 tx=0x7fb340008040 comp rx=0 tx=0).stop 2023-05-22T15:43:28.753 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.747+0000 7fb355b0b640 1 -- 172.21.15.16:0/4284909931 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb35015d650 msgr2=0x7fb3501371e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:28.754 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.747+0000 7fb355b0b640 1 --2- 172.21.15.16:0/4284909931 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb35015d650 0x7fb3501371e0 secure :-1 s=READY pgs=1438 cs=0 l=1 rev1=1 crypto rx=0x7fb348007e60 tx=0x7fb34800e4b0 comp rx=0 tx=0).stop 2023-05-22T15:43:28.754 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.747+0000 7fb355b0b640 1 -- 172.21.15.16:0/4284909931 shutdown_connections 2023-05-22T15:43:28.754 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.747+0000 7fb355b0b640 1 --2- 172.21.15.16:0/4284909931 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb35015d650 0x7fb3501371e0 unknown :-1 s=CLOSED pgs=1438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:28.754 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.747+0000 7fb355b0b640 1 --2- 172.21.15.16:0/4284909931 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb3500c9e40 0x7fb35014fc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:28.754 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.747+0000 7fb355b0b640 1 --2- 172.21.15.16:0/4284909931 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fb33c0654d0 0x7fb33c067980 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:28.755 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.747+0000 7fb355b0b640 1 --2- 172.21.15.16:0/4284909931 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb350070c10 0x7fb35006b680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:28.755 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.747+0000 7fb355b0b640 1 -- 172.21.15.16:0/4284909931 >> 172.21.15.16:0/4284909931 conn(0x7fb350069720 msgr2=0x7fb350158f10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:43:28.755 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.747+0000 7fb355b0b640 1 -- 172.21.15.16:0/4284909931 shutdown_connections 2023-05-22T15:43:28.756 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.751+0000 7fb355b0b640 1 -- 172.21.15.16:0/4284909931 wait complete. 2023-05-22T15:43:28.777 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-05-22T15:43:28.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.907+0000 7fb0c1756640 1 Processor -- start 2023-05-22T15:43:28.911 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.907+0000 7fb0c1756640 1 -- start start 2023-05-22T15:43:28.912 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.907+0000 7fb0c1756640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fb0bc160170 0x7fb0bc0c8660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:28.912 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.907+0000 7fb0c1756640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0bc164090 con 0x7fb0bc0ca530 2023-05-22T15:43:28.912 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.907+0000 7fb0c1756640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0bc164370 con 0x7fb0bc0c1380 2023-05-22T15:43:28.912 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.907+0000 7fb0c1756640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0bc164650 con 0x7fb0bc160170 2023-05-22T15:43:28.912 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.907+0000 7fb0bb7fe640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fb0bc0ca530 0x7fb0bc06f8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:54050/0 (socket says 172.21.15.16:54050) 2023-05-22T15:43:28.913 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.907+0000 7fb0bb7fe640 1 -- 172.21.15.16:0/2498549626 learned_addr learned my addr 172.21.15.16:0/2498549626 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:43:28.913 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.907+0000 7fb0ba7fc640 1 -- 172.21.15.16:0/2498549626 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3126865551 0 0) 0x7fb0bc164090 con 0x7fb0bc0ca530 2023-05-22T15:43:28.913 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.907+0000 7fb0baffd640 1 --2- 172.21.15.16:0/2498549626 >> v2:172.21.15.62:3300/0 conn(0x7fb0bc160170 0x7fb0bc0c8660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:28.913 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.907+0000 7fb0ba7fc640 1 -- 172.21.15.16:0/2498549626 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb09c003530 con 0x7fb0bc0ca530 2023-05-22T15:43:28.917 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.907+0000 7fb0ba7fc640 1 -- 172.21.15.16:0/2498549626 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3672087539 0 0) 0x7fb09c003530 con 0x7fb0bc0ca530 2023-05-22T15:43:28.917 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.907+0000 7fb0ba7fc640 1 -- 172.21.15.16:0/2498549626 >> v2:172.21.15.62:3300/0 conn(0x7fb0bc160170 msgr2=0x7fb0bc0c8660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:43:28.917 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.907+0000 7fb0ba7fc640 1 --2- 172.21.15.16:0/2498549626 >> v2:172.21.15.62:3300/0 conn(0x7fb0bc160170 0x7fb0bc0c8660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:28.918 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.907+0000 7fb0ba7fc640 1 -- 172.21.15.16:0/2498549626 >> v1:172.21.15.62:6789/0 conn(0x7fb0bc0c1380 legacy=0x7fb0bc0c9e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:28.918 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.907+0000 7fb0ba7fc640 1 -- 172.21.15.16:0/2498549626 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0bc164930 con 0x7fb0bc0ca530 2023-05-22T15:43:28.918 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.907+0000 7fb0ba7fc640 1 -- 172.21.15.16:0/2498549626 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7fb0b4001a00 con 0x7fb0bc0ca530 2023-05-22T15:43:28.918 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.907+0000 7fb0ba7fc640 1 -- 172.21.15.16:0/2498549626 >> v1:172.21.15.16:6789/0 conn(0x7fb0bc0ca530 legacy=0x7fb0bc06f8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:28.918 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.907+0000 7fb0ba7fc640 1 --2- 172.21.15.16:0/2498549626 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb09c0039d0 0x7fb09c003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:28.918 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.907+0000 7fb0ba7fc640 1 --2- 172.21.15.16:0/2498549626 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb09c0047d0 0x7fb09c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:28.918 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.907+0000 7fb0ba7fc640 1 --2- 172.21.15.16:0/2498549626 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb09c0051d0 0x7fb09c00b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:28.919 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.907+0000 7fb0ba7fc640 1 -- 172.21.15.16:0/2498549626 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0bc164650 con 0x7fb09c0047d0 2023-05-22T15:43:28.919 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.907+0000 7fb0ba7fc640 1 -- 172.21.15.16:0/2498549626 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb09c00c110 con 0x7fb09c0051d0 2023-05-22T15:43:28.919 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.907+0000 7fb0ba7fc640 1 -- 172.21.15.16:0/2498549626 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb09c00c2d0 con 0x7fb09c0039d0 2023-05-22T15:43:28.919 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.907+0000 7fb0b3fff640 1 --2- 172.21.15.16:0/2498549626 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb09c0047d0 0x7fb09c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:28.919 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.907+0000 7fb0baffd640 1 --2- 172.21.15.16:0/2498549626 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb09c0039d0 0x7fb09c003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:28.919 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.907+0000 7fb0bb7fe640 1 --2- 172.21.15.16:0/2498549626 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb09c0051d0 0x7fb09c00b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:28.919 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.907+0000 7fb0b3fff640 1 -- 172.21.15.16:0/2498549626 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb09c0039d0 msgr2=0x7fb09c003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:43:28.919 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.907+0000 7fb0b3fff640 1 --2- 172.21.15.16:0/2498549626 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb09c0039d0 0x7fb09c003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:28.920 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.907+0000 7fb0b3fff640 1 -- 172.21.15.16:0/2498549626 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb09c0051d0 msgr2=0x7fb09c00b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:43:28.920 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.907+0000 7fb0b3fff640 1 --2- 172.21.15.16:0/2498549626 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb09c0051d0 0x7fb09c00b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:28.920 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.907+0000 7fb0b3fff640 1 -- 172.21.15.16:0/2498549626 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb09c003530 con 0x7fb09c0047d0 2023-05-22T15:43:28.920 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.907+0000 7fb0bb7fe640 1 --2- 172.21.15.16:0/2498549626 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb09c0051d0 0x7fb09c00b990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:43:28.920 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.907+0000 7fb0bb7fe640 1 -- 172.21.15.16:0/2498549626 reap_dead start 2023-05-22T15:43:28.920 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.911+0000 7fb0b3fff640 1 --2- 172.21.15.16:0/2498549626 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb09c0047d0 0x7fb09c004c40 secure :-1 s=READY pgs=1440 cs=0 l=1 rev1=1 crypto rx=0x7fb09c004750 tx=0x7fb0ac01c380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c02f12d52441c631 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:43:28.920 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.911+0000 7fb0ba7fc640 1 -- 172.21.15.16:0/2498549626 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0ac0052d0 con 0x7fb09c0047d0 2023-05-22T15:43:28.921 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.911+0000 7fb0ba7fc640 1 -- 172.21.15.16:0/2498549626 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0ac0a7050 con 0x7fb09c0047d0 2023-05-22T15:43:28.921 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.911+0000 7fb0ba7fc640 1 -- 172.21.15.16:0/2498549626 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0ac003600 con 0x7fb09c0047d0 2023-05-22T15:43:28.921 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.911+0000 7fb0c1756640 1 -- 172.21.15.16:0/2498549626 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb09c0047d0 msgr2=0x7fb09c004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:28.921 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.911+0000 7fb0c1756640 1 --2- 172.21.15.16:0/2498549626 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb09c0047d0 0x7fb09c004c40 secure :-1 s=READY pgs=1440 cs=0 l=1 rev1=1 crypto rx=0x7fb09c004750 tx=0x7fb0ac01c380 comp rx=0 tx=0).stop 2023-05-22T15:43:28.921 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.911+0000 7fb0c1756640 1 -- 172.21.15.16:0/2498549626 shutdown_connections 2023-05-22T15:43:28.921 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.911+0000 7fb0c1756640 1 --2- 172.21.15.16:0/2498549626 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb09c0047d0 0x7fb09c004c40 unknown :-1 s=CLOSED pgs=1440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:28.921 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.911+0000 7fb0c1756640 1 -- 172.21.15.16:0/2498549626 >> 172.21.15.16:0/2498549626 conn(0x7fb0bc069720 msgr2=0x7fb0bc069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:43:28.922 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.911+0000 7fb0c1756640 1 -- 172.21.15.16:0/2498549626 shutdown_connections 2023-05-22T15:43:28.922 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.911+0000 7fb0c1756640 1 -- 172.21.15.16:0/2498549626 wait complete. 2023-05-22T15:43:28.922 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.911+0000 7fb0c1756640 1 Processor -- start 2023-05-22T15:43:28.922 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.911+0000 7fb0c1756640 1 -- start start 2023-05-22T15:43:28.924 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.911+0000 7fb0c1756640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb09c0047d0 0x7fb0bc1630d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:28.924 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.911+0000 7fb0c1756640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb0bc0c50e0 0x7fb0bc0c4830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:28.924 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.911+0000 7fb0c1756640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb0bc15d2f0 0x7fb0bc0707b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:28.924 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.911+0000 7fb0c1756640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0bc0c4d70 con 0x7fb0bc0c50e0 2023-05-22T15:43:28.925 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.911+0000 7fb0c1756640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0bc0c5530 con 0x7fb0bc15d2f0 2023-05-22T15:43:28.925 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.911+0000 7fb0c1756640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0bc0c29a0 con 0x7fb09c0047d0 2023-05-22T15:43:28.925 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.915+0000 7fb0b3fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb0bc0c50e0 0x7fb0bc0c4830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:28.925 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.915+0000 7fb0b3fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb0bc0c50e0 0x7fb0bc0c4830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58000/0 (socket says 172.21.15.16:58000) 2023-05-22T15:43:28.925 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.915+0000 7fb0b3fff640 1 -- 172.21.15.16:0/2424868093 learned_addr learned my addr 172.21.15.16:0/2424868093 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:43:28.925 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.915+0000 7fb0baffd640 1 --2- 172.21.15.16:0/2424868093 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb09c0047d0 0x7fb0bc1630d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:28.925 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.915+0000 7fb0bb7fe640 1 --2- 172.21.15.16:0/2424868093 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb0bc15d2f0 0x7fb0bc0707b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:28.926 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.915+0000 7fb0b3fff640 1 -- 172.21.15.16:0/2424868093 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb09c0047d0 msgr2=0x7fb0bc1630d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:28.926 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.915+0000 7fb0b3fff640 1 --2- 172.21.15.16:0/2424868093 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb09c0047d0 0x7fb0bc1630d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:28.926 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.915+0000 7fb0b3fff640 1 -- 172.21.15.16:0/2424868093 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb0bc15d2f0 msgr2=0x7fb0bc0707b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:28.926 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.915+0000 7fb0b3fff640 1 --2- 172.21.15.16:0/2424868093 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb0bc15d2f0 0x7fb0bc0707b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:28.926 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.915+0000 7fb0b3fff640 1 -- 172.21.15.16:0/2424868093 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0bc163d60 con 0x7fb0bc0c50e0 2023-05-22T15:43:28.926 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.915+0000 7fb0baffd640 1 --2- 172.21.15.16:0/2424868093 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb09c0047d0 0x7fb0bc1630d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T15:43:28.926 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.915+0000 7fb0b3fff640 1 --2- 172.21.15.16:0/2424868093 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb0bc0c50e0 0x7fb0bc0c4830 secure :-1 s=READY pgs=1441 cs=0 l=1 rev1=1 crypto rx=0x7fb0ac01c1b0 tx=0x7fb09c005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:43:28.926 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.915+0000 7fb0b37fe640 1 -- 172.21.15.16:0/2424868093 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0ac007440 con 0x7fb0bc0c50e0 2023-05-22T15:43:28.927 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.915+0000 7fb0b37fe640 1 -- 172.21.15.16:0/2424868093 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0ac0075a0 con 0x7fb0bc0c50e0 2023-05-22T15:43:28.927 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.915+0000 7fb0c1756640 1 -- 172.21.15.16:0/2424868093 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0bc164c10 con 0x7fb0bc0c50e0 2023-05-22T15:43:28.927 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.915+0000 7fb0b37fe640 1 -- 172.21.15.16:0/2424868093 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0ac006a00 con 0x7fb0bc0c50e0 2023-05-22T15:43:28.927 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.915+0000 7fb0c1756640 1 -- 172.21.15.16:0/2424868093 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0bc160540 con 0x7fb0bc0c50e0 2023-05-22T15:43:28.927 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.915+0000 7fb0c1756640 1 -- 172.21.15.16:0/2424868093 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0bc04dbe0 con 0x7fb0bc0c50e0 2023-05-22T15:43:28.927 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.915+0000 7fb0b37fe640 1 -- 172.21.15.16:0/2424868093 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 41) v1 ==== 84332+0+0 (secure 0 0 0) 0x7fb0ac006b60 con 0x7fb0bc0c50e0 2023-05-22T15:43:28.927 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.915+0000 7fb0b37fe640 1 --2- 172.21.15.16:0/2424868093 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fb0a00654a0 0x7fb0a0067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:28.928 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.915+0000 7fb0b37fe640 1 -- 172.21.15.16:0/2424868093 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 5900+0+0 (secure 0 0 0) 0x7fb0ac11d830 con 0x7fb0bc0c50e0 2023-05-22T15:43:28.928 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.915+0000 7fb0baffd640 1 --2- 172.21.15.16:0/2424868093 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fb0a00654a0 0x7fb0a0067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:28.928 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.915+0000 7fb0baffd640 1 --2- 172.21.15.16:0/2424868093 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fb0a00654a0 0x7fb0a0067950 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fb0a400baf0 tx=0x7fb0a4008040 comp rx=0 tx=0).ready entity=mgr.7788 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:43:28.928 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:28.919+0000 7fb0b37fe640 1 -- 172.21.15.16:0/2424868093 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fb0ac0eb330 con 0x7fb0bc0c50e0 2023-05-22T15:43:29.076 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:29.071+0000 7fb0c1756640 1 -- 172.21.15.16:0/2424868093 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fb0bc0c1160 con 0x7fb0bc0c50e0 2023-05-22T15:43:29.760 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:29.755+0000 7fb0b37fe640 1 -- 172.21.15.16:0/2424868093 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v116) v1 ==== 170+0+0 (secure 0 0 0) 0x7fb0ac0ee950 con 0x7fb0bc0c50e0 2023-05-22T15:43:29.763 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'cephfs_data' created 2023-05-22T15:43:29.770 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:29.763+0000 7fb0c1756640 1 -- 172.21.15.16:0/2424868093 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fb0a00654a0 msgr2=0x7fb0a0067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:29.771 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:29.763+0000 7fb0c1756640 1 --2- 172.21.15.16:0/2424868093 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fb0a00654a0 0x7fb0a0067950 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fb0a400baf0 tx=0x7fb0a4008040 comp rx=0 tx=0).stop 2023-05-22T15:43:29.771 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:29.763+0000 7fb0c1756640 1 -- 172.21.15.16:0/2424868093 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb0bc0c50e0 msgr2=0x7fb0bc0c4830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:29.771 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:29.763+0000 7fb0c1756640 1 --2- 172.21.15.16:0/2424868093 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb0bc0c50e0 0x7fb0bc0c4830 secure :-1 s=READY pgs=1441 cs=0 l=1 rev1=1 crypto rx=0x7fb0ac01c1b0 tx=0x7fb09c005800 comp rx=0 tx=0).stop 2023-05-22T15:43:29.773 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:29.767+0000 7fb0c1756640 1 -- 172.21.15.16:0/2424868093 shutdown_connections 2023-05-22T15:43:29.773 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:29.767+0000 7fb0c1756640 1 --2- 172.21.15.16:0/2424868093 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb0bc15d2f0 0x7fb0bc0707b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:29.773 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:29.767+0000 7fb0c1756640 1 --2- 172.21.15.16:0/2424868093 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb0bc0c50e0 0x7fb0bc0c4830 unknown :-1 s=CLOSED pgs=1441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:29.773 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:29.767+0000 7fb0c1756640 1 --2- 172.21.15.16:0/2424868093 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fb0a00654a0 0x7fb0a0067950 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:29.773 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:29.767+0000 7fb0c1756640 1 --2- 172.21.15.16:0/2424868093 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb09c0047d0 0x7fb0bc1630d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:29.774 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:29.767+0000 7fb0c1756640 1 -- 172.21.15.16:0/2424868093 >> 172.21.15.16:0/2424868093 conn(0x7fb0bc069720 msgr2=0x7fb0bc0b1100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:43:29.774 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:29.767+0000 7fb0c1756640 1 -- 172.21.15.16:0/2424868093 shutdown_connections 2023-05-22T15:43:29.774 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:29.771+0000 7fb0c1756640 1 -- 172.21.15.16:0/2424868093 wait complete. 2023-05-22T15:43:29.810 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-05-22T15:43:29.930 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:29.923+0000 7f05e834a640 1 Processor -- start 2023-05-22T15:43:29.930 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:29.923+0000 7f05e834a640 1 -- start start 2023-05-22T15:43:29.931 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:29.927+0000 7f05e834a640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f05e01596e0 0x7f05e0163950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:29.931 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:29.927+0000 7f05e834a640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05e0159f70 con 0x7f05e00cb4c0 2023-05-22T15:43:29.931 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:29.927+0000 7f05e834a640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05e00cb030 con 0x7f05e006b6c0 2023-05-22T15:43:29.932 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:29.927+0000 7f05e834a640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05e0071760 con 0x7f05e01596e0 2023-05-22T15:43:29.932 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:29.927+0000 7f05e58e5640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f05e00cb4c0 0x7f05e0158fd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:51736/0 (socket says 172.21.15.16:51736) 2023-05-22T15:43:29.932 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:29.927+0000 7f05e58e5640 1 -- 172.21.15.16:0/24819001 learned_addr learned my addr 172.21.15.16:0/24819001 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:43:29.932 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:29.927+0000 7f05e68e7640 1 --2- 172.21.15.16:0/24819001 >> v2:172.21.15.62:3300/0 conn(0x7f05e01596e0 0x7f05e0163950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:29.933 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:29.927+0000 7f05e50e4640 1 -- 172.21.15.16:0/24819001 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2982462857 0 0) 0x7f05e0159f70 con 0x7f05e00cb4c0 2023-05-22T15:43:29.933 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:29.927+0000 7f05e50e4640 1 -- 172.21.15.16:0/24819001 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f05cc003530 con 0x7f05e00cb4c0 2023-05-22T15:43:29.933 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:29.927+0000 7f05e68e7640 1 -- 172.21.15.16:0/24819001 >> v1:172.21.15.62:6789/0 conn(0x7f05e006b6c0 legacy=0x7f05e006ba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:29.933 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:29.927+0000 7f05e68e7640 1 -- 172.21.15.16:0/24819001 >> v1:172.21.15.16:6789/0 conn(0x7f05e00cb4c0 legacy=0x7f05e0158fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:29.933 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:29.927+0000 7f05e50e4640 1 -- 172.21.15.16:0/24819001 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 436252921 0 0) 0x7f05cc003530 con 0x7f05e00cb4c0 2023-05-22T15:43:29.933 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:29.927+0000 7f05e68e7640 1 -- 172.21.15.16:0/24819001 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05e0159c90 con 0x7f05e01596e0 2023-05-22T15:43:29.934 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:29.931+0000 7f05e68e7640 1 --2- 172.21.15.16:0/24819001 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f05e01596e0 0x7f05e0163950 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f05d80097c0 tx=0x7f05d801e520 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ae0be244643b3505 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:43:29.934 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:29.931+0000 7f05e50e4640 1 -- 172.21.15.16:0/24819001 <== mon.1 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05dc001a00 con 0x7f05e01596e0 2023-05-22T15:43:29.935 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:29.931+0000 7f05e50e4640 1 -- 172.21.15.16:0/24819001 <== mon.1 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05d80a9050 con 0x7f05e01596e0 2023-05-22T15:43:29.935 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:29.931+0000 7f05e50e4640 1 -- 172.21.15.16:0/24819001 <== mon.1 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05d80ad620 con 0x7f05e01596e0 2023-05-22T15:43:29.935 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:29.931+0000 7f05e834a640 1 -- 172.21.15.16:0/24819001 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f05e01596e0 msgr2=0x7f05e0163950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:29.936 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:29.931+0000 7f05e834a640 1 --2- 172.21.15.16:0/24819001 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f05e01596e0 0x7f05e0163950 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f05d80097c0 tx=0x7f05d801e520 comp rx=0 tx=0).stop 2023-05-22T15:43:29.936 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:29.931+0000 7f05e834a640 1 -- 172.21.15.16:0/24819001 shutdown_connections 2023-05-22T15:43:29.936 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:29.931+0000 7f05e834a640 1 --2- 172.21.15.16:0/24819001 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f05e01596e0 0x7f05e0163950 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:29.936 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:29.931+0000 7f05e834a640 1 -- 172.21.15.16:0/24819001 >> 172.21.15.16:0/24819001 conn(0x7f05e0069720 msgr2=0x7f05e0069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:43:29.937 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:29.931+0000 7f05e834a640 1 -- 172.21.15.16:0/24819001 shutdown_connections 2023-05-22T15:43:29.937 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:29.931+0000 7f05e834a640 1 -- 172.21.15.16:0/24819001 wait complete. 2023-05-22T15:43:29.938 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:29.931+0000 7f05e834a640 1 Processor -- start 2023-05-22T15:43:29.938 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:29.931+0000 7f05e834a640 1 -- start start 2023-05-22T15:43:29.939 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:29.931+0000 7f05e834a640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f05e006b6c0 0x7f05e0154f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:29.939 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:29.931+0000 7f05e834a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f05e00cb4c0 0x7f05e00c4440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:29.939 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:29.931+0000 7f05e834a640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f05e01596e0 0x7f05e00b35d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:29.939 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:29.931+0000 7f05e834a640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05e0154a70 con 0x7f05e00cb4c0 2023-05-22T15:43:29.939 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:29.931+0000 7f05e834a640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05e014e410 con 0x7f05e006b6c0 2023-05-22T15:43:29.939 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:29.931+0000 7f05e834a640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05e0155440 con 0x7f05e01596e0 2023-05-22T15:43:29.939 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:29.931+0000 7f05e58e5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f05e00cb4c0 0x7f05e00c4440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:29.940 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:29.931+0000 7f05e58e5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f05e00cb4c0 0x7f05e00c4440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35950/0 (socket says 172.21.15.16:35950) 2023-05-22T15:43:29.940 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:29.931+0000 7f05e58e5640 1 -- 172.21.15.16:0/3203097559 learned_addr learned my addr 172.21.15.16:0/3203097559 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:43:29.940 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:29.931+0000 7f05e60e6640 1 --2- 172.21.15.16:0/3203097559 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f05e006b6c0 0x7f05e0154f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:29.940 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:29.931+0000 7f05e68e7640 1 --2- 172.21.15.16:0/3203097559 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f05e01596e0 0x7f05e00b35d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:29.940 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:29.931+0000 7f05e58e5640 1 -- 172.21.15.16:0/3203097559 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f05e01596e0 msgr2=0x7f05e00b35d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:29.940 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:29.931+0000 7f05e58e5640 1 --2- 172.21.15.16:0/3203097559 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f05e01596e0 0x7f05e00b35d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:29.940 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:29.931+0000 7f05e58e5640 1 -- 172.21.15.16:0/3203097559 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f05e006b6c0 msgr2=0x7f05e0154f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:29.940 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:29.931+0000 7f05e58e5640 1 --2- 172.21.15.16:0/3203097559 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f05e006b6c0 0x7f05e0154f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:29.941 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:29.931+0000 7f05e58e5640 1 -- 172.21.15.16:0/3203097559 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05d8009420 con 0x7f05e00cb4c0 2023-05-22T15:43:29.941 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:29.931+0000 7f05e60e6640 1 --2- 172.21.15.16:0/3203097559 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f05e006b6c0 0x7f05e0154f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T15:43:29.941 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:29.935+0000 7f05e58e5640 1 --2- 172.21.15.16:0/3203097559 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f05e00cb4c0 0x7f05e00c4440 secure :-1 s=READY pgs=1443 cs=0 l=1 rev1=1 crypto rx=0x7f05dc002030 tx=0x7f05dc00a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:43:29.941 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:29.935+0000 7f05d2ffd640 1 -- 172.21.15.16:0/3203097559 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05dc012400 con 0x7f05e00cb4c0 2023-05-22T15:43:29.941 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:29.935+0000 7f05d2ffd640 1 -- 172.21.15.16:0/3203097559 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05dc012560 con 0x7f05e00cb4c0 2023-05-22T15:43:29.941 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:29.935+0000 7f05d2ffd640 1 -- 172.21.15.16:0/3203097559 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05dc0128f0 con 0x7f05e00cb4c0 2023-05-22T15:43:29.942 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:29.935+0000 7f05e834a640 1 -- 172.21.15.16:0/3203097559 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05e01569d0 con 0x7f05e00cb4c0 2023-05-22T15:43:29.942 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:29.935+0000 7f05e834a640 1 -- 172.21.15.16:0/3203097559 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05e0156840 con 0x7f05e00cb4c0 2023-05-22T15:43:29.942 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:29.935+0000 7f05e834a640 1 -- 172.21.15.16:0/3203097559 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05e00ce820 con 0x7f05e00cb4c0 2023-05-22T15:43:29.943 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:29.935+0000 7f05d2ffd640 1 -- 172.21.15.16:0/3203097559 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 41) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f05dc012a50 con 0x7f05e00cb4c0 2023-05-22T15:43:29.943 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:29.935+0000 7f05d2ffd640 1 --2- 172.21.15.16:0/3203097559 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f05c40654d0 0x7f05c4067980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:29.943 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:29.935+0000 7f05d2ffd640 1 -- 172.21.15.16:0/3203097559 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 6282+0+0 (secure 0 0 0) 0x7f05dc083ca0 con 0x7f05e00cb4c0 2023-05-22T15:43:29.943 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:29.935+0000 7f05e60e6640 1 --2- 172.21.15.16:0/3203097559 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f05c40654d0 0x7f05c4067980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:29.943 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:29.935+0000 7f05e60e6640 1 --2- 172.21.15.16:0/3203097559 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f05c40654d0 0x7f05c4067980 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f05d4000f70 tx=0x7f05d4009040 comp rx=0 tx=0).ready entity=mgr.7788 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:43:29.945 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:29.939+0000 7f05d2ffd640 1 -- 172.21.15.16:0/3203097559 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f05dc051820 con 0x7f05e00cb4c0 2023-05-22T15:43:30.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.127+0000 7f05e834a640 1 -- 172.21.15.16:0/3203097559 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f05e004dbe0 con 0x7f05e00cb4c0 2023-05-22T15:43:30.149 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.143+0000 7f05d2ffd640 1 -- 172.21.15.16:0/3203097559 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 14 and data pool 15 v40) v1 ==== 174+0+0 (secure 0 0 0) 0x7f05dc054e40 con 0x7f05e00cb4c0 2023-05-22T15:43:30.149 INFO:teuthology.orchestra.run.smithi016.stderr:new fs with metadata pool 14 and data pool 15 2023-05-22T15:43:30.154 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.147+0000 7f05e834a640 1 -- 172.21.15.16:0/3203097559 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f05c40654d0 msgr2=0x7f05c4067980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:30.155 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.147+0000 7f05e834a640 1 --2- 172.21.15.16:0/3203097559 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f05c40654d0 0x7f05c4067980 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f05d4000f70 tx=0x7f05d4009040 comp rx=0 tx=0).stop 2023-05-22T15:43:30.155 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.147+0000 7f05e834a640 1 -- 172.21.15.16:0/3203097559 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f05e00cb4c0 msgr2=0x7f05e00c4440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:30.155 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.147+0000 7f05e834a640 1 --2- 172.21.15.16:0/3203097559 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f05e00cb4c0 0x7f05e00c4440 secure :-1 s=READY pgs=1443 cs=0 l=1 rev1=1 crypto rx=0x7f05dc002030 tx=0x7f05dc00a430 comp rx=0 tx=0).stop 2023-05-22T15:43:30.160 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.155+0000 7f05e834a640 1 -- 172.21.15.16:0/3203097559 shutdown_connections 2023-05-22T15:43:30.161 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.155+0000 7f05e834a640 1 --2- 172.21.15.16:0/3203097559 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f05c40654d0 0x7f05c4067980 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:30.161 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.155+0000 7f05e834a640 1 --2- 172.21.15.16:0/3203097559 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f05e01596e0 0x7f05e00b35d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:30.161 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.155+0000 7f05e834a640 1 --2- 172.21.15.16:0/3203097559 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f05e00cb4c0 0x7f05e00c4440 unknown :-1 s=CLOSED pgs=1443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:30.161 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.155+0000 7f05e834a640 1 --2- 172.21.15.16:0/3203097559 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f05e006b6c0 0x7f05e0154f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:30.161 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.155+0000 7f05e834a640 1 -- 172.21.15.16:0/3203097559 >> 172.21.15.16:0/3203097559 conn(0x7f05e0069720 msgr2=0x7f05e0156bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:43:30.161 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.155+0000 7f05e834a640 1 -- 172.21.15.16:0/3203097559 shutdown_connections 2023-05-22T15:43:30.162 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.155+0000 7f05e834a640 1 -- 172.21.15.16:0/3203097559 wait complete. 2023-05-22T15:43:30.187 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-05-22T15:43:30.305 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.299+0000 7fd2b578e640 1 Processor -- start 2023-05-22T15:43:30.306 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.299+0000 7fd2b578e640 1 -- start start 2023-05-22T15:43:30.308 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.299+0000 7fd2b578e640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fd2b01531b0 0x7fd2b00afb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:30.308 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.299+0000 7fd2b578e640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2b015a0a0 con 0x7fd2b00b0080 2023-05-22T15:43:30.308 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.303+0000 7fd2b578e640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2b0159e50 con 0x7fd2b0159a40 2023-05-22T15:43:30.309 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.303+0000 7fd2b578e640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2b00b0520 con 0x7fd2b01531b0 2023-05-22T15:43:30.309 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.303+0000 7fd2ae7fc640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fd2b00b0080 0x7fd2b0159330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:51740/0 (socket says 172.21.15.16:51740) 2023-05-22T15:43:30.309 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.303+0000 7fd2ae7fc640 1 -- 172.21.15.16:0/3239464837 learned_addr learned my addr 172.21.15.16:0/3239464837 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:43:30.309 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.303+0000 7fd2aeffd640 1 --2- 172.21.15.16:0/3239464837 >> v2:172.21.15.62:3300/0 conn(0x7fd2b01531b0 0x7fd2b00afb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:30.309 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.303+0000 7fd2adffb640 1 -- 172.21.15.16:0/3239464837 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2137427948 0 0) 0x7fd2b015a0a0 con 0x7fd2b00b0080 2023-05-22T15:43:30.309 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.303+0000 7fd2adffb640 1 -- 172.21.15.16:0/3239464837 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd298003530 con 0x7fd2b00b0080 2023-05-22T15:43:30.309 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.303+0000 7fd2adffb640 1 -- 172.21.15.16:0/3239464837 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 766200195 0 0) 0x7fd298003530 con 0x7fd2b00b0080 2023-05-22T15:43:30.310 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.303+0000 7fd2adffb640 1 -- 172.21.15.16:0/3239464837 >> v2:172.21.15.62:3300/0 conn(0x7fd2b01531b0 msgr2=0x7fd2b00afb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:43:30.310 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.303+0000 7fd2adffb640 1 --2- 172.21.15.16:0/3239464837 >> v2:172.21.15.62:3300/0 conn(0x7fd2b01531b0 0x7fd2b00afb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:30.310 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.303+0000 7fd2adffb640 1 -- 172.21.15.16:0/3239464837 >> v1:172.21.15.62:6789/0 conn(0x7fd2b0159a40 legacy=0x7fd2b00c8ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:30.310 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.303+0000 7fd2adffb640 1 -- 172.21.15.16:0/3239464837 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2b0153580 con 0x7fd2b00b0080 2023-05-22T15:43:30.310 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.303+0000 7fd2adffb640 1 -- 172.21.15.16:0/3239464837 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7fd2a4001680 con 0x7fd2b00b0080 2023-05-22T15:43:30.310 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.303+0000 7fd2adffb640 1 -- 172.21.15.16:0/3239464837 >> v1:172.21.15.16:6789/0 conn(0x7fd2b00b0080 legacy=0x7fd2b0159330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:30.311 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.303+0000 7fd2adffb640 1 --2- 172.21.15.16:0/3239464837 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd298003980 0x7fd298003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:30.311 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.303+0000 7fd2adffb640 1 --2- 172.21.15.16:0/3239464837 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd2980047d0 0x7fd298004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:30.311 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.303+0000 7fd2adffb640 1 --2- 172.21.15.16:0/3239464837 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd298005180 0x7fd29800b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:30.311 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.303+0000 7fd2adffb640 1 -- 172.21.15.16:0/3239464837 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd29800bfd0 con 0x7fd298003980 2023-05-22T15:43:30.311 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.303+0000 7fd2adffb640 1 -- 172.21.15.16:0/3239464837 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd29800c140 con 0x7fd2980047d0 2023-05-22T15:43:30.311 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.303+0000 7fd2adffb640 1 -- 172.21.15.16:0/3239464837 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd29800c300 con 0x7fd298005180 2023-05-22T15:43:30.311 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.303+0000 7fd2aeffd640 1 --2- 172.21.15.16:0/3239464837 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd298003980 0x7fd298003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:30.311 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.303+0000 7fd2ae7fc640 1 --2- 172.21.15.16:0/3239464837 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd2980047d0 0x7fd298004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:30.312 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.303+0000 7fd2af7fe640 1 --2- 172.21.15.16:0/3239464837 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd298005180 0x7fd29800b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:30.312 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.303+0000 7fd2aeffd640 1 -- 172.21.15.16:0/3239464837 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd298005180 msgr2=0x7fd29800b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:43:30.312 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.303+0000 7fd2aeffd640 1 --2- 172.21.15.16:0/3239464837 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd298005180 0x7fd29800b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:30.312 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.303+0000 7fd2aeffd640 1 -- 172.21.15.16:0/3239464837 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd2980047d0 msgr2=0x7fd298004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:43:30.312 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.303+0000 7fd2aeffd640 1 --2- 172.21.15.16:0/3239464837 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd2980047d0 0x7fd298004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:30.312 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.303+0000 7fd2aeffd640 1 -- 172.21.15.16:0/3239464837 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd298003530 con 0x7fd298003980 2023-05-22T15:43:30.313 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.303+0000 7fd2ae7fc640 1 --2- 172.21.15.16:0/3239464837 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd2980047d0 0x7fd298004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T15:43:30.313 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.303+0000 7fd2af7fe640 1 --2- 172.21.15.16:0/3239464837 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd298005180 0x7fd29800b990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:43:30.313 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.303+0000 7fd2af7fe640 1 -- 172.21.15.16:0/3239464837 reap_dead start 2023-05-22T15:43:30.318 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.303+0000 7fd2aeffd640 1 --2- 172.21.15.16:0/3239464837 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd298003980 0x7fd298003d90 secure :-1 s=READY pgs=1445 cs=0 l=1 rev1=1 crypto rx=0x7fd2a0009ae0 tx=0x7fd2a001ca50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=781a89d84c376586 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:43:30.319 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.303+0000 7fd2adffb640 1 -- 172.21.15.16:0/3239464837 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2a00a9450 con 0x7fd298003980 2023-05-22T15:43:30.319 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.303+0000 7fd2adffb640 1 -- 172.21.15.16:0/3239464837 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2a00a95b0 con 0x7fd298003980 2023-05-22T15:43:30.319 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.303+0000 7fd2adffb640 1 -- 172.21.15.16:0/3239464837 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2a00a9960 con 0x7fd298003980 2023-05-22T15:43:30.320 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.307+0000 7fd2b578e640 1 -- 172.21.15.16:0/3239464837 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd298003980 msgr2=0x7fd298003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:30.320 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.307+0000 7fd2b578e640 1 --2- 172.21.15.16:0/3239464837 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd298003980 0x7fd298003d90 secure :-1 s=READY pgs=1445 cs=0 l=1 rev1=1 crypto rx=0x7fd2a0009ae0 tx=0x7fd2a001ca50 comp rx=0 tx=0).stop 2023-05-22T15:43:30.320 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.307+0000 7fd2b578e640 1 -- 172.21.15.16:0/3239464837 shutdown_connections 2023-05-22T15:43:30.320 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.307+0000 7fd2b578e640 1 --2- 172.21.15.16:0/3239464837 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd298003980 0x7fd298003d90 unknown :-1 s=CLOSED pgs=1445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:30.321 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.307+0000 7fd2b578e640 1 -- 172.21.15.16:0/3239464837 >> 172.21.15.16:0/3239464837 conn(0x7fd2b0069720 msgr2=0x7fd2b0069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:43:30.321 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.307+0000 7fd2b578e640 1 -- 172.21.15.16:0/3239464837 shutdown_connections 2023-05-22T15:43:30.321 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.307+0000 7fd2b578e640 1 -- 172.21.15.16:0/3239464837 wait complete. 2023-05-22T15:43:30.321 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.307+0000 7fd2b578e640 1 Processor -- start 2023-05-22T15:43:30.322 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.307+0000 7fd2b578e640 1 -- start start 2023-05-22T15:43:30.322 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.307+0000 7fd2b578e640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd298003980 0x7fd2b014aa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:30.323 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.307+0000 7fd2b578e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd2b014fdf0 0x7fd2b0150970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:30.323 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.307+0000 7fd2b578e640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd2b01457b0 0x7fd2b014f1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:30.323 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.307+0000 7fd2b578e640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2b014a610 con 0x7fd2b014fdf0 2023-05-22T15:43:30.323 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.307+0000 7fd2b578e640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2b0150600 con 0x7fd298003980 2023-05-22T15:43:30.324 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.307+0000 7fd2b578e640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2b0150320 con 0x7fd2b01457b0 2023-05-22T15:43:30.324 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.307+0000 7fd2ae7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd2b014fdf0 0x7fd2b0150970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:30.324 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.307+0000 7fd2ae7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd2b014fdf0 0x7fd2b0150970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35978/0 (socket says 172.21.15.16:35978) 2023-05-22T15:43:30.324 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.307+0000 7fd2ae7fc640 1 -- 172.21.15.16:0/1361611650 learned_addr learned my addr 172.21.15.16:0/1361611650 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:43:30.325 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.307+0000 7fd2aeffd640 1 --2- 172.21.15.16:0/1361611650 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd298003980 0x7fd2b014aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:30.325 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.307+0000 7fd2af7fe640 1 --2- 172.21.15.16:0/1361611650 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd2b01457b0 0x7fd2b014f1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:30.325 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.307+0000 7fd2ae7fc640 1 -- 172.21.15.16:0/1361611650 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd2b01457b0 msgr2=0x7fd2b014f1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:30.325 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.307+0000 7fd2ae7fc640 1 --2- 172.21.15.16:0/1361611650 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd2b01457b0 0x7fd2b014f1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:30.326 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.307+0000 7fd2ae7fc640 1 -- 172.21.15.16:0/1361611650 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd298003980 msgr2=0x7fd2b014aa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:30.326 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.307+0000 7fd2ae7fc640 1 --2- 172.21.15.16:0/1361611650 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd298003980 0x7fd2b014aa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:30.326 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.307+0000 7fd2ae7fc640 1 -- 172.21.15.16:0/1361611650 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2b015d0c0 con 0x7fd2b014fdf0 2023-05-22T15:43:30.326 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.307+0000 7fd2aeffd640 1 --2- 172.21.15.16:0/1361611650 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd298003980 0x7fd2b014aa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:43:30.327 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.307+0000 7fd2af7fe640 1 --2- 172.21.15.16:0/1361611650 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd2b01457b0 0x7fd2b014f1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:43:30.327 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.307+0000 7fd2ae7fc640 1 --2- 172.21.15.16:0/1361611650 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd2b014fdf0 0x7fd2b0150970 secure :-1 s=READY pgs=1446 cs=0 l=1 rev1=1 crypto rx=0x7fd2a400e850 tx=0x7fd2a400ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:43:30.327 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.307+0000 7fd2977fe640 1 -- 172.21.15.16:0/1361611650 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2a400c8d0 con 0x7fd2b014fdf0 2023-05-22T15:43:30.328 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.307+0000 7fd2977fe640 1 -- 172.21.15.16:0/1361611650 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2a400ca30 con 0x7fd2b014fdf0 2023-05-22T15:43:30.328 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.307+0000 7fd2b578e640 1 -- 172.21.15.16:0/1361611650 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2b015cde0 con 0x7fd2b014fdf0 2023-05-22T15:43:30.328 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.307+0000 7fd2977fe640 1 -- 172.21.15.16:0/1361611650 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2a400cda0 con 0x7fd2b014fdf0 2023-05-22T15:43:30.328 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.307+0000 7fd2b578e640 1 -- 172.21.15.16:0/1361611650 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2b0159740 con 0x7fd2b014fdf0 2023-05-22T15:43:30.329 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.311+0000 7fd2b578e640 1 -- 172.21.15.16:0/1361611650 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2b00cba60 con 0x7fd2b014fdf0 2023-05-22T15:43:30.329 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.311+0000 7fd2977fe640 1 -- 172.21.15.16:0/1361611650 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 41) v1 ==== 84332+0+0 (secure 0 0 0) 0x7fd2a4013420 con 0x7fd2b014fdf0 2023-05-22T15:43:30.329 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.311+0000 7fd2977fe640 1 --2- 172.21.15.16:0/1361611650 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd29c0654d0 0x7fd29c067980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:30.330 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.311+0000 7fd2977fe640 1 -- 172.21.15.16:0/1361611650 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 6382+0+0 (secure 0 0 0) 0x7fd2a40828c0 con 0x7fd2b014fdf0 2023-05-22T15:43:30.330 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.311+0000 7fd2aeffd640 1 --2- 172.21.15.16:0/1361611650 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd29c0654d0 0x7fd29c067980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:30.330 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.311+0000 7fd2aeffd640 1 --2- 172.21.15.16:0/1361611650 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd29c0654d0 0x7fd29c067980 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fd2a00ae000 tx=0x7fd2a00b3040 comp rx=0 tx=0).ready entity=mgr.7788 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:43:30.331 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.315+0000 7fd2977fe640 1 -- 172.21.15.16:0/1361611650 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fd2a4050280 con 0x7fd2b014fdf0 2023-05-22T15:43:30.474 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.471+0000 7fd2b578e640 1 -- 172.21.15.16:0/1361611650 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd2b014a750 con 0x7fd2b014fdf0 2023-05-22T15:43:30.476 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.471+0000 7fd2977fe640 1 -- 172.21.15.16:0/1361611650 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v117) v1 ==== 74+0+18003 (secure 0 0 0) 0x7fd2b014a750 con 0x7fd2b014fdf0 2023-05-22T15:43:30.476 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:43:30.477 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":117,"fsid":"b2719834-7847-4d98-90a6-f65d57f85c49","created":"2023-05-22T15:26:25.259314+0000","modified":"2023-05-22T15:43:30.133972+0000","last_up_change":"2023-05-22T15:26:36.448524+0000","last_in_change":"2023-05-22T15:26:27.675668+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-05-22T15:26:38.019781+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-05-22T15:26:43.570514+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2023-05-22T15:43:28.684367+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"117","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2023-05-22T15:43:29.119294+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"117","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9e7ff854-b25f-4cbb-a5c2-1799d9c53d2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":37021},{"type":"v1","addr":"172.21.15.16:6825","nonce":37021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":37021},{"type":"v1","addr":"172.21.15.16:6827","nonce":37021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":37021},{"type":"v1","addr":"172.21.15.16:6831","nonce":37021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":37021},{"type":"v1","addr":"172.21.15.16:6829","nonce":37021}]},"public_addr":"172.21.15.16:6825/37021","cluster_addr":"172.21.15.16:6827/37021","heartbeat_back_addr":"172.21.15.16:6831/37021","heartbeat_front_addr":"172.21.15.16:6829/37021","state":["exists","up"]},{"osd":1,"uuid":"4702fbc9-a927-4565-aca5-41dc69cc8264","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":37008},{"type":"v1","addr":"172.21.15.16:6801","nonce":37008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":37008},{"type":"v1","addr":"172.21.15.16:6803","nonce":37008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":37008},{"type":"v1","addr":"172.21.15.16:6807","nonce":37008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":37008},{"type":"v1","addr":"172.21.15.16:6805","nonce":37008}]},"public_addr":"172.21.15.16:6801/37008","cluster_addr":"172.21.15.16:6803/37008","heartbeat_back_addr":"172.21.15.16:6807/37008","heartbeat_front_addr":"172.21.15.16:6805/37008","state":["exists","up"]},{"osd":2,"uuid":"1e7bba46-4eba-47ef-bb81-5e2d371638ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":37009},{"type":"v1","addr":"172.21.15.16:6809","nonce":37009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":37009},{"type":"v1","addr":"172.21.15.16:6811","nonce":37009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":37009},{"type":"v1","addr":"172.21.15.16:6815","nonce":37009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":37009},{"type":"v1","addr":"172.21.15.16:6813","nonce":37009}]},"public_addr":"172.21.15.16:6809/37009","cluster_addr":"172.21.15.16:6811/37009","heartbeat_back_addr":"172.21.15.16:6815/37009","heartbeat_front_addr":"172.21.15.16:6813/37009","state":["exists","up"]},{"osd":3,"uuid":"383cacc0-5a88-4904-990e-498ca59ef48a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":37020},{"type":"v1","addr":"172.21.15.16:6817","nonce":37020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":37020},{"type":"v1","addr":"172.21.15.16:6819","nonce":37020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":37020},{"type":"v1","addr":"172.21.15.16:6823","nonce":37020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":37020},{"type":"v1","addr":"172.21.15.16:6821","nonce":37020}]},"public_addr":"172.21.15.16:6817/37020","cluster_addr":"172.21.15.16:6819/37020","heartbeat_back_addr":"172.21.15.16:6823/37020","heartbeat_front_addr":"172.21.15.16:6821/37020","state":["exists","up"]},{"osd":4,"uuid":"970500ef-95b7-42c1-b080-eed4fae7013d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":37353},{"type":"v1","addr":"172.21.15.62:6817","nonce":37353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6819","nonce":37353},{"type":"v1","addr":"172.21.15.62:6822","nonce":37353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37353},{"type":"v1","addr":"172.21.15.62:6831","nonce":37353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37353},{"type":"v1","addr":"172.21.15.62:6826","nonce":37353}]},"public_addr":"172.21.15.62:6817/37353","cluster_addr":"172.21.15.62:6822/37353","heartbeat_back_addr":"172.21.15.62:6831/37353","heartbeat_front_addr":"172.21.15.62:6826/37353","state":["exists","up"]},{"osd":5,"uuid":"022d67db-d4d8-445b-b1b9-8b446a00f5c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":37352},{"type":"v1","addr":"172.21.15.62:6801","nonce":37352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":37352},{"type":"v1","addr":"172.21.15.62:6803","nonce":37352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":37352},{"type":"v1","addr":"172.21.15.62:6807","nonce":37352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":37352},{"type":"v1","addr":"172.21.15.62:6805","nonce":37352}]},"public_addr":"172.21.15.62:6801/37352","cluster_addr":"172.21.15.62:6803/37352","heartbeat_back_addr":"172.21.15.62:6807/37352","heartbeat_front_addr":"172.21.15.62:6805/37352","state":["exists","up"]},{"osd":6,"uuid":"42bd372b-6688-4a1b-8407-e16e8c266e33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":37354},{"type":"v1","addr":"172.21.15.62:6820","nonce":37354}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":37354},{"type":"v1","addr":"172.21.15.62:6823","nonce":37354}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6829","nonce":37354},{"type":"v1","addr":"172.21.15.62:6830","nonce":37354}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6825","nonce":37354},{"type":"v1","addr":"172.21.15.62:6827","nonce":37354}]},"public_addr":"172.21.15.62:6820/37354","cluster_addr":"172.21.15.62:6823/37354","heartbeat_back_addr":"172.21.15.62:6830/37354","heartbeat_front_addr":"172.21.15.62:6827/37354","state":["exists","up"]},{"osd":7,"uuid":"ac70b2a3-f41b-4bf2-b227-7f2cbb55426d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37355},{"type":"v1","addr":"172.21.15.62:6809","nonce":37355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37355},{"type":"v1","addr":"172.21.15.62:6811","nonce":37355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":37355},{"type":"v1","addr":"172.21.15.62:6815","nonce":37355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":37355},{"type":"v1","addr":"172.21.15.62:6813","nonce":37355}]},"public_addr":"172.21.15.62:6809/37355","cluster_addr":"172.21.15.62:6811/37355","heartbeat_back_addr":"172.21.15.62:6815/37355","heartbeat_front_addr":"172.21.15.62:6813/37355","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.368896+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.107273+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.276888+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.287502+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.124791+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.251937+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.255679+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.232797+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-05-22T15:43:30.478 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.475+0000 7fd2b578e640 1 -- 172.21.15.16:0/1361611650 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd29c0654d0 msgr2=0x7fd29c067980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:30.479 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.475+0000 7fd2b578e640 1 --2- 172.21.15.16:0/1361611650 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd29c0654d0 0x7fd29c067980 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fd2a00ae000 tx=0x7fd2a00b3040 comp rx=0 tx=0).stop 2023-05-22T15:43:30.479 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.475+0000 7fd2b578e640 1 -- 172.21.15.16:0/1361611650 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd2b014fdf0 msgr2=0x7fd2b0150970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:30.479 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.475+0000 7fd2b578e640 1 --2- 172.21.15.16:0/1361611650 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd2b014fdf0 0x7fd2b0150970 secure :-1 s=READY pgs=1446 cs=0 l=1 rev1=1 crypto rx=0x7fd2a400e850 tx=0x7fd2a400ed10 comp rx=0 tx=0).stop 2023-05-22T15:43:30.479 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.475+0000 7fd2b578e640 1 -- 172.21.15.16:0/1361611650 shutdown_connections 2023-05-22T15:43:30.479 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.475+0000 7fd2b578e640 1 --2- 172.21.15.16:0/1361611650 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd29c0654d0 0x7fd29c067980 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:30.479 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.475+0000 7fd2b578e640 1 --2- 172.21.15.16:0/1361611650 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd2b01457b0 0x7fd2b014f1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:30.479 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.475+0000 7fd2b578e640 1 --2- 172.21.15.16:0/1361611650 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd2b014fdf0 0x7fd2b0150970 unknown :-1 s=CLOSED pgs=1446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:30.480 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.475+0000 7fd2b578e640 1 --2- 172.21.15.16:0/1361611650 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd298003980 0x7fd2b014aa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:30.480 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.475+0000 7fd2b578e640 1 -- 172.21.15.16:0/1361611650 >> 172.21.15.16:0/1361611650 conn(0x7fd2b0069720 msgr2=0x7fd2b015e030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:43:30.480 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.475+0000 7fd2b578e640 1 -- 172.21.15.16:0/1361611650 shutdown_connections 2023-05-22T15:43:30.480 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.475+0000 7fd2b578e640 1 -- 172.21.15.16:0/1361611650 wait complete. 2023-05-22T15:43:30.496 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-05-22T15:43:30.621 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.615+0000 7fe6bd822640 1 Processor -- start 2023-05-22T15:43:30.622 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.615+0000 7fe6bd822640 1 -- start start 2023-05-22T15:43:30.622 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.619+0000 7fe6bd822640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe6b8071500 0x7fe6b815e460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:30.623 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.619+0000 7fe6bd822640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6b80b0650 con 0x7fe6b80c9b40 2023-05-22T15:43:30.623 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.619+0000 7fe6bd822640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6b80b0370 con 0x7fe6b80ca4e0 2023-05-22T15:43:30.623 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.619+0000 7fe6bd822640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6b815d980 con 0x7fe6b8071500 2023-05-22T15:43:30.623 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.619+0000 7fe6b67fc640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fe6b80c9b40 0x7fe6b8070df0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:51746/0 (socket says 172.21.15.16:51746) 2023-05-22T15:43:30.623 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.619+0000 7fe6b67fc640 1 -- 172.21.15.16:0/407867293 learned_addr learned my addr 172.21.15.16:0/407867293 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:43:30.624 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.619+0000 7fe6b77fe640 1 --2- 172.21.15.16:0/407867293 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe6b8071500 0x7fe6b815e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:30.624 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.619+0000 7fe6b5ffb640 1 -- 172.21.15.16:0/407867293 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2768030503 0 0) 0x7fe6b80b0650 con 0x7fe6b80c9b40 2023-05-22T15:43:30.625 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.619+0000 7fe6b5ffb640 1 -- 172.21.15.16:0/407867293 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe6a0003530 con 0x7fe6b80c9b40 2023-05-22T15:43:30.625 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.619+0000 7fe6b5ffb640 1 -- 172.21.15.16:0/407867293 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4017208996 0 0) 0x7fe6a0003530 con 0x7fe6b80c9b40 2023-05-22T15:43:30.625 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.619+0000 7fe6b5ffb640 1 -- 172.21.15.16:0/407867293 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe6b8071500 msgr2=0x7fe6b815e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:43:30.625 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.619+0000 7fe6b5ffb640 1 --2- 172.21.15.16:0/407867293 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe6b8071500 0x7fe6b815e460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:30.625 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.619+0000 7fe6b5ffb640 1 -- 172.21.15.16:0/407867293 >> v1:172.21.15.62:6789/0 conn(0x7fe6b80ca4e0 legacy=0x7fe6b80afb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:30.625 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.619+0000 7fe6b5ffb640 1 -- 172.21.15.16:0/407867293 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6b815dc60 con 0x7fe6b80c9b40 2023-05-22T15:43:30.625 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.619+0000 7fe6b5ffb640 1 -- 172.21.15.16:0/407867293 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7fe6ac0016a0 con 0x7fe6b80c9b40 2023-05-22T15:43:30.626 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.619+0000 7fe6b5ffb640 1 -- 172.21.15.16:0/407867293 >> v1:172.21.15.16:6789/0 conn(0x7fe6b80c9b40 legacy=0x7fe6b8070df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:30.626 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.619+0000 7fe6b5ffb640 1 --2- 172.21.15.16:0/407867293 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe6a00039b0 0x7fe6a0003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:30.626 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.619+0000 7fe6b5ffb640 1 --2- 172.21.15.16:0/407867293 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe6b8071500 0x7fe6a00047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:30.626 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.619+0000 7fe6b5ffb640 1 --2- 172.21.15.16:0/407867293 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe6a0004d10 0x7fe6a0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:30.626 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.619+0000 7fe6b5ffb640 1 -- 172.21.15.16:0/407867293 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6b815d980 con 0x7fe6a0004d10 2023-05-22T15:43:30.626 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.619+0000 7fe6b5ffb640 1 -- 172.21.15.16:0/407867293 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6a000ba40 con 0x7fe6a00039b0 2023-05-22T15:43:30.626 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.619+0000 7fe6b5ffb640 1 -- 172.21.15.16:0/407867293 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6a000bc00 con 0x7fe6b8071500 2023-05-22T15:43:30.627 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.619+0000 7fe6b77fe640 1 --2- 172.21.15.16:0/407867293 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe6a0004d10 0x7fe6a0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:30.627 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.619+0000 7fe6b6ffd640 1 --2- 172.21.15.16:0/407867293 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe6a00039b0 0x7fe6a0003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:30.627 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.619+0000 7fe6b77fe640 1 -- 172.21.15.16:0/407867293 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe6b8071500 msgr2=0x7fe6a00047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:43:30.627 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.619+0000 7fe6b67fc640 1 --2- 172.21.15.16:0/407867293 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe6b8071500 0x7fe6a00047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:30.627 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.619+0000 7fe6b77fe640 1 --2- 172.21.15.16:0/407867293 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe6b8071500 0x7fe6a00047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:30.627 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.619+0000 7fe6b77fe640 1 -- 172.21.15.16:0/407867293 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe6a00039b0 msgr2=0x7fe6a0003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:43:30.627 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.619+0000 7fe6b77fe640 1 --2- 172.21.15.16:0/407867293 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe6a00039b0 0x7fe6a0003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:30.627 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.619+0000 7fe6b77fe640 1 -- 172.21.15.16:0/407867293 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6a0003530 con 0x7fe6a0004d10 2023-05-22T15:43:30.628 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.619+0000 7fe6b6ffd640 1 --2- 172.21.15.16:0/407867293 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe6a00039b0 0x7fe6a0003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T15:43:30.628 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.619+0000 7fe6b67fc640 1 --2- 172.21.15.16:0/407867293 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe6b8071500 0x7fe6a00047d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:43:30.629 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.623+0000 7fe6b77fe640 1 --2- 172.21.15.16:0/407867293 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe6a0004d10 0x7fe6a0005180 secure :-1 s=READY pgs=1448 cs=0 l=1 rev1=1 crypto rx=0x7fe6b0009fc0 tx=0x7fe6b001cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c1d015f7816a298 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:43:30.629 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.623+0000 7fe6b5ffb640 1 -- 172.21.15.16:0/407867293 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6b00a5c70 con 0x7fe6a0004d10 2023-05-22T15:43:30.629 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.623+0000 7fe6b5ffb640 1 -- 172.21.15.16:0/407867293 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6b000f040 con 0x7fe6a0004d10 2023-05-22T15:43:30.629 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.623+0000 7fe6b5ffb640 1 -- 172.21.15.16:0/407867293 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6b00ad440 con 0x7fe6a0004d10 2023-05-22T15:43:30.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.623+0000 7fe6bd822640 1 -- 172.21.15.16:0/407867293 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe6a0004d10 msgr2=0x7fe6a0005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:30.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.623+0000 7fe6bd822640 1 --2- 172.21.15.16:0/407867293 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe6a0004d10 0x7fe6a0005180 secure :-1 s=READY pgs=1448 cs=0 l=1 rev1=1 crypto rx=0x7fe6b0009fc0 tx=0x7fe6b001cc60 comp rx=0 tx=0).stop 2023-05-22T15:43:30.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.623+0000 7fe6b77fe640 1 -- 172.21.15.16:0/407867293 reap_dead start 2023-05-22T15:43:30.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.623+0000 7fe6bd822640 1 -- 172.21.15.16:0/407867293 shutdown_connections 2023-05-22T15:43:30.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.623+0000 7fe6bd822640 1 -- 172.21.15.16:0/407867293 >> 172.21.15.16:0/407867293 conn(0x7fe6b8069720 msgr2=0x7fe6b8069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:43:30.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.623+0000 7fe6bd822640 1 -- 172.21.15.16:0/407867293 shutdown_connections 2023-05-22T15:43:30.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.623+0000 7fe6bd822640 1 -- 172.21.15.16:0/407867293 wait complete. 2023-05-22T15:43:30.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.623+0000 7fe6bd822640 1 Processor -- start 2023-05-22T15:43:30.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.623+0000 7fe6bd822640 1 -- start start 2023-05-22T15:43:30.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.623+0000 7fe6bd822640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe6b81514e0 0x7fe6b81f0460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:30.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.623+0000 7fe6bd822640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe6b81549a0 0x7fe6b80c3640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:30.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.623+0000 7fe6bd822640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe6b80c3b80 0x7fe6b81f2fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:30.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.623+0000 7fe6bd822640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6b80c4020 con 0x7fe6b81514e0 2023-05-22T15:43:30.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.623+0000 7fe6bd822640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6b80c1720 con 0x7fe6b81549a0 2023-05-22T15:43:30.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.623+0000 7fe6bd822640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6b81f09a0 con 0x7fe6b80c3b80 2023-05-22T15:43:30.633 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.623+0000 7fe6b6ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe6b81514e0 0x7fe6b81f0460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:30.633 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.623+0000 7fe6b6ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe6b81514e0 0x7fe6b81f0460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35986/0 (socket says 172.21.15.16:35986) 2023-05-22T15:43:30.633 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.623+0000 7fe6b6ffd640 1 -- 172.21.15.16:0/2984191700 learned_addr learned my addr 172.21.15.16:0/2984191700 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:43:30.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.623+0000 7fe6b67fc640 1 --2- 172.21.15.16:0/2984191700 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe6b81549a0 0x7fe6b80c3640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:30.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.623+0000 7fe6b77fe640 1 --2- 172.21.15.16:0/2984191700 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe6b80c3b80 0x7fe6b81f2fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:30.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.623+0000 7fe6b6ffd640 1 -- 172.21.15.16:0/2984191700 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe6b80c3b80 msgr2=0x7fe6b81f2fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:30.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.623+0000 7fe6b6ffd640 1 --2- 172.21.15.16:0/2984191700 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe6b80c3b80 0x7fe6b81f2fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:30.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.623+0000 7fe6b6ffd640 1 -- 172.21.15.16:0/2984191700 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe6b81549a0 msgr2=0x7fe6b80c3640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:30.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.623+0000 7fe6b6ffd640 1 --2- 172.21.15.16:0/2984191700 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe6b81549a0 0x7fe6b80c3640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:30.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.623+0000 7fe6b6ffd640 1 -- 172.21.15.16:0/2984191700 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6b80ca8b0 con 0x7fe6b81514e0 2023-05-22T15:43:30.635 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.623+0000 7fe6b6ffd640 1 --2- 172.21.15.16:0/2984191700 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe6b81514e0 0x7fe6b81f0460 secure :-1 s=READY pgs=1449 cs=0 l=1 rev1=1 crypto rx=0x7fe6a800e580 tx=0x7fe6a800ea40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:43:30.635 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.627+0000 7fe69ffff640 1 -- 172.21.15.16:0/2984191700 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6a80052e0 con 0x7fe6b81514e0 2023-05-22T15:43:30.635 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.627+0000 7fe69ffff640 1 -- 172.21.15.16:0/2984191700 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6a8005440 con 0x7fe6b81514e0 2023-05-22T15:43:30.635 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.627+0000 7fe6bd822640 1 -- 172.21.15.16:0/2984191700 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6b815df40 con 0x7fe6b81514e0 2023-05-22T15:43:30.635 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.627+0000 7fe69ffff640 1 -- 172.21.15.16:0/2984191700 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6a8003820 con 0x7fe6b81514e0 2023-05-22T15:43:30.635 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.627+0000 7fe6bd822640 1 -- 172.21.15.16:0/2984191700 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6b815e460 con 0x7fe6b81514e0 2023-05-22T15:43:30.635 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.627+0000 7fe6bd822640 1 -- 172.21.15.16:0/2984191700 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6b804dbe0 con 0x7fe6b81514e0 2023-05-22T15:43:30.635 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.627+0000 7fe69ffff640 1 -- 172.21.15.16:0/2984191700 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 41) v1 ==== 84332+0+0 (secure 0 0 0) 0x7fe6a8003980 con 0x7fe6b81514e0 2023-05-22T15:43:30.636 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.627+0000 7fe69ffff640 1 --2- 172.21.15.16:0/2984191700 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fe6a40654d0 0x7fe6a4067980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:30.636 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.627+0000 7fe69ffff640 1 -- 172.21.15.16:0/2984191700 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 6382+0+0 (secure 0 0 0) 0x7fe6a8082980 con 0x7fe6b81514e0 2023-05-22T15:43:30.636 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.627+0000 7fe6b67fc640 1 --2- 172.21.15.16:0/2984191700 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fe6a40654d0 0x7fe6a4067980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:30.636 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.627+0000 7fe6b67fc640 1 --2- 172.21.15.16:0/2984191700 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fe6a40654d0 0x7fe6a4067980 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fe6b80c4a50 tx=0x7fe6ac009040 comp rx=0 tx=0).ready entity=mgr.7788 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:43:30.637 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.631+0000 7fe69ffff640 1 -- 172.21.15.16:0/2984191700 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fe6a80503c0 con 0x7fe6b81514e0 2023-05-22T15:43:30.791 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.787+0000 7fe6bd822640 1 -- 172.21.15.16:0/2984191700 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe6b80b0eb0 con 0x7fe6b81514e0 2023-05-22T15:43:30.794 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.791+0000 7fe69ffff640 1 -- 172.21.15.16:0/2984191700 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v117) v1 ==== 74+0+18003 (secure 0 0 0) 0x7fe6a80539e0 con 0x7fe6b81514e0 2023-05-22T15:43:30.795 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:43:30.795 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":117,"fsid":"b2719834-7847-4d98-90a6-f65d57f85c49","created":"2023-05-22T15:26:25.259314+0000","modified":"2023-05-22T15:43:30.133972+0000","last_up_change":"2023-05-22T15:26:36.448524+0000","last_in_change":"2023-05-22T15:26:27.675668+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-05-22T15:26:38.019781+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-05-22T15:26:43.570514+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2023-05-22T15:43:28.684367+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"117","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2023-05-22T15:43:29.119294+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"117","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9e7ff854-b25f-4cbb-a5c2-1799d9c53d2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":37021},{"type":"v1","addr":"172.21.15.16:6825","nonce":37021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":37021},{"type":"v1","addr":"172.21.15.16:6827","nonce":37021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":37021},{"type":"v1","addr":"172.21.15.16:6831","nonce":37021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":37021},{"type":"v1","addr":"172.21.15.16:6829","nonce":37021}]},"public_addr":"172.21.15.16:6825/37021","cluster_addr":"172.21.15.16:6827/37021","heartbeat_back_addr":"172.21.15.16:6831/37021","heartbeat_front_addr":"172.21.15.16:6829/37021","state":["exists","up"]},{"osd":1,"uuid":"4702fbc9-a927-4565-aca5-41dc69cc8264","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":37008},{"type":"v1","addr":"172.21.15.16:6801","nonce":37008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":37008},{"type":"v1","addr":"172.21.15.16:6803","nonce":37008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":37008},{"type":"v1","addr":"172.21.15.16:6807","nonce":37008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":37008},{"type":"v1","addr":"172.21.15.16:6805","nonce":37008}]},"public_addr":"172.21.15.16:6801/37008","cluster_addr":"172.21.15.16:6803/37008","heartbeat_back_addr":"172.21.15.16:6807/37008","heartbeat_front_addr":"172.21.15.16:6805/37008","state":["exists","up"]},{"osd":2,"uuid":"1e7bba46-4eba-47ef-bb81-5e2d371638ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":37009},{"type":"v1","addr":"172.21.15.16:6809","nonce":37009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":37009},{"type":"v1","addr":"172.21.15.16:6811","nonce":37009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":37009},{"type":"v1","addr":"172.21.15.16:6815","nonce":37009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":37009},{"type":"v1","addr":"172.21.15.16:6813","nonce":37009}]},"public_addr":"172.21.15.16:6809/37009","cluster_addr":"172.21.15.16:6811/37009","heartbeat_back_addr":"172.21.15.16:6815/37009","heartbeat_front_addr":"172.21.15.16:6813/37009","state":["exists","up"]},{"osd":3,"uuid":"383cacc0-5a88-4904-990e-498ca59ef48a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":37020},{"type":"v1","addr":"172.21.15.16:6817","nonce":37020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":37020},{"type":"v1","addr":"172.21.15.16:6819","nonce":37020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":37020},{"type":"v1","addr":"172.21.15.16:6823","nonce":37020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":37020},{"type":"v1","addr":"172.21.15.16:6821","nonce":37020}]},"public_addr":"172.21.15.16:6817/37020","cluster_addr":"172.21.15.16:6819/37020","heartbeat_back_addr":"172.21.15.16:6823/37020","heartbeat_front_addr":"172.21.15.16:6821/37020","state":["exists","up"]},{"osd":4,"uuid":"970500ef-95b7-42c1-b080-eed4fae7013d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":37353},{"type":"v1","addr":"172.21.15.62:6817","nonce":37353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6819","nonce":37353},{"type":"v1","addr":"172.21.15.62:6822","nonce":37353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37353},{"type":"v1","addr":"172.21.15.62:6831","nonce":37353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37353},{"type":"v1","addr":"172.21.15.62:6826","nonce":37353}]},"public_addr":"172.21.15.62:6817/37353","cluster_addr":"172.21.15.62:6822/37353","heartbeat_back_addr":"172.21.15.62:6831/37353","heartbeat_front_addr":"172.21.15.62:6826/37353","state":["exists","up"]},{"osd":5,"uuid":"022d67db-d4d8-445b-b1b9-8b446a00f5c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":37352},{"type":"v1","addr":"172.21.15.62:6801","nonce":37352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":37352},{"type":"v1","addr":"172.21.15.62:6803","nonce":37352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":37352},{"type":"v1","addr":"172.21.15.62:6807","nonce":37352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":37352},{"type":"v1","addr":"172.21.15.62:6805","nonce":37352}]},"public_addr":"172.21.15.62:6801/37352","cluster_addr":"172.21.15.62:6803/37352","heartbeat_back_addr":"172.21.15.62:6807/37352","heartbeat_front_addr":"172.21.15.62:6805/37352","state":["exists","up"]},{"osd":6,"uuid":"42bd372b-6688-4a1b-8407-e16e8c266e33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":37354},{"type":"v1","addr":"172.21.15.62:6820","nonce":37354}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":37354},{"type":"v1","addr":"172.21.15.62:6823","nonce":37354}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6829","nonce":37354},{"type":"v1","addr":"172.21.15.62:6830","nonce":37354}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6825","nonce":37354},{"type":"v1","addr":"172.21.15.62:6827","nonce":37354}]},"public_addr":"172.21.15.62:6820/37354","cluster_addr":"172.21.15.62:6823/37354","heartbeat_back_addr":"172.21.15.62:6830/37354","heartbeat_front_addr":"172.21.15.62:6827/37354","state":["exists","up"]},{"osd":7,"uuid":"ac70b2a3-f41b-4bf2-b227-7f2cbb55426d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37355},{"type":"v1","addr":"172.21.15.62:6809","nonce":37355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37355},{"type":"v1","addr":"172.21.15.62:6811","nonce":37355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":37355},{"type":"v1","addr":"172.21.15.62:6815","nonce":37355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":37355},{"type":"v1","addr":"172.21.15.62:6813","nonce":37355}]},"public_addr":"172.21.15.62:6809/37355","cluster_addr":"172.21.15.62:6811/37355","heartbeat_back_addr":"172.21.15.62:6815/37355","heartbeat_front_addr":"172.21.15.62:6813/37355","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.368896+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.107273+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.276888+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.287502+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.124791+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.251937+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.255679+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.232797+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-05-22T15:43:30.796 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.791+0000 7fe6bd822640 1 -- 172.21.15.16:0/2984191700 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fe6a40654d0 msgr2=0x7fe6a4067980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:30.796 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.791+0000 7fe6bd822640 1 --2- 172.21.15.16:0/2984191700 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fe6a40654d0 0x7fe6a4067980 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fe6b80c4a50 tx=0x7fe6ac009040 comp rx=0 tx=0).stop 2023-05-22T15:43:30.797 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.791+0000 7fe6bd822640 1 -- 172.21.15.16:0/2984191700 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe6b81514e0 msgr2=0x7fe6b81f0460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:30.797 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.791+0000 7fe6bd822640 1 --2- 172.21.15.16:0/2984191700 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe6b81514e0 0x7fe6b81f0460 secure :-1 s=READY pgs=1449 cs=0 l=1 rev1=1 crypto rx=0x7fe6a800e580 tx=0x7fe6a800ea40 comp rx=0 tx=0).stop 2023-05-22T15:43:30.797 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.791+0000 7fe6bd822640 1 -- 172.21.15.16:0/2984191700 shutdown_connections 2023-05-22T15:43:30.797 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.791+0000 7fe6bd822640 1 --2- 172.21.15.16:0/2984191700 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fe6a40654d0 0x7fe6a4067980 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:30.797 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.791+0000 7fe6bd822640 1 --2- 172.21.15.16:0/2984191700 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe6b80c3b80 0x7fe6b81f2fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:30.797 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.791+0000 7fe6bd822640 1 --2- 172.21.15.16:0/2984191700 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe6b81549a0 0x7fe6b80c3640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:30.797 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.791+0000 7fe6bd822640 1 --2- 172.21.15.16:0/2984191700 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe6b81514e0 0x7fe6b81f0460 unknown :-1 s=CLOSED pgs=1449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:30.798 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.791+0000 7fe6bd822640 1 -- 172.21.15.16:0/2984191700 >> 172.21.15.16:0/2984191700 conn(0x7fe6b8069720 msgr2=0x7fe6b80b10d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:43:30.798 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.791+0000 7fe6bd822640 1 -- 172.21.15.16:0/2984191700 shutdown_connections 2023-05-22T15:43:30.798 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.791+0000 7fe6bd822640 1 -- 172.21.15.16:0/2984191700 wait complete. 2023-05-22T15:43:30.813 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-05-22T15:43:30.939 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.935+0000 7f469425f640 1 Processor -- start 2023-05-22T15:43:30.939 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.935+0000 7f469425f640 1 -- start start 2023-05-22T15:43:30.941 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.935+0000 7f469425f640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f468c15cfa0 0x7f468c06db00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:30.941 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.935+0000 7f469425f640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f468c0cf870 0x7f468c0b1fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:30.942 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.935+0000 7f469425f640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f468c06e040 con 0x7f468c06e740 2023-05-22T15:43:30.942 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.935+0000 7f469425f640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f468c0cb9f0 con 0x7f468c15cfa0 2023-05-22T15:43:30.942 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.935+0000 7f469425f640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f468c0b24f0 con 0x7f468c0cf870 2023-05-22T15:43:30.942 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.935+0000 7f46927fc640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f468c06e740 0x7f468c070de0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:51758/0 (socket says 172.21.15.16:51758) 2023-05-22T15:43:30.942 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.935+0000 7f46927fc640 1 -- 172.21.15.16:0/773908367 learned_addr learned my addr 172.21.15.16:0/773908367 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:43:30.942 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.935+0000 7f4691ffb640 1 --2- 172.21.15.16:0/773908367 >> v2:172.21.15.62:3300/0 conn(0x7f468c15cfa0 0x7f468c06db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:30.942 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.935+0000 7f46917fa640 1 --2- 172.21.15.16:0/773908367 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f468c0cf870 0x7f468c0b1fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:30.942 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.935+0000 7f4690ff9640 1 -- 172.21.15.16:0/773908367 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2281802978 0 0) 0x7f4688001120 con 0x7f468c06e740 2023-05-22T15:43:30.943 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.935+0000 7f4690ff9640 1 -- 172.21.15.16:0/773908367 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4678003530 con 0x7f468c06e740 2023-05-22T15:43:30.943 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.935+0000 7f4690ff9640 1 -- 172.21.15.16:0/773908367 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3109419403 0 0) 0x7f4678003530 con 0x7f468c06e740 2023-05-22T15:43:30.943 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.935+0000 7f4690ff9640 1 -- 172.21.15.16:0/773908367 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f468c0cf870 msgr2=0x7f468c0b1fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:43:30.943 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.935+0000 7f4690ff9640 1 --2- 172.21.15.16:0/773908367 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f468c0cf870 0x7f468c0b1fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:30.943 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.935+0000 7f4690ff9640 1 -- 172.21.15.16:0/773908367 >> v2:172.21.15.62:3300/0 conn(0x7f468c15cfa0 msgr2=0x7f468c06db00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:43:30.943 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.935+0000 7f4690ff9640 1 --2- 172.21.15.16:0/773908367 >> v2:172.21.15.62:3300/0 conn(0x7f468c15cfa0 0x7f468c06db00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:30.943 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.935+0000 7f4690ff9640 1 -- 172.21.15.16:0/773908367 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f468c06e490 con 0x7f468c06e740 2023-05-22T15:43:30.944 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.935+0000 7f4690ff9640 1 -- 172.21.15.16:0/773908367 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7f4688001810 con 0x7f468c06e740 2023-05-22T15:43:30.944 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.935+0000 7f4690ff9640 1 -- 172.21.15.16:0/773908367 >> v1:172.21.15.16:6789/0 conn(0x7f468c06e740 legacy=0x7f468c070de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:30.944 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.935+0000 7f4690ff9640 1 --2- 172.21.15.16:0/773908367 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f468c0cf870 0x7f4678003980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:30.944 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.935+0000 7f4690ff9640 1 --2- 172.21.15.16:0/773908367 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4678003f00 0x7f46780047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:30.944 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.935+0000 7f4690ff9640 1 --2- 172.21.15.16:0/773908367 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4678004d10 0x7f4678005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:30.944 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.935+0000 7f4690ff9640 1 -- 172.21.15.16:0/773908367 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f468c0cb9f0 con 0x7f4678003f00 2023-05-22T15:43:30.944 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.935+0000 7f4690ff9640 1 -- 172.21.15.16:0/773908367 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f468c0b24f0 con 0x7f4678004d10 2023-05-22T15:43:30.944 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.935+0000 7f4690ff9640 1 -- 172.21.15.16:0/773908367 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f467800ba70 con 0x7f468c0cf870 2023-05-22T15:43:30.945 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.935+0000 7f4691ffb640 1 --2- 172.21.15.16:0/773908367 >> v2:172.21.15.62:3300/0 conn(0x7f468c15cfa0 0x7f468c06db00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T15:43:30.945 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.935+0000 7f46917fa640 1 --2- 172.21.15.16:0/773908367 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4678003f00 0x7f46780047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:30.945 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.935+0000 7f4691ffb640 1 --2- 172.21.15.16:0/773908367 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f468c0cf870 0x7f4678003980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:30.945 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.935+0000 7f46917fa640 1 -- 172.21.15.16:0/773908367 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f468c0cf870 msgr2=0x7f4678003980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:43:30.945 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.935+0000 7f46917fa640 1 --2- 172.21.15.16:0/773908367 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f468c0cf870 0x7f4678003980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:30.945 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.935+0000 7f46917fa640 1 -- 172.21.15.16:0/773908367 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4678004d10 msgr2=0x7f4678005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:43:30.945 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.935+0000 7f46917fa640 1 --2- 172.21.15.16:0/773908367 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4678004d10 0x7f4678005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:30.945 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.935+0000 7f46917fa640 1 -- 172.21.15.16:0/773908367 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4678003530 con 0x7f4678003f00 2023-05-22T15:43:30.948 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.939+0000 7f46917fa640 1 --2- 172.21.15.16:0/773908367 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4678003f00 0x7f46780047d0 secure :-1 s=READY pgs=1451 cs=0 l=1 rev1=1 crypto rx=0x7f467c009ab0 tx=0x7f467c01ca00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f84a961af06305a0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:43:30.948 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.939+0000 7f4690ff9640 1 -- 172.21.15.16:0/773908367 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f467c0a64f0 con 0x7f4678003f00 2023-05-22T15:43:30.948 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.939+0000 7f4690ff9640 1 -- 172.21.15.16:0/773908367 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f467c0a6650 con 0x7f4678003f00 2023-05-22T15:43:30.948 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.939+0000 7f4690ff9640 1 -- 172.21.15.16:0/773908367 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f467c0a6a00 con 0x7f4678003f00 2023-05-22T15:43:30.948 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.939+0000 7f469425f640 1 -- 172.21.15.16:0/773908367 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4678003f00 msgr2=0x7f46780047d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:30.949 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.939+0000 7f469425f640 1 --2- 172.21.15.16:0/773908367 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4678003f00 0x7f46780047d0 secure :-1 s=READY pgs=1451 cs=0 l=1 rev1=1 crypto rx=0x7f467c009ab0 tx=0x7f467c01ca00 comp rx=0 tx=0).stop 2023-05-22T15:43:30.949 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.939+0000 7f46927fc640 1 -- 172.21.15.16:0/773908367 reap_dead start 2023-05-22T15:43:30.949 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.939+0000 7f469425f640 1 -- 172.21.15.16:0/773908367 shutdown_connections 2023-05-22T15:43:30.949 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.939+0000 7f469425f640 1 -- 172.21.15.16:0/773908367 >> 172.21.15.16:0/773908367 conn(0x7f468c0cc870 msgr2=0x7f468c0cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:43:30.949 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.939+0000 7f469425f640 1 -- 172.21.15.16:0/773908367 shutdown_connections 2023-05-22T15:43:30.949 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.939+0000 7f469425f640 1 -- 172.21.15.16:0/773908367 wait complete. 2023-05-22T15:43:30.949 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.939+0000 7f469425f640 1 Processor -- start 2023-05-22T15:43:30.950 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.939+0000 7f469425f640 1 -- start start 2023-05-22T15:43:30.950 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.943+0000 7f469425f640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f468c14c5c0 0x7f468c155c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:30.950 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.943+0000 7f469425f640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f468c14c990 0x7f468c1f5d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:30.950 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.943+0000 7f469425f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f468c151030 0x7f468c14f480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:30.950 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.943+0000 7f469425f640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f468c0b2a80 con 0x7f468c151030 2023-05-22T15:43:30.951 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.943+0000 7f469425f640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f468c1f62b0 con 0x7f468c14c5c0 2023-05-22T15:43:30.951 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.943+0000 7f469425f640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f468c1542b0 con 0x7f468c14c990 2023-05-22T15:43:30.951 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.943+0000 7f46927fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f468c151030 0x7f468c14f480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:30.951 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.943+0000 7f46927fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f468c151030 0x7f468c14f480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36010/0 (socket says 172.21.15.16:36010) 2023-05-22T15:43:30.951 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.943+0000 7f46927fc640 1 -- 172.21.15.16:0/3093265085 learned_addr learned my addr 172.21.15.16:0/3093265085 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:43:30.951 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.943+0000 7f46917fa640 1 --2- 172.21.15.16:0/3093265085 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f468c14c990 0x7f468c1f5d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:30.951 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.943+0000 7f4691ffb640 1 --2- 172.21.15.16:0/3093265085 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f468c14c5c0 0x7f468c155c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:30.951 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.943+0000 7f46927fc640 1 -- 172.21.15.16:0/3093265085 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f468c14c990 msgr2=0x7f468c1f5d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:30.952 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.943+0000 7f46927fc640 1 --2- 172.21.15.16:0/3093265085 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f468c14c990 0x7f468c1f5d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:30.952 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.943+0000 7f46927fc640 1 -- 172.21.15.16:0/3093265085 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f468c14c5c0 msgr2=0x7f468c155c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:30.952 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.943+0000 7f46927fc640 1 --2- 172.21.15.16:0/3093265085 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f468c14c5c0 0x7f468c155c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:30.952 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.943+0000 7f46927fc640 1 -- 172.21.15.16:0/3093265085 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f468c0cb610 con 0x7f468c151030 2023-05-22T15:43:30.952 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.943+0000 7f46917fa640 1 --2- 172.21.15.16:0/3093265085 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f468c14c990 0x7f468c1f5d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T15:43:30.952 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.943+0000 7f46927fc640 1 --2- 172.21.15.16:0/3093265085 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f468c151030 0x7f468c14f480 secure :-1 s=READY pgs=1452 cs=0 l=1 rev1=1 crypto rx=0x7f468c06fbb0 tx=0x7f468800c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:43:30.952 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.943+0000 7f46827fc640 1 -- 172.21.15.16:0/3093265085 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4688004cc0 con 0x7f468c151030 2023-05-22T15:43:30.953 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.943+0000 7f46827fc640 1 -- 172.21.15.16:0/3093265085 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4688004e20 con 0x7f468c151030 2023-05-22T15:43:30.953 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.943+0000 7f469425f640 1 -- 172.21.15.16:0/3093265085 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f468c0cb330 con 0x7f468c151030 2023-05-22T15:43:30.953 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.943+0000 7f46827fc640 1 -- 172.21.15.16:0/3093265085 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f468800ca60 con 0x7f468c151030 2023-05-22T15:43:30.953 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.943+0000 7f469425f640 1 -- 172.21.15.16:0/3093265085 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f468c071900 con 0x7f468c151030 2023-05-22T15:43:30.954 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.943+0000 7f469425f640 1 -- 172.21.15.16:0/3093265085 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f468c1544d0 con 0x7f468c151030 2023-05-22T15:43:30.954 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.943+0000 7f46827fc640 1 -- 172.21.15.16:0/3093265085 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 41) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f468800cbc0 con 0x7f468c151030 2023-05-22T15:43:30.954 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.943+0000 7f46827fc640 1 --2- 172.21.15.16:0/3093265085 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f46700654d0 0x7f4670067980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:30.954 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.943+0000 7f46827fc640 1 -- 172.21.15.16:0/3093265085 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f46880818c0 con 0x7f468c151030 2023-05-22T15:43:30.954 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.943+0000 7f4691ffb640 1 --2- 172.21.15.16:0/3093265085 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f46700654d0 0x7f4670067980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:30.955 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.943+0000 7f4691ffb640 1 --2- 172.21.15.16:0/3093265085 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f46700654d0 0x7f4670067980 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f4684003fd0 tx=0x7f468400b040 comp rx=0 tx=0).ready entity=mgr.7788 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:43:30.955 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:30.947+0000 7f46827fc640 1 -- 172.21.15.16:0/3093265085 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f468c1544d0 con 0x7f468c151030 2023-05-22T15:43:31.156 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:31.151+0000 7f469425f640 1 -- 172.21.15.16:0/3093265085 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f468c06e040 con 0x7f468c151030 2023-05-22T15:43:31.596 INFO:tasks.ceph.mds.a.smithi016.stderr:2023-05-22T15:43:31.591+0000 7ffaf59f5640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-05-22T15:43:32.149 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:32.143+0000 7f46827fc640 1 -- 172.21.15.16:0/3093265085 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v42) v1 ==== 118+0+0 (secure 0 0 0) 0x7f468c06e040 con 0x7f468c151030 2023-05-22T15:43:32.153 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:32.147+0000 7f469425f640 1 -- 172.21.15.16:0/3093265085 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f46700654d0 msgr2=0x7f4670067980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:32.153 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:32.147+0000 7f469425f640 1 --2- 172.21.15.16:0/3093265085 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f46700654d0 0x7f4670067980 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f4684003fd0 tx=0x7f468400b040 comp rx=0 tx=0).stop 2023-05-22T15:43:32.153 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:32.147+0000 7f469425f640 1 -- 172.21.15.16:0/3093265085 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f468c151030 msgr2=0x7f468c14f480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:32.153 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:32.147+0000 7f469425f640 1 --2- 172.21.15.16:0/3093265085 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f468c151030 0x7f468c14f480 secure :-1 s=READY pgs=1452 cs=0 l=1 rev1=1 crypto rx=0x7f468c06fbb0 tx=0x7f468800c450 comp rx=0 tx=0).stop 2023-05-22T15:43:32.154 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:32.147+0000 7f469425f640 1 -- 172.21.15.16:0/3093265085 shutdown_connections 2023-05-22T15:43:32.154 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:32.147+0000 7f469425f640 1 --2- 172.21.15.16:0/3093265085 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f468c151030 0x7f468c14f480 unknown :-1 s=CLOSED pgs=1452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:32.154 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:32.147+0000 7f469425f640 1 --2- 172.21.15.16:0/3093265085 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f46700654d0 0x7f4670067980 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:32.154 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:32.147+0000 7f469425f640 1 --2- 172.21.15.16:0/3093265085 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f468c14c990 0x7f468c1f5d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:32.154 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:32.147+0000 7f469425f640 1 --2- 172.21.15.16:0/3093265085 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f468c14c5c0 0x7f468c155c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:32.154 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:32.147+0000 7f469425f640 1 -- 172.21.15.16:0/3093265085 >> 172.21.15.16:0/3093265085 conn(0x7f468c0cc870 msgr2=0x7f468c162800 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:43:32.154 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:32.147+0000 7f469425f640 1 -- 172.21.15.16:0/3093265085 shutdown_connections 2023-05-22T15:43:32.155 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:32.147+0000 7f469425f640 1 -- 172.21.15.16:0/3093265085 wait complete. 2023-05-22T15:43:32.175 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2023-05-22T15:43:32.176 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-05-22T15:43:32.307 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:32.303+0000 7f8989fd0640 1 Processor -- start 2023-05-22T15:43:32.308 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:32.303+0000 7f8989fd0640 1 -- start start 2023-05-22T15:43:32.308 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:32.303+0000 7f8989fd0640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f8984159970 0x7f8984159d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:32.308 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:32.303+0000 7f8989fd0640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f898415d680 0x7f89840cb260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:32.308 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:32.303+0000 7f8989fd0640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8984070c30 con 0x7f898415dad0 2023-05-22T15:43:32.309 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:32.303+0000 7f8989fd0640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8984070e60 con 0x7f8984159970 2023-05-22T15:43:32.309 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:32.303+0000 7f8989fd0640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8984071140 con 0x7f898415d680 2023-05-22T15:43:32.309 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:32.303+0000 7f89837fe640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f8984159970 0x7f8984159d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:32.310 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:32.303+0000 7f89837fe640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f8984159970 0x7f8984159d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36022/0 (socket says 172.21.15.16:36022) 2023-05-22T15:43:32.310 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:32.303+0000 7f89837fe640 1 -- 172.21.15.16:0/109453531 learned_addr learned my addr 172.21.15.16:0/109453531 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:43:32.310 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:32.303+0000 7f8982ffd640 1 --2- 172.21.15.16:0/109453531 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f898415d680 0x7f89840cb260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:32.310 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:32.303+0000 7f89837fe640 1 -- 172.21.15.16:0/109453531 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f898415d680 msgr2=0x7f89840cb260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:43:32.310 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:32.303+0000 7f89837fe640 1 --2- 172.21.15.16:0/109453531 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f898415d680 0x7f89840cb260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:32.310 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:32.303+0000 7f89837fe640 1 -- 172.21.15.16:0/109453531 >> v1:172.21.15.62:6789/0 conn(0x7f898415dad0 legacy=0x7f898406b6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:32.311 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:32.303+0000 7f89837fe640 1 -- 172.21.15.16:0/109453531 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8984071420 con 0x7f8984159970 2023-05-22T15:43:32.312 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:32.307+0000 7f89837fe640 1 --2- 172.21.15.16:0/109453531 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8984159970 0x7f8984159d40 secure :-1 s=READY pgs=1453 cs=0 l=1 rev1=1 crypto rx=0x7f8974009870 tx=0x7f897401c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce556e005bff71c8 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:43:32.314 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:32.307+0000 7f89827fc640 1 -- 172.21.15.16:0/109453531 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89740a6040 con 0x7f8984159970 2023-05-22T15:43:32.314 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:32.307+0000 7f89827fc640 1 -- 172.21.15.16:0/109453531 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89740aa600 con 0x7f8984159970 2023-05-22T15:43:32.314 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:32.307+0000 7f89827fc640 1 -- 172.21.15.16:0/109453531 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89740aa990 con 0x7f8984159970 2023-05-22T15:43:32.315 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:32.307+0000 7f8989fd0640 1 -- 172.21.15.16:0/109453531 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8984159970 msgr2=0x7f8984159d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:32.315 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:32.307+0000 7f8989fd0640 1 --2- 172.21.15.16:0/109453531 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8984159970 0x7f8984159d40 secure :-1 s=READY pgs=1453 cs=0 l=1 rev1=1 crypto rx=0x7f8974009870 tx=0x7f897401c480 comp rx=0 tx=0).stop 2023-05-22T15:43:32.315 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:32.307+0000 7f8989fd0640 1 -- 172.21.15.16:0/109453531 shutdown_connections 2023-05-22T15:43:32.315 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:32.307+0000 7f8989fd0640 1 --2- 172.21.15.16:0/109453531 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f898415d680 0x7f89840cb260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:32.316 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:32.307+0000 7f8989fd0640 1 --2- 172.21.15.16:0/109453531 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8984159970 0x7f8984159d40 unknown :-1 s=CLOSED pgs=1453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:32.316 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:32.307+0000 7f8989fd0640 1 -- 172.21.15.16:0/109453531 >> 172.21.15.16:0/109453531 conn(0x7f8984069720 msgr2=0x7f8984069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:43:32.316 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:32.307+0000 7f8989fd0640 1 -- 172.21.15.16:0/109453531 shutdown_connections 2023-05-22T15:43:32.317 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:32.307+0000 7f8989fd0640 1 -- 172.21.15.16:0/109453531 wait complete. 2023-05-22T15:43:32.319 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:32.311+0000 7f8989fd0640 1 Processor -- start 2023-05-22T15:43:32.319 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:32.311+0000 7f8989fd0640 1 -- start start 2023-05-22T15:43:32.320 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:32.311+0000 7f8989fd0640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8984159970 0x7f898414fa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:32.320 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:32.311+0000 7f8989fd0640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f898415d680 0x7f89840b2b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:32.320 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:32.311+0000 7f8989fd0640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f898415dad0 0x7f89840b30b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:32.321 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:32.311+0000 7f8989fd0640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f898414e1a0 con 0x7f8984159970 2023-05-22T15:43:32.321 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:32.311+0000 7f8989fd0640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f898414e480 con 0x7f898415dad0 2023-05-22T15:43:32.321 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:32.311+0000 7f8989fd0640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f898414e760 con 0x7f898415d680 2023-05-22T15:43:32.321 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:32.311+0000 7f89837fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8984159970 0x7f898414fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:32.322 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:32.311+0000 7f89837fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8984159970 0x7f898414fa30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36028/0 (socket says 172.21.15.16:36028) 2023-05-22T15:43:32.322 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:32.311+0000 7f89837fe640 1 -- 172.21.15.16:0/4248076019 learned_addr learned my addr 172.21.15.16:0/4248076019 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:43:32.322 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:32.311+0000 7f8982ffd640 1 --2- 172.21.15.16:0/4248076019 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f898415d680 0x7f89840b2b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:32.323 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:32.311+0000 7f8983fff640 1 --2- 172.21.15.16:0/4248076019 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f898415dad0 0x7f89840b30b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:32.323 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:32.311+0000 7f89837fe640 1 -- 172.21.15.16:0/4248076019 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f898415d680 msgr2=0x7f89840b2b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:32.323 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:32.311+0000 7f89837fe640 1 --2- 172.21.15.16:0/4248076019 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f898415d680 0x7f89840b2b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:32.324 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:32.311+0000 7f89837fe640 1 -- 172.21.15.16:0/4248076019 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f898415dad0 msgr2=0x7f89840b30b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:32.324 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:32.311+0000 7f89837fe640 1 --2- 172.21.15.16:0/4248076019 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f898415dad0 0x7f89840b30b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:32.324 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:32.311+0000 7f89837fe640 1 -- 172.21.15.16:0/4248076019 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89740094d0 con 0x7f8984159970 2023-05-22T15:43:32.324 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:32.311+0000 7f89837fe640 1 --2- 172.21.15.16:0/4248076019 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8984159970 0x7f898414fa30 secure :-1 s=READY pgs=1454 cs=0 l=1 rev1=1 crypto rx=0x7f897401a0b0 tx=0x7f897401acc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:43:32.325 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:32.311+0000 7f896bfff640 1 -- 172.21.15.16:0/4248076019 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89740b3600 con 0x7f8984159970 2023-05-22T15:43:32.325 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:32.311+0000 7f896bfff640 1 -- 172.21.15.16:0/4248076019 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89740b3760 con 0x7f8984159970 2023-05-22T15:43:32.325 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:32.311+0000 7f896bfff640 1 -- 172.21.15.16:0/4248076019 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89740b3af0 con 0x7f8984159970 2023-05-22T15:43:32.325 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:32.311+0000 7f8989fd0640 1 -- 172.21.15.16:0/4248076019 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89840cae20 con 0x7f8984159970 2023-05-22T15:43:32.326 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:32.311+0000 7f8989fd0640 1 -- 172.21.15.16:0/4248076019 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89840cac90 con 0x7f8984159970 2023-05-22T15:43:32.326 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:32.311+0000 7f8989fd0640 1 -- 172.21.15.16:0/4248076019 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f898406bdd0 con 0x7f8984159970 2023-05-22T15:43:32.326 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:32.315+0000 7f896bfff640 1 -- 172.21.15.16:0/4248076019 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 41) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f89740b3c50 con 0x7f8984159970 2023-05-22T15:43:32.326 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:32.315+0000 7f896bfff640 1 --2- 172.21.15.16:0/4248076019 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f89700654a0 0x7f8970067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:32.327 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:32.315+0000 7f896bfff640 1 -- 172.21.15.16:0/4248076019 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f897411d070 con 0x7f8984159970 2023-05-22T15:43:32.327 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:32.315+0000 7f8982ffd640 1 --2- 172.21.15.16:0/4248076019 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f89700654a0 0x7f8970067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:32.327 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:32.315+0000 7f8982ffd640 1 --2- 172.21.15.16:0/4248076019 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f89700654a0 0x7f8970067950 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f8978009af0 tx=0x7f8978008040 comp rx=0 tx=0).ready entity=mgr.7788 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:43:32.328 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:32.319+0000 7f896bfff640 1 -- 172.21.15.16:0/4248076019 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f89740ea8f0 con 0x7f8984159970 2023-05-22T15:43:32.511 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:32.507+0000 7f8989fd0640 1 -- 172.21.15.16:0/4248076019 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f898404dbe0 con 0x7f8984159970 2023-05-22T15:43:33.162 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.155+0000 7f896bfff640 1 -- 172.21.15.16:0/4248076019 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v43) v1 ==== 122+0+0 (secure 0 0 0) 0x7f89740edf10 con 0x7f8984159970 2023-05-22T15:43:33.165 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.159+0000 7f8989fd0640 1 -- 172.21.15.16:0/4248076019 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f89700654a0 msgr2=0x7f8970067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:33.165 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.159+0000 7f8989fd0640 1 --2- 172.21.15.16:0/4248076019 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f89700654a0 0x7f8970067950 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f8978009af0 tx=0x7f8978008040 comp rx=0 tx=0).stop 2023-05-22T15:43:33.165 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.159+0000 7f8989fd0640 1 -- 172.21.15.16:0/4248076019 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8984159970 msgr2=0x7f898414fa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:33.165 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.159+0000 7f8989fd0640 1 --2- 172.21.15.16:0/4248076019 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8984159970 0x7f898414fa30 secure :-1 s=READY pgs=1454 cs=0 l=1 rev1=1 crypto rx=0x7f897401a0b0 tx=0x7f897401acc0 comp rx=0 tx=0).stop 2023-05-22T15:43:33.166 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.159+0000 7f8989fd0640 1 -- 172.21.15.16:0/4248076019 shutdown_connections 2023-05-22T15:43:33.166 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.159+0000 7f8989fd0640 1 --2- 172.21.15.16:0/4248076019 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f898415dad0 0x7f89840b30b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:33.166 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.159+0000 7f8989fd0640 1 --2- 172.21.15.16:0/4248076019 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f89700654a0 0x7f8970067950 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:33.166 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.159+0000 7f8989fd0640 1 --2- 172.21.15.16:0/4248076019 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f898415d680 0x7f89840b2b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:33.166 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.159+0000 7f8989fd0640 1 --2- 172.21.15.16:0/4248076019 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8984159970 0x7f898414fa30 unknown :-1 s=CLOSED pgs=1454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:33.167 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.159+0000 7f8989fd0640 1 -- 172.21.15.16:0/4248076019 >> 172.21.15.16:0/4248076019 conn(0x7f8984069720 msgr2=0x7f8984069af0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:43:33.167 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.159+0000 7f8989fd0640 1 -- 172.21.15.16:0/4248076019 shutdown_connections 2023-05-22T15:43:33.167 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.159+0000 7f8989fd0640 1 -- 172.21.15.16:0/4248076019 wait complete. 2023-05-22T15:43:33.185 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2023-05-22T15:43:33.186 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-05-22T15:43:33.307 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.303+0000 7f490e123640 1 Processor -- start 2023-05-22T15:43:33.307 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.303+0000 7f490e123640 1 -- start start 2023-05-22T15:43:33.308 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.303+0000 7f490e123640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f49080ccab0 0x7f490815fe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:33.308 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.303+0000 7f490e123640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f49080d0410 0x7f49080d0830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:33.308 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.303+0000 7f490e123640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49080ccef0 con 0x7f49080b7dd0 2023-05-22T15:43:33.309 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.303+0000 7f490e123640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49080d3360 con 0x7f49080ccab0 2023-05-22T15:43:33.309 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.303+0000 7f490e123640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49080cd170 con 0x7f49080d0410 2023-05-22T15:43:33.309 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.303+0000 7f4906ffd640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f49080ccab0 0x7f490815fe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:33.309 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.303+0000 7f4906ffd640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f49080ccab0 0x7f490815fe10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36034/0 (socket says 172.21.15.16:36034) 2023-05-22T15:43:33.310 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.303+0000 7f4906ffd640 1 -- 172.21.15.16:0/263850693 learned_addr learned my addr 172.21.15.16:0/263850693 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:43:33.310 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.303+0000 7f4907fff640 1 --2- 172.21.15.16:0/263850693 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f49080d0410 0x7f49080d0830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:33.310 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.303+0000 7f4906ffd640 1 -- 172.21.15.16:0/263850693 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f49080d0410 msgr2=0x7f49080d0830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:43:33.310 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.303+0000 7f4906ffd640 1 --2- 172.21.15.16:0/263850693 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f49080d0410 0x7f49080d0830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:33.311 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.303+0000 7f4906ffd640 1 -- 172.21.15.16:0/263850693 >> v1:172.21.15.62:6789/0 conn(0x7f49080b7dd0 legacy=0x7f490815f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:33.311 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.303+0000 7f4906ffd640 1 -- 172.21.15.16:0/263850693 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49080d2ac0 con 0x7f49080ccab0 2023-05-22T15:43:33.312 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.307+0000 7f4906ffd640 1 --2- 172.21.15.16:0/263850693 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f49080ccab0 0x7f490815fe10 secure :-1 s=READY pgs=1455 cs=0 l=1 rev1=1 crypto rx=0x7f48fc009870 tx=0x7f48fc01c510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=792f7c39544d19ac server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:43:33.312 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.307+0000 7f49067fc640 1 -- 172.21.15.16:0/263850693 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48fc0aa670 con 0x7f49080ccab0 2023-05-22T15:43:33.312 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.307+0000 7f49067fc640 1 -- 172.21.15.16:0/263850693 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48fc0aa7d0 con 0x7f49080ccab0 2023-05-22T15:43:33.313 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.307+0000 7f49067fc640 1 -- 172.21.15.16:0/263850693 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48fc0aab60 con 0x7f49080ccab0 2023-05-22T15:43:33.313 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.307+0000 7f490e123640 1 -- 172.21.15.16:0/263850693 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f49080ccab0 msgr2=0x7f490815fe10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:33.313 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.307+0000 7f490e123640 1 --2- 172.21.15.16:0/263850693 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f49080ccab0 0x7f490815fe10 secure :-1 s=READY pgs=1455 cs=0 l=1 rev1=1 crypto rx=0x7f48fc009870 tx=0x7f48fc01c510 comp rx=0 tx=0).stop 2023-05-22T15:43:33.313 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.307+0000 7f490e123640 1 -- 172.21.15.16:0/263850693 shutdown_connections 2023-05-22T15:43:33.313 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.307+0000 7f490e123640 1 --2- 172.21.15.16:0/263850693 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f49080d0410 0x7f49080d0830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:33.313 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.307+0000 7f490e123640 1 --2- 172.21.15.16:0/263850693 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f49080ccab0 0x7f490815fe10 unknown :-1 s=CLOSED pgs=1455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:33.313 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.307+0000 7f490e123640 1 -- 172.21.15.16:0/263850693 >> 172.21.15.16:0/263850693 conn(0x7f49080655c0 msgr2=0x7f4908070e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:43:33.313 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.307+0000 7f490e123640 1 -- 172.21.15.16:0/263850693 shutdown_connections 2023-05-22T15:43:33.314 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.307+0000 7f490e123640 1 -- 172.21.15.16:0/263850693 wait complete. 2023-05-22T15:43:33.315 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.307+0000 7f490e123640 1 Processor -- start 2023-05-22T15:43:33.315 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.307+0000 7f490e123640 1 -- start start 2023-05-22T15:43:33.316 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.307+0000 7f490e123640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f49080b7dd0 0x7f49080c5f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:33.316 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.307+0000 7f490e123640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f49080ccab0 0x7f4908153a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:33.316 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.307+0000 7f490e123640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f49080d0410 0x7f49080cbaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:33.316 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.307+0000 7f490e123640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4908155940 con 0x7f49080d0410 2023-05-22T15:43:33.316 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.307+0000 7f490e123640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4908155c20 con 0x7f49080b7dd0 2023-05-22T15:43:33.316 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.307+0000 7f490e123640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4908155f00 con 0x7f49080ccab0 2023-05-22T15:43:33.316 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.307+0000 7f4907fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f49080d0410 0x7f49080cbaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:33.317 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.307+0000 7f4907fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f49080d0410 0x7f49080cbaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36036/0 (socket says 172.21.15.16:36036) 2023-05-22T15:43:33.317 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.307+0000 7f4907fff640 1 -- 172.21.15.16:0/252589582 learned_addr learned my addr 172.21.15.16:0/252589582 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:43:33.317 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.311+0000 7f49077fe640 1 --2- 172.21.15.16:0/252589582 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f49080b7dd0 0x7f49080c5f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:33.317 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.311+0000 7f4906ffd640 1 --2- 172.21.15.16:0/252589582 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f49080ccab0 0x7f4908153a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:33.317 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.311+0000 7f4907fff640 1 -- 172.21.15.16:0/252589582 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f49080ccab0 msgr2=0x7f4908153a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:33.317 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.311+0000 7f4907fff640 1 --2- 172.21.15.16:0/252589582 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f49080ccab0 0x7f4908153a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:33.317 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.311+0000 7f4907fff640 1 -- 172.21.15.16:0/252589582 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f49080b7dd0 msgr2=0x7f49080c5f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:33.318 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.311+0000 7f4907fff640 1 --2- 172.21.15.16:0/252589582 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f49080b7dd0 0x7f49080c5f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:33.318 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.311+0000 7f4907fff640 1 -- 172.21.15.16:0/252589582 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48fc0094d0 con 0x7f49080d0410 2023-05-22T15:43:33.318 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.311+0000 7f4906ffd640 1 --2- 172.21.15.16:0/252589582 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f49080ccab0 0x7f4908153a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:43:33.318 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.311+0000 7f4907fff640 1 --2- 172.21.15.16:0/252589582 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f49080d0410 0x7f49080cbaa0 secure :-1 s=READY pgs=1456 cs=0 l=1 rev1=1 crypto rx=0x7f4900007e60 tx=0x7f490000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:43:33.318 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.311+0000 7f48effff640 1 -- 172.21.15.16:0/252589582 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4900014420 con 0x7f49080d0410 2023-05-22T15:43:33.318 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.311+0000 7f48effff640 1 -- 172.21.15.16:0/252589582 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49000145a0 con 0x7f49080d0410 2023-05-22T15:43:33.318 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.311+0000 7f490e123640 1 -- 172.21.15.16:0/252589582 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4908160350 con 0x7f49080d0410 2023-05-22T15:43:33.318 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.311+0000 7f48effff640 1 -- 172.21.15.16:0/252589582 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4900014950 con 0x7f49080d0410 2023-05-22T15:43:33.319 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.311+0000 7f490e123640 1 -- 172.21.15.16:0/252589582 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49080cf730 con 0x7f49080d0410 2023-05-22T15:43:33.319 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.311+0000 7f490e123640 1 -- 172.21.15.16:0/252589582 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49081604e0 con 0x7f49080d0410 2023-05-22T15:43:33.320 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.311+0000 7f48effff640 1 -- 172.21.15.16:0/252589582 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 41) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f4900014ab0 con 0x7f49080d0410 2023-05-22T15:43:33.320 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.311+0000 7f48effff640 1 --2- 172.21.15.16:0/252589582 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f48f40654a0 0x7f48f4067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:33.320 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.311+0000 7f49077fe640 1 --2- 172.21.15.16:0/252589582 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f48f40654a0 0x7f48f4067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:33.320 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.311+0000 7f48effff640 1 -- 172.21.15.16:0/252589582 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f4900083470 con 0x7f49080d0410 2023-05-22T15:43:33.321 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.311+0000 7f49077fe640 1 --2- 172.21.15.16:0/252589582 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f48f40654a0 0x7f48f4067950 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f49080d1ac0 tx=0x7f48f8009040 comp rx=0 tx=0).ready entity=mgr.7788 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:43:33.321 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.315+0000 7f48effff640 1 -- 172.21.15.16:0/252589582 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f49000548d0 con 0x7f49080d0410 2023-05-22T15:43:33.506 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.499+0000 7f490e123640 1 -- 172.21.15.16:0/252589582 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f490804dbe0 con 0x7f49080d0410 2023-05-22T15:43:33.506 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.503+0000 7f48effff640 1 -- 172.21.15.16:0/252589582 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 43 v43) v1 ==== 94+0+4650 (secure 0 0 0) 0x7f49000544c0 con 0x7f49080d0410 2023-05-22T15:43:33.507 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:43:33.507 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":43,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6350,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6833/1554108010","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":1554108010},{"type":"v1","addr":"172.21.15.62:6833","nonce":1554108010}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":39},{"gid":6979,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/2667219028","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":2667219028},{"type":"v1","addr":"172.21.15.62:6835","nonce":2667219028}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":39},{"gid":8013,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6836/1892737356","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1892737356},{"type":"v1","addr":"172.21.15.16:6836","nonce":1892737356}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":39}],"filesystems":[{"mdsmap":{"epoch":43,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-05-22T15:43:30.134589+0000","modified":"2023-05-22T15:43:33.144847+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8016},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8016":{"gid":8016,"name":"a","rank":0,"incarnation":40,"state":"up:active","state_seq":3,"addr":"172.21.15.16:6837/2085936428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6835","nonce":2085936428},{"type":"v1","addr":"172.21.15.16:6837","nonce":2085936428}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2023-05-22T15:43:33.507 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 43 2023-05-22T15:43:33.508 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.503+0000 7f490e123640 1 -- 172.21.15.16:0/252589582 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f48f40654a0 msgr2=0x7f48f4067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:33.509 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.503+0000 7f490e123640 1 --2- 172.21.15.16:0/252589582 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f48f40654a0 0x7f48f4067950 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f49080d1ac0 tx=0x7f48f8009040 comp rx=0 tx=0).stop 2023-05-22T15:43:33.509 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.503+0000 7f490e123640 1 -- 172.21.15.16:0/252589582 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f49080d0410 msgr2=0x7f49080cbaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:33.509 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.503+0000 7f490e123640 1 --2- 172.21.15.16:0/252589582 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f49080d0410 0x7f49080cbaa0 secure :-1 s=READY pgs=1456 cs=0 l=1 rev1=1 crypto rx=0x7f4900007e60 tx=0x7f490000e4b0 comp rx=0 tx=0).stop 2023-05-22T15:43:33.509 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.503+0000 7f490e123640 1 -- 172.21.15.16:0/252589582 shutdown_connections 2023-05-22T15:43:33.509 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.503+0000 7f490e123640 1 --2- 172.21.15.16:0/252589582 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f49080d0410 0x7f49080cbaa0 unknown :-1 s=CLOSED pgs=1456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:33.509 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.503+0000 7f490e123640 1 --2- 172.21.15.16:0/252589582 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f48f40654a0 0x7f48f4067950 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:33.509 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.503+0000 7f490e123640 1 --2- 172.21.15.16:0/252589582 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f49080ccab0 0x7f4908153a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:33.510 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.503+0000 7f490e123640 1 --2- 172.21.15.16:0/252589582 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f49080b7dd0 0x7f49080c5f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:33.510 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.503+0000 7f490e123640 1 -- 172.21.15.16:0/252589582 >> 172.21.15.16:0/252589582 conn(0x7f49080655c0 msgr2=0x7f490815f690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:43:33.510 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.503+0000 7f490e123640 1 -- 172.21.15.16:0/252589582 shutdown_connections 2023-05-22T15:43:33.510 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.503+0000 7f490e123640 1 -- 172.21.15.16:0/252589582 wait complete. 2023-05-22T15:43:33.524 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-05-22T15:43:33.633 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.627+0000 7fd350700640 1 Processor -- start 2023-05-22T15:43:33.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.627+0000 7fd350700640 1 -- start start 2023-05-22T15:43:33.635 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.631+0000 7fd350700640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd348071500 0x7fd34815e460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:33.635 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.631+0000 7fd350700640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3480b0650 con 0x7fd3480ca4e0 2023-05-22T15:43:33.635 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.631+0000 7fd350700640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3480b0370 con 0x7fd3480c9b40 2023-05-22T15:43:33.635 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.631+0000 7fd350700640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd34815d980 con 0x7fd348071500 2023-05-22T15:43:33.635 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.631+0000 7fd34e49c640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fd3480ca4e0 0x7fd3480afb40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:51768/0 (socket says 172.21.15.16:51768) 2023-05-22T15:43:33.635 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.631+0000 7fd34e49c640 1 -- 172.21.15.16:0/763527127 learned_addr learned my addr 172.21.15.16:0/763527127 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:43:33.636 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.631+0000 7fd34ec9d640 1 --2- 172.21.15.16:0/763527127 >> v2:172.21.15.16:3300/0 conn(0x7fd348071500 0x7fd34815e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:33.636 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.631+0000 7fd34d49a640 1 -- 172.21.15.16:0/763527127 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3555554339 0 0) 0x7fd3480b0650 con 0x7fd3480ca4e0 2023-05-22T15:43:33.636 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.631+0000 7fd34d49a640 1 -- 172.21.15.16:0/763527127 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd330003530 con 0x7fd3480ca4e0 2023-05-22T15:43:33.636 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.631+0000 7fd34ec9d640 1 -- 172.21.15.16:0/763527127 >> v1:172.21.15.62:6789/0 conn(0x7fd3480c9b40 legacy=0x7fd348070df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:43:33.637 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.631+0000 7fd34ec9d640 1 -- 172.21.15.16:0/763527127 >> v1:172.21.15.16:6789/0 conn(0x7fd3480ca4e0 legacy=0x7fd3480afb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:33.637 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.631+0000 7fd34ec9d640 1 -- 172.21.15.16:0/763527127 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd34815dc60 con 0x7fd348071500 2023-05-22T15:43:33.637 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.631+0000 7fd34ec9d640 1 --2- 172.21.15.16:0/763527127 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd348071500 0x7fd34815e460 secure :-1 s=READY pgs=1458 cs=0 l=1 rev1=1 crypto rx=0x7fd338009d90 tx=0x7fd33801c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e350a4adf9c8d8 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:43:33.638 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.631+0000 7fd34d49a640 1 -- 172.21.15.16:0/763527127 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3380a5900 con 0x7fd348071500 2023-05-22T15:43:33.639 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.631+0000 7fd34d49a640 1 -- 172.21.15.16:0/763527127 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3380a5a60 con 0x7fd348071500 2023-05-22T15:43:33.639 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.631+0000 7fd34d49a640 1 -- 172.21.15.16:0/763527127 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3380a5dd0 con 0x7fd348071500 2023-05-22T15:43:33.639 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.635+0000 7fd350700640 1 -- 172.21.15.16:0/763527127 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd348071500 msgr2=0x7fd34815e460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:33.639 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.635+0000 7fd350700640 1 --2- 172.21.15.16:0/763527127 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd348071500 0x7fd34815e460 secure :-1 s=READY pgs=1458 cs=0 l=1 rev1=1 crypto rx=0x7fd338009d90 tx=0x7fd33801c920 comp rx=0 tx=0).stop 2023-05-22T15:43:33.639 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.635+0000 7fd350700640 1 -- 172.21.15.16:0/763527127 shutdown_connections 2023-05-22T15:43:33.639 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.635+0000 7fd350700640 1 --2- 172.21.15.16:0/763527127 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd348071500 0x7fd34815e460 unknown :-1 s=CLOSED pgs=1458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:33.639 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.635+0000 7fd350700640 1 -- 172.21.15.16:0/763527127 >> 172.21.15.16:0/763527127 conn(0x7fd348069720 msgr2=0x7fd348069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:43:33.640 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.635+0000 7fd350700640 1 -- 172.21.15.16:0/763527127 shutdown_connections 2023-05-22T15:43:33.640 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.635+0000 7fd350700640 1 -- 172.21.15.16:0/763527127 wait complete. 2023-05-22T15:43:33.641 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.635+0000 7fd350700640 1 Processor -- start 2023-05-22T15:43:33.641 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.635+0000 7fd350700640 1 -- start start 2023-05-22T15:43:33.642 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.635+0000 7fd350700640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd348071500 0x7fd348151220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:33.642 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.635+0000 7fd350700640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd3480c9b40 0x7fd34815a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:33.642 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.635+0000 7fd350700640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd3480ca4e0 0x7fd348150780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:33.642 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.635+0000 7fd350700640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd34815b680 con 0x7fd3480c9b40 2023-05-22T15:43:33.642 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.635+0000 7fd350700640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd34815a7f0 con 0x7fd3480ca4e0 2023-05-22T15:43:33.642 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.635+0000 7fd350700640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd34815aa70 con 0x7fd348071500 2023-05-22T15:43:33.642 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.635+0000 7fd34dc9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd3480c9b40 0x7fd34815a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:33.642 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.635+0000 7fd34dc9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd3480c9b40 0x7fd34815a2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36052/0 (socket says 172.21.15.16:36052) 2023-05-22T15:43:33.643 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.635+0000 7fd34dc9b640 1 -- 172.21.15.16:0/527056557 learned_addr learned my addr 172.21.15.16:0/527056557 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:43:33.643 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.635+0000 7fd34ec9d640 1 --2- 172.21.15.16:0/527056557 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd3480ca4e0 0x7fd348150780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:33.643 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.635+0000 7fd34dc9b640 1 -- 172.21.15.16:0/527056557 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd348071500 msgr2=0x7fd348151220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:33.643 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.635+0000 7fd34dc9b640 1 --2- 172.21.15.16:0/527056557 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd348071500 0x7fd348151220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:33.643 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.635+0000 7fd34dc9b640 1 -- 172.21.15.16:0/527056557 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd3480ca4e0 msgr2=0x7fd348150780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:33.643 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.635+0000 7fd34dc9b640 1 --2- 172.21.15.16:0/527056557 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd3480ca4e0 0x7fd348150780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:33.643 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.635+0000 7fd34dc9b640 1 -- 172.21.15.16:0/527056557 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd338009a70 con 0x7fd3480c9b40 2023-05-22T15:43:33.644 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.635+0000 7fd34ec9d640 1 --2- 172.21.15.16:0/527056557 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd3480ca4e0 0x7fd348150780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:43:33.644 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.635+0000 7fd34dc9b640 1 --2- 172.21.15.16:0/527056557 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd3480c9b40 0x7fd34815a2b0 secure :-1 s=READY pgs=1459 cs=0 l=1 rev1=1 crypto rx=0x7fd33c00c450 tx=0x7fd33c00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:43:33.644 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.635+0000 7fd346ffd640 1 -- 172.21.15.16:0/527056557 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd33c0052d0 con 0x7fd3480c9b40 2023-05-22T15:43:33.644 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.635+0000 7fd346ffd640 1 -- 172.21.15.16:0/527056557 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd33c005430 con 0x7fd3480c9b40 2023-05-22T15:43:33.644 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.635+0000 7fd350700640 1 -- 172.21.15.16:0/527056557 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3480c9010 con 0x7fd3480c9b40 2023-05-22T15:43:33.644 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.635+0000 7fd346ffd640 1 -- 172.21.15.16:0/527056557 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd33c007260 con 0x7fd3480c9b40 2023-05-22T15:43:33.644 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.635+0000 7fd350700640 1 -- 172.21.15.16:0/527056557 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3480c8e80 con 0x7fd3480c9b40 2023-05-22T15:43:33.645 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.635+0000 7fd350700640 1 -- 172.21.15.16:0/527056557 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3480c91a0 con 0x7fd3480c9b40 2023-05-22T15:43:33.645 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.639+0000 7fd346ffd640 1 -- 172.21.15.16:0/527056557 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 41) v1 ==== 84332+0+0 (secure 0 0 0) 0x7fd33c0073c0 con 0x7fd3480c9b40 2023-05-22T15:43:33.645 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.639+0000 7fd346ffd640 1 --2- 172.21.15.16:0/527056557 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd3340654a0 0x7fd334067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:33.646 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.639+0000 7fd34e49c640 1 --2- 172.21.15.16:0/527056557 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd3340654a0 0x7fd334067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:33.646 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.639+0000 7fd346ffd640 1 -- 172.21.15.16:0/527056557 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 6382+0+0 (secure 0 0 0) 0x7fd33c082370 con 0x7fd3480c9b40 2023-05-22T15:43:33.646 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.639+0000 7fd34e49c640 1 --2- 172.21.15.16:0/527056557 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd3340654a0 0x7fd334067950 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fd340007fd0 tx=0x7fd340007940 comp rx=0 tx=0).ready entity=mgr.7788 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:43:33.647 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.643+0000 7fd346ffd640 1 -- 172.21.15.16:0/527056557 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fd33c04fe30 con 0x7fd3480c9b40 2023-05-22T15:43:33.801 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.795+0000 7fd350700640 1 -- 172.21.15.16:0/527056557 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd34804dbe0 con 0x7fd3480c9b40 2023-05-22T15:43:33.805 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.799+0000 7fd346ffd640 1 -- 172.21.15.16:0/527056557 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v118) v1 ==== 74+0+17990 (secure 0 0 0) 0x7fd33c053450 con 0x7fd3480c9b40 2023-05-22T15:43:33.805 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:43:33.805 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":118,"fsid":"b2719834-7847-4d98-90a6-f65d57f85c49","created":"2023-05-22T15:26:25.259314+0000","modified":"2023-05-22T15:43:31.138085+0000","last_up_change":"2023-05-22T15:26:36.448524+0000","last_in_change":"2023-05-22T15:26:27.675668+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-05-22T15:26:38.019781+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-05-22T15:26:43.570514+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2023-05-22T15:43:28.684367+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"117","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2023-05-22T15:43:29.119294+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"118","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9e7ff854-b25f-4cbb-a5c2-1799d9c53d2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":37021},{"type":"v1","addr":"172.21.15.16:6825","nonce":37021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":37021},{"type":"v1","addr":"172.21.15.16:6827","nonce":37021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":37021},{"type":"v1","addr":"172.21.15.16:6831","nonce":37021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":37021},{"type":"v1","addr":"172.21.15.16:6829","nonce":37021}]},"public_addr":"172.21.15.16:6825/37021","cluster_addr":"172.21.15.16:6827/37021","heartbeat_back_addr":"172.21.15.16:6831/37021","heartbeat_front_addr":"172.21.15.16:6829/37021","state":["exists","up"]},{"osd":1,"uuid":"4702fbc9-a927-4565-aca5-41dc69cc8264","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":37008},{"type":"v1","addr":"172.21.15.16:6801","nonce":37008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":37008},{"type":"v1","addr":"172.21.15.16:6803","nonce":37008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":37008},{"type":"v1","addr":"172.21.15.16:6807","nonce":37008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":37008},{"type":"v1","addr":"172.21.15.16:6805","nonce":37008}]},"public_addr":"172.21.15.16:6801/37008","cluster_addr":"172.21.15.16:6803/37008","heartbeat_back_addr":"172.21.15.16:6807/37008","heartbeat_front_addr":"172.21.15.16:6805/37008","state":["exists","up"]},{"osd":2,"uuid":"1e7bba46-4eba-47ef-bb81-5e2d371638ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":37009},{"type":"v1","addr":"172.21.15.16:6809","nonce":37009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":37009},{"type":"v1","addr":"172.21.15.16:6811","nonce":37009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":37009},{"type":"v1","addr":"172.21.15.16:6815","nonce":37009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":37009},{"type":"v1","addr":"172.21.15.16:6813","nonce":37009}]},"public_addr":"172.21.15.16:6809/37009","cluster_addr":"172.21.15.16:6811/37009","heartbeat_back_addr":"172.21.15.16:6815/37009","heartbeat_front_addr":"172.21.15.16:6813/37009","state":["exists","up"]},{"osd":3,"uuid":"383cacc0-5a88-4904-990e-498ca59ef48a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":37020},{"type":"v1","addr":"172.21.15.16:6817","nonce":37020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":37020},{"type":"v1","addr":"172.21.15.16:6819","nonce":37020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":37020},{"type":"v1","addr":"172.21.15.16:6823","nonce":37020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":37020},{"type":"v1","addr":"172.21.15.16:6821","nonce":37020}]},"public_addr":"172.21.15.16:6817/37020","cluster_addr":"172.21.15.16:6819/37020","heartbeat_back_addr":"172.21.15.16:6823/37020","heartbeat_front_addr":"172.21.15.16:6821/37020","state":["exists","up"]},{"osd":4,"uuid":"970500ef-95b7-42c1-b080-eed4fae7013d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":37353},{"type":"v1","addr":"172.21.15.62:6817","nonce":37353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6819","nonce":37353},{"type":"v1","addr":"172.21.15.62:6822","nonce":37353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37353},{"type":"v1","addr":"172.21.15.62:6831","nonce":37353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37353},{"type":"v1","addr":"172.21.15.62:6826","nonce":37353}]},"public_addr":"172.21.15.62:6817/37353","cluster_addr":"172.21.15.62:6822/37353","heartbeat_back_addr":"172.21.15.62:6831/37353","heartbeat_front_addr":"172.21.15.62:6826/37353","state":["exists","up"]},{"osd":5,"uuid":"022d67db-d4d8-445b-b1b9-8b446a00f5c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":37352},{"type":"v1","addr":"172.21.15.62:6801","nonce":37352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":37352},{"type":"v1","addr":"172.21.15.62:6803","nonce":37352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":37352},{"type":"v1","addr":"172.21.15.62:6807","nonce":37352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":37352},{"type":"v1","addr":"172.21.15.62:6805","nonce":37352}]},"public_addr":"172.21.15.62:6801/37352","cluster_addr":"172.21.15.62:6803/37352","heartbeat_back_addr":"172.21.15.62:6807/37352","heartbeat_front_addr":"172.21.15.62:6805/37352","state":["exists","up"]},{"osd":6,"uuid":"42bd372b-6688-4a1b-8407-e16e8c266e33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":37354},{"type":"v1","addr":"172.21.15.62:6820","nonce":37354}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":37354},{"type":"v1","addr":"172.21.15.62:6823","nonce":37354}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6829","nonce":37354},{"type":"v1","addr":"172.21.15.62:6830","nonce":37354}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6825","nonce":37354},{"type":"v1","addr":"172.21.15.62:6827","nonce":37354}]},"public_addr":"172.21.15.62:6820/37354","cluster_addr":"172.21.15.62:6823/37354","heartbeat_back_addr":"172.21.15.62:6830/37354","heartbeat_front_addr":"172.21.15.62:6827/37354","state":["exists","up"]},{"osd":7,"uuid":"ac70b2a3-f41b-4bf2-b227-7f2cbb55426d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37355},{"type":"v1","addr":"172.21.15.62:6809","nonce":37355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37355},{"type":"v1","addr":"172.21.15.62:6811","nonce":37355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":37355},{"type":"v1","addr":"172.21.15.62:6815","nonce":37355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":37355},{"type":"v1","addr":"172.21.15.62:6813","nonce":37355}]},"public_addr":"172.21.15.62:6809/37355","cluster_addr":"172.21.15.62:6811/37355","heartbeat_back_addr":"172.21.15.62:6815/37355","heartbeat_front_addr":"172.21.15.62:6813/37355","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.368896+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.107273+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.276888+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.287502+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.124791+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.251937+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.255679+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.232797+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-05-22T15:43:33.806 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.803+0000 7fd350700640 1 -- 172.21.15.16:0/527056557 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd3340654a0 msgr2=0x7fd334067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:33.807 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.803+0000 7fd350700640 1 --2- 172.21.15.16:0/527056557 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd3340654a0 0x7fd334067950 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fd340007fd0 tx=0x7fd340007940 comp rx=0 tx=0).stop 2023-05-22T15:43:33.807 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.803+0000 7fd350700640 1 -- 172.21.15.16:0/527056557 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd3480c9b40 msgr2=0x7fd34815a2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:33.807 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.803+0000 7fd350700640 1 --2- 172.21.15.16:0/527056557 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd3480c9b40 0x7fd34815a2b0 secure :-1 s=READY pgs=1459 cs=0 l=1 rev1=1 crypto rx=0x7fd33c00c450 tx=0x7fd33c00c910 comp rx=0 tx=0).stop 2023-05-22T15:43:33.807 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.803+0000 7fd350700640 1 -- 172.21.15.16:0/527056557 shutdown_connections 2023-05-22T15:43:33.807 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.803+0000 7fd350700640 1 --2- 172.21.15.16:0/527056557 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd3480ca4e0 0x7fd348150780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:33.807 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.803+0000 7fd350700640 1 --2- 172.21.15.16:0/527056557 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd3480c9b40 0x7fd34815a2b0 unknown :-1 s=CLOSED pgs=1459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:33.808 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.803+0000 7fd350700640 1 --2- 172.21.15.16:0/527056557 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd3340654a0 0x7fd334067950 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:33.808 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.803+0000 7fd350700640 1 --2- 172.21.15.16:0/527056557 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd348071500 0x7fd348151220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:33.808 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.803+0000 7fd350700640 1 -- 172.21.15.16:0/527056557 >> 172.21.15.16:0/527056557 conn(0x7fd348069720 msgr2=0x7fd3480aedf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:43:33.808 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.803+0000 7fd350700640 1 -- 172.21.15.16:0/527056557 shutdown_connections 2023-05-22T15:43:33.808 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.803+0000 7fd350700640 1 -- 172.21.15.16:0/527056557 wait complete. 2023-05-22T15:43:33.823 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-05-22T15:43:33.824 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-05-22T15:43:33.937 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.931+0000 7fd40e09a640 1 Processor -- start 2023-05-22T15:43:33.938 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.931+0000 7fd40e09a640 1 -- start start 2023-05-22T15:43:33.938 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.935+0000 7fd40e09a640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd4080cbf80 0x7fd40806b210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:33.938 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.935+0000 7fd40e09a640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fd4080b22e0 0x7fd40806bec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:33.939 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.935+0000 7fd40e09a640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd40806db40 con 0x7fd40806c400 2023-05-22T15:43:33.939 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.935+0000 7fd40e09a640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd40806c880 con 0x7fd4080b22e0 2023-05-22T15:43:33.939 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.935+0000 7fd40e09a640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4080cf3f0 con 0x7fd4080cbf80 2023-05-22T15:43:33.943 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.935+0000 7fd4077fe640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd4080cbf80 0x7fd40806b210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:33.943 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.935+0000 7fd406ffd640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fd4080b22e0 0x7fd40806bec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:33.944 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.935+0000 7fd4077fe640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd4080cbf80 0x7fd40806b210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3301/0 says I am v2:172.21.15.16:35176/0 (socket says 172.21.15.16:35176) 2023-05-22T15:43:33.944 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.935+0000 7fd407fff640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7fd40806c400 0x7fd40815f810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.16:34198/0 (socket says 172.21.15.16:34198) 2023-05-22T15:43:33.944 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.935+0000 7fd4077fe640 1 -- 172.21.15.16:0/1265511981 learned_addr learned my addr 172.21.15.16:0/1265511981 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:43:33.945 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.935+0000 7fd4067fc640 1 -- 172.21.15.16:0/1265511981 <== mon.1 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1010255117 0 0) 0x7fd40806db40 con 0x7fd40806c400 2023-05-22T15:43:33.945 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.935+0000 7fd4067fc640 1 -- 172.21.15.16:0/1265511981 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd3f0003530 con 0x7fd40806c400 2023-05-22T15:43:33.945 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.935+0000 7fd406ffd640 1 -- 172.21.15.16:0/1265511981 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd4080cbf80 msgr2=0x7fd40806b210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:43:33.945 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.935+0000 7fd406ffd640 1 --2- 172.21.15.16:0/1265511981 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd4080cbf80 0x7fd40806b210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:33.946 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.935+0000 7fd406ffd640 1 -- 172.21.15.16:0/1265511981 >> v1:172.21.15.62:6789/0 conn(0x7fd40806c400 legacy=0x7fd40815f810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:33.946 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.935+0000 7fd406ffd640 1 -- 172.21.15.16:0/1265511981 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd40815c080 con 0x7fd4080b22e0 2023-05-22T15:43:33.946 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.935+0000 7fd4077fe640 1 --2- 172.21.15.16:0/1265511981 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd4080cbf80 0x7fd40806b210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-05-22T15:43:33.946 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.939+0000 7fd406ffd640 1 --2- 172.21.15.16:0/1265511981 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd4080b22e0 0x7fd40806bec0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fd3fc0097c0 tx=0x7fd3fc01c450 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1e00d03696c8b554 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:43:33.946 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.939+0000 7fd4067fc640 1 -- 172.21.15.16:0/1265511981 <== mon.1 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3fc0ab650 con 0x7fd4080b22e0 2023-05-22T15:43:33.947 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.939+0000 7fd4067fc640 1 -- 172.21.15.16:0/1265511981 <== mon.1 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3fc0ab7b0 con 0x7fd4080b22e0 2023-05-22T15:43:33.947 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.939+0000 7fd4067fc640 1 -- 172.21.15.16:0/1265511981 <== mon.1 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3fc0abb20 con 0x7fd4080b22e0 2023-05-22T15:43:33.952 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.939+0000 7fd40e09a640 1 -- 172.21.15.16:0/1265511981 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd4080b22e0 msgr2=0x7fd40806bec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:33.952 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.939+0000 7fd40e09a640 1 --2- 172.21.15.16:0/1265511981 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd4080b22e0 0x7fd40806bec0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fd3fc0097c0 tx=0x7fd3fc01c450 comp rx=0 tx=0).stop 2023-05-22T15:43:33.953 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.939+0000 7fd40e09a640 1 -- 172.21.15.16:0/1265511981 shutdown_connections 2023-05-22T15:43:33.953 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.939+0000 7fd40e09a640 1 --2- 172.21.15.16:0/1265511981 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd4080b22e0 0x7fd40806bec0 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:33.953 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.939+0000 7fd40e09a640 1 --2- 172.21.15.16:0/1265511981 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd4080cbf80 0x7fd40806b210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:33.953 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.939+0000 7fd40e09a640 1 -- 172.21.15.16:0/1265511981 >> 172.21.15.16:0/1265511981 conn(0x7fd40806f560 msgr2=0x7fd40806f960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:43:33.953 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.939+0000 7fd40e09a640 1 -- 172.21.15.16:0/1265511981 shutdown_connections 2023-05-22T15:43:33.953 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.939+0000 7fd40e09a640 1 -- 172.21.15.16:0/1265511981 wait complete. 2023-05-22T15:43:33.953 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.939+0000 7fd40e09a640 1 Processor -- start 2023-05-22T15:43:33.954 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.939+0000 7fd40e09a640 1 -- start start 2023-05-22T15:43:33.954 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.943+0000 7fd40e09a640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd40806c400 0x7fd4080cb300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:33.954 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.943+0000 7fd40e09a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd4080b22e0 0x7fd4080c9a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:33.954 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.943+0000 7fd40e09a640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd4080cbf80 0x7fd4081557b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:33.954 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.943+0000 7fd40e09a640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4080b4550 con 0x7fd4080b22e0 2023-05-22T15:43:33.955 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.943+0000 7fd40e09a640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd408155cf0 con 0x7fd40806c400 2023-05-22T15:43:33.955 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.943+0000 7fd40e09a640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4080c74d0 con 0x7fd4080cbf80 2023-05-22T15:43:33.955 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.943+0000 7fd406ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd4080b22e0 0x7fd4080c9a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:33.955 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.943+0000 7fd406ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd4080b22e0 0x7fd4080c9a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36056/0 (socket says 172.21.15.16:36056) 2023-05-22T15:43:33.956 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.943+0000 7fd406ffd640 1 -- 172.21.15.16:0/3508921988 learned_addr learned my addr 172.21.15.16:0/3508921988 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:43:33.956 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.943+0000 7fd4077fe640 1 --2- 172.21.15.16:0/3508921988 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd40806c400 0x7fd4080cb300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:33.956 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.943+0000 7fd407fff640 1 --2- 172.21.15.16:0/3508921988 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd4080cbf80 0x7fd4081557b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:33.956 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.943+0000 7fd406ffd640 1 -- 172.21.15.16:0/3508921988 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd4080cbf80 msgr2=0x7fd4081557b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:33.957 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.943+0000 7fd406ffd640 1 --2- 172.21.15.16:0/3508921988 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd4080cbf80 0x7fd4081557b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:33.957 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.943+0000 7fd406ffd640 1 -- 172.21.15.16:0/3508921988 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd40806c400 msgr2=0x7fd4080cb300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:33.957 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.943+0000 7fd406ffd640 1 --2- 172.21.15.16:0/3508921988 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd40806c400 0x7fd4080cb300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:33.957 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.943+0000 7fd406ffd640 1 -- 172.21.15.16:0/3508921988 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3f80094d0 con 0x7fd4080b22e0 2023-05-22T15:43:33.958 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.943+0000 7fd406ffd640 1 --2- 172.21.15.16:0/3508921988 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd4080b22e0 0x7fd4080c9a50 secure :-1 s=READY pgs=1460 cs=0 l=1 rev1=1 crypto rx=0x7fd3fc0ad040 tx=0x7fd3fc0191f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:43:33.958 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.943+0000 7fd3effff640 1 -- 172.21.15.16:0/3508921988 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3fc004450 con 0x7fd4080b22e0 2023-05-22T15:43:33.958 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.943+0000 7fd3effff640 1 -- 172.21.15.16:0/3508921988 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3fc0ab920 con 0x7fd4080b22e0 2023-05-22T15:43:33.958 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.943+0000 7fd40e09a640 1 -- 172.21.15.16:0/3508921988 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3fc009420 con 0x7fd4080b22e0 2023-05-22T15:43:33.958 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.943+0000 7fd3effff640 1 -- 172.21.15.16:0/3508921988 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3fc0abcb0 con 0x7fd4080b22e0 2023-05-22T15:43:33.958 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.943+0000 7fd40e09a640 1 -- 172.21.15.16:0/3508921988 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd40806b750 con 0x7fd4080b22e0 2023-05-22T15:43:33.959 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.943+0000 7fd40e09a640 1 -- 172.21.15.16:0/3508921988 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd40806e380 con 0x7fd4080b22e0 2023-05-22T15:43:33.959 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.943+0000 7fd3effff640 1 -- 172.21.15.16:0/3508921988 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 41) v1 ==== 84332+0+0 (secure 0 0 0) 0x7fd3fc003620 con 0x7fd4080b22e0 2023-05-22T15:43:33.959 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.943+0000 7fd3effff640 1 --2- 172.21.15.16:0/3508921988 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd3f40654a0 0x7fd3f4067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:33.959 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.943+0000 7fd3effff640 1 -- 172.21.15.16:0/3508921988 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 6382+0+0 (secure 0 0 0) 0x7fd3fc11cdf0 con 0x7fd4080b22e0 2023-05-22T15:43:33.959 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.943+0000 7fd4077fe640 1 --2- 172.21.15.16:0/3508921988 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd3f40654a0 0x7fd3f4067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:33.959 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.943+0000 7fd4077fe640 1 --2- 172.21.15.16:0/3508921988 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd3f40654a0 0x7fd3f4067950 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fd3f8009bd0 tx=0x7fd3f8008040 comp rx=0 tx=0).ready entity=mgr.7788 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:43:33.960 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:33.947+0000 7fd3effff640 1 -- 172.21.15.16:0/3508921988 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fd3fc0ea7c0 con 0x7fd4080b22e0 2023-05-22T15:43:34.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:34.099+0000 7fd40e09a640 1 -- 172.21.15.16:0/3508921988 --> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd4080cc350 con 0x7fd3f40654a0 2023-05-22T15:43:34.127 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:34.123+0000 7fd3effff640 1 -- 172.21.15.16:0/3508921988 <== mgr.7788 v2:172.21.15.16:6832/36921 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+310237 (secure 0 0 0) 0x7fd4080cc350 con 0x7fd3f40654a0 2023-05-22T15:43:34.128 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:43:34.131 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2023-05-22T15:43:34.132 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:34.127+0000 7fd40e09a640 1 -- 172.21.15.16:0/3508921988 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd3f40654a0 msgr2=0x7fd3f4067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:34.132 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:34.127+0000 7fd40e09a640 1 --2- 172.21.15.16:0/3508921988 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd3f40654a0 0x7fd3f4067950 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fd3f8009bd0 tx=0x7fd3f8008040 comp rx=0 tx=0).stop 2023-05-22T15:43:34.132 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:34.127+0000 7fd40e09a640 1 -- 172.21.15.16:0/3508921988 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd4080b22e0 msgr2=0x7fd4080c9a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:34.132 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:34.127+0000 7fd40e09a640 1 --2- 172.21.15.16:0/3508921988 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd4080b22e0 0x7fd4080c9a50 secure :-1 s=READY pgs=1460 cs=0 l=1 rev1=1 crypto rx=0x7fd3fc0ad040 tx=0x7fd3fc0191f0 comp rx=0 tx=0).stop 2023-05-22T15:43:34.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:34.127+0000 7fd40e09a640 1 -- 172.21.15.16:0/3508921988 shutdown_connections 2023-05-22T15:43:34.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:34.127+0000 7fd40e09a640 1 --2- 172.21.15.16:0/3508921988 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd3f40654a0 0x7fd3f4067950 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:34.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:34.127+0000 7fd40e09a640 1 --2- 172.21.15.16:0/3508921988 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd4080cbf80 0x7fd4081557b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:34.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:34.127+0000 7fd40e09a640 1 --2- 172.21.15.16:0/3508921988 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd4080b22e0 0x7fd4080c9a50 unknown :-1 s=CLOSED pgs=1460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:34.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:34.127+0000 7fd40e09a640 1 --2- 172.21.15.16:0/3508921988 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd40806c400 0x7fd4080cb300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:34.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:34.127+0000 7fd40e09a640 1 -- 172.21.15.16:0/3508921988 >> 172.21.15.16:0/3508921988 conn(0x7fd40806f560 msgr2=0x7fd4081583e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:43:34.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:34.127+0000 7fd40e09a640 1 -- 172.21.15.16:0/3508921988 shutdown_connections 2023-05-22T15:43:34.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:34.127+0000 7fd40e09a640 1 -- 172.21.15.16:0/3508921988 wait complete. 2023-05-22T15:43:34.150 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2023-05-22T15:43:32.996157+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592698,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":884,"num_read_kb":1748,"num_write":364,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":244,"ondisk_log_size":244,"up":210,"acting":210,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":98,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219912,"kb_used_data":12488,"kb_used_omap":0,"kb_used_meta":207360,"kb_avail":754754808,"statfs":{"total":773094113280,"available":772868923392,"internally_reserved":0,"allocated":12787712,"data_stored":7247012,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":212336640},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,4,0,1,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":73,"apply_latency_ms":73,"commit_latency_ns":73000000,"apply_latency_ns":73000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2330,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.256482"},"pg_stats":[{"pgid":"15.3f","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-05-22T15:43:30.184101+0000","last_change":"2023-05-22T15:43:30.183119+0000","last_active":"2023-05-22T15:43:30.184101+0000","last_peered":"2023-05-22T15:43:30.184101+0000","last_clean":"2023-05-22T15:43:30.184101+0000","last_became_active":"2023-05-22T15:43:30.182859+0000","last_became_peered":"2023-05-22T15:43:30.182859+0000","last_unstale":"2023-05-22T15:43:30.184101+0000","last_undegraded":"2023-05-22T15:43:30.184101+0000","last_fullsized":"2023-05-22T15:43:30.184101+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_clean_scrub_stamp":"2023-05-22T15:43:29.738376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:52:09.288277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.3e","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-05-22T15:43:30.164706+0000","last_change":"2023-05-22T15:43:30.163639+0000","last_active":"2023-05-22T15:43:30.164706+0000","last_peered":"2023-05-22T15:43:30.164706+0000","last_clean":"2023-05-22T15:43:30.164706+0000","last_became_active":"2023-05-22T15:43:30.163136+0000","last_became_peered":"2023-05-22T15:43:30.163136+0000","last_unstale":"2023-05-22T15:43:30.164706+0000","last_undegraded":"2023-05-22T15:43:30.164706+0000","last_fullsized":"2023-05-22T15:43:30.164706+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_clean_scrub_stamp":"2023-05-22T15:43:29.738376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T03:40:48.926917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.3d","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-05-22T15:43:30.187909+0000","last_change":"2023-05-22T15:43:30.187039+0000","last_active":"2023-05-22T15:43:30.187909+0000","last_peered":"2023-05-22T15:43:30.187909+0000","last_clean":"2023-05-22T15:43:30.187909+0000","last_became_active":"2023-05-22T15:43:30.186793+0000","last_became_peered":"2023-05-22T15:43:30.186793+0000","last_unstale":"2023-05-22T15:43:30.187909+0000","last_undegraded":"2023-05-22T15:43:30.187909+0000","last_fullsized":"2023-05-22T15:43:30.187909+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_clean_scrub_stamp":"2023-05-22T15:43:29.738376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:44:35.350206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.3c","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-05-22T15:43:30.163798+0000","last_change":"2023-05-22T15:43:30.162829+0000","last_active":"2023-05-22T15:43:30.163798+0000","last_peered":"2023-05-22T15:43:30.163798+0000","last_clean":"2023-05-22T15:43:30.163798+0000","last_became_active":"2023-05-22T15:43:30.162517+0000","last_became_peered":"2023-05-22T15:43:30.162517+0000","last_unstale":"2023-05-22T15:43:30.163798+0000","last_undegraded":"2023-05-22T15:43:30.163798+0000","last_fullsized":"2023-05-22T15:43:30.163798+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_clean_scrub_stamp":"2023-05-22T15:43:29.738376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:38:15.549682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.3b","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-05-22T15:43:30.211224+0000","last_change":"2023-05-22T15:43:30.181145+0000","last_active":"2023-05-22T15:43:30.211224+0000","last_peered":"2023-05-22T15:43:30.211224+0000","last_clean":"2023-05-22T15:43:30.211224+0000","last_became_active":"2023-05-22T15:43:30.180391+0000","last_became_peered":"2023-05-22T15:43:30.180391+0000","last_unstale":"2023-05-22T15:43:30.211224+0000","last_undegraded":"2023-05-22T15:43:30.211224+0000","last_fullsized":"2023-05-22T15:43:30.211224+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_clean_scrub_stamp":"2023-05-22T15:43:29.738376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:34:00.384115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.3a","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-05-22T15:43:30.214374+0000","last_change":"2023-05-22T15:43:30.206158+0000","last_active":"2023-05-22T15:43:30.214374+0000","last_peered":"2023-05-22T15:43:30.214374+0000","last_clean":"2023-05-22T15:43:30.214374+0000","last_became_active":"2023-05-22T15:43:30.205344+0000","last_became_peered":"2023-05-22T15:43:30.205344+0000","last_unstale":"2023-05-22T15:43:30.214374+0000","last_undegraded":"2023-05-22T15:43:30.214374+0000","last_fullsized":"2023-05-22T15:43:30.214374+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_clean_scrub_stamp":"2023-05-22T15:43:29.738376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:20:57.149814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.39","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-05-22T15:43:30.210122+0000","last_change":"2023-05-22T15:43:30.203885+0000","last_active":"2023-05-22T15:43:30.210122+0000","last_peered":"2023-05-22T15:43:30.210122+0000","last_clean":"2023-05-22T15:43:30.210122+0000","last_became_active":"2023-05-22T15:43:30.203150+0000","last_became_peered":"2023-05-22T15:43:30.203150+0000","last_unstale":"2023-05-22T15:43:30.210122+0000","last_undegraded":"2023-05-22T15:43:30.210122+0000","last_fullsized":"2023-05-22T15:43:30.210122+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_clean_scrub_stamp":"2023-05-22T15:43:29.738376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:04:28.418159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.38","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-05-22T15:43:30.190236+0000","last_change":"2023-05-22T15:43:30.189579+0000","last_active":"2023-05-22T15:43:30.190236+0000","last_peered":"2023-05-22T15:43:30.190236+0000","last_clean":"2023-05-22T15:43:30.190236+0000","last_became_active":"2023-05-22T15:43:30.189264+0000","last_became_peered":"2023-05-22T15:43:30.189264+0000","last_unstale":"2023-05-22T15:43:30.190236+0000","last_undegraded":"2023-05-22T15:43:30.190236+0000","last_fullsized":"2023-05-22T15:43:30.190236+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_clean_scrub_stamp":"2023-05-22T15:43:29.738376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:03:33.396793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.37","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-05-22T15:43:30.224021+0000","last_change":"2023-05-22T15:43:30.211619+0000","last_active":"2023-05-22T15:43:30.224021+0000","last_peered":"2023-05-22T15:43:30.224021+0000","last_clean":"2023-05-22T15:43:30.224021+0000","last_became_active":"2023-05-22T15:43:30.210848+0000","last_became_peered":"2023-05-22T15:43:30.210848+0000","last_unstale":"2023-05-22T15:43:30.224021+0000","last_undegraded":"2023-05-22T15:43:30.224021+0000","last_fullsized":"2023-05-22T15:43:30.224021+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_clean_scrub_stamp":"2023-05-22T15:43:29.738376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:36:26.720817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.36","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-05-22T15:43:30.211139+0000","last_change":"2023-05-22T15:43:30.197809+0000","last_active":"2023-05-22T15:43:30.211139+0000","last_peered":"2023-05-22T15:43:30.211139+0000","last_clean":"2023-05-22T15:43:30.211139+0000","last_became_active":"2023-05-22T15:43:30.197078+0000","last_became_peered":"2023-05-22T15:43:30.197078+0000","last_unstale":"2023-05-22T15:43:30.211139+0000","last_undegraded":"2023-05-22T15:43:30.211139+0000","last_fullsized":"2023-05-22T15:43:30.211139+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_clean_scrub_stamp":"2023-05-22T15:43:29.738376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T16:04:25.079184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.35","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-05-22T15:43:30.222171+0000","last_change":"2023-05-22T15:43:30.220667+0000","last_active":"2023-05-22T15:43:30.222171+0000","last_peered":"2023-05-22T15:43:30.222171+0000","last_clean":"2023-05-22T15:43:30.222171+0000","last_became_active":"2023-05-22T15:43:30.219182+0000","last_became_peered":"2023-05-22T15:43:30.219182+0000","last_unstale":"2023-05-22T15:43:30.222171+0000","last_undegraded":"2023-05-22T15:43:30.222171+0000","last_fullsized":"2023-05-22T15:43:30.222171+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_clean_scrub_stamp":"2023-05-22T15:43:29.738376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:38:15.549682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.34","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-05-22T15:43:30.223901+0000","last_change":"2023-05-22T15:43:30.209645+0000","last_active":"2023-05-22T15:43:30.223901+0000","last_peered":"2023-05-22T15:43:30.223901+0000","last_clean":"2023-05-22T15:43:30.223901+0000","last_became_active":"2023-05-22T15:43:30.208670+0000","last_became_peered":"2023-05-22T15:43:30.208670+0000","last_unstale":"2023-05-22T15:43:30.223901+0000","last_undegraded":"2023-05-22T15:43:30.223901+0000","last_fullsized":"2023-05-22T15:43:30.223901+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_clean_scrub_stamp":"2023-05-22T15:43:29.738376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:38:15.549682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.33","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-05-22T15:43:30.207751+0000","last_change":"2023-05-22T15:43:30.207287+0000","last_active":"2023-05-22T15:43:30.207751+0000","last_peered":"2023-05-22T15:43:30.207751+0000","last_clean":"2023-05-22T15:43:30.207751+0000","last_became_active":"2023-05-22T15:43:30.207040+0000","last_became_peered":"2023-05-22T15:43:30.207040+0000","last_unstale":"2023-05-22T15:43:30.207751+0000","last_undegraded":"2023-05-22T15:43:30.207751+0000","last_fullsized":"2023-05-22T15:43:30.207751+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_clean_scrub_stamp":"2023-05-22T15:43:29.738376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T16:34:27.749652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.32","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-05-22T15:43:30.181218+0000","last_change":"2023-05-22T15:43:30.180424+0000","last_active":"2023-05-22T15:43:30.181218+0000","last_peered":"2023-05-22T15:43:30.181218+0000","last_clean":"2023-05-22T15:43:30.181218+0000","last_became_active":"2023-05-22T15:43:30.180064+0000","last_became_peered":"2023-05-22T15:43:30.180064+0000","last_unstale":"2023-05-22T15:43:30.181218+0000","last_undegraded":"2023-05-22T15:43:30.181218+0000","last_fullsized":"2023-05-22T15:43:30.181218+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_clean_scrub_stamp":"2023-05-22T15:43:29.738376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:57:34.706301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.31","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-05-22T15:43:30.171923+0000","last_change":"2023-05-22T15:43:30.169125+0000","last_active":"2023-05-22T15:43:30.171923+0000","last_peered":"2023-05-22T15:43:30.171923+0000","last_clean":"2023-05-22T15:43:30.171923+0000","last_became_active":"2023-05-22T15:43:30.168764+0000","last_became_peered":"2023-05-22T15:43:30.168764+0000","last_unstale":"2023-05-22T15:43:30.171923+0000","last_undegraded":"2023-05-22T15:43:30.171923+0000","last_fullsized":"2023-05-22T15:43:30.171923+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_clean_scrub_stamp":"2023-05-22T15:43:29.738376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:20:06.441496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.30","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-05-22T15:43:30.206221+0000","last_change":"2023-05-22T15:43:30.193444+0000","last_active":"2023-05-22T15:43:30.206221+0000","last_peered":"2023-05-22T15:43:30.206221+0000","last_clean":"2023-05-22T15:43:30.206221+0000","last_became_active":"2023-05-22T15:43:30.192461+0000","last_became_peered":"2023-05-22T15:43:30.192461+0000","last_unstale":"2023-05-22T15:43:30.206221+0000","last_undegraded":"2023-05-22T15:43:30.206221+0000","last_fullsized":"2023-05-22T15:43:30.206221+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_clean_scrub_stamp":"2023-05-22T15:43:29.738376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:58:34.403442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.2f","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-05-22T15:43:30.153787+0000","last_change":"2023-05-22T15:43:30.151800+0000","last_active":"2023-05-22T15:43:30.153787+0000","last_peered":"2023-05-22T15:43:30.153787+0000","last_clean":"2023-05-22T15:43:30.153787+0000","last_became_active":"2023-05-22T15:43:30.151509+0000","last_became_peered":"2023-05-22T15:43:30.151509+0000","last_unstale":"2023-05-22T15:43:30.153787+0000","last_undegraded":"2023-05-22T15:43:30.153787+0000","last_fullsized":"2023-05-22T15:43:30.153787+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_clean_scrub_stamp":"2023-05-22T15:43:29.738376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T18:09:47.059168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.2e","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-05-22T15:43:30.165194+0000","last_change":"2023-05-22T15:43:30.162990+0000","last_active":"2023-05-22T15:43:30.165194+0000","last_peered":"2023-05-22T15:43:30.165194+0000","last_clean":"2023-05-22T15:43:30.165194+0000","last_became_active":"2023-05-22T15:43:30.162647+0000","last_became_peered":"2023-05-22T15:43:30.162647+0000","last_unstale":"2023-05-22T15:43:30.165194+0000","last_undegraded":"2023-05-22T15:43:30.165194+0000","last_fullsized":"2023-05-22T15:43:30.165194+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_clean_scrub_stamp":"2023-05-22T15:43:29.738376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:20:06.684861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.2d","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-05-22T15:43:30.175878+0000","last_change":"2023-05-22T15:43:30.170512+0000","last_active":"2023-05-22T15:43:30.175878+0000","last_peered":"2023-05-22T15:43:30.175878+0000","last_clean":"2023-05-22T15:43:30.175878+0000","last_became_active":"2023-05-22T15:43:30.170187+0000","last_became_peered":"2023-05-22T15:43:30.170187+0000","last_unstale":"2023-05-22T15:43:30.175878+0000","last_undegraded":"2023-05-22T15:43:30.175878+0000","last_fullsized":"2023-05-22T15:43:30.175878+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_clean_scrub_stamp":"2023-05-22T15:43:29.738376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:05:02.019815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2c","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-05-22T15:43:30.223926+0000","last_change":"2023-05-22T15:43:30.209849+0000","last_active":"2023-05-22T15:43:30.223926+0000","last_peered":"2023-05-22T15:43:30.223926+0000","last_clean":"2023-05-22T15:43:30.223926+0000","last_became_active":"2023-05-22T15:43:30.209224+0000","last_became_peered":"2023-05-22T15:43:30.209224+0000","last_unstale":"2023-05-22T15:43:30.223926+0000","last_undegraded":"2023-05-22T15:43:30.223926+0000","last_fullsized":"2023-05-22T15:43:30.223926+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_clean_scrub_stamp":"2023-05-22T15:43:29.738376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:09:50.754411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.2b","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-05-22T15:43:30.197970+0000","last_change":"2023-05-22T15:43:30.197011+0000","last_active":"2023-05-22T15:43:30.197970+0000","last_peered":"2023-05-22T15:43:30.197970+0000","last_clean":"2023-05-22T15:43:30.197970+0000","last_became_active":"2023-05-22T15:43:30.196709+0000","last_became_peered":"2023-05-22T15:43:30.196709+0000","last_unstale":"2023-05-22T15:43:30.197970+0000","last_undegraded":"2023-05-22T15:43:30.197970+0000","last_fullsized":"2023-05-22T15:43:30.197970+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_clean_scrub_stamp":"2023-05-22T15:43:29.738376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T03:33:14.051311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2a","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-05-22T15:43:30.161058+0000","last_change":"2023-05-22T15:43:30.156401+0000","last_active":"2023-05-22T15:43:30.161058+0000","last_peered":"2023-05-22T15:43:30.161058+0000","last_clean":"2023-05-22T15:43:30.161058+0000","last_became_active":"2023-05-22T15:43:30.156023+0000","last_became_peered":"2023-05-22T15:43:30.156023+0000","last_unstale":"2023-05-22T15:43:30.161058+0000","last_undegraded":"2023-05-22T15:43:30.161058+0000","last_fullsized":"2023-05-22T15:43:30.161058+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_clean_scrub_stamp":"2023-05-22T15:43:29.738376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:59:43.398245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.29","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-05-22T15:43:30.161024+0000","last_change":"2023-05-22T15:43:30.155803+0000","last_active":"2023-05-22T15:43:30.161024+0000","last_peered":"2023-05-22T15:43:30.161024+0000","last_clean":"2023-05-22T15:43:30.161024+0000","last_became_active":"2023-05-22T15:43:30.155488+0000","last_became_peered":"2023-05-22T15:43:30.155488+0000","last_unstale":"2023-05-22T15:43:30.161024+0000","last_undegraded":"2023-05-22T15:43:30.161024+0000","last_fullsized":"2023-05-22T15:43:30.161024+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_clean_scrub_stamp":"2023-05-22T15:43:29.738376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:35:30.801478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.28","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-05-22T15:43:30.206165+0000","last_change":"2023-05-22T15:43:30.191603+0000","last_active":"2023-05-22T15:43:30.206165+0000","last_peered":"2023-05-22T15:43:30.206165+0000","last_clean":"2023-05-22T15:43:30.206165+0000","last_became_active":"2023-05-22T15:43:30.190473+0000","last_became_peered":"2023-05-22T15:43:30.190473+0000","last_unstale":"2023-05-22T15:43:30.206165+0000","last_undegraded":"2023-05-22T15:43:30.206165+0000","last_fullsized":"2023-05-22T15:43:30.206165+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_clean_scrub_stamp":"2023-05-22T15:43:29.738376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:48:02.716268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.27","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-05-22T15:43:30.212521+0000","last_change":"2023-05-22T15:43:30.181035+0000","last_active":"2023-05-22T15:43:30.212521+0000","last_peered":"2023-05-22T15:43:30.212521+0000","last_clean":"2023-05-22T15:43:30.212521+0000","last_became_active":"2023-05-22T15:43:30.179544+0000","last_became_peered":"2023-05-22T15:43:30.179544+0000","last_unstale":"2023-05-22T15:43:30.212521+0000","last_undegraded":"2023-05-22T15:43:30.212521+0000","last_fullsized":"2023-05-22T15:43:30.212521+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_clean_scrub_stamp":"2023-05-22T15:43:29.738376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:46:18.725589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.26","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-05-22T15:43:30.172660+0000","last_change":"2023-05-22T15:43:30.168435+0000","last_active":"2023-05-22T15:43:30.172660+0000","last_peered":"2023-05-22T15:43:30.172660+0000","last_clean":"2023-05-22T15:43:30.172660+0000","last_became_active":"2023-05-22T15:43:30.168152+0000","last_became_peered":"2023-05-22T15:43:30.168152+0000","last_unstale":"2023-05-22T15:43:30.172660+0000","last_undegraded":"2023-05-22T15:43:30.172660+0000","last_fullsized":"2023-05-22T15:43:30.172660+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_clean_scrub_stamp":"2023-05-22T15:43:29.738376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:19:08.559088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.25","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-05-22T15:43:30.208217+0000","last_change":"2023-05-22T15:43:30.190690+0000","last_active":"2023-05-22T15:43:30.208217+0000","last_peered":"2023-05-22T15:43:30.208217+0000","last_clean":"2023-05-22T15:43:30.208217+0000","last_became_active":"2023-05-22T15:43:30.190035+0000","last_became_peered":"2023-05-22T15:43:30.190035+0000","last_unstale":"2023-05-22T15:43:30.208217+0000","last_undegraded":"2023-05-22T15:43:30.208217+0000","last_fullsized":"2023-05-22T15:43:30.208217+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_clean_scrub_stamp":"2023-05-22T15:43:29.738376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:18:19.994700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.24","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-05-22T15:43:30.211187+0000","last_change":"2023-05-22T15:43:30.184195+0000","last_active":"2023-05-22T15:43:30.211187+0000","last_peered":"2023-05-22T15:43:30.211187+0000","last_clean":"2023-05-22T15:43:30.211187+0000","last_became_active":"2023-05-22T15:43:30.180854+0000","last_became_peered":"2023-05-22T15:43:30.180854+0000","last_unstale":"2023-05-22T15:43:30.211187+0000","last_undegraded":"2023-05-22T15:43:30.211187+0000","last_fullsized":"2023-05-22T15:43:30.211187+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_clean_scrub_stamp":"2023-05-22T15:43:29.738376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T15:59:51.980766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.23","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-05-22T15:43:30.172717+0000","last_change":"2023-05-22T15:43:30.168398+0000","last_active":"2023-05-22T15:43:30.172717+0000","last_peered":"2023-05-22T15:43:30.172717+0000","last_clean":"2023-05-22T15:43:30.172717+0000","last_became_active":"2023-05-22T15:43:30.168063+0000","last_became_peered":"2023-05-22T15:43:30.168063+0000","last_unstale":"2023-05-22T15:43:30.172717+0000","last_undegraded":"2023-05-22T15:43:30.172717+0000","last_fullsized":"2023-05-22T15:43:30.172717+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_clean_scrub_stamp":"2023-05-22T15:43:29.738376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:46:43.160476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.22","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-05-22T15:43:30.175849+0000","last_change":"2023-05-22T15:43:30.170436+0000","last_active":"2023-05-22T15:43:30.175849+0000","last_peered":"2023-05-22T15:43:30.175849+0000","last_clean":"2023-05-22T15:43:30.175849+0000","last_became_active":"2023-05-22T15:43:30.170139+0000","last_became_peered":"2023-05-22T15:43:30.170139+0000","last_unstale":"2023-05-22T15:43:30.175849+0000","last_undegraded":"2023-05-22T15:43:30.175849+0000","last_fullsized":"2023-05-22T15:43:30.175849+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_clean_scrub_stamp":"2023-05-22T15:43:29.738376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:59:49.774332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.21","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-05-22T15:43:30.186714+0000","last_change":"2023-05-22T15:43:30.185698+0000","last_active":"2023-05-22T15:43:30.186714+0000","last_peered":"2023-05-22T15:43:30.186714+0000","last_clean":"2023-05-22T15:43:30.186714+0000","last_became_active":"2023-05-22T15:43:30.185389+0000","last_became_peered":"2023-05-22T15:43:30.185389+0000","last_unstale":"2023-05-22T15:43:30.186714+0000","last_undegraded":"2023-05-22T15:43:30.186714+0000","last_fullsized":"2023-05-22T15:43:30.186714+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_clean_scrub_stamp":"2023-05-22T15:43:29.738376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:52:20.517587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.20","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-05-22T15:43:30.175792+0000","last_change":"2023-05-22T15:43:30.169903+0000","last_active":"2023-05-22T15:43:30.175792+0000","last_peered":"2023-05-22T15:43:30.175792+0000","last_clean":"2023-05-22T15:43:30.175792+0000","last_became_active":"2023-05-22T15:43:30.169322+0000","last_became_peered":"2023-05-22T15:43:30.169322+0000","last_unstale":"2023-05-22T15:43:30.175792+0000","last_undegraded":"2023-05-22T15:43:30.175792+0000","last_fullsized":"2023-05-22T15:43:30.175792+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_clean_scrub_stamp":"2023-05-22T15:43:29.738376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T16:59:50.668128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1e","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-05-22T15:43:30.203107+0000","last_change":"2023-05-22T15:43:30.202292+0000","last_active":"2023-05-22T15:43:30.203107+0000","last_peered":"2023-05-22T15:43:30.203107+0000","last_clean":"2023-05-22T15:43:30.203107+0000","last_became_active":"2023-05-22T15:43:30.202084+0000","last_became_peered":"2023-05-22T15:43:30.202084+0000","last_unstale":"2023-05-22T15:43:30.203107+0000","last_undegraded":"2023-05-22T15:43:30.203107+0000","last_fullsized":"2023-05-22T15:43:30.203107+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_clean_scrub_stamp":"2023-05-22T15:43:29.738376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T16:59:50.668128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1f","version":"117'4","reported_seq":14,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-05-22T15:43:30.195460+0000","last_change":"2023-05-22T15:43:29.802221+0000","last_active":"2023-05-22T15:43:30.195460+0000","last_peered":"2023-05-22T15:43:30.195460+0000","last_clean":"2023-05-22T15:43:30.195460+0000","last_became_active":"2023-05-22T15:43:29.801774+0000","last_became_peered":"2023-05-22T15:43:29.801774+0000","last_unstale":"2023-05-22T15:43:30.195460+0000","last_undegraded":"2023-05-22T15:43:30.195460+0000","last_fullsized":"2023-05-22T15:43:30.195460+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:43:28.734545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:43:28.734545+0000","last_clean_scrub_stamp":"2023-05-22T15:43:28.734545+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:20:05.437665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":566,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1f","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-05-22T15:43:30.186706+0000","last_change":"2023-05-22T15:43:30.185658+0000","last_active":"2023-05-22T15:43:30.186706+0000","last_peered":"2023-05-22T15:43:30.186706+0000","last_clean":"2023-05-22T15:43:30.186706+0000","last_became_active":"2023-05-22T15:43:30.185349+0000","last_became_peered":"2023-05-22T15:43:30.185349+0000","last_unstale":"2023-05-22T15:43:30.186706+0000","last_undegraded":"2023-05-22T15:43:30.186706+0000","last_fullsized":"2023-05-22T15:43:30.186706+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_clean_scrub_stamp":"2023-05-22T15:43:29.738376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:09:50.754411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.1e","version":"117'1","reported_seq":11,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-05-22T15:43:30.202739+0000","last_change":"2023-05-22T15:43:29.803088+0000","last_active":"2023-05-22T15:43:30.202739+0000","last_peered":"2023-05-22T15:43:30.202739+0000","last_clean":"2023-05-22T15:43:30.202739+0000","last_became_active":"2023-05-22T15:43:29.802113+0000","last_became_peered":"2023-05-22T15:43:29.802113+0000","last_unstale":"2023-05-22T15:43:30.202739+0000","last_undegraded":"2023-05-22T15:43:30.202739+0000","last_fullsized":"2023-05-22T15:43:30.202739+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:43:28.734545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:43:28.734545+0000","last_clean_scrub_stamp":"2023-05-22T15:43:28.734545+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:12:32.488160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.1c","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-05-22T15:43:30.211272+0000","last_change":"2023-05-22T15:43:30.209315+0000","last_active":"2023-05-22T15:43:30.211272+0000","last_peered":"2023-05-22T15:43:30.211272+0000","last_clean":"2023-05-22T15:43:30.211272+0000","last_became_active":"2023-05-22T15:43:30.209018+0000","last_became_peered":"2023-05-22T15:43:30.209018+0000","last_unstale":"2023-05-22T15:43:30.211272+0000","last_undegraded":"2023-05-22T15:43:30.211272+0000","last_fullsized":"2023-05-22T15:43:30.211272+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_clean_scrub_stamp":"2023-05-22T15:43:29.738376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:12:33.491991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.1d","version":"117'2","reported_seq":12,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-05-22T15:43:30.170842+0000","last_change":"2023-05-22T15:43:29.778644+0000","last_active":"2023-05-22T15:43:30.170842+0000","last_peered":"2023-05-22T15:43:30.170842+0000","last_clean":"2023-05-22T15:43:30.170842+0000","last_became_active":"2023-05-22T15:43:29.778325+0000","last_became_peered":"2023-05-22T15:43:29.778325+0000","last_unstale":"2023-05-22T15:43:30.170842+0000","last_undegraded":"2023-05-22T15:43:30.170842+0000","last_fullsized":"2023-05-22T15:43:30.170842+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:43:28.734545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:43:28.734545+0000","last_clean_scrub_stamp":"2023-05-22T15:43:28.734545+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:18:00.068884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1d","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-05-22T15:43:30.223688+0000","last_change":"2023-05-22T15:43:30.205970+0000","last_active":"2023-05-22T15:43:30.223688+0000","last_peered":"2023-05-22T15:43:30.223688+0000","last_clean":"2023-05-22T15:43:30.223688+0000","last_became_active":"2023-05-22T15:43:30.205168+0000","last_became_peered":"2023-05-22T15:43:30.205168+0000","last_unstale":"2023-05-22T15:43:30.223688+0000","last_undegraded":"2023-05-22T15:43:30.223688+0000","last_fullsized":"2023-05-22T15:43:30.223688+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_clean_scrub_stamp":"2023-05-22T15:43:29.738376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:20:06.684861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.1c","version":"117'2","reported_seq":12,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-05-22T15:43:30.217505+0000","last_change":"2023-05-22T15:43:29.807740+0000","last_active":"2023-05-22T15:43:30.217505+0000","last_peered":"2023-05-22T15:43:30.217505+0000","last_clean":"2023-05-22T15:43:30.217505+0000","last_became_active":"2023-05-22T15:43:29.807280+0000","last_became_peered":"2023-05-22T15:43:29.807280+0000","last_unstale":"2023-05-22T15:43:30.217505+0000","last_undegraded":"2023-05-22T15:43:30.217505+0000","last_fullsized":"2023-05-22T15:43:30.217505+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:43:28.734545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:43:28.734545+0000","last_clean_scrub_stamp":"2023-05-22T15:43:28.734545+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:09:49.750580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1a","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-05-22T15:43:30.215960+0000","last_change":"2023-05-22T15:43:30.211412+0000","last_active":"2023-05-22T15:43:30.215960+0000","last_peered":"2023-05-22T15:43:30.215960+0000","last_clean":"2023-05-22T15:43:30.215960+0000","last_became_active":"2023-05-22T15:43:30.210786+0000","last_became_peered":"2023-05-22T15:43:30.210786+0000","last_unstale":"2023-05-22T15:43:30.215960+0000","last_undegraded":"2023-05-22T15:43:30.215960+0000","last_fullsized":"2023-05-22T15:43:30.215960+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_clean_scrub_stamp":"2023-05-22T15:43:29.738376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T16:59:50.668128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.1b","version":"117'1","reported_seq":11,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-05-22T15:43:30.197240+0000","last_change":"2023-05-22T15:43:29.789519+0000","last_active":"2023-05-22T15:43:30.197240+0000","last_peered":"2023-05-22T15:43:30.197240+0000","last_clean":"2023-05-22T15:43:30.197240+0000","last_became_active":"2023-05-22T15:43:29.789209+0000","last_became_peered":"2023-05-22T15:43:29.789209+0000","last_unstale":"2023-05-22T15:43:30.197240+0000","last_undegraded":"2023-05-22T15:43:30.197240+0000","last_fullsized":"2023-05-22T15:43:30.197240+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:43:28.734545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:43:28.734545+0000","last_clean_scrub_stamp":"2023-05-22T15:43:28.734545+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:45:12.725224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1b","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-05-22T15:43:30.206262+0000","last_change":"2023-05-22T15:43:30.199558+0000","last_active":"2023-05-22T15:43:30.206262+0000","last_peered":"2023-05-22T15:43:30.206262+0000","last_clean":"2023-05-22T15:43:30.206262+0000","last_became_active":"2023-05-22T15:43:30.197829+0000","last_became_peered":"2023-05-22T15:43:30.197829+0000","last_unstale":"2023-05-22T15:43:30.206262+0000","last_undegraded":"2023-05-22T15:43:30.206262+0000","last_fullsized":"2023-05-22T15:43:30.206262+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_clean_scrub_stamp":"2023-05-22T15:43:29.738376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T03:40:58.697755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.1a","version":"117'1","reported_seq":11,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-05-22T15:43:30.165399+0000","last_change":"2023-05-22T15:43:29.786409+0000","last_active":"2023-05-22T15:43:30.165399+0000","last_peered":"2023-05-22T15:43:30.165399+0000","last_clean":"2023-05-22T15:43:30.165399+0000","last_became_active":"2023-05-22T15:43:29.786082+0000","last_became_peered":"2023-05-22T15:43:29.786082+0000","last_unstale":"2023-05-22T15:43:30.165399+0000","last_undegraded":"2023-05-22T15:43:30.165399+0000","last_fullsized":"2023-05-22T15:43:30.165399+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:43:28.734545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:43:28.734545+0000","last_clean_scrub_stamp":"2023-05-22T15:43:28.734545+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:33:59.380284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.18","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-05-22T15:43:30.172181+0000","last_change":"2023-05-22T15:43:30.169856+0000","last_active":"2023-05-22T15:43:30.172181+0000","last_peered":"2023-05-22T15:43:30.172181+0000","last_clean":"2023-05-22T15:43:30.172181+0000","last_became_active":"2023-05-22T15:43:30.169581+0000","last_became_peered":"2023-05-22T15:43:30.169581+0000","last_unstale":"2023-05-22T15:43:30.172181+0000","last_undegraded":"2023-05-22T15:43:30.172181+0000","last_fullsized":"2023-05-22T15:43:30.172181+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_clean_scrub_stamp":"2023-05-22T15:43:29.738376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:36:26.720817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.19","version":"0'0","reported_seq":10,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-05-22T15:43:30.171375+0000","last_change":"2023-05-22T15:43:29.790291+0000","last_active":"2023-05-22T15:43:30.171375+0000","last_peered":"2023-05-22T15:43:30.171375+0000","last_clean":"2023-05-22T15:43:30.171375+0000","last_became_active":"2023-05-22T15:43:29.789473+0000","last_became_peered":"2023-05-22T15:43:29.789473+0000","last_unstale":"2023-05-22T15:43:30.171375+0000","last_undegraded":"2023-05-22T15:43:30.171375+0000","last_fullsized":"2023-05-22T15:43:30.171375+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:43:28.734545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:43:28.734545+0000","last_clean_scrub_stamp":"2023-05-22T15:43:28.734545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T03:33:13.047480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.19","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-05-22T15:43:30.223768+0000","last_change":"2023-05-22T15:43:30.206409+0000","last_active":"2023-05-22T15:43:30.223768+0000","last_peered":"2023-05-22T15:43:30.223768+0000","last_clean":"2023-05-22T15:43:30.223768+0000","last_became_active":"2023-05-22T15:43:30.205299+0000","last_became_peered":"2023-05-22T15:43:30.205299+0000","last_unstale":"2023-05-22T15:43:30.223768+0000","last_undegraded":"2023-05-22T15:43:30.223768+0000","last_fullsized":"2023-05-22T15:43:30.223768+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_clean_scrub_stamp":"2023-05-22T15:43:29.738376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:18:19.994700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.18","version":"0'0","reported_seq":10,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-05-22T15:43:30.156027+0000","last_change":"2023-05-22T15:43:29.776421+0000","last_active":"2023-05-22T15:43:30.156027+0000","last_peered":"2023-05-22T15:43:30.156027+0000","last_clean":"2023-05-22T15:43:30.156027+0000","last_became_active":"2023-05-22T15:43:29.776114+0000","last_became_peered":"2023-05-22T15:43:29.776114+0000","last_unstale":"2023-05-22T15:43:30.156027+0000","last_undegraded":"2023-05-22T15:43:30.156027+0000","last_fullsized":"2023-05-22T15:43:30.156027+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:43:28.734545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:43:28.734545+0000","last_clean_scrub_stamp":"2023-05-22T15:43:28.734545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:29:55.479369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.16","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-05-22T15:43:30.205067+0000","last_change":"2023-05-22T15:43:30.204449+0000","last_active":"2023-05-22T15:43:30.205067+0000","last_peered":"2023-05-22T15:43:30.205067+0000","last_clean":"2023-05-22T15:43:30.205067+0000","last_became_active":"2023-05-22T15:43:30.204175+0000","last_became_peered":"2023-05-22T15:43:30.204175+0000","last_unstale":"2023-05-22T15:43:30.205067+0000","last_undegraded":"2023-05-22T15:43:30.205067+0000","last_fullsized":"2023-05-22T15:43:30.205067+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_clean_scrub_stamp":"2023-05-22T15:43:29.738376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:38:19.788691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.17","version":"0'0","reported_seq":10,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-05-22T15:43:30.158574+0000","last_change":"2023-05-22T15:43:29.791430+0000","last_active":"2023-05-22T15:43:30.158574+0000","last_peered":"2023-05-22T15:43:30.158574+0000","last_clean":"2023-05-22T15:43:30.158574+0000","last_became_active":"2023-05-22T15:43:29.791156+0000","last_became_peered":"2023-05-22T15:43:29.791156+0000","last_unstale":"2023-05-22T15:43:30.158574+0000","last_undegraded":"2023-05-22T15:43:30.158574+0000","last_fullsized":"2023-05-22T15:43:30.158574+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:43:28.734545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:43:28.734545+0000","last_clean_scrub_stamp":"2023-05-22T15:43:28.734545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:13:19.208630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.17","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-05-22T15:43:30.223768+0000","last_change":"2023-05-22T15:43:30.206797+0000","last_active":"2023-05-22T15:43:30.223768+0000","last_peered":"2023-05-22T15:43:30.223768+0000","last_clean":"2023-05-22T15:43:30.223768+0000","last_became_active":"2023-05-22T15:43:30.206222+0000","last_became_peered":"2023-05-22T15:43:30.206222+0000","last_unstale":"2023-05-22T15:43:30.223768+0000","last_undegraded":"2023-05-22T15:43:30.223768+0000","last_fullsized":"2023-05-22T15:43:30.223768+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_clean_scrub_stamp":"2023-05-22T15:43:29.738376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:20:06.441496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.16","version":"0'0","reported_seq":10,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-05-22T15:43:30.148238+0000","last_change":"2023-05-22T15:43:29.785747+0000","last_active":"2023-05-22T15:43:30.148238+0000","last_peered":"2023-05-22T15:43:30.148238+0000","last_clean":"2023-05-22T15:43:30.148238+0000","last_became_active":"2023-05-22T15:43:29.785179+0000","last_became_peered":"2023-05-22T15:43:29.785179+0000","last_unstale":"2023-05-22T15:43:30.148238+0000","last_undegraded":"2023-05-22T15:43:30.148238+0000","last_fullsized":"2023-05-22T15:43:30.148238+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:43:28.734545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:43:28.734545+0000","last_clean_scrub_stamp":"2023-05-22T15:43:28.734545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T15:59:50.976935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.14","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-05-22T15:43:30.196588+0000","last_change":"2023-05-22T15:43:30.169946+0000","last_active":"2023-05-22T15:43:30.196588+0000","last_peered":"2023-05-22T15:43:30.196588+0000","last_clean":"2023-05-22T15:43:30.196588+0000","last_became_active":"2023-05-22T15:43:30.169159+0000","last_became_peered":"2023-05-22T15:43:30.169159+0000","last_unstale":"2023-05-22T15:43:30.196588+0000","last_undegraded":"2023-05-22T15:43:30.196588+0000","last_fullsized":"2023-05-22T15:43:30.196588+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_clean_scrub_stamp":"2023-05-22T15:43:29.738376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:30:29.500628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.15","version":"117'3","reported_seq":13,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-05-22T15:43:30.220527+0000","last_change":"2023-05-22T15:43:29.810062+0000","last_active":"2023-05-22T15:43:30.220527+0000","last_peered":"2023-05-22T15:43:30.220527+0000","last_clean":"2023-05-22T15:43:30.220527+0000","last_became_active":"2023-05-22T15:43:29.809287+0000","last_became_peered":"2023-05-22T15:43:29.809287+0000","last_unstale":"2023-05-22T15:43:30.220527+0000","last_undegraded":"2023-05-22T15:43:30.220527+0000","last_fullsized":"2023-05-22T15:43:30.220527+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:43:28.734545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:43:28.734545+0000","last_clean_scrub_stamp":"2023-05-22T15:43:28.734545+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:44:15.881912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.15","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-05-22T15:43:30.208267+0000","last_change":"2023-05-22T15:43:30.191994+0000","last_active":"2023-05-22T15:43:30.208267+0000","last_peered":"2023-05-22T15:43:30.208267+0000","last_clean":"2023-05-22T15:43:30.208267+0000","last_became_active":"2023-05-22T15:43:30.191302+0000","last_became_peered":"2023-05-22T15:43:30.191302+0000","last_unstale":"2023-05-22T15:43:30.208267+0000","last_undegraded":"2023-05-22T15:43:30.208267+0000","last_fullsized":"2023-05-22T15:43:30.208267+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_clean_scrub_stamp":"2023-05-22T15:43:29.738376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:18:01.072715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.14","version":"117'6","reported_seq":16,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-05-22T15:43:30.225024+0000","last_change":"2023-05-22T15:43:29.807258+0000","last_active":"2023-05-22T15:43:30.225024+0000","last_peered":"2023-05-22T15:43:30.225024+0000","last_clean":"2023-05-22T15:43:30.225024+0000","last_became_active":"2023-05-22T15:43:29.806533+0000","last_became_peered":"2023-05-22T15:43:29.806533+0000","last_unstale":"2023-05-22T15:43:30.225024+0000","last_undegraded":"2023-05-22T15:43:30.225024+0000","last_fullsized":"2023-05-22T15:43:30.225024+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:43:28.734545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:43:28.734545+0000","last_clean_scrub_stamp":"2023-05-22T15:43:28.734545+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T03:40:57.693924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1548,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.12","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-05-22T15:43:30.206352+0000","last_change":"2023-05-22T15:43:30.205777+0000","last_active":"2023-05-22T15:43:30.206352+0000","last_peered":"2023-05-22T15:43:30.206352+0000","last_clean":"2023-05-22T15:43:30.206352+0000","last_became_active":"2023-05-22T15:43:30.205565+0000","last_became_peered":"2023-05-22T15:43:30.205565+0000","last_unstale":"2023-05-22T15:43:30.206352+0000","last_undegraded":"2023-05-22T15:43:30.206352+0000","last_fullsized":"2023-05-22T15:43:30.206352+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_clean_scrub_stamp":"2023-05-22T15:43:29.738376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T16:53:26.129417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.13","version":"117'2","reported_seq":12,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-05-22T15:43:30.206194+0000","last_change":"2023-05-22T15:43:29.806812+0000","last_active":"2023-05-22T15:43:30.206194+0000","last_peered":"2023-05-22T15:43:30.206194+0000","last_clean":"2023-05-22T15:43:30.206194+0000","last_became_active":"2023-05-22T15:43:29.806402+0000","last_became_peered":"2023-05-22T15:43:29.806402+0000","last_unstale":"2023-05-22T15:43:30.206194+0000","last_undegraded":"2023-05-22T15:43:30.206194+0000","last_fullsized":"2023-05-22T15:43:30.206194+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:43:28.734545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:43:28.734545+0000","last_clean_scrub_stamp":"2023-05-22T15:43:28.734545+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T18:18:31.388215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.13","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-05-22T15:43:30.173056+0000","last_change":"2023-05-22T15:43:30.168305+0000","last_active":"2023-05-22T15:43:30.173056+0000","last_peered":"2023-05-22T15:43:30.173056+0000","last_clean":"2023-05-22T15:43:30.173056+0000","last_became_active":"2023-05-22T15:43:30.167986+0000","last_became_peered":"2023-05-22T15:43:30.167986+0000","last_unstale":"2023-05-22T15:43:30.173056+0000","last_undegraded":"2023-05-22T15:43:30.173056+0000","last_fullsized":"2023-05-22T15:43:30.173056+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_clean_scrub_stamp":"2023-05-22T15:43:29.738376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T03:33:14.051311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.12","version":"0'0","reported_seq":10,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-05-22T15:43:30.157575+0000","last_change":"2023-05-22T15:43:29.776238+0000","last_active":"2023-05-22T15:43:30.157575+0000","last_peered":"2023-05-22T15:43:30.157575+0000","last_clean":"2023-05-22T15:43:30.157575+0000","last_became_active":"2023-05-22T15:43:29.775974+0000","last_became_peered":"2023-05-22T15:43:29.775974+0000","last_unstale":"2023-05-22T15:43:30.157575+0000","last_undegraded":"2023-05-22T15:43:30.157575+0000","last_fullsized":"2023-05-22T15:43:30.157575+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:43:28.734545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:43:28.734545+0000","last_clean_scrub_stamp":"2023-05-22T15:43:28.734545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:45:03.727507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.10","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-05-22T15:43:30.223699+0000","last_change":"2023-05-22T15:43:30.206181+0000","last_active":"2023-05-22T15:43:30.223699+0000","last_peered":"2023-05-22T15:43:30.223699+0000","last_clean":"2023-05-22T15:43:30.223699+0000","last_became_active":"2023-05-22T15:43:30.205355+0000","last_became_peered":"2023-05-22T15:43:30.205355+0000","last_unstale":"2023-05-22T15:43:30.223699+0000","last_undegraded":"2023-05-22T15:43:30.223699+0000","last_fullsized":"2023-05-22T15:43:30.223699+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_clean_scrub_stamp":"2023-05-22T15:43:29.738376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:00:19.974693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.11","version":"0'0","reported_seq":10,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-05-22T15:43:30.181766+0000","last_change":"2023-05-22T15:43:29.800529+0000","last_active":"2023-05-22T15:43:30.181766+0000","last_peered":"2023-05-22T15:43:30.181766+0000","last_clean":"2023-05-22T15:43:30.181766+0000","last_became_active":"2023-05-22T15:43:29.799819+0000","last_became_peered":"2023-05-22T15:43:29.799819+0000","last_unstale":"2023-05-22T15:43:30.181766+0000","last_undegraded":"2023-05-22T15:43:30.181766+0000","last_fullsized":"2023-05-22T15:43:30.181766+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:43:28.734545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:43:28.734545+0000","last_clean_scrub_stamp":"2023-05-22T15:43:28.734545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:19:18.535583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.11","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-05-22T15:43:30.211039+0000","last_change":"2023-05-22T15:43:30.209771+0000","last_active":"2023-05-22T15:43:30.211039+0000","last_peered":"2023-05-22T15:43:30.211039+0000","last_clean":"2023-05-22T15:43:30.211039+0000","last_became_active":"2023-05-22T15:43:30.209482+0000","last_became_peered":"2023-05-22T15:43:30.209482+0000","last_unstale":"2023-05-22T15:43:30.211039+0000","last_undegraded":"2023-05-22T15:43:30.211039+0000","last_fullsized":"2023-05-22T15:43:30.211039+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_clean_scrub_stamp":"2023-05-22T15:43:29.738376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:44:35.350206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.10","version":"117'8","reported_seq":18,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-05-22T15:43:30.203024+0000","last_change":"2023-05-22T15:43:29.798352+0000","last_active":"2023-05-22T15:43:30.203024+0000","last_peered":"2023-05-22T15:43:30.203024+0000","last_clean":"2023-05-22T15:43:30.203024+0000","last_became_active":"2023-05-22T15:43:29.798066+0000","last_became_peered":"2023-05-22T15:43:29.798066+0000","last_unstale":"2023-05-22T15:43:30.203024+0000","last_undegraded":"2023-05-22T15:43:30.203024+0000","last_fullsized":"2023-05-22T15:43:30.203024+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:43:28.734545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:43:28.734545+0000","last_clean_scrub_stamp":"2023-05-22T15:43:28.734545+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:27:48.345151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.6","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-05-22T15:43:30.164679+0000","last_change":"2023-05-22T15:43:30.163320+0000","last_active":"2023-05-22T15:43:30.164679+0000","last_peered":"2023-05-22T15:43:30.164679+0000","last_clean":"2023-05-22T15:43:30.164679+0000","last_became_active":"2023-05-22T15:43:30.163046+0000","last_became_peered":"2023-05-22T15:43:30.163046+0000","last_unstale":"2023-05-22T15:43:30.164679+0000","last_undegraded":"2023-05-22T15:43:30.164679+0000","last_fullsized":"2023-05-22T15:43:30.164679+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_clean_scrub_stamp":"2023-05-22T15:43:29.738376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:50:18.281158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.7","version":"117'2","reported_seq":12,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-05-22T15:43:30.211222+0000","last_change":"2023-05-22T15:43:29.808187+0000","last_active":"2023-05-22T15:43:30.211222+0000","last_peered":"2023-05-22T15:43:30.211222+0000","last_clean":"2023-05-22T15:43:30.211222+0000","last_became_active":"2023-05-22T15:43:29.807549+0000","last_became_peered":"2023-05-22T15:43:29.807549+0000","last_unstale":"2023-05-22T15:43:30.211222+0000","last_undegraded":"2023-05-22T15:43:30.211222+0000","last_fullsized":"2023-05-22T15:43:30.211222+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:43:28.734545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:43:28.734545+0000","last_clean_scrub_stamp":"2023-05-22T15:43:28.734545+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:08:05.729807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.7","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-05-22T15:43:30.173040+0000","last_change":"2023-05-22T15:43:30.168243+0000","last_active":"2023-05-22T15:43:30.173040+0000","last_peered":"2023-05-22T15:43:30.173040+0000","last_clean":"2023-05-22T15:43:30.173040+0000","last_became_active":"2023-05-22T15:43:30.167889+0000","last_became_peered":"2023-05-22T15:43:30.167889+0000","last_unstale":"2023-05-22T15:43:30.173040+0000","last_undegraded":"2023-05-22T15:43:30.173040+0000","last_fullsized":"2023-05-22T15:43:30.173040+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_clean_scrub_stamp":"2023-05-22T15:43:29.738376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:59:49.774332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.6","version":"117'1","reported_seq":11,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-05-22T15:43:30.597094+0000","last_change":"2023-05-22T15:43:29.784036+0000","last_active":"2023-05-22T15:43:30.597094+0000","last_peered":"2023-05-22T15:43:30.597094+0000","last_clean":"2023-05-22T15:43:30.597094+0000","last_became_active":"2023-05-22T15:43:29.783744+0000","last_became_peered":"2023-05-22T15:43:29.783744+0000","last_unstale":"2023-05-22T15:43:30.597094+0000","last_undegraded":"2023-05-22T15:43:30.597094+0000","last_fullsized":"2023-05-22T15:43:30.597094+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:43:28.734545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:43:28.734545+0000","last_clean_scrub_stamp":"2023-05-22T15:43:28.734545+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:04:27.414328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.4","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-05-22T15:43:30.198927+0000","last_change":"2023-05-22T15:43:30.197767+0000","last_active":"2023-05-22T15:43:30.198927+0000","last_peered":"2023-05-22T15:43:30.198927+0000","last_clean":"2023-05-22T15:43:30.198927+0000","last_became_active":"2023-05-22T15:43:30.197470+0000","last_became_peered":"2023-05-22T15:43:30.197470+0000","last_unstale":"2023-05-22T15:43:30.198927+0000","last_undegraded":"2023-05-22T15:43:30.198927+0000","last_fullsized":"2023-05-22T15:43:30.198927+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_clean_scrub_stamp":"2023-05-22T15:43:29.738376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:05:33.735688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.5","version":"0'0","reported_seq":10,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-05-22T15:43:30.147025+0000","last_change":"2023-05-22T15:43:29.806276+0000","last_active":"2023-05-22T15:43:30.147025+0000","last_peered":"2023-05-22T15:43:30.147025+0000","last_clean":"2023-05-22T15:43:30.147025+0000","last_became_active":"2023-05-22T15:43:29.805860+0000","last_became_peered":"2023-05-22T15:43:29.805860+0000","last_unstale":"2023-05-22T15:43:30.147025+0000","last_undegraded":"2023-05-22T15:43:30.147025+0000","last_fullsized":"2023-05-22T15:43:30.147025+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:43:28.734545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:43:28.734545+0000","last_clean_scrub_stamp":"2023-05-22T15:43:28.734545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:38:14.545852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.5","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-05-22T15:43:30.215694+0000","last_change":"2023-05-22T15:43:30.199769+0000","last_active":"2023-05-22T15:43:30.215694+0000","last_peered":"2023-05-22T15:43:30.215694+0000","last_clean":"2023-05-22T15:43:30.215694+0000","last_became_active":"2023-05-22T15:43:30.199109+0000","last_became_peered":"2023-05-22T15:43:30.199109+0000","last_unstale":"2023-05-22T15:43:30.215694+0000","last_undegraded":"2023-05-22T15:43:30.215694+0000","last_fullsized":"2023-05-22T15:43:30.215694+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_clean_scrub_stamp":"2023-05-22T15:43:29.738376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:47:24.012002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.4","version":"0'0","reported_seq":10,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-05-22T15:43:30.159908+0000","last_change":"2023-05-22T15:43:29.790575+0000","last_active":"2023-05-22T15:43:30.159908+0000","last_peered":"2023-05-22T15:43:30.159908+0000","last_clean":"2023-05-22T15:43:30.159908+0000","last_became_active":"2023-05-22T15:43:29.789764+0000","last_became_peered":"2023-05-22T15:43:29.789764+0000","last_unstale":"2023-05-22T15:43:30.159908+0000","last_undegraded":"2023-05-22T15:43:30.159908+0000","last_fullsized":"2023-05-22T15:43:30.159908+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:43:28.734545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:43:28.734545+0000","last_clean_scrub_stamp":"2023-05-22T15:43:28.734545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:05:01.015984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.2","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-05-22T15:43:30.211007+0000","last_change":"2023-05-22T15:43:30.209732+0000","last_active":"2023-05-22T15:43:30.211007+0000","last_peered":"2023-05-22T15:43:30.211007+0000","last_clean":"2023-05-22T15:43:30.211007+0000","last_became_active":"2023-05-22T15:43:30.209409+0000","last_became_peered":"2023-05-22T15:43:30.209409+0000","last_unstale":"2023-05-22T15:43:30.211007+0000","last_undegraded":"2023-05-22T15:43:30.211007+0000","last_fullsized":"2023-05-22T15:43:30.211007+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_clean_scrub_stamp":"2023-05-22T15:43:29.738376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:08:06.733638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.3","version":"0'0","reported_seq":10,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-05-22T15:43:30.145467+0000","last_change":"2023-05-22T15:43:29.781583+0000","last_active":"2023-05-22T15:43:30.145467+0000","last_peered":"2023-05-22T15:43:30.145467+0000","last_clean":"2023-05-22T15:43:30.145467+0000","last_became_active":"2023-05-22T15:43:29.781276+0000","last_became_peered":"2023-05-22T15:43:29.781276+0000","last_unstale":"2023-05-22T15:43:30.145467+0000","last_undegraded":"2023-05-22T15:43:30.145467+0000","last_fullsized":"2023-05-22T15:43:30.145467+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:43:28.734545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:43:28.734545+0000","last_clean_scrub_stamp":"2023-05-22T15:43:28.734545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:20:56.145983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.3","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-05-22T15:43:30.223804+0000","last_change":"2023-05-22T15:43:30.208056+0000","last_active":"2023-05-22T15:43:30.223804+0000","last_peered":"2023-05-22T15:43:30.223804+0000","last_clean":"2023-05-22T15:43:30.223804+0000","last_became_active":"2023-05-22T15:43:30.207454+0000","last_became_peered":"2023-05-22T15:43:30.207454+0000","last_unstale":"2023-05-22T15:43:30.223804+0000","last_undegraded":"2023-05-22T15:43:30.223804+0000","last_fullsized":"2023-05-22T15:43:30.223804+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_clean_scrub_stamp":"2023-05-22T15:43:29.738376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:52:20.517587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2","version":"0'0","reported_seq":10,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-05-22T15:43:30.156924+0000","last_change":"2023-05-22T15:43:29.779390+0000","last_active":"2023-05-22T15:43:30.156924+0000","last_peered":"2023-05-22T15:43:30.156924+0000","last_clean":"2023-05-22T15:43:30.156924+0000","last_became_active":"2023-05-22T15:43:29.778569+0000","last_became_peered":"2023-05-22T15:43:29.778569+0000","last_unstale":"2023-05-22T15:43:30.156924+0000","last_undegraded":"2023-05-22T15:43:30.156924+0000","last_fullsized":"2023-05-22T15:43:30.156924+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:43:28.734545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:43:28.734545+0000","last_clean_scrub_stamp":"2023-05-22T15:43:28.734545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:59:48.770501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.0","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-05-22T15:43:30.211054+0000","last_change":"2023-05-22T15:43:30.210031+0000","last_active":"2023-05-22T15:43:30.211054+0000","last_peered":"2023-05-22T15:43:30.211054+0000","last_clean":"2023-05-22T15:43:30.211054+0000","last_became_active":"2023-05-22T15:43:30.209727+0000","last_became_peered":"2023-05-22T15:43:30.209727+0000","last_unstale":"2023-05-22T15:43:30.211054+0000","last_undegraded":"2023-05-22T15:43:30.211054+0000","last_fullsized":"2023-05-22T15:43:30.211054+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_clean_scrub_stamp":"2023-05-22T15:43:29.738376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:05:33.735688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1","version":"0'0","reported_seq":10,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-05-22T15:43:30.148534+0000","last_change":"2023-05-22T15:43:29.776927+0000","last_active":"2023-05-22T15:43:30.148534+0000","last_peered":"2023-05-22T15:43:30.148534+0000","last_clean":"2023-05-22T15:43:30.148534+0000","last_became_active":"2023-05-22T15:43:29.776007+0000","last_became_peered":"2023-05-22T15:43:29.776007+0000","last_unstale":"2023-05-22T15:43:30.148534+0000","last_undegraded":"2023-05-22T15:43:30.148534+0000","last_fullsized":"2023-05-22T15:43:30.148534+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:43:28.734545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:43:28.734545+0000","last_clean_scrub_stamp":"2023-05-22T15:43:28.734545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:47:23.008171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.1","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-05-22T15:43:30.211077+0000","last_change":"2023-05-22T15:43:30.178779+0000","last_active":"2023-05-22T15:43:30.211077+0000","last_peered":"2023-05-22T15:43:30.211077+0000","last_clean":"2023-05-22T15:43:30.211077+0000","last_became_active":"2023-05-22T15:43:30.177907+0000","last_became_peered":"2023-05-22T15:43:30.177907+0000","last_unstale":"2023-05-22T15:43:30.211077+0000","last_undegraded":"2023-05-22T15:43:30.211077+0000","last_fullsized":"2023-05-22T15:43:30.211077+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_clean_scrub_stamp":"2023-05-22T15:43:29.738376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T03:40:58.697755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.0","version":"0'0","reported_seq":10,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-05-22T15:43:30.150999+0000","last_change":"2023-05-22T15:43:29.786297+0000","last_active":"2023-05-22T15:43:30.150999+0000","last_peered":"2023-05-22T15:43:30.150999+0000","last_clean":"2023-05-22T15:43:30.150999+0000","last_became_active":"2023-05-22T15:43:29.786016+0000","last_became_peered":"2023-05-22T15:43:29.786016+0000","last_unstale":"2023-05-22T15:43:30.150999+0000","last_undegraded":"2023-05-22T15:43:30.150999+0000","last_fullsized":"2023-05-22T15:43:30.150999+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:43:28.734545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:43:28.734545+0000","last_clean_scrub_stamp":"2023-05-22T15:43:28.734545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T16:05:36.115819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.e","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-05-22T15:43:30.174625+0000","last_change":"2023-05-22T15:43:30.168122+0000","last_active":"2023-05-22T15:43:30.174625+0000","last_peered":"2023-05-22T15:43:30.174625+0000","last_clean":"2023-05-22T15:43:30.174625+0000","last_became_active":"2023-05-22T15:43:30.166491+0000","last_became_peered":"2023-05-22T15:43:30.166491+0000","last_unstale":"2023-05-22T15:43:30.174625+0000","last_undegraded":"2023-05-22T15:43:30.174625+0000","last_fullsized":"2023-05-22T15:43:30.174625+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_clean_scrub_stamp":"2023-05-22T15:43:29.738376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:45:35.107137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.f","version":"117'1","reported_seq":11,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-05-22T15:43:30.202749+0000","last_change":"2023-05-22T15:43:29.797109+0000","last_active":"2023-05-22T15:43:30.202749+0000","last_peered":"2023-05-22T15:43:30.202749+0000","last_clean":"2023-05-22T15:43:30.202749+0000","last_became_active":"2023-05-22T15:43:29.796390+0000","last_became_peered":"2023-05-22T15:43:29.796390+0000","last_unstale":"2023-05-22T15:43:30.202749+0000","last_undegraded":"2023-05-22T15:43:30.202749+0000","last_fullsized":"2023-05-22T15:43:30.202749+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:43:28.734545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:43:28.734545+0000","last_clean_scrub_stamp":"2023-05-22T15:43:28.734545+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T18:41:58.613554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"106'205","reported_seq":570,"reported_epoch":116,"state":"active+clean","last_fresh":"2023-05-22T15:43:29.786247+0000","last_change":"2023-05-22T15:26:39.318204+0000","last_active":"2023-05-22T15:43:29.786247+0000","last_peered":"2023-05-22T15:43:29.786247+0000","last_clean":"2023-05-22T15:43:29.786247+0000","last_became_active":"2023-05-22T15:26:39.315370+0000","last_became_peered":"2023-05-22T15:26:39.315370+0000","last_unstale":"2023-05-22T15:43:29.786247+0000","last_undegraded":"2023-05-22T15:43:29.786247+0000","last_fullsized":"2023-05-22T15:43:29.786247+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:38.020076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:38.020076+0000","last_clean_scrub_stamp":"2023-05-22T15:26:38.020076+0000","objects_scrubbed":0,"log_size":205,"log_dups_size":0,"ondisk_log_size":205,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:06:39.207543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":884,"num_read_kb":1748,"num_write":319,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":111,"reported_epoch":116,"state":"active+clean","last_fresh":"2023-05-22T15:43:29.778340+0000","last_change":"2023-05-22T15:26:44.971951+0000","last_active":"2023-05-22T15:43:29.778340+0000","last_peered":"2023-05-22T15:43:29.778340+0000","last_clean":"2023-05-22T15:43:29.778340+0000","last_became_active":"2023-05-22T15:26:44.970872+0000","last_became_peered":"2023-05-22T15:26:44.970872+0000","last_unstale":"2023-05-22T15:43:29.778340+0000","last_undegraded":"2023-05-22T15:43:29.778340+0000","last_fullsized":"2023-05-22T15:43:29.778340+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:55:34.752949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.d","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-05-22T15:43:30.222151+0000","last_change":"2023-05-22T15:43:30.220279+0000","last_active":"2023-05-22T15:43:30.222151+0000","last_peered":"2023-05-22T15:43:30.222151+0000","last_clean":"2023-05-22T15:43:30.222151+0000","last_became_active":"2023-05-22T15:43:30.219191+0000","last_became_peered":"2023-05-22T15:43:30.219191+0000","last_unstale":"2023-05-22T15:43:30.222151+0000","last_undegraded":"2023-05-22T15:43:30.222151+0000","last_fullsized":"2023-05-22T15:43:30.222151+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_clean_scrub_stamp":"2023-05-22T15:43:29.738376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:20:06.441496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.c","version":"0'0","reported_seq":10,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-05-22T15:43:30.145281+0000","last_change":"2023-05-22T15:43:29.807154+0000","last_active":"2023-05-22T15:43:30.145281+0000","last_peered":"2023-05-22T15:43:30.145281+0000","last_clean":"2023-05-22T15:43:30.145281+0000","last_became_active":"2023-05-22T15:43:29.806884+0000","last_became_peered":"2023-05-22T15:43:29.806884+0000","last_unstale":"2023-05-22T15:43:30.145281+0000","last_undegraded":"2023-05-22T15:43:30.145281+0000","last_fullsized":"2023-05-22T15:43:30.145281+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:43:28.734545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:43:28.734545+0000","last_clean_scrub_stamp":"2023-05-22T15:43:28.734545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:20:05.681030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":105,"reported_epoch":116,"state":"active+clean","last_fresh":"2023-05-22T15:43:29.786011+0000","last_change":"2023-05-22T15:26:44.977198+0000","last_active":"2023-05-22T15:43:29.786011+0000","last_peered":"2023-05-22T15:43:29.786011+0000","last_clean":"2023-05-22T15:43:29.786011+0000","last_became_active":"2023-05-22T15:26:44.976445+0000","last_became_peered":"2023-05-22T15:26:44.976445+0000","last_unstale":"2023-05-22T15:43:29.786011+0000","last_undegraded":"2023-05-22T15:43:29.786011+0000","last_fullsized":"2023-05-22T15:43:29.786011+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:12:41.464054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.c","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-05-22T15:43:30.204628+0000","last_change":"2023-05-22T15:43:30.187095+0000","last_active":"2023-05-22T15:43:30.204628+0000","last_peered":"2023-05-22T15:43:30.204628+0000","last_clean":"2023-05-22T15:43:30.204628+0000","last_became_active":"2023-05-22T15:43:30.186352+0000","last_became_peered":"2023-05-22T15:43:30.186352+0000","last_unstale":"2023-05-22T15:43:30.204628+0000","last_undegraded":"2023-05-22T15:43:30.204628+0000","last_fullsized":"2023-05-22T15:43:30.204628+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_clean_scrub_stamp":"2023-05-22T15:43:29.738376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:12:33.491991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.d","version":"117'1","reported_seq":11,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-05-22T15:43:30.198947+0000","last_change":"2023-05-22T15:43:29.794522+0000","last_active":"2023-05-22T15:43:30.198947+0000","last_peered":"2023-05-22T15:43:30.198947+0000","last_clean":"2023-05-22T15:43:30.198947+0000","last_became_active":"2023-05-22T15:43:29.793688+0000","last_became_peered":"2023-05-22T15:43:29.793688+0000","last_unstale":"2023-05-22T15:43:30.198947+0000","last_undegraded":"2023-05-22T15:43:30.198947+0000","last_fullsized":"2023-05-22T15:43:30.198947+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:43:28.734545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:43:28.734545+0000","last_clean_scrub_stamp":"2023-05-22T15:43:28.734545+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:57:33.702470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":105,"reported_epoch":116,"state":"active+clean","last_fresh":"2023-05-22T15:43:29.755329+0000","last_change":"2023-05-22T15:26:44.969469+0000","last_active":"2023-05-22T15:43:29.755329+0000","last_peered":"2023-05-22T15:43:29.755329+0000","last_clean":"2023-05-22T15:43:29.755329+0000","last_became_active":"2023-05-22T15:26:44.969083+0000","last_became_peered":"2023-05-22T15:26:44.969083+0000","last_unstale":"2023-05-22T15:43:29.755329+0000","last_undegraded":"2023-05-22T15:43:29.755329+0000","last_fullsized":"2023-05-22T15:43:29.755329+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:29:23.670067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.f","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-05-22T15:43:30.216502+0000","last_change":"2023-05-22T15:43:30.212809+0000","last_active":"2023-05-22T15:43:30.216502+0000","last_peered":"2023-05-22T15:43:30.216502+0000","last_clean":"2023-05-22T15:43:30.216502+0000","last_became_active":"2023-05-22T15:43:30.212094+0000","last_became_peered":"2023-05-22T15:43:30.212094+0000","last_unstale":"2023-05-22T15:43:30.216502+0000","last_undegraded":"2023-05-22T15:43:30.216502+0000","last_fullsized":"2023-05-22T15:43:30.216502+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_clean_scrub_stamp":"2023-05-22T15:43:29.738376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:00:19.974693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.e","version":"117'2","reported_seq":12,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-05-22T15:43:30.218642+0000","last_change":"2023-05-22T15:43:29.807795+0000","last_active":"2023-05-22T15:43:30.218642+0000","last_peered":"2023-05-22T15:43:30.218642+0000","last_clean":"2023-05-22T15:43:30.218642+0000","last_became_active":"2023-05-22T15:43:29.806892+0000","last_became_peered":"2023-05-22T15:43:29.806892+0000","last_unstale":"2023-05-22T15:43:30.218642+0000","last_undegraded":"2023-05-22T15:43:30.218642+0000","last_fullsized":"2023-05-22T15:43:30.218642+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:43:28.734545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:43:28.734545+0000","last_clean_scrub_stamp":"2023-05-22T15:43:28.734545+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:47:23.008171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":111,"reported_epoch":116,"state":"active+clean","last_fresh":"2023-05-22T15:43:29.778708+0000","last_change":"2023-05-22T15:26:44.976632+0000","last_active":"2023-05-22T15:43:29.778708+0000","last_peered":"2023-05-22T15:43:29.778708+0000","last_clean":"2023-05-22T15:43:29.778708+0000","last_became_active":"2023-05-22T15:26:44.975546+0000","last_became_peered":"2023-05-22T15:26:44.975546+0000","last_unstale":"2023-05-22T15:43:29.778708+0000","last_undegraded":"2023-05-22T15:43:29.778708+0000","last_fullsized":"2023-05-22T15:43:29.778708+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:42:57.973030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.9","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-05-22T15:43:30.211346+0000","last_change":"2023-05-22T15:43:30.198152+0000","last_active":"2023-05-22T15:43:30.211346+0000","last_peered":"2023-05-22T15:43:30.211346+0000","last_clean":"2023-05-22T15:43:30.211346+0000","last_became_active":"2023-05-22T15:43:30.197390+0000","last_became_peered":"2023-05-22T15:43:30.197390+0000","last_unstale":"2023-05-22T15:43:30.211346+0000","last_undegraded":"2023-05-22T15:43:30.211346+0000","last_fullsized":"2023-05-22T15:43:30.211346+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_clean_scrub_stamp":"2023-05-22T15:43:29.738376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:19:19.539413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.8","version":"0'0","reported_seq":10,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-05-22T15:43:30.144067+0000","last_change":"2023-05-22T15:43:29.786288+0000","last_active":"2023-05-22T15:43:30.144067+0000","last_peered":"2023-05-22T15:43:30.144067+0000","last_clean":"2023-05-22T15:43:30.144067+0000","last_became_active":"2023-05-22T15:43:29.786009+0000","last_became_peered":"2023-05-22T15:43:29.786009+0000","last_unstale":"2023-05-22T15:43:30.144067+0000","last_undegraded":"2023-05-22T15:43:30.144067+0000","last_fullsized":"2023-05-22T15:43:30.144067+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:43:28.734545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:43:28.734545+0000","last_clean_scrub_stamp":"2023-05-22T15:43:28.734545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:48:01.712437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":99,"reported_epoch":116,"state":"active+clean","last_fresh":"2023-05-22T15:43:29.765352+0000","last_change":"2023-05-22T15:26:45.091475+0000","last_active":"2023-05-22T15:43:29.765352+0000","last_peered":"2023-05-22T15:43:29.765352+0000","last_clean":"2023-05-22T15:43:29.765352+0000","last_became_active":"2023-05-22T15:26:45.091143+0000","last_became_peered":"2023-05-22T15:26:45.091143+0000","last_unstale":"2023-05-22T15:43:29.765352+0000","last_undegraded":"2023-05-22T15:43:29.765352+0000","last_fullsized":"2023-05-22T15:43:29.765352+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:12:41.464054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.8","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-05-22T15:43:30.174494+0000","last_change":"2023-05-22T15:43:30.166073+0000","last_active":"2023-05-22T15:43:30.174494+0000","last_peered":"2023-05-22T15:43:30.174494+0000","last_clean":"2023-05-22T15:43:30.174494+0000","last_became_active":"2023-05-22T15:43:30.165458+0000","last_became_peered":"2023-05-22T15:43:30.165458+0000","last_unstale":"2023-05-22T15:43:30.174494+0000","last_undegraded":"2023-05-22T15:43:30.174494+0000","last_fullsized":"2023-05-22T15:43:30.174494+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_clean_scrub_stamp":"2023-05-22T15:43:29.738376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T03:17:16.869920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.9","version":"0'0","reported_seq":10,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-05-22T15:43:30.152917+0000","last_change":"2023-05-22T15:43:29.781790+0000","last_active":"2023-05-22T15:43:30.152917+0000","last_peered":"2023-05-22T15:43:30.152917+0000","last_clean":"2023-05-22T15:43:30.152917+0000","last_became_active":"2023-05-22T15:43:29.779832+0000","last_became_peered":"2023-05-22T15:43:29.779832+0000","last_unstale":"2023-05-22T15:43:30.152917+0000","last_undegraded":"2023-05-22T15:43:30.152917+0000","last_fullsized":"2023-05-22T15:43:30.152917+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:43:28.734545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:43:28.734545+0000","last_clean_scrub_stamp":"2023-05-22T15:43:28.734545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:59:48.770501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":104,"reported_epoch":116,"state":"active+clean","last_fresh":"2023-05-22T15:43:29.791838+0000","last_change":"2023-05-22T15:26:45.092367+0000","last_active":"2023-05-22T15:43:29.791838+0000","last_peered":"2023-05-22T15:43:29.791838+0000","last_clean":"2023-05-22T15:43:29.791838+0000","last_became_active":"2023-05-22T15:26:45.091951+0000","last_became_peered":"2023-05-22T15:26:45.091951+0000","last_unstale":"2023-05-22T15:43:29.791838+0000","last_undegraded":"2023-05-22T15:43:29.791838+0000","last_fullsized":"2023-05-22T15:43:29.791838+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:42:57.973030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.b","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-05-22T15:43:30.174523+0000","last_change":"2023-05-22T15:43:30.166496+0000","last_active":"2023-05-22T15:43:30.174523+0000","last_peered":"2023-05-22T15:43:30.174523+0000","last_clean":"2023-05-22T15:43:30.174523+0000","last_became_active":"2023-05-22T15:43:30.166174+0000","last_became_peered":"2023-05-22T15:43:30.166174+0000","last_unstale":"2023-05-22T15:43:30.174523+0000","last_undegraded":"2023-05-22T15:43:30.174523+0000","last_fullsized":"2023-05-22T15:43:30.174523+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_clean_scrub_stamp":"2023-05-22T15:43:29.738376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:52:09.288277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.a","version":"0'0","reported_seq":10,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-05-22T15:43:30.153359+0000","last_change":"2023-05-22T15:43:29.786377+0000","last_active":"2023-05-22T15:43:30.153359+0000","last_peered":"2023-05-22T15:43:30.153359+0000","last_clean":"2023-05-22T15:43:30.153359+0000","last_became_active":"2023-05-22T15:43:29.785660+0000","last_became_peered":"2023-05-22T15:43:29.785660+0000","last_unstale":"2023-05-22T15:43:30.153359+0000","last_undegraded":"2023-05-22T15:43:30.153359+0000","last_fullsized":"2023-05-22T15:43:30.153359+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:43:28.734545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:43:28.734545+0000","last_clean_scrub_stamp":"2023-05-22T15:43:28.734545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:57:33.702470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":98,"reported_epoch":116,"state":"active+clean","last_fresh":"2023-05-22T15:43:29.759473+0000","last_change":"2023-05-22T15:26:44.978858+0000","last_active":"2023-05-22T15:43:29.759473+0000","last_peered":"2023-05-22T15:43:29.759473+0000","last_clean":"2023-05-22T15:43:29.759473+0000","last_became_active":"2023-05-22T15:26:44.978443+0000","last_became_peered":"2023-05-22T15:26:44.978443+0000","last_unstale":"2023-05-22T15:43:29.759473+0000","last_undegraded":"2023-05-22T15:43:29.759473+0000","last_fullsized":"2023-05-22T15:43:29.759473+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:46:29.290243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.a","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-05-22T15:43:30.174457+0000","last_change":"2023-05-22T15:43:30.165980+0000","last_active":"2023-05-22T15:43:30.174457+0000","last_peered":"2023-05-22T15:43:30.174457+0000","last_clean":"2023-05-22T15:43:30.174457+0000","last_became_active":"2023-05-22T15:43:30.165677+0000","last_became_peered":"2023-05-22T15:43:30.165677+0000","last_unstale":"2023-05-22T15:43:30.174457+0000","last_undegraded":"2023-05-22T15:43:30.174457+0000","last_fullsized":"2023-05-22T15:43:30.174457+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_clean_scrub_stamp":"2023-05-22T15:43:29.738376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:21:06.992004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.b","version":"117'2","reported_seq":12,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-05-22T15:43:30.166870+0000","last_change":"2023-05-22T15:43:29.782081+0000","last_active":"2023-05-22T15:43:30.166870+0000","last_peered":"2023-05-22T15:43:30.166870+0000","last_clean":"2023-05-22T15:43:30.166870+0000","last_became_active":"2023-05-22T15:43:29.781811+0000","last_became_peered":"2023-05-22T15:43:29.781811+0000","last_unstale":"2023-05-22T15:43:30.166870+0000","last_undegraded":"2023-05-22T15:43:30.166870+0000","last_fullsized":"2023-05-22T15:43:30.166870+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:43:28.734545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:43:28.734545+0000","last_clean_scrub_stamp":"2023-05-22T15:43:28.734545+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:58:33.399611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":99,"reported_epoch":116,"state":"active+clean","last_fresh":"2023-05-22T15:43:29.773979+0000","last_change":"2023-05-22T15:26:44.982421+0000","last_active":"2023-05-22T15:43:29.773979+0000","last_peered":"2023-05-22T15:43:29.773979+0000","last_clean":"2023-05-22T15:43:29.773979+0000","last_became_active":"2023-05-22T15:26:44.981586+0000","last_became_peered":"2023-05-22T15:26:44.981586+0000","last_unstale":"2023-05-22T15:43:29.773979+0000","last_undegraded":"2023-05-22T15:43:29.773979+0000","last_fullsized":"2023-05-22T15:43:29.773979+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:14:11.156544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":15,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":14,"num_pg":32,"stat_sum":{"num_bytes":2330,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":884,"num_read_kb":1748,"num_write":319,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":205,"ondisk_log_size":205,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575051,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35156,"kb_used_data":1996,"kb_used_omap":0,"kb_used_meta":33152,"kb_avail":94336684,"statfs":{"total":96636764160,"available":96600764416,"internally_reserved":0,"allocated":2043904,"data_stored":1348182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":33947648},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,4,0,1,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 15:43:00 2023","interfaces":[{"interface":"back","average":{"1min":0.78900000000000003,"5min":0.84199999999999997,"15min":0.86199999999999999},"min":{"1min":0.51300000000000001,"5min":0.44800000000000001,"15min":0.42599999999999999},"max":{"1min":1.151,"5min":1.4199999999999999,"15min":1.7709999999999999},"last":2.637},{"interface":"front","average":{"1min":0.87,"5min":0.86399999999999999,"15min":0.879},"min":{"1min":0.52900000000000003,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":1.107,"5min":1.2090000000000001,"15min":1.3169999999999999},"last":3.0449999999999999}]},{"osd":1,"last update":"Mon May 22 15:43:00 2023","interfaces":[{"interface":"back","average":{"1min":0.94899999999999995,"5min":0.95599999999999996,"15min":0.98099999999999998},"min":{"1min":0.60999999999999999,"5min":0.5,"15min":0.5},"max":{"1min":1.2589999999999999,"5min":1.385,"15min":1.8300000000000001},"last":0.505},{"interface":"front","average":{"1min":0.95699999999999996,"5min":1.002,"15min":1.02},"min":{"1min":0.63,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":1.1950000000000001,"5min":1.3169999999999999,"15min":2.161},"last":2.7269999999999999}]},{"osd":2,"last update":"Mon May 22 15:43:00 2023","interfaces":[{"interface":"back","average":{"1min":0.84699999999999998,"5min":0.85099999999999998,"15min":0.90300000000000002},"min":{"1min":0.60899999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.2130000000000001,"5min":1.2130000000000001,"15min":6.7279999999999998},"last":0.67800000000000005},{"interface":"front","average":{"1min":0.90900000000000003,"5min":0.93600000000000005,"15min":0.96299999999999997},"min":{"1min":0.56599999999999995,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":1.2809999999999999,"5min":1.2809999999999999,"15min":6.8810000000000002},"last":3.141}]},{"osd":3,"last update":"Mon May 22 15:43:00 2023","interfaces":[{"interface":"back","average":{"1min":0.92300000000000004,"5min":0.94299999999999995,"15min":0.95599999999999996},"min":{"1min":0.56399999999999995,"5min":0.55900000000000005,"15min":0.53800000000000003},"max":{"1min":1.214,"5min":1.3360000000000001,"15min":3.0569999999999999},"last":2.8149999999999999},{"interface":"front","average":{"1min":0.91900000000000004,"5min":0.95299999999999996,"15min":0.97699999999999998},"min":{"1min":0.72099999999999997,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":1.1120000000000001,"5min":1.3029999999999999,"15min":1.4710000000000001},"last":0.75900000000000001}]},{"osd":4,"last update":"Mon May 22 15:43:00 2023","interfaces":[{"interface":"back","average":{"1min":1.014,"5min":0.91300000000000003,"15min":0.89000000000000001},"min":{"1min":0.69599999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"max":{"1min":1.3460000000000001,"5min":1.621,"15min":1.621},"last":3.472},{"interface":"front","average":{"1min":1.006,"5min":0.99299999999999999,"15min":0.97599999999999998},"min":{"1min":0.65900000000000003,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.2829999999999999,"5min":1.4319999999999999,"15min":1.958},"last":15.627000000000001}]},{"osd":5,"last update":"Mon May 22 15:43:00 2023","interfaces":[{"interface":"back","average":{"1min":1.0669999999999999,"5min":1.026,"15min":0.999},"min":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.56699999999999995},"max":{"1min":1.3340000000000001,"5min":1.409,"15min":1.5029999999999999},"last":4.5599999999999996},{"interface":"front","average":{"1min":1.123,"5min":1.125,"15min":1.135},"min":{"1min":0.72199999999999998,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":1.4039999999999999,"5min":1.5229999999999999,"15min":5.0839999999999996},"last":4.3799999999999999}]},{"osd":6,"last update":"Mon May 22 15:43:00 2023","interfaces":[{"interface":"back","average":{"1min":1.1120000000000001,"5min":1.0529999999999999,"15min":1.0720000000000001},"min":{"1min":0.80800000000000005,"5min":0.65800000000000003,"15min":0.65800000000000003},"max":{"1min":1.4950000000000001,"5min":1.4950000000000001,"15min":5.1719999999999997},"last":6.5069999999999997},{"interface":"front","average":{"1min":1.0960000000000001,"5min":1.1200000000000001,"15min":1.1279999999999999},"min":{"1min":0.73699999999999999,"5min":0.70099999999999996,"15min":0.70099999999999996},"max":{"1min":1.3360000000000001,"5min":1.702,"15min":1.879},"last":3.3500000000000001}]}]},{"osd":6,"up_from":13,"seq":55834575052,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30928,"kb_used_data":1416,"kb_used_omap":0,"kb_used_meta":29504,"kb_avail":94340912,"statfs":{"total":96636764160,"available":96605093888,"internally_reserved":0,"allocated":1449984,"data_stored":758338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":30212096},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 15:43:11 2023","interfaces":[{"interface":"back","average":{"1min":0.89500000000000002,"5min":0.88100000000000001,"15min":0.90200000000000002},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":1.486,"5min":1.486,"15min":1.486},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.93799999999999994,"5min":0.91300000000000003,"15min":0.90700000000000003},"min":{"1min":0.53500000000000003,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":1.395,"5min":1.5680000000000001,"15min":1.5680000000000001},"last":0.93700000000000006}]},{"osd":1,"last update":"Mon May 22 15:43:11 2023","interfaces":[{"interface":"back","average":{"1min":0.85399999999999998,"5min":0.91700000000000004,"15min":0.91200000000000003},"min":{"1min":0.504,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":1.1499999999999999,"5min":1.3169999999999999,"15min":1.375},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.95099999999999996},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":1.325,"5min":1.554,"15min":1.554},"last":0.65500000000000003}]},{"osd":2,"last update":"Mon May 22 15:43:11 2023","interfaces":[{"interface":"back","average":{"1min":0.91000000000000003,"5min":0.95899999999999996,"15min":0.97999999999999998},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":1.276,"5min":1.425,"15min":1.425},"last":1.0609999999999999},{"interface":"front","average":{"1min":0.92200000000000004,"5min":1.022,"15min":1.042},"min":{"1min":0.5,"5min":0.442,"15min":0.442},"max":{"1min":1.6020000000000001,"5min":1.6020000000000001,"15min":1.6020000000000001},"last":0.78300000000000003}]},{"osd":3,"last update":"Mon May 22 15:43:11 2023","interfaces":[{"interface":"back","average":{"1min":0.90100000000000002,"5min":0.98399999999999999,"15min":0.98899999999999999},"min":{"1min":0.55700000000000005,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":1.333,"5min":1.679,"15min":1.679},"last":0.753},{"interface":"front","average":{"1min":0.91400000000000003,"5min":0.95399999999999996,"15min":0.95799999999999996},"min":{"1min":0.54700000000000004,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.2150000000000001,"5min":1.4570000000000001,"15min":1.4570000000000001},"last":0.84799999999999998}]},{"osd":4,"last update":"Mon May 22 15:43:11 2023","interfaces":[{"interface":"back","average":{"1min":0.91700000000000004,"5min":0.92600000000000005,"15min":0.92300000000000004},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":1.393,"5min":1.6859999999999999,"15min":1.6859999999999999},"last":1.0049999999999999},{"interface":"front","average":{"1min":1.0469999999999999,"5min":1.02,"15min":1.026},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":1.3799999999999999,"5min":1.8300000000000001,"15min":1.8300000000000001},"last":1.137}]},{"osd":5,"last update":"Mon May 22 15:43:11 2023","interfaces":[{"interface":"back","average":{"1min":0.94099999999999995,"5min":0.98799999999999999,"15min":0.97099999999999997},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":1.419,"5min":1.6499999999999999,"15min":1.6499999999999999},"last":1.2929999999999999},{"interface":"front","average":{"1min":1.139,"5min":1.1160000000000001,"15min":1.099},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.728,"5min":1.728,"15min":1.728},"last":1.1000000000000001}]},{"osd":7,"last update":"Mon May 22 15:43:11 2023","interfaces":[{"interface":"back","average":{"1min":1.135,"5min":1.1200000000000001,"15min":1.127},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":1.651,"5min":1.9179999999999999,"15min":1.9179999999999999},"last":1.2629999999999999},{"interface":"front","average":{"1min":1.1080000000000001,"5min":1.1319999999999999,"15min":1.1279999999999999},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":1.5640000000000001,"5min":1.8520000000000001,"15min":1.8520000000000001},"last":1.2110000000000001}]}]},{"osd":5,"up_from":13,"seq":55834575052,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":24780,"kb_used_data":1412,"kb_used_omap":0,"kb_used_meta":23360,"kb_avail":94347060,"statfs":{"total":96636764160,"available":96611389440,"internally_reserved":0,"allocated":1445888,"data_stored":757772,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":23920640},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 15:43:08 2023","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.82999999999999996,"15min":0.878},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.016,"5min":1.3440000000000001,"15min":1.3440000000000001},"last":1.0860000000000001},{"interface":"front","average":{"1min":0.81100000000000005,"5min":0.877,"15min":0.88100000000000001},"min":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.497},"max":{"1min":1.353,"5min":2.1699999999999999,"15min":2.1699999999999999},"last":0.754}]},{"osd":1,"last update":"Mon May 22 15:43:08 2023","interfaces":[{"interface":"back","average":{"1min":0.85899999999999999,"5min":0.96499999999999997,"15min":0.99299999999999999},"min":{"1min":0.52400000000000002,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":1.163,"5min":1.9370000000000001,"15min":1.9370000000000001},"last":1.042},{"interface":"front","average":{"1min":0.95299999999999996,"5min":0.98699999999999999,"15min":1.022},"min":{"1min":0.55300000000000005,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.2430000000000001,"5min":1.3680000000000001,"15min":1.3680000000000001},"last":1.179}]},{"osd":2,"last update":"Mon May 22 15:43:08 2023","interfaces":[{"interface":"back","average":{"1min":0.95399999999999996,"5min":0.95899999999999996,"15min":0.93700000000000006},"min":{"1min":0.65100000000000002,"5min":0.504,"15min":0.504},"max":{"1min":1.1870000000000001,"5min":1.9830000000000001,"15min":1.9830000000000001},"last":0.94599999999999995},{"interface":"front","average":{"1min":0.90500000000000003,"5min":0.94999999999999996,"15min":0.94099999999999995},"min":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.56599999999999995},"max":{"1min":1.304,"5min":2.1120000000000001,"15min":2.1120000000000001},"last":0.91200000000000003}]},{"osd":3,"last update":"Mon May 22 15:43:08 2023","interfaces":[{"interface":"back","average":{"1min":0.86199999999999999,"5min":0.96699999999999997,"15min":0.95199999999999996},"min":{"1min":0.65900000000000003,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.149,"5min":1.4419999999999999,"15min":1.4419999999999999},"last":0.97899999999999998},{"interface":"front","average":{"1min":0.96799999999999997,"5min":1.038,"15min":1.034},"min":{"1min":0.55000000000000004,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":1.1479999999999999,"5min":2.52,"15min":2.52},"last":0.86499999999999999}]},{"osd":4,"last update":"Mon May 22 15:43:08 2023","interfaces":[{"interface":"back","average":{"1min":0.92200000000000004,"5min":0.96799999999999997,"15min":0.93500000000000005},"min":{"1min":0.65200000000000002,"5min":0.60599999999999998,"15min":0.57599999999999996},"max":{"1min":1.2210000000000001,"5min":2.226,"15min":2.226},"last":1.131},{"interface":"front","average":{"1min":0.88200000000000001,"5min":1.0389999999999999,"15min":1.026},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.1639999999999999,"5min":1.6859999999999999,"15min":1.6859999999999999},"last":1.2849999999999999}]},{"osd":6,"last update":"Mon May 22 15:43:08 2023","interfaces":[{"interface":"back","average":{"1min":1.0009999999999999,"5min":1.077,"15min":1.0389999999999999},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.2929999999999999,"5min":2.831,"15min":2.831},"last":1.1080000000000001},{"interface":"front","average":{"1min":1.0089999999999999,"5min":1.0880000000000001,"15min":1.071},"min":{"1min":0.629,"5min":0.629,"15min":0.629},"max":{"1min":1.302,"5min":2.3690000000000002,"15min":2.3690000000000002},"last":1.236}]},{"osd":7,"last update":"Mon May 22 15:43:08 2023","interfaces":[{"interface":"back","average":{"1min":1.0660000000000001,"5min":1.151,"15min":1.1140000000000001},"min":{"1min":0.70999999999999996,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":1.2609999999999999,"5min":1.849,"15min":1.849},"last":1.3640000000000001},{"interface":"front","average":{"1min":1.0980000000000001,"5min":1.198,"15min":1.1659999999999999},"min":{"1min":0.76100000000000001,"5min":0.75700000000000001,"15min":0.71699999999999997},"max":{"1min":1.4159999999999999,"5min":2.7429999999999999,"15min":2.7429999999999999},"last":1.397}]}]},{"osd":4,"up_from":13,"seq":55834575050,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":26588,"kb_used_data":1428,"kb_used_omap":0,"kb_used_meta":25152,"kb_avail":94345252,"statfs":{"total":96636764160,"available":96609538048,"internally_reserved":0,"allocated":1462272,"data_stored":759374,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":25755648},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 15:43:11 2023","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.89900000000000002,"15min":0.90000000000000002},"min":{"1min":0.505,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":1.048,"5min":1.9910000000000001,"15min":1.9910000000000001},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.89000000000000001,"5min":0.89700000000000002,"15min":0.93500000000000005},"min":{"1min":0.51600000000000001,"5min":0.373,"15min":0.373},"max":{"1min":1.264,"5min":1.3779999999999999,"15min":1.633},"last":0.68200000000000005}]},{"osd":1,"last update":"Mon May 22 15:43:11 2023","interfaces":[{"interface":"back","average":{"1min":0.82099999999999995,"5min":0.95899999999999996,"15min":0.97299999999999998},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":1.1779999999999999,"5min":1.383,"15min":1.3859999999999999},"last":0.78500000000000003},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.872,"15min":0.90200000000000002},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1,"5min":1.3120000000000001,"15min":1.3640000000000001},"last":0.74099999999999999}]},{"osd":2,"last update":"Mon May 22 15:43:11 2023","interfaces":[{"interface":"back","average":{"1min":0.95799999999999996,"5min":0.95699999999999996,"15min":0.94699999999999995},"min":{"1min":0.50800000000000001,"5min":0.495,"15min":0.495},"max":{"1min":1.96,"5min":2.1019999999999999,"15min":2.1019999999999999},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.93200000000000005,"5min":0.996,"15min":1.002},"min":{"1min":0.48799999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":1.4399999999999999,"5min":1.4399999999999999,"15min":1.6000000000000001},"last":0.59299999999999997}]},{"osd":3,"last update":"Mon May 22 15:43:11 2023","interfaces":[{"interface":"back","average":{"1min":0.872,"5min":0.91200000000000003,"15min":0.93799999999999994},"min":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":1.2310000000000001,"5min":1.29,"15min":1.665},"last":0.51600000000000001},{"interface":"front","average":{"1min":1.016,"5min":1.0489999999999999,"15min":1.0309999999999999},"min":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":1.4179999999999999,"5min":2.1720000000000002,"15min":2.1720000000000002},"last":0.71299999999999997}]},{"osd":5,"last update":"Mon May 22 15:43:11 2023","interfaces":[{"interface":"back","average":{"1min":0.96799999999999997,"5min":1.022,"15min":0.98999999999999999},"min":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.51200000000000001},"max":{"1min":1.286,"5min":1.472,"15min":1.6830000000000001},"last":1.0109999999999999},{"interface":"front","average":{"1min":0.95899999999999996,"5min":0.97499999999999998,"15min":0.97499999999999998},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.3420000000000001,"5min":1.401,"15min":1.6719999999999999},"last":0.86099999999999999}]},{"osd":6,"last update":"Mon May 22 15:43:11 2023","interfaces":[{"interface":"back","average":{"1min":0.98199999999999998,"5min":1.0249999999999999,"15min":0.98899999999999999},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":1.3620000000000001,"5min":2.2349999999999999,"15min":2.2349999999999999},"last":0.93899999999999995},{"interface":"front","average":{"1min":1.0740000000000001,"5min":1.1299999999999999,"15min":1.107},"min":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":1.4339999999999999,"5min":1.8220000000000001,"15min":1.8220000000000001},"last":0.90700000000000003}]},{"osd":7,"last update":"Mon May 22 15:43:11 2023","interfaces":[{"interface":"back","average":{"1min":1.0189999999999999,"5min":1.0600000000000001,"15min":1.0569999999999999},"min":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.67700000000000005},"max":{"1min":1.452,"5min":1.5800000000000001,"15min":1.5800000000000001},"last":0.97999999999999998},{"interface":"front","average":{"1min":1.163,"5min":1.1639999999999999,"15min":1.1419999999999999},"min":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"max":{"1min":1.53,"5min":2.306,"15min":2.306},"last":0.82699999999999996}]}]},{"osd":3,"up_from":13,"seq":55834575050,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35672,"kb_used_data":1424,"kb_used_omap":0,"kb_used_meta":34240,"kb_avail":94336168,"statfs":{"total":96636764160,"available":96600236032,"internally_reserved":0,"allocated":1458176,"data_stored":759272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":35061760},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 15:42:54 2023","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.61199999999999999,"15min":0.628},"min":{"1min":0.312,"5min":0.19,"15min":0.19},"max":{"1min":1.1519999999999999,"5min":1.194,"15min":1.194},"last":0.70099999999999996},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.68700000000000006,"15min":0.69999999999999996},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.88300000000000001,"5min":1.1799999999999999,"15min":1.296},"last":0.439}]},{"osd":1,"last update":"Mon May 22 15:42:54 2023","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.61899999999999999,"15min":0.66300000000000003},"min":{"1min":0.315,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.80000000000000004,"5min":1.008,"15min":1.0600000000000001},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.71899999999999997,"15min":0.75800000000000001},"min":{"1min":0.34399999999999997,"5min":0.313,"15min":0.216},"max":{"1min":0.82299999999999995,"5min":1.222,"15min":1.222},"last":0.56999999999999995}]},{"osd":2,"last update":"Mon May 22 15:42:54 2023","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.76700000000000002,"15min":0.81200000000000006},"min":{"1min":0.41599999999999998,"5min":0.36599999999999999,"15min":0.32200000000000001},"max":{"1min":1.506,"5min":1.506,"15min":1.506},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.748,"15min":0.80200000000000005},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.29699999999999999},"max":{"1min":1.097,"5min":1.181,"15min":1.468},"last":0.62}]},{"osd":4,"last update":"Mon May 22 15:42:54 2023","interfaces":[{"interface":"back","average":{"1min":0.84399999999999997,"5min":0.92300000000000004,"15min":0.95299999999999996},"min":{"1min":0.54500000000000004,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":1.413,"5min":1.413,"15min":1.4470000000000001},"last":0.89200000000000002},{"interface":"front","average":{"1min":0.81799999999999995,"5min":0.94699999999999995,"15min":1.0009999999999999},"min":{"1min":0.64800000000000002,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.077,"5min":1.3740000000000001,"15min":1.4470000000000001},"last":0.84899999999999998}]},{"osd":5,"last update":"Mon May 22 15:42:54 2023","interfaces":[{"interface":"back","average":{"1min":0.83199999999999996,"5min":0.96099999999999997,"15min":1.01},"min":{"1min":0.53200000000000003,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":1.2070000000000001,"5min":1.3149999999999999,"15min":1.4339999999999999},"last":0.97299999999999998},{"interface":"front","average":{"1min":0.94899999999999995,"5min":0.98999999999999999,"15min":1.0429999999999999},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.4299999999999999,"5min":1.4299999999999999,"15min":1.643},"last":0.81499999999999995}]},{"osd":6,"last update":"Mon May 22 15:42:54 2023","interfaces":[{"interface":"back","average":{"1min":0.85099999999999998,"5min":0.98099999999999998,"15min":1.0369999999999999},"min":{"1min":0.47699999999999998,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.1539999999999999,"5min":1.4139999999999999,"15min":1.863},"last":1.0549999999999999},{"interface":"front","average":{"1min":0.93100000000000005,"5min":1.0229999999999999,"15min":1.0720000000000001},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":1.2909999999999999,"5min":1.3109999999999999,"15min":1.704},"last":0.92600000000000005}]},{"osd":7,"last update":"Mon May 22 15:42:54 2023","interfaces":[{"interface":"back","average":{"1min":0.96699999999999997,"5min":1.0469999999999999,"15min":1.1000000000000001},"min":{"1min":0.58599999999999997,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":1.407,"5min":1.407,"15min":1.5640000000000001},"last":1.0289999999999999},{"interface":"front","average":{"1min":0.94099999999999995,"5min":1.024,"15min":1.0700000000000001},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":1.8700000000000001,"5min":1.8700000000000001,"15min":1.8700000000000001},"last":0.73899999999999999}]}]},{"osd":2,"up_from":13,"seq":55834575051,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":17804,"kb_used_data":1412,"kb_used_omap":0,"kb_used_meta":16384,"kb_avail":94354036,"statfs":{"total":96636764160,"available":96618532864,"internally_reserved":0,"allocated":1445888,"data_stored":757736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":16777216},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 15:42:58 2023","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.65500000000000003,"15min":0.65400000000000003},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":1.1080000000000001,"5min":1.8240000000000001,"15min":1.8240000000000001},"last":0.83299999999999996},{"interface":"front","average":{"1min":0.71099999999999997,"5min":0.75600000000000001,"15min":0.746},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.253},"max":{"1min":1.0369999999999999,"5min":1.8160000000000001,"15min":1.8160000000000001},"last":0.48199999999999998}]},{"osd":1,"last update":"Mon May 22 15:42:58 2023","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.73099999999999998,"15min":0.73399999999999999},"min":{"1min":0.34799999999999998,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.96599999999999997,"5min":1.2,"15min":1.2},"last":0.72499999999999998},{"interface":"front","average":{"1min":0.73899999999999999,"5min":0.81899999999999995,"15min":0.81200000000000006},"min":{"1min":0.317,"5min":0.317,"15min":0.29099999999999998},"max":{"1min":0.99299999999999999,"5min":2.5089999999999999,"15min":2.5089999999999999},"last":0.57999999999999996}]},{"osd":3,"last update":"Mon May 22 15:42:58 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.72399999999999998,"15min":0.72099999999999997},"min":{"1min":0.36699999999999999,"5min":0.36299999999999999,"15min":0.26700000000000002},"max":{"1min":0.95699999999999996,"5min":2.7629999999999999,"15min":2.7629999999999999},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.871,"5min":0.88600000000000001,"15min":0.88100000000000001},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.33300000000000002},"max":{"1min":1.256,"5min":2.6840000000000002,"15min":2.6840000000000002},"last":1.143}]},{"osd":4,"last update":"Mon May 22 15:42:58 2023","interfaces":[{"interface":"back","average":{"1min":1.0069999999999999,"5min":1.03,"15min":1.016},"min":{"1min":0.76000000000000001,"5min":0.65800000000000003,"15min":0.55500000000000005},"max":{"1min":1.536,"5min":2.8039999999999998,"15min":2.8039999999999998},"last":1.012},{"interface":"front","average":{"1min":1.0249999999999999,"5min":1.036,"15min":1.028},"min":{"1min":0.59199999999999997,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":1.4019999999999999,"5min":2.0819999999999999,"15min":2.0819999999999999},"last":1.121}]},{"osd":5,"last update":"Mon May 22 15:42:58 2023","interfaces":[{"interface":"back","average":{"1min":1.0700000000000001,"5min":1.0669999999999999,"15min":1.036},"min":{"1min":0.622,"5min":0.622,"15min":0.55200000000000005},"max":{"1min":1.4570000000000001,"5min":2.3860000000000001,"15min":2.3860000000000001},"last":0.90800000000000003},{"interface":"front","average":{"1min":0.95799999999999996,"5min":0.996,"15min":0.995},"min":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.56200000000000006},"max":{"1min":1.327,"5min":3.0299999999999998,"15min":3.0299999999999998},"last":1.0680000000000001}]},{"osd":6,"last update":"Mon May 22 15:42:58 2023","interfaces":[{"interface":"back","average":{"1min":0.97999999999999998,"5min":1.024,"15min":1.0309999999999999},"min":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.56399999999999995},"max":{"1min":1.49,"5min":2.883,"15min":2.883},"last":0.98399999999999999},{"interface":"front","average":{"1min":1.0660000000000001,"5min":1.107,"15min":1.095},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.53400000000000003},"max":{"1min":1.421,"5min":3.1459999999999999,"15min":3.1459999999999999},"last":1.0920000000000001}]},{"osd":7,"last update":"Mon May 22 15:42:58 2023","interfaces":[{"interface":"back","average":{"1min":1.083,"5min":1.1359999999999999,"15min":1.1220000000000001},"min":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.58299999999999996},"max":{"1min":1.5029999999999999,"5min":2.5990000000000002,"15min":2.5990000000000002},"last":1.0389999999999999},{"interface":"front","average":{"1min":1.024,"5min":1.0960000000000001,"15min":1.0980000000000001},"min":{"1min":0.60499999999999998,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":1.381,"5min":3.8799999999999999,"15min":3.8799999999999999},"last":1.0269999999999999}]}]},{"osd":1,"up_from":13,"seq":55834575051,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":24840,"kb_used_data":1408,"kb_used_omap":0,"kb_used_meta":23424,"kb_avail":94347000,"statfs":{"total":96636764160,"available":96611328000,"internally_reserved":0,"allocated":1441792,"data_stored":757702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":23986176},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 15:43:13 2023","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.61599999999999999,"15min":0.63400000000000001},"min":{"1min":0.26300000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.90900000000000003,"5min":1.2270000000000001,"15min":1.7689999999999999},"last":0.439},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.746,"15min":0.76400000000000001},"min":{"1min":0.29299999999999998,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":1.0860000000000001,"5min":1.2290000000000001,"15min":1.2529999999999999},"last":0.47299999999999998}]},{"osd":2,"last update":"Mon May 22 15:43:13 2023","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.747,"15min":0.73499999999999999},"min":{"1min":0.39400000000000002,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":1.1639999999999999,"5min":1.391,"15min":1.7290000000000001},"last":0.503},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.753,"15min":0.76100000000000001},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.27800000000000002},"max":{"1min":1.155,"5min":1.276,"15min":2.4049999999999998},"last":0.45900000000000002}]},{"osd":3,"last update":"Mon May 22 15:43:13 2023","interfaces":[{"interface":"back","average":{"1min":0.69999999999999996,"5min":0.80100000000000005,"15min":0.82699999999999996},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.317},"max":{"1min":0.996,"5min":1.657,"15min":1.657},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.80200000000000005,"15min":0.82799999999999996},"min":{"1min":0.35499999999999998,"5min":0.315,"15min":0.315},"max":{"1min":1.1910000000000001,"5min":2.0569999999999999,"15min":7.5620000000000003},"last":0.53800000000000003}]},{"osd":4,"last update":"Mon May 22 15:43:13 2023","interfaces":[{"interface":"back","average":{"1min":0.96299999999999997,"5min":0.97099999999999997,"15min":0.95799999999999996},"min":{"1min":0.55200000000000005,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":2.9529999999999998,"5min":2.9529999999999998,"15min":2.9529999999999998},"last":0.95099999999999996},{"interface":"front","average":{"1min":1.012,"5min":0.98799999999999999,"15min":0.996},"min":{"1min":0.504,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":2.7759999999999998,"5min":2.7759999999999998,"15min":2.7759999999999998},"last":1.147}]},{"osd":5,"last update":"Mon May 22 15:43:13 2023","interfaces":[{"interface":"back","average":{"1min":0.92200000000000004,"5min":1.0169999999999999,"15min":1.03},"min":{"1min":0.58399999999999996,"5min":0.443,"15min":0.443},"max":{"1min":1.409,"5min":1.9570000000000001,"15min":1.9570000000000001},"last":0.76300000000000001},{"interface":"front","average":{"1min":0.91600000000000004,"5min":1.0189999999999999,"15min":1.0449999999999999},"min":{"1min":0.53400000000000003,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":1.3069999999999999,"5min":2.1200000000000001,"15min":4.4710000000000001},"last":0.90400000000000003}]},{"osd":6,"last update":"Mon May 22 15:43:13 2023","interfaces":[{"interface":"back","average":{"1min":1.075,"5min":1.018,"15min":1.077},"min":{"1min":0.56299999999999994,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":2.831,"5min":2.831,"15min":14.968999999999999},"last":1.018},{"interface":"front","average":{"1min":0.97599999999999998,"5min":1.0620000000000001,"15min":1.127},"min":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":1.456,"5min":1.7629999999999999,"15min":13.242000000000001},"last":0.96299999999999997}]},{"osd":7,"last update":"Mon May 22 15:43:13 2023","interfaces":[{"interface":"back","average":{"1min":1.056,"5min":1.1000000000000001,"15min":1.105},"min":{"1min":0.60299999999999998,"5min":0.56799999999999995,"15min":0.56799999999999995},"max":{"1min":2.4199999999999999,"5min":2.4199999999999999,"15min":2.4199999999999999},"last":0.93700000000000006},{"interface":"front","average":{"1min":1.0700000000000001,"5min":1.129,"15min":1.133},"min":{"1min":0.47299999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":3.8799999999999999,"5min":3.8799999999999999,"15min":5.681},"last":1.0529999999999999}]}]},{"osd":0,"up_from":13,"seq":55834575052,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":24144,"kb_used_data":1992,"kb_used_omap":0,"kb_used_meta":22144,"kb_avail":94347696,"statfs":{"total":96636764160,"available":96612040704,"internally_reserved":0,"allocated":2039808,"data_stored":1348636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":22675456},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon May 22 15:43:08 2023","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":0.66600000000000004,"15min":0.67800000000000005},"min":{"1min":0.33500000000000002,"5min":0.223,"15min":0.223},"max":{"1min":1.466,"5min":1.466,"15min":1.466},"last":0.81499999999999995},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.70399999999999996,"15min":0.73199999999999998},"min":{"1min":0.34200000000000003,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":1.71,"5min":1.71,"15min":1.71},"last":0.73799999999999999}]},{"osd":2,"last update":"Mon May 22 15:43:08 2023","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.72499999999999998,"15min":0.746},"min":{"1min":0.38600000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":1.165,"5min":1.165,"15min":1.4059999999999999},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.75700000000000001,"15min":0.79600000000000004},"min":{"1min":0.38500000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":1.0189999999999999,"5min":1.1850000000000001,"15min":1.3999999999999999},"last":0.95599999999999996}]},{"osd":3,"last update":"Mon May 22 15:43:08 2023","interfaces":[{"interface":"back","average":{"1min":0.754,"5min":0.77300000000000002,"15min":0.76500000000000001},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":1.923,"5min":1.9299999999999999,"15min":1.9299999999999999},"last":0.77800000000000002},{"interface":"front","average":{"1min":0.82599999999999996,"5min":0.85099999999999998,"15min":0.875},"min":{"1min":0.38600000000000001,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":1.758,"5min":1.758,"15min":1.758},"last":0.88600000000000001}]},{"osd":4,"last update":"Mon May 22 15:43:08 2023","interfaces":[{"interface":"back","average":{"1min":0.93300000000000005,"5min":0.94399999999999995,"15min":0.95199999999999996},"min":{"1min":0.63700000000000001,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":1.6000000000000001,"5min":2.923,"15min":2.923},"last":0.92900000000000005},{"interface":"front","average":{"1min":1.0169999999999999,"5min":1.028,"15min":1.028},"min":{"1min":0.64900000000000002,"5min":0.64600000000000002,"15min":0.54900000000000004},"max":{"1min":2.0129999999999999,"5min":2.0129999999999999,"15min":2.0129999999999999},"last":1.0429999999999999}]},{"osd":5,"last update":"Mon May 22 15:43:08 2023","interfaces":[{"interface":"back","average":{"1min":1.0449999999999999,"5min":1.0109999999999999,"15min":1.0409999999999999},"min":{"1min":0.71199999999999997,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.97,"5min":1.97,"15min":1.97},"last":1.147},{"interface":"front","average":{"1min":0.94799999999999995,"5min":1.0249999999999999,"15min":1.01},"min":{"1min":0.57199999999999995,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.804,"5min":5.0519999999999996,"15min":5.0519999999999996},"last":0.86799999999999999}]},{"osd":6,"last update":"Mon May 22 15:43:08 2023","interfaces":[{"interface":"back","average":{"1min":1.1379999999999999,"5min":1.0880000000000001,"15min":1.0880000000000001},"min":{"1min":0.58699999999999997,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":2.117,"5min":2.117,"15min":2.117},"last":0.997},{"interface":"front","average":{"1min":1.123,"5min":1.1399999999999999,"15min":1.1140000000000001},"min":{"1min":0.74299999999999999,"5min":0.499,"15min":0.499},"max":{"1min":2.052,"5min":5.2640000000000002,"15min":5.2640000000000002},"last":1.2390000000000001}]},{"osd":7,"last update":"Mon May 22 15:43:08 2023","interfaces":[{"interface":"back","average":{"1min":1.1519999999999999,"5min":1.1200000000000001,"15min":1.125},"min":{"1min":0.77500000000000002,"5min":0.65700000000000003,"15min":0.58099999999999996},"max":{"1min":1.839,"5min":1.839,"15min":1.839},"last":1.3149999999999999},{"interface":"front","average":{"1min":1.0669999999999999,"5min":1.099,"15min":1.093},"min":{"1min":0.63300000000000001,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":2.0800000000000001,"5min":3.9060000000000001,"15min":3.9060000000000001},"last":1.02}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1570,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-05-22T15:43:34.159 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-05-22T15:43:34.324 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:34.319+0000 7faabb56b640 1 Processor -- start 2023-05-22T15:43:34.324 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:34.319+0000 7faabb56b640 1 -- start start 2023-05-22T15:43:34.325 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:34.319+0000 7faabb56b640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7faab40ca4e0 0x7faab415e460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:34.325 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:34.319+0000 7faabb56b640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7faab4070df0 0x7faab4071210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:34.326 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:34.319+0000 7faabb56b640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faab40b0650 con 0x7faab40c9b40 2023-05-22T15:43:34.326 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:34.319+0000 7faabb56b640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faab40b0370 con 0x7faab4070df0 2023-05-22T15:43:34.326 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:34.319+0000 7faabb56b640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faab415d980 con 0x7faab40ca4e0 2023-05-22T15:43:34.326 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:34.319+0000 7faab9b08640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7faab4070df0 0x7faab4071210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:34.326 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:34.319+0000 7faab9b08640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7faab4070df0 0x7faab4071210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36064/0 (socket says 172.21.15.16:36064) 2023-05-22T15:43:34.326 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:34.319+0000 7faab9b08640 1 -- 172.21.15.16:0/4072439204 learned_addr learned my addr 172.21.15.16:0/4072439204 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:43:34.326 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:34.319+0000 7faab9307640 1 --2- 172.21.15.16:0/4072439204 >> v2:172.21.15.62:3300/0 conn(0x7faab40ca4e0 0x7faab415e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:34.327 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:34.319+0000 7faab9b08640 1 -- 172.21.15.16:0/4072439204 >> v2:172.21.15.62:3300/0 conn(0x7faab40ca4e0 msgr2=0x7faab415e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:43:34.327 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:34.319+0000 7faab9b08640 1 --2- 172.21.15.16:0/4072439204 >> v2:172.21.15.62:3300/0 conn(0x7faab40ca4e0 0x7faab415e460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:34.327 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:34.319+0000 7faab9b08640 1 -- 172.21.15.16:0/4072439204 >> v1:172.21.15.62:6789/0 conn(0x7faab40c9b40 legacy=0x7faab40afb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:43:34.327 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:34.319+0000 7faab9b08640 1 -- 172.21.15.16:0/4072439204 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faab415dc60 con 0x7faab4070df0 2023-05-22T15:43:34.327 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:34.323+0000 7faab9b08640 1 --2- 172.21.15.16:0/4072439204 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faab4070df0 0x7faab4071210 secure :-1 s=READY pgs=1461 cs=0 l=1 rev1=1 crypto rx=0x7faab40cad40 tx=0x7faaac01c9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66b2f492c2e35b6f server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:43:34.328 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:34.323+0000 7faaabfff640 1 -- 172.21.15.16:0/4072439204 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faaac0a57c0 con 0x7faab4070df0 2023-05-22T15:43:34.328 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:34.323+0000 7faaabfff640 1 -- 172.21.15.16:0/4072439204 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faaac0a5920 con 0x7faab4070df0 2023-05-22T15:43:34.328 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:34.323+0000 7faaabfff640 1 -- 172.21.15.16:0/4072439204 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faaac0a5cb0 con 0x7faab4070df0 2023-05-22T15:43:34.329 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:34.323+0000 7faabb56b640 1 -- 172.21.15.16:0/4072439204 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faab4070df0 msgr2=0x7faab4071210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:34.329 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:34.323+0000 7faabb56b640 1 --2- 172.21.15.16:0/4072439204 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faab4070df0 0x7faab4071210 secure :-1 s=READY pgs=1461 cs=0 l=1 rev1=1 crypto rx=0x7faab40cad40 tx=0x7faaac01c9a0 comp rx=0 tx=0).stop 2023-05-22T15:43:34.329 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:34.323+0000 7faabb56b640 1 -- 172.21.15.16:0/4072439204 shutdown_connections 2023-05-22T15:43:34.330 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:34.323+0000 7faabb56b640 1 --2- 172.21.15.16:0/4072439204 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faab4070df0 0x7faab4071210 unknown :-1 s=CLOSED pgs=1461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:34.330 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:34.323+0000 7faabb56b640 1 --2- 172.21.15.16:0/4072439204 >> v2:172.21.15.62:3300/0 conn(0x7faab40ca4e0 0x7faab415e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:34.330 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:34.323+0000 7faabb56b640 1 -- 172.21.15.16:0/4072439204 >> 172.21.15.16:0/4072439204 conn(0x7faab4069720 msgr2=0x7faab4069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:43:34.330 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:34.323+0000 7faabb56b640 1 -- 172.21.15.16:0/4072439204 shutdown_connections 2023-05-22T15:43:34.331 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:34.323+0000 7faabb56b640 1 -- 172.21.15.16:0/4072439204 wait complete. 2023-05-22T15:43:34.334 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:34.323+0000 7faabb56b640 1 Processor -- start 2023-05-22T15:43:34.334 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:34.323+0000 7faabb56b640 1 -- start start 2023-05-22T15:43:34.334 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:34.327+0000 7faabb56b640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faab4070df0 0x7faab4158d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:34.335 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:34.327+0000 7faabb56b640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7faab40c9b40 0x7faab40c1a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:34.335 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:34.327+0000 7faabb56b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faab40ca4e0 0x7faab40c3400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:34.335 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:34.327+0000 7faabb56b640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faab414aff0 con 0x7faab40ca4e0 2023-05-22T15:43:34.335 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:34.327+0000 7faabb56b640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faab4156ed0 con 0x7faab4070df0 2023-05-22T15:43:34.336 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:34.327+0000 7faabb56b640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faab40c1fb0 con 0x7faab40c9b40 2023-05-22T15:43:34.336 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:34.327+0000 7faab9b08640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faab40ca4e0 0x7faab40c3400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:34.336 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:34.327+0000 7faab9b08640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faab40ca4e0 0x7faab40c3400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36080/0 (socket says 172.21.15.16:36080) 2023-05-22T15:43:34.336 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:34.327+0000 7faab9b08640 1 -- 172.21.15.16:0/1269782788 learned_addr learned my addr 172.21.15.16:0/1269782788 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:43:34.337 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:34.327+0000 7faab9307640 1 --2- 172.21.15.16:0/1269782788 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faab4070df0 0x7faab4158d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:34.337 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:34.327+0000 7faab8b06640 1 --2- 172.21.15.16:0/1269782788 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7faab40c9b40 0x7faab40c1a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:34.337 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:34.327+0000 7faab9b08640 1 -- 172.21.15.16:0/1269782788 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7faab40c9b40 msgr2=0x7faab40c1a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:34.337 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:34.327+0000 7faab9b08640 1 --2- 172.21.15.16:0/1269782788 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7faab40c9b40 0x7faab40c1a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:34.338 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:34.327+0000 7faab9b08640 1 -- 172.21.15.16:0/1269782788 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faab4070df0 msgr2=0x7faab4158d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:34.338 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:34.327+0000 7faab9b08640 1 --2- 172.21.15.16:0/1269782788 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faab4070df0 0x7faab4158d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:34.338 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:34.327+0000 7faab9b08640 1 -- 172.21.15.16:0/1269782788 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faaac009b30 con 0x7faab40ca4e0 2023-05-22T15:43:34.338 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:34.327+0000 7faab9b08640 1 --2- 172.21.15.16:0/1269782788 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faab40ca4e0 0x7faab40c3400 secure :-1 s=READY pgs=1462 cs=0 l=1 rev1=1 crypto rx=0x7faaac0af040 tx=0x7faaac019740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:43:34.338 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:34.327+0000 7faaa9ffb640 1 -- 172.21.15.16:0/1269782788 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faaac0b8670 con 0x7faab40ca4e0 2023-05-22T15:43:34.339 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:34.327+0000 7faaa9ffb640 1 -- 172.21.15.16:0/1269782788 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faaac0b87d0 con 0x7faab40ca4e0 2023-05-22T15:43:34.339 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:34.327+0000 7faabb56b640 1 -- 172.21.15.16:0/1269782788 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faab40c8e80 con 0x7faab40ca4e0 2023-05-22T15:43:34.339 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:34.327+0000 7faaa9ffb640 1 -- 172.21.15.16:0/1269782788 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faaac0b8b60 con 0x7faab40ca4e0 2023-05-22T15:43:34.339 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:34.327+0000 7faabb56b640 1 -- 172.21.15.16:0/1269782788 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faab40ca8b0 con 0x7faab40ca4e0 2023-05-22T15:43:34.340 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:34.327+0000 7faabb56b640 1 -- 172.21.15.16:0/1269782788 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faab40c9010 con 0x7faab40ca4e0 2023-05-22T15:43:34.340 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:34.327+0000 7faaa9ffb640 1 -- 172.21.15.16:0/1269782788 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 41) v1 ==== 84332+0+0 (secure 0 0 0) 0x7faaac0b8cc0 con 0x7faab40ca4e0 2023-05-22T15:43:34.340 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:34.327+0000 7faaa9ffb640 1 --2- 172.21.15.16:0/1269782788 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7faa980654a0 0x7faa98067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:34.340 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:34.327+0000 7faaa9ffb640 1 -- 172.21.15.16:0/1269782788 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 6382+0+0 (secure 0 0 0) 0x7faaac11e040 con 0x7faab40ca4e0 2023-05-22T15:43:34.341 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:34.327+0000 7faab9307640 1 --2- 172.21.15.16:0/1269782788 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7faa980654a0 0x7faa98067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:34.341 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:34.327+0000 7faab9307640 1 --2- 172.21.15.16:0/1269782788 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7faa980654a0 0x7faa98067950 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7faab000b930 tx=0x7faab0008040 comp rx=0 tx=0).ready entity=mgr.7788 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:43:34.342 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:34.331+0000 7faaa9ffb640 1 -- 172.21.15.16:0/1269782788 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7faaac0ebbc0 con 0x7faab40ca4e0 2023-05-22T15:43:34.488 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:34.483+0000 7faabb56b640 1 -- 172.21.15.16:0/1269782788 --> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7faab4157080 con 0x7faa980654a0 2023-05-22T15:43:34.505 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:34.499+0000 7faaa9ffb640 1 -- 172.21.15.16:0/1269782788 <== mgr.7788 v2:172.21.15.16:6832/36921 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+310237 (secure 0 0 0) 0x7faab4157080 con 0x7faa980654a0 2023-05-22T15:43:34.507 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:43:34.510 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2023-05-22T15:43:34.511 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:34.507+0000 7faabb56b640 1 -- 172.21.15.16:0/1269782788 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7faa980654a0 msgr2=0x7faa98067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:34.511 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:34.507+0000 7faabb56b640 1 --2- 172.21.15.16:0/1269782788 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7faa980654a0 0x7faa98067950 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7faab000b930 tx=0x7faab0008040 comp rx=0 tx=0).stop 2023-05-22T15:43:34.511 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:34.507+0000 7faabb56b640 1 -- 172.21.15.16:0/1269782788 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faab40ca4e0 msgr2=0x7faab40c3400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:34.511 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:34.507+0000 7faabb56b640 1 --2- 172.21.15.16:0/1269782788 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faab40ca4e0 0x7faab40c3400 secure :-1 s=READY pgs=1462 cs=0 l=1 rev1=1 crypto rx=0x7faaac0af040 tx=0x7faaac019740 comp rx=0 tx=0).stop 2023-05-22T15:43:34.511 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:34.507+0000 7faabb56b640 1 -- 172.21.15.16:0/1269782788 shutdown_connections 2023-05-22T15:43:34.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:34.507+0000 7faabb56b640 1 --2- 172.21.15.16:0/1269782788 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faab40ca4e0 0x7faab40c3400 unknown :-1 s=CLOSED pgs=1462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:34.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:34.507+0000 7faabb56b640 1 --2- 172.21.15.16:0/1269782788 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7faa980654a0 0x7faa98067950 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:34.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:34.507+0000 7faabb56b640 1 --2- 172.21.15.16:0/1269782788 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7faab40c9b40 0x7faab40c1a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:34.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:34.507+0000 7faabb56b640 1 --2- 172.21.15.16:0/1269782788 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faab4070df0 0x7faab4158d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:34.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:34.507+0000 7faabb56b640 1 -- 172.21.15.16:0/1269782788 >> 172.21.15.16:0/1269782788 conn(0x7faab4069720 msgr2=0x7faab40c4960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:43:34.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:34.507+0000 7faabb56b640 1 -- 172.21.15.16:0/1269782788 shutdown_connections 2023-05-22T15:43:34.513 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:34.507+0000 7faabb56b640 1 -- 172.21.15.16:0/1269782788 wait complete. 2023-05-22T15:43:34.529 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2023-05-22T15:43:32.996157+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592698,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":884,"num_read_kb":1748,"num_write":364,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":244,"ondisk_log_size":244,"up":210,"acting":210,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":98,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219912,"kb_used_data":12488,"kb_used_omap":0,"kb_used_meta":207360,"kb_avail":754754808,"statfs":{"total":773094113280,"available":772868923392,"internally_reserved":0,"allocated":12787712,"data_stored":7247012,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":212336640},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,4,0,1,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":73,"apply_latency_ms":73,"commit_latency_ns":73000000,"apply_latency_ns":73000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2330,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.256482"},"pg_stats":[{"pgid":"15.3f","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-05-22T15:43:30.184101+0000","last_change":"2023-05-22T15:43:30.183119+0000","last_active":"2023-05-22T15:43:30.184101+0000","last_peered":"2023-05-22T15:43:30.184101+0000","last_clean":"2023-05-22T15:43:30.184101+0000","last_became_active":"2023-05-22T15:43:30.182859+0000","last_became_peered":"2023-05-22T15:43:30.182859+0000","last_unstale":"2023-05-22T15:43:30.184101+0000","last_undegraded":"2023-05-22T15:43:30.184101+0000","last_fullsized":"2023-05-22T15:43:30.184101+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_clean_scrub_stamp":"2023-05-22T15:43:29.738376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:52:09.288277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.3e","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-05-22T15:43:30.164706+0000","last_change":"2023-05-22T15:43:30.163639+0000","last_active":"2023-05-22T15:43:30.164706+0000","last_peered":"2023-05-22T15:43:30.164706+0000","last_clean":"2023-05-22T15:43:30.164706+0000","last_became_active":"2023-05-22T15:43:30.163136+0000","last_became_peered":"2023-05-22T15:43:30.163136+0000","last_unstale":"2023-05-22T15:43:30.164706+0000","last_undegraded":"2023-05-22T15:43:30.164706+0000","last_fullsized":"2023-05-22T15:43:30.164706+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_clean_scrub_stamp":"2023-05-22T15:43:29.738376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T03:40:48.926917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.3d","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-05-22T15:43:30.187909+0000","last_change":"2023-05-22T15:43:30.187039+0000","last_active":"2023-05-22T15:43:30.187909+0000","last_peered":"2023-05-22T15:43:30.187909+0000","last_clean":"2023-05-22T15:43:30.187909+0000","last_became_active":"2023-05-22T15:43:30.186793+0000","last_became_peered":"2023-05-22T15:43:30.186793+0000","last_unstale":"2023-05-22T15:43:30.187909+0000","last_undegraded":"2023-05-22T15:43:30.187909+0000","last_fullsized":"2023-05-22T15:43:30.187909+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_clean_scrub_stamp":"2023-05-22T15:43:29.738376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:44:35.350206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.3c","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-05-22T15:43:30.163798+0000","last_change":"2023-05-22T15:43:30.162829+0000","last_active":"2023-05-22T15:43:30.163798+0000","last_peered":"2023-05-22T15:43:30.163798+0000","last_clean":"2023-05-22T15:43:30.163798+0000","last_became_active":"2023-05-22T15:43:30.162517+0000","last_became_peered":"2023-05-22T15:43:30.162517+0000","last_unstale":"2023-05-22T15:43:30.163798+0000","last_undegraded":"2023-05-22T15:43:30.163798+0000","last_fullsized":"2023-05-22T15:43:30.163798+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_clean_scrub_stamp":"2023-05-22T15:43:29.738376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:38:15.549682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.3b","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-05-22T15:43:30.211224+0000","last_change":"2023-05-22T15:43:30.181145+0000","last_active":"2023-05-22T15:43:30.211224+0000","last_peered":"2023-05-22T15:43:30.211224+0000","last_clean":"2023-05-22T15:43:30.211224+0000","last_became_active":"2023-05-22T15:43:30.180391+0000","last_became_peered":"2023-05-22T15:43:30.180391+0000","last_unstale":"2023-05-22T15:43:30.211224+0000","last_undegraded":"2023-05-22T15:43:30.211224+0000","last_fullsized":"2023-05-22T15:43:30.211224+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_clean_scrub_stamp":"2023-05-22T15:43:29.738376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:34:00.384115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.3a","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-05-22T15:43:30.214374+0000","last_change":"2023-05-22T15:43:30.206158+0000","last_active":"2023-05-22T15:43:30.214374+0000","last_peered":"2023-05-22T15:43:30.214374+0000","last_clean":"2023-05-22T15:43:30.214374+0000","last_became_active":"2023-05-22T15:43:30.205344+0000","last_became_peered":"2023-05-22T15:43:30.205344+0000","last_unstale":"2023-05-22T15:43:30.214374+0000","last_undegraded":"2023-05-22T15:43:30.214374+0000","last_fullsized":"2023-05-22T15:43:30.214374+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_clean_scrub_stamp":"2023-05-22T15:43:29.738376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:20:57.149814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.39","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-05-22T15:43:30.210122+0000","last_change":"2023-05-22T15:43:30.203885+0000","last_active":"2023-05-22T15:43:30.210122+0000","last_peered":"2023-05-22T15:43:30.210122+0000","last_clean":"2023-05-22T15:43:30.210122+0000","last_became_active":"2023-05-22T15:43:30.203150+0000","last_became_peered":"2023-05-22T15:43:30.203150+0000","last_unstale":"2023-05-22T15:43:30.210122+0000","last_undegraded":"2023-05-22T15:43:30.210122+0000","last_fullsized":"2023-05-22T15:43:30.210122+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_clean_scrub_stamp":"2023-05-22T15:43:29.738376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:04:28.418159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.38","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-05-22T15:43:30.190236+0000","last_change":"2023-05-22T15:43:30.189579+0000","last_active":"2023-05-22T15:43:30.190236+0000","last_peered":"2023-05-22T15:43:30.190236+0000","last_clean":"2023-05-22T15:43:30.190236+0000","last_became_active":"2023-05-22T15:43:30.189264+0000","last_became_peered":"2023-05-22T15:43:30.189264+0000","last_unstale":"2023-05-22T15:43:30.190236+0000","last_undegraded":"2023-05-22T15:43:30.190236+0000","last_fullsized":"2023-05-22T15:43:30.190236+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_clean_scrub_stamp":"2023-05-22T15:43:29.738376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:03:33.396793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.37","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-05-22T15:43:30.224021+0000","last_change":"2023-05-22T15:43:30.211619+0000","last_active":"2023-05-22T15:43:30.224021+0000","last_peered":"2023-05-22T15:43:30.224021+0000","last_clean":"2023-05-22T15:43:30.224021+0000","last_became_active":"2023-05-22T15:43:30.210848+0000","last_became_peered":"2023-05-22T15:43:30.210848+0000","last_unstale":"2023-05-22T15:43:30.224021+0000","last_undegraded":"2023-05-22T15:43:30.224021+0000","last_fullsized":"2023-05-22T15:43:30.224021+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_clean_scrub_stamp":"2023-05-22T15:43:29.738376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:36:26.720817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.36","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-05-22T15:43:30.211139+0000","last_change":"2023-05-22T15:43:30.197809+0000","last_active":"2023-05-22T15:43:30.211139+0000","last_peered":"2023-05-22T15:43:30.211139+0000","last_clean":"2023-05-22T15:43:30.211139+0000","last_became_active":"2023-05-22T15:43:30.197078+0000","last_became_peered":"2023-05-22T15:43:30.197078+0000","last_unstale":"2023-05-22T15:43:30.211139+0000","last_undegraded":"2023-05-22T15:43:30.211139+0000","last_fullsized":"2023-05-22T15:43:30.211139+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_clean_scrub_stamp":"2023-05-22T15:43:29.738376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T16:04:25.079184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.35","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-05-22T15:43:30.222171+0000","last_change":"2023-05-22T15:43:30.220667+0000","last_active":"2023-05-22T15:43:30.222171+0000","last_peered":"2023-05-22T15:43:30.222171+0000","last_clean":"2023-05-22T15:43:30.222171+0000","last_became_active":"2023-05-22T15:43:30.219182+0000","last_became_peered":"2023-05-22T15:43:30.219182+0000","last_unstale":"2023-05-22T15:43:30.222171+0000","last_undegraded":"2023-05-22T15:43:30.222171+0000","last_fullsized":"2023-05-22T15:43:30.222171+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_clean_scrub_stamp":"2023-05-22T15:43:29.738376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:38:15.549682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.34","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-05-22T15:43:30.223901+0000","last_change":"2023-05-22T15:43:30.209645+0000","last_active":"2023-05-22T15:43:30.223901+0000","last_peered":"2023-05-22T15:43:30.223901+0000","last_clean":"2023-05-22T15:43:30.223901+0000","last_became_active":"2023-05-22T15:43:30.208670+0000","last_became_peered":"2023-05-22T15:43:30.208670+0000","last_unstale":"2023-05-22T15:43:30.223901+0000","last_undegraded":"2023-05-22T15:43:30.223901+0000","last_fullsized":"2023-05-22T15:43:30.223901+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_clean_scrub_stamp":"2023-05-22T15:43:29.738376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:38:15.549682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.33","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-05-22T15:43:30.207751+0000","last_change":"2023-05-22T15:43:30.207287+0000","last_active":"2023-05-22T15:43:30.207751+0000","last_peered":"2023-05-22T15:43:30.207751+0000","last_clean":"2023-05-22T15:43:30.207751+0000","last_became_active":"2023-05-22T15:43:30.207040+0000","last_became_peered":"2023-05-22T15:43:30.207040+0000","last_unstale":"2023-05-22T15:43:30.207751+0000","last_undegraded":"2023-05-22T15:43:30.207751+0000","last_fullsized":"2023-05-22T15:43:30.207751+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_clean_scrub_stamp":"2023-05-22T15:43:29.738376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T16:34:27.749652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.32","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-05-22T15:43:30.181218+0000","last_change":"2023-05-22T15:43:30.180424+0000","last_active":"2023-05-22T15:43:30.181218+0000","last_peered":"2023-05-22T15:43:30.181218+0000","last_clean":"2023-05-22T15:43:30.181218+0000","last_became_active":"2023-05-22T15:43:30.180064+0000","last_became_peered":"2023-05-22T15:43:30.180064+0000","last_unstale":"2023-05-22T15:43:30.181218+0000","last_undegraded":"2023-05-22T15:43:30.181218+0000","last_fullsized":"2023-05-22T15:43:30.181218+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_clean_scrub_stamp":"2023-05-22T15:43:29.738376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:57:34.706301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.31","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-05-22T15:43:30.171923+0000","last_change":"2023-05-22T15:43:30.169125+0000","last_active":"2023-05-22T15:43:30.171923+0000","last_peered":"2023-05-22T15:43:30.171923+0000","last_clean":"2023-05-22T15:43:30.171923+0000","last_became_active":"2023-05-22T15:43:30.168764+0000","last_became_peered":"2023-05-22T15:43:30.168764+0000","last_unstale":"2023-05-22T15:43:30.171923+0000","last_undegraded":"2023-05-22T15:43:30.171923+0000","last_fullsized":"2023-05-22T15:43:30.171923+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_clean_scrub_stamp":"2023-05-22T15:43:29.738376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:20:06.441496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.30","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-05-22T15:43:30.206221+0000","last_change":"2023-05-22T15:43:30.193444+0000","last_active":"2023-05-22T15:43:30.206221+0000","last_peered":"2023-05-22T15:43:30.206221+0000","last_clean":"2023-05-22T15:43:30.206221+0000","last_became_active":"2023-05-22T15:43:30.192461+0000","last_became_peered":"2023-05-22T15:43:30.192461+0000","last_unstale":"2023-05-22T15:43:30.206221+0000","last_undegraded":"2023-05-22T15:43:30.206221+0000","last_fullsized":"2023-05-22T15:43:30.206221+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_clean_scrub_stamp":"2023-05-22T15:43:29.738376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:58:34.403442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.2f","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-05-22T15:43:30.153787+0000","last_change":"2023-05-22T15:43:30.151800+0000","last_active":"2023-05-22T15:43:30.153787+0000","last_peered":"2023-05-22T15:43:30.153787+0000","last_clean":"2023-05-22T15:43:30.153787+0000","last_became_active":"2023-05-22T15:43:30.151509+0000","last_became_peered":"2023-05-22T15:43:30.151509+0000","last_unstale":"2023-05-22T15:43:30.153787+0000","last_undegraded":"2023-05-22T15:43:30.153787+0000","last_fullsized":"2023-05-22T15:43:30.153787+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_clean_scrub_stamp":"2023-05-22T15:43:29.738376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T18:09:47.059168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.2e","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-05-22T15:43:30.165194+0000","last_change":"2023-05-22T15:43:30.162990+0000","last_active":"2023-05-22T15:43:30.165194+0000","last_peered":"2023-05-22T15:43:30.165194+0000","last_clean":"2023-05-22T15:43:30.165194+0000","last_became_active":"2023-05-22T15:43:30.162647+0000","last_became_peered":"2023-05-22T15:43:30.162647+0000","last_unstale":"2023-05-22T15:43:30.165194+0000","last_undegraded":"2023-05-22T15:43:30.165194+0000","last_fullsized":"2023-05-22T15:43:30.165194+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_clean_scrub_stamp":"2023-05-22T15:43:29.738376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:20:06.684861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.2d","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-05-22T15:43:30.175878+0000","last_change":"2023-05-22T15:43:30.170512+0000","last_active":"2023-05-22T15:43:30.175878+0000","last_peered":"2023-05-22T15:43:30.175878+0000","last_clean":"2023-05-22T15:43:30.175878+0000","last_became_active":"2023-05-22T15:43:30.170187+0000","last_became_peered":"2023-05-22T15:43:30.170187+0000","last_unstale":"2023-05-22T15:43:30.175878+0000","last_undegraded":"2023-05-22T15:43:30.175878+0000","last_fullsized":"2023-05-22T15:43:30.175878+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_clean_scrub_stamp":"2023-05-22T15:43:29.738376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:05:02.019815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2c","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-05-22T15:43:30.223926+0000","last_change":"2023-05-22T15:43:30.209849+0000","last_active":"2023-05-22T15:43:30.223926+0000","last_peered":"2023-05-22T15:43:30.223926+0000","last_clean":"2023-05-22T15:43:30.223926+0000","last_became_active":"2023-05-22T15:43:30.209224+0000","last_became_peered":"2023-05-22T15:43:30.209224+0000","last_unstale":"2023-05-22T15:43:30.223926+0000","last_undegraded":"2023-05-22T15:43:30.223926+0000","last_fullsized":"2023-05-22T15:43:30.223926+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_clean_scrub_stamp":"2023-05-22T15:43:29.738376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:09:50.754411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.2b","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-05-22T15:43:30.197970+0000","last_change":"2023-05-22T15:43:30.197011+0000","last_active":"2023-05-22T15:43:30.197970+0000","last_peered":"2023-05-22T15:43:30.197970+0000","last_clean":"2023-05-22T15:43:30.197970+0000","last_became_active":"2023-05-22T15:43:30.196709+0000","last_became_peered":"2023-05-22T15:43:30.196709+0000","last_unstale":"2023-05-22T15:43:30.197970+0000","last_undegraded":"2023-05-22T15:43:30.197970+0000","last_fullsized":"2023-05-22T15:43:30.197970+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_clean_scrub_stamp":"2023-05-22T15:43:29.738376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T03:33:14.051311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2a","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-05-22T15:43:30.161058+0000","last_change":"2023-05-22T15:43:30.156401+0000","last_active":"2023-05-22T15:43:30.161058+0000","last_peered":"2023-05-22T15:43:30.161058+0000","last_clean":"2023-05-22T15:43:30.161058+0000","last_became_active":"2023-05-22T15:43:30.156023+0000","last_became_peered":"2023-05-22T15:43:30.156023+0000","last_unstale":"2023-05-22T15:43:30.161058+0000","last_undegraded":"2023-05-22T15:43:30.161058+0000","last_fullsized":"2023-05-22T15:43:30.161058+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_clean_scrub_stamp":"2023-05-22T15:43:29.738376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:59:43.398245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.29","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-05-22T15:43:30.161024+0000","last_change":"2023-05-22T15:43:30.155803+0000","last_active":"2023-05-22T15:43:30.161024+0000","last_peered":"2023-05-22T15:43:30.161024+0000","last_clean":"2023-05-22T15:43:30.161024+0000","last_became_active":"2023-05-22T15:43:30.155488+0000","last_became_peered":"2023-05-22T15:43:30.155488+0000","last_unstale":"2023-05-22T15:43:30.161024+0000","last_undegraded":"2023-05-22T15:43:30.161024+0000","last_fullsized":"2023-05-22T15:43:30.161024+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_clean_scrub_stamp":"2023-05-22T15:43:29.738376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:35:30.801478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.28","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-05-22T15:43:30.206165+0000","last_change":"2023-05-22T15:43:30.191603+0000","last_active":"2023-05-22T15:43:30.206165+0000","last_peered":"2023-05-22T15:43:30.206165+0000","last_clean":"2023-05-22T15:43:30.206165+0000","last_became_active":"2023-05-22T15:43:30.190473+0000","last_became_peered":"2023-05-22T15:43:30.190473+0000","last_unstale":"2023-05-22T15:43:30.206165+0000","last_undegraded":"2023-05-22T15:43:30.206165+0000","last_fullsized":"2023-05-22T15:43:30.206165+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_clean_scrub_stamp":"2023-05-22T15:43:29.738376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:48:02.716268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.27","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-05-22T15:43:30.212521+0000","last_change":"2023-05-22T15:43:30.181035+0000","last_active":"2023-05-22T15:43:30.212521+0000","last_peered":"2023-05-22T15:43:30.212521+0000","last_clean":"2023-05-22T15:43:30.212521+0000","last_became_active":"2023-05-22T15:43:30.179544+0000","last_became_peered":"2023-05-22T15:43:30.179544+0000","last_unstale":"2023-05-22T15:43:30.212521+0000","last_undegraded":"2023-05-22T15:43:30.212521+0000","last_fullsized":"2023-05-22T15:43:30.212521+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_clean_scrub_stamp":"2023-05-22T15:43:29.738376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:46:18.725589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.26","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-05-22T15:43:30.172660+0000","last_change":"2023-05-22T15:43:30.168435+0000","last_active":"2023-05-22T15:43:30.172660+0000","last_peered":"2023-05-22T15:43:30.172660+0000","last_clean":"2023-05-22T15:43:30.172660+0000","last_became_active":"2023-05-22T15:43:30.168152+0000","last_became_peered":"2023-05-22T15:43:30.168152+0000","last_unstale":"2023-05-22T15:43:30.172660+0000","last_undegraded":"2023-05-22T15:43:30.172660+0000","last_fullsized":"2023-05-22T15:43:30.172660+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_clean_scrub_stamp":"2023-05-22T15:43:29.738376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:19:08.559088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.25","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-05-22T15:43:30.208217+0000","last_change":"2023-05-22T15:43:30.190690+0000","last_active":"2023-05-22T15:43:30.208217+0000","last_peered":"2023-05-22T15:43:30.208217+0000","last_clean":"2023-05-22T15:43:30.208217+0000","last_became_active":"2023-05-22T15:43:30.190035+0000","last_became_peered":"2023-05-22T15:43:30.190035+0000","last_unstale":"2023-05-22T15:43:30.208217+0000","last_undegraded":"2023-05-22T15:43:30.208217+0000","last_fullsized":"2023-05-22T15:43:30.208217+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_clean_scrub_stamp":"2023-05-22T15:43:29.738376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:18:19.994700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.24","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-05-22T15:43:30.211187+0000","last_change":"2023-05-22T15:43:30.184195+0000","last_active":"2023-05-22T15:43:30.211187+0000","last_peered":"2023-05-22T15:43:30.211187+0000","last_clean":"2023-05-22T15:43:30.211187+0000","last_became_active":"2023-05-22T15:43:30.180854+0000","last_became_peered":"2023-05-22T15:43:30.180854+0000","last_unstale":"2023-05-22T15:43:30.211187+0000","last_undegraded":"2023-05-22T15:43:30.211187+0000","last_fullsized":"2023-05-22T15:43:30.211187+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_clean_scrub_stamp":"2023-05-22T15:43:29.738376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T15:59:51.980766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.23","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-05-22T15:43:30.172717+0000","last_change":"2023-05-22T15:43:30.168398+0000","last_active":"2023-05-22T15:43:30.172717+0000","last_peered":"2023-05-22T15:43:30.172717+0000","last_clean":"2023-05-22T15:43:30.172717+0000","last_became_active":"2023-05-22T15:43:30.168063+0000","last_became_peered":"2023-05-22T15:43:30.168063+0000","last_unstale":"2023-05-22T15:43:30.172717+0000","last_undegraded":"2023-05-22T15:43:30.172717+0000","last_fullsized":"2023-05-22T15:43:30.172717+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_clean_scrub_stamp":"2023-05-22T15:43:29.738376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:46:43.160476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.22","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-05-22T15:43:30.175849+0000","last_change":"2023-05-22T15:43:30.170436+0000","last_active":"2023-05-22T15:43:30.175849+0000","last_peered":"2023-05-22T15:43:30.175849+0000","last_clean":"2023-05-22T15:43:30.175849+0000","last_became_active":"2023-05-22T15:43:30.170139+0000","last_became_peered":"2023-05-22T15:43:30.170139+0000","last_unstale":"2023-05-22T15:43:30.175849+0000","last_undegraded":"2023-05-22T15:43:30.175849+0000","last_fullsized":"2023-05-22T15:43:30.175849+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_clean_scrub_stamp":"2023-05-22T15:43:29.738376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:59:49.774332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.21","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-05-22T15:43:30.186714+0000","last_change":"2023-05-22T15:43:30.185698+0000","last_active":"2023-05-22T15:43:30.186714+0000","last_peered":"2023-05-22T15:43:30.186714+0000","last_clean":"2023-05-22T15:43:30.186714+0000","last_became_active":"2023-05-22T15:43:30.185389+0000","last_became_peered":"2023-05-22T15:43:30.185389+0000","last_unstale":"2023-05-22T15:43:30.186714+0000","last_undegraded":"2023-05-22T15:43:30.186714+0000","last_fullsized":"2023-05-22T15:43:30.186714+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_clean_scrub_stamp":"2023-05-22T15:43:29.738376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:52:20.517587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.20","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-05-22T15:43:30.175792+0000","last_change":"2023-05-22T15:43:30.169903+0000","last_active":"2023-05-22T15:43:30.175792+0000","last_peered":"2023-05-22T15:43:30.175792+0000","last_clean":"2023-05-22T15:43:30.175792+0000","last_became_active":"2023-05-22T15:43:30.169322+0000","last_became_peered":"2023-05-22T15:43:30.169322+0000","last_unstale":"2023-05-22T15:43:30.175792+0000","last_undegraded":"2023-05-22T15:43:30.175792+0000","last_fullsized":"2023-05-22T15:43:30.175792+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_clean_scrub_stamp":"2023-05-22T15:43:29.738376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T16:59:50.668128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1e","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-05-22T15:43:30.203107+0000","last_change":"2023-05-22T15:43:30.202292+0000","last_active":"2023-05-22T15:43:30.203107+0000","last_peered":"2023-05-22T15:43:30.203107+0000","last_clean":"2023-05-22T15:43:30.203107+0000","last_became_active":"2023-05-22T15:43:30.202084+0000","last_became_peered":"2023-05-22T15:43:30.202084+0000","last_unstale":"2023-05-22T15:43:30.203107+0000","last_undegraded":"2023-05-22T15:43:30.203107+0000","last_fullsized":"2023-05-22T15:43:30.203107+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_clean_scrub_stamp":"2023-05-22T15:43:29.738376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T16:59:50.668128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1f","version":"117'4","reported_seq":14,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-05-22T15:43:30.195460+0000","last_change":"2023-05-22T15:43:29.802221+0000","last_active":"2023-05-22T15:43:30.195460+0000","last_peered":"2023-05-22T15:43:30.195460+0000","last_clean":"2023-05-22T15:43:30.195460+0000","last_became_active":"2023-05-22T15:43:29.801774+0000","last_became_peered":"2023-05-22T15:43:29.801774+0000","last_unstale":"2023-05-22T15:43:30.195460+0000","last_undegraded":"2023-05-22T15:43:30.195460+0000","last_fullsized":"2023-05-22T15:43:30.195460+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:43:28.734545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:43:28.734545+0000","last_clean_scrub_stamp":"2023-05-22T15:43:28.734545+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:20:05.437665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":566,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1f","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-05-22T15:43:30.186706+0000","last_change":"2023-05-22T15:43:30.185658+0000","last_active":"2023-05-22T15:43:30.186706+0000","last_peered":"2023-05-22T15:43:30.186706+0000","last_clean":"2023-05-22T15:43:30.186706+0000","last_became_active":"2023-05-22T15:43:30.185349+0000","last_became_peered":"2023-05-22T15:43:30.185349+0000","last_unstale":"2023-05-22T15:43:30.186706+0000","last_undegraded":"2023-05-22T15:43:30.186706+0000","last_fullsized":"2023-05-22T15:43:30.186706+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_clean_scrub_stamp":"2023-05-22T15:43:29.738376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:09:50.754411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.1e","version":"117'1","reported_seq":11,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-05-22T15:43:30.202739+0000","last_change":"2023-05-22T15:43:29.803088+0000","last_active":"2023-05-22T15:43:30.202739+0000","last_peered":"2023-05-22T15:43:30.202739+0000","last_clean":"2023-05-22T15:43:30.202739+0000","last_became_active":"2023-05-22T15:43:29.802113+0000","last_became_peered":"2023-05-22T15:43:29.802113+0000","last_unstale":"2023-05-22T15:43:30.202739+0000","last_undegraded":"2023-05-22T15:43:30.202739+0000","last_fullsized":"2023-05-22T15:43:30.202739+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:43:28.734545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:43:28.734545+0000","last_clean_scrub_stamp":"2023-05-22T15:43:28.734545+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:12:32.488160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.1c","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-05-22T15:43:30.211272+0000","last_change":"2023-05-22T15:43:30.209315+0000","last_active":"2023-05-22T15:43:30.211272+0000","last_peered":"2023-05-22T15:43:30.211272+0000","last_clean":"2023-05-22T15:43:30.211272+0000","last_became_active":"2023-05-22T15:43:30.209018+0000","last_became_peered":"2023-05-22T15:43:30.209018+0000","last_unstale":"2023-05-22T15:43:30.211272+0000","last_undegraded":"2023-05-22T15:43:30.211272+0000","last_fullsized":"2023-05-22T15:43:30.211272+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_clean_scrub_stamp":"2023-05-22T15:43:29.738376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:12:33.491991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.1d","version":"117'2","reported_seq":12,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-05-22T15:43:30.170842+0000","last_change":"2023-05-22T15:43:29.778644+0000","last_active":"2023-05-22T15:43:30.170842+0000","last_peered":"2023-05-22T15:43:30.170842+0000","last_clean":"2023-05-22T15:43:30.170842+0000","last_became_active":"2023-05-22T15:43:29.778325+0000","last_became_peered":"2023-05-22T15:43:29.778325+0000","last_unstale":"2023-05-22T15:43:30.170842+0000","last_undegraded":"2023-05-22T15:43:30.170842+0000","last_fullsized":"2023-05-22T15:43:30.170842+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:43:28.734545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:43:28.734545+0000","last_clean_scrub_stamp":"2023-05-22T15:43:28.734545+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:18:00.068884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1d","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-05-22T15:43:30.223688+0000","last_change":"2023-05-22T15:43:30.205970+0000","last_active":"2023-05-22T15:43:30.223688+0000","last_peered":"2023-05-22T15:43:30.223688+0000","last_clean":"2023-05-22T15:43:30.223688+0000","last_became_active":"2023-05-22T15:43:30.205168+0000","last_became_peered":"2023-05-22T15:43:30.205168+0000","last_unstale":"2023-05-22T15:43:30.223688+0000","last_undegraded":"2023-05-22T15:43:30.223688+0000","last_fullsized":"2023-05-22T15:43:30.223688+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_clean_scrub_stamp":"2023-05-22T15:43:29.738376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:20:06.684861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.1c","version":"117'2","reported_seq":12,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-05-22T15:43:30.217505+0000","last_change":"2023-05-22T15:43:29.807740+0000","last_active":"2023-05-22T15:43:30.217505+0000","last_peered":"2023-05-22T15:43:30.217505+0000","last_clean":"2023-05-22T15:43:30.217505+0000","last_became_active":"2023-05-22T15:43:29.807280+0000","last_became_peered":"2023-05-22T15:43:29.807280+0000","last_unstale":"2023-05-22T15:43:30.217505+0000","last_undegraded":"2023-05-22T15:43:30.217505+0000","last_fullsized":"2023-05-22T15:43:30.217505+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:43:28.734545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:43:28.734545+0000","last_clean_scrub_stamp":"2023-05-22T15:43:28.734545+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:09:49.750580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1a","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-05-22T15:43:30.215960+0000","last_change":"2023-05-22T15:43:30.211412+0000","last_active":"2023-05-22T15:43:30.215960+0000","last_peered":"2023-05-22T15:43:30.215960+0000","last_clean":"2023-05-22T15:43:30.215960+0000","last_became_active":"2023-05-22T15:43:30.210786+0000","last_became_peered":"2023-05-22T15:43:30.210786+0000","last_unstale":"2023-05-22T15:43:30.215960+0000","last_undegraded":"2023-05-22T15:43:30.215960+0000","last_fullsized":"2023-05-22T15:43:30.215960+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_clean_scrub_stamp":"2023-05-22T15:43:29.738376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T16:59:50.668128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.1b","version":"117'1","reported_seq":11,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-05-22T15:43:30.197240+0000","last_change":"2023-05-22T15:43:29.789519+0000","last_active":"2023-05-22T15:43:30.197240+0000","last_peered":"2023-05-22T15:43:30.197240+0000","last_clean":"2023-05-22T15:43:30.197240+0000","last_became_active":"2023-05-22T15:43:29.789209+0000","last_became_peered":"2023-05-22T15:43:29.789209+0000","last_unstale":"2023-05-22T15:43:30.197240+0000","last_undegraded":"2023-05-22T15:43:30.197240+0000","last_fullsized":"2023-05-22T15:43:30.197240+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:43:28.734545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:43:28.734545+0000","last_clean_scrub_stamp":"2023-05-22T15:43:28.734545+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:45:12.725224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1b","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-05-22T15:43:30.206262+0000","last_change":"2023-05-22T15:43:30.199558+0000","last_active":"2023-05-22T15:43:30.206262+0000","last_peered":"2023-05-22T15:43:30.206262+0000","last_clean":"2023-05-22T15:43:30.206262+0000","last_became_active":"2023-05-22T15:43:30.197829+0000","last_became_peered":"2023-05-22T15:43:30.197829+0000","last_unstale":"2023-05-22T15:43:30.206262+0000","last_undegraded":"2023-05-22T15:43:30.206262+0000","last_fullsized":"2023-05-22T15:43:30.206262+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_clean_scrub_stamp":"2023-05-22T15:43:29.738376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T03:40:58.697755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.1a","version":"117'1","reported_seq":11,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-05-22T15:43:30.165399+0000","last_change":"2023-05-22T15:43:29.786409+0000","last_active":"2023-05-22T15:43:30.165399+0000","last_peered":"2023-05-22T15:43:30.165399+0000","last_clean":"2023-05-22T15:43:30.165399+0000","last_became_active":"2023-05-22T15:43:29.786082+0000","last_became_peered":"2023-05-22T15:43:29.786082+0000","last_unstale":"2023-05-22T15:43:30.165399+0000","last_undegraded":"2023-05-22T15:43:30.165399+0000","last_fullsized":"2023-05-22T15:43:30.165399+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:43:28.734545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:43:28.734545+0000","last_clean_scrub_stamp":"2023-05-22T15:43:28.734545+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:33:59.380284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.18","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-05-22T15:43:30.172181+0000","last_change":"2023-05-22T15:43:30.169856+0000","last_active":"2023-05-22T15:43:30.172181+0000","last_peered":"2023-05-22T15:43:30.172181+0000","last_clean":"2023-05-22T15:43:30.172181+0000","last_became_active":"2023-05-22T15:43:30.169581+0000","last_became_peered":"2023-05-22T15:43:30.169581+0000","last_unstale":"2023-05-22T15:43:30.172181+0000","last_undegraded":"2023-05-22T15:43:30.172181+0000","last_fullsized":"2023-05-22T15:43:30.172181+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_clean_scrub_stamp":"2023-05-22T15:43:29.738376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:36:26.720817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.19","version":"0'0","reported_seq":10,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-05-22T15:43:30.171375+0000","last_change":"2023-05-22T15:43:29.790291+0000","last_active":"2023-05-22T15:43:30.171375+0000","last_peered":"2023-05-22T15:43:30.171375+0000","last_clean":"2023-05-22T15:43:30.171375+0000","last_became_active":"2023-05-22T15:43:29.789473+0000","last_became_peered":"2023-05-22T15:43:29.789473+0000","last_unstale":"2023-05-22T15:43:30.171375+0000","last_undegraded":"2023-05-22T15:43:30.171375+0000","last_fullsized":"2023-05-22T15:43:30.171375+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:43:28.734545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:43:28.734545+0000","last_clean_scrub_stamp":"2023-05-22T15:43:28.734545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T03:33:13.047480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.19","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-05-22T15:43:30.223768+0000","last_change":"2023-05-22T15:43:30.206409+0000","last_active":"2023-05-22T15:43:30.223768+0000","last_peered":"2023-05-22T15:43:30.223768+0000","last_clean":"2023-05-22T15:43:30.223768+0000","last_became_active":"2023-05-22T15:43:30.205299+0000","last_became_peered":"2023-05-22T15:43:30.205299+0000","last_unstale":"2023-05-22T15:43:30.223768+0000","last_undegraded":"2023-05-22T15:43:30.223768+0000","last_fullsized":"2023-05-22T15:43:30.223768+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_clean_scrub_stamp":"2023-05-22T15:43:29.738376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:18:19.994700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.18","version":"0'0","reported_seq":10,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-05-22T15:43:30.156027+0000","last_change":"2023-05-22T15:43:29.776421+0000","last_active":"2023-05-22T15:43:30.156027+0000","last_peered":"2023-05-22T15:43:30.156027+0000","last_clean":"2023-05-22T15:43:30.156027+0000","last_became_active":"2023-05-22T15:43:29.776114+0000","last_became_peered":"2023-05-22T15:43:29.776114+0000","last_unstale":"2023-05-22T15:43:30.156027+0000","last_undegraded":"2023-05-22T15:43:30.156027+0000","last_fullsized":"2023-05-22T15:43:30.156027+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:43:28.734545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:43:28.734545+0000","last_clean_scrub_stamp":"2023-05-22T15:43:28.734545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:29:55.479369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.16","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-05-22T15:43:30.205067+0000","last_change":"2023-05-22T15:43:30.204449+0000","last_active":"2023-05-22T15:43:30.205067+0000","last_peered":"2023-05-22T15:43:30.205067+0000","last_clean":"2023-05-22T15:43:30.205067+0000","last_became_active":"2023-05-22T15:43:30.204175+0000","last_became_peered":"2023-05-22T15:43:30.204175+0000","last_unstale":"2023-05-22T15:43:30.205067+0000","last_undegraded":"2023-05-22T15:43:30.205067+0000","last_fullsized":"2023-05-22T15:43:30.205067+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_clean_scrub_stamp":"2023-05-22T15:43:29.738376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:38:19.788691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.17","version":"0'0","reported_seq":10,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-05-22T15:43:30.158574+0000","last_change":"2023-05-22T15:43:29.791430+0000","last_active":"2023-05-22T15:43:30.158574+0000","last_peered":"2023-05-22T15:43:30.158574+0000","last_clean":"2023-05-22T15:43:30.158574+0000","last_became_active":"2023-05-22T15:43:29.791156+0000","last_became_peered":"2023-05-22T15:43:29.791156+0000","last_unstale":"2023-05-22T15:43:30.158574+0000","last_undegraded":"2023-05-22T15:43:30.158574+0000","last_fullsized":"2023-05-22T15:43:30.158574+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:43:28.734545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:43:28.734545+0000","last_clean_scrub_stamp":"2023-05-22T15:43:28.734545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:13:19.208630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.17","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-05-22T15:43:30.223768+0000","last_change":"2023-05-22T15:43:30.206797+0000","last_active":"2023-05-22T15:43:30.223768+0000","last_peered":"2023-05-22T15:43:30.223768+0000","last_clean":"2023-05-22T15:43:30.223768+0000","last_became_active":"2023-05-22T15:43:30.206222+0000","last_became_peered":"2023-05-22T15:43:30.206222+0000","last_unstale":"2023-05-22T15:43:30.223768+0000","last_undegraded":"2023-05-22T15:43:30.223768+0000","last_fullsized":"2023-05-22T15:43:30.223768+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_clean_scrub_stamp":"2023-05-22T15:43:29.738376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:20:06.441496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.16","version":"0'0","reported_seq":10,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-05-22T15:43:30.148238+0000","last_change":"2023-05-22T15:43:29.785747+0000","last_active":"2023-05-22T15:43:30.148238+0000","last_peered":"2023-05-22T15:43:30.148238+0000","last_clean":"2023-05-22T15:43:30.148238+0000","last_became_active":"2023-05-22T15:43:29.785179+0000","last_became_peered":"2023-05-22T15:43:29.785179+0000","last_unstale":"2023-05-22T15:43:30.148238+0000","last_undegraded":"2023-05-22T15:43:30.148238+0000","last_fullsized":"2023-05-22T15:43:30.148238+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:43:28.734545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:43:28.734545+0000","last_clean_scrub_stamp":"2023-05-22T15:43:28.734545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T15:59:50.976935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.14","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-05-22T15:43:30.196588+0000","last_change":"2023-05-22T15:43:30.169946+0000","last_active":"2023-05-22T15:43:30.196588+0000","last_peered":"2023-05-22T15:43:30.196588+0000","last_clean":"2023-05-22T15:43:30.196588+0000","last_became_active":"2023-05-22T15:43:30.169159+0000","last_became_peered":"2023-05-22T15:43:30.169159+0000","last_unstale":"2023-05-22T15:43:30.196588+0000","last_undegraded":"2023-05-22T15:43:30.196588+0000","last_fullsized":"2023-05-22T15:43:30.196588+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_clean_scrub_stamp":"2023-05-22T15:43:29.738376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:30:29.500628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.15","version":"117'3","reported_seq":13,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-05-22T15:43:30.220527+0000","last_change":"2023-05-22T15:43:29.810062+0000","last_active":"2023-05-22T15:43:30.220527+0000","last_peered":"2023-05-22T15:43:30.220527+0000","last_clean":"2023-05-22T15:43:30.220527+0000","last_became_active":"2023-05-22T15:43:29.809287+0000","last_became_peered":"2023-05-22T15:43:29.809287+0000","last_unstale":"2023-05-22T15:43:30.220527+0000","last_undegraded":"2023-05-22T15:43:30.220527+0000","last_fullsized":"2023-05-22T15:43:30.220527+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:43:28.734545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:43:28.734545+0000","last_clean_scrub_stamp":"2023-05-22T15:43:28.734545+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:44:15.881912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.15","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-05-22T15:43:30.208267+0000","last_change":"2023-05-22T15:43:30.191994+0000","last_active":"2023-05-22T15:43:30.208267+0000","last_peered":"2023-05-22T15:43:30.208267+0000","last_clean":"2023-05-22T15:43:30.208267+0000","last_became_active":"2023-05-22T15:43:30.191302+0000","last_became_peered":"2023-05-22T15:43:30.191302+0000","last_unstale":"2023-05-22T15:43:30.208267+0000","last_undegraded":"2023-05-22T15:43:30.208267+0000","last_fullsized":"2023-05-22T15:43:30.208267+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_clean_scrub_stamp":"2023-05-22T15:43:29.738376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:18:01.072715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.14","version":"117'6","reported_seq":16,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-05-22T15:43:30.225024+0000","last_change":"2023-05-22T15:43:29.807258+0000","last_active":"2023-05-22T15:43:30.225024+0000","last_peered":"2023-05-22T15:43:30.225024+0000","last_clean":"2023-05-22T15:43:30.225024+0000","last_became_active":"2023-05-22T15:43:29.806533+0000","last_became_peered":"2023-05-22T15:43:29.806533+0000","last_unstale":"2023-05-22T15:43:30.225024+0000","last_undegraded":"2023-05-22T15:43:30.225024+0000","last_fullsized":"2023-05-22T15:43:30.225024+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:43:28.734545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:43:28.734545+0000","last_clean_scrub_stamp":"2023-05-22T15:43:28.734545+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T03:40:57.693924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1548,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.12","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-05-22T15:43:30.206352+0000","last_change":"2023-05-22T15:43:30.205777+0000","last_active":"2023-05-22T15:43:30.206352+0000","last_peered":"2023-05-22T15:43:30.206352+0000","last_clean":"2023-05-22T15:43:30.206352+0000","last_became_active":"2023-05-22T15:43:30.205565+0000","last_became_peered":"2023-05-22T15:43:30.205565+0000","last_unstale":"2023-05-22T15:43:30.206352+0000","last_undegraded":"2023-05-22T15:43:30.206352+0000","last_fullsized":"2023-05-22T15:43:30.206352+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_clean_scrub_stamp":"2023-05-22T15:43:29.738376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T16:53:26.129417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.13","version":"117'2","reported_seq":12,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-05-22T15:43:30.206194+0000","last_change":"2023-05-22T15:43:29.806812+0000","last_active":"2023-05-22T15:43:30.206194+0000","last_peered":"2023-05-22T15:43:30.206194+0000","last_clean":"2023-05-22T15:43:30.206194+0000","last_became_active":"2023-05-22T15:43:29.806402+0000","last_became_peered":"2023-05-22T15:43:29.806402+0000","last_unstale":"2023-05-22T15:43:30.206194+0000","last_undegraded":"2023-05-22T15:43:30.206194+0000","last_fullsized":"2023-05-22T15:43:30.206194+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:43:28.734545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:43:28.734545+0000","last_clean_scrub_stamp":"2023-05-22T15:43:28.734545+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T18:18:31.388215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.13","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-05-22T15:43:30.173056+0000","last_change":"2023-05-22T15:43:30.168305+0000","last_active":"2023-05-22T15:43:30.173056+0000","last_peered":"2023-05-22T15:43:30.173056+0000","last_clean":"2023-05-22T15:43:30.173056+0000","last_became_active":"2023-05-22T15:43:30.167986+0000","last_became_peered":"2023-05-22T15:43:30.167986+0000","last_unstale":"2023-05-22T15:43:30.173056+0000","last_undegraded":"2023-05-22T15:43:30.173056+0000","last_fullsized":"2023-05-22T15:43:30.173056+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_clean_scrub_stamp":"2023-05-22T15:43:29.738376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T03:33:14.051311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.12","version":"0'0","reported_seq":10,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-05-22T15:43:30.157575+0000","last_change":"2023-05-22T15:43:29.776238+0000","last_active":"2023-05-22T15:43:30.157575+0000","last_peered":"2023-05-22T15:43:30.157575+0000","last_clean":"2023-05-22T15:43:30.157575+0000","last_became_active":"2023-05-22T15:43:29.775974+0000","last_became_peered":"2023-05-22T15:43:29.775974+0000","last_unstale":"2023-05-22T15:43:30.157575+0000","last_undegraded":"2023-05-22T15:43:30.157575+0000","last_fullsized":"2023-05-22T15:43:30.157575+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:43:28.734545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:43:28.734545+0000","last_clean_scrub_stamp":"2023-05-22T15:43:28.734545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:45:03.727507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.10","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-05-22T15:43:30.223699+0000","last_change":"2023-05-22T15:43:30.206181+0000","last_active":"2023-05-22T15:43:30.223699+0000","last_peered":"2023-05-22T15:43:30.223699+0000","last_clean":"2023-05-22T15:43:30.223699+0000","last_became_active":"2023-05-22T15:43:30.205355+0000","last_became_peered":"2023-05-22T15:43:30.205355+0000","last_unstale":"2023-05-22T15:43:30.223699+0000","last_undegraded":"2023-05-22T15:43:30.223699+0000","last_fullsized":"2023-05-22T15:43:30.223699+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_clean_scrub_stamp":"2023-05-22T15:43:29.738376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:00:19.974693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.11","version":"0'0","reported_seq":10,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-05-22T15:43:30.181766+0000","last_change":"2023-05-22T15:43:29.800529+0000","last_active":"2023-05-22T15:43:30.181766+0000","last_peered":"2023-05-22T15:43:30.181766+0000","last_clean":"2023-05-22T15:43:30.181766+0000","last_became_active":"2023-05-22T15:43:29.799819+0000","last_became_peered":"2023-05-22T15:43:29.799819+0000","last_unstale":"2023-05-22T15:43:30.181766+0000","last_undegraded":"2023-05-22T15:43:30.181766+0000","last_fullsized":"2023-05-22T15:43:30.181766+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:43:28.734545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:43:28.734545+0000","last_clean_scrub_stamp":"2023-05-22T15:43:28.734545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:19:18.535583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.11","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-05-22T15:43:30.211039+0000","last_change":"2023-05-22T15:43:30.209771+0000","last_active":"2023-05-22T15:43:30.211039+0000","last_peered":"2023-05-22T15:43:30.211039+0000","last_clean":"2023-05-22T15:43:30.211039+0000","last_became_active":"2023-05-22T15:43:30.209482+0000","last_became_peered":"2023-05-22T15:43:30.209482+0000","last_unstale":"2023-05-22T15:43:30.211039+0000","last_undegraded":"2023-05-22T15:43:30.211039+0000","last_fullsized":"2023-05-22T15:43:30.211039+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_clean_scrub_stamp":"2023-05-22T15:43:29.738376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:44:35.350206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.10","version":"117'8","reported_seq":18,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-05-22T15:43:30.203024+0000","last_change":"2023-05-22T15:43:29.798352+0000","last_active":"2023-05-22T15:43:30.203024+0000","last_peered":"2023-05-22T15:43:30.203024+0000","last_clean":"2023-05-22T15:43:30.203024+0000","last_became_active":"2023-05-22T15:43:29.798066+0000","last_became_peered":"2023-05-22T15:43:29.798066+0000","last_unstale":"2023-05-22T15:43:30.203024+0000","last_undegraded":"2023-05-22T15:43:30.203024+0000","last_fullsized":"2023-05-22T15:43:30.203024+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:43:28.734545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:43:28.734545+0000","last_clean_scrub_stamp":"2023-05-22T15:43:28.734545+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:27:48.345151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.6","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-05-22T15:43:30.164679+0000","last_change":"2023-05-22T15:43:30.163320+0000","last_active":"2023-05-22T15:43:30.164679+0000","last_peered":"2023-05-22T15:43:30.164679+0000","last_clean":"2023-05-22T15:43:30.164679+0000","last_became_active":"2023-05-22T15:43:30.163046+0000","last_became_peered":"2023-05-22T15:43:30.163046+0000","last_unstale":"2023-05-22T15:43:30.164679+0000","last_undegraded":"2023-05-22T15:43:30.164679+0000","last_fullsized":"2023-05-22T15:43:30.164679+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_clean_scrub_stamp":"2023-05-22T15:43:29.738376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:50:18.281158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.7","version":"117'2","reported_seq":12,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-05-22T15:43:30.211222+0000","last_change":"2023-05-22T15:43:29.808187+0000","last_active":"2023-05-22T15:43:30.211222+0000","last_peered":"2023-05-22T15:43:30.211222+0000","last_clean":"2023-05-22T15:43:30.211222+0000","last_became_active":"2023-05-22T15:43:29.807549+0000","last_became_peered":"2023-05-22T15:43:29.807549+0000","last_unstale":"2023-05-22T15:43:30.211222+0000","last_undegraded":"2023-05-22T15:43:30.211222+0000","last_fullsized":"2023-05-22T15:43:30.211222+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:43:28.734545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:43:28.734545+0000","last_clean_scrub_stamp":"2023-05-22T15:43:28.734545+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:08:05.729807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.7","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-05-22T15:43:30.173040+0000","last_change":"2023-05-22T15:43:30.168243+0000","last_active":"2023-05-22T15:43:30.173040+0000","last_peered":"2023-05-22T15:43:30.173040+0000","last_clean":"2023-05-22T15:43:30.173040+0000","last_became_active":"2023-05-22T15:43:30.167889+0000","last_became_peered":"2023-05-22T15:43:30.167889+0000","last_unstale":"2023-05-22T15:43:30.173040+0000","last_undegraded":"2023-05-22T15:43:30.173040+0000","last_fullsized":"2023-05-22T15:43:30.173040+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_clean_scrub_stamp":"2023-05-22T15:43:29.738376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:59:49.774332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.6","version":"117'1","reported_seq":11,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-05-22T15:43:30.597094+0000","last_change":"2023-05-22T15:43:29.784036+0000","last_active":"2023-05-22T15:43:30.597094+0000","last_peered":"2023-05-22T15:43:30.597094+0000","last_clean":"2023-05-22T15:43:30.597094+0000","last_became_active":"2023-05-22T15:43:29.783744+0000","last_became_peered":"2023-05-22T15:43:29.783744+0000","last_unstale":"2023-05-22T15:43:30.597094+0000","last_undegraded":"2023-05-22T15:43:30.597094+0000","last_fullsized":"2023-05-22T15:43:30.597094+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:43:28.734545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:43:28.734545+0000","last_clean_scrub_stamp":"2023-05-22T15:43:28.734545+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:04:27.414328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.4","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-05-22T15:43:30.198927+0000","last_change":"2023-05-22T15:43:30.197767+0000","last_active":"2023-05-22T15:43:30.198927+0000","last_peered":"2023-05-22T15:43:30.198927+0000","last_clean":"2023-05-22T15:43:30.198927+0000","last_became_active":"2023-05-22T15:43:30.197470+0000","last_became_peered":"2023-05-22T15:43:30.197470+0000","last_unstale":"2023-05-22T15:43:30.198927+0000","last_undegraded":"2023-05-22T15:43:30.198927+0000","last_fullsized":"2023-05-22T15:43:30.198927+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_clean_scrub_stamp":"2023-05-22T15:43:29.738376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:05:33.735688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.5","version":"0'0","reported_seq":10,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-05-22T15:43:30.147025+0000","last_change":"2023-05-22T15:43:29.806276+0000","last_active":"2023-05-22T15:43:30.147025+0000","last_peered":"2023-05-22T15:43:30.147025+0000","last_clean":"2023-05-22T15:43:30.147025+0000","last_became_active":"2023-05-22T15:43:29.805860+0000","last_became_peered":"2023-05-22T15:43:29.805860+0000","last_unstale":"2023-05-22T15:43:30.147025+0000","last_undegraded":"2023-05-22T15:43:30.147025+0000","last_fullsized":"2023-05-22T15:43:30.147025+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:43:28.734545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:43:28.734545+0000","last_clean_scrub_stamp":"2023-05-22T15:43:28.734545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:38:14.545852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.5","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-05-22T15:43:30.215694+0000","last_change":"2023-05-22T15:43:30.199769+0000","last_active":"2023-05-22T15:43:30.215694+0000","last_peered":"2023-05-22T15:43:30.215694+0000","last_clean":"2023-05-22T15:43:30.215694+0000","last_became_active":"2023-05-22T15:43:30.199109+0000","last_became_peered":"2023-05-22T15:43:30.199109+0000","last_unstale":"2023-05-22T15:43:30.215694+0000","last_undegraded":"2023-05-22T15:43:30.215694+0000","last_fullsized":"2023-05-22T15:43:30.215694+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_clean_scrub_stamp":"2023-05-22T15:43:29.738376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:47:24.012002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.4","version":"0'0","reported_seq":10,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-05-22T15:43:30.159908+0000","last_change":"2023-05-22T15:43:29.790575+0000","last_active":"2023-05-22T15:43:30.159908+0000","last_peered":"2023-05-22T15:43:30.159908+0000","last_clean":"2023-05-22T15:43:30.159908+0000","last_became_active":"2023-05-22T15:43:29.789764+0000","last_became_peered":"2023-05-22T15:43:29.789764+0000","last_unstale":"2023-05-22T15:43:30.159908+0000","last_undegraded":"2023-05-22T15:43:30.159908+0000","last_fullsized":"2023-05-22T15:43:30.159908+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:43:28.734545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:43:28.734545+0000","last_clean_scrub_stamp":"2023-05-22T15:43:28.734545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:05:01.015984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.2","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-05-22T15:43:30.211007+0000","last_change":"2023-05-22T15:43:30.209732+0000","last_active":"2023-05-22T15:43:30.211007+0000","last_peered":"2023-05-22T15:43:30.211007+0000","last_clean":"2023-05-22T15:43:30.211007+0000","last_became_active":"2023-05-22T15:43:30.209409+0000","last_became_peered":"2023-05-22T15:43:30.209409+0000","last_unstale":"2023-05-22T15:43:30.211007+0000","last_undegraded":"2023-05-22T15:43:30.211007+0000","last_fullsized":"2023-05-22T15:43:30.211007+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_clean_scrub_stamp":"2023-05-22T15:43:29.738376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:08:06.733638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.3","version":"0'0","reported_seq":10,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-05-22T15:43:30.145467+0000","last_change":"2023-05-22T15:43:29.781583+0000","last_active":"2023-05-22T15:43:30.145467+0000","last_peered":"2023-05-22T15:43:30.145467+0000","last_clean":"2023-05-22T15:43:30.145467+0000","last_became_active":"2023-05-22T15:43:29.781276+0000","last_became_peered":"2023-05-22T15:43:29.781276+0000","last_unstale":"2023-05-22T15:43:30.145467+0000","last_undegraded":"2023-05-22T15:43:30.145467+0000","last_fullsized":"2023-05-22T15:43:30.145467+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:43:28.734545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:43:28.734545+0000","last_clean_scrub_stamp":"2023-05-22T15:43:28.734545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:20:56.145983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.3","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-05-22T15:43:30.223804+0000","last_change":"2023-05-22T15:43:30.208056+0000","last_active":"2023-05-22T15:43:30.223804+0000","last_peered":"2023-05-22T15:43:30.223804+0000","last_clean":"2023-05-22T15:43:30.223804+0000","last_became_active":"2023-05-22T15:43:30.207454+0000","last_became_peered":"2023-05-22T15:43:30.207454+0000","last_unstale":"2023-05-22T15:43:30.223804+0000","last_undegraded":"2023-05-22T15:43:30.223804+0000","last_fullsized":"2023-05-22T15:43:30.223804+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_clean_scrub_stamp":"2023-05-22T15:43:29.738376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:52:20.517587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2","version":"0'0","reported_seq":10,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-05-22T15:43:30.156924+0000","last_change":"2023-05-22T15:43:29.779390+0000","last_active":"2023-05-22T15:43:30.156924+0000","last_peered":"2023-05-22T15:43:30.156924+0000","last_clean":"2023-05-22T15:43:30.156924+0000","last_became_active":"2023-05-22T15:43:29.778569+0000","last_became_peered":"2023-05-22T15:43:29.778569+0000","last_unstale":"2023-05-22T15:43:30.156924+0000","last_undegraded":"2023-05-22T15:43:30.156924+0000","last_fullsized":"2023-05-22T15:43:30.156924+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:43:28.734545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:43:28.734545+0000","last_clean_scrub_stamp":"2023-05-22T15:43:28.734545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:59:48.770501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.0","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-05-22T15:43:30.211054+0000","last_change":"2023-05-22T15:43:30.210031+0000","last_active":"2023-05-22T15:43:30.211054+0000","last_peered":"2023-05-22T15:43:30.211054+0000","last_clean":"2023-05-22T15:43:30.211054+0000","last_became_active":"2023-05-22T15:43:30.209727+0000","last_became_peered":"2023-05-22T15:43:30.209727+0000","last_unstale":"2023-05-22T15:43:30.211054+0000","last_undegraded":"2023-05-22T15:43:30.211054+0000","last_fullsized":"2023-05-22T15:43:30.211054+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_clean_scrub_stamp":"2023-05-22T15:43:29.738376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:05:33.735688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1","version":"0'0","reported_seq":10,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-05-22T15:43:30.148534+0000","last_change":"2023-05-22T15:43:29.776927+0000","last_active":"2023-05-22T15:43:30.148534+0000","last_peered":"2023-05-22T15:43:30.148534+0000","last_clean":"2023-05-22T15:43:30.148534+0000","last_became_active":"2023-05-22T15:43:29.776007+0000","last_became_peered":"2023-05-22T15:43:29.776007+0000","last_unstale":"2023-05-22T15:43:30.148534+0000","last_undegraded":"2023-05-22T15:43:30.148534+0000","last_fullsized":"2023-05-22T15:43:30.148534+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:43:28.734545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:43:28.734545+0000","last_clean_scrub_stamp":"2023-05-22T15:43:28.734545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:47:23.008171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.1","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-05-22T15:43:30.211077+0000","last_change":"2023-05-22T15:43:30.178779+0000","last_active":"2023-05-22T15:43:30.211077+0000","last_peered":"2023-05-22T15:43:30.211077+0000","last_clean":"2023-05-22T15:43:30.211077+0000","last_became_active":"2023-05-22T15:43:30.177907+0000","last_became_peered":"2023-05-22T15:43:30.177907+0000","last_unstale":"2023-05-22T15:43:30.211077+0000","last_undegraded":"2023-05-22T15:43:30.211077+0000","last_fullsized":"2023-05-22T15:43:30.211077+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_clean_scrub_stamp":"2023-05-22T15:43:29.738376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T03:40:58.697755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.0","version":"0'0","reported_seq":10,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-05-22T15:43:30.150999+0000","last_change":"2023-05-22T15:43:29.786297+0000","last_active":"2023-05-22T15:43:30.150999+0000","last_peered":"2023-05-22T15:43:30.150999+0000","last_clean":"2023-05-22T15:43:30.150999+0000","last_became_active":"2023-05-22T15:43:29.786016+0000","last_became_peered":"2023-05-22T15:43:29.786016+0000","last_unstale":"2023-05-22T15:43:30.150999+0000","last_undegraded":"2023-05-22T15:43:30.150999+0000","last_fullsized":"2023-05-22T15:43:30.150999+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:43:28.734545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:43:28.734545+0000","last_clean_scrub_stamp":"2023-05-22T15:43:28.734545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T16:05:36.115819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.e","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-05-22T15:43:30.174625+0000","last_change":"2023-05-22T15:43:30.168122+0000","last_active":"2023-05-22T15:43:30.174625+0000","last_peered":"2023-05-22T15:43:30.174625+0000","last_clean":"2023-05-22T15:43:30.174625+0000","last_became_active":"2023-05-22T15:43:30.166491+0000","last_became_peered":"2023-05-22T15:43:30.166491+0000","last_unstale":"2023-05-22T15:43:30.174625+0000","last_undegraded":"2023-05-22T15:43:30.174625+0000","last_fullsized":"2023-05-22T15:43:30.174625+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_clean_scrub_stamp":"2023-05-22T15:43:29.738376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:45:35.107137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.f","version":"117'1","reported_seq":11,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-05-22T15:43:30.202749+0000","last_change":"2023-05-22T15:43:29.797109+0000","last_active":"2023-05-22T15:43:30.202749+0000","last_peered":"2023-05-22T15:43:30.202749+0000","last_clean":"2023-05-22T15:43:30.202749+0000","last_became_active":"2023-05-22T15:43:29.796390+0000","last_became_peered":"2023-05-22T15:43:29.796390+0000","last_unstale":"2023-05-22T15:43:30.202749+0000","last_undegraded":"2023-05-22T15:43:30.202749+0000","last_fullsized":"2023-05-22T15:43:30.202749+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:43:28.734545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:43:28.734545+0000","last_clean_scrub_stamp":"2023-05-22T15:43:28.734545+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T18:41:58.613554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"106'205","reported_seq":570,"reported_epoch":116,"state":"active+clean","last_fresh":"2023-05-22T15:43:29.786247+0000","last_change":"2023-05-22T15:26:39.318204+0000","last_active":"2023-05-22T15:43:29.786247+0000","last_peered":"2023-05-22T15:43:29.786247+0000","last_clean":"2023-05-22T15:43:29.786247+0000","last_became_active":"2023-05-22T15:26:39.315370+0000","last_became_peered":"2023-05-22T15:26:39.315370+0000","last_unstale":"2023-05-22T15:43:29.786247+0000","last_undegraded":"2023-05-22T15:43:29.786247+0000","last_fullsized":"2023-05-22T15:43:29.786247+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:38.020076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:38.020076+0000","last_clean_scrub_stamp":"2023-05-22T15:26:38.020076+0000","objects_scrubbed":0,"log_size":205,"log_dups_size":0,"ondisk_log_size":205,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:06:39.207543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":884,"num_read_kb":1748,"num_write":319,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":111,"reported_epoch":116,"state":"active+clean","last_fresh":"2023-05-22T15:43:29.778340+0000","last_change":"2023-05-22T15:26:44.971951+0000","last_active":"2023-05-22T15:43:29.778340+0000","last_peered":"2023-05-22T15:43:29.778340+0000","last_clean":"2023-05-22T15:43:29.778340+0000","last_became_active":"2023-05-22T15:26:44.970872+0000","last_became_peered":"2023-05-22T15:26:44.970872+0000","last_unstale":"2023-05-22T15:43:29.778340+0000","last_undegraded":"2023-05-22T15:43:29.778340+0000","last_fullsized":"2023-05-22T15:43:29.778340+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:55:34.752949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.d","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-05-22T15:43:30.222151+0000","last_change":"2023-05-22T15:43:30.220279+0000","last_active":"2023-05-22T15:43:30.222151+0000","last_peered":"2023-05-22T15:43:30.222151+0000","last_clean":"2023-05-22T15:43:30.222151+0000","last_became_active":"2023-05-22T15:43:30.219191+0000","last_became_peered":"2023-05-22T15:43:30.219191+0000","last_unstale":"2023-05-22T15:43:30.222151+0000","last_undegraded":"2023-05-22T15:43:30.222151+0000","last_fullsized":"2023-05-22T15:43:30.222151+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_clean_scrub_stamp":"2023-05-22T15:43:29.738376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:20:06.441496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.c","version":"0'0","reported_seq":10,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-05-22T15:43:30.145281+0000","last_change":"2023-05-22T15:43:29.807154+0000","last_active":"2023-05-22T15:43:30.145281+0000","last_peered":"2023-05-22T15:43:30.145281+0000","last_clean":"2023-05-22T15:43:30.145281+0000","last_became_active":"2023-05-22T15:43:29.806884+0000","last_became_peered":"2023-05-22T15:43:29.806884+0000","last_unstale":"2023-05-22T15:43:30.145281+0000","last_undegraded":"2023-05-22T15:43:30.145281+0000","last_fullsized":"2023-05-22T15:43:30.145281+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:43:28.734545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:43:28.734545+0000","last_clean_scrub_stamp":"2023-05-22T15:43:28.734545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:20:05.681030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":105,"reported_epoch":116,"state":"active+clean","last_fresh":"2023-05-22T15:43:29.786011+0000","last_change":"2023-05-22T15:26:44.977198+0000","last_active":"2023-05-22T15:43:29.786011+0000","last_peered":"2023-05-22T15:43:29.786011+0000","last_clean":"2023-05-22T15:43:29.786011+0000","last_became_active":"2023-05-22T15:26:44.976445+0000","last_became_peered":"2023-05-22T15:26:44.976445+0000","last_unstale":"2023-05-22T15:43:29.786011+0000","last_undegraded":"2023-05-22T15:43:29.786011+0000","last_fullsized":"2023-05-22T15:43:29.786011+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:12:41.464054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.c","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-05-22T15:43:30.204628+0000","last_change":"2023-05-22T15:43:30.187095+0000","last_active":"2023-05-22T15:43:30.204628+0000","last_peered":"2023-05-22T15:43:30.204628+0000","last_clean":"2023-05-22T15:43:30.204628+0000","last_became_active":"2023-05-22T15:43:30.186352+0000","last_became_peered":"2023-05-22T15:43:30.186352+0000","last_unstale":"2023-05-22T15:43:30.204628+0000","last_undegraded":"2023-05-22T15:43:30.204628+0000","last_fullsized":"2023-05-22T15:43:30.204628+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_clean_scrub_stamp":"2023-05-22T15:43:29.738376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:12:33.491991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.d","version":"117'1","reported_seq":11,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-05-22T15:43:30.198947+0000","last_change":"2023-05-22T15:43:29.794522+0000","last_active":"2023-05-22T15:43:30.198947+0000","last_peered":"2023-05-22T15:43:30.198947+0000","last_clean":"2023-05-22T15:43:30.198947+0000","last_became_active":"2023-05-22T15:43:29.793688+0000","last_became_peered":"2023-05-22T15:43:29.793688+0000","last_unstale":"2023-05-22T15:43:30.198947+0000","last_undegraded":"2023-05-22T15:43:30.198947+0000","last_fullsized":"2023-05-22T15:43:30.198947+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:43:28.734545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:43:28.734545+0000","last_clean_scrub_stamp":"2023-05-22T15:43:28.734545+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:57:33.702470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":105,"reported_epoch":116,"state":"active+clean","last_fresh":"2023-05-22T15:43:29.755329+0000","last_change":"2023-05-22T15:26:44.969469+0000","last_active":"2023-05-22T15:43:29.755329+0000","last_peered":"2023-05-22T15:43:29.755329+0000","last_clean":"2023-05-22T15:43:29.755329+0000","last_became_active":"2023-05-22T15:26:44.969083+0000","last_became_peered":"2023-05-22T15:26:44.969083+0000","last_unstale":"2023-05-22T15:43:29.755329+0000","last_undegraded":"2023-05-22T15:43:29.755329+0000","last_fullsized":"2023-05-22T15:43:29.755329+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:29:23.670067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.f","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-05-22T15:43:30.216502+0000","last_change":"2023-05-22T15:43:30.212809+0000","last_active":"2023-05-22T15:43:30.216502+0000","last_peered":"2023-05-22T15:43:30.216502+0000","last_clean":"2023-05-22T15:43:30.216502+0000","last_became_active":"2023-05-22T15:43:30.212094+0000","last_became_peered":"2023-05-22T15:43:30.212094+0000","last_unstale":"2023-05-22T15:43:30.216502+0000","last_undegraded":"2023-05-22T15:43:30.216502+0000","last_fullsized":"2023-05-22T15:43:30.216502+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_clean_scrub_stamp":"2023-05-22T15:43:29.738376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:00:19.974693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.e","version":"117'2","reported_seq":12,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-05-22T15:43:30.218642+0000","last_change":"2023-05-22T15:43:29.807795+0000","last_active":"2023-05-22T15:43:30.218642+0000","last_peered":"2023-05-22T15:43:30.218642+0000","last_clean":"2023-05-22T15:43:30.218642+0000","last_became_active":"2023-05-22T15:43:29.806892+0000","last_became_peered":"2023-05-22T15:43:29.806892+0000","last_unstale":"2023-05-22T15:43:30.218642+0000","last_undegraded":"2023-05-22T15:43:30.218642+0000","last_fullsized":"2023-05-22T15:43:30.218642+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:43:28.734545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:43:28.734545+0000","last_clean_scrub_stamp":"2023-05-22T15:43:28.734545+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:47:23.008171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":111,"reported_epoch":116,"state":"active+clean","last_fresh":"2023-05-22T15:43:29.778708+0000","last_change":"2023-05-22T15:26:44.976632+0000","last_active":"2023-05-22T15:43:29.778708+0000","last_peered":"2023-05-22T15:43:29.778708+0000","last_clean":"2023-05-22T15:43:29.778708+0000","last_became_active":"2023-05-22T15:26:44.975546+0000","last_became_peered":"2023-05-22T15:26:44.975546+0000","last_unstale":"2023-05-22T15:43:29.778708+0000","last_undegraded":"2023-05-22T15:43:29.778708+0000","last_fullsized":"2023-05-22T15:43:29.778708+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:42:57.973030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.9","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-05-22T15:43:30.211346+0000","last_change":"2023-05-22T15:43:30.198152+0000","last_active":"2023-05-22T15:43:30.211346+0000","last_peered":"2023-05-22T15:43:30.211346+0000","last_clean":"2023-05-22T15:43:30.211346+0000","last_became_active":"2023-05-22T15:43:30.197390+0000","last_became_peered":"2023-05-22T15:43:30.197390+0000","last_unstale":"2023-05-22T15:43:30.211346+0000","last_undegraded":"2023-05-22T15:43:30.211346+0000","last_fullsized":"2023-05-22T15:43:30.211346+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_clean_scrub_stamp":"2023-05-22T15:43:29.738376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:19:19.539413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.8","version":"0'0","reported_seq":10,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-05-22T15:43:30.144067+0000","last_change":"2023-05-22T15:43:29.786288+0000","last_active":"2023-05-22T15:43:30.144067+0000","last_peered":"2023-05-22T15:43:30.144067+0000","last_clean":"2023-05-22T15:43:30.144067+0000","last_became_active":"2023-05-22T15:43:29.786009+0000","last_became_peered":"2023-05-22T15:43:29.786009+0000","last_unstale":"2023-05-22T15:43:30.144067+0000","last_undegraded":"2023-05-22T15:43:30.144067+0000","last_fullsized":"2023-05-22T15:43:30.144067+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:43:28.734545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:43:28.734545+0000","last_clean_scrub_stamp":"2023-05-22T15:43:28.734545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:48:01.712437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":99,"reported_epoch":116,"state":"active+clean","last_fresh":"2023-05-22T15:43:29.765352+0000","last_change":"2023-05-22T15:26:45.091475+0000","last_active":"2023-05-22T15:43:29.765352+0000","last_peered":"2023-05-22T15:43:29.765352+0000","last_clean":"2023-05-22T15:43:29.765352+0000","last_became_active":"2023-05-22T15:26:45.091143+0000","last_became_peered":"2023-05-22T15:26:45.091143+0000","last_unstale":"2023-05-22T15:43:29.765352+0000","last_undegraded":"2023-05-22T15:43:29.765352+0000","last_fullsized":"2023-05-22T15:43:29.765352+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:12:41.464054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.8","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-05-22T15:43:30.174494+0000","last_change":"2023-05-22T15:43:30.166073+0000","last_active":"2023-05-22T15:43:30.174494+0000","last_peered":"2023-05-22T15:43:30.174494+0000","last_clean":"2023-05-22T15:43:30.174494+0000","last_became_active":"2023-05-22T15:43:30.165458+0000","last_became_peered":"2023-05-22T15:43:30.165458+0000","last_unstale":"2023-05-22T15:43:30.174494+0000","last_undegraded":"2023-05-22T15:43:30.174494+0000","last_fullsized":"2023-05-22T15:43:30.174494+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_clean_scrub_stamp":"2023-05-22T15:43:29.738376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T03:17:16.869920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.9","version":"0'0","reported_seq":10,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-05-22T15:43:30.152917+0000","last_change":"2023-05-22T15:43:29.781790+0000","last_active":"2023-05-22T15:43:30.152917+0000","last_peered":"2023-05-22T15:43:30.152917+0000","last_clean":"2023-05-22T15:43:30.152917+0000","last_became_active":"2023-05-22T15:43:29.779832+0000","last_became_peered":"2023-05-22T15:43:29.779832+0000","last_unstale":"2023-05-22T15:43:30.152917+0000","last_undegraded":"2023-05-22T15:43:30.152917+0000","last_fullsized":"2023-05-22T15:43:30.152917+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:43:28.734545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:43:28.734545+0000","last_clean_scrub_stamp":"2023-05-22T15:43:28.734545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:59:48.770501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":104,"reported_epoch":116,"state":"active+clean","last_fresh":"2023-05-22T15:43:29.791838+0000","last_change":"2023-05-22T15:26:45.092367+0000","last_active":"2023-05-22T15:43:29.791838+0000","last_peered":"2023-05-22T15:43:29.791838+0000","last_clean":"2023-05-22T15:43:29.791838+0000","last_became_active":"2023-05-22T15:26:45.091951+0000","last_became_peered":"2023-05-22T15:26:45.091951+0000","last_unstale":"2023-05-22T15:43:29.791838+0000","last_undegraded":"2023-05-22T15:43:29.791838+0000","last_fullsized":"2023-05-22T15:43:29.791838+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:42:57.973030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.b","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-05-22T15:43:30.174523+0000","last_change":"2023-05-22T15:43:30.166496+0000","last_active":"2023-05-22T15:43:30.174523+0000","last_peered":"2023-05-22T15:43:30.174523+0000","last_clean":"2023-05-22T15:43:30.174523+0000","last_became_active":"2023-05-22T15:43:30.166174+0000","last_became_peered":"2023-05-22T15:43:30.166174+0000","last_unstale":"2023-05-22T15:43:30.174523+0000","last_undegraded":"2023-05-22T15:43:30.174523+0000","last_fullsized":"2023-05-22T15:43:30.174523+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_clean_scrub_stamp":"2023-05-22T15:43:29.738376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:52:09.288277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.a","version":"0'0","reported_seq":10,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-05-22T15:43:30.153359+0000","last_change":"2023-05-22T15:43:29.786377+0000","last_active":"2023-05-22T15:43:30.153359+0000","last_peered":"2023-05-22T15:43:30.153359+0000","last_clean":"2023-05-22T15:43:30.153359+0000","last_became_active":"2023-05-22T15:43:29.785660+0000","last_became_peered":"2023-05-22T15:43:29.785660+0000","last_unstale":"2023-05-22T15:43:30.153359+0000","last_undegraded":"2023-05-22T15:43:30.153359+0000","last_fullsized":"2023-05-22T15:43:30.153359+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:43:28.734545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:43:28.734545+0000","last_clean_scrub_stamp":"2023-05-22T15:43:28.734545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:57:33.702470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":98,"reported_epoch":116,"state":"active+clean","last_fresh":"2023-05-22T15:43:29.759473+0000","last_change":"2023-05-22T15:26:44.978858+0000","last_active":"2023-05-22T15:43:29.759473+0000","last_peered":"2023-05-22T15:43:29.759473+0000","last_clean":"2023-05-22T15:43:29.759473+0000","last_became_active":"2023-05-22T15:26:44.978443+0000","last_became_peered":"2023-05-22T15:26:44.978443+0000","last_unstale":"2023-05-22T15:43:29.759473+0000","last_undegraded":"2023-05-22T15:43:29.759473+0000","last_fullsized":"2023-05-22T15:43:29.759473+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:46:29.290243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.a","version":"0'0","reported_seq":9,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-05-22T15:43:30.174457+0000","last_change":"2023-05-22T15:43:30.165980+0000","last_active":"2023-05-22T15:43:30.174457+0000","last_peered":"2023-05-22T15:43:30.174457+0000","last_clean":"2023-05-22T15:43:30.174457+0000","last_became_active":"2023-05-22T15:43:30.165677+0000","last_became_peered":"2023-05-22T15:43:30.165677+0000","last_unstale":"2023-05-22T15:43:30.174457+0000","last_undegraded":"2023-05-22T15:43:30.174457+0000","last_fullsized":"2023-05-22T15:43:30.174457+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:43:29.738376+0000","last_clean_scrub_stamp":"2023-05-22T15:43:29.738376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:21:06.992004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.b","version":"117'2","reported_seq":12,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-05-22T15:43:30.166870+0000","last_change":"2023-05-22T15:43:29.782081+0000","last_active":"2023-05-22T15:43:30.166870+0000","last_peered":"2023-05-22T15:43:30.166870+0000","last_clean":"2023-05-22T15:43:30.166870+0000","last_became_active":"2023-05-22T15:43:29.781811+0000","last_became_peered":"2023-05-22T15:43:29.781811+0000","last_unstale":"2023-05-22T15:43:30.166870+0000","last_undegraded":"2023-05-22T15:43:30.166870+0000","last_fullsized":"2023-05-22T15:43:30.166870+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:43:28.734545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:43:28.734545+0000","last_clean_scrub_stamp":"2023-05-22T15:43:28.734545+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:58:33.399611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":99,"reported_epoch":116,"state":"active+clean","last_fresh":"2023-05-22T15:43:29.773979+0000","last_change":"2023-05-22T15:26:44.982421+0000","last_active":"2023-05-22T15:43:29.773979+0000","last_peered":"2023-05-22T15:43:29.773979+0000","last_clean":"2023-05-22T15:43:29.773979+0000","last_became_active":"2023-05-22T15:26:44.981586+0000","last_became_peered":"2023-05-22T15:26:44.981586+0000","last_unstale":"2023-05-22T15:43:29.773979+0000","last_undegraded":"2023-05-22T15:43:29.773979+0000","last_fullsized":"2023-05-22T15:43:29.773979+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:14:11.156544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":15,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":14,"num_pg":32,"stat_sum":{"num_bytes":2330,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":884,"num_read_kb":1748,"num_write":319,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":205,"ondisk_log_size":205,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575051,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35156,"kb_used_data":1996,"kb_used_omap":0,"kb_used_meta":33152,"kb_avail":94336684,"statfs":{"total":96636764160,"available":96600764416,"internally_reserved":0,"allocated":2043904,"data_stored":1348182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":33947648},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,4,0,1,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 15:43:00 2023","interfaces":[{"interface":"back","average":{"1min":0.78900000000000003,"5min":0.84199999999999997,"15min":0.86199999999999999},"min":{"1min":0.51300000000000001,"5min":0.44800000000000001,"15min":0.42599999999999999},"max":{"1min":1.151,"5min":1.4199999999999999,"15min":1.7709999999999999},"last":2.637},{"interface":"front","average":{"1min":0.87,"5min":0.86399999999999999,"15min":0.879},"min":{"1min":0.52900000000000003,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":1.107,"5min":1.2090000000000001,"15min":1.3169999999999999},"last":3.0449999999999999}]},{"osd":1,"last update":"Mon May 22 15:43:00 2023","interfaces":[{"interface":"back","average":{"1min":0.94899999999999995,"5min":0.95599999999999996,"15min":0.98099999999999998},"min":{"1min":0.60999999999999999,"5min":0.5,"15min":0.5},"max":{"1min":1.2589999999999999,"5min":1.385,"15min":1.8300000000000001},"last":0.505},{"interface":"front","average":{"1min":0.95699999999999996,"5min":1.002,"15min":1.02},"min":{"1min":0.63,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":1.1950000000000001,"5min":1.3169999999999999,"15min":2.161},"last":2.7269999999999999}]},{"osd":2,"last update":"Mon May 22 15:43:00 2023","interfaces":[{"interface":"back","average":{"1min":0.84699999999999998,"5min":0.85099999999999998,"15min":0.90300000000000002},"min":{"1min":0.60899999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.2130000000000001,"5min":1.2130000000000001,"15min":6.7279999999999998},"last":0.67800000000000005},{"interface":"front","average":{"1min":0.90900000000000003,"5min":0.93600000000000005,"15min":0.96299999999999997},"min":{"1min":0.56599999999999995,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":1.2809999999999999,"5min":1.2809999999999999,"15min":6.8810000000000002},"last":3.141}]},{"osd":3,"last update":"Mon May 22 15:43:00 2023","interfaces":[{"interface":"back","average":{"1min":0.92300000000000004,"5min":0.94299999999999995,"15min":0.95599999999999996},"min":{"1min":0.56399999999999995,"5min":0.55900000000000005,"15min":0.53800000000000003},"max":{"1min":1.214,"5min":1.3360000000000001,"15min":3.0569999999999999},"last":2.8149999999999999},{"interface":"front","average":{"1min":0.91900000000000004,"5min":0.95299999999999996,"15min":0.97699999999999998},"min":{"1min":0.72099999999999997,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":1.1120000000000001,"5min":1.3029999999999999,"15min":1.4710000000000001},"last":0.75900000000000001}]},{"osd":4,"last update":"Mon May 22 15:43:00 2023","interfaces":[{"interface":"back","average":{"1min":1.014,"5min":0.91300000000000003,"15min":0.89000000000000001},"min":{"1min":0.69599999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"max":{"1min":1.3460000000000001,"5min":1.621,"15min":1.621},"last":3.472},{"interface":"front","average":{"1min":1.006,"5min":0.99299999999999999,"15min":0.97599999999999998},"min":{"1min":0.65900000000000003,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.2829999999999999,"5min":1.4319999999999999,"15min":1.958},"last":15.627000000000001}]},{"osd":5,"last update":"Mon May 22 15:43:00 2023","interfaces":[{"interface":"back","average":{"1min":1.0669999999999999,"5min":1.026,"15min":0.999},"min":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.56699999999999995},"max":{"1min":1.3340000000000001,"5min":1.409,"15min":1.5029999999999999},"last":4.5599999999999996},{"interface":"front","average":{"1min":1.123,"5min":1.125,"15min":1.135},"min":{"1min":0.72199999999999998,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":1.4039999999999999,"5min":1.5229999999999999,"15min":5.0839999999999996},"last":4.3799999999999999}]},{"osd":6,"last update":"Mon May 22 15:43:00 2023","interfaces":[{"interface":"back","average":{"1min":1.1120000000000001,"5min":1.0529999999999999,"15min":1.0720000000000001},"min":{"1min":0.80800000000000005,"5min":0.65800000000000003,"15min":0.65800000000000003},"max":{"1min":1.4950000000000001,"5min":1.4950000000000001,"15min":5.1719999999999997},"last":6.5069999999999997},{"interface":"front","average":{"1min":1.0960000000000001,"5min":1.1200000000000001,"15min":1.1279999999999999},"min":{"1min":0.73699999999999999,"5min":0.70099999999999996,"15min":0.70099999999999996},"max":{"1min":1.3360000000000001,"5min":1.702,"15min":1.879},"last":3.3500000000000001}]}]},{"osd":6,"up_from":13,"seq":55834575052,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30928,"kb_used_data":1416,"kb_used_omap":0,"kb_used_meta":29504,"kb_avail":94340912,"statfs":{"total":96636764160,"available":96605093888,"internally_reserved":0,"allocated":1449984,"data_stored":758338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":30212096},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 15:43:11 2023","interfaces":[{"interface":"back","average":{"1min":0.89500000000000002,"5min":0.88100000000000001,"15min":0.90200000000000002},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":1.486,"5min":1.486,"15min":1.486},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.93799999999999994,"5min":0.91300000000000003,"15min":0.90700000000000003},"min":{"1min":0.53500000000000003,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":1.395,"5min":1.5680000000000001,"15min":1.5680000000000001},"last":0.93700000000000006}]},{"osd":1,"last update":"Mon May 22 15:43:11 2023","interfaces":[{"interface":"back","average":{"1min":0.85399999999999998,"5min":0.91700000000000004,"15min":0.91200000000000003},"min":{"1min":0.504,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":1.1499999999999999,"5min":1.3169999999999999,"15min":1.375},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.95099999999999996},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":1.325,"5min":1.554,"15min":1.554},"last":0.65500000000000003}]},{"osd":2,"last update":"Mon May 22 15:43:11 2023","interfaces":[{"interface":"back","average":{"1min":0.91000000000000003,"5min":0.95899999999999996,"15min":0.97999999999999998},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":1.276,"5min":1.425,"15min":1.425},"last":1.0609999999999999},{"interface":"front","average":{"1min":0.92200000000000004,"5min":1.022,"15min":1.042},"min":{"1min":0.5,"5min":0.442,"15min":0.442},"max":{"1min":1.6020000000000001,"5min":1.6020000000000001,"15min":1.6020000000000001},"last":0.78300000000000003}]},{"osd":3,"last update":"Mon May 22 15:43:11 2023","interfaces":[{"interface":"back","average":{"1min":0.90100000000000002,"5min":0.98399999999999999,"15min":0.98899999999999999},"min":{"1min":0.55700000000000005,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":1.333,"5min":1.679,"15min":1.679},"last":0.753},{"interface":"front","average":{"1min":0.91400000000000003,"5min":0.95399999999999996,"15min":0.95799999999999996},"min":{"1min":0.54700000000000004,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.2150000000000001,"5min":1.4570000000000001,"15min":1.4570000000000001},"last":0.84799999999999998}]},{"osd":4,"last update":"Mon May 22 15:43:11 2023","interfaces":[{"interface":"back","average":{"1min":0.91700000000000004,"5min":0.92600000000000005,"15min":0.92300000000000004},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":1.393,"5min":1.6859999999999999,"15min":1.6859999999999999},"last":1.0049999999999999},{"interface":"front","average":{"1min":1.0469999999999999,"5min":1.02,"15min":1.026},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":1.3799999999999999,"5min":1.8300000000000001,"15min":1.8300000000000001},"last":1.137}]},{"osd":5,"last update":"Mon May 22 15:43:11 2023","interfaces":[{"interface":"back","average":{"1min":0.94099999999999995,"5min":0.98799999999999999,"15min":0.97099999999999997},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":1.419,"5min":1.6499999999999999,"15min":1.6499999999999999},"last":1.2929999999999999},{"interface":"front","average":{"1min":1.139,"5min":1.1160000000000001,"15min":1.099},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.728,"5min":1.728,"15min":1.728},"last":1.1000000000000001}]},{"osd":7,"last update":"Mon May 22 15:43:11 2023","interfaces":[{"interface":"back","average":{"1min":1.135,"5min":1.1200000000000001,"15min":1.127},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":1.651,"5min":1.9179999999999999,"15min":1.9179999999999999},"last":1.2629999999999999},{"interface":"front","average":{"1min":1.1080000000000001,"5min":1.1319999999999999,"15min":1.1279999999999999},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":1.5640000000000001,"5min":1.8520000000000001,"15min":1.8520000000000001},"last":1.2110000000000001}]}]},{"osd":5,"up_from":13,"seq":55834575052,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":24780,"kb_used_data":1412,"kb_used_omap":0,"kb_used_meta":23360,"kb_avail":94347060,"statfs":{"total":96636764160,"available":96611389440,"internally_reserved":0,"allocated":1445888,"data_stored":757772,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":23920640},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 15:43:08 2023","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.82999999999999996,"15min":0.878},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.016,"5min":1.3440000000000001,"15min":1.3440000000000001},"last":1.0860000000000001},{"interface":"front","average":{"1min":0.81100000000000005,"5min":0.877,"15min":0.88100000000000001},"min":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.497},"max":{"1min":1.353,"5min":2.1699999999999999,"15min":2.1699999999999999},"last":0.754}]},{"osd":1,"last update":"Mon May 22 15:43:08 2023","interfaces":[{"interface":"back","average":{"1min":0.85899999999999999,"5min":0.96499999999999997,"15min":0.99299999999999999},"min":{"1min":0.52400000000000002,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":1.163,"5min":1.9370000000000001,"15min":1.9370000000000001},"last":1.042},{"interface":"front","average":{"1min":0.95299999999999996,"5min":0.98699999999999999,"15min":1.022},"min":{"1min":0.55300000000000005,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.2430000000000001,"5min":1.3680000000000001,"15min":1.3680000000000001},"last":1.179}]},{"osd":2,"last update":"Mon May 22 15:43:08 2023","interfaces":[{"interface":"back","average":{"1min":0.95399999999999996,"5min":0.95899999999999996,"15min":0.93700000000000006},"min":{"1min":0.65100000000000002,"5min":0.504,"15min":0.504},"max":{"1min":1.1870000000000001,"5min":1.9830000000000001,"15min":1.9830000000000001},"last":0.94599999999999995},{"interface":"front","average":{"1min":0.90500000000000003,"5min":0.94999999999999996,"15min":0.94099999999999995},"min":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.56599999999999995},"max":{"1min":1.304,"5min":2.1120000000000001,"15min":2.1120000000000001},"last":0.91200000000000003}]},{"osd":3,"last update":"Mon May 22 15:43:08 2023","interfaces":[{"interface":"back","average":{"1min":0.86199999999999999,"5min":0.96699999999999997,"15min":0.95199999999999996},"min":{"1min":0.65900000000000003,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.149,"5min":1.4419999999999999,"15min":1.4419999999999999},"last":0.97899999999999998},{"interface":"front","average":{"1min":0.96799999999999997,"5min":1.038,"15min":1.034},"min":{"1min":0.55000000000000004,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":1.1479999999999999,"5min":2.52,"15min":2.52},"last":0.86499999999999999}]},{"osd":4,"last update":"Mon May 22 15:43:08 2023","interfaces":[{"interface":"back","average":{"1min":0.92200000000000004,"5min":0.96799999999999997,"15min":0.93500000000000005},"min":{"1min":0.65200000000000002,"5min":0.60599999999999998,"15min":0.57599999999999996},"max":{"1min":1.2210000000000001,"5min":2.226,"15min":2.226},"last":1.131},{"interface":"front","average":{"1min":0.88200000000000001,"5min":1.0389999999999999,"15min":1.026},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.1639999999999999,"5min":1.6859999999999999,"15min":1.6859999999999999},"last":1.2849999999999999}]},{"osd":6,"last update":"Mon May 22 15:43:08 2023","interfaces":[{"interface":"back","average":{"1min":1.0009999999999999,"5min":1.077,"15min":1.0389999999999999},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.2929999999999999,"5min":2.831,"15min":2.831},"last":1.1080000000000001},{"interface":"front","average":{"1min":1.0089999999999999,"5min":1.0880000000000001,"15min":1.071},"min":{"1min":0.629,"5min":0.629,"15min":0.629},"max":{"1min":1.302,"5min":2.3690000000000002,"15min":2.3690000000000002},"last":1.236}]},{"osd":7,"last update":"Mon May 22 15:43:08 2023","interfaces":[{"interface":"back","average":{"1min":1.0660000000000001,"5min":1.151,"15min":1.1140000000000001},"min":{"1min":0.70999999999999996,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":1.2609999999999999,"5min":1.849,"15min":1.849},"last":1.3640000000000001},{"interface":"front","average":{"1min":1.0980000000000001,"5min":1.198,"15min":1.1659999999999999},"min":{"1min":0.76100000000000001,"5min":0.75700000000000001,"15min":0.71699999999999997},"max":{"1min":1.4159999999999999,"5min":2.7429999999999999,"15min":2.7429999999999999},"last":1.397}]}]},{"osd":4,"up_from":13,"seq":55834575050,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":26588,"kb_used_data":1428,"kb_used_omap":0,"kb_used_meta":25152,"kb_avail":94345252,"statfs":{"total":96636764160,"available":96609538048,"internally_reserved":0,"allocated":1462272,"data_stored":759374,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":25755648},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 15:43:11 2023","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.89900000000000002,"15min":0.90000000000000002},"min":{"1min":0.505,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":1.048,"5min":1.9910000000000001,"15min":1.9910000000000001},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.89000000000000001,"5min":0.89700000000000002,"15min":0.93500000000000005},"min":{"1min":0.51600000000000001,"5min":0.373,"15min":0.373},"max":{"1min":1.264,"5min":1.3779999999999999,"15min":1.633},"last":0.68200000000000005}]},{"osd":1,"last update":"Mon May 22 15:43:11 2023","interfaces":[{"interface":"back","average":{"1min":0.82099999999999995,"5min":0.95899999999999996,"15min":0.97299999999999998},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":1.1779999999999999,"5min":1.383,"15min":1.3859999999999999},"last":0.78500000000000003},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.872,"15min":0.90200000000000002},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1,"5min":1.3120000000000001,"15min":1.3640000000000001},"last":0.74099999999999999}]},{"osd":2,"last update":"Mon May 22 15:43:11 2023","interfaces":[{"interface":"back","average":{"1min":0.95799999999999996,"5min":0.95699999999999996,"15min":0.94699999999999995},"min":{"1min":0.50800000000000001,"5min":0.495,"15min":0.495},"max":{"1min":1.96,"5min":2.1019999999999999,"15min":2.1019999999999999},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.93200000000000005,"5min":0.996,"15min":1.002},"min":{"1min":0.48799999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":1.4399999999999999,"5min":1.4399999999999999,"15min":1.6000000000000001},"last":0.59299999999999997}]},{"osd":3,"last update":"Mon May 22 15:43:11 2023","interfaces":[{"interface":"back","average":{"1min":0.872,"5min":0.91200000000000003,"15min":0.93799999999999994},"min":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":1.2310000000000001,"5min":1.29,"15min":1.665},"last":0.51600000000000001},{"interface":"front","average":{"1min":1.016,"5min":1.0489999999999999,"15min":1.0309999999999999},"min":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":1.4179999999999999,"5min":2.1720000000000002,"15min":2.1720000000000002},"last":0.71299999999999997}]},{"osd":5,"last update":"Mon May 22 15:43:11 2023","interfaces":[{"interface":"back","average":{"1min":0.96799999999999997,"5min":1.022,"15min":0.98999999999999999},"min":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.51200000000000001},"max":{"1min":1.286,"5min":1.472,"15min":1.6830000000000001},"last":1.0109999999999999},{"interface":"front","average":{"1min":0.95899999999999996,"5min":0.97499999999999998,"15min":0.97499999999999998},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.3420000000000001,"5min":1.401,"15min":1.6719999999999999},"last":0.86099999999999999}]},{"osd":6,"last update":"Mon May 22 15:43:11 2023","interfaces":[{"interface":"back","average":{"1min":0.98199999999999998,"5min":1.0249999999999999,"15min":0.98899999999999999},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":1.3620000000000001,"5min":2.2349999999999999,"15min":2.2349999999999999},"last":0.93899999999999995},{"interface":"front","average":{"1min":1.0740000000000001,"5min":1.1299999999999999,"15min":1.107},"min":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":1.4339999999999999,"5min":1.8220000000000001,"15min":1.8220000000000001},"last":0.90700000000000003}]},{"osd":7,"last update":"Mon May 22 15:43:11 2023","interfaces":[{"interface":"back","average":{"1min":1.0189999999999999,"5min":1.0600000000000001,"15min":1.0569999999999999},"min":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.67700000000000005},"max":{"1min":1.452,"5min":1.5800000000000001,"15min":1.5800000000000001},"last":0.97999999999999998},{"interface":"front","average":{"1min":1.163,"5min":1.1639999999999999,"15min":1.1419999999999999},"min":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"max":{"1min":1.53,"5min":2.306,"15min":2.306},"last":0.82699999999999996}]}]},{"osd":3,"up_from":13,"seq":55834575050,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35672,"kb_used_data":1424,"kb_used_omap":0,"kb_used_meta":34240,"kb_avail":94336168,"statfs":{"total":96636764160,"available":96600236032,"internally_reserved":0,"allocated":1458176,"data_stored":759272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":35061760},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 15:42:54 2023","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.61199999999999999,"15min":0.628},"min":{"1min":0.312,"5min":0.19,"15min":0.19},"max":{"1min":1.1519999999999999,"5min":1.194,"15min":1.194},"last":0.70099999999999996},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.68700000000000006,"15min":0.69999999999999996},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.88300000000000001,"5min":1.1799999999999999,"15min":1.296},"last":0.439}]},{"osd":1,"last update":"Mon May 22 15:42:54 2023","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.61899999999999999,"15min":0.66300000000000003},"min":{"1min":0.315,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.80000000000000004,"5min":1.008,"15min":1.0600000000000001},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.71899999999999997,"15min":0.75800000000000001},"min":{"1min":0.34399999999999997,"5min":0.313,"15min":0.216},"max":{"1min":0.82299999999999995,"5min":1.222,"15min":1.222},"last":0.56999999999999995}]},{"osd":2,"last update":"Mon May 22 15:42:54 2023","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.76700000000000002,"15min":0.81200000000000006},"min":{"1min":0.41599999999999998,"5min":0.36599999999999999,"15min":0.32200000000000001},"max":{"1min":1.506,"5min":1.506,"15min":1.506},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.748,"15min":0.80200000000000005},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.29699999999999999},"max":{"1min":1.097,"5min":1.181,"15min":1.468},"last":0.62}]},{"osd":4,"last update":"Mon May 22 15:42:54 2023","interfaces":[{"interface":"back","average":{"1min":0.84399999999999997,"5min":0.92300000000000004,"15min":0.95299999999999996},"min":{"1min":0.54500000000000004,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":1.413,"5min":1.413,"15min":1.4470000000000001},"last":0.89200000000000002},{"interface":"front","average":{"1min":0.81799999999999995,"5min":0.94699999999999995,"15min":1.0009999999999999},"min":{"1min":0.64800000000000002,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.077,"5min":1.3740000000000001,"15min":1.4470000000000001},"last":0.84899999999999998}]},{"osd":5,"last update":"Mon May 22 15:42:54 2023","interfaces":[{"interface":"back","average":{"1min":0.83199999999999996,"5min":0.96099999999999997,"15min":1.01},"min":{"1min":0.53200000000000003,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":1.2070000000000001,"5min":1.3149999999999999,"15min":1.4339999999999999},"last":0.97299999999999998},{"interface":"front","average":{"1min":0.94899999999999995,"5min":0.98999999999999999,"15min":1.0429999999999999},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.4299999999999999,"5min":1.4299999999999999,"15min":1.643},"last":0.81499999999999995}]},{"osd":6,"last update":"Mon May 22 15:42:54 2023","interfaces":[{"interface":"back","average":{"1min":0.85099999999999998,"5min":0.98099999999999998,"15min":1.0369999999999999},"min":{"1min":0.47699999999999998,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.1539999999999999,"5min":1.4139999999999999,"15min":1.863},"last":1.0549999999999999},{"interface":"front","average":{"1min":0.93100000000000005,"5min":1.0229999999999999,"15min":1.0720000000000001},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":1.2909999999999999,"5min":1.3109999999999999,"15min":1.704},"last":0.92600000000000005}]},{"osd":7,"last update":"Mon May 22 15:42:54 2023","interfaces":[{"interface":"back","average":{"1min":0.96699999999999997,"5min":1.0469999999999999,"15min":1.1000000000000001},"min":{"1min":0.58599999999999997,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":1.407,"5min":1.407,"15min":1.5640000000000001},"last":1.0289999999999999},{"interface":"front","average":{"1min":0.94099999999999995,"5min":1.024,"15min":1.0700000000000001},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":1.8700000000000001,"5min":1.8700000000000001,"15min":1.8700000000000001},"last":0.73899999999999999}]}]},{"osd":2,"up_from":13,"seq":55834575051,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":17804,"kb_used_data":1412,"kb_used_omap":0,"kb_used_meta":16384,"kb_avail":94354036,"statfs":{"total":96636764160,"available":96618532864,"internally_reserved":0,"allocated":1445888,"data_stored":757736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":16777216},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 15:42:58 2023","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.65500000000000003,"15min":0.65400000000000003},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":1.1080000000000001,"5min":1.8240000000000001,"15min":1.8240000000000001},"last":0.83299999999999996},{"interface":"front","average":{"1min":0.71099999999999997,"5min":0.75600000000000001,"15min":0.746},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.253},"max":{"1min":1.0369999999999999,"5min":1.8160000000000001,"15min":1.8160000000000001},"last":0.48199999999999998}]},{"osd":1,"last update":"Mon May 22 15:42:58 2023","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.73099999999999998,"15min":0.73399999999999999},"min":{"1min":0.34799999999999998,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.96599999999999997,"5min":1.2,"15min":1.2},"last":0.72499999999999998},{"interface":"front","average":{"1min":0.73899999999999999,"5min":0.81899999999999995,"15min":0.81200000000000006},"min":{"1min":0.317,"5min":0.317,"15min":0.29099999999999998},"max":{"1min":0.99299999999999999,"5min":2.5089999999999999,"15min":2.5089999999999999},"last":0.57999999999999996}]},{"osd":3,"last update":"Mon May 22 15:42:58 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.72399999999999998,"15min":0.72099999999999997},"min":{"1min":0.36699999999999999,"5min":0.36299999999999999,"15min":0.26700000000000002},"max":{"1min":0.95699999999999996,"5min":2.7629999999999999,"15min":2.7629999999999999},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.871,"5min":0.88600000000000001,"15min":0.88100000000000001},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.33300000000000002},"max":{"1min":1.256,"5min":2.6840000000000002,"15min":2.6840000000000002},"last":1.143}]},{"osd":4,"last update":"Mon May 22 15:42:58 2023","interfaces":[{"interface":"back","average":{"1min":1.0069999999999999,"5min":1.03,"15min":1.016},"min":{"1min":0.76000000000000001,"5min":0.65800000000000003,"15min":0.55500000000000005},"max":{"1min":1.536,"5min":2.8039999999999998,"15min":2.8039999999999998},"last":1.012},{"interface":"front","average":{"1min":1.0249999999999999,"5min":1.036,"15min":1.028},"min":{"1min":0.59199999999999997,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":1.4019999999999999,"5min":2.0819999999999999,"15min":2.0819999999999999},"last":1.121}]},{"osd":5,"last update":"Mon May 22 15:42:58 2023","interfaces":[{"interface":"back","average":{"1min":1.0700000000000001,"5min":1.0669999999999999,"15min":1.036},"min":{"1min":0.622,"5min":0.622,"15min":0.55200000000000005},"max":{"1min":1.4570000000000001,"5min":2.3860000000000001,"15min":2.3860000000000001},"last":0.90800000000000003},{"interface":"front","average":{"1min":0.95799999999999996,"5min":0.996,"15min":0.995},"min":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.56200000000000006},"max":{"1min":1.327,"5min":3.0299999999999998,"15min":3.0299999999999998},"last":1.0680000000000001}]},{"osd":6,"last update":"Mon May 22 15:42:58 2023","interfaces":[{"interface":"back","average":{"1min":0.97999999999999998,"5min":1.024,"15min":1.0309999999999999},"min":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.56399999999999995},"max":{"1min":1.49,"5min":2.883,"15min":2.883},"last":0.98399999999999999},{"interface":"front","average":{"1min":1.0660000000000001,"5min":1.107,"15min":1.095},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.53400000000000003},"max":{"1min":1.421,"5min":3.1459999999999999,"15min":3.1459999999999999},"last":1.0920000000000001}]},{"osd":7,"last update":"Mon May 22 15:42:58 2023","interfaces":[{"interface":"back","average":{"1min":1.083,"5min":1.1359999999999999,"15min":1.1220000000000001},"min":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.58299999999999996},"max":{"1min":1.5029999999999999,"5min":2.5990000000000002,"15min":2.5990000000000002},"last":1.0389999999999999},{"interface":"front","average":{"1min":1.024,"5min":1.0960000000000001,"15min":1.0980000000000001},"min":{"1min":0.60499999999999998,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":1.381,"5min":3.8799999999999999,"15min":3.8799999999999999},"last":1.0269999999999999}]}]},{"osd":1,"up_from":13,"seq":55834575051,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":24840,"kb_used_data":1408,"kb_used_omap":0,"kb_used_meta":23424,"kb_avail":94347000,"statfs":{"total":96636764160,"available":96611328000,"internally_reserved":0,"allocated":1441792,"data_stored":757702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":23986176},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 15:43:13 2023","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.61599999999999999,"15min":0.63400000000000001},"min":{"1min":0.26300000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.90900000000000003,"5min":1.2270000000000001,"15min":1.7689999999999999},"last":0.439},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.746,"15min":0.76400000000000001},"min":{"1min":0.29299999999999998,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":1.0860000000000001,"5min":1.2290000000000001,"15min":1.2529999999999999},"last":0.47299999999999998}]},{"osd":2,"last update":"Mon May 22 15:43:13 2023","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.747,"15min":0.73499999999999999},"min":{"1min":0.39400000000000002,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":1.1639999999999999,"5min":1.391,"15min":1.7290000000000001},"last":0.503},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.753,"15min":0.76100000000000001},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.27800000000000002},"max":{"1min":1.155,"5min":1.276,"15min":2.4049999999999998},"last":0.45900000000000002}]},{"osd":3,"last update":"Mon May 22 15:43:13 2023","interfaces":[{"interface":"back","average":{"1min":0.69999999999999996,"5min":0.80100000000000005,"15min":0.82699999999999996},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.317},"max":{"1min":0.996,"5min":1.657,"15min":1.657},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.80200000000000005,"15min":0.82799999999999996},"min":{"1min":0.35499999999999998,"5min":0.315,"15min":0.315},"max":{"1min":1.1910000000000001,"5min":2.0569999999999999,"15min":7.5620000000000003},"last":0.53800000000000003}]},{"osd":4,"last update":"Mon May 22 15:43:13 2023","interfaces":[{"interface":"back","average":{"1min":0.96299999999999997,"5min":0.97099999999999997,"15min":0.95799999999999996},"min":{"1min":0.55200000000000005,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":2.9529999999999998,"5min":2.9529999999999998,"15min":2.9529999999999998},"last":0.95099999999999996},{"interface":"front","average":{"1min":1.012,"5min":0.98799999999999999,"15min":0.996},"min":{"1min":0.504,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":2.7759999999999998,"5min":2.7759999999999998,"15min":2.7759999999999998},"last":1.147}]},{"osd":5,"last update":"Mon May 22 15:43:13 2023","interfaces":[{"interface":"back","average":{"1min":0.92200000000000004,"5min":1.0169999999999999,"15min":1.03},"min":{"1min":0.58399999999999996,"5min":0.443,"15min":0.443},"max":{"1min":1.409,"5min":1.9570000000000001,"15min":1.9570000000000001},"last":0.76300000000000001},{"interface":"front","average":{"1min":0.91600000000000004,"5min":1.0189999999999999,"15min":1.0449999999999999},"min":{"1min":0.53400000000000003,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":1.3069999999999999,"5min":2.1200000000000001,"15min":4.4710000000000001},"last":0.90400000000000003}]},{"osd":6,"last update":"Mon May 22 15:43:13 2023","interfaces":[{"interface":"back","average":{"1min":1.075,"5min":1.018,"15min":1.077},"min":{"1min":0.56299999999999994,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":2.831,"5min":2.831,"15min":14.968999999999999},"last":1.018},{"interface":"front","average":{"1min":0.97599999999999998,"5min":1.0620000000000001,"15min":1.127},"min":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":1.456,"5min":1.7629999999999999,"15min":13.242000000000001},"last":0.96299999999999997}]},{"osd":7,"last update":"Mon May 22 15:43:13 2023","interfaces":[{"interface":"back","average":{"1min":1.056,"5min":1.1000000000000001,"15min":1.105},"min":{"1min":0.60299999999999998,"5min":0.56799999999999995,"15min":0.56799999999999995},"max":{"1min":2.4199999999999999,"5min":2.4199999999999999,"15min":2.4199999999999999},"last":0.93700000000000006},{"interface":"front","average":{"1min":1.0700000000000001,"5min":1.129,"15min":1.133},"min":{"1min":0.47299999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":3.8799999999999999,"5min":3.8799999999999999,"15min":5.681},"last":1.0529999999999999}]}]},{"osd":0,"up_from":13,"seq":55834575052,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":24144,"kb_used_data":1992,"kb_used_omap":0,"kb_used_meta":22144,"kb_avail":94347696,"statfs":{"total":96636764160,"available":96612040704,"internally_reserved":0,"allocated":2039808,"data_stored":1348636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":22675456},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon May 22 15:43:08 2023","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":0.66600000000000004,"15min":0.67800000000000005},"min":{"1min":0.33500000000000002,"5min":0.223,"15min":0.223},"max":{"1min":1.466,"5min":1.466,"15min":1.466},"last":0.81499999999999995},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.70399999999999996,"15min":0.73199999999999998},"min":{"1min":0.34200000000000003,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":1.71,"5min":1.71,"15min":1.71},"last":0.73799999999999999}]},{"osd":2,"last update":"Mon May 22 15:43:08 2023","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.72499999999999998,"15min":0.746},"min":{"1min":0.38600000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":1.165,"5min":1.165,"15min":1.4059999999999999},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.75700000000000001,"15min":0.79600000000000004},"min":{"1min":0.38500000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":1.0189999999999999,"5min":1.1850000000000001,"15min":1.3999999999999999},"last":0.95599999999999996}]},{"osd":3,"last update":"Mon May 22 15:43:08 2023","interfaces":[{"interface":"back","average":{"1min":0.754,"5min":0.77300000000000002,"15min":0.76500000000000001},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":1.923,"5min":1.9299999999999999,"15min":1.9299999999999999},"last":0.77800000000000002},{"interface":"front","average":{"1min":0.82599999999999996,"5min":0.85099999999999998,"15min":0.875},"min":{"1min":0.38600000000000001,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":1.758,"5min":1.758,"15min":1.758},"last":0.88600000000000001}]},{"osd":4,"last update":"Mon May 22 15:43:08 2023","interfaces":[{"interface":"back","average":{"1min":0.93300000000000005,"5min":0.94399999999999995,"15min":0.95199999999999996},"min":{"1min":0.63700000000000001,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":1.6000000000000001,"5min":2.923,"15min":2.923},"last":0.92900000000000005},{"interface":"front","average":{"1min":1.0169999999999999,"5min":1.028,"15min":1.028},"min":{"1min":0.64900000000000002,"5min":0.64600000000000002,"15min":0.54900000000000004},"max":{"1min":2.0129999999999999,"5min":2.0129999999999999,"15min":2.0129999999999999},"last":1.0429999999999999}]},{"osd":5,"last update":"Mon May 22 15:43:08 2023","interfaces":[{"interface":"back","average":{"1min":1.0449999999999999,"5min":1.0109999999999999,"15min":1.0409999999999999},"min":{"1min":0.71199999999999997,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.97,"5min":1.97,"15min":1.97},"last":1.147},{"interface":"front","average":{"1min":0.94799999999999995,"5min":1.0249999999999999,"15min":1.01},"min":{"1min":0.57199999999999995,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.804,"5min":5.0519999999999996,"15min":5.0519999999999996},"last":0.86799999999999999}]},{"osd":6,"last update":"Mon May 22 15:43:08 2023","interfaces":[{"interface":"back","average":{"1min":1.1379999999999999,"5min":1.0880000000000001,"15min":1.0880000000000001},"min":{"1min":0.58699999999999997,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":2.117,"5min":2.117,"15min":2.117},"last":0.997},{"interface":"front","average":{"1min":1.123,"5min":1.1399999999999999,"15min":1.1140000000000001},"min":{"1min":0.74299999999999999,"5min":0.499,"15min":0.499},"max":{"1min":2.052,"5min":5.2640000000000002,"15min":5.2640000000000002},"last":1.2390000000000001}]},{"osd":7,"last update":"Mon May 22 15:43:08 2023","interfaces":[{"interface":"back","average":{"1min":1.1519999999999999,"5min":1.1200000000000001,"15min":1.125},"min":{"1min":0.77500000000000002,"5min":0.65700000000000003,"15min":0.58099999999999996},"max":{"1min":1.839,"5min":1.839,"15min":1.839},"last":1.3149999999999999},{"interface":"front","average":{"1min":1.0669999999999999,"5min":1.099,"15min":1.093},"min":{"1min":0.63300000000000001,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":2.0800000000000001,"5min":3.9060000000000001,"15min":3.9060000000000001},"last":1.02}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1570,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-05-22T15:43:34.536 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-05-22T15:43:34.537 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2023-05-22T15:43:34.700 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:34.695+0000 7fd407073640 1 Processor -- start 2023-05-22T15:43:34.700 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:34.695+0000 7fd407073640 1 -- start start 2023-05-22T15:43:34.701 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:34.695+0000 7fd407073640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd40006a830 0x7fd40015fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:34.701 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:34.695+0000 7fd407073640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd40015bfc0 con 0x7fd40006c5f0 2023-05-22T15:43:34.701 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:34.695+0000 7fd407073640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4000cf620 con 0x7fd40006db10 2023-05-22T15:43:34.702 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:34.695+0000 7fd407073640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd40015f790 con 0x7fd40006a830 2023-05-22T15:43:34.702 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:34.695+0000 7fd3fffff640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fd40006c5f0 0x7fd4000cc7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:51784/0 (socket says 172.21.15.16:51784) 2023-05-22T15:43:34.702 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:34.695+0000 7fd3fffff640 1 -- 172.21.15.16:0/364251293 learned_addr learned my addr 172.21.15.16:0/364251293 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:43:34.702 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:34.695+0000 7fd405610640 1 --2- 172.21.15.16:0/364251293 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd40006a830 0x7fd40015fbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:34.703 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:34.695+0000 7fd3ff7fe640 1 -- 172.21.15.16:0/364251293 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2804468184 0 0) 0x7fd3f0001120 con 0x7fd40006c5f0 2023-05-22T15:43:34.704 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:34.695+0000 7fd3ff7fe640 1 -- 172.21.15.16:0/364251293 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd3e8003530 con 0x7fd40006c5f0 2023-05-22T15:43:34.704 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:34.695+0000 7fd3ff7fe640 1 -- 172.21.15.16:0/364251293 <== mon.1 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1301855663 0 0) 0x7fd4000cf620 con 0x7fd40006db10 2023-05-22T15:43:34.704 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:34.695+0000 7fd3ff7fe640 1 -- 172.21.15.16:0/364251293 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd3f0001120 con 0x7fd40006db10 2023-05-22T15:43:34.704 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:34.695+0000 7fd3ff7fe640 1 -- 172.21.15.16:0/364251293 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 432839304 0 0) 0x7fd3e8003530 con 0x7fd40006c5f0 2023-05-22T15:43:34.704 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:34.695+0000 7fd3ff7fe640 1 -- 172.21.15.16:0/364251293 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd40006a830 msgr2=0x7fd40015fbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:43:34.704 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:34.695+0000 7fd3ff7fe640 1 --2- 172.21.15.16:0/364251293 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd40006a830 0x7fd40015fbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:34.704 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:34.695+0000 7fd3ff7fe640 1 -- 172.21.15.16:0/364251293 >> v1:172.21.15.62:6789/0 conn(0x7fd40006db10 legacy=0x7fd40006dee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:34.705 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:34.695+0000 7fd3ff7fe640 1 -- 172.21.15.16:0/364251293 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4000cf900 con 0x7fd40006c5f0 2023-05-22T15:43:34.705 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:34.695+0000 7fd3ff7fe640 1 -- 172.21.15.16:0/364251293 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7fd3f0001810 con 0x7fd40006c5f0 2023-05-22T15:43:34.705 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:34.695+0000 7fd3ff7fe640 1 -- 172.21.15.16:0/364251293 >> v1:172.21.15.16:6789/0 conn(0x7fd40006c5f0 legacy=0x7fd4000cc7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:34.705 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:34.695+0000 7fd3ff7fe640 1 --2- 172.21.15.16:0/364251293 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd40006a830 0x7fd3e8003b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:34.705 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:34.695+0000 7fd3ff7fe640 1 --2- 172.21.15.16:0/364251293 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd3e80040a0 0x7fd3e80047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:34.705 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:34.695+0000 7fd3ff7fe640 1 --2- 172.21.15.16:0/364251293 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd3e8004d10 0x7fd3e8005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:34.705 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:34.695+0000 7fd3ff7fe640 1 -- 172.21.15.16:0/364251293 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd40015f790 con 0x7fd3e8004d10 2023-05-22T15:43:34.706 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:34.695+0000 7fd3ff7fe640 1 -- 172.21.15.16:0/364251293 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3e800ba40 con 0x7fd3e80040a0 2023-05-22T15:43:34.706 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:34.695+0000 7fd3ff7fe640 1 -- 172.21.15.16:0/364251293 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3e800bc00 con 0x7fd40006a830 2023-05-22T15:43:34.706 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:34.695+0000 7fd405610640 1 --2- 172.21.15.16:0/364251293 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd3e8004d10 0x7fd3e8005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:34.706 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:34.699+0000 7fd3fffff640 1 --2- 172.21.15.16:0/364251293 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd3e80040a0 0x7fd3e80047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:34.706 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:34.699+0000 7fd404e0f640 1 --2- 172.21.15.16:0/364251293 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd40006a830 0x7fd3e8003b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:34.706 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:34.699+0000 7fd405610640 1 -- 172.21.15.16:0/364251293 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd40006a830 msgr2=0x7fd3e8003b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:43:34.706 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:34.699+0000 7fd405610640 1 --2- 172.21.15.16:0/364251293 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd40006a830 0x7fd3e8003b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:34.707 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:34.699+0000 7fd405610640 1 -- 172.21.15.16:0/364251293 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd3e80040a0 msgr2=0x7fd3e80047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:43:34.707 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:34.699+0000 7fd405610640 1 --2- 172.21.15.16:0/364251293 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd3e80040a0 0x7fd3e80047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:34.707 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:34.699+0000 7fd405610640 1 -- 172.21.15.16:0/364251293 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3e8003530 con 0x7fd3e8004d10 2023-05-22T15:43:34.709 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:34.699+0000 7fd405610640 1 --2- 172.21.15.16:0/364251293 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd3e8004d10 0x7fd3e8005180 secure :-1 s=READY pgs=1464 cs=0 l=1 rev1=1 crypto rx=0x7fd3f8009f90 tx=0x7fd3f801cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9bb572365be75284 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:43:34.709 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:34.699+0000 7fd3ff7fe640 1 -- 172.21.15.16:0/364251293 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3f80a5a10 con 0x7fd3e8004d10 2023-05-22T15:43:34.709 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:34.699+0000 7fd3ff7fe640 1 -- 172.21.15.16:0/364251293 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3f800f040 con 0x7fd3e8004d10 2023-05-22T15:43:34.710 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:34.699+0000 7fd3ff7fe640 1 -- 172.21.15.16:0/364251293 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3f80a5db0 con 0x7fd3e8004d10 2023-05-22T15:43:34.710 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:34.699+0000 7fd407073640 1 -- 172.21.15.16:0/364251293 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd3e8004d10 msgr2=0x7fd3e8005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:34.710 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:34.699+0000 7fd407073640 1 --2- 172.21.15.16:0/364251293 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd3e8004d10 0x7fd3e8005180 secure :-1 s=READY pgs=1464 cs=0 l=1 rev1=1 crypto rx=0x7fd3f8009f90 tx=0x7fd3f801cc20 comp rx=0 tx=0).stop 2023-05-22T15:43:34.710 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:34.699+0000 7fd405610640 1 -- 172.21.15.16:0/364251293 reap_dead start 2023-05-22T15:43:34.710 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:34.699+0000 7fd407073640 1 -- 172.21.15.16:0/364251293 shutdown_connections 2023-05-22T15:43:34.710 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:34.699+0000 7fd407073640 1 -- 172.21.15.16:0/364251293 >> 172.21.15.16:0/364251293 conn(0x7fd40006f4f0 msgr2=0x7fd40006f8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:43:34.710 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:34.699+0000 7fd407073640 1 -- 172.21.15.16:0/364251293 shutdown_connections 2023-05-22T15:43:34.711 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:34.699+0000 7fd407073640 1 -- 172.21.15.16:0/364251293 wait complete. 2023-05-22T15:43:34.711 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:34.703+0000 7fd407073640 1 Processor -- start 2023-05-22T15:43:34.711 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:34.703+0000 7fd407073640 1 -- start start 2023-05-22T15:43:34.711 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:34.703+0000 7fd407073640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd3e8004d10 0x7fd400143350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:34.711 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:34.703+0000 7fd407073640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd4000b3cd0 0x7fd4000b4140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:34.711 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:34.703+0000 7fd407073640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd400155210 0x7fd400155680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:34.711 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:34.703+0000 7fd407073640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4000b4680 con 0x7fd3e8004d10 2023-05-22T15:43:34.711 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:34.703+0000 7fd407073640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd400157690 con 0x7fd400155210 2023-05-22T15:43:34.712 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:34.703+0000 7fd407073640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd400157970 con 0x7fd4000b3cd0 2023-05-22T15:43:34.712 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:34.703+0000 7fd404e0f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd3e8004d10 0x7fd400143350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:34.712 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:34.703+0000 7fd404e0f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd3e8004d10 0x7fd400143350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36104/0 (socket says 172.21.15.16:36104) 2023-05-22T15:43:34.712 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:34.703+0000 7fd404e0f640 1 -- 172.21.15.16:0/206183787 learned_addr learned my addr 172.21.15.16:0/206183787 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:43:34.712 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:34.703+0000 7fd405610640 1 --2- 172.21.15.16:0/206183787 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd400155210 0x7fd400155680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:34.712 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:34.703+0000 7fd3fffff640 1 --2- 172.21.15.16:0/206183787 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd4000b3cd0 0x7fd4000b4140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:34.713 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:34.703+0000 7fd404e0f640 1 -- 172.21.15.16:0/206183787 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd4000b3cd0 msgr2=0x7fd4000b4140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:34.713 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:34.703+0000 7fd404e0f640 1 --2- 172.21.15.16:0/206183787 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd4000b3cd0 0x7fd4000b4140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:34.713 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:34.703+0000 7fd404e0f640 1 -- 172.21.15.16:0/206183787 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd400155210 msgr2=0x7fd400155680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:34.713 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:34.703+0000 7fd404e0f640 1 --2- 172.21.15.16:0/206183787 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd400155210 0x7fd400155680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:34.713 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:34.703+0000 7fd404e0f640 1 -- 172.21.15.16:0/206183787 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4000cfec0 con 0x7fd3e8004d10 2023-05-22T15:43:34.713 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:34.703+0000 7fd3fffff640 1 --2- 172.21.15.16:0/206183787 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd4000b3cd0 0x7fd4000b4140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:43:34.713 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:34.703+0000 7fd405610640 1 --2- 172.21.15.16:0/206183787 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd400155210 0x7fd400155680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T15:43:34.713 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:34.703+0000 7fd404e0f640 1 --2- 172.21.15.16:0/206183787 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd3e8004d10 0x7fd400143350 secure :-1 s=READY pgs=1465 cs=0 l=1 rev1=1 crypto rx=0x7fd3f400f770 tx=0x7fd3f400fc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:43:34.714 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:34.703+0000 7fd3fd7fa640 1 -- 172.21.15.16:0/206183787 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3f40014b0 con 0x7fd3e8004d10 2023-05-22T15:43:34.714 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:34.703+0000 7fd3fd7fa640 1 -- 172.21.15.16:0/206183787 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3f400fe80 con 0x7fd3e8004d10 2023-05-22T15:43:34.714 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:34.703+0000 7fd3fd7fa640 1 -- 172.21.15.16:0/206183787 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3f400d730 con 0x7fd3e8004d10 2023-05-22T15:43:34.715 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:34.703+0000 7fd407073640 1 -- 172.21.15.16:0/206183787 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4000cfbe0 con 0x7fd3e8004d10 2023-05-22T15:43:34.715 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:34.703+0000 7fd407073640 1 -- 172.21.15.16:0/206183787 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4000cf260 con 0x7fd3e8004d10 2023-05-22T15:43:34.715 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:34.703+0000 7fd407073640 1 -- 172.21.15.16:0/206183787 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd40004dbe0 con 0x7fd3e8004d10 2023-05-22T15:43:34.716 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:34.703+0000 7fd3fd7fa640 1 -- 172.21.15.16:0/206183787 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 41) v1 ==== 84332+0+0 (secure 0 0 0) 0x7fd3f400d890 con 0x7fd3e8004d10 2023-05-22T15:43:34.716 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:34.707+0000 7fd3fd7fa640 1 --2- 172.21.15.16:0/206183787 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd3ec0654d0 0x7fd3ec067980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:34.716 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:34.707+0000 7fd3fd7fa640 1 -- 172.21.15.16:0/206183787 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 6382+0+0 (secure 0 0 0) 0x7fd3f4082ba0 con 0x7fd3e8004d10 2023-05-22T15:43:34.716 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:34.707+0000 7fd3fffff640 1 --2- 172.21.15.16:0/206183787 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd3ec0654d0 0x7fd3ec067980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:34.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:34.707+0000 7fd3fffff640 1 --2- 172.21.15.16:0/206183787 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd3ec0654d0 0x7fd3ec067980 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fd3f0007fd0 tx=0x7fd3f0007a00 comp rx=0 tx=0).ready entity=mgr.7788 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:43:34.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:34.711+0000 7fd3fd7fa640 1 -- 172.21.15.16:0/206183787 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fd3f40505d0 con 0x7fd3e8004d10 2023-05-22T15:43:34.914 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:34.907+0000 7fd407073640 1 -- 172.21.15.16:0/206183787 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7fd4000b6140 con 0x7fd3e8004d10 2023-05-22T15:43:34.921 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:34.915+0000 7fd3fd7fa640 1 -- 172.21.15.16:0/206183787 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v22) v1 ==== 190+0+0 (secure 0 0 0) 0x7fd3f4053bf0 con 0x7fd3e8004d10 2023-05-22T15:43:34.921 INFO:teuthology.orchestra.run.smithi016.stderr:updated caps for client.0 2023-05-22T15:43:34.923 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:34.919+0000 7fd407073640 1 -- 172.21.15.16:0/206183787 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd3ec0654d0 msgr2=0x7fd3ec067980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:34.924 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:34.919+0000 7fd407073640 1 --2- 172.21.15.16:0/206183787 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd3ec0654d0 0x7fd3ec067980 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fd3f0007fd0 tx=0x7fd3f0007a00 comp rx=0 tx=0).stop 2023-05-22T15:43:34.924 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:34.919+0000 7fd407073640 1 -- 172.21.15.16:0/206183787 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd3e8004d10 msgr2=0x7fd400143350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:34.924 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:34.919+0000 7fd407073640 1 --2- 172.21.15.16:0/206183787 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd3e8004d10 0x7fd400143350 secure :-1 s=READY pgs=1465 cs=0 l=1 rev1=1 crypto rx=0x7fd3f400f770 tx=0x7fd3f400fc30 comp rx=0 tx=0).stop 2023-05-22T15:43:34.925 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:34.919+0000 7fd407073640 1 -- 172.21.15.16:0/206183787 shutdown_connections 2023-05-22T15:43:34.925 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:34.919+0000 7fd407073640 1 --2- 172.21.15.16:0/206183787 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd400155210 0x7fd400155680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:34.925 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:34.919+0000 7fd407073640 1 --2- 172.21.15.16:0/206183787 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd3ec0654d0 0x7fd3ec067980 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:34.926 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:34.919+0000 7fd407073640 1 --2- 172.21.15.16:0/206183787 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd4000b3cd0 0x7fd4000b4140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:34.926 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:34.919+0000 7fd407073640 1 --2- 172.21.15.16:0/206183787 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd3e8004d10 0x7fd400143350 unknown :-1 s=CLOSED pgs=1465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:34.926 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:34.919+0000 7fd407073640 1 -- 172.21.15.16:0/206183787 >> 172.21.15.16:0/206183787 conn(0x7fd40006f4f0 msgr2=0x7fd4000afc40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:43:34.926 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:34.919+0000 7fd407073640 1 -- 172.21.15.16:0/206183787 shutdown_connections 2023-05-22T15:43:34.926 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:34.919+0000 7fd407073640 1 -- 172.21.15.16:0/206183787 wait complete. 2023-05-22T15:43:34.938 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-05-22T15:43:35.053 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.047+0000 7f08c1858640 1 Processor -- start 2023-05-22T15:43:35.053 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.047+0000 7f08c1858640 1 -- start start 2023-05-22T15:43:35.055 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.047+0000 7f08c1858640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f08bc0cb720 0x7f08bc0cda30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:35.055 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.047+0000 7f08c1858640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f08bc15c520 0x7f08bc15c9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:35.056 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.047+0000 7f08c1858640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f08bc06c6c0 0x7f08bc06cae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:35.056 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.047+0000 7f08c1858640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08bc0cbaf0 con 0x7f08bc0cb720 2023-05-22T15:43:35.056 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.047+0000 7f08c1858640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08bc0cd020 con 0x7f08bc15c520 2023-05-22T15:43:35.057 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.047+0000 7f08c1858640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08bc06d020 con 0x7f08bc06c6c0 2023-05-22T15:43:35.057 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.047+0000 7f08baffd640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f08bc0cb720 0x7f08bc0cda30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:35.057 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.047+0000 7f08baffd640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f08bc0cb720 0x7f08bc0cda30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36118/0 (socket says 172.21.15.16:36118) 2023-05-22T15:43:35.057 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.047+0000 7f08baffd640 1 -- 172.21.15.16:0/3977553288 learned_addr learned my addr 172.21.15.16:0/3977553288 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:43:35.058 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.047+0000 7f08bb7fe640 1 --2- 172.21.15.16:0/3977553288 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f08bc06c6c0 0x7f08bc06cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:35.058 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.047+0000 7f08baffd640 1 -- 172.21.15.16:0/3977553288 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f08bc06c6c0 msgr2=0x7f08bc06cae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:43:35.058 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.047+0000 7f08baffd640 1 --2- 172.21.15.16:0/3977553288 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f08bc06c6c0 0x7f08bc06cae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:35.059 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.047+0000 7f08baffd640 1 -- 172.21.15.16:0/3977553288 >> v2:172.21.15.62:3300/0 conn(0x7f08bc15c520 msgr2=0x7f08bc15c9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:43:35.059 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.047+0000 7f08baffd640 1 --2- 172.21.15.16:0/3977553288 >> v2:172.21.15.62:3300/0 conn(0x7f08bc15c520 0x7f08bc15c9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:35.059 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.047+0000 7f08baffd640 1 -- 172.21.15.16:0/3977553288 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08bc0cdf70 con 0x7f08bc0cb720 2023-05-22T15:43:35.063 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.051+0000 7f08baffd640 1 --2- 172.21.15.16:0/3977553288 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08bc0cb720 0x7f08bc0cda30 secure :-1 s=READY pgs=1466 cs=0 l=1 rev1=1 crypto rx=0x7f08bc065ed0 tx=0x7f08ac01eba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c66c4d1cc6dc015 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:43:35.064 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.051+0000 7f08b9ffb640 1 -- 172.21.15.16:0/3977553288 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08ac0a7c30 con 0x7f08bc0cb720 2023-05-22T15:43:35.064 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.051+0000 7f08b9ffb640 1 -- 172.21.15.16:0/3977553288 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08ac0a7d90 con 0x7f08bc0cb720 2023-05-22T15:43:35.065 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.051+0000 7f08b9ffb640 1 -- 172.21.15.16:0/3977553288 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08ac0af660 con 0x7f08bc0cb720 2023-05-22T15:43:35.065 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.051+0000 7f08c1858640 1 -- 172.21.15.16:0/3977553288 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08bc0cb720 msgr2=0x7f08bc0cda30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:35.065 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.051+0000 7f08c1858640 1 --2- 172.21.15.16:0/3977553288 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08bc0cb720 0x7f08bc0cda30 secure :-1 s=READY pgs=1466 cs=0 l=1 rev1=1 crypto rx=0x7f08bc065ed0 tx=0x7f08ac01eba0 comp rx=0 tx=0).stop 2023-05-22T15:43:35.065 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.051+0000 7f08c1858640 1 -- 172.21.15.16:0/3977553288 shutdown_connections 2023-05-22T15:43:35.065 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.051+0000 7f08c1858640 1 --2- 172.21.15.16:0/3977553288 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f08bc06c6c0 0x7f08bc06cae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:35.065 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.051+0000 7f08c1858640 1 --2- 172.21.15.16:0/3977553288 >> v2:172.21.15.62:3300/0 conn(0x7f08bc15c520 0x7f08bc15c9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:35.066 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.051+0000 7f08c1858640 1 --2- 172.21.15.16:0/3977553288 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08bc0cb720 0x7f08bc0cda30 unknown :-1 s=CLOSED pgs=1466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:35.066 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.051+0000 7f08c1858640 1 -- 172.21.15.16:0/3977553288 >> 172.21.15.16:0/3977553288 conn(0x7f08bc0b1950 msgr2=0x7f08bc0cf1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:43:35.066 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.051+0000 7f08c1858640 1 -- 172.21.15.16:0/3977553288 shutdown_connections 2023-05-22T15:43:35.066 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.051+0000 7f08c1858640 1 -- 172.21.15.16:0/3977553288 wait complete. 2023-05-22T15:43:35.066 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.055+0000 7f08c1858640 1 Processor -- start 2023-05-22T15:43:35.066 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.055+0000 7f08c1858640 1 -- start start 2023-05-22T15:43:35.066 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.055+0000 7f08c1858640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08bc06c6c0 0x7f08bc1f3e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:35.067 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.055+0000 7f08c1858640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f08bc0cb720 0x7f08bc14e670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:35.067 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.055+0000 7f08c1858640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f08bc15c520 0x7f08bc151fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:35.067 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.055+0000 7f08c1858640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08bc162f30 con 0x7f08bc06c6c0 2023-05-22T15:43:35.067 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.055+0000 7f08c1858640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08bc1f4360 con 0x7f08bc15c520 2023-05-22T15:43:35.067 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.055+0000 7f08c1858640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08bc162320 con 0x7f08bc0cb720 2023-05-22T15:43:35.067 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.055+0000 7f08baffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08bc06c6c0 0x7f08bc1f3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:35.067 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.055+0000 7f08baffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08bc06c6c0 0x7f08bc1f3e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36122/0 (socket says 172.21.15.16:36122) 2023-05-22T15:43:35.067 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.055+0000 7f08baffd640 1 -- 172.21.15.16:0/2991603553 learned_addr learned my addr 172.21.15.16:0/2991603553 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:43:35.068 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.055+0000 7f08bb7fe640 1 --2- 172.21.15.16:0/2991603553 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f08bc15c520 0x7f08bc151fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:35.068 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.055+0000 7f08ba7fc640 1 --2- 172.21.15.16:0/2991603553 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f08bc0cb720 0x7f08bc14e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:35.068 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.055+0000 7f08baffd640 1 -- 172.21.15.16:0/2991603553 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f08bc0cb720 msgr2=0x7f08bc14e670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:35.068 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.055+0000 7f08baffd640 1 --2- 172.21.15.16:0/2991603553 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f08bc0cb720 0x7f08bc14e670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:35.068 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.055+0000 7f08baffd640 1 -- 172.21.15.16:0/2991603553 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f08bc15c520 msgr2=0x7f08bc151fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:35.068 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.055+0000 7f08baffd640 1 --2- 172.21.15.16:0/2991603553 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f08bc15c520 0x7f08bc151fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:35.069 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.055+0000 7f08baffd640 1 -- 172.21.15.16:0/2991603553 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08ac00b770 con 0x7f08bc06c6c0 2023-05-22T15:43:35.069 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.055+0000 7f08ba7fc640 1 --2- 172.21.15.16:0/2991603553 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f08bc0cb720 0x7f08bc14e670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:43:35.069 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.055+0000 7f08baffd640 1 --2- 172.21.15.16:0/2991603553 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08bc06c6c0 0x7f08bc1f3e20 secure :-1 s=READY pgs=1467 cs=0 l=1 rev1=1 crypto rx=0x7f08ac00b2c0 tx=0x7f08ac01b940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:43:35.069 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.055+0000 7f08a3fff640 1 -- 172.21.15.16:0/2991603553 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08ac0ba660 con 0x7f08bc06c6c0 2023-05-22T15:43:35.069 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.055+0000 7f08a3fff640 1 -- 172.21.15.16:0/2991603553 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08ac0ba7e0 con 0x7f08bc06c6c0 2023-05-22T15:43:35.070 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.055+0000 7f08c1858640 1 -- 172.21.15.16:0/2991603553 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08bc0cd510 con 0x7f08bc06c6c0 2023-05-22T15:43:35.070 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.055+0000 7f08a3fff640 1 -- 172.21.15.16:0/2991603553 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08ac0bab90 con 0x7f08bc06c6c0 2023-05-22T15:43:35.070 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.055+0000 7f08c1858640 1 -- 172.21.15.16:0/2991603553 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08bc0cd230 con 0x7f08bc06c6c0 2023-05-22T15:43:35.070 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.055+0000 7f08c1858640 1 -- 172.21.15.16:0/2991603553 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08bc159190 con 0x7f08bc06c6c0 2023-05-22T15:43:35.070 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.055+0000 7f08a3fff640 1 -- 172.21.15.16:0/2991603553 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 41) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f08ac0c2360 con 0x7f08bc06c6c0 2023-05-22T15:43:35.070 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.055+0000 7f08a3fff640 1 --2- 172.21.15.16:0/2991603553 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f08a80654a0 0x7f08a8067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:35.070 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.059+0000 7f08a3fff640 1 -- 172.21.15.16:0/2991603553 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f08ac11da80 con 0x7f08bc06c6c0 2023-05-22T15:43:35.070 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.059+0000 7f08ba7fc640 1 --2- 172.21.15.16:0/2991603553 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f08a80654a0 0x7f08a8067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:35.071 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.059+0000 7f08ba7fc640 1 --2- 172.21.15.16:0/2991603553 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f08a80654a0 0x7f08a8067950 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f08bc1f79a0 tx=0x7f08b0008040 comp rx=0 tx=0).ready entity=mgr.7788 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:43:35.071 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.063+0000 7f08a3fff640 1 -- 172.21.15.16:0/2991603553 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f08ac0eb4e0 con 0x7f08bc06c6c0 2023-05-22T15:43:35.257 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.251+0000 7f08c1858640 1 -- 172.21.15.16:0/2991603553 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f08bc04dbe0 con 0x7f08bc06c6c0 2023-05-22T15:43:35.258 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.251+0000 7f08a3fff640 1 -- 172.21.15.16:0/2991603553 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 43 v43) v1 ==== 94+0+4650 (secure 0 0 0) 0x7f08ac0b4090 con 0x7f08bc06c6c0 2023-05-22T15:43:35.258 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:43:35.258 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":43,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6350,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6833/1554108010","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":1554108010},{"type":"v1","addr":"172.21.15.62:6833","nonce":1554108010}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":39},{"gid":6979,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/2667219028","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":2667219028},{"type":"v1","addr":"172.21.15.62:6835","nonce":2667219028}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":39},{"gid":8013,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6836/1892737356","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1892737356},{"type":"v1","addr":"172.21.15.16:6836","nonce":1892737356}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":39}],"filesystems":[{"mdsmap":{"epoch":43,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-05-22T15:43:30.134589+0000","modified":"2023-05-22T15:43:33.144847+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8016},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8016":{"gid":8016,"name":"a","rank":0,"incarnation":40,"state":"up:active","state_seq":3,"addr":"172.21.15.16:6837/2085936428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6835","nonce":2085936428},{"type":"v1","addr":"172.21.15.16:6837","nonce":2085936428}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2023-05-22T15:43:35.259 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 43 2023-05-22T15:43:35.260 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.255+0000 7f08c1858640 1 -- 172.21.15.16:0/2991603553 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f08a80654a0 msgr2=0x7f08a8067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:35.260 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.255+0000 7f08c1858640 1 --2- 172.21.15.16:0/2991603553 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f08a80654a0 0x7f08a8067950 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f08bc1f79a0 tx=0x7f08b0008040 comp rx=0 tx=0).stop 2023-05-22T15:43:35.260 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.255+0000 7f08c1858640 1 -- 172.21.15.16:0/2991603553 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08bc06c6c0 msgr2=0x7f08bc1f3e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:35.261 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.255+0000 7f08c1858640 1 --2- 172.21.15.16:0/2991603553 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08bc06c6c0 0x7f08bc1f3e20 secure :-1 s=READY pgs=1467 cs=0 l=1 rev1=1 crypto rx=0x7f08ac00b2c0 tx=0x7f08ac01b940 comp rx=0 tx=0).stop 2023-05-22T15:43:35.261 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.255+0000 7f08c1858640 1 -- 172.21.15.16:0/2991603553 shutdown_connections 2023-05-22T15:43:35.261 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.255+0000 7f08c1858640 1 --2- 172.21.15.16:0/2991603553 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f08bc15c520 0x7f08bc151fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:35.261 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.255+0000 7f08c1858640 1 --2- 172.21.15.16:0/2991603553 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f08a80654a0 0x7f08a8067950 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:35.261 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.255+0000 7f08c1858640 1 --2- 172.21.15.16:0/2991603553 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f08bc0cb720 0x7f08bc14e670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:35.261 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.255+0000 7f08c1858640 1 --2- 172.21.15.16:0/2991603553 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08bc06c6c0 0x7f08bc1f3e20 unknown :-1 s=CLOSED pgs=1467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:35.261 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.255+0000 7f08c1858640 1 -- 172.21.15.16:0/2991603553 >> 172.21.15.16:0/2991603553 conn(0x7f08bc0b1950 msgr2=0x7f08bc1512b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:43:35.262 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.255+0000 7f08c1858640 1 -- 172.21.15.16:0/2991603553 shutdown_connections 2023-05-22T15:43:35.262 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.255+0000 7f08c1858640 1 -- 172.21.15.16:0/2991603553 wait complete. 2023-05-22T15:43:35.679 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 43, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-05-22T15:43:30.134589+0000', 'modified': '2023-05-22T15:43:33.144847+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8016}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8016': {'gid': 8016, 'name': 'a', 'rank': 0, 'incarnation': 40, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.16:6837/2085936428', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6835', 'nonce': 2085936428}, {'type': 'v1', 'addr': '172.21.15.16:6837', 'nonce': 2085936428}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-05-22T15:43:35.679 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-05-22T15:43:35.680 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2023-05-22T15:43:35.814 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.811+0000 7faed22ab640 1 Processor -- start 2023-05-22T15:43:35.815 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.811+0000 7faed22ab640 1 -- start start 2023-05-22T15:43:35.815 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.811+0000 7faed22ab640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7faecc0ca4e0 0x7faecc15e460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:35.816 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.811+0000 7faed22ab640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7faecc070df0 0x7faecc071210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:35.816 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.811+0000 7faed22ab640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faecc0b0650 con 0x7faecc0c9b40 2023-05-22T15:43:35.816 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.811+0000 7faed22ab640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faecc0b0370 con 0x7faecc070df0 2023-05-22T15:43:35.816 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.811+0000 7faed22ab640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faecc15d980 con 0x7faecc0ca4e0 2023-05-22T15:43:35.817 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.811+0000 7faed0848640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7faecc070df0 0x7faecc071210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:35.817 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.811+0000 7faecb7fe640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7faecc0c9b40 0x7faecc0afb40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.16:34226/0 (socket says 172.21.15.16:34226) 2023-05-22T15:43:35.817 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.811+0000 7faecbfff640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7faecc0ca4e0 0x7faecc15e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:35.817 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.811+0000 7faecb7fe640 1 -- 172.21.15.16:0/1385563829 learned_addr learned my addr 172.21.15.16:0/1385563829 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:43:35.817 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.811+0000 7faed0848640 1 --2- 172.21.15.16:0/1385563829 >> v2:172.21.15.62:3300/0 conn(0x7faecc070df0 0x7faecc071210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.16:49530/0 (socket says 172.21.15.16:49530) 2023-05-22T15:43:35.818 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.811+0000 7faed0848640 1 -- 172.21.15.16:0/1385563829 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7faecc0ca4e0 msgr2=0x7faecc15e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:43:35.818 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.811+0000 7faed0848640 1 --2- 172.21.15.16:0/1385563829 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7faecc0ca4e0 0x7faecc15e460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:35.818 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.811+0000 7faed0848640 1 -- 172.21.15.16:0/1385563829 >> v1:172.21.15.62:6789/0 conn(0x7faecc0c9b40 legacy=0x7faecc0afb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:35.818 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.811+0000 7faed0848640 1 -- 172.21.15.16:0/1385563829 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faecc15dc60 con 0x7faecc070df0 2023-05-22T15:43:35.818 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.815+0000 7faed0848640 1 --2- 172.21.15.16:0/1385563829 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faecc070df0 0x7faecc071210 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7faeb40097c0 tx=0x7faeb401c3c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=22a3410d0d7b1b96 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:43:35.819 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.815+0000 7faecaffd640 1 -- 172.21.15.16:0/1385563829 <== mon.1 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faeb40ab600 con 0x7faecc070df0 2023-05-22T15:43:35.819 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.815+0000 7faecaffd640 1 -- 172.21.15.16:0/1385563829 <== mon.1 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faeb40a7050 con 0x7faecc070df0 2023-05-22T15:43:35.819 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.815+0000 7faecaffd640 1 -- 172.21.15.16:0/1385563829 <== mon.1 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faeb40ab980 con 0x7faecc070df0 2023-05-22T15:43:35.820 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.815+0000 7faed22ab640 1 -- 172.21.15.16:0/1385563829 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faecc070df0 msgr2=0x7faecc071210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:35.820 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.815+0000 7faed22ab640 1 --2- 172.21.15.16:0/1385563829 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faecc070df0 0x7faecc071210 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7faeb40097c0 tx=0x7faeb401c3c0 comp rx=0 tx=0).stop 2023-05-22T15:43:35.820 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.815+0000 7faed22ab640 1 -- 172.21.15.16:0/1385563829 shutdown_connections 2023-05-22T15:43:35.821 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.815+0000 7faed22ab640 1 --2- 172.21.15.16:0/1385563829 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faecc070df0 0x7faecc071210 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:35.821 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.815+0000 7faed22ab640 1 --2- 172.21.15.16:0/1385563829 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7faecc0ca4e0 0x7faecc15e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:35.821 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.815+0000 7faed22ab640 1 -- 172.21.15.16:0/1385563829 >> 172.21.15.16:0/1385563829 conn(0x7faecc069720 msgr2=0x7faecc069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:43:35.821 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.815+0000 7faed22ab640 1 -- 172.21.15.16:0/1385563829 shutdown_connections 2023-05-22T15:43:35.821 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.815+0000 7faed22ab640 1 -- 172.21.15.16:0/1385563829 wait complete. 2023-05-22T15:43:35.823 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.815+0000 7faed22ab640 1 Processor -- start 2023-05-22T15:43:35.823 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.815+0000 7faed22ab640 1 -- start start 2023-05-22T15:43:35.823 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.815+0000 7faed22ab640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faecc070df0 0x7faecc0c6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:35.823 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.815+0000 7faed22ab640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7faecc0c9b40 0x7faecc0b1110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:35.823 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.815+0000 7faed22ab640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faecc0ca4e0 0x7faecc1f4a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:35.823 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.819+0000 7faed22ab640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faecc157700 con 0x7faecc070df0 2023-05-22T15:43:35.824 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.819+0000 7faed22ab640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faecc0c67a0 con 0x7faecc0ca4e0 2023-05-22T15:43:35.824 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.819+0000 7faed22ab640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faecc15c170 con 0x7faecc0c9b40 2023-05-22T15:43:35.824 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.819+0000 7faecbfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faecc070df0 0x7faecc0c6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:35.824 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.819+0000 7faecbfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faecc070df0 0x7faecc0c6260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36136/0 (socket says 172.21.15.16:36136) 2023-05-22T15:43:35.824 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.819+0000 7faecbfff640 1 -- 172.21.15.16:0/2360619448 learned_addr learned my addr 172.21.15.16:0/2360619448 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:43:35.824 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.819+0000 7faed0848640 1 --2- 172.21.15.16:0/2360619448 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faecc0ca4e0 0x7faecc1f4a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:35.824 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.819+0000 7faecb7fe640 1 --2- 172.21.15.16:0/2360619448 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7faecc0c9b40 0x7faecc0b1110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:35.825 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.819+0000 7faecbfff640 1 -- 172.21.15.16:0/2360619448 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7faecc0c9b40 msgr2=0x7faecc0b1110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:35.825 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.819+0000 7faecbfff640 1 --2- 172.21.15.16:0/2360619448 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7faecc0c9b40 0x7faecc0b1110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:35.825 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.819+0000 7faecbfff640 1 -- 172.21.15.16:0/2360619448 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faecc0ca4e0 msgr2=0x7faecc1f4a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:35.825 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.819+0000 7faecbfff640 1 --2- 172.21.15.16:0/2360619448 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faecc0ca4e0 0x7faecc1f4a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:35.826 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.819+0000 7faecbfff640 1 -- 172.21.15.16:0/2360619448 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faeb4009420 con 0x7faecc070df0 2023-05-22T15:43:35.826 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.819+0000 7faecbfff640 1 --2- 172.21.15.16:0/2360619448 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faecc070df0 0x7faecc0c6260 secure :-1 s=READY pgs=1468 cs=0 l=1 rev1=1 crypto rx=0x7faec4009d10 tx=0x7faec400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:43:35.829 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.819+0000 7faec8ff9640 1 -- 172.21.15.16:0/2360619448 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faec4009b80 con 0x7faecc070df0 2023-05-22T15:43:35.829 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.819+0000 7faec8ff9640 1 -- 172.21.15.16:0/2360619448 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faec400f040 con 0x7faecc070df0 2023-05-22T15:43:35.830 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.819+0000 7faed22ab640 1 -- 172.21.15.16:0/2360619448 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faecc0c8e80 con 0x7faecc070df0 2023-05-22T15:43:35.830 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.819+0000 7faec8ff9640 1 -- 172.21.15.16:0/2360619448 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faec4014680 con 0x7faecc070df0 2023-05-22T15:43:35.830 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.819+0000 7faed22ab640 1 -- 172.21.15.16:0/2360619448 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faecc0ca8b0 con 0x7faecc070df0 2023-05-22T15:43:35.830 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.819+0000 7faec8ff9640 1 -- 172.21.15.16:0/2360619448 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 41) v1 ==== 84332+0+0 (secure 0 0 0) 0x7faec40147e0 con 0x7faecc070df0 2023-05-22T15:43:35.830 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.819+0000 7faec8ff9640 1 --2- 172.21.15.16:0/2360619448 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7faeac0654a0 0x7faeac067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:35.830 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.819+0000 7faec8ff9640 1 -- 172.21.15.16:0/2360619448 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 6382+0+0 (secure 0 0 0) 0x7faec40839d0 con 0x7faecc070df0 2023-05-22T15:43:35.830 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.819+0000 7faecb7fe640 1 --2- 172.21.15.16:0/2360619448 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7faeac0654a0 0x7faeac067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:35.831 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.819+0000 7faecb7fe640 1 --2- 172.21.15.16:0/2360619448 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7faeac0654a0 0x7faeac067950 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7faecc0b2230 tx=0x7faebc009040 comp rx=0 tx=0).ready entity=mgr.7788 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:43:35.831 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.819+0000 7faed53cd000 1 Processor -- start 2023-05-22T15:43:35.831 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.819+0000 7faed53cd000 1 -- start start 2023-05-22T15:43:35.831 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.823+0000 7faed53cd000 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x5633121f7fd0 0x56331246ac30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:35.832 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.823+0000 7faed53cd000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56331246b170 0x56331246b540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:35.832 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.823+0000 7faed53cd000 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56331241ea10 0x56331241ede0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:35.832 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.823+0000 7faed53cd000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x5633122e8bd0 con 0x56331246b170 2023-05-22T15:43:35.832 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.823+0000 7faed53cd000 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x563312205dd0 con 0x56331241ea10 2023-05-22T15:43:35.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.823+0000 7faed53cd000 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x563312276640 con 0x5633121f7fd0 2023-05-22T15:43:35.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.823+0000 7faecb7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56331246b170 0x56331246b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:35.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.823+0000 7faecb7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56331246b170 0x56331246b540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36138/0 (socket says 172.21.15.16:36138) 2023-05-22T15:43:35.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.823+0000 7faecb7fe640 1 -- 172.21.15.16:0/2246586381 learned_addr learned my addr 172.21.15.16:0/2246586381 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:43:35.834 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.823+0000 7faecbfff640 1 --2- 172.21.15.16:0/2246586381 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x5633121f7fd0 0x56331246ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:35.834 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.823+0000 7faed0848640 1 --2- 172.21.15.16:0/2246586381 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56331241ea10 0x56331241ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:35.834 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.823+0000 7faecb7fe640 1 -- 172.21.15.16:0/2246586381 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x5633121f7fd0 msgr2=0x56331246ac30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:43:35.834 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.823+0000 7faecb7fe640 1 --2- 172.21.15.16:0/2246586381 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x5633121f7fd0 0x56331246ac30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:35.834 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.823+0000 7faecb7fe640 1 -- 172.21.15.16:0/2246586381 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56331241ea10 msgr2=0x56331241ede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:43:35.835 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.823+0000 7faecb7fe640 1 --2- 172.21.15.16:0/2246586381 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56331241ea10 0x56331241ede0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:35.835 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.823+0000 7faecb7fe640 1 -- 172.21.15.16:0/2246586381 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5633122350b0 con 0x56331246b170 2023-05-22T15:43:35.835 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.823+0000 7faecb7fe640 1 --2- 172.21.15.16:0/2246586381 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56331246b170 0x56331246b540 secure :-1 s=READY pgs=1469 cs=0 l=1 rev1=1 crypto rx=0x7faebc001e70 tx=0x7faebc011000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a85cd1d7aa38a8ae server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:43:35.835 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.823+0000 7faec1ffb640 1 -- 172.21.15.16:0/2246586381 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faebc005e10 con 0x56331246b170 2023-05-22T15:43:35.835 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.823+0000 7faec1ffb640 1 -- 172.21.15.16:0/2246586381 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faebc00e7b0 con 0x56331246b170 2023-05-22T15:43:35.836 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.823+0000 7faec1ffb640 1 -- 172.21.15.16:0/2246586381 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faebc00eb20 con 0x56331246b170 2023-05-22T15:43:35.836 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.823+0000 7faed53cd000 1 -- 172.21.15.16:0/2246586381 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56331246b170 msgr2=0x56331246b540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:35.836 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.823+0000 7faed53cd000 1 --2- 172.21.15.16:0/2246586381 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56331246b170 0x56331246b540 secure :-1 s=READY pgs=1469 cs=0 l=1 rev1=1 crypto rx=0x7faebc001e70 tx=0x7faebc011000 comp rx=0 tx=0).stop 2023-05-22T15:43:35.836 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.823+0000 7faed53cd000 1 -- 172.21.15.16:0/2246586381 shutdown_connections 2023-05-22T15:43:35.837 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.823+0000 7faed53cd000 1 --2- 172.21.15.16:0/2246586381 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56331241ea10 0x56331241ede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:35.837 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.823+0000 7faed53cd000 1 --2- 172.21.15.16:0/2246586381 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56331246b170 0x56331246b540 unknown :-1 s=CLOSED pgs=1469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:35.837 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.823+0000 7faed53cd000 1 --2- 172.21.15.16:0/2246586381 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x5633121f7fd0 0x56331246ac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:35.837 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.823+0000 7faed53cd000 1 -- 172.21.15.16:0/2246586381 >> 172.21.15.16:0/2246586381 conn(0x5633122c7270 msgr2=0x56331244e220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:43:35.838 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.823+0000 7faed53cd000 1 -- 172.21.15.16:0/2246586381 shutdown_connections 2023-05-22T15:43:35.838 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.823+0000 7faed53cd000 1 -- 172.21.15.16:0/2246586381 wait complete. 2023-05-22T15:43:35.838 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.823+0000 7faed53cd000 1 -- start start 2023-05-22T15:43:35.838 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.823+0000 7faed53cd000 1 Processor -- start 2023-05-22T15:43:35.843 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.823+0000 7faed53cd000 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x5633121f7fd0 0x563312471ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:35.843 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.823+0000 7faed53cd000 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56331241ea10 0x5633124720e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:35.843 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.823+0000 7faed53cd000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56331246b170 0x56331244f450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:35.843 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.823+0000 7faed53cd000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x56331226f780 con 0x56331246b170 2023-05-22T15:43:35.844 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.823+0000 7faed53cd000 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x56331222d250 con 0x56331241ea10 2023-05-22T15:43:35.844 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.823+0000 7faed53cd000 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x56331222cef0 con 0x5633121f7fd0 2023-05-22T15:43:35.844 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.823+0000 7faed53cd000 10 client.0 _ll_register_callbacks cb 0x56331240cb50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-05-22T15:43:35.844 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.823+0000 7faed0848640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56331246b170 0x56331244f450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:35.845 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.823+0000 7faed0848640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56331246b170 0x56331244f450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36154/0 (socket says 172.21.15.16:36154) 2023-05-22T15:43:35.845 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.823+0000 7faed0848640 1 -- 172.21.15.16:0/2045979496 learned_addr learned my addr 172.21.15.16:0/2045979496 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:43:35.845 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.827+0000 7faecb7fe640 1 --2- 172.21.15.16:0/2045979496 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56331241ea10 0x5633124720e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:35.845 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.827+0000 7faecbfff640 1 --2- 172.21.15.16:0/2045979496 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x5633121f7fd0 0x563312471ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:35.846 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.827+0000 7faed0848640 1 -- 172.21.15.16:0/2045979496 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x5633121f7fd0 msgr2=0x563312471ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:43:35.846 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.827+0000 7faed0848640 1 --2- 172.21.15.16:0/2045979496 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x5633121f7fd0 0x563312471ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:35.846 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.827+0000 7faed0848640 1 -- 172.21.15.16:0/2045979496 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56331241ea10 msgr2=0x5633124720e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:43:35.846 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.827+0000 7faed0848640 1 --2- 172.21.15.16:0/2045979496 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56331241ea10 0x5633124720e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:35.846 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.827+0000 7faed0848640 1 -- 172.21.15.16:0/2045979496 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faebc00f040 con 0x56331246b170 2023-05-22T15:43:35.847 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.827+0000 7faecb7fe640 1 --2- 172.21.15.16:0/2045979496 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56331241ea10 0x5633124720e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T15:43:35.847 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.827+0000 7faed0848640 1 --2- 172.21.15.16:0/2045979496 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56331246b170 0x56331244f450 secure :-1 s=READY pgs=1470 cs=0 l=1 rev1=1 crypto rx=0x7faeb401aa70 tx=0x7faeb401ac00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ada50051a0e5fb3 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:43:35.847 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.827+0000 7faeaaffd640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2023-05-22T15:43:35.848 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.827+0000 7faeaaffd640 1 -- 172.21.15.16:0/2045979496 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faeb40abb00 con 0x56331246b170 2023-05-22T15:43:35.848 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.827+0000 7faeaaffd640 1 -- 172.21.15.16:0/2045979496 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(118..118 src has 1..118) v4 ==== 6382+0+0 (secure 0 0 0) 0x7faeb40abc60 con 0x56331246b170 2023-05-22T15:43:35.848 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.827+0000 7faed53cd000 1 -- 172.21.15.16:0/2045979496 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x563312233c60 con 0x56331246b170 2023-05-22T15:43:35.848 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.827+0000 7faeaaffd640 1 -- 172.21.15.16:0/2045979496 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) v2 ==== 24+0+0 (secure 0 0 0) 0x7faeb40abe20 con 0x56331246b170 2023-05-22T15:43:35.849 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.827+0000 7faed53cd000 10 client.8148 fetch_fsmap learned FSMap version 43 2023-05-22T15:43:35.849 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.827+0000 7faed53cd000 1 -- 172.21.15.16:0/2045979496 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=2+}) v3 -- 0x563311f6c890 con 0x56331246b170 2023-05-22T15:43:35.849 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.827+0000 7faeaaffd640 1 -- 172.21.15.16:0/2045979496 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x563312233c60 con 0x56331246b170 2023-05-22T15:43:35.849 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.827+0000 7faeaaffd640 1 -- 172.21.15.16:0/2045979496 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 43) v1 ==== 3106+0+0 (secure 0 0 0) 0x7faeb40be020 con 0x56331246b170 2023-05-22T15:43:35.850 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.827+0000 7faed53cd000 10 client.8148 fetch_fsmap finished waiting for FSMap version 43 2023-05-22T15:43:35.850 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.827+0000 7faed53cd000 10 client.8148 resolve_mds: resolved name 'a' to daemon mds.a 2023-05-22T15:43:35.850 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.827+0000 7faed53cd000 20 client.8148 populate_metadata read hostname 'smithi016' 2023-05-22T15:43:35.850 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.827+0000 7faed53cd000 1 --2- 172.21.15.16:0/2045979496 >> [v2:172.21.15.16:6835/2085936428,v1:172.21.15.16:6837/2085936428] conn(0x5633124501c0 0x563312450590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:35.851 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.827+0000 7faed53cd000 4 client.8148 mds_command: new command op to 8016 tid=0[{"prefix": "get_command_descriptions"}] 2023-05-22T15:43:35.851 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.827+0000 7faed53cd000 1 -- 172.21.15.16:0/2045979496 --> [v2:172.21.15.16:6835/2085936428,v1:172.21.15.16:6837/2085936428] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56331226e0d0 con 0x5633124501c0 2023-05-22T15:43:35.851 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.827+0000 7faecbfff640 1 --2- 172.21.15.16:0/2045979496 >> [v2:172.21.15.16:6835/2085936428,v1:172.21.15.16:6837/2085936428] conn(0x5633124501c0 0x563312450590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:35.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.827+0000 7faecbfff640 1 --2- 172.21.15.16:0/2045979496 >> [v2:172.21.15.16:6835/2085936428,v1:172.21.15.16:6837/2085936428] conn(0x5633124501c0 0x563312450590 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ce0ad3ad99765c7d server_cookie=2aafae0b3cd89f43 in_seq=0 out_seq=0 2023-05-22T15:43:35.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.827+0000 7faeaaffd640 10 client.8148 ms_handle_connect on v2:172.21.15.16:6835/2085936428 2023-05-22T15:43:35.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.827+0000 7faeaaffd640 1 -- 172.21.15.16:0/2045979496 <== mds.0 v2:172.21.15.16:6835/2085936428 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x56331226e0d0 con 0x5633124501c0 2023-05-22T15:43:35.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.827+0000 7faeaaffd640 10 client.8148 handle_command_reply: tid=0 2023-05-22T15:43:35.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.827+0000 7faed53cd000 1 client.8148 shutdown 2023-05-22T15:43:35.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.827+0000 7faed53cd000 1 -- 172.21.15.16:0/2045979496 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56331246b170 msgr2=0x56331244f450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:35.853 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.827+0000 7faed53cd000 1 --2- 172.21.15.16:0/2045979496 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56331246b170 0x56331244f450 secure :-1 s=READY pgs=1470 cs=0 l=1 rev1=1 crypto rx=0x7faeb401aa70 tx=0x7faeb401ac00 comp rx=0 tx=0).stop 2023-05-22T15:43:35.853 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.827+0000 7faed53cd000 1 -- 172.21.15.16:0/2045979496 shutdown_connections 2023-05-22T15:43:35.853 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.827+0000 7faed53cd000 1 --2- 172.21.15.16:0/2045979496 >> [v2:172.21.15.16:6835/2085936428,v1:172.21.15.16:6837/2085936428] conn(0x5633124501c0 0x563312450590 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:35.853 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.827+0000 7faed53cd000 1 --2- 172.21.15.16:0/2045979496 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56331246b170 0x56331244f450 unknown :-1 s=CLOSED pgs=1470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:35.853 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.827+0000 7faeaaffd640 0 client.8148 ms_handle_reset on v2:172.21.15.16:6835/2085936428 2023-05-22T15:43:35.853 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.827+0000 7faed53cd000 1 --2- 172.21.15.16:0/2045979496 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56331241ea10 0x5633124720e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:35.853 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.827+0000 7faed53cd000 1 --2- 172.21.15.16:0/2045979496 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x5633121f7fd0 0x563312471ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:35.854 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.827+0000 7faed53cd000 1 -- 172.21.15.16:0/2045979496 >> 172.21.15.16:0/2045979496 conn(0x5633122c7270 msgr2=0x56331241e100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:43:35.854 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.827+0000 7faed53cd000 1 -- 172.21.15.16:0/2045979496 shutdown_connections 2023-05-22T15:43:35.854 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.827+0000 7faed53cd000 1 -- 172.21.15.16:0/2045979496 wait complete. 2023-05-22T15:43:35.854 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.827+0000 7faed53cd000 20 client.8148 trim_cache size 0 max 16384 2023-05-22T15:43:35.860 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.843+0000 7faed53cd000 1 Processor -- start 2023-05-22T15:43:35.860 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.843+0000 7faed53cd000 1 -- start start 2023-05-22T15:43:35.861 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.843+0000 7faed53cd000 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5633121f7fd0 0x563312421e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:35.861 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.843+0000 7faed53cd000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56331241ea10 0x563312422340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:35.861 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.843+0000 7faed53cd000 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x56331246b170 0x563312447ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:35.861 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.843+0000 7faed53cd000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x563311f2d940 con 0x56331241ea10 2023-05-22T15:43:35.861 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.843+0000 7faed53cd000 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x563312478620 con 0x5633121f7fd0 2023-05-22T15:43:35.861 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.843+0000 7faed53cd000 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x5633122722e0 con 0x56331246b170 2023-05-22T15:43:35.861 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.843+0000 7faecb7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56331241ea10 0x563312422340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:35.861 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.843+0000 7faecb7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56331241ea10 0x563312422340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36160/0 (socket says 172.21.15.16:36160) 2023-05-22T15:43:35.862 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.843+0000 7faecb7fe640 1 -- 172.21.15.16:0/2126598170 learned_addr learned my addr 172.21.15.16:0/2126598170 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:43:35.862 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.843+0000 7faecbfff640 1 --2- 172.21.15.16:0/2126598170 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5633121f7fd0 0x563312421e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:35.862 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.843+0000 7faecb7fe640 1 -- 172.21.15.16:0/2126598170 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x56331246b170 msgr2=0x563312447ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:43:35.862 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.843+0000 7faecb7fe640 1 --2- 172.21.15.16:0/2126598170 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x56331246b170 0x563312447ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:35.862 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.843+0000 7faecb7fe640 1 -- 172.21.15.16:0/2126598170 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5633121f7fd0 msgr2=0x563312421e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:43:35.863 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.843+0000 7faecb7fe640 1 --2- 172.21.15.16:0/2126598170 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5633121f7fd0 0x563312421e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:35.863 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.843+0000 7faecb7fe640 1 -- 172.21.15.16:0/2126598170 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563312235360 con 0x56331241ea10 2023-05-22T15:43:35.863 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.843+0000 7faecb7fe640 1 --2- 172.21.15.16:0/2126598170 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56331241ea10 0x563312422340 secure :-1 s=READY pgs=1471 cs=0 l=1 rev1=1 crypto rx=0x7faebc005890 tx=0x7faebc007000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e53b307c0cef351d server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:43:35.863 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.843+0000 7faec27fc640 1 -- 172.21.15.16:0/2126598170 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faebc005e10 con 0x56331241ea10 2023-05-22T15:43:35.864 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.843+0000 7faec27fc640 1 -- 172.21.15.16:0/2126598170 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faebc019030 con 0x56331241ea10 2023-05-22T15:43:35.864 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.843+0000 7faec27fc640 1 -- 172.21.15.16:0/2126598170 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faebc00e9d0 con 0x56331241ea10 2023-05-22T15:43:35.864 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.847+0000 7faed53cd000 1 -- 172.21.15.16:0/2126598170 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56331241ea10 msgr2=0x563312422340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:35.865 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.847+0000 7faed53cd000 1 --2- 172.21.15.16:0/2126598170 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56331241ea10 0x563312422340 secure :-1 s=READY pgs=1471 cs=0 l=1 rev1=1 crypto rx=0x7faebc005890 tx=0x7faebc007000 comp rx=0 tx=0).stop 2023-05-22T15:43:35.865 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.847+0000 7faed53cd000 1 -- 172.21.15.16:0/2126598170 shutdown_connections 2023-05-22T15:43:35.865 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.847+0000 7faed53cd000 1 --2- 172.21.15.16:0/2126598170 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x56331246b170 0x563312447ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:35.865 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.847+0000 7faed53cd000 1 --2- 172.21.15.16:0/2126598170 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56331241ea10 0x563312422340 unknown :-1 s=CLOSED pgs=1471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:35.866 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.847+0000 7faed53cd000 1 --2- 172.21.15.16:0/2126598170 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5633121f7fd0 0x563312421e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:35.866 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.847+0000 7faed53cd000 1 -- 172.21.15.16:0/2126598170 >> 172.21.15.16:0/2126598170 conn(0x5633122c7270 msgr2=0x56331241e080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:43:35.866 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.847+0000 7faed53cd000 1 -- 172.21.15.16:0/2126598170 shutdown_connections 2023-05-22T15:43:35.866 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.847+0000 7faed53cd000 1 -- 172.21.15.16:0/2126598170 wait complete. 2023-05-22T15:43:35.867 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.847+0000 7faed53cd000 1 -- start start 2023-05-22T15:43:35.867 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.847+0000 7faed53cd000 1 Processor -- start 2023-05-22T15:43:35.867 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.847+0000 7faed53cd000 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5633121f7fd0 0x5633124c3c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:35.868 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.847+0000 7faed53cd000 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x56331241ea10 0x5633124c41d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:35.868 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.847+0000 7faed53cd000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56331246b170 0x5633124c4710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:35.868 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.847+0000 7faed53cd000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x5633122778d0 con 0x56331246b170 2023-05-22T15:43:35.868 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.847+0000 7faed53cd000 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x56331220e720 con 0x5633121f7fd0 2023-05-22T15:43:35.869 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.847+0000 7faed53cd000 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x563312269b20 con 0x56331241ea10 2023-05-22T15:43:35.869 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.847+0000 7faed53cd000 10 client.0 _ll_register_callbacks cb 0x5633122bfeb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-05-22T15:43:35.869 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.847+0000 7faed0848640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56331246b170 0x5633124c4710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:35.870 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.847+0000 7faed0848640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56331246b170 0x5633124c4710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36168/0 (socket says 172.21.15.16:36168) 2023-05-22T15:43:35.870 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.847+0000 7faed0848640 1 -- 172.21.15.16:0/2481207069 learned_addr learned my addr 172.21.15.16:0/2481207069 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:43:35.870 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.847+0000 7faecbfff640 1 --2- 172.21.15.16:0/2481207069 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5633121f7fd0 0x5633124c3c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:35.871 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.847+0000 7faecb7fe640 1 --2- 172.21.15.16:0/2481207069 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x56331241ea10 0x5633124c41d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:35.871 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.847+0000 7faed0848640 1 -- 172.21.15.16:0/2481207069 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x56331241ea10 msgr2=0x5633124c41d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:43:35.871 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.847+0000 7faed0848640 1 --2- 172.21.15.16:0/2481207069 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x56331241ea10 0x5633124c41d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:35.871 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.847+0000 7faed0848640 1 -- 172.21.15.16:0/2481207069 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5633121f7fd0 msgr2=0x5633124c3c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:43:35.871 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.847+0000 7faed0848640 1 --2- 172.21.15.16:0/2481207069 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5633121f7fd0 0x5633124c3c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:35.872 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.847+0000 7faed0848640 1 -- 172.21.15.16:0/2481207069 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5633122350b0 con 0x56331246b170 2023-05-22T15:43:35.872 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.847+0000 7faed0848640 1 --2- 172.21.15.16:0/2481207069 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56331246b170 0x5633124c4710 secure :-1 s=READY pgs=1472 cs=0 l=1 rev1=1 crypto rx=0x7faeb4008010 tx=0x7faeb4019160 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7439ada89c8b2bab server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:43:35.872 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.847+0000 7faeaaffd640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2023-05-22T15:43:35.872 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.847+0000 7faeaaffd640 1 -- 172.21.15.16:0/2481207069 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faeb40c0a70 con 0x56331246b170 2023-05-22T15:43:35.873 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.847+0000 7faeaaffd640 1 -- 172.21.15.16:0/2481207069 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(118..118 src has 1..118) v4 ==== 6382+0+0 (secure 0 0 0) 0x7faeb40cca60 con 0x56331246b170 2023-05-22T15:43:35.873 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.847+0000 7faed53cd000 1 -- 172.21.15.16:0/2481207069 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5633124251c0 con 0x56331246b170 2023-05-22T15:43:35.873 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.847+0000 7faeaaffd640 1 -- 172.21.15.16:0/2481207069 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) v2 ==== 24+0+0 (secure 0 0 0) 0x7faeb4006a70 con 0x56331246b170 2023-05-22T15:43:35.873 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.847+0000 7faed53cd000 10 client.8154 fetch_fsmap learned FSMap version 43 2023-05-22T15:43:35.874 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.847+0000 7faed53cd000 1 -- 172.21.15.16:0/2481207069 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=2+}) v3 -- 0x56331221abe0 con 0x56331246b170 2023-05-22T15:43:35.874 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.847+0000 7faeaaffd640 1 -- 172.21.15.16:0/2481207069 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x5633124251c0 con 0x56331246b170 2023-05-22T15:43:35.874 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.847+0000 7faeaaffd640 1 -- 172.21.15.16:0/2481207069 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 43) v1 ==== 3106+0+0 (secure 0 0 0) 0x7faeb40be040 con 0x56331246b170 2023-05-22T15:43:35.874 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.847+0000 7faed53cd000 10 client.8154 fetch_fsmap finished waiting for FSMap version 43 2023-05-22T15:43:35.875 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.847+0000 7faed53cd000 10 client.8154 resolve_mds: resolved name 'a' to daemon mds.a 2023-05-22T15:43:35.875 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.847+0000 7faed53cd000 20 client.8154 populate_metadata read hostname 'smithi016' 2023-05-22T15:43:35.875 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.847+0000 7faed53cd000 1 --2- 172.21.15.16:0/2481207069 >> [v2:172.21.15.16:6835/2085936428,v1:172.21.15.16:6837/2085936428] conn(0x5633124c4c50 0x5633124c5020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:35.875 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.847+0000 7faed53cd000 4 client.8154 mds_command: new command op to 8016 tid=0[{"prefix": "status"}] 2023-05-22T15:43:35.875 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.847+0000 7faed53cd000 1 -- 172.21.15.16:0/2481207069 --> [v2:172.21.15.16:6835/2085936428,v1:172.21.15.16:6837/2085936428] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5633122164d0 con 0x5633124c4c50 2023-05-22T15:43:35.875 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.847+0000 7faecbfff640 1 --2- 172.21.15.16:0/2481207069 >> [v2:172.21.15.16:6835/2085936428,v1:172.21.15.16:6837/2085936428] conn(0x5633124c4c50 0x5633124c5020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:35.875 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.847+0000 7faeaaffd640 10 client.8154 ms_handle_connect on v2:172.21.15.16:6835/2085936428 2023-05-22T15:43:35.876 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.847+0000 7faecbfff640 1 --2- 172.21.15.16:0/2481207069 >> [v2:172.21.15.16:6835/2085936428,v1:172.21.15.16:6837/2085936428] conn(0x5633124c4c50 0x5633124c5020 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cdb2a8c562053d13 server_cookie=2af9b842357c490 in_seq=0 out_seq=0 2023-05-22T15:43:35.876 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.847+0000 7faeaaffd640 1 -- 172.21.15.16:0/2481207069 <== mds.0 v2:172.21.15.16:6835/2085936428 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+333 (crc 0 0 0) 0x5633122164d0 con 0x5633124c4c50 2023-05-22T15:43:35.876 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.847+0000 7faeaaffd640 10 client.8154 handle_command_reply: tid=0 2023-05-22T15:43:35.876 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.847+0000 7faed53cd000 1 client.8154 shutdown 2023-05-22T15:43:35.877 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.847+0000 7faed53cd000 1 -- 172.21.15.16:0/2481207069 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56331246b170 msgr2=0x5633124c4710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:35.877 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.847+0000 7faed53cd000 1 --2- 172.21.15.16:0/2481207069 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56331246b170 0x5633124c4710 secure :-1 s=READY pgs=1472 cs=0 l=1 rev1=1 crypto rx=0x7faeb4008010 tx=0x7faeb4019160 comp rx=0 tx=0).stop 2023-05-22T15:43:35.877 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.847+0000 7faed53cd000 1 -- 172.21.15.16:0/2481207069 shutdown_connections 2023-05-22T15:43:35.877 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.847+0000 7faed53cd000 1 --2- 172.21.15.16:0/2481207069 >> [v2:172.21.15.16:6835/2085936428,v1:172.21.15.16:6837/2085936428] conn(0x5633124c4c50 0x5633124c5020 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:35.877 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.847+0000 7faed53cd000 1 --2- 172.21.15.16:0/2481207069 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56331246b170 0x5633124c4710 unknown :-1 s=CLOSED pgs=1472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:35.877 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.847+0000 7faed53cd000 1 --2- 172.21.15.16:0/2481207069 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x56331241ea10 0x5633124c41d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:35.877 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.847+0000 7faed53cd000 1 --2- 172.21.15.16:0/2481207069 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5633121f7fd0 0x5633124c3c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:35.877 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.847+0000 7faed53cd000 1 -- 172.21.15.16:0/2481207069 >> 172.21.15.16:0/2481207069 conn(0x5633122c7270 msgr2=0x563312420df0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:43:35.878 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.851+0000 7faeaaffd640 0 client.8154 ms_handle_reset on v2:172.21.15.16:6835/2085936428 2023-05-22T15:43:35.878 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.851+0000 7faed53cd000 1 -- 172.21.15.16:0/2481207069 shutdown_connections 2023-05-22T15:43:35.878 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.851+0000 7faed53cd000 1 -- 172.21.15.16:0/2481207069 wait complete. 2023-05-22T15:43:35.878 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.851+0000 7faed53cd000 20 client.8154 trim_cache size 0 max 16384 2023-05-22T15:43:35.879 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-05-22T15:43:35.879 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_fsid": "b2719834-7847-4d98-90a6-f65d57f85c49", 2023-05-22T15:43:35.879 INFO:teuthology.orchestra.run.smithi016.stdout: "whoami": 0, 2023-05-22T15:43:35.879 INFO:teuthology.orchestra.run.smithi016.stdout: "id": 8016, 2023-05-22T15:43:35.879 INFO:teuthology.orchestra.run.smithi016.stdout: "want_state": "up:active", 2023-05-22T15:43:35.880 INFO:teuthology.orchestra.run.smithi016.stdout: "state": "up:active", 2023-05-22T15:43:35.880 INFO:teuthology.orchestra.run.smithi016.stdout: "fs_name": "cephfs", 2023-05-22T15:43:35.880 INFO:teuthology.orchestra.run.smithi016.stdout: "rank_uptime": 5.7146188049999997, 2023-05-22T15:43:35.880 INFO:teuthology.orchestra.run.smithi016.stdout: "mdsmap_epoch": 43, 2023-05-22T15:43:35.880 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch": 118, 2023-05-22T15:43:35.880 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch_barrier": 118, 2023-05-22T15:43:35.880 INFO:teuthology.orchestra.run.smithi016.stdout: "uptime": 10.262002367999999 2023-05-22T15:43:35.881 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-05-22T15:43:35.882 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.851+0000 7faed22ab640 1 -- 172.21.15.16:0/2360619448 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7faeac0654a0 msgr2=0x7faeac067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:35.882 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.851+0000 7faed22ab640 1 --2- 172.21.15.16:0/2360619448 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7faeac0654a0 0x7faeac067950 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7faecc0b2230 tx=0x7faebc009040 comp rx=0 tx=0).stop 2023-05-22T15:43:35.882 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.851+0000 7faed22ab640 1 -- 172.21.15.16:0/2360619448 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faecc070df0 msgr2=0x7faecc0c6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:35.882 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.851+0000 7faed22ab640 1 --2- 172.21.15.16:0/2360619448 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faecc070df0 0x7faecc0c6260 secure :-1 s=READY pgs=1468 cs=0 l=1 rev1=1 crypto rx=0x7faec4009d10 tx=0x7faec400c490 comp rx=0 tx=0).stop 2023-05-22T15:43:35.882 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.851+0000 7faed22ab640 1 -- 172.21.15.16:0/2360619448 shutdown_connections 2023-05-22T15:43:35.883 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.851+0000 7faed22ab640 1 --2- 172.21.15.16:0/2360619448 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faecc0ca4e0 0x7faecc1f4a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:35.883 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.851+0000 7faed22ab640 1 --2- 172.21.15.16:0/2360619448 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7faeac0654a0 0x7faeac067950 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:35.883 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.851+0000 7faed22ab640 1 --2- 172.21.15.16:0/2360619448 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7faecc0c9b40 0x7faecc0b1110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:35.883 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.851+0000 7faed22ab640 1 --2- 172.21.15.16:0/2360619448 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faecc070df0 0x7faecc0c6260 unknown :-1 s=CLOSED pgs=1468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:35.883 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.851+0000 7faed22ab640 1 -- 172.21.15.16:0/2360619448 >> 172.21.15.16:0/2360619448 conn(0x7faecc069720 msgr2=0x7faecc0b2260 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:43:35.883 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.851+0000 7faed22ab640 1 -- 172.21.15.16:0/2360619448 shutdown_connections 2023-05-22T15:43:35.883 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.851+0000 7faed22ab640 1 -- 172.21.15.16:0/2360619448 wait complete. 2023-05-22T15:43:35.884 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-05-22T15:43:35.998 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.995+0000 7f963a44c640 1 Processor -- start 2023-05-22T15:43:35.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.995+0000 7f963a44c640 1 -- start start 2023-05-22T15:43:36.001 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.995+0000 7f963a44c640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f963406dd50 0x7f963415cce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:36.001 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.995+0000 7f963a44c640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f96340ceee0 0x7f963406bf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:36.001 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.995+0000 7f963a44c640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96340ce160 con 0x7f96340b29a0 2023-05-22T15:43:36.002 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.995+0000 7f963a44c640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f963415d2f0 con 0x7f963406dd50 2023-05-22T15:43:36.002 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.995+0000 7f963a44c640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96340ce400 con 0x7f96340ceee0 2023-05-22T15:43:36.002 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.995+0000 7f96389e9640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f96340ceee0 0x7f963406bf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:36.002 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.995+0000 7f9633fff640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f963406dd50 0x7f963415cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:36.003 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.995+0000 7f96337fe640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f96340b29a0 0x7f963406e120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.16:34242/0 (socket says 172.21.15.16:34242) 2023-05-22T15:43:36.003 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.995+0000 7f9633fff640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f963406dd50 0x7f963415cce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.16:49570/0 (socket says 172.21.15.16:49570) 2023-05-22T15:43:36.003 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.995+0000 7f96389e9640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f96340ceee0 0x7f963406bf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3301/0 says I am v2:172.21.15.16:35292/0 (socket says 172.21.15.16:35292) 2023-05-22T15:43:36.003 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.995+0000 7f96337fe640 1 -- 172.21.15.16:0/2350553705 learned_addr learned my addr 172.21.15.16:0/2350553705 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:43:36.004 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.995+0000 7f9633fff640 1 -- 172.21.15.16:0/2350553705 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f96340ceee0 msgr2=0x7f963406bf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:43:36.004 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.995+0000 7f9633fff640 1 --2- 172.21.15.16:0/2350553705 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f96340ceee0 0x7f963406bf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:36.004 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.995+0000 7f9633fff640 1 -- 172.21.15.16:0/2350553705 >> v1:172.21.15.62:6789/0 conn(0x7f96340b29a0 legacy=0x7f963406e120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:36.005 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.995+0000 7f9633fff640 1 -- 172.21.15.16:0/2350553705 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96340ce6e0 con 0x7f963406dd50 2023-05-22T15:43:36.009 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.999+0000 7f9633fff640 1 --2- 172.21.15.16:0/2350553705 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f963406dd50 0x7f963415cce0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f96240097c0 tx=0x7f962401c360 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1992dc818b4a5dd1 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:43:36.009 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.999+0000 7f9632ffd640 1 -- 172.21.15.16:0/2350553705 <== mon.1 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96240ab600 con 0x7f963406dd50 2023-05-22T15:43:36.010 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.999+0000 7f9632ffd640 1 -- 172.21.15.16:0/2350553705 <== mon.1 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96240a7050 con 0x7f963406dd50 2023-05-22T15:43:36.010 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.999+0000 7f9632ffd640 1 -- 172.21.15.16:0/2350553705 <== mon.1 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96240ab980 con 0x7f963406dd50 2023-05-22T15:43:36.010 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.999+0000 7f963a44c640 1 -- 172.21.15.16:0/2350553705 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f963406dd50 msgr2=0x7f963415cce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:36.010 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.999+0000 7f963a44c640 1 --2- 172.21.15.16:0/2350553705 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f963406dd50 0x7f963415cce0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f96240097c0 tx=0x7f962401c360 comp rx=0 tx=0).stop 2023-05-22T15:43:36.011 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.999+0000 7f963a44c640 1 -- 172.21.15.16:0/2350553705 shutdown_connections 2023-05-22T15:43:36.011 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.999+0000 7f963a44c640 1 --2- 172.21.15.16:0/2350553705 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f96340ceee0 0x7f963406bf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:36.011 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.999+0000 7f963a44c640 1 --2- 172.21.15.16:0/2350553705 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f963406dd50 0x7f963415cce0 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:36.011 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.999+0000 7f963a44c640 1 -- 172.21.15.16:0/2350553705 >> 172.21.15.16:0/2350553705 conn(0x7f963406f5c0 msgr2=0x7f963406f9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:43:36.012 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.999+0000 7f963a44c640 1 -- 172.21.15.16:0/2350553705 shutdown_connections 2023-05-22T15:43:36.012 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.999+0000 7f963a44c640 1 -- 172.21.15.16:0/2350553705 wait complete. 2023-05-22T15:43:36.012 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.999+0000 7f963a44c640 1 Processor -- start 2023-05-22T15:43:36.013 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.999+0000 7f963a44c640 1 -- start start 2023-05-22T15:43:36.013 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.999+0000 7f963a44c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f963406dd50 0x7f9634153ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:36.013 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.999+0000 7f963a44c640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f96340b29a0 0x7f963415b0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:36.013 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.999+0000 7f963a44c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f96340ceee0 0x7f963415b620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:36.013 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.999+0000 7f963a44c640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9634158d20 con 0x7f963406dd50 2023-05-22T15:43:36.014 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.999+0000 7f963a44c640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96341580a0 con 0x7f96340ceee0 2023-05-22T15:43:36.014 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.999+0000 7f963a44c640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9634158380 con 0x7f96340b29a0 2023-05-22T15:43:36.014 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.999+0000 7f9633fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f963406dd50 0x7f9634153ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:36.015 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.999+0000 7f9633fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f963406dd50 0x7f9634153ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36172/0 (socket says 172.21.15.16:36172) 2023-05-22T15:43:36.015 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.999+0000 7f9633fff640 1 -- 172.21.15.16:0/155517150 learned_addr learned my addr 172.21.15.16:0/155517150 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:43:36.015 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.999+0000 7f96337fe640 1 --2- 172.21.15.16:0/155517150 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f96340b29a0 0x7f963415b0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:36.015 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.999+0000 7f96389e9640 1 --2- 172.21.15.16:0/155517150 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f96340ceee0 0x7f963415b620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:36.016 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:35.999+0000 7f9633fff640 1 -- 172.21.15.16:0/155517150 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f96340b29a0 msgr2=0x7f963415b0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:36.016 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:36.003+0000 7f9633fff640 1 --2- 172.21.15.16:0/155517150 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f96340b29a0 0x7f963415b0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:36.016 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:36.003+0000 7f9633fff640 1 -- 172.21.15.16:0/155517150 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f96340ceee0 msgr2=0x7f963415b620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:36.016 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:36.003+0000 7f9633fff640 1 --2- 172.21.15.16:0/155517150 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f96340ceee0 0x7f963415b620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:36.017 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:36.003+0000 7f9633fff640 1 -- 172.21.15.16:0/155517150 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9624009420 con 0x7f963406dd50 2023-05-22T15:43:36.017 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:36.003+0000 7f96389e9640 1 --2- 172.21.15.16:0/155517150 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f96340ceee0 0x7f963415b620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T15:43:36.017 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:36.003+0000 7f9633fff640 1 --2- 172.21.15.16:0/155517150 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f963406dd50 0x7f9634153ab0 secure :-1 s=READY pgs=1473 cs=0 l=1 rev1=1 crypto rx=0x7f9624019f90 tx=0x7f962401aba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:43:36.018 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:36.003+0000 7f9630ff9640 1 -- 172.21.15.16:0/155517150 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96240b7650 con 0x7f963406dd50 2023-05-22T15:43:36.018 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:36.003+0000 7f9630ff9640 1 -- 172.21.15.16:0/155517150 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96240b77d0 con 0x7f963406dd50 2023-05-22T15:43:36.018 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:36.003+0000 7f963a44c640 1 -- 172.21.15.16:0/155517150 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96340d0190 con 0x7f963406dd50 2023-05-22T15:43:36.018 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:36.003+0000 7f9630ff9640 1 -- 172.21.15.16:0/155517150 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96240b7b80 con 0x7f963406dd50 2023-05-22T15:43:36.019 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:36.003+0000 7f963a44c640 1 -- 172.21.15.16:0/155517150 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96340d0000 con 0x7f963406dd50 2023-05-22T15:43:36.019 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:36.003+0000 7f963a44c640 1 -- 172.21.15.16:0/155517150 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96340d0320 con 0x7f963406dd50 2023-05-22T15:43:36.019 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:36.003+0000 7f9630ff9640 1 -- 172.21.15.16:0/155517150 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 41) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f96240b7ce0 con 0x7f963406dd50 2023-05-22T15:43:36.020 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:36.003+0000 7f9630ff9640 1 --2- 172.21.15.16:0/155517150 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f96200654a0 0x7f9620067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:36.020 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:36.003+0000 7f9630ff9640 1 -- 172.21.15.16:0/155517150 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f962411d5e0 con 0x7f963406dd50 2023-05-22T15:43:36.020 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:36.003+0000 7f96337fe640 1 --2- 172.21.15.16:0/155517150 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f96200654a0 0x7f9620067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:36.020 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:36.003+0000 7f96337fe640 1 --2- 172.21.15.16:0/155517150 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f96200654a0 0x7f9620067950 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f96340b1ac0 tx=0x7f9628009040 comp rx=0 tx=0).ready entity=mgr.7788 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:43:36.021 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:36.007+0000 7f9630ff9640 1 -- 172.21.15.16:0/155517150 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f96240eaf90 con 0x7f963406dd50 2023-05-22T15:43:36.196 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:36.191+0000 7f963a44c640 1 -- 172.21.15.16:0/155517150 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f96340ca690 con 0x7f963406dd50 2023-05-22T15:43:36.197 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:36.191+0000 7f9630ff9640 1 -- 172.21.15.16:0/155517150 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v43) v1 ==== 53+0+73 (secure 0 0 0) 0x7f96240ee5b0 con 0x7f963406dd50 2023-05-22T15:43:36.198 INFO:teuthology.orchestra.run.smithi016.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-05-22T15:43:36.200 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:36.195+0000 7f963a44c640 1 -- 172.21.15.16:0/155517150 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f96200654a0 msgr2=0x7f9620067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:36.200 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:36.195+0000 7f963a44c640 1 --2- 172.21.15.16:0/155517150 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f96200654a0 0x7f9620067950 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f96340b1ac0 tx=0x7f9628009040 comp rx=0 tx=0).stop 2023-05-22T15:43:36.201 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:36.195+0000 7f963a44c640 1 -- 172.21.15.16:0/155517150 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f963406dd50 msgr2=0x7f9634153ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:36.201 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:36.195+0000 7f963a44c640 1 --2- 172.21.15.16:0/155517150 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f963406dd50 0x7f9634153ab0 secure :-1 s=READY pgs=1473 cs=0 l=1 rev1=1 crypto rx=0x7f9624019f90 tx=0x7f962401aba0 comp rx=0 tx=0).stop 2023-05-22T15:43:36.201 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:36.195+0000 7f963a44c640 1 -- 172.21.15.16:0/155517150 shutdown_connections 2023-05-22T15:43:36.201 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:36.195+0000 7f963a44c640 1 --2- 172.21.15.16:0/155517150 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f96340ceee0 0x7f963415b620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:36.201 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:36.195+0000 7f963a44c640 1 --2- 172.21.15.16:0/155517150 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f96200654a0 0x7f9620067950 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:36.201 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:36.195+0000 7f963a44c640 1 --2- 172.21.15.16:0/155517150 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f96340b29a0 0x7f963415b0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:36.201 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:36.195+0000 7f963a44c640 1 --2- 172.21.15.16:0/155517150 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f963406dd50 0x7f9634153ab0 unknown :-1 s=CLOSED pgs=1473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:36.202 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:36.195+0000 7f963a44c640 1 -- 172.21.15.16:0/155517150 >> 172.21.15.16:0/155517150 conn(0x7f963406f5c0 msgr2=0x7f963415ee80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:43:36.202 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:36.195+0000 7f963a44c640 1 -- 172.21.15.16:0/155517150 shutdown_connections 2023-05-22T15:43:36.202 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:36.195+0000 7f963a44c640 1 -- 172.21.15.16:0/155517150 wait complete. 2023-05-22T15:43:36.215 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-05-22T15:43:36.215 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-05-22T15:43:36.215 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi016.front.sepia.ceph.com 2023-05-22T15:43:36.216 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-05-22T15:43:36.216 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-05-22T15:43:36.216 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-05-22T15:43:36.217 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-05-22T15:43:36.217 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-05-22T15:43:36.217 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-05-22T15:43:36.217 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:43:36.218 DEBUG:teuthology.orchestra.run.smithi016:> ip addr 2023-05-22T15:43:36.224 INFO:teuthology.orchestra.run.smithi016.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-05-22T15:43:36.225 INFO:teuthology.orchestra.run.smithi016.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-05-22T15:43:36.225 INFO:teuthology.orchestra.run.smithi016.stdout: inet 127.0.0.1/8 scope host lo 2023-05-22T15:43:36.225 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-05-22T15:43:36.225 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 ::1/128 scope host 2023-05-22T15:43:36.225 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-05-22T15:43:36.225 INFO:teuthology.orchestra.run.smithi016.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-05-22T15:43:36.225 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:6b:fd:e0 brd ff:ff:ff:ff:ff:ff 2023-05-22T15:43:36.226 INFO:teuthology.orchestra.run.smithi016.stdout: altname enp7s0f0 2023-05-22T15:43:36.226 INFO:teuthology.orchestra.run.smithi016.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-05-22T15:43:36.226 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:6b:fd:e1 brd ff:ff:ff:ff:ff:ff 2023-05-22T15:43:36.226 INFO:teuthology.orchestra.run.smithi016.stdout: altname enp7s0f1 2023-05-22T15:43:36.226 INFO:teuthology.orchestra.run.smithi016.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2023-05-22T15:43:36.226 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:bd:15:64 brd ff:ff:ff:ff:ff:ff 2023-05-22T15:43:36.226 INFO:teuthology.orchestra.run.smithi016.stdout: altname enp1s0f0 2023-05-22T15:43:36.227 INFO:teuthology.orchestra.run.smithi016.stdout: inet 172.21.15.16/20 brd 172.21.15.255 scope global dynamic ens1f0 2023-05-22T15:43:36.227 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft 41500sec preferred_lft 41500sec 2023-05-22T15:43:36.227 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::ec4:7aff:febd:1564/64 scope link 2023-05-22T15:43:36.227 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-05-22T15:43:36.227 INFO:teuthology.orchestra.run.smithi016.stdout:5: ens1f1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-05-22T15:43:36.227 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:bd:15:65 brd ff:ff:ff:ff:ff:ff 2023-05-22T15:43:36.227 INFO:teuthology.orchestra.run.smithi016.stdout: altname enp1s0f1 2023-05-22T15:43:36.228 INFO:teuthology.orchestra.run.smithi016.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2023-05-22T15:43:36.228 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 02:42:1f:b8:98:0a brd ff:ff:ff:ff:ff:ff 2023-05-22T15:43:36.228 INFO:teuthology.orchestra.run.smithi016.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2023-05-22T15:43:36.228 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-05-22T15:43:36.228 INFO:teuthology.orchestra.run.smithi016.stdout:9: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-05-22T15:43:36.228 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 5e:1f:84:6d:79:ae brd ff:ff:ff:ff:ff:ff 2023-05-22T15:43:36.228 INFO:teuthology.orchestra.run.smithi016.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-05-22T15:43:36.228 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-05-22T15:43:36.229 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::5c1f:84ff:fe6d:79ae/64 scope link 2023-05-22T15:43:36.229 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-05-22T15:43:36.229 INFO:teuthology.orchestra.run.smithi016.stdout:10: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-05-22T15:43:36.229 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0e:5c:f6:1f:82:2f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-05-22T15:43:36.229 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::c5c:f6ff:fe1f:822f/64 scope link 2023-05-22T15:43:36.229 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-05-22T15:43:36.229 INFO:teuthology.orchestra.run.smithi016.stdout:11: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-05-22T15:43:36.230 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether c6:a6:3d:69:d0:db brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-05-22T15:43:36.230 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::c4a6:3dff:fe69:d0db/64 scope link 2023-05-22T15:43:36.230 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-05-22T15:43:36.230 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:43:36.231 DEBUG:teuthology.orchestra.run.smithi016:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-05-22T15:43:36.276 INFO:teuthology.orchestra.run.smithi016.stdout:1 2023-05-22T15:43:36.277 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:43:36.278 DEBUG:teuthology.orchestra.run.smithi016:> route 2023-05-22T15:43:36.324 INFO:teuthology.orchestra.run.smithi016.stdout:Kernel IP routing table 2023-05-22T15:43:36.324 INFO:teuthology.orchestra.run.smithi016.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-05-22T15:43:36.324 INFO:teuthology.orchestra.run.smithi016.stdout:default _gateway 0.0.0.0 UG 0 0 0 ens1f0 2023-05-22T15:43:36.325 INFO:teuthology.orchestra.run.smithi016.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2023-05-22T15:43:36.325 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 0 0 0 ens1f0 2023-05-22T15:43:36.325 INFO:teuthology.orchestra.run.smithi016.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-05-22T15:43:36.325 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:43:36.326 DEBUG:teuthology.orchestra.run.smithi016:> (cd / && exec bash -c ' 2023-05-22T15:43:36.326 DEBUG:teuthology.orchestra.run.smithi016:> set -e 2023-05-22T15:43:36.326 DEBUG:teuthology.orchestra.run.smithi016:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2023-05-22T15:43:36.326 DEBUG:teuthology.orchestra.run.smithi016:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2023-05-22T15:43:36.327 DEBUG:teuthology.orchestra.run.smithi016:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2023-05-22T15:43:36.327 DEBUG:teuthology.orchestra.run.smithi016:> ') 2023-05-22T15:43:36.400 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:43:36.400 DEBUG:teuthology.orchestra.run.smithi016:> ip netns list 2023-05-22T15:43:36.446 INFO:teuthology.orchestra.run.smithi016.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-05-22T15:43:36.447 INFO:teuthology.orchestra.run.smithi016.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-05-22T15:43:36.447 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-05-22T15:43:36.447 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:43:36.448 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip link set brx.0 up 2023-05-22T15:43:36.507 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2023-05-22T15:43:36.508 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-05-22T15:43:36.508 DEBUG:teuthology.orchestra.run.smithi016:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-05-22T15:43:36.556 INFO:teuthology.orchestra.run.smithi016.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-05-22T15:43:36.556 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-05-22T15:43:36.557 DEBUG:teuthology.orchestra.run.smithi016:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-05-22T15:43:36.602 DEBUG:teuthology.orchestra.run.smithi016:> sudo modprobe fuse 2023-05-22T15:43:36.660 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-05-22T15:43:36.661 DEBUG:teuthology.orchestra.run.smithi016:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-05-22T15:43:36.715 INFO:teuthology.orchestra.run.smithi016.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-05-22T15:43:36.716 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-05-22T15:43:36.717 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:43:36.718 DEBUG:teuthology.orchestra.run.smithi016:> ls /sys/fs/fuse/connections 2023-05-22T15:43:36.763 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2023-05-22T15:43:36.763 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2023-05-22T15:43:36.808 DEBUG:teuthology.orchestra.run.smithi016:> sudo modprobe fuse 2023-05-22T15:43:36.825 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-05-22T15:43:36.826 DEBUG:teuthology.orchestra.run.smithi016:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-05-22T15:43:36.881 INFO:teuthology.orchestra.run.smithi016.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-05-22T15:43:36.882 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-05-22T15:43:36.883 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:43:36.883 DEBUG:teuthology.orchestra.run.smithi016:> ls /sys/fs/fuse/connections 2023-05-22T15:43:36.895 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi016.stderr:2023-05-22T15:43:36.890+0000 7f2900c0c580 -1 init, newargv = 0x55d5e9dc5e40 newargc=15 2023-05-22T15:43:36.895 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi016.stderr:ceph-fuse[59097]: starting ceph client 2023-05-22T15:43:36.902 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi016.stderr:ceph-fuse[59097]: starting fuse 2023-05-22T15:43:38.001 DEBUG:teuthology.orchestra.run.smithi016:> sudo modprobe fuse 2023-05-22T15:43:38.017 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-05-22T15:43:38.018 DEBUG:teuthology.orchestra.run.smithi016:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-05-22T15:43:38.072 INFO:teuthology.orchestra.run.smithi016.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-05-22T15:43:38.073 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-05-22T15:43:38.073 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:43:38.074 DEBUG:teuthology.orchestra.run.smithi016:> ls /sys/fs/fuse/connections 2023-05-22T15:43:38.119 INFO:teuthology.orchestra.run.smithi016.stdout:45 2023-05-22T15:43:38.120 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [45] 2023-05-22T15:43:38.121 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2023-05-22T15:43:38.122 DEBUG:teuthology.orchestra.run.smithi016:> import glob 2023-05-22T15:43:38.122 DEBUG:teuthology.orchestra.run.smithi016:> import re 2023-05-22T15:43:38.123 DEBUG:teuthology.orchestra.run.smithi016:> import os 2023-05-22T15:43:38.123 DEBUG:teuthology.orchestra.run.smithi016:> import subprocess 2023-05-22T15:43:38.124 DEBUG:teuthology.orchestra.run.smithi016:> 2023-05-22T15:43:38.124 DEBUG:teuthology.orchestra.run.smithi016:> def _find_admin_socket(client_name): 2023-05-22T15:43:38.124 DEBUG:teuthology.orchestra.run.smithi016:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2023-05-22T15:43:38.125 DEBUG:teuthology.orchestra.run.smithi016:> files = glob.glob(asok_path) 2023-05-22T15:43:38.125 DEBUG:teuthology.orchestra.run.smithi016:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2023-05-22T15:43:38.125 DEBUG:teuthology.orchestra.run.smithi016:> 2023-05-22T15:43:38.126 DEBUG:teuthology.orchestra.run.smithi016:> # Given a non-glob path, it better be there 2023-05-22T15:43:38.126 DEBUG:teuthology.orchestra.run.smithi016:> if "*" not in asok_path: 2023-05-22T15:43:38.127 DEBUG:teuthology.orchestra.run.smithi016:> assert(len(files) == 1) 2023-05-22T15:43:38.127 DEBUG:teuthology.orchestra.run.smithi016:> return files[0] 2023-05-22T15:43:38.128 DEBUG:teuthology.orchestra.run.smithi016:> 2023-05-22T15:43:38.129 DEBUG:teuthology.orchestra.run.smithi016:> for f in files: 2023-05-22T15:43:38.129 DEBUG:teuthology.orchestra.run.smithi016:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-05-22T15:43:38.130 DEBUG:teuthology.orchestra.run.smithi016:> if os.path.exists("/proc/{0}".format(pid)): 2023-05-22T15:43:38.131 DEBUG:teuthology.orchestra.run.smithi016:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-05-22T15:43:38.133 DEBUG:teuthology.orchestra.run.smithi016:> contents = proc_f.read() 2023-05-22T15:43:38.133 DEBUG:teuthology.orchestra.run.smithi016:> if mountpoint in contents: 2023-05-22T15:43:38.133 DEBUG:teuthology.orchestra.run.smithi016:> return f 2023-05-22T15:43:38.134 DEBUG:teuthology.orchestra.run.smithi016:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-05-22T15:43:38.134 DEBUG:teuthology.orchestra.run.smithi016:> 2023-05-22T15:43:38.134 DEBUG:teuthology.orchestra.run.smithi016:> print(_find_admin_socket("client.0")) 2023-05-22T15:43:38.135 DEBUG:teuthology.orchestra.run.smithi016:> ' 2023-05-22T15:43:38.247 INFO:teuthology.orchestra.run.smithi016.stdout:/var/run/ceph/ceph-client.0.59097.asok 2023-05-22T15:43:38.427 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.59097.asok 2023-05-22T15:43:38.428 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:43:38.428 DEBUG:teuthology.orchestra.run.smithi016:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.59097.asok status 2023-05-22T15:43:38.590 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-05-22T15:43:38.591 INFO:teuthology.orchestra.run.smithi016.stdout: "metadata": { 2023-05-22T15:43:38.591 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph_sha1": "9c9ba4df22024d9d8d6cf9f98a4b96751fc78d0c", 2023-05-22T15:43:38.591 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph_version": "ceph version 18.0.0-3192-g9c9ba4df (9c9ba4df22024d9d8d6cf9f98a4b96751fc78d0c) reef (dev)", 2023-05-22T15:43:38.591 INFO:teuthology.orchestra.run.smithi016.stdout: "entity_id": "0", 2023-05-22T15:43:38.591 INFO:teuthology.orchestra.run.smithi016.stdout: "hostname": "smithi016", 2023-05-22T15:43:38.591 INFO:teuthology.orchestra.run.smithi016.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2023-05-22T15:43:38.592 INFO:teuthology.orchestra.run.smithi016.stdout: "pid": "59097", 2023-05-22T15:43:38.592 INFO:teuthology.orchestra.run.smithi016.stdout: "root": "/" 2023-05-22T15:43:38.592 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:43:38.592 INFO:teuthology.orchestra.run.smithi016.stdout: "dentry_count": 0, 2023-05-22T15:43:38.592 INFO:teuthology.orchestra.run.smithi016.stdout: "dentry_pinned_count": 0, 2023-05-22T15:43:38.592 INFO:teuthology.orchestra.run.smithi016.stdout: "id": 8163, 2023-05-22T15:43:38.592 INFO:teuthology.orchestra.run.smithi016.stdout: "inst": { 2023-05-22T15:43:38.592 INFO:teuthology.orchestra.run.smithi016.stdout: "name": { 2023-05-22T15:43:38.593 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "client", 2023-05-22T15:43:38.593 INFO:teuthology.orchestra.run.smithi016.stdout: "num": 8163 2023-05-22T15:43:38.593 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:43:38.593 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": { 2023-05-22T15:43:38.593 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-05-22T15:43:38.593 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "192.168.0.1:0", 2023-05-22T15:43:38.593 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 1371959459 2023-05-22T15:43:38.593 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T15:43:38.594 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:43:38.594 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": { 2023-05-22T15:43:38.594 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-05-22T15:43:38.594 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "192.168.0.1:0", 2023-05-22T15:43:38.594 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 1371959459 2023-05-22T15:43:38.594 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:43:38.594 INFO:teuthology.orchestra.run.smithi016.stdout: "inst_str": "client.8163 192.168.0.1:0/1371959459", 2023-05-22T15:43:38.594 INFO:teuthology.orchestra.run.smithi016.stdout: "addr_str": "192.168.0.1:0/1371959459", 2023-05-22T15:43:38.595 INFO:teuthology.orchestra.run.smithi016.stdout: "inode_count": 1, 2023-05-22T15:43:38.595 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_epoch": 43, 2023-05-22T15:43:38.595 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_epoch": 118, 2023-05-22T15:43:38.595 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_epoch_barrier": 0, 2023-05-22T15:43:38.595 INFO:teuthology.orchestra.run.smithi016.stdout: "blocklisted": false, 2023-05-22T15:43:38.595 INFO:teuthology.orchestra.run.smithi016.stdout: "fs_name": "cephfs" 2023-05-22T15:43:38.595 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-05-22T15:43:38.605 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:43:38.605 DEBUG:teuthology.orchestra.run.smithi016:> stat --file-system '--printf=%T 2023-05-22T15:43:38.606 DEBUG:teuthology.orchestra.run.smithi016:> ' -- /home/ubuntu/cephtest/mnt.0 2023-05-22T15:43:38.611 INFO:teuthology.orchestra.run.smithi016.stdout:fuseblk 2023-05-22T15:43:38.611 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2023-05-22T15:43:38.612 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:43:38.612 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-05-22T15:43:38.667 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2023-05-22T15:43:38.839 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:38.834+0000 7f3b46d14640 1 Processor -- start 2023-05-22T15:43:38.839 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:38.834+0000 7f3b46d14640 1 -- start start 2023-05-22T15:43:38.841 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:38.834+0000 7f3b46d14640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f3b4015c2f0 0x7f3b40162e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:38.842 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:38.834+0000 7f3b46d14640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b40158eb0 con 0x7f3b40070600 2023-05-22T15:43:38.842 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:38.834+0000 7f3b46d14640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b4006cf20 con 0x7f3b4006dab0 2023-05-22T15:43:38.842 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:38.834+0000 7f3b46d14640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b4006de80 con 0x7f3b4015c2f0 2023-05-22T15:43:38.842 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:38.834+0000 7f3b452b1640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f3b4015c2f0 0x7f3b40162e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:38.842 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:38.834+0000 7f3b3ffff640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f3b40070600 0x7f3b4015bbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:51796/0 (socket says 172.21.15.16:51796) 2023-05-22T15:43:38.842 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:38.834+0000 7f3b452b1640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f3b4015c2f0 0x7f3b40162e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36178/0 (socket says 172.21.15.16:36178) 2023-05-22T15:43:38.842 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:38.834+0000 7f3b3ffff640 1 -- 172.21.15.16:0/2635039498 learned_addr learned my addr 172.21.15.16:0/2635039498 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:43:38.843 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:38.834+0000 7f3b452b1640 1 -- 172.21.15.16:0/2635039498 >> v1:172.21.15.62:6789/0 conn(0x7f3b4006dab0 legacy=0x7f3b4006fef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:38.843 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:38.834+0000 7f3b452b1640 1 -- 172.21.15.16:0/2635039498 >> v1:172.21.15.16:6789/0 conn(0x7f3b40070600 legacy=0x7f3b4015bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:38.844 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:38.834+0000 7f3b452b1640 1 -- 172.21.15.16:0/2635039498 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b4015c950 con 0x7f3b4015c2f0 2023-05-22T15:43:38.844 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:38.834+0000 7f3b3f7fe640 1 -- 172.21.15.16:0/2635039498 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3094351824 0 0) 0x7f3b40158eb0 con 0x7f3b40070600 2023-05-22T15:43:38.847 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:38.838+0000 7f3b452b1640 1 --2- 172.21.15.16:0/2635039498 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3b4015c2f0 0x7f3b40162e90 secure :-1 s=READY pgs=1477 cs=0 l=1 rev1=1 crypto rx=0x7f3b3800aea0 tx=0x7f3b3801da50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=646c624dfda403cd server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:43:38.848 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:38.838+0000 7f3b3f7fe640 1 -- 172.21.15.16:0/2635039498 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b3800a2d0 con 0x7f3b4015c2f0 2023-05-22T15:43:38.848 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:38.838+0000 7f3b3f7fe640 1 -- 172.21.15.16:0/2635039498 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b380a6780 con 0x7f3b4015c2f0 2023-05-22T15:43:38.849 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:38.838+0000 7f3b3f7fe640 1 -- 172.21.15.16:0/2635039498 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b380a6b10 con 0x7f3b4015c2f0 2023-05-22T15:43:38.849 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:38.838+0000 7f3b46d14640 1 -- 172.21.15.16:0/2635039498 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3b4015c2f0 msgr2=0x7f3b40162e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:38.849 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:38.838+0000 7f3b46d14640 1 --2- 172.21.15.16:0/2635039498 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3b4015c2f0 0x7f3b40162e90 secure :-1 s=READY pgs=1477 cs=0 l=1 rev1=1 crypto rx=0x7f3b3800aea0 tx=0x7f3b3801da50 comp rx=0 tx=0).stop 2023-05-22T15:43:38.849 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:38.838+0000 7f3b46d14640 1 -- 172.21.15.16:0/2635039498 shutdown_connections 2023-05-22T15:43:38.850 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:38.838+0000 7f3b46d14640 1 --2- 172.21.15.16:0/2635039498 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3b4015c2f0 0x7f3b40162e90 unknown :-1 s=CLOSED pgs=1477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:38.850 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:38.838+0000 7f3b46d14640 1 -- 172.21.15.16:0/2635039498 >> 172.21.15.16:0/2635039498 conn(0x7f3b400b1110 msgr2=0x7f3b400ce830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:43:38.850 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:38.838+0000 7f3b46d14640 1 -- 172.21.15.16:0/2635039498 shutdown_connections 2023-05-22T15:43:38.850 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:38.838+0000 7f3b46d14640 1 -- 172.21.15.16:0/2635039498 wait complete. 2023-05-22T15:43:38.850 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:38.838+0000 7f3b46d14640 1 Processor -- start 2023-05-22T15:43:38.850 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:38.838+0000 7f3b46d14640 1 -- start start 2023-05-22T15:43:38.850 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:38.842+0000 7f3b46d14640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3b4006dab0 0x7f3b401344f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:38.851 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:38.842+0000 7f3b46d14640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3b40070600 0x7f3b40136370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:38.851 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:38.842+0000 7f3b46d14640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3b4015c2f0 0x7f3b400b17c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:38.851 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:38.842+0000 7f3b46d14640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b40138830 con 0x7f3b4015c2f0 2023-05-22T15:43:38.851 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:38.842+0000 7f3b46d14640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b40134a30 con 0x7f3b40070600 2023-05-22T15:43:38.851 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:38.842+0000 7f3b46d14640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b401368b0 con 0x7f3b4006dab0 2023-05-22T15:43:38.851 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:38.842+0000 7f3b452b1640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3b4015c2f0 0x7f3b400b17c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:38.851 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:38.842+0000 7f3b452b1640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3b4015c2f0 0x7f3b400b17c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36182/0 (socket says 172.21.15.16:36182) 2023-05-22T15:43:38.851 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:38.842+0000 7f3b452b1640 1 -- 172.21.15.16:0/2849443206 learned_addr learned my addr 172.21.15.16:0/2849443206 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:43:38.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:38.842+0000 7f3b44ab0640 1 --2- 172.21.15.16:0/2849443206 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3b4006dab0 0x7f3b401344f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:38.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:38.842+0000 7f3b3ffff640 1 --2- 172.21.15.16:0/2849443206 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3b40070600 0x7f3b40136370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:38.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:38.842+0000 7f3b452b1640 1 -- 172.21.15.16:0/2849443206 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3b4006dab0 msgr2=0x7f3b401344f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:38.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:38.842+0000 7f3b452b1640 1 --2- 172.21.15.16:0/2849443206 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3b4006dab0 0x7f3b401344f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:38.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:38.842+0000 7f3b452b1640 1 -- 172.21.15.16:0/2849443206 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3b40070600 msgr2=0x7f3b40136370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:38.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:38.842+0000 7f3b452b1640 1 --2- 172.21.15.16:0/2849443206 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3b40070600 0x7f3b40136370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:38.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:38.842+0000 7f3b452b1640 1 -- 172.21.15.16:0/2849443206 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b3800ab80 con 0x7f3b4015c2f0 2023-05-22T15:43:38.853 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:38.842+0000 7f3b3ffff640 1 --2- 172.21.15.16:0/2849443206 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3b40070600 0x7f3b40136370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:43:38.853 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:38.842+0000 7f3b452b1640 1 --2- 172.21.15.16:0/2849443206 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3b4015c2f0 0x7f3b400b17c0 secure :-1 s=READY pgs=1478 cs=0 l=1 rev1=1 crypto rx=0x7f3b3801a660 tx=0x7f3b3801a7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:43:38.853 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:38.842+0000 7f3b3d7fa640 1 -- 172.21.15.16:0/2849443206 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b380bc070 con 0x7f3b4015c2f0 2023-05-22T15:43:38.853 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:38.842+0000 7f3b3d7fa640 1 -- 172.21.15.16:0/2849443206 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b380b7480 con 0x7f3b4015c2f0 2023-05-22T15:43:38.853 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:38.842+0000 7f3b46d14640 1 -- 172.21.15.16:0/2849443206 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b40159320 con 0x7f3b4015c2f0 2023-05-22T15:43:38.853 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:38.842+0000 7f3b3d7fa640 1 -- 172.21.15.16:0/2849443206 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b380b7810 con 0x7f3b4015c2f0 2023-05-22T15:43:38.853 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:38.842+0000 7f3b46d14640 1 -- 172.21.15.16:0/2849443206 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b40158510 con 0x7f3b4015c2f0 2023-05-22T15:43:38.854 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:38.842+0000 7f3b46d14640 1 -- 172.21.15.16:0/2849443206 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b401594b0 con 0x7f3b4015c2f0 2023-05-22T15:43:38.854 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:38.842+0000 7f3b3d7fa640 1 -- 172.21.15.16:0/2849443206 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 41) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f3b380b7970 con 0x7f3b4015c2f0 2023-05-22T15:43:38.855 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:38.842+0000 7f3b3d7fa640 1 --2- 172.21.15.16:0/2849443206 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f3b200654a0 0x7f3b20067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:38.855 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:38.842+0000 7f3b3d7fa640 1 -- 172.21.15.16:0/2849443206 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f3b3811d680 con 0x7f3b4015c2f0 2023-05-22T15:43:38.855 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:38.842+0000 7f3b44ab0640 1 --2- 172.21.15.16:0/2849443206 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f3b200654a0 0x7f3b20067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:38.855 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:38.842+0000 7f3b44ab0640 1 --2- 172.21.15.16:0/2849443206 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f3b200654a0 0x7f3b20067950 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f3b30000f70 tx=0x7f3b30009040 comp rx=0 tx=0).ready entity=mgr.7788 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:43:38.855 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:38.846+0000 7f3b3d7fa640 1 -- 172.21.15.16:0/2849443206 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f3b380eb0b0 con 0x7f3b4015c2f0 2023-05-22T15:43:39.030 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:39.022+0000 7f3b46d14640 1 -- 172.21.15.16:0/2849443206 --> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f3b4006b0d0 con 0x7f3b200654a0 2023-05-22T15:43:39.031 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:39.026+0000 7f3b3d7fa640 1 -- 172.21.15.16:0/2849443206 <== mgr.7788 v2:172.21.15.16:6832/36921 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f3b4006b0d0 con 0x7f3b200654a0 2023-05-22T15:43:39.032 INFO:teuthology.orchestra.run.smithi016.stdout:[ 2023-05-22T15:43:39.032 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T15:43:39.032 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "cephfs" 2023-05-22T15:43:39.032 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T15:43:39.032 INFO:teuthology.orchestra.run.smithi016.stdout:] 2023-05-22T15:43:39.034 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:39.026+0000 7f3b46d14640 1 -- 172.21.15.16:0/2849443206 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f3b200654a0 msgr2=0x7f3b20067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:39.034 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:39.026+0000 7f3b46d14640 1 --2- 172.21.15.16:0/2849443206 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f3b200654a0 0x7f3b20067950 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f3b30000f70 tx=0x7f3b30009040 comp rx=0 tx=0).stop 2023-05-22T15:43:39.034 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:39.026+0000 7f3b46d14640 1 -- 172.21.15.16:0/2849443206 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3b4015c2f0 msgr2=0x7f3b400b17c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:39.034 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:39.026+0000 7f3b46d14640 1 --2- 172.21.15.16:0/2849443206 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3b4015c2f0 0x7f3b400b17c0 secure :-1 s=READY pgs=1478 cs=0 l=1 rev1=1 crypto rx=0x7f3b3801a660 tx=0x7f3b3801a7f0 comp rx=0 tx=0).stop 2023-05-22T15:43:39.034 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:39.026+0000 7f3b46d14640 1 -- 172.21.15.16:0/2849443206 shutdown_connections 2023-05-22T15:43:39.035 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:39.026+0000 7f3b46d14640 1 --2- 172.21.15.16:0/2849443206 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3b4015c2f0 0x7f3b400b17c0 unknown :-1 s=CLOSED pgs=1478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:39.035 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:39.026+0000 7f3b46d14640 1 --2- 172.21.15.16:0/2849443206 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3b40070600 0x7f3b40136370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:39.035 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:39.026+0000 7f3b46d14640 1 --2- 172.21.15.16:0/2849443206 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f3b200654a0 0x7f3b20067950 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:39.035 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:39.026+0000 7f3b46d14640 1 --2- 172.21.15.16:0/2849443206 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3b4006dab0 0x7f3b401344f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:39.035 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:39.026+0000 7f3b46d14640 1 -- 172.21.15.16:0/2849443206 >> 172.21.15.16:0/2849443206 conn(0x7f3b400b1110 msgr2=0x7f3b400cea00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:43:39.035 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:39.026+0000 7f3b46d14640 1 -- 172.21.15.16:0/2849443206 shutdown_connections 2023-05-22T15:43:39.036 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:39.026+0000 7f3b46d14640 1 -- 172.21.15.16:0/2849443206 wait complete. 2023-05-22T15:43:39.051 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2023-05-22T15:43:39.177 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:39.170+0000 7f9521029640 1 Processor -- start 2023-05-22T15:43:39.178 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:39.170+0000 7f9521029640 1 -- start start 2023-05-22T15:43:39.178 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:39.174+0000 7f9521029640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f951c071530 0x7f951c15c580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:39.179 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:39.174+0000 7f9521029640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f951c06b6c0 0x7f951c06bb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:39.179 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:39.174+0000 7f9521029640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f951c15d520 con 0x7f951c070c70 2023-05-22T15:43:39.179 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:39.174+0000 7f9521029640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f951c071980 con 0x7f951c06b6c0 2023-05-22T15:43:39.179 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:39.174+0000 7f9521029640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f951c06c080 con 0x7f951c071530 2023-05-22T15:43:39.179 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:39.174+0000 7f951b59e640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f951c06b6c0 0x7f951c06bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:39.179 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:39.174+0000 7f951ad9d640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f951c070c70 0x7f951c06fba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:43764/0 (socket says 172.21.15.16:43764) 2023-05-22T15:43:39.180 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:39.174+0000 7f951ad9d640 1 -- 172.21.15.16:0/173842375 learned_addr learned my addr 172.21.15.16:0/173842375 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:43:39.180 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:39.174+0000 7f951a59c640 1 --2- 172.21.15.16:0/173842375 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f951c071530 0x7f951c15c580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:39.181 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:39.174+0000 7f951b59e640 1 -- 172.21.15.16:0/173842375 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f951c071530 msgr2=0x7f951c15c580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:43:39.181 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:39.174+0000 7f951b59e640 1 --2- 172.21.15.16:0/173842375 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f951c071530 0x7f951c15c580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:39.181 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:39.174+0000 7f951b59e640 1 -- 172.21.15.16:0/173842375 >> v1:172.21.15.16:6789/0 conn(0x7f951c070c70 legacy=0x7f951c06fba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:39.181 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:39.174+0000 7f951b59e640 1 -- 172.21.15.16:0/173842375 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f951c0c8b10 con 0x7f951c06b6c0 2023-05-22T15:43:39.182 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:39.174+0000 7f951b59e640 1 --2- 172.21.15.16:0/173842375 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f951c06b6c0 0x7f951c06bb40 secure :-1 s=READY pgs=1480 cs=0 l=1 rev1=1 crypto rx=0x7f9514009d90 tx=0x7f951401c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7947dcfbd4dc997e server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:43:39.182 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:39.174+0000 7f9519d9b640 1 -- 172.21.15.16:0/173842375 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95140a5980 con 0x7f951c06b6c0 2023-05-22T15:43:39.182 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:39.174+0000 7f9519d9b640 1 -- 172.21.15.16:0/173842375 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95140a5ae0 con 0x7f951c06b6c0 2023-05-22T15:43:39.182 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:39.174+0000 7f9519d9b640 1 -- 172.21.15.16:0/173842375 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95140a5e70 con 0x7f951c06b6c0 2023-05-22T15:43:39.183 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:39.178+0000 7f9521029640 1 -- 172.21.15.16:0/173842375 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f951c06b6c0 msgr2=0x7f951c06bb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:39.184 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:39.178+0000 7f9521029640 1 --2- 172.21.15.16:0/173842375 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f951c06b6c0 0x7f951c06bb40 secure :-1 s=READY pgs=1480 cs=0 l=1 rev1=1 crypto rx=0x7f9514009d90 tx=0x7f951401c8c0 comp rx=0 tx=0).stop 2023-05-22T15:43:39.184 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:39.178+0000 7f9521029640 1 -- 172.21.15.16:0/173842375 shutdown_connections 2023-05-22T15:43:39.184 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:39.178+0000 7f9521029640 1 --2- 172.21.15.16:0/173842375 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f951c06b6c0 0x7f951c06bb40 unknown :-1 s=CLOSED pgs=1480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:39.184 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:39.178+0000 7f9521029640 1 --2- 172.21.15.16:0/173842375 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f951c071530 0x7f951c15c580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:39.184 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:39.178+0000 7f9521029640 1 -- 172.21.15.16:0/173842375 >> 172.21.15.16:0/173842375 conn(0x7f951c069720 msgr2=0x7f951c069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:43:39.184 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:39.178+0000 7f9521029640 1 -- 172.21.15.16:0/173842375 shutdown_connections 2023-05-22T15:43:39.184 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:39.178+0000 7f9521029640 1 -- 172.21.15.16:0/173842375 wait complete. 2023-05-22T15:43:39.185 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:39.178+0000 7f9521029640 1 Processor -- start 2023-05-22T15:43:39.185 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:39.178+0000 7f9521029640 1 -- start start 2023-05-22T15:43:39.186 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:39.178+0000 7f9521029640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f951c06b6c0 0x7f951c0c3020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:39.186 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:39.178+0000 7f9521029640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f951c070c70 0x7f951c15b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:39.186 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:39.178+0000 7f9521029640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f951c071530 0x7f951c1522e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:39.187 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:39.178+0000 7f9521029640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f951c0c3690 con 0x7f951c070c70 2023-05-22T15:43:39.187 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:39.178+0000 7f9521029640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f951c0c3970 con 0x7f951c06b6c0 2023-05-22T15:43:39.187 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:39.178+0000 7f9521029640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f951c0c3c50 con 0x7f951c071530 2023-05-22T15:43:39.187 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:39.178+0000 7f951a59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f951c070c70 0x7f951c15b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:39.187 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:39.178+0000 7f951a59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f951c070c70 0x7f951c15b480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33148/0 (socket says 172.21.15.16:33148) 2023-05-22T15:43:39.187 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:39.178+0000 7f951a59c640 1 -- 172.21.15.16:0/732620736 learned_addr learned my addr 172.21.15.16:0/732620736 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:43:39.187 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:39.178+0000 7f951ad9d640 1 --2- 172.21.15.16:0/732620736 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f951c06b6c0 0x7f951c0c3020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:39.188 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:39.178+0000 7f951b59e640 1 --2- 172.21.15.16:0/732620736 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f951c071530 0x7f951c1522e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:39.188 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:39.178+0000 7f951a59c640 1 -- 172.21.15.16:0/732620736 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f951c071530 msgr2=0x7f951c1522e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:39.188 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:39.178+0000 7f951a59c640 1 --2- 172.21.15.16:0/732620736 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f951c071530 0x7f951c1522e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:39.188 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:39.178+0000 7f951a59c640 1 -- 172.21.15.16:0/732620736 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f951c06b6c0 msgr2=0x7f951c0c3020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:39.188 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:39.178+0000 7f951a59c640 1 --2- 172.21.15.16:0/732620736 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f951c06b6c0 0x7f951c0c3020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:39.188 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:39.178+0000 7f951a59c640 1 -- 172.21.15.16:0/732620736 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9514009a70 con 0x7f951c070c70 2023-05-22T15:43:39.188 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:39.178+0000 7f951a59c640 1 --2- 172.21.15.16:0/732620736 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f951c070c70 0x7f951c15b480 secure :-1 s=READY pgs=1481 cs=0 l=1 rev1=1 crypto rx=0x7f951000d5d0 tx=0x7f951000da90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:43:39.188 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:39.178+0000 7f95037fe640 1 -- 172.21.15.16:0/732620736 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9510004240 con 0x7f951c070c70 2023-05-22T15:43:39.189 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:39.178+0000 7f95037fe640 1 -- 172.21.15.16:0/732620736 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95100043a0 con 0x7f951c070c70 2023-05-22T15:43:39.189 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:39.178+0000 7f9521029640 1 -- 172.21.15.16:0/732620736 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f951c0cc510 con 0x7f951c070c70 2023-05-22T15:43:39.189 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:39.178+0000 7f95037fe640 1 -- 172.21.15.16:0/732620736 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9510003840 con 0x7f951c070c70 2023-05-22T15:43:39.189 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:39.178+0000 7f9521029640 1 -- 172.21.15.16:0/732620736 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f951c15d240 con 0x7f951c070c70 2023-05-22T15:43:39.189 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:39.182+0000 7f9521029640 1 -- 172.21.15.16:0/732620736 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f951c071040 con 0x7f951c070c70 2023-05-22T15:43:39.190 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:39.182+0000 7f95037fe640 1 -- 172.21.15.16:0/732620736 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 41) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f95100039a0 con 0x7f951c070c70 2023-05-22T15:43:39.190 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:39.182+0000 7f95037fe640 1 --2- 172.21.15.16:0/732620736 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f95080654a0 0x7f9508067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:39.190 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:39.182+0000 7f95037fe640 1 -- 172.21.15.16:0/732620736 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f9510082c30 con 0x7f951c070c70 2023-05-22T15:43:39.190 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:39.182+0000 7f951ad9d640 1 --2- 172.21.15.16:0/732620736 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f95080654a0 0x7f9508067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:39.190 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:39.182+0000 7f951ad9d640 1 --2- 172.21.15.16:0/732620736 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f95080654a0 0x7f9508067950 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f950c005890 tx=0x7f950c009040 comp rx=0 tx=0).ready entity=mgr.7788 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:43:39.192 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:39.186+0000 7f95037fe640 1 -- 172.21.15.16:0/732620736 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f9510050690 con 0x7f951c070c70 2023-05-22T15:43:39.384 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:39.378+0000 7f9521029640 1 -- 172.21.15.16:0/732620736 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7f951c04dbe0 con 0x7f951c070c70 2023-05-22T15:43:40.205 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.198+0000 7f95037fe640 1 -- 172.21.15.16:0/732620736 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mgrmap(e 42) v1 ==== 84349+0+0 (secure 0 0 0) 0x7f9510050a60 con 0x7f951c070c70 2023-05-22T15:43:40.207 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.202+0000 7f95037fe640 1 -- 172.21.15.16:0/732620736 <== mon.0 v2:172.21.15.16:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v42) v1 ==== 92+0+0 (secure 0 0 0) 0x7f951005c690 con 0x7f951c070c70 2023-05-22T15:43:40.209 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.202+0000 7f9521029640 1 -- 172.21.15.16:0/732620736 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f95080654a0 msgr2=0x7f9508067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:40.209 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.202+0000 7f9521029640 1 --2- 172.21.15.16:0/732620736 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f95080654a0 0x7f9508067950 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f950c005890 tx=0x7f950c009040 comp rx=0 tx=0).stop 2023-05-22T15:43:40.210 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.202+0000 7f9521029640 1 -- 172.21.15.16:0/732620736 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f951c070c70 msgr2=0x7f951c15b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:40.210 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.202+0000 7f9521029640 1 --2- 172.21.15.16:0/732620736 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f951c070c70 0x7f951c15b480 secure :-1 s=READY pgs=1481 cs=0 l=1 rev1=1 crypto rx=0x7f951000d5d0 tx=0x7f951000da90 comp rx=0 tx=0).stop 2023-05-22T15:43:40.210 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.202+0000 7f9521029640 1 -- 172.21.15.16:0/732620736 shutdown_connections 2023-05-22T15:43:40.210 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.202+0000 7f9521029640 1 --2- 172.21.15.16:0/732620736 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f95080654a0 0x7f9508067950 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:40.210 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.202+0000 7f9521029640 1 --2- 172.21.15.16:0/732620736 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f951c071530 0x7f951c1522e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:40.210 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.202+0000 7f9521029640 1 --2- 172.21.15.16:0/732620736 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f951c070c70 0x7f951c15b480 unknown :-1 s=CLOSED pgs=1481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:40.210 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.202+0000 7f9521029640 1 --2- 172.21.15.16:0/732620736 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f951c06b6c0 0x7f951c0c3020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:40.210 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.202+0000 7f9521029640 1 -- 172.21.15.16:0/732620736 >> 172.21.15.16:0/732620736 conn(0x7f951c069720 msgr2=0x7f951c0b1070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:43:40.211 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.202+0000 7f9521029640 1 -- 172.21.15.16:0/732620736 shutdown_connections 2023-05-22T15:43:40.211 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.202+0000 7f9521029640 1 -- 172.21.15.16:0/732620736 wait complete. 2023-05-22T15:43:40.229 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2023-05-22T15:43:40.330 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:43:40.326+0000 7fa394942180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-05-22T15:43:40.339 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:43:40.334+0000 7fc5b6f34180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-05-22T15:43:40.354 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.346+0000 7fbcb1f84640 1 Processor -- start 2023-05-22T15:43:40.354 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.346+0000 7fbcb1f84640 1 -- start start 2023-05-22T15:43:40.355 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.350+0000 7fbcb1f84640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbcac071640 0x7fbcac160090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:40.355 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.350+0000 7fbcb1f84640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fbcac1605d0 0x7fbcac15c270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:40.355 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.350+0000 7fbcb1f84640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbcac069ae0 con 0x7fbcac0ce480 2023-05-22T15:43:40.355 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.350+0000 7fbcb1f84640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbcac15f990 con 0x7fbcac1605d0 2023-05-22T15:43:40.356 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.350+0000 7fbcb1f84640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbcac071a90 con 0x7fbcac071640 2023-05-22T15:43:40.356 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.350+0000 7fbcab7fe640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fbcac0ce480 0x7fbcac0ce850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:43790/0 (socket says 172.21.15.16:43790) 2023-05-22T15:43:40.356 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.350+0000 7fbcabfff640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fbcac1605d0 0x7fbcac15c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:40.356 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.350+0000 7fbcab7fe640 1 -- 172.21.15.16:0/2485292697 learned_addr learned my addr 172.21.15.16:0/2485292697 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:43:40.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.350+0000 7fbcaaffd640 1 --2- 172.21.15.16:0/2485292697 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbcac071640 0x7fbcac160090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:40.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.350+0000 7fbcaa7fc640 1 -- 172.21.15.16:0/2485292697 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3035552859 0 0) 0x7fbcac069ae0 con 0x7fbcac0ce480 2023-05-22T15:43:40.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.350+0000 7fbcaa7fc640 1 -- 172.21.15.16:0/2485292697 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc98003530 con 0x7fbcac0ce480 2023-05-22T15:43:40.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.350+0000 7fbcabfff640 1 -- 172.21.15.16:0/2485292697 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbcac071640 msgr2=0x7fbcac160090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:43:40.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.350+0000 7fbcabfff640 1 --2- 172.21.15.16:0/2485292697 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbcac071640 0x7fbcac160090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:40.358 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.350+0000 7fbcabfff640 1 -- 172.21.15.16:0/2485292697 >> v1:172.21.15.16:6789/0 conn(0x7fbcac0ce480 legacy=0x7fbcac0ce850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:40.358 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.350+0000 7fbcabfff640 1 -- 172.21.15.16:0/2485292697 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcac069dc0 con 0x7fbcac1605d0 2023-05-22T15:43:40.359 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.354+0000 7fbcabfff640 1 --2- 172.21.15.16:0/2485292697 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbcac1605d0 0x7fbcac15c270 secure :-1 s=READY pgs=1486 cs=0 l=1 rev1=1 crypto rx=0x7fbc9c009d90 tx=0x7fbc9c01c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79b01ac34a9b8c14 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:43:40.359 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.354+0000 7fbcaa7fc640 1 -- 172.21.15.16:0/2485292697 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc9c0a59e0 con 0x7fbcac1605d0 2023-05-22T15:43:40.359 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.354+0000 7fbcaa7fc640 1 -- 172.21.15.16:0/2485292697 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc9c0a5b40 con 0x7fbcac1605d0 2023-05-22T15:43:40.359 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.354+0000 7fbcaa7fc640 1 -- 172.21.15.16:0/2485292697 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc9c0ad440 con 0x7fbcac1605d0 2023-05-22T15:43:40.360 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.354+0000 7fbcb1f84640 1 -- 172.21.15.16:0/2485292697 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbcac1605d0 msgr2=0x7fbcac15c270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:40.360 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.354+0000 7fbcb1f84640 1 --2- 172.21.15.16:0/2485292697 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbcac1605d0 0x7fbcac15c270 secure :-1 s=READY pgs=1486 cs=0 l=1 rev1=1 crypto rx=0x7fbc9c009d90 tx=0x7fbc9c01c920 comp rx=0 tx=0).stop 2023-05-22T15:43:40.360 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.354+0000 7fbcb1f84640 1 -- 172.21.15.16:0/2485292697 shutdown_connections 2023-05-22T15:43:40.361 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.354+0000 7fbcb1f84640 1 --2- 172.21.15.16:0/2485292697 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbcac1605d0 0x7fbcac15c270 unknown :-1 s=CLOSED pgs=1486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:40.361 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.354+0000 7fbcb1f84640 1 --2- 172.21.15.16:0/2485292697 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbcac071640 0x7fbcac160090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:40.361 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.354+0000 7fbcb1f84640 1 -- 172.21.15.16:0/2485292697 >> 172.21.15.16:0/2485292697 conn(0x7fbcac0cc870 msgr2=0x7fbcac0cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:43:40.361 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.354+0000 7fbcb1f84640 1 -- 172.21.15.16:0/2485292697 shutdown_connections 2023-05-22T15:43:40.361 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.354+0000 7fbcb1f84640 1 -- 172.21.15.16:0/2485292697 wait complete. 2023-05-22T15:43:40.362 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.354+0000 7fbcb1f84640 1 Processor -- start 2023-05-22T15:43:40.363 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.354+0000 7fbcb1f84640 1 -- start start 2023-05-22T15:43:40.363 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.354+0000 7fbcb1f84640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbcac071640 0x7fbcac1f8080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:40.363 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.354+0000 7fbcb1f84640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbcac0ce480 0x7fbcac15a770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:40.363 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.354+0000 7fbcb1f84640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbcac1605d0 0x7fbcac152d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:40.363 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.354+0000 7fbcb1f84640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcac0af0d0 con 0x7fbcac0ce480 2023-05-22T15:43:40.363 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.354+0000 7fbcb1f84640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcac153910 con 0x7fbcac1605d0 2023-05-22T15:43:40.363 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.354+0000 7fbcb1f84640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbcac1f8f70 con 0x7fbcac071640 2023-05-22T15:43:40.364 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.358+0000 7fbcaaffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbcac0ce480 0x7fbcac15a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:40.364 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.358+0000 7fbcaaffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbcac0ce480 0x7fbcac15a770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33182/0 (socket says 172.21.15.16:33182) 2023-05-22T15:43:40.364 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.358+0000 7fbcaaffd640 1 -- 172.21.15.16:0/1190645360 learned_addr learned my addr 172.21.15.16:0/1190645360 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:43:40.364 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.358+0000 7fbcabfff640 1 --2- 172.21.15.16:0/1190645360 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbcac1605d0 0x7fbcac152d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:40.364 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.358+0000 7fbcab7fe640 1 --2- 172.21.15.16:0/1190645360 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbcac071640 0x7fbcac1f8080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:40.364 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.358+0000 7fbcaaffd640 1 -- 172.21.15.16:0/1190645360 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbcac071640 msgr2=0x7fbcac1f8080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:40.364 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.358+0000 7fbcaaffd640 1 --2- 172.21.15.16:0/1190645360 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbcac071640 0x7fbcac1f8080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:40.365 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.358+0000 7fbcaaffd640 1 -- 172.21.15.16:0/1190645360 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbcac1605d0 msgr2=0x7fbcac152d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:40.365 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.358+0000 7fbcaaffd640 1 --2- 172.21.15.16:0/1190645360 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbcac1605d0 0x7fbcac152d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:40.365 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.358+0000 7fbcaaffd640 1 -- 172.21.15.16:0/1190645360 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc9c009a70 con 0x7fbcac0ce480 2023-05-22T15:43:40.365 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.358+0000 7fbcabfff640 1 --2- 172.21.15.16:0/1190645360 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbcac1605d0 0x7fbcac152d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T15:43:40.366 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.358+0000 7fbcaaffd640 1 --2- 172.21.15.16:0/1190645360 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbcac0ce480 0x7fbcac15a770 secure :-1 s=READY pgs=1487 cs=0 l=1 rev1=1 crypto rx=0x7fbc9400d5f0 tx=0x7fbc9400dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:43:40.366 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.358+0000 7fbca3fff640 1 -- 172.21.15.16:0/1190645360 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc940125f0 con 0x7fbcac0ce480 2023-05-22T15:43:40.366 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.358+0000 7fbca3fff640 1 -- 172.21.15.16:0/1190645360 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc94012750 con 0x7fbcac0ce480 2023-05-22T15:43:40.367 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.358+0000 7fbcb1f84640 1 -- 172.21.15.16:0/1190645360 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbcac0698b0 con 0x7fbcac0ce480 2023-05-22T15:43:40.367 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.358+0000 7fbca3fff640 1 -- 172.21.15.16:0/1190645360 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc94012ae0 con 0x7fbcac0ce480 2023-05-22T15:43:40.367 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.358+0000 7fbcb1f84640 1 -- 172.21.15.16:0/1190645360 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbcac069720 con 0x7fbcac0ce480 2023-05-22T15:43:40.367 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.358+0000 7fbcb1f84640 1 -- 172.21.15.16:0/1190645360 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbcac0c5e70 con 0x7fbcac0ce480 2023-05-22T15:43:40.367 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.358+0000 7fbca3fff640 1 -- 172.21.15.16:0/1190645360 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 42) v1 ==== 84349+0+0 (secure 0 0 0) 0x7fbc94012c40 con 0x7fbcac0ce480 2023-05-22T15:43:40.367 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.358+0000 7fbca3fff640 1 --2- 172.21.15.16:0/1190645360 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fbc8c0654f0 0x7fbc8c0679a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:40.367 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.358+0000 7fbcab7fe640 1 -- 172.21.15.16:0/1190645360 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fbc8c0654f0 msgr2=0x7fbc8c0679a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/36921 2023-05-22T15:43:40.368 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.358+0000 7fbca3fff640 1 -- 172.21.15.16:0/1190645360 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 6382+0+0 (secure 0 0 0) 0x7fbc940830a0 con 0x7fbcac0ce480 2023-05-22T15:43:40.368 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.358+0000 7fbcab7fe640 1 --2- 172.21.15.16:0/1190645360 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fbc8c0654f0 0x7fbc8c0679a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-05-22T15:43:40.369 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.362+0000 7fbca3fff640 1 -- 172.21.15.16:0/1190645360 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fbc94050ac0 con 0x7fbcac0ce480 2023-05-22T15:43:40.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.518+0000 7fbcb1f84640 1 -- 172.21.15.16:0/1190645360 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7fbcac04dbe0 con 0x7fbcac0ce480 2023-05-22T15:43:40.530 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.522+0000 7fbca3fff640 1 -- 172.21.15.16:0/1190645360 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v35)=0 v35) v1 ==== 138+0+0 (secure 0 0 0) 0x7fbc940540e0 con 0x7fbcac0ce480 2023-05-22T15:43:40.532 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.526+0000 7fbcb1f84640 1 -- 172.21.15.16:0/1190645360 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fbc8c0654f0 msgr2=0x7fbc8c0679a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-05-22T15:43:40.532 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.526+0000 7fbcb1f84640 1 --2- 172.21.15.16:0/1190645360 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fbc8c0654f0 0x7fbc8c0679a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:40.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.526+0000 7fbcb1f84640 1 -- 172.21.15.16:0/1190645360 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbcac0ce480 msgr2=0x7fbcac15a770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:40.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.526+0000 7fbcb1f84640 1 --2- 172.21.15.16:0/1190645360 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbcac0ce480 0x7fbcac15a770 secure :-1 s=READY pgs=1487 cs=0 l=1 rev1=1 crypto rx=0x7fbc9400d5f0 tx=0x7fbc9400dab0 comp rx=0 tx=0).stop 2023-05-22T15:43:40.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.526+0000 7fbcb1f84640 1 -- 172.21.15.16:0/1190645360 shutdown_connections 2023-05-22T15:43:40.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.526+0000 7fbcb1f84640 1 --2- 172.21.15.16:0/1190645360 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbcac1605d0 0x7fbcac152d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:40.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.526+0000 7fbcb1f84640 1 --2- 172.21.15.16:0/1190645360 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbcac0ce480 0x7fbcac15a770 unknown :-1 s=CLOSED pgs=1487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:40.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.526+0000 7fbcb1f84640 1 --2- 172.21.15.16:0/1190645360 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fbc8c0654f0 0x7fbc8c0679a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:40.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.526+0000 7fbcb1f84640 1 --2- 172.21.15.16:0/1190645360 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbcac071640 0x7fbcac1f8080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:40.534 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.526+0000 7fbcb1f84640 1 -- 172.21.15.16:0/1190645360 >> 172.21.15.16:0/1190645360 conn(0x7fbcac0cc870 msgr2=0x7fbcac15cbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:43:40.534 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.526+0000 7fbcb1f84640 1 -- 172.21.15.16:0/1190645360 shutdown_connections 2023-05-22T15:43:40.534 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.526+0000 7fbcb1f84640 1 -- 172.21.15.16:0/1190645360 wait complete. 2023-05-22T15:43:40.548 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2023-05-22T15:43:40.666 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.662+0000 7f9fc0860640 1 Processor -- start 2023-05-22T15:43:40.667 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.662+0000 7f9fc0860640 1 -- start start 2023-05-22T15:43:40.668 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.662+0000 7f9fc0860640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f9fbc0cc6f0 0x7f9fbc164a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:40.668 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.662+0000 7f9fc0860640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fbc0c4dc0 con 0x7f9fbc0b2880 2023-05-22T15:43:40.668 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.662+0000 7f9fc0860640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fbc0c6ba0 con 0x7f9fbc0c67d0 2023-05-22T15:43:40.668 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.662+0000 7f9fc0860640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9fbc15a300 con 0x7f9fbc0cc6f0 2023-05-22T15:43:40.669 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.662+0000 7f9fb9d9b640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f9fbc0b2880 0x7f9fbc0cbfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:43802/0 (socket says 172.21.15.16:43802) 2023-05-22T15:43:40.669 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.662+0000 7f9fb9d9b640 1 -- 172.21.15.16:0/3223972372 learned_addr learned my addr 172.21.15.16:0/3223972372 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:43:40.669 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.662+0000 7f9fbad9d640 1 --2- 172.21.15.16:0/3223972372 >> v2:172.21.15.16:3300/0 conn(0x7f9fbc0cc6f0 0x7f9fbc164a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:40.669 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.662+0000 7f9fb959a640 1 -- 172.21.15.16:0/3223972372 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3168736857 0 0) 0x7f9fbc0c4dc0 con 0x7f9fbc0b2880 2023-05-22T15:43:40.669 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.662+0000 7f9fb959a640 1 -- 172.21.15.16:0/3223972372 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9fa4003530 con 0x7f9fbc0b2880 2023-05-22T15:43:40.669 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.662+0000 7f9fbad9d640 1 -- 172.21.15.16:0/3223972372 >> v1:172.21.15.62:6789/0 conn(0x7f9fbc0c67d0 legacy=0x7f9fbc0cf730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:40.669 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.662+0000 7f9fbad9d640 1 -- 172.21.15.16:0/3223972372 >> v1:172.21.15.16:6789/0 conn(0x7f9fbc0b2880 legacy=0x7f9fbc0cbfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:40.670 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.662+0000 7f9fbad9d640 1 -- 172.21.15.16:0/3223972372 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fbc0cfef0 con 0x7f9fbc0cc6f0 2023-05-22T15:43:40.670 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.662+0000 7f9fbad9d640 1 --2- 172.21.15.16:0/3223972372 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9fbc0cc6f0 0x7f9fbc164a20 secure :-1 s=READY pgs=1489 cs=0 l=1 rev1=1 crypto rx=0x7f9fb4009d90 tx=0x7f9fb401c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca0de8fe8642f4d0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:43:40.670 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.666+0000 7f9fb959a640 1 -- 172.21.15.16:0/3223972372 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fb40a5950 con 0x7f9fbc0cc6f0 2023-05-22T15:43:40.671 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.666+0000 7f9fb959a640 1 -- 172.21.15.16:0/3223972372 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9fb40a5ab0 con 0x7f9fbc0cc6f0 2023-05-22T15:43:40.671 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.666+0000 7f9fb959a640 1 -- 172.21.15.16:0/3223972372 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fb40a5e40 con 0x7f9fbc0cc6f0 2023-05-22T15:43:40.672 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.666+0000 7f9fc0860640 1 -- 172.21.15.16:0/3223972372 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9fbc0cc6f0 msgr2=0x7f9fbc164a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:40.672 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.666+0000 7f9fc0860640 1 --2- 172.21.15.16:0/3223972372 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9fbc0cc6f0 0x7f9fbc164a20 secure :-1 s=READY pgs=1489 cs=0 l=1 rev1=1 crypto rx=0x7f9fb4009d90 tx=0x7f9fb401c890 comp rx=0 tx=0).stop 2023-05-22T15:43:40.672 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.666+0000 7f9fc0860640 1 -- 172.21.15.16:0/3223972372 shutdown_connections 2023-05-22T15:43:40.672 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.666+0000 7f9fc0860640 1 --2- 172.21.15.16:0/3223972372 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9fbc0cc6f0 0x7f9fbc164a20 unknown :-1 s=CLOSED pgs=1489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:40.672 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.666+0000 7f9fc0860640 1 -- 172.21.15.16:0/3223972372 >> 172.21.15.16:0/3223972372 conn(0x7f9fbc06f5a0 msgr2=0x7f9fbc06f9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:43:40.672 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.666+0000 7f9fc0860640 1 -- 172.21.15.16:0/3223972372 shutdown_connections 2023-05-22T15:43:40.673 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.666+0000 7f9fc0860640 1 -- 172.21.15.16:0/3223972372 wait complete. 2023-05-22T15:43:40.674 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.666+0000 7f9fc0860640 1 Processor -- start 2023-05-22T15:43:40.674 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.666+0000 7f9fc0860640 1 -- start start 2023-05-22T15:43:40.675 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.666+0000 7f9fc0860640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9fbc0b2880 0x7f9fbc0c9120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:40.675 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.666+0000 7f9fc0860640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9fbc0c67d0 0x7f9fbc1f5770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:40.675 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.666+0000 7f9fc0860640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9fbc0cc6f0 0x7f9fbc1f5cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:40.675 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.666+0000 7f9fc0860640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fbc0b3090 con 0x7f9fbc0cc6f0 2023-05-22T15:43:40.675 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.666+0000 7f9fc0860640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fbc0b3930 con 0x7f9fbc0c67d0 2023-05-22T15:43:40.675 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.666+0000 7f9fc0860640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9fbc158e90 con 0x7f9fbc0b2880 2023-05-22T15:43:40.675 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.666+0000 7f9fbad9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9fbc0cc6f0 0x7f9fbc1f5cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:40.676 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.666+0000 7f9fbad9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9fbc0cc6f0 0x7f9fbc1f5cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33192/0 (socket says 172.21.15.16:33192) 2023-05-22T15:43:40.676 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.666+0000 7f9fbad9d640 1 -- 172.21.15.16:0/1248304290 learned_addr learned my addr 172.21.15.16:0/1248304290 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:43:40.676 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.666+0000 7f9fb9d9b640 1 --2- 172.21.15.16:0/1248304290 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9fbc0c67d0 0x7f9fbc1f5770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:40.676 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.666+0000 7f9fba59c640 1 --2- 172.21.15.16:0/1248304290 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9fbc0b2880 0x7f9fbc0c9120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:40.676 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.666+0000 7f9fbad9d640 1 -- 172.21.15.16:0/1248304290 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9fbc0b2880 msgr2=0x7f9fbc0c9120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:40.676 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.666+0000 7f9fbad9d640 1 --2- 172.21.15.16:0/1248304290 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9fbc0b2880 0x7f9fbc0c9120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:40.676 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.666+0000 7f9fbad9d640 1 -- 172.21.15.16:0/1248304290 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9fbc0c67d0 msgr2=0x7f9fbc1f5770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:40.677 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.666+0000 7f9fbad9d640 1 --2- 172.21.15.16:0/1248304290 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9fbc0c67d0 0x7f9fbc1f5770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:40.677 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.666+0000 7f9fbad9d640 1 -- 172.21.15.16:0/1248304290 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fb4009a70 con 0x7f9fbc0cc6f0 2023-05-22T15:43:40.677 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.666+0000 7f9fbad9d640 1 --2- 172.21.15.16:0/1248304290 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9fbc0cc6f0 0x7f9fbc1f5cb0 secure :-1 s=READY pgs=1490 cs=0 l=1 rev1=1 crypto rx=0x7f9fb40b0040 tx=0x7f9fb4019630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:43:40.677 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.666+0000 7f9faaffd640 1 -- 172.21.15.16:0/1248304290 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fb40add90 con 0x7f9fbc0cc6f0 2023-05-22T15:43:40.677 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.666+0000 7f9faaffd640 1 -- 172.21.15.16:0/1248304290 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9fb4009450 con 0x7f9fbc0cc6f0 2023-05-22T15:43:40.677 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.666+0000 7f9fc0860640 1 -- 172.21.15.16:0/1248304290 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9fbc0d0a70 con 0x7f9fbc0cc6f0 2023-05-22T15:43:40.677 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.666+0000 7f9faaffd640 1 -- 172.21.15.16:0/1248304290 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fb40b9670 con 0x7f9fbc0cc6f0 2023-05-22T15:43:40.678 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.666+0000 7f9fc0860640 1 -- 172.21.15.16:0/1248304290 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9fbc0d0790 con 0x7f9fbc0cc6f0 2023-05-22T15:43:40.678 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.670+0000 7f9fc0860640 1 -- 172.21.15.16:0/1248304290 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9fbc0d0f90 con 0x7f9fbc0cc6f0 2023-05-22T15:43:40.679 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.670+0000 7f9faaffd640 1 -- 172.21.15.16:0/1248304290 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 42) v1 ==== 84349+0+0 (secure 0 0 0) 0x7f9fb40b97d0 con 0x7f9fbc0cc6f0 2023-05-22T15:43:40.679 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.670+0000 7f9faaffd640 1 --2- 172.21.15.16:0/1248304290 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f9f9c0654f0 0x7f9f9c0679a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:40.679 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.670+0000 7f9fba59c640 1 -- 172.21.15.16:0/1248304290 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f9f9c0654f0 msgr2=0x7f9f9c0679a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/36921 2023-05-22T15:43:40.679 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.670+0000 7f9faaffd640 1 -- 172.21.15.16:0/1248304290 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f9fb411e470 con 0x7f9fbc0cc6f0 2023-05-22T15:43:40.679 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.670+0000 7f9fba59c640 1 --2- 172.21.15.16:0/1248304290 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f9f9c0654f0 0x7f9f9c0679a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-05-22T15:43:40.684 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.678+0000 7f9faaffd640 1 -- 172.21.15.16:0/1248304290 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f9fb40ebe20 con 0x7f9fbc0cc6f0 2023-05-22T15:43:40.717 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:43:40.714+0000 7fa394942180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-05-22T15:43:40.742 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:43:40.734+0000 7fc5b6f34180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-05-22T15:43:40.841 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.834+0000 7f9fc0860640 1 -- 172.21.15.16:0/1248304290 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7f9fbc04dbe0 con 0x7f9fbc0cc6f0 2023-05-22T15:43:40.853 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.846+0000 7f9faaffd640 1 -- 172.21.15.16:0/1248304290 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v36)=0 v36) v1 ==== 133+0+0 (secure 0 0 0) 0x7f9fb40ef440 con 0x7f9fbc0cc6f0 2023-05-22T15:43:40.855 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:43:40.850+0000 7fa394942180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-05-22T15:43:40.856 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.850+0000 7f9fc0860640 1 -- 172.21.15.16:0/1248304290 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f9f9c0654f0 msgr2=0x7f9f9c0679a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-05-22T15:43:40.856 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.850+0000 7f9fc0860640 1 --2- 172.21.15.16:0/1248304290 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f9f9c0654f0 0x7f9f9c0679a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:40.856 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.850+0000 7f9fc0860640 1 -- 172.21.15.16:0/1248304290 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9fbc0cc6f0 msgr2=0x7f9fbc1f5cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:40.856 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.850+0000 7f9fc0860640 1 --2- 172.21.15.16:0/1248304290 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9fbc0cc6f0 0x7f9fbc1f5cb0 secure :-1 s=READY pgs=1490 cs=0 l=1 rev1=1 crypto rx=0x7f9fb40b0040 tx=0x7f9fb4019630 comp rx=0 tx=0).stop 2023-05-22T15:43:40.856 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.850+0000 7f9fc0860640 1 -- 172.21.15.16:0/1248304290 shutdown_connections 2023-05-22T15:43:40.857 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.850+0000 7f9fc0860640 1 --2- 172.21.15.16:0/1248304290 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9fbc0cc6f0 0x7f9fbc1f5cb0 unknown :-1 s=CLOSED pgs=1490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:40.857 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.850+0000 7f9fc0860640 1 --2- 172.21.15.16:0/1248304290 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9fbc0c67d0 0x7f9fbc1f5770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:40.857 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.850+0000 7f9fc0860640 1 --2- 172.21.15.16:0/1248304290 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f9f9c0654f0 0x7f9f9c0679a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:40.857 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.850+0000 7f9fc0860640 1 --2- 172.21.15.16:0/1248304290 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9fbc0b2880 0x7f9fbc0c9120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:40.857 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.850+0000 7f9fc0860640 1 -- 172.21.15.16:0/1248304290 >> 172.21.15.16:0/1248304290 conn(0x7f9fbc06f5a0 msgr2=0x7f9fbc0c4fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:43:40.857 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.850+0000 7f9fc0860640 1 -- 172.21.15.16:0/1248304290 shutdown_connections 2023-05-22T15:43:40.857 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.850+0000 7f9fc0860640 1 -- 172.21.15.16:0/1248304290 wait complete. 2023-05-22T15:43:40.872 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add --fs cephfs /bad-path 1M 2023-05-22T15:43:40.880 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:43:40.874+0000 7fc5b6f34180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-05-22T15:43:40.989 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.982+0000 7f0262783640 1 Processor -- start 2023-05-22T15:43:40.989 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.982+0000 7f0262783640 1 -- start start 2023-05-22T15:43:40.990 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.986+0000 7f0262783640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f025c159ac0 0x7f025c0b1700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:40.990 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.986+0000 7f0262783640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f025c06b9c0 con 0x7f025c0b40b0 2023-05-22T15:43:40.991 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.986+0000 7f0262783640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f025c06c820 con 0x7f025c070600 2023-05-22T15:43:40.991 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.986+0000 7f0262783640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f025c070e80 con 0x7f025c159ac0 2023-05-22T15:43:40.991 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.986+0000 7f025b7fe640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f025c0b40b0 0x7f025c06fef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:43812/0 (socket says 172.21.15.16:43812) 2023-05-22T15:43:40.991 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.986+0000 7f025b7fe640 1 -- 172.21.15.16:0/3273889799 learned_addr learned my addr 172.21.15.16:0/3273889799 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:43:40.992 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.986+0000 7f025bfff640 1 --2- 172.21.15.16:0/3273889799 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f025c159ac0 0x7f025c0b1700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:40.993 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.986+0000 7f025affd640 1 -- 172.21.15.16:0/3273889799 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2019727728 0 0) 0x7f025c06b9c0 con 0x7f025c0b40b0 2023-05-22T15:43:40.993 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.986+0000 7f025affd640 1 -- 172.21.15.16:0/3273889799 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0244003530 con 0x7f025c0b40b0 2023-05-22T15:43:40.993 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.986+0000 7f025affd640 1 -- 172.21.15.16:0/3273889799 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3372523216 0 0) 0x7f0244003530 con 0x7f025c0b40b0 2023-05-22T15:43:40.993 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.986+0000 7f025affd640 1 -- 172.21.15.16:0/3273889799 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f025c159ac0 msgr2=0x7f025c0b1700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:43:40.993 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.986+0000 7f025affd640 1 --2- 172.21.15.16:0/3273889799 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f025c159ac0 0x7f025c0b1700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:40.993 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.986+0000 7f025affd640 1 -- 172.21.15.16:0/3273889799 >> v1:172.21.15.62:6789/0 conn(0x7f025c070600 legacy=0x7f025c06ccb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:40.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.986+0000 7f025affd640 1 -- 172.21.15.16:0/3273889799 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f025c06bc10 con 0x7f025c0b40b0 2023-05-22T15:43:40.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.986+0000 7f025affd640 1 -- 172.21.15.16:0/3273889799 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7f024c001680 con 0x7f025c0b40b0 2023-05-22T15:43:40.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.986+0000 7f025affd640 1 -- 172.21.15.16:0/3273889799 >> v1:172.21.15.16:6789/0 conn(0x7f025c0b40b0 legacy=0x7f025c06fef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:40.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.986+0000 7f025affd640 1 --2- 172.21.15.16:0/3273889799 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f025c159ac0 0x7f02440039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:40.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.986+0000 7f025affd640 1 --2- 172.21.15.16:0/3273889799 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0244003f30 0x7f02440047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:40.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.986+0000 7f025affd640 1 --2- 172.21.15.16:0/3273889799 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0244004d10 0x7f0244005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:40.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.986+0000 7f025affd640 1 -- 172.21.15.16:0/3273889799 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f025c070e80 con 0x7f0244004d10 2023-05-22T15:43:40.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.986+0000 7f025affd640 1 -- 172.21.15.16:0/3273889799 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f024400ba70 con 0x7f0244003f30 2023-05-22T15:43:40.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.986+0000 7f025affd640 1 -- 172.21.15.16:0/3273889799 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f024400bc30 con 0x7f025c159ac0 2023-05-22T15:43:40.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.986+0000 7f0260d20640 1 --2- 172.21.15.16:0/3273889799 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0244004d10 0x7f0244005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:40.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.986+0000 7f025bfff640 1 --2- 172.21.15.16:0/3273889799 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f025c159ac0 0x7f02440039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:40.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.986+0000 7f025b7fe640 1 --2- 172.21.15.16:0/3273889799 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0244003f30 0x7f02440047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:40.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.986+0000 7f0260d20640 1 -- 172.21.15.16:0/3273889799 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f025c159ac0 msgr2=0x7f02440039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:43:40.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.986+0000 7f0260d20640 1 --2- 172.21.15.16:0/3273889799 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f025c159ac0 0x7f02440039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:40.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.986+0000 7f0260d20640 1 -- 172.21.15.16:0/3273889799 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0244003f30 msgr2=0x7f02440047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:43:40.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.986+0000 7f0260d20640 1 --2- 172.21.15.16:0/3273889799 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0244003f30 0x7f02440047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:40.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.986+0000 7f0260d20640 1 -- 172.21.15.16:0/3273889799 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0244003530 con 0x7f0244004d10 2023-05-22T15:43:40.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.986+0000 7f025bfff640 1 --2- 172.21.15.16:0/3273889799 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f025c159ac0 0x7f02440039b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T15:43:40.998 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.990+0000 7f0260d20640 1 --2- 172.21.15.16:0/3273889799 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0244004d10 0x7f0244005180 secure :-1 s=READY pgs=1492 cs=0 l=1 rev1=1 crypto rx=0x7f0254001e40 tx=0x7f025401c740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1662b10c7920ebba server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:43:40.998 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.990+0000 7f025affd640 1 -- 172.21.15.16:0/3273889799 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0254003b80 con 0x7f0244004d10 2023-05-22T15:43:40.998 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.990+0000 7f025affd640 1 -- 172.21.15.16:0/3273889799 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0254003ce0 con 0x7f0244004d10 2023-05-22T15:43:40.998 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.990+0000 7f025affd640 1 -- 172.21.15.16:0/3273889799 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02540052c0 con 0x7f0244004d10 2023-05-22T15:43:40.998 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.990+0000 7f0262783640 1 -- 172.21.15.16:0/3273889799 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0244004d10 msgr2=0x7f0244005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:40.998 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.990+0000 7f0262783640 1 --2- 172.21.15.16:0/3273889799 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0244004d10 0x7f0244005180 secure :-1 s=READY pgs=1492 cs=0 l=1 rev1=1 crypto rx=0x7f0254001e40 tx=0x7f025401c740 comp rx=0 tx=0).stop 2023-05-22T15:43:40.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.990+0000 7f0260d20640 1 -- 172.21.15.16:0/3273889799 reap_dead start 2023-05-22T15:43:40.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.990+0000 7f0262783640 1 -- 172.21.15.16:0/3273889799 shutdown_connections 2023-05-22T15:43:40.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.990+0000 7f0262783640 1 -- 172.21.15.16:0/3273889799 >> 172.21.15.16:0/3273889799 conn(0x7f025c0ce8d0 msgr2=0x7f025c0cecd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:43:40.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.990+0000 7f0262783640 1 -- 172.21.15.16:0/3273889799 shutdown_connections 2023-05-22T15:43:40.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.990+0000 7f0262783640 1 -- 172.21.15.16:0/3273889799 wait complete. 2023-05-22T15:43:40.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.990+0000 7f0262783640 1 Processor -- start 2023-05-22T15:43:40.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.990+0000 7f0262783640 1 -- start start 2023-05-22T15:43:40.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.990+0000 7f0262783640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0244004d10 0x7f025c0cdc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:41.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.990+0000 7f0262783640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f025c0ab7d0 0x7f025c0c6540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:41.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.990+0000 7f0262783640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f025c0ace10 0x7f025c0c6a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:41.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.990+0000 7f0262783640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f025c0ac980 con 0x7f025c0ace10 2023-05-22T15:43:41.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.990+0000 7f0262783640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f025c0abd70 con 0x7f0244004d10 2023-05-22T15:43:41.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.990+0000 7f0262783640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f025c0ac050 con 0x7f025c0ab7d0 2023-05-22T15:43:41.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.990+0000 7f0260d20640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f025c0ace10 0x7f025c0c6a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:41.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.990+0000 7f0260d20640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f025c0ace10 0x7f025c0c6a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33206/0 (socket says 172.21.15.16:33206) 2023-05-22T15:43:41.001 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.990+0000 7f0260d20640 1 -- 172.21.15.16:0/1918168753 learned_addr learned my addr 172.21.15.16:0/1918168753 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:43:41.001 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.990+0000 7f025bfff640 1 --2- 172.21.15.16:0/1918168753 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0244004d10 0x7f025c0cdc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:41.002 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.990+0000 7f025b7fe640 1 --2- 172.21.15.16:0/1918168753 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f025c0ab7d0 0x7f025c0c6540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:41.002 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.994+0000 7f0260d20640 1 -- 172.21.15.16:0/1918168753 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f025c0ab7d0 msgr2=0x7f025c0c6540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:41.003 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.994+0000 7f0260d20640 1 --2- 172.21.15.16:0/1918168753 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f025c0ab7d0 0x7f025c0c6540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:41.003 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.994+0000 7f0260d20640 1 -- 172.21.15.16:0/1918168753 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0244004d10 msgr2=0x7f025c0cdc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:41.003 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.994+0000 7f0260d20640 1 --2- 172.21.15.16:0/1918168753 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0244004d10 0x7f025c0cdc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:41.003 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.994+0000 7f0260d20640 1 -- 172.21.15.16:0/1918168753 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f025c06c1d0 con 0x7f025c0ace10 2023-05-22T15:43:41.003 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.994+0000 7f0260d20640 1 --2- 172.21.15.16:0/1918168753 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f025c0ace10 0x7f025c0c6a80 secure :-1 s=READY pgs=1493 cs=0 l=1 rev1=1 crypto rx=0x7f025401cc20 tx=0x7f0244005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:43:41.003 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.994+0000 7f0258ff9640 1 -- 172.21.15.16:0/1918168753 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02540012e0 con 0x7f025c0ace10 2023-05-22T15:43:41.003 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.994+0000 7f0258ff9640 1 -- 172.21.15.16:0/1918168753 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0254005030 con 0x7f025c0ace10 2023-05-22T15:43:41.003 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.994+0000 7f0262783640 1 -- 172.21.15.16:0/1918168753 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f025c06bef0 con 0x7f025c0ace10 2023-05-22T15:43:41.004 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.994+0000 7f0258ff9640 1 -- 172.21.15.16:0/1918168753 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02540053a0 con 0x7f025c0ace10 2023-05-22T15:43:41.004 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.994+0000 7f0262783640 1 -- 172.21.15.16:0/1918168753 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f025c06c4b0 con 0x7f025c0ace10 2023-05-22T15:43:41.004 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.994+0000 7f0262783640 1 -- 172.21.15.16:0/1918168753 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f025c0acb30 con 0x7f025c0ace10 2023-05-22T15:43:41.004 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.994+0000 7f0258ff9640 1 -- 172.21.15.16:0/1918168753 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 42) v1 ==== 84349+0+0 (secure 0 0 0) 0x7f0254005500 con 0x7f025c0ace10 2023-05-22T15:43:41.004 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.994+0000 7f0258ff9640 1 --2- 172.21.15.16:0/1918168753 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f0248065520 0x7f02480679d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:41.004 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.994+0000 7f025bfff640 1 -- 172.21.15.16:0/1918168753 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f0248065520 msgr2=0x7f02480679d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/36921 2023-05-22T15:43:41.004 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.994+0000 7f0258ff9640 1 -- 172.21.15.16:0/1918168753 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f025411c0e0 con 0x7f025c0ace10 2023-05-22T15:43:41.005 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.994+0000 7f025bfff640 1 --2- 172.21.15.16:0/1918168753 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f0248065520 0x7f02480679d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-05-22T15:43:41.005 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:40.998+0000 7f0258ff9640 1 -- 172.21.15.16:0/1918168753 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f025c0acb30 con 0x7f025c0ace10 2023-05-22T15:43:41.172 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:41.166+0000 7f0262783640 1 -- 172.21.15.16:0/1918168753 --> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/bad-path", "snap_schedule": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f025c04dbe0 con 0x7f0248065520 2023-05-22T15:43:41.203 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:41.198+0000 7f025bfff640 1 -- 172.21.15.16:0/1918168753 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f0248065520 msgr2=0x7f02480679d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/36921 2023-05-22T15:43:41.203 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:41.198+0000 7f025bfff640 1 --2- 172.21.15.16:0/1918168753 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f0248065520 0x7f02480679d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-05-22T15:43:41.298 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:43:41.294+0000 7fa394942180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-05-22T15:43:41.329 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:43:41.322+0000 7fc5b6f34180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-05-22T15:43:41.351 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:43:41.346+0000 7fa394942180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-05-22T15:43:41.382 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:43:41.374+0000 7fc5b6f34180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-05-22T15:43:41.401 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:43:41.398+0000 7fa394942180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-05-22T15:43:41.432 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:43:41.426+0000 7fc5b6f34180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-05-22T15:43:41.478 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:43:41.474+0000 7fa394942180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-05-22T15:43:41.510 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:43:41.502+0000 7fc5b6f34180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-05-22T15:43:41.530 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:43:41.526+0000 7fa394942180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-05-22T15:43:41.561 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:43:41.554+0000 7fc5b6f34180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-05-22T15:43:41.605 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:43:41.602+0000 7fa394942180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-05-22T15:43:41.606 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:41.602+0000 7f025bfff640 1 -- 172.21.15.16:0/1918168753 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f0248065520 msgr2=0x7f02480679d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/36921 2023-05-22T15:43:41.606 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:41.602+0000 7f025bfff640 1 --2- 172.21.15.16:0/1918168753 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f0248065520 0x7f02480679d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-05-22T15:43:41.638 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:43:41.630+0000 7fc5b6f34180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-05-22T15:43:41.658 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:43:41.654+0000 7fa394942180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-05-22T15:43:41.691 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:43:41.686+0000 7fc5b6f34180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-05-22T15:43:41.790 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:43:41.786+0000 7fa394942180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-05-22T15:43:41.824 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:43:41.818+0000 7fc5b6f34180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-05-22T15:43:41.843 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:43:41.838+0000 7fa394942180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-05-22T15:43:41.876 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:43:41.870+0000 7fc5b6f34180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-05-22T15:43:42.067 INFO:tasks.ceph.mgr.x.smithi062.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2023-05-22T15:43:42.067 INFO:tasks.ceph.mgr.x.smithi062.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2023-05-22T15:43:42.068 INFO:tasks.ceph.mgr.x.smithi062.stderr: from numpy import show_config as show_numpy_config 2023-05-22T15:43:42.075 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:43:42.070+0000 7fa394942180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-05-22T15:43:42.100 INFO:tasks.ceph.mgr.y.smithi016.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2023-05-22T15:43:42.100 INFO:tasks.ceph.mgr.y.smithi016.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2023-05-22T15:43:42.101 INFO:tasks.ceph.mgr.y.smithi016.stderr: from numpy import show_config as show_numpy_config 2023-05-22T15:43:42.108 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:43:42.102+0000 7fc5b6f34180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-05-22T15:43:42.200 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:43:42.194+0000 7fa394942180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-05-22T15:43:42.233 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:43:42.226+0000 7fc5b6f34180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-05-22T15:43:42.312 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:43:42.306+0000 7fa394942180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-05-22T15:43:42.346 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:43:42.338+0000 7fc5b6f34180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-05-22T15:43:42.376 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:43:42.370+0000 7fa394942180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-05-22T15:43:42.409 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:43:42.402+0000 7fc5b6f34180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-05-22T15:43:42.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:42.406+0000 7f025bfff640 1 -- 172.21.15.16:0/1918168753 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f0248065520 msgr2=0x7f02480679d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/36921 2023-05-22T15:43:42.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:42.406+0000 7f025bfff640 1 --2- 172.21.15.16:0/1918168753 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f0248065520 0x7f02480679d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-05-22T15:43:42.501 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:43:42.498+0000 7fa394942180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-05-22T15:43:42.535 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:43:42.530+0000 7fc5b6f34180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-05-22T15:43:42.994 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:43:42.990+0000 7fa394942180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-05-22T15:43:43.029 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:43:43.022+0000 7fc5b6f34180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-05-22T15:43:43.070 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:43:43.066+0000 7fa394942180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-05-22T15:43:43.105 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:43:43.098+0000 7fc5b6f34180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-05-22T15:43:43.206 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:43:43.202+0000 7fa394942180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-05-22T15:43:43.241 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:43:43.234+0000 7fc5b6f34180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-05-22T15:43:43.283 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:43:43.278+0000 7fa394942180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-05-22T15:43:43.318 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:43:43.310+0000 7fc5b6f34180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-05-22T15:43:43.531 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:43:43.526+0000 7fa394942180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-05-22T15:43:43.565 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:43:43.558+0000 7fc5b6f34180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-05-22T15:43:43.586 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:43:43.582+0000 7fa394942180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-05-22T15:43:43.620 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:43:43.614+0000 7fc5b6f34180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-05-22T15:43:43.655 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:43.650+0000 7f0258ff9640 1 -- 172.21.15.16:0/1918168753 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mgrmap(e 43) v1 ==== 83958+0+0 (secure 0 0 0) 0x7f025411a360 con 0x7f025c0ace10 2023-05-22T15:43:43.655 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:43.650+0000 7f0258ff9640 1 -- 172.21.15.16:0/1918168753 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f0248065520 msgr2=0x7f02480679d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-05-22T15:43:43.656 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:43.650+0000 7f0258ff9640 1 --2- 172.21.15.16:0/1918168753 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f0248065520 0x7f02480679d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:43.814 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:43:43.806+0000 7fc5390b1640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T15:43:43.814 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:43:43.806+0000 7fc5390b1640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T15:43:43.814 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:43:43.806+0000 7fc5390b1640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T15:43:43.814 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:43:43.806+0000 7fc5390b1640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T15:43:43.814 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:43:43.806+0000 7fc5390b1640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T15:43:43.814 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:43:43.806+0000 7fc5360ab640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T15:43:43.815 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:43:43.806+0000 7fc5360ab640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T15:43:43.815 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:43:43.806+0000 7fc5360ab640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T15:43:43.815 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:43:43.806+0000 7fc5360ab640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T15:43:43.815 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:43:43.806+0000 7fc5360ab640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T15:43:44.685 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:44.678+0000 7f0258ff9640 1 -- 172.21.15.16:0/1918168753 <== mon.0 v2:172.21.15.16:3300/0 8 ==== mgrmap(e 44) v1 ==== 84085+0+0 (secure 0 0 0) 0x7f02540e9e00 con 0x7f025c0ace10 2023-05-22T15:43:44.686 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:44.682+0000 7f0258ff9640 1 --2- 172.21.15.16:0/1918168753 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f0248065520 0x7f0248067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:44.686 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:44.682+0000 7f0258ff9640 1 -- 172.21.15.16:0/1918168753 --> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/bad-path", "snap_schedule": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f025c04dbe0 con 0x7f0248065520 2023-05-22T15:43:44.687 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:44.682+0000 7f025bfff640 1 --2- 172.21.15.16:0/1918168753 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f0248065520 0x7f0248067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:44.692 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:44.686+0000 7f025bfff640 1 --2- 172.21.15.16:0/1918168753 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f0248065520 0x7f0248067950 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f0250006a80 tx=0x7f0250008040 comp rx=0 tx=0).ready entity=mgr.6470 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:43:44.832 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:44.826+0000 7f0258ff9640 1 -- 172.21.15.16:0/1918168753 <== mgr.6470 v2:172.21.15.16:6832/36921 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7f025c04dbe0 con 0x7f0248065520 2023-05-22T15:43:44.833 INFO:teuthology.orchestra.run.smithi016.stdout:Schedule set for path /bad-path 2023-05-22T15:43:44.836 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:44.830+0000 7f0262783640 1 -- 172.21.15.16:0/1918168753 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f0248065520 msgr2=0x7f0248067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:44.836 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:44.830+0000 7f0262783640 1 --2- 172.21.15.16:0/1918168753 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f0248065520 0x7f0248067950 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f0250006a80 tx=0x7f0250008040 comp rx=0 tx=0).stop 2023-05-22T15:43:44.836 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:44.830+0000 7f0262783640 1 -- 172.21.15.16:0/1918168753 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f025c0ace10 msgr2=0x7f025c0c6a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:44.836 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:44.830+0000 7f0262783640 1 --2- 172.21.15.16:0/1918168753 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f025c0ace10 0x7f025c0c6a80 secure :-1 s=READY pgs=1493 cs=0 l=1 rev1=1 crypto rx=0x7f025401cc20 tx=0x7f0244005800 comp rx=0 tx=0).stop 2023-05-22T15:43:44.836 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:44.830+0000 7f0262783640 1 -- 172.21.15.16:0/1918168753 shutdown_connections 2023-05-22T15:43:44.837 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:44.830+0000 7f0262783640 1 --2- 172.21.15.16:0/1918168753 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f025c0ace10 0x7f025c0c6a80 unknown :-1 s=CLOSED pgs=1493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:44.837 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:44.830+0000 7f0262783640 1 --2- 172.21.15.16:0/1918168753 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f0248065520 0x7f0248067950 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:44.837 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:44.830+0000 7f0262783640 1 --2- 172.21.15.16:0/1918168753 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f025c0ab7d0 0x7f025c0c6540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:44.837 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:44.830+0000 7f0262783640 1 --2- 172.21.15.16:0/1918168753 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0244004d10 0x7f025c0cdc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:44.837 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:44.830+0000 7f0262783640 1 -- 172.21.15.16:0/1918168753 >> 172.21.15.16:0/1918168753 conn(0x7f025c0ce8d0 msgr2=0x7f025c06a170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:43:44.837 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:44.830+0000 7f0262783640 1 -- 172.21.15.16:0/1918168753 shutdown_connections 2023-05-22T15:43:44.837 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:44.830+0000 7f0262783640 1 -- 172.21.15.16:0/1918168753 wait complete. 2023-05-22T15:43:44.856 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /bad-path 2023-05-22T15:43:44.856 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --fs cephfs --format json /bad-path 2023-05-22T15:43:45.032 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:45.026+0000 7fcdb31d1640 1 Processor -- start 2023-05-22T15:43:45.032 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:45.026+0000 7fcdb31d1640 1 -- start start 2023-05-22T15:43:45.033 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:45.026+0000 7fcdb31d1640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fcdac0b02c0 0x7fcdac0ce060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:45.033 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:45.026+0000 7fcdb31d1640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fcdac0b3910 0x7fcdac0ce5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:45.033 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:45.026+0000 7fcdb31d1640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcdac0712d0 con 0x7fcdac159b60 2023-05-22T15:43:45.033 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:45.026+0000 7fcdb31d1640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcdac0cbb10 con 0x7fcdac0b3910 2023-05-22T15:43:45.033 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:45.026+0000 7fcdb31d1640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcdac070cc0 con 0x7fcdac0b02c0 2023-05-22T15:43:45.034 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:45.026+0000 7fcdb176e640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fcdac0b3910 0x7fcdac0ce5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:45.034 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:45.026+0000 7fcdb176e640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fcdac0b3910 0x7fcdac0ce5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33368/0 (socket says 172.21.15.16:33368) 2023-05-22T15:43:45.034 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:45.026+0000 7fcdb176e640 1 -- 172.21.15.16:0/155989875 learned_addr learned my addr 172.21.15.16:0/155989875 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:43:45.034 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:45.026+0000 7fcdabfff640 1 --2- 172.21.15.16:0/155989875 >> v2:172.21.15.62:3300/0 conn(0x7fcdac0b02c0 0x7fcdac0ce060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:45.035 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:45.026+0000 7fcdb176e640 1 -- 172.21.15.16:0/155989875 >> v2:172.21.15.62:3300/0 conn(0x7fcdac0b02c0 msgr2=0x7fcdac0ce060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:43:45.035 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:45.026+0000 7fcdb176e640 1 --2- 172.21.15.16:0/155989875 >> v2:172.21.15.62:3300/0 conn(0x7fcdac0b02c0 0x7fcdac0ce060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:45.035 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:45.026+0000 7fcdb176e640 1 -- 172.21.15.16:0/155989875 >> v1:172.21.15.62:6789/0 conn(0x7fcdac159b60 legacy=0x7fcdac0afbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:43:45.035 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:45.026+0000 7fcdb176e640 1 -- 172.21.15.16:0/155989875 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdac0cb550 con 0x7fcdac0b3910 2023-05-22T15:43:45.035 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:45.026+0000 7fcdabfff640 1 --2- 172.21.15.16:0/155989875 >> v2:172.21.15.62:3300/0 conn(0x7fcdac0b02c0 0x7fcdac0ce060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:43:45.036 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:45.030+0000 7fcdb176e640 1 --2- 172.21.15.16:0/155989875 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcdac0b3910 0x7fcdac0ce5a0 secure :-1 s=READY pgs=1508 cs=0 l=1 rev1=1 crypto rx=0x7fcdac0c1bd0 tx=0x7fcda401c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c53aad4809a7da2 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:43:45.036 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:45.030+0000 7fcdab7fe640 1 -- 172.21.15.16:0/155989875 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcda40a5820 con 0x7fcdac0b3910 2023-05-22T15:43:45.037 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:45.030+0000 7fcdab7fe640 1 -- 172.21.15.16:0/155989875 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcda40a5980 con 0x7fcdac0b3910 2023-05-22T15:43:45.037 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:45.030+0000 7fcdab7fe640 1 -- 172.21.15.16:0/155989875 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcda40a5d10 con 0x7fcdac0b3910 2023-05-22T15:43:45.037 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:45.030+0000 7fcdb31d1640 1 -- 172.21.15.16:0/155989875 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcdac0b3910 msgr2=0x7fcdac0ce5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:45.037 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:45.030+0000 7fcdb31d1640 1 --2- 172.21.15.16:0/155989875 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcdac0b3910 0x7fcdac0ce5a0 secure :-1 s=READY pgs=1508 cs=0 l=1 rev1=1 crypto rx=0x7fcdac0c1bd0 tx=0x7fcda401c960 comp rx=0 tx=0).stop 2023-05-22T15:43:45.037 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:45.030+0000 7fcdb31d1640 1 -- 172.21.15.16:0/155989875 shutdown_connections 2023-05-22T15:43:45.037 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:45.030+0000 7fcdb31d1640 1 --2- 172.21.15.16:0/155989875 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcdac0b3910 0x7fcdac0ce5a0 unknown :-1 s=CLOSED pgs=1508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:45.037 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:45.030+0000 7fcdb31d1640 1 --2- 172.21.15.16:0/155989875 >> v2:172.21.15.62:3300/0 conn(0x7fcdac0b02c0 0x7fcdac0ce060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:45.038 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:45.030+0000 7fcdb31d1640 1 -- 172.21.15.16:0/155989875 >> 172.21.15.16:0/155989875 conn(0x7fcdac069720 msgr2=0x7fcdac069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:43:45.038 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:45.030+0000 7fcdb31d1640 1 -- 172.21.15.16:0/155989875 shutdown_connections 2023-05-22T15:43:45.038 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:45.030+0000 7fcdb31d1640 1 -- 172.21.15.16:0/155989875 wait complete. 2023-05-22T15:43:45.040 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:45.030+0000 7fcdb31d1640 1 Processor -- start 2023-05-22T15:43:45.040 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:45.030+0000 7fcdb31d1640 1 -- start start 2023-05-22T15:43:45.040 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:45.034+0000 7fcdb31d1640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcdac0b02c0 0x7fcdac1f2e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:45.040 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:45.034+0000 7fcdb31d1640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fcdac0b3910 0x7fcdac0c4260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:45.040 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:45.034+0000 7fcdb31d1640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcdac159b60 0x7fcdac14f000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:45.041 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:45.034+0000 7fcdb31d1640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdac1f3fb0 con 0x7fcdac159b60 2023-05-22T15:43:45.041 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:45.034+0000 7fcdb31d1640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdac1f3430 con 0x7fcdac0b02c0 2023-05-22T15:43:45.041 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:45.034+0000 7fcdb31d1640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcdac1f3710 con 0x7fcdac0b3910 2023-05-22T15:43:45.041 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:45.034+0000 7fcdb176e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcdac159b60 0x7fcdac14f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:45.041 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:45.034+0000 7fcdb176e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcdac159b60 0x7fcdac14f000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33380/0 (socket says 172.21.15.16:33380) 2023-05-22T15:43:45.043 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:45.034+0000 7fcdb176e640 1 -- 172.21.15.16:0/2057482737 learned_addr learned my addr 172.21.15.16:0/2057482737 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:43:45.044 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:45.034+0000 7fcdb0f6d640 1 --2- 172.21.15.16:0/2057482737 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcdac0b02c0 0x7fcdac1f2e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:45.044 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:45.034+0000 7fcdabfff640 1 --2- 172.21.15.16:0/2057482737 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fcdac0b3910 0x7fcdac0c4260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:45.044 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:45.034+0000 7fcdb176e640 1 -- 172.21.15.16:0/2057482737 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fcdac0b3910 msgr2=0x7fcdac0c4260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:45.044 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:45.034+0000 7fcdb176e640 1 --2- 172.21.15.16:0/2057482737 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fcdac0b3910 0x7fcdac0c4260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:45.044 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:45.034+0000 7fcdb176e640 1 -- 172.21.15.16:0/2057482737 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcdac0b02c0 msgr2=0x7fcdac1f2e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:45.045 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:45.034+0000 7fcdb176e640 1 --2- 172.21.15.16:0/2057482737 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcdac0b02c0 0x7fcdac1f2e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:45.045 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:45.034+0000 7fcdb176e640 1 -- 172.21.15.16:0/2057482737 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcda4009b30 con 0x7fcdac159b60 2023-05-22T15:43:45.045 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:45.034+0000 7fcdb176e640 1 --2- 172.21.15.16:0/2057482737 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcdac159b60 0x7fcdac14f000 secure :-1 s=READY pgs=1509 cs=0 l=1 rev1=1 crypto rx=0x7fcda401a590 tx=0x7fcda4019700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:43:45.045 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:45.034+0000 7fcda97fa640 1 -- 172.21.15.16:0/2057482737 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcda40b8880 con 0x7fcdac159b60 2023-05-22T15:43:45.045 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:45.034+0000 7fcda97fa640 1 -- 172.21.15.16:0/2057482737 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcda40b8a00 con 0x7fcdac159b60 2023-05-22T15:43:45.045 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:45.034+0000 7fcdb31d1640 1 -- 172.21.15.16:0/2057482737 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcdac159f30 con 0x7fcdac159b60 2023-05-22T15:43:45.045 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:45.034+0000 7fcda97fa640 1 -- 172.21.15.16:0/2057482737 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcda40b8db0 con 0x7fcdac159b60 2023-05-22T15:43:45.045 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:45.034+0000 7fcdb31d1640 1 -- 172.21.15.16:0/2057482737 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcdac0caeb0 con 0x7fcdac159b60 2023-05-22T15:43:45.046 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:45.034+0000 7fcdb31d1640 1 -- 172.21.15.16:0/2057482737 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcdac0ceef0 con 0x7fcdac159b60 2023-05-22T15:43:45.047 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:45.034+0000 7fcda97fa640 1 -- 172.21.15.16:0/2057482737 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 44) v1 ==== 84085+0+0 (secure 0 0 0) 0x7fcda40b6440 con 0x7fcdac159b60 2023-05-22T15:43:45.047 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:45.034+0000 7fcda97fa640 1 --2- 172.21.15.16:0/2057482737 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fcd98065220 0x7fcd980676d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:43:45.047 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:45.034+0000 7fcda97fa640 1 -- 172.21.15.16:0/2057482737 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(119..119 src has 1..119) v4 ==== 6726+0+0 (secure 0 0 0) 0x7fcda411e010 con 0x7fcdac159b60 2023-05-22T15:43:45.047 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:45.034+0000 7fcdb0f6d640 1 --2- 172.21.15.16:0/2057482737 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fcd98065220 0x7fcd980676d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:43:45.047 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:45.034+0000 7fcdb0f6d640 1 --2- 172.21.15.16:0/2057482737 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fcd98065220 0x7fcd980676d0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fcda000b810 tx=0x7fcda0008040 comp rx=0 tx=0).ready entity=mgr.6470 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:43:45.047 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:45.042+0000 7fcda97fa640 1 -- 172.21.15.16:0/2057482737 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fcdac0ceef0 con 0x7fcdac159b60 2023-05-22T15:43:45.213 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:45.206+0000 7fcdb31d1640 1 -- 172.21.15.16:0/2057482737 --> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/bad-path", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fcdac1f3880 con 0x7fcd98065220 2023-05-22T15:43:45.224 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:45.218+0000 7fcda97fa640 1 -- 172.21.15.16:0/2057482737 <== mgr.6470 v2:172.21.15.16:6832/36921 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+287 (secure 0 0 0) 0x7fcdac1f3880 con 0x7fcd98065220 2023-05-22T15:43:45.225 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:43:45.225 INFO:teuthology.orchestra.run.smithi016.stdout:[{"fs": "cephfs", "subvol": null, "path": "/bad-path", "rel_path": "/bad-path", "schedule": "1M", "retention": {}, "start": "2023-05-22T00:00:00", "created": "2023-05-22T15:43:44", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": true}] 2023-05-22T15:43:45.227 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:45.222+0000 7fcdb31d1640 1 -- 172.21.15.16:0/2057482737 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fcd98065220 msgr2=0x7fcd980676d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:45.227 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:45.222+0000 7fcdb31d1640 1 --2- 172.21.15.16:0/2057482737 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fcd98065220 0x7fcd980676d0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fcda000b810 tx=0x7fcda0008040 comp rx=0 tx=0).stop 2023-05-22T15:43:45.227 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:45.222+0000 7fcdb31d1640 1 -- 172.21.15.16:0/2057482737 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcdac159b60 msgr2=0x7fcdac14f000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:43:45.227 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:45.222+0000 7fcdb31d1640 1 --2- 172.21.15.16:0/2057482737 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcdac159b60 0x7fcdac14f000 secure :-1 s=READY pgs=1509 cs=0 l=1 rev1=1 crypto rx=0x7fcda401a590 tx=0x7fcda4019700 comp rx=0 tx=0).stop 2023-05-22T15:43:45.227 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:45.222+0000 7fcdb31d1640 1 -- 172.21.15.16:0/2057482737 shutdown_connections 2023-05-22T15:43:45.227 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:45.222+0000 7fcdb31d1640 1 --2- 172.21.15.16:0/2057482737 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcdac159b60 0x7fcdac14f000 unknown :-1 s=CLOSED pgs=1509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:45.228 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:45.222+0000 7fcdb31d1640 1 --2- 172.21.15.16:0/2057482737 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fcd98065220 0x7fcd980676d0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:45.228 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:45.222+0000 7fcdb31d1640 1 --2- 172.21.15.16:0/2057482737 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fcdac0b3910 0x7fcdac0c4260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:45.228 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:45.222+0000 7fcdb31d1640 1 --2- 172.21.15.16:0/2057482737 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcdac0b02c0 0x7fcdac1f2e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:43:45.228 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:45.222+0000 7fcdb31d1640 1 -- 172.21.15.16:0/2057482737 >> 172.21.15.16:0/2057482737 conn(0x7fcdac069720 msgr2=0x7fcdac153420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:43:45.228 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:45.222+0000 7fcdb31d1640 1 -- 172.21.15.16:0/2057482737 shutdown_connections 2023-05-22T15:43:45.228 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:43:45.222+0000 7fcdb31d1640 1 -- 172.21.15.16:0/2057482737 wait complete. 2023-05-22T15:43:45.243 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "path": "/bad-path", "rel_path": "/bad-path", "schedule": "1M", "retention": {}, "start": "2023-05-22T00:00:00", "created": "2023-05-22T15:43:44", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": true}] 2023-05-22T15:43:52.198 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-05-22T15:43:52.201 DEBUG:teuthology.orchestra.run.smithi062:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-05-22T15:44:00.850 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:44:00.842+0000 7fc528890640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T15:44:00.851 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:44:00.842+0000 7fc528890640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T15:44:00.851 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:44:00.842+0000 7fc528890640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T15:44:00.851 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:44:00.842+0000 7fc528890640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T15:44:00.851 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:44:00.842+0000 7fc528890640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T15:44:22.246 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-05-22T15:44:22.250 DEBUG:teuthology.orchestra.run.smithi062:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-05-22T15:44:45.246 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --fs cephfs --format json /bad-path 2023-05-22T15:44:45.388 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:45.385+0000 7f9da0d70640 1 Processor -- start 2023-05-22T15:44:45.388 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:45.385+0000 7f9da0d70640 1 -- start start 2023-05-22T15:44:45.389 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:45.385+0000 7f9da0d70640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9d9c159a30 0x7f9d9c06e380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:45.389 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:45.385+0000 7f9da0d70640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d9c0cde70 con 0x7f9d9c0cf570 2023-05-22T15:44:45.389 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:45.385+0000 7f9da0d70640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d9c15db20 con 0x7f9d9c15d610 2023-05-22T15:44:45.389 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:45.385+0000 7f9da0d70640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d9c0cc520 con 0x7f9d9c159a30 2023-05-22T15:44:45.389 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:45.385+0000 7f9d9ad9d640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f9d9c0cf570 0x7f9d9c0ce300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:47674/0 (socket says 172.21.15.16:47674) 2023-05-22T15:44:45.389 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:45.385+0000 7f9d9ad9d640 1 -- 172.21.15.16:0/2443988582 learned_addr learned my addr 172.21.15.16:0/2443988582 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:44:45.391 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:45.385+0000 7f9d9a59c640 1 --2- 172.21.15.16:0/2443988582 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9d9c159a30 0x7f9d9c06e380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:44:45.391 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:45.385+0000 7f9d9959a640 1 -- 172.21.15.16:0/2443988582 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1408578955 0 0) 0x7f9d9c0cde70 con 0x7f9d9c0cf570 2023-05-22T15:44:45.391 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:45.385+0000 7f9d9959a640 1 -- 172.21.15.16:0/2443988582 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d84003530 con 0x7f9d9c0cf570 2023-05-22T15:44:45.392 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:45.385+0000 7f9d9959a640 1 -- 172.21.15.16:0/2443988582 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 589489292 0 0) 0x7f9d84003530 con 0x7f9d9c0cf570 2023-05-22T15:44:45.392 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:45.385+0000 7f9d9959a640 1 -- 172.21.15.16:0/2443988582 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9d9c159a30 msgr2=0x7f9d9c06e380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:44:45.392 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:45.385+0000 7f9d9959a640 1 --2- 172.21.15.16:0/2443988582 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9d9c159a30 0x7f9d9c06e380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:45.392 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:45.385+0000 7f9d9959a640 1 -- 172.21.15.16:0/2443988582 >> v1:172.21.15.62:6789/0 conn(0x7f9d9c15d610 legacy=0x7f9d9c0cbe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:44:45.392 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:45.385+0000 7f9d9959a640 1 -- 172.21.15.16:0/2443988582 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d9c06e8c0 con 0x7f9d9c0cf570 2023-05-22T15:44:45.392 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:45.385+0000 7f9d9959a640 1 -- 172.21.15.16:0/2443988582 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7f9d94001a00 con 0x7f9d9c0cf570 2023-05-22T15:44:45.392 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:45.385+0000 7f9d9959a640 1 -- 172.21.15.16:0/2443988582 >> v1:172.21.15.16:6789/0 conn(0x7f9d9c0cf570 legacy=0x7f9d9c0ce300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:44:45.392 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:45.385+0000 7f9d9959a640 1 --2- 172.21.15.16:0/2443988582 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9d840039b0 0x7f9d84003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:45.393 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:45.385+0000 7f9d9959a640 1 --2- 172.21.15.16:0/2443988582 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9d9c159a30 0x7f9d840047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:45.393 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:45.385+0000 7f9d9959a640 1 --2- 172.21.15.16:0/2443988582 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9d84004d10 0x7f9d84005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:45.393 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:45.385+0000 7f9d9959a640 1 -- 172.21.15.16:0/2443988582 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d9c0cc520 con 0x7f9d84004d10 2023-05-22T15:44:45.393 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:45.385+0000 7f9d9959a640 1 -- 172.21.15.16:0/2443988582 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d8400ba40 con 0x7f9d840039b0 2023-05-22T15:44:45.393 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:45.385+0000 7f9d9959a640 1 -- 172.21.15.16:0/2443988582 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d8400bbd0 con 0x7f9d9c159a30 2023-05-22T15:44:45.393 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:45.385+0000 7f9d9ad9d640 1 --2- 172.21.15.16:0/2443988582 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9d84004d10 0x7f9d84005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:44:45.393 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:45.385+0000 7f9d9a59c640 1 --2- 172.21.15.16:0/2443988582 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9d840039b0 0x7f9d84003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:44:45.394 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:45.385+0000 7f9d99d9b640 1 --2- 172.21.15.16:0/2443988582 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9d9c159a30 0x7f9d840047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:44:45.394 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:45.389+0000 7f9d9ad9d640 1 -- 172.21.15.16:0/2443988582 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9d9c159a30 msgr2=0x7f9d840047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:44:45.394 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:45.389+0000 7f9d9ad9d640 1 --2- 172.21.15.16:0/2443988582 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9d9c159a30 0x7f9d840047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:45.394 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:45.389+0000 7f9d9ad9d640 1 -- 172.21.15.16:0/2443988582 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9d840039b0 msgr2=0x7f9d84003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:44:45.394 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:45.389+0000 7f9d9ad9d640 1 --2- 172.21.15.16:0/2443988582 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9d840039b0 0x7f9d84003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:45.394 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:45.389+0000 7f9d9ad9d640 1 -- 172.21.15.16:0/2443988582 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d84003530 con 0x7f9d84004d10 2023-05-22T15:44:45.394 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:45.389+0000 7f9d9a59c640 1 --2- 172.21.15.16:0/2443988582 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9d840039b0 0x7f9d84003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T15:44:45.398 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:45.389+0000 7f9d9ad9d640 1 --2- 172.21.15.16:0/2443988582 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9d84004d10 0x7f9d84005180 secure :-1 s=READY pgs=1513 cs=0 l=1 rev1=1 crypto rx=0x7f9d9400a830 tx=0x7f9d9401d340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f8fbd370d612f1b server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:44:45.398 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:45.389+0000 7f9d9959a640 1 -- 172.21.15.16:0/2443988582 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d940072d0 con 0x7f9d84004d10 2023-05-22T15:44:45.398 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:45.389+0000 7f9d9959a640 1 -- 172.21.15.16:0/2443988582 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d940a8070 con 0x7f9d84004d10 2023-05-22T15:44:45.399 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:45.389+0000 7f9d9959a640 1 -- 172.21.15.16:0/2443988582 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d940076b0 con 0x7f9d84004d10 2023-05-22T15:44:45.399 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:45.389+0000 7f9da0d70640 1 -- 172.21.15.16:0/2443988582 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9d84004d10 msgr2=0x7f9d84005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:44:45.399 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:45.389+0000 7f9da0d70640 1 --2- 172.21.15.16:0/2443988582 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9d84004d10 0x7f9d84005180 secure :-1 s=READY pgs=1513 cs=0 l=1 rev1=1 crypto rx=0x7f9d9400a830 tx=0x7f9d9401d340 comp rx=0 tx=0).stop 2023-05-22T15:44:45.399 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:45.389+0000 7f9d9ad9d640 1 -- 172.21.15.16:0/2443988582 reap_dead start 2023-05-22T15:44:45.399 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:45.389+0000 7f9da0d70640 1 -- 172.21.15.16:0/2443988582 shutdown_connections 2023-05-22T15:44:45.400 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:45.389+0000 7f9da0d70640 1 -- 172.21.15.16:0/2443988582 >> 172.21.15.16:0/2443988582 conn(0x7f9d9c0700b0 msgr2=0x7f9d9c06d9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:44:45.400 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:45.389+0000 7f9da0d70640 1 -- 172.21.15.16:0/2443988582 shutdown_connections 2023-05-22T15:44:45.400 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:45.389+0000 7f9da0d70640 1 -- 172.21.15.16:0/2443988582 wait complete. 2023-05-22T15:44:45.400 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:45.393+0000 7f9da0d70640 1 Processor -- start 2023-05-22T15:44:45.400 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:45.393+0000 7f9da0d70640 1 -- start start 2023-05-22T15:44:45.401 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:45.393+0000 7f9da0d70640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9d84004d10 0x7f9d9c06c1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:45.401 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:45.393+0000 7f9da0d70640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9d9c0c83c0 0x7f9d9c1f4b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:45.401 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:45.393+0000 7f9da0d70640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9d9c0c99d0 0x7f9d9c155660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:45.401 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:45.393+0000 7f9da0d70640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d9c0c8930 con 0x7f9d9c0c99d0 2023-05-22T15:44:45.401 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:45.393+0000 7f9da0d70640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d9c0c8c10 con 0x7f9d84004d10 2023-05-22T15:44:45.401 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:45.393+0000 7f9da0d70640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d9c0c8ef0 con 0x7f9d9c0c83c0 2023-05-22T15:44:45.401 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:45.393+0000 7f9d9ad9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9d9c0c99d0 0x7f9d9c155660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:44:45.402 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:45.393+0000 7f9d9ad9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9d9c0c99d0 0x7f9d9c155660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35560/0 (socket says 172.21.15.16:35560) 2023-05-22T15:44:45.402 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:45.393+0000 7f9d9ad9d640 1 -- 172.21.15.16:0/2930494579 learned_addr learned my addr 172.21.15.16:0/2930494579 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:44:45.402 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:45.393+0000 7f9d99d9b640 1 --2- 172.21.15.16:0/2930494579 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9d9c0c83c0 0x7f9d9c1f4b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:44:45.402 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:45.393+0000 7f9d9a59c640 1 --2- 172.21.15.16:0/2930494579 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9d84004d10 0x7f9d9c06c1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:44:45.402 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:45.393+0000 7f9d9ad9d640 1 -- 172.21.15.16:0/2930494579 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9d9c0c83c0 msgr2=0x7f9d9c1f4b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:44:45.402 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:45.393+0000 7f9d9ad9d640 1 --2- 172.21.15.16:0/2930494579 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9d9c0c83c0 0x7f9d9c1f4b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:45.403 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:45.393+0000 7f9d9ad9d640 1 -- 172.21.15.16:0/2930494579 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9d84004d10 msgr2=0x7f9d9c06c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:44:45.403 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:45.393+0000 7f9d9ad9d640 1 --2- 172.21.15.16:0/2930494579 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9d84004d10 0x7f9d9c06c1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:45.403 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:45.393+0000 7f9d9ad9d640 1 -- 172.21.15.16:0/2930494579 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d9c15e0c0 con 0x7f9d9c0c99d0 2023-05-22T15:44:45.403 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:45.393+0000 7f9d9ad9d640 1 --2- 172.21.15.16:0/2930494579 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9d9c0c99d0 0x7f9d9c155660 secure :-1 s=READY pgs=1514 cs=0 l=1 rev1=1 crypto rx=0x7f9d9401d170 tx=0x7f9d84005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:44:45.403 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:45.393+0000 7f9d8affd640 1 -- 172.21.15.16:0/2930494579 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d940bb070 con 0x7f9d9c0c99d0 2023-05-22T15:44:45.403 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:45.393+0000 7f9d8affd640 1 -- 172.21.15.16:0/2930494579 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d940011d0 con 0x7f9d9c0c99d0 2023-05-22T15:44:45.403 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:45.393+0000 7f9d8affd640 1 -- 172.21.15.16:0/2930494579 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d940019e0 con 0x7f9d9c0c99d0 2023-05-22T15:44:45.403 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:45.393+0000 7f9da0d70640 1 -- 172.21.15.16:0/2930494579 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d9c15dde0 con 0x7f9d9c0c99d0 2023-05-22T15:44:45.404 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:45.393+0000 7f9da0d70640 1 -- 172.21.15.16:0/2930494579 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d9c15e3a0 con 0x7f9d9c0c99d0 2023-05-22T15:44:45.404 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:45.393+0000 7f9da0d70640 1 -- 172.21.15.16:0/2930494579 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d9c066910 con 0x7f9d9c0c99d0 2023-05-22T15:44:45.404 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:45.393+0000 7f9d8affd640 1 -- 172.21.15.16:0/2930494579 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 46) v1 ==== 84463+0+0 (secure 0 0 0) 0x7f9d940aa030 con 0x7f9d9c0c99d0 2023-05-22T15:44:45.404 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:45.397+0000 7f9d8affd640 1 --2- 172.21.15.16:0/2930494579 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f9d7c065610 0x7f9d7c067ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:45.405 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:45.397+0000 7f9d8affd640 1 -- 172.21.15.16:0/2930494579 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(119..119 src has 1..119) v4 ==== 6726+0+0 (secure 0 0 0) 0x7f9d9411c980 con 0x7f9d9c0c99d0 2023-05-22T15:44:45.405 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:45.397+0000 7f9d9a59c640 1 --2- 172.21.15.16:0/2930494579 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f9d7c065610 0x7f9d7c067ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:44:45.405 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:45.397+0000 7f9d9a59c640 1 --2- 172.21.15.16:0/2930494579 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f9d7c065610 0x7f9d7c067ac0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f9d90009a30 tx=0x7f9d90008040 comp rx=0 tx=0).ready entity=mgr.6470 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:44:45.405 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:45.401+0000 7f9d8affd640 1 -- 172.21.15.16:0/2930494579 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f9d940ea050 con 0x7f9d9c0c99d0 2023-05-22T15:44:45.579 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:45.577+0000 7f9da0d70640 1 -- 172.21.15.16:0/2930494579 --> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/bad-path", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f9d9c04dbe0 con 0x7f9d7c065610 2023-05-22T15:44:45.589 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:45.585+0000 7f9d8affd640 1 -- 172.21.15.16:0/2930494579 <== mgr.6470 v2:172.21.15.16:6832/36921 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+288 (secure 0 0 0) 0x7f9d9c04dbe0 con 0x7f9d7c065610 2023-05-22T15:44:45.590 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:44:45.590 INFO:teuthology.orchestra.run.smithi016.stdout:[{"fs": "cephfs", "subvol": null, "path": "/bad-path", "rel_path": "/bad-path", "schedule": "1M", "retention": {}, "start": "2023-05-22T00:00:00", "created": "2023-05-22T15:43:44", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": false}] 2023-05-22T15:44:45.592 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:45.589+0000 7f9da0d70640 1 -- 172.21.15.16:0/2930494579 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f9d7c065610 msgr2=0x7f9d7c067ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:44:45.592 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:45.589+0000 7f9da0d70640 1 --2- 172.21.15.16:0/2930494579 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f9d7c065610 0x7f9d7c067ac0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f9d90009a30 tx=0x7f9d90008040 comp rx=0 tx=0).stop 2023-05-22T15:44:45.592 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:45.589+0000 7f9da0d70640 1 -- 172.21.15.16:0/2930494579 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9d9c0c99d0 msgr2=0x7f9d9c155660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:44:45.592 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:45.589+0000 7f9da0d70640 1 --2- 172.21.15.16:0/2930494579 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9d9c0c99d0 0x7f9d9c155660 secure :-1 s=READY pgs=1514 cs=0 l=1 rev1=1 crypto rx=0x7f9d9401d170 tx=0x7f9d84005800 comp rx=0 tx=0).stop 2023-05-22T15:44:45.593 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:45.589+0000 7f9da0d70640 1 -- 172.21.15.16:0/2930494579 shutdown_connections 2023-05-22T15:44:45.593 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:45.589+0000 7f9da0d70640 1 --2- 172.21.15.16:0/2930494579 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9d9c0c99d0 0x7f9d9c155660 unknown :-1 s=CLOSED pgs=1514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:45.593 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:45.589+0000 7f9da0d70640 1 --2- 172.21.15.16:0/2930494579 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f9d7c065610 0x7f9d7c067ac0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:45.593 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:45.589+0000 7f9da0d70640 1 --2- 172.21.15.16:0/2930494579 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9d9c0c83c0 0x7f9d9c1f4b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:45.593 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:45.589+0000 7f9da0d70640 1 --2- 172.21.15.16:0/2930494579 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9d84004d10 0x7f9d9c06c1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:45.593 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:45.589+0000 7f9da0d70640 1 -- 172.21.15.16:0/2930494579 >> 172.21.15.16:0/2930494579 conn(0x7f9d9c0700b0 msgr2=0x7f9d9c0cbe10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:44:45.593 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:45.589+0000 7f9da0d70640 1 -- 172.21.15.16:0/2930494579 shutdown_connections 2023-05-22T15:44:45.593 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:45.589+0000 7f9da0d70640 1 -- 172.21.15.16:0/2930494579 wait complete. 2023-05-22T15:44:45.610 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "path": "/bad-path", "rel_path": "/bad-path", "schedule": "1M", "retention": {}, "start": "2023-05-22T00:00:00", "created": "2023-05-22T15:43:44", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": false}] 2023-05-22T15:44:45.610 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove --fs cephfs /bad-path 2023-05-22T15:44:45.737 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:45.733+0000 7fb7b02d5640 1 Processor -- start 2023-05-22T15:44:45.738 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:45.733+0000 7fb7b02d5640 1 -- start start 2023-05-22T15:44:45.740 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:45.737+0000 7fb7b02d5640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb7a8157f70 0x7fb7a80c33f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:45.741 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:45.737+0000 7fb7b02d5640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fb7a8158340 0x7fb7a80cbc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:45.741 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:45.737+0000 7fb7b02d5640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb7a815bcf0 0x7fb7a80cc1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:45.741 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:45.737+0000 7fb7b02d5640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7a806f770 con 0x7fb7a8157f70 2023-05-22T15:44:45.741 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:45.737+0000 7fb7b02d5640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7a80c30e0 con 0x7fb7a8158340 2023-05-22T15:44:45.742 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:45.737+0000 7fb7b02d5640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7a80c3a40 con 0x7fb7a815bcf0 2023-05-22T15:44:45.742 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:45.737+0000 7fb7ae071640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb7a8157f70 0x7fb7a80c33f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:44:45.742 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:45.737+0000 7fb7ae071640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb7a8157f70 0x7fb7a80c33f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35562/0 (socket says 172.21.15.16:35562) 2023-05-22T15:44:45.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:45.737+0000 7fb7ae071640 1 -- 172.21.15.16:0/1908307207 learned_addr learned my addr 172.21.15.16:0/1908307207 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:44:45.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:45.737+0000 7fb7ae872640 1 --2- 172.21.15.16:0/1908307207 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb7a815bcf0 0x7fb7a80cc1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:44:45.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:45.737+0000 7fb7ad870640 1 --2- 172.21.15.16:0/1908307207 >> v2:172.21.15.62:3300/0 conn(0x7fb7a8158340 0x7fb7a80cbc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:44:45.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:45.737+0000 7fb7ae071640 1 -- 172.21.15.16:0/1908307207 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb7a815bcf0 msgr2=0x7fb7a80cc1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:44:45.744 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:45.737+0000 7fb7ae071640 1 --2- 172.21.15.16:0/1908307207 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb7a815bcf0 0x7fb7a80cc1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:45.744 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:45.737+0000 7fb7ae071640 1 -- 172.21.15.16:0/1908307207 >> v2:172.21.15.62:3300/0 conn(0x7fb7a8158340 msgr2=0x7fb7a80cbc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:44:45.744 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:45.737+0000 7fb7ae071640 1 --2- 172.21.15.16:0/1908307207 >> v2:172.21.15.62:3300/0 conn(0x7fb7a8158340 0x7fb7a80cbc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:45.744 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:45.737+0000 7fb7ae071640 1 -- 172.21.15.16:0/1908307207 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7a80c3d20 con 0x7fb7a8157f70 2023-05-22T15:44:45.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:45.737+0000 7fb7ae872640 1 --2- 172.21.15.16:0/1908307207 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb7a815bcf0 0x7fb7a80cc1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:44:45.749 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:45.737+0000 7fb7ae071640 1 --2- 172.21.15.16:0/1908307207 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb7a8157f70 0x7fb7a80c33f0 secure :-1 s=READY pgs=1515 cs=0 l=1 rev1=1 crypto rx=0x7fb7a80adbf0 tx=0x7fb7a401c9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45eefd4382b00e58 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:44:45.749 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:45.737+0000 7fb7ad06f640 1 -- 172.21.15.16:0/1908307207 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7a40a5820 con 0x7fb7a8157f70 2023-05-22T15:44:45.750 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:45.737+0000 7fb7ad06f640 1 -- 172.21.15.16:0/1908307207 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7a40a5980 con 0x7fb7a8157f70 2023-05-22T15:44:45.750 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:45.737+0000 7fb7ad06f640 1 -- 172.21.15.16:0/1908307207 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7a40a5d10 con 0x7fb7a8157f70 2023-05-22T15:44:45.751 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:45.737+0000 7fb7b02d5640 1 -- 172.21.15.16:0/1908307207 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb7a8157f70 msgr2=0x7fb7a80c33f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:44:45.751 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:45.737+0000 7fb7b02d5640 1 --2- 172.21.15.16:0/1908307207 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb7a8157f70 0x7fb7a80c33f0 secure :-1 s=READY pgs=1515 cs=0 l=1 rev1=1 crypto rx=0x7fb7a80adbf0 tx=0x7fb7a401c9a0 comp rx=0 tx=0).stop 2023-05-22T15:44:45.751 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:45.737+0000 7fb7b02d5640 1 -- 172.21.15.16:0/1908307207 shutdown_connections 2023-05-22T15:44:45.751 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:45.737+0000 7fb7b02d5640 1 --2- 172.21.15.16:0/1908307207 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb7a815bcf0 0x7fb7a80cc1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:45.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:45.737+0000 7fb7b02d5640 1 --2- 172.21.15.16:0/1908307207 >> v2:172.21.15.62:3300/0 conn(0x7fb7a8158340 0x7fb7a80cbc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:45.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:45.741+0000 7fb7b02d5640 1 --2- 172.21.15.16:0/1908307207 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb7a8157f70 0x7fb7a80c33f0 unknown :-1 s=CLOSED pgs=1515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:45.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:45.741+0000 7fb7b02d5640 1 -- 172.21.15.16:0/1908307207 >> 172.21.15.16:0/1908307207 conn(0x7fb7a8069720 msgr2=0x7fb7a8069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:44:45.753 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:45.741+0000 7fb7b02d5640 1 -- 172.21.15.16:0/1908307207 shutdown_connections 2023-05-22T15:44:45.753 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:45.741+0000 7fb7b02d5640 1 -- 172.21.15.16:0/1908307207 wait complete. 2023-05-22T15:44:45.753 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:45.741+0000 7fb7b02d5640 1 Processor -- start 2023-05-22T15:44:45.753 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:45.741+0000 7fb7b02d5640 1 -- start start 2023-05-22T15:44:45.754 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:45.741+0000 7fb7b02d5640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb7a8157f70 0x7fb7a8153b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:45.754 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:45.741+0000 7fb7b02d5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb7a8158340 0x7fb7a80b1d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:45.754 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:45.741+0000 7fb7b02d5640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb7a815bcf0 0x7fb7a80b2250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:45.754 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:45.741+0000 7fb7b02d5640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7a80c7f10 con 0x7fb7a8158340 2023-05-22T15:44:45.755 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:45.741+0000 7fb7b02d5640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7a80c8770 con 0x7fb7a815bcf0 2023-05-22T15:44:45.755 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:45.741+0000 7fb7b02d5640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7a80c8a50 con 0x7fb7a8157f70 2023-05-22T15:44:45.755 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:45.741+0000 7fb7ad870640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb7a8158340 0x7fb7a80b1d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:44:45.755 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:45.741+0000 7fb7ad870640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb7a8158340 0x7fb7a80b1d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35572/0 (socket says 172.21.15.16:35572) 2023-05-22T15:44:45.756 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:45.741+0000 7fb7ad870640 1 -- 172.21.15.16:0/985065868 learned_addr learned my addr 172.21.15.16:0/985065868 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:44:45.756 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:45.741+0000 7fb7ae071640 1 --2- 172.21.15.16:0/985065868 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb7a8157f70 0x7fb7a8153b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:44:45.756 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:45.741+0000 7fb7ae872640 1 --2- 172.21.15.16:0/985065868 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb7a815bcf0 0x7fb7a80b2250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:44:45.756 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:45.741+0000 7fb7ad870640 1 -- 172.21.15.16:0/985065868 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb7a8157f70 msgr2=0x7fb7a8153b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:44:45.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:45.741+0000 7fb7ad870640 1 --2- 172.21.15.16:0/985065868 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb7a8157f70 0x7fb7a8153b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:45.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:45.741+0000 7fb7ad870640 1 -- 172.21.15.16:0/985065868 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb7a815bcf0 msgr2=0x7fb7a80b2250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:44:45.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:45.741+0000 7fb7ad870640 1 --2- 172.21.15.16:0/985065868 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb7a815bcf0 0x7fb7a80b2250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:45.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:45.741+0000 7fb7ad870640 1 -- 172.21.15.16:0/985065868 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7a4009b30 con 0x7fb7a8158340 2023-05-22T15:44:45.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:45.741+0000 7fb7ae872640 1 --2- 172.21.15.16:0/985065868 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb7a815bcf0 0x7fb7a80b2250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T15:44:45.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:45.741+0000 7fb7ad870640 1 --2- 172.21.15.16:0/985065868 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb7a8158340 0x7fb7a80b1d10 secure :-1 s=READY pgs=1516 cs=0 l=1 rev1=1 crypto rx=0x7fb79800c7f0 tx=0x7fb79800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:44:45.758 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:45.741+0000 7fb79effd640 1 -- 172.21.15.16:0/985065868 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb798004220 con 0x7fb7a8158340 2023-05-22T15:44:45.758 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:45.741+0000 7fb79effd640 1 -- 172.21.15.16:0/985065868 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb798004380 con 0x7fb7a8158340 2023-05-22T15:44:45.758 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:45.741+0000 7fb79effd640 1 -- 172.21.15.16:0/985065868 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb798003860 con 0x7fb7a8158340 2023-05-22T15:44:45.759 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:45.741+0000 7fb7b02d5640 1 -- 172.21.15.16:0/985065868 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7a806c020 con 0x7fb7a8158340 2023-05-22T15:44:45.759 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:45.741+0000 7fb7b02d5640 1 -- 172.21.15.16:0/985065868 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7a806be90 con 0x7fb7a8158340 2023-05-22T15:44:45.759 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:45.745+0000 7fb7b02d5640 1 -- 172.21.15.16:0/985065868 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7a806c1b0 con 0x7fb7a8158340 2023-05-22T15:44:45.759 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:45.745+0000 7fb79effd640 1 -- 172.21.15.16:0/985065868 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 46) v1 ==== 84463+0+0 (secure 0 0 0) 0x7fb7980039c0 con 0x7fb7a8158340 2023-05-22T15:44:45.760 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:45.745+0000 7fb79effd640 1 --2- 172.21.15.16:0/985065868 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fb78c065610 0x7fb78c067ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:45.760 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:45.745+0000 7fb79effd640 1 -- 172.21.15.16:0/985065868 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(119..119 src has 1..119) v4 ==== 6726+0+0 (secure 0 0 0) 0x7fb798082dd0 con 0x7fb7a8158340 2023-05-22T15:44:45.760 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:45.745+0000 7fb7ae071640 1 --2- 172.21.15.16:0/985065868 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fb78c065610 0x7fb78c067ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:44:45.760 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:45.745+0000 7fb7ae071640 1 --2- 172.21.15.16:0/985065868 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fb78c065610 0x7fb78c067ac0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fb7a40ae000 tx=0x7fb7a40a7070 comp rx=0 tx=0).ready entity=mgr.6470 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:44:45.761 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:45.749+0000 7fb79effd640 1 -- 172.21.15.16:0/985065868 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fb7980502e0 con 0x7fb7a8158340 2023-05-22T15:44:45.926 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:45.921+0000 7fb7b02d5640 1 -- 172.21.15.16:0/985065868 --> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "fs": "cephfs", "path": "/bad-path", "target": ["mon-mgr", ""]}) v1 -- 0x7fb7a804dbe0 con 0x7fb78c065610 2023-05-22T15:44:46.058 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:46.053+0000 7fb79effd640 1 -- 172.21.15.16:0/985065868 <== mgr.6470 v2:172.21.15.16:6832/36921 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+36 (secure 0 0 0) 0x7fb7a804dbe0 con 0x7fb78c065610 2023-05-22T15:44:46.058 INFO:teuthology.orchestra.run.smithi016.stdout:Schedule removed for path /bad-path 2023-05-22T15:44:46.061 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:46.057+0000 7fb7b02d5640 1 -- 172.21.15.16:0/985065868 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fb78c065610 msgr2=0x7fb78c067ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:44:46.061 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:46.057+0000 7fb7b02d5640 1 --2- 172.21.15.16:0/985065868 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fb78c065610 0x7fb78c067ac0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fb7a40ae000 tx=0x7fb7a40a7070 comp rx=0 tx=0).stop 2023-05-22T15:44:46.061 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:46.057+0000 7fb7b02d5640 1 -- 172.21.15.16:0/985065868 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb7a8158340 msgr2=0x7fb7a80b1d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:44:46.061 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:46.057+0000 7fb7b02d5640 1 --2- 172.21.15.16:0/985065868 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb7a8158340 0x7fb7a80b1d10 secure :-1 s=READY pgs=1516 cs=0 l=1 rev1=1 crypto rx=0x7fb79800c7f0 tx=0x7fb79800ccb0 comp rx=0 tx=0).stop 2023-05-22T15:44:46.062 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:46.057+0000 7fb7b02d5640 1 -- 172.21.15.16:0/985065868 shutdown_connections 2023-05-22T15:44:46.062 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:46.057+0000 7fb7b02d5640 1 --2- 172.21.15.16:0/985065868 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb7a815bcf0 0x7fb7a80b2250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:46.062 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:46.057+0000 7fb7b02d5640 1 --2- 172.21.15.16:0/985065868 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb7a8158340 0x7fb7a80b1d10 unknown :-1 s=CLOSED pgs=1516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:46.062 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:46.057+0000 7fb7b02d5640 1 --2- 172.21.15.16:0/985065868 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fb78c065610 0x7fb78c067ac0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:46.062 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:46.057+0000 7fb7b02d5640 1 --2- 172.21.15.16:0/985065868 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb7a8157f70 0x7fb7a8153b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:46.062 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:46.057+0000 7fb7b02d5640 1 -- 172.21.15.16:0/985065868 >> 172.21.15.16:0/985065868 conn(0x7fb7a8069720 msgr2=0x7fb7a8069af0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:44:46.062 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:46.057+0000 7fb7b02d5640 1 -- 172.21.15.16:0/985065868 shutdown_connections 2023-05-22T15:44:46.063 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:46.057+0000 7fb7b02d5640 1 -- 172.21.15.16:0/985065868 wait complete. 2023-05-22T15:44:46.082 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /bad-path 2023-05-22T15:44:46.083 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2023-05-22T15:44:46.214 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:46.209+0000 7f0faf9c1640 1 Processor -- start 2023-05-22T15:44:46.214 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:46.209+0000 7f0faf9c1640 1 -- start start 2023-05-22T15:44:46.215 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:46.213+0000 7f0faf9c1640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f0fa8158210 0x7f0fa8159160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:46.215 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:46.213+0000 7f0faf9c1640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f0fa815bcd0 0x7f0fa815c120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:46.216 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:46.213+0000 7f0faf9c1640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0fa8158cd0 con 0x7f0fa80c8660 2023-05-22T15:44:46.216 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:46.213+0000 7f0faf9c1640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0fa81596a0 con 0x7f0fa815bcd0 2023-05-22T15:44:46.216 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:46.213+0000 7f0faf9c1640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0fa815c790 con 0x7f0fa8158210 2023-05-22T15:44:46.216 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:46.213+0000 7f0facf5c640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f0fa815bcd0 0x7f0fa815c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:44:46.216 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:46.213+0000 7f0fadf5e640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f0fa80c8660 0x7f0fa80ca690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:47682/0 (socket says 172.21.15.16:47682) 2023-05-22T15:44:46.216 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:46.213+0000 7f0fadf5e640 1 -- 172.21.15.16:0/1206410353 learned_addr learned my addr 172.21.15.16:0/1206410353 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:44:46.216 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:46.213+0000 7f0fad75d640 1 --2- 172.21.15.16:0/1206410353 >> v2:172.21.15.62:3300/0 conn(0x7f0fa8158210 0x7f0fa8159160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:44:46.217 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:46.213+0000 7f0facf5c640 1 -- 172.21.15.16:0/1206410353 >> v2:172.21.15.62:3300/0 conn(0x7f0fa8158210 msgr2=0x7f0fa8159160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:44:46.217 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:46.213+0000 7f0facf5c640 1 --2- 172.21.15.16:0/1206410353 >> v2:172.21.15.62:3300/0 conn(0x7f0fa8158210 0x7f0fa8159160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:46.217 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:46.213+0000 7f0facf5c640 1 -- 172.21.15.16:0/1206410353 >> v1:172.21.15.16:6789/0 conn(0x7f0fa80c8660 legacy=0x7f0fa80ca690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:44:46.217 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:46.213+0000 7f0f9ffff640 1 -- 172.21.15.16:0/1206410353 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3084437399 0 0) 0x7f0fa8158cd0 con 0x7f0fa80c8660 2023-05-22T15:44:46.217 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:46.213+0000 7f0facf5c640 1 -- 172.21.15.16:0/1206410353 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fa815ca70 con 0x7f0fa815bcd0 2023-05-22T15:44:46.217 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:46.213+0000 7f0fad75d640 1 --2- 172.21.15.16:0/1206410353 >> v2:172.21.15.62:3300/0 conn(0x7f0fa8158210 0x7f0fa8159160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:44:46.218 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:46.213+0000 7f0facf5c640 1 --2- 172.21.15.16:0/1206410353 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0fa815bcd0 0x7f0fa815c120 secure :-1 s=READY pgs=1518 cs=0 l=1 rev1=1 crypto rx=0x7f0fa806f2d0 tx=0x7f0f9801d770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad8237f24a4802c4 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:44:46.218 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:46.213+0000 7f0f9ffff640 1 -- 172.21.15.16:0/1206410353 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f980a6860 con 0x7f0fa815bcd0 2023-05-22T15:44:46.219 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:46.213+0000 7f0f9ffff640 1 -- 172.21.15.16:0/1206410353 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f980a69c0 con 0x7f0fa815bcd0 2023-05-22T15:44:46.219 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:46.213+0000 7f0f9ffff640 1 -- 172.21.15.16:0/1206410353 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f980a6d50 con 0x7f0fa815bcd0 2023-05-22T15:44:46.219 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:46.213+0000 7f0faf9c1640 1 -- 172.21.15.16:0/1206410353 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0fa815bcd0 msgr2=0x7f0fa815c120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:44:46.220 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:46.213+0000 7f0faf9c1640 1 --2- 172.21.15.16:0/1206410353 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0fa815bcd0 0x7f0fa815c120 secure :-1 s=READY pgs=1518 cs=0 l=1 rev1=1 crypto rx=0x7f0fa806f2d0 tx=0x7f0f9801d770 comp rx=0 tx=0).stop 2023-05-22T15:44:46.220 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:46.213+0000 7f0faf9c1640 1 -- 172.21.15.16:0/1206410353 shutdown_connections 2023-05-22T15:44:46.220 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:46.213+0000 7f0faf9c1640 1 --2- 172.21.15.16:0/1206410353 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0fa815bcd0 0x7f0fa815c120 unknown :-1 s=CLOSED pgs=1518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:46.220 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:46.213+0000 7f0faf9c1640 1 --2- 172.21.15.16:0/1206410353 >> v2:172.21.15.62:3300/0 conn(0x7f0fa8158210 0x7f0fa8159160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:46.221 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:46.213+0000 7f0faf9c1640 1 -- 172.21.15.16:0/1206410353 >> 172.21.15.16:0/1206410353 conn(0x7f0fa8069720 msgr2=0x7f0fa8069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:44:46.221 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:46.217+0000 7f0faf9c1640 1 -- 172.21.15.16:0/1206410353 shutdown_connections 2023-05-22T15:44:46.221 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:46.217+0000 7f0faf9c1640 1 -- 172.21.15.16:0/1206410353 wait complete. 2023-05-22T15:44:46.223 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:46.217+0000 7f0faf9c1640 1 Processor -- start 2023-05-22T15:44:46.223 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:46.217+0000 7f0faf9c1640 1 -- start start 2023-05-22T15:44:46.223 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:46.217+0000 7f0faf9c1640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0fa80c8660 0x7f0fa80c6080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:46.223 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:46.217+0000 7f0faf9c1640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0fa8158210 0x7f0fa8151330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:46.223 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:46.217+0000 7f0faf9c1640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0fa815bcd0 0x7f0fa80c72f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:46.223 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:46.217+0000 7f0faf9c1640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fa80c50e0 con 0x7f0fa815bcd0 2023-05-22T15:44:46.223 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:46.217+0000 7f0faf9c1640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fa80c7e00 con 0x7f0fa80c8660 2023-05-22T15:44:46.224 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:46.217+0000 7f0faf9c1640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0fa80c52c0 con 0x7f0fa8158210 2023-05-22T15:44:46.224 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:46.217+0000 7f0fadf5e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0fa815bcd0 0x7f0fa80c72f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:44:46.224 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:46.217+0000 7f0fadf5e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0fa815bcd0 0x7f0fa80c72f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35586/0 (socket says 172.21.15.16:35586) 2023-05-22T15:44:46.224 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:46.217+0000 7f0fadf5e640 1 -- 172.21.15.16:0/4175474527 learned_addr learned my addr 172.21.15.16:0/4175474527 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:44:46.224 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:46.217+0000 7f0fad75d640 1 --2- 172.21.15.16:0/4175474527 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0fa80c8660 0x7f0fa80c6080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:44:46.224 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:46.217+0000 7f0facf5c640 1 --2- 172.21.15.16:0/4175474527 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0fa8158210 0x7f0fa8151330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:44:46.225 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:46.217+0000 7f0fadf5e640 1 -- 172.21.15.16:0/4175474527 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0fa8158210 msgr2=0x7f0fa8151330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:44:46.225 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:46.217+0000 7f0fadf5e640 1 --2- 172.21.15.16:0/4175474527 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0fa8158210 0x7f0fa8151330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:46.225 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:46.217+0000 7f0fadf5e640 1 -- 172.21.15.16:0/4175474527 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0fa80c8660 msgr2=0x7f0fa80c6080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:44:46.225 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:46.217+0000 7f0fadf5e640 1 --2- 172.21.15.16:0/4175474527 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0fa80c8660 0x7f0fa80c6080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:46.226 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:46.217+0000 7f0fadf5e640 1 -- 172.21.15.16:0/4175474527 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f9800a5c0 con 0x7f0fa815bcd0 2023-05-22T15:44:46.226 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:46.217+0000 7f0facf5c640 1 --2- 172.21.15.16:0/4175474527 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0fa8158210 0x7f0fa8151330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:44:46.226 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:46.217+0000 7f0fadf5e640 1 --2- 172.21.15.16:0/4175474527 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0fa815bcd0 0x7f0fa80c72f0 secure :-1 s=READY pgs=1519 cs=0 l=1 rev1=1 crypto rx=0x7f0fa000c780 tx=0x7f0fa000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:44:46.226 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:46.217+0000 7f0f9dffb640 1 -- 172.21.15.16:0/4175474527 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fa000ce90 con 0x7f0fa815bcd0 2023-05-22T15:44:46.226 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:46.217+0000 7f0f9dffb640 1 -- 172.21.15.16:0/4175474527 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fa000a640 con 0x7f0fa815bcd0 2023-05-22T15:44:46.226 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:46.217+0000 7f0faf9c1640 1 -- 172.21.15.16:0/4175474527 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0fa80c8eb0 con 0x7f0fa815bcd0 2023-05-22T15:44:46.226 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:46.217+0000 7f0f9dffb640 1 -- 172.21.15.16:0/4175474527 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fa000a9d0 con 0x7f0fa815bcd0 2023-05-22T15:44:46.226 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:46.217+0000 7f0faf9c1640 1 -- 172.21.15.16:0/4175474527 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0fa81585e0 con 0x7f0fa815bcd0 2023-05-22T15:44:46.227 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:46.217+0000 7f0faf9c1640 1 -- 172.21.15.16:0/4175474527 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0fa80cb1b0 con 0x7f0fa815bcd0 2023-05-22T15:44:46.227 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:46.221+0000 7f0f9dffb640 1 -- 172.21.15.16:0/4175474527 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 46) v1 ==== 84463+0+0 (secure 0 0 0) 0x7f0fa000ab30 con 0x7f0fa815bcd0 2023-05-22T15:44:46.228 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:46.221+0000 7f0f9dffb640 1 --2- 172.21.15.16:0/4175474527 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f0f8c065640 0x7f0f8c067af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:46.228 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:46.221+0000 7f0f9dffb640 1 -- 172.21.15.16:0/4175474527 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(119..119 src has 1..119) v4 ==== 6726+0+0 (secure 0 0 0) 0x7f0fa00823c0 con 0x7f0fa815bcd0 2023-05-22T15:44:46.228 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:46.221+0000 7f0fad75d640 1 --2- 172.21.15.16:0/4175474527 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f0f8c065640 0x7f0f8c067af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:44:46.228 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:46.221+0000 7f0fad75d640 1 --2- 172.21.15.16:0/4175474527 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f0f8c065640 0x7f0f8c067af0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f0fa4006fd0 tx=0x7f0fa4008040 comp rx=0 tx=0).ready entity=mgr.6470 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:44:46.229 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:46.225+0000 7f0f9dffb640 1 -- 172.21.15.16:0/4175474527 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f0fa004fa40 con 0x7f0fa815bcd0 2023-05-22T15:44:46.418 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:46.413+0000 7f0faf9c1640 1 -- 172.21.15.16:0/4175474527 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7f0fa80cb5c0 con 0x7f0fa815bcd0 2023-05-22T15:44:46.539 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:46.537+0000 7f0f9dffb640 1 -- 172.21.15.16:0/4175474527 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mgrmap(e 47) v1 ==== 84446+0+0 (secure 0 0 0) 0x7f0fa00820d0 con 0x7f0fa815bcd0 2023-05-22T15:44:46.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:46.541+0000 7f0f9dffb640 1 -- 172.21.15.16:0/4175474527 <== mon.0 v2:172.21.15.16:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v47) v1 ==== 93+0+0 (secure 0 0 0) 0x7f0fa004fa40 con 0x7f0fa815bcd0 2023-05-22T15:44:46.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:46.541+0000 7f0faf9c1640 1 -- 172.21.15.16:0/4175474527 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f0f8c065640 msgr2=0x7f0f8c067af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:44:46.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:46.541+0000 7f0faf9c1640 1 --2- 172.21.15.16:0/4175474527 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f0f8c065640 0x7f0f8c067af0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f0fa4006fd0 tx=0x7f0fa4008040 comp rx=0 tx=0).stop 2023-05-22T15:44:46.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:46.541+0000 7f0faf9c1640 1 -- 172.21.15.16:0/4175474527 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0fa815bcd0 msgr2=0x7f0fa80c72f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:44:46.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:46.541+0000 7f0faf9c1640 1 --2- 172.21.15.16:0/4175474527 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0fa815bcd0 0x7f0fa80c72f0 secure :-1 s=READY pgs=1519 cs=0 l=1 rev1=1 crypto rx=0x7f0fa000c780 tx=0x7f0fa000cc40 comp rx=0 tx=0).stop 2023-05-22T15:44:46.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:46.541+0000 7f0faf9c1640 1 -- 172.21.15.16:0/4175474527 shutdown_connections 2023-05-22T15:44:46.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:46.541+0000 7f0faf9c1640 1 --2- 172.21.15.16:0/4175474527 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0fa815bcd0 0x7f0fa80c72f0 unknown :-1 s=CLOSED pgs=1519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:46.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:46.541+0000 7f0faf9c1640 1 --2- 172.21.15.16:0/4175474527 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f0f8c065640 0x7f0f8c067af0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:46.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:46.541+0000 7f0faf9c1640 1 --2- 172.21.15.16:0/4175474527 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0fa8158210 0x7f0fa8151330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:46.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:46.541+0000 7f0faf9c1640 1 --2- 172.21.15.16:0/4175474527 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0fa80c8660 0x7f0fa80c6080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:46.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:46.541+0000 7f0faf9c1640 1 -- 172.21.15.16:0/4175474527 >> 172.21.15.16:0/4175474527 conn(0x7f0fa8069720 msgr2=0x7f0fa8155540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:44:46.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:46.541+0000 7f0faf9c1640 1 -- 172.21.15.16:0/4175474527 shutdown_connections 2023-05-22T15:44:46.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:46.541+0000 7f0faf9c1640 1 -- 172.21.15.16:0/4175474527 wait complete. 2023-05-22T15:44:46.566 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-05-22T15:44:46.567 DEBUG:teuthology.orchestra.run.smithi016:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-05-22T15:44:46.600 DEBUG:teuthology.orchestra.run.smithi062:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-05-22T15:44:46.641 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-05-22T15:44:46.642 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-05-22T15:44:46.642 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2023-05-22T15:44:46.647 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-05-22T15:44:46.670 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:44:46.666+0000 7f8f9a67d180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-05-22T15:44:46.674 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:44:46.673+0000 7f4d405c0180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-05-22T15:44:46.797 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:46.793+0000 7f2de976c640 1 Processor -- start 2023-05-22T15:44:46.798 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:46.793+0000 7f2de976c640 1 -- start start 2023-05-22T15:44:46.800 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:46.793+0000 7f2de976c640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2de4159cd0 0x7f2de415a0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:46.800 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:46.793+0000 7f2de976c640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2de40b2880 0x7f2de40cbfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:46.800 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:46.793+0000 7f2de976c640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f2de415a5e0 0x7f2de40cc520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:46.801 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:46.793+0000 7f2de976c640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2de40cf9a0 con 0x7f2de4159cd0 2023-05-22T15:44:46.801 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:46.793+0000 7f2de976c640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2de40cca60 con 0x7f2de415a5e0 2023-05-22T15:44:46.801 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:46.793+0000 7f2de976c640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2de415aba0 con 0x7f2de40b2880 2023-05-22T15:44:46.801 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:46.793+0000 7f2de2ffd640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2de4159cd0 0x7f2de415a0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:44:46.801 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:46.793+0000 7f2de2ffd640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2de4159cd0 0x7f2de415a0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35608/0 (socket says 172.21.15.16:35608) 2023-05-22T15:44:46.801 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:46.793+0000 7f2de2ffd640 1 -- 172.21.15.16:0/3424854853 learned_addr learned my addr 172.21.15.16:0/3424854853 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:44:46.801 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:46.797+0000 7f2de27fc640 1 --2- 172.21.15.16:0/3424854853 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2de40b2880 0x7f2de40cbfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:44:46.802 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:46.797+0000 7f2de37fe640 1 --2- 172.21.15.16:0/3424854853 >> v2:172.21.15.62:3300/0 conn(0x7f2de415a5e0 0x7f2de40cc520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:44:46.802 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:46.797+0000 7f2de2ffd640 1 -- 172.21.15.16:0/3424854853 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2de40b2880 msgr2=0x7f2de40cbfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:44:46.802 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:46.797+0000 7f2de2ffd640 1 --2- 172.21.15.16:0/3424854853 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2de40b2880 0x7f2de40cbfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:46.802 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:46.797+0000 7f2de2ffd640 1 -- 172.21.15.16:0/3424854853 >> v2:172.21.15.62:3300/0 conn(0x7f2de415a5e0 msgr2=0x7f2de40cc520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:44:46.802 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:46.797+0000 7f2de2ffd640 1 --2- 172.21.15.16:0/3424854853 >> v2:172.21.15.62:3300/0 conn(0x7f2de415a5e0 0x7f2de40cc520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:46.802 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:46.797+0000 7f2de2ffd640 1 -- 172.21.15.16:0/3424854853 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2de415ae80 con 0x7f2de4159cd0 2023-05-22T15:44:46.804 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:46.797+0000 7f2de2ffd640 1 --2- 172.21.15.16:0/3424854853 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2de4159cd0 0x7f2de415a0a0 secure :-1 s=READY pgs=1524 cs=0 l=1 rev1=1 crypto rx=0x7f2de4070360 tx=0x7f2dd401c9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5736efc6529542f0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:44:46.805 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:46.797+0000 7f2de1ffb640 1 -- 172.21.15.16:0/3424854853 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dd40a5730 con 0x7f2de4159cd0 2023-05-22T15:44:46.805 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:46.797+0000 7f2de1ffb640 1 -- 172.21.15.16:0/3424854853 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2dd40a5890 con 0x7f2de4159cd0 2023-05-22T15:44:46.805 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:46.797+0000 7f2de1ffb640 1 -- 172.21.15.16:0/3424854853 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dd40a5c20 con 0x7f2de4159cd0 2023-05-22T15:44:46.805 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:46.797+0000 7f2de976c640 1 -- 172.21.15.16:0/3424854853 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2de4159cd0 msgr2=0x7f2de415a0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:44:46.805 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:46.797+0000 7f2de976c640 1 --2- 172.21.15.16:0/3424854853 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2de4159cd0 0x7f2de415a0a0 secure :-1 s=READY pgs=1524 cs=0 l=1 rev1=1 crypto rx=0x7f2de4070360 tx=0x7f2dd401c9a0 comp rx=0 tx=0).stop 2023-05-22T15:44:46.805 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:46.797+0000 7f2de976c640 1 -- 172.21.15.16:0/3424854853 shutdown_connections 2023-05-22T15:44:46.805 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:46.797+0000 7f2de976c640 1 --2- 172.21.15.16:0/3424854853 >> v2:172.21.15.62:3300/0 conn(0x7f2de415a5e0 0x7f2de40cc520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:46.805 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:46.797+0000 7f2de976c640 1 --2- 172.21.15.16:0/3424854853 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2de40b2880 0x7f2de40cbfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:46.806 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:46.797+0000 7f2de976c640 1 --2- 172.21.15.16:0/3424854853 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2de4159cd0 0x7f2de415a0a0 unknown :-1 s=CLOSED pgs=1524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:46.806 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:46.797+0000 7f2de976c640 1 -- 172.21.15.16:0/3424854853 >> 172.21.15.16:0/3424854853 conn(0x7f2de406f5c0 msgr2=0x7f2de406f9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:44:46.806 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:46.797+0000 7f2de976c640 1 -- 172.21.15.16:0/3424854853 shutdown_connections 2023-05-22T15:44:46.806 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:46.797+0000 7f2de976c640 1 -- 172.21.15.16:0/3424854853 wait complete. 2023-05-22T15:44:46.806 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:46.801+0000 7f2de976c640 1 Processor -- start 2023-05-22T15:44:46.806 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:46.801+0000 7f2de976c640 1 -- start start 2023-05-22T15:44:46.806 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:46.801+0000 7f2de976c640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2de40b2880 0x7f2de40c3ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:46.807 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:46.801+0000 7f2de976c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2de4159cd0 0x7f2de4154330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:46.807 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:46.801+0000 7f2de976c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2de415a5e0 0x7f2de414d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:46.807 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:46.801+0000 7f2de976c640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2de40c4420 con 0x7f2de415a5e0 2023-05-22T15:44:46.807 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:46.801+0000 7f2de976c640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2de41573b0 con 0x7f2de4159cd0 2023-05-22T15:44:46.807 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:46.801+0000 7f2de976c640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2de40c7610 con 0x7f2de40b2880 2023-05-22T15:44:46.808 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:46.801+0000 7f2de37fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2de415a5e0 0x7f2de414d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:44:46.808 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:46.801+0000 7f2de37fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2de415a5e0 0x7f2de414d410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35624/0 (socket says 172.21.15.16:35624) 2023-05-22T15:44:46.808 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:46.801+0000 7f2de37fe640 1 -- 172.21.15.16:0/2921929862 learned_addr learned my addr 172.21.15.16:0/2921929862 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:44:46.808 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:46.801+0000 7f2de27fc640 1 --2- 172.21.15.16:0/2921929862 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2de4159cd0 0x7f2de4154330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:44:46.808 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:46.801+0000 7f2de2ffd640 1 --2- 172.21.15.16:0/2921929862 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2de40b2880 0x7f2de40c3ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:44:46.809 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:46.801+0000 7f2de37fe640 1 -- 172.21.15.16:0/2921929862 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2de40b2880 msgr2=0x7f2de40c3ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:44:46.809 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:46.801+0000 7f2de37fe640 1 --2- 172.21.15.16:0/2921929862 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2de40b2880 0x7f2de40c3ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:46.809 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:46.801+0000 7f2de37fe640 1 -- 172.21.15.16:0/2921929862 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2de4159cd0 msgr2=0x7f2de4154330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:44:46.809 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:46.801+0000 7f2de37fe640 1 --2- 172.21.15.16:0/2921929862 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2de4159cd0 0x7f2de4154330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:46.809 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:46.801+0000 7f2de37fe640 1 -- 172.21.15.16:0/2921929862 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dd4009b30 con 0x7f2de415a5e0 2023-05-22T15:44:46.809 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:46.801+0000 7f2de27fc640 1 --2- 172.21.15.16:0/2921929862 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2de4159cd0 0x7f2de4154330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T15:44:46.809 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:46.801+0000 7f2de2ffd640 1 --2- 172.21.15.16:0/2921929862 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2de40b2880 0x7f2de40c3ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T15:44:46.810 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:46.801+0000 7f2de37fe640 1 --2- 172.21.15.16:0/2921929862 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2de415a5e0 0x7f2de414d410 secure :-1 s=READY pgs=1525 cs=0 l=1 rev1=1 crypto rx=0x7f2ddc00d590 tx=0x7f2ddc00da50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:44:46.810 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:46.801+0000 7f2ddb7fe640 1 -- 172.21.15.16:0/2921929862 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ddc0125f0 con 0x7f2de415a5e0 2023-05-22T15:44:46.810 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:46.801+0000 7f2ddb7fe640 1 -- 172.21.15.16:0/2921929862 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ddc012770 con 0x7f2de415a5e0 2023-05-22T15:44:46.810 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:46.801+0000 7f2ddb7fe640 1 -- 172.21.15.16:0/2921929862 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ddc012b20 con 0x7f2de415a5e0 2023-05-22T15:44:46.811 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:46.801+0000 7f2de976c640 1 -- 172.21.15.16:0/2921929862 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2de40ceb00 con 0x7f2de415a5e0 2023-05-22T15:44:46.811 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:46.801+0000 7f2de976c640 1 -- 172.21.15.16:0/2921929862 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2de40ce970 con 0x7f2de415a5e0 2023-05-22T15:44:46.811 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:46.805+0000 7f2de976c640 1 -- 172.21.15.16:0/2921929862 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2de40cec90 con 0x7f2de415a5e0 2023-05-22T15:44:46.811 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:46.805+0000 7f2ddb7fe640 1 -- 172.21.15.16:0/2921929862 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 47) v1 ==== 84446+0+0 (secure 0 0 0) 0x7f2ddc012c80 con 0x7f2de415a5e0 2023-05-22T15:44:46.812 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:46.805+0000 7f2ddb7fe640 1 --2- 172.21.15.16:0/2921929862 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f2dc40655c0 0x7f2dc4067a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:46.812 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:46.805+0000 7f2de2ffd640 1 -- 172.21.15.16:0/2921929862 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f2dc40655c0 msgr2=0x7f2dc4067a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/36921 2023-05-22T15:44:46.812 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:46.805+0000 7f2ddb7fe640 1 -- 172.21.15.16:0/2921929862 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(119..119 src has 1..119) v4 ==== 6726+0+0 (secure 0 0 0) 0x7f2ddc083320 con 0x7f2de415a5e0 2023-05-22T15:44:46.812 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:46.805+0000 7f2de2ffd640 1 --2- 172.21.15.16:0/2921929862 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f2dc40655c0 0x7f2dc4067a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-05-22T15:44:46.812 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:46.809+0000 7f2ddb7fe640 1 -- 172.21.15.16:0/2921929862 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f2ddc050a30 con 0x7f2de415a5e0 2023-05-22T15:44:46.965 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:46.961+0000 7f2de976c640 1 -- 172.21.15.16:0/2921929862 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2de404dbe0 con 0x7f2de415a5e0 2023-05-22T15:44:46.968 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:46.965+0000 7f2ddb7fe640 1 -- 172.21.15.16:0/2921929862 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v119) v1 ==== 74+0+18480 (secure 0 0 0) 0x7f2ddc054050 con 0x7f2de415a5e0 2023-05-22T15:44:46.968 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:44:46.969 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":119,"fsid":"b2719834-7847-4d98-90a6-f65d57f85c49","created":"2023-05-22T15:26:25.259314+0000","modified":"2023-05-22T15:43:43.641801+0000","last_up_change":"2023-05-22T15:26:36.448524+0000","last_in_change":"2023-05-22T15:26:27.675668+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-05-22T15:26:38.019781+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-05-22T15:26:43.570514+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2023-05-22T15:43:28.684367+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"117","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2023-05-22T15:43:29.119294+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"118","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9e7ff854-b25f-4cbb-a5c2-1799d9c53d2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":37021},{"type":"v1","addr":"172.21.15.16:6825","nonce":37021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":37021},{"type":"v1","addr":"172.21.15.16:6827","nonce":37021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":37021},{"type":"v1","addr":"172.21.15.16:6831","nonce":37021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":37021},{"type":"v1","addr":"172.21.15.16:6829","nonce":37021}]},"public_addr":"172.21.15.16:6825/37021","cluster_addr":"172.21.15.16:6827/37021","heartbeat_back_addr":"172.21.15.16:6831/37021","heartbeat_front_addr":"172.21.15.16:6829/37021","state":["exists","up"]},{"osd":1,"uuid":"4702fbc9-a927-4565-aca5-41dc69cc8264","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":37008},{"type":"v1","addr":"172.21.15.16:6801","nonce":37008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":37008},{"type":"v1","addr":"172.21.15.16:6803","nonce":37008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":37008},{"type":"v1","addr":"172.21.15.16:6807","nonce":37008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":37008},{"type":"v1","addr":"172.21.15.16:6805","nonce":37008}]},"public_addr":"172.21.15.16:6801/37008","cluster_addr":"172.21.15.16:6803/37008","heartbeat_back_addr":"172.21.15.16:6807/37008","heartbeat_front_addr":"172.21.15.16:6805/37008","state":["exists","up"]},{"osd":2,"uuid":"1e7bba46-4eba-47ef-bb81-5e2d371638ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":37009},{"type":"v1","addr":"172.21.15.16:6809","nonce":37009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":37009},{"type":"v1","addr":"172.21.15.16:6811","nonce":37009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":37009},{"type":"v1","addr":"172.21.15.16:6815","nonce":37009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":37009},{"type":"v1","addr":"172.21.15.16:6813","nonce":37009}]},"public_addr":"172.21.15.16:6809/37009","cluster_addr":"172.21.15.16:6811/37009","heartbeat_back_addr":"172.21.15.16:6815/37009","heartbeat_front_addr":"172.21.15.16:6813/37009","state":["exists","up"]},{"osd":3,"uuid":"383cacc0-5a88-4904-990e-498ca59ef48a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":37020},{"type":"v1","addr":"172.21.15.16:6817","nonce":37020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":37020},{"type":"v1","addr":"172.21.15.16:6819","nonce":37020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":37020},{"type":"v1","addr":"172.21.15.16:6823","nonce":37020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":37020},{"type":"v1","addr":"172.21.15.16:6821","nonce":37020}]},"public_addr":"172.21.15.16:6817/37020","cluster_addr":"172.21.15.16:6819/37020","heartbeat_back_addr":"172.21.15.16:6823/37020","heartbeat_front_addr":"172.21.15.16:6821/37020","state":["exists","up"]},{"osd":4,"uuid":"970500ef-95b7-42c1-b080-eed4fae7013d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":37353},{"type":"v1","addr":"172.21.15.62:6817","nonce":37353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6819","nonce":37353},{"type":"v1","addr":"172.21.15.62:6822","nonce":37353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37353},{"type":"v1","addr":"172.21.15.62:6831","nonce":37353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37353},{"type":"v1","addr":"172.21.15.62:6826","nonce":37353}]},"public_addr":"172.21.15.62:6817/37353","cluster_addr":"172.21.15.62:6822/37353","heartbeat_back_addr":"172.21.15.62:6831/37353","heartbeat_front_addr":"172.21.15.62:6826/37353","state":["exists","up"]},{"osd":5,"uuid":"022d67db-d4d8-445b-b1b9-8b446a00f5c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":37352},{"type":"v1","addr":"172.21.15.62:6801","nonce":37352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":37352},{"type":"v1","addr":"172.21.15.62:6803","nonce":37352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":37352},{"type":"v1","addr":"172.21.15.62:6807","nonce":37352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":37352},{"type":"v1","addr":"172.21.15.62:6805","nonce":37352}]},"public_addr":"172.21.15.62:6801/37352","cluster_addr":"172.21.15.62:6803/37352","heartbeat_back_addr":"172.21.15.62:6807/37352","heartbeat_front_addr":"172.21.15.62:6805/37352","state":["exists","up"]},{"osd":6,"uuid":"42bd372b-6688-4a1b-8407-e16e8c266e33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":37354},{"type":"v1","addr":"172.21.15.62:6820","nonce":37354}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":37354},{"type":"v1","addr":"172.21.15.62:6823","nonce":37354}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6829","nonce":37354},{"type":"v1","addr":"172.21.15.62:6830","nonce":37354}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6825","nonce":37354},{"type":"v1","addr":"172.21.15.62:6827","nonce":37354}]},"public_addr":"172.21.15.62:6820/37354","cluster_addr":"172.21.15.62:6823/37354","heartbeat_back_addr":"172.21.15.62:6830/37354","heartbeat_front_addr":"172.21.15.62:6827/37354","state":["exists","up"]},{"osd":7,"uuid":"ac70b2a3-f41b-4bf2-b227-7f2cbb55426d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37355},{"type":"v1","addr":"172.21.15.62:6809","nonce":37355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37355},{"type":"v1","addr":"172.21.15.62:6811","nonce":37355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":37355},{"type":"v1","addr":"172.21.15.62:6815","nonce":37355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":37355},{"type":"v1","addr":"172.21.15.62:6813","nonce":37355}]},"public_addr":"172.21.15.62:6809/37355","cluster_addr":"172.21.15.62:6811/37355","heartbeat_back_addr":"172.21.15.62:6815/37355","heartbeat_front_addr":"172.21.15.62:6813/37355","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.368896+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.107273+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.276888+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.287502+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.124791+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.251937+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.255679+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.232797+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:0/2218849021":"2023-05-23T15:43:43.641715+0000","172.21.15.16:0/2339322867":"2023-05-23T15:43:43.641715+0000","172.21.15.16:0/155043049":"2023-05-23T15:43:43.641715+0000","172.21.15.16:0/4168497577":"2023-05-23T15:43:43.641715+0000","172.21.15.16:0/3302517394":"2023-05-23T15:43:43.641715+0000","172.21.15.16:0/4100157757":"2023-05-23T15:43:43.641715+0000","172.21.15.16:6833/36921":"2023-05-23T15:43:43.641715+0000","172.21.15.16:6832/36921":"2023-05-23T15:43:43.641715+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-05-22T15:44:46.970 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:46.965+0000 7f2de976c640 1 -- 172.21.15.16:0/2921929862 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f2dc40655c0 msgr2=0x7f2dc4067a70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-05-22T15:44:46.971 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:46.965+0000 7f2de976c640 1 --2- 172.21.15.16:0/2921929862 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f2dc40655c0 0x7f2dc4067a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:46.971 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:46.969+0000 7f2de976c640 1 -- 172.21.15.16:0/2921929862 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2de415a5e0 msgr2=0x7f2de414d410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:44:46.971 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:46.969+0000 7f2de976c640 1 --2- 172.21.15.16:0/2921929862 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2de415a5e0 0x7f2de414d410 secure :-1 s=READY pgs=1525 cs=0 l=1 rev1=1 crypto rx=0x7f2ddc00d590 tx=0x7f2ddc00da50 comp rx=0 tx=0).stop 2023-05-22T15:44:46.971 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:46.969+0000 7f2de976c640 1 -- 172.21.15.16:0/2921929862 shutdown_connections 2023-05-22T15:44:46.971 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:46.969+0000 7f2de976c640 1 --2- 172.21.15.16:0/2921929862 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2de415a5e0 0x7f2de414d410 unknown :-1 s=CLOSED pgs=1525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:46.971 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:46.969+0000 7f2de976c640 1 --2- 172.21.15.16:0/2921929862 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2de4159cd0 0x7f2de4154330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:46.971 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:46.969+0000 7f2de976c640 1 --2- 172.21.15.16:0/2921929862 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f2dc40655c0 0x7f2dc4067a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:46.972 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:46.969+0000 7f2de976c640 1 --2- 172.21.15.16:0/2921929862 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2de40b2880 0x7f2de40c3ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:46.972 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:46.969+0000 7f2de976c640 1 -- 172.21.15.16:0/2921929862 >> 172.21.15.16:0/2921929862 conn(0x7f2de406f5c0 msgr2=0x7f2de4158e00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:44:46.972 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:46.969+0000 7f2de976c640 1 -- 172.21.15.16:0/2921929862 shutdown_connections 2023-05-22T15:44:46.972 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:46.969+0000 7f2de976c640 1 -- 172.21.15.16:0/2921929862 wait complete. 2023-05-22T15:44:46.988 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-05-22T15:26:38.019781+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-05-22T15:26:43.570514+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 14, 'pool_name': 'cephfs_metadata', 'create_time': '2023-05-22T15:43:28.684367+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '117', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 15, 'pool_name': 'cephfs_data', 'create_time': '2023-05-22T15:43:29.119294+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '118', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-05-22T15:44:46.990 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-05-22T15:44:47.061 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:44:47.058+0000 7f8f9a67d180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-05-22T15:44:47.072 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:44:47.069+0000 7f4d405c0180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-05-22T15:44:47.111 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.105+0000 7f48f8447640 1 Processor -- start 2023-05-22T15:44:47.111 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.109+0000 7f48f8447640 1 -- start start 2023-05-22T15:44:47.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.109+0000 7f48f8447640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f48f0071640 0x7f48f0160090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:47.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.109+0000 7f48f8447640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f48f01605d0 0x7f48f015c270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:47.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.109+0000 7f48f8447640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48f0069ae0 con 0x7f48f00ce480 2023-05-22T15:44:47.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.109+0000 7f48f8447640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48f015f990 con 0x7f48f01605d0 2023-05-22T15:44:47.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.109+0000 7f48f8447640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48f0071a90 con 0x7f48f0071640 2023-05-22T15:44:47.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.109+0000 7f48f61e3640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f48f00ce480 0x7f48f00ce850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:47700/0 (socket says 172.21.15.16:47700) 2023-05-22T15:44:47.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.109+0000 7f48f61e3640 1 -- 172.21.15.16:0/3367800622 learned_addr learned my addr 172.21.15.16:0/3367800622 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:44:47.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.109+0000 7f48f69e4640 1 --2- 172.21.15.16:0/3367800622 >> v2:172.21.15.16:3300/0 conn(0x7f48f01605d0 0x7f48f015c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:44:47.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.109+0000 7f48f59e2640 1 --2- 172.21.15.16:0/3367800622 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f48f0071640 0x7f48f0160090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:44:47.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.109+0000 7f48f69e4640 1 -- 172.21.15.16:0/3367800622 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f48f0071640 msgr2=0x7f48f0160090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:44:47.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.109+0000 7f48f69e4640 1 --2- 172.21.15.16:0/3367800622 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f48f0071640 0x7f48f0160090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:47.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.109+0000 7f48f69e4640 1 -- 172.21.15.16:0/3367800622 >> v1:172.21.15.16:6789/0 conn(0x7f48f00ce480 legacy=0x7f48f00ce850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:44:47.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.109+0000 7f48f51e1640 1 -- 172.21.15.16:0/3367800622 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 105847287 0 0) 0x7f48f0069ae0 con 0x7f48f00ce480 2023-05-22T15:44:47.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.109+0000 7f48f69e4640 1 -- 172.21.15.16:0/3367800622 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48f0069dc0 con 0x7f48f01605d0 2023-05-22T15:44:47.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.109+0000 7f48f69e4640 1 --2- 172.21.15.16:0/3367800622 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f48f01605d0 0x7f48f015c270 secure :-1 s=READY pgs=1527 cs=0 l=1 rev1=1 crypto rx=0x7f48e80097c0 tx=0x7f48e801c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=afe6b2f7e647654f server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:44:47.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.109+0000 7f48f51e1640 1 -- 172.21.15.16:0/3367800622 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48e80ab650 con 0x7f48f01605d0 2023-05-22T15:44:47.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.109+0000 7f48f51e1640 1 -- 172.21.15.16:0/3367800622 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48e80ab7b0 con 0x7f48f01605d0 2023-05-22T15:44:47.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.109+0000 7f48f51e1640 1 -- 172.21.15.16:0/3367800622 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48e80abb40 con 0x7f48f01605d0 2023-05-22T15:44:47.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.113+0000 7f48f8447640 1 -- 172.21.15.16:0/3367800622 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f48f01605d0 msgr2=0x7f48f015c270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:44:47.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.113+0000 7f48f8447640 1 --2- 172.21.15.16:0/3367800622 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f48f01605d0 0x7f48f015c270 secure :-1 s=READY pgs=1527 cs=0 l=1 rev1=1 crypto rx=0x7f48e80097c0 tx=0x7f48e801c450 comp rx=0 tx=0).stop 2023-05-22T15:44:47.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.113+0000 7f48f8447640 1 -- 172.21.15.16:0/3367800622 shutdown_connections 2023-05-22T15:44:47.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.113+0000 7f48f8447640 1 --2- 172.21.15.16:0/3367800622 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f48f01605d0 0x7f48f015c270 unknown :-1 s=CLOSED pgs=1527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:47.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.113+0000 7f48f8447640 1 --2- 172.21.15.16:0/3367800622 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f48f0071640 0x7f48f0160090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:47.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.113+0000 7f48f8447640 1 -- 172.21.15.16:0/3367800622 >> 172.21.15.16:0/3367800622 conn(0x7f48f00cc870 msgr2=0x7f48f00cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:44:47.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.113+0000 7f48f8447640 1 -- 172.21.15.16:0/3367800622 shutdown_connections 2023-05-22T15:44:47.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.113+0000 7f48f8447640 1 -- 172.21.15.16:0/3367800622 wait complete. 2023-05-22T15:44:47.118 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.113+0000 7f48f8447640 1 Processor -- start 2023-05-22T15:44:47.119 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.113+0000 7f48f8447640 1 -- start start 2023-05-22T15:44:47.119 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.113+0000 7f48f8447640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f48f0071640 0x7f48f0154fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:47.119 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.113+0000 7f48f8447640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f48f00ce480 0x7f48f015d580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:47.119 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.113+0000 7f48f8447640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f48f01605d0 0x7f48f015dac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:47.119 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.113+0000 7f48f8447640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48f00ab870 con 0x7f48f01605d0 2023-05-22T15:44:47.119 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.113+0000 7f48f8447640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48f00abb10 con 0x7f48f0071640 2023-05-22T15:44:47.119 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.113+0000 7f48f8447640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48f00abdf0 con 0x7f48f00ce480 2023-05-22T15:44:47.119 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.113+0000 7f48f69e4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f48f01605d0 0x7f48f015dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:44:47.120 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.113+0000 7f48f69e4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f48f01605d0 0x7f48f015dac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35652/0 (socket says 172.21.15.16:35652) 2023-05-22T15:44:47.120 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.113+0000 7f48f69e4640 1 -- 172.21.15.16:0/3738475856 learned_addr learned my addr 172.21.15.16:0/3738475856 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:44:47.120 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.113+0000 7f48f61e3640 1 --2- 172.21.15.16:0/3738475856 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f48f0071640 0x7f48f0154fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:44:47.120 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.113+0000 7f48f59e2640 1 --2- 172.21.15.16:0/3738475856 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f48f00ce480 0x7f48f015d580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:44:47.120 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.113+0000 7f48f69e4640 1 -- 172.21.15.16:0/3738475856 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f48f00ce480 msgr2=0x7f48f015d580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:44:47.120 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.113+0000 7f48f69e4640 1 --2- 172.21.15.16:0/3738475856 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f48f00ce480 0x7f48f015d580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:47.121 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.113+0000 7f48f69e4640 1 -- 172.21.15.16:0/3738475856 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f48f0071640 msgr2=0x7f48f0154fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:44:47.121 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.113+0000 7f48f69e4640 1 --2- 172.21.15.16:0/3738475856 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f48f0071640 0x7f48f0154fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:47.121 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.113+0000 7f48f69e4640 1 -- 172.21.15.16:0/3738475856 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48e8009420 con 0x7f48f01605d0 2023-05-22T15:44:47.121 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.113+0000 7f48f69e4640 1 --2- 172.21.15.16:0/3738475856 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f48f01605d0 0x7f48f015dac0 secure :-1 s=READY pgs=1528 cs=0 l=1 rev1=1 crypto rx=0x7f48e80ad040 tx=0x7f48e801ac90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:44:47.121 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.113+0000 7f48e6ffd640 1 -- 172.21.15.16:0/3738475856 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48e80bf610 con 0x7f48f01605d0 2023-05-22T15:44:47.121 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.113+0000 7f48e6ffd640 1 -- 172.21.15.16:0/3738475856 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48e80bf770 con 0x7f48f01605d0 2023-05-22T15:44:47.121 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.113+0000 7f48f8447640 1 -- 172.21.15.16:0/3738475856 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48f00698b0 con 0x7f48f01605d0 2023-05-22T15:44:47.122 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.113+0000 7f48e6ffd640 1 -- 172.21.15.16:0/3738475856 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48e80bfb00 con 0x7f48f01605d0 2023-05-22T15:44:47.122 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.113+0000 7f48f8447640 1 -- 172.21.15.16:0/3738475856 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48f0069720 con 0x7f48f01605d0 2023-05-22T15:44:47.122 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.117+0000 7f48f8447640 1 -- 172.21.15.16:0/3738475856 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48f00c5e70 con 0x7f48f01605d0 2023-05-22T15:44:47.122 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.117+0000 7f48e6ffd640 1 -- 172.21.15.16:0/3738475856 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 47) v1 ==== 84446+0+0 (secure 0 0 0) 0x7f48e80b5400 con 0x7f48f01605d0 2023-05-22T15:44:47.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.117+0000 7f48e6ffd640 1 --2- 172.21.15.16:0/3738475856 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f48d40655c0 0x7f48d4067a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:47.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.117+0000 7f48f61e3640 1 -- 172.21.15.16:0/3738475856 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f48d40655c0 msgr2=0x7f48d4067a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/36921 2023-05-22T15:44:47.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.117+0000 7f48e6ffd640 1 -- 172.21.15.16:0/3738475856 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(119..119 src has 1..119) v4 ==== 6726+0+0 (secure 0 0 0) 0x7f48e811dac0 con 0x7f48f01605d0 2023-05-22T15:44:47.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.117+0000 7f48f61e3640 1 --2- 172.21.15.16:0/3738475856 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f48d40655c0 0x7f48d4067a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-05-22T15:44:47.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.121+0000 7f48e6ffd640 1 -- 172.21.15.16:0/3738475856 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f48e80eb060 con 0x7f48f01605d0 2023-05-22T15:44:47.199 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:44:47.194+0000 7f8f9a67d180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-05-22T15:44:47.212 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:44:47.209+0000 7f4d405c0180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-05-22T15:44:47.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.273+0000 7f48f8447640 1 -- 172.21.15.16:0/3738475856 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f48f00ac310 con 0x7f48f01605d0 2023-05-22T15:44:47.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.273+0000 7f48e6ffd640 1 -- 172.21.15.16:0/3738475856 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v119) v1 ==== 93+0+10 (secure 0 0 0) 0x7f48f00ac310 con 0x7f48f01605d0 2023-05-22T15:44:47.277 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2023-05-22T15:44:47.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.277+0000 7f48f8447640 1 -- 172.21.15.16:0/3738475856 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f48d40655c0 msgr2=0x7f48d4067a70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-05-22T15:44:47.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.277+0000 7f48f8447640 1 --2- 172.21.15.16:0/3738475856 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f48d40655c0 0x7f48d4067a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:47.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.277+0000 7f48f8447640 1 -- 172.21.15.16:0/3738475856 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f48f01605d0 msgr2=0x7f48f015dac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:44:47.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.277+0000 7f48f8447640 1 --2- 172.21.15.16:0/3738475856 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f48f01605d0 0x7f48f015dac0 secure :-1 s=READY pgs=1528 cs=0 l=1 rev1=1 crypto rx=0x7f48e80ad040 tx=0x7f48e801ac90 comp rx=0 tx=0).stop 2023-05-22T15:44:47.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.277+0000 7f48f8447640 1 -- 172.21.15.16:0/3738475856 shutdown_connections 2023-05-22T15:44:47.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.277+0000 7f48f8447640 1 --2- 172.21.15.16:0/3738475856 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f48f01605d0 0x7f48f015dac0 unknown :-1 s=CLOSED pgs=1528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:47.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.277+0000 7f48f8447640 1 --2- 172.21.15.16:0/3738475856 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f48d40655c0 0x7f48d4067a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:47.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.277+0000 7f48f8447640 1 --2- 172.21.15.16:0/3738475856 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f48f00ce480 0x7f48f015d580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:47.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.277+0000 7f48f8447640 1 --2- 172.21.15.16:0/3738475856 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f48f0071640 0x7f48f0154fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:47.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.277+0000 7f48f8447640 1 -- 172.21.15.16:0/3738475856 >> 172.21.15.16:0/3738475856 conn(0x7f48f00cc870 msgr2=0x7f48f00cd100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:44:47.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.277+0000 7f48f8447640 1 -- 172.21.15.16:0/3738475856 shutdown_connections 2023-05-22T15:44:47.282 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.277+0000 7f48f8447640 1 -- 172.21.15.16:0/3738475856 wait complete. 2023-05-22T15:44:47.296 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-05-22T15:44:47.420 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.413+0000 7f2f112b1640 1 Processor -- start 2023-05-22T15:44:47.420 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.413+0000 7f2f112b1640 1 -- start start 2023-05-22T15:44:47.420 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.417+0000 7f2f112b1640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2f0c0b2880 0x7f2f0c0cbfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:47.420 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.417+0000 7f2f112b1640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f2f0c0cc520 0x7f2f0c0c3db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:47.420 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.417+0000 7f2f112b1640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f0c0c6ba0 con 0x7f2f0c0c67d0 2023-05-22T15:44:47.420 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.417+0000 7f2f112b1640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f0c15a300 con 0x7f2f0c0cc520 2023-05-22T15:44:47.421 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.417+0000 7f2f112b1640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f0c0cfef0 con 0x7f2f0c0b2880 2023-05-22T15:44:47.421 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.417+0000 7f2f0a7fc640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2f0c0b2880 0x7f2f0c0cbfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:44:47.421 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.417+0000 7f2f0affd640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f2f0c0c67d0 0x7f2f0c0cf730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.16:55996/0 (socket says 172.21.15.16:55996) 2023-05-22T15:44:47.421 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.417+0000 7f2f0affd640 1 -- 172.21.15.16:0/1186472253 learned_addr learned my addr 172.21.15.16:0/1186472253 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:44:47.421 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.417+0000 7f2f0b7fe640 1 --2- 172.21.15.16:0/1186472253 >> v2:172.21.15.62:3300/0 conn(0x7f2f0c0cc520 0x7f2f0c0c3db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:44:47.421 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.417+0000 7f2f0a7fc640 1 -- 172.21.15.16:0/1186472253 >> v2:172.21.15.62:3300/0 conn(0x7f2f0c0cc520 msgr2=0x7f2f0c0c3db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:44:47.421 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.417+0000 7f2f0a7fc640 1 --2- 172.21.15.16:0/1186472253 >> v2:172.21.15.62:3300/0 conn(0x7f2f0c0cc520 0x7f2f0c0c3db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:47.422 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.417+0000 7f2f0a7fc640 1 -- 172.21.15.16:0/1186472253 >> v1:172.21.15.62:6789/0 conn(0x7f2f0c0c67d0 legacy=0x7f2f0c0cf730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:44:47.422 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.417+0000 7f2f0a7fc640 1 -- 172.21.15.16:0/1186472253 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f0c0d01d0 con 0x7f2f0c0b2880 2023-05-22T15:44:47.422 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.421+0000 7f2f0a7fc640 1 --2- 172.21.15.16:0/1186472253 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2f0c0b2880 0x7f2f0c0cbfe0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f2f000097c0 tx=0x7f2f0001c3c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=98fb05af2c7adbb2 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:44:47.423 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.421+0000 7f2f09ffb640 1 -- 172.21.15.16:0/1186472253 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f000ac620 con 0x7f2f0c0b2880 2023-05-22T15:44:47.423 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.421+0000 7f2f09ffb640 1 -- 172.21.15.16:0/1186472253 <== mon.2 v2:172.21.15.62:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f000ac780 con 0x7f2f0c0b2880 2023-05-22T15:44:47.423 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.421+0000 7f2f09ffb640 1 -- 172.21.15.16:0/1186472253 <== mon.2 v2:172.21.15.62:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f000acb10 con 0x7f2f0c0b2880 2023-05-22T15:44:47.424 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.421+0000 7f2f112b1640 1 -- 172.21.15.16:0/1186472253 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2f0c0b2880 msgr2=0x7f2f0c0cbfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:44:47.424 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.421+0000 7f2f112b1640 1 --2- 172.21.15.16:0/1186472253 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2f0c0b2880 0x7f2f0c0cbfe0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f2f000097c0 tx=0x7f2f0001c3c0 comp rx=0 tx=0).stop 2023-05-22T15:44:47.424 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.421+0000 7f2f112b1640 1 -- 172.21.15.16:0/1186472253 shutdown_connections 2023-05-22T15:44:47.424 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.421+0000 7f2f112b1640 1 --2- 172.21.15.16:0/1186472253 >> v2:172.21.15.62:3300/0 conn(0x7f2f0c0cc520 0x7f2f0c0c3db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:47.425 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.421+0000 7f2f112b1640 1 --2- 172.21.15.16:0/1186472253 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2f0c0b2880 0x7f2f0c0cbfe0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:47.425 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.421+0000 7f2f112b1640 1 -- 172.21.15.16:0/1186472253 >> 172.21.15.16:0/1186472253 conn(0x7f2f0c06f5a0 msgr2=0x7f2f0c06f9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:44:47.425 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.421+0000 7f2f112b1640 1 -- 172.21.15.16:0/1186472253 shutdown_connections 2023-05-22T15:44:47.425 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.421+0000 7f2f112b1640 1 -- 172.21.15.16:0/1186472253 wait complete. 2023-05-22T15:44:47.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.421+0000 7f2f112b1640 1 Processor -- start 2023-05-22T15:44:47.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.421+0000 7f2f112b1640 1 -- start start 2023-05-22T15:44:47.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.421+0000 7f2f112b1640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2f0c0b2880 0x7f2f0c0c7860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:47.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.421+0000 7f2f112b1640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f0c0c67d0 0x7f2f0c150af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:47.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.421+0000 7f2f112b1640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2f0c0cc520 0x7f2f0c0b3090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:47.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.421+0000 7f2f112b1640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f0c151f30 con 0x7f2f0c0c67d0 2023-05-22T15:44:47.432 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.421+0000 7f2f112b1640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f0c151030 con 0x7f2f0c0cc520 2023-05-22T15:44:47.432 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.421+0000 7f2f112b1640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f0c1512b0 con 0x7f2f0c0b2880 2023-05-22T15:44:47.432 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.421+0000 7f2f0a7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f0c0c67d0 0x7f2f0c150af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:44:47.432 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.421+0000 7f2f0a7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f0c0c67d0 0x7f2f0c150af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35654/0 (socket says 172.21.15.16:35654) 2023-05-22T15:44:47.432 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.421+0000 7f2f0a7fc640 1 -- 172.21.15.16:0/1582085595 learned_addr learned my addr 172.21.15.16:0/1582085595 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:44:47.432 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.421+0000 7f2f0affd640 1 --2- 172.21.15.16:0/1582085595 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2f0c0b2880 0x7f2f0c0c7860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:44:47.432 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.421+0000 7f2f0b7fe640 1 --2- 172.21.15.16:0/1582085595 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2f0c0cc520 0x7f2f0c0b3090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:44:47.432 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.425+0000 7f2f0a7fc640 1 -- 172.21.15.16:0/1582085595 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2f0c0b2880 msgr2=0x7f2f0c0c7860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:44:47.433 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.425+0000 7f2f0a7fc640 1 --2- 172.21.15.16:0/1582085595 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2f0c0b2880 0x7f2f0c0c7860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:47.433 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.425+0000 7f2f0a7fc640 1 -- 172.21.15.16:0/1582085595 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2f0c0cc520 msgr2=0x7f2f0c0b3090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:44:47.433 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.425+0000 7f2f0a7fc640 1 --2- 172.21.15.16:0/1582085595 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2f0c0cc520 0x7f2f0c0b3090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:47.433 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.425+0000 7f2f0a7fc640 1 -- 172.21.15.16:0/1582085595 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f00009420 con 0x7f2f0c0c67d0 2023-05-22T15:44:47.433 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.425+0000 7f2f0a7fc640 1 --2- 172.21.15.16:0/1582085595 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f0c0c67d0 0x7f2f0c150af0 secure :-1 s=READY pgs=1529 cs=0 l=1 rev1=1 crypto rx=0x7f2f000af040 tx=0x7f2f00019160 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:44:47.433 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.425+0000 7f2ef37fe640 1 -- 172.21.15.16:0/1582085595 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f000b5410 con 0x7f2f0c0c67d0 2023-05-22T15:44:47.433 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.425+0000 7f2ef37fe640 1 -- 172.21.15.16:0/1582085595 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f000b5590 con 0x7f2f0c0c67d0 2023-05-22T15:44:47.434 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.425+0000 7f2f112b1640 1 -- 172.21.15.16:0/1582085595 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f0c0d0f90 con 0x7f2f0c0c67d0 2023-05-22T15:44:47.434 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.425+0000 7f2ef37fe640 1 -- 172.21.15.16:0/1582085595 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f000b5940 con 0x7f2f0c0c67d0 2023-05-22T15:44:47.434 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.425+0000 7f2f112b1640 1 -- 172.21.15.16:0/1582085595 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f0c0d0a70 con 0x7f2f0c0c67d0 2023-05-22T15:44:47.434 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.425+0000 7f2f112b1640 1 -- 172.21.15.16:0/1582085595 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f0c0cc920 con 0x7f2f0c0c67d0 2023-05-22T15:44:47.434 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.425+0000 7f2ef37fe640 1 -- 172.21.15.16:0/1582085595 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 47) v1 ==== 84446+0+0 (secure 0 0 0) 0x7f2f000b3040 con 0x7f2f0c0c67d0 2023-05-22T15:44:47.434 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.425+0000 7f2ef37fe640 1 --2- 172.21.15.16:0/1582085595 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f2ef80655c0 0x7f2ef8067a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:47.434 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.425+0000 7f2f0affd640 1 -- 172.21.15.16:0/1582085595 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f2ef80655c0 msgr2=0x7f2ef8067a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/36921 2023-05-22T15:44:47.434 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.425+0000 7f2ef37fe640 1 -- 172.21.15.16:0/1582085595 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(119..119 src has 1..119) v4 ==== 6726+0+0 (secure 0 0 0) 0x7f2f0011d070 con 0x7f2f0c0c67d0 2023-05-22T15:44:47.435 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.425+0000 7f2f0affd640 1 --2- 172.21.15.16:0/1582085595 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f2ef80655c0 0x7f2ef8067a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-05-22T15:44:47.435 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.429+0000 7f2ef37fe640 1 -- 172.21.15.16:0/1582085595 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f2f0014d620 con 0x7f2f0c0c67d0 2023-05-22T15:44:47.584 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.581+0000 7f2f112b1640 1 -- 172.21.15.16:0/1582085595 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f2f0c152070 con 0x7f2f0c0c67d0 2023-05-22T15:44:47.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.581+0000 7f2ef37fe640 1 -- 172.21.15.16:0/1582085595 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v119) v1 ==== 92+0+10 (secure 0 0 0) 0x7f2f0c152070 con 0x7f2f0c0c67d0 2023-05-22T15:44:47.585 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 8 2023-05-22T15:44:47.587 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.585+0000 7f2f112b1640 1 -- 172.21.15.16:0/1582085595 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f2ef80655c0 msgr2=0x7f2ef8067a70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-05-22T15:44:47.587 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.585+0000 7f2f112b1640 1 --2- 172.21.15.16:0/1582085595 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f2ef80655c0 0x7f2ef8067a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:47.587 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.585+0000 7f2f112b1640 1 -- 172.21.15.16:0/1582085595 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f0c0c67d0 msgr2=0x7f2f0c150af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:44:47.587 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.585+0000 7f2f112b1640 1 --2- 172.21.15.16:0/1582085595 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f0c0c67d0 0x7f2f0c150af0 secure :-1 s=READY pgs=1529 cs=0 l=1 rev1=1 crypto rx=0x7f2f000af040 tx=0x7f2f00019160 comp rx=0 tx=0).stop 2023-05-22T15:44:47.587 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.585+0000 7f2f112b1640 1 -- 172.21.15.16:0/1582085595 shutdown_connections 2023-05-22T15:44:47.588 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.585+0000 7f2f112b1640 1 --2- 172.21.15.16:0/1582085595 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2f0c0cc520 0x7f2f0c0b3090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:47.588 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.585+0000 7f2f112b1640 1 --2- 172.21.15.16:0/1582085595 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f0c0c67d0 0x7f2f0c150af0 unknown :-1 s=CLOSED pgs=1529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:47.588 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.585+0000 7f2f112b1640 1 --2- 172.21.15.16:0/1582085595 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f2ef80655c0 0x7f2ef8067a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:47.588 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.585+0000 7f2f112b1640 1 --2- 172.21.15.16:0/1582085595 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2f0c0b2880 0x7f2f0c0c7860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:47.588 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.585+0000 7f2f112b1640 1 -- 172.21.15.16:0/1582085595 >> 172.21.15.16:0/1582085595 conn(0x7f2f0c06f5a0 msgr2=0x7f2f0c156530 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:44:47.588 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.585+0000 7f2f112b1640 1 -- 172.21.15.16:0/1582085595 shutdown_connections 2023-05-22T15:44:47.588 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.585+0000 7f2f112b1640 1 -- 172.21.15.16:0/1582085595 wait complete. 2023-05-22T15:44:47.606 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-05-22T15:44:47.638 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:44:47.634+0000 7f8f9a67d180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-05-22T15:44:47.665 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:44:47.661+0000 7f4d405c0180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-05-22T15:44:47.691 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:44:47.686+0000 7f8f9a67d180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-05-22T15:44:47.720 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:44:47.717+0000 7f4d405c0180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-05-22T15:44:47.738 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.733+0000 7ffa6f144640 1 Processor -- start 2023-05-22T15:44:47.739 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.737+0000 7ffa6f144640 1 -- start start 2023-05-22T15:44:47.739 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.737+0000 7ffa6f144640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7ffa680c67d0 0x7ffa680cf730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:47.740 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.737+0000 7ffa6f144640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ffa680b2880 0x7ffa680cbfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:47.740 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.737+0000 7ffa6f144640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa680cfc70 con 0x7ffa680cc520 2023-05-22T15:44:47.740 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.737+0000 7ffa6f144640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffa6815a300 con 0x7ffa680c67d0 2023-05-22T15:44:47.740 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.737+0000 7ffa6f144640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa680c4dc0 con 0x7ffa680b2880 2023-05-22T15:44:47.740 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.737+0000 7ffa6cee0640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7ffa680c67d0 0x7ffa680cf730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:44:47.740 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.737+0000 7ffa6cee0640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7ffa680c67d0 0x7ffa680cf730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.16:37134/0 (socket says 172.21.15.16:37134) 2023-05-22T15:44:47.741 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.737+0000 7ffa6cee0640 1 -- 172.21.15.16:0/1947896642 learned_addr learned my addr 172.21.15.16:0/1947896642 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:44:47.741 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.737+0000 7ffa67fff640 1 --2- 172.21.15.16:0/1947896642 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ffa680b2880 0x7ffa680cbfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:44:47.741 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.737+0000 7ffa6cee0640 1 -- 172.21.15.16:0/1947896642 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ffa680b2880 msgr2=0x7ffa680cbfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:44:47.741 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.737+0000 7ffa6cee0640 1 --2- 172.21.15.16:0/1947896642 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ffa680b2880 0x7ffa680cbfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:47.742 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.737+0000 7ffa6cee0640 1 -- 172.21.15.16:0/1947896642 >> v1:172.21.15.62:6789/0 conn(0x7ffa680cc520 legacy=0x7ffa680ce160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:44:47.742 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.737+0000 7ffa6cee0640 1 -- 172.21.15.16:0/1947896642 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa680c6ba0 con 0x7ffa680c67d0 2023-05-22T15:44:47.742 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:44:47.738+0000 7f8f9a67d180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-05-22T15:44:47.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.741+0000 7ffa6cee0640 1 --2- 172.21.15.16:0/1947896642 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffa680c67d0 0x7ffa680cf730 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7ffa60009e50 tx=0x7ffa6001c950 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f9f6ff94a632f29c server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:44:47.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.741+0000 7ffa677fe640 1 -- 172.21.15.16:0/1947896642 <== mon.1 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa600a5730 con 0x7ffa680c67d0 2023-05-22T15:44:47.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.741+0000 7ffa677fe640 1 -- 172.21.15.16:0/1947896642 <== mon.1 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa600a5890 con 0x7ffa680c67d0 2023-05-22T15:44:47.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.741+0000 7ffa677fe640 1 -- 172.21.15.16:0/1947896642 <== mon.1 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa600a5c20 con 0x7ffa680c67d0 2023-05-22T15:44:47.744 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.741+0000 7ffa6f144640 1 -- 172.21.15.16:0/1947896642 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffa680c67d0 msgr2=0x7ffa680cf730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:44:47.744 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.741+0000 7ffa6f144640 1 --2- 172.21.15.16:0/1947896642 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffa680c67d0 0x7ffa680cf730 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7ffa60009e50 tx=0x7ffa6001c950 comp rx=0 tx=0).stop 2023-05-22T15:44:47.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.741+0000 7ffa6f144640 1 -- 172.21.15.16:0/1947896642 shutdown_connections 2023-05-22T15:44:47.746 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.741+0000 7ffa6f144640 1 --2- 172.21.15.16:0/1947896642 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ffa680b2880 0x7ffa680cbfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:47.746 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.741+0000 7ffa6f144640 1 --2- 172.21.15.16:0/1947896642 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffa680c67d0 0x7ffa680cf730 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:47.746 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.741+0000 7ffa6f144640 1 -- 172.21.15.16:0/1947896642 >> 172.21.15.16:0/1947896642 conn(0x7ffa6806f5a0 msgr2=0x7ffa6806f9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:44:47.746 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.741+0000 7ffa6f144640 1 -- 172.21.15.16:0/1947896642 shutdown_connections 2023-05-22T15:44:47.746 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.741+0000 7ffa6f144640 1 -- 172.21.15.16:0/1947896642 wait complete. 2023-05-22T15:44:47.748 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.741+0000 7ffa6f144640 1 Processor -- start 2023-05-22T15:44:47.749 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.741+0000 7ffa6f144640 1 -- start start 2023-05-22T15:44:47.749 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.741+0000 7ffa6f144640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ffa680b2880 0x7ffa6815a740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:47.749 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.741+0000 7ffa6f144640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffa680cc520 0x7ffa6815cd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:47.749 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.741+0000 7ffa6f144640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffa6806c4b0 0x7ffa680b1050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:47.749 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.741+0000 7ffa6f144640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa68161e60 con 0x7ffa6806c4b0 2023-05-22T15:44:47.749 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.741+0000 7ffa6f144640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa680b3090 con 0x7ffa680cc520 2023-05-22T15:44:47.750 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.741+0000 7ffa6f144640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa6815d240 con 0x7ffa680b2880 2023-05-22T15:44:47.750 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.741+0000 7ffa6d6e1640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffa6806c4b0 0x7ffa680b1050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:44:47.750 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.741+0000 7ffa6d6e1640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffa6806c4b0 0x7ffa680b1050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35670/0 (socket says 172.21.15.16:35670) 2023-05-22T15:44:47.750 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.741+0000 7ffa6d6e1640 1 -- 172.21.15.16:0/2678774030 learned_addr learned my addr 172.21.15.16:0/2678774030 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:44:47.750 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.741+0000 7ffa67fff640 1 --2- 172.21.15.16:0/2678774030 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffa680cc520 0x7ffa6815cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:44:47.750 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.741+0000 7ffa6cee0640 1 --2- 172.21.15.16:0/2678774030 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ffa680b2880 0x7ffa6815a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:44:47.751 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.745+0000 7ffa6d6e1640 1 -- 172.21.15.16:0/2678774030 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ffa680b2880 msgr2=0x7ffa6815a740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:44:47.751 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.745+0000 7ffa6d6e1640 1 --2- 172.21.15.16:0/2678774030 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ffa680b2880 0x7ffa6815a740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:47.751 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.745+0000 7ffa6d6e1640 1 -- 172.21.15.16:0/2678774030 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffa680cc520 msgr2=0x7ffa6815cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:44:47.751 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.745+0000 7ffa6d6e1640 1 --2- 172.21.15.16:0/2678774030 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffa680cc520 0x7ffa6815cd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:47.751 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.745+0000 7ffa6d6e1640 1 -- 172.21.15.16:0/2678774030 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa60009b30 con 0x7ffa6806c4b0 2023-05-22T15:44:47.751 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.745+0000 7ffa6d6e1640 1 --2- 172.21.15.16:0/2678774030 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffa6806c4b0 0x7ffa680b1050 secure :-1 s=READY pgs=1530 cs=0 l=1 rev1=1 crypto rx=0x7ffa5800c390 tx=0x7ffa5800c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:44:47.751 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.745+0000 7ffa657fa640 1 -- 172.21.15.16:0/2678774030 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa580052d0 con 0x7ffa6806c4b0 2023-05-22T15:44:47.751 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.745+0000 7ffa657fa640 1 -- 172.21.15.16:0/2678774030 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa58005430 con 0x7ffa6806c4b0 2023-05-22T15:44:47.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.745+0000 7ffa6f144640 1 -- 172.21.15.16:0/2678774030 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa680d0790 con 0x7ffa6806c4b0 2023-05-22T15:44:47.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.745+0000 7ffa657fa640 1 -- 172.21.15.16:0/2678774030 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa58007240 con 0x7ffa6806c4b0 2023-05-22T15:44:47.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.745+0000 7ffa6f144640 1 -- 172.21.15.16:0/2678774030 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa680d04b0 con 0x7ffa6806c4b0 2023-05-22T15:44:47.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.745+0000 7ffa6f144640 1 -- 172.21.15.16:0/2678774030 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa680d0a70 con 0x7ffa6806c4b0 2023-05-22T15:44:47.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.745+0000 7ffa657fa640 1 -- 172.21.15.16:0/2678774030 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 47) v1 ==== 84446+0+0 (secure 0 0 0) 0x7ffa580073a0 con 0x7ffa6806c4b0 2023-05-22T15:44:47.753 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.745+0000 7ffa657fa640 1 --2- 172.21.15.16:0/2678774030 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ffa480655c0 0x7ffa48067a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:47.753 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.745+0000 7ffa6cee0640 1 -- 172.21.15.16:0/2678774030 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ffa480655c0 msgr2=0x7ffa48067a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/36921 2023-05-22T15:44:47.753 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.745+0000 7ffa657fa640 1 -- 172.21.15.16:0/2678774030 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(119..119 src has 1..119) v4 ==== 6726+0+0 (secure 0 0 0) 0x7ffa580820a0 con 0x7ffa6806c4b0 2023-05-22T15:44:47.753 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.745+0000 7ffa6cee0640 1 --2- 172.21.15.16:0/2678774030 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ffa480655c0 0x7ffa48067a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-05-22T15:44:47.753 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.749+0000 7ffa657fa640 1 -- 172.21.15.16:0/2678774030 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7ffa58049620 con 0x7ffa6806c4b0 2023-05-22T15:44:47.773 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:44:47.769+0000 7f4d405c0180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-05-22T15:44:47.820 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:44:47.818+0000 7f8f9a67d180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-05-22T15:44:47.851 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:44:47.849+0000 7f4d405c0180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-05-22T15:44:47.871 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:44:47.866+0000 7f8f9a67d180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-05-22T15:44:47.903 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:44:47.901+0000 7f4d405c0180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-05-22T15:44:47.905 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.901+0000 7ffa6f144640 1 -- 172.21.15.16:0/2678774030 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7ffa6804dbe0 con 0x7ffa6806c4b0 2023-05-22T15:44:47.906 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.901+0000 7ffa657fa640 1 -- 172.21.15.16:0/2678774030 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v119) v1 ==== 104+0+11 (secure 0 0 0) 0x7ffa5804f820 con 0x7ffa6806c4b0 2023-05-22T15:44:47.906 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2023-05-22T15:44:47.908 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.905+0000 7ffa6f144640 1 -- 172.21.15.16:0/2678774030 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ffa480655c0 msgr2=0x7ffa48067a70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-05-22T15:44:47.908 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.905+0000 7ffa6f144640 1 --2- 172.21.15.16:0/2678774030 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ffa480655c0 0x7ffa48067a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:47.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.905+0000 7ffa6f144640 1 -- 172.21.15.16:0/2678774030 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffa6806c4b0 msgr2=0x7ffa680b1050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:44:47.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.905+0000 7ffa6f144640 1 --2- 172.21.15.16:0/2678774030 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffa6806c4b0 0x7ffa680b1050 secure :-1 s=READY pgs=1530 cs=0 l=1 rev1=1 crypto rx=0x7ffa5800c390 tx=0x7ffa5800c850 comp rx=0 tx=0).stop 2023-05-22T15:44:47.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.905+0000 7ffa6f144640 1 -- 172.21.15.16:0/2678774030 shutdown_connections 2023-05-22T15:44:47.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.905+0000 7ffa6f144640 1 --2- 172.21.15.16:0/2678774030 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffa6806c4b0 0x7ffa680b1050 unknown :-1 s=CLOSED pgs=1530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:47.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.905+0000 7ffa6f144640 1 --2- 172.21.15.16:0/2678774030 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffa680cc520 0x7ffa6815cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:47.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.905+0000 7ffa6f144640 1 --2- 172.21.15.16:0/2678774030 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ffa480655c0 0x7ffa48067a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:47.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.905+0000 7ffa6f144640 1 --2- 172.21.15.16:0/2678774030 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ffa680b2880 0x7ffa6815a740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:47.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.905+0000 7ffa6f144640 1 -- 172.21.15.16:0/2678774030 >> 172.21.15.16:0/2678774030 conn(0x7ffa6806f5a0 msgr2=0x7ffa68159570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:44:47.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.905+0000 7ffa6f144640 1 -- 172.21.15.16:0/2678774030 shutdown_connections 2023-05-22T15:44:47.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:47.905+0000 7ffa6f144640 1 -- 172.21.15.16:0/2678774030 wait complete. 2023-05-22T15:44:47.925 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-05-22T15:44:47.948 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:44:47.942+0000 7f8f9a67d180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-05-22T15:44:47.984 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:44:47.981+0000 7f4d405c0180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-05-22T15:44:48.001 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:44:47.998+0000 7f8f9a67d180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-05-22T15:44:48.036 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:44:48.033+0000 7f4d405c0180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-05-22T15:44:48.048 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:48.045+0000 7f398f13c640 1 Processor -- start 2023-05-22T15:44:48.048 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:48.045+0000 7f398f13c640 1 -- start start 2023-05-22T15:44:48.050 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:48.045+0000 7f398f13c640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f39880ce480 0x7f3988071640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:48.050 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:48.045+0000 7f398f13c640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f39880ce850 0x7f3988160090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:48.050 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:48.045+0000 7f398f13c640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3988069ae0 con 0x7f39881605d0 2023-05-22T15:44:48.050 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:48.045+0000 7f398f13c640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f398815f990 con 0x7f39880ce850 2023-05-22T15:44:48.051 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:48.045+0000 7f398f13c640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3988069dc0 con 0x7f39880ce480 2023-05-22T15:44:48.051 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:48.045+0000 7f3987fff640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f39880ce850 0x7f3988160090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:44:48.051 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:48.045+0000 7f398d6d9640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f39881605d0 0x7f398815c270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:47706/0 (socket says 172.21.15.16:47706) 2023-05-22T15:44:48.051 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:48.045+0000 7f398d6d9640 1 -- 172.21.15.16:0/2784262859 learned_addr learned my addr 172.21.15.16:0/2784262859 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:44:48.051 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:48.045+0000 7f3987fff640 1 --2- 172.21.15.16:0/2784262859 >> v2:172.21.15.16:3300/0 conn(0x7f39880ce850 0x7f3988160090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35678/0 (socket says 172.21.15.16:35678) 2023-05-22T15:44:48.051 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:48.045+0000 7f398ced8640 1 --2- 172.21.15.16:0/2784262859 >> v2:172.21.15.62:3300/0 conn(0x7f39880ce480 0x7f3988071640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:44:48.052 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:48.045+0000 7f3987fff640 1 -- 172.21.15.16:0/2784262859 >> v2:172.21.15.62:3300/0 conn(0x7f39880ce480 msgr2=0x7f3988071640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:44:48.052 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:48.045+0000 7f3987fff640 1 --2- 172.21.15.16:0/2784262859 >> v2:172.21.15.62:3300/0 conn(0x7f39880ce480 0x7f3988071640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:48.052 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:48.045+0000 7f3987fff640 1 -- 172.21.15.16:0/2784262859 >> v1:172.21.15.16:6789/0 conn(0x7f39881605d0 legacy=0x7f398815c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:44:48.052 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:48.045+0000 7f3987fff640 1 -- 172.21.15.16:0/2784262859 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f398806a0a0 con 0x7f39880ce850 2023-05-22T15:44:48.052 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:48.045+0000 7f39877fe640 1 -- 172.21.15.16:0/2784262859 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1816902772 0 0) 0x7f3988069ae0 con 0x7f39881605d0 2023-05-22T15:44:48.053 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:48.049+0000 7f3987fff640 1 --2- 172.21.15.16:0/2784262859 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f39880ce850 0x7f3988160090 secure :-1 s=READY pgs=1532 cs=0 l=1 rev1=1 crypto rx=0x7f398806f310 tx=0x7f397c01c430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c25e7af1083c3487 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:44:48.053 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:48.049+0000 7f39877fe640 1 -- 172.21.15.16:0/2784262859 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f397c0ab600 con 0x7f39880ce850 2023-05-22T15:44:48.054 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:48.049+0000 7f39877fe640 1 -- 172.21.15.16:0/2784262859 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f397c0a7050 con 0x7f39880ce850 2023-05-22T15:44:48.056 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:48.049+0000 7f39877fe640 1 -- 172.21.15.16:0/2784262859 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f397c0ab980 con 0x7f39880ce850 2023-05-22T15:44:48.056 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:48.049+0000 7f398f13c640 1 -- 172.21.15.16:0/2784262859 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f39880ce850 msgr2=0x7f3988160090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:44:48.057 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:48.049+0000 7f398f13c640 1 --2- 172.21.15.16:0/2784262859 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f39880ce850 0x7f3988160090 secure :-1 s=READY pgs=1532 cs=0 l=1 rev1=1 crypto rx=0x7f398806f310 tx=0x7f397c01c430 comp rx=0 tx=0).stop 2023-05-22T15:44:48.057 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:48.049+0000 7f398f13c640 1 -- 172.21.15.16:0/2784262859 shutdown_connections 2023-05-22T15:44:48.057 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:48.049+0000 7f398f13c640 1 --2- 172.21.15.16:0/2784262859 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f39880ce850 0x7f3988160090 unknown :-1 s=CLOSED pgs=1532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:48.057 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:48.049+0000 7f398f13c640 1 --2- 172.21.15.16:0/2784262859 >> v2:172.21.15.62:3300/0 conn(0x7f39880ce480 0x7f3988071640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:48.057 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:48.049+0000 7f398f13c640 1 -- 172.21.15.16:0/2784262859 >> 172.21.15.16:0/2784262859 conn(0x7f39880cc870 msgr2=0x7f39880cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:44:48.057 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:48.049+0000 7f398f13c640 1 -- 172.21.15.16:0/2784262859 shutdown_connections 2023-05-22T15:44:48.057 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:48.049+0000 7f398f13c640 1 -- 172.21.15.16:0/2784262859 wait complete. 2023-05-22T15:44:48.060 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:48.049+0000 7f398f13c640 1 Processor -- start 2023-05-22T15:44:48.060 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:48.049+0000 7f398f13c640 1 -- start start 2023-05-22T15:44:48.060 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:48.053+0000 7f398f13c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f39880ce480 0x7f398815f220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:48.061 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:48.053+0000 7f398f13c640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f39880ce850 0x7f39880c4fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:48.061 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:48.053+0000 7f398f13c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f39881605d0 0x7f398815d630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:48.061 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:48.053+0000 7f398f13c640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39880c5520 con 0x7f39881605d0 2023-05-22T15:44:48.061 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:48.053+0000 7f398f13c640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39880b1b20 con 0x7f39880ce480 2023-05-22T15:44:48.061 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:48.053+0000 7f398f13c640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39880ab870 con 0x7f39880ce850 2023-05-22T15:44:48.061 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:48.053+0000 7f398d6d9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f39881605d0 0x7f398815d630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:44:48.061 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:48.053+0000 7f398d6d9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f39881605d0 0x7f398815d630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35686/0 (socket says 172.21.15.16:35686) 2023-05-22T15:44:48.062 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:48.053+0000 7f398d6d9640 1 -- 172.21.15.16:0/3495970419 learned_addr learned my addr 172.21.15.16:0/3495970419 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:44:48.062 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:48.053+0000 7f3987fff640 1 --2- 172.21.15.16:0/3495970419 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f39880ce850 0x7f39880c4fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:44:48.062 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:48.053+0000 7f398ced8640 1 --2- 172.21.15.16:0/3495970419 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f39880ce480 0x7f398815f220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:44:48.062 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:48.053+0000 7f398d6d9640 1 -- 172.21.15.16:0/3495970419 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f39880ce850 msgr2=0x7f39880c4fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:44:48.062 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:48.053+0000 7f398d6d9640 1 --2- 172.21.15.16:0/3495970419 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f39880ce850 0x7f39880c4fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:48.062 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:48.053+0000 7f398d6d9640 1 -- 172.21.15.16:0/3495970419 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f39880ce480 msgr2=0x7f398815f220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:44:48.062 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:48.053+0000 7f398d6d9640 1 --2- 172.21.15.16:0/3495970419 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f39880ce480 0x7f398815f220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:48.062 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:48.053+0000 7f398d6d9640 1 -- 172.21.15.16:0/3495970419 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f397c009420 con 0x7f39881605d0 2023-05-22T15:44:48.063 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:48.053+0000 7f398d6d9640 1 --2- 172.21.15.16:0/3495970419 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f39881605d0 0x7f398815d630 secure :-1 s=READY pgs=1533 cs=0 l=1 rev1=1 crypto rx=0x7f398000c780 tx=0x7f398000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:44:48.063 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:48.053+0000 7f39857fa640 1 -- 172.21.15.16:0/3495970419 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f398000ce90 con 0x7f39881605d0 2023-05-22T15:44:48.063 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:48.053+0000 7f39857fa640 1 -- 172.21.15.16:0/3495970419 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f398000a5e0 con 0x7f39881605d0 2023-05-22T15:44:48.063 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:48.053+0000 7f398f13c640 1 -- 172.21.15.16:0/3495970419 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39880698b0 con 0x7f39881605d0 2023-05-22T15:44:48.063 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:48.053+0000 7f39857fa640 1 -- 172.21.15.16:0/3495970419 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f398000a970 con 0x7f39881605d0 2023-05-22T15:44:48.063 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:48.053+0000 7f398f13c640 1 -- 172.21.15.16:0/3495970419 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3988069720 con 0x7f39881605d0 2023-05-22T15:44:48.063 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:48.053+0000 7f398f13c640 1 -- 172.21.15.16:0/3495970419 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39880c5e70 con 0x7f39881605d0 2023-05-22T15:44:48.064 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:48.053+0000 7f39857fa640 1 -- 172.21.15.16:0/3495970419 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 47) v1 ==== 84446+0+0 (secure 0 0 0) 0x7f398000aad0 con 0x7f39881605d0 2023-05-22T15:44:48.064 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:48.053+0000 7f39857fa640 1 --2- 172.21.15.16:0/3495970419 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f39740655c0 0x7f3974067a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:48.064 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:48.053+0000 7f398ced8640 1 -- 172.21.15.16:0/3495970419 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f39740655c0 msgr2=0x7f3974067a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/36921 2023-05-22T15:44:48.064 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:48.053+0000 7f39857fa640 1 -- 172.21.15.16:0/3495970419 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(119..119 src has 1..119) v4 ==== 6726+0+0 (secure 0 0 0) 0x7f3980082630 con 0x7f39881605d0 2023-05-22T15:44:48.064 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:48.053+0000 7f398ced8640 1 --2- 172.21.15.16:0/3495970419 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f39740655c0 0x7f3974067a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-05-22T15:44:48.064 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:48.057+0000 7f39857fa640 1 -- 172.21.15.16:0/3495970419 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f398004fd40 con 0x7f39881605d0 2023-05-22T15:44:48.134 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:44:48.130+0000 7f8f9a67d180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-05-22T15:44:48.172 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:44:48.169+0000 7f4d405c0180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-05-22T15:44:48.186 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:44:48.182+0000 7f8f9a67d180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-05-22T15:44:48.213 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:48.209+0000 7f398f13c640 1 -- 172.21.15.16:0/3495970419 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f39880657b0 con 0x7f39881605d0 2023-05-22T15:44:48.213 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:48.209+0000 7f39857fa640 1 -- 172.21.15.16:0/3495970419 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v119) v1 ==== 100+0+11 (secure 0 0 0) 0x7f3980053360 con 0x7f39881605d0 2023-05-22T15:44:48.214 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 64 2023-05-22T15:44:48.216 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:48.213+0000 7f398f13c640 1 -- 172.21.15.16:0/3495970419 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f39740655c0 msgr2=0x7f3974067a70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-05-22T15:44:48.216 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:48.213+0000 7f398f13c640 1 --2- 172.21.15.16:0/3495970419 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f39740655c0 0x7f3974067a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:48.216 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:48.213+0000 7f398f13c640 1 -- 172.21.15.16:0/3495970419 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f39881605d0 msgr2=0x7f398815d630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:44:48.216 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:48.213+0000 7f398f13c640 1 --2- 172.21.15.16:0/3495970419 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f39881605d0 0x7f398815d630 secure :-1 s=READY pgs=1533 cs=0 l=1 rev1=1 crypto rx=0x7f398000c780 tx=0x7f398000cc40 comp rx=0 tx=0).stop 2023-05-22T15:44:48.216 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:48.213+0000 7f398f13c640 1 -- 172.21.15.16:0/3495970419 shutdown_connections 2023-05-22T15:44:48.217 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:48.213+0000 7f398f13c640 1 --2- 172.21.15.16:0/3495970419 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f39881605d0 0x7f398815d630 unknown :-1 s=CLOSED pgs=1533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:48.217 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:48.213+0000 7f398f13c640 1 --2- 172.21.15.16:0/3495970419 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f39740655c0 0x7f3974067a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:48.217 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:48.213+0000 7f398f13c640 1 --2- 172.21.15.16:0/3495970419 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f39880ce850 0x7f39880c4fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:48.217 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:48.213+0000 7f398f13c640 1 --2- 172.21.15.16:0/3495970419 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f39880ce480 0x7f398815f220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:48.217 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:48.213+0000 7f398f13c640 1 -- 172.21.15.16:0/3495970419 >> 172.21.15.16:0/3495970419 conn(0x7f39880cc870 msgr2=0x7f398815b900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:44:48.217 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:48.213+0000 7f398f13c640 1 -- 172.21.15.16:0/3495970419 shutdown_connections 2023-05-22T15:44:48.217 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:48.213+0000 7f398f13c640 1 -- 172.21.15.16:0/3495970419 wait complete. 2023-05-22T15:44:48.226 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:44:48.221+0000 7f4d405c0180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-05-22T15:44:48.233 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-05-22T15:44:48.347 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:48.345+0000 7f766ad45640 1 Processor -- start 2023-05-22T15:44:48.347 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:48.345+0000 7f766ad45640 1 -- start start 2023-05-22T15:44:48.349 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:48.345+0000 7f766ad45640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f766415c2f0 0x7f7664162e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:48.350 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:48.345+0000 7f766ad45640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7664158eb0 con 0x7f7664070600 2023-05-22T15:44:48.350 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:48.345+0000 7f766ad45640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f766406cf20 con 0x7f766406dab0 2023-05-22T15:44:48.350 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:48.345+0000 7f766ad45640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f766406de80 con 0x7f766415c2f0 2023-05-22T15:44:48.350 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:48.345+0000 7f765bfff640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f7664070600 0x7f766415bbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:47716/0 (socket says 172.21.15.16:47716) 2023-05-22T15:44:48.350 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:48.345+0000 7f765bfff640 1 -- 172.21.15.16:0/2180080052 learned_addr learned my addr 172.21.15.16:0/2180080052 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:44:48.350 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:48.345+0000 7f76692e2640 1 --2- 172.21.15.16:0/2180080052 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f766415c2f0 0x7f7664162e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:44:48.350 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:48.345+0000 7f765b7fe640 1 -- 172.21.15.16:0/2180080052 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1409974133 0 0) 0x7f7664158eb0 con 0x7f7664070600 2023-05-22T15:44:48.351 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:48.345+0000 7f765b7fe640 1 -- 172.21.15.16:0/2180080052 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f764c003530 con 0x7f7664070600 2023-05-22T15:44:48.351 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:48.345+0000 7f765b7fe640 1 -- 172.21.15.16:0/2180080052 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3831002144 0 0) 0x7f764c003530 con 0x7f7664070600 2023-05-22T15:44:48.351 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:48.345+0000 7f765b7fe640 1 -- 172.21.15.16:0/2180080052 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f766415c2f0 msgr2=0x7f7664162e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:44:48.351 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:48.345+0000 7f765b7fe640 1 --2- 172.21.15.16:0/2180080052 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f766415c2f0 0x7f7664162e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:48.351 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:48.345+0000 7f765b7fe640 1 -- 172.21.15.16:0/2180080052 >> v1:172.21.15.62:6789/0 conn(0x7f766406dab0 legacy=0x7f766406fef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:44:48.351 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:48.345+0000 7f765b7fe640 1 -- 172.21.15.16:0/2180080052 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f766415c950 con 0x7f7664070600 2023-05-22T15:44:48.352 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:48.345+0000 7f765b7fe640 1 -- 172.21.15.16:0/2180080052 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7f7654001680 con 0x7f7664070600 2023-05-22T15:44:48.352 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:48.345+0000 7f765b7fe640 1 -- 172.21.15.16:0/2180080052 >> v1:172.21.15.16:6789/0 conn(0x7f7664070600 legacy=0x7f766415bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:44:48.352 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:48.345+0000 7f765b7fe640 1 --2- 172.21.15.16:0/2180080052 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f766415c2f0 0x7f764c0039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:48.352 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:48.345+0000 7f765b7fe640 1 --2- 172.21.15.16:0/2180080052 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f764c003ef0 0x7f764c0047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:48.352 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:48.345+0000 7f765b7fe640 1 --2- 172.21.15.16:0/2180080052 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f764c004d10 0x7f764c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:48.352 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:48.345+0000 7f765b7fe640 1 -- 172.21.15.16:0/2180080052 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f766406de80 con 0x7f764c003ef0 2023-05-22T15:44:48.352 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:48.345+0000 7f765b7fe640 1 -- 172.21.15.16:0/2180080052 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f764c00b9c0 con 0x7f764c004d10 2023-05-22T15:44:48.352 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:48.345+0000 7f765b7fe640 1 -- 172.21.15.16:0/2180080052 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f764c00bb80 con 0x7f766415c2f0 2023-05-22T15:44:48.353 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:48.345+0000 7f765bfff640 1 --2- 172.21.15.16:0/2180080052 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f764c003ef0 0x7f764c0047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:44:48.353 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:48.345+0000 7f76692e2640 1 --2- 172.21.15.16:0/2180080052 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f764c004d10 0x7f764c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:44:48.353 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:48.345+0000 7f7668ae1640 1 --2- 172.21.15.16:0/2180080052 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f766415c2f0 0x7f764c0039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:44:48.353 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:48.345+0000 7f765bfff640 1 -- 172.21.15.16:0/2180080052 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f766415c2f0 msgr2=0x7f764c0039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:44:48.353 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:48.345+0000 7f765bfff640 1 --2- 172.21.15.16:0/2180080052 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f766415c2f0 0x7f764c0039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:48.353 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:48.345+0000 7f765bfff640 1 -- 172.21.15.16:0/2180080052 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f764c004d10 msgr2=0x7f764c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:44:48.353 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:48.345+0000 7f765bfff640 1 --2- 172.21.15.16:0/2180080052 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f764c004d10 0x7f764c005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:48.354 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:48.345+0000 7f765bfff640 1 -- 172.21.15.16:0/2180080052 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f764c003530 con 0x7f764c003ef0 2023-05-22T15:44:48.354 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:48.345+0000 7f76692e2640 1 --2- 172.21.15.16:0/2180080052 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f764c004d10 0x7f764c005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T15:44:48.356 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:48.349+0000 7f765bfff640 1 --2- 172.21.15.16:0/2180080052 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f764c003ef0 0x7f764c0047d0 secure :-1 s=READY pgs=1535 cs=0 l=1 rev1=1 crypto rx=0x7f765400c210 tx=0x7f765401cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b135be963a129c52 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:44:48.356 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:48.349+0000 7f765b7fe640 1 -- 172.21.15.16:0/2180080052 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76540a5b40 con 0x7f764c003ef0 2023-05-22T15:44:48.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:48.349+0000 7f765b7fe640 1 -- 172.21.15.16:0/2180080052 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76540a5ca0 con 0x7f764c003ef0 2023-05-22T15:44:48.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:48.349+0000 7f765b7fe640 1 -- 172.21.15.16:0/2180080052 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76540a5b40 con 0x7f764c003ef0 2023-05-22T15:44:48.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:48.349+0000 7f766ad45640 1 -- 172.21.15.16:0/2180080052 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f764c003ef0 msgr2=0x7f764c0047d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:44:48.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:48.349+0000 7f766ad45640 1 --2- 172.21.15.16:0/2180080052 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f764c003ef0 0x7f764c0047d0 secure :-1 s=READY pgs=1535 cs=0 l=1 rev1=1 crypto rx=0x7f765400c210 tx=0x7f765401cc40 comp rx=0 tx=0).stop 2023-05-22T15:44:48.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:48.349+0000 7f76692e2640 1 -- 172.21.15.16:0/2180080052 reap_dead start 2023-05-22T15:44:48.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:48.349+0000 7f766ad45640 1 -- 172.21.15.16:0/2180080052 shutdown_connections 2023-05-22T15:44:48.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:48.349+0000 7f766ad45640 1 -- 172.21.15.16:0/2180080052 >> 172.21.15.16:0/2180080052 conn(0x7f76640b1110 msgr2=0x7f76640ce830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:44:48.358 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:48.353+0000 7f766ad45640 1 -- 172.21.15.16:0/2180080052 shutdown_connections 2023-05-22T15:44:48.358 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:48.353+0000 7f766ad45640 1 -- 172.21.15.16:0/2180080052 wait complete. 2023-05-22T15:44:48.358 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:48.353+0000 7f766ad45640 1 Processor -- start 2023-05-22T15:44:48.358 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:48.353+0000 7f766ad45640 1 -- start start 2023-05-22T15:44:48.359 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:48.353+0000 7f766ad45640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7664156330 0x7f76640acf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:48.359 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:48.353+0000 7f766ad45640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f76640b17c0 0x7f76641e1940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:48.359 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:48.353+0000 7f766ad45640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f766406bdf0 0x7f766406c260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:48.359 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:48.353+0000 7f766ad45640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f766415c770 con 0x7f766406bdf0 2023-05-22T15:44:48.360 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:48.353+0000 7f766ad45640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f766406c7a0 con 0x7f76640b17c0 2023-05-22T15:44:48.360 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:48.353+0000 7f766ad45640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7664156700 con 0x7f7664156330 2023-05-22T15:44:48.360 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:48.353+0000 7f7668ae1640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7664156330 0x7f76640acf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:44:48.360 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:48.353+0000 7f7668ae1640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7664156330 0x7f76640acf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3301/0 says I am v2:172.21.15.16:53452/0 (socket says 172.21.15.16:53452) 2023-05-22T15:44:48.360 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:48.353+0000 7f7668ae1640 1 -- 172.21.15.16:0/1746306943 learned_addr learned my addr 172.21.15.16:0/1746306943 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:44:48.360 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:48.353+0000 7f765bfff640 1 --2- 172.21.15.16:0/1746306943 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f76640b17c0 0x7f76641e1940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:44:48.361 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:48.353+0000 7f76692e2640 1 --2- 172.21.15.16:0/1746306943 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f766406bdf0 0x7f766406c260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:44:48.361 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:48.353+0000 7f7668ae1640 1 -- 172.21.15.16:0/1746306943 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f76640b17c0 msgr2=0x7f76641e1940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:44:48.361 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:48.353+0000 7f7668ae1640 1 --2- 172.21.15.16:0/1746306943 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f76640b17c0 0x7f76641e1940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:48.361 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:48.353+0000 7f7668ae1640 1 -- 172.21.15.16:0/1746306943 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f766406bdf0 msgr2=0x7f766406c260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:44:48.361 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:48.353+0000 7f7668ae1640 1 --2- 172.21.15.16:0/1746306943 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f766406bdf0 0x7f766406c260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:48.361 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:48.353+0000 7f7668ae1640 1 -- 172.21.15.16:0/1746306943 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7664158b40 con 0x7f7664156330 2023-05-22T15:44:48.361 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:48.353+0000 7f7668ae1640 1 --2- 172.21.15.16:0/1746306943 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7664156330 0x7f76640acf00 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f7660010fa0 tx=0x7f766000e530 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:44:48.362 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:48.357+0000 7f76597fa640 1 -- 172.21.15.16:0/1746306943 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7660005270 con 0x7f7664156330 2023-05-22T15:44:48.362 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:48.357+0000 7f76597fa640 1 -- 172.21.15.16:0/1746306943 <== mon.2 v2:172.21.15.62:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f766000d030 con 0x7f7664156330 2023-05-22T15:44:48.362 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:48.357+0000 7f76597fa640 1 -- 172.21.15.16:0/1746306943 <== mon.2 v2:172.21.15.62:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f766000ee80 con 0x7f7664156330 2023-05-22T15:44:48.363 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:48.357+0000 7f766ad45640 1 -- 172.21.15.16:0/1746306943 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7664158860 con 0x7f7664156330 2023-05-22T15:44:48.363 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:48.357+0000 7f766ad45640 1 -- 172.21.15.16:0/1746306943 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7664158510 con 0x7f7664156330 2023-05-22T15:44:48.363 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:48.357+0000 7f766ad45640 1 -- 172.21.15.16:0/1746306943 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f766406b0d0 con 0x7f7664156330 2023-05-22T15:44:48.363 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:48.361+0000 7f76597fa640 1 -- 172.21.15.16:0/1746306943 <== mon.2 v2:172.21.15.62:3301/0 4 ==== mgrmap(e 47) v1 ==== 84446+0+0 (secure 0 0 0) 0x7f7660003600 con 0x7f7664156330 2023-05-22T15:44:48.364 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:48.361+0000 7f76597fa640 1 --2- 172.21.15.16:0/1746306943 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f7650065610 0x7f7650067ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:48.364 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:48.361+0000 7f765bfff640 1 -- 172.21.15.16:0/1746306943 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f7650065610 msgr2=0x7f7650067ac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/36921 2023-05-22T15:44:48.365 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:48.361+0000 7f76597fa640 1 -- 172.21.15.16:0/1746306943 <== mon.2 v2:172.21.15.62:3301/0 5 ==== osd_map(119..119 src has 1..119) v4 ==== 6726+0+0 (secure 0 0 0) 0x7f7660086ce0 con 0x7f7664156330 2023-05-22T15:44:48.365 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:48.361+0000 7f765bfff640 1 --2- 172.21.15.16:0/1746306943 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f7650065610 0x7f7650067ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-05-22T15:44:48.373 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:48.369+0000 7f76597fa640 1 -- 172.21.15.16:0/1746306943 <== mon.2 v2:172.21.15.62:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f76600543e0 con 0x7f7664156330 2023-05-22T15:44:48.412 INFO:tasks.ceph.mgr.x.smithi062.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2023-05-22T15:44:48.413 INFO:tasks.ceph.mgr.x.smithi062.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2023-05-22T15:44:48.413 INFO:tasks.ceph.mgr.x.smithi062.stderr: from numpy import show_config as show_numpy_config 2023-05-22T15:44:48.420 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:44:48.418+0000 7f8f9a67d180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-05-22T15:44:48.455 INFO:tasks.ceph.mgr.y.smithi016.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2023-05-22T15:44:48.455 INFO:tasks.ceph.mgr.y.smithi016.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2023-05-22T15:44:48.455 INFO:tasks.ceph.mgr.y.smithi016.stderr: from numpy import show_config as show_numpy_config 2023-05-22T15:44:48.463 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:44:48.461+0000 7f4d405c0180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-05-22T15:44:48.546 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:44:48.542+0000 7f8f9a67d180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-05-22T15:44:48.566 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:48.565+0000 7f765bfff640 1 -- 172.21.15.16:0/1746306943 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f7650065610 msgr2=0x7f7650067ac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/36921 2023-05-22T15:44:48.567 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:48.565+0000 7f765bfff640 1 --2- 172.21.15.16:0/1746306943 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f7650065610 0x7f7650067ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-05-22T15:44:48.572 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:48.569+0000 7f766ad45640 1 -- 172.21.15.16:0/1746306943 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f76640b39d0 con 0x7f7664156330 2023-05-22T15:44:48.573 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:48.569+0000 7f76597fa640 1 -- 172.21.15.16:0/1746306943 <== mon.2 v2:172.21.15.62:3301/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 8 entries v119) v1 ==== 80+0+459 (secure 0 0 0) 0x7f7660057a00 con 0x7f7664156330 2023-05-22T15:44:48.574 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/2218849021 2023-05-23T15:43:43.641715+0000 2023-05-22T15:44:48.574 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/2339322867 2023-05-23T15:43:43.641715+0000 2023-05-22T15:44:48.574 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/155043049 2023-05-23T15:43:43.641715+0000 2023-05-22T15:44:48.574 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/4168497577 2023-05-23T15:43:43.641715+0000 2023-05-22T15:44:48.574 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/3302517394 2023-05-23T15:43:43.641715+0000 2023-05-22T15:44:48.574 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/4100157757 2023-05-23T15:43:43.641715+0000 2023-05-22T15:44:48.575 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:6833/36921 2023-05-23T15:43:43.641715+0000 2023-05-22T15:44:48.575 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:6832/36921 2023-05-23T15:43:43.641715+0000 2023-05-22T15:44:48.577 INFO:teuthology.orchestra.run.smithi016.stderr:listed 8 entries 2023-05-22T15:44:48.577 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:48.573+0000 7f766ad45640 1 -- 172.21.15.16:0/1746306943 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f7650065610 msgr2=0x7f7650067ac0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-05-22T15:44:48.577 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:48.573+0000 7f766ad45640 1 --2- 172.21.15.16:0/1746306943 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f7650065610 0x7f7650067ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:48.577 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:48.573+0000 7f766ad45640 1 -- 172.21.15.16:0/1746306943 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7664156330 msgr2=0x7f76640acf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:44:48.578 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:48.573+0000 7f766ad45640 1 --2- 172.21.15.16:0/1746306943 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7664156330 0x7f76640acf00 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f7660010fa0 tx=0x7f766000e530 comp rx=0 tx=0).stop 2023-05-22T15:44:48.578 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:48.573+0000 7f766ad45640 1 -- 172.21.15.16:0/1746306943 shutdown_connections 2023-05-22T15:44:48.578 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:48.573+0000 7f766ad45640 1 --2- 172.21.15.16:0/1746306943 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f766406bdf0 0x7f766406c260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:48.578 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:48.573+0000 7f766ad45640 1 --2- 172.21.15.16:0/1746306943 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f76640b17c0 0x7f76641e1940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:48.578 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:48.573+0000 7f766ad45640 1 --2- 172.21.15.16:0/1746306943 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f7650065610 0x7f7650067ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:48.579 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:48.573+0000 7f766ad45640 1 --2- 172.21.15.16:0/1746306943 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7664156330 0x7f76640acf00 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:48.579 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:48.573+0000 7f766ad45640 1 -- 172.21.15.16:0/1746306943 >> 172.21.15.16:0/1746306943 conn(0x7f76640b1110 msgr2=0x7f76640ae680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:44:48.579 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:48.573+0000 7f766ad45640 1 -- 172.21.15.16:0/1746306943 shutdown_connections 2023-05-22T15:44:48.579 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:48.573+0000 7f766ad45640 1 -- 172.21.15.16:0/1746306943 wait complete. 2023-05-22T15:44:48.590 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:44:48.585+0000 7f4d405c0180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-05-22T15:44:48.591 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi016.front.sepia.ceph.com... 2023-05-22T15:44:48.591 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:44:48.592 DEBUG:teuthology.orchestra.run.smithi016:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2023-05-22T15:44:48.602 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi016.stderr:ceph-fuse[59097]: fuse finished with error 0 and tester_r 0 2023-05-22T15:44:48.603 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi016.front.sepia.ceph.com 2023-05-22T15:44:48.604 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:44:48.604 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-05-22T15:44:48.660 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:44:48.658+0000 7f8f9a67d180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-05-22T15:44:48.706 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:44:48.701+0000 7f4d405c0180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-05-22T15:44:48.724 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:44:48.718+0000 7f8f9a67d180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-05-22T15:44:48.764 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2023-05-22T15:44:48.765 DEBUG:teuthology.orchestra.run.smithi062:> dd if=/proc/self/mounts of=/dev/stdout 2023-05-22T15:44:48.771 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16342116k,nr_inodes=4085529,mode=755,inode64 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275932k,mode=755,inode64 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev,inode64 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=13250 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275928k,nr_inodes=818982,mode=700,uid=1000,gid=1271,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275932k,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-05-22T15:44:48.772 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi062.front.sepia.ceph.com 2023-05-22T15:44:48.772 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:44:48.772 DEBUG:teuthology.orchestra.run.smithi062:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2023-05-22T15:44:48.773 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:44:48.769+0000 7f4d405c0180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-05-22T15:44:48.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-05-22T15:44:48.817 INFO:teuthology.orchestra.run.smithi062.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2023-05-22T15:44:48.818 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-05-22T15:44:48.818 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2023-05-22T15:44:48.824 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16345192k,nr_inodes=4086298,mode=755,inode64 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3276544k,mode=755,inode64 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev,inode64 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=15732 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3276540k,nr_inodes=819135,mode=700,uid=1000,gid=1271,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3276544k,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-05-22T15:44:48.824 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi016.front.sepia.ceph.com 2023-05-22T15:44:48.825 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:44:48.825 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2023-05-22T15:44:48.850 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:44:48.846+0000 7f8f9a67d180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-05-22T15:44:48.871 INFO:teuthology.orchestra.run.smithi016.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2023-05-22T15:44:48.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-05-22T15:44:48.872 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2023-05-22T15:44:48.873 DEBUG:teuthology.orchestra.run.smithi062:> dd if=/proc/self/mounts of=/dev/stdout 2023-05-22T15:44:48.879 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16342116k,nr_inodes=4085529,mode=755,inode64 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275932k,mode=755,inode64 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev,inode64 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=13250 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275928k,nr_inodes=818982,mode=700,uid=1000,gid=1271,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275932k,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-05-22T15:44:48.879 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi062.front.sepia.ceph.com 2023-05-22T15:44:48.880 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:44:48.880 DEBUG:teuthology.orchestra.run.smithi062:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2023-05-22T15:44:48.897 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:44:48.893+0000 7f4d405c0180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-05-22T15:44:48.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-05-22T15:44:48.926 INFO:teuthology.orchestra.run.smithi062.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2023-05-22T15:44:48.927 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-05-22T15:44:49.039 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.037+0000 7fe12693f640 1 Processor -- start 2023-05-22T15:44:49.039 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.037+0000 7fe12693f640 1 -- start start 2023-05-22T15:44:49.040 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.037+0000 7fe12693f640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe120070600 0x7fe12015bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:49.040 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.037+0000 7fe12693f640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fe12015c120 0x7fe12006bf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:49.040 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.037+0000 7fe12693f640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe120158eb0 con 0x7fe12006dab0 2023-05-22T15:44:49.040 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.037+0000 7fe12693f640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe12006de80 con 0x7fe120070600 2023-05-22T15:44:49.040 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.037+0000 7fe12693f640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe12015c950 con 0x7fe12015c120 2023-05-22T15:44:49.041 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.037+0000 7fe11ffff640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fe12006dab0 0x7fe12006fef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:47730/0 (socket says 172.21.15.16:47730) 2023-05-22T15:44:49.041 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.037+0000 7fe11ffff640 1 -- 172.21.15.16:0/3533958446 learned_addr learned my addr 172.21.15.16:0/3533958446 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:44:49.041 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.037+0000 7fe11f7fe640 1 --2- 172.21.15.16:0/3533958446 >> v2:172.21.15.16:3300/0 conn(0x7fe120070600 0x7fe12015bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:44:49.042 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.037+0000 7fe124edc640 1 --2- 172.21.15.16:0/3533958446 >> v2:172.21.15.62:3300/0 conn(0x7fe12015c120 0x7fe12006bf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:44:49.043 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.037+0000 7fe11effd640 1 -- 172.21.15.16:0/3533958446 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1837224923 0 0) 0x7fe120158eb0 con 0x7fe12006dab0 2023-05-22T15:44:49.043 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.037+0000 7fe11effd640 1 -- 172.21.15.16:0/3533958446 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe108003430 con 0x7fe12006dab0 2023-05-22T15:44:49.043 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.037+0000 7fe11f7fe640 1 -- 172.21.15.16:0/3533958446 >> v2:172.21.15.62:3300/0 conn(0x7fe12015c120 msgr2=0x7fe12006bf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:44:49.043 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.037+0000 7fe11f7fe640 1 --2- 172.21.15.16:0/3533958446 >> v2:172.21.15.62:3300/0 conn(0x7fe12015c120 0x7fe12006bf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:49.043 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.037+0000 7fe11f7fe640 1 -- 172.21.15.16:0/3533958446 >> v1:172.21.15.16:6789/0 conn(0x7fe12006dab0 legacy=0x7fe12006fef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:44:49.043 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.037+0000 7fe11f7fe640 1 -- 172.21.15.16:0/3533958446 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe120158860 con 0x7fe120070600 2023-05-22T15:44:49.043 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.041+0000 7fe11f7fe640 1 --2- 172.21.15.16:0/3533958446 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe120070600 0x7fe12015bbe0 secure :-1 s=READY pgs=1537 cs=0 l=1 rev1=1 crypto rx=0x7fe1200cb260 tx=0x7fe11401c430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c05471dacebfb9a server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:44:49.044 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.041+0000 7fe11effd640 1 -- 172.21.15.16:0/3533958446 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1140ac630 con 0x7fe120070600 2023-05-22T15:44:49.044 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.041+0000 7fe11effd640 1 -- 172.21.15.16:0/3533958446 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1140ac790 con 0x7fe120070600 2023-05-22T15:44:49.044 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.041+0000 7fe11effd640 1 -- 172.21.15.16:0/3533958446 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1140acb20 con 0x7fe120070600 2023-05-22T15:44:49.045 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.041+0000 7fe12693f640 1 -- 172.21.15.16:0/3533958446 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe120070600 msgr2=0x7fe12015bbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:44:49.045 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.041+0000 7fe12693f640 1 --2- 172.21.15.16:0/3533958446 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe120070600 0x7fe12015bbe0 secure :-1 s=READY pgs=1537 cs=0 l=1 rev1=1 crypto rx=0x7fe1200cb260 tx=0x7fe11401c430 comp rx=0 tx=0).stop 2023-05-22T15:44:49.045 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.041+0000 7fe12693f640 1 -- 172.21.15.16:0/3533958446 shutdown_connections 2023-05-22T15:44:49.045 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.041+0000 7fe12693f640 1 --2- 172.21.15.16:0/3533958446 >> v2:172.21.15.62:3300/0 conn(0x7fe12015c120 0x7fe12006bf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:49.045 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.041+0000 7fe12693f640 1 --2- 172.21.15.16:0/3533958446 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe120070600 0x7fe12015bbe0 unknown :-1 s=CLOSED pgs=1537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:49.045 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.041+0000 7fe12693f640 1 -- 172.21.15.16:0/3533958446 >> 172.21.15.16:0/3533958446 conn(0x7fe1200b1110 msgr2=0x7fe1200ce830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:44:49.045 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.041+0000 7fe12693f640 1 -- 172.21.15.16:0/3533958446 shutdown_connections 2023-05-22T15:44:49.046 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.041+0000 7fe12693f640 1 -- 172.21.15.16:0/3533958446 wait complete. 2023-05-22T15:44:49.046 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.041+0000 7fe12693f640 1 Processor -- start 2023-05-22T15:44:49.046 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.041+0000 7fe12693f640 1 -- start start 2023-05-22T15:44:49.048 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.041+0000 7fe12693f640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe12006dab0 0x7fe1201577b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:49.049 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.041+0000 7fe12693f640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe120070600 0x7fe120140490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:49.049 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.041+0000 7fe12693f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe12015c120 0x7fe1201409d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:49.049 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.041+0000 7fe12693f640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe120157cf0 con 0x7fe12015c120 2023-05-22T15:44:49.049 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.041+0000 7fe12693f640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1200b3d70 con 0x7fe12006dab0 2023-05-22T15:44:49.049 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.041+0000 7fe12693f640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1200b4050 con 0x7fe120070600 2023-05-22T15:44:49.049 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.041+0000 7fe124edc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe12015c120 0x7fe1201409d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:44:49.049 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.041+0000 7fe124edc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe12015c120 0x7fe1201409d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35728/0 (socket says 172.21.15.16:35728) 2023-05-22T15:44:49.050 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.041+0000 7fe124edc640 1 -- 172.21.15.16:0/4132728522 learned_addr learned my addr 172.21.15.16:0/4132728522 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:44:49.050 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.041+0000 7fe11ffff640 1 --2- 172.21.15.16:0/4132728522 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe12006dab0 0x7fe1201577b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:44:49.050 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.041+0000 7fe11f7fe640 1 --2- 172.21.15.16:0/4132728522 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe120070600 0x7fe120140490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:44:49.050 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.041+0000 7fe124edc640 1 -- 172.21.15.16:0/4132728522 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe120070600 msgr2=0x7fe120140490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:44:49.050 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.041+0000 7fe124edc640 1 --2- 172.21.15.16:0/4132728522 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe120070600 0x7fe120140490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:49.050 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.041+0000 7fe124edc640 1 -- 172.21.15.16:0/4132728522 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe12006dab0 msgr2=0x7fe1201577b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:44:49.050 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.041+0000 7fe124edc640 1 --2- 172.21.15.16:0/4132728522 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe12006dab0 0x7fe1201577b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:49.051 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.041+0000 7fe124edc640 1 -- 172.21.15.16:0/4132728522 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe114009420 con 0x7fe12015c120 2023-05-22T15:44:49.051 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.041+0000 7fe11ffff640 1 --2- 172.21.15.16:0/4132728522 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe12006dab0 0x7fe1201577b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:44:49.051 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.041+0000 7fe124edc640 1 --2- 172.21.15.16:0/4132728522 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe12015c120 0x7fe1201409d0 secure :-1 s=READY pgs=1538 cs=0 l=1 rev1=1 crypto rx=0x7fe118007e60 tx=0x7fe11800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:44:49.051 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.041+0000 7fe11cff9640 1 -- 172.21.15.16:0/4132728522 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe118014420 con 0x7fe12015c120 2023-05-22T15:44:49.051 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.041+0000 7fe11cff9640 1 -- 172.21.15.16:0/4132728522 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1180145a0 con 0x7fe12015c120 2023-05-22T15:44:49.051 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.041+0000 7fe11cff9640 1 -- 172.21.15.16:0/4132728522 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe118014950 con 0x7fe12015c120 2023-05-22T15:44:49.051 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.045+0000 7fe12693f640 1 -- 172.21.15.16:0/4132728522 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe12015c6b0 con 0x7fe12015c120 2023-05-22T15:44:49.051 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.045+0000 7fe12693f640 1 -- 172.21.15.16:0/4132728522 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe12015c520 con 0x7fe12015c120 2023-05-22T15:44:49.052 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.045+0000 7fe12693f640 1 -- 172.21.15.16:0/4132728522 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe120159320 con 0x7fe12015c120 2023-05-22T15:44:49.052 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.045+0000 7fe11cff9640 1 -- 172.21.15.16:0/4132728522 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 47) v1 ==== 84446+0+0 (secure 0 0 0) 0x7fe118014ab0 con 0x7fe12015c120 2023-05-22T15:44:49.052 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.045+0000 7fe11cff9640 1 --2- 172.21.15.16:0/4132728522 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fe10c0655c0 0x7fe10c067a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:49.052 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.045+0000 7fe11ffff640 1 -- 172.21.15.16:0/4132728522 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fe10c0655c0 msgr2=0x7fe10c067a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/36921 2023-05-22T15:44:49.052 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.045+0000 7fe11cff9640 1 -- 172.21.15.16:0/4132728522 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(119..119 src has 1..119) v4 ==== 6726+0+0 (secure 0 0 0) 0x7fe1180837c0 con 0x7fe12015c120 2023-05-22T15:44:49.052 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.045+0000 7fe11ffff640 1 --2- 172.21.15.16:0/4132728522 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fe10c0655c0 0x7fe10c067a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-05-22T15:44:49.053 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.049+0000 7fe11cff9640 1 -- 172.21.15.16:0/4132728522 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fe118054940 con 0x7fe12015c120 2023-05-22T15:44:49.239 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.237+0000 7fe12693f640 1 -- 172.21.15.16:0/4132728522 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe12006b0d0 con 0x7fe12015c120 2023-05-22T15:44:49.240 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.237+0000 7fe11cff9640 1 -- 172.21.15.16:0/4132728522 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 43 v43) v1 ==== 94+0+4650 (secure 0 0 0) 0x7fe118054530 con 0x7fe12015c120 2023-05-22T15:44:49.240 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:44:49.241 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":43,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6350,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6833/1554108010","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":1554108010},{"type":"v1","addr":"172.21.15.62:6833","nonce":1554108010}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":39},{"gid":6979,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/2667219028","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":2667219028},{"type":"v1","addr":"172.21.15.62:6835","nonce":2667219028}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":39},{"gid":8013,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6836/1892737356","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1892737356},{"type":"v1","addr":"172.21.15.16:6836","nonce":1892737356}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":39}],"filesystems":[{"mdsmap":{"epoch":43,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-05-22T15:43:30.134589+0000","modified":"2023-05-22T15:43:33.144847+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8016},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8016":{"gid":8016,"name":"a","rank":0,"incarnation":40,"state":"up:active","state_seq":3,"addr":"172.21.15.16:6837/2085936428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6835","nonce":2085936428},{"type":"v1","addr":"172.21.15.16:6837","nonce":2085936428}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2023-05-22T15:44:49.241 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 43 2023-05-22T15:44:49.242 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.237+0000 7fe12693f640 1 -- 172.21.15.16:0/4132728522 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fe10c0655c0 msgr2=0x7fe10c067a70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-05-22T15:44:49.242 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.237+0000 7fe12693f640 1 --2- 172.21.15.16:0/4132728522 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fe10c0655c0 0x7fe10c067a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:49.243 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.237+0000 7fe12693f640 1 -- 172.21.15.16:0/4132728522 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe12015c120 msgr2=0x7fe1201409d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:44:49.243 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.237+0000 7fe12693f640 1 --2- 172.21.15.16:0/4132728522 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe12015c120 0x7fe1201409d0 secure :-1 s=READY pgs=1538 cs=0 l=1 rev1=1 crypto rx=0x7fe118007e60 tx=0x7fe11800e4b0 comp rx=0 tx=0).stop 2023-05-22T15:44:49.243 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.237+0000 7fe12693f640 1 -- 172.21.15.16:0/4132728522 shutdown_connections 2023-05-22T15:44:49.243 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.237+0000 7fe12693f640 1 --2- 172.21.15.16:0/4132728522 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe12015c120 0x7fe1201409d0 unknown :-1 s=CLOSED pgs=1538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:49.243 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.237+0000 7fe12693f640 1 --2- 172.21.15.16:0/4132728522 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fe10c0655c0 0x7fe10c067a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:49.243 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.237+0000 7fe12693f640 1 --2- 172.21.15.16:0/4132728522 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe120070600 0x7fe120140490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:49.243 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.237+0000 7fe12693f640 1 --2- 172.21.15.16:0/4132728522 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe12006dab0 0x7fe1201577b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:49.243 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.237+0000 7fe12693f640 1 -- 172.21.15.16:0/4132728522 >> 172.21.15.16:0/4132728522 conn(0x7fe1200b1110 msgr2=0x7fe1200ae410 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:44:49.244 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.237+0000 7fe12693f640 1 -- 172.21.15.16:0/4132728522 shutdown_connections 2023-05-22T15:44:49.244 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.241+0000 7fe12693f640 1 -- 172.21.15.16:0/4132728522 wait complete. 2023-05-22T15:44:49.259 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-05-22T15:44:49.348 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:44:49.342+0000 7f8f9a67d180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-05-22T15:44:49.377 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.373+0000 7f8cbf26e640 1 Processor -- start 2023-05-22T15:44:49.378 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.373+0000 7f8cbf26e640 1 -- start start 2023-05-22T15:44:49.379 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.373+0000 7f8cbf26e640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f8cb815c920 0x7f8cb80d0030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:49.379 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.373+0000 7f8cbf26e640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f8cb815cd70 0x7f8cb80d0570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:49.379 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.373+0000 7f8cbf26e640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cb815c490 con 0x7f8cb815fc50 2023-05-22T15:44:49.379 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.373+0000 7f8cbf26e640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8cb80cf7a0 con 0x7f8cb815c920 2023-05-22T15:44:49.380 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.373+0000 7f8cbf26e640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8cb815c140 con 0x7f8cb815cd70 2023-05-22T15:44:49.380 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.377+0000 7f8cbc809640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f8cb815c920 0x7f8cb80d0030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:44:49.380 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.377+0000 7f8cbc809640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f8cb815c920 0x7f8cb80d0030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59042/0 (socket says 172.21.15.16:59042) 2023-05-22T15:44:49.380 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.377+0000 7f8cbc809640 1 -- 172.21.15.16:0/2679423451 learned_addr learned my addr 172.21.15.16:0/2679423451 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:44:49.381 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.377+0000 7f8cbd80b640 1 --2- 172.21.15.16:0/2679423451 >> v2:172.21.15.62:3300/0 conn(0x7f8cb815cd70 0x7f8cb80d0570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:44:49.381 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.377+0000 7f8cbc809640 1 -- 172.21.15.16:0/2679423451 >> v2:172.21.15.62:3300/0 conn(0x7f8cb815cd70 msgr2=0x7f8cb80d0570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:44:49.381 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.377+0000 7f8cbc809640 1 --2- 172.21.15.16:0/2679423451 >> v2:172.21.15.62:3300/0 conn(0x7f8cb815cd70 0x7f8cb80d0570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:49.382 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.377+0000 7f8cbc809640 1 -- 172.21.15.16:0/2679423451 >> v1:172.21.15.62:6789/0 conn(0x7f8cb815fc50 legacy=0x7f8cb8160020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:44:49.382 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.377+0000 7f8cbc809640 1 -- 172.21.15.16:0/2679423451 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cb80ccb50 con 0x7f8cb815c920 2023-05-22T15:44:49.388 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.377+0000 7f8cbc809640 1 --2- 172.21.15.16:0/2679423451 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8cb815c920 0x7f8cb80d0030 secure :-1 s=READY pgs=1539 cs=0 l=1 rev1=1 crypto rx=0x7f8cb80c5f30 tx=0x7f8ca801c4c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93e7fe5cf83b8124 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:44:49.388 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.377+0000 7f8caffff640 1 -- 172.21.15.16:0/2679423451 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ca80ac630 con 0x7f8cb815c920 2023-05-22T15:44:49.389 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.377+0000 7f8caffff640 1 -- 172.21.15.16:0/2679423451 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ca80ac790 con 0x7f8cb815c920 2023-05-22T15:44:49.389 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.377+0000 7f8caffff640 1 -- 172.21.15.16:0/2679423451 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ca80acb20 con 0x7f8cb815c920 2023-05-22T15:44:49.389 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.381+0000 7f8cbf26e640 1 -- 172.21.15.16:0/2679423451 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8cb815c920 msgr2=0x7f8cb80d0030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:44:49.389 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.381+0000 7f8cbf26e640 1 --2- 172.21.15.16:0/2679423451 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8cb815c920 0x7f8cb80d0030 secure :-1 s=READY pgs=1539 cs=0 l=1 rev1=1 crypto rx=0x7f8cb80c5f30 tx=0x7f8ca801c4c0 comp rx=0 tx=0).stop 2023-05-22T15:44:49.390 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.381+0000 7f8cbf26e640 1 -- 172.21.15.16:0/2679423451 shutdown_connections 2023-05-22T15:44:49.390 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.381+0000 7f8cbf26e640 1 --2- 172.21.15.16:0/2679423451 >> v2:172.21.15.62:3300/0 conn(0x7f8cb815cd70 0x7f8cb80d0570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:49.390 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.381+0000 7f8cbf26e640 1 --2- 172.21.15.16:0/2679423451 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8cb815c920 0x7f8cb80d0030 unknown :-1 s=CLOSED pgs=1539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:49.391 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.381+0000 7f8cbf26e640 1 -- 172.21.15.16:0/2679423451 >> 172.21.15.16:0/2679423451 conn(0x7f8cb80657f0 msgr2=0x7f8cb8070e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:44:49.391 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.381+0000 7f8cbf26e640 1 -- 172.21.15.16:0/2679423451 shutdown_connections 2023-05-22T15:44:49.391 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.381+0000 7f8cbf26e640 1 -- 172.21.15.16:0/2679423451 wait complete. 2023-05-22T15:44:49.392 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.381+0000 7f8cbf26e640 1 Processor -- start 2023-05-22T15:44:49.392 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.381+0000 7f8cbf26e640 1 -- start start 2023-05-22T15:44:49.392 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.381+0000 7f8cbf26e640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8cb815c920 0x7f8cb814ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:49.393 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.381+0000 7f8cbf26e640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8cb815cd70 0x7f8cb8154ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:49.393 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.381+0000 7f8cbf26e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8cb815fc50 0x7f8cb815e2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:49.393 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.381+0000 7f8cbf26e640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cb814f110 con 0x7f8cb815fc50 2023-05-22T15:44:49.393 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.381+0000 7f8cbf26e640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cb8153e50 con 0x7f8cb815cd70 2023-05-22T15:44:49.394 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.381+0000 7f8cbf26e640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8cb8154130 con 0x7f8cb815c920 2023-05-22T15:44:49.394 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.381+0000 7f8cbd80b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8cb815fc50 0x7f8cb815e2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:44:49.394 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.381+0000 7f8cbd80b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8cb815fc50 0x7f8cb815e2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59046/0 (socket says 172.21.15.16:59046) 2023-05-22T15:44:49.395 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.381+0000 7f8cbd80b640 1 -- 172.21.15.16:0/3127060238 learned_addr learned my addr 172.21.15.16:0/3127060238 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:44:49.395 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.381+0000 7f8cbd00a640 1 --2- 172.21.15.16:0/3127060238 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8cb815c920 0x7f8cb814ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:44:49.395 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.381+0000 7f8cbc809640 1 --2- 172.21.15.16:0/3127060238 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8cb815cd70 0x7f8cb8154ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:44:49.395 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.381+0000 7f8cbd80b640 1 -- 172.21.15.16:0/3127060238 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8cb815c920 msgr2=0x7f8cb814ebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:44:49.395 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.381+0000 7f8cbd80b640 1 --2- 172.21.15.16:0/3127060238 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8cb815c920 0x7f8cb814ebd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:49.395 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.381+0000 7f8cbd80b640 1 -- 172.21.15.16:0/3127060238 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8cb815cd70 msgr2=0x7f8cb8154ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:44:49.396 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.381+0000 7f8cbd80b640 1 --2- 172.21.15.16:0/3127060238 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8cb815cd70 0x7f8cb8154ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:49.396 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.381+0000 7f8cbd80b640 1 -- 172.21.15.16:0/3127060238 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ca80094d0 con 0x7f8cb815fc50 2023-05-22T15:44:49.396 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.381+0000 7f8cbd80b640 1 --2- 172.21.15.16:0/3127060238 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8cb815fc50 0x7f8cb815e2a0 secure :-1 s=READY pgs=1540 cs=0 l=1 rev1=1 crypto rx=0x7f8cb000d800 tx=0x7f8cb000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:44:49.396 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.381+0000 7f8cadffb640 1 -- 172.21.15.16:0/3127060238 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cb000b6c0 con 0x7f8cb815fc50 2023-05-22T15:44:49.396 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.381+0000 7f8cadffb640 1 -- 172.21.15.16:0/3127060238 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8cb000b820 con 0x7f8cb815fc50 2023-05-22T15:44:49.397 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.381+0000 7f8cbf26e640 1 -- 172.21.15.16:0/3127060238 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8cb80cf130 con 0x7f8cb815fc50 2023-05-22T15:44:49.397 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.381+0000 7f8cadffb640 1 -- 172.21.15.16:0/3127060238 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cb000bbd0 con 0x7f8cb815fc50 2023-05-22T15:44:49.397 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.381+0000 7f8cbf26e640 1 -- 172.21.15.16:0/3127060238 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8cb80cefa0 con 0x7f8cb815fc50 2023-05-22T15:44:49.397 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.385+0000 7f8cbf26e640 1 -- 172.21.15.16:0/3127060238 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8cb80cf2c0 con 0x7f8cb815fc50 2023-05-22T15:44:49.397 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.385+0000 7f8cadffb640 1 -- 172.21.15.16:0/3127060238 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 47) v1 ==== 84446+0+0 (secure 0 0 0) 0x7f8cb000bd30 con 0x7f8cb815fc50 2023-05-22T15:44:49.397 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.385+0000 7f8cadffb640 1 --2- 172.21.15.16:0/3127060238 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f8c9c0655c0 0x7f8c9c067a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:49.397 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.385+0000 7f8cbd00a640 1 -- 172.21.15.16:0/3127060238 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f8c9c0655c0 msgr2=0x7f8c9c067a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/36921 2023-05-22T15:44:49.398 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.385+0000 7f8cadffb640 1 -- 172.21.15.16:0/3127060238 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(119..119 src has 1..119) v4 ==== 6726+0+0 (secure 0 0 0) 0x7f8cb0083340 con 0x7f8cb815fc50 2023-05-22T15:44:49.398 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.385+0000 7f8cbd00a640 1 --2- 172.21.15.16:0/3127060238 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f8c9c0655c0 0x7f8c9c067a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-05-22T15:44:49.398 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.389+0000 7f8cadffb640 1 -- 172.21.15.16:0/3127060238 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f8cb0050a70 con 0x7f8cb815fc50 2023-05-22T15:44:49.406 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:44:49.401+0000 7f4d405c0180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-05-22T15:44:49.424 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:44:49.422+0000 7f8f9a67d180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-05-22T15:44:49.484 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:44:49.481+0000 7f4d405c0180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-05-22T15:44:49.544 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.541+0000 7f8cbf26e640 1 -- 172.21.15.16:0/3127060238 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8cb804dbe0 con 0x7f8cb815fc50 2023-05-22T15:44:49.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.541+0000 7f8cadffb640 1 -- 172.21.15.16:0/3127060238 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v119) v1 ==== 74+0+18480 (secure 0 0 0) 0x7f8cb0054090 con 0x7f8cb815fc50 2023-05-22T15:44:49.546 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:44:49.546 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":119,"fsid":"b2719834-7847-4d98-90a6-f65d57f85c49","created":"2023-05-22T15:26:25.259314+0000","modified":"2023-05-22T15:43:43.641801+0000","last_up_change":"2023-05-22T15:26:36.448524+0000","last_in_change":"2023-05-22T15:26:27.675668+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-05-22T15:26:38.019781+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-05-22T15:26:43.570514+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2023-05-22T15:43:28.684367+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"117","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2023-05-22T15:43:29.119294+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"118","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9e7ff854-b25f-4cbb-a5c2-1799d9c53d2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":37021},{"type":"v1","addr":"172.21.15.16:6825","nonce":37021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":37021},{"type":"v1","addr":"172.21.15.16:6827","nonce":37021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":37021},{"type":"v1","addr":"172.21.15.16:6831","nonce":37021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":37021},{"type":"v1","addr":"172.21.15.16:6829","nonce":37021}]},"public_addr":"172.21.15.16:6825/37021","cluster_addr":"172.21.15.16:6827/37021","heartbeat_back_addr":"172.21.15.16:6831/37021","heartbeat_front_addr":"172.21.15.16:6829/37021","state":["exists","up"]},{"osd":1,"uuid":"4702fbc9-a927-4565-aca5-41dc69cc8264","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":37008},{"type":"v1","addr":"172.21.15.16:6801","nonce":37008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":37008},{"type":"v1","addr":"172.21.15.16:6803","nonce":37008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":37008},{"type":"v1","addr":"172.21.15.16:6807","nonce":37008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":37008},{"type":"v1","addr":"172.21.15.16:6805","nonce":37008}]},"public_addr":"172.21.15.16:6801/37008","cluster_addr":"172.21.15.16:6803/37008","heartbeat_back_addr":"172.21.15.16:6807/37008","heartbeat_front_addr":"172.21.15.16:6805/37008","state":["exists","up"]},{"osd":2,"uuid":"1e7bba46-4eba-47ef-bb81-5e2d371638ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":37009},{"type":"v1","addr":"172.21.15.16:6809","nonce":37009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":37009},{"type":"v1","addr":"172.21.15.16:6811","nonce":37009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":37009},{"type":"v1","addr":"172.21.15.16:6815","nonce":37009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":37009},{"type":"v1","addr":"172.21.15.16:6813","nonce":37009}]},"public_addr":"172.21.15.16:6809/37009","cluster_addr":"172.21.15.16:6811/37009","heartbeat_back_addr":"172.21.15.16:6815/37009","heartbeat_front_addr":"172.21.15.16:6813/37009","state":["exists","up"]},{"osd":3,"uuid":"383cacc0-5a88-4904-990e-498ca59ef48a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":37020},{"type":"v1","addr":"172.21.15.16:6817","nonce":37020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":37020},{"type":"v1","addr":"172.21.15.16:6819","nonce":37020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":37020},{"type":"v1","addr":"172.21.15.16:6823","nonce":37020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":37020},{"type":"v1","addr":"172.21.15.16:6821","nonce":37020}]},"public_addr":"172.21.15.16:6817/37020","cluster_addr":"172.21.15.16:6819/37020","heartbeat_back_addr":"172.21.15.16:6823/37020","heartbeat_front_addr":"172.21.15.16:6821/37020","state":["exists","up"]},{"osd":4,"uuid":"970500ef-95b7-42c1-b080-eed4fae7013d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":37353},{"type":"v1","addr":"172.21.15.62:6817","nonce":37353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6819","nonce":37353},{"type":"v1","addr":"172.21.15.62:6822","nonce":37353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37353},{"type":"v1","addr":"172.21.15.62:6831","nonce":37353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37353},{"type":"v1","addr":"172.21.15.62:6826","nonce":37353}]},"public_addr":"172.21.15.62:6817/37353","cluster_addr":"172.21.15.62:6822/37353","heartbeat_back_addr":"172.21.15.62:6831/37353","heartbeat_front_addr":"172.21.15.62:6826/37353","state":["exists","up"]},{"osd":5,"uuid":"022d67db-d4d8-445b-b1b9-8b446a00f5c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":37352},{"type":"v1","addr":"172.21.15.62:6801","nonce":37352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":37352},{"type":"v1","addr":"172.21.15.62:6803","nonce":37352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":37352},{"type":"v1","addr":"172.21.15.62:6807","nonce":37352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":37352},{"type":"v1","addr":"172.21.15.62:6805","nonce":37352}]},"public_addr":"172.21.15.62:6801/37352","cluster_addr":"172.21.15.62:6803/37352","heartbeat_back_addr":"172.21.15.62:6807/37352","heartbeat_front_addr":"172.21.15.62:6805/37352","state":["exists","up"]},{"osd":6,"uuid":"42bd372b-6688-4a1b-8407-e16e8c266e33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":37354},{"type":"v1","addr":"172.21.15.62:6820","nonce":37354}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":37354},{"type":"v1","addr":"172.21.15.62:6823","nonce":37354}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6829","nonce":37354},{"type":"v1","addr":"172.21.15.62:6830","nonce":37354}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6825","nonce":37354},{"type":"v1","addr":"172.21.15.62:6827","nonce":37354}]},"public_addr":"172.21.15.62:6820/37354","cluster_addr":"172.21.15.62:6823/37354","heartbeat_back_addr":"172.21.15.62:6830/37354","heartbeat_front_addr":"172.21.15.62:6827/37354","state":["exists","up"]},{"osd":7,"uuid":"ac70b2a3-f41b-4bf2-b227-7f2cbb55426d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37355},{"type":"v1","addr":"172.21.15.62:6809","nonce":37355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37355},{"type":"v1","addr":"172.21.15.62:6811","nonce":37355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":37355},{"type":"v1","addr":"172.21.15.62:6815","nonce":37355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":37355},{"type":"v1","addr":"172.21.15.62:6813","nonce":37355}]},"public_addr":"172.21.15.62:6809/37355","cluster_addr":"172.21.15.62:6811/37355","heartbeat_back_addr":"172.21.15.62:6815/37355","heartbeat_front_addr":"172.21.15.62:6813/37355","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.368896+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.107273+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.276888+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.287502+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.124791+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.251937+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.255679+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.232797+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:0/2218849021":"2023-05-23T15:43:43.641715+0000","172.21.15.16:0/2339322867":"2023-05-23T15:43:43.641715+0000","172.21.15.16:0/155043049":"2023-05-23T15:43:43.641715+0000","172.21.15.16:0/4168497577":"2023-05-23T15:43:43.641715+0000","172.21.15.16:0/3302517394":"2023-05-23T15:43:43.641715+0000","172.21.15.16:0/4100157757":"2023-05-23T15:43:43.641715+0000","172.21.15.16:6833/36921":"2023-05-23T15:43:43.641715+0000","172.21.15.16:6832/36921":"2023-05-23T15:43:43.641715+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-05-22T15:44:49.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.545+0000 7f8cbf26e640 1 -- 172.21.15.16:0/3127060238 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f8c9c0655c0 msgr2=0x7f8c9c067a70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-05-22T15:44:49.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.545+0000 7f8cbf26e640 1 --2- 172.21.15.16:0/3127060238 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f8c9c0655c0 0x7f8c9c067a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:49.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.545+0000 7f8cbf26e640 1 -- 172.21.15.16:0/3127060238 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8cb815fc50 msgr2=0x7f8cb815e2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:44:49.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.545+0000 7f8cbf26e640 1 --2- 172.21.15.16:0/3127060238 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8cb815fc50 0x7f8cb815e2a0 secure :-1 s=READY pgs=1540 cs=0 l=1 rev1=1 crypto rx=0x7f8cb000d800 tx=0x7f8cb000dcc0 comp rx=0 tx=0).stop 2023-05-22T15:44:49.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.545+0000 7f8cbf26e640 1 -- 172.21.15.16:0/3127060238 shutdown_connections 2023-05-22T15:44:49.549 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.545+0000 7f8cbf26e640 1 --2- 172.21.15.16:0/3127060238 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8cb815fc50 0x7f8cb815e2a0 unknown :-1 s=CLOSED pgs=1540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:49.549 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.545+0000 7f8cbf26e640 1 --2- 172.21.15.16:0/3127060238 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8cb815cd70 0x7f8cb8154ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:49.549 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.545+0000 7f8cbf26e640 1 --2- 172.21.15.16:0/3127060238 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f8c9c0655c0 0x7f8c9c067a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:49.549 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.545+0000 7f8cbf26e640 1 --2- 172.21.15.16:0/3127060238 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8cb815c920 0x7f8cb814ebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:49.549 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.545+0000 7f8cbf26e640 1 -- 172.21.15.16:0/3127060238 >> 172.21.15.16:0/3127060238 conn(0x7f8cb80657f0 msgr2=0x7f8cb80b7470 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:44:49.549 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.545+0000 7f8cbf26e640 1 -- 172.21.15.16:0/3127060238 shutdown_connections 2023-05-22T15:44:49.549 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.545+0000 7f8cbf26e640 1 -- 172.21.15.16:0/3127060238 wait complete. 2023-05-22T15:44:49.562 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:44:49.558+0000 7f8f9a67d180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-05-22T15:44:49.564 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-05-22T15:26:38.019781+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-05-22T15:26:43.570514+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 14, 'pool_name': 'cephfs_metadata', 'create_time': '2023-05-22T15:43:28.684367+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '117', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 15, 'pool_name': 'cephfs_data', 'create_time': '2023-05-22T15:43:29.119294+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '118', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-05-22T15:44:49.565 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-05-22T15:44:49.625 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:44:49.621+0000 7f4d405c0180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-05-22T15:44:49.639 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:44:49.634+0000 7f8f9a67d180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-05-22T15:44:49.683 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.681+0000 7f79d8862640 1 Processor -- start 2023-05-22T15:44:49.683 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.681+0000 7f79d8862640 1 -- start start 2023-05-22T15:44:49.685 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.681+0000 7f79d8862640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f79d4071640 0x7f79d4160090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:49.685 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.681+0000 7f79d8862640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f79d41605d0 0x7f79d415c270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:49.686 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.681+0000 7f79d8862640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79d4069ae0 con 0x7f79d40ce480 2023-05-22T15:44:49.686 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.681+0000 7f79d8862640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79d415f990 con 0x7f79d41605d0 2023-05-22T15:44:49.686 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.681+0000 7f79d8862640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79d4071a90 con 0x7f79d4071640 2023-05-22T15:44:49.686 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.681+0000 7f79d259c640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f79d40ce480 0x7f79d40ce850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:51990/0 (socket says 172.21.15.16:51990) 2023-05-22T15:44:49.686 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.681+0000 7f79d259c640 1 -- 172.21.15.16:0/3209021162 learned_addr learned my addr 172.21.15.16:0/3209021162 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:44:49.686 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.681+0000 7f79d1d9b640 1 --2- 172.21.15.16:0/3209021162 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f79d4071640 0x7f79d4160090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:44:49.686 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.681+0000 7f79d2d9d640 1 --2- 172.21.15.16:0/3209021162 >> v2:172.21.15.62:3300/0 conn(0x7f79d41605d0 0x7f79d415c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:44:49.687 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.681+0000 7f79d159a640 1 -- 172.21.15.16:0/3209021162 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1326338664 0 0) 0x7f79d4069ae0 con 0x7f79d40ce480 2023-05-22T15:44:49.687 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.681+0000 7f79d159a640 1 -- 172.21.15.16:0/3209021162 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f79bc003530 con 0x7f79d40ce480 2023-05-22T15:44:49.687 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.681+0000 7f79d159a640 1 -- 172.21.15.16:0/3209021162 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2025894774 0 0) 0x7f79bc003530 con 0x7f79d40ce480 2023-05-22T15:44:49.687 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.681+0000 7f79d159a640 1 -- 172.21.15.16:0/3209021162 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f79d4071640 msgr2=0x7f79d4160090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:44:49.687 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.681+0000 7f79d159a640 1 --2- 172.21.15.16:0/3209021162 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f79d4071640 0x7f79d4160090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:49.687 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.681+0000 7f79d159a640 1 -- 172.21.15.16:0/3209021162 >> v2:172.21.15.62:3300/0 conn(0x7f79d41605d0 msgr2=0x7f79d415c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:44:49.687 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.681+0000 7f79d159a640 1 --2- 172.21.15.16:0/3209021162 >> v2:172.21.15.62:3300/0 conn(0x7f79d41605d0 0x7f79d415c270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:49.688 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.681+0000 7f79d159a640 1 -- 172.21.15.16:0/3209021162 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79d4069dc0 con 0x7f79d40ce480 2023-05-22T15:44:49.688 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.681+0000 7f79d159a640 1 -- 172.21.15.16:0/3209021162 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7f79c4001680 con 0x7f79d40ce480 2023-05-22T15:44:49.688 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.681+0000 7f79d159a640 1 -- 172.21.15.16:0/3209021162 >> v1:172.21.15.16:6789/0 conn(0x7f79d40ce480 legacy=0x7f79d40ce850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:44:49.688 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.681+0000 7f79d159a640 1 --2- 172.21.15.16:0/3209021162 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f79bc003980 0x7f79bc003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:49.688 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.681+0000 7f79d159a640 1 --2- 172.21.15.16:0/3209021162 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f79d4071640 0x7f79bc0047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:49.688 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.681+0000 7f79d159a640 1 --2- 172.21.15.16:0/3209021162 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f79bc004d10 0x7f79bc005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:49.688 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.681+0000 7f79d159a640 1 -- 172.21.15.16:0/3209021162 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79d415f990 con 0x7f79bc003980 2023-05-22T15:44:49.689 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.681+0000 7f79d159a640 1 -- 172.21.15.16:0/3209021162 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79d4071a90 con 0x7f79bc004d10 2023-05-22T15:44:49.689 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.681+0000 7f79d159a640 1 -- 172.21.15.16:0/3209021162 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79bc00b990 con 0x7f79d4071640 2023-05-22T15:44:49.689 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.681+0000 7f79d259c640 1 --2- 172.21.15.16:0/3209021162 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f79bc003980 0x7f79bc003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:44:49.689 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.681+0000 7f79d1d9b640 1 --2- 172.21.15.16:0/3209021162 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f79d4071640 0x7f79bc0047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:44:49.689 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.681+0000 7f79d2d9d640 1 --2- 172.21.15.16:0/3209021162 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f79bc004d10 0x7f79bc005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:44:49.689 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.681+0000 7f79d259c640 1 -- 172.21.15.16:0/3209021162 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f79d4071640 msgr2=0x7f79bc0047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:44:49.689 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.681+0000 7f79d259c640 1 --2- 172.21.15.16:0/3209021162 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f79d4071640 0x7f79bc0047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:49.689 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.681+0000 7f79d259c640 1 -- 172.21.15.16:0/3209021162 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f79bc004d10 msgr2=0x7f79bc005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:44:49.690 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.681+0000 7f79d259c640 1 --2- 172.21.15.16:0/3209021162 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f79bc004d10 0x7f79bc005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:49.690 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.681+0000 7f79d259c640 1 -- 172.21.15.16:0/3209021162 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79bc003530 con 0x7f79bc003980 2023-05-22T15:44:49.690 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.681+0000 7f79d1d9b640 1 --2- 172.21.15.16:0/3209021162 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f79d4071640 0x7f79bc0047d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T15:44:49.690 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.681+0000 7f79d2d9d640 1 --2- 172.21.15.16:0/3209021162 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f79bc004d10 0x7f79bc005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:44:49.692 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.685+0000 7f79d259c640 1 --2- 172.21.15.16:0/3209021162 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f79bc003980 0x7f79bc003d90 secure :-1 s=READY pgs=1542 cs=0 l=1 rev1=1 crypto rx=0x7f79c400c290 tx=0x7f79c401cc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bed4c01b87d0ad08 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:44:49.693 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.685+0000 7f79d159a640 1 -- 172.21.15.16:0/3209021162 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79c40058e0 con 0x7f79bc003980 2023-05-22T15:44:49.693 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.685+0000 7f79d159a640 1 -- 172.21.15.16:0/3209021162 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79c4005a40 con 0x7f79bc003980 2023-05-22T15:44:49.693 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.685+0000 7f79d159a640 1 -- 172.21.15.16:0/3209021162 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79c4005df0 con 0x7f79bc003980 2023-05-22T15:44:49.693 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.685+0000 7f79d8862640 1 -- 172.21.15.16:0/3209021162 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f79bc003980 msgr2=0x7f79bc003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:44:49.694 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.685+0000 7f79d8862640 1 --2- 172.21.15.16:0/3209021162 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f79bc003980 0x7f79bc003d90 secure :-1 s=READY pgs=1542 cs=0 l=1 rev1=1 crypto rx=0x7f79c400c290 tx=0x7f79c401cc50 comp rx=0 tx=0).stop 2023-05-22T15:44:49.694 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.685+0000 7f79d2d9d640 1 -- 172.21.15.16:0/3209021162 reap_dead start 2023-05-22T15:44:49.694 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.685+0000 7f79d8862640 1 -- 172.21.15.16:0/3209021162 shutdown_connections 2023-05-22T15:44:49.694 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.685+0000 7f79d8862640 1 -- 172.21.15.16:0/3209021162 >> 172.21.15.16:0/3209021162 conn(0x7f79d40cc870 msgr2=0x7f79d40cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:44:49.694 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.685+0000 7f79d8862640 1 -- 172.21.15.16:0/3209021162 shutdown_connections 2023-05-22T15:44:49.694 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.685+0000 7f79d8862640 1 -- 172.21.15.16:0/3209021162 wait complete. 2023-05-22T15:44:49.694 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.685+0000 7f79d8862640 1 Processor -- start 2023-05-22T15:44:49.694 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.685+0000 7f79d8862640 1 -- start start 2023-05-22T15:44:49.695 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.685+0000 7f79d8862640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f79bc003980 0x7f79d415df40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:49.695 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.685+0000 7f79d8862640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f79d40c72f0 0x7f79d413ff50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:49.695 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.685+0000 7f79d8862640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f79d40c7760 0x7f79d4140490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:49.695 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.685+0000 7f79d8862640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79d41409d0 con 0x7f79d40c7760 2023-05-22T15:44:49.696 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.685+0000 7f79d8862640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79d414d6d0 con 0x7f79bc003980 2023-05-22T15:44:49.696 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.685+0000 7f79d8862640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79d415cbc0 con 0x7f79d40c72f0 2023-05-22T15:44:49.696 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.685+0000 7f79d2d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f79d40c7760 0x7f79d4140490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:44:49.696 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.685+0000 7f79d2d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f79d40c7760 0x7f79d4140490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59060/0 (socket says 172.21.15.16:59060) 2023-05-22T15:44:49.696 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.685+0000 7f79d2d9d640 1 -- 172.21.15.16:0/4292491982 learned_addr learned my addr 172.21.15.16:0/4292491982 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:44:49.696 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.685+0000 7f79d1d9b640 1 --2- 172.21.15.16:0/4292491982 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f79d40c72f0 0x7f79d413ff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:44:49.697 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.685+0000 7f79d259c640 1 --2- 172.21.15.16:0/4292491982 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f79bc003980 0x7f79d415df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:44:49.697 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.689+0000 7f79d2d9d640 1 -- 172.21.15.16:0/4292491982 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f79d40c72f0 msgr2=0x7f79d413ff50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:44:49.697 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.689+0000 7f79d2d9d640 1 --2- 172.21.15.16:0/4292491982 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f79d40c72f0 0x7f79d413ff50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:49.697 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.689+0000 7f79d2d9d640 1 -- 172.21.15.16:0/4292491982 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f79bc003980 msgr2=0x7f79d415df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:44:49.697 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.689+0000 7f79d2d9d640 1 --2- 172.21.15.16:0/4292491982 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f79bc003980 0x7f79d415df40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:49.697 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.689+0000 7f79d2d9d640 1 -- 172.21.15.16:0/4292491982 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79d40698b0 con 0x7f79d40c7760 2023-05-22T15:44:49.697 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.689+0000 7f79d2d9d640 1 --2- 172.21.15.16:0/4292491982 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f79d40c7760 0x7f79d4140490 secure :-1 s=READY pgs=1543 cs=0 l=1 rev1=1 crypto rx=0x7f79d40cd850 tx=0x7f79cc00f150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:44:49.697 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.689+0000 7f79c2ffd640 1 -- 172.21.15.16:0/4292491982 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79cc00f910 con 0x7f79d40c7760 2023-05-22T15:44:49.698 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.689+0000 7f79c2ffd640 1 -- 172.21.15.16:0/4292491982 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79cc00fa90 con 0x7f79d40c7760 2023-05-22T15:44:49.698 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.689+0000 7f79d8862640 1 -- 172.21.15.16:0/4292491982 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79d4069720 con 0x7f79d40c7760 2023-05-22T15:44:49.698 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.689+0000 7f79c2ffd640 1 -- 172.21.15.16:0/4292491982 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79cc00fe40 con 0x7f79d40c7760 2023-05-22T15:44:49.698 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.689+0000 7f79d8862640 1 -- 172.21.15.16:0/4292491982 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79d40c5e70 con 0x7f79d40c7760 2023-05-22T15:44:49.698 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.689+0000 7f79d8862640 1 -- 172.21.15.16:0/4292491982 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79d404dbe0 con 0x7f79d40c7760 2023-05-22T15:44:49.698 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.689+0000 7f79c2ffd640 1 -- 172.21.15.16:0/4292491982 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 47) v1 ==== 84446+0+0 (secure 0 0 0) 0x7f79cc016420 con 0x7f79d40c7760 2023-05-22T15:44:49.698 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.689+0000 7f79c2ffd640 1 --2- 172.21.15.16:0/4292491982 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f79b40655c0 0x7f79b4067a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:49.699 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.689+0000 7f79d259c640 1 -- 172.21.15.16:0/4292491982 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f79b40655c0 msgr2=0x7f79b4067a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/36921 2023-05-22T15:44:49.699 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.689+0000 7f79c2ffd640 1 -- 172.21.15.16:0/4292491982 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(119..119 src has 1..119) v4 ==== 6726+0+0 (secure 0 0 0) 0x7f79cc0835d0 con 0x7f79d40c7760 2023-05-22T15:44:49.699 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.689+0000 7f79d259c640 1 --2- 172.21.15.16:0/4292491982 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f79b40655c0 0x7f79b4067a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-05-22T15:44:49.700 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.697+0000 7f79c2ffd640 1 -- 172.21.15.16:0/4292491982 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f79cc050ae0 con 0x7f79d40c7760 2023-05-22T15:44:49.707 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:44:49.705+0000 7f4d405c0180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-05-22T15:44:49.856 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.853+0000 7f79d8862640 1 -- 172.21.15.16:0/4292491982 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f79d4070a10 con 0x7f79d40c7760 2023-05-22T15:44:49.857 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.853+0000 7f79c2ffd640 1 -- 172.21.15.16:0/4292491982 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v119) v1 ==== 93+0+10 (secure 0 0 0) 0x7f79cc054100 con 0x7f79d40c7760 2023-05-22T15:44:49.857 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2023-05-22T15:44:49.859 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.857+0000 7f79d8862640 1 -- 172.21.15.16:0/4292491982 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f79b40655c0 msgr2=0x7f79b4067a70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-05-22T15:44:49.859 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.857+0000 7f79d8862640 1 --2- 172.21.15.16:0/4292491982 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f79b40655c0 0x7f79b4067a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:49.859 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.857+0000 7f79d8862640 1 -- 172.21.15.16:0/4292491982 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f79d40c7760 msgr2=0x7f79d4140490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:44:49.859 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.857+0000 7f79d8862640 1 --2- 172.21.15.16:0/4292491982 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f79d40c7760 0x7f79d4140490 secure :-1 s=READY pgs=1543 cs=0 l=1 rev1=1 crypto rx=0x7f79d40cd850 tx=0x7f79cc00f150 comp rx=0 tx=0).stop 2023-05-22T15:44:49.859 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.857+0000 7f79d8862640 1 -- 172.21.15.16:0/4292491982 shutdown_connections 2023-05-22T15:44:49.860 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.857+0000 7f79d8862640 1 --2- 172.21.15.16:0/4292491982 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f79d40c7760 0x7f79d4140490 unknown :-1 s=CLOSED pgs=1543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:49.860 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.857+0000 7f79d8862640 1 --2- 172.21.15.16:0/4292491982 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f79b40655c0 0x7f79b4067a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:49.860 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.857+0000 7f79d8862640 1 --2- 172.21.15.16:0/4292491982 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f79d40c72f0 0x7f79d413ff50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:49.860 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.857+0000 7f79d8862640 1 --2- 172.21.15.16:0/4292491982 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f79bc003980 0x7f79d415df40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:49.860 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.857+0000 7f79d8862640 1 -- 172.21.15.16:0/4292491982 >> 172.21.15.16:0/4292491982 conn(0x7f79d40cc870 msgr2=0x7f79d415b850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:44:49.860 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.857+0000 7f79d8862640 1 -- 172.21.15.16:0/4292491982 shutdown_connections 2023-05-22T15:44:49.860 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.857+0000 7f79d8862640 1 -- 172.21.15.16:0/4292491982 wait complete. 2023-05-22T15:44:49.875 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-05-22T15:44:49.890 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:44:49.886+0000 7f8f9a67d180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-05-22T15:44:49.944 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:44:49.942+0000 7f8f9a67d180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-05-22T15:44:49.963 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:44:49.961+0000 7f4d405c0180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-05-22T15:44:49.990 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.985+0000 7fc588365640 1 Processor -- start 2023-05-22T15:44:49.991 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.989+0000 7fc588365640 1 -- start start 2023-05-22T15:44:49.993 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.989+0000 7fc588365640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fc580158a10 0x7fc58006e480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:49.993 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.989+0000 7fc588365640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc580158de0 con 0x7fc58006cdf0 2023-05-22T15:44:49.993 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.989+0000 7fc588365640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5800711e0 con 0x7fc580070760 2023-05-22T15:44:49.993 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.989+0000 7fc588365640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc58006e060 con 0x7fc580158a10 2023-05-22T15:44:49.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.989+0000 7fc586902640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fc58006cdf0 0x7fc5800c9a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:51992/0 (socket says 172.21.15.16:51992) 2023-05-22T15:44:49.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.989+0000 7fc586902640 1 -- 172.21.15.16:0/2866043448 learned_addr learned my addr 172.21.15.16:0/2866043448 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:44:49.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.989+0000 7fc586101640 1 --2- 172.21.15.16:0/2866043448 >> v2:172.21.15.62:3300/0 conn(0x7fc580158a10 0x7fc58006e480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:44:49.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.989+0000 7fc5850ff640 1 -- 172.21.15.16:0/2866043448 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3818839592 0 0) 0x7fc580158de0 con 0x7fc58006cdf0 2023-05-22T15:44:49.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.989+0000 7fc5850ff640 1 -- 172.21.15.16:0/2866043448 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc56c003430 con 0x7fc58006cdf0 2023-05-22T15:44:49.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.989+0000 7fc5850ff640 1 -- 172.21.15.16:0/2866043448 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3377469345 0 0) 0x7fc56c003430 con 0x7fc58006cdf0 2023-05-22T15:44:49.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.989+0000 7fc5850ff640 1 -- 172.21.15.16:0/2866043448 >> v2:172.21.15.62:3300/0 conn(0x7fc580158a10 msgr2=0x7fc58006e480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:44:49.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.989+0000 7fc5850ff640 1 --2- 172.21.15.16:0/2866043448 >> v2:172.21.15.62:3300/0 conn(0x7fc580158a10 0x7fc58006e480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:49.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.989+0000 7fc5850ff640 1 -- 172.21.15.16:0/2866043448 >> v1:172.21.15.62:6789/0 conn(0x7fc580070760 legacy=0x7fc58006c6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:44:49.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.989+0000 7fc5850ff640 1 -- 172.21.15.16:0/2866043448 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5800714c0 con 0x7fc58006cdf0 2023-05-22T15:44:49.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.989+0000 7fc5850ff640 1 -- 172.21.15.16:0/2866043448 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7fc578001a00 con 0x7fc58006cdf0 2023-05-22T15:44:49.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.989+0000 7fc5850ff640 1 -- 172.21.15.16:0/2866043448 >> v1:172.21.15.16:6789/0 conn(0x7fc58006cdf0 legacy=0x7fc5800c9a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:44:49.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.989+0000 7fc5850ff640 1 --2- 172.21.15.16:0/2866043448 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc56c004a60 0x7fc56c004e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:49.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.989+0000 7fc5850ff640 1 --2- 172.21.15.16:0/2866043448 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc56c003670 0x7fc56c003ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:49.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.989+0000 7fc5850ff640 1 --2- 172.21.15.16:0/2866043448 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc56c004220 0x7fc56c00fbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:49.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.989+0000 7fc5850ff640 1 -- 172.21.15.16:0/2866043448 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc58006e060 con 0x7fc56c004a60 2023-05-22T15:44:49.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.989+0000 7fc5850ff640 1 -- 172.21.15.16:0/2866043448 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc56c0104a0 con 0x7fc56c003670 2023-05-22T15:44:49.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.989+0000 7fc5850ff640 1 -- 172.21.15.16:0/2866043448 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc56c010660 con 0x7fc56c004220 2023-05-22T15:44:49.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.989+0000 7fc586101640 1 --2- 172.21.15.16:0/2866043448 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc56c004a60 0x7fc56c004e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:44:49.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.989+0000 7fc586902640 1 --2- 172.21.15.16:0/2866043448 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc56c004220 0x7fc56c00fbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:44:49.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.989+0000 7fc586101640 1 -- 172.21.15.16:0/2866043448 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc56c004220 msgr2=0x7fc56c00fbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:44:49.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.989+0000 7fc586101640 1 --2- 172.21.15.16:0/2866043448 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc56c004220 0x7fc56c00fbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:49.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.989+0000 7fc586101640 1 -- 172.21.15.16:0/2866043448 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc56c003670 msgr2=0x7fc56c003ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:44:49.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.989+0000 7fc585900640 1 --2- 172.21.15.16:0/2866043448 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc56c003670 0x7fc56c003ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:44:49.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.989+0000 7fc586101640 1 --2- 172.21.15.16:0/2866043448 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc56c003670 0x7fc56c003ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:49.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.989+0000 7fc586902640 1 -- 172.21.15.16:0/2866043448 reap_dead start 2023-05-22T15:44:49.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.989+0000 7fc586101640 1 -- 172.21.15.16:0/2866043448 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc56c003430 con 0x7fc56c004a60 2023-05-22T15:44:49.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.989+0000 7fc585900640 1 --2- 172.21.15.16:0/2866043448 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc56c003670 0x7fc56c003ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:44:49.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.993+0000 7fc586101640 1 --2- 172.21.15.16:0/2866043448 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc56c004a60 0x7fc56c004e70 secure :-1 s=READY pgs=1545 cs=0 l=1 rev1=1 crypto rx=0x7fc57c009ae0 tx=0x7fc57c01ca70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b9b6537ffb84ec0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:44:49.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.993+0000 7fc5850ff640 1 -- 172.21.15.16:0/2866043448 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc57c0a5550 con 0x7fc56c004a60 2023-05-22T15:44:50.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.993+0000 7fc5850ff640 1 -- 172.21.15.16:0/2866043448 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc57c0a56b0 con 0x7fc56c004a60 2023-05-22T15:44:50.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.993+0000 7fc5850ff640 1 -- 172.21.15.16:0/2866043448 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc57c0a5a60 con 0x7fc56c004a60 2023-05-22T15:44:50.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.993+0000 7fc588365640 1 -- 172.21.15.16:0/2866043448 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc56c004a60 msgr2=0x7fc56c004e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:44:50.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.993+0000 7fc588365640 1 --2- 172.21.15.16:0/2866043448 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc56c004a60 0x7fc56c004e70 secure :-1 s=READY pgs=1545 cs=0 l=1 rev1=1 crypto rx=0x7fc57c009ae0 tx=0x7fc57c01ca70 comp rx=0 tx=0).stop 2023-05-22T15:44:50.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.993+0000 7fc588365640 1 -- 172.21.15.16:0/2866043448 shutdown_connections 2023-05-22T15:44:50.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.993+0000 7fc588365640 1 --2- 172.21.15.16:0/2866043448 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc56c004a60 0x7fc56c004e70 unknown :-1 s=CLOSED pgs=1545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:50.001 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.993+0000 7fc588365640 1 -- 172.21.15.16:0/2866043448 >> 172.21.15.16:0/2866043448 conn(0x7fc5800cc870 msgr2=0x7fc5800cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:44:50.001 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.993+0000 7fc588365640 1 -- 172.21.15.16:0/2866043448 shutdown_connections 2023-05-22T15:44:50.001 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.993+0000 7fc588365640 1 -- 172.21.15.16:0/2866043448 wait complete. 2023-05-22T15:44:50.001 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.993+0000 7fc588365640 1 Processor -- start 2023-05-22T15:44:50.001 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.993+0000 7fc588365640 1 -- start start 2023-05-22T15:44:50.001 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.993+0000 7fc588365640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc56c004a60 0x7fc5800afbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:50.001 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.993+0000 7fc588365640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc580151ed0 0x7fc580151300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:50.001 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.993+0000 7fc588365640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc580151840 0x7fc5800b2240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:50.002 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.993+0000 7fc588365640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc580151cb0 con 0x7fc580151ed0 2023-05-22T15:44:50.002 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.993+0000 7fc588365640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5800b2fd0 con 0x7fc56c004a60 2023-05-22T15:44:50.002 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.993+0000 7fc588365640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5800b32b0 con 0x7fc580151840 2023-05-22T15:44:50.002 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.997+0000 7fc585900640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc580151ed0 0x7fc580151300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:44:50.002 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.997+0000 7fc585900640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc580151ed0 0x7fc580151300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59086/0 (socket says 172.21.15.16:59086) 2023-05-22T15:44:50.002 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.997+0000 7fc585900640 1 -- 172.21.15.16:0/3091932604 learned_addr learned my addr 172.21.15.16:0/3091932604 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:44:50.002 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.997+0000 7fc586101640 1 --2- 172.21.15.16:0/3091932604 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc56c004a60 0x7fc5800afbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:44:50.002 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.997+0000 7fc586902640 1 --2- 172.21.15.16:0/3091932604 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc580151840 0x7fc5800b2240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:44:50.003 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.997+0000 7fc585900640 1 -- 172.21.15.16:0/3091932604 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc580151840 msgr2=0x7fc5800b2240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:44:50.003 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.997+0000 7fc585900640 1 --2- 172.21.15.16:0/3091932604 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc580151840 0x7fc5800b2240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:50.003 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.997+0000 7fc585900640 1 -- 172.21.15.16:0/3091932604 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc56c004a60 msgr2=0x7fc5800afbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:44:50.003 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.997+0000 7fc585900640 1 --2- 172.21.15.16:0/3091932604 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc56c004a60 0x7fc5800afbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:50.003 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.997+0000 7fc585900640 1 -- 172.21.15.16:0/3091932604 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5800c9a30 con 0x7fc580151ed0 2023-05-22T15:44:50.003 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.997+0000 7fc586101640 1 --2- 172.21.15.16:0/3091932604 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc56c004a60 0x7fc5800afbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T15:44:50.003 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.997+0000 7fc585900640 1 --2- 172.21.15.16:0/3091932604 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc580151ed0 0x7fc580151300 secure :-1 s=READY pgs=1546 cs=0 l=1 rev1=1 crypto rx=0x7fc57000d4a0 tx=0x7fc57000d590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:44:50.004 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.997+0000 7fc576ffd640 1 -- 172.21.15.16:0/3091932604 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc57000b070 con 0x7fc580151ed0 2023-05-22T15:44:50.004 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.997+0000 7fc576ffd640 1 -- 172.21.15.16:0/3091932604 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc570005330 con 0x7fc580151ed0 2023-05-22T15:44:50.004 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.997+0000 7fc576ffd640 1 -- 172.21.15.16:0/3091932604 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc570007270 con 0x7fc580151ed0 2023-05-22T15:44:50.004 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.997+0000 7fc588365640 1 -- 172.21.15.16:0/3091932604 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc58006caf0 con 0x7fc580151ed0 2023-05-22T15:44:50.004 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.997+0000 7fc588365640 1 -- 172.21.15.16:0/3091932604 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5800c9bc0 con 0x7fc580151ed0 2023-05-22T15:44:50.005 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.997+0000 7fc588365640 1 -- 172.21.15.16:0/3091932604 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc58004dbe0 con 0x7fc580151ed0 2023-05-22T15:44:50.005 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.997+0000 7fc576ffd640 1 -- 172.21.15.16:0/3091932604 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 47) v1 ==== 84446+0+0 (secure 0 0 0) 0x7fc5700073d0 con 0x7fc580151ed0 2023-05-22T15:44:50.005 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.997+0000 7fc576ffd640 1 --2- 172.21.15.16:0/3091932604 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fc5640655f0 0x7fc564067aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:50.005 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.997+0000 7fc586101640 1 -- 172.21.15.16:0/3091932604 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fc5640655f0 msgr2=0x7fc564067aa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/36921 2023-05-22T15:44:50.005 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.997+0000 7fc576ffd640 1 -- 172.21.15.16:0/3091932604 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(119..119 src has 1..119) v4 ==== 6726+0+0 (secure 0 0 0) 0x7fc570083080 con 0x7fc580151ed0 2023-05-22T15:44:50.006 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:49.997+0000 7fc586101640 1 --2- 172.21.15.16:0/3091932604 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fc5640655f0 0x7fc564067aa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-05-22T15:44:50.006 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.001+0000 7fc576ffd640 1 -- 172.21.15.16:0/3091932604 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fc5700541c0 con 0x7fc580151ed0 2023-05-22T15:44:50.021 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:44:50.017+0000 7f4d405c0180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-05-22T15:44:50.056 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.049+0000 7fc576ffd640 1 -- 172.21.15.16:0/3091932604 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mgrmap(e 48) v1 ==== 83941+0+0 (secure 0 0 0) 0x7fc570050b60 con 0x7fc580151ed0 2023-05-22T15:44:50.057 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.053+0000 7fc576ffd640 1 -- 172.21.15.16:0/3091932604 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fc5640655f0 msgr2=0x7fc564067aa0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-05-22T15:44:50.057 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.053+0000 7fc576ffd640 1 --2- 172.21.15.16:0/3091932604 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fc5640655f0 0x7fc564067aa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:50.121 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:44:50.117+0000 7f4ccc752640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T15:44:50.121 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:44:50.117+0000 7f4ccc752640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T15:44:50.122 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:44:50.117+0000 7f4ccc752640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T15:44:50.122 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:44:50.117+0000 7f4ccc752640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T15:44:50.122 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:44:50.117+0000 7f4ccc752640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T15:44:50.122 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:44:50.121+0000 7f4cc874a640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T15:44:50.123 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:44:50.121+0000 7f4cc874a640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T15:44:50.123 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:44:50.121+0000 7f4cc874a640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T15:44:50.123 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:44:50.121+0000 7f4cc874a640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T15:44:50.123 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:44:50.121+0000 7f4cc874a640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T15:44:50.188 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.185+0000 7fc588365640 1 -- 172.21.15.16:0/3091932604 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fc5800b50e0 con 0x7fc580151ed0 2023-05-22T15:44:50.189 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.185+0000 7fc576ffd640 1 -- 172.21.15.16:0/3091932604 <== mon.0 v2:172.21.15.16:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v120) v1 ==== 92+0+10 (secure 0 0 0) 0x7fc5700617a0 con 0x7fc580151ed0 2023-05-22T15:44:50.189 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 8 2023-05-22T15:44:50.191 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.189+0000 7fc588365640 1 -- 172.21.15.16:0/3091932604 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc580151ed0 msgr2=0x7fc580151300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:44:50.191 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.189+0000 7fc588365640 1 --2- 172.21.15.16:0/3091932604 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc580151ed0 0x7fc580151300 secure :-1 s=READY pgs=1546 cs=0 l=1 rev1=1 crypto rx=0x7fc57000d4a0 tx=0x7fc57000d590 comp rx=0 tx=0).stop 2023-05-22T15:44:50.191 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.189+0000 7fc588365640 1 -- 172.21.15.16:0/3091932604 shutdown_connections 2023-05-22T15:44:50.191 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.189+0000 7fc588365640 1 --2- 172.21.15.16:0/3091932604 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fc5640655f0 0x7fc564067aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:50.191 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.189+0000 7fc588365640 1 --2- 172.21.15.16:0/3091932604 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc580151840 0x7fc5800b2240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:50.191 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.189+0000 7fc588365640 1 --2- 172.21.15.16:0/3091932604 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc580151ed0 0x7fc580151300 unknown :-1 s=CLOSED pgs=1546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:50.191 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.189+0000 7fc588365640 1 --2- 172.21.15.16:0/3091932604 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc56c004a60 0x7fc5800afbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:50.192 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.189+0000 7fc588365640 1 -- 172.21.15.16:0/3091932604 >> 172.21.15.16:0/3091932604 conn(0x7fc5800cc870 msgr2=0x7fc5800aebc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:44:50.192 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.189+0000 7fc588365640 1 -- 172.21.15.16:0/3091932604 shutdown_connections 2023-05-22T15:44:50.192 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.189+0000 7fc588365640 1 -- 172.21.15.16:0/3091932604 wait complete. 2023-05-22T15:44:50.206 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-05-22T15:44:50.323 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.321+0000 7f5b25a5d640 1 Processor -- start 2023-05-22T15:44:50.323 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.321+0000 7f5b25a5d640 1 -- start start 2023-05-22T15:44:50.324 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.321+0000 7f5b25a5d640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f5b2015bee0 0x7f5b2006b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:50.324 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.321+0000 7f5b25a5d640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b200c8660 con 0x7f5b200cb730 2023-05-22T15:44:50.324 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.321+0000 7f5b25a5d640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b20160670 con 0x7f5b200c9b00 2023-05-22T15:44:50.324 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.321+0000 7f5b25a5d640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b200ce4a0 con 0x7f5b2015bee0 2023-05-22T15:44:50.324 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.321+0000 7f5b1effd640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f5b2015bee0 0x7f5b2006b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:44:50.325 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.321+0000 7f5b1e7fc640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f5b200cb730 0x7f5b2015f850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:51998/0 (socket says 172.21.15.16:51998) 2023-05-22T15:44:50.325 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.321+0000 7f5b1e7fc640 1 -- 172.21.15.16:0/2288606832 learned_addr learned my addr 172.21.15.16:0/2288606832 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:44:50.325 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.321+0000 7f5b1effd640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f5b2015bee0 0x7f5b2006b6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59226/0 (socket says 172.21.15.16:59226) 2023-05-22T15:44:50.326 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.321+0000 7f5b1effd640 1 -- 172.21.15.16:0/2288606832 >> v1:172.21.15.62:6789/0 conn(0x7f5b200c9b00 legacy=0x7f5b2015ff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:44:50.326 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.321+0000 7f5b1effd640 1 -- 172.21.15.16:0/2288606832 >> v1:172.21.15.16:6789/0 conn(0x7f5b200cb730 legacy=0x7f5b2015f850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:44:50.326 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.321+0000 7f5b1effd640 1 -- 172.21.15.16:0/2288606832 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b200ce780 con 0x7f5b2015bee0 2023-05-22T15:44:50.327 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.321+0000 7f5b1dffb640 1 -- 172.21.15.16:0/2288606832 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 238842772 0 0) 0x7f5b200c8660 con 0x7f5b200cb730 2023-05-22T15:44:50.327 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.325+0000 7f5b1effd640 1 --2- 172.21.15.16:0/2288606832 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b2015bee0 0x7f5b2006b6c0 secure :-1 s=READY pgs=1559 cs=0 l=1 rev1=1 crypto rx=0x7f5b100097c0 tx=0x7f5b1001c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cbd47a872621e197 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:44:50.328 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.325+0000 7f5b1dffb640 1 -- 172.21.15.16:0/2288606832 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b100ab650 con 0x7f5b2015bee0 2023-05-22T15:44:50.328 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.325+0000 7f5b1dffb640 1 -- 172.21.15.16:0/2288606832 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b100ab7b0 con 0x7f5b2015bee0 2023-05-22T15:44:50.328 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.325+0000 7f5b1dffb640 1 -- 172.21.15.16:0/2288606832 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b100abb40 con 0x7f5b2015bee0 2023-05-22T15:44:50.329 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.325+0000 7f5b25a5d640 1 -- 172.21.15.16:0/2288606832 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b2015bee0 msgr2=0x7f5b2006b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:44:50.329 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.325+0000 7f5b25a5d640 1 --2- 172.21.15.16:0/2288606832 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b2015bee0 0x7f5b2006b6c0 secure :-1 s=READY pgs=1559 cs=0 l=1 rev1=1 crypto rx=0x7f5b100097c0 tx=0x7f5b1001c450 comp rx=0 tx=0).stop 2023-05-22T15:44:50.329 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.325+0000 7f5b25a5d640 1 -- 172.21.15.16:0/2288606832 shutdown_connections 2023-05-22T15:44:50.329 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.325+0000 7f5b25a5d640 1 --2- 172.21.15.16:0/2288606832 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b2015bee0 0x7f5b2006b6c0 unknown :-1 s=CLOSED pgs=1559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:50.329 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.325+0000 7f5b25a5d640 1 -- 172.21.15.16:0/2288606832 >> 172.21.15.16:0/2288606832 conn(0x7f5b20069720 msgr2=0x7f5b20069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:44:50.329 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.325+0000 7f5b25a5d640 1 -- 172.21.15.16:0/2288606832 shutdown_connections 2023-05-22T15:44:50.329 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.325+0000 7f5b25a5d640 1 -- 172.21.15.16:0/2288606832 wait complete. 2023-05-22T15:44:50.330 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.325+0000 7f5b25a5d640 1 Processor -- start 2023-05-22T15:44:50.330 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.325+0000 7f5b25a5d640 1 -- start start 2023-05-22T15:44:50.332 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.329+0000 7f5b25a5d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b200c9b00 0x7f5b20146b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:50.332 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.329+0000 7f5b25a5d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5b200cb730 0x7f5b200c2f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:50.332 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.329+0000 7f5b25a5d640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5b2015bee0 0x7f5b20070020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:50.332 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.329+0000 7f5b25a5d640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b20147090 con 0x7f5b200c9b00 2023-05-22T15:44:50.332 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.329+0000 7f5b25a5d640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b200c4010 con 0x7f5b200cb730 2023-05-22T15:44:50.333 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.329+0000 7f5b25a5d640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b200c3770 con 0x7f5b2015bee0 2023-05-22T15:44:50.333 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.329+0000 7f5b1effd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b200c9b00 0x7f5b20146b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:44:50.333 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.329+0000 7f5b1effd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b200c9b00 0x7f5b20146b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59232/0 (socket says 172.21.15.16:59232) 2023-05-22T15:44:50.333 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.329+0000 7f5b1effd640 1 -- 172.21.15.16:0/723775977 learned_addr learned my addr 172.21.15.16:0/723775977 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:44:50.333 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.329+0000 7f5b1e7fc640 1 --2- 172.21.15.16:0/723775977 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5b200cb730 0x7f5b200c2f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:44:50.333 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.329+0000 7f5b1effd640 1 -- 172.21.15.16:0/723775977 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5b2015bee0 msgr2=0x7f5b20070020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:44:50.333 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.329+0000 7f5b1f7fe640 1 --2- 172.21.15.16:0/723775977 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5b2015bee0 0x7f5b20070020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:44:50.333 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.329+0000 7f5b1effd640 1 --2- 172.21.15.16:0/723775977 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5b2015bee0 0x7f5b20070020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:50.334 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.329+0000 7f5b1effd640 1 -- 172.21.15.16:0/723775977 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5b200cb730 msgr2=0x7f5b200c2f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:44:50.334 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.329+0000 7f5b1effd640 1 --2- 172.21.15.16:0/723775977 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5b200cb730 0x7f5b200c2f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:50.334 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.329+0000 7f5b1effd640 1 -- 172.21.15.16:0/723775977 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b10009420 con 0x7f5b200c9b00 2023-05-22T15:44:50.334 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.329+0000 7f5b1e7fc640 1 --2- 172.21.15.16:0/723775977 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5b200cb730 0x7f5b200c2f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:44:50.334 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.329+0000 7f5b1effd640 1 --2- 172.21.15.16:0/723775977 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b200c9b00 0x7f5b20146b50 secure :-1 s=READY pgs=1560 cs=0 l=1 rev1=1 crypto rx=0x7f5b100ad040 tx=0x7f5b1001ac90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:44:50.334 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.329+0000 7f5b07fff640 1 -- 172.21.15.16:0/723775977 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b100b5810 con 0x7f5b200c9b00 2023-05-22T15:44:50.334 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.329+0000 7f5b07fff640 1 -- 172.21.15.16:0/723775977 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b100b5990 con 0x7f5b200c9b00 2023-05-22T15:44:50.335 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.329+0000 7f5b25a5d640 1 -- 172.21.15.16:0/723775977 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b200ce0c0 con 0x7f5b200c9b00 2023-05-22T15:44:50.335 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.329+0000 7f5b07fff640 1 -- 172.21.15.16:0/723775977 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b100b5d40 con 0x7f5b200c9b00 2023-05-22T15:44:50.335 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.329+0000 7f5b25a5d640 1 -- 172.21.15.16:0/723775977 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b2006c010 con 0x7f5b200c9b00 2023-05-22T15:44:50.335 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.329+0000 7f5b25a5d640 1 -- 172.21.15.16:0/723775977 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b200ce250 con 0x7f5b200c9b00 2023-05-22T15:44:50.336 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.329+0000 7f5b07fff640 1 -- 172.21.15.16:0/723775977 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 48) v1 ==== 83941+0+0 (secure 0 0 0) 0x7f5b100bf410 con 0x7f5b200c9b00 2023-05-22T15:44:50.336 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.329+0000 7f5b07fff640 1 -- 172.21.15.16:0/723775977 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 7070+0+0 (secure 0 0 0) 0x7f5b1011d470 con 0x7f5b200c9b00 2023-05-22T15:44:50.341 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.333+0000 7f5b07fff640 1 -- 172.21.15.16:0/723775977 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f5b100eb0f0 con 0x7f5b200c9b00 2023-05-22T15:44:50.486 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.481+0000 7f5b25a5d640 1 -- 172.21.15.16:0/723775977 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f5b2004dbe0 con 0x7f5b200c9b00 2023-05-22T15:44:50.487 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.485+0000 7f5b07fff640 1 -- 172.21.15.16:0/723775977 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v120) v1 ==== 104+0+11 (secure 0 0 0) 0x7f5b100ee710 con 0x7f5b200c9b00 2023-05-22T15:44:50.487 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2023-05-22T15:44:50.489 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.485+0000 7f5b25a5d640 1 -- 172.21.15.16:0/723775977 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b200c9b00 msgr2=0x7f5b20146b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:44:50.489 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.485+0000 7f5b25a5d640 1 --2- 172.21.15.16:0/723775977 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b200c9b00 0x7f5b20146b50 secure :-1 s=READY pgs=1560 cs=0 l=1 rev1=1 crypto rx=0x7f5b100ad040 tx=0x7f5b1001ac90 comp rx=0 tx=0).stop 2023-05-22T15:44:50.489 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.485+0000 7f5b25a5d640 1 -- 172.21.15.16:0/723775977 shutdown_connections 2023-05-22T15:44:50.489 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.485+0000 7f5b25a5d640 1 --2- 172.21.15.16:0/723775977 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5b2015bee0 0x7f5b20070020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:50.490 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.485+0000 7f5b25a5d640 1 --2- 172.21.15.16:0/723775977 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5b200cb730 0x7f5b200c2f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:50.490 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.485+0000 7f5b25a5d640 1 --2- 172.21.15.16:0/723775977 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b200c9b00 0x7f5b20146b50 unknown :-1 s=CLOSED pgs=1560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:50.490 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.485+0000 7f5b25a5d640 1 -- 172.21.15.16:0/723775977 >> 172.21.15.16:0/723775977 conn(0x7f5b20069720 msgr2=0x7f5b200b3230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:44:50.490 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.485+0000 7f5b25a5d640 1 -- 172.21.15.16:0/723775977 shutdown_connections 2023-05-22T15:44:50.490 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.485+0000 7f5b25a5d640 1 -- 172.21.15.16:0/723775977 wait complete. 2023-05-22T15:44:50.505 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-05-22T15:44:50.618 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.617+0000 7fbda6693640 1 Processor -- start 2023-05-22T15:44:50.619 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.617+0000 7fbda6693640 1 -- start start 2023-05-22T15:44:50.620 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.617+0000 7fbda6693640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fbda00c8d70 0x7fbda015c120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:50.620 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.617+0000 7fbda6693640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbda0158cd0 con 0x7fbda0158210 2023-05-22T15:44:50.620 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.617+0000 7fbda6693640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbda015c790 con 0x7fbda015bcd0 2023-05-22T15:44:50.621 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.617+0000 7fbda6693640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbda015ca70 con 0x7fbda00c8d70 2023-05-22T15:44:50.621 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.617+0000 7fbd9ffff640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fbda0158210 0x7fbda0159160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:52002/0 (socket says 172.21.15.16:52002) 2023-05-22T15:44:50.621 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.617+0000 7fbd9ffff640 1 -- 172.21.15.16:0/2650514105 learned_addr learned my addr 172.21.15.16:0/2650514105 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:44:50.621 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.617+0000 7fbda4c30640 1 --2- 172.21.15.16:0/2650514105 >> v2:172.21.15.16:3300/0 conn(0x7fbda00c8d70 0x7fbda015c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:44:50.622 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.617+0000 7fbd9f7fe640 1 -- 172.21.15.16:0/2650514105 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 749020399 0 0) 0x7fbda0158cd0 con 0x7fbda0158210 2023-05-22T15:44:50.622 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.617+0000 7fbd9f7fe640 1 -- 172.21.15.16:0/2650514105 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd8c005740 con 0x7fbda0158210 2023-05-22T15:44:50.622 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.617+0000 7fbda4c30640 1 -- 172.21.15.16:0/2650514105 >> v1:172.21.15.62:6789/0 conn(0x7fbda015bcd0 legacy=0x7fbda00c8660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:44:50.622 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.617+0000 7fbda4c30640 1 -- 172.21.15.16:0/2650514105 >> v1:172.21.15.16:6789/0 conn(0x7fbda0158210 legacy=0x7fbda0159160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:44:50.622 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.617+0000 7fbda4c30640 1 -- 172.21.15.16:0/2650514105 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbda0158960 con 0x7fbda00c8d70 2023-05-22T15:44:50.623 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.621+0000 7fbda4c30640 1 --2- 172.21.15.16:0/2650514105 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbda00c8d70 0x7fbda015c120 secure :-1 s=READY pgs=1562 cs=0 l=1 rev1=1 crypto rx=0x7fbd98009d90 tx=0x7fbd9801c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ab34cff783c63fd server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:44:50.623 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.621+0000 7fbd9f7fe640 1 -- 172.21.15.16:0/2650514105 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd980a5920 con 0x7fbda00c8d70 2023-05-22T15:44:50.623 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.621+0000 7fbd9f7fe640 1 -- 172.21.15.16:0/2650514105 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd980a5a80 con 0x7fbda00c8d70 2023-05-22T15:44:50.624 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.621+0000 7fbd9f7fe640 1 -- 172.21.15.16:0/2650514105 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd980a5df0 con 0x7fbda00c8d70 2023-05-22T15:44:50.624 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.621+0000 7fbda6693640 1 -- 172.21.15.16:0/2650514105 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbda00c8d70 msgr2=0x7fbda015c120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:44:50.624 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.621+0000 7fbda6693640 1 --2- 172.21.15.16:0/2650514105 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbda00c8d70 0x7fbda015c120 secure :-1 s=READY pgs=1562 cs=0 l=1 rev1=1 crypto rx=0x7fbd98009d90 tx=0x7fbd9801c8c0 comp rx=0 tx=0).stop 2023-05-22T15:44:50.624 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.621+0000 7fbda6693640 1 -- 172.21.15.16:0/2650514105 shutdown_connections 2023-05-22T15:44:50.624 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.621+0000 7fbda6693640 1 --2- 172.21.15.16:0/2650514105 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbda00c8d70 0x7fbda015c120 unknown :-1 s=CLOSED pgs=1562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:50.624 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.621+0000 7fbda6693640 1 -- 172.21.15.16:0/2650514105 >> 172.21.15.16:0/2650514105 conn(0x7fbda0069720 msgr2=0x7fbda0069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:44:50.624 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.621+0000 7fbda6693640 1 -- 172.21.15.16:0/2650514105 shutdown_connections 2023-05-22T15:44:50.625 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.621+0000 7fbda6693640 1 -- 172.21.15.16:0/2650514105 wait complete. 2023-05-22T15:44:50.626 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.621+0000 7fbda6693640 1 Processor -- start 2023-05-22T15:44:50.626 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.621+0000 7fbda6693640 1 -- start start 2023-05-22T15:44:50.626 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.621+0000 7fbda6693640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbda00c8d70 0x7fbda0149a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:50.626 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.621+0000 7fbda6693640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbda0158210 0x7fbda014e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:50.627 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.621+0000 7fbda6693640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbda015bcd0 0x7fbda015ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:50.627 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.621+0000 7fbda6693640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbda014eb10 con 0x7fbda00c8d70 2023-05-22T15:44:50.627 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.621+0000 7fbda6693640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbda014edf0 con 0x7fbda0158210 2023-05-22T15:44:50.627 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.621+0000 7fbda6693640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbda015a420 con 0x7fbda015bcd0 2023-05-22T15:44:50.627 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.621+0000 7fbd9ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbda00c8d70 0x7fbda0149a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:44:50.627 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.621+0000 7fbd9ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbda00c8d70 0x7fbda0149a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59248/0 (socket says 172.21.15.16:59248) 2023-05-22T15:44:50.627 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.621+0000 7fbd9ffff640 1 -- 172.21.15.16:0/2957737272 learned_addr learned my addr 172.21.15.16:0/2957737272 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:44:50.628 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.621+0000 7fbd97fff640 1 --2- 172.21.15.16:0/2957737272 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbda0158210 0x7fbda014e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:44:50.628 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.621+0000 7fbda4c30640 1 --2- 172.21.15.16:0/2957737272 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbda015bcd0 0x7fbda015ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:44:50.628 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.621+0000 7fbd9ffff640 1 -- 172.21.15.16:0/2957737272 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbda015bcd0 msgr2=0x7fbda015ac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:44:50.628 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.621+0000 7fbd9ffff640 1 --2- 172.21.15.16:0/2957737272 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbda015bcd0 0x7fbda015ac20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:50.628 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.621+0000 7fbd9ffff640 1 -- 172.21.15.16:0/2957737272 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbda0158210 msgr2=0x7fbda014e1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:44:50.628 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.621+0000 7fbd9ffff640 1 --2- 172.21.15.16:0/2957737272 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbda0158210 0x7fbda014e1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:50.628 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.621+0000 7fbd9ffff640 1 -- 172.21.15.16:0/2957737272 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd98009a70 con 0x7fbda00c8d70 2023-05-22T15:44:50.628 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.621+0000 7fbd9ffff640 1 --2- 172.21.15.16:0/2957737272 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbda00c8d70 0x7fbda0149a10 secure :-1 s=READY pgs=1563 cs=0 l=1 rev1=1 crypto rx=0x7fbd9000c4d0 tx=0x7fbd9000c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:44:50.629 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.621+0000 7fbd9d7fa640 1 -- 172.21.15.16:0/2957737272 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd90005cf0 con 0x7fbda00c8d70 2023-05-22T15:44:50.629 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.621+0000 7fbd9d7fa640 1 -- 172.21.15.16:0/2957737272 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd90005e50 con 0x7fbda00c8d70 2023-05-22T15:44:50.629 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.621+0000 7fbda6693640 1 -- 172.21.15.16:0/2957737272 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbda00ca820 con 0x7fbda00c8d70 2023-05-22T15:44:50.629 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.621+0000 7fbd9d7fa640 1 -- 172.21.15.16:0/2957737272 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd90007270 con 0x7fbda00c8d70 2023-05-22T15:44:50.629 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.621+0000 7fbda6693640 1 -- 172.21.15.16:0/2957737272 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbda00ca690 con 0x7fbda00c8d70 2023-05-22T15:44:50.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.625+0000 7fbda6693640 1 -- 172.21.15.16:0/2957737272 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbda00ca9b0 con 0x7fbda00c8d70 2023-05-22T15:44:50.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.625+0000 7fbd9d7fa640 1 -- 172.21.15.16:0/2957737272 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 48) v1 ==== 83941+0+0 (secure 0 0 0) 0x7fbd900073d0 con 0x7fbda00c8d70 2023-05-22T15:44:50.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.625+0000 7fbd9d7fa640 1 -- 172.21.15.16:0/2957737272 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 7070+0+0 (secure 0 0 0) 0x7fbd90081c80 con 0x7fbda00c8d70 2023-05-22T15:44:50.632 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.629+0000 7fbd9d7fa640 1 -- 172.21.15.16:0/2957737272 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fbd9004a620 con 0x7fbda00c8d70 2023-05-22T15:44:50.785 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.781+0000 7fbda6693640 1 -- 172.21.15.16:0/2957737272 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fbda015a900 con 0x7fbda00c8d70 2023-05-22T15:44:50.785 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.781+0000 7fbd9d7fa640 1 -- 172.21.15.16:0/2957737272 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v120) v1 ==== 100+0+11 (secure 0 0 0) 0x7fbd9004f920 con 0x7fbda00c8d70 2023-05-22T15:44:50.786 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 64 2023-05-22T15:44:50.788 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.785+0000 7fbda6693640 1 -- 172.21.15.16:0/2957737272 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbda00c8d70 msgr2=0x7fbda0149a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:44:50.788 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.785+0000 7fbda6693640 1 --2- 172.21.15.16:0/2957737272 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbda00c8d70 0x7fbda0149a10 secure :-1 s=READY pgs=1563 cs=0 l=1 rev1=1 crypto rx=0x7fbd9000c4d0 tx=0x7fbd9000c990 comp rx=0 tx=0).stop 2023-05-22T15:44:50.789 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.785+0000 7fbda6693640 1 -- 172.21.15.16:0/2957737272 shutdown_connections 2023-05-22T15:44:50.789 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.785+0000 7fbda6693640 1 --2- 172.21.15.16:0/2957737272 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbda015bcd0 0x7fbda015ac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:50.789 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.785+0000 7fbda6693640 1 --2- 172.21.15.16:0/2957737272 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbda0158210 0x7fbda014e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:50.789 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.785+0000 7fbda6693640 1 --2- 172.21.15.16:0/2957737272 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbda00c8d70 0x7fbda0149a10 unknown :-1 s=CLOSED pgs=1563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:50.790 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.785+0000 7fbda6693640 1 -- 172.21.15.16:0/2957737272 >> 172.21.15.16:0/2957737272 conn(0x7fbda0069720 msgr2=0x7fbda006f4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:44:50.790 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.785+0000 7fbda6693640 1 -- 172.21.15.16:0/2957737272 shutdown_connections 2023-05-22T15:44:50.790 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.785+0000 7fbda6693640 1 -- 172.21.15.16:0/2957737272 wait complete. 2023-05-22T15:44:50.805 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-05-22T15:44:50.927 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.925+0000 7fcdabb68640 1 Processor -- start 2023-05-22T15:44:50.928 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.925+0000 7fcdabb68640 1 -- start start 2023-05-22T15:44:50.930 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.925+0000 7fcdabb68640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fcda40b7dd0 0x7fcda40ccab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:50.930 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.925+0000 7fcdabb68640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fcda415f700 0x7fcda415fb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:50.930 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.925+0000 7fcdabb68640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fcda4160050 0x7fcda40ce720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:50.931 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.925+0000 7fcdabb68640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcda40d3360 con 0x7fcda415f700 2023-05-22T15:44:50.931 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.925+0000 7fcdabb68640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcda40cd170 con 0x7fcda4160050 2023-05-22T15:44:50.931 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.925+0000 7fcdabb68640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcda40d2ac0 con 0x7fcda40b7dd0 2023-05-22T15:44:50.931 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.925+0000 7fcda9103640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fcda415f700 0x7fcda415fb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:44:50.931 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.925+0000 7fcda9103640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fcda415f700 0x7fcda415fb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59262/0 (socket says 172.21.15.16:59262) 2023-05-22T15:44:50.932 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.925+0000 7fcda9103640 1 -- 172.21.15.16:0/660165306 learned_addr learned my addr 172.21.15.16:0/660165306 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:44:50.932 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.925+0000 7fcdaa105640 1 --2- 172.21.15.16:0/660165306 >> v2:172.21.15.62:3300/0 conn(0x7fcda4160050 0x7fcda40ce720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:44:50.932 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.925+0000 7fcda9904640 1 --2- 172.21.15.16:0/660165306 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fcda40b7dd0 0x7fcda40ccab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:44:50.933 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.925+0000 7fcda9103640 1 -- 172.21.15.16:0/660165306 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fcda40b7dd0 msgr2=0x7fcda40ccab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:44:50.933 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.925+0000 7fcda9103640 1 --2- 172.21.15.16:0/660165306 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fcda40b7dd0 0x7fcda40ccab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:50.933 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.925+0000 7fcda9103640 1 -- 172.21.15.16:0/660165306 >> v2:172.21.15.62:3300/0 conn(0x7fcda4160050 msgr2=0x7fcda40ce720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:44:50.933 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.925+0000 7fcda9103640 1 --2- 172.21.15.16:0/660165306 >> v2:172.21.15.62:3300/0 conn(0x7fcda4160050 0x7fcda40ce720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:50.933 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.925+0000 7fcda9103640 1 -- 172.21.15.16:0/660165306 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcda40d2da0 con 0x7fcda415f700 2023-05-22T15:44:50.935 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.929+0000 7fcda9103640 1 --2- 172.21.15.16:0/660165306 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcda415f700 0x7fcda415fb10 secure :-1 s=READY pgs=1564 cs=0 l=1 rev1=1 crypto rx=0x7fcda41607a0 tx=0x7fcd9401c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6a8752c60aa1adb server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:44:50.935 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.929+0000 7fcda8902640 1 -- 172.21.15.16:0/660165306 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd940aa610 con 0x7fcda415f700 2023-05-22T15:44:50.936 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.929+0000 7fcda8902640 1 -- 172.21.15.16:0/660165306 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd940a6040 con 0x7fcda415f700 2023-05-22T15:44:50.936 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.929+0000 7fcda8902640 1 -- 172.21.15.16:0/660165306 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd940aa990 con 0x7fcda415f700 2023-05-22T15:44:50.936 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.929+0000 7fcdabb68640 1 -- 172.21.15.16:0/660165306 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcda415f700 msgr2=0x7fcda415fb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:44:50.936 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.929+0000 7fcdabb68640 1 --2- 172.21.15.16:0/660165306 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcda415f700 0x7fcda415fb10 secure :-1 s=READY pgs=1564 cs=0 l=1 rev1=1 crypto rx=0x7fcda41607a0 tx=0x7fcd9401c4d0 comp rx=0 tx=0).stop 2023-05-22T15:44:50.936 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.929+0000 7fcdabb68640 1 -- 172.21.15.16:0/660165306 shutdown_connections 2023-05-22T15:44:50.936 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.929+0000 7fcdabb68640 1 --2- 172.21.15.16:0/660165306 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcda415f700 0x7fcda415fb10 unknown :-1 s=CLOSED pgs=1564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:50.937 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.929+0000 7fcdabb68640 1 --2- 172.21.15.16:0/660165306 >> v2:172.21.15.62:3300/0 conn(0x7fcda4160050 0x7fcda40ce720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:50.937 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.929+0000 7fcdabb68640 1 --2- 172.21.15.16:0/660165306 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fcda40b7dd0 0x7fcda40ccab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:50.937 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.929+0000 7fcdabb68640 1 -- 172.21.15.16:0/660165306 >> 172.21.15.16:0/660165306 conn(0x7fcda40655c0 msgr2=0x7fcda4070e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:44:50.937 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.929+0000 7fcdabb68640 1 -- 172.21.15.16:0/660165306 shutdown_connections 2023-05-22T15:44:50.937 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.929+0000 7fcdabb68640 1 -- 172.21.15.16:0/660165306 wait complete. 2023-05-22T15:44:50.937 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.929+0000 7fcdabb68640 1 Processor -- start 2023-05-22T15:44:50.938 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.929+0000 7fcdabb68640 1 -- start start 2023-05-22T15:44:50.938 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.929+0000 7fcdabb68640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcda40b7dd0 0x7fcda4158570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:50.938 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.929+0000 7fcdabb68640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcda415f700 0x7fcda40c8a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:50.938 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.929+0000 7fcdabb68640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fcda4160050 0x7fcda40c9530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:50.938 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.929+0000 7fcdabb68640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcda40c8590 con 0x7fcda40b7dd0 2023-05-22T15:44:50.938 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.929+0000 7fcdabb68640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcda40c7980 con 0x7fcda415f700 2023-05-22T15:44:50.938 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.929+0000 7fcdabb68640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcda40c7c60 con 0x7fcda4160050 2023-05-22T15:44:50.938 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.929+0000 7fcda9904640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcda40b7dd0 0x7fcda4158570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:44:50.939 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.929+0000 7fcda9904640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcda40b7dd0 0x7fcda4158570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59272/0 (socket says 172.21.15.16:59272) 2023-05-22T15:44:50.939 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.933+0000 7fcda9904640 1 -- 172.21.15.16:0/447709052 learned_addr learned my addr 172.21.15.16:0/447709052 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:44:50.939 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.933+0000 7fcdaa105640 1 --2- 172.21.15.16:0/447709052 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fcda4160050 0x7fcda40c9530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:44:50.939 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.933+0000 7fcda9103640 1 --2- 172.21.15.16:0/447709052 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcda415f700 0x7fcda40c8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:44:50.939 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.933+0000 7fcda9904640 1 -- 172.21.15.16:0/447709052 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fcda4160050 msgr2=0x7fcda40c9530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:44:50.940 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.933+0000 7fcda9904640 1 --2- 172.21.15.16:0/447709052 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fcda4160050 0x7fcda40c9530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:50.940 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.933+0000 7fcda9904640 1 -- 172.21.15.16:0/447709052 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcda415f700 msgr2=0x7fcda40c8a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:44:50.940 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.933+0000 7fcda9904640 1 --2- 172.21.15.16:0/447709052 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcda415f700 0x7fcda40c8a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:50.940 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.933+0000 7fcda9904640 1 -- 172.21.15.16:0/447709052 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd940094d0 con 0x7fcda40b7dd0 2023-05-22T15:44:50.940 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.933+0000 7fcda9904640 1 --2- 172.21.15.16:0/447709052 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcda40b7dd0 0x7fcda4158570 secure :-1 s=READY pgs=1565 cs=0 l=1 rev1=1 crypto rx=0x7fcda000d800 tx=0x7fcda000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:44:50.940 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.933+0000 7fcd9a7fc640 1 -- 172.21.15.16:0/447709052 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcda000b6c0 con 0x7fcda40b7dd0 2023-05-22T15:44:50.940 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.933+0000 7fcd9a7fc640 1 -- 172.21.15.16:0/447709052 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcda000b820 con 0x7fcda40b7dd0 2023-05-22T15:44:50.941 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.933+0000 7fcdabb68640 1 -- 172.21.15.16:0/447709052 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcda40cf600 con 0x7fcda40b7dd0 2023-05-22T15:44:50.941 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.933+0000 7fcd9a7fc640 1 -- 172.21.15.16:0/447709052 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcda000bb90 con 0x7fcda40b7dd0 2023-05-22T15:44:50.941 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.933+0000 7fcdabb68640 1 -- 172.21.15.16:0/447709052 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcda40cf470 con 0x7fcda40b7dd0 2023-05-22T15:44:50.941 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.933+0000 7fcdabb68640 1 -- 172.21.15.16:0/447709052 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcda40cf790 con 0x7fcda40b7dd0 2023-05-22T15:44:50.942 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.933+0000 7fcd9a7fc640 1 -- 172.21.15.16:0/447709052 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 48) v1 ==== 83941+0+0 (secure 0 0 0) 0x7fcda000bcf0 con 0x7fcda40b7dd0 2023-05-22T15:44:50.942 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.933+0000 7fcd9a7fc640 1 -- 172.21.15.16:0/447709052 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 7070+0+0 (secure 0 0 0) 0x7fcda0082ce0 con 0x7fcda40b7dd0 2023-05-22T15:44:50.942 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:50.937+0000 7fcd9a7fc640 1 -- 172.21.15.16:0/447709052 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fcda00507f0 con 0x7fcda40b7dd0 2023-05-22T15:44:51.065 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.061+0000 7fcd9a7fc640 1 -- 172.21.15.16:0/447709052 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mgrmap(e 49) v1 ==== 84068+0+0 (secure 0 0 0) 0x7fcda0050470 con 0x7fcda40b7dd0 2023-05-22T15:44:51.066 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.061+0000 7fcd9a7fc640 1 --2- 172.21.15.16:0/447709052 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fcd8806b6d0 0x7fcd8806dab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:51.067 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.065+0000 7fcda9103640 1 --2- 172.21.15.16:0/447709052 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fcd8806b6d0 0x7fcd8806dab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:44:51.070 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.065+0000 7fcda9103640 1 --2- 172.21.15.16:0/447709052 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fcd8806b6d0 0x7fcd8806dab0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fcd940ae000 tx=0x7fcd940a7000 comp rx=0 tx=0).ready entity=mgr.8304 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:44:51.132 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.129+0000 7fcdabb68640 1 -- 172.21.15.16:0/447709052 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcda404dbe0 con 0x7fcda40b7dd0 2023-05-22T15:44:51.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.129+0000 7fcd9a7fc640 1 -- 172.21.15.16:0/447709052 <== mon.0 v2:172.21.15.16:3300/0 8 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 43 v43) v1 ==== 94+0+4650 (secure 0 0 0) 0x7fcda007db90 con 0x7fcda40b7dd0 2023-05-22T15:44:51.133 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:44:51.134 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":43,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6350,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6833/1554108010","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":1554108010},{"type":"v1","addr":"172.21.15.62:6833","nonce":1554108010}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":39},{"gid":6979,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/2667219028","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":2667219028},{"type":"v1","addr":"172.21.15.62:6835","nonce":2667219028}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":39},{"gid":8013,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6836/1892737356","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1892737356},{"type":"v1","addr":"172.21.15.16:6836","nonce":1892737356}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":39}],"filesystems":[{"mdsmap":{"epoch":43,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-05-22T15:43:30.134589+0000","modified":"2023-05-22T15:43:33.144847+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8016},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8016":{"gid":8016,"name":"a","rank":0,"incarnation":40,"state":"up:active","state_seq":3,"addr":"172.21.15.16:6837/2085936428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6835","nonce":2085936428},{"type":"v1","addr":"172.21.15.16:6837","nonce":2085936428}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2023-05-22T15:44:51.135 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 43 2023-05-22T15:44:51.135 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.133+0000 7fcdabb68640 1 -- 172.21.15.16:0/447709052 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fcd8806b6d0 msgr2=0x7fcd8806dab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:44:51.136 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.133+0000 7fcdabb68640 1 --2- 172.21.15.16:0/447709052 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fcd8806b6d0 0x7fcd8806dab0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fcd940ae000 tx=0x7fcd940a7000 comp rx=0 tx=0).stop 2023-05-22T15:44:51.136 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.133+0000 7fcdabb68640 1 -- 172.21.15.16:0/447709052 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcda40b7dd0 msgr2=0x7fcda4158570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:44:51.136 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.133+0000 7fcdabb68640 1 --2- 172.21.15.16:0/447709052 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcda40b7dd0 0x7fcda4158570 secure :-1 s=READY pgs=1565 cs=0 l=1 rev1=1 crypto rx=0x7fcda000d800 tx=0x7fcda000dcc0 comp rx=0 tx=0).stop 2023-05-22T15:44:51.136 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.133+0000 7fcdabb68640 1 -- 172.21.15.16:0/447709052 shutdown_connections 2023-05-22T15:44:51.136 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.133+0000 7fcdabb68640 1 --2- 172.21.15.16:0/447709052 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fcd8806b6d0 0x7fcd8806dab0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:51.136 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.133+0000 7fcdabb68640 1 --2- 172.21.15.16:0/447709052 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fcda4160050 0x7fcda40c9530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:51.137 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.133+0000 7fcdabb68640 1 --2- 172.21.15.16:0/447709052 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcda415f700 0x7fcda40c8a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:51.137 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.133+0000 7fcdabb68640 1 --2- 172.21.15.16:0/447709052 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcda40b7dd0 0x7fcda4158570 unknown :-1 s=CLOSED pgs=1565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:51.137 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.133+0000 7fcdabb68640 1 -- 172.21.15.16:0/447709052 >> 172.21.15.16:0/447709052 conn(0x7fcda40655c0 msgr2=0x7fcda40b6980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:44:51.137 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.133+0000 7fcdabb68640 1 -- 172.21.15.16:0/447709052 shutdown_connections 2023-05-22T15:44:51.137 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.133+0000 7fcdabb68640 1 -- 172.21.15.16:0/447709052 wait complete. 2023-05-22T15:44:51.151 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-05-22T15:44:51.274 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.269+0000 7f26938de640 1 Processor -- start 2023-05-22T15:44:51.274 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.269+0000 7f26938de640 1 -- start start 2023-05-22T15:44:51.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.273+0000 7f26938de640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f268c0b0250 0x7f268c0cd1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:51.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.273+0000 7f26938de640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f268c0cd8d0 con 0x7f268c15fcf0 2023-05-22T15:44:51.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.273+0000 7f26938de640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f268c164de0 con 0x7f268c070fb0 2023-05-22T15:44:51.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.273+0000 7f26938de640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f268c0ca400 con 0x7f268c0b0250 2023-05-22T15:44:51.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.273+0000 7f2691e7b640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f268c15fcf0 0x7f268c160190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:52004/0 (socket says 172.21.15.16:52004) 2023-05-22T15:44:51.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.273+0000 7f2691e7b640 1 -- 172.21.15.16:0/271649294 learned_addr learned my addr 172.21.15.16:0/271649294 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:44:51.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.273+0000 7f2690e79640 1 --2- 172.21.15.16:0/271649294 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f268c0b0250 0x7f268c0cd1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:44:51.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.273+0000 7f2683fff640 1 -- 172.21.15.16:0/271649294 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2672645367 0 0) 0x7f268c0cd8d0 con 0x7f268c15fcf0 2023-05-22T15:44:51.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.273+0000 7f2683fff640 1 -- 172.21.15.16:0/271649294 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2678003530 con 0x7f268c15fcf0 2023-05-22T15:44:51.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.273+0000 7f2683fff640 1 -- 172.21.15.16:0/271649294 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1773488226 0 0) 0x7f2678003530 con 0x7f268c15fcf0 2023-05-22T15:44:51.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.273+0000 7f2683fff640 1 -- 172.21.15.16:0/271649294 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f268c0b0250 msgr2=0x7f268c0cd1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:44:51.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.273+0000 7f2683fff640 1 --2- 172.21.15.16:0/271649294 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f268c0b0250 0x7f268c0cd1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:51.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.273+0000 7f2683fff640 1 -- 172.21.15.16:0/271649294 >> v1:172.21.15.62:6789/0 conn(0x7f268c070fb0 legacy=0x7f268c0afb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:44:51.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.273+0000 7f2683fff640 1 -- 172.21.15.16:0/271649294 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f268c0ca6e0 con 0x7f268c15fcf0 2023-05-22T15:44:51.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.273+0000 7f2683fff640 1 -- 172.21.15.16:0/271649294 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7f2684001a00 con 0x7f268c15fcf0 2023-05-22T15:44:51.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.273+0000 7f2683fff640 1 -- 172.21.15.16:0/271649294 >> v1:172.21.15.16:6789/0 conn(0x7f268c15fcf0 legacy=0x7f268c160190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:44:51.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.273+0000 7f2683fff640 1 --2- 172.21.15.16:0/271649294 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f26780039b0 0x7f2678003d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:51.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.273+0000 7f2683fff640 1 --2- 172.21.15.16:0/271649294 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2678004350 0x7f2678004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:51.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.273+0000 7f2683fff640 1 --2- 172.21.15.16:0/271649294 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f268c0b0250 0x7f26780050a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:51.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.273+0000 7f2683fff640 1 -- 172.21.15.16:0/271649294 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f268c0ca400 con 0x7f2678004350 2023-05-22T15:44:51.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.273+0000 7f2683fff640 1 -- 172.21.15.16:0/271649294 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f267800b9c0 con 0x7f26780039b0 2023-05-22T15:44:51.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.273+0000 7f2683fff640 1 -- 172.21.15.16:0/271649294 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f267800bb80 con 0x7f268c0b0250 2023-05-22T15:44:51.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.273+0000 7f2690e79640 1 --2- 172.21.15.16:0/271649294 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2678004350 0x7f2678004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:44:51.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.273+0000 7f269167a640 1 --2- 172.21.15.16:0/271649294 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f26780039b0 0x7f2678003d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:44:51.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.273+0000 7f2691e7b640 1 --2- 172.21.15.16:0/271649294 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f268c0b0250 0x7f26780050a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:44:51.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.273+0000 7f2690e79640 1 -- 172.21.15.16:0/271649294 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f268c0b0250 msgr2=0x7f26780050a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:44:51.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.273+0000 7f2690e79640 1 --2- 172.21.15.16:0/271649294 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f268c0b0250 0x7f26780050a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:51.282 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.273+0000 7f2690e79640 1 -- 172.21.15.16:0/271649294 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f26780039b0 msgr2=0x7f2678003d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:44:51.282 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.273+0000 7f2690e79640 1 --2- 172.21.15.16:0/271649294 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f26780039b0 0x7f2678003d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:51.282 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.273+0000 7f2690e79640 1 -- 172.21.15.16:0/271649294 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2678003530 con 0x7f2678004350 2023-05-22T15:44:51.282 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.273+0000 7f2691e7b640 1 --2- 172.21.15.16:0/271649294 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f268c0b0250 0x7f26780050a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T15:44:51.282 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.273+0000 7f269167a640 1 --2- 172.21.15.16:0/271649294 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f26780039b0 0x7f2678003d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T15:44:51.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.277+0000 7f2690e79640 1 --2- 172.21.15.16:0/271649294 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2678004350 0x7f2678004810 secure :-1 s=READY pgs=1567 cs=0 l=1 rev1=1 crypto rx=0x7f267c009ab0 tx=0x7f267c01ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25f0d0cfb27734b2 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:44:51.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.277+0000 7f2683fff640 1 -- 172.21.15.16:0/271649294 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f267c0a5620 con 0x7f2678004350 2023-05-22T15:44:51.285 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.277+0000 7f2683fff640 1 -- 172.21.15.16:0/271649294 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f267c0a5780 con 0x7f2678004350 2023-05-22T15:44:51.285 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.277+0000 7f2683fff640 1 -- 172.21.15.16:0/271649294 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f267c0a5b30 con 0x7f2678004350 2023-05-22T15:44:51.285 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.277+0000 7f26938de640 1 -- 172.21.15.16:0/271649294 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2678004350 msgr2=0x7f2678004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:44:51.285 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.277+0000 7f26938de640 1 --2- 172.21.15.16:0/271649294 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2678004350 0x7f2678004810 secure :-1 s=READY pgs=1567 cs=0 l=1 rev1=1 crypto rx=0x7f267c009ab0 tx=0x7f267c01ca30 comp rx=0 tx=0).stop 2023-05-22T15:44:51.285 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.277+0000 7f2691e7b640 1 -- 172.21.15.16:0/271649294 reap_dead start 2023-05-22T15:44:51.286 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.277+0000 7f26938de640 1 -- 172.21.15.16:0/271649294 shutdown_connections 2023-05-22T15:44:51.286 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.277+0000 7f26938de640 1 -- 172.21.15.16:0/271649294 >> 172.21.15.16:0/271649294 conn(0x7f268c069720 msgr2=0x7f268c069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:44:51.286 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.277+0000 7f26938de640 1 -- 172.21.15.16:0/271649294 shutdown_connections 2023-05-22T15:44:51.286 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.277+0000 7f26938de640 1 -- 172.21.15.16:0/271649294 wait complete. 2023-05-22T15:44:51.286 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.277+0000 7f26938de640 1 Processor -- start 2023-05-22T15:44:51.286 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.277+0000 7f26938de640 1 -- start start 2023-05-22T15:44:51.286 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.281+0000 7f26938de640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f268c14e360 0x7f268c0c6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:51.286 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.281+0000 7f26938de640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f268c14e730 0x7f268c0c67b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:51.287 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.281+0000 7f26938de640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f268c0b2980 0x7f268c0b2df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:51.287 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.281+0000 7f26938de640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f268c0b36b0 con 0x7f268c0b2980 2023-05-22T15:44:51.287 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.281+0000 7f26938de640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f268c0b3990 con 0x7f268c14e730 2023-05-22T15:44:51.287 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.281+0000 7f26938de640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f268c160ad0 con 0x7f268c14e360 2023-05-22T15:44:51.287 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.281+0000 7f2691e7b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f268c0b2980 0x7f268c0b2df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:44:51.287 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.281+0000 7f2691e7b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f268c0b2980 0x7f268c0b2df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59292/0 (socket says 172.21.15.16:59292) 2023-05-22T15:44:51.287 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.281+0000 7f2691e7b640 1 -- 172.21.15.16:0/3781439726 learned_addr learned my addr 172.21.15.16:0/3781439726 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:44:51.288 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.281+0000 7f2690e79640 1 --2- 172.21.15.16:0/3781439726 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f268c14e730 0x7f268c0c67b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:44:51.288 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.281+0000 7f269167a640 1 --2- 172.21.15.16:0/3781439726 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f268c14e360 0x7f268c0c6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:44:51.288 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.281+0000 7f2691e7b640 1 -- 172.21.15.16:0/3781439726 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f268c14e360 msgr2=0x7f268c0c6270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:44:51.288 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.281+0000 7f2691e7b640 1 --2- 172.21.15.16:0/3781439726 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f268c14e360 0x7f268c0c6270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:51.288 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.281+0000 7f2691e7b640 1 -- 172.21.15.16:0/3781439726 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f268c14e730 msgr2=0x7f268c0c67b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:44:51.288 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.281+0000 7f2691e7b640 1 --2- 172.21.15.16:0/3781439726 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f268c14e730 0x7f268c0c67b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:51.288 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.281+0000 7f2691e7b640 1 -- 172.21.15.16:0/3781439726 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f268c1608a0 con 0x7f268c0b2980 2023-05-22T15:44:51.289 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.281+0000 7f2691e7b640 1 --2- 172.21.15.16:0/3781439726 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f268c0b2980 0x7f268c0b2df0 secure :-1 s=READY pgs=1568 cs=0 l=1 rev1=1 crypto rx=0x7f2684001fc0 tx=0x7f268400e160 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:44:51.289 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.281+0000 7f2681ffb640 1 -- 172.21.15.16:0/3781439726 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f268400e720 con 0x7f268c0b2980 2023-05-22T15:44:51.289 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.281+0000 7f2681ffb640 1 -- 172.21.15.16:0/3781439726 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f268400e880 con 0x7f268c0b2980 2023-05-22T15:44:51.289 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.281+0000 7f2681ffb640 1 -- 172.21.15.16:0/3781439726 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f268400ec10 con 0x7f268c0b2980 2023-05-22T15:44:51.289 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.281+0000 7f26938de640 1 -- 172.21.15.16:0/3781439726 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f268c06f630 con 0x7f268c0b2980 2023-05-22T15:44:51.289 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.281+0000 7f26938de640 1 -- 172.21.15.16:0/3781439726 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f268c0cedc0 con 0x7f268c0b2980 2023-05-22T15:44:51.289 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.281+0000 7f26938de640 1 -- 172.21.15.16:0/3781439726 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f268c04dbe0 con 0x7f268c0b2980 2023-05-22T15:44:51.290 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.281+0000 7f2681ffb640 1 -- 172.21.15.16:0/3781439726 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 49) v1 ==== 84068+0+0 (secure 0 0 0) 0x7f268400ed70 con 0x7f268c0b2980 2023-05-22T15:44:51.290 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.281+0000 7f2681ffb640 1 --2- 172.21.15.16:0/3781439726 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f2670065200 0x7f26700676b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:51.290 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.281+0000 7f2681ffb640 1 -- 172.21.15.16:0/3781439726 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 7070+0+0 (secure 0 0 0) 0x7f26840832c0 con 0x7f268c0b2980 2023-05-22T15:44:51.290 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.281+0000 7f269167a640 1 --2- 172.21.15.16:0/3781439726 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f2670065200 0x7f26700676b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:44:51.291 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.281+0000 7f269167a640 1 --2- 172.21.15.16:0/3781439726 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f2670065200 0x7f26700676b0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f2688012450 tx=0x7f2688000bd0 comp rx=0 tx=0).ready entity=mgr.8304 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:44:51.291 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.285+0000 7f2681ffb640 1 -- 172.21.15.16:0/3781439726 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f2684050ef0 con 0x7f268c0b2980 2023-05-22T15:44:51.443 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.441+0000 7f26938de640 1 -- 172.21.15.16:0/3781439726 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f268c0cb210 con 0x7f268c0b2980 2023-05-22T15:44:51.445 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.441+0000 7f2681ffb640 1 -- 172.21.15.16:0/3781439726 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v120) v1 ==== 74+0+18973 (secure 0 0 0) 0x7f2684054510 con 0x7f268c0b2980 2023-05-22T15:44:51.446 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:44:51.446 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":120,"fsid":"b2719834-7847-4d98-90a6-f65d57f85c49","created":"2023-05-22T15:26:25.259314+0000","modified":"2023-05-22T15:44:50.039971+0000","last_up_change":"2023-05-22T15:26:36.448524+0000","last_in_change":"2023-05-22T15:26:27.675668+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-05-22T15:26:38.019781+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-05-22T15:26:43.570514+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2023-05-22T15:43:28.684367+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"117","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2023-05-22T15:43:29.119294+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"118","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9e7ff854-b25f-4cbb-a5c2-1799d9c53d2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":37021},{"type":"v1","addr":"172.21.15.16:6825","nonce":37021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":37021},{"type":"v1","addr":"172.21.15.16:6827","nonce":37021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":37021},{"type":"v1","addr":"172.21.15.16:6831","nonce":37021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":37021},{"type":"v1","addr":"172.21.15.16:6829","nonce":37021}]},"public_addr":"172.21.15.16:6825/37021","cluster_addr":"172.21.15.16:6827/37021","heartbeat_back_addr":"172.21.15.16:6831/37021","heartbeat_front_addr":"172.21.15.16:6829/37021","state":["exists","up"]},{"osd":1,"uuid":"4702fbc9-a927-4565-aca5-41dc69cc8264","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":37008},{"type":"v1","addr":"172.21.15.16:6801","nonce":37008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":37008},{"type":"v1","addr":"172.21.15.16:6803","nonce":37008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":37008},{"type":"v1","addr":"172.21.15.16:6807","nonce":37008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":37008},{"type":"v1","addr":"172.21.15.16:6805","nonce":37008}]},"public_addr":"172.21.15.16:6801/37008","cluster_addr":"172.21.15.16:6803/37008","heartbeat_back_addr":"172.21.15.16:6807/37008","heartbeat_front_addr":"172.21.15.16:6805/37008","state":["exists","up"]},{"osd":2,"uuid":"1e7bba46-4eba-47ef-bb81-5e2d371638ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":37009},{"type":"v1","addr":"172.21.15.16:6809","nonce":37009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":37009},{"type":"v1","addr":"172.21.15.16:6811","nonce":37009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":37009},{"type":"v1","addr":"172.21.15.16:6815","nonce":37009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":37009},{"type":"v1","addr":"172.21.15.16:6813","nonce":37009}]},"public_addr":"172.21.15.16:6809/37009","cluster_addr":"172.21.15.16:6811/37009","heartbeat_back_addr":"172.21.15.16:6815/37009","heartbeat_front_addr":"172.21.15.16:6813/37009","state":["exists","up"]},{"osd":3,"uuid":"383cacc0-5a88-4904-990e-498ca59ef48a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":37020},{"type":"v1","addr":"172.21.15.16:6817","nonce":37020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":37020},{"type":"v1","addr":"172.21.15.16:6819","nonce":37020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":37020},{"type":"v1","addr":"172.21.15.16:6823","nonce":37020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":37020},{"type":"v1","addr":"172.21.15.16:6821","nonce":37020}]},"public_addr":"172.21.15.16:6817/37020","cluster_addr":"172.21.15.16:6819/37020","heartbeat_back_addr":"172.21.15.16:6823/37020","heartbeat_front_addr":"172.21.15.16:6821/37020","state":["exists","up"]},{"osd":4,"uuid":"970500ef-95b7-42c1-b080-eed4fae7013d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":37353},{"type":"v1","addr":"172.21.15.62:6817","nonce":37353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6819","nonce":37353},{"type":"v1","addr":"172.21.15.62:6822","nonce":37353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37353},{"type":"v1","addr":"172.21.15.62:6831","nonce":37353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37353},{"type":"v1","addr":"172.21.15.62:6826","nonce":37353}]},"public_addr":"172.21.15.62:6817/37353","cluster_addr":"172.21.15.62:6822/37353","heartbeat_back_addr":"172.21.15.62:6831/37353","heartbeat_front_addr":"172.21.15.62:6826/37353","state":["exists","up"]},{"osd":5,"uuid":"022d67db-d4d8-445b-b1b9-8b446a00f5c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":37352},{"type":"v1","addr":"172.21.15.62:6801","nonce":37352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":37352},{"type":"v1","addr":"172.21.15.62:6803","nonce":37352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":37352},{"type":"v1","addr":"172.21.15.62:6807","nonce":37352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":37352},{"type":"v1","addr":"172.21.15.62:6805","nonce":37352}]},"public_addr":"172.21.15.62:6801/37352","cluster_addr":"172.21.15.62:6803/37352","heartbeat_back_addr":"172.21.15.62:6807/37352","heartbeat_front_addr":"172.21.15.62:6805/37352","state":["exists","up"]},{"osd":6,"uuid":"42bd372b-6688-4a1b-8407-e16e8c266e33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":37354},{"type":"v1","addr":"172.21.15.62:6820","nonce":37354}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":37354},{"type":"v1","addr":"172.21.15.62:6823","nonce":37354}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6829","nonce":37354},{"type":"v1","addr":"172.21.15.62:6830","nonce":37354}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6825","nonce":37354},{"type":"v1","addr":"172.21.15.62:6827","nonce":37354}]},"public_addr":"172.21.15.62:6820/37354","cluster_addr":"172.21.15.62:6823/37354","heartbeat_back_addr":"172.21.15.62:6830/37354","heartbeat_front_addr":"172.21.15.62:6827/37354","state":["exists","up"]},{"osd":7,"uuid":"ac70b2a3-f41b-4bf2-b227-7f2cbb55426d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37355},{"type":"v1","addr":"172.21.15.62:6809","nonce":37355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37355},{"type":"v1","addr":"172.21.15.62:6811","nonce":37355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":37355},{"type":"v1","addr":"172.21.15.62:6815","nonce":37355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":37355},{"type":"v1","addr":"172.21.15.62:6813","nonce":37355}]},"public_addr":"172.21.15.62:6809/37355","cluster_addr":"172.21.15.62:6811/37355","heartbeat_back_addr":"172.21.15.62:6815/37355","heartbeat_front_addr":"172.21.15.62:6813/37355","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.368896+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.107273+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.276888+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.287502+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.124791+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.251937+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.255679+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.232797+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:0/2189763318":"2023-05-23T15:44:50.039898+0000","172.21.15.16:0/384246348":"2023-05-23T15:44:50.039898+0000","172.21.15.16:0/3752134468":"2023-05-23T15:44:50.039898+0000","172.21.15.16:0/2658406406":"2023-05-23T15:44:50.039898+0000","172.21.15.16:0/2218849021":"2023-05-23T15:43:43.641715+0000","172.21.15.16:0/1980969896":"2023-05-23T15:44:50.039898+0000","172.21.15.16:0/2339322867":"2023-05-23T15:43:43.641715+0000","172.21.15.16:0/155043049":"2023-05-23T15:43:43.641715+0000","172.21.15.16:0/4168497577":"2023-05-23T15:43:43.641715+0000","172.21.15.16:0/3302517394":"2023-05-23T15:43:43.641715+0000","172.21.15.16:0/4100157757":"2023-05-23T15:43:43.641715+0000","172.21.15.16:0/2834828521":"2023-05-23T15:44:50.039898+0000","172.21.15.16:0/579180365":"2023-05-23T15:44:50.039898+0000","172.21.15.16:0/177878289":"2023-05-23T15:44:50.039898+0000","172.21.15.16:6833/36921":"2023-05-23T15:43:43.641715+0000","172.21.15.16:6832/36921":"2023-05-23T15:43:43.641715+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-05-22T15:44:51.448 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.445+0000 7f26938de640 1 -- 172.21.15.16:0/3781439726 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f2670065200 msgr2=0x7f26700676b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:44:51.448 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.445+0000 7f26938de640 1 --2- 172.21.15.16:0/3781439726 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f2670065200 0x7f26700676b0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f2688012450 tx=0x7f2688000bd0 comp rx=0 tx=0).stop 2023-05-22T15:44:51.449 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.445+0000 7f26938de640 1 -- 172.21.15.16:0/3781439726 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f268c0b2980 msgr2=0x7f268c0b2df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:44:51.449 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.445+0000 7f26938de640 1 --2- 172.21.15.16:0/3781439726 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f268c0b2980 0x7f268c0b2df0 secure :-1 s=READY pgs=1568 cs=0 l=1 rev1=1 crypto rx=0x7f2684001fc0 tx=0x7f268400e160 comp rx=0 tx=0).stop 2023-05-22T15:44:51.449 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.445+0000 7f26938de640 1 -- 172.21.15.16:0/3781439726 shutdown_connections 2023-05-22T15:44:51.449 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.445+0000 7f26938de640 1 --2- 172.21.15.16:0/3781439726 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f268c0b2980 0x7f268c0b2df0 unknown :-1 s=CLOSED pgs=1568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:51.449 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.445+0000 7f26938de640 1 --2- 172.21.15.16:0/3781439726 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f268c14e730 0x7f268c0c67b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:51.449 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.445+0000 7f26938de640 1 --2- 172.21.15.16:0/3781439726 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f2670065200 0x7f26700676b0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:51.449 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.445+0000 7f26938de640 1 --2- 172.21.15.16:0/3781439726 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f268c14e360 0x7f268c0c6270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:51.450 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.445+0000 7f26938de640 1 -- 172.21.15.16:0/3781439726 >> 172.21.15.16:0/3781439726 conn(0x7f268c069720 msgr2=0x7f268c069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:44:51.450 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.445+0000 7f26938de640 1 -- 172.21.15.16:0/3781439726 shutdown_connections 2023-05-22T15:44:51.450 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.445+0000 7f26938de640 1 -- 172.21.15.16:0/3781439726 wait complete. 2023-05-22T15:44:51.466 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-05-22T15:44:51.467 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-05-22T15:44:51.583 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.581+0000 7fdde4c1d640 1 Processor -- start 2023-05-22T15:44:51.583 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.581+0000 7fdde4c1d640 1 -- start start 2023-05-22T15:44:51.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.581+0000 7fdde4c1d640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fdde00b7dd0 0x7fdde00ccab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:51.587 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.581+0000 7fdde4c1d640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fdde0160250 0x7fdde00d0410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:51.587 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.581+0000 7fdde4c1d640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdde00cd170 con 0x7fdde015f700 2023-05-22T15:44:51.588 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.581+0000 7fdde4c1d640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdde00d3360 con 0x7fdde00b7dd0 2023-05-22T15:44:51.589 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.581+0000 7fdde4c1d640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdde00d2ac0 con 0x7fdde0160250 2023-05-22T15:44:51.589 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.581+0000 7fddde59c640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fdde00b7dd0 0x7fdde00ccab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:44:51.589 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.581+0000 7fddde59c640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fdde00b7dd0 0x7fdde00ccab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59308/0 (socket says 172.21.15.16:59308) 2023-05-22T15:44:51.589 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.581+0000 7fddde59c640 1 -- 172.21.15.16:0/3441038647 learned_addr learned my addr 172.21.15.16:0/3441038647 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:44:51.589 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.581+0000 7fddded9d640 1 --2- 172.21.15.16:0/3441038647 >> v2:172.21.15.62:3300/0 conn(0x7fdde0160250 0x7fdde00d0410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:44:51.589 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.581+0000 7fddde59c640 1 -- 172.21.15.16:0/3441038647 >> v2:172.21.15.62:3300/0 conn(0x7fdde0160250 msgr2=0x7fdde00d0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:44:51.589 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.581+0000 7fddde59c640 1 --2- 172.21.15.16:0/3441038647 >> v2:172.21.15.62:3300/0 conn(0x7fdde0160250 0x7fdde00d0410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:51.590 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.581+0000 7fddde59c640 1 -- 172.21.15.16:0/3441038647 >> v1:172.21.15.62:6789/0 conn(0x7fdde015f700 legacy=0x7fdde015fb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:44:51.590 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.581+0000 7fddde59c640 1 -- 172.21.15.16:0/3441038647 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdde00d2da0 con 0x7fdde00b7dd0 2023-05-22T15:44:51.590 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.581+0000 7fddded9d640 1 --2- 172.21.15.16:0/3441038647 >> v2:172.21.15.62:3300/0 conn(0x7fdde0160250 0x7fdde00d0410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:44:51.592 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.585+0000 7fddde59c640 1 --2- 172.21.15.16:0/3441038647 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdde00b7dd0 0x7fdde00ccab0 secure :-1 s=READY pgs=1569 cs=0 l=1 rev1=1 crypto rx=0x7fdde015c190 tx=0x7fddd001c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b7fdbc3c06ceba5 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:44:51.593 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.585+0000 7fddddd9b640 1 -- 172.21.15.16:0/3441038647 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddd00ab650 con 0x7fdde00b7dd0 2023-05-22T15:44:51.593 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.585+0000 7fddddd9b640 1 -- 172.21.15.16:0/3441038647 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddd00ab7b0 con 0x7fdde00b7dd0 2023-05-22T15:44:51.593 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.585+0000 7fddddd9b640 1 -- 172.21.15.16:0/3441038647 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddd00abb40 con 0x7fdde00b7dd0 2023-05-22T15:44:51.593 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.585+0000 7fdde4c1d640 1 -- 172.21.15.16:0/3441038647 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdde00b7dd0 msgr2=0x7fdde00ccab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:44:51.593 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.585+0000 7fdde4c1d640 1 --2- 172.21.15.16:0/3441038647 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdde00b7dd0 0x7fdde00ccab0 secure :-1 s=READY pgs=1569 cs=0 l=1 rev1=1 crypto rx=0x7fdde015c190 tx=0x7fddd001c490 comp rx=0 tx=0).stop 2023-05-22T15:44:51.593 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.585+0000 7fdde4c1d640 1 -- 172.21.15.16:0/3441038647 shutdown_connections 2023-05-22T15:44:51.593 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.585+0000 7fdde4c1d640 1 --2- 172.21.15.16:0/3441038647 >> v2:172.21.15.62:3300/0 conn(0x7fdde0160250 0x7fdde00d0410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:51.593 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.585+0000 7fdde4c1d640 1 --2- 172.21.15.16:0/3441038647 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdde00b7dd0 0x7fdde00ccab0 unknown :-1 s=CLOSED pgs=1569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:51.594 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.585+0000 7fdde4c1d640 1 -- 172.21.15.16:0/3441038647 >> 172.21.15.16:0/3441038647 conn(0x7fdde00655c0 msgr2=0x7fdde0070e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:44:51.594 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.585+0000 7fdde4c1d640 1 -- 172.21.15.16:0/3441038647 shutdown_connections 2023-05-22T15:44:51.594 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.585+0000 7fdde4c1d640 1 -- 172.21.15.16:0/3441038647 wait complete. 2023-05-22T15:44:51.594 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.585+0000 7fdde4c1d640 1 Processor -- start 2023-05-22T15:44:51.594 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.585+0000 7fdde4c1d640 1 -- start start 2023-05-22T15:44:51.594 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.585+0000 7fdde4c1d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdde00b7dd0 0x7fdde015de00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:51.594 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.585+0000 7fdde4c1d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdde015f700 0x7fdde0156d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:51.595 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.585+0000 7fdde4c1d640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fdde0160250 0x7fdde014a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:51.595 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.585+0000 7fdde4c1d640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdde00b5f90 con 0x7fdde015f700 2023-05-22T15:44:51.595 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.585+0000 7fdde4c1d640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdde00b6550 con 0x7fdde00b7dd0 2023-05-22T15:44:51.595 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.585+0000 7fdde4c1d640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdde01555f0 con 0x7fdde0160250 2023-05-22T15:44:51.595 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.589+0000 7fddd7fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdde015f700 0x7fdde0156d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:44:51.595 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.589+0000 7fddd7fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdde015f700 0x7fdde0156d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59310/0 (socket says 172.21.15.16:59310) 2023-05-22T15:44:51.595 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.589+0000 7fddd7fff640 1 -- 172.21.15.16:0/2663604853 learned_addr learned my addr 172.21.15.16:0/2663604853 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:44:51.595 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.589+0000 7fddde59c640 1 --2- 172.21.15.16:0/2663604853 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdde00b7dd0 0x7fdde015de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:44:51.596 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.589+0000 7fddded9d640 1 --2- 172.21.15.16:0/2663604853 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fdde0160250 0x7fdde014a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:44:51.596 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.589+0000 7fddd7fff640 1 -- 172.21.15.16:0/2663604853 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fdde0160250 msgr2=0x7fdde014a4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:44:51.596 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.589+0000 7fddd7fff640 1 --2- 172.21.15.16:0/2663604853 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fdde0160250 0x7fdde014a4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:51.596 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.589+0000 7fddd7fff640 1 -- 172.21.15.16:0/2663604853 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdde00b7dd0 msgr2=0x7fdde015de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:44:51.596 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.589+0000 7fddd7fff640 1 --2- 172.21.15.16:0/2663604853 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdde00b7dd0 0x7fdde015de00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:51.596 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.589+0000 7fddd7fff640 1 -- 172.21.15.16:0/2663604853 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddd00094d0 con 0x7fdde015f700 2023-05-22T15:44:51.596 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.589+0000 7fddd7fff640 1 --2- 172.21.15.16:0/2663604853 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdde015f700 0x7fdde0156d70 secure :-1 s=READY pgs=1570 cs=0 l=1 rev1=1 crypto rx=0x7fddc800c4c0 tx=0x7fddc8001e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:44:51.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.589+0000 7fddd6ffd640 1 -- 172.21.15.16:0/2663604853 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddc8005480 con 0x7fdde015f700 2023-05-22T15:44:51.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.589+0000 7fddd6ffd640 1 -- 172.21.15.16:0/2663604853 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddc8003600 con 0x7fdde015f700 2023-05-22T15:44:51.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.589+0000 7fdde4c1d640 1 -- 172.21.15.16:0/2663604853 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdde00d12f0 con 0x7fdde015f700 2023-05-22T15:44:51.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.589+0000 7fddd6ffd640 1 -- 172.21.15.16:0/2663604853 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddc8003990 con 0x7fdde015f700 2023-05-22T15:44:51.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.589+0000 7fdde4c1d640 1 -- 172.21.15.16:0/2663604853 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdde00d1160 con 0x7fdde015f700 2023-05-22T15:44:51.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.589+0000 7fdde4c1d640 1 -- 172.21.15.16:0/2663604853 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdde00d1480 con 0x7fdde015f700 2023-05-22T15:44:51.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.589+0000 7fddd6ffd640 1 -- 172.21.15.16:0/2663604853 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 49) v1 ==== 84068+0+0 (secure 0 0 0) 0x7fddc8003af0 con 0x7fdde015f700 2023-05-22T15:44:51.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.589+0000 7fddd6ffd640 1 --2- 172.21.15.16:0/2663604853 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fddc0065200 0x7fddc00676b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:51.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.589+0000 7fddd6ffd640 1 -- 172.21.15.16:0/2663604853 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 7070+0+0 (secure 0 0 0) 0x7fddc8081d50 con 0x7fdde015f700 2023-05-22T15:44:51.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.589+0000 7fddde59c640 1 --2- 172.21.15.16:0/2663604853 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fddc0065200 0x7fddc00676b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:44:51.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.589+0000 7fddde59c640 1 --2- 172.21.15.16:0/2663604853 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fddc0065200 0x7fddc00676b0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fddd00af000 tx=0x7fddd00a7000 comp rx=0 tx=0).ready entity=mgr.8304 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:44:51.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.593+0000 7fddd6ffd640 1 -- 172.21.15.16:0/2663604853 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fddc804f8c0 con 0x7fdde015f700 2023-05-22T15:44:51.782 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.777+0000 7fdde4c1d640 1 -- 172.21.15.16:0/2663604853 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdde004dbe0 con 0x7fdde015f700 2023-05-22T15:44:51.783 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.781+0000 7fddd6ffd640 1 -- 172.21.15.16:0/2663604853 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 43 v43) v1 ==== 94+0+4650 (secure 0 0 0) 0x7fddc8017020 con 0x7fdde015f700 2023-05-22T15:44:51.783 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:44:51.783 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":43,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6350,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6833/1554108010","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":1554108010},{"type":"v1","addr":"172.21.15.62:6833","nonce":1554108010}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":39},{"gid":6979,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/2667219028","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":2667219028},{"type":"v1","addr":"172.21.15.62:6835","nonce":2667219028}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":39},{"gid":8013,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6836/1892737356","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1892737356},{"type":"v1","addr":"172.21.15.16:6836","nonce":1892737356}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":39}],"filesystems":[{"mdsmap":{"epoch":43,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-05-22T15:43:30.134589+0000","modified":"2023-05-22T15:43:33.144847+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8016},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8016":{"gid":8016,"name":"a","rank":0,"incarnation":40,"state":"up:active","state_seq":3,"addr":"172.21.15.16:6837/2085936428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6835","nonce":2085936428},{"type":"v1","addr":"172.21.15.16:6837","nonce":2085936428}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2023-05-22T15:44:51.784 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 43 2023-05-22T15:44:51.785 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.781+0000 7fdde4c1d640 1 -- 172.21.15.16:0/2663604853 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fddc0065200 msgr2=0x7fddc00676b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:44:51.785 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.781+0000 7fdde4c1d640 1 --2- 172.21.15.16:0/2663604853 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fddc0065200 0x7fddc00676b0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fddd00af000 tx=0x7fddd00a7000 comp rx=0 tx=0).stop 2023-05-22T15:44:51.785 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.781+0000 7fdde4c1d640 1 -- 172.21.15.16:0/2663604853 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdde015f700 msgr2=0x7fdde0156d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:44:51.785 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.781+0000 7fdde4c1d640 1 --2- 172.21.15.16:0/2663604853 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdde015f700 0x7fdde0156d70 secure :-1 s=READY pgs=1570 cs=0 l=1 rev1=1 crypto rx=0x7fddc800c4c0 tx=0x7fddc8001e80 comp rx=0 tx=0).stop 2023-05-22T15:44:51.785 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.781+0000 7fdde4c1d640 1 -- 172.21.15.16:0/2663604853 shutdown_connections 2023-05-22T15:44:51.785 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.781+0000 7fdde4c1d640 1 --2- 172.21.15.16:0/2663604853 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fddc0065200 0x7fddc00676b0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:51.786 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.781+0000 7fdde4c1d640 1 --2- 172.21.15.16:0/2663604853 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fdde0160250 0x7fdde014a4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:51.786 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.781+0000 7fdde4c1d640 1 --2- 172.21.15.16:0/2663604853 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdde015f700 0x7fdde0156d70 unknown :-1 s=CLOSED pgs=1570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:51.786 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.781+0000 7fdde4c1d640 1 --2- 172.21.15.16:0/2663604853 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdde00b7dd0 0x7fdde015de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:51.786 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.781+0000 7fdde4c1d640 1 -- 172.21.15.16:0/2663604853 >> 172.21.15.16:0/2663604853 conn(0x7fdde00655c0 msgr2=0x7fdde00d21a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:44:51.786 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.781+0000 7fdde4c1d640 1 -- 172.21.15.16:0/2663604853 shutdown_connections 2023-05-22T15:44:51.786 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.781+0000 7fdde4c1d640 1 -- 172.21.15.16:0/2663604853 wait complete. 2023-05-22T15:44:51.801 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-05-22T15:44:51.911 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.909+0000 7fad03bea640 1 Processor -- start 2023-05-22T15:44:51.911 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.909+0000 7fad03bea640 1 -- start start 2023-05-22T15:44:51.912 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.909+0000 7fad03bea640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7facfc06c5d0 0x7facfc166ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:51.912 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.909+0000 7fad03bea640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7facfc06db40 con 0x7facfc0cbf80 2023-05-22T15:44:51.912 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.909+0000 7fad03bea640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7facfc0cf3f0 con 0x7facfc0b22e0 2023-05-22T15:44:51.912 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.909+0000 7fad03bea640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7facfc15c080 con 0x7facfc06c5d0 2023-05-22T15:44:51.912 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.909+0000 7fad01986640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7facfc0cbf80 0x7facfc06b210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:52018/0 (socket says 172.21.15.16:52018) 2023-05-22T15:44:51.913 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.909+0000 7fad01986640 1 -- 172.21.15.16:0/1385311189 learned_addr learned my addr 172.21.15.16:0/1385311189 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:44:51.914 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.909+0000 7fad02187640 1 --2- 172.21.15.16:0/1385311189 >> v2:172.21.15.62:3300/0 conn(0x7facfc06c5d0 0x7facfc166ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:44:51.914 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.909+0000 7fad00984640 1 -- 172.21.15.16:0/1385311189 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 983001593 0 0) 0x7facfc06db40 con 0x7facfc0cbf80 2023-05-22T15:44:51.914 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.909+0000 7fad00984640 1 -- 172.21.15.16:0/1385311189 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7face8003530 con 0x7facfc0cbf80 2023-05-22T15:44:51.914 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.909+0000 7fad00984640 1 -- 172.21.15.16:0/1385311189 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2921559159 0 0) 0x7face8003530 con 0x7facfc0cbf80 2023-05-22T15:44:51.915 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.909+0000 7fad00984640 1 -- 172.21.15.16:0/1385311189 >> v2:172.21.15.62:3300/0 conn(0x7facfc06c5d0 msgr2=0x7facfc166ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:44:51.915 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.909+0000 7fad00984640 1 --2- 172.21.15.16:0/1385311189 >> v2:172.21.15.62:3300/0 conn(0x7facfc06c5d0 0x7facfc166ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:51.915 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.909+0000 7fad00984640 1 -- 172.21.15.16:0/1385311189 >> v1:172.21.15.62:6789/0 conn(0x7facfc0b22e0 legacy=0x7facfc06bec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:44:51.915 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.909+0000 7fad00984640 1 -- 172.21.15.16:0/1385311189 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facfc06ddc0 con 0x7facfc0cbf80 2023-05-22T15:44:51.915 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.909+0000 7fad00984640 1 -- 172.21.15.16:0/1385311189 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7facf8001680 con 0x7facfc0cbf80 2023-05-22T15:44:51.915 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.909+0000 7fad00984640 1 -- 172.21.15.16:0/1385311189 >> v1:172.21.15.16:6789/0 conn(0x7facfc0cbf80 legacy=0x7facfc06b210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:44:51.915 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.909+0000 7fad00984640 1 --2- 172.21.15.16:0/1385311189 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7face8003980 0x7face8003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:51.915 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.909+0000 7fad00984640 1 --2- 172.21.15.16:0/1385311189 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7face80047d0 0x7face8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:51.916 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.909+0000 7fad00984640 1 --2- 172.21.15.16:0/1385311189 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7face8005180 0x7face800b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:51.916 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.909+0000 7fad00984640 1 -- 172.21.15.16:0/1385311189 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facfc15c080 con 0x7face8005180 2023-05-22T15:44:51.916 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.909+0000 7fad00984640 1 -- 172.21.15.16:0/1385311189 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7face800c090 con 0x7face8003980 2023-05-22T15:44:51.916 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.909+0000 7fad00984640 1 -- 172.21.15.16:0/1385311189 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7face800c250 con 0x7face80047d0 2023-05-22T15:44:51.916 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.909+0000 7fad02187640 1 --2- 172.21.15.16:0/1385311189 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7face8005180 0x7face800b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:44:51.916 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.909+0000 7fad01986640 1 --2- 172.21.15.16:0/1385311189 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7face8003980 0x7face8003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:44:51.917 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.909+0000 7fad01185640 1 --2- 172.21.15.16:0/1385311189 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7face80047d0 0x7face8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:44:51.917 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.909+0000 7fad02187640 1 -- 172.21.15.16:0/1385311189 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7face80047d0 msgr2=0x7face8004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:44:51.917 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.909+0000 7fad02187640 1 --2- 172.21.15.16:0/1385311189 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7face80047d0 0x7face8004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:51.917 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.909+0000 7fad02187640 1 -- 172.21.15.16:0/1385311189 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7face8003980 msgr2=0x7face8003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:44:51.917 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.909+0000 7fad02187640 1 --2- 172.21.15.16:0/1385311189 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7face8003980 0x7face8003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:51.918 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.909+0000 7fad02187640 1 -- 172.21.15.16:0/1385311189 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7face8003530 con 0x7face8005180 2023-05-22T15:44:51.918 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.909+0000 7fad01986640 1 --2- 172.21.15.16:0/1385311189 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7face8003980 0x7face8003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:44:51.920 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.913+0000 7fad02187640 1 -- 172.21.15.16:0/1385311189 reap_dead start 2023-05-22T15:44:51.921 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.913+0000 7fad02187640 1 --2- 172.21.15.16:0/1385311189 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7face8005180 0x7face800b990 secure :-1 s=READY pgs=1572 cs=0 l=1 rev1=1 crypto rx=0x7facf4009f90 tx=0x7facf401cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2de0cda64c3857fd server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:44:51.921 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.913+0000 7fad00984640 1 -- 172.21.15.16:0/1385311189 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facf40a5c70 con 0x7face8005180 2023-05-22T15:44:51.921 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.913+0000 7fad00984640 1 -- 172.21.15.16:0/1385311189 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facf8001d80 con 0x7face8005180 2023-05-22T15:44:51.921 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.913+0000 7fad00984640 1 -- 172.21.15.16:0/1385311189 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facf40039a0 con 0x7face8005180 2023-05-22T15:44:51.921 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.913+0000 7fad03bea640 1 -- 172.21.15.16:0/1385311189 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7face8005180 msgr2=0x7face800b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:44:51.921 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.913+0000 7fad03bea640 1 --2- 172.21.15.16:0/1385311189 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7face8005180 0x7face800b990 secure :-1 s=READY pgs=1572 cs=0 l=1 rev1=1 crypto rx=0x7facf4009f90 tx=0x7facf401cc60 comp rx=0 tx=0).stop 2023-05-22T15:44:51.921 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.913+0000 7fad03bea640 1 -- 172.21.15.16:0/1385311189 shutdown_connections 2023-05-22T15:44:51.922 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.913+0000 7fad03bea640 1 --2- 172.21.15.16:0/1385311189 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7face8005180 0x7face800b990 unknown :-1 s=CLOSED pgs=1572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:51.922 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.913+0000 7fad03bea640 1 -- 172.21.15.16:0/1385311189 >> 172.21.15.16:0/1385311189 conn(0x7facfc06f560 msgr2=0x7facfc06f960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:44:51.922 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.913+0000 7fad03bea640 1 -- 172.21.15.16:0/1385311189 shutdown_connections 2023-05-22T15:44:51.922 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.913+0000 7fad03bea640 1 -- 172.21.15.16:0/1385311189 wait complete. 2023-05-22T15:44:51.922 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.913+0000 7fad03bea640 1 Processor -- start 2023-05-22T15:44:51.922 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.913+0000 7fad03bea640 1 -- start start 2023-05-22T15:44:51.922 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.913+0000 7fad03bea640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7face8005180 0x7facfc153ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:51.922 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.913+0000 7fad03bea640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7facfc0c9ed0 0x7facfc0b4af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:51.923 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.913+0000 7fad03bea640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7facfc150f40 0x7facfc1513b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:51.923 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.913+0000 7fad03bea640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facfc1518f0 con 0x7facfc150f40 2023-05-22T15:44:51.923 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.913+0000 7fad03bea640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facfc0b5030 con 0x7facfc0c9ed0 2023-05-22T15:44:51.923 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.913+0000 7fad03bea640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7facfc0b52b0 con 0x7face8005180 2023-05-22T15:44:51.923 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.913+0000 7fad02187640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7facfc150f40 0x7facfc1513b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:44:51.923 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.913+0000 7fad02187640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7facfc150f40 0x7facfc1513b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59328/0 (socket says 172.21.15.16:59328) 2023-05-22T15:44:51.924 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.913+0000 7fad02187640 1 -- 172.21.15.16:0/1559620874 learned_addr learned my addr 172.21.15.16:0/1559620874 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:44:51.924 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.913+0000 7fad01986640 1 --2- 172.21.15.16:0/1559620874 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7face8005180 0x7facfc153ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:44:51.924 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.913+0000 7fad01185640 1 --2- 172.21.15.16:0/1559620874 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7facfc0c9ed0 0x7facfc0b4af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:44:51.924 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.913+0000 7fad02187640 1 -- 172.21.15.16:0/1559620874 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7face8005180 msgr2=0x7facfc153ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:44:51.924 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.913+0000 7fad02187640 1 --2- 172.21.15.16:0/1559620874 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7face8005180 0x7facfc153ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:51.925 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.913+0000 7fad02187640 1 -- 172.21.15.16:0/1559620874 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7facfc0c9ed0 msgr2=0x7facfc0b4af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:44:51.925 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.913+0000 7fad02187640 1 --2- 172.21.15.16:0/1559620874 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7facfc0c9ed0 0x7facfc0b4af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:51.925 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.913+0000 7fad02187640 1 -- 172.21.15.16:0/1559620874 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facfc06e380 con 0x7facfc150f40 2023-05-22T15:44:51.925 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.917+0000 7fad01185640 1 --2- 172.21.15.16:0/1559620874 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7facfc0c9ed0 0x7facfc0b4af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:44:51.926 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.917+0000 7fad02187640 1 --2- 172.21.15.16:0/1559620874 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7facfc150f40 0x7facfc1513b0 secure :-1 s=READY pgs=1573 cs=0 l=1 rev1=1 crypto rx=0x7facf401ca90 tx=0x7facf4009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:44:51.926 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.917+0000 7facf27fc640 1 -- 172.21.15.16:0/1559620874 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facf40039a0 con 0x7facfc150f40 2023-05-22T15:44:51.926 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.917+0000 7facf27fc640 1 -- 172.21.15.16:0/1559620874 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facf400f040 con 0x7facfc150f40 2023-05-22T15:44:51.926 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.917+0000 7facf27fc640 1 -- 172.21.15.16:0/1559620874 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facf4003d60 con 0x7facfc150f40 2023-05-22T15:44:51.926 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.917+0000 7fad03bea640 1 -- 172.21.15.16:0/1559620874 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7facfc06e0a0 con 0x7facfc150f40 2023-05-22T15:44:51.927 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.917+0000 7fad03bea640 1 -- 172.21.15.16:0/1559620874 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7facfc0cc350 con 0x7facfc150f40 2023-05-22T15:44:51.927 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.917+0000 7fad03bea640 1 -- 172.21.15.16:0/1559620874 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7facfc04dbe0 con 0x7facfc150f40 2023-05-22T15:44:51.927 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.917+0000 7facf27fc640 1 -- 172.21.15.16:0/1559620874 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 49) v1 ==== 84068+0+0 (secure 0 0 0) 0x7facf4003ec0 con 0x7facfc150f40 2023-05-22T15:44:51.928 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.917+0000 7facf27fc640 1 --2- 172.21.15.16:0/1559620874 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7face00651d0 0x7face0067680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:51.928 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.917+0000 7facf27fc640 1 -- 172.21.15.16:0/1559620874 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 7070+0+0 (secure 0 0 0) 0x7facf411d050 con 0x7facfc150f40 2023-05-22T15:44:51.928 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.917+0000 7fad01986640 1 --2- 172.21.15.16:0/1559620874 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7face00651d0 0x7face0067680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:44:51.928 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.917+0000 7fad01986640 1 --2- 172.21.15.16:0/1559620874 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7face00651d0 0x7face0067680 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7facf80017b0 tx=0x7facf8001480 comp rx=0 tx=0).ready entity=mgr.8304 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:44:51.929 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:51.921+0000 7facf27fc640 1 -- 172.21.15.16:0/1559620874 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7facf40eabc0 con 0x7facfc150f40 2023-05-22T15:44:52.103 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:52.101+0000 7fad03bea640 1 -- 172.21.15.16:0/1559620874 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7facfc0b61e0 con 0x7facfc150f40 2023-05-22T15:44:52.103 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:52.101+0000 7facf27fc640 1 -- 172.21.15.16:0/1559620874 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 43 v43) v1 ==== 94+0+4650 (secure 0 0 0) 0x7facf40ee1e0 con 0x7facfc150f40 2023-05-22T15:44:52.104 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:44:52.104 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":43,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6350,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6833/1554108010","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":1554108010},{"type":"v1","addr":"172.21.15.62:6833","nonce":1554108010}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":39},{"gid":6979,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/2667219028","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":2667219028},{"type":"v1","addr":"172.21.15.62:6835","nonce":2667219028}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":39},{"gid":8013,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6836/1892737356","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1892737356},{"type":"v1","addr":"172.21.15.16:6836","nonce":1892737356}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":39}],"filesystems":[{"mdsmap":{"epoch":43,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-05-22T15:43:30.134589+0000","modified":"2023-05-22T15:43:33.144847+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8016},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8016":{"gid":8016,"name":"a","rank":0,"incarnation":40,"state":"up:active","state_seq":3,"addr":"172.21.15.16:6837/2085936428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6835","nonce":2085936428},{"type":"v1","addr":"172.21.15.16:6837","nonce":2085936428}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2023-05-22T15:44:52.104 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 43 2023-05-22T15:44:52.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:52.101+0000 7fad03bea640 1 -- 172.21.15.16:0/1559620874 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7face00651d0 msgr2=0x7face0067680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:44:52.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:52.101+0000 7fad03bea640 1 --2- 172.21.15.16:0/1559620874 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7face00651d0 0x7face0067680 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7facf80017b0 tx=0x7facf8001480 comp rx=0 tx=0).stop 2023-05-22T15:44:52.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:52.101+0000 7fad03bea640 1 -- 172.21.15.16:0/1559620874 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7facfc150f40 msgr2=0x7facfc1513b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:44:52.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:52.101+0000 7fad03bea640 1 --2- 172.21.15.16:0/1559620874 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7facfc150f40 0x7facfc1513b0 secure :-1 s=READY pgs=1573 cs=0 l=1 rev1=1 crypto rx=0x7facf401ca90 tx=0x7facf4009750 comp rx=0 tx=0).stop 2023-05-22T15:44:52.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:52.101+0000 7fad03bea640 1 -- 172.21.15.16:0/1559620874 shutdown_connections 2023-05-22T15:44:52.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:52.101+0000 7fad03bea640 1 --2- 172.21.15.16:0/1559620874 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7facfc150f40 0x7facfc1513b0 unknown :-1 s=CLOSED pgs=1573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:52.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:52.101+0000 7fad03bea640 1 --2- 172.21.15.16:0/1559620874 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7facfc0c9ed0 0x7facfc0b4af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:52.107 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:52.101+0000 7fad03bea640 1 --2- 172.21.15.16:0/1559620874 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7face00651d0 0x7face0067680 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:52.107 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:52.101+0000 7fad03bea640 1 --2- 172.21.15.16:0/1559620874 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7face8005180 0x7facfc153ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:52.107 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:52.101+0000 7fad03bea640 1 -- 172.21.15.16:0/1559620874 >> 172.21.15.16:0/1559620874 conn(0x7facfc06f560 msgr2=0x7facfc06f930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:44:52.107 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:52.101+0000 7fad03bea640 1 -- 172.21.15.16:0/1559620874 shutdown_connections 2023-05-22T15:44:52.107 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:52.101+0000 7fad03bea640 1 -- 172.21.15.16:0/1559620874 wait complete. 2023-05-22T15:44:52.121 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-05-22T15:44:52.242 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:52.237+0000 7f98d18a7640 1 Processor -- start 2023-05-22T15:44:52.242 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:52.237+0000 7f98d18a7640 1 -- start start 2023-05-22T15:44:52.243 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:52.241+0000 7f98d18a7640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f98cc159cd0 0x7f98cc15a0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:52.243 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:52.241+0000 7f98d18a7640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f98cc0cc6f0 0x7f98cc0cfe30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:52.243 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:52.241+0000 7f98d18a7640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98cc0cf9a0 con 0x7f98cc0b2880 2023-05-22T15:44:52.243 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:52.241+0000 7f98d18a7640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98cc15aba0 con 0x7f98cc0cc6f0 2023-05-22T15:44:52.244 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:52.241+0000 7f98d18a7640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98cc15ae80 con 0x7f98cc159cd0 2023-05-22T15:44:52.244 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:52.241+0000 7f98cb7fe640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f98cc0cc6f0 0x7f98cc0cfe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:44:52.244 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:52.241+0000 7f98cb7fe640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f98cc0cc6f0 0x7f98cc0cfe30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59334/0 (socket says 172.21.15.16:59334) 2023-05-22T15:44:52.244 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:52.241+0000 7f98cb7fe640 1 -- 172.21.15.16:0/2499537262 learned_addr learned my addr 172.21.15.16:0/2499537262 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:44:52.245 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:52.241+0000 7f98caffd640 1 --2- 172.21.15.16:0/2499537262 >> v2:172.21.15.62:3300/0 conn(0x7f98cc159cd0 0x7f98cc15a0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:44:52.245 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:52.241+0000 7f98cb7fe640 1 -- 172.21.15.16:0/2499537262 >> v2:172.21.15.62:3300/0 conn(0x7f98cc159cd0 msgr2=0x7f98cc15a0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:44:52.245 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:52.241+0000 7f98cb7fe640 1 --2- 172.21.15.16:0/2499537262 >> v2:172.21.15.62:3300/0 conn(0x7f98cc159cd0 0x7f98cc15a0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:52.245 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:52.241+0000 7f98cb7fe640 1 -- 172.21.15.16:0/2499537262 >> v1:172.21.15.62:6789/0 conn(0x7f98cc0b2880 legacy=0x7f98cc0cbfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:44:52.245 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:52.241+0000 7f98cb7fe640 1 -- 172.21.15.16:0/2499537262 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98cc15b160 con 0x7f98cc0cc6f0 2023-05-22T15:44:52.246 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:52.241+0000 7f98cb7fe640 1 --2- 172.21.15.16:0/2499537262 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f98cc0cc6f0 0x7f98cc0cfe30 secure :-1 s=READY pgs=1574 cs=0 l=1 rev1=1 crypto rx=0x7f98cc0d1330 tx=0x7f98c401c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8fe6502873d3b5a server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:44:52.247 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:52.245+0000 7f98c9ffb640 1 -- 172.21.15.16:0/2499537262 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98c40a5730 con 0x7f98cc0cc6f0 2023-05-22T15:44:52.247 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:52.245+0000 7f98c9ffb640 1 -- 172.21.15.16:0/2499537262 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98c40a5890 con 0x7f98cc0cc6f0 2023-05-22T15:44:52.247 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:52.245+0000 7f98c9ffb640 1 -- 172.21.15.16:0/2499537262 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98c40a5c20 con 0x7f98cc0cc6f0 2023-05-22T15:44:52.247 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:52.245+0000 7f98d18a7640 1 -- 172.21.15.16:0/2499537262 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f98cc0cc6f0 msgr2=0x7f98cc0cfe30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:44:52.247 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:52.245+0000 7f98d18a7640 1 --2- 172.21.15.16:0/2499537262 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f98cc0cc6f0 0x7f98cc0cfe30 secure :-1 s=READY pgs=1574 cs=0 l=1 rev1=1 crypto rx=0x7f98cc0d1330 tx=0x7f98c401c960 comp rx=0 tx=0).stop 2023-05-22T15:44:52.248 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:52.245+0000 7f98d18a7640 1 -- 172.21.15.16:0/2499537262 shutdown_connections 2023-05-22T15:44:52.248 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:52.245+0000 7f98d18a7640 1 --2- 172.21.15.16:0/2499537262 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f98cc0cc6f0 0x7f98cc0cfe30 unknown :-1 s=CLOSED pgs=1574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:52.248 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:52.245+0000 7f98d18a7640 1 --2- 172.21.15.16:0/2499537262 >> v2:172.21.15.62:3300/0 conn(0x7f98cc159cd0 0x7f98cc15a0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:52.248 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:52.245+0000 7f98d18a7640 1 -- 172.21.15.16:0/2499537262 >> 172.21.15.16:0/2499537262 conn(0x7f98cc06f5c0 msgr2=0x7f98cc06f9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:44:52.248 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:52.245+0000 7f98d18a7640 1 -- 172.21.15.16:0/2499537262 shutdown_connections 2023-05-22T15:44:52.248 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:52.245+0000 7f98d18a7640 1 -- 172.21.15.16:0/2499537262 wait complete. 2023-05-22T15:44:52.250 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:52.245+0000 7f98d18a7640 1 Processor -- start 2023-05-22T15:44:52.250 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:52.245+0000 7f98d18a7640 1 -- start start 2023-05-22T15:44:52.250 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:52.245+0000 7f98d18a7640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f98cc0b2880 0x7f98cc151ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:52.250 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:52.245+0000 7f98d18a7640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f98cc0cc6f0 0x7f98cc0c96a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:52.251 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:52.245+0000 7f98d18a7640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f98cc159cd0 0x7f98cc0c9be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:52.251 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:52.245+0000 7f98d18a7640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98cc155100 con 0x7f98cc0b2880 2023-05-22T15:44:52.251 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:52.245+0000 7f98d18a7640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98cc0ca120 con 0x7f98cc0cc6f0 2023-05-22T15:44:52.251 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:52.245+0000 7f98d18a7640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98cc1526f0 con 0x7f98cc159cd0 2023-05-22T15:44:52.251 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:52.245+0000 7f98caffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f98cc0b2880 0x7f98cc151ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:44:52.251 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:52.245+0000 7f98caffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f98cc0b2880 0x7f98cc151ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59348/0 (socket says 172.21.15.16:59348) 2023-05-22T15:44:52.251 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:52.245+0000 7f98caffd640 1 -- 172.21.15.16:0/1477284056 learned_addr learned my addr 172.21.15.16:0/1477284056 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:44:52.252 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:52.245+0000 7f98ca7fc640 1 --2- 172.21.15.16:0/1477284056 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f98cc0cc6f0 0x7f98cc0c96a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:44:52.252 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:52.245+0000 7f98cb7fe640 1 --2- 172.21.15.16:0/1477284056 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f98cc159cd0 0x7f98cc0c9be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:44:52.252 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:52.245+0000 7f98caffd640 1 -- 172.21.15.16:0/1477284056 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f98cc159cd0 msgr2=0x7f98cc0c9be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:44:52.252 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:52.245+0000 7f98caffd640 1 --2- 172.21.15.16:0/1477284056 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f98cc159cd0 0x7f98cc0c9be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:52.252 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:52.245+0000 7f98caffd640 1 -- 172.21.15.16:0/1477284056 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f98cc0cc6f0 msgr2=0x7f98cc0c96a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:44:52.252 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:52.245+0000 7f98caffd640 1 --2- 172.21.15.16:0/1477284056 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f98cc0cc6f0 0x7f98cc0c96a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:52.252 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:52.245+0000 7f98caffd640 1 -- 172.21.15.16:0/1477284056 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98c4009b30 con 0x7f98cc0b2880 2023-05-22T15:44:52.252 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:52.245+0000 7f98ca7fc640 1 --2- 172.21.15.16:0/1477284056 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f98cc0cc6f0 0x7f98cc0c96a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:44:52.253 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:52.245+0000 7f98caffd640 1 --2- 172.21.15.16:0/1477284056 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f98cc0b2880 0x7f98cc151ea0 secure :-1 s=READY pgs=1575 cs=0 l=1 rev1=1 crypto rx=0x7f98bc00d830 tx=0x7f98bc00dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:44:52.253 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:52.245+0000 7f98b3fff640 1 -- 172.21.15.16:0/1477284056 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98bc00b6c0 con 0x7f98cc0b2880 2023-05-22T15:44:52.253 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:52.245+0000 7f98b3fff640 1 -- 172.21.15.16:0/1477284056 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98bc00b820 con 0x7f98cc0b2880 2023-05-22T15:44:52.253 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:52.245+0000 7f98d18a7640 1 -- 172.21.15.16:0/1477284056 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98cc0d0d10 con 0x7f98cc0b2880 2023-05-22T15:44:52.253 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:52.245+0000 7f98b3fff640 1 -- 172.21.15.16:0/1477284056 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98bc00bb90 con 0x7f98cc0b2880 2023-05-22T15:44:52.253 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:52.245+0000 7f98d18a7640 1 -- 172.21.15.16:0/1477284056 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98cc0d0b80 con 0x7f98cc0b2880 2023-05-22T15:44:52.254 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:52.249+0000 7f98d18a7640 1 -- 172.21.15.16:0/1477284056 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98cc0d0ea0 con 0x7f98cc0b2880 2023-05-22T15:44:52.254 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:52.249+0000 7f98b3fff640 1 -- 172.21.15.16:0/1477284056 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 49) v1 ==== 84068+0+0 (secure 0 0 0) 0x7f98bc01e460 con 0x7f98cc0b2880 2023-05-22T15:44:52.254 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:52.249+0000 7f98b3fff640 1 --2- 172.21.15.16:0/1477284056 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f98b8065200 0x7f98b80676b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:52.254 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:52.249+0000 7f98b3fff640 1 -- 172.21.15.16:0/1477284056 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 7070+0+0 (secure 0 0 0) 0x7f98bc083050 con 0x7f98cc0b2880 2023-05-22T15:44:52.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:52.249+0000 7f98ca7fc640 1 --2- 172.21.15.16:0/1477284056 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f98b8065200 0x7f98b80676b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:44:52.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:52.249+0000 7f98ca7fc640 1 --2- 172.21.15.16:0/1477284056 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f98b8065200 0x7f98b80676b0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f98cc156e30 tx=0x7f98c0008040 comp rx=0 tx=0).ready entity=mgr.8304 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:44:52.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:52.253+0000 7f98b3fff640 1 -- 172.21.15.16:0/1477284056 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f98bc050bc0 con 0x7f98cc0b2880 2023-05-22T15:44:52.309 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-05-22T15:44:52.311 DEBUG:teuthology.orchestra.run.smithi062:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-05-22T15:44:52.402 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:52.397+0000 7f98d18a7640 1 -- 172.21.15.16:0/1477284056 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f98cc04dbe0 con 0x7f98cc0b2880 2023-05-22T15:44:52.531 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:52.401+0000 7f98b3fff640 1 -- 172.21.15.16:0/1477284056 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v120) v1 ==== 74+0+18973 (secure 0 0 0) 0x7f98bc0541e0 con 0x7f98cc0b2880 2023-05-22T15:44:52.532 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:44:52.532 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":120,"fsid":"b2719834-7847-4d98-90a6-f65d57f85c49","created":"2023-05-22T15:26:25.259314+0000","modified":"2023-05-22T15:44:50.039971+0000","last_up_change":"2023-05-22T15:26:36.448524+0000","last_in_change":"2023-05-22T15:26:27.675668+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-05-22T15:26:38.019781+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-05-22T15:26:43.570514+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2023-05-22T15:43:28.684367+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"117","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2023-05-22T15:43:29.119294+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"118","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9e7ff854-b25f-4cbb-a5c2-1799d9c53d2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":37021},{"type":"v1","addr":"172.21.15.16:6825","nonce":37021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":37021},{"type":"v1","addr":"172.21.15.16:6827","nonce":37021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":37021},{"type":"v1","addr":"172.21.15.16:6831","nonce":37021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":37021},{"type":"v1","addr":"172.21.15.16:6829","nonce":37021}]},"public_addr":"172.21.15.16:6825/37021","cluster_addr":"172.21.15.16:6827/37021","heartbeat_back_addr":"172.21.15.16:6831/37021","heartbeat_front_addr":"172.21.15.16:6829/37021","state":["exists","up"]},{"osd":1,"uuid":"4702fbc9-a927-4565-aca5-41dc69cc8264","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":37008},{"type":"v1","addr":"172.21.15.16:6801","nonce":37008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":37008},{"type":"v1","addr":"172.21.15.16:6803","nonce":37008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":37008},{"type":"v1","addr":"172.21.15.16:6807","nonce":37008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":37008},{"type":"v1","addr":"172.21.15.16:6805","nonce":37008}]},"public_addr":"172.21.15.16:6801/37008","cluster_addr":"172.21.15.16:6803/37008","heartbeat_back_addr":"172.21.15.16:6807/37008","heartbeat_front_addr":"172.21.15.16:6805/37008","state":["exists","up"]},{"osd":2,"uuid":"1e7bba46-4eba-47ef-bb81-5e2d371638ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":37009},{"type":"v1","addr":"172.21.15.16:6809","nonce":37009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":37009},{"type":"v1","addr":"172.21.15.16:6811","nonce":37009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":37009},{"type":"v1","addr":"172.21.15.16:6815","nonce":37009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":37009},{"type":"v1","addr":"172.21.15.16:6813","nonce":37009}]},"public_addr":"172.21.15.16:6809/37009","cluster_addr":"172.21.15.16:6811/37009","heartbeat_back_addr":"172.21.15.16:6815/37009","heartbeat_front_addr":"172.21.15.16:6813/37009","state":["exists","up"]},{"osd":3,"uuid":"383cacc0-5a88-4904-990e-498ca59ef48a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":37020},{"type":"v1","addr":"172.21.15.16:6817","nonce":37020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":37020},{"type":"v1","addr":"172.21.15.16:6819","nonce":37020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":37020},{"type":"v1","addr":"172.21.15.16:6823","nonce":37020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":37020},{"type":"v1","addr":"172.21.15.16:6821","nonce":37020}]},"public_addr":"172.21.15.16:6817/37020","cluster_addr":"172.21.15.16:6819/37020","heartbeat_back_addr":"172.21.15.16:6823/37020","heartbeat_front_addr":"172.21.15.16:6821/37020","state":["exists","up"]},{"osd":4,"uuid":"970500ef-95b7-42c1-b080-eed4fae7013d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":37353},{"type":"v1","addr":"172.21.15.62:6817","nonce":37353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6819","nonce":37353},{"type":"v1","addr":"172.21.15.62:6822","nonce":37353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37353},{"type":"v1","addr":"172.21.15.62:6831","nonce":37353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37353},{"type":"v1","addr":"172.21.15.62:6826","nonce":37353}]},"public_addr":"172.21.15.62:6817/37353","cluster_addr":"172.21.15.62:6822/37353","heartbeat_back_addr":"172.21.15.62:6831/37353","heartbeat_front_addr":"172.21.15.62:6826/37353","state":["exists","up"]},{"osd":5,"uuid":"022d67db-d4d8-445b-b1b9-8b446a00f5c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":37352},{"type":"v1","addr":"172.21.15.62:6801","nonce":37352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":37352},{"type":"v1","addr":"172.21.15.62:6803","nonce":37352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":37352},{"type":"v1","addr":"172.21.15.62:6807","nonce":37352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":37352},{"type":"v1","addr":"172.21.15.62:6805","nonce":37352}]},"public_addr":"172.21.15.62:6801/37352","cluster_addr":"172.21.15.62:6803/37352","heartbeat_back_addr":"172.21.15.62:6807/37352","heartbeat_front_addr":"172.21.15.62:6805/37352","state":["exists","up"]},{"osd":6,"uuid":"42bd372b-6688-4a1b-8407-e16e8c266e33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":37354},{"type":"v1","addr":"172.21.15.62:6820","nonce":37354}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":37354},{"type":"v1","addr":"172.21.15.62:6823","nonce":37354}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6829","nonce":37354},{"type":"v1","addr":"172.21.15.62:6830","nonce":37354}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6825","nonce":37354},{"type":"v1","addr":"172.21.15.62:6827","nonce":37354}]},"public_addr":"172.21.15.62:6820/37354","cluster_addr":"172.21.15.62:6823/37354","heartbeat_back_addr":"172.21.15.62:6830/37354","heartbeat_front_addr":"172.21.15.62:6827/37354","state":["exists","up"]},{"osd":7,"uuid":"ac70b2a3-f41b-4bf2-b227-7f2cbb55426d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37355},{"type":"v1","addr":"172.21.15.62:6809","nonce":37355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37355},{"type":"v1","addr":"172.21.15.62:6811","nonce":37355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":37355},{"type":"v1","addr":"172.21.15.62:6815","nonce":37355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":37355},{"type":"v1","addr":"172.21.15.62:6813","nonce":37355}]},"public_addr":"172.21.15.62:6809/37355","cluster_addr":"172.21.15.62:6811/37355","heartbeat_back_addr":"172.21.15.62:6815/37355","heartbeat_front_addr":"172.21.15.62:6813/37355","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.368896+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.107273+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.276888+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.287502+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.124791+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.251937+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.255679+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.232797+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:0/2189763318":"2023-05-23T15:44:50.039898+0000","172.21.15.16:0/384246348":"2023-05-23T15:44:50.039898+0000","172.21.15.16:0/3752134468":"2023-05-23T15:44:50.039898+0000","172.21.15.16:0/2658406406":"2023-05-23T15:44:50.039898+0000","172.21.15.16:0/2218849021":"2023-05-23T15:43:43.641715+0000","172.21.15.16:0/1980969896":"2023-05-23T15:44:50.039898+0000","172.21.15.16:0/2339322867":"2023-05-23T15:43:43.641715+0000","172.21.15.16:0/155043049":"2023-05-23T15:43:43.641715+0000","172.21.15.16:0/4168497577":"2023-05-23T15:43:43.641715+0000","172.21.15.16:0/3302517394":"2023-05-23T15:43:43.641715+0000","172.21.15.16:0/4100157757":"2023-05-23T15:43:43.641715+0000","172.21.15.16:0/2834828521":"2023-05-23T15:44:50.039898+0000","172.21.15.16:0/579180365":"2023-05-23T15:44:50.039898+0000","172.21.15.16:0/177878289":"2023-05-23T15:44:50.039898+0000","172.21.15.16:6833/36921":"2023-05-23T15:43:43.641715+0000","172.21.15.16:6832/36921":"2023-05-23T15:43:43.641715+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-05-22T15:44:52.534 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:52.405+0000 7f98d18a7640 1 -- 172.21.15.16:0/1477284056 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f98b8065200 msgr2=0x7f98b80676b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:44:52.534 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:52.405+0000 7f98d18a7640 1 --2- 172.21.15.16:0/1477284056 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f98b8065200 0x7f98b80676b0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f98cc156e30 tx=0x7f98c0008040 comp rx=0 tx=0).stop 2023-05-22T15:44:52.534 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:52.405+0000 7f98d18a7640 1 -- 172.21.15.16:0/1477284056 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f98cc0b2880 msgr2=0x7f98cc151ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:44:52.534 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:52.405+0000 7f98d18a7640 1 --2- 172.21.15.16:0/1477284056 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f98cc0b2880 0x7f98cc151ea0 secure :-1 s=READY pgs=1575 cs=0 l=1 rev1=1 crypto rx=0x7f98bc00d830 tx=0x7f98bc00dcf0 comp rx=0 tx=0).stop 2023-05-22T15:44:52.535 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:52.405+0000 7f98d18a7640 1 -- 172.21.15.16:0/1477284056 shutdown_connections 2023-05-22T15:44:52.535 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:52.405+0000 7f98d18a7640 1 --2- 172.21.15.16:0/1477284056 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f98b8065200 0x7f98b80676b0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:52.535 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:52.405+0000 7f98d18a7640 1 --2- 172.21.15.16:0/1477284056 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f98cc159cd0 0x7f98cc0c9be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:52.535 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:52.405+0000 7f98d18a7640 1 --2- 172.21.15.16:0/1477284056 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f98cc0cc6f0 0x7f98cc0c96a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:52.535 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:52.405+0000 7f98d18a7640 1 --2- 172.21.15.16:0/1477284056 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f98cc0b2880 0x7f98cc151ea0 unknown :-1 s=CLOSED pgs=1575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:52.536 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:52.405+0000 7f98d18a7640 1 -- 172.21.15.16:0/1477284056 >> 172.21.15.16:0/1477284056 conn(0x7f98cc06f5c0 msgr2=0x7f98cc06de00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:44:52.536 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:52.405+0000 7f98d18a7640 1 -- 172.21.15.16:0/1477284056 shutdown_connections 2023-05-22T15:44:52.536 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:52.405+0000 7f98d18a7640 1 -- 172.21.15.16:0/1477284056 wait complete. 2023-05-22T15:44:52.537 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-05-22T15:44:52.668 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:52.665+0000 7efdd5d72640 1 Processor -- start 2023-05-22T15:44:52.669 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:52.665+0000 7efdd5d72640 1 -- start start 2023-05-22T15:44:52.669 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:52.665+0000 7efdd5d72640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7efdd00b0250 0x7efdd0159330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:52.669 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:52.665+0000 7efdd5d72640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efdd015a0a0 con 0x7efdd0159870 2023-05-22T15:44:52.670 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:52.665+0000 7efdd5d72640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efdd0153580 con 0x7efdd01531b0 2023-05-22T15:44:52.670 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:52.665+0000 7efdd5d72640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efdd015d5e0 con 0x7efdd00b0250 2023-05-22T15:44:52.670 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:52.665+0000 7efdcffff640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7efdd0159870 0x7efdd00c8ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:52034/0 (socket says 172.21.15.16:52034) 2023-05-22T15:44:52.670 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:52.665+0000 7efdcffff640 1 -- 172.21.15.16:0/314485893 learned_addr learned my addr 172.21.15.16:0/314485893 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:44:52.671 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:52.665+0000 7efdce7fc640 1 -- 172.21.15.16:0/314485893 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 878933101 0 0) 0x7efdd015a0a0 con 0x7efdd0159870 2023-05-22T15:44:52.671 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:52.665+0000 7efdceffd640 1 --2- 172.21.15.16:0/314485893 >> v2:172.21.15.62:3300/0 conn(0x7efdd00b0250 0x7efdd0159330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:44:52.672 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:52.665+0000 7efdce7fc640 1 -- 172.21.15.16:0/314485893 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efdbc003530 con 0x7efdd0159870 2023-05-22T15:44:52.672 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:52.665+0000 7efdce7fc640 1 -- 172.21.15.16:0/314485893 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2959025164 0 0) 0x7efdbc003530 con 0x7efdd0159870 2023-05-22T15:44:52.672 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:52.665+0000 7efdce7fc640 1 -- 172.21.15.16:0/314485893 >> v2:172.21.15.62:3300/0 conn(0x7efdd00b0250 msgr2=0x7efdd0159330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:44:52.672 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:52.665+0000 7efdce7fc640 1 --2- 172.21.15.16:0/314485893 >> v2:172.21.15.62:3300/0 conn(0x7efdd00b0250 0x7efdd0159330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:52.672 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:52.665+0000 7efdce7fc640 1 -- 172.21.15.16:0/314485893 >> v1:172.21.15.62:6789/0 conn(0x7efdd01531b0 legacy=0x7efdd00afb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:44:52.673 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:52.665+0000 7efdce7fc640 1 -- 172.21.15.16:0/314485893 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdd015c820 con 0x7efdd0159870 2023-05-22T15:44:52.673 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:52.665+0000 7efdce7fc640 1 -- 172.21.15.16:0/314485893 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7efdc8001680 con 0x7efdd0159870 2023-05-22T15:44:52.673 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:52.665+0000 7efdce7fc640 1 -- 172.21.15.16:0/314485893 >> v1:172.21.15.16:6789/0 conn(0x7efdd0159870 legacy=0x7efdd00c8ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:44:52.673 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:52.665+0000 7efdce7fc640 1 --2- 172.21.15.16:0/314485893 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efdbc003980 0x7efdbc003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:52.674 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:52.665+0000 7efdce7fc640 1 --2- 172.21.15.16:0/314485893 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efdbc0047d0 0x7efdbc004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:52.674 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:52.665+0000 7efdce7fc640 1 --2- 172.21.15.16:0/314485893 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7efdbc005180 0x7efdbc00dba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:52.674 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:52.665+0000 7efdce7fc640 1 -- 172.21.15.16:0/314485893 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdd015d5e0 con 0x7efdbc0047d0 2023-05-22T15:44:52.674 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:52.665+0000 7efdce7fc640 1 -- 172.21.15.16:0/314485893 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdbc005960 con 0x7efdbc003980 2023-05-22T15:44:52.675 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:52.665+0000 7efdce7fc640 1 -- 172.21.15.16:0/314485893 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efdbc005b20 con 0x7efdbc005180 2023-05-22T15:44:52.675 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:52.669+0000 7efdceffd640 1 --2- 172.21.15.16:0/314485893 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efdbc0047d0 0x7efdbc004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:44:52.675 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:52.669+0000 7efdcf7fe640 1 --2- 172.21.15.16:0/314485893 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efdbc003980 0x7efdbc003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:44:52.675 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:52.669+0000 7efdcffff640 1 --2- 172.21.15.16:0/314485893 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7efdbc005180 0x7efdbc00dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:44:52.676 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:52.669+0000 7efdceffd640 1 -- 172.21.15.16:0/314485893 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7efdbc005180 msgr2=0x7efdbc00dba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:44:52.677 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:52.669+0000 7efdceffd640 1 --2- 172.21.15.16:0/314485893 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7efdbc005180 0x7efdbc00dba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:52.677 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:52.669+0000 7efdceffd640 1 -- 172.21.15.16:0/314485893 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efdbc003980 msgr2=0x7efdbc003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:44:52.677 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:52.669+0000 7efdceffd640 1 --2- 172.21.15.16:0/314485893 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efdbc003980 0x7efdbc003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:52.678 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:52.669+0000 7efdcffff640 1 -- 172.21.15.16:0/314485893 reap_dead start 2023-05-22T15:44:52.678 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:52.669+0000 7efdceffd640 1 -- 172.21.15.16:0/314485893 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdbc003530 con 0x7efdbc0047d0 2023-05-22T15:44:52.681 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:52.669+0000 7efdceffd640 1 --2- 172.21.15.16:0/314485893 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efdbc0047d0 0x7efdbc004c40 secure :-1 s=READY pgs=1577 cs=0 l=1 rev1=1 crypto rx=0x7efdd006f400 tx=0x7efdb801cc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6366679a22a13a5 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:44:52.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:52.669+0000 7efdce7fc640 1 -- 172.21.15.16:0/314485893 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdb80a5a20 con 0x7efdbc0047d0 2023-05-22T15:44:52.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:52.669+0000 7efdce7fc640 1 -- 172.21.15.16:0/314485893 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efdb800f040 con 0x7efdbc0047d0 2023-05-22T15:44:52.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:52.669+0000 7efdce7fc640 1 -- 172.21.15.16:0/314485893 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdb80a5dc0 con 0x7efdbc0047d0 2023-05-22T15:44:52.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:52.669+0000 7efdd5d72640 1 -- 172.21.15.16:0/314485893 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efdbc0047d0 msgr2=0x7efdbc004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:44:52.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:52.669+0000 7efdd5d72640 1 --2- 172.21.15.16:0/314485893 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efdbc0047d0 0x7efdbc004c40 secure :-1 s=READY pgs=1577 cs=0 l=1 rev1=1 crypto rx=0x7efdd006f400 tx=0x7efdb801cc00 comp rx=0 tx=0).stop 2023-05-22T15:44:52.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:52.669+0000 7efdd5d72640 1 -- 172.21.15.16:0/314485893 shutdown_connections 2023-05-22T15:44:52.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:52.669+0000 7efdd5d72640 1 --2- 172.21.15.16:0/314485893 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efdbc0047d0 0x7efdbc004c40 unknown :-1 s=CLOSED pgs=1577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:52.683 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:52.669+0000 7efdd5d72640 1 -- 172.21.15.16:0/314485893 >> 172.21.15.16:0/314485893 conn(0x7efdd0069720 msgr2=0x7efdd0069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:44:52.683 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:52.669+0000 7efdd5d72640 1 -- 172.21.15.16:0/314485893 shutdown_connections 2023-05-22T15:44:52.683 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:52.669+0000 7efdd5d72640 1 -- 172.21.15.16:0/314485893 wait complete. 2023-05-22T15:44:52.683 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:52.673+0000 7efdd5d72640 1 Processor -- start 2023-05-22T15:44:52.683 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:52.673+0000 7efdd5d72640 1 -- start start 2023-05-22T15:44:52.683 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:52.673+0000 7efdd5d72640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efdbc0047d0 0x7efdd00c50f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:52.683 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:52.673+0000 7efdd5d72640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efdd00c7b90 0x7efdd0140750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:52.683 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:52.673+0000 7efdd5d72640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7efdd00b3690 0x7efdd00cc4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:52.684 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:52.673+0000 7efdd5d72640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdd00c8000 con 0x7efdbc0047d0 2023-05-22T15:44:52.684 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:52.673+0000 7efdd5d72640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdd0140c90 con 0x7efdd00c7b90 2023-05-22T15:44:52.684 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:52.673+0000 7efdd5d72640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efdd00c56d0 con 0x7efdd00b3690 2023-05-22T15:44:52.684 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:52.673+0000 7efdcf7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efdbc0047d0 0x7efdd00c50f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:44:52.684 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:52.673+0000 7efdcf7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efdbc0047d0 0x7efdd00c50f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59376/0 (socket says 172.21.15.16:59376) 2023-05-22T15:44:52.684 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:52.673+0000 7efdcf7fe640 1 -- 172.21.15.16:0/1563434969 learned_addr learned my addr 172.21.15.16:0/1563434969 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:44:52.684 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:52.673+0000 7efdceffd640 1 --2- 172.21.15.16:0/1563434969 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efdd00c7b90 0x7efdd0140750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:44:52.685 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:52.673+0000 7efdcffff640 1 --2- 172.21.15.16:0/1563434969 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7efdd00b3690 0x7efdd00cc4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:44:52.685 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:52.673+0000 7efdcf7fe640 1 -- 172.21.15.16:0/1563434969 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7efdd00b3690 msgr2=0x7efdd00cc4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:44:52.685 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:52.673+0000 7efdcf7fe640 1 --2- 172.21.15.16:0/1563434969 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7efdd00b3690 0x7efdd00cc4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:52.685 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:52.673+0000 7efdcf7fe640 1 -- 172.21.15.16:0/1563434969 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efdd00c7b90 msgr2=0x7efdd0140750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:44:52.685 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:52.673+0000 7efdcf7fe640 1 --2- 172.21.15.16:0/1563434969 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efdd00c7b90 0x7efdd0140750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:52.685 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:52.673+0000 7efdcf7fe640 1 -- 172.21.15.16:0/1563434969 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdd015cde0 con 0x7efdbc0047d0 2023-05-22T15:44:52.685 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:52.673+0000 7efdcffff640 1 --2- 172.21.15.16:0/1563434969 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7efdd00b3690 0x7efdd00cc4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:44:52.685 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:52.673+0000 7efdcf7fe640 1 --2- 172.21.15.16:0/1563434969 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efdbc0047d0 0x7efdd00c50f0 secure :-1 s=READY pgs=1578 cs=0 l=1 rev1=1 crypto rx=0x7efdc0001ec0 tx=0x7efdc000e4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:44:52.686 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:52.673+0000 7efdc7fff640 1 -- 172.21.15.16:0/1563434969 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdc00050d0 con 0x7efdbc0047d0 2023-05-22T15:44:52.686 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:52.673+0000 7efdc7fff640 1 -- 172.21.15.16:0/1563434969 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efdc0005230 con 0x7efdbc0047d0 2023-05-22T15:44:52.686 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:52.673+0000 7efdc7fff640 1 -- 172.21.15.16:0/1563434969 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdc0003600 con 0x7efdbc0047d0 2023-05-22T15:44:52.686 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:52.673+0000 7efdd5d72640 1 -- 172.21.15.16:0/1563434969 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efdd015cb00 con 0x7efdbc0047d0 2023-05-22T15:44:52.686 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:52.673+0000 7efdd5d72640 1 -- 172.21.15.16:0/1563434969 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efdd015d0c0 con 0x7efdbc0047d0 2023-05-22T15:44:52.687 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:52.673+0000 7efdd5d72640 1 -- 172.21.15.16:0/1563434969 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efdd004dbe0 con 0x7efdbc0047d0 2023-05-22T15:44:52.687 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:52.673+0000 7efdc7fff640 1 -- 172.21.15.16:0/1563434969 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 49) v1 ==== 84068+0+0 (secure 0 0 0) 0x7efdc0003760 con 0x7efdbc0047d0 2023-05-22T15:44:52.687 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:52.677+0000 7efdc7fff640 1 --2- 172.21.15.16:0/1563434969 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7efdb0065200 0x7efdb00676b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:52.687 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:52.677+0000 7efdc7fff640 1 -- 172.21.15.16:0/1563434969 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 7070+0+0 (secure 0 0 0) 0x7efdc0081c80 con 0x7efdbc0047d0 2023-05-22T15:44:52.687 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:52.677+0000 7efdceffd640 1 --2- 172.21.15.16:0/1563434969 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7efdb0065200 0x7efdb00676b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:44:52.687 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:52.677+0000 7efdceffd640 1 --2- 172.21.15.16:0/1563434969 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7efdb0065200 0x7efdb00676b0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7efdb80af000 tx=0x7efdb80a7040 comp rx=0 tx=0).ready entity=mgr.8304 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:44:52.688 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:52.681+0000 7efdc7fff640 1 -- 172.21.15.16:0/1563434969 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7efdc004c620 con 0x7efdbc0047d0 2023-05-22T15:44:52.862 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:52.857+0000 7efdd5d72640 1 -- 172.21.15.16:0/1563434969 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7efdd00cc2d0 con 0x7efdbc0047d0 2023-05-22T15:44:53.079 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:53.077+0000 7efdc7fff640 1 -- 172.21.15.16:0/1563434969 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mgrmap(e 50) v1 ==== 84332+0+0 (secure 0 0 0) 0x7efdc004fc00 con 0x7efdbc0047d0 2023-05-22T15:44:53.545 INFO:tasks.ceph.mon.a.smithi016.stderr:2023-05-22T15:44:53.541+0000 7f95dd48a640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-05-22T15:44:53.555 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:53.553+0000 7efdc7fff640 1 -- 172.21.15.16:0/1563434969 <== mon.0 v2:172.21.15.16:3300/0 8 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v44) v1 ==== 161+0+0 (secure 0 0 0) 0x7efdc005b720 con 0x7efdbc0047d0 2023-05-22T15:44:53.556 INFO:teuthology.orchestra.run.smithi016.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-05-22T15:44:53.558 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:53.553+0000 7efdd5d72640 1 -- 172.21.15.16:0/1563434969 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7efdb0065200 msgr2=0x7efdb00676b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:44:53.558 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:53.553+0000 7efdd5d72640 1 --2- 172.21.15.16:0/1563434969 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7efdb0065200 0x7efdb00676b0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7efdb80af000 tx=0x7efdb80a7040 comp rx=0 tx=0).stop 2023-05-22T15:44:53.559 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:53.553+0000 7efdd5d72640 1 -- 172.21.15.16:0/1563434969 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efdbc0047d0 msgr2=0x7efdd00c50f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:44:53.559 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:53.553+0000 7efdd5d72640 1 --2- 172.21.15.16:0/1563434969 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efdbc0047d0 0x7efdd00c50f0 secure :-1 s=READY pgs=1578 cs=0 l=1 rev1=1 crypto rx=0x7efdc0001ec0 tx=0x7efdc000e4d0 comp rx=0 tx=0).stop 2023-05-22T15:44:53.559 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:53.553+0000 7efdd5d72640 1 -- 172.21.15.16:0/1563434969 shutdown_connections 2023-05-22T15:44:53.559 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:53.553+0000 7efdd5d72640 1 --2- 172.21.15.16:0/1563434969 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7efdb0065200 0x7efdb00676b0 secure :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7efdb80af000 tx=0x7efdb80a7040 comp rx=0 tx=0).stop 2023-05-22T15:44:53.559 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:53.553+0000 7efdd5d72640 1 --2- 172.21.15.16:0/1563434969 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7efdd00b3690 0x7efdd00cc4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:53.559 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:53.553+0000 7efdd5d72640 1 --2- 172.21.15.16:0/1563434969 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efdd00c7b90 0x7efdd0140750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:53.559 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:53.553+0000 7efdd5d72640 1 --2- 172.21.15.16:0/1563434969 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efdbc0047d0 0x7efdd00c50f0 unknown :-1 s=CLOSED pgs=1578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:53.560 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:53.553+0000 7efdd5d72640 1 -- 172.21.15.16:0/1563434969 >> 172.21.15.16:0/1563434969 conn(0x7efdd0069720 msgr2=0x7efdd00afb30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:44:53.560 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:53.557+0000 7efdd5d72640 1 -- 172.21.15.16:0/1563434969 shutdown_connections 2023-05-22T15:44:53.560 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:53.557+0000 7efdd5d72640 1 -- 172.21.15.16:0/1563434969 wait complete. 2023-05-22T15:44:53.562 INFO:tasks.ceph.mds.a.smithi016.stderr: -6312> 2023-05-22T15:43:31.591+0000 7ffaf59f5640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-05-22T15:44:53.580 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-05-22T15:44:53.607 INFO:tasks.ceph.mds.a.smithi016.stdout:starting mds.a at 2023-05-22T15:44:53.705 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:53.701+0000 7fec767f0640 1 Processor -- start 2023-05-22T15:44:53.705 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:53.701+0000 7fec767f0640 1 -- start start 2023-05-22T15:44:53.707 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:53.701+0000 7fec767f0640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fec7015bca0 0x7fec700cda80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:53.707 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:53.701+0000 7fec767f0640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fec700d2480 0x7fec700d28d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:53.707 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:53.701+0000 7fec767f0640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec700d2e10 con 0x7fec700b5340 2023-05-22T15:44:53.707 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:53.701+0000 7fec767f0640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec700cd660 con 0x7fec700d2480 2023-05-22T15:44:53.708 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:53.705+0000 7fec767f0640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec700cccc0 con 0x7fec7015bca0 2023-05-22T15:44:53.708 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:53.705+0000 7fec74d8d640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fec700b5340 0x7fec700b57c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:52056/0 (socket says 172.21.15.16:52056) 2023-05-22T15:44:53.708 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:53.705+0000 7fec67fff640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fec700d2480 0x7fec700d28d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:44:53.708 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:53.705+0000 7fec74d8d640 1 -- 172.21.15.16:0/3439961327 learned_addr learned my addr 172.21.15.16:0/3439961327 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:44:53.708 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:53.705+0000 7fec6ffff640 1 --2- 172.21.15.16:0/3439961327 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fec7015bca0 0x7fec700cda80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:44:53.708 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:53.705+0000 7fec67fff640 1 -- 172.21.15.16:0/3439961327 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fec7015bca0 msgr2=0x7fec700cda80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:44:53.708 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:53.705+0000 7fec67fff640 1 --2- 172.21.15.16:0/3439961327 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fec7015bca0 0x7fec700cda80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:53.709 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:53.705+0000 7fec67fff640 1 -- 172.21.15.16:0/3439961327 >> v1:172.21.15.16:6789/0 conn(0x7fec700b5340 legacy=0x7fec700b57c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:44:53.709 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:53.705+0000 7fec6f7fe640 1 -- 172.21.15.16:0/3439961327 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3001224911 0 0) 0x7fec680014a0 con 0x7fec700b5340 2023-05-22T15:44:53.710 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:53.705+0000 7fec67fff640 1 -- 172.21.15.16:0/3439961327 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec700ccfa0 con 0x7fec700d2480 2023-05-22T15:44:53.711 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:53.705+0000 7fec67fff640 1 --2- 172.21.15.16:0/3439961327 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fec700d2480 0x7fec700d28d0 secure :-1 s=READY pgs=1583 cs=0 l=1 rev1=1 crypto rx=0x7fec580097c0 tx=0x7fec5801c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52d45f169e0d5b90 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:44:53.711 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:53.705+0000 7fec6f7fe640 1 -- 172.21.15.16:0/3439961327 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec580ab670 con 0x7fec700d2480 2023-05-22T15:44:53.711 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:53.705+0000 7fec6f7fe640 1 -- 172.21.15.16:0/3439961327 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec580ab7d0 con 0x7fec700d2480 2023-05-22T15:44:53.712 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:53.705+0000 7fec6f7fe640 1 -- 172.21.15.16:0/3439961327 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec580abb60 con 0x7fec700d2480 2023-05-22T15:44:53.712 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:53.705+0000 7fec767f0640 1 -- 172.21.15.16:0/3439961327 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fec700d2480 msgr2=0x7fec700d28d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:44:53.712 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:53.705+0000 7fec767f0640 1 --2- 172.21.15.16:0/3439961327 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fec700d2480 0x7fec700d28d0 secure :-1 s=READY pgs=1583 cs=0 l=1 rev1=1 crypto rx=0x7fec580097c0 tx=0x7fec5801c450 comp rx=0 tx=0).stop 2023-05-22T15:44:53.712 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:53.705+0000 7fec767f0640 1 -- 172.21.15.16:0/3439961327 shutdown_connections 2023-05-22T15:44:53.712 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:53.705+0000 7fec767f0640 1 --2- 172.21.15.16:0/3439961327 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fec700d2480 0x7fec700d28d0 unknown :-1 s=CLOSED pgs=1583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:53.712 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:53.705+0000 7fec767f0640 1 --2- 172.21.15.16:0/3439961327 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fec7015bca0 0x7fec700cda80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:53.713 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:53.705+0000 7fec767f0640 1 -- 172.21.15.16:0/3439961327 >> 172.21.15.16:0/3439961327 conn(0x7fec700655c0 msgr2=0x7fec70070e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:44:53.713 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:53.705+0000 7fec767f0640 1 -- 172.21.15.16:0/3439961327 shutdown_connections 2023-05-22T15:44:53.713 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:53.705+0000 7fec767f0640 1 -- 172.21.15.16:0/3439961327 wait complete. 2023-05-22T15:44:53.713 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:53.709+0000 7fec767f0640 1 Processor -- start 2023-05-22T15:44:53.713 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:53.709+0000 7fec767f0640 1 -- start start 2023-05-22T15:44:53.715 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:53.709+0000 7fec767f0640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fec700b5340 0x7fec70159400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:53.715 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:53.709+0000 7fec767f0640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fec700d2480 0x7fec700c7bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:53.715 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:53.709+0000 7fec767f0640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fec7015bca0 0x7fec7015c330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:53.716 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:53.709+0000 7fec767f0640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec70158f70 con 0x7fec7015bca0 2023-05-22T15:44:53.716 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:53.709+0000 7fec767f0640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec70153a70 con 0x7fec700d2480 2023-05-22T15:44:53.716 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:53.709+0000 7fec767f0640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec7015cea0 con 0x7fec700b5340 2023-05-22T15:44:53.716 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:53.709+0000 7fec74d8d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fec7015bca0 0x7fec7015c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:44:53.716 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:53.709+0000 7fec74d8d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fec7015bca0 0x7fec7015c330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59416/0 (socket says 172.21.15.16:59416) 2023-05-22T15:44:53.716 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:53.709+0000 7fec74d8d640 1 -- 172.21.15.16:0/2727810677 learned_addr learned my addr 172.21.15.16:0/2727810677 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:44:53.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:53.709+0000 7fec67fff640 1 --2- 172.21.15.16:0/2727810677 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fec700d2480 0x7fec700c7bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:44:53.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:53.709+0000 7fec6ffff640 1 --2- 172.21.15.16:0/2727810677 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fec700b5340 0x7fec70159400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:44:53.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:53.709+0000 7fec74d8d640 1 -- 172.21.15.16:0/2727810677 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fec700b5340 msgr2=0x7fec70159400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:44:53.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:53.709+0000 7fec74d8d640 1 --2- 172.21.15.16:0/2727810677 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fec700b5340 0x7fec70159400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:53.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:53.709+0000 7fec74d8d640 1 -- 172.21.15.16:0/2727810677 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fec700d2480 msgr2=0x7fec700c7bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:44:53.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:53.709+0000 7fec74d8d640 1 --2- 172.21.15.16:0/2727810677 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fec700d2480 0x7fec700c7bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:53.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:53.709+0000 7fec74d8d640 1 -- 172.21.15.16:0/2727810677 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec58009420 con 0x7fec7015bca0 2023-05-22T15:44:53.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:53.709+0000 7fec6ffff640 1 --2- 172.21.15.16:0/2727810677 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fec700b5340 0x7fec70159400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:44:53.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:53.709+0000 7fec74d8d640 1 --2- 172.21.15.16:0/2727810677 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fec7015bca0 0x7fec7015c330 secure :-1 s=READY pgs=1584 cs=0 l=1 rev1=1 crypto rx=0x7fec6800c7e0 tx=0x7fec6800cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:44:53.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:53.709+0000 7fec6d7fa640 1 -- 172.21.15.16:0/2727810677 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec6800a640 con 0x7fec7015bca0 2023-05-22T15:44:53.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:53.709+0000 7fec6d7fa640 1 -- 172.21.15.16:0/2727810677 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec6800a7c0 con 0x7fec7015bca0 2023-05-22T15:44:53.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:53.709+0000 7fec6d7fa640 1 -- 172.21.15.16:0/2727810677 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec6800abb0 con 0x7fec7015bca0 2023-05-22T15:44:53.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:53.709+0000 7fec767f0640 1 -- 172.21.15.16:0/2727810677 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec70160360 con 0x7fec7015bca0 2023-05-22T15:44:53.719 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:53.709+0000 7fec767f0640 1 -- 172.21.15.16:0/2727810677 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec700cdfc0 con 0x7fec7015bca0 2023-05-22T15:44:53.719 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:53.709+0000 7fec767f0640 1 -- 172.21.15.16:0/2727810677 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec701604f0 con 0x7fec7015bca0 2023-05-22T15:44:53.719 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:53.713+0000 7fec6d7fa640 1 -- 172.21.15.16:0/2727810677 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 50) v1 ==== 84332+0+0 (secure 0 0 0) 0x7fec6800ad10 con 0x7fec7015bca0 2023-05-22T15:44:53.719 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:53.713+0000 7fec6d7fa640 1 --2- 172.21.15.16:0/2727810677 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fec500654d0 0x7fec50067980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:53.719 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:53.713+0000 7fec6ffff640 1 --2- 172.21.15.16:0/2727810677 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fec500654d0 0x7fec50067980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:44:53.720 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:53.713+0000 7fec6d7fa640 1 -- 172.21.15.16:0/2727810677 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 7156+0+0 (secure 0 0 0) 0x7fec68082120 con 0x7fec7015bca0 2023-05-22T15:44:53.720 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:53.713+0000 7fec6ffff640 1 --2- 172.21.15.16:0/2727810677 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fec500654d0 0x7fec50067980 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fec600097a0 tx=0x7fec60008040 comp rx=0 tx=0).ready entity=mgr.8304 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:44:53.720 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:53.717+0000 7fec6d7fa640 1 -- 172.21.15.16:0/2727810677 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fec6804f840 con 0x7fec7015bca0 2023-05-22T15:44:53.908 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:53.905+0000 7fec767f0640 1 -- 172.21.15.16:0/2727810677 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fec700d2e10 con 0x7fec7015bca0 2023-05-22T15:44:54.569 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:54.565+0000 7fec6d7fa640 1 -- 172.21.15.16:0/2727810677 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v45) v1 ==== 104+0+0 (secure 0 0 0) 0x7fec700d2e10 con 0x7fec7015bca0 2023-05-22T15:44:54.573 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:54.569+0000 7fec767f0640 1 -- 172.21.15.16:0/2727810677 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fec500654d0 msgr2=0x7fec50067980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:44:54.573 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:54.569+0000 7fec767f0640 1 --2- 172.21.15.16:0/2727810677 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fec500654d0 0x7fec50067980 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fec600097a0 tx=0x7fec60008040 comp rx=0 tx=0).stop 2023-05-22T15:44:54.573 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:54.569+0000 7fec767f0640 1 -- 172.21.15.16:0/2727810677 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fec7015bca0 msgr2=0x7fec7015c330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:44:54.573 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:54.569+0000 7fec767f0640 1 --2- 172.21.15.16:0/2727810677 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fec7015bca0 0x7fec7015c330 secure :-1 s=READY pgs=1584 cs=0 l=1 rev1=1 crypto rx=0x7fec6800c7e0 tx=0x7fec6800cca0 comp rx=0 tx=0).stop 2023-05-22T15:44:54.573 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:54.569+0000 7fec767f0640 1 -- 172.21.15.16:0/2727810677 shutdown_connections 2023-05-22T15:44:54.574 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:54.569+0000 7fec767f0640 1 --2- 172.21.15.16:0/2727810677 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fec7015bca0 0x7fec7015c330 unknown :-1 s=CLOSED pgs=1584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:54.574 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:54.569+0000 7fec767f0640 1 --2- 172.21.15.16:0/2727810677 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fec700d2480 0x7fec700c7bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:54.574 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:54.569+0000 7fec767f0640 1 --2- 172.21.15.16:0/2727810677 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fec500654d0 0x7fec50067980 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:54.574 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:54.569+0000 7fec767f0640 1 --2- 172.21.15.16:0/2727810677 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fec700b5340 0x7fec70159400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:54.574 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:54.569+0000 7fec767f0640 1 -- 172.21.15.16:0/2727810677 >> 172.21.15.16:0/2727810677 conn(0x7fec700655c0 msgr2=0x7fec700b7b00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:44:54.574 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:54.569+0000 7fec767f0640 1 -- 172.21.15.16:0/2727810677 shutdown_connections 2023-05-22T15:44:54.574 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:54.569+0000 7fec767f0640 1 -- 172.21.15.16:0/2727810677 wait complete. 2023-05-22T15:44:54.596 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-05-22T15:44:54.773 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:54.769+0000 7ff7d63dc640 1 Processor -- start 2023-05-22T15:44:54.773 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:54.769+0000 7ff7d63dc640 1 -- start start 2023-05-22T15:44:54.774 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:54.769+0000 7ff7d63dc640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7ff7d015ba90 0x7ff7d006b240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:54.775 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:54.769+0000 7ff7d63dc640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff7d015be60 0x7ff7d006dc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:54.775 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:54.769+0000 7ff7d63dc640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff7d006e1d0 0x7ff7d015f740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:54.775 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:54.769+0000 7ff7d63dc640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7d006c8d0 con 0x7ff7d006e1d0 2023-05-22T15:44:54.776 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:54.769+0000 7ff7d63dc640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7d006e650 con 0x7ff7d015ba90 2023-05-22T15:44:54.776 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:54.769+0000 7ff7d63dc640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7d00b25d0 con 0x7ff7d015be60 2023-05-22T15:44:54.776 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:54.773+0000 7ff7d4979640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff7d006e1d0 0x7ff7d015f740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:44:54.776 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:54.773+0000 7ff7d4979640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff7d006e1d0 0x7ff7d015f740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59428/0 (socket says 172.21.15.16:59428) 2023-05-22T15:44:54.776 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:54.773+0000 7ff7d4979640 1 -- 172.21.15.16:0/2139728625 learned_addr learned my addr 172.21.15.16:0/2139728625 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:44:54.776 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:54.773+0000 7ff7cf7fe640 1 --2- 172.21.15.16:0/2139728625 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff7d015be60 0x7ff7d006dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:44:54.777 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:54.773+0000 7ff7cffff640 1 --2- 172.21.15.16:0/2139728625 >> v2:172.21.15.62:3300/0 conn(0x7ff7d015ba90 0x7ff7d006b240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:44:54.777 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:54.773+0000 7ff7d4979640 1 -- 172.21.15.16:0/2139728625 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff7d015be60 msgr2=0x7ff7d006dc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:44:54.777 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:54.773+0000 7ff7d4979640 1 --2- 172.21.15.16:0/2139728625 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff7d015be60 0x7ff7d006dc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:54.777 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:54.773+0000 7ff7d4979640 1 -- 172.21.15.16:0/2139728625 >> v2:172.21.15.62:3300/0 conn(0x7ff7d015ba90 msgr2=0x7ff7d006b240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:44:54.777 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:54.773+0000 7ff7d4979640 1 --2- 172.21.15.16:0/2139728625 >> v2:172.21.15.62:3300/0 conn(0x7ff7d015ba90 0x7ff7d006b240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:54.777 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:54.773+0000 7ff7d4979640 1 -- 172.21.15.16:0/2139728625 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7d00cf9d0 con 0x7ff7d006e1d0 2023-05-22T15:44:54.778 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:54.773+0000 7ff7d4979640 1 --2- 172.21.15.16:0/2139728625 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff7d006e1d0 0x7ff7d015f740 secure :-1 s=READY pgs=1585 cs=0 l=1 rev1=1 crypto rx=0x7ff7d00b2e20 tx=0x7ff7c801c710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7161363c84dc5f88 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:44:54.778 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:54.773+0000 7ff7ceffd640 1 -- 172.21.15.16:0/2139728625 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7c80a6490 con 0x7ff7d006e1d0 2023-05-22T15:44:54.778 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:54.773+0000 7ff7ceffd640 1 -- 172.21.15.16:0/2139728625 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7c80a65f0 con 0x7ff7d006e1d0 2023-05-22T15:44:54.779 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:54.773+0000 7ff7ceffd640 1 -- 172.21.15.16:0/2139728625 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7c80a6980 con 0x7ff7d006e1d0 2023-05-22T15:44:54.779 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:54.773+0000 7ff7d63dc640 1 -- 172.21.15.16:0/2139728625 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff7d006e1d0 msgr2=0x7ff7d015f740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:44:54.779 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:54.773+0000 7ff7d63dc640 1 --2- 172.21.15.16:0/2139728625 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff7d006e1d0 0x7ff7d015f740 secure :-1 s=READY pgs=1585 cs=0 l=1 rev1=1 crypto rx=0x7ff7d00b2e20 tx=0x7ff7c801c710 comp rx=0 tx=0).stop 2023-05-22T15:44:54.779 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:54.773+0000 7ff7d63dc640 1 -- 172.21.15.16:0/2139728625 shutdown_connections 2023-05-22T15:44:54.779 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:54.773+0000 7ff7d63dc640 1 --2- 172.21.15.16:0/2139728625 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff7d006e1d0 0x7ff7d015f740 unknown :-1 s=CLOSED pgs=1585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:54.779 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:54.773+0000 7ff7d63dc640 1 --2- 172.21.15.16:0/2139728625 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff7d015be60 0x7ff7d006dc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:54.779 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:54.773+0000 7ff7d63dc640 1 --2- 172.21.15.16:0/2139728625 >> v2:172.21.15.62:3300/0 conn(0x7ff7d015ba90 0x7ff7d006b240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:54.780 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:54.773+0000 7ff7d63dc640 1 -- 172.21.15.16:0/2139728625 >> 172.21.15.16:0/2139728625 conn(0x7ff7d006ffc0 msgr2=0x7ff7d00653f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:44:54.780 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:54.773+0000 7ff7d63dc640 1 -- 172.21.15.16:0/2139728625 shutdown_connections 2023-05-22T15:44:54.780 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:54.773+0000 7ff7d63dc640 1 -- 172.21.15.16:0/2139728625 wait complete. 2023-05-22T15:44:54.781 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:54.777+0000 7ff7d63dc640 1 Processor -- start 2023-05-22T15:44:54.781 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:54.777+0000 7ff7d63dc640 1 -- start start 2023-05-22T15:44:54.782 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:54.777+0000 7ff7d63dc640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff7d006e1d0 0x7ff7d00b3060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:54.782 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:54.777+0000 7ff7d63dc640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff7d015ba90 0x7ff7d014c0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:54.782 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:54.777+0000 7ff7d63dc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff7d014b2c0 0x7ff7d014ab20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:54.782 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:54.777+0000 7ff7d63dc640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7d014ba60 con 0x7ff7d014b2c0 2023-05-22T15:44:54.782 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:54.777+0000 7ff7d63dc640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7d00b35a0 con 0x7ff7d015ba90 2023-05-22T15:44:54.782 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:54.777+0000 7ff7d63dc640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7d014c600 con 0x7ff7d006e1d0 2023-05-22T15:44:54.782 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:54.777+0000 7ff7d4979640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff7d014b2c0 0x7ff7d014ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:44:54.783 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:54.777+0000 7ff7d4979640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff7d014b2c0 0x7ff7d014ab20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59436/0 (socket says 172.21.15.16:59436) 2023-05-22T15:44:54.783 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:54.777+0000 7ff7d4979640 1 -- 172.21.15.16:0/3847006256 learned_addr learned my addr 172.21.15.16:0/3847006256 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:44:54.783 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:54.777+0000 7ff7cffff640 1 --2- 172.21.15.16:0/3847006256 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff7d006e1d0 0x7ff7d00b3060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:44:54.783 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:54.777+0000 7ff7cf7fe640 1 --2- 172.21.15.16:0/3847006256 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff7d015ba90 0x7ff7d014c0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:44:54.783 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:54.777+0000 7ff7d4979640 1 -- 172.21.15.16:0/3847006256 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff7d006e1d0 msgr2=0x7ff7d00b3060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:44:54.783 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:54.777+0000 7ff7d4979640 1 --2- 172.21.15.16:0/3847006256 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff7d006e1d0 0x7ff7d00b3060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:54.783 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:54.777+0000 7ff7d4979640 1 -- 172.21.15.16:0/3847006256 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff7d015ba90 msgr2=0x7ff7d014c0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:44:54.783 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:54.777+0000 7ff7d4979640 1 --2- 172.21.15.16:0/3847006256 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff7d015ba90 0x7ff7d014c0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:54.784 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:54.777+0000 7ff7d4979640 1 -- 172.21.15.16:0/3847006256 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7c8009b30 con 0x7ff7d014b2c0 2023-05-22T15:44:54.784 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:54.777+0000 7ff7d4979640 1 --2- 172.21.15.16:0/3847006256 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff7d014b2c0 0x7ff7d014ab20 secure :-1 s=READY pgs=1586 cs=0 l=1 rev1=1 crypto rx=0x7ff7c801adc0 tx=0x7ff7c80194b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:44:54.784 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:54.777+0000 7ff7ccff9640 1 -- 172.21.15.16:0/3847006256 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7c80a6490 con 0x7ff7d014b2c0 2023-05-22T15:44:54.784 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:54.777+0000 7ff7ccff9640 1 -- 172.21.15.16:0/3847006256 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7c80a7670 con 0x7ff7d014b2c0 2023-05-22T15:44:54.784 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:54.777+0000 7ff7d63dc640 1 -- 172.21.15.16:0/3847006256 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7d006b780 con 0x7ff7d014b2c0 2023-05-22T15:44:54.784 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:54.777+0000 7ff7ccff9640 1 -- 172.21.15.16:0/3847006256 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7c80a7a00 con 0x7ff7d014b2c0 2023-05-22T15:44:54.784 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:54.777+0000 7ff7d63dc640 1 -- 172.21.15.16:0/3847006256 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7d00cf660 con 0x7ff7d014b2c0 2023-05-22T15:44:54.785 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:54.777+0000 7ff7d63dc640 1 -- 172.21.15.16:0/3847006256 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7d006bf30 con 0x7ff7d014b2c0 2023-05-22T15:44:54.786 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:54.781+0000 7ff7ccff9640 1 -- 172.21.15.16:0/3847006256 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 50) v1 ==== 84332+0+0 (secure 0 0 0) 0x7ff7c80a7b60 con 0x7ff7d014b2c0 2023-05-22T15:44:54.786 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:54.781+0000 7ff7ccff9640 1 --2- 172.21.15.16:0/3847006256 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ff7b00654d0 0x7ff7b0067980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:54.786 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:54.781+0000 7ff7ccff9640 1 -- 172.21.15.16:0/3847006256 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 7156+0+0 (secure 0 0 0) 0x7ff7c811cf20 con 0x7ff7d014b2c0 2023-05-22T15:44:54.786 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:54.781+0000 7ff7cffff640 1 --2- 172.21.15.16:0/3847006256 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ff7b00654d0 0x7ff7b0067980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:44:54.786 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:54.781+0000 7ff7cffff640 1 --2- 172.21.15.16:0/3847006256 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ff7b00654d0 0x7ff7b0067980 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7ff7c000b9d0 tx=0x7ff7c0008040 comp rx=0 tx=0).ready entity=mgr.8304 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:44:54.787 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:54.785+0000 7ff7ccff9640 1 -- 172.21.15.16:0/3847006256 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7ff7c80ea530 con 0x7ff7d014b2c0 2023-05-22T15:44:54.942 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:54.937+0000 7ff7d63dc640 1 -- 172.21.15.16:0/3847006256 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7ff7d004dbe0 con 0x7ff7d014b2c0 2023-05-22T15:44:55.654 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:55.649+0000 7ff7ccff9640 1 -- 172.21.15.16:0/3847006256 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v122) v1 ==== 181+0+0 (secure 0 0 0) 0x7ff7c8009510 con 0x7ff7d014b2c0 2023-05-22T15:44:55.654 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'cephfs_metadata' removed 2023-05-22T15:44:55.657 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:55.653+0000 7ff7d63dc640 1 -- 172.21.15.16:0/3847006256 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ff7b00654d0 msgr2=0x7ff7b0067980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:44:55.657 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:55.653+0000 7ff7d63dc640 1 --2- 172.21.15.16:0/3847006256 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ff7b00654d0 0x7ff7b0067980 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7ff7c000b9d0 tx=0x7ff7c0008040 comp rx=0 tx=0).stop 2023-05-22T15:44:55.658 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:55.653+0000 7ff7d63dc640 1 -- 172.21.15.16:0/3847006256 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff7d014b2c0 msgr2=0x7ff7d014ab20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:44:55.658 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:55.653+0000 7ff7d63dc640 1 --2- 172.21.15.16:0/3847006256 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff7d014b2c0 0x7ff7d014ab20 secure :-1 s=READY pgs=1586 cs=0 l=1 rev1=1 crypto rx=0x7ff7c801adc0 tx=0x7ff7c80194b0 comp rx=0 tx=0).stop 2023-05-22T15:44:55.658 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:55.653+0000 7ff7d63dc640 1 -- 172.21.15.16:0/3847006256 shutdown_connections 2023-05-22T15:44:55.658 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:55.653+0000 7ff7d63dc640 1 --2- 172.21.15.16:0/3847006256 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff7d014b2c0 0x7ff7d014ab20 unknown :-1 s=CLOSED pgs=1586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:55.658 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:55.653+0000 7ff7d63dc640 1 --2- 172.21.15.16:0/3847006256 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff7d015ba90 0x7ff7d014c0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:55.659 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:55.653+0000 7ff7d63dc640 1 --2- 172.21.15.16:0/3847006256 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ff7b00654d0 0x7ff7b0067980 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:55.659 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:55.653+0000 7ff7d63dc640 1 --2- 172.21.15.16:0/3847006256 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff7d006e1d0 0x7ff7d00b3060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:55.659 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:55.653+0000 7ff7d63dc640 1 -- 172.21.15.16:0/3847006256 >> 172.21.15.16:0/3847006256 conn(0x7ff7d006ffc0 msgr2=0x7ff7d00b1350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:44:55.659 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:55.653+0000 7ff7d63dc640 1 -- 172.21.15.16:0/3847006256 shutdown_connections 2023-05-22T15:44:55.659 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:55.653+0000 7ff7d63dc640 1 -- 172.21.15.16:0/3847006256 wait complete. 2023-05-22T15:44:55.678 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-05-22T15:44:55.815 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:55.813+0000 7f0e16886640 1 Processor -- start 2023-05-22T15:44:55.815 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:55.813+0000 7f0e16886640 1 -- start start 2023-05-22T15:44:55.818 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:55.813+0000 7f0e16886640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f0e100ccc50 0x7f0e1015ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:55.818 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:55.813+0000 7f0e16886640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e100d2a20 con 0x7f0e1015c8e0 2023-05-22T15:44:55.818 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:55.813+0000 7f0e16886640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e100cd230 con 0x7f0e100cef10 2023-05-22T15:44:55.819 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:55.813+0000 7f0e16886640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e100d23d0 con 0x7f0e100ccc50 2023-05-22T15:44:55.819 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:55.813+0000 7f0e0ffff640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f0e1015c8e0 0x7f0e100d2eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:52058/0 (socket says 172.21.15.16:52058) 2023-05-22T15:44:55.819 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:55.813+0000 7f0e0ffff640 1 -- 172.21.15.16:0/2576765148 learned_addr learned my addr 172.21.15.16:0/2576765148 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:44:55.819 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:55.813+0000 7f0e0f7fe640 1 --2- 172.21.15.16:0/2576765148 >> v2:172.21.15.62:3300/0 conn(0x7f0e100ccc50 0x7f0e1015ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:44:55.820 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:55.813+0000 7f0e0effd640 1 -- 172.21.15.16:0/2576765148 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 885592087 0 0) 0x7f0e100d2a20 con 0x7f0e1015c8e0 2023-05-22T15:44:55.820 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:55.813+0000 7f0e0effd640 1 -- 172.21.15.16:0/2576765148 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0df8003530 con 0x7f0e1015c8e0 2023-05-22T15:44:55.820 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:55.813+0000 7f0e0effd640 1 -- 172.21.15.16:0/2576765148 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2736803427 0 0) 0x7f0df8003530 con 0x7f0e1015c8e0 2023-05-22T15:44:55.820 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:55.813+0000 7f0e0effd640 1 -- 172.21.15.16:0/2576765148 >> v2:172.21.15.62:3300/0 conn(0x7f0e100ccc50 msgr2=0x7f0e1015ccb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:44:55.821 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:55.813+0000 7f0e0effd640 1 --2- 172.21.15.16:0/2576765148 >> v2:172.21.15.62:3300/0 conn(0x7f0e100ccc50 0x7f0e1015ccb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:55.821 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:55.813+0000 7f0e0effd640 1 -- 172.21.15.16:0/2576765148 >> v1:172.21.15.62:6789/0 conn(0x7f0e100cef10 legacy=0x7f0e100cf310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:44:55.821 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:55.813+0000 7f0e0effd640 1 -- 172.21.15.16:0/2576765148 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e100d26b0 con 0x7f0e1015c8e0 2023-05-22T15:44:55.821 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:55.813+0000 7f0e0effd640 1 -- 172.21.15.16:0/2576765148 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7f0e00001680 con 0x7f0e1015c8e0 2023-05-22T15:44:55.822 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:55.813+0000 7f0e0effd640 1 -- 172.21.15.16:0/2576765148 >> v1:172.21.15.16:6789/0 conn(0x7f0e1015c8e0 legacy=0x7f0e100d2eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:44:55.822 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:55.813+0000 7f0e0effd640 1 --2- 172.21.15.16:0/2576765148 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0df8003980 0x7f0df8003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:55.822 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:55.813+0000 7f0e0effd640 1 --2- 172.21.15.16:0/2576765148 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0df80047d0 0x7f0df8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:55.822 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:55.813+0000 7f0e0effd640 1 --2- 172.21.15.16:0/2576765148 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0df8005180 0x7f0df800b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:55.822 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:55.813+0000 7f0e0effd640 1 -- 172.21.15.16:0/2576765148 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e100d23d0 con 0x7f0df8003980 2023-05-22T15:44:55.823 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:55.813+0000 7f0e0effd640 1 -- 172.21.15.16:0/2576765148 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0df800c090 con 0x7f0df80047d0 2023-05-22T15:44:55.823 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:55.813+0000 7f0e0effd640 1 -- 172.21.15.16:0/2576765148 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0df800c250 con 0x7f0df8005180 2023-05-22T15:44:55.823 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:55.813+0000 7f0e0ffff640 1 --2- 172.21.15.16:0/2576765148 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0df8003980 0x7f0df8003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:44:55.823 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:55.813+0000 7f0e0f7fe640 1 --2- 172.21.15.16:0/2576765148 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0df80047d0 0x7f0df8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:44:55.823 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:55.813+0000 7f0e14e23640 1 --2- 172.21.15.16:0/2576765148 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0df8005180 0x7f0df800b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:44:55.824 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:55.813+0000 7f0e0ffff640 1 -- 172.21.15.16:0/2576765148 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0df8005180 msgr2=0x7f0df800b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:44:55.824 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:55.813+0000 7f0e0ffff640 1 --2- 172.21.15.16:0/2576765148 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0df8005180 0x7f0df800b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:55.824 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:55.813+0000 7f0e0ffff640 1 -- 172.21.15.16:0/2576765148 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0df80047d0 msgr2=0x7f0df8004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:44:55.824 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:55.813+0000 7f0e0ffff640 1 --2- 172.21.15.16:0/2576765148 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0df80047d0 0x7f0df8004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:55.824 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:55.813+0000 7f0e14e23640 1 -- 172.21.15.16:0/2576765148 reap_dead start 2023-05-22T15:44:55.824 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:55.813+0000 7f0e0ffff640 1 -- 172.21.15.16:0/2576765148 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0df8003530 con 0x7f0df8003980 2023-05-22T15:44:55.825 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:55.821+0000 7f0e0ffff640 1 --2- 172.21.15.16:0/2576765148 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0df8003980 0x7f0df8003d90 secure :-1 s=READY pgs=1588 cs=0 l=1 rev1=1 crypto rx=0x7f0e0000c240 tx=0x7f0e0001cc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16095e7d2a0fe109 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:44:55.826 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:55.821+0000 7f0e0effd640 1 -- 172.21.15.16:0/2576765148 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e00005ae0 con 0x7f0df8003980 2023-05-22T15:44:55.826 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:55.821+0000 7f0e0effd640 1 -- 172.21.15.16:0/2576765148 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e00005c40 con 0x7f0df8003980 2023-05-22T15:44:55.826 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:55.821+0000 7f0e0effd640 1 -- 172.21.15.16:0/2576765148 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e000072b0 con 0x7f0df8003980 2023-05-22T15:44:55.826 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:55.821+0000 7f0e16886640 1 -- 172.21.15.16:0/2576765148 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0df8003980 msgr2=0x7f0df8003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:44:55.826 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:55.821+0000 7f0e16886640 1 --2- 172.21.15.16:0/2576765148 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0df8003980 0x7f0df8003d90 secure :-1 s=READY pgs=1588 cs=0 l=1 rev1=1 crypto rx=0x7f0e0000c240 tx=0x7f0e0001cc00 comp rx=0 tx=0).stop 2023-05-22T15:44:55.826 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:55.821+0000 7f0e16886640 1 -- 172.21.15.16:0/2576765148 shutdown_connections 2023-05-22T15:44:55.826 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:55.821+0000 7f0e16886640 1 --2- 172.21.15.16:0/2576765148 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0df8003980 0x7f0df8003d90 unknown :-1 s=CLOSED pgs=1588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:55.827 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:55.821+0000 7f0e16886640 1 -- 172.21.15.16:0/2576765148 >> 172.21.15.16:0/2576765148 conn(0x7f0e10070e50 msgr2=0x7f0e10071250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:44:55.827 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:55.821+0000 7f0e16886640 1 -- 172.21.15.16:0/2576765148 shutdown_connections 2023-05-22T15:44:55.827 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:55.821+0000 7f0e16886640 1 -- 172.21.15.16:0/2576765148 wait complete. 2023-05-22T15:44:55.827 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:55.821+0000 7f0e16886640 1 Processor -- start 2023-05-22T15:44:55.827 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:55.821+0000 7f0e16886640 1 -- start start 2023-05-22T15:44:55.829 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:55.821+0000 7f0e16886640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0df8003980 0x7f0e100cbfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:55.829 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:55.821+0000 7f0e16886640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e10153350 0x7f0e10155700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:55.829 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:55.821+0000 7f0e16886640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0e101537c0 0x7f0e10151540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:55.829 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:55.821+0000 7f0e16886640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e100c9870 con 0x7f0e10153350 2023-05-22T15:44:55.829 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:55.821+0000 7f0e16886640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e100c9b50 con 0x7f0e101537c0 2023-05-22T15:44:55.830 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:55.821+0000 7f0e16886640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e100c9e30 con 0x7f0df8003980 2023-05-22T15:44:55.830 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:55.821+0000 7f0e0f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e10153350 0x7f0e10155700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:44:55.830 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:55.821+0000 7f0e0f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e10153350 0x7f0e10155700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59454/0 (socket says 172.21.15.16:59454) 2023-05-22T15:44:55.830 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:55.821+0000 7f0e0f7fe640 1 -- 172.21.15.16:0/1258930557 learned_addr learned my addr 172.21.15.16:0/1258930557 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:44:55.830 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:55.825+0000 7f0e14e23640 1 --2- 172.21.15.16:0/1258930557 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0e101537c0 0x7f0e10151540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:44:55.830 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:55.825+0000 7f0e0ffff640 1 --2- 172.21.15.16:0/1258930557 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0df8003980 0x7f0e100cbfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:44:55.831 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:55.825+0000 7f0e0f7fe640 1 -- 172.21.15.16:0/1258930557 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0df8003980 msgr2=0x7f0e100cbfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:44:55.831 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:55.825+0000 7f0e0f7fe640 1 --2- 172.21.15.16:0/1258930557 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0df8003980 0x7f0e100cbfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:55.831 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:55.825+0000 7f0e0f7fe640 1 -- 172.21.15.16:0/1258930557 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0e101537c0 msgr2=0x7f0e10151540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:44:55.831 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:55.825+0000 7f0e0f7fe640 1 --2- 172.21.15.16:0/1258930557 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0e101537c0 0x7f0e10151540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:55.831 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:55.825+0000 7f0e0f7fe640 1 -- 172.21.15.16:0/1258930557 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e1015d250 con 0x7f0e10153350 2023-05-22T15:44:55.831 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:55.825+0000 7f0e14e23640 1 --2- 172.21.15.16:0/1258930557 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0e101537c0 0x7f0e10151540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:44:55.832 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:55.825+0000 7f0e0f7fe640 1 --2- 172.21.15.16:0/1258930557 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e10153350 0x7f0e10155700 secure :-1 s=READY pgs=1589 cs=0 l=1 rev1=1 crypto rx=0x7f0e04011900 tx=0x7f0e040119f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:44:55.832 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:55.825+0000 7f0e0cff9640 1 -- 172.21.15.16:0/1258930557 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e04004240 con 0x7f0e10153350 2023-05-22T15:44:55.832 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:55.825+0000 7f0e0cff9640 1 -- 172.21.15.16:0/1258930557 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e040043a0 con 0x7f0e10153350 2023-05-22T15:44:55.832 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:55.825+0000 7f0e16886640 1 -- 172.21.15.16:0/1258930557 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e1015d0c0 con 0x7f0e10153350 2023-05-22T15:44:55.832 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:55.825+0000 7f0e0cff9640 1 -- 172.21.15.16:0/1258930557 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e04003860 con 0x7f0e10153350 2023-05-22T15:44:55.832 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:55.825+0000 7f0e16886640 1 -- 172.21.15.16:0/1258930557 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e100d03e0 con 0x7f0e10153350 2023-05-22T15:44:55.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:55.825+0000 7f0e16886640 1 -- 172.21.15.16:0/1258930557 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e1004dbe0 con 0x7f0e10153350 2023-05-22T15:44:55.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:55.825+0000 7f0e0cff9640 1 -- 172.21.15.16:0/1258930557 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 50) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f0e040039c0 con 0x7f0e10153350 2023-05-22T15:44:55.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:55.825+0000 7f0e0cff9640 1 --2- 172.21.15.16:0/1258930557 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f0dfc0654d0 0x7f0dfc067980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:55.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:55.825+0000 7f0e0cff9640 1 -- 172.21.15.16:0/1258930557 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(122..122 src has 1..122) v4 ==== 6718+0+0 (secure 0 0 0) 0x7f0e04082dd0 con 0x7f0e10153350 2023-05-22T15:44:55.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:55.825+0000 7f0e0ffff640 1 --2- 172.21.15.16:0/1258930557 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f0dfc0654d0 0x7f0dfc067980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:44:55.834 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:55.825+0000 7f0e0ffff640 1 --2- 172.21.15.16:0/1258930557 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f0dfc0654d0 0x7f0dfc067980 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f0e000ab000 tx=0x7f0e000a7070 comp rx=0 tx=0).ready entity=mgr.8304 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:44:55.834 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:55.829+0000 7f0e0cff9640 1 -- 172.21.15.16:0/1258930557 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f0e040540e0 con 0x7f0e10153350 2023-05-22T15:44:55.988 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:55.985+0000 7f0e16886640 1 -- 172.21.15.16:0/1258930557 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f0e100b4be0 con 0x7f0e10153350 2023-05-22T15:44:56.674 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:56.669+0000 7f0e0cff9640 1 -- 172.21.15.16:0/1258930557 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v123) v1 ==== 169+0+0 (secure 0 0 0) 0x7f0e040230d0 con 0x7f0e10153350 2023-05-22T15:44:56.676 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'cephfs_data' removed 2023-05-22T15:44:56.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:56.677+0000 7f0e16886640 1 -- 172.21.15.16:0/1258930557 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f0dfc0654d0 msgr2=0x7f0dfc067980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:44:56.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:56.677+0000 7f0e16886640 1 --2- 172.21.15.16:0/1258930557 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f0dfc0654d0 0x7f0dfc067980 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f0e000ab000 tx=0x7f0e000a7070 comp rx=0 tx=0).stop 2023-05-22T15:44:56.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:56.677+0000 7f0e16886640 1 -- 172.21.15.16:0/1258930557 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e10153350 msgr2=0x7f0e10155700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:44:56.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:56.677+0000 7f0e16886640 1 --2- 172.21.15.16:0/1258930557 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e10153350 0x7f0e10155700 secure :-1 s=READY pgs=1589 cs=0 l=1 rev1=1 crypto rx=0x7f0e04011900 tx=0x7f0e040119f0 comp rx=0 tx=0).stop 2023-05-22T15:44:56.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:56.677+0000 7f0e16886640 1 -- 172.21.15.16:0/1258930557 shutdown_connections 2023-05-22T15:44:56.683 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:56.677+0000 7f0e16886640 1 --2- 172.21.15.16:0/1258930557 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0e101537c0 0x7f0e10151540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:56.683 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:56.677+0000 7f0e16886640 1 --2- 172.21.15.16:0/1258930557 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e10153350 0x7f0e10155700 unknown :-1 s=CLOSED pgs=1589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:56.683 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:56.677+0000 7f0e16886640 1 --2- 172.21.15.16:0/1258930557 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f0dfc0654d0 0x7f0dfc067980 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:56.683 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:56.677+0000 7f0e16886640 1 --2- 172.21.15.16:0/1258930557 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0df8003980 0x7f0e100cbfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:56.684 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:56.677+0000 7f0e16886640 1 -- 172.21.15.16:0/1258930557 >> 172.21.15.16:0/1258930557 conn(0x7f0e10070e50 msgr2=0x7f0e1015d3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:44:56.684 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:56.681+0000 7f0e16886640 1 -- 172.21.15.16:0/1258930557 shutdown_connections 2023-05-22T15:44:56.684 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:56.681+0000 7f0e16886640 1 -- 172.21.15.16:0/1258930557 wait complete. 2023-05-22T15:44:56.705 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2023-05-22T15:44:56.827 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:56.825+0000 7f0d4fecc640 1 Processor -- start 2023-05-22T15:44:56.828 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:56.825+0000 7f0d4fecc640 1 -- start start 2023-05-22T15:44:56.832 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:56.825+0000 7f0d4fecc640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0d48158a10 0x7f0d4806e480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:56.832 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:56.825+0000 7f0d4fecc640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f0d4806cdf0 0x7f0d48070bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:56.832 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:56.825+0000 7f0d4fecc640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d48158de0 con 0x7f0d48070760 2023-05-22T15:44:56.832 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:56.825+0000 7f0d4fecc640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d4806e060 con 0x7f0d4806cdf0 2023-05-22T15:44:56.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:56.825+0000 7f0d4fecc640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d480711e0 con 0x7f0d48158a10 2023-05-22T15:44:56.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:56.825+0000 7f0d4d467640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f0d48070760 0x7f0d4806c6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:52070/0 (socket says 172.21.15.16:52070) 2023-05-22T15:44:56.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:56.825+0000 7f0d4d467640 1 -- 172.21.15.16:0/286632031 learned_addr learned my addr 172.21.15.16:0/286632031 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:44:56.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:56.825+0000 7f0d4dc68640 1 --2- 172.21.15.16:0/286632031 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0d48158a10 0x7f0d4806e480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:44:56.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:56.825+0000 7f0d4e469640 1 --2- 172.21.15.16:0/286632031 >> v2:172.21.15.62:3300/0 conn(0x7f0d4806cdf0 0x7f0d48070bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:44:56.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:56.825+0000 7f0d4cc66640 1 -- 172.21.15.16:0/286632031 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1507507947 0 0) 0x7f0d48158de0 con 0x7f0d48070760 2023-05-22T15:44:56.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:56.825+0000 7f0d4cc66640 1 -- 172.21.15.16:0/286632031 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d34003530 con 0x7f0d48070760 2023-05-22T15:44:56.834 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:56.825+0000 7f0d4cc66640 1 -- 172.21.15.16:0/286632031 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1580603796 0 0) 0x7f0d34003530 con 0x7f0d48070760 2023-05-22T15:44:56.834 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:56.825+0000 7f0d4cc66640 1 -- 172.21.15.16:0/286632031 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0d48158a10 msgr2=0x7f0d4806e480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:44:56.834 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:56.825+0000 7f0d4cc66640 1 --2- 172.21.15.16:0/286632031 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0d48158a10 0x7f0d4806e480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:56.834 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:56.825+0000 7f0d4cc66640 1 -- 172.21.15.16:0/286632031 >> v2:172.21.15.62:3300/0 conn(0x7f0d4806cdf0 msgr2=0x7f0d48070bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:44:56.834 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:56.825+0000 7f0d4cc66640 1 --2- 172.21.15.16:0/286632031 >> v2:172.21.15.62:3300/0 conn(0x7f0d4806cdf0 0x7f0d48070bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:56.834 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:56.825+0000 7f0d4cc66640 1 -- 172.21.15.16:0/286632031 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d480714c0 con 0x7f0d48070760 2023-05-22T15:44:56.834 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:56.825+0000 7f0d4cc66640 1 -- 172.21.15.16:0/286632031 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7f0d38001680 con 0x7f0d48070760 2023-05-22T15:44:56.835 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:56.825+0000 7f0d4cc66640 1 -- 172.21.15.16:0/286632031 >> v1:172.21.15.16:6789/0 conn(0x7f0d48070760 legacy=0x7f0d4806c6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:44:56.835 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:56.825+0000 7f0d4cc66640 1 --2- 172.21.15.16:0/286632031 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0d34003980 0x7f0d34003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:56.835 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:56.825+0000 7f0d4cc66640 1 --2- 172.21.15.16:0/286632031 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0d48158a10 0x7f0d340047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:56.835 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:56.825+0000 7f0d4cc66640 1 --2- 172.21.15.16:0/286632031 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d34004d10 0x7f0d34005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:56.835 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:56.825+0000 7f0d4cc66640 1 -- 172.21.15.16:0/286632031 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d4806e060 con 0x7f0d34004d10 2023-05-22T15:44:56.835 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:56.825+0000 7f0d4cc66640 1 -- 172.21.15.16:0/286632031 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d480711e0 con 0x7f0d34003980 2023-05-22T15:44:56.835 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:56.825+0000 7f0d4cc66640 1 -- 172.21.15.16:0/286632031 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d3400b990 con 0x7f0d48158a10 2023-05-22T15:44:56.835 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:56.825+0000 7f0d4e469640 1 --2- 172.21.15.16:0/286632031 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d34004d10 0x7f0d34005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:44:56.836 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:56.825+0000 7f0d4d467640 1 --2- 172.21.15.16:0/286632031 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0d48158a10 0x7f0d340047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:44:56.836 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:56.825+0000 7f0d4dc68640 1 --2- 172.21.15.16:0/286632031 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0d34003980 0x7f0d34003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:44:56.836 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:56.825+0000 7f0d4e469640 1 -- 172.21.15.16:0/286632031 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0d48158a10 msgr2=0x7f0d340047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:44:56.836 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:56.825+0000 7f0d4e469640 1 --2- 172.21.15.16:0/286632031 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0d48158a10 0x7f0d340047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:56.836 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:56.825+0000 7f0d4e469640 1 -- 172.21.15.16:0/286632031 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0d34003980 msgr2=0x7f0d34003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:44:56.836 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:56.825+0000 7f0d4e469640 1 --2- 172.21.15.16:0/286632031 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0d34003980 0x7f0d34003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:56.836 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:56.825+0000 7f0d4e469640 1 -- 172.21.15.16:0/286632031 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d34003530 con 0x7f0d34004d10 2023-05-22T15:44:56.837 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:56.825+0000 7f0d4d467640 1 --2- 172.21.15.16:0/286632031 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0d48158a10 0x7f0d340047d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:44:56.837 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:56.825+0000 7f0d4dc68640 1 --2- 172.21.15.16:0/286632031 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0d34003980 0x7f0d34003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:44:56.837 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:56.829+0000 7f0d4e469640 1 --2- 172.21.15.16:0/286632031 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d34004d10 0x7f0d34005180 secure :-1 s=READY pgs=1591 cs=0 l=1 rev1=1 crypto rx=0x7f0d40009f90 tx=0x7f0d4001cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24bc15785d5b45aa server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:44:56.839 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:56.829+0000 7f0d4cc66640 1 -- 172.21.15.16:0/286632031 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d400a5c70 con 0x7f0d34004d10 2023-05-22T15:44:56.839 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:56.829+0000 7f0d4cc66640 1 -- 172.21.15.16:0/286632031 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d4000f040 con 0x7f0d34004d10 2023-05-22T15:44:56.839 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:56.829+0000 7f0d4cc66640 1 -- 172.21.15.16:0/286632031 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d400ad440 con 0x7f0d34004d10 2023-05-22T15:44:56.840 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:56.829+0000 7f0d4fecc640 1 -- 172.21.15.16:0/286632031 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d34004d10 msgr2=0x7f0d34005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:44:56.840 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:56.829+0000 7f0d4fecc640 1 --2- 172.21.15.16:0/286632031 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d34004d10 0x7f0d34005180 secure :-1 s=READY pgs=1591 cs=0 l=1 rev1=1 crypto rx=0x7f0d40009f90 tx=0x7f0d4001cc60 comp rx=0 tx=0).stop 2023-05-22T15:44:56.840 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:56.829+0000 7f0d4e469640 1 -- 172.21.15.16:0/286632031 reap_dead start 2023-05-22T15:44:56.840 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:56.829+0000 7f0d4fecc640 1 -- 172.21.15.16:0/286632031 shutdown_connections 2023-05-22T15:44:56.840 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:56.829+0000 7f0d4fecc640 1 -- 172.21.15.16:0/286632031 >> 172.21.15.16:0/286632031 conn(0x7f0d480cc870 msgr2=0x7f0d480cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:44:56.840 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:56.829+0000 7f0d4fecc640 1 -- 172.21.15.16:0/286632031 shutdown_connections 2023-05-22T15:44:56.840 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:56.829+0000 7f0d4fecc640 1 -- 172.21.15.16:0/286632031 wait complete. 2023-05-22T15:44:56.841 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:56.829+0000 7f0d4fecc640 1 Processor -- start 2023-05-22T15:44:56.841 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:56.829+0000 7f0d4fecc640 1 -- start start 2023-05-22T15:44:56.841 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:56.833+0000 7f0d4fecc640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0d4815b470 0x7f0d480c88e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:56.841 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:56.833+0000 7f0d4fecc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d4815b840 0x7f0d480b3f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:56.841 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:56.833+0000 7f0d4fecc640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0d480c7280 0x7f0d48152720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:56.841 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:56.833+0000 7f0d4fecc640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d480c8450 con 0x7f0d4815b840 2023-05-22T15:44:56.841 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:56.833+0000 7f0d4fecc640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d480c7840 con 0x7f0d4815b470 2023-05-22T15:44:56.842 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:56.833+0000 7f0d4fecc640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d480c7b20 con 0x7f0d480c7280 2023-05-22T15:44:56.842 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:56.833+0000 7f0d4d467640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d4815b840 0x7f0d480b3f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:44:56.842 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:56.833+0000 7f0d4d467640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d4815b840 0x7f0d480b3f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59470/0 (socket says 172.21.15.16:59470) 2023-05-22T15:44:56.842 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:56.833+0000 7f0d4d467640 1 -- 172.21.15.16:0/3439280577 learned_addr learned my addr 172.21.15.16:0/3439280577 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:44:56.842 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:56.833+0000 7f0d4e469640 1 --2- 172.21.15.16:0/3439280577 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0d480c7280 0x7f0d48152720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:44:56.842 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:56.833+0000 7f0d4dc68640 1 --2- 172.21.15.16:0/3439280577 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0d4815b470 0x7f0d480c88e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:44:56.842 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:56.833+0000 7f0d4d467640 1 -- 172.21.15.16:0/3439280577 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0d480c7280 msgr2=0x7f0d48152720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:44:56.842 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:56.833+0000 7f0d4d467640 1 --2- 172.21.15.16:0/3439280577 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0d480c7280 0x7f0d48152720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:56.843 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:56.833+0000 7f0d4d467640 1 -- 172.21.15.16:0/3439280577 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0d4815b470 msgr2=0x7f0d480c88e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:44:56.843 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:56.833+0000 7f0d4d467640 1 --2- 172.21.15.16:0/3439280577 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0d4815b470 0x7f0d480c88e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:56.843 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:56.833+0000 7f0d4d467640 1 -- 172.21.15.16:0/3439280577 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d480cfe40 con 0x7f0d4815b840 2023-05-22T15:44:56.843 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:56.833+0000 7f0d4d467640 1 --2- 172.21.15.16:0/3439280577 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d4815b840 0x7f0d480b3f90 secure :-1 s=READY pgs=1592 cs=0 l=1 rev1=1 crypto rx=0x7f0d38010c50 tx=0x7f0d3800e4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:44:56.843 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:56.833+0000 7f0d3e7fc640 1 -- 172.21.15.16:0/3439280577 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d3800edf0 con 0x7f0d4815b840 2023-05-22T15:44:56.843 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:56.833+0000 7f0d3e7fc640 1 -- 172.21.15.16:0/3439280577 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d380143a0 con 0x7f0d4815b840 2023-05-22T15:44:56.843 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:56.833+0000 7f0d3e7fc640 1 -- 172.21.15.16:0/3439280577 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d38014710 con 0x7f0d4815b840 2023-05-22T15:44:56.843 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:56.833+0000 7f0d4fecc640 1 -- 172.21.15.16:0/3439280577 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d4806dc80 con 0x7f0d4815b840 2023-05-22T15:44:56.844 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:56.833+0000 7f0d4fecc640 1 -- 172.21.15.16:0/3439280577 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d4806e480 con 0x7f0d4815b840 2023-05-22T15:44:56.844 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:56.833+0000 7f0d4fecc640 1 -- 172.21.15.16:0/3439280577 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d480c8600 con 0x7f0d4815b840 2023-05-22T15:44:56.844 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:56.833+0000 7f0d3e7fc640 1 -- 172.21.15.16:0/3439280577 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 50) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f0d38014870 con 0x7f0d4815b840 2023-05-22T15:44:56.844 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:56.833+0000 7f0d3e7fc640 1 --2- 172.21.15.16:0/3439280577 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f0d2c0654d0 0x7f0d2c067980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:56.844 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:56.833+0000 7f0d3e7fc640 1 -- 172.21.15.16:0/3439280577 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(123..123 src has 1..123) v4 ==== 6304+0+0 (secure 0 0 0) 0x7f0d38085740 con 0x7f0d4815b840 2023-05-22T15:44:56.844 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:56.833+0000 7f0d4dc68640 1 --2- 172.21.15.16:0/3439280577 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f0d2c0654d0 0x7f0d2c067980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:44:56.845 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:56.837+0000 7f0d4dc68640 1 --2- 172.21.15.16:0/3439280577 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f0d2c0654d0 0x7f0d2c067980 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f0d48158130 tx=0x7f0d44008040 comp rx=0 tx=0).ready entity=mgr.8304 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:44:56.845 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:56.837+0000 7f0d3e7fc640 1 -- 172.21.15.16:0/3439280577 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f0d480c8600 con 0x7f0d4815b840 2023-05-22T15:44:57.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:56.997+0000 7f0d4fecc640 1 -- 172.21.15.16:0/3439280577 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7f0d480c8600 con 0x7f0d4815b840 2023-05-22T15:44:57.010 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.005+0000 7f0d3e7fc640 1 -- 172.21.15.16:0/3439280577 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v37) v1 ==== 120+0+0 (secure 0 0 0) 0x7f0d480c8600 con 0x7f0d4815b840 2023-05-22T15:44:57.013 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.009+0000 7f0d4fecc640 1 -- 172.21.15.16:0/3439280577 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f0d2c0654d0 msgr2=0x7f0d2c067980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:44:57.014 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.009+0000 7f0d4fecc640 1 --2- 172.21.15.16:0/3439280577 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f0d2c0654d0 0x7f0d2c067980 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f0d48158130 tx=0x7f0d44008040 comp rx=0 tx=0).stop 2023-05-22T15:44:57.014 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.009+0000 7f0d4fecc640 1 -- 172.21.15.16:0/3439280577 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d4815b840 msgr2=0x7f0d480b3f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:44:57.014 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.009+0000 7f0d4fecc640 1 --2- 172.21.15.16:0/3439280577 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d4815b840 0x7f0d480b3f90 secure :-1 s=READY pgs=1592 cs=0 l=1 rev1=1 crypto rx=0x7f0d38010c50 tx=0x7f0d3800e4d0 comp rx=0 tx=0).stop 2023-05-22T15:44:57.014 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.009+0000 7f0d4fecc640 1 -- 172.21.15.16:0/3439280577 shutdown_connections 2023-05-22T15:44:57.014 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.009+0000 7f0d4fecc640 1 --2- 172.21.15.16:0/3439280577 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f0d2c0654d0 0x7f0d2c067980 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:57.014 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.009+0000 7f0d4fecc640 1 --2- 172.21.15.16:0/3439280577 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0d480c7280 0x7f0d48152720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:57.015 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.009+0000 7f0d4fecc640 1 --2- 172.21.15.16:0/3439280577 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d4815b840 0x7f0d480b3f90 unknown :-1 s=CLOSED pgs=1592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:57.015 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.009+0000 7f0d4fecc640 1 --2- 172.21.15.16:0/3439280577 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0d4815b470 0x7f0d480c88e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:57.015 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.009+0000 7f0d4fecc640 1 -- 172.21.15.16:0/3439280577 >> 172.21.15.16:0/3439280577 conn(0x7f0d480cc870 msgr2=0x7f0d4815c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:44:57.015 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.009+0000 7f0d4fecc640 1 -- 172.21.15.16:0/3439280577 shutdown_connections 2023-05-22T15:44:57.015 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.009+0000 7f0d4fecc640 1 -- 172.21.15.16:0/3439280577 wait complete. 2023-05-22T15:44:57.033 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2023-05-22T15:44:57.161 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.157+0000 7fca9911d640 1 Processor -- start 2023-05-22T15:44:57.161 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.157+0000 7fca9911d640 1 -- start start 2023-05-22T15:44:57.162 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.157+0000 7fca9911d640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fca94159970 0x7fca94159d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:57.163 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.157+0000 7fca9911d640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fca9415d680 0x7fca940cb260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:57.163 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.157+0000 7fca9911d640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fca9415dad0 0x7fca9406b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:57.163 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.157+0000 7fca9911d640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca94070c30 con 0x7fca9415d680 2023-05-22T15:44:57.163 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.157+0000 7fca9911d640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca94070e60 con 0x7fca9415dad0 2023-05-22T15:44:57.163 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.157+0000 7fca9911d640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca94071140 con 0x7fca94159970 2023-05-22T15:44:57.163 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.161+0000 7fca9259c640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fca9415d680 0x7fca940cb260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:44:57.163 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.161+0000 7fca9259c640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fca9415d680 0x7fca940cb260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59476/0 (socket says 172.21.15.16:59476) 2023-05-22T15:44:57.164 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.161+0000 7fca9259c640 1 -- 172.21.15.16:0/414336040 learned_addr learned my addr 172.21.15.16:0/414336040 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:44:57.164 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.161+0000 7fca92d9d640 1 --2- 172.21.15.16:0/414336040 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fca94159970 0x7fca94159d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:44:57.164 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.161+0000 7fca9359e640 1 --2- 172.21.15.16:0/414336040 >> v2:172.21.15.62:3300/0 conn(0x7fca9415dad0 0x7fca9406b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:44:57.165 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.161+0000 7fca9259c640 1 -- 172.21.15.16:0/414336040 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fca94159970 msgr2=0x7fca94159d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:44:57.165 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.161+0000 7fca9259c640 1 --2- 172.21.15.16:0/414336040 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fca94159970 0x7fca94159d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:57.165 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.161+0000 7fca9259c640 1 -- 172.21.15.16:0/414336040 >> v2:172.21.15.62:3300/0 conn(0x7fca9415dad0 msgr2=0x7fca9406b6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:44:57.166 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.161+0000 7fca9259c640 1 --2- 172.21.15.16:0/414336040 >> v2:172.21.15.62:3300/0 conn(0x7fca9415dad0 0x7fca9406b6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:57.166 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.161+0000 7fca9259c640 1 -- 172.21.15.16:0/414336040 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca94071420 con 0x7fca9415d680 2023-05-22T15:44:57.166 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.161+0000 7fca9259c640 1 --2- 172.21.15.16:0/414336040 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fca9415d680 0x7fca940cb260 secure :-1 s=READY pgs=1593 cs=0 l=1 rev1=1 crypto rx=0x7fca940b40e0 tx=0x7fca8801c440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c34e6943148b0f31 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:44:57.166 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.161+0000 7fca91d9b640 1 -- 172.21.15.16:0/414336040 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca880ac640 con 0x7fca9415d680 2023-05-22T15:44:57.166 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.161+0000 7fca91d9b640 1 -- 172.21.15.16:0/414336040 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca880ac7a0 con 0x7fca9415d680 2023-05-22T15:44:57.166 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.161+0000 7fca91d9b640 1 -- 172.21.15.16:0/414336040 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca880acb30 con 0x7fca9415d680 2023-05-22T15:44:57.167 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.161+0000 7fca9911d640 1 -- 172.21.15.16:0/414336040 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fca9415d680 msgr2=0x7fca940cb260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:44:57.168 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.161+0000 7fca9911d640 1 --2- 172.21.15.16:0/414336040 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fca9415d680 0x7fca940cb260 secure :-1 s=READY pgs=1593 cs=0 l=1 rev1=1 crypto rx=0x7fca940b40e0 tx=0x7fca8801c440 comp rx=0 tx=0).stop 2023-05-22T15:44:57.168 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.161+0000 7fca9911d640 1 -- 172.21.15.16:0/414336040 shutdown_connections 2023-05-22T15:44:57.168 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.161+0000 7fca9911d640 1 --2- 172.21.15.16:0/414336040 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fca9415d680 0x7fca940cb260 unknown :-1 s=CLOSED pgs=1593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:57.168 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.161+0000 7fca9911d640 1 --2- 172.21.15.16:0/414336040 >> v2:172.21.15.62:3300/0 conn(0x7fca9415dad0 0x7fca9406b6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:57.168 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.161+0000 7fca9911d640 1 --2- 172.21.15.16:0/414336040 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fca94159970 0x7fca94159d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:57.169 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.161+0000 7fca9911d640 1 -- 172.21.15.16:0/414336040 >> 172.21.15.16:0/414336040 conn(0x7fca94069720 msgr2=0x7fca94069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:44:57.169 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.161+0000 7fca9911d640 1 -- 172.21.15.16:0/414336040 shutdown_connections 2023-05-22T15:44:57.169 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.161+0000 7fca9911d640 1 -- 172.21.15.16:0/414336040 wait complete. 2023-05-22T15:44:57.169 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.165+0000 7fca9911d640 1 Processor -- start 2023-05-22T15:44:57.169 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.165+0000 7fca9911d640 1 -- start start 2023-05-22T15:44:57.172 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.165+0000 7fca9911d640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fca94159970 0x7fca9414d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:57.172 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.165+0000 7fca9911d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fca9415d680 0x7fca9414c730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:57.172 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.165+0000 7fca9911d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fca9415dad0 0x7fca94152340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:57.172 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.165+0000 7fca9911d640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca9414d300 con 0x7fca9415dad0 2023-05-22T15:44:57.172 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.165+0000 7fca9911d640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca941635d0 con 0x7fca9415d680 2023-05-22T15:44:57.172 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.165+0000 7fca9911d640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca94150ab0 con 0x7fca94159970 2023-05-22T15:44:57.173 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.165+0000 7fca9359e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fca9415dad0 0x7fca94152340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:44:57.173 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.165+0000 7fca92d9d640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fca94159970 0x7fca9414d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:44:57.173 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.165+0000 7fca9259c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fca9415d680 0x7fca9414c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:44:57.173 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.165+0000 7fca9359e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fca9415dad0 0x7fca94152340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59490/0 (socket says 172.21.15.16:59490) 2023-05-22T15:44:57.173 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.165+0000 7fca9359e640 1 -- 172.21.15.16:0/3151608868 learned_addr learned my addr 172.21.15.16:0/3151608868 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:44:57.173 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.165+0000 7fca9359e640 1 -- 172.21.15.16:0/3151608868 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fca94159970 msgr2=0x7fca9414d4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:44:57.174 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.165+0000 7fca9359e640 1 --2- 172.21.15.16:0/3151608868 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fca94159970 0x7fca9414d4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:57.174 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.165+0000 7fca9359e640 1 -- 172.21.15.16:0/3151608868 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fca9415d680 msgr2=0x7fca9414c730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:44:57.174 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.165+0000 7fca9359e640 1 --2- 172.21.15.16:0/3151608868 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fca9415d680 0x7fca9414c730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:57.174 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.165+0000 7fca9359e640 1 -- 172.21.15.16:0/3151608868 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca880094d0 con 0x7fca9415dad0 2023-05-22T15:44:57.175 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.165+0000 7fca9359e640 1 --2- 172.21.15.16:0/3151608868 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fca9415dad0 0x7fca94152340 secure :-1 s=READY pgs=1594 cs=0 l=1 rev1=1 crypto rx=0x7fca8400d500 tx=0x7fca8400d9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:44:57.175 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.165+0000 7fca7b7fe640 1 -- 172.21.15.16:0/3151608868 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca840125f0 con 0x7fca9415dad0 2023-05-22T15:44:57.175 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.165+0000 7fca7b7fe640 1 -- 172.21.15.16:0/3151608868 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca84012750 con 0x7fca9415dad0 2023-05-22T15:44:57.176 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.165+0000 7fca9911d640 1 -- 172.21.15.16:0/3151608868 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca940cae20 con 0x7fca9415dad0 2023-05-22T15:44:57.176 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.165+0000 7fca7b7fe640 1 -- 172.21.15.16:0/3151608868 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca84012b00 con 0x7fca9415dad0 2023-05-22T15:44:57.176 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.165+0000 7fca9911d640 1 -- 172.21.15.16:0/3151608868 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca940cac90 con 0x7fca9415dad0 2023-05-22T15:44:57.176 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.169+0000 7fca9911d640 1 -- 172.21.15.16:0/3151608868 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca9415adc0 con 0x7fca9415dad0 2023-05-22T15:44:57.177 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.169+0000 7fca7b7fe640 1 -- 172.21.15.16:0/3151608868 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 50) v1 ==== 84332+0+0 (secure 0 0 0) 0x7fca84012c60 con 0x7fca9415dad0 2023-05-22T15:44:57.177 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.169+0000 7fca7b7fe640 1 --2- 172.21.15.16:0/3151608868 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fca800654d0 0x7fca80067980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:57.177 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.169+0000 7fca7b7fe640 1 -- 172.21.15.16:0/3151608868 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(123..123 src has 1..123) v4 ==== 6304+0+0 (secure 0 0 0) 0x7fca84082be0 con 0x7fca9415dad0 2023-05-22T15:44:57.178 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.169+0000 7fca92d9d640 1 --2- 172.21.15.16:0/3151608868 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fca800654d0 0x7fca80067980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:44:57.178 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.169+0000 7fca92d9d640 1 --2- 172.21.15.16:0/3151608868 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fca800654d0 0x7fca80067980 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fca9414ee80 tx=0x7fca8c009830 comp rx=0 tx=0).ready entity=mgr.8304 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:44:57.179 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.173+0000 7fca7b7fe640 1 -- 172.21.15.16:0/3151608868 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fca840506c0 con 0x7fca9415dad0 2023-05-22T15:44:57.329 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.325+0000 7fca9911d640 1 -- 172.21.15.16:0/3151608868 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fca9404dbe0 con 0x7fca9415dad0 2023-05-22T15:44:57.339 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.337+0000 7fca7b7fe640 1 -- 172.21.15.16:0/3151608868 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v38) v1 ==== 104+0+0 (secure 0 0 0) 0x7fca84053ce0 con 0x7fca9415dad0 2023-05-22T15:44:57.342 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.337+0000 7fca9911d640 1 -- 172.21.15.16:0/3151608868 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fca800654d0 msgr2=0x7fca80067980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:44:57.342 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.337+0000 7fca9911d640 1 --2- 172.21.15.16:0/3151608868 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fca800654d0 0x7fca80067980 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fca9414ee80 tx=0x7fca8c009830 comp rx=0 tx=0).stop 2023-05-22T15:44:57.342 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.337+0000 7fca9911d640 1 -- 172.21.15.16:0/3151608868 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fca9415dad0 msgr2=0x7fca94152340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:44:57.342 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.337+0000 7fca9911d640 1 --2- 172.21.15.16:0/3151608868 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fca9415dad0 0x7fca94152340 secure :-1 s=READY pgs=1594 cs=0 l=1 rev1=1 crypto rx=0x7fca8400d500 tx=0x7fca8400d9c0 comp rx=0 tx=0).stop 2023-05-22T15:44:57.343 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.337+0000 7fca9911d640 1 -- 172.21.15.16:0/3151608868 shutdown_connections 2023-05-22T15:44:57.343 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.337+0000 7fca9911d640 1 --2- 172.21.15.16:0/3151608868 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fca9415dad0 0x7fca94152340 unknown :-1 s=CLOSED pgs=1594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:57.343 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.337+0000 7fca9911d640 1 --2- 172.21.15.16:0/3151608868 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fca9415d680 0x7fca9414c730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:57.344 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.337+0000 7fca9911d640 1 --2- 172.21.15.16:0/3151608868 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fca800654d0 0x7fca80067980 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:57.344 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.337+0000 7fca9911d640 1 --2- 172.21.15.16:0/3151608868 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fca94159970 0x7fca9414d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:57.345 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.337+0000 7fca9911d640 1 -- 172.21.15.16:0/3151608868 >> 172.21.15.16:0/3151608868 conn(0x7fca94069720 msgr2=0x7fca94158140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:44:57.345 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.337+0000 7fca9911d640 1 -- 172.21.15.16:0/3151608868 shutdown_connections 2023-05-22T15:44:57.345 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.337+0000 7fca9911d640 1 -- 172.21.15.16:0/3151608868 wait complete. 2023-05-22T15:44:57.360 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2023-05-22T15:44:57.494 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.489+0000 7f21d1607640 1 Processor -- start 2023-05-22T15:44:57.495 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.489+0000 7f21d1607640 1 -- start start 2023-05-22T15:44:57.497 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.493+0000 7f21d1607640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f21cc15bca0 0x7f21cc160160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:57.497 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.493+0000 7f21d1607640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f21cc0b3ee0 0x7f21cc0b4330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:57.497 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.493+0000 7f21d1607640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f21cc0b4870 0x7f21cc0ce900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:57.498 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.493+0000 7f21d1607640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21cc15c070 con 0x7f21cc0b4870 2023-05-22T15:44:57.498 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.493+0000 7f21d1607640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21cc0d0400 con 0x7f21cc0b3ee0 2023-05-22T15:44:57.498 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.493+0000 7f21d1607640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc1606a0 con 0x7f21cc15bca0 2023-05-22T15:44:57.498 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.493+0000 7f21cb7fe640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f21cc0b4870 0x7f21cc0ce900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:44:57.499 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.493+0000 7f21cb7fe640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f21cc0b4870 0x7f21cc0ce900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59494/0 (socket says 172.21.15.16:59494) 2023-05-22T15:44:57.499 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.493+0000 7f21cb7fe640 1 -- 172.21.15.16:0/76208827 learned_addr learned my addr 172.21.15.16:0/76208827 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:44:57.499 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.493+0000 7f21ca7fc640 1 --2- 172.21.15.16:0/76208827 >> v2:172.21.15.62:3300/0 conn(0x7f21cc0b3ee0 0x7f21cc0b4330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:44:57.500 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.493+0000 7f21caffd640 1 --2- 172.21.15.16:0/76208827 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f21cc15bca0 0x7f21cc160160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:44:57.500 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.493+0000 7f21cb7fe640 1 -- 172.21.15.16:0/76208827 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f21cc15bca0 msgr2=0x7f21cc160160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:44:57.500 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.493+0000 7f21cb7fe640 1 --2- 172.21.15.16:0/76208827 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f21cc15bca0 0x7f21cc160160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:57.500 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.493+0000 7f21cb7fe640 1 -- 172.21.15.16:0/76208827 >> v2:172.21.15.62:3300/0 conn(0x7f21cc0b3ee0 msgr2=0x7f21cc0b4330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:44:57.501 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.493+0000 7f21cb7fe640 1 --2- 172.21.15.16:0/76208827 >> v2:172.21.15.62:3300/0 conn(0x7f21cc0b3ee0 0x7f21cc0b4330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:57.501 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.493+0000 7f21cb7fe640 1 -- 172.21.15.16:0/76208827 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21cc0d00d0 con 0x7f21cc0b4870 2023-05-22T15:44:57.505 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.493+0000 7f21cb7fe640 1 --2- 172.21.15.16:0/76208827 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f21cc0b4870 0x7f21cc0ce900 secure :-1 s=READY pgs=1595 cs=0 l=1 rev1=1 crypto rx=0x7f21cc065e50 tx=0x7f21c401c440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c83717ee912f544d server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:44:57.505 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.493+0000 7f21c9ffb640 1 -- 172.21.15.16:0/76208827 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21c40ab600 con 0x7f21cc0b4870 2023-05-22T15:44:57.506 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.493+0000 7f21c9ffb640 1 -- 172.21.15.16:0/76208827 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21c40ab760 con 0x7f21cc0b4870 2023-05-22T15:44:57.506 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.493+0000 7f21c9ffb640 1 -- 172.21.15.16:0/76208827 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21c40abaf0 con 0x7f21cc0b4870 2023-05-22T15:44:57.506 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.493+0000 7f21d1607640 1 -- 172.21.15.16:0/76208827 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f21cc0b4870 msgr2=0x7f21cc0ce900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:44:57.506 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.493+0000 7f21d1607640 1 --2- 172.21.15.16:0/76208827 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f21cc0b4870 0x7f21cc0ce900 secure :-1 s=READY pgs=1595 cs=0 l=1 rev1=1 crypto rx=0x7f21cc065e50 tx=0x7f21c401c440 comp rx=0 tx=0).stop 2023-05-22T15:44:57.507 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.493+0000 7f21d1607640 1 -- 172.21.15.16:0/76208827 shutdown_connections 2023-05-22T15:44:57.507 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.493+0000 7f21d1607640 1 --2- 172.21.15.16:0/76208827 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f21cc0b4870 0x7f21cc0ce900 unknown :-1 s=CLOSED pgs=1595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:57.507 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.493+0000 7f21d1607640 1 --2- 172.21.15.16:0/76208827 >> v2:172.21.15.62:3300/0 conn(0x7f21cc0b3ee0 0x7f21cc0b4330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:57.507 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.493+0000 7f21d1607640 1 --2- 172.21.15.16:0/76208827 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f21cc15bca0 0x7f21cc160160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:57.508 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.493+0000 7f21d1607640 1 -- 172.21.15.16:0/76208827 >> 172.21.15.16:0/76208827 conn(0x7f21cc0657f0 msgr2=0x7f21cc070e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:44:57.508 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.497+0000 7f21d1607640 1 -- 172.21.15.16:0/76208827 shutdown_connections 2023-05-22T15:44:57.509 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.497+0000 7f21d1607640 1 -- 172.21.15.16:0/76208827 wait complete. 2023-05-22T15:44:57.509 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.497+0000 7f21d1607640 1 Processor -- start 2023-05-22T15:44:57.509 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.497+0000 7f21d1607640 1 -- start start 2023-05-22T15:44:57.510 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.497+0000 7f21d1607640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f21cc0b3ee0 0x7f21cc152280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:57.510 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.497+0000 7f21d1607640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f21cc0b4870 0x7f21cc150170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:57.511 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.497+0000 7f21d1607640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f21cc15bca0 0x7f21cc0c6cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:57.511 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.497+0000 7f21d1607640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc1511d0 con 0x7f21cc15bca0 2023-05-22T15:44:57.511 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.497+0000 7f21d1607640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc151470 con 0x7f21cc0b4870 2023-05-22T15:44:57.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.497+0000 7f21d1607640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc151750 con 0x7f21cc0b3ee0 2023-05-22T15:44:57.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.497+0000 7f21cb7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f21cc15bca0 0x7f21cc0c6cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:44:57.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.497+0000 7f21cb7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f21cc15bca0 0x7f21cc0c6cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59498/0 (socket says 172.21.15.16:59498) 2023-05-22T15:44:57.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.497+0000 7f21cb7fe640 1 -- 172.21.15.16:0/1553462461 learned_addr learned my addr 172.21.15.16:0/1553462461 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:44:57.513 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.497+0000 7f21caffd640 1 --2- 172.21.15.16:0/1553462461 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f21cc0b3ee0 0x7f21cc152280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:44:57.513 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.497+0000 7f21ca7fc640 1 --2- 172.21.15.16:0/1553462461 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f21cc0b4870 0x7f21cc150170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:44:57.513 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.497+0000 7f21cb7fe640 1 -- 172.21.15.16:0/1553462461 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f21cc0b3ee0 msgr2=0x7f21cc152280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:44:57.513 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.497+0000 7f21cb7fe640 1 --2- 172.21.15.16:0/1553462461 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f21cc0b3ee0 0x7f21cc152280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:57.513 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.497+0000 7f21cb7fe640 1 -- 172.21.15.16:0/1553462461 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f21cc0b4870 msgr2=0x7f21cc150170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:44:57.514 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.497+0000 7f21cb7fe640 1 --2- 172.21.15.16:0/1553462461 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f21cc0b4870 0x7f21cc150170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:57.514 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.497+0000 7f21cb7fe640 1 -- 172.21.15.16:0/1553462461 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21c40094d0 con 0x7f21cc15bca0 2023-05-22T15:44:57.514 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.497+0000 7f21ca7fc640 1 --2- 172.21.15.16:0/1553462461 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f21cc0b4870 0x7f21cc150170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:44:57.514 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.497+0000 7f21cb7fe640 1 --2- 172.21.15.16:0/1553462461 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f21cc15bca0 0x7f21cc0c6cf0 secure :-1 s=READY pgs=1596 cs=0 l=1 rev1=1 crypto rx=0x7f21c40ad040 tx=0x7f21c401ac80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:44:57.514 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.497+0000 7f21b37fe640 1 -- 172.21.15.16:0/1553462461 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21c40b4800 con 0x7f21cc15bca0 2023-05-22T15:44:57.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.497+0000 7f21b37fe640 1 -- 172.21.15.16:0/1553462461 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21c40b4980 con 0x7f21cc15bca0 2023-05-22T15:44:57.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.497+0000 7f21d1607640 1 -- 172.21.15.16:0/1553462461 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21cc0cd1a0 con 0x7f21cc15bca0 2023-05-22T15:44:57.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.497+0000 7f21b37fe640 1 -- 172.21.15.16:0/1553462461 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21c40b4d30 con 0x7f21cc15bca0 2023-05-22T15:44:57.516 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.497+0000 7f21d1607640 1 -- 172.21.15.16:0/1553462461 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21cc0b4cb0 con 0x7f21cc15bca0 2023-05-22T15:44:57.516 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.497+0000 7f21d1607640 1 -- 172.21.15.16:0/1553462461 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21cc0d0870 con 0x7f21cc15bca0 2023-05-22T15:44:57.517 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.501+0000 7f21b37fe640 1 -- 172.21.15.16:0/1553462461 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 50) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f21c40c7440 con 0x7f21cc15bca0 2023-05-22T15:44:57.517 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.501+0000 7f21b37fe640 1 --2- 172.21.15.16:0/1553462461 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f21b80654a0 0x7f21b8067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:57.517 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.501+0000 7f21b37fe640 1 -- 172.21.15.16:0/1553462461 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(123..123 src has 1..123) v4 ==== 6304+0+0 (secure 0 0 0) 0x7f21c411d7f0 con 0x7f21cc15bca0 2023-05-22T15:44:57.517 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.501+0000 7f21caffd640 1 --2- 172.21.15.16:0/1553462461 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f21b80654a0 0x7f21b8067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:44:57.517 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.501+0000 7f21caffd640 1 --2- 172.21.15.16:0/1553462461 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f21b80654a0 0x7f21b8067950 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f21bc003fd0 tx=0x7f21bc0039a0 comp rx=0 tx=0).ready entity=mgr.8304 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:44:57.518 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.505+0000 7f21b37fe640 1 -- 172.21.15.16:0/1553462461 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f21c40eb110 con 0x7f21cc15bca0 2023-05-22T15:44:57.668 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.665+0000 7f21d1607640 1 -- 172.21.15.16:0/1553462461 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7f21cc04dbe0 con 0x7f21cc15bca0 2023-05-22T15:44:57.680 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.677+0000 7f21b37fe640 1 -- 172.21.15.16:0/1553462461 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v39) v1 ==== 115+0+0 (secure 0 0 0) 0x7f21c40ee730 con 0x7f21cc15bca0 2023-05-22T15:44:57.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.681+0000 7f21d1607640 1 -- 172.21.15.16:0/1553462461 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f21b80654a0 msgr2=0x7f21b8067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:44:57.683 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.681+0000 7f21d1607640 1 --2- 172.21.15.16:0/1553462461 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f21b80654a0 0x7f21b8067950 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f21bc003fd0 tx=0x7f21bc0039a0 comp rx=0 tx=0).stop 2023-05-22T15:44:57.683 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.681+0000 7f21d1607640 1 -- 172.21.15.16:0/1553462461 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f21cc15bca0 msgr2=0x7f21cc0c6cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:44:57.683 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.681+0000 7f21d1607640 1 --2- 172.21.15.16:0/1553462461 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f21cc15bca0 0x7f21cc0c6cf0 secure :-1 s=READY pgs=1596 cs=0 l=1 rev1=1 crypto rx=0x7f21c40ad040 tx=0x7f21c401ac80 comp rx=0 tx=0).stop 2023-05-22T15:44:57.683 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.681+0000 7f21d1607640 1 -- 172.21.15.16:0/1553462461 shutdown_connections 2023-05-22T15:44:57.683 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.681+0000 7f21d1607640 1 --2- 172.21.15.16:0/1553462461 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f21cc15bca0 0x7f21cc0c6cf0 unknown :-1 s=CLOSED pgs=1596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:57.683 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.681+0000 7f21d1607640 1 --2- 172.21.15.16:0/1553462461 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f21cc0b4870 0x7f21cc150170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:57.683 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.681+0000 7f21d1607640 1 --2- 172.21.15.16:0/1553462461 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f21b80654a0 0x7f21b8067950 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:57.684 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.681+0000 7f21d1607640 1 --2- 172.21.15.16:0/1553462461 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f21cc0b3ee0 0x7f21cc152280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:57.684 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.681+0000 7f21d1607640 1 -- 172.21.15.16:0/1553462461 >> 172.21.15.16:0/1553462461 conn(0x7f21cc0657f0 msgr2=0x7f21cc0d2c50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:44:57.684 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.681+0000 7f21d1607640 1 -- 172.21.15.16:0/1553462461 shutdown_connections 2023-05-22T15:44:57.684 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.681+0000 7f21d1607640 1 -- 172.21.15.16:0/1553462461 wait complete. 2023-05-22T15:44:57.700 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_schedule_auto_deactivation_for_non_existent_path' 2023-05-22T15:44:57.827 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.825+0000 7f7a67392640 1 Processor -- start 2023-05-22T15:44:57.827 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.825+0000 7f7a67392640 1 -- start start 2023-05-22T15:44:57.828 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.825+0000 7f7a67392640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f7a6015cfa0 0x7f7a6006db00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:57.828 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.825+0000 7f7a67392640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a6006e040 con 0x7f7a6006e740 2023-05-22T15:44:57.828 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.825+0000 7f7a67392640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a6006e490 con 0x7f7a600cf870 2023-05-22T15:44:57.828 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.825+0000 7f7a67392640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a600cb9f0 con 0x7f7a6015cfa0 2023-05-22T15:44:57.828 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.825+0000 7f7a6512e640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f7a6015cfa0 0x7f7a6006db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:44:57.829 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.825+0000 7f7a6512e640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f7a6015cfa0 0x7f7a6006db00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59504/0 (socket says 172.21.15.16:59504) 2023-05-22T15:44:57.829 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.825+0000 7f7a6512e640 1 -- 172.21.15.16:0/3872076436 learned_addr learned my addr 172.21.15.16:0/3872076436 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:44:57.829 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.825+0000 7f7a6512e640 1 -- 172.21.15.16:0/3872076436 >> v1:172.21.15.62:6789/0 conn(0x7f7a600cf870 legacy=0x7f7a600b1fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:44:57.829 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.825+0000 7f7a6512e640 1 -- 172.21.15.16:0/3872076436 >> v1:172.21.15.16:6789/0 conn(0x7f7a6006e740 legacy=0x7f7a60070de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:44:57.830 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.825+0000 7f7a6512e640 1 -- 172.21.15.16:0/3872076436 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a600cad70 con 0x7f7a6015cfa0 2023-05-22T15:44:57.830 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.825+0000 7f7a57fff640 1 -- 172.21.15.16:0/3872076436 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4033489165 0 0) 0x7f7a580014a0 con 0x7f7a6006e740 2023-05-22T15:44:57.830 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.829+0000 7f7a6512e640 1 --2- 172.21.15.16:0/3872076436 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7a6015cfa0 0x7f7a6006db00 secure :-1 s=READY pgs=1598 cs=0 l=1 rev1=1 crypto rx=0x7f7a5c009870 tx=0x7f7a5c01c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8972bdd9727c0a6 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:44:57.831 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.829+0000 7f7a57fff640 1 -- 172.21.15.16:0/3872076436 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a5c0aa430 con 0x7f7a6015cfa0 2023-05-22T15:44:57.831 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.829+0000 7f7a57fff640 1 -- 172.21.15.16:0/3872076436 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a5c0aa590 con 0x7f7a6015cfa0 2023-05-22T15:44:57.831 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.829+0000 7f7a57fff640 1 -- 172.21.15.16:0/3872076436 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a5c0aa920 con 0x7f7a6015cfa0 2023-05-22T15:44:57.832 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.829+0000 7f7a67392640 1 -- 172.21.15.16:0/3872076436 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7a6015cfa0 msgr2=0x7f7a6006db00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:44:57.832 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.829+0000 7f7a67392640 1 --2- 172.21.15.16:0/3872076436 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7a6015cfa0 0x7f7a6006db00 secure :-1 s=READY pgs=1598 cs=0 l=1 rev1=1 crypto rx=0x7f7a5c009870 tx=0x7f7a5c01c4d0 comp rx=0 tx=0).stop 2023-05-22T15:44:57.832 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.829+0000 7f7a67392640 1 -- 172.21.15.16:0/3872076436 shutdown_connections 2023-05-22T15:44:57.832 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.829+0000 7f7a67392640 1 --2- 172.21.15.16:0/3872076436 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7a6015cfa0 0x7f7a6006db00 unknown :-1 s=CLOSED pgs=1598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:57.832 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.829+0000 7f7a67392640 1 -- 172.21.15.16:0/3872076436 >> 172.21.15.16:0/3872076436 conn(0x7f7a600cc870 msgr2=0x7f7a600cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:44:57.832 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.829+0000 7f7a67392640 1 -- 172.21.15.16:0/3872076436 shutdown_connections 2023-05-22T15:44:57.832 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.829+0000 7f7a67392640 1 -- 172.21.15.16:0/3872076436 wait complete. 2023-05-22T15:44:57.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.829+0000 7f7a67392640 1 Processor -- start 2023-05-22T15:44:57.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.829+0000 7f7a67392640 1 -- start start 2023-05-22T15:44:57.834 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.829+0000 7f7a67392640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7a6006e740 0x7f7a60148d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:57.835 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.829+0000 7f7a67392640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7a600cf870 0x7f7a60158f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:57.835 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.829+0000 7f7a67392640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7a6015cfa0 0x7f7a60159480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:57.835 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.829+0000 7f7a67392640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a60149280 con 0x7f7a6006e740 2023-05-22T15:44:57.835 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.829+0000 7f7a67392640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a600c6f20 con 0x7f7a6015cfa0 2023-05-22T15:44:57.835 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.829+0000 7f7a67392640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a600c7200 con 0x7f7a600cf870 2023-05-22T15:44:57.835 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.829+0000 7f7a6512e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7a6006e740 0x7f7a60148d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:44:57.836 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.829+0000 7f7a6512e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7a6006e740 0x7f7a60148d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59506/0 (socket says 172.21.15.16:59506) 2023-05-22T15:44:57.836 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.829+0000 7f7a6512e640 1 -- 172.21.15.16:0/3705287854 learned_addr learned my addr 172.21.15.16:0/3705287854 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:44:57.836 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.829+0000 7f7a6492d640 1 --2- 172.21.15.16:0/3705287854 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7a600cf870 0x7f7a60158f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:44:57.836 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.829+0000 7f7a6512e640 1 -- 172.21.15.16:0/3705287854 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7a600cf870 msgr2=0x7f7a60158f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:44:57.836 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.829+0000 7f7a6512e640 1 --2- 172.21.15.16:0/3705287854 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7a600cf870 0x7f7a60158f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:57.836 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.829+0000 7f7a6512e640 1 -- 172.21.15.16:0/3705287854 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7a6015cfa0 msgr2=0x7f7a60159480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:44:57.837 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.829+0000 7f7a6512e640 1 --2- 172.21.15.16:0/3705287854 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7a6015cfa0 0x7f7a60159480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:57.837 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.829+0000 7f7a6512e640 1 -- 172.21.15.16:0/3705287854 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a5c0094d0 con 0x7f7a6006e740 2023-05-22T15:44:57.837 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.829+0000 7f7a6492d640 1 --2- 172.21.15.16:0/3705287854 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7a600cf870 0x7f7a60158f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:44:57.837 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.833+0000 7f7a6512e640 1 --2- 172.21.15.16:0/3705287854 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7a6006e740 0x7f7a60148d40 secure :-1 s=READY pgs=1599 cs=0 l=1 rev1=1 crypto rx=0x7f7a5c0ac040 tx=0x7f7a5c01ad10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:44:57.837 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.833+0000 7f7a55ffb640 1 -- 172.21.15.16:0/3705287854 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a5c0b38c0 con 0x7f7a6006e740 2023-05-22T15:44:57.837 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.833+0000 7f7a55ffb640 1 -- 172.21.15.16:0/3705287854 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a5c0b3a40 con 0x7f7a6006e740 2023-05-22T15:44:57.837 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.833+0000 7f7a67392640 1 -- 172.21.15.16:0/3705287854 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a60071900 con 0x7f7a6006e740 2023-05-22T15:44:57.837 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.833+0000 7f7a55ffb640 1 -- 172.21.15.16:0/3705287854 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a5c0b3df0 con 0x7f7a6006e740 2023-05-22T15:44:57.838 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.833+0000 7f7a67392640 1 -- 172.21.15.16:0/3705287854 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a600cb610 con 0x7f7a6006e740 2023-05-22T15:44:57.838 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.833+0000 7f7a67392640 1 -- 172.21.15.16:0/3705287854 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a6006ef90 con 0x7f7a6006e740 2023-05-22T15:44:57.838 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.833+0000 7f7a55ffb640 1 -- 172.21.15.16:0/3705287854 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 50) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f7a5c0c6390 con 0x7f7a6006e740 2023-05-22T15:44:57.839 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.833+0000 7f7a55ffb640 1 --2- 172.21.15.16:0/3705287854 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f7a440654a0 0x7f7a44067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:57.839 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.833+0000 7f7a6492d640 1 --2- 172.21.15.16:0/3705287854 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f7a440654a0 0x7f7a44067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:44:57.839 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.833+0000 7f7a55ffb640 1 -- 172.21.15.16:0/3705287854 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(123..123 src has 1..123) v4 ==== 6304+0+0 (secure 0 0 0) 0x7f7a5c11d690 con 0x7f7a6006e740 2023-05-22T15:44:57.839 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.833+0000 7f7a6492d640 1 --2- 172.21.15.16:0/3705287854 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f7a440654a0 0x7f7a44067950 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f7a60155850 tx=0x7f7a50009040 comp rx=0 tx=0).ready entity=mgr.8304 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:44:57.841 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:57.837+0000 7f7a55ffb640 1 -- 172.21.15.16:0/3705287854 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f7a5c0eb030 con 0x7f7a6006e740 2023-05-22T15:44:58.051 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:58.049+0000 7f7a67392640 1 -- 172.21.15.16:0/3705287854 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_schedule_auto_deactivation_for_non_existent_path"]} v 0) v1 -- 0x7f7a6006ef90 con 0x7f7a6006e740 2023-05-22T15:44:58.671 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:58.669+0000 7f7a55ffb640 1 -- 172.21.15.16:0/3705287854 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_schedule_auto_deactivation_for_non_existent_path"]}]=0 v686) v1 ==== 183+0+0 (secure 0 0 0) 0x7f7a5c0ee650 con 0x7f7a6006e740 2023-05-22T15:44:58.674 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:58.669+0000 7f7a67392640 1 -- 172.21.15.16:0/3705287854 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f7a440654a0 msgr2=0x7f7a44067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:44:58.674 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:58.669+0000 7f7a67392640 1 --2- 172.21.15.16:0/3705287854 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f7a440654a0 0x7f7a44067950 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f7a60155850 tx=0x7f7a50009040 comp rx=0 tx=0).stop 2023-05-22T15:44:58.674 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:58.669+0000 7f7a67392640 1 -- 172.21.15.16:0/3705287854 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7a6006e740 msgr2=0x7f7a60148d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:44:58.674 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:58.669+0000 7f7a67392640 1 --2- 172.21.15.16:0/3705287854 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7a6006e740 0x7f7a60148d40 secure :-1 s=READY pgs=1599 cs=0 l=1 rev1=1 crypto rx=0x7f7a5c0ac040 tx=0x7f7a5c01ad10 comp rx=0 tx=0).stop 2023-05-22T15:44:58.674 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:58.669+0000 7f7a67392640 1 -- 172.21.15.16:0/3705287854 shutdown_connections 2023-05-22T15:44:58.675 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:58.669+0000 7f7a67392640 1 --2- 172.21.15.16:0/3705287854 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7a6015cfa0 0x7f7a60159480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:58.675 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:58.669+0000 7f7a67392640 1 --2- 172.21.15.16:0/3705287854 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f7a440654a0 0x7f7a44067950 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:58.676 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:58.669+0000 7f7a67392640 1 --2- 172.21.15.16:0/3705287854 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7a600cf870 0x7f7a60158f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:58.676 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:58.669+0000 7f7a67392640 1 --2- 172.21.15.16:0/3705287854 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7a6006e740 0x7f7a60148d40 unknown :-1 s=CLOSED pgs=1599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:58.676 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:58.669+0000 7f7a67392640 1 -- 172.21.15.16:0/3705287854 >> 172.21.15.16:0/3705287854 conn(0x7f7a600cc870 msgr2=0x7f7a6015abd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:44:58.676 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:58.669+0000 7f7a67392640 1 -- 172.21.15.16:0/3705287854 shutdown_connections 2023-05-22T15:44:58.676 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:58.669+0000 7f7a67392640 1 -- 172.21.15.16:0/3705287854 wait complete. 2023-05-22T15:44:58.696 INFO:tasks.cephfs_test_runner:test_schedule_auto_deactivation_for_non_existent_path (tasks.cephfs.test_snap_schedules.TestSnapSchedules) ... ok 2023-05-22T15:44:58.697 INFO:tasks.cephfs_test_runner:Starting test: test_snap_schedule (tasks.cephfs.test_snap_schedules.TestSnapSchedules) Test existence of a scheduled snapshot 2023-05-22T15:44:58.697 INFO:tasks.cephfs_test_runner:test_snap_schedule (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2023-05-22T15:44:58.697 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule' 2023-05-22T15:44:58.827 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:58.821+0000 7fafc8dfb640 1 Processor -- start 2023-05-22T15:44:58.827 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:58.825+0000 7fafc8dfb640 1 -- start start 2023-05-22T15:44:58.831 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:58.825+0000 7fafc8dfb640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fafc40ca4e0 0x7fafc415e460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:58.831 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:58.825+0000 7fafc8dfb640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fafc4070df0 0x7fafc4071210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:58.831 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:58.825+0000 7fafc8dfb640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafc40b0650 con 0x7fafc40c9b40 2023-05-22T15:44:58.832 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:58.825+0000 7fafc8dfb640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fafc40b0370 con 0x7fafc4070df0 2023-05-22T15:44:58.832 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:58.825+0000 7fafc8dfb640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafc415d980 con 0x7fafc40ca4e0 2023-05-22T15:44:58.832 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:58.825+0000 7fafc1d9b640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fafc40c9b40 0x7fafc40afb40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:52086/0 (socket says 172.21.15.16:52086) 2023-05-22T15:44:58.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:58.825+0000 7fafc1d9b640 1 -- 172.21.15.16:0/3261045960 learned_addr learned my addr 172.21.15.16:0/3261045960 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:44:58.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:58.825+0000 7fafc259c640 1 --2- 172.21.15.16:0/3261045960 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fafc40ca4e0 0x7fafc415e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:44:58.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:58.825+0000 7fafc2d9d640 1 --2- 172.21.15.16:0/3261045960 >> v2:172.21.15.62:3300/0 conn(0x7fafc4070df0 0x7fafc4071210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:44:58.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:58.825+0000 7fafc159a640 1 -- 172.21.15.16:0/3261045960 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 428936515 0 0) 0x7fafc40b0650 con 0x7fafc40c9b40 2023-05-22T15:44:58.834 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:58.825+0000 7fafc159a640 1 -- 172.21.15.16:0/3261045960 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fafac003530 con 0x7fafc40c9b40 2023-05-22T15:44:58.834 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:58.825+0000 7fafc159a640 1 -- 172.21.15.16:0/3261045960 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 580236988 0 0) 0x7fafac003530 con 0x7fafc40c9b40 2023-05-22T15:44:58.834 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:58.825+0000 7fafc159a640 1 -- 172.21.15.16:0/3261045960 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fafc40ca4e0 msgr2=0x7fafc415e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:44:58.834 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:58.825+0000 7fafc159a640 1 --2- 172.21.15.16:0/3261045960 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fafc40ca4e0 0x7fafc415e460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:58.834 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:58.825+0000 7fafc159a640 1 -- 172.21.15.16:0/3261045960 >> v2:172.21.15.62:3300/0 conn(0x7fafc4070df0 msgr2=0x7fafc4071210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:44:58.834 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:58.825+0000 7fafc159a640 1 --2- 172.21.15.16:0/3261045960 >> v2:172.21.15.62:3300/0 conn(0x7fafc4070df0 0x7fafc4071210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:58.834 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:58.825+0000 7fafc159a640 1 -- 172.21.15.16:0/3261045960 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafc415dc60 con 0x7fafc40c9b40 2023-05-22T15:44:58.835 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:58.825+0000 7fafc159a640 1 -- 172.21.15.16:0/3261045960 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7fafb8001680 con 0x7fafc40c9b40 2023-05-22T15:44:58.835 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:58.825+0000 7fafc159a640 1 -- 172.21.15.16:0/3261045960 >> v1:172.21.15.16:6789/0 conn(0x7fafc40c9b40 legacy=0x7fafc40afb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:44:58.835 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:58.825+0000 7fafc159a640 1 --2- 172.21.15.16:0/3261045960 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fafac003980 0x7fafac003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:58.835 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:58.825+0000 7fafc159a640 1 --2- 172.21.15.16:0/3261045960 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fafc40ca4e0 0x7fafac0047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:58.835 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:58.825+0000 7fafc159a640 1 --2- 172.21.15.16:0/3261045960 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fafac004d10 0x7fafac005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:58.835 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:58.825+0000 7fafc159a640 1 -- 172.21.15.16:0/3261045960 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafc40b0370 con 0x7fafac003980 2023-05-22T15:44:58.835 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:58.825+0000 7fafc159a640 1 -- 172.21.15.16:0/3261045960 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafc415d980 con 0x7fafac004d10 2023-05-22T15:44:58.835 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:58.825+0000 7fafc159a640 1 -- 172.21.15.16:0/3261045960 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafac00b990 con 0x7fafc40ca4e0 2023-05-22T15:44:58.836 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:58.825+0000 7fafc259c640 1 --2- 172.21.15.16:0/3261045960 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fafac003980 0x7fafac003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:44:58.836 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:58.825+0000 7fafc1d9b640 1 --2- 172.21.15.16:0/3261045960 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fafc40ca4e0 0x7fafac0047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:44:58.836 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:58.825+0000 7fafc2d9d640 1 --2- 172.21.15.16:0/3261045960 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fafac004d10 0x7fafac005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:44:58.836 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:58.825+0000 7fafc259c640 1 -- 172.21.15.16:0/3261045960 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fafc40ca4e0 msgr2=0x7fafac0047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:44:58.836 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:58.825+0000 7fafc259c640 1 --2- 172.21.15.16:0/3261045960 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fafc40ca4e0 0x7fafac0047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:58.836 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:58.825+0000 7fafc259c640 1 -- 172.21.15.16:0/3261045960 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fafac004d10 msgr2=0x7fafac005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:44:58.836 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:58.825+0000 7fafc259c640 1 --2- 172.21.15.16:0/3261045960 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fafac004d10 0x7fafac005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:58.837 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:58.825+0000 7fafc259c640 1 -- 172.21.15.16:0/3261045960 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafac003530 con 0x7fafac003980 2023-05-22T15:44:58.837 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:58.825+0000 7fafc2d9d640 1 --2- 172.21.15.16:0/3261045960 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fafac004d10 0x7fafac005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:44:58.837 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:58.825+0000 7fafc1d9b640 1 --2- 172.21.15.16:0/3261045960 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fafc40ca4e0 0x7fafac0047d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:44:58.837 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:58.829+0000 7fafc259c640 1 --2- 172.21.15.16:0/3261045960 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fafac003980 0x7fafac003d90 secure :-1 s=READY pgs=1601 cs=0 l=1 rev1=1 crypto rx=0x7fafb4009ae0 tx=0x7fafb401ca20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=703225cb97d7caa7 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:44:58.837 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:58.829+0000 7fafc159a640 1 -- 172.21.15.16:0/3261045960 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafb40a6510 con 0x7fafac003980 2023-05-22T15:44:58.837 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:58.829+0000 7fafc159a640 1 -- 172.21.15.16:0/3261045960 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafb40a6670 con 0x7fafac003980 2023-05-22T15:44:58.838 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:58.829+0000 7fafc159a640 1 -- 172.21.15.16:0/3261045960 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafb40a6a20 con 0x7fafac003980 2023-05-22T15:44:58.840 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:58.829+0000 7fafc8dfb640 1 -- 172.21.15.16:0/3261045960 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fafac003980 msgr2=0x7fafac003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:44:58.840 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:58.829+0000 7fafc8dfb640 1 --2- 172.21.15.16:0/3261045960 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fafac003980 0x7fafac003d90 secure :-1 s=READY pgs=1601 cs=0 l=1 rev1=1 crypto rx=0x7fafb4009ae0 tx=0x7fafb401ca20 comp rx=0 tx=0).stop 2023-05-22T15:44:58.840 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:58.829+0000 7fafc2d9d640 1 -- 172.21.15.16:0/3261045960 reap_dead start 2023-05-22T15:44:58.840 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:58.829+0000 7fafc8dfb640 1 -- 172.21.15.16:0/3261045960 shutdown_connections 2023-05-22T15:44:58.841 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:58.829+0000 7fafc8dfb640 1 -- 172.21.15.16:0/3261045960 >> 172.21.15.16:0/3261045960 conn(0x7fafc4069720 msgr2=0x7fafc4069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:44:58.841 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:58.829+0000 7fafc8dfb640 1 -- 172.21.15.16:0/3261045960 shutdown_connections 2023-05-22T15:44:58.841 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:58.829+0000 7fafc8dfb640 1 -- 172.21.15.16:0/3261045960 wait complete. 2023-05-22T15:44:58.841 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:58.829+0000 7fafc8dfb640 1 Processor -- start 2023-05-22T15:44:58.841 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:58.829+0000 7fafc8dfb640 1 -- start start 2023-05-22T15:44:58.841 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:58.829+0000 7fafc8dfb640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fafac003980 0x7fafc40c3ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:58.841 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:58.829+0000 7fafc8dfb640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fafc40c3620 0x7fafc415b6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:58.842 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:58.829+0000 7fafc8dfb640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fafc415bbe0 0x7fafc4151030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:58.842 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:58.829+0000 7fafc8dfb640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafc415d2e0 con 0x7fafac003980 2023-05-22T15:44:58.842 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:58.829+0000 7fafc8dfb640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafc40c2580 con 0x7fafc415bbe0 2023-05-22T15:44:58.842 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:58.829+0000 7fafc8dfb640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafc40c2860 con 0x7fafc40c3620 2023-05-22T15:44:58.842 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:58.829+0000 7fafc259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fafac003980 0x7fafc40c3ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:44:58.842 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:58.829+0000 7fafc259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fafac003980 0x7fafc40c3ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59526/0 (socket says 172.21.15.16:59526) 2023-05-22T15:44:58.842 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:58.829+0000 7fafc259c640 1 -- 172.21.15.16:0/4218799511 learned_addr learned my addr 172.21.15.16:0/4218799511 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:44:58.843 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:58.833+0000 7fafc2d9d640 1 --2- 172.21.15.16:0/4218799511 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fafc415bbe0 0x7fafc4151030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:44:58.843 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:58.833+0000 7fafc1d9b640 1 --2- 172.21.15.16:0/4218799511 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fafc40c3620 0x7fafc415b6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:44:58.843 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:58.833+0000 7fafc259c640 1 -- 172.21.15.16:0/4218799511 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fafc40c3620 msgr2=0x7fafc415b6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:44:58.843 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:58.833+0000 7fafc259c640 1 --2- 172.21.15.16:0/4218799511 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fafc40c3620 0x7fafc415b6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:58.843 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:58.833+0000 7fafc259c640 1 -- 172.21.15.16:0/4218799511 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fafc415bbe0 msgr2=0x7fafc4151030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:44:58.843 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:58.833+0000 7fafc259c640 1 --2- 172.21.15.16:0/4218799511 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fafc415bbe0 0x7fafc4151030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:58.843 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:58.833+0000 7fafc259c640 1 -- 172.21.15.16:0/4218799511 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafc40c8e80 con 0x7fafac003980 2023-05-22T15:44:58.843 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:58.833+0000 7fafc259c640 1 --2- 172.21.15.16:0/4218799511 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fafac003980 0x7fafc40c3ed0 secure :-1 s=READY pgs=1602 cs=0 l=1 rev1=1 crypto rx=0x7fafb401c850 tx=0x7fafac005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:44:58.844 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:58.833+0000 7fafb2ffd640 1 -- 172.21.15.16:0/4218799511 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafb400f000 con 0x7fafac003980 2023-05-22T15:44:58.844 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:58.833+0000 7fafb2ffd640 1 -- 172.21.15.16:0/4218799511 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafb40a67e0 con 0x7fafac003980 2023-05-22T15:44:58.844 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:58.833+0000 7fafc8dfb640 1 -- 172.21.15.16:0/4218799511 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafc40ca8b0 con 0x7fafac003980 2023-05-22T15:44:58.844 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:58.833+0000 7fafb2ffd640 1 -- 172.21.15.16:0/4218799511 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafb40a6b90 con 0x7fafac003980 2023-05-22T15:44:58.844 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:58.833+0000 7fafc8dfb640 1 -- 172.21.15.16:0/4218799511 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fafc40c9010 con 0x7fafac003980 2023-05-22T15:44:58.844 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:58.833+0000 7fafc8dfb640 1 -- 172.21.15.16:0/4218799511 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fafc404dbe0 con 0x7fafac003980 2023-05-22T15:44:58.845 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:58.833+0000 7fafb2ffd640 1 -- 172.21.15.16:0/4218799511 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 50) v1 ==== 84332+0+0 (secure 0 0 0) 0x7fafb40a6cf0 con 0x7fafac003980 2023-05-22T15:44:58.845 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:58.833+0000 7fafb2ffd640 1 --2- 172.21.15.16:0/4218799511 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fafa40654a0 0x7fafa4067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:58.845 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:58.833+0000 7fafb2ffd640 1 -- 172.21.15.16:0/4218799511 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(123..123 src has 1..123) v4 ==== 6304+0+0 (secure 0 0 0) 0x7fafb40aa070 con 0x7fafac003980 2023-05-22T15:44:58.845 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:58.833+0000 7fafc1d9b640 1 --2- 172.21.15.16:0/4218799511 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fafa40654a0 0x7fafa4067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:44:58.845 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:58.833+0000 7fafc1d9b640 1 --2- 172.21.15.16:0/4218799511 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fafa40654a0 0x7fafa4067950 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fafb8001830 tx=0x7fafb8009040 comp rx=0 tx=0).ready entity=mgr.8304 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:44:58.846 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:58.837+0000 7fafb2ffd640 1 -- 172.21.15.16:0/4218799511 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fafb40eb2b0 con 0x7fafac003980 2023-05-22T15:44:59.050 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:59.049+0000 7fafc8dfb640 1 -- 172.21.15.16:0/4218799511 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule"]} v 0) v1 -- 0x7fafc40b3140 con 0x7fafac003980 2023-05-22T15:44:59.673 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:59.669+0000 7fafb2ffd640 1 -- 172.21.15.16:0/4218799511 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule"]}]=0 v687) v1 ==== 151+0+0 (secure 0 0 0) 0x7fafb40ee8d0 con 0x7fafac003980 2023-05-22T15:44:59.677 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:59.673+0000 7fafc8dfb640 1 -- 172.21.15.16:0/4218799511 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fafa40654a0 msgr2=0x7fafa4067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:44:59.677 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:59.673+0000 7fafc8dfb640 1 --2- 172.21.15.16:0/4218799511 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fafa40654a0 0x7fafa4067950 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fafb8001830 tx=0x7fafb8009040 comp rx=0 tx=0).stop 2023-05-22T15:44:59.677 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:59.673+0000 7fafc8dfb640 1 -- 172.21.15.16:0/4218799511 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fafac003980 msgr2=0x7fafc40c3ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:44:59.677 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:59.673+0000 7fafc8dfb640 1 --2- 172.21.15.16:0/4218799511 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fafac003980 0x7fafc40c3ed0 secure :-1 s=READY pgs=1602 cs=0 l=1 rev1=1 crypto rx=0x7fafb401c850 tx=0x7fafac005800 comp rx=0 tx=0).stop 2023-05-22T15:44:59.678 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:59.673+0000 7fafc8dfb640 1 -- 172.21.15.16:0/4218799511 shutdown_connections 2023-05-22T15:44:59.678 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:59.673+0000 7fafc8dfb640 1 --2- 172.21.15.16:0/4218799511 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fafc415bbe0 0x7fafc4151030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:59.678 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:59.673+0000 7fafc8dfb640 1 --2- 172.21.15.16:0/4218799511 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fafa40654a0 0x7fafa4067950 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:59.678 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:59.673+0000 7fafc8dfb640 1 --2- 172.21.15.16:0/4218799511 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fafc40c3620 0x7fafc415b6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:59.679 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:59.673+0000 7fafc8dfb640 1 --2- 172.21.15.16:0/4218799511 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fafac003980 0x7fafc40c3ed0 unknown :-1 s=CLOSED pgs=1602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:59.679 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:59.673+0000 7fafc8dfb640 1 -- 172.21.15.16:0/4218799511 >> 172.21.15.16:0/4218799511 conn(0x7fafc4069720 msgr2=0x7fafc40b14c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:44:59.679 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:59.673+0000 7fafc8dfb640 1 -- 172.21.15.16:0/4218799511 shutdown_connections 2023-05-22T15:44:59.680 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:59.673+0000 7fafc8dfb640 1 -- 172.21.15.16:0/4218799511 wait complete. 2023-05-22T15:44:59.699 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-05-22T15:44:59.831 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:59.829+0000 7fca66f41640 1 Processor -- start 2023-05-22T15:44:59.831 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:59.829+0000 7fca66f41640 1 -- start start 2023-05-22T15:44:59.832 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:59.829+0000 7fca66f41640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fca600b5a50 0x7fca600d2900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:59.832 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:59.829+0000 7fca66f41640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca600cccc0 con 0x7fca6015bca0 2023-05-22T15:44:59.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:59.829+0000 7fca66f41640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca600ccfa0 con 0x7fca600d2480 2023-05-22T15:44:59.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:59.829+0000 7fca66f41640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca600cd660 con 0x7fca600b5a50 2023-05-22T15:44:59.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:59.829+0000 7fca654de640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fca600b5a50 0x7fca600d2900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:44:59.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:59.829+0000 7fca64cdd640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fca6015bca0 0x7fca600cda80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:44828/0 (socket says 172.21.15.16:44828) 2023-05-22T15:44:59.834 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:59.829+0000 7fca64cdd640 1 -- 172.21.15.16:0/1624411280 learned_addr learned my addr 172.21.15.16:0/1624411280 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:44:59.834 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:59.829+0000 7fca654de640 1 -- 172.21.15.16:0/1624411280 >> v1:172.21.15.62:6789/0 conn(0x7fca600d2480 legacy=0x7fca600b5340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:44:59.834 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:59.829+0000 7fca654de640 1 -- 172.21.15.16:0/1624411280 >> v1:172.21.15.16:6789/0 conn(0x7fca6015bca0 legacy=0x7fca600cda80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:44:59.834 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:59.829+0000 7fca5f7fe640 1 -- 172.21.15.16:0/1624411280 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 926344108 0 0) 0x7fca600cccc0 con 0x7fca6015bca0 2023-05-22T15:44:59.835 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:59.829+0000 7fca654de640 1 -- 172.21.15.16:0/1624411280 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca600cd280 con 0x7fca600b5a50 2023-05-22T15:44:59.835 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:59.833+0000 7fca654de640 1 --2- 172.21.15.16:0/1624411280 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fca600b5a50 0x7fca600d2900 secure :-1 s=READY pgs=1604 cs=0 l=1 rev1=1 crypto rx=0x7fca58009d90 tx=0x7fca5801c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13f6036628da839c server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:44:59.836 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:59.833+0000 7fca5f7fe640 1 -- 172.21.15.16:0/1624411280 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca580a5920 con 0x7fca600b5a50 2023-05-22T15:44:59.837 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:59.833+0000 7fca5f7fe640 1 -- 172.21.15.16:0/1624411280 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca580a5a80 con 0x7fca600b5a50 2023-05-22T15:44:59.837 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:59.833+0000 7fca5f7fe640 1 -- 172.21.15.16:0/1624411280 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca580a5e10 con 0x7fca600b5a50 2023-05-22T15:44:59.837 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:59.833+0000 7fca66f41640 1 -- 172.21.15.16:0/1624411280 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fca600b5a50 msgr2=0x7fca600d2900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:44:59.837 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:59.833+0000 7fca66f41640 1 --2- 172.21.15.16:0/1624411280 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fca600b5a50 0x7fca600d2900 secure :-1 s=READY pgs=1604 cs=0 l=1 rev1=1 crypto rx=0x7fca58009d90 tx=0x7fca5801c950 comp rx=0 tx=0).stop 2023-05-22T15:44:59.837 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:59.833+0000 7fca66f41640 1 -- 172.21.15.16:0/1624411280 shutdown_connections 2023-05-22T15:44:59.837 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:59.833+0000 7fca66f41640 1 --2- 172.21.15.16:0/1624411280 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fca600b5a50 0x7fca600d2900 unknown :-1 s=CLOSED pgs=1604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:59.838 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:59.833+0000 7fca66f41640 1 -- 172.21.15.16:0/1624411280 >> 172.21.15.16:0/1624411280 conn(0x7fca600655c0 msgr2=0x7fca60070e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:44:59.838 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:59.833+0000 7fca66f41640 1 -- 172.21.15.16:0/1624411280 shutdown_connections 2023-05-22T15:44:59.838 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:59.833+0000 7fca66f41640 1 -- 172.21.15.16:0/1624411280 wait complete. 2023-05-22T15:44:59.840 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:59.833+0000 7fca66f41640 1 Processor -- start 2023-05-22T15:44:59.840 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:59.833+0000 7fca66f41640 1 -- start start 2023-05-22T15:44:59.840 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:59.833+0000 7fca66f41640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fca600b5a50 0x7fca6014ee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:59.840 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:59.833+0000 7fca66f41640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fca600d2480 0x7fca600c7340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:59.841 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:59.833+0000 7fca66f41640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fca6015bca0 0x7fca600c7880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:59.841 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:59.833+0000 7fca66f41640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca600ce190 con 0x7fca600d2480 2023-05-22T15:44:59.841 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:59.833+0000 7fca66f41640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca601539c0 con 0x7fca600b5a50 2023-05-22T15:44:59.841 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:59.833+0000 7fca66f41640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca60153c60 con 0x7fca6015bca0 2023-05-22T15:44:59.841 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:59.833+0000 7fca5ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fca600d2480 0x7fca600c7340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:44:59.841 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:59.833+0000 7fca5ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fca600d2480 0x7fca600c7340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44978/0 (socket says 172.21.15.16:44978) 2023-05-22T15:44:59.842 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:59.833+0000 7fca5ffff640 1 -- 172.21.15.16:0/2866526392 learned_addr learned my addr 172.21.15.16:0/2866526392 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:44:59.842 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:59.833+0000 7fca654de640 1 --2- 172.21.15.16:0/2866526392 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fca6015bca0 0x7fca600c7880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:44:59.842 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:59.833+0000 7fca64cdd640 1 --2- 172.21.15.16:0/2866526392 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fca600b5a50 0x7fca6014ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:44:59.842 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:59.837+0000 7fca5ffff640 1 -- 172.21.15.16:0/2866526392 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fca6015bca0 msgr2=0x7fca600c7880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:44:59.842 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:59.837+0000 7fca5ffff640 1 --2- 172.21.15.16:0/2866526392 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fca6015bca0 0x7fca600c7880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:59.842 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:59.837+0000 7fca5ffff640 1 -- 172.21.15.16:0/2866526392 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fca600b5a50 msgr2=0x7fca6014ee60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:44:59.842 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:59.837+0000 7fca5ffff640 1 --2- 172.21.15.16:0/2866526392 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fca600b5a50 0x7fca6014ee60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:44:59.842 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:59.837+0000 7fca5ffff640 1 -- 172.21.15.16:0/2866526392 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca58009a70 con 0x7fca600d2480 2023-05-22T15:44:59.843 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:59.837+0000 7fca5ffff640 1 --2- 172.21.15.16:0/2866526392 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fca600d2480 0x7fca600c7340 secure :-1 s=READY pgs=1605 cs=0 l=1 rev1=1 crypto rx=0x7fca5000c500 tx=0x7fca5000c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:44:59.843 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:59.837+0000 7fca5d7fa640 1 -- 172.21.15.16:0/2866526392 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca50005220 con 0x7fca600d2480 2023-05-22T15:44:59.843 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:59.837+0000 7fca5d7fa640 1 -- 172.21.15.16:0/2866526392 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca50005380 con 0x7fca600d2480 2023-05-22T15:44:59.843 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:59.837+0000 7fca5d7fa640 1 -- 172.21.15.16:0/2866526392 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca50007270 con 0x7fca600d2480 2023-05-22T15:44:59.843 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:59.837+0000 7fca66f41640 1 -- 172.21.15.16:0/2866526392 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca601604f0 con 0x7fca600d2480 2023-05-22T15:44:59.843 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:59.837+0000 7fca66f41640 1 -- 172.21.15.16:0/2866526392 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca60160360 con 0x7fca600d2480 2023-05-22T15:44:59.843 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:59.837+0000 7fca66f41640 1 -- 172.21.15.16:0/2866526392 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca60169c60 con 0x7fca600d2480 2023-05-22T15:44:59.845 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:59.837+0000 7fca5d7fa640 1 -- 172.21.15.16:0/2866526392 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 50) v1 ==== 84332+0+0 (secure 0 0 0) 0x7fca500073d0 con 0x7fca600d2480 2023-05-22T15:44:59.846 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:59.837+0000 7fca5d7fa640 1 --2- 172.21.15.16:0/2866526392 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fca400654a0 0x7fca40067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:44:59.846 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:59.837+0000 7fca5d7fa640 1 -- 172.21.15.16:0/2866526392 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(123..123 src has 1..123) v4 ==== 6304+0+0 (secure 0 0 0) 0x7fca50082400 con 0x7fca600d2480 2023-05-22T15:44:59.846 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:59.837+0000 7fca64cdd640 1 --2- 172.21.15.16:0/2866526392 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fca400654a0 0x7fca40067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:44:59.847 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:59.837+0000 7fca64cdd640 1 --2- 172.21.15.16:0/2866526392 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fca400654a0 0x7fca40067950 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fca48000f70 tx=0x7fca48007d30 comp rx=0 tx=0).ready entity=mgr.8304 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:44:59.849 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:59.841+0000 7fca5d7fa640 1 -- 172.21.15.16:0/2866526392 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fca5004fe10 con 0x7fca600d2480 2023-05-22T15:45:00.001 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:44:59.997+0000 7fca66f41640 1 -- 172.21.15.16:0/2866526392 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fca60157350 con 0x7fca600d2480 2023-05-22T15:45:00.009 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:00.005+0000 7fca5d7fa640 1 -- 172.21.15.16:0/2866526392 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v40)=0 v40) v1 ==== 122+0+0 (secure 0 0 0) 0x7fca50053430 con 0x7fca600d2480 2023-05-22T15:45:00.012 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:00.009+0000 7fca66f41640 1 -- 172.21.15.16:0/2866526392 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fca400654a0 msgr2=0x7fca40067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:45:00.012 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:00.009+0000 7fca66f41640 1 --2- 172.21.15.16:0/2866526392 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fca400654a0 0x7fca40067950 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fca48000f70 tx=0x7fca48007d30 comp rx=0 tx=0).stop 2023-05-22T15:45:00.012 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:00.009+0000 7fca66f41640 1 -- 172.21.15.16:0/2866526392 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fca600d2480 msgr2=0x7fca600c7340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:45:00.013 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:00.009+0000 7fca66f41640 1 --2- 172.21.15.16:0/2866526392 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fca600d2480 0x7fca600c7340 secure :-1 s=READY pgs=1605 cs=0 l=1 rev1=1 crypto rx=0x7fca5000c500 tx=0x7fca5000c9c0 comp rx=0 tx=0).stop 2023-05-22T15:45:00.013 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:00.009+0000 7fca66f41640 1 -- 172.21.15.16:0/2866526392 shutdown_connections 2023-05-22T15:45:00.013 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:00.009+0000 7fca66f41640 1 --2- 172.21.15.16:0/2866526392 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fca400654a0 0x7fca40067950 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:00.013 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:00.009+0000 7fca66f41640 1 --2- 172.21.15.16:0/2866526392 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fca6015bca0 0x7fca600c7880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:00.013 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:00.009+0000 7fca66f41640 1 --2- 172.21.15.16:0/2866526392 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fca600d2480 0x7fca600c7340 unknown :-1 s=CLOSED pgs=1605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:00.014 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:00.009+0000 7fca66f41640 1 --2- 172.21.15.16:0/2866526392 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fca600b5a50 0x7fca6014ee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:00.014 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:00.009+0000 7fca66f41640 1 -- 172.21.15.16:0/2866526392 >> 172.21.15.16:0/2866526392 conn(0x7fca600655c0 msgr2=0x7fca600c8c40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:45:00.014 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:00.009+0000 7fca66f41640 1 -- 172.21.15.16:0/2866526392 shutdown_connections 2023-05-22T15:45:00.014 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:00.009+0000 7fca66f41640 1 -- 172.21.15.16:0/2866526392 wait complete. 2023-05-22T15:45:00.029 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-05-22T15:45:00.030 DEBUG:teuthology.orchestra.run.smithi016:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-05-22T15:45:00.089 DEBUG:teuthology.orchestra.run.smithi062:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-05-22T15:45:00.142 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-05-22T15:45:00.142 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-05-22T15:45:00.143 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2023-05-22T15:45:00.149 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16345192k,nr_inodes=4086298,mode=755,inode64 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3276544k,mode=755,inode64 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev,inode64 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=15732 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3276540k,nr_inodes=819135,mode=700,uid=1000,gid=1271,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3276544k,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-05-22T15:45:00.150 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2023-05-22T15:45:00.151 DEBUG:teuthology.orchestra.run.smithi062:> dd if=/proc/self/mounts of=/dev/stdout 2023-05-22T15:45:00.161 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16342116k,nr_inodes=4085529,mode=755,inode64 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275932k,mode=755,inode64 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev,inode64 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=13250 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275928k,nr_inodes=818982,mode=700,uid=1000,gid=1271,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275932k,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-05-22T15:45:00.162 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-05-22T15:45:00.162 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2023-05-22T15:45:00.195 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16345192k,nr_inodes=4086298,mode=755,inode64 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3276544k,mode=755,inode64 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev,inode64 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=15732 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3276540k,nr_inodes=819135,mode=700,uid=1000,gid=1271,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3276544k,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-05-22T15:45:00.196 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2023-05-22T15:45:00.197 DEBUG:teuthology.orchestra.run.smithi062:> dd if=/proc/self/mounts of=/dev/stdout 2023-05-22T15:45:00.208 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16342116k,nr_inodes=4085529,mode=755,inode64 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275932k,mode=755,inode64 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev,inode64 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=13250 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275928k,nr_inodes=818982,mode=700,uid=1000,gid=1271,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275932k,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-05-22T15:45:00.209 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2023-05-22T15:45:00.209 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-05-22T15:45:00.374 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:00.369+0000 7fb28b1a0640 1 Processor -- start 2023-05-22T15:45:00.374 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:00.369+0000 7fb28b1a0640 1 -- start start 2023-05-22T15:45:00.375 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:00.373+0000 7fb28b1a0640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fb284158290 0x7fb2840c8880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:00.375 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:00.373+0000 7fb28b1a0640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb2840ce6c0 0x7fb28406f630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:00.376 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:00.373+0000 7fb28b1a0640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2840c92b0 con 0x7fb284158660 2023-05-22T15:45:00.376 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:00.373+0000 7fb28b1a0640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2840cc060 con 0x7fb284158290 2023-05-22T15:45:00.376 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:00.373+0000 7fb28b1a0640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2840cc340 con 0x7fb2840ce6c0 2023-05-22T15:45:00.376 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:00.373+0000 7fb28973d640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb2840ce6c0 0x7fb28406f630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:00.377 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:00.373+0000 7fb288f3c640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fb284158290 0x7fb2840c8880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:00.377 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:00.373+0000 7fb288f3c640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fb284158290 0x7fb2840c8880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.16:56144/0 (socket says 172.21.15.16:56144) 2023-05-22T15:45:00.377 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:00.373+0000 7fb28973d640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb2840ce6c0 0x7fb28406f630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3301/0 says I am v2:172.21.15.16:57506/0 (socket says 172.21.15.16:57506) 2023-05-22T15:45:00.377 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:00.373+0000 7fb288f3c640 1 -- 172.21.15.16:0/1931256562 learned_addr learned my addr 172.21.15.16:0/1931256562 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:45:00.378 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:00.373+0000 7fb288f3c640 1 -- 172.21.15.16:0/1931256562 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb2840ce6c0 msgr2=0x7fb28406f630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:45:00.378 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:00.373+0000 7fb288f3c640 1 --2- 172.21.15.16:0/1931256562 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb2840ce6c0 0x7fb28406f630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:00.378 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:00.373+0000 7fb288f3c640 1 -- 172.21.15.16:0/1931256562 >> v1:172.21.15.62:6789/0 conn(0x7fb284158660 legacy=0x7fb2840cdfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:45:00.378 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:00.373+0000 7fb288f3c640 1 -- 172.21.15.16:0/1931256562 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2840cc620 con 0x7fb284158290 2023-05-22T15:45:00.379 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:00.373+0000 7fb28973d640 1 --2- 172.21.15.16:0/1931256562 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb2840ce6c0 0x7fb28406f630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-05-22T15:45:00.379 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:00.377+0000 7fb288f3c640 1 --2- 172.21.15.16:0/1931256562 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb284158290 0x7fb2840c8880 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fb274009870 tx=0x7fb27401c510 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c0179d211c3ea411 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:45:00.380 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:00.377+0000 7fb2837fe640 1 -- 172.21.15.16:0/1931256562 <== mon.1 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2740a95f0 con 0x7fb284158290 2023-05-22T15:45:00.380 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:00.377+0000 7fb2837fe640 1 -- 172.21.15.16:0/1931256562 <== mon.1 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2740a9750 con 0x7fb284158290 2023-05-22T15:45:00.380 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:00.377+0000 7fb2837fe640 1 -- 172.21.15.16:0/1931256562 <== mon.1 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2740a9ae0 con 0x7fb284158290 2023-05-22T15:45:00.381 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:00.377+0000 7fb28b1a0640 1 -- 172.21.15.16:0/1931256562 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb284158290 msgr2=0x7fb2840c8880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:45:00.381 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:00.377+0000 7fb28b1a0640 1 --2- 172.21.15.16:0/1931256562 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb284158290 0x7fb2840c8880 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fb274009870 tx=0x7fb27401c510 comp rx=0 tx=0).stop 2023-05-22T15:45:00.381 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:00.377+0000 7fb28b1a0640 1 -- 172.21.15.16:0/1931256562 shutdown_connections 2023-05-22T15:45:00.381 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:00.377+0000 7fb28b1a0640 1 --2- 172.21.15.16:0/1931256562 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb2840ce6c0 0x7fb28406f630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:00.381 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:00.377+0000 7fb28b1a0640 1 --2- 172.21.15.16:0/1931256562 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb284158290 0x7fb2840c8880 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:00.382 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:00.377+0000 7fb28b1a0640 1 -- 172.21.15.16:0/1931256562 >> 172.21.15.16:0/1931256562 conn(0x7fb284069720 msgr2=0x7fb284069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:45:00.382 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:00.377+0000 7fb28b1a0640 1 -- 172.21.15.16:0/1931256562 shutdown_connections 2023-05-22T15:45:00.382 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:00.377+0000 7fb28b1a0640 1 -- 172.21.15.16:0/1931256562 wait complete. 2023-05-22T15:45:00.383 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:00.377+0000 7fb28b1a0640 1 Processor -- start 2023-05-22T15:45:00.383 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:00.377+0000 7fb28b1a0640 1 -- start start 2023-05-22T15:45:00.383 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:00.381+0000 7fb28b1a0640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb2840ce6c0 0x7fb28414e270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:00.383 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:00.381+0000 7fb28b1a0640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb284158290 0x7fb284155230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:00.383 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:00.381+0000 7fb28b1a0640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb284158660 0x7fb28414f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:00.384 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:00.381+0000 7fb28b1a0640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb28414e7b0 con 0x7fb2840ce6c0 2023-05-22T15:45:00.384 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:00.381+0000 7fb28b1a0640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb284155770 con 0x7fb284158660 2023-05-22T15:45:00.384 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:00.381+0000 7fb28b1a0640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb284154470 con 0x7fb284158290 2023-05-22T15:45:00.384 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:00.381+0000 7fb288f3c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb2840ce6c0 0x7fb28414e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:00.384 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:00.381+0000 7fb288f3c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb2840ce6c0 0x7fb28414e270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44992/0 (socket says 172.21.15.16:44992) 2023-05-22T15:45:00.384 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:00.381+0000 7fb288f3c640 1 -- 172.21.15.16:0/462200634 learned_addr learned my addr 172.21.15.16:0/462200634 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:45:00.384 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:00.381+0000 7fb28973d640 1 --2- 172.21.15.16:0/462200634 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb284158660 0x7fb28414f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:00.385 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:00.381+0000 7fb283fff640 1 --2- 172.21.15.16:0/462200634 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb284158290 0x7fb284155230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:00.386 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:00.381+0000 7fb288f3c640 1 -- 172.21.15.16:0/462200634 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb284158290 msgr2=0x7fb284155230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:45:00.386 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:00.381+0000 7fb288f3c640 1 --2- 172.21.15.16:0/462200634 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb284158290 0x7fb284155230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:00.386 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:00.381+0000 7fb288f3c640 1 -- 172.21.15.16:0/462200634 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb284158660 msgr2=0x7fb28414f010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:45:00.386 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:00.381+0000 7fb288f3c640 1 --2- 172.21.15.16:0/462200634 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb284158660 0x7fb28414f010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:00.387 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:00.381+0000 7fb288f3c640 1 -- 172.21.15.16:0/462200634 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb27c009a70 con 0x7fb2840ce6c0 2023-05-22T15:45:00.387 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:00.381+0000 7fb288f3c640 1 --2- 172.21.15.16:0/462200634 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb2840ce6c0 0x7fb28414e270 secure :-1 s=READY pgs=1606 cs=0 l=1 rev1=1 crypto rx=0x7fb2740ab040 tx=0x7fb27401ad50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:45:00.387 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:00.381+0000 7fb2817fa640 1 -- 172.21.15.16:0/462200634 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2740b1800 con 0x7fb2840ce6c0 2023-05-22T15:45:00.387 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:00.381+0000 7fb2817fa640 1 -- 172.21.15.16:0/462200634 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2740b1980 con 0x7fb2840ce6c0 2023-05-22T15:45:00.387 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:00.381+0000 7fb28b1a0640 1 -- 172.21.15.16:0/462200634 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2740094d0 con 0x7fb2840ce6c0 2023-05-22T15:45:00.387 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:00.381+0000 7fb2817fa640 1 -- 172.21.15.16:0/462200634 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2740b1d30 con 0x7fb2840ce6c0 2023-05-22T15:45:00.388 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:00.381+0000 7fb28b1a0640 1 -- 172.21.15.16:0/462200634 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2840ccfb0 con 0x7fb2840ce6c0 2023-05-22T15:45:00.388 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:00.381+0000 7fb28b1a0640 1 -- 172.21.15.16:0/462200634 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2840cce20 con 0x7fb2840ce6c0 2023-05-22T15:45:00.388 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:00.381+0000 7fb2817fa640 1 -- 172.21.15.16:0/462200634 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 50) v1 ==== 84332+0+0 (secure 0 0 0) 0x7fb2740a8350 con 0x7fb2840ce6c0 2023-05-22T15:45:00.388 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:00.381+0000 7fb2817fa640 1 --2- 172.21.15.16:0/462200634 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fb2700654a0 0x7fb270067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:00.388 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:00.381+0000 7fb2817fa640 1 -- 172.21.15.16:0/462200634 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(123..123 src has 1..123) v4 ==== 6304+0+0 (secure 0 0 0) 0x7fb27411d500 con 0x7fb2840ce6c0 2023-05-22T15:45:00.388 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:00.381+0000 7fb283fff640 1 --2- 172.21.15.16:0/462200634 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fb2700654a0 0x7fb270067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:00.389 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:00.385+0000 7fb283fff640 1 --2- 172.21.15.16:0/462200634 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fb2700654a0 0x7fb270067950 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fb28414fc30 tx=0x7fb278009040 comp rx=0 tx=0).ready entity=mgr.8304 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:45:00.390 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:00.385+0000 7fb2817fa640 1 -- 172.21.15.16:0/462200634 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fb2740eae40 con 0x7fb2840ce6c0 2023-05-22T15:45:00.575 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:00.573+0000 7fb28b1a0640 1 -- 172.21.15.16:0/462200634 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb2840c8dc0 con 0x7fb2840ce6c0 2023-05-22T15:45:00.576 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:00.573+0000 7fb2817fa640 1 -- 172.21.15.16:0/462200634 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) v1 ==== 94+0+3529 (secure 0 0 0) 0x7fb2740ee460 con 0x7fb2840ce6c0 2023-05-22T15:45:00.576 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:45:00.576 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":45,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6350,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6833/1554108010","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":1554108010},{"type":"v1","addr":"172.21.15.62:6833","nonce":1554108010}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":39},{"gid":6979,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/2667219028","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":2667219028},{"type":"v1","addr":"172.21.15.62:6835","nonce":2667219028}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":39},{"gid":8013,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6836/1892737356","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1892737356},{"type":"v1","addr":"172.21.15.16:6836","nonce":1892737356}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":39},{"gid":8466,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/2971455165","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6835","nonce":2971455165},{"type":"v1","addr":"172.21.15.16:6837","nonce":2971455165}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":45}],"filesystems":[]} 2023-05-22T15:45:00.577 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 45 2023-05-22T15:45:00.578 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:00.573+0000 7fb28b1a0640 1 -- 172.21.15.16:0/462200634 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fb2700654a0 msgr2=0x7fb270067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:45:00.578 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:00.573+0000 7fb28b1a0640 1 --2- 172.21.15.16:0/462200634 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fb2700654a0 0x7fb270067950 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fb28414fc30 tx=0x7fb278009040 comp rx=0 tx=0).stop 2023-05-22T15:45:00.578 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:00.573+0000 7fb28b1a0640 1 -- 172.21.15.16:0/462200634 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb2840ce6c0 msgr2=0x7fb28414e270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:45:00.578 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:00.573+0000 7fb28b1a0640 1 --2- 172.21.15.16:0/462200634 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb2840ce6c0 0x7fb28414e270 secure :-1 s=READY pgs=1606 cs=0 l=1 rev1=1 crypto rx=0x7fb2740ab040 tx=0x7fb27401ad50 comp rx=0 tx=0).stop 2023-05-22T15:45:00.578 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:00.573+0000 7fb28b1a0640 1 -- 172.21.15.16:0/462200634 shutdown_connections 2023-05-22T15:45:00.579 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:00.573+0000 7fb28b1a0640 1 --2- 172.21.15.16:0/462200634 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb284158660 0x7fb28414f010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:00.579 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:00.573+0000 7fb28b1a0640 1 --2- 172.21.15.16:0/462200634 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fb2700654a0 0x7fb270067950 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:00.579 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:00.573+0000 7fb28b1a0640 1 --2- 172.21.15.16:0/462200634 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb284158290 0x7fb284155230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:00.579 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:00.573+0000 7fb28b1a0640 1 --2- 172.21.15.16:0/462200634 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb2840ce6c0 0x7fb28414e270 unknown :-1 s=CLOSED pgs=1606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:00.579 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:00.573+0000 7fb28b1a0640 1 -- 172.21.15.16:0/462200634 >> 172.21.15.16:0/462200634 conn(0x7fb284069720 msgr2=0x7fb284069bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:45:00.579 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:00.573+0000 7fb28b1a0640 1 -- 172.21.15.16:0/462200634 shutdown_connections 2023-05-22T15:45:00.579 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:00.573+0000 7fb28b1a0640 1 -- 172.21.15.16:0/462200634 wait complete. 2023-05-22T15:45:00.594 INFO:tasks.ceph.mds.a:Restarting daemon 2023-05-22T15:45:00.594 INFO:tasks.ceph.mds.a:Stopping old one... 2023-05-22T15:45:00.594 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-05-22T15:45:00.595 INFO:teuthology.orchestra.run:waiting for 300 2023-05-22T15:45:00.595 INFO:tasks.ceph.mds.c:Restarting daemon 2023-05-22T15:45:00.595 INFO:tasks.ceph.mds.c:Stopping old one... 2023-05-22T15:45:00.595 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-05-22T15:45:00.595 INFO:teuthology.orchestra.run:waiting for 300 2023-05-22T15:45:00.596 INFO:tasks.ceph.mds.b:Restarting daemon 2023-05-22T15:45:00.596 INFO:tasks.ceph.mds.b:Stopping old one... 2023-05-22T15:45:00.596 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-05-22T15:45:00.596 INFO:teuthology.orchestra.run:waiting for 300 2023-05-22T15:45:00.597 INFO:tasks.ceph.mds.d:Restarting daemon 2023-05-22T15:45:00.597 INFO:tasks.ceph.mds.d:Stopping old one... 2023-05-22T15:45:00.597 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-05-22T15:45:00.597 INFO:teuthology.orchestra.run:waiting for 300 2023-05-22T15:45:00.604 INFO:tasks.ceph.mds.a:Stopped 2023-05-22T15:45:00.604 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-05-22T15:45:00.607 INFO:tasks.ceph.mds.c:Stopped 2023-05-22T15:45:00.607 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-05-22T15:45:00.611 INFO:tasks.ceph.mds.b:Stopped 2023-05-22T15:45:00.611 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-05-22T15:45:00.616 INFO:tasks.ceph.mds.d:Stopped 2023-05-22T15:45:00.616 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-05-22T15:45:00.655 INFO:tasks.ceph.mds.a:Started 2023-05-22T15:45:00.656 DEBUG:teuthology.parallel:result is None 2023-05-22T15:45:00.656 INFO:tasks.ceph.mds.c:Started 2023-05-22T15:45:00.657 DEBUG:teuthology.parallel:result is None 2023-05-22T15:45:00.664 INFO:tasks.ceph.mds.b:Started 2023-05-22T15:45:00.665 DEBUG:teuthology.parallel:result is None 2023-05-22T15:45:00.666 INFO:tasks.ceph.mds.d:Started 2023-05-22T15:45:00.666 DEBUG:teuthology.parallel:result is None 2023-05-22T15:45:00.667 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2023-05-22T15:45:00.714 INFO:tasks.ceph.mds.c.smithi016.stdout:starting mds.c at 2023-05-22T15:45:00.727 INFO:tasks.ceph.mds.a.smithi016.stdout:starting mds.a at 2023-05-22T15:45:00.748 INFO:tasks.ceph.mds.b.smithi062.stdout:starting mds.b at 2023-05-22T15:45:00.749 INFO:tasks.ceph.mds.d.smithi062.stdout:starting mds.d at 2023-05-22T15:45:00.782 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:00.777+0000 7f524eb09640 1 Processor -- start 2023-05-22T15:45:00.783 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:00.777+0000 7f524eb09640 1 -- start start 2023-05-22T15:45:00.785 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:00.781+0000 7f524eb09640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f52481563c0 0x7f52480c7b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:00.785 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:00.781+0000 7f524eb09640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f52480cc6c0 0x7f52480c8080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:00.786 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:00.781+0000 7f524eb09640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f524806c220 0x7f524806c6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:00.786 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:00.781+0000 7f524eb09640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f524815c210 con 0x7f524806c220 2023-05-22T15:45:00.786 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:00.781+0000 7f524eb09640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f524815c480 con 0x7f52480cc6c0 2023-05-22T15:45:00.787 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:00.781+0000 7f524eb09640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f524815c760 con 0x7f52481563c0 2023-05-22T15:45:00.787 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:00.781+0000 7f524d0a6640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f524806c220 0x7f524806c6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:00.787 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:00.781+0000 7f524d0a6640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f524806c220 0x7f524806c6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45040/0 (socket says 172.21.15.16:45040) 2023-05-22T15:45:00.787 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:00.781+0000 7f524d0a6640 1 -- 172.21.15.16:0/3846657269 learned_addr learned my addr 172.21.15.16:0/3846657269 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:45:00.788 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:00.781+0000 7f5247fff640 1 --2- 172.21.15.16:0/3846657269 >> v2:172.21.15.62:3300/0 conn(0x7f52480cc6c0 0x7f52480c8080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:00.788 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:00.781+0000 7f524c8a5640 1 --2- 172.21.15.16:0/3846657269 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f52481563c0 0x7f52480c7b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:00.788 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:00.781+0000 7f524d0a6640 1 -- 172.21.15.16:0/3846657269 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f52481563c0 msgr2=0x7f52480c7b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:45:00.789 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:00.781+0000 7f524d0a6640 1 --2- 172.21.15.16:0/3846657269 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f52481563c0 0x7f52480c7b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:00.789 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:00.781+0000 7f524d0a6640 1 -- 172.21.15.16:0/3846657269 >> v2:172.21.15.62:3300/0 conn(0x7f52480cc6c0 msgr2=0x7f52480c8080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:45:00.789 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:00.781+0000 7f524d0a6640 1 --2- 172.21.15.16:0/3846657269 >> v2:172.21.15.62:3300/0 conn(0x7f52480cc6c0 0x7f52480c8080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:00.789 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:00.781+0000 7f524d0a6640 1 -- 172.21.15.16:0/3846657269 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f524815ca40 con 0x7f524806c220 2023-05-22T15:45:00.792 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:00.781+0000 7f524d0a6640 1 --2- 172.21.15.16:0/3846657269 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f524806c220 0x7f524806c6c0 secure :-1 s=READY pgs=1613 cs=0 l=1 rev1=1 crypto rx=0x7f5248070f00 tx=0x7f524001c9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1107d63a6ee42f88 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:45:00.792 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:00.785+0000 7f52477fe640 1 -- 172.21.15.16:0/3846657269 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52400a5840 con 0x7f524806c220 2023-05-22T15:45:00.792 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:00.785+0000 7f52477fe640 1 -- 172.21.15.16:0/3846657269 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52400a59a0 con 0x7f524806c220 2023-05-22T15:45:00.793 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:00.785+0000 7f52477fe640 1 -- 172.21.15.16:0/3846657269 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52400a5d30 con 0x7f524806c220 2023-05-22T15:45:00.793 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:00.785+0000 7f524eb09640 1 -- 172.21.15.16:0/3846657269 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f524806c220 msgr2=0x7f524806c6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:45:00.793 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:00.785+0000 7f524eb09640 1 --2- 172.21.15.16:0/3846657269 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f524806c220 0x7f524806c6c0 secure :-1 s=READY pgs=1613 cs=0 l=1 rev1=1 crypto rx=0x7f5248070f00 tx=0x7f524001c9a0 comp rx=0 tx=0).stop 2023-05-22T15:45:00.793 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:00.785+0000 7f524eb09640 1 -- 172.21.15.16:0/3846657269 shutdown_connections 2023-05-22T15:45:00.793 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:00.785+0000 7f524eb09640 1 --2- 172.21.15.16:0/3846657269 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f524806c220 0x7f524806c6c0 unknown :-1 s=CLOSED pgs=1613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:00.793 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:00.785+0000 7f524eb09640 1 --2- 172.21.15.16:0/3846657269 >> v2:172.21.15.62:3300/0 conn(0x7f52480cc6c0 0x7f52480c8080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:00.794 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:00.785+0000 7f524eb09640 1 --2- 172.21.15.16:0/3846657269 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f52481563c0 0x7f52480c7b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:00.794 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:00.785+0000 7f524eb09640 1 -- 172.21.15.16:0/3846657269 >> 172.21.15.16:0/3846657269 conn(0x7f52480b1110 msgr2=0x7f52480ce830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:45:00.794 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:00.785+0000 7f524eb09640 1 -- 172.21.15.16:0/3846657269 shutdown_connections 2023-05-22T15:45:00.794 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:00.785+0000 7f524eb09640 1 -- 172.21.15.16:0/3846657269 wait complete. 2023-05-22T15:45:00.794 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:00.785+0000 7f524eb09640 1 Processor -- start 2023-05-22T15:45:00.794 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:00.785+0000 7f524eb09640 1 -- start start 2023-05-22T15:45:00.794 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:00.789+0000 7f524eb09640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f524806c220 0x7f5248153430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:00.795 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:00.789+0000 7f524eb09640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f52480cc6c0 0x7f524814e0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:00.795 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:00.789+0000 7f524eb09640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f52480ad580 0x7f52480c7b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:00.795 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:00.789+0000 7f524eb09640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52481f8620 con 0x7f524806c220 2023-05-22T15:45:00.795 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:00.789+0000 7f524eb09640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52480c9730 con 0x7f52480ad580 2023-05-22T15:45:00.795 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:00.789+0000 7f524eb09640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52481f80c0 con 0x7f52480cc6c0 2023-05-22T15:45:00.795 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:00.789+0000 7f524c8a5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f524806c220 0x7f5248153430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:00.795 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:00.789+0000 7f524c8a5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f524806c220 0x7f5248153430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45044/0 (socket says 172.21.15.16:45044) 2023-05-22T15:45:00.796 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:00.789+0000 7f524c8a5640 1 -- 172.21.15.16:0/1323128831 learned_addr learned my addr 172.21.15.16:0/1323128831 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:45:00.796 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:00.789+0000 7f524d0a6640 1 --2- 172.21.15.16:0/1323128831 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f52480ad580 0x7f52480c7b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:00.796 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:00.789+0000 7f5247fff640 1 --2- 172.21.15.16:0/1323128831 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f52480cc6c0 0x7f524814e0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:00.796 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:00.789+0000 7f524c8a5640 1 -- 172.21.15.16:0/1323128831 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f52480cc6c0 msgr2=0x7f524814e0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:45:00.796 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:00.789+0000 7f524c8a5640 1 --2- 172.21.15.16:0/1323128831 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f52480cc6c0 0x7f524814e0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:00.796 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:00.789+0000 7f524c8a5640 1 -- 172.21.15.16:0/1323128831 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f52480ad580 msgr2=0x7f52480c7b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:45:00.797 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:00.789+0000 7f524c8a5640 1 --2- 172.21.15.16:0/1323128831 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f52480ad580 0x7f52480c7b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:00.797 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:00.789+0000 7f524c8a5640 1 -- 172.21.15.16:0/1323128831 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5240009b30 con 0x7f524806c220 2023-05-22T15:45:00.797 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:00.789+0000 7f524c8a5640 1 --2- 172.21.15.16:0/1323128831 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f524806c220 0x7f5248153430 secure :-1 s=READY pgs=1614 cs=0 l=1 rev1=1 crypto rx=0x7f523c00e8a0 tx=0x7f523c00eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:45:00.797 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:00.789+0000 7f52457fa640 1 -- 172.21.15.16:0/1323128831 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f523c009510 con 0x7f524806c220 2023-05-22T15:45:00.797 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:00.789+0000 7f52457fa640 1 -- 172.21.15.16:0/1323128831 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f523c00c940 con 0x7f524806c220 2023-05-22T15:45:00.797 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:00.789+0000 7f524eb09640 1 -- 172.21.15.16:0/1323128831 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5248070890 con 0x7f524806c220 2023-05-22T15:45:00.797 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:00.789+0000 7f52457fa640 1 -- 172.21.15.16:0/1323128831 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f523c00ccb0 con 0x7f524806c220 2023-05-22T15:45:00.798 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:00.789+0000 7f524eb09640 1 -- 172.21.15.16:0/1323128831 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5248070700 con 0x7f524806c220 2023-05-22T15:45:00.798 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:00.789+0000 7f524eb09640 1 -- 172.21.15.16:0/1323128831 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5248070a20 con 0x7f524806c220 2023-05-22T15:45:00.798 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:00.789+0000 7f52457fa640 1 -- 172.21.15.16:0/1323128831 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 50) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f523c013470 con 0x7f524806c220 2023-05-22T15:45:00.799 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:00.789+0000 7f52457fa640 1 --2- 172.21.15.16:0/1323128831 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f522c0654a0 0x7f522c067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:00.799 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:00.789+0000 7f52457fa640 1 -- 172.21.15.16:0/1323128831 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 6304+0+0 (secure 0 0 0) 0x7f523c0829e0 con 0x7f524806c220 2023-05-22T15:45:00.799 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:00.789+0000 7f5247fff640 1 --2- 172.21.15.16:0/1323128831 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f522c0654a0 0x7f522c067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:00.799 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:00.789+0000 7f5247fff640 1 --2- 172.21.15.16:0/1323128831 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f522c0654a0 0x7f522c067950 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f52481f7610 tx=0x7f5234009190 comp rx=0 tx=0).ready entity=mgr.8304 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:45:00.800 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:00.797+0000 7f52457fa640 1 -- 172.21.15.16:0/1323128831 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f523c050540 con 0x7f524806c220 2023-05-22T15:45:00.958 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:00.953+0000 7f524eb09640 1 -- 172.21.15.16:0/1323128831 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f524804dbe0 con 0x7f524806c220 2023-05-22T15:45:01.804 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:01.801+0000 7f52457fa640 1 -- 172.21.15.16:0/1323128831 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v128) v1 ==== 97+0+0 (secure 0 0 0) 0x7f523c053b60 con 0x7f524806c220 2023-05-22T15:45:01.804 INFO:teuthology.orchestra.run.smithi016.stderr: removed all blocklist entries 2023-05-22T15:45:01.806 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:01.805+0000 7f524eb09640 1 -- 172.21.15.16:0/1323128831 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f522c0654a0 msgr2=0x7f522c067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:45:01.807 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:01.805+0000 7f524eb09640 1 --2- 172.21.15.16:0/1323128831 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f522c0654a0 0x7f522c067950 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f52481f7610 tx=0x7f5234009190 comp rx=0 tx=0).stop 2023-05-22T15:45:01.807 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:01.805+0000 7f524eb09640 1 -- 172.21.15.16:0/1323128831 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f524806c220 msgr2=0x7f5248153430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:45:01.807 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:01.805+0000 7f524eb09640 1 --2- 172.21.15.16:0/1323128831 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f524806c220 0x7f5248153430 secure :-1 s=READY pgs=1614 cs=0 l=1 rev1=1 crypto rx=0x7f523c00e8a0 tx=0x7f523c00eca0 comp rx=0 tx=0).stop 2023-05-22T15:45:01.807 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:01.805+0000 7f524eb09640 1 -- 172.21.15.16:0/1323128831 shutdown_connections 2023-05-22T15:45:01.807 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:01.805+0000 7f524eb09640 1 --2- 172.21.15.16:0/1323128831 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f52480ad580 0x7f52480c7b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:01.808 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:01.805+0000 7f524eb09640 1 --2- 172.21.15.16:0/1323128831 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f522c0654a0 0x7f522c067950 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:01.808 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:01.805+0000 7f524eb09640 1 --2- 172.21.15.16:0/1323128831 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f52480cc6c0 0x7f524814e0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:01.808 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:01.805+0000 7f524eb09640 1 --2- 172.21.15.16:0/1323128831 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f524806c220 0x7f5248153430 unknown :-1 s=CLOSED pgs=1614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:01.808 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:01.805+0000 7f524eb09640 1 -- 172.21.15.16:0/1323128831 >> 172.21.15.16:0/1323128831 conn(0x7f52480b1110 msgr2=0x7f52480c8e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:45:01.808 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:01.805+0000 7f524eb09640 1 -- 172.21.15.16:0/1323128831 shutdown_connections 2023-05-22T15:45:01.808 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:01.805+0000 7f524eb09640 1 -- 172.21.15.16:0/1323128831 wait complete. 2023-05-22T15:45:01.826 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2023-05-22T15:45:01.947 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:01.945+0000 7f0cbb973640 1 Processor -- start 2023-05-22T15:45:01.947 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:01.945+0000 7f0cbb973640 1 -- start start 2023-05-22T15:45:01.948 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:01.945+0000 7f0cbb973640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f0cb40cd730 0x7f0cb40afb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:01.948 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:01.945+0000 7f0cbb973640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f0cb40b0080 0x7f0cb40acba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:01.949 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:01.945+0000 7f0cbb973640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0cb40cc300 con 0x7f0cb41531d0 2023-05-22T15:45:01.949 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:01.945+0000 7f0cbb973640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0cb415d330 con 0x7f0cb40b0080 2023-05-22T15:45:01.949 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:01.945+0000 7f0cbb973640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0cb415c4a0 con 0x7f0cb40cd730 2023-05-22T15:45:01.949 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:01.945+0000 7f0cb9f10640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f0cb40b0080 0x7f0cb40acba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:01.949 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:01.945+0000 7f0cb9f10640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f0cb40b0080 0x7f0cb40acba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45050/0 (socket says 172.21.15.16:45050) 2023-05-22T15:45:01.949 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:01.945+0000 7f0cb970f640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f0cb41531d0 0x7f0cb40cbbf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:44856/0 (socket says 172.21.15.16:44856) 2023-05-22T15:45:01.950 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:01.945+0000 7f0cb9f10640 1 -- 172.21.15.16:0/2516125956 learned_addr learned my addr 172.21.15.16:0/2516125956 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:45:01.950 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:01.945+0000 7f0cb8f0e640 1 --2- 172.21.15.16:0/2516125956 >> v2:172.21.15.62:3300/0 conn(0x7f0cb40cd730 0x7f0cb40afb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:01.950 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:01.945+0000 7f0cb9f10640 1 -- 172.21.15.16:0/2516125956 >> v2:172.21.15.62:3300/0 conn(0x7f0cb40cd730 msgr2=0x7f0cb40afb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:45:01.950 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:01.945+0000 7f0cb9f10640 1 --2- 172.21.15.16:0/2516125956 >> v2:172.21.15.62:3300/0 conn(0x7f0cb40cd730 0x7f0cb40afb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:01.950 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:01.945+0000 7f0cb9f10640 1 -- 172.21.15.16:0/2516125956 >> v1:172.21.15.16:6789/0 conn(0x7f0cb41531d0 legacy=0x7f0cb40cbbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:45:01.950 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:01.945+0000 7f0cabfff640 1 -- 172.21.15.16:0/2516125956 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2842012891 0 0) 0x7f0cb40cc300 con 0x7f0cb41531d0 2023-05-22T15:45:01.950 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:01.945+0000 7f0cb9f10640 1 -- 172.21.15.16:0/2516125956 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cb40b04c0 con 0x7f0cb40b0080 2023-05-22T15:45:01.951 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:01.949+0000 7f0cb9f10640 1 --2- 172.21.15.16:0/2516125956 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0cb40b0080 0x7f0cb40acba0 secure :-1 s=READY pgs=1616 cs=0 l=1 rev1=1 crypto rx=0x7f0cb40b0a70 tx=0x7f0cac01c4e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96dcd4259c66e9a1 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:45:01.951 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:01.949+0000 7f0cabfff640 1 -- 172.21.15.16:0/2516125956 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cac0aa440 con 0x7f0cb40b0080 2023-05-22T15:45:01.951 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:01.949+0000 7f0cabfff640 1 -- 172.21.15.16:0/2516125956 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0cac0aa5a0 con 0x7f0cb40b0080 2023-05-22T15:45:01.952 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:01.949+0000 7f0cabfff640 1 -- 172.21.15.16:0/2516125956 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cac0aa930 con 0x7f0cb40b0080 2023-05-22T15:45:01.952 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:01.949+0000 7f0cbb973640 1 -- 172.21.15.16:0/2516125956 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0cb40b0080 msgr2=0x7f0cb40acba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:45:01.953 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:01.949+0000 7f0cbb973640 1 --2- 172.21.15.16:0/2516125956 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0cb40b0080 0x7f0cb40acba0 secure :-1 s=READY pgs=1616 cs=0 l=1 rev1=1 crypto rx=0x7f0cb40b0a70 tx=0x7f0cac01c4e0 comp rx=0 tx=0).stop 2023-05-22T15:45:01.953 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:01.949+0000 7f0cbb973640 1 -- 172.21.15.16:0/2516125956 shutdown_connections 2023-05-22T15:45:01.953 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:01.949+0000 7f0cbb973640 1 --2- 172.21.15.16:0/2516125956 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0cb40b0080 0x7f0cb40acba0 unknown :-1 s=CLOSED pgs=1616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:01.953 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:01.949+0000 7f0cbb973640 1 --2- 172.21.15.16:0/2516125956 >> v2:172.21.15.62:3300/0 conn(0x7f0cb40cd730 0x7f0cb40afb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:01.953 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:01.949+0000 7f0cbb973640 1 -- 172.21.15.16:0/2516125956 >> 172.21.15.16:0/2516125956 conn(0x7f0cb4069720 msgr2=0x7f0cb4069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:45:01.953 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:01.949+0000 7f0cbb973640 1 -- 172.21.15.16:0/2516125956 shutdown_connections 2023-05-22T15:45:01.953 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:01.949+0000 7f0cbb973640 1 -- 172.21.15.16:0/2516125956 wait complete. 2023-05-22T15:45:01.955 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:01.949+0000 7f0cbb973640 1 Processor -- start 2023-05-22T15:45:01.955 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:01.949+0000 7f0cbb973640 1 -- start start 2023-05-22T15:45:01.955 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:01.949+0000 7f0cbb973640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0cb40b0080 0x7f0cb414a240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:01.955 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:01.949+0000 7f0cbb973640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0cb40cd730 0x7f0cb40ae830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:01.956 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:01.949+0000 7f0cbb973640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0cb41531d0 0x7f0cb414cf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:01.956 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:01.949+0000 7f0cbb973640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cb414ab10 con 0x7f0cb40cd730 2023-05-22T15:45:01.956 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:01.949+0000 7f0cbb973640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cb414ad60 con 0x7f0cb41531d0 2023-05-22T15:45:01.956 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:01.949+0000 7f0cbb973640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0cb414b040 con 0x7f0cb40b0080 2023-05-22T15:45:01.956 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:01.949+0000 7f0cb8f0e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0cb40cd730 0x7f0cb40ae830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:01.956 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:01.949+0000 7f0cb8f0e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0cb40cd730 0x7f0cb40ae830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45062/0 (socket says 172.21.15.16:45062) 2023-05-22T15:45:01.956 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:01.949+0000 7f0cb8f0e640 1 -- 172.21.15.16:0/3648002705 learned_addr learned my addr 172.21.15.16:0/3648002705 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:45:01.957 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:01.953+0000 7f0cb970f640 1 --2- 172.21.15.16:0/3648002705 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0cb40b0080 0x7f0cb414a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:01.957 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:01.953+0000 7f0cb9f10640 1 --2- 172.21.15.16:0/3648002705 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0cb41531d0 0x7f0cb414cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:01.957 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:01.953+0000 7f0cb8f0e640 1 -- 172.21.15.16:0/3648002705 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0cb40b0080 msgr2=0x7f0cb414a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:45:01.957 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:01.953+0000 7f0cb8f0e640 1 --2- 172.21.15.16:0/3648002705 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0cb40b0080 0x7f0cb414a240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:01.957 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:01.953+0000 7f0cb8f0e640 1 -- 172.21.15.16:0/3648002705 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0cb41531d0 msgr2=0x7f0cb414cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:45:01.957 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:01.953+0000 7f0cb8f0e640 1 --2- 172.21.15.16:0/3648002705 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0cb41531d0 0x7f0cb414cf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:01.957 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:01.953+0000 7f0cb8f0e640 1 -- 172.21.15.16:0/3648002705 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cac0094d0 con 0x7f0cb40cd730 2023-05-22T15:45:01.958 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:01.953+0000 7f0cb8f0e640 1 --2- 172.21.15.16:0/3648002705 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0cb40cd730 0x7f0cb40ae830 secure :-1 s=READY pgs=1617 cs=0 l=1 rev1=1 crypto rx=0x7f0ca400d800 tx=0x7f0ca400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:45:01.958 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:01.953+0000 7f0ca9ffb640 1 -- 172.21.15.16:0/3648002705 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ca400b6c0 con 0x7f0cb40cd730 2023-05-22T15:45:01.958 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:01.953+0000 7f0ca9ffb640 1 -- 172.21.15.16:0/3648002705 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ca400b820 con 0x7f0cb40cd730 2023-05-22T15:45:01.958 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:01.953+0000 7f0ca9ffb640 1 -- 172.21.15.16:0/3648002705 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ca400bb90 con 0x7f0cb40cd730 2023-05-22T15:45:01.958 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:01.953+0000 7f0cbb973640 1 -- 172.21.15.16:0/3648002705 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0cb415cc70 con 0x7f0cb40cd730 2023-05-22T15:45:01.958 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:01.953+0000 7f0cbb973640 1 -- 172.21.15.16:0/3648002705 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0cb415c990 con 0x7f0cb40cd730 2023-05-22T15:45:01.959 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:01.953+0000 7f0cbb973640 1 -- 172.21.15.16:0/3648002705 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0cb415cf50 con 0x7f0cb40cd730 2023-05-22T15:45:01.959 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:01.953+0000 7f0ca9ffb640 1 -- 172.21.15.16:0/3648002705 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 50) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f0ca400bcf0 con 0x7f0cb40cd730 2023-05-22T15:45:01.959 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:01.953+0000 7f0ca9ffb640 1 --2- 172.21.15.16:0/3648002705 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f0c980654a0 0x7f0c98067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:01.959 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:01.953+0000 7f0cb970f640 1 --2- 172.21.15.16:0/3648002705 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f0c980654a0 0x7f0c98067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:01.960 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:01.953+0000 7f0ca9ffb640 1 -- 172.21.15.16:0/3648002705 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 5530+0+0 (secure 0 0 0) 0x7f0ca4082dc0 con 0x7f0cb40cd730 2023-05-22T15:45:01.960 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:01.953+0000 7f0cb970f640 1 --2- 172.21.15.16:0/3648002705 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f0c980654a0 0x7f0c98067950 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f0cb0007fd0 tx=0x7f0cb0009040 comp rx=0 tx=0).ready entity=mgr.8304 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:45:01.961 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:01.957+0000 7f0ca9ffb640 1 -- 172.21.15.16:0/3648002705 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f0ca4054570 con 0x7f0cb40cd730 2023-05-22T15:45:02.166 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.161+0000 7f0cbb973640 1 -- 172.21.15.16:0/3648002705 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f0cb404dbe0 con 0x7f0cb40cd730 2023-05-22T15:45:02.166 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.161+0000 7f0ca9ffb640 1 -- 172.21.15.16:0/3648002705 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v22) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f0ca4054160 con 0x7f0cb40cd730 2023-05-22T15:45:02.167 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:45:02.167 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-05-22T15:45:02.167 INFO:teuthology.orchestra.run.smithi016.stdout: "auth_dump": [ 2023-05-22T15:45:02.167 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T15:45:02.167 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "mds.a", 2023-05-22T15:45:02.168 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQCAiWtkw/K0GRAA9hvTMKMmcbgKV68GeoC4ag==", 2023-05-22T15:45:02.168 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-05-22T15:45:02.168 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2023-05-22T15:45:02.168 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow *", 2023-05-22T15:45:02.168 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow *", 2023-05-22T15:45:02.168 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-05-22T15:45:02.168 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T15:45:02.169 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:45:02.169 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T15:45:02.169 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "mds.b", 2023-05-22T15:45:02.169 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQCAiWtkoFzfJRAAxCiB0MGM5apMVA81WDq+dA==", 2023-05-22T15:45:02.169 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-05-22T15:45:02.169 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2023-05-22T15:45:02.169 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow *", 2023-05-22T15:45:02.169 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow *", 2023-05-22T15:45:02.170 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-05-22T15:45:02.170 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T15:45:02.170 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:45:02.171 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T15:45:02.171 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "mds.c", 2023-05-22T15:45:02.171 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQCAiWtkvgOKHxAAMuASPtMqwyoqpZ5I+MRS7A==", 2023-05-22T15:45:02.171 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-05-22T15:45:02.171 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2023-05-22T15:45:02.171 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow *", 2023-05-22T15:45:02.171 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow *", 2023-05-22T15:45:02.172 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-05-22T15:45:02.172 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T15:45:02.172 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:45:02.172 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T15:45:02.173 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "mds.d", 2023-05-22T15:45:02.173 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQCAiWtk6hPQKxAAlCtfC36vja5ftz6uOqUhHg==", 2023-05-22T15:45:02.173 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-05-22T15:45:02.174 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2023-05-22T15:45:02.174 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow *", 2023-05-22T15:45:02.174 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow *", 2023-05-22T15:45:02.174 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-05-22T15:45:02.174 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T15:45:02.174 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:45:02.175 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T15:45:02.175 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.0", 2023-05-22T15:45:02.175 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQCCiWtk6ZLoCRAAKdCQitZuPnWcIpkiXhJeww==", 2023-05-22T15:45:02.175 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-05-22T15:45:02.175 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2023-05-22T15:45:02.175 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2023-05-22T15:45:02.175 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-05-22T15:45:02.176 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T15:45:02.176 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:45:02.176 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T15:45:02.176 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.1", 2023-05-22T15:45:02.176 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQCFiWtkXtVeERAAkgPeDKuFmWXrDiCyqWOYwQ==", 2023-05-22T15:45:02.177 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-05-22T15:45:02.177 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2023-05-22T15:45:02.177 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2023-05-22T15:45:02.177 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-05-22T15:45:02.177 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T15:45:02.177 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:45:02.177 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T15:45:02.178 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.2", 2023-05-22T15:45:02.178 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQCIiWtkdGaRHRAAx3POIp3mnz7p7XcCVMXm5Q==", 2023-05-22T15:45:02.178 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-05-22T15:45:02.178 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2023-05-22T15:45:02.178 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2023-05-22T15:45:02.178 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-05-22T15:45:02.178 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T15:45:02.178 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:45:02.179 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T15:45:02.179 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.3", 2023-05-22T15:45:02.179 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQCLiWtkSBydJhAAsgc19Xaf/GOyy9nm9MLdJQ==", 2023-05-22T15:45:02.179 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-05-22T15:45:02.179 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2023-05-22T15:45:02.179 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2023-05-22T15:45:02.179 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-05-22T15:45:02.179 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T15:45:02.180 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:45:02.180 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T15:45:02.180 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.4", 2023-05-22T15:45:02.180 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQCRiWtkueYvJRAARXbHlKHrTbgTqBArRJfvBA==", 2023-05-22T15:45:02.180 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-05-22T15:45:02.180 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2023-05-22T15:45:02.180 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2023-05-22T15:45:02.181 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-05-22T15:45:02.181 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T15:45:02.181 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:45:02.181 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T15:45:02.181 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.5", 2023-05-22T15:45:02.181 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQCUiWtkH3hwLRAAUekP6y+wkJVPh+OPDiZbzQ==", 2023-05-22T15:45:02.181 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-05-22T15:45:02.181 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2023-05-22T15:45:02.182 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2023-05-22T15:45:02.182 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-05-22T15:45:02.182 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T15:45:02.182 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:45:02.182 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T15:45:02.182 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.6", 2023-05-22T15:45:02.182 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQCXiWtkLsnBNhAAQ8ilX8ZU4ncJ2JJU0I+U4w==", 2023-05-22T15:45:02.182 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-05-22T15:45:02.183 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2023-05-22T15:45:02.183 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2023-05-22T15:45:02.183 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-05-22T15:45:02.183 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T15:45:02.183 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:45:02.183 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T15:45:02.183 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.7", 2023-05-22T15:45:02.183 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQCbiWtkUvXjBBAApMWw7wdfxWq0WEQDL2qprA==", 2023-05-22T15:45:02.184 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-05-22T15:45:02.184 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2023-05-22T15:45:02.184 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2023-05-22T15:45:02.184 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-05-22T15:45:02.184 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T15:45:02.184 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:45:02.184 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T15:45:02.185 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "client.0", 2023-05-22T15:45:02.185 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQCAiWtkZwuUMRAAnYqSFiaT/QRsajdjNgTetA==", 2023-05-22T15:45:02.185 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-05-22T15:45:02.185 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2023-05-22T15:45:02.185 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow r", 2023-05-22T15:45:02.185 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow rw tag cephfs data=cephfs" 2023-05-22T15:45:02.185 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T15:45:02.185 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:45:02.186 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T15:45:02.186 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "client.1", 2023-05-22T15:45:02.186 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQCAiWtkPimmORAABzNL2rANz5TgCl3p++GVng==", 2023-05-22T15:45:02.186 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-05-22T15:45:02.186 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2023-05-22T15:45:02.186 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow r", 2023-05-22T15:45:02.186 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow rw", 2023-05-22T15:45:02.187 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow rwx" 2023-05-22T15:45:02.187 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T15:45:02.187 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:45:02.187 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T15:45:02.187 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "client.2", 2023-05-22T15:45:02.187 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQCAiWtkqkPMNhAAqugrnD4Mh/4At8QAFmzqcg==", 2023-05-22T15:45:02.187 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-05-22T15:45:02.187 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2023-05-22T15:45:02.188 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow r", 2023-05-22T15:45:02.188 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow rw", 2023-05-22T15:45:02.188 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow rwx" 2023-05-22T15:45:02.188 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T15:45:02.188 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:45:02.188 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T15:45:02.188 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "client.3", 2023-05-22T15:45:02.188 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQCBiWtkpvlWAxAAPuedivXBGZGHIO6hSnz8PQ==", 2023-05-22T15:45:02.189 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-05-22T15:45:02.189 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2023-05-22T15:45:02.189 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow r", 2023-05-22T15:45:02.189 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow rw", 2023-05-22T15:45:02.189 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow rwx" 2023-05-22T15:45:02.189 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T15:45:02.189 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:45:02.189 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T15:45:02.190 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "client.admin", 2023-05-22T15:45:02.190 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQCAiWtkxG9hAhAAfnUnRrBRUFoCEtsb2riNOQ==", 2023-05-22T15:45:02.190 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-05-22T15:45:02.190 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow *", 2023-05-22T15:45:02.190 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow *", 2023-05-22T15:45:02.190 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow *", 2023-05-22T15:45:02.190 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-05-22T15:45:02.191 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T15:45:02.191 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:45:02.191 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T15:45:02.191 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "mgr.x", 2023-05-22T15:45:02.191 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQCAiWtkkaIqFhAAb/4BI6FYWOy3qOSaylXLSw==", 2023-05-22T15:45:02.191 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-05-22T15:45:02.191 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow *", 2023-05-22T15:45:02.192 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile mgr", 2023-05-22T15:45:02.192 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-05-22T15:45:02.192 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T15:45:02.192 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:45:02.192 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T15:45:02.192 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "mgr.y", 2023-05-22T15:45:02.193 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQCAiWtkhgVWExAAjDoh7cexFVfsigsYfGkmVg==", 2023-05-22T15:45:02.193 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-05-22T15:45:02.193 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow *", 2023-05-22T15:45:02.193 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile mgr", 2023-05-22T15:45:02.193 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-05-22T15:45:02.193 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T15:45:02.193 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T15:45:02.193 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-05-22T15:45:02.194 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-05-22T15:45:02.195 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.165+0000 7f0cbb973640 1 -- 172.21.15.16:0/3648002705 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f0c980654a0 msgr2=0x7f0c98067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:45:02.195 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.165+0000 7f0cbb973640 1 --2- 172.21.15.16:0/3648002705 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f0c980654a0 0x7f0c98067950 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f0cb0007fd0 tx=0x7f0cb0009040 comp rx=0 tx=0).stop 2023-05-22T15:45:02.195 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.165+0000 7f0cbb973640 1 -- 172.21.15.16:0/3648002705 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0cb40cd730 msgr2=0x7f0cb40ae830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:45:02.195 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.165+0000 7f0cbb973640 1 --2- 172.21.15.16:0/3648002705 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0cb40cd730 0x7f0cb40ae830 secure :-1 s=READY pgs=1617 cs=0 l=1 rev1=1 crypto rx=0x7f0ca400d800 tx=0x7f0ca400dcc0 comp rx=0 tx=0).stop 2023-05-22T15:45:02.196 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.165+0000 7f0cbb973640 1 -- 172.21.15.16:0/3648002705 shutdown_connections 2023-05-22T15:45:02.196 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.165+0000 7f0cbb973640 1 --2- 172.21.15.16:0/3648002705 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0cb41531d0 0x7f0cb414cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:02.196 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.165+0000 7f0cbb973640 1 --2- 172.21.15.16:0/3648002705 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0cb40cd730 0x7f0cb40ae830 unknown :-1 s=CLOSED pgs=1617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:02.196 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.165+0000 7f0cbb973640 1 --2- 172.21.15.16:0/3648002705 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f0c980654a0 0x7f0c98067950 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:02.196 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.165+0000 7f0cbb973640 1 --2- 172.21.15.16:0/3648002705 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0cb40b0080 0x7f0cb414a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:02.196 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.165+0000 7f0cbb973640 1 -- 172.21.15.16:0/3648002705 >> 172.21.15.16:0/3648002705 conn(0x7f0cb4069720 msgr2=0x7f0cb415b7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:45:02.196 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.165+0000 7f0cbb973640 1 -- 172.21.15.16:0/3648002705 shutdown_connections 2023-05-22T15:45:02.197 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.165+0000 7f0cbb973640 1 -- 172.21.15.16:0/3648002705 wait complete. 2023-05-22T15:45:02.197 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-05-22T15:45:02.306 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.301+0000 7f14c1852640 1 Processor -- start 2023-05-22T15:45:02.307 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.305+0000 7f14c1852640 1 -- start start 2023-05-22T15:45:02.308 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.305+0000 7f14c1852640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f14bc0ca4e0 0x7f14bc15e460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:02.308 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.305+0000 7f14c1852640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f14bc0c9b40 0x7f14bc0afb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:02.308 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.305+0000 7f14c1852640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f14bc070df0 0x7f14bc071210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:02.308 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.305+0000 7f14c1852640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14bc0b0650 con 0x7f14bc0ca4e0 2023-05-22T15:45:02.309 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.305+0000 7f14c1852640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14bc0b0370 con 0x7f14bc070df0 2023-05-22T15:45:02.309 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.305+0000 7f14c1852640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14bc15d980 con 0x7f14bc0c9b40 2023-05-22T15:45:02.309 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.305+0000 7f14baffd640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f14bc0ca4e0 0x7f14bc15e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:02.309 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.305+0000 7f14baffd640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f14bc0ca4e0 0x7f14bc15e460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45070/0 (socket says 172.21.15.16:45070) 2023-05-22T15:45:02.309 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.305+0000 7f14baffd640 1 -- 172.21.15.16:0/2739567807 learned_addr learned my addr 172.21.15.16:0/2739567807 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:45:02.309 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.305+0000 7f14bb7fe640 1 --2- 172.21.15.16:0/2739567807 >> v2:172.21.15.62:3300/0 conn(0x7f14bc070df0 0x7f14bc071210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:02.309 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.305+0000 7f14baffd640 1 -- 172.21.15.16:0/2739567807 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f14bc0c9b40 msgr2=0x7f14bc0afb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:45:02.309 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.305+0000 7f14ba7fc640 1 --2- 172.21.15.16:0/2739567807 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f14bc0c9b40 0x7f14bc0afb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:02.310 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.305+0000 7f14baffd640 1 --2- 172.21.15.16:0/2739567807 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f14bc0c9b40 0x7f14bc0afb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:02.310 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.305+0000 7f14baffd640 1 -- 172.21.15.16:0/2739567807 >> v2:172.21.15.62:3300/0 conn(0x7f14bc070df0 msgr2=0x7f14bc071210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:45:02.310 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.305+0000 7f14baffd640 1 --2- 172.21.15.16:0/2739567807 >> v2:172.21.15.62:3300/0 conn(0x7f14bc070df0 0x7f14bc071210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:02.310 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.305+0000 7f14baffd640 1 -- 172.21.15.16:0/2739567807 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14bc15dc60 con 0x7f14bc0ca4e0 2023-05-22T15:45:02.310 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.305+0000 7f14bb7fe640 1 --2- 172.21.15.16:0/2739567807 >> v2:172.21.15.62:3300/0 conn(0x7f14bc070df0 0x7f14bc071210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:45:02.311 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.305+0000 7f14baffd640 1 --2- 172.21.15.16:0/2739567807 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f14bc0ca4e0 0x7f14bc15e460 secure :-1 s=READY pgs=1618 cs=0 l=1 rev1=1 crypto rx=0x7f14bc0b0a40 tx=0x7f14ac01c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2d5fea6b32d9442 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:45:02.311 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.305+0000 7f14b9ffb640 1 -- 172.21.15.16:0/2739567807 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14ac0a5770 con 0x7f14bc0ca4e0 2023-05-22T15:45:02.312 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.305+0000 7f14b9ffb640 1 -- 172.21.15.16:0/2739567807 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14ac0a58d0 con 0x7f14bc0ca4e0 2023-05-22T15:45:02.312 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.305+0000 7f14b9ffb640 1 -- 172.21.15.16:0/2739567807 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14ac0a5c60 con 0x7f14bc0ca4e0 2023-05-22T15:45:02.312 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.309+0000 7f14c1852640 1 -- 172.21.15.16:0/2739567807 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f14bc0ca4e0 msgr2=0x7f14bc15e460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:45:02.312 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.309+0000 7f14c1852640 1 --2- 172.21.15.16:0/2739567807 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f14bc0ca4e0 0x7f14bc15e460 secure :-1 s=READY pgs=1618 cs=0 l=1 rev1=1 crypto rx=0x7f14bc0b0a40 tx=0x7f14ac01c910 comp rx=0 tx=0).stop 2023-05-22T15:45:02.312 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.309+0000 7f14c1852640 1 -- 172.21.15.16:0/2739567807 shutdown_connections 2023-05-22T15:45:02.312 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.309+0000 7f14c1852640 1 --2- 172.21.15.16:0/2739567807 >> v2:172.21.15.62:3300/0 conn(0x7f14bc070df0 0x7f14bc071210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:02.312 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.309+0000 7f14c1852640 1 --2- 172.21.15.16:0/2739567807 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f14bc0c9b40 0x7f14bc0afb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:02.313 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.309+0000 7f14c1852640 1 --2- 172.21.15.16:0/2739567807 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f14bc0ca4e0 0x7f14bc15e460 unknown :-1 s=CLOSED pgs=1618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:02.313 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.309+0000 7f14c1852640 1 -- 172.21.15.16:0/2739567807 >> 172.21.15.16:0/2739567807 conn(0x7f14bc069720 msgr2=0x7f14bc069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:45:02.313 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.309+0000 7f14c1852640 1 -- 172.21.15.16:0/2739567807 shutdown_connections 2023-05-22T15:45:02.313 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.309+0000 7f14c1852640 1 -- 172.21.15.16:0/2739567807 wait complete. 2023-05-22T15:45:02.314 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.309+0000 7f14c1852640 1 Processor -- start 2023-05-22T15:45:02.314 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.309+0000 7f14c1852640 1 -- start start 2023-05-22T15:45:02.314 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.309+0000 7f14c1852640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f14bc070df0 0x7f14bc1f5190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:02.315 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.309+0000 7f14c1852640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f14bc0c9b40 0x7f14bc0c2f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:02.315 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.309+0000 7f14c1852640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f14bc0ca4e0 0x7f14bc15c730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:02.315 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.309+0000 7f14c1852640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14bc153ff0 con 0x7f14bc0c9b40 2023-05-22T15:45:02.315 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.309+0000 7f14c1852640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14bc0c4600 con 0x7f14bc070df0 2023-05-22T15:45:02.315 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.309+0000 7f14c1852640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14bc0c39f0 con 0x7f14bc0ca4e0 2023-05-22T15:45:02.315 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.309+0000 7f14ba7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f14bc0c9b40 0x7f14bc0c2f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:02.315 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.309+0000 7f14ba7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f14bc0c9b40 0x7f14bc0c2f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45074/0 (socket says 172.21.15.16:45074) 2023-05-22T15:45:02.316 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.309+0000 7f14ba7fc640 1 -- 172.21.15.16:0/2959801932 learned_addr learned my addr 172.21.15.16:0/2959801932 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:45:02.316 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.309+0000 7f14baffd640 1 --2- 172.21.15.16:0/2959801932 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f14bc070df0 0x7f14bc1f5190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:02.316 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.309+0000 7f14bb7fe640 1 --2- 172.21.15.16:0/2959801932 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f14bc0ca4e0 0x7f14bc15c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:02.316 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.309+0000 7f14ba7fc640 1 -- 172.21.15.16:0/2959801932 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f14bc0ca4e0 msgr2=0x7f14bc15c730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:45:02.316 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.309+0000 7f14ba7fc640 1 --2- 172.21.15.16:0/2959801932 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f14bc0ca4e0 0x7f14bc15c730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:02.316 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.309+0000 7f14ba7fc640 1 -- 172.21.15.16:0/2959801932 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f14bc070df0 msgr2=0x7f14bc1f5190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:45:02.316 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.309+0000 7f14ba7fc640 1 --2- 172.21.15.16:0/2959801932 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f14bc070df0 0x7f14bc1f5190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:02.316 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.309+0000 7f14ba7fc640 1 -- 172.21.15.16:0/2959801932 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14ac009b30 con 0x7f14bc0c9b40 2023-05-22T15:45:02.317 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.309+0000 7f14bb7fe640 1 --2- 172.21.15.16:0/2959801932 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f14bc0ca4e0 0x7f14bc15c730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:45:02.317 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.313+0000 7f14ba7fc640 1 --2- 172.21.15.16:0/2959801932 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f14bc0c9b40 0x7f14bc0c2f30 secure :-1 s=READY pgs=1619 cs=0 l=1 rev1=1 crypto rx=0x7f14b000c7f0 tx=0x7f14b000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:45:02.318 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.313+0000 7f14a3fff640 1 -- 172.21.15.16:0/2959801932 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14b0004220 con 0x7f14bc0c9b40 2023-05-22T15:45:02.318 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.313+0000 7f14a3fff640 1 -- 172.21.15.16:0/2959801932 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14b00043a0 con 0x7f14bc0c9b40 2023-05-22T15:45:02.318 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.313+0000 7f14c1852640 1 -- 172.21.15.16:0/2959801932 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14bc0ca8b0 con 0x7f14bc0c9b40 2023-05-22T15:45:02.318 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.313+0000 7f14a3fff640 1 -- 172.21.15.16:0/2959801932 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14b0004510 con 0x7f14bc0c9b40 2023-05-22T15:45:02.318 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.313+0000 7f14c1852640 1 -- 172.21.15.16:0/2959801932 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14bc071750 con 0x7f14bc0c9b40 2023-05-22T15:45:02.318 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.313+0000 7f14c1852640 1 -- 172.21.15.16:0/2959801932 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14bc0c8e80 con 0x7f14bc0c9b40 2023-05-22T15:45:02.319 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.313+0000 7f14a3fff640 1 -- 172.21.15.16:0/2959801932 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 50) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f14b0003860 con 0x7f14bc0c9b40 2023-05-22T15:45:02.319 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.313+0000 7f14a3fff640 1 --2- 172.21.15.16:0/2959801932 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f14a80654d0 0x7f14a8067980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:02.319 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.313+0000 7f14a3fff640 1 -- 172.21.15.16:0/2959801932 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 5530+0+0 (secure 0 0 0) 0x7f14b0082690 con 0x7f14bc0c9b40 2023-05-22T15:45:02.319 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.313+0000 7f14baffd640 1 --2- 172.21.15.16:0/2959801932 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f14a80654d0 0x7f14a8067980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:02.319 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.313+0000 7f14baffd640 1 --2- 172.21.15.16:0/2959801932 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f14a80654d0 0x7f14a8067980 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f14ac0ae000 tx=0x7f14ac0a7070 comp rx=0 tx=0).ready entity=mgr.8304 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:45:02.320 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.317+0000 7f14a3fff640 1 -- 172.21.15.16:0/2959801932 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f14b0050300 con 0x7f14bc0c9b40 2023-05-22T15:45:02.468 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.465+0000 7f14c1852640 1 -- 172.21.15.16:0/2959801932 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f14bc072160 con 0x7f14bc0c9b40 2023-05-22T15:45:02.470 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.465+0000 7f14a3fff640 1 -- 172.21.15.16:0/2959801932 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v128) v1 ==== 74+0+14338 (secure 0 0 0) 0x7f14b001e090 con 0x7f14bc0c9b40 2023-05-22T15:45:02.470 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:45:02.470 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":128,"fsid":"b2719834-7847-4d98-90a6-f65d57f85c49","created":"2023-05-22T15:26:25.259314+0000","modified":"2023-05-22T15:45:01.792870+0000","last_up_change":"2023-05-22T15:26:36.448524+0000","last_in_change":"2023-05-22T15:26:27.675668+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-05-22T15:26:38.019781+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-05-22T15:26:43.570514+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9e7ff854-b25f-4cbb-a5c2-1799d9c53d2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":37021},{"type":"v1","addr":"172.21.15.16:6825","nonce":37021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":37021},{"type":"v1","addr":"172.21.15.16:6827","nonce":37021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":37021},{"type":"v1","addr":"172.21.15.16:6831","nonce":37021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":37021},{"type":"v1","addr":"172.21.15.16:6829","nonce":37021}]},"public_addr":"172.21.15.16:6825/37021","cluster_addr":"172.21.15.16:6827/37021","heartbeat_back_addr":"172.21.15.16:6831/37021","heartbeat_front_addr":"172.21.15.16:6829/37021","state":["exists","up"]},{"osd":1,"uuid":"4702fbc9-a927-4565-aca5-41dc69cc8264","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":37008},{"type":"v1","addr":"172.21.15.16:6801","nonce":37008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":37008},{"type":"v1","addr":"172.21.15.16:6803","nonce":37008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":37008},{"type":"v1","addr":"172.21.15.16:6807","nonce":37008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":37008},{"type":"v1","addr":"172.21.15.16:6805","nonce":37008}]},"public_addr":"172.21.15.16:6801/37008","cluster_addr":"172.21.15.16:6803/37008","heartbeat_back_addr":"172.21.15.16:6807/37008","heartbeat_front_addr":"172.21.15.16:6805/37008","state":["exists","up"]},{"osd":2,"uuid":"1e7bba46-4eba-47ef-bb81-5e2d371638ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":37009},{"type":"v1","addr":"172.21.15.16:6809","nonce":37009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":37009},{"type":"v1","addr":"172.21.15.16:6811","nonce":37009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":37009},{"type":"v1","addr":"172.21.15.16:6815","nonce":37009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":37009},{"type":"v1","addr":"172.21.15.16:6813","nonce":37009}]},"public_addr":"172.21.15.16:6809/37009","cluster_addr":"172.21.15.16:6811/37009","heartbeat_back_addr":"172.21.15.16:6815/37009","heartbeat_front_addr":"172.21.15.16:6813/37009","state":["exists","up"]},{"osd":3,"uuid":"383cacc0-5a88-4904-990e-498ca59ef48a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":37020},{"type":"v1","addr":"172.21.15.16:6817","nonce":37020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":37020},{"type":"v1","addr":"172.21.15.16:6819","nonce":37020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":37020},{"type":"v1","addr":"172.21.15.16:6823","nonce":37020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":37020},{"type":"v1","addr":"172.21.15.16:6821","nonce":37020}]},"public_addr":"172.21.15.16:6817/37020","cluster_addr":"172.21.15.16:6819/37020","heartbeat_back_addr":"172.21.15.16:6823/37020","heartbeat_front_addr":"172.21.15.16:6821/37020","state":["exists","up"]},{"osd":4,"uuid":"970500ef-95b7-42c1-b080-eed4fae7013d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":37353},{"type":"v1","addr":"172.21.15.62:6817","nonce":37353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6819","nonce":37353},{"type":"v1","addr":"172.21.15.62:6822","nonce":37353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37353},{"type":"v1","addr":"172.21.15.62:6831","nonce":37353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37353},{"type":"v1","addr":"172.21.15.62:6826","nonce":37353}]},"public_addr":"172.21.15.62:6817/37353","cluster_addr":"172.21.15.62:6822/37353","heartbeat_back_addr":"172.21.15.62:6831/37353","heartbeat_front_addr":"172.21.15.62:6826/37353","state":["exists","up"]},{"osd":5,"uuid":"022d67db-d4d8-445b-b1b9-8b446a00f5c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":37352},{"type":"v1","addr":"172.21.15.62:6801","nonce":37352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":37352},{"type":"v1","addr":"172.21.15.62:6803","nonce":37352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":37352},{"type":"v1","addr":"172.21.15.62:6807","nonce":37352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":37352},{"type":"v1","addr":"172.21.15.62:6805","nonce":37352}]},"public_addr":"172.21.15.62:6801/37352","cluster_addr":"172.21.15.62:6803/37352","heartbeat_back_addr":"172.21.15.62:6807/37352","heartbeat_front_addr":"172.21.15.62:6805/37352","state":["exists","up"]},{"osd":6,"uuid":"42bd372b-6688-4a1b-8407-e16e8c266e33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":37354},{"type":"v1","addr":"172.21.15.62:6820","nonce":37354}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":37354},{"type":"v1","addr":"172.21.15.62:6823","nonce":37354}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6829","nonce":37354},{"type":"v1","addr":"172.21.15.62:6830","nonce":37354}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6825","nonce":37354},{"type":"v1","addr":"172.21.15.62:6827","nonce":37354}]},"public_addr":"172.21.15.62:6820/37354","cluster_addr":"172.21.15.62:6823/37354","heartbeat_back_addr":"172.21.15.62:6830/37354","heartbeat_front_addr":"172.21.15.62:6827/37354","state":["exists","up"]},{"osd":7,"uuid":"ac70b2a3-f41b-4bf2-b227-7f2cbb55426d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37355},{"type":"v1","addr":"172.21.15.62:6809","nonce":37355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37355},{"type":"v1","addr":"172.21.15.62:6811","nonce":37355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":37355},{"type":"v1","addr":"172.21.15.62:6815","nonce":37355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":37355},{"type":"v1","addr":"172.21.15.62:6813","nonce":37355}]},"public_addr":"172.21.15.62:6809/37355","cluster_addr":"172.21.15.62:6811/37355","heartbeat_back_addr":"172.21.15.62:6815/37355","heartbeat_front_addr":"172.21.15.62:6813/37355","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.368896+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.107273+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.276888+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.287502+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.124791+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.251937+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.255679+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.232797+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-05-22T15:45:02.471 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.469+0000 7f14c1852640 1 -- 172.21.15.16:0/2959801932 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f14a80654d0 msgr2=0x7f14a8067980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:45:02.472 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.469+0000 7f14c1852640 1 --2- 172.21.15.16:0/2959801932 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f14a80654d0 0x7f14a8067980 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f14ac0ae000 tx=0x7f14ac0a7070 comp rx=0 tx=0).stop 2023-05-22T15:45:02.472 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.469+0000 7f14c1852640 1 -- 172.21.15.16:0/2959801932 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f14bc0c9b40 msgr2=0x7f14bc0c2f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:45:02.472 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.469+0000 7f14c1852640 1 --2- 172.21.15.16:0/2959801932 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f14bc0c9b40 0x7f14bc0c2f30 secure :-1 s=READY pgs=1619 cs=0 l=1 rev1=1 crypto rx=0x7f14b000c7f0 tx=0x7f14b000ccb0 comp rx=0 tx=0).stop 2023-05-22T15:45:02.472 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.469+0000 7f14c1852640 1 -- 172.21.15.16:0/2959801932 shutdown_connections 2023-05-22T15:45:02.472 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.469+0000 7f14c1852640 1 --2- 172.21.15.16:0/2959801932 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f14a80654d0 0x7f14a8067980 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:02.472 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.469+0000 7f14c1852640 1 --2- 172.21.15.16:0/2959801932 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f14bc0ca4e0 0x7f14bc15c730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:02.472 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.469+0000 7f14c1852640 1 --2- 172.21.15.16:0/2959801932 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f14bc0c9b40 0x7f14bc0c2f30 unknown :-1 s=CLOSED pgs=1619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:02.473 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.469+0000 7f14c1852640 1 --2- 172.21.15.16:0/2959801932 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f14bc070df0 0x7f14bc1f5190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:02.473 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.469+0000 7f14c1852640 1 -- 172.21.15.16:0/2959801932 >> 172.21.15.16:0/2959801932 conn(0x7f14bc069720 msgr2=0x7f14bc157890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:45:02.473 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.469+0000 7f14c1852640 1 -- 172.21.15.16:0/2959801932 shutdown_connections 2023-05-22T15:45:02.473 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.469+0000 7f14c1852640 1 -- 172.21.15.16:0/2959801932 wait complete. 2023-05-22T15:45:02.487 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-05-22T15:26:38.019781+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-05-22T15:26:43.570514+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-05-22T15:45:02.488 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-05-22T15:45:02.603 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.601+0000 7f7ced3be640 1 Processor -- start 2023-05-22T15:45:02.603 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.601+0000 7f7ced3be640 1 -- start start 2023-05-22T15:45:02.606 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.601+0000 7f7ced3be640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7ce80b2880 0x7f7ce80cbfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:02.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.601+0000 7f7ced3be640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f7ce80cc520 0x7f7ce80c3db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:02.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.601+0000 7f7ced3be640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ce80c6ba0 con 0x7f7ce80c67d0 2023-05-22T15:45:02.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.601+0000 7f7ced3be640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ce815a300 con 0x7f7ce80cc520 2023-05-22T15:45:02.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.601+0000 7f7ced3be640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ce80cfef0 con 0x7f7ce80b2880 2023-05-22T15:45:02.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.601+0000 7f7ce77fe640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f7ce80cc520 0x7f7ce80c3db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:02.609 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.601+0000 7f7cdffff640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7ce80b2880 0x7f7ce80cbfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:02.609 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.601+0000 7f7cdffff640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7ce80b2880 0x7f7ce80cbfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3301/0 says I am v2:172.21.15.16:57586/0 (socket says 172.21.15.16:57586) 2023-05-22T15:45:02.609 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.601+0000 7f7cdffff640 1 -- 172.21.15.16:0/2729035641 learned_addr learned my addr 172.21.15.16:0/2729035641 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:45:02.610 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.601+0000 7f7cdffff640 1 -- 172.21.15.16:0/2729035641 >> v2:172.21.15.62:3300/0 conn(0x7f7ce80cc520 msgr2=0x7f7ce80c3db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:45:02.610 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.601+0000 7f7cdffff640 1 --2- 172.21.15.16:0/2729035641 >> v2:172.21.15.62:3300/0 conn(0x7f7ce80cc520 0x7f7ce80c3db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:02.610 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.601+0000 7f7cdffff640 1 -- 172.21.15.16:0/2729035641 >> v1:172.21.15.62:6789/0 conn(0x7f7ce80c67d0 legacy=0x7f7ce80cf730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:45:02.610 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.601+0000 7f7cdffff640 1 -- 172.21.15.16:0/2729035641 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ce80d01d0 con 0x7f7ce80b2880 2023-05-22T15:45:02.619 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.605+0000 7f7cdffff640 1 --2- 172.21.15.16:0/2729035641 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7ce80b2880 0x7f7ce80cbfe0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f7cd0009870 tx=0x7f7cd001c310 comp rx=0 tx=0).ready entity=mon.2 client_cookie=151a38a0048e958c server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:45:02.619 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.605+0000 7f7ce67fc640 1 -- 172.21.15.16:0/2729035641 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cd00ab600 con 0x7f7ce80b2880 2023-05-22T15:45:02.620 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.605+0000 7f7ce67fc640 1 -- 172.21.15.16:0/2729035641 <== mon.2 v2:172.21.15.62:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7cd00a7050 con 0x7f7ce80b2880 2023-05-22T15:45:02.620 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.605+0000 7f7ce67fc640 1 -- 172.21.15.16:0/2729035641 <== mon.2 v2:172.21.15.62:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cd00ab980 con 0x7f7ce80b2880 2023-05-22T15:45:02.620 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.605+0000 7f7ced3be640 1 -- 172.21.15.16:0/2729035641 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7ce80b2880 msgr2=0x7f7ce80cbfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:45:02.621 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.605+0000 7f7ced3be640 1 --2- 172.21.15.16:0/2729035641 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7ce80b2880 0x7f7ce80cbfe0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f7cd0009870 tx=0x7f7cd001c310 comp rx=0 tx=0).stop 2023-05-22T15:45:02.621 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.605+0000 7f7ced3be640 1 -- 172.21.15.16:0/2729035641 shutdown_connections 2023-05-22T15:45:02.621 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.605+0000 7f7ced3be640 1 --2- 172.21.15.16:0/2729035641 >> v2:172.21.15.62:3300/0 conn(0x7f7ce80cc520 0x7f7ce80c3db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:02.621 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.605+0000 7f7ced3be640 1 --2- 172.21.15.16:0/2729035641 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7ce80b2880 0x7f7ce80cbfe0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:02.622 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.605+0000 7f7ced3be640 1 -- 172.21.15.16:0/2729035641 >> 172.21.15.16:0/2729035641 conn(0x7f7ce806f5a0 msgr2=0x7f7ce806f9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:45:02.622 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.605+0000 7f7ced3be640 1 -- 172.21.15.16:0/2729035641 shutdown_connections 2023-05-22T15:45:02.622 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.605+0000 7f7ced3be640 1 -- 172.21.15.16:0/2729035641 wait complete. 2023-05-22T15:45:02.622 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.605+0000 7f7ced3be640 1 Processor -- start 2023-05-22T15:45:02.622 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.605+0000 7f7ced3be640 1 -- start start 2023-05-22T15:45:02.623 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.605+0000 7f7ced3be640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7ce80b2880 0x7f7ce81f54b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:02.623 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.605+0000 7f7ced3be640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7ce80c67d0 0x7f7ce8157f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:02.623 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.605+0000 7f7ced3be640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ce80cc520 0x7f7ce80ce6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:02.624 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.605+0000 7f7ced3be640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ce81f5330 con 0x7f7ce80cc520 2023-05-22T15:45:02.624 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.605+0000 7f7ced3be640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ce81584d0 con 0x7f7ce80c67d0 2023-05-22T15:45:02.624 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.605+0000 7f7ced3be640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ce81f5050 con 0x7f7ce80b2880 2023-05-22T15:45:02.624 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.605+0000 7f7ce77fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ce80cc520 0x7f7ce80ce6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:02.625 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.609+0000 7f7ce77fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ce80cc520 0x7f7ce80ce6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45090/0 (socket says 172.21.15.16:45090) 2023-05-22T15:45:02.625 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.609+0000 7f7ce77fe640 1 -- 172.21.15.16:0/999555290 learned_addr learned my addr 172.21.15.16:0/999555290 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:45:02.625 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.609+0000 7f7cdffff640 1 --2- 172.21.15.16:0/999555290 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7ce80c67d0 0x7f7ce8157f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:02.626 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.609+0000 7f7ce6ffd640 1 --2- 172.21.15.16:0/999555290 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7ce80b2880 0x7f7ce81f54b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:02.626 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.609+0000 7f7ce77fe640 1 -- 172.21.15.16:0/999555290 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7ce80b2880 msgr2=0x7f7ce81f54b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:45:02.626 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.609+0000 7f7ce77fe640 1 --2- 172.21.15.16:0/999555290 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7ce80b2880 0x7f7ce81f54b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:02.627 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.609+0000 7f7ce77fe640 1 -- 172.21.15.16:0/999555290 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7ce80c67d0 msgr2=0x7f7ce8157f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:45:02.627 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.609+0000 7f7ce77fe640 1 --2- 172.21.15.16:0/999555290 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7ce80c67d0 0x7f7ce8157f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:02.628 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.609+0000 7f7ce77fe640 1 -- 172.21.15.16:0/999555290 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cd00094d0 con 0x7f7ce80cc520 2023-05-22T15:45:02.628 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.609+0000 7f7ce77fe640 1 --2- 172.21.15.16:0/999555290 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ce80cc520 0x7f7ce80ce6b0 secure :-1 s=READY pgs=1620 cs=0 l=1 rev1=1 crypto rx=0x7f7ce000efc0 tx=0x7f7ce000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:45:02.628 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.609+0000 7f7cdf7fe640 1 -- 172.21.15.16:0/999555290 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ce0014480 con 0x7f7ce80cc520 2023-05-22T15:45:02.629 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.609+0000 7f7cdf7fe640 1 -- 172.21.15.16:0/999555290 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ce000f040 con 0x7f7ce80cc520 2023-05-22T15:45:02.629 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.609+0000 7f7ced3be640 1 -- 172.21.15.16:0/999555290 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ce80d0f90 con 0x7f7ce80cc520 2023-05-22T15:45:02.629 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.609+0000 7f7cdf7fe640 1 -- 172.21.15.16:0/999555290 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ce0014820 con 0x7f7ce80cc520 2023-05-22T15:45:02.629 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.609+0000 7f7ced3be640 1 -- 172.21.15.16:0/999555290 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ce80d0a70 con 0x7f7ce80cc520 2023-05-22T15:45:02.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.609+0000 7f7ced3be640 1 -- 172.21.15.16:0/999555290 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ce80cc920 con 0x7f7ce80cc520 2023-05-22T15:45:02.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.609+0000 7f7cdf7fe640 1 -- 172.21.15.16:0/999555290 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 50) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f7ce0014980 con 0x7f7ce80cc520 2023-05-22T15:45:02.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.609+0000 7f7cdf7fe640 1 --2- 172.21.15.16:0/999555290 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f7cc80654a0 0x7f7cc8067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:02.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.609+0000 7f7ce6ffd640 1 --2- 172.21.15.16:0/999555290 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f7cc80654a0 0x7f7cc8067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:02.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.609+0000 7f7cdf7fe640 1 -- 172.21.15.16:0/999555290 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 5530+0+0 (secure 0 0 0) 0x7f7ce0083e30 con 0x7f7ce80cc520 2023-05-22T15:45:02.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.609+0000 7f7ce6ffd640 1 --2- 172.21.15.16:0/999555290 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f7cc80654a0 0x7f7cc8067950 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f7cd8000f70 tx=0x7f7cd8009040 comp rx=0 tx=0).ready entity=mgr.8304 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:45:02.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.613+0000 7f7cdf7fe640 1 -- 172.21.15.16:0/999555290 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f7ce0051a00 con 0x7f7ce80cc520 2023-05-22T15:45:02.764 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.761+0000 7f7ced3be640 1 -- 172.21.15.16:0/999555290 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f7ce804dbe0 con 0x7f7ce80cc520 2023-05-22T15:45:02.765 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.761+0000 7f7cdf7fe640 1 -- 172.21.15.16:0/999555290 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v128) v1 ==== 93+0+10 (secure 0 0 0) 0x7f7ce0055020 con 0x7f7ce80cc520 2023-05-22T15:45:02.765 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2023-05-22T15:45:02.766 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.761+0000 7f7ced3be640 1 -- 172.21.15.16:0/999555290 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f7cc80654a0 msgr2=0x7f7cc8067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:45:02.766 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.761+0000 7f7ced3be640 1 --2- 172.21.15.16:0/999555290 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f7cc80654a0 0x7f7cc8067950 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f7cd8000f70 tx=0x7f7cd8009040 comp rx=0 tx=0).stop 2023-05-22T15:45:02.767 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.761+0000 7f7ced3be640 1 -- 172.21.15.16:0/999555290 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ce80cc520 msgr2=0x7f7ce80ce6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:45:02.767 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.761+0000 7f7ced3be640 1 --2- 172.21.15.16:0/999555290 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ce80cc520 0x7f7ce80ce6b0 secure :-1 s=READY pgs=1620 cs=0 l=1 rev1=1 crypto rx=0x7f7ce000efc0 tx=0x7f7ce000c490 comp rx=0 tx=0).stop 2023-05-22T15:45:02.767 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.761+0000 7f7ced3be640 1 -- 172.21.15.16:0/999555290 shutdown_connections 2023-05-22T15:45:02.767 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.761+0000 7f7ced3be640 1 --2- 172.21.15.16:0/999555290 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ce80cc520 0x7f7ce80ce6b0 unknown :-1 s=CLOSED pgs=1620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:02.767 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.761+0000 7f7ced3be640 1 --2- 172.21.15.16:0/999555290 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7ce80c67d0 0x7f7ce8157f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:02.767 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.761+0000 7f7ced3be640 1 --2- 172.21.15.16:0/999555290 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f7cc80654a0 0x7f7cc8067950 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:02.767 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.761+0000 7f7ced3be640 1 --2- 172.21.15.16:0/999555290 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7ce80b2880 0x7f7ce81f54b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:02.768 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.761+0000 7f7ced3be640 1 -- 172.21.15.16:0/999555290 >> 172.21.15.16:0/999555290 conn(0x7f7ce806f5a0 msgr2=0x7f7ce806f970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:45:02.768 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.761+0000 7f7ced3be640 1 -- 172.21.15.16:0/999555290 shutdown_connections 2023-05-22T15:45:02.768 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.761+0000 7f7ced3be640 1 -- 172.21.15.16:0/999555290 wait complete. 2023-05-22T15:45:02.782 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-05-22T15:45:02.899 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.897+0000 7fc45847e640 1 Processor -- start 2023-05-22T15:45:02.899 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.897+0000 7fc45847e640 1 -- start start 2023-05-22T15:45:02.905 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.897+0000 7fc45847e640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc4500ce480 0x7fc450071640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:02.905 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.897+0000 7fc45847e640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fc4500ce850 0x7fc450160090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:02.905 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.897+0000 7fc45847e640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc450069ae0 con 0x7fc4501605d0 2023-05-22T15:45:02.905 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.897+0000 7fc45847e640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc45015f990 con 0x7fc4500ce850 2023-05-22T15:45:02.906 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.897+0000 7fc45847e640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc450069dc0 con 0x7fc4500ce480 2023-05-22T15:45:02.906 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.897+0000 7fc456a1b640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fc4501605d0 0x7fc45015c270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:44862/0 (socket says 172.21.15.16:44862) 2023-05-22T15:45:02.906 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.897+0000 7fc456a1b640 1 -- 172.21.15.16:0/2840683264 learned_addr learned my addr 172.21.15.16:0/2840683264 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:45:02.907 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.897+0000 7fc455a19640 1 --2- 172.21.15.16:0/2840683264 >> v2:172.21.15.62:3300/0 conn(0x7fc4500ce850 0x7fc450160090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:02.907 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.897+0000 7fc45621a640 1 --2- 172.21.15.16:0/2840683264 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc4500ce480 0x7fc450071640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:02.907 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.897+0000 7fc455218640 1 -- 172.21.15.16:0/2840683264 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1416225463 0 0) 0x7fc450069ae0 con 0x7fc4501605d0 2023-05-22T15:45:02.907 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.897+0000 7fc455218640 1 -- 172.21.15.16:0/2840683264 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc43c003530 con 0x7fc4501605d0 2023-05-22T15:45:02.908 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.897+0000 7fc455218640 1 -- 172.21.15.16:0/2840683264 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4051754596 0 0) 0x7fc43c003530 con 0x7fc4501605d0 2023-05-22T15:45:02.908 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.897+0000 7fc455218640 1 -- 172.21.15.16:0/2840683264 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc4500ce480 msgr2=0x7fc450071640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:45:02.908 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.897+0000 7fc455218640 1 --2- 172.21.15.16:0/2840683264 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc4500ce480 0x7fc450071640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:02.908 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.897+0000 7fc455218640 1 -- 172.21.15.16:0/2840683264 >> v2:172.21.15.62:3300/0 conn(0x7fc4500ce850 msgr2=0x7fc450160090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:45:02.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.897+0000 7fc455218640 1 --2- 172.21.15.16:0/2840683264 >> v2:172.21.15.62:3300/0 conn(0x7fc4500ce850 0x7fc450160090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:02.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.897+0000 7fc455218640 1 -- 172.21.15.16:0/2840683264 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc45006a0a0 con 0x7fc4501605d0 2023-05-22T15:45:02.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.897+0000 7fc455218640 1 -- 172.21.15.16:0/2840683264 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7fc448001a00 con 0x7fc4501605d0 2023-05-22T15:45:02.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.897+0000 7fc455218640 1 -- 172.21.15.16:0/2840683264 >> v1:172.21.15.16:6789/0 conn(0x7fc4501605d0 legacy=0x7fc45015c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:45:02.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.897+0000 7fc455218640 1 --2- 172.21.15.16:0/2840683264 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc4500ce480 0x7fc43c003980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:02.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.897+0000 7fc455218640 1 --2- 172.21.15.16:0/2840683264 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc43c003f00 0x7fc43c0047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:02.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.897+0000 7fc455218640 1 --2- 172.21.15.16:0/2840683264 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc43c004d10 0x7fc43c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:02.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.897+0000 7fc455218640 1 -- 172.21.15.16:0/2840683264 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc45015f990 con 0x7fc43c004d10 2023-05-22T15:45:02.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.897+0000 7fc455218640 1 -- 172.21.15.16:0/2840683264 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc450069dc0 con 0x7fc43c003f00 2023-05-22T15:45:02.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.897+0000 7fc455218640 1 -- 172.21.15.16:0/2840683264 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc43c00b990 con 0x7fc4500ce480 2023-05-22T15:45:02.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.897+0000 7fc456a1b640 1 --2- 172.21.15.16:0/2840683264 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc43c004d10 0x7fc43c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:02.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.897+0000 7fc455a19640 1 --2- 172.21.15.16:0/2840683264 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc43c003f00 0x7fc43c0047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:02.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.897+0000 7fc45621a640 1 --2- 172.21.15.16:0/2840683264 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc4500ce480 0x7fc43c003980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:02.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.897+0000 7fc456a1b640 1 -- 172.21.15.16:0/2840683264 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc4500ce480 msgr2=0x7fc43c003980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:45:02.911 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.897+0000 7fc456a1b640 1 --2- 172.21.15.16:0/2840683264 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc4500ce480 0x7fc43c003980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:02.911 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.897+0000 7fc456a1b640 1 -- 172.21.15.16:0/2840683264 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc43c003f00 msgr2=0x7fc43c0047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:45:02.911 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.897+0000 7fc456a1b640 1 --2- 172.21.15.16:0/2840683264 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc43c003f00 0x7fc43c0047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:02.911 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.897+0000 7fc456a1b640 1 -- 172.21.15.16:0/2840683264 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc43c003530 con 0x7fc43c004d10 2023-05-22T15:45:02.911 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.897+0000 7fc45621a640 1 --2- 172.21.15.16:0/2840683264 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc4500ce480 0x7fc43c003980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:45:02.911 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.901+0000 7fc456a1b640 1 --2- 172.21.15.16:0/2840683264 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc43c004d10 0x7fc43c005180 secure :-1 s=READY pgs=1622 cs=0 l=1 rev1=1 crypto rx=0x7fc44800a830 tx=0x7fc44801d340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43284a5907b25436 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:45:02.911 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.901+0000 7fc455218640 1 -- 172.21.15.16:0/2840683264 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4480ad630 con 0x7fc43c004d10 2023-05-22T15:45:02.911 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.901+0000 7fc455218640 1 -- 172.21.15.16:0/2840683264 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4480a8070 con 0x7fc43c004d10 2023-05-22T15:45:02.912 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.901+0000 7fc455218640 1 -- 172.21.15.16:0/2840683264 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4480ada30 con 0x7fc43c004d10 2023-05-22T15:45:02.912 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.901+0000 7fc45847e640 1 -- 172.21.15.16:0/2840683264 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc43c004d10 msgr2=0x7fc43c005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:45:02.912 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.901+0000 7fc45847e640 1 --2- 172.21.15.16:0/2840683264 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc43c004d10 0x7fc43c005180 secure :-1 s=READY pgs=1622 cs=0 l=1 rev1=1 crypto rx=0x7fc44800a830 tx=0x7fc44801d340 comp rx=0 tx=0).stop 2023-05-22T15:45:02.912 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.901+0000 7fc456a1b640 1 -- 172.21.15.16:0/2840683264 reap_dead start 2023-05-22T15:45:02.912 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.901+0000 7fc45847e640 1 -- 172.21.15.16:0/2840683264 shutdown_connections 2023-05-22T15:45:02.912 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.901+0000 7fc45847e640 1 -- 172.21.15.16:0/2840683264 >> 172.21.15.16:0/2840683264 conn(0x7fc4500cc870 msgr2=0x7fc4500cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:45:02.912 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.901+0000 7fc45847e640 1 -- 172.21.15.16:0/2840683264 shutdown_connections 2023-05-22T15:45:02.913 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.901+0000 7fc45847e640 1 -- 172.21.15.16:0/2840683264 wait complete. 2023-05-22T15:45:02.914 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.901+0000 7fc45847e640 1 Processor -- start 2023-05-22T15:45:02.915 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.901+0000 7fc45847e640 1 -- start start 2023-05-22T15:45:02.915 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.905+0000 7fc45847e640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc43c004d10 0x7fc4501503d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:02.915 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.905+0000 7fc45847e640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc4500b0af0 0x7fc4500b0f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:02.915 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.905+0000 7fc45847e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4500afde0 0x7fc4500b0250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:02.915 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.905+0000 7fc45847e640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4500b14a0 con 0x7fc4500afde0 2023-05-22T15:45:02.915 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.905+0000 7fc45847e640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4500b0790 con 0x7fc43c004d10 2023-05-22T15:45:02.916 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.905+0000 7fc45847e640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4500b1f90 con 0x7fc4500b0af0 2023-05-22T15:45:02.916 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.905+0000 7fc456a1b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4500afde0 0x7fc4500b0250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:02.916 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.905+0000 7fc456a1b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4500afde0 0x7fc4500b0250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45102/0 (socket says 172.21.15.16:45102) 2023-05-22T15:45:02.916 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.905+0000 7fc456a1b640 1 -- 172.21.15.16:0/3611944385 learned_addr learned my addr 172.21.15.16:0/3611944385 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:45:02.916 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.905+0000 7fc45621a640 1 --2- 172.21.15.16:0/3611944385 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc43c004d10 0x7fc4501503d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:02.916 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.905+0000 7fc455a19640 1 --2- 172.21.15.16:0/3611944385 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc4500b0af0 0x7fc4500b0f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:02.916 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.905+0000 7fc456a1b640 1 -- 172.21.15.16:0/3611944385 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc4500b0af0 msgr2=0x7fc4500b0f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:45:02.917 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.905+0000 7fc456a1b640 1 --2- 172.21.15.16:0/3611944385 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc4500b0af0 0x7fc4500b0f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:02.917 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.905+0000 7fc456a1b640 1 -- 172.21.15.16:0/3611944385 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc43c004d10 msgr2=0x7fc4501503d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:45:02.917 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.905+0000 7fc456a1b640 1 --2- 172.21.15.16:0/3611944385 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc43c004d10 0x7fc4501503d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:02.917 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.905+0000 7fc456a1b640 1 -- 172.21.15.16:0/3611944385 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4500ceca0 con 0x7fc4500afde0 2023-05-22T15:45:02.917 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.905+0000 7fc45621a640 1 --2- 172.21.15.16:0/3611944385 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc43c004d10 0x7fc4501503d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:45:02.918 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.905+0000 7fc456a1b640 1 --2- 172.21.15.16:0/3611944385 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4500afde0 0x7fc4500b0250 secure :-1 s=READY pgs=1623 cs=0 l=1 rev1=1 crypto rx=0x7fc44801d170 tx=0x7fc43c005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:45:02.918 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.905+0000 7fc446ffd640 1 -- 172.21.15.16:0/3611944385 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4480019c0 con 0x7fc4500afde0 2023-05-22T15:45:02.918 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.905+0000 7fc446ffd640 1 -- 172.21.15.16:0/3611944385 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4480011d0 con 0x7fc4500afde0 2023-05-22T15:45:02.918 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.905+0000 7fc45847e640 1 -- 172.21.15.16:0/3611944385 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc45015fb70 con 0x7fc4500afde0 2023-05-22T15:45:02.918 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.905+0000 7fc446ffd640 1 -- 172.21.15.16:0/3611944385 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4480ad630 con 0x7fc4500afde0 2023-05-22T15:45:02.918 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.905+0000 7fc45847e640 1 -- 172.21.15.16:0/3611944385 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc450069720 con 0x7fc4500afde0 2023-05-22T15:45:02.919 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.905+0000 7fc45847e640 1 -- 172.21.15.16:0/3611944385 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc45004dbe0 con 0x7fc4500afde0 2023-05-22T15:45:02.919 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.905+0000 7fc446ffd640 1 -- 172.21.15.16:0/3611944385 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 50) v1 ==== 84332+0+0 (secure 0 0 0) 0x7fc4480aa030 con 0x7fc4500afde0 2023-05-22T15:45:02.919 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.905+0000 7fc446ffd640 1 --2- 172.21.15.16:0/3611944385 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fc4340654a0 0x7fc434067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:02.919 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.905+0000 7fc446ffd640 1 -- 172.21.15.16:0/3611944385 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 5530+0+0 (secure 0 0 0) 0x7fc44811c4f0 con 0x7fc4500afde0 2023-05-22T15:45:02.919 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.905+0000 7fc45621a640 1 --2- 172.21.15.16:0/3611944385 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fc4340654a0 0x7fc434067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:02.919 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.905+0000 7fc45621a640 1 --2- 172.21.15.16:0/3611944385 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fc4340654a0 0x7fc434067950 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fc45006f1f0 tx=0x7fc44c008040 comp rx=0 tx=0).ready entity=mgr.8304 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:45:02.920 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:02.909+0000 7fc446ffd640 1 -- 172.21.15.16:0/3611944385 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fc4480ea2d0 con 0x7fc4500afde0 2023-05-22T15:45:03.064 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.061+0000 7fc45847e640 1 -- 172.21.15.16:0/3611944385 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fc450070a10 con 0x7fc4500afde0 2023-05-22T15:45:03.065 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.061+0000 7fc446ffd640 1 -- 172.21.15.16:0/3611944385 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v128) v1 ==== 92+0+10 (secure 0 0 0) 0x7fc4480ed8f0 con 0x7fc4500afde0 2023-05-22T15:45:03.065 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 8 2023-05-22T15:45:03.066 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.061+0000 7fc45847e640 1 -- 172.21.15.16:0/3611944385 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fc4340654a0 msgr2=0x7fc434067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:45:03.066 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.061+0000 7fc45847e640 1 --2- 172.21.15.16:0/3611944385 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fc4340654a0 0x7fc434067950 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fc45006f1f0 tx=0x7fc44c008040 comp rx=0 tx=0).stop 2023-05-22T15:45:03.067 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.061+0000 7fc45847e640 1 -- 172.21.15.16:0/3611944385 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4500afde0 msgr2=0x7fc4500b0250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:45:03.067 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.061+0000 7fc45847e640 1 --2- 172.21.15.16:0/3611944385 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4500afde0 0x7fc4500b0250 secure :-1 s=READY pgs=1623 cs=0 l=1 rev1=1 crypto rx=0x7fc44801d170 tx=0x7fc43c005800 comp rx=0 tx=0).stop 2023-05-22T15:45:03.067 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.065+0000 7fc45847e640 1 -- 172.21.15.16:0/3611944385 shutdown_connections 2023-05-22T15:45:03.067 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.065+0000 7fc45847e640 1 --2- 172.21.15.16:0/3611944385 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4500afde0 0x7fc4500b0250 unknown :-1 s=CLOSED pgs=1623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:03.067 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.065+0000 7fc45847e640 1 --2- 172.21.15.16:0/3611944385 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fc4340654a0 0x7fc434067950 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:03.067 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.065+0000 7fc45847e640 1 --2- 172.21.15.16:0/3611944385 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc4500b0af0 0x7fc4500b0f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:03.068 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.065+0000 7fc45847e640 1 --2- 172.21.15.16:0/3611944385 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc43c004d10 0x7fc4501503d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:03.068 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.065+0000 7fc45847e640 1 -- 172.21.15.16:0/3611944385 >> 172.21.15.16:0/3611944385 conn(0x7fc4500cc870 msgr2=0x7fc45015ed90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:45:03.068 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.065+0000 7fc45847e640 1 -- 172.21.15.16:0/3611944385 shutdown_connections 2023-05-22T15:45:03.068 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.065+0000 7fc45847e640 1 -- 172.21.15.16:0/3611944385 wait complete. 2023-05-22T15:45:03.083 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-05-22T15:45:03.206 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.201+0000 7fd28e994640 1 Processor -- start 2023-05-22T15:45:03.207 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.201+0000 7fd28e994640 1 -- start start 2023-05-22T15:45:03.210 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.205+0000 7fd28e994640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fd2880ca4e0 0x7fd28815e460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:03.210 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.205+0000 7fd28e994640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd288070df0 0x7fd288071210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:03.210 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.205+0000 7fd28e994640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2880b0650 con 0x7fd2880c9b40 2023-05-22T15:45:03.210 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.205+0000 7fd28e994640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2880b0370 con 0x7fd2880ca4e0 2023-05-22T15:45:03.210 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.205+0000 7fd28e994640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd28815d980 con 0x7fd288070df0 2023-05-22T15:45:03.211 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.205+0000 7fd2877fe640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fd2880c9b40 0x7fd2880afb40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:44874/0 (socket says 172.21.15.16:44874) 2023-05-22T15:45:03.211 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.205+0000 7fd2877fe640 1 -- 172.21.15.16:0/2450461966 learned_addr learned my addr 172.21.15.16:0/2450461966 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:45:03.211 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.205+0000 7fd28cf31640 1 --2- 172.21.15.16:0/2450461966 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd288070df0 0x7fd288071210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:03.211 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.205+0000 7fd287fff640 1 --2- 172.21.15.16:0/2450461966 >> v2:172.21.15.62:3300/0 conn(0x7fd2880ca4e0 0x7fd28815e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:03.211 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.205+0000 7fd286ffd640 1 -- 172.21.15.16:0/2450461966 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 9203777 0 0) 0x7fd2880b0650 con 0x7fd2880c9b40 2023-05-22T15:45:03.211 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.205+0000 7fd286ffd640 1 -- 172.21.15.16:0/2450461966 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd270003530 con 0x7fd2880c9b40 2023-05-22T15:45:03.211 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.205+0000 7fd286ffd640 1 -- 172.21.15.16:0/2450461966 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1703058815 0 0) 0x7fd270003530 con 0x7fd2880c9b40 2023-05-22T15:45:03.212 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.205+0000 7fd286ffd640 1 -- 172.21.15.16:0/2450461966 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd288070df0 msgr2=0x7fd288071210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:45:03.212 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.205+0000 7fd286ffd640 1 --2- 172.21.15.16:0/2450461966 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd288070df0 0x7fd288071210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:03.212 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.205+0000 7fd286ffd640 1 -- 172.21.15.16:0/2450461966 >> v2:172.21.15.62:3300/0 conn(0x7fd2880ca4e0 msgr2=0x7fd28815e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:45:03.212 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.205+0000 7fd286ffd640 1 --2- 172.21.15.16:0/2450461966 >> v2:172.21.15.62:3300/0 conn(0x7fd2880ca4e0 0x7fd28815e460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:03.212 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.205+0000 7fd286ffd640 1 -- 172.21.15.16:0/2450461966 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd28815dc60 con 0x7fd2880c9b40 2023-05-22T15:45:03.212 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.205+0000 7fd286ffd640 1 -- 172.21.15.16:0/2450461966 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7fd27c0016a0 con 0x7fd2880c9b40 2023-05-22T15:45:03.212 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.205+0000 7fd286ffd640 1 -- 172.21.15.16:0/2450461966 >> v1:172.21.15.16:6789/0 conn(0x7fd2880c9b40 legacy=0x7fd2880afb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:45:03.213 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.205+0000 7fd286ffd640 1 --2- 172.21.15.16:0/2450461966 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd270003980 0x7fd270003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:03.213 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.205+0000 7fd286ffd640 1 --2- 172.21.15.16:0/2450461966 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd2700043b0 0x7fd270004870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:03.213 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.205+0000 7fd286ffd640 1 --2- 172.21.15.16:0/2450461966 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd288070df0 0x7fd270004f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:03.213 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.205+0000 7fd286ffd640 1 -- 172.21.15.16:0/2450461966 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2880b0370 con 0x7fd2700043b0 2023-05-22T15:45:03.213 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.205+0000 7fd286ffd640 1 -- 172.21.15.16:0/2450461966 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd28815d980 con 0x7fd270003980 2023-05-22T15:45:03.213 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.205+0000 7fd286ffd640 1 -- 172.21.15.16:0/2450461966 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd27000b990 con 0x7fd288070df0 2023-05-22T15:45:03.213 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.205+0000 7fd2877fe640 1 --2- 172.21.15.16:0/2450461966 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd2700043b0 0x7fd270004870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:03.213 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.205+0000 7fd28cf31640 1 --2- 172.21.15.16:0/2450461966 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd288070df0 0x7fd270004f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:03.214 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.205+0000 7fd2877fe640 1 -- 172.21.15.16:0/2450461966 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd288070df0 msgr2=0x7fd270004f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:45:03.214 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.205+0000 7fd2877fe640 1 --2- 172.21.15.16:0/2450461966 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd288070df0 0x7fd270004f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:03.214 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.205+0000 7fd2877fe640 1 -- 172.21.15.16:0/2450461966 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd270003980 msgr2=0x7fd270003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:45:03.214 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.205+0000 7fd287fff640 1 --2- 172.21.15.16:0/2450461966 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd270003980 0x7fd270003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:03.214 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.205+0000 7fd2877fe640 1 --2- 172.21.15.16:0/2450461966 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd270003980 0x7fd270003d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:03.214 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.205+0000 7fd2877fe640 1 -- 172.21.15.16:0/2450461966 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd270003530 con 0x7fd2700043b0 2023-05-22T15:45:03.214 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.205+0000 7fd28cf31640 1 --2- 172.21.15.16:0/2450461966 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd288070df0 0x7fd270004f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:45:03.217 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.205+0000 7fd2877fe640 1 --2- 172.21.15.16:0/2450461966 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd2700043b0 0x7fd270004870 secure :-1 s=READY pgs=1625 cs=0 l=1 rev1=1 crypto rx=0x7fd27c00c210 tx=0x7fd27c01f390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1406ee5d562d58e server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:45:03.217 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.209+0000 7fd286ffd640 1 -- 172.21.15.16:0/2450461966 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd27c005ae0 con 0x7fd2700043b0 2023-05-22T15:45:03.217 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.209+0000 7fd286ffd640 1 -- 172.21.15.16:0/2450461966 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd27c0ab070 con 0x7fd2700043b0 2023-05-22T15:45:03.218 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.209+0000 7fd286ffd640 1 -- 172.21.15.16:0/2450461966 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd27c005ea0 con 0x7fd2700043b0 2023-05-22T15:45:03.218 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.209+0000 7fd28e994640 1 -- 172.21.15.16:0/2450461966 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd2700043b0 msgr2=0x7fd270004870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:45:03.218 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.209+0000 7fd28e994640 1 --2- 172.21.15.16:0/2450461966 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd2700043b0 0x7fd270004870 secure :-1 s=READY pgs=1625 cs=0 l=1 rev1=1 crypto rx=0x7fd27c00c210 tx=0x7fd27c01f390 comp rx=0 tx=0).stop 2023-05-22T15:45:03.218 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.209+0000 7fd28cf31640 1 -- 172.21.15.16:0/2450461966 reap_dead start 2023-05-22T15:45:03.218 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.209+0000 7fd28e994640 1 -- 172.21.15.16:0/2450461966 shutdown_connections 2023-05-22T15:45:03.218 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.209+0000 7fd28e994640 1 -- 172.21.15.16:0/2450461966 >> 172.21.15.16:0/2450461966 conn(0x7fd288069720 msgr2=0x7fd288069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:45:03.218 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.209+0000 7fd28e994640 1 -- 172.21.15.16:0/2450461966 shutdown_connections 2023-05-22T15:45:03.218 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.209+0000 7fd28e994640 1 -- 172.21.15.16:0/2450461966 wait complete. 2023-05-22T15:45:03.219 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.209+0000 7fd28e994640 1 Processor -- start 2023-05-22T15:45:03.219 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.209+0000 7fd28e994640 1 -- start start 2023-05-22T15:45:03.220 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.209+0000 7fd28e994640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd2700043b0 0x7fd28814e380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:03.220 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.209+0000 7fd28e994640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd2880c3f50 0x7fd28814f450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:03.220 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.209+0000 7fd28e994640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd28814f990 0x7fd2880c1720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:03.220 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.209+0000 7fd28e994640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd288150ed0 con 0x7fd28814f990 2023-05-22T15:45:03.220 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.209+0000 7fd28e994640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd28814fe30 con 0x7fd2700043b0 2023-05-22T15:45:03.220 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.209+0000 7fd28e994640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd288150110 con 0x7fd2880c3f50 2023-05-22T15:45:03.220 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.209+0000 7fd28cf31640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd28814f990 0x7fd2880c1720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:03.221 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.209+0000 7fd28cf31640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd28814f990 0x7fd2880c1720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45126/0 (socket says 172.21.15.16:45126) 2023-05-22T15:45:03.221 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.209+0000 7fd28cf31640 1 -- 172.21.15.16:0/3892255332 learned_addr learned my addr 172.21.15.16:0/3892255332 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:45:03.221 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.209+0000 7fd287fff640 1 --2- 172.21.15.16:0/3892255332 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd2700043b0 0x7fd28814e380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:03.221 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.209+0000 7fd2877fe640 1 --2- 172.21.15.16:0/3892255332 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd2880c3f50 0x7fd28814f450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:03.221 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.209+0000 7fd28cf31640 1 -- 172.21.15.16:0/3892255332 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd2880c3f50 msgr2=0x7fd28814f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:45:03.221 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.209+0000 7fd28cf31640 1 --2- 172.21.15.16:0/3892255332 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd2880c3f50 0x7fd28814f450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:03.221 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.209+0000 7fd28cf31640 1 -- 172.21.15.16:0/3892255332 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd2700043b0 msgr2=0x7fd28814e380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:45:03.222 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.209+0000 7fd28cf31640 1 --2- 172.21.15.16:0/3892255332 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd2700043b0 0x7fd28814e380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:03.222 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.209+0000 7fd28cf31640 1 -- 172.21.15.16:0/3892255332 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2880ca8b0 con 0x7fd28814f990 2023-05-22T15:45:03.222 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.209+0000 7fd287fff640 1 --2- 172.21.15.16:0/3892255332 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd2700043b0 0x7fd28814e380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:45:03.222 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.209+0000 7fd28cf31640 1 --2- 172.21.15.16:0/3892255332 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd28814f990 0x7fd2880c1720 secure :-1 s=READY pgs=1626 cs=0 l=1 rev1=1 crypto rx=0x7fd2880cdef0 tx=0x7fd28000f0d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:45:03.222 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.209+0000 7fd284ff9640 1 -- 172.21.15.16:0/3892255332 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd28000f810 con 0x7fd28814f990 2023-05-22T15:45:03.222 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.209+0000 7fd284ff9640 1 -- 172.21.15.16:0/3892255332 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd280011040 con 0x7fd28814f990 2023-05-22T15:45:03.222 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.209+0000 7fd28e994640 1 -- 172.21.15.16:0/3892255332 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd28815df40 con 0x7fd28814f990 2023-05-22T15:45:03.222 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.209+0000 7fd284ff9640 1 -- 172.21.15.16:0/3892255332 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd28000fb90 con 0x7fd28814f990 2023-05-22T15:45:03.223 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.209+0000 7fd28e994640 1 -- 172.21.15.16:0/3892255332 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd288071210 con 0x7fd28814f990 2023-05-22T15:45:03.223 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.213+0000 7fd28e994640 1 -- 172.21.15.16:0/3892255332 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd28804dbe0 con 0x7fd28814f990 2023-05-22T15:45:03.223 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.213+0000 7fd284ff9640 1 -- 172.21.15.16:0/3892255332 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 50) v1 ==== 84332+0+0 (secure 0 0 0) 0x7fd28000fcf0 con 0x7fd28814f990 2023-05-22T15:45:03.223 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.213+0000 7fd284ff9640 1 --2- 172.21.15.16:0/3892255332 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd2740654a0 0x7fd274067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:03.223 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.213+0000 7fd284ff9640 1 -- 172.21.15.16:0/3892255332 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 5530+0+0 (secure 0 0 0) 0x7fd280082470 con 0x7fd28814f990 2023-05-22T15:45:03.223 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.213+0000 7fd287fff640 1 --2- 172.21.15.16:0/3892255332 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd2740654a0 0x7fd274067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:03.223 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.213+0000 7fd287fff640 1 --2- 172.21.15.16:0/3892255332 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd2740654a0 0x7fd274067950 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fd2880afbb0 tx=0x7fd278008040 comp rx=0 tx=0).ready entity=mgr.8304 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:45:03.224 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.217+0000 7fd284ff9640 1 -- 172.21.15.16:0/3892255332 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fd280050110 con 0x7fd28814f990 2023-05-22T15:45:03.375 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.373+0000 7fd28e994640 1 -- 172.21.15.16:0/3892255332 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fd2880cb010 con 0x7fd28814f990 2023-05-22T15:45:03.375 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.373+0000 7fd284ff9640 1 -- 172.21.15.16:0/3892255332 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v128) v1 ==== 84+0+126 (secure 0 0 0) 0x7fd280053730 con 0x7fd28814f990 2023-05-22T15:45:03.376 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:45:03.376 INFO:teuthology.orchestra.run.smithi016.stdout:[ 2023-05-22T15:45:03.376 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T15:45:03.376 INFO:teuthology.orchestra.run.smithi016.stdout: "poolnum": 1, 2023-05-22T15:45:03.376 INFO:teuthology.orchestra.run.smithi016.stdout: "poolname": ".mgr" 2023-05-22T15:45:03.377 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:45:03.377 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T15:45:03.377 INFO:teuthology.orchestra.run.smithi016.stdout: "poolnum": 2, 2023-05-22T15:45:03.377 INFO:teuthology.orchestra.run.smithi016.stdout: "poolname": "rbd" 2023-05-22T15:45:03.377 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T15:45:03.377 INFO:teuthology.orchestra.run.smithi016.stdout:] 2023-05-22T15:45:03.378 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.373+0000 7fd28e994640 1 -- 172.21.15.16:0/3892255332 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd2740654a0 msgr2=0x7fd274067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:45:03.378 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.373+0000 7fd28e994640 1 --2- 172.21.15.16:0/3892255332 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd2740654a0 0x7fd274067950 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fd2880afbb0 tx=0x7fd278008040 comp rx=0 tx=0).stop 2023-05-22T15:45:03.378 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.373+0000 7fd28e994640 1 -- 172.21.15.16:0/3892255332 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd28814f990 msgr2=0x7fd2880c1720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:45:03.379 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.373+0000 7fd28e994640 1 --2- 172.21.15.16:0/3892255332 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd28814f990 0x7fd2880c1720 secure :-1 s=READY pgs=1626 cs=0 l=1 rev1=1 crypto rx=0x7fd2880cdef0 tx=0x7fd28000f0d0 comp rx=0 tx=0).stop 2023-05-22T15:45:03.379 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.373+0000 7fd28e994640 1 -- 172.21.15.16:0/3892255332 shutdown_connections 2023-05-22T15:45:03.379 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.373+0000 7fd28e994640 1 --2- 172.21.15.16:0/3892255332 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd28814f990 0x7fd2880c1720 unknown :-1 s=CLOSED pgs=1626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:03.379 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.373+0000 7fd28e994640 1 --2- 172.21.15.16:0/3892255332 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd2740654a0 0x7fd274067950 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:03.379 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.373+0000 7fd28e994640 1 --2- 172.21.15.16:0/3892255332 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd2880c3f50 0x7fd28814f450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:03.379 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.373+0000 7fd28e994640 1 --2- 172.21.15.16:0/3892255332 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd2700043b0 0x7fd28814e380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:03.379 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.373+0000 7fd28e994640 1 -- 172.21.15.16:0/3892255332 >> 172.21.15.16:0/3892255332 conn(0x7fd288069720 msgr2=0x7fd28815bee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:45:03.380 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.373+0000 7fd28e994640 1 -- 172.21.15.16:0/3892255332 shutdown_connections 2023-05-22T15:45:03.380 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.373+0000 7fd28e994640 1 -- 172.21.15.16:0/3892255332 wait complete. 2023-05-22T15:45:03.395 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-05-22T15:45:03.395 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-05-22T15:45:03.511 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.509+0000 7fe0a23e7640 1 Processor -- start 2023-05-22T15:45:03.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.509+0000 7fe0a23e7640 1 -- start start 2023-05-22T15:45:03.513 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.509+0000 7fe0a23e7640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fe09c159a30 0x7fe09c06e380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:03.513 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.509+0000 7fe0a23e7640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe09c15d610 0x7fe09c0cbe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:03.513 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.509+0000 7fe0a23e7640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe09c0cde70 con 0x7fe09c0cf570 2023-05-22T15:45:03.513 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.509+0000 7fe0a23e7640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe09c15db20 con 0x7fe09c159a30 2023-05-22T15:45:03.513 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.509+0000 7fe0a23e7640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe09c06e8c0 con 0x7fe09c15d610 2023-05-22T15:45:03.513 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.509+0000 7fe0a0984640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fe09c0cf570 0x7fe09c0ce300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:44878/0 (socket says 172.21.15.16:44878) 2023-05-22T15:45:03.513 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.509+0000 7fe0a0984640 1 -- 172.21.15.16:0/3199532000 learned_addr learned my addr 172.21.15.16:0/3199532000 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:45:03.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.509+0000 7fe09bfff640 1 --2- 172.21.15.16:0/3199532000 >> v2:172.21.15.62:3300/0 conn(0x7fe09c159a30 0x7fe09c06e380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:03.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.509+0000 7fe09b7fe640 1 --2- 172.21.15.16:0/3199532000 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe09c15d610 0x7fe09c0cbe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:03.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.509+0000 7fe09affd640 1 -- 172.21.15.16:0/3199532000 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 189118382 0 0) 0x7fe09c0cde70 con 0x7fe09c0cf570 2023-05-22T15:45:03.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.509+0000 7fe09affd640 1 -- 172.21.15.16:0/3199532000 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe084003530 con 0x7fe09c0cf570 2023-05-22T15:45:03.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.509+0000 7fe09affd640 1 -- 172.21.15.16:0/3199532000 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2045273934 0 0) 0x7fe084003530 con 0x7fe09c0cf570 2023-05-22T15:45:03.516 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.509+0000 7fe09affd640 1 -- 172.21.15.16:0/3199532000 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe09c15d610 msgr2=0x7fe09c0cbe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:45:03.516 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.509+0000 7fe09affd640 1 --2- 172.21.15.16:0/3199532000 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe09c15d610 0x7fe09c0cbe10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:03.516 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.509+0000 7fe09affd640 1 -- 172.21.15.16:0/3199532000 >> v2:172.21.15.62:3300/0 conn(0x7fe09c159a30 msgr2=0x7fe09c06e380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:45:03.516 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.509+0000 7fe09affd640 1 --2- 172.21.15.16:0/3199532000 >> v2:172.21.15.62:3300/0 conn(0x7fe09c159a30 0x7fe09c06e380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:03.516 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.509+0000 7fe09affd640 1 -- 172.21.15.16:0/3199532000 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe09c15dde0 con 0x7fe09c0cf570 2023-05-22T15:45:03.516 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.509+0000 7fe09affd640 1 -- 172.21.15.16:0/3199532000 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7fe094001a00 con 0x7fe09c0cf570 2023-05-22T15:45:03.516 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.509+0000 7fe09affd640 1 -- 172.21.15.16:0/3199532000 >> v1:172.21.15.16:6789/0 conn(0x7fe09c0cf570 legacy=0x7fe09c0ce300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:45:03.517 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.509+0000 7fe09affd640 1 --2- 172.21.15.16:0/3199532000 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe09c15d610 0x7fe084003980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:03.517 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.509+0000 7fe09affd640 1 --2- 172.21.15.16:0/3199532000 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe084003f00 0x7fe0840047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:03.517 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.509+0000 7fe09affd640 1 --2- 172.21.15.16:0/3199532000 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe084004d10 0x7fe084005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:03.517 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.509+0000 7fe09affd640 1 -- 172.21.15.16:0/3199532000 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe09c15db20 con 0x7fe084004d10 2023-05-22T15:45:03.517 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.509+0000 7fe09affd640 1 -- 172.21.15.16:0/3199532000 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe09c06e8c0 con 0x7fe084003f00 2023-05-22T15:45:03.517 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.509+0000 7fe09affd640 1 -- 172.21.15.16:0/3199532000 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe08400b990 con 0x7fe09c15d610 2023-05-22T15:45:03.517 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.509+0000 7fe0a0984640 1 --2- 172.21.15.16:0/3199532000 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe084004d10 0x7fe084005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:03.517 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.509+0000 7fe0a0984640 1 -- 172.21.15.16:0/3199532000 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe09c15d610 msgr2=0x7fe084003980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:45:03.518 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.509+0000 7fe0a0984640 1 --2- 172.21.15.16:0/3199532000 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe09c15d610 0x7fe084003980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:03.518 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.509+0000 7fe09b7fe640 1 --2- 172.21.15.16:0/3199532000 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe084003f00 0x7fe0840047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:03.518 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.509+0000 7fe0a0984640 1 -- 172.21.15.16:0/3199532000 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe084003f00 msgr2=0x7fe0840047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:45:03.518 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.509+0000 7fe0a0984640 1 --2- 172.21.15.16:0/3199532000 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe084003f00 0x7fe0840047d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:03.518 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.509+0000 7fe0a0984640 1 -- 172.21.15.16:0/3199532000 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe084003530 con 0x7fe084004d10 2023-05-22T15:45:03.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.513+0000 7fe0a0984640 1 --2- 172.21.15.16:0/3199532000 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe084004d10 0x7fe084005180 secure :-1 s=READY pgs=1628 cs=0 l=1 rev1=1 crypto rx=0x7fe09400a8d0 tx=0x7fe09401d3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2aa9ee23c9fbafd2 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:45:03.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.513+0000 7fe09affd640 1 -- 172.21.15.16:0/3199532000 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0940ad630 con 0x7fe084004d10 2023-05-22T15:45:03.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.513+0000 7fe09affd640 1 -- 172.21.15.16:0/3199532000 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0940a8070 con 0x7fe084004d10 2023-05-22T15:45:03.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.513+0000 7fe09affd640 1 -- 172.21.15.16:0/3199532000 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0940ad9d0 con 0x7fe084004d10 2023-05-22T15:45:03.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.513+0000 7fe0a23e7640 1 -- 172.21.15.16:0/3199532000 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe084004d10 msgr2=0x7fe084005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:45:03.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.513+0000 7fe0a23e7640 1 --2- 172.21.15.16:0/3199532000 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe084004d10 0x7fe084005180 secure :-1 s=READY pgs=1628 cs=0 l=1 rev1=1 crypto rx=0x7fe09400a8d0 tx=0x7fe09401d3e0 comp rx=0 tx=0).stop 2023-05-22T15:45:03.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.513+0000 7fe0a0984640 1 -- 172.21.15.16:0/3199532000 reap_dead start 2023-05-22T15:45:03.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.513+0000 7fe0a23e7640 1 -- 172.21.15.16:0/3199532000 shutdown_connections 2023-05-22T15:45:03.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.513+0000 7fe0a23e7640 1 -- 172.21.15.16:0/3199532000 >> 172.21.15.16:0/3199532000 conn(0x7fe09c0700b0 msgr2=0x7fe09c06d9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:45:03.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.513+0000 7fe0a23e7640 1 -- 172.21.15.16:0/3199532000 shutdown_connections 2023-05-22T15:45:03.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.513+0000 7fe0a23e7640 1 -- 172.21.15.16:0/3199532000 wait complete. 2023-05-22T15:45:03.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.513+0000 7fe0a23e7640 1 Processor -- start 2023-05-22T15:45:03.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.513+0000 7fe0a23e7640 1 -- start start 2023-05-22T15:45:03.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.517+0000 7fe0a23e7640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe09c151840 0x7fe09c15bbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:03.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.517+0000 7fe0a23e7640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe09c0c9ee0 0x7fe09c15c110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:03.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.517+0000 7fe0a23e7640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe09c0c9890 0x7fe09c0b04f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:03.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.517+0000 7fe0a23e7640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe09c0c83c0 con 0x7fe09c0c9890 2023-05-22T15:45:03.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.517+0000 7fe0a23e7640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe09c0c96b0 con 0x7fe09c0c9ee0 2023-05-22T15:45:03.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.517+0000 7fe0a23e7640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe09c0c8610 con 0x7fe09c151840 2023-05-22T15:45:03.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.517+0000 7fe0a0984640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe09c0c9890 0x7fe09c0b04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:03.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.517+0000 7fe0a0984640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe09c0c9890 0x7fe09c0b04f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45138/0 (socket says 172.21.15.16:45138) 2023-05-22T15:45:03.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.517+0000 7fe0a0984640 1 -- 172.21.15.16:0/3903671644 learned_addr learned my addr 172.21.15.16:0/3903671644 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:45:03.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.517+0000 7fe09bfff640 1 --2- 172.21.15.16:0/3903671644 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe09c151840 0x7fe09c15bbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:03.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.517+0000 7fe09b7fe640 1 --2- 172.21.15.16:0/3903671644 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe09c0c9ee0 0x7fe09c15c110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:03.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.517+0000 7fe0a0984640 1 -- 172.21.15.16:0/3903671644 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe09c151840 msgr2=0x7fe09c15bbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:45:03.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.517+0000 7fe0a0984640 1 --2- 172.21.15.16:0/3903671644 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe09c151840 0x7fe09c15bbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:03.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.517+0000 7fe0a0984640 1 -- 172.21.15.16:0/3903671644 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe09c0c9ee0 msgr2=0x7fe09c15c110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:45:03.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.517+0000 7fe0a0984640 1 --2- 172.21.15.16:0/3903671644 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe09c0c9ee0 0x7fe09c15c110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:03.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.517+0000 7fe0a0984640 1 -- 172.21.15.16:0/3903671644 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe09c15e3a0 con 0x7fe09c0c9890 2023-05-22T15:45:03.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.517+0000 7fe0a0984640 1 --2- 172.21.15.16:0/3903671644 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe09c0c9890 0x7fe09c0b04f0 secure :-1 s=READY pgs=1629 cs=0 l=1 rev1=1 crypto rx=0x7fe094098d70 tx=0x7fe084005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:45:03.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.517+0000 7fe098ff9640 1 -- 172.21.15.16:0/3903671644 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0940019d0 con 0x7fe09c0c9890 2023-05-22T15:45:03.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.517+0000 7fe098ff9640 1 -- 172.21.15.16:0/3903671644 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe094002010 con 0x7fe09c0c9890 2023-05-22T15:45:03.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.517+0000 7fe0a23e7640 1 -- 172.21.15.16:0/3903671644 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe09c15e0c0 con 0x7fe09c0c9890 2023-05-22T15:45:03.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.517+0000 7fe098ff9640 1 -- 172.21.15.16:0/3903671644 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0940b66b0 con 0x7fe09c0c9890 2023-05-22T15:45:03.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.517+0000 7fe0a23e7640 1 -- 172.21.15.16:0/3903671644 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe09c159e00 con 0x7fe09c0c9890 2023-05-22T15:45:03.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.517+0000 7fe0a23e7640 1 -- 172.21.15.16:0/3903671644 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe09c066910 con 0x7fe09c0c9890 2023-05-22T15:45:03.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.517+0000 7fe098ff9640 1 -- 172.21.15.16:0/3903671644 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 50) v1 ==== 84332+0+0 (secure 0 0 0) 0x7fe0940aa030 con 0x7fe09c0c9890 2023-05-22T15:45:03.526 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.517+0000 7fe098ff9640 1 --2- 172.21.15.16:0/3903671644 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fe0880654d0 0x7fe088067980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:03.526 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.517+0000 7fe098ff9640 1 -- 172.21.15.16:0/3903671644 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 5530+0+0 (secure 0 0 0) 0x7fe09411c960 con 0x7fe09c0c9890 2023-05-22T15:45:03.526 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.517+0000 7fe09bfff640 1 --2- 172.21.15.16:0/3903671644 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fe0880654d0 0x7fe088067980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:03.526 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.517+0000 7fe09bfff640 1 --2- 172.21.15.16:0/3903671644 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fe0880654d0 0x7fe088067980 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fe08c0047c0 tx=0x7fe08c008040 comp rx=0 tx=0).ready entity=mgr.8304 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:45:03.526 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.521+0000 7fe098ff9640 1 -- 172.21.15.16:0/3903671644 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fe0940ea530 con 0x7fe09c0c9890 2023-05-22T15:45:03.676 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.673+0000 7fe0a23e7640 1 -- 172.21.15.16:0/3903671644 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fe09c04dbe0 con 0x7fe09c0c9890 2023-05-22T15:45:03.808 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.805+0000 7fe098ff9640 1 -- 172.21.15.16:0/3903671644 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v129) v1 ==== 138+0+0 (secure 0 0 0) 0x7fe0940bb0b0 con 0x7fe09c0c9890 2023-05-22T15:45:03.808 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'cephfs_metadata' created 2023-05-22T15:45:03.812 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.809+0000 7fe0a23e7640 1 -- 172.21.15.16:0/3903671644 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fe0880654d0 msgr2=0x7fe088067980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:45:03.813 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.809+0000 7fe0a23e7640 1 --2- 172.21.15.16:0/3903671644 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fe0880654d0 0x7fe088067980 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fe08c0047c0 tx=0x7fe08c008040 comp rx=0 tx=0).stop 2023-05-22T15:45:03.813 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.809+0000 7fe0a23e7640 1 -- 172.21.15.16:0/3903671644 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe09c0c9890 msgr2=0x7fe09c0b04f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:45:03.813 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.809+0000 7fe0a23e7640 1 --2- 172.21.15.16:0/3903671644 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe09c0c9890 0x7fe09c0b04f0 secure :-1 s=READY pgs=1629 cs=0 l=1 rev1=1 crypto rx=0x7fe094098d70 tx=0x7fe084005800 comp rx=0 tx=0).stop 2023-05-22T15:45:03.814 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.809+0000 7fe0a23e7640 1 -- 172.21.15.16:0/3903671644 shutdown_connections 2023-05-22T15:45:03.814 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.809+0000 7fe0a23e7640 1 --2- 172.21.15.16:0/3903671644 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe09c0c9890 0x7fe09c0b04f0 unknown :-1 s=CLOSED pgs=1629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:03.814 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.809+0000 7fe0a23e7640 1 --2- 172.21.15.16:0/3903671644 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe09c0c9ee0 0x7fe09c15c110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:03.814 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.809+0000 7fe0a23e7640 1 --2- 172.21.15.16:0/3903671644 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fe0880654d0 0x7fe088067980 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:03.814 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.809+0000 7fe0a23e7640 1 --2- 172.21.15.16:0/3903671644 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe09c151840 0x7fe09c15bbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:03.814 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.809+0000 7fe0a23e7640 1 -- 172.21.15.16:0/3903671644 >> 172.21.15.16:0/3903671644 conn(0x7fe09c0700b0 msgr2=0x7fe09c0cbdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:45:03.815 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.809+0000 7fe0a23e7640 1 -- 172.21.15.16:0/3903671644 shutdown_connections 2023-05-22T15:45:03.816 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.813+0000 7fe0a23e7640 1 -- 172.21.15.16:0/3903671644 wait complete. 2023-05-22T15:45:03.835 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-05-22T15:45:03.966 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.961+0000 7fa381dd4640 1 Processor -- start 2023-05-22T15:45:03.967 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.961+0000 7fa381dd4640 1 -- start start 2023-05-22T15:45:03.967 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.961+0000 7fa381dd4640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa37c0cf870 0x7fa37c06db00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:03.967 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.961+0000 7fa381dd4640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fa37c06e740 0x7fa37c06ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:03.968 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.961+0000 7fa381dd4640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa37c06e040 con 0x7fa37c15cfa0 2023-05-22T15:45:03.968 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.961+0000 7fa381dd4640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa37c0cb9f0 con 0x7fa37c06e740 2023-05-22T15:45:03.968 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.961+0000 7fa381dd4640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa37c06f100 con 0x7fa37c0cf870 2023-05-22T15:45:03.969 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.961+0000 7fa37bfff640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fa37c06e740 0x7fa37c06ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:03.969 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.961+0000 7fa37affd640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa37c0cf870 0x7fa37c06db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:03.970 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.961+0000 7fa37affd640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa37c0cf870 0x7fa37c06db00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3301/0 says I am v2:172.21.15.16:57690/0 (socket says 172.21.15.16:57690) 2023-05-22T15:45:03.970 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.961+0000 7fa37affd640 1 -- 172.21.15.16:0/1607236403 learned_addr learned my addr 172.21.15.16:0/1607236403 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:45:03.971 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.961+0000 7fa37affd640 1 -- 172.21.15.16:0/1607236403 >> v2:172.21.15.62:3300/0 conn(0x7fa37c06e740 msgr2=0x7fa37c06ebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:45:03.971 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.961+0000 7fa37affd640 1 --2- 172.21.15.16:0/1607236403 >> v2:172.21.15.62:3300/0 conn(0x7fa37c06e740 0x7fa37c06ebc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:03.971 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.961+0000 7fa37affd640 1 -- 172.21.15.16:0/1607236403 >> v1:172.21.15.62:6789/0 conn(0x7fa37c15cfa0 legacy=0x7fa37c0b1fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:45:03.971 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.961+0000 7fa37affd640 1 -- 172.21.15.16:0/1607236403 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa37c06e490 con 0x7fa37c0cf870 2023-05-22T15:45:03.971 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.961+0000 7fa37bfff640 1 --2- 172.21.15.16:0/1607236403 >> v2:172.21.15.62:3300/0 conn(0x7fa37c06e740 0x7fa37c06ebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-05-22T15:45:03.974 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.965+0000 7fa37affd640 1 --2- 172.21.15.16:0/1607236403 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa37c0cf870 0x7fa37c06db00 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fa36c009870 tx=0x7fa36c01c480 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6619cdf0b7d949fe server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:45:03.974 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.965+0000 7fa37a7fc640 1 -- 172.21.15.16:0/1607236403 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa36c0ab680 con 0x7fa37c0cf870 2023-05-22T15:45:03.974 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.965+0000 7fa37a7fc640 1 -- 172.21.15.16:0/1607236403 <== mon.2 v2:172.21.15.62:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa36c0ab7e0 con 0x7fa37c0cf870 2023-05-22T15:45:03.974 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.965+0000 7fa37a7fc640 1 -- 172.21.15.16:0/1607236403 <== mon.2 v2:172.21.15.62:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa36c0abb70 con 0x7fa37c0cf870 2023-05-22T15:45:03.974 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.965+0000 7fa381dd4640 1 -- 172.21.15.16:0/1607236403 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa37c0cf870 msgr2=0x7fa37c06db00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:45:03.975 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.965+0000 7fa381dd4640 1 --2- 172.21.15.16:0/1607236403 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa37c0cf870 0x7fa37c06db00 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fa36c009870 tx=0x7fa36c01c480 comp rx=0 tx=0).stop 2023-05-22T15:45:03.975 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.965+0000 7fa381dd4640 1 -- 172.21.15.16:0/1607236403 shutdown_connections 2023-05-22T15:45:03.975 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.965+0000 7fa381dd4640 1 --2- 172.21.15.16:0/1607236403 >> v2:172.21.15.62:3300/0 conn(0x7fa37c06e740 0x7fa37c06ebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:03.975 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.965+0000 7fa381dd4640 1 --2- 172.21.15.16:0/1607236403 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa37c0cf870 0x7fa37c06db00 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:03.975 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.965+0000 7fa381dd4640 1 -- 172.21.15.16:0/1607236403 >> 172.21.15.16:0/1607236403 conn(0x7fa37c0cc870 msgr2=0x7fa37c0cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:45:03.975 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.965+0000 7fa381dd4640 1 -- 172.21.15.16:0/1607236403 shutdown_connections 2023-05-22T15:45:03.976 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.965+0000 7fa381dd4640 1 -- 172.21.15.16:0/1607236403 wait complete. 2023-05-22T15:45:03.976 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.965+0000 7fa381dd4640 1 Processor -- start 2023-05-22T15:45:03.976 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.969+0000 7fa381dd4640 1 -- start start 2023-05-22T15:45:03.976 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.969+0000 7fa381dd4640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa37c06e740 0x7fa37c157290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:03.977 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.969+0000 7fa381dd4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa37c0cf870 0x7fa37c0cd860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:03.977 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.969+0000 7fa381dd4640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa37c15cfa0 0x7fa37c14d960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:03.977 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.969+0000 7fa381dd4640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa37c1577d0 con 0x7fa37c0cf870 2023-05-22T15:45:03.977 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.969+0000 7fa381dd4640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa37c0cdda0 con 0x7fa37c06e740 2023-05-22T15:45:03.977 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.969+0000 7fa381dd4640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa37c157a00 con 0x7fa37c15cfa0 2023-05-22T15:45:03.977 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.969+0000 7fa37affd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa37c0cf870 0x7fa37c0cd860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:03.977 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.969+0000 7fa37affd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa37c0cf870 0x7fa37c0cd860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45152/0 (socket says 172.21.15.16:45152) 2023-05-22T15:45:03.977 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.969+0000 7fa37affd640 1 -- 172.21.15.16:0/1810680935 learned_addr learned my addr 172.21.15.16:0/1810680935 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:45:03.978 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.969+0000 7fa37bfff640 1 --2- 172.21.15.16:0/1810680935 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa37c15cfa0 0x7fa37c14d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:03.978 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.969+0000 7fa37b7fe640 1 --2- 172.21.15.16:0/1810680935 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa37c06e740 0x7fa37c157290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:03.978 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.969+0000 7fa37affd640 1 -- 172.21.15.16:0/1810680935 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa37c15cfa0 msgr2=0x7fa37c14d960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:45:03.978 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.969+0000 7fa37affd640 1 --2- 172.21.15.16:0/1810680935 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa37c15cfa0 0x7fa37c14d960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:03.978 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.969+0000 7fa37affd640 1 -- 172.21.15.16:0/1810680935 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa37c06e740 msgr2=0x7fa37c157290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:45:03.978 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.969+0000 7fa37affd640 1 --2- 172.21.15.16:0/1810680935 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa37c06e740 0x7fa37c157290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:03.978 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.969+0000 7fa37affd640 1 -- 172.21.15.16:0/1810680935 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa374009420 con 0x7fa37c0cf870 2023-05-22T15:45:03.979 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.969+0000 7fa37affd640 1 --2- 172.21.15.16:0/1810680935 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa37c0cf870 0x7fa37c0cd860 secure :-1 s=READY pgs=1630 cs=0 l=1 rev1=1 crypto rx=0x7fa36c0ad040 tx=0x7fa36c01acc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:45:03.979 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.969+0000 7fa373fff640 1 -- 172.21.15.16:0/1810680935 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa36c0ab680 con 0x7fa37c0cf870 2023-05-22T15:45:03.979 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.969+0000 7fa373fff640 1 -- 172.21.15.16:0/1810680935 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa36c0b4820 con 0x7fa37c0cf870 2023-05-22T15:45:03.980 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.969+0000 7fa373fff640 1 -- 172.21.15.16:0/1810680935 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa36c0b4bd0 con 0x7fa37c0cf870 2023-05-22T15:45:03.980 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.969+0000 7fa381dd4640 1 -- 172.21.15.16:0/1810680935 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa36c0094d0 con 0x7fa37c0cf870 2023-05-22T15:45:03.980 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.969+0000 7fa381dd4640 1 -- 172.21.15.16:0/1810680935 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa37c0cb610 con 0x7fa37c0cf870 2023-05-22T15:45:03.980 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.969+0000 7fa381dd4640 1 -- 172.21.15.16:0/1810680935 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa37c0cb330 con 0x7fa37c0cf870 2023-05-22T15:45:03.980 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.969+0000 7fa373fff640 1 -- 172.21.15.16:0/1810680935 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 50) v1 ==== 84332+0+0 (secure 0 0 0) 0x7fa36c0b4d30 con 0x7fa37c0cf870 2023-05-22T15:45:03.981 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.969+0000 7fa373fff640 1 --2- 172.21.15.16:0/1810680935 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fa35c0654a0 0x7fa35c067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:03.981 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.973+0000 7fa37b7fe640 1 --2- 172.21.15.16:0/1810680935 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fa35c0654a0 0x7fa35c067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:03.981 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.973+0000 7fa373fff640 1 -- 172.21.15.16:0/1810680935 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5900+0+0 (secure 0 0 0) 0x7fa36c11d9e0 con 0x7fa37c0cf870 2023-05-22T15:45:03.982 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.973+0000 7fa37b7fe640 1 --2- 172.21.15.16:0/1810680935 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fa35c0654a0 0x7fa35c067950 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fa37c157fe0 tx=0x7fa364009040 comp rx=0 tx=0).ready entity=mgr.8304 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:45:03.982 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:03.977+0000 7fa373fff640 1 -- 172.21.15.16:0/1810680935 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fa36c0eb590 con 0x7fa37c0cf870 2023-05-22T15:45:04.127 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:04.125+0000 7fa381dd4640 1 -- 172.21.15.16:0/1810680935 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fa37c15d370 con 0x7fa37c0cf870 2023-05-22T15:45:04.824 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:04.821+0000 7fa373fff640 1 -- 172.21.15.16:0/1810680935 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v130) v1 ==== 170+0+0 (secure 0 0 0) 0x7fa36c0c40d0 con 0x7fa37c0cf870 2023-05-22T15:45:04.825 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'cephfs_data' created 2023-05-22T15:45:04.832 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:04.829+0000 7fa381dd4640 1 -- 172.21.15.16:0/1810680935 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fa35c0654a0 msgr2=0x7fa35c067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:45:04.832 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:04.829+0000 7fa381dd4640 1 --2- 172.21.15.16:0/1810680935 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fa35c0654a0 0x7fa35c067950 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fa37c157fe0 tx=0x7fa364009040 comp rx=0 tx=0).stop 2023-05-22T15:45:04.832 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:04.829+0000 7fa381dd4640 1 -- 172.21.15.16:0/1810680935 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa37c0cf870 msgr2=0x7fa37c0cd860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:45:04.832 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:04.829+0000 7fa381dd4640 1 --2- 172.21.15.16:0/1810680935 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa37c0cf870 0x7fa37c0cd860 secure :-1 s=READY pgs=1630 cs=0 l=1 rev1=1 crypto rx=0x7fa36c0ad040 tx=0x7fa36c01acc0 comp rx=0 tx=0).stop 2023-05-22T15:45:04.835 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:04.829+0000 7fa381dd4640 1 -- 172.21.15.16:0/1810680935 shutdown_connections 2023-05-22T15:45:04.835 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:04.829+0000 7fa381dd4640 1 --2- 172.21.15.16:0/1810680935 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fa35c0654a0 0x7fa35c067950 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:04.835 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:04.829+0000 7fa381dd4640 1 --2- 172.21.15.16:0/1810680935 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa37c15cfa0 0x7fa37c14d960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:04.836 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:04.829+0000 7fa381dd4640 1 --2- 172.21.15.16:0/1810680935 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa37c0cf870 0x7fa37c0cd860 secure :-1 s=CLOSED pgs=1630 cs=0 l=1 rev1=1 crypto rx=0x7fa36c0ad040 tx=0x7fa36c01acc0 comp rx=0 tx=0).stop 2023-05-22T15:45:04.836 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:04.829+0000 7fa381dd4640 1 --2- 172.21.15.16:0/1810680935 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa37c06e740 0x7fa37c157290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:04.836 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:04.829+0000 7fa381dd4640 1 -- 172.21.15.16:0/1810680935 >> 172.21.15.16:0/1810680935 conn(0x7fa37c0cc870 msgr2=0x7fa37c0aef60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:45:04.848 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:04.845+0000 7fa381dd4640 1 -- 172.21.15.16:0/1810680935 shutdown_connections 2023-05-22T15:45:04.850 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:04.845+0000 7fa381dd4640 1 -- 172.21.15.16:0/1810680935 wait complete. 2023-05-22T15:45:04.875 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-05-22T15:45:04.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:04.997+0000 7f1ca5298640 1 Processor -- start 2023-05-22T15:45:04.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:04.997+0000 7f1ca5298640 1 -- start start 2023-05-22T15:45:05.003 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:04.997+0000 7f1ca5298640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1ca0159a90 0x7f1ca00cbb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:05.003 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:04.997+0000 7f1ca5298640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f1ca00cd770 0x7f1ca015dbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:05.004 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:04.997+0000 7f1ca5298640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ca0159e60 con 0x7f1ca015e130 2023-05-22T15:45:05.004 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:04.997+0000 7f1ca5298640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ca015d4f0 con 0x7f1ca00cd770 2023-05-22T15:45:05.004 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:04.997+0000 7f1ca5298640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ca00cc1d0 con 0x7f1ca0159a90 2023-05-22T15:45:05.004 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:04.997+0000 7f1c9f7fe640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f1ca015e130 0x7f1ca00cdd90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:44890/0 (socket says 172.21.15.16:44890) 2023-05-22T15:45:05.004 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:04.997+0000 7f1c9f7fe640 1 -- 172.21.15.16:0/3800522034 learned_addr learned my addr 172.21.15.16:0/3800522034 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:45:05.005 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:04.997+0000 7f1c9e7fc640 1 --2- 172.21.15.16:0/3800522034 >> v2:172.21.15.62:3300/0 conn(0x7f1ca00cd770 0x7f1ca015dbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:05.005 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:04.997+0000 7f1c9effd640 1 --2- 172.21.15.16:0/3800522034 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1ca0159a90 0x7f1ca00cbb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:05.005 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:04.997+0000 7f1c9dffb640 1 -- 172.21.15.16:0/3800522034 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3556668972 0 0) 0x7f1ca0159e60 con 0x7f1ca015e130 2023-05-22T15:45:05.005 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:04.997+0000 7f1c9dffb640 1 -- 172.21.15.16:0/3800522034 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c88003530 con 0x7f1ca015e130 2023-05-22T15:45:05.005 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:04.997+0000 7f1c9dffb640 1 -- 172.21.15.16:0/3800522034 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 120786827 0 0) 0x7f1c88003530 con 0x7f1ca015e130 2023-05-22T15:45:05.005 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:04.997+0000 7f1c9dffb640 1 -- 172.21.15.16:0/3800522034 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1ca0159a90 msgr2=0x7f1ca00cbb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:45:05.005 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:04.997+0000 7f1c9dffb640 1 --2- 172.21.15.16:0/3800522034 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1ca0159a90 0x7f1ca00cbb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:05.006 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:04.997+0000 7f1c9dffb640 1 -- 172.21.15.16:0/3800522034 >> v2:172.21.15.62:3300/0 conn(0x7f1ca00cd770 msgr2=0x7f1ca015dbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:45:05.006 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:04.997+0000 7f1c9dffb640 1 --2- 172.21.15.16:0/3800522034 >> v2:172.21.15.62:3300/0 conn(0x7f1ca00cd770 0x7f1ca015dbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:05.006 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:04.997+0000 7f1c9dffb640 1 -- 172.21.15.16:0/3800522034 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ca00cc4b0 con 0x7f1ca015e130 2023-05-22T15:45:05.006 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:04.997+0000 7f1c9dffb640 1 -- 172.21.15.16:0/3800522034 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7f1c98001a00 con 0x7f1ca015e130 2023-05-22T15:45:05.006 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:04.997+0000 7f1c9e7fc640 1 --2- 172.21.15.16:0/3800522034 >> v2:172.21.15.62:3300/0 conn(0x7f1ca00cd770 0x7f1ca015dbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T15:45:05.006 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:04.997+0000 7f1c9dffb640 1 -- 172.21.15.16:0/3800522034 >> v1:172.21.15.16:6789/0 conn(0x7f1ca015e130 legacy=0x7f1ca00cdd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:45:05.007 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:04.997+0000 7f1c9dffb640 1 --2- 172.21.15.16:0/3800522034 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1c88003980 0x7f1c88003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:05.007 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:04.997+0000 7f1c9dffb640 1 --2- 172.21.15.16:0/3800522034 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c880047d0 0x7f1c88004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:05.007 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:04.997+0000 7f1c9dffb640 1 --2- 172.21.15.16:0/3800522034 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1ca0159a90 0x7f1c88005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:05.007 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:04.997+0000 7f1c9dffb640 1 -- 172.21.15.16:0/3800522034 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ca015d4f0 con 0x7f1c880047d0 2023-05-22T15:45:05.007 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:04.997+0000 7f1c9dffb640 1 -- 172.21.15.16:0/3800522034 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ca00cc1d0 con 0x7f1c88003980 2023-05-22T15:45:05.008 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:04.997+0000 7f1c9dffb640 1 -- 172.21.15.16:0/3800522034 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c8800bac0 con 0x7f1ca0159a90 2023-05-22T15:45:05.008 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:04.997+0000 7f1c9e7fc640 1 --2- 172.21.15.16:0/3800522034 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c880047d0 0x7f1c88004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:05.008 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:04.997+0000 7f1c9effd640 1 --2- 172.21.15.16:0/3800522034 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1c88003980 0x7f1c88003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:05.008 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:04.997+0000 7f1c9f7fe640 1 --2- 172.21.15.16:0/3800522034 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1ca0159a90 0x7f1c88005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:05.008 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:04.997+0000 7f1c9e7fc640 1 -- 172.21.15.16:0/3800522034 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1ca0159a90 msgr2=0x7f1c88005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:45:05.008 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:04.997+0000 7f1c9e7fc640 1 --2- 172.21.15.16:0/3800522034 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1ca0159a90 0x7f1c88005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:05.009 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:04.997+0000 7f1c9e7fc640 1 -- 172.21.15.16:0/3800522034 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1c88003980 msgr2=0x7f1c88003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:45:05.009 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:04.997+0000 7f1c9e7fc640 1 --2- 172.21.15.16:0/3800522034 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1c88003980 0x7f1c88003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:05.009 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:04.997+0000 7f1c9e7fc640 1 -- 172.21.15.16:0/3800522034 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c88003530 con 0x7f1c880047d0 2023-05-22T15:45:05.009 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:04.997+0000 7f1c9effd640 1 --2- 172.21.15.16:0/3800522034 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1c88003980 0x7f1c88003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T15:45:05.012 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.001+0000 7f1c9e7fc640 1 --2- 172.21.15.16:0/3800522034 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c880047d0 0x7f1c88004c40 secure :-1 s=READY pgs=1632 cs=0 l=1 rev1=1 crypto rx=0x7f1c940098a0 tx=0x7f1c9401c5a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f22859e8a5d19ae server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:45:05.012 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.001+0000 7f1c9dffb640 1 -- 172.21.15.16:0/3800522034 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c940aa640 con 0x7f1c880047d0 2023-05-22T15:45:05.012 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.001+0000 7f1c9dffb640 1 -- 172.21.15.16:0/3800522034 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c940aa7a0 con 0x7f1c880047d0 2023-05-22T15:45:05.012 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.001+0000 7f1c9dffb640 1 -- 172.21.15.16:0/3800522034 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c940aab10 con 0x7f1c880047d0 2023-05-22T15:45:05.013 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.001+0000 7f1ca5298640 1 -- 172.21.15.16:0/3800522034 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c880047d0 msgr2=0x7f1c88004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:45:05.013 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.001+0000 7f1ca5298640 1 --2- 172.21.15.16:0/3800522034 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c880047d0 0x7f1c88004c40 secure :-1 s=READY pgs=1632 cs=0 l=1 rev1=1 crypto rx=0x7f1c940098a0 tx=0x7f1c9401c5a0 comp rx=0 tx=0).stop 2023-05-22T15:45:05.013 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.001+0000 7f1c9f7fe640 1 -- 172.21.15.16:0/3800522034 reap_dead start 2023-05-22T15:45:05.013 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.001+0000 7f1ca5298640 1 -- 172.21.15.16:0/3800522034 shutdown_connections 2023-05-22T15:45:05.013 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.001+0000 7f1ca5298640 1 -- 172.21.15.16:0/3800522034 >> 172.21.15.16:0/3800522034 conn(0x7f1ca006f4f0 msgr2=0x7f1ca006f8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:45:05.013 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.001+0000 7f1ca5298640 1 -- 172.21.15.16:0/3800522034 shutdown_connections 2023-05-22T15:45:05.013 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.001+0000 7f1ca5298640 1 -- 172.21.15.16:0/3800522034 wait complete. 2023-05-22T15:45:05.014 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.001+0000 7f1ca5298640 1 Processor -- start 2023-05-22T15:45:05.014 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.001+0000 7f1ca5298640 1 -- start start 2023-05-22T15:45:05.014 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.005+0000 7f1ca5298640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1c880047d0 0x7f1ca015a0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:05.014 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.005+0000 7f1ca5298640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1ca015bd70 0x7f1ca015c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:05.014 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.005+0000 7f1ca5298640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1ca01f13e0 0x7f1ca01f1850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:05.014 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.005+0000 7f1ca5298640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ca013ffa0 con 0x7f1ca01f13e0 2023-05-22T15:45:05.015 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.005+0000 7f1ca5298640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ca015c850 con 0x7f1ca015bd70 2023-05-22T15:45:05.015 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.005+0000 7f1ca5298640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ca015cb30 con 0x7f1c880047d0 2023-05-22T15:45:05.015 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.005+0000 7f1c9f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1ca01f13e0 0x7f1ca01f1850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:05.015 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.005+0000 7f1c9f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1ca01f13e0 0x7f1ca01f1850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45172/0 (socket says 172.21.15.16:45172) 2023-05-22T15:45:05.015 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.005+0000 7f1c9f7fe640 1 -- 172.21.15.16:0/3646930779 learned_addr learned my addr 172.21.15.16:0/3646930779 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:45:05.015 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.005+0000 7f1c9effd640 1 --2- 172.21.15.16:0/3646930779 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1c880047d0 0x7f1ca015a0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:05.016 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.005+0000 7f1c9e7fc640 1 --2- 172.21.15.16:0/3646930779 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1ca015bd70 0x7f1ca015c1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:05.016 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.005+0000 7f1c9f7fe640 1 -- 172.21.15.16:0/3646930779 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1c880047d0 msgr2=0x7f1ca015a0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:45:05.016 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.005+0000 7f1c9f7fe640 1 --2- 172.21.15.16:0/3646930779 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1c880047d0 0x7f1ca015a0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:05.016 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.005+0000 7f1c9f7fe640 1 -- 172.21.15.16:0/3646930779 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1ca015bd70 msgr2=0x7f1ca015c1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:45:05.016 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.005+0000 7f1c9f7fe640 1 --2- 172.21.15.16:0/3646930779 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1ca015bd70 0x7f1ca015c1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:05.016 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.005+0000 7f1c9f7fe640 1 -- 172.21.15.16:0/3646930779 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ca00cd2e0 con 0x7f1ca01f13e0 2023-05-22T15:45:05.016 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.005+0000 7f1c9f7fe640 1 --2- 172.21.15.16:0/3646930779 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1ca01f13e0 0x7f1ca01f1850 secure :-1 s=READY pgs=1633 cs=0 l=1 rev1=1 crypto rx=0x7f1ca015e500 tx=0x7f1c98007a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:45:05.017 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.005+0000 7f1c877fe640 1 -- 172.21.15.16:0/3646930779 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c980115b0 con 0x7f1ca01f13e0 2023-05-22T15:45:05.017 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.005+0000 7f1c877fe640 1 -- 172.21.15.16:0/3646930779 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c98011750 con 0x7f1ca01f13e0 2023-05-22T15:45:05.017 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.005+0000 7f1c877fe640 1 -- 172.21.15.16:0/3646930779 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c98011b00 con 0x7f1ca01f13e0 2023-05-22T15:45:05.017 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.005+0000 7f1ca5298640 1 -- 172.21.15.16:0/3646930779 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ca00cbf40 con 0x7f1ca01f13e0 2023-05-22T15:45:05.017 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.005+0000 7f1ca5298640 1 -- 172.21.15.16:0/3646930779 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ca00ce4a0 con 0x7f1ca01f13e0 2023-05-22T15:45:05.017 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.005+0000 7f1ca5298640 1 -- 172.21.15.16:0/3646930779 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ca004dbe0 con 0x7f1ca01f13e0 2023-05-22T15:45:05.017 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.005+0000 7f1c877fe640 1 -- 172.21.15.16:0/3646930779 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 50) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f1c98011c60 con 0x7f1ca01f13e0 2023-05-22T15:45:05.018 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.005+0000 7f1c877fe640 1 --2- 172.21.15.16:0/3646930779 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f1c8c0654a0 0x7f1c8c067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:05.018 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.005+0000 7f1c877fe640 1 -- 172.21.15.16:0/3646930779 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 6282+0+0 (secure 0 0 0) 0x7f1c98082270 con 0x7f1ca01f13e0 2023-05-22T15:45:05.018 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.005+0000 7f1c9effd640 1 --2- 172.21.15.16:0/3646930779 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f1c8c0654a0 0x7f1c8c067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:05.018 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.005+0000 7f1c9effd640 1 --2- 172.21.15.16:0/3646930779 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f1c8c0654a0 0x7f1c8c067950 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f1c900099d0 tx=0x7f1c90008040 comp rx=0 tx=0).ready entity=mgr.8304 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:45:05.019 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.009+0000 7f1c877fe640 1 -- 172.21.15.16:0/3646930779 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f1c9804c620 con 0x7f1ca01f13e0 2023-05-22T15:45:05.200 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.197+0000 7f1ca5298640 1 -- 172.21.15.16:0/3646930779 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f1ca006dcc0 con 0x7f1ca01f13e0 2023-05-22T15:45:05.215 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.213+0000 7f1c877fe640 1 -- 172.21.15.16:0/3646930779 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 16 and data pool 17 v47) v1 ==== 174+0+0 (secure 0 0 0) 0x7f1c9804fda0 con 0x7f1ca01f13e0 2023-05-22T15:45:05.216 INFO:teuthology.orchestra.run.smithi016.stderr:new fs with metadata pool 16 and data pool 17 2023-05-22T15:45:05.220 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.217+0000 7f1ca5298640 1 -- 172.21.15.16:0/3646930779 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f1c8c0654a0 msgr2=0x7f1c8c067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:45:05.220 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.217+0000 7f1ca5298640 1 --2- 172.21.15.16:0/3646930779 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f1c8c0654a0 0x7f1c8c067950 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f1c900099d0 tx=0x7f1c90008040 comp rx=0 tx=0).stop 2023-05-22T15:45:05.220 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.217+0000 7f1ca5298640 1 -- 172.21.15.16:0/3646930779 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1ca01f13e0 msgr2=0x7f1ca01f1850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:45:05.221 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.217+0000 7f1ca5298640 1 --2- 172.21.15.16:0/3646930779 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1ca01f13e0 0x7f1ca01f1850 secure :-1 s=READY pgs=1633 cs=0 l=1 rev1=1 crypto rx=0x7f1ca015e500 tx=0x7f1c98007a20 comp rx=0 tx=0).stop 2023-05-22T15:45:05.227 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.221+0000 7f1ca5298640 1 -- 172.21.15.16:0/3646930779 shutdown_connections 2023-05-22T15:45:05.227 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.225+0000 7f1ca5298640 1 --2- 172.21.15.16:0/3646930779 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1ca01f13e0 0x7f1ca01f1850 unknown :-1 s=CLOSED pgs=1633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:05.228 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.225+0000 7f1ca5298640 1 --2- 172.21.15.16:0/3646930779 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1ca015bd70 0x7f1ca015c1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:05.228 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.225+0000 7f1ca5298640 1 --2- 172.21.15.16:0/3646930779 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f1c8c0654a0 0x7f1c8c067950 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:05.228 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.225+0000 7f1ca5298640 1 --2- 172.21.15.16:0/3646930779 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1c880047d0 0x7f1ca015a0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:05.229 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.225+0000 7f1ca5298640 1 -- 172.21.15.16:0/3646930779 >> 172.21.15.16:0/3646930779 conn(0x7f1ca006f4f0 msgr2=0x7f1ca00c4110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:45:05.229 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.225+0000 7f1ca5298640 1 -- 172.21.15.16:0/3646930779 shutdown_connections 2023-05-22T15:45:05.229 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.225+0000 7f1ca5298640 1 -- 172.21.15.16:0/3646930779 wait complete. 2023-05-22T15:45:05.255 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-05-22T15:45:05.370 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.369+0000 7fbc714db640 1 Processor -- start 2023-05-22T15:45:05.371 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.369+0000 7fbc714db640 1 -- start start 2023-05-22T15:45:05.371 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.369+0000 7fbc714db640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fbc6c159cd0 0x7fbc6c15a0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:05.372 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.369+0000 7fbc714db640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbc6c0cc6f0 0x7fbc6c0cfe30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:05.372 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.369+0000 7fbc714db640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc6c0cf9a0 con 0x7fbc6c0b2880 2023-05-22T15:45:05.372 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.369+0000 7fbc714db640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc6c15aba0 con 0x7fbc6c159cd0 2023-05-22T15:45:05.372 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.369+0000 7fbc714db640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc6c15ae80 con 0x7fbc6c0cc6f0 2023-05-22T15:45:05.372 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.369+0000 7fbc6affd640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fbc6c159cd0 0x7fbc6c15a0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:05.372 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.369+0000 7fbc6a7fc640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fbc6c0b2880 0x7fbc6c0cbfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:44902/0 (socket says 172.21.15.16:44902) 2023-05-22T15:45:05.373 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.369+0000 7fbc6affd640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fbc6c159cd0 0x7fbc6c15a0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45176/0 (socket says 172.21.15.16:45176) 2023-05-22T15:45:05.373 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.369+0000 7fbc6a7fc640 1 -- 172.21.15.16:0/856833805 learned_addr learned my addr 172.21.15.16:0/856833805 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:45:05.373 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.369+0000 7fbc6affd640 1 -- 172.21.15.16:0/856833805 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbc6c0cc6f0 msgr2=0x7fbc6c0cfe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:45:05.373 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.369+0000 7fbc6b7fe640 1 --2- 172.21.15.16:0/856833805 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbc6c0cc6f0 0x7fbc6c0cfe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:05.374 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.369+0000 7fbc6affd640 1 --2- 172.21.15.16:0/856833805 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbc6c0cc6f0 0x7fbc6c0cfe30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:05.374 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.369+0000 7fbc6affd640 1 -- 172.21.15.16:0/856833805 >> v1:172.21.15.16:6789/0 conn(0x7fbc6c0b2880 legacy=0x7fbc6c0cbfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:45:05.374 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.369+0000 7fbc69ffb640 1 -- 172.21.15.16:0/856833805 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3374047272 0 0) 0x7fbc6c0cf9a0 con 0x7fbc6c0b2880 2023-05-22T15:45:05.374 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.369+0000 7fbc6affd640 1 -- 172.21.15.16:0/856833805 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc6c15b160 con 0x7fbc6c159cd0 2023-05-22T15:45:05.375 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.369+0000 7fbc6affd640 1 --2- 172.21.15.16:0/856833805 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc6c159cd0 0x7fbc6c15a0a0 secure :-1 s=READY pgs=1635 cs=0 l=1 rev1=1 crypto rx=0x7fbc5c0097c0 tx=0x7fbc5c01c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c275ef038f693978 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:45:05.375 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.369+0000 7fbc69ffb640 1 -- 172.21.15.16:0/856833805 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc5c0ad660 con 0x7fbc6c159cd0 2023-05-22T15:45:05.375 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.373+0000 7fbc69ffb640 1 -- 172.21.15.16:0/856833805 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc5c0ad7c0 con 0x7fbc6c159cd0 2023-05-22T15:45:05.375 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.373+0000 7fbc69ffb640 1 -- 172.21.15.16:0/856833805 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc5c0adb50 con 0x7fbc6c159cd0 2023-05-22T15:45:05.376 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.373+0000 7fbc714db640 1 -- 172.21.15.16:0/856833805 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc6c159cd0 msgr2=0x7fbc6c15a0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:45:05.376 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.373+0000 7fbc714db640 1 --2- 172.21.15.16:0/856833805 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc6c159cd0 0x7fbc6c15a0a0 secure :-1 s=READY pgs=1635 cs=0 l=1 rev1=1 crypto rx=0x7fbc5c0097c0 tx=0x7fbc5c01c450 comp rx=0 tx=0).stop 2023-05-22T15:45:05.376 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.373+0000 7fbc714db640 1 -- 172.21.15.16:0/856833805 shutdown_connections 2023-05-22T15:45:05.376 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.373+0000 7fbc714db640 1 --2- 172.21.15.16:0/856833805 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbc6c0cc6f0 0x7fbc6c0cfe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:05.376 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.373+0000 7fbc714db640 1 --2- 172.21.15.16:0/856833805 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc6c159cd0 0x7fbc6c15a0a0 unknown :-1 s=CLOSED pgs=1635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:05.376 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.373+0000 7fbc714db640 1 -- 172.21.15.16:0/856833805 >> 172.21.15.16:0/856833805 conn(0x7fbc6c06f5c0 msgr2=0x7fbc6c06f9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:45:05.376 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.373+0000 7fbc714db640 1 -- 172.21.15.16:0/856833805 shutdown_connections 2023-05-22T15:45:05.377 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.373+0000 7fbc714db640 1 -- 172.21.15.16:0/856833805 wait complete. 2023-05-22T15:45:05.378 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.373+0000 7fbc714db640 1 Processor -- start 2023-05-22T15:45:05.378 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.373+0000 7fbc714db640 1 -- start start 2023-05-22T15:45:05.378 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.373+0000 7fbc714db640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc6c0b2880 0x7fbc6c1476f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:05.379 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.373+0000 7fbc714db640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbc6c0cc6f0 0x7fbc6c14c1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:05.379 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.373+0000 7fbc714db640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbc6c159cd0 0x7fbc6c14d370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:05.379 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.373+0000 7fbc714db640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc6c1491f0 con 0x7fbc6c0b2880 2023-05-22T15:45:05.379 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.373+0000 7fbc714db640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc6c147c30 con 0x7fbc6c159cd0 2023-05-22T15:45:05.379 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.373+0000 7fbc714db640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc6c149680 con 0x7fbc6c0cc6f0 2023-05-22T15:45:05.379 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.373+0000 7fbc6affd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc6c0b2880 0x7fbc6c1476f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:05.379 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.373+0000 7fbc6affd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc6c0b2880 0x7fbc6c1476f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45182/0 (socket says 172.21.15.16:45182) 2023-05-22T15:45:05.380 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.373+0000 7fbc6affd640 1 -- 172.21.15.16:0/3829845170 learned_addr learned my addr 172.21.15.16:0/3829845170 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:45:05.380 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.373+0000 7fbc6b7fe640 1 --2- 172.21.15.16:0/3829845170 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbc6c159cd0 0x7fbc6c14d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:05.380 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.373+0000 7fbc6affd640 1 -- 172.21.15.16:0/3829845170 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbc6c0cc6f0 msgr2=0x7fbc6c14c1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:45:05.380 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.373+0000 7fbc6a7fc640 1 --2- 172.21.15.16:0/3829845170 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbc6c0cc6f0 0x7fbc6c14c1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:05.380 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.373+0000 7fbc6affd640 1 --2- 172.21.15.16:0/3829845170 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbc6c0cc6f0 0x7fbc6c14c1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:05.380 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.373+0000 7fbc6affd640 1 -- 172.21.15.16:0/3829845170 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbc6c159cd0 msgr2=0x7fbc6c14d370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:45:05.380 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.373+0000 7fbc6affd640 1 --2- 172.21.15.16:0/3829845170 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbc6c159cd0 0x7fbc6c14d370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:05.381 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.373+0000 7fbc6affd640 1 -- 172.21.15.16:0/3829845170 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc5c009420 con 0x7fbc6c0b2880 2023-05-22T15:45:05.381 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.373+0000 7fbc6a7fc640 1 --2- 172.21.15.16:0/3829845170 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbc6c0cc6f0 0x7fbc6c14c1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:45:05.381 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.373+0000 7fbc6b7fe640 1 --2- 172.21.15.16:0/3829845170 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbc6c159cd0 0x7fbc6c14d370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:45:05.381 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.373+0000 7fbc6affd640 1 --2- 172.21.15.16:0/3829845170 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc6c0b2880 0x7fbc6c1476f0 secure :-1 s=READY pgs=1636 cs=0 l=1 rev1=1 crypto rx=0x7fbc5c01a080 tx=0x7fbc5c01ac90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:45:05.381 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.373+0000 7fbc537fe640 1 -- 172.21.15.16:0/3829845170 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc5c004220 con 0x7fbc6c0b2880 2023-05-22T15:45:05.381 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.373+0000 7fbc537fe640 1 -- 172.21.15.16:0/3829845170 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc5c004380 con 0x7fbc6c0b2880 2023-05-22T15:45:05.381 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.373+0000 7fbc714db640 1 -- 172.21.15.16:0/3829845170 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc6c0d0d10 con 0x7fbc6c0b2880 2023-05-22T15:45:05.381 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.373+0000 7fbc537fe640 1 -- 172.21.15.16:0/3829845170 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc5c0b8870 con 0x7fbc6c0b2880 2023-05-22T15:45:05.382 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.373+0000 7fbc714db640 1 -- 172.21.15.16:0/3829845170 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc6c0d0b80 con 0x7fbc6c0b2880 2023-05-22T15:45:05.382 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.377+0000 7fbc714db640 1 -- 172.21.15.16:0/3829845170 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc6c0d0ea0 con 0x7fbc6c0b2880 2023-05-22T15:45:05.382 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.377+0000 7fbc537fe640 1 -- 172.21.15.16:0/3829845170 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 50) v1 ==== 84332+0+0 (secure 0 0 0) 0x7fbc5c0b89d0 con 0x7fbc6c0b2880 2023-05-22T15:45:05.383 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.377+0000 7fbc537fe640 1 --2- 172.21.15.16:0/3829845170 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fbc580654a0 0x7fbc58067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:05.383 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.377+0000 7fbc537fe640 1 -- 172.21.15.16:0/3829845170 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 6382+0+0 (secure 0 0 0) 0x7fbc5c11dd40 con 0x7fbc6c0b2880 2023-05-22T15:45:05.383 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.377+0000 7fbc6a7fc640 1 --2- 172.21.15.16:0/3829845170 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fbc580654a0 0x7fbc58067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:05.383 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.377+0000 7fbc6a7fc640 1 --2- 172.21.15.16:0/3829845170 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fbc580654a0 0x7fbc58067950 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fbc60007fd0 tx=0x7fbc60009040 comp rx=0 tx=0).ready entity=mgr.8304 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:45:05.384 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.381+0000 7fbc537fe640 1 -- 172.21.15.16:0/3829845170 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fbc5c0eb5c0 con 0x7fbc6c0b2880 2023-05-22T15:45:05.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.537+0000 7fbc714db640 1 -- 172.21.15.16:0/3829845170 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbc6c04dbe0 con 0x7fbc6c0b2880 2023-05-22T15:45:05.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.541+0000 7fbc537fe640 1 -- 172.21.15.16:0/3829845170 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v131) v1 ==== 74+0+18011 (secure 0 0 0) 0x7fbc5c0c5090 con 0x7fbc6c0b2880 2023-05-22T15:45:05.546 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:45:05.546 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":131,"fsid":"b2719834-7847-4d98-90a6-f65d57f85c49","created":"2023-05-22T15:26:25.259314+0000","modified":"2023-05-22T15:45:05.200265+0000","last_up_change":"2023-05-22T15:26:36.448524+0000","last_in_change":"2023-05-22T15:26:27.675668+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-05-22T15:26:38.019781+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-05-22T15:26:43.570514+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2023-05-22T15:45:03.718869+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"131","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2023-05-22T15:45:04.168135+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"131","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9e7ff854-b25f-4cbb-a5c2-1799d9c53d2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":37021},{"type":"v1","addr":"172.21.15.16:6825","nonce":37021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":37021},{"type":"v1","addr":"172.21.15.16:6827","nonce":37021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":37021},{"type":"v1","addr":"172.21.15.16:6831","nonce":37021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":37021},{"type":"v1","addr":"172.21.15.16:6829","nonce":37021}]},"public_addr":"172.21.15.16:6825/37021","cluster_addr":"172.21.15.16:6827/37021","heartbeat_back_addr":"172.21.15.16:6831/37021","heartbeat_front_addr":"172.21.15.16:6829/37021","state":["exists","up"]},{"osd":1,"uuid":"4702fbc9-a927-4565-aca5-41dc69cc8264","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":37008},{"type":"v1","addr":"172.21.15.16:6801","nonce":37008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":37008},{"type":"v1","addr":"172.21.15.16:6803","nonce":37008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":37008},{"type":"v1","addr":"172.21.15.16:6807","nonce":37008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":37008},{"type":"v1","addr":"172.21.15.16:6805","nonce":37008}]},"public_addr":"172.21.15.16:6801/37008","cluster_addr":"172.21.15.16:6803/37008","heartbeat_back_addr":"172.21.15.16:6807/37008","heartbeat_front_addr":"172.21.15.16:6805/37008","state":["exists","up"]},{"osd":2,"uuid":"1e7bba46-4eba-47ef-bb81-5e2d371638ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":37009},{"type":"v1","addr":"172.21.15.16:6809","nonce":37009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":37009},{"type":"v1","addr":"172.21.15.16:6811","nonce":37009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":37009},{"type":"v1","addr":"172.21.15.16:6815","nonce":37009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":37009},{"type":"v1","addr":"172.21.15.16:6813","nonce":37009}]},"public_addr":"172.21.15.16:6809/37009","cluster_addr":"172.21.15.16:6811/37009","heartbeat_back_addr":"172.21.15.16:6815/37009","heartbeat_front_addr":"172.21.15.16:6813/37009","state":["exists","up"]},{"osd":3,"uuid":"383cacc0-5a88-4904-990e-498ca59ef48a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":37020},{"type":"v1","addr":"172.21.15.16:6817","nonce":37020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":37020},{"type":"v1","addr":"172.21.15.16:6819","nonce":37020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":37020},{"type":"v1","addr":"172.21.15.16:6823","nonce":37020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":37020},{"type":"v1","addr":"172.21.15.16:6821","nonce":37020}]},"public_addr":"172.21.15.16:6817/37020","cluster_addr":"172.21.15.16:6819/37020","heartbeat_back_addr":"172.21.15.16:6823/37020","heartbeat_front_addr":"172.21.15.16:6821/37020","state":["exists","up"]},{"osd":4,"uuid":"970500ef-95b7-42c1-b080-eed4fae7013d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":37353},{"type":"v1","addr":"172.21.15.62:6817","nonce":37353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6819","nonce":37353},{"type":"v1","addr":"172.21.15.62:6822","nonce":37353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37353},{"type":"v1","addr":"172.21.15.62:6831","nonce":37353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37353},{"type":"v1","addr":"172.21.15.62:6826","nonce":37353}]},"public_addr":"172.21.15.62:6817/37353","cluster_addr":"172.21.15.62:6822/37353","heartbeat_back_addr":"172.21.15.62:6831/37353","heartbeat_front_addr":"172.21.15.62:6826/37353","state":["exists","up"]},{"osd":5,"uuid":"022d67db-d4d8-445b-b1b9-8b446a00f5c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":37352},{"type":"v1","addr":"172.21.15.62:6801","nonce":37352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":37352},{"type":"v1","addr":"172.21.15.62:6803","nonce":37352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":37352},{"type":"v1","addr":"172.21.15.62:6807","nonce":37352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":37352},{"type":"v1","addr":"172.21.15.62:6805","nonce":37352}]},"public_addr":"172.21.15.62:6801/37352","cluster_addr":"172.21.15.62:6803/37352","heartbeat_back_addr":"172.21.15.62:6807/37352","heartbeat_front_addr":"172.21.15.62:6805/37352","state":["exists","up"]},{"osd":6,"uuid":"42bd372b-6688-4a1b-8407-e16e8c266e33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":37354},{"type":"v1","addr":"172.21.15.62:6820","nonce":37354}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":37354},{"type":"v1","addr":"172.21.15.62:6823","nonce":37354}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6829","nonce":37354},{"type":"v1","addr":"172.21.15.62:6830","nonce":37354}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6825","nonce":37354},{"type":"v1","addr":"172.21.15.62:6827","nonce":37354}]},"public_addr":"172.21.15.62:6820/37354","cluster_addr":"172.21.15.62:6823/37354","heartbeat_back_addr":"172.21.15.62:6830/37354","heartbeat_front_addr":"172.21.15.62:6827/37354","state":["exists","up"]},{"osd":7,"uuid":"ac70b2a3-f41b-4bf2-b227-7f2cbb55426d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37355},{"type":"v1","addr":"172.21.15.62:6809","nonce":37355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37355},{"type":"v1","addr":"172.21.15.62:6811","nonce":37355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":37355},{"type":"v1","addr":"172.21.15.62:6815","nonce":37355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":37355},{"type":"v1","addr":"172.21.15.62:6813","nonce":37355}]},"public_addr":"172.21.15.62:6809/37355","cluster_addr":"172.21.15.62:6811/37355","heartbeat_back_addr":"172.21.15.62:6815/37355","heartbeat_front_addr":"172.21.15.62:6813/37355","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.368896+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.107273+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.276888+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.287502+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.124791+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.251937+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.255679+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.232797+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-05-22T15:45:05.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.541+0000 7fbc714db640 1 -- 172.21.15.16:0/3829845170 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fbc580654a0 msgr2=0x7fbc58067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:45:05.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.541+0000 7fbc714db640 1 --2- 172.21.15.16:0/3829845170 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fbc580654a0 0x7fbc58067950 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fbc60007fd0 tx=0x7fbc60009040 comp rx=0 tx=0).stop 2023-05-22T15:45:05.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.541+0000 7fbc714db640 1 -- 172.21.15.16:0/3829845170 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc6c0b2880 msgr2=0x7fbc6c1476f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:45:05.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.541+0000 7fbc714db640 1 --2- 172.21.15.16:0/3829845170 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc6c0b2880 0x7fbc6c1476f0 secure :-1 s=READY pgs=1636 cs=0 l=1 rev1=1 crypto rx=0x7fbc5c01a080 tx=0x7fbc5c01ac90 comp rx=0 tx=0).stop 2023-05-22T15:45:05.550 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.541+0000 7fbc714db640 1 -- 172.21.15.16:0/3829845170 shutdown_connections 2023-05-22T15:45:05.550 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.541+0000 7fbc714db640 1 --2- 172.21.15.16:0/3829845170 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbc6c159cd0 0x7fbc6c14d370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:05.550 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.541+0000 7fbc714db640 1 --2- 172.21.15.16:0/3829845170 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fbc580654a0 0x7fbc58067950 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:05.551 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.541+0000 7fbc714db640 1 --2- 172.21.15.16:0/3829845170 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbc6c0cc6f0 0x7fbc6c14c1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:05.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.541+0000 7fbc714db640 1 --2- 172.21.15.16:0/3829845170 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc6c0b2880 0x7fbc6c1476f0 unknown :-1 s=CLOSED pgs=1636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:05.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.541+0000 7fbc714db640 1 -- 172.21.15.16:0/3829845170 >> 172.21.15.16:0/3829845170 conn(0x7fbc6c06f5c0 msgr2=0x7fbc6c157920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:45:05.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.541+0000 7fbc714db640 1 -- 172.21.15.16:0/3829845170 shutdown_connections 2023-05-22T15:45:05.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.541+0000 7fbc714db640 1 -- 172.21.15.16:0/3829845170 wait complete. 2023-05-22T15:45:05.563 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-05-22T15:45:05.679 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.677+0000 7f13019b5640 1 Processor -- start 2023-05-22T15:45:05.679 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.677+0000 7f13019b5640 1 -- start start 2023-05-22T15:45:05.680 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.677+0000 7f13019b5640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f12fc15cfa0 0x7f12fc06db00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:05.680 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.677+0000 7f13019b5640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f12fc0cf870 0x7f12fc0b1fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:05.680 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.677+0000 7f13019b5640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12fc06e040 con 0x7f12fc06e740 2023-05-22T15:45:05.681 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.677+0000 7f13019b5640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12fc0cb9f0 con 0x7f12fc0cf870 2023-05-22T15:45:05.681 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.677+0000 7f13019b5640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12fc0b24f0 con 0x7f12fc15cfa0 2023-05-22T15:45:05.681 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.677+0000 7f12faffd640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f12fc15cfa0 0x7f12fc06db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:05.681 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.677+0000 7f12fa7fc640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f12fc0cf870 0x7f12fc0b1fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:05.681 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.677+0000 7f12fa7fc640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f12fc0cf870 0x7f12fc0b1fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.16:56380/0 (socket says 172.21.15.16:56380) 2023-05-22T15:45:05.681 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.677+0000 7f12fa7fc640 1 -- 172.21.15.16:0/1282113239 learned_addr learned my addr 172.21.15.16:0/1282113239 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:45:05.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.677+0000 7f12fa7fc640 1 -- 172.21.15.16:0/1282113239 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f12fc15cfa0 msgr2=0x7f12fc06db00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:45:05.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.677+0000 7f12fa7fc640 1 --2- 172.21.15.16:0/1282113239 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f12fc15cfa0 0x7f12fc06db00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:05.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.677+0000 7f12fa7fc640 1 -- 172.21.15.16:0/1282113239 >> v1:172.21.15.62:6789/0 conn(0x7f12fc06e740 legacy=0x7f12fc070de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:45:05.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.677+0000 7f12fa7fc640 1 -- 172.21.15.16:0/1282113239 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12fc06e490 con 0x7f12fc0cf870 2023-05-22T15:45:05.683 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.681+0000 7f12fa7fc640 1 --2- 172.21.15.16:0/1282113239 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f12fc0cf870 0x7f12fc0b1fb0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f12e4009870 tx=0x7f12e401c480 comp rx=0 tx=0).ready entity=mon.1 client_cookie=50cec8fa2c9f5fc0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:45:05.683 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.681+0000 7f12f9ffb640 1 -- 172.21.15.16:0/1282113239 <== mon.1 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12e40ab680 con 0x7f12fc0cf870 2023-05-22T15:45:05.684 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.681+0000 7f12f9ffb640 1 -- 172.21.15.16:0/1282113239 <== mon.1 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12e40ab7e0 con 0x7f12fc0cf870 2023-05-22T15:45:05.684 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.681+0000 7f12f9ffb640 1 -- 172.21.15.16:0/1282113239 <== mon.1 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12e40abb70 con 0x7f12fc0cf870 2023-05-22T15:45:05.684 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.681+0000 7f13019b5640 1 -- 172.21.15.16:0/1282113239 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f12fc0cf870 msgr2=0x7f12fc0b1fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:45:05.685 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.681+0000 7f13019b5640 1 --2- 172.21.15.16:0/1282113239 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f12fc0cf870 0x7f12fc0b1fb0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f12e4009870 tx=0x7f12e401c480 comp rx=0 tx=0).stop 2023-05-22T15:45:05.685 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.681+0000 7f13019b5640 1 -- 172.21.15.16:0/1282113239 shutdown_connections 2023-05-22T15:45:05.685 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.681+0000 7f13019b5640 1 --2- 172.21.15.16:0/1282113239 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f12fc0cf870 0x7f12fc0b1fb0 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:05.685 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.681+0000 7f13019b5640 1 --2- 172.21.15.16:0/1282113239 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f12fc15cfa0 0x7f12fc06db00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:05.685 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.681+0000 7f13019b5640 1 -- 172.21.15.16:0/1282113239 >> 172.21.15.16:0/1282113239 conn(0x7f12fc0cc870 msgr2=0x7f12fc0cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:45:05.685 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.681+0000 7f13019b5640 1 -- 172.21.15.16:0/1282113239 shutdown_connections 2023-05-22T15:45:05.685 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.681+0000 7f13019b5640 1 -- 172.21.15.16:0/1282113239 wait complete. 2023-05-22T15:45:05.690 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.681+0000 7f13019b5640 1 Processor -- start 2023-05-22T15:45:05.690 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.681+0000 7f13019b5640 1 -- start start 2023-05-22T15:45:05.690 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.685+0000 7f13019b5640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f12fc06e740 0x7f12fc14f030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:05.691 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.685+0000 7f13019b5640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f12fc0cf870 0x7f12fc156790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:05.691 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.685+0000 7f13019b5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f12fc15cfa0 0x7f12fc14a610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:05.691 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.685+0000 7f13019b5640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12fc156cd0 con 0x7f12fc15cfa0 2023-05-22T15:45:05.691 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.685+0000 7f13019b5640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12fc14ab50 con 0x7f12fc06e740 2023-05-22T15:45:05.691 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.685+0000 7f13019b5640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12fc14f6f0 con 0x7f12fc0cf870 2023-05-22T15:45:05.691 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.685+0000 7f12fb7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f12fc15cfa0 0x7f12fc14a610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:05.691 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.685+0000 7f12fb7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f12fc15cfa0 0x7f12fc14a610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45192/0 (socket says 172.21.15.16:45192) 2023-05-22T15:45:05.691 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.685+0000 7f12fb7fe640 1 -- 172.21.15.16:0/1671379394 learned_addr learned my addr 172.21.15.16:0/1671379394 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:45:05.692 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.685+0000 7f12faffd640 1 --2- 172.21.15.16:0/1671379394 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f12fc06e740 0x7f12fc14f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:05.692 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.685+0000 7f12fa7fc640 1 --2- 172.21.15.16:0/1671379394 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f12fc0cf870 0x7f12fc156790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:05.692 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.685+0000 7f12fb7fe640 1 -- 172.21.15.16:0/1671379394 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f12fc0cf870 msgr2=0x7f12fc156790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:45:05.692 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.685+0000 7f12fb7fe640 1 --2- 172.21.15.16:0/1671379394 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f12fc0cf870 0x7f12fc156790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:05.692 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.685+0000 7f12fb7fe640 1 -- 172.21.15.16:0/1671379394 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f12fc06e740 msgr2=0x7f12fc14f030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:45:05.692 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.685+0000 7f12fb7fe640 1 --2- 172.21.15.16:0/1671379394 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f12fc06e740 0x7f12fc14f030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:05.692 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.685+0000 7f12fb7fe640 1 -- 172.21.15.16:0/1671379394 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12e40094d0 con 0x7f12fc15cfa0 2023-05-22T15:45:05.693 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.685+0000 7f12fb7fe640 1 --2- 172.21.15.16:0/1671379394 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f12fc15cfa0 0x7f12fc14a610 secure :-1 s=READY pgs=1637 cs=0 l=1 rev1=1 crypto rx=0x7f12ec00c710 tx=0x7f12ec00cbd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:45:05.693 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.685+0000 7f12f3fff640 1 -- 172.21.15.16:0/1671379394 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12ec006d60 con 0x7f12fc15cfa0 2023-05-22T15:45:05.693 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.685+0000 7f12f3fff640 1 -- 172.21.15.16:0/1671379394 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12ec00ce20 con 0x7f12fc15cfa0 2023-05-22T15:45:05.693 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.685+0000 7f13019b5640 1 -- 172.21.15.16:0/1671379394 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12fc0cb610 con 0x7f12fc15cfa0 2023-05-22T15:45:05.693 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.685+0000 7f12f3fff640 1 -- 172.21.15.16:0/1671379394 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12ec003bc0 con 0x7f12fc15cfa0 2023-05-22T15:45:05.693 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.685+0000 7f13019b5640 1 -- 172.21.15.16:0/1671379394 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12fc0cb330 con 0x7f12fc15cfa0 2023-05-22T15:45:05.693 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.685+0000 7f13019b5640 1 -- 172.21.15.16:0/1671379394 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12fc071900 con 0x7f12fc15cfa0 2023-05-22T15:45:05.694 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.685+0000 7f12f3fff640 1 -- 172.21.15.16:0/1671379394 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 50) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f12ec003d20 con 0x7f12fc15cfa0 2023-05-22T15:45:05.694 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.685+0000 7f12f3fff640 1 --2- 172.21.15.16:0/1671379394 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f12dc0654a0 0x7f12dc067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:05.694 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.685+0000 7f12f3fff640 1 -- 172.21.15.16:0/1671379394 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f12ec082500 con 0x7f12fc15cfa0 2023-05-22T15:45:05.694 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.685+0000 7f12faffd640 1 --2- 172.21.15.16:0/1671379394 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f12dc0654a0 0x7f12dc067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:05.694 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.685+0000 7f12faffd640 1 --2- 172.21.15.16:0/1671379394 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f12dc0654a0 0x7f12dc067950 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f12f4009a50 tx=0x7f12f4008040 comp rx=0 tx=0).ready entity=mgr.8304 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:45:05.694 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.689+0000 7f12f3fff640 1 -- 172.21.15.16:0/1671379394 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f12ec0539f0 con 0x7f12fc15cfa0 2023-05-22T15:45:05.849 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.845+0000 7f13019b5640 1 -- 172.21.15.16:0/1671379394 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f12fc06e040 con 0x7f12fc15cfa0 2023-05-22T15:45:05.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.849+0000 7f12f3fff640 1 -- 172.21.15.16:0/1671379394 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v131) v1 ==== 74+0+18011 (secure 0 0 0) 0x7f12fc06e040 con 0x7f12fc15cfa0 2023-05-22T15:45:05.853 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:45:05.853 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":131,"fsid":"b2719834-7847-4d98-90a6-f65d57f85c49","created":"2023-05-22T15:26:25.259314+0000","modified":"2023-05-22T15:45:05.200265+0000","last_up_change":"2023-05-22T15:26:36.448524+0000","last_in_change":"2023-05-22T15:26:27.675668+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-05-22T15:26:38.019781+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-05-22T15:26:43.570514+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2023-05-22T15:45:03.718869+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"131","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2023-05-22T15:45:04.168135+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"131","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9e7ff854-b25f-4cbb-a5c2-1799d9c53d2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":37021},{"type":"v1","addr":"172.21.15.16:6825","nonce":37021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":37021},{"type":"v1","addr":"172.21.15.16:6827","nonce":37021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":37021},{"type":"v1","addr":"172.21.15.16:6831","nonce":37021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":37021},{"type":"v1","addr":"172.21.15.16:6829","nonce":37021}]},"public_addr":"172.21.15.16:6825/37021","cluster_addr":"172.21.15.16:6827/37021","heartbeat_back_addr":"172.21.15.16:6831/37021","heartbeat_front_addr":"172.21.15.16:6829/37021","state":["exists","up"]},{"osd":1,"uuid":"4702fbc9-a927-4565-aca5-41dc69cc8264","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":37008},{"type":"v1","addr":"172.21.15.16:6801","nonce":37008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":37008},{"type":"v1","addr":"172.21.15.16:6803","nonce":37008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":37008},{"type":"v1","addr":"172.21.15.16:6807","nonce":37008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":37008},{"type":"v1","addr":"172.21.15.16:6805","nonce":37008}]},"public_addr":"172.21.15.16:6801/37008","cluster_addr":"172.21.15.16:6803/37008","heartbeat_back_addr":"172.21.15.16:6807/37008","heartbeat_front_addr":"172.21.15.16:6805/37008","state":["exists","up"]},{"osd":2,"uuid":"1e7bba46-4eba-47ef-bb81-5e2d371638ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":37009},{"type":"v1","addr":"172.21.15.16:6809","nonce":37009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":37009},{"type":"v1","addr":"172.21.15.16:6811","nonce":37009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":37009},{"type":"v1","addr":"172.21.15.16:6815","nonce":37009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":37009},{"type":"v1","addr":"172.21.15.16:6813","nonce":37009}]},"public_addr":"172.21.15.16:6809/37009","cluster_addr":"172.21.15.16:6811/37009","heartbeat_back_addr":"172.21.15.16:6815/37009","heartbeat_front_addr":"172.21.15.16:6813/37009","state":["exists","up"]},{"osd":3,"uuid":"383cacc0-5a88-4904-990e-498ca59ef48a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":37020},{"type":"v1","addr":"172.21.15.16:6817","nonce":37020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":37020},{"type":"v1","addr":"172.21.15.16:6819","nonce":37020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":37020},{"type":"v1","addr":"172.21.15.16:6823","nonce":37020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":37020},{"type":"v1","addr":"172.21.15.16:6821","nonce":37020}]},"public_addr":"172.21.15.16:6817/37020","cluster_addr":"172.21.15.16:6819/37020","heartbeat_back_addr":"172.21.15.16:6823/37020","heartbeat_front_addr":"172.21.15.16:6821/37020","state":["exists","up"]},{"osd":4,"uuid":"970500ef-95b7-42c1-b080-eed4fae7013d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":37353},{"type":"v1","addr":"172.21.15.62:6817","nonce":37353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6819","nonce":37353},{"type":"v1","addr":"172.21.15.62:6822","nonce":37353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37353},{"type":"v1","addr":"172.21.15.62:6831","nonce":37353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37353},{"type":"v1","addr":"172.21.15.62:6826","nonce":37353}]},"public_addr":"172.21.15.62:6817/37353","cluster_addr":"172.21.15.62:6822/37353","heartbeat_back_addr":"172.21.15.62:6831/37353","heartbeat_front_addr":"172.21.15.62:6826/37353","state":["exists","up"]},{"osd":5,"uuid":"022d67db-d4d8-445b-b1b9-8b446a00f5c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":37352},{"type":"v1","addr":"172.21.15.62:6801","nonce":37352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":37352},{"type":"v1","addr":"172.21.15.62:6803","nonce":37352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":37352},{"type":"v1","addr":"172.21.15.62:6807","nonce":37352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":37352},{"type":"v1","addr":"172.21.15.62:6805","nonce":37352}]},"public_addr":"172.21.15.62:6801/37352","cluster_addr":"172.21.15.62:6803/37352","heartbeat_back_addr":"172.21.15.62:6807/37352","heartbeat_front_addr":"172.21.15.62:6805/37352","state":["exists","up"]},{"osd":6,"uuid":"42bd372b-6688-4a1b-8407-e16e8c266e33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":37354},{"type":"v1","addr":"172.21.15.62:6820","nonce":37354}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":37354},{"type":"v1","addr":"172.21.15.62:6823","nonce":37354}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6829","nonce":37354},{"type":"v1","addr":"172.21.15.62:6830","nonce":37354}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6825","nonce":37354},{"type":"v1","addr":"172.21.15.62:6827","nonce":37354}]},"public_addr":"172.21.15.62:6820/37354","cluster_addr":"172.21.15.62:6823/37354","heartbeat_back_addr":"172.21.15.62:6830/37354","heartbeat_front_addr":"172.21.15.62:6827/37354","state":["exists","up"]},{"osd":7,"uuid":"ac70b2a3-f41b-4bf2-b227-7f2cbb55426d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37355},{"type":"v1","addr":"172.21.15.62:6809","nonce":37355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37355},{"type":"v1","addr":"172.21.15.62:6811","nonce":37355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":37355},{"type":"v1","addr":"172.21.15.62:6815","nonce":37355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":37355},{"type":"v1","addr":"172.21.15.62:6813","nonce":37355}]},"public_addr":"172.21.15.62:6809/37355","cluster_addr":"172.21.15.62:6811/37355","heartbeat_back_addr":"172.21.15.62:6815/37355","heartbeat_front_addr":"172.21.15.62:6813/37355","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.368896+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.107273+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.276888+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.287502+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.124791+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.251937+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.255679+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.232797+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-05-22T15:45:05.854 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.849+0000 7f13019b5640 1 -- 172.21.15.16:0/1671379394 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f12dc0654a0 msgr2=0x7f12dc067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:45:05.854 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.849+0000 7f13019b5640 1 --2- 172.21.15.16:0/1671379394 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f12dc0654a0 0x7f12dc067950 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f12f4009a50 tx=0x7f12f4008040 comp rx=0 tx=0).stop 2023-05-22T15:45:05.854 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.849+0000 7f13019b5640 1 -- 172.21.15.16:0/1671379394 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f12fc15cfa0 msgr2=0x7f12fc14a610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:45:05.854 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.849+0000 7f13019b5640 1 --2- 172.21.15.16:0/1671379394 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f12fc15cfa0 0x7f12fc14a610 secure :-1 s=READY pgs=1637 cs=0 l=1 rev1=1 crypto rx=0x7f12ec00c710 tx=0x7f12ec00cbd0 comp rx=0 tx=0).stop 2023-05-22T15:45:05.855 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.849+0000 7f13019b5640 1 -- 172.21.15.16:0/1671379394 shutdown_connections 2023-05-22T15:45:05.855 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.849+0000 7f13019b5640 1 --2- 172.21.15.16:0/1671379394 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f12fc15cfa0 0x7f12fc14a610 unknown :-1 s=CLOSED pgs=1637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:05.856 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.849+0000 7f13019b5640 1 --2- 172.21.15.16:0/1671379394 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f12dc0654a0 0x7f12dc067950 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:05.856 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.849+0000 7f13019b5640 1 --2- 172.21.15.16:0/1671379394 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f12fc0cf870 0x7f12fc156790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:05.856 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.849+0000 7f13019b5640 1 --2- 172.21.15.16:0/1671379394 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f12fc06e740 0x7f12fc14f030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:05.856 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.849+0000 7f13019b5640 1 -- 172.21.15.16:0/1671379394 >> 172.21.15.16:0/1671379394 conn(0x7f12fc0cc870 msgr2=0x7f12fc0aebc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:45:05.857 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.849+0000 7f13019b5640 1 -- 172.21.15.16:0/1671379394 shutdown_connections 2023-05-22T15:45:05.857 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.849+0000 7f13019b5640 1 -- 172.21.15.16:0/1671379394 wait complete. 2023-05-22T15:45:05.872 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-05-22T15:45:05.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.989+0000 7f26f4e11640 1 Processor -- start 2023-05-22T15:45:05.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.989+0000 7f26f4e11640 1 -- start start 2023-05-22T15:45:05.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.993+0000 7f26f4e11640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f26f015bca0 0x7f26f00cda80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:05.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.993+0000 7f26f4e11640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f26f00b5a50 0x7f26f00d2900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:05.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.993+0000 7f26f4e11640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26f00cccc0 con 0x7f26f00d2480 2023-05-22T15:45:05.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.993+0000 7f26f4e11640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26f00cd660 con 0x7f26f015bca0 2023-05-22T15:45:05.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.993+0000 7f26f4e11640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26f00d2e40 con 0x7f26f00b5a50 2023-05-22T15:45:05.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.993+0000 7f26ee59c640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f26f015bca0 0x7f26f00cda80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:05.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.993+0000 7f26ee59c640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f26f015bca0 0x7f26f00cda80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45206/0 (socket says 172.21.15.16:45206) 2023-05-22T15:45:05.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.993+0000 7f26ee59c640 1 -- 172.21.15.16:0/2418321003 learned_addr learned my addr 172.21.15.16:0/2418321003 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:45:05.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.993+0000 7f26eed9d640 1 --2- 172.21.15.16:0/2418321003 >> v2:172.21.15.62:3300/0 conn(0x7f26f00b5a50 0x7f26f00d2900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:05.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.993+0000 7f26ee59c640 1 -- 172.21.15.16:0/2418321003 >> v2:172.21.15.62:3300/0 conn(0x7f26f00b5a50 msgr2=0x7f26f00d2900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:45:05.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.993+0000 7f26ee59c640 1 --2- 172.21.15.16:0/2418321003 >> v2:172.21.15.62:3300/0 conn(0x7f26f00b5a50 0x7f26f00d2900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:05.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.993+0000 7f26ee59c640 1 -- 172.21.15.16:0/2418321003 >> v1:172.21.15.62:6789/0 conn(0x7f26f00d2480 legacy=0x7f26f00b5340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:45:05.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.993+0000 7f26ee59c640 1 -- 172.21.15.16:0/2418321003 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26f00ccfa0 con 0x7f26f015bca0 2023-05-22T15:45:05.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.993+0000 7f26ee59c640 1 --2- 172.21.15.16:0/2418321003 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f26f015bca0 0x7f26f00cda80 secure :-1 s=READY pgs=1638 cs=0 l=1 rev1=1 crypto rx=0x7f26f00c5fc0 tx=0x7f26e001c4c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4e891b12dcb6d1a server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:45:05.998 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.993+0000 7f26ed59a640 1 -- 172.21.15.16:0/2418321003 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26e00aa640 con 0x7f26f015bca0 2023-05-22T15:45:05.998 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.993+0000 7f26ed59a640 1 -- 172.21.15.16:0/2418321003 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26e00aa7a0 con 0x7f26f015bca0 2023-05-22T15:45:05.998 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.993+0000 7f26ed59a640 1 -- 172.21.15.16:0/2418321003 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26e00aab30 con 0x7f26f015bca0 2023-05-22T15:45:05.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.993+0000 7f26f4e11640 1 -- 172.21.15.16:0/2418321003 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f26f015bca0 msgr2=0x7f26f00cda80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:45:05.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.993+0000 7f26f4e11640 1 --2- 172.21.15.16:0/2418321003 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f26f015bca0 0x7f26f00cda80 secure :-1 s=READY pgs=1638 cs=0 l=1 rev1=1 crypto rx=0x7f26f00c5fc0 tx=0x7f26e001c4c0 comp rx=0 tx=0).stop 2023-05-22T15:45:05.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.993+0000 7f26f4e11640 1 -- 172.21.15.16:0/2418321003 shutdown_connections 2023-05-22T15:45:05.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.993+0000 7f26f4e11640 1 --2- 172.21.15.16:0/2418321003 >> v2:172.21.15.62:3300/0 conn(0x7f26f00b5a50 0x7f26f00d2900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:05.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.993+0000 7f26f4e11640 1 --2- 172.21.15.16:0/2418321003 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f26f015bca0 0x7f26f00cda80 unknown :-1 s=CLOSED pgs=1638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:05.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.993+0000 7f26f4e11640 1 -- 172.21.15.16:0/2418321003 >> 172.21.15.16:0/2418321003 conn(0x7f26f00655c0 msgr2=0x7f26f0070e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:45:06.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.997+0000 7f26f4e11640 1 -- 172.21.15.16:0/2418321003 shutdown_connections 2023-05-22T15:45:06.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.997+0000 7f26f4e11640 1 -- 172.21.15.16:0/2418321003 wait complete. 2023-05-22T15:45:06.002 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.997+0000 7f26f4e11640 1 Processor -- start 2023-05-22T15:45:06.002 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.997+0000 7f26f4e11640 1 -- start start 2023-05-22T15:45:06.002 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.997+0000 7f26f4e11640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f26f00b5a50 0x7f26f0153e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:06.003 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.997+0000 7f26f4e11640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f26f00d2480 0x7f26f0154680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:06.003 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.997+0000 7f26f4e11640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f26f015bca0 0x7f26f00d0030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:06.004 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.997+0000 7f26f4e11640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26f00cbf40 con 0x7f26f00b5a50 2023-05-22T15:45:06.004 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.997+0000 7f26f4e11640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26f00cb6a0 con 0x7f26f00d2480 2023-05-22T15:45:06.004 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.997+0000 7f26f4e11640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26f00cb980 con 0x7f26f015bca0 2023-05-22T15:45:06.004 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.997+0000 7f26ee59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f26f00b5a50 0x7f26f0153e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:06.004 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.997+0000 7f26ee59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f26f00b5a50 0x7f26f0153e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45220/0 (socket says 172.21.15.16:45220) 2023-05-22T15:45:06.004 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.997+0000 7f26ee59c640 1 -- 172.21.15.16:0/2285687985 learned_addr learned my addr 172.21.15.16:0/2285687985 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:45:06.004 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.997+0000 7f26edd9b640 1 --2- 172.21.15.16:0/2285687985 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f26f00d2480 0x7f26f0154680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:06.005 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.997+0000 7f26eed9d640 1 --2- 172.21.15.16:0/2285687985 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f26f015bca0 0x7f26f00d0030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:06.005 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.997+0000 7f26ee59c640 1 -- 172.21.15.16:0/2285687985 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f26f015bca0 msgr2=0x7f26f00d0030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:45:06.005 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.997+0000 7f26ee59c640 1 --2- 172.21.15.16:0/2285687985 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f26f015bca0 0x7f26f00d0030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:06.005 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.997+0000 7f26ee59c640 1 -- 172.21.15.16:0/2285687985 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f26f00d2480 msgr2=0x7f26f0154680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:45:06.005 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.997+0000 7f26ee59c640 1 --2- 172.21.15.16:0/2285687985 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f26f00d2480 0x7f26f0154680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:06.005 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.997+0000 7f26ee59c640 1 -- 172.21.15.16:0/2285687985 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26e00094d0 con 0x7f26f00b5a50 2023-05-22T15:45:06.005 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.997+0000 7f26ee59c640 1 --2- 172.21.15.16:0/2285687985 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f26f00b5a50 0x7f26f0153e30 secure :-1 s=READY pgs=1639 cs=0 l=1 rev1=1 crypto rx=0x7f26e001a0f0 tx=0x7f26e001ad00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:45:06.006 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.997+0000 7f26deffd640 1 -- 172.21.15.16:0/2285687985 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26e00bd5f0 con 0x7f26f00b5a50 2023-05-22T15:45:06.006 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.997+0000 7f26deffd640 1 -- 172.21.15.16:0/2285687985 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26e00bd750 con 0x7f26f00b5a50 2023-05-22T15:45:06.006 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.997+0000 7f26f4e11640 1 -- 172.21.15.16:0/2285687985 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26f0160360 con 0x7f26f00b5a50 2023-05-22T15:45:06.006 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.997+0000 7f26deffd640 1 -- 172.21.15.16:0/2285687985 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26e00bdb00 con 0x7f26f00b5a50 2023-05-22T15:45:06.006 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:05.997+0000 7f26f4e11640 1 -- 172.21.15.16:0/2285687985 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26f00cdfc0 con 0x7f26f00b5a50 2023-05-22T15:45:06.007 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:06.001+0000 7f26f4e11640 1 -- 172.21.15.16:0/2285687985 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26f01604f0 con 0x7f26f00b5a50 2023-05-22T15:45:06.007 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:06.001+0000 7f26deffd640 1 -- 172.21.15.16:0/2285687985 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 50) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f26e00a8050 con 0x7f26f00b5a50 2023-05-22T15:45:06.008 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:06.001+0000 7f26deffd640 1 --2- 172.21.15.16:0/2285687985 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f26d00654a0 0x7f26d0067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:06.008 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:06.001+0000 7f26deffd640 1 -- 172.21.15.16:0/2285687985 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f26e011d800 con 0x7f26f00b5a50 2023-05-22T15:45:06.008 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:06.001+0000 7f26edd9b640 1 --2- 172.21.15.16:0/2285687985 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f26d00654a0 0x7f26d0067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:06.008 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:06.001+0000 7f26edd9b640 1 --2- 172.21.15.16:0/2285687985 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f26d00654a0 0x7f26d0067950 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f26f015b230 tx=0x7f26e4009040 comp rx=0 tx=0).ready entity=mgr.8304 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:45:06.008 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:06.005+0000 7f26deffd640 1 -- 172.21.15.16:0/2285687985 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f26e00eb200 con 0x7f26f00b5a50 2023-05-22T15:45:06.194 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:06.189+0000 7f26f4e11640 1 -- 172.21.15.16:0/2285687985 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f26f004dbe0 con 0x7f26f00b5a50 2023-05-22T15:45:06.697 INFO:tasks.ceph.mds.a.smithi016.stderr:2023-05-22T15:45:06.693+0000 7fa0fae19640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-05-22T15:45:06.839 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:06.837+0000 7f26deffd640 1 -- 172.21.15.16:0/2285687985 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v49) v1 ==== 118+0+0 (secure 0 0 0) 0x7f26e00a6020 con 0x7f26f00b5a50 2023-05-22T15:45:06.842 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:06.837+0000 7f26f4e11640 1 -- 172.21.15.16:0/2285687985 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f26d00654a0 msgr2=0x7f26d0067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:45:06.843 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:06.837+0000 7f26f4e11640 1 --2- 172.21.15.16:0/2285687985 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f26d00654a0 0x7f26d0067950 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f26f015b230 tx=0x7f26e4009040 comp rx=0 tx=0).stop 2023-05-22T15:45:06.843 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:06.837+0000 7f26f4e11640 1 -- 172.21.15.16:0/2285687985 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f26f00b5a50 msgr2=0x7f26f0153e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:45:06.843 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:06.837+0000 7f26f4e11640 1 --2- 172.21.15.16:0/2285687985 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f26f00b5a50 0x7f26f0153e30 secure :-1 s=READY pgs=1639 cs=0 l=1 rev1=1 crypto rx=0x7f26e001a0f0 tx=0x7f26e001ad00 comp rx=0 tx=0).stop 2023-05-22T15:45:06.843 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:06.841+0000 7f26f4e11640 1 -- 172.21.15.16:0/2285687985 shutdown_connections 2023-05-22T15:45:06.843 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:06.841+0000 7f26f4e11640 1 --2- 172.21.15.16:0/2285687985 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f26d00654a0 0x7f26d0067950 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:06.843 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:06.841+0000 7f26f4e11640 1 --2- 172.21.15.16:0/2285687985 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f26f015bca0 0x7f26f00d0030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:06.844 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:06.841+0000 7f26f4e11640 1 --2- 172.21.15.16:0/2285687985 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f26f00d2480 0x7f26f0154680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:06.844 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:06.841+0000 7f26f4e11640 1 --2- 172.21.15.16:0/2285687985 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f26f00b5a50 0x7f26f0153e30 unknown :-1 s=CLOSED pgs=1639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:06.844 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:06.841+0000 7f26f4e11640 1 -- 172.21.15.16:0/2285687985 >> 172.21.15.16:0/2285687985 conn(0x7f26f00655c0 msgr2=0x7f26f00b5320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:45:06.844 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:06.841+0000 7f26f4e11640 1 -- 172.21.15.16:0/2285687985 shutdown_connections 2023-05-22T15:45:06.844 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:06.841+0000 7f26f4e11640 1 -- 172.21.15.16:0/2285687985 wait complete. 2023-05-22T15:45:06.865 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2023-05-22T15:45:06.866 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-05-22T15:45:06.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:06.993+0000 7f918efb8640 1 Processor -- start 2023-05-22T15:45:06.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:06.993+0000 7f918efb8640 1 -- start start 2023-05-22T15:45:06.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:06.993+0000 7f918efb8640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f9188159970 0x7f9188159d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:06.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:06.993+0000 7f918efb8640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f918815db00 0x7f918806b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:06.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:06.993+0000 7f918efb8640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9188070c30 con 0x7f918815d680 2023-05-22T15:45:06.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:06.993+0000 7f918efb8640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91880cb970 con 0x7f9188159970 2023-05-22T15:45:06.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:06.993+0000 7f918efb8640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9188070e60 con 0x7f918815db00 2023-05-22T15:45:06.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:06.993+0000 7f918cd54640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f9188159970 0x7f9188159d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:06.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:06.993+0000 7f918cd54640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f9188159970 0x7f9188159d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45232/0 (socket says 172.21.15.16:45232) 2023-05-22T15:45:06.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:06.993+0000 7f918cd54640 1 -- 172.21.15.16:0/1135054940 learned_addr learned my addr 172.21.15.16:0/1135054940 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:45:06.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:06.993+0000 7f918d555640 1 --2- 172.21.15.16:0/1135054940 >> v2:172.21.15.62:3300/0 conn(0x7f918815db00 0x7f918806b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:06.998 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:06.993+0000 7f918cd54640 1 -- 172.21.15.16:0/1135054940 >> v2:172.21.15.62:3300/0 conn(0x7f918815db00 msgr2=0x7f918806b6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:45:06.998 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:06.993+0000 7f918cd54640 1 --2- 172.21.15.16:0/1135054940 >> v2:172.21.15.62:3300/0 conn(0x7f918815db00 0x7f918806b6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:06.998 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:06.993+0000 7f918cd54640 1 -- 172.21.15.16:0/1135054940 >> v1:172.21.15.62:6789/0 conn(0x7f918815d680 legacy=0x7f91880cb260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:45:06.998 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:06.993+0000 7f918cd54640 1 -- 172.21.15.16:0/1135054940 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9188071140 con 0x7f9188159970 2023-05-22T15:45:07.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:06.997+0000 7f918cd54640 1 --2- 172.21.15.16:0/1135054940 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9188159970 0x7f9188159d40 secure :-1 s=READY pgs=1640 cs=0 l=1 rev1=1 crypto rx=0x7f9188070770 tx=0x7f917801c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=361b8e77bde2e8a server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:45:07.001 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:06.997+0000 7f91877fe640 1 -- 172.21.15.16:0/1135054940 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91780a56a0 con 0x7f9188159970 2023-05-22T15:45:07.001 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:06.997+0000 7f91877fe640 1 -- 172.21.15.16:0/1135054940 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91780a5800 con 0x7f9188159970 2023-05-22T15:45:07.001 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:06.997+0000 7f91877fe640 1 -- 172.21.15.16:0/1135054940 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91780a5b90 con 0x7f9188159970 2023-05-22T15:45:07.001 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:06.997+0000 7f918efb8640 1 -- 172.21.15.16:0/1135054940 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9188159970 msgr2=0x7f9188159d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:45:07.001 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:06.997+0000 7f918efb8640 1 --2- 172.21.15.16:0/1135054940 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9188159970 0x7f9188159d40 secure :-1 s=READY pgs=1640 cs=0 l=1 rev1=1 crypto rx=0x7f9188070770 tx=0x7f917801c8d0 comp rx=0 tx=0).stop 2023-05-22T15:45:07.001 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:06.997+0000 7f918efb8640 1 -- 172.21.15.16:0/1135054940 shutdown_connections 2023-05-22T15:45:07.002 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:06.997+0000 7f918efb8640 1 --2- 172.21.15.16:0/1135054940 >> v2:172.21.15.62:3300/0 conn(0x7f918815db00 0x7f918806b6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:07.002 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:06.997+0000 7f918efb8640 1 --2- 172.21.15.16:0/1135054940 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9188159970 0x7f9188159d40 unknown :-1 s=CLOSED pgs=1640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:07.002 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:06.997+0000 7f918efb8640 1 -- 172.21.15.16:0/1135054940 >> 172.21.15.16:0/1135054940 conn(0x7f9188069720 msgr2=0x7f9188069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:45:07.002 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:06.997+0000 7f918efb8640 1 -- 172.21.15.16:0/1135054940 shutdown_connections 2023-05-22T15:45:07.002 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:06.997+0000 7f918efb8640 1 -- 172.21.15.16:0/1135054940 wait complete. 2023-05-22T15:45:07.003 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:06.997+0000 7f918efb8640 1 Processor -- start 2023-05-22T15:45:07.004 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:06.997+0000 7f918efb8640 1 -- start start 2023-05-22T15:45:07.004 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:07.001+0000 7f918efb8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9188159970 0x7f91880c2a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:07.004 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:07.001+0000 7f918efb8640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f918815d680 0x7f91881544c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:07.004 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:07.001+0000 7f918efb8640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f918815db00 0x7f9188069af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:07.004 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:07.001+0000 7f918efb8640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91880c2f60 con 0x7f9188159970 2023-05-22T15:45:07.004 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:07.001+0000 7f918efb8640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91880c7950 con 0x7f918815db00 2023-05-22T15:45:07.005 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:07.001+0000 7f918efb8640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91880c5210 con 0x7f918815d680 2023-05-22T15:45:07.005 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:07.001+0000 7f918cd54640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9188159970 0x7f91880c2a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:07.005 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:07.001+0000 7f918cd54640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9188159970 0x7f91880c2a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45242/0 (socket says 172.21.15.16:45242) 2023-05-22T15:45:07.005 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:07.001+0000 7f918cd54640 1 -- 172.21.15.16:0/2344431279 learned_addr learned my addr 172.21.15.16:0/2344431279 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:45:07.005 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:07.001+0000 7f918d555640 1 --2- 172.21.15.16:0/2344431279 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f918815db00 0x7f9188069af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:07.005 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:07.001+0000 7f9187fff640 1 --2- 172.21.15.16:0/2344431279 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f918815d680 0x7f91881544c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:07.005 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:07.001+0000 7f918cd54640 1 -- 172.21.15.16:0/2344431279 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f918815d680 msgr2=0x7f91881544c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:45:07.005 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:07.001+0000 7f918cd54640 1 --2- 172.21.15.16:0/2344431279 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f918815d680 0x7f91881544c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:07.006 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:07.001+0000 7f918cd54640 1 -- 172.21.15.16:0/2344431279 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f918815db00 msgr2=0x7f9188069af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:45:07.006 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:07.001+0000 7f918cd54640 1 --2- 172.21.15.16:0/2344431279 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f918815db00 0x7f9188069af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:07.006 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:07.001+0000 7f918cd54640 1 -- 172.21.15.16:0/2344431279 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9178009b30 con 0x7f9188159970 2023-05-22T15:45:07.006 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:07.001+0000 7f918d555640 1 --2- 172.21.15.16:0/2344431279 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f918815db00 0x7f9188069af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:45:07.006 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:07.001+0000 7f918cd54640 1 --2- 172.21.15.16:0/2344431279 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9188159970 0x7f91880c2a20 secure :-1 s=READY pgs=1641 cs=0 l=1 rev1=1 crypto rx=0x7f917801a500 tx=0x7f9178019670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:45:07.006 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:07.001+0000 7f91857fa640 1 -- 172.21.15.16:0/2344431279 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91780b8670 con 0x7f9188159970 2023-05-22T15:45:07.006 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:07.001+0000 7f91857fa640 1 -- 172.21.15.16:0/2344431279 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91780b87d0 con 0x7f9188159970 2023-05-22T15:45:07.007 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:07.001+0000 7f91857fa640 1 -- 172.21.15.16:0/2344431279 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91780b8b80 con 0x7f9188159970 2023-05-22T15:45:07.007 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:07.001+0000 7f918efb8640 1 -- 172.21.15.16:0/2344431279 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91880cae20 con 0x7f9188159970 2023-05-22T15:45:07.007 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:07.001+0000 7f918efb8640 1 -- 172.21.15.16:0/2344431279 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91880cac90 con 0x7f9188159970 2023-05-22T15:45:07.007 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:07.001+0000 7f918efb8640 1 -- 172.21.15.16:0/2344431279 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f918815adc0 con 0x7f9188159970 2023-05-22T15:45:07.008 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:07.001+0000 7f91857fa640 1 -- 172.21.15.16:0/2344431279 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 50) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f91780b8ce0 con 0x7f9188159970 2023-05-22T15:45:07.008 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:07.001+0000 7f91857fa640 1 --2- 172.21.15.16:0/2344431279 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f91740654a0 0x7f9174067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:07.008 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:07.001+0000 7f9187fff640 1 --2- 172.21.15.16:0/2344431279 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f91740654a0 0x7f9174067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:07.008 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:07.001+0000 7f91857fa640 1 -- 172.21.15.16:0/2344431279 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f917811e000 con 0x7f9188159970 2023-05-22T15:45:07.008 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:07.001+0000 7f9187fff640 1 --2- 172.21.15.16:0/2344431279 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f91740654a0 0x7f9174067950 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f91880c27d0 tx=0x7f917c009040 comp rx=0 tx=0).ready entity=mgr.8304 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:45:07.010 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:07.005+0000 7f91857fa640 1 -- 172.21.15.16:0/2344431279 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f91780ebac0 con 0x7f9188159970 2023-05-22T15:45:07.189 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:07.185+0000 7f918efb8640 1 -- 172.21.15.16:0/2344431279 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f918804dbe0 con 0x7f9188159970 2023-05-22T15:45:07.842 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:07.837+0000 7f91857fa640 1 -- 172.21.15.16:0/2344431279 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v50) v1 ==== 122+0+0 (secure 0 0 0) 0x7f91780ef0e0 con 0x7f9188159970 2023-05-22T15:45:07.845 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:07.841+0000 7f918efb8640 1 -- 172.21.15.16:0/2344431279 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f91740654a0 msgr2=0x7f9174067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:45:07.845 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:07.841+0000 7f918efb8640 1 --2- 172.21.15.16:0/2344431279 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f91740654a0 0x7f9174067950 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f91880c27d0 tx=0x7f917c009040 comp rx=0 tx=0).stop 2023-05-22T15:45:07.845 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:07.841+0000 7f918efb8640 1 -- 172.21.15.16:0/2344431279 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9188159970 msgr2=0x7f91880c2a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:45:07.845 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:07.841+0000 7f918efb8640 1 --2- 172.21.15.16:0/2344431279 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9188159970 0x7f91880c2a20 secure :-1 s=READY pgs=1641 cs=0 l=1 rev1=1 crypto rx=0x7f917801a500 tx=0x7f9178019670 comp rx=0 tx=0).stop 2023-05-22T15:45:07.846 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:07.841+0000 7f918efb8640 1 -- 172.21.15.16:0/2344431279 shutdown_connections 2023-05-22T15:45:07.846 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:07.841+0000 7f918efb8640 1 --2- 172.21.15.16:0/2344431279 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f918815db00 0x7f9188069af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:07.846 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:07.841+0000 7f918efb8640 1 --2- 172.21.15.16:0/2344431279 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f91740654a0 0x7f9174067950 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:07.846 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:07.841+0000 7f918efb8640 1 --2- 172.21.15.16:0/2344431279 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f918815d680 0x7f91881544c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:07.846 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:07.841+0000 7f918efb8640 1 --2- 172.21.15.16:0/2344431279 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9188159970 0x7f91880c2a20 unknown :-1 s=CLOSED pgs=1641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:07.846 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:07.841+0000 7f918efb8640 1 -- 172.21.15.16:0/2344431279 >> 172.21.15.16:0/2344431279 conn(0x7f9188069720 msgr2=0x7f91880c3e30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:45:07.846 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:07.841+0000 7f918efb8640 1 -- 172.21.15.16:0/2344431279 shutdown_connections 2023-05-22T15:45:07.846 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:07.841+0000 7f918efb8640 1 -- 172.21.15.16:0/2344431279 wait complete. 2023-05-22T15:45:07.866 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2023-05-22T15:45:07.866 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-05-22T15:45:07.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:07.989+0000 7f5062397640 1 Processor -- start 2023-05-22T15:45:07.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:07.989+0000 7f5062397640 1 -- start start 2023-05-22T15:45:07.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:07.993+0000 7f5062397640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f505c15c920 0x7f505c0d0030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:07.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:07.993+0000 7f5062397640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f505c15c490 con 0x7f505c15cd70 2023-05-22T15:45:07.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:07.993+0000 7f5062397640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f505c15c140 con 0x7f505c15fc50 2023-05-22T15:45:07.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:07.993+0000 7f5062397640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f505c0ccb50 con 0x7f505c15c920 2023-05-22T15:45:07.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:07.993+0000 7f5060934640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f505c15cd70 0x7f505c0ce790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:44908/0 (socket says 172.21.15.16:44908) 2023-05-22T15:45:07.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:07.993+0000 7f5060934640 1 -- 172.21.15.16:0/3253390117 learned_addr learned my addr 172.21.15.16:0/3253390117 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:45:07.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:07.993+0000 7f5053fff640 1 --2- 172.21.15.16:0/3253390117 >> v2:172.21.15.62:3300/0 conn(0x7f505c15c920 0x7f505c0d0030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:07.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:07.993+0000 7f505b7fe640 1 -- 172.21.15.16:0/3253390117 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3423357020 0 0) 0x7f505c15c490 con 0x7f505c15cd70 2023-05-22T15:45:07.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:07.993+0000 7f505b7fe640 1 -- 172.21.15.16:0/3253390117 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5048003530 con 0x7f505c15cd70 2023-05-22T15:45:07.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:07.993+0000 7f505b7fe640 1 -- 172.21.15.16:0/3253390117 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 296084205 0 0) 0x7f5048003530 con 0x7f505c15cd70 2023-05-22T15:45:07.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:07.993+0000 7f505b7fe640 1 -- 172.21.15.16:0/3253390117 >> v2:172.21.15.62:3300/0 conn(0x7f505c15c920 msgr2=0x7f505c0d0030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:45:07.998 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:07.993+0000 7f505b7fe640 1 --2- 172.21.15.16:0/3253390117 >> v2:172.21.15.62:3300/0 conn(0x7f505c15c920 0x7f505c0d0030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:07.998 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:07.993+0000 7f505b7fe640 1 -- 172.21.15.16:0/3253390117 >> v1:172.21.15.62:6789/0 conn(0x7f505c15fc50 legacy=0x7f505c160020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:45:07.998 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:07.993+0000 7f505b7fe640 1 -- 172.21.15.16:0/3253390117 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f505c0cce30 con 0x7f505c15cd70 2023-05-22T15:45:07.998 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:07.993+0000 7f505b7fe640 1 -- 172.21.15.16:0/3253390117 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7f5054001a00 con 0x7f505c15cd70 2023-05-22T15:45:07.998 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:07.993+0000 7f505b7fe640 1 -- 172.21.15.16:0/3253390117 >> v1:172.21.15.16:6789/0 conn(0x7f505c15cd70 legacy=0x7f505c0ce790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:45:07.998 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:07.993+0000 7f505b7fe640 1 --2- 172.21.15.16:0/3253390117 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f50480039d0 0x7f5048003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:07.998 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:07.993+0000 7f505b7fe640 1 --2- 172.21.15.16:0/3253390117 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f50480047d0 0x7f5048004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:07.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:07.993+0000 7f505b7fe640 1 --2- 172.21.15.16:0/3253390117 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5048005180 0x7f504800b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:07.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:07.993+0000 7f505b7fe640 1 -- 172.21.15.16:0/3253390117 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f505c0ccb50 con 0x7f5048005180 2023-05-22T15:45:07.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:07.993+0000 7f505b7fe640 1 -- 172.21.15.16:0/3253390117 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f504800c060 con 0x7f50480039d0 2023-05-22T15:45:07.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:07.993+0000 7f505b7fe640 1 -- 172.21.15.16:0/3253390117 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f504800c220 con 0x7f50480047d0 2023-05-22T15:45:07.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:07.993+0000 7f5060934640 1 --2- 172.21.15.16:0/3253390117 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5048005180 0x7f504800b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:08.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:07.993+0000 7f505bfff640 1 --2- 172.21.15.16:0/3253390117 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f50480039d0 0x7f5048003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:08.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:07.993+0000 7f5053fff640 1 --2- 172.21.15.16:0/3253390117 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f50480047d0 0x7f5048004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:08.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:07.993+0000 7f5060934640 1 -- 172.21.15.16:0/3253390117 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f50480047d0 msgr2=0x7f5048004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:45:08.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:07.993+0000 7f5060934640 1 --2- 172.21.15.16:0/3253390117 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f50480047d0 0x7f5048004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:08.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:07.993+0000 7f5060934640 1 -- 172.21.15.16:0/3253390117 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f50480039d0 msgr2=0x7f5048003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:45:08.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:07.993+0000 7f5060934640 1 --2- 172.21.15.16:0/3253390117 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f50480039d0 0x7f5048003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:08.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:07.993+0000 7f5060934640 1 -- 172.21.15.16:0/3253390117 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5048003530 con 0x7f5048005180 2023-05-22T15:45:08.003 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:07.993+0000 7f5060934640 1 -- 172.21.15.16:0/3253390117 reap_dead start 2023-05-22T15:45:08.003 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:07.993+0000 7f5060934640 1 --2- 172.21.15.16:0/3253390117 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5048005180 0x7f504800b990 secure :-1 s=READY pgs=1643 cs=0 l=1 rev1=1 crypto rx=0x7f5054002240 tx=0x7f505401d240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42293c7939deb070 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:45:08.003 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:07.997+0000 7f505b7fe640 1 -- 172.21.15.16:0/3253390117 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50540ad630 con 0x7f5048005180 2023-05-22T15:45:08.003 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:07.997+0000 7f505b7fe640 1 -- 172.21.15.16:0/3253390117 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50540a8070 con 0x7f5048005180 2023-05-22T15:45:08.004 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:07.997+0000 7f505b7fe640 1 -- 172.21.15.16:0/3253390117 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50540ad9d0 con 0x7f5048005180 2023-05-22T15:45:08.004 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:07.997+0000 7f5062397640 1 -- 172.21.15.16:0/3253390117 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5048005180 msgr2=0x7f504800b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:45:08.004 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:07.997+0000 7f5062397640 1 --2- 172.21.15.16:0/3253390117 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5048005180 0x7f504800b990 secure :-1 s=READY pgs=1643 cs=0 l=1 rev1=1 crypto rx=0x7f5054002240 tx=0x7f505401d240 comp rx=0 tx=0).stop 2023-05-22T15:45:08.004 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:07.997+0000 7f5062397640 1 -- 172.21.15.16:0/3253390117 shutdown_connections 2023-05-22T15:45:08.004 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:07.997+0000 7f5062397640 1 --2- 172.21.15.16:0/3253390117 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5048005180 0x7f504800b990 unknown :-1 s=CLOSED pgs=1643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:08.004 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:07.997+0000 7f5062397640 1 -- 172.21.15.16:0/3253390117 >> 172.21.15.16:0/3253390117 conn(0x7f505c0657f0 msgr2=0x7f505c070e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:45:08.004 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:07.997+0000 7f5062397640 1 -- 172.21.15.16:0/3253390117 shutdown_connections 2023-05-22T15:45:08.005 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:07.997+0000 7f5062397640 1 -- 172.21.15.16:0/3253390117 wait complete. 2023-05-22T15:45:08.005 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:07.997+0000 7f5062397640 1 Processor -- start 2023-05-22T15:45:08.005 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:07.997+0000 7f5062397640 1 -- start start 2023-05-22T15:45:08.005 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:07.997+0000 7f5062397640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5048005180 0x7f505c146680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:08.005 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:07.997+0000 7f5062397640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f505c1531b0 0x7f505c1f46b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:08.005 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:07.997+0000 7f5062397640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f505c153620 0x7f505c1f4bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:08.005 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:07.997+0000 7f5062397640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f505c0cf850 con 0x7f505c1531b0 2023-05-22T15:45:08.005 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:07.997+0000 7f5062397640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f505c0cebd0 con 0x7f5048005180 2023-05-22T15:45:08.006 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:07.997+0000 7f5062397640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f505c0ceeb0 con 0x7f505c153620 2023-05-22T15:45:08.006 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:07.997+0000 7f5053fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f505c1531b0 0x7f505c1f46b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:08.006 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:07.997+0000 7f5053fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f505c1531b0 0x7f505c1f46b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45262/0 (socket says 172.21.15.16:45262) 2023-05-22T15:45:08.006 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:07.997+0000 7f5053fff640 1 -- 172.21.15.16:0/3204553902 learned_addr learned my addr 172.21.15.16:0/3204553902 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:45:08.006 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:07.997+0000 7f505bfff640 1 --2- 172.21.15.16:0/3204553902 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5048005180 0x7f505c146680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:08.007 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:07.997+0000 7f5060934640 1 --2- 172.21.15.16:0/3204553902 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f505c153620 0x7f505c1f4bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:08.007 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:07.997+0000 7f5053fff640 1 -- 172.21.15.16:0/3204553902 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f505c153620 msgr2=0x7f505c1f4bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:45:08.007 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:07.997+0000 7f5053fff640 1 --2- 172.21.15.16:0/3204553902 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f505c153620 0x7f505c1f4bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:08.007 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:07.997+0000 7f5053fff640 1 -- 172.21.15.16:0/3204553902 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5048005180 msgr2=0x7f505c146680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:45:08.007 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:07.997+0000 7f5053fff640 1 --2- 172.21.15.16:0/3204553902 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5048005180 0x7f505c146680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:08.007 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:07.997+0000 7f5053fff640 1 -- 172.21.15.16:0/3204553902 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f505c160020 con 0x7f505c1531b0 2023-05-22T15:45:08.007 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:07.997+0000 7f505bfff640 1 --2- 172.21.15.16:0/3204553902 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5048005180 0x7f505c146680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:45:08.007 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:07.997+0000 7f5053fff640 1 --2- 172.21.15.16:0/3204553902 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f505c1531b0 0x7f505c1f46b0 secure :-1 s=READY pgs=1644 cs=0 l=1 rev1=1 crypto rx=0x7f5044011590 tx=0x7f5044011a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:45:08.008 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:07.997+0000 7f50597fa640 1 -- 172.21.15.16:0/3204553902 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5044015870 con 0x7f505c1531b0 2023-05-22T15:45:08.008 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:07.997+0000 7f50597fa640 1 -- 172.21.15.16:0/3204553902 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50440159d0 con 0x7f505c1531b0 2023-05-22T15:45:08.008 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:07.997+0000 7f5062397640 1 -- 172.21.15.16:0/3204553902 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f505c0cd110 con 0x7f505c1531b0 2023-05-22T15:45:08.008 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:07.997+0000 7f50597fa640 1 -- 172.21.15.16:0/3204553902 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5044015d40 con 0x7f505c1531b0 2023-05-22T15:45:08.008 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:07.997+0000 7f5062397640 1 -- 172.21.15.16:0/3204553902 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f505c1601b0 con 0x7f505c1531b0 2023-05-22T15:45:08.008 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:08.001+0000 7f5062397640 1 -- 172.21.15.16:0/3204553902 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f505c04dbe0 con 0x7f505c1531b0 2023-05-22T15:45:08.008 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:08.001+0000 7f50597fa640 1 -- 172.21.15.16:0/3204553902 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 50) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f5044013400 con 0x7f505c1531b0 2023-05-22T15:45:08.009 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:08.001+0000 7f50597fa640 1 --2- 172.21.15.16:0/3204553902 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f503c0654d0 0x7f503c067980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:08.009 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:08.001+0000 7f50597fa640 1 -- 172.21.15.16:0/3204553902 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f5044082e40 con 0x7f505c1531b0 2023-05-22T15:45:08.009 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:08.001+0000 7f505bfff640 1 --2- 172.21.15.16:0/3204553902 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f503c0654d0 0x7f503c067980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:08.010 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:08.001+0000 7f505bfff640 1 --2- 172.21.15.16:0/3204553902 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f503c0654d0 0x7f503c067980 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f504c001660 tx=0x7f504c009040 comp rx=0 tx=0).ready entity=mgr.8304 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:45:08.010 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:08.005+0000 7f50597fa640 1 -- 172.21.15.16:0/3204553902 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f5044054330 con 0x7f505c1531b0 2023-05-22T15:45:08.193 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:08.189+0000 7f5062397640 1 -- 172.21.15.16:0/3204553902 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f505c06b260 con 0x7f505c1531b0 2023-05-22T15:45:08.195 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:08.189+0000 7f50597fa640 1 -- 172.21.15.16:0/3204553902 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 50 v50) v1 ==== 94+0+4644 (secure 0 0 0) 0x7f5044053ec0 con 0x7f505c1531b0 2023-05-22T15:45:08.195 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:45:08.195 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":50,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6689,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/296693182","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6833","nonce":296693182},{"type":"v1","addr":"172.21.15.62:6835","nonce":296693182}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":46},{"gid":7402,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6834/856426578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":856426578},{"type":"v1","addr":"172.21.15.62:6834","nonce":856426578}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":46},{"gid":8541,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/2260895833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2260895833},{"type":"v1","addr":"172.21.15.16:6835","nonce":2260895833}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":46}],"filesystems":[{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-05-22T15:45:05.200760+0000","modified":"2023-05-22T15:45:07.832845+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8547},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8547":{"gid":8547,"name":"a","rank":0,"incarnation":47,"state":"up:active","state_seq":3,"addr":"172.21.15.16:6837/1566663454","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":1566663454},{"type":"v1","addr":"172.21.15.16:6837","nonce":1566663454}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":7}]} 2023-05-22T15:45:08.197 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 50 2023-05-22T15:45:08.197 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:08.189+0000 7f5062397640 1 -- 172.21.15.16:0/3204553902 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f503c0654d0 msgr2=0x7f503c067980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:45:08.198 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:08.189+0000 7f5062397640 1 --2- 172.21.15.16:0/3204553902 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f503c0654d0 0x7f503c067980 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f504c001660 tx=0x7f504c009040 comp rx=0 tx=0).stop 2023-05-22T15:45:08.198 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:08.189+0000 7f5062397640 1 -- 172.21.15.16:0/3204553902 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f505c1531b0 msgr2=0x7f505c1f46b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:45:08.198 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:08.189+0000 7f5062397640 1 --2- 172.21.15.16:0/3204553902 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f505c1531b0 0x7f505c1f46b0 secure :-1 s=READY pgs=1644 cs=0 l=1 rev1=1 crypto rx=0x7f5044011590 tx=0x7f5044011a50 comp rx=0 tx=0).stop 2023-05-22T15:45:08.198 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:08.193+0000 7f5062397640 1 -- 172.21.15.16:0/3204553902 shutdown_connections 2023-05-22T15:45:08.199 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:08.193+0000 7f5062397640 1 --2- 172.21.15.16:0/3204553902 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f503c0654d0 0x7f503c067980 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:08.199 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:08.193+0000 7f5062397640 1 --2- 172.21.15.16:0/3204553902 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f505c153620 0x7f505c1f4bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:08.199 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:08.193+0000 7f5062397640 1 --2- 172.21.15.16:0/3204553902 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f505c1531b0 0x7f505c1f46b0 unknown :-1 s=CLOSED pgs=1644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:08.200 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:08.193+0000 7f5062397640 1 --2- 172.21.15.16:0/3204553902 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5048005180 0x7f505c146680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:08.200 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:08.193+0000 7f5062397640 1 -- 172.21.15.16:0/3204553902 >> 172.21.15.16:0/3204553902 conn(0x7f505c0657f0 msgr2=0x7f505c0b7860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:45:08.200 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:08.193+0000 7f5062397640 1 -- 172.21.15.16:0/3204553902 shutdown_connections 2023-05-22T15:45:08.201 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:08.193+0000 7f5062397640 1 -- 172.21.15.16:0/3204553902 wait complete. 2023-05-22T15:45:08.210 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-05-22T15:45:08.331 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:08.329+0000 7f16ea647640 1 Processor -- start 2023-05-22T15:45:08.331 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:08.329+0000 7f16ea647640 1 -- start start 2023-05-22T15:45:08.332 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:08.329+0000 7f16ea647640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f16e4158a10 0x7f16e406e480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:08.332 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:08.329+0000 7f16ea647640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f16e406cdf0 0x7f16e4070be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:08.333 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:08.329+0000 7f16ea647640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16e4158de0 con 0x7f16e4070760 2023-05-22T15:45:08.333 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:08.329+0000 7f16ea647640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16e406e060 con 0x7f16e4158a10 2023-05-22T15:45:08.333 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:08.329+0000 7f16ea647640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16e40711e0 con 0x7f16e406cdf0 2023-05-22T15:45:08.334 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:08.329+0000 7f16e3fff640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f16e4158a10 0x7f16e406e480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:08.334 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:08.329+0000 7f16db7fe640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f16e4070760 0x7f16e406c6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:44910/0 (socket says 172.21.15.16:44910) 2023-05-22T15:45:08.334 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:08.329+0000 7f16db7fe640 1 -- 172.21.15.16:0/3083166398 learned_addr learned my addr 172.21.15.16:0/3083166398 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:45:08.334 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:08.329+0000 7f16e8be4640 1 --2- 172.21.15.16:0/3083166398 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f16e406cdf0 0x7f16e4070be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:08.335 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:08.329+0000 7f16e37fe640 1 -- 172.21.15.16:0/3083166398 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1444872175 0 0) 0x7f16e4158de0 con 0x7f16e4070760 2023-05-22T15:45:08.336 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:08.329+0000 7f16e37fe640 1 -- 172.21.15.16:0/3083166398 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f16d0003530 con 0x7f16e4070760 2023-05-22T15:45:08.336 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:08.329+0000 7f16e3fff640 1 -- 172.21.15.16:0/3083166398 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f16e406cdf0 msgr2=0x7f16e4070be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:45:08.336 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:08.329+0000 7f16e3fff640 1 --2- 172.21.15.16:0/3083166398 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f16e406cdf0 0x7f16e4070be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:08.336 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:08.329+0000 7f16e3fff640 1 -- 172.21.15.16:0/3083166398 >> v1:172.21.15.16:6789/0 conn(0x7f16e4070760 legacy=0x7f16e406c6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:45:08.336 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:08.329+0000 7f16e3fff640 1 -- 172.21.15.16:0/3083166398 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16e40714c0 con 0x7f16e4158a10 2023-05-22T15:45:08.336 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:08.333+0000 7f16e3fff640 1 --2- 172.21.15.16:0/3083166398 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16e4158a10 0x7f16e406e480 secure :-1 s=READY pgs=1646 cs=0 l=1 rev1=1 crypto rx=0x7f16cc0097c0 tx=0x7f16cc01c3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4774de8c75ab8a02 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:45:08.337 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:08.333+0000 7f16e37fe640 1 -- 172.21.15.16:0/3083166398 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16cc0ab610 con 0x7f16e4158a10 2023-05-22T15:45:08.337 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:08.333+0000 7f16e37fe640 1 -- 172.21.15.16:0/3083166398 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16cc0ab770 con 0x7f16e4158a10 2023-05-22T15:45:08.337 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:08.333+0000 7f16e37fe640 1 -- 172.21.15.16:0/3083166398 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16cc0abae0 con 0x7f16e4158a10 2023-05-22T15:45:08.337 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:08.333+0000 7f16ea647640 1 -- 172.21.15.16:0/3083166398 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16e4158a10 msgr2=0x7f16e406e480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:45:08.337 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:08.333+0000 7f16ea647640 1 --2- 172.21.15.16:0/3083166398 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16e4158a10 0x7f16e406e480 secure :-1 s=READY pgs=1646 cs=0 l=1 rev1=1 crypto rx=0x7f16cc0097c0 tx=0x7f16cc01c3f0 comp rx=0 tx=0).stop 2023-05-22T15:45:08.337 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:08.333+0000 7f16ea647640 1 -- 172.21.15.16:0/3083166398 shutdown_connections 2023-05-22T15:45:08.337 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:08.333+0000 7f16ea647640 1 --2- 172.21.15.16:0/3083166398 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f16e406cdf0 0x7f16e4070be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:08.338 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:08.333+0000 7f16ea647640 1 --2- 172.21.15.16:0/3083166398 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16e4158a10 0x7f16e406e480 unknown :-1 s=CLOSED pgs=1646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:08.338 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:08.333+0000 7f16ea647640 1 -- 172.21.15.16:0/3083166398 >> 172.21.15.16:0/3083166398 conn(0x7f16e40cc870 msgr2=0x7f16e40cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:45:08.338 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:08.333+0000 7f16ea647640 1 -- 172.21.15.16:0/3083166398 shutdown_connections 2023-05-22T15:45:08.338 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:08.333+0000 7f16ea647640 1 -- 172.21.15.16:0/3083166398 wait complete. 2023-05-22T15:45:08.341 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:08.333+0000 7f16ea647640 1 Processor -- start 2023-05-22T15:45:08.341 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:08.333+0000 7f16ea647640 1 -- start start 2023-05-22T15:45:08.341 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:08.333+0000 7f16ea647640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f16e406cdf0 0x7f16e415aec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:08.341 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:08.333+0000 7f16ea647640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16e4070760 0x7f16e414b740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:08.341 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:08.333+0000 7f16ea647640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f16e4158a10 0x7f16e40aee30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:08.341 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:08.333+0000 7f16ea647640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16e415ac50 con 0x7f16e4070760 2023-05-22T15:45:08.341 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:08.333+0000 7f16ea647640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16e4160d00 con 0x7f16e406cdf0 2023-05-22T15:45:08.342 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:08.333+0000 7f16ea647640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16e4160fe0 con 0x7f16e4158a10 2023-05-22T15:45:08.342 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:08.333+0000 7f16db7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16e4070760 0x7f16e414b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:08.342 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:08.333+0000 7f16db7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16e4070760 0x7f16e414b740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45284/0 (socket says 172.21.15.16:45284) 2023-05-22T15:45:08.342 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:08.333+0000 7f16db7fe640 1 -- 172.21.15.16:0/2689621907 learned_addr learned my addr 172.21.15.16:0/2689621907 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:45:08.342 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:08.333+0000 7f16e8be4640 1 --2- 172.21.15.16:0/2689621907 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f16e4158a10 0x7f16e40aee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:08.342 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:08.333+0000 7f16e3fff640 1 --2- 172.21.15.16:0/2689621907 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f16e406cdf0 0x7f16e415aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:08.342 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:08.333+0000 7f16db7fe640 1 -- 172.21.15.16:0/2689621907 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f16e4158a10 msgr2=0x7f16e40aee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:45:08.343 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:08.333+0000 7f16db7fe640 1 --2- 172.21.15.16:0/2689621907 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f16e4158a10 0x7f16e40aee30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:08.343 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:08.333+0000 7f16db7fe640 1 -- 172.21.15.16:0/2689621907 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f16e406cdf0 msgr2=0x7f16e415aec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:45:08.343 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:08.333+0000 7f16db7fe640 1 --2- 172.21.15.16:0/2689621907 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f16e406cdf0 0x7f16e415aec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:08.343 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:08.333+0000 7f16db7fe640 1 -- 172.21.15.16:0/2689621907 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16cc009420 con 0x7f16e4070760 2023-05-22T15:45:08.343 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:08.333+0000 7f16db7fe640 1 --2- 172.21.15.16:0/2689621907 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16e4070760 0x7f16e414b740 secure :-1 s=READY pgs=1647 cs=0 l=1 rev1=1 crypto rx=0x7f16d400c4d0 tx=0x7f16d400c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:45:08.343 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:08.337+0000 7f16e17fa640 1 -- 172.21.15.16:0/2689621907 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16d4005cf0 con 0x7f16e4070760 2023-05-22T15:45:08.343 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:08.337+0000 7f16e17fa640 1 -- 172.21.15.16:0/2689621907 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16d4005e50 con 0x7f16e4070760 2023-05-22T15:45:08.344 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:08.337+0000 7f16ea647640 1 -- 172.21.15.16:0/2689621907 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16e406eb50 con 0x7f16e4070760 2023-05-22T15:45:08.344 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:08.337+0000 7f16e17fa640 1 -- 172.21.15.16:0/2689621907 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16d4007270 con 0x7f16e4070760 2023-05-22T15:45:08.344 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:08.337+0000 7f16ea647640 1 -- 172.21.15.16:0/2689621907 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16e406e9c0 con 0x7f16e4070760 2023-05-22T15:45:08.344 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:08.337+0000 7f16ea647640 1 -- 172.21.15.16:0/2689621907 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16e40ca280 con 0x7f16e4070760 2023-05-22T15:45:08.344 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:08.337+0000 7f16e17fa640 1 -- 172.21.15.16:0/2689621907 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 50) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f16d40073d0 con 0x7f16e4070760 2023-05-22T15:45:08.345 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:08.337+0000 7f16e17fa640 1 --2- 172.21.15.16:0/2689621907 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f16c40654a0 0x7f16c4067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:08.345 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:08.337+0000 7f16e17fa640 1 -- 172.21.15.16:0/2689621907 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f16d40820b0 con 0x7f16e4070760 2023-05-22T15:45:08.345 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:08.337+0000 7f16e3fff640 1 --2- 172.21.15.16:0/2689621907 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f16c40654a0 0x7f16c4067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:08.345 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:08.337+0000 7f16e3fff640 1 --2- 172.21.15.16:0/2689621907 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f16c40654a0 0x7f16c4067950 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f16cc0af000 tx=0x7f16cc0a7000 comp rx=0 tx=0).ready entity=mgr.8304 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:45:08.345 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:08.341+0000 7f16e17fa640 1 -- 172.21.15.16:0/2689621907 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f16d404a620 con 0x7f16e4070760 2023-05-22T15:45:08.494 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:08.489+0000 7f16ea647640 1 -- 172.21.15.16:0/2689621907 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f16e404dbe0 con 0x7f16e4070760 2023-05-22T15:45:08.497 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:08.493+0000 7f16e17fa640 1 -- 172.21.15.16:0/2689621907 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v132) v1 ==== 74+0+17998 (secure 0 0 0) 0x7f16d4017090 con 0x7f16e4070760 2023-05-22T15:45:08.497 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:45:08.497 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":132,"fsid":"b2719834-7847-4d98-90a6-f65d57f85c49","created":"2023-05-22T15:26:25.259314+0000","modified":"2023-05-22T15:45:06.203912+0000","last_up_change":"2023-05-22T15:26:36.448524+0000","last_in_change":"2023-05-22T15:26:27.675668+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-05-22T15:26:38.019781+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-05-22T15:26:43.570514+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2023-05-22T15:45:03.718869+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"131","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2023-05-22T15:45:04.168135+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"132","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9e7ff854-b25f-4cbb-a5c2-1799d9c53d2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":37021},{"type":"v1","addr":"172.21.15.16:6825","nonce":37021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":37021},{"type":"v1","addr":"172.21.15.16:6827","nonce":37021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":37021},{"type":"v1","addr":"172.21.15.16:6831","nonce":37021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":37021},{"type":"v1","addr":"172.21.15.16:6829","nonce":37021}]},"public_addr":"172.21.15.16:6825/37021","cluster_addr":"172.21.15.16:6827/37021","heartbeat_back_addr":"172.21.15.16:6831/37021","heartbeat_front_addr":"172.21.15.16:6829/37021","state":["exists","up"]},{"osd":1,"uuid":"4702fbc9-a927-4565-aca5-41dc69cc8264","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":37008},{"type":"v1","addr":"172.21.15.16:6801","nonce":37008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":37008},{"type":"v1","addr":"172.21.15.16:6803","nonce":37008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":37008},{"type":"v1","addr":"172.21.15.16:6807","nonce":37008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":37008},{"type":"v1","addr":"172.21.15.16:6805","nonce":37008}]},"public_addr":"172.21.15.16:6801/37008","cluster_addr":"172.21.15.16:6803/37008","heartbeat_back_addr":"172.21.15.16:6807/37008","heartbeat_front_addr":"172.21.15.16:6805/37008","state":["exists","up"]},{"osd":2,"uuid":"1e7bba46-4eba-47ef-bb81-5e2d371638ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":37009},{"type":"v1","addr":"172.21.15.16:6809","nonce":37009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":37009},{"type":"v1","addr":"172.21.15.16:6811","nonce":37009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":37009},{"type":"v1","addr":"172.21.15.16:6815","nonce":37009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":37009},{"type":"v1","addr":"172.21.15.16:6813","nonce":37009}]},"public_addr":"172.21.15.16:6809/37009","cluster_addr":"172.21.15.16:6811/37009","heartbeat_back_addr":"172.21.15.16:6815/37009","heartbeat_front_addr":"172.21.15.16:6813/37009","state":["exists","up"]},{"osd":3,"uuid":"383cacc0-5a88-4904-990e-498ca59ef48a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":37020},{"type":"v1","addr":"172.21.15.16:6817","nonce":37020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":37020},{"type":"v1","addr":"172.21.15.16:6819","nonce":37020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":37020},{"type":"v1","addr":"172.21.15.16:6823","nonce":37020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":37020},{"type":"v1","addr":"172.21.15.16:6821","nonce":37020}]},"public_addr":"172.21.15.16:6817/37020","cluster_addr":"172.21.15.16:6819/37020","heartbeat_back_addr":"172.21.15.16:6823/37020","heartbeat_front_addr":"172.21.15.16:6821/37020","state":["exists","up"]},{"osd":4,"uuid":"970500ef-95b7-42c1-b080-eed4fae7013d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":37353},{"type":"v1","addr":"172.21.15.62:6817","nonce":37353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6819","nonce":37353},{"type":"v1","addr":"172.21.15.62:6822","nonce":37353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37353},{"type":"v1","addr":"172.21.15.62:6831","nonce":37353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37353},{"type":"v1","addr":"172.21.15.62:6826","nonce":37353}]},"public_addr":"172.21.15.62:6817/37353","cluster_addr":"172.21.15.62:6822/37353","heartbeat_back_addr":"172.21.15.62:6831/37353","heartbeat_front_addr":"172.21.15.62:6826/37353","state":["exists","up"]},{"osd":5,"uuid":"022d67db-d4d8-445b-b1b9-8b446a00f5c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":37352},{"type":"v1","addr":"172.21.15.62:6801","nonce":37352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":37352},{"type":"v1","addr":"172.21.15.62:6803","nonce":37352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":37352},{"type":"v1","addr":"172.21.15.62:6807","nonce":37352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":37352},{"type":"v1","addr":"172.21.15.62:6805","nonce":37352}]},"public_addr":"172.21.15.62:6801/37352","cluster_addr":"172.21.15.62:6803/37352","heartbeat_back_addr":"172.21.15.62:6807/37352","heartbeat_front_addr":"172.21.15.62:6805/37352","state":["exists","up"]},{"osd":6,"uuid":"42bd372b-6688-4a1b-8407-e16e8c266e33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":37354},{"type":"v1","addr":"172.21.15.62:6820","nonce":37354}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":37354},{"type":"v1","addr":"172.21.15.62:6823","nonce":37354}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6829","nonce":37354},{"type":"v1","addr":"172.21.15.62:6830","nonce":37354}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6825","nonce":37354},{"type":"v1","addr":"172.21.15.62:6827","nonce":37354}]},"public_addr":"172.21.15.62:6820/37354","cluster_addr":"172.21.15.62:6823/37354","heartbeat_back_addr":"172.21.15.62:6830/37354","heartbeat_front_addr":"172.21.15.62:6827/37354","state":["exists","up"]},{"osd":7,"uuid":"ac70b2a3-f41b-4bf2-b227-7f2cbb55426d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37355},{"type":"v1","addr":"172.21.15.62:6809","nonce":37355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37355},{"type":"v1","addr":"172.21.15.62:6811","nonce":37355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":37355},{"type":"v1","addr":"172.21.15.62:6815","nonce":37355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":37355},{"type":"v1","addr":"172.21.15.62:6813","nonce":37355}]},"public_addr":"172.21.15.62:6809/37355","cluster_addr":"172.21.15.62:6811/37355","heartbeat_back_addr":"172.21.15.62:6815/37355","heartbeat_front_addr":"172.21.15.62:6813/37355","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.368896+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.107273+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.276888+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.287502+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.124791+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.251937+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.255679+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.232797+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-05-22T15:45:08.499 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:08.493+0000 7f16ea647640 1 -- 172.21.15.16:0/2689621907 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f16c40654a0 msgr2=0x7f16c4067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:45:08.499 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:08.493+0000 7f16ea647640 1 --2- 172.21.15.16:0/2689621907 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f16c40654a0 0x7f16c4067950 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f16cc0af000 tx=0x7f16cc0a7000 comp rx=0 tx=0).stop 2023-05-22T15:45:08.499 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:08.493+0000 7f16ea647640 1 -- 172.21.15.16:0/2689621907 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16e4070760 msgr2=0x7f16e414b740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:45:08.499 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:08.493+0000 7f16ea647640 1 --2- 172.21.15.16:0/2689621907 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16e4070760 0x7f16e414b740 secure :-1 s=READY pgs=1647 cs=0 l=1 rev1=1 crypto rx=0x7f16d400c4d0 tx=0x7f16d400c990 comp rx=0 tx=0).stop 2023-05-22T15:45:08.499 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:08.493+0000 7f16ea647640 1 -- 172.21.15.16:0/2689621907 shutdown_connections 2023-05-22T15:45:08.499 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:08.493+0000 7f16ea647640 1 --2- 172.21.15.16:0/2689621907 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f16c40654a0 0x7f16c4067950 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:08.499 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:08.493+0000 7f16ea647640 1 --2- 172.21.15.16:0/2689621907 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f16e4158a10 0x7f16e40aee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:08.500 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:08.493+0000 7f16ea647640 1 --2- 172.21.15.16:0/2689621907 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16e4070760 0x7f16e414b740 secure :-1 s=CLOSED pgs=1647 cs=0 l=1 rev1=1 crypto rx=0x7f16d400c4d0 tx=0x7f16d400c990 comp rx=0 tx=0).stop 2023-05-22T15:45:08.500 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:08.493+0000 7f16ea647640 1 --2- 172.21.15.16:0/2689621907 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f16e406cdf0 0x7f16e415aec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:08.500 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:08.493+0000 7f16ea647640 1 -- 172.21.15.16:0/2689621907 >> 172.21.15.16:0/2689621907 conn(0x7f16e40cc870 msgr2=0x7f16e41579e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:45:08.500 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:08.493+0000 7f16ea647640 1 -- 172.21.15.16:0/2689621907 shutdown_connections 2023-05-22T15:45:08.500 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:08.493+0000 7f16ea647640 1 -- 172.21.15.16:0/2689621907 wait complete. 2023-05-22T15:45:08.514 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-05-22T15:45:08.515 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-05-22T15:45:08.626 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:08.625+0000 7fe0e3c31640 1 Processor -- start 2023-05-22T15:45:08.627 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:08.625+0000 7fe0e3c31640 1 -- start start 2023-05-22T15:45:08.628 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:08.625+0000 7fe0e3c31640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fe0dc1531b0 0x7fe0dc0afb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:08.628 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:08.625+0000 7fe0e3c31640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe0dc159a40 0x7fe0dc0c8ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:08.628 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:08.625+0000 7fe0e3c31640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0dc15a0a0 con 0x7fe0dc0b0080 2023-05-22T15:45:08.629 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:08.625+0000 7fe0e3c31640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0dc0b0520 con 0x7fe0dc1531b0 2023-05-22T15:45:08.629 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:08.625+0000 7fe0e3c31640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0dc159e50 con 0x7fe0dc159a40 2023-05-22T15:45:08.629 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:08.625+0000 7fe0e21ce640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe0dc159a40 0x7fe0dc0c8ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:08.629 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:08.625+0000 7fe0e19cd640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fe0dc1531b0 0x7fe0dc0afb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:08.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:08.625+0000 7fe0e21ce640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe0dc159a40 0x7fe0dc0c8ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3301/0 says I am v2:172.21.15.16:57802/0 (socket says 172.21.15.16:57802) 2023-05-22T15:45:08.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:08.625+0000 7fe0e21ce640 1 -- 172.21.15.16:0/3266288114 learned_addr learned my addr 172.21.15.16:0/3266288114 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:45:08.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:08.625+0000 7fe0e21ce640 1 -- 172.21.15.16:0/3266288114 >> v2:172.21.15.62:3300/0 conn(0x7fe0dc1531b0 msgr2=0x7fe0dc0afb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:45:08.632 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:08.625+0000 7fe0e21ce640 1 --2- 172.21.15.16:0/3266288114 >> v2:172.21.15.62:3300/0 conn(0x7fe0dc1531b0 0x7fe0dc0afb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:08.633 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:08.625+0000 7fe0e21ce640 1 -- 172.21.15.16:0/3266288114 >> v1:172.21.15.62:6789/0 conn(0x7fe0dc0b0080 legacy=0x7fe0dc159330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:45:08.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:08.625+0000 7fe0e21ce640 1 -- 172.21.15.16:0/3266288114 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0dc153580 con 0x7fe0dc159a40 2023-05-22T15:45:08.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:08.629+0000 7fe0e21ce640 1 --2- 172.21.15.16:0/3266288114 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe0dc159a40 0x7fe0dc0c8ba0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fe0d400d150 tx=0x7fe0d401ec10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2aff34ea05d9b90b server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:45:08.636 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:08.629+0000 7fe0e09cb640 1 -- 172.21.15.16:0/3266288114 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0d400b2d0 con 0x7fe0dc159a40 2023-05-22T15:45:08.636 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:08.629+0000 7fe0e09cb640 1 -- 172.21.15.16:0/3266288114 <== mon.2 v2:172.21.15.62:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0d400b430 con 0x7fe0dc159a40 2023-05-22T15:45:08.637 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:08.629+0000 7fe0e09cb640 1 -- 172.21.15.16:0/3266288114 <== mon.2 v2:172.21.15.62:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0d40a7bf0 con 0x7fe0dc159a40 2023-05-22T15:45:08.637 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:08.629+0000 7fe0e3c31640 1 -- 172.21.15.16:0/3266288114 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe0dc159a40 msgr2=0x7fe0dc0c8ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:45:08.637 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:08.629+0000 7fe0e3c31640 1 --2- 172.21.15.16:0/3266288114 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe0dc159a40 0x7fe0dc0c8ba0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fe0d400d150 tx=0x7fe0d401ec10 comp rx=0 tx=0).stop 2023-05-22T15:45:08.637 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:08.629+0000 7fe0e3c31640 1 -- 172.21.15.16:0/3266288114 shutdown_connections 2023-05-22T15:45:08.637 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:08.629+0000 7fe0e3c31640 1 --2- 172.21.15.16:0/3266288114 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe0dc159a40 0x7fe0dc0c8ba0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:08.637 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:08.629+0000 7fe0e3c31640 1 --2- 172.21.15.16:0/3266288114 >> v2:172.21.15.62:3300/0 conn(0x7fe0dc1531b0 0x7fe0dc0afb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:08.637 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:08.629+0000 7fe0e3c31640 1 -- 172.21.15.16:0/3266288114 >> 172.21.15.16:0/3266288114 conn(0x7fe0dc069720 msgr2=0x7fe0dc069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:45:08.638 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:08.629+0000 7fe0e3c31640 1 -- 172.21.15.16:0/3266288114 shutdown_connections 2023-05-22T15:45:08.638 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:08.629+0000 7fe0e3c31640 1 -- 172.21.15.16:0/3266288114 wait complete. 2023-05-22T15:45:08.638 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:08.629+0000 7fe0e3c31640 1 Processor -- start 2023-05-22T15:45:08.638 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:08.629+0000 7fe0e3c31640 1 -- start start 2023-05-22T15:45:08.638 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:08.633+0000 7fe0e3c31640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe0dc0b0080 0x7fe0dc0c1b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:08.638 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:08.633+0000 7fe0e3c31640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe0dc1531b0 0x7fe0dc1495f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:08.638 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:08.633+0000 7fe0e3c31640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe0dc159a40 0x7fe0dc0b19e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:08.638 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:08.633+0000 7fe0e3c31640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0dc150220 con 0x7fe0dc0b0080 2023-05-22T15:45:08.639 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:08.633+0000 7fe0e3c31640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0dc14f610 con 0x7fe0dc1531b0 2023-05-22T15:45:08.639 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:08.633+0000 7fe0e3c31640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0dc14f8f0 con 0x7fe0dc159a40 2023-05-22T15:45:08.639 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:08.633+0000 7fe0e19cd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe0dc0b0080 0x7fe0dc0c1b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:08.639 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:08.633+0000 7fe0e19cd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe0dc0b0080 0x7fe0dc0c1b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45298/0 (socket says 172.21.15.16:45298) 2023-05-22T15:45:08.639 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:08.633+0000 7fe0e19cd640 1 -- 172.21.15.16:0/2587125525 learned_addr learned my addr 172.21.15.16:0/2587125525 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:45:08.639 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:08.633+0000 7fe0e21ce640 1 --2- 172.21.15.16:0/2587125525 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe0dc159a40 0x7fe0dc0b19e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:08.639 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:08.633+0000 7fe0e11cc640 1 --2- 172.21.15.16:0/2587125525 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe0dc1531b0 0x7fe0dc1495f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:08.640 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:08.633+0000 7fe0e19cd640 1 -- 172.21.15.16:0/2587125525 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe0dc159a40 msgr2=0x7fe0dc0b19e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:45:08.640 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:08.633+0000 7fe0e19cd640 1 --2- 172.21.15.16:0/2587125525 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe0dc159a40 0x7fe0dc0b19e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:08.640 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:08.633+0000 7fe0e19cd640 1 -- 172.21.15.16:0/2587125525 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe0dc1531b0 msgr2=0x7fe0dc1495f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:45:08.640 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:08.633+0000 7fe0e19cd640 1 --2- 172.21.15.16:0/2587125525 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe0dc1531b0 0x7fe0dc1495f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:08.640 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:08.633+0000 7fe0e19cd640 1 -- 172.21.15.16:0/2587125525 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0d400bd40 con 0x7fe0dc0b0080 2023-05-22T15:45:08.640 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:08.633+0000 7fe0e11cc640 1 --2- 172.21.15.16:0/2587125525 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe0dc1531b0 0x7fe0dc1495f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:45:08.641 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:08.633+0000 7fe0e19cd640 1 --2- 172.21.15.16:0/2587125525 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe0dc0b0080 0x7fe0dc0c1b50 secure :-1 s=READY pgs=1648 cs=0 l=1 rev1=1 crypto rx=0x7fe0d800e930 tx=0x7fe0d800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:45:08.641 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:08.633+0000 7fe0d27fc640 1 -- 172.21.15.16:0/2587125525 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0d8009690 con 0x7fe0dc0b0080 2023-05-22T15:45:08.641 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:08.633+0000 7fe0d27fc640 1 -- 172.21.15.16:0/2587125525 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0d800ca00 con 0x7fe0dc0b0080 2023-05-22T15:45:08.641 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:08.633+0000 7fe0e3c31640 1 -- 172.21.15.16:0/2587125525 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0dc15d0c0 con 0x7fe0dc0b0080 2023-05-22T15:45:08.641 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:08.633+0000 7fe0d27fc640 1 -- 172.21.15.16:0/2587125525 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0d800cd70 con 0x7fe0dc0b0080 2023-05-22T15:45:08.641 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:08.633+0000 7fe0e3c31640 1 -- 172.21.15.16:0/2587125525 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0dc15cde0 con 0x7fe0dc0b0080 2023-05-22T15:45:08.642 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:08.633+0000 7fe0e3c31640 1 -- 172.21.15.16:0/2587125525 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0dc0ce020 con 0x7fe0dc0b0080 2023-05-22T15:45:08.642 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:08.633+0000 7fe0d27fc640 1 -- 172.21.15.16:0/2587125525 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 50) v1 ==== 84332+0+0 (secure 0 0 0) 0x7fe0d8013470 con 0x7fe0dc0b0080 2023-05-22T15:45:08.643 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:08.633+0000 7fe0d27fc640 1 --2- 172.21.15.16:0/2587125525 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fe0c00654a0 0x7fe0c0067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:08.643 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:08.633+0000 7fe0d27fc640 1 -- 172.21.15.16:0/2587125525 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 6382+0+0 (secure 0 0 0) 0x7fe0d8082d70 con 0x7fe0dc0b0080 2023-05-22T15:45:08.643 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:08.633+0000 7fe0e11cc640 1 --2- 172.21.15.16:0/2587125525 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fe0c00654a0 0x7fe0c0067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:08.643 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:08.633+0000 7fe0e11cc640 1 --2- 172.21.15.16:0/2587125525 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fe0c00654a0 0x7fe0c0067950 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fe0dc0b3fb0 tx=0x7fe0cc009040 comp rx=0 tx=0).ready entity=mgr.8304 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:45:08.643 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:08.637+0000 7fe0d27fc640 1 -- 172.21.15.16:0/2587125525 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fe0d8050800 con 0x7fe0dc0b0080 2023-05-22T15:45:08.793 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:08.789+0000 7fe0e3c31640 1 -- 172.21.15.16:0/2587125525 --> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fe0dc150360 con 0x7fe0c00654a0 2023-05-22T15:45:08.813 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:08.809+0000 7fe0d27fc640 1 -- 172.21.15.16:0/2587125525 <== mgr.8304 v2:172.21.15.16:6832/36921 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+310104 (secure 0 0 0) 0x7fe0cc001ce0 con 0x7fe0c00654a0 2023-05-22T15:45:08.815 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:45:08.817 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2023-05-22T15:45:08.818 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:08.813+0000 7fe0e3c31640 1 -- 172.21.15.16:0/2587125525 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fe0c00654a0 msgr2=0x7fe0c0067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:45:08.818 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:08.813+0000 7fe0e3c31640 1 --2- 172.21.15.16:0/2587125525 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fe0c00654a0 0x7fe0c0067950 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fe0dc0b3fb0 tx=0x7fe0cc009040 comp rx=0 tx=0).stop 2023-05-22T15:45:08.818 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:08.813+0000 7fe0e3c31640 1 -- 172.21.15.16:0/2587125525 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe0dc0b0080 msgr2=0x7fe0dc0c1b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:45:08.818 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:08.813+0000 7fe0e3c31640 1 --2- 172.21.15.16:0/2587125525 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe0dc0b0080 0x7fe0dc0c1b50 secure :-1 s=READY pgs=1648 cs=0 l=1 rev1=1 crypto rx=0x7fe0d800e930 tx=0x7fe0d800edf0 comp rx=0 tx=0).stop 2023-05-22T15:45:08.819 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:08.813+0000 7fe0e3c31640 1 -- 172.21.15.16:0/2587125525 shutdown_connections 2023-05-22T15:45:08.819 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:08.813+0000 7fe0e3c31640 1 --2- 172.21.15.16:0/2587125525 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fe0c00654a0 0x7fe0c0067950 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:08.819 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:08.813+0000 7fe0e3c31640 1 --2- 172.21.15.16:0/2587125525 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe0dc159a40 0x7fe0dc0b19e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:08.819 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:08.813+0000 7fe0e3c31640 1 --2- 172.21.15.16:0/2587125525 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe0dc1531b0 0x7fe0dc1495f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:08.819 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:08.813+0000 7fe0e3c31640 1 --2- 172.21.15.16:0/2587125525 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe0dc0b0080 0x7fe0dc0c1b50 unknown :-1 s=CLOSED pgs=1648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:08.819 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:08.813+0000 7fe0e3c31640 1 -- 172.21.15.16:0/2587125525 >> 172.21.15.16:0/2587125525 conn(0x7fe0dc069720 msgr2=0x7fe0dc069e00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:45:08.819 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:08.813+0000 7fe0e3c31640 1 -- 172.21.15.16:0/2587125525 shutdown_connections 2023-05-22T15:45:08.820 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:08.813+0000 7fe0e3c31640 1 -- 172.21.15.16:0/2587125525 wait complete. 2023-05-22T15:45:08.835 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2023-05-22T15:45:08.064469+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592698,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":1076,"num_read_kb":2160,"num_write":404,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":266,"ondisk_log_size":266,"up":210,"acting":210,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":40,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":224564,"kb_used_data":13876,"kb_used_omap":0,"kb_used_meta":210624,"kb_avail":754750156,"statfs":{"total":773094113280,"available":772864159744,"internally_reserved":0,"allocated":14209024,"data_stored":8014936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":215678976},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":61,"apply_latency_ms":61,"commit_latency_ns":61000000,"apply_latency_ns":61000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2330,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.261557"},"pg_stats":[{"pgid":"17.3f","version":"0'0","reported_seq":9,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-05-22T15:45:05.266425+0000","last_change":"2023-05-22T15:45:05.265785+0000","last_active":"2023-05-22T15:45:05.266425+0000","last_peered":"2023-05-22T15:45:05.266425+0000","last_clean":"2023-05-22T15:45:05.266425+0000","last_became_active":"2023-05-22T15:45:05.265486+0000","last_became_peered":"2023-05-22T15:45:05.265486+0000","last_unstale":"2023-05-22T15:45:05.266425+0000","last_undegraded":"2023-05-22T15:45:05.266425+0000","last_fullsized":"2023-05-22T15:45:05.266425+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_clean_scrub_stamp":"2023-05-22T15:45:04.802083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:36:11.193934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3e","version":"0'0","reported_seq":9,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-05-22T15:45:05.261440+0000","last_change":"2023-05-22T15:45:05.252520+0000","last_active":"2023-05-22T15:45:05.261440+0000","last_peered":"2023-05-22T15:45:05.261440+0000","last_clean":"2023-05-22T15:45:05.261440+0000","last_became_active":"2023-05-22T15:45:05.251768+0000","last_became_peered":"2023-05-22T15:45:05.251768+0000","last_unstale":"2023-05-22T15:45:05.261440+0000","last_undegraded":"2023-05-22T15:45:05.261440+0000","last_fullsized":"2023-05-22T15:45:05.261440+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_clean_scrub_stamp":"2023-05-22T15:45:04.802083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:09:12.891783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.3d","version":"0'0","reported_seq":9,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-05-22T15:45:05.289805+0000","last_change":"2023-05-22T15:45:05.281371+0000","last_active":"2023-05-22T15:45:05.289805+0000","last_peered":"2023-05-22T15:45:05.289805+0000","last_clean":"2023-05-22T15:45:05.289805+0000","last_became_active":"2023-05-22T15:45:05.279345+0000","last_became_peered":"2023-05-22T15:45:05.279345+0000","last_unstale":"2023-05-22T15:45:05.289805+0000","last_undegraded":"2023-05-22T15:45:05.289805+0000","last_fullsized":"2023-05-22T15:45:05.289805+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_clean_scrub_stamp":"2023-05-22T15:45:04.802083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:22:50.401810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.3c","version":"0'0","reported_seq":9,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-05-22T15:45:05.252795+0000","last_change":"2023-05-22T15:45:05.251350+0000","last_active":"2023-05-22T15:45:05.252795+0000","last_peered":"2023-05-22T15:45:05.252795+0000","last_clean":"2023-05-22T15:45:05.252795+0000","last_became_active":"2023-05-22T15:45:05.251091+0000","last_became_peered":"2023-05-22T15:45:05.251091+0000","last_unstale":"2023-05-22T15:45:05.252795+0000","last_undegraded":"2023-05-22T15:45:05.252795+0000","last_fullsized":"2023-05-22T15:45:05.252795+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_clean_scrub_stamp":"2023-05-22T15:45:04.802083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:51:15.372468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3b","version":"0'0","reported_seq":9,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-05-22T15:45:05.217549+0000","last_change":"2023-05-22T15:45:05.216706+0000","last_active":"2023-05-22T15:45:05.217549+0000","last_peered":"2023-05-22T15:45:05.217549+0000","last_clean":"2023-05-22T15:45:05.217549+0000","last_became_active":"2023-05-22T15:45:05.216316+0000","last_became_peered":"2023-05-22T15:45:05.216316+0000","last_unstale":"2023-05-22T15:45:05.217549+0000","last_undegraded":"2023-05-22T15:45:05.217549+0000","last_fullsized":"2023-05-22T15:45:05.217549+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_clean_scrub_stamp":"2023-05-22T15:45:04.802083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:00:52.830469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.3a","version":"0'0","reported_seq":9,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-05-22T15:45:05.227672+0000","last_change":"2023-05-22T15:45:05.217866+0000","last_active":"2023-05-22T15:45:05.227672+0000","last_peered":"2023-05-22T15:45:05.227672+0000","last_clean":"2023-05-22T15:45:05.227672+0000","last_became_active":"2023-05-22T15:45:05.217582+0000","last_became_peered":"2023-05-22T15:45:05.217582+0000","last_unstale":"2023-05-22T15:45:05.227672+0000","last_undegraded":"2023-05-22T15:45:05.227672+0000","last_fullsized":"2023-05-22T15:45:05.227672+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_clean_scrub_stamp":"2023-05-22T15:45:04.802083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T16:55:32.318704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.39","version":"0'0","reported_seq":9,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-05-22T15:45:05.239851+0000","last_change":"2023-05-22T15:45:05.234774+0000","last_active":"2023-05-22T15:45:05.239851+0000","last_peered":"2023-05-22T15:45:05.239851+0000","last_clean":"2023-05-22T15:45:05.239851+0000","last_became_active":"2023-05-22T15:45:05.234054+0000","last_became_peered":"2023-05-22T15:45:05.234054+0000","last_unstale":"2023-05-22T15:45:05.239851+0000","last_undegraded":"2023-05-22T15:45:05.239851+0000","last_fullsized":"2023-05-22T15:45:05.239851+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_clean_scrub_stamp":"2023-05-22T15:45:04.802083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T18:38:17.673900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.38","version":"0'0","reported_seq":9,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-05-22T15:45:05.289844+0000","last_change":"2023-05-22T15:45:05.285050+0000","last_active":"2023-05-22T15:45:05.289844+0000","last_peered":"2023-05-22T15:45:05.289844+0000","last_clean":"2023-05-22T15:45:05.289844+0000","last_became_active":"2023-05-22T15:45:05.281072+0000","last_became_peered":"2023-05-22T15:45:05.281072+0000","last_unstale":"2023-05-22T15:45:05.289844+0000","last_undegraded":"2023-05-22T15:45:05.289844+0000","last_fullsized":"2023-05-22T15:45:05.289844+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_clean_scrub_stamp":"2023-05-22T15:45:04.802083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:29:06.460933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.37","version":"0'0","reported_seq":9,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-05-22T15:45:05.241024+0000","last_change":"2023-05-22T15:45:05.234127+0000","last_active":"2023-05-22T15:45:05.241024+0000","last_peered":"2023-05-22T15:45:05.241024+0000","last_clean":"2023-05-22T15:45:05.241024+0000","last_became_active":"2023-05-22T15:45:05.233876+0000","last_became_peered":"2023-05-22T15:45:05.233876+0000","last_unstale":"2023-05-22T15:45:05.241024+0000","last_undegraded":"2023-05-22T15:45:05.241024+0000","last_fullsized":"2023-05-22T15:45:05.241024+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_clean_scrub_stamp":"2023-05-22T15:45:04.802083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T16:14:35.323782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.36","version":"0'0","reported_seq":9,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-05-22T15:45:05.241092+0000","last_change":"2023-05-22T15:45:05.234783+0000","last_active":"2023-05-22T15:45:05.241092+0000","last_peered":"2023-05-22T15:45:05.241092+0000","last_clean":"2023-05-22T15:45:05.241092+0000","last_became_active":"2023-05-22T15:45:05.234523+0000","last_became_peered":"2023-05-22T15:45:05.234523+0000","last_unstale":"2023-05-22T15:45:05.241092+0000","last_undegraded":"2023-05-22T15:45:05.241092+0000","last_fullsized":"2023-05-22T15:45:05.241092+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_clean_scrub_stamp":"2023-05-22T15:45:04.802083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:25:01.333952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.35","version":"0'0","reported_seq":9,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-05-22T15:45:05.264970+0000","last_change":"2023-05-22T15:45:05.253945+0000","last_active":"2023-05-22T15:45:05.264970+0000","last_peered":"2023-05-22T15:45:05.264970+0000","last_clean":"2023-05-22T15:45:05.264970+0000","last_became_active":"2023-05-22T15:45:05.253119+0000","last_became_peered":"2023-05-22T15:45:05.253119+0000","last_unstale":"2023-05-22T15:45:05.264970+0000","last_undegraded":"2023-05-22T15:45:05.264970+0000","last_fullsized":"2023-05-22T15:45:05.264970+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_clean_scrub_stamp":"2023-05-22T15:45:04.802083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:17:05.563955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.34","version":"0'0","reported_seq":9,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-05-22T15:45:05.239014+0000","last_change":"2023-05-22T15:45:05.233397+0000","last_active":"2023-05-22T15:45:05.239014+0000","last_peered":"2023-05-22T15:45:05.239014+0000","last_clean":"2023-05-22T15:45:05.239014+0000","last_became_active":"2023-05-22T15:45:05.233083+0000","last_became_peered":"2023-05-22T15:45:05.233083+0000","last_unstale":"2023-05-22T15:45:05.239014+0000","last_undegraded":"2023-05-22T15:45:05.239014+0000","last_fullsized":"2023-05-22T15:45:05.239014+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_clean_scrub_stamp":"2023-05-22T15:45:04.802083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T18:46:17.237523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.33","version":"0'0","reported_seq":9,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-05-22T15:45:05.289787+0000","last_change":"2023-05-22T15:45:05.281691+0000","last_active":"2023-05-22T15:45:05.289787+0000","last_peered":"2023-05-22T15:45:05.289787+0000","last_clean":"2023-05-22T15:45:05.289787+0000","last_became_active":"2023-05-22T15:45:05.280285+0000","last_became_peered":"2023-05-22T15:45:05.280285+0000","last_unstale":"2023-05-22T15:45:05.289787+0000","last_undegraded":"2023-05-22T15:45:05.289787+0000","last_fullsized":"2023-05-22T15:45:05.289787+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_clean_scrub_stamp":"2023-05-22T15:45:04.802083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T16:42:50.060919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.32","version":"0'0","reported_seq":9,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-05-22T15:45:05.289585+0000","last_change":"2023-05-22T15:45:05.279318+0000","last_active":"2023-05-22T15:45:05.289585+0000","last_peered":"2023-05-22T15:45:05.289585+0000","last_clean":"2023-05-22T15:45:05.289585+0000","last_became_active":"2023-05-22T15:45:05.278848+0000","last_became_peered":"2023-05-22T15:45:05.278848+0000","last_unstale":"2023-05-22T15:45:05.289585+0000","last_undegraded":"2023-05-22T15:45:05.289585+0000","last_fullsized":"2023-05-22T15:45:05.289585+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_clean_scrub_stamp":"2023-05-22T15:45:04.802083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:09:12.891783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.31","version":"0'0","reported_seq":9,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-05-22T15:45:05.223267+0000","last_change":"2023-05-22T15:45:05.217316+0000","last_active":"2023-05-22T15:45:05.223267+0000","last_peered":"2023-05-22T15:45:05.223267+0000","last_clean":"2023-05-22T15:45:05.223267+0000","last_became_active":"2023-05-22T15:45:05.216853+0000","last_became_peered":"2023-05-22T15:45:05.216853+0000","last_unstale":"2023-05-22T15:45:05.223267+0000","last_undegraded":"2023-05-22T15:45:05.223267+0000","last_fullsized":"2023-05-22T15:45:05.223267+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_clean_scrub_stamp":"2023-05-22T15:45:04.802083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:00:52.830469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.30","version":"0'0","reported_seq":9,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-05-22T15:45:05.243075+0000","last_change":"2023-05-22T15:45:05.241967+0000","last_active":"2023-05-22T15:45:05.243075+0000","last_peered":"2023-05-22T15:45:05.243075+0000","last_clean":"2023-05-22T15:45:05.243075+0000","last_became_active":"2023-05-22T15:45:05.241757+0000","last_became_peered":"2023-05-22T15:45:05.241757+0000","last_unstale":"2023-05-22T15:45:05.243075+0000","last_undegraded":"2023-05-22T15:45:05.243075+0000","last_fullsized":"2023-05-22T15:45:05.243075+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_clean_scrub_stamp":"2023-05-22T15:45:04.802083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:57:36.081992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.2f","version":"0'0","reported_seq":9,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-05-22T15:45:05.271840+0000","last_change":"2023-05-22T15:45:05.268751+0000","last_active":"2023-05-22T15:45:05.271840+0000","last_peered":"2023-05-22T15:45:05.271840+0000","last_clean":"2023-05-22T15:45:05.271840+0000","last_became_active":"2023-05-22T15:45:05.267736+0000","last_became_peered":"2023-05-22T15:45:05.267736+0000","last_unstale":"2023-05-22T15:45:05.271840+0000","last_undegraded":"2023-05-22T15:45:05.271840+0000","last_fullsized":"2023-05-22T15:45:05.271840+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_clean_scrub_stamp":"2023-05-22T15:45:04.802083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:41:01.090364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2e","version":"0'0","reported_seq":9,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-05-22T15:45:05.274103+0000","last_change":"2023-05-22T15:45:05.253174+0000","last_active":"2023-05-22T15:45:05.274103+0000","last_peered":"2023-05-22T15:45:05.274103+0000","last_clean":"2023-05-22T15:45:05.274103+0000","last_became_active":"2023-05-22T15:45:05.252455+0000","last_became_peered":"2023-05-22T15:45:05.252455+0000","last_unstale":"2023-05-22T15:45:05.274103+0000","last_undegraded":"2023-05-22T15:45:05.274103+0000","last_fullsized":"2023-05-22T15:45:05.274103+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_clean_scrub_stamp":"2023-05-22T15:45:04.802083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:46:36.321475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.2d","version":"0'0","reported_seq":9,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-05-22T15:45:05.289770+0000","last_change":"2023-05-22T15:45:05.281430+0000","last_active":"2023-05-22T15:45:05.289770+0000","last_peered":"2023-05-22T15:45:05.289770+0000","last_clean":"2023-05-22T15:45:05.289770+0000","last_became_active":"2023-05-22T15:45:05.280421+0000","last_became_peered":"2023-05-22T15:45:05.280421+0000","last_unstale":"2023-05-22T15:45:05.289770+0000","last_undegraded":"2023-05-22T15:45:05.289770+0000","last_fullsized":"2023-05-22T15:45:05.289770+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_clean_scrub_stamp":"2023-05-22T15:45:04.802083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T03:18:50.763567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.2c","version":"0'0","reported_seq":9,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-05-22T15:45:05.261159+0000","last_change":"2023-05-22T15:45:05.245275+0000","last_active":"2023-05-22T15:45:05.261159+0000","last_peered":"2023-05-22T15:45:05.261159+0000","last_clean":"2023-05-22T15:45:05.261159+0000","last_became_active":"2023-05-22T15:45:05.244543+0000","last_became_peered":"2023-05-22T15:45:05.244543+0000","last_unstale":"2023-05-22T15:45:05.261159+0000","last_undegraded":"2023-05-22T15:45:05.261159+0000","last_fullsized":"2023-05-22T15:45:05.261159+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_clean_scrub_stamp":"2023-05-22T15:45:04.802083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:34:38.791933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2b","version":"0'0","reported_seq":9,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-05-22T15:45:05.269669+0000","last_change":"2023-05-22T15:45:05.262231+0000","last_active":"2023-05-22T15:45:05.269669+0000","last_peered":"2023-05-22T15:45:05.269669+0000","last_clean":"2023-05-22T15:45:05.269669+0000","last_became_active":"2023-05-22T15:45:05.261471+0000","last_became_peered":"2023-05-22T15:45:05.261471+0000","last_unstale":"2023-05-22T15:45:05.269669+0000","last_undegraded":"2023-05-22T15:45:05.269669+0000","last_fullsized":"2023-05-22T15:45:05.269669+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_clean_scrub_stamp":"2023-05-22T15:45:04.802083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:23:52.692138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2a","version":"0'0","reported_seq":9,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-05-22T15:45:05.269723+0000","last_change":"2023-05-22T15:45:05.269213+0000","last_active":"2023-05-22T15:45:05.269723+0000","last_peered":"2023-05-22T15:45:05.269723+0000","last_clean":"2023-05-22T15:45:05.269723+0000","last_became_active":"2023-05-22T15:45:05.268886+0000","last_became_peered":"2023-05-22T15:45:05.268886+0000","last_unstale":"2023-05-22T15:45:05.269723+0000","last_undegraded":"2023-05-22T15:45:05.269723+0000","last_fullsized":"2023-05-22T15:45:05.269723+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_clean_scrub_stamp":"2023-05-22T15:45:04.802083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:34:38.791933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.29","version":"0'0","reported_seq":9,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-05-22T15:45:05.276886+0000","last_change":"2023-05-22T15:45:05.273703+0000","last_active":"2023-05-22T15:45:05.276886+0000","last_peered":"2023-05-22T15:45:05.276886+0000","last_clean":"2023-05-22T15:45:05.276886+0000","last_became_active":"2023-05-22T15:45:05.273181+0000","last_became_peered":"2023-05-22T15:45:05.273181+0000","last_unstale":"2023-05-22T15:45:05.276886+0000","last_undegraded":"2023-05-22T15:45:05.276886+0000","last_fullsized":"2023-05-22T15:45:05.276886+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_clean_scrub_stamp":"2023-05-22T15:45:04.802083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:22:50.401810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.28","version":"0'0","reported_seq":9,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-05-22T15:45:05.242870+0000","last_change":"2023-05-22T15:45:05.241904+0000","last_active":"2023-05-22T15:45:05.242870+0000","last_peered":"2023-05-22T15:45:05.242870+0000","last_clean":"2023-05-22T15:45:05.242870+0000","last_became_active":"2023-05-22T15:45:05.241604+0000","last_became_peered":"2023-05-22T15:45:05.241604+0000","last_unstale":"2023-05-22T15:45:05.242870+0000","last_undegraded":"2023-05-22T15:45:05.242870+0000","last_fullsized":"2023-05-22T15:45:05.242870+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_clean_scrub_stamp":"2023-05-22T15:45:04.802083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:52:29.459195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.27","version":"0'0","reported_seq":9,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-05-22T15:45:05.271809+0000","last_change":"2023-05-22T15:45:05.268365+0000","last_active":"2023-05-22T15:45:05.271809+0000","last_peered":"2023-05-22T15:45:05.271809+0000","last_clean":"2023-05-22T15:45:05.271809+0000","last_became_active":"2023-05-22T15:45:05.267647+0000","last_became_peered":"2023-05-22T15:45:05.267647+0000","last_unstale":"2023-05-22T15:45:05.271809+0000","last_undegraded":"2023-05-22T15:45:05.271809+0000","last_fullsized":"2023-05-22T15:45:05.271809+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_clean_scrub_stamp":"2023-05-22T15:45:04.802083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:06:40.672215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.26","version":"0'0","reported_seq":9,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-05-22T15:45:05.242614+0000","last_change":"2023-05-22T15:45:05.239295+0000","last_active":"2023-05-22T15:45:05.242614+0000","last_peered":"2023-05-22T15:45:05.242614+0000","last_clean":"2023-05-22T15:45:05.242614+0000","last_became_active":"2023-05-22T15:45:05.239002+0000","last_became_peered":"2023-05-22T15:45:05.239002+0000","last_unstale":"2023-05-22T15:45:05.242614+0000","last_undegraded":"2023-05-22T15:45:05.242614+0000","last_fullsized":"2023-05-22T15:45:05.242614+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_clean_scrub_stamp":"2023-05-22T15:45:04.802083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T03:03:20.741835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.25","version":"0'0","reported_seq":9,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-05-22T15:45:05.235190+0000","last_change":"2023-05-22T15:45:05.231336+0000","last_active":"2023-05-22T15:45:05.235190+0000","last_peered":"2023-05-22T15:45:05.235190+0000","last_clean":"2023-05-22T15:45:05.235190+0000","last_became_active":"2023-05-22T15:45:05.230379+0000","last_became_peered":"2023-05-22T15:45:05.230379+0000","last_unstale":"2023-05-22T15:45:05.235190+0000","last_undegraded":"2023-05-22T15:45:05.235190+0000","last_fullsized":"2023-05-22T15:45:05.235190+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_clean_scrub_stamp":"2023-05-22T15:45:04.802083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:41:05.163840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.24","version":"0'0","reported_seq":9,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-05-22T15:45:05.236226+0000","last_change":"2023-05-22T15:45:05.234054+0000","last_active":"2023-05-22T15:45:05.236226+0000","last_peered":"2023-05-22T15:45:05.236226+0000","last_clean":"2023-05-22T15:45:05.236226+0000","last_became_active":"2023-05-22T15:45:05.233789+0000","last_became_peered":"2023-05-22T15:45:05.233789+0000","last_unstale":"2023-05-22T15:45:05.236226+0000","last_undegraded":"2023-05-22T15:45:05.236226+0000","last_fullsized":"2023-05-22T15:45:05.236226+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_clean_scrub_stamp":"2023-05-22T15:45:04.802083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:01:01.088020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.23","version":"0'0","reported_seq":9,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-05-22T15:45:05.285893+0000","last_change":"2023-05-22T15:45:05.280361+0000","last_active":"2023-05-22T15:45:05.285893+0000","last_peered":"2023-05-22T15:45:05.285893+0000","last_clean":"2023-05-22T15:45:05.285893+0000","last_became_active":"2023-05-22T15:45:05.279623+0000","last_became_peered":"2023-05-22T15:45:05.279623+0000","last_unstale":"2023-05-22T15:45:05.285893+0000","last_undegraded":"2023-05-22T15:45:05.285893+0000","last_fullsized":"2023-05-22T15:45:05.285893+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_clean_scrub_stamp":"2023-05-22T15:45:04.802083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:52:29.459195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.22","version":"0'0","reported_seq":9,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-05-22T15:45:05.239577+0000","last_change":"2023-05-22T15:45:05.231856+0000","last_active":"2023-05-22T15:45:05.239577+0000","last_peered":"2023-05-22T15:45:05.239577+0000","last_clean":"2023-05-22T15:45:05.239577+0000","last_became_active":"2023-05-22T15:45:05.231053+0000","last_became_peered":"2023-05-22T15:45:05.231053+0000","last_unstale":"2023-05-22T15:45:05.239577+0000","last_undegraded":"2023-05-22T15:45:05.239577+0000","last_fullsized":"2023-05-22T15:45:05.239577+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_clean_scrub_stamp":"2023-05-22T15:45:04.802083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:08:32.792658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.21","version":"0'0","reported_seq":9,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-05-22T15:45:05.241689+0000","last_change":"2023-05-22T15:45:05.239247+0000","last_active":"2023-05-22T15:45:05.241689+0000","last_peered":"2023-05-22T15:45:05.241689+0000","last_clean":"2023-05-22T15:45:05.241689+0000","last_became_active":"2023-05-22T15:45:05.238970+0000","last_became_peered":"2023-05-22T15:45:05.238970+0000","last_unstale":"2023-05-22T15:45:05.241689+0000","last_undegraded":"2023-05-22T15:45:05.241689+0000","last_fullsized":"2023-05-22T15:45:05.241689+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_clean_scrub_stamp":"2023-05-22T15:45:04.802083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:52:07.479851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.20","version":"0'0","reported_seq":9,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-05-22T15:45:05.223278+0000","last_change":"2023-05-22T15:45:05.217281+0000","last_active":"2023-05-22T15:45:05.223278+0000","last_peered":"2023-05-22T15:45:05.223278+0000","last_clean":"2023-05-22T15:45:05.223278+0000","last_became_active":"2023-05-22T15:45:05.216989+0000","last_became_peered":"2023-05-22T15:45:05.216989+0000","last_unstale":"2023-05-22T15:45:05.223278+0000","last_undegraded":"2023-05-22T15:45:05.223278+0000","last_fullsized":"2023-05-22T15:45:05.223278+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_clean_scrub_stamp":"2023-05-22T15:45:04.802083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:08:32.792658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1e","version":"0'0","reported_seq":9,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-05-22T15:45:05.215467+0000","last_change":"2023-05-22T15:45:05.213555+0000","last_active":"2023-05-22T15:45:05.215467+0000","last_peered":"2023-05-22T15:45:05.215467+0000","last_clean":"2023-05-22T15:45:05.215467+0000","last_became_active":"2023-05-22T15:45:05.213311+0000","last_became_peered":"2023-05-22T15:45:05.213311+0000","last_unstale":"2023-05-22T15:45:05.215467+0000","last_undegraded":"2023-05-22T15:45:05.215467+0000","last_fullsized":"2023-05-22T15:45:05.215467+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_clean_scrub_stamp":"2023-05-22T15:45:04.802083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:11:24.311549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1f","version":"131'4","reported_seq":14,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-05-22T15:45:05.282593+0000","last_change":"2023-05-22T15:45:04.869973+0000","last_active":"2023-05-22T15:45:05.282593+0000","last_peered":"2023-05-22T15:45:05.282593+0000","last_clean":"2023-05-22T15:45:05.282593+0000","last_became_active":"2023-05-22T15:45:04.869509+0000","last_became_peered":"2023-05-22T15:45:04.869509+0000","last_unstale":"2023-05-22T15:45:05.282593+0000","last_undegraded":"2023-05-22T15:45:05.282593+0000","last_fullsized":"2023-05-22T15:45:05.282593+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:45:03.797855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:45:03.797855+0000","last_clean_scrub_stamp":"2023-05-22T15:45:03.797855+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:45:07.119273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":566,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1f","version":"0'0","reported_seq":9,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-05-22T15:45:05.276159+0000","last_change":"2023-05-22T15:45:05.274094+0000","last_active":"2023-05-22T15:45:05.276159+0000","last_peered":"2023-05-22T15:45:05.276159+0000","last_clean":"2023-05-22T15:45:05.276159+0000","last_became_active":"2023-05-22T15:45:05.273255+0000","last_became_peered":"2023-05-22T15:45:05.273255+0000","last_unstale":"2023-05-22T15:45:05.276159+0000","last_undegraded":"2023-05-22T15:45:05.276159+0000","last_fullsized":"2023-05-22T15:45:05.276159+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_clean_scrub_stamp":"2023-05-22T15:45:04.802083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:50:34.136689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1e","version":"131'1","reported_seq":11,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-05-22T15:45:05.278868+0000","last_change":"2023-05-22T15:45:04.871159+0000","last_active":"2023-05-22T15:45:05.278868+0000","last_peered":"2023-05-22T15:45:05.278868+0000","last_clean":"2023-05-22T15:45:05.278868+0000","last_became_active":"2023-05-22T15:45:04.870377+0000","last_became_peered":"2023-05-22T15:45:04.870377+0000","last_unstale":"2023-05-22T15:45:05.278868+0000","last_undegraded":"2023-05-22T15:45:05.278868+0000","last_fullsized":"2023-05-22T15:45:05.278868+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:45:03.797855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:45:03.797855+0000","last_clean_scrub_stamp":"2023-05-22T15:45:03.797855+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:03:57.988869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"0'0","reported_seq":9,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-05-22T15:45:05.223245+0000","last_change":"2023-05-22T15:45:05.217299+0000","last_active":"2023-05-22T15:45:05.223245+0000","last_peered":"2023-05-22T15:45:05.223245+0000","last_clean":"2023-05-22T15:45:05.223245+0000","last_became_active":"2023-05-22T15:45:05.217078+0000","last_became_peered":"2023-05-22T15:45:05.217078+0000","last_unstale":"2023-05-22T15:45:05.223245+0000","last_undegraded":"2023-05-22T15:45:05.223245+0000","last_fullsized":"2023-05-22T15:45:05.223245+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_clean_scrub_stamp":"2023-05-22T15:45:04.802083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:17:53.852853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1d","version":"131'2","reported_seq":12,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-05-22T15:45:05.242366+0000","last_change":"2023-05-22T15:45:04.842007+0000","last_active":"2023-05-22T15:45:05.242366+0000","last_peered":"2023-05-22T15:45:05.242366+0000","last_clean":"2023-05-22T15:45:05.242366+0000","last_became_active":"2023-05-22T15:45:04.841559+0000","last_became_peered":"2023-05-22T15:45:04.841559+0000","last_unstale":"2023-05-22T15:45:05.242366+0000","last_undegraded":"2023-05-22T15:45:05.242366+0000","last_fullsized":"2023-05-22T15:45:05.242366+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:45:03.797855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:45:03.797855+0000","last_clean_scrub_stamp":"2023-05-22T15:45:03.797855+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:45:07.119273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1d","version":"0'0","reported_seq":9,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-05-22T15:45:05.289852+0000","last_change":"2023-05-22T15:45:05.282125+0000","last_active":"2023-05-22T15:45:05.289852+0000","last_peered":"2023-05-22T15:45:05.289852+0000","last_clean":"2023-05-22T15:45:05.289852+0000","last_became_active":"2023-05-22T15:45:05.280933+0000","last_became_peered":"2023-05-22T15:45:05.280933+0000","last_unstale":"2023-05-22T15:45:05.289852+0000","last_undegraded":"2023-05-22T15:45:05.289852+0000","last_fullsized":"2023-05-22T15:45:05.289852+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_clean_scrub_stamp":"2023-05-22T15:45:04.802083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T18:44:18.645468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1c","version":"131'2","reported_seq":12,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-05-22T15:45:05.242370+0000","last_change":"2023-05-22T15:45:04.842614+0000","last_active":"2023-05-22T15:45:05.242370+0000","last_peered":"2023-05-22T15:45:05.242370+0000","last_clean":"2023-05-22T15:45:05.242370+0000","last_became_active":"2023-05-22T15:45:04.842097+0000","last_became_peered":"2023-05-22T15:45:04.842097+0000","last_unstale":"2023-05-22T15:45:05.242370+0000","last_undegraded":"2023-05-22T15:45:05.242370+0000","last_fullsized":"2023-05-22T15:45:05.242370+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:45:03.797855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:45:03.797855+0000","last_clean_scrub_stamp":"2023-05-22T15:45:03.797855+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T16:34:14.156084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1a","version":"0'0","reported_seq":9,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-05-22T15:45:05.229405+0000","last_change":"2023-05-22T15:45:05.224946+0000","last_active":"2023-05-22T15:45:05.229405+0000","last_peered":"2023-05-22T15:45:05.229405+0000","last_clean":"2023-05-22T15:45:05.229405+0000","last_became_active":"2023-05-22T15:45:05.224665+0000","last_became_peered":"2023-05-22T15:45:05.224665+0000","last_unstale":"2023-05-22T15:45:05.229405+0000","last_undegraded":"2023-05-22T15:45:05.229405+0000","last_fullsized":"2023-05-22T15:45:05.229405+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_clean_scrub_stamp":"2023-05-22T15:45:04.802083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T03:12:25.788950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1b","version":"131'1","reported_seq":11,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-05-22T15:45:05.260842+0000","last_change":"2023-05-22T15:45:04.858720+0000","last_active":"2023-05-22T15:45:05.260842+0000","last_peered":"2023-05-22T15:45:05.260842+0000","last_clean":"2023-05-22T15:45:05.260842+0000","last_became_active":"2023-05-22T15:45:04.857943+0000","last_became_peered":"2023-05-22T15:45:04.857943+0000","last_unstale":"2023-05-22T15:45:05.260842+0000","last_undegraded":"2023-05-22T15:45:05.260842+0000","last_fullsized":"2023-05-22T15:45:05.260842+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:45:03.797855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:45:03.797855+0000","last_clean_scrub_stamp":"2023-05-22T15:45:03.797855+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:37:19.722085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1b","version":"0'0","reported_seq":9,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-05-22T15:45:05.264903+0000","last_change":"2023-05-22T15:45:05.253189+0000","last_active":"2023-05-22T15:45:05.264903+0000","last_peered":"2023-05-22T15:45:05.264903+0000","last_clean":"2023-05-22T15:45:05.264903+0000","last_became_active":"2023-05-22T15:45:05.252282+0000","last_became_peered":"2023-05-22T15:45:05.252282+0000","last_unstale":"2023-05-22T15:45:05.264903+0000","last_undegraded":"2023-05-22T15:45:05.264903+0000","last_fullsized":"2023-05-22T15:45:05.264903+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_clean_scrub_stamp":"2023-05-22T15:45:04.802083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:00:24.701583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1a","version":"131'1","reported_seq":11,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-05-22T15:45:05.268632+0000","last_change":"2023-05-22T15:45:04.861987+0000","last_active":"2023-05-22T15:45:05.268632+0000","last_peered":"2023-05-22T15:45:05.268632+0000","last_clean":"2023-05-22T15:45:05.268632+0000","last_became_active":"2023-05-22T15:45:04.861703+0000","last_became_peered":"2023-05-22T15:45:04.861703+0000","last_unstale":"2023-05-22T15:45:05.268632+0000","last_undegraded":"2023-05-22T15:45:05.268632+0000","last_fullsized":"2023-05-22T15:45:05.268632+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:45:03.797855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:45:03.797855+0000","last_clean_scrub_stamp":"2023-05-22T15:45:03.797855+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T03:42:25.864815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.18","version":"0'0","reported_seq":9,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-05-22T15:45:05.241105+0000","last_change":"2023-05-22T15:45:05.235148+0000","last_active":"2023-05-22T15:45:05.241105+0000","last_peered":"2023-05-22T15:45:05.241105+0000","last_clean":"2023-05-22T15:45:05.241105+0000","last_became_active":"2023-05-22T15:45:05.234900+0000","last_became_peered":"2023-05-22T15:45:05.234900+0000","last_unstale":"2023-05-22T15:45:05.241105+0000","last_undegraded":"2023-05-22T15:45:05.241105+0000","last_fullsized":"2023-05-22T15:45:05.241105+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_clean_scrub_stamp":"2023-05-22T15:45:04.802083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:00:57.027365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":10,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-05-22T15:45:05.244747+0000","last_change":"2023-05-22T15:45:04.853214+0000","last_active":"2023-05-22T15:45:05.244747+0000","last_peered":"2023-05-22T15:45:05.244747+0000","last_clean":"2023-05-22T15:45:05.244747+0000","last_became_active":"2023-05-22T15:45:04.852332+0000","last_became_peered":"2023-05-22T15:45:04.852332+0000","last_unstale":"2023-05-22T15:45:05.244747+0000","last_undegraded":"2023-05-22T15:45:05.244747+0000","last_fullsized":"2023-05-22T15:45:05.244747+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:45:03.797855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:45:03.797855+0000","last_clean_scrub_stamp":"2023-05-22T15:45:03.797855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:30:53.313453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":9,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-05-22T15:45:05.259132+0000","last_change":"2023-05-22T15:45:05.244170+0000","last_active":"2023-05-22T15:45:05.259132+0000","last_peered":"2023-05-22T15:45:05.259132+0000","last_clean":"2023-05-22T15:45:05.259132+0000","last_became_active":"2023-05-22T15:45:05.243422+0000","last_became_peered":"2023-05-22T15:45:05.243422+0000","last_unstale":"2023-05-22T15:45:05.259132+0000","last_undegraded":"2023-05-22T15:45:05.259132+0000","last_fullsized":"2023-05-22T15:45:05.259132+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_clean_scrub_stamp":"2023-05-22T15:45:04.802083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T16:17:06.682365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":10,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-05-22T15:45:05.219570+0000","last_change":"2023-05-22T15:45:04.849089+0000","last_active":"2023-05-22T15:45:05.219570+0000","last_peered":"2023-05-22T15:45:05.219570+0000","last_clean":"2023-05-22T15:45:05.219570+0000","last_became_active":"2023-05-22T15:45:04.848736+0000","last_became_peered":"2023-05-22T15:45:04.848736+0000","last_unstale":"2023-05-22T15:45:05.219570+0000","last_undegraded":"2023-05-22T15:45:05.219570+0000","last_fullsized":"2023-05-22T15:45:05.219570+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:45:03.797855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:45:03.797855+0000","last_clean_scrub_stamp":"2023-05-22T15:45:03.797855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:01:00.083792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.e","version":"0'0","reported_seq":9,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-05-22T15:45:05.242260+0000","last_change":"2023-05-22T15:45:05.230275+0000","last_active":"2023-05-22T15:45:05.242260+0000","last_peered":"2023-05-22T15:45:05.242260+0000","last_clean":"2023-05-22T15:45:05.242260+0000","last_became_active":"2023-05-22T15:45:05.229991+0000","last_became_peered":"2023-05-22T15:45:05.229991+0000","last_unstale":"2023-05-22T15:45:05.242260+0000","last_undegraded":"2023-05-22T15:45:05.242260+0000","last_fullsized":"2023-05-22T15:45:05.242260+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_clean_scrub_stamp":"2023-05-22T15:45:04.802083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:08:32.792658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.f","version":"131'1","reported_seq":11,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-05-22T15:45:05.238814+0000","last_change":"2023-05-22T15:45:04.852319+0000","last_active":"2023-05-22T15:45:05.238814+0000","last_peered":"2023-05-22T15:45:05.238814+0000","last_clean":"2023-05-22T15:45:05.238814+0000","last_became_active":"2023-05-22T15:45:04.851954+0000","last_became_peered":"2023-05-22T15:45:04.851954+0000","last_unstale":"2023-05-22T15:45:05.238814+0000","last_undegraded":"2023-05-22T15:45:05.238814+0000","last_fullsized":"2023-05-22T15:45:05.238814+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:45:03.797855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:45:03.797855+0000","last_clean_scrub_stamp":"2023-05-22T15:45:03.797855+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T18:58:40.362754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.f","version":"0'0","reported_seq":9,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-05-22T15:45:05.236255+0000","last_change":"2023-05-22T15:45:05.234321+0000","last_active":"2023-05-22T15:45:05.236255+0000","last_peered":"2023-05-22T15:45:05.236255+0000","last_clean":"2023-05-22T15:45:05.236255+0000","last_became_active":"2023-05-22T15:45:05.234015+0000","last_became_peered":"2023-05-22T15:45:05.234015+0000","last_unstale":"2023-05-22T15:45:05.236255+0000","last_undegraded":"2023-05-22T15:45:05.236255+0000","last_fullsized":"2023-05-22T15:45:05.236255+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_clean_scrub_stamp":"2023-05-22T15:45:04.802083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T18:45:31.668005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.e","version":"131'2","reported_seq":12,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-05-22T15:45:05.252138+0000","last_change":"2023-05-22T15:45:04.841561+0000","last_active":"2023-05-22T15:45:05.252138+0000","last_peered":"2023-05-22T15:45:05.252138+0000","last_clean":"2023-05-22T15:45:05.252138+0000","last_became_active":"2023-05-22T15:45:04.841227+0000","last_became_peered":"2023-05-22T15:45:04.841227+0000","last_unstale":"2023-05-22T15:45:05.252138+0000","last_undegraded":"2023-05-22T15:45:05.252138+0000","last_fullsized":"2023-05-22T15:45:05.252138+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:45:03.797855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:45:03.797855+0000","last_clean_scrub_stamp":"2023-05-22T15:45:03.797855+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:11:23.307322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":9,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-05-22T15:45:05.289638+0000","last_change":"2023-05-22T15:45:05.279979+0000","last_active":"2023-05-22T15:45:05.289638+0000","last_peered":"2023-05-22T15:45:05.289638+0000","last_clean":"2023-05-22T15:45:05.289638+0000","last_became_active":"2023-05-22T15:45:05.279184+0000","last_became_peered":"2023-05-22T15:45:05.279184+0000","last_unstale":"2023-05-22T15:45:05.289638+0000","last_undegraded":"2023-05-22T15:45:05.289638+0000","last_fullsized":"2023-05-22T15:45:05.289638+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_clean_scrub_stamp":"2023-05-22T15:45:04.802083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:33:29.003697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.d","version":"131'1","reported_seq":11,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-05-22T15:45:05.273720+0000","last_change":"2023-05-22T15:45:04.852056+0000","last_active":"2023-05-22T15:45:05.273720+0000","last_peered":"2023-05-22T15:45:05.273720+0000","last_clean":"2023-05-22T15:45:05.273720+0000","last_became_active":"2023-05-22T15:45:04.851291+0000","last_became_peered":"2023-05-22T15:45:04.851291+0000","last_unstale":"2023-05-22T15:45:05.273720+0000","last_undegraded":"2023-05-22T15:45:05.273720+0000","last_fullsized":"2023-05-22T15:45:05.273720+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:45:03.797855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:45:03.797855+0000","last_clean_scrub_stamp":"2023-05-22T15:45:03.797855+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T18:38:16.669672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.d","version":"0'0","reported_seq":9,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-05-22T15:45:05.289836+0000","last_change":"2023-05-22T15:45:05.281784+0000","last_active":"2023-05-22T15:45:05.289836+0000","last_peered":"2023-05-22T15:45:05.289836+0000","last_clean":"2023-05-22T15:45:05.289836+0000","last_became_active":"2023-05-22T15:45:05.280662+0000","last_became_peered":"2023-05-22T15:45:05.280662+0000","last_unstale":"2023-05-22T15:45:05.289836+0000","last_undegraded":"2023-05-22T15:45:05.289836+0000","last_fullsized":"2023-05-22T15:45:05.289836+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_clean_scrub_stamp":"2023-05-22T15:45:04.802083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:00:24.701583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":10,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-05-22T15:45:05.214239+0000","last_change":"2023-05-22T15:45:04.860453+0000","last_active":"2023-05-22T15:45:05.214239+0000","last_peered":"2023-05-22T15:45:05.214239+0000","last_clean":"2023-05-22T15:45:05.214239+0000","last_became_active":"2023-05-22T15:45:04.859619+0000","last_became_peered":"2023-05-22T15:45:04.859619+0000","last_unstale":"2023-05-22T15:45:05.214239+0000","last_undegraded":"2023-05-22T15:45:05.214239+0000","last_fullsized":"2023-05-22T15:45:05.214239+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:45:03.797855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:45:03.797855+0000","last_clean_scrub_stamp":"2023-05-22T15:45:03.797855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T15:48:38.551891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.a","version":"0'0","reported_seq":9,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-05-22T15:45:05.289663+0000","last_change":"2023-05-22T15:45:05.280179+0000","last_active":"2023-05-22T15:45:05.289663+0000","last_peered":"2023-05-22T15:45:05.289663+0000","last_clean":"2023-05-22T15:45:05.289663+0000","last_became_active":"2023-05-22T15:45:05.279701+0000","last_became_peered":"2023-05-22T15:45:05.279701+0000","last_unstale":"2023-05-22T15:45:05.289663+0000","last_undegraded":"2023-05-22T15:45:05.289663+0000","last_fullsized":"2023-05-22T15:45:05.289663+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_clean_scrub_stamp":"2023-05-22T15:45:04.802083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:28:24.072722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.b","version":"131'2","reported_seq":12,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-05-22T15:45:05.231793+0000","last_change":"2023-05-22T15:45:04.842639+0000","last_active":"2023-05-22T15:45:05.231793+0000","last_peered":"2023-05-22T15:45:05.231793+0000","last_clean":"2023-05-22T15:45:05.231793+0000","last_became_active":"2023-05-22T15:45:04.842205+0000","last_became_peered":"2023-05-22T15:45:04.842205+0000","last_unstale":"2023-05-22T15:45:05.231793+0000","last_undegraded":"2023-05-22T15:45:05.231793+0000","last_fullsized":"2023-05-22T15:45:05.231793+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:45:03.797855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:45:03.797855+0000","last_clean_scrub_stamp":"2023-05-22T15:45:03.797855+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:26:15.565494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.b","version":"0'0","reported_seq":9,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-05-22T15:45:05.243067+0000","last_change":"2023-05-22T15:45:05.241929+0000","last_active":"2023-05-22T15:45:05.243067+0000","last_peered":"2023-05-22T15:45:05.243067+0000","last_clean":"2023-05-22T15:45:05.243067+0000","last_became_active":"2023-05-22T15:45:05.241689+0000","last_became_peered":"2023-05-22T15:45:05.241689+0000","last_unstale":"2023-05-22T15:45:05.243067+0000","last_undegraded":"2023-05-22T15:45:05.243067+0000","last_fullsized":"2023-05-22T15:45:05.243067+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_clean_scrub_stamp":"2023-05-22T15:45:04.802083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:40:25.027078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":10,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-05-22T15:45:05.207791+0000","last_change":"2023-05-22T15:45:04.845246+0000","last_active":"2023-05-22T15:45:05.207791+0000","last_peered":"2023-05-22T15:45:05.207791+0000","last_clean":"2023-05-22T15:45:05.207791+0000","last_became_active":"2023-05-22T15:45:04.844828+0000","last_became_peered":"2023-05-22T15:45:04.844828+0000","last_unstale":"2023-05-22T15:45:05.207791+0000","last_undegraded":"2023-05-22T15:45:05.207791+0000","last_fullsized":"2023-05-22T15:45:05.207791+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:45:03.797855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:45:03.797855+0000","last_clean_scrub_stamp":"2023-05-22T15:45:03.797855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:37:19.722085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.8","version":"0'0","reported_seq":9,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-05-22T15:45:05.289596+0000","last_change":"2023-05-22T15:45:05.279697+0000","last_active":"2023-05-22T15:45:05.289596+0000","last_peered":"2023-05-22T15:45:05.289596+0000","last_clean":"2023-05-22T15:45:05.289596+0000","last_became_active":"2023-05-22T15:45:05.278972+0000","last_became_peered":"2023-05-22T15:45:05.278972+0000","last_unstale":"2023-05-22T15:45:05.289596+0000","last_undegraded":"2023-05-22T15:45:05.289596+0000","last_fullsized":"2023-05-22T15:45:05.289596+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_clean_scrub_stamp":"2023-05-22T15:45:04.802083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:59:57.835660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":10,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-05-22T15:45:05.210256+0000","last_change":"2023-05-22T15:45:04.844292+0000","last_active":"2023-05-22T15:45:05.210256+0000","last_peered":"2023-05-22T15:45:05.210256+0000","last_clean":"2023-05-22T15:45:05.210256+0000","last_became_active":"2023-05-22T15:45:04.843144+0000","last_became_peered":"2023-05-22T15:45:04.843144+0000","last_unstale":"2023-05-22T15:45:05.210256+0000","last_undegraded":"2023-05-22T15:45:05.210256+0000","last_fullsized":"2023-05-22T15:45:05.210256+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:45:03.797855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:45:03.797855+0000","last_clean_scrub_stamp":"2023-05-22T15:45:03.797855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:19:28.324586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":9,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-05-22T15:45:05.234450+0000","last_change":"2023-05-22T15:45:05.231757+0000","last_active":"2023-05-22T15:45:05.234450+0000","last_peered":"2023-05-22T15:45:05.234450+0000","last_clean":"2023-05-22T15:45:05.234450+0000","last_became_active":"2023-05-22T15:45:05.231377+0000","last_became_peered":"2023-05-22T15:45:05.231377+0000","last_unstale":"2023-05-22T15:45:05.234450+0000","last_undegraded":"2023-05-22T15:45:05.234450+0000","last_fullsized":"2023-05-22T15:45:05.234450+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_clean_scrub_stamp":"2023-05-22T15:45:04.802083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:06:40.672215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":10,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-05-22T15:45:05.219571+0000","last_change":"2023-05-22T15:45:04.853025+0000","last_active":"2023-05-22T15:45:05.219571+0000","last_peered":"2023-05-22T15:45:05.219571+0000","last_clean":"2023-05-22T15:45:05.219571+0000","last_became_active":"2023-05-22T15:45:04.852645+0000","last_became_peered":"2023-05-22T15:45:04.852645+0000","last_unstale":"2023-05-22T15:45:05.219571+0000","last_undegraded":"2023-05-22T15:45:05.219571+0000","last_fullsized":"2023-05-22T15:45:05.219571+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:45:03.797855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:45:03.797855+0000","last_clean_scrub_stamp":"2023-05-22T15:45:03.797855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T03:35:58.750761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.6","version":"0'0","reported_seq":9,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-05-22T15:45:05.269921+0000","last_change":"2023-05-22T15:45:05.243786+0000","last_active":"2023-05-22T15:45:05.269921+0000","last_peered":"2023-05-22T15:45:05.269921+0000","last_clean":"2023-05-22T15:45:05.269921+0000","last_became_active":"2023-05-22T15:45:05.242992+0000","last_became_peered":"2023-05-22T15:45:05.242992+0000","last_unstale":"2023-05-22T15:45:05.269921+0000","last_undegraded":"2023-05-22T15:45:05.269921+0000","last_fullsized":"2023-05-22T15:45:05.269921+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_clean_scrub_stamp":"2023-05-22T15:45:04.802083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:21:00.848206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.7","version":"131'2","reported_seq":12,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-05-22T15:45:05.232831+0000","last_change":"2023-05-22T15:45:04.853894+0000","last_active":"2023-05-22T15:45:05.232831+0000","last_peered":"2023-05-22T15:45:05.232831+0000","last_clean":"2023-05-22T15:45:05.232831+0000","last_became_active":"2023-05-22T15:45:04.853572+0000","last_became_peered":"2023-05-22T15:45:04.853572+0000","last_unstale":"2023-05-22T15:45:05.232831+0000","last_undegraded":"2023-05-22T15:45:05.232831+0000","last_fullsized":"2023-05-22T15:45:05.232831+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:45:03.797855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:45:03.797855+0000","last_clean_scrub_stamp":"2023-05-22T15:45:03.797855+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:20:59.843979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.7","version":"0'0","reported_seq":9,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-05-22T15:45:05.252803+0000","last_change":"2023-05-22T15:45:05.251361+0000","last_active":"2023-05-22T15:45:05.252803+0000","last_peered":"2023-05-22T15:45:05.252803+0000","last_clean":"2023-05-22T15:45:05.252803+0000","last_became_active":"2023-05-22T15:45:05.251007+0000","last_became_peered":"2023-05-22T15:45:05.251007+0000","last_unstale":"2023-05-22T15:45:05.252803+0000","last_undegraded":"2023-05-22T15:45:05.252803+0000","last_fullsized":"2023-05-22T15:45:05.252803+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_clean_scrub_stamp":"2023-05-22T15:45:04.802083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T03:23:00.475670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.6","version":"131'1","reported_seq":11,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-05-22T15:45:05.735502+0000","last_change":"2023-05-22T15:45:04.855381+0000","last_active":"2023-05-22T15:45:05.735502+0000","last_peered":"2023-05-22T15:45:05.735502+0000","last_clean":"2023-05-22T15:45:05.735502+0000","last_became_active":"2023-05-22T15:45:04.853669+0000","last_became_peered":"2023-05-22T15:45:04.853669+0000","last_unstale":"2023-05-22T15:45:05.735502+0000","last_undegraded":"2023-05-22T15:45:05.735502+0000","last_fullsized":"2023-05-22T15:45:05.735502+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:45:03.797855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:45:03.797855+0000","last_clean_scrub_stamp":"2023-05-22T15:45:03.797855+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:33:35.429024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":9,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-05-22T15:45:05.241244+0000","last_change":"2023-05-22T15:45:05.238182+0000","last_active":"2023-05-22T15:45:05.241244+0000","last_peered":"2023-05-22T15:45:05.241244+0000","last_clean":"2023-05-22T15:45:05.241244+0000","last_became_active":"2023-05-22T15:45:05.237126+0000","last_became_peered":"2023-05-22T15:45:05.237126+0000","last_unstale":"2023-05-22T15:45:05.241244+0000","last_undegraded":"2023-05-22T15:45:05.241244+0000","last_fullsized":"2023-05-22T15:45:05.241244+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_clean_scrub_stamp":"2023-05-22T15:45:04.802083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:05:00.739104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":10,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-05-22T15:45:05.223114+0000","last_change":"2023-05-22T15:45:04.862699+0000","last_active":"2023-05-22T15:45:05.223114+0000","last_peered":"2023-05-22T15:45:05.223114+0000","last_clean":"2023-05-22T15:45:05.223114+0000","last_became_active":"2023-05-22T15:45:04.862334+0000","last_became_peered":"2023-05-22T15:45:04.862334+0000","last_unstale":"2023-05-22T15:45:05.223114+0000","last_undegraded":"2023-05-22T15:45:05.223114+0000","last_fullsized":"2023-05-22T15:45:05.223114+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:45:03.797855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:45:03.797855+0000","last_clean_scrub_stamp":"2023-05-22T15:45:03.797855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:41:00.086136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":9,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-05-22T15:45:05.266564+0000","last_change":"2023-05-22T15:45:05.237005+0000","last_active":"2023-05-22T15:45:05.266564+0000","last_peered":"2023-05-22T15:45:05.266564+0000","last_clean":"2023-05-22T15:45:05.266564+0000","last_became_active":"2023-05-22T15:45:05.236224+0000","last_became_peered":"2023-05-22T15:45:05.236224+0000","last_unstale":"2023-05-22T15:45:05.266564+0000","last_undegraded":"2023-05-22T15:45:05.266564+0000","last_fullsized":"2023-05-22T15:45:05.266564+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_clean_scrub_stamp":"2023-05-22T15:45:04.802083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T03:42:26.869043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":10,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-05-22T15:45:05.213382+0000","last_change":"2023-05-22T15:45:04.848269+0000","last_active":"2023-05-22T15:45:05.213382+0000","last_peered":"2023-05-22T15:45:05.213382+0000","last_clean":"2023-05-22T15:45:05.213382+0000","last_became_active":"2023-05-22T15:45:04.847276+0000","last_became_peered":"2023-05-22T15:45:04.847276+0000","last_unstale":"2023-05-22T15:45:05.213382+0000","last_undegraded":"2023-05-22T15:45:05.213382+0000","last_fullsized":"2023-05-22T15:45:05.213382+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:45:03.797855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:45:03.797855+0000","last_clean_scrub_stamp":"2023-05-22T15:45:03.797855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:50:33.132462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":9,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-05-22T15:45:05.269769+0000","last_change":"2023-05-22T15:45:05.263355+0000","last_active":"2023-05-22T15:45:05.269769+0000","last_peered":"2023-05-22T15:45:05.269769+0000","last_clean":"2023-05-22T15:45:05.269769+0000","last_became_active":"2023-05-22T15:45:05.262691+0000","last_became_peered":"2023-05-22T15:45:05.262691+0000","last_unstale":"2023-05-22T15:45:05.269769+0000","last_undegraded":"2023-05-22T15:45:05.269769+0000","last_fullsized":"2023-05-22T15:45:05.269769+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_clean_scrub_stamp":"2023-05-22T15:45:04.802083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T18:19:40.408404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":10,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-05-22T15:45:05.213848+0000","last_change":"2023-05-22T15:45:04.868721+0000","last_active":"2023-05-22T15:45:05.213848+0000","last_peered":"2023-05-22T15:45:05.213848+0000","last_clean":"2023-05-22T15:45:05.213848+0000","last_became_active":"2023-05-22T15:45:04.868048+0000","last_became_peered":"2023-05-22T15:45:04.868048+0000","last_unstale":"2023-05-22T15:45:05.213848+0000","last_undegraded":"2023-05-22T15:45:05.213848+0000","last_fullsized":"2023-05-22T15:45:05.213848+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:45:03.797855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:45:03.797855+0000","last_clean_scrub_stamp":"2023-05-22T15:45:03.797855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T16:34:14.156084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.3","version":"0'0","reported_seq":9,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-05-22T15:45:05.264965+0000","last_change":"2023-05-22T15:45:05.253288+0000","last_active":"2023-05-22T15:45:05.264965+0000","last_peered":"2023-05-22T15:45:05.264965+0000","last_clean":"2023-05-22T15:45:05.264965+0000","last_became_active":"2023-05-22T15:45:05.251454+0000","last_became_peered":"2023-05-22T15:45:05.251454+0000","last_unstale":"2023-05-22T15:45:05.264965+0000","last_undegraded":"2023-05-22T15:45:05.264965+0000","last_fullsized":"2023-05-22T15:45:05.264965+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_clean_scrub_stamp":"2023-05-22T15:45:04.802083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T03:18:50.763567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":10,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-05-22T15:45:05.245450+0000","last_change":"2023-05-22T15:45:04.846123+0000","last_active":"2023-05-22T15:45:05.245450+0000","last_peered":"2023-05-22T15:45:05.245450+0000","last_clean":"2023-05-22T15:45:05.245450+0000","last_became_active":"2023-05-22T15:45:04.845353+0000","last_became_peered":"2023-05-22T15:45:04.845353+0000","last_unstale":"2023-05-22T15:45:05.245450+0000","last_undegraded":"2023-05-22T15:45:05.245450+0000","last_fullsized":"2023-05-22T15:45:05.245450+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:45:03.797855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:45:03.797855+0000","last_clean_scrub_stamp":"2023-05-22T15:45:03.797855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T16:46:49.576208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.0","version":"0'0","reported_seq":9,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-05-22T15:45:05.289712+0000","last_change":"2023-05-22T15:45:05.280628+0000","last_active":"2023-05-22T15:45:05.289712+0000","last_peered":"2023-05-22T15:45:05.289712+0000","last_clean":"2023-05-22T15:45:05.289712+0000","last_became_active":"2023-05-22T15:45:05.279793+0000","last_became_peered":"2023-05-22T15:45:05.279793+0000","last_unstale":"2023-05-22T15:45:05.289712+0000","last_undegraded":"2023-05-22T15:45:05.289712+0000","last_fullsized":"2023-05-22T15:45:05.289712+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_clean_scrub_stamp":"2023-05-22T15:45:04.802083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T03:42:53.382337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1","version":"0'0","reported_seq":10,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-05-22T15:45:05.207719+0000","last_change":"2023-05-22T15:45:04.846570+0000","last_active":"2023-05-22T15:45:05.207719+0000","last_peered":"2023-05-22T15:45:05.207719+0000","last_clean":"2023-05-22T15:45:05.207719+0000","last_became_active":"2023-05-22T15:45:04.846265+0000","last_became_peered":"2023-05-22T15:45:04.846265+0000","last_unstale":"2023-05-22T15:45:05.207719+0000","last_undegraded":"2023-05-22T15:45:05.207719+0000","last_fullsized":"2023-05-22T15:45:05.207719+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:45:03.797855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:45:03.797855+0000","last_clean_scrub_stamp":"2023-05-22T15:45:03.797855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T03:34:25.195968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1","version":"0'0","reported_seq":9,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-05-22T15:45:05.241158+0000","last_change":"2023-05-22T15:45:05.235400+0000","last_active":"2023-05-22T15:45:05.241158+0000","last_peered":"2023-05-22T15:45:05.241158+0000","last_clean":"2023-05-22T15:45:05.241158+0000","last_became_active":"2023-05-22T15:45:05.235120+0000","last_became_peered":"2023-05-22T15:45:05.235120+0000","last_unstale":"2023-05-22T15:45:05.241158+0000","last_undegraded":"2023-05-22T15:45:05.241158+0000","last_fullsized":"2023-05-22T15:45:05.241158+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_clean_scrub_stamp":"2023-05-22T15:45:04.802083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:01:37.137003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":10,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-05-22T15:45:05.219684+0000","last_change":"2023-05-22T15:45:04.845876+0000","last_active":"2023-05-22T15:45:05.219684+0000","last_peered":"2023-05-22T15:45:05.219684+0000","last_clean":"2023-05-22T15:45:05.219684+0000","last_became_active":"2023-05-22T15:45:04.845574+0000","last_became_peered":"2023-05-22T15:45:04.845574+0000","last_unstale":"2023-05-22T15:45:05.219684+0000","last_undegraded":"2023-05-22T15:45:05.219684+0000","last_fullsized":"2023-05-22T15:45:05.219684+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:45:03.797855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:45:03.797855+0000","last_clean_scrub_stamp":"2023-05-22T15:45:03.797855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:06:39.667988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.10","version":"0'0","reported_seq":9,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-05-22T15:45:05.243392+0000","last_change":"2023-05-22T15:45:05.241940+0000","last_active":"2023-05-22T15:45:05.243392+0000","last_peered":"2023-05-22T15:45:05.243392+0000","last_clean":"2023-05-22T15:45:05.243392+0000","last_became_active":"2023-05-22T15:45:05.241683+0000","last_became_peered":"2023-05-22T15:45:05.241683+0000","last_unstale":"2023-05-22T15:45:05.243392+0000","last_undegraded":"2023-05-22T15:45:05.243392+0000","last_fullsized":"2023-05-22T15:45:05.243392+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_clean_scrub_stamp":"2023-05-22T15:45:04.802083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:23:52.692138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":10,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-05-22T15:45:05.216101+0000","last_change":"2023-05-22T15:45:04.860420+0000","last_active":"2023-05-22T15:45:05.216101+0000","last_peered":"2023-05-22T15:45:05.216101+0000","last_clean":"2023-05-22T15:45:05.216101+0000","last_became_active":"2023-05-22T15:45:04.859659+0000","last_became_peered":"2023-05-22T15:45:04.859659+0000","last_unstale":"2023-05-22T15:45:05.216101+0000","last_undegraded":"2023-05-22T15:45:05.216101+0000","last_fullsized":"2023-05-22T15:45:05.216101+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:45:03.797855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:45:03.797855+0000","last_clean_scrub_stamp":"2023-05-22T15:45:03.797855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:44:21.409644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"120'227","reported_seq":662,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-05-22T15:45:04.855004+0000","last_change":"2023-05-22T15:26:39.318204+0000","last_active":"2023-05-22T15:45:04.855004+0000","last_peered":"2023-05-22T15:45:04.855004+0000","last_clean":"2023-05-22T15:45:04.855004+0000","last_became_active":"2023-05-22T15:26:39.315370+0000","last_became_peered":"2023-05-22T15:26:39.315370+0000","last_unstale":"2023-05-22T15:45:04.855004+0000","last_undegraded":"2023-05-22T15:45:04.855004+0000","last_fullsized":"2023-05-22T15:45:04.855004+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:38.020076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:38.020076+0000","last_clean_scrub_stamp":"2023-05-22T15:26:38.020076+0000","objects_scrubbed":0,"log_size":227,"log_dups_size":0,"ondisk_log_size":227,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:06:39.207543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1076,"num_read_kb":2160,"num_write":359,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":125,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-05-22T15:45:04.853121+0000","last_change":"2023-05-22T15:26:44.971951+0000","last_active":"2023-05-22T15:45:04.853121+0000","last_peered":"2023-05-22T15:45:04.853121+0000","last_clean":"2023-05-22T15:45:04.853121+0000","last_became_active":"2023-05-22T15:26:44.970872+0000","last_became_peered":"2023-05-22T15:26:44.970872+0000","last_unstale":"2023-05-22T15:45:04.853121+0000","last_undegraded":"2023-05-22T15:45:04.853121+0000","last_fullsized":"2023-05-22T15:45:04.853121+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:55:34.752949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.13","version":"0'0","reported_seq":9,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-05-22T15:45:05.271605+0000","last_change":"2023-05-22T15:45:05.271178+0000","last_active":"2023-05-22T15:45:05.271605+0000","last_peered":"2023-05-22T15:45:05.271605+0000","last_clean":"2023-05-22T15:45:05.271605+0000","last_became_active":"2023-05-22T15:45:05.270957+0000","last_became_peered":"2023-05-22T15:45:05.270957+0000","last_unstale":"2023-05-22T15:45:05.271605+0000","last_undegraded":"2023-05-22T15:45:05.271605+0000","last_fullsized":"2023-05-22T15:45:05.271605+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_clean_scrub_stamp":"2023-05-22T15:45:04.802083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T18:19:40.408404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":10,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-05-22T15:45:05.250590+0000","last_change":"2023-05-22T15:45:04.845921+0000","last_active":"2023-05-22T15:45:05.250590+0000","last_peered":"2023-05-22T15:45:05.250590+0000","last_clean":"2023-05-22T15:45:05.250590+0000","last_became_active":"2023-05-22T15:45:04.845130+0000","last_became_peered":"2023-05-22T15:45:04.845130+0000","last_unstale":"2023-05-22T15:45:05.250590+0000","last_undegraded":"2023-05-22T15:45:05.250590+0000","last_fullsized":"2023-05-22T15:45:05.250590+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:45:03.797855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:45:03.797855+0000","last_clean_scrub_stamp":"2023-05-22T15:45:03.797855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T16:17:05.678138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":119,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-05-22T15:45:04.846752+0000","last_change":"2023-05-22T15:26:44.977198+0000","last_active":"2023-05-22T15:45:04.846752+0000","last_peered":"2023-05-22T15:45:04.846752+0000","last_clean":"2023-05-22T15:45:04.846752+0000","last_became_active":"2023-05-22T15:26:44.976445+0000","last_became_peered":"2023-05-22T15:26:44.976445+0000","last_unstale":"2023-05-22T15:45:04.846752+0000","last_undegraded":"2023-05-22T15:45:04.846752+0000","last_fullsized":"2023-05-22T15:45:04.846752+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:12:41.464054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":9,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-05-22T15:45:05.228187+0000","last_change":"2023-05-22T15:45:05.225003+0000","last_active":"2023-05-22T15:45:05.228187+0000","last_peered":"2023-05-22T15:45:05.228187+0000","last_clean":"2023-05-22T15:45:05.228187+0000","last_became_active":"2023-05-22T15:45:05.223711+0000","last_became_peered":"2023-05-22T15:45:05.223711+0000","last_unstale":"2023-05-22T15:45:05.228187+0000","last_undegraded":"2023-05-22T15:45:05.228187+0000","last_fullsized":"2023-05-22T15:45:05.228187+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_clean_scrub_stamp":"2023-05-22T15:45:04.802083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:41:01.090364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.13","version":"131'2","reported_seq":12,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-05-22T15:45:05.271146+0000","last_change":"2023-05-22T15:45:04.867655+0000","last_active":"2023-05-22T15:45:05.271146+0000","last_peered":"2023-05-22T15:45:05.271146+0000","last_clean":"2023-05-22T15:45:05.271146+0000","last_became_active":"2023-05-22T15:45:04.867350+0000","last_became_peered":"2023-05-22T15:45:04.867350+0000","last_unstale":"2023-05-22T15:45:05.271146+0000","last_undegraded":"2023-05-22T15:45:05.271146+0000","last_fullsized":"2023-05-22T15:45:05.271146+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:45:03.797855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:45:03.797855+0000","last_clean_scrub_stamp":"2023-05-22T15:45:03.797855+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:53:11.065522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":119,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-05-22T15:45:04.820605+0000","last_change":"2023-05-22T15:26:44.969469+0000","last_active":"2023-05-22T15:45:04.820605+0000","last_peered":"2023-05-22T15:45:04.820605+0000","last_clean":"2023-05-22T15:45:04.820605+0000","last_became_active":"2023-05-22T15:26:44.969083+0000","last_became_peered":"2023-05-22T15:26:44.969083+0000","last_unstale":"2023-05-22T15:45:04.820605+0000","last_undegraded":"2023-05-22T15:45:04.820605+0000","last_fullsized":"2023-05-22T15:45:04.820605+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:29:23.670067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.11","version":"0'0","reported_seq":9,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-05-22T15:45:05.237152+0000","last_change":"2023-05-22T15:45:05.236102+0000","last_active":"2023-05-22T15:45:05.237152+0000","last_peered":"2023-05-22T15:45:05.237152+0000","last_clean":"2023-05-22T15:45:05.237152+0000","last_became_active":"2023-05-22T15:45:05.235808+0000","last_became_peered":"2023-05-22T15:45:05.235808+0000","last_unstale":"2023-05-22T15:45:05.237152+0000","last_undegraded":"2023-05-22T15:45:05.237152+0000","last_fullsized":"2023-05-22T15:45:05.237152+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_clean_scrub_stamp":"2023-05-22T15:45:04.802083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:17:05.563955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.10","version":"131'8","reported_seq":18,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-05-22T15:45:05.240165+0000","last_change":"2023-05-22T15:45:04.841261+0000","last_active":"2023-05-22T15:45:05.240165+0000","last_peered":"2023-05-22T15:45:05.240165+0000","last_clean":"2023-05-22T15:45:05.240165+0000","last_became_active":"2023-05-22T15:45:04.840898+0000","last_became_peered":"2023-05-22T15:45:04.840898+0000","last_unstale":"2023-05-22T15:45:05.240165+0000","last_undegraded":"2023-05-22T15:45:05.240165+0000","last_fullsized":"2023-05-22T15:45:05.240165+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:45:03.797855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:45:03.797855+0000","last_clean_scrub_stamp":"2023-05-22T15:45:03.797855+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:16:37.560794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":125,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-05-22T15:45:04.852215+0000","last_change":"2023-05-22T15:26:44.976632+0000","last_active":"2023-05-22T15:45:04.852215+0000","last_peered":"2023-05-22T15:45:04.852215+0000","last_clean":"2023-05-22T15:45:04.852215+0000","last_became_active":"2023-05-22T15:26:44.975546+0000","last_became_peered":"2023-05-22T15:26:44.975546+0000","last_unstale":"2023-05-22T15:45:04.852215+0000","last_undegraded":"2023-05-22T15:45:04.852215+0000","last_fullsized":"2023-05-22T15:45:04.852215+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:42:57.973030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.17","version":"0'0","reported_seq":9,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-05-22T15:45:05.274342+0000","last_change":"2023-05-22T15:45:05.266421+0000","last_active":"2023-05-22T15:45:05.274342+0000","last_peered":"2023-05-22T15:45:05.274342+0000","last_clean":"2023-05-22T15:45:05.274342+0000","last_became_active":"2023-05-22T15:45:05.254074+0000","last_became_peered":"2023-05-22T15:45:05.254074+0000","last_unstale":"2023-05-22T15:45:05.274342+0000","last_undegraded":"2023-05-22T15:45:05.274342+0000","last_fullsized":"2023-05-22T15:45:05.274342+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_clean_scrub_stamp":"2023-05-22T15:45:04.802083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:25:31.706898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":10,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-05-22T15:45:05.212610+0000","last_change":"2023-05-22T15:45:04.852022+0000","last_active":"2023-05-22T15:45:05.212610+0000","last_peered":"2023-05-22T15:45:05.212610+0000","last_clean":"2023-05-22T15:45:05.212610+0000","last_became_active":"2023-05-22T15:45:04.851734+0000","last_became_peered":"2023-05-22T15:45:04.851734+0000","last_unstale":"2023-05-22T15:45:05.212610+0000","last_undegraded":"2023-05-22T15:45:05.212610+0000","last_fullsized":"2023-05-22T15:45:05.212610+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:45:03.797855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:45:03.797855+0000","last_clean_scrub_stamp":"2023-05-22T15:45:03.797855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:25:30.702670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":111,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-05-22T15:45:04.838954+0000","last_change":"2023-05-22T15:26:45.091475+0000","last_active":"2023-05-22T15:45:04.838954+0000","last_peered":"2023-05-22T15:45:04.838954+0000","last_clean":"2023-05-22T15:45:04.838954+0000","last_became_active":"2023-05-22T15:26:45.091143+0000","last_became_peered":"2023-05-22T15:26:45.091143+0000","last_unstale":"2023-05-22T15:45:04.838954+0000","last_undegraded":"2023-05-22T15:45:04.838954+0000","last_fullsized":"2023-05-22T15:45:04.838954+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:12:41.464054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.16","version":"0'0","reported_seq":9,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-05-22T15:45:05.221872+0000","last_change":"2023-05-22T15:45:05.219266+0000","last_active":"2023-05-22T15:45:05.221872+0000","last_peered":"2023-05-22T15:45:05.221872+0000","last_clean":"2023-05-22T15:45:05.221872+0000","last_became_active":"2023-05-22T15:45:05.218279+0000","last_became_peered":"2023-05-22T15:45:05.218279+0000","last_unstale":"2023-05-22T15:45:05.221872+0000","last_undegraded":"2023-05-22T15:45:05.221872+0000","last_fullsized":"2023-05-22T15:45:05.221872+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_clean_scrub_stamp":"2023-05-22T15:45:04.802083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:17:53.852853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":10,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-05-22T15:45:05.208422+0000","last_change":"2023-05-22T15:45:04.846152+0000","last_active":"2023-05-22T15:45:05.208422+0000","last_peered":"2023-05-22T15:45:05.208422+0000","last_clean":"2023-05-22T15:45:05.208422+0000","last_became_active":"2023-05-22T15:45:04.845709+0000","last_became_peered":"2023-05-22T15:45:04.845709+0000","last_unstale":"2023-05-22T15:45:05.208422+0000","last_undegraded":"2023-05-22T15:45:05.208422+0000","last_fullsized":"2023-05-22T15:45:05.208422+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:45:03.797855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:45:03.797855+0000","last_clean_scrub_stamp":"2023-05-22T15:45:03.797855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:44:21.409644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":118,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-05-22T15:45:04.856499+0000","last_change":"2023-05-22T15:26:45.092367+0000","last_active":"2023-05-22T15:45:04.856499+0000","last_peered":"2023-05-22T15:45:04.856499+0000","last_clean":"2023-05-22T15:45:04.856499+0000","last_became_active":"2023-05-22T15:26:45.091951+0000","last_became_peered":"2023-05-22T15:26:45.091951+0000","last_unstale":"2023-05-22T15:45:04.856499+0000","last_undegraded":"2023-05-22T15:45:04.856499+0000","last_fullsized":"2023-05-22T15:45:04.856499+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:42:57.973030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.15","version":"0'0","reported_seq":9,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-05-22T15:45:05.285475+0000","last_change":"2023-05-22T15:45:05.283232+0000","last_active":"2023-05-22T15:45:05.285475+0000","last_peered":"2023-05-22T15:45:05.285475+0000","last_clean":"2023-05-22T15:45:05.285475+0000","last_became_active":"2023-05-22T15:45:05.282448+0000","last_became_peered":"2023-05-22T15:45:05.282448+0000","last_unstale":"2023-05-22T15:45:05.285475+0000","last_undegraded":"2023-05-22T15:45:05.285475+0000","last_fullsized":"2023-05-22T15:45:05.285475+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_clean_scrub_stamp":"2023-05-22T15:45:04.802083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:33:36.433252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.14","version":"131'6","reported_seq":16,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-05-22T15:45:05.260939+0000","last_change":"2023-05-22T15:45:04.849926+0000","last_active":"2023-05-22T15:45:05.260939+0000","last_peered":"2023-05-22T15:45:05.260939+0000","last_clean":"2023-05-22T15:45:05.260939+0000","last_became_active":"2023-05-22T15:45:04.849626+0000","last_became_peered":"2023-05-22T15:45:04.849626+0000","last_unstale":"2023-05-22T15:45:05.260939+0000","last_undegraded":"2023-05-22T15:45:05.260939+0000","last_fullsized":"2023-05-22T15:45:05.260939+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:45:03.797855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:45:03.797855+0000","last_clean_scrub_stamp":"2023-05-22T15:45:03.797855+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:02:33.642419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1548,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":111,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-05-22T15:45:04.838884+0000","last_change":"2023-05-22T15:26:44.978858+0000","last_active":"2023-05-22T15:45:04.838884+0000","last_peered":"2023-05-22T15:45:04.838884+0000","last_clean":"2023-05-22T15:45:04.838884+0000","last_became_active":"2023-05-22T15:26:44.978443+0000","last_became_peered":"2023-05-22T15:26:44.978443+0000","last_unstale":"2023-05-22T15:45:04.838884+0000","last_undegraded":"2023-05-22T15:45:04.838884+0000","last_fullsized":"2023-05-22T15:45:04.838884+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:46:29.290243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.14","version":"0'0","reported_seq":9,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-05-22T15:45:05.249290+0000","last_change":"2023-05-22T15:45:05.248517+0000","last_active":"2023-05-22T15:45:05.249290+0000","last_peered":"2023-05-22T15:45:05.249290+0000","last_clean":"2023-05-22T15:45:05.249290+0000","last_became_active":"2023-05-22T15:45:05.248227+0000","last_became_peered":"2023-05-22T15:45:05.248227+0000","last_unstale":"2023-05-22T15:45:05.249290+0000","last_undegraded":"2023-05-22T15:45:05.249290+0000","last_fullsized":"2023-05-22T15:45:05.249290+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_clean_scrub_stamp":"2023-05-22T15:45:04.802083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T03:35:59.754988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.15","version":"131'3","reported_seq":13,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-05-22T15:45:05.237375+0000","last_change":"2023-05-22T15:45:04.848230+0000","last_active":"2023-05-22T15:45:05.237375+0000","last_peered":"2023-05-22T15:45:05.237375+0000","last_clean":"2023-05-22T15:45:05.237375+0000","last_became_active":"2023-05-22T15:45:04.847778+0000","last_became_peered":"2023-05-22T15:45:04.847778+0000","last_unstale":"2023-05-22T15:45:05.237375+0000","last_undegraded":"2023-05-22T15:45:05.237375+0000","last_fullsized":"2023-05-22T15:45:05.237375+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:45:03.797855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:45:03.797855+0000","last_clean_scrub_stamp":"2023-05-22T15:45:03.797855+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:17:04.559727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":112,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-05-22T15:45:04.833279+0000","last_change":"2023-05-22T15:26:44.982421+0000","last_active":"2023-05-22T15:45:04.833279+0000","last_peered":"2023-05-22T15:45:04.833279+0000","last_clean":"2023-05-22T15:45:04.833279+0000","last_became_active":"2023-05-22T15:26:44.981586+0000","last_became_peered":"2023-05-22T15:26:44.981586+0000","last_unstale":"2023-05-22T15:45:04.833279+0000","last_undegraded":"2023-05-22T15:45:04.833279+0000","last_fullsized":"2023-05-22T15:45:04.833279+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:14:11.156544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":17,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":2330,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1076,"num_read_kb":2160,"num_write":359,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":227,"ondisk_log_size":227,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575069,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35652,"kb_used_data":2172,"kb_used_omap":0,"kb_used_meta":33472,"kb_avail":94336188,"statfs":{"total":96636764160,"available":96600256512,"internally_reserved":0,"allocated":2224128,"data_stored":1445607,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":34275328},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 15:44:03 2023","interfaces":[{"interface":"back","average":{"1min":0.999,"5min":0.874,"15min":0.874},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":2.637,"5min":2.637,"15min":2.637},"last":1.1830000000000001},{"interface":"front","average":{"1min":0.93899999999999995,"5min":0.875,"15min":0.89100000000000001},"min":{"1min":0.502,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":3.0449999999999999,"5min":3.0449999999999999,"15min":3.0449999999999999},"last":1.1040000000000001}]},{"osd":1,"last update":"Mon May 22 15:44:03 2023","interfaces":[{"interface":"back","average":{"1min":0.88900000000000001,"5min":0.93500000000000005,"15min":0.97699999999999998},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":1.2270000000000001,"5min":1.385,"15min":1.389},"last":0.83799999999999997},{"interface":"front","average":{"1min":1.0169999999999999,"5min":0.997,"15min":1.0209999999999999},"min":{"1min":0.63800000000000001,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":2.7269999999999999,"5min":2.7269999999999999,"15min":2.7269999999999999},"last":1.333}]},{"osd":2,"last update":"Mon May 22 15:44:03 2023","interfaces":[{"interface":"back","average":{"1min":0.78900000000000003,"5min":0.83999999999999997,"15min":0.877},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.1519999999999999,"5min":1.2130000000000001,"15min":1.244},"last":0.76200000000000001},{"interface":"front","average":{"1min":0.99099999999999999,"5min":0.92700000000000005,"15min":0.95599999999999996},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":3.141,"5min":3.141,"15min":3.141},"last":1.5049999999999999}]},{"osd":3,"last update":"Mon May 22 15:44:03 2023","interfaces":[{"interface":"back","average":{"1min":1.0289999999999999,"5min":0.95799999999999996,"15min":0.95999999999999996},"min":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":2.8149999999999999,"5min":2.8149999999999999,"15min":2.8149999999999999},"last":1.569},{"interface":"front","average":{"1min":0.91500000000000004,"5min":0.93500000000000005,"15min":0.97299999999999998},"min":{"1min":0.504,"5min":0.504,"15min":0.504},"max":{"1min":1.101,"5min":1.3029999999999999,"15min":1.4710000000000001},"last":1.389}]},{"osd":4,"last update":"Mon May 22 15:44:03 2023","interfaces":[{"interface":"back","average":{"1min":1.0429999999999999,"5min":0.93799999999999994,"15min":0.89900000000000002},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":3.472,"5min":3.472,"15min":3.472},"last":1.591},{"interface":"front","average":{"1min":1.915,"5min":1.175,"15min":1.036},"min":{"1min":0.56000000000000005,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":15.627000000000001,"5min":15.627000000000001,"15min":15.627000000000001},"last":1.4199999999999999}]},{"osd":5,"last update":"Mon May 22 15:44:03 2023","interfaces":[{"interface":"back","average":{"1min":1.202,"5min":1.0569999999999999,"15min":1.0069999999999999},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":4.5599999999999996,"5min":4.5599999999999996,"15min":4.5599999999999996},"last":1.2370000000000001},{"interface":"front","average":{"1min":1.2709999999999999,"5min":1.1439999999999999,"15min":1.1339999999999999},"min":{"1min":0.63800000000000001,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":4.3799999999999999,"5min":4.3799999999999999,"15min":4.3799999999999999},"last":1.7649999999999999}]},{"osd":6,"last update":"Mon May 22 15:44:03 2023","interfaces":[{"interface":"back","average":{"1min":1.381,"5min":1.113,"15min":1.077},"min":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":6.5069999999999997,"5min":6.5069999999999997,"15min":6.5069999999999997},"last":1.8660000000000001},{"interface":"front","average":{"1min":1.2849999999999999,"5min":1.1439999999999999,"15min":1.135},"min":{"1min":0.62,"5min":0.62,"15min":0.62},"max":{"1min":3.3500000000000001,"5min":3.3500000000000001,"15min":3.3500000000000001},"last":2.0129999999999999}]}]},{"osd":6,"up_from":13,"seq":55834575070,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31676,"kb_used_data":1588,"kb_used_omap":0,"kb_used_meta":30080,"kb_avail":94340164,"statfs":{"total":96636764160,"available":96604327936,"internally_reserved":0,"allocated":1626112,"data_stored":854709,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":30801920},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 15:44:14 2023","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":0.85799999999999998,"15min":0.89800000000000002},"min":{"1min":0.57499999999999996,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":1.26,"5min":1.486,"15min":1.486},"last":0.73099999999999998},{"interface":"front","average":{"1min":0.97599999999999998,"5min":0.92000000000000004,"15min":0.91200000000000003},"min":{"1min":0.64300000000000002,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":1.6240000000000001,"5min":1.6240000000000001,"15min":1.6240000000000001},"last":1.1240000000000001}]},{"osd":1,"last update":"Mon May 22 15:44:14 2023","interfaces":[{"interface":"back","average":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.92300000000000004},"min":{"1min":0.64600000000000002,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":1.4039999999999999,"5min":1.4039999999999999,"15min":1.4039999999999999},"last":0.997},{"interface":"front","average":{"1min":0.879,"5min":0.91500000000000004,"15min":0.94499999999999995},"min":{"1min":0.65500000000000003,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":1.2509999999999999,"5min":1.554,"15min":1.554},"last":0.78300000000000003}]},{"osd":2,"last update":"Mon May 22 15:44:14 2023","interfaces":[{"interface":"back","average":{"1min":0.97499999999999998,"5min":0.95499999999999996,"15min":0.98399999999999999},"min":{"1min":0.68100000000000005,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":1.3200000000000001,"5min":1.425,"15min":1.425},"last":1.036},{"interface":"front","average":{"1min":1.0580000000000001,"5min":1.024,"15min":1.044},"min":{"1min":0.78300000000000003,"5min":0.442,"15min":0.442},"max":{"1min":1.673,"5min":1.673,"15min":1.673},"last":1.0860000000000001}]},{"osd":3,"last update":"Mon May 22 15:44:14 2023","interfaces":[{"interface":"back","average":{"1min":0.97699999999999998,"5min":0.98099999999999998,"15min":0.98399999999999999},"min":{"1min":0.753,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":1.329,"5min":1.679,"15min":1.679},"last":0.879},{"interface":"front","average":{"1min":0.96799999999999997,"5min":0.94899999999999995,"15min":0.95899999999999996},"min":{"1min":0.53200000000000003,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.3009999999999999,"5min":1.4570000000000001,"15min":1.4570000000000001},"last":1.1559999999999999}]},{"osd":4,"last update":"Mon May 22 15:44:14 2023","interfaces":[{"interface":"back","average":{"1min":1.0109999999999999,"5min":0.95399999999999996,"15min":0.92400000000000004},"min":{"1min":0.59699999999999998,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":1.581,"5min":1.6859999999999999,"15min":1.6859999999999999},"last":0.91700000000000004},{"interface":"front","average":{"1min":1.1020000000000001,"5min":1.03,"15min":1.0229999999999999},"min":{"1min":0.65800000000000003,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":1.452,"5min":1.8300000000000001,"15min":1.8300000000000001},"last":0.95799999999999996}]},{"osd":5,"last update":"Mon May 22 15:44:14 2023","interfaces":[{"interface":"back","average":{"1min":1.0880000000000001,"5min":1.0029999999999999,"15min":0.97499999999999998},"min":{"1min":0.76600000000000001,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":1.296,"5min":1.6499999999999999,"15min":1.6499999999999999},"last":1.2529999999999999},{"interface":"front","average":{"1min":1.1830000000000001,"5min":1.139,"15min":1.1020000000000001},"min":{"1min":0.71699999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.7689999999999999,"5min":1.7689999999999999,"15min":1.7689999999999999},"last":1.1950000000000001}]},{"osd":7,"last update":"Mon May 22 15:44:14 2023","interfaces":[{"interface":"back","average":{"1min":1.1779999999999999,"5min":1.1359999999999999,"15min":1.1259999999999999},"min":{"1min":0.76700000000000002,"5min":0.37,"15min":0.37},"max":{"1min":1.853,"5min":1.9179999999999999,"15min":1.9179999999999999},"last":1.224},{"interface":"front","average":{"1min":1.1619999999999999,"5min":1.1359999999999999,"15min":1.1279999999999999},"min":{"1min":0.78700000000000003,"5min":0.442,"15min":0.442},"max":{"1min":1.518,"5min":1.8520000000000001,"15min":1.8520000000000001},"last":1.3200000000000001}]}]},{"osd":5,"up_from":13,"seq":55834575070,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":25536,"kb_used_data":1592,"kb_used_omap":0,"kb_used_meta":23936,"kb_avail":94346304,"statfs":{"total":96636764160,"available":96610615296,"internally_reserved":0,"allocated":1630208,"data_stored":855227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":24510464},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 15:44:09 2023","interfaces":[{"interface":"back","average":{"1min":0.85799999999999998,"5min":0.83899999999999997,"15min":0.88600000000000001},"min":{"1min":0.52000000000000002,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.171,"5min":1.3440000000000001,"15min":1.3440000000000001},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.81299999999999994,"5min":0.86399999999999999,"15min":0.88200000000000001},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":1.095,"5min":2.1699999999999999,"15min":2.1699999999999999},"last":1.0269999999999999}]},{"osd":1,"last update":"Mon May 22 15:44:09 2023","interfaces":[{"interface":"back","average":{"1min":0.95999999999999996,"5min":0.96199999999999997,"15min":0.997},"min":{"1min":0.61499999999999999,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":1.248,"5min":1.9370000000000001,"15min":1.9370000000000001},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.97099999999999997,"5min":0.96799999999999997,"15min":1.0249999999999999},"min":{"1min":0.65000000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.375,"5min":1.375,"15min":1.375},"last":0.89700000000000002}]},{"osd":2,"last update":"Mon May 22 15:44:09 2023","interfaces":[{"interface":"back","average":{"1min":0.88600000000000001,"5min":0.94399999999999995,"15min":0.93799999999999994},"min":{"1min":0.57499999999999996,"5min":0.504,"15min":0.504},"max":{"1min":1.1659999999999999,"5min":1.9830000000000001,"15min":1.9830000000000001},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.95699999999999996,"5min":0.94999999999999996,"15min":0.94999999999999996},"min":{"1min":0.72499999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"max":{"1min":1.121,"5min":2.1120000000000001,"15min":2.1120000000000001},"last":0.69399999999999995}]},{"osd":3,"last update":"Mon May 22 15:44:09 2023","interfaces":[{"interface":"back","average":{"1min":0.92700000000000005,"5min":0.94299999999999995,"15min":0.95599999999999996},"min":{"1min":0.69099999999999995,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.2629999999999999,"5min":1.4419999999999999,"15min":1.4419999999999999},"last":0.80700000000000005},{"interface":"front","average":{"1min":1.024,"5min":1.036,"15min":1.034},"min":{"1min":0.69699999999999995,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":1.337,"5min":2.52,"15min":2.52},"last":0.93600000000000005}]},{"osd":4,"last update":"Mon May 22 15:44:09 2023","interfaces":[{"interface":"back","average":{"1min":0.93799999999999994,"5min":0.97299999999999998,"15min":0.93200000000000005},"min":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.57599999999999996},"max":{"1min":1.28,"5min":2.226,"15min":2.226},"last":0.77400000000000002},{"interface":"front","average":{"1min":1.0609999999999999,"5min":1.04,"15min":1.0269999999999999},"min":{"1min":0.78200000000000003,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.3859999999999999,"5min":1.6859999999999999,"15min":1.6859999999999999},"last":0.98599999999999999}]},{"osd":6,"last update":"Mon May 22 15:44:09 2023","interfaces":[{"interface":"back","average":{"1min":1.034,"5min":1.0620000000000001,"15min":1.032},"min":{"1min":0.68400000000000005,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.246,"5min":2.831,"15min":2.831},"last":1.0860000000000001},{"interface":"front","average":{"1min":1.119,"5min":1.0980000000000001,"15min":1.0720000000000001},"min":{"1min":0.65000000000000002,"5min":0.629,"15min":0.629},"max":{"1min":1.3560000000000001,"5min":2.3690000000000002,"15min":2.3690000000000002},"last":0.87}]},{"osd":7,"last update":"Mon May 22 15:44:09 2023","interfaces":[{"interface":"back","average":{"1min":1.1259999999999999,"5min":1.1279999999999999,"15min":1.113},"min":{"1min":0.72899999999999998,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":1.4690000000000001,"5min":1.849,"15min":1.849},"last":1.0609999999999999},{"interface":"front","average":{"1min":1.1739999999999999,"5min":1.1899999999999999,"15min":1.1619999999999999},"min":{"1min":0.78500000000000003,"5min":0.75700000000000001,"15min":0.71699999999999997},"max":{"1min":1.4319999999999999,"5min":2.7429999999999999,"15min":2.7429999999999999},"last":1.1319999999999999}]}]},{"osd":4,"up_from":13,"seq":55834575068,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27064,"kb_used_data":1584,"kb_used_omap":0,"kb_used_meta":25472,"kb_avail":94344776,"statfs":{"total":96636764160,"available":96609050624,"internally_reserved":0,"allocated":1622016,"data_stored":854639,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":26083328},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 15:44:15 2023","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":0.85999999999999999,"15min":0.90000000000000002},"min":{"1min":0.48999999999999999,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":1.1220000000000001,"5min":1.9910000000000001,"15min":1.9910000000000001},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.85099999999999998,"5min":0.89500000000000002,"15min":0.93400000000000005},"min":{"1min":0.499,"5min":0.373,"15min":0.373},"max":{"1min":1.1539999999999999,"5min":1.3779999999999999,"15min":1.633},"last":0.59899999999999998}]},{"osd":1,"last update":"Mon May 22 15:44:15 2023","interfaces":[{"interface":"back","average":{"1min":0.876,"5min":0.93400000000000005,"15min":0.96999999999999997},"min":{"1min":0.70899999999999996,"5min":0.436,"15min":0.436},"max":{"1min":1.24,"5min":1.325,"15min":1.383},"last":0.81599999999999995},{"interface":"front","average":{"1min":0.86699999999999999,"5min":0.85899999999999999,"15min":0.90800000000000003},"min":{"1min":0.48699999999999999,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.2410000000000001,"5min":1.298,"15min":1.3640000000000001},"last":0.68700000000000006}]},{"osd":2,"last update":"Mon May 22 15:44:15 2023","interfaces":[{"interface":"back","average":{"1min":0.88700000000000001,"5min":0.92800000000000005,"15min":0.94999999999999996},"min":{"1min":0.55800000000000005,"5min":0.495,"15min":0.495},"max":{"1min":1.1459999999999999,"5min":2.1019999999999999,"15min":2.1019999999999999},"last":0.879},{"interface":"front","average":{"1min":0.93500000000000005,"5min":0.97999999999999998,"15min":1.002},"min":{"1min":0.59299999999999997,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":1.1850000000000001,"5min":1.4399999999999999,"15min":1.6000000000000001},"last":0.67000000000000004}]},{"osd":3,"last update":"Mon May 22 15:44:15 2023","interfaces":[{"interface":"back","average":{"1min":0.93200000000000005,"5min":0.90400000000000003,"15min":0.94499999999999995},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":1.194,"5min":1.29,"15min":1.665},"last":0.92300000000000004},{"interface":"front","average":{"1min":0.93600000000000005,"5min":1.0189999999999999,"15min":1.028},"min":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":1.21,"5min":2.1720000000000002,"15min":2.1720000000000002},"last":1.048}]},{"osd":5,"last update":"Mon May 22 15:44:15 2023","interfaces":[{"interface":"back","average":{"1min":0.90000000000000002,"5min":0.99399999999999999,"15min":0.98099999999999998},"min":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.1830000000000001,"5min":1.472,"15min":1.6830000000000001},"last":1.1439999999999999},{"interface":"front","average":{"1min":1.0069999999999999,"5min":0.97899999999999998,"15min":0.97799999999999998},"min":{"1min":0.68100000000000005,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.365,"5min":1.401,"15min":1.6719999999999999},"last":1.103}]},{"osd":6,"last update":"Mon May 22 15:44:15 2023","interfaces":[{"interface":"back","average":{"1min":0.91700000000000004,"5min":0.995,"15min":0.97999999999999998},"min":{"1min":0.55700000000000005,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":1.3200000000000001,"5min":2.2349999999999999,"15min":2.2349999999999999},"last":1.3420000000000001},{"interface":"front","average":{"1min":1.0580000000000001,"5min":1.1160000000000001,"15min":1.1040000000000001},"min":{"1min":0.73999999999999999,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":1.284,"5min":1.8220000000000001,"15min":1.8220000000000001},"last":1.2769999999999999}]},{"osd":7,"last update":"Mon May 22 15:44:15 2023","interfaces":[{"interface":"back","average":{"1min":1.0349999999999999,"5min":1.052,"15min":1.054},"min":{"1min":0.76000000000000001,"5min":0.70799999999999996,"15min":0.67700000000000005},"max":{"1min":1.46,"5min":1.5800000000000001,"15min":1.5800000000000001},"last":1.2989999999999999},{"interface":"front","average":{"1min":1.1100000000000001,"5min":1.1539999999999999,"15min":1.139},"min":{"1min":0.78300000000000003,"5min":0.66500000000000004,"15min":0.66500000000000004},"max":{"1min":1.6559999999999999,"5min":2.306,"15min":2.306},"last":1.204}]}]},{"osd":3,"up_from":13,"seq":55834575068,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36168,"kb_used_data":1600,"kb_used_omap":0,"kb_used_meta":34560,"kb_avail":94335672,"statfs":{"total":96636764160,"available":96599728128,"internally_reserved":0,"allocated":1638400,"data_stored":856257,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":35389440},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 15:45:03 2023","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.60199999999999998,"15min":0.63200000000000001},"min":{"1min":0.252,"5min":0.252,"15min":0.19},"max":{"1min":1.1299999999999999,"5min":1.194,"15min":1.194},"last":0.97999999999999998},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.66800000000000004,"15min":0.70799999999999996},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":1.296,"5min":1.296,"15min":1.296},"last":0.66200000000000003}]},{"osd":1,"last update":"Mon May 22 15:45:03 2023","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.623,"15min":0.66300000000000003},"min":{"1min":0.36799999999999999,"5min":0.314,"15min":0.29499999999999998},"max":{"1min":1.2230000000000001,"5min":1.2609999999999999,"15min":1.2609999999999999},"last":0.77300000000000002},{"interface":"front","average":{"1min":0.746,"5min":0.72999999999999998,"15min":0.76900000000000002},"min":{"1min":0.35199999999999998,"5min":0.34399999999999997,"15min":0.313},"max":{"1min":1.091,"5min":1.222,"15min":1.222},"last":0.84999999999999998}]},{"osd":2,"last update":"Mon May 22 15:45:03 2023","interfaces":[{"interface":"back","average":{"1min":0.78700000000000003,"5min":0.76200000000000001,"15min":0.81899999999999995},"min":{"1min":0.42399999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":1.359,"5min":1.506,"15min":1.506},"last":0.89900000000000002},{"interface":"front","average":{"1min":0.79200000000000004,"5min":0.745,"15min":0.80600000000000005},"min":{"1min":0.32000000000000001,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":1.0960000000000001,"5min":1.181,"15min":1.468},"last":0.71099999999999997}]},{"osd":4,"last update":"Mon May 22 15:45:03 2023","interfaces":[{"interface":"back","average":{"1min":0.94399999999999995,"5min":0.91800000000000004,"15min":0.95599999999999996},"min":{"1min":0.67700000000000005,"5min":0.53700000000000003,"15min":0.48999999999999999},"max":{"1min":1.2290000000000001,"5min":1.413,"15min":1.4470000000000001},"last":1.012},{"interface":"front","average":{"1min":0.98199999999999998,"5min":0.95099999999999996,"15min":1.004},"min":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.49199999999999999},"max":{"1min":1.3540000000000001,"5min":1.3540000000000001,"15min":1.4470000000000001},"last":0.95099999999999996}]},{"osd":5,"last update":"Mon May 22 15:45:03 2023","interfaces":[{"interface":"back","average":{"1min":1.036,"5min":0.97599999999999998,"15min":1.0169999999999999},"min":{"1min":0.64100000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.5089999999999999,"5min":1.5089999999999999,"15min":1.5089999999999999},"last":1.0429999999999999},{"interface":"front","average":{"1min":1.05,"5min":0.995,"15min":1.0429999999999999},"min":{"1min":0.61199999999999999,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.3069999999999999,"5min":1.4299999999999999,"15min":1.643},"last":1.1499999999999999}]},{"osd":6,"last update":"Mon May 22 15:45:03 2023","interfaces":[{"interface":"back","average":{"1min":1.0640000000000001,"5min":1.006,"15min":1.044},"min":{"1min":0.72599999999999998,"5min":0.47699999999999998,"15min":0.46999999999999997},"max":{"1min":1.4570000000000001,"5min":1.4570000000000001,"15min":1.49},"last":1.1870000000000001},{"interface":"front","average":{"1min":1.05,"5min":1.0149999999999999,"15min":1.0720000000000001},"min":{"1min":0.55200000000000005,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":1.409,"5min":1.4330000000000001,"15min":1.704},"last":1.25}]},{"osd":7,"last update":"Mon May 22 15:45:03 2023","interfaces":[{"interface":"back","average":{"1min":1.107,"5min":1.0649999999999999,"15min":1.1060000000000001},"min":{"1min":0.76500000000000001,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.4870000000000001,"5min":1.6719999999999999,"15min":1.6719999999999999},"last":1.212},{"interface":"front","average":{"1min":1.1000000000000001,"5min":1.05,"15min":1.0860000000000001},"min":{"1min":0.81499999999999995,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":1.5720000000000001,"5min":1.8700000000000001,"15min":1.8700000000000001},"last":1.105}]}]},{"osd":2,"up_from":13,"seq":55834575069,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":18424,"kb_used_data":1584,"kb_used_omap":0,"kb_used_meta":16832,"kb_avail":94353416,"statfs":{"total":96636764160,"available":96617897984,"internally_reserved":0,"allocated":1622016,"data_stored":854639,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":17235968},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 15:45:00 2023","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.63800000000000001,"15min":0.65500000000000003},"min":{"1min":0.28899999999999998,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.94299999999999995,"5min":1.1539999999999999,"15min":1.8240000000000001},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.71399999999999997,"15min":0.748},"min":{"1min":0.46999999999999997,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.92300000000000004,"5min":1.0740000000000001,"15min":1.8160000000000001},"last":0.54800000000000004}]},{"osd":1,"last update":"Mon May 22 15:45:00 2023","interfaces":[{"interface":"back","average":{"1min":0.72899999999999998,"5min":0.72799999999999998,"15min":0.745},"min":{"1min":0.40300000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":1.1359999999999999,"5min":1.2,"15min":1.2},"last":1.1359999999999999},{"interface":"front","average":{"1min":0.71999999999999997,"5min":0.76300000000000001,"15min":0.81200000000000006},"min":{"1min":0.374,"5min":0.317,"15min":0.317},"max":{"1min":1.052,"5min":1.246,"15min":2.5089999999999999},"last":0.72399999999999998}]},{"osd":3,"last update":"Mon May 22 15:45:00 2023","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.69199999999999995,"15min":0.72699999999999998},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.96999999999999997,"5min":1.077,"15min":2.7629999999999999},"last":0.69699999999999995},{"interface":"front","average":{"1min":0.82799999999999996,"5min":0.85299999999999998,"15min":0.88800000000000001},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":1.2410000000000001,"5min":1.304,"15min":2.6840000000000002},"last":0.90900000000000003}]},{"osd":4,"last update":"Mon May 22 15:45:00 2023","interfaces":[{"interface":"back","average":{"1min":0.99099999999999999,"5min":0.995,"15min":1.02},"min":{"1min":0.73899999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":1.3799999999999999,"5min":1.536,"15min":2.8039999999999998},"last":1.0900000000000001},{"interface":"front","average":{"1min":0.98799999999999999,"5min":1.004,"15min":1.0329999999999999},"min":{"1min":0.62,"5min":0.58699999999999997,"15min":0.56000000000000005},"max":{"1min":1.4299999999999999,"5min":1.4299999999999999,"15min":2.0819999999999999},"last":0.76200000000000001}]},{"osd":5,"last update":"Mon May 22 15:45:00 2023","interfaces":[{"interface":"back","average":{"1min":1.0369999999999999,"5min":1.0429999999999999,"15min":1.0489999999999999},"min":{"1min":0.65000000000000002,"5min":0.622,"15min":0.622},"max":{"1min":1.5049999999999999,"5min":1.5049999999999999,"15min":2.3860000000000001},"last":1.0009999999999999},{"interface":"front","average":{"1min":0.94299999999999995,"5min":0.97199999999999998,"15min":0.999},"min":{"1min":0.60999999999999999,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":1.2989999999999999,"5min":1.327,"15min":3.0299999999999998},"last":0.89500000000000002}]},{"osd":6,"last update":"Mon May 22 15:45:00 2023","interfaces":[{"interface":"back","average":{"1min":0.97699999999999998,"5min":0.996,"15min":1.04},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.2529999999999999,"5min":1.49,"15min":2.883},"last":0.81599999999999995},{"interface":"front","average":{"1min":1.069,"5min":1.069,"15min":1.103},"min":{"1min":0.69299999999999995,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.5660000000000001,"5min":1.5660000000000001,"15min":3.1459999999999999},"last":1.0289999999999999}]},{"osd":7,"last update":"Mon May 22 15:45:00 2023","interfaces":[{"interface":"back","average":{"1min":1.022,"5min":1.0800000000000001,"15min":1.127},"min":{"1min":0.499,"5min":0.499,"15min":0.499},"max":{"1min":1.55,"5min":1.55,"15min":2.5990000000000002},"last":0.94099999999999995},{"interface":"front","average":{"1min":1.026,"5min":1.046,"15min":1.1060000000000001},"min":{"1min":0.60199999999999998,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":1.3540000000000001,"5min":1.3839999999999999,"15min":3.8799999999999999},"last":0.97199999999999998}]}]},{"osd":1,"up_from":13,"seq":55834575069,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":25196,"kb_used_data":1572,"kb_used_omap":0,"kb_used_meta":23616,"kb_avail":94346644,"statfs":{"total":96636764160,"available":96610963456,"internally_reserved":0,"allocated":1609728,"data_stored":847191,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":24182784},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 15:44:14 2023","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.61399999999999999,"15min":0.63600000000000001},"min":{"1min":0.34799999999999998,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.97799999999999998,"5min":1.2270000000000001,"15min":1.2270000000000001},"last":0.374},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.72899999999999998,"15min":0.77200000000000002},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":1.151,"5min":1.2290000000000001,"15min":1.2529999999999999},"last":0.61799999999999999}]},{"osd":2,"last update":"Mon May 22 15:44:14 2023","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.73699999999999999,"15min":0.74199999999999999},"min":{"1min":0.38,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.92900000000000005,"5min":1.391,"15min":1.391},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.755,"5min":0.748,"15min":0.77100000000000002},"min":{"1min":0.36399999999999999,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":1.01,"5min":1.276,"15min":1.276},"last":0.57799999999999996}]},{"osd":3,"last update":"Mon May 22 15:44:14 2023","interfaces":[{"interface":"back","average":{"1min":0.873,"5min":0.80300000000000005,"15min":0.84299999999999997},"min":{"1min":0.32900000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.6220000000000001,"5min":1.657,"15min":1.657},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.81599999999999995,"5min":0.79600000000000004,"15min":0.82199999999999995},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":1.165,"5min":2.0569999999999999,"15min":2.0569999999999999},"last":0.53600000000000003}]},{"osd":4,"last update":"Mon May 22 15:44:14 2023","interfaces":[{"interface":"back","average":{"1min":1.0029999999999999,"5min":0.96899999999999997,"15min":0.96599999999999997},"min":{"1min":0.70299999999999996,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":1.2989999999999999,"5min":2.9529999999999998,"15min":2.9529999999999998},"last":0.82499999999999996},{"interface":"front","average":{"1min":1.0349999999999999,"5min":0.997,"15min":1.0089999999999999},"min":{"1min":0.58599999999999997,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":1.429,"5min":2.7759999999999998,"15min":2.7759999999999998},"last":0.874}]},{"osd":5,"last update":"Mon May 22 15:44:14 2023","interfaces":[{"interface":"back","average":{"1min":1.0309999999999999,"5min":1.0089999999999999,"15min":1.0409999999999999},"min":{"1min":0.60599999999999998,"5min":0.443,"15min":0.443},"max":{"1min":1.5649999999999999,"5min":1.9570000000000001,"15min":1.9570000000000001},"last":0.90500000000000003},{"interface":"front","average":{"1min":1.095,"5min":1.0249999999999999,"15min":1.0449999999999999},"min":{"1min":0.76100000000000001,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":1.486,"5min":2.1200000000000001,"15min":2.1200000000000001},"last":0.74299999999999999}]},{"osd":6,"last update":"Mon May 22 15:44:14 2023","interfaces":[{"interface":"back","average":{"1min":1.0549999999999999,"5min":1.0209999999999999,"15min":1.04},"min":{"1min":0.746,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.3260000000000001,"5min":2.831,"15min":2.831},"last":0.96699999999999997},{"interface":"front","average":{"1min":1.1299999999999999,"5min":1.0720000000000001,"15min":1.099},"min":{"1min":0.63500000000000001,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":1.6779999999999999,"5min":1.7629999999999999,"15min":1.778},"last":0.79700000000000004}]},{"osd":7,"last update":"Mon May 22 15:44:14 2023","interfaces":[{"interface":"back","average":{"1min":1.1679999999999999,"5min":1.1080000000000001,"15min":1.1200000000000001},"min":{"1min":0.78600000000000003,"5min":0.56799999999999995,"15min":0.56799999999999995},"max":{"1min":1.446,"5min":2.4199999999999999,"15min":2.4199999999999999},"last":0.94699999999999995},{"interface":"front","average":{"1min":1.163,"5min":1.1259999999999999,"15min":1.1299999999999999},"min":{"1min":0.59899999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":1.466,"5min":3.8799999999999999,"15min":3.8799999999999999},"last":0.68100000000000005}]}]},{"osd":0,"up_from":13,"seq":55834575070,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":24848,"kb_used_data":2184,"kb_used_omap":0,"kb_used_meta":22656,"kb_avail":94346992,"statfs":{"total":96636764160,"available":96611319808,"internally_reserved":0,"allocated":2236416,"data_stored":1446667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":23199744},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon May 22 15:44:13 2023","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.68700000000000006},"min":{"1min":0.45600000000000002,"5min":0.223,"15min":0.223},"max":{"1min":1.071,"5min":1.466,"15min":1.466},"last":0.32900000000000001},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.69399999999999995,"15min":0.74099999999999999},"min":{"1min":0.39000000000000001,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.81499999999999995,"5min":1.71,"15min":1.71},"last":0.35199999999999998}]},{"osd":2,"last update":"Mon May 22 15:44:13 2023","interfaces":[{"interface":"back","average":{"1min":0.71799999999999997,"5min":0.71999999999999997,"15min":0.754},"min":{"1min":0.40699999999999997,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.97899999999999998,"5min":1.165,"15min":1.4059999999999999},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.751,"5min":0.745,"15min":0.80500000000000005},"min":{"1min":0.34999999999999998,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":1.123,"5min":1.147,"15min":1.3999999999999999},"last":0.40999999999999998}]},{"osd":3,"last update":"Mon May 22 15:44:13 2023","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.75700000000000001,"15min":0.77300000000000002},"min":{"1min":0.314,"5min":0.31,"15min":0.31},"max":{"1min":0.95999999999999996,"5min":1.9299999999999999,"15min":1.9299999999999999},"last":0.38800000000000001},{"interface":"front","average":{"1min":0.81499999999999995,"5min":0.83399999999999996,"15min":0.88700000000000001},"min":{"1min":0.36499999999999999,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":1.2050000000000001,"5min":1.758,"15min":1.758},"last":0.375}]},{"osd":4,"last update":"Mon May 22 15:44:13 2023","interfaces":[{"interface":"back","average":{"1min":0.91900000000000004,"5min":0.93700000000000006,"15min":0.96099999999999997},"min":{"1min":0.68100000000000005,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":1.1739999999999999,"5min":2.923,"15min":2.923},"last":0.83099999999999996},{"interface":"front","average":{"1min":0.95699999999999996,"5min":1.012,"15min":1.0329999999999999},"min":{"1min":0.64700000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"max":{"1min":1.268,"5min":2.0129999999999999,"15min":2.0129999999999999},"last":0.66300000000000003}]},{"osd":5,"last update":"Mon May 22 15:44:13 2023","interfaces":[{"interface":"back","average":{"1min":1.002,"5min":0.99399999999999999,"15min":1.0489999999999999},"min":{"1min":0.63500000000000001,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.4179999999999999,"5min":1.97,"15min":1.97},"last":0.85699999999999998},{"interface":"front","average":{"1min":0.96399999999999997,"5min":1.0129999999999999,"15min":1.0169999999999999},"min":{"1min":0.67900000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.5129999999999999,"5min":5.0519999999999996,"15min":5.0519999999999996},"last":0.72899999999999998}]},{"osd":6,"last update":"Mon May 22 15:44:13 2023","interfaces":[{"interface":"back","average":{"1min":1.0129999999999999,"5min":1.0629999999999999,"15min":1.095},"min":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":1.46,"5min":2.117,"15min":2.117},"last":0.77100000000000002},{"interface":"front","average":{"1min":1.054,"5min":1.123,"15min":1.121},"min":{"1min":0.70599999999999996,"5min":0.499,"15min":0.499},"max":{"1min":1.5609999999999999,"5min":5.2640000000000002,"15min":5.2640000000000002},"last":0.69499999999999995}]},{"osd":7,"last update":"Mon May 22 15:44:13 2023","interfaces":[{"interface":"back","average":{"1min":1.091,"5min":1.105,"15min":1.135},"min":{"1min":0.65900000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"max":{"1min":1.6599999999999999,"5min":1.839,"15min":1.839},"last":0.63600000000000001},{"interface":"front","average":{"1min":1.0229999999999999,"5min":1.0820000000000001,"15min":1.101},"min":{"1min":0.79700000000000004,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":1.591,"5min":3.9060000000000001,"15min":3.9060000000000001},"last":0.79600000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-05-22T15:45:08.842 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-05-22T15:45:09.011 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.009+0000 7ff8c5511640 1 Processor -- start 2023-05-22T15:45:09.011 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.009+0000 7ff8c5511640 1 -- start start 2023-05-22T15:45:09.012 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.009+0000 7ff8c5511640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7ff8c0159b10 0x7ff8c00cc780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:09.012 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.009+0000 7ff8c5511640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff8c006e250 0x7ff8c00b3e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:09.012 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.009+0000 7ff8c5511640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8c0159ee0 con 0x7ff8c00b5e80 2023-05-22T15:45:09.013 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.009+0000 7ff8c5511640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8c00b30b0 con 0x7ff8c0159b10 2023-05-22T15:45:09.013 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.009+0000 7ff8c5511640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8c00b2810 con 0x7ff8c006e250 2023-05-22T15:45:09.013 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.009+0000 7ff8b7fff640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7ff8c00b5e80 0x7ff8c006db40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:44912/0 (socket says 172.21.15.16:44912) 2023-05-22T15:45:09.013 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.009+0000 7ff8b7fff640 1 -- 172.21.15.16:0/1677075000 learned_addr learned my addr 172.21.15.16:0/1677075000 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:45:09.016 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.009+0000 7ff8bf7fe640 1 --2- 172.21.15.16:0/1677075000 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff8c006e250 0x7ff8c00b3e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:09.016 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.009+0000 7ff8beffd640 1 --2- 172.21.15.16:0/1677075000 >> v2:172.21.15.62:3300/0 conn(0x7ff8c0159b10 0x7ff8c00cc780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:09.016 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.009+0000 7ff8be7fc640 1 -- 172.21.15.16:0/1677075000 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1508527991 0 0) 0x7ff8c0159ee0 con 0x7ff8c00b5e80 2023-05-22T15:45:09.016 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.009+0000 7ff8be7fc640 1 -- 172.21.15.16:0/1677075000 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8ac003530 con 0x7ff8c00b5e80 2023-05-22T15:45:09.016 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.009+0000 7ff8be7fc640 1 -- 172.21.15.16:0/1677075000 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1664411487 0 0) 0x7ff8ac003530 con 0x7ff8c00b5e80 2023-05-22T15:45:09.016 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.009+0000 7ff8be7fc640 1 -- 172.21.15.16:0/1677075000 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff8c006e250 msgr2=0x7ff8c00b3e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:45:09.017 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.009+0000 7ff8be7fc640 1 --2- 172.21.15.16:0/1677075000 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff8c006e250 0x7ff8c00b3e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:09.017 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.009+0000 7ff8be7fc640 1 -- 172.21.15.16:0/1677075000 >> v2:172.21.15.62:3300/0 conn(0x7ff8c0159b10 msgr2=0x7ff8c00cc780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:45:09.017 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.009+0000 7ff8be7fc640 1 --2- 172.21.15.16:0/1677075000 >> v2:172.21.15.62:3300/0 conn(0x7ff8c0159b10 0x7ff8c00cc780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:09.017 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.009+0000 7ff8be7fc640 1 -- 172.21.15.16:0/1677075000 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8c00b2af0 con 0x7ff8c00b5e80 2023-05-22T15:45:09.017 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.009+0000 7ff8be7fc640 1 -- 172.21.15.16:0/1677075000 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7ff8a8001680 con 0x7ff8c00b5e80 2023-05-22T15:45:09.017 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.009+0000 7ff8be7fc640 1 -- 172.21.15.16:0/1677075000 >> v1:172.21.15.16:6789/0 conn(0x7ff8c00b5e80 legacy=0x7ff8c006db40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:45:09.017 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.009+0000 7ff8be7fc640 1 --2- 172.21.15.16:0/1677075000 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff8c006e250 0x7ff8ac003980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:09.018 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.009+0000 7ff8be7fc640 1 --2- 172.21.15.16:0/1677075000 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff8ac003f00 0x7ff8ac0047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:09.018 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.009+0000 7ff8be7fc640 1 --2- 172.21.15.16:0/1677075000 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff8ac004d10 0x7ff8ac005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:09.018 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.009+0000 7ff8be7fc640 1 -- 172.21.15.16:0/1677075000 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8c00b30b0 con 0x7ff8ac003f00 2023-05-22T15:45:09.018 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.009+0000 7ff8be7fc640 1 -- 172.21.15.16:0/1677075000 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8c00b2810 con 0x7ff8ac004d10 2023-05-22T15:45:09.018 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.009+0000 7ff8be7fc640 1 -- 172.21.15.16:0/1677075000 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8ac00b990 con 0x7ff8c006e250 2023-05-22T15:45:09.018 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.009+0000 7ff8b7fff640 1 --2- 172.21.15.16:0/1677075000 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff8ac003f00 0x7ff8ac0047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:09.018 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.009+0000 7ff8beffd640 1 --2- 172.21.15.16:0/1677075000 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff8c006e250 0x7ff8ac003980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:09.018 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.009+0000 7ff8bf7fe640 1 --2- 172.21.15.16:0/1677075000 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff8ac004d10 0x7ff8ac005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:09.019 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.009+0000 7ff8b7fff640 1 -- 172.21.15.16:0/1677075000 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff8c006e250 msgr2=0x7ff8ac003980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:45:09.019 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.009+0000 7ff8b7fff640 1 --2- 172.21.15.16:0/1677075000 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff8c006e250 0x7ff8ac003980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:09.019 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.009+0000 7ff8b7fff640 1 -- 172.21.15.16:0/1677075000 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff8ac004d10 msgr2=0x7ff8ac005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:45:09.019 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.009+0000 7ff8b7fff640 1 --2- 172.21.15.16:0/1677075000 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff8ac004d10 0x7ff8ac005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:09.019 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.009+0000 7ff8b7fff640 1 -- 172.21.15.16:0/1677075000 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8ac003530 con 0x7ff8ac003f00 2023-05-22T15:45:09.019 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.009+0000 7ff8bf7fe640 1 --2- 172.21.15.16:0/1677075000 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff8ac004d10 0x7ff8ac005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:45:09.019 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.013+0000 7ff8b7fff640 1 --2- 172.21.15.16:0/1677075000 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff8ac003f00 0x7ff8ac0047d0 secure :-1 s=READY pgs=1650 cs=0 l=1 rev1=1 crypto rx=0x7ff8a800c290 tx=0x7ff8a801f430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63d94ca20f08bb7e server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:45:09.020 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.013+0000 7ff8be7fc640 1 -- 172.21.15.16:0/1677075000 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8a8005ae0 con 0x7ff8ac003f00 2023-05-22T15:45:09.020 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.013+0000 7ff8be7fc640 1 -- 172.21.15.16:0/1677075000 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8a80ab070 con 0x7ff8ac003f00 2023-05-22T15:45:09.021 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.013+0000 7ff8be7fc640 1 -- 172.21.15.16:0/1677075000 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8a8005e80 con 0x7ff8ac003f00 2023-05-22T15:45:09.023 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.013+0000 7ff8c5511640 1 -- 172.21.15.16:0/1677075000 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff8ac003f00 msgr2=0x7ff8ac0047d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:45:09.023 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.013+0000 7ff8c5511640 1 --2- 172.21.15.16:0/1677075000 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff8ac003f00 0x7ff8ac0047d0 secure :-1 s=READY pgs=1650 cs=0 l=1 rev1=1 crypto rx=0x7ff8a800c290 tx=0x7ff8a801f430 comp rx=0 tx=0).stop 2023-05-22T15:45:09.023 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.013+0000 7ff8bf7fe640 1 -- 172.21.15.16:0/1677075000 reap_dead start 2023-05-22T15:45:09.023 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.013+0000 7ff8c5511640 1 -- 172.21.15.16:0/1677075000 shutdown_connections 2023-05-22T15:45:09.024 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.013+0000 7ff8c5511640 1 -- 172.21.15.16:0/1677075000 >> 172.21.15.16:0/1677075000 conn(0x7ff8c006f560 msgr2=0x7ff8c006f960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:45:09.024 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.013+0000 7ff8c5511640 1 -- 172.21.15.16:0/1677075000 shutdown_connections 2023-05-22T15:45:09.024 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.013+0000 7ff8c5511640 1 -- 172.21.15.16:0/1677075000 wait complete. 2023-05-22T15:45:09.024 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.013+0000 7ff8c5511640 1 Processor -- start 2023-05-22T15:45:09.024 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.013+0000 7ff8c5511640 1 -- start start 2023-05-22T15:45:09.025 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.017+0000 7ff8c5511640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff8c014d6c0 0x7ff8c00c4d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:09.025 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.017+0000 7ff8c5511640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff8c014da90 0x7ff8c00c9340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:09.025 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.017+0000 7ff8c5511640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff8c00c9880 0x7ff8c00c5c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:09.025 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.017+0000 7ff8c5511640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8c00c9d20 con 0x7ff8c014d6c0 2023-05-22T15:45:09.025 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.017+0000 7ff8c5511640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8c00c3d20 con 0x7ff8c014da90 2023-05-22T15:45:09.025 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.017+0000 7ff8c5511640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8c00c3f70 con 0x7ff8c00c9880 2023-05-22T15:45:09.025 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.017+0000 7ff8beffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff8c014d6c0 0x7ff8c00c4d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:09.026 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.017+0000 7ff8beffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff8c014d6c0 0x7ff8c00c4d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45314/0 (socket says 172.21.15.16:45314) 2023-05-22T15:45:09.026 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.017+0000 7ff8beffd640 1 -- 172.21.15.16:0/4074562806 learned_addr learned my addr 172.21.15.16:0/4074562806 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:45:09.026 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.017+0000 7ff8b7fff640 1 --2- 172.21.15.16:0/4074562806 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff8c014da90 0x7ff8c00c9340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:09.026 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.017+0000 7ff8bf7fe640 1 --2- 172.21.15.16:0/4074562806 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff8c00c9880 0x7ff8c00c5c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:09.026 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.017+0000 7ff8beffd640 1 -- 172.21.15.16:0/4074562806 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff8c00c9880 msgr2=0x7ff8c00c5c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:45:09.026 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.017+0000 7ff8beffd640 1 --2- 172.21.15.16:0/4074562806 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff8c00c9880 0x7ff8c00c5c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:09.026 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.017+0000 7ff8beffd640 1 -- 172.21.15.16:0/4074562806 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff8c014da90 msgr2=0x7ff8c00c9340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:45:09.026 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.017+0000 7ff8beffd640 1 --2- 172.21.15.16:0/4074562806 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff8c014da90 0x7ff8c00c9340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:09.027 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.017+0000 7ff8beffd640 1 -- 172.21.15.16:0/4074562806 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8c00b3630 con 0x7ff8c014d6c0 2023-05-22T15:45:09.027 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.017+0000 7ff8beffd640 1 --2- 172.21.15.16:0/4074562806 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff8c014d6c0 0x7ff8c00c4d30 secure :-1 s=READY pgs=1651 cs=0 l=1 rev1=1 crypto rx=0x7ff8b0011560 tx=0x7ff8b0011a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:45:09.027 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.017+0000 7ff8b77fe640 1 -- 172.21.15.16:0/4074562806 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8b0015870 con 0x7ff8c014d6c0 2023-05-22T15:45:09.027 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.017+0000 7ff8b77fe640 1 -- 172.21.15.16:0/4074562806 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8b00159d0 con 0x7ff8c014d6c0 2023-05-22T15:45:09.027 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.017+0000 7ff8c5511640 1 -- 172.21.15.16:0/4074562806 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8c00b3350 con 0x7ff8c014d6c0 2023-05-22T15:45:09.027 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.017+0000 7ff8b77fe640 1 -- 172.21.15.16:0/4074562806 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8b0015d40 con 0x7ff8c014d6c0 2023-05-22T15:45:09.027 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.017+0000 7ff8c5511640 1 -- 172.21.15.16:0/4074562806 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8c00b3910 con 0x7ff8c014d6c0 2023-05-22T15:45:09.028 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.017+0000 7ff8c5511640 1 -- 172.21.15.16:0/4074562806 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8c004dbe0 con 0x7ff8c014d6c0 2023-05-22T15:45:09.028 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.017+0000 7ff8b77fe640 1 -- 172.21.15.16:0/4074562806 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 50) v1 ==== 84332+0+0 (secure 0 0 0) 0x7ff8b000f460 con 0x7ff8c014d6c0 2023-05-22T15:45:09.028 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.017+0000 7ff8b77fe640 1 --2- 172.21.15.16:0/4074562806 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ff8a00654a0 0x7ff8a0067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:09.028 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.017+0000 7ff8b77fe640 1 -- 172.21.15.16:0/4074562806 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 6382+0+0 (secure 0 0 0) 0x7ff8b0083380 con 0x7ff8c014d6c0 2023-05-22T15:45:09.028 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.017+0000 7ff8b7fff640 1 --2- 172.21.15.16:0/4074562806 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ff8a00654a0 0x7ff8a0067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:09.028 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.017+0000 7ff8b7fff640 1 --2- 172.21.15.16:0/4074562806 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ff8a00654a0 0x7ff8a0067950 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7ff8a80ae000 tx=0x7ff8a8007040 comp rx=0 tx=0).ready entity=mgr.8304 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:45:09.029 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.021+0000 7ff8b77fe640 1 -- 172.21.15.16:0/4074562806 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7ff8b00547e0 con 0x7ff8c014d6c0 2023-05-22T15:45:09.178 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.173+0000 7ff8c5511640 1 -- 172.21.15.16:0/4074562806 --> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff8c0154c00 con 0x7ff8a00654a0 2023-05-22T15:45:09.186 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.185+0000 7ff8b77fe640 1 -- 172.21.15.16:0/4074562806 <== mgr.8304 v2:172.21.15.16:6832/36921 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+310104 (secure 0 0 0) 0x7ff8c0154c00 con 0x7ff8a00654a0 2023-05-22T15:45:09.187 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:45:09.190 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2023-05-22T15:45:09.192 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.189+0000 7ff8c5511640 1 -- 172.21.15.16:0/4074562806 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ff8a00654a0 msgr2=0x7ff8a0067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:45:09.192 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.189+0000 7ff8c5511640 1 --2- 172.21.15.16:0/4074562806 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ff8a00654a0 0x7ff8a0067950 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7ff8a80ae000 tx=0x7ff8a8007040 comp rx=0 tx=0).stop 2023-05-22T15:45:09.192 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.189+0000 7ff8c5511640 1 -- 172.21.15.16:0/4074562806 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff8c014d6c0 msgr2=0x7ff8c00c4d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:45:09.192 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.189+0000 7ff8c5511640 1 --2- 172.21.15.16:0/4074562806 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff8c014d6c0 0x7ff8c00c4d30 secure :-1 s=READY pgs=1651 cs=0 l=1 rev1=1 crypto rx=0x7ff8b0011560 tx=0x7ff8b0011a20 comp rx=0 tx=0).stop 2023-05-22T15:45:09.192 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.189+0000 7ff8c5511640 1 -- 172.21.15.16:0/4074562806 shutdown_connections 2023-05-22T15:45:09.192 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.189+0000 7ff8c5511640 1 --2- 172.21.15.16:0/4074562806 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ff8a00654a0 0x7ff8a0067950 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:09.193 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.189+0000 7ff8c5511640 1 --2- 172.21.15.16:0/4074562806 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff8c00c9880 0x7ff8c00c5c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:09.193 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.189+0000 7ff8c5511640 1 --2- 172.21.15.16:0/4074562806 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff8c014da90 0x7ff8c00c9340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:09.193 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.189+0000 7ff8c5511640 1 --2- 172.21.15.16:0/4074562806 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff8c014d6c0 0x7ff8c00c4d30 unknown :-1 s=CLOSED pgs=1651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:09.193 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.189+0000 7ff8c5511640 1 -- 172.21.15.16:0/4074562806 >> 172.21.15.16:0/4074562806 conn(0x7ff8c006f560 msgr2=0x7ff8c00b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:45:09.193 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.189+0000 7ff8c5511640 1 -- 172.21.15.16:0/4074562806 shutdown_connections 2023-05-22T15:45:09.193 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.189+0000 7ff8c5511640 1 -- 172.21.15.16:0/4074562806 wait complete. 2023-05-22T15:45:09.209 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2023-05-22T15:45:08.064469+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592698,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":1076,"num_read_kb":2160,"num_write":404,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":266,"ondisk_log_size":266,"up":210,"acting":210,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":40,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":224564,"kb_used_data":13876,"kb_used_omap":0,"kb_used_meta":210624,"kb_avail":754750156,"statfs":{"total":773094113280,"available":772864159744,"internally_reserved":0,"allocated":14209024,"data_stored":8014936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":215678976},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":61,"apply_latency_ms":61,"commit_latency_ns":61000000,"apply_latency_ns":61000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2330,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.261557"},"pg_stats":[{"pgid":"17.3f","version":"0'0","reported_seq":9,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-05-22T15:45:05.266425+0000","last_change":"2023-05-22T15:45:05.265785+0000","last_active":"2023-05-22T15:45:05.266425+0000","last_peered":"2023-05-22T15:45:05.266425+0000","last_clean":"2023-05-22T15:45:05.266425+0000","last_became_active":"2023-05-22T15:45:05.265486+0000","last_became_peered":"2023-05-22T15:45:05.265486+0000","last_unstale":"2023-05-22T15:45:05.266425+0000","last_undegraded":"2023-05-22T15:45:05.266425+0000","last_fullsized":"2023-05-22T15:45:05.266425+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_clean_scrub_stamp":"2023-05-22T15:45:04.802083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:36:11.193934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3e","version":"0'0","reported_seq":9,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-05-22T15:45:05.261440+0000","last_change":"2023-05-22T15:45:05.252520+0000","last_active":"2023-05-22T15:45:05.261440+0000","last_peered":"2023-05-22T15:45:05.261440+0000","last_clean":"2023-05-22T15:45:05.261440+0000","last_became_active":"2023-05-22T15:45:05.251768+0000","last_became_peered":"2023-05-22T15:45:05.251768+0000","last_unstale":"2023-05-22T15:45:05.261440+0000","last_undegraded":"2023-05-22T15:45:05.261440+0000","last_fullsized":"2023-05-22T15:45:05.261440+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_clean_scrub_stamp":"2023-05-22T15:45:04.802083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:09:12.891783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.3d","version":"0'0","reported_seq":9,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-05-22T15:45:05.289805+0000","last_change":"2023-05-22T15:45:05.281371+0000","last_active":"2023-05-22T15:45:05.289805+0000","last_peered":"2023-05-22T15:45:05.289805+0000","last_clean":"2023-05-22T15:45:05.289805+0000","last_became_active":"2023-05-22T15:45:05.279345+0000","last_became_peered":"2023-05-22T15:45:05.279345+0000","last_unstale":"2023-05-22T15:45:05.289805+0000","last_undegraded":"2023-05-22T15:45:05.289805+0000","last_fullsized":"2023-05-22T15:45:05.289805+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_clean_scrub_stamp":"2023-05-22T15:45:04.802083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:22:50.401810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.3c","version":"0'0","reported_seq":9,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-05-22T15:45:05.252795+0000","last_change":"2023-05-22T15:45:05.251350+0000","last_active":"2023-05-22T15:45:05.252795+0000","last_peered":"2023-05-22T15:45:05.252795+0000","last_clean":"2023-05-22T15:45:05.252795+0000","last_became_active":"2023-05-22T15:45:05.251091+0000","last_became_peered":"2023-05-22T15:45:05.251091+0000","last_unstale":"2023-05-22T15:45:05.252795+0000","last_undegraded":"2023-05-22T15:45:05.252795+0000","last_fullsized":"2023-05-22T15:45:05.252795+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_clean_scrub_stamp":"2023-05-22T15:45:04.802083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:51:15.372468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3b","version":"0'0","reported_seq":9,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-05-22T15:45:05.217549+0000","last_change":"2023-05-22T15:45:05.216706+0000","last_active":"2023-05-22T15:45:05.217549+0000","last_peered":"2023-05-22T15:45:05.217549+0000","last_clean":"2023-05-22T15:45:05.217549+0000","last_became_active":"2023-05-22T15:45:05.216316+0000","last_became_peered":"2023-05-22T15:45:05.216316+0000","last_unstale":"2023-05-22T15:45:05.217549+0000","last_undegraded":"2023-05-22T15:45:05.217549+0000","last_fullsized":"2023-05-22T15:45:05.217549+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_clean_scrub_stamp":"2023-05-22T15:45:04.802083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:00:52.830469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.3a","version":"0'0","reported_seq":9,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-05-22T15:45:05.227672+0000","last_change":"2023-05-22T15:45:05.217866+0000","last_active":"2023-05-22T15:45:05.227672+0000","last_peered":"2023-05-22T15:45:05.227672+0000","last_clean":"2023-05-22T15:45:05.227672+0000","last_became_active":"2023-05-22T15:45:05.217582+0000","last_became_peered":"2023-05-22T15:45:05.217582+0000","last_unstale":"2023-05-22T15:45:05.227672+0000","last_undegraded":"2023-05-22T15:45:05.227672+0000","last_fullsized":"2023-05-22T15:45:05.227672+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_clean_scrub_stamp":"2023-05-22T15:45:04.802083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T16:55:32.318704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.39","version":"0'0","reported_seq":9,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-05-22T15:45:05.239851+0000","last_change":"2023-05-22T15:45:05.234774+0000","last_active":"2023-05-22T15:45:05.239851+0000","last_peered":"2023-05-22T15:45:05.239851+0000","last_clean":"2023-05-22T15:45:05.239851+0000","last_became_active":"2023-05-22T15:45:05.234054+0000","last_became_peered":"2023-05-22T15:45:05.234054+0000","last_unstale":"2023-05-22T15:45:05.239851+0000","last_undegraded":"2023-05-22T15:45:05.239851+0000","last_fullsized":"2023-05-22T15:45:05.239851+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_clean_scrub_stamp":"2023-05-22T15:45:04.802083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T18:38:17.673900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.38","version":"0'0","reported_seq":9,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-05-22T15:45:05.289844+0000","last_change":"2023-05-22T15:45:05.285050+0000","last_active":"2023-05-22T15:45:05.289844+0000","last_peered":"2023-05-22T15:45:05.289844+0000","last_clean":"2023-05-22T15:45:05.289844+0000","last_became_active":"2023-05-22T15:45:05.281072+0000","last_became_peered":"2023-05-22T15:45:05.281072+0000","last_unstale":"2023-05-22T15:45:05.289844+0000","last_undegraded":"2023-05-22T15:45:05.289844+0000","last_fullsized":"2023-05-22T15:45:05.289844+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_clean_scrub_stamp":"2023-05-22T15:45:04.802083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:29:06.460933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.37","version":"0'0","reported_seq":9,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-05-22T15:45:05.241024+0000","last_change":"2023-05-22T15:45:05.234127+0000","last_active":"2023-05-22T15:45:05.241024+0000","last_peered":"2023-05-22T15:45:05.241024+0000","last_clean":"2023-05-22T15:45:05.241024+0000","last_became_active":"2023-05-22T15:45:05.233876+0000","last_became_peered":"2023-05-22T15:45:05.233876+0000","last_unstale":"2023-05-22T15:45:05.241024+0000","last_undegraded":"2023-05-22T15:45:05.241024+0000","last_fullsized":"2023-05-22T15:45:05.241024+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_clean_scrub_stamp":"2023-05-22T15:45:04.802083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T16:14:35.323782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.36","version":"0'0","reported_seq":9,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-05-22T15:45:05.241092+0000","last_change":"2023-05-22T15:45:05.234783+0000","last_active":"2023-05-22T15:45:05.241092+0000","last_peered":"2023-05-22T15:45:05.241092+0000","last_clean":"2023-05-22T15:45:05.241092+0000","last_became_active":"2023-05-22T15:45:05.234523+0000","last_became_peered":"2023-05-22T15:45:05.234523+0000","last_unstale":"2023-05-22T15:45:05.241092+0000","last_undegraded":"2023-05-22T15:45:05.241092+0000","last_fullsized":"2023-05-22T15:45:05.241092+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_clean_scrub_stamp":"2023-05-22T15:45:04.802083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:25:01.333952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.35","version":"0'0","reported_seq":9,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-05-22T15:45:05.264970+0000","last_change":"2023-05-22T15:45:05.253945+0000","last_active":"2023-05-22T15:45:05.264970+0000","last_peered":"2023-05-22T15:45:05.264970+0000","last_clean":"2023-05-22T15:45:05.264970+0000","last_became_active":"2023-05-22T15:45:05.253119+0000","last_became_peered":"2023-05-22T15:45:05.253119+0000","last_unstale":"2023-05-22T15:45:05.264970+0000","last_undegraded":"2023-05-22T15:45:05.264970+0000","last_fullsized":"2023-05-22T15:45:05.264970+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_clean_scrub_stamp":"2023-05-22T15:45:04.802083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:17:05.563955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.34","version":"0'0","reported_seq":9,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-05-22T15:45:05.239014+0000","last_change":"2023-05-22T15:45:05.233397+0000","last_active":"2023-05-22T15:45:05.239014+0000","last_peered":"2023-05-22T15:45:05.239014+0000","last_clean":"2023-05-22T15:45:05.239014+0000","last_became_active":"2023-05-22T15:45:05.233083+0000","last_became_peered":"2023-05-22T15:45:05.233083+0000","last_unstale":"2023-05-22T15:45:05.239014+0000","last_undegraded":"2023-05-22T15:45:05.239014+0000","last_fullsized":"2023-05-22T15:45:05.239014+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_clean_scrub_stamp":"2023-05-22T15:45:04.802083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T18:46:17.237523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.33","version":"0'0","reported_seq":9,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-05-22T15:45:05.289787+0000","last_change":"2023-05-22T15:45:05.281691+0000","last_active":"2023-05-22T15:45:05.289787+0000","last_peered":"2023-05-22T15:45:05.289787+0000","last_clean":"2023-05-22T15:45:05.289787+0000","last_became_active":"2023-05-22T15:45:05.280285+0000","last_became_peered":"2023-05-22T15:45:05.280285+0000","last_unstale":"2023-05-22T15:45:05.289787+0000","last_undegraded":"2023-05-22T15:45:05.289787+0000","last_fullsized":"2023-05-22T15:45:05.289787+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_clean_scrub_stamp":"2023-05-22T15:45:04.802083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T16:42:50.060919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.32","version":"0'0","reported_seq":9,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-05-22T15:45:05.289585+0000","last_change":"2023-05-22T15:45:05.279318+0000","last_active":"2023-05-22T15:45:05.289585+0000","last_peered":"2023-05-22T15:45:05.289585+0000","last_clean":"2023-05-22T15:45:05.289585+0000","last_became_active":"2023-05-22T15:45:05.278848+0000","last_became_peered":"2023-05-22T15:45:05.278848+0000","last_unstale":"2023-05-22T15:45:05.289585+0000","last_undegraded":"2023-05-22T15:45:05.289585+0000","last_fullsized":"2023-05-22T15:45:05.289585+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_clean_scrub_stamp":"2023-05-22T15:45:04.802083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:09:12.891783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.31","version":"0'0","reported_seq":9,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-05-22T15:45:05.223267+0000","last_change":"2023-05-22T15:45:05.217316+0000","last_active":"2023-05-22T15:45:05.223267+0000","last_peered":"2023-05-22T15:45:05.223267+0000","last_clean":"2023-05-22T15:45:05.223267+0000","last_became_active":"2023-05-22T15:45:05.216853+0000","last_became_peered":"2023-05-22T15:45:05.216853+0000","last_unstale":"2023-05-22T15:45:05.223267+0000","last_undegraded":"2023-05-22T15:45:05.223267+0000","last_fullsized":"2023-05-22T15:45:05.223267+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_clean_scrub_stamp":"2023-05-22T15:45:04.802083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:00:52.830469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.30","version":"0'0","reported_seq":9,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-05-22T15:45:05.243075+0000","last_change":"2023-05-22T15:45:05.241967+0000","last_active":"2023-05-22T15:45:05.243075+0000","last_peered":"2023-05-22T15:45:05.243075+0000","last_clean":"2023-05-22T15:45:05.243075+0000","last_became_active":"2023-05-22T15:45:05.241757+0000","last_became_peered":"2023-05-22T15:45:05.241757+0000","last_unstale":"2023-05-22T15:45:05.243075+0000","last_undegraded":"2023-05-22T15:45:05.243075+0000","last_fullsized":"2023-05-22T15:45:05.243075+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_clean_scrub_stamp":"2023-05-22T15:45:04.802083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:57:36.081992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.2f","version":"0'0","reported_seq":9,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-05-22T15:45:05.271840+0000","last_change":"2023-05-22T15:45:05.268751+0000","last_active":"2023-05-22T15:45:05.271840+0000","last_peered":"2023-05-22T15:45:05.271840+0000","last_clean":"2023-05-22T15:45:05.271840+0000","last_became_active":"2023-05-22T15:45:05.267736+0000","last_became_peered":"2023-05-22T15:45:05.267736+0000","last_unstale":"2023-05-22T15:45:05.271840+0000","last_undegraded":"2023-05-22T15:45:05.271840+0000","last_fullsized":"2023-05-22T15:45:05.271840+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_clean_scrub_stamp":"2023-05-22T15:45:04.802083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:41:01.090364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2e","version":"0'0","reported_seq":9,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-05-22T15:45:05.274103+0000","last_change":"2023-05-22T15:45:05.253174+0000","last_active":"2023-05-22T15:45:05.274103+0000","last_peered":"2023-05-22T15:45:05.274103+0000","last_clean":"2023-05-22T15:45:05.274103+0000","last_became_active":"2023-05-22T15:45:05.252455+0000","last_became_peered":"2023-05-22T15:45:05.252455+0000","last_unstale":"2023-05-22T15:45:05.274103+0000","last_undegraded":"2023-05-22T15:45:05.274103+0000","last_fullsized":"2023-05-22T15:45:05.274103+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_clean_scrub_stamp":"2023-05-22T15:45:04.802083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:46:36.321475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.2d","version":"0'0","reported_seq":9,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-05-22T15:45:05.289770+0000","last_change":"2023-05-22T15:45:05.281430+0000","last_active":"2023-05-22T15:45:05.289770+0000","last_peered":"2023-05-22T15:45:05.289770+0000","last_clean":"2023-05-22T15:45:05.289770+0000","last_became_active":"2023-05-22T15:45:05.280421+0000","last_became_peered":"2023-05-22T15:45:05.280421+0000","last_unstale":"2023-05-22T15:45:05.289770+0000","last_undegraded":"2023-05-22T15:45:05.289770+0000","last_fullsized":"2023-05-22T15:45:05.289770+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_clean_scrub_stamp":"2023-05-22T15:45:04.802083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T03:18:50.763567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.2c","version":"0'0","reported_seq":9,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-05-22T15:45:05.261159+0000","last_change":"2023-05-22T15:45:05.245275+0000","last_active":"2023-05-22T15:45:05.261159+0000","last_peered":"2023-05-22T15:45:05.261159+0000","last_clean":"2023-05-22T15:45:05.261159+0000","last_became_active":"2023-05-22T15:45:05.244543+0000","last_became_peered":"2023-05-22T15:45:05.244543+0000","last_unstale":"2023-05-22T15:45:05.261159+0000","last_undegraded":"2023-05-22T15:45:05.261159+0000","last_fullsized":"2023-05-22T15:45:05.261159+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_clean_scrub_stamp":"2023-05-22T15:45:04.802083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:34:38.791933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2b","version":"0'0","reported_seq":9,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-05-22T15:45:05.269669+0000","last_change":"2023-05-22T15:45:05.262231+0000","last_active":"2023-05-22T15:45:05.269669+0000","last_peered":"2023-05-22T15:45:05.269669+0000","last_clean":"2023-05-22T15:45:05.269669+0000","last_became_active":"2023-05-22T15:45:05.261471+0000","last_became_peered":"2023-05-22T15:45:05.261471+0000","last_unstale":"2023-05-22T15:45:05.269669+0000","last_undegraded":"2023-05-22T15:45:05.269669+0000","last_fullsized":"2023-05-22T15:45:05.269669+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_clean_scrub_stamp":"2023-05-22T15:45:04.802083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:23:52.692138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2a","version":"0'0","reported_seq":9,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-05-22T15:45:05.269723+0000","last_change":"2023-05-22T15:45:05.269213+0000","last_active":"2023-05-22T15:45:05.269723+0000","last_peered":"2023-05-22T15:45:05.269723+0000","last_clean":"2023-05-22T15:45:05.269723+0000","last_became_active":"2023-05-22T15:45:05.268886+0000","last_became_peered":"2023-05-22T15:45:05.268886+0000","last_unstale":"2023-05-22T15:45:05.269723+0000","last_undegraded":"2023-05-22T15:45:05.269723+0000","last_fullsized":"2023-05-22T15:45:05.269723+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_clean_scrub_stamp":"2023-05-22T15:45:04.802083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:34:38.791933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.29","version":"0'0","reported_seq":9,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-05-22T15:45:05.276886+0000","last_change":"2023-05-22T15:45:05.273703+0000","last_active":"2023-05-22T15:45:05.276886+0000","last_peered":"2023-05-22T15:45:05.276886+0000","last_clean":"2023-05-22T15:45:05.276886+0000","last_became_active":"2023-05-22T15:45:05.273181+0000","last_became_peered":"2023-05-22T15:45:05.273181+0000","last_unstale":"2023-05-22T15:45:05.276886+0000","last_undegraded":"2023-05-22T15:45:05.276886+0000","last_fullsized":"2023-05-22T15:45:05.276886+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_clean_scrub_stamp":"2023-05-22T15:45:04.802083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:22:50.401810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.28","version":"0'0","reported_seq":9,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-05-22T15:45:05.242870+0000","last_change":"2023-05-22T15:45:05.241904+0000","last_active":"2023-05-22T15:45:05.242870+0000","last_peered":"2023-05-22T15:45:05.242870+0000","last_clean":"2023-05-22T15:45:05.242870+0000","last_became_active":"2023-05-22T15:45:05.241604+0000","last_became_peered":"2023-05-22T15:45:05.241604+0000","last_unstale":"2023-05-22T15:45:05.242870+0000","last_undegraded":"2023-05-22T15:45:05.242870+0000","last_fullsized":"2023-05-22T15:45:05.242870+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_clean_scrub_stamp":"2023-05-22T15:45:04.802083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:52:29.459195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.27","version":"0'0","reported_seq":9,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-05-22T15:45:05.271809+0000","last_change":"2023-05-22T15:45:05.268365+0000","last_active":"2023-05-22T15:45:05.271809+0000","last_peered":"2023-05-22T15:45:05.271809+0000","last_clean":"2023-05-22T15:45:05.271809+0000","last_became_active":"2023-05-22T15:45:05.267647+0000","last_became_peered":"2023-05-22T15:45:05.267647+0000","last_unstale":"2023-05-22T15:45:05.271809+0000","last_undegraded":"2023-05-22T15:45:05.271809+0000","last_fullsized":"2023-05-22T15:45:05.271809+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_clean_scrub_stamp":"2023-05-22T15:45:04.802083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:06:40.672215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.26","version":"0'0","reported_seq":9,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-05-22T15:45:05.242614+0000","last_change":"2023-05-22T15:45:05.239295+0000","last_active":"2023-05-22T15:45:05.242614+0000","last_peered":"2023-05-22T15:45:05.242614+0000","last_clean":"2023-05-22T15:45:05.242614+0000","last_became_active":"2023-05-22T15:45:05.239002+0000","last_became_peered":"2023-05-22T15:45:05.239002+0000","last_unstale":"2023-05-22T15:45:05.242614+0000","last_undegraded":"2023-05-22T15:45:05.242614+0000","last_fullsized":"2023-05-22T15:45:05.242614+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_clean_scrub_stamp":"2023-05-22T15:45:04.802083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T03:03:20.741835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.25","version":"0'0","reported_seq":9,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-05-22T15:45:05.235190+0000","last_change":"2023-05-22T15:45:05.231336+0000","last_active":"2023-05-22T15:45:05.235190+0000","last_peered":"2023-05-22T15:45:05.235190+0000","last_clean":"2023-05-22T15:45:05.235190+0000","last_became_active":"2023-05-22T15:45:05.230379+0000","last_became_peered":"2023-05-22T15:45:05.230379+0000","last_unstale":"2023-05-22T15:45:05.235190+0000","last_undegraded":"2023-05-22T15:45:05.235190+0000","last_fullsized":"2023-05-22T15:45:05.235190+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_clean_scrub_stamp":"2023-05-22T15:45:04.802083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:41:05.163840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.24","version":"0'0","reported_seq":9,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-05-22T15:45:05.236226+0000","last_change":"2023-05-22T15:45:05.234054+0000","last_active":"2023-05-22T15:45:05.236226+0000","last_peered":"2023-05-22T15:45:05.236226+0000","last_clean":"2023-05-22T15:45:05.236226+0000","last_became_active":"2023-05-22T15:45:05.233789+0000","last_became_peered":"2023-05-22T15:45:05.233789+0000","last_unstale":"2023-05-22T15:45:05.236226+0000","last_undegraded":"2023-05-22T15:45:05.236226+0000","last_fullsized":"2023-05-22T15:45:05.236226+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_clean_scrub_stamp":"2023-05-22T15:45:04.802083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:01:01.088020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.23","version":"0'0","reported_seq":9,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-05-22T15:45:05.285893+0000","last_change":"2023-05-22T15:45:05.280361+0000","last_active":"2023-05-22T15:45:05.285893+0000","last_peered":"2023-05-22T15:45:05.285893+0000","last_clean":"2023-05-22T15:45:05.285893+0000","last_became_active":"2023-05-22T15:45:05.279623+0000","last_became_peered":"2023-05-22T15:45:05.279623+0000","last_unstale":"2023-05-22T15:45:05.285893+0000","last_undegraded":"2023-05-22T15:45:05.285893+0000","last_fullsized":"2023-05-22T15:45:05.285893+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_clean_scrub_stamp":"2023-05-22T15:45:04.802083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:52:29.459195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.22","version":"0'0","reported_seq":9,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-05-22T15:45:05.239577+0000","last_change":"2023-05-22T15:45:05.231856+0000","last_active":"2023-05-22T15:45:05.239577+0000","last_peered":"2023-05-22T15:45:05.239577+0000","last_clean":"2023-05-22T15:45:05.239577+0000","last_became_active":"2023-05-22T15:45:05.231053+0000","last_became_peered":"2023-05-22T15:45:05.231053+0000","last_unstale":"2023-05-22T15:45:05.239577+0000","last_undegraded":"2023-05-22T15:45:05.239577+0000","last_fullsized":"2023-05-22T15:45:05.239577+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_clean_scrub_stamp":"2023-05-22T15:45:04.802083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:08:32.792658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.21","version":"0'0","reported_seq":9,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-05-22T15:45:05.241689+0000","last_change":"2023-05-22T15:45:05.239247+0000","last_active":"2023-05-22T15:45:05.241689+0000","last_peered":"2023-05-22T15:45:05.241689+0000","last_clean":"2023-05-22T15:45:05.241689+0000","last_became_active":"2023-05-22T15:45:05.238970+0000","last_became_peered":"2023-05-22T15:45:05.238970+0000","last_unstale":"2023-05-22T15:45:05.241689+0000","last_undegraded":"2023-05-22T15:45:05.241689+0000","last_fullsized":"2023-05-22T15:45:05.241689+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_clean_scrub_stamp":"2023-05-22T15:45:04.802083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:52:07.479851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.20","version":"0'0","reported_seq":9,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-05-22T15:45:05.223278+0000","last_change":"2023-05-22T15:45:05.217281+0000","last_active":"2023-05-22T15:45:05.223278+0000","last_peered":"2023-05-22T15:45:05.223278+0000","last_clean":"2023-05-22T15:45:05.223278+0000","last_became_active":"2023-05-22T15:45:05.216989+0000","last_became_peered":"2023-05-22T15:45:05.216989+0000","last_unstale":"2023-05-22T15:45:05.223278+0000","last_undegraded":"2023-05-22T15:45:05.223278+0000","last_fullsized":"2023-05-22T15:45:05.223278+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_clean_scrub_stamp":"2023-05-22T15:45:04.802083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:08:32.792658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1e","version":"0'0","reported_seq":9,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-05-22T15:45:05.215467+0000","last_change":"2023-05-22T15:45:05.213555+0000","last_active":"2023-05-22T15:45:05.215467+0000","last_peered":"2023-05-22T15:45:05.215467+0000","last_clean":"2023-05-22T15:45:05.215467+0000","last_became_active":"2023-05-22T15:45:05.213311+0000","last_became_peered":"2023-05-22T15:45:05.213311+0000","last_unstale":"2023-05-22T15:45:05.215467+0000","last_undegraded":"2023-05-22T15:45:05.215467+0000","last_fullsized":"2023-05-22T15:45:05.215467+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_clean_scrub_stamp":"2023-05-22T15:45:04.802083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:11:24.311549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1f","version":"131'4","reported_seq":14,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-05-22T15:45:05.282593+0000","last_change":"2023-05-22T15:45:04.869973+0000","last_active":"2023-05-22T15:45:05.282593+0000","last_peered":"2023-05-22T15:45:05.282593+0000","last_clean":"2023-05-22T15:45:05.282593+0000","last_became_active":"2023-05-22T15:45:04.869509+0000","last_became_peered":"2023-05-22T15:45:04.869509+0000","last_unstale":"2023-05-22T15:45:05.282593+0000","last_undegraded":"2023-05-22T15:45:05.282593+0000","last_fullsized":"2023-05-22T15:45:05.282593+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:45:03.797855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:45:03.797855+0000","last_clean_scrub_stamp":"2023-05-22T15:45:03.797855+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:45:07.119273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":566,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1f","version":"0'0","reported_seq":9,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-05-22T15:45:05.276159+0000","last_change":"2023-05-22T15:45:05.274094+0000","last_active":"2023-05-22T15:45:05.276159+0000","last_peered":"2023-05-22T15:45:05.276159+0000","last_clean":"2023-05-22T15:45:05.276159+0000","last_became_active":"2023-05-22T15:45:05.273255+0000","last_became_peered":"2023-05-22T15:45:05.273255+0000","last_unstale":"2023-05-22T15:45:05.276159+0000","last_undegraded":"2023-05-22T15:45:05.276159+0000","last_fullsized":"2023-05-22T15:45:05.276159+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_clean_scrub_stamp":"2023-05-22T15:45:04.802083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:50:34.136689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1e","version":"131'1","reported_seq":11,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-05-22T15:45:05.278868+0000","last_change":"2023-05-22T15:45:04.871159+0000","last_active":"2023-05-22T15:45:05.278868+0000","last_peered":"2023-05-22T15:45:05.278868+0000","last_clean":"2023-05-22T15:45:05.278868+0000","last_became_active":"2023-05-22T15:45:04.870377+0000","last_became_peered":"2023-05-22T15:45:04.870377+0000","last_unstale":"2023-05-22T15:45:05.278868+0000","last_undegraded":"2023-05-22T15:45:05.278868+0000","last_fullsized":"2023-05-22T15:45:05.278868+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:45:03.797855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:45:03.797855+0000","last_clean_scrub_stamp":"2023-05-22T15:45:03.797855+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:03:57.988869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"0'0","reported_seq":9,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-05-22T15:45:05.223245+0000","last_change":"2023-05-22T15:45:05.217299+0000","last_active":"2023-05-22T15:45:05.223245+0000","last_peered":"2023-05-22T15:45:05.223245+0000","last_clean":"2023-05-22T15:45:05.223245+0000","last_became_active":"2023-05-22T15:45:05.217078+0000","last_became_peered":"2023-05-22T15:45:05.217078+0000","last_unstale":"2023-05-22T15:45:05.223245+0000","last_undegraded":"2023-05-22T15:45:05.223245+0000","last_fullsized":"2023-05-22T15:45:05.223245+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_clean_scrub_stamp":"2023-05-22T15:45:04.802083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:17:53.852853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1d","version":"131'2","reported_seq":12,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-05-22T15:45:05.242366+0000","last_change":"2023-05-22T15:45:04.842007+0000","last_active":"2023-05-22T15:45:05.242366+0000","last_peered":"2023-05-22T15:45:05.242366+0000","last_clean":"2023-05-22T15:45:05.242366+0000","last_became_active":"2023-05-22T15:45:04.841559+0000","last_became_peered":"2023-05-22T15:45:04.841559+0000","last_unstale":"2023-05-22T15:45:05.242366+0000","last_undegraded":"2023-05-22T15:45:05.242366+0000","last_fullsized":"2023-05-22T15:45:05.242366+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:45:03.797855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:45:03.797855+0000","last_clean_scrub_stamp":"2023-05-22T15:45:03.797855+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:45:07.119273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1d","version":"0'0","reported_seq":9,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-05-22T15:45:05.289852+0000","last_change":"2023-05-22T15:45:05.282125+0000","last_active":"2023-05-22T15:45:05.289852+0000","last_peered":"2023-05-22T15:45:05.289852+0000","last_clean":"2023-05-22T15:45:05.289852+0000","last_became_active":"2023-05-22T15:45:05.280933+0000","last_became_peered":"2023-05-22T15:45:05.280933+0000","last_unstale":"2023-05-22T15:45:05.289852+0000","last_undegraded":"2023-05-22T15:45:05.289852+0000","last_fullsized":"2023-05-22T15:45:05.289852+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_clean_scrub_stamp":"2023-05-22T15:45:04.802083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T18:44:18.645468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1c","version":"131'2","reported_seq":12,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-05-22T15:45:05.242370+0000","last_change":"2023-05-22T15:45:04.842614+0000","last_active":"2023-05-22T15:45:05.242370+0000","last_peered":"2023-05-22T15:45:05.242370+0000","last_clean":"2023-05-22T15:45:05.242370+0000","last_became_active":"2023-05-22T15:45:04.842097+0000","last_became_peered":"2023-05-22T15:45:04.842097+0000","last_unstale":"2023-05-22T15:45:05.242370+0000","last_undegraded":"2023-05-22T15:45:05.242370+0000","last_fullsized":"2023-05-22T15:45:05.242370+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:45:03.797855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:45:03.797855+0000","last_clean_scrub_stamp":"2023-05-22T15:45:03.797855+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T16:34:14.156084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1a","version":"0'0","reported_seq":9,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-05-22T15:45:05.229405+0000","last_change":"2023-05-22T15:45:05.224946+0000","last_active":"2023-05-22T15:45:05.229405+0000","last_peered":"2023-05-22T15:45:05.229405+0000","last_clean":"2023-05-22T15:45:05.229405+0000","last_became_active":"2023-05-22T15:45:05.224665+0000","last_became_peered":"2023-05-22T15:45:05.224665+0000","last_unstale":"2023-05-22T15:45:05.229405+0000","last_undegraded":"2023-05-22T15:45:05.229405+0000","last_fullsized":"2023-05-22T15:45:05.229405+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_clean_scrub_stamp":"2023-05-22T15:45:04.802083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T03:12:25.788950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1b","version":"131'1","reported_seq":11,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-05-22T15:45:05.260842+0000","last_change":"2023-05-22T15:45:04.858720+0000","last_active":"2023-05-22T15:45:05.260842+0000","last_peered":"2023-05-22T15:45:05.260842+0000","last_clean":"2023-05-22T15:45:05.260842+0000","last_became_active":"2023-05-22T15:45:04.857943+0000","last_became_peered":"2023-05-22T15:45:04.857943+0000","last_unstale":"2023-05-22T15:45:05.260842+0000","last_undegraded":"2023-05-22T15:45:05.260842+0000","last_fullsized":"2023-05-22T15:45:05.260842+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:45:03.797855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:45:03.797855+0000","last_clean_scrub_stamp":"2023-05-22T15:45:03.797855+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:37:19.722085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1b","version":"0'0","reported_seq":9,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-05-22T15:45:05.264903+0000","last_change":"2023-05-22T15:45:05.253189+0000","last_active":"2023-05-22T15:45:05.264903+0000","last_peered":"2023-05-22T15:45:05.264903+0000","last_clean":"2023-05-22T15:45:05.264903+0000","last_became_active":"2023-05-22T15:45:05.252282+0000","last_became_peered":"2023-05-22T15:45:05.252282+0000","last_unstale":"2023-05-22T15:45:05.264903+0000","last_undegraded":"2023-05-22T15:45:05.264903+0000","last_fullsized":"2023-05-22T15:45:05.264903+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_clean_scrub_stamp":"2023-05-22T15:45:04.802083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:00:24.701583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1a","version":"131'1","reported_seq":11,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-05-22T15:45:05.268632+0000","last_change":"2023-05-22T15:45:04.861987+0000","last_active":"2023-05-22T15:45:05.268632+0000","last_peered":"2023-05-22T15:45:05.268632+0000","last_clean":"2023-05-22T15:45:05.268632+0000","last_became_active":"2023-05-22T15:45:04.861703+0000","last_became_peered":"2023-05-22T15:45:04.861703+0000","last_unstale":"2023-05-22T15:45:05.268632+0000","last_undegraded":"2023-05-22T15:45:05.268632+0000","last_fullsized":"2023-05-22T15:45:05.268632+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:45:03.797855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:45:03.797855+0000","last_clean_scrub_stamp":"2023-05-22T15:45:03.797855+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T03:42:25.864815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.18","version":"0'0","reported_seq":9,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-05-22T15:45:05.241105+0000","last_change":"2023-05-22T15:45:05.235148+0000","last_active":"2023-05-22T15:45:05.241105+0000","last_peered":"2023-05-22T15:45:05.241105+0000","last_clean":"2023-05-22T15:45:05.241105+0000","last_became_active":"2023-05-22T15:45:05.234900+0000","last_became_peered":"2023-05-22T15:45:05.234900+0000","last_unstale":"2023-05-22T15:45:05.241105+0000","last_undegraded":"2023-05-22T15:45:05.241105+0000","last_fullsized":"2023-05-22T15:45:05.241105+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_clean_scrub_stamp":"2023-05-22T15:45:04.802083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:00:57.027365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":10,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-05-22T15:45:05.244747+0000","last_change":"2023-05-22T15:45:04.853214+0000","last_active":"2023-05-22T15:45:05.244747+0000","last_peered":"2023-05-22T15:45:05.244747+0000","last_clean":"2023-05-22T15:45:05.244747+0000","last_became_active":"2023-05-22T15:45:04.852332+0000","last_became_peered":"2023-05-22T15:45:04.852332+0000","last_unstale":"2023-05-22T15:45:05.244747+0000","last_undegraded":"2023-05-22T15:45:05.244747+0000","last_fullsized":"2023-05-22T15:45:05.244747+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:45:03.797855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:45:03.797855+0000","last_clean_scrub_stamp":"2023-05-22T15:45:03.797855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:30:53.313453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":9,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-05-22T15:45:05.259132+0000","last_change":"2023-05-22T15:45:05.244170+0000","last_active":"2023-05-22T15:45:05.259132+0000","last_peered":"2023-05-22T15:45:05.259132+0000","last_clean":"2023-05-22T15:45:05.259132+0000","last_became_active":"2023-05-22T15:45:05.243422+0000","last_became_peered":"2023-05-22T15:45:05.243422+0000","last_unstale":"2023-05-22T15:45:05.259132+0000","last_undegraded":"2023-05-22T15:45:05.259132+0000","last_fullsized":"2023-05-22T15:45:05.259132+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_clean_scrub_stamp":"2023-05-22T15:45:04.802083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T16:17:06.682365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":10,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-05-22T15:45:05.219570+0000","last_change":"2023-05-22T15:45:04.849089+0000","last_active":"2023-05-22T15:45:05.219570+0000","last_peered":"2023-05-22T15:45:05.219570+0000","last_clean":"2023-05-22T15:45:05.219570+0000","last_became_active":"2023-05-22T15:45:04.848736+0000","last_became_peered":"2023-05-22T15:45:04.848736+0000","last_unstale":"2023-05-22T15:45:05.219570+0000","last_undegraded":"2023-05-22T15:45:05.219570+0000","last_fullsized":"2023-05-22T15:45:05.219570+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:45:03.797855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:45:03.797855+0000","last_clean_scrub_stamp":"2023-05-22T15:45:03.797855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:01:00.083792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.e","version":"0'0","reported_seq":9,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-05-22T15:45:05.242260+0000","last_change":"2023-05-22T15:45:05.230275+0000","last_active":"2023-05-22T15:45:05.242260+0000","last_peered":"2023-05-22T15:45:05.242260+0000","last_clean":"2023-05-22T15:45:05.242260+0000","last_became_active":"2023-05-22T15:45:05.229991+0000","last_became_peered":"2023-05-22T15:45:05.229991+0000","last_unstale":"2023-05-22T15:45:05.242260+0000","last_undegraded":"2023-05-22T15:45:05.242260+0000","last_fullsized":"2023-05-22T15:45:05.242260+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_clean_scrub_stamp":"2023-05-22T15:45:04.802083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:08:32.792658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.f","version":"131'1","reported_seq":11,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-05-22T15:45:05.238814+0000","last_change":"2023-05-22T15:45:04.852319+0000","last_active":"2023-05-22T15:45:05.238814+0000","last_peered":"2023-05-22T15:45:05.238814+0000","last_clean":"2023-05-22T15:45:05.238814+0000","last_became_active":"2023-05-22T15:45:04.851954+0000","last_became_peered":"2023-05-22T15:45:04.851954+0000","last_unstale":"2023-05-22T15:45:05.238814+0000","last_undegraded":"2023-05-22T15:45:05.238814+0000","last_fullsized":"2023-05-22T15:45:05.238814+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:45:03.797855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:45:03.797855+0000","last_clean_scrub_stamp":"2023-05-22T15:45:03.797855+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T18:58:40.362754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.f","version":"0'0","reported_seq":9,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-05-22T15:45:05.236255+0000","last_change":"2023-05-22T15:45:05.234321+0000","last_active":"2023-05-22T15:45:05.236255+0000","last_peered":"2023-05-22T15:45:05.236255+0000","last_clean":"2023-05-22T15:45:05.236255+0000","last_became_active":"2023-05-22T15:45:05.234015+0000","last_became_peered":"2023-05-22T15:45:05.234015+0000","last_unstale":"2023-05-22T15:45:05.236255+0000","last_undegraded":"2023-05-22T15:45:05.236255+0000","last_fullsized":"2023-05-22T15:45:05.236255+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_clean_scrub_stamp":"2023-05-22T15:45:04.802083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T18:45:31.668005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.e","version":"131'2","reported_seq":12,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-05-22T15:45:05.252138+0000","last_change":"2023-05-22T15:45:04.841561+0000","last_active":"2023-05-22T15:45:05.252138+0000","last_peered":"2023-05-22T15:45:05.252138+0000","last_clean":"2023-05-22T15:45:05.252138+0000","last_became_active":"2023-05-22T15:45:04.841227+0000","last_became_peered":"2023-05-22T15:45:04.841227+0000","last_unstale":"2023-05-22T15:45:05.252138+0000","last_undegraded":"2023-05-22T15:45:05.252138+0000","last_fullsized":"2023-05-22T15:45:05.252138+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:45:03.797855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:45:03.797855+0000","last_clean_scrub_stamp":"2023-05-22T15:45:03.797855+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:11:23.307322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":9,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-05-22T15:45:05.289638+0000","last_change":"2023-05-22T15:45:05.279979+0000","last_active":"2023-05-22T15:45:05.289638+0000","last_peered":"2023-05-22T15:45:05.289638+0000","last_clean":"2023-05-22T15:45:05.289638+0000","last_became_active":"2023-05-22T15:45:05.279184+0000","last_became_peered":"2023-05-22T15:45:05.279184+0000","last_unstale":"2023-05-22T15:45:05.289638+0000","last_undegraded":"2023-05-22T15:45:05.289638+0000","last_fullsized":"2023-05-22T15:45:05.289638+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_clean_scrub_stamp":"2023-05-22T15:45:04.802083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:33:29.003697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.d","version":"131'1","reported_seq":11,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-05-22T15:45:05.273720+0000","last_change":"2023-05-22T15:45:04.852056+0000","last_active":"2023-05-22T15:45:05.273720+0000","last_peered":"2023-05-22T15:45:05.273720+0000","last_clean":"2023-05-22T15:45:05.273720+0000","last_became_active":"2023-05-22T15:45:04.851291+0000","last_became_peered":"2023-05-22T15:45:04.851291+0000","last_unstale":"2023-05-22T15:45:05.273720+0000","last_undegraded":"2023-05-22T15:45:05.273720+0000","last_fullsized":"2023-05-22T15:45:05.273720+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:45:03.797855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:45:03.797855+0000","last_clean_scrub_stamp":"2023-05-22T15:45:03.797855+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T18:38:16.669672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.d","version":"0'0","reported_seq":9,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-05-22T15:45:05.289836+0000","last_change":"2023-05-22T15:45:05.281784+0000","last_active":"2023-05-22T15:45:05.289836+0000","last_peered":"2023-05-22T15:45:05.289836+0000","last_clean":"2023-05-22T15:45:05.289836+0000","last_became_active":"2023-05-22T15:45:05.280662+0000","last_became_peered":"2023-05-22T15:45:05.280662+0000","last_unstale":"2023-05-22T15:45:05.289836+0000","last_undegraded":"2023-05-22T15:45:05.289836+0000","last_fullsized":"2023-05-22T15:45:05.289836+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_clean_scrub_stamp":"2023-05-22T15:45:04.802083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:00:24.701583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":10,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-05-22T15:45:05.214239+0000","last_change":"2023-05-22T15:45:04.860453+0000","last_active":"2023-05-22T15:45:05.214239+0000","last_peered":"2023-05-22T15:45:05.214239+0000","last_clean":"2023-05-22T15:45:05.214239+0000","last_became_active":"2023-05-22T15:45:04.859619+0000","last_became_peered":"2023-05-22T15:45:04.859619+0000","last_unstale":"2023-05-22T15:45:05.214239+0000","last_undegraded":"2023-05-22T15:45:05.214239+0000","last_fullsized":"2023-05-22T15:45:05.214239+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:45:03.797855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:45:03.797855+0000","last_clean_scrub_stamp":"2023-05-22T15:45:03.797855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T15:48:38.551891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.a","version":"0'0","reported_seq":9,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-05-22T15:45:05.289663+0000","last_change":"2023-05-22T15:45:05.280179+0000","last_active":"2023-05-22T15:45:05.289663+0000","last_peered":"2023-05-22T15:45:05.289663+0000","last_clean":"2023-05-22T15:45:05.289663+0000","last_became_active":"2023-05-22T15:45:05.279701+0000","last_became_peered":"2023-05-22T15:45:05.279701+0000","last_unstale":"2023-05-22T15:45:05.289663+0000","last_undegraded":"2023-05-22T15:45:05.289663+0000","last_fullsized":"2023-05-22T15:45:05.289663+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_clean_scrub_stamp":"2023-05-22T15:45:04.802083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:28:24.072722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.b","version":"131'2","reported_seq":12,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-05-22T15:45:05.231793+0000","last_change":"2023-05-22T15:45:04.842639+0000","last_active":"2023-05-22T15:45:05.231793+0000","last_peered":"2023-05-22T15:45:05.231793+0000","last_clean":"2023-05-22T15:45:05.231793+0000","last_became_active":"2023-05-22T15:45:04.842205+0000","last_became_peered":"2023-05-22T15:45:04.842205+0000","last_unstale":"2023-05-22T15:45:05.231793+0000","last_undegraded":"2023-05-22T15:45:05.231793+0000","last_fullsized":"2023-05-22T15:45:05.231793+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:45:03.797855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:45:03.797855+0000","last_clean_scrub_stamp":"2023-05-22T15:45:03.797855+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:26:15.565494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.b","version":"0'0","reported_seq":9,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-05-22T15:45:05.243067+0000","last_change":"2023-05-22T15:45:05.241929+0000","last_active":"2023-05-22T15:45:05.243067+0000","last_peered":"2023-05-22T15:45:05.243067+0000","last_clean":"2023-05-22T15:45:05.243067+0000","last_became_active":"2023-05-22T15:45:05.241689+0000","last_became_peered":"2023-05-22T15:45:05.241689+0000","last_unstale":"2023-05-22T15:45:05.243067+0000","last_undegraded":"2023-05-22T15:45:05.243067+0000","last_fullsized":"2023-05-22T15:45:05.243067+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_clean_scrub_stamp":"2023-05-22T15:45:04.802083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:40:25.027078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":10,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-05-22T15:45:05.207791+0000","last_change":"2023-05-22T15:45:04.845246+0000","last_active":"2023-05-22T15:45:05.207791+0000","last_peered":"2023-05-22T15:45:05.207791+0000","last_clean":"2023-05-22T15:45:05.207791+0000","last_became_active":"2023-05-22T15:45:04.844828+0000","last_became_peered":"2023-05-22T15:45:04.844828+0000","last_unstale":"2023-05-22T15:45:05.207791+0000","last_undegraded":"2023-05-22T15:45:05.207791+0000","last_fullsized":"2023-05-22T15:45:05.207791+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:45:03.797855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:45:03.797855+0000","last_clean_scrub_stamp":"2023-05-22T15:45:03.797855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:37:19.722085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.8","version":"0'0","reported_seq":9,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-05-22T15:45:05.289596+0000","last_change":"2023-05-22T15:45:05.279697+0000","last_active":"2023-05-22T15:45:05.289596+0000","last_peered":"2023-05-22T15:45:05.289596+0000","last_clean":"2023-05-22T15:45:05.289596+0000","last_became_active":"2023-05-22T15:45:05.278972+0000","last_became_peered":"2023-05-22T15:45:05.278972+0000","last_unstale":"2023-05-22T15:45:05.289596+0000","last_undegraded":"2023-05-22T15:45:05.289596+0000","last_fullsized":"2023-05-22T15:45:05.289596+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_clean_scrub_stamp":"2023-05-22T15:45:04.802083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:59:57.835660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":10,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-05-22T15:45:05.210256+0000","last_change":"2023-05-22T15:45:04.844292+0000","last_active":"2023-05-22T15:45:05.210256+0000","last_peered":"2023-05-22T15:45:05.210256+0000","last_clean":"2023-05-22T15:45:05.210256+0000","last_became_active":"2023-05-22T15:45:04.843144+0000","last_became_peered":"2023-05-22T15:45:04.843144+0000","last_unstale":"2023-05-22T15:45:05.210256+0000","last_undegraded":"2023-05-22T15:45:05.210256+0000","last_fullsized":"2023-05-22T15:45:05.210256+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:45:03.797855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:45:03.797855+0000","last_clean_scrub_stamp":"2023-05-22T15:45:03.797855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:19:28.324586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":9,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-05-22T15:45:05.234450+0000","last_change":"2023-05-22T15:45:05.231757+0000","last_active":"2023-05-22T15:45:05.234450+0000","last_peered":"2023-05-22T15:45:05.234450+0000","last_clean":"2023-05-22T15:45:05.234450+0000","last_became_active":"2023-05-22T15:45:05.231377+0000","last_became_peered":"2023-05-22T15:45:05.231377+0000","last_unstale":"2023-05-22T15:45:05.234450+0000","last_undegraded":"2023-05-22T15:45:05.234450+0000","last_fullsized":"2023-05-22T15:45:05.234450+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_clean_scrub_stamp":"2023-05-22T15:45:04.802083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:06:40.672215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":10,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-05-22T15:45:05.219571+0000","last_change":"2023-05-22T15:45:04.853025+0000","last_active":"2023-05-22T15:45:05.219571+0000","last_peered":"2023-05-22T15:45:05.219571+0000","last_clean":"2023-05-22T15:45:05.219571+0000","last_became_active":"2023-05-22T15:45:04.852645+0000","last_became_peered":"2023-05-22T15:45:04.852645+0000","last_unstale":"2023-05-22T15:45:05.219571+0000","last_undegraded":"2023-05-22T15:45:05.219571+0000","last_fullsized":"2023-05-22T15:45:05.219571+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:45:03.797855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:45:03.797855+0000","last_clean_scrub_stamp":"2023-05-22T15:45:03.797855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T03:35:58.750761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.6","version":"0'0","reported_seq":9,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-05-22T15:45:05.269921+0000","last_change":"2023-05-22T15:45:05.243786+0000","last_active":"2023-05-22T15:45:05.269921+0000","last_peered":"2023-05-22T15:45:05.269921+0000","last_clean":"2023-05-22T15:45:05.269921+0000","last_became_active":"2023-05-22T15:45:05.242992+0000","last_became_peered":"2023-05-22T15:45:05.242992+0000","last_unstale":"2023-05-22T15:45:05.269921+0000","last_undegraded":"2023-05-22T15:45:05.269921+0000","last_fullsized":"2023-05-22T15:45:05.269921+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_clean_scrub_stamp":"2023-05-22T15:45:04.802083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:21:00.848206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.7","version":"131'2","reported_seq":12,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-05-22T15:45:05.232831+0000","last_change":"2023-05-22T15:45:04.853894+0000","last_active":"2023-05-22T15:45:05.232831+0000","last_peered":"2023-05-22T15:45:05.232831+0000","last_clean":"2023-05-22T15:45:05.232831+0000","last_became_active":"2023-05-22T15:45:04.853572+0000","last_became_peered":"2023-05-22T15:45:04.853572+0000","last_unstale":"2023-05-22T15:45:05.232831+0000","last_undegraded":"2023-05-22T15:45:05.232831+0000","last_fullsized":"2023-05-22T15:45:05.232831+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:45:03.797855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:45:03.797855+0000","last_clean_scrub_stamp":"2023-05-22T15:45:03.797855+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:20:59.843979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.7","version":"0'0","reported_seq":9,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-05-22T15:45:05.252803+0000","last_change":"2023-05-22T15:45:05.251361+0000","last_active":"2023-05-22T15:45:05.252803+0000","last_peered":"2023-05-22T15:45:05.252803+0000","last_clean":"2023-05-22T15:45:05.252803+0000","last_became_active":"2023-05-22T15:45:05.251007+0000","last_became_peered":"2023-05-22T15:45:05.251007+0000","last_unstale":"2023-05-22T15:45:05.252803+0000","last_undegraded":"2023-05-22T15:45:05.252803+0000","last_fullsized":"2023-05-22T15:45:05.252803+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_clean_scrub_stamp":"2023-05-22T15:45:04.802083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T03:23:00.475670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.6","version":"131'1","reported_seq":11,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-05-22T15:45:05.735502+0000","last_change":"2023-05-22T15:45:04.855381+0000","last_active":"2023-05-22T15:45:05.735502+0000","last_peered":"2023-05-22T15:45:05.735502+0000","last_clean":"2023-05-22T15:45:05.735502+0000","last_became_active":"2023-05-22T15:45:04.853669+0000","last_became_peered":"2023-05-22T15:45:04.853669+0000","last_unstale":"2023-05-22T15:45:05.735502+0000","last_undegraded":"2023-05-22T15:45:05.735502+0000","last_fullsized":"2023-05-22T15:45:05.735502+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:45:03.797855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:45:03.797855+0000","last_clean_scrub_stamp":"2023-05-22T15:45:03.797855+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:33:35.429024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":9,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-05-22T15:45:05.241244+0000","last_change":"2023-05-22T15:45:05.238182+0000","last_active":"2023-05-22T15:45:05.241244+0000","last_peered":"2023-05-22T15:45:05.241244+0000","last_clean":"2023-05-22T15:45:05.241244+0000","last_became_active":"2023-05-22T15:45:05.237126+0000","last_became_peered":"2023-05-22T15:45:05.237126+0000","last_unstale":"2023-05-22T15:45:05.241244+0000","last_undegraded":"2023-05-22T15:45:05.241244+0000","last_fullsized":"2023-05-22T15:45:05.241244+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_clean_scrub_stamp":"2023-05-22T15:45:04.802083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:05:00.739104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":10,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-05-22T15:45:05.223114+0000","last_change":"2023-05-22T15:45:04.862699+0000","last_active":"2023-05-22T15:45:05.223114+0000","last_peered":"2023-05-22T15:45:05.223114+0000","last_clean":"2023-05-22T15:45:05.223114+0000","last_became_active":"2023-05-22T15:45:04.862334+0000","last_became_peered":"2023-05-22T15:45:04.862334+0000","last_unstale":"2023-05-22T15:45:05.223114+0000","last_undegraded":"2023-05-22T15:45:05.223114+0000","last_fullsized":"2023-05-22T15:45:05.223114+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:45:03.797855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:45:03.797855+0000","last_clean_scrub_stamp":"2023-05-22T15:45:03.797855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:41:00.086136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":9,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-05-22T15:45:05.266564+0000","last_change":"2023-05-22T15:45:05.237005+0000","last_active":"2023-05-22T15:45:05.266564+0000","last_peered":"2023-05-22T15:45:05.266564+0000","last_clean":"2023-05-22T15:45:05.266564+0000","last_became_active":"2023-05-22T15:45:05.236224+0000","last_became_peered":"2023-05-22T15:45:05.236224+0000","last_unstale":"2023-05-22T15:45:05.266564+0000","last_undegraded":"2023-05-22T15:45:05.266564+0000","last_fullsized":"2023-05-22T15:45:05.266564+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_clean_scrub_stamp":"2023-05-22T15:45:04.802083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T03:42:26.869043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":10,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-05-22T15:45:05.213382+0000","last_change":"2023-05-22T15:45:04.848269+0000","last_active":"2023-05-22T15:45:05.213382+0000","last_peered":"2023-05-22T15:45:05.213382+0000","last_clean":"2023-05-22T15:45:05.213382+0000","last_became_active":"2023-05-22T15:45:04.847276+0000","last_became_peered":"2023-05-22T15:45:04.847276+0000","last_unstale":"2023-05-22T15:45:05.213382+0000","last_undegraded":"2023-05-22T15:45:05.213382+0000","last_fullsized":"2023-05-22T15:45:05.213382+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:45:03.797855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:45:03.797855+0000","last_clean_scrub_stamp":"2023-05-22T15:45:03.797855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:50:33.132462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":9,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-05-22T15:45:05.269769+0000","last_change":"2023-05-22T15:45:05.263355+0000","last_active":"2023-05-22T15:45:05.269769+0000","last_peered":"2023-05-22T15:45:05.269769+0000","last_clean":"2023-05-22T15:45:05.269769+0000","last_became_active":"2023-05-22T15:45:05.262691+0000","last_became_peered":"2023-05-22T15:45:05.262691+0000","last_unstale":"2023-05-22T15:45:05.269769+0000","last_undegraded":"2023-05-22T15:45:05.269769+0000","last_fullsized":"2023-05-22T15:45:05.269769+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_clean_scrub_stamp":"2023-05-22T15:45:04.802083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T18:19:40.408404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":10,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-05-22T15:45:05.213848+0000","last_change":"2023-05-22T15:45:04.868721+0000","last_active":"2023-05-22T15:45:05.213848+0000","last_peered":"2023-05-22T15:45:05.213848+0000","last_clean":"2023-05-22T15:45:05.213848+0000","last_became_active":"2023-05-22T15:45:04.868048+0000","last_became_peered":"2023-05-22T15:45:04.868048+0000","last_unstale":"2023-05-22T15:45:05.213848+0000","last_undegraded":"2023-05-22T15:45:05.213848+0000","last_fullsized":"2023-05-22T15:45:05.213848+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:45:03.797855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:45:03.797855+0000","last_clean_scrub_stamp":"2023-05-22T15:45:03.797855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T16:34:14.156084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.3","version":"0'0","reported_seq":9,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-05-22T15:45:05.264965+0000","last_change":"2023-05-22T15:45:05.253288+0000","last_active":"2023-05-22T15:45:05.264965+0000","last_peered":"2023-05-22T15:45:05.264965+0000","last_clean":"2023-05-22T15:45:05.264965+0000","last_became_active":"2023-05-22T15:45:05.251454+0000","last_became_peered":"2023-05-22T15:45:05.251454+0000","last_unstale":"2023-05-22T15:45:05.264965+0000","last_undegraded":"2023-05-22T15:45:05.264965+0000","last_fullsized":"2023-05-22T15:45:05.264965+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_clean_scrub_stamp":"2023-05-22T15:45:04.802083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T03:18:50.763567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":10,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-05-22T15:45:05.245450+0000","last_change":"2023-05-22T15:45:04.846123+0000","last_active":"2023-05-22T15:45:05.245450+0000","last_peered":"2023-05-22T15:45:05.245450+0000","last_clean":"2023-05-22T15:45:05.245450+0000","last_became_active":"2023-05-22T15:45:04.845353+0000","last_became_peered":"2023-05-22T15:45:04.845353+0000","last_unstale":"2023-05-22T15:45:05.245450+0000","last_undegraded":"2023-05-22T15:45:05.245450+0000","last_fullsized":"2023-05-22T15:45:05.245450+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:45:03.797855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:45:03.797855+0000","last_clean_scrub_stamp":"2023-05-22T15:45:03.797855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T16:46:49.576208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.0","version":"0'0","reported_seq":9,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-05-22T15:45:05.289712+0000","last_change":"2023-05-22T15:45:05.280628+0000","last_active":"2023-05-22T15:45:05.289712+0000","last_peered":"2023-05-22T15:45:05.289712+0000","last_clean":"2023-05-22T15:45:05.289712+0000","last_became_active":"2023-05-22T15:45:05.279793+0000","last_became_peered":"2023-05-22T15:45:05.279793+0000","last_unstale":"2023-05-22T15:45:05.289712+0000","last_undegraded":"2023-05-22T15:45:05.289712+0000","last_fullsized":"2023-05-22T15:45:05.289712+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_clean_scrub_stamp":"2023-05-22T15:45:04.802083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T03:42:53.382337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1","version":"0'0","reported_seq":10,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-05-22T15:45:05.207719+0000","last_change":"2023-05-22T15:45:04.846570+0000","last_active":"2023-05-22T15:45:05.207719+0000","last_peered":"2023-05-22T15:45:05.207719+0000","last_clean":"2023-05-22T15:45:05.207719+0000","last_became_active":"2023-05-22T15:45:04.846265+0000","last_became_peered":"2023-05-22T15:45:04.846265+0000","last_unstale":"2023-05-22T15:45:05.207719+0000","last_undegraded":"2023-05-22T15:45:05.207719+0000","last_fullsized":"2023-05-22T15:45:05.207719+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:45:03.797855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:45:03.797855+0000","last_clean_scrub_stamp":"2023-05-22T15:45:03.797855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T03:34:25.195968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1","version":"0'0","reported_seq":9,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-05-22T15:45:05.241158+0000","last_change":"2023-05-22T15:45:05.235400+0000","last_active":"2023-05-22T15:45:05.241158+0000","last_peered":"2023-05-22T15:45:05.241158+0000","last_clean":"2023-05-22T15:45:05.241158+0000","last_became_active":"2023-05-22T15:45:05.235120+0000","last_became_peered":"2023-05-22T15:45:05.235120+0000","last_unstale":"2023-05-22T15:45:05.241158+0000","last_undegraded":"2023-05-22T15:45:05.241158+0000","last_fullsized":"2023-05-22T15:45:05.241158+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_clean_scrub_stamp":"2023-05-22T15:45:04.802083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:01:37.137003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":10,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-05-22T15:45:05.219684+0000","last_change":"2023-05-22T15:45:04.845876+0000","last_active":"2023-05-22T15:45:05.219684+0000","last_peered":"2023-05-22T15:45:05.219684+0000","last_clean":"2023-05-22T15:45:05.219684+0000","last_became_active":"2023-05-22T15:45:04.845574+0000","last_became_peered":"2023-05-22T15:45:04.845574+0000","last_unstale":"2023-05-22T15:45:05.219684+0000","last_undegraded":"2023-05-22T15:45:05.219684+0000","last_fullsized":"2023-05-22T15:45:05.219684+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:45:03.797855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:45:03.797855+0000","last_clean_scrub_stamp":"2023-05-22T15:45:03.797855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:06:39.667988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.10","version":"0'0","reported_seq":9,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-05-22T15:45:05.243392+0000","last_change":"2023-05-22T15:45:05.241940+0000","last_active":"2023-05-22T15:45:05.243392+0000","last_peered":"2023-05-22T15:45:05.243392+0000","last_clean":"2023-05-22T15:45:05.243392+0000","last_became_active":"2023-05-22T15:45:05.241683+0000","last_became_peered":"2023-05-22T15:45:05.241683+0000","last_unstale":"2023-05-22T15:45:05.243392+0000","last_undegraded":"2023-05-22T15:45:05.243392+0000","last_fullsized":"2023-05-22T15:45:05.243392+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_clean_scrub_stamp":"2023-05-22T15:45:04.802083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:23:52.692138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":10,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-05-22T15:45:05.216101+0000","last_change":"2023-05-22T15:45:04.860420+0000","last_active":"2023-05-22T15:45:05.216101+0000","last_peered":"2023-05-22T15:45:05.216101+0000","last_clean":"2023-05-22T15:45:05.216101+0000","last_became_active":"2023-05-22T15:45:04.859659+0000","last_became_peered":"2023-05-22T15:45:04.859659+0000","last_unstale":"2023-05-22T15:45:05.216101+0000","last_undegraded":"2023-05-22T15:45:05.216101+0000","last_fullsized":"2023-05-22T15:45:05.216101+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:45:03.797855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:45:03.797855+0000","last_clean_scrub_stamp":"2023-05-22T15:45:03.797855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:44:21.409644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"120'227","reported_seq":662,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-05-22T15:45:04.855004+0000","last_change":"2023-05-22T15:26:39.318204+0000","last_active":"2023-05-22T15:45:04.855004+0000","last_peered":"2023-05-22T15:45:04.855004+0000","last_clean":"2023-05-22T15:45:04.855004+0000","last_became_active":"2023-05-22T15:26:39.315370+0000","last_became_peered":"2023-05-22T15:26:39.315370+0000","last_unstale":"2023-05-22T15:45:04.855004+0000","last_undegraded":"2023-05-22T15:45:04.855004+0000","last_fullsized":"2023-05-22T15:45:04.855004+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:38.020076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:38.020076+0000","last_clean_scrub_stamp":"2023-05-22T15:26:38.020076+0000","objects_scrubbed":0,"log_size":227,"log_dups_size":0,"ondisk_log_size":227,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:06:39.207543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1076,"num_read_kb":2160,"num_write":359,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":125,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-05-22T15:45:04.853121+0000","last_change":"2023-05-22T15:26:44.971951+0000","last_active":"2023-05-22T15:45:04.853121+0000","last_peered":"2023-05-22T15:45:04.853121+0000","last_clean":"2023-05-22T15:45:04.853121+0000","last_became_active":"2023-05-22T15:26:44.970872+0000","last_became_peered":"2023-05-22T15:26:44.970872+0000","last_unstale":"2023-05-22T15:45:04.853121+0000","last_undegraded":"2023-05-22T15:45:04.853121+0000","last_fullsized":"2023-05-22T15:45:04.853121+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:55:34.752949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.13","version":"0'0","reported_seq":9,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-05-22T15:45:05.271605+0000","last_change":"2023-05-22T15:45:05.271178+0000","last_active":"2023-05-22T15:45:05.271605+0000","last_peered":"2023-05-22T15:45:05.271605+0000","last_clean":"2023-05-22T15:45:05.271605+0000","last_became_active":"2023-05-22T15:45:05.270957+0000","last_became_peered":"2023-05-22T15:45:05.270957+0000","last_unstale":"2023-05-22T15:45:05.271605+0000","last_undegraded":"2023-05-22T15:45:05.271605+0000","last_fullsized":"2023-05-22T15:45:05.271605+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_clean_scrub_stamp":"2023-05-22T15:45:04.802083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T18:19:40.408404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":10,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-05-22T15:45:05.250590+0000","last_change":"2023-05-22T15:45:04.845921+0000","last_active":"2023-05-22T15:45:05.250590+0000","last_peered":"2023-05-22T15:45:05.250590+0000","last_clean":"2023-05-22T15:45:05.250590+0000","last_became_active":"2023-05-22T15:45:04.845130+0000","last_became_peered":"2023-05-22T15:45:04.845130+0000","last_unstale":"2023-05-22T15:45:05.250590+0000","last_undegraded":"2023-05-22T15:45:05.250590+0000","last_fullsized":"2023-05-22T15:45:05.250590+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:45:03.797855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:45:03.797855+0000","last_clean_scrub_stamp":"2023-05-22T15:45:03.797855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T16:17:05.678138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":119,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-05-22T15:45:04.846752+0000","last_change":"2023-05-22T15:26:44.977198+0000","last_active":"2023-05-22T15:45:04.846752+0000","last_peered":"2023-05-22T15:45:04.846752+0000","last_clean":"2023-05-22T15:45:04.846752+0000","last_became_active":"2023-05-22T15:26:44.976445+0000","last_became_peered":"2023-05-22T15:26:44.976445+0000","last_unstale":"2023-05-22T15:45:04.846752+0000","last_undegraded":"2023-05-22T15:45:04.846752+0000","last_fullsized":"2023-05-22T15:45:04.846752+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:12:41.464054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":9,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-05-22T15:45:05.228187+0000","last_change":"2023-05-22T15:45:05.225003+0000","last_active":"2023-05-22T15:45:05.228187+0000","last_peered":"2023-05-22T15:45:05.228187+0000","last_clean":"2023-05-22T15:45:05.228187+0000","last_became_active":"2023-05-22T15:45:05.223711+0000","last_became_peered":"2023-05-22T15:45:05.223711+0000","last_unstale":"2023-05-22T15:45:05.228187+0000","last_undegraded":"2023-05-22T15:45:05.228187+0000","last_fullsized":"2023-05-22T15:45:05.228187+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_clean_scrub_stamp":"2023-05-22T15:45:04.802083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:41:01.090364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.13","version":"131'2","reported_seq":12,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-05-22T15:45:05.271146+0000","last_change":"2023-05-22T15:45:04.867655+0000","last_active":"2023-05-22T15:45:05.271146+0000","last_peered":"2023-05-22T15:45:05.271146+0000","last_clean":"2023-05-22T15:45:05.271146+0000","last_became_active":"2023-05-22T15:45:04.867350+0000","last_became_peered":"2023-05-22T15:45:04.867350+0000","last_unstale":"2023-05-22T15:45:05.271146+0000","last_undegraded":"2023-05-22T15:45:05.271146+0000","last_fullsized":"2023-05-22T15:45:05.271146+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:45:03.797855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:45:03.797855+0000","last_clean_scrub_stamp":"2023-05-22T15:45:03.797855+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:53:11.065522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":119,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-05-22T15:45:04.820605+0000","last_change":"2023-05-22T15:26:44.969469+0000","last_active":"2023-05-22T15:45:04.820605+0000","last_peered":"2023-05-22T15:45:04.820605+0000","last_clean":"2023-05-22T15:45:04.820605+0000","last_became_active":"2023-05-22T15:26:44.969083+0000","last_became_peered":"2023-05-22T15:26:44.969083+0000","last_unstale":"2023-05-22T15:45:04.820605+0000","last_undegraded":"2023-05-22T15:45:04.820605+0000","last_fullsized":"2023-05-22T15:45:04.820605+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:29:23.670067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.11","version":"0'0","reported_seq":9,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-05-22T15:45:05.237152+0000","last_change":"2023-05-22T15:45:05.236102+0000","last_active":"2023-05-22T15:45:05.237152+0000","last_peered":"2023-05-22T15:45:05.237152+0000","last_clean":"2023-05-22T15:45:05.237152+0000","last_became_active":"2023-05-22T15:45:05.235808+0000","last_became_peered":"2023-05-22T15:45:05.235808+0000","last_unstale":"2023-05-22T15:45:05.237152+0000","last_undegraded":"2023-05-22T15:45:05.237152+0000","last_fullsized":"2023-05-22T15:45:05.237152+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_clean_scrub_stamp":"2023-05-22T15:45:04.802083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:17:05.563955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.10","version":"131'8","reported_seq":18,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-05-22T15:45:05.240165+0000","last_change":"2023-05-22T15:45:04.841261+0000","last_active":"2023-05-22T15:45:05.240165+0000","last_peered":"2023-05-22T15:45:05.240165+0000","last_clean":"2023-05-22T15:45:05.240165+0000","last_became_active":"2023-05-22T15:45:04.840898+0000","last_became_peered":"2023-05-22T15:45:04.840898+0000","last_unstale":"2023-05-22T15:45:05.240165+0000","last_undegraded":"2023-05-22T15:45:05.240165+0000","last_fullsized":"2023-05-22T15:45:05.240165+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:45:03.797855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:45:03.797855+0000","last_clean_scrub_stamp":"2023-05-22T15:45:03.797855+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:16:37.560794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":125,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-05-22T15:45:04.852215+0000","last_change":"2023-05-22T15:26:44.976632+0000","last_active":"2023-05-22T15:45:04.852215+0000","last_peered":"2023-05-22T15:45:04.852215+0000","last_clean":"2023-05-22T15:45:04.852215+0000","last_became_active":"2023-05-22T15:26:44.975546+0000","last_became_peered":"2023-05-22T15:26:44.975546+0000","last_unstale":"2023-05-22T15:45:04.852215+0000","last_undegraded":"2023-05-22T15:45:04.852215+0000","last_fullsized":"2023-05-22T15:45:04.852215+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:42:57.973030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.17","version":"0'0","reported_seq":9,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-05-22T15:45:05.274342+0000","last_change":"2023-05-22T15:45:05.266421+0000","last_active":"2023-05-22T15:45:05.274342+0000","last_peered":"2023-05-22T15:45:05.274342+0000","last_clean":"2023-05-22T15:45:05.274342+0000","last_became_active":"2023-05-22T15:45:05.254074+0000","last_became_peered":"2023-05-22T15:45:05.254074+0000","last_unstale":"2023-05-22T15:45:05.274342+0000","last_undegraded":"2023-05-22T15:45:05.274342+0000","last_fullsized":"2023-05-22T15:45:05.274342+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_clean_scrub_stamp":"2023-05-22T15:45:04.802083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:25:31.706898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":10,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-05-22T15:45:05.212610+0000","last_change":"2023-05-22T15:45:04.852022+0000","last_active":"2023-05-22T15:45:05.212610+0000","last_peered":"2023-05-22T15:45:05.212610+0000","last_clean":"2023-05-22T15:45:05.212610+0000","last_became_active":"2023-05-22T15:45:04.851734+0000","last_became_peered":"2023-05-22T15:45:04.851734+0000","last_unstale":"2023-05-22T15:45:05.212610+0000","last_undegraded":"2023-05-22T15:45:05.212610+0000","last_fullsized":"2023-05-22T15:45:05.212610+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:45:03.797855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:45:03.797855+0000","last_clean_scrub_stamp":"2023-05-22T15:45:03.797855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:25:30.702670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":111,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-05-22T15:45:04.838954+0000","last_change":"2023-05-22T15:26:45.091475+0000","last_active":"2023-05-22T15:45:04.838954+0000","last_peered":"2023-05-22T15:45:04.838954+0000","last_clean":"2023-05-22T15:45:04.838954+0000","last_became_active":"2023-05-22T15:26:45.091143+0000","last_became_peered":"2023-05-22T15:26:45.091143+0000","last_unstale":"2023-05-22T15:45:04.838954+0000","last_undegraded":"2023-05-22T15:45:04.838954+0000","last_fullsized":"2023-05-22T15:45:04.838954+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:12:41.464054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.16","version":"0'0","reported_seq":9,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-05-22T15:45:05.221872+0000","last_change":"2023-05-22T15:45:05.219266+0000","last_active":"2023-05-22T15:45:05.221872+0000","last_peered":"2023-05-22T15:45:05.221872+0000","last_clean":"2023-05-22T15:45:05.221872+0000","last_became_active":"2023-05-22T15:45:05.218279+0000","last_became_peered":"2023-05-22T15:45:05.218279+0000","last_unstale":"2023-05-22T15:45:05.221872+0000","last_undegraded":"2023-05-22T15:45:05.221872+0000","last_fullsized":"2023-05-22T15:45:05.221872+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_clean_scrub_stamp":"2023-05-22T15:45:04.802083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:17:53.852853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":10,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-05-22T15:45:05.208422+0000","last_change":"2023-05-22T15:45:04.846152+0000","last_active":"2023-05-22T15:45:05.208422+0000","last_peered":"2023-05-22T15:45:05.208422+0000","last_clean":"2023-05-22T15:45:05.208422+0000","last_became_active":"2023-05-22T15:45:04.845709+0000","last_became_peered":"2023-05-22T15:45:04.845709+0000","last_unstale":"2023-05-22T15:45:05.208422+0000","last_undegraded":"2023-05-22T15:45:05.208422+0000","last_fullsized":"2023-05-22T15:45:05.208422+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:45:03.797855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:45:03.797855+0000","last_clean_scrub_stamp":"2023-05-22T15:45:03.797855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:44:21.409644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":118,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-05-22T15:45:04.856499+0000","last_change":"2023-05-22T15:26:45.092367+0000","last_active":"2023-05-22T15:45:04.856499+0000","last_peered":"2023-05-22T15:45:04.856499+0000","last_clean":"2023-05-22T15:45:04.856499+0000","last_became_active":"2023-05-22T15:26:45.091951+0000","last_became_peered":"2023-05-22T15:26:45.091951+0000","last_unstale":"2023-05-22T15:45:04.856499+0000","last_undegraded":"2023-05-22T15:45:04.856499+0000","last_fullsized":"2023-05-22T15:45:04.856499+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:42:57.973030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.15","version":"0'0","reported_seq":9,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-05-22T15:45:05.285475+0000","last_change":"2023-05-22T15:45:05.283232+0000","last_active":"2023-05-22T15:45:05.285475+0000","last_peered":"2023-05-22T15:45:05.285475+0000","last_clean":"2023-05-22T15:45:05.285475+0000","last_became_active":"2023-05-22T15:45:05.282448+0000","last_became_peered":"2023-05-22T15:45:05.282448+0000","last_unstale":"2023-05-22T15:45:05.285475+0000","last_undegraded":"2023-05-22T15:45:05.285475+0000","last_fullsized":"2023-05-22T15:45:05.285475+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_clean_scrub_stamp":"2023-05-22T15:45:04.802083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:33:36.433252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.14","version":"131'6","reported_seq":16,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-05-22T15:45:05.260939+0000","last_change":"2023-05-22T15:45:04.849926+0000","last_active":"2023-05-22T15:45:05.260939+0000","last_peered":"2023-05-22T15:45:05.260939+0000","last_clean":"2023-05-22T15:45:05.260939+0000","last_became_active":"2023-05-22T15:45:04.849626+0000","last_became_peered":"2023-05-22T15:45:04.849626+0000","last_unstale":"2023-05-22T15:45:05.260939+0000","last_undegraded":"2023-05-22T15:45:05.260939+0000","last_fullsized":"2023-05-22T15:45:05.260939+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:45:03.797855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:45:03.797855+0000","last_clean_scrub_stamp":"2023-05-22T15:45:03.797855+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:02:33.642419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1548,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":111,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-05-22T15:45:04.838884+0000","last_change":"2023-05-22T15:26:44.978858+0000","last_active":"2023-05-22T15:45:04.838884+0000","last_peered":"2023-05-22T15:45:04.838884+0000","last_clean":"2023-05-22T15:45:04.838884+0000","last_became_active":"2023-05-22T15:26:44.978443+0000","last_became_peered":"2023-05-22T15:26:44.978443+0000","last_unstale":"2023-05-22T15:45:04.838884+0000","last_undegraded":"2023-05-22T15:45:04.838884+0000","last_fullsized":"2023-05-22T15:45:04.838884+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:46:29.290243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.14","version":"0'0","reported_seq":9,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-05-22T15:45:05.249290+0000","last_change":"2023-05-22T15:45:05.248517+0000","last_active":"2023-05-22T15:45:05.249290+0000","last_peered":"2023-05-22T15:45:05.249290+0000","last_clean":"2023-05-22T15:45:05.249290+0000","last_became_active":"2023-05-22T15:45:05.248227+0000","last_became_peered":"2023-05-22T15:45:05.248227+0000","last_unstale":"2023-05-22T15:45:05.249290+0000","last_undegraded":"2023-05-22T15:45:05.249290+0000","last_fullsized":"2023-05-22T15:45:05.249290+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:45:04.802083+0000","last_clean_scrub_stamp":"2023-05-22T15:45:04.802083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T03:35:59.754988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.15","version":"131'3","reported_seq":13,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-05-22T15:45:05.237375+0000","last_change":"2023-05-22T15:45:04.848230+0000","last_active":"2023-05-22T15:45:05.237375+0000","last_peered":"2023-05-22T15:45:05.237375+0000","last_clean":"2023-05-22T15:45:05.237375+0000","last_became_active":"2023-05-22T15:45:04.847778+0000","last_became_peered":"2023-05-22T15:45:04.847778+0000","last_unstale":"2023-05-22T15:45:05.237375+0000","last_undegraded":"2023-05-22T15:45:05.237375+0000","last_fullsized":"2023-05-22T15:45:05.237375+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:45:03.797855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:45:03.797855+0000","last_clean_scrub_stamp":"2023-05-22T15:45:03.797855+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:17:04.559727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":112,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-05-22T15:45:04.833279+0000","last_change":"2023-05-22T15:26:44.982421+0000","last_active":"2023-05-22T15:45:04.833279+0000","last_peered":"2023-05-22T15:45:04.833279+0000","last_clean":"2023-05-22T15:45:04.833279+0000","last_became_active":"2023-05-22T15:26:44.981586+0000","last_became_peered":"2023-05-22T15:26:44.981586+0000","last_unstale":"2023-05-22T15:45:04.833279+0000","last_undegraded":"2023-05-22T15:45:04.833279+0000","last_fullsized":"2023-05-22T15:45:04.833279+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:14:11.156544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":17,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":2330,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1076,"num_read_kb":2160,"num_write":359,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":227,"ondisk_log_size":227,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575069,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35652,"kb_used_data":2172,"kb_used_omap":0,"kb_used_meta":33472,"kb_avail":94336188,"statfs":{"total":96636764160,"available":96600256512,"internally_reserved":0,"allocated":2224128,"data_stored":1445607,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":34275328},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 15:44:03 2023","interfaces":[{"interface":"back","average":{"1min":0.999,"5min":0.874,"15min":0.874},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":2.637,"5min":2.637,"15min":2.637},"last":1.1830000000000001},{"interface":"front","average":{"1min":0.93899999999999995,"5min":0.875,"15min":0.89100000000000001},"min":{"1min":0.502,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":3.0449999999999999,"5min":3.0449999999999999,"15min":3.0449999999999999},"last":1.1040000000000001}]},{"osd":1,"last update":"Mon May 22 15:44:03 2023","interfaces":[{"interface":"back","average":{"1min":0.88900000000000001,"5min":0.93500000000000005,"15min":0.97699999999999998},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":1.2270000000000001,"5min":1.385,"15min":1.389},"last":0.83799999999999997},{"interface":"front","average":{"1min":1.0169999999999999,"5min":0.997,"15min":1.0209999999999999},"min":{"1min":0.63800000000000001,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":2.7269999999999999,"5min":2.7269999999999999,"15min":2.7269999999999999},"last":1.333}]},{"osd":2,"last update":"Mon May 22 15:44:03 2023","interfaces":[{"interface":"back","average":{"1min":0.78900000000000003,"5min":0.83999999999999997,"15min":0.877},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.1519999999999999,"5min":1.2130000000000001,"15min":1.244},"last":0.76200000000000001},{"interface":"front","average":{"1min":0.99099999999999999,"5min":0.92700000000000005,"15min":0.95599999999999996},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":3.141,"5min":3.141,"15min":3.141},"last":1.5049999999999999}]},{"osd":3,"last update":"Mon May 22 15:44:03 2023","interfaces":[{"interface":"back","average":{"1min":1.0289999999999999,"5min":0.95799999999999996,"15min":0.95999999999999996},"min":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":2.8149999999999999,"5min":2.8149999999999999,"15min":2.8149999999999999},"last":1.569},{"interface":"front","average":{"1min":0.91500000000000004,"5min":0.93500000000000005,"15min":0.97299999999999998},"min":{"1min":0.504,"5min":0.504,"15min":0.504},"max":{"1min":1.101,"5min":1.3029999999999999,"15min":1.4710000000000001},"last":1.389}]},{"osd":4,"last update":"Mon May 22 15:44:03 2023","interfaces":[{"interface":"back","average":{"1min":1.0429999999999999,"5min":0.93799999999999994,"15min":0.89900000000000002},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":3.472,"5min":3.472,"15min":3.472},"last":1.591},{"interface":"front","average":{"1min":1.915,"5min":1.175,"15min":1.036},"min":{"1min":0.56000000000000005,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":15.627000000000001,"5min":15.627000000000001,"15min":15.627000000000001},"last":1.4199999999999999}]},{"osd":5,"last update":"Mon May 22 15:44:03 2023","interfaces":[{"interface":"back","average":{"1min":1.202,"5min":1.0569999999999999,"15min":1.0069999999999999},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":4.5599999999999996,"5min":4.5599999999999996,"15min":4.5599999999999996},"last":1.2370000000000001},{"interface":"front","average":{"1min":1.2709999999999999,"5min":1.1439999999999999,"15min":1.1339999999999999},"min":{"1min":0.63800000000000001,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":4.3799999999999999,"5min":4.3799999999999999,"15min":4.3799999999999999},"last":1.7649999999999999}]},{"osd":6,"last update":"Mon May 22 15:44:03 2023","interfaces":[{"interface":"back","average":{"1min":1.381,"5min":1.113,"15min":1.077},"min":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":6.5069999999999997,"5min":6.5069999999999997,"15min":6.5069999999999997},"last":1.8660000000000001},{"interface":"front","average":{"1min":1.2849999999999999,"5min":1.1439999999999999,"15min":1.135},"min":{"1min":0.62,"5min":0.62,"15min":0.62},"max":{"1min":3.3500000000000001,"5min":3.3500000000000001,"15min":3.3500000000000001},"last":2.0129999999999999}]}]},{"osd":6,"up_from":13,"seq":55834575070,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31676,"kb_used_data":1588,"kb_used_omap":0,"kb_used_meta":30080,"kb_avail":94340164,"statfs":{"total":96636764160,"available":96604327936,"internally_reserved":0,"allocated":1626112,"data_stored":854709,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":30801920},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 15:44:14 2023","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":0.85799999999999998,"15min":0.89800000000000002},"min":{"1min":0.57499999999999996,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":1.26,"5min":1.486,"15min":1.486},"last":0.73099999999999998},{"interface":"front","average":{"1min":0.97599999999999998,"5min":0.92000000000000004,"15min":0.91200000000000003},"min":{"1min":0.64300000000000002,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":1.6240000000000001,"5min":1.6240000000000001,"15min":1.6240000000000001},"last":1.1240000000000001}]},{"osd":1,"last update":"Mon May 22 15:44:14 2023","interfaces":[{"interface":"back","average":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.92300000000000004},"min":{"1min":0.64600000000000002,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":1.4039999999999999,"5min":1.4039999999999999,"15min":1.4039999999999999},"last":0.997},{"interface":"front","average":{"1min":0.879,"5min":0.91500000000000004,"15min":0.94499999999999995},"min":{"1min":0.65500000000000003,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":1.2509999999999999,"5min":1.554,"15min":1.554},"last":0.78300000000000003}]},{"osd":2,"last update":"Mon May 22 15:44:14 2023","interfaces":[{"interface":"back","average":{"1min":0.97499999999999998,"5min":0.95499999999999996,"15min":0.98399999999999999},"min":{"1min":0.68100000000000005,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":1.3200000000000001,"5min":1.425,"15min":1.425},"last":1.036},{"interface":"front","average":{"1min":1.0580000000000001,"5min":1.024,"15min":1.044},"min":{"1min":0.78300000000000003,"5min":0.442,"15min":0.442},"max":{"1min":1.673,"5min":1.673,"15min":1.673},"last":1.0860000000000001}]},{"osd":3,"last update":"Mon May 22 15:44:14 2023","interfaces":[{"interface":"back","average":{"1min":0.97699999999999998,"5min":0.98099999999999998,"15min":0.98399999999999999},"min":{"1min":0.753,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":1.329,"5min":1.679,"15min":1.679},"last":0.879},{"interface":"front","average":{"1min":0.96799999999999997,"5min":0.94899999999999995,"15min":0.95899999999999996},"min":{"1min":0.53200000000000003,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.3009999999999999,"5min":1.4570000000000001,"15min":1.4570000000000001},"last":1.1559999999999999}]},{"osd":4,"last update":"Mon May 22 15:44:14 2023","interfaces":[{"interface":"back","average":{"1min":1.0109999999999999,"5min":0.95399999999999996,"15min":0.92400000000000004},"min":{"1min":0.59699999999999998,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":1.581,"5min":1.6859999999999999,"15min":1.6859999999999999},"last":0.91700000000000004},{"interface":"front","average":{"1min":1.1020000000000001,"5min":1.03,"15min":1.0229999999999999},"min":{"1min":0.65800000000000003,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":1.452,"5min":1.8300000000000001,"15min":1.8300000000000001},"last":0.95799999999999996}]},{"osd":5,"last update":"Mon May 22 15:44:14 2023","interfaces":[{"interface":"back","average":{"1min":1.0880000000000001,"5min":1.0029999999999999,"15min":0.97499999999999998},"min":{"1min":0.76600000000000001,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":1.296,"5min":1.6499999999999999,"15min":1.6499999999999999},"last":1.2529999999999999},{"interface":"front","average":{"1min":1.1830000000000001,"5min":1.139,"15min":1.1020000000000001},"min":{"1min":0.71699999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.7689999999999999,"5min":1.7689999999999999,"15min":1.7689999999999999},"last":1.1950000000000001}]},{"osd":7,"last update":"Mon May 22 15:44:14 2023","interfaces":[{"interface":"back","average":{"1min":1.1779999999999999,"5min":1.1359999999999999,"15min":1.1259999999999999},"min":{"1min":0.76700000000000002,"5min":0.37,"15min":0.37},"max":{"1min":1.853,"5min":1.9179999999999999,"15min":1.9179999999999999},"last":1.224},{"interface":"front","average":{"1min":1.1619999999999999,"5min":1.1359999999999999,"15min":1.1279999999999999},"min":{"1min":0.78700000000000003,"5min":0.442,"15min":0.442},"max":{"1min":1.518,"5min":1.8520000000000001,"15min":1.8520000000000001},"last":1.3200000000000001}]}]},{"osd":5,"up_from":13,"seq":55834575070,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":25536,"kb_used_data":1592,"kb_used_omap":0,"kb_used_meta":23936,"kb_avail":94346304,"statfs":{"total":96636764160,"available":96610615296,"internally_reserved":0,"allocated":1630208,"data_stored":855227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":24510464},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 15:44:09 2023","interfaces":[{"interface":"back","average":{"1min":0.85799999999999998,"5min":0.83899999999999997,"15min":0.88600000000000001},"min":{"1min":0.52000000000000002,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.171,"5min":1.3440000000000001,"15min":1.3440000000000001},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.81299999999999994,"5min":0.86399999999999999,"15min":0.88200000000000001},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":1.095,"5min":2.1699999999999999,"15min":2.1699999999999999},"last":1.0269999999999999}]},{"osd":1,"last update":"Mon May 22 15:44:09 2023","interfaces":[{"interface":"back","average":{"1min":0.95999999999999996,"5min":0.96199999999999997,"15min":0.997},"min":{"1min":0.61499999999999999,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":1.248,"5min":1.9370000000000001,"15min":1.9370000000000001},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.97099999999999997,"5min":0.96799999999999997,"15min":1.0249999999999999},"min":{"1min":0.65000000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.375,"5min":1.375,"15min":1.375},"last":0.89700000000000002}]},{"osd":2,"last update":"Mon May 22 15:44:09 2023","interfaces":[{"interface":"back","average":{"1min":0.88600000000000001,"5min":0.94399999999999995,"15min":0.93799999999999994},"min":{"1min":0.57499999999999996,"5min":0.504,"15min":0.504},"max":{"1min":1.1659999999999999,"5min":1.9830000000000001,"15min":1.9830000000000001},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.95699999999999996,"5min":0.94999999999999996,"15min":0.94999999999999996},"min":{"1min":0.72499999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"max":{"1min":1.121,"5min":2.1120000000000001,"15min":2.1120000000000001},"last":0.69399999999999995}]},{"osd":3,"last update":"Mon May 22 15:44:09 2023","interfaces":[{"interface":"back","average":{"1min":0.92700000000000005,"5min":0.94299999999999995,"15min":0.95599999999999996},"min":{"1min":0.69099999999999995,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.2629999999999999,"5min":1.4419999999999999,"15min":1.4419999999999999},"last":0.80700000000000005},{"interface":"front","average":{"1min":1.024,"5min":1.036,"15min":1.034},"min":{"1min":0.69699999999999995,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":1.337,"5min":2.52,"15min":2.52},"last":0.93600000000000005}]},{"osd":4,"last update":"Mon May 22 15:44:09 2023","interfaces":[{"interface":"back","average":{"1min":0.93799999999999994,"5min":0.97299999999999998,"15min":0.93200000000000005},"min":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.57599999999999996},"max":{"1min":1.28,"5min":2.226,"15min":2.226},"last":0.77400000000000002},{"interface":"front","average":{"1min":1.0609999999999999,"5min":1.04,"15min":1.0269999999999999},"min":{"1min":0.78200000000000003,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.3859999999999999,"5min":1.6859999999999999,"15min":1.6859999999999999},"last":0.98599999999999999}]},{"osd":6,"last update":"Mon May 22 15:44:09 2023","interfaces":[{"interface":"back","average":{"1min":1.034,"5min":1.0620000000000001,"15min":1.032},"min":{"1min":0.68400000000000005,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.246,"5min":2.831,"15min":2.831},"last":1.0860000000000001},{"interface":"front","average":{"1min":1.119,"5min":1.0980000000000001,"15min":1.0720000000000001},"min":{"1min":0.65000000000000002,"5min":0.629,"15min":0.629},"max":{"1min":1.3560000000000001,"5min":2.3690000000000002,"15min":2.3690000000000002},"last":0.87}]},{"osd":7,"last update":"Mon May 22 15:44:09 2023","interfaces":[{"interface":"back","average":{"1min":1.1259999999999999,"5min":1.1279999999999999,"15min":1.113},"min":{"1min":0.72899999999999998,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":1.4690000000000001,"5min":1.849,"15min":1.849},"last":1.0609999999999999},{"interface":"front","average":{"1min":1.1739999999999999,"5min":1.1899999999999999,"15min":1.1619999999999999},"min":{"1min":0.78500000000000003,"5min":0.75700000000000001,"15min":0.71699999999999997},"max":{"1min":1.4319999999999999,"5min":2.7429999999999999,"15min":2.7429999999999999},"last":1.1319999999999999}]}]},{"osd":4,"up_from":13,"seq":55834575068,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27064,"kb_used_data":1584,"kb_used_omap":0,"kb_used_meta":25472,"kb_avail":94344776,"statfs":{"total":96636764160,"available":96609050624,"internally_reserved":0,"allocated":1622016,"data_stored":854639,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":26083328},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 15:44:15 2023","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":0.85999999999999999,"15min":0.90000000000000002},"min":{"1min":0.48999999999999999,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":1.1220000000000001,"5min":1.9910000000000001,"15min":1.9910000000000001},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.85099999999999998,"5min":0.89500000000000002,"15min":0.93400000000000005},"min":{"1min":0.499,"5min":0.373,"15min":0.373},"max":{"1min":1.1539999999999999,"5min":1.3779999999999999,"15min":1.633},"last":0.59899999999999998}]},{"osd":1,"last update":"Mon May 22 15:44:15 2023","interfaces":[{"interface":"back","average":{"1min":0.876,"5min":0.93400000000000005,"15min":0.96999999999999997},"min":{"1min":0.70899999999999996,"5min":0.436,"15min":0.436},"max":{"1min":1.24,"5min":1.325,"15min":1.383},"last":0.81599999999999995},{"interface":"front","average":{"1min":0.86699999999999999,"5min":0.85899999999999999,"15min":0.90800000000000003},"min":{"1min":0.48699999999999999,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.2410000000000001,"5min":1.298,"15min":1.3640000000000001},"last":0.68700000000000006}]},{"osd":2,"last update":"Mon May 22 15:44:15 2023","interfaces":[{"interface":"back","average":{"1min":0.88700000000000001,"5min":0.92800000000000005,"15min":0.94999999999999996},"min":{"1min":0.55800000000000005,"5min":0.495,"15min":0.495},"max":{"1min":1.1459999999999999,"5min":2.1019999999999999,"15min":2.1019999999999999},"last":0.879},{"interface":"front","average":{"1min":0.93500000000000005,"5min":0.97999999999999998,"15min":1.002},"min":{"1min":0.59299999999999997,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":1.1850000000000001,"5min":1.4399999999999999,"15min":1.6000000000000001},"last":0.67000000000000004}]},{"osd":3,"last update":"Mon May 22 15:44:15 2023","interfaces":[{"interface":"back","average":{"1min":0.93200000000000005,"5min":0.90400000000000003,"15min":0.94499999999999995},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":1.194,"5min":1.29,"15min":1.665},"last":0.92300000000000004},{"interface":"front","average":{"1min":0.93600000000000005,"5min":1.0189999999999999,"15min":1.028},"min":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":1.21,"5min":2.1720000000000002,"15min":2.1720000000000002},"last":1.048}]},{"osd":5,"last update":"Mon May 22 15:44:15 2023","interfaces":[{"interface":"back","average":{"1min":0.90000000000000002,"5min":0.99399999999999999,"15min":0.98099999999999998},"min":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.1830000000000001,"5min":1.472,"15min":1.6830000000000001},"last":1.1439999999999999},{"interface":"front","average":{"1min":1.0069999999999999,"5min":0.97899999999999998,"15min":0.97799999999999998},"min":{"1min":0.68100000000000005,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.365,"5min":1.401,"15min":1.6719999999999999},"last":1.103}]},{"osd":6,"last update":"Mon May 22 15:44:15 2023","interfaces":[{"interface":"back","average":{"1min":0.91700000000000004,"5min":0.995,"15min":0.97999999999999998},"min":{"1min":0.55700000000000005,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":1.3200000000000001,"5min":2.2349999999999999,"15min":2.2349999999999999},"last":1.3420000000000001},{"interface":"front","average":{"1min":1.0580000000000001,"5min":1.1160000000000001,"15min":1.1040000000000001},"min":{"1min":0.73999999999999999,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":1.284,"5min":1.8220000000000001,"15min":1.8220000000000001},"last":1.2769999999999999}]},{"osd":7,"last update":"Mon May 22 15:44:15 2023","interfaces":[{"interface":"back","average":{"1min":1.0349999999999999,"5min":1.052,"15min":1.054},"min":{"1min":0.76000000000000001,"5min":0.70799999999999996,"15min":0.67700000000000005},"max":{"1min":1.46,"5min":1.5800000000000001,"15min":1.5800000000000001},"last":1.2989999999999999},{"interface":"front","average":{"1min":1.1100000000000001,"5min":1.1539999999999999,"15min":1.139},"min":{"1min":0.78300000000000003,"5min":0.66500000000000004,"15min":0.66500000000000004},"max":{"1min":1.6559999999999999,"5min":2.306,"15min":2.306},"last":1.204}]}]},{"osd":3,"up_from":13,"seq":55834575068,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36168,"kb_used_data":1600,"kb_used_omap":0,"kb_used_meta":34560,"kb_avail":94335672,"statfs":{"total":96636764160,"available":96599728128,"internally_reserved":0,"allocated":1638400,"data_stored":856257,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":35389440},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 15:45:03 2023","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.60199999999999998,"15min":0.63200000000000001},"min":{"1min":0.252,"5min":0.252,"15min":0.19},"max":{"1min":1.1299999999999999,"5min":1.194,"15min":1.194},"last":0.97999999999999998},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.66800000000000004,"15min":0.70799999999999996},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":1.296,"5min":1.296,"15min":1.296},"last":0.66200000000000003}]},{"osd":1,"last update":"Mon May 22 15:45:03 2023","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.623,"15min":0.66300000000000003},"min":{"1min":0.36799999999999999,"5min":0.314,"15min":0.29499999999999998},"max":{"1min":1.2230000000000001,"5min":1.2609999999999999,"15min":1.2609999999999999},"last":0.77300000000000002},{"interface":"front","average":{"1min":0.746,"5min":0.72999999999999998,"15min":0.76900000000000002},"min":{"1min":0.35199999999999998,"5min":0.34399999999999997,"15min":0.313},"max":{"1min":1.091,"5min":1.222,"15min":1.222},"last":0.84999999999999998}]},{"osd":2,"last update":"Mon May 22 15:45:03 2023","interfaces":[{"interface":"back","average":{"1min":0.78700000000000003,"5min":0.76200000000000001,"15min":0.81899999999999995},"min":{"1min":0.42399999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":1.359,"5min":1.506,"15min":1.506},"last":0.89900000000000002},{"interface":"front","average":{"1min":0.79200000000000004,"5min":0.745,"15min":0.80600000000000005},"min":{"1min":0.32000000000000001,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":1.0960000000000001,"5min":1.181,"15min":1.468},"last":0.71099999999999997}]},{"osd":4,"last update":"Mon May 22 15:45:03 2023","interfaces":[{"interface":"back","average":{"1min":0.94399999999999995,"5min":0.91800000000000004,"15min":0.95599999999999996},"min":{"1min":0.67700000000000005,"5min":0.53700000000000003,"15min":0.48999999999999999},"max":{"1min":1.2290000000000001,"5min":1.413,"15min":1.4470000000000001},"last":1.012},{"interface":"front","average":{"1min":0.98199999999999998,"5min":0.95099999999999996,"15min":1.004},"min":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.49199999999999999},"max":{"1min":1.3540000000000001,"5min":1.3540000000000001,"15min":1.4470000000000001},"last":0.95099999999999996}]},{"osd":5,"last update":"Mon May 22 15:45:03 2023","interfaces":[{"interface":"back","average":{"1min":1.036,"5min":0.97599999999999998,"15min":1.0169999999999999},"min":{"1min":0.64100000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.5089999999999999,"5min":1.5089999999999999,"15min":1.5089999999999999},"last":1.0429999999999999},{"interface":"front","average":{"1min":1.05,"5min":0.995,"15min":1.0429999999999999},"min":{"1min":0.61199999999999999,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.3069999999999999,"5min":1.4299999999999999,"15min":1.643},"last":1.1499999999999999}]},{"osd":6,"last update":"Mon May 22 15:45:03 2023","interfaces":[{"interface":"back","average":{"1min":1.0640000000000001,"5min":1.006,"15min":1.044},"min":{"1min":0.72599999999999998,"5min":0.47699999999999998,"15min":0.46999999999999997},"max":{"1min":1.4570000000000001,"5min":1.4570000000000001,"15min":1.49},"last":1.1870000000000001},{"interface":"front","average":{"1min":1.05,"5min":1.0149999999999999,"15min":1.0720000000000001},"min":{"1min":0.55200000000000005,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":1.409,"5min":1.4330000000000001,"15min":1.704},"last":1.25}]},{"osd":7,"last update":"Mon May 22 15:45:03 2023","interfaces":[{"interface":"back","average":{"1min":1.107,"5min":1.0649999999999999,"15min":1.1060000000000001},"min":{"1min":0.76500000000000001,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.4870000000000001,"5min":1.6719999999999999,"15min":1.6719999999999999},"last":1.212},{"interface":"front","average":{"1min":1.1000000000000001,"5min":1.05,"15min":1.0860000000000001},"min":{"1min":0.81499999999999995,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":1.5720000000000001,"5min":1.8700000000000001,"15min":1.8700000000000001},"last":1.105}]}]},{"osd":2,"up_from":13,"seq":55834575069,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":18424,"kb_used_data":1584,"kb_used_omap":0,"kb_used_meta":16832,"kb_avail":94353416,"statfs":{"total":96636764160,"available":96617897984,"internally_reserved":0,"allocated":1622016,"data_stored":854639,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":17235968},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 15:45:00 2023","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.63800000000000001,"15min":0.65500000000000003},"min":{"1min":0.28899999999999998,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.94299999999999995,"5min":1.1539999999999999,"15min":1.8240000000000001},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.71399999999999997,"15min":0.748},"min":{"1min":0.46999999999999997,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.92300000000000004,"5min":1.0740000000000001,"15min":1.8160000000000001},"last":0.54800000000000004}]},{"osd":1,"last update":"Mon May 22 15:45:00 2023","interfaces":[{"interface":"back","average":{"1min":0.72899999999999998,"5min":0.72799999999999998,"15min":0.745},"min":{"1min":0.40300000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":1.1359999999999999,"5min":1.2,"15min":1.2},"last":1.1359999999999999},{"interface":"front","average":{"1min":0.71999999999999997,"5min":0.76300000000000001,"15min":0.81200000000000006},"min":{"1min":0.374,"5min":0.317,"15min":0.317},"max":{"1min":1.052,"5min":1.246,"15min":2.5089999999999999},"last":0.72399999999999998}]},{"osd":3,"last update":"Mon May 22 15:45:00 2023","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.69199999999999995,"15min":0.72699999999999998},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.96999999999999997,"5min":1.077,"15min":2.7629999999999999},"last":0.69699999999999995},{"interface":"front","average":{"1min":0.82799999999999996,"5min":0.85299999999999998,"15min":0.88800000000000001},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":1.2410000000000001,"5min":1.304,"15min":2.6840000000000002},"last":0.90900000000000003}]},{"osd":4,"last update":"Mon May 22 15:45:00 2023","interfaces":[{"interface":"back","average":{"1min":0.99099999999999999,"5min":0.995,"15min":1.02},"min":{"1min":0.73899999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":1.3799999999999999,"5min":1.536,"15min":2.8039999999999998},"last":1.0900000000000001},{"interface":"front","average":{"1min":0.98799999999999999,"5min":1.004,"15min":1.0329999999999999},"min":{"1min":0.62,"5min":0.58699999999999997,"15min":0.56000000000000005},"max":{"1min":1.4299999999999999,"5min":1.4299999999999999,"15min":2.0819999999999999},"last":0.76200000000000001}]},{"osd":5,"last update":"Mon May 22 15:45:00 2023","interfaces":[{"interface":"back","average":{"1min":1.0369999999999999,"5min":1.0429999999999999,"15min":1.0489999999999999},"min":{"1min":0.65000000000000002,"5min":0.622,"15min":0.622},"max":{"1min":1.5049999999999999,"5min":1.5049999999999999,"15min":2.3860000000000001},"last":1.0009999999999999},{"interface":"front","average":{"1min":0.94299999999999995,"5min":0.97199999999999998,"15min":0.999},"min":{"1min":0.60999999999999999,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":1.2989999999999999,"5min":1.327,"15min":3.0299999999999998},"last":0.89500000000000002}]},{"osd":6,"last update":"Mon May 22 15:45:00 2023","interfaces":[{"interface":"back","average":{"1min":0.97699999999999998,"5min":0.996,"15min":1.04},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.2529999999999999,"5min":1.49,"15min":2.883},"last":0.81599999999999995},{"interface":"front","average":{"1min":1.069,"5min":1.069,"15min":1.103},"min":{"1min":0.69299999999999995,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.5660000000000001,"5min":1.5660000000000001,"15min":3.1459999999999999},"last":1.0289999999999999}]},{"osd":7,"last update":"Mon May 22 15:45:00 2023","interfaces":[{"interface":"back","average":{"1min":1.022,"5min":1.0800000000000001,"15min":1.127},"min":{"1min":0.499,"5min":0.499,"15min":0.499},"max":{"1min":1.55,"5min":1.55,"15min":2.5990000000000002},"last":0.94099999999999995},{"interface":"front","average":{"1min":1.026,"5min":1.046,"15min":1.1060000000000001},"min":{"1min":0.60199999999999998,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":1.3540000000000001,"5min":1.3839999999999999,"15min":3.8799999999999999},"last":0.97199999999999998}]}]},{"osd":1,"up_from":13,"seq":55834575069,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":25196,"kb_used_data":1572,"kb_used_omap":0,"kb_used_meta":23616,"kb_avail":94346644,"statfs":{"total":96636764160,"available":96610963456,"internally_reserved":0,"allocated":1609728,"data_stored":847191,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":24182784},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 15:44:14 2023","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.61399999999999999,"15min":0.63600000000000001},"min":{"1min":0.34799999999999998,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.97799999999999998,"5min":1.2270000000000001,"15min":1.2270000000000001},"last":0.374},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.72899999999999998,"15min":0.77200000000000002},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":1.151,"5min":1.2290000000000001,"15min":1.2529999999999999},"last":0.61799999999999999}]},{"osd":2,"last update":"Mon May 22 15:44:14 2023","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.73699999999999999,"15min":0.74199999999999999},"min":{"1min":0.38,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.92900000000000005,"5min":1.391,"15min":1.391},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.755,"5min":0.748,"15min":0.77100000000000002},"min":{"1min":0.36399999999999999,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":1.01,"5min":1.276,"15min":1.276},"last":0.57799999999999996}]},{"osd":3,"last update":"Mon May 22 15:44:14 2023","interfaces":[{"interface":"back","average":{"1min":0.873,"5min":0.80300000000000005,"15min":0.84299999999999997},"min":{"1min":0.32900000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.6220000000000001,"5min":1.657,"15min":1.657},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.81599999999999995,"5min":0.79600000000000004,"15min":0.82199999999999995},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":1.165,"5min":2.0569999999999999,"15min":2.0569999999999999},"last":0.53600000000000003}]},{"osd":4,"last update":"Mon May 22 15:44:14 2023","interfaces":[{"interface":"back","average":{"1min":1.0029999999999999,"5min":0.96899999999999997,"15min":0.96599999999999997},"min":{"1min":0.70299999999999996,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":1.2989999999999999,"5min":2.9529999999999998,"15min":2.9529999999999998},"last":0.82499999999999996},{"interface":"front","average":{"1min":1.0349999999999999,"5min":0.997,"15min":1.0089999999999999},"min":{"1min":0.58599999999999997,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":1.429,"5min":2.7759999999999998,"15min":2.7759999999999998},"last":0.874}]},{"osd":5,"last update":"Mon May 22 15:44:14 2023","interfaces":[{"interface":"back","average":{"1min":1.0309999999999999,"5min":1.0089999999999999,"15min":1.0409999999999999},"min":{"1min":0.60599999999999998,"5min":0.443,"15min":0.443},"max":{"1min":1.5649999999999999,"5min":1.9570000000000001,"15min":1.9570000000000001},"last":0.90500000000000003},{"interface":"front","average":{"1min":1.095,"5min":1.0249999999999999,"15min":1.0449999999999999},"min":{"1min":0.76100000000000001,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":1.486,"5min":2.1200000000000001,"15min":2.1200000000000001},"last":0.74299999999999999}]},{"osd":6,"last update":"Mon May 22 15:44:14 2023","interfaces":[{"interface":"back","average":{"1min":1.0549999999999999,"5min":1.0209999999999999,"15min":1.04},"min":{"1min":0.746,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.3260000000000001,"5min":2.831,"15min":2.831},"last":0.96699999999999997},{"interface":"front","average":{"1min":1.1299999999999999,"5min":1.0720000000000001,"15min":1.099},"min":{"1min":0.63500000000000001,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":1.6779999999999999,"5min":1.7629999999999999,"15min":1.778},"last":0.79700000000000004}]},{"osd":7,"last update":"Mon May 22 15:44:14 2023","interfaces":[{"interface":"back","average":{"1min":1.1679999999999999,"5min":1.1080000000000001,"15min":1.1200000000000001},"min":{"1min":0.78600000000000003,"5min":0.56799999999999995,"15min":0.56799999999999995},"max":{"1min":1.446,"5min":2.4199999999999999,"15min":2.4199999999999999},"last":0.94699999999999995},{"interface":"front","average":{"1min":1.163,"5min":1.1259999999999999,"15min":1.1299999999999999},"min":{"1min":0.59899999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":1.466,"5min":3.8799999999999999,"15min":3.8799999999999999},"last":0.68100000000000005}]}]},{"osd":0,"up_from":13,"seq":55834575070,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":24848,"kb_used_data":2184,"kb_used_omap":0,"kb_used_meta":22656,"kb_avail":94346992,"statfs":{"total":96636764160,"available":96611319808,"internally_reserved":0,"allocated":2236416,"data_stored":1446667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":23199744},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon May 22 15:44:13 2023","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.68700000000000006},"min":{"1min":0.45600000000000002,"5min":0.223,"15min":0.223},"max":{"1min":1.071,"5min":1.466,"15min":1.466},"last":0.32900000000000001},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.69399999999999995,"15min":0.74099999999999999},"min":{"1min":0.39000000000000001,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.81499999999999995,"5min":1.71,"15min":1.71},"last":0.35199999999999998}]},{"osd":2,"last update":"Mon May 22 15:44:13 2023","interfaces":[{"interface":"back","average":{"1min":0.71799999999999997,"5min":0.71999999999999997,"15min":0.754},"min":{"1min":0.40699999999999997,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.97899999999999998,"5min":1.165,"15min":1.4059999999999999},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.751,"5min":0.745,"15min":0.80500000000000005},"min":{"1min":0.34999999999999998,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":1.123,"5min":1.147,"15min":1.3999999999999999},"last":0.40999999999999998}]},{"osd":3,"last update":"Mon May 22 15:44:13 2023","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.75700000000000001,"15min":0.77300000000000002},"min":{"1min":0.314,"5min":0.31,"15min":0.31},"max":{"1min":0.95999999999999996,"5min":1.9299999999999999,"15min":1.9299999999999999},"last":0.38800000000000001},{"interface":"front","average":{"1min":0.81499999999999995,"5min":0.83399999999999996,"15min":0.88700000000000001},"min":{"1min":0.36499999999999999,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":1.2050000000000001,"5min":1.758,"15min":1.758},"last":0.375}]},{"osd":4,"last update":"Mon May 22 15:44:13 2023","interfaces":[{"interface":"back","average":{"1min":0.91900000000000004,"5min":0.93700000000000006,"15min":0.96099999999999997},"min":{"1min":0.68100000000000005,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":1.1739999999999999,"5min":2.923,"15min":2.923},"last":0.83099999999999996},{"interface":"front","average":{"1min":0.95699999999999996,"5min":1.012,"15min":1.0329999999999999},"min":{"1min":0.64700000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"max":{"1min":1.268,"5min":2.0129999999999999,"15min":2.0129999999999999},"last":0.66300000000000003}]},{"osd":5,"last update":"Mon May 22 15:44:13 2023","interfaces":[{"interface":"back","average":{"1min":1.002,"5min":0.99399999999999999,"15min":1.0489999999999999},"min":{"1min":0.63500000000000001,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.4179999999999999,"5min":1.97,"15min":1.97},"last":0.85699999999999998},{"interface":"front","average":{"1min":0.96399999999999997,"5min":1.0129999999999999,"15min":1.0169999999999999},"min":{"1min":0.67900000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.5129999999999999,"5min":5.0519999999999996,"15min":5.0519999999999996},"last":0.72899999999999998}]},{"osd":6,"last update":"Mon May 22 15:44:13 2023","interfaces":[{"interface":"back","average":{"1min":1.0129999999999999,"5min":1.0629999999999999,"15min":1.095},"min":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":1.46,"5min":2.117,"15min":2.117},"last":0.77100000000000002},{"interface":"front","average":{"1min":1.054,"5min":1.123,"15min":1.121},"min":{"1min":0.70599999999999996,"5min":0.499,"15min":0.499},"max":{"1min":1.5609999999999999,"5min":5.2640000000000002,"15min":5.2640000000000002},"last":0.69499999999999995}]},{"osd":7,"last update":"Mon May 22 15:44:13 2023","interfaces":[{"interface":"back","average":{"1min":1.091,"5min":1.105,"15min":1.135},"min":{"1min":0.65900000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"max":{"1min":1.6599999999999999,"5min":1.839,"15min":1.839},"last":0.63600000000000001},{"interface":"front","average":{"1min":1.0229999999999999,"5min":1.0820000000000001,"15min":1.101},"min":{"1min":0.79700000000000004,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":1.591,"5min":3.9060000000000001,"15min":3.9060000000000001},"last":0.79600000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-05-22T15:45:09.216 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-05-22T15:45:09.217 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2023-05-22T15:45:09.394 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.389+0000 7f4d0998f640 1 Processor -- start 2023-05-22T15:45:09.394 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.389+0000 7f4d0998f640 1 -- start start 2023-05-22T15:45:09.394 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.389+0000 7f4d0998f640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f4d0415c8b0 0x7f4d040cee30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:09.394 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.389+0000 7f4d0998f640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d0406c3e0 con 0x7f4d0406bf60 2023-05-22T15:45:09.395 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.389+0000 7f4d0998f640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d0406c640 con 0x7f4d0406db80 2023-05-22T15:45:09.395 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.389+0000 7f4d0998f640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d0415a0c0 con 0x7f4d0415c8b0 2023-05-22T15:45:09.395 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.389+0000 7f4d02ffd640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f4d0415c8b0 0x7f4d040cee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:09.396 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.389+0000 7f4d027fc640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f4d0406bf60 0x7f4d0415cc80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:42204/0 (socket says 172.21.15.16:42204) 2023-05-22T15:45:09.396 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.389+0000 7f4d027fc640 1 -- 172.21.15.16:0/3854461855 learned_addr learned my addr 172.21.15.16:0/3854461855 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:45:09.396 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.393+0000 7f4d02ffd640 1 -- 172.21.15.16:0/3854461855 >> v1:172.21.15.62:6789/0 conn(0x7f4d0406db80 legacy=0x7f4d040cf8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:45:09.397 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.393+0000 7f4d02ffd640 1 -- 172.21.15.16:0/3854461855 >> v1:172.21.15.16:6789/0 conn(0x7f4d0406bf60 legacy=0x7f4d0415cc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:45:09.397 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.393+0000 7f4d01ffb640 1 -- 172.21.15.16:0/3854461855 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3529984190 0 0) 0x7f4d0406c3e0 con 0x7f4d0406bf60 2023-05-22T15:45:09.397 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.393+0000 7f4d02ffd640 1 -- 172.21.15.16:0/3854461855 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d040ce370 con 0x7f4d0415c8b0 2023-05-22T15:45:09.398 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.393+0000 7f4d02ffd640 1 --2- 172.21.15.16:0/3854461855 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4d0415c8b0 0x7f4d040cee30 secure :-1 s=READY pgs=1653 cs=0 l=1 rev1=1 crypto rx=0x7f4cf40097f0 tx=0x7f4d040d0810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14e90eb54de57811 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:45:09.398 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.393+0000 7f4d01ffb640 1 -- 172.21.15.16:0/3854461855 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cf40ab600 con 0x7f4d0415c8b0 2023-05-22T15:45:09.398 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.393+0000 7f4d01ffb640 1 -- 172.21.15.16:0/3854461855 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4cf40a7050 con 0x7f4d0415c8b0 2023-05-22T15:45:09.398 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.393+0000 7f4d01ffb640 1 -- 172.21.15.16:0/3854461855 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cf40ab980 con 0x7f4d0415c8b0 2023-05-22T15:45:09.399 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.397+0000 7f4d0998f640 1 -- 172.21.15.16:0/3854461855 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4d0415c8b0 msgr2=0x7f4d040cee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:45:09.399 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.397+0000 7f4d0998f640 1 --2- 172.21.15.16:0/3854461855 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4d0415c8b0 0x7f4d040cee30 secure :-1 s=READY pgs=1653 cs=0 l=1 rev1=1 crypto rx=0x7f4cf40097f0 tx=0x7f4d040d0810 comp rx=0 tx=0).stop 2023-05-22T15:45:09.399 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.397+0000 7f4d0998f640 1 -- 172.21.15.16:0/3854461855 shutdown_connections 2023-05-22T15:45:09.399 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.397+0000 7f4d0998f640 1 --2- 172.21.15.16:0/3854461855 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4d0415c8b0 0x7f4d040cee30 unknown :-1 s=CLOSED pgs=1653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:09.400 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.397+0000 7f4d0998f640 1 -- 172.21.15.16:0/3854461855 >> 172.21.15.16:0/3854461855 conn(0x7f4d0406f5a0 msgr2=0x7f4d0406f9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:45:09.400 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.397+0000 7f4d0998f640 1 -- 172.21.15.16:0/3854461855 shutdown_connections 2023-05-22T15:45:09.400 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.397+0000 7f4d0998f640 1 -- 172.21.15.16:0/3854461855 wait complete. 2023-05-22T15:45:09.401 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.397+0000 7f4d0998f640 1 Processor -- start 2023-05-22T15:45:09.401 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.397+0000 7f4d0998f640 1 -- start start 2023-05-22T15:45:09.401 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.397+0000 7f4d0998f640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4d0406bf60 0x7f4d0414e0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:09.401 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.397+0000 7f4d0998f640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4d0406db80 0x7f4d040b32a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:09.401 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.397+0000 7f4d0998f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4d0415c8b0 0x7f4d0414d6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:09.401 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.397+0000 7f4d0998f640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d0414cc30 con 0x7f4d0415c8b0 2023-05-22T15:45:09.401 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.397+0000 7f4d0998f640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d040b5ca0 con 0x7f4d0406bf60 2023-05-22T15:45:09.402 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.397+0000 7f4d0998f640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d0414ced0 con 0x7f4d0406db80 2023-05-22T15:45:09.403 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.397+0000 7f4d037fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4d0415c8b0 0x7f4d0414d6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:09.403 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.397+0000 7f4d037fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4d0415c8b0 0x7f4d0414d6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36684/0 (socket says 172.21.15.16:36684) 2023-05-22T15:45:09.403 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.397+0000 7f4d037fe640 1 -- 172.21.15.16:0/2796256742 learned_addr learned my addr 172.21.15.16:0/2796256742 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:45:09.403 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.397+0000 7f4d02ffd640 1 --2- 172.21.15.16:0/2796256742 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4d0406bf60 0x7f4d0414e0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:09.404 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.397+0000 7f4d037fe640 1 -- 172.21.15.16:0/2796256742 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4d0406db80 msgr2=0x7f4d040b32a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:45:09.404 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.397+0000 7f4d027fc640 1 --2- 172.21.15.16:0/2796256742 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4d0406db80 0x7f4d040b32a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:09.404 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.397+0000 7f4d037fe640 1 --2- 172.21.15.16:0/2796256742 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4d0406db80 0x7f4d040b32a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:09.404 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.397+0000 7f4d037fe640 1 -- 172.21.15.16:0/2796256742 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4d0406bf60 msgr2=0x7f4d0414e0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:45:09.404 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.397+0000 7f4d037fe640 1 --2- 172.21.15.16:0/2796256742 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4d0406bf60 0x7f4d0414e0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:09.404 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.397+0000 7f4d037fe640 1 -- 172.21.15.16:0/2796256742 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cf4009420 con 0x7f4d0415c8b0 2023-05-22T15:45:09.405 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.397+0000 7f4d02ffd640 1 --2- 172.21.15.16:0/2796256742 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4d0406bf60 0x7f4d0414e0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T15:45:09.405 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.397+0000 7f4d027fc640 1 --2- 172.21.15.16:0/2796256742 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4d0406db80 0x7f4d040b32a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:45:09.405 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.397+0000 7f4d037fe640 1 --2- 172.21.15.16:0/2796256742 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4d0415c8b0 0x7f4d0414d6d0 secure :-1 s=READY pgs=1654 cs=0 l=1 rev1=1 crypto rx=0x7f4cfc00c770 tx=0x7f4cfc00cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:45:09.405 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.397+0000 7f4cebfff640 1 -- 172.21.15.16:0/2796256742 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cfc003b80 con 0x7f4d0415c8b0 2023-05-22T15:45:09.405 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.397+0000 7f4cebfff640 1 -- 172.21.15.16:0/2796256742 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4cfc003ce0 con 0x7f4d0415c8b0 2023-05-22T15:45:09.405 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.397+0000 7f4d0998f640 1 -- 172.21.15.16:0/2796256742 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d0406e520 con 0x7f4d0415c8b0 2023-05-22T15:45:09.405 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.397+0000 7f4cebfff640 1 -- 172.21.15.16:0/2796256742 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cfc00aa90 con 0x7f4d0415c8b0 2023-05-22T15:45:09.405 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.397+0000 7f4d0998f640 1 -- 172.21.15.16:0/2796256742 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d0406e390 con 0x7f4d0415c8b0 2023-05-22T15:45:09.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.401+0000 7f4d0998f640 1 -- 172.21.15.16:0/2796256742 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d0415d390 con 0x7f4d0415c8b0 2023-05-22T15:45:09.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.401+0000 7f4cebfff640 1 -- 172.21.15.16:0/2796256742 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 50) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f4cfc00abf0 con 0x7f4d0415c8b0 2023-05-22T15:45:09.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.401+0000 7f4cebfff640 1 --2- 172.21.15.16:0/2796256742 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f4cf00654a0 0x7f4cf0067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:09.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.401+0000 7f4cebfff640 1 -- 172.21.15.16:0/2796256742 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f4cfc082a40 con 0x7f4d0415c8b0 2023-05-22T15:45:09.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.401+0000 7f4d02ffd640 1 --2- 172.21.15.16:0/2796256742 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f4cf00654a0 0x7f4cf0067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:09.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.401+0000 7f4d02ffd640 1 --2- 172.21.15.16:0/2796256742 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f4cf00654a0 0x7f4cf0067950 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f4cf40ad000 tx=0x7f4cf4004740 comp rx=0 tx=0).ready entity=mgr.8304 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:45:09.408 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.405+0000 7f4cebfff640 1 -- 172.21.15.16:0/2796256742 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f4cfc0503d0 con 0x7f4d0415c8b0 2023-05-22T15:45:09.615 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.613+0000 7f4d0998f640 1 -- 172.21.15.16:0/2796256742 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f4d0414d3b0 con 0x7f4d0415c8b0 2023-05-22T15:45:09.626 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.621+0000 7f4cebfff640 1 -- 172.21.15.16:0/2796256742 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v23) v1 ==== 190+0+0 (secure 0 0 0) 0x7f4cfc0539f0 con 0x7f4d0415c8b0 2023-05-22T15:45:09.626 INFO:teuthology.orchestra.run.smithi016.stderr:updated caps for client.0 2023-05-22T15:45:09.629 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.625+0000 7f4d0998f640 1 -- 172.21.15.16:0/2796256742 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f4cf00654a0 msgr2=0x7f4cf0067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:45:09.629 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.625+0000 7f4d0998f640 1 --2- 172.21.15.16:0/2796256742 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f4cf00654a0 0x7f4cf0067950 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f4cf40ad000 tx=0x7f4cf4004740 comp rx=0 tx=0).stop 2023-05-22T15:45:09.629 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.625+0000 7f4d0998f640 1 -- 172.21.15.16:0/2796256742 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4d0415c8b0 msgr2=0x7f4d0414d6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:45:09.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.625+0000 7f4d0998f640 1 --2- 172.21.15.16:0/2796256742 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4d0415c8b0 0x7f4d0414d6d0 secure :-1 s=READY pgs=1654 cs=0 l=1 rev1=1 crypto rx=0x7f4cfc00c770 tx=0x7f4cfc00cc30 comp rx=0 tx=0).stop 2023-05-22T15:45:09.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.625+0000 7f4d0998f640 1 -- 172.21.15.16:0/2796256742 shutdown_connections 2023-05-22T15:45:09.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.625+0000 7f4d0998f640 1 --2- 172.21.15.16:0/2796256742 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4d0415c8b0 0x7f4d0414d6d0 unknown :-1 s=CLOSED pgs=1654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:09.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.625+0000 7f4d0998f640 1 --2- 172.21.15.16:0/2796256742 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f4cf00654a0 0x7f4cf0067950 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:09.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.625+0000 7f4d0998f640 1 --2- 172.21.15.16:0/2796256742 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4d0406db80 0x7f4d040b32a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:09.632 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.625+0000 7f4d0998f640 1 --2- 172.21.15.16:0/2796256742 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4d0406bf60 0x7f4d0414e0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:09.632 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.625+0000 7f4d0998f640 1 -- 172.21.15.16:0/2796256742 >> 172.21.15.16:0/2796256742 conn(0x7f4d0406f5a0 msgr2=0x7f4d040cb960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:45:09.633 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.625+0000 7f4d0998f640 1 -- 172.21.15.16:0/2796256742 shutdown_connections 2023-05-22T15:45:09.633 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.625+0000 7f4d0998f640 1 -- 172.21.15.16:0/2796256742 wait complete. 2023-05-22T15:45:09.643 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-05-22T15:45:09.763 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.761+0000 7f0495755640 1 Processor -- start 2023-05-22T15:45:09.763 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.761+0000 7f0495755640 1 -- start start 2023-05-22T15:45:09.765 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.761+0000 7f0495755640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f04900ce480 0x7f0490071640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:09.765 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.761+0000 7f0495755640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0490069ae0 con 0x7f04900ce850 2023-05-22T15:45:09.765 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.761+0000 7f0495755640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0490069dc0 con 0x7f049015c270 2023-05-22T15:45:09.765 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.761+0000 7f0495755640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f049015f990 con 0x7f04900ce480 2023-05-22T15:45:09.766 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.761+0000 7f04867fc640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f04900ce850 0x7f0490160090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:42212/0 (socket says 172.21.15.16:42212) 2023-05-22T15:45:09.766 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.761+0000 7f04867fc640 1 -- 172.21.15.16:0/3012540589 learned_addr learned my addr 172.21.15.16:0/3012540589 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:45:09.766 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.761+0000 7f048effd640 1 --2- 172.21.15.16:0/3012540589 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f04900ce480 0x7f0490071640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:09.766 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.761+0000 7f048e7fc640 1 -- 172.21.15.16:0/3012540589 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2211146510 0 0) 0x7f0490069ae0 con 0x7f04900ce850 2023-05-22T15:45:09.766 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.761+0000 7f048e7fc640 1 -- 172.21.15.16:0/3012540589 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f047c003530 con 0x7f04900ce850 2023-05-22T15:45:09.766 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.761+0000 7f048e7fc640 1 -- 172.21.15.16:0/3012540589 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 813145766 0 0) 0x7f047c003530 con 0x7f04900ce850 2023-05-22T15:45:09.766 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.761+0000 7f048e7fc640 1 -- 172.21.15.16:0/3012540589 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f04900ce480 msgr2=0x7f0490071640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:45:09.767 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.761+0000 7f048e7fc640 1 --2- 172.21.15.16:0/3012540589 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f04900ce480 0x7f0490071640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:09.767 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.761+0000 7f048e7fc640 1 -- 172.21.15.16:0/3012540589 >> v1:172.21.15.62:6789/0 conn(0x7f049015c270 legacy=0x7f049015c6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:45:09.767 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.761+0000 7f048e7fc640 1 -- 172.21.15.16:0/3012540589 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f049006a0a0 con 0x7f04900ce850 2023-05-22T15:45:09.767 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.761+0000 7f048e7fc640 1 -- 172.21.15.16:0/3012540589 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7f0478001680 con 0x7f04900ce850 2023-05-22T15:45:09.767 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.761+0000 7f048e7fc640 1 -- 172.21.15.16:0/3012540589 >> v1:172.21.15.16:6789/0 conn(0x7f04900ce850 legacy=0x7f0490160090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:45:09.767 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.761+0000 7f048e7fc640 1 --2- 172.21.15.16:0/3012540589 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f047c0039b0 0x7f047c003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:09.767 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.761+0000 7f048e7fc640 1 --2- 172.21.15.16:0/3012540589 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f04900ce480 0x7f047c0047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:09.767 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.761+0000 7f048e7fc640 1 --2- 172.21.15.16:0/3012540589 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f047c004d10 0x7f047c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:09.768 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.761+0000 7f048e7fc640 1 -- 172.21.15.16:0/3012540589 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f049015f990 con 0x7f047c004d10 2023-05-22T15:45:09.768 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.761+0000 7f048e7fc640 1 -- 172.21.15.16:0/3012540589 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f047c00ba40 con 0x7f047c0039b0 2023-05-22T15:45:09.768 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.761+0000 7f048e7fc640 1 -- 172.21.15.16:0/3012540589 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f047c00bbd0 con 0x7f04900ce480 2023-05-22T15:45:09.768 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.761+0000 7f048f7fe640 1 --2- 172.21.15.16:0/3012540589 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f047c004d10 0x7f047c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:09.768 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.761+0000 7f04867fc640 1 --2- 172.21.15.16:0/3012540589 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f04900ce480 0x7f047c0047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:09.768 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.761+0000 7f048effd640 1 --2- 172.21.15.16:0/3012540589 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f047c0039b0 0x7f047c003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:09.768 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.761+0000 7f048f7fe640 1 -- 172.21.15.16:0/3012540589 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f04900ce480 msgr2=0x7f047c0047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:45:09.769 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.761+0000 7f048f7fe640 1 --2- 172.21.15.16:0/3012540589 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f04900ce480 0x7f047c0047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:09.769 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.761+0000 7f048f7fe640 1 -- 172.21.15.16:0/3012540589 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f047c0039b0 msgr2=0x7f047c003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:45:09.769 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.761+0000 7f048f7fe640 1 --2- 172.21.15.16:0/3012540589 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f047c0039b0 0x7f047c003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:09.769 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.761+0000 7f048f7fe640 1 -- 172.21.15.16:0/3012540589 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f047c003530 con 0x7f047c004d10 2023-05-22T15:45:09.771 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.765+0000 7f048f7fe640 1 --2- 172.21.15.16:0/3012540589 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f047c004d10 0x7f047c005180 secure :-1 s=READY pgs=1656 cs=0 l=1 rev1=1 crypto rx=0x7f0488001e70 tx=0x7f048801c760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5cb5c302cf564d90 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:45:09.771 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.765+0000 7f048e7fc640 1 -- 172.21.15.16:0/3012540589 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0488003b80 con 0x7f047c004d10 2023-05-22T15:45:09.771 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.765+0000 7f048e7fc640 1 -- 172.21.15.16:0/3012540589 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0488003ce0 con 0x7f047c004d10 2023-05-22T15:45:09.772 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.765+0000 7f048e7fc640 1 -- 172.21.15.16:0/3012540589 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04880052c0 con 0x7f047c004d10 2023-05-22T15:45:09.772 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.765+0000 7f0495755640 1 -- 172.21.15.16:0/3012540589 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f047c004d10 msgr2=0x7f047c005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:45:09.772 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.765+0000 7f0495755640 1 --2- 172.21.15.16:0/3012540589 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f047c004d10 0x7f047c005180 secure :-1 s=READY pgs=1656 cs=0 l=1 rev1=1 crypto rx=0x7f0488001e70 tx=0x7f048801c760 comp rx=0 tx=0).stop 2023-05-22T15:45:09.772 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.765+0000 7f048f7fe640 1 -- 172.21.15.16:0/3012540589 reap_dead start 2023-05-22T15:45:09.772 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.765+0000 7f0495755640 1 -- 172.21.15.16:0/3012540589 shutdown_connections 2023-05-22T15:45:09.772 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.765+0000 7f0495755640 1 -- 172.21.15.16:0/3012540589 >> 172.21.15.16:0/3012540589 conn(0x7f04900cc870 msgr2=0x7f04900cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:45:09.772 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.765+0000 7f0495755640 1 -- 172.21.15.16:0/3012540589 shutdown_connections 2023-05-22T15:45:09.773 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.765+0000 7f0495755640 1 -- 172.21.15.16:0/3012540589 wait complete. 2023-05-22T15:45:09.773 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.765+0000 7f0495755640 1 Processor -- start 2023-05-22T15:45:09.773 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.765+0000 7f0495755640 1 -- start start 2023-05-22T15:45:09.773 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.765+0000 7f0495755640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f047c004d10 0x7f04901440f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:09.773 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.765+0000 7f0495755640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f04901477c0 0x7f04900b1f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:09.773 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.765+0000 7f0495755640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0490146a80 0x7f0490146f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:09.773 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.765+0000 7f0495755640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04900ae710 con 0x7f04901477c0 2023-05-22T15:45:09.773 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.765+0000 7f0495755640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0490156070 con 0x7f0490146a80 2023-05-22T15:45:09.774 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.765+0000 7f0495755640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0490156bf0 con 0x7f047c004d10 2023-05-22T15:45:09.774 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.765+0000 7f04867fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f04901477c0 0x7f04900b1f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:09.774 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.765+0000 7f04867fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f04901477c0 0x7f04900b1f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36698/0 (socket says 172.21.15.16:36698) 2023-05-22T15:45:09.774 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.765+0000 7f04867fc640 1 -- 172.21.15.16:0/1278305512 learned_addr learned my addr 172.21.15.16:0/1278305512 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:45:09.774 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.765+0000 7f048effd640 1 --2- 172.21.15.16:0/1278305512 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f047c004d10 0x7f04901440f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:09.774 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.765+0000 7f048f7fe640 1 --2- 172.21.15.16:0/1278305512 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0490146a80 0x7f0490146f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:09.774 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.765+0000 7f04867fc640 1 -- 172.21.15.16:0/1278305512 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f047c004d10 msgr2=0x7f04901440f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:45:09.775 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.765+0000 7f04867fc640 1 --2- 172.21.15.16:0/1278305512 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f047c004d10 0x7f04901440f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:09.775 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.765+0000 7f04867fc640 1 -- 172.21.15.16:0/1278305512 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0490146a80 msgr2=0x7f0490146f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:45:09.775 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.765+0000 7f04867fc640 1 --2- 172.21.15.16:0/1278305512 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0490146a80 0x7f0490146f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:09.775 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.765+0000 7f04867fc640 1 -- 172.21.15.16:0/1278305512 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04901607a0 con 0x7f04901477c0 2023-05-22T15:45:09.775 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.769+0000 7f04867fc640 1 --2- 172.21.15.16:0/1278305512 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f04901477c0 0x7f04900b1f90 secure :-1 s=READY pgs=1657 cs=0 l=1 rev1=1 crypto rx=0x7f047800e7b0 tx=0x7f047800ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:45:09.776 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.769+0000 7f0487fff640 1 -- 172.21.15.16:0/1278305512 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f047800c8d0 con 0x7f04901477c0 2023-05-22T15:45:09.776 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.769+0000 7f0487fff640 1 -- 172.21.15.16:0/1278305512 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f047800ca50 con 0x7f04901477c0 2023-05-22T15:45:09.776 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.769+0000 7f0495755640 1 -- 172.21.15.16:0/1278305512 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f049015fb70 con 0x7f04901477c0 2023-05-22T15:45:09.776 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.769+0000 7f0487fff640 1 -- 172.21.15.16:0/1278305512 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f047800ce00 con 0x7f04901477c0 2023-05-22T15:45:09.776 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.769+0000 7f0495755640 1 -- 172.21.15.16:0/1278305512 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04900ceca0 con 0x7f04901477c0 2023-05-22T15:45:09.777 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.769+0000 7f0495755640 1 -- 172.21.15.16:0/1278305512 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f049004dbe0 con 0x7f04901477c0 2023-05-22T15:45:09.777 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.769+0000 7f0487fff640 1 -- 172.21.15.16:0/1278305512 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 50) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f0478013420 con 0x7f04901477c0 2023-05-22T15:45:09.777 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.769+0000 7f0487fff640 1 --2- 172.21.15.16:0/1278305512 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f04700654d0 0x7f0470067980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:09.778 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.769+0000 7f0487fff640 1 -- 172.21.15.16:0/1278305512 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f0478082800 con 0x7f04901477c0 2023-05-22T15:45:09.778 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.769+0000 7f048effd640 1 --2- 172.21.15.16:0/1278305512 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f04700654d0 0x7f0470067980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:09.778 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.769+0000 7f048effd640 1 --2- 172.21.15.16:0/1278305512 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f04700654d0 0x7f0470067980 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f0480009a30 tx=0x7f0480008040 comp rx=0 tx=0).ready entity=mgr.8304 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:45:09.778 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.773+0000 7f0487fff640 1 -- 172.21.15.16:0/1278305512 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f04780501d0 con 0x7f04901477c0 2023-05-22T15:45:09.962 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.957+0000 7f0495755640 1 -- 172.21.15.16:0/1278305512 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0490156d30 con 0x7f04901477c0 2023-05-22T15:45:09.963 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.961+0000 7f0487fff640 1 -- 172.21.15.16:0/1278305512 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 50 v50) v1 ==== 94+0+4644 (secure 0 0 0) 0x7f04780537f0 con 0x7f04901477c0 2023-05-22T15:45:09.963 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:45:09.963 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":50,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6689,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/296693182","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6833","nonce":296693182},{"type":"v1","addr":"172.21.15.62:6835","nonce":296693182}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":46},{"gid":7402,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6834/856426578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":856426578},{"type":"v1","addr":"172.21.15.62:6834","nonce":856426578}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":46},{"gid":8541,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/2260895833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2260895833},{"type":"v1","addr":"172.21.15.16:6835","nonce":2260895833}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":46}],"filesystems":[{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-05-22T15:45:05.200760+0000","modified":"2023-05-22T15:45:07.832845+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8547},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8547":{"gid":8547,"name":"a","rank":0,"incarnation":47,"state":"up:active","state_seq":3,"addr":"172.21.15.16:6837/1566663454","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":1566663454},{"type":"v1","addr":"172.21.15.16:6837","nonce":1566663454}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":7}]} 2023-05-22T15:45:09.964 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 50 2023-05-22T15:45:09.965 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.961+0000 7f0495755640 1 -- 172.21.15.16:0/1278305512 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f04700654d0 msgr2=0x7f0470067980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:45:09.965 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.961+0000 7f0495755640 1 --2- 172.21.15.16:0/1278305512 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f04700654d0 0x7f0470067980 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f0480009a30 tx=0x7f0480008040 comp rx=0 tx=0).stop 2023-05-22T15:45:09.965 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.961+0000 7f0495755640 1 -- 172.21.15.16:0/1278305512 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f04901477c0 msgr2=0x7f04900b1f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:45:09.965 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.961+0000 7f0495755640 1 --2- 172.21.15.16:0/1278305512 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f04901477c0 0x7f04900b1f90 secure :-1 s=READY pgs=1657 cs=0 l=1 rev1=1 crypto rx=0x7f047800e7b0 tx=0x7f047800ec70 comp rx=0 tx=0).stop 2023-05-22T15:45:09.965 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.961+0000 7f0495755640 1 -- 172.21.15.16:0/1278305512 shutdown_connections 2023-05-22T15:45:09.965 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.961+0000 7f0495755640 1 --2- 172.21.15.16:0/1278305512 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0490146a80 0x7f0490146f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:09.966 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.961+0000 7f0495755640 1 --2- 172.21.15.16:0/1278305512 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f04901477c0 0x7f04900b1f90 unknown :-1 s=CLOSED pgs=1657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:09.966 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.961+0000 7f0495755640 1 --2- 172.21.15.16:0/1278305512 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f04700654d0 0x7f0470067980 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:09.966 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.961+0000 7f0495755640 1 --2- 172.21.15.16:0/1278305512 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f047c004d10 0x7f04901440f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:09.966 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.961+0000 7f0495755640 1 -- 172.21.15.16:0/1278305512 >> 172.21.15.16:0/1278305512 conn(0x7f04900cc870 msgr2=0x7f049015d560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:45:09.966 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.961+0000 7f0495755640 1 -- 172.21.15.16:0/1278305512 shutdown_connections 2023-05-22T15:45:09.966 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:09.961+0000 7f0495755640 1 -- 172.21.15.16:0/1278305512 wait complete. 2023-05-22T15:45:10.383 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 50, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-05-22T15:45:05.200760+0000', 'modified': '2023-05-22T15:45:07.832845+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8547}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8547': {'gid': 8547, 'name': 'a', 'rank': 0, 'incarnation': 47, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.16:6837/1566663454', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6836', 'nonce': 1566663454}, {'type': 'v1', 'addr': '172.21.15.16:6837', 'nonce': 1566663454}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [17], 'metadata_pool': 16, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-05-22T15:45:10.384 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-05-22T15:45:10.384 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2023-05-22T15:45:10.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.517+0000 7fae9327c640 1 Processor -- start 2023-05-22T15:45:10.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.517+0000 7fae9327c640 1 -- start start 2023-05-22T15:45:10.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.517+0000 7fae9327c640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fae8c15c130 0x7fae8c0ceb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:10.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.517+0000 7fae9327c640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fae8c06e440 0x7fae8c0cb1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:10.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.517+0000 7fae9327c640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae8c06bfd0 con 0x7fae8c0b24a0 2023-05-22T15:45:10.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.517+0000 7fae9327c640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae8c0d1170 con 0x7fae8c06e440 2023-05-22T15:45:10.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.517+0000 7fae9327c640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae8c06c830 con 0x7fae8c15c130 2023-05-22T15:45:10.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.517+0000 7fae90817640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fae8c0b24a0 0x7fae8c06dd30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:42228/0 (socket says 172.21.15.16:42228) 2023-05-22T15:45:10.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.517+0000 7fae90817640 1 -- 172.21.15.16:0/3799961110 learned_addr learned my addr 172.21.15.16:0/3799961110 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:45:10.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.517+0000 7fae91819640 1 --2- 172.21.15.16:0/3799961110 >> v2:172.21.15.16:3300/0 conn(0x7fae8c06e440 0x7fae8c0cb1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:10.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.517+0000 7fae91018640 1 --2- 172.21.15.16:0/3799961110 >> v2:172.21.15.62:3300/0 conn(0x7fae8c15c130 0x7fae8c0ceb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:10.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.517+0000 7fae91819640 1 -- 172.21.15.16:0/3799961110 >> v2:172.21.15.62:3300/0 conn(0x7fae8c15c130 msgr2=0x7fae8c0ceb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:45:10.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.517+0000 7fae91819640 1 --2- 172.21.15.16:0/3799961110 >> v2:172.21.15.62:3300/0 conn(0x7fae8c15c130 0x7fae8c0ceb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:10.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.517+0000 7fae91819640 1 -- 172.21.15.16:0/3799961110 >> v1:172.21.15.16:6789/0 conn(0x7fae8c0b24a0 legacy=0x7fae8c06dd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:45:10.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.517+0000 7fae91819640 1 -- 172.21.15.16:0/3799961110 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae8c0cf510 con 0x7fae8c06e440 2023-05-22T15:45:10.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.517+0000 7fae8bfff640 1 -- 172.21.15.16:0/3799961110 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 600143037 0 0) 0x7fae8c06bfd0 con 0x7fae8c0b24a0 2023-05-22T15:45:10.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.517+0000 7fae91018640 1 --2- 172.21.15.16:0/3799961110 >> v2:172.21.15.62:3300/0 conn(0x7fae8c15c130 0x7fae8c0ceb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T15:45:10.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.521+0000 7fae91819640 1 --2- 172.21.15.16:0/3799961110 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fae8c06e440 0x7fae8c0cb1e0 secure :-1 s=READY pgs=1659 cs=0 l=1 rev1=1 crypto rx=0x7fae8c0cd610 tx=0x7fae7c01c3d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fac888259ba6a05d server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:45:10.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.521+0000 7fae8bfff640 1 -- 172.21.15.16:0/3799961110 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae7c0ab600 con 0x7fae8c06e440 2023-05-22T15:45:10.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.521+0000 7fae8bfff640 1 -- 172.21.15.16:0/3799961110 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae7c0a7050 con 0x7fae8c06e440 2023-05-22T15:45:10.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.521+0000 7fae8bfff640 1 -- 172.21.15.16:0/3799961110 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae7c0ab980 con 0x7fae8c06e440 2023-05-22T15:45:10.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.521+0000 7fae9327c640 1 -- 172.21.15.16:0/3799961110 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fae8c06e440 msgr2=0x7fae8c0cb1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:45:10.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.521+0000 7fae9327c640 1 --2- 172.21.15.16:0/3799961110 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fae8c06e440 0x7fae8c0cb1e0 secure :-1 s=READY pgs=1659 cs=0 l=1 rev1=1 crypto rx=0x7fae8c0cd610 tx=0x7fae7c01c3d0 comp rx=0 tx=0).stop 2023-05-22T15:45:10.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.521+0000 7fae9327c640 1 -- 172.21.15.16:0/3799961110 shutdown_connections 2023-05-22T15:45:10.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.521+0000 7fae9327c640 1 --2- 172.21.15.16:0/3799961110 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fae8c06e440 0x7fae8c0cb1e0 unknown :-1 s=CLOSED pgs=1659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:10.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.521+0000 7fae9327c640 1 --2- 172.21.15.16:0/3799961110 >> v2:172.21.15.62:3300/0 conn(0x7fae8c15c130 0x7fae8c0ceb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:10.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.521+0000 7fae9327c640 1 -- 172.21.15.16:0/3799961110 >> 172.21.15.16:0/3799961110 conn(0x7fae8c06f5c0 msgr2=0x7fae8c06f9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:45:10.530 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.521+0000 7fae9327c640 1 -- 172.21.15.16:0/3799961110 shutdown_connections 2023-05-22T15:45:10.530 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.521+0000 7fae9327c640 1 -- 172.21.15.16:0/3799961110 wait complete. 2023-05-22T15:45:10.530 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.521+0000 7fae9327c640 1 Processor -- start 2023-05-22T15:45:10.530 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.521+0000 7fae9327c640 1 -- start start 2023-05-22T15:45:10.530 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.525+0000 7fae9327c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fae8c06e440 0x7fae8c0acce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:10.530 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.525+0000 7fae9327c640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fae8c0b24a0 0x7fae8c0b4b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:10.530 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.525+0000 7fae9327c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fae8c15c130 0x7fae8c0aff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:10.531 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.525+0000 7fae9327c640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae8c0abc40 con 0x7fae8c06e440 2023-05-22T15:45:10.531 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.525+0000 7fae9327c640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae8c0abf20 con 0x7fae8c15c130 2023-05-22T15:45:10.531 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.525+0000 7fae9327c640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae8c0ac200 con 0x7fae8c0b24a0 2023-05-22T15:45:10.531 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.525+0000 7fae91018640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fae8c06e440 0x7fae8c0acce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:10.531 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.525+0000 7fae91018640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fae8c06e440 0x7fae8c0acce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36710/0 (socket says 172.21.15.16:36710) 2023-05-22T15:45:10.531 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.525+0000 7fae91018640 1 -- 172.21.15.16:0/3189844944 learned_addr learned my addr 172.21.15.16:0/3189844944 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:45:10.531 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.525+0000 7fae90817640 1 --2- 172.21.15.16:0/3189844944 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fae8c0b24a0 0x7fae8c0b4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:10.532 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.525+0000 7fae91819640 1 --2- 172.21.15.16:0/3189844944 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fae8c15c130 0x7fae8c0aff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:10.532 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.525+0000 7fae91018640 1 -- 172.21.15.16:0/3189844944 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fae8c0b24a0 msgr2=0x7fae8c0b4b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:45:10.532 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.525+0000 7fae91018640 1 --2- 172.21.15.16:0/3189844944 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fae8c0b24a0 0x7fae8c0b4b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:10.532 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.525+0000 7fae91018640 1 -- 172.21.15.16:0/3189844944 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fae8c15c130 msgr2=0x7fae8c0aff20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:45:10.532 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.525+0000 7fae91018640 1 --2- 172.21.15.16:0/3189844944 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fae8c15c130 0x7fae8c0aff20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:10.532 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.525+0000 7fae91018640 1 -- 172.21.15.16:0/3189844944 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae7c009420 con 0x7fae8c06e440 2023-05-22T15:45:10.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.525+0000 7fae91018640 1 --2- 172.21.15.16:0/3189844944 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fae8c06e440 0x7fae8c0acce0 secure :-1 s=READY pgs=1660 cs=0 l=1 rev1=1 crypto rx=0x7fae84009fc0 tx=0x7fae8400d560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:45:10.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.525+0000 7fae89ffb640 1 -- 172.21.15.16:0/3189844944 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae84003ba0 con 0x7fae8c06e440 2023-05-22T15:45:10.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.525+0000 7fae89ffb640 1 -- 172.21.15.16:0/3189844944 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae8400f040 con 0x7fae8c06e440 2023-05-22T15:45:10.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.525+0000 7fae9327c640 1 -- 172.21.15.16:0/3189844944 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae8c0cacc0 con 0x7fae8c06e440 2023-05-22T15:45:10.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.525+0000 7fae89ffb640 1 -- 172.21.15.16:0/3189844944 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae84004030 con 0x7fae8c06e440 2023-05-22T15:45:10.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.525+0000 7fae9327c640 1 -- 172.21.15.16:0/3189844944 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae8c0ca9e0 con 0x7fae8c06e440 2023-05-22T15:45:10.540 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.525+0000 7fae89ffb640 1 -- 172.21.15.16:0/3189844944 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 50) v1 ==== 84332+0+0 (secure 0 0 0) 0x7fae84004190 con 0x7fae8c06e440 2023-05-22T15:45:10.540 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.525+0000 7fae89ffb640 1 --2- 172.21.15.16:0/3189844944 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fae780654a0 0x7fae78067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:10.540 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.525+0000 7fae89ffb640 1 -- 172.21.15.16:0/3189844944 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 6382+0+0 (secure 0 0 0) 0x7fae84082ef0 con 0x7fae8c06e440 2023-05-22T15:45:10.541 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.525+0000 7fae90817640 1 --2- 172.21.15.16:0/3189844944 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fae780654a0 0x7fae78067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:10.541 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.529+0000 7fae90817640 1 --2- 172.21.15.16:0/3189844944 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fae780654a0 0x7fae78067950 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fae8c1f7aa0 tx=0x7fae80007b60 comp rx=0 tx=0).ready entity=mgr.8304 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:45:10.541 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.529+0000 7fae9639e000 1 Processor -- start 2023-05-22T15:45:10.541 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.529+0000 7fae9639e000 1 -- start start 2023-05-22T15:45:10.541 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.529+0000 7fae9639e000 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x556a2fa5bfd0 0x556a2fccec30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:10.541 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.529+0000 7fae9639e000 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556a2fccf170 0x556a2fccf540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:10.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.529+0000 7fae9639e000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556a2fc82a10 0x556a2fc82de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:10.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.529+0000 7fae9639e000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x556a2fb4cbd0 con 0x556a2fc82a10 2023-05-22T15:45:10.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.529+0000 7fae9639e000 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x556a2fa69dd0 con 0x556a2fccf170 2023-05-22T15:45:10.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.529+0000 7fae9639e000 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x556a2fada640 con 0x556a2fa5bfd0 2023-05-22T15:45:10.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.529+0000 7fae91819640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556a2fc82a10 0x556a2fc82de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:10.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.529+0000 7fae91819640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556a2fc82a10 0x556a2fc82de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36720/0 (socket says 172.21.15.16:36720) 2023-05-22T15:45:10.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.529+0000 7fae91819640 1 -- 172.21.15.16:0/1966993504 learned_addr learned my addr 172.21.15.16:0/1966993504 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:45:10.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.529+0000 7fae91018640 1 --2- 172.21.15.16:0/1966993504 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x556a2fa5bfd0 0x556a2fccec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:10.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.529+0000 7fae90817640 1 --2- 172.21.15.16:0/1966993504 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556a2fccf170 0x556a2fccf540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:10.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.529+0000 7fae91819640 1 -- 172.21.15.16:0/1966993504 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x556a2fa5bfd0 msgr2=0x556a2fccec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:45:10.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.529+0000 7fae91819640 1 --2- 172.21.15.16:0/1966993504 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x556a2fa5bfd0 0x556a2fccec30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:10.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.529+0000 7fae91819640 1 -- 172.21.15.16:0/1966993504 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556a2fccf170 msgr2=0x556a2fccf540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:45:10.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.529+0000 7fae91819640 1 --2- 172.21.15.16:0/1966993504 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556a2fccf170 0x556a2fccf540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:10.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.529+0000 7fae91819640 1 -- 172.21.15.16:0/1966993504 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556a2fa990b0 con 0x556a2fc82a10 2023-05-22T15:45:10.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.529+0000 7fae90817640 1 --2- 172.21.15.16:0/1966993504 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556a2fccf170 0x556a2fccf540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:45:10.544 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.529+0000 7fae91819640 1 --2- 172.21.15.16:0/1966993504 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556a2fc82a10 0x556a2fc82de0 secure :-1 s=READY pgs=1661 cs=0 l=1 rev1=1 crypto rx=0x7fae7c01a000 tx=0x7fae7c0a6000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a8db8ceb28abc52 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:45:10.544 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.529+0000 7fae72ffd640 1 -- 172.21.15.16:0/1966993504 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae7c0ab7c0 con 0x556a2fc82a10 2023-05-22T15:45:10.544 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.529+0000 7fae72ffd640 1 -- 172.21.15.16:0/1966993504 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae7c0ab920 con 0x556a2fc82a10 2023-05-22T15:45:10.544 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.529+0000 7fae72ffd640 1 -- 172.21.15.16:0/1966993504 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae7c0abcb0 con 0x556a2fc82a10 2023-05-22T15:45:10.544 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.529+0000 7fae9639e000 1 -- 172.21.15.16:0/1966993504 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556a2fc82a10 msgr2=0x556a2fc82de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:45:10.544 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.529+0000 7fae9639e000 1 --2- 172.21.15.16:0/1966993504 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556a2fc82a10 0x556a2fc82de0 secure :-1 s=READY pgs=1661 cs=0 l=1 rev1=1 crypto rx=0x7fae7c01a000 tx=0x7fae7c0a6000 comp rx=0 tx=0).stop 2023-05-22T15:45:10.544 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.529+0000 7fae9639e000 1 -- 172.21.15.16:0/1966993504 shutdown_connections 2023-05-22T15:45:10.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.529+0000 7fae9639e000 1 --2- 172.21.15.16:0/1966993504 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556a2fc82a10 0x556a2fc82de0 unknown :-1 s=CLOSED pgs=1661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:10.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.529+0000 7fae9639e000 1 --2- 172.21.15.16:0/1966993504 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556a2fccf170 0x556a2fccf540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:10.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.529+0000 7fae9639e000 1 --2- 172.21.15.16:0/1966993504 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x556a2fa5bfd0 0x556a2fccec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:10.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.529+0000 7fae9639e000 1 -- 172.21.15.16:0/1966993504 >> 172.21.15.16:0/1966993504 conn(0x556a2fb2b270 msgr2=0x556a2fcb2220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:45:10.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.529+0000 7fae9639e000 1 -- 172.21.15.16:0/1966993504 shutdown_connections 2023-05-22T15:45:10.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.529+0000 7fae9639e000 1 -- 172.21.15.16:0/1966993504 wait complete. 2023-05-22T15:45:10.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.529+0000 7fae9639e000 1 -- start start 2023-05-22T15:45:10.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.529+0000 7fae9639e000 1 Processor -- start 2023-05-22T15:45:10.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.533+0000 7fae9639e000 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x556a2fa5bfd0 0x556a2fc8aa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:10.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.533+0000 7fae9639e000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556a2fc82a10 0x556a2fc8af60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:10.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.533+0000 7fae9639e000 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556a2fccf170 0x556a2fcb44a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:10.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.533+0000 7fae9639e000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x556a2fad3780 con 0x556a2fc82a10 2023-05-22T15:45:10.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.533+0000 7fae9639e000 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x556a2fa91250 con 0x556a2fccf170 2023-05-22T15:45:10.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.533+0000 7fae9639e000 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x556a2fa90ef0 con 0x556a2fa5bfd0 2023-05-22T15:45:10.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.533+0000 7fae9639e000 10 client.0 _ll_register_callbacks cb 0x556a2fc70b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-05-22T15:45:10.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.533+0000 7fae90817640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556a2fc82a10 0x556a2fc8af60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:10.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.533+0000 7fae90817640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556a2fc82a10 0x556a2fc8af60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36732/0 (socket says 172.21.15.16:36732) 2023-05-22T15:45:10.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.533+0000 7fae90817640 1 -- 172.21.15.16:0/2316364797 learned_addr learned my addr 172.21.15.16:0/2316364797 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:45:10.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.533+0000 7fae91018640 1 --2- 172.21.15.16:0/2316364797 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x556a2fa5bfd0 0x556a2fc8aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:10.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.533+0000 7fae91819640 1 --2- 172.21.15.16:0/2316364797 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556a2fccf170 0x556a2fcb44a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:10.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.533+0000 7fae90817640 1 -- 172.21.15.16:0/2316364797 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x556a2fa5bfd0 msgr2=0x556a2fc8aa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:45:10.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.533+0000 7fae90817640 1 --2- 172.21.15.16:0/2316364797 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x556a2fa5bfd0 0x556a2fc8aa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:10.549 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.533+0000 7fae90817640 1 -- 172.21.15.16:0/2316364797 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556a2fccf170 msgr2=0x556a2fcb44a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:45:10.549 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.533+0000 7fae90817640 1 --2- 172.21.15.16:0/2316364797 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556a2fccf170 0x556a2fcb44a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:10.549 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.533+0000 7fae90817640 1 -- 172.21.15.16:0/2316364797 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae8c0cf510 con 0x556a2fc82a10 2023-05-22T15:45:10.549 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.533+0000 7fae91819640 1 --2- 172.21.15.16:0/2316364797 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556a2fccf170 0x556a2fcb44a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:45:10.549 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.533+0000 7fae91018640 1 --2- 172.21.15.16:0/2316364797 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x556a2fa5bfd0 0x556a2fc8aa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:45:10.549 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.533+0000 7fae90817640 1 --2- 172.21.15.16:0/2316364797 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556a2fc82a10 0x556a2fc8af60 secure :-1 s=READY pgs=1662 cs=0 l=1 rev1=1 crypto rx=0x7fae80001e90 tx=0x7fae80011000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54caacb5b8f86c2f server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:45:10.549 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.533+0000 7fae6bfff640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2023-05-22T15:45:10.549 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.533+0000 7fae6bfff640 1 -- 172.21.15.16:0/2316364797 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae80005df0 con 0x556a2fc82a10 2023-05-22T15:45:10.550 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.533+0000 7fae6bfff640 1 -- 172.21.15.16:0/2316364797 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(132..132 src has 1..132) v4 ==== 6382+0+0 (secure 0 0 0) 0x7fae80021d20 con 0x556a2fc82a10 2023-05-22T15:45:10.550 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.533+0000 7fae9639e000 1 -- 172.21.15.16:0/2316364797 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x556a2fa97c60 con 0x556a2fc82a10 2023-05-22T15:45:10.550 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.533+0000 7fae6bfff640 1 -- 172.21.15.16:0/2316364797 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=50) v2 ==== 24+0+0 (secure 0 0 0) 0x7fae80022430 con 0x556a2fc82a10 2023-05-22T15:45:10.550 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.533+0000 7fae9639e000 10 client.8691 fetch_fsmap learned FSMap version 50 2023-05-22T15:45:10.550 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.533+0000 7fae9639e000 1 -- 172.21.15.16:0/2316364797 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=50,monmap=2+}) v3 -- 0x556a2f7d0890 con 0x556a2fc82a10 2023-05-22T15:45:10.550 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.533+0000 7fae6bfff640 1 -- 172.21.15.16:0/2316364797 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x556a2fa97c60 con 0x556a2fc82a10 2023-05-22T15:45:10.550 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.533+0000 7fae6bfff640 1 -- 172.21.15.16:0/2316364797 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 50) v1 ==== 3106+0+0 (secure 0 0 0) 0x7fae80021420 con 0x556a2fc82a10 2023-05-22T15:45:10.550 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.533+0000 7fae9639e000 10 client.8691 fetch_fsmap finished waiting for FSMap version 50 2023-05-22T15:45:10.551 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.533+0000 7fae9639e000 10 client.8691 resolve_mds: resolved name 'a' to daemon mds.a 2023-05-22T15:45:10.551 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.533+0000 7fae9639e000 20 client.8691 populate_metadata read hostname 'smithi016' 2023-05-22T15:45:10.551 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.533+0000 7fae9639e000 1 --2- 172.21.15.16:0/2316364797 >> [v2:172.21.15.16:6836/1566663454,v1:172.21.15.16:6837/1566663454] conn(0x556a2fcde8e0 0x556a2fce0cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:10.551 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.533+0000 7fae9639e000 4 client.8691 mds_command: new command op to 8547 tid=0[{"prefix": "get_command_descriptions"}] 2023-05-22T15:45:10.551 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.533+0000 7fae9639e000 1 -- 172.21.15.16:0/2316364797 --> [v2:172.21.15.16:6836/1566663454,v1:172.21.15.16:6837/1566663454] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x556a2fad20d0 con 0x556a2fcde8e0 2023-05-22T15:45:10.551 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.533+0000 7fae91018640 1 --2- 172.21.15.16:0/2316364797 >> [v2:172.21.15.16:6836/1566663454,v1:172.21.15.16:6837/1566663454] conn(0x556a2fcde8e0 0x556a2fce0cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:10.551 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.533+0000 7fae6bfff640 10 client.8691 ms_handle_connect on v2:172.21.15.16:6836/1566663454 2023-05-22T15:45:10.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.533+0000 7fae91018640 1 --2- 172.21.15.16:0/2316364797 >> [v2:172.21.15.16:6836/1566663454,v1:172.21.15.16:6837/1566663454] conn(0x556a2fcde8e0 0x556a2fce0cc0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5e90f60403cc90c1 server_cookie=e926047659496d65 in_seq=0 out_seq=0 2023-05-22T15:45:10.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.533+0000 7fae6bfff640 1 -- 172.21.15.16:0/2316364797 <== mds.0 v2:172.21.15.16:6836/1566663454 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x556a2fad20d0 con 0x556a2fcde8e0 2023-05-22T15:45:10.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.533+0000 7fae6bfff640 10 client.8691 handle_command_reply: tid=0 2023-05-22T15:45:10.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.533+0000 7fae9639e000 1 client.8691 shutdown 2023-05-22T15:45:10.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.533+0000 7fae9639e000 1 -- 172.21.15.16:0/2316364797 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556a2fc82a10 msgr2=0x556a2fc8af60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:45:10.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.533+0000 7fae9639e000 1 --2- 172.21.15.16:0/2316364797 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556a2fc82a10 0x556a2fc8af60 secure :-1 s=READY pgs=1662 cs=0 l=1 rev1=1 crypto rx=0x7fae80001e90 tx=0x7fae80011000 comp rx=0 tx=0).stop 2023-05-22T15:45:10.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.533+0000 7fae9639e000 1 -- 172.21.15.16:0/2316364797 shutdown_connections 2023-05-22T15:45:10.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.533+0000 7fae9639e000 1 --2- 172.21.15.16:0/2316364797 >> [v2:172.21.15.16:6836/1566663454,v1:172.21.15.16:6837/1566663454] conn(0x556a2fcde8e0 0x556a2fce0cc0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:10.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.533+0000 7fae9639e000 1 --2- 172.21.15.16:0/2316364797 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556a2fccf170 0x556a2fcb44a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:10.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.533+0000 7fae9639e000 1 --2- 172.21.15.16:0/2316364797 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556a2fc82a10 0x556a2fc8af60 unknown :-1 s=CLOSED pgs=1662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:10.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.533+0000 7fae9639e000 1 --2- 172.21.15.16:0/2316364797 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x556a2fa5bfd0 0x556a2fc8aa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:10.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.533+0000 7fae9639e000 1 -- 172.21.15.16:0/2316364797 >> 172.21.15.16:0/2316364797 conn(0x556a2fb2b270 msgr2=0x556a2fc82100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:45:10.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.533+0000 7fae6bfff640 0 client.8691 ms_handle_reset on v2:172.21.15.16:6836/1566663454 2023-05-22T15:45:10.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.533+0000 7fae9639e000 1 -- 172.21.15.16:0/2316364797 shutdown_connections 2023-05-22T15:45:10.554 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.533+0000 7fae9639e000 1 -- 172.21.15.16:0/2316364797 wait complete. 2023-05-22T15:45:10.554 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.533+0000 7fae9639e000 20 client.8691 trim_cache size 0 max 16384 2023-05-22T15:45:10.556 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.549+0000 7fae9639e000 1 Processor -- start 2023-05-22T15:45:10.556 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.549+0000 7fae9639e000 1 -- start start 2023-05-22T15:45:10.556 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.549+0000 7fae9639e000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556a2fa5bfd0 0x556a2fcb3450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:10.556 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.549+0000 7fae9639e000 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x556a2fc82a10 0x556a2fcb3990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:10.556 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.549+0000 7fae9639e000 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556a2fccf170 0x556a2fcce310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:10.556 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.549+0000 7fae9639e000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x556a2f791940 con 0x556a2fa5bfd0 2023-05-22T15:45:10.557 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.549+0000 7fae9639e000 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x556a2facdb20 con 0x556a2fccf170 2023-05-22T15:45:10.557 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.549+0000 7fae9639e000 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x556a2fc7e490 con 0x556a2fc82a10 2023-05-22T15:45:10.557 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.549+0000 7fae91018640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556a2fa5bfd0 0x556a2fcb3450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:10.557 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.549+0000 7fae91018640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556a2fa5bfd0 0x556a2fcb3450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36744/0 (socket says 172.21.15.16:36744) 2023-05-22T15:45:10.557 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.549+0000 7fae91018640 1 -- 172.21.15.16:0/3203190410 learned_addr learned my addr 172.21.15.16:0/3203190410 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:45:10.557 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.549+0000 7fae91819640 1 --2- 172.21.15.16:0/3203190410 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556a2fccf170 0x556a2fcce310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:10.557 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.553+0000 7fae90817640 1 --2- 172.21.15.16:0/3203190410 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x556a2fc82a10 0x556a2fcb3990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:10.557 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.553+0000 7fae91018640 1 -- 172.21.15.16:0/3203190410 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x556a2fc82a10 msgr2=0x556a2fcb3990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:45:10.558 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.553+0000 7fae91018640 1 --2- 172.21.15.16:0/3203190410 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x556a2fc82a10 0x556a2fcb3990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:10.558 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.553+0000 7fae91018640 1 -- 172.21.15.16:0/3203190410 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556a2fccf170 msgr2=0x556a2fcce310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:45:10.558 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.553+0000 7fae91018640 1 --2- 172.21.15.16:0/3203190410 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556a2fccf170 0x556a2fcce310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:10.558 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.553+0000 7fae91018640 1 -- 172.21.15.16:0/3203190410 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556a2fa99360 con 0x556a2fa5bfd0 2023-05-22T15:45:10.558 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.553+0000 7fae91819640 1 --2- 172.21.15.16:0/3203190410 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556a2fccf170 0x556a2fcce310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T15:45:10.558 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.553+0000 7fae91018640 1 --2- 172.21.15.16:0/3203190410 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556a2fa5bfd0 0x556a2fcb3450 secure :-1 s=READY pgs=1663 cs=0 l=1 rev1=1 crypto rx=0x7fae8405a1f0 tx=0x7fae84004a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c40aebf441e96a1c server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:45:10.558 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.553+0000 7fae737fe640 1 -- 172.21.15.16:0/3203190410 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae840529e0 con 0x556a2fa5bfd0 2023-05-22T15:45:10.559 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.553+0000 7fae737fe640 1 -- 172.21.15.16:0/3203190410 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae8406b070 con 0x556a2fa5bfd0 2023-05-22T15:45:10.559 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.553+0000 7fae737fe640 1 -- 172.21.15.16:0/3203190410 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae84050c90 con 0x556a2fa5bfd0 2023-05-22T15:45:10.559 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.553+0000 7fae9639e000 1 -- 172.21.15.16:0/3203190410 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556a2fa5bfd0 msgr2=0x556a2fcb3450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:45:10.559 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.553+0000 7fae9639e000 1 --2- 172.21.15.16:0/3203190410 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556a2fa5bfd0 0x556a2fcb3450 secure :-1 s=READY pgs=1663 cs=0 l=1 rev1=1 crypto rx=0x7fae8405a1f0 tx=0x7fae84004a50 comp rx=0 tx=0).stop 2023-05-22T15:45:10.559 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.553+0000 7fae9639e000 1 -- 172.21.15.16:0/3203190410 shutdown_connections 2023-05-22T15:45:10.559 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.553+0000 7fae9639e000 1 --2- 172.21.15.16:0/3203190410 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556a2fccf170 0x556a2fcce310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:10.559 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.553+0000 7fae9639e000 1 --2- 172.21.15.16:0/3203190410 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x556a2fc82a10 0x556a2fcb3990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:10.559 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.553+0000 7fae9639e000 1 --2- 172.21.15.16:0/3203190410 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556a2fa5bfd0 0x556a2fcb3450 unknown :-1 s=CLOSED pgs=1663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:10.560 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.553+0000 7fae9639e000 1 -- 172.21.15.16:0/3203190410 >> 172.21.15.16:0/3203190410 conn(0x556a2fb2b270 msgr2=0x556a2fcdbe30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:45:10.560 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.553+0000 7fae9639e000 1 -- 172.21.15.16:0/3203190410 shutdown_connections 2023-05-22T15:45:10.560 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.553+0000 7fae9639e000 1 -- 172.21.15.16:0/3203190410 wait complete. 2023-05-22T15:45:10.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.553+0000 7fae9639e000 1 -- start start 2023-05-22T15:45:10.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.553+0000 7fae9639e000 1 Processor -- start 2023-05-22T15:45:10.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.553+0000 7fae9639e000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556a2fa5bfd0 0x556a2fd2ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:10.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.553+0000 7fae9639e000 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x556a2fc82a10 0x556a2fd2dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:10.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.553+0000 7fae9639e000 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556a2fccf170 0x556a2fd30500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:10.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.553+0000 7fae9639e000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x556a2fb5d250 con 0x556a2fa5bfd0 2023-05-22T15:45:10.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.553+0000 7fae9639e000 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x556a2fad62e0 con 0x556a2fccf170 2023-05-22T15:45:10.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.553+0000 7fae9639e000 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x556a2f7d11b0 con 0x556a2fc82a10 2023-05-22T15:45:10.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.553+0000 7fae9639e000 10 client.0 _ll_register_callbacks cb 0x556a2fb23eb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-05-22T15:45:10.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.553+0000 7fae91018640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556a2fa5bfd0 0x556a2fd2ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:10.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.553+0000 7fae91018640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556a2fa5bfd0 0x556a2fd2ba60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36758/0 (socket says 172.21.15.16:36758) 2023-05-22T15:45:10.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.553+0000 7fae91018640 1 -- 172.21.15.16:0/2238863283 learned_addr learned my addr 172.21.15.16:0/2238863283 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:45:10.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.553+0000 7fae91018640 1 -- 172.21.15.16:0/2238863283 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x556a2fc82a10 msgr2=0x556a2fd2dfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:45:10.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.553+0000 7fae90817640 1 --2- 172.21.15.16:0/2238863283 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x556a2fc82a10 0x556a2fd2dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:10.565 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.553+0000 7fae91018640 1 --2- 172.21.15.16:0/2238863283 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x556a2fc82a10 0x556a2fd2dfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:10.565 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.553+0000 7fae91018640 1 -- 172.21.15.16:0/2238863283 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556a2fccf170 msgr2=0x556a2fd30500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:45:10.565 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.553+0000 7fae91819640 1 --2- 172.21.15.16:0/2238863283 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556a2fccf170 0x556a2fd30500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:10.565 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.553+0000 7fae91018640 1 --2- 172.21.15.16:0/2238863283 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556a2fccf170 0x556a2fd30500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:10.565 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.553+0000 7fae91018640 1 -- 172.21.15.16:0/2238863283 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae8c0ca9e0 con 0x556a2fa5bfd0 2023-05-22T15:45:10.565 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.553+0000 7fae91018640 1 --2- 172.21.15.16:0/2238863283 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556a2fa5bfd0 0x556a2fd2ba60 secure :-1 s=READY pgs=1664 cs=0 l=1 rev1=1 crypto rx=0x7fae8400da60 tx=0x7fae80009030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5f5036b59967bf1 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:45:10.565 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.553+0000 7fae6bfff640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2023-05-22T15:45:10.565 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.553+0000 7fae6bfff640 1 -- 172.21.15.16:0/2238863283 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae8406bbe0 con 0x556a2fa5bfd0 2023-05-22T15:45:10.566 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.553+0000 7fae6bfff640 1 -- 172.21.15.16:0/2238863283 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(132..132 src has 1..132) v4 ==== 6382+0+0 (secure 0 0 0) 0x7fae8406bd60 con 0x556a2fa5bfd0 2023-05-22T15:45:10.566 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.553+0000 7fae9639e000 1 -- 172.21.15.16:0/2238863283 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x556a2fa7a4d0 con 0x556a2fa5bfd0 2023-05-22T15:45:10.566 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.553+0000 7fae6bfff640 1 -- 172.21.15.16:0/2238863283 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=50) v2 ==== 24+0+0 (secure 0 0 0) 0x7fae84053a70 con 0x556a2fa5bfd0 2023-05-22T15:45:10.566 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.553+0000 7fae9639e000 10 client.8697 fetch_fsmap learned FSMap version 50 2023-05-22T15:45:10.566 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.553+0000 7fae9639e000 1 -- 172.21.15.16:0/2238863283 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=50,monmap=2+}) v3 -- 0x556a2fa7ebe0 con 0x556a2fa5bfd0 2023-05-22T15:45:10.566 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.553+0000 7fae6bfff640 1 -- 172.21.15.16:0/2238863283 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x556a2fa7a4d0 con 0x556a2fa5bfd0 2023-05-22T15:45:10.566 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.553+0000 7fae6bfff640 1 -- 172.21.15.16:0/2238863283 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 50) v1 ==== 3106+0+0 (secure 0 0 0) 0x7fae84064030 con 0x556a2fa5bfd0 2023-05-22T15:45:10.567 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.553+0000 7fae9639e000 10 client.8697 fetch_fsmap finished waiting for FSMap version 50 2023-05-22T15:45:10.567 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.553+0000 7fae9639e000 10 client.8697 resolve_mds: resolved name 'a' to daemon mds.a 2023-05-22T15:45:10.567 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.553+0000 7fae9639e000 20 client.8697 populate_metadata read hostname 'smithi016' 2023-05-22T15:45:10.567 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.553+0000 7fae9639e000 1 --2- 172.21.15.16:0/2238863283 >> [v2:172.21.15.16:6836/1566663454,v1:172.21.15.16:6837/1566663454] conn(0x556a2fd30a40 0x556a2fd32e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:10.567 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.557+0000 7fae9639e000 4 client.8697 mds_command: new command op to 8547 tid=0[{"prefix": "status"}] 2023-05-22T15:45:10.567 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.557+0000 7fae9639e000 1 -- 172.21.15.16:0/2238863283 --> [v2:172.21.15.16:6836/1566663454,v1:172.21.15.16:6837/1566663454] -- command(tid 0: {"prefix": "status"}) v1 -- 0x556a2facc800 con 0x556a2fd30a40 2023-05-22T15:45:10.567 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.557+0000 7fae90817640 1 --2- 172.21.15.16:0/2238863283 >> [v2:172.21.15.16:6836/1566663454,v1:172.21.15.16:6837/1566663454] conn(0x556a2fd30a40 0x556a2fd32e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:10.567 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.557+0000 7fae90817640 1 --2- 172.21.15.16:0/2238863283 >> [v2:172.21.15.16:6836/1566663454,v1:172.21.15.16:6837/1566663454] conn(0x556a2fd30a40 0x556a2fd32e20 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c371b5bf2ee29f2d server_cookie=e0b0593730a4a782 in_seq=0 out_seq=0 2023-05-22T15:45:10.568 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.557+0000 7fae6bfff640 10 client.8697 ms_handle_connect on v2:172.21.15.16:6836/1566663454 2023-05-22T15:45:10.568 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.557+0000 7fae6bfff640 1 -- 172.21.15.16:0/2238863283 <== mds.0 v2:172.21.15.16:6836/1566663454 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+333 (crc 0 0 0) 0x556a2facc800 con 0x556a2fd30a40 2023-05-22T15:45:10.568 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.557+0000 7fae6bfff640 10 client.8697 handle_command_reply: tid=0 2023-05-22T15:45:10.568 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.557+0000 7fae9639e000 1 client.8697 shutdown 2023-05-22T15:45:10.568 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.557+0000 7fae9639e000 1 -- 172.21.15.16:0/2238863283 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556a2fa5bfd0 msgr2=0x556a2fd2ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:45:10.568 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.557+0000 7fae9639e000 1 --2- 172.21.15.16:0/2238863283 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556a2fa5bfd0 0x556a2fd2ba60 secure :-1 s=READY pgs=1664 cs=0 l=1 rev1=1 crypto rx=0x7fae8400da60 tx=0x7fae80009030 comp rx=0 tx=0).stop 2023-05-22T15:45:10.568 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.557+0000 7fae9639e000 1 -- 172.21.15.16:0/2238863283 shutdown_connections 2023-05-22T15:45:10.568 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.557+0000 7fae9639e000 1 --2- 172.21.15.16:0/2238863283 >> [v2:172.21.15.16:6836/1566663454,v1:172.21.15.16:6837/1566663454] conn(0x556a2fd30a40 0x556a2fd32e20 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:10.569 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.557+0000 7fae9639e000 1 --2- 172.21.15.16:0/2238863283 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556a2fccf170 0x556a2fd30500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:10.569 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.557+0000 7fae9639e000 1 --2- 172.21.15.16:0/2238863283 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x556a2fc82a10 0x556a2fd2dfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:10.569 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.557+0000 7fae6bfff640 0 client.8697 ms_handle_reset on v2:172.21.15.16:6836/1566663454 2023-05-22T15:45:10.569 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.557+0000 7fae9639e000 1 --2- 172.21.15.16:0/2238863283 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556a2fa5bfd0 0x556a2fd2ba60 unknown :-1 s=CLOSED pgs=1664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:10.569 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.557+0000 7fae9639e000 1 -- 172.21.15.16:0/2238863283 >> 172.21.15.16:0/2238863283 conn(0x556a2fb2b270 msgr2=0x556a2fcdb5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:45:10.569 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.557+0000 7fae9639e000 1 -- 172.21.15.16:0/2238863283 shutdown_connections 2023-05-22T15:45:10.569 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.557+0000 7fae9639e000 1 -- 172.21.15.16:0/2238863283 wait complete. 2023-05-22T15:45:10.570 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.557+0000 7fae9639e000 20 client.8697 trim_cache size 0 max 16384 2023-05-22T15:45:10.570 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-05-22T15:45:10.570 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_fsid": "b2719834-7847-4d98-90a6-f65d57f85c49", 2023-05-22T15:45:10.571 INFO:teuthology.orchestra.run.smithi016.stdout: "whoami": 0, 2023-05-22T15:45:10.571 INFO:teuthology.orchestra.run.smithi016.stdout: "id": 8547, 2023-05-22T15:45:10.571 INFO:teuthology.orchestra.run.smithi016.stdout: "want_state": "up:active", 2023-05-22T15:45:10.571 INFO:teuthology.orchestra.run.smithi016.stdout: "state": "up:active", 2023-05-22T15:45:10.571 INFO:teuthology.orchestra.run.smithi016.stdout: "fs_name": "cephfs", 2023-05-22T15:45:10.571 INFO:teuthology.orchestra.run.smithi016.stdout: "rank_uptime": 5.3536655949999998, 2023-05-22T15:45:10.571 INFO:teuthology.orchestra.run.smithi016.stdout: "mdsmap_epoch": 50, 2023-05-22T15:45:10.571 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch": 132, 2023-05-22T15:45:10.572 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch_barrier": 132, 2023-05-22T15:45:10.572 INFO:teuthology.orchestra.run.smithi016.stdout: "uptime": 9.8308010120000002 2023-05-22T15:45:10.572 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-05-22T15:45:10.573 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.557+0000 7fae9327c640 1 -- 172.21.15.16:0/3189844944 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fae780654a0 msgr2=0x7fae78067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:45:10.573 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.557+0000 7fae9327c640 1 --2- 172.21.15.16:0/3189844944 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fae780654a0 0x7fae78067950 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fae8c1f7aa0 tx=0x7fae80007b60 comp rx=0 tx=0).stop 2023-05-22T15:45:10.573 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.557+0000 7fae9327c640 1 -- 172.21.15.16:0/3189844944 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fae8c06e440 msgr2=0x7fae8c0acce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:45:10.573 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.557+0000 7fae9327c640 1 --2- 172.21.15.16:0/3189844944 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fae8c06e440 0x7fae8c0acce0 secure :-1 s=READY pgs=1660 cs=0 l=1 rev1=1 crypto rx=0x7fae84009fc0 tx=0x7fae8400d560 comp rx=0 tx=0).stop 2023-05-22T15:45:10.573 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.557+0000 7fae9327c640 1 -- 172.21.15.16:0/3189844944 shutdown_connections 2023-05-22T15:45:10.573 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.557+0000 7fae9327c640 1 --2- 172.21.15.16:0/3189844944 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fae8c15c130 0x7fae8c0aff20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:10.573 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.557+0000 7fae9327c640 1 --2- 172.21.15.16:0/3189844944 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fae780654a0 0x7fae78067950 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:10.574 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.557+0000 7fae9327c640 1 --2- 172.21.15.16:0/3189844944 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fae8c0b24a0 0x7fae8c0b4b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:10.574 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.557+0000 7fae9327c640 1 --2- 172.21.15.16:0/3189844944 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fae8c06e440 0x7fae8c0acce0 unknown :-1 s=CLOSED pgs=1660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:10.574 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.557+0000 7fae9327c640 1 -- 172.21.15.16:0/3189844944 >> 172.21.15.16:0/3189844944 conn(0x7fae8c06f5c0 msgr2=0x7fae8c0cbf30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:45:10.574 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.557+0000 7fae9327c640 1 -- 172.21.15.16:0/3189844944 shutdown_connections 2023-05-22T15:45:10.574 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.557+0000 7fae9327c640 1 -- 172.21.15.16:0/3189844944 wait complete. 2023-05-22T15:45:10.576 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-05-22T15:45:10.690 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.685+0000 7f707e32f640 1 Processor -- start 2023-05-22T15:45:10.690 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.685+0000 7f707e32f640 1 -- start start 2023-05-22T15:45:10.692 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.689+0000 7f707e32f640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f70780ca4e0 0x7f707815e460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:10.692 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.689+0000 7f707e32f640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f7078070df0 0x7f7078071210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:10.692 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.689+0000 7f707e32f640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70780b0650 con 0x7f70780c9b40 2023-05-22T15:45:10.692 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.689+0000 7f707e32f640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70780b0370 con 0x7f7078070df0 2023-05-22T15:45:10.693 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.689+0000 7f707e32f640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f707815d980 con 0x7f70780ca4e0 2023-05-22T15:45:10.693 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.689+0000 7f707c8cc640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f7078070df0 0x7f7078071210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:10.693 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.689+0000 7f707c8cc640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f7078070df0 0x7f7078071210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36764/0 (socket says 172.21.15.16:36764) 2023-05-22T15:45:10.693 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.689+0000 7f707c8cc640 1 -- 172.21.15.16:0/78938327 learned_addr learned my addr 172.21.15.16:0/78938327 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:45:10.693 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.689+0000 7f7077fff640 1 --2- 172.21.15.16:0/78938327 >> v2:172.21.15.62:3300/0 conn(0x7f70780ca4e0 0x7f707815e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:10.693 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.689+0000 7f707c8cc640 1 -- 172.21.15.16:0/78938327 >> v2:172.21.15.62:3300/0 conn(0x7f70780ca4e0 msgr2=0x7f707815e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:45:10.693 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.689+0000 7f707c8cc640 1 --2- 172.21.15.16:0/78938327 >> v2:172.21.15.62:3300/0 conn(0x7f70780ca4e0 0x7f707815e460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:10.694 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.689+0000 7f707c8cc640 1 -- 172.21.15.16:0/78938327 >> v1:172.21.15.62:6789/0 conn(0x7f70780c9b40 legacy=0x7f70780afb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:45:10.694 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.689+0000 7f707c8cc640 1 -- 172.21.15.16:0/78938327 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f707815dc60 con 0x7f7078070df0 2023-05-22T15:45:10.694 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.689+0000 7f7077fff640 1 --2- 172.21.15.16:0/78938327 >> v2:172.21.15.62:3300/0 conn(0x7f70780ca4e0 0x7f707815e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:45:10.695 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.689+0000 7f707c8cc640 1 --2- 172.21.15.16:0/78938327 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7078070df0 0x7f7078071210 secure :-1 s=READY pgs=1665 cs=0 l=1 rev1=1 crypto rx=0x7f70780cad40 tx=0x7f707001c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f9828a4feca9df1 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:45:10.695 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.689+0000 7f7076ffd640 1 -- 172.21.15.16:0/78938327 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70700a5780 con 0x7f7078070df0 2023-05-22T15:45:10.695 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.689+0000 7f7076ffd640 1 -- 172.21.15.16:0/78938327 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70700a58e0 con 0x7f7078070df0 2023-05-22T15:45:10.696 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.689+0000 7f7076ffd640 1 -- 172.21.15.16:0/78938327 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70700a5c70 con 0x7f7078070df0 2023-05-22T15:45:10.696 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.693+0000 7f707e32f640 1 -- 172.21.15.16:0/78938327 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7078070df0 msgr2=0x7f7078071210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:45:10.696 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.693+0000 7f707e32f640 1 --2- 172.21.15.16:0/78938327 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7078070df0 0x7f7078071210 secure :-1 s=READY pgs=1665 cs=0 l=1 rev1=1 crypto rx=0x7f70780cad40 tx=0x7f707001c960 comp rx=0 tx=0).stop 2023-05-22T15:45:10.696 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.693+0000 7f707e32f640 1 -- 172.21.15.16:0/78938327 shutdown_connections 2023-05-22T15:45:10.696 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.693+0000 7f707e32f640 1 --2- 172.21.15.16:0/78938327 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7078070df0 0x7f7078071210 unknown :-1 s=CLOSED pgs=1665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:10.697 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.693+0000 7f707e32f640 1 --2- 172.21.15.16:0/78938327 >> v2:172.21.15.62:3300/0 conn(0x7f70780ca4e0 0x7f707815e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:10.697 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.693+0000 7f707e32f640 1 -- 172.21.15.16:0/78938327 >> 172.21.15.16:0/78938327 conn(0x7f7078069720 msgr2=0x7f7078069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:45:10.697 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.693+0000 7f707e32f640 1 -- 172.21.15.16:0/78938327 shutdown_connections 2023-05-22T15:45:10.697 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.693+0000 7f707e32f640 1 -- 172.21.15.16:0/78938327 wait complete. 2023-05-22T15:45:10.699 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.693+0000 7f707e32f640 1 Processor -- start 2023-05-22T15:45:10.699 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.693+0000 7f707e32f640 1 -- start start 2023-05-22T15:45:10.699 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.693+0000 7f707e32f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7078070df0 0x7f707815cc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:10.699 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.693+0000 7f707e32f640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f70780c9b40 0x7f707815a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:10.700 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.693+0000 7f707e32f640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f70780ca4e0 0x7f7078159160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:10.700 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.693+0000 7f707e32f640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f707815d150 con 0x7f7078070df0 2023-05-22T15:45:10.700 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.693+0000 7f707e32f640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70780c61a0 con 0x7f70780ca4e0 2023-05-22T15:45:10.700 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.693+0000 7f707e32f640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f707815ab90 con 0x7f70780c9b40 2023-05-22T15:45:10.700 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.693+0000 7f7077fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7078070df0 0x7f707815cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:10.700 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.693+0000 7f7077fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7078070df0 0x7f707815cc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36780/0 (socket says 172.21.15.16:36780) 2023-05-22T15:45:10.700 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.693+0000 7f7077fff640 1 -- 172.21.15.16:0/4013563648 learned_addr learned my addr 172.21.15.16:0/4013563648 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:45:10.700 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.693+0000 7f70777fe640 1 --2- 172.21.15.16:0/4013563648 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f70780c9b40 0x7f707815a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:10.701 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.693+0000 7f7077fff640 1 -- 172.21.15.16:0/4013563648 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f70780c9b40 msgr2=0x7f707815a2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:45:10.701 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.693+0000 7f707c8cc640 1 --2- 172.21.15.16:0/4013563648 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f70780ca4e0 0x7f7078159160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:10.701 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.693+0000 7f7077fff640 1 --2- 172.21.15.16:0/4013563648 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f70780c9b40 0x7f707815a2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:10.701 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.693+0000 7f7077fff640 1 -- 172.21.15.16:0/4013563648 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f70780ca4e0 msgr2=0x7f7078159160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:45:10.701 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.693+0000 7f7077fff640 1 --2- 172.21.15.16:0/4013563648 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f70780ca4e0 0x7f7078159160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:10.701 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.693+0000 7f7077fff640 1 -- 172.21.15.16:0/4013563648 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7070009b30 con 0x7f7078070df0 2023-05-22T15:45:10.702 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.693+0000 7f70777fe640 1 --2- 172.21.15.16:0/4013563648 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f70780c9b40 0x7f707815a2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:45:10.702 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.693+0000 7f7077fff640 1 --2- 172.21.15.16:0/4013563648 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7078070df0 0x7f707815cc10 secure :-1 s=READY pgs=1666 cs=0 l=1 rev1=1 crypto rx=0x7f706c00c7f0 tx=0x7f706c00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:45:10.702 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.693+0000 7f7074ff9640 1 -- 172.21.15.16:0/4013563648 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f706c007820 con 0x7f7078070df0 2023-05-22T15:45:10.702 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.693+0000 7f7074ff9640 1 -- 172.21.15.16:0/4013563648 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f706c007980 con 0x7f7078070df0 2023-05-22T15:45:10.702 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.693+0000 7f7074ff9640 1 -- 172.21.15.16:0/4013563648 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f706c007d30 con 0x7f7078070df0 2023-05-22T15:45:10.702 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.693+0000 7f707e32f640 1 -- 172.21.15.16:0/4013563648 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70780c8e80 con 0x7f7078070df0 2023-05-22T15:45:10.702 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.693+0000 7f707e32f640 1 -- 172.21.15.16:0/4013563648 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70780ca8b0 con 0x7f7078070df0 2023-05-22T15:45:10.703 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.697+0000 7f707e32f640 1 -- 172.21.15.16:0/4013563648 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70780c9010 con 0x7f7078070df0 2023-05-22T15:45:10.703 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.697+0000 7f7074ff9640 1 -- 172.21.15.16:0/4013563648 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 50) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f706c0113f0 con 0x7f7078070df0 2023-05-22T15:45:10.704 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.697+0000 7f7074ff9640 1 --2- 172.21.15.16:0/4013563648 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f70640654a0 0x7f7064067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:10.704 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.697+0000 7f7074ff9640 1 -- 172.21.15.16:0/4013563648 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f706c082fa0 con 0x7f7078070df0 2023-05-22T15:45:10.704 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.697+0000 7f70777fe640 1 --2- 172.21.15.16:0/4013563648 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f70640654a0 0x7f7064067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:10.704 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.697+0000 7f70777fe640 1 --2- 172.21.15.16:0/4013563648 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f70640654a0 0x7f7064067950 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f70780acff0 tx=0x7f7068009040 comp rx=0 tx=0).ready entity=mgr.8304 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:45:10.704 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.701+0000 7f7074ff9640 1 -- 172.21.15.16:0/4013563648 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f706c050860 con 0x7f7078070df0 2023-05-22T15:45:10.899 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.897+0000 7f707e32f640 1 -- 172.21.15.16:0/4013563648 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f707804dbe0 con 0x7f7078070df0 2023-05-22T15:45:10.900 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.897+0000 7f7074ff9640 1 -- 172.21.15.16:0/4013563648 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v50) v1 ==== 53+0+73 (secure 0 0 0) 0x7f706c053e80 con 0x7f7078070df0 2023-05-22T15:45:10.901 INFO:teuthology.orchestra.run.smithi016.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-05-22T15:45:10.902 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.897+0000 7f707e32f640 1 -- 172.21.15.16:0/4013563648 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f70640654a0 msgr2=0x7f7064067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:45:10.902 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.897+0000 7f707e32f640 1 --2- 172.21.15.16:0/4013563648 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f70640654a0 0x7f7064067950 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f70780acff0 tx=0x7f7068009040 comp rx=0 tx=0).stop 2023-05-22T15:45:10.903 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.897+0000 7f707e32f640 1 -- 172.21.15.16:0/4013563648 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7078070df0 msgr2=0x7f707815cc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:45:10.903 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.897+0000 7f707e32f640 1 --2- 172.21.15.16:0/4013563648 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7078070df0 0x7f707815cc10 secure :-1 s=READY pgs=1666 cs=0 l=1 rev1=1 crypto rx=0x7f706c00c7f0 tx=0x7f706c00ccb0 comp rx=0 tx=0).stop 2023-05-22T15:45:10.903 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.897+0000 7f707e32f640 1 -- 172.21.15.16:0/4013563648 shutdown_connections 2023-05-22T15:45:10.903 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.897+0000 7f707e32f640 1 --2- 172.21.15.16:0/4013563648 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f70780ca4e0 0x7f7078159160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:10.903 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.897+0000 7f707e32f640 1 --2- 172.21.15.16:0/4013563648 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f70640654a0 0x7f7064067950 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:10.904 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.897+0000 7f707e32f640 1 --2- 172.21.15.16:0/4013563648 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f70780c9b40 0x7f707815a2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:10.904 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.897+0000 7f707e32f640 1 --2- 172.21.15.16:0/4013563648 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7078070df0 0x7f707815cc10 unknown :-1 s=CLOSED pgs=1666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:10.904 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.897+0000 7f707e32f640 1 -- 172.21.15.16:0/4013563648 >> 172.21.15.16:0/4013563648 conn(0x7f7078069720 msgr2=0x7f7078158010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:45:10.904 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.897+0000 7f707e32f640 1 -- 172.21.15.16:0/4013563648 shutdown_connections 2023-05-22T15:45:10.905 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:10.897+0000 7f707e32f640 1 -- 172.21.15.16:0/4013563648 wait complete. 2023-05-22T15:45:10.918 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-05-22T15:45:10.918 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-05-22T15:45:10.919 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi016.front.sepia.ceph.com 2023-05-22T15:45:10.919 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-05-22T15:45:10.919 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-05-22T15:45:10.919 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-05-22T15:45:10.920 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-05-22T15:45:10.920 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-05-22T15:45:10.921 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-05-22T15:45:10.921 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:45:10.921 DEBUG:teuthology.orchestra.run.smithi016:> ip addr 2023-05-22T15:45:10.927 INFO:teuthology.orchestra.run.smithi016.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-05-22T15:45:10.928 INFO:teuthology.orchestra.run.smithi016.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-05-22T15:45:10.928 INFO:teuthology.orchestra.run.smithi016.stdout: inet 127.0.0.1/8 scope host lo 2023-05-22T15:45:10.929 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-05-22T15:45:10.929 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 ::1/128 scope host 2023-05-22T15:45:10.929 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-05-22T15:45:10.929 INFO:teuthology.orchestra.run.smithi016.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-05-22T15:45:10.930 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:6b:fd:e0 brd ff:ff:ff:ff:ff:ff 2023-05-22T15:45:10.930 INFO:teuthology.orchestra.run.smithi016.stdout: altname enp7s0f0 2023-05-22T15:45:10.930 INFO:teuthology.orchestra.run.smithi016.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-05-22T15:45:10.930 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:6b:fd:e1 brd ff:ff:ff:ff:ff:ff 2023-05-22T15:45:10.930 INFO:teuthology.orchestra.run.smithi016.stdout: altname enp7s0f1 2023-05-22T15:45:10.930 INFO:teuthology.orchestra.run.smithi016.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2023-05-22T15:45:10.930 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:bd:15:64 brd ff:ff:ff:ff:ff:ff 2023-05-22T15:45:10.931 INFO:teuthology.orchestra.run.smithi016.stdout: altname enp1s0f0 2023-05-22T15:45:10.931 INFO:teuthology.orchestra.run.smithi016.stdout: inet 172.21.15.16/20 brd 172.21.15.255 scope global dynamic ens1f0 2023-05-22T15:45:10.931 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft 41405sec preferred_lft 41405sec 2023-05-22T15:45:10.932 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::ec4:7aff:febd:1564/64 scope link 2023-05-22T15:45:10.932 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-05-22T15:45:10.932 INFO:teuthology.orchestra.run.smithi016.stdout:5: ens1f1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-05-22T15:45:10.932 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:bd:15:65 brd ff:ff:ff:ff:ff:ff 2023-05-22T15:45:10.932 INFO:teuthology.orchestra.run.smithi016.stdout: altname enp1s0f1 2023-05-22T15:45:10.932 INFO:teuthology.orchestra.run.smithi016.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2023-05-22T15:45:10.932 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 02:42:1f:b8:98:0a brd ff:ff:ff:ff:ff:ff 2023-05-22T15:45:10.933 INFO:teuthology.orchestra.run.smithi016.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2023-05-22T15:45:10.933 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-05-22T15:45:10.933 INFO:teuthology.orchestra.run.smithi016.stdout:9: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-05-22T15:45:10.933 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 5e:1f:84:6d:79:ae brd ff:ff:ff:ff:ff:ff 2023-05-22T15:45:10.933 INFO:teuthology.orchestra.run.smithi016.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-05-22T15:45:10.933 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-05-22T15:45:10.933 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::5c1f:84ff:fe6d:79ae/64 scope link 2023-05-22T15:45:10.933 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-05-22T15:45:10.934 INFO:teuthology.orchestra.run.smithi016.stdout:10: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-05-22T15:45:10.934 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0e:5c:f6:1f:82:2f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-05-22T15:45:10.934 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::c5c:f6ff:fe1f:822f/64 scope link 2023-05-22T15:45:10.934 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-05-22T15:45:10.934 INFO:teuthology.orchestra.run.smithi016.stdout:11: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-05-22T15:45:10.934 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether c6:a6:3d:69:d0:db brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-05-22T15:45:10.934 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::c4a6:3dff:fe69:d0db/64 scope link 2023-05-22T15:45:10.935 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-05-22T15:45:10.935 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:45:10.935 DEBUG:teuthology.orchestra.run.smithi016:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-05-22T15:45:10.979 INFO:teuthology.orchestra.run.smithi016.stdout:1 2023-05-22T15:45:10.980 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:45:10.981 DEBUG:teuthology.orchestra.run.smithi016:> route 2023-05-22T15:45:11.028 INFO:teuthology.orchestra.run.smithi016.stdout:Kernel IP routing table 2023-05-22T15:45:11.028 INFO:teuthology.orchestra.run.smithi016.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-05-22T15:45:11.028 INFO:teuthology.orchestra.run.smithi016.stdout:default _gateway 0.0.0.0 UG 0 0 0 ens1f0 2023-05-22T15:45:11.028 INFO:teuthology.orchestra.run.smithi016.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2023-05-22T15:45:11.029 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 0 0 0 ens1f0 2023-05-22T15:45:11.029 INFO:teuthology.orchestra.run.smithi016.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-05-22T15:45:11.029 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:45:11.030 DEBUG:teuthology.orchestra.run.smithi016:> (cd / && exec bash -c ' 2023-05-22T15:45:11.030 DEBUG:teuthology.orchestra.run.smithi016:> set -e 2023-05-22T15:45:11.030 DEBUG:teuthology.orchestra.run.smithi016:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2023-05-22T15:45:11.031 DEBUG:teuthology.orchestra.run.smithi016:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2023-05-22T15:45:11.031 DEBUG:teuthology.orchestra.run.smithi016:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2023-05-22T15:45:11.031 DEBUG:teuthology.orchestra.run.smithi016:> ') 2023-05-22T15:45:11.098 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:45:11.099 DEBUG:teuthology.orchestra.run.smithi016:> ip netns list 2023-05-22T15:45:11.148 INFO:teuthology.orchestra.run.smithi016.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-05-22T15:45:11.148 INFO:teuthology.orchestra.run.smithi016.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-05-22T15:45:11.148 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-05-22T15:45:11.149 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:45:11.150 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip link set brx.0 up 2023-05-22T15:45:11.204 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2023-05-22T15:45:11.204 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-05-22T15:45:11.205 DEBUG:teuthology.orchestra.run.smithi016:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-05-22T15:45:11.251 INFO:teuthology.orchestra.run.smithi016.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-05-22T15:45:11.252 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-05-22T15:45:11.252 DEBUG:teuthology.orchestra.run.smithi016:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-05-22T15:45:11.299 DEBUG:teuthology.orchestra.run.smithi016:> sudo modprobe fuse 2023-05-22T15:45:11.357 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-05-22T15:45:11.358 DEBUG:teuthology.orchestra.run.smithi016:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-05-22T15:45:11.410 INFO:teuthology.orchestra.run.smithi016.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-05-22T15:45:11.411 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-05-22T15:45:11.412 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:45:11.412 DEBUG:teuthology.orchestra.run.smithi016:> ls /sys/fs/fuse/connections 2023-05-22T15:45:11.459 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2023-05-22T15:45:11.459 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2023-05-22T15:45:11.509 DEBUG:teuthology.orchestra.run.smithi016:> sudo modprobe fuse 2023-05-22T15:45:11.524 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-05-22T15:45:11.526 DEBUG:teuthology.orchestra.run.smithi016:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-05-22T15:45:11.576 INFO:teuthology.orchestra.run.smithi016.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-05-22T15:45:11.577 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-05-22T15:45:11.578 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:45:11.578 DEBUG:teuthology.orchestra.run.smithi016:> ls /sys/fs/fuse/connections 2023-05-22T15:45:11.594 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi016.stderr:2023-05-22T15:45:11.589+0000 7f6e5b9b0580 -1 init, newargv = 0x55ca3d09cd50 newargc=15 2023-05-22T15:45:11.594 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi016.stderr:ceph-fuse[61146]: starting ceph client 2023-05-22T15:45:11.598 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi016.stderr:ceph-fuse[61146]: starting fuse 2023-05-22T15:45:12.700 DEBUG:teuthology.orchestra.run.smithi016:> sudo modprobe fuse 2023-05-22T15:45:12.716 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-05-22T15:45:12.717 DEBUG:teuthology.orchestra.run.smithi016:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-05-22T15:45:12.771 INFO:teuthology.orchestra.run.smithi016.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-05-22T15:45:12.772 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-05-22T15:45:12.773 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:45:12.773 DEBUG:teuthology.orchestra.run.smithi016:> ls /sys/fs/fuse/connections 2023-05-22T15:45:12.822 INFO:teuthology.orchestra.run.smithi016.stdout:45 2023-05-22T15:45:12.824 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [45] 2023-05-22T15:45:12.825 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2023-05-22T15:45:12.826 DEBUG:teuthology.orchestra.run.smithi016:> import glob 2023-05-22T15:45:12.826 DEBUG:teuthology.orchestra.run.smithi016:> import re 2023-05-22T15:45:12.827 DEBUG:teuthology.orchestra.run.smithi016:> import os 2023-05-22T15:45:12.828 DEBUG:teuthology.orchestra.run.smithi016:> import subprocess 2023-05-22T15:45:12.829 DEBUG:teuthology.orchestra.run.smithi016:> 2023-05-22T15:45:12.829 DEBUG:teuthology.orchestra.run.smithi016:> def _find_admin_socket(client_name): 2023-05-22T15:45:12.830 DEBUG:teuthology.orchestra.run.smithi016:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2023-05-22T15:45:12.831 DEBUG:teuthology.orchestra.run.smithi016:> files = glob.glob(asok_path) 2023-05-22T15:45:12.831 DEBUG:teuthology.orchestra.run.smithi016:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2023-05-22T15:45:12.832 DEBUG:teuthology.orchestra.run.smithi016:> 2023-05-22T15:45:12.833 DEBUG:teuthology.orchestra.run.smithi016:> # Given a non-glob path, it better be there 2023-05-22T15:45:12.833 DEBUG:teuthology.orchestra.run.smithi016:> if "*" not in asok_path: 2023-05-22T15:45:12.834 DEBUG:teuthology.orchestra.run.smithi016:> assert(len(files) == 1) 2023-05-22T15:45:12.835 DEBUG:teuthology.orchestra.run.smithi016:> return files[0] 2023-05-22T15:45:12.835 DEBUG:teuthology.orchestra.run.smithi016:> 2023-05-22T15:45:12.836 DEBUG:teuthology.orchestra.run.smithi016:> for f in files: 2023-05-22T15:45:12.836 DEBUG:teuthology.orchestra.run.smithi016:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-05-22T15:45:12.837 DEBUG:teuthology.orchestra.run.smithi016:> if os.path.exists("/proc/{0}".format(pid)): 2023-05-22T15:45:12.838 DEBUG:teuthology.orchestra.run.smithi016:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-05-22T15:45:12.838 DEBUG:teuthology.orchestra.run.smithi016:> contents = proc_f.read() 2023-05-22T15:45:12.839 DEBUG:teuthology.orchestra.run.smithi016:> if mountpoint in contents: 2023-05-22T15:45:12.839 DEBUG:teuthology.orchestra.run.smithi016:> return f 2023-05-22T15:45:12.840 DEBUG:teuthology.orchestra.run.smithi016:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-05-22T15:45:12.840 DEBUG:teuthology.orchestra.run.smithi016:> 2023-05-22T15:45:12.840 DEBUG:teuthology.orchestra.run.smithi016:> print(_find_admin_socket("client.0")) 2023-05-22T15:45:12.841 DEBUG:teuthology.orchestra.run.smithi016:> ' 2023-05-22T15:45:12.963 INFO:teuthology.orchestra.run.smithi016.stdout:/var/run/ceph/ceph-client.0.61146.asok 2023-05-22T15:45:13.143 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.61146.asok 2023-05-22T15:45:13.144 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:45:13.145 DEBUG:teuthology.orchestra.run.smithi016:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.61146.asok status 2023-05-22T15:45:13.309 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-05-22T15:45:13.309 INFO:teuthology.orchestra.run.smithi016.stdout: "metadata": { 2023-05-22T15:45:13.309 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph_sha1": "9c9ba4df22024d9d8d6cf9f98a4b96751fc78d0c", 2023-05-22T15:45:13.309 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph_version": "ceph version 18.0.0-3192-g9c9ba4df (9c9ba4df22024d9d8d6cf9f98a4b96751fc78d0c) reef (dev)", 2023-05-22T15:45:13.309 INFO:teuthology.orchestra.run.smithi016.stdout: "entity_id": "0", 2023-05-22T15:45:13.310 INFO:teuthology.orchestra.run.smithi016.stdout: "hostname": "smithi016", 2023-05-22T15:45:13.310 INFO:teuthology.orchestra.run.smithi016.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2023-05-22T15:45:13.310 INFO:teuthology.orchestra.run.smithi016.stdout: "pid": "61146", 2023-05-22T15:45:13.310 INFO:teuthology.orchestra.run.smithi016.stdout: "root": "/" 2023-05-22T15:45:13.310 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:45:13.310 INFO:teuthology.orchestra.run.smithi016.stdout: "dentry_count": 0, 2023-05-22T15:45:13.310 INFO:teuthology.orchestra.run.smithi016.stdout: "dentry_pinned_count": 0, 2023-05-22T15:45:13.311 INFO:teuthology.orchestra.run.smithi016.stdout: "id": 8712, 2023-05-22T15:45:13.311 INFO:teuthology.orchestra.run.smithi016.stdout: "inst": { 2023-05-22T15:45:13.311 INFO:teuthology.orchestra.run.smithi016.stdout: "name": { 2023-05-22T15:45:13.311 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "client", 2023-05-22T15:45:13.311 INFO:teuthology.orchestra.run.smithi016.stdout: "num": 8712 2023-05-22T15:45:13.311 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:45:13.311 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": { 2023-05-22T15:45:13.312 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-05-22T15:45:13.312 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "192.168.0.1:0", 2023-05-22T15:45:13.312 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2383605255 2023-05-22T15:45:13.312 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T15:45:13.312 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:45:13.312 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": { 2023-05-22T15:45:13.312 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-05-22T15:45:13.313 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "192.168.0.1:0", 2023-05-22T15:45:13.313 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2383605255 2023-05-22T15:45:13.313 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:45:13.313 INFO:teuthology.orchestra.run.smithi016.stdout: "inst_str": "client.8712 192.168.0.1:0/2383605255", 2023-05-22T15:45:13.313 INFO:teuthology.orchestra.run.smithi016.stdout: "addr_str": "192.168.0.1:0/2383605255", 2023-05-22T15:45:13.313 INFO:teuthology.orchestra.run.smithi016.stdout: "inode_count": 1, 2023-05-22T15:45:13.313 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_epoch": 50, 2023-05-22T15:45:13.314 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_epoch": 132, 2023-05-22T15:45:13.314 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_epoch_barrier": 0, 2023-05-22T15:45:13.314 INFO:teuthology.orchestra.run.smithi016.stdout: "blocklisted": false, 2023-05-22T15:45:13.314 INFO:teuthology.orchestra.run.smithi016.stdout: "fs_name": "cephfs" 2023-05-22T15:45:13.314 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-05-22T15:45:13.329 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:45:13.330 DEBUG:teuthology.orchestra.run.smithi016:> stat --file-system '--printf=%T 2023-05-22T15:45:13.331 DEBUG:teuthology.orchestra.run.smithi016:> ' -- /home/ubuntu/cephtest/mnt.0 2023-05-22T15:45:13.337 INFO:teuthology.orchestra.run.smithi016.stdout:fuseblk 2023-05-22T15:45:13.338 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2023-05-22T15:45:13.338 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:45:13.339 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-05-22T15:45:13.392 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2023-05-22T15:45:13.567 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:13.565+0000 7f82ee6ae640 1 Processor -- start 2023-05-22T15:45:13.567 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:13.565+0000 7f82ee6ae640 1 -- start start 2023-05-22T15:45:13.569 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:13.565+0000 7f82ee6ae640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f82e80cbf80 0x7f82e806b210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:13.569 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:13.565+0000 7f82ee6ae640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f82e80b22e0 0x7f82e806bec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:13.569 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:13.565+0000 7f82ee6ae640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f82e806c400 0x7f82e815f810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:13.570 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:13.565+0000 7f82ee6ae640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82e806c880 con 0x7f82e80cbf80 2023-05-22T15:45:13.570 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:13.565+0000 7f82ee6ae640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82e806db40 con 0x7f82e806c400 2023-05-22T15:45:13.570 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:13.565+0000 7f82ee6ae640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82e80cf3f0 con 0x7f82e80b22e0 2023-05-22T15:45:13.570 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:13.565+0000 7f82e7fff640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f82e80cbf80 0x7f82e806b210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:13.570 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:13.565+0000 7f82e7fff640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f82e80cbf80 0x7f82e806b210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36792/0 (socket says 172.21.15.16:36792) 2023-05-22T15:45:13.570 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:13.565+0000 7f82e7fff640 1 -- 172.21.15.16:0/2195057410 learned_addr learned my addr 172.21.15.16:0/2195057410 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:45:13.570 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:13.565+0000 7f82ecc4b640 1 --2- 172.21.15.16:0/2195057410 >> v2:172.21.15.62:3300/0 conn(0x7f82e806c400 0x7f82e815f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:13.571 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:13.565+0000 7f82e77fe640 1 --2- 172.21.15.16:0/2195057410 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f82e80b22e0 0x7f82e806bec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:13.571 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:13.565+0000 7f82e7fff640 1 -- 172.21.15.16:0/2195057410 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f82e80b22e0 msgr2=0x7f82e806bec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:45:13.571 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:13.565+0000 7f82e7fff640 1 --2- 172.21.15.16:0/2195057410 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f82e80b22e0 0x7f82e806bec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:13.571 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:13.565+0000 7f82e7fff640 1 -- 172.21.15.16:0/2195057410 >> v2:172.21.15.62:3300/0 conn(0x7f82e806c400 msgr2=0x7f82e815f810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:45:13.572 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:13.565+0000 7f82e7fff640 1 --2- 172.21.15.16:0/2195057410 >> v2:172.21.15.62:3300/0 conn(0x7f82e806c400 0x7f82e815f810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:13.572 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:13.565+0000 7f82e7fff640 1 -- 172.21.15.16:0/2195057410 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82e815c080 con 0x7f82e80cbf80 2023-05-22T15:45:13.573 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:13.569+0000 7f82e7fff640 1 --2- 172.21.15.16:0/2195057410 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82e80cbf80 0x7f82e806b210 secure :-1 s=READY pgs=1670 cs=0 l=1 rev1=1 crypto rx=0x7f82e80cec50 tx=0x7f82d801c210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3637e4fe465727fb server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:45:13.573 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:13.569+0000 7f82e6ffd640 1 -- 172.21.15.16:0/2195057410 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82d80ab600 con 0x7f82e80cbf80 2023-05-22T15:45:13.573 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:13.569+0000 7f82e6ffd640 1 -- 172.21.15.16:0/2195057410 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82d80a7050 con 0x7f82e80cbf80 2023-05-22T15:45:13.573 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:13.569+0000 7f82e6ffd640 1 -- 172.21.15.16:0/2195057410 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82d80ab980 con 0x7f82e80cbf80 2023-05-22T15:45:13.574 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:13.569+0000 7f82ee6ae640 1 -- 172.21.15.16:0/2195057410 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82e80cbf80 msgr2=0x7f82e806b210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:45:13.574 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:13.569+0000 7f82ee6ae640 1 --2- 172.21.15.16:0/2195057410 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82e80cbf80 0x7f82e806b210 secure :-1 s=READY pgs=1670 cs=0 l=1 rev1=1 crypto rx=0x7f82e80cec50 tx=0x7f82d801c210 comp rx=0 tx=0).stop 2023-05-22T15:45:13.574 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:13.569+0000 7f82ee6ae640 1 -- 172.21.15.16:0/2195057410 shutdown_connections 2023-05-22T15:45:13.574 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:13.569+0000 7f82ee6ae640 1 --2- 172.21.15.16:0/2195057410 >> v2:172.21.15.62:3300/0 conn(0x7f82e806c400 0x7f82e815f810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:13.574 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:13.569+0000 7f82ee6ae640 1 --2- 172.21.15.16:0/2195057410 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f82e80b22e0 0x7f82e806bec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:13.574 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:13.569+0000 7f82ee6ae640 1 --2- 172.21.15.16:0/2195057410 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82e80cbf80 0x7f82e806b210 unknown :-1 s=CLOSED pgs=1670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:13.575 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:13.569+0000 7f82ee6ae640 1 -- 172.21.15.16:0/2195057410 >> 172.21.15.16:0/2195057410 conn(0x7f82e806f560 msgr2=0x7f82e806f960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:45:13.575 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:13.569+0000 7f82ee6ae640 1 -- 172.21.15.16:0/2195057410 shutdown_connections 2023-05-22T15:45:13.575 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:13.569+0000 7f82ee6ae640 1 -- 172.21.15.16:0/2195057410 wait complete. 2023-05-22T15:45:13.575 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:13.569+0000 7f82ee6ae640 1 Processor -- start 2023-05-22T15:45:13.575 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:13.569+0000 7f82ee6ae640 1 -- start start 2023-05-22T15:45:13.577 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:13.569+0000 7f82ee6ae640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f82e806c400 0x7f82e80b3c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:13.578 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:13.569+0000 7f82ee6ae640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f82e80b22e0 0x7f82e8155ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:13.578 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:13.569+0000 7f82ee6ae640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82e80cbf80 0x7f82e80b43e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:13.578 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:13.569+0000 7f82ee6ae640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82e80b4150 con 0x7f82e80cbf80 2023-05-22T15:45:13.578 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:13.569+0000 7f82ee6ae640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82e80b4f50 con 0x7f82e80b22e0 2023-05-22T15:45:13.578 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:13.569+0000 7f82ee6ae640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82e80b3130 con 0x7f82e806c400 2023-05-22T15:45:13.578 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:13.569+0000 7f82ecc4b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82e80cbf80 0x7f82e80b43e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:13.579 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:13.569+0000 7f82ecc4b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82e80cbf80 0x7f82e80b43e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36804/0 (socket says 172.21.15.16:36804) 2023-05-22T15:45:13.579 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:13.569+0000 7f82ecc4b640 1 -- 172.21.15.16:0/4168632212 learned_addr learned my addr 172.21.15.16:0/4168632212 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:45:13.579 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:13.569+0000 7f82e77fe640 1 --2- 172.21.15.16:0/4168632212 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f82e80b22e0 0x7f82e8155ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:13.579 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:13.569+0000 7f82e7fff640 1 --2- 172.21.15.16:0/4168632212 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f82e806c400 0x7f82e80b3c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:13.579 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:13.573+0000 7f82ecc4b640 1 -- 172.21.15.16:0/4168632212 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f82e806c400 msgr2=0x7f82e80b3c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:45:13.579 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:13.573+0000 7f82ecc4b640 1 --2- 172.21.15.16:0/4168632212 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f82e806c400 0x7f82e80b3c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:13.579 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:13.573+0000 7f82ecc4b640 1 -- 172.21.15.16:0/4168632212 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f82e80b22e0 msgr2=0x7f82e8155ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:45:13.580 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:13.573+0000 7f82ecc4b640 1 --2- 172.21.15.16:0/4168632212 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f82e80b22e0 0x7f82e8155ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:13.580 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:13.573+0000 7f82ecc4b640 1 -- 172.21.15.16:0/4168632212 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82d80094d0 con 0x7f82e80cbf80 2023-05-22T15:45:13.580 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:13.573+0000 7f82ecc4b640 1 --2- 172.21.15.16:0/4168632212 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82e80cbf80 0x7f82e80b43e0 secure :-1 s=READY pgs=1671 cs=0 l=1 rev1=1 crypto rx=0x7f82e0007e40 tx=0x7f82e000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:45:13.580 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:13.573+0000 7f82e4ff9640 1 -- 172.21.15.16:0/4168632212 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82e0014620 con 0x7f82e80cbf80 2023-05-22T15:45:13.580 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:13.573+0000 7f82e4ff9640 1 -- 172.21.15.16:0/4168632212 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82e000ee90 con 0x7f82e80cbf80 2023-05-22T15:45:13.580 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:13.573+0000 7f82ee6ae640 1 -- 172.21.15.16:0/4168632212 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82e806b750 con 0x7f82e80cbf80 2023-05-22T15:45:13.580 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:13.573+0000 7f82e4ff9640 1 -- 172.21.15.16:0/4168632212 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82e00149e0 con 0x7f82e80cbf80 2023-05-22T15:45:13.581 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:13.573+0000 7f82ee6ae640 1 -- 172.21.15.16:0/4168632212 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82e806e380 con 0x7f82e80cbf80 2023-05-22T15:45:13.581 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:13.573+0000 7f82ee6ae640 1 -- 172.21.15.16:0/4168632212 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82e80cc350 con 0x7f82e80cbf80 2023-05-22T15:45:13.581 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:13.573+0000 7f82e4ff9640 1 -- 172.21.15.16:0/4168632212 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 50) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f82e0014b40 con 0x7f82e80cbf80 2023-05-22T15:45:13.581 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:13.573+0000 7f82e4ff9640 1 --2- 172.21.15.16:0/4168632212 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f82d40654d0 0x7f82d4067980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:13.581 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:13.573+0000 7f82e4ff9640 1 -- 172.21.15.16:0/4168632212 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f82e00833d0 con 0x7f82e80cbf80 2023-05-22T15:45:13.581 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:13.573+0000 7f82e7fff640 1 --2- 172.21.15.16:0/4168632212 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f82d40654d0 0x7f82d4067980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:13.582 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:13.573+0000 7f82e7fff640 1 --2- 172.21.15.16:0/4168632212 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f82d40654d0 0x7f82d4067980 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f82d80ad000 tx=0x7f82d80a6000 comp rx=0 tx=0).ready entity=mgr.8304 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:45:13.582 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:13.577+0000 7f82e4ff9640 1 -- 172.21.15.16:0/4168632212 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f82e0050e00 con 0x7f82e80cbf80 2023-05-22T15:45:13.756 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:13.753+0000 7f82ee6ae640 1 -- 172.21.15.16:0/4168632212 --> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f82e804dbe0 con 0x7f82d40654d0 2023-05-22T15:45:13.759 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:13.753+0000 7f82e4ff9640 1 -- 172.21.15.16:0/4168632212 <== mgr.8304 v2:172.21.15.16:6832/36921 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f82e804dbe0 con 0x7f82d40654d0 2023-05-22T15:45:13.759 INFO:teuthology.orchestra.run.smithi016.stdout:[ 2023-05-22T15:45:13.760 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T15:45:13.760 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "cephfs" 2023-05-22T15:45:13.760 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T15:45:13.760 INFO:teuthology.orchestra.run.smithi016.stdout:] 2023-05-22T15:45:13.761 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:13.757+0000 7f82ee6ae640 1 -- 172.21.15.16:0/4168632212 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f82d40654d0 msgr2=0x7f82d4067980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:45:13.761 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:13.757+0000 7f82ee6ae640 1 --2- 172.21.15.16:0/4168632212 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f82d40654d0 0x7f82d4067980 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f82d80ad000 tx=0x7f82d80a6000 comp rx=0 tx=0).stop 2023-05-22T15:45:13.761 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:13.757+0000 7f82ee6ae640 1 -- 172.21.15.16:0/4168632212 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82e80cbf80 msgr2=0x7f82e80b43e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:45:13.761 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:13.757+0000 7f82ee6ae640 1 --2- 172.21.15.16:0/4168632212 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82e80cbf80 0x7f82e80b43e0 secure :-1 s=READY pgs=1671 cs=0 l=1 rev1=1 crypto rx=0x7f82e0007e40 tx=0x7f82e000e4b0 comp rx=0 tx=0).stop 2023-05-22T15:45:13.762 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:13.757+0000 7f82ee6ae640 1 -- 172.21.15.16:0/4168632212 shutdown_connections 2023-05-22T15:45:13.762 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:13.757+0000 7f82ee6ae640 1 --2- 172.21.15.16:0/4168632212 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82e80cbf80 0x7f82e80b43e0 unknown :-1 s=CLOSED pgs=1671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:13.762 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:13.757+0000 7f82ee6ae640 1 --2- 172.21.15.16:0/4168632212 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f82e80b22e0 0x7f82e8155ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:13.762 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:13.757+0000 7f82ee6ae640 1 --2- 172.21.15.16:0/4168632212 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f82d40654d0 0x7f82d4067980 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:13.762 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:13.757+0000 7f82ee6ae640 1 --2- 172.21.15.16:0/4168632212 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f82e806c400 0x7f82e80b3c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:13.762 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:13.757+0000 7f82ee6ae640 1 -- 172.21.15.16:0/4168632212 >> 172.21.15.16:0/4168632212 conn(0x7f82e806f560 msgr2=0x7f82e80ca610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:45:13.762 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:13.757+0000 7f82ee6ae640 1 -- 172.21.15.16:0/4168632212 shutdown_connections 2023-05-22T15:45:13.763 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:13.757+0000 7f82ee6ae640 1 -- 172.21.15.16:0/4168632212 wait complete. 2023-05-22T15:45:13.779 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2023-05-22T15:45:13.896 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:13.893+0000 7fc962c14640 1 Processor -- start 2023-05-22T15:45:13.896 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:13.893+0000 7fc962c14640 1 -- start start 2023-05-22T15:45:13.897 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:13.893+0000 7fc962c14640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fc95c0cd770 0x7fc95c0cbb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:13.897 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:13.893+0000 7fc962c14640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc95c0cdd90 0x7fc95c0ce1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:13.897 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:13.893+0000 7fc962c14640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc95c159e60 con 0x7fc95c159a90 2023-05-22T15:45:13.897 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:13.893+0000 7fc962c14640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc95c15d4f0 con 0x7fc95c0cd770 2023-05-22T15:45:13.898 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:13.893+0000 7fc962c14640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc95c0ce6f0 con 0x7fc95c0cdd90 2023-05-22T15:45:13.898 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:13.893+0000 7fc9609b0640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fc95c159a90 0x7fc95c15dbf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:42238/0 (socket says 172.21.15.16:42238) 2023-05-22T15:45:13.898 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:13.893+0000 7fc9609b0640 1 -- 172.21.15.16:0/726671382 learned_addr learned my addr 172.21.15.16:0/726671382 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:45:13.898 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:13.893+0000 7fc95bfff640 1 --2- 172.21.15.16:0/726671382 >> v2:172.21.15.62:3300/0 conn(0x7fc95c0cd770 0x7fc95c0cbb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:13.898 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:13.893+0000 7fc95b7fe640 1 -- 172.21.15.16:0/726671382 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3153041103 0 0) 0x7fc95c159e60 con 0x7fc95c159a90 2023-05-22T15:45:13.898 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:13.893+0000 7fc9611b1640 1 --2- 172.21.15.16:0/726671382 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc95c0cdd90 0x7fc95c0ce1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:13.899 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:13.893+0000 7fc95b7fe640 1 -- 172.21.15.16:0/726671382 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc944003530 con 0x7fc95c159a90 2023-05-22T15:45:13.900 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:13.893+0000 7fc95b7fe640 1 -- 172.21.15.16:0/726671382 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1377140884 0 0) 0x7fc944003530 con 0x7fc95c159a90 2023-05-22T15:45:13.900 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:13.893+0000 7fc95b7fe640 1 -- 172.21.15.16:0/726671382 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc95c0cdd90 msgr2=0x7fc95c0ce1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:45:13.900 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:13.893+0000 7fc95b7fe640 1 --2- 172.21.15.16:0/726671382 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc95c0cdd90 0x7fc95c0ce1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:13.901 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:13.893+0000 7fc95b7fe640 1 -- 172.21.15.16:0/726671382 >> v2:172.21.15.62:3300/0 conn(0x7fc95c0cd770 msgr2=0x7fc95c0cbb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:45:13.901 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:13.893+0000 7fc95b7fe640 1 --2- 172.21.15.16:0/726671382 >> v2:172.21.15.62:3300/0 conn(0x7fc95c0cd770 0x7fc95c0cbb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:13.901 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:13.893+0000 7fc95b7fe640 1 -- 172.21.15.16:0/726671382 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc95c0cc1d0 con 0x7fc95c159a90 2023-05-22T15:45:13.901 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:13.893+0000 7fc95b7fe640 1 -- 172.21.15.16:0/726671382 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7fc94c0016a0 con 0x7fc95c159a90 2023-05-22T15:45:13.901 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:13.893+0000 7fc95b7fe640 1 -- 172.21.15.16:0/726671382 >> v1:172.21.15.16:6789/0 conn(0x7fc95c159a90 legacy=0x7fc95c15dbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:45:13.901 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:13.893+0000 7fc95b7fe640 1 --2- 172.21.15.16:0/726671382 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc9440039b0 0x7fc944003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:13.902 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:13.893+0000 7fc95b7fe640 1 --2- 172.21.15.16:0/726671382 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc95c0cdd90 0x7fc9440047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:13.902 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:13.893+0000 7fc95b7fe640 1 --2- 172.21.15.16:0/726671382 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc944004d10 0x7fc944005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:13.902 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:13.893+0000 7fc95b7fe640 1 -- 172.21.15.16:0/726671382 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc95c15d4f0 con 0x7fc9440039b0 2023-05-22T15:45:13.902 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:13.893+0000 7fc95b7fe640 1 -- 172.21.15.16:0/726671382 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc95c0ce6f0 con 0x7fc944004d10 2023-05-22T15:45:13.902 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:13.893+0000 7fc95b7fe640 1 -- 172.21.15.16:0/726671382 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc94400b9c0 con 0x7fc95c0cdd90 2023-05-22T15:45:13.903 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:13.893+0000 7fc9609b0640 1 --2- 172.21.15.16:0/726671382 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc9440039b0 0x7fc944003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:13.903 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:13.893+0000 7fc9611b1640 1 --2- 172.21.15.16:0/726671382 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc944004d10 0x7fc944005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:13.903 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:13.893+0000 7fc95bfff640 1 --2- 172.21.15.16:0/726671382 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc95c0cdd90 0x7fc9440047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:13.903 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:13.893+0000 7fc9609b0640 1 -- 172.21.15.16:0/726671382 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc95c0cdd90 msgr2=0x7fc9440047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:45:13.903 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:13.893+0000 7fc9609b0640 1 --2- 172.21.15.16:0/726671382 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc95c0cdd90 0x7fc9440047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:13.903 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:13.893+0000 7fc9609b0640 1 -- 172.21.15.16:0/726671382 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc944004d10 msgr2=0x7fc944005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:45:13.903 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:13.893+0000 7fc9609b0640 1 --2- 172.21.15.16:0/726671382 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc944004d10 0x7fc944005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:13.904 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:13.893+0000 7fc9609b0640 1 -- 172.21.15.16:0/726671382 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc944003530 con 0x7fc9440039b0 2023-05-22T15:45:13.904 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:13.893+0000 7fc9611b1640 1 --2- 172.21.15.16:0/726671382 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc944004d10 0x7fc944005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T15:45:13.904 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:13.893+0000 7fc95bfff640 1 --2- 172.21.15.16:0/726671382 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc95c0cdd90 0x7fc9440047d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:45:13.905 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:13.901+0000 7fc9609b0640 1 --2- 172.21.15.16:0/726671382 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc9440039b0 0x7fc944003dc0 secure :-1 s=READY pgs=1673 cs=0 l=1 rev1=1 crypto rx=0x7fc95c0b6220 tx=0x7fc94c01cbd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=464b4ccc5a11e5ab server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:45:13.905 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:13.901+0000 7fc95b7fe640 1 -- 172.21.15.16:0/726671382 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc94c005ae0 con 0x7fc9440039b0 2023-05-22T15:45:13.905 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:13.901+0000 7fc95b7fe640 1 -- 172.21.15.16:0/726671382 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc94c005c40 con 0x7fc9440039b0 2023-05-22T15:45:13.905 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:13.901+0000 7fc95b7fe640 1 -- 172.21.15.16:0/726671382 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc94c0a57c0 con 0x7fc9440039b0 2023-05-22T15:45:13.906 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:13.901+0000 7fc962c14640 1 -- 172.21.15.16:0/726671382 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc9440039b0 msgr2=0x7fc944003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:45:13.906 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:13.901+0000 7fc962c14640 1 --2- 172.21.15.16:0/726671382 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc9440039b0 0x7fc944003dc0 secure :-1 s=READY pgs=1673 cs=0 l=1 rev1=1 crypto rx=0x7fc95c0b6220 tx=0x7fc94c01cbd0 comp rx=0 tx=0).stop 2023-05-22T15:45:13.906 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:13.901+0000 7fc9611b1640 1 -- 172.21.15.16:0/726671382 reap_dead start 2023-05-22T15:45:13.906 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:13.901+0000 7fc962c14640 1 -- 172.21.15.16:0/726671382 shutdown_connections 2023-05-22T15:45:13.906 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:13.901+0000 7fc962c14640 1 -- 172.21.15.16:0/726671382 >> 172.21.15.16:0/726671382 conn(0x7fc95c06f4f0 msgr2=0x7fc95c06f8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:45:13.906 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:13.901+0000 7fc962c14640 1 -- 172.21.15.16:0/726671382 shutdown_connections 2023-05-22T15:45:13.906 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:13.901+0000 7fc962c14640 1 -- 172.21.15.16:0/726671382 wait complete. 2023-05-22T15:45:13.908 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:13.901+0000 7fc962c14640 1 Processor -- start 2023-05-22T15:45:13.908 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:13.901+0000 7fc962c14640 1 -- start start 2023-05-22T15:45:13.908 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:13.905+0000 7fc962c14640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc95c0c6190 0x7fc95c0b26c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:13.908 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:13.905+0000 7fc962c14640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc95c0afc40 0x7fc95c150c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:13.908 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:13.905+0000 7fc962c14640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc95c0c9440 0x7fc95c0c98b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:13.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:13.905+0000 7fc962c14640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc95c0b2c00 con 0x7fc95c0c9440 2023-05-22T15:45:13.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:13.905+0000 7fc962c14640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc95c0c9df0 con 0x7fc95c0afc40 2023-05-22T15:45:13.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:13.905+0000 7fc962c14640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc95c0b01b0 con 0x7fc95c0c6190 2023-05-22T15:45:13.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:13.905+0000 7fc9611b1640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc95c0c9440 0x7fc95c0c98b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:13.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:13.905+0000 7fc9611b1640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc95c0c9440 0x7fc95c0c98b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36812/0 (socket says 172.21.15.16:36812) 2023-05-22T15:45:13.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:13.905+0000 7fc9611b1640 1 -- 172.21.15.16:0/841685254 learned_addr learned my addr 172.21.15.16:0/841685254 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:45:13.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:13.905+0000 7fc95bfff640 1 --2- 172.21.15.16:0/841685254 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc95c0afc40 0x7fc95c150c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:13.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:13.905+0000 7fc9609b0640 1 --2- 172.21.15.16:0/841685254 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc95c0c6190 0x7fc95c0b26c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:13.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:13.905+0000 7fc9611b1640 1 -- 172.21.15.16:0/841685254 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc95c0c6190 msgr2=0x7fc95c0b26c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:45:13.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:13.905+0000 7fc9611b1640 1 --2- 172.21.15.16:0/841685254 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc95c0c6190 0x7fc95c0b26c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:13.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:13.905+0000 7fc9611b1640 1 -- 172.21.15.16:0/841685254 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc95c0afc40 msgr2=0x7fc95c150c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:45:13.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:13.905+0000 7fc9611b1640 1 --2- 172.21.15.16:0/841685254 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc95c0afc40 0x7fc95c150c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:13.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:13.905+0000 7fc9611b1640 1 -- 172.21.15.16:0/841685254 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc95c0cd2e0 con 0x7fc95c0c9440 2023-05-22T15:45:13.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:13.905+0000 7fc9611b1640 1 --2- 172.21.15.16:0/841685254 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc95c0c9440 0x7fc95c0c98b0 secure :-1 s=READY pgs=1674 cs=0 l=1 rev1=1 crypto rx=0x7fc95c0d0120 tx=0x7fc954007db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:45:13.911 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:13.905+0000 7fc9597fa640 1 -- 172.21.15.16:0/841685254 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc95400e070 con 0x7fc95c0c9440 2023-05-22T15:45:13.911 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:13.905+0000 7fc9597fa640 1 -- 172.21.15.16:0/841685254 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc95400f040 con 0x7fc95c0c9440 2023-05-22T15:45:13.911 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:13.905+0000 7fc962c14640 1 -- 172.21.15.16:0/841685254 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc95c15d6d0 con 0x7fc95c0c9440 2023-05-22T15:45:13.911 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:13.905+0000 7fc9597fa640 1 -- 172.21.15.16:0/841685254 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc954003bc0 con 0x7fc95c0c9440 2023-05-22T15:45:13.911 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:13.905+0000 7fc962c14640 1 -- 172.21.15.16:0/841685254 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc95c15e300 con 0x7fc95c0c9440 2023-05-22T15:45:13.912 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:13.905+0000 7fc962c14640 1 -- 172.21.15.16:0/841685254 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc95c04dbe0 con 0x7fc95c0c9440 2023-05-22T15:45:13.912 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:13.905+0000 7fc9597fa640 1 -- 172.21.15.16:0/841685254 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 50) v1 ==== 84332+0+0 (secure 0 0 0) 0x7fc954015650 con 0x7fc95c0c9440 2023-05-22T15:45:13.912 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:13.905+0000 7fc9597fa640 1 --2- 172.21.15.16:0/841685254 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fc9480654a0 0x7fc948067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:13.912 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:13.905+0000 7fc9597fa640 1 -- 172.21.15.16:0/841685254 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 6382+0+0 (secure 0 0 0) 0x7fc9540827b0 con 0x7fc95c0c9440 2023-05-22T15:45:13.912 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:13.905+0000 7fc9609b0640 1 --2- 172.21.15.16:0/841685254 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fc9480654a0 0x7fc948067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:13.913 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:13.905+0000 7fc9609b0640 1 --2- 172.21.15.16:0/841685254 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fc9480654a0 0x7fc948067950 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fc94c0ab000 tx=0x7fc94c0a7070 comp rx=0 tx=0).ready entity=mgr.8304 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:45:13.913 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:13.909+0000 7fc9597fa640 1 -- 172.21.15.16:0/841685254 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fc954053a70 con 0x7fc95c0c9440 2023-05-22T15:45:14.103 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.101+0000 7fc962c14640 1 -- 172.21.15.16:0/841685254 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7fc95c0af9e0 con 0x7fc95c0c9440 2023-05-22T15:45:14.168 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.165+0000 7fc9597fa640 1 -- 172.21.15.16:0/841685254 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mgrmap(e 51) v1 ==== 84349+0+0 (secure 0 0 0) 0x7fc954050410 con 0x7fc95c0c9440 2023-05-22T15:45:14.169 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.165+0000 7fc9597fa640 1 -- 172.21.15.16:0/841685254 <== mon.0 v2:172.21.15.16:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v51) v1 ==== 92+0+0 (secure 0 0 0) 0x7fc954053a70 con 0x7fc95c0c9440 2023-05-22T15:45:14.172 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.169+0000 7fc962c14640 1 -- 172.21.15.16:0/841685254 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fc9480654a0 msgr2=0x7fc948067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:45:14.173 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.169+0000 7fc962c14640 1 --2- 172.21.15.16:0/841685254 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fc9480654a0 0x7fc948067950 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fc94c0ab000 tx=0x7fc94c0a7070 comp rx=0 tx=0).stop 2023-05-22T15:45:14.173 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.169+0000 7fc962c14640 1 -- 172.21.15.16:0/841685254 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc95c0c9440 msgr2=0x7fc95c0c98b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:45:14.173 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.169+0000 7fc962c14640 1 --2- 172.21.15.16:0/841685254 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc95c0c9440 0x7fc95c0c98b0 secure :-1 s=READY pgs=1674 cs=0 l=1 rev1=1 crypto rx=0x7fc95c0d0120 tx=0x7fc954007db0 comp rx=0 tx=0).stop 2023-05-22T15:45:14.173 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.169+0000 7fc962c14640 1 -- 172.21.15.16:0/841685254 shutdown_connections 2023-05-22T15:45:14.173 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.169+0000 7fc962c14640 1 --2- 172.21.15.16:0/841685254 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc95c0c9440 0x7fc95c0c98b0 unknown :-1 s=CLOSED pgs=1674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:14.173 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.169+0000 7fc962c14640 1 --2- 172.21.15.16:0/841685254 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc95c0afc40 0x7fc95c150c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:14.173 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.169+0000 7fc962c14640 1 --2- 172.21.15.16:0/841685254 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fc9480654a0 0x7fc948067950 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:14.174 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.169+0000 7fc962c14640 1 --2- 172.21.15.16:0/841685254 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc95c0c6190 0x7fc95c0b26c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:14.174 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.169+0000 7fc962c14640 1 -- 172.21.15.16:0/841685254 >> 172.21.15.16:0/841685254 conn(0x7fc95c06f4f0 msgr2=0x7fc95c0c7290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:45:14.174 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.169+0000 7fc962c14640 1 -- 172.21.15.16:0/841685254 shutdown_connections 2023-05-22T15:45:14.174 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.169+0000 7fc962c14640 1 -- 172.21.15.16:0/841685254 wait complete. 2023-05-22T15:45:14.191 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2023-05-22T15:45:14.293 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:45:14.289+0000 7f98ab67d180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-05-22T15:45:14.310 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.309+0000 7f158ab99640 1 Processor -- start 2023-05-22T15:45:14.311 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.309+0000 7f158ab99640 1 -- start start 2023-05-22T15:45:14.312 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.309+0000 7f158ab99640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1584153fe0 0x7f158406d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:14.312 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.309+0000 7f158ab99640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1584069720 con 0x7f15840cae20 2023-05-22T15:45:14.312 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.309+0000 7f158ab99640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15841543b0 con 0x7f1584070600 2023-05-22T15:45:14.312 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.309+0000 7f158ab99640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f158415e360 con 0x7f1584153fe0 2023-05-22T15:45:14.312 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.309+0000 7f1583fff640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f15840cae20 0x7f158406fef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:42250/0 (socket says 172.21.15.16:42250) 2023-05-22T15:45:14.312 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.309+0000 7f1583fff640 1 -- 172.21.15.16:0/2331854487 learned_addr learned my addr 172.21.15.16:0/2331854487 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:45:14.313 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.309+0000 7f1588935640 1 --2- 172.21.15.16:0/2331854487 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1584153fe0 0x7f158406d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:14.313 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.309+0000 7f15837fe640 1 -- 172.21.15.16:0/2331854487 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2078616866 0 0) 0x7f1584069720 con 0x7f15840cae20 2023-05-22T15:45:14.313 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.309+0000 7f15837fe640 1 -- 172.21.15.16:0/2331854487 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1574003530 con 0x7f15840cae20 2023-05-22T15:45:14.314 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.309+0000 7f15837fe640 1 -- 172.21.15.16:0/2331854487 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3735298859 0 0) 0x7f1574003530 con 0x7f15840cae20 2023-05-22T15:45:14.314 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.309+0000 7f15837fe640 1 -- 172.21.15.16:0/2331854487 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1584153fe0 msgr2=0x7f158406d3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:45:14.314 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.309+0000 7f15837fe640 1 --2- 172.21.15.16:0/2331854487 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1584153fe0 0x7f158406d3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:14.314 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.309+0000 7f15837fe640 1 -- 172.21.15.16:0/2331854487 >> v1:172.21.15.62:6789/0 conn(0x7f1584070600 legacy=0x7f15840ad410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:45:14.315 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.309+0000 7f15837fe640 1 -- 172.21.15.16:0/2331854487 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f158415d5a0 con 0x7f15840cae20 2023-05-22T15:45:14.315 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.309+0000 7f15837fe640 1 -- 172.21.15.16:0/2331854487 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7f1570001680 con 0x7f15840cae20 2023-05-22T15:45:14.315 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.309+0000 7f1588935640 1 --2- 172.21.15.16:0/2331854487 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1584153fe0 0x7f158406d3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-05-22T15:45:14.315 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.309+0000 7f15837fe640 1 -- 172.21.15.16:0/2331854487 >> v1:172.21.15.16:6789/0 conn(0x7f15840cae20 legacy=0x7f158406fef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:45:14.315 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.309+0000 7f15837fe640 1 --2- 172.21.15.16:0/2331854487 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1584153fe0 0x7f15740039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:14.315 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.309+0000 7f15837fe640 1 --2- 172.21.15.16:0/2331854487 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1574003ef0 0x7f15740047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:14.315 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.309+0000 7f15837fe640 1 --2- 172.21.15.16:0/2331854487 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1574004d10 0x7f1574005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:14.316 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.309+0000 7f15837fe640 1 -- 172.21.15.16:0/2331854487 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f158415e360 con 0x7f1574004d10 2023-05-22T15:45:14.316 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.309+0000 7f15837fe640 1 -- 172.21.15.16:0/2331854487 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f157400b9c0 con 0x7f1574003ef0 2023-05-22T15:45:14.316 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.309+0000 7f15837fe640 1 -- 172.21.15.16:0/2331854487 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f157400bb80 con 0x7f1584153fe0 2023-05-22T15:45:14.316 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.309+0000 7f1589136640 1 --2- 172.21.15.16:0/2331854487 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1574004d10 0x7f1574005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:14.316 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.309+0000 7f1588935640 1 --2- 172.21.15.16:0/2331854487 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1584153fe0 0x7f15740039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:14.316 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.309+0000 7f1583fff640 1 --2- 172.21.15.16:0/2331854487 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1574003ef0 0x7f15740047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:14.316 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.309+0000 7f1589136640 1 -- 172.21.15.16:0/2331854487 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1584153fe0 msgr2=0x7f15740039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:45:14.317 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.309+0000 7f1589136640 1 --2- 172.21.15.16:0/2331854487 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1584153fe0 0x7f15740039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:14.317 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.309+0000 7f1589136640 1 -- 172.21.15.16:0/2331854487 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1574003ef0 msgr2=0x7f15740047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:45:14.317 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.309+0000 7f1589136640 1 --2- 172.21.15.16:0/2331854487 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1574003ef0 0x7f15740047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:14.317 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.309+0000 7f1589136640 1 -- 172.21.15.16:0/2331854487 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1578009420 con 0x7f1574004d10 2023-05-22T15:45:14.320 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.313+0000 7f1589136640 1 --2- 172.21.15.16:0/2331854487 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1574004d10 0x7f1574005180 secure :-1 s=READY pgs=1678 cs=0 l=1 rev1=1 crypto rx=0x7f15840ce750 tx=0x7f157c01c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb89be21521f7b74 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:45:14.320 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.313+0000 7f15837fe640 1 -- 172.21.15.16:0/2331854487 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f157c0a8650 con 0x7f1574004d10 2023-05-22T15:45:14.320 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.313+0000 7f15837fe640 1 -- 172.21.15.16:0/2331854487 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f157c0a87b0 con 0x7f1574004d10 2023-05-22T15:45:14.320 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.313+0000 7f15837fe640 1 -- 172.21.15.16:0/2331854487 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f157c0a8ba0 con 0x7f1574004d10 2023-05-22T15:45:14.320 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.313+0000 7f158ab99640 1 -- 172.21.15.16:0/2331854487 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1574004d10 msgr2=0x7f1574005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:45:14.320 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.313+0000 7f158ab99640 1 --2- 172.21.15.16:0/2331854487 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1574004d10 0x7f1574005180 secure :-1 s=READY pgs=1678 cs=0 l=1 rev1=1 crypto rx=0x7f15840ce750 tx=0x7f157c01c800 comp rx=0 tx=0).stop 2023-05-22T15:45:14.321 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.313+0000 7f1589136640 1 -- 172.21.15.16:0/2331854487 reap_dead start 2023-05-22T15:45:14.321 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.313+0000 7f158ab99640 1 -- 172.21.15.16:0/2331854487 shutdown_connections 2023-05-22T15:45:14.322 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.313+0000 7f158ab99640 1 -- 172.21.15.16:0/2331854487 >> 172.21.15.16:0/2331854487 conn(0x7f15840b1110 msgr2=0x7f15840ce830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:45:14.322 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.313+0000 7f158ab99640 1 -- 172.21.15.16:0/2331854487 shutdown_connections 2023-05-22T15:45:14.322 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.313+0000 7f158ab99640 1 -- 172.21.15.16:0/2331854487 wait complete. 2023-05-22T15:45:14.322 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.313+0000 7f158ab99640 1 Processor -- start 2023-05-22T15:45:14.322 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.313+0000 7f158ab99640 1 -- start start 2023-05-22T15:45:14.322 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.313+0000 7f158ab99640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1574004d10 0x7f1584148780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:14.322 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.313+0000 7f158ab99640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1584159f20 0x7f158415b8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:14.323 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.313+0000 7f158ab99640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f158406d3f0 0x7f158406d890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:14.323 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.313+0000 7f158ab99640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f158406ddd0 con 0x7f158406d3f0 2023-05-22T15:45:14.323 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.313+0000 7f158ab99640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f158415ca00 con 0x7f1574004d10 2023-05-22T15:45:14.323 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.313+0000 7f158ab99640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15840cc5c0 con 0x7f1584159f20 2023-05-22T15:45:14.323 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.313+0000 7f1589136640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f158406d3f0 0x7f158406d890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:14.323 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.313+0000 7f1589136640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f158406d3f0 0x7f158406d890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36840/0 (socket says 172.21.15.16:36840) 2023-05-22T15:45:14.323 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.313+0000 7f1589136640 1 -- 172.21.15.16:0/918987687 learned_addr learned my addr 172.21.15.16:0/918987687 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:45:14.323 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.313+0000 7f1583fff640 1 --2- 172.21.15.16:0/918987687 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1584159f20 0x7f158415b8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:14.324 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.313+0000 7f1588935640 1 --2- 172.21.15.16:0/918987687 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1574004d10 0x7f1584148780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:14.324 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.317+0000 7f1589136640 1 -- 172.21.15.16:0/918987687 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1584159f20 msgr2=0x7f158415b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:45:14.324 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.317+0000 7f1589136640 1 --2- 172.21.15.16:0/918987687 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1584159f20 0x7f158415b8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:14.324 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.317+0000 7f1589136640 1 -- 172.21.15.16:0/918987687 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1574004d10 msgr2=0x7f1584148780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:45:14.324 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.317+0000 7f1589136640 1 --2- 172.21.15.16:0/918987687 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1574004d10 0x7f1584148780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:14.325 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.317+0000 7f1589136640 1 -- 172.21.15.16:0/918987687 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f158415db60 con 0x7f158406d3f0 2023-05-22T15:45:14.325 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.317+0000 7f1583fff640 1 --2- 172.21.15.16:0/918987687 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1584159f20 0x7f158415b8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T15:45:14.325 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.317+0000 7f1589136640 1 --2- 172.21.15.16:0/918987687 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f158406d3f0 0x7f158406d890 secure :-1 s=READY pgs=1679 cs=0 l=1 rev1=1 crypto rx=0x7f157c098100 tx=0x7f1574005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:45:14.325 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.317+0000 7f15817fa640 1 -- 172.21.15.16:0/918987687 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f157c0b6070 con 0x7f158406d3f0 2023-05-22T15:45:14.325 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.317+0000 7f15817fa640 1 -- 172.21.15.16:0/918987687 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f157c0011d0 con 0x7f158406d3f0 2023-05-22T15:45:14.325 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.317+0000 7f158ab99640 1 -- 172.21.15.16:0/918987687 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f158415d880 con 0x7f158406d3f0 2023-05-22T15:45:14.326 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.317+0000 7f15817fa640 1 -- 172.21.15.16:0/918987687 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f157c0a8d50 con 0x7f158406d3f0 2023-05-22T15:45:14.326 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.317+0000 7f158ab99640 1 -- 172.21.15.16:0/918987687 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f158415de40 con 0x7f158406d3f0 2023-05-22T15:45:14.327 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.317+0000 7f158ab99640 1 -- 172.21.15.16:0/918987687 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f158404dbe0 con 0x7f158406d3f0 2023-05-22T15:45:14.327 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.317+0000 7f15817fa640 1 -- 172.21.15.16:0/918987687 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 51) v1 ==== 84349+0+0 (secure 0 0 0) 0x7f157c0050c0 con 0x7f158406d3f0 2023-05-22T15:45:14.327 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.317+0000 7f15817fa640 1 --2- 172.21.15.16:0/918987687 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f1568065520 0x7f15680679d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:14.327 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.317+0000 7f1588935640 1 -- 172.21.15.16:0/918987687 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f1568065520 msgr2=0x7f15680679d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/36921 2023-05-22T15:45:14.327 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.317+0000 7f15817fa640 1 -- 172.21.15.16:0/918987687 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f157c11cff0 con 0x7f158406d3f0 2023-05-22T15:45:14.327 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.317+0000 7f1588935640 1 --2- 172.21.15.16:0/918987687 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f1568065520 0x7f15680679d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-05-22T15:45:14.328 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.321+0000 7f15817fa640 1 -- 172.21.15.16:0/918987687 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f157c0ea910 con 0x7f158406d3f0 2023-05-22T15:45:14.329 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:45:14.325+0000 7f4f29a78180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-05-22T15:45:14.484 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.481+0000 7f158ab99640 1 -- 172.21.15.16:0/918987687 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7f15840cc700 con 0x7f158406d3f0 2023-05-22T15:45:14.489 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.485+0000 7f15817fa640 1 -- 172.21.15.16:0/918987687 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v41)=0 v41) v1 ==== 138+0+0 (secure 0 0 0) 0x7f157c0edf30 con 0x7f158406d3f0 2023-05-22T15:45:14.492 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.489+0000 7f158ab99640 1 -- 172.21.15.16:0/918987687 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f1568065520 msgr2=0x7f15680679d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-05-22T15:45:14.492 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.489+0000 7f158ab99640 1 --2- 172.21.15.16:0/918987687 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f1568065520 0x7f15680679d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:14.492 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.489+0000 7f158ab99640 1 -- 172.21.15.16:0/918987687 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f158406d3f0 msgr2=0x7f158406d890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:45:14.492 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.489+0000 7f158ab99640 1 --2- 172.21.15.16:0/918987687 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f158406d3f0 0x7f158406d890 secure :-1 s=READY pgs=1679 cs=0 l=1 rev1=1 crypto rx=0x7f157c098100 tx=0x7f1574005800 comp rx=0 tx=0).stop 2023-05-22T15:45:14.493 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.489+0000 7f158ab99640 1 -- 172.21.15.16:0/918987687 shutdown_connections 2023-05-22T15:45:14.493 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.489+0000 7f158ab99640 1 --2- 172.21.15.16:0/918987687 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f158406d3f0 0x7f158406d890 unknown :-1 s=CLOSED pgs=1679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:14.493 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.489+0000 7f158ab99640 1 --2- 172.21.15.16:0/918987687 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f1568065520 0x7f15680679d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:14.493 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.489+0000 7f158ab99640 1 --2- 172.21.15.16:0/918987687 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1584159f20 0x7f158415b8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:14.493 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.489+0000 7f158ab99640 1 --2- 172.21.15.16:0/918987687 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1574004d10 0x7f1584148780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:14.494 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.489+0000 7f158ab99640 1 -- 172.21.15.16:0/918987687 >> 172.21.15.16:0/918987687 conn(0x7f15840b1110 msgr2=0x7f158406c6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:45:14.494 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.489+0000 7f158ab99640 1 -- 172.21.15.16:0/918987687 shutdown_connections 2023-05-22T15:45:14.494 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.489+0000 7f158ab99640 1 -- 172.21.15.16:0/918987687 wait complete. 2023-05-22T15:45:14.508 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2023-05-22T15:45:14.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.625+0000 7f632aba3640 1 Processor -- start 2023-05-22T15:45:14.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.625+0000 7f632aba3640 1 -- start start 2023-05-22T15:45:14.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.625+0000 7f632aba3640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f632415cfa0 0x7f632406db00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:14.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.625+0000 7f632aba3640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f632406e040 con 0x7f63240cf870 2023-05-22T15:45:14.632 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.625+0000 7f632aba3640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f632406e490 con 0x7f632406e740 2023-05-22T15:45:14.632 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.625+0000 7f632aba3640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63240cb9f0 con 0x7f632415cfa0 2023-05-22T15:45:14.633 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.625+0000 7f6323fff640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f63240cf870 0x7f63240b1fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:42262/0 (socket says 172.21.15.16:42262) 2023-05-22T15:45:14.633 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.625+0000 7f6323fff640 1 -- 172.21.15.16:0/2169283184 learned_addr learned my addr 172.21.15.16:0/2169283184 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:45:14.633 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.625+0000 7f632893f640 1 --2- 172.21.15.16:0/2169283184 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f632415cfa0 0x7f632406db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:14.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.625+0000 7f63237fe640 1 -- 172.21.15.16:0/2169283184 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2418299296 0 0) 0x7f6318001120 con 0x7f63240cf870 2023-05-22T15:45:14.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.625+0000 7f63237fe640 1 -- 172.21.15.16:0/2169283184 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f630c003530 con 0x7f63240cf870 2023-05-22T15:45:14.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.625+0000 7f632893f640 1 -- 172.21.15.16:0/2169283184 >> v1:172.21.15.62:6789/0 conn(0x7f632406e740 legacy=0x7f6324070de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:45:14.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.625+0000 7f632893f640 1 -- 172.21.15.16:0/2169283184 >> v1:172.21.15.16:6789/0 conn(0x7f63240cf870 legacy=0x7f63240b1fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:45:14.635 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.625+0000 7f632893f640 1 -- 172.21.15.16:0/2169283184 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63240cad70 con 0x7f632415cfa0 2023-05-22T15:45:14.639 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.629+0000 7f632893f640 1 --2- 172.21.15.16:0/2169283184 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f632415cfa0 0x7f632406db00 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f63140097f0 tx=0x7f631401c400 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2ab61c730933c15a server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:45:14.639 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.629+0000 7f63237fe640 1 -- 172.21.15.16:0/2169283184 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63140ab600 con 0x7f632415cfa0 2023-05-22T15:45:14.639 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.629+0000 7f63237fe640 1 -- 172.21.15.16:0/2169283184 <== mon.2 v2:172.21.15.62:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63140ab760 con 0x7f632415cfa0 2023-05-22T15:45:14.639 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.629+0000 7f63237fe640 1 -- 172.21.15.16:0/2169283184 <== mon.2 v2:172.21.15.62:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63140abad0 con 0x7f632415cfa0 2023-05-22T15:45:14.640 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.629+0000 7f632aba3640 1 -- 172.21.15.16:0/2169283184 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f632415cfa0 msgr2=0x7f632406db00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:45:14.640 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.629+0000 7f632aba3640 1 --2- 172.21.15.16:0/2169283184 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f632415cfa0 0x7f632406db00 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f63140097f0 tx=0x7f631401c400 comp rx=0 tx=0).stop 2023-05-22T15:45:14.640 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.629+0000 7f632aba3640 1 -- 172.21.15.16:0/2169283184 shutdown_connections 2023-05-22T15:45:14.640 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.629+0000 7f632aba3640 1 --2- 172.21.15.16:0/2169283184 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f632415cfa0 0x7f632406db00 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:14.640 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.629+0000 7f632aba3640 1 -- 172.21.15.16:0/2169283184 >> 172.21.15.16:0/2169283184 conn(0x7f63240cc870 msgr2=0x7f63240cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:45:14.640 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.629+0000 7f632aba3640 1 -- 172.21.15.16:0/2169283184 shutdown_connections 2023-05-22T15:45:14.641 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.629+0000 7f632aba3640 1 -- 172.21.15.16:0/2169283184 wait complete. 2023-05-22T15:45:14.641 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.629+0000 7f632aba3640 1 Processor -- start 2023-05-22T15:45:14.641 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.629+0000 7f632aba3640 1 -- start start 2023-05-22T15:45:14.641 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.633+0000 7f632aba3640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f632406e740 0x7f6324159780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:14.641 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.633+0000 7f632aba3640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f63240cf870 0x7f6324154720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:14.641 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.633+0000 7f632aba3640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f632415cfa0 0x7f6324156700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:14.641 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.633+0000 7f632aba3640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6324151500 con 0x7f632415cfa0 2023-05-22T15:45:14.642 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.633+0000 7f632aba3640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63241586e0 con 0x7f63240cf870 2023-05-22T15:45:14.642 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.633+0000 7f632aba3640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63241589c0 con 0x7f632406e740 2023-05-22T15:45:14.642 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.633+0000 7f6329140640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f632415cfa0 0x7f6324156700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:14.642 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.633+0000 7f6329140640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f632415cfa0 0x7f6324156700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36842/0 (socket says 172.21.15.16:36842) 2023-05-22T15:45:14.642 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.633+0000 7f6329140640 1 -- 172.21.15.16:0/4057348872 learned_addr learned my addr 172.21.15.16:0/4057348872 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:45:14.642 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.633+0000 7f632893f640 1 --2- 172.21.15.16:0/4057348872 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f632406e740 0x7f6324159780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:14.642 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.633+0000 7f6323fff640 1 --2- 172.21.15.16:0/4057348872 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f63240cf870 0x7f6324154720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:14.643 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.633+0000 7f6329140640 1 -- 172.21.15.16:0/4057348872 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f632406e740 msgr2=0x7f6324159780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:45:14.643 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.633+0000 7f6329140640 1 --2- 172.21.15.16:0/4057348872 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f632406e740 0x7f6324159780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:14.643 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.633+0000 7f6329140640 1 -- 172.21.15.16:0/4057348872 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f63240cf870 msgr2=0x7f6324154720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:45:14.643 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.633+0000 7f6329140640 1 --2- 172.21.15.16:0/4057348872 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f63240cf870 0x7f6324154720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:14.643 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.633+0000 7f6329140640 1 -- 172.21.15.16:0/4057348872 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6314009420 con 0x7f632415cfa0 2023-05-22T15:45:14.643 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.633+0000 7f632893f640 1 --2- 172.21.15.16:0/4057348872 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f632406e740 0x7f6324159780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T15:45:14.643 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.633+0000 7f6329140640 1 --2- 172.21.15.16:0/4057348872 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f632415cfa0 0x7f6324156700 secure :-1 s=READY pgs=1681 cs=0 l=1 rev1=1 crypto rx=0x7f631c00c770 tx=0x7f631c00cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:45:14.644 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.633+0000 7f63217fa640 1 -- 172.21.15.16:0/4057348872 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f631c003b80 con 0x7f632415cfa0 2023-05-22T15:45:14.644 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.633+0000 7f63217fa640 1 -- 172.21.15.16:0/4057348872 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f631c003ce0 con 0x7f632415cfa0 2023-05-22T15:45:14.644 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.633+0000 7f63217fa640 1 -- 172.21.15.16:0/4057348872 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f631c00aa90 con 0x7f632415cfa0 2023-05-22T15:45:14.644 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.633+0000 7f632aba3640 1 -- 172.21.15.16:0/4057348872 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6324071900 con 0x7f632415cfa0 2023-05-22T15:45:14.645 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.633+0000 7f632aba3640 1 -- 172.21.15.16:0/4057348872 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63240cb610 con 0x7f632415cfa0 2023-05-22T15:45:14.645 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.633+0000 7f632aba3640 1 -- 172.21.15.16:0/4057348872 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f632406ef50 con 0x7f632415cfa0 2023-05-22T15:45:14.645 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.633+0000 7f63217fa640 1 -- 172.21.15.16:0/4057348872 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 51) v1 ==== 84349+0+0 (secure 0 0 0) 0x7f631c00abf0 con 0x7f632415cfa0 2023-05-22T15:45:14.645 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.633+0000 7f63217fa640 1 --2- 172.21.15.16:0/4057348872 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f63100654f0 0x7f63100679a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:14.646 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.633+0000 7f632893f640 1 -- 172.21.15.16:0/4057348872 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f63100654f0 msgr2=0x7f63100679a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/36921 2023-05-22T15:45:14.646 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.633+0000 7f63217fa640 1 -- 172.21.15.16:0/4057348872 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f631c082970 con 0x7f632415cfa0 2023-05-22T15:45:14.646 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.633+0000 7f632893f640 1 --2- 172.21.15.16:0/4057348872 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f63100654f0 0x7f63100679a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-05-22T15:45:14.647 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.641+0000 7f63217fa640 1 -- 172.21.15.16:0/4057348872 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f631c050430 con 0x7f632415cfa0 2023-05-22T15:45:14.689 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:45:14.685+0000 7f98ab67d180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-05-22T15:45:14.729 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:45:14.725+0000 7f4f29a78180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-05-22T15:45:14.799 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.797+0000 7f632aba3640 1 -- 172.21.15.16:0/4057348872 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7f632404dbe0 con 0x7f632415cfa0 2023-05-22T15:45:14.806 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.801+0000 7f63217fa640 1 -- 172.21.15.16:0/4057348872 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v42)=0 v42) v1 ==== 133+0+0 (secure 0 0 0) 0x7f631c053a50 con 0x7f632415cfa0 2023-05-22T15:45:14.808 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.805+0000 7f632aba3640 1 -- 172.21.15.16:0/4057348872 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f63100654f0 msgr2=0x7f63100679a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-05-22T15:45:14.808 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.805+0000 7f632aba3640 1 --2- 172.21.15.16:0/4057348872 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f63100654f0 0x7f63100679a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:14.809 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.805+0000 7f632aba3640 1 -- 172.21.15.16:0/4057348872 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f632415cfa0 msgr2=0x7f6324156700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:45:14.809 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.805+0000 7f632aba3640 1 --2- 172.21.15.16:0/4057348872 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f632415cfa0 0x7f6324156700 secure :-1 s=READY pgs=1681 cs=0 l=1 rev1=1 crypto rx=0x7f631c00c770 tx=0x7f631c00cc30 comp rx=0 tx=0).stop 2023-05-22T15:45:14.809 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.805+0000 7f632aba3640 1 -- 172.21.15.16:0/4057348872 shutdown_connections 2023-05-22T15:45:14.809 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.805+0000 7f632aba3640 1 --2- 172.21.15.16:0/4057348872 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f632415cfa0 0x7f6324156700 unknown :-1 s=CLOSED pgs=1681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:14.809 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.805+0000 7f632aba3640 1 --2- 172.21.15.16:0/4057348872 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f63240cf870 0x7f6324154720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:14.810 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.805+0000 7f632aba3640 1 --2- 172.21.15.16:0/4057348872 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f63100654f0 0x7f63100679a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:14.810 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.805+0000 7f632aba3640 1 --2- 172.21.15.16:0/4057348872 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f632406e740 0x7f6324159780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:14.810 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.805+0000 7f632aba3640 1 -- 172.21.15.16:0/4057348872 >> 172.21.15.16:0/4057348872 conn(0x7f63240cc870 msgr2=0x7f63240b3fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:45:14.810 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.805+0000 7f632aba3640 1 -- 172.21.15.16:0/4057348872 shutdown_connections 2023-05-22T15:45:14.810 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.805+0000 7f632aba3640 1 -- 172.21.15.16:0/4057348872 wait complete. 2023-05-22T15:45:14.825 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:45:14.826 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1) 2023-05-22T15:45:14.827 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:45:14.821+0000 7f98ab67d180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-05-22T15:45:14.838 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add --fs cephfs snap_test_dir1 1M 2023-05-22T15:45:14.869 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:45:14.865+0000 7f4f29a78180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-05-22T15:45:14.990 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.985+0000 7fd63e931640 1 Processor -- start 2023-05-22T15:45:14.990 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.985+0000 7fd63e931640 1 -- start start 2023-05-22T15:45:14.991 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.989+0000 7fd63e931640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd6380cb340 0x7fd6380b17d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:14.992 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.989+0000 7fd63e931640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fd638069720 0x7fd638069bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:14.992 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.989+0000 7fd63e931640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6380ce350 con 0x7fd6380cae40 2023-05-22T15:45:14.992 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.989+0000 7fd63e931640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd63806a100 con 0x7fd6380cb340 2023-05-22T15:45:14.993 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.989+0000 7fd63e931640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd63815f7e0 con 0x7fd638069720 2023-05-22T15:45:14.993 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.989+0000 7fd6377fe640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd6380cb340 0x7fd6380b17d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:14.993 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.989+0000 7fd6377fe640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd6380cb340 0x7fd6380b17d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36854/0 (socket says 172.21.15.16:36854) 2023-05-22T15:45:14.993 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.989+0000 7fd6377fe640 1 -- 172.21.15.16:0/2422775823 learned_addr learned my addr 172.21.15.16:0/2422775823 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:45:14.993 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.989+0000 7fd63cece640 1 --2- 172.21.15.16:0/2422775823 >> v2:172.21.15.62:3300/0 conn(0x7fd638069720 0x7fd638069bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:14.993 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.989+0000 7fd6377fe640 1 -- 172.21.15.16:0/2422775823 >> v2:172.21.15.62:3300/0 conn(0x7fd638069720 msgr2=0x7fd638069bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:45:14.993 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.989+0000 7fd6377fe640 1 --2- 172.21.15.16:0/2422775823 >> v2:172.21.15.62:3300/0 conn(0x7fd638069720 0x7fd638069bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:14.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.989+0000 7fd6377fe640 1 -- 172.21.15.16:0/2422775823 >> v1:172.21.15.62:6789/0 conn(0x7fd6380cae40 legacy=0x7fd63806c700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:45:14.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.989+0000 7fd6377fe640 1 -- 172.21.15.16:0/2422775823 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd63806ce10 con 0x7fd6380cb340 2023-05-22T15:45:14.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.989+0000 7fd6377fe640 1 --2- 172.21.15.16:0/2422775823 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd6380cb340 0x7fd6380b17d0 secure :-1 s=READY pgs=1682 cs=0 l=1 rev1=1 crypto rx=0x7fd6380c9710 tx=0x7fd62c01c4c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a2206c5f74da3b1 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:45:14.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.989+0000 7fd636ffd640 1 -- 172.21.15.16:0/2422775823 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd62c0ab600 con 0x7fd6380cb340 2023-05-22T15:45:14.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.989+0000 7fd636ffd640 1 -- 172.21.15.16:0/2422775823 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd62c0a7050 con 0x7fd6380cb340 2023-05-22T15:45:14.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.989+0000 7fd636ffd640 1 -- 172.21.15.16:0/2422775823 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd62c0ab980 con 0x7fd6380cb340 2023-05-22T15:45:14.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.989+0000 7fd63e931640 1 -- 172.21.15.16:0/2422775823 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd6380cb340 msgr2=0x7fd6380b17d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:45:14.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.989+0000 7fd63e931640 1 --2- 172.21.15.16:0/2422775823 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd6380cb340 0x7fd6380b17d0 secure :-1 s=READY pgs=1682 cs=0 l=1 rev1=1 crypto rx=0x7fd6380c9710 tx=0x7fd62c01c4c0 comp rx=0 tx=0).stop 2023-05-22T15:45:14.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.989+0000 7fd63e931640 1 -- 172.21.15.16:0/2422775823 shutdown_connections 2023-05-22T15:45:14.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.989+0000 7fd63e931640 1 --2- 172.21.15.16:0/2422775823 >> v2:172.21.15.62:3300/0 conn(0x7fd638069720 0x7fd638069bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:14.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.989+0000 7fd63e931640 1 --2- 172.21.15.16:0/2422775823 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd6380cb340 0x7fd6380b17d0 unknown :-1 s=CLOSED pgs=1682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:14.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.989+0000 7fd63e931640 1 -- 172.21.15.16:0/2422775823 >> 172.21.15.16:0/2422775823 conn(0x7fd6380b0cc0 msgr2=0x7fd6380b10c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:45:14.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.993+0000 7fd63e931640 1 -- 172.21.15.16:0/2422775823 shutdown_connections 2023-05-22T15:45:14.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.993+0000 7fd63e931640 1 -- 172.21.15.16:0/2422775823 wait complete. 2023-05-22T15:45:15.001 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.993+0000 7fd63e931640 1 Processor -- start 2023-05-22T15:45:15.001 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.993+0000 7fd63e931640 1 -- start start 2023-05-22T15:45:15.002 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.993+0000 7fd63e931640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd638069720 0x7fd638152880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:15.002 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.993+0000 7fd63e931640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd6380cae40 0x7fd6380b4910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:15.002 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.993+0000 7fd63e931640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd6380cb340 0x7fd6380b29d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:15.002 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.993+0000 7fd63e931640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6380ad1f0 con 0x7fd638069720 2023-05-22T15:45:15.003 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.993+0000 7fd63e931640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd638152dc0 con 0x7fd6380cb340 2023-05-22T15:45:15.003 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.993+0000 7fd63e931640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6380ac5e0 con 0x7fd6380cae40 2023-05-22T15:45:15.007 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.993+0000 7fd637fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd638069720 0x7fd638152880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:15.007 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.993+0000 7fd637fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd638069720 0x7fd638152880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36868/0 (socket says 172.21.15.16:36868) 2023-05-22T15:45:15.008 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.993+0000 7fd637fff640 1 -- 172.21.15.16:0/69993484 learned_addr learned my addr 172.21.15.16:0/69993484 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:45:15.008 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.993+0000 7fd63cece640 1 --2- 172.21.15.16:0/69993484 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd6380cb340 0x7fd6380b29d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:15.009 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.993+0000 7fd6377fe640 1 --2- 172.21.15.16:0/69993484 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd6380cae40 0x7fd6380b4910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:15.009 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.993+0000 7fd637fff640 1 -- 172.21.15.16:0/69993484 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd6380cae40 msgr2=0x7fd6380b4910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:45:15.009 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.993+0000 7fd637fff640 1 --2- 172.21.15.16:0/69993484 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd6380cae40 0x7fd6380b4910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:15.009 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.993+0000 7fd637fff640 1 -- 172.21.15.16:0/69993484 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd6380cb340 msgr2=0x7fd6380b29d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:45:15.010 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.993+0000 7fd637fff640 1 --2- 172.21.15.16:0/69993484 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd6380cb340 0x7fd6380b29d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:15.010 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.993+0000 7fd637fff640 1 -- 172.21.15.16:0/69993484 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd62c0094d0 con 0x7fd638069720 2023-05-22T15:45:15.010 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.993+0000 7fd63cece640 1 --2- 172.21.15.16:0/69993484 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd6380cb340 0x7fd6380b29d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T15:45:15.010 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.993+0000 7fd637fff640 1 --2- 172.21.15.16:0/69993484 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd638069720 0x7fd638152880 secure :-1 s=READY pgs=1683 cs=0 l=1 rev1=1 crypto rx=0x7fd62800c450 tx=0x7fd62800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:45:15.011 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.993+0000 7fd634ff9640 1 -- 172.21.15.16:0/69993484 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd628012830 con 0x7fd638069720 2023-05-22T15:45:15.011 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.993+0000 7fd634ff9640 1 -- 172.21.15.16:0/69993484 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6280129b0 con 0x7fd638069720 2023-05-22T15:45:15.011 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.993+0000 7fd63e931640 1 -- 172.21.15.16:0/69993484 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6380cee20 con 0x7fd638069720 2023-05-22T15:45:15.011 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.993+0000 7fd634ff9640 1 -- 172.21.15.16:0/69993484 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd628012d60 con 0x7fd638069720 2023-05-22T15:45:15.012 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.993+0000 7fd63e931640 1 -- 172.21.15.16:0/69993484 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6380ceb40 con 0x7fd638069720 2023-05-22T15:45:15.012 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.997+0000 7fd63e931640 1 -- 172.21.15.16:0/69993484 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd638168af0 con 0x7fd638069720 2023-05-22T15:45:15.012 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.997+0000 7fd634ff9640 1 -- 172.21.15.16:0/69993484 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 51) v1 ==== 84349+0+0 (secure 0 0 0) 0x7fd62801c460 con 0x7fd638069720 2023-05-22T15:45:15.012 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.997+0000 7fd634ff9640 1 --2- 172.21.15.16:0/69993484 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd6240654f0 0x7fd6240679a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:15.013 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.997+0000 7fd6377fe640 1 -- 172.21.15.16:0/69993484 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd6240654f0 msgr2=0x7fd6240679a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/36921 2023-05-22T15:45:15.013 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.997+0000 7fd634ff9640 1 -- 172.21.15.16:0/69993484 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 6382+0+0 (secure 0 0 0) 0x7fd6280829b0 con 0x7fd638069720 2023-05-22T15:45:15.013 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:14.997+0000 7fd6377fe640 1 --2- 172.21.15.16:0/69993484 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd6240654f0 0x7fd6240679a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-05-22T15:45:15.014 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:15.005+0000 7fd634ff9640 1 -- 172.21.15.16:0/69993484 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fd628050410 con 0x7fd638069720 2023-05-22T15:45:15.180 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:15.177+0000 7fd63e931640 1 -- 172.21.15.16:0/69993484 --> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "snap_test_dir1", "snap_schedule": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7fd6380ad330 con 0x7fd6240654f0 2023-05-22T15:45:15.202 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:15.201+0000 7fd6377fe640 1 -- 172.21.15.16:0/69993484 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd6240654f0 msgr2=0x7fd6240679a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/36921 2023-05-22T15:45:15.203 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:15.201+0000 7fd6377fe640 1 --2- 172.21.15.16:0/69993484 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd6240654f0 0x7fd6240679a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-05-22T15:45:15.268 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:45:15.265+0000 7f98ab67d180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-05-22T15:45:15.321 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:45:15.317+0000 7f98ab67d180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-05-22T15:45:15.321 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:45:15.317+0000 7f4f29a78180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-05-22T15:45:15.371 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:45:15.365+0000 7f98ab67d180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-05-22T15:45:15.374 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:45:15.369+0000 7f4f29a78180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-05-22T15:45:15.424 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:45:15.421+0000 7f4f29a78180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-05-22T15:45:15.448 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:45:15.445+0000 7f98ab67d180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-05-22T15:45:15.499 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:45:15.493+0000 7f98ab67d180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-05-22T15:45:15.502 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:45:15.497+0000 7f4f29a78180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-05-22T15:45:15.554 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:45:15.549+0000 7f4f29a78180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-05-22T15:45:15.577 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:45:15.573+0000 7f98ab67d180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-05-22T15:45:15.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:15.605+0000 7fd6377fe640 1 -- 172.21.15.16:0/69993484 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd6240654f0 msgr2=0x7fd6240679a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/36921 2023-05-22T15:45:15.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:15.605+0000 7fd6377fe640 1 --2- 172.21.15.16:0/69993484 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd6240654f0 0x7fd6240679a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-05-22T15:45:15.629 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:45:15.625+0000 7f98ab67d180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-05-22T15:45:15.632 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:45:15.629+0000 7f4f29a78180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-05-22T15:45:15.683 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:45:15.681+0000 7f4f29a78180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-05-22T15:45:15.762 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:45:15.757+0000 7f98ab67d180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-05-22T15:45:15.814 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:45:15.809+0000 7f98ab67d180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-05-22T15:45:15.816 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:45:15.813+0000 7f4f29a78180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-05-22T15:45:15.868 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:45:15.865+0000 7f4f29a78180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-05-22T15:45:16.038 INFO:tasks.ceph.mgr.x.smithi062.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2023-05-22T15:45:16.039 INFO:tasks.ceph.mgr.x.smithi062.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2023-05-22T15:45:16.039 INFO:tasks.ceph.mgr.x.smithi062.stderr: from numpy import show_config as show_numpy_config 2023-05-22T15:45:16.046 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:45:16.041+0000 7f98ab67d180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-05-22T15:45:16.093 INFO:tasks.ceph.mgr.y.smithi016.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2023-05-22T15:45:16.094 INFO:tasks.ceph.mgr.y.smithi016.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2023-05-22T15:45:16.094 INFO:tasks.ceph.mgr.y.smithi016.stderr: from numpy import show_config as show_numpy_config 2023-05-22T15:45:16.101 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:45:16.097+0000 7f4f29a78180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-05-22T15:45:16.171 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:45:16.165+0000 7f98ab67d180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-05-22T15:45:16.226 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:45:16.221+0000 7f4f29a78180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-05-22T15:45:16.283 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:45:16.277+0000 7f98ab67d180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-05-22T15:45:16.339 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:45:16.337+0000 7f4f29a78180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-05-22T15:45:16.346 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:45:16.341+0000 7f98ab67d180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-05-22T15:45:16.403 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:45:16.401+0000 7f4f29a78180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-05-22T15:45:16.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:16.409+0000 7fd6377fe640 1 -- 172.21.15.16:0/69993484 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd6240654f0 msgr2=0x7fd6240679a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/36921 2023-05-22T15:45:16.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:16.409+0000 7fd6377fe640 1 --2- 172.21.15.16:0/69993484 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd6240654f0 0x7fd6240679a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-05-22T15:45:16.471 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:45:16.465+0000 7f98ab67d180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-05-22T15:45:16.528 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:45:16.525+0000 7f4f29a78180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-05-22T15:45:16.964 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:45:16.957+0000 7f98ab67d180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-05-22T15:45:17.025 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:45:17.021+0000 7f4f29a78180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-05-22T15:45:17.038 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:45:17.033+0000 7f98ab67d180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-05-22T15:45:17.101 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:45:17.097+0000 7f4f29a78180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-05-22T15:45:17.175 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:45:17.169+0000 7f98ab67d180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-05-22T15:45:17.240 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:45:17.237+0000 7f4f29a78180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-05-22T15:45:17.252 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:45:17.245+0000 7f98ab67d180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-05-22T15:45:17.317 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:45:17.313+0000 7f4f29a78180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-05-22T15:45:17.497 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:45:17.493+0000 7f98ab67d180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-05-22T15:45:17.552 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:45:17.545+0000 7f98ab67d180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-05-22T15:45:17.567 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:45:17.565+0000 7f4f29a78180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-05-22T15:45:17.623 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:45:17.621+0000 7f4f29a78180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-05-22T15:45:17.636 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:17.633+0000 7fd634ff9640 1 -- 172.21.15.16:0/69993484 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mgrmap(e 52) v1 ==== 84349+0+0 (secure 0 0 0) 0x7fd6280507e0 con 0x7fd638069720 2023-05-22T15:45:17.665 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:17.661+0000 7fd634ff9640 1 -- 172.21.15.16:0/69993484 <== mon.0 v2:172.21.15.16:3300/0 8 ==== mgrmap(e 53) v1 ==== 83958+0+0 (secure 0 0 0) 0x7fd628080c00 con 0x7fd638069720 2023-05-22T15:45:17.666 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:17.661+0000 7fd634ff9640 1 -- 172.21.15.16:0/69993484 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd6240654f0 msgr2=0x7fd6240679a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-05-22T15:45:17.666 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:17.661+0000 7fd634ff9640 1 --2- 172.21.15.16:0/69993484 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd6240654f0 0x7fd6240679a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:17.845 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:45:17.841+0000 7f4eabbf6640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T15:45:17.846 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:45:17.841+0000 7f4eabbf6640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T15:45:17.846 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:45:17.841+0000 7f4eabbf6640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T15:45:17.846 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:45:17.841+0000 7f4eabbf6640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T15:45:17.846 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:45:17.841+0000 7f4eabbf6640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T15:45:17.847 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:45:17.841+0000 7f4ea8bf0640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T15:45:17.847 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:45:17.841+0000 7f4ea8bf0640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T15:45:17.847 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:45:17.841+0000 7f4ea8bf0640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T15:45:17.847 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:45:17.841+0000 7f4ea8bf0640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T15:45:17.847 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:45:17.841+0000 7f4ea8bf0640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T15:45:18.671 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:18.669+0000 7fd634ff9640 1 -- 172.21.15.16:0/69993484 <== mon.0 v2:172.21.15.16:3300/0 9 ==== mgrmap(e 54) v1 ==== 84085+0+0 (secure 0 0 0) 0x7fd628057d70 con 0x7fd638069720 2023-05-22T15:45:18.671 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:18.669+0000 7fd634ff9640 1 --2- 172.21.15.16:0/69993484 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd6240654f0 0x7fd624067920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:18.671 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:18.669+0000 7fd634ff9640 1 -- 172.21.15.16:0/69993484 --> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "snap_test_dir1", "snap_schedule": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7fd624031f70 con 0x7fd6240654f0 2023-05-22T15:45:18.674 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:18.669+0000 7fd6377fe640 1 --2- 172.21.15.16:0/69993484 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd6240654f0 0x7fd624067920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:18.677 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:18.673+0000 7fd6377fe640 1 --2- 172.21.15.16:0/69993484 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd6240654f0 0x7fd624067920 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fd62c0ad000 tx=0x7fd62c0a6000 comp rx=0 tx=0).ready entity=mgr.8733 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:45:18.821 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:18.817+0000 7fd634ff9640 1 -- 172.21.15.16:0/69993484 <== mgr.8733 v2:172.21.15.16:6832/36921 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fd624031f70 con 0x7fd6240654f0 2023-05-22T15:45:18.821 INFO:teuthology.orchestra.run.smithi016.stdout:Schedule set for path snap_test_dir1 2023-05-22T15:45:18.824 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:18.821+0000 7fd63e931640 1 -- 172.21.15.16:0/69993484 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd6240654f0 msgr2=0x7fd624067920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:45:18.825 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:18.821+0000 7fd63e931640 1 --2- 172.21.15.16:0/69993484 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd6240654f0 0x7fd624067920 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fd62c0ad000 tx=0x7fd62c0a6000 comp rx=0 tx=0).stop 2023-05-22T15:45:18.825 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:18.821+0000 7fd63e931640 1 -- 172.21.15.16:0/69993484 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd638069720 msgr2=0x7fd638152880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:45:18.825 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:18.821+0000 7fd63e931640 1 --2- 172.21.15.16:0/69993484 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd638069720 0x7fd638152880 secure :-1 s=READY pgs=1683 cs=0 l=1 rev1=1 crypto rx=0x7fd62800c450 tx=0x7fd62800c910 comp rx=0 tx=0).stop 2023-05-22T15:45:18.825 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:18.821+0000 7fd63e931640 1 -- 172.21.15.16:0/69993484 shutdown_connections 2023-05-22T15:45:18.825 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:18.821+0000 7fd63e931640 1 --2- 172.21.15.16:0/69993484 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd6380cb340 0x7fd6380b29d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:18.825 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:18.821+0000 7fd63e931640 1 --2- 172.21.15.16:0/69993484 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd6240654f0 0x7fd624067920 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:18.825 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:18.821+0000 7fd63e931640 1 --2- 172.21.15.16:0/69993484 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd6380cae40 0x7fd6380b4910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:18.826 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:18.821+0000 7fd63e931640 1 --2- 172.21.15.16:0/69993484 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd638069720 0x7fd638152880 unknown :-1 s=CLOSED pgs=1683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:18.826 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:18.821+0000 7fd63e931640 1 -- 172.21.15.16:0/69993484 >> 172.21.15.16:0/69993484 conn(0x7fd6380b0cc0 msgr2=0x7fd6380b00b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:45:18.826 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:18.821+0000 7fd63e931640 1 -- 172.21.15.16:0/69993484 shutdown_connections 2023-05-22T15:45:18.826 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:18.821+0000 7fd63e931640 1 -- 172.21.15.16:0/69993484 wait complete. 2023-05-22T15:45:18.847 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1 2023-05-22T15:45:18.848 DEBUG:tasks.cephfs.test_snap_schedules:expecting snap snap_test_dir1/.snap/scheduled-2023-05-22-15_46_00 in ~42s... 2023-05-22T15:45:18.849 DEBUG:tasks.cephfs.test_snap_schedules:expected_schedule: ['1M'], expected_retention: [] 2023-05-22T15:45:18.849 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule list --fs cephfs --format json snap_test_dir1 2023-05-22T15:45:18.981 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:18.977+0000 7f71aa51b640 1 Processor -- start 2023-05-22T15:45:18.982 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:18.977+0000 7f71aa51b640 1 -- start start 2023-05-22T15:45:18.991 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:18.977+0000 7f71aa51b640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f71a40cc880 0x7f71a40cad20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:18.992 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:18.977+0000 7f71aa51b640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71a4070d20 con 0x7f71a40718e0 2023-05-22T15:45:18.992 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:18.977+0000 7f71aa51b640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71a415e5d0 con 0x7f71a4069720 2023-05-22T15:45:18.992 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:18.977+0000 7f71aa51b640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71a40b1c70 con 0x7f71a40cc880 2023-05-22T15:45:18.992 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:18.981+0000 7f71a8ab8640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f71a40718e0 0x7f71a40cb260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:42268/0 (socket says 172.21.15.16:42268) 2023-05-22T15:45:18.992 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:18.981+0000 7f71a8ab8640 1 -- 172.21.15.16:0/3357201310 learned_addr learned my addr 172.21.15.16:0/3357201310 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:45:18.992 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:18.981+0000 7f71a37fe640 1 --2- 172.21.15.16:0/3357201310 >> v2:172.21.15.62:3300/0 conn(0x7f71a40cc880 0x7f71a40cad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:18.992 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:18.981+0000 7f71a2ffd640 1 -- 172.21.15.16:0/3357201310 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 16908023 0 0) 0x7f71a4070d20 con 0x7f71a40718e0 2023-05-22T15:45:18.993 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:18.981+0000 7f71a2ffd640 1 -- 172.21.15.16:0/3357201310 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7184003530 con 0x7f71a40718e0 2023-05-22T15:45:18.993 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:18.981+0000 7f71a2ffd640 1 -- 172.21.15.16:0/3357201310 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1801423633 0 0) 0x7f7184003530 con 0x7f71a40718e0 2023-05-22T15:45:18.993 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:18.981+0000 7f71a2ffd640 1 -- 172.21.15.16:0/3357201310 >> v2:172.21.15.62:3300/0 conn(0x7f71a40cc880 msgr2=0x7f71a40cad20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:45:18.993 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:18.981+0000 7f71a2ffd640 1 --2- 172.21.15.16:0/3357201310 >> v2:172.21.15.62:3300/0 conn(0x7f71a40cc880 0x7f71a40cad20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:18.993 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:18.981+0000 7f71a2ffd640 1 -- 172.21.15.16:0/3357201310 >> v1:172.21.15.62:6789/0 conn(0x7f71a4069720 legacy=0x7f71a4069af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:45:18.993 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:18.981+0000 7f71a2ffd640 1 -- 172.21.15.16:0/3357201310 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71a4159f40 con 0x7f71a40718e0 2023-05-22T15:45:18.993 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:18.981+0000 7f71a2ffd640 1 -- 172.21.15.16:0/3357201310 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7f719c001680 con 0x7f71a40718e0 2023-05-22T15:45:18.993 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:18.981+0000 7f71a2ffd640 1 -- 172.21.15.16:0/3357201310 >> v1:172.21.15.16:6789/0 conn(0x7f71a40718e0 legacy=0x7f71a40cb260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:45:18.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:18.981+0000 7f71a2ffd640 1 --2- 172.21.15.16:0/3357201310 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7184003980 0x7f7184003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:18.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:18.981+0000 7f71a2ffd640 1 --2- 172.21.15.16:0/3357201310 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f71840047d0 0x7f7184004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:18.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:18.981+0000 7f71a2ffd640 1 --2- 172.21.15.16:0/3357201310 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7184005180 0x7f718400b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:18.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:18.981+0000 7f71a2ffd640 1 -- 172.21.15.16:0/3357201310 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71a40b1c70 con 0x7f7184005180 2023-05-22T15:45:18.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:18.981+0000 7f71a2ffd640 1 -- 172.21.15.16:0/3357201310 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f718400c090 con 0x7f7184003980 2023-05-22T15:45:18.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:18.981+0000 7f71a2ffd640 1 -- 172.21.15.16:0/3357201310 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f718400c250 con 0x7f71840047d0 2023-05-22T15:45:18.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:18.981+0000 7f71a8ab8640 1 --2- 172.21.15.16:0/3357201310 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7184005180 0x7f718400b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:18.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:18.981+0000 7f71a37fe640 1 --2- 172.21.15.16:0/3357201310 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f71840047d0 0x7f7184004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:18.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:18.981+0000 7f71a3fff640 1 --2- 172.21.15.16:0/3357201310 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7184003980 0x7f7184003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:18.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:18.981+0000 7f71a8ab8640 1 -- 172.21.15.16:0/3357201310 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f71840047d0 msgr2=0x7f7184004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:45:18.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:18.981+0000 7f71a8ab8640 1 --2- 172.21.15.16:0/3357201310 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f71840047d0 0x7f7184004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:18.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:18.981+0000 7f71a8ab8640 1 -- 172.21.15.16:0/3357201310 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7184003980 msgr2=0x7f7184003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:45:18.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:18.981+0000 7f71a8ab8640 1 --2- 172.21.15.16:0/3357201310 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7184003980 0x7f7184003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:18.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:18.981+0000 7f71a8ab8640 1 -- 172.21.15.16:0/3357201310 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7184003530 con 0x7f7184005180 2023-05-22T15:45:18.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:18.981+0000 7f71a8ab8640 1 -- 172.21.15.16:0/3357201310 reap_dead start 2023-05-22T15:45:18.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:18.981+0000 7f71a8ab8640 1 --2- 172.21.15.16:0/3357201310 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7184005180 0x7f718400b990 secure :-1 s=READY pgs=1700 cs=0 l=1 rev1=1 crypto rx=0x7f719c00c240 tx=0x7f719c01f3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd96866dd44ff575 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:45:18.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:18.981+0000 7f71a2ffd640 1 -- 172.21.15.16:0/3357201310 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f719c005ae0 con 0x7f7184005180 2023-05-22T15:45:18.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:18.981+0000 7f71a2ffd640 1 -- 172.21.15.16:0/3357201310 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f719c0ab070 con 0x7f7184005180 2023-05-22T15:45:18.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:18.981+0000 7f71a2ffd640 1 -- 172.21.15.16:0/3357201310 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f719c007070 con 0x7f7184005180 2023-05-22T15:45:18.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:18.985+0000 7f71aa51b640 1 -- 172.21.15.16:0/3357201310 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7184005180 msgr2=0x7f718400b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:45:18.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:18.985+0000 7f71aa51b640 1 --2- 172.21.15.16:0/3357201310 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7184005180 0x7f718400b990 secure :-1 s=READY pgs=1700 cs=0 l=1 rev1=1 crypto rx=0x7f719c00c240 tx=0x7f719c01f3b0 comp rx=0 tx=0).stop 2023-05-22T15:45:18.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:18.985+0000 7f71aa51b640 1 -- 172.21.15.16:0/3357201310 shutdown_connections 2023-05-22T15:45:18.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:18.985+0000 7f71aa51b640 1 --2- 172.21.15.16:0/3357201310 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7184005180 0x7f718400b990 unknown :-1 s=CLOSED pgs=1700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:18.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:18.985+0000 7f71aa51b640 1 -- 172.21.15.16:0/3357201310 >> 172.21.15.16:0/3357201310 conn(0x7f71a40b0cc0 msgr2=0x7f71a40b10c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:45:18.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:18.985+0000 7f71aa51b640 1 -- 172.21.15.16:0/3357201310 shutdown_connections 2023-05-22T15:45:18.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:18.985+0000 7f71aa51b640 1 -- 172.21.15.16:0/3357201310 wait complete. 2023-05-22T15:45:18.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:18.985+0000 7f71aa51b640 1 Processor -- start 2023-05-22T15:45:18.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:18.985+0000 7f71aa51b640 1 -- start start 2023-05-22T15:45:18.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:18.985+0000 7f71aa51b640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7184005180 0x7f71a41521d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:18.998 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:18.985+0000 7f71aa51b640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f71a4154e00 0x7f71a4150f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:18.998 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:18.985+0000 7f71aa51b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f71a415b3e0 0x7f71a40b3140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:18.998 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:18.985+0000 7f71aa51b640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71a41514a0 con 0x7f71a415b3e0 2023-05-22T15:45:18.998 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:18.985+0000 7f71aa51b640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71a40c8f80 con 0x7f71a4154e00 2023-05-22T15:45:18.998 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:18.985+0000 7f71aa51b640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71a40c9310 con 0x7f7184005180 2023-05-22T15:45:18.998 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:18.985+0000 7f71a8ab8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f71a415b3e0 0x7f71a40b3140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:18.998 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:18.985+0000 7f71a8ab8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f71a415b3e0 0x7f71a40b3140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36998/0 (socket says 172.21.15.16:36998) 2023-05-22T15:45:18.998 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:18.985+0000 7f71a8ab8640 1 -- 172.21.15.16:0/3111432715 learned_addr learned my addr 172.21.15.16:0/3111432715 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:45:18.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:18.985+0000 7f71a37fe640 1 --2- 172.21.15.16:0/3111432715 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f71a4154e00 0x7f71a4150f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:18.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:18.985+0000 7f71a3fff640 1 --2- 172.21.15.16:0/3111432715 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7184005180 0x7f71a41521d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:18.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:18.985+0000 7f71a8ab8640 1 -- 172.21.15.16:0/3111432715 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7184005180 msgr2=0x7f71a41521d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:45:18.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:18.985+0000 7f71a8ab8640 1 --2- 172.21.15.16:0/3111432715 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7184005180 0x7f71a41521d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:18.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:18.985+0000 7f71a8ab8640 1 -- 172.21.15.16:0/3111432715 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f71a4154e00 msgr2=0x7f71a4150f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:45:18.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:18.985+0000 7f71a8ab8640 1 --2- 172.21.15.16:0/3111432715 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f71a4154e00 0x7f71a4150f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:18.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:18.985+0000 7f71a8ab8640 1 -- 172.21.15.16:0/3111432715 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71a415daf0 con 0x7f71a415b3e0 2023-05-22T15:45:19.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:18.985+0000 7f71a37fe640 1 --2- 172.21.15.16:0/3111432715 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f71a4154e00 0x7f71a4150f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T15:45:19.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:18.985+0000 7f71a8ab8640 1 --2- 172.21.15.16:0/3111432715 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f71a415b3e0 0x7f71a40b3140 secure :-1 s=READY pgs=1701 cs=0 l=1 rev1=1 crypto rx=0x7f719c01fc50 tx=0x7f7184005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:45:19.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:18.985+0000 7f71a0ff9640 1 -- 172.21.15.16:0/3111432715 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f719c0b5070 con 0x7f71a415b3e0 2023-05-22T15:45:19.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:18.985+0000 7f71a0ff9640 1 -- 172.21.15.16:0/3111432715 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f719c001480 con 0x7f71a415b3e0 2023-05-22T15:45:19.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:18.985+0000 7f71aa51b640 1 -- 172.21.15.16:0/3111432715 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71a415d810 con 0x7f71a415b3e0 2023-05-22T15:45:19.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:18.985+0000 7f71a0ff9640 1 -- 172.21.15.16:0/3111432715 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f719c0049f0 con 0x7f71a415b3e0 2023-05-22T15:45:19.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:18.985+0000 7f71aa51b640 1 -- 172.21.15.16:0/3111432715 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71a415ddd0 con 0x7f71a415b3e0 2023-05-22T15:45:19.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:18.989+0000 7f71aa51b640 1 -- 172.21.15.16:0/3111432715 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71a404dbe0 con 0x7f71a415b3e0 2023-05-22T15:45:19.001 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:18.989+0000 7f71a0ff9640 1 -- 172.21.15.16:0/3111432715 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 54) v1 ==== 84085+0+0 (secure 0 0 0) 0x7f719c0ad030 con 0x7f71a415b3e0 2023-05-22T15:45:19.001 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:18.989+0000 7f71a0ff9640 1 --2- 172.21.15.16:0/3111432715 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f7188065250 0x7f7188067700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:45:19.001 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:18.989+0000 7f71a0ff9640 1 -- 172.21.15.16:0/3111432715 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(133..133 src has 1..133) v4 ==== 6726+0+0 (secure 0 0 0) 0x7f719c11ef50 con 0x7f71a415b3e0 2023-05-22T15:45:19.001 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:18.989+0000 7f71a3fff640 1 --2- 172.21.15.16:0/3111432715 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f7188065250 0x7f7188067700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:45:19.002 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:18.989+0000 7f71a3fff640 1 --2- 172.21.15.16:0/3111432715 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f7188065250 0x7f7188067700 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f7194000c10 tx=0x7f7194009040 comp rx=0 tx=0).ready entity=mgr.8733 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:45:19.002 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:18.993+0000 7f71a0ff9640 1 -- 172.21.15.16:0/3111432715 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f719c0ecc70 con 0x7f71a415b3e0 2023-05-22T15:45:19.167 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:19.165+0000 7f71aa51b640 1 -- 172.21.15.16:0/3111432715 --> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "snap_test_dir1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f71a415e010 con 0x7f7188065250 2023-05-22T15:45:19.181 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:19.177+0000 7f71a0ff9640 1 -- 172.21.15.16:0/3111432715 <== mgr.8733 v2:172.21.15.16:6832/36921 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+66 (secure 0 0 0) 0x7f71a415e010 con 0x7f7188065250 2023-05-22T15:45:19.181 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:45:19.181 INFO:teuthology.orchestra.run.smithi016.stdout:{"path": "snap_test_dir1", "schedule": ["1M"], "retention": [{}]} 2023-05-22T15:45:19.183 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:19.181+0000 7f71aa51b640 1 -- 172.21.15.16:0/3111432715 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f7188065250 msgr2=0x7f7188067700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:45:19.183 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:19.181+0000 7f71aa51b640 1 --2- 172.21.15.16:0/3111432715 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f7188065250 0x7f7188067700 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f7194000c10 tx=0x7f7194009040 comp rx=0 tx=0).stop 2023-05-22T15:45:19.184 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:19.181+0000 7f71aa51b640 1 -- 172.21.15.16:0/3111432715 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f71a415b3e0 msgr2=0x7f71a40b3140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:45:19.184 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:19.181+0000 7f71aa51b640 1 --2- 172.21.15.16:0/3111432715 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f71a415b3e0 0x7f71a40b3140 secure :-1 s=READY pgs=1701 cs=0 l=1 rev1=1 crypto rx=0x7f719c01fc50 tx=0x7f7184005800 comp rx=0 tx=0).stop 2023-05-22T15:45:19.184 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:19.181+0000 7f71aa51b640 1 -- 172.21.15.16:0/3111432715 shutdown_connections 2023-05-22T15:45:19.184 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:19.181+0000 7f71aa51b640 1 --2- 172.21.15.16:0/3111432715 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f71a415b3e0 0x7f71a40b3140 unknown :-1 s=CLOSED pgs=1701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:19.184 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:19.181+0000 7f71aa51b640 1 --2- 172.21.15.16:0/3111432715 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f71a4154e00 0x7f71a4150f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:19.185 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:19.181+0000 7f71aa51b640 1 --2- 172.21.15.16:0/3111432715 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f7188065250 0x7f7188067700 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:19.185 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:19.181+0000 7f71aa51b640 1 --2- 172.21.15.16:0/3111432715 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7184005180 0x7f71a41521d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:45:19.185 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:19.181+0000 7f71aa51b640 1 -- 172.21.15.16:0/3111432715 >> 172.21.15.16:0/3111432715 conn(0x7f71a40b0cc0 msgr2=0x7f71a40afb80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:45:19.185 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:19.181+0000 7f71aa51b640 1 -- 172.21.15.16:0/3111432715 shutdown_connections 2023-05-22T15:45:19.185 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:45:19.181+0000 7f71aa51b640 1 -- 172.21.15.16:0/3111432715 wait complete. 2023-05-22T15:45:19.201 DEBUG:tasks.cephfs.test_snap_schedules:res= {"path": "snap_test_dir1", "schedule": ["1M"], "retention": [{}]} 2023-05-22T15:45:19.201 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'path': 'snap_test_dir1', 'schedule': ['1M'], 'retention': [{}]} 2023-05-22T15:45:19.201 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:45:19.202 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T15:45:19.212 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-05-22T15:45:19.213 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T15:45:19.213 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T15:45:20.214 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:45:20.215 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T15:45:20.231 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-05-22T15:45:20.231 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T15:45:20.232 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T15:45:21.233 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:45:21.234 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T15:45:21.249 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-05-22T15:45:21.250 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T15:45:21.250 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T15:45:22.251 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:45:22.252 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T15:45:22.266 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-05-22T15:45:22.267 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T15:45:22.267 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T15:45:22.359 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-05-22T15:45:22.361 DEBUG:teuthology.orchestra.run.smithi062:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-05-22T15:45:23.268 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:45:23.269 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T15:45:23.285 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-05-22T15:45:23.285 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T15:45:23.285 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T15:45:24.287 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:45:24.287 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T15:45:24.301 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-05-22T15:45:24.301 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T15:45:24.302 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T15:45:25.302 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:45:25.303 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T15:45:25.323 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-05-22T15:45:25.323 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T15:45:25.324 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T15:45:26.324 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:45:26.325 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T15:45:26.338 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-05-22T15:45:26.338 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T15:45:26.339 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T15:45:27.339 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:45:27.340 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T15:45:27.355 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-05-22T15:45:27.355 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T15:45:27.355 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T15:45:28.357 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:45:28.358 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T15:45:28.374 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-05-22T15:45:28.374 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T15:45:28.375 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T15:45:29.376 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:45:29.376 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T15:45:29.393 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-05-22T15:45:29.394 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T15:45:29.394 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T15:45:30.395 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:45:30.396 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T15:45:30.412 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-05-22T15:45:30.413 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T15:45:30.413 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T15:45:31.414 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:45:31.415 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T15:45:31.431 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-05-22T15:45:31.431 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T15:45:31.432 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T15:45:32.433 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:45:32.434 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T15:45:32.449 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-05-22T15:45:32.449 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T15:45:32.450 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T15:45:33.450 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:45:33.451 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T15:45:33.466 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-05-22T15:45:33.467 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T15:45:33.467 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T15:45:34.469 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:45:34.470 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T15:45:34.485 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-05-22T15:45:34.485 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T15:45:34.485 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T15:45:35.487 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:45:35.488 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T15:45:35.504 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-05-22T15:45:35.505 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T15:45:35.505 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T15:45:36.506 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:45:36.506 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T15:45:36.522 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-05-22T15:45:36.523 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T15:45:36.523 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T15:45:37.525 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:45:37.526 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T15:45:37.540 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-05-22T15:45:37.541 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T15:45:37.541 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T15:45:38.542 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:45:38.543 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T15:45:38.558 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-05-22T15:45:38.558 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T15:45:38.559 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T15:45:39.560 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:45:39.560 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T15:45:39.575 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-05-22T15:45:39.575 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T15:45:39.575 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T15:45:40.576 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:45:40.577 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T15:45:40.593 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-05-22T15:45:40.594 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T15:45:40.594 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T15:45:41.596 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:45:41.597 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T15:45:41.615 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-05-22T15:45:41.616 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T15:45:41.616 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T15:45:42.618 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:45:42.618 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T15:45:42.632 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-05-22T15:45:42.632 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T15:45:42.633 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T15:45:43.634 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:45:43.635 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T15:45:43.652 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-05-22T15:45:43.652 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T15:45:43.652 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T15:45:44.653 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:45:44.654 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T15:45:44.670 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-05-22T15:45:44.670 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T15:45:44.671 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T15:45:45.671 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:45:45.672 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T15:45:45.690 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-05-22T15:45:45.690 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T15:45:45.691 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T15:45:46.692 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:45:46.693 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T15:45:46.708 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-05-22T15:45:46.709 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T15:45:46.709 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T15:45:47.710 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:45:47.711 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T15:45:47.724 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-05-22T15:45:47.724 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T15:45:47.724 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T15:45:48.726 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:45:48.727 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T15:45:48.743 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-05-22T15:45:48.744 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T15:45:48.744 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T15:45:49.745 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:45:49.746 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T15:45:49.762 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-05-22T15:45:49.762 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T15:45:49.762 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T15:45:50.763 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:45:50.764 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T15:45:50.778 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-05-22T15:45:50.778 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T15:45:50.779 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T15:45:51.780 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:45:51.781 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T15:45:51.795 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-05-22T15:45:51.796 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T15:45:51.796 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T15:45:52.399 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-05-22T15:45:52.401 DEBUG:teuthology.orchestra.run.smithi062:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-05-22T15:45:52.798 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:45:52.798 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T15:45:52.813 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-05-22T15:45:52.813 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T15:45:52.813 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T15:45:53.814 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:45:53.855 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T15:45:53.871 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-05-22T15:45:53.872 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T15:45:53.872 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T15:45:54.873 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:45:54.875 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T15:45:54.890 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-05-22T15:45:54.891 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T15:45:54.891 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T15:45:55.892 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:45:55.893 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T15:45:55.910 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-05-22T15:45:55.911 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T15:45:55.911 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T15:45:56.912 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:45:56.913 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T15:45:56.931 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-05-22T15:45:56.931 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T15:45:56.932 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T15:45:57.932 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:45:57.933 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T15:45:57.950 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-05-22T15:45:57.951 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T15:45:57.951 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T15:45:58.952 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:45:58.953 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T15:45:58.967 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-05-22T15:45:58.967 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T15:45:58.967 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T15:45:59.968 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:45:59.969 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T15:45:59.983 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-05-22T15:45:59.984 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T15:45:59.984 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T15:46:00.840 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:46:00.836+0000 7f4e9b3d5640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T15:46:00.841 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:46:00.836+0000 7f4e9b3d5640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T15:46:00.841 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:46:00.836+0000 7f4e9b3d5640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T15:46:00.841 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:46:00.836+0000 7f4e9b3d5640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T15:46:00.841 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:46:00.836+0000 7f4e9b3d5640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T15:46:00.985 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:46:00.986 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T15:46:00.999 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_46_00_UTC 2023-05-22T15:46:01.000 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-05-22-15_46_00_UTC'} 2023-05-22T15:46:01.001 INFO:tasks.cephfs.test_snap_schedules:added: {'scheduled-2023-05-22-15_46_00_UTC'} 2023-05-22T15:46:01.001 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T15:46:01.001 DEBUG:tasks.cephfs.test_snap_schedules:snapshots added=['scheduled-2023-05-22-15_46_00_UTC'] 2023-05-22T15:46:01.001 DEBUG:tasks.cephfs.test_snap_schedules:exec=1684770318.8484778, now = 1684770361.0019655, timo = 42 2023-05-22T15:46:02.003 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove --fs cephfs snap_test_dir1 2023-05-22T15:46:02.144 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:02.140+0000 7fd213fa2640 1 Processor -- start 2023-05-22T15:46:02.144 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:02.140+0000 7fd213fa2640 1 -- start start 2023-05-22T15:46:02.145 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:02.140+0000 7fd213fa2640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fd20c070c10 0x7fd20c0cc4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:02.145 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:02.140+0000 7fd213fa2640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd20c0c9e40 0x7fd20c06bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:02.146 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:02.140+0000 7fd213fa2640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd20c0ccbb0 con 0x7fd20c15d650 2023-05-22T15:46:02.146 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:02.140+0000 7fd213fa2640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd20c0cd7c0 con 0x7fd20c0c9e40 2023-05-22T15:46:02.146 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:02.140+0000 7fd213fa2640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd20c0cdaa0 con 0x7fd20c070c10 2023-05-22T15:46:02.146 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:02.140+0000 7fd21153d640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd20c0c9e40 0x7fd20c06bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:02.146 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:02.140+0000 7fd21253f640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fd20c15d650 0x7fd20c15da70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:32794/0 (socket says 172.21.15.16:32794) 2023-05-22T15:46:02.146 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:02.140+0000 7fd21153d640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd20c0c9e40 0x7fd20c06bfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59288/0 (socket says 172.21.15.16:59288) 2023-05-22T15:46:02.146 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:02.140+0000 7fd21253f640 1 -- 172.21.15.16:0/2775120356 learned_addr learned my addr 172.21.15.16:0/2775120356 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:46:02.146 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:02.140+0000 7fd211d3e640 1 --2- 172.21.15.16:0/2775120356 >> v2:172.21.15.62:3300/0 conn(0x7fd20c070c10 0x7fd20c0cc4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:02.147 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:02.140+0000 7fd21153d640 1 -- 172.21.15.16:0/2775120356 >> v2:172.21.15.62:3300/0 conn(0x7fd20c070c10 msgr2=0x7fd20c0cc4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:46:02.147 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:02.140+0000 7fd21153d640 1 --2- 172.21.15.16:0/2775120356 >> v2:172.21.15.62:3300/0 conn(0x7fd20c070c10 0x7fd20c0cc4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:02.147 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:02.140+0000 7fd21153d640 1 -- 172.21.15.16:0/2775120356 >> v1:172.21.15.16:6789/0 conn(0x7fd20c15d650 legacy=0x7fd20c15da70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:02.148 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:02.140+0000 7fd21153d640 1 -- 172.21.15.16:0/2775120356 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd20c0cdd80 con 0x7fd20c0c9e40 2023-05-22T15:46:02.148 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:02.140+0000 7fd210d3c640 1 -- 172.21.15.16:0/2775120356 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2741227745 0 0) 0x7fd20c0ccbb0 con 0x7fd20c15d650 2023-05-22T15:46:02.150 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:02.144+0000 7fd21153d640 1 --2- 172.21.15.16:0/2775120356 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd20c0c9e40 0x7fd20c06bfa0 secure :-1 s=READY pgs=1705 cs=0 l=1 rev1=1 crypto rx=0x7fd20c0cf110 tx=0x7fd20001c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f12b7e16781bbf85 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:02.151 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:02.144+0000 7fd210d3c640 1 -- 172.21.15.16:0/2775120356 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2000ab680 con 0x7fd20c0c9e40 2023-05-22T15:46:02.151 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:02.144+0000 7fd210d3c640 1 -- 172.21.15.16:0/2775120356 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2000ab7e0 con 0x7fd20c0c9e40 2023-05-22T15:46:02.151 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:02.144+0000 7fd210d3c640 1 -- 172.21.15.16:0/2775120356 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2000abb70 con 0x7fd20c0c9e40 2023-05-22T15:46:02.151 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:02.144+0000 7fd213fa2640 1 -- 172.21.15.16:0/2775120356 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd20c0c9e40 msgr2=0x7fd20c06bfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:02.152 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:02.144+0000 7fd213fa2640 1 --2- 172.21.15.16:0/2775120356 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd20c0c9e40 0x7fd20c06bfa0 secure :-1 s=READY pgs=1705 cs=0 l=1 rev1=1 crypto rx=0x7fd20c0cf110 tx=0x7fd20001c460 comp rx=0 tx=0).stop 2023-05-22T15:46:02.152 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:02.144+0000 7fd213fa2640 1 -- 172.21.15.16:0/2775120356 shutdown_connections 2023-05-22T15:46:02.152 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:02.144+0000 7fd213fa2640 1 --2- 172.21.15.16:0/2775120356 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd20c0c9e40 0x7fd20c06bfa0 unknown :-1 s=CLOSED pgs=1705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:02.152 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:02.144+0000 7fd213fa2640 1 --2- 172.21.15.16:0/2775120356 >> v2:172.21.15.62:3300/0 conn(0x7fd20c070c10 0x7fd20c0cc4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:02.153 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:02.144+0000 7fd213fa2640 1 -- 172.21.15.16:0/2775120356 >> 172.21.15.16:0/2775120356 conn(0x7fd20c069720 msgr2=0x7fd20c069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:46:02.153 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:02.144+0000 7fd213fa2640 1 -- 172.21.15.16:0/2775120356 shutdown_connections 2023-05-22T15:46:02.153 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:02.144+0000 7fd213fa2640 1 -- 172.21.15.16:0/2775120356 wait complete. 2023-05-22T15:46:02.153 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:02.144+0000 7fd213fa2640 1 Processor -- start 2023-05-22T15:46:02.154 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:02.144+0000 7fd213fa2640 1 -- start start 2023-05-22T15:46:02.156 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:02.148+0000 7fd213fa2640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd20c070c10 0x7fd20c0c7910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:02.157 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:02.148+0000 7fd213fa2640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd20c0c9e40 0x7fd20c13f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:02.157 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:02.148+0000 7fd213fa2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd20c15d650 0x7fd20c0c5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:02.157 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:02.148+0000 7fd213fa2640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd20c0c7fb0 con 0x7fd20c15d650 2023-05-22T15:46:02.157 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:02.148+0000 7fd213fa2640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd20c15bd00 con 0x7fd20c0c9e40 2023-05-22T15:46:02.158 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:02.148+0000 7fd213fa2640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd20c15bf60 con 0x7fd20c070c10 2023-05-22T15:46:02.158 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:02.148+0000 7fd21253f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd20c15d650 0x7fd20c0c5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:02.158 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:02.148+0000 7fd21253f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd20c15d650 0x7fd20c0c5170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59300/0 (socket says 172.21.15.16:59300) 2023-05-22T15:46:02.158 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:02.148+0000 7fd21253f640 1 -- 172.21.15.16:0/840300645 learned_addr learned my addr 172.21.15.16:0/840300645 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:46:02.158 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:02.148+0000 7fd21153d640 1 --2- 172.21.15.16:0/840300645 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd20c0c9e40 0x7fd20c13f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:02.159 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:02.148+0000 7fd211d3e640 1 --2- 172.21.15.16:0/840300645 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd20c070c10 0x7fd20c0c7910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:02.159 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:02.148+0000 7fd21253f640 1 -- 172.21.15.16:0/840300645 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd20c070c10 msgr2=0x7fd20c0c7910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:02.159 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:02.148+0000 7fd21253f640 1 --2- 172.21.15.16:0/840300645 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd20c070c10 0x7fd20c0c7910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:02.159 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:02.148+0000 7fd21253f640 1 -- 172.21.15.16:0/840300645 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd20c0c9e40 msgr2=0x7fd20c13f1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:02.159 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:02.148+0000 7fd21253f640 1 --2- 172.21.15.16:0/840300645 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd20c0c9e40 0x7fd20c13f1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:02.160 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:02.148+0000 7fd21253f640 1 -- 172.21.15.16:0/840300645 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd200009420 con 0x7fd20c15d650 2023-05-22T15:46:02.160 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:02.148+0000 7fd21253f640 1 --2- 172.21.15.16:0/840300645 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd20c15d650 0x7fd20c0c5170 secure :-1 s=READY pgs=1706 cs=0 l=1 rev1=1 crypto rx=0x7fd20400c780 tx=0x7fd20400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:02.160 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:02.148+0000 7fd20a7fc640 1 -- 172.21.15.16:0/840300645 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd20400ce90 con 0x7fd20c15d650 2023-05-22T15:46:02.160 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:02.148+0000 7fd20a7fc640 1 -- 172.21.15.16:0/840300645 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd20400a640 con 0x7fd20c15d650 2023-05-22T15:46:02.160 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:02.148+0000 7fd20a7fc640 1 -- 172.21.15.16:0/840300645 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd20400a9d0 con 0x7fd20c15d650 2023-05-22T15:46:02.160 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:02.148+0000 7fd213fa2640 1 -- 172.21.15.16:0/840300645 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd20c0cce90 con 0x7fd20c15d650 2023-05-22T15:46:02.160 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:02.148+0000 7fd213fa2640 1 -- 172.21.15.16:0/840300645 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd20c06c4e0 con 0x7fd20c15d650 2023-05-22T15:46:02.161 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:02.148+0000 7fd213fa2640 1 -- 172.21.15.16:0/840300645 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd20c0ceab0 con 0x7fd20c15d650 2023-05-22T15:46:02.161 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:02.148+0000 7fd20a7fc640 1 -- 172.21.15.16:0/840300645 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 55) v1 ==== 84406+0+0 (secure 0 0 0) 0x7fd20400ab30 con 0x7fd20c15d650 2023-05-22T15:46:02.161 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:02.148+0000 7fd20a7fc640 1 --2- 172.21.15.16:0/840300645 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd1f8065580 0x7fd1f8067a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:02.161 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:02.148+0000 7fd20a7fc640 1 -- 172.21.15.16:0/840300645 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(133..133 src has 1..133) v4 ==== 6726+0+0 (secure 0 0 0) 0x7fd204082330 con 0x7fd20c15d650 2023-05-22T15:46:02.161 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:02.148+0000 7fd211d3e640 1 --2- 172.21.15.16:0/840300645 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd1f8065580 0x7fd1f8067a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:02.161 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:02.152+0000 7fd211d3e640 1 --2- 172.21.15.16:0/840300645 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd1f8065580 0x7fd1f8067a30 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fd1fc00b9d0 tx=0x7fd1fc008040 comp rx=0 tx=0).ready entity=mgr.8733 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:02.162 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:02.152+0000 7fd20a7fc640 1 -- 172.21.15.16:0/840300645 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fd20404fb40 con 0x7fd20c15d650 2023-05-22T15:46:02.331 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:02.328+0000 7fd213fa2640 1 -- 172.21.15.16:0/840300645 --> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "fs": "cephfs", "path": "snap_test_dir1", "target": ["mon-mgr", ""]}) v1 -- 0x7fd20c04dbe0 con 0x7fd1f8065580 2023-05-22T15:46:02.475 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:02.468+0000 7fd20a7fc640 1 -- 172.21.15.16:0/840300645 <== mgr.8733 v2:172.21.15.16:6832/36921 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7fd20c04dbe0 con 0x7fd1f8065580 2023-05-22T15:46:02.475 INFO:teuthology.orchestra.run.smithi016.stdout:Schedule removed for path snap_test_dir1 2023-05-22T15:46:02.478 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:02.472+0000 7fd213fa2640 1 -- 172.21.15.16:0/840300645 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd1f8065580 msgr2=0x7fd1f8067a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:02.478 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:02.472+0000 7fd213fa2640 1 --2- 172.21.15.16:0/840300645 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd1f8065580 0x7fd1f8067a30 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fd1fc00b9d0 tx=0x7fd1fc008040 comp rx=0 tx=0).stop 2023-05-22T15:46:02.478 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:02.472+0000 7fd213fa2640 1 -- 172.21.15.16:0/840300645 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd20c15d650 msgr2=0x7fd20c0c5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:02.479 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:02.472+0000 7fd213fa2640 1 --2- 172.21.15.16:0/840300645 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd20c15d650 0x7fd20c0c5170 secure :-1 s=READY pgs=1706 cs=0 l=1 rev1=1 crypto rx=0x7fd20400c780 tx=0x7fd20400cc40 comp rx=0 tx=0).stop 2023-05-22T15:46:02.479 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:02.472+0000 7fd213fa2640 1 -- 172.21.15.16:0/840300645 shutdown_connections 2023-05-22T15:46:02.479 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:02.472+0000 7fd213fa2640 1 --2- 172.21.15.16:0/840300645 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd20c15d650 0x7fd20c0c5170 unknown :-1 s=CLOSED pgs=1706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:02.479 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:02.472+0000 7fd213fa2640 1 --2- 172.21.15.16:0/840300645 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd20c0c9e40 0x7fd20c13f1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:02.479 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:02.472+0000 7fd213fa2640 1 --2- 172.21.15.16:0/840300645 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd1f8065580 0x7fd1f8067a30 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:02.479 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:02.472+0000 7fd213fa2640 1 --2- 172.21.15.16:0/840300645 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd20c070c10 0x7fd20c0c7910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:02.480 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:02.472+0000 7fd213fa2640 1 -- 172.21.15.16:0/840300645 >> 172.21.15.16:0/840300645 conn(0x7fd20c069720 msgr2=0x7fd20c14f820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:46:02.480 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:02.472+0000 7fd213fa2640 1 -- 172.21.15.16:0/840300645 shutdown_connections 2023-05-22T15:46:02.480 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:02.472+0000 7fd213fa2640 1 -- 172.21.15.16:0/840300645 wait complete. 2023-05-22T15:46:02.498 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path snap_test_dir1 2023-05-22T15:46:02.498 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:46:02.499 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T15:46:02.510 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_46_00_UTC 2023-05-22T15:46:02.511 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-05-22-15_46_00_UTC 2023-05-22T15:46:02.511 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:46:02.512 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-05-22-15_46_00_UTC) 2023-05-22T15:46:02.571 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:46:02.572 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1) 2023-05-22T15:46:02.629 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2023-05-22T15:46:02.809 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:02.804+0000 7f4ea8b05640 1 Processor -- start 2023-05-22T15:46:02.810 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:02.804+0000 7f4ea8b05640 1 -- start start 2023-05-22T15:46:02.812 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:02.804+0000 7f4ea8b05640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f4ea4159970 0x7f4ea4159d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:02.812 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:02.804+0000 7f4ea8b05640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f4ea415d680 0x7f4ea40cb260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:02.813 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:02.804+0000 7f4ea8b05640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ea4070c30 con 0x7f4ea415db00 2023-05-22T15:46:02.813 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:02.804+0000 7f4ea8b05640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ea4070e60 con 0x7f4ea415d680 2023-05-22T15:46:02.813 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:02.804+0000 7f4ea8b05640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ea4071140 con 0x7f4ea4159970 2023-05-22T15:46:02.813 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:02.804+0000 7f4ea1d9b640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f4ea415d680 0x7f4ea40cb260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:02.814 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:02.804+0000 7f4ea2d9d640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f4ea415db00 0x7f4ea406b6c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:32810/0 (socket says 172.21.15.16:32810) 2023-05-22T15:46:02.814 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:02.804+0000 7f4ea1d9b640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f4ea415d680 0x7f4ea40cb260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59308/0 (socket says 172.21.15.16:59308) 2023-05-22T15:46:02.814 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:02.804+0000 7f4ea2d9d640 1 -- 172.21.15.16:0/25690808 learned_addr learned my addr 172.21.15.16:0/25690808 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:46:02.814 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:02.808+0000 7f4ea259c640 1 --2- 172.21.15.16:0/25690808 >> v2:172.21.15.62:3300/0 conn(0x7f4ea4159970 0x7f4ea4159d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:02.814 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:02.808+0000 7f4ea1d9b640 1 -- 172.21.15.16:0/25690808 >> v2:172.21.15.62:3300/0 conn(0x7f4ea4159970 msgr2=0x7f4ea4159d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:46:02.815 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:02.808+0000 7f4ea1d9b640 1 --2- 172.21.15.16:0/25690808 >> v2:172.21.15.62:3300/0 conn(0x7f4ea4159970 0x7f4ea4159d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:02.815 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:02.808+0000 7f4ea1d9b640 1 -- 172.21.15.16:0/25690808 >> v1:172.21.15.16:6789/0 conn(0x7f4ea415db00 legacy=0x7f4ea406b6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:02.815 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:02.808+0000 7f4ea1d9b640 1 -- 172.21.15.16:0/25690808 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ea4071420 con 0x7f4ea415d680 2023-05-22T15:46:02.815 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:02.808+0000 7f4ea159a640 1 -- 172.21.15.16:0/25690808 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2329198205 0 0) 0x7f4ea4070c30 con 0x7f4ea415db00 2023-05-22T15:46:02.817 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:02.808+0000 7f4ea1d9b640 1 --2- 172.21.15.16:0/25690808 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ea415d680 0x7f4ea40cb260 secure :-1 s=READY pgs=1708 cs=0 l=1 rev1=1 crypto rx=0x7f4ea406daf0 tx=0x7f4e9801c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d74871accd1fd053 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:02.818 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:02.808+0000 7f4ea159a640 1 -- 172.21.15.16:0/25690808 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e980ac620 con 0x7f4ea415d680 2023-05-22T15:46:02.818 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:02.808+0000 7f4ea159a640 1 -- 172.21.15.16:0/25690808 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e980ac780 con 0x7f4ea415d680 2023-05-22T15:46:02.818 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:02.808+0000 7f4ea159a640 1 -- 172.21.15.16:0/25690808 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e980acb10 con 0x7f4ea415d680 2023-05-22T15:46:02.818 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:02.808+0000 7f4ea8b05640 1 -- 172.21.15.16:0/25690808 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ea415d680 msgr2=0x7f4ea40cb260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:02.818 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:02.808+0000 7f4ea8b05640 1 --2- 172.21.15.16:0/25690808 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ea415d680 0x7f4ea40cb260 secure :-1 s=READY pgs=1708 cs=0 l=1 rev1=1 crypto rx=0x7f4ea406daf0 tx=0x7f4e9801c3a0 comp rx=0 tx=0).stop 2023-05-22T15:46:02.818 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:02.808+0000 7f4ea8b05640 1 -- 172.21.15.16:0/25690808 shutdown_connections 2023-05-22T15:46:02.818 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:02.808+0000 7f4ea8b05640 1 --2- 172.21.15.16:0/25690808 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ea415d680 0x7f4ea40cb260 unknown :-1 s=CLOSED pgs=1708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:02.819 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:02.808+0000 7f4ea8b05640 1 --2- 172.21.15.16:0/25690808 >> v2:172.21.15.62:3300/0 conn(0x7f4ea4159970 0x7f4ea4159d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:02.819 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:02.808+0000 7f4ea8b05640 1 -- 172.21.15.16:0/25690808 >> 172.21.15.16:0/25690808 conn(0x7f4ea4069720 msgr2=0x7f4ea4069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:46:02.819 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:02.808+0000 7f4ea8b05640 1 -- 172.21.15.16:0/25690808 shutdown_connections 2023-05-22T15:46:02.819 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:02.808+0000 7f4ea8b05640 1 -- 172.21.15.16:0/25690808 wait complete. 2023-05-22T15:46:02.819 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:02.812+0000 7f4ea8b05640 1 Processor -- start 2023-05-22T15:46:02.819 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:02.812+0000 7f4ea8b05640 1 -- start start 2023-05-22T15:46:02.819 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:02.812+0000 7f4ea8b05640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ea4159970 0x7f4ea4145b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:02.820 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:02.812+0000 7f4ea8b05640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4ea415d680 0x7f4ea4152fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:02.820 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:02.812+0000 7f4ea8b05640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4ea415db00 0x7f4ea4153510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:02.820 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:02.812+0000 7f4ea8b05640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ea4155a60 con 0x7f4ea4159970 2023-05-22T15:46:02.820 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:02.812+0000 7f4ea8b05640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ea4154ca0 con 0x7f4ea415d680 2023-05-22T15:46:02.820 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:02.812+0000 7f4ea8b05640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ea4154f80 con 0x7f4ea415db00 2023-05-22T15:46:02.820 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:02.812+0000 7f4ea259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ea4159970 0x7f4ea4145b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:02.820 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:02.812+0000 7f4ea259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ea4159970 0x7f4ea4145b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59324/0 (socket says 172.21.15.16:59324) 2023-05-22T15:46:02.821 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:02.812+0000 7f4ea259c640 1 -- 172.21.15.16:0/3603082917 learned_addr learned my addr 172.21.15.16:0/3603082917 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:46:02.821 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:02.812+0000 7f4ea1d9b640 1 --2- 172.21.15.16:0/3603082917 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4ea415d680 0x7f4ea4152fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:02.821 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:02.812+0000 7f4ea2d9d640 1 --2- 172.21.15.16:0/3603082917 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4ea415db00 0x7f4ea4153510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:02.821 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:02.812+0000 7f4ea259c640 1 -- 172.21.15.16:0/3603082917 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4ea415db00 msgr2=0x7f4ea4153510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:02.821 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:02.812+0000 7f4ea259c640 1 --2- 172.21.15.16:0/3603082917 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4ea415db00 0x7f4ea4153510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:02.821 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:02.812+0000 7f4ea259c640 1 -- 172.21.15.16:0/3603082917 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4ea415d680 msgr2=0x7f4ea4152fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:02.821 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:02.812+0000 7f4ea259c640 1 --2- 172.21.15.16:0/3603082917 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4ea415d680 0x7f4ea4152fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:02.822 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:02.812+0000 7f4ea259c640 1 -- 172.21.15.16:0/3603082917 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e98009420 con 0x7f4ea4159970 2023-05-22T15:46:02.822 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:02.812+0000 7f4ea2d9d640 1 --2- 172.21.15.16:0/3603082917 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4ea415db00 0x7f4ea4153510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:46:02.822 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:02.812+0000 7f4ea259c640 1 --2- 172.21.15.16:0/3603082917 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ea4159970 0x7f4ea4145b40 secure :-1 s=READY pgs=1709 cs=0 l=1 rev1=1 crypto rx=0x7f4e9400d800 tx=0x7f4e9400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:02.822 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:02.812+0000 7f4e92ffd640 1 -- 172.21.15.16:0/3603082917 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e9400b6c0 con 0x7f4ea4159970 2023-05-22T15:46:02.822 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:02.812+0000 7f4e92ffd640 1 -- 172.21.15.16:0/3603082917 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e9400b820 con 0x7f4ea4159970 2023-05-22T15:46:02.822 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:02.812+0000 7f4e92ffd640 1 -- 172.21.15.16:0/3603082917 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e9400bb90 con 0x7f4ea4159970 2023-05-22T15:46:02.822 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:02.812+0000 7f4ea8b05640 1 -- 172.21.15.16:0/3603082917 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ea40cae20 con 0x7f4ea4159970 2023-05-22T15:46:02.823 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:02.812+0000 7f4ea8b05640 1 -- 172.21.15.16:0/3603082917 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ea40cac90 con 0x7f4ea4159970 2023-05-22T15:46:02.827 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:02.812+0000 7f4ea8b05640 1 -- 172.21.15.16:0/3603082917 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ea406bdd0 con 0x7f4ea4159970 2023-05-22T15:46:02.828 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:02.816+0000 7f4e92ffd640 1 -- 172.21.15.16:0/3603082917 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 56) v1 ==== 84463+0+0 (secure 0 0 0) 0x7f4e9400bcf0 con 0x7f4ea4159970 2023-05-22T15:46:02.828 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:02.816+0000 7f4e92ffd640 1 --2- 172.21.15.16:0/3603082917 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f4e84065610 0x7f4e84067ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:02.828 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:02.816+0000 7f4e92ffd640 1 -- 172.21.15.16:0/3603082917 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(133..133 src has 1..133) v4 ==== 6726+0+0 (secure 0 0 0) 0x7f4e94083650 con 0x7f4ea4159970 2023-05-22T15:46:02.828 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:02.816+0000 7f4ea1d9b640 1 --2- 172.21.15.16:0/3603082917 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f4e84065610 0x7f4e84067ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:02.828 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:02.816+0000 7f4ea1d9b640 1 --2- 172.21.15.16:0/3603082917 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f4e84065610 0x7f4e84067ac0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f4e980ae000 tx=0x7f4e980a7000 comp rx=0 tx=0).ready entity=mgr.8733 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:02.828 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:02.820+0000 7f4e92ffd640 1 -- 172.21.15.16:0/3603082917 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f4e94054660 con 0x7f4ea4159970 2023-05-22T15:46:03.017 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:03.012+0000 7f4ea8b05640 1 -- 172.21.15.16:0/3603082917 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7f4ea404dbe0 con 0x7f4ea4159970 2023-05-22T15:46:03.814 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:03.808+0000 7f4e92ffd640 1 -- 172.21.15.16:0/3603082917 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mgrmap(e 57) v1 ==== 84446+0+0 (secure 0 0 0) 0x7f4e94051000 con 0x7f4ea4159970 2023-05-22T15:46:03.814 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:03.812+0000 7f4e92ffd640 1 -- 172.21.15.16:0/3603082917 <== mon.0 v2:172.21.15.16:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v57) v1 ==== 93+0+0 (secure 0 0 0) 0x7f4e94054660 con 0x7f4ea4159970 2023-05-22T15:46:03.817 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:03.812+0000 7f4ea8b05640 1 -- 172.21.15.16:0/3603082917 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f4e84065610 msgr2=0x7f4e84067ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:03.817 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:03.812+0000 7f4ea8b05640 1 --2- 172.21.15.16:0/3603082917 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f4e84065610 0x7f4e84067ac0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f4e980ae000 tx=0x7f4e980a7000 comp rx=0 tx=0).stop 2023-05-22T15:46:03.818 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:03.812+0000 7f4ea8b05640 1 -- 172.21.15.16:0/3603082917 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ea4159970 msgr2=0x7f4ea4145b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:03.818 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:03.812+0000 7f4ea8b05640 1 --2- 172.21.15.16:0/3603082917 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ea4159970 0x7f4ea4145b40 secure :-1 s=READY pgs=1709 cs=0 l=1 rev1=1 crypto rx=0x7f4e9400d800 tx=0x7f4e9400dcc0 comp rx=0 tx=0).stop 2023-05-22T15:46:03.818 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:03.812+0000 7f4ea8b05640 1 -- 172.21.15.16:0/3603082917 shutdown_connections 2023-05-22T15:46:03.818 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:03.812+0000 7f4ea8b05640 1 --2- 172.21.15.16:0/3603082917 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f4e84065610 0x7f4e84067ac0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:03.818 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:03.812+0000 7f4ea8b05640 1 --2- 172.21.15.16:0/3603082917 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4ea415db00 0x7f4ea4153510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:03.818 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:03.812+0000 7f4ea8b05640 1 --2- 172.21.15.16:0/3603082917 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4ea415d680 0x7f4ea4152fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:03.818 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:03.812+0000 7f4ea8b05640 1 --2- 172.21.15.16:0/3603082917 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ea4159970 0x7f4ea4145b40 unknown :-1 s=CLOSED pgs=1709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:03.819 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:03.812+0000 7f4ea8b05640 1 -- 172.21.15.16:0/3603082917 >> 172.21.15.16:0/3603082917 conn(0x7f4ea4069720 msgr2=0x7f4ea40c9550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:46:03.819 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:03.812+0000 7f4ea8b05640 1 -- 172.21.15.16:0/3603082917 shutdown_connections 2023-05-22T15:46:03.819 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:03.812+0000 7f4ea8b05640 1 -- 172.21.15.16:0/3603082917 wait complete. 2023-05-22T15:46:03.837 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-05-22T15:46:03.837 DEBUG:teuthology.orchestra.run.smithi016:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-05-22T15:46:03.888 DEBUG:teuthology.orchestra.run.smithi062:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-05-22T15:46:03.917 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-05-22T15:46:03.918 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-05-22T15:46:03.918 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2023-05-22T15:46:03.924 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-05-22T15:46:03.941 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:46:03.940+0000 7f4a1c01d180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-05-22T15:46:03.942 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:46:03.936+0000 7f614e4ed180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-05-22T15:46:04.057 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.052+0000 7f2012a2c640 1 Processor -- start 2023-05-22T15:46:04.057 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.052+0000 7f2012a2c640 1 -- start start 2023-05-22T15:46:04.058 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.052+0000 7f2012a2c640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f200c06e220 0x7f200c06e900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:04.058 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.052+0000 7f2012a2c640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f200c069720 0x7f200c069bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:04.059 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.052+0000 7f2012a2c640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f200c06ee40 con 0x7f200c15bcc0 2023-05-22T15:46:04.059 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.052+0000 7f2012a2c640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f200c06a100 con 0x7f200c069720 2023-05-22T15:46:04.059 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.052+0000 7f2012a2c640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f200c06f0e0 con 0x7f200c06e220 2023-05-22T15:46:04.059 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.052+0000 7f200bfff640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f200c15bcc0 0x7f200c0715d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:32824/0 (socket says 172.21.15.16:32824) 2023-05-22T15:46:04.059 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.052+0000 7f200bfff640 1 -- 172.21.15.16:0/3673078325 learned_addr learned my addr 172.21.15.16:0/3673078325 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:46:04.059 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.052+0000 7f2010fc9640 1 --2- 172.21.15.16:0/3673078325 >> v2:172.21.15.16:3300/0 conn(0x7f200c069720 0x7f200c069bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:04.059 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.052+0000 7f200b7fe640 1 --2- 172.21.15.16:0/3673078325 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f200c06e220 0x7f200c06e900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:04.060 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.056+0000 7f2010fc9640 1 -- 172.21.15.16:0/3673078325 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f200c06e220 msgr2=0x7f200c06e900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:46:04.060 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.056+0000 7f2010fc9640 1 --2- 172.21.15.16:0/3673078325 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f200c06e220 0x7f200c06e900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:04.060 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.056+0000 7f2010fc9640 1 -- 172.21.15.16:0/3673078325 >> v1:172.21.15.16:6789/0 conn(0x7f200c15bcc0 legacy=0x7f200c0715d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:04.060 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.056+0000 7f200affd640 1 -- 172.21.15.16:0/3673078325 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2710847694 0 0) 0x7f200c06ee40 con 0x7f200c15bcc0 2023-05-22T15:46:04.060 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.056+0000 7f2010fc9640 1 -- 172.21.15.16:0/3673078325 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f200c070810 con 0x7f200c069720 2023-05-22T15:46:04.063 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.056+0000 7f2010fc9640 1 --2- 172.21.15.16:0/3673078325 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f200c069720 0x7f200c069bc0 secure :-1 s=READY pgs=1714 cs=0 l=1 rev1=1 crypto rx=0x7f2004009d90 tx=0x7f200401c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90c55adad68cb459 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:04.063 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.056+0000 7f200affd640 1 -- 172.21.15.16:0/3673078325 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20040a59e0 con 0x7f200c069720 2023-05-22T15:46:04.063 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.056+0000 7f200affd640 1 -- 172.21.15.16:0/3673078325 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20040a5b40 con 0x7f200c069720 2023-05-22T15:46:04.063 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.056+0000 7f200affd640 1 -- 172.21.15.16:0/3673078325 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20040ad440 con 0x7f200c069720 2023-05-22T15:46:04.063 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.056+0000 7f2012a2c640 1 -- 172.21.15.16:0/3673078325 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f200c069720 msgr2=0x7f200c069bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:04.063 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.056+0000 7f2012a2c640 1 --2- 172.21.15.16:0/3673078325 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f200c069720 0x7f200c069bc0 secure :-1 s=READY pgs=1714 cs=0 l=1 rev1=1 crypto rx=0x7f2004009d90 tx=0x7f200401c920 comp rx=0 tx=0).stop 2023-05-22T15:46:04.064 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.060+0000 7f2012a2c640 1 -- 172.21.15.16:0/3673078325 shutdown_connections 2023-05-22T15:46:04.064 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.060+0000 7f2012a2c640 1 --2- 172.21.15.16:0/3673078325 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f200c069720 0x7f200c069bc0 unknown :-1 s=CLOSED pgs=1714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:04.064 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.060+0000 7f2012a2c640 1 --2- 172.21.15.16:0/3673078325 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f200c06e220 0x7f200c06e900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:04.064 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.060+0000 7f2012a2c640 1 -- 172.21.15.16:0/3673078325 >> 172.21.15.16:0/3673078325 conn(0x7f200c0b0d30 msgr2=0x7f200c0b1130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:46:04.064 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.060+0000 7f2012a2c640 1 -- 172.21.15.16:0/3673078325 shutdown_connections 2023-05-22T15:46:04.064 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.060+0000 7f2012a2c640 1 -- 172.21.15.16:0/3673078325 wait complete. 2023-05-22T15:46:04.066 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.060+0000 7f2012a2c640 1 Processor -- start 2023-05-22T15:46:04.067 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.060+0000 7f2012a2c640 1 -- start start 2023-05-22T15:46:04.067 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.060+0000 7f2012a2c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f200c069720 0x7f200c0ae270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:04.067 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.060+0000 7f2012a2c640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f200c06e220 0x7f200c147ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:04.067 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.060+0000 7f2012a2c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f200c15bcc0 0x7f200c14ff30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:04.067 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.060+0000 7f2012a2c640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f200c151790 con 0x7f200c069720 2023-05-22T15:46:04.067 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.060+0000 7f2012a2c640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f200c14add0 con 0x7f200c15bcc0 2023-05-22T15:46:04.067 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.060+0000 7f2012a2c640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f200c150cb0 con 0x7f200c06e220 2023-05-22T15:46:04.068 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.060+0000 7f200bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f200c069720 0x7f200c0ae270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:04.068 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.060+0000 7f200bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f200c069720 0x7f200c0ae270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59354/0 (socket says 172.21.15.16:59354) 2023-05-22T15:46:04.068 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.060+0000 7f200bfff640 1 -- 172.21.15.16:0/1975643542 learned_addr learned my addr 172.21.15.16:0/1975643542 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:46:04.068 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.060+0000 7f200b7fe640 1 --2- 172.21.15.16:0/1975643542 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f200c06e220 0x7f200c147ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:04.068 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.060+0000 7f2010fc9640 1 --2- 172.21.15.16:0/1975643542 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f200c15bcc0 0x7f200c14ff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:04.068 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.060+0000 7f200bfff640 1 -- 172.21.15.16:0/1975643542 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f200c06e220 msgr2=0x7f200c147ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:04.068 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.060+0000 7f200bfff640 1 --2- 172.21.15.16:0/1975643542 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f200c06e220 0x7f200c147ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:04.069 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.060+0000 7f200bfff640 1 -- 172.21.15.16:0/1975643542 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f200c15bcc0 msgr2=0x7f200c14ff30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:04.069 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.060+0000 7f200bfff640 1 --2- 172.21.15.16:0/1975643542 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f200c15bcc0 0x7f200c14ff30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:04.069 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.060+0000 7f200bfff640 1 -- 172.21.15.16:0/1975643542 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2004009a70 con 0x7f200c069720 2023-05-22T15:46:04.069 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.060+0000 7f2010fc9640 1 --2- 172.21.15.16:0/1975643542 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f200c15bcc0 0x7f200c14ff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T15:46:04.069 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.060+0000 7f200bfff640 1 --2- 172.21.15.16:0/1975643542 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f200c069720 0x7f200c0ae270 secure :-1 s=READY pgs=1715 cs=0 l=1 rev1=1 crypto rx=0x7f200000c480 tx=0x7f200000c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:04.069 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.060+0000 7f2008ff9640 1 -- 172.21.15.16:0/1975643542 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20000074e0 con 0x7f200c069720 2023-05-22T15:46:04.069 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.060+0000 7f2008ff9640 1 -- 172.21.15.16:0/1975643542 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2000007660 con 0x7f200c069720 2023-05-22T15:46:04.069 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.060+0000 7f2012a2c640 1 -- 172.21.15.16:0/1975643542 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f200c0cd4e0 con 0x7f200c069720 2023-05-22T15:46:04.070 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.060+0000 7f2008ff9640 1 -- 172.21.15.16:0/1975643542 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2000004a30 con 0x7f200c069720 2023-05-22T15:46:04.070 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.060+0000 7f2012a2c640 1 -- 172.21.15.16:0/1975643542 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f200c0710b0 con 0x7f200c069720 2023-05-22T15:46:04.070 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.064+0000 7f2012a2c640 1 -- 172.21.15.16:0/1975643542 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f200c0cd670 con 0x7f200c069720 2023-05-22T15:46:04.071 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.064+0000 7f2008ff9640 1 -- 172.21.15.16:0/1975643542 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 57) v1 ==== 84446+0+0 (secure 0 0 0) 0x7f2000004b90 con 0x7f200c069720 2023-05-22T15:46:04.071 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.064+0000 7f2008ff9640 1 --2- 172.21.15.16:0/1975643542 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f1ff80655c0 0x7f1ff8067a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:04.071 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.064+0000 7f200b7fe640 1 -- 172.21.15.16:0/1975643542 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f1ff80655c0 msgr2=0x7f1ff8067a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/36921 2023-05-22T15:46:04.071 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.064+0000 7f2008ff9640 1 -- 172.21.15.16:0/1975643542 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(133..133 src has 1..133) v4 ==== 6726+0+0 (secure 0 0 0) 0x7f20000826e0 con 0x7f200c069720 2023-05-22T15:46:04.071 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.064+0000 7f200b7fe640 1 --2- 172.21.15.16:0/1975643542 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f1ff80655c0 0x7f1ff8067a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-05-22T15:46:04.072 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.068+0000 7f2008ff9640 1 -- 172.21.15.16:0/1975643542 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f200004fed0 con 0x7f200c069720 2023-05-22T15:46:04.232 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.228+0000 7f2012a2c640 1 -- 172.21.15.16:0/1975643542 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f200c0738f0 con 0x7f200c069720 2023-05-22T15:46:04.233 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.228+0000 7f2008ff9640 1 -- 172.21.15.16:0/1975643542 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v133) v1 ==== 74+0+18487 (secure 0 0 0) 0x7f20000534f0 con 0x7f200c069720 2023-05-22T15:46:04.234 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:46:04.234 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":133,"fsid":"b2719834-7847-4d98-90a6-f65d57f85c49","created":"2023-05-22T15:26:25.259314+0000","modified":"2023-05-22T15:45:17.648468+0000","last_up_change":"2023-05-22T15:26:36.448524+0000","last_in_change":"2023-05-22T15:26:27.675668+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-05-22T15:26:38.019781+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-05-22T15:26:43.570514+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2023-05-22T15:45:03.718869+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"131","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2023-05-22T15:45:04.168135+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"132","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9e7ff854-b25f-4cbb-a5c2-1799d9c53d2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":37021},{"type":"v1","addr":"172.21.15.16:6825","nonce":37021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":37021},{"type":"v1","addr":"172.21.15.16:6827","nonce":37021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":37021},{"type":"v1","addr":"172.21.15.16:6831","nonce":37021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":37021},{"type":"v1","addr":"172.21.15.16:6829","nonce":37021}]},"public_addr":"172.21.15.16:6825/37021","cluster_addr":"172.21.15.16:6827/37021","heartbeat_back_addr":"172.21.15.16:6831/37021","heartbeat_front_addr":"172.21.15.16:6829/37021","state":["exists","up"]},{"osd":1,"uuid":"4702fbc9-a927-4565-aca5-41dc69cc8264","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":37008},{"type":"v1","addr":"172.21.15.16:6801","nonce":37008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":37008},{"type":"v1","addr":"172.21.15.16:6803","nonce":37008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":37008},{"type":"v1","addr":"172.21.15.16:6807","nonce":37008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":37008},{"type":"v1","addr":"172.21.15.16:6805","nonce":37008}]},"public_addr":"172.21.15.16:6801/37008","cluster_addr":"172.21.15.16:6803/37008","heartbeat_back_addr":"172.21.15.16:6807/37008","heartbeat_front_addr":"172.21.15.16:6805/37008","state":["exists","up"]},{"osd":2,"uuid":"1e7bba46-4eba-47ef-bb81-5e2d371638ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":37009},{"type":"v1","addr":"172.21.15.16:6809","nonce":37009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":37009},{"type":"v1","addr":"172.21.15.16:6811","nonce":37009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":37009},{"type":"v1","addr":"172.21.15.16:6815","nonce":37009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":37009},{"type":"v1","addr":"172.21.15.16:6813","nonce":37009}]},"public_addr":"172.21.15.16:6809/37009","cluster_addr":"172.21.15.16:6811/37009","heartbeat_back_addr":"172.21.15.16:6815/37009","heartbeat_front_addr":"172.21.15.16:6813/37009","state":["exists","up"]},{"osd":3,"uuid":"383cacc0-5a88-4904-990e-498ca59ef48a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":37020},{"type":"v1","addr":"172.21.15.16:6817","nonce":37020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":37020},{"type":"v1","addr":"172.21.15.16:6819","nonce":37020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":37020},{"type":"v1","addr":"172.21.15.16:6823","nonce":37020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":37020},{"type":"v1","addr":"172.21.15.16:6821","nonce":37020}]},"public_addr":"172.21.15.16:6817/37020","cluster_addr":"172.21.15.16:6819/37020","heartbeat_back_addr":"172.21.15.16:6823/37020","heartbeat_front_addr":"172.21.15.16:6821/37020","state":["exists","up"]},{"osd":4,"uuid":"970500ef-95b7-42c1-b080-eed4fae7013d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":37353},{"type":"v1","addr":"172.21.15.62:6817","nonce":37353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6819","nonce":37353},{"type":"v1","addr":"172.21.15.62:6822","nonce":37353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37353},{"type":"v1","addr":"172.21.15.62:6831","nonce":37353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37353},{"type":"v1","addr":"172.21.15.62:6826","nonce":37353}]},"public_addr":"172.21.15.62:6817/37353","cluster_addr":"172.21.15.62:6822/37353","heartbeat_back_addr":"172.21.15.62:6831/37353","heartbeat_front_addr":"172.21.15.62:6826/37353","state":["exists","up"]},{"osd":5,"uuid":"022d67db-d4d8-445b-b1b9-8b446a00f5c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":37352},{"type":"v1","addr":"172.21.15.62:6801","nonce":37352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":37352},{"type":"v1","addr":"172.21.15.62:6803","nonce":37352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":37352},{"type":"v1","addr":"172.21.15.62:6807","nonce":37352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":37352},{"type":"v1","addr":"172.21.15.62:6805","nonce":37352}]},"public_addr":"172.21.15.62:6801/37352","cluster_addr":"172.21.15.62:6803/37352","heartbeat_back_addr":"172.21.15.62:6807/37352","heartbeat_front_addr":"172.21.15.62:6805/37352","state":["exists","up"]},{"osd":6,"uuid":"42bd372b-6688-4a1b-8407-e16e8c266e33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":37354},{"type":"v1","addr":"172.21.15.62:6820","nonce":37354}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":37354},{"type":"v1","addr":"172.21.15.62:6823","nonce":37354}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6829","nonce":37354},{"type":"v1","addr":"172.21.15.62:6830","nonce":37354}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6825","nonce":37354},{"type":"v1","addr":"172.21.15.62:6827","nonce":37354}]},"public_addr":"172.21.15.62:6820/37354","cluster_addr":"172.21.15.62:6823/37354","heartbeat_back_addr":"172.21.15.62:6830/37354","heartbeat_front_addr":"172.21.15.62:6827/37354","state":["exists","up"]},{"osd":7,"uuid":"ac70b2a3-f41b-4bf2-b227-7f2cbb55426d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37355},{"type":"v1","addr":"172.21.15.62:6809","nonce":37355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37355},{"type":"v1","addr":"172.21.15.62:6811","nonce":37355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":37355},{"type":"v1","addr":"172.21.15.62:6815","nonce":37355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":37355},{"type":"v1","addr":"172.21.15.62:6813","nonce":37355}]},"public_addr":"172.21.15.62:6809/37355","cluster_addr":"172.21.15.62:6811/37355","heartbeat_back_addr":"172.21.15.62:6815/37355","heartbeat_front_addr":"172.21.15.62:6813/37355","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.368896+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.107273+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.276888+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.287502+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.124791+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.251937+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.255679+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.232797+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:0/1950855917":"2023-05-23T15:45:17.648374+0000","172.21.15.16:0/1889818530":"2023-05-23T15:45:17.648374+0000","172.21.15.16:6833/36921":"2023-05-23T15:45:17.648374+0000","172.21.15.16:0/4160466619":"2023-05-23T15:45:17.648374+0000","172.21.15.16:0/55601757":"2023-05-23T15:45:17.648374+0000","172.21.15.16:6832/36921":"2023-05-23T15:45:17.648374+0000","172.21.15.16:0/4127127856":"2023-05-23T15:45:17.648374+0000","172.21.15.16:0/2333289485":"2023-05-23T15:45:17.648374+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-05-22T15:46:04.235 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.232+0000 7f2012a2c640 1 -- 172.21.15.16:0/1975643542 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f1ff80655c0 msgr2=0x7f1ff8067a70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-05-22T15:46:04.236 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.232+0000 7f2012a2c640 1 --2- 172.21.15.16:0/1975643542 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f1ff80655c0 0x7f1ff8067a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:04.236 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.232+0000 7f2012a2c640 1 -- 172.21.15.16:0/1975643542 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f200c069720 msgr2=0x7f200c0ae270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:04.236 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.232+0000 7f2012a2c640 1 --2- 172.21.15.16:0/1975643542 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f200c069720 0x7f200c0ae270 secure :-1 s=READY pgs=1715 cs=0 l=1 rev1=1 crypto rx=0x7f200000c480 tx=0x7f200000c940 comp rx=0 tx=0).stop 2023-05-22T15:46:04.236 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.232+0000 7f2012a2c640 1 -- 172.21.15.16:0/1975643542 shutdown_connections 2023-05-22T15:46:04.236 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.232+0000 7f2012a2c640 1 --2- 172.21.15.16:0/1975643542 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f200c15bcc0 0x7f200c14ff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:04.237 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.232+0000 7f2012a2c640 1 --2- 172.21.15.16:0/1975643542 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f1ff80655c0 0x7f1ff8067a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:04.237 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.232+0000 7f2012a2c640 1 --2- 172.21.15.16:0/1975643542 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f200c06e220 0x7f200c147ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:04.237 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.232+0000 7f2012a2c640 1 --2- 172.21.15.16:0/1975643542 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f200c069720 0x7f200c0ae270 unknown :-1 s=CLOSED pgs=1715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:04.237 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.232+0000 7f2012a2c640 1 -- 172.21.15.16:0/1975643542 >> 172.21.15.16:0/1975643542 conn(0x7f200c0b0d30 msgr2=0x7f200c15a210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:46:04.237 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.232+0000 7f2012a2c640 1 -- 172.21.15.16:0/1975643542 shutdown_connections 2023-05-22T15:46:04.237 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.232+0000 7f2012a2c640 1 -- 172.21.15.16:0/1975643542 wait complete. 2023-05-22T15:46:04.253 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-05-22T15:26:38.019781+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-05-22T15:26:43.570514+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 16, 'pool_name': 'cephfs_metadata', 'create_time': '2023-05-22T15:45:03.718869+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '131', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 17, 'pool_name': 'cephfs_data', 'create_time': '2023-05-22T15:45:04.168135+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '132', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-05-22T15:46:04.253 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-05-22T15:46:04.329 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:46:04.328+0000 7f4a1c01d180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-05-22T15:46:04.337 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:46:04.332+0000 7f614e4ed180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-05-22T15:46:04.387 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.384+0000 7fe5ca3d9640 1 Processor -- start 2023-05-22T15:46:04.387 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.384+0000 7fe5ca3d9640 1 -- start start 2023-05-22T15:46:04.388 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.384+0000 7fe5ca3d9640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fe5c415cfa0 0x7fe5c406db00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:04.389 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.384+0000 7fe5ca3d9640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe5c406e740 0x7fe5c406ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:04.389 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.384+0000 7fe5ca3d9640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5c406e040 con 0x7fe5c40cf870 2023-05-22T15:46:04.389 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.384+0000 7fe5ca3d9640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5c40cb9f0 con 0x7fe5c406e740 2023-05-22T15:46:04.389 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.384+0000 7fe5ca3d9640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5c406f100 con 0x7fe5c415cfa0 2023-05-22T15:46:04.389 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.384+0000 7fe5c8976640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe5c406e740 0x7fe5c406ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:04.389 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.384+0000 7fe5c8976640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe5c406e740 0x7fe5c406ebc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59368/0 (socket says 172.21.15.16:59368) 2023-05-22T15:46:04.390 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.384+0000 7fe5c8976640 1 -- 172.21.15.16:0/219397897 learned_addr learned my addr 172.21.15.16:0/219397897 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:46:04.390 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.384+0000 7fe5c3fff640 1 --2- 172.21.15.16:0/219397897 >> v2:172.21.15.62:3300/0 conn(0x7fe5c415cfa0 0x7fe5c406db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:04.390 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.384+0000 7fe5c8976640 1 -- 172.21.15.16:0/219397897 >> v2:172.21.15.62:3300/0 conn(0x7fe5c415cfa0 msgr2=0x7fe5c406db00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:46:04.390 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.384+0000 7fe5c8976640 1 --2- 172.21.15.16:0/219397897 >> v2:172.21.15.62:3300/0 conn(0x7fe5c415cfa0 0x7fe5c406db00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:04.390 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.384+0000 7fe5c8976640 1 -- 172.21.15.16:0/219397897 >> v1:172.21.15.62:6789/0 conn(0x7fe5c40cf870 legacy=0x7fe5c40b1fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:46:04.390 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.384+0000 7fe5c8976640 1 -- 172.21.15.16:0/219397897 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5c406e490 con 0x7fe5c406e740 2023-05-22T15:46:04.391 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.388+0000 7fe5c8976640 1 --2- 172.21.15.16:0/219397897 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe5c406e740 0x7fe5c406ebc0 secure :-1 s=READY pgs=1716 cs=0 l=1 rev1=1 crypto rx=0x7fe5c40b1d50 tx=0x7fe5bc01c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38858ed7aeb48fb7 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:04.391 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.388+0000 7fe5c2ffd640 1 -- 172.21.15.16:0/219397897 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5bc0a5820 con 0x7fe5c406e740 2023-05-22T15:46:04.391 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.388+0000 7fe5c2ffd640 1 -- 172.21.15.16:0/219397897 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5bc0a5980 con 0x7fe5c406e740 2023-05-22T15:46:04.391 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.388+0000 7fe5c2ffd640 1 -- 172.21.15.16:0/219397897 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5bc0a5d10 con 0x7fe5c406e740 2023-05-22T15:46:04.392 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.388+0000 7fe5ca3d9640 1 -- 172.21.15.16:0/219397897 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe5c406e740 msgr2=0x7fe5c406ebc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:04.392 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.388+0000 7fe5ca3d9640 1 --2- 172.21.15.16:0/219397897 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe5c406e740 0x7fe5c406ebc0 secure :-1 s=READY pgs=1716 cs=0 l=1 rev1=1 crypto rx=0x7fe5c40b1d50 tx=0x7fe5bc01c960 comp rx=0 tx=0).stop 2023-05-22T15:46:04.393 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.388+0000 7fe5ca3d9640 1 -- 172.21.15.16:0/219397897 shutdown_connections 2023-05-22T15:46:04.393 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.388+0000 7fe5ca3d9640 1 --2- 172.21.15.16:0/219397897 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe5c406e740 0x7fe5c406ebc0 unknown :-1 s=CLOSED pgs=1716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:04.393 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.388+0000 7fe5ca3d9640 1 --2- 172.21.15.16:0/219397897 >> v2:172.21.15.62:3300/0 conn(0x7fe5c415cfa0 0x7fe5c406db00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:04.393 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.388+0000 7fe5ca3d9640 1 -- 172.21.15.16:0/219397897 >> 172.21.15.16:0/219397897 conn(0x7fe5c40cc870 msgr2=0x7fe5c40cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:46:04.393 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.388+0000 7fe5ca3d9640 1 -- 172.21.15.16:0/219397897 shutdown_connections 2023-05-22T15:46:04.393 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.388+0000 7fe5ca3d9640 1 -- 172.21.15.16:0/219397897 wait complete. 2023-05-22T15:46:04.395 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.388+0000 7fe5ca3d9640 1 Processor -- start 2023-05-22T15:46:04.395 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.388+0000 7fe5ca3d9640 1 -- start start 2023-05-22T15:46:04.395 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.388+0000 7fe5ca3d9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe5c406e740 0x7fe5c4156ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:04.395 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.388+0000 7fe5ca3d9640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe5c40cf870 0x7fe5c40b2960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:04.395 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.388+0000 7fe5ca3d9640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe5c415cfa0 0x7fe5c4153f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:04.395 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.388+0000 7fe5ca3d9640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5c4157420 con 0x7fe5c406e740 2023-05-22T15:46:04.396 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.388+0000 7fe5ca3d9640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5c40b2ea0 con 0x7fe5c415cfa0 2023-05-22T15:46:04.396 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.388+0000 7fe5ca3d9640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5c40b3970 con 0x7fe5c40cf870 2023-05-22T15:46:04.396 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.388+0000 7fe5c3fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe5c406e740 0x7fe5c4156ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:04.396 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.388+0000 7fe5c3fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe5c406e740 0x7fe5c4156ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59384/0 (socket says 172.21.15.16:59384) 2023-05-22T15:46:04.396 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.388+0000 7fe5c3fff640 1 -- 172.21.15.16:0/4071612667 learned_addr learned my addr 172.21.15.16:0/4071612667 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:46:04.396 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.388+0000 7fe5c8976640 1 --2- 172.21.15.16:0/4071612667 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe5c415cfa0 0x7fe5c4153f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:04.396 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.388+0000 7fe5c37fe640 1 --2- 172.21.15.16:0/4071612667 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe5c40cf870 0x7fe5c40b2960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:04.396 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.388+0000 7fe5c3fff640 1 -- 172.21.15.16:0/4071612667 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe5c40cf870 msgr2=0x7fe5c40b2960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:04.397 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.388+0000 7fe5c3fff640 1 --2- 172.21.15.16:0/4071612667 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe5c40cf870 0x7fe5c40b2960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:04.397 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.388+0000 7fe5c3fff640 1 -- 172.21.15.16:0/4071612667 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe5c415cfa0 msgr2=0x7fe5c4153f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:04.397 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.388+0000 7fe5c3fff640 1 --2- 172.21.15.16:0/4071612667 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe5c415cfa0 0x7fe5c4153f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:04.397 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.388+0000 7fe5c3fff640 1 -- 172.21.15.16:0/4071612667 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5bc009b30 con 0x7fe5c406e740 2023-05-22T15:46:04.397 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.388+0000 7fe5c37fe640 1 --2- 172.21.15.16:0/4071612667 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe5c40cf870 0x7fe5c40b2960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T15:46:04.397 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.388+0000 7fe5c3fff640 1 --2- 172.21.15.16:0/4071612667 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe5c406e740 0x7fe5c4156ee0 secure :-1 s=READY pgs=1717 cs=0 l=1 rev1=1 crypto rx=0x7fe5b400d5f0 tx=0x7fe5b400dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:04.397 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.388+0000 7fe5c0ff9640 1 -- 172.21.15.16:0/4071612667 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5b40125f0 con 0x7fe5c406e740 2023-05-22T15:46:04.398 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.388+0000 7fe5c0ff9640 1 -- 172.21.15.16:0/4071612667 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5b4012750 con 0x7fe5c406e740 2023-05-22T15:46:04.398 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.392+0000 7fe5c0ff9640 1 -- 172.21.15.16:0/4071612667 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5b4012ae0 con 0x7fe5c406e740 2023-05-22T15:46:04.398 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.392+0000 7fe5ca3d9640 1 -- 172.21.15.16:0/4071612667 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5c40cb610 con 0x7fe5c406e740 2023-05-22T15:46:04.398 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.392+0000 7fe5ca3d9640 1 -- 172.21.15.16:0/4071612667 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5c40cb330 con 0x7fe5c406e740 2023-05-22T15:46:04.399 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.392+0000 7fe5ca3d9640 1 -- 172.21.15.16:0/4071612667 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5c415d370 con 0x7fe5c406e740 2023-05-22T15:46:04.399 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.392+0000 7fe5c0ff9640 1 -- 172.21.15.16:0/4071612667 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 57) v1 ==== 84446+0+0 (secure 0 0 0) 0x7fe5b4012c40 con 0x7fe5c406e740 2023-05-22T15:46:04.399 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.392+0000 7fe5c0ff9640 1 --2- 172.21.15.16:0/4071612667 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fe5b00655c0 0x7fe5b0067a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:04.399 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.392+0000 7fe5c37fe640 1 -- 172.21.15.16:0/4071612667 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fe5b00655c0 msgr2=0x7fe5b0067a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/36921 2023-05-22T15:46:04.399 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.392+0000 7fe5c0ff9640 1 -- 172.21.15.16:0/4071612667 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(133..133 src has 1..133) v4 ==== 6726+0+0 (secure 0 0 0) 0x7fe5b4083430 con 0x7fe5c406e740 2023-05-22T15:46:04.399 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.392+0000 7fe5c37fe640 1 --2- 172.21.15.16:0/4071612667 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fe5b00655c0 0x7fe5b0067a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-05-22T15:46:04.404 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.400+0000 7fe5c0ff9640 1 -- 172.21.15.16:0/4071612667 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fe5b40509c0 con 0x7fe5c406e740 2023-05-22T15:46:04.465 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:46:04.464+0000 7f4a1c01d180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-05-22T15:46:04.479 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:46:04.476+0000 7f614e4ed180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-05-22T15:46:04.559 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.552+0000 7fe5ca3d9640 1 -- 172.21.15.16:0/4071612667 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fe5c415cb70 con 0x7fe5c406e740 2023-05-22T15:46:04.559 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.556+0000 7fe5c0ff9640 1 -- 172.21.15.16:0/4071612667 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v133) v1 ==== 93+0+10 (secure 0 0 0) 0x7fe5b4053fe0 con 0x7fe5c406e740 2023-05-22T15:46:04.559 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2023-05-22T15:46:04.561 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.556+0000 7fe5ca3d9640 1 -- 172.21.15.16:0/4071612667 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fe5b00655c0 msgr2=0x7fe5b0067a70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-05-22T15:46:04.561 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.556+0000 7fe5ca3d9640 1 --2- 172.21.15.16:0/4071612667 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fe5b00655c0 0x7fe5b0067a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:04.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.556+0000 7fe5ca3d9640 1 -- 172.21.15.16:0/4071612667 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe5c406e740 msgr2=0x7fe5c4156ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:04.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.556+0000 7fe5ca3d9640 1 --2- 172.21.15.16:0/4071612667 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe5c406e740 0x7fe5c4156ee0 secure :-1 s=READY pgs=1717 cs=0 l=1 rev1=1 crypto rx=0x7fe5b400d5f0 tx=0x7fe5b400dab0 comp rx=0 tx=0).stop 2023-05-22T15:46:04.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.556+0000 7fe5ca3d9640 1 -- 172.21.15.16:0/4071612667 shutdown_connections 2023-05-22T15:46:04.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.556+0000 7fe5ca3d9640 1 --2- 172.21.15.16:0/4071612667 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe5c415cfa0 0x7fe5c4153f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:04.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.556+0000 7fe5ca3d9640 1 --2- 172.21.15.16:0/4071612667 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fe5b00655c0 0x7fe5b0067a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:04.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.556+0000 7fe5ca3d9640 1 --2- 172.21.15.16:0/4071612667 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe5c40cf870 0x7fe5c40b2960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:04.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.556+0000 7fe5ca3d9640 1 --2- 172.21.15.16:0/4071612667 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe5c406e740 0x7fe5c4156ee0 unknown :-1 s=CLOSED pgs=1717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:04.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.556+0000 7fe5ca3d9640 1 -- 172.21.15.16:0/4071612667 >> 172.21.15.16:0/4071612667 conn(0x7fe5c40cc870 msgr2=0x7fe5c41576a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:46:04.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.556+0000 7fe5ca3d9640 1 -- 172.21.15.16:0/4071612667 shutdown_connections 2023-05-22T15:46:04.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.556+0000 7fe5ca3d9640 1 -- 172.21.15.16:0/4071612667 wait complete. 2023-05-22T15:46:04.577 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-05-22T15:46:04.707 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.704+0000 7f2b6b37d640 1 Processor -- start 2023-05-22T15:46:04.708 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.704+0000 7f2b6b37d640 1 -- start start 2023-05-22T15:46:04.709 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.704+0000 7f2b6b37d640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2b640ce480 0x7f2b64071640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:04.709 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.704+0000 7f2b6b37d640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f2b6415c270 0x7f2b6415c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:04.709 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.704+0000 7f2b6b37d640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b64069ae0 con 0x7f2b640ce850 2023-05-22T15:46:04.709 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.704+0000 7f2b6b37d640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b6415f990 con 0x7f2b640ce480 2023-05-22T15:46:04.710 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.704+0000 7f2b6b37d640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b64069dc0 con 0x7f2b6415c270 2023-05-22T15:46:04.710 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.704+0000 7f2b69119640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2b640ce480 0x7f2b64071640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:04.710 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.704+0000 7f2b69119640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2b640ce480 0x7f2b64071640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59386/0 (socket says 172.21.15.16:59386) 2023-05-22T15:46:04.710 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.704+0000 7f2b68918640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f2b640ce850 0x7f2b64160090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:32828/0 (socket says 172.21.15.16:32828) 2023-05-22T15:46:04.710 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.704+0000 7f2b69119640 1 -- 172.21.15.16:0/1841416580 learned_addr learned my addr 172.21.15.16:0/1841416580 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:46:04.710 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.704+0000 7f2b6991a640 1 --2- 172.21.15.16:0/1841416580 >> v2:172.21.15.62:3300/0 conn(0x7f2b6415c270 0x7f2b6415c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:04.710 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.704+0000 7f2b69119640 1 -- 172.21.15.16:0/1841416580 >> v2:172.21.15.62:3300/0 conn(0x7f2b6415c270 msgr2=0x7f2b6415c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:46:04.711 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.704+0000 7f2b69119640 1 --2- 172.21.15.16:0/1841416580 >> v2:172.21.15.62:3300/0 conn(0x7f2b6415c270 0x7f2b6415c6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:04.711 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.704+0000 7f2b69119640 1 -- 172.21.15.16:0/1841416580 >> v1:172.21.15.16:6789/0 conn(0x7f2b640ce850 legacy=0x7f2b64160090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:04.711 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.704+0000 7f2b69119640 1 -- 172.21.15.16:0/1841416580 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b6406a0a0 con 0x7f2b640ce480 2023-05-22T15:46:04.711 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.704+0000 7f2b5bfff640 1 -- 172.21.15.16:0/1841416580 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 274973617 0 0) 0x7f2b64069ae0 con 0x7f2b640ce850 2023-05-22T15:46:04.711 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.704+0000 7f2b6991a640 1 --2- 172.21.15.16:0/1841416580 >> v2:172.21.15.62:3300/0 conn(0x7f2b6415c270 0x7f2b6415c6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:46:04.715 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.708+0000 7f2b69119640 1 --2- 172.21.15.16:0/1841416580 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b640ce480 0x7f2b64071640 secure :-1 s=READY pgs=1719 cs=0 l=1 rev1=1 crypto rx=0x7f2b640aeb30 tx=0x7f2b5c01c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61991a3c0eaff5b3 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:04.715 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.708+0000 7f2b5bfff640 1 -- 172.21.15.16:0/1841416580 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b5c0aa650 con 0x7f2b640ce480 2023-05-22T15:46:04.715 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.708+0000 7f2b5bfff640 1 -- 172.21.15.16:0/1841416580 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b5c0aa7b0 con 0x7f2b640ce480 2023-05-22T15:46:04.716 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.708+0000 7f2b5bfff640 1 -- 172.21.15.16:0/1841416580 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b5c0aab40 con 0x7f2b640ce480 2023-05-22T15:46:04.716 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.708+0000 7f2b6b37d640 1 -- 172.21.15.16:0/1841416580 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b640ce480 msgr2=0x7f2b64071640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:04.716 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.708+0000 7f2b6b37d640 1 --2- 172.21.15.16:0/1841416580 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b640ce480 0x7f2b64071640 secure :-1 s=READY pgs=1719 cs=0 l=1 rev1=1 crypto rx=0x7f2b640aeb30 tx=0x7f2b5c01c4d0 comp rx=0 tx=0).stop 2023-05-22T15:46:04.716 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.708+0000 7f2b6b37d640 1 -- 172.21.15.16:0/1841416580 shutdown_connections 2023-05-22T15:46:04.716 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.708+0000 7f2b6b37d640 1 --2- 172.21.15.16:0/1841416580 >> v2:172.21.15.62:3300/0 conn(0x7f2b6415c270 0x7f2b6415c6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:04.716 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.708+0000 7f2b6b37d640 1 --2- 172.21.15.16:0/1841416580 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b640ce480 0x7f2b64071640 unknown :-1 s=CLOSED pgs=1719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:04.716 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.708+0000 7f2b6b37d640 1 -- 172.21.15.16:0/1841416580 >> 172.21.15.16:0/1841416580 conn(0x7f2b640cc870 msgr2=0x7f2b640cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:46:04.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.708+0000 7f2b6b37d640 1 -- 172.21.15.16:0/1841416580 shutdown_connections 2023-05-22T15:46:04.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.708+0000 7f2b6b37d640 1 -- 172.21.15.16:0/1841416580 wait complete. 2023-05-22T15:46:04.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.708+0000 7f2b6b37d640 1 Processor -- start 2023-05-22T15:46:04.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.708+0000 7f2b6b37d640 1 -- start start 2023-05-22T15:46:04.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.708+0000 7f2b6b37d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b640ce480 0x7f2b64152540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:04.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.708+0000 7f2b6b37d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2b640ce850 0x7f2b64158170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:04.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.708+0000 7f2b6b37d640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2b6415c270 0x7f2b6414e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:04.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.708+0000 7f2b6b37d640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b64152d90 con 0x7f2b640ce480 2023-05-22T15:46:04.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.708+0000 7f2b6b37d640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b64153070 con 0x7f2b640ce850 2023-05-22T15:46:04.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.708+0000 7f2b6b37d640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b64153350 con 0x7f2b6415c270 2023-05-22T15:46:04.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.708+0000 7f2b69119640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b640ce480 0x7f2b64152540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:04.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.708+0000 7f2b69119640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b640ce480 0x7f2b64152540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59396/0 (socket says 172.21.15.16:59396) 2023-05-22T15:46:04.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.708+0000 7f2b69119640 1 -- 172.21.15.16:0/1915762324 learned_addr learned my addr 172.21.15.16:0/1915762324 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:46:04.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.708+0000 7f2b6991a640 1 --2- 172.21.15.16:0/1915762324 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2b6415c270 0x7f2b6414e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:04.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.708+0000 7f2b68918640 1 --2- 172.21.15.16:0/1915762324 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2b640ce850 0x7f2b64158170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:04.719 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.708+0000 7f2b69119640 1 -- 172.21.15.16:0/1915762324 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2b6415c270 msgr2=0x7f2b6414e0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:04.719 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.708+0000 7f2b69119640 1 --2- 172.21.15.16:0/1915762324 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2b6415c270 0x7f2b6414e0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:04.719 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.708+0000 7f2b69119640 1 -- 172.21.15.16:0/1915762324 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2b640ce850 msgr2=0x7f2b64158170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:04.719 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.708+0000 7f2b69119640 1 --2- 172.21.15.16:0/1915762324 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2b640ce850 0x7f2b64158170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:04.719 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.708+0000 7f2b69119640 1 -- 172.21.15.16:0/1915762324 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b5c0094d0 con 0x7f2b640ce480 2023-05-22T15:46:04.719 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.708+0000 7f2b6991a640 1 --2- 172.21.15.16:0/1915762324 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2b6415c270 0x7f2b6414e0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:46:04.719 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.708+0000 7f2b68918640 1 --2- 172.21.15.16:0/1915762324 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2b640ce850 0x7f2b64158170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:46:04.719 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.708+0000 7f2b69119640 1 --2- 172.21.15.16:0/1915762324 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b640ce480 0x7f2b64152540 secure :-1 s=READY pgs=1720 cs=0 l=1 rev1=1 crypto rx=0x7f2b5c0ac040 tx=0x7f2b5c01ad10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:04.720 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.712+0000 7f2b59ffb640 1 -- 172.21.15.16:0/1915762324 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b5c0aae70 con 0x7f2b640ce480 2023-05-22T15:46:04.720 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.712+0000 7f2b59ffb640 1 -- 172.21.15.16:0/1915762324 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b5c0b7070 con 0x7f2b640ce480 2023-05-22T15:46:04.720 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.712+0000 7f2b6b37d640 1 -- 172.21.15.16:0/1915762324 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b640698b0 con 0x7f2b640ce480 2023-05-22T15:46:04.720 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.712+0000 7f2b59ffb640 1 -- 172.21.15.16:0/1915762324 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b5c004220 con 0x7f2b640ce480 2023-05-22T15:46:04.720 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.712+0000 7f2b6b37d640 1 -- 172.21.15.16:0/1915762324 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b64069720 con 0x7f2b640ce480 2023-05-22T15:46:04.721 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.712+0000 7f2b6b37d640 1 -- 172.21.15.16:0/1915762324 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b640cec70 con 0x7f2b640ce480 2023-05-22T15:46:04.721 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.712+0000 7f2b59ffb640 1 -- 172.21.15.16:0/1915762324 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 57) v1 ==== 84446+0+0 (secure 0 0 0) 0x7f2b5c004380 con 0x7f2b640ce480 2023-05-22T15:46:04.721 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.712+0000 7f2b59ffb640 1 --2- 172.21.15.16:0/1915762324 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f2b480655f0 0x7f2b48067aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:04.721 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.712+0000 7f2b68918640 1 -- 172.21.15.16:0/1915762324 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f2b480655f0 msgr2=0x7f2b48067aa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/36921 2023-05-22T15:46:04.722 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.712+0000 7f2b59ffb640 1 -- 172.21.15.16:0/1915762324 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(133..133 src has 1..133) v4 ==== 6726+0+0 (secure 0 0 0) 0x7f2b5c11d600 con 0x7f2b640ce480 2023-05-22T15:46:04.722 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.712+0000 7f2b68918640 1 --2- 172.21.15.16:0/1915762324 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f2b480655f0 0x7f2b48067aa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-05-22T15:46:04.722 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.716+0000 7f2b59ffb640 1 -- 172.21.15.16:0/1915762324 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f2b5c0ead60 con 0x7f2b640ce480 2023-05-22T15:46:04.874 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.868+0000 7f2b6b37d640 1 -- 172.21.15.16:0/1915762324 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f2b6404dbe0 con 0x7f2b640ce480 2023-05-22T15:46:04.875 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.872+0000 7f2b59ffb640 1 -- 172.21.15.16:0/1915762324 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v133) v1 ==== 92+0+10 (secure 0 0 0) 0x7f2b5c0ee380 con 0x7f2b640ce480 2023-05-22T15:46:04.875 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 8 2023-05-22T15:46:04.877 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.872+0000 7f2b6b37d640 1 -- 172.21.15.16:0/1915762324 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f2b480655f0 msgr2=0x7f2b48067aa0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-05-22T15:46:04.877 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.872+0000 7f2b6b37d640 1 --2- 172.21.15.16:0/1915762324 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f2b480655f0 0x7f2b48067aa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:04.878 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.872+0000 7f2b6b37d640 1 -- 172.21.15.16:0/1915762324 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b640ce480 msgr2=0x7f2b64152540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:04.878 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.872+0000 7f2b6b37d640 1 --2- 172.21.15.16:0/1915762324 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b640ce480 0x7f2b64152540 secure :-1 s=READY pgs=1720 cs=0 l=1 rev1=1 crypto rx=0x7f2b5c0ac040 tx=0x7f2b5c01ad10 comp rx=0 tx=0).stop 2023-05-22T15:46:04.878 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.872+0000 7f2b6b37d640 1 -- 172.21.15.16:0/1915762324 shutdown_connections 2023-05-22T15:46:04.878 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.872+0000 7f2b6b37d640 1 --2- 172.21.15.16:0/1915762324 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f2b480655f0 0x7f2b48067aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:04.878 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.872+0000 7f2b6b37d640 1 --2- 172.21.15.16:0/1915762324 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2b6415c270 0x7f2b6414e0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:04.878 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.872+0000 7f2b6b37d640 1 --2- 172.21.15.16:0/1915762324 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2b640ce850 0x7f2b64158170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:04.879 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.872+0000 7f2b6b37d640 1 --2- 172.21.15.16:0/1915762324 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b640ce480 0x7f2b64152540 unknown :-1 s=CLOSED pgs=1720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:04.879 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.872+0000 7f2b6b37d640 1 -- 172.21.15.16:0/1915762324 >> 172.21.15.16:0/1915762324 conn(0x7f2b640cc870 msgr2=0x7f2b64155190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:46:04.879 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.872+0000 7f2b6b37d640 1 -- 172.21.15.16:0/1915762324 shutdown_connections 2023-05-22T15:46:04.879 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:04.872+0000 7f2b6b37d640 1 -- 172.21.15.16:0/1915762324 wait complete. 2023-05-22T15:46:04.894 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-05-22T15:46:04.906 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:46:04.904+0000 7f4a1c01d180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-05-22T15:46:04.930 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:46:04.924+0000 7f614e4ed180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-05-22T15:46:04.959 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:46:04.956+0000 7f4a1c01d180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-05-22T15:46:04.984 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:46:04.980+0000 7f614e4ed180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-05-22T15:46:05.009 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:46:05.008+0000 7f4a1c01d180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-05-22T15:46:05.011 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.008+0000 7f25f2e71640 1 Processor -- start 2023-05-22T15:46:05.011 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.008+0000 7f25f2e71640 1 -- start start 2023-05-22T15:46:05.012 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.008+0000 7f25f2e71640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f25ec0cae20 0x7f25ec06fef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:05.012 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.008+0000 7f25f2e71640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25ec069720 con 0x7f25ec070430 2023-05-22T15:46:05.012 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.008+0000 7f25f2e71640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25ec1543b0 con 0x7f25ec153fe0 2023-05-22T15:46:05.012 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.008+0000 7f25f2e71640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25ec0708d0 con 0x7f25ec0cae20 2023-05-22T15:46:05.012 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.008+0000 7f25f140e640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f25ec070430 0x7f25ec0ad410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:32842/0 (socket says 172.21.15.16:32842) 2023-05-22T15:46:05.013 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.008+0000 7f25f140e640 1 -- 172.21.15.16:0/3382076649 learned_addr learned my addr 172.21.15.16:0/3382076649 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:46:05.014 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.008+0000 7f25ebfff640 1 --2- 172.21.15.16:0/3382076649 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f25ec0cae20 0x7f25ec06fef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:05.014 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.008+0000 7f25eb7fe640 1 -- 172.21.15.16:0/3382076649 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 201402919 0 0) 0x7f25ec069720 con 0x7f25ec070430 2023-05-22T15:46:05.014 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.008+0000 7f25eb7fe640 1 -- 172.21.15.16:0/3382076649 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f25dc003530 con 0x7f25ec070430 2023-05-22T15:46:05.014 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.008+0000 7f25eb7fe640 1 -- 172.21.15.16:0/3382076649 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4067010881 0 0) 0x7f25dc003530 con 0x7f25ec070430 2023-05-22T15:46:05.015 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.008+0000 7f25eb7fe640 1 -- 172.21.15.16:0/3382076649 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f25ec0cae20 msgr2=0x7f25ec06fef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:46:05.015 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.008+0000 7f25eb7fe640 1 --2- 172.21.15.16:0/3382076649 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f25ec0cae20 0x7f25ec06fef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:05.015 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.008+0000 7f25eb7fe640 1 -- 172.21.15.16:0/3382076649 >> v1:172.21.15.62:6789/0 conn(0x7f25ec153fe0 legacy=0x7f25ec06d3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:05.015 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.008+0000 7f25eb7fe640 1 -- 172.21.15.16:0/3382076649 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25ec15e360 con 0x7f25ec070430 2023-05-22T15:46:05.015 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.008+0000 7f25eb7fe640 1 -- 172.21.15.16:0/3382076649 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7f25e4001a00 con 0x7f25ec070430 2023-05-22T15:46:05.015 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.008+0000 7f25eb7fe640 1 -- 172.21.15.16:0/3382076649 >> v1:172.21.15.16:6789/0 conn(0x7f25ec070430 legacy=0x7f25ec0ad410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:05.015 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.008+0000 7f25eb7fe640 1 --2- 172.21.15.16:0/3382076649 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f25ec0cae20 0x7f25dc0039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:05.015 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.008+0000 7f25eb7fe640 1 --2- 172.21.15.16:0/3382076649 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f25dc003f30 0x7f25dc0047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:05.016 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.008+0000 7f25eb7fe640 1 --2- 172.21.15.16:0/3382076649 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f25dc004d10 0x7f25dc005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:05.016 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.008+0000 7f25eb7fe640 1 -- 172.21.15.16:0/3382076649 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25dc00ba40 con 0x7f25dc003f30 2023-05-22T15:46:05.016 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.008+0000 7f25eb7fe640 1 -- 172.21.15.16:0/3382076649 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25dc00bbb0 con 0x7f25dc004d10 2023-05-22T15:46:05.016 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.008+0000 7f25eb7fe640 1 -- 172.21.15.16:0/3382076649 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25dc00bd70 con 0x7f25ec0cae20 2023-05-22T15:46:05.016 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.008+0000 7f25ebfff640 1 --2- 172.21.15.16:0/3382076649 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f25dc003f30 0x7f25dc0047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:05.016 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.008+0000 7f25f0c0d640 1 --2- 172.21.15.16:0/3382076649 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f25ec0cae20 0x7f25dc0039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:05.017 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.008+0000 7f25f140e640 1 --2- 172.21.15.16:0/3382076649 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f25dc004d10 0x7f25dc005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:05.017 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.008+0000 7f25ebfff640 1 -- 172.21.15.16:0/3382076649 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f25ec0cae20 msgr2=0x7f25dc0039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:46:05.017 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.008+0000 7f25ebfff640 1 --2- 172.21.15.16:0/3382076649 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f25ec0cae20 0x7f25dc0039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:05.017 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.008+0000 7f25ebfff640 1 -- 172.21.15.16:0/3382076649 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f25dc004d10 msgr2=0x7f25dc005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:46:05.017 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.008+0000 7f25ebfff640 1 --2- 172.21.15.16:0/3382076649 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f25dc004d10 0x7f25dc005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:05.017 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.008+0000 7f25ebfff640 1 -- 172.21.15.16:0/3382076649 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25dc003530 con 0x7f25dc003f30 2023-05-22T15:46:05.019 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.012+0000 7f25ebfff640 1 --2- 172.21.15.16:0/3382076649 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f25dc003f30 0x7f25dc0047d0 secure :-1 s=READY pgs=1722 cs=0 l=1 rev1=1 crypto rx=0x7f25d8009ab0 tx=0x7f25d801ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2b2a6aa92f137de server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:05.019 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.012+0000 7f25eb7fe640 1 -- 172.21.15.16:0/3382076649 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25d80a5510 con 0x7f25dc003f30 2023-05-22T15:46:05.020 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.012+0000 7f25eb7fe640 1 -- 172.21.15.16:0/3382076649 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25d80a5670 con 0x7f25dc003f30 2023-05-22T15:46:05.020 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.012+0000 7f25eb7fe640 1 -- 172.21.15.16:0/3382076649 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25d80a5a20 con 0x7f25dc003f30 2023-05-22T15:46:05.020 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.012+0000 7f25f2e71640 1 -- 172.21.15.16:0/3382076649 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f25dc003f30 msgr2=0x7f25dc0047d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:05.020 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.012+0000 7f25f2e71640 1 --2- 172.21.15.16:0/3382076649 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f25dc003f30 0x7f25dc0047d0 secure :-1 s=READY pgs=1722 cs=0 l=1 rev1=1 crypto rx=0x7f25d8009ab0 tx=0x7f25d801ca30 comp rx=0 tx=0).stop 2023-05-22T15:46:05.020 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.012+0000 7f25f140e640 1 -- 172.21.15.16:0/3382076649 reap_dead start 2023-05-22T15:46:05.020 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.012+0000 7f25f2e71640 1 -- 172.21.15.16:0/3382076649 shutdown_connections 2023-05-22T15:46:05.020 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.012+0000 7f25f2e71640 1 -- 172.21.15.16:0/3382076649 >> 172.21.15.16:0/3382076649 conn(0x7f25ec0b1110 msgr2=0x7f25ec0ce830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:46:05.021 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.012+0000 7f25f2e71640 1 -- 172.21.15.16:0/3382076649 shutdown_connections 2023-05-22T15:46:05.021 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.012+0000 7f25f2e71640 1 -- 172.21.15.16:0/3382076649 wait complete. 2023-05-22T15:46:05.021 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.012+0000 7f25f2e71640 1 Processor -- start 2023-05-22T15:46:05.021 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.012+0000 7f25f2e71640 1 -- start start 2023-05-22T15:46:05.021 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.012+0000 7f25f2e71640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f25dc003f30 0x7f25ec152940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:05.021 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.012+0000 7f25f2e71640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f25ec15bb30 0x7f25ec158b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:05.021 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.012+0000 7f25f2e71640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f25ec15bfa0 0x7f25ec1590c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:05.022 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.012+0000 7f25f2e71640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25ec159ce0 con 0x7f25dc003f30 2023-05-22T15:46:05.022 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.012+0000 7f25f2e71640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25ec159ef0 con 0x7f25ec15bfa0 2023-05-22T15:46:05.022 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.012+0000 7f25f2e71640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25ec15a1d0 con 0x7f25ec15bb30 2023-05-22T15:46:05.022 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.016+0000 7f25f0c0d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f25dc003f30 0x7f25ec152940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:05.022 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.016+0000 7f25ebfff640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f25ec15bb30 0x7f25ec158b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:05.022 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.016+0000 7f25f0c0d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f25dc003f30 0x7f25ec152940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59412/0 (socket says 172.21.15.16:59412) 2023-05-22T15:46:05.022 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.016+0000 7f25f0c0d640 1 -- 172.21.15.16:0/23838636 learned_addr learned my addr 172.21.15.16:0/23838636 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:46:05.022 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.016+0000 7f25f140e640 1 --2- 172.21.15.16:0/23838636 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f25ec15bfa0 0x7f25ec1590c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:05.023 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.016+0000 7f25f0c0d640 1 -- 172.21.15.16:0/23838636 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f25ec15bb30 msgr2=0x7f25ec158b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:05.023 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.016+0000 7f25f0c0d640 1 --2- 172.21.15.16:0/23838636 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f25ec15bb30 0x7f25ec158b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:05.023 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.016+0000 7f25f0c0d640 1 -- 172.21.15.16:0/23838636 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f25ec15bfa0 msgr2=0x7f25ec1590c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:05.023 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.016+0000 7f25f0c0d640 1 --2- 172.21.15.16:0/23838636 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f25ec15bfa0 0x7f25ec1590c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:05.023 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.016+0000 7f25f0c0d640 1 -- 172.21.15.16:0/23838636 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25ec15d880 con 0x7f25dc003f30 2023-05-22T15:46:05.023 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.016+0000 7f25f140e640 1 --2- 172.21.15.16:0/23838636 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f25ec15bfa0 0x7f25ec1590c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T15:46:05.023 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.016+0000 7f25f0c0d640 1 --2- 172.21.15.16:0/23838636 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f25dc003f30 0x7f25ec152940 secure :-1 s=READY pgs=1723 cs=0 l=1 rev1=1 crypto rx=0x7f25e000f7f0 tx=0x7f25e000fcb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:05.023 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.016+0000 7f25e97fa640 1 -- 172.21.15.16:0/23838636 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25e00050d0 con 0x7f25dc003f30 2023-05-22T15:46:05.024 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.016+0000 7f25e97fa640 1 -- 172.21.15.16:0/23838636 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25e0005230 con 0x7f25dc003f30 2023-05-22T15:46:05.024 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.016+0000 7f25f2e71640 1 -- 172.21.15.16:0/23838636 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25ec15d5a0 con 0x7f25dc003f30 2023-05-22T15:46:05.024 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.016+0000 7f25e97fa640 1 -- 172.21.15.16:0/23838636 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25e000d730 con 0x7f25dc003f30 2023-05-22T15:46:05.024 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.016+0000 7f25f2e71640 1 -- 172.21.15.16:0/23838636 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25ec15db60 con 0x7f25dc003f30 2023-05-22T15:46:05.025 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.016+0000 7f25f2e71640 1 -- 172.21.15.16:0/23838636 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25ec04dbe0 con 0x7f25dc003f30 2023-05-22T15:46:05.025 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.016+0000 7f25e97fa640 1 -- 172.21.15.16:0/23838636 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 57) v1 ==== 84446+0+0 (secure 0 0 0) 0x7f25e000d890 con 0x7f25dc003f30 2023-05-22T15:46:05.025 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.016+0000 7f25e97fa640 1 --2- 172.21.15.16:0/23838636 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f25d00655f0 0x7f25d0067aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:05.025 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.016+0000 7f25ebfff640 1 -- 172.21.15.16:0/23838636 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f25d00655f0 msgr2=0x7f25d0067aa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/36921 2023-05-22T15:46:05.025 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.016+0000 7f25e97fa640 1 -- 172.21.15.16:0/23838636 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(133..133 src has 1..133) v4 ==== 6726+0+0 (secure 0 0 0) 0x7f25e0082f00 con 0x7f25dc003f30 2023-05-22T15:46:05.025 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.016+0000 7f25ebfff640 1 --2- 172.21.15.16:0/23838636 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f25d00655f0 0x7f25d0067aa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-05-22T15:46:05.026 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.020+0000 7f25e97fa640 1 -- 172.21.15.16:0/23838636 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f25e0050580 con 0x7f25dc003f30 2023-05-22T15:46:05.047 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:46:05.044+0000 7f614e4ed180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-05-22T15:46:05.087 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:46:05.084+0000 7f4a1c01d180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-05-22T15:46:05.128 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:46:05.124+0000 7f614e4ed180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-05-22T15:46:05.138 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:46:05.136+0000 7f4a1c01d180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-05-22T15:46:05.179 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:46:05.176+0000 7f614e4ed180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-05-22T15:46:05.190 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.184+0000 7f25f2e71640 1 -- 172.21.15.16:0/23838636 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f25ec0adf00 con 0x7f25dc003f30 2023-05-22T15:46:05.191 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.188+0000 7f25e97fa640 1 -- 172.21.15.16:0/23838636 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v133) v1 ==== 104+0+11 (secure 0 0 0) 0x7f25e0053ba0 con 0x7f25dc003f30 2023-05-22T15:46:05.191 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2023-05-22T15:46:05.193 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.188+0000 7f25f2e71640 1 -- 172.21.15.16:0/23838636 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f25d00655f0 msgr2=0x7f25d0067aa0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-05-22T15:46:05.194 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.188+0000 7f25f2e71640 1 --2- 172.21.15.16:0/23838636 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f25d00655f0 0x7f25d0067aa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:05.194 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.188+0000 7f25f2e71640 1 -- 172.21.15.16:0/23838636 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f25dc003f30 msgr2=0x7f25ec152940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:05.194 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.188+0000 7f25f2e71640 1 --2- 172.21.15.16:0/23838636 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f25dc003f30 0x7f25ec152940 secure :-1 s=READY pgs=1723 cs=0 l=1 rev1=1 crypto rx=0x7f25e000f7f0 tx=0x7f25e000fcb0 comp rx=0 tx=0).stop 2023-05-22T15:46:05.194 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.188+0000 7f25f2e71640 1 -- 172.21.15.16:0/23838636 shutdown_connections 2023-05-22T15:46:05.194 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.188+0000 7f25f2e71640 1 --2- 172.21.15.16:0/23838636 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f25ec15bfa0 0x7f25ec1590c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:05.194 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.188+0000 7f25f2e71640 1 --2- 172.21.15.16:0/23838636 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f25d00655f0 0x7f25d0067aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:05.195 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.188+0000 7f25f2e71640 1 --2- 172.21.15.16:0/23838636 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f25ec15bb30 0x7f25ec158b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:05.195 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.188+0000 7f25f2e71640 1 --2- 172.21.15.16:0/23838636 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f25dc003f30 0x7f25ec152940 unknown :-1 s=CLOSED pgs=1723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:05.195 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.188+0000 7f25f2e71640 1 -- 172.21.15.16:0/23838636 >> 172.21.15.16:0/23838636 conn(0x7f25ec0b1110 msgr2=0x7f25ec0c7b40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:46:05.195 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.188+0000 7f25f2e71640 1 -- 172.21.15.16:0/23838636 shutdown_connections 2023-05-22T15:46:05.195 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.188+0000 7f25f2e71640 1 -- 172.21.15.16:0/23838636 wait complete. 2023-05-22T15:46:05.210 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-05-22T15:46:05.213 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:46:05.212+0000 7f4a1c01d180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-05-22T15:46:05.259 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:46:05.256+0000 7f614e4ed180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-05-22T15:46:05.265 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:46:05.264+0000 7f4a1c01d180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-05-22T15:46:05.312 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:46:05.308+0000 7f614e4ed180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-05-22T15:46:05.375 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.368+0000 7fb068648640 1 Processor -- start 2023-05-22T15:46:05.375 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.372+0000 7fb068648640 1 -- start start 2023-05-22T15:46:05.376 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.372+0000 7fb068648640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fb0600ce480 0x7fb060071640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:05.377 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.372+0000 7fb068648640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb06015c270 0x7fb06015c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:05.378 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.372+0000 7fb068648640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb060069ae0 con 0x7fb0600ce850 2023-05-22T15:46:05.378 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.372+0000 7fb068648640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb06015f990 con 0x7fb0600ce480 2023-05-22T15:46:05.378 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.372+0000 7fb068648640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb060069dc0 con 0x7fb06015c270 2023-05-22T15:46:05.378 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.372+0000 7fb065be3640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fb0600ce850 0x7fb060160090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:32854/0 (socket says 172.21.15.16:32854) 2023-05-22T15:46:05.378 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.372+0000 7fb065be3640 1 -- 172.21.15.16:0/1187761591 learned_addr learned my addr 172.21.15.16:0/1187761591 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:46:05.378 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.372+0000 7fb066be5640 1 --2- 172.21.15.16:0/1187761591 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb06015c270 0x7fb06015c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:05.378 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.372+0000 7fb0663e4640 1 --2- 172.21.15.16:0/1187761591 >> v2:172.21.15.62:3300/0 conn(0x7fb0600ce480 0x7fb060071640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:05.379 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.372+0000 7fb0653e2640 1 -- 172.21.15.16:0/1187761591 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3535413355 0 0) 0x7fb060069ae0 con 0x7fb0600ce850 2023-05-22T15:46:05.379 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.372+0000 7fb0653e2640 1 -- 172.21.15.16:0/1187761591 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb04c003530 con 0x7fb0600ce850 2023-05-22T15:46:05.379 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.372+0000 7fb066be5640 1 -- 172.21.15.16:0/1187761591 >> v2:172.21.15.62:3300/0 conn(0x7fb0600ce480 msgr2=0x7fb060071640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:46:05.379 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.372+0000 7fb066be5640 1 --2- 172.21.15.16:0/1187761591 >> v2:172.21.15.62:3300/0 conn(0x7fb0600ce480 0x7fb060071640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:05.379 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.372+0000 7fb066be5640 1 -- 172.21.15.16:0/1187761591 >> v1:172.21.15.16:6789/0 conn(0x7fb0600ce850 legacy=0x7fb060160090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:05.379 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.372+0000 7fb066be5640 1 -- 172.21.15.16:0/1187761591 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb06006a0a0 con 0x7fb06015c270 2023-05-22T15:46:05.385 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.376+0000 7fb066be5640 1 --2- 172.21.15.16:0/1187761591 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb06015c270 0x7fb06015c6f0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fb058009d90 tx=0x7fb05801c920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6381609c77d7b10d server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:05.386 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.376+0000 7fb0653e2640 1 -- 172.21.15.16:0/1187761591 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0580a5a00 con 0x7fb06015c270 2023-05-22T15:46:05.386 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.376+0000 7fb0653e2640 1 -- 172.21.15.16:0/1187761591 <== mon.2 v2:172.21.15.62:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0580a5b60 con 0x7fb06015c270 2023-05-22T15:46:05.386 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.376+0000 7fb0653e2640 1 -- 172.21.15.16:0/1187761591 <== mon.2 v2:172.21.15.62:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0580ad440 con 0x7fb06015c270 2023-05-22T15:46:05.387 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.376+0000 7fb068648640 1 -- 172.21.15.16:0/1187761591 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb06015c270 msgr2=0x7fb06015c6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:05.387 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.376+0000 7fb068648640 1 --2- 172.21.15.16:0/1187761591 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb06015c270 0x7fb06015c6f0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fb058009d90 tx=0x7fb05801c920 comp rx=0 tx=0).stop 2023-05-22T15:46:05.387 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.376+0000 7fb068648640 1 -- 172.21.15.16:0/1187761591 shutdown_connections 2023-05-22T15:46:05.387 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.376+0000 7fb068648640 1 --2- 172.21.15.16:0/1187761591 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb06015c270 0x7fb06015c6f0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:05.388 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.376+0000 7fb068648640 1 --2- 172.21.15.16:0/1187761591 >> v2:172.21.15.62:3300/0 conn(0x7fb0600ce480 0x7fb060071640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:05.388 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.376+0000 7fb068648640 1 -- 172.21.15.16:0/1187761591 >> 172.21.15.16:0/1187761591 conn(0x7fb0600cc870 msgr2=0x7fb0600cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:46:05.388 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.376+0000 7fb068648640 1 -- 172.21.15.16:0/1187761591 shutdown_connections 2023-05-22T15:46:05.388 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.376+0000 7fb068648640 1 -- 172.21.15.16:0/1187761591 wait complete. 2023-05-22T15:46:05.388 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.376+0000 7fb068648640 1 Processor -- start 2023-05-22T15:46:05.388 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.376+0000 7fb068648640 1 -- start start 2023-05-22T15:46:05.389 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.376+0000 7fb068648640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb0600ce480 0x7fb0600ad9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:05.389 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.376+0000 7fb068648640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb0600ce850 0x7fb06014b8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:05.389 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.376+0000 7fb068648640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb06015c270 0x7fb0600ab7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:05.389 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.376+0000 7fb068648640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb06014d880 con 0x7fb0600ce480 2023-05-22T15:46:05.389 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.376+0000 7fb068648640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0600ae260 con 0x7fb0600ce850 2023-05-22T15:46:05.389 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.376+0000 7fb068648640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0600ae540 con 0x7fb06015c270 2023-05-22T15:46:05.389 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.376+0000 7fb0663e4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb0600ce480 0x7fb0600ad9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:05.389 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.376+0000 7fb0663e4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb0600ce480 0x7fb0600ad9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59428/0 (socket says 172.21.15.16:59428) 2023-05-22T15:46:05.390 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.376+0000 7fb0663e4640 1 -- 172.21.15.16:0/316173238 learned_addr learned my addr 172.21.15.16:0/316173238 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:46:05.390 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.376+0000 7fb066be5640 1 --2- 172.21.15.16:0/316173238 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb06015c270 0x7fb0600ab7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:05.390 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.376+0000 7fb065be3640 1 --2- 172.21.15.16:0/316173238 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb0600ce850 0x7fb06014b8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:05.390 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.376+0000 7fb0663e4640 1 -- 172.21.15.16:0/316173238 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb06015c270 msgr2=0x7fb0600ab7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:05.390 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.376+0000 7fb0663e4640 1 --2- 172.21.15.16:0/316173238 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb06015c270 0x7fb0600ab7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:05.390 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.376+0000 7fb0663e4640 1 -- 172.21.15.16:0/316173238 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb0600ce850 msgr2=0x7fb06014b8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:05.391 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.376+0000 7fb0663e4640 1 --2- 172.21.15.16:0/316173238 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb0600ce850 0x7fb06014b8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:05.391 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.376+0000 7fb0663e4640 1 -- 172.21.15.16:0/316173238 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb058009a70 con 0x7fb0600ce480 2023-05-22T15:46:05.391 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.376+0000 7fb0663e4640 1 --2- 172.21.15.16:0/316173238 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb0600ce480 0x7fb0600ad9f0 secure :-1 s=READY pgs=1725 cs=0 l=1 rev1=1 crypto rx=0x7fb05000e930 tx=0x7fb05000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:05.391 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.376+0000 7fb056ffd640 1 -- 172.21.15.16:0/316173238 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb050009690 con 0x7fb0600ce480 2023-05-22T15:46:05.391 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.376+0000 7fb056ffd640 1 -- 172.21.15.16:0/316173238 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb05000ca00 con 0x7fb0600ce480 2023-05-22T15:46:05.391 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.376+0000 7fb068648640 1 -- 172.21.15.16:0/316173238 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0600698b0 con 0x7fb0600ce480 2023-05-22T15:46:05.391 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.376+0000 7fb056ffd640 1 -- 172.21.15.16:0/316173238 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb05000cd70 con 0x7fb0600ce480 2023-05-22T15:46:05.391 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.376+0000 7fb068648640 1 -- 172.21.15.16:0/316173238 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb060069720 con 0x7fb0600ce480 2023-05-22T15:46:05.392 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.380+0000 7fb068648640 1 -- 172.21.15.16:0/316173238 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb06015cc30 con 0x7fb0600ce480 2023-05-22T15:46:05.392 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.380+0000 7fb056ffd640 1 -- 172.21.15.16:0/316173238 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 57) v1 ==== 84446+0+0 (secure 0 0 0) 0x7fb050013470 con 0x7fb0600ce480 2023-05-22T15:46:05.392 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.380+0000 7fb056ffd640 1 --2- 172.21.15.16:0/316173238 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fb0440655f0 0x7fb044067aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:05.392 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.380+0000 7fb065be3640 1 -- 172.21.15.16:0/316173238 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fb0440655f0 msgr2=0x7fb044067aa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/36921 2023-05-22T15:46:05.392 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.380+0000 7fb056ffd640 1 -- 172.21.15.16:0/316173238 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(133..133 src has 1..133) v4 ==== 6726+0+0 (secure 0 0 0) 0x7fb0500834f0 con 0x7fb0600ce480 2023-05-22T15:46:05.392 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.380+0000 7fb065be3640 1 --2- 172.21.15.16:0/316173238 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fb0440655f0 0x7fb044067aa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-05-22T15:46:05.393 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.384+0000 7fb056ffd640 1 -- 172.21.15.16:0/316173238 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fb050050bc0 con 0x7fb0600ce480 2023-05-22T15:46:05.397 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:46:05.396+0000 7f4a1c01d180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-05-22T15:46:05.447 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:46:05.444+0000 7f614e4ed180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-05-22T15:46:05.449 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:46:05.448+0000 7f4a1c01d180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-05-22T15:46:05.500 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:46:05.496+0000 7f614e4ed180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-05-22T15:46:05.539 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.536+0000 7fb068648640 1 -- 172.21.15.16:0/316173238 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fb0600ae700 con 0x7fb0600ce480 2023-05-22T15:46:05.540 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.536+0000 7fb056ffd640 1 -- 172.21.15.16:0/316173238 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v133) v1 ==== 100+0+11 (secure 0 0 0) 0x7fb0500541e0 con 0x7fb0600ce480 2023-05-22T15:46:05.540 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 64 2023-05-22T15:46:05.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.536+0000 7fb068648640 1 -- 172.21.15.16:0/316173238 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fb0440655f0 msgr2=0x7fb044067aa0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-05-22T15:46:05.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.536+0000 7fb068648640 1 --2- 172.21.15.16:0/316173238 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fb0440655f0 0x7fb044067aa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:05.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.536+0000 7fb068648640 1 -- 172.21.15.16:0/316173238 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb0600ce480 msgr2=0x7fb0600ad9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:05.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.536+0000 7fb068648640 1 --2- 172.21.15.16:0/316173238 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb0600ce480 0x7fb0600ad9f0 secure :-1 s=READY pgs=1725 cs=0 l=1 rev1=1 crypto rx=0x7fb05000e930 tx=0x7fb05000edf0 comp rx=0 tx=0).stop 2023-05-22T15:46:05.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.536+0000 7fb068648640 1 -- 172.21.15.16:0/316173238 shutdown_connections 2023-05-22T15:46:05.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.536+0000 7fb068648640 1 --2- 172.21.15.16:0/316173238 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fb0440655f0 0x7fb044067aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:05.544 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.536+0000 7fb068648640 1 --2- 172.21.15.16:0/316173238 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb06015c270 0x7fb0600ab7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:05.544 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.536+0000 7fb068648640 1 --2- 172.21.15.16:0/316173238 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb0600ce850 0x7fb06014b8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:05.544 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.536+0000 7fb068648640 1 --2- 172.21.15.16:0/316173238 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb0600ce480 0x7fb0600ad9f0 unknown :-1 s=CLOSED pgs=1725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:05.544 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.536+0000 7fb068648640 1 -- 172.21.15.16:0/316173238 >> 172.21.15.16:0/316173238 conn(0x7fb0600cc870 msgr2=0x7fb06006ec40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:46:05.544 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.536+0000 7fb068648640 1 -- 172.21.15.16:0/316173238 shutdown_connections 2023-05-22T15:46:05.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.536+0000 7fb068648640 1 -- 172.21.15.16:0/316173238 wait complete. 2023-05-22T15:46:05.559 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-05-22T15:46:05.676 INFO:tasks.ceph.mgr.x.smithi062.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2023-05-22T15:46:05.677 INFO:tasks.ceph.mgr.x.smithi062.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2023-05-22T15:46:05.677 INFO:tasks.ceph.mgr.x.smithi062.stderr: from numpy import show_config as show_numpy_config 2023-05-22T15:46:05.684 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:46:05.680+0000 7f4a1c01d180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-05-22T15:46:05.731 INFO:tasks.ceph.mgr.y.smithi016.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2023-05-22T15:46:05.732 INFO:tasks.ceph.mgr.y.smithi016.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2023-05-22T15:46:05.732 INFO:tasks.ceph.mgr.y.smithi016.stderr: from numpy import show_config as show_numpy_config 2023-05-22T15:46:05.741 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:46:05.736+0000 7f614e4ed180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-05-22T15:46:05.758 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.752+0000 7fd2945bc640 1 Processor -- start 2023-05-22T15:46:05.759 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.752+0000 7fd2945bc640 1 -- start start 2023-05-22T15:46:05.762 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.756+0000 7fd2945bc640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd28c158a10 0x7fd28c06e480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:05.762 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.756+0000 7fd2945bc640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd28c158de0 con 0x7fd28c070760 2023-05-22T15:46:05.763 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.756+0000 7fd2945bc640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd28c0711e0 con 0x7fd28c06cdf0 2023-05-22T15:46:05.763 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.756+0000 7fd2945bc640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd28c06e060 con 0x7fd28c158a10 2023-05-22T15:46:05.763 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.756+0000 7fd291b57640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fd28c070760 0x7fd28c06c6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:32864/0 (socket says 172.21.15.16:32864) 2023-05-22T15:46:05.763 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.756+0000 7fd291b57640 1 -- 172.21.15.16:0/2558298895 learned_addr learned my addr 172.21.15.16:0/2558298895 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:46:05.763 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.756+0000 7fd292358640 1 --2- 172.21.15.16:0/2558298895 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd28c158a10 0x7fd28c06e480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:05.763 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.756+0000 7fd291356640 1 -- 172.21.15.16:0/2558298895 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1324337542 0 0) 0x7fd28c158de0 con 0x7fd28c070760 2023-05-22T15:46:05.764 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.756+0000 7fd291356640 1 -- 172.21.15.16:0/2558298895 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd278003530 con 0x7fd28c070760 2023-05-22T15:46:05.764 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.756+0000 7fd291356640 1 -- 172.21.15.16:0/2558298895 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 366619818 0 0) 0x7fd278003530 con 0x7fd28c070760 2023-05-22T15:46:05.764 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.756+0000 7fd291356640 1 -- 172.21.15.16:0/2558298895 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd28c158a10 msgr2=0x7fd28c06e480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:46:05.764 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.756+0000 7fd291356640 1 --2- 172.21.15.16:0/2558298895 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd28c158a10 0x7fd28c06e480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:05.764 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.756+0000 7fd291356640 1 -- 172.21.15.16:0/2558298895 >> v1:172.21.15.62:6789/0 conn(0x7fd28c06cdf0 legacy=0x7fd28c0c9a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:05.764 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.756+0000 7fd291356640 1 -- 172.21.15.16:0/2558298895 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd28c0714c0 con 0x7fd28c070760 2023-05-22T15:46:05.764 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.756+0000 7fd291356640 1 -- 172.21.15.16:0/2558298895 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7fd27c001680 con 0x7fd28c070760 2023-05-22T15:46:05.765 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.756+0000 7fd291356640 1 -- 172.21.15.16:0/2558298895 >> v1:172.21.15.16:6789/0 conn(0x7fd28c070760 legacy=0x7fd28c06c6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:05.765 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.756+0000 7fd291356640 1 --2- 172.21.15.16:0/2558298895 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd2780039b0 0x7fd278003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:05.765 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.756+0000 7fd291356640 1 --2- 172.21.15.16:0/2558298895 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd28c158a10 0x7fd2780047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:05.765 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.756+0000 7fd291356640 1 --2- 172.21.15.16:0/2558298895 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd278004d10 0x7fd278005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:05.765 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.756+0000 7fd291356640 1 -- 172.21.15.16:0/2558298895 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd28c06e060 con 0x7fd2780039b0 2023-05-22T15:46:05.765 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.756+0000 7fd291356640 1 -- 172.21.15.16:0/2558298895 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd27800baa0 con 0x7fd278004d10 2023-05-22T15:46:05.765 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.756+0000 7fd291356640 1 -- 172.21.15.16:0/2558298895 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd27800bc60 con 0x7fd28c158a10 2023-05-22T15:46:05.766 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.756+0000 7fd292358640 1 --2- 172.21.15.16:0/2558298895 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd2780039b0 0x7fd278003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:05.766 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.756+0000 7fd291b57640 1 --2- 172.21.15.16:0/2558298895 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd28c158a10 0x7fd2780047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:05.766 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.756+0000 7fd292b59640 1 --2- 172.21.15.16:0/2558298895 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd278004d10 0x7fd278005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:05.766 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.756+0000 7fd292358640 1 -- 172.21.15.16:0/2558298895 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd28c158a10 msgr2=0x7fd2780047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:46:05.766 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.756+0000 7fd292358640 1 --2- 172.21.15.16:0/2558298895 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd28c158a10 0x7fd2780047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:05.766 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.756+0000 7fd292358640 1 -- 172.21.15.16:0/2558298895 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd278004d10 msgr2=0x7fd278005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:46:05.766 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.756+0000 7fd292358640 1 --2- 172.21.15.16:0/2558298895 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd278004d10 0x7fd278005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:05.766 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.756+0000 7fd292358640 1 -- 172.21.15.16:0/2558298895 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd278003530 con 0x7fd2780039b0 2023-05-22T15:46:05.770 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.760+0000 7fd292358640 1 --2- 172.21.15.16:0/2558298895 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd2780039b0 0x7fd278003dc0 secure :-1 s=READY pgs=1727 cs=0 l=1 rev1=1 crypto rx=0x7fd284009fc0 tx=0x7fd28401cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a233e7f772b725f7 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:05.770 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.760+0000 7fd291356640 1 -- 172.21.15.16:0/2558298895 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2840a5b60 con 0x7fd2780039b0 2023-05-22T15:46:05.770 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.760+0000 7fd291356640 1 -- 172.21.15.16:0/2558298895 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd28400f040 con 0x7fd2780039b0 2023-05-22T15:46:05.771 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.760+0000 7fd291356640 1 -- 172.21.15.16:0/2558298895 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2840ad440 con 0x7fd2780039b0 2023-05-22T15:46:05.771 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.760+0000 7fd2945bc640 1 -- 172.21.15.16:0/2558298895 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd2780039b0 msgr2=0x7fd278003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:05.771 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.760+0000 7fd2945bc640 1 --2- 172.21.15.16:0/2558298895 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd2780039b0 0x7fd278003dc0 secure :-1 s=READY pgs=1727 cs=0 l=1 rev1=1 crypto rx=0x7fd284009fc0 tx=0x7fd28401cc60 comp rx=0 tx=0).stop 2023-05-22T15:46:05.771 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.760+0000 7fd292b59640 1 -- 172.21.15.16:0/2558298895 reap_dead start 2023-05-22T15:46:05.771 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.760+0000 7fd2945bc640 1 -- 172.21.15.16:0/2558298895 shutdown_connections 2023-05-22T15:46:05.771 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.760+0000 7fd2945bc640 1 -- 172.21.15.16:0/2558298895 >> 172.21.15.16:0/2558298895 conn(0x7fd28c0cc870 msgr2=0x7fd28c0cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:46:05.772 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.760+0000 7fd2945bc640 1 -- 172.21.15.16:0/2558298895 shutdown_connections 2023-05-22T15:46:05.772 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.760+0000 7fd2945bc640 1 -- 172.21.15.16:0/2558298895 wait complete. 2023-05-22T15:46:05.772 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.760+0000 7fd2945bc640 1 Processor -- start 2023-05-22T15:46:05.772 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.760+0000 7fd2945bc640 1 -- start start 2023-05-22T15:46:05.772 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.760+0000 7fd2945bc640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd2780039b0 0x7fd28c0ad5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:05.772 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.760+0000 7fd2945bc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd28c0c5610 0x7fd28c145410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:05.772 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.760+0000 7fd2945bc640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd28c145950 0x7fd28c145dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:05.773 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.760+0000 7fd2945bc640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd28c0ae830 con 0x7fd28c0c5610 2023-05-22T15:46:05.773 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.760+0000 7fd2945bc640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd28c0adc20 con 0x7fd28c145950 2023-05-22T15:46:05.773 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.760+0000 7fd2945bc640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd28c0adf00 con 0x7fd2780039b0 2023-05-22T15:46:05.773 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.760+0000 7fd291b57640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd28c0c5610 0x7fd28c145410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:05.773 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.760+0000 7fd291b57640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd28c0c5610 0x7fd28c145410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59444/0 (socket says 172.21.15.16:59444) 2023-05-22T15:46:05.773 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.760+0000 7fd291b57640 1 -- 172.21.15.16:0/3207465786 learned_addr learned my addr 172.21.15.16:0/3207465786 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:46:05.773 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.760+0000 7fd292358640 1 --2- 172.21.15.16:0/3207465786 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd2780039b0 0x7fd28c0ad5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:05.774 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.760+0000 7fd292b59640 1 --2- 172.21.15.16:0/3207465786 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd28c145950 0x7fd28c145dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:05.774 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.760+0000 7fd291b57640 1 -- 172.21.15.16:0/3207465786 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd2780039b0 msgr2=0x7fd28c0ad5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:05.774 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.760+0000 7fd291b57640 1 --2- 172.21.15.16:0/3207465786 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd2780039b0 0x7fd28c0ad5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:05.774 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.760+0000 7fd291b57640 1 -- 172.21.15.16:0/3207465786 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd28c145950 msgr2=0x7fd28c145dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:05.774 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.760+0000 7fd291b57640 1 --2- 172.21.15.16:0/3207465786 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd28c145950 0x7fd28c145dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:05.774 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.760+0000 7fd291b57640 1 -- 172.21.15.16:0/3207465786 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd28c164900 con 0x7fd28c0c5610 2023-05-22T15:46:05.774 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.760+0000 7fd292358640 1 --2- 172.21.15.16:0/3207465786 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd2780039b0 0x7fd28c0ad5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T15:46:05.774 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.764+0000 7fd291b57640 1 --2- 172.21.15.16:0/3207465786 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd28c0c5610 0x7fd28c145410 secure :-1 s=READY pgs=1728 cs=0 l=1 rev1=1 crypto rx=0x7fd27c00e8c0 tx=0x7fd27c00ed80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:05.775 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.764+0000 7fd282ffd640 1 -- 172.21.15.16:0/3207465786 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd27c00c870 con 0x7fd28c0c5610 2023-05-22T15:46:05.775 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.764+0000 7fd282ffd640 1 -- 172.21.15.16:0/3207465786 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd27c00c9d0 con 0x7fd28c0c5610 2023-05-22T15:46:05.775 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.764+0000 7fd282ffd640 1 -- 172.21.15.16:0/3207465786 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd27c00cd80 con 0x7fd28c0c5610 2023-05-22T15:46:05.775 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.764+0000 7fd2945bc640 1 -- 172.21.15.16:0/3207465786 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd28c070f40 con 0x7fd28c0c5610 2023-05-22T15:46:05.775 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.764+0000 7fd2945bc640 1 -- 172.21.15.16:0/3207465786 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd28c164a90 con 0x7fd28c0c5610 2023-05-22T15:46:05.775 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.764+0000 7fd2945bc640 1 -- 172.21.15.16:0/3207465786 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd28c143380 con 0x7fd28c0c5610 2023-05-22T15:46:05.775 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.764+0000 7fd282ffd640 1 -- 172.21.15.16:0/3207465786 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 57) v1 ==== 84446+0+0 (secure 0 0 0) 0x7fd27c01d460 con 0x7fd28c0c5610 2023-05-22T15:46:05.776 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.764+0000 7fd282ffd640 1 --2- 172.21.15.16:0/3207465786 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd2700655c0 0x7fd270067a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:05.776 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.764+0000 7fd292358640 1 -- 172.21.15.16:0/3207465786 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd2700655c0 msgr2=0x7fd270067a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/36921 2023-05-22T15:46:05.776 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.764+0000 7fd282ffd640 1 -- 172.21.15.16:0/3207465786 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(133..133 src has 1..133) v4 ==== 6726+0+0 (secure 0 0 0) 0x7fd27c082e10 con 0x7fd28c0c5610 2023-05-22T15:46:05.776 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.764+0000 7fd292358640 1 --2- 172.21.15.16:0/3207465786 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd2700655c0 0x7fd270067a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-05-22T15:46:05.777 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.768+0000 7fd282ffd640 1 -- 172.21.15.16:0/3207465786 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fd27c050590 con 0x7fd28c0c5610 2023-05-22T15:46:05.812 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:46:05.808+0000 7f4a1c01d180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-05-22T15:46:05.899 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:46:05.896+0000 7f614e4ed180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-05-22T15:46:05.925 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:46:05.924+0000 7f4a1c01d180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-05-22T15:46:05.963 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.960+0000 7fd2945bc640 1 -- 172.21.15.16:0/3207465786 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fd28c04dbe0 con 0x7fd28c0c5610 2023-05-22T15:46:05.964 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.960+0000 7fd282ffd640 1 -- 172.21.15.16:0/3207465786 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 8 entries v133) v1 ==== 80+0+458 (secure 0 0 0) 0x7fd27c001620 con 0x7fd28c0c5610 2023-05-22T15:46:05.964 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/1950855917 2023-05-23T15:45:17.648374+0000 2023-05-22T15:46:05.964 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/1889818530 2023-05-23T15:45:17.648374+0000 2023-05-22T15:46:05.965 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:6833/36921 2023-05-23T15:45:17.648374+0000 2023-05-22T15:46:05.965 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/4160466619 2023-05-23T15:45:17.648374+0000 2023-05-22T15:46:05.965 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/55601757 2023-05-23T15:45:17.648374+0000 2023-05-22T15:46:05.965 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:6832/36921 2023-05-23T15:45:17.648374+0000 2023-05-22T15:46:05.965 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/4127127856 2023-05-23T15:45:17.648374+0000 2023-05-22T15:46:05.965 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/2333289485 2023-05-23T15:45:17.648374+0000 2023-05-22T15:46:05.966 INFO:teuthology.orchestra.run.smithi016.stderr:listed 8 entries 2023-05-22T15:46:05.966 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.960+0000 7fd2945bc640 1 -- 172.21.15.16:0/3207465786 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd2700655c0 msgr2=0x7fd270067a70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-05-22T15:46:05.966 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.960+0000 7fd2945bc640 1 --2- 172.21.15.16:0/3207465786 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd2700655c0 0x7fd270067a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:05.966 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.960+0000 7fd2945bc640 1 -- 172.21.15.16:0/3207465786 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd28c0c5610 msgr2=0x7fd28c145410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:05.967 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.960+0000 7fd2945bc640 1 --2- 172.21.15.16:0/3207465786 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd28c0c5610 0x7fd28c145410 secure :-1 s=READY pgs=1728 cs=0 l=1 rev1=1 crypto rx=0x7fd27c00e8c0 tx=0x7fd27c00ed80 comp rx=0 tx=0).stop 2023-05-22T15:46:05.967 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.960+0000 7fd2945bc640 1 -- 172.21.15.16:0/3207465786 shutdown_connections 2023-05-22T15:46:05.967 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.960+0000 7fd2945bc640 1 --2- 172.21.15.16:0/3207465786 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd28c145950 0x7fd28c145dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:05.967 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.960+0000 7fd2945bc640 1 --2- 172.21.15.16:0/3207465786 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd28c0c5610 0x7fd28c145410 unknown :-1 s=CLOSED pgs=1728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:05.967 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.960+0000 7fd2945bc640 1 --2- 172.21.15.16:0/3207465786 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd2700655c0 0x7fd270067a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:05.967 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.960+0000 7fd2945bc640 1 --2- 172.21.15.16:0/3207465786 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd2780039b0 0x7fd28c0ad5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:05.967 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.960+0000 7fd2945bc640 1 -- 172.21.15.16:0/3207465786 >> 172.21.15.16:0/3207465786 conn(0x7fd28c0cc870 msgr2=0x7fd28c157bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:46:05.968 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.960+0000 7fd2945bc640 1 -- 172.21.15.16:0/3207465786 shutdown_connections 2023-05-22T15:46:05.968 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:05.960+0000 7fd2945bc640 1 -- 172.21.15.16:0/3207465786 wait complete. 2023-05-22T15:46:05.983 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi016.front.sepia.ceph.com... 2023-05-22T15:46:05.983 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:46:05.984 DEBUG:teuthology.orchestra.run.smithi016:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2023-05-22T15:46:05.989 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:46:05.988+0000 7f4a1c01d180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-05-22T15:46:05.995 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi016.stderr:ceph-fuse[61146]: fuse finished with error 0 and tester_r 0 2023-05-22T15:46:05.996 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi016.front.sepia.ceph.com 2023-05-22T15:46:05.996 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:46:05.997 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-05-22T15:46:06.017 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:46:06.012+0000 7f614e4ed180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-05-22T15:46:06.081 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:46:06.076+0000 7f614e4ed180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-05-22T15:46:06.115 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:46:06.112+0000 7f4a1c01d180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-05-22T15:46:06.143 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2023-05-22T15:46:06.143 DEBUG:teuthology.orchestra.run.smithi062:> dd if=/proc/self/mounts of=/dev/stdout 2023-05-22T15:46:06.149 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16342116k,nr_inodes=4085529,mode=755,inode64 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275932k,mode=755,inode64 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev,inode64 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=13250 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275928k,nr_inodes=818982,mode=700,uid=1000,gid=1271,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275932k,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-05-22T15:46:06.149 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi062.front.sepia.ceph.com 2023-05-22T15:46:06.149 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:46:06.150 DEBUG:teuthology.orchestra.run.smithi062:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2023-05-22T15:46:06.196 INFO:teuthology.orchestra.run.smithi062.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2023-05-22T15:46:06.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-05-22T15:46:06.198 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-05-22T15:46:06.198 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2023-05-22T15:46:06.204 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16345192k,nr_inodes=4086298,mode=755,inode64 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3276544k,mode=755,inode64 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev,inode64 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=15732 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3276540k,nr_inodes=819135,mode=700,uid=1000,gid=1271,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3276544k,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-05-22T15:46:06.205 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi016.front.sepia.ceph.com 2023-05-22T15:46:06.205 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:46:06.205 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2023-05-22T15:46:06.207 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:46:06.204+0000 7f614e4ed180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-05-22T15:46:06.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-05-22T15:46:06.212 INFO:teuthology.orchestra.run.smithi016.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2023-05-22T15:46:06.212 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2023-05-22T15:46:06.212 DEBUG:teuthology.orchestra.run.smithi062:> dd if=/proc/self/mounts of=/dev/stdout 2023-05-22T15:46:06.245 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16342116k,nr_inodes=4085529,mode=755,inode64 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275932k,mode=755,inode64 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev,inode64 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=13250 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275928k,nr_inodes=818982,mode=700,uid=1000,gid=1271,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275932k,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-05-22T15:46:06.245 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi062.front.sepia.ceph.com 2023-05-22T15:46:06.246 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:46:06.246 DEBUG:teuthology.orchestra.run.smithi062:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2023-05-22T15:46:06.292 INFO:teuthology.orchestra.run.smithi062.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2023-05-22T15:46:06.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-05-22T15:46:06.295 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-05-22T15:46:06.412 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:06.408+0000 7f48064b0640 1 Processor -- start 2023-05-22T15:46:06.412 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:06.408+0000 7f48064b0640 1 -- start start 2023-05-22T15:46:06.412 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:06.408+0000 7f48064b0640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f48000c8250 0x7f480006c220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:06.412 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:06.408+0000 7f48064b0640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f48000cc6c0 0x7f480006c760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:06.413 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:06.408+0000 7f48064b0640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f480015c210 con 0x7f48001563c0 2023-05-22T15:46:06.413 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:06.408+0000 7f48064b0640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4800160700 con 0x7f48000cc6c0 2023-05-22T15:46:06.413 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:06.408+0000 7f48064b0640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f480015c480 con 0x7f48000c8250 2023-05-22T15:46:06.413 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:06.408+0000 7f4804a4d640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f48000cc6c0 0x7f480006c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:06.414 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:06.408+0000 7f47f7fff640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f48000c8250 0x7f480006c220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:06.414 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:06.408+0000 7f4804a4d640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f48000cc6c0 0x7f480006c760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.16:57676/0 (socket says 172.21.15.16:57676) 2023-05-22T15:46:06.414 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:06.408+0000 7f4804a4d640 1 -- 172.21.15.16:0/266427787 learned_addr learned my addr 172.21.15.16:0/266427787 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:46:06.414 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:06.408+0000 7f47f7fff640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f48000c8250 0x7f480006c220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3301/0 says I am v2:172.21.15.16:34744/0 (socket says 172.21.15.16:34744) 2023-05-22T15:46:06.414 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:06.408+0000 7f4804a4d640 1 -- 172.21.15.16:0/266427787 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f48000c8250 msgr2=0x7f480006c220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:46:06.414 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:06.408+0000 7f4804a4d640 1 --2- 172.21.15.16:0/266427787 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f48000c8250 0x7f480006c220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:06.415 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:06.408+0000 7f4804a4d640 1 -- 172.21.15.16:0/266427787 >> v1:172.21.15.62:6789/0 conn(0x7f48001563c0 legacy=0x7f48000c7b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:06.415 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:06.408+0000 7f4804a4d640 1 -- 172.21.15.16:0/266427787 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f480015c760 con 0x7f48000cc6c0 2023-05-22T15:46:06.415 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:06.412+0000 7f4804a4d640 1 --2- 172.21.15.16:0/266427787 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f48000cc6c0 0x7f480006c760 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f47f8009e50 tx=0x7f47f801c9e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a3aef9005a01af16 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:06.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:06.412+0000 7f47ff7fe640 1 -- 172.21.15.16:0/266427787 <== mon.1 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47f80a58a0 con 0x7f48000cc6c0 2023-05-22T15:46:06.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:06.412+0000 7f47ff7fe640 1 -- 172.21.15.16:0/266427787 <== mon.1 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47f80a5a00 con 0x7f48000cc6c0 2023-05-22T15:46:06.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:06.412+0000 7f47ff7fe640 1 -- 172.21.15.16:0/266427787 <== mon.1 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47f80a5d90 con 0x7f48000cc6c0 2023-05-22T15:46:06.417 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:06.412+0000 7f48064b0640 1 -- 172.21.15.16:0/266427787 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f48000cc6c0 msgr2=0x7f480006c760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:06.417 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:06.412+0000 7f48064b0640 1 --2- 172.21.15.16:0/266427787 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f48000cc6c0 0x7f480006c760 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f47f8009e50 tx=0x7f47f801c9e0 comp rx=0 tx=0).stop 2023-05-22T15:46:06.417 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:06.412+0000 7f48064b0640 1 -- 172.21.15.16:0/266427787 shutdown_connections 2023-05-22T15:46:06.417 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:06.412+0000 7f48064b0640 1 --2- 172.21.15.16:0/266427787 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f48000cc6c0 0x7f480006c760 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:06.417 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:06.412+0000 7f48064b0640 1 --2- 172.21.15.16:0/266427787 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f48000c8250 0x7f480006c220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:06.417 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:06.412+0000 7f48064b0640 1 -- 172.21.15.16:0/266427787 >> 172.21.15.16:0/266427787 conn(0x7f48000b1110 msgr2=0x7f48000ce830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:46:06.417 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:06.412+0000 7f48064b0640 1 -- 172.21.15.16:0/266427787 shutdown_connections 2023-05-22T15:46:06.418 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:06.412+0000 7f48064b0640 1 -- 172.21.15.16:0/266427787 wait complete. 2023-05-22T15:46:06.418 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:06.412+0000 7f48064b0640 1 Processor -- start 2023-05-22T15:46:06.418 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:06.412+0000 7f48064b0640 1 -- start start 2023-05-22T15:46:06.420 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:06.416+0000 7f48064b0640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f48000c8250 0x7f480015abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:06.420 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:06.416+0000 7f48064b0640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f48000cc6c0 0x7f4800146670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:06.420 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:06.416+0000 7f48064b0640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f48001558e0 0x7f480014cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:06.420 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:06.416+0000 7f48064b0640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4800151da0 con 0x7f48001558e0 2023-05-22T15:46:06.420 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:06.416+0000 7f48064b0640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48000c9190 con 0x7f48000c8250 2023-05-22T15:46:06.420 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:06.416+0000 7f48064b0640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48000caa00 con 0x7f48000cc6c0 2023-05-22T15:46:06.421 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:06.416+0000 7f4804a4d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f48001558e0 0x7f480014cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:06.421 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:06.416+0000 7f4804a4d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f48001558e0 0x7f480014cce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59450/0 (socket says 172.21.15.16:59450) 2023-05-22T15:46:06.421 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:06.416+0000 7f4804a4d640 1 -- 172.21.15.16:0/556993209 learned_addr learned my addr 172.21.15.16:0/556993209 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:46:06.421 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:06.416+0000 7f47fffff640 1 --2- 172.21.15.16:0/556993209 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f48000c8250 0x7f480015abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:06.421 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:06.416+0000 7f47f7fff640 1 --2- 172.21.15.16:0/556993209 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f48000cc6c0 0x7f4800146670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:06.421 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:06.416+0000 7f4804a4d640 1 -- 172.21.15.16:0/556993209 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f48000cc6c0 msgr2=0x7f4800146670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:06.421 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:06.416+0000 7f4804a4d640 1 --2- 172.21.15.16:0/556993209 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f48000cc6c0 0x7f4800146670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:06.421 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:06.416+0000 7f4804a4d640 1 -- 172.21.15.16:0/556993209 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f48000c8250 msgr2=0x7f480015abf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:06.422 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:06.416+0000 7f4804a4d640 1 --2- 172.21.15.16:0/556993209 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f48000c8250 0x7f480015abf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:06.422 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:06.416+0000 7f4804a4d640 1 -- 172.21.15.16:0/556993209 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47f8009b30 con 0x7f48001558e0 2023-05-22T15:46:06.422 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:06.416+0000 7f4804a4d640 1 --2- 172.21.15.16:0/556993209 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f48001558e0 0x7f480014cce0 secure :-1 s=READY pgs=1729 cs=0 l=1 rev1=1 crypto rx=0x7f47f801a610 tx=0x7f47f8019780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:06.423 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:06.416+0000 7f47fd7fa640 1 -- 172.21.15.16:0/556993209 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47f80b8670 con 0x7f48001558e0 2023-05-22T15:46:06.423 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:06.416+0000 7f47fd7fa640 1 -- 172.21.15.16:0/556993209 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47f80b87d0 con 0x7f48001558e0 2023-05-22T15:46:06.423 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:06.416+0000 7f48064b0640 1 -- 172.21.15.16:0/556993209 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f480006ce30 con 0x7f48001558e0 2023-05-22T15:46:06.423 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:06.416+0000 7f47fd7fa640 1 -- 172.21.15.16:0/556993209 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47f80b8b80 con 0x7f48001558e0 2023-05-22T15:46:06.423 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:06.416+0000 7f48064b0640 1 -- 172.21.15.16:0/556993209 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f480006cca0 con 0x7f48001558e0 2023-05-22T15:46:06.424 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:06.416+0000 7f48064b0640 1 -- 172.21.15.16:0/556993209 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f480015ff00 con 0x7f48001558e0 2023-05-22T15:46:06.424 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:06.416+0000 7f47fd7fa640 1 -- 172.21.15.16:0/556993209 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 57) v1 ==== 84446+0+0 (secure 0 0 0) 0x7f47f80b8ce0 con 0x7f48001558e0 2023-05-22T15:46:06.424 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:06.416+0000 7f47fd7fa640 1 --2- 172.21.15.16:0/556993209 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f47e00655f0 0x7f47e0067aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:06.424 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:06.416+0000 7f47fffff640 1 -- 172.21.15.16:0/556993209 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f47e00655f0 msgr2=0x7f47e0067aa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/36921 2023-05-22T15:46:06.424 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:06.416+0000 7f47fd7fa640 1 -- 172.21.15.16:0/556993209 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(133..133 src has 1..133) v4 ==== 6726+0+0 (secure 0 0 0) 0x7f47f811e4c0 con 0x7f48001558e0 2023-05-22T15:46:06.424 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:06.416+0000 7f47fffff640 1 --2- 172.21.15.16:0/556993209 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f47e00655f0 0x7f47e0067aa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-05-22T15:46:06.426 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:06.420+0000 7f47fd7fa640 1 -- 172.21.15.16:0/556993209 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f47f80ebba0 con 0x7f48001558e0 2023-05-22T15:46:06.607 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:46:06.604+0000 7f4a1c01d180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-05-22T15:46:06.612 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:06.608+0000 7f48064b0640 1 -- 172.21.15.16:0/556993209 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f480004dbe0 con 0x7f48001558e0 2023-05-22T15:46:06.613 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:06.608+0000 7f47fd7fa640 1 -- 172.21.15.16:0/556993209 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 50 v50) v1 ==== 94+0+4644 (secure 0 0 0) 0x7f47f80ef1c0 con 0x7f48001558e0 2023-05-22T15:46:06.614 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:46:06.614 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":50,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6689,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/296693182","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6833","nonce":296693182},{"type":"v1","addr":"172.21.15.62:6835","nonce":296693182}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":46},{"gid":7402,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6834/856426578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":856426578},{"type":"v1","addr":"172.21.15.62:6834","nonce":856426578}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":46},{"gid":8541,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/2260895833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2260895833},{"type":"v1","addr":"172.21.15.16:6835","nonce":2260895833}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":46}],"filesystems":[{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-05-22T15:45:05.200760+0000","modified":"2023-05-22T15:45:07.832845+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8547},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8547":{"gid":8547,"name":"a","rank":0,"incarnation":47,"state":"up:active","state_seq":3,"addr":"172.21.15.16:6837/1566663454","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":1566663454},{"type":"v1","addr":"172.21.15.16:6837","nonce":1566663454}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":7}]} 2023-05-22T15:46:06.614 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 50 2023-05-22T15:46:06.615 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:06.612+0000 7f48064b0640 1 -- 172.21.15.16:0/556993209 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f47e00655f0 msgr2=0x7f47e0067aa0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-05-22T15:46:06.615 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:06.612+0000 7f48064b0640 1 --2- 172.21.15.16:0/556993209 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f47e00655f0 0x7f47e0067aa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:06.615 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:06.612+0000 7f48064b0640 1 -- 172.21.15.16:0/556993209 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f48001558e0 msgr2=0x7f480014cce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:06.615 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:06.612+0000 7f48064b0640 1 --2- 172.21.15.16:0/556993209 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f48001558e0 0x7f480014cce0 secure :-1 s=READY pgs=1729 cs=0 l=1 rev1=1 crypto rx=0x7f47f801a610 tx=0x7f47f8019780 comp rx=0 tx=0).stop 2023-05-22T15:46:06.616 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:06.612+0000 7f48064b0640 1 -- 172.21.15.16:0/556993209 shutdown_connections 2023-05-22T15:46:06.616 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:06.612+0000 7f48064b0640 1 --2- 172.21.15.16:0/556993209 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f48001558e0 0x7f480014cce0 unknown :-1 s=CLOSED pgs=1729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:06.616 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:06.612+0000 7f48064b0640 1 --2- 172.21.15.16:0/556993209 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f47e00655f0 0x7f47e0067aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:06.616 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:06.612+0000 7f48064b0640 1 --2- 172.21.15.16:0/556993209 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f48000cc6c0 0x7f4800146670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:06.616 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:06.612+0000 7f48064b0640 1 --2- 172.21.15.16:0/556993209 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f48000c8250 0x7f480015abf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:06.616 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:06.612+0000 7f48064b0640 1 -- 172.21.15.16:0/556993209 >> 172.21.15.16:0/556993209 conn(0x7f48000b1110 msgr2=0x7f48000ccb40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:46:06.616 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:06.612+0000 7f48064b0640 1 -- 172.21.15.16:0/556993209 shutdown_connections 2023-05-22T15:46:06.617 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:06.612+0000 7f48064b0640 1 -- 172.21.15.16:0/556993209 wait complete. 2023-05-22T15:46:06.631 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-05-22T15:46:06.683 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:46:06.680+0000 7f4a1c01d180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-05-22T15:46:06.708 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:46:06.704+0000 7f614e4ed180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-05-22T15:46:06.751 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:06.748+0000 7f1e0fde7640 1 Processor -- start 2023-05-22T15:46:06.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:06.748+0000 7f1e0fde7640 1 -- start start 2023-05-22T15:46:06.753 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:06.748+0000 7f1e0fde7640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1e0815c130 0x7f1e080b2330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:06.753 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:06.748+0000 7f1e0fde7640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f1e080cfc20 0x7f1e080ca8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:06.753 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:06.748+0000 7f1e0fde7640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e080b2cc0 con 0x7f1e080b2870 2023-05-22T15:46:06.753 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:06.748+0000 7f1e0fde7640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e0806c4a0 con 0x7f1e080cfc20 2023-05-22T15:46:06.753 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:06.748+0000 7f1e0fde7640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e080ce160 con 0x7f1e0815c130 2023-05-22T15:46:06.754 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:06.748+0000 7f1e0e384640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f1e080cfc20 0x7f1e080ca8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:06.754 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:06.748+0000 7f1e0db83640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1e0815c130 0x7f1e080b2330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:06.754 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:06.748+0000 7f1e0d382640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f1e080b2870 0x7f1e080cf510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.16:41768/0 (socket says 172.21.15.16:41768) 2023-05-22T15:46:06.754 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:06.748+0000 7f1e0d382640 1 -- 172.21.15.16:0/1687208740 learned_addr learned my addr 172.21.15.16:0/1687208740 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:46:06.754 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:06.748+0000 7f1e0db83640 1 -- 172.21.15.16:0/1687208740 >> v2:172.21.15.62:3300/0 conn(0x7f1e080cfc20 msgr2=0x7f1e080ca8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:46:06.754 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:06.748+0000 7f1e0db83640 1 --2- 172.21.15.16:0/1687208740 >> v2:172.21.15.62:3300/0 conn(0x7f1e080cfc20 0x7f1e080ca8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:06.754 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:06.748+0000 7f1e0db83640 1 -- 172.21.15.16:0/1687208740 >> v1:172.21.15.62:6789/0 conn(0x7f1e080b2870 legacy=0x7f1e080cf510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:06.755 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:06.748+0000 7f1e0db83640 1 -- 172.21.15.16:0/1687208740 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e080cece0 con 0x7f1e0815c130 2023-05-22T15:46:06.755 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:06.752+0000 7f1e0db83640 1 --2- 172.21.15.16:0/1687208740 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1e0815c130 0x7f1e080b2330 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f1e040097c0 tx=0x7f1e0401c480 comp rx=0 tx=0).ready entity=mon.2 client_cookie=982d1cdd12d1d685 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:06.756 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:06.752+0000 7f1e0cb81640 1 -- 172.21.15.16:0/1687208740 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e040ab600 con 0x7f1e0815c130 2023-05-22T15:46:06.756 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:06.752+0000 7f1e0cb81640 1 -- 172.21.15.16:0/1687208740 <== mon.2 v2:172.21.15.62:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e040a7050 con 0x7f1e0815c130 2023-05-22T15:46:06.756 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:06.752+0000 7f1e0cb81640 1 -- 172.21.15.16:0/1687208740 <== mon.2 v2:172.21.15.62:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e040ab980 con 0x7f1e0815c130 2023-05-22T15:46:06.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:06.752+0000 7f1e0fde7640 1 -- 172.21.15.16:0/1687208740 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1e0815c130 msgr2=0x7f1e080b2330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:06.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:06.752+0000 7f1e0fde7640 1 --2- 172.21.15.16:0/1687208740 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1e0815c130 0x7f1e080b2330 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f1e040097c0 tx=0x7f1e0401c480 comp rx=0 tx=0).stop 2023-05-22T15:46:06.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:06.752+0000 7f1e0fde7640 1 -- 172.21.15.16:0/1687208740 shutdown_connections 2023-05-22T15:46:06.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:06.752+0000 7f1e0fde7640 1 --2- 172.21.15.16:0/1687208740 >> v2:172.21.15.62:3300/0 conn(0x7f1e080cfc20 0x7f1e080ca8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:06.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:06.752+0000 7f1e0fde7640 1 --2- 172.21.15.16:0/1687208740 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1e0815c130 0x7f1e080b2330 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:06.758 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:06.752+0000 7f1e0fde7640 1 -- 172.21.15.16:0/1687208740 >> 172.21.15.16:0/1687208740 conn(0x7f1e0806f5c0 msgr2=0x7f1e0806f9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:46:06.758 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:06.752+0000 7f1e0fde7640 1 -- 172.21.15.16:0/1687208740 shutdown_connections 2023-05-22T15:46:06.758 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:06.752+0000 7f1e0fde7640 1 -- 172.21.15.16:0/1687208740 wait complete. 2023-05-22T15:46:06.761 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:06.752+0000 7f1e0fde7640 1 Processor -- start 2023-05-22T15:46:06.762 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:06.752+0000 7f1e0fde7640 1 -- start start 2023-05-22T15:46:06.762 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:06.756+0000 7f1e0fde7640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1e080b2870 0x7f1e081546f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:06.762 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:06.756+0000 7f1e0fde7640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1e080cfc20 0x7f1e081572a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:06.762 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:06.756+0000 7f1e0fde7640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1e0815c130 0x7f1e081577e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:06.763 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:06.756+0000 7f1e0fde7640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e080b4940 con 0x7f1e0815c130 2023-05-22T15:46:06.763 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:06.756+0000 7f1e0fde7640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e080b3d30 con 0x7f1e080b2870 2023-05-22T15:46:06.763 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:06.756+0000 7f1e0fde7640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e080b4010 con 0x7f1e080cfc20 2023-05-22T15:46:06.764 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:06.756+0000 7f1e0e384640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1e0815c130 0x7f1e081577e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:06.764 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:06.756+0000 7f1e0e384640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1e0815c130 0x7f1e081577e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59460/0 (socket says 172.21.15.16:59460) 2023-05-22T15:46:06.764 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:06.756+0000 7f1e0e384640 1 -- 172.21.15.16:0/2285661583 learned_addr learned my addr 172.21.15.16:0/2285661583 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:46:06.764 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:06.756+0000 7f1e0db83640 1 --2- 172.21.15.16:0/2285661583 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1e080b2870 0x7f1e081546f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:06.765 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:06.756+0000 7f1e0d382640 1 --2- 172.21.15.16:0/2285661583 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1e080cfc20 0x7f1e081572a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:06.765 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:06.756+0000 7f1e0e384640 1 -- 172.21.15.16:0/2285661583 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1e080cfc20 msgr2=0x7f1e081572a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:06.765 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:06.756+0000 7f1e0e384640 1 --2- 172.21.15.16:0/2285661583 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1e080cfc20 0x7f1e081572a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:06.765 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:06.756+0000 7f1e0e384640 1 -- 172.21.15.16:0/2285661583 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1e080b2870 msgr2=0x7f1e081546f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:06.765 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:06.756+0000 7f1e0e384640 1 --2- 172.21.15.16:0/2285661583 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1e080b2870 0x7f1e081546f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:06.765 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:06.756+0000 7f1e0e384640 1 -- 172.21.15.16:0/2285661583 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e04009420 con 0x7f1e0815c130 2023-05-22T15:46:06.765 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:06.756+0000 7f1e0db83640 1 --2- 172.21.15.16:0/2285661583 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1e080b2870 0x7f1e081546f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T15:46:06.766 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:06.756+0000 7f1e0e384640 1 --2- 172.21.15.16:0/2285661583 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1e0815c130 0x7f1e081577e0 secure :-1 s=READY pgs=1730 cs=0 l=1 rev1=1 crypto rx=0x7f1e0000efc0 tx=0x7f1e0000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:06.766 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:06.756+0000 7f1dfe7fc640 1 -- 172.21.15.16:0/2285661583 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e00009ba0 con 0x7f1e0815c130 2023-05-22T15:46:06.766 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:06.756+0000 7f1dfe7fc640 1 -- 172.21.15.16:0/2285661583 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e0000f040 con 0x7f1e0815c130 2023-05-22T15:46:06.766 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:06.756+0000 7f1dfe7fc640 1 -- 172.21.15.16:0/2285661583 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e000146c0 con 0x7f1e0815c130 2023-05-22T15:46:06.766 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:06.756+0000 7f1e0fde7640 1 -- 172.21.15.16:0/2285661583 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e080cb600 con 0x7f1e0815c130 2023-05-22T15:46:06.766 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:06.756+0000 7f1e0fde7640 1 -- 172.21.15.16:0/2285661583 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e080ce970 con 0x7f1e0815c130 2023-05-22T15:46:06.766 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:06.756+0000 7f1e0fde7640 1 -- 172.21.15.16:0/2285661583 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e080cb790 con 0x7f1e0815c130 2023-05-22T15:46:06.768 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:06.756+0000 7f1dfe7fc640 1 -- 172.21.15.16:0/2285661583 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 57) v1 ==== 84446+0+0 (secure 0 0 0) 0x7f1e00014820 con 0x7f1e0815c130 2023-05-22T15:46:06.768 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:06.756+0000 7f1dfe7fc640 1 --2- 172.21.15.16:0/2285661583 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f1dec0655c0 0x7f1dec067a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:06.769 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:06.756+0000 7f1e0db83640 1 -- 172.21.15.16:0/2285661583 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f1dec0655c0 msgr2=0x7f1dec067a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/36921 2023-05-22T15:46:06.769 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:06.756+0000 7f1dfe7fc640 1 -- 172.21.15.16:0/2285661583 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(133..133 src has 1..133) v4 ==== 6726+0+0 (secure 0 0 0) 0x7f1e00083e70 con 0x7f1e0815c130 2023-05-22T15:46:06.769 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:06.756+0000 7f1e0db83640 1 --2- 172.21.15.16:0/2285661583 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f1dec0655c0 0x7f1dec067a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-05-22T15:46:06.769 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:06.760+0000 7f1dfe7fc640 1 -- 172.21.15.16:0/2285661583 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f1e00051340 con 0x7f1e0815c130 2023-05-22T15:46:06.787 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:46:06.784+0000 7f614e4ed180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-05-22T15:46:06.821 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:46:06.820+0000 7f4a1c01d180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-05-22T15:46:06.897 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:46:06.896+0000 7f4a1c01d180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-05-22T15:46:06.922 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:06.916+0000 7f1e0fde7640 1 -- 172.21.15.16:0/2285661583 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1e080b4af0 con 0x7f1e0815c130 2023-05-22T15:46:06.923 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:06.920+0000 7f1dfe7fc640 1 -- 172.21.15.16:0/2285661583 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v133) v1 ==== 74+0+18487 (secure 0 0 0) 0x7f1e080b4af0 con 0x7f1e0815c130 2023-05-22T15:46:06.924 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:46:06.924 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":133,"fsid":"b2719834-7847-4d98-90a6-f65d57f85c49","created":"2023-05-22T15:26:25.259314+0000","modified":"2023-05-22T15:45:17.648468+0000","last_up_change":"2023-05-22T15:26:36.448524+0000","last_in_change":"2023-05-22T15:26:27.675668+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-05-22T15:26:38.019781+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-05-22T15:26:43.570514+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2023-05-22T15:45:03.718869+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"131","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2023-05-22T15:45:04.168135+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"132","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9e7ff854-b25f-4cbb-a5c2-1799d9c53d2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":37021},{"type":"v1","addr":"172.21.15.16:6825","nonce":37021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":37021},{"type":"v1","addr":"172.21.15.16:6827","nonce":37021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":37021},{"type":"v1","addr":"172.21.15.16:6831","nonce":37021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":37021},{"type":"v1","addr":"172.21.15.16:6829","nonce":37021}]},"public_addr":"172.21.15.16:6825/37021","cluster_addr":"172.21.15.16:6827/37021","heartbeat_back_addr":"172.21.15.16:6831/37021","heartbeat_front_addr":"172.21.15.16:6829/37021","state":["exists","up"]},{"osd":1,"uuid":"4702fbc9-a927-4565-aca5-41dc69cc8264","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":37008},{"type":"v1","addr":"172.21.15.16:6801","nonce":37008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":37008},{"type":"v1","addr":"172.21.15.16:6803","nonce":37008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":37008},{"type":"v1","addr":"172.21.15.16:6807","nonce":37008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":37008},{"type":"v1","addr":"172.21.15.16:6805","nonce":37008}]},"public_addr":"172.21.15.16:6801/37008","cluster_addr":"172.21.15.16:6803/37008","heartbeat_back_addr":"172.21.15.16:6807/37008","heartbeat_front_addr":"172.21.15.16:6805/37008","state":["exists","up"]},{"osd":2,"uuid":"1e7bba46-4eba-47ef-bb81-5e2d371638ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":37009},{"type":"v1","addr":"172.21.15.16:6809","nonce":37009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":37009},{"type":"v1","addr":"172.21.15.16:6811","nonce":37009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":37009},{"type":"v1","addr":"172.21.15.16:6815","nonce":37009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":37009},{"type":"v1","addr":"172.21.15.16:6813","nonce":37009}]},"public_addr":"172.21.15.16:6809/37009","cluster_addr":"172.21.15.16:6811/37009","heartbeat_back_addr":"172.21.15.16:6815/37009","heartbeat_front_addr":"172.21.15.16:6813/37009","state":["exists","up"]},{"osd":3,"uuid":"383cacc0-5a88-4904-990e-498ca59ef48a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":37020},{"type":"v1","addr":"172.21.15.16:6817","nonce":37020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":37020},{"type":"v1","addr":"172.21.15.16:6819","nonce":37020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":37020},{"type":"v1","addr":"172.21.15.16:6823","nonce":37020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":37020},{"type":"v1","addr":"172.21.15.16:6821","nonce":37020}]},"public_addr":"172.21.15.16:6817/37020","cluster_addr":"172.21.15.16:6819/37020","heartbeat_back_addr":"172.21.15.16:6823/37020","heartbeat_front_addr":"172.21.15.16:6821/37020","state":["exists","up"]},{"osd":4,"uuid":"970500ef-95b7-42c1-b080-eed4fae7013d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":37353},{"type":"v1","addr":"172.21.15.62:6817","nonce":37353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6819","nonce":37353},{"type":"v1","addr":"172.21.15.62:6822","nonce":37353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37353},{"type":"v1","addr":"172.21.15.62:6831","nonce":37353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37353},{"type":"v1","addr":"172.21.15.62:6826","nonce":37353}]},"public_addr":"172.21.15.62:6817/37353","cluster_addr":"172.21.15.62:6822/37353","heartbeat_back_addr":"172.21.15.62:6831/37353","heartbeat_front_addr":"172.21.15.62:6826/37353","state":["exists","up"]},{"osd":5,"uuid":"022d67db-d4d8-445b-b1b9-8b446a00f5c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":37352},{"type":"v1","addr":"172.21.15.62:6801","nonce":37352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":37352},{"type":"v1","addr":"172.21.15.62:6803","nonce":37352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":37352},{"type":"v1","addr":"172.21.15.62:6807","nonce":37352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":37352},{"type":"v1","addr":"172.21.15.62:6805","nonce":37352}]},"public_addr":"172.21.15.62:6801/37352","cluster_addr":"172.21.15.62:6803/37352","heartbeat_back_addr":"172.21.15.62:6807/37352","heartbeat_front_addr":"172.21.15.62:6805/37352","state":["exists","up"]},{"osd":6,"uuid":"42bd372b-6688-4a1b-8407-e16e8c266e33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":37354},{"type":"v1","addr":"172.21.15.62:6820","nonce":37354}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":37354},{"type":"v1","addr":"172.21.15.62:6823","nonce":37354}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6829","nonce":37354},{"type":"v1","addr":"172.21.15.62:6830","nonce":37354}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6825","nonce":37354},{"type":"v1","addr":"172.21.15.62:6827","nonce":37354}]},"public_addr":"172.21.15.62:6820/37354","cluster_addr":"172.21.15.62:6823/37354","heartbeat_back_addr":"172.21.15.62:6830/37354","heartbeat_front_addr":"172.21.15.62:6827/37354","state":["exists","up"]},{"osd":7,"uuid":"ac70b2a3-f41b-4bf2-b227-7f2cbb55426d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37355},{"type":"v1","addr":"172.21.15.62:6809","nonce":37355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37355},{"type":"v1","addr":"172.21.15.62:6811","nonce":37355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":37355},{"type":"v1","addr":"172.21.15.62:6815","nonce":37355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":37355},{"type":"v1","addr":"172.21.15.62:6813","nonce":37355}]},"public_addr":"172.21.15.62:6809/37355","cluster_addr":"172.21.15.62:6811/37355","heartbeat_back_addr":"172.21.15.62:6815/37355","heartbeat_front_addr":"172.21.15.62:6813/37355","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.368896+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.107273+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.276888+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.287502+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.124791+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.251937+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.255679+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.232797+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:0/1950855917":"2023-05-23T15:45:17.648374+0000","172.21.15.16:0/1889818530":"2023-05-23T15:45:17.648374+0000","172.21.15.16:6833/36921":"2023-05-23T15:45:17.648374+0000","172.21.15.16:0/4160466619":"2023-05-23T15:45:17.648374+0000","172.21.15.16:0/55601757":"2023-05-23T15:45:17.648374+0000","172.21.15.16:6832/36921":"2023-05-23T15:45:17.648374+0000","172.21.15.16:0/4127127856":"2023-05-23T15:45:17.648374+0000","172.21.15.16:0/2333289485":"2023-05-23T15:45:17.648374+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-05-22T15:46:06.925 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:06.920+0000 7f1e0fde7640 1 -- 172.21.15.16:0/2285661583 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f1dec0655c0 msgr2=0x7f1dec067a70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-05-22T15:46:06.926 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:06.920+0000 7f1e0fde7640 1 --2- 172.21.15.16:0/2285661583 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f1dec0655c0 0x7f1dec067a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:06.926 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:06.920+0000 7f1e0fde7640 1 -- 172.21.15.16:0/2285661583 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1e0815c130 msgr2=0x7f1e081577e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:06.926 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:06.920+0000 7f1e0fde7640 1 --2- 172.21.15.16:0/2285661583 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1e0815c130 0x7f1e081577e0 secure :-1 s=READY pgs=1730 cs=0 l=1 rev1=1 crypto rx=0x7f1e0000efc0 tx=0x7f1e0000c490 comp rx=0 tx=0).stop 2023-05-22T15:46:06.926 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:06.920+0000 7f1e0fde7640 1 -- 172.21.15.16:0/2285661583 shutdown_connections 2023-05-22T15:46:06.926 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:06.920+0000 7f1e0fde7640 1 --2- 172.21.15.16:0/2285661583 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1e0815c130 0x7f1e081577e0 unknown :-1 s=CLOSED pgs=1730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:06.926 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:06.920+0000 7f1e0fde7640 1 --2- 172.21.15.16:0/2285661583 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f1dec0655c0 0x7f1dec067a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:06.927 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:06.920+0000 7f1e0fde7640 1 --2- 172.21.15.16:0/2285661583 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1e080cfc20 0x7f1e081572a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:06.927 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:06.920+0000 7f1e0fde7640 1 --2- 172.21.15.16:0/2285661583 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1e080b2870 0x7f1e081546f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:06.927 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:06.920+0000 7f1e0fde7640 1 -- 172.21.15.16:0/2285661583 >> 172.21.15.16:0/2285661583 conn(0x7f1e0806f5c0 msgr2=0x7f1e080cf300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:46:06.927 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:06.920+0000 7f1e0fde7640 1 -- 172.21.15.16:0/2285661583 shutdown_connections 2023-05-22T15:46:06.927 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:06.920+0000 7f1e0fde7640 1 -- 172.21.15.16:0/2285661583 wait complete. 2023-05-22T15:46:06.930 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:46:06.924+0000 7f614e4ed180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-05-22T15:46:06.941 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-05-22T15:26:38.019781+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-05-22T15:26:43.570514+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 16, 'pool_name': 'cephfs_metadata', 'create_time': '2023-05-22T15:45:03.718869+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '131', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 17, 'pool_name': 'cephfs_data', 'create_time': '2023-05-22T15:45:04.168135+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '132', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-05-22T15:46:06.942 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-05-22T15:46:07.009 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:46:07.004+0000 7f614e4ed180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-05-22T15:46:07.066 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.060+0000 7fd3a4b14640 1 Processor -- start 2023-05-22T15:46:07.067 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.060+0000 7fd3a4b14640 1 -- start start 2023-05-22T15:46:07.068 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.064+0000 7fd3a4b14640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fd3a015cfa0 0x7fd3a006db00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:07.068 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.064+0000 7fd3a4b14640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd3a00cf870 0x7fd3a00b1fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:07.068 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.064+0000 7fd3a4b14640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd3a006e740 0x7fd3a006ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:07.068 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.064+0000 7fd3a4b14640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3a00cb9f0 con 0x7fd3a00cf870 2023-05-22T15:46:07.068 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.064+0000 7fd3a4b14640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3a006f100 con 0x7fd3a015cfa0 2023-05-22T15:46:07.068 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.064+0000 7fd3a4b14640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3a006e040 con 0x7fd3a006e740 2023-05-22T15:46:07.068 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.064+0000 7fd39dd9b640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd3a00cf870 0x7fd3a00b1fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:07.069 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.064+0000 7fd39dd9b640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd3a00cf870 0x7fd3a00b1fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59476/0 (socket says 172.21.15.16:59476) 2023-05-22T15:46:07.069 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.064+0000 7fd39dd9b640 1 -- 172.21.15.16:0/2931844890 learned_addr learned my addr 172.21.15.16:0/2931844890 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:46:07.069 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.064+0000 7fd39ed9d640 1 --2- 172.21.15.16:0/2931844890 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd3a006e740 0x7fd3a006ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:07.069 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.064+0000 7fd39e59c640 1 --2- 172.21.15.16:0/2931844890 >> v2:172.21.15.62:3300/0 conn(0x7fd3a015cfa0 0x7fd3a006db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:07.069 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.064+0000 7fd39dd9b640 1 -- 172.21.15.16:0/2931844890 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd3a006e740 msgr2=0x7fd3a006ebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:46:07.069 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.064+0000 7fd39dd9b640 1 --2- 172.21.15.16:0/2931844890 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd3a006e740 0x7fd3a006ebc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:07.069 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.064+0000 7fd39dd9b640 1 -- 172.21.15.16:0/2931844890 >> v2:172.21.15.62:3300/0 conn(0x7fd3a015cfa0 msgr2=0x7fd3a006db00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:46:07.070 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.064+0000 7fd39dd9b640 1 --2- 172.21.15.16:0/2931844890 >> v2:172.21.15.62:3300/0 conn(0x7fd3a015cfa0 0x7fd3a006db00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:07.070 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.064+0000 7fd39dd9b640 1 -- 172.21.15.16:0/2931844890 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3a00b24f0 con 0x7fd3a00cf870 2023-05-22T15:46:07.070 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.064+0000 7fd39dd9b640 1 --2- 172.21.15.16:0/2931844890 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd3a00cf870 0x7fd3a00b1fb0 secure :-1 s=READY pgs=1731 cs=0 l=1 rev1=1 crypto rx=0x7fd3a00b1d50 tx=0x7fd38801c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80c50e798f60ab40 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:07.071 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.068+0000 7fd39d59a640 1 -- 172.21.15.16:0/2931844890 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3880ac410 con 0x7fd3a00cf870 2023-05-22T15:46:07.072 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.068+0000 7fd39d59a640 1 -- 172.21.15.16:0/2931844890 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3880ac570 con 0x7fd3a00cf870 2023-05-22T15:46:07.072 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.068+0000 7fd39d59a640 1 -- 172.21.15.16:0/2931844890 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3880ac900 con 0x7fd3a00cf870 2023-05-22T15:46:07.072 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.068+0000 7fd3a4b14640 1 -- 172.21.15.16:0/2931844890 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd3a00cf870 msgr2=0x7fd3a00b1fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:07.072 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.068+0000 7fd3a4b14640 1 --2- 172.21.15.16:0/2931844890 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd3a00cf870 0x7fd3a00b1fb0 secure :-1 s=READY pgs=1731 cs=0 l=1 rev1=1 crypto rx=0x7fd3a00b1d50 tx=0x7fd38801c500 comp rx=0 tx=0).stop 2023-05-22T15:46:07.072 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.068+0000 7fd3a4b14640 1 -- 172.21.15.16:0/2931844890 shutdown_connections 2023-05-22T15:46:07.072 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.068+0000 7fd3a4b14640 1 --2- 172.21.15.16:0/2931844890 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd3a00cf870 0x7fd3a00b1fb0 unknown :-1 s=CLOSED pgs=1731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:07.072 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.068+0000 7fd3a4b14640 1 --2- 172.21.15.16:0/2931844890 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd3a006e740 0x7fd3a006ebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:07.073 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.068+0000 7fd3a4b14640 1 --2- 172.21.15.16:0/2931844890 >> v2:172.21.15.62:3300/0 conn(0x7fd3a015cfa0 0x7fd3a006db00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:07.073 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.068+0000 7fd3a4b14640 1 -- 172.21.15.16:0/2931844890 >> 172.21.15.16:0/2931844890 conn(0x7fd3a00cc870 msgr2=0x7fd3a00cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:46:07.073 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.068+0000 7fd3a4b14640 1 -- 172.21.15.16:0/2931844890 shutdown_connections 2023-05-22T15:46:07.073 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.068+0000 7fd3a4b14640 1 -- 172.21.15.16:0/2931844890 wait complete. 2023-05-22T15:46:07.073 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.068+0000 7fd3a4b14640 1 Processor -- start 2023-05-22T15:46:07.074 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.068+0000 7fd3a4b14640 1 -- start start 2023-05-22T15:46:07.075 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.068+0000 7fd3a4b14640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd3a006e740 0x7fd3a01f5b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:07.075 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.068+0000 7fd3a4b14640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd3a00cf870 0x7fd3a0157230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:07.075 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.068+0000 7fd3a4b14640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd3a015cfa0 0x7fd3a015a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:07.075 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.068+0000 7fd3a4b14640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3a0136b90 con 0x7fd3a015cfa0 2023-05-22T15:46:07.075 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.068+0000 7fd3a4b14640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3a0157770 con 0x7fd3a006e740 2023-05-22T15:46:07.075 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.068+0000 7fd3a4b14640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3a0157a40 con 0x7fd3a00cf870 2023-05-22T15:46:07.075 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.068+0000 7fd39ed9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd3a015cfa0 0x7fd3a015a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:07.076 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.068+0000 7fd39ed9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd3a015cfa0 0x7fd3a015a950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59490/0 (socket says 172.21.15.16:59490) 2023-05-22T15:46:07.076 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.068+0000 7fd39ed9d640 1 -- 172.21.15.16:0/1003939556 learned_addr learned my addr 172.21.15.16:0/1003939556 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:46:07.076 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.068+0000 7fd39e59c640 1 --2- 172.21.15.16:0/1003939556 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd3a006e740 0x7fd3a01f5b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:07.076 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.068+0000 7fd39dd9b640 1 --2- 172.21.15.16:0/1003939556 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd3a00cf870 0x7fd3a0157230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:07.076 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.072+0000 7fd39ed9d640 1 -- 172.21.15.16:0/1003939556 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd3a00cf870 msgr2=0x7fd3a0157230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:07.076 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.072+0000 7fd39ed9d640 1 --2- 172.21.15.16:0/1003939556 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd3a00cf870 0x7fd3a0157230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:07.076 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.072+0000 7fd39ed9d640 1 -- 172.21.15.16:0/1003939556 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd3a006e740 msgr2=0x7fd3a01f5b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:07.076 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.072+0000 7fd39ed9d640 1 --2- 172.21.15.16:0/1003939556 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd3a006e740 0x7fd3a01f5b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:07.077 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.072+0000 7fd39ed9d640 1 -- 172.21.15.16:0/1003939556 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3880094d0 con 0x7fd3a015cfa0 2023-05-22T15:46:07.077 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.072+0000 7fd39ed9d640 1 --2- 172.21.15.16:0/1003939556 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd3a015cfa0 0x7fd3a015a950 secure :-1 s=READY pgs=1732 cs=0 l=1 rev1=1 crypto rx=0x7fd39000d800 tx=0x7fd39000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:07.078 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.072+0000 7fd396ffd640 1 -- 172.21.15.16:0/1003939556 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3900096c0 con 0x7fd3a015cfa0 2023-05-22T15:46:07.078 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.072+0000 7fd396ffd640 1 -- 172.21.15.16:0/1003939556 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd390009820 con 0x7fd3a015cfa0 2023-05-22T15:46:07.078 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.072+0000 7fd3a4b14640 1 -- 172.21.15.16:0/1003939556 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3a00cb610 con 0x7fd3a015cfa0 2023-05-22T15:46:07.078 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.072+0000 7fd396ffd640 1 -- 172.21.15.16:0/1003939556 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd390009bd0 con 0x7fd3a015cfa0 2023-05-22T15:46:07.078 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.072+0000 7fd3a4b14640 1 -- 172.21.15.16:0/1003939556 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3a00cb330 con 0x7fd3a015cfa0 2023-05-22T15:46:07.078 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.072+0000 7fd3a4b14640 1 -- 172.21.15.16:0/1003939556 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3a015d370 con 0x7fd3a015cfa0 2023-05-22T15:46:07.079 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.072+0000 7fd396ffd640 1 -- 172.21.15.16:0/1003939556 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 57) v1 ==== 84446+0+0 (secure 0 0 0) 0x7fd390009d30 con 0x7fd3a015cfa0 2023-05-22T15:46:07.079 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.072+0000 7fd396ffd640 1 --2- 172.21.15.16:0/1003939556 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd3800655c0 0x7fd380067a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:07.079 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.072+0000 7fd39e59c640 1 -- 172.21.15.16:0/1003939556 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd3800655c0 msgr2=0x7fd380067a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/36921 2023-05-22T15:46:07.079 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.072+0000 7fd396ffd640 1 -- 172.21.15.16:0/1003939556 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(133..133 src has 1..133) v4 ==== 6726+0+0 (secure 0 0 0) 0x7fd390083670 con 0x7fd3a015cfa0 2023-05-22T15:46:07.079 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.072+0000 7fd39e59c640 1 --2- 172.21.15.16:0/1003939556 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd3800655c0 0x7fd380067a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-05-22T15:46:07.081 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.076+0000 7fd396ffd640 1 -- 172.21.15.16:0/1003939556 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fd390054730 con 0x7fd3a015cfa0 2023-05-22T15:46:07.143 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:46:07.140+0000 7f4a1c01d180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-05-22T15:46:07.197 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:46:07.196+0000 7f4a1c01d180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-05-22T15:46:07.232 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.228+0000 7fd3a4b14640 1 -- 172.21.15.16:0/1003939556 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fd3a004dbe0 con 0x7fd3a015cfa0 2023-05-22T15:46:07.233 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.228+0000 7fd396ffd640 1 -- 172.21.15.16:0/1003939556 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v133) v1 ==== 93+0+10 (secure 0 0 0) 0x7fd39000b3c0 con 0x7fd3a015cfa0 2023-05-22T15:46:07.233 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2023-05-22T15:46:07.235 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.232+0000 7fd3a4b14640 1 -- 172.21.15.16:0/1003939556 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd3800655c0 msgr2=0x7fd380067a70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-05-22T15:46:07.235 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.232+0000 7fd3a4b14640 1 --2- 172.21.15.16:0/1003939556 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd3800655c0 0x7fd380067a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:07.235 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.232+0000 7fd3a4b14640 1 -- 172.21.15.16:0/1003939556 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd3a015cfa0 msgr2=0x7fd3a015a950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:07.235 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.232+0000 7fd3a4b14640 1 --2- 172.21.15.16:0/1003939556 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd3a015cfa0 0x7fd3a015a950 secure :-1 s=READY pgs=1732 cs=0 l=1 rev1=1 crypto rx=0x7fd39000d800 tx=0x7fd39000dcc0 comp rx=0 tx=0).stop 2023-05-22T15:46:07.235 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.232+0000 7fd3a4b14640 1 -- 172.21.15.16:0/1003939556 shutdown_connections 2023-05-22T15:46:07.236 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.232+0000 7fd3a4b14640 1 --2- 172.21.15.16:0/1003939556 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd3a015cfa0 0x7fd3a015a950 unknown :-1 s=CLOSED pgs=1732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:07.236 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.232+0000 7fd3a4b14640 1 --2- 172.21.15.16:0/1003939556 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd3800655c0 0x7fd380067a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:07.236 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.232+0000 7fd3a4b14640 1 --2- 172.21.15.16:0/1003939556 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd3a00cf870 0x7fd3a0157230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:07.236 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.232+0000 7fd3a4b14640 1 --2- 172.21.15.16:0/1003939556 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd3a006e740 0x7fd3a01f5b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:07.236 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.232+0000 7fd3a4b14640 1 -- 172.21.15.16:0/1003939556 >> 172.21.15.16:0/1003939556 conn(0x7fd3a00cc870 msgr2=0x7fd3a00b14c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:46:07.236 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.232+0000 7fd3a4b14640 1 -- 172.21.15.16:0/1003939556 shutdown_connections 2023-05-22T15:46:07.237 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.232+0000 7fd3a4b14640 1 -- 172.21.15.16:0/1003939556 wait complete. 2023-05-22T15:46:07.251 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-05-22T15:46:07.260 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:46:07.256+0000 7f614e4ed180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-05-22T15:46:07.316 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:46:07.312+0000 7f614e4ed180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-05-22T15:46:07.377 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.372+0000 7fe0164b9640 1 Processor -- start 2023-05-22T15:46:07.377 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.372+0000 7fe0164b9640 1 -- start start 2023-05-22T15:46:07.378 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.372+0000 7fe0164b9640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe010071640 0x7fe010160090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:07.378 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.372+0000 7fe0164b9640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fe0101605d0 0x7fe01015c270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:07.379 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.372+0000 7fe0164b9640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe010069ae0 con 0x7fe0100ce480 2023-05-22T15:46:07.379 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.372+0000 7fe0164b9640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe01015f990 con 0x7fe010071640 2023-05-22T15:46:07.379 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.372+0000 7fe0164b9640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe010071ac0 con 0x7fe0101605d0 2023-05-22T15:46:07.379 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.372+0000 7fe00ffff640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fe0100ce480 0x7fe0100ce850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:32876/0 (socket says 172.21.15.16:32876) 2023-05-22T15:46:07.379 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.372+0000 7fe00ffff640 1 -- 172.21.15.16:0/1139216986 learned_addr learned my addr 172.21.15.16:0/1139216986 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:46:07.379 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.372+0000 7fe00f7fe640 1 --2- 172.21.15.16:0/1139216986 >> v2:172.21.15.16:3300/0 conn(0x7fe010071640 0x7fe010160090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:07.379 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.372+0000 7fe014a56640 1 --2- 172.21.15.16:0/1139216986 >> v2:172.21.15.62:3300/0 conn(0x7fe0101605d0 0x7fe01015c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:07.380 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.372+0000 7fe00f7fe640 1 -- 172.21.15.16:0/1139216986 >> v2:172.21.15.62:3300/0 conn(0x7fe0101605d0 msgr2=0x7fe01015c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:46:07.380 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.372+0000 7fe00f7fe640 1 --2- 172.21.15.16:0/1139216986 >> v2:172.21.15.62:3300/0 conn(0x7fe0101605d0 0x7fe01015c270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:07.380 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.372+0000 7fe00f7fe640 1 -- 172.21.15.16:0/1139216986 >> v1:172.21.15.16:6789/0 conn(0x7fe0100ce480 legacy=0x7fe0100ce850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:07.380 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.372+0000 7fe00effd640 1 -- 172.21.15.16:0/1139216986 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3569200614 0 0) 0x7fe010069ae0 con 0x7fe0100ce480 2023-05-22T15:46:07.380 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.372+0000 7fe00f7fe640 1 -- 172.21.15.16:0/1139216986 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe010069dc0 con 0x7fe010071640 2023-05-22T15:46:07.380 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.372+0000 7fe014a56640 1 --2- 172.21.15.16:0/1139216986 >> v2:172.21.15.62:3300/0 conn(0x7fe0101605d0 0x7fe01015c270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:46:07.381 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.376+0000 7fe00f7fe640 1 --2- 172.21.15.16:0/1139216986 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe010071640 0x7fe010160090 secure :-1 s=READY pgs=1737 cs=0 l=1 rev1=1 crypto rx=0x7fe0100aeb30 tx=0x7fe00401c430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ac0c0c6de5ff8ce server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:07.381 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.376+0000 7fe00effd640 1 -- 172.21.15.16:0/1139216986 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0040a7070 con 0x7fe010071640 2023-05-22T15:46:07.382 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.376+0000 7fe00effd640 1 -- 172.21.15.16:0/1139216986 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0040ab430 con 0x7fe010071640 2023-05-22T15:46:07.382 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.376+0000 7fe00effd640 1 -- 172.21.15.16:0/1139216986 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0040ab7c0 con 0x7fe010071640 2023-05-22T15:46:07.382 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.376+0000 7fe0164b9640 1 -- 172.21.15.16:0/1139216986 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe010071640 msgr2=0x7fe010160090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:07.382 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.376+0000 7fe0164b9640 1 --2- 172.21.15.16:0/1139216986 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe010071640 0x7fe010160090 secure :-1 s=READY pgs=1737 cs=0 l=1 rev1=1 crypto rx=0x7fe0100aeb30 tx=0x7fe00401c430 comp rx=0 tx=0).stop 2023-05-22T15:46:07.382 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.376+0000 7fe0164b9640 1 -- 172.21.15.16:0/1139216986 shutdown_connections 2023-05-22T15:46:07.382 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.376+0000 7fe0164b9640 1 --2- 172.21.15.16:0/1139216986 >> v2:172.21.15.62:3300/0 conn(0x7fe0101605d0 0x7fe01015c270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:07.382 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.376+0000 7fe0164b9640 1 --2- 172.21.15.16:0/1139216986 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe010071640 0x7fe010160090 unknown :-1 s=CLOSED pgs=1737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:07.383 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.376+0000 7fe0164b9640 1 -- 172.21.15.16:0/1139216986 >> 172.21.15.16:0/1139216986 conn(0x7fe0100cc870 msgr2=0x7fe0100cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:46:07.383 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.376+0000 7fe0164b9640 1 -- 172.21.15.16:0/1139216986 shutdown_connections 2023-05-22T15:46:07.383 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.376+0000 7fe0164b9640 1 -- 172.21.15.16:0/1139216986 wait complete. 2023-05-22T15:46:07.384 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.380+0000 7fe0164b9640 1 Processor -- start 2023-05-22T15:46:07.385 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.380+0000 7fe0164b9640 1 -- start start 2023-05-22T15:46:07.385 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.380+0000 7fe0164b9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe010071640 0x7fe01015a640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:07.385 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.380+0000 7fe0164b9640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe0100ce480 0x7fe010155110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:07.385 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.380+0000 7fe0164b9640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe0101605d0 0x7fe0100ac070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:07.385 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.380+0000 7fe0164b9640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0100b0080 con 0x7fe010071640 2023-05-22T15:46:07.385 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.380+0000 7fe0164b9640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0100b0360 con 0x7fe0100ce480 2023-05-22T15:46:07.385 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.380+0000 7fe0164b9640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0100ab870 con 0x7fe0101605d0 2023-05-22T15:46:07.386 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.380+0000 7fe00ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe010071640 0x7fe01015a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:07.386 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.380+0000 7fe00ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe010071640 0x7fe01015a640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59528/0 (socket says 172.21.15.16:59528) 2023-05-22T15:46:07.386 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.380+0000 7fe00ffff640 1 -- 172.21.15.16:0/478787339 learned_addr learned my addr 172.21.15.16:0/478787339 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:46:07.386 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.380+0000 7fe00f7fe640 1 --2- 172.21.15.16:0/478787339 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe0100ce480 0x7fe010155110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:07.386 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.380+0000 7fe014a56640 1 --2- 172.21.15.16:0/478787339 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe0101605d0 0x7fe0100ac070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:07.386 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.380+0000 7fe00ffff640 1 -- 172.21.15.16:0/478787339 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe0101605d0 msgr2=0x7fe0100ac070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:07.386 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.380+0000 7fe00ffff640 1 --2- 172.21.15.16:0/478787339 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe0101605d0 0x7fe0100ac070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:07.386 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.380+0000 7fe00ffff640 1 -- 172.21.15.16:0/478787339 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe0100ce480 msgr2=0x7fe010155110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:07.387 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.380+0000 7fe00ffff640 1 --2- 172.21.15.16:0/478787339 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe0100ce480 0x7fe010155110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:07.387 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.380+0000 7fe00ffff640 1 -- 172.21.15.16:0/478787339 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe004009420 con 0x7fe010071640 2023-05-22T15:46:07.387 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.380+0000 7fe00ffff640 1 --2- 172.21.15.16:0/478787339 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe010071640 0x7fe01015a640 secure :-1 s=READY pgs=1738 cs=0 l=1 rev1=1 crypto rx=0x7fe000001570 tx=0x7fe00000cd20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:07.387 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.380+0000 7fe00cff9640 1 -- 172.21.15.16:0/478787339 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe00000a640 con 0x7fe010071640 2023-05-22T15:46:07.387 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.380+0000 7fe00cff9640 1 -- 172.21.15.16:0/478787339 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe00000a7c0 con 0x7fe010071640 2023-05-22T15:46:07.387 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.380+0000 7fe0164b9640 1 -- 172.21.15.16:0/478787339 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0100698b0 con 0x7fe010071640 2023-05-22T15:46:07.387 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.380+0000 7fe00cff9640 1 -- 172.21.15.16:0/478787339 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe00000ab70 con 0x7fe010071640 2023-05-22T15:46:07.387 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.380+0000 7fe0164b9640 1 -- 172.21.15.16:0/478787339 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe010069720 con 0x7fe010071640 2023-05-22T15:46:07.388 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.380+0000 7fe0164b9640 1 -- 172.21.15.16:0/478787339 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0100c5e70 con 0x7fe010071640 2023-05-22T15:46:07.388 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.380+0000 7fe00cff9640 1 -- 172.21.15.16:0/478787339 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 58) v1 ==== 83941+0+0 (secure 0 0 0) 0x7fe00000acd0 con 0x7fe010071640 2023-05-22T15:46:07.388 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.380+0000 7fe00cff9640 1 -- 172.21.15.16:0/478787339 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 7070+0+0 (secure 0 0 0) 0x7fe000081fc0 con 0x7fe010071640 2023-05-22T15:46:07.390 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.384+0000 7fe00cff9640 1 -- 172.21.15.16:0/478787339 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fe00004fca0 con 0x7fe010071640 2023-05-22T15:46:07.428 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:46:07.424+0000 7f60da77d640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T15:46:07.429 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:46:07.424+0000 7f60da77d640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T15:46:07.429 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:46:07.424+0000 7f60d7777640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T15:46:07.429 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:46:07.424+0000 7f60d7777640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T15:46:07.429 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:46:07.424+0000 7f60d7777640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T15:46:07.429 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:46:07.424+0000 7f60d7777640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T15:46:07.430 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:46:07.424+0000 7f60d7777640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T15:46:07.430 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:46:07.424+0000 7f60da77d640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T15:46:07.430 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:46:07.424+0000 7f60da77d640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T15:46:07.430 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:46:07.424+0000 7f60da77d640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T15:46:07.555 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.552+0000 7fe0164b9640 1 -- 172.21.15.16:0/478787339 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fe0100abd90 con 0x7fe010071640 2023-05-22T15:46:07.556 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.552+0000 7fe00cff9640 1 -- 172.21.15.16:0/478787339 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v134) v1 ==== 92+0+10 (secure 0 0 0) 0x7fe0100abd90 con 0x7fe010071640 2023-05-22T15:46:07.556 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 8 2023-05-22T15:46:07.558 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.552+0000 7fe0164b9640 1 -- 172.21.15.16:0/478787339 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe010071640 msgr2=0x7fe01015a640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:07.558 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.552+0000 7fe0164b9640 1 --2- 172.21.15.16:0/478787339 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe010071640 0x7fe01015a640 secure :-1 s=READY pgs=1738 cs=0 l=1 rev1=1 crypto rx=0x7fe000001570 tx=0x7fe00000cd20 comp rx=0 tx=0).stop 2023-05-22T15:46:07.558 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.552+0000 7fe0164b9640 1 -- 172.21.15.16:0/478787339 shutdown_connections 2023-05-22T15:46:07.558 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.552+0000 7fe0164b9640 1 --2- 172.21.15.16:0/478787339 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe0101605d0 0x7fe0100ac070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:07.558 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.552+0000 7fe0164b9640 1 --2- 172.21.15.16:0/478787339 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe0100ce480 0x7fe010155110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:07.559 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.552+0000 7fe0164b9640 1 --2- 172.21.15.16:0/478787339 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe010071640 0x7fe01015a640 unknown :-1 s=CLOSED pgs=1738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:07.559 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.552+0000 7fe0164b9640 1 -- 172.21.15.16:0/478787339 >> 172.21.15.16:0/478787339 conn(0x7fe0100cc870 msgr2=0x7fe01006e510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:46:07.559 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.552+0000 7fe0164b9640 1 -- 172.21.15.16:0/478787339 shutdown_connections 2023-05-22T15:46:07.559 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.552+0000 7fe0164b9640 1 -- 172.21.15.16:0/478787339 wait complete. 2023-05-22T15:46:07.573 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-05-22T15:46:07.740 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.736+0000 7f2c0ce56640 1 Processor -- start 2023-05-22T15:46:07.740 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.736+0000 7f2c0ce56640 1 -- start start 2023-05-22T15:46:07.741 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.736+0000 7f2c0ce56640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2c0815bee0 0x7f2c0806b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:07.741 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.736+0000 7f2c0ce56640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2c080cb730 0x7f2c0806bc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:07.742 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.736+0000 7f2c0ce56640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c080c8660 con 0x7f2c080cbbe0 2023-05-22T15:46:07.742 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.736+0000 7f2c0ce56640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c080ce4a0 con 0x7f2c0815bee0 2023-05-22T15:46:07.742 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.736+0000 7f2c0ce56640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c080ce780 con 0x7f2c080cb730 2023-05-22T15:46:07.742 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.736+0000 7f2c0659c640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2c0815bee0 0x7f2c0806b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:07.742 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.736+0000 7f2c06d9d640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f2c080cbbe0 0x7f2c0815f850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:32890/0 (socket says 172.21.15.16:32890) 2023-05-22T15:46:07.742 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.736+0000 7f2c06d9d640 1 -- 172.21.15.16:0/317036671 learned_addr learned my addr 172.21.15.16:0/317036671 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:46:07.742 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.736+0000 7f2c0659c640 1 -- 172.21.15.16:0/317036671 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2c080cb730 msgr2=0x7f2c0806bc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:46:07.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.736+0000 7f2c05d9b640 1 --2- 172.21.15.16:0/317036671 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2c080cb730 0x7f2c0806bc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:07.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.736+0000 7f2c0559a640 1 -- 172.21.15.16:0/317036671 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3326281541 0 0) 0x7f2c080c8660 con 0x7f2c080cbbe0 2023-05-22T15:46:07.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.736+0000 7f2c0659c640 1 --2- 172.21.15.16:0/317036671 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2c080cb730 0x7f2c0806bc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:07.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.736+0000 7f2c0659c640 1 -- 172.21.15.16:0/317036671 >> v1:172.21.15.16:6789/0 conn(0x7f2c080cbbe0 legacy=0x7f2c0815f850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:07.744 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.736+0000 7f2c0659c640 1 -- 172.21.15.16:0/317036671 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c080cea60 con 0x7f2c0815bee0 2023-05-22T15:46:07.744 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.736+0000 7f2c05d9b640 1 --2- 172.21.15.16:0/317036671 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2c080cb730 0x7f2c0806bc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:46:07.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.740+0000 7f2c0659c640 1 --2- 172.21.15.16:0/317036671 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2c0815bee0 0x7f2c0806b6c0 secure :-1 s=READY pgs=1748 cs=0 l=1 rev1=1 crypto rx=0x7f2bf8009d90 tx=0x7f2bf801c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97b341de94884cf7 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:07.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.740+0000 7f2c0559a640 1 -- 172.21.15.16:0/317036671 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bf80a5a00 con 0x7f2c0815bee0 2023-05-22T15:46:07.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.740+0000 7f2c0559a640 1 -- 172.21.15.16:0/317036671 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2bf80a5b60 con 0x7f2c0815bee0 2023-05-22T15:46:07.746 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.740+0000 7f2c0559a640 1 -- 172.21.15.16:0/317036671 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bf80ad440 con 0x7f2c0815bee0 2023-05-22T15:46:07.746 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.740+0000 7f2c0ce56640 1 -- 172.21.15.16:0/317036671 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2c0815bee0 msgr2=0x7f2c0806b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:07.746 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.740+0000 7f2c0ce56640 1 --2- 172.21.15.16:0/317036671 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2c0815bee0 0x7f2c0806b6c0 secure :-1 s=READY pgs=1748 cs=0 l=1 rev1=1 crypto rx=0x7f2bf8009d90 tx=0x7f2bf801c920 comp rx=0 tx=0).stop 2023-05-22T15:46:07.746 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.740+0000 7f2c0ce56640 1 -- 172.21.15.16:0/317036671 shutdown_connections 2023-05-22T15:46:07.746 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.740+0000 7f2c0ce56640 1 --2- 172.21.15.16:0/317036671 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2c080cb730 0x7f2c0806bc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:07.746 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.740+0000 7f2c0ce56640 1 --2- 172.21.15.16:0/317036671 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2c0815bee0 0x7f2c0806b6c0 unknown :-1 s=CLOSED pgs=1748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:07.746 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.740+0000 7f2c0ce56640 1 -- 172.21.15.16:0/317036671 >> 172.21.15.16:0/317036671 conn(0x7f2c08069720 msgr2=0x7f2c08069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:46:07.747 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.740+0000 7f2c0ce56640 1 -- 172.21.15.16:0/317036671 shutdown_connections 2023-05-22T15:46:07.747 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.740+0000 7f2c0ce56640 1 -- 172.21.15.16:0/317036671 wait complete. 2023-05-22T15:46:07.748 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.740+0000 7f2c0ce56640 1 Processor -- start 2023-05-22T15:46:07.749 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.740+0000 7f2c0ce56640 1 -- start start 2023-05-22T15:46:07.749 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.744+0000 7f2c0ce56640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2c080cb730 0x7f2c0814c350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:07.749 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.744+0000 7f2c0ce56640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2c080cbbe0 0x7f2c0814dc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:07.749 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.744+0000 7f2c0ce56640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2c0815bee0 0x7f2c0815e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:07.749 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.744+0000 7f2c0ce56640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c0814d860 con 0x7f2c080cb730 2023-05-22T15:46:07.749 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.744+0000 7f2c0ce56640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c0814c890 con 0x7f2c080cbbe0 2023-05-22T15:46:07.750 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.744+0000 7f2c0ce56640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c0815c590 con 0x7f2c0815bee0 2023-05-22T15:46:07.750 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.744+0000 7f2c0659c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2c080cb730 0x7f2c0814c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:07.750 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.744+0000 7f2c0659c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2c080cb730 0x7f2c0814c350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59644/0 (socket says 172.21.15.16:59644) 2023-05-22T15:46:07.750 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.744+0000 7f2c0659c640 1 -- 172.21.15.16:0/2920838116 learned_addr learned my addr 172.21.15.16:0/2920838116 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:46:07.750 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.744+0000 7f2c06d9d640 1 --2- 172.21.15.16:0/2920838116 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2c0815bee0 0x7f2c0815e6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:07.750 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.744+0000 7f2c0659c640 1 -- 172.21.15.16:0/2920838116 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2c0815bee0 msgr2=0x7f2c0815e6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:07.750 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.744+0000 7f2c05d9b640 1 --2- 172.21.15.16:0/2920838116 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2c080cbbe0 0x7f2c0814dc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:07.751 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.744+0000 7f2c0659c640 1 --2- 172.21.15.16:0/2920838116 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2c0815bee0 0x7f2c0815e6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:07.751 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.744+0000 7f2c0659c640 1 -- 172.21.15.16:0/2920838116 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2c080cbbe0 msgr2=0x7f2c0814dc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:07.751 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.744+0000 7f2c0659c640 1 --2- 172.21.15.16:0/2920838116 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2c080cbbe0 0x7f2c0814dc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:07.751 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.744+0000 7f2c0659c640 1 -- 172.21.15.16:0/2920838116 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bf8009a70 con 0x7f2c080cb730 2023-05-22T15:46:07.751 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.744+0000 7f2c0659c640 1 --2- 172.21.15.16:0/2920838116 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2c080cb730 0x7f2c0814c350 secure :-1 s=READY pgs=1749 cs=0 l=1 rev1=1 crypto rx=0x7f2bf8009d90 tx=0x7f2bf80196c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:07.751 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.744+0000 7f2bf6ffd640 1 -- 172.21.15.16:0/2920838116 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bf80aee90 con 0x7f2c080cb730 2023-05-22T15:46:07.751 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.744+0000 7f2bf6ffd640 1 -- 172.21.15.16:0/2920838116 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2bf80039a0 con 0x7f2c080cb730 2023-05-22T15:46:07.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.744+0000 7f2c0ce56640 1 -- 172.21.15.16:0/2920838116 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c08168d40 con 0x7f2c080cb730 2023-05-22T15:46:07.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.744+0000 7f2bf6ffd640 1 -- 172.21.15.16:0/2920838116 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bf8003d50 con 0x7f2c080cb730 2023-05-22T15:46:07.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.744+0000 7f2c0ce56640 1 -- 172.21.15.16:0/2920838116 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c08168bb0 con 0x7f2c080cb730 2023-05-22T15:46:07.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.744+0000 7f2c0ce56640 1 -- 172.21.15.16:0/2920838116 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c08168ed0 con 0x7f2c080cb730 2023-05-22T15:46:07.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.744+0000 7f2bf6ffd640 1 -- 172.21.15.16:0/2920838116 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 58) v1 ==== 83941+0+0 (secure 0 0 0) 0x7f2bf80b4050 con 0x7f2c080cb730 2023-05-22T15:46:07.753 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.744+0000 7f2bf6ffd640 1 -- 172.21.15.16:0/2920838116 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 7070+0+0 (secure 0 0 0) 0x7f2bf811dbf0 con 0x7f2c080cb730 2023-05-22T15:46:07.753 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.748+0000 7f2bf6ffd640 1 -- 172.21.15.16:0/2920838116 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f2bf80eb970 con 0x7f2c080cb730 2023-05-22T15:46:07.905 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.900+0000 7f2c0ce56640 1 -- 172.21.15.16:0/2920838116 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f2c0814d9a0 con 0x7f2c080cb730 2023-05-22T15:46:07.906 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.900+0000 7f2bf6ffd640 1 -- 172.21.15.16:0/2920838116 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v134) v1 ==== 104+0+11 (secure 0 0 0) 0x7f2bf80eef90 con 0x7f2c080cb730 2023-05-22T15:46:07.906 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2023-05-22T15:46:07.908 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.904+0000 7f2c0ce56640 1 -- 172.21.15.16:0/2920838116 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2c080cb730 msgr2=0x7f2c0814c350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:07.908 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.904+0000 7f2c0ce56640 1 --2- 172.21.15.16:0/2920838116 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2c080cb730 0x7f2c0814c350 secure :-1 s=READY pgs=1749 cs=0 l=1 rev1=1 crypto rx=0x7f2bf8009d90 tx=0x7f2bf80196c0 comp rx=0 tx=0).stop 2023-05-22T15:46:07.908 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.904+0000 7f2c0ce56640 1 -- 172.21.15.16:0/2920838116 shutdown_connections 2023-05-22T15:46:07.908 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.904+0000 7f2c0ce56640 1 --2- 172.21.15.16:0/2920838116 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2c0815bee0 0x7f2c0815e6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:07.908 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.904+0000 7f2c0ce56640 1 --2- 172.21.15.16:0/2920838116 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2c080cbbe0 0x7f2c0814dc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:07.908 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.904+0000 7f2c0ce56640 1 --2- 172.21.15.16:0/2920838116 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2c080cb730 0x7f2c0814c350 unknown :-1 s=CLOSED pgs=1749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:07.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.904+0000 7f2c0ce56640 1 -- 172.21.15.16:0/2920838116 >> 172.21.15.16:0/2920838116 conn(0x7f2c08069720 msgr2=0x7f2c080b0ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:46:07.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.904+0000 7f2c0ce56640 1 -- 172.21.15.16:0/2920838116 shutdown_connections 2023-05-22T15:46:07.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:07.904+0000 7f2c0ce56640 1 -- 172.21.15.16:0/2920838116 wait complete. 2023-05-22T15:46:07.925 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-05-22T15:46:08.051 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.048+0000 7fdeab631640 1 Processor -- start 2023-05-22T15:46:08.051 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.048+0000 7fdeab631640 1 -- start start 2023-05-22T15:46:08.052 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.048+0000 7fdeab631640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fdea415bca0 0x7fdea4160160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:08.053 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.048+0000 7fdeab631640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fdea40b3ee0 0x7fdea40b4330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:08.053 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.048+0000 7fdeab631640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdea40d0400 con 0x7fdea40b4870 2023-05-22T15:46:08.053 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.048+0000 7fdeab631640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdea415c070 con 0x7fdea40b3ee0 2023-05-22T15:46:08.053 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.048+0000 7fdeab631640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdea41606a0 con 0x7fdea415bca0 2023-05-22T15:46:08.053 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.048+0000 7fdea8bcc640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fdea40b3ee0 0x7fdea40b4330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:08.054 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.048+0000 7fdea8bcc640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fdea40b3ee0 0x7fdea40b4330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59650/0 (socket says 172.21.15.16:59650) 2023-05-22T15:46:08.054 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.048+0000 7fdea8bcc640 1 -- 172.21.15.16:0/3240879734 learned_addr learned my addr 172.21.15.16:0/3240879734 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:46:08.054 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.048+0000 7fdea93cd640 1 --2- 172.21.15.16:0/3240879734 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fdea415bca0 0x7fdea4160160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:08.054 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.048+0000 7fdea8bcc640 1 -- 172.21.15.16:0/3240879734 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fdea415bca0 msgr2=0x7fdea4160160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:46:08.055 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.048+0000 7fdea8bcc640 1 --2- 172.21.15.16:0/3240879734 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fdea415bca0 0x7fdea4160160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:08.055 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.048+0000 7fdea8bcc640 1 -- 172.21.15.16:0/3240879734 >> v1:172.21.15.62:6789/0 conn(0x7fdea40b4870 legacy=0x7fdea40ce900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:46:08.055 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.048+0000 7fdea8bcc640 1 -- 172.21.15.16:0/3240879734 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdea40d00d0 con 0x7fdea40b3ee0 2023-05-22T15:46:08.057 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.052+0000 7fdea8bcc640 1 --2- 172.21.15.16:0/3240879734 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdea40b3ee0 0x7fdea40b4330 secure :-1 s=READY pgs=1750 cs=0 l=1 rev1=1 crypto rx=0x7fde94009870 tx=0x7fde9401c440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aab0b9b8c621d51b server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:08.057 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.052+0000 7fde9bfff640 1 -- 172.21.15.16:0/3240879734 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde940ab600 con 0x7fdea40b3ee0 2023-05-22T15:46:08.057 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.052+0000 7fde9bfff640 1 -- 172.21.15.16:0/3240879734 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde940ab760 con 0x7fdea40b3ee0 2023-05-22T15:46:08.057 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.052+0000 7fde9bfff640 1 -- 172.21.15.16:0/3240879734 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde940abaf0 con 0x7fdea40b3ee0 2023-05-22T15:46:08.057 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.052+0000 7fdeab631640 1 -- 172.21.15.16:0/3240879734 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdea40b3ee0 msgr2=0x7fdea40b4330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:08.058 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.052+0000 7fdeab631640 1 --2- 172.21.15.16:0/3240879734 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdea40b3ee0 0x7fdea40b4330 secure :-1 s=READY pgs=1750 cs=0 l=1 rev1=1 crypto rx=0x7fde94009870 tx=0x7fde9401c440 comp rx=0 tx=0).stop 2023-05-22T15:46:08.058 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.052+0000 7fdeab631640 1 -- 172.21.15.16:0/3240879734 shutdown_connections 2023-05-22T15:46:08.058 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.052+0000 7fdeab631640 1 --2- 172.21.15.16:0/3240879734 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdea40b3ee0 0x7fdea40b4330 unknown :-1 s=CLOSED pgs=1750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:08.058 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.052+0000 7fdeab631640 1 --2- 172.21.15.16:0/3240879734 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fdea415bca0 0x7fdea4160160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:08.058 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.052+0000 7fdeab631640 1 -- 172.21.15.16:0/3240879734 >> 172.21.15.16:0/3240879734 conn(0x7fdea40657f0 msgr2=0x7fdea4070e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:46:08.058 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.052+0000 7fdeab631640 1 -- 172.21.15.16:0/3240879734 shutdown_connections 2023-05-22T15:46:08.058 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.052+0000 7fdeab631640 1 -- 172.21.15.16:0/3240879734 wait complete. 2023-05-22T15:46:08.059 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.052+0000 7fdeab631640 1 Processor -- start 2023-05-22T15:46:08.059 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.052+0000 7fdeab631640 1 -- start start 2023-05-22T15:46:08.060 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.052+0000 7fdeab631640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdea40b3ee0 0x7fdea40c9530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:08.061 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.052+0000 7fdeab631640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fdea40b4870 0x7fdea40c9b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:08.061 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.052+0000 7fdeab631640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdea415bca0 0x7fdea41519e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:08.061 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.052+0000 7fdeab631640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdea40cae10 con 0x7fdea415bca0 2023-05-22T15:46:08.061 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.052+0000 7fdeab631640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdea415ae90 con 0x7fdea40b3ee0 2023-05-22T15:46:08.061 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.052+0000 7fdeab631640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdea4151f20 con 0x7fdea40b4870 2023-05-22T15:46:08.061 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.052+0000 7fdea9bce640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdea415bca0 0x7fdea41519e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:08.061 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.052+0000 7fdea9bce640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdea415bca0 0x7fdea41519e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59666/0 (socket says 172.21.15.16:59666) 2023-05-22T15:46:08.061 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.052+0000 7fdea9bce640 1 -- 172.21.15.16:0/2872490490 learned_addr learned my addr 172.21.15.16:0/2872490490 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:46:08.062 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.052+0000 7fdea93cd640 1 --2- 172.21.15.16:0/2872490490 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdea40b3ee0 0x7fdea40c9530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:08.062 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.052+0000 7fdea8bcc640 1 --2- 172.21.15.16:0/2872490490 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fdea40b4870 0x7fdea40c9b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:08.062 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.052+0000 7fdea9bce640 1 -- 172.21.15.16:0/2872490490 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fdea40b4870 msgr2=0x7fdea40c9b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:08.062 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.052+0000 7fdea9bce640 1 --2- 172.21.15.16:0/2872490490 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fdea40b4870 0x7fdea40c9b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:08.062 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.052+0000 7fdea9bce640 1 -- 172.21.15.16:0/2872490490 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdea40b3ee0 msgr2=0x7fdea40c9530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:08.062 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.052+0000 7fdea9bce640 1 --2- 172.21.15.16:0/2872490490 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdea40b3ee0 0x7fdea40c9530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:08.062 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.052+0000 7fdea9bce640 1 -- 172.21.15.16:0/2872490490 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde940094d0 con 0x7fdea415bca0 2023-05-22T15:46:08.063 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.052+0000 7fdea9bce640 1 --2- 172.21.15.16:0/2872490490 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdea415bca0 0x7fdea41519e0 secure :-1 s=READY pgs=1751 cs=0 l=1 rev1=1 crypto rx=0x7fde9c00c7b0 tx=0x7fde9c00cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:08.063 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.052+0000 7fde99ffb640 1 -- 172.21.15.16:0/2872490490 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde9c004b70 con 0x7fdea415bca0 2023-05-22T15:46:08.063 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.052+0000 7fde99ffb640 1 -- 172.21.15.16:0/2872490490 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde9c004cd0 con 0x7fdea415bca0 2023-05-22T15:46:08.063 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.052+0000 7fdeab631640 1 -- 172.21.15.16:0/2872490490 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdea40cd1a0 con 0x7fdea415bca0 2023-05-22T15:46:08.063 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.052+0000 7fde99ffb640 1 -- 172.21.15.16:0/2872490490 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde9c00aa90 con 0x7fdea415bca0 2023-05-22T15:46:08.063 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.052+0000 7fdeab631640 1 -- 172.21.15.16:0/2872490490 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdea40b4cb0 con 0x7fdea415bca0 2023-05-22T15:46:08.063 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.056+0000 7fdeab631640 1 -- 172.21.15.16:0/2872490490 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdea4168af0 con 0x7fdea415bca0 2023-05-22T15:46:08.063 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.056+0000 7fde99ffb640 1 -- 172.21.15.16:0/2872490490 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 58) v1 ==== 83941+0+0 (secure 0 0 0) 0x7fde9c00abf0 con 0x7fdea415bca0 2023-05-22T15:46:08.064 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.056+0000 7fde99ffb640 1 -- 172.21.15.16:0/2872490490 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 7070+0+0 (secure 0 0 0) 0x7fde9c0829a0 con 0x7fdea415bca0 2023-05-22T15:46:08.064 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.060+0000 7fde99ffb640 1 -- 172.21.15.16:0/2872490490 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fde9c050590 con 0x7fdea415bca0 2023-05-22T15:46:08.209 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.204+0000 7fdeab631640 1 -- 172.21.15.16:0/2872490490 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fdea40caf50 con 0x7fdea415bca0 2023-05-22T15:46:08.210 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.204+0000 7fde99ffb640 1 -- 172.21.15.16:0/2872490490 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v134) v1 ==== 100+0+11 (secure 0 0 0) 0x7fde9c053bb0 con 0x7fdea415bca0 2023-05-22T15:46:08.210 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 64 2023-05-22T15:46:08.211 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.208+0000 7fdeab631640 1 -- 172.21.15.16:0/2872490490 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdea415bca0 msgr2=0x7fdea41519e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:08.212 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.208+0000 7fdeab631640 1 --2- 172.21.15.16:0/2872490490 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdea415bca0 0x7fdea41519e0 secure :-1 s=READY pgs=1751 cs=0 l=1 rev1=1 crypto rx=0x7fde9c00c7b0 tx=0x7fde9c00cc70 comp rx=0 tx=0).stop 2023-05-22T15:46:08.212 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.208+0000 7fdeab631640 1 -- 172.21.15.16:0/2872490490 shutdown_connections 2023-05-22T15:46:08.212 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.208+0000 7fdeab631640 1 --2- 172.21.15.16:0/2872490490 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdea415bca0 0x7fdea41519e0 unknown :-1 s=CLOSED pgs=1751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:08.212 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.208+0000 7fdeab631640 1 --2- 172.21.15.16:0/2872490490 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fdea40b4870 0x7fdea40c9b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:08.212 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.208+0000 7fdeab631640 1 --2- 172.21.15.16:0/2872490490 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdea40b3ee0 0x7fdea40c9530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:08.212 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.208+0000 7fdeab631640 1 -- 172.21.15.16:0/2872490490 >> 172.21.15.16:0/2872490490 conn(0x7fdea40657f0 msgr2=0x7fdea40b6ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:46:08.212 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.208+0000 7fdeab631640 1 -- 172.21.15.16:0/2872490490 shutdown_connections 2023-05-22T15:46:08.213 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.208+0000 7fdeab631640 1 -- 172.21.15.16:0/2872490490 wait complete. 2023-05-22T15:46:08.227 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-05-22T15:46:08.355 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.352+0000 7ff69dac8640 1 Processor -- start 2023-05-22T15:46:08.356 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.352+0000 7ff69dac8640 1 -- start start 2023-05-22T15:46:08.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.352+0000 7ff69dac8640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff698070760 0x7ff698070be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:08.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.352+0000 7ff69dac8640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff69806c6e0 0x7ff69806cb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:08.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.352+0000 7ff69dac8640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff69806eb90 con 0x7ff698158a10 2023-05-22T15:46:08.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.352+0000 7ff69dac8640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff69806e060 con 0x7ff698070760 2023-05-22T15:46:08.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.352+0000 7ff69dac8640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff69806d0c0 con 0x7ff69806c6e0 2023-05-22T15:46:08.358 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.352+0000 7ff6977fe640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7ff698158a10 0x7ff69806e480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:32892/0 (socket says 172.21.15.16:32892) 2023-05-22T15:46:08.358 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.352+0000 7ff6977fe640 1 -- 172.21.15.16:0/2738269327 learned_addr learned my addr 172.21.15.16:0/2738269327 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:46:08.358 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.356+0000 7ff696ffd640 1 --2- 172.21.15.16:0/2738269327 >> v2:172.21.15.16:3300/0 conn(0x7ff698070760 0x7ff698070be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:08.363 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.356+0000 7ff696ffd640 1 -- 172.21.15.16:0/2738269327 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff69806c6e0 msgr2=0x7ff69806cb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:46:08.363 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.356+0000 7ff696ffd640 1 --2- 172.21.15.16:0/2738269327 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff69806c6e0 0x7ff69806cb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:08.363 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.356+0000 7ff696ffd640 1 -- 172.21.15.16:0/2738269327 >> v1:172.21.15.16:6789/0 conn(0x7ff698158a10 legacy=0x7ff69806e480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:08.363 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.356+0000 7ff696ffd640 1 -- 172.21.15.16:0/2738269327 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff698158de0 con 0x7ff698070760 2023-05-22T15:46:08.364 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.360+0000 7ff696ffd640 1 --2- 172.21.15.16:0/2738269327 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff698070760 0x7ff698070be0 secure :-1 s=READY pgs=1753 cs=0 l=1 rev1=1 crypto rx=0x7ff690009d90 tx=0x7ff69001c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ca5a9256d737127 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:08.364 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.360+0000 7ff6967fc640 1 -- 172.21.15.16:0/2738269327 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6900a5980 con 0x7ff698070760 2023-05-22T15:46:08.364 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.360+0000 7ff6967fc640 1 -- 172.21.15.16:0/2738269327 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6900a5ae0 con 0x7ff698070760 2023-05-22T15:46:08.365 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.360+0000 7ff6967fc640 1 -- 172.21.15.16:0/2738269327 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6900a5e70 con 0x7ff698070760 2023-05-22T15:46:08.365 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.360+0000 7ff69dac8640 1 -- 172.21.15.16:0/2738269327 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff698070760 msgr2=0x7ff698070be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:08.366 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.360+0000 7ff69dac8640 1 --2- 172.21.15.16:0/2738269327 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff698070760 0x7ff698070be0 secure :-1 s=READY pgs=1753 cs=0 l=1 rev1=1 crypto rx=0x7ff690009d90 tx=0x7ff69001c8c0 comp rx=0 tx=0).stop 2023-05-22T15:46:08.366 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.360+0000 7ff69dac8640 1 -- 172.21.15.16:0/2738269327 shutdown_connections 2023-05-22T15:46:08.366 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.360+0000 7ff69dac8640 1 --2- 172.21.15.16:0/2738269327 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff69806c6e0 0x7ff69806cb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:08.366 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.360+0000 7ff69dac8640 1 --2- 172.21.15.16:0/2738269327 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff698070760 0x7ff698070be0 unknown :-1 s=CLOSED pgs=1753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:08.366 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.360+0000 7ff69dac8640 1 -- 172.21.15.16:0/2738269327 >> 172.21.15.16:0/2738269327 conn(0x7ff6980cc870 msgr2=0x7ff6980cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:46:08.366 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.360+0000 7ff69dac8640 1 -- 172.21.15.16:0/2738269327 shutdown_connections 2023-05-22T15:46:08.366 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.360+0000 7ff69dac8640 1 -- 172.21.15.16:0/2738269327 wait complete. 2023-05-22T15:46:08.367 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.360+0000 7ff69dac8640 1 Processor -- start 2023-05-22T15:46:08.367 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.360+0000 7ff69dac8640 1 -- start start 2023-05-22T15:46:08.367 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.364+0000 7ff69dac8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff69806c6e0 0x7ff69814e1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:08.368 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.364+0000 7ff69dac8640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff698070760 0x7ff69814f210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:08.368 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.364+0000 7ff69dac8640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff698158a10 0x7ff698159d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:08.368 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.364+0000 7ff69dac8640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff69806a510 con 0x7ff69806c6e0 2023-05-22T15:46:08.369 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.364+0000 7ff69dac8640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff698069c70 con 0x7ff698070760 2023-05-22T15:46:08.369 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.364+0000 7ff69dac8640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff698069f50 con 0x7ff698158a10 2023-05-22T15:46:08.370 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.364+0000 7ff697fff640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff698158a10 0x7ff698159d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:08.370 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.364+0000 7ff696ffd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff698070760 0x7ff69814f210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:08.370 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.364+0000 7ff697fff640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff698158a10 0x7ff698159d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3301/0 says I am v2:172.21.15.16:34956/0 (socket says 172.21.15.16:34956) 2023-05-22T15:46:08.370 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.364+0000 7ff697fff640 1 -- 172.21.15.16:0/3855537482 learned_addr learned my addr 172.21.15.16:0/3855537482 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:46:08.370 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.364+0000 7ff6977fe640 1 --2- 172.21.15.16:0/3855537482 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff69806c6e0 0x7ff69814e1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:08.370 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.364+0000 7ff6977fe640 1 -- 172.21.15.16:0/3855537482 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff698158a10 msgr2=0x7ff698159d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:08.370 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.364+0000 7ff6977fe640 1 --2- 172.21.15.16:0/3855537482 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff698158a10 0x7ff698159d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:08.371 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.364+0000 7ff6977fe640 1 -- 172.21.15.16:0/3855537482 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff698070760 msgr2=0x7ff69814f210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:08.371 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.364+0000 7ff6977fe640 1 --2- 172.21.15.16:0/3855537482 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff698070760 0x7ff69814f210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:08.371 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.364+0000 7ff6977fe640 1 -- 172.21.15.16:0/3855537482 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff690009a70 con 0x7ff69806c6e0 2023-05-22T15:46:08.371 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.364+0000 7ff6977fe640 1 --2- 172.21.15.16:0/3855537482 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff69806c6e0 0x7ff69814e1c0 secure :-1 s=READY pgs=1754 cs=0 l=1 rev1=1 crypto rx=0x7ff68c00c340 tx=0x7ff68c00c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:08.372 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.364+0000 7ff67ffff640 1 -- 172.21.15.16:0/3855537482 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff68c005cf0 con 0x7ff69806c6e0 2023-05-22T15:46:08.372 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.364+0000 7ff67ffff640 1 -- 172.21.15.16:0/3855537482 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff68c005e50 con 0x7ff69806c6e0 2023-05-22T15:46:08.372 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.364+0000 7ff67ffff640 1 -- 172.21.15.16:0/3855537482 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff68c0072d0 con 0x7ff69806c6e0 2023-05-22T15:46:08.372 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.368+0000 7ff69dac8640 1 -- 172.21.15.16:0/3855537482 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6980cfe40 con 0x7ff69806c6e0 2023-05-22T15:46:08.373 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.368+0000 7ff69dac8640 1 -- 172.21.15.16:0/3855537482 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff69806dc80 con 0x7ff69806c6e0 2023-05-22T15:46:08.373 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.368+0000 7ff69dac8640 1 -- 172.21.15.16:0/3855537482 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6980ca280 con 0x7ff69806c6e0 2023-05-22T15:46:08.373 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.368+0000 7ff67ffff640 1 -- 172.21.15.16:0/3855537482 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 59) v1 ==== 84068+0+0 (secure 0 0 0) 0x7ff68c007430 con 0x7ff69806c6e0 2023-05-22T15:46:08.373 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.368+0000 7ff67ffff640 1 --2- 172.21.15.16:0/3855537482 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ff6840651d0 0x7ff684067680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:08.374 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.368+0000 7ff67ffff640 1 -- 172.21.15.16:0/3855537482 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(135..135 src has 1..135) v4 ==== 7126+0+0 (secure 0 0 0) 0x7ff68c081d30 con 0x7ff69806c6e0 2023-05-22T15:46:08.374 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.368+0000 7ff696ffd640 1 --2- 172.21.15.16:0/3855537482 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ff6840651d0 0x7ff684067680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:08.374 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.368+0000 7ff696ffd640 1 --2- 172.21.15.16:0/3855537482 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ff6840651d0 0x7ff684067680 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7ff6900af000 tx=0x7ff6900a7070 comp rx=0 tx=0).ready entity=mgr.8835 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:08.376 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.372+0000 7ff67ffff640 1 -- 172.21.15.16:0/3855537482 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7ff68c04a620 con 0x7ff69806c6e0 2023-05-22T15:46:08.566 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.560+0000 7ff69dac8640 1 -- 172.21.15.16:0/3855537482 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff69804dbe0 con 0x7ff69806c6e0 2023-05-22T15:46:08.567 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.564+0000 7ff67ffff640 1 -- 172.21.15.16:0/3855537482 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 50 v50) v1 ==== 94+0+4644 (secure 0 0 0) 0x7ff68c04f8f0 con 0x7ff69806c6e0 2023-05-22T15:46:08.568 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:46:08.568 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":50,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6689,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/296693182","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6833","nonce":296693182},{"type":"v1","addr":"172.21.15.62:6835","nonce":296693182}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":46},{"gid":7402,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6834/856426578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":856426578},{"type":"v1","addr":"172.21.15.62:6834","nonce":856426578}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":46},{"gid":8541,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/2260895833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2260895833},{"type":"v1","addr":"172.21.15.16:6835","nonce":2260895833}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":46}],"filesystems":[{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-05-22T15:45:05.200760+0000","modified":"2023-05-22T15:45:07.832845+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8547},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8547":{"gid":8547,"name":"a","rank":0,"incarnation":47,"state":"up:active","state_seq":3,"addr":"172.21.15.16:6837/1566663454","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":1566663454},{"type":"v1","addr":"172.21.15.16:6837","nonce":1566663454}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":7}]} 2023-05-22T15:46:08.568 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 50 2023-05-22T15:46:08.569 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.564+0000 7ff69dac8640 1 -- 172.21.15.16:0/3855537482 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ff6840651d0 msgr2=0x7ff684067680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:08.569 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.564+0000 7ff69dac8640 1 --2- 172.21.15.16:0/3855537482 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ff6840651d0 0x7ff684067680 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7ff6900af000 tx=0x7ff6900a7070 comp rx=0 tx=0).stop 2023-05-22T15:46:08.570 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.564+0000 7ff69dac8640 1 -- 172.21.15.16:0/3855537482 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff69806c6e0 msgr2=0x7ff69814e1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:08.570 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.564+0000 7ff69dac8640 1 --2- 172.21.15.16:0/3855537482 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff69806c6e0 0x7ff69814e1c0 secure :-1 s=READY pgs=1754 cs=0 l=1 rev1=1 crypto rx=0x7ff68c00c340 tx=0x7ff68c00c800 comp rx=0 tx=0).stop 2023-05-22T15:46:08.570 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.564+0000 7ff69dac8640 1 -- 172.21.15.16:0/3855537482 shutdown_connections 2023-05-22T15:46:08.570 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.564+0000 7ff69dac8640 1 --2- 172.21.15.16:0/3855537482 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ff6840651d0 0x7ff684067680 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:08.570 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.564+0000 7ff69dac8640 1 --2- 172.21.15.16:0/3855537482 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff698158a10 0x7ff698159d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:08.570 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.564+0000 7ff69dac8640 1 --2- 172.21.15.16:0/3855537482 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff698070760 0x7ff69814f210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:08.570 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.564+0000 7ff69dac8640 1 --2- 172.21.15.16:0/3855537482 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff69806c6e0 0x7ff69814e1c0 unknown :-1 s=CLOSED pgs=1754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:08.571 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.564+0000 7ff69dac8640 1 -- 172.21.15.16:0/3855537482 >> 172.21.15.16:0/3855537482 conn(0x7ff6980cc870 msgr2=0x7ff698152dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:46:08.571 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.564+0000 7ff69dac8640 1 -- 172.21.15.16:0/3855537482 shutdown_connections 2023-05-22T15:46:08.571 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.564+0000 7ff69dac8640 1 -- 172.21.15.16:0/3855537482 wait complete. 2023-05-22T15:46:08.586 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-05-22T15:46:08.701 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.696+0000 7f42331f8640 1 Processor -- start 2023-05-22T15:46:08.701 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.696+0000 7f42331f8640 1 -- start start 2023-05-22T15:46:08.702 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.696+0000 7f42331f8640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f422c0c9b40 0x7f422c15e460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:08.703 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.696+0000 7f42331f8640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f422c070df0 0x7f422c071210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:08.703 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.696+0000 7f42331f8640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f422c0b0650 con 0x7f422c0ca4e0 2023-05-22T15:46:08.703 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.696+0000 7f42331f8640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f422c0b0370 con 0x7f422c070df0 2023-05-22T15:46:08.703 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.696+0000 7f42331f8640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f422c15d980 con 0x7f422c0c9b40 2023-05-22T15:46:08.703 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.696+0000 7f4231795640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f422c070df0 0x7f422c071210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:08.703 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.696+0000 7f4231795640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f422c070df0 0x7f422c071210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59676/0 (socket says 172.21.15.16:59676) 2023-05-22T15:46:08.703 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.696+0000 7f4231795640 1 -- 172.21.15.16:0/3192779422 learned_addr learned my addr 172.21.15.16:0/3192779422 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:46:08.704 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.696+0000 7f422bfff640 1 --2- 172.21.15.16:0/3192779422 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f422c0c9b40 0x7f422c15e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:08.704 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.700+0000 7f4231795640 1 -- 172.21.15.16:0/3192779422 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f422c0c9b40 msgr2=0x7f422c15e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:46:08.704 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.700+0000 7f4231795640 1 --2- 172.21.15.16:0/3192779422 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f422c0c9b40 0x7f422c15e460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:08.704 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.700+0000 7f4231795640 1 -- 172.21.15.16:0/3192779422 >> v1:172.21.15.62:6789/0 conn(0x7f422c0ca4e0 legacy=0x7f422c0afb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:46:08.704 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.700+0000 7f4231795640 1 -- 172.21.15.16:0/3192779422 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f422c15dc60 con 0x7f422c070df0 2023-05-22T15:46:08.705 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.700+0000 7f4231795640 1 --2- 172.21.15.16:0/3192779422 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f422c070df0 0x7f422c071210 secure :-1 s=READY pgs=1755 cs=0 l=1 rev1=1 crypto rx=0x7f4224009e50 tx=0x7f422401c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=769be4160ec8f244 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:08.705 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.700+0000 7f422b7fe640 1 -- 172.21.15.16:0/3192779422 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42240a56f0 con 0x7f422c070df0 2023-05-22T15:46:08.705 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.700+0000 7f422b7fe640 1 -- 172.21.15.16:0/3192779422 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42240a5850 con 0x7f422c070df0 2023-05-22T15:46:08.706 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.700+0000 7f422b7fe640 1 -- 172.21.15.16:0/3192779422 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42240a5be0 con 0x7f422c070df0 2023-05-22T15:46:08.706 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.700+0000 7f42331f8640 1 -- 172.21.15.16:0/3192779422 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f422c070df0 msgr2=0x7f422c071210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:08.706 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.700+0000 7f42331f8640 1 --2- 172.21.15.16:0/3192779422 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f422c070df0 0x7f422c071210 secure :-1 s=READY pgs=1755 cs=0 l=1 rev1=1 crypto rx=0x7f4224009e50 tx=0x7f422401c910 comp rx=0 tx=0).stop 2023-05-22T15:46:08.706 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.700+0000 7f42331f8640 1 -- 172.21.15.16:0/3192779422 shutdown_connections 2023-05-22T15:46:08.706 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.700+0000 7f42331f8640 1 --2- 172.21.15.16:0/3192779422 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f422c070df0 0x7f422c071210 unknown :-1 s=CLOSED pgs=1755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:08.706 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.700+0000 7f42331f8640 1 --2- 172.21.15.16:0/3192779422 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f422c0c9b40 0x7f422c15e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:08.706 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.700+0000 7f42331f8640 1 -- 172.21.15.16:0/3192779422 >> 172.21.15.16:0/3192779422 conn(0x7f422c069720 msgr2=0x7f422c069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:46:08.707 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.700+0000 7f42331f8640 1 -- 172.21.15.16:0/3192779422 shutdown_connections 2023-05-22T15:46:08.707 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.700+0000 7f42331f8640 1 -- 172.21.15.16:0/3192779422 wait complete. 2023-05-22T15:46:08.708 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.704+0000 7f42331f8640 1 Processor -- start 2023-05-22T15:46:08.708 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.704+0000 7f42331f8640 1 -- start start 2023-05-22T15:46:08.709 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.704+0000 7f42331f8640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f422c070df0 0x7f422c0c7b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:08.709 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.704+0000 7f42331f8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f422c0c9b40 0x7f422c14f5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:08.709 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.704+0000 7f42331f8640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f422c0ca4e0 0x7f422c069af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:08.709 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.704+0000 7f42331f8640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f422c15b430 con 0x7f422c0c9b40 2023-05-22T15:46:08.709 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.704+0000 7f42331f8640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f422c0c6670 con 0x7f422c0ca4e0 2023-05-22T15:46:08.709 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.704+0000 7f42331f8640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f422c0c61e0 con 0x7f422c070df0 2023-05-22T15:46:08.710 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.704+0000 7f422bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f422c0c9b40 0x7f422c14f5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:08.710 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.704+0000 7f422bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f422c0c9b40 0x7f422c14f5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59682/0 (socket says 172.21.15.16:59682) 2023-05-22T15:46:08.710 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.704+0000 7f422bfff640 1 -- 172.21.15.16:0/1947497559 learned_addr learned my addr 172.21.15.16:0/1947497559 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:46:08.710 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.704+0000 7f4230f94640 1 --2- 172.21.15.16:0/1947497559 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f422c070df0 0x7f422c0c7b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:08.710 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.704+0000 7f4231795640 1 --2- 172.21.15.16:0/1947497559 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f422c0ca4e0 0x7f422c069af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:08.710 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.704+0000 7f422bfff640 1 -- 172.21.15.16:0/1947497559 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f422c070df0 msgr2=0x7f422c0c7b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:08.711 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.704+0000 7f422bfff640 1 --2- 172.21.15.16:0/1947497559 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f422c070df0 0x7f422c0c7b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:08.711 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.704+0000 7f422bfff640 1 -- 172.21.15.16:0/1947497559 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f422c0ca4e0 msgr2=0x7f422c069af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:08.711 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.704+0000 7f422bfff640 1 --2- 172.21.15.16:0/1947497559 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f422c0ca4e0 0x7f422c069af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:08.711 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.704+0000 7f422bfff640 1 -- 172.21.15.16:0/1947497559 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4224009b30 con 0x7f422c0c9b40 2023-05-22T15:46:08.711 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.704+0000 7f422bfff640 1 --2- 172.21.15.16:0/1947497559 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f422c0c9b40 0x7f422c14f5c0 secure :-1 s=READY pgs=1756 cs=0 l=1 rev1=1 crypto rx=0x7f422000d800 tx=0x7f422000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:08.711 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.704+0000 7f42297fa640 1 -- 172.21.15.16:0/1947497559 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f422000b660 con 0x7f422c0c9b40 2023-05-22T15:46:08.711 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.704+0000 7f42297fa640 1 -- 172.21.15.16:0/1947497559 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f422000b7c0 con 0x7f422c0c9b40 2023-05-22T15:46:08.712 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.704+0000 7f42331f8640 1 -- 172.21.15.16:0/1947497559 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f422c0c8e80 con 0x7f422c0c9b40 2023-05-22T15:46:08.712 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.704+0000 7f42297fa640 1 -- 172.21.15.16:0/1947497559 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f422000bb30 con 0x7f422c0c9b40 2023-05-22T15:46:08.712 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.704+0000 7f42331f8640 1 -- 172.21.15.16:0/1947497559 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f422c0ca8b0 con 0x7f422c0c9b40 2023-05-22T15:46:08.713 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.704+0000 7f42331f8640 1 -- 172.21.15.16:0/1947497559 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f422c0c9010 con 0x7f422c0c9b40 2023-05-22T15:46:08.713 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.704+0000 7f42297fa640 1 -- 172.21.15.16:0/1947497559 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 59) v1 ==== 84068+0+0 (secure 0 0 0) 0x7f422000bc90 con 0x7f422c0c9b40 2023-05-22T15:46:08.713 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.704+0000 7f42297fa640 1 --2- 172.21.15.16:0/1947497559 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f42180651d0 0x7f4218067680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:08.713 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.704+0000 7f42297fa640 1 -- 172.21.15.16:0/1947497559 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(135..135 src has 1..135) v4 ==== 7126+0+0 (secure 0 0 0) 0x7f4220082ca0 con 0x7f422c0c9b40 2023-05-22T15:46:08.713 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.704+0000 7f4230f94640 1 --2- 172.21.15.16:0/1947497559 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f42180651d0 0x7f4218067680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:08.713 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.708+0000 7f4230f94640 1 --2- 172.21.15.16:0/1947497559 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f42180651d0 0x7f4218067680 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f421c0098e0 tx=0x7f421c009060 comp rx=0 tx=0).ready entity=mgr.8835 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:08.714 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.708+0000 7f42297fa640 1 -- 172.21.15.16:0/1947497559 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f42200508e0 con 0x7f422c0c9b40 2023-05-22T15:46:08.866 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.860+0000 7f42331f8640 1 -- 172.21.15.16:0/1947497559 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f422c0cb010 con 0x7f422c0c9b40 2023-05-22T15:46:08.868 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.864+0000 7f42297fa640 1 -- 172.21.15.16:0/1947497559 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v135) v1 ==== 74+0+19091 (secure 0 0 0) 0x7f4220053f00 con 0x7f422c0c9b40 2023-05-22T15:46:08.869 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:46:08.869 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":135,"fsid":"b2719834-7847-4d98-90a6-f65d57f85c49","created":"2023-05-22T15:26:25.259314+0000","modified":"2023-05-22T15:46:08.343543+0000","last_up_change":"2023-05-22T15:26:36.448524+0000","last_in_change":"2023-05-22T15:26:27.675668+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-05-22T15:26:38.019781+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-05-22T15:26:43.570514+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2023-05-22T15:45:03.718869+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"131","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2023-05-22T15:45:04.168135+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"135","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":135,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9e7ff854-b25f-4cbb-a5c2-1799d9c53d2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":37021},{"type":"v1","addr":"172.21.15.16:6825","nonce":37021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":37021},{"type":"v1","addr":"172.21.15.16:6827","nonce":37021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":37021},{"type":"v1","addr":"172.21.15.16:6831","nonce":37021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":37021},{"type":"v1","addr":"172.21.15.16:6829","nonce":37021}]},"public_addr":"172.21.15.16:6825/37021","cluster_addr":"172.21.15.16:6827/37021","heartbeat_back_addr":"172.21.15.16:6831/37021","heartbeat_front_addr":"172.21.15.16:6829/37021","state":["exists","up"]},{"osd":1,"uuid":"4702fbc9-a927-4565-aca5-41dc69cc8264","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":37008},{"type":"v1","addr":"172.21.15.16:6801","nonce":37008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":37008},{"type":"v1","addr":"172.21.15.16:6803","nonce":37008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":37008},{"type":"v1","addr":"172.21.15.16:6807","nonce":37008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":37008},{"type":"v1","addr":"172.21.15.16:6805","nonce":37008}]},"public_addr":"172.21.15.16:6801/37008","cluster_addr":"172.21.15.16:6803/37008","heartbeat_back_addr":"172.21.15.16:6807/37008","heartbeat_front_addr":"172.21.15.16:6805/37008","state":["exists","up"]},{"osd":2,"uuid":"1e7bba46-4eba-47ef-bb81-5e2d371638ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":37009},{"type":"v1","addr":"172.21.15.16:6809","nonce":37009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":37009},{"type":"v1","addr":"172.21.15.16:6811","nonce":37009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":37009},{"type":"v1","addr":"172.21.15.16:6815","nonce":37009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":37009},{"type":"v1","addr":"172.21.15.16:6813","nonce":37009}]},"public_addr":"172.21.15.16:6809/37009","cluster_addr":"172.21.15.16:6811/37009","heartbeat_back_addr":"172.21.15.16:6815/37009","heartbeat_front_addr":"172.21.15.16:6813/37009","state":["exists","up"]},{"osd":3,"uuid":"383cacc0-5a88-4904-990e-498ca59ef48a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":37020},{"type":"v1","addr":"172.21.15.16:6817","nonce":37020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":37020},{"type":"v1","addr":"172.21.15.16:6819","nonce":37020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":37020},{"type":"v1","addr":"172.21.15.16:6823","nonce":37020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":37020},{"type":"v1","addr":"172.21.15.16:6821","nonce":37020}]},"public_addr":"172.21.15.16:6817/37020","cluster_addr":"172.21.15.16:6819/37020","heartbeat_back_addr":"172.21.15.16:6823/37020","heartbeat_front_addr":"172.21.15.16:6821/37020","state":["exists","up"]},{"osd":4,"uuid":"970500ef-95b7-42c1-b080-eed4fae7013d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":37353},{"type":"v1","addr":"172.21.15.62:6817","nonce":37353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6819","nonce":37353},{"type":"v1","addr":"172.21.15.62:6822","nonce":37353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37353},{"type":"v1","addr":"172.21.15.62:6831","nonce":37353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37353},{"type":"v1","addr":"172.21.15.62:6826","nonce":37353}]},"public_addr":"172.21.15.62:6817/37353","cluster_addr":"172.21.15.62:6822/37353","heartbeat_back_addr":"172.21.15.62:6831/37353","heartbeat_front_addr":"172.21.15.62:6826/37353","state":["exists","up"]},{"osd":5,"uuid":"022d67db-d4d8-445b-b1b9-8b446a00f5c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":37352},{"type":"v1","addr":"172.21.15.62:6801","nonce":37352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":37352},{"type":"v1","addr":"172.21.15.62:6803","nonce":37352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":37352},{"type":"v1","addr":"172.21.15.62:6807","nonce":37352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":37352},{"type":"v1","addr":"172.21.15.62:6805","nonce":37352}]},"public_addr":"172.21.15.62:6801/37352","cluster_addr":"172.21.15.62:6803/37352","heartbeat_back_addr":"172.21.15.62:6807/37352","heartbeat_front_addr":"172.21.15.62:6805/37352","state":["exists","up"]},{"osd":6,"uuid":"42bd372b-6688-4a1b-8407-e16e8c266e33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":37354},{"type":"v1","addr":"172.21.15.62:6820","nonce":37354}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":37354},{"type":"v1","addr":"172.21.15.62:6823","nonce":37354}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6829","nonce":37354},{"type":"v1","addr":"172.21.15.62:6830","nonce":37354}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6825","nonce":37354},{"type":"v1","addr":"172.21.15.62:6827","nonce":37354}]},"public_addr":"172.21.15.62:6820/37354","cluster_addr":"172.21.15.62:6823/37354","heartbeat_back_addr":"172.21.15.62:6830/37354","heartbeat_front_addr":"172.21.15.62:6827/37354","state":["exists","up"]},{"osd":7,"uuid":"ac70b2a3-f41b-4bf2-b227-7f2cbb55426d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37355},{"type":"v1","addr":"172.21.15.62:6809","nonce":37355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37355},{"type":"v1","addr":"172.21.15.62:6811","nonce":37355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":37355},{"type":"v1","addr":"172.21.15.62:6815","nonce":37355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":37355},{"type":"v1","addr":"172.21.15.62:6813","nonce":37355}]},"public_addr":"172.21.15.62:6809/37355","cluster_addr":"172.21.15.62:6811/37355","heartbeat_back_addr":"172.21.15.62:6815/37355","heartbeat_front_addr":"172.21.15.62:6813/37355","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.368896+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.107273+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.276888+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.287502+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.124791+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.251937+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.255679+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.232797+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:0/79646455":"2023-05-23T15:46:07.335649+0000","172.21.15.16:0/2657546211":"2023-05-23T15:46:07.335649+0000","172.21.15.16:0/3313824983":"2023-05-23T15:46:07.335649+0000","172.21.15.16:0/3149639262":"2023-05-23T15:46:07.335649+0000","172.21.15.16:0/2513263958":"2023-05-23T15:46:07.335649+0000","172.21.15.16:0/2722517029":"2023-05-23T15:46:07.335649+0000","172.21.15.16:0/736449025":"2023-05-23T15:46:07.335649+0000","172.21.15.16:0/3623988153":"2023-05-23T15:46:07.335649+0000","172.21.15.16:0/1950855917":"2023-05-23T15:45:17.648374+0000","172.21.15.16:0/1889818530":"2023-05-23T15:45:17.648374+0000","172.21.15.16:6833/36921":"2023-05-23T15:45:17.648374+0000","172.21.15.16:0/4160466619":"2023-05-23T15:45:17.648374+0000","172.21.15.16:0/55601757":"2023-05-23T15:45:17.648374+0000","172.21.15.16:6832/36921":"2023-05-23T15:45:17.648374+0000","172.21.15.16:0/4127127856":"2023-05-23T15:45:17.648374+0000","172.21.15.16:0/2333289485":"2023-05-23T15:45:17.648374+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":17,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[{"pool":17,"snaps":[{"begin":2,"length":2}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-05-22T15:46:08.870 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.864+0000 7f42331f8640 1 -- 172.21.15.16:0/1947497559 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f42180651d0 msgr2=0x7f4218067680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:08.871 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.864+0000 7f42331f8640 1 --2- 172.21.15.16:0/1947497559 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f42180651d0 0x7f4218067680 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f421c0098e0 tx=0x7f421c009060 comp rx=0 tx=0).stop 2023-05-22T15:46:08.871 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.864+0000 7f42331f8640 1 -- 172.21.15.16:0/1947497559 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f422c0c9b40 msgr2=0x7f422c14f5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:08.871 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.864+0000 7f42331f8640 1 --2- 172.21.15.16:0/1947497559 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f422c0c9b40 0x7f422c14f5c0 secure :-1 s=READY pgs=1756 cs=0 l=1 rev1=1 crypto rx=0x7f422000d800 tx=0x7f422000dcc0 comp rx=0 tx=0).stop 2023-05-22T15:46:08.871 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.864+0000 7f42331f8640 1 -- 172.21.15.16:0/1947497559 shutdown_connections 2023-05-22T15:46:08.871 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.864+0000 7f42331f8640 1 --2- 172.21.15.16:0/1947497559 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f422c0ca4e0 0x7f422c069af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:08.871 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.864+0000 7f42331f8640 1 --2- 172.21.15.16:0/1947497559 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f422c0c9b40 0x7f422c14f5c0 unknown :-1 s=CLOSED pgs=1756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:08.872 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.864+0000 7f42331f8640 1 --2- 172.21.15.16:0/1947497559 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f42180651d0 0x7f4218067680 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:08.872 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.864+0000 7f42331f8640 1 --2- 172.21.15.16:0/1947497559 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f422c070df0 0x7f422c0c7b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:08.872 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.864+0000 7f42331f8640 1 -- 172.21.15.16:0/1947497559 >> 172.21.15.16:0/1947497559 conn(0x7f422c069720 msgr2=0x7f422c156640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:46:08.872 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.868+0000 7f42331f8640 1 -- 172.21.15.16:0/1947497559 shutdown_connections 2023-05-22T15:46:08.872 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:08.868+0000 7f42331f8640 1 -- 172.21.15.16:0/1947497559 wait complete. 2023-05-22T15:46:08.886 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-05-22T15:46:08.887 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-05-22T15:46:09.007 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.000+0000 7f7ef9e9c640 1 Processor -- start 2023-05-22T15:46:09.007 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.004+0000 7f7ef9e9c640 1 -- start start 2023-05-22T15:46:09.009 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.004+0000 7f7ef9e9c640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f7ef41531b0 0x7f7ef40afb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:09.009 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.004+0000 7f7ef9e9c640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7ef4159a40 0x7f7ef40c8ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:09.009 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.004+0000 7f7ef9e9c640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ef415a0a0 con 0x7f7ef40b0080 2023-05-22T15:46:09.009 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.004+0000 7f7ef9e9c640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ef40b0520 con 0x7f7ef41531b0 2023-05-22T15:46:09.009 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.004+0000 7f7ef9e9c640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ef4159e50 con 0x7f7ef4159a40 2023-05-22T15:46:09.009 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.004+0000 7f7ef37fe640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f7ef41531b0 0x7f7ef40afb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:09.010 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.004+0000 7f7ef37fe640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f7ef41531b0 0x7f7ef40afb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59690/0 (socket says 172.21.15.16:59690) 2023-05-22T15:46:09.010 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.004+0000 7f7ef37fe640 1 -- 172.21.15.16:0/3989908322 learned_addr learned my addr 172.21.15.16:0/3989908322 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:46:09.010 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.004+0000 7f7ef3fff640 1 --2- 172.21.15.16:0/3989908322 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7ef4159a40 0x7f7ef40c8ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:09.010 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.004+0000 7f7ef37fe640 1 -- 172.21.15.16:0/3989908322 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7ef4159a40 msgr2=0x7f7ef40c8ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:46:09.010 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.004+0000 7f7ef37fe640 1 --2- 172.21.15.16:0/3989908322 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7ef4159a40 0x7f7ef40c8ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:09.011 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.004+0000 7f7ef37fe640 1 -- 172.21.15.16:0/3989908322 >> v1:172.21.15.16:6789/0 conn(0x7f7ef40b0080 legacy=0x7f7ef4159330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:09.011 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.004+0000 7f7ef37fe640 1 -- 172.21.15.16:0/3989908322 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ef4153580 con 0x7f7ef41531b0 2023-05-22T15:46:09.012 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.004+0000 7f7ef37fe640 1 --2- 172.21.15.16:0/3989908322 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ef41531b0 0x7f7ef40afb40 secure :-1 s=READY pgs=1758 cs=0 l=1 rev1=1 crypto rx=0x7f7edc009870 tx=0x7f7edc01c540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63fc781918713108 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:09.012 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.008+0000 7f7ef27fc640 1 -- 172.21.15.16:0/3989908322 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7edc0a7070 con 0x7f7ef41531b0 2023-05-22T15:46:09.012 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.008+0000 7f7ef27fc640 1 -- 172.21.15.16:0/3989908322 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7edc0ab430 con 0x7f7ef41531b0 2023-05-22T15:46:09.013 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.008+0000 7f7ef27fc640 1 -- 172.21.15.16:0/3989908322 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7edc0ab7c0 con 0x7f7ef41531b0 2023-05-22T15:46:09.013 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.008+0000 7f7ef9e9c640 1 -- 172.21.15.16:0/3989908322 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ef41531b0 msgr2=0x7f7ef40afb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:09.013 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.008+0000 7f7ef9e9c640 1 --2- 172.21.15.16:0/3989908322 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ef41531b0 0x7f7ef40afb40 secure :-1 s=READY pgs=1758 cs=0 l=1 rev1=1 crypto rx=0x7f7edc009870 tx=0x7f7edc01c540 comp rx=0 tx=0).stop 2023-05-22T15:46:09.013 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.008+0000 7f7ef9e9c640 1 -- 172.21.15.16:0/3989908322 shutdown_connections 2023-05-22T15:46:09.013 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.008+0000 7f7ef9e9c640 1 --2- 172.21.15.16:0/3989908322 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7ef4159a40 0x7f7ef40c8ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:09.013 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.008+0000 7f7ef9e9c640 1 --2- 172.21.15.16:0/3989908322 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ef41531b0 0x7f7ef40afb40 unknown :-1 s=CLOSED pgs=1758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:09.013 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.008+0000 7f7ef9e9c640 1 -- 172.21.15.16:0/3989908322 >> 172.21.15.16:0/3989908322 conn(0x7f7ef4069720 msgr2=0x7f7ef4069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:46:09.014 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.008+0000 7f7ef9e9c640 1 -- 172.21.15.16:0/3989908322 shutdown_connections 2023-05-22T15:46:09.014 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.008+0000 7f7ef9e9c640 1 -- 172.21.15.16:0/3989908322 wait complete. 2023-05-22T15:46:09.017 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.008+0000 7f7ef9e9c640 1 Processor -- start 2023-05-22T15:46:09.017 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.008+0000 7f7ef9e9c640 1 -- start start 2023-05-22T15:46:09.017 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.008+0000 7f7ef9e9c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7ef40b0080 0x7f7ef4146650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:09.017 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.008+0000 7f7ef9e9c640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7ef41531b0 0x7f7ef40adfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:09.018 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.008+0000 7f7ef9e9c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ef4159a40 0x7f7ef40ae500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:09.018 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.008+0000 7f7ef9e9c640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ef40c4a40 con 0x7f7ef4159a40 2023-05-22T15:46:09.018 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.008+0000 7f7ef9e9c640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ef40c3e30 con 0x7f7ef40b0080 2023-05-22T15:46:09.018 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.008+0000 7f7ef9e9c640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ef40c4110 con 0x7f7ef41531b0 2023-05-22T15:46:09.018 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.008+0000 7f7ef3fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ef4159a40 0x7f7ef40ae500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:09.018 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.008+0000 7f7ef3fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ef4159a40 0x7f7ef40ae500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59698/0 (socket says 172.21.15.16:59698) 2023-05-22T15:46:09.018 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.008+0000 7f7ef3fff640 1 -- 172.21.15.16:0/4094474822 learned_addr learned my addr 172.21.15.16:0/4094474822 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:46:09.019 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.008+0000 7f7ef37fe640 1 --2- 172.21.15.16:0/4094474822 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7ef40b0080 0x7f7ef4146650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:09.019 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.008+0000 7f7ef2ffd640 1 --2- 172.21.15.16:0/4094474822 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7ef41531b0 0x7f7ef40adfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:09.019 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.008+0000 7f7ef3fff640 1 -- 172.21.15.16:0/4094474822 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7ef41531b0 msgr2=0x7f7ef40adfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:09.019 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.008+0000 7f7ef3fff640 1 --2- 172.21.15.16:0/4094474822 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7ef41531b0 0x7f7ef40adfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:09.019 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.008+0000 7f7ef3fff640 1 -- 172.21.15.16:0/4094474822 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7ef40b0080 msgr2=0x7f7ef4146650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:09.019 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.008+0000 7f7ef3fff640 1 --2- 172.21.15.16:0/4094474822 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7ef40b0080 0x7f7ef4146650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:09.019 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.008+0000 7f7ef3fff640 1 -- 172.21.15.16:0/4094474822 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7edc0094d0 con 0x7f7ef4159a40 2023-05-22T15:46:09.019 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.008+0000 7f7ef2ffd640 1 --2- 172.21.15.16:0/4094474822 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7ef41531b0 0x7f7ef40adfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:46:09.020 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.008+0000 7f7ef3fff640 1 --2- 172.21.15.16:0/4094474822 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ef4159a40 0x7f7ef40ae500 secure :-1 s=READY pgs=1759 cs=0 l=1 rev1=1 crypto rx=0x7f7eec00d560 tx=0x7f7eec00da20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:09.020 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.008+0000 7f7eebfff640 1 -- 172.21.15.16:0/4094474822 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7eec0125f0 con 0x7f7ef4159a40 2023-05-22T15:46:09.020 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.008+0000 7f7eebfff640 1 -- 172.21.15.16:0/4094474822 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7eec012770 con 0x7f7ef4159a40 2023-05-22T15:46:09.020 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.008+0000 7f7eebfff640 1 -- 172.21.15.16:0/4094474822 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7eec012b20 con 0x7f7ef4159a40 2023-05-22T15:46:09.020 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.008+0000 7f7ef9e9c640 1 -- 172.21.15.16:0/4094474822 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ef415d0c0 con 0x7f7ef4159a40 2023-05-22T15:46:09.020 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.012+0000 7f7ef9e9c640 1 -- 172.21.15.16:0/4094474822 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ef415cde0 con 0x7f7ef4159a40 2023-05-22T15:46:09.021 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.012+0000 7f7ef9e9c640 1 -- 172.21.15.16:0/4094474822 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ef40ce020 con 0x7f7ef4159a40 2023-05-22T15:46:09.021 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.012+0000 7f7eebfff640 1 -- 172.21.15.16:0/4094474822 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 59) v1 ==== 84068+0+0 (secure 0 0 0) 0x7f7eec012c80 con 0x7f7ef4159a40 2023-05-22T15:46:09.021 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.012+0000 7f7eebfff640 1 --2- 172.21.15.16:0/4094474822 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f7ed40651d0 0x7f7ed4067680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:09.021 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.012+0000 7f7eebfff640 1 -- 172.21.15.16:0/4094474822 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(135..135 src has 1..135) v4 ==== 7126+0+0 (secure 0 0 0) 0x7f7eec082d80 con 0x7f7ef4159a40 2023-05-22T15:46:09.021 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.012+0000 7f7ef37fe640 1 --2- 172.21.15.16:0/4094474822 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f7ed40651d0 0x7f7ed4067680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:09.022 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.012+0000 7f7ef37fe640 1 --2- 172.21.15.16:0/4094474822 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f7ed40651d0 0x7f7ed4067680 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f7edc0af000 tx=0x7f7edc008040 comp rx=0 tx=0).ready entity=mgr.8835 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:09.022 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.016+0000 7f7eebfff640 1 -- 172.21.15.16:0/4094474822 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f7eec050910 con 0x7f7ef4159a40 2023-05-22T15:46:09.203 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.200+0000 7f7ef9e9c640 1 -- 172.21.15.16:0/4094474822 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7ef404dbe0 con 0x7f7ef4159a40 2023-05-22T15:46:09.204 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.200+0000 7f7eebfff640 1 -- 172.21.15.16:0/4094474822 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 50 v50) v1 ==== 94+0+4644 (secure 0 0 0) 0x7f7eec053f30 con 0x7f7ef4159a40 2023-05-22T15:46:09.204 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:46:09.205 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":50,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6689,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/296693182","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6833","nonce":296693182},{"type":"v1","addr":"172.21.15.62:6835","nonce":296693182}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":46},{"gid":7402,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6834/856426578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":856426578},{"type":"v1","addr":"172.21.15.62:6834","nonce":856426578}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":46},{"gid":8541,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/2260895833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2260895833},{"type":"v1","addr":"172.21.15.16:6835","nonce":2260895833}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":46}],"filesystems":[{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-05-22T15:45:05.200760+0000","modified":"2023-05-22T15:45:07.832845+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8547},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8547":{"gid":8547,"name":"a","rank":0,"incarnation":47,"state":"up:active","state_seq":3,"addr":"172.21.15.16:6837/1566663454","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":1566663454},{"type":"v1","addr":"172.21.15.16:6837","nonce":1566663454}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":7}]} 2023-05-22T15:46:09.205 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 50 2023-05-22T15:46:09.206 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.200+0000 7f7ef9e9c640 1 -- 172.21.15.16:0/4094474822 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f7ed40651d0 msgr2=0x7f7ed4067680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:09.206 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.200+0000 7f7ef9e9c640 1 --2- 172.21.15.16:0/4094474822 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f7ed40651d0 0x7f7ed4067680 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f7edc0af000 tx=0x7f7edc008040 comp rx=0 tx=0).stop 2023-05-22T15:46:09.206 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.200+0000 7f7ef9e9c640 1 -- 172.21.15.16:0/4094474822 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ef4159a40 msgr2=0x7f7ef40ae500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:09.206 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.200+0000 7f7ef9e9c640 1 --2- 172.21.15.16:0/4094474822 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ef4159a40 0x7f7ef40ae500 secure :-1 s=READY pgs=1759 cs=0 l=1 rev1=1 crypto rx=0x7f7eec00d560 tx=0x7f7eec00da20 comp rx=0 tx=0).stop 2023-05-22T15:46:09.207 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.200+0000 7f7ef9e9c640 1 -- 172.21.15.16:0/4094474822 shutdown_connections 2023-05-22T15:46:09.207 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.200+0000 7f7ef9e9c640 1 --2- 172.21.15.16:0/4094474822 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ef4159a40 0x7f7ef40ae500 unknown :-1 s=CLOSED pgs=1759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:09.207 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.200+0000 7f7ef9e9c640 1 --2- 172.21.15.16:0/4094474822 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f7ed40651d0 0x7f7ed4067680 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:09.207 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.200+0000 7f7ef9e9c640 1 --2- 172.21.15.16:0/4094474822 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7ef41531b0 0x7f7ef40adfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:09.207 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.200+0000 7f7ef9e9c640 1 --2- 172.21.15.16:0/4094474822 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7ef40b0080 0x7f7ef4146650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:09.207 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.200+0000 7f7ef9e9c640 1 -- 172.21.15.16:0/4094474822 >> 172.21.15.16:0/4094474822 conn(0x7f7ef4069720 msgr2=0x7f7ef40af220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:46:09.207 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.200+0000 7f7ef9e9c640 1 -- 172.21.15.16:0/4094474822 shutdown_connections 2023-05-22T15:46:09.207 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.200+0000 7f7ef9e9c640 1 -- 172.21.15.16:0/4094474822 wait complete. 2023-05-22T15:46:09.223 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-05-22T15:46:09.337 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.332+0000 7fa61c424640 1 Processor -- start 2023-05-22T15:46:09.338 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.332+0000 7fa61c424640 1 -- start start 2023-05-22T15:46:09.338 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.332+0000 7fa61c424640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fa6140cb6a0 0x7fa614159340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:09.339 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.332+0000 7fa61c424640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fa6140b0250 0x7fa614159880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:09.339 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.332+0000 7fa61c424640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa614156b90 con 0x7fa6141567c0 2023-05-22T15:46:09.339 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.332+0000 7fa61c424640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa61415a0b0 con 0x7fa6140cb6a0 2023-05-22T15:46:09.339 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.332+0000 7fa61c424640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6140ca8e0 con 0x7fa6140b0250 2023-05-22T15:46:09.339 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.332+0000 7fa6199bf640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fa6140cb6a0 0x7fa614159340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:09.340 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.332+0000 7fa6199bf640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fa6140cb6a0 0x7fa614159340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43366/0 (socket says 172.21.15.16:43366) 2023-05-22T15:46:09.340 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.332+0000 7fa6199bf640 1 -- 172.21.15.16:0/2695733203 learned_addr learned my addr 172.21.15.16:0/2695733203 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:46:09.340 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.332+0000 7fa61a9c1640 1 --2- 172.21.15.16:0/2695733203 >> v2:172.21.15.62:3300/0 conn(0x7fa6140b0250 0x7fa614159880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:09.340 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.332+0000 7fa6199bf640 1 -- 172.21.15.16:0/2695733203 >> v2:172.21.15.62:3300/0 conn(0x7fa6140b0250 msgr2=0x7fa614159880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:46:09.340 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.332+0000 7fa6199bf640 1 --2- 172.21.15.16:0/2695733203 >> v2:172.21.15.62:3300/0 conn(0x7fa6140b0250 0x7fa614159880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:09.340 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.332+0000 7fa6199bf640 1 -- 172.21.15.16:0/2695733203 >> v1:172.21.15.16:6789/0 conn(0x7fa6141567c0 legacy=0x7fa6140afb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:09.341 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.332+0000 7fa6199bf640 1 -- 172.21.15.16:0/2695733203 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6140cabc0 con 0x7fa6140cb6a0 2023-05-22T15:46:09.341 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.336+0000 7fa6199bf640 1 --2- 172.21.15.16:0/2695733203 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa6140cb6a0 0x7fa614159340 secure :-1 s=READY pgs=1761 cs=0 l=1 rev1=1 crypto rx=0x7fa6140cbc70 tx=0x7fa60401c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35d0ba165d900881 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:09.342 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.336+0000 7fa6191be640 1 -- 172.21.15.16:0/2695733203 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6040ab600 con 0x7fa6140cb6a0 2023-05-22T15:46:09.342 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.336+0000 7fa6191be640 1 -- 172.21.15.16:0/2695733203 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6040a7050 con 0x7fa6140cb6a0 2023-05-22T15:46:09.342 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.336+0000 7fa6191be640 1 -- 172.21.15.16:0/2695733203 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6040ab980 con 0x7fa6140cb6a0 2023-05-22T15:46:09.342 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.336+0000 7fa61c424640 1 -- 172.21.15.16:0/2695733203 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa6140cb6a0 msgr2=0x7fa614159340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:09.343 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.336+0000 7fa61c424640 1 --2- 172.21.15.16:0/2695733203 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa6140cb6a0 0x7fa614159340 secure :-1 s=READY pgs=1761 cs=0 l=1 rev1=1 crypto rx=0x7fa6140cbc70 tx=0x7fa60401c500 comp rx=0 tx=0).stop 2023-05-22T15:46:09.343 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.336+0000 7fa61c424640 1 -- 172.21.15.16:0/2695733203 shutdown_connections 2023-05-22T15:46:09.343 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.336+0000 7fa61c424640 1 --2- 172.21.15.16:0/2695733203 >> v2:172.21.15.62:3300/0 conn(0x7fa6140b0250 0x7fa614159880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:09.343 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.336+0000 7fa61c424640 1 --2- 172.21.15.16:0/2695733203 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa6140cb6a0 0x7fa614159340 unknown :-1 s=CLOSED pgs=1761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:09.343 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.336+0000 7fa61c424640 1 -- 172.21.15.16:0/2695733203 >> 172.21.15.16:0/2695733203 conn(0x7fa614069720 msgr2=0x7fa614069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:46:09.343 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.336+0000 7fa61c424640 1 -- 172.21.15.16:0/2695733203 shutdown_connections 2023-05-22T15:46:09.343 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.336+0000 7fa61c424640 1 -- 172.21.15.16:0/2695733203 wait complete. 2023-05-22T15:46:09.348 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.336+0000 7fa61c424640 1 Processor -- start 2023-05-22T15:46:09.348 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.336+0000 7fa61c424640 1 -- start start 2023-05-22T15:46:09.349 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.340+0000 7fa61c424640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa6140b0250 0x7fa61415b7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:09.349 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.340+0000 7fa61c424640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa6140cb6a0 0x7fa61415bd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:09.349 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.340+0000 7fa61c424640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa6141567c0 0x7fa6140adb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:09.350 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.340+0000 7fa61c424640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6140c71f0 con 0x7fa6140cb6a0 2023-05-22T15:46:09.350 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.340+0000 7fa61c424640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa614144d60 con 0x7fa6140b0250 2023-05-22T15:46:09.350 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.340+0000 7fa61c424640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6140afcb0 con 0x7fa6141567c0 2023-05-22T15:46:09.351 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.340+0000 7fa6199bf640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa6140cb6a0 0x7fa61415bd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:09.351 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.340+0000 7fa6199bf640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa6140cb6a0 0x7fa61415bd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43370/0 (socket says 172.21.15.16:43370) 2023-05-22T15:46:09.351 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.340+0000 7fa6199bf640 1 -- 172.21.15.16:0/2785359069 learned_addr learned my addr 172.21.15.16:0/2785359069 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:46:09.352 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.340+0000 7fa61a1c0640 1 --2- 172.21.15.16:0/2785359069 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa6140b0250 0x7fa61415b7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:09.352 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.340+0000 7fa61a9c1640 1 --2- 172.21.15.16:0/2785359069 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa6141567c0 0x7fa6140adb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:09.352 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.340+0000 7fa6199bf640 1 -- 172.21.15.16:0/2785359069 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa6141567c0 msgr2=0x7fa6140adb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:09.353 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.340+0000 7fa6199bf640 1 --2- 172.21.15.16:0/2785359069 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa6141567c0 0x7fa6140adb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:09.353 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.340+0000 7fa6199bf640 1 -- 172.21.15.16:0/2785359069 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa6140b0250 msgr2=0x7fa61415b7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:09.353 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.340+0000 7fa6199bf640 1 --2- 172.21.15.16:0/2785359069 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa6140b0250 0x7fa61415b7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:09.353 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.340+0000 7fa6199bf640 1 -- 172.21.15.16:0/2785359069 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6040094d0 con 0x7fa6140cb6a0 2023-05-22T15:46:09.354 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.340+0000 7fa6199bf640 1 --2- 172.21.15.16:0/2785359069 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa6140cb6a0 0x7fa61415bd20 secure :-1 s=READY pgs=1762 cs=0 l=1 rev1=1 crypto rx=0x7fa604019110 tx=0x7fa6040192a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:09.354 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.340+0000 7fa60affd640 1 -- 172.21.15.16:0/2785359069 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6040b37d0 con 0x7fa6140cb6a0 2023-05-22T15:46:09.354 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.340+0000 7fa60affd640 1 -- 172.21.15.16:0/2785359069 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6040b3930 con 0x7fa6140cb6a0 2023-05-22T15:46:09.354 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.340+0000 7fa61c424640 1 -- 172.21.15.16:0/2785359069 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6140c8e80 con 0x7fa6140cb6a0 2023-05-22T15:46:09.355 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.340+0000 7fa60affd640 1 -- 172.21.15.16:0/2785359069 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6040b3ce0 con 0x7fa6140cb6a0 2023-05-22T15:46:09.355 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.340+0000 7fa61c424640 1 -- 172.21.15.16:0/2785359069 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa614159dc0 con 0x7fa6140cb6a0 2023-05-22T15:46:09.355 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.340+0000 7fa61c424640 1 -- 172.21.15.16:0/2785359069 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6140c9010 con 0x7fa6140cb6a0 2023-05-22T15:46:09.356 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.340+0000 7fa60affd640 1 -- 172.21.15.16:0/2785359069 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 59) v1 ==== 84068+0+0 (secure 0 0 0) 0x7fa6040b2070 con 0x7fa6140cb6a0 2023-05-22T15:46:09.356 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.340+0000 7fa60affd640 1 --2- 172.21.15.16:0/2785359069 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fa5f80651d0 0x7fa5f8067680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:09.356 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.340+0000 7fa60affd640 1 -- 172.21.15.16:0/2785359069 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(135..135 src has 1..135) v4 ==== 7126+0+0 (secure 0 0 0) 0x7fa60411cd30 con 0x7fa6140cb6a0 2023-05-22T15:46:09.356 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.340+0000 7fa61a1c0640 1 --2- 172.21.15.16:0/2785359069 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fa5f80651d0 0x7fa5f8067680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:09.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.340+0000 7fa61a1c0640 1 --2- 172.21.15.16:0/2785359069 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fa5f80651d0 0x7fa5f8067680 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fa610000f70 tx=0x7fa610009040 comp rx=0 tx=0).ready entity=mgr.8835 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:09.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.344+0000 7fa60affd640 1 -- 172.21.15.16:0/2785359069 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fa6040ea8c0 con 0x7fa6140cb6a0 2023-05-22T15:46:09.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.524+0000 7fa61c424640 1 -- 172.21.15.16:0/2785359069 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa61404dbe0 con 0x7fa6140cb6a0 2023-05-22T15:46:09.530 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.524+0000 7fa60affd640 1 -- 172.21.15.16:0/2785359069 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 50 v50) v1 ==== 94+0+4644 (secure 0 0 0) 0x7fa6040edee0 con 0x7fa6140cb6a0 2023-05-22T15:46:09.531 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:46:09.531 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":50,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6689,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/296693182","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6833","nonce":296693182},{"type":"v1","addr":"172.21.15.62:6835","nonce":296693182}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":46},{"gid":7402,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6834/856426578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":856426578},{"type":"v1","addr":"172.21.15.62:6834","nonce":856426578}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":46},{"gid":8541,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/2260895833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2260895833},{"type":"v1","addr":"172.21.15.16:6835","nonce":2260895833}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":46}],"filesystems":[{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-05-22T15:45:05.200760+0000","modified":"2023-05-22T15:45:07.832845+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8547},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8547":{"gid":8547,"name":"a","rank":0,"incarnation":47,"state":"up:active","state_seq":3,"addr":"172.21.15.16:6837/1566663454","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":1566663454},{"type":"v1","addr":"172.21.15.16:6837","nonce":1566663454}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":7}]} 2023-05-22T15:46:09.531 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 50 2023-05-22T15:46:09.532 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.528+0000 7fa61c424640 1 -- 172.21.15.16:0/2785359069 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fa5f80651d0 msgr2=0x7fa5f8067680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:09.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.528+0000 7fa61c424640 1 --2- 172.21.15.16:0/2785359069 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fa5f80651d0 0x7fa5f8067680 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fa610000f70 tx=0x7fa610009040 comp rx=0 tx=0).stop 2023-05-22T15:46:09.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.528+0000 7fa61c424640 1 -- 172.21.15.16:0/2785359069 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa6140cb6a0 msgr2=0x7fa61415bd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:09.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.528+0000 7fa61c424640 1 --2- 172.21.15.16:0/2785359069 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa6140cb6a0 0x7fa61415bd20 secure :-1 s=READY pgs=1762 cs=0 l=1 rev1=1 crypto rx=0x7fa604019110 tx=0x7fa6040192a0 comp rx=0 tx=0).stop 2023-05-22T15:46:09.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.528+0000 7fa61c424640 1 -- 172.21.15.16:0/2785359069 shutdown_connections 2023-05-22T15:46:09.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.528+0000 7fa61c424640 1 --2- 172.21.15.16:0/2785359069 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fa5f80651d0 0x7fa5f8067680 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:09.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.528+0000 7fa61c424640 1 --2- 172.21.15.16:0/2785359069 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa6141567c0 0x7fa6140adb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:09.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.528+0000 7fa61c424640 1 --2- 172.21.15.16:0/2785359069 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa6140cb6a0 0x7fa61415bd20 unknown :-1 s=CLOSED pgs=1762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:09.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.528+0000 7fa61c424640 1 --2- 172.21.15.16:0/2785359069 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa6140b0250 0x7fa61415b7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:09.534 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.528+0000 7fa61c424640 1 -- 172.21.15.16:0/2785359069 >> 172.21.15.16:0/2785359069 conn(0x7fa614069720 msgr2=0x7fa6140cccd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:46:09.534 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.528+0000 7fa61c424640 1 -- 172.21.15.16:0/2785359069 shutdown_connections 2023-05-22T15:46:09.534 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.528+0000 7fa61c424640 1 -- 172.21.15.16:0/2785359069 wait complete. 2023-05-22T15:46:09.549 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-05-22T15:46:09.658 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.652+0000 7fa87f08c640 1 Processor -- start 2023-05-22T15:46:09.659 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.652+0000 7fa87f08c640 1 -- start start 2023-05-22T15:46:09.660 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.656+0000 7fa87f08c640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fa8781531d0 0x7fa8780cd730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:09.660 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.656+0000 7fa87f08c640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fa8780cbbf0 0x7fa8780afb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:09.660 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.656+0000 7fa87f08c640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa8780cc020 0x7fa8780b0080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:09.660 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.656+0000 7fa87f08c640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa87815d330 con 0x7fa8781531d0 2023-05-22T15:46:09.660 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.656+0000 7fa87f08c640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa87815c4a0 con 0x7fa8780cbbf0 2023-05-22T15:46:09.661 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.656+0000 7fa87f08c640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa87815c6b0 con 0x7fa8780cc020 2023-05-22T15:46:09.661 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.656+0000 7fa87ce28640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fa8781531d0 0x7fa8780cd730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:09.661 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.656+0000 7fa87ce28640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fa8781531d0 0x7fa8780cd730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43382/0 (socket says 172.21.15.16:43382) 2023-05-22T15:46:09.661 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.656+0000 7fa87ce28640 1 -- 172.21.15.16:0/3755839509 learned_addr learned my addr 172.21.15.16:0/3755839509 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:46:09.661 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.656+0000 7fa87ce28640 1 -- 172.21.15.16:0/3755839509 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa8780cc020 msgr2=0x7fa8780b0080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:46:09.661 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.656+0000 7fa87ce28640 1 --2- 172.21.15.16:0/3755839509 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa8780cc020 0x7fa8780b0080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:09.661 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.656+0000 7fa87ce28640 1 -- 172.21.15.16:0/3755839509 >> v2:172.21.15.62:3300/0 conn(0x7fa8780cbbf0 msgr2=0x7fa8780afb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:46:09.662 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.656+0000 7fa87ce28640 1 --2- 172.21.15.16:0/3755839509 >> v2:172.21.15.62:3300/0 conn(0x7fa8780cbbf0 0x7fa8780afb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:09.662 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.656+0000 7fa87ce28640 1 -- 172.21.15.16:0/3755839509 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa87815c990 con 0x7fa8781531d0 2023-05-22T15:46:09.664 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.660+0000 7fa87ce28640 1 --2- 172.21.15.16:0/3755839509 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa8781531d0 0x7fa8780cd730 secure :-1 s=READY pgs=1763 cs=0 l=1 rev1=1 crypto rx=0x7fa8780b0a70 tx=0x7fa86001c9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7407a74cf4d48285 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:09.664 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.660+0000 7fa8777fe640 1 -- 172.21.15.16:0/3755839509 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8600a5850 con 0x7fa8781531d0 2023-05-22T15:46:09.664 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.660+0000 7fa8777fe640 1 -- 172.21.15.16:0/3755839509 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8600a59b0 con 0x7fa8781531d0 2023-05-22T15:46:09.665 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.660+0000 7fa8777fe640 1 -- 172.21.15.16:0/3755839509 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8600a5d40 con 0x7fa8781531d0 2023-05-22T15:46:09.665 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.660+0000 7fa87f08c640 1 -- 172.21.15.16:0/3755839509 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa8781531d0 msgr2=0x7fa8780cd730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:09.665 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.660+0000 7fa87f08c640 1 --2- 172.21.15.16:0/3755839509 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa8781531d0 0x7fa8780cd730 secure :-1 s=READY pgs=1763 cs=0 l=1 rev1=1 crypto rx=0x7fa8780b0a70 tx=0x7fa86001c9d0 comp rx=0 tx=0).stop 2023-05-22T15:46:09.666 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.660+0000 7fa87f08c640 1 -- 172.21.15.16:0/3755839509 shutdown_connections 2023-05-22T15:46:09.666 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.660+0000 7fa87f08c640 1 --2- 172.21.15.16:0/3755839509 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa8780cc020 0x7fa8780b0080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:09.666 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.660+0000 7fa87f08c640 1 --2- 172.21.15.16:0/3755839509 >> v2:172.21.15.62:3300/0 conn(0x7fa8780cbbf0 0x7fa8780afb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:09.666 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.660+0000 7fa87f08c640 1 --2- 172.21.15.16:0/3755839509 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa8781531d0 0x7fa8780cd730 unknown :-1 s=CLOSED pgs=1763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:09.666 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.660+0000 7fa87f08c640 1 -- 172.21.15.16:0/3755839509 >> 172.21.15.16:0/3755839509 conn(0x7fa878069720 msgr2=0x7fa878069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:46:09.666 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.660+0000 7fa87f08c640 1 -- 172.21.15.16:0/3755839509 shutdown_connections 2023-05-22T15:46:09.666 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.660+0000 7fa87f08c640 1 -- 172.21.15.16:0/3755839509 wait complete. 2023-05-22T15:46:09.668 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.660+0000 7fa87f08c640 1 Processor -- start 2023-05-22T15:46:09.668 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.660+0000 7fa87f08c640 1 -- start start 2023-05-22T15:46:09.668 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.664+0000 7fa87f08c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa8780cbbf0 0x7fa878148b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:09.668 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.664+0000 7fa87f08c640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa8780cc020 0x7fa8780aecb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:09.669 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.664+0000 7fa87f08c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa8781531d0 0x7fa87815a030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:09.669 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.664+0000 7fa87f08c640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa87814a6f0 con 0x7fa8780cbbf0 2023-05-22T15:46:09.669 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.664+0000 7fa87f08c640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa878149cf0 con 0x7fa8781531d0 2023-05-22T15:46:09.669 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.664+0000 7fa87f08c640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa878149f10 con 0x7fa8780cc020 2023-05-22T15:46:09.669 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.664+0000 7fa87ce28640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa8780cbbf0 0x7fa878148b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:09.669 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.664+0000 7fa87ce28640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa8780cbbf0 0x7fa878148b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43398/0 (socket says 172.21.15.16:43398) 2023-05-22T15:46:09.669 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.664+0000 7fa87ce28640 1 -- 172.21.15.16:0/2967606626 learned_addr learned my addr 172.21.15.16:0/2967606626 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:46:09.669 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.664+0000 7fa87ce28640 1 -- 172.21.15.16:0/2967606626 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa8780cc020 msgr2=0x7fa8780aecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:09.670 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.664+0000 7fa877fff640 1 --2- 172.21.15.16:0/2967606626 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa8780cc020 0x7fa8780aecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:09.670 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.664+0000 7fa87d629640 1 --2- 172.21.15.16:0/2967606626 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa8781531d0 0x7fa87815a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:09.670 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.664+0000 7fa87ce28640 1 --2- 172.21.15.16:0/2967606626 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa8780cc020 0x7fa8780aecb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:09.670 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.664+0000 7fa87ce28640 1 -- 172.21.15.16:0/2967606626 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa8781531d0 msgr2=0x7fa87815a030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:09.670 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.664+0000 7fa87ce28640 1 --2- 172.21.15.16:0/2967606626 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa8781531d0 0x7fa87815a030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:09.670 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.664+0000 7fa87ce28640 1 -- 172.21.15.16:0/2967606626 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa860009b30 con 0x7fa8780cbbf0 2023-05-22T15:46:09.670 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.664+0000 7fa87ce28640 1 --2- 172.21.15.16:0/2967606626 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa8780cbbf0 0x7fa878148b80 secure :-1 s=READY pgs=1764 cs=0 l=1 rev1=1 crypto rx=0x7fa86001d270 tx=0x7fa860019770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:09.670 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.664+0000 7fa8757fa640 1 -- 172.21.15.16:0/2967606626 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8600bd070 con 0x7fa8780cbbf0 2023-05-22T15:46:09.671 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.664+0000 7fa8757fa640 1 -- 172.21.15.16:0/2967606626 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8600b8470 con 0x7fa8780cbbf0 2023-05-22T15:46:09.671 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.664+0000 7fa8757fa640 1 -- 172.21.15.16:0/2967606626 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8600b8800 con 0x7fa8780cbbf0 2023-05-22T15:46:09.671 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.664+0000 7fa87f08c640 1 -- 172.21.15.16:0/2967606626 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8781608d0 con 0x7fa8780cbbf0 2023-05-22T15:46:09.671 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.664+0000 7fa87f08c640 1 -- 172.21.15.16:0/2967606626 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa878160740 con 0x7fa8780cbbf0 2023-05-22T15:46:09.672 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.664+0000 7fa87f08c640 1 -- 172.21.15.16:0/2967606626 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8780ad3b0 con 0x7fa8780cbbf0 2023-05-22T15:46:09.672 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.664+0000 7fa8757fa640 1 -- 172.21.15.16:0/2967606626 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 59) v1 ==== 84068+0+0 (secure 0 0 0) 0x7fa8600b8960 con 0x7fa8780cbbf0 2023-05-22T15:46:09.672 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.664+0000 7fa8757fa640 1 --2- 172.21.15.16:0/2967606626 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fa8580651d0 0x7fa858067680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:09.673 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.664+0000 7fa8757fa640 1 -- 172.21.15.16:0/2967606626 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(135..135 src has 1..135) v4 ==== 7126+0+0 (secure 0 0 0) 0x7fa86011e8e0 con 0x7fa8780cbbf0 2023-05-22T15:46:09.673 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.664+0000 7fa877fff640 1 --2- 172.21.15.16:0/2967606626 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fa8580651d0 0x7fa858067680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:09.673 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.668+0000 7fa877fff640 1 --2- 172.21.15.16:0/2967606626 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fa8580651d0 0x7fa858067680 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fa87815ba70 tx=0x7fa868008040 comp rx=0 tx=0).ready entity=mgr.8835 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:09.673 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.668+0000 7fa8757fa640 1 -- 172.21.15.16:0/2967606626 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fa8600efea0 con 0x7fa8780cbbf0 2023-05-22T15:46:09.830 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.824+0000 7fa87f08c640 1 -- 172.21.15.16:0/2967606626 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa8780cb750 con 0x7fa8780cbbf0 2023-05-22T15:46:09.830 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.824+0000 7fa8757fa640 1 -- 172.21.15.16:0/2967606626 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mgrmap(e 60) v1 ==== 84332+0+0 (secure 0 0 0) 0x7fa8600ec840 con 0x7fa8780cbbf0 2023-05-22T15:46:09.834 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.832+0000 7fa8757fa640 1 -- 172.21.15.16:0/2967606626 <== mon.0 v2:172.21.15.16:3300/0 8 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v135) v1 ==== 74+0+19091 (secure 0 0 0) 0x7fa8600efc70 con 0x7fa8780cbbf0 2023-05-22T15:46:09.835 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:46:09.835 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":135,"fsid":"b2719834-7847-4d98-90a6-f65d57f85c49","created":"2023-05-22T15:26:25.259314+0000","modified":"2023-05-22T15:46:08.343543+0000","last_up_change":"2023-05-22T15:26:36.448524+0000","last_in_change":"2023-05-22T15:26:27.675668+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-05-22T15:26:38.019781+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-05-22T15:26:43.570514+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2023-05-22T15:45:03.718869+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"131","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2023-05-22T15:45:04.168135+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"135","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":135,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9e7ff854-b25f-4cbb-a5c2-1799d9c53d2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":37021},{"type":"v1","addr":"172.21.15.16:6825","nonce":37021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":37021},{"type":"v1","addr":"172.21.15.16:6827","nonce":37021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":37021},{"type":"v1","addr":"172.21.15.16:6831","nonce":37021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":37021},{"type":"v1","addr":"172.21.15.16:6829","nonce":37021}]},"public_addr":"172.21.15.16:6825/37021","cluster_addr":"172.21.15.16:6827/37021","heartbeat_back_addr":"172.21.15.16:6831/37021","heartbeat_front_addr":"172.21.15.16:6829/37021","state":["exists","up"]},{"osd":1,"uuid":"4702fbc9-a927-4565-aca5-41dc69cc8264","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":37008},{"type":"v1","addr":"172.21.15.16:6801","nonce":37008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":37008},{"type":"v1","addr":"172.21.15.16:6803","nonce":37008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":37008},{"type":"v1","addr":"172.21.15.16:6807","nonce":37008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":37008},{"type":"v1","addr":"172.21.15.16:6805","nonce":37008}]},"public_addr":"172.21.15.16:6801/37008","cluster_addr":"172.21.15.16:6803/37008","heartbeat_back_addr":"172.21.15.16:6807/37008","heartbeat_front_addr":"172.21.15.16:6805/37008","state":["exists","up"]},{"osd":2,"uuid":"1e7bba46-4eba-47ef-bb81-5e2d371638ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":37009},{"type":"v1","addr":"172.21.15.16:6809","nonce":37009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":37009},{"type":"v1","addr":"172.21.15.16:6811","nonce":37009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":37009},{"type":"v1","addr":"172.21.15.16:6815","nonce":37009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":37009},{"type":"v1","addr":"172.21.15.16:6813","nonce":37009}]},"public_addr":"172.21.15.16:6809/37009","cluster_addr":"172.21.15.16:6811/37009","heartbeat_back_addr":"172.21.15.16:6815/37009","heartbeat_front_addr":"172.21.15.16:6813/37009","state":["exists","up"]},{"osd":3,"uuid":"383cacc0-5a88-4904-990e-498ca59ef48a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":37020},{"type":"v1","addr":"172.21.15.16:6817","nonce":37020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":37020},{"type":"v1","addr":"172.21.15.16:6819","nonce":37020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":37020},{"type":"v1","addr":"172.21.15.16:6823","nonce":37020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":37020},{"type":"v1","addr":"172.21.15.16:6821","nonce":37020}]},"public_addr":"172.21.15.16:6817/37020","cluster_addr":"172.21.15.16:6819/37020","heartbeat_back_addr":"172.21.15.16:6823/37020","heartbeat_front_addr":"172.21.15.16:6821/37020","state":["exists","up"]},{"osd":4,"uuid":"970500ef-95b7-42c1-b080-eed4fae7013d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":37353},{"type":"v1","addr":"172.21.15.62:6817","nonce":37353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6819","nonce":37353},{"type":"v1","addr":"172.21.15.62:6822","nonce":37353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37353},{"type":"v1","addr":"172.21.15.62:6831","nonce":37353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37353},{"type":"v1","addr":"172.21.15.62:6826","nonce":37353}]},"public_addr":"172.21.15.62:6817/37353","cluster_addr":"172.21.15.62:6822/37353","heartbeat_back_addr":"172.21.15.62:6831/37353","heartbeat_front_addr":"172.21.15.62:6826/37353","state":["exists","up"]},{"osd":5,"uuid":"022d67db-d4d8-445b-b1b9-8b446a00f5c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":37352},{"type":"v1","addr":"172.21.15.62:6801","nonce":37352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":37352},{"type":"v1","addr":"172.21.15.62:6803","nonce":37352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":37352},{"type":"v1","addr":"172.21.15.62:6807","nonce":37352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":37352},{"type":"v1","addr":"172.21.15.62:6805","nonce":37352}]},"public_addr":"172.21.15.62:6801/37352","cluster_addr":"172.21.15.62:6803/37352","heartbeat_back_addr":"172.21.15.62:6807/37352","heartbeat_front_addr":"172.21.15.62:6805/37352","state":["exists","up"]},{"osd":6,"uuid":"42bd372b-6688-4a1b-8407-e16e8c266e33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":37354},{"type":"v1","addr":"172.21.15.62:6820","nonce":37354}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":37354},{"type":"v1","addr":"172.21.15.62:6823","nonce":37354}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6829","nonce":37354},{"type":"v1","addr":"172.21.15.62:6830","nonce":37354}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6825","nonce":37354},{"type":"v1","addr":"172.21.15.62:6827","nonce":37354}]},"public_addr":"172.21.15.62:6820/37354","cluster_addr":"172.21.15.62:6823/37354","heartbeat_back_addr":"172.21.15.62:6830/37354","heartbeat_front_addr":"172.21.15.62:6827/37354","state":["exists","up"]},{"osd":7,"uuid":"ac70b2a3-f41b-4bf2-b227-7f2cbb55426d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37355},{"type":"v1","addr":"172.21.15.62:6809","nonce":37355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37355},{"type":"v1","addr":"172.21.15.62:6811","nonce":37355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":37355},{"type":"v1","addr":"172.21.15.62:6815","nonce":37355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":37355},{"type":"v1","addr":"172.21.15.62:6813","nonce":37355}]},"public_addr":"172.21.15.62:6809/37355","cluster_addr":"172.21.15.62:6811/37355","heartbeat_back_addr":"172.21.15.62:6815/37355","heartbeat_front_addr":"172.21.15.62:6813/37355","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.368896+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.107273+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.276888+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.287502+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.124791+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.251937+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.255679+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.232797+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:0/79646455":"2023-05-23T15:46:07.335649+0000","172.21.15.16:0/2657546211":"2023-05-23T15:46:07.335649+0000","172.21.15.16:0/3313824983":"2023-05-23T15:46:07.335649+0000","172.21.15.16:0/3149639262":"2023-05-23T15:46:07.335649+0000","172.21.15.16:0/2513263958":"2023-05-23T15:46:07.335649+0000","172.21.15.16:0/2722517029":"2023-05-23T15:46:07.335649+0000","172.21.15.16:0/736449025":"2023-05-23T15:46:07.335649+0000","172.21.15.16:0/3623988153":"2023-05-23T15:46:07.335649+0000","172.21.15.16:0/1950855917":"2023-05-23T15:45:17.648374+0000","172.21.15.16:0/1889818530":"2023-05-23T15:45:17.648374+0000","172.21.15.16:6833/36921":"2023-05-23T15:45:17.648374+0000","172.21.15.16:0/4160466619":"2023-05-23T15:45:17.648374+0000","172.21.15.16:0/55601757":"2023-05-23T15:45:17.648374+0000","172.21.15.16:6832/36921":"2023-05-23T15:45:17.648374+0000","172.21.15.16:0/4127127856":"2023-05-23T15:45:17.648374+0000","172.21.15.16:0/2333289485":"2023-05-23T15:45:17.648374+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":17,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[{"pool":17,"snaps":[{"begin":2,"length":2}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-05-22T15:46:09.837 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.832+0000 7fa87f08c640 1 -- 172.21.15.16:0/2967606626 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fa8580651d0 msgr2=0x7fa858067680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:09.837 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.832+0000 7fa87f08c640 1 --2- 172.21.15.16:0/2967606626 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fa8580651d0 0x7fa858067680 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fa87815ba70 tx=0x7fa868008040 comp rx=0 tx=0).stop 2023-05-22T15:46:09.837 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.832+0000 7fa87f08c640 1 -- 172.21.15.16:0/2967606626 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa8780cbbf0 msgr2=0x7fa878148b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:09.837 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.832+0000 7fa87f08c640 1 --2- 172.21.15.16:0/2967606626 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa8780cbbf0 0x7fa878148b80 secure :-1 s=READY pgs=1764 cs=0 l=1 rev1=1 crypto rx=0x7fa86001d270 tx=0x7fa860019770 comp rx=0 tx=0).stop 2023-05-22T15:46:09.838 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.832+0000 7fa87f08c640 1 -- 172.21.15.16:0/2967606626 shutdown_connections 2023-05-22T15:46:09.838 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.832+0000 7fa87f08c640 1 --2- 172.21.15.16:0/2967606626 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa8781531d0 0x7fa87815a030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:09.838 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.832+0000 7fa87f08c640 1 --2- 172.21.15.16:0/2967606626 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fa8580651d0 0x7fa858067680 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:09.838 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.832+0000 7fa87f08c640 1 --2- 172.21.15.16:0/2967606626 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa8780cc020 0x7fa8780aecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:09.839 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.832+0000 7fa87f08c640 1 --2- 172.21.15.16:0/2967606626 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa8780cbbf0 0x7fa878148b80 unknown :-1 s=CLOSED pgs=1764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:09.839 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.832+0000 7fa87f08c640 1 -- 172.21.15.16:0/2967606626 >> 172.21.15.16:0/2967606626 conn(0x7fa878069720 msgr2=0x7fa87806f530 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:46:09.839 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.832+0000 7fa87f08c640 1 -- 172.21.15.16:0/2967606626 shutdown_connections 2023-05-22T15:46:09.839 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.832+0000 7fa87f08c640 1 -- 172.21.15.16:0/2967606626 wait complete. 2023-05-22T15:46:09.854 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-05-22T15:46:09.977 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.972+0000 7fe4365d5640 1 Processor -- start 2023-05-22T15:46:09.977 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.972+0000 7fe4365d5640 1 -- start start 2023-05-22T15:46:09.978 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.972+0000 7fe4365d5640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe4300d1c70 0x7fe4300ce6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:09.979 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.972+0000 7fe4365d5640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe4300cec00 0x7fe4300b3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:09.979 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.972+0000 7fe4365d5640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4300b4de0 con 0x7fe430159740 2023-05-22T15:46:09.979 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.972+0000 7fe4365d5640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4300d21e0 con 0x7fe4300cec00 2023-05-22T15:46:09.979 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.972+0000 7fe4365d5640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4300ccbf0 con 0x7fe4300d1c70 2023-05-22T15:46:09.979 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.972+0000 7fe434b72640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe4300cec00 0x7fe4300b3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:09.979 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.972+0000 7fe434b72640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe4300cec00 0x7fe4300b3ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43412/0 (socket says 172.21.15.16:43412) 2023-05-22T15:46:09.980 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.972+0000 7fe434b72640 1 -- 172.21.15.16:0/2791240235 learned_addr learned my addr 172.21.15.16:0/2791240235 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:46:09.980 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.972+0000 7fe42f7fe640 1 --2- 172.21.15.16:0/2791240235 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe4300d1c70 0x7fe4300ce6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:09.980 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.972+0000 7fe434b72640 1 -- 172.21.15.16:0/2791240235 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe4300d1c70 msgr2=0x7fe4300ce6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:46:09.980 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.972+0000 7fe434b72640 1 --2- 172.21.15.16:0/2791240235 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe4300d1c70 0x7fe4300ce6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:09.980 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.972+0000 7fe434b72640 1 -- 172.21.15.16:0/2791240235 >> v1:172.21.15.62:6789/0 conn(0x7fe430159740 legacy=0x7fe43015cc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:09.980 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.972+0000 7fe434b72640 1 -- 172.21.15.16:0/2791240235 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4300cce90 con 0x7fe4300cec00 2023-05-22T15:46:09.980 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.972+0000 7fe42f7fe640 1 --2- 172.21.15.16:0/2791240235 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe4300d1c70 0x7fe4300ce6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T15:46:09.981 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.976+0000 7fe434b72640 1 --2- 172.21.15.16:0/2791240235 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe4300cec00 0x7fe4300b3ef0 secure :-1 s=READY pgs=1765 cs=0 l=1 rev1=1 crypto rx=0x7fe420009870 tx=0x7fe42001c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf1e30d1b1b863ef server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:09.982 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.976+0000 7fe42effd640 1 -- 172.21.15.16:0/2791240235 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4200ad640 con 0x7fe4300cec00 2023-05-22T15:46:09.982 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.976+0000 7fe42effd640 1 -- 172.21.15.16:0/2791240235 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4200ad7a0 con 0x7fe4300cec00 2023-05-22T15:46:09.982 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.976+0000 7fe42effd640 1 -- 172.21.15.16:0/2791240235 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4200adb30 con 0x7fe4300cec00 2023-05-22T15:46:09.982 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.976+0000 7fe4365d5640 1 -- 172.21.15.16:0/2791240235 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe4300cec00 msgr2=0x7fe4300b3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:09.982 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.976+0000 7fe4365d5640 1 --2- 172.21.15.16:0/2791240235 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe4300cec00 0x7fe4300b3ef0 secure :-1 s=READY pgs=1765 cs=0 l=1 rev1=1 crypto rx=0x7fe420009870 tx=0x7fe42001c480 comp rx=0 tx=0).stop 2023-05-22T15:46:09.982 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.976+0000 7fe4365d5640 1 -- 172.21.15.16:0/2791240235 shutdown_connections 2023-05-22T15:46:09.983 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.976+0000 7fe4365d5640 1 --2- 172.21.15.16:0/2791240235 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe4300cec00 0x7fe4300b3ef0 unknown :-1 s=CLOSED pgs=1765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:09.983 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.976+0000 7fe4365d5640 1 --2- 172.21.15.16:0/2791240235 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe4300d1c70 0x7fe4300ce6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:09.983 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.976+0000 7fe4365d5640 1 -- 172.21.15.16:0/2791240235 >> 172.21.15.16:0/2791240235 conn(0x7fe43006c2f0 msgr2=0x7fe430070e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:46:09.983 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.976+0000 7fe4365d5640 1 -- 172.21.15.16:0/2791240235 shutdown_connections 2023-05-22T15:46:09.983 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.976+0000 7fe4365d5640 1 -- 172.21.15.16:0/2791240235 wait complete. 2023-05-22T15:46:09.985 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.980+0000 7fe4365d5640 1 Processor -- start 2023-05-22T15:46:09.985 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.980+0000 7fe4365d5640 1 -- start start 2023-05-22T15:46:09.985 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.980+0000 7fe4365d5640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe4300cec00 0x7fe4300cc340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:09.985 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.980+0000 7fe4365d5640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe4300d1c70 0x7fe4300c81f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:09.986 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.980+0000 7fe4365d5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe430159740 0x7fe430153b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:09.986 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.980+0000 7fe4365d5640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4300cc880 con 0x7fe430159740 2023-05-22T15:46:09.986 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.980+0000 7fe4365d5640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4300cb050 con 0x7fe4300cec00 2023-05-22T15:46:09.986 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.980+0000 7fe4365d5640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4300cb2a0 con 0x7fe4300d1c70 2023-05-22T15:46:09.986 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.980+0000 7fe434b72640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe430159740 0x7fe430153b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:09.986 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.980+0000 7fe434b72640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe430159740 0x7fe430153b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43416/0 (socket says 172.21.15.16:43416) 2023-05-22T15:46:09.986 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.980+0000 7fe434b72640 1 -- 172.21.15.16:0/2060258102 learned_addr learned my addr 172.21.15.16:0/2060258102 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:46:09.986 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.980+0000 7fe42f7fe640 1 --2- 172.21.15.16:0/2060258102 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe4300d1c70 0x7fe4300c81f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:09.987 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.980+0000 7fe42ffff640 1 --2- 172.21.15.16:0/2060258102 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe4300cec00 0x7fe4300cc340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:09.987 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.980+0000 7fe434b72640 1 -- 172.21.15.16:0/2060258102 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe4300d1c70 msgr2=0x7fe4300c81f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:09.987 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.980+0000 7fe434b72640 1 --2- 172.21.15.16:0/2060258102 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe4300d1c70 0x7fe4300c81f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:09.987 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.980+0000 7fe434b72640 1 -- 172.21.15.16:0/2060258102 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe4300cec00 msgr2=0x7fe4300cc340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:09.987 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.980+0000 7fe434b72640 1 --2- 172.21.15.16:0/2060258102 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe4300cec00 0x7fe4300cc340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:09.987 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.980+0000 7fe434b72640 1 -- 172.21.15.16:0/2060258102 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4200094d0 con 0x7fe430159740 2023-05-22T15:46:09.987 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.980+0000 7fe434b72640 1 --2- 172.21.15.16:0/2060258102 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe430159740 0x7fe430153b50 secure :-1 s=READY pgs=1766 cs=0 l=1 rev1=1 crypto rx=0x7fe42001a0b0 tx=0x7fe420019220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:09.988 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.980+0000 7fe42cff9640 1 -- 172.21.15.16:0/2060258102 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4200b8680 con 0x7fe430159740 2023-05-22T15:46:09.988 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.980+0000 7fe42cff9640 1 -- 172.21.15.16:0/2060258102 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4200a6040 con 0x7fe430159740 2023-05-22T15:46:09.988 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.980+0000 7fe42cff9640 1 -- 172.21.15.16:0/2060258102 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4200b89e0 con 0x7fe430159740 2023-05-22T15:46:09.988 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.980+0000 7fe4365d5640 1 -- 172.21.15.16:0/2060258102 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4300d0e10 con 0x7fe430159740 2023-05-22T15:46:09.988 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.980+0000 7fe4365d5640 1 -- 172.21.15.16:0/2060258102 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe430159b10 con 0x7fe430159740 2023-05-22T15:46:09.989 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.980+0000 7fe4365d5640 1 -- 172.21.15.16:0/2060258102 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4300d0fa0 con 0x7fe430159740 2023-05-22T15:46:09.989 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.980+0000 7fe42cff9640 1 -- 172.21.15.16:0/2060258102 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 60) v1 ==== 84332+0+0 (secure 0 0 0) 0x7fe4200b8b40 con 0x7fe430159740 2023-05-22T15:46:09.989 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.980+0000 7fe42cff9640 1 --2- 172.21.15.16:0/2060258102 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fe4100654a0 0x7fe410067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:09.989 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.980+0000 7fe42cff9640 1 -- 172.21.15.16:0/2060258102 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(135..135 src has 1..135) v4 ==== 7126+0+0 (secure 0 0 0) 0x7fe42011d9c0 con 0x7fe430159740 2023-05-22T15:46:09.989 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.984+0000 7fe42ffff640 1 --2- 172.21.15.16:0/2060258102 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fe4100654a0 0x7fe410067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:09.990 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.984+0000 7fe42ffff640 1 --2- 172.21.15.16:0/2060258102 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fe4100654a0 0x7fe410067950 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fe4280012f0 tx=0x7fe428009040 comp rx=0 tx=0).ready entity=mgr.8835 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:09.991 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:09.988+0000 7fe42cff9640 1 -- 172.21.15.16:0/2060258102 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fe4200eb110 con 0x7fe430159740 2023-05-22T15:46:10.178 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:10.172+0000 7fe4365d5640 1 -- 172.21.15.16:0/2060258102 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fe43004dbe0 con 0x7fe430159740 2023-05-22T15:46:10.733 INFO:tasks.ceph.mds.a.smithi016.stderr:2023-05-22T15:46:10.728+0000 7fa0f7612640 -1 mds.0.journaler.mdlog(rw) _finish_flush got (108) Cannot send after transport endpoint shutdown 2023-05-22T15:46:10.733 INFO:tasks.ceph.mds.a.smithi016.stderr:2023-05-22T15:46:10.728+0000 7fa0f7612640 -1 mds.0.journaler.mdlog(rw) handle_write_error (108) Cannot send after transport endpoint shutdown 2023-05-22T15:46:10.733 INFO:tasks.ceph.mds.a.smithi016.stderr:2023-05-22T15:46:10.728+0000 7fa0f7612640 -1 MDSIOContextBase: failed with -108, restarting... 2023-05-22T15:46:10.770 INFO:tasks.ceph.mds.a.smithi016.stderr: -4> 2023-05-22T15:46:10.728+0000 7fa0f7612640 -1 mds.0.journaler.mdlog(rw) _finish_flush got (108) Cannot send after transport endpoint shutdown 2023-05-22T15:46:10.771 INFO:tasks.ceph.mds.a.smithi016.stderr: -3> 2023-05-22T15:46:10.728+0000 7fa0f7612640 -1 mds.0.journaler.mdlog(rw) handle_write_error (108) Cannot send after transport endpoint shutdown 2023-05-22T15:46:10.771 INFO:tasks.ceph.mds.a.smithi016.stderr: -1> 2023-05-22T15:46:10.728+0000 7fa0f7612640 -1 MDSIOContextBase: failed with -108, restarting... 2023-05-22T15:46:10.823 INFO:tasks.ceph.mds.a.smithi016.stdout:starting mds.a at 2023-05-22T15:46:10.836 INFO:tasks.ceph.mon.a.smithi016.stderr:2023-05-22T15:46:10.832+0000 7f95dd48a640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-05-22T15:46:10.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:10.848+0000 7fe42cff9640 1 -- 172.21.15.16:0/2060258102 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v51) v1 ==== 161+0+0 (secure 0 0 0) 0x7fe4200b2090 con 0x7fe430159740 2023-05-22T15:46:10.852 INFO:teuthology.orchestra.run.smithi016.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-05-22T15:46:10.855 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:10.848+0000 7fe4365d5640 1 -- 172.21.15.16:0/2060258102 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fe4100654a0 msgr2=0x7fe410067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:10.855 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:10.848+0000 7fe4365d5640 1 --2- 172.21.15.16:0/2060258102 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fe4100654a0 0x7fe410067950 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fe4280012f0 tx=0x7fe428009040 comp rx=0 tx=0).stop 2023-05-22T15:46:10.855 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:10.848+0000 7fe4365d5640 1 -- 172.21.15.16:0/2060258102 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe430159740 msgr2=0x7fe430153b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:10.855 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:10.848+0000 7fe4365d5640 1 --2- 172.21.15.16:0/2060258102 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe430159740 0x7fe430153b50 secure :-1 s=READY pgs=1766 cs=0 l=1 rev1=1 crypto rx=0x7fe42001a0b0 tx=0x7fe420019220 comp rx=0 tx=0).stop 2023-05-22T15:46:10.855 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:10.848+0000 7fe4365d5640 1 -- 172.21.15.16:0/2060258102 shutdown_connections 2023-05-22T15:46:10.855 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:10.848+0000 7fe4365d5640 1 --2- 172.21.15.16:0/2060258102 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe430159740 0x7fe430153b50 unknown :-1 s=CLOSED pgs=1766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:10.856 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:10.848+0000 7fe4365d5640 1 --2- 172.21.15.16:0/2060258102 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fe4100654a0 0x7fe410067950 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:10.856 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:10.848+0000 7fe4365d5640 1 --2- 172.21.15.16:0/2060258102 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe4300d1c70 0x7fe4300c81f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:10.856 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:10.848+0000 7fe4365d5640 1 --2- 172.21.15.16:0/2060258102 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe4300cec00 0x7fe4300cc340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:10.856 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:10.848+0000 7fe4365d5640 1 -- 172.21.15.16:0/2060258102 >> 172.21.15.16:0/2060258102 conn(0x7fe43006c2f0 msgr2=0x7fe4300b1f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:46:10.856 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:10.852+0000 7fe4365d5640 1 -- 172.21.15.16:0/2060258102 shutdown_connections 2023-05-22T15:46:10.856 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:10.852+0000 7fe4365d5640 1 -- 172.21.15.16:0/2060258102 wait complete. 2023-05-22T15:46:10.876 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-05-22T15:46:11.005 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:11.000+0000 7f19c9c80640 1 Processor -- start 2023-05-22T15:46:11.006 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:11.000+0000 7f19c9c80640 1 -- start start 2023-05-22T15:46:11.006 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:11.000+0000 7f19c9c80640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f19c40ade80 0x7f19c40cc400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:11.007 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:11.000+0000 7f19c9c80640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f19c415c350 0x7f19c415c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:11.007 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:11.000+0000 7f19c9c80640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19c40c95f0 con 0x7f19c4158ca0 2023-05-22T15:46:11.007 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:11.000+0000 7f19c9c80640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19c40ca270 con 0x7f19c415c350 2023-05-22T15:46:11.007 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:11.000+0000 7f19c9c80640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19c40ca690 con 0x7f19c40ade80 2023-05-22T15:46:11.007 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:11.004+0000 7f19c2ffd640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f19c415c350 0x7f19c415c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:11.007 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:11.004+0000 7f19c2ffd640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f19c415c350 0x7f19c415c7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43446/0 (socket says 172.21.15.16:43446) 2023-05-22T15:46:11.007 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:11.004+0000 7f19c2ffd640 1 -- 172.21.15.16:0/1839362605 learned_addr learned my addr 172.21.15.16:0/1839362605 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:46:11.008 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:11.004+0000 7f19c3fff640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f19c4158ca0 0x7f19c41590e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:59172/0 (socket says 172.21.15.16:59172) 2023-05-22T15:46:11.008 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:11.004+0000 7f19c37fe640 1 --2- 172.21.15.16:0/1839362605 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f19c40ade80 0x7f19c40cc400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:11.008 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:11.004+0000 7f19c2ffd640 1 -- 172.21.15.16:0/1839362605 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f19c40ade80 msgr2=0x7f19c40cc400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:46:11.008 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:11.004+0000 7f19c2ffd640 1 --2- 172.21.15.16:0/1839362605 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f19c40ade80 0x7f19c40cc400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:11.009 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:11.004+0000 7f19c2ffd640 1 -- 172.21.15.16:0/1839362605 >> v1:172.21.15.16:6789/0 conn(0x7f19c4158ca0 legacy=0x7f19c41590e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:11.009 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:11.004+0000 7f19c2ffd640 1 -- 172.21.15.16:0/1839362605 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19c40c98d0 con 0x7f19c415c350 2023-05-22T15:46:11.010 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:11.004+0000 7f19c2ffd640 1 --2- 172.21.15.16:0/1839362605 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f19c415c350 0x7f19c415c7c0 secure :-1 s=READY pgs=1771 cs=0 l=1 rev1=1 crypto rx=0x7f19b80098a0 tx=0x7f19b801c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=576ec320f3dfaad2 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:11.010 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:11.004+0000 7f19c27fc640 1 -- 172.21.15.16:0/1839362605 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19b80ab600 con 0x7f19c415c350 2023-05-22T15:46:11.010 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:11.004+0000 7f19c27fc640 1 -- 172.21.15.16:0/1839362605 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19b80a7050 con 0x7f19c415c350 2023-05-22T15:46:11.010 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:11.004+0000 7f19c27fc640 1 -- 172.21.15.16:0/1839362605 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19b80ab980 con 0x7f19c415c350 2023-05-22T15:46:11.010 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:11.004+0000 7f19c9c80640 1 -- 172.21.15.16:0/1839362605 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f19c415c350 msgr2=0x7f19c415c7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:11.010 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:11.004+0000 7f19c9c80640 1 --2- 172.21.15.16:0/1839362605 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f19c415c350 0x7f19c415c7c0 secure :-1 s=READY pgs=1771 cs=0 l=1 rev1=1 crypto rx=0x7f19b80098a0 tx=0x7f19b801c490 comp rx=0 tx=0).stop 2023-05-22T15:46:11.011 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:11.004+0000 7f19c9c80640 1 -- 172.21.15.16:0/1839362605 shutdown_connections 2023-05-22T15:46:11.011 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:11.004+0000 7f19c9c80640 1 --2- 172.21.15.16:0/1839362605 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f19c415c350 0x7f19c415c7c0 unknown :-1 s=CLOSED pgs=1771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:11.011 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:11.004+0000 7f19c9c80640 1 --2- 172.21.15.16:0/1839362605 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f19c40ade80 0x7f19c40cc400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:11.011 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:11.004+0000 7f19c9c80640 1 -- 172.21.15.16:0/1839362605 >> 172.21.15.16:0/1839362605 conn(0x7f19c40b1110 msgr2=0x7f19c40ce830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:46:11.011 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:11.004+0000 7f19c9c80640 1 -- 172.21.15.16:0/1839362605 shutdown_connections 2023-05-22T15:46:11.011 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:11.004+0000 7f19c9c80640 1 -- 172.21.15.16:0/1839362605 wait complete. 2023-05-22T15:46:11.013 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:11.008+0000 7f19c9c80640 1 Processor -- start 2023-05-22T15:46:11.013 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:11.008+0000 7f19c9c80640 1 -- start start 2023-05-22T15:46:11.013 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:11.008+0000 7f19c9c80640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f19c40ade80 0x7f19c414a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:11.013 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:11.008+0000 7f19c9c80640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f19c4158ca0 0x7f19c415a540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:11.014 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:11.008+0000 7f19c9c80640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f19c415c350 0x7f19c414a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:11.014 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:11.008+0000 7f19c9c80640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19c414a830 con 0x7f19c415c350 2023-05-22T15:46:11.014 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:11.008+0000 7f19c9c80640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19c414b9a0 con 0x7f19c40ade80 2023-05-22T15:46:11.014 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:11.008+0000 7f19c9c80640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19c414b3e0 con 0x7f19c4158ca0 2023-05-22T15:46:11.014 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:11.008+0000 7f19c3fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f19c415c350 0x7f19c414a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:11.014 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:11.008+0000 7f19c3fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f19c415c350 0x7f19c414a990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43448/0 (socket says 172.21.15.16:43448) 2023-05-22T15:46:11.014 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:11.008+0000 7f19c3fff640 1 -- 172.21.15.16:0/967314261 learned_addr learned my addr 172.21.15.16:0/967314261 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:46:11.014 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:11.008+0000 7f19c2ffd640 1 --2- 172.21.15.16:0/967314261 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f19c4158ca0 0x7f19c415a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:11.015 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:11.008+0000 7f19c37fe640 1 --2- 172.21.15.16:0/967314261 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f19c40ade80 0x7f19c414a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:11.015 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:11.008+0000 7f19c3fff640 1 -- 172.21.15.16:0/967314261 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f19c4158ca0 msgr2=0x7f19c415a540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:11.015 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:11.008+0000 7f19c3fff640 1 --2- 172.21.15.16:0/967314261 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f19c4158ca0 0x7f19c415a540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:11.015 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:11.008+0000 7f19c3fff640 1 -- 172.21.15.16:0/967314261 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f19c40ade80 msgr2=0x7f19c414a2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:11.015 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:11.008+0000 7f19c3fff640 1 --2- 172.21.15.16:0/967314261 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f19c40ade80 0x7f19c414a2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:11.015 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:11.008+0000 7f19c3fff640 1 -- 172.21.15.16:0/967314261 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19b80094d0 con 0x7f19c415c350 2023-05-22T15:46:11.015 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:11.008+0000 7f19c37fe640 1 --2- 172.21.15.16:0/967314261 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f19c40ade80 0x7f19c414a2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:46:11.016 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:11.008+0000 7f19c3fff640 1 --2- 172.21.15.16:0/967314261 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f19c415c350 0x7f19c414a990 secure :-1 s=READY pgs=1772 cs=0 l=1 rev1=1 crypto rx=0x7f19bc00c7a0 tx=0x7f19bc00cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:11.016 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:11.008+0000 7f19abfff640 1 -- 172.21.15.16:0/967314261 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19bc00a640 con 0x7f19c415c350 2023-05-22T15:46:11.016 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:11.008+0000 7f19abfff640 1 -- 172.21.15.16:0/967314261 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19bc00a7a0 con 0x7f19c415c350 2023-05-22T15:46:11.016 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:11.008+0000 7f19c9c80640 1 -- 172.21.15.16:0/967314261 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19c415d090 con 0x7f19c415c350 2023-05-22T15:46:11.016 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:11.008+0000 7f19abfff640 1 -- 172.21.15.16:0/967314261 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19bc00ab10 con 0x7f19c415c350 2023-05-22T15:46:11.016 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:11.008+0000 7f19c9c80640 1 -- 172.21.15.16:0/967314261 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19c40cc940 con 0x7f19c415c350 2023-05-22T15:46:11.017 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:11.008+0000 7f19c9c80640 1 -- 172.21.15.16:0/967314261 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19c40c9210 con 0x7f19c415c350 2023-05-22T15:46:11.017 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:11.008+0000 7f19abfff640 1 -- 172.21.15.16:0/967314261 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 60) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f19bc00ac70 con 0x7f19c415c350 2023-05-22T15:46:11.017 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:11.012+0000 7f19abfff640 1 --2- 172.21.15.16:0/967314261 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f19b00654a0 0x7f19b0067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:11.017 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:11.012+0000 7f19abfff640 1 -- 172.21.15.16:0/967314261 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(136..136 src has 1..136) v4 ==== 7184+0+0 (secure 0 0 0) 0x7f19bc082530 con 0x7f19c415c350 2023-05-22T15:46:11.018 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:11.012+0000 7f19c37fe640 1 --2- 172.21.15.16:0/967314261 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f19b00654a0 0x7f19b0067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:11.018 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:11.012+0000 7f19c37fe640 1 --2- 172.21.15.16:0/967314261 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f19b00654a0 0x7f19b0067950 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f19b40097a0 tx=0x7f19b4008040 comp rx=0 tx=0).ready entity=mgr.8835 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:11.019 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:11.016+0000 7f19abfff640 1 -- 172.21.15.16:0/967314261 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f19bc04a620 con 0x7f19c415c350 2023-05-22T15:46:11.199 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:11.192+0000 7f19c9c80640 1 -- 172.21.15.16:0/967314261 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f19c404dbe0 con 0x7f19c415c350 2023-05-22T15:46:11.858 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:11.852+0000 7f19abfff640 1 -- 172.21.15.16:0/967314261 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v52) v1 ==== 104+0+0 (secure 0 0 0) 0x7f19bc04fd80 con 0x7f19c415c350 2023-05-22T15:46:11.861 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:11.856+0000 7f19c9c80640 1 -- 172.21.15.16:0/967314261 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f19b00654a0 msgr2=0x7f19b0067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:11.861 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:11.856+0000 7f19c9c80640 1 --2- 172.21.15.16:0/967314261 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f19b00654a0 0x7f19b0067950 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f19b40097a0 tx=0x7f19b4008040 comp rx=0 tx=0).stop 2023-05-22T15:46:11.861 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:11.856+0000 7f19c9c80640 1 -- 172.21.15.16:0/967314261 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f19c415c350 msgr2=0x7f19c414a990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:11.862 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:11.856+0000 7f19c9c80640 1 --2- 172.21.15.16:0/967314261 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f19c415c350 0x7f19c414a990 secure :-1 s=READY pgs=1772 cs=0 l=1 rev1=1 crypto rx=0x7f19bc00c7a0 tx=0x7f19bc00cc60 comp rx=0 tx=0).stop 2023-05-22T15:46:11.862 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:11.856+0000 7f19c9c80640 1 -- 172.21.15.16:0/967314261 shutdown_connections 2023-05-22T15:46:11.862 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:11.856+0000 7f19c9c80640 1 --2- 172.21.15.16:0/967314261 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f19c415c350 0x7f19c414a990 unknown :-1 s=CLOSED pgs=1772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:11.862 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:11.856+0000 7f19c9c80640 1 --2- 172.21.15.16:0/967314261 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f19b00654a0 0x7f19b0067950 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:11.862 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:11.856+0000 7f19c9c80640 1 --2- 172.21.15.16:0/967314261 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f19c4158ca0 0x7f19c415a540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:11.862 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:11.856+0000 7f19c9c80640 1 --2- 172.21.15.16:0/967314261 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f19c40ade80 0x7f19c414a2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:11.862 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:11.856+0000 7f19c9c80640 1 -- 172.21.15.16:0/967314261 >> 172.21.15.16:0/967314261 conn(0x7f19c40b1110 msgr2=0x7f19c406cef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:46:11.863 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:11.856+0000 7f19c9c80640 1 -- 172.21.15.16:0/967314261 shutdown_connections 2023-05-22T15:46:11.863 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:11.856+0000 7f19c9c80640 1 -- 172.21.15.16:0/967314261 wait complete. 2023-05-22T15:46:11.885 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-05-22T15:46:12.010 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:12.004+0000 7f5b6494b640 1 Processor -- start 2023-05-22T15:46:12.010 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:12.004+0000 7f5b6494b640 1 -- start start 2023-05-22T15:46:12.011 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:12.008+0000 7f5b6494b640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f5b600cf670 0x7f5b601601d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:12.011 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:12.008+0000 7f5b6494b640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5b600cfa40 0x7f5b600cd260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:12.011 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:12.008+0000 7f5b6494b640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b6015c0a0 con 0x7f5b6006a830 2023-05-22T15:46:12.012 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:12.008+0000 7f5b6494b640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b600cc850 con 0x7f5b600cf670 2023-05-22T15:46:12.012 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:12.008+0000 7f5b6494b640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b6006b3c0 con 0x7f5b600cfa40 2023-05-22T15:46:12.012 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:12.008+0000 7f5b5e59c640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f5b600cf670 0x7f5b601601d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:12.012 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:12.008+0000 7f5b5e59c640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f5b600cf670 0x7f5b601601d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43450/0 (socket says 172.21.15.16:43450) 2023-05-22T15:46:12.012 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:12.008+0000 7f5b5e59c640 1 -- 172.21.15.16:0/2090820756 learned_addr learned my addr 172.21.15.16:0/2090820756 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:46:12.012 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:12.008+0000 7f5b57fff640 1 --2- 172.21.15.16:0/2090820756 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5b600cfa40 0x7f5b600cd260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:12.013 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:12.008+0000 7f5b5e59c640 1 -- 172.21.15.16:0/2090820756 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5b600cfa40 msgr2=0x7f5b600cd260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:46:12.013 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:12.008+0000 7f5b5e59c640 1 --2- 172.21.15.16:0/2090820756 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5b600cfa40 0x7f5b600cd260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:12.013 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:12.008+0000 7f5b5e59c640 1 -- 172.21.15.16:0/2090820756 >> v1:172.21.15.62:6789/0 conn(0x7f5b6006a830 legacy=0x7f5b6006acb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:12.013 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:12.008+0000 7f5b5e59c640 1 -- 172.21.15.16:0/2090820756 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b600ccdd0 con 0x7f5b600cf670 2023-05-22T15:46:12.014 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:12.008+0000 7f5b5e59c640 1 --2- 172.21.15.16:0/2090820756 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b600cf670 0x7f5b601601d0 secure :-1 s=READY pgs=1773 cs=0 l=1 rev1=1 crypto rx=0x7f5b500098a0 tx=0x7f5b5001c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f917c8da2cabb04e server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:12.014 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:12.008+0000 7f5b5dd9b640 1 -- 172.21.15.16:0/2090820756 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b500ab610 con 0x7f5b600cf670 2023-05-22T15:46:12.014 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:12.008+0000 7f5b5dd9b640 1 -- 172.21.15.16:0/2090820756 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b500ab770 con 0x7f5b600cf670 2023-05-22T15:46:12.014 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:12.008+0000 7f5b5dd9b640 1 -- 172.21.15.16:0/2090820756 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b500abb00 con 0x7f5b600cf670 2023-05-22T15:46:12.015 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:12.008+0000 7f5b6494b640 1 -- 172.21.15.16:0/2090820756 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b600cf670 msgr2=0x7f5b601601d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:12.015 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:12.008+0000 7f5b6494b640 1 --2- 172.21.15.16:0/2090820756 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b600cf670 0x7f5b601601d0 secure :-1 s=READY pgs=1773 cs=0 l=1 rev1=1 crypto rx=0x7f5b500098a0 tx=0x7f5b5001c520 comp rx=0 tx=0).stop 2023-05-22T15:46:12.015 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:12.008+0000 7f5b6494b640 1 -- 172.21.15.16:0/2090820756 shutdown_connections 2023-05-22T15:46:12.015 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:12.008+0000 7f5b6494b640 1 --2- 172.21.15.16:0/2090820756 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5b600cfa40 0x7f5b600cd260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:12.016 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:12.008+0000 7f5b6494b640 1 --2- 172.21.15.16:0/2090820756 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b600cf670 0x7f5b601601d0 unknown :-1 s=CLOSED pgs=1773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:12.016 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:12.008+0000 7f5b6494b640 1 -- 172.21.15.16:0/2090820756 >> 172.21.15.16:0/2090820756 conn(0x7f5b600700b0 msgr2=0x7f5b6006d9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:46:12.016 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:12.008+0000 7f5b6494b640 1 -- 172.21.15.16:0/2090820756 shutdown_connections 2023-05-22T15:46:12.017 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:12.008+0000 7f5b6494b640 1 -- 172.21.15.16:0/2090820756 wait complete. 2023-05-22T15:46:12.018 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:12.012+0000 7f5b6494b640 1 Processor -- start 2023-05-22T15:46:12.018 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:12.012+0000 7f5b6494b640 1 -- start start 2023-05-22T15:46:12.019 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:12.012+0000 7f5b6494b640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5b6006a830 0x7f5b6014f580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:12.019 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:12.012+0000 7f5b6494b640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5b600cf670 0x7f5b600b30d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:12.019 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:12.012+0000 7f5b6494b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b600cfa40 0x7f5b60157e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:12.019 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:12.012+0000 7f5b6494b640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b600b3610 con 0x7f5b600cfa40 2023-05-22T15:46:12.019 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:12.012+0000 7f5b6494b640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b600b58c0 con 0x7f5b600cf670 2023-05-22T15:46:12.019 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:12.012+0000 7f5b6494b640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b600b5ba0 con 0x7f5b6006a830 2023-05-22T15:46:12.019 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:12.012+0000 7f5b5ed9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b600cfa40 0x7f5b60157e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:12.019 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:12.012+0000 7f5b5ed9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b600cfa40 0x7f5b60157e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43452/0 (socket says 172.21.15.16:43452) 2023-05-22T15:46:12.020 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:12.012+0000 7f5b5ed9d640 1 -- 172.21.15.16:0/882548012 learned_addr learned my addr 172.21.15.16:0/882548012 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:46:12.020 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:12.012+0000 7f5b57fff640 1 --2- 172.21.15.16:0/882548012 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5b600cf670 0x7f5b600b30d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:12.020 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:12.012+0000 7f5b5e59c640 1 --2- 172.21.15.16:0/882548012 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5b6006a830 0x7f5b6014f580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:12.020 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:12.012+0000 7f5b5ed9d640 1 -- 172.21.15.16:0/882548012 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5b6006a830 msgr2=0x7f5b6014f580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:12.020 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:12.012+0000 7f5b5ed9d640 1 --2- 172.21.15.16:0/882548012 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5b6006a830 0x7f5b6014f580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:12.020 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:12.012+0000 7f5b5ed9d640 1 -- 172.21.15.16:0/882548012 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5b600cf670 msgr2=0x7f5b600b30d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:12.020 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:12.012+0000 7f5b5ed9d640 1 --2- 172.21.15.16:0/882548012 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5b600cf670 0x7f5b600b30d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:12.021 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:12.012+0000 7f5b5ed9d640 1 -- 172.21.15.16:0/882548012 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b500094d0 con 0x7f5b600cfa40 2023-05-22T15:46:12.021 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:12.012+0000 7f5b5ed9d640 1 --2- 172.21.15.16:0/882548012 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b600cfa40 0x7f5b60157e90 secure :-1 s=READY pgs=1774 cs=0 l=1 rev1=1 crypto rx=0x7f5b5800c770 tx=0x7f5b5800cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:12.021 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:12.012+0000 7f5b56ffd640 1 -- 172.21.15.16:0/882548012 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b58003b80 con 0x7f5b600cfa40 2023-05-22T15:46:12.021 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:12.012+0000 7f5b56ffd640 1 -- 172.21.15.16:0/882548012 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b58003ce0 con 0x7f5b600cfa40 2023-05-22T15:46:12.021 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:12.012+0000 7f5b6494b640 1 -- 172.21.15.16:0/882548012 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b6006c0e0 con 0x7f5b600cfa40 2023-05-22T15:46:12.021 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:12.012+0000 7f5b56ffd640 1 -- 172.21.15.16:0/882548012 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b5800aa90 con 0x7f5b600cfa40 2023-05-22T15:46:12.021 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:12.012+0000 7f5b6494b640 1 -- 172.21.15.16:0/882548012 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b600cca60 con 0x7f5b600cfa40 2023-05-22T15:46:12.022 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:12.012+0000 7f5b6494b640 1 -- 172.21.15.16:0/882548012 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b600cd7a0 con 0x7f5b600cfa40 2023-05-22T15:46:12.022 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:12.016+0000 7f5b56ffd640 1 -- 172.21.15.16:0/882548012 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 60) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f5b5800abf0 con 0x7f5b600cfa40 2023-05-22T15:46:12.022 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:12.016+0000 7f5b56ffd640 1 --2- 172.21.15.16:0/882548012 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f5b400654a0 0x7f5b40067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:12.023 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:12.016+0000 7f5b5e59c640 1 --2- 172.21.15.16:0/882548012 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f5b400654a0 0x7f5b40067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:12.023 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:12.016+0000 7f5b56ffd640 1 -- 172.21.15.16:0/882548012 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(136..136 src has 1..136) v4 ==== 7184+0+0 (secure 0 0 0) 0x7f5b58082b10 con 0x7f5b600cfa40 2023-05-22T15:46:12.023 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:12.016+0000 7f5b5e59c640 1 --2- 172.21.15.16:0/882548012 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f5b400654a0 0x7f5b40067950 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f5b500af000 tx=0x7f5b500a7000 comp rx=0 tx=0).ready entity=mgr.8835 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:12.023 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:12.020+0000 7f5b56ffd640 1 -- 172.21.15.16:0/882548012 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f5b58050220 con 0x7f5b600cfa40 2023-05-22T15:46:12.176 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:12.172+0000 7f5b6494b640 1 -- 172.21.15.16:0/882548012 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f5b6015c0a0 con 0x7f5b600cfa40 2023-05-22T15:46:12.877 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:12.872+0000 7f5b56ffd640 1 -- 172.21.15.16:0/882548012 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v137) v1 ==== 181+0+0 (secure 0 0 0) 0x7f5b6015c0a0 con 0x7f5b600cfa40 2023-05-22T15:46:12.881 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'cephfs_metadata' removed 2023-05-22T15:46:12.888 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:12.884+0000 7f5b6494b640 1 -- 172.21.15.16:0/882548012 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f5b400654a0 msgr2=0x7f5b40067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:12.888 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:12.884+0000 7f5b6494b640 1 --2- 172.21.15.16:0/882548012 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f5b400654a0 0x7f5b40067950 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f5b500af000 tx=0x7f5b500a7000 comp rx=0 tx=0).stop 2023-05-22T15:46:12.888 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:12.884+0000 7f5b6494b640 1 -- 172.21.15.16:0/882548012 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b600cfa40 msgr2=0x7f5b60157e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:12.889 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:12.884+0000 7f5b6494b640 1 --2- 172.21.15.16:0/882548012 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b600cfa40 0x7f5b60157e90 secure :-1 s=READY pgs=1774 cs=0 l=1 rev1=1 crypto rx=0x7f5b5800c770 tx=0x7f5b5800cc30 comp rx=0 tx=0).stop 2023-05-22T15:46:12.889 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:12.884+0000 7f5b6494b640 1 -- 172.21.15.16:0/882548012 shutdown_connections 2023-05-22T15:46:12.889 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:12.884+0000 7f5b6494b640 1 --2- 172.21.15.16:0/882548012 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b600cfa40 0x7f5b60157e90 unknown :-1 s=CLOSED pgs=1774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:12.890 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:12.884+0000 7f5b6494b640 1 --2- 172.21.15.16:0/882548012 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5b600cf670 0x7f5b600b30d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:12.890 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:12.884+0000 7f5b6494b640 1 --2- 172.21.15.16:0/882548012 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f5b400654a0 0x7f5b40067950 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:12.890 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:12.884+0000 7f5b6494b640 1 --2- 172.21.15.16:0/882548012 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5b6006a830 0x7f5b6014f580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:12.890 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:12.884+0000 7f5b6494b640 1 -- 172.21.15.16:0/882548012 >> 172.21.15.16:0/882548012 conn(0x7f5b600700b0 msgr2=0x7f5b6006acb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:46:12.890 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:12.884+0000 7f5b6494b640 1 -- 172.21.15.16:0/882548012 shutdown_connections 2023-05-22T15:46:12.891 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:12.884+0000 7f5b6494b640 1 -- 172.21.15.16:0/882548012 wait complete. 2023-05-22T15:46:12.909 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-05-22T15:46:13.040 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:13.036+0000 7fbf2e741640 1 Processor -- start 2023-05-22T15:46:13.041 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:13.036+0000 7fbf2e741640 1 -- start start 2023-05-22T15:46:13.043 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:13.036+0000 7fbf2e741640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbf28070c10 0x7fbf280cc4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:13.043 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:13.036+0000 7fbf2e741640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf280ccbb0 con 0x7fbf280c9e40 2023-05-22T15:46:13.044 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:13.036+0000 7fbf2e741640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf280cd7c0 con 0x7fbf2815d650 2023-05-22T15:46:13.044 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:13.036+0000 7fbf2e741640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf280cdaa0 con 0x7fbf28070c10 2023-05-22T15:46:13.044 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:13.036+0000 7fbf277fe640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fbf280c9e40 0x7fbf2806bfa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:59176/0 (socket says 172.21.15.16:59176) 2023-05-22T15:46:13.045 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:13.036+0000 7fbf277fe640 1 -- 172.21.15.16:0/1081938947 learned_addr learned my addr 172.21.15.16:0/1081938947 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:46:13.045 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:13.036+0000 7fbf27fff640 1 --2- 172.21.15.16:0/1081938947 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbf28070c10 0x7fbf280cc4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:13.045 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:13.036+0000 7fbf26ffd640 1 -- 172.21.15.16:0/1081938947 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 872820678 0 0) 0x7fbf280ccbb0 con 0x7fbf280c9e40 2023-05-22T15:46:13.045 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:13.036+0000 7fbf26ffd640 1 -- 172.21.15.16:0/1081938947 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf10003530 con 0x7fbf280c9e40 2023-05-22T15:46:13.045 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:13.036+0000 7fbf26ffd640 1 -- 172.21.15.16:0/1081938947 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3300987709 0 0) 0x7fbf10003530 con 0x7fbf280c9e40 2023-05-22T15:46:13.045 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:13.036+0000 7fbf26ffd640 1 -- 172.21.15.16:0/1081938947 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbf28070c10 msgr2=0x7fbf280cc4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:46:13.046 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:13.036+0000 7fbf26ffd640 1 --2- 172.21.15.16:0/1081938947 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbf28070c10 0x7fbf280cc4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:13.046 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:13.036+0000 7fbf26ffd640 1 -- 172.21.15.16:0/1081938947 >> v1:172.21.15.62:6789/0 conn(0x7fbf2815d650 legacy=0x7fbf280ce2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:13.046 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:13.036+0000 7fbf26ffd640 1 -- 172.21.15.16:0/1081938947 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf280cdd80 con 0x7fbf280c9e40 2023-05-22T15:46:13.046 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:13.036+0000 7fbf26ffd640 1 -- 172.21.15.16:0/1081938947 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7fbf1c001680 con 0x7fbf280c9e40 2023-05-22T15:46:13.046 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:13.036+0000 7fbf26ffd640 1 -- 172.21.15.16:0/1081938947 >> v1:172.21.15.16:6789/0 conn(0x7fbf280c9e40 legacy=0x7fbf2806bfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:13.046 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:13.036+0000 7fbf26ffd640 1 --2- 172.21.15.16:0/1081938947 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbf100039b0 0x7fbf10003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:13.046 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:13.036+0000 7fbf26ffd640 1 --2- 172.21.15.16:0/1081938947 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbf28070c10 0x7fbf100047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:13.047 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:13.036+0000 7fbf26ffd640 1 --2- 172.21.15.16:0/1081938947 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbf10004d10 0x7fbf10005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:13.047 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:13.036+0000 7fbf26ffd640 1 -- 172.21.15.16:0/1081938947 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf280cdaa0 con 0x7fbf100039b0 2023-05-22T15:46:13.047 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:13.036+0000 7fbf26ffd640 1 -- 172.21.15.16:0/1081938947 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf1000ba40 con 0x7fbf10004d10 2023-05-22T15:46:13.047 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:13.036+0000 7fbf26ffd640 1 -- 172.21.15.16:0/1081938947 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf1000bbd0 con 0x7fbf28070c10 2023-05-22T15:46:13.047 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:13.036+0000 7fbf27fff640 1 --2- 172.21.15.16:0/1081938947 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbf100039b0 0x7fbf10003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:13.047 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:13.036+0000 7fbf277fe640 1 --2- 172.21.15.16:0/1081938947 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbf28070c10 0x7fbf100047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:13.047 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:13.036+0000 7fbf27fff640 1 -- 172.21.15.16:0/1081938947 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbf28070c10 msgr2=0x7fbf100047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:46:13.048 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:13.036+0000 7fbf2ccde640 1 --2- 172.21.15.16:0/1081938947 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbf10004d10 0x7fbf10005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:13.048 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:13.036+0000 7fbf27fff640 1 --2- 172.21.15.16:0/1081938947 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbf28070c10 0x7fbf100047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:13.048 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:13.036+0000 7fbf27fff640 1 -- 172.21.15.16:0/1081938947 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbf10004d10 msgr2=0x7fbf10005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:46:13.048 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:13.036+0000 7fbf27fff640 1 --2- 172.21.15.16:0/1081938947 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbf10004d10 0x7fbf10005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:13.048 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:13.036+0000 7fbf27fff640 1 -- 172.21.15.16:0/1081938947 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf10003530 con 0x7fbf100039b0 2023-05-22T15:46:13.048 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:13.040+0000 7fbf277fe640 1 --2- 172.21.15.16:0/1081938947 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbf28070c10 0x7fbf100047d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:46:13.048 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:13.040+0000 7fbf2ccde640 1 --2- 172.21.15.16:0/1081938947 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbf10004d10 0x7fbf10005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:46:13.051 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:13.040+0000 7fbf27fff640 1 --2- 172.21.15.16:0/1081938947 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbf100039b0 0x7fbf10003dc0 secure :-1 s=READY pgs=1776 cs=0 l=1 rev1=1 crypto rx=0x7fbf18009ae0 tx=0x7fbf1801ca50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e95b0da70c65647 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:13.051 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:13.040+0000 7fbf26ffd640 1 -- 172.21.15.16:0/1081938947 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf180a5620 con 0x7fbf100039b0 2023-05-22T15:46:13.051 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:13.040+0000 7fbf26ffd640 1 -- 172.21.15.16:0/1081938947 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf180a5780 con 0x7fbf100039b0 2023-05-22T15:46:13.051 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:13.040+0000 7fbf26ffd640 1 -- 172.21.15.16:0/1081938947 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf180a5b30 con 0x7fbf100039b0 2023-05-22T15:46:13.052 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:13.040+0000 7fbf2e741640 1 -- 172.21.15.16:0/1081938947 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbf100039b0 msgr2=0x7fbf10003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:13.052 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:13.040+0000 7fbf2e741640 1 --2- 172.21.15.16:0/1081938947 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbf100039b0 0x7fbf10003dc0 secure :-1 s=READY pgs=1776 cs=0 l=1 rev1=1 crypto rx=0x7fbf18009ae0 tx=0x7fbf1801ca50 comp rx=0 tx=0).stop 2023-05-22T15:46:13.052 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:13.040+0000 7fbf2ccde640 1 -- 172.21.15.16:0/1081938947 reap_dead start 2023-05-22T15:46:13.052 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:13.040+0000 7fbf2e741640 1 -- 172.21.15.16:0/1081938947 shutdown_connections 2023-05-22T15:46:13.052 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:13.040+0000 7fbf2e741640 1 -- 172.21.15.16:0/1081938947 >> 172.21.15.16:0/1081938947 conn(0x7fbf28069720 msgr2=0x7fbf28069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:46:13.052 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:13.040+0000 7fbf2e741640 1 -- 172.21.15.16:0/1081938947 shutdown_connections 2023-05-22T15:46:13.052 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:13.040+0000 7fbf2e741640 1 -- 172.21.15.16:0/1081938947 wait complete. 2023-05-22T15:46:13.053 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:13.040+0000 7fbf2e741640 1 Processor -- start 2023-05-22T15:46:13.053 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:13.044+0000 7fbf2e741640 1 -- start start 2023-05-22T15:46:13.053 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:13.044+0000 7fbf2e741640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbf100039b0 0x7fbf280b00e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:13.053 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:13.044+0000 7fbf2e741640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbf281f2390 0x7fbf281f2800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:13.053 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:13.044+0000 7fbf2e741640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbf2806f750 0x7fbf2806fbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:13.053 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:13.044+0000 7fbf2e741640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf281f37f0 con 0x7fbf2806f750 2023-05-22T15:46:13.053 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:13.044+0000 7fbf2e741640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf28159660 con 0x7fbf281f2390 2023-05-22T15:46:13.054 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:13.044+0000 7fbf2e741640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf281f2d40 con 0x7fbf100039b0 2023-05-22T15:46:13.054 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:13.044+0000 7fbf2ccde640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbf2806f750 0x7fbf2806fbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:13.054 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:13.044+0000 7fbf2ccde640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbf2806f750 0x7fbf2806fbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43482/0 (socket says 172.21.15.16:43482) 2023-05-22T15:46:13.054 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:13.044+0000 7fbf2ccde640 1 -- 172.21.15.16:0/822658140 learned_addr learned my addr 172.21.15.16:0/822658140 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:46:13.054 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:13.044+0000 7fbf277fe640 1 --2- 172.21.15.16:0/822658140 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbf281f2390 0x7fbf281f2800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:13.054 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:13.044+0000 7fbf27fff640 1 --2- 172.21.15.16:0/822658140 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbf100039b0 0x7fbf280b00e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:13.054 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:13.044+0000 7fbf2ccde640 1 -- 172.21.15.16:0/822658140 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbf100039b0 msgr2=0x7fbf280b00e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:13.055 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:13.044+0000 7fbf2ccde640 1 --2- 172.21.15.16:0/822658140 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbf100039b0 0x7fbf280b00e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:13.055 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:13.044+0000 7fbf2ccde640 1 -- 172.21.15.16:0/822658140 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbf281f2390 msgr2=0x7fbf281f2800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:13.055 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:13.044+0000 7fbf2ccde640 1 --2- 172.21.15.16:0/822658140 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbf281f2390 0x7fbf281f2800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:13.055 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:13.044+0000 7fbf2ccde640 1 -- 172.21.15.16:0/822658140 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf280cedc0 con 0x7fbf2806f750 2023-05-22T15:46:13.055 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:13.044+0000 7fbf277fe640 1 --2- 172.21.15.16:0/822658140 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbf281f2390 0x7fbf281f2800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:46:13.055 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:13.044+0000 7fbf27fff640 1 --2- 172.21.15.16:0/822658140 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbf100039b0 0x7fbf280b00e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:46:13.055 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:13.044+0000 7fbf2ccde640 1 --2- 172.21.15.16:0/822658140 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbf2806f750 0x7fbf2806fbc0 secure :-1 s=READY pgs=1777 cs=0 l=1 rev1=1 crypto rx=0x7fbf2806f700 tx=0x7fbf2000d830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:13.056 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:13.044+0000 7fbf24ff9640 1 -- 172.21.15.16:0/822658140 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf2000b070 con 0x7fbf2806f750 2023-05-22T15:46:13.056 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:13.044+0000 7fbf24ff9640 1 -- 172.21.15.16:0/822658140 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf2000db40 con 0x7fbf2806f750 2023-05-22T15:46:13.056 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:13.044+0000 7fbf24ff9640 1 -- 172.21.15.16:0/822658140 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf20003980 con 0x7fbf2806f750 2023-05-22T15:46:13.056 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:13.044+0000 7fbf2e741640 1 -- 172.21.15.16:0/822658140 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf280cce90 con 0x7fbf2806f750 2023-05-22T15:46:13.056 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:13.044+0000 7fbf2e741640 1 -- 172.21.15.16:0/822658140 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf280ca230 con 0x7fbf2806f750 2023-05-22T15:46:13.056 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:13.044+0000 7fbf2e741640 1 -- 172.21.15.16:0/822658140 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf281f3c80 con 0x7fbf2806f750 2023-05-22T15:46:13.056 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:13.044+0000 7fbf24ff9640 1 -- 172.21.15.16:0/822658140 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 60) v1 ==== 84332+0+0 (secure 0 0 0) 0x7fbf20003ae0 con 0x7fbf2806f750 2023-05-22T15:46:13.057 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:13.044+0000 7fbf24ff9640 1 --2- 172.21.15.16:0/822658140 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fbf140654a0 0x7fbf14067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:13.057 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:13.044+0000 7fbf27fff640 1 --2- 172.21.15.16:0/822658140 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fbf140654a0 0x7fbf14067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:13.057 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:13.044+0000 7fbf24ff9640 1 -- 172.21.15.16:0/822658140 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(137..137 src has 1..137) v4 ==== 6746+0+0 (secure 0 0 0) 0x7fbf20082300 con 0x7fbf2806f750 2023-05-22T15:46:13.057 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:13.044+0000 7fbf27fff640 1 --2- 172.21.15.16:0/822658140 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fbf140654a0 0x7fbf14067950 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fbf180af000 tx=0x7fbf180046d0 comp rx=0 tx=0).ready entity=mgr.8835 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:13.057 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:13.048+0000 7fbf24ff9640 1 -- 172.21.15.16:0/822658140 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fbf281f3c80 con 0x7fbf2806f750 2023-05-22T15:46:13.211 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:13.208+0000 7fbf2e741640 1 -- 172.21.15.16:0/822658140 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fbf281f3c80 con 0x7fbf2806f750 2023-05-22T15:46:13.896 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:13.892+0000 7fbf24ff9640 1 -- 172.21.15.16:0/822658140 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v138) v1 ==== 169+0+0 (secure 0 0 0) 0x7fbf281f3c80 con 0x7fbf2806f750 2023-05-22T15:46:13.898 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'cephfs_data' removed 2023-05-22T15:46:13.903 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:13.900+0000 7fbf2e741640 1 -- 172.21.15.16:0/822658140 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fbf140654a0 msgr2=0x7fbf14067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:13.903 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:13.900+0000 7fbf2e741640 1 --2- 172.21.15.16:0/822658140 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fbf140654a0 0x7fbf14067950 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fbf180af000 tx=0x7fbf180046d0 comp rx=0 tx=0).stop 2023-05-22T15:46:13.904 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:13.900+0000 7fbf2e741640 1 -- 172.21.15.16:0/822658140 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbf2806f750 msgr2=0x7fbf2806fbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:13.904 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:13.900+0000 7fbf2e741640 1 --2- 172.21.15.16:0/822658140 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbf2806f750 0x7fbf2806fbc0 secure :-1 s=READY pgs=1777 cs=0 l=1 rev1=1 crypto rx=0x7fbf2806f700 tx=0x7fbf2000d830 comp rx=0 tx=0).stop 2023-05-22T15:46:13.904 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:13.900+0000 7fbf2e741640 1 -- 172.21.15.16:0/822658140 shutdown_connections 2023-05-22T15:46:13.905 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:13.900+0000 7fbf2e741640 1 --2- 172.21.15.16:0/822658140 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbf2806f750 0x7fbf2806fbc0 unknown :-1 s=CLOSED pgs=1777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:13.905 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:13.900+0000 7fbf2e741640 1 --2- 172.21.15.16:0/822658140 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbf281f2390 0x7fbf281f2800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:13.905 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:13.900+0000 7fbf2e741640 1 --2- 172.21.15.16:0/822658140 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fbf140654a0 0x7fbf14067950 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:13.905 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:13.900+0000 7fbf2e741640 1 --2- 172.21.15.16:0/822658140 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbf100039b0 0x7fbf280b00e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:13.905 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:13.900+0000 7fbf2e741640 1 -- 172.21.15.16:0/822658140 >> 172.21.15.16:0/822658140 conn(0x7fbf28069720 msgr2=0x7fbf28069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:46:13.905 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:13.900+0000 7fbf2e741640 1 -- 172.21.15.16:0/822658140 shutdown_connections 2023-05-22T15:46:13.905 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:13.900+0000 7fbf2e741640 1 -- 172.21.15.16:0/822658140 wait complete. 2023-05-22T15:46:13.930 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2023-05-22T15:46:14.061 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.056+0000 7f994beba640 1 Processor -- start 2023-05-22T15:46:14.061 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.056+0000 7f994beba640 1 -- start start 2023-05-22T15:46:14.062 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.056+0000 7f994beba640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f99440cb4c0 0x7f9944158fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:14.062 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.056+0000 7f994beba640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f9944159510 0x7f994406fba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:14.062 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.056+0000 7f994beba640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9944159f70 con 0x7f994406b6c0 2023-05-22T15:46:14.062 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.056+0000 7f994beba640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9944071760 con 0x7f9944159510 2023-05-22T15:46:14.063 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.056+0000 7f994beba640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99440cb910 con 0x7f99440cb4c0 2023-05-22T15:46:14.063 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.056+0000 7f9949c56640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f994406b6c0 0x7f994406ba90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:59186/0 (socket says 172.21.15.16:59186) 2023-05-22T15:46:14.063 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.056+0000 7f9949c56640 1 -- 172.21.15.16:0/161049352 learned_addr learned my addr 172.21.15.16:0/161049352 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:46:14.064 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.056+0000 7f9949455640 1 --2- 172.21.15.16:0/161049352 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f99440cb4c0 0x7f9944158fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:14.065 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.056+0000 7f994a457640 1 --2- 172.21.15.16:0/161049352 >> v2:172.21.15.62:3300/0 conn(0x7f9944159510 0x7f994406fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:14.065 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.056+0000 7f9948c54640 1 -- 172.21.15.16:0/161049352 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2808519280 0 0) 0x7f9944159f70 con 0x7f994406b6c0 2023-05-22T15:46:14.065 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.060+0000 7f9948c54640 1 -- 172.21.15.16:0/161049352 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9930003530 con 0x7f994406b6c0 2023-05-22T15:46:14.065 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.060+0000 7f9948c54640 1 -- 172.21.15.16:0/161049352 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2452897621 0 0) 0x7f9930003530 con 0x7f994406b6c0 2023-05-22T15:46:14.065 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.060+0000 7f9948c54640 1 -- 172.21.15.16:0/161049352 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f99440cb4c0 msgr2=0x7f9944158fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:46:14.065 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.060+0000 7f9948c54640 1 --2- 172.21.15.16:0/161049352 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f99440cb4c0 0x7f9944158fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:14.065 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.060+0000 7f9948c54640 1 -- 172.21.15.16:0/161049352 >> v2:172.21.15.62:3300/0 conn(0x7f9944159510 msgr2=0x7f994406fba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:46:14.066 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.060+0000 7f9948c54640 1 --2- 172.21.15.16:0/161049352 >> v2:172.21.15.62:3300/0 conn(0x7f9944159510 0x7f994406fba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:14.066 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.060+0000 7f9948c54640 1 -- 172.21.15.16:0/161049352 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99440cb030 con 0x7f994406b6c0 2023-05-22T15:46:14.066 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.060+0000 7f9948c54640 1 -- 172.21.15.16:0/161049352 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7f9940001680 con 0x7f994406b6c0 2023-05-22T15:46:14.066 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.060+0000 7f9948c54640 1 -- 172.21.15.16:0/161049352 >> v1:172.21.15.16:6789/0 conn(0x7f994406b6c0 legacy=0x7f994406ba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:14.066 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.060+0000 7f9948c54640 1 --2- 172.21.15.16:0/161049352 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9930003980 0x7f9930003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:14.066 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.060+0000 7f9948c54640 1 --2- 172.21.15.16:0/161049352 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f99300043b0 0x7f9930004870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:14.066 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.060+0000 7f9948c54640 1 --2- 172.21.15.16:0/161049352 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f99440cb4c0 0x7f9930004f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:14.066 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.060+0000 7f9948c54640 1 -- 172.21.15.16:0/161049352 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9944071760 con 0x7f9930003980 2023-05-22T15:46:14.067 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.060+0000 7f9948c54640 1 -- 172.21.15.16:0/161049352 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99440cb910 con 0x7f99300043b0 2023-05-22T15:46:14.067 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.060+0000 7f9948c54640 1 -- 172.21.15.16:0/161049352 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9930005500 con 0x7f99440cb4c0 2023-05-22T15:46:14.067 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.060+0000 7f9949c56640 1 --2- 172.21.15.16:0/161049352 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9930003980 0x7f9930003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:14.067 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.060+0000 7f9949455640 1 --2- 172.21.15.16:0/161049352 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f99300043b0 0x7f9930004870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:14.067 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.060+0000 7f9949c56640 1 -- 172.21.15.16:0/161049352 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f99440cb4c0 msgr2=0x7f9930004f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:46:14.067 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.060+0000 7f994a457640 1 --2- 172.21.15.16:0/161049352 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f99440cb4c0 0x7f9930004f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:14.067 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.060+0000 7f9949c56640 1 --2- 172.21.15.16:0/161049352 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f99440cb4c0 0x7f9930004f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:14.068 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.060+0000 7f9949c56640 1 -- 172.21.15.16:0/161049352 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f99300043b0 msgr2=0x7f9930004870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:46:14.068 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.060+0000 7f9949c56640 1 --2- 172.21.15.16:0/161049352 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f99300043b0 0x7f9930004870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:14.068 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.060+0000 7f9949c56640 1 -- 172.21.15.16:0/161049352 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9930003530 con 0x7f9930003980 2023-05-22T15:46:14.068 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.060+0000 7f994a457640 1 --2- 172.21.15.16:0/161049352 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f99440cb4c0 0x7f9930004f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:46:14.070 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.060+0000 7f9949c56640 1 --2- 172.21.15.16:0/161049352 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9930003980 0x7f9930003d90 secure :-1 s=READY pgs=1779 cs=0 l=1 rev1=1 crypto rx=0x7f994000c210 tx=0x7f994001cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f77a4d791f1f96f server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:14.070 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.060+0000 7f9948c54640 1 -- 172.21.15.16:0/161049352 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99400058e0 con 0x7f9930003980 2023-05-22T15:46:14.070 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.060+0000 7f9948c54640 1 -- 172.21.15.16:0/161049352 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9940005a40 con 0x7f9930003980 2023-05-22T15:46:14.071 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.060+0000 7f9948c54640 1 -- 172.21.15.16:0/161049352 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9940005e30 con 0x7f9930003980 2023-05-22T15:46:14.071 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.060+0000 7f994beba640 1 -- 172.21.15.16:0/161049352 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9930003980 msgr2=0x7f9930003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:14.071 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.060+0000 7f994beba640 1 --2- 172.21.15.16:0/161049352 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9930003980 0x7f9930003d90 secure :-1 s=READY pgs=1779 cs=0 l=1 rev1=1 crypto rx=0x7f994000c210 tx=0x7f994001cc60 comp rx=0 tx=0).stop 2023-05-22T15:46:14.071 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.060+0000 7f994a457640 1 -- 172.21.15.16:0/161049352 reap_dead start 2023-05-22T15:46:14.071 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.060+0000 7f994beba640 1 -- 172.21.15.16:0/161049352 shutdown_connections 2023-05-22T15:46:14.071 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.060+0000 7f994beba640 1 -- 172.21.15.16:0/161049352 >> 172.21.15.16:0/161049352 conn(0x7f9944069720 msgr2=0x7f9944069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:46:14.071 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.060+0000 7f994beba640 1 -- 172.21.15.16:0/161049352 shutdown_connections 2023-05-22T15:46:14.071 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.060+0000 7f994beba640 1 -- 172.21.15.16:0/161049352 wait complete. 2023-05-22T15:46:14.072 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.064+0000 7f994beba640 1 Processor -- start 2023-05-22T15:46:14.072 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.064+0000 7f994beba640 1 -- start start 2023-05-22T15:46:14.072 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.064+0000 7f994beba640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f994414f660 0x7f99440c2570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:14.072 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.064+0000 7f994beba640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f994414fa30 0x7f99441f05a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:14.072 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.064+0000 7f994beba640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f99441f4330 0x7f994414cc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:14.072 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.064+0000 7f994beba640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99441f3bc0 con 0x7f994414f660 2023-05-22T15:46:14.072 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.064+0000 7f994beba640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99441f0ae0 con 0x7f994414fa30 2023-05-22T15:46:14.073 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.064+0000 7f994beba640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99441f0db0 con 0x7f99441f4330 2023-05-22T15:46:14.073 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.064+0000 7f9949c56640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f994414f660 0x7f99440c2570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:14.073 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.064+0000 7f9949c56640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f994414f660 0x7f99440c2570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43494/0 (socket says 172.21.15.16:43494) 2023-05-22T15:46:14.073 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.064+0000 7f9949c56640 1 -- 172.21.15.16:0/4190187484 learned_addr learned my addr 172.21.15.16:0/4190187484 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:46:14.073 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.064+0000 7f994a457640 1 --2- 172.21.15.16:0/4190187484 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f99441f4330 0x7f994414cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:14.073 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.064+0000 7f9949c56640 1 -- 172.21.15.16:0/4190187484 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f99441f4330 msgr2=0x7f994414cc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:14.073 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.064+0000 7f9949c56640 1 --2- 172.21.15.16:0/4190187484 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f99441f4330 0x7f994414cc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:14.073 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.064+0000 7f9949c56640 1 -- 172.21.15.16:0/4190187484 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f994414fa30 msgr2=0x7f99441f05a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:14.074 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.064+0000 7f9949455640 1 --2- 172.21.15.16:0/4190187484 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f994414fa30 0x7f99441f05a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:14.074 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.064+0000 7f9949c56640 1 --2- 172.21.15.16:0/4190187484 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f994414fa30 0x7f99441f05a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:14.074 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.064+0000 7f9949c56640 1 -- 172.21.15.16:0/4190187484 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9944159910 con 0x7f994414f660 2023-05-22T15:46:14.074 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.064+0000 7f994a457640 1 --2- 172.21.15.16:0/4190187484 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f99441f4330 0x7f994414cc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:46:14.074 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.064+0000 7f9949c56640 1 --2- 172.21.15.16:0/4190187484 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f994414f660 0x7f99440c2570 secure :-1 s=READY pgs=1780 cs=0 l=1 rev1=1 crypto rx=0x7f994000c210 tx=0x7f9930005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:14.074 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.064+0000 7f993a7fc640 1 -- 172.21.15.16:0/4190187484 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9940005e70 con 0x7f994414f660 2023-05-22T15:46:14.074 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.064+0000 7f993a7fc640 1 -- 172.21.15.16:0/4190187484 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99400048b0 con 0x7f994414f660 2023-05-22T15:46:14.075 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.064+0000 7f994beba640 1 -- 172.21.15.16:0/4190187484 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99440cacc0 con 0x7f994414f660 2023-05-22T15:46:14.075 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.064+0000 7f993a7fc640 1 -- 172.21.15.16:0/4190187484 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9940004c60 con 0x7f994414f660 2023-05-22T15:46:14.075 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.064+0000 7f994beba640 1 -- 172.21.15.16:0/4190187484 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9944156840 con 0x7f994414f660 2023-05-22T15:46:14.075 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.064+0000 7f994beba640 1 -- 172.21.15.16:0/4190187484 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f994404dbe0 con 0x7f994414f660 2023-05-22T15:46:14.076 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.068+0000 7f993a7fc640 1 -- 172.21.15.16:0/4190187484 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 60) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f9940007090 con 0x7f994414f660 2023-05-22T15:46:14.076 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.068+0000 7f993a7fc640 1 --2- 172.21.15.16:0/4190187484 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f99280654d0 0x7f9928067980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:14.076 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.068+0000 7f993a7fc640 1 -- 172.21.15.16:0/4190187484 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(138..138 src has 1..138) v4 ==== 6332+0+0 (secure 0 0 0) 0x7f994011c570 con 0x7f994414f660 2023-05-22T15:46:14.076 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.068+0000 7f9949455640 1 --2- 172.21.15.16:0/4190187484 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f99280654d0 0x7f9928067980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:14.076 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.068+0000 7f9949455640 1 --2- 172.21.15.16:0/4190187484 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f99280654d0 0x7f9928067980 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f994414e020 tx=0x7f9934008040 comp rx=0 tx=0).ready entity=mgr.8835 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:14.076 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.072+0000 7f993a7fc640 1 -- 172.21.15.16:0/4190187484 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f99400ea040 con 0x7f994414f660 2023-05-22T15:46:14.241 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.236+0000 7f994beba640 1 -- 172.21.15.16:0/4190187484 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7f99440cc0e0 con 0x7f994414f660 2023-05-22T15:46:14.250 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.244+0000 7f993a7fc640 1 -- 172.21.15.16:0/4190187484 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v43) v1 ==== 120+0+0 (secure 0 0 0) 0x7f99400ed660 con 0x7f994414f660 2023-05-22T15:46:14.253 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.248+0000 7f994beba640 1 -- 172.21.15.16:0/4190187484 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f99280654d0 msgr2=0x7f9928067980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:14.254 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.248+0000 7f994beba640 1 --2- 172.21.15.16:0/4190187484 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f99280654d0 0x7f9928067980 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f994414e020 tx=0x7f9934008040 comp rx=0 tx=0).stop 2023-05-22T15:46:14.254 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.248+0000 7f994beba640 1 -- 172.21.15.16:0/4190187484 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f994414f660 msgr2=0x7f99440c2570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:14.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.248+0000 7f994beba640 1 --2- 172.21.15.16:0/4190187484 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f994414f660 0x7f99440c2570 secure :-1 s=READY pgs=1780 cs=0 l=1 rev1=1 crypto rx=0x7f994000c210 tx=0x7f9930005800 comp rx=0 tx=0).stop 2023-05-22T15:46:14.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.248+0000 7f994beba640 1 -- 172.21.15.16:0/4190187484 shutdown_connections 2023-05-22T15:46:14.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.248+0000 7f994beba640 1 --2- 172.21.15.16:0/4190187484 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f99280654d0 0x7f9928067980 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:14.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.248+0000 7f994beba640 1 --2- 172.21.15.16:0/4190187484 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f99441f4330 0x7f994414cc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:14.256 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.248+0000 7f994beba640 1 --2- 172.21.15.16:0/4190187484 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f994414fa30 0x7f99441f05a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:14.256 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.248+0000 7f994beba640 1 --2- 172.21.15.16:0/4190187484 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f994414f660 0x7f99440c2570 unknown :-1 s=CLOSED pgs=1780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:14.256 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.248+0000 7f994beba640 1 -- 172.21.15.16:0/4190187484 >> 172.21.15.16:0/4190187484 conn(0x7f9944069720 msgr2=0x7f994415a170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:46:14.256 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.248+0000 7f994beba640 1 -- 172.21.15.16:0/4190187484 shutdown_connections 2023-05-22T15:46:14.256 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.248+0000 7f994beba640 1 -- 172.21.15.16:0/4190187484 wait complete. 2023-05-22T15:46:14.273 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2023-05-22T15:46:14.392 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.388+0000 7f570781e640 1 Processor -- start 2023-05-22T15:46:14.392 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.388+0000 7f570781e640 1 -- start start 2023-05-22T15:46:14.394 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.388+0000 7f570781e640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f57000c67d0 0x7f57000cf730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:14.394 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.388+0000 7f570781e640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f57000cc6f0 0x7f57000c3db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:14.395 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.388+0000 7f570781e640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57000cfc70 con 0x7f57000b2880 2023-05-22T15:46:14.395 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.388+0000 7f570781e640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f570015a300 con 0x7f57000cc6f0 2023-05-22T15:46:14.395 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.388+0000 7f570781e640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57000c6ba0 con 0x7f57000c67d0 2023-05-22T15:46:14.395 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.388+0000 7f5705dbb640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f57000cc6f0 0x7f57000c3db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:14.396 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.388+0000 7f5705dbb640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f57000cc6f0 0x7f57000c3db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43496/0 (socket says 172.21.15.16:43496) 2023-05-22T15:46:14.396 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.388+0000 7f5705dbb640 1 -- 172.21.15.16:0/1787521017 learned_addr learned my addr 172.21.15.16:0/1787521017 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:46:14.396 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.388+0000 7f57055ba640 1 --2- 172.21.15.16:0/1787521017 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f57000c67d0 0x7f57000cf730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:14.396 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.388+0000 7f5705dbb640 1 -- 172.21.15.16:0/1787521017 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f57000c67d0 msgr2=0x7f57000cf730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:46:14.396 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.388+0000 7f5705dbb640 1 --2- 172.21.15.16:0/1787521017 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f57000c67d0 0x7f57000cf730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:14.397 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.388+0000 7f5705dbb640 1 -- 172.21.15.16:0/1787521017 >> v1:172.21.15.62:6789/0 conn(0x7f57000b2880 legacy=0x7f57000cbfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:46:14.397 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.388+0000 7f5705dbb640 1 -- 172.21.15.16:0/1787521017 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57000cfef0 con 0x7f57000cc6f0 2023-05-22T15:46:14.397 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.388+0000 7f57055ba640 1 --2- 172.21.15.16:0/1787521017 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f57000c67d0 0x7f57000cf730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:46:14.402 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.392+0000 7f5705dbb640 1 --2- 172.21.15.16:0/1787521017 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f57000cc6f0 0x7f57000c3db0 secure :-1 s=READY pgs=1781 cs=0 l=1 rev1=1 crypto rx=0x7f56f8009e50 tx=0x7f56f801c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c52f24ff1f928976 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:14.402 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.392+0000 7f56f7fff640 1 -- 172.21.15.16:0/1787521017 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56f80a5790 con 0x7f57000cc6f0 2023-05-22T15:46:14.402 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.392+0000 7f56f7fff640 1 -- 172.21.15.16:0/1787521017 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56f80a58f0 con 0x7f57000cc6f0 2023-05-22T15:46:14.403 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.392+0000 7f56f7fff640 1 -- 172.21.15.16:0/1787521017 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56f80a5c80 con 0x7f57000cc6f0 2023-05-22T15:46:14.403 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.392+0000 7f570781e640 1 -- 172.21.15.16:0/1787521017 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f57000cc6f0 msgr2=0x7f57000c3db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:14.403 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.392+0000 7f570781e640 1 --2- 172.21.15.16:0/1787521017 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f57000cc6f0 0x7f57000c3db0 secure :-1 s=READY pgs=1781 cs=0 l=1 rev1=1 crypto rx=0x7f56f8009e50 tx=0x7f56f801c910 comp rx=0 tx=0).stop 2023-05-22T15:46:14.403 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.392+0000 7f570781e640 1 -- 172.21.15.16:0/1787521017 shutdown_connections 2023-05-22T15:46:14.404 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.392+0000 7f570781e640 1 --2- 172.21.15.16:0/1787521017 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f57000cc6f0 0x7f57000c3db0 unknown :-1 s=CLOSED pgs=1781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:14.404 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.392+0000 7f570781e640 1 --2- 172.21.15.16:0/1787521017 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f57000c67d0 0x7f57000cf730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:14.404 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.392+0000 7f570781e640 1 -- 172.21.15.16:0/1787521017 >> 172.21.15.16:0/1787521017 conn(0x7f570006f5a0 msgr2=0x7f570006f9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:46:14.404 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.392+0000 7f570781e640 1 -- 172.21.15.16:0/1787521017 shutdown_connections 2023-05-22T15:46:14.404 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.392+0000 7f570781e640 1 -- 172.21.15.16:0/1787521017 wait complete. 2023-05-22T15:46:14.405 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.392+0000 7f570781e640 1 Processor -- start 2023-05-22T15:46:14.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.392+0000 7f570781e640 1 -- start start 2023-05-22T15:46:14.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.392+0000 7f570781e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f57000b2880 0x7f57000c98d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:14.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.392+0000 7f570781e640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f57000c67d0 0x7f57000b06d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:14.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.392+0000 7f570781e640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f57000cc6f0 0x7f57001587f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:14.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.392+0000 7f570781e640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f570015c040 con 0x7f57000b2880 2023-05-22T15:46:14.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.392+0000 7f570781e640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57000c9e10 con 0x7f57000cc6f0 2023-05-22T15:46:14.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.392+0000 7f570781e640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f570015b710 con 0x7f57000c67d0 2023-05-22T15:46:14.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.392+0000 7f57055ba640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f57000b2880 0x7f57000c98d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:14.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.392+0000 7f57055ba640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f57000b2880 0x7f57000c98d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43504/0 (socket says 172.21.15.16:43504) 2023-05-22T15:46:14.408 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.396+0000 7f57055ba640 1 -- 172.21.15.16:0/1533008050 learned_addr learned my addr 172.21.15.16:0/1533008050 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:46:14.408 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.396+0000 7f5704db9640 1 --2- 172.21.15.16:0/1533008050 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f57000c67d0 0x7f57000b06d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:14.408 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.396+0000 7f5705dbb640 1 --2- 172.21.15.16:0/1533008050 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f57000cc6f0 0x7f57001587f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:14.408 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.396+0000 7f57055ba640 1 -- 172.21.15.16:0/1533008050 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f57000c67d0 msgr2=0x7f57000b06d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:14.408 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.396+0000 7f57055ba640 1 --2- 172.21.15.16:0/1533008050 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f57000c67d0 0x7f57000b06d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:14.408 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.396+0000 7f57055ba640 1 -- 172.21.15.16:0/1533008050 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f57000cc6f0 msgr2=0x7f57001587f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:14.409 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.396+0000 7f57055ba640 1 --2- 172.21.15.16:0/1533008050 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f57000cc6f0 0x7f57001587f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:14.409 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.396+0000 7f57055ba640 1 -- 172.21.15.16:0/1533008050 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56f8009b30 con 0x7f57000b2880 2023-05-22T15:46:14.409 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.396+0000 7f5704db9640 1 --2- 172.21.15.16:0/1533008050 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f57000c67d0 0x7f57000b06d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:46:14.409 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.396+0000 7f57055ba640 1 --2- 172.21.15.16:0/1533008050 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f57000b2880 0x7f57000c98d0 secure :-1 s=READY pgs=1782 cs=0 l=1 rev1=1 crypto rx=0x7f56f000c7f0 tx=0x7f56f000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:14.409 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.396+0000 7f56f5ffb640 1 -- 172.21.15.16:0/1533008050 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56f0007820 con 0x7f57000b2880 2023-05-22T15:46:14.409 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.396+0000 7f56f5ffb640 1 -- 172.21.15.16:0/1533008050 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56f0007980 con 0x7f57000b2880 2023-05-22T15:46:14.409 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.396+0000 7f570781e640 1 -- 172.21.15.16:0/1533008050 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57000d0a70 con 0x7f57000b2880 2023-05-22T15:46:14.409 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.396+0000 7f56f5ffb640 1 -- 172.21.15.16:0/1533008050 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56f0007d30 con 0x7f57000b2880 2023-05-22T15:46:14.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.396+0000 7f570781e640 1 -- 172.21.15.16:0/1533008050 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57000d0790 con 0x7f57000b2880 2023-05-22T15:46:14.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.396+0000 7f570781e640 1 -- 172.21.15.16:0/1533008050 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57000d0f90 con 0x7f57000b2880 2023-05-22T15:46:14.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.396+0000 7f56f5ffb640 1 -- 172.21.15.16:0/1533008050 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 60) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f56f00113f0 con 0x7f57000b2880 2023-05-22T15:46:14.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.396+0000 7f56f5ffb640 1 --2- 172.21.15.16:0/1533008050 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f56e40654d0 0x7f56e4067980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:14.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.396+0000 7f56f5ffb640 1 -- 172.21.15.16:0/1533008050 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(138..138 src has 1..138) v4 ==== 6332+0+0 (secure 0 0 0) 0x7f56f0082f00 con 0x7f57000b2880 2023-05-22T15:46:14.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.396+0000 7f5704db9640 1 --2- 172.21.15.16:0/1533008050 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f56e40654d0 0x7f56e4067980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:14.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.396+0000 7f5704db9640 1 --2- 172.21.15.16:0/1533008050 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f56e40654d0 0x7f56e4067980 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f57001f8790 tx=0x7f56fc008040 comp rx=0 tx=0).ready entity=mgr.8835 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:14.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.400+0000 7f56f5ffb640 1 -- 172.21.15.16:0/1533008050 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f56f00507f0 con 0x7f57000b2880 2023-05-22T15:46:14.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.560+0000 7f570781e640 1 -- 172.21.15.16:0/1533008050 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f570004dbe0 con 0x7f57000b2880 2023-05-22T15:46:14.570 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.564+0000 7f56f5ffb640 1 -- 172.21.15.16:0/1533008050 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v44) v1 ==== 104+0+0 (secure 0 0 0) 0x7f56f0053e10 con 0x7f57000b2880 2023-05-22T15:46:14.573 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.568+0000 7f570781e640 1 -- 172.21.15.16:0/1533008050 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f56e40654d0 msgr2=0x7f56e4067980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:14.573 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.568+0000 7f570781e640 1 --2- 172.21.15.16:0/1533008050 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f56e40654d0 0x7f56e4067980 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f57001f8790 tx=0x7f56fc008040 comp rx=0 tx=0).stop 2023-05-22T15:46:14.573 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.568+0000 7f570781e640 1 -- 172.21.15.16:0/1533008050 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f57000b2880 msgr2=0x7f57000c98d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:14.573 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.568+0000 7f570781e640 1 --2- 172.21.15.16:0/1533008050 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f57000b2880 0x7f57000c98d0 secure :-1 s=READY pgs=1782 cs=0 l=1 rev1=1 crypto rx=0x7f56f000c7f0 tx=0x7f56f000ccb0 comp rx=0 tx=0).stop 2023-05-22T15:46:14.573 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.568+0000 7f570781e640 1 -- 172.21.15.16:0/1533008050 shutdown_connections 2023-05-22T15:46:14.573 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.568+0000 7f570781e640 1 --2- 172.21.15.16:0/1533008050 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f57000cc6f0 0x7f57001587f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:14.574 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.568+0000 7f570781e640 1 --2- 172.21.15.16:0/1533008050 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f56e40654d0 0x7f56e4067980 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:14.574 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.568+0000 7f570781e640 1 --2- 172.21.15.16:0/1533008050 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f57000c67d0 0x7f57000b06d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:14.574 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.568+0000 7f570781e640 1 --2- 172.21.15.16:0/1533008050 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f57000b2880 0x7f57000c98d0 unknown :-1 s=CLOSED pgs=1782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:14.574 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.568+0000 7f570781e640 1 -- 172.21.15.16:0/1533008050 >> 172.21.15.16:0/1533008050 conn(0x7f570006f5a0 msgr2=0x7f57001640b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:46:14.574 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.568+0000 7f570781e640 1 -- 172.21.15.16:0/1533008050 shutdown_connections 2023-05-22T15:46:14.574 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.568+0000 7f570781e640 1 -- 172.21.15.16:0/1533008050 wait complete. 2023-05-22T15:46:14.591 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2023-05-22T15:46:14.721 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.716+0000 7f8d24e30640 1 Processor -- start 2023-05-22T15:46:14.721 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.716+0000 7f8d24e30640 1 -- start start 2023-05-22T15:46:14.723 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.716+0000 7f8d24e30640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8d2006b6c0 0x7f8d200cb4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:14.723 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.716+0000 7f8d24e30640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f8d2006ba90 0x7f8d20158fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:14.723 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.716+0000 7f8d24e30640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f8d20159510 0x7f8d200cc740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:14.724 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.716+0000 7f8d24e30640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d20071760 con 0x7f8d2006ba90 2023-05-22T15:46:14.724 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.716+0000 7f8d24e30640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d200ca7d0 con 0x7f8d20159510 2023-05-22T15:46:14.724 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.716+0000 7f8d24e30640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d20159f70 con 0x7f8d2006b6c0 2023-05-22T15:46:14.724 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.716+0000 7f8d1dd9b640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f8d2006ba90 0x7f8d20158fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:14.724 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.716+0000 7f8d1dd9b640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f8d2006ba90 0x7f8d20158fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43518/0 (socket says 172.21.15.16:43518) 2023-05-22T15:46:14.725 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.716+0000 7f8d1dd9b640 1 -- 172.21.15.16:0/1712167707 learned_addr learned my addr 172.21.15.16:0/1712167707 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:46:14.725 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.720+0000 7f8d1e59c640 1 --2- 172.21.15.16:0/1712167707 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8d2006b6c0 0x7f8d200cb4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:14.725 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.720+0000 7f8d1ed9d640 1 --2- 172.21.15.16:0/1712167707 >> v2:172.21.15.62:3300/0 conn(0x7f8d20159510 0x7f8d200cc740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:14.726 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.720+0000 7f8d1dd9b640 1 -- 172.21.15.16:0/1712167707 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8d2006b6c0 msgr2=0x7f8d200cb4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:46:14.726 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.720+0000 7f8d1dd9b640 1 --2- 172.21.15.16:0/1712167707 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8d2006b6c0 0x7f8d200cb4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:14.726 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.720+0000 7f8d1dd9b640 1 -- 172.21.15.16:0/1712167707 >> v2:172.21.15.62:3300/0 conn(0x7f8d20159510 msgr2=0x7f8d200cc740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:46:14.726 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.720+0000 7f8d1dd9b640 1 --2- 172.21.15.16:0/1712167707 >> v2:172.21.15.62:3300/0 conn(0x7f8d20159510 0x7f8d200cc740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:14.726 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.720+0000 7f8d1dd9b640 1 -- 172.21.15.16:0/1712167707 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d200cb030 con 0x7f8d2006ba90 2023-05-22T15:46:14.726 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.720+0000 7f8d1e59c640 1 --2- 172.21.15.16:0/1712167707 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8d2006b6c0 0x7f8d200cb4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T15:46:14.729 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.720+0000 7f8d1dd9b640 1 --2- 172.21.15.16:0/1712167707 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8d2006ba90 0x7f8d20158fd0 secure :-1 s=READY pgs=1783 cs=0 l=1 rev1=1 crypto rx=0x7f8d2006ecd0 tx=0x7f8d1401c210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30bdfd69cf9933f0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:14.729 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.720+0000 7f8d1d59a640 1 -- 172.21.15.16:0/1712167707 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d140ab600 con 0x7f8d2006ba90 2023-05-22T15:46:14.730 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.720+0000 7f8d1d59a640 1 -- 172.21.15.16:0/1712167707 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d140a7050 con 0x7f8d2006ba90 2023-05-22T15:46:14.730 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.720+0000 7f8d1d59a640 1 -- 172.21.15.16:0/1712167707 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d140ab980 con 0x7f8d2006ba90 2023-05-22T15:46:14.730 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.720+0000 7f8d24e30640 1 -- 172.21.15.16:0/1712167707 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8d2006ba90 msgr2=0x7f8d20158fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:14.730 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.720+0000 7f8d24e30640 1 --2- 172.21.15.16:0/1712167707 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8d2006ba90 0x7f8d20158fd0 secure :-1 s=READY pgs=1783 cs=0 l=1 rev1=1 crypto rx=0x7f8d2006ecd0 tx=0x7f8d1401c210 comp rx=0 tx=0).stop 2023-05-22T15:46:14.730 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.720+0000 7f8d24e30640 1 -- 172.21.15.16:0/1712167707 shutdown_connections 2023-05-22T15:46:14.730 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.720+0000 7f8d24e30640 1 --2- 172.21.15.16:0/1712167707 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8d2006ba90 0x7f8d20158fd0 unknown :-1 s=CLOSED pgs=1783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:14.730 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.720+0000 7f8d24e30640 1 --2- 172.21.15.16:0/1712167707 >> v2:172.21.15.62:3300/0 conn(0x7f8d20159510 0x7f8d200cc740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:14.731 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.720+0000 7f8d24e30640 1 --2- 172.21.15.16:0/1712167707 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8d2006b6c0 0x7f8d200cb4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:14.731 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.720+0000 7f8d24e30640 1 -- 172.21.15.16:0/1712167707 >> 172.21.15.16:0/1712167707 conn(0x7f8d20069720 msgr2=0x7f8d20069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:46:14.731 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.720+0000 7f8d24e30640 1 -- 172.21.15.16:0/1712167707 shutdown_connections 2023-05-22T15:46:14.731 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.720+0000 7f8d24e30640 1 -- 172.21.15.16:0/1712167707 wait complete. 2023-05-22T15:46:14.731 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.724+0000 7f8d24e30640 1 Processor -- start 2023-05-22T15:46:14.731 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.724+0000 7f8d24e30640 1 -- start start 2023-05-22T15:46:14.731 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.724+0000 7f8d24e30640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8d2006b6c0 0x7f8d2014d820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:14.732 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.724+0000 7f8d24e30640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8d2006ba90 0x7f8d20156ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:14.732 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.724+0000 7f8d24e30640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8d20159510 0x7f8d20157020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:14.732 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.724+0000 7f8d24e30640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d200b2b60 con 0x7f8d2006b6c0 2023-05-22T15:46:14.732 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.724+0000 7f8d24e30640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d200b2e40 con 0x7f8d2006ba90 2023-05-22T15:46:14.733 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.724+0000 7f8d24e30640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d200b3120 con 0x7f8d20159510 2023-05-22T15:46:14.733 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.724+0000 7f8d1e59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8d2006b6c0 0x7f8d2014d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:14.733 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.724+0000 7f8d1e59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8d2006b6c0 0x7f8d2014d820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43530/0 (socket says 172.21.15.16:43530) 2023-05-22T15:46:14.733 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.724+0000 7f8d1e59c640 1 -- 172.21.15.16:0/834202061 learned_addr learned my addr 172.21.15.16:0/834202061 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:46:14.734 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.724+0000 7f8d1dd9b640 1 --2- 172.21.15.16:0/834202061 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8d2006ba90 0x7f8d20156ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:14.734 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.724+0000 7f8d1ed9d640 1 --2- 172.21.15.16:0/834202061 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8d20159510 0x7f8d20157020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:14.734 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.724+0000 7f8d1e59c640 1 -- 172.21.15.16:0/834202061 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8d20159510 msgr2=0x7f8d20157020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:14.734 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.724+0000 7f8d1e59c640 1 --2- 172.21.15.16:0/834202061 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8d20159510 0x7f8d20157020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:14.734 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.724+0000 7f8d1e59c640 1 -- 172.21.15.16:0/834202061 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8d2006ba90 msgr2=0x7f8d20156ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:14.735 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.724+0000 7f8d1e59c640 1 --2- 172.21.15.16:0/834202061 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8d2006ba90 0x7f8d20156ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:14.735 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.724+0000 7f8d1e59c640 1 -- 172.21.15.16:0/834202061 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d140094d0 con 0x7f8d2006b6c0 2023-05-22T15:46:14.736 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.724+0000 7f8d1e59c640 1 --2- 172.21.15.16:0/834202061 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8d2006b6c0 0x7f8d2014d820 secure :-1 s=READY pgs=1784 cs=0 l=1 rev1=1 crypto rx=0x7f8d1000e810 tx=0x7f8d1000ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:14.736 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.724+0000 7f8d0effd640 1 -- 172.21.15.16:0/834202061 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d10004030 con 0x7f8d2006b6c0 2023-05-22T15:46:14.736 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.724+0000 7f8d0effd640 1 -- 172.21.15.16:0/834202061 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d10004190 con 0x7f8d2006b6c0 2023-05-22T15:46:14.736 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.724+0000 7f8d0effd640 1 -- 172.21.15.16:0/834202061 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d10004520 con 0x7f8d2006b6c0 2023-05-22T15:46:14.737 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.724+0000 7f8d24e30640 1 -- 172.21.15.16:0/834202061 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d20159910 con 0x7f8d2006b6c0 2023-05-22T15:46:14.737 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.724+0000 7f8d24e30640 1 -- 172.21.15.16:0/834202061 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d200cacc0 con 0x7f8d2006b6c0 2023-05-22T15:46:14.739 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.724+0000 7f8d24e30640 1 -- 172.21.15.16:0/834202061 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d2006bee0 con 0x7f8d2006b6c0 2023-05-22T15:46:14.739 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.728+0000 7f8d0effd640 1 -- 172.21.15.16:0/834202061 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 60) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f8d1000c6d0 con 0x7f8d2006b6c0 2023-05-22T15:46:14.739 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.728+0000 7f8d0effd640 1 --2- 172.21.15.16:0/834202061 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f8d000654a0 0x7f8d00067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:14.740 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.728+0000 7f8d0effd640 1 -- 172.21.15.16:0/834202061 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(138..138 src has 1..138) v4 ==== 6332+0+0 (secure 0 0 0) 0x7f8d10082a80 con 0x7f8d2006b6c0 2023-05-22T15:46:14.740 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.728+0000 7f8d1dd9b640 1 --2- 172.21.15.16:0/834202061 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f8d000654a0 0x7f8d00067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:14.740 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.728+0000 7f8d1dd9b640 1 --2- 172.21.15.16:0/834202061 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f8d000654a0 0x7f8d00067950 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f8d140ad000 tx=0x7f8d14008040 comp rx=0 tx=0).ready entity=mgr.8835 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:14.741 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.732+0000 7f8d0effd640 1 -- 172.21.15.16:0/834202061 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f8d100502a0 con 0x7f8d2006b6c0 2023-05-22T15:46:14.892 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.888+0000 7f8d24e30640 1 -- 172.21.15.16:0/834202061 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7f8d2004dbe0 con 0x7f8d2006b6c0 2023-05-22T15:46:14.899 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.896+0000 7f8d0effd640 1 -- 172.21.15.16:0/834202061 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v45) v1 ==== 115+0+0 (secure 0 0 0) 0x7f8d100200b0 con 0x7f8d2006b6c0 2023-05-22T15:46:14.901 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.896+0000 7f8d24e30640 1 -- 172.21.15.16:0/834202061 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f8d000654a0 msgr2=0x7f8d00067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:14.902 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.896+0000 7f8d24e30640 1 --2- 172.21.15.16:0/834202061 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f8d000654a0 0x7f8d00067950 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f8d140ad000 tx=0x7f8d14008040 comp rx=0 tx=0).stop 2023-05-22T15:46:14.902 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.896+0000 7f8d24e30640 1 -- 172.21.15.16:0/834202061 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8d2006b6c0 msgr2=0x7f8d2014d820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:14.902 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.896+0000 7f8d24e30640 1 --2- 172.21.15.16:0/834202061 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8d2006b6c0 0x7f8d2014d820 secure :-1 s=READY pgs=1784 cs=0 l=1 rev1=1 crypto rx=0x7f8d1000e810 tx=0x7f8d1000ecd0 comp rx=0 tx=0).stop 2023-05-22T15:46:14.902 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.896+0000 7f8d24e30640 1 -- 172.21.15.16:0/834202061 shutdown_connections 2023-05-22T15:46:14.902 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.896+0000 7f8d24e30640 1 --2- 172.21.15.16:0/834202061 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f8d000654a0 0x7f8d00067950 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:14.902 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.896+0000 7f8d24e30640 1 --2- 172.21.15.16:0/834202061 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8d20159510 0x7f8d20157020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:14.903 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.896+0000 7f8d24e30640 1 --2- 172.21.15.16:0/834202061 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8d2006ba90 0x7f8d20156ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:14.903 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.896+0000 7f8d24e30640 1 --2- 172.21.15.16:0/834202061 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8d2006b6c0 0x7f8d2014d820 unknown :-1 s=CLOSED pgs=1784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:14.903 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.896+0000 7f8d24e30640 1 -- 172.21.15.16:0/834202061 >> 172.21.15.16:0/834202061 conn(0x7f8d20069720 msgr2=0x7f8d200c94f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:46:14.903 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.896+0000 7f8d24e30640 1 -- 172.21.15.16:0/834202061 shutdown_connections 2023-05-22T15:46:14.903 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:14.896+0000 7f8d24e30640 1 -- 172.21.15.16:0/834202061 wait complete. 2023-05-22T15:46:14.920 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule' 2023-05-22T15:46:15.037 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:15.032+0000 7f2b0652e640 1 Processor -- start 2023-05-22T15:46:15.038 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:15.032+0000 7f2b0652e640 1 -- start start 2023-05-22T15:46:15.039 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:15.032+0000 7f2b0652e640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2b00159a40 0x7f2b000c8ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:15.040 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:15.032+0000 7f2b0652e640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b0015a0a0 con 0x7f2b000b0250 2023-05-22T15:46:15.040 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:15.032+0000 7f2b0652e640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b00153580 con 0x7f2b001531b0 2023-05-22T15:46:15.040 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:15.032+0000 7f2b0652e640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b00159e90 con 0x7f2b00159a40 2023-05-22T15:46:15.040 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:15.032+0000 7f2af77fe640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f2b000b0250 0x7f2b00159330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:59188/0 (socket says 172.21.15.16:59188) 2023-05-22T15:46:15.041 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:15.032+0000 7f2af77fe640 1 -- 172.21.15.16:0/2831031494 learned_addr learned my addr 172.21.15.16:0/2831031494 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:46:15.041 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:15.032+0000 7f2b04acb640 1 --2- 172.21.15.16:0/2831031494 >> v2:172.21.15.16:3300/0 conn(0x7f2b00159a40 0x7f2b000c8ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:15.041 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:15.036+0000 7f2aff7fe640 1 -- 172.21.15.16:0/2831031494 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3386371551 0 0) 0x7f2b0015a0a0 con 0x7f2b000b0250 2023-05-22T15:46:15.042 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:15.036+0000 7f2aff7fe640 1 -- 172.21.15.16:0/2831031494 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ae0003530 con 0x7f2b000b0250 2023-05-22T15:46:15.042 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:15.036+0000 7f2b04acb640 1 -- 172.21.15.16:0/2831031494 >> v1:172.21.15.62:6789/0 conn(0x7f2b001531b0 legacy=0x7f2b000afb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:15.042 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:15.036+0000 7f2b04acb640 1 -- 172.21.15.16:0/2831031494 >> v1:172.21.15.16:6789/0 conn(0x7f2b000b0250 legacy=0x7f2b00159330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:15.042 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:15.036+0000 7f2b04acb640 1 -- 172.21.15.16:0/2831031494 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b0015d5e0 con 0x7f2b00159a40 2023-05-22T15:46:15.048 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:15.036+0000 7f2b04acb640 1 --2- 172.21.15.16:0/2831031494 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b00159a40 0x7f2b000c8ba0 secure :-1 s=READY pgs=1786 cs=0 l=1 rev1=1 crypto rx=0x7f2af8009d90 tx=0x7f2af801c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8903514a4dbe194 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:15.048 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:15.036+0000 7f2aff7fe640 1 -- 172.21.15.16:0/2831031494 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2af80a59e0 con 0x7f2b00159a40 2023-05-22T15:46:15.049 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:15.036+0000 7f2aff7fe640 1 -- 172.21.15.16:0/2831031494 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2af80a5b40 con 0x7f2b00159a40 2023-05-22T15:46:15.049 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:15.036+0000 7f2aff7fe640 1 -- 172.21.15.16:0/2831031494 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2af80ad440 con 0x7f2b00159a40 2023-05-22T15:46:15.049 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:15.036+0000 7f2b0652e640 1 -- 172.21.15.16:0/2831031494 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b00159a40 msgr2=0x7f2b000c8ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:15.049 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:15.036+0000 7f2b0652e640 1 --2- 172.21.15.16:0/2831031494 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b00159a40 0x7f2b000c8ba0 secure :-1 s=READY pgs=1786 cs=0 l=1 rev1=1 crypto rx=0x7f2af8009d90 tx=0x7f2af801c920 comp rx=0 tx=0).stop 2023-05-22T15:46:15.050 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:15.036+0000 7f2b0652e640 1 -- 172.21.15.16:0/2831031494 shutdown_connections 2023-05-22T15:46:15.050 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:15.036+0000 7f2b0652e640 1 --2- 172.21.15.16:0/2831031494 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b00159a40 0x7f2b000c8ba0 unknown :-1 s=CLOSED pgs=1786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:15.050 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:15.036+0000 7f2b0652e640 1 -- 172.21.15.16:0/2831031494 >> 172.21.15.16:0/2831031494 conn(0x7f2b00069720 msgr2=0x7f2b00069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:46:15.050 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:15.040+0000 7f2b0652e640 1 -- 172.21.15.16:0/2831031494 shutdown_connections 2023-05-22T15:46:15.051 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:15.040+0000 7f2b0652e640 1 -- 172.21.15.16:0/2831031494 wait complete. 2023-05-22T15:46:15.051 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:15.040+0000 7f2b0652e640 1 Processor -- start 2023-05-22T15:46:15.051 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:15.040+0000 7f2b0652e640 1 -- start start 2023-05-22T15:46:15.051 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:15.040+0000 7f2b0652e640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2b000b0250 0x7f2b0014a320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:15.052 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:15.040+0000 7f2b0652e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b001531b0 0x7f2b000c4150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:15.052 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:15.040+0000 7f2b0652e640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2b00159a40 0x7f2b000c4690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:15.052 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:15.040+0000 7f2b0652e640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b0014ac50 con 0x7f2b001531b0 2023-05-22T15:46:15.052 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:15.040+0000 7f2b0652e640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b0014af30 con 0x7f2b00159a40 2023-05-22T15:46:15.053 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:15.040+0000 7f2b0652e640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b0014b210 con 0x7f2b000b0250 2023-05-22T15:46:15.053 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:15.040+0000 7f2af77fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b001531b0 0x7f2b000c4150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:15.054 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:15.040+0000 7f2af77fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b001531b0 0x7f2b000c4150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43550/0 (socket says 172.21.15.16:43550) 2023-05-22T15:46:15.054 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:15.040+0000 7f2af77fe640 1 -- 172.21.15.16:0/3510114306 learned_addr learned my addr 172.21.15.16:0/3510114306 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:46:15.054 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:15.040+0000 7f2afffff640 1 --2- 172.21.15.16:0/3510114306 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2b000b0250 0x7f2b0014a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:15.055 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:15.040+0000 7f2b04acb640 1 --2- 172.21.15.16:0/3510114306 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2b00159a40 0x7f2b000c4690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:15.055 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:15.040+0000 7f2af77fe640 1 -- 172.21.15.16:0/3510114306 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2b000b0250 msgr2=0x7f2b0014a320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:15.055 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:15.040+0000 7f2af77fe640 1 --2- 172.21.15.16:0/3510114306 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2b000b0250 0x7f2b0014a320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:15.055 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:15.040+0000 7f2af77fe640 1 -- 172.21.15.16:0/3510114306 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2b00159a40 msgr2=0x7f2b000c4690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:15.056 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:15.040+0000 7f2af77fe640 1 --2- 172.21.15.16:0/3510114306 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2b00159a40 0x7f2b000c4690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:15.056 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:15.040+0000 7f2af77fe640 1 -- 172.21.15.16:0/3510114306 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2af8009a70 con 0x7f2b001531b0 2023-05-22T15:46:15.056 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:15.040+0000 7f2af77fe640 1 --2- 172.21.15.16:0/3510114306 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b001531b0 0x7f2b000c4150 secure :-1 s=READY pgs=1787 cs=0 l=1 rev1=1 crypto rx=0x7f2ae800c430 tx=0x7f2ae800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:15.057 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:15.040+0000 7f2afd7fa640 1 -- 172.21.15.16:0/3510114306 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ae8005cf0 con 0x7f2b001531b0 2023-05-22T15:46:15.057 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:15.040+0000 7f2afd7fa640 1 -- 172.21.15.16:0/3510114306 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ae8005e70 con 0x7f2b001531b0 2023-05-22T15:46:15.057 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:15.040+0000 7f2b0652e640 1 -- 172.21.15.16:0/3510114306 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b0015d0c0 con 0x7f2b001531b0 2023-05-22T15:46:15.057 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:15.040+0000 7f2afd7fa640 1 -- 172.21.15.16:0/3510114306 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ae80072d0 con 0x7f2b001531b0 2023-05-22T15:46:15.058 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:15.040+0000 7f2b0652e640 1 -- 172.21.15.16:0/3510114306 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b0015cde0 con 0x7f2b001531b0 2023-05-22T15:46:15.058 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:15.040+0000 7f2b0652e640 1 -- 172.21.15.16:0/3510114306 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b00070d40 con 0x7f2b001531b0 2023-05-22T15:46:15.058 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:15.044+0000 7f2afd7fa640 1 -- 172.21.15.16:0/3510114306 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 60) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f2ae8007430 con 0x7f2b001531b0 2023-05-22T15:46:15.058 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:15.044+0000 7f2afd7fa640 1 --2- 172.21.15.16:0/3510114306 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f2ae40654a0 0x7f2ae4067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:15.059 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:15.044+0000 7f2afd7fa640 1 -- 172.21.15.16:0/3510114306 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(138..138 src has 1..138) v4 ==== 6332+0+0 (secure 0 0 0) 0x7f2ae8081db0 con 0x7f2b001531b0 2023-05-22T15:46:15.059 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:15.044+0000 7f2afffff640 1 --2- 172.21.15.16:0/3510114306 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f2ae40654a0 0x7f2ae4067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:15.059 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:15.044+0000 7f2afffff640 1 --2- 172.21.15.16:0/3510114306 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f2ae40654a0 0x7f2ae4067950 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f2af0000f70 tx=0x7f2af0009040 comp rx=0 tx=0).ready entity=mgr.8835 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:15.060 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:15.048+0000 7f2afd7fa640 1 -- 172.21.15.16:0/3510114306 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f2ae804f780 con 0x7f2b001531b0 2023-05-22T15:46:15.257 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:15.252+0000 7f2b0652e640 1 -- 172.21.15.16:0/3510114306 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule"]} v 0) v1 -- 0x7f2b000cb8e0 con 0x7f2b001531b0 2023-05-22T15:46:15.894 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:15.888+0000 7f2afd7fa640 1 -- 172.21.15.16:0/3510114306 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule"]}]=0 v740) v1 ==== 148+0+0 (secure 0 0 0) 0x7f2ae8017020 con 0x7f2b001531b0 2023-05-22T15:46:15.900 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:15.896+0000 7f2b0652e640 1 -- 172.21.15.16:0/3510114306 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f2ae40654a0 msgr2=0x7f2ae4067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:15.900 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:15.896+0000 7f2b0652e640 1 --2- 172.21.15.16:0/3510114306 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f2ae40654a0 0x7f2ae4067950 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f2af0000f70 tx=0x7f2af0009040 comp rx=0 tx=0).stop 2023-05-22T15:46:15.900 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:15.896+0000 7f2b0652e640 1 -- 172.21.15.16:0/3510114306 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b001531b0 msgr2=0x7f2b000c4150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:15.900 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:15.896+0000 7f2b0652e640 1 --2- 172.21.15.16:0/3510114306 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b001531b0 0x7f2b000c4150 secure :-1 s=READY pgs=1787 cs=0 l=1 rev1=1 crypto rx=0x7f2ae800c430 tx=0x7f2ae800c8f0 comp rx=0 tx=0).stop 2023-05-22T15:46:15.901 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:15.896+0000 7f2b0652e640 1 -- 172.21.15.16:0/3510114306 shutdown_connections 2023-05-22T15:46:15.901 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:15.896+0000 7f2b0652e640 1 --2- 172.21.15.16:0/3510114306 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2b00159a40 0x7f2b000c4690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:15.901 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:15.896+0000 7f2b0652e640 1 --2- 172.21.15.16:0/3510114306 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b001531b0 0x7f2b000c4150 unknown :-1 s=CLOSED pgs=1787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:15.901 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:15.896+0000 7f2b0652e640 1 --2- 172.21.15.16:0/3510114306 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f2ae40654a0 0x7f2ae4067950 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:15.901 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:15.896+0000 7f2b0652e640 1 --2- 172.21.15.16:0/3510114306 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2b000b0250 0x7f2b0014a320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:15.901 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:15.896+0000 7f2b0652e640 1 -- 172.21.15.16:0/3510114306 >> 172.21.15.16:0/3510114306 conn(0x7f2b00069720 msgr2=0x7f2b000caa30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:46:15.901 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:15.896+0000 7f2b0652e640 1 -- 172.21.15.16:0/3510114306 shutdown_connections 2023-05-22T15:46:15.901 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:15.896+0000 7f2b0652e640 1 -- 172.21.15.16:0/3510114306 wait complete. 2023-05-22T15:46:15.921 INFO:tasks.cephfs_test_runner:Test existence of a scheduled snapshot ... ok 2023-05-22T15:46:15.922 INFO:tasks.cephfs_test_runner:Starting test: test_snap_schedule_list_post_schedule_remove (tasks.cephfs.test_snap_schedules.TestSnapSchedules) Test listing snap schedules post removal of a schedule 2023-05-22T15:46:15.922 INFO:tasks.cephfs_test_runner:test_snap_schedule_list_post_schedule_remove (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2023-05-22T15:46:15.923 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_list_post_schedule_remove' 2023-05-22T15:46:16.044 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:16.040+0000 7f9098e4f640 1 Processor -- start 2023-05-22T15:46:16.044 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:16.040+0000 7f9098e4f640 1 -- start start 2023-05-22T15:46:16.046 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:16.040+0000 7f9098e4f640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f909415c130 0x7f90940ceb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:16.046 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:16.040+0000 7f9098e4f640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f90940b24a0 0x7f909406dd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:16.046 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:16.040+0000 7f9098e4f640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f909406bfd0 con 0x7f909406e270 2023-05-22T15:46:16.046 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:16.040+0000 7f9098e4f640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90940d1170 con 0x7f909415c130 2023-05-22T15:46:16.047 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:16.040+0000 7f9098e4f640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f909406e6f0 con 0x7f90940b24a0 2023-05-22T15:46:16.047 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:16.040+0000 7f909259c640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f909415c130 0x7f90940ceb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:16.047 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:16.040+0000 7f909259c640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f909415c130 0x7f90940ceb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43554/0 (socket says 172.21.15.16:43554) 2023-05-22T15:46:16.048 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:16.040+0000 7f909259c640 1 -- 172.21.15.16:0/3502418291 learned_addr learned my addr 172.21.15.16:0/3502418291 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:46:16.048 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:16.040+0000 7f9091d9b640 1 --2- 172.21.15.16:0/3502418291 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f90940b24a0 0x7f909406dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:16.048 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:16.040+0000 7f909259c640 1 -- 172.21.15.16:0/3502418291 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f90940b24a0 msgr2=0x7f909406dd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:46:16.048 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:16.040+0000 7f909259c640 1 --2- 172.21.15.16:0/3502418291 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f90940b24a0 0x7f909406dd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:16.048 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:16.040+0000 7f909259c640 1 -- 172.21.15.16:0/3502418291 >> v1:172.21.15.16:6789/0 conn(0x7f909406e270 legacy=0x7f909415fb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:16.048 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:16.040+0000 7f909259c640 1 -- 172.21.15.16:0/3502418291 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f909406c830 con 0x7f909415c130 2023-05-22T15:46:16.049 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:16.044+0000 7f909259c640 1 --2- 172.21.15.16:0/3502418291 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f909415c130 0x7f90940ceb70 secure :-1 s=READY pgs=1789 cs=0 l=1 rev1=1 crypto rx=0x7f9084009e50 tx=0x7f908401c980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75661c18afe6aae5 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:16.050 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:16.044+0000 7f909159a640 1 -- 172.21.15.16:0/3502418291 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90840a5650 con 0x7f909415c130 2023-05-22T15:46:16.050 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:16.044+0000 7f909159a640 1 -- 172.21.15.16:0/3502418291 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90840a57b0 con 0x7f909415c130 2023-05-22T15:46:16.050 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:16.044+0000 7f909159a640 1 -- 172.21.15.16:0/3502418291 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90840a5b40 con 0x7f909415c130 2023-05-22T15:46:16.050 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:16.044+0000 7f9098e4f640 1 -- 172.21.15.16:0/3502418291 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f909415c130 msgr2=0x7f90940ceb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:16.050 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:16.044+0000 7f9098e4f640 1 --2- 172.21.15.16:0/3502418291 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f909415c130 0x7f90940ceb70 secure :-1 s=READY pgs=1789 cs=0 l=1 rev1=1 crypto rx=0x7f9084009e50 tx=0x7f908401c980 comp rx=0 tx=0).stop 2023-05-22T15:46:16.050 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:16.044+0000 7f9098e4f640 1 -- 172.21.15.16:0/3502418291 shutdown_connections 2023-05-22T15:46:16.051 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:16.044+0000 7f9098e4f640 1 --2- 172.21.15.16:0/3502418291 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f90940b24a0 0x7f909406dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:16.051 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:16.044+0000 7f9098e4f640 1 --2- 172.21.15.16:0/3502418291 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f909415c130 0x7f90940ceb70 unknown :-1 s=CLOSED pgs=1789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:16.051 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:16.044+0000 7f9098e4f640 1 -- 172.21.15.16:0/3502418291 >> 172.21.15.16:0/3502418291 conn(0x7f909406f5c0 msgr2=0x7f909406f9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:46:16.051 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:16.044+0000 7f9098e4f640 1 -- 172.21.15.16:0/3502418291 shutdown_connections 2023-05-22T15:46:16.051 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:16.044+0000 7f9098e4f640 1 -- 172.21.15.16:0/3502418291 wait complete. 2023-05-22T15:46:16.051 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:16.044+0000 7f9098e4f640 1 Processor -- start 2023-05-22T15:46:16.051 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:16.044+0000 7f9098e4f640 1 -- start start 2023-05-22T15:46:16.053 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:16.044+0000 7f9098e4f640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f909406e270 0x7f9094155790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:16.054 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:16.044+0000 7f9098e4f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f90940b24a0 0x7f9094153a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:16.054 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:16.044+0000 7f9098e4f640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f909415c130 0x7f9094158690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:16.054 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:16.044+0000 7f9098e4f640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9094157980 con 0x7f90940b24a0 2023-05-22T15:46:16.054 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:16.048+0000 7f9098e4f640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9094156d00 con 0x7f909406e270 2023-05-22T15:46:16.054 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:16.048+0000 7f9098e4f640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9094156fe0 con 0x7f909415c130 2023-05-22T15:46:16.054 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:16.048+0000 7f9091d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f90940b24a0 0x7f9094153a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:16.054 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:16.048+0000 7f9091d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f90940b24a0 0x7f9094153a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43564/0 (socket says 172.21.15.16:43564) 2023-05-22T15:46:16.055 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:16.048+0000 7f9091d9b640 1 -- 172.21.15.16:0/1325044972 learned_addr learned my addr 172.21.15.16:0/1325044972 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:46:16.055 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:16.048+0000 7f909259c640 1 --2- 172.21.15.16:0/1325044972 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f909406e270 0x7f9094155790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:16.055 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:16.048+0000 7f9092d9d640 1 --2- 172.21.15.16:0/1325044972 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f909415c130 0x7f9094158690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:16.055 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:16.048+0000 7f9091d9b640 1 -- 172.21.15.16:0/1325044972 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f909415c130 msgr2=0x7f9094158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:16.055 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:16.048+0000 7f9091d9b640 1 --2- 172.21.15.16:0/1325044972 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f909415c130 0x7f9094158690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:16.055 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:16.048+0000 7f9091d9b640 1 -- 172.21.15.16:0/1325044972 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f909406e270 msgr2=0x7f9094155790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:16.055 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:16.048+0000 7f9091d9b640 1 --2- 172.21.15.16:0/1325044972 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f909406e270 0x7f9094155790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:16.055 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:16.048+0000 7f9091d9b640 1 -- 172.21.15.16:0/1325044972 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9084009b30 con 0x7f90940b24a0 2023-05-22T15:46:16.056 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:16.048+0000 7f909259c640 1 --2- 172.21.15.16:0/1325044972 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f909406e270 0x7f9094155790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:46:16.056 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:16.048+0000 7f9091d9b640 1 --2- 172.21.15.16:0/1325044972 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f90940b24a0 0x7f9094153a40 secure :-1 s=READY pgs=1790 cs=0 l=1 rev1=1 crypto rx=0x7f908800d7d0 tx=0x7f908800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:16.056 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:16.048+0000 7f9082ffd640 1 -- 172.21.15.16:0/1325044972 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f908800b6c0 con 0x7f90940b24a0 2023-05-22T15:46:16.056 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:16.048+0000 7f9082ffd640 1 -- 172.21.15.16:0/1325044972 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f908800b820 con 0x7f90940b24a0 2023-05-22T15:46:16.056 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:16.048+0000 7f9098e4f640 1 -- 172.21.15.16:0/1325044972 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90940ca9e0 con 0x7f90940b24a0 2023-05-22T15:46:16.056 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:16.048+0000 7f9082ffd640 1 -- 172.21.15.16:0/1325044972 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f908800bbb0 con 0x7f90940b24a0 2023-05-22T15:46:16.056 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:16.048+0000 7f9098e4f640 1 -- 172.21.15.16:0/1325044972 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f909406c550 con 0x7f90940b24a0 2023-05-22T15:46:16.056 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:16.048+0000 7f9098e4f640 1 -- 172.21.15.16:0/1325044972 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90940cacc0 con 0x7f90940b24a0 2023-05-22T15:46:16.057 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:16.048+0000 7f9082ffd640 1 -- 172.21.15.16:0/1325044972 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 60) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f908800bd10 con 0x7f90940b24a0 2023-05-22T15:46:16.057 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:16.048+0000 7f9082ffd640 1 --2- 172.21.15.16:0/1325044972 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f90740654a0 0x7f9074067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:16.058 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:16.048+0000 7f9082ffd640 1 -- 172.21.15.16:0/1325044972 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(138..138 src has 1..138) v4 ==== 6332+0+0 (secure 0 0 0) 0x7f9088082fb0 con 0x7f90940b24a0 2023-05-22T15:46:16.058 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:16.048+0000 7f909259c640 1 --2- 172.21.15.16:0/1325044972 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f90740654a0 0x7f9074067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:16.058 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:16.048+0000 7f909259c640 1 --2- 172.21.15.16:0/1325044972 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f90740654a0 0x7f9074067950 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f90840ae000 tx=0x7f90840a7070 comp rx=0 tx=0).ready entity=mgr.8835 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:16.058 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:16.052+0000 7f9082ffd640 1 -- 172.21.15.16:0/1325044972 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f9088050a10 con 0x7f90940b24a0 2023-05-22T15:46:16.257 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:16.252+0000 7f9098e4f640 1 -- 172.21.15.16:0/1325044972 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_list_post_schedule_remove"]} v 0) v1 -- 0x7f9094157ac0 con 0x7f90940b24a0 2023-05-22T15:46:16.897 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:16.892+0000 7f9082ffd640 1 -- 172.21.15.16:0/1325044972 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_list_post_schedule_remove"]}]=0 v741) v1 ==== 177+0+0 (secure 0 0 0) 0x7f9088054030 con 0x7f90940b24a0 2023-05-22T15:46:16.901 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:16.896+0000 7f9098e4f640 1 -- 172.21.15.16:0/1325044972 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f90740654a0 msgr2=0x7f9074067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:16.901 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:16.896+0000 7f9098e4f640 1 --2- 172.21.15.16:0/1325044972 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f90740654a0 0x7f9074067950 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f90840ae000 tx=0x7f90840a7070 comp rx=0 tx=0).stop 2023-05-22T15:46:16.901 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:16.896+0000 7f9098e4f640 1 -- 172.21.15.16:0/1325044972 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f90940b24a0 msgr2=0x7f9094153a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:16.901 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:16.896+0000 7f9098e4f640 1 --2- 172.21.15.16:0/1325044972 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f90940b24a0 0x7f9094153a40 secure :-1 s=READY pgs=1790 cs=0 l=1 rev1=1 crypto rx=0x7f908800d7d0 tx=0x7f908800dc90 comp rx=0 tx=0).stop 2023-05-22T15:46:16.901 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:16.896+0000 7f9098e4f640 1 -- 172.21.15.16:0/1325044972 shutdown_connections 2023-05-22T15:46:16.901 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:16.896+0000 7f9098e4f640 1 --2- 172.21.15.16:0/1325044972 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f90740654a0 0x7f9074067950 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:16.901 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:16.896+0000 7f9098e4f640 1 --2- 172.21.15.16:0/1325044972 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f909415c130 0x7f9094158690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:16.902 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:16.896+0000 7f9098e4f640 1 --2- 172.21.15.16:0/1325044972 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f90940b24a0 0x7f9094153a40 unknown :-1 s=CLOSED pgs=1790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:16.902 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:16.896+0000 7f9098e4f640 1 --2- 172.21.15.16:0/1325044972 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f909406e270 0x7f9094155790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:16.902 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:16.896+0000 7f9098e4f640 1 -- 172.21.15.16:0/1325044972 >> 172.21.15.16:0/1325044972 conn(0x7f909406f5c0 msgr2=0x7f90940b5760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:46:16.902 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:16.896+0000 7f9098e4f640 1 -- 172.21.15.16:0/1325044972 shutdown_connections 2023-05-22T15:46:16.903 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:16.896+0000 7f9098e4f640 1 -- 172.21.15.16:0/1325044972 wait complete. 2023-05-22T15:46:16.924 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-05-22T15:46:17.059 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:17.052+0000 7fcda2ef4640 1 Processor -- start 2023-05-22T15:46:17.059 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:17.052+0000 7fcda2ef4640 1 -- start start 2023-05-22T15:46:17.061 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:17.056+0000 7fcda2ef4640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fcd9c159970 0x7fcd9c159d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:17.062 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:17.056+0000 7fcda2ef4640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fcd9c15d680 0x7fcd9c0cb260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:17.062 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:17.056+0000 7fcda2ef4640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fcd9c15dad0 0x7fcd9c06b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:17.062 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:17.056+0000 7fcda2ef4640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd9c070c30 con 0x7fcd9c15dad0 2023-05-22T15:46:17.062 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:17.056+0000 7fcda2ef4640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd9c070e60 con 0x7fcd9c159970 2023-05-22T15:46:17.063 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:17.056+0000 7fcda2ef4640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd9c071140 con 0x7fcd9c15d680 2023-05-22T15:46:17.063 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:17.056+0000 7fcda1491640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fcd9c15dad0 0x7fcd9c06b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:17.063 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:17.056+0000 7fcda1491640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fcd9c15dad0 0x7fcd9c06b6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43574/0 (socket says 172.21.15.16:43574) 2023-05-22T15:46:17.063 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:17.056+0000 7fcda1491640 1 -- 172.21.15.16:0/3089021187 learned_addr learned my addr 172.21.15.16:0/3089021187 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:46:17.063 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:17.056+0000 7fcd9bfff640 1 --2- 172.21.15.16:0/3089021187 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fcd9c15d680 0x7fcd9c0cb260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:17.064 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:17.056+0000 7fcda0c90640 1 --2- 172.21.15.16:0/3089021187 >> v2:172.21.15.62:3300/0 conn(0x7fcd9c159970 0x7fcd9c159d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:17.064 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:17.056+0000 7fcda1491640 1 -- 172.21.15.16:0/3089021187 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fcd9c15d680 msgr2=0x7fcd9c0cb260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:46:17.064 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:17.056+0000 7fcda1491640 1 --2- 172.21.15.16:0/3089021187 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fcd9c15d680 0x7fcd9c0cb260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:17.064 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:17.056+0000 7fcda1491640 1 -- 172.21.15.16:0/3089021187 >> v2:172.21.15.62:3300/0 conn(0x7fcd9c159970 msgr2=0x7fcd9c159d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:46:17.064 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:17.056+0000 7fcda1491640 1 --2- 172.21.15.16:0/3089021187 >> v2:172.21.15.62:3300/0 conn(0x7fcd9c159970 0x7fcd9c159d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:17.064 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:17.056+0000 7fcda1491640 1 -- 172.21.15.16:0/3089021187 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd9c071420 con 0x7fcd9c15dad0 2023-05-22T15:46:17.065 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:17.056+0000 7fcda0c90640 1 --2- 172.21.15.16:0/3089021187 >> v2:172.21.15.62:3300/0 conn(0x7fcd9c159970 0x7fcd9c159d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:46:17.067 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:17.056+0000 7fcda1491640 1 --2- 172.21.15.16:0/3089021187 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcd9c15dad0 0x7fcd9c06b6c0 secure :-1 s=READY pgs=1791 cs=0 l=1 rev1=1 crypto rx=0x7fcd9c06daf0 tx=0x7fcd9401c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=431b1f5b46e3ecbc server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:17.067 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:17.056+0000 7fcd9b7fe640 1 -- 172.21.15.16:0/3089021187 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd940a56a0 con 0x7fcd9c15dad0 2023-05-22T15:46:17.068 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:17.056+0000 7fcd9b7fe640 1 -- 172.21.15.16:0/3089021187 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd940a5800 con 0x7fcd9c15dad0 2023-05-22T15:46:17.068 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:17.056+0000 7fcd9b7fe640 1 -- 172.21.15.16:0/3089021187 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd940a5b90 con 0x7fcd9c15dad0 2023-05-22T15:46:17.068 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:17.060+0000 7fcda2ef4640 1 -- 172.21.15.16:0/3089021187 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcd9c15dad0 msgr2=0x7fcd9c06b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:17.068 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:17.060+0000 7fcda2ef4640 1 --2- 172.21.15.16:0/3089021187 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcd9c15dad0 0x7fcd9c06b6c0 secure :-1 s=READY pgs=1791 cs=0 l=1 rev1=1 crypto rx=0x7fcd9c06daf0 tx=0x7fcd9401c910 comp rx=0 tx=0).stop 2023-05-22T15:46:17.068 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:17.060+0000 7fcda2ef4640 1 -- 172.21.15.16:0/3089021187 shutdown_connections 2023-05-22T15:46:17.068 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:17.060+0000 7fcda2ef4640 1 --2- 172.21.15.16:0/3089021187 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcd9c15dad0 0x7fcd9c06b6c0 unknown :-1 s=CLOSED pgs=1791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:17.069 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:17.060+0000 7fcda2ef4640 1 --2- 172.21.15.16:0/3089021187 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fcd9c15d680 0x7fcd9c0cb260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:17.069 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:17.060+0000 7fcda2ef4640 1 --2- 172.21.15.16:0/3089021187 >> v2:172.21.15.62:3300/0 conn(0x7fcd9c159970 0x7fcd9c159d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:17.069 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:17.060+0000 7fcda2ef4640 1 -- 172.21.15.16:0/3089021187 >> 172.21.15.16:0/3089021187 conn(0x7fcd9c069720 msgr2=0x7fcd9c069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:46:17.069 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:17.060+0000 7fcda2ef4640 1 -- 172.21.15.16:0/3089021187 shutdown_connections 2023-05-22T15:46:17.069 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:17.060+0000 7fcda2ef4640 1 -- 172.21.15.16:0/3089021187 wait complete. 2023-05-22T15:46:17.069 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:17.060+0000 7fcda2ef4640 1 Processor -- start 2023-05-22T15:46:17.069 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:17.060+0000 7fcda2ef4640 1 -- start start 2023-05-22T15:46:17.070 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:17.060+0000 7fcda2ef4640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcd9c159970 0x7fcd9c164240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:17.070 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:17.060+0000 7fcda2ef4640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fcd9c15d680 0x7fcd9c0cbbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:17.070 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:17.060+0000 7fcda2ef4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcd9c15dad0 0x7fcd9c1f3a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:17.070 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:17.060+0000 7fcda2ef4640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd9c0cc2e0 con 0x7fcd9c15dad0 2023-05-22T15:46:17.070 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:17.060+0000 7fcda2ef4640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd9c0cc5c0 con 0x7fcd9c159970 2023-05-22T15:46:17.070 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:17.060+0000 7fcda2ef4640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd9c0cc8a0 con 0x7fcd9c15d680 2023-05-22T15:46:17.070 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:17.060+0000 7fcda1491640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcd9c15dad0 0x7fcd9c1f3a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:17.070 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:17.060+0000 7fcda1491640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcd9c15dad0 0x7fcd9c1f3a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43582/0 (socket says 172.21.15.16:43582) 2023-05-22T15:46:17.071 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:17.060+0000 7fcda1491640 1 -- 172.21.15.16:0/2631905308 learned_addr learned my addr 172.21.15.16:0/2631905308 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:46:17.071 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:17.060+0000 7fcda0c90640 1 --2- 172.21.15.16:0/2631905308 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcd9c159970 0x7fcd9c164240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:17.071 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:17.060+0000 7fcd9bfff640 1 --2- 172.21.15.16:0/2631905308 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fcd9c15d680 0x7fcd9c0cbbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:17.071 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:17.060+0000 7fcda1491640 1 -- 172.21.15.16:0/2631905308 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fcd9c15d680 msgr2=0x7fcd9c0cbbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:17.071 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:17.060+0000 7fcda1491640 1 --2- 172.21.15.16:0/2631905308 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fcd9c15d680 0x7fcd9c0cbbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:17.071 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:17.060+0000 7fcda1491640 1 -- 172.21.15.16:0/2631905308 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcd9c159970 msgr2=0x7fcd9c164240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:17.071 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:17.060+0000 7fcda1491640 1 --2- 172.21.15.16:0/2631905308 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcd9c159970 0x7fcd9c164240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:17.072 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:17.060+0000 7fcda1491640 1 -- 172.21.15.16:0/2631905308 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd94009b30 con 0x7fcd9c15dad0 2023-05-22T15:46:17.072 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:17.060+0000 7fcda0c90640 1 --2- 172.21.15.16:0/2631905308 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcd9c159970 0x7fcd9c164240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T15:46:17.072 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:17.060+0000 7fcda1491640 1 --2- 172.21.15.16:0/2631905308 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcd9c15dad0 0x7fcd9c1f3a80 secure :-1 s=READY pgs=1792 cs=0 l=1 rev1=1 crypto rx=0x7fcd940af040 tx=0x7fcd940196b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:17.072 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:17.060+0000 7fcd997fa640 1 -- 172.21.15.16:0/2631905308 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd940b8670 con 0x7fcd9c15dad0 2023-05-22T15:46:17.072 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:17.060+0000 7fcd997fa640 1 -- 172.21.15.16:0/2631905308 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd940b87f0 con 0x7fcd9c15dad0 2023-05-22T15:46:17.072 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:17.060+0000 7fcda2ef4640 1 -- 172.21.15.16:0/2631905308 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd9c0cae20 con 0x7fcd9c15dad0 2023-05-22T15:46:17.073 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:17.060+0000 7fcd997fa640 1 -- 172.21.15.16:0/2631905308 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd940b8ba0 con 0x7fcd9c15dad0 2023-05-22T15:46:17.073 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:17.060+0000 7fcda2ef4640 1 -- 172.21.15.16:0/2631905308 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd9c0cac90 con 0x7fcd9c15dad0 2023-05-22T15:46:17.073 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:17.064+0000 7fcda2ef4640 1 -- 172.21.15.16:0/2631905308 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd9c15adc0 con 0x7fcd9c15dad0 2023-05-22T15:46:17.073 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:17.064+0000 7fcd997fa640 1 -- 172.21.15.16:0/2631905308 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 60) v1 ==== 84332+0+0 (secure 0 0 0) 0x7fcd940b8d00 con 0x7fcd9c15dad0 2023-05-22T15:46:17.074 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:17.064+0000 7fcd997fa640 1 --2- 172.21.15.16:0/2631905308 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fcd880654a0 0x7fcd88067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:17.074 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:17.064+0000 7fcda0c90640 1 --2- 172.21.15.16:0/2631905308 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fcd880654a0 0x7fcd88067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:17.075 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:17.064+0000 7fcd997fa640 1 -- 172.21.15.16:0/2631905308 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(138..138 src has 1..138) v4 ==== 6332+0+0 (secure 0 0 0) 0x7fcd9411e0a0 con 0x7fcd9c15dad0 2023-05-22T15:46:17.075 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:17.064+0000 7fcda0c90640 1 --2- 172.21.15.16:0/2631905308 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fcd880654a0 0x7fcd88067950 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fcd8c006b20 tx=0x7fcd8c008040 comp rx=0 tx=0).ready entity=mgr.8835 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:17.075 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:17.068+0000 7fcd997fa640 1 -- 172.21.15.16:0/2631905308 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fcd940ebbb0 con 0x7fcd9c15dad0 2023-05-22T15:46:17.236 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:17.232+0000 7fcda2ef4640 1 -- 172.21.15.16:0/2631905308 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fcd9c04dbe0 con 0x7fcd9c15dad0 2023-05-22T15:46:17.242 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:17.236+0000 7fcd997fa640 1 -- 172.21.15.16:0/2631905308 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v46)=0 v46) v1 ==== 122+0+0 (secure 0 0 0) 0x7fcd940ef1d0 con 0x7fcd9c15dad0 2023-05-22T15:46:17.245 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:17.240+0000 7fcda2ef4640 1 -- 172.21.15.16:0/2631905308 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fcd880654a0 msgr2=0x7fcd88067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:17.245 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:17.240+0000 7fcda2ef4640 1 --2- 172.21.15.16:0/2631905308 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fcd880654a0 0x7fcd88067950 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fcd8c006b20 tx=0x7fcd8c008040 comp rx=0 tx=0).stop 2023-05-22T15:46:17.245 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:17.240+0000 7fcda2ef4640 1 -- 172.21.15.16:0/2631905308 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcd9c15dad0 msgr2=0x7fcd9c1f3a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:17.245 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:17.240+0000 7fcda2ef4640 1 --2- 172.21.15.16:0/2631905308 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcd9c15dad0 0x7fcd9c1f3a80 secure :-1 s=READY pgs=1792 cs=0 l=1 rev1=1 crypto rx=0x7fcd940af040 tx=0x7fcd940196b0 comp rx=0 tx=0).stop 2023-05-22T15:46:17.245 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:17.240+0000 7fcda2ef4640 1 -- 172.21.15.16:0/2631905308 shutdown_connections 2023-05-22T15:46:17.245 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:17.240+0000 7fcda2ef4640 1 --2- 172.21.15.16:0/2631905308 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcd9c15dad0 0x7fcd9c1f3a80 unknown :-1 s=CLOSED pgs=1792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:17.246 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:17.240+0000 7fcda2ef4640 1 --2- 172.21.15.16:0/2631905308 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fcd880654a0 0x7fcd88067950 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:17.246 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:17.240+0000 7fcda2ef4640 1 --2- 172.21.15.16:0/2631905308 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fcd9c15d680 0x7fcd9c0cbbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:17.246 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:17.240+0000 7fcda2ef4640 1 --2- 172.21.15.16:0/2631905308 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcd9c159970 0x7fcd9c164240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:17.246 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:17.240+0000 7fcda2ef4640 1 -- 172.21.15.16:0/2631905308 >> 172.21.15.16:0/2631905308 conn(0x7fcd9c069720 msgr2=0x7fcd9c158c80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:46:17.246 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:17.240+0000 7fcda2ef4640 1 -- 172.21.15.16:0/2631905308 shutdown_connections 2023-05-22T15:46:17.246 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:17.240+0000 7fcda2ef4640 1 -- 172.21.15.16:0/2631905308 wait complete. 2023-05-22T15:46:17.262 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-05-22T15:46:17.262 DEBUG:teuthology.orchestra.run.smithi016:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-05-22T15:46:17.313 DEBUG:teuthology.orchestra.run.smithi062:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-05-22T15:46:17.374 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-05-22T15:46:17.375 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-05-22T15:46:17.375 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2023-05-22T15:46:17.381 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16345192k,nr_inodes=4086298,mode=755,inode64 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3276544k,mode=755,inode64 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev,inode64 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=15732 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3276540k,nr_inodes=819135,mode=700,uid=1000,gid=1271,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3276544k,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-05-22T15:46:17.381 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2023-05-22T15:46:17.382 DEBUG:teuthology.orchestra.run.smithi062:> dd if=/proc/self/mounts of=/dev/stdout 2023-05-22T15:46:17.388 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16342116k,nr_inodes=4085529,mode=755,inode64 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275932k,mode=755,inode64 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev,inode64 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=13250 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275928k,nr_inodes=818982,mode=700,uid=1000,gid=1271,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275932k,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-05-22T15:46:17.389 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-05-22T15:46:17.390 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2023-05-22T15:46:17.432 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16345192k,nr_inodes=4086298,mode=755,inode64 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3276544k,mode=755,inode64 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev,inode64 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=15732 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3276540k,nr_inodes=819135,mode=700,uid=1000,gid=1271,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3276544k,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-05-22T15:46:17.432 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2023-05-22T15:46:17.433 DEBUG:teuthology.orchestra.run.smithi062:> dd if=/proc/self/mounts of=/dev/stdout 2023-05-22T15:46:17.439 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16342116k,nr_inodes=4085529,mode=755,inode64 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275932k,mode=755,inode64 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev,inode64 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=13250 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275928k,nr_inodes=818982,mode=700,uid=1000,gid=1271,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275932k,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-05-22T15:46:17.439 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2023-05-22T15:46:17.440 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-05-22T15:46:17.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:17.604+0000 7fe679a84640 1 Processor -- start 2023-05-22T15:46:17.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:17.604+0000 7fe679a84640 1 -- start start 2023-05-22T15:46:17.610 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:17.604+0000 7fe679a84640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fe6741567c0 0x7fe6740afb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:17.610 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:17.604+0000 7fe679a84640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe6740cb6a0 0x7fe6740b0080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:17.610 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:17.604+0000 7fe679a84640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe674156b90 con 0x7fe674159340 2023-05-22T15:46:17.610 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:17.604+0000 7fe679a84640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe674159ec0 con 0x7fe6741567c0 2023-05-22T15:46:17.610 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:17.604+0000 7fe679a84640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe67415a0b0 con 0x7fe6740cb6a0 2023-05-22T15:46:17.610 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:17.604+0000 7fe672ffd640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe6740cb6a0 0x7fe6740b0080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:17.611 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:17.604+0000 7fe672ffd640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe6740cb6a0 0x7fe6740b0080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3301/0 says I am v2:172.21.15.16:40696/0 (socket says 172.21.15.16:40696) 2023-05-22T15:46:17.611 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:17.604+0000 7fe672ffd640 1 -- 172.21.15.16:0/2002825137 learned_addr learned my addr 172.21.15.16:0/2002825137 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:46:17.611 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:17.604+0000 7fe6737fe640 1 --2- 172.21.15.16:0/2002825137 >> v2:172.21.15.62:3300/0 conn(0x7fe6741567c0 0x7fe6740afb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:17.611 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:17.604+0000 7fe672ffd640 1 -- 172.21.15.16:0/2002825137 >> v2:172.21.15.62:3300/0 conn(0x7fe6741567c0 msgr2=0x7fe6740afb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:46:17.611 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:17.604+0000 7fe672ffd640 1 --2- 172.21.15.16:0/2002825137 >> v2:172.21.15.62:3300/0 conn(0x7fe6741567c0 0x7fe6740afb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:17.611 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:17.604+0000 7fe672ffd640 1 -- 172.21.15.16:0/2002825137 >> v1:172.21.15.62:6789/0 conn(0x7fe674159340 legacy=0x7fe6741597b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:17.611 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:17.604+0000 7fe672ffd640 1 -- 172.21.15.16:0/2002825137 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6740ca8e0 con 0x7fe6740cb6a0 2023-05-22T15:46:17.612 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:17.608+0000 7fe672ffd640 1 --2- 172.21.15.16:0/2002825137 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe6740cb6a0 0x7fe6740b0080 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fe65c009e50 tx=0x7fe65c01ca10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7b904d618f5109ac server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:17.613 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:17.608+0000 7fe6727fc640 1 -- 172.21.15.16:0/2002825137 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe65c0a56c0 con 0x7fe6740cb6a0 2023-05-22T15:46:17.614 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:17.608+0000 7fe6727fc640 1 -- 172.21.15.16:0/2002825137 <== mon.2 v2:172.21.15.62:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe65c0a5820 con 0x7fe6740cb6a0 2023-05-22T15:46:17.614 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:17.608+0000 7fe6727fc640 1 -- 172.21.15.16:0/2002825137 <== mon.2 v2:172.21.15.62:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe65c0a5bb0 con 0x7fe6740cb6a0 2023-05-22T15:46:17.615 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:17.608+0000 7fe679a84640 1 -- 172.21.15.16:0/2002825137 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe6740cb6a0 msgr2=0x7fe6740b0080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:17.615 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:17.608+0000 7fe679a84640 1 --2- 172.21.15.16:0/2002825137 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe6740cb6a0 0x7fe6740b0080 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fe65c009e50 tx=0x7fe65c01ca10 comp rx=0 tx=0).stop 2023-05-22T15:46:17.616 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:17.608+0000 7fe679a84640 1 -- 172.21.15.16:0/2002825137 shutdown_connections 2023-05-22T15:46:17.616 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:17.608+0000 7fe679a84640 1 --2- 172.21.15.16:0/2002825137 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe6740cb6a0 0x7fe6740b0080 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:17.616 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:17.608+0000 7fe679a84640 1 --2- 172.21.15.16:0/2002825137 >> v2:172.21.15.62:3300/0 conn(0x7fe6741567c0 0x7fe6740afb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:17.616 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:17.608+0000 7fe679a84640 1 -- 172.21.15.16:0/2002825137 >> 172.21.15.16:0/2002825137 conn(0x7fe674069720 msgr2=0x7fe674069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:46:17.616 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:17.608+0000 7fe679a84640 1 -- 172.21.15.16:0/2002825137 shutdown_connections 2023-05-22T15:46:17.616 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:17.608+0000 7fe679a84640 1 -- 172.21.15.16:0/2002825137 wait complete. 2023-05-22T15:46:17.618 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:17.608+0000 7fe679a84640 1 Processor -- start 2023-05-22T15:46:17.619 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:17.608+0000 7fe679a84640 1 -- start start 2023-05-22T15:46:17.619 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:17.612+0000 7fe679a84640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe6740cb6a0 0x7fe674156d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:17.619 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:17.612+0000 7fe679a84640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe6741567c0 0x7fe6741f77e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:17.619 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:17.612+0000 7fe679a84640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe674159340 0x7fe6741f7d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:17.619 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:17.612+0000 7fe679a84640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe674153200 con 0x7fe674159340 2023-05-22T15:46:17.619 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:17.612+0000 7fe679a84640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe67415a8f0 con 0x7fe6741567c0 2023-05-22T15:46:17.620 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:17.612+0000 7fe679a84640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe67415abd0 con 0x7fe6740cb6a0 2023-05-22T15:46:17.620 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:17.612+0000 7fe673fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe674159340 0x7fe6741f7d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:17.620 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:17.612+0000 7fe673fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe674159340 0x7fe6741f7d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43586/0 (socket says 172.21.15.16:43586) 2023-05-22T15:46:17.620 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:17.612+0000 7fe673fff640 1 -- 172.21.15.16:0/3503701220 learned_addr learned my addr 172.21.15.16:0/3503701220 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:46:17.620 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:17.612+0000 7fe6737fe640 1 --2- 172.21.15.16:0/3503701220 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe6740cb6a0 0x7fe674156d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:17.620 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:17.612+0000 7fe672ffd640 1 --2- 172.21.15.16:0/3503701220 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe6741567c0 0x7fe6741f77e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:17.620 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:17.612+0000 7fe673fff640 1 -- 172.21.15.16:0/3503701220 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe6740cb6a0 msgr2=0x7fe674156d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:17.621 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:17.612+0000 7fe673fff640 1 --2- 172.21.15.16:0/3503701220 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe6740cb6a0 0x7fe674156d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:17.621 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:17.612+0000 7fe673fff640 1 -- 172.21.15.16:0/3503701220 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe6741567c0 msgr2=0x7fe6741f77e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:17.621 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:17.612+0000 7fe673fff640 1 --2- 172.21.15.16:0/3503701220 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe6741567c0 0x7fe6741f77e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:17.621 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:17.612+0000 7fe673fff640 1 -- 172.21.15.16:0/3503701220 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe65c009b30 con 0x7fe674159340 2023-05-22T15:46:17.621 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:17.612+0000 7fe673fff640 1 --2- 172.21.15.16:0/3503701220 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe674159340 0x7fe6741f7d20 secure :-1 s=READY pgs=1793 cs=0 l=1 rev1=1 crypto rx=0x7fe66c00c310 tx=0x7fe66c00c7d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:17.621 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:17.612+0000 7fe66bfff640 1 -- 172.21.15.16:0/3503701220 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe66c0052d0 con 0x7fe674159340 2023-05-22T15:46:17.621 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:17.612+0000 7fe66bfff640 1 -- 172.21.15.16:0/3503701220 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe66c005430 con 0x7fe674159340 2023-05-22T15:46:17.621 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:17.612+0000 7fe66bfff640 1 -- 172.21.15.16:0/3503701220 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe66c003800 con 0x7fe674159340 2023-05-22T15:46:17.622 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:17.612+0000 7fe679a84640 1 -- 172.21.15.16:0/3503701220 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6740c8e80 con 0x7fe674159340 2023-05-22T15:46:17.622 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:17.612+0000 7fe679a84640 1 -- 172.21.15.16:0/3503701220 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6740cb180 con 0x7fe674159340 2023-05-22T15:46:17.622 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:17.612+0000 7fe679a84640 1 -- 172.21.15.16:0/3503701220 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6740c9010 con 0x7fe674159340 2023-05-22T15:46:17.622 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:17.612+0000 7fe66bfff640 1 -- 172.21.15.16:0/3503701220 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 60) v1 ==== 84332+0+0 (secure 0 0 0) 0x7fe66c003960 con 0x7fe674159340 2023-05-22T15:46:17.622 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:17.612+0000 7fe66bfff640 1 --2- 172.21.15.16:0/3503701220 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fe6540654a0 0x7fe654067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:17.622 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:17.612+0000 7fe66bfff640 1 -- 172.21.15.16:0/3503701220 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(138..138 src has 1..138) v4 ==== 6332+0+0 (secure 0 0 0) 0x7fe66c081c40 con 0x7fe674159340 2023-05-22T15:46:17.622 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:17.612+0000 7fe6737fe640 1 --2- 172.21.15.16:0/3503701220 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fe6540654a0 0x7fe654067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:17.623 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:17.616+0000 7fe6737fe640 1 --2- 172.21.15.16:0/3503701220 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fe6540654a0 0x7fe654067950 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fe664009b60 tx=0x7fe664009190 comp rx=0 tx=0).ready entity=mgr.8835 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:17.623 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:17.620+0000 7fe66bfff640 1 -- 172.21.15.16:0/3503701220 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fe66c04a620 con 0x7fe674159340 2023-05-22T15:46:17.810 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:17.804+0000 7fe679a84640 1 -- 172.21.15.16:0/3503701220 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe67415a750 con 0x7fe674159340 2023-05-22T15:46:17.811 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:17.808+0000 7fe66bfff640 1 -- 172.21.15.16:0/3503701220 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 52 v52) v1 ==== 94+0+3520 (secure 0 0 0) 0x7fe67415a750 con 0x7fe674159340 2023-05-22T15:46:17.811 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:46:17.812 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":52,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6689,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/296693182","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6833","nonce":296693182},{"type":"v1","addr":"172.21.15.62:6835","nonce":296693182}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":46},{"gid":7402,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6834/856426578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":856426578},{"type":"v1","addr":"172.21.15.62:6834","nonce":856426578}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":46},{"gid":8541,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/2260895833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2260895833},{"type":"v1","addr":"172.21.15.16:6835","nonce":2260895833}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":46},{"gid":8997,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/202189782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":202189782},{"type":"v1","addr":"172.21.15.16:6837","nonce":202189782}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":52}],"filesystems":[]} 2023-05-22T15:46:17.812 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 52 2023-05-22T15:46:17.813 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:17.808+0000 7fe679a84640 1 -- 172.21.15.16:0/3503701220 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fe6540654a0 msgr2=0x7fe654067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:17.813 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:17.808+0000 7fe679a84640 1 --2- 172.21.15.16:0/3503701220 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fe6540654a0 0x7fe654067950 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fe664009b60 tx=0x7fe664009190 comp rx=0 tx=0).stop 2023-05-22T15:46:17.814 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:17.808+0000 7fe679a84640 1 -- 172.21.15.16:0/3503701220 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe674159340 msgr2=0x7fe6741f7d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:17.814 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:17.808+0000 7fe679a84640 1 --2- 172.21.15.16:0/3503701220 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe674159340 0x7fe6741f7d20 secure :-1 s=READY pgs=1793 cs=0 l=1 rev1=1 crypto rx=0x7fe66c00c310 tx=0x7fe66c00c7d0 comp rx=0 tx=0).stop 2023-05-22T15:46:17.814 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:17.808+0000 7fe679a84640 1 -- 172.21.15.16:0/3503701220 shutdown_connections 2023-05-22T15:46:17.814 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:17.808+0000 7fe679a84640 1 --2- 172.21.15.16:0/3503701220 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe674159340 0x7fe6741f7d20 unknown :-1 s=CLOSED pgs=1793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:17.814 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:17.808+0000 7fe679a84640 1 --2- 172.21.15.16:0/3503701220 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe6741567c0 0x7fe6741f77e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:17.814 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:17.808+0000 7fe679a84640 1 --2- 172.21.15.16:0/3503701220 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fe6540654a0 0x7fe654067950 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:17.814 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:17.808+0000 7fe679a84640 1 --2- 172.21.15.16:0/3503701220 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe6740cb6a0 0x7fe674156d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:17.815 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:17.808+0000 7fe679a84640 1 -- 172.21.15.16:0/3503701220 >> 172.21.15.16:0/3503701220 conn(0x7fe674069720 msgr2=0x7fe6740ce020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:46:17.815 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:17.808+0000 7fe679a84640 1 -- 172.21.15.16:0/3503701220 shutdown_connections 2023-05-22T15:46:17.815 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:17.808+0000 7fe679a84640 1 -- 172.21.15.16:0/3503701220 wait complete. 2023-05-22T15:46:17.830 INFO:tasks.ceph.mds.a:Restarting daemon 2023-05-22T15:46:17.830 INFO:tasks.ceph.mds.a:Stopping old one... 2023-05-22T15:46:17.831 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-05-22T15:46:17.832 INFO:teuthology.orchestra.run:waiting for 300 2023-05-22T15:46:17.832 INFO:tasks.ceph.mds.c:Restarting daemon 2023-05-22T15:46:17.833 INFO:tasks.ceph.mds.c:Stopping old one... 2023-05-22T15:46:17.833 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-05-22T15:46:17.833 INFO:teuthology.orchestra.run:waiting for 300 2023-05-22T15:46:17.834 INFO:tasks.ceph.mds.b:Restarting daemon 2023-05-22T15:46:17.834 INFO:tasks.ceph.mds.b:Stopping old one... 2023-05-22T15:46:17.834 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-05-22T15:46:17.834 INFO:teuthology.orchestra.run:waiting for 300 2023-05-22T15:46:17.834 INFO:tasks.ceph.mds.d:Restarting daemon 2023-05-22T15:46:17.835 INFO:tasks.ceph.mds.d:Stopping old one... 2023-05-22T15:46:17.835 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-05-22T15:46:17.835 INFO:teuthology.orchestra.run:waiting for 300 2023-05-22T15:46:17.840 INFO:tasks.ceph.mds.a:Stopped 2023-05-22T15:46:17.841 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-05-22T15:46:17.844 INFO:tasks.ceph.mds.c:Stopped 2023-05-22T15:46:17.844 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-05-22T15:46:17.850 INFO:tasks.ceph.mds.b:Stopped 2023-05-22T15:46:17.851 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-05-22T15:46:17.854 INFO:tasks.ceph.mds.d:Stopped 2023-05-22T15:46:17.854 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-05-22T15:46:17.887 INFO:tasks.ceph.mds.a:Started 2023-05-22T15:46:17.888 DEBUG:teuthology.parallel:result is None 2023-05-22T15:46:17.889 INFO:tasks.ceph.mds.c:Started 2023-05-22T15:46:17.890 DEBUG:teuthology.parallel:result is None 2023-05-22T15:46:17.901 INFO:tasks.ceph.mds.b:Started 2023-05-22T15:46:17.901 DEBUG:teuthology.parallel:result is None 2023-05-22T15:46:17.902 INFO:tasks.ceph.mds.d:Started 2023-05-22T15:46:17.903 DEBUG:teuthology.parallel:result is None 2023-05-22T15:46:17.903 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2023-05-22T15:46:17.949 INFO:tasks.ceph.mds.c.smithi016.stdout:starting mds.c at 2023-05-22T15:46:17.959 INFO:tasks.ceph.mds.a.smithi016.stdout:starting mds.a at 2023-05-22T15:46:17.979 INFO:tasks.ceph.mds.d.smithi062.stdout:starting mds.d at 2023-05-22T15:46:17.979 INFO:tasks.ceph.mds.b.smithi062.stdout:starting mds.b at 2023-05-22T15:46:18.022 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:18.016+0000 7fe3c4589640 1 Processor -- start 2023-05-22T15:46:18.022 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:18.016+0000 7fe3c4589640 1 -- start start 2023-05-22T15:46:18.023 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:18.020+0000 7fe3c4589640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe3bc1563c0 0x7fe3bc0c7b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:18.024 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:18.020+0000 7fe3c4589640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fe3bc0cc6c0 0x7fe3bc0c8080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:18.024 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:18.020+0000 7fe3c4589640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3bc15c210 con 0x7fe3bc06c220 2023-05-22T15:46:18.024 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:18.020+0000 7fe3c4589640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3bc160700 con 0x7fe3bc1563c0 2023-05-22T15:46:18.024 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:18.020+0000 7fe3c4589640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3bc06cdd0 con 0x7fe3bc0cc6c0 2023-05-22T15:46:18.024 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:18.020+0000 7fe3c2325640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe3bc1563c0 0x7fe3bc0c7b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:18.024 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:18.020+0000 7fe3c2325640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe3bc1563c0 0x7fe3bc0c7b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43642/0 (socket says 172.21.15.16:43642) 2023-05-22T15:46:18.025 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:18.020+0000 7fe3c2325640 1 -- 172.21.15.16:0/3938532712 learned_addr learned my addr 172.21.15.16:0/3938532712 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:46:18.025 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:18.020+0000 7fe3c1b24640 1 --2- 172.21.15.16:0/3938532712 >> v2:172.21.15.62:3300/0 conn(0x7fe3bc0cc6c0 0x7fe3bc0c8080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:18.025 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:18.020+0000 7fe3c2325640 1 -- 172.21.15.16:0/3938532712 >> v2:172.21.15.62:3300/0 conn(0x7fe3bc0cc6c0 msgr2=0x7fe3bc0c8080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:46:18.025 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:18.020+0000 7fe3c2325640 1 --2- 172.21.15.16:0/3938532712 >> v2:172.21.15.62:3300/0 conn(0x7fe3bc0cc6c0 0x7fe3bc0c8080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:18.025 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:18.020+0000 7fe3c2325640 1 -- 172.21.15.16:0/3938532712 >> v1:172.21.15.62:6789/0 conn(0x7fe3bc06c220 legacy=0x7fe3bc06c6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:18.026 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:18.020+0000 7fe3c2325640 1 -- 172.21.15.16:0/3938532712 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3bc15c480 con 0x7fe3bc1563c0 2023-05-22T15:46:18.026 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:18.020+0000 7fe3c2325640 1 --2- 172.21.15.16:0/3938532712 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe3bc1563c0 0x7fe3bc0c7b40 secure :-1 s=READY pgs=1800 cs=0 l=1 rev1=1 crypto rx=0x7fe3bc070ff0 tx=0x7fe3b801c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7846fdc418626a46 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:18.028 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:18.024+0000 7fe3c1323640 1 -- 172.21.15.16:0/3938532712 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3b80aa430 con 0x7fe3bc1563c0 2023-05-22T15:46:18.028 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:18.024+0000 7fe3c1323640 1 -- 172.21.15.16:0/3938532712 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3b80aa590 con 0x7fe3bc1563c0 2023-05-22T15:46:18.028 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:18.024+0000 7fe3c1323640 1 -- 172.21.15.16:0/3938532712 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3b80aa920 con 0x7fe3bc1563c0 2023-05-22T15:46:18.029 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:18.024+0000 7fe3c4589640 1 -- 172.21.15.16:0/3938532712 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe3bc1563c0 msgr2=0x7fe3bc0c7b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:18.029 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:18.024+0000 7fe3c4589640 1 --2- 172.21.15.16:0/3938532712 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe3bc1563c0 0x7fe3bc0c7b40 secure :-1 s=READY pgs=1800 cs=0 l=1 rev1=1 crypto rx=0x7fe3bc070ff0 tx=0x7fe3b801c4d0 comp rx=0 tx=0).stop 2023-05-22T15:46:18.029 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:18.024+0000 7fe3c4589640 1 -- 172.21.15.16:0/3938532712 shutdown_connections 2023-05-22T15:46:18.029 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:18.024+0000 7fe3c4589640 1 --2- 172.21.15.16:0/3938532712 >> v2:172.21.15.62:3300/0 conn(0x7fe3bc0cc6c0 0x7fe3bc0c8080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:18.030 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:18.024+0000 7fe3c4589640 1 --2- 172.21.15.16:0/3938532712 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe3bc1563c0 0x7fe3bc0c7b40 unknown :-1 s=CLOSED pgs=1800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:18.030 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:18.024+0000 7fe3c4589640 1 -- 172.21.15.16:0/3938532712 >> 172.21.15.16:0/3938532712 conn(0x7fe3bc0b1110 msgr2=0x7fe3bc0ce830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:46:18.030 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:18.024+0000 7fe3c4589640 1 -- 172.21.15.16:0/3938532712 shutdown_connections 2023-05-22T15:46:18.030 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:18.024+0000 7fe3c4589640 1 -- 172.21.15.16:0/3938532712 wait complete. 2023-05-22T15:46:18.030 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:18.024+0000 7fe3c4589640 1 Processor -- start 2023-05-22T15:46:18.031 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:18.024+0000 7fe3c4589640 1 -- start start 2023-05-22T15:46:18.032 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:18.028+0000 7fe3c4589640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe3bc06c220 0x7fe3bc14e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:18.032 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:18.028+0000 7fe3c4589640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe3bc0cc6c0 0x7fe3bc0c9420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:18.032 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:18.028+0000 7fe3c4589640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe3bc153840 0x7fe3bc153cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:18.033 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:18.028+0000 7fe3c4589640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3bc1f4ee0 con 0x7fe3bc0cc6c0 2023-05-22T15:46:18.033 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:18.028+0000 7fe3c4589640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3bc1f4010 con 0x7fe3bc06c220 2023-05-22T15:46:18.033 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:18.028+0000 7fe3c4589640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3bc1f4260 con 0x7fe3bc153840 2023-05-22T15:46:18.033 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:18.028+0000 7fe3c1b24640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe3bc0cc6c0 0x7fe3bc0c9420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:18.033 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:18.028+0000 7fe3c1b24640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe3bc0cc6c0 0x7fe3bc0c9420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43656/0 (socket says 172.21.15.16:43656) 2023-05-22T15:46:18.033 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:18.028+0000 7fe3c1b24640 1 -- 172.21.15.16:0/1057331147 learned_addr learned my addr 172.21.15.16:0/1057331147 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:46:18.033 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:18.028+0000 7fe3c2325640 1 --2- 172.21.15.16:0/1057331147 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe3bc06c220 0x7fe3bc14e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:18.034 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:18.028+0000 7fe3c2b26640 1 --2- 172.21.15.16:0/1057331147 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe3bc153840 0x7fe3bc153cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:18.034 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:18.028+0000 7fe3c1b24640 1 -- 172.21.15.16:0/1057331147 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe3bc153840 msgr2=0x7fe3bc153cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:18.034 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:18.028+0000 7fe3c1b24640 1 --2- 172.21.15.16:0/1057331147 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe3bc153840 0x7fe3bc153cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:18.034 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:18.028+0000 7fe3c1b24640 1 -- 172.21.15.16:0/1057331147 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe3bc06c220 msgr2=0x7fe3bc14e0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:18.034 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:18.028+0000 7fe3c1b24640 1 --2- 172.21.15.16:0/1057331147 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe3bc06c220 0x7fe3bc14e0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:18.034 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:18.028+0000 7fe3c1b24640 1 -- 172.21.15.16:0/1057331147 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3b80094d0 con 0x7fe3bc0cc6c0 2023-05-22T15:46:18.034 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:18.028+0000 7fe3c1b24640 1 --2- 172.21.15.16:0/1057331147 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe3bc0cc6c0 0x7fe3bc0c9420 secure :-1 s=READY pgs=1801 cs=0 l=1 rev1=1 crypto rx=0x7fe3ac00e900 tx=0x7fe3ac00edc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:18.034 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:18.028+0000 7fe3b2ffd640 1 -- 172.21.15.16:0/1057331147 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3ac009690 con 0x7fe3bc0cc6c0 2023-05-22T15:46:18.035 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:18.028+0000 7fe3b2ffd640 1 -- 172.21.15.16:0/1057331147 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3ac00ca00 con 0x7fe3bc0cc6c0 2023-05-22T15:46:18.035 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:18.028+0000 7fe3b2ffd640 1 -- 172.21.15.16:0/1057331147 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3ac00cd90 con 0x7fe3bc0cc6c0 2023-05-22T15:46:18.035 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:18.028+0000 7fe3c4589640 1 -- 172.21.15.16:0/1057331147 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3bc15ff00 con 0x7fe3bc0cc6c0 2023-05-22T15:46:18.035 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:18.028+0000 7fe3c4589640 1 -- 172.21.15.16:0/1057331147 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3bc15cd20 con 0x7fe3bc0cc6c0 2023-05-22T15:46:18.036 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:18.028+0000 7fe3c4589640 1 -- 172.21.15.16:0/1057331147 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3bc160090 con 0x7fe3bc0cc6c0 2023-05-22T15:46:18.036 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:18.028+0000 7fe3b2ffd640 1 -- 172.21.15.16:0/1057331147 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 60) v1 ==== 84332+0+0 (secure 0 0 0) 0x7fe3ac013470 con 0x7fe3bc0cc6c0 2023-05-22T15:46:18.036 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:18.028+0000 7fe3b2ffd640 1 --2- 172.21.15.16:0/1057331147 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fe3a00654a0 0x7fe3a0067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:18.036 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:18.028+0000 7fe3b2ffd640 1 -- 172.21.15.16:0/1057331147 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(142..142 src has 1..142) v4 ==== 6332+0+0 (secure 0 0 0) 0x7fe3ac082c80 con 0x7fe3bc0cc6c0 2023-05-22T15:46:18.036 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:18.028+0000 7fe3c2325640 1 --2- 172.21.15.16:0/1057331147 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fe3a00654a0 0x7fe3a0067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:18.037 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:18.028+0000 7fe3c2325640 1 --2- 172.21.15.16:0/1057331147 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fe3a00654a0 0x7fe3a0067950 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fe3b80ae000 tx=0x7fe3b80b5040 comp rx=0 tx=0).ready entity=mgr.8835 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:18.038 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:18.032+0000 7fe3b2ffd640 1 -- 172.21.15.16:0/1057331147 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fe3ac050730 con 0x7fe3bc0cc6c0 2023-05-22T15:46:18.196 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:18.192+0000 7fe3c4589640 1 -- 172.21.15.16:0/1057331147 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7fe3bc04dbe0 con 0x7fe3bc0cc6c0 2023-05-22T15:46:19.037 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.031+0000 7fe3b2ffd640 1 -- 172.21.15.16:0/1057331147 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v143) v1 ==== 97+0+0 (secure 0 0 0) 0x7fe3ac053d50 con 0x7fe3bc0cc6c0 2023-05-22T15:46:19.037 INFO:teuthology.orchestra.run.smithi016.stderr: removed all blocklist entries 2023-05-22T15:46:19.041 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.035+0000 7fe3c4589640 1 -- 172.21.15.16:0/1057331147 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fe3a00654a0 msgr2=0x7fe3a0067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:19.041 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.035+0000 7fe3c4589640 1 --2- 172.21.15.16:0/1057331147 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fe3a00654a0 0x7fe3a0067950 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fe3b80ae000 tx=0x7fe3b80b5040 comp rx=0 tx=0).stop 2023-05-22T15:46:19.041 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.035+0000 7fe3c4589640 1 -- 172.21.15.16:0/1057331147 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe3bc0cc6c0 msgr2=0x7fe3bc0c9420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:19.041 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.035+0000 7fe3c4589640 1 --2- 172.21.15.16:0/1057331147 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe3bc0cc6c0 0x7fe3bc0c9420 secure :-1 s=READY pgs=1801 cs=0 l=1 rev1=1 crypto rx=0x7fe3ac00e900 tx=0x7fe3ac00edc0 comp rx=0 tx=0).stop 2023-05-22T15:46:19.041 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.035+0000 7fe3c4589640 1 -- 172.21.15.16:0/1057331147 shutdown_connections 2023-05-22T15:46:19.042 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.035+0000 7fe3c4589640 1 --2- 172.21.15.16:0/1057331147 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fe3a00654a0 0x7fe3a0067950 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:19.042 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.035+0000 7fe3c4589640 1 --2- 172.21.15.16:0/1057331147 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe3bc153840 0x7fe3bc153cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:19.042 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.035+0000 7fe3c4589640 1 --2- 172.21.15.16:0/1057331147 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe3bc0cc6c0 0x7fe3bc0c9420 unknown :-1 s=CLOSED pgs=1801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:19.042 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.035+0000 7fe3c4589640 1 --2- 172.21.15.16:0/1057331147 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe3bc06c220 0x7fe3bc14e0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:19.042 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.035+0000 7fe3c4589640 1 -- 172.21.15.16:0/1057331147 >> 172.21.15.16:0/1057331147 conn(0x7fe3bc0b1110 msgr2=0x7fe3bc151f20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:46:19.042 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.035+0000 7fe3c4589640 1 -- 172.21.15.16:0/1057331147 shutdown_connections 2023-05-22T15:46:19.042 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.035+0000 7fe3c4589640 1 -- 172.21.15.16:0/1057331147 wait complete. 2023-05-22T15:46:19.066 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2023-05-22T15:46:19.243 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.239+0000 7f065e1a8640 1 Processor -- start 2023-05-22T15:46:19.244 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.239+0000 7f065e1a8640 1 -- start start 2023-05-22T15:46:19.244 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.239+0000 7f065e1a8640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f065815c270 0x7f065815c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:19.245 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.239+0000 7f065e1a8640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0658069ae0 con 0x7f0658071640 2023-05-22T15:46:19.245 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.239+0000 7f065e1a8640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0658069dc0 con 0x7f06580ce480 2023-05-22T15:46:19.245 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.239+0000 7f065e1a8640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f065815f990 con 0x7f065815c270 2023-05-22T15:46:19.245 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.239+0000 7f0657fff640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f065815c270 0x7f065815c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:19.245 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.239+0000 7f0656ffd640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f0658071640 0x7f0658160090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:38788/0 (socket says 172.21.15.16:38788) 2023-05-22T15:46:19.245 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.239+0000 7f0656ffd640 1 -- 172.21.15.16:0/2370419897 learned_addr learned my addr 172.21.15.16:0/2370419897 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:46:19.246 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.239+0000 7f0657fff640 1 -- 172.21.15.16:0/2370419897 >> v1:172.21.15.62:6789/0 conn(0x7f06580ce480 legacy=0x7f06580ce850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:19.246 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.239+0000 7f0657fff640 1 -- 172.21.15.16:0/2370419897 >> v1:172.21.15.16:6789/0 conn(0x7f0658071640 legacy=0x7f0658160090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:19.246 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.239+0000 7f0657fff640 1 -- 172.21.15.16:0/2370419897 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f065806a0a0 con 0x7f065815c270 2023-05-22T15:46:19.247 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.243+0000 7f0657fff640 1 --2- 172.21.15.16:0/2370419897 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f065815c270 0x7f065815c6d0 secure :-1 s=READY pgs=1803 cs=0 l=1 rev1=1 crypto rx=0x7f0650009d90 tx=0x7f065001c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6adc1fcb87e8092e server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:19.247 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.243+0000 7f06567fc640 1 -- 172.21.15.16:0/2370419897 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06500a5a00 con 0x7f065815c270 2023-05-22T15:46:19.248 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.243+0000 7f06567fc640 1 -- 172.21.15.16:0/2370419897 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06500a5b60 con 0x7f065815c270 2023-05-22T15:46:19.248 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.243+0000 7f06567fc640 1 -- 172.21.15.16:0/2370419897 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06500ad440 con 0x7f065815c270 2023-05-22T15:46:19.248 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.243+0000 7f065e1a8640 1 -- 172.21.15.16:0/2370419897 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f065815c270 msgr2=0x7f065815c6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:19.249 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.243+0000 7f065e1a8640 1 --2- 172.21.15.16:0/2370419897 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f065815c270 0x7f065815c6d0 secure :-1 s=READY pgs=1803 cs=0 l=1 rev1=1 crypto rx=0x7f0650009d90 tx=0x7f065001c920 comp rx=0 tx=0).stop 2023-05-22T15:46:19.249 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.243+0000 7f065e1a8640 1 -- 172.21.15.16:0/2370419897 shutdown_connections 2023-05-22T15:46:19.249 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.243+0000 7f065e1a8640 1 --2- 172.21.15.16:0/2370419897 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f065815c270 0x7f065815c6d0 unknown :-1 s=CLOSED pgs=1803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:19.249 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.243+0000 7f065e1a8640 1 -- 172.21.15.16:0/2370419897 >> 172.21.15.16:0/2370419897 conn(0x7f06580cc870 msgr2=0x7f06580cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:46:19.250 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.243+0000 7f065e1a8640 1 -- 172.21.15.16:0/2370419897 shutdown_connections 2023-05-22T15:46:19.250 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.243+0000 7f065e1a8640 1 -- 172.21.15.16:0/2370419897 wait complete. 2023-05-22T15:46:19.251 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.243+0000 7f065e1a8640 1 Processor -- start 2023-05-22T15:46:19.251 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.243+0000 7f065e1a8640 1 -- start start 2023-05-22T15:46:19.251 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.243+0000 7f065e1a8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0658071640 0x7f0658158380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:19.252 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.243+0000 7f065e1a8640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f06580ce480 0x7f065815ea00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:19.252 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.243+0000 7f065e1a8640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f065815c270 0x7f06580ac950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:19.252 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.243+0000 7f065e1a8640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0658071ac0 con 0x7f0658071640 2023-05-22T15:46:19.252 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.243+0000 7f065e1a8640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f065815b070 con 0x7f065815c270 2023-05-22T15:46:19.252 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.243+0000 7f065e1a8640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f065815a780 con 0x7f06580ce480 2023-05-22T15:46:19.252 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.243+0000 7f06577fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0658071640 0x7f0658158380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:19.252 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.243+0000 7f06577fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0658071640 0x7f0658158380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34306/0 (socket says 172.21.15.16:34306) 2023-05-22T15:46:19.253 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.243+0000 7f06577fe640 1 -- 172.21.15.16:0/1451962329 learned_addr learned my addr 172.21.15.16:0/1451962329 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:46:19.253 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.243+0000 7f0656ffd640 1 --2- 172.21.15.16:0/1451962329 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f06580ce480 0x7f065815ea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:19.253 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.243+0000 7f0657fff640 1 --2- 172.21.15.16:0/1451962329 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f065815c270 0x7f06580ac950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:19.253 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.247+0000 7f06577fe640 1 -- 172.21.15.16:0/1451962329 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f06580ce480 msgr2=0x7f065815ea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:19.253 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.247+0000 7f06577fe640 1 --2- 172.21.15.16:0/1451962329 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f06580ce480 0x7f065815ea00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:19.253 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.247+0000 7f06577fe640 1 -- 172.21.15.16:0/1451962329 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f065815c270 msgr2=0x7f06580ac950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:19.253 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.247+0000 7f06577fe640 1 --2- 172.21.15.16:0/1451962329 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f065815c270 0x7f06580ac950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:19.253 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.247+0000 7f06577fe640 1 -- 172.21.15.16:0/1451962329 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0650009a70 con 0x7f0658071640 2023-05-22T15:46:19.254 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.247+0000 7f0657fff640 1 --2- 172.21.15.16:0/1451962329 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f065815c270 0x7f06580ac950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T15:46:19.254 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.247+0000 7f06577fe640 1 --2- 172.21.15.16:0/1451962329 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0658071640 0x7f0658158380 secure :-1 s=READY pgs=1804 cs=0 l=1 rev1=1 crypto rx=0x7f064800c3c0 tx=0x7f064800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:19.254 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.247+0000 7f063ffff640 1 -- 172.21.15.16:0/1451962329 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06480052d0 con 0x7f0658071640 2023-05-22T15:46:19.254 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.247+0000 7f063ffff640 1 -- 172.21.15.16:0/1451962329 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0648005430 con 0x7f0658071640 2023-05-22T15:46:19.254 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.247+0000 7f065e1a8640 1 -- 172.21.15.16:0/1451962329 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06580698b0 con 0x7f0658071640 2023-05-22T15:46:19.254 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.247+0000 7f063ffff640 1 -- 172.21.15.16:0/1451962329 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0648003840 con 0x7f0658071640 2023-05-22T15:46:19.254 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.247+0000 7f065e1a8640 1 -- 172.21.15.16:0/1451962329 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0658069720 con 0x7f0658071640 2023-05-22T15:46:19.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.247+0000 7f065e1a8640 1 -- 172.21.15.16:0/1451962329 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06580c5e70 con 0x7f0658071640 2023-05-22T15:46:19.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.247+0000 7f063ffff640 1 -- 172.21.15.16:0/1451962329 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 60) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f06480039a0 con 0x7f0658071640 2023-05-22T15:46:19.256 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.247+0000 7f063ffff640 1 --2- 172.21.15.16:0/1451962329 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f06440654a0 0x7f0644067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:19.256 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.247+0000 7f0656ffd640 1 --2- 172.21.15.16:0/1451962329 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f06440654a0 0x7f0644067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:19.256 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.247+0000 7f063ffff640 1 -- 172.21.15.16:0/1451962329 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5558+0+0 (secure 0 0 0) 0x7f0648081fa0 con 0x7f0658071640 2023-05-22T15:46:19.256 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.247+0000 7f0656ffd640 1 --2- 172.21.15.16:0/1451962329 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f06440654a0 0x7f0644067950 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f065815e390 tx=0x7f064c009040 comp rx=0 tx=0).ready entity=mgr.8835 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:19.257 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.251+0000 7f063ffff640 1 -- 172.21.15.16:0/1451962329 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f064804fd10 con 0x7f0658071640 2023-05-22T15:46:19.465 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.459+0000 7f065e1a8640 1 -- 172.21.15.16:0/1451962329 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f065804dbe0 con 0x7f0658071640 2023-05-22T15:46:19.465 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.459+0000 7f063ffff640 1 -- 172.21.15.16:0/1451962329 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v23) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f0648053330 con 0x7f0658071640 2023-05-22T15:46:19.465 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:46:19.465 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-05-22T15:46:19.466 INFO:teuthology.orchestra.run.smithi016.stdout: "auth_dump": [ 2023-05-22T15:46:19.466 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T15:46:19.466 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "mds.a", 2023-05-22T15:46:19.466 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQCAiWtkw/K0GRAA9hvTMKMmcbgKV68GeoC4ag==", 2023-05-22T15:46:19.466 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-05-22T15:46:19.466 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2023-05-22T15:46:19.466 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow *", 2023-05-22T15:46:19.467 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow *", 2023-05-22T15:46:19.467 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-05-22T15:46:19.467 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T15:46:19.467 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:46:19.467 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T15:46:19.467 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "mds.b", 2023-05-22T15:46:19.467 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQCAiWtkoFzfJRAAxCiB0MGM5apMVA81WDq+dA==", 2023-05-22T15:46:19.468 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-05-22T15:46:19.468 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2023-05-22T15:46:19.468 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow *", 2023-05-22T15:46:19.468 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow *", 2023-05-22T15:46:19.468 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-05-22T15:46:19.468 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T15:46:19.468 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:46:19.468 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T15:46:19.469 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "mds.c", 2023-05-22T15:46:19.469 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQCAiWtkvgOKHxAAMuASPtMqwyoqpZ5I+MRS7A==", 2023-05-22T15:46:19.469 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-05-22T15:46:19.469 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2023-05-22T15:46:19.469 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow *", 2023-05-22T15:46:19.469 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow *", 2023-05-22T15:46:19.469 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-05-22T15:46:19.469 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T15:46:19.470 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:46:19.470 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T15:46:19.470 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "mds.d", 2023-05-22T15:46:19.470 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQCAiWtk6hPQKxAAlCtfC36vja5ftz6uOqUhHg==", 2023-05-22T15:46:19.470 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-05-22T15:46:19.470 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2023-05-22T15:46:19.470 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow *", 2023-05-22T15:46:19.470 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow *", 2023-05-22T15:46:19.471 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-05-22T15:46:19.471 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T15:46:19.471 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:46:19.471 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T15:46:19.471 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.0", 2023-05-22T15:46:19.471 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQCCiWtk6ZLoCRAAKdCQitZuPnWcIpkiXhJeww==", 2023-05-22T15:46:19.471 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-05-22T15:46:19.472 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2023-05-22T15:46:19.472 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2023-05-22T15:46:19.472 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-05-22T15:46:19.472 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T15:46:19.472 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:46:19.472 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T15:46:19.472 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.1", 2023-05-22T15:46:19.472 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQCFiWtkXtVeERAAkgPeDKuFmWXrDiCyqWOYwQ==", 2023-05-22T15:46:19.473 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-05-22T15:46:19.473 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2023-05-22T15:46:19.473 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2023-05-22T15:46:19.473 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-05-22T15:46:19.473 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T15:46:19.473 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:46:19.473 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T15:46:19.473 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.2", 2023-05-22T15:46:19.474 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQCIiWtkdGaRHRAAx3POIp3mnz7p7XcCVMXm5Q==", 2023-05-22T15:46:19.474 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-05-22T15:46:19.474 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2023-05-22T15:46:19.474 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2023-05-22T15:46:19.474 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-05-22T15:46:19.474 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T15:46:19.474 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:46:19.474 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T15:46:19.475 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.3", 2023-05-22T15:46:19.475 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQCLiWtkSBydJhAAsgc19Xaf/GOyy9nm9MLdJQ==", 2023-05-22T15:46:19.475 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-05-22T15:46:19.475 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2023-05-22T15:46:19.475 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2023-05-22T15:46:19.475 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-05-22T15:46:19.475 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T15:46:19.475 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:46:19.476 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T15:46:19.476 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.4", 2023-05-22T15:46:19.476 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQCRiWtkueYvJRAARXbHlKHrTbgTqBArRJfvBA==", 2023-05-22T15:46:19.476 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-05-22T15:46:19.476 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2023-05-22T15:46:19.476 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2023-05-22T15:46:19.476 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-05-22T15:46:19.477 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T15:46:19.477 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:46:19.477 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T15:46:19.477 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.5", 2023-05-22T15:46:19.477 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQCUiWtkH3hwLRAAUekP6y+wkJVPh+OPDiZbzQ==", 2023-05-22T15:46:19.477 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-05-22T15:46:19.477 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2023-05-22T15:46:19.477 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2023-05-22T15:46:19.478 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-05-22T15:46:19.478 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T15:46:19.478 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:46:19.478 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T15:46:19.478 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.6", 2023-05-22T15:46:19.478 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQCXiWtkLsnBNhAAQ8ilX8ZU4ncJ2JJU0I+U4w==", 2023-05-22T15:46:19.478 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-05-22T15:46:19.479 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2023-05-22T15:46:19.479 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2023-05-22T15:46:19.479 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-05-22T15:46:19.479 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T15:46:19.479 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:46:19.479 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T15:46:19.479 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.7", 2023-05-22T15:46:19.479 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQCbiWtkUvXjBBAApMWw7wdfxWq0WEQDL2qprA==", 2023-05-22T15:46:19.480 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-05-22T15:46:19.480 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2023-05-22T15:46:19.480 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2023-05-22T15:46:19.480 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-05-22T15:46:19.480 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T15:46:19.480 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:46:19.480 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T15:46:19.480 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "client.0", 2023-05-22T15:46:19.481 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQCAiWtkZwuUMRAAnYqSFiaT/QRsajdjNgTetA==", 2023-05-22T15:46:19.481 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-05-22T15:46:19.481 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2023-05-22T15:46:19.481 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow r", 2023-05-22T15:46:19.481 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow rw tag cephfs data=cephfs" 2023-05-22T15:46:19.481 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T15:46:19.481 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:46:19.481 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T15:46:19.482 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "client.1", 2023-05-22T15:46:19.482 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQCAiWtkPimmORAABzNL2rANz5TgCl3p++GVng==", 2023-05-22T15:46:19.482 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-05-22T15:46:19.482 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2023-05-22T15:46:19.482 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow r", 2023-05-22T15:46:19.482 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow rw", 2023-05-22T15:46:19.482 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow rwx" 2023-05-22T15:46:19.483 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T15:46:19.483 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:46:19.483 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T15:46:19.483 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "client.2", 2023-05-22T15:46:19.483 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQCAiWtkqkPMNhAAqugrnD4Mh/4At8QAFmzqcg==", 2023-05-22T15:46:19.483 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-05-22T15:46:19.483 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2023-05-22T15:46:19.483 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow r", 2023-05-22T15:46:19.484 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow rw", 2023-05-22T15:46:19.484 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow rwx" 2023-05-22T15:46:19.484 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T15:46:19.484 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:46:19.484 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T15:46:19.484 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "client.3", 2023-05-22T15:46:19.484 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQCBiWtkpvlWAxAAPuedivXBGZGHIO6hSnz8PQ==", 2023-05-22T15:46:19.484 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-05-22T15:46:19.485 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2023-05-22T15:46:19.485 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow r", 2023-05-22T15:46:19.485 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow rw", 2023-05-22T15:46:19.485 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow rwx" 2023-05-22T15:46:19.485 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T15:46:19.485 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:46:19.485 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T15:46:19.485 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "client.admin", 2023-05-22T15:46:19.486 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQCAiWtkxG9hAhAAfnUnRrBRUFoCEtsb2riNOQ==", 2023-05-22T15:46:19.486 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-05-22T15:46:19.486 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow *", 2023-05-22T15:46:19.486 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow *", 2023-05-22T15:46:19.486 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow *", 2023-05-22T15:46:19.486 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-05-22T15:46:19.486 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T15:46:19.486 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:46:19.487 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T15:46:19.487 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "mgr.x", 2023-05-22T15:46:19.487 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQCAiWtkkaIqFhAAb/4BI6FYWOy3qOSaylXLSw==", 2023-05-22T15:46:19.487 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-05-22T15:46:19.487 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow *", 2023-05-22T15:46:19.487 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile mgr", 2023-05-22T15:46:19.487 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-05-22T15:46:19.487 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T15:46:19.488 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:46:19.488 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T15:46:19.488 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "mgr.y", 2023-05-22T15:46:19.488 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQCAiWtkhgVWExAAjDoh7cexFVfsigsYfGkmVg==", 2023-05-22T15:46:19.488 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-05-22T15:46:19.488 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow *", 2023-05-22T15:46:19.488 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile mgr", 2023-05-22T15:46:19.489 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-05-22T15:46:19.489 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T15:46:19.489 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T15:46:19.489 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-05-22T15:46:19.489 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-05-22T15:46:19.491 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.463+0000 7f065e1a8640 1 -- 172.21.15.16:0/1451962329 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f06440654a0 msgr2=0x7f0644067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:19.491 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.463+0000 7f065e1a8640 1 --2- 172.21.15.16:0/1451962329 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f06440654a0 0x7f0644067950 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f065815e390 tx=0x7f064c009040 comp rx=0 tx=0).stop 2023-05-22T15:46:19.491 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.463+0000 7f065e1a8640 1 -- 172.21.15.16:0/1451962329 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0658071640 msgr2=0x7f0658158380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:19.491 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.463+0000 7f065e1a8640 1 --2- 172.21.15.16:0/1451962329 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0658071640 0x7f0658158380 secure :-1 s=READY pgs=1804 cs=0 l=1 rev1=1 crypto rx=0x7f064800c3c0 tx=0x7f064800c880 comp rx=0 tx=0).stop 2023-05-22T15:46:19.491 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.463+0000 7f065e1a8640 1 -- 172.21.15.16:0/1451962329 shutdown_connections 2023-05-22T15:46:19.491 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.463+0000 7f065e1a8640 1 --2- 172.21.15.16:0/1451962329 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f065815c270 0x7f06580ac950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:19.492 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.463+0000 7f065e1a8640 1 --2- 172.21.15.16:0/1451962329 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f06440654a0 0x7f0644067950 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:19.492 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.463+0000 7f065e1a8640 1 --2- 172.21.15.16:0/1451962329 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f06580ce480 0x7f065815ea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:19.492 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.463+0000 7f065e1a8640 1 --2- 172.21.15.16:0/1451962329 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0658071640 0x7f0658158380 unknown :-1 s=CLOSED pgs=1804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:19.492 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.463+0000 7f065e1a8640 1 -- 172.21.15.16:0/1451962329 >> 172.21.15.16:0/1451962329 conn(0x7f06580cc870 msgr2=0x7f06580cef90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:46:19.492 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.463+0000 7f065e1a8640 1 -- 172.21.15.16:0/1451962329 shutdown_connections 2023-05-22T15:46:19.492 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.463+0000 7f065e1a8640 1 -- 172.21.15.16:0/1451962329 wait complete. 2023-05-22T15:46:19.493 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-05-22T15:46:19.619 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.611+0000 7f07e4b1d640 1 Processor -- start 2023-05-22T15:46:19.620 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.611+0000 7f07e4b1d640 1 -- start start 2023-05-22T15:46:19.621 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.615+0000 7f07e4b1d640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f07e00afbb0 0x7f07e00c8880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:19.621 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.615+0000 7f07e4b1d640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f07e00b0030 0x7f07e00cdfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:19.621 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.615+0000 7f07e4b1d640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07e00c92b0 con 0x7f07e00cce20 2023-05-22T15:46:19.622 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.615+0000 7f07e4b1d640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07e00cc060 con 0x7f07e00b0030 2023-05-22T15:46:19.622 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.615+0000 7f07e4b1d640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07e00cc340 con 0x7f07e00afbb0 2023-05-22T15:46:19.622 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.615+0000 7f07ded9d640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f07e00b0030 0x7f07e00cdfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:19.622 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.615+0000 7f07ded9d640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f07e00b0030 0x7f07e00cdfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34318/0 (socket says 172.21.15.16:34318) 2023-05-22T15:46:19.622 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.615+0000 7f07ded9d640 1 -- 172.21.15.16:0/2838041009 learned_addr learned my addr 172.21.15.16:0/2838041009 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:46:19.622 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.615+0000 7f07ddd9b640 1 --2- 172.21.15.16:0/2838041009 >> v2:172.21.15.62:3300/0 conn(0x7f07e00afbb0 0x7f07e00c8880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:19.622 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.615+0000 7f07ded9d640 1 -- 172.21.15.16:0/2838041009 >> v2:172.21.15.62:3300/0 conn(0x7f07e00afbb0 msgr2=0x7f07e00c8880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:46:19.623 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.615+0000 7f07ded9d640 1 --2- 172.21.15.16:0/2838041009 >> v2:172.21.15.62:3300/0 conn(0x7f07e00afbb0 0x7f07e00c8880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:19.623 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.615+0000 7f07ded9d640 1 -- 172.21.15.16:0/2838041009 >> v1:172.21.15.62:6789/0 conn(0x7f07e00cce20 legacy=0x7f07e0158290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:46:19.623 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.615+0000 7f07ded9d640 1 -- 172.21.15.16:0/2838041009 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07e00cc620 con 0x7f07e00b0030 2023-05-22T15:46:19.623 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.615+0000 7f07ddd9b640 1 --2- 172.21.15.16:0/2838041009 >> v2:172.21.15.62:3300/0 conn(0x7f07e00afbb0 0x7f07e00c8880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T15:46:19.624 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.615+0000 7f07ded9d640 1 --2- 172.21.15.16:0/2838041009 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f07e00b0030 0x7f07e00cdfb0 secure :-1 s=READY pgs=1805 cs=0 l=1 rev1=1 crypto rx=0x7f07e006d810 tx=0x7f07d801c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5cfb8c6599902d76 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:19.624 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.615+0000 7f07dd59a640 1 -- 172.21.15.16:0/2838041009 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07d80a57b0 con 0x7f07e00b0030 2023-05-22T15:46:19.624 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.615+0000 7f07dd59a640 1 -- 172.21.15.16:0/2838041009 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07d80a5910 con 0x7f07e00b0030 2023-05-22T15:46:19.624 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.615+0000 7f07dd59a640 1 -- 172.21.15.16:0/2838041009 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07d80a5ca0 con 0x7f07e00b0030 2023-05-22T15:46:19.625 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.619+0000 7f07e4b1d640 1 -- 172.21.15.16:0/2838041009 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f07e00b0030 msgr2=0x7f07e00cdfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:19.625 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.619+0000 7f07e4b1d640 1 --2- 172.21.15.16:0/2838041009 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f07e00b0030 0x7f07e00cdfb0 secure :-1 s=READY pgs=1805 cs=0 l=1 rev1=1 crypto rx=0x7f07e006d810 tx=0x7f07d801c8d0 comp rx=0 tx=0).stop 2023-05-22T15:46:19.625 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.619+0000 7f07e4b1d640 1 -- 172.21.15.16:0/2838041009 shutdown_connections 2023-05-22T15:46:19.625 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.619+0000 7f07e4b1d640 1 --2- 172.21.15.16:0/2838041009 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f07e00b0030 0x7f07e00cdfb0 unknown :-1 s=CLOSED pgs=1805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:19.625 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.619+0000 7f07e4b1d640 1 --2- 172.21.15.16:0/2838041009 >> v2:172.21.15.62:3300/0 conn(0x7f07e00afbb0 0x7f07e00c8880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:19.625 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.619+0000 7f07e4b1d640 1 -- 172.21.15.16:0/2838041009 >> 172.21.15.16:0/2838041009 conn(0x7f07e0069720 msgr2=0x7f07e0069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:46:19.626 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.619+0000 7f07e4b1d640 1 -- 172.21.15.16:0/2838041009 shutdown_connections 2023-05-22T15:46:19.626 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.619+0000 7f07e4b1d640 1 -- 172.21.15.16:0/2838041009 wait complete. 2023-05-22T15:46:19.627 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.619+0000 7f07e4b1d640 1 Processor -- start 2023-05-22T15:46:19.628 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.619+0000 7f07e4b1d640 1 -- start start 2023-05-22T15:46:19.628 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.619+0000 7f07e4b1d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f07e00afbb0 0x7f07e014b230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:19.628 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.619+0000 7f07e4b1d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f07e00b0030 0x7f07e014be40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:19.628 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.619+0000 7f07e4b1d640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f07e00cce20 0x7f07e0158a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:19.629 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.619+0000 7f07e4b1d640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07e014d310 con 0x7f07e00afbb0 2023-05-22T15:46:19.629 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.619+0000 7f07e4b1d640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07e014c690 con 0x7f07e00b0030 2023-05-22T15:46:19.629 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.619+0000 7f07e4b1d640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07e014c970 con 0x7f07e00cce20 2023-05-22T15:46:19.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.619+0000 7f07de59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f07e00afbb0 0x7f07e014b230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:19.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.619+0000 7f07de59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f07e00afbb0 0x7f07e014b230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34330/0 (socket says 172.21.15.16:34330) 2023-05-22T15:46:19.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.619+0000 7f07de59c640 1 -- 172.21.15.16:0/1592028907 learned_addr learned my addr 172.21.15.16:0/1592028907 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:46:19.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.619+0000 7f07ddd9b640 1 --2- 172.21.15.16:0/1592028907 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f07e00b0030 0x7f07e014be40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:19.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.619+0000 7f07ded9d640 1 --2- 172.21.15.16:0/1592028907 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f07e00cce20 0x7f07e0158a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:19.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.619+0000 7f07de59c640 1 -- 172.21.15.16:0/1592028907 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f07e00cce20 msgr2=0x7f07e0158a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:19.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.619+0000 7f07de59c640 1 --2- 172.21.15.16:0/1592028907 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f07e00cce20 0x7f07e0158a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:19.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.619+0000 7f07de59c640 1 -- 172.21.15.16:0/1592028907 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f07e00b0030 msgr2=0x7f07e014be40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:19.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.619+0000 7f07de59c640 1 --2- 172.21.15.16:0/1592028907 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f07e00b0030 0x7f07e014be40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:19.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.619+0000 7f07de59c640 1 -- 172.21.15.16:0/1592028907 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07d8009b30 con 0x7f07e00afbb0 2023-05-22T15:46:19.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.619+0000 7f07ddd9b640 1 --2- 172.21.15.16:0/1592028907 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f07e00b0030 0x7f07e014be40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:46:19.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.619+0000 7f07de59c640 1 --2- 172.21.15.16:0/1592028907 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f07e00afbb0 0x7f07e014b230 secure :-1 s=READY pgs=1806 cs=0 l=1 rev1=1 crypto rx=0x7f07d000b410 tx=0x7f07d000b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:19.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.619+0000 7f07d6ffd640 1 -- 172.21.15.16:0/1592028907 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07d0004020 con 0x7f07e00afbb0 2023-05-22T15:46:19.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.619+0000 7f07d6ffd640 1 -- 172.21.15.16:0/1592028907 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07d0004180 con 0x7f07e00afbb0 2023-05-22T15:46:19.632 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.619+0000 7f07e4b1d640 1 -- 172.21.15.16:0/1592028907 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07e006fe40 con 0x7f07e00afbb0 2023-05-22T15:46:19.632 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.619+0000 7f07d6ffd640 1 -- 172.21.15.16:0/1592028907 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07d0011630 con 0x7f07e00afbb0 2023-05-22T15:46:19.632 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.619+0000 7f07e4b1d640 1 -- 172.21.15.16:0/1592028907 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07e00ce900 con 0x7f07e00afbb0 2023-05-22T15:46:19.632 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.623+0000 7f07e4b1d640 1 -- 172.21.15.16:0/1592028907 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07e006ffd0 con 0x7f07e00afbb0 2023-05-22T15:46:19.633 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.623+0000 7f07d6ffd640 1 -- 172.21.15.16:0/1592028907 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 60) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f07d0011790 con 0x7f07e00afbb0 2023-05-22T15:46:19.633 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.623+0000 7f07d6ffd640 1 --2- 172.21.15.16:0/1592028907 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f07c00654a0 0x7f07c0067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:19.633 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.623+0000 7f07d6ffd640 1 -- 172.21.15.16:0/1592028907 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5558+0+0 (secure 0 0 0) 0x7f07d00824c0 con 0x7f07e00afbb0 2023-05-22T15:46:19.633 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.623+0000 7f07ddd9b640 1 --2- 172.21.15.16:0/1592028907 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f07c00654a0 0x7f07c0067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:19.633 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.623+0000 7f07ddd9b640 1 --2- 172.21.15.16:0/1592028907 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f07c00654a0 0x7f07c0067950 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f07e00c50a0 tx=0x7f07c8008040 comp rx=0 tx=0).ready entity=mgr.8835 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:19.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.627+0000 7f07d6ffd640 1 -- 172.21.15.16:0/1592028907 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f07d00502b0 con 0x7f07e00afbb0 2023-05-22T15:46:19.787 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.783+0000 7f07e4b1d640 1 -- 172.21.15.16:0/1592028907 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f07e004dbe0 con 0x7f07e00afbb0 2023-05-22T15:46:19.789 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.783+0000 7f07d6ffd640 1 -- 172.21.15.16:0/1592028907 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v143) v1 ==== 74+0+14382 (secure 0 0 0) 0x7f07d00538d0 con 0x7f07e00afbb0 2023-05-22T15:46:19.790 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:46:19.790 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":143,"fsid":"b2719834-7847-4d98-90a6-f65d57f85c49","created":"2023-05-22T15:26:25.259314+0000","modified":"2023-05-22T15:46:19.025292+0000","last_up_change":"2023-05-22T15:26:36.448524+0000","last_in_change":"2023-05-22T15:26:27.675668+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-05-22T15:26:38.019781+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-05-22T15:26:43.570514+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9e7ff854-b25f-4cbb-a5c2-1799d9c53d2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":37021},{"type":"v1","addr":"172.21.15.16:6825","nonce":37021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":37021},{"type":"v1","addr":"172.21.15.16:6827","nonce":37021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":37021},{"type":"v1","addr":"172.21.15.16:6831","nonce":37021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":37021},{"type":"v1","addr":"172.21.15.16:6829","nonce":37021}]},"public_addr":"172.21.15.16:6825/37021","cluster_addr":"172.21.15.16:6827/37021","heartbeat_back_addr":"172.21.15.16:6831/37021","heartbeat_front_addr":"172.21.15.16:6829/37021","state":["exists","up"]},{"osd":1,"uuid":"4702fbc9-a927-4565-aca5-41dc69cc8264","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":37008},{"type":"v1","addr":"172.21.15.16:6801","nonce":37008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":37008},{"type":"v1","addr":"172.21.15.16:6803","nonce":37008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":37008},{"type":"v1","addr":"172.21.15.16:6807","nonce":37008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":37008},{"type":"v1","addr":"172.21.15.16:6805","nonce":37008}]},"public_addr":"172.21.15.16:6801/37008","cluster_addr":"172.21.15.16:6803/37008","heartbeat_back_addr":"172.21.15.16:6807/37008","heartbeat_front_addr":"172.21.15.16:6805/37008","state":["exists","up"]},{"osd":2,"uuid":"1e7bba46-4eba-47ef-bb81-5e2d371638ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":37009},{"type":"v1","addr":"172.21.15.16:6809","nonce":37009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":37009},{"type":"v1","addr":"172.21.15.16:6811","nonce":37009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":37009},{"type":"v1","addr":"172.21.15.16:6815","nonce":37009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":37009},{"type":"v1","addr":"172.21.15.16:6813","nonce":37009}]},"public_addr":"172.21.15.16:6809/37009","cluster_addr":"172.21.15.16:6811/37009","heartbeat_back_addr":"172.21.15.16:6815/37009","heartbeat_front_addr":"172.21.15.16:6813/37009","state":["exists","up"]},{"osd":3,"uuid":"383cacc0-5a88-4904-990e-498ca59ef48a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":37020},{"type":"v1","addr":"172.21.15.16:6817","nonce":37020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":37020},{"type":"v1","addr":"172.21.15.16:6819","nonce":37020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":37020},{"type":"v1","addr":"172.21.15.16:6823","nonce":37020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":37020},{"type":"v1","addr":"172.21.15.16:6821","nonce":37020}]},"public_addr":"172.21.15.16:6817/37020","cluster_addr":"172.21.15.16:6819/37020","heartbeat_back_addr":"172.21.15.16:6823/37020","heartbeat_front_addr":"172.21.15.16:6821/37020","state":["exists","up"]},{"osd":4,"uuid":"970500ef-95b7-42c1-b080-eed4fae7013d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":37353},{"type":"v1","addr":"172.21.15.62:6817","nonce":37353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6819","nonce":37353},{"type":"v1","addr":"172.21.15.62:6822","nonce":37353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37353},{"type":"v1","addr":"172.21.15.62:6831","nonce":37353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37353},{"type":"v1","addr":"172.21.15.62:6826","nonce":37353}]},"public_addr":"172.21.15.62:6817/37353","cluster_addr":"172.21.15.62:6822/37353","heartbeat_back_addr":"172.21.15.62:6831/37353","heartbeat_front_addr":"172.21.15.62:6826/37353","state":["exists","up"]},{"osd":5,"uuid":"022d67db-d4d8-445b-b1b9-8b446a00f5c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":37352},{"type":"v1","addr":"172.21.15.62:6801","nonce":37352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":37352},{"type":"v1","addr":"172.21.15.62:6803","nonce":37352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":37352},{"type":"v1","addr":"172.21.15.62:6807","nonce":37352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":37352},{"type":"v1","addr":"172.21.15.62:6805","nonce":37352}]},"public_addr":"172.21.15.62:6801/37352","cluster_addr":"172.21.15.62:6803/37352","heartbeat_back_addr":"172.21.15.62:6807/37352","heartbeat_front_addr":"172.21.15.62:6805/37352","state":["exists","up"]},{"osd":6,"uuid":"42bd372b-6688-4a1b-8407-e16e8c266e33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":37354},{"type":"v1","addr":"172.21.15.62:6820","nonce":37354}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":37354},{"type":"v1","addr":"172.21.15.62:6823","nonce":37354}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6829","nonce":37354},{"type":"v1","addr":"172.21.15.62:6830","nonce":37354}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6825","nonce":37354},{"type":"v1","addr":"172.21.15.62:6827","nonce":37354}]},"public_addr":"172.21.15.62:6820/37354","cluster_addr":"172.21.15.62:6823/37354","heartbeat_back_addr":"172.21.15.62:6830/37354","heartbeat_front_addr":"172.21.15.62:6827/37354","state":["exists","up"]},{"osd":7,"uuid":"ac70b2a3-f41b-4bf2-b227-7f2cbb55426d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37355},{"type":"v1","addr":"172.21.15.62:6809","nonce":37355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37355},{"type":"v1","addr":"172.21.15.62:6811","nonce":37355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":37355},{"type":"v1","addr":"172.21.15.62:6815","nonce":37355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":37355},{"type":"v1","addr":"172.21.15.62:6813","nonce":37355}]},"public_addr":"172.21.15.62:6809/37355","cluster_addr":"172.21.15.62:6811/37355","heartbeat_back_addr":"172.21.15.62:6815/37355","heartbeat_front_addr":"172.21.15.62:6813/37355","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.368896+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.107273+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.276888+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.287502+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.124791+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.251937+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.255679+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.232797+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":17,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-05-22T15:46:19.792 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.787+0000 7f07e4b1d640 1 -- 172.21.15.16:0/1592028907 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f07c00654a0 msgr2=0x7f07c0067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:19.793 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.787+0000 7f07e4b1d640 1 --2- 172.21.15.16:0/1592028907 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f07c00654a0 0x7f07c0067950 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f07e00c50a0 tx=0x7f07c8008040 comp rx=0 tx=0).stop 2023-05-22T15:46:19.793 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.787+0000 7f07e4b1d640 1 -- 172.21.15.16:0/1592028907 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f07e00afbb0 msgr2=0x7f07e014b230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:19.793 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.787+0000 7f07e4b1d640 1 --2- 172.21.15.16:0/1592028907 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f07e00afbb0 0x7f07e014b230 secure :-1 s=READY pgs=1806 cs=0 l=1 rev1=1 crypto rx=0x7f07d000b410 tx=0x7f07d000b8d0 comp rx=0 tx=0).stop 2023-05-22T15:46:19.793 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.787+0000 7f07e4b1d640 1 -- 172.21.15.16:0/1592028907 shutdown_connections 2023-05-22T15:46:19.794 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.787+0000 7f07e4b1d640 1 --2- 172.21.15.16:0/1592028907 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f07c00654a0 0x7f07c0067950 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:19.794 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.787+0000 7f07e4b1d640 1 --2- 172.21.15.16:0/1592028907 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f07e00cce20 0x7f07e0158a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:19.794 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.787+0000 7f07e4b1d640 1 --2- 172.21.15.16:0/1592028907 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f07e00b0030 0x7f07e014be40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:19.794 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.787+0000 7f07e4b1d640 1 --2- 172.21.15.16:0/1592028907 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f07e00afbb0 0x7f07e014b230 unknown :-1 s=CLOSED pgs=1806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:19.795 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.787+0000 7f07e4b1d640 1 -- 172.21.15.16:0/1592028907 >> 172.21.15.16:0/1592028907 conn(0x7f07e0069720 msgr2=0x7f07e0155730 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:46:19.795 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.787+0000 7f07e4b1d640 1 -- 172.21.15.16:0/1592028907 shutdown_connections 2023-05-22T15:46:19.795 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.787+0000 7f07e4b1d640 1 -- 172.21.15.16:0/1592028907 wait complete. 2023-05-22T15:46:19.810 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-05-22T15:26:38.019781+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-05-22T15:26:43.570514+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-05-22T15:46:19.811 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-05-22T15:46:19.943 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.935+0000 7fb5ebcaf640 1 Processor -- start 2023-05-22T15:46:19.943 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.939+0000 7fb5ebcaf640 1 -- start start 2023-05-22T15:46:19.944 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.939+0000 7fb5ebcaf640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb5e40b5a50 0x7fb5e40d2900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:19.944 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.939+0000 7fb5ebcaf640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5e40cccc0 con 0x7fb5e40d2480 2023-05-22T15:46:19.944 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.939+0000 7fb5ebcaf640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5e40ccfa0 con 0x7fb5e415bca0 2023-05-22T15:46:19.944 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.939+0000 7fb5ebcaf640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5e40cd660 con 0x7fb5e40b5a50 2023-05-22T15:46:19.944 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.939+0000 7fb5ea24c640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb5e40b5a50 0x7fb5e40d2900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:19.945 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.939+0000 7fb5e924a640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fb5e40d2480 0x7fb5e40b5340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:38794/0 (socket says 172.21.15.16:38794) 2023-05-22T15:46:19.945 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.939+0000 7fb5e924a640 1 -- 172.21.15.16:0/905718765 learned_addr learned my addr 172.21.15.16:0/905718765 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:46:19.945 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.939+0000 7fb5ea24c640 1 --2- 172.21.15.16:0/905718765 >> v2:172.21.15.16:3300/0 conn(0x7fb5e40b5a50 0x7fb5e40d2900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34346/0 (socket says 172.21.15.16:34346) 2023-05-22T15:46:19.945 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.939+0000 7fb5ea24c640 1 -- 172.21.15.16:0/905718765 >> v1:172.21.15.62:6789/0 conn(0x7fb5e415bca0 legacy=0x7fb5e40cda80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:46:19.945 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.939+0000 7fb5ea24c640 1 -- 172.21.15.16:0/905718765 >> v1:172.21.15.16:6789/0 conn(0x7fb5e40d2480 legacy=0x7fb5e40b5340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:19.946 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.939+0000 7fb5ea24c640 1 -- 172.21.15.16:0/905718765 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5e40cd280 con 0x7fb5e40b5a50 2023-05-22T15:46:19.946 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.939+0000 7fb5ea24c640 1 --2- 172.21.15.16:0/905718765 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb5e40b5a50 0x7fb5e40d2900 secure :-1 s=READY pgs=1808 cs=0 l=1 rev1=1 crypto rx=0x7fb5dc009d90 tx=0x7fb5dc01c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a75b02e272cb75d4 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:19.947 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.943+0000 7fb5e8a49640 1 -- 172.21.15.16:0/905718765 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5dc0a58e0 con 0x7fb5e40b5a50 2023-05-22T15:46:19.947 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.943+0000 7fb5e8a49640 1 -- 172.21.15.16:0/905718765 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5dc0a5a40 con 0x7fb5e40b5a50 2023-05-22T15:46:19.947 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.943+0000 7fb5e8a49640 1 -- 172.21.15.16:0/905718765 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5dc0a5dd0 con 0x7fb5e40b5a50 2023-05-22T15:46:19.948 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.943+0000 7fb5ebcaf640 1 -- 172.21.15.16:0/905718765 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb5e40b5a50 msgr2=0x7fb5e40d2900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:19.948 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.943+0000 7fb5ebcaf640 1 --2- 172.21.15.16:0/905718765 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb5e40b5a50 0x7fb5e40d2900 secure :-1 s=READY pgs=1808 cs=0 l=1 rev1=1 crypto rx=0x7fb5dc009d90 tx=0x7fb5dc01c910 comp rx=0 tx=0).stop 2023-05-22T15:46:19.948 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.943+0000 7fb5ebcaf640 1 -- 172.21.15.16:0/905718765 shutdown_connections 2023-05-22T15:46:19.948 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.943+0000 7fb5ebcaf640 1 --2- 172.21.15.16:0/905718765 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb5e40b5a50 0x7fb5e40d2900 unknown :-1 s=CLOSED pgs=1808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:19.948 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.943+0000 7fb5ebcaf640 1 -- 172.21.15.16:0/905718765 >> 172.21.15.16:0/905718765 conn(0x7fb5e40655c0 msgr2=0x7fb5e4070e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:46:19.949 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.943+0000 7fb5ebcaf640 1 -- 172.21.15.16:0/905718765 shutdown_connections 2023-05-22T15:46:19.949 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.943+0000 7fb5ebcaf640 1 -- 172.21.15.16:0/905718765 wait complete. 2023-05-22T15:46:19.950 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.943+0000 7fb5ebcaf640 1 Processor -- start 2023-05-22T15:46:19.950 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.943+0000 7fb5ebcaf640 1 -- start start 2023-05-22T15:46:19.950 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.943+0000 7fb5ebcaf640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb5e40b5a50 0x7fb5e415f1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:19.950 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.943+0000 7fb5ebcaf640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb5e40d2480 0x7fb5e4159ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:19.950 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.943+0000 7fb5ebcaf640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb5e415bca0 0x7fb5e414f310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:19.950 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.943+0000 7fb5ebcaf640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5e40ce190 con 0x7fb5e415bca0 2023-05-22T15:46:19.951 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.943+0000 7fb5ebcaf640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5e4155810 con 0x7fb5e40d2480 2023-05-22T15:46:19.951 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.943+0000 7fb5ebcaf640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5e40c95c0 con 0x7fb5e40b5a50 2023-05-22T15:46:19.951 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.943+0000 7fb5ea24c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb5e415bca0 0x7fb5e414f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:19.951 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.943+0000 7fb5ea24c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb5e415bca0 0x7fb5e414f310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34356/0 (socket says 172.21.15.16:34356) 2023-05-22T15:46:19.951 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.943+0000 7fb5ea24c640 1 -- 172.21.15.16:0/1433312362 learned_addr learned my addr 172.21.15.16:0/1433312362 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:46:19.951 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.943+0000 7fb5e9a4b640 1 --2- 172.21.15.16:0/1433312362 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb5e40b5a50 0x7fb5e415f1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:19.951 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.943+0000 7fb5e924a640 1 --2- 172.21.15.16:0/1433312362 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb5e40d2480 0x7fb5e4159ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:19.952 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.943+0000 7fb5ea24c640 1 -- 172.21.15.16:0/1433312362 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb5e40b5a50 msgr2=0x7fb5e415f1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:19.952 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.943+0000 7fb5ea24c640 1 --2- 172.21.15.16:0/1433312362 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb5e40b5a50 0x7fb5e415f1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:19.952 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.943+0000 7fb5ea24c640 1 -- 172.21.15.16:0/1433312362 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb5e40d2480 msgr2=0x7fb5e4159ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:19.952 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.943+0000 7fb5ea24c640 1 --2- 172.21.15.16:0/1433312362 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb5e40d2480 0x7fb5e4159ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:19.952 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.943+0000 7fb5ea24c640 1 -- 172.21.15.16:0/1433312362 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5dc009a70 con 0x7fb5e415bca0 2023-05-22T15:46:19.952 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.943+0000 7fb5ea24c640 1 --2- 172.21.15.16:0/1433312362 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb5e415bca0 0x7fb5e414f310 secure :-1 s=READY pgs=1809 cs=0 l=1 rev1=1 crypto rx=0x7fb5dc01a540 tx=0x7fb5dc0196b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:19.953 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.943+0000 7fb5da7fc640 1 -- 172.21.15.16:0/1433312362 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5dc0adc00 con 0x7fb5e415bca0 2023-05-22T15:46:19.953 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.943+0000 7fb5da7fc640 1 -- 172.21.15.16:0/1433312362 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5dc0add60 con 0x7fb5e415bca0 2023-05-22T15:46:19.954 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.943+0000 7fb5ebcaf640 1 -- 172.21.15.16:0/1433312362 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5e41604f0 con 0x7fb5e415bca0 2023-05-22T15:46:19.954 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.943+0000 7fb5da7fc640 1 -- 172.21.15.16:0/1433312362 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5dc0b86a0 con 0x7fb5e415bca0 2023-05-22T15:46:19.954 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.943+0000 7fb5ebcaf640 1 -- 172.21.15.16:0/1433312362 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5e4160360 con 0x7fb5e415bca0 2023-05-22T15:46:19.954 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.947+0000 7fb5ebcaf640 1 -- 172.21.15.16:0/1433312362 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5e4169c60 con 0x7fb5e415bca0 2023-05-22T15:46:19.954 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.947+0000 7fb5da7fc640 1 -- 172.21.15.16:0/1433312362 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 60) v1 ==== 84332+0+0 (secure 0 0 0) 0x7fb5dc0b4040 con 0x7fb5e415bca0 2023-05-22T15:46:19.954 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.947+0000 7fb5da7fc640 1 --2- 172.21.15.16:0/1433312362 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fb5c80654d0 0x7fb5c8067980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:19.954 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.947+0000 7fb5da7fc640 1 -- 172.21.15.16:0/1433312362 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5558+0+0 (secure 0 0 0) 0x7fb5dc11da80 con 0x7fb5e415bca0 2023-05-22T15:46:19.955 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.947+0000 7fb5e9a4b640 1 --2- 172.21.15.16:0/1433312362 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fb5c80654d0 0x7fb5c8067980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:19.955 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.947+0000 7fb5e9a4b640 1 --2- 172.21.15.16:0/1433312362 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fb5c80654d0 0x7fb5c8067980 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fb5e000b810 tx=0x7fb5e0008040 comp rx=0 tx=0).ready entity=mgr.8835 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:19.957 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:19.951+0000 7fb5da7fc640 1 -- 172.21.15.16:0/1433312362 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fb5dc0eb7d0 con 0x7fb5e415bca0 2023-05-22T15:46:20.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.107+0000 7fb5ebcaf640 1 -- 172.21.15.16:0/1433312362 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fb5e4169c60 con 0x7fb5e415bca0 2023-05-22T15:46:20.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.107+0000 7fb5da7fc640 1 -- 172.21.15.16:0/1433312362 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v143) v1 ==== 93+0+10 (secure 0 0 0) 0x7fb5dc0eedf0 con 0x7fb5e415bca0 2023-05-22T15:46:20.114 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2023-05-22T15:46:20.120 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.111+0000 7fb5ebcaf640 1 -- 172.21.15.16:0/1433312362 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fb5c80654d0 msgr2=0x7fb5c8067980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:20.120 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.111+0000 7fb5ebcaf640 1 --2- 172.21.15.16:0/1433312362 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fb5c80654d0 0x7fb5c8067980 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fb5e000b810 tx=0x7fb5e0008040 comp rx=0 tx=0).stop 2023-05-22T15:46:20.120 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.111+0000 7fb5ebcaf640 1 -- 172.21.15.16:0/1433312362 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb5e415bca0 msgr2=0x7fb5e414f310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:20.121 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.111+0000 7fb5ebcaf640 1 --2- 172.21.15.16:0/1433312362 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb5e415bca0 0x7fb5e414f310 secure :-1 s=READY pgs=1809 cs=0 l=1 rev1=1 crypto rx=0x7fb5dc01a540 tx=0x7fb5dc0196b0 comp rx=0 tx=0).stop 2023-05-22T15:46:20.121 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.111+0000 7fb5ebcaf640 1 -- 172.21.15.16:0/1433312362 shutdown_connections 2023-05-22T15:46:20.121 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.111+0000 7fb5ebcaf640 1 --2- 172.21.15.16:0/1433312362 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb5e415bca0 0x7fb5e414f310 unknown :-1 s=CLOSED pgs=1809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:20.121 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.111+0000 7fb5ebcaf640 1 --2- 172.21.15.16:0/1433312362 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb5e40d2480 0x7fb5e4159ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:20.121 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.111+0000 7fb5ebcaf640 1 --2- 172.21.15.16:0/1433312362 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fb5c80654d0 0x7fb5c8067980 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:20.121 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.111+0000 7fb5ebcaf640 1 --2- 172.21.15.16:0/1433312362 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb5e40b5a50 0x7fb5e415f1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:20.122 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.111+0000 7fb5ebcaf640 1 -- 172.21.15.16:0/1433312362 >> 172.21.15.16:0/1433312362 conn(0x7fb5e40655c0 msgr2=0x7fb5e4070e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:46:20.122 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.111+0000 7fb5ebcaf640 1 -- 172.21.15.16:0/1433312362 shutdown_connections 2023-05-22T15:46:20.122 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.111+0000 7fb5ebcaf640 1 -- 172.21.15.16:0/1433312362 wait complete. 2023-05-22T15:46:20.134 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-05-22T15:46:20.256 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.251+0000 7f7509dca640 1 Processor -- start 2023-05-22T15:46:20.257 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.251+0000 7f7509dca640 1 -- start start 2023-05-22T15:46:20.258 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.251+0000 7f7509dca640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f75040b5a50 0x7f75040d2900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:20.258 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.251+0000 7f7509dca640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75040cccc0 con 0x7f750415bca0 2023-05-22T15:46:20.259 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.251+0000 7f7509dca640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75040ccfa0 con 0x7f75040d2480 2023-05-22T15:46:20.259 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.251+0000 7f7509dca640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75040cd660 con 0x7f75040b5a50 2023-05-22T15:46:20.259 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.251+0000 7f7503fff640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f75040b5a50 0x7f75040d2900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:20.260 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.251+0000 7f75037fe640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f750415bca0 0x7f75040cda80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:38796/0 (socket says 172.21.15.16:38796) 2023-05-22T15:46:20.260 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.251+0000 7f7503fff640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f75040b5a50 0x7f75040d2900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34370/0 (socket says 172.21.15.16:34370) 2023-05-22T15:46:20.260 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.251+0000 7f75037fe640 1 -- 172.21.15.16:0/2069188245 learned_addr learned my addr 172.21.15.16:0/2069188245 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:46:20.261 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.251+0000 7f7503fff640 1 -- 172.21.15.16:0/2069188245 >> v1:172.21.15.62:6789/0 conn(0x7f75040d2480 legacy=0x7f75040b5340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:20.261 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.251+0000 7f7503fff640 1 -- 172.21.15.16:0/2069188245 >> v1:172.21.15.16:6789/0 conn(0x7f750415bca0 legacy=0x7f75040cda80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:20.261 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.251+0000 7f7503fff640 1 -- 172.21.15.16:0/2069188245 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75040cd280 con 0x7f75040b5a50 2023-05-22T15:46:20.265 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.255+0000 7f7503fff640 1 --2- 172.21.15.16:0/2069188245 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f75040b5a50 0x7f75040d2900 secure :-1 s=READY pgs=1811 cs=0 l=1 rev1=1 crypto rx=0x7f74fc009d90 tx=0x7f74fc01c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6a56c46723c286b server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:20.265 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.255+0000 7f75027fc640 1 -- 172.21.15.16:0/2069188245 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74fc0a5920 con 0x7f75040b5a50 2023-05-22T15:46:20.265 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.255+0000 7f75027fc640 1 -- 172.21.15.16:0/2069188245 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74fc0a5a80 con 0x7f75040b5a50 2023-05-22T15:46:20.265 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.255+0000 7f75027fc640 1 -- 172.21.15.16:0/2069188245 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74fc0a5e10 con 0x7f75040b5a50 2023-05-22T15:46:20.265 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.255+0000 7f7509dca640 1 -- 172.21.15.16:0/2069188245 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f75040b5a50 msgr2=0x7f75040d2900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:20.265 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.255+0000 7f7509dca640 1 --2- 172.21.15.16:0/2069188245 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f75040b5a50 0x7f75040d2900 secure :-1 s=READY pgs=1811 cs=0 l=1 rev1=1 crypto rx=0x7f74fc009d90 tx=0x7f74fc01c950 comp rx=0 tx=0).stop 2023-05-22T15:46:20.266 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.255+0000 7f7509dca640 1 -- 172.21.15.16:0/2069188245 shutdown_connections 2023-05-22T15:46:20.266 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.255+0000 7f7509dca640 1 --2- 172.21.15.16:0/2069188245 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f75040b5a50 0x7f75040d2900 unknown :-1 s=CLOSED pgs=1811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:20.266 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.255+0000 7f7509dca640 1 -- 172.21.15.16:0/2069188245 >> 172.21.15.16:0/2069188245 conn(0x7f75040655c0 msgr2=0x7f7504070e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:46:20.266 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.255+0000 7f7509dca640 1 -- 172.21.15.16:0/2069188245 shutdown_connections 2023-05-22T15:46:20.266 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.255+0000 7f7509dca640 1 -- 172.21.15.16:0/2069188245 wait complete. 2023-05-22T15:46:20.266 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.255+0000 7f7509dca640 1 Processor -- start 2023-05-22T15:46:20.266 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.255+0000 7f7509dca640 1 -- start start 2023-05-22T15:46:20.267 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.259+0000 7f7509dca640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f75040b5a50 0x7f75041529f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:20.267 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.259+0000 7f7509dca640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f75040d2480 0x7f75041532b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:20.267 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.259+0000 7f7509dca640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f750415bca0 0x7f75041537f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:20.267 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.259+0000 7f7509dca640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75040ce190 con 0x7f75040d2480 2023-05-22T15:46:20.267 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.259+0000 7f7509dca640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75041592f0 con 0x7f750415bca0 2023-05-22T15:46:20.267 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.259+0000 7f7509dca640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7504152f30 con 0x7f75040b5a50 2023-05-22T15:46:20.267 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.259+0000 7f7502ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f75040d2480 0x7f75041532b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:20.267 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.259+0000 7f7502ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f75040d2480 0x7f75041532b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34374/0 (socket says 172.21.15.16:34374) 2023-05-22T15:46:20.268 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.259+0000 7f7502ffd640 1 -- 172.21.15.16:0/2601291011 learned_addr learned my addr 172.21.15.16:0/2601291011 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:46:20.268 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.259+0000 7f75037fe640 1 --2- 172.21.15.16:0/2601291011 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f75040b5a50 0x7f75041529f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:20.268 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.259+0000 7f7503fff640 1 --2- 172.21.15.16:0/2601291011 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f750415bca0 0x7f75041537f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:20.268 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.259+0000 7f7502ffd640 1 -- 172.21.15.16:0/2601291011 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f75040b5a50 msgr2=0x7f75041529f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:20.268 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.259+0000 7f7502ffd640 1 --2- 172.21.15.16:0/2601291011 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f75040b5a50 0x7f75041529f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:20.268 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.259+0000 7f7502ffd640 1 -- 172.21.15.16:0/2601291011 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f750415bca0 msgr2=0x7f75041537f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:20.268 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.259+0000 7f7502ffd640 1 --2- 172.21.15.16:0/2601291011 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f750415bca0 0x7f75041537f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:20.269 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.259+0000 7f7502ffd640 1 -- 172.21.15.16:0/2601291011 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74fc009a70 con 0x7f75040d2480 2023-05-22T15:46:20.269 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.259+0000 7f7503fff640 1 --2- 172.21.15.16:0/2601291011 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f750415bca0 0x7f75041537f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:46:20.269 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.259+0000 7f7502ffd640 1 --2- 172.21.15.16:0/2601291011 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f75040d2480 0x7f75041532b0 secure :-1 s=READY pgs=1812 cs=0 l=1 rev1=1 crypto rx=0x7f74f800c500 tx=0x7f74f800c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:20.269 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.259+0000 7f74ebfff640 1 -- 172.21.15.16:0/2601291011 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74f8005220 con 0x7f75040d2480 2023-05-22T15:46:20.269 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.259+0000 7f74ebfff640 1 -- 172.21.15.16:0/2601291011 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74f8005380 con 0x7f75040d2480 2023-05-22T15:46:20.269 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.259+0000 7f7509dca640 1 -- 172.21.15.16:0/2601291011 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75041604f0 con 0x7f75040d2480 2023-05-22T15:46:20.269 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.259+0000 7f7509dca640 1 -- 172.21.15.16:0/2601291011 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7504160360 con 0x7f75040d2480 2023-05-22T15:46:20.270 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.259+0000 7f74ebfff640 1 -- 172.21.15.16:0/2601291011 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74f8007270 con 0x7f75040d2480 2023-05-22T15:46:20.270 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.259+0000 7f7509dca640 1 -- 172.21.15.16:0/2601291011 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74e4005260 con 0x7f75040d2480 2023-05-22T15:46:20.271 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.259+0000 7f74ebfff640 1 -- 172.21.15.16:0/2601291011 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 60) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f74f80073d0 con 0x7f75040d2480 2023-05-22T15:46:20.271 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.259+0000 7f74ebfff640 1 --2- 172.21.15.16:0/2601291011 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f74f00654a0 0x7f74f0067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:20.271 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.259+0000 7f74ebfff640 1 -- 172.21.15.16:0/2601291011 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5558+0+0 (secure 0 0 0) 0x7f74f8082070 con 0x7f75040d2480 2023-05-22T15:46:20.271 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.259+0000 7f75037fe640 1 --2- 172.21.15.16:0/2601291011 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f74f00654a0 0x7f74f0067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:20.271 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.263+0000 7f75037fe640 1 --2- 172.21.15.16:0/2601291011 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f74f00654a0 0x7f74f0067950 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f74f4005890 tx=0x7f74f4009040 comp rx=0 tx=0).ready entity=mgr.8835 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:20.271 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.263+0000 7f74ebfff640 1 -- 172.21.15.16:0/2601291011 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f74f804fde0 con 0x7f75040d2480 2023-05-22T15:46:20.424 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.419+0000 7f7509dca640 1 -- 172.21.15.16:0/2601291011 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f74e4005a90 con 0x7f75040d2480 2023-05-22T15:46:20.428 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.419+0000 7f74ebfff640 1 -- 172.21.15.16:0/2601291011 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v143) v1 ==== 92+0+10 (secure 0 0 0) 0x7f74f8053400 con 0x7f75040d2480 2023-05-22T15:46:20.429 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 8 2023-05-22T15:46:20.429 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.419+0000 7f7509dca640 1 -- 172.21.15.16:0/2601291011 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f74f00654a0 msgr2=0x7f74f0067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:20.430 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.419+0000 7f7509dca640 1 --2- 172.21.15.16:0/2601291011 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f74f00654a0 0x7f74f0067950 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f74f4005890 tx=0x7f74f4009040 comp rx=0 tx=0).stop 2023-05-22T15:46:20.430 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.419+0000 7f7509dca640 1 -- 172.21.15.16:0/2601291011 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f75040d2480 msgr2=0x7f75041532b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:20.430 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.419+0000 7f7509dca640 1 --2- 172.21.15.16:0/2601291011 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f75040d2480 0x7f75041532b0 secure :-1 s=READY pgs=1812 cs=0 l=1 rev1=1 crypto rx=0x7f74f800c500 tx=0x7f74f800c9c0 comp rx=0 tx=0).stop 2023-05-22T15:46:20.430 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.419+0000 7f7509dca640 1 -- 172.21.15.16:0/2601291011 shutdown_connections 2023-05-22T15:46:20.430 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.419+0000 7f7509dca640 1 --2- 172.21.15.16:0/2601291011 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f750415bca0 0x7f75041537f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:20.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.419+0000 7f7509dca640 1 --2- 172.21.15.16:0/2601291011 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f75040d2480 0x7f75041532b0 unknown :-1 s=CLOSED pgs=1812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:20.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.419+0000 7f7509dca640 1 --2- 172.21.15.16:0/2601291011 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f74f00654a0 0x7f74f0067950 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:20.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.419+0000 7f7509dca640 1 --2- 172.21.15.16:0/2601291011 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f75040b5a50 0x7f75041529f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:20.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.419+0000 7f7509dca640 1 -- 172.21.15.16:0/2601291011 >> 172.21.15.16:0/2601291011 conn(0x7f75040655c0 msgr2=0x7f750415af70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:46:20.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.423+0000 7f7509dca640 1 -- 172.21.15.16:0/2601291011 shutdown_connections 2023-05-22T15:46:20.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.423+0000 7f7509dca640 1 -- 172.21.15.16:0/2601291011 wait complete. 2023-05-22T15:46:20.445 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-05-22T15:46:20.558 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.551+0000 7fc269ffb640 1 Processor -- start 2023-05-22T15:46:20.558 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.551+0000 7fc269ffb640 1 -- start start 2023-05-22T15:46:20.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.555+0000 7fc269ffb640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fc2640ca4e0 0x7fc26415e460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:20.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.555+0000 7fc269ffb640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc264070df0 0x7fc264071210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:20.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.555+0000 7fc269ffb640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2640b0650 con 0x7fc2640c9b40 2023-05-22T15:46:20.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.555+0000 7fc269ffb640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2640b0370 con 0x7fc2640ca4e0 2023-05-22T15:46:20.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.555+0000 7fc269ffb640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc26415d980 con 0x7fc264070df0 2023-05-22T15:46:20.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.555+0000 7fc25affd640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fc2640c9b40 0x7fc2640afb40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:38802/0 (socket says 172.21.15.16:38802) 2023-05-22T15:46:20.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.555+0000 7fc25affd640 1 -- 172.21.15.16:0/3298274765 learned_addr learned my addr 172.21.15.16:0/3298274765 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:46:20.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.555+0000 7fc2637fe640 1 --2- 172.21.15.16:0/3298274765 >> v2:172.21.15.62:3300/0 conn(0x7fc2640ca4e0 0x7fc26415e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:20.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.555+0000 7fc263fff640 1 --2- 172.21.15.16:0/3298274765 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc264070df0 0x7fc264071210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:20.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.555+0000 7fc262ffd640 1 -- 172.21.15.16:0/3298274765 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 131221656 0 0) 0x7fc2640b0650 con 0x7fc2640c9b40 2023-05-22T15:46:20.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.555+0000 7fc262ffd640 1 -- 172.21.15.16:0/3298274765 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc250003530 con 0x7fc2640c9b40 2023-05-22T15:46:20.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.555+0000 7fc262ffd640 1 -- 172.21.15.16:0/3298274765 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1892255766 0 0) 0x7fc250003530 con 0x7fc2640c9b40 2023-05-22T15:46:20.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.555+0000 7fc262ffd640 1 -- 172.21.15.16:0/3298274765 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc264070df0 msgr2=0x7fc264071210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:46:20.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.555+0000 7fc262ffd640 1 --2- 172.21.15.16:0/3298274765 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc264070df0 0x7fc264071210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:20.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.555+0000 7fc262ffd640 1 -- 172.21.15.16:0/3298274765 >> v2:172.21.15.62:3300/0 conn(0x7fc2640ca4e0 msgr2=0x7fc26415e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:46:20.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.555+0000 7fc262ffd640 1 --2- 172.21.15.16:0/3298274765 >> v2:172.21.15.62:3300/0 conn(0x7fc2640ca4e0 0x7fc26415e460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:20.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.555+0000 7fc262ffd640 1 -- 172.21.15.16:0/3298274765 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc26415dc60 con 0x7fc2640c9b40 2023-05-22T15:46:20.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.555+0000 7fc262ffd640 1 -- 172.21.15.16:0/3298274765 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7fc24c0016a0 con 0x7fc2640c9b40 2023-05-22T15:46:20.565 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.555+0000 7fc262ffd640 1 -- 172.21.15.16:0/3298274765 >> v1:172.21.15.16:6789/0 conn(0x7fc2640c9b40 legacy=0x7fc2640afb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:20.565 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.555+0000 7fc262ffd640 1 --2- 172.21.15.16:0/3298274765 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc264070df0 0x7fc250003980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:20.565 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.555+0000 7fc262ffd640 1 --2- 172.21.15.16:0/3298274765 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc250003f00 0x7fc2500047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:20.565 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.555+0000 7fc262ffd640 1 --2- 172.21.15.16:0/3298274765 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc250004d10 0x7fc250005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:20.565 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.555+0000 7fc262ffd640 1 -- 172.21.15.16:0/3298274765 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2640b0370 con 0x7fc250003f00 2023-05-22T15:46:20.565 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.555+0000 7fc262ffd640 1 -- 172.21.15.16:0/3298274765 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc26415d980 con 0x7fc250004d10 2023-05-22T15:46:20.565 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.555+0000 7fc262ffd640 1 -- 172.21.15.16:0/3298274765 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc25000b990 con 0x7fc264070df0 2023-05-22T15:46:20.565 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.555+0000 7fc25affd640 1 --2- 172.21.15.16:0/3298274765 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc250003f00 0x7fc2500047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:20.566 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.555+0000 7fc2637fe640 1 --2- 172.21.15.16:0/3298274765 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc264070df0 0x7fc250003980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:20.566 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.555+0000 7fc25affd640 1 -- 172.21.15.16:0/3298274765 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc264070df0 msgr2=0x7fc250003980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:46:20.566 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.555+0000 7fc25affd640 1 --2- 172.21.15.16:0/3298274765 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc264070df0 0x7fc250003980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:20.566 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.555+0000 7fc25affd640 1 -- 172.21.15.16:0/3298274765 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc250004d10 msgr2=0x7fc250005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:46:20.566 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.555+0000 7fc25affd640 1 --2- 172.21.15.16:0/3298274765 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc250004d10 0x7fc250005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:20.566 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.555+0000 7fc25affd640 1 -- 172.21.15.16:0/3298274765 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc250003530 con 0x7fc250003f00 2023-05-22T15:46:20.567 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.555+0000 7fc2637fe640 1 --2- 172.21.15.16:0/3298274765 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc264070df0 0x7fc250003980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:46:20.569 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.555+0000 7fc25affd640 1 --2- 172.21.15.16:0/3298274765 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc250003f00 0x7fc2500047d0 secure :-1 s=READY pgs=1814 cs=0 l=1 rev1=1 crypto rx=0x7fc24c00c210 tx=0x7fc24c01f390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27a09d91250f6adc server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:20.570 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.559+0000 7fc262ffd640 1 -- 172.21.15.16:0/3298274765 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc24c005ae0 con 0x7fc250003f00 2023-05-22T15:46:20.570 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.559+0000 7fc262ffd640 1 -- 172.21.15.16:0/3298274765 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc24c0ab070 con 0x7fc250003f00 2023-05-22T15:46:20.570 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.559+0000 7fc262ffd640 1 -- 172.21.15.16:0/3298274765 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc24c005e80 con 0x7fc250003f00 2023-05-22T15:46:20.570 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.559+0000 7fc269ffb640 1 -- 172.21.15.16:0/3298274765 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc250003f00 msgr2=0x7fc2500047d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:20.570 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.559+0000 7fc269ffb640 1 --2- 172.21.15.16:0/3298274765 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc250003f00 0x7fc2500047d0 secure :-1 s=READY pgs=1814 cs=0 l=1 rev1=1 crypto rx=0x7fc24c00c210 tx=0x7fc24c01f390 comp rx=0 tx=0).stop 2023-05-22T15:46:20.570 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.559+0000 7fc263fff640 1 -- 172.21.15.16:0/3298274765 reap_dead start 2023-05-22T15:46:20.570 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.559+0000 7fc269ffb640 1 -- 172.21.15.16:0/3298274765 shutdown_connections 2023-05-22T15:46:20.571 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.559+0000 7fc269ffb640 1 -- 172.21.15.16:0/3298274765 >> 172.21.15.16:0/3298274765 conn(0x7fc264069720 msgr2=0x7fc264069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:46:20.571 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.559+0000 7fc269ffb640 1 -- 172.21.15.16:0/3298274765 shutdown_connections 2023-05-22T15:46:20.571 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.559+0000 7fc269ffb640 1 -- 172.21.15.16:0/3298274765 wait complete. 2023-05-22T15:46:20.571 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.559+0000 7fc269ffb640 1 Processor -- start 2023-05-22T15:46:20.571 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.559+0000 7fc269ffb640 1 -- start start 2023-05-22T15:46:20.571 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.559+0000 7fc269ffb640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc250003f00 0x7fc2641f0530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:20.571 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.559+0000 7fc269ffb640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc264157300 0x7fc2641ec430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:20.571 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.559+0000 7fc269ffb640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc264157770 0x7fc2641ec970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:20.572 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.559+0000 7fc269ffb640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc26414f2e0 con 0x7fc264157770 2023-05-22T15:46:20.572 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.559+0000 7fc269ffb640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc26414f5c0 con 0x7fc250003f00 2023-05-22T15:46:20.572 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.559+0000 7fc269ffb640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc26414f8a0 con 0x7fc264157300 2023-05-22T15:46:20.572 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.559+0000 7fc263fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc264157770 0x7fc2641ec970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:20.572 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.559+0000 7fc263fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc264157770 0x7fc2641ec970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34392/0 (socket says 172.21.15.16:34392) 2023-05-22T15:46:20.572 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.559+0000 7fc263fff640 1 -- 172.21.15.16:0/491250923 learned_addr learned my addr 172.21.15.16:0/491250923 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:46:20.572 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.559+0000 7fc2637fe640 1 --2- 172.21.15.16:0/491250923 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc250003f00 0x7fc2641f0530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:20.573 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.559+0000 7fc25affd640 1 --2- 172.21.15.16:0/491250923 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc264157300 0x7fc2641ec430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:20.573 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.559+0000 7fc263fff640 1 -- 172.21.15.16:0/491250923 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc264157300 msgr2=0x7fc2641ec430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:20.573 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.559+0000 7fc263fff640 1 --2- 172.21.15.16:0/491250923 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc264157300 0x7fc2641ec430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:20.573 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.559+0000 7fc263fff640 1 -- 172.21.15.16:0/491250923 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc250003f00 msgr2=0x7fc2641f0530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:20.573 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.559+0000 7fc263fff640 1 --2- 172.21.15.16:0/491250923 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc250003f00 0x7fc2641f0530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:20.573 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.559+0000 7fc263fff640 1 -- 172.21.15.16:0/491250923 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2640ca8b0 con 0x7fc264157770 2023-05-22T15:46:20.573 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.559+0000 7fc2637fe640 1 --2- 172.21.15.16:0/491250923 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc250003f00 0x7fc2641f0530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:46:20.573 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.559+0000 7fc25affd640 1 --2- 172.21.15.16:0/491250923 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc264157300 0x7fc2641ec430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:46:20.574 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.559+0000 7fc263fff640 1 --2- 172.21.15.16:0/491250923 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc264157770 0x7fc2641ec970 secure :-1 s=READY pgs=1815 cs=0 l=1 rev1=1 crypto rx=0x7fc2640cdef0 tx=0x7fc25c00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:20.574 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.559+0000 7fc260ff9640 1 -- 172.21.15.16:0/491250923 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc25c010040 con 0x7fc264157770 2023-05-22T15:46:20.574 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.559+0000 7fc260ff9640 1 -- 172.21.15.16:0/491250923 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc25c014810 con 0x7fc264157770 2023-05-22T15:46:20.574 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.559+0000 7fc260ff9640 1 -- 172.21.15.16:0/491250923 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc25c009b10 con 0x7fc264157770 2023-05-22T15:46:20.574 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.559+0000 7fc269ffb640 1 -- 172.21.15.16:0/491250923 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc26415df40 con 0x7fc264157770 2023-05-22T15:46:20.574 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.559+0000 7fc269ffb640 1 -- 172.21.15.16:0/491250923 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc264071210 con 0x7fc264157770 2023-05-22T15:46:20.574 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.563+0000 7fc269ffb640 1 -- 172.21.15.16:0/491250923 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc26404dbe0 con 0x7fc264157770 2023-05-22T15:46:20.574 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.563+0000 7fc260ff9640 1 -- 172.21.15.16:0/491250923 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 60) v1 ==== 84332+0+0 (secure 0 0 0) 0x7fc25c014bc0 con 0x7fc264157770 2023-05-22T15:46:20.575 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.563+0000 7fc260ff9640 1 --2- 172.21.15.16:0/491250923 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fc2440654d0 0x7fc244067980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:20.575 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.563+0000 7fc260ff9640 1 -- 172.21.15.16:0/491250923 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5558+0+0 (secure 0 0 0) 0x7fc25c0822e0 con 0x7fc264157770 2023-05-22T15:46:20.575 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.563+0000 7fc2637fe640 1 --2- 172.21.15.16:0/491250923 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fc2440654d0 0x7fc244067980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:20.575 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.563+0000 7fc2637fe640 1 --2- 172.21.15.16:0/491250923 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fc2440654d0 0x7fc244067980 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fc264157cb0 tx=0x7fc254008040 comp rx=0 tx=0).ready entity=mgr.8835 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:20.575 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.567+0000 7fc260ff9640 1 -- 172.21.15.16:0/491250923 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fc25c053960 con 0x7fc264157770 2023-05-22T15:46:20.725 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.719+0000 7fc269ffb640 1 -- 172.21.15.16:0/491250923 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fc2640cb010 con 0x7fc264157770 2023-05-22T15:46:20.725 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.719+0000 7fc260ff9640 1 -- 172.21.15.16:0/491250923 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v143) v1 ==== 84+0+126 (secure 0 0 0) 0x7fc25c053550 con 0x7fc264157770 2023-05-22T15:46:20.726 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:46:20.726 INFO:teuthology.orchestra.run.smithi016.stdout:[ 2023-05-22T15:46:20.726 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T15:46:20.726 INFO:teuthology.orchestra.run.smithi016.stdout: "poolnum": 1, 2023-05-22T15:46:20.726 INFO:teuthology.orchestra.run.smithi016.stdout: "poolname": ".mgr" 2023-05-22T15:46:20.727 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:46:20.727 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T15:46:20.727 INFO:teuthology.orchestra.run.smithi016.stdout: "poolnum": 2, 2023-05-22T15:46:20.727 INFO:teuthology.orchestra.run.smithi016.stdout: "poolname": "rbd" 2023-05-22T15:46:20.727 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T15:46:20.727 INFO:teuthology.orchestra.run.smithi016.stdout:] 2023-05-22T15:46:20.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.723+0000 7fc269ffb640 1 -- 172.21.15.16:0/491250923 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fc2440654d0 msgr2=0x7fc244067980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:20.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.723+0000 7fc269ffb640 1 --2- 172.21.15.16:0/491250923 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fc2440654d0 0x7fc244067980 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fc264157cb0 tx=0x7fc254008040 comp rx=0 tx=0).stop 2023-05-22T15:46:20.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.723+0000 7fc269ffb640 1 -- 172.21.15.16:0/491250923 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc264157770 msgr2=0x7fc2641ec970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:20.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.723+0000 7fc269ffb640 1 --2- 172.21.15.16:0/491250923 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc264157770 0x7fc2641ec970 secure :-1 s=READY pgs=1815 cs=0 l=1 rev1=1 crypto rx=0x7fc2640cdef0 tx=0x7fc25c00ee80 comp rx=0 tx=0).stop 2023-05-22T15:46:20.729 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.723+0000 7fc269ffb640 1 -- 172.21.15.16:0/491250923 shutdown_connections 2023-05-22T15:46:20.729 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.723+0000 7fc269ffb640 1 --2- 172.21.15.16:0/491250923 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc264157770 0x7fc2641ec970 unknown :-1 s=CLOSED pgs=1815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:20.729 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.723+0000 7fc269ffb640 1 --2- 172.21.15.16:0/491250923 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fc2440654d0 0x7fc244067980 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:20.729 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.723+0000 7fc269ffb640 1 --2- 172.21.15.16:0/491250923 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc264157300 0x7fc2641ec430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:20.729 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.723+0000 7fc269ffb640 1 --2- 172.21.15.16:0/491250923 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc250003f00 0x7fc2641f0530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:20.730 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.723+0000 7fc269ffb640 1 -- 172.21.15.16:0/491250923 >> 172.21.15.16:0/491250923 conn(0x7fc264069720 msgr2=0x7fc2640b30e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:46:20.730 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.723+0000 7fc269ffb640 1 -- 172.21.15.16:0/491250923 shutdown_connections 2023-05-22T15:46:20.731 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.723+0000 7fc269ffb640 1 -- 172.21.15.16:0/491250923 wait complete. 2023-05-22T15:46:20.745 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-05-22T15:46:20.746 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-05-22T15:46:20.880 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.875+0000 7f5e54d29640 1 Processor -- start 2023-05-22T15:46:20.881 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.875+0000 7f5e54d29640 1 -- start start 2023-05-22T15:46:20.882 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.875+0000 7f5e54d29640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f5e501647d0 0x7f5e50163d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:20.882 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.875+0000 7f5e54d29640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5e500caa00 0x7f5e500b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:20.882 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.875+0000 7f5e54d29640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e500c1b30 con 0x7f5e500affb0 2023-05-22T15:46:20.883 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.875+0000 7f5e54d29640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e500b0930 con 0x7f5e501647d0 2023-05-22T15:46:20.883 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.875+0000 7f5e54d29640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e500b0b50 con 0x7f5e500caa00 2023-05-22T15:46:20.883 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.875+0000 7f5e4dd9b640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f5e500affb0 0x7f5e500ca2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:38810/0 (socket says 172.21.15.16:38810) 2023-05-22T15:46:20.883 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.875+0000 7f5e4dd9b640 1 -- 172.21.15.16:0/1268980094 learned_addr learned my addr 172.21.15.16:0/1268980094 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:46:20.883 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.875+0000 7f5e4e59c640 1 --2- 172.21.15.16:0/1268980094 >> v2:172.21.15.16:3300/0 conn(0x7f5e501647d0 0x7f5e50163d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:20.883 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.875+0000 7f5e4ed9d640 1 --2- 172.21.15.16:0/1268980094 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5e500caa00 0x7f5e500b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:20.883 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.875+0000 7f5e4e59c640 1 -- 172.21.15.16:0/1268980094 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5e500caa00 msgr2=0x7f5e500b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:46:20.884 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.875+0000 7f5e4e59c640 1 --2- 172.21.15.16:0/1268980094 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5e500caa00 0x7f5e500b03f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:20.884 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.875+0000 7f5e4e59c640 1 -- 172.21.15.16:0/1268980094 >> v1:172.21.15.16:6789/0 conn(0x7f5e500affb0 legacy=0x7f5e500ca2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:20.884 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.875+0000 7f5e4e59c640 1 -- 172.21.15.16:0/1268980094 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e500cd7a0 con 0x7f5e501647d0 2023-05-22T15:46:20.884 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.875+0000 7f5e4d59a640 1 -- 172.21.15.16:0/1268980094 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 551240615 0 0) 0x7f5e500c1b30 con 0x7f5e500affb0 2023-05-22T15:46:20.885 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.879+0000 7f5e4e59c640 1 --2- 172.21.15.16:0/1268980094 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5e501647d0 0x7f5e50163d40 secure :-1 s=READY pgs=1817 cs=0 l=1 rev1=1 crypto rx=0x7f5e400097c0 tx=0x7f5e4001c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8a576aaeb90889c server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:20.885 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.879+0000 7f5e4d59a640 1 -- 172.21.15.16:0/1268980094 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e40004230 con 0x7f5e501647d0 2023-05-22T15:46:20.885 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.879+0000 7f5e4d59a640 1 -- 172.21.15.16:0/1268980094 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e400a7050 con 0x7f5e501647d0 2023-05-22T15:46:20.885 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.879+0000 7f5e4d59a640 1 -- 172.21.15.16:0/1268980094 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e40003620 con 0x7f5e501647d0 2023-05-22T15:46:20.886 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.879+0000 7f5e54d29640 1 -- 172.21.15.16:0/1268980094 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5e501647d0 msgr2=0x7f5e50163d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:20.886 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.879+0000 7f5e54d29640 1 --2- 172.21.15.16:0/1268980094 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5e501647d0 0x7f5e50163d40 secure :-1 s=READY pgs=1817 cs=0 l=1 rev1=1 crypto rx=0x7f5e400097c0 tx=0x7f5e4001c330 comp rx=0 tx=0).stop 2023-05-22T15:46:20.886 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.879+0000 7f5e54d29640 1 -- 172.21.15.16:0/1268980094 shutdown_connections 2023-05-22T15:46:20.886 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.879+0000 7f5e54d29640 1 --2- 172.21.15.16:0/1268980094 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5e500caa00 0x7f5e500b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:20.886 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.879+0000 7f5e54d29640 1 --2- 172.21.15.16:0/1268980094 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5e501647d0 0x7f5e50163d40 unknown :-1 s=CLOSED pgs=1817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:20.887 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.879+0000 7f5e54d29640 1 -- 172.21.15.16:0/1268980094 >> 172.21.15.16:0/1268980094 conn(0x7f5e500cafb0 msgr2=0x7f5e500cb3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:46:20.887 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.879+0000 7f5e54d29640 1 -- 172.21.15.16:0/1268980094 shutdown_connections 2023-05-22T15:46:20.887 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.879+0000 7f5e54d29640 1 -- 172.21.15.16:0/1268980094 wait complete. 2023-05-22T15:46:20.890 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.879+0000 7f5e54d29640 1 Processor -- start 2023-05-22T15:46:20.890 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.879+0000 7f5e54d29640 1 -- start start 2023-05-22T15:46:20.891 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.883+0000 7f5e54d29640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5e500affb0 0x7f5e500b2940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:20.891 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.883+0000 7f5e54d29640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5e500caa00 0x7f5e50155f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:20.891 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.883+0000 7f5e54d29640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5e501647d0 0x7f5e500adf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:20.891 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.883+0000 7f5e54d29640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e500c4ef0 con 0x7f5e500affb0 2023-05-22T15:46:20.891 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.883+0000 7f5e54d29640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e501f83c0 con 0x7f5e501647d0 2023-05-22T15:46:20.891 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.883+0000 7f5e54d29640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e500b2e80 con 0x7f5e500caa00 2023-05-22T15:46:20.891 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.883+0000 7f5e4e59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5e500affb0 0x7f5e500b2940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:20.891 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.883+0000 7f5e4e59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5e500affb0 0x7f5e500b2940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34404/0 (socket says 172.21.15.16:34404) 2023-05-22T15:46:20.892 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.883+0000 7f5e4e59c640 1 -- 172.21.15.16:0/2641190847 learned_addr learned my addr 172.21.15.16:0/2641190847 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:46:20.892 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.883+0000 7f5e4ed9d640 1 --2- 172.21.15.16:0/2641190847 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5e501647d0 0x7f5e500adf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:20.892 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.883+0000 7f5e4dd9b640 1 --2- 172.21.15.16:0/2641190847 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5e500caa00 0x7f5e50155f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:20.892 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.883+0000 7f5e4e59c640 1 -- 172.21.15.16:0/2641190847 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5e500caa00 msgr2=0x7f5e50155f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:20.892 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.883+0000 7f5e4e59c640 1 --2- 172.21.15.16:0/2641190847 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5e500caa00 0x7f5e50155f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:20.892 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.883+0000 7f5e4e59c640 1 -- 172.21.15.16:0/2641190847 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5e501647d0 msgr2=0x7f5e500adf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:20.892 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.883+0000 7f5e4e59c640 1 --2- 172.21.15.16:0/2641190847 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5e501647d0 0x7f5e500adf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:20.893 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.883+0000 7f5e4e59c640 1 -- 172.21.15.16:0/2641190847 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e40009420 con 0x7f5e500affb0 2023-05-22T15:46:20.893 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.883+0000 7f5e4e59c640 1 --2- 172.21.15.16:0/2641190847 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5e500affb0 0x7f5e500b2940 secure :-1 s=READY pgs=1818 cs=0 l=1 rev1=1 crypto rx=0x7f5e40019f60 tx=0x7f5e400190d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:20.893 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.883+0000 7f5e46ffd640 1 -- 172.21.15.16:0/2641190847 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e40003c70 con 0x7f5e500affb0 2023-05-22T15:46:20.893 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.883+0000 7f5e46ffd640 1 -- 172.21.15.16:0/2641190847 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e40003df0 con 0x7f5e500affb0 2023-05-22T15:46:20.893 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.883+0000 7f5e46ffd640 1 -- 172.21.15.16:0/2641190847 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e400ab680 con 0x7f5e500affb0 2023-05-22T15:46:20.893 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.883+0000 7f5e54d29640 1 -- 172.21.15.16:0/2641190847 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e50164ba0 con 0x7f5e500affb0 2023-05-22T15:46:20.893 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.883+0000 7f5e54d29640 1 -- 172.21.15.16:0/2641190847 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e500ce040 con 0x7f5e500affb0 2023-05-22T15:46:20.893 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.883+0000 7f5e54d29640 1 -- 172.21.15.16:0/2641190847 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e5006a010 con 0x7f5e500affb0 2023-05-22T15:46:20.894 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.883+0000 7f5e46ffd640 1 -- 172.21.15.16:0/2641190847 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 60) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f5e400b2040 con 0x7f5e500affb0 2023-05-22T15:46:20.894 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.883+0000 7f5e46ffd640 1 --2- 172.21.15.16:0/2641190847 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f5e340654a0 0x7f5e34067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:20.894 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.883+0000 7f5e46ffd640 1 -- 172.21.15.16:0/2641190847 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5558+0+0 (secure 0 0 0) 0x7f5e4011c9e0 con 0x7f5e500affb0 2023-05-22T15:46:20.894 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.883+0000 7f5e4dd9b640 1 --2- 172.21.15.16:0/2641190847 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f5e340654a0 0x7f5e34067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:20.894 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.883+0000 7f5e4dd9b640 1 --2- 172.21.15.16:0/2641190847 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f5e340654a0 0x7f5e34067950 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f5e5015c250 tx=0x7f5e48005ec0 comp rx=0 tx=0).ready entity=mgr.8835 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:20.894 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:20.887+0000 7f5e46ffd640 1 -- 172.21.15.16:0/2641190847 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f5e400ea5d0 con 0x7f5e500affb0 2023-05-22T15:46:21.045 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:21.039+0000 7f5e54d29640 1 -- 172.21.15.16:0/2641190847 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f5e5004dbe0 con 0x7f5e500affb0 2023-05-22T15:46:22.056 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:22.051+0000 7f5e46ffd640 1 -- 172.21.15.16:0/2641190847 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v144) v1 ==== 138+0+0 (secure 0 0 0) 0x7f5e400b00b0 con 0x7f5e500affb0 2023-05-22T15:46:22.067 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'cephfs_metadata' created 2023-05-22T15:46:22.071 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:22.063+0000 7f5e54d29640 1 -- 172.21.15.16:0/2641190847 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f5e340654a0 msgr2=0x7f5e34067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:22.071 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:22.063+0000 7f5e54d29640 1 --2- 172.21.15.16:0/2641190847 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f5e340654a0 0x7f5e34067950 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f5e5015c250 tx=0x7f5e48005ec0 comp rx=0 tx=0).stop 2023-05-22T15:46:22.071 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:22.063+0000 7f5e54d29640 1 -- 172.21.15.16:0/2641190847 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5e500affb0 msgr2=0x7f5e500b2940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:22.072 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:22.063+0000 7f5e54d29640 1 --2- 172.21.15.16:0/2641190847 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5e500affb0 0x7f5e500b2940 secure :-1 s=READY pgs=1818 cs=0 l=1 rev1=1 crypto rx=0x7f5e40019f60 tx=0x7f5e400190d0 comp rx=0 tx=0).stop 2023-05-22T15:46:22.072 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:22.063+0000 7f5e54d29640 1 -- 172.21.15.16:0/2641190847 shutdown_connections 2023-05-22T15:46:22.073 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:22.063+0000 7f5e54d29640 1 --2- 172.21.15.16:0/2641190847 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5e501647d0 0x7f5e500adf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:22.073 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:22.063+0000 7f5e54d29640 1 --2- 172.21.15.16:0/2641190847 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f5e340654a0 0x7f5e34067950 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:22.073 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:22.063+0000 7f5e54d29640 1 --2- 172.21.15.16:0/2641190847 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5e500caa00 0x7f5e50155f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:22.074 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:22.063+0000 7f5e54d29640 1 --2- 172.21.15.16:0/2641190847 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5e500affb0 0x7f5e500b2940 unknown :-1 s=CLOSED pgs=1818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:22.074 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:22.063+0000 7f5e54d29640 1 -- 172.21.15.16:0/2641190847 >> 172.21.15.16:0/2641190847 conn(0x7f5e500cafb0 msgr2=0x7f5e500cb380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:46:22.074 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:22.063+0000 7f5e54d29640 1 -- 172.21.15.16:0/2641190847 shutdown_connections 2023-05-22T15:46:22.075 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:22.063+0000 7f5e54d29640 1 -- 172.21.15.16:0/2641190847 wait complete. 2023-05-22T15:46:22.091 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-05-22T15:46:22.232 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:22.227+0000 7efea4059640 1 Processor -- start 2023-05-22T15:46:22.232 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:22.227+0000 7efea4059640 1 -- start start 2023-05-22T15:46:22.233 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:22.227+0000 7efea4059640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7efe9c15bca0 0x7efe9c0cda80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:22.234 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:22.227+0000 7efea4059640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7efe9c0b5a50 0x7efe9c0d2900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:22.234 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:22.227+0000 7efea4059640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe9c0cccc0 con 0x7efe9c0d2480 2023-05-22T15:46:22.234 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:22.227+0000 7efea4059640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe9c0cd660 con 0x7efe9c15bca0 2023-05-22T15:46:22.234 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:22.227+0000 7efea4059640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe9c0d2e40 con 0x7efe9c0b5a50 2023-05-22T15:46:22.234 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:22.227+0000 7efea1df5640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7efe9c15bca0 0x7efe9c0cda80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:22.234 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:22.227+0000 7efea1df5640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7efe9c15bca0 0x7efe9c0cda80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34420/0 (socket says 172.21.15.16:34420) 2023-05-22T15:46:22.234 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:22.227+0000 7efea1df5640 1 -- 172.21.15.16:0/2359257120 learned_addr learned my addr 172.21.15.16:0/2359257120 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:46:22.235 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:22.227+0000 7efea25f6640 1 --2- 172.21.15.16:0/2359257120 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7efe9c0b5a50 0x7efe9c0d2900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:22.235 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:22.227+0000 7efea1df5640 1 -- 172.21.15.16:0/2359257120 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7efe9c0b5a50 msgr2=0x7efe9c0d2900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:46:22.235 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:22.227+0000 7efea1df5640 1 --2- 172.21.15.16:0/2359257120 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7efe9c0b5a50 0x7efe9c0d2900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:22.235 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:22.227+0000 7efea1df5640 1 -- 172.21.15.16:0/2359257120 >> v1:172.21.15.62:6789/0 conn(0x7efe9c0d2480 legacy=0x7efe9c0b5340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:22.235 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:22.227+0000 7efea1df5640 1 -- 172.21.15.16:0/2359257120 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe9c0ccfa0 con 0x7efe9c15bca0 2023-05-22T15:46:22.235 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:22.231+0000 7efea1df5640 1 --2- 172.21.15.16:0/2359257120 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efe9c15bca0 0x7efe9c0cda80 secure :-1 s=READY pgs=1819 cs=0 l=1 rev1=1 crypto rx=0x7efe98009e50 tx=0x7efe9801ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5759c1242dc9a74 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:22.236 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:22.231+0000 7efea0df3640 1 -- 172.21.15.16:0/2359257120 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe980a5900 con 0x7efe9c15bca0 2023-05-22T15:46:22.236 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:22.231+0000 7efea0df3640 1 -- 172.21.15.16:0/2359257120 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe980a5a60 con 0x7efe9c15bca0 2023-05-22T15:46:22.236 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:22.231+0000 7efea0df3640 1 -- 172.21.15.16:0/2359257120 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe980a5df0 con 0x7efe9c15bca0 2023-05-22T15:46:22.237 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:22.231+0000 7efea4059640 1 -- 172.21.15.16:0/2359257120 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efe9c15bca0 msgr2=0x7efe9c0cda80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:22.237 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:22.231+0000 7efea4059640 1 --2- 172.21.15.16:0/2359257120 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efe9c15bca0 0x7efe9c0cda80 secure :-1 s=READY pgs=1819 cs=0 l=1 rev1=1 crypto rx=0x7efe98009e50 tx=0x7efe9801ca10 comp rx=0 tx=0).stop 2023-05-22T15:46:22.237 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:22.231+0000 7efea4059640 1 -- 172.21.15.16:0/2359257120 shutdown_connections 2023-05-22T15:46:22.237 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:22.231+0000 7efea4059640 1 --2- 172.21.15.16:0/2359257120 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7efe9c0b5a50 0x7efe9c0d2900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:22.238 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:22.231+0000 7efea4059640 1 --2- 172.21.15.16:0/2359257120 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efe9c15bca0 0x7efe9c0cda80 unknown :-1 s=CLOSED pgs=1819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:22.238 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:22.231+0000 7efea4059640 1 -- 172.21.15.16:0/2359257120 >> 172.21.15.16:0/2359257120 conn(0x7efe9c0655c0 msgr2=0x7efe9c070e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:46:22.238 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:22.231+0000 7efea4059640 1 -- 172.21.15.16:0/2359257120 shutdown_connections 2023-05-22T15:46:22.238 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:22.231+0000 7efea4059640 1 -- 172.21.15.16:0/2359257120 wait complete. 2023-05-22T15:46:22.239 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:22.231+0000 7efea4059640 1 Processor -- start 2023-05-22T15:46:22.239 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:22.231+0000 7efea4059640 1 -- start start 2023-05-22T15:46:22.239 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:22.235+0000 7efea4059640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efe9c0b5a50 0x7efe9c158800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:22.240 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:22.235+0000 7efea4059640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efe9c0d2480 0x7efe9c156e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:22.240 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:22.235+0000 7efea4059640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7efe9c15bca0 0x7efe9c152360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:22.240 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:22.235+0000 7efea4059640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe9c158de0 con 0x7efe9c0d2480 2023-05-22T15:46:22.240 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:22.235+0000 7efea4059640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe9c1590c0 con 0x7efe9c0b5a50 2023-05-22T15:46:22.240 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:22.235+0000 7efea4059640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe9c1593a0 con 0x7efe9c15bca0 2023-05-22T15:46:22.240 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:22.235+0000 7efea15f4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efe9c0d2480 0x7efe9c156e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:22.240 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:22.235+0000 7efea15f4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efe9c0d2480 0x7efe9c156e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34434/0 (socket says 172.21.15.16:34434) 2023-05-22T15:46:22.241 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:22.235+0000 7efea15f4640 1 -- 172.21.15.16:0/377703825 learned_addr learned my addr 172.21.15.16:0/377703825 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:46:22.241 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:22.235+0000 7efea25f6640 1 --2- 172.21.15.16:0/377703825 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7efe9c15bca0 0x7efe9c152360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:22.241 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:22.235+0000 7efea1df5640 1 --2- 172.21.15.16:0/377703825 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efe9c0b5a50 0x7efe9c158800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:22.241 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:22.235+0000 7efea15f4640 1 -- 172.21.15.16:0/377703825 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7efe9c15bca0 msgr2=0x7efe9c152360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:22.241 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:22.235+0000 7efea15f4640 1 --2- 172.21.15.16:0/377703825 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7efe9c15bca0 0x7efe9c152360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:22.241 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:22.235+0000 7efea15f4640 1 -- 172.21.15.16:0/377703825 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efe9c0b5a50 msgr2=0x7efe9c158800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:22.241 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:22.235+0000 7efea15f4640 1 --2- 172.21.15.16:0/377703825 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efe9c0b5a50 0x7efe9c158800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:22.242 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:22.235+0000 7efea15f4640 1 -- 172.21.15.16:0/377703825 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe98009b30 con 0x7efe9c0d2480 2023-05-22T15:46:22.242 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:22.235+0000 7efea1df5640 1 --2- 172.21.15.16:0/377703825 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efe9c0b5a50 0x7efe9c158800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T15:46:22.243 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:22.235+0000 7efea15f4640 1 --2- 172.21.15.16:0/377703825 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efe9c0d2480 0x7efe9c156e90 secure :-1 s=READY pgs=1820 cs=0 l=1 rev1=1 crypto rx=0x7efe8c00c450 tx=0x7efe8c00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:22.243 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:22.235+0000 7efe927fc640 1 -- 172.21.15.16:0/377703825 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe8c0052d0 con 0x7efe9c0d2480 2023-05-22T15:46:22.243 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:22.235+0000 7efe927fc640 1 -- 172.21.15.16:0/377703825 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe8c005450 con 0x7efe9c0d2480 2023-05-22T15:46:22.244 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:22.235+0000 7efea4059640 1 -- 172.21.15.16:0/377703825 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe9c160360 con 0x7efe9c0d2480 2023-05-22T15:46:22.244 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:22.235+0000 7efe927fc640 1 -- 172.21.15.16:0/377703825 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe8c012870 con 0x7efe9c0d2480 2023-05-22T15:46:22.244 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:22.235+0000 7efea4059640 1 -- 172.21.15.16:0/377703825 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe9c0cdfc0 con 0x7efe9c0d2480 2023-05-22T15:46:22.244 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:22.235+0000 7efea4059640 1 -- 172.21.15.16:0/377703825 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe9c1604f0 con 0x7efe9c0d2480 2023-05-22T15:46:22.244 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:22.235+0000 7efe927fc640 1 -- 172.21.15.16:0/377703825 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 60) v1 ==== 84332+0+0 (secure 0 0 0) 0x7efe8c0129d0 con 0x7efe9c0d2480 2023-05-22T15:46:22.244 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:22.235+0000 7efe927fc640 1 --2- 172.21.15.16:0/377703825 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7efe800654a0 0x7efe80067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:22.244 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:22.235+0000 7efe927fc640 1 -- 172.21.15.16:0/377703825 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 5928+0+0 (secure 0 0 0) 0x7efe8c082f10 con 0x7efe9c0d2480 2023-05-22T15:46:22.244 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:22.235+0000 7efea1df5640 1 --2- 172.21.15.16:0/377703825 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7efe800654a0 0x7efe80067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:22.245 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:22.239+0000 7efea1df5640 1 --2- 172.21.15.16:0/377703825 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7efe800654a0 0x7efe80067950 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7efe980ae000 tx=0x7efe980a7070 comp rx=0 tx=0).ready entity=mgr.8835 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:22.246 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:22.239+0000 7efe927fc640 1 -- 172.21.15.16:0/377703825 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7efe8c050be0 con 0x7efe9c0d2480 2023-05-22T15:46:22.400 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:22.395+0000 7efea4059640 1 -- 172.21.15.16:0/377703825 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7efe9c04dbe0 con 0x7efe9c0d2480 2023-05-22T15:46:22.463 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-05-22T15:46:22.466 DEBUG:teuthology.orchestra.run.smithi062:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-05-22T15:46:23.067 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.059+0000 7efe927fc640 1 -- 172.21.15.16:0/377703825 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v145) v1 ==== 170+0+0 (secure 0 0 0) 0x7efe8c054200 con 0x7efe9c0d2480 2023-05-22T15:46:23.068 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'cephfs_data' created 2023-05-22T15:46:23.078 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.071+0000 7efea4059640 1 -- 172.21.15.16:0/377703825 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7efe800654a0 msgr2=0x7efe80067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:23.079 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.071+0000 7efea4059640 1 --2- 172.21.15.16:0/377703825 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7efe800654a0 0x7efe80067950 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7efe980ae000 tx=0x7efe980a7070 comp rx=0 tx=0).stop 2023-05-22T15:46:23.079 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.071+0000 7efea4059640 1 -- 172.21.15.16:0/377703825 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efe9c0d2480 msgr2=0x7efe9c156e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:23.079 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.071+0000 7efea4059640 1 --2- 172.21.15.16:0/377703825 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efe9c0d2480 0x7efe9c156e90 secure :-1 s=READY pgs=1820 cs=0 l=1 rev1=1 crypto rx=0x7efe8c00c450 tx=0x7efe8c00c910 comp rx=0 tx=0).stop 2023-05-22T15:46:23.080 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.071+0000 7efea4059640 1 -- 172.21.15.16:0/377703825 shutdown_connections 2023-05-22T15:46:23.080 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.071+0000 7efea4059640 1 --2- 172.21.15.16:0/377703825 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7efe800654a0 0x7efe80067950 secure :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7efe980ae000 tx=0x7efe980a7070 comp rx=0 tx=0).stop 2023-05-22T15:46:23.080 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.071+0000 7efea4059640 1 --2- 172.21.15.16:0/377703825 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7efe9c15bca0 0x7efe9c152360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:23.081 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.071+0000 7efea4059640 1 --2- 172.21.15.16:0/377703825 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efe9c0d2480 0x7efe9c156e90 unknown :-1 s=CLOSED pgs=1820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:23.081 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.071+0000 7efea4059640 1 --2- 172.21.15.16:0/377703825 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efe9c0b5a50 0x7efe9c158800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:23.081 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.071+0000 7efea4059640 1 -- 172.21.15.16:0/377703825 >> 172.21.15.16:0/377703825 conn(0x7efe9c0655c0 msgr2=0x7efe9c15f2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:46:23.081 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.071+0000 7efea4059640 1 -- 172.21.15.16:0/377703825 shutdown_connections 2023-05-22T15:46:23.082 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.071+0000 7efea4059640 1 -- 172.21.15.16:0/377703825 wait complete. 2023-05-22T15:46:23.109 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-05-22T15:46:23.237 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.231+0000 7fdfd5518640 1 Processor -- start 2023-05-22T15:46:23.238 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.231+0000 7fdfd5518640 1 -- start start 2023-05-22T15:46:23.241 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.231+0000 7fdfd5518640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fdfd01596e0 0x7fdfd0163950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:23.241 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.231+0000 7fdfd5518640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdfd0159f70 con 0x7fdfd00cb4c0 2023-05-22T15:46:23.242 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.231+0000 7fdfd5518640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdfd00cb030 con 0x7fdfd006b6c0 2023-05-22T15:46:23.242 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.231+0000 7fdfd5518640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdfd0071760 con 0x7fdfd01596e0 2023-05-22T15:46:23.242 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.231+0000 7fdfce7fc640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fdfd00cb4c0 0x7fdfd0158fd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:38814/0 (socket says 172.21.15.16:38814) 2023-05-22T15:46:23.242 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.231+0000 7fdfce7fc640 1 -- 172.21.15.16:0/1887490082 learned_addr learned my addr 172.21.15.16:0/1887490082 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:46:23.243 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.235+0000 7fdfcf7fe640 1 --2- 172.21.15.16:0/1887490082 >> v2:172.21.15.62:3300/0 conn(0x7fdfd01596e0 0x7fdfd0163950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:23.243 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.235+0000 7fdfcdffb640 1 -- 172.21.15.16:0/1887490082 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4192460780 0 0) 0x7fdfd0159f70 con 0x7fdfd00cb4c0 2023-05-22T15:46:23.243 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.235+0000 7fdfcdffb640 1 -- 172.21.15.16:0/1887490082 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdfb8003530 con 0x7fdfd00cb4c0 2023-05-22T15:46:23.244 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.235+0000 7fdfcdffb640 1 -- 172.21.15.16:0/1887490082 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3916963846 0 0) 0x7fdfb8003530 con 0x7fdfd00cb4c0 2023-05-22T15:46:23.244 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.235+0000 7fdfcdffb640 1 -- 172.21.15.16:0/1887490082 >> v2:172.21.15.62:3300/0 conn(0x7fdfd01596e0 msgr2=0x7fdfd0163950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:46:23.244 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.235+0000 7fdfcdffb640 1 --2- 172.21.15.16:0/1887490082 >> v2:172.21.15.62:3300/0 conn(0x7fdfd01596e0 0x7fdfd0163950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:23.244 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.235+0000 7fdfcdffb640 1 -- 172.21.15.16:0/1887490082 >> v1:172.21.15.62:6789/0 conn(0x7fdfd006b6c0 legacy=0x7fdfd006ba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:23.244 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.235+0000 7fdfcdffb640 1 -- 172.21.15.16:0/1887490082 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfd0159c90 con 0x7fdfd00cb4c0 2023-05-22T15:46:23.244 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.235+0000 7fdfcdffb640 1 -- 172.21.15.16:0/1887490082 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7fdfc4001a00 con 0x7fdfd00cb4c0 2023-05-22T15:46:23.245 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.235+0000 7fdfcdffb640 1 -- 172.21.15.16:0/1887490082 >> v1:172.21.15.16:6789/0 conn(0x7fdfd00cb4c0 legacy=0x7fdfd0158fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:23.245 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.235+0000 7fdfcdffb640 1 --2- 172.21.15.16:0/1887490082 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdfb8003980 0x7fdfb8003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:23.245 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.235+0000 7fdfcdffb640 1 --2- 172.21.15.16:0/1887490082 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fdfb80047d0 0x7fdfb8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:23.245 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.235+0000 7fdfcdffb640 1 --2- 172.21.15.16:0/1887490082 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdfb8005180 0x7fdfb800b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:23.245 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.235+0000 7fdfcdffb640 1 -- 172.21.15.16:0/1887490082 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfd0071760 con 0x7fdfb8003980 2023-05-22T15:46:23.245 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.235+0000 7fdfcdffb640 1 -- 172.21.15.16:0/1887490082 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfb800c090 con 0x7fdfb8005180 2023-05-22T15:46:23.245 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.235+0000 7fdfcdffb640 1 -- 172.21.15.16:0/1887490082 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdfb800c250 con 0x7fdfb80047d0 2023-05-22T15:46:23.245 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.235+0000 7fdfceffd640 1 --2- 172.21.15.16:0/1887490082 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdfb8003980 0x7fdfb8003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:23.246 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.235+0000 7fdfcf7fe640 1 --2- 172.21.15.16:0/1887490082 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdfb8005180 0x7fdfb800b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:23.246 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.235+0000 7fdfce7fc640 1 --2- 172.21.15.16:0/1887490082 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fdfb80047d0 0x7fdfb8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:23.246 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.235+0000 7fdfceffd640 1 -- 172.21.15.16:0/1887490082 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fdfb80047d0 msgr2=0x7fdfb8004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:46:23.246 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.235+0000 7fdfceffd640 1 --2- 172.21.15.16:0/1887490082 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fdfb80047d0 0x7fdfb8004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:23.246 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.235+0000 7fdfceffd640 1 -- 172.21.15.16:0/1887490082 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdfb8005180 msgr2=0x7fdfb800b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:46:23.246 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.235+0000 7fdfceffd640 1 --2- 172.21.15.16:0/1887490082 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdfb8005180 0x7fdfb800b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:23.246 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.235+0000 7fdfceffd640 1 -- 172.21.15.16:0/1887490082 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfb8003530 con 0x7fdfb8003980 2023-05-22T15:46:23.247 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.235+0000 7fdfcf7fe640 1 --2- 172.21.15.16:0/1887490082 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdfb8005180 0x7fdfb800b990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:46:23.247 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.235+0000 7fdfce7fc640 1 --2- 172.21.15.16:0/1887490082 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fdfb80047d0 0x7fdfb8004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:46:23.247 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.235+0000 7fdfcf7fe640 1 -- 172.21.15.16:0/1887490082 reap_dead start 2023-05-22T15:46:23.249 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.235+0000 7fdfceffd640 1 --2- 172.21.15.16:0/1887490082 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdfb8003980 0x7fdfb8003d90 secure :-1 s=READY pgs=1822 cs=0 l=1 rev1=1 crypto rx=0x7fdfc0001780 tx=0x7fdfc001c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83ffb22400143d59 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:23.250 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.235+0000 7fdfcdffb640 1 -- 172.21.15.16:0/1887490082 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfc00052d0 con 0x7fdfb8003980 2023-05-22T15:46:23.250 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.235+0000 7fdfcdffb640 1 -- 172.21.15.16:0/1887490082 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdfc00a7050 con 0x7fdfb8003980 2023-05-22T15:46:23.250 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.235+0000 7fdfcdffb640 1 -- 172.21.15.16:0/1887490082 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfc0003600 con 0x7fdfb8003980 2023-05-22T15:46:23.250 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.239+0000 7fdfd5518640 1 -- 172.21.15.16:0/1887490082 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdfb8003980 msgr2=0x7fdfb8003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:23.250 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.239+0000 7fdfd5518640 1 --2- 172.21.15.16:0/1887490082 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdfb8003980 0x7fdfb8003d90 secure :-1 s=READY pgs=1822 cs=0 l=1 rev1=1 crypto rx=0x7fdfc0001780 tx=0x7fdfc001c3e0 comp rx=0 tx=0).stop 2023-05-22T15:46:23.251 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.239+0000 7fdfd5518640 1 -- 172.21.15.16:0/1887490082 shutdown_connections 2023-05-22T15:46:23.251 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.239+0000 7fdfd5518640 1 --2- 172.21.15.16:0/1887490082 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdfb8003980 0x7fdfb8003d90 unknown :-1 s=CLOSED pgs=1822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:23.251 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.239+0000 7fdfd5518640 1 -- 172.21.15.16:0/1887490082 >> 172.21.15.16:0/1887490082 conn(0x7fdfd0069720 msgr2=0x7fdfd0069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:46:23.251 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.239+0000 7fdfd5518640 1 -- 172.21.15.16:0/1887490082 shutdown_connections 2023-05-22T15:46:23.251 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.239+0000 7fdfd5518640 1 -- 172.21.15.16:0/1887490082 wait complete. 2023-05-22T15:46:23.252 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.239+0000 7fdfd5518640 1 Processor -- start 2023-05-22T15:46:23.252 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.239+0000 7fdfd5518640 1 -- start start 2023-05-22T15:46:23.252 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.239+0000 7fdfd5518640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdfb8003980 0x7fdfd00b29c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:23.252 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.239+0000 7fdfd5518640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdfd00c94f0 0x7fdfd01f7b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:23.253 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.239+0000 7fdfd5518640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fdfd01f80b0 0x7fdfd01f4bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:23.253 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.239+0000 7fdfd5518640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfd00c9a60 con 0x7fdfd00c94f0 2023-05-22T15:46:23.253 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.239+0000 7fdfd5518640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfd00c9d40 con 0x7fdfb8003980 2023-05-22T15:46:23.253 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.239+0000 7fdfd5518640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdfd00ca020 con 0x7fdfd01f80b0 2023-05-22T15:46:23.254 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.239+0000 7fdfce7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdfd00c94f0 0x7fdfd01f7b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:23.254 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.239+0000 7fdfce7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdfd00c94f0 0x7fdfd01f7b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34456/0 (socket says 172.21.15.16:34456) 2023-05-22T15:46:23.254 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.239+0000 7fdfce7fc640 1 -- 172.21.15.16:0/649262428 learned_addr learned my addr 172.21.15.16:0/649262428 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:46:23.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.239+0000 7fdfcf7fe640 1 --2- 172.21.15.16:0/649262428 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fdfd01f80b0 0x7fdfd01f4bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:23.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.239+0000 7fdfceffd640 1 --2- 172.21.15.16:0/649262428 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdfb8003980 0x7fdfd00b29c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:23.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.239+0000 7fdfce7fc640 1 -- 172.21.15.16:0/649262428 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fdfd01f80b0 msgr2=0x7fdfd01f4bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:23.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.239+0000 7fdfce7fc640 1 --2- 172.21.15.16:0/649262428 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fdfd01f80b0 0x7fdfd01f4bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:23.256 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.239+0000 7fdfce7fc640 1 -- 172.21.15.16:0/649262428 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdfb8003980 msgr2=0x7fdfd00b29c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:23.256 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.239+0000 7fdfce7fc640 1 --2- 172.21.15.16:0/649262428 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdfb8003980 0x7fdfd00b29c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:23.256 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.239+0000 7fdfce7fc640 1 -- 172.21.15.16:0/649262428 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfd01593e0 con 0x7fdfd00c94f0 2023-05-22T15:46:23.256 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.239+0000 7fdfce7fc640 1 --2- 172.21.15.16:0/649262428 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdfd00c94f0 0x7fdfd01f7b70 secure :-1 s=READY pgs=1823 cs=0 l=1 rev1=1 crypto rx=0x7fdfc4002140 tx=0x7fdfc400b0d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:23.257 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.239+0000 7fdfb77fe640 1 -- 172.21.15.16:0/649262428 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfc400b900 con 0x7fdfd00c94f0 2023-05-22T15:46:23.257 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.239+0000 7fdfb77fe640 1 -- 172.21.15.16:0/649262428 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdfc400ba60 con 0x7fdfd00c94f0 2023-05-22T15:46:23.257 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.239+0000 7fdfd5518640 1 -- 172.21.15.16:0/649262428 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdfd00cacc0 con 0x7fdfd00c94f0 2023-05-22T15:46:23.257 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.239+0000 7fdfb77fe640 1 -- 172.21.15.16:0/649262428 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfc400bdd0 con 0x7fdfd00c94f0 2023-05-22T15:46:23.257 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.239+0000 7fdfd5518640 1 -- 172.21.15.16:0/649262428 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdfd0156840 con 0x7fdfd00c94f0 2023-05-22T15:46:23.258 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.243+0000 7fdfd5518640 1 -- 172.21.15.16:0/649262428 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdfd004dbe0 con 0x7fdfd00c94f0 2023-05-22T15:46:23.258 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.243+0000 7fdfb77fe640 1 -- 172.21.15.16:0/649262428 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 60) v1 ==== 84332+0+0 (secure 0 0 0) 0x7fdfc40070d0 con 0x7fdfd00c94f0 2023-05-22T15:46:23.258 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.243+0000 7fdfb77fe640 1 --2- 172.21.15.16:0/649262428 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fdfbc0654a0 0x7fdfbc067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:23.258 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.243+0000 7fdfb77fe640 1 -- 172.21.15.16:0/649262428 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 6310+0+0 (secure 0 0 0) 0x7fdfc4082bf0 con 0x7fdfd00c94f0 2023-05-22T15:46:23.258 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.243+0000 7fdfceffd640 1 --2- 172.21.15.16:0/649262428 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fdfbc0654a0 0x7fdfbc067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:23.258 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.243+0000 7fdfceffd640 1 --2- 172.21.15.16:0/649262428 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fdfbc0654a0 0x7fdfbc067950 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fdfc0001750 tx=0x7fdfc00011a0 comp rx=0 tx=0).ready entity=mgr.8835 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:23.259 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.247+0000 7fdfb77fe640 1 -- 172.21.15.16:0/649262428 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fdfc40504e0 con 0x7fdfd00c94f0 2023-05-22T15:46:23.443 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.435+0000 7fdfd5518640 1 -- 172.21.15.16:0/649262428 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fdfd00cc0e0 con 0x7fdfd00c94f0 2023-05-22T15:46:23.476 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.471+0000 7fdfb77fe640 1 -- 172.21.15.16:0/649262428 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 18 and data pool 19 v54) v1 ==== 174+0+0 (secure 0 0 0) 0x7fdfc4019020 con 0x7fdfd00c94f0 2023-05-22T15:46:23.477 INFO:teuthology.orchestra.run.smithi016.stderr:new fs with metadata pool 18 and data pool 19 2023-05-22T15:46:23.482 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.475+0000 7fdfd5518640 1 -- 172.21.15.16:0/649262428 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fdfbc0654a0 msgr2=0x7fdfbc067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:23.483 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.475+0000 7fdfd5518640 1 --2- 172.21.15.16:0/649262428 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fdfbc0654a0 0x7fdfbc067950 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fdfc0001750 tx=0x7fdfc00011a0 comp rx=0 tx=0).stop 2023-05-22T15:46:23.483 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.475+0000 7fdfd5518640 1 -- 172.21.15.16:0/649262428 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdfd00c94f0 msgr2=0x7fdfd01f7b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:23.483 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.475+0000 7fdfd5518640 1 --2- 172.21.15.16:0/649262428 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdfd00c94f0 0x7fdfd01f7b70 secure :-1 s=READY pgs=1823 cs=0 l=1 rev1=1 crypto rx=0x7fdfc4002140 tx=0x7fdfc400b0d0 comp rx=0 tx=0).stop 2023-05-22T15:46:23.484 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.479+0000 7fdfd5518640 1 -- 172.21.15.16:0/649262428 shutdown_connections 2023-05-22T15:46:23.485 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.479+0000 7fdfd5518640 1 --2- 172.21.15.16:0/649262428 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fdfbc0654a0 0x7fdfbc067950 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:23.485 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.479+0000 7fdfd5518640 1 --2- 172.21.15.16:0/649262428 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fdfd01f80b0 0x7fdfd01f4bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:23.485 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.479+0000 7fdfd5518640 1 --2- 172.21.15.16:0/649262428 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdfd00c94f0 0x7fdfd01f7b70 unknown :-1 s=CLOSED pgs=1823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:23.485 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.479+0000 7fdfd5518640 1 --2- 172.21.15.16:0/649262428 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdfb8003980 0x7fdfd00b29c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:23.486 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.479+0000 7fdfd5518640 1 -- 172.21.15.16:0/649262428 >> 172.21.15.16:0/649262428 conn(0x7fdfd0069720 msgr2=0x7fdfd0155960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:46:23.486 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.479+0000 7fdfd5518640 1 -- 172.21.15.16:0/649262428 shutdown_connections 2023-05-22T15:46:23.486 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.479+0000 7fdfd5518640 1 -- 172.21.15.16:0/649262428 wait complete. 2023-05-22T15:46:23.507 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-05-22T15:46:23.627 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.623+0000 7f9dafc51640 1 Processor -- start 2023-05-22T15:46:23.627 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.623+0000 7f9dafc51640 1 -- start start 2023-05-22T15:46:23.628 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.623+0000 7f9dafc51640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f9da81531b0 0x7f9da80afb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:23.628 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.623+0000 7f9dafc51640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f9da80b0080 0x7f9da8159330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:23.628 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.623+0000 7f9dafc51640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9da815a0a0 con 0x7f9da8159870 2023-05-22T15:46:23.629 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.623+0000 7f9dafc51640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9da80b0520 con 0x7f9da80b0080 2023-05-22T15:46:23.629 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.623+0000 7f9dafc51640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9da8153580 con 0x7f9da81531b0 2023-05-22T15:46:23.629 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.623+0000 7f9dad1ec640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f9da80b0080 0x7f9da8159330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:23.629 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.623+0000 7f9dad1ec640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f9da80b0080 0x7f9da8159330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34468/0 (socket says 172.21.15.16:34468) 2023-05-22T15:46:23.629 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.623+0000 7f9dad1ec640 1 -- 172.21.15.16:0/3337572887 learned_addr learned my addr 172.21.15.16:0/3337572887 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:46:23.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.623+0000 7f9dad9ed640 1 --2- 172.21.15.16:0/3337572887 >> v2:172.21.15.62:3300/0 conn(0x7f9da81531b0 0x7f9da80afb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:23.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.623+0000 7f9dad1ec640 1 -- 172.21.15.16:0/3337572887 >> v2:172.21.15.62:3300/0 conn(0x7f9da81531b0 msgr2=0x7f9da80afb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:46:23.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.623+0000 7f9dad1ec640 1 --2- 172.21.15.16:0/3337572887 >> v2:172.21.15.62:3300/0 conn(0x7f9da81531b0 0x7f9da80afb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:23.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.623+0000 7f9dad1ec640 1 -- 172.21.15.16:0/3337572887 >> v1:172.21.15.62:6789/0 conn(0x7f9da8159870 legacy=0x7f9da80c8ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:46:23.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.623+0000 7f9dad1ec640 1 -- 172.21.15.16:0/3337572887 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9da815d5e0 con 0x7f9da80b0080 2023-05-22T15:46:23.632 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.623+0000 7f9dad9ed640 1 --2- 172.21.15.16:0/3337572887 >> v2:172.21.15.62:3300/0 conn(0x7f9da81531b0 0x7f9da80afb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:46:23.637 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.627+0000 7f9dad1ec640 1 --2- 172.21.15.16:0/3337572887 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9da80b0080 0x7f9da8159330 secure :-1 s=READY pgs=1824 cs=0 l=1 rev1=1 crypto rx=0x7f9da80b0790 tx=0x7f9d9801eb60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54c728b66b6885d0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:23.637 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.627+0000 7f9dac9eb640 1 -- 172.21.15.16:0/3337572887 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d980a7c30 con 0x7f9da80b0080 2023-05-22T15:46:23.638 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.627+0000 7f9dac9eb640 1 -- 172.21.15.16:0/3337572887 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d980a7d90 con 0x7f9da80b0080 2023-05-22T15:46:23.638 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.627+0000 7f9dac9eb640 1 -- 172.21.15.16:0/3337572887 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d980af660 con 0x7f9da80b0080 2023-05-22T15:46:23.638 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.627+0000 7f9dafc51640 1 -- 172.21.15.16:0/3337572887 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9da80b0080 msgr2=0x7f9da8159330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:23.638 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.627+0000 7f9dafc51640 1 --2- 172.21.15.16:0/3337572887 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9da80b0080 0x7f9da8159330 secure :-1 s=READY pgs=1824 cs=0 l=1 rev1=1 crypto rx=0x7f9da80b0790 tx=0x7f9d9801eb60 comp rx=0 tx=0).stop 2023-05-22T15:46:23.638 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.627+0000 7f9dafc51640 1 -- 172.21.15.16:0/3337572887 shutdown_connections 2023-05-22T15:46:23.638 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.627+0000 7f9dafc51640 1 --2- 172.21.15.16:0/3337572887 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9da80b0080 0x7f9da8159330 unknown :-1 s=CLOSED pgs=1824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:23.638 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.627+0000 7f9dafc51640 1 --2- 172.21.15.16:0/3337572887 >> v2:172.21.15.62:3300/0 conn(0x7f9da81531b0 0x7f9da80afb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:23.639 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.627+0000 7f9dafc51640 1 -- 172.21.15.16:0/3337572887 >> 172.21.15.16:0/3337572887 conn(0x7f9da8069720 msgr2=0x7f9da8069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:46:23.639 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.627+0000 7f9dafc51640 1 -- 172.21.15.16:0/3337572887 shutdown_connections 2023-05-22T15:46:23.639 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.627+0000 7f9dafc51640 1 -- 172.21.15.16:0/3337572887 wait complete. 2023-05-22T15:46:23.639 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.627+0000 7f9dafc51640 1 Processor -- start 2023-05-22T15:46:23.639 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.627+0000 7f9dafc51640 1 -- start start 2023-05-22T15:46:23.639 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.627+0000 7f9dafc51640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9da80b0080 0x7f9da80af760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:23.639 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.627+0000 7f9dafc51640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9da81531b0 0x7f9da80aecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:23.640 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.627+0000 7f9dafc51640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9da8159870 0x7f9da8150620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:23.640 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.627+0000 7f9dafc51640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9da80ae6b0 con 0x7f9da80b0080 2023-05-22T15:46:23.640 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.627+0000 7f9dafc51640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9da80ae990 con 0x7f9da81531b0 2023-05-22T15:46:23.640 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.627+0000 7f9dafc51640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9da80ce5a0 con 0x7f9da8159870 2023-05-22T15:46:23.640 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.627+0000 7f9dad9ed640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9da80b0080 0x7f9da80af760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:23.640 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.627+0000 7f9dad9ed640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9da80b0080 0x7f9da80af760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34470/0 (socket says 172.21.15.16:34470) 2023-05-22T15:46:23.640 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.627+0000 7f9dad9ed640 1 -- 172.21.15.16:0/782169945 learned_addr learned my addr 172.21.15.16:0/782169945 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:46:23.640 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.627+0000 7f9dad1ec640 1 --2- 172.21.15.16:0/782169945 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9da81531b0 0x7f9da80aecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:23.641 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.627+0000 7f9dae1ee640 1 --2- 172.21.15.16:0/782169945 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9da8159870 0x7f9da8150620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:23.641 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.627+0000 7f9dad9ed640 1 -- 172.21.15.16:0/782169945 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9da8159870 msgr2=0x7f9da8150620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:23.641 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.627+0000 7f9dad9ed640 1 --2- 172.21.15.16:0/782169945 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9da8159870 0x7f9da8150620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:23.641 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.627+0000 7f9dad9ed640 1 -- 172.21.15.16:0/782169945 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9da81531b0 msgr2=0x7f9da80aecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:23.641 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.627+0000 7f9dad9ed640 1 --2- 172.21.15.16:0/782169945 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9da81531b0 0x7f9da80aecc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:23.641 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.627+0000 7f9dad9ed640 1 -- 172.21.15.16:0/782169945 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d9800b770 con 0x7f9da80b0080 2023-05-22T15:46:23.641 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.627+0000 7f9dad1ec640 1 --2- 172.21.15.16:0/782169945 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9da81531b0 0x7f9da80aecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:46:23.641 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.631+0000 7f9dad9ed640 1 --2- 172.21.15.16:0/782169945 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9da80b0080 0x7f9da80af760 secure :-1 s=READY pgs=1825 cs=0 l=1 rev1=1 crypto rx=0x7f9da400c7f0 tx=0x7f9da400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:23.642 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.631+0000 7f9d9e7fc640 1 -- 172.21.15.16:0/782169945 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9da4007820 con 0x7f9da80b0080 2023-05-22T15:46:23.642 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.631+0000 7f9d9e7fc640 1 -- 172.21.15.16:0/782169945 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9da4007980 con 0x7f9da80b0080 2023-05-22T15:46:23.642 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.631+0000 7f9d9e7fc640 1 -- 172.21.15.16:0/782169945 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9da4007d30 con 0x7f9da80b0080 2023-05-22T15:46:23.642 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.631+0000 7f9dafc51640 1 -- 172.21.15.16:0/782169945 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9da815d0c0 con 0x7f9da80b0080 2023-05-22T15:46:23.642 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.631+0000 7f9dafc51640 1 -- 172.21.15.16:0/782169945 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9da815cde0 con 0x7f9da80b0080 2023-05-22T15:46:23.643 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.631+0000 7f9dafc51640 1 -- 172.21.15.16:0/782169945 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9da8159c80 con 0x7f9da80b0080 2023-05-22T15:46:23.643 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.631+0000 7f9d9e7fc640 1 -- 172.21.15.16:0/782169945 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 60) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f9da40113f0 con 0x7f9da80b0080 2023-05-22T15:46:23.643 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.631+0000 7f9d9e7fc640 1 --2- 172.21.15.16:0/782169945 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f9d8c0654a0 0x7f9d8c067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:23.643 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.631+0000 7f9dad1ec640 1 --2- 172.21.15.16:0/782169945 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f9d8c0654a0 0x7f9d8c067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:23.643 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.631+0000 7f9d9e7fc640 1 -- 172.21.15.16:0/782169945 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(146..146 src has 1..146) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f9da4082fe0 con 0x7f9da80b0080 2023-05-22T15:46:23.643 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.631+0000 7f9dad1ec640 1 --2- 172.21.15.16:0/782169945 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f9d8c0654a0 0x7f9d8c067950 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f9d980b1000 tx=0x7f9d98008040 comp rx=0 tx=0).ready entity=mgr.8835 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:23.644 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.635+0000 7f9d9e7fc640 1 -- 172.21.15.16:0/782169945 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f9da4050820 con 0x7f9da80b0080 2023-05-22T15:46:23.796 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.791+0000 7f9dafc51640 1 -- 172.21.15.16:0/782169945 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9da80ae7f0 con 0x7f9da80b0080 2023-05-22T15:46:23.799 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.795+0000 7f9d9e7fc640 1 -- 172.21.15.16:0/782169945 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v146) v1 ==== 74+0+18107 (secure 0 0 0) 0x7f9da80ae7f0 con 0x7f9da80b0080 2023-05-22T15:46:23.800 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:46:23.800 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":146,"fsid":"b2719834-7847-4d98-90a6-f65d57f85c49","created":"2023-05-22T15:26:25.259314+0000","modified":"2023-05-22T15:46:23.442883+0000","last_up_change":"2023-05-22T15:26:36.448524+0000","last_in_change":"2023-05-22T15:26:27.675668+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-05-22T15:26:38.019781+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-05-22T15:26:43.570514+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2023-05-22T15:46:21.084615+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"146","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_data","create_time":"2023-05-22T15:46:22.440577+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"146","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9e7ff854-b25f-4cbb-a5c2-1799d9c53d2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":37021},{"type":"v1","addr":"172.21.15.16:6825","nonce":37021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":37021},{"type":"v1","addr":"172.21.15.16:6827","nonce":37021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":37021},{"type":"v1","addr":"172.21.15.16:6831","nonce":37021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":37021},{"type":"v1","addr":"172.21.15.16:6829","nonce":37021}]},"public_addr":"172.21.15.16:6825/37021","cluster_addr":"172.21.15.16:6827/37021","heartbeat_back_addr":"172.21.15.16:6831/37021","heartbeat_front_addr":"172.21.15.16:6829/37021","state":["exists","up"]},{"osd":1,"uuid":"4702fbc9-a927-4565-aca5-41dc69cc8264","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":37008},{"type":"v1","addr":"172.21.15.16:6801","nonce":37008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":37008},{"type":"v1","addr":"172.21.15.16:6803","nonce":37008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":37008},{"type":"v1","addr":"172.21.15.16:6807","nonce":37008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":37008},{"type":"v1","addr":"172.21.15.16:6805","nonce":37008}]},"public_addr":"172.21.15.16:6801/37008","cluster_addr":"172.21.15.16:6803/37008","heartbeat_back_addr":"172.21.15.16:6807/37008","heartbeat_front_addr":"172.21.15.16:6805/37008","state":["exists","up"]},{"osd":2,"uuid":"1e7bba46-4eba-47ef-bb81-5e2d371638ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":37009},{"type":"v1","addr":"172.21.15.16:6809","nonce":37009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":37009},{"type":"v1","addr":"172.21.15.16:6811","nonce":37009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":37009},{"type":"v1","addr":"172.21.15.16:6815","nonce":37009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":37009},{"type":"v1","addr":"172.21.15.16:6813","nonce":37009}]},"public_addr":"172.21.15.16:6809/37009","cluster_addr":"172.21.15.16:6811/37009","heartbeat_back_addr":"172.21.15.16:6815/37009","heartbeat_front_addr":"172.21.15.16:6813/37009","state":["exists","up"]},{"osd":3,"uuid":"383cacc0-5a88-4904-990e-498ca59ef48a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":37020},{"type":"v1","addr":"172.21.15.16:6817","nonce":37020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":37020},{"type":"v1","addr":"172.21.15.16:6819","nonce":37020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":37020},{"type":"v1","addr":"172.21.15.16:6823","nonce":37020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":37020},{"type":"v1","addr":"172.21.15.16:6821","nonce":37020}]},"public_addr":"172.21.15.16:6817/37020","cluster_addr":"172.21.15.16:6819/37020","heartbeat_back_addr":"172.21.15.16:6823/37020","heartbeat_front_addr":"172.21.15.16:6821/37020","state":["exists","up"]},{"osd":4,"uuid":"970500ef-95b7-42c1-b080-eed4fae7013d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":37353},{"type":"v1","addr":"172.21.15.62:6817","nonce":37353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6819","nonce":37353},{"type":"v1","addr":"172.21.15.62:6822","nonce":37353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37353},{"type":"v1","addr":"172.21.15.62:6831","nonce":37353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37353},{"type":"v1","addr":"172.21.15.62:6826","nonce":37353}]},"public_addr":"172.21.15.62:6817/37353","cluster_addr":"172.21.15.62:6822/37353","heartbeat_back_addr":"172.21.15.62:6831/37353","heartbeat_front_addr":"172.21.15.62:6826/37353","state":["exists","up"]},{"osd":5,"uuid":"022d67db-d4d8-445b-b1b9-8b446a00f5c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":37352},{"type":"v1","addr":"172.21.15.62:6801","nonce":37352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":37352},{"type":"v1","addr":"172.21.15.62:6803","nonce":37352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":37352},{"type":"v1","addr":"172.21.15.62:6807","nonce":37352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":37352},{"type":"v1","addr":"172.21.15.62:6805","nonce":37352}]},"public_addr":"172.21.15.62:6801/37352","cluster_addr":"172.21.15.62:6803/37352","heartbeat_back_addr":"172.21.15.62:6807/37352","heartbeat_front_addr":"172.21.15.62:6805/37352","state":["exists","up"]},{"osd":6,"uuid":"42bd372b-6688-4a1b-8407-e16e8c266e33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":37354},{"type":"v1","addr":"172.21.15.62:6820","nonce":37354}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":37354},{"type":"v1","addr":"172.21.15.62:6823","nonce":37354}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6829","nonce":37354},{"type":"v1","addr":"172.21.15.62:6830","nonce":37354}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6825","nonce":37354},{"type":"v1","addr":"172.21.15.62:6827","nonce":37354}]},"public_addr":"172.21.15.62:6820/37354","cluster_addr":"172.21.15.62:6823/37354","heartbeat_back_addr":"172.21.15.62:6830/37354","heartbeat_front_addr":"172.21.15.62:6827/37354","state":["exists","up"]},{"osd":7,"uuid":"ac70b2a3-f41b-4bf2-b227-7f2cbb55426d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37355},{"type":"v1","addr":"172.21.15.62:6809","nonce":37355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37355},{"type":"v1","addr":"172.21.15.62:6811","nonce":37355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":37355},{"type":"v1","addr":"172.21.15.62:6815","nonce":37355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":37355},{"type":"v1","addr":"172.21.15.62:6813","nonce":37355}]},"public_addr":"172.21.15.62:6809/37355","cluster_addr":"172.21.15.62:6811/37355","heartbeat_back_addr":"172.21.15.62:6815/37355","heartbeat_front_addr":"172.21.15.62:6813/37355","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.368896+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.107273+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.276888+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.287502+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.124791+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.251937+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.255679+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.232797+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":17,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-05-22T15:46:23.803 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.795+0000 7f9dafc51640 1 -- 172.21.15.16:0/782169945 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f9d8c0654a0 msgr2=0x7f9d8c067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:23.803 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.795+0000 7f9dafc51640 1 --2- 172.21.15.16:0/782169945 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f9d8c0654a0 0x7f9d8c067950 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f9d980b1000 tx=0x7f9d98008040 comp rx=0 tx=0).stop 2023-05-22T15:46:23.803 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.795+0000 7f9dafc51640 1 -- 172.21.15.16:0/782169945 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9da80b0080 msgr2=0x7f9da80af760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:23.803 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.795+0000 7f9dafc51640 1 --2- 172.21.15.16:0/782169945 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9da80b0080 0x7f9da80af760 secure :-1 s=READY pgs=1825 cs=0 l=1 rev1=1 crypto rx=0x7f9da400c7f0 tx=0x7f9da400ccb0 comp rx=0 tx=0).stop 2023-05-22T15:46:23.803 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.795+0000 7f9dafc51640 1 -- 172.21.15.16:0/782169945 shutdown_connections 2023-05-22T15:46:23.804 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.795+0000 7f9dafc51640 1 --2- 172.21.15.16:0/782169945 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f9d8c0654a0 0x7f9d8c067950 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:23.804 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.795+0000 7f9dafc51640 1 --2- 172.21.15.16:0/782169945 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9da8159870 0x7f9da8150620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:23.804 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.795+0000 7f9dafc51640 1 --2- 172.21.15.16:0/782169945 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9da81531b0 0x7f9da80aecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:23.804 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.795+0000 7f9dafc51640 1 --2- 172.21.15.16:0/782169945 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9da80b0080 0x7f9da80af760 unknown :-1 s=CLOSED pgs=1825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:23.804 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.795+0000 7f9dafc51640 1 -- 172.21.15.16:0/782169945 >> 172.21.15.16:0/782169945 conn(0x7f9da8069720 msgr2=0x7f9da8157ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:46:23.804 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.795+0000 7f9dafc51640 1 -- 172.21.15.16:0/782169945 shutdown_connections 2023-05-22T15:46:23.804 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.795+0000 7f9dafc51640 1 -- 172.21.15.16:0/782169945 wait complete. 2023-05-22T15:46:23.818 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-05-22T15:46:23.946 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.935+0000 7f3f1bbbb640 1 Processor -- start 2023-05-22T15:46:23.946 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.939+0000 7f3f1bbbb640 1 -- start start 2023-05-22T15:46:23.946 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.939+0000 7f3f1bbbb640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f3f1415fc50 0x7f3f14160020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:23.947 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.939+0000 7f3f1bbbb640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3f140d0740 0x7f3f140ce790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:23.947 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.939+0000 7f3f1bbbb640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f1415c490 con 0x7f3f1415c920 2023-05-22T15:46:23.947 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.939+0000 7f3f1bbbb640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f1415c140 con 0x7f3f1415fc50 2023-05-22T15:46:23.947 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.939+0000 7f3f1bbbb640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f140ccb50 con 0x7f3f140d0740 2023-05-22T15:46:23.947 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.939+0000 7f3f19957640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f3f1415fc50 0x7f3f14160020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:23.947 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.939+0000 7f3f19957640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f3f1415fc50 0x7f3f14160020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34480/0 (socket says 172.21.15.16:34480) 2023-05-22T15:46:23.947 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.939+0000 7f3f19957640 1 -- 172.21.15.16:0/713982532 learned_addr learned my addr 172.21.15.16:0/713982532 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:46:23.948 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.939+0000 7f3f1a158640 1 --2- 172.21.15.16:0/713982532 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3f140d0740 0x7f3f140ce790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:23.948 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.939+0000 7f3f19957640 1 -- 172.21.15.16:0/713982532 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3f140d0740 msgr2=0x7f3f140ce790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:46:23.948 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.939+0000 7f3f19957640 1 --2- 172.21.15.16:0/713982532 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3f140d0740 0x7f3f140ce790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:23.948 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.939+0000 7f3f19957640 1 -- 172.21.15.16:0/713982532 >> v1:172.21.15.62:6789/0 conn(0x7f3f1415c920 legacy=0x7f3f140d0030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:23.948 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.939+0000 7f3f19957640 1 -- 172.21.15.16:0/713982532 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f140cce30 con 0x7f3f1415fc50 2023-05-22T15:46:23.950 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.939+0000 7f3f19957640 1 --2- 172.21.15.16:0/713982532 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3f1415fc50 0x7f3f14160020 secure :-1 s=READY pgs=1826 cs=0 l=1 rev1=1 crypto rx=0x7f3f10009870 tx=0x7f3f1001c510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7fadb27dca8ddda2 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:23.951 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.939+0000 7f3f18955640 1 -- 172.21.15.16:0/713982532 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f100ab620 con 0x7f3f1415fc50 2023-05-22T15:46:23.951 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.939+0000 7f3f18955640 1 -- 172.21.15.16:0/713982532 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f100ab780 con 0x7f3f1415fc50 2023-05-22T15:46:23.951 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.943+0000 7f3f18955640 1 -- 172.21.15.16:0/713982532 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f100abb10 con 0x7f3f1415fc50 2023-05-22T15:46:23.951 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.943+0000 7f3f1bbbb640 1 -- 172.21.15.16:0/713982532 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3f1415fc50 msgr2=0x7f3f14160020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:23.951 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.943+0000 7f3f1bbbb640 1 --2- 172.21.15.16:0/713982532 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3f1415fc50 0x7f3f14160020 secure :-1 s=READY pgs=1826 cs=0 l=1 rev1=1 crypto rx=0x7f3f10009870 tx=0x7f3f1001c510 comp rx=0 tx=0).stop 2023-05-22T15:46:23.951 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.943+0000 7f3f1bbbb640 1 -- 172.21.15.16:0/713982532 shutdown_connections 2023-05-22T15:46:23.952 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.943+0000 7f3f1bbbb640 1 --2- 172.21.15.16:0/713982532 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3f140d0740 0x7f3f140ce790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:23.952 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.943+0000 7f3f1bbbb640 1 --2- 172.21.15.16:0/713982532 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3f1415fc50 0x7f3f14160020 unknown :-1 s=CLOSED pgs=1826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:23.952 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.943+0000 7f3f1bbbb640 1 -- 172.21.15.16:0/713982532 >> 172.21.15.16:0/713982532 conn(0x7f3f140657f0 msgr2=0x7f3f14070e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:46:23.952 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.943+0000 7f3f1bbbb640 1 -- 172.21.15.16:0/713982532 shutdown_connections 2023-05-22T15:46:23.952 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.943+0000 7f3f1bbbb640 1 -- 172.21.15.16:0/713982532 wait complete. 2023-05-22T15:46:23.952 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.943+0000 7f3f1bbbb640 1 Processor -- start 2023-05-22T15:46:23.952 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.943+0000 7f3f1bbbb640 1 -- start start 2023-05-22T15:46:23.953 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.943+0000 7f3f1bbbb640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3f140d0740 0x7f3f140b3390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:23.953 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.943+0000 7f3f1bbbb640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3f1415c920 0x7f3f14153240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:23.953 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.943+0000 7f3f1bbbb640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3f1415fc50 0x7f3f140c7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:23.953 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.943+0000 7f3f1bbbb640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f140c6c10 con 0x7f3f140d0740 2023-05-22T15:46:23.953 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.943+0000 7f3f1bbbb640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f140c6070 con 0x7f3f1415fc50 2023-05-22T15:46:23.954 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.943+0000 7f3f1bbbb640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f140c6e20 con 0x7f3f1415c920 2023-05-22T15:46:23.954 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.943+0000 7f3f19957640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3f140d0740 0x7f3f140b3390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:23.954 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.943+0000 7f3f19957640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3f140d0740 0x7f3f140b3390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34486/0 (socket says 172.21.15.16:34486) 2023-05-22T15:46:23.954 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.943+0000 7f3f19957640 1 -- 172.21.15.16:0/1240026638 learned_addr learned my addr 172.21.15.16:0/1240026638 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:46:23.954 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.943+0000 7f3f19156640 1 --2- 172.21.15.16:0/1240026638 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3f1415c920 0x7f3f14153240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:23.954 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.943+0000 7f3f1a158640 1 --2- 172.21.15.16:0/1240026638 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3f1415fc50 0x7f3f140c7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:23.954 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.943+0000 7f3f19957640 1 -- 172.21.15.16:0/1240026638 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3f1415c920 msgr2=0x7f3f14153240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:23.955 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.943+0000 7f3f19957640 1 --2- 172.21.15.16:0/1240026638 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3f1415c920 0x7f3f14153240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:23.955 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.943+0000 7f3f19957640 1 -- 172.21.15.16:0/1240026638 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3f1415fc50 msgr2=0x7f3f140c7be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:23.955 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.943+0000 7f3f19957640 1 --2- 172.21.15.16:0/1240026638 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3f1415fc50 0x7f3f140c7be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:23.956 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.943+0000 7f3f19957640 1 -- 172.21.15.16:0/1240026638 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f100094d0 con 0x7f3f140d0740 2023-05-22T15:46:23.956 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.943+0000 7f3f19957640 1 --2- 172.21.15.16:0/1240026638 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3f140d0740 0x7f3f140b3390 secure :-1 s=READY pgs=1827 cs=0 l=1 rev1=1 crypto rx=0x7f3f100ad040 tx=0x7f3f1001ad50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:23.956 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.943+0000 7f3f0a7fc640 1 -- 172.21.15.16:0/1240026638 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f100be3f0 con 0x7f3f140d0740 2023-05-22T15:46:23.956 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.943+0000 7f3f0a7fc640 1 -- 172.21.15.16:0/1240026638 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f100be550 con 0x7f3f140d0740 2023-05-22T15:46:23.956 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.943+0000 7f3f1bbbb640 1 -- 172.21.15.16:0/1240026638 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f140cf670 con 0x7f3f140d0740 2023-05-22T15:46:23.956 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.943+0000 7f3f0a7fc640 1 -- 172.21.15.16:0/1240026638 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f100be900 con 0x7f3f140d0740 2023-05-22T15:46:23.956 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.943+0000 7f3f1bbbb640 1 -- 172.21.15.16:0/1240026638 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f140cf4e0 con 0x7f3f140d0740 2023-05-22T15:46:23.957 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.947+0000 7f3f1bbbb640 1 -- 172.21.15.16:0/1240026638 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f140cf800 con 0x7f3f140d0740 2023-05-22T15:46:23.958 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.947+0000 7f3f0a7fc640 1 -- 172.21.15.16:0/1240026638 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 60) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f3f100b43f0 con 0x7f3f140d0740 2023-05-22T15:46:23.958 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.947+0000 7f3f0a7fc640 1 --2- 172.21.15.16:0/1240026638 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f3ef80654d0 0x7f3ef8067980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:23.958 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.947+0000 7f3f0a7fc640 1 -- 172.21.15.16:0/1240026638 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(146..146 src has 1..146) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f3f1011c9b0 con 0x7f3f140d0740 2023-05-22T15:46:23.958 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.947+0000 7f3f19156640 1 --2- 172.21.15.16:0/1240026638 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f3ef80654d0 0x7f3ef8067980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:23.958 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.947+0000 7f3f19156640 1 --2- 172.21.15.16:0/1240026638 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f3ef80654d0 0x7f3ef8067980 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f3f140b3100 tx=0x7f3f04009040 comp rx=0 tx=0).ready entity=mgr.8835 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:23.958 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:23.951+0000 7f3f0a7fc640 1 -- 172.21.15.16:0/1240026638 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f3f140cf800 con 0x7f3f140d0740 2023-05-22T15:46:24.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:24.107+0000 7f3f1bbbb640 1 -- 172.21.15.16:0/1240026638 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3f140cf800 con 0x7f3f140d0740 2023-05-22T15:46:24.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:24.111+0000 7f3f0a7fc640 1 -- 172.21.15.16:0/1240026638 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v146) v1 ==== 74+0+18107 (secure 0 0 0) 0x7f3f140cf800 con 0x7f3f140d0740 2023-05-22T15:46:24.115 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:46:24.115 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":146,"fsid":"b2719834-7847-4d98-90a6-f65d57f85c49","created":"2023-05-22T15:26:25.259314+0000","modified":"2023-05-22T15:46:23.442883+0000","last_up_change":"2023-05-22T15:26:36.448524+0000","last_in_change":"2023-05-22T15:26:27.675668+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-05-22T15:26:38.019781+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-05-22T15:26:43.570514+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2023-05-22T15:46:21.084615+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"146","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_data","create_time":"2023-05-22T15:46:22.440577+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"146","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9e7ff854-b25f-4cbb-a5c2-1799d9c53d2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":37021},{"type":"v1","addr":"172.21.15.16:6825","nonce":37021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":37021},{"type":"v1","addr":"172.21.15.16:6827","nonce":37021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":37021},{"type":"v1","addr":"172.21.15.16:6831","nonce":37021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":37021},{"type":"v1","addr":"172.21.15.16:6829","nonce":37021}]},"public_addr":"172.21.15.16:6825/37021","cluster_addr":"172.21.15.16:6827/37021","heartbeat_back_addr":"172.21.15.16:6831/37021","heartbeat_front_addr":"172.21.15.16:6829/37021","state":["exists","up"]},{"osd":1,"uuid":"4702fbc9-a927-4565-aca5-41dc69cc8264","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":37008},{"type":"v1","addr":"172.21.15.16:6801","nonce":37008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":37008},{"type":"v1","addr":"172.21.15.16:6803","nonce":37008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":37008},{"type":"v1","addr":"172.21.15.16:6807","nonce":37008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":37008},{"type":"v1","addr":"172.21.15.16:6805","nonce":37008}]},"public_addr":"172.21.15.16:6801/37008","cluster_addr":"172.21.15.16:6803/37008","heartbeat_back_addr":"172.21.15.16:6807/37008","heartbeat_front_addr":"172.21.15.16:6805/37008","state":["exists","up"]},{"osd":2,"uuid":"1e7bba46-4eba-47ef-bb81-5e2d371638ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":37009},{"type":"v1","addr":"172.21.15.16:6809","nonce":37009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":37009},{"type":"v1","addr":"172.21.15.16:6811","nonce":37009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":37009},{"type":"v1","addr":"172.21.15.16:6815","nonce":37009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":37009},{"type":"v1","addr":"172.21.15.16:6813","nonce":37009}]},"public_addr":"172.21.15.16:6809/37009","cluster_addr":"172.21.15.16:6811/37009","heartbeat_back_addr":"172.21.15.16:6815/37009","heartbeat_front_addr":"172.21.15.16:6813/37009","state":["exists","up"]},{"osd":3,"uuid":"383cacc0-5a88-4904-990e-498ca59ef48a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":37020},{"type":"v1","addr":"172.21.15.16:6817","nonce":37020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":37020},{"type":"v1","addr":"172.21.15.16:6819","nonce":37020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":37020},{"type":"v1","addr":"172.21.15.16:6823","nonce":37020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":37020},{"type":"v1","addr":"172.21.15.16:6821","nonce":37020}]},"public_addr":"172.21.15.16:6817/37020","cluster_addr":"172.21.15.16:6819/37020","heartbeat_back_addr":"172.21.15.16:6823/37020","heartbeat_front_addr":"172.21.15.16:6821/37020","state":["exists","up"]},{"osd":4,"uuid":"970500ef-95b7-42c1-b080-eed4fae7013d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":37353},{"type":"v1","addr":"172.21.15.62:6817","nonce":37353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6819","nonce":37353},{"type":"v1","addr":"172.21.15.62:6822","nonce":37353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37353},{"type":"v1","addr":"172.21.15.62:6831","nonce":37353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37353},{"type":"v1","addr":"172.21.15.62:6826","nonce":37353}]},"public_addr":"172.21.15.62:6817/37353","cluster_addr":"172.21.15.62:6822/37353","heartbeat_back_addr":"172.21.15.62:6831/37353","heartbeat_front_addr":"172.21.15.62:6826/37353","state":["exists","up"]},{"osd":5,"uuid":"022d67db-d4d8-445b-b1b9-8b446a00f5c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":37352},{"type":"v1","addr":"172.21.15.62:6801","nonce":37352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":37352},{"type":"v1","addr":"172.21.15.62:6803","nonce":37352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":37352},{"type":"v1","addr":"172.21.15.62:6807","nonce":37352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":37352},{"type":"v1","addr":"172.21.15.62:6805","nonce":37352}]},"public_addr":"172.21.15.62:6801/37352","cluster_addr":"172.21.15.62:6803/37352","heartbeat_back_addr":"172.21.15.62:6807/37352","heartbeat_front_addr":"172.21.15.62:6805/37352","state":["exists","up"]},{"osd":6,"uuid":"42bd372b-6688-4a1b-8407-e16e8c266e33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":37354},{"type":"v1","addr":"172.21.15.62:6820","nonce":37354}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":37354},{"type":"v1","addr":"172.21.15.62:6823","nonce":37354}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6829","nonce":37354},{"type":"v1","addr":"172.21.15.62:6830","nonce":37354}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6825","nonce":37354},{"type":"v1","addr":"172.21.15.62:6827","nonce":37354}]},"public_addr":"172.21.15.62:6820/37354","cluster_addr":"172.21.15.62:6823/37354","heartbeat_back_addr":"172.21.15.62:6830/37354","heartbeat_front_addr":"172.21.15.62:6827/37354","state":["exists","up"]},{"osd":7,"uuid":"ac70b2a3-f41b-4bf2-b227-7f2cbb55426d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37355},{"type":"v1","addr":"172.21.15.62:6809","nonce":37355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37355},{"type":"v1","addr":"172.21.15.62:6811","nonce":37355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":37355},{"type":"v1","addr":"172.21.15.62:6815","nonce":37355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":37355},{"type":"v1","addr":"172.21.15.62:6813","nonce":37355}]},"public_addr":"172.21.15.62:6809/37355","cluster_addr":"172.21.15.62:6811/37355","heartbeat_back_addr":"172.21.15.62:6815/37355","heartbeat_front_addr":"172.21.15.62:6813/37355","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.368896+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.107273+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.276888+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.287502+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.124791+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.251937+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.255679+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.232797+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":17,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-05-22T15:46:24.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:24.111+0000 7f3f1bbbb640 1 -- 172.21.15.16:0/1240026638 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f3ef80654d0 msgr2=0x7f3ef8067980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:24.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:24.111+0000 7f3f1bbbb640 1 --2- 172.21.15.16:0/1240026638 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f3ef80654d0 0x7f3ef8067980 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f3f140b3100 tx=0x7f3f04009040 comp rx=0 tx=0).stop 2023-05-22T15:46:24.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:24.111+0000 7f3f1bbbb640 1 -- 172.21.15.16:0/1240026638 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3f140d0740 msgr2=0x7f3f140b3390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:24.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:24.111+0000 7f3f1bbbb640 1 --2- 172.21.15.16:0/1240026638 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3f140d0740 0x7f3f140b3390 secure :-1 s=READY pgs=1827 cs=0 l=1 rev1=1 crypto rx=0x7f3f100ad040 tx=0x7f3f1001ad50 comp rx=0 tx=0).stop 2023-05-22T15:46:24.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:24.111+0000 7f3f1bbbb640 1 -- 172.21.15.16:0/1240026638 shutdown_connections 2023-05-22T15:46:24.118 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:24.111+0000 7f3f1bbbb640 1 --2- 172.21.15.16:0/1240026638 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3f1415fc50 0x7f3f140c7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:24.118 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:24.111+0000 7f3f1bbbb640 1 --2- 172.21.15.16:0/1240026638 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f3ef80654d0 0x7f3ef8067980 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:24.118 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:24.111+0000 7f3f1bbbb640 1 --2- 172.21.15.16:0/1240026638 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3f1415c920 0x7f3f14153240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:24.118 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:24.111+0000 7f3f1bbbb640 1 --2- 172.21.15.16:0/1240026638 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3f140d0740 0x7f3f140b3390 unknown :-1 s=CLOSED pgs=1827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:24.118 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:24.111+0000 7f3f1bbbb640 1 -- 172.21.15.16:0/1240026638 >> 172.21.15.16:0/1240026638 conn(0x7f3f140657f0 msgr2=0x7f3f140b71a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:46:24.118 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:24.111+0000 7f3f1bbbb640 1 -- 172.21.15.16:0/1240026638 shutdown_connections 2023-05-22T15:46:24.118 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:24.111+0000 7f3f1bbbb640 1 -- 172.21.15.16:0/1240026638 wait complete. 2023-05-22T15:46:24.135 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-05-22T15:46:24.267 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:24.259+0000 7f3bf5c05640 1 Processor -- start 2023-05-22T15:46:24.267 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:24.259+0000 7f3bf5c05640 1 -- start start 2023-05-22T15:46:24.270 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:24.263+0000 7f3bf5c05640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f3bf0071640 0x7f3bf0160090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:24.270 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:24.263+0000 7f3bf5c05640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f3bf01605d0 0x7f3bf015c270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:24.271 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:24.263+0000 7f3bf5c05640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3bf0069ae0 con 0x7f3bf00ce480 2023-05-22T15:46:24.271 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:24.263+0000 7f3bf5c05640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3bf015f990 con 0x7f3bf01605d0 2023-05-22T15:46:24.271 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:24.263+0000 7f3bf5c05640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3bf0071ac0 con 0x7f3bf0071640 2023-05-22T15:46:24.271 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:24.263+0000 7f3bef7fe640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f3bf00ce480 0x7f3bf00ce850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:38830/0 (socket says 172.21.15.16:38830) 2023-05-22T15:46:24.272 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:24.263+0000 7f3bef7fe640 1 -- 172.21.15.16:0/272316039 learned_addr learned my addr 172.21.15.16:0/272316039 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:46:24.272 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:24.263+0000 7f3beffff640 1 --2- 172.21.15.16:0/272316039 >> v2:172.21.15.16:3300/0 conn(0x7f3bf01605d0 0x7f3bf015c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:24.272 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:24.263+0000 7f3beeffd640 1 --2- 172.21.15.16:0/272316039 >> v2:172.21.15.62:3300/0 conn(0x7f3bf0071640 0x7f3bf0160090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:24.272 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:24.263+0000 7f3bee7fc640 1 -- 172.21.15.16:0/272316039 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2991685138 0 0) 0x7f3bf0069ae0 con 0x7f3bf00ce480 2023-05-22T15:46:24.273 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:24.263+0000 7f3bee7fc640 1 -- 172.21.15.16:0/272316039 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3bdc003530 con 0x7f3bf00ce480 2023-05-22T15:46:24.273 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:24.263+0000 7f3beffff640 1 -- 172.21.15.16:0/272316039 >> v2:172.21.15.62:3300/0 conn(0x7f3bf0071640 msgr2=0x7f3bf0160090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:46:24.273 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:24.263+0000 7f3beffff640 1 --2- 172.21.15.16:0/272316039 >> v2:172.21.15.62:3300/0 conn(0x7f3bf0071640 0x7f3bf0160090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:24.273 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:24.263+0000 7f3beffff640 1 -- 172.21.15.16:0/272316039 >> v1:172.21.15.16:6789/0 conn(0x7f3bf00ce480 legacy=0x7f3bf00ce850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:24.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:24.263+0000 7f3beffff640 1 -- 172.21.15.16:0/272316039 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bf0069dc0 con 0x7f3bf01605d0 2023-05-22T15:46:24.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:24.263+0000 7f3beffff640 1 --2- 172.21.15.16:0/272316039 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3bf01605d0 0x7f3bf015c270 secure :-1 s=READY pgs=1829 cs=0 l=1 rev1=1 crypto rx=0x7f3bf00b1d50 tx=0x7f3be801c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ad5d4390d69d6d6 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:24.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:24.263+0000 7f3bee7fc640 1 -- 172.21.15.16:0/272316039 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3be80a59a0 con 0x7f3bf01605d0 2023-05-22T15:46:24.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:24.263+0000 7f3bee7fc640 1 -- 172.21.15.16:0/272316039 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3be80a5b00 con 0x7f3bf01605d0 2023-05-22T15:46:24.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:24.263+0000 7f3bee7fc640 1 -- 172.21.15.16:0/272316039 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3be80a5e90 con 0x7f3bf01605d0 2023-05-22T15:46:24.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:24.267+0000 7f3bf5c05640 1 -- 172.21.15.16:0/272316039 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3bf01605d0 msgr2=0x7f3bf015c270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:24.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:24.267+0000 7f3bf5c05640 1 --2- 172.21.15.16:0/272316039 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3bf01605d0 0x7f3bf015c270 secure :-1 s=READY pgs=1829 cs=0 l=1 rev1=1 crypto rx=0x7f3bf00b1d50 tx=0x7f3be801c8e0 comp rx=0 tx=0).stop 2023-05-22T15:46:24.282 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:24.267+0000 7f3bf5c05640 1 -- 172.21.15.16:0/272316039 shutdown_connections 2023-05-22T15:46:24.282 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:24.267+0000 7f3bf5c05640 1 --2- 172.21.15.16:0/272316039 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3bf01605d0 0x7f3bf015c270 unknown :-1 s=CLOSED pgs=1829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:24.282 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:24.267+0000 7f3bf5c05640 1 --2- 172.21.15.16:0/272316039 >> v2:172.21.15.62:3300/0 conn(0x7f3bf0071640 0x7f3bf0160090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:24.282 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:24.267+0000 7f3bf5c05640 1 -- 172.21.15.16:0/272316039 >> 172.21.15.16:0/272316039 conn(0x7f3bf00cc870 msgr2=0x7f3bf00cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:46:24.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:24.267+0000 7f3bf5c05640 1 -- 172.21.15.16:0/272316039 shutdown_connections 2023-05-22T15:46:24.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:24.267+0000 7f3bf5c05640 1 -- 172.21.15.16:0/272316039 wait complete. 2023-05-22T15:46:24.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:24.267+0000 7f3bf5c05640 1 Processor -- start 2023-05-22T15:46:24.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:24.267+0000 7f3bf5c05640 1 -- start start 2023-05-22T15:46:24.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:24.267+0000 7f3bf5c05640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3bf0071640 0x7f3bf00ad0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:24.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:24.267+0000 7f3bf5c05640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3bf00ce480 0x7f3bf015e740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:24.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:24.267+0000 7f3bf5c05640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3bf01605d0 0x7f3bf015ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:24.285 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:24.267+0000 7f3bf5c05640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bf00aebc0 con 0x7f3bf0071640 2023-05-22T15:46:24.285 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:24.267+0000 7f3bf5c05640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bf00af3f0 con 0x7f3bf00ce480 2023-05-22T15:46:24.285 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:24.267+0000 7f3bf5c05640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3bf00ada10 con 0x7f3bf01605d0 2023-05-22T15:46:24.285 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:24.267+0000 7f3bef7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3bf0071640 0x7f3bf00ad0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:24.286 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:24.267+0000 7f3bef7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3bf0071640 0x7f3bf00ad0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34508/0 (socket says 172.21.15.16:34508) 2023-05-22T15:46:24.286 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:24.267+0000 7f3bef7fe640 1 -- 172.21.15.16:0/294674154 learned_addr learned my addr 172.21.15.16:0/294674154 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:46:24.286 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:24.267+0000 7f3beeffd640 1 --2- 172.21.15.16:0/294674154 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3bf00ce480 0x7f3bf015e740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:24.287 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:24.267+0000 7f3bef7fe640 1 -- 172.21.15.16:0/294674154 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3bf01605d0 msgr2=0x7f3bf015ec80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:24.287 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:24.267+0000 7f3bef7fe640 1 --2- 172.21.15.16:0/294674154 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3bf01605d0 0x7f3bf015ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:24.288 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:24.267+0000 7f3bef7fe640 1 -- 172.21.15.16:0/294674154 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3bf00ce480 msgr2=0x7f3bf015e740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:24.288 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:24.267+0000 7f3bef7fe640 1 --2- 172.21.15.16:0/294674154 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3bf00ce480 0x7f3bf015e740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:24.288 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:24.267+0000 7f3bef7fe640 1 -- 172.21.15.16:0/294674154 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3be8009a70 con 0x7f3bf0071640 2023-05-22T15:46:24.289 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:24.267+0000 7f3bef7fe640 1 --2- 172.21.15.16:0/294674154 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3bf0071640 0x7f3bf00ad0e0 secure :-1 s=READY pgs=1830 cs=0 l=1 rev1=1 crypto rx=0x7f3bd800c4d0 tx=0x7f3bd800c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:24.289 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:24.267+0000 7f3be7fff640 1 -- 172.21.15.16:0/294674154 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bd8005cf0 con 0x7f3bf0071640 2023-05-22T15:46:24.289 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:24.267+0000 7f3be7fff640 1 -- 172.21.15.16:0/294674154 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3bd8005e50 con 0x7f3bf0071640 2023-05-22T15:46:24.290 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:24.267+0000 7f3be7fff640 1 -- 172.21.15.16:0/294674154 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bd8007270 con 0x7f3bf0071640 2023-05-22T15:46:24.290 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:24.267+0000 7f3bf5c05640 1 -- 172.21.15.16:0/294674154 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3bf00698b0 con 0x7f3bf0071640 2023-05-22T15:46:24.290 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:24.267+0000 7f3bf5c05640 1 -- 172.21.15.16:0/294674154 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3bf0069720 con 0x7f3bf0071640 2023-05-22T15:46:24.290 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:24.271+0000 7f3bf5c05640 1 -- 172.21.15.16:0/294674154 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3bf00c5e70 con 0x7f3bf0071640 2023-05-22T15:46:24.290 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:24.271+0000 7f3be7fff640 1 -- 172.21.15.16:0/294674154 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 60) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f3bd80073d0 con 0x7f3bf0071640 2023-05-22T15:46:24.290 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:24.271+0000 7f3be7fff640 1 --2- 172.21.15.16:0/294674154 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f3bd00654a0 0x7f3bd0067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:24.291 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:24.271+0000 7f3be7fff640 1 -- 172.21.15.16:0/294674154 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(146..146 src has 1..146) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f3bd8081f80 con 0x7f3bf0071640 2023-05-22T15:46:24.291 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:24.271+0000 7f3beeffd640 1 --2- 172.21.15.16:0/294674154 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f3bd00654a0 0x7f3bd0067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:24.291 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:24.271+0000 7f3beeffd640 1 --2- 172.21.15.16:0/294674154 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f3bd00654a0 0x7f3bd0067950 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f3bf00abdb0 tx=0x7f3be0008040 comp rx=0 tx=0).ready entity=mgr.8835 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:24.292 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:24.275+0000 7f3be7fff640 1 -- 172.21.15.16:0/294674154 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f3bd804a620 con 0x7f3bf0071640 2023-05-22T15:46:24.462 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:24.455+0000 7f3bf5c05640 1 -- 172.21.15.16:0/294674154 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f3bf004dbe0 con 0x7f3bf0071640 2023-05-22T15:46:25.473 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:25.467+0000 7f3be7fff640 1 -- 172.21.15.16:0/294674154 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v57) v1 ==== 118+0+0 (secure 0 0 0) 0x7f3bd80170b0 con 0x7f3bf0071640 2023-05-22T15:46:25.476 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:25.471+0000 7f3bf5c05640 1 -- 172.21.15.16:0/294674154 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f3bd00654a0 msgr2=0x7f3bd0067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:25.477 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:25.471+0000 7f3bf5c05640 1 --2- 172.21.15.16:0/294674154 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f3bd00654a0 0x7f3bd0067950 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f3bf00abdb0 tx=0x7f3be0008040 comp rx=0 tx=0).stop 2023-05-22T15:46:25.477 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:25.471+0000 7f3bf5c05640 1 -- 172.21.15.16:0/294674154 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3bf0071640 msgr2=0x7f3bf00ad0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:25.477 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:25.471+0000 7f3bf5c05640 1 --2- 172.21.15.16:0/294674154 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3bf0071640 0x7f3bf00ad0e0 secure :-1 s=READY pgs=1830 cs=0 l=1 rev1=1 crypto rx=0x7f3bd800c4d0 tx=0x7f3bd800c990 comp rx=0 tx=0).stop 2023-05-22T15:46:25.477 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:25.471+0000 7f3bf5c05640 1 -- 172.21.15.16:0/294674154 shutdown_connections 2023-05-22T15:46:25.477 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:25.471+0000 7f3bf5c05640 1 --2- 172.21.15.16:0/294674154 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f3bd00654a0 0x7f3bd0067950 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:25.477 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:25.471+0000 7f3bf5c05640 1 --2- 172.21.15.16:0/294674154 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3bf01605d0 0x7f3bf015ec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:25.477 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:25.471+0000 7f3bf5c05640 1 --2- 172.21.15.16:0/294674154 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3bf00ce480 0x7f3bf015e740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:25.478 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:25.471+0000 7f3bf5c05640 1 --2- 172.21.15.16:0/294674154 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3bf0071640 0x7f3bf00ad0e0 unknown :-1 s=CLOSED pgs=1830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:25.478 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:25.471+0000 7f3bf5c05640 1 -- 172.21.15.16:0/294674154 >> 172.21.15.16:0/294674154 conn(0x7f3bf00cc870 msgr2=0x7f3bf015b900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:46:25.478 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:25.471+0000 7f3bf5c05640 1 -- 172.21.15.16:0/294674154 shutdown_connections 2023-05-22T15:46:25.478 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:25.471+0000 7f3bf5c05640 1 -- 172.21.15.16:0/294674154 wait complete. 2023-05-22T15:46:25.496 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2023-05-22T15:46:25.496 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-05-22T15:46:25.628 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:25.623+0000 7fbd38340640 1 Processor -- start 2023-05-22T15:46:25.628 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:25.623+0000 7fbd38340640 1 -- start start 2023-05-22T15:46:25.629 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:25.623+0000 7fbd38340640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fbd300d2480 0x7fbd300d2900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:25.629 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:25.623+0000 7fbd38340640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd300cccc0 con 0x7fbd3015bca0 2023-05-22T15:46:25.629 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:25.623+0000 7fbd38340640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd300ccfa0 con 0x7fbd300b5340 2023-05-22T15:46:25.629 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:25.623+0000 7fbd38340640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd300cd660 con 0x7fbd300d2480 2023-05-22T15:46:25.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:25.623+0000 7fbd360dc640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fbd3015bca0 0x7fbd300cda80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:38834/0 (socket says 172.21.15.16:38834) 2023-05-22T15:46:25.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:25.623+0000 7fbd360dc640 1 -- 172.21.15.16:0/4284931890 learned_addr learned my addr 172.21.15.16:0/4284931890 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:46:25.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:25.623+0000 7fbd358db640 1 --2- 172.21.15.16:0/4284931890 >> v2:172.21.15.16:3300/0 conn(0x7fbd300d2480 0x7fbd300d2900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:25.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:25.623+0000 7fbd350da640 1 -- 172.21.15.16:0/4284931890 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 333582460 0 0) 0x7fbd300cccc0 con 0x7fbd3015bca0 2023-05-22T15:46:25.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:25.623+0000 7fbd350da640 1 -- 172.21.15.16:0/4284931890 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd1c003530 con 0x7fbd3015bca0 2023-05-22T15:46:25.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:25.623+0000 7fbd358db640 1 -- 172.21.15.16:0/4284931890 >> v1:172.21.15.62:6789/0 conn(0x7fbd300b5340 legacy=0x7fbd300b5790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:25.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:25.623+0000 7fbd358db640 1 -- 172.21.15.16:0/4284931890 >> v1:172.21.15.16:6789/0 conn(0x7fbd3015bca0 legacy=0x7fbd300cda80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:25.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:25.623+0000 7fbd358db640 1 -- 172.21.15.16:0/4284931890 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd300cd280 con 0x7fbd300d2480 2023-05-22T15:46:25.632 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:25.627+0000 7fbd358db640 1 --2- 172.21.15.16:0/4284931890 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbd300d2480 0x7fbd300d2900 secure :-1 s=READY pgs=1832 cs=0 l=1 rev1=1 crypto rx=0x7fbd240097c0 tx=0x7fbd2401c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1604c562ce969b4 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:25.632 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:25.627+0000 7fbd350da640 1 -- 172.21.15.16:0/4284931890 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd240a8050 con 0x7fbd300d2480 2023-05-22T15:46:25.632 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:25.627+0000 7fbd350da640 1 -- 172.21.15.16:0/4284931890 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd240ac5f0 con 0x7fbd300d2480 2023-05-22T15:46:25.632 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:25.627+0000 7fbd350da640 1 -- 172.21.15.16:0/4284931890 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd240ac980 con 0x7fbd300d2480 2023-05-22T15:46:25.633 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:25.627+0000 7fbd38340640 1 -- 172.21.15.16:0/4284931890 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbd300d2480 msgr2=0x7fbd300d2900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:25.633 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:25.627+0000 7fbd38340640 1 --2- 172.21.15.16:0/4284931890 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbd300d2480 0x7fbd300d2900 secure :-1 s=READY pgs=1832 cs=0 l=1 rev1=1 crypto rx=0x7fbd240097c0 tx=0x7fbd2401c480 comp rx=0 tx=0).stop 2023-05-22T15:46:25.633 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:25.627+0000 7fbd38340640 1 -- 172.21.15.16:0/4284931890 shutdown_connections 2023-05-22T15:46:25.633 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:25.627+0000 7fbd38340640 1 --2- 172.21.15.16:0/4284931890 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbd300d2480 0x7fbd300d2900 unknown :-1 s=CLOSED pgs=1832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:25.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:25.627+0000 7fbd38340640 1 -- 172.21.15.16:0/4284931890 >> 172.21.15.16:0/4284931890 conn(0x7fbd300655c0 msgr2=0x7fbd30070e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:46:25.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:25.627+0000 7fbd38340640 1 -- 172.21.15.16:0/4284931890 shutdown_connections 2023-05-22T15:46:25.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:25.627+0000 7fbd38340640 1 -- 172.21.15.16:0/4284931890 wait complete. 2023-05-22T15:46:25.635 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:25.627+0000 7fbd38340640 1 Processor -- start 2023-05-22T15:46:25.636 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:25.627+0000 7fbd38340640 1 -- start start 2023-05-22T15:46:25.636 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:25.627+0000 7fbd38340640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbd300b5340 0x7fbd3015f150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:25.636 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:25.627+0000 7fbd38340640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbd300d2480 0x7fbd3014f8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:25.636 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:25.631+0000 7fbd38340640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbd3015bca0 0x7fbd300b2ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:25.636 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:25.631+0000 7fbd38340640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd300ce190 con 0x7fbd3015bca0 2023-05-22T15:46:25.636 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:25.631+0000 7fbd38340640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd3015f690 con 0x7fbd300b5340 2023-05-22T15:46:25.637 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:25.631+0000 7fbd38340640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd30157fc0 con 0x7fbd300d2480 2023-05-22T15:46:25.637 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:25.631+0000 7fbd368dd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbd3015bca0 0x7fbd300b2ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:25.637 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:25.631+0000 7fbd368dd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbd3015bca0 0x7fbd300b2ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34536/0 (socket says 172.21.15.16:34536) 2023-05-22T15:46:25.637 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:25.631+0000 7fbd368dd640 1 -- 172.21.15.16:0/4280241667 learned_addr learned my addr 172.21.15.16:0/4280241667 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:46:25.637 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:25.631+0000 7fbd360dc640 1 --2- 172.21.15.16:0/4280241667 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbd300b5340 0x7fbd3015f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:25.637 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:25.631+0000 7fbd358db640 1 --2- 172.21.15.16:0/4280241667 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbd300d2480 0x7fbd3014f8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:25.637 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:25.631+0000 7fbd368dd640 1 -- 172.21.15.16:0/4280241667 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbd300d2480 msgr2=0x7fbd3014f8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:25.638 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:25.631+0000 7fbd368dd640 1 --2- 172.21.15.16:0/4280241667 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbd300d2480 0x7fbd3014f8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:25.638 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:25.631+0000 7fbd368dd640 1 -- 172.21.15.16:0/4280241667 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbd300b5340 msgr2=0x7fbd3015f150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:25.638 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:25.631+0000 7fbd368dd640 1 --2- 172.21.15.16:0/4280241667 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbd300b5340 0x7fbd3015f150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:25.638 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:25.631+0000 7fbd368dd640 1 -- 172.21.15.16:0/4280241667 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd24009420 con 0x7fbd3015bca0 2023-05-22T15:46:25.638 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:25.631+0000 7fbd368dd640 1 --2- 172.21.15.16:0/4280241667 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbd3015bca0 0x7fbd300b2ac0 secure :-1 s=READY pgs=1833 cs=0 l=1 rev1=1 crypto rx=0x7fbd2800c770 tx=0x7fbd2800cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:25.638 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:25.631+0000 7fbd22ffd640 1 -- 172.21.15.16:0/4280241667 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd28003b80 con 0x7fbd3015bca0 2023-05-22T15:46:25.639 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:25.631+0000 7fbd22ffd640 1 -- 172.21.15.16:0/4280241667 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd28003ce0 con 0x7fbd3015bca0 2023-05-22T15:46:25.639 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:25.631+0000 7fbd38340640 1 -- 172.21.15.16:0/4280241667 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd301604f0 con 0x7fbd3015bca0 2023-05-22T15:46:25.639 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:25.631+0000 7fbd22ffd640 1 -- 172.21.15.16:0/4280241667 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd2800aa90 con 0x7fbd3015bca0 2023-05-22T15:46:25.639 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:25.631+0000 7fbd38340640 1 -- 172.21.15.16:0/4280241667 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd30160360 con 0x7fbd3015bca0 2023-05-22T15:46:25.640 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:25.631+0000 7fbd38340640 1 -- 172.21.15.16:0/4280241667 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd30169c60 con 0x7fbd3015bca0 2023-05-22T15:46:25.640 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:25.631+0000 7fbd22ffd640 1 -- 172.21.15.16:0/4280241667 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 60) v1 ==== 84332+0+0 (secure 0 0 0) 0x7fbd2800abf0 con 0x7fbd3015bca0 2023-05-22T15:46:25.640 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:25.631+0000 7fbd22ffd640 1 --2- 172.21.15.16:0/4280241667 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fbd14065470 0x7fbd14067920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:25.640 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:25.631+0000 7fbd22ffd640 1 -- 172.21.15.16:0/4280241667 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(147..147 src has 1..147) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fbd280828d0 con 0x7fbd3015bca0 2023-05-22T15:46:25.640 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:25.631+0000 7fbd360dc640 1 --2- 172.21.15.16:0/4280241667 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fbd14065470 0x7fbd14067920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:25.641 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:25.631+0000 7fbd360dc640 1 --2- 172.21.15.16:0/4280241667 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fbd14065470 0x7fbd14067920 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fbd2c0011f0 tx=0x7fbd2c009040 comp rx=0 tx=0).ready entity=mgr.8835 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:25.641 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:25.635+0000 7fbd22ffd640 1 -- 172.21.15.16:0/4280241667 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fbd28050370 con 0x7fbd3015bca0 2023-05-22T15:46:25.831 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:25.823+0000 7fbd38340640 1 -- 172.21.15.16:0/4280241667 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fbd3004dbe0 con 0x7fbd3015bca0 2023-05-22T15:46:26.567 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:26.563+0000 7fbd22ffd640 1 -- 172.21.15.16:0/4280241667 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v58) v1 ==== 122+0+0 (secure 0 0 0) 0x7fbd28053990 con 0x7fbd3015bca0 2023-05-22T15:46:26.570 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:26.563+0000 7fbd38340640 1 -- 172.21.15.16:0/4280241667 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fbd14065470 msgr2=0x7fbd14067920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:26.571 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:26.563+0000 7fbd38340640 1 --2- 172.21.15.16:0/4280241667 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fbd14065470 0x7fbd14067920 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fbd2c0011f0 tx=0x7fbd2c009040 comp rx=0 tx=0).stop 2023-05-22T15:46:26.571 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:26.563+0000 7fbd38340640 1 -- 172.21.15.16:0/4280241667 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbd3015bca0 msgr2=0x7fbd300b2ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:26.571 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:26.563+0000 7fbd38340640 1 --2- 172.21.15.16:0/4280241667 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbd3015bca0 0x7fbd300b2ac0 secure :-1 s=READY pgs=1833 cs=0 l=1 rev1=1 crypto rx=0x7fbd2800c770 tx=0x7fbd2800cc30 comp rx=0 tx=0).stop 2023-05-22T15:46:26.571 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:26.563+0000 7fbd38340640 1 -- 172.21.15.16:0/4280241667 shutdown_connections 2023-05-22T15:46:26.571 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:26.563+0000 7fbd38340640 1 --2- 172.21.15.16:0/4280241667 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbd3015bca0 0x7fbd300b2ac0 unknown :-1 s=CLOSED pgs=1833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:26.571 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:26.563+0000 7fbd38340640 1 --2- 172.21.15.16:0/4280241667 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fbd14065470 0x7fbd14067920 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:26.572 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:26.563+0000 7fbd38340640 1 --2- 172.21.15.16:0/4280241667 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbd300d2480 0x7fbd3014f8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:26.572 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:26.563+0000 7fbd38340640 1 --2- 172.21.15.16:0/4280241667 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbd300b5340 0x7fbd3015f150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:26.572 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:26.563+0000 7fbd38340640 1 -- 172.21.15.16:0/4280241667 >> 172.21.15.16:0/4280241667 conn(0x7fbd300655c0 msgr2=0x7fbd300cda80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:46:26.572 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:26.563+0000 7fbd38340640 1 -- 172.21.15.16:0/4280241667 shutdown_connections 2023-05-22T15:46:26.572 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:26.563+0000 7fbd38340640 1 -- 172.21.15.16:0/4280241667 wait complete. 2023-05-22T15:46:26.590 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2023-05-22T15:46:26.591 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-05-22T15:46:26.719 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:26.715+0000 7f389446c640 1 Processor -- start 2023-05-22T15:46:26.719 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:26.715+0000 7f389446c640 1 -- start start 2023-05-22T15:46:26.720 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:26.715+0000 7f389446c640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f388c0d1930 0x7f388c1668b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:26.720 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:26.715+0000 7f389446c640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f388c0b80c0 con 0x7f388c15f340 2023-05-22T15:46:26.720 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:26.715+0000 7f389446c640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f388c1605b0 con 0x7f388c0d2dc0 2023-05-22T15:46:26.720 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:26.715+0000 7f389446c640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f388c0d0160 con 0x7f388c0d1930 2023-05-22T15:46:26.720 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:26.715+0000 7f3892208640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f388c15f340 0x7f388c15f790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:38846/0 (socket says 172.21.15.16:38846) 2023-05-22T15:46:26.721 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:26.715+0000 7f3892208640 1 -- 172.21.15.16:0/664595904 learned_addr learned my addr 172.21.15.16:0/664595904 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:46:26.722 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:26.715+0000 7f3891206640 1 -- 172.21.15.16:0/664595904 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3325952019 0 0) 0x7f388c0b80c0 con 0x7f388c15f340 2023-05-22T15:46:26.722 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:26.715+0000 7f3891206640 1 -- 172.21.15.16:0/664595904 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3878003530 con 0x7f388c15f340 2023-05-22T15:46:26.722 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:26.715+0000 7f3892a09640 1 --2- 172.21.15.16:0/664595904 >> v2:172.21.15.62:3300/0 conn(0x7f388c0d1930 0x7f388c1668b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:26.722 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:26.715+0000 7f3891206640 1 -- 172.21.15.16:0/664595904 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3790216523 0 0) 0x7f3878003530 con 0x7f388c15f340 2023-05-22T15:46:26.723 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:26.715+0000 7f3891206640 1 -- 172.21.15.16:0/664595904 >> v2:172.21.15.62:3300/0 conn(0x7f388c0d1930 msgr2=0x7f388c1668b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:46:26.723 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:26.715+0000 7f3891206640 1 --2- 172.21.15.16:0/664595904 >> v2:172.21.15.62:3300/0 conn(0x7f388c0d1930 0x7f388c1668b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:26.723 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:26.715+0000 7f3891206640 1 -- 172.21.15.16:0/664595904 >> v1:172.21.15.62:6789/0 conn(0x7f388c0d2dc0 legacy=0x7f388c15fea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:26.723 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:26.715+0000 7f3891206640 1 -- 172.21.15.16:0/664595904 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f388c0ccc50 con 0x7f388c15f340 2023-05-22T15:46:26.723 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:26.715+0000 7f3891206640 1 -- 172.21.15.16:0/664595904 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7f38880016a0 con 0x7f388c15f340 2023-05-22T15:46:26.723 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:26.715+0000 7f3891206640 1 -- 172.21.15.16:0/664595904 >> v1:172.21.15.16:6789/0 conn(0x7f388c15f340 legacy=0x7f388c15f790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:26.723 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:26.715+0000 7f3891206640 1 --2- 172.21.15.16:0/664595904 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3878003980 0x7f3878003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:26.724 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:26.715+0000 7f3891206640 1 --2- 172.21.15.16:0/664595904 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38780047d0 0x7f3878004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:26.724 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:26.715+0000 7f3891206640 1 --2- 172.21.15.16:0/664595904 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f38780052a0 0x7f387800b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:26.724 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:26.715+0000 7f3891206640 1 -- 172.21.15.16:0/664595904 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f388c0d0160 con 0x7f38780047d0 2023-05-22T15:46:26.724 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:26.715+0000 7f3891206640 1 -- 172.21.15.16:0/664595904 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f387800c160 con 0x7f38780052a0 2023-05-22T15:46:26.724 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:26.715+0000 7f3891206640 1 -- 172.21.15.16:0/664595904 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f387800c320 con 0x7f3878003980 2023-05-22T15:46:26.724 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:26.715+0000 7f3891a07640 1 --2- 172.21.15.16:0/664595904 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38780047d0 0x7f3878004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:26.724 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:26.715+0000 7f3892208640 1 --2- 172.21.15.16:0/664595904 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3878003980 0x7f3878003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:26.725 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:26.715+0000 7f3892a09640 1 --2- 172.21.15.16:0/664595904 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f38780052a0 0x7f387800b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:26.725 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:26.715+0000 7f3891a07640 1 -- 172.21.15.16:0/664595904 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3878003980 msgr2=0x7f3878003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:46:26.725 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:26.715+0000 7f3891a07640 1 --2- 172.21.15.16:0/664595904 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3878003980 0x7f3878003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:26.725 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:26.715+0000 7f3891a07640 1 -- 172.21.15.16:0/664595904 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f38780052a0 msgr2=0x7f387800b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:46:26.725 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:26.715+0000 7f3891a07640 1 --2- 172.21.15.16:0/664595904 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f38780052a0 0x7f387800b990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:26.725 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:26.715+0000 7f3892a09640 1 -- 172.21.15.16:0/664595904 reap_dead start 2023-05-22T15:46:26.725 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:26.715+0000 7f3891a07640 1 -- 172.21.15.16:0/664595904 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3878003530 con 0x7f38780047d0 2023-05-22T15:46:26.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:26.719+0000 7f3891a07640 1 --2- 172.21.15.16:0/664595904 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38780047d0 0x7f3878004c40 secure :-1 s=READY pgs=1835 cs=0 l=1 rev1=1 crypto rx=0x7f387c001780 tx=0x7f387c01c3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3362bd26615618a3 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:26.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:26.719+0000 7f3891206640 1 -- 172.21.15.16:0/664595904 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f387c0052d0 con 0x7f38780047d0 2023-05-22T15:46:26.729 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:26.719+0000 7f3891206640 1 -- 172.21.15.16:0/664595904 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f387c0a7050 con 0x7f38780047d0 2023-05-22T15:46:26.729 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:26.719+0000 7f3891206640 1 -- 172.21.15.16:0/664595904 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f387c003600 con 0x7f38780047d0 2023-05-22T15:46:26.729 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:26.719+0000 7f389446c640 1 -- 172.21.15.16:0/664595904 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38780047d0 msgr2=0x7f3878004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:26.729 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:26.719+0000 7f389446c640 1 --2- 172.21.15.16:0/664595904 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38780047d0 0x7f3878004c40 secure :-1 s=READY pgs=1835 cs=0 l=1 rev1=1 crypto rx=0x7f387c001780 tx=0x7f387c01c3b0 comp rx=0 tx=0).stop 2023-05-22T15:46:26.729 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:26.719+0000 7f389446c640 1 -- 172.21.15.16:0/664595904 shutdown_connections 2023-05-22T15:46:26.729 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:26.719+0000 7f389446c640 1 --2- 172.21.15.16:0/664595904 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38780047d0 0x7f3878004c40 unknown :-1 s=CLOSED pgs=1835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:26.729 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:26.719+0000 7f389446c640 1 -- 172.21.15.16:0/664595904 >> 172.21.15.16:0/664595904 conn(0x7f388c0655c0 msgr2=0x7f388c070e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:46:26.729 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:26.719+0000 7f389446c640 1 -- 172.21.15.16:0/664595904 shutdown_connections 2023-05-22T15:46:26.730 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:26.719+0000 7f389446c640 1 -- 172.21.15.16:0/664595904 wait complete. 2023-05-22T15:46:26.730 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:26.719+0000 7f389446c640 1 Processor -- start 2023-05-22T15:46:26.730 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:26.719+0000 7f389446c640 1 -- start start 2023-05-22T15:46:26.730 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:26.719+0000 7f389446c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f38780047d0 0x7f388c0b5450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:26.730 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:26.719+0000 7f389446c640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f388c0c5b40 0x7f388c0b5990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:26.730 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:26.719+0000 7f389446c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f388c1e85b0 0x7f388c1e8a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:26.730 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:26.719+0000 7f389446c640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f388c144030 con 0x7f388c1e85b0 2023-05-22T15:46:26.731 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:26.719+0000 7f389446c640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f388c1f4210 con 0x7f38780047d0 2023-05-22T15:46:26.731 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:26.719+0000 7f389446c640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f388c0b61c0 con 0x7f388c0c5b40 2023-05-22T15:46:26.731 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:26.719+0000 7f3892a09640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f388c1e85b0 0x7f388c1e8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:26.731 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:26.719+0000 7f3892a09640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f388c1e85b0 0x7f388c1e8a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34564/0 (socket says 172.21.15.16:34564) 2023-05-22T15:46:26.731 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:26.723+0000 7f3892a09640 1 -- 172.21.15.16:0/2233710620 learned_addr learned my addr 172.21.15.16:0/2233710620 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:46:26.731 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:26.723+0000 7f3892208640 1 --2- 172.21.15.16:0/2233710620 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f38780047d0 0x7f388c0b5450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:26.731 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:26.723+0000 7f3891a07640 1 --2- 172.21.15.16:0/2233710620 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f388c0c5b40 0x7f388c0b5990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:26.731 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:26.723+0000 7f3892a09640 1 -- 172.21.15.16:0/2233710620 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f388c0c5b40 msgr2=0x7f388c0b5990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:26.732 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:26.723+0000 7f3892a09640 1 --2- 172.21.15.16:0/2233710620 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f388c0c5b40 0x7f388c0b5990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:26.732 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:26.723+0000 7f3892a09640 1 -- 172.21.15.16:0/2233710620 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f38780047d0 msgr2=0x7f388c0b5450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:26.732 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:26.723+0000 7f3892a09640 1 --2- 172.21.15.16:0/2233710620 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f38780047d0 0x7f388c0b5450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:26.732 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:26.723+0000 7f3892a09640 1 -- 172.21.15.16:0/2233710620 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f388c0d31c0 con 0x7f388c1e85b0 2023-05-22T15:46:26.732 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:26.723+0000 7f3891a07640 1 --2- 172.21.15.16:0/2233710620 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f388c0c5b40 0x7f388c0b5990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:46:26.732 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:26.723+0000 7f3892a09640 1 --2- 172.21.15.16:0/2233710620 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f388c1e85b0 0x7f388c1e8a20 secure :-1 s=READY pgs=1836 cs=0 l=1 rev1=1 crypto rx=0x7f388c0b3ee0 tx=0x7f3884009970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:26.732 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:26.723+0000 7f3882ffd640 1 -- 172.21.15.16:0/2233710620 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3884014600 con 0x7f388c1e85b0 2023-05-22T15:46:26.733 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:26.723+0000 7f3882ffd640 1 -- 172.21.15.16:0/2233710620 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3884014760 con 0x7f388c1e85b0 2023-05-22T15:46:26.733 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:26.723+0000 7f389446c640 1 -- 172.21.15.16:0/2233710620 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f388c0cd210 con 0x7f388c1e85b0 2023-05-22T15:46:26.733 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:26.723+0000 7f3882ffd640 1 -- 172.21.15.16:0/2233710620 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3884014b10 con 0x7f388c1e85b0 2023-05-22T15:46:26.733 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:26.723+0000 7f389446c640 1 -- 172.21.15.16:0/2233710620 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f388c15ffa0 con 0x7f388c1e85b0 2023-05-22T15:46:26.733 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:26.723+0000 7f389446c640 1 -- 172.21.15.16:0/2233710620 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f388c04dbe0 con 0x7f388c1e85b0 2023-05-22T15:46:26.734 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:26.723+0000 7f3882ffd640 1 -- 172.21.15.16:0/2233710620 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 60) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f388401d3b0 con 0x7f388c1e85b0 2023-05-22T15:46:26.734 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:26.723+0000 7f3882ffd640 1 --2- 172.21.15.16:0/2233710620 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f38700654a0 0x7f3870067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:26.734 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:26.723+0000 7f3882ffd640 1 -- 172.21.15.16:0/2233710620 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(147..147 src has 1..147) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f38840824c0 con 0x7f388c1e85b0 2023-05-22T15:46:26.734 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:26.723+0000 7f3892208640 1 --2- 172.21.15.16:0/2233710620 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f38700654a0 0x7f3870067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:26.734 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:26.723+0000 7f3892208640 1 --2- 172.21.15.16:0/2233710620 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f38700654a0 0x7f3870067950 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f3888001cc0 tx=0x7f3888009040 comp rx=0 tx=0).ready entity=mgr.8835 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:26.735 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:26.727+0000 7f3882ffd640 1 -- 172.21.15.16:0/2233710620 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f388404ff00 con 0x7f388c1e85b0 2023-05-22T15:46:26.920 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:26.915+0000 7f389446c640 1 -- 172.21.15.16:0/2233710620 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f388c0c92f0 con 0x7f388c1e85b0 2023-05-22T15:46:26.923 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:26.915+0000 7f3882ffd640 1 -- 172.21.15.16:0/2233710620 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 58 v58) v1 ==== 94+0+4650 (secure 0 0 0) 0x7f3884053520 con 0x7f388c1e85b0 2023-05-22T15:46:26.924 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:46:26.924 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":58,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7753,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6834/3192074681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":3192074681},{"type":"v1","addr":"172.21.15.62:6834","nonce":3192074681}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":53},{"gid":7756,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/1344243469","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6833","nonce":1344243469},{"type":"v1","addr":"172.21.15.62:6835","nonce":1344243469}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":53},{"gid":9069,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/3636102434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":3636102434},{"type":"v1","addr":"172.21.15.16:6835","nonce":3636102434}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":53}],"filesystems":[{"mdsmap":{"epoch":58,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-05-22T15:46:23.443662+0000","modified":"2023-05-22T15:46:26.557591+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9075},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9075":{"gid":9075,"name":"a","rank":0,"incarnation":54,"state":"up:active","state_seq":3,"addr":"172.21.15.16:6837/2858899325","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":2858899325},{"type":"v1","addr":"172.21.15.16:6837","nonce":2858899325}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2023-05-22T15:46:26.925 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 58 2023-05-22T15:46:26.926 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:26.919+0000 7f389446c640 1 -- 172.21.15.16:0/2233710620 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f38700654a0 msgr2=0x7f3870067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:26.926 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:26.919+0000 7f389446c640 1 --2- 172.21.15.16:0/2233710620 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f38700654a0 0x7f3870067950 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f3888001cc0 tx=0x7f3888009040 comp rx=0 tx=0).stop 2023-05-22T15:46:26.927 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:26.919+0000 7f389446c640 1 -- 172.21.15.16:0/2233710620 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f388c1e85b0 msgr2=0x7f388c1e8a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:26.927 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:26.919+0000 7f389446c640 1 --2- 172.21.15.16:0/2233710620 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f388c1e85b0 0x7f388c1e8a20 secure :-1 s=READY pgs=1836 cs=0 l=1 rev1=1 crypto rx=0x7f388c0b3ee0 tx=0x7f3884009970 comp rx=0 tx=0).stop 2023-05-22T15:46:26.927 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:26.919+0000 7f389446c640 1 -- 172.21.15.16:0/2233710620 shutdown_connections 2023-05-22T15:46:26.928 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:26.919+0000 7f389446c640 1 --2- 172.21.15.16:0/2233710620 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f388c1e85b0 0x7f388c1e8a20 unknown :-1 s=CLOSED pgs=1836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:26.929 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:26.919+0000 7f389446c640 1 --2- 172.21.15.16:0/2233710620 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f38700654a0 0x7f3870067950 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:26.929 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:26.919+0000 7f389446c640 1 --2- 172.21.15.16:0/2233710620 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f388c0c5b40 0x7f388c0b5990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:26.929 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:26.919+0000 7f389446c640 1 --2- 172.21.15.16:0/2233710620 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f38780047d0 0x7f388c0b5450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:26.929 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:26.919+0000 7f389446c640 1 -- 172.21.15.16:0/2233710620 >> 172.21.15.16:0/2233710620 conn(0x7f388c0655c0 msgr2=0x7f388c157a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:46:26.930 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:26.919+0000 7f389446c640 1 -- 172.21.15.16:0/2233710620 shutdown_connections 2023-05-22T15:46:26.930 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:26.919+0000 7f389446c640 1 -- 172.21.15.16:0/2233710620 wait complete. 2023-05-22T15:46:26.941 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-05-22T15:46:27.066 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.059+0000 7f3067c08640 1 Processor -- start 2023-05-22T15:46:27.067 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.059+0000 7f3067c08640 1 -- start start 2023-05-22T15:46:27.070 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.063+0000 7f3067c08640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3060159a40 0x7f30600c8ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:27.070 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.063+0000 7f3067c08640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f306015a0a0 con 0x7f30601531b0 2023-05-22T15:46:27.071 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.063+0000 7f3067c08640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3060153580 con 0x7f30600b0250 2023-05-22T15:46:27.071 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.063+0000 7f3067c08640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3060159e90 con 0x7f3060159a40 2023-05-22T15:46:27.071 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.063+0000 7f30659a4640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f30601531b0 0x7f30600afb40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:38860/0 (socket says 172.21.15.16:38860) 2023-05-22T15:46:27.071 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.063+0000 7f30659a4640 1 -- 172.21.15.16:0/4190167472 learned_addr learned my addr 172.21.15.16:0/4190167472 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:46:27.072 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.063+0000 7f30661a5640 1 --2- 172.21.15.16:0/4190167472 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3060159a40 0x7f30600c8ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:27.072 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.063+0000 7f30649a2640 1 -- 172.21.15.16:0/4190167472 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4276567214 0 0) 0x7f306015a0a0 con 0x7f30601531b0 2023-05-22T15:46:27.072 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.063+0000 7f30649a2640 1 -- 172.21.15.16:0/4190167472 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3048003530 con 0x7f30601531b0 2023-05-22T15:46:27.073 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.063+0000 7f30649a2640 1 -- 172.21.15.16:0/4190167472 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1105112974 0 0) 0x7f3048003530 con 0x7f30601531b0 2023-05-22T15:46:27.073 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.063+0000 7f30649a2640 1 -- 172.21.15.16:0/4190167472 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3060159a40 msgr2=0x7f30600c8ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:46:27.073 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.063+0000 7f30649a2640 1 --2- 172.21.15.16:0/4190167472 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3060159a40 0x7f30600c8ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:27.073 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.063+0000 7f30649a2640 1 -- 172.21.15.16:0/4190167472 >> v1:172.21.15.62:6789/0 conn(0x7f30600b0250 legacy=0x7f3060159330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:27.074 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.063+0000 7f30649a2640 1 -- 172.21.15.16:0/4190167472 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f306015d5e0 con 0x7f30601531b0 2023-05-22T15:46:27.074 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.063+0000 7f30649a2640 1 -- 172.21.15.16:0/4190167472 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7f3050001680 con 0x7f30601531b0 2023-05-22T15:46:27.074 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.063+0000 7f30649a2640 1 -- 172.21.15.16:0/4190167472 >> v1:172.21.15.16:6789/0 conn(0x7f30601531b0 legacy=0x7f30600afb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:27.074 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.063+0000 7f30649a2640 1 --2- 172.21.15.16:0/4190167472 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3060159a40 0x7f30480039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:27.075 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.063+0000 7f30649a2640 1 --2- 172.21.15.16:0/4190167472 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3048003f30 0x7f30480047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:27.075 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.063+0000 7f30649a2640 1 --2- 172.21.15.16:0/4190167472 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3048004d10 0x7f3048005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:27.075 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.063+0000 7f30649a2640 1 -- 172.21.15.16:0/4190167472 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3060159e90 con 0x7f3048003f30 2023-05-22T15:46:27.075 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.063+0000 7f30649a2640 1 -- 172.21.15.16:0/4190167472 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f304800ba70 con 0x7f3048004d10 2023-05-22T15:46:27.075 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.063+0000 7f30649a2640 1 -- 172.21.15.16:0/4190167472 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f304800bc30 con 0x7f3060159a40 2023-05-22T15:46:27.075 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.063+0000 7f30651a3640 1 --2- 172.21.15.16:0/4190167472 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3048003f30 0x7f30480047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:27.076 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.063+0000 7f30659a4640 1 --2- 172.21.15.16:0/4190167472 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3060159a40 0x7f30480039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:27.076 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.063+0000 7f30661a5640 1 --2- 172.21.15.16:0/4190167472 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3048004d10 0x7f3048005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:27.076 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.063+0000 7f30651a3640 1 -- 172.21.15.16:0/4190167472 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3060159a40 msgr2=0x7f30480039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:46:27.076 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.063+0000 7f30651a3640 1 --2- 172.21.15.16:0/4190167472 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3060159a40 0x7f30480039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:27.076 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.063+0000 7f30651a3640 1 -- 172.21.15.16:0/4190167472 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3048004d10 msgr2=0x7f3048005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:46:27.076 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.063+0000 7f30651a3640 1 --2- 172.21.15.16:0/4190167472 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3048004d10 0x7f3048005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:27.077 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.063+0000 7f30651a3640 1 -- 172.21.15.16:0/4190167472 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3048003530 con 0x7f3048003f30 2023-05-22T15:46:27.082 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.067+0000 7f30651a3640 1 --2- 172.21.15.16:0/4190167472 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3048003f30 0x7f30480047d0 secure :-1 s=READY pgs=1838 cs=0 l=1 rev1=1 crypto rx=0x7f306015c5e0 tx=0x7f305401c6e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e0c0c7459dda4d6 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:27.082 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.067+0000 7f30649a2640 1 -- 172.21.15.16:0/4190167472 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3054003b80 con 0x7f3048003f30 2023-05-22T15:46:27.082 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.067+0000 7f30649a2640 1 -- 172.21.15.16:0/4190167472 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3054003ce0 con 0x7f3048003f30 2023-05-22T15:46:27.083 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.067+0000 7f30649a2640 1 -- 172.21.15.16:0/4190167472 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3054005310 con 0x7f3048003f30 2023-05-22T15:46:27.083 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.067+0000 7f3067c08640 1 -- 172.21.15.16:0/4190167472 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3048003f30 msgr2=0x7f30480047d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:27.083 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.067+0000 7f3067c08640 1 --2- 172.21.15.16:0/4190167472 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3048003f30 0x7f30480047d0 secure :-1 s=READY pgs=1838 cs=0 l=1 rev1=1 crypto rx=0x7f306015c5e0 tx=0x7f305401c6e0 comp rx=0 tx=0).stop 2023-05-22T15:46:27.084 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.067+0000 7f30661a5640 1 -- 172.21.15.16:0/4190167472 reap_dead start 2023-05-22T15:46:27.084 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.067+0000 7f3067c08640 1 -- 172.21.15.16:0/4190167472 shutdown_connections 2023-05-22T15:46:27.084 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.067+0000 7f3067c08640 1 -- 172.21.15.16:0/4190167472 >> 172.21.15.16:0/4190167472 conn(0x7f3060069720 msgr2=0x7f3060069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:46:27.084 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.067+0000 7f3067c08640 1 -- 172.21.15.16:0/4190167472 shutdown_connections 2023-05-22T15:46:27.084 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.067+0000 7f3067c08640 1 -- 172.21.15.16:0/4190167472 wait complete. 2023-05-22T15:46:27.084 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.067+0000 7f3067c08640 1 Processor -- start 2023-05-22T15:46:27.085 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.067+0000 7f3067c08640 1 -- start start 2023-05-22T15:46:27.085 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.067+0000 7f3067c08640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3048003f30 0x7f306014abe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:27.085 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.067+0000 7f3067c08640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f30600c6430 0x7f30600c6880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:27.085 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.067+0000 7f3067c08640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f306014b9d0 0x7f306014be40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:27.085 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.067+0000 7f3067c08640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f306014c380 con 0x7f3048003f30 2023-05-22T15:46:27.085 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.067+0000 7f3067c08640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30600c6e80 con 0x7f30600c6430 2023-05-22T15:46:27.085 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.067+0000 7f3067c08640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30600c7160 con 0x7f306014b9d0 2023-05-22T15:46:27.086 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.067+0000 7f30659a4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3048003f30 0x7f306014abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:27.086 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.067+0000 7f30659a4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3048003f30 0x7f306014abe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34588/0 (socket says 172.21.15.16:34588) 2023-05-22T15:46:27.086 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.067+0000 7f30659a4640 1 -- 172.21.15.16:0/998347760 learned_addr learned my addr 172.21.15.16:0/998347760 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:46:27.086 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.067+0000 7f30651a3640 1 --2- 172.21.15.16:0/998347760 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f30600c6430 0x7f30600c6880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:27.086 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.071+0000 7f30661a5640 1 --2- 172.21.15.16:0/998347760 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f306014b9d0 0x7f306014be40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:27.086 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.071+0000 7f30659a4640 1 -- 172.21.15.16:0/998347760 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f306014b9d0 msgr2=0x7f306014be40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:27.086 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.071+0000 7f30659a4640 1 --2- 172.21.15.16:0/998347760 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f306014b9d0 0x7f306014be40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:27.086 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.071+0000 7f30659a4640 1 -- 172.21.15.16:0/998347760 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f30600c6430 msgr2=0x7f30600c6880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:27.087 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.071+0000 7f30659a4640 1 --2- 172.21.15.16:0/998347760 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f30600c6430 0x7f30600c6880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:27.087 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.071+0000 7f30659a4640 1 -- 172.21.15.16:0/998347760 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f306015cb00 con 0x7f3048003f30 2023-05-22T15:46:27.087 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.071+0000 7f30659a4640 1 --2- 172.21.15.16:0/998347760 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3048003f30 0x7f306014abe0 secure :-1 s=READY pgs=1839 cs=0 l=1 rev1=1 crypto rx=0x7f30500058b0 tx=0x7f3050011d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:27.087 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.071+0000 7f305e7fc640 1 -- 172.21.15.16:0/998347760 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f305000fb60 con 0x7f3048003f30 2023-05-22T15:46:27.087 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.071+0000 7f305e7fc640 1 -- 172.21.15.16:0/998347760 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f305000fcc0 con 0x7f3048003f30 2023-05-22T15:46:27.087 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.071+0000 7f305e7fc640 1 -- 172.21.15.16:0/998347760 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3050015630 con 0x7f3048003f30 2023-05-22T15:46:27.087 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.071+0000 7f3067c08640 1 -- 172.21.15.16:0/998347760 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f306015c820 con 0x7f3048003f30 2023-05-22T15:46:27.088 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.071+0000 7f3067c08640 1 -- 172.21.15.16:0/998347760 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f306015cde0 con 0x7f3048003f30 2023-05-22T15:46:27.088 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.071+0000 7f3067c08640 1 -- 172.21.15.16:0/998347760 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30600afac0 con 0x7f3048003f30 2023-05-22T15:46:27.088 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.071+0000 7f305e7fc640 1 -- 172.21.15.16:0/998347760 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 60) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f3050015790 con 0x7f3048003f30 2023-05-22T15:46:27.088 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.071+0000 7f305e7fc640 1 --2- 172.21.15.16:0/998347760 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f304c0654a0 0x7f304c067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:27.088 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.071+0000 7f305e7fc640 1 -- 172.21.15.16:0/998347760 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(147..147 src has 1..147) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f3050085f30 con 0x7f3048003f30 2023-05-22T15:46:27.088 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.071+0000 7f30651a3640 1 --2- 172.21.15.16:0/998347760 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f304c0654a0 0x7f304c067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:27.088 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.071+0000 7f30651a3640 1 --2- 172.21.15.16:0/998347760 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f304c0654a0 0x7f304c067950 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f30540ac000 tx=0x7f3054001520 comp rx=0 tx=0).ready entity=mgr.8835 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:27.089 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.075+0000 7f305e7fc640 1 -- 172.21.15.16:0/998347760 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f3050057370 con 0x7f3048003f30 2023-05-22T15:46:27.236 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.231+0000 7f3067c08640 1 -- 172.21.15.16:0/998347760 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f30600c7d70 con 0x7f3048003f30 2023-05-22T15:46:27.238 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.231+0000 7f305e7fc640 1 -- 172.21.15.16:0/998347760 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v147) v1 ==== 74+0+18094 (secure 0 0 0) 0x7f3050056f60 con 0x7f3048003f30 2023-05-22T15:46:27.238 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:46:27.238 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":147,"fsid":"b2719834-7847-4d98-90a6-f65d57f85c49","created":"2023-05-22T15:26:25.259314+0000","modified":"2023-05-22T15:46:24.447924+0000","last_up_change":"2023-05-22T15:26:36.448524+0000","last_in_change":"2023-05-22T15:26:27.675668+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-05-22T15:26:38.019781+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-05-22T15:26:43.570514+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2023-05-22T15:46:21.084615+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"146","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_data","create_time":"2023-05-22T15:46:22.440577+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"147","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9e7ff854-b25f-4cbb-a5c2-1799d9c53d2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":37021},{"type":"v1","addr":"172.21.15.16:6825","nonce":37021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":37021},{"type":"v1","addr":"172.21.15.16:6827","nonce":37021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":37021},{"type":"v1","addr":"172.21.15.16:6831","nonce":37021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":37021},{"type":"v1","addr":"172.21.15.16:6829","nonce":37021}]},"public_addr":"172.21.15.16:6825/37021","cluster_addr":"172.21.15.16:6827/37021","heartbeat_back_addr":"172.21.15.16:6831/37021","heartbeat_front_addr":"172.21.15.16:6829/37021","state":["exists","up"]},{"osd":1,"uuid":"4702fbc9-a927-4565-aca5-41dc69cc8264","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":37008},{"type":"v1","addr":"172.21.15.16:6801","nonce":37008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":37008},{"type":"v1","addr":"172.21.15.16:6803","nonce":37008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":37008},{"type":"v1","addr":"172.21.15.16:6807","nonce":37008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":37008},{"type":"v1","addr":"172.21.15.16:6805","nonce":37008}]},"public_addr":"172.21.15.16:6801/37008","cluster_addr":"172.21.15.16:6803/37008","heartbeat_back_addr":"172.21.15.16:6807/37008","heartbeat_front_addr":"172.21.15.16:6805/37008","state":["exists","up"]},{"osd":2,"uuid":"1e7bba46-4eba-47ef-bb81-5e2d371638ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":37009},{"type":"v1","addr":"172.21.15.16:6809","nonce":37009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":37009},{"type":"v1","addr":"172.21.15.16:6811","nonce":37009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":37009},{"type":"v1","addr":"172.21.15.16:6815","nonce":37009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":37009},{"type":"v1","addr":"172.21.15.16:6813","nonce":37009}]},"public_addr":"172.21.15.16:6809/37009","cluster_addr":"172.21.15.16:6811/37009","heartbeat_back_addr":"172.21.15.16:6815/37009","heartbeat_front_addr":"172.21.15.16:6813/37009","state":["exists","up"]},{"osd":3,"uuid":"383cacc0-5a88-4904-990e-498ca59ef48a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":37020},{"type":"v1","addr":"172.21.15.16:6817","nonce":37020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":37020},{"type":"v1","addr":"172.21.15.16:6819","nonce":37020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":37020},{"type":"v1","addr":"172.21.15.16:6823","nonce":37020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":37020},{"type":"v1","addr":"172.21.15.16:6821","nonce":37020}]},"public_addr":"172.21.15.16:6817/37020","cluster_addr":"172.21.15.16:6819/37020","heartbeat_back_addr":"172.21.15.16:6823/37020","heartbeat_front_addr":"172.21.15.16:6821/37020","state":["exists","up"]},{"osd":4,"uuid":"970500ef-95b7-42c1-b080-eed4fae7013d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":37353},{"type":"v1","addr":"172.21.15.62:6817","nonce":37353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6819","nonce":37353},{"type":"v1","addr":"172.21.15.62:6822","nonce":37353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37353},{"type":"v1","addr":"172.21.15.62:6831","nonce":37353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37353},{"type":"v1","addr":"172.21.15.62:6826","nonce":37353}]},"public_addr":"172.21.15.62:6817/37353","cluster_addr":"172.21.15.62:6822/37353","heartbeat_back_addr":"172.21.15.62:6831/37353","heartbeat_front_addr":"172.21.15.62:6826/37353","state":["exists","up"]},{"osd":5,"uuid":"022d67db-d4d8-445b-b1b9-8b446a00f5c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":37352},{"type":"v1","addr":"172.21.15.62:6801","nonce":37352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":37352},{"type":"v1","addr":"172.21.15.62:6803","nonce":37352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":37352},{"type":"v1","addr":"172.21.15.62:6807","nonce":37352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":37352},{"type":"v1","addr":"172.21.15.62:6805","nonce":37352}]},"public_addr":"172.21.15.62:6801/37352","cluster_addr":"172.21.15.62:6803/37352","heartbeat_back_addr":"172.21.15.62:6807/37352","heartbeat_front_addr":"172.21.15.62:6805/37352","state":["exists","up"]},{"osd":6,"uuid":"42bd372b-6688-4a1b-8407-e16e8c266e33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":37354},{"type":"v1","addr":"172.21.15.62:6820","nonce":37354}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":37354},{"type":"v1","addr":"172.21.15.62:6823","nonce":37354}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6829","nonce":37354},{"type":"v1","addr":"172.21.15.62:6830","nonce":37354}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6825","nonce":37354},{"type":"v1","addr":"172.21.15.62:6827","nonce":37354}]},"public_addr":"172.21.15.62:6820/37354","cluster_addr":"172.21.15.62:6823/37354","heartbeat_back_addr":"172.21.15.62:6830/37354","heartbeat_front_addr":"172.21.15.62:6827/37354","state":["exists","up"]},{"osd":7,"uuid":"ac70b2a3-f41b-4bf2-b227-7f2cbb55426d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37355},{"type":"v1","addr":"172.21.15.62:6809","nonce":37355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37355},{"type":"v1","addr":"172.21.15.62:6811","nonce":37355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":37355},{"type":"v1","addr":"172.21.15.62:6815","nonce":37355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":37355},{"type":"v1","addr":"172.21.15.62:6813","nonce":37355}]},"public_addr":"172.21.15.62:6809/37355","cluster_addr":"172.21.15.62:6811/37355","heartbeat_back_addr":"172.21.15.62:6815/37355","heartbeat_front_addr":"172.21.15.62:6813/37355","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.368896+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.107273+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.276888+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.287502+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.124791+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.251937+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.255679+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.232797+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":17,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-05-22T15:46:27.239 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.235+0000 7f3067c08640 1 -- 172.21.15.16:0/998347760 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f304c0654a0 msgr2=0x7f304c067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:27.239 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.235+0000 7f3067c08640 1 --2- 172.21.15.16:0/998347760 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f304c0654a0 0x7f304c067950 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f30540ac000 tx=0x7f3054001520 comp rx=0 tx=0).stop 2023-05-22T15:46:27.240 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.235+0000 7f3067c08640 1 -- 172.21.15.16:0/998347760 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3048003f30 msgr2=0x7f306014abe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:27.240 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.235+0000 7f3067c08640 1 --2- 172.21.15.16:0/998347760 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3048003f30 0x7f306014abe0 secure :-1 s=READY pgs=1839 cs=0 l=1 rev1=1 crypto rx=0x7f30500058b0 tx=0x7f3050011d10 comp rx=0 tx=0).stop 2023-05-22T15:46:27.240 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.235+0000 7f3067c08640 1 -- 172.21.15.16:0/998347760 shutdown_connections 2023-05-22T15:46:27.240 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.235+0000 7f3067c08640 1 --2- 172.21.15.16:0/998347760 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f304c0654a0 0x7f304c067950 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:27.240 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.235+0000 7f3067c08640 1 --2- 172.21.15.16:0/998347760 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f306014b9d0 0x7f306014be40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:27.240 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.235+0000 7f3067c08640 1 --2- 172.21.15.16:0/998347760 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f30600c6430 0x7f30600c6880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:27.240 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.235+0000 7f3067c08640 1 --2- 172.21.15.16:0/998347760 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3048003f30 0x7f306014abe0 unknown :-1 s=CLOSED pgs=1839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:27.241 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.235+0000 7f3067c08640 1 -- 172.21.15.16:0/998347760 >> 172.21.15.16:0/998347760 conn(0x7f3060069720 msgr2=0x7f30600ce000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:46:27.241 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.235+0000 7f3067c08640 1 -- 172.21.15.16:0/998347760 shutdown_connections 2023-05-22T15:46:27.241 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.235+0000 7f3067c08640 1 -- 172.21.15.16:0/998347760 wait complete. 2023-05-22T15:46:27.257 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-05-22T15:46:27.257 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-05-22T15:46:27.385 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.375+0000 7eff2bd61640 1 Processor -- start 2023-05-22T15:46:27.385 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.375+0000 7eff2bd61640 1 -- start start 2023-05-22T15:46:27.386 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.379+0000 7eff2bd61640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7eff240ca4e0 0x7eff2415e460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:27.386 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.379+0000 7eff2bd61640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7eff240c9b40 0x7eff240afb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:27.387 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.379+0000 7eff2bd61640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7eff24070df0 0x7eff24071210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:27.387 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.379+0000 7eff2bd61640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff240b0650 con 0x7eff240c9b40 2023-05-22T15:46:27.387 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.379+0000 7eff2bd61640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff240b0370 con 0x7eff240ca4e0 2023-05-22T15:46:27.388 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.379+0000 7eff2bd61640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff2415d980 con 0x7eff24070df0 2023-05-22T15:46:27.388 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.379+0000 7eff292fc640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7eff240c9b40 0x7eff240afb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:27.388 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.379+0000 7eff292fc640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7eff240c9b40 0x7eff240afb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34590/0 (socket says 172.21.15.16:34590) 2023-05-22T15:46:27.389 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.379+0000 7eff292fc640 1 -- 172.21.15.16:0/1239356799 learned_addr learned my addr 172.21.15.16:0/1239356799 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:46:27.389 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.379+0000 7eff2a2fe640 1 --2- 172.21.15.16:0/1239356799 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7eff24070df0 0x7eff24071210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:27.389 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.379+0000 7eff29afd640 1 --2- 172.21.15.16:0/1239356799 >> v2:172.21.15.62:3300/0 conn(0x7eff240ca4e0 0x7eff2415e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:27.389 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.379+0000 7eff292fc640 1 -- 172.21.15.16:0/1239356799 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7eff24070df0 msgr2=0x7eff24071210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:46:27.389 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.379+0000 7eff292fc640 1 --2- 172.21.15.16:0/1239356799 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7eff24070df0 0x7eff24071210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:27.389 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.379+0000 7eff292fc640 1 -- 172.21.15.16:0/1239356799 >> v2:172.21.15.62:3300/0 conn(0x7eff240ca4e0 msgr2=0x7eff2415e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:46:27.389 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.379+0000 7eff292fc640 1 --2- 172.21.15.16:0/1239356799 >> v2:172.21.15.62:3300/0 conn(0x7eff240ca4e0 0x7eff2415e460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:27.389 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.379+0000 7eff292fc640 1 -- 172.21.15.16:0/1239356799 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff2415dc60 con 0x7eff240c9b40 2023-05-22T15:46:27.393 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.379+0000 7eff292fc640 1 --2- 172.21.15.16:0/1239356799 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7eff240c9b40 0x7eff240afb40 secure :-1 s=READY pgs=1840 cs=0 l=1 rev1=1 crypto rx=0x7eff240cad40 tx=0x7eff1401c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d844f325cc5e2950 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:27.393 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.379+0000 7eff28afb640 1 -- 172.21.15.16:0/1239356799 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff140ab670 con 0x7eff240c9b40 2023-05-22T15:46:27.393 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.379+0000 7eff28afb640 1 -- 172.21.15.16:0/1239356799 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff140ab7d0 con 0x7eff240c9b40 2023-05-22T15:46:27.393 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.379+0000 7eff28afb640 1 -- 172.21.15.16:0/1239356799 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff140abb60 con 0x7eff240c9b40 2023-05-22T15:46:27.393 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.383+0000 7eff2bd61640 1 -- 172.21.15.16:0/1239356799 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7eff240c9b40 msgr2=0x7eff240afb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:27.394 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.383+0000 7eff2bd61640 1 --2- 172.21.15.16:0/1239356799 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7eff240c9b40 0x7eff240afb40 secure :-1 s=READY pgs=1840 cs=0 l=1 rev1=1 crypto rx=0x7eff240cad40 tx=0x7eff1401c4d0 comp rx=0 tx=0).stop 2023-05-22T15:46:27.394 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.383+0000 7eff2bd61640 1 -- 172.21.15.16:0/1239356799 shutdown_connections 2023-05-22T15:46:27.394 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.383+0000 7eff2bd61640 1 --2- 172.21.15.16:0/1239356799 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7eff240c9b40 0x7eff240afb40 unknown :-1 s=CLOSED pgs=1840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:27.394 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.383+0000 7eff2bd61640 1 --2- 172.21.15.16:0/1239356799 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7eff24070df0 0x7eff24071210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:27.394 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.383+0000 7eff2bd61640 1 --2- 172.21.15.16:0/1239356799 >> v2:172.21.15.62:3300/0 conn(0x7eff240ca4e0 0x7eff2415e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:27.394 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.383+0000 7eff2bd61640 1 -- 172.21.15.16:0/1239356799 >> 172.21.15.16:0/1239356799 conn(0x7eff24069720 msgr2=0x7eff24069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:46:27.394 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.383+0000 7eff2bd61640 1 -- 172.21.15.16:0/1239356799 shutdown_connections 2023-05-22T15:46:27.395 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.383+0000 7eff2bd61640 1 -- 172.21.15.16:0/1239356799 wait complete. 2023-05-22T15:46:27.395 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.383+0000 7eff2bd61640 1 Processor -- start 2023-05-22T15:46:27.395 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.383+0000 7eff2bd61640 1 -- start start 2023-05-22T15:46:27.395 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.383+0000 7eff2bd61640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7eff24070df0 0x7eff241f2390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:27.395 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.383+0000 7eff2bd61640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7eff240c9b40 0x7eff24153020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:27.395 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.383+0000 7eff2bd61640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7eff240ca4e0 0x7eff240c56e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:27.395 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.383+0000 7eff2bd61640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff240c2670 con 0x7eff240c9b40 2023-05-22T15:46:27.395 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.383+0000 7eff2bd61640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff240c2950 con 0x7eff24070df0 2023-05-22T15:46:27.396 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.383+0000 7eff2bd61640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff240c2c30 con 0x7eff240ca4e0 2023-05-22T15:46:27.396 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.383+0000 7eff292fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7eff240c9b40 0x7eff24153020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:27.396 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.383+0000 7eff292fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7eff240c9b40 0x7eff24153020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34592/0 (socket says 172.21.15.16:34592) 2023-05-22T15:46:27.396 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.383+0000 7eff292fc640 1 -- 172.21.15.16:0/1861754105 learned_addr learned my addr 172.21.15.16:0/1861754105 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:46:27.396 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.383+0000 7eff29afd640 1 --2- 172.21.15.16:0/1861754105 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7eff24070df0 0x7eff241f2390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:27.397 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.383+0000 7eff2a2fe640 1 --2- 172.21.15.16:0/1861754105 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7eff240ca4e0 0x7eff240c56e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:27.397 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.383+0000 7eff292fc640 1 -- 172.21.15.16:0/1861754105 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7eff240ca4e0 msgr2=0x7eff240c56e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:27.397 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.383+0000 7eff292fc640 1 --2- 172.21.15.16:0/1861754105 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7eff240ca4e0 0x7eff240c56e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:27.397 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.383+0000 7eff292fc640 1 -- 172.21.15.16:0/1861754105 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7eff24070df0 msgr2=0x7eff241f2390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:27.397 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.383+0000 7eff292fc640 1 --2- 172.21.15.16:0/1861754105 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7eff24070df0 0x7eff241f2390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:27.397 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.383+0000 7eff292fc640 1 -- 172.21.15.16:0/1861754105 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff140094d0 con 0x7eff240c9b40 2023-05-22T15:46:27.398 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.383+0000 7eff292fc640 1 --2- 172.21.15.16:0/1861754105 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7eff240c9b40 0x7eff24153020 secure :-1 s=READY pgs=1841 cs=0 l=1 rev1=1 crypto rx=0x7eff1401ab80 tx=0x7eff1401ad10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:27.398 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.383+0000 7eff1a7fc640 1 -- 172.21.15.16:0/1861754105 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff140b4800 con 0x7eff240c9b40 2023-05-22T15:46:27.398 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.383+0000 7eff1a7fc640 1 -- 172.21.15.16:0/1861754105 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff140b4980 con 0x7eff240c9b40 2023-05-22T15:46:27.398 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.383+0000 7eff2bd61640 1 -- 172.21.15.16:0/1861754105 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff240ca8b0 con 0x7eff240c9b40 2023-05-22T15:46:27.398 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.383+0000 7eff1a7fc640 1 -- 172.21.15.16:0/1861754105 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff140b4d30 con 0x7eff240c9b40 2023-05-22T15:46:27.398 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.383+0000 7eff2bd61640 1 -- 172.21.15.16:0/1861754105 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff24071750 con 0x7eff240c9b40 2023-05-22T15:46:27.398 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.387+0000 7eff2bd61640 1 -- 172.21.15.16:0/1861754105 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff240c8e80 con 0x7eff240c9b40 2023-05-22T15:46:27.399 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.387+0000 7eff1a7fc640 1 -- 172.21.15.16:0/1861754105 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 60) v1 ==== 84332+0+0 (secure 0 0 0) 0x7eff140059a0 con 0x7eff240c9b40 2023-05-22T15:46:27.399 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.387+0000 7eff1a7fc640 1 --2- 172.21.15.16:0/1861754105 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7eff080654a0 0x7eff08067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:27.399 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.387+0000 7eff1a7fc640 1 -- 172.21.15.16:0/1861754105 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(147..147 src has 1..147) v4 ==== 6410+0+0 (secure 0 0 0) 0x7eff1411db30 con 0x7eff240c9b40 2023-05-22T15:46:27.399 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.387+0000 7eff29afd640 1 --2- 172.21.15.16:0/1861754105 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7eff080654a0 0x7eff08067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:27.399 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.387+0000 7eff29afd640 1 --2- 172.21.15.16:0/1861754105 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7eff080654a0 0x7eff08067950 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7eff2000b930 tx=0x7eff20008040 comp rx=0 tx=0).ready entity=mgr.8835 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:27.400 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.391+0000 7eff1a7fc640 1 -- 172.21.15.16:0/1861754105 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7eff140eb3b0 con 0x7eff240c9b40 2023-05-22T15:46:27.549 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.543+0000 7eff2bd61640 1 -- 172.21.15.16:0/1861754105 --> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7eff240c2ac0 con 0x7eff080654a0 2023-05-22T15:46:27.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.555+0000 7eff1a7fc640 1 -- 172.21.15.16:0/1861754105 <== mgr.8835 v2:172.21.15.16:6832/36921 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+307001 (secure 0 0 0) 0x7eff240c2ac0 con 0x7eff080654a0 2023-05-22T15:46:27.563 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:46:27.567 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2023-05-22T15:46:27.568 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.563+0000 7eff2bd61640 1 -- 172.21.15.16:0/1861754105 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7eff080654a0 msgr2=0x7eff08067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:27.569 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.563+0000 7eff2bd61640 1 --2- 172.21.15.16:0/1861754105 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7eff080654a0 0x7eff08067950 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7eff2000b930 tx=0x7eff20008040 comp rx=0 tx=0).stop 2023-05-22T15:46:27.569 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.563+0000 7eff2bd61640 1 -- 172.21.15.16:0/1861754105 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7eff240c9b40 msgr2=0x7eff24153020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:27.569 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.563+0000 7eff2bd61640 1 --2- 172.21.15.16:0/1861754105 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7eff240c9b40 0x7eff24153020 secure :-1 s=READY pgs=1841 cs=0 l=1 rev1=1 crypto rx=0x7eff1401ab80 tx=0x7eff1401ad10 comp rx=0 tx=0).stop 2023-05-22T15:46:27.569 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.563+0000 7eff2bd61640 1 -- 172.21.15.16:0/1861754105 shutdown_connections 2023-05-22T15:46:27.569 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.563+0000 7eff2bd61640 1 --2- 172.21.15.16:0/1861754105 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7eff080654a0 0x7eff08067950 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:27.569 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.563+0000 7eff2bd61640 1 --2- 172.21.15.16:0/1861754105 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7eff240ca4e0 0x7eff240c56e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:27.570 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.563+0000 7eff2bd61640 1 --2- 172.21.15.16:0/1861754105 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7eff240c9b40 0x7eff24153020 secure :-1 s=CLOSED pgs=1841 cs=0 l=1 rev1=1 crypto rx=0x7eff1401ab80 tx=0x7eff1401ad10 comp rx=0 tx=0).stop 2023-05-22T15:46:27.570 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.563+0000 7eff2bd61640 1 --2- 172.21.15.16:0/1861754105 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7eff24070df0 0x7eff241f2390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:27.570 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.563+0000 7eff2bd61640 1 -- 172.21.15.16:0/1861754105 >> 172.21.15.16:0/1861754105 conn(0x7eff24069720 msgr2=0x7eff240b27e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:46:27.570 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.563+0000 7eff2bd61640 1 -- 172.21.15.16:0/1861754105 shutdown_connections 2023-05-22T15:46:27.570 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.563+0000 7eff2bd61640 1 -- 172.21.15.16:0/1861754105 wait complete. 2023-05-22T15:46:27.586 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":26,"stamp":"2023-05-22T15:46:27.369508+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1268,"num_read_kb":2572,"num_write":399,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":249,"ondisk_log_size":249,"up":210,"acting":210,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":33,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":224768,"kb_used_data":15168,"kb_used_omap":0,"kb_used_meta":209536,"kb_avail":754749952,"statfs":{"total":773094113280,"available":772863950848,"internally_reserved":0,"allocated":15532032,"data_stored":8756104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":214564864},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":62,"apply_latency_ms":62,"commit_latency_ns":62000000,"apply_latency_ns":62000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.321201"},"pg_stats":[{"pgid":"19.3f","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.079122+0000","last_change":"2023-05-22T15:46:23.079122+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.079122+0000","last_undegraded":"2023-05-22T15:46:23.079122+0000","last_fullsized":"2023-05-22T15:46:23.079122+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:57:19.852836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.3e","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.076282+0000","last_change":"2023-05-22T15:46:23.076282+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.076282+0000","last_undegraded":"2023-05-22T15:46:23.076282+0000","last_fullsized":"2023-05-22T15:46:23.076282+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T03:20:39.433281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.3d","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.072443+0000","last_change":"2023-05-22T15:46:23.072443+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.072443+0000","last_undegraded":"2023-05-22T15:46:23.072443+0000","last_fullsized":"2023-05-22T15:46:23.072443+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:46:28.227459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.3c","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.077931+0000","last_change":"2023-05-22T15:46:23.077931+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.077931+0000","last_undegraded":"2023-05-22T15:46:23.077931+0000","last_fullsized":"2023-05-22T15:46:23.077931+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:42:45.119566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.3b","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.090321+0000","last_change":"2023-05-22T15:46:23.090321+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.090321+0000","last_undegraded":"2023-05-22T15:46:23.090321+0000","last_fullsized":"2023-05-22T15:46:23.090321+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:59:24.451915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.3a","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.072441+0000","last_change":"2023-05-22T15:46:23.072441+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.072441+0000","last_undegraded":"2023-05-22T15:46:23.072441+0000","last_fullsized":"2023-05-22T15:46:23.072441+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:25:55.107784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.39","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.076980+0000","last_change":"2023-05-22T15:46:23.076980+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.076980+0000","last_undegraded":"2023-05-22T15:46:23.076980+0000","last_fullsized":"2023-05-22T15:46:23.076980+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:29:20.717884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.38","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.078358+0000","last_change":"2023-05-22T15:46:23.078358+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.078358+0000","last_undegraded":"2023-05-22T15:46:23.078358+0000","last_fullsized":"2023-05-22T15:46:23.078358+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:51:08.858142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.37","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.074307+0000","last_change":"2023-05-22T15:46:23.074307+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.074307+0000","last_undegraded":"2023-05-22T15:46:23.074307+0000","last_fullsized":"2023-05-22T15:46:23.074307+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:28:23.397061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.36","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.075645+0000","last_change":"2023-05-22T15:46:23.075645+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.075645+0000","last_undegraded":"2023-05-22T15:46:23.075645+0000","last_fullsized":"2023-05-22T15:46:23.075645+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:46:28.227459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.35","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.069852+0000","last_change":"2023-05-22T15:46:23.069852+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.069852+0000","last_undegraded":"2023-05-22T15:46:23.069852+0000","last_fullsized":"2023-05-22T15:46:23.069852+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T15:53:20.317234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.34","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.073019+0000","last_change":"2023-05-22T15:46:23.073019+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.073019+0000","last_undegraded":"2023-05-22T15:46:23.073019+0000","last_fullsized":"2023-05-22T15:46:23.073019+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T18:52:29.352539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.33","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.078530+0000","last_change":"2023-05-22T15:46:23.078530+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.078530+0000","last_undegraded":"2023-05-22T15:46:23.078530+0000","last_fullsized":"2023-05-22T15:46:23.078530+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:22:06.668010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.32","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.072337+0000","last_change":"2023-05-22T15:46:23.072337+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.072337+0000","last_undegraded":"2023-05-22T15:46:23.072337+0000","last_fullsized":"2023-05-22T15:46:23.072337+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T18:05:16.988218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.31","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.074359+0000","last_change":"2023-05-22T15:46:23.074359+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.074359+0000","last_undegraded":"2023-05-22T15:46:23.074359+0000","last_fullsized":"2023-05-22T15:46:23.074359+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T18:05:16.988218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.30","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.063646+0000","last_change":"2023-05-22T15:46:23.063646+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.063646+0000","last_undegraded":"2023-05-22T15:46:23.063646+0000","last_fullsized":"2023-05-22T15:46:23.063646+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:30:04.941241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2f","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.075803+0000","last_change":"2023-05-22T15:46:23.075803+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.075803+0000","last_undegraded":"2023-05-22T15:46:23.075803+0000","last_fullsized":"2023-05-22T15:46:23.075803+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:23:42.844820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.2e","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.064448+0000","last_change":"2023-05-22T15:46:23.064448+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.064448+0000","last_undegraded":"2023-05-22T15:46:23.064448+0000","last_fullsized":"2023-05-22T15:46:23.064448+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:23:58.635276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2d","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.068994+0000","last_change":"2023-05-22T15:46:23.068994+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.068994+0000","last_undegraded":"2023-05-22T15:46:23.068994+0000","last_fullsized":"2023-05-22T15:46:23.068994+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:38:41.516579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.2c","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.062047+0000","last_change":"2023-05-22T15:46:23.062047+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.062047+0000","last_undegraded":"2023-05-22T15:46:23.062047+0000","last_fullsized":"2023-05-22T15:46:23.062047+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:52:08.264737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2b","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.093524+0000","last_change":"2023-05-22T15:46:23.093524+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.093524+0000","last_undegraded":"2023-05-22T15:46:23.093524+0000","last_fullsized":"2023-05-22T15:46:23.093524+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:55:41.010860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.2a","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.069379+0000","last_change":"2023-05-22T15:46:23.069379+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.069379+0000","last_undegraded":"2023-05-22T15:46:23.069379+0000","last_fullsized":"2023-05-22T15:46:23.069379+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:36:49.326554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.29","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.081591+0000","last_change":"2023-05-22T15:46:23.081591+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.081591+0000","last_undegraded":"2023-05-22T15:46:23.081591+0000","last_fullsized":"2023-05-22T15:46:23.081591+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:55:51.486171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.28","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.070163+0000","last_change":"2023-05-22T15:46:23.070163+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.070163+0000","last_undegraded":"2023-05-22T15:46:23.070163+0000","last_fullsized":"2023-05-22T15:46:23.070163+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:21:39.187868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.27","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.071496+0000","last_change":"2023-05-22T15:46:23.071496+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.071496+0000","last_undegraded":"2023-05-22T15:46:23.071496+0000","last_fullsized":"2023-05-22T15:46:23.071496+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:37:26.183234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.26","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.063189+0000","last_change":"2023-05-22T15:46:23.063189+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.063189+0000","last_undegraded":"2023-05-22T15:46:23.063189+0000","last_fullsized":"2023-05-22T15:46:23.063189+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:43:11.401376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.25","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.074484+0000","last_change":"2023-05-22T15:46:23.074484+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.074484+0000","last_undegraded":"2023-05-22T15:46:23.074484+0000","last_fullsized":"2023-05-22T15:46:23.074484+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T03:43:40.124163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.24","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.064681+0000","last_change":"2023-05-22T15:46:23.064681+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.064681+0000","last_undegraded":"2023-05-22T15:46:23.064681+0000","last_fullsized":"2023-05-22T15:46:23.064681+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:20:29.148898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.23","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.077284+0000","last_change":"2023-05-22T15:46:23.077284+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.077284+0000","last_undegraded":"2023-05-22T15:46:23.077284+0000","last_fullsized":"2023-05-22T15:46:23.077284+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:34:43.855049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.22","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.073667+0000","last_change":"2023-05-22T15:46:23.073667+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.073667+0000","last_undegraded":"2023-05-22T15:46:23.073667+0000","last_fullsized":"2023-05-22T15:46:23.073667+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T16:10:54.516325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.21","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.086294+0000","last_change":"2023-05-22T15:46:23.086294+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.086294+0000","last_undegraded":"2023-05-22T15:46:23.086294+0000","last_fullsized":"2023-05-22T15:46:23.086294+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:20:12.734869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.20","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.074629+0000","last_change":"2023-05-22T15:46:23.074629+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.074629+0000","last_undegraded":"2023-05-22T15:46:23.074629+0000","last_fullsized":"2023-05-22T15:46:23.074629+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:52:03.083872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.1e","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.061067+0000","last_change":"2023-05-22T15:46:23.061067+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.061067+0000","last_undegraded":"2023-05-22T15:46:23.061067+0000","last_fullsized":"2023-05-22T15:46:23.061067+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:29:20.717884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1f","version":"0'0","reported_seq":9,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.120601+0000","last_change":"2023-05-22T15:46:23.112442+0000","last_active":"2023-05-22T15:46:23.120601+0000","last_peered":"2023-05-22T15:46:23.120601+0000","last_clean":"2023-05-22T15:46:23.120601+0000","last_became_active":"2023-05-22T15:46:23.111561+0000","last_became_peered":"2023-05-22T15:46:23.111561+0000","last_unstale":"2023-05-22T15:46:23.120601+0000","last_undegraded":"2023-05-22T15:46:23.120601+0000","last_fullsized":"2023-05-22T15:46:23.120601+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_clean_scrub_stamp":"2023-05-22T15:46:22.034488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:28:22.384974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.1f","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.071619+0000","last_change":"2023-05-22T15:46:23.071619+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.071619+0000","last_undegraded":"2023-05-22T15:46:23.071619+0000","last_fullsized":"2023-05-22T15:46:23.071619+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:22:11.366110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.1e","version":"0'0","reported_seq":9,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.092179+0000","last_change":"2023-05-22T15:46:23.078407+0000","last_active":"2023-05-22T15:46:23.092179+0000","last_peered":"2023-05-22T15:46:23.092179+0000","last_clean":"2023-05-22T15:46:23.092179+0000","last_became_active":"2023-05-22T15:46:23.077477+0000","last_became_peered":"2023-05-22T15:46:23.077477+0000","last_unstale":"2023-05-22T15:46:23.092179+0000","last_undegraded":"2023-05-22T15:46:23.092179+0000","last_fullsized":"2023-05-22T15:46:23.092179+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_clean_scrub_stamp":"2023-05-22T15:46:22.034488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:20:11.712441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.1c","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.089330+0000","last_change":"2023-05-22T15:46:23.089330+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.089330+0000","last_undegraded":"2023-05-22T15:46:23.089330+0000","last_fullsized":"2023-05-22T15:46:23.089330+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:47:45.260369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1d","version":"0'0","reported_seq":9,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.090669+0000","last_change":"2023-05-22T15:46:23.083700+0000","last_active":"2023-05-22T15:46:23.090669+0000","last_peered":"2023-05-22T15:46:23.090669+0000","last_clean":"2023-05-22T15:46:23.090669+0000","last_became_active":"2023-05-22T15:46:23.083350+0000","last_became_peered":"2023-05-22T15:46:23.083350+0000","last_unstale":"2023-05-22T15:46:23.090669+0000","last_undegraded":"2023-05-22T15:46:23.090669+0000","last_fullsized":"2023-05-22T15:46:23.090669+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_clean_scrub_stamp":"2023-05-22T15:46:22.034488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:03:14.130053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.1d","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.090978+0000","last_change":"2023-05-22T15:46:23.090978+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.090978+0000","last_undegraded":"2023-05-22T15:46:23.090978+0000","last_fullsized":"2023-05-22T15:46:23.090978+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:07:01.166140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1c","version":"0'0","reported_seq":9,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.094786+0000","last_change":"2023-05-22T15:46:23.092200+0000","last_active":"2023-05-22T15:46:23.094786+0000","last_peered":"2023-05-22T15:46:23.094786+0000","last_clean":"2023-05-22T15:46:23.094786+0000","last_became_active":"2023-05-22T15:46:23.089522+0000","last_became_peered":"2023-05-22T15:46:23.089522+0000","last_unstale":"2023-05-22T15:46:23.094786+0000","last_undegraded":"2023-05-22T15:46:23.094786+0000","last_fullsized":"2023-05-22T15:46:23.094786+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_clean_scrub_stamp":"2023-05-22T15:46:22.034488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:16:51.789402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.1a","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.061711+0000","last_change":"2023-05-22T15:46:23.061711+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.061711+0000","last_undegraded":"2023-05-22T15:46:23.061711+0000","last_fullsized":"2023-05-22T15:46:23.061711+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:23:42.844820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1b","version":"0'0","reported_seq":9,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.094709+0000","last_change":"2023-05-22T15:46:23.088945+0000","last_active":"2023-05-22T15:46:23.094709+0000","last_peered":"2023-05-22T15:46:23.094709+0000","last_clean":"2023-05-22T15:46:23.094709+0000","last_became_active":"2023-05-22T15:46:23.088536+0000","last_became_peered":"2023-05-22T15:46:23.088536+0000","last_unstale":"2023-05-22T15:46:23.094709+0000","last_undegraded":"2023-05-22T15:46:23.094709+0000","last_fullsized":"2023-05-22T15:46:23.094709+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_clean_scrub_stamp":"2023-05-22T15:46:22.034488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:12:41.914428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.1b","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.073120+0000","last_change":"2023-05-22T15:46:23.073120+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.073120+0000","last_undegraded":"2023-05-22T15:46:23.073120+0000","last_fullsized":"2023-05-22T15:46:23.073120+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:59:24.451915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.1a","version":"0'0","reported_seq":9,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.090491+0000","last_change":"2023-05-22T15:46:23.089182+0000","last_active":"2023-05-22T15:46:23.090491+0000","last_peered":"2023-05-22T15:46:23.090491+0000","last_clean":"2023-05-22T15:46:23.090491+0000","last_became_active":"2023-05-22T15:46:23.088748+0000","last_became_peered":"2023-05-22T15:46:23.088748+0000","last_unstale":"2023-05-22T15:46:23.090491+0000","last_undegraded":"2023-05-22T15:46:23.090491+0000","last_fullsized":"2023-05-22T15:46:23.090491+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_clean_scrub_stamp":"2023-05-22T15:46:22.034488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:49:33.716090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.18","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.077114+0000","last_change":"2023-05-22T15:46:23.077114+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.077114+0000","last_undegraded":"2023-05-22T15:46:23.077114+0000","last_fullsized":"2023-05-22T15:46:23.077114+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T03:20:39.103248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.19","version":"0'0","reported_seq":9,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.117596+0000","last_change":"2023-05-22T15:46:23.105440+0000","last_active":"2023-05-22T15:46:23.117596+0000","last_peered":"2023-05-22T15:46:23.117596+0000","last_clean":"2023-05-22T15:46:23.117596+0000","last_became_active":"2023-05-22T15:46:23.104687+0000","last_became_peered":"2023-05-22T15:46:23.104687+0000","last_unstale":"2023-05-22T15:46:23.117596+0000","last_undegraded":"2023-05-22T15:46:23.117596+0000","last_fullsized":"2023-05-22T15:46:23.117596+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_clean_scrub_stamp":"2023-05-22T15:46:22.034488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:51:07.846056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.19","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.075186+0000","last_change":"2023-05-22T15:46:23.075186+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.075186+0000","last_undegraded":"2023-05-22T15:46:23.075186+0000","last_fullsized":"2023-05-22T15:46:23.075186+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:22:20.631045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.18","version":"0'0","reported_seq":9,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.108549+0000","last_change":"2023-05-22T15:46:23.106544+0000","last_active":"2023-05-22T15:46:23.108549+0000","last_peered":"2023-05-22T15:46:23.108549+0000","last_clean":"2023-05-22T15:46:23.108549+0000","last_became_active":"2023-05-22T15:46:23.106098+0000","last_became_peered":"2023-05-22T15:46:23.106098+0000","last_unstale":"2023-05-22T15:46:23.108549+0000","last_undegraded":"2023-05-22T15:46:23.108549+0000","last_fullsized":"2023-05-22T15:46:23.108549+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_clean_scrub_stamp":"2023-05-22T15:46:22.034488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:42:03.486863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.e","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.083614+0000","last_change":"2023-05-22T15:46:23.083614+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.083614+0000","last_undegraded":"2023-05-22T15:46:23.083614+0000","last_fullsized":"2023-05-22T15:46:23.083614+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:20:55.836997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.f","version":"0'0","reported_seq":9,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.116428+0000","last_change":"2023-05-22T15:46:23.112873+0000","last_active":"2023-05-22T15:46:23.116428+0000","last_peered":"2023-05-22T15:46:23.116428+0000","last_clean":"2023-05-22T15:46:23.116428+0000","last_became_active":"2023-05-22T15:46:23.112082+0000","last_became_peered":"2023-05-22T15:46:23.112082+0000","last_unstale":"2023-05-22T15:46:23.116428+0000","last_undegraded":"2023-05-22T15:46:23.116428+0000","last_fullsized":"2023-05-22T15:46:23.116428+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_clean_scrub_stamp":"2023-05-22T15:46:22.034488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:24:32.309552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.f","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.063425+0000","last_change":"2023-05-22T15:46:23.063425+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.063425+0000","last_undegraded":"2023-05-22T15:46:23.063425+0000","last_fullsized":"2023-05-22T15:46:23.063425+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T16:10:54.516325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.e","version":"0'0","reported_seq":9,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.109486+0000","last_change":"2023-05-22T15:46:23.107165+0000","last_active":"2023-05-22T15:46:23.109486+0000","last_peered":"2023-05-22T15:46:23.109486+0000","last_clean":"2023-05-22T15:46:23.109486+0000","last_became_active":"2023-05-22T15:46:23.106681+0000","last_became_peered":"2023-05-22T15:46:23.106681+0000","last_unstale":"2023-05-22T15:46:23.109486+0000","last_undegraded":"2023-05-22T15:46:23.109486+0000","last_fullsized":"2023-05-22T15:46:23.109486+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_clean_scrub_stamp":"2023-05-22T15:46:22.034488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T15:59:17.155128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.c","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.060697+0000","last_change":"2023-05-22T15:46:23.060697+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.060697+0000","last_undegraded":"2023-05-22T15:46:23.060697+0000","last_fullsized":"2023-05-22T15:46:23.060697+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:33:34.943631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.d","version":"0'0","reported_seq":9,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.111448+0000","last_change":"2023-05-22T15:46:23.103043+0000","last_active":"2023-05-22T15:46:23.111448+0000","last_peered":"2023-05-22T15:46:23.111448+0000","last_clean":"2023-05-22T15:46:23.111448+0000","last_became_active":"2023-05-22T15:46:23.102220+0000","last_became_peered":"2023-05-22T15:46:23.102220+0000","last_unstale":"2023-05-22T15:46:23.111448+0000","last_undegraded":"2023-05-22T15:46:23.111448+0000","last_fullsized":"2023-05-22T15:46:23.111448+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_clean_scrub_stamp":"2023-05-22T15:46:22.034488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:20:54.824911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.d","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.073232+0000","last_change":"2023-05-22T15:46:23.073232+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.073232+0000","last_undegraded":"2023-05-22T15:46:23.073232+0000","last_fullsized":"2023-05-22T15:46:23.073232+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:49:48.656654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.c","version":"0'0","reported_seq":9,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.082235+0000","last_change":"2023-05-22T15:46:23.078278+0000","last_active":"2023-05-22T15:46:23.082235+0000","last_peered":"2023-05-22T15:46:23.082235+0000","last_clean":"2023-05-22T15:46:23.082235+0000","last_became_active":"2023-05-22T15:46:23.077343+0000","last_became_peered":"2023-05-22T15:46:23.077343+0000","last_unstale":"2023-05-22T15:46:23.082235+0000","last_undegraded":"2023-05-22T15:46:23.082235+0000","last_fullsized":"2023-05-22T15:46:23.082235+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_clean_scrub_stamp":"2023-05-22T15:46:22.034488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:17:43.848098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.a","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.072991+0000","last_change":"2023-05-22T15:46:23.072991+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.072991+0000","last_undegraded":"2023-05-22T15:46:23.072991+0000","last_fullsized":"2023-05-22T15:46:23.072991+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:24:33.321639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.b","version":"0'0","reported_seq":9,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.092758+0000","last_change":"2023-05-22T15:46:23.086857+0000","last_active":"2023-05-22T15:46:23.092758+0000","last_peered":"2023-05-22T15:46:23.092758+0000","last_clean":"2023-05-22T15:46:23.092758+0000","last_became_active":"2023-05-22T15:46:23.086098+0000","last_became_peered":"2023-05-22T15:46:23.086098+0000","last_unstale":"2023-05-22T15:46:23.092758+0000","last_undegraded":"2023-05-22T15:46:23.092758+0000","last_fullsized":"2023-05-22T15:46:23.092758+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_clean_scrub_stamp":"2023-05-22T15:46:22.034488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:45:50.595714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.b","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.071529+0000","last_change":"2023-05-22T15:46:23.071529+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.071529+0000","last_undegraded":"2023-05-22T15:46:23.071529+0000","last_fullsized":"2023-05-22T15:46:23.071529+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:20:12.734869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.a","version":"0'0","reported_seq":9,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.107500+0000","last_change":"2023-05-22T15:46:23.106615+0000","last_active":"2023-05-22T15:46:23.107500+0000","last_peered":"2023-05-22T15:46:23.107500+0000","last_clean":"2023-05-22T15:46:23.107500+0000","last_became_active":"2023-05-22T15:46:23.106155+0000","last_became_peered":"2023-05-22T15:46:23.106155+0000","last_unstale":"2023-05-22T15:46:23.107500+0000","last_undegraded":"2023-05-22T15:46:23.107500+0000","last_fullsized":"2023-05-22T15:46:23.107500+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_clean_scrub_stamp":"2023-05-22T15:46:22.034488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:55:28.593183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.8","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.071955+0000","last_change":"2023-05-22T15:46:23.071955+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.071955+0000","last_undegraded":"2023-05-22T15:46:23.071955+0000","last_fullsized":"2023-05-22T15:46:23.071955+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T16:27:47.997551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.9","version":"0'0","reported_seq":9,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.082561+0000","last_change":"2023-05-22T15:46:23.080736+0000","last_active":"2023-05-22T15:46:23.082561+0000","last_peered":"2023-05-22T15:46:23.082561+0000","last_clean":"2023-05-22T15:46:23.082561+0000","last_became_active":"2023-05-22T15:46:23.080251+0000","last_became_peered":"2023-05-22T15:46:23.080251+0000","last_unstale":"2023-05-22T15:46:23.082561+0000","last_undegraded":"2023-05-22T15:46:23.082561+0000","last_fullsized":"2023-05-22T15:46:23.082561+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_clean_scrub_stamp":"2023-05-22T15:46:22.034488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T16:52:22.575852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.9","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.089066+0000","last_change":"2023-05-22T15:46:23.089066+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.089066+0000","last_undegraded":"2023-05-22T15:46:23.089066+0000","last_fullsized":"2023-05-22T15:46:23.089066+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:31:09.541110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.8","version":"0'0","reported_seq":9,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.109492+0000","last_change":"2023-05-22T15:46:23.107191+0000","last_active":"2023-05-22T15:46:23.109492+0000","last_peered":"2023-05-22T15:46:23.109492+0000","last_clean":"2023-05-22T15:46:23.109492+0000","last_became_active":"2023-05-22T15:46:23.106653+0000","last_became_peered":"2023-05-22T15:46:23.106653+0000","last_unstale":"2023-05-22T15:46:23.109492+0000","last_undegraded":"2023-05-22T15:46:23.109492+0000","last_fullsized":"2023-05-22T15:46:23.109492+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_clean_scrub_stamp":"2023-05-22T15:46:22.034488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:00:00.382995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.6","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.076413+0000","last_change":"2023-05-22T15:46:23.076413+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.076413+0000","last_undegraded":"2023-05-22T15:46:23.076413+0000","last_fullsized":"2023-05-22T15:46:23.076413+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:55:41.010860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.7","version":"0'0","reported_seq":9,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.092881+0000","last_change":"2023-05-22T15:46:23.091817+0000","last_active":"2023-05-22T15:46:23.092881+0000","last_peered":"2023-05-22T15:46:23.092881+0000","last_clean":"2023-05-22T15:46:23.092881+0000","last_became_active":"2023-05-22T15:46:23.091434+0000","last_became_peered":"2023-05-22T15:46:23.091434+0000","last_unstale":"2023-05-22T15:46:23.092881+0000","last_undegraded":"2023-05-22T15:46:23.092881+0000","last_fullsized":"2023-05-22T15:46:23.092881+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_clean_scrub_stamp":"2023-05-22T15:46:22.034488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T16:37:19.990243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.7","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.079702+0000","last_change":"2023-05-22T15:46:23.079702+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.079702+0000","last_undegraded":"2023-05-22T15:46:23.079702+0000","last_fullsized":"2023-05-22T15:46:23.079702+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:20:29.148898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.6","version":"0'0","reported_seq":9,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.120724+0000","last_change":"2023-05-22T15:46:23.114945+0000","last_active":"2023-05-22T15:46:23.120724+0000","last_peered":"2023-05-22T15:46:23.120724+0000","last_clean":"2023-05-22T15:46:23.120724+0000","last_became_active":"2023-05-22T15:46:23.114027+0000","last_became_peered":"2023-05-22T15:46:23.114027+0000","last_unstale":"2023-05-22T15:46:23.120724+0000","last_undegraded":"2023-05-22T15:46:23.120724+0000","last_fullsized":"2023-05-22T15:46:23.120724+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_clean_scrub_stamp":"2023-05-22T15:46:22.034488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T18:52:28.340453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.4","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.075841+0000","last_change":"2023-05-22T15:46:23.075841+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.075841+0000","last_undegraded":"2023-05-22T15:46:23.075841+0000","last_fullsized":"2023-05-22T15:46:23.075841+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:46:38.837010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.5","version":"0'0","reported_seq":9,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.109205+0000","last_change":"2023-05-22T15:46:23.108380+0000","last_active":"2023-05-22T15:46:23.109205+0000","last_peered":"2023-05-22T15:46:23.109205+0000","last_clean":"2023-05-22T15:46:23.109205+0000","last_became_active":"2023-05-22T15:46:23.108014+0000","last_became_peered":"2023-05-22T15:46:23.108014+0000","last_unstale":"2023-05-22T15:46:23.109205+0000","last_undegraded":"2023-05-22T15:46:23.109205+0000","last_fullsized":"2023-05-22T15:46:23.109205+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_clean_scrub_stamp":"2023-05-22T15:46:22.034488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T03:37:19.248335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.5","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.073829+0000","last_change":"2023-05-22T15:46:23.073829+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.073829+0000","last_undegraded":"2023-05-22T15:46:23.073829+0000","last_fullsized":"2023-05-22T15:46:23.073829+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:47:45.260369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.4","version":"0'0","reported_seq":9,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.094601+0000","last_change":"2023-05-22T15:46:23.093730+0000","last_active":"2023-05-22T15:46:23.094601+0000","last_peered":"2023-05-22T15:46:23.094601+0000","last_clean":"2023-05-22T15:46:23.094601+0000","last_became_active":"2023-05-22T15:46:23.093418+0000","last_became_peered":"2023-05-22T15:46:23.093418+0000","last_unstale":"2023-05-22T15:46:23.094601+0000","last_undegraded":"2023-05-22T15:46:23.094601+0000","last_fullsized":"2023-05-22T15:46:23.094601+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_clean_scrub_stamp":"2023-05-22T15:46:22.034488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T16:38:30.323718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.2","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.084350+0000","last_change":"2023-05-22T15:46:23.084350+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.084350+0000","last_undegraded":"2023-05-22T15:46:23.084350+0000","last_fullsized":"2023-05-22T15:46:23.084350+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:25:55.107784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.3","version":"0'0","reported_seq":9,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.111036+0000","last_change":"2023-05-22T15:46:23.110053+0000","last_active":"2023-05-22T15:46:23.111036+0000","last_peered":"2023-05-22T15:46:23.111036+0000","last_clean":"2023-05-22T15:46:23.111036+0000","last_became_active":"2023-05-22T15:46:23.109753+0000","last_became_peered":"2023-05-22T15:46:23.109753+0000","last_unstale":"2023-05-22T15:46:23.111036+0000","last_undegraded":"2023-05-22T15:46:23.111036+0000","last_fullsized":"2023-05-22T15:46:23.111036+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_clean_scrub_stamp":"2023-05-22T15:46:22.034488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T18:29:44.257708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.3","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.062480+0000","last_change":"2023-05-22T15:46:23.062480+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.062480+0000","last_undegraded":"2023-05-22T15:46:23.062480+0000","last_fullsized":"2023-05-22T15:46:23.062480+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T18:05:16.988218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.2","version":"0'0","reported_seq":9,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.120623+0000","last_change":"2023-05-22T15:46:23.112868+0000","last_active":"2023-05-22T15:46:23.120623+0000","last_peered":"2023-05-22T15:46:23.120623+0000","last_clean":"2023-05-22T15:46:23.120623+0000","last_became_active":"2023-05-22T15:46:23.112288+0000","last_became_peered":"2023-05-22T15:46:23.112288+0000","last_unstale":"2023-05-22T15:46:23.120623+0000","last_undegraded":"2023-05-22T15:46:23.120623+0000","last_fullsized":"2023-05-22T15:46:23.120623+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_clean_scrub_stamp":"2023-05-22T15:46:22.034488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:16:19.873549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.0","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.062429+0000","last_change":"2023-05-22T15:46:23.062429+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.062429+0000","last_undegraded":"2023-05-22T15:46:23.062429+0000","last_fullsized":"2023-05-22T15:46:23.062429+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:03:37.104570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1","version":"0'0","reported_seq":9,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.092294+0000","last_change":"2023-05-22T15:46:23.091423+0000","last_active":"2023-05-22T15:46:23.092294+0000","last_peered":"2023-05-22T15:46:23.092294+0000","last_clean":"2023-05-22T15:46:23.092294+0000","last_became_active":"2023-05-22T15:46:23.091088+0000","last_became_peered":"2023-05-22T15:46:23.091088+0000","last_unstale":"2023-05-22T15:46:23.092294+0000","last_undegraded":"2023-05-22T15:46:23.092294+0000","last_fullsized":"2023-05-22T15:46:23.092294+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_clean_scrub_stamp":"2023-05-22T15:46:22.034488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:52:16.638521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.1","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.072920+0000","last_change":"2023-05-22T15:46:23.072920+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.072920+0000","last_undegraded":"2023-05-22T15:46:23.072920+0000","last_fullsized":"2023-05-22T15:46:23.072920+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:16:20.885635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.0","version":"0'0","reported_seq":9,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.107395+0000","last_change":"2023-05-22T15:46:23.105214+0000","last_active":"2023-05-22T15:46:23.107395+0000","last_peered":"2023-05-22T15:46:23.107395+0000","last_clean":"2023-05-22T15:46:23.107395+0000","last_became_active":"2023-05-22T15:46:23.104909+0000","last_became_peered":"2023-05-22T15:46:23.104909+0000","last_unstale":"2023-05-22T15:46:23.107395+0000","last_undegraded":"2023-05-22T15:46:23.107395+0000","last_fullsized":"2023-05-22T15:46:23.107395+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_clean_scrub_stamp":"2023-05-22T15:46:22.034488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:37:03.818368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.12","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.085873+0000","last_change":"2023-05-22T15:46:23.085873+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.085873+0000","last_undegraded":"2023-05-22T15:46:23.085873+0000","last_fullsized":"2023-05-22T15:46:23.085873+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:20:55.836997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.13","version":"0'0","reported_seq":9,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.120574+0000","last_change":"2023-05-22T15:46:23.112061+0000","last_active":"2023-05-22T15:46:23.120574+0000","last_peered":"2023-05-22T15:46:23.120574+0000","last_clean":"2023-05-22T15:46:23.120574+0000","last_became_active":"2023-05-22T15:46:23.111315+0000","last_became_peered":"2023-05-22T15:46:23.111315+0000","last_unstale":"2023-05-22T15:46:23.120574+0000","last_undegraded":"2023-05-22T15:46:23.120574+0000","last_fullsized":"2023-05-22T15:46:23.120574+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_clean_scrub_stamp":"2023-05-22T15:46:22.034488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:37:25.171147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"134'249","reported_seq":753,"reported_epoch":144,"state":"active+clean","last_fresh":"2023-05-22T15:46:22.069225+0000","last_change":"2023-05-22T15:26:39.318204+0000","last_active":"2023-05-22T15:46:22.069225+0000","last_peered":"2023-05-22T15:46:22.069225+0000","last_clean":"2023-05-22T15:46:22.069225+0000","last_became_active":"2023-05-22T15:26:39.315370+0000","last_became_peered":"2023-05-22T15:26:39.315370+0000","last_unstale":"2023-05-22T15:46:22.069225+0000","last_undegraded":"2023-05-22T15:46:22.069225+0000","last_fullsized":"2023-05-22T15:46:22.069225+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:38.020076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:38.020076+0000","last_clean_scrub_stamp":"2023-05-22T15:26:38.020076+0000","objects_scrubbed":0,"log_size":249,"log_dups_size":0,"ondisk_log_size":249,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:06:39.207543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1268,"num_read_kb":2572,"num_write":399,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":139,"reported_epoch":144,"state":"active+clean","last_fresh":"2023-05-22T15:46:22.059466+0000","last_change":"2023-05-22T15:26:44.971951+0000","last_active":"2023-05-22T15:46:22.059466+0000","last_peered":"2023-05-22T15:46:22.059466+0000","last_clean":"2023-05-22T15:46:22.059466+0000","last_became_active":"2023-05-22T15:26:44.970872+0000","last_became_peered":"2023-05-22T15:26:44.970872+0000","last_unstale":"2023-05-22T15:46:22.059466+0000","last_undegraded":"2023-05-22T15:46:22.059466+0000","last_fullsized":"2023-05-22T15:46:22.059466+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:55:34.752949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.11","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.082774+0000","last_change":"2023-05-22T15:46:23.082774+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.082774+0000","last_undegraded":"2023-05-22T15:46:23.082774+0000","last_fullsized":"2023-05-22T15:46:23.082774+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:37:33.613777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.10","version":"0'0","reported_seq":9,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.107838+0000","last_change":"2023-05-22T15:46:23.091282+0000","last_active":"2023-05-22T15:46:23.107838+0000","last_peered":"2023-05-22T15:46:23.107838+0000","last_clean":"2023-05-22T15:46:23.107838+0000","last_became_active":"2023-05-22T15:46:23.086640+0000","last_became_peered":"2023-05-22T15:46:23.086640+0000","last_unstale":"2023-05-22T15:46:23.107838+0000","last_undegraded":"2023-05-22T15:46:23.107838+0000","last_fullsized":"2023-05-22T15:46:23.107838+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_clean_scrub_stamp":"2023-05-22T15:46:22.034488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T16:27:46.985465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":132,"reported_epoch":144,"state":"active+clean","last_fresh":"2023-05-22T15:46:22.068980+0000","last_change":"2023-05-22T15:26:44.977198+0000","last_active":"2023-05-22T15:46:22.068980+0000","last_peered":"2023-05-22T15:46:22.068980+0000","last_clean":"2023-05-22T15:46:22.068980+0000","last_became_active":"2023-05-22T15:26:44.976445+0000","last_became_peered":"2023-05-22T15:26:44.976445+0000","last_unstale":"2023-05-22T15:46:22.068980+0000","last_undegraded":"2023-05-22T15:46:22.068980+0000","last_fullsized":"2023-05-22T15:46:22.068980+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:12:41.464054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.10","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.073353+0000","last_change":"2023-05-22T15:46:23.073353+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.073353+0000","last_undegraded":"2023-05-22T15:46:23.073353+0000","last_fullsized":"2023-05-22T15:46:23.073353+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:42:45.119566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.11","version":"0'0","reported_seq":9,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.103893+0000","last_change":"2023-05-22T15:46:23.102815+0000","last_active":"2023-05-22T15:46:23.103893+0000","last_peered":"2023-05-22T15:46:23.103893+0000","last_clean":"2023-05-22T15:46:23.103893+0000","last_became_active":"2023-05-22T15:46:23.102405+0000","last_became_peered":"2023-05-22T15:46:23.102405+0000","last_unstale":"2023-05-22T15:46:23.103893+0000","last_undegraded":"2023-05-22T15:46:23.103893+0000","last_fullsized":"2023-05-22T15:46:23.103893+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_clean_scrub_stamp":"2023-05-22T15:46:22.034488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:45:50.595714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":132,"reported_epoch":144,"state":"active+clean","last_fresh":"2023-05-22T15:46:22.051914+0000","last_change":"2023-05-22T15:26:44.969469+0000","last_active":"2023-05-22T15:46:22.051914+0000","last_peered":"2023-05-22T15:46:22.051914+0000","last_clean":"2023-05-22T15:46:22.051914+0000","last_became_active":"2023-05-22T15:26:44.969083+0000","last_became_peered":"2023-05-22T15:26:44.969083+0000","last_unstale":"2023-05-22T15:46:22.051914+0000","last_undegraded":"2023-05-22T15:46:22.051914+0000","last_fullsized":"2023-05-22T15:46:22.051914+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:29:23.670067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.13","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.076508+0000","last_change":"2023-05-22T15:46:23.076508+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.076508+0000","last_undegraded":"2023-05-22T15:46:23.076508+0000","last_fullsized":"2023-05-22T15:46:23.076508+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T16:27:47.997551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.12","version":"0'0","reported_seq":9,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.107682+0000","last_change":"2023-05-22T15:46:23.105368+0000","last_active":"2023-05-22T15:46:23.107682+0000","last_peered":"2023-05-22T15:46:23.107682+0000","last_clean":"2023-05-22T15:46:23.107682+0000","last_became_active":"2023-05-22T15:46:23.104771+0000","last_became_peered":"2023-05-22T15:46:23.104771+0000","last_unstale":"2023-05-22T15:46:23.107682+0000","last_undegraded":"2023-05-22T15:46:23.107682+0000","last_fullsized":"2023-05-22T15:46:23.107682+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_clean_scrub_stamp":"2023-05-22T15:46:22.034488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:42:03.486863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":139,"reported_epoch":144,"state":"active+clean","last_fresh":"2023-05-22T15:46:22.059265+0000","last_change":"2023-05-22T15:26:44.976632+0000","last_active":"2023-05-22T15:46:22.059265+0000","last_peered":"2023-05-22T15:46:22.059265+0000","last_clean":"2023-05-22T15:46:22.059265+0000","last_became_active":"2023-05-22T15:26:44.975546+0000","last_became_peered":"2023-05-22T15:26:44.975546+0000","last_unstale":"2023-05-22T15:46:22.059265+0000","last_undegraded":"2023-05-22T15:46:22.059265+0000","last_fullsized":"2023-05-22T15:46:22.059265+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:42:57.973030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.15","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.084525+0000","last_change":"2023-05-22T15:46:23.084525+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.084525+0000","last_undegraded":"2023-05-22T15:46:23.084525+0000","last_fullsized":"2023-05-22T15:46:23.084525+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T03:25:18.349611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.14","version":"0'0","reported_seq":9,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.109240+0000","last_change":"2023-05-22T15:46:23.104870+0000","last_active":"2023-05-22T15:46:23.109240+0000","last_peered":"2023-05-22T15:46:23.109240+0000","last_clean":"2023-05-22T15:46:23.109240+0000","last_became_active":"2023-05-22T15:46:23.104126+0000","last_became_peered":"2023-05-22T15:46:23.104126+0000","last_unstale":"2023-05-22T15:46:23.109240+0000","last_undegraded":"2023-05-22T15:46:23.109240+0000","last_fullsized":"2023-05-22T15:46:23.109240+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_clean_scrub_stamp":"2023-05-22T15:46:22.034488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:20:28.136812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":123,"reported_epoch":144,"state":"active+clean","last_fresh":"2023-05-22T15:46:22.065910+0000","last_change":"2023-05-22T15:26:45.091475+0000","last_active":"2023-05-22T15:46:22.065910+0000","last_peered":"2023-05-22T15:46:22.065910+0000","last_clean":"2023-05-22T15:46:22.065910+0000","last_became_active":"2023-05-22T15:26:45.091143+0000","last_became_peered":"2023-05-22T15:26:45.091143+0000","last_unstale":"2023-05-22T15:46:22.065910+0000","last_undegraded":"2023-05-22T15:46:22.065910+0000","last_fullsized":"2023-05-22T15:46:22.065910+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:12:41.464054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.14","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.088461+0000","last_change":"2023-05-22T15:46:23.088461+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.088461+0000","last_undegraded":"2023-05-22T15:46:23.088461+0000","last_fullsized":"2023-05-22T15:46:23.088461+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:49:34.728176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.15","version":"0'0","reported_seq":9,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.092614+0000","last_change":"2023-05-22T15:46:23.083261+0000","last_active":"2023-05-22T15:46:23.092614+0000","last_peered":"2023-05-22T15:46:23.092614+0000","last_clean":"2023-05-22T15:46:23.092614+0000","last_became_active":"2023-05-22T15:46:23.082317+0000","last_became_peered":"2023-05-22T15:46:23.082317+0000","last_unstale":"2023-05-22T15:46:23.092614+0000","last_undegraded":"2023-05-22T15:46:23.092614+0000","last_fullsized":"2023-05-22T15:46:23.092614+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_clean_scrub_stamp":"2023-05-22T15:46:22.034488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T18:29:44.257708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":132,"reported_epoch":144,"state":"active+clean","last_fresh":"2023-05-22T15:46:22.069477+0000","last_change":"2023-05-22T15:26:45.092367+0000","last_active":"2023-05-22T15:46:22.069477+0000","last_peered":"2023-05-22T15:46:22.069477+0000","last_clean":"2023-05-22T15:46:22.069477+0000","last_became_active":"2023-05-22T15:26:45.091951+0000","last_became_peered":"2023-05-22T15:26:45.091951+0000","last_unstale":"2023-05-22T15:46:22.069477+0000","last_undegraded":"2023-05-22T15:46:22.069477+0000","last_fullsized":"2023-05-22T15:46:22.069477+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:42:57.973030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.17","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.065336+0000","last_change":"2023-05-22T15:46:23.065336+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.065336+0000","last_undegraded":"2023-05-22T15:46:23.065336+0000","last_fullsized":"2023-05-22T15:46:23.065336+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T16:27:47.997551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.16","version":"0'0","reported_seq":9,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.107509+0000","last_change":"2023-05-22T15:46:23.106558+0000","last_active":"2023-05-22T15:46:23.107509+0000","last_peered":"2023-05-22T15:46:23.107509+0000","last_clean":"2023-05-22T15:46:23.107509+0000","last_became_active":"2023-05-22T15:46:23.106098+0000","last_became_peered":"2023-05-22T15:46:23.106098+0000","last_unstale":"2023-05-22T15:46:23.107509+0000","last_undegraded":"2023-05-22T15:46:23.107509+0000","last_fullsized":"2023-05-22T15:46:23.107509+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_clean_scrub_stamp":"2023-05-22T15:46:22.034488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:30:32.850917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":123,"reported_epoch":144,"state":"active+clean","last_fresh":"2023-05-22T15:46:22.065875+0000","last_change":"2023-05-22T15:26:44.978858+0000","last_active":"2023-05-22T15:46:22.065875+0000","last_peered":"2023-05-22T15:46:22.065875+0000","last_clean":"2023-05-22T15:46:22.065875+0000","last_became_active":"2023-05-22T15:26:44.978443+0000","last_became_peered":"2023-05-22T15:26:44.978443+0000","last_unstale":"2023-05-22T15:46:22.065875+0000","last_undegraded":"2023-05-22T15:46:22.065875+0000","last_fullsized":"2023-05-22T15:46:22.065875+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:46:29.290243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.16","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.071629+0000","last_change":"2023-05-22T15:46:23.071629+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.071629+0000","last_undegraded":"2023-05-22T15:46:23.071629+0000","last_fullsized":"2023-05-22T15:46:23.071629+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T18:29:45.269794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.17","version":"0'0","reported_seq":9,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.092889+0000","last_change":"2023-05-22T15:46:23.084909+0000","last_active":"2023-05-22T15:46:23.092889+0000","last_peered":"2023-05-22T15:46:23.092889+0000","last_clean":"2023-05-22T15:46:23.092889+0000","last_became_active":"2023-05-22T15:46:23.084184+0000","last_became_peered":"2023-05-22T15:46:23.084184+0000","last_unstale":"2023-05-22T15:46:23.092889+0000","last_undegraded":"2023-05-22T15:46:23.092889+0000","last_fullsized":"2023-05-22T15:46:23.092889+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_clean_scrub_stamp":"2023-05-22T15:46:22.034488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:47:36.764285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":125,"reported_epoch":144,"state":"active+clean","last_fresh":"2023-05-22T15:46:22.063472+0000","last_change":"2023-05-22T15:26:44.982421+0000","last_active":"2023-05-22T15:46:22.063472+0000","last_peered":"2023-05-22T15:46:22.063472+0000","last_clean":"2023-05-22T15:46:22.063472+0000","last_became_active":"2023-05-22T15:26:44.981586+0000","last_became_peered":"2023-05-22T15:26:44.981586+0000","last_unstale":"2023-05-22T15:46:22.063472+0000","last_undegraded":"2023-05-22T15:46:22.063472+0000","last_fullsized":"2023-05-22T15:46:22.063472+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:14:11.156544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":19,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":18,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1268,"num_read_kb":2572,"num_write":399,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":249,"ondisk_log_size":249,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575084,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":19680,"kb_used_data":2328,"kb_used_omap":0,"kb_used_meta":17344,"kb_avail":94352160,"statfs":{"total":96636764160,"available":96616611840,"internally_reserved":0,"allocated":2383872,"data_stored":1535545,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":17760256},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 15:46:11 2023","interfaces":[{"interface":"back","average":{"1min":0.86699999999999999,"5min":0.89200000000000002,"15min":0.876},"min":{"1min":0.53600000000000003,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":1.294,"5min":2.637,"15min":2.637},"last":0.76600000000000001},{"interface":"front","average":{"1min":0.88,"5min":0.88700000000000001,"15min":0.89200000000000002},"min":{"1min":0.60799999999999998,"5min":0.502,"15min":0.35799999999999998},"max":{"1min":1.1519999999999999,"5min":3.0449999999999999,"15min":3.0449999999999999},"last":0.86199999999999999}]},{"osd":1,"last update":"Mon May 22 15:46:11 2023","interfaces":[{"interface":"back","average":{"1min":0.96799999999999997,"5min":0.95399999999999996,"15min":0.97899999999999998},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.1899999999999999,"5min":1.385,"15min":1.389},"last":0.73799999999999999},{"interface":"front","average":{"1min":0.96799999999999997,"5min":1.006,"15min":1.022},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.54900000000000004},"max":{"1min":1.226,"5min":2.7269999999999999,"15min":2.7269999999999999},"last":0.99299999999999999}]},{"osd":2,"last update":"Mon May 22 15:46:11 2023","interfaces":[{"interface":"back","average":{"1min":0.86199999999999999,"5min":0.85099999999999998,"15min":0.871},"min":{"1min":0.55600000000000005,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.109,"5min":1.252,"15min":1.252},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.88100000000000001,"5min":0.93999999999999995,"15min":0.95699999999999996},"min":{"1min":0.504,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.143,"5min":3.141,"15min":3.141},"last":0.68500000000000005}]},{"osd":3,"last update":"Mon May 22 15:46:11 2023","interfaces":[{"interface":"back","average":{"1min":0.98199999999999998,"5min":0.98899999999999999,"15min":0.96299999999999997},"min":{"1min":0.67600000000000005,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":1.3839999999999999,"5min":2.8149999999999999,"15min":2.8149999999999999},"last":1.109},{"interface":"front","average":{"1min":0.94399999999999995,"5min":0.92100000000000004,"15min":0.96199999999999997},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.2709999999999999,"5min":1.389,"15min":1.4710000000000001},"last":0.82899999999999996}]},{"osd":4,"last update":"Mon May 22 15:46:11 2023","interfaces":[{"interface":"back","average":{"1min":0.87,"5min":0.94399999999999995,"15min":0.90400000000000003},"min":{"1min":0.60299999999999998,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.5089999999999999,"5min":3.472,"15min":3.472},"last":0.94699999999999995},{"interface":"front","average":{"1min":0.997,"5min":1.1850000000000001,"15min":1.05},"min":{"1min":0.59699999999999998,"5min":0.56000000000000005,"15min":0.44600000000000001},"max":{"1min":1.2949999999999999,"5min":15.627000000000001,"15min":15.627000000000001},"last":1.163}]},{"osd":5,"last update":"Mon May 22 15:46:11 2023","interfaces":[{"interface":"back","average":{"1min":1.012,"5min":1.071,"15min":1.02},"min":{"1min":0.60899999999999999,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":1.4750000000000001,"5min":4.5599999999999996,"15min":4.5599999999999996},"last":0.90900000000000003},{"interface":"front","average":{"1min":1.115,"5min":1.161,"15min":1.139},"min":{"1min":0.82999999999999996,"5min":0.63800000000000001,"15min":0.58799999999999997},"max":{"1min":1.302,"5min":4.3799999999999999,"15min":4.3799999999999999},"last":1.1919999999999999}]},{"osd":6,"last update":"Mon May 22 15:46:11 2023","interfaces":[{"interface":"back","average":{"1min":1.073,"5min":1.1399999999999999,"15min":1.0840000000000001},"min":{"1min":0.80300000000000005,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.415,"5min":6.5069999999999997,"15min":6.5069999999999997},"last":1.1339999999999999},{"interface":"front","average":{"1min":1.1200000000000001,"5min":1.1659999999999999,"15min":1.1419999999999999},"min":{"1min":0.70299999999999996,"5min":0.62,"15min":0.62},"max":{"1min":1.458,"5min":3.3500000000000001,"15min":3.3500000000000001},"last":1.214}]}]},{"osd":6,"up_from":13,"seq":55834575085,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36572,"kb_used_data":1748,"kb_used_omap":0,"kb_used_meta":34816,"kb_avail":94335268,"statfs":{"total":96636764160,"available":96599314432,"internally_reserved":0,"allocated":1789952,"data_stored":945177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":35651584},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 15:46:20 2023","interfaces":[{"interface":"back","average":{"1min":0.90000000000000002,"5min":0.86699999999999999,"15min":0.89900000000000002},"min":{"1min":0.499,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":1.4039999999999999,"5min":1.486,"15min":1.486},"last":0.81899999999999995},{"interface":"front","average":{"1min":0.89200000000000002,"5min":0.94599999999999995,"15min":0.91100000000000003},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.36299999999999999},"max":{"1min":1.2729999999999999,"5min":1.6240000000000001,"15min":1.6240000000000001},"last":1.054}]},{"osd":1,"last update":"Mon May 22 15:46:20 2023","interfaces":[{"interface":"back","average":{"1min":0.93200000000000005,"5min":0.89100000000000001,"15min":0.91200000000000003},"min":{"1min":0.59799999999999998,"5min":0.504,"15min":0.48299999999999998},"max":{"1min":1.8799999999999999,"5min":1.8799999999999999,"15min":1.8799999999999999},"last":1.03},{"interface":"front","average":{"1min":0.95899999999999996,"5min":0.92400000000000004,"15min":0.93999999999999995},"min":{"1min":0.70299999999999996,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":1.1950000000000001,"5min":1.3320000000000001,"15min":1.554},"last":0.77300000000000002}]},{"osd":2,"last update":"Mon May 22 15:46:20 2023","interfaces":[{"interface":"back","average":{"1min":1.05,"5min":0.97399999999999998,"15min":0.98999999999999999},"min":{"1min":0.55400000000000005,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":3.2429999999999999,"5min":3.2429999999999999,"15min":3.2429999999999999},"last":1.099},{"interface":"front","average":{"1min":1.032,"5min":1.0269999999999999,"15min":1.0429999999999999},"min":{"1min":0.55300000000000005,"5min":0.5,"15min":0.442},"max":{"1min":3.3969999999999998,"5min":3.3969999999999998,"15min":3.3969999999999998},"last":1.1220000000000001}]},{"osd":3,"last update":"Mon May 22 15:46:20 2023","interfaces":[{"interface":"back","average":{"1min":0.88900000000000001,"5min":0.94599999999999995,"15min":0.97899999999999998},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":1.2470000000000001,"5min":1.5780000000000001,"15min":1.679},"last":1.155},{"interface":"front","average":{"1min":0.93500000000000005,"5min":0.95099999999999996,"15min":0.96299999999999997},"min":{"1min":0.69099999999999995,"5min":0.53200000000000003,"15min":0.50800000000000001},"max":{"1min":1.2430000000000001,"5min":1.383,"15min":1.4570000000000001},"last":1.006}]},{"osd":4,"last update":"Mon May 22 15:46:20 2023","interfaces":[{"interface":"back","average":{"1min":0.91300000000000003,"5min":0.93200000000000005,"15min":0.92500000000000004},"min":{"1min":0.61699999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":1.3169999999999999,"5min":1.581,"15min":1.6859999999999999},"last":0.91700000000000004},{"interface":"front","average":{"1min":0.93200000000000005,"5min":1.0209999999999999,"15min":1.0169999999999999},"min":{"1min":0.65500000000000003,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":1.242,"5min":1.452,"15min":1.8300000000000001},"last":0.88500000000000001}]},{"osd":5,"last update":"Mon May 22 15:46:20 2023","interfaces":[{"interface":"back","average":{"1min":0.98799999999999999,"5min":1.0169999999999999,"15min":0.98899999999999999},"min":{"1min":0.67600000000000005,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":1.3200000000000001,"5min":1.4530000000000001,"15min":1.6499999999999999},"last":0.85399999999999998},{"interface":"front","average":{"1min":1.1459999999999999,"5min":1.141,"15min":1.1120000000000001},"min":{"1min":0.81999999999999995,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.9990000000000001,"5min":1.9990000000000001,"15min":1.9990000000000001},"last":0.97799999999999998}]},{"osd":7,"last update":"Mon May 22 15:46:20 2023","interfaces":[{"interface":"back","average":{"1min":1.0860000000000001,"5min":1.131,"15min":1.125},"min":{"1min":0.85499999999999998,"5min":0.37,"15min":0.37},"max":{"1min":1.5529999999999999,"5min":1.853,"15min":1.9179999999999999},"last":1.23},{"interface":"front","average":{"1min":1.131,"5min":1.133,"15min":1.135},"min":{"1min":0.80300000000000005,"5min":0.442,"15min":0.442},"max":{"1min":1.579,"5min":1.579,"15min":1.8520000000000001},"last":0.95199999999999996}]}]},{"osd":5,"up_from":13,"seq":55834575085,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":26012,"kb_used_data":1748,"kb_used_omap":0,"kb_used_meta":24256,"kb_avail":94345828,"statfs":{"total":96636764160,"available":96610127872,"internally_reserved":0,"allocated":1789952,"data_stored":945177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":24838144},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 15:46:10 2023","interfaces":[{"interface":"back","average":{"1min":0.90900000000000003,"5min":0.85599999999999998,"15min":0.88300000000000001},"min":{"1min":0.61399999999999999,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.2889999999999999,"5min":1.3140000000000001,"15min":1.3440000000000001},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.91800000000000004,"5min":0.88600000000000001,"15min":0.88500000000000001},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":1.2709999999999999,"5min":1.3899999999999999,"15min":2.1699999999999999},"last":0.72599999999999998}]},{"osd":1,"last update":"Mon May 22 15:46:10 2023","interfaces":[{"interface":"back","average":{"1min":0.94999999999999996,"5min":0.96099999999999997,"15min":0.995},"min":{"1min":0.64800000000000002,"5min":0.52400000000000002,"15min":0.51000000000000001},"max":{"1min":1.3109999999999999,"5min":1.337,"15min":1.9370000000000001},"last":0.78000000000000003},{"interface":"front","average":{"1min":1.073,"5min":1.008,"15min":1.026},"min":{"1min":0.501,"5min":0.501,"15min":0.501},"max":{"1min":1.3089999999999999,"5min":1.6970000000000001,"15min":1.6970000000000001},"last":0.96899999999999997}]},{"osd":2,"last update":"Mon May 22 15:46:10 2023","interfaces":[{"interface":"back","average":{"1min":0.89100000000000001,"5min":0.91600000000000004,"15min":0.93400000000000005},"min":{"1min":0.505,"5min":0.505,"15min":0.504},"max":{"1min":1.343,"5min":1.5549999999999999,"15min":1.9830000000000001},"last":1.0549999999999999},{"interface":"front","average":{"1min":0.91800000000000004,"5min":0.94999999999999996,"15min":0.95199999999999996},"min":{"1min":0.56100000000000005,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.202,"5min":1.321,"15min":2.1120000000000001},"last":1.0169999999999999}]},{"osd":3,"last update":"Mon May 22 15:46:10 2023","interfaces":[{"interface":"back","average":{"1min":0.92900000000000005,"5min":0.92000000000000004,"15min":0.94899999999999995},"min":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.48199999999999998},"max":{"1min":1.206,"5min":1.2809999999999999,"15min":1.4419999999999999},"last":0.88300000000000001},{"interface":"front","average":{"1min":1.028,"5min":1.008,"15min":1.036},"min":{"1min":0.626,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":1.2490000000000001,"5min":1.371,"15min":2.52},"last":1.161}]},{"osd":4,"last update":"Mon May 22 15:46:10 2023","interfaces":[{"interface":"back","average":{"1min":0.96999999999999997,"5min":0.95699999999999996,"15min":0.95199999999999996},"min":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":1.405,"5min":1.405,"15min":2.226},"last":0.93500000000000005},{"interface":"front","average":{"1min":1.0629999999999999,"5min":1.024,"15min":1.0309999999999999},"min":{"1min":0.76000000000000001,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.3740000000000001,"5min":1.863,"15min":1.863},"last":1.1120000000000001}]},{"osd":6,"last update":"Mon May 22 15:46:10 2023","interfaces":[{"interface":"back","average":{"1min":1.008,"5min":1.024,"15min":1.0349999999999999},"min":{"1min":0.51900000000000002,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.397,"5min":1.4330000000000001,"15min":2.831},"last":1.246},{"interface":"front","average":{"1min":1.0369999999999999,"5min":1.0589999999999999,"15min":1.071},"min":{"1min":0.67000000000000004,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":1.4239999999999999,"5min":1.4239999999999999,"15min":2.3690000000000002},"last":1.296}]},{"osd":7,"last update":"Mon May 22 15:46:10 2023","interfaces":[{"interface":"back","average":{"1min":1.081,"5min":1.0900000000000001,"15min":1.1120000000000001},"min":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.42499999999999999},"max":{"1min":1.4359999999999999,"5min":1.47,"15min":1.849},"last":1.206},{"interface":"front","average":{"1min":1.1599999999999999,"5min":1.149,"15min":1.1659999999999999},"min":{"1min":0.86899999999999999,"5min":0.64000000000000001,"15min":0.64000000000000001},"max":{"1min":1.4339999999999999,"5min":1.575,"15min":2.7429999999999999},"last":1.3360000000000001}]}]},{"osd":4,"up_from":13,"seq":55834575083,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31592,"kb_used_data":1760,"kb_used_omap":0,"kb_used_meta":29824,"kb_avail":94340248,"statfs":{"total":96636764160,"available":96604413952,"internally_reserved":0,"allocated":1802240,"data_stored":952153,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":30539776},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 15:46:19 2023","interfaces":[{"interface":"back","average":{"1min":0.93200000000000005,"5min":0.86799999999999999,"15min":0.90500000000000003},"min":{"1min":0.63400000000000001,"5min":0.48999999999999999,"15min":0.34100000000000003},"max":{"1min":1.2629999999999999,"5min":1.4119999999999999,"15min":1.9910000000000001},"last":0.66600000000000004},{"interface":"front","average":{"1min":0.91300000000000003,"5min":0.90000000000000002,"15min":0.92200000000000004},"min":{"1min":0.51100000000000001,"5min":0.499,"15min":0.373},"max":{"1min":1.284,"5min":1.369,"15min":1.3779999999999999},"last":0.81000000000000005}]},{"osd":1,"last update":"Mon May 22 15:46:19 2023","interfaces":[{"interface":"back","average":{"1min":0.94799999999999995,"5min":0.91800000000000004,"15min":0.96399999999999997},"min":{"1min":0.60499999999999998,"5min":0.436,"15min":0.436},"max":{"1min":1.1970000000000001,"5min":1.3899999999999999,"15min":1.3899999999999999},"last":0.72099999999999997},{"interface":"front","average":{"1min":0.91700000000000004,"5min":0.84499999999999997,"15min":0.90100000000000002},"min":{"1min":0.56699999999999995,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.157,"5min":1.2889999999999999,"15min":1.3640000000000001},"last":0.97799999999999998}]},{"osd":2,"last update":"Mon May 22 15:46:19 2023","interfaces":[{"interface":"back","average":{"1min":0.90100000000000002,"5min":0.91600000000000004,"15min":0.94199999999999995},"min":{"1min":0.56299999999999994,"5min":0.50800000000000001,"15min":0.495},"max":{"1min":1.244,"5min":1.96,"15min":2.1019999999999999},"last":0.76500000000000001},{"interface":"front","average":{"1min":0.97799999999999998,"5min":0.98399999999999999,"15min":0.99199999999999999},"min":{"1min":0.58099999999999996,"5min":0.48799999999999999,"15min":0.48399999999999999},"max":{"1min":1.329,"5min":1.4399999999999999,"15min":1.4399999999999999},"last":0.60299999999999998}]},{"osd":3,"last update":"Mon May 22 15:46:19 2023","interfaces":[{"interface":"back","average":{"1min":0.91100000000000003,"5min":0.90200000000000002,"15min":0.93200000000000005},"min":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":1.224,"5min":1.28,"15min":1.3169999999999999},"last":1.0580000000000001},{"interface":"front","average":{"1min":1.0469999999999999,"5min":1.014,"15min":1.0209999999999999},"min":{"1min":0.72199999999999998,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":1.3420000000000001,"5min":1.4179999999999999,"15min":2.1720000000000002},"last":0.85599999999999998}]},{"osd":5,"last update":"Mon May 22 15:46:19 2023","interfaces":[{"interface":"back","average":{"1min":1.0209999999999999,"5min":0.97199999999999998,"15min":0.97699999999999998},"min":{"1min":0.55000000000000004,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.357,"5min":1.373,"15min":1.472},"last":1.0369999999999999},{"interface":"front","average":{"1min":1.0389999999999999,"5min":0.98999999999999999,"15min":0.97499999999999998},"min":{"1min":0.59999999999999998,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.262,"5min":1.365,"15min":1.4139999999999999},"last":1.155}]},{"osd":6,"last update":"Mon May 22 15:46:19 2023","interfaces":[{"interface":"back","average":{"1min":1.0649999999999999,"5min":0.99299999999999999,"15min":0.98499999999999999},"min":{"1min":0.72099999999999997,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":1.456,"5min":1.456,"15min":2.2349999999999999},"last":0.93500000000000005},{"interface":"front","average":{"1min":1.131,"5min":1.103,"15min":1.1100000000000001},"min":{"1min":0.66500000000000004,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":1.425,"5min":1.486,"15min":1.8220000000000001},"last":0.89800000000000002}]},{"osd":7,"last update":"Mon May 22 15:46:19 2023","interfaces":[{"interface":"back","average":{"1min":1.137,"5min":1.0629999999999999,"15min":1.052},"min":{"1min":0.85699999999999998,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":1.389,"5min":1.46,"15min":1.5800000000000001},"last":1.109},{"interface":"front","average":{"1min":1.1930000000000001,"5min":1.167,"15min":1.143},"min":{"1min":0.78900000000000003,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":1.5620000000000001,"5min":1.6559999999999999,"15min":2.306},"last":1.018}]}]},{"osd":3,"up_from":13,"seq":55834575083,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36956,"kb_used_data":1748,"kb_used_omap":0,"kb_used_meta":35200,"kb_avail":94334884,"statfs":{"total":96636764160,"available":96598921216,"internally_reserved":0,"allocated":1789952,"data_stored":945177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":36044800},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 15:46:03 2023","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.61699999999999999,"15min":0.63600000000000001},"min":{"1min":0.50700000000000001,"5min":0.252,"15min":0.19},"max":{"1min":1.169,"5min":1.169,"15min":1.194},"last":0.64600000000000002},{"interface":"front","average":{"1min":0.746,"5min":0.68500000000000005,"15min":0.70899999999999996},"min":{"1min":0.48899999999999999,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":1.085,"5min":1.296,"15min":1.296},"last":0.83599999999999997}]},{"osd":1,"last update":"Mon May 22 15:46:03 2023","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.64200000000000002,"15min":0.66200000000000003},"min":{"1min":0.52100000000000002,"5min":0.314,"15min":0.29499999999999998},"max":{"1min":0.90600000000000003,"5min":1.2609999999999999,"15min":1.2609999999999999},"last":0.93000000000000005},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.72899999999999998,"15min":0.76800000000000002},"min":{"1min":0.47499999999999998,"5min":0.34399999999999997,"15min":0.313},"max":{"1min":1.081,"5min":1.1359999999999999,"15min":1.222},"last":0.60099999999999998}]},{"osd":2,"last update":"Mon May 22 15:46:03 2023","interfaces":[{"interface":"back","average":{"1min":0.84199999999999997,"5min":0.78700000000000003,"15min":0.81699999999999995},"min":{"1min":0.45600000000000002,"5min":0.40699999999999997,"15min":0.36599999999999999},"max":{"1min":1.0529999999999999,"5min":1.506,"15min":1.506},"last":0.75800000000000001},{"interface":"front","average":{"1min":0.81899999999999995,"5min":0.75600000000000001,"15min":0.80700000000000005},"min":{"1min":0.65300000000000002,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":1.032,"5min":1.147,"15min":1.359},"last":0.89400000000000002}]},{"osd":4,"last update":"Mon May 22 15:46:03 2023","interfaces":[{"interface":"back","average":{"1min":0.97499999999999998,"5min":0.92500000000000004,"15min":0.95599999999999996},"min":{"1min":0.71999999999999997,"5min":0.54500000000000004,"15min":0.48999999999999999},"max":{"1min":1.238,"5min":1.413,"15min":1.4470000000000001},"last":1.0569999999999999},{"interface":"front","average":{"1min":1.0529999999999999,"5min":0.97499999999999998,"15min":1.0029999999999999},"min":{"1min":0.83299999999999996,"5min":0.54400000000000004,"15min":0.49199999999999999},"max":{"1min":1.407,"5min":1.407,"15min":1.4470000000000001},"last":1.0289999999999999}]},{"osd":5,"last update":"Mon May 22 15:46:03 2023","interfaces":[{"interface":"back","average":{"1min":1.0509999999999999,"5min":0.98399999999999999,"15min":1.016},"min":{"1min":0.80900000000000005,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.3149999999999999,"5min":1.5089999999999999,"15min":1.5089999999999999},"last":0.95899999999999996},{"interface":"front","average":{"1min":1.0349999999999999,"5min":1.006,"15min":1.038},"min":{"1min":0.90200000000000002,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.2130000000000001,"5min":1.4299999999999999,"15min":1.4359999999999999},"last":1.347}]},{"osd":6,"last update":"Mon May 22 15:46:03 2023","interfaces":[{"interface":"back","average":{"1min":1.0580000000000001,"5min":1.0129999999999999,"15min":1.042},"min":{"1min":0.873,"5min":0.47699999999999998,"15min":0.46999999999999997},"max":{"1min":1.2989999999999999,"5min":1.4570000000000001,"15min":1.49},"last":1.101},{"interface":"front","average":{"1min":1.083,"5min":1.0269999999999999,"15min":1.0700000000000001},"min":{"1min":0.79800000000000004,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":1.2949999999999999,"5min":1.4330000000000001,"15min":1.5009999999999999},"last":1.403}]},{"osd":7,"last update":"Mon May 22 15:46:03 2023","interfaces":[{"interface":"back","average":{"1min":1.113,"5min":1.0780000000000001,"15min":1.1040000000000001},"min":{"1min":0.83599999999999997,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.3779999999999999,"5min":1.6719999999999999,"15min":1.6719999999999999},"last":1.454},{"interface":"front","average":{"1min":1.117,"5min":1.071,"15min":1.085},"min":{"1min":0.90200000000000002,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":1.361,"5min":1.8700000000000001,"15min":1.8700000000000001},"last":1.0029999999999999}]}]},{"osd":2,"up_from":13,"seq":55834575084,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":18780,"kb_used_data":1748,"kb_used_omap":0,"kb_used_meta":17024,"kb_avail":94353060,"statfs":{"total":96636764160,"available":96617533440,"internally_reserved":0,"allocated":1789952,"data_stored":945177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":17432576},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 15:46:03 2023","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.64400000000000002,"15min":0.65600000000000003},"min":{"1min":0.28699999999999998,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":1.097,"5min":1.1539999999999999,"15min":1.8240000000000001},"last":0.70299999999999996},{"interface":"front","average":{"1min":0.69899999999999995,"5min":0.70499999999999996,"15min":0.746},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":1.079,"5min":1.079,"15min":1.8160000000000001},"last":0.46300000000000002}]},{"osd":1,"last update":"Mon May 22 15:46:03 2023","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.72999999999999998,"15min":0.74399999999999999},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.27000000000000002},"max":{"1min":1.196,"5min":1.196,"15min":1.2},"last":0.92000000000000004},{"interface":"front","average":{"1min":0.754,"5min":0.75900000000000001,"15min":0.80800000000000005},"min":{"1min":0.32400000000000001,"5min":0.317,"15min":0.317},"max":{"1min":1.0309999999999999,"5min":1.246,"15min":2.5089999999999999},"last":0.52500000000000002}]},{"osd":3,"last update":"Mon May 22 15:46:03 2023","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":0.69399999999999995,"15min":0.72599999999999998},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.96199999999999997,"5min":1.077,"15min":2.7629999999999999},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.82699999999999996,"5min":0.84599999999999997,"15min":0.88300000000000001},"min":{"1min":0.39600000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":1.1699999999999999,"5min":1.256,"15min":2.6840000000000002},"last":0.85699999999999998}]},{"osd":4,"last update":"Mon May 22 15:46:03 2023","interfaces":[{"interface":"back","average":{"1min":0.97499999999999998,"5min":0.98499999999999999,"15min":1.018},"min":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":1.274,"5min":1.536,"15min":2.8039999999999998},"last":1.1459999999999999},{"interface":"front","average":{"1min":1.0069999999999999,"5min":1.0009999999999999,"15min":1.028},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.2509999999999999,"5min":1.4299999999999999,"15min":2.0819999999999999},"last":0.81799999999999995}]},{"osd":5,"last update":"Mon May 22 15:46:03 2023","interfaces":[{"interface":"back","average":{"1min":1.0569999999999999,"5min":1.056,"15min":1.052},"min":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.3959999999999999,"5min":1.5049999999999999,"15min":2.3860000000000001},"last":1.175},{"interface":"front","average":{"1min":0.99099999999999999,"5min":0.97399999999999998,"15min":0.998},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.331,"5min":1.331,"15min":3.0299999999999998},"last":0.78100000000000003}]},{"osd":6,"last update":"Mon May 22 15:46:03 2023","interfaces":[{"interface":"back","average":{"1min":1.038,"5min":1.012,"15min":1.036},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":1.3320000000000001,"5min":1.49,"15min":2.883},"last":1.1020000000000001},{"interface":"front","average":{"1min":1.129,"5min":1.081,"15min":1.1040000000000001},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":1.4450000000000001,"5min":1.5660000000000001,"15min":3.1459999999999999},"last":1.0209999999999999}]},{"osd":7,"last update":"Mon May 22 15:46:03 2023","interfaces":[{"interface":"back","average":{"1min":1.1639999999999999,"5min":1.097,"15min":1.1279999999999999},"min":{"1min":0.55000000000000004,"5min":0.499,"15min":0.499},"max":{"1min":1.429,"5min":1.55,"15min":2.5990000000000002},"last":1.052},{"interface":"front","average":{"1min":1.1240000000000001,"5min":1.0640000000000001,"15min":1.1060000000000001},"min":{"1min":0.5,"5min":0.5,"15min":0.5},"max":{"1min":1.4650000000000001,"5min":1.4650000000000001,"15min":3.8799999999999999},"last":0.97699999999999998}]}]},{"osd":1,"up_from":13,"seq":55834575084,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29800,"kb_used_data":1760,"kb_used_omap":0,"kb_used_meta":28032,"kb_avail":94342040,"statfs":{"total":96636764160,"available":96606248960,"internally_reserved":0,"allocated":1802240,"data_stored":952153,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":28704768},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 15:46:17 2023","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.63200000000000001,"15min":0.63400000000000001},"min":{"1min":0.40600000000000003,"5min":0.26300000000000001,"15min":0.22700000000000001},"max":{"1min":1.1200000000000001,"5min":1.2270000000000001,"15min":1.2270000000000001},"last":0.98699999999999999},{"interface":"front","average":{"1min":0.77100000000000002,"5min":0.70699999999999996,"15min":0.76400000000000001},"min":{"1min":0.35999999999999999,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":1.1839999999999999,"5min":1.1839999999999999,"15min":1.2529999999999999},"last":0.58699999999999997}]},{"osd":2,"last update":"Mon May 22 15:46:17 2023","interfaces":[{"interface":"back","average":{"1min":0.76400000000000001,"5min":0.71399999999999997,"15min":0.73799999999999999},"min":{"1min":0.44800000000000001,"5min":0.317,"15min":0.28999999999999998},"max":{"1min":1.0660000000000001,"5min":1.391,"15min":1.391},"last":0.73899999999999999},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.72999999999999998,"15min":0.76300000000000001},"min":{"1min":0.49199999999999999,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.98699999999999999,"5min":1.2270000000000001,"15min":1.276},"last":0.77100000000000002}]},{"osd":3,"last update":"Mon May 22 15:46:17 2023","interfaces":[{"interface":"back","average":{"1min":0.80100000000000005,"5min":0.79900000000000004,"15min":0.82799999999999996},"min":{"1min":0.51200000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.153,"5min":1.657,"15min":1.657},"last":0.93200000000000005},{"interface":"front","average":{"1min":0.81100000000000005,"5min":0.80800000000000005,"15min":0.81499999999999995},"min":{"1min":0.33700000000000002,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":1.095,"5min":2.0569999999999999,"15min":2.0569999999999999},"last":0.876}]},{"osd":4,"last update":"Mon May 22 15:46:17 2023","interfaces":[{"interface":"back","average":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.96499999999999997},"min":{"1min":0.64500000000000002,"5min":0.55200000000000005,"15min":0.34000000000000002},"max":{"1min":1.502,"5min":2.9529999999999998,"15min":2.9529999999999998},"last":1.2450000000000001},{"interface":"front","average":{"1min":0.97499999999999998,"5min":0.999,"15min":1.002},"min":{"1min":0.61299999999999999,"5min":0.504,"15min":0.49299999999999999},"max":{"1min":1.292,"5min":2.7759999999999998,"15min":2.7759999999999998},"last":1.0469999999999999}]},{"osd":5,"last update":"Mon May 22 15:46:17 2023","interfaces":[{"interface":"back","average":{"1min":1.0349999999999999,"5min":1.006,"15min":1.034},"min":{"1min":0.627,"5min":0.58399999999999996,"15min":0.443},"max":{"1min":1.556,"5min":1.9570000000000001,"15min":1.9570000000000001},"last":1.0269999999999999},{"interface":"front","average":{"1min":1.1060000000000001,"5min":1.034,"15min":1.0449999999999999},"min":{"1min":0.59399999999999997,"5min":0.53400000000000003,"15min":0.47299999999999998},"max":{"1min":1.577,"5min":2.1200000000000001,"15min":2.1200000000000001},"last":1.177}]},{"osd":6,"last update":"Mon May 22 15:46:17 2023","interfaces":[{"interface":"back","average":{"1min":1.02,"5min":1.0369999999999999,"15min":1.036},"min":{"1min":0.65900000000000003,"5min":0.54000000000000004,"15min":0.45300000000000001},"max":{"1min":1.3180000000000001,"5min":2.831,"15min":2.831},"last":1.1399999999999999},{"interface":"front","average":{"1min":1.1499999999999999,"5min":1.085,"15min":1.0940000000000001},"min":{"1min":0.55800000000000005,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":1.6299999999999999,"5min":1.7629999999999999,"15min":1.7629999999999999},"last":1.0920000000000001}]},{"osd":7,"last update":"Mon May 22 15:46:17 2023","interfaces":[{"interface":"back","average":{"1min":1.1759999999999999,"5min":1.135,"15min":1.1180000000000001},"min":{"1min":0.61299999999999999,"5min":0.60299999999999998,"15min":0.56799999999999995},"max":{"1min":1.679,"5min":2.4199999999999999,"15min":2.4199999999999999},"last":1.1299999999999999},{"interface":"front","average":{"1min":1.1819999999999999,"5min":1.1259999999999999,"15min":1.1299999999999999},"min":{"1min":0.76300000000000001,"5min":0.47299999999999998,"15min":0.41599999999999998},"max":{"1min":1.619,"5min":3.8799999999999999,"15min":3.8799999999999999},"last":1.337}]}]},{"osd":0,"up_from":13,"seq":55834575085,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":25376,"kb_used_data":2328,"kb_used_omap":0,"kb_used_meta":23040,"kb_avail":94346464,"statfs":{"total":96636764160,"available":96610779136,"internally_reserved":0,"allocated":2383872,"data_stored":1535545,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":23592960},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon May 22 15:46:21 2023","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.65900000000000003,"15min":0.68100000000000005},"min":{"1min":0.307,"5min":0.248,"15min":0.223},"max":{"1min":1.2190000000000001,"5min":1.466,"15min":1.466},"last":0.78400000000000003},{"interface":"front","average":{"1min":0.69899999999999995,"5min":0.70599999999999996,"15min":0.72899999999999998},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.24099999999999999},"max":{"1min":1.2230000000000001,"5min":1.71,"15min":1.71},"last":0.55200000000000005}]},{"osd":2,"last update":"Mon May 22 15:46:21 2023","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":0.70699999999999996,"15min":0.74099999999999999},"min":{"1min":0.39700000000000002,"5min":0.318,"15min":0.26500000000000001},"max":{"1min":0.96799999999999997,"5min":1.165,"15min":1.3220000000000001},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.82799999999999996,"5min":0.75900000000000001,"15min":0.78800000000000003},"min":{"1min":0.375,"5min":0.34999999999999998,"15min":0.32900000000000001},"max":{"1min":1.1699999999999999,"5min":1.3129999999999999,"15min":1.3129999999999999},"last":0.747}]},{"osd":3,"last update":"Mon May 22 15:46:21 2023","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":0.74099999999999999,"15min":0.76000000000000001},"min":{"1min":0.35499999999999998,"5min":0.31,"15min":0.31},"max":{"1min":0.96199999999999997,"5min":1.9299999999999999,"15min":1.9299999999999999},"last":0.88600000000000001},{"interface":"front","average":{"1min":0.83499999999999996,"5min":0.83299999999999996,"15min":0.871},"min":{"1min":0.35599999999999998,"5min":0.35199999999999998,"15min":0.34399999999999997},"max":{"1min":1.1299999999999999,"5min":1.758,"15min":1.758},"last":0.85999999999999999}]},{"osd":4,"last update":"Mon May 22 15:46:21 2023","interfaces":[{"interface":"back","average":{"1min":0.92800000000000005,"5min":0.95099999999999996,"15min":0.95099999999999996},"min":{"1min":0.57599999999999996,"5min":0.57399999999999995,"15min":0.48099999999999998},"max":{"1min":1.149,"5min":2.923,"15min":2.923},"last":1.091},{"interface":"front","average":{"1min":0.98999999999999999,"5min":1.006,"15min":1.026},"min":{"1min":0.624,"5min":0.624,"15min":0.624},"max":{"1min":1.3839999999999999,"5min":2.0129999999999999,"15min":2.0129999999999999},"last":0.93600000000000005}]},{"osd":5,"last update":"Mon May 22 15:46:21 2023","interfaces":[{"interface":"back","average":{"1min":1,"5min":1.004,"15min":1.0409999999999999},"min":{"1min":0.59099999999999997,"5min":0.56000000000000005,"15min":0.46000000000000002},"max":{"1min":1.238,"5min":1.97,"15min":1.97},"last":1.006},{"interface":"front","average":{"1min":1.02,"5min":1.016,"15min":1.008},"min":{"1min":0.67600000000000005,"5min":0.56599999999999995,"15min":0.55300000000000005},"max":{"1min":1.5349999999999999,"5min":5.0519999999999996,"15min":5.0519999999999996},"last":0.98199999999999998}]},{"osd":6,"last update":"Mon May 22 15:46:21 2023","interfaces":[{"interface":"back","average":{"1min":1.044,"5min":1.0589999999999999,"15min":1.089},"min":{"1min":0.58899999999999997,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":1.306,"5min":2.117,"15min":2.117},"last":1.1439999999999999},{"interface":"front","average":{"1min":1.085,"5min":1.1299999999999999,"15min":1.119},"min":{"1min":0.69199999999999995,"5min":0.65500000000000003,"15min":0.499},"max":{"1min":1.3100000000000001,"5min":5.2640000000000002,"15min":5.2640000000000002},"last":1.1910000000000001}]},{"osd":7,"last update":"Mon May 22 15:46:21 2023","interfaces":[{"interface":"back","average":{"1min":1.075,"5min":1.1040000000000001,"15min":1.129},"min":{"1min":0.63800000000000001,"5min":0.61299999999999999,"15min":0.61299999999999999},"max":{"1min":1.3640000000000001,"5min":2.0550000000000002,"15min":2.0550000000000002},"last":1.0680000000000001},{"interface":"front","average":{"1min":1.089,"5min":1.0900000000000001,"15min":1.0960000000000001},"min":{"1min":0.73099999999999998,"5min":0.63300000000000001,"15min":0.60299999999999998},"max":{"1min":1.6240000000000001,"5min":3.9060000000000001,"15min":3.9060000000000001},"last":1.0409999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-05-22T15:46:27.594 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-05-22T15:46:27.755 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.751+0000 7f6601209640 1 Processor -- start 2023-05-22T15:46:27.755 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.751+0000 7f6601209640 1 -- start start 2023-05-22T15:46:27.756 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.751+0000 7f6601209640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f65fc0b0250 0x7f65fc159330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:27.756 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.751+0000 7f6601209640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f65fc159870 0x7f65fc0c8ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:27.756 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.751+0000 7f6601209640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65fc15a0a0 con 0x7f65fc1531b0 2023-05-22T15:46:27.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.751+0000 7f6601209640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65fc153580 con 0x7f65fc159870 2023-05-22T15:46:27.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.751+0000 7f6601209640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65fc15d5e0 con 0x7f65fc0b0250 2023-05-22T15:46:27.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.751+0000 7f65fad9d640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f65fc1531b0 0x7f65fc0afb40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:38870/0 (socket says 172.21.15.16:38870) 2023-05-22T15:46:27.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.751+0000 7f65fad9d640 1 -- 172.21.15.16:0/2798601649 learned_addr learned my addr 172.21.15.16:0/2798601649 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:46:27.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.751+0000 7f65fb59e640 1 --2- 172.21.15.16:0/2798601649 >> v2:172.21.15.16:3300/0 conn(0x7f65fc159870 0x7f65fc0c8ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:27.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.751+0000 7f65fa59c640 1 --2- 172.21.15.16:0/2798601649 >> v2:172.21.15.62:3300/0 conn(0x7f65fc0b0250 0x7f65fc159330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:27.758 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.751+0000 7f65fb59e640 1 -- 172.21.15.16:0/2798601649 >> v2:172.21.15.62:3300/0 conn(0x7f65fc0b0250 msgr2=0x7f65fc159330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:46:27.758 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.751+0000 7f65fb59e640 1 --2- 172.21.15.16:0/2798601649 >> v2:172.21.15.62:3300/0 conn(0x7f65fc0b0250 0x7f65fc159330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:27.758 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.751+0000 7f65fb59e640 1 -- 172.21.15.16:0/2798601649 >> v1:172.21.15.16:6789/0 conn(0x7f65fc1531b0 legacy=0x7f65fc0afb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:27.758 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.751+0000 7f65fb59e640 1 -- 172.21.15.16:0/2798601649 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65fc15c820 con 0x7f65fc159870 2023-05-22T15:46:27.759 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.751+0000 7f65f9d9b640 1 -- 172.21.15.16:0/2798601649 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1154841494 0 0) 0x7f65fc15a0a0 con 0x7f65fc1531b0 2023-05-22T15:46:27.759 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.755+0000 7f65fb59e640 1 --2- 172.21.15.16:0/2798601649 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f65fc159870 0x7f65fc0c8ba0 secure :-1 s=READY pgs=1843 cs=0 l=1 rev1=1 crypto rx=0x7f65fc06d880 tx=0x7f65f401c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45b5d8003419be1a server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:27.759 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.755+0000 7f65f9d9b640 1 -- 172.21.15.16:0/2798601649 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65f40a59a0 con 0x7f65fc159870 2023-05-22T15:46:27.759 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.755+0000 7f65f9d9b640 1 -- 172.21.15.16:0/2798601649 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65f40a5b00 con 0x7f65fc159870 2023-05-22T15:46:27.760 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.755+0000 7f65f9d9b640 1 -- 172.21.15.16:0/2798601649 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65f40a5e90 con 0x7f65fc159870 2023-05-22T15:46:27.760 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.755+0000 7f6601209640 1 -- 172.21.15.16:0/2798601649 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f65fc159870 msgr2=0x7f65fc0c8ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:27.760 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.755+0000 7f6601209640 1 --2- 172.21.15.16:0/2798601649 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f65fc159870 0x7f65fc0c8ba0 secure :-1 s=READY pgs=1843 cs=0 l=1 rev1=1 crypto rx=0x7f65fc06d880 tx=0x7f65f401c8e0 comp rx=0 tx=0).stop 2023-05-22T15:46:27.761 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.755+0000 7f6601209640 1 -- 172.21.15.16:0/2798601649 shutdown_connections 2023-05-22T15:46:27.761 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.755+0000 7f6601209640 1 --2- 172.21.15.16:0/2798601649 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f65fc159870 0x7f65fc0c8ba0 unknown :-1 s=CLOSED pgs=1843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:27.761 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.755+0000 7f6601209640 1 --2- 172.21.15.16:0/2798601649 >> v2:172.21.15.62:3300/0 conn(0x7f65fc0b0250 0x7f65fc159330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:27.761 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.755+0000 7f6601209640 1 -- 172.21.15.16:0/2798601649 >> 172.21.15.16:0/2798601649 conn(0x7f65fc069720 msgr2=0x7f65fc069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:46:27.761 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.755+0000 7f6601209640 1 -- 172.21.15.16:0/2798601649 shutdown_connections 2023-05-22T15:46:27.761 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.755+0000 7f6601209640 1 -- 172.21.15.16:0/2798601649 wait complete. 2023-05-22T15:46:27.763 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.755+0000 7f6601209640 1 Processor -- start 2023-05-22T15:46:27.763 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.755+0000 7f6601209640 1 -- start start 2023-05-22T15:46:27.763 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.755+0000 7f6601209640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f65fc0b0250 0x7f65fc0b2ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:27.763 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.755+0000 7f6601209640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f65fc1531b0 0x7f65fc14ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:27.763 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.755+0000 7f6601209640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f65fc159870 0x7f65fc14b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:27.764 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.755+0000 7f6601209640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65fc14b6b0 con 0x7f65fc0b0250 2023-05-22T15:46:27.764 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.755+0000 7f6601209640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65fc0b2770 con 0x7f65fc159870 2023-05-22T15:46:27.764 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.755+0000 7f6601209640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65fc0b2a50 con 0x7f65fc1531b0 2023-05-22T15:46:27.764 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.755+0000 7f65fad9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f65fc0b0250 0x7f65fc0b2ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:27.764 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.755+0000 7f65fad9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f65fc0b0250 0x7f65fc0b2ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34610/0 (socket says 172.21.15.16:34610) 2023-05-22T15:46:27.764 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.755+0000 7f65fad9d640 1 -- 172.21.15.16:0/235625735 learned_addr learned my addr 172.21.15.16:0/235625735 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:46:27.764 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.755+0000 7f65fb59e640 1 --2- 172.21.15.16:0/235625735 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f65fc159870 0x7f65fc14b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:27.765 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.755+0000 7f65fad9d640 1 -- 172.21.15.16:0/235625735 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f65fc1531b0 msgr2=0x7f65fc14ac30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:27.765 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.755+0000 7f65fad9d640 1 --2- 172.21.15.16:0/235625735 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f65fc1531b0 0x7f65fc14ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:27.765 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.755+0000 7f65fad9d640 1 -- 172.21.15.16:0/235625735 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f65fc159870 msgr2=0x7f65fc14b170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:27.765 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.755+0000 7f65fad9d640 1 --2- 172.21.15.16:0/235625735 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f65fc159870 0x7f65fc14b170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:27.765 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.755+0000 7f65fad9d640 1 -- 172.21.15.16:0/235625735 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65f4009a70 con 0x7f65fc0b0250 2023-05-22T15:46:27.765 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.759+0000 7f65fad9d640 1 --2- 172.21.15.16:0/235625735 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f65fc0b0250 0x7f65fc0b2ee0 secure :-1 s=READY pgs=1844 cs=0 l=1 rev1=1 crypto rx=0x7f65ec00c520 tx=0x7f65ec00c9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:27.765 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.759+0000 7f65e37fe640 1 -- 172.21.15.16:0/235625735 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65ec0074e0 con 0x7f65fc0b0250 2023-05-22T15:46:27.765 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.759+0000 7f65e37fe640 1 -- 172.21.15.16:0/235625735 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65ec007640 con 0x7f65fc0b0250 2023-05-22T15:46:27.766 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.759+0000 7f6601209640 1 -- 172.21.15.16:0/235625735 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65fc159cc0 con 0x7f65fc0b0250 2023-05-22T15:46:27.766 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.759+0000 7f65e37fe640 1 -- 172.21.15.16:0/235625735 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65ec0049f0 con 0x7f65fc0b0250 2023-05-22T15:46:27.766 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.759+0000 7f6601209640 1 -- 172.21.15.16:0/235625735 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65fc15d0c0 con 0x7f65fc0b0250 2023-05-22T15:46:27.767 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.759+0000 7f6601209640 1 -- 172.21.15.16:0/235625735 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65fc159e50 con 0x7f65fc0b0250 2023-05-22T15:46:27.767 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.759+0000 7f65e37fe640 1 -- 172.21.15.16:0/235625735 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 60) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f65ec004b50 con 0x7f65fc0b0250 2023-05-22T15:46:27.767 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.759+0000 7f65e37fe640 1 --2- 172.21.15.16:0/235625735 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f65e80654a0 0x7f65e8067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:27.767 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.759+0000 7f65e37fe640 1 -- 172.21.15.16:0/235625735 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(147..147 src has 1..147) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f65ec081eb0 con 0x7f65fc0b0250 2023-05-22T15:46:27.767 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.759+0000 7f65fa59c640 1 --2- 172.21.15.16:0/235625735 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f65e80654a0 0x7f65e8067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:27.767 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.759+0000 7f65fa59c640 1 --2- 172.21.15.16:0/235625735 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f65e80654a0 0x7f65e8067950 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f65fc14d610 tx=0x7f65f0008040 comp rx=0 tx=0).ready entity=mgr.8835 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:27.769 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.763+0000 7f65e37fe640 1 -- 172.21.15.16:0/235625735 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f65ec04f820 con 0x7f65fc0b0250 2023-05-22T15:46:27.919 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.915+0000 7f6601209640 1 -- 172.21.15.16:0/235625735 --> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f65fc0b2b90 con 0x7f65e80654a0 2023-05-22T15:46:27.928 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.923+0000 7f65e37fe640 1 -- 172.21.15.16:0/235625735 <== mgr.8835 v2:172.21.15.16:6832/36921 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+307001 (secure 0 0 0) 0x7f65f000be00 con 0x7f65e80654a0 2023-05-22T15:46:27.931 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:46:27.932 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2023-05-22T15:46:27.933 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.927+0000 7f6601209640 1 -- 172.21.15.16:0/235625735 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f65e80654a0 msgr2=0x7f65e8067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:27.933 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.927+0000 7f6601209640 1 --2- 172.21.15.16:0/235625735 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f65e80654a0 0x7f65e8067950 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f65fc14d610 tx=0x7f65f0008040 comp rx=0 tx=0).stop 2023-05-22T15:46:27.933 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.927+0000 7f6601209640 1 -- 172.21.15.16:0/235625735 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f65fc0b0250 msgr2=0x7f65fc0b2ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:27.934 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.927+0000 7f6601209640 1 --2- 172.21.15.16:0/235625735 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f65fc0b0250 0x7f65fc0b2ee0 secure :-1 s=READY pgs=1844 cs=0 l=1 rev1=1 crypto rx=0x7f65ec00c520 tx=0x7f65ec00c9e0 comp rx=0 tx=0).stop 2023-05-22T15:46:27.934 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.927+0000 7f6601209640 1 -- 172.21.15.16:0/235625735 shutdown_connections 2023-05-22T15:46:27.934 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.927+0000 7f6601209640 1 --2- 172.21.15.16:0/235625735 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f65fc159870 0x7f65fc14b170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:27.934 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.927+0000 7f6601209640 1 --2- 172.21.15.16:0/235625735 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f65e80654a0 0x7f65e8067950 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:27.934 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.927+0000 7f6601209640 1 --2- 172.21.15.16:0/235625735 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f65fc1531b0 0x7f65fc14ac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:27.934 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.927+0000 7f6601209640 1 --2- 172.21.15.16:0/235625735 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f65fc0b0250 0x7f65fc0b2ee0 unknown :-1 s=CLOSED pgs=1844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:27.934 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.927+0000 7f6601209640 1 -- 172.21.15.16:0/235625735 >> 172.21.15.16:0/235625735 conn(0x7f65fc069720 msgr2=0x7f65fc06b730 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:46:27.934 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.927+0000 7f6601209640 1 -- 172.21.15.16:0/235625735 shutdown_connections 2023-05-22T15:46:27.935 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:27.927+0000 7f6601209640 1 -- 172.21.15.16:0/235625735 wait complete. 2023-05-22T15:46:27.949 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":26,"stamp":"2023-05-22T15:46:27.369508+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1268,"num_read_kb":2572,"num_write":399,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":249,"ondisk_log_size":249,"up":210,"acting":210,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":33,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":224768,"kb_used_data":15168,"kb_used_omap":0,"kb_used_meta":209536,"kb_avail":754749952,"statfs":{"total":773094113280,"available":772863950848,"internally_reserved":0,"allocated":15532032,"data_stored":8756104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":214564864},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":62,"apply_latency_ms":62,"commit_latency_ns":62000000,"apply_latency_ns":62000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.321201"},"pg_stats":[{"pgid":"19.3f","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.079122+0000","last_change":"2023-05-22T15:46:23.079122+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.079122+0000","last_undegraded":"2023-05-22T15:46:23.079122+0000","last_fullsized":"2023-05-22T15:46:23.079122+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:57:19.852836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.3e","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.076282+0000","last_change":"2023-05-22T15:46:23.076282+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.076282+0000","last_undegraded":"2023-05-22T15:46:23.076282+0000","last_fullsized":"2023-05-22T15:46:23.076282+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T03:20:39.433281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.3d","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.072443+0000","last_change":"2023-05-22T15:46:23.072443+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.072443+0000","last_undegraded":"2023-05-22T15:46:23.072443+0000","last_fullsized":"2023-05-22T15:46:23.072443+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:46:28.227459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.3c","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.077931+0000","last_change":"2023-05-22T15:46:23.077931+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.077931+0000","last_undegraded":"2023-05-22T15:46:23.077931+0000","last_fullsized":"2023-05-22T15:46:23.077931+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:42:45.119566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.3b","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.090321+0000","last_change":"2023-05-22T15:46:23.090321+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.090321+0000","last_undegraded":"2023-05-22T15:46:23.090321+0000","last_fullsized":"2023-05-22T15:46:23.090321+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:59:24.451915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.3a","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.072441+0000","last_change":"2023-05-22T15:46:23.072441+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.072441+0000","last_undegraded":"2023-05-22T15:46:23.072441+0000","last_fullsized":"2023-05-22T15:46:23.072441+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:25:55.107784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.39","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.076980+0000","last_change":"2023-05-22T15:46:23.076980+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.076980+0000","last_undegraded":"2023-05-22T15:46:23.076980+0000","last_fullsized":"2023-05-22T15:46:23.076980+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:29:20.717884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.38","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.078358+0000","last_change":"2023-05-22T15:46:23.078358+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.078358+0000","last_undegraded":"2023-05-22T15:46:23.078358+0000","last_fullsized":"2023-05-22T15:46:23.078358+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:51:08.858142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.37","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.074307+0000","last_change":"2023-05-22T15:46:23.074307+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.074307+0000","last_undegraded":"2023-05-22T15:46:23.074307+0000","last_fullsized":"2023-05-22T15:46:23.074307+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:28:23.397061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.36","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.075645+0000","last_change":"2023-05-22T15:46:23.075645+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.075645+0000","last_undegraded":"2023-05-22T15:46:23.075645+0000","last_fullsized":"2023-05-22T15:46:23.075645+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:46:28.227459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.35","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.069852+0000","last_change":"2023-05-22T15:46:23.069852+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.069852+0000","last_undegraded":"2023-05-22T15:46:23.069852+0000","last_fullsized":"2023-05-22T15:46:23.069852+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T15:53:20.317234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.34","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.073019+0000","last_change":"2023-05-22T15:46:23.073019+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.073019+0000","last_undegraded":"2023-05-22T15:46:23.073019+0000","last_fullsized":"2023-05-22T15:46:23.073019+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T18:52:29.352539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.33","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.078530+0000","last_change":"2023-05-22T15:46:23.078530+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.078530+0000","last_undegraded":"2023-05-22T15:46:23.078530+0000","last_fullsized":"2023-05-22T15:46:23.078530+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:22:06.668010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.32","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.072337+0000","last_change":"2023-05-22T15:46:23.072337+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.072337+0000","last_undegraded":"2023-05-22T15:46:23.072337+0000","last_fullsized":"2023-05-22T15:46:23.072337+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T18:05:16.988218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.31","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.074359+0000","last_change":"2023-05-22T15:46:23.074359+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.074359+0000","last_undegraded":"2023-05-22T15:46:23.074359+0000","last_fullsized":"2023-05-22T15:46:23.074359+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T18:05:16.988218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.30","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.063646+0000","last_change":"2023-05-22T15:46:23.063646+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.063646+0000","last_undegraded":"2023-05-22T15:46:23.063646+0000","last_fullsized":"2023-05-22T15:46:23.063646+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:30:04.941241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2f","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.075803+0000","last_change":"2023-05-22T15:46:23.075803+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.075803+0000","last_undegraded":"2023-05-22T15:46:23.075803+0000","last_fullsized":"2023-05-22T15:46:23.075803+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:23:42.844820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.2e","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.064448+0000","last_change":"2023-05-22T15:46:23.064448+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.064448+0000","last_undegraded":"2023-05-22T15:46:23.064448+0000","last_fullsized":"2023-05-22T15:46:23.064448+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:23:58.635276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2d","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.068994+0000","last_change":"2023-05-22T15:46:23.068994+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.068994+0000","last_undegraded":"2023-05-22T15:46:23.068994+0000","last_fullsized":"2023-05-22T15:46:23.068994+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:38:41.516579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.2c","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.062047+0000","last_change":"2023-05-22T15:46:23.062047+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.062047+0000","last_undegraded":"2023-05-22T15:46:23.062047+0000","last_fullsized":"2023-05-22T15:46:23.062047+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:52:08.264737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2b","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.093524+0000","last_change":"2023-05-22T15:46:23.093524+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.093524+0000","last_undegraded":"2023-05-22T15:46:23.093524+0000","last_fullsized":"2023-05-22T15:46:23.093524+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:55:41.010860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.2a","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.069379+0000","last_change":"2023-05-22T15:46:23.069379+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.069379+0000","last_undegraded":"2023-05-22T15:46:23.069379+0000","last_fullsized":"2023-05-22T15:46:23.069379+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:36:49.326554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.29","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.081591+0000","last_change":"2023-05-22T15:46:23.081591+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.081591+0000","last_undegraded":"2023-05-22T15:46:23.081591+0000","last_fullsized":"2023-05-22T15:46:23.081591+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:55:51.486171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.28","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.070163+0000","last_change":"2023-05-22T15:46:23.070163+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.070163+0000","last_undegraded":"2023-05-22T15:46:23.070163+0000","last_fullsized":"2023-05-22T15:46:23.070163+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:21:39.187868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.27","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.071496+0000","last_change":"2023-05-22T15:46:23.071496+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.071496+0000","last_undegraded":"2023-05-22T15:46:23.071496+0000","last_fullsized":"2023-05-22T15:46:23.071496+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:37:26.183234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.26","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.063189+0000","last_change":"2023-05-22T15:46:23.063189+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.063189+0000","last_undegraded":"2023-05-22T15:46:23.063189+0000","last_fullsized":"2023-05-22T15:46:23.063189+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:43:11.401376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.25","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.074484+0000","last_change":"2023-05-22T15:46:23.074484+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.074484+0000","last_undegraded":"2023-05-22T15:46:23.074484+0000","last_fullsized":"2023-05-22T15:46:23.074484+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T03:43:40.124163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.24","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.064681+0000","last_change":"2023-05-22T15:46:23.064681+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.064681+0000","last_undegraded":"2023-05-22T15:46:23.064681+0000","last_fullsized":"2023-05-22T15:46:23.064681+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:20:29.148898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.23","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.077284+0000","last_change":"2023-05-22T15:46:23.077284+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.077284+0000","last_undegraded":"2023-05-22T15:46:23.077284+0000","last_fullsized":"2023-05-22T15:46:23.077284+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:34:43.855049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.22","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.073667+0000","last_change":"2023-05-22T15:46:23.073667+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.073667+0000","last_undegraded":"2023-05-22T15:46:23.073667+0000","last_fullsized":"2023-05-22T15:46:23.073667+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T16:10:54.516325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.21","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.086294+0000","last_change":"2023-05-22T15:46:23.086294+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.086294+0000","last_undegraded":"2023-05-22T15:46:23.086294+0000","last_fullsized":"2023-05-22T15:46:23.086294+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:20:12.734869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.20","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.074629+0000","last_change":"2023-05-22T15:46:23.074629+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.074629+0000","last_undegraded":"2023-05-22T15:46:23.074629+0000","last_fullsized":"2023-05-22T15:46:23.074629+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:52:03.083872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.1e","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.061067+0000","last_change":"2023-05-22T15:46:23.061067+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.061067+0000","last_undegraded":"2023-05-22T15:46:23.061067+0000","last_fullsized":"2023-05-22T15:46:23.061067+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:29:20.717884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1f","version":"0'0","reported_seq":9,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.120601+0000","last_change":"2023-05-22T15:46:23.112442+0000","last_active":"2023-05-22T15:46:23.120601+0000","last_peered":"2023-05-22T15:46:23.120601+0000","last_clean":"2023-05-22T15:46:23.120601+0000","last_became_active":"2023-05-22T15:46:23.111561+0000","last_became_peered":"2023-05-22T15:46:23.111561+0000","last_unstale":"2023-05-22T15:46:23.120601+0000","last_undegraded":"2023-05-22T15:46:23.120601+0000","last_fullsized":"2023-05-22T15:46:23.120601+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_clean_scrub_stamp":"2023-05-22T15:46:22.034488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:28:22.384974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.1f","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.071619+0000","last_change":"2023-05-22T15:46:23.071619+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.071619+0000","last_undegraded":"2023-05-22T15:46:23.071619+0000","last_fullsized":"2023-05-22T15:46:23.071619+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:22:11.366110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.1e","version":"0'0","reported_seq":9,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.092179+0000","last_change":"2023-05-22T15:46:23.078407+0000","last_active":"2023-05-22T15:46:23.092179+0000","last_peered":"2023-05-22T15:46:23.092179+0000","last_clean":"2023-05-22T15:46:23.092179+0000","last_became_active":"2023-05-22T15:46:23.077477+0000","last_became_peered":"2023-05-22T15:46:23.077477+0000","last_unstale":"2023-05-22T15:46:23.092179+0000","last_undegraded":"2023-05-22T15:46:23.092179+0000","last_fullsized":"2023-05-22T15:46:23.092179+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_clean_scrub_stamp":"2023-05-22T15:46:22.034488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:20:11.712441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.1c","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.089330+0000","last_change":"2023-05-22T15:46:23.089330+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.089330+0000","last_undegraded":"2023-05-22T15:46:23.089330+0000","last_fullsized":"2023-05-22T15:46:23.089330+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:47:45.260369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1d","version":"0'0","reported_seq":9,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.090669+0000","last_change":"2023-05-22T15:46:23.083700+0000","last_active":"2023-05-22T15:46:23.090669+0000","last_peered":"2023-05-22T15:46:23.090669+0000","last_clean":"2023-05-22T15:46:23.090669+0000","last_became_active":"2023-05-22T15:46:23.083350+0000","last_became_peered":"2023-05-22T15:46:23.083350+0000","last_unstale":"2023-05-22T15:46:23.090669+0000","last_undegraded":"2023-05-22T15:46:23.090669+0000","last_fullsized":"2023-05-22T15:46:23.090669+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_clean_scrub_stamp":"2023-05-22T15:46:22.034488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:03:14.130053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.1d","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.090978+0000","last_change":"2023-05-22T15:46:23.090978+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.090978+0000","last_undegraded":"2023-05-22T15:46:23.090978+0000","last_fullsized":"2023-05-22T15:46:23.090978+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:07:01.166140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1c","version":"0'0","reported_seq":9,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.094786+0000","last_change":"2023-05-22T15:46:23.092200+0000","last_active":"2023-05-22T15:46:23.094786+0000","last_peered":"2023-05-22T15:46:23.094786+0000","last_clean":"2023-05-22T15:46:23.094786+0000","last_became_active":"2023-05-22T15:46:23.089522+0000","last_became_peered":"2023-05-22T15:46:23.089522+0000","last_unstale":"2023-05-22T15:46:23.094786+0000","last_undegraded":"2023-05-22T15:46:23.094786+0000","last_fullsized":"2023-05-22T15:46:23.094786+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_clean_scrub_stamp":"2023-05-22T15:46:22.034488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:16:51.789402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.1a","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.061711+0000","last_change":"2023-05-22T15:46:23.061711+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.061711+0000","last_undegraded":"2023-05-22T15:46:23.061711+0000","last_fullsized":"2023-05-22T15:46:23.061711+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:23:42.844820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1b","version":"0'0","reported_seq":9,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.094709+0000","last_change":"2023-05-22T15:46:23.088945+0000","last_active":"2023-05-22T15:46:23.094709+0000","last_peered":"2023-05-22T15:46:23.094709+0000","last_clean":"2023-05-22T15:46:23.094709+0000","last_became_active":"2023-05-22T15:46:23.088536+0000","last_became_peered":"2023-05-22T15:46:23.088536+0000","last_unstale":"2023-05-22T15:46:23.094709+0000","last_undegraded":"2023-05-22T15:46:23.094709+0000","last_fullsized":"2023-05-22T15:46:23.094709+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_clean_scrub_stamp":"2023-05-22T15:46:22.034488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:12:41.914428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.1b","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.073120+0000","last_change":"2023-05-22T15:46:23.073120+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.073120+0000","last_undegraded":"2023-05-22T15:46:23.073120+0000","last_fullsized":"2023-05-22T15:46:23.073120+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:59:24.451915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.1a","version":"0'0","reported_seq":9,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.090491+0000","last_change":"2023-05-22T15:46:23.089182+0000","last_active":"2023-05-22T15:46:23.090491+0000","last_peered":"2023-05-22T15:46:23.090491+0000","last_clean":"2023-05-22T15:46:23.090491+0000","last_became_active":"2023-05-22T15:46:23.088748+0000","last_became_peered":"2023-05-22T15:46:23.088748+0000","last_unstale":"2023-05-22T15:46:23.090491+0000","last_undegraded":"2023-05-22T15:46:23.090491+0000","last_fullsized":"2023-05-22T15:46:23.090491+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_clean_scrub_stamp":"2023-05-22T15:46:22.034488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:49:33.716090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.18","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.077114+0000","last_change":"2023-05-22T15:46:23.077114+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.077114+0000","last_undegraded":"2023-05-22T15:46:23.077114+0000","last_fullsized":"2023-05-22T15:46:23.077114+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T03:20:39.103248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.19","version":"0'0","reported_seq":9,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.117596+0000","last_change":"2023-05-22T15:46:23.105440+0000","last_active":"2023-05-22T15:46:23.117596+0000","last_peered":"2023-05-22T15:46:23.117596+0000","last_clean":"2023-05-22T15:46:23.117596+0000","last_became_active":"2023-05-22T15:46:23.104687+0000","last_became_peered":"2023-05-22T15:46:23.104687+0000","last_unstale":"2023-05-22T15:46:23.117596+0000","last_undegraded":"2023-05-22T15:46:23.117596+0000","last_fullsized":"2023-05-22T15:46:23.117596+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_clean_scrub_stamp":"2023-05-22T15:46:22.034488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:51:07.846056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.19","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.075186+0000","last_change":"2023-05-22T15:46:23.075186+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.075186+0000","last_undegraded":"2023-05-22T15:46:23.075186+0000","last_fullsized":"2023-05-22T15:46:23.075186+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:22:20.631045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.18","version":"0'0","reported_seq":9,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.108549+0000","last_change":"2023-05-22T15:46:23.106544+0000","last_active":"2023-05-22T15:46:23.108549+0000","last_peered":"2023-05-22T15:46:23.108549+0000","last_clean":"2023-05-22T15:46:23.108549+0000","last_became_active":"2023-05-22T15:46:23.106098+0000","last_became_peered":"2023-05-22T15:46:23.106098+0000","last_unstale":"2023-05-22T15:46:23.108549+0000","last_undegraded":"2023-05-22T15:46:23.108549+0000","last_fullsized":"2023-05-22T15:46:23.108549+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_clean_scrub_stamp":"2023-05-22T15:46:22.034488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:42:03.486863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.e","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.083614+0000","last_change":"2023-05-22T15:46:23.083614+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.083614+0000","last_undegraded":"2023-05-22T15:46:23.083614+0000","last_fullsized":"2023-05-22T15:46:23.083614+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:20:55.836997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.f","version":"0'0","reported_seq":9,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.116428+0000","last_change":"2023-05-22T15:46:23.112873+0000","last_active":"2023-05-22T15:46:23.116428+0000","last_peered":"2023-05-22T15:46:23.116428+0000","last_clean":"2023-05-22T15:46:23.116428+0000","last_became_active":"2023-05-22T15:46:23.112082+0000","last_became_peered":"2023-05-22T15:46:23.112082+0000","last_unstale":"2023-05-22T15:46:23.116428+0000","last_undegraded":"2023-05-22T15:46:23.116428+0000","last_fullsized":"2023-05-22T15:46:23.116428+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_clean_scrub_stamp":"2023-05-22T15:46:22.034488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:24:32.309552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.f","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.063425+0000","last_change":"2023-05-22T15:46:23.063425+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.063425+0000","last_undegraded":"2023-05-22T15:46:23.063425+0000","last_fullsized":"2023-05-22T15:46:23.063425+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T16:10:54.516325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.e","version":"0'0","reported_seq":9,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.109486+0000","last_change":"2023-05-22T15:46:23.107165+0000","last_active":"2023-05-22T15:46:23.109486+0000","last_peered":"2023-05-22T15:46:23.109486+0000","last_clean":"2023-05-22T15:46:23.109486+0000","last_became_active":"2023-05-22T15:46:23.106681+0000","last_became_peered":"2023-05-22T15:46:23.106681+0000","last_unstale":"2023-05-22T15:46:23.109486+0000","last_undegraded":"2023-05-22T15:46:23.109486+0000","last_fullsized":"2023-05-22T15:46:23.109486+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_clean_scrub_stamp":"2023-05-22T15:46:22.034488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T15:59:17.155128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.c","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.060697+0000","last_change":"2023-05-22T15:46:23.060697+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.060697+0000","last_undegraded":"2023-05-22T15:46:23.060697+0000","last_fullsized":"2023-05-22T15:46:23.060697+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:33:34.943631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.d","version":"0'0","reported_seq":9,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.111448+0000","last_change":"2023-05-22T15:46:23.103043+0000","last_active":"2023-05-22T15:46:23.111448+0000","last_peered":"2023-05-22T15:46:23.111448+0000","last_clean":"2023-05-22T15:46:23.111448+0000","last_became_active":"2023-05-22T15:46:23.102220+0000","last_became_peered":"2023-05-22T15:46:23.102220+0000","last_unstale":"2023-05-22T15:46:23.111448+0000","last_undegraded":"2023-05-22T15:46:23.111448+0000","last_fullsized":"2023-05-22T15:46:23.111448+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_clean_scrub_stamp":"2023-05-22T15:46:22.034488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:20:54.824911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.d","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.073232+0000","last_change":"2023-05-22T15:46:23.073232+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.073232+0000","last_undegraded":"2023-05-22T15:46:23.073232+0000","last_fullsized":"2023-05-22T15:46:23.073232+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:49:48.656654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.c","version":"0'0","reported_seq":9,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.082235+0000","last_change":"2023-05-22T15:46:23.078278+0000","last_active":"2023-05-22T15:46:23.082235+0000","last_peered":"2023-05-22T15:46:23.082235+0000","last_clean":"2023-05-22T15:46:23.082235+0000","last_became_active":"2023-05-22T15:46:23.077343+0000","last_became_peered":"2023-05-22T15:46:23.077343+0000","last_unstale":"2023-05-22T15:46:23.082235+0000","last_undegraded":"2023-05-22T15:46:23.082235+0000","last_fullsized":"2023-05-22T15:46:23.082235+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_clean_scrub_stamp":"2023-05-22T15:46:22.034488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:17:43.848098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.a","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.072991+0000","last_change":"2023-05-22T15:46:23.072991+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.072991+0000","last_undegraded":"2023-05-22T15:46:23.072991+0000","last_fullsized":"2023-05-22T15:46:23.072991+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:24:33.321639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.b","version":"0'0","reported_seq":9,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.092758+0000","last_change":"2023-05-22T15:46:23.086857+0000","last_active":"2023-05-22T15:46:23.092758+0000","last_peered":"2023-05-22T15:46:23.092758+0000","last_clean":"2023-05-22T15:46:23.092758+0000","last_became_active":"2023-05-22T15:46:23.086098+0000","last_became_peered":"2023-05-22T15:46:23.086098+0000","last_unstale":"2023-05-22T15:46:23.092758+0000","last_undegraded":"2023-05-22T15:46:23.092758+0000","last_fullsized":"2023-05-22T15:46:23.092758+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_clean_scrub_stamp":"2023-05-22T15:46:22.034488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:45:50.595714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.b","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.071529+0000","last_change":"2023-05-22T15:46:23.071529+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.071529+0000","last_undegraded":"2023-05-22T15:46:23.071529+0000","last_fullsized":"2023-05-22T15:46:23.071529+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:20:12.734869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.a","version":"0'0","reported_seq":9,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.107500+0000","last_change":"2023-05-22T15:46:23.106615+0000","last_active":"2023-05-22T15:46:23.107500+0000","last_peered":"2023-05-22T15:46:23.107500+0000","last_clean":"2023-05-22T15:46:23.107500+0000","last_became_active":"2023-05-22T15:46:23.106155+0000","last_became_peered":"2023-05-22T15:46:23.106155+0000","last_unstale":"2023-05-22T15:46:23.107500+0000","last_undegraded":"2023-05-22T15:46:23.107500+0000","last_fullsized":"2023-05-22T15:46:23.107500+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_clean_scrub_stamp":"2023-05-22T15:46:22.034488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:55:28.593183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.8","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.071955+0000","last_change":"2023-05-22T15:46:23.071955+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.071955+0000","last_undegraded":"2023-05-22T15:46:23.071955+0000","last_fullsized":"2023-05-22T15:46:23.071955+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T16:27:47.997551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.9","version":"0'0","reported_seq":9,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.082561+0000","last_change":"2023-05-22T15:46:23.080736+0000","last_active":"2023-05-22T15:46:23.082561+0000","last_peered":"2023-05-22T15:46:23.082561+0000","last_clean":"2023-05-22T15:46:23.082561+0000","last_became_active":"2023-05-22T15:46:23.080251+0000","last_became_peered":"2023-05-22T15:46:23.080251+0000","last_unstale":"2023-05-22T15:46:23.082561+0000","last_undegraded":"2023-05-22T15:46:23.082561+0000","last_fullsized":"2023-05-22T15:46:23.082561+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_clean_scrub_stamp":"2023-05-22T15:46:22.034488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T16:52:22.575852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.9","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.089066+0000","last_change":"2023-05-22T15:46:23.089066+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.089066+0000","last_undegraded":"2023-05-22T15:46:23.089066+0000","last_fullsized":"2023-05-22T15:46:23.089066+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:31:09.541110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.8","version":"0'0","reported_seq":9,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.109492+0000","last_change":"2023-05-22T15:46:23.107191+0000","last_active":"2023-05-22T15:46:23.109492+0000","last_peered":"2023-05-22T15:46:23.109492+0000","last_clean":"2023-05-22T15:46:23.109492+0000","last_became_active":"2023-05-22T15:46:23.106653+0000","last_became_peered":"2023-05-22T15:46:23.106653+0000","last_unstale":"2023-05-22T15:46:23.109492+0000","last_undegraded":"2023-05-22T15:46:23.109492+0000","last_fullsized":"2023-05-22T15:46:23.109492+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_clean_scrub_stamp":"2023-05-22T15:46:22.034488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:00:00.382995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.6","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.076413+0000","last_change":"2023-05-22T15:46:23.076413+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.076413+0000","last_undegraded":"2023-05-22T15:46:23.076413+0000","last_fullsized":"2023-05-22T15:46:23.076413+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:55:41.010860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.7","version":"0'0","reported_seq":9,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.092881+0000","last_change":"2023-05-22T15:46:23.091817+0000","last_active":"2023-05-22T15:46:23.092881+0000","last_peered":"2023-05-22T15:46:23.092881+0000","last_clean":"2023-05-22T15:46:23.092881+0000","last_became_active":"2023-05-22T15:46:23.091434+0000","last_became_peered":"2023-05-22T15:46:23.091434+0000","last_unstale":"2023-05-22T15:46:23.092881+0000","last_undegraded":"2023-05-22T15:46:23.092881+0000","last_fullsized":"2023-05-22T15:46:23.092881+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_clean_scrub_stamp":"2023-05-22T15:46:22.034488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T16:37:19.990243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.7","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.079702+0000","last_change":"2023-05-22T15:46:23.079702+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.079702+0000","last_undegraded":"2023-05-22T15:46:23.079702+0000","last_fullsized":"2023-05-22T15:46:23.079702+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:20:29.148898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.6","version":"0'0","reported_seq":9,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.120724+0000","last_change":"2023-05-22T15:46:23.114945+0000","last_active":"2023-05-22T15:46:23.120724+0000","last_peered":"2023-05-22T15:46:23.120724+0000","last_clean":"2023-05-22T15:46:23.120724+0000","last_became_active":"2023-05-22T15:46:23.114027+0000","last_became_peered":"2023-05-22T15:46:23.114027+0000","last_unstale":"2023-05-22T15:46:23.120724+0000","last_undegraded":"2023-05-22T15:46:23.120724+0000","last_fullsized":"2023-05-22T15:46:23.120724+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_clean_scrub_stamp":"2023-05-22T15:46:22.034488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T18:52:28.340453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.4","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.075841+0000","last_change":"2023-05-22T15:46:23.075841+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.075841+0000","last_undegraded":"2023-05-22T15:46:23.075841+0000","last_fullsized":"2023-05-22T15:46:23.075841+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:46:38.837010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.5","version":"0'0","reported_seq":9,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.109205+0000","last_change":"2023-05-22T15:46:23.108380+0000","last_active":"2023-05-22T15:46:23.109205+0000","last_peered":"2023-05-22T15:46:23.109205+0000","last_clean":"2023-05-22T15:46:23.109205+0000","last_became_active":"2023-05-22T15:46:23.108014+0000","last_became_peered":"2023-05-22T15:46:23.108014+0000","last_unstale":"2023-05-22T15:46:23.109205+0000","last_undegraded":"2023-05-22T15:46:23.109205+0000","last_fullsized":"2023-05-22T15:46:23.109205+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_clean_scrub_stamp":"2023-05-22T15:46:22.034488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T03:37:19.248335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.5","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.073829+0000","last_change":"2023-05-22T15:46:23.073829+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.073829+0000","last_undegraded":"2023-05-22T15:46:23.073829+0000","last_fullsized":"2023-05-22T15:46:23.073829+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:47:45.260369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.4","version":"0'0","reported_seq":9,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.094601+0000","last_change":"2023-05-22T15:46:23.093730+0000","last_active":"2023-05-22T15:46:23.094601+0000","last_peered":"2023-05-22T15:46:23.094601+0000","last_clean":"2023-05-22T15:46:23.094601+0000","last_became_active":"2023-05-22T15:46:23.093418+0000","last_became_peered":"2023-05-22T15:46:23.093418+0000","last_unstale":"2023-05-22T15:46:23.094601+0000","last_undegraded":"2023-05-22T15:46:23.094601+0000","last_fullsized":"2023-05-22T15:46:23.094601+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_clean_scrub_stamp":"2023-05-22T15:46:22.034488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T16:38:30.323718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.2","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.084350+0000","last_change":"2023-05-22T15:46:23.084350+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.084350+0000","last_undegraded":"2023-05-22T15:46:23.084350+0000","last_fullsized":"2023-05-22T15:46:23.084350+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:25:55.107784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.3","version":"0'0","reported_seq":9,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.111036+0000","last_change":"2023-05-22T15:46:23.110053+0000","last_active":"2023-05-22T15:46:23.111036+0000","last_peered":"2023-05-22T15:46:23.111036+0000","last_clean":"2023-05-22T15:46:23.111036+0000","last_became_active":"2023-05-22T15:46:23.109753+0000","last_became_peered":"2023-05-22T15:46:23.109753+0000","last_unstale":"2023-05-22T15:46:23.111036+0000","last_undegraded":"2023-05-22T15:46:23.111036+0000","last_fullsized":"2023-05-22T15:46:23.111036+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_clean_scrub_stamp":"2023-05-22T15:46:22.034488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T18:29:44.257708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.3","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.062480+0000","last_change":"2023-05-22T15:46:23.062480+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.062480+0000","last_undegraded":"2023-05-22T15:46:23.062480+0000","last_fullsized":"2023-05-22T15:46:23.062480+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T18:05:16.988218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.2","version":"0'0","reported_seq":9,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.120623+0000","last_change":"2023-05-22T15:46:23.112868+0000","last_active":"2023-05-22T15:46:23.120623+0000","last_peered":"2023-05-22T15:46:23.120623+0000","last_clean":"2023-05-22T15:46:23.120623+0000","last_became_active":"2023-05-22T15:46:23.112288+0000","last_became_peered":"2023-05-22T15:46:23.112288+0000","last_unstale":"2023-05-22T15:46:23.120623+0000","last_undegraded":"2023-05-22T15:46:23.120623+0000","last_fullsized":"2023-05-22T15:46:23.120623+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_clean_scrub_stamp":"2023-05-22T15:46:22.034488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:16:19.873549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.0","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.062429+0000","last_change":"2023-05-22T15:46:23.062429+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.062429+0000","last_undegraded":"2023-05-22T15:46:23.062429+0000","last_fullsized":"2023-05-22T15:46:23.062429+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:03:37.104570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1","version":"0'0","reported_seq":9,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.092294+0000","last_change":"2023-05-22T15:46:23.091423+0000","last_active":"2023-05-22T15:46:23.092294+0000","last_peered":"2023-05-22T15:46:23.092294+0000","last_clean":"2023-05-22T15:46:23.092294+0000","last_became_active":"2023-05-22T15:46:23.091088+0000","last_became_peered":"2023-05-22T15:46:23.091088+0000","last_unstale":"2023-05-22T15:46:23.092294+0000","last_undegraded":"2023-05-22T15:46:23.092294+0000","last_fullsized":"2023-05-22T15:46:23.092294+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_clean_scrub_stamp":"2023-05-22T15:46:22.034488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:52:16.638521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.1","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.072920+0000","last_change":"2023-05-22T15:46:23.072920+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.072920+0000","last_undegraded":"2023-05-22T15:46:23.072920+0000","last_fullsized":"2023-05-22T15:46:23.072920+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:16:20.885635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.0","version":"0'0","reported_seq":9,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.107395+0000","last_change":"2023-05-22T15:46:23.105214+0000","last_active":"2023-05-22T15:46:23.107395+0000","last_peered":"2023-05-22T15:46:23.107395+0000","last_clean":"2023-05-22T15:46:23.107395+0000","last_became_active":"2023-05-22T15:46:23.104909+0000","last_became_peered":"2023-05-22T15:46:23.104909+0000","last_unstale":"2023-05-22T15:46:23.107395+0000","last_undegraded":"2023-05-22T15:46:23.107395+0000","last_fullsized":"2023-05-22T15:46:23.107395+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_clean_scrub_stamp":"2023-05-22T15:46:22.034488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:37:03.818368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.12","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.085873+0000","last_change":"2023-05-22T15:46:23.085873+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.085873+0000","last_undegraded":"2023-05-22T15:46:23.085873+0000","last_fullsized":"2023-05-22T15:46:23.085873+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:20:55.836997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.13","version":"0'0","reported_seq":9,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.120574+0000","last_change":"2023-05-22T15:46:23.112061+0000","last_active":"2023-05-22T15:46:23.120574+0000","last_peered":"2023-05-22T15:46:23.120574+0000","last_clean":"2023-05-22T15:46:23.120574+0000","last_became_active":"2023-05-22T15:46:23.111315+0000","last_became_peered":"2023-05-22T15:46:23.111315+0000","last_unstale":"2023-05-22T15:46:23.120574+0000","last_undegraded":"2023-05-22T15:46:23.120574+0000","last_fullsized":"2023-05-22T15:46:23.120574+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_clean_scrub_stamp":"2023-05-22T15:46:22.034488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:37:25.171147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"134'249","reported_seq":753,"reported_epoch":144,"state":"active+clean","last_fresh":"2023-05-22T15:46:22.069225+0000","last_change":"2023-05-22T15:26:39.318204+0000","last_active":"2023-05-22T15:46:22.069225+0000","last_peered":"2023-05-22T15:46:22.069225+0000","last_clean":"2023-05-22T15:46:22.069225+0000","last_became_active":"2023-05-22T15:26:39.315370+0000","last_became_peered":"2023-05-22T15:26:39.315370+0000","last_unstale":"2023-05-22T15:46:22.069225+0000","last_undegraded":"2023-05-22T15:46:22.069225+0000","last_fullsized":"2023-05-22T15:46:22.069225+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:38.020076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:38.020076+0000","last_clean_scrub_stamp":"2023-05-22T15:26:38.020076+0000","objects_scrubbed":0,"log_size":249,"log_dups_size":0,"ondisk_log_size":249,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:06:39.207543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1268,"num_read_kb":2572,"num_write":399,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":139,"reported_epoch":144,"state":"active+clean","last_fresh":"2023-05-22T15:46:22.059466+0000","last_change":"2023-05-22T15:26:44.971951+0000","last_active":"2023-05-22T15:46:22.059466+0000","last_peered":"2023-05-22T15:46:22.059466+0000","last_clean":"2023-05-22T15:46:22.059466+0000","last_became_active":"2023-05-22T15:26:44.970872+0000","last_became_peered":"2023-05-22T15:26:44.970872+0000","last_unstale":"2023-05-22T15:46:22.059466+0000","last_undegraded":"2023-05-22T15:46:22.059466+0000","last_fullsized":"2023-05-22T15:46:22.059466+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:55:34.752949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.11","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.082774+0000","last_change":"2023-05-22T15:46:23.082774+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.082774+0000","last_undegraded":"2023-05-22T15:46:23.082774+0000","last_fullsized":"2023-05-22T15:46:23.082774+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:37:33.613777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.10","version":"0'0","reported_seq":9,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.107838+0000","last_change":"2023-05-22T15:46:23.091282+0000","last_active":"2023-05-22T15:46:23.107838+0000","last_peered":"2023-05-22T15:46:23.107838+0000","last_clean":"2023-05-22T15:46:23.107838+0000","last_became_active":"2023-05-22T15:46:23.086640+0000","last_became_peered":"2023-05-22T15:46:23.086640+0000","last_unstale":"2023-05-22T15:46:23.107838+0000","last_undegraded":"2023-05-22T15:46:23.107838+0000","last_fullsized":"2023-05-22T15:46:23.107838+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_clean_scrub_stamp":"2023-05-22T15:46:22.034488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T16:27:46.985465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":132,"reported_epoch":144,"state":"active+clean","last_fresh":"2023-05-22T15:46:22.068980+0000","last_change":"2023-05-22T15:26:44.977198+0000","last_active":"2023-05-22T15:46:22.068980+0000","last_peered":"2023-05-22T15:46:22.068980+0000","last_clean":"2023-05-22T15:46:22.068980+0000","last_became_active":"2023-05-22T15:26:44.976445+0000","last_became_peered":"2023-05-22T15:26:44.976445+0000","last_unstale":"2023-05-22T15:46:22.068980+0000","last_undegraded":"2023-05-22T15:46:22.068980+0000","last_fullsized":"2023-05-22T15:46:22.068980+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:12:41.464054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.10","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.073353+0000","last_change":"2023-05-22T15:46:23.073353+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.073353+0000","last_undegraded":"2023-05-22T15:46:23.073353+0000","last_fullsized":"2023-05-22T15:46:23.073353+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:42:45.119566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.11","version":"0'0","reported_seq":9,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.103893+0000","last_change":"2023-05-22T15:46:23.102815+0000","last_active":"2023-05-22T15:46:23.103893+0000","last_peered":"2023-05-22T15:46:23.103893+0000","last_clean":"2023-05-22T15:46:23.103893+0000","last_became_active":"2023-05-22T15:46:23.102405+0000","last_became_peered":"2023-05-22T15:46:23.102405+0000","last_unstale":"2023-05-22T15:46:23.103893+0000","last_undegraded":"2023-05-22T15:46:23.103893+0000","last_fullsized":"2023-05-22T15:46:23.103893+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_clean_scrub_stamp":"2023-05-22T15:46:22.034488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:45:50.595714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":132,"reported_epoch":144,"state":"active+clean","last_fresh":"2023-05-22T15:46:22.051914+0000","last_change":"2023-05-22T15:26:44.969469+0000","last_active":"2023-05-22T15:46:22.051914+0000","last_peered":"2023-05-22T15:46:22.051914+0000","last_clean":"2023-05-22T15:46:22.051914+0000","last_became_active":"2023-05-22T15:26:44.969083+0000","last_became_peered":"2023-05-22T15:26:44.969083+0000","last_unstale":"2023-05-22T15:46:22.051914+0000","last_undegraded":"2023-05-22T15:46:22.051914+0000","last_fullsized":"2023-05-22T15:46:22.051914+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:29:23.670067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.13","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.076508+0000","last_change":"2023-05-22T15:46:23.076508+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.076508+0000","last_undegraded":"2023-05-22T15:46:23.076508+0000","last_fullsized":"2023-05-22T15:46:23.076508+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T16:27:47.997551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.12","version":"0'0","reported_seq":9,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.107682+0000","last_change":"2023-05-22T15:46:23.105368+0000","last_active":"2023-05-22T15:46:23.107682+0000","last_peered":"2023-05-22T15:46:23.107682+0000","last_clean":"2023-05-22T15:46:23.107682+0000","last_became_active":"2023-05-22T15:46:23.104771+0000","last_became_peered":"2023-05-22T15:46:23.104771+0000","last_unstale":"2023-05-22T15:46:23.107682+0000","last_undegraded":"2023-05-22T15:46:23.107682+0000","last_fullsized":"2023-05-22T15:46:23.107682+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_clean_scrub_stamp":"2023-05-22T15:46:22.034488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:42:03.486863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":139,"reported_epoch":144,"state":"active+clean","last_fresh":"2023-05-22T15:46:22.059265+0000","last_change":"2023-05-22T15:26:44.976632+0000","last_active":"2023-05-22T15:46:22.059265+0000","last_peered":"2023-05-22T15:46:22.059265+0000","last_clean":"2023-05-22T15:46:22.059265+0000","last_became_active":"2023-05-22T15:26:44.975546+0000","last_became_peered":"2023-05-22T15:26:44.975546+0000","last_unstale":"2023-05-22T15:46:22.059265+0000","last_undegraded":"2023-05-22T15:46:22.059265+0000","last_fullsized":"2023-05-22T15:46:22.059265+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:42:57.973030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.15","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.084525+0000","last_change":"2023-05-22T15:46:23.084525+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.084525+0000","last_undegraded":"2023-05-22T15:46:23.084525+0000","last_fullsized":"2023-05-22T15:46:23.084525+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T03:25:18.349611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.14","version":"0'0","reported_seq":9,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.109240+0000","last_change":"2023-05-22T15:46:23.104870+0000","last_active":"2023-05-22T15:46:23.109240+0000","last_peered":"2023-05-22T15:46:23.109240+0000","last_clean":"2023-05-22T15:46:23.109240+0000","last_became_active":"2023-05-22T15:46:23.104126+0000","last_became_peered":"2023-05-22T15:46:23.104126+0000","last_unstale":"2023-05-22T15:46:23.109240+0000","last_undegraded":"2023-05-22T15:46:23.109240+0000","last_fullsized":"2023-05-22T15:46:23.109240+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_clean_scrub_stamp":"2023-05-22T15:46:22.034488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:20:28.136812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":123,"reported_epoch":144,"state":"active+clean","last_fresh":"2023-05-22T15:46:22.065910+0000","last_change":"2023-05-22T15:26:45.091475+0000","last_active":"2023-05-22T15:46:22.065910+0000","last_peered":"2023-05-22T15:46:22.065910+0000","last_clean":"2023-05-22T15:46:22.065910+0000","last_became_active":"2023-05-22T15:26:45.091143+0000","last_became_peered":"2023-05-22T15:26:45.091143+0000","last_unstale":"2023-05-22T15:46:22.065910+0000","last_undegraded":"2023-05-22T15:46:22.065910+0000","last_fullsized":"2023-05-22T15:46:22.065910+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:12:41.464054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.14","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.088461+0000","last_change":"2023-05-22T15:46:23.088461+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.088461+0000","last_undegraded":"2023-05-22T15:46:23.088461+0000","last_fullsized":"2023-05-22T15:46:23.088461+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:49:34.728176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.15","version":"0'0","reported_seq":9,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.092614+0000","last_change":"2023-05-22T15:46:23.083261+0000","last_active":"2023-05-22T15:46:23.092614+0000","last_peered":"2023-05-22T15:46:23.092614+0000","last_clean":"2023-05-22T15:46:23.092614+0000","last_became_active":"2023-05-22T15:46:23.082317+0000","last_became_peered":"2023-05-22T15:46:23.082317+0000","last_unstale":"2023-05-22T15:46:23.092614+0000","last_undegraded":"2023-05-22T15:46:23.092614+0000","last_fullsized":"2023-05-22T15:46:23.092614+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_clean_scrub_stamp":"2023-05-22T15:46:22.034488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T18:29:44.257708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":132,"reported_epoch":144,"state":"active+clean","last_fresh":"2023-05-22T15:46:22.069477+0000","last_change":"2023-05-22T15:26:45.092367+0000","last_active":"2023-05-22T15:46:22.069477+0000","last_peered":"2023-05-22T15:46:22.069477+0000","last_clean":"2023-05-22T15:46:22.069477+0000","last_became_active":"2023-05-22T15:26:45.091951+0000","last_became_peered":"2023-05-22T15:26:45.091951+0000","last_unstale":"2023-05-22T15:46:22.069477+0000","last_undegraded":"2023-05-22T15:46:22.069477+0000","last_fullsized":"2023-05-22T15:46:22.069477+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:42:57.973030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.17","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.065336+0000","last_change":"2023-05-22T15:46:23.065336+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.065336+0000","last_undegraded":"2023-05-22T15:46:23.065336+0000","last_fullsized":"2023-05-22T15:46:23.065336+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T16:27:47.997551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.16","version":"0'0","reported_seq":9,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.107509+0000","last_change":"2023-05-22T15:46:23.106558+0000","last_active":"2023-05-22T15:46:23.107509+0000","last_peered":"2023-05-22T15:46:23.107509+0000","last_clean":"2023-05-22T15:46:23.107509+0000","last_became_active":"2023-05-22T15:46:23.106098+0000","last_became_peered":"2023-05-22T15:46:23.106098+0000","last_unstale":"2023-05-22T15:46:23.107509+0000","last_undegraded":"2023-05-22T15:46:23.107509+0000","last_fullsized":"2023-05-22T15:46:23.107509+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_clean_scrub_stamp":"2023-05-22T15:46:22.034488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:30:32.850917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":123,"reported_epoch":144,"state":"active+clean","last_fresh":"2023-05-22T15:46:22.065875+0000","last_change":"2023-05-22T15:26:44.978858+0000","last_active":"2023-05-22T15:46:22.065875+0000","last_peered":"2023-05-22T15:46:22.065875+0000","last_clean":"2023-05-22T15:46:22.065875+0000","last_became_active":"2023-05-22T15:26:44.978443+0000","last_became_peered":"2023-05-22T15:26:44.978443+0000","last_unstale":"2023-05-22T15:46:22.065875+0000","last_undegraded":"2023-05-22T15:46:22.065875+0000","last_fullsized":"2023-05-22T15:46:22.065875+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:46:29.290243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.16","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.071629+0000","last_change":"2023-05-22T15:46:23.071629+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.071629+0000","last_undegraded":"2023-05-22T15:46:23.071629+0000","last_fullsized":"2023-05-22T15:46:23.071629+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T18:29:45.269794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.17","version":"0'0","reported_seq":9,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.092889+0000","last_change":"2023-05-22T15:46:23.084909+0000","last_active":"2023-05-22T15:46:23.092889+0000","last_peered":"2023-05-22T15:46:23.092889+0000","last_clean":"2023-05-22T15:46:23.092889+0000","last_became_active":"2023-05-22T15:46:23.084184+0000","last_became_peered":"2023-05-22T15:46:23.084184+0000","last_unstale":"2023-05-22T15:46:23.092889+0000","last_undegraded":"2023-05-22T15:46:23.092889+0000","last_fullsized":"2023-05-22T15:46:23.092889+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_clean_scrub_stamp":"2023-05-22T15:46:22.034488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:47:36.764285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":125,"reported_epoch":144,"state":"active+clean","last_fresh":"2023-05-22T15:46:22.063472+0000","last_change":"2023-05-22T15:26:44.982421+0000","last_active":"2023-05-22T15:46:22.063472+0000","last_peered":"2023-05-22T15:46:22.063472+0000","last_clean":"2023-05-22T15:46:22.063472+0000","last_became_active":"2023-05-22T15:26:44.981586+0000","last_became_peered":"2023-05-22T15:26:44.981586+0000","last_unstale":"2023-05-22T15:46:22.063472+0000","last_undegraded":"2023-05-22T15:46:22.063472+0000","last_fullsized":"2023-05-22T15:46:22.063472+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:14:11.156544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":19,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":18,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1268,"num_read_kb":2572,"num_write":399,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":249,"ondisk_log_size":249,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575084,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":19680,"kb_used_data":2328,"kb_used_omap":0,"kb_used_meta":17344,"kb_avail":94352160,"statfs":{"total":96636764160,"available":96616611840,"internally_reserved":0,"allocated":2383872,"data_stored":1535545,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":17760256},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 15:46:11 2023","interfaces":[{"interface":"back","average":{"1min":0.86699999999999999,"5min":0.89200000000000002,"15min":0.876},"min":{"1min":0.53600000000000003,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":1.294,"5min":2.637,"15min":2.637},"last":0.76600000000000001},{"interface":"front","average":{"1min":0.88,"5min":0.88700000000000001,"15min":0.89200000000000002},"min":{"1min":0.60799999999999998,"5min":0.502,"15min":0.35799999999999998},"max":{"1min":1.1519999999999999,"5min":3.0449999999999999,"15min":3.0449999999999999},"last":0.86199999999999999}]},{"osd":1,"last update":"Mon May 22 15:46:11 2023","interfaces":[{"interface":"back","average":{"1min":0.96799999999999997,"5min":0.95399999999999996,"15min":0.97899999999999998},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.1899999999999999,"5min":1.385,"15min":1.389},"last":0.73799999999999999},{"interface":"front","average":{"1min":0.96799999999999997,"5min":1.006,"15min":1.022},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.54900000000000004},"max":{"1min":1.226,"5min":2.7269999999999999,"15min":2.7269999999999999},"last":0.99299999999999999}]},{"osd":2,"last update":"Mon May 22 15:46:11 2023","interfaces":[{"interface":"back","average":{"1min":0.86199999999999999,"5min":0.85099999999999998,"15min":0.871},"min":{"1min":0.55600000000000005,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.109,"5min":1.252,"15min":1.252},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.88100000000000001,"5min":0.93999999999999995,"15min":0.95699999999999996},"min":{"1min":0.504,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.143,"5min":3.141,"15min":3.141},"last":0.68500000000000005}]},{"osd":3,"last update":"Mon May 22 15:46:11 2023","interfaces":[{"interface":"back","average":{"1min":0.98199999999999998,"5min":0.98899999999999999,"15min":0.96299999999999997},"min":{"1min":0.67600000000000005,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":1.3839999999999999,"5min":2.8149999999999999,"15min":2.8149999999999999},"last":1.109},{"interface":"front","average":{"1min":0.94399999999999995,"5min":0.92100000000000004,"15min":0.96199999999999997},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.2709999999999999,"5min":1.389,"15min":1.4710000000000001},"last":0.82899999999999996}]},{"osd":4,"last update":"Mon May 22 15:46:11 2023","interfaces":[{"interface":"back","average":{"1min":0.87,"5min":0.94399999999999995,"15min":0.90400000000000003},"min":{"1min":0.60299999999999998,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.5089999999999999,"5min":3.472,"15min":3.472},"last":0.94699999999999995},{"interface":"front","average":{"1min":0.997,"5min":1.1850000000000001,"15min":1.05},"min":{"1min":0.59699999999999998,"5min":0.56000000000000005,"15min":0.44600000000000001},"max":{"1min":1.2949999999999999,"5min":15.627000000000001,"15min":15.627000000000001},"last":1.163}]},{"osd":5,"last update":"Mon May 22 15:46:11 2023","interfaces":[{"interface":"back","average":{"1min":1.012,"5min":1.071,"15min":1.02},"min":{"1min":0.60899999999999999,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":1.4750000000000001,"5min":4.5599999999999996,"15min":4.5599999999999996},"last":0.90900000000000003},{"interface":"front","average":{"1min":1.115,"5min":1.161,"15min":1.139},"min":{"1min":0.82999999999999996,"5min":0.63800000000000001,"15min":0.58799999999999997},"max":{"1min":1.302,"5min":4.3799999999999999,"15min":4.3799999999999999},"last":1.1919999999999999}]},{"osd":6,"last update":"Mon May 22 15:46:11 2023","interfaces":[{"interface":"back","average":{"1min":1.073,"5min":1.1399999999999999,"15min":1.0840000000000001},"min":{"1min":0.80300000000000005,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.415,"5min":6.5069999999999997,"15min":6.5069999999999997},"last":1.1339999999999999},{"interface":"front","average":{"1min":1.1200000000000001,"5min":1.1659999999999999,"15min":1.1419999999999999},"min":{"1min":0.70299999999999996,"5min":0.62,"15min":0.62},"max":{"1min":1.458,"5min":3.3500000000000001,"15min":3.3500000000000001},"last":1.214}]}]},{"osd":6,"up_from":13,"seq":55834575085,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36572,"kb_used_data":1748,"kb_used_omap":0,"kb_used_meta":34816,"kb_avail":94335268,"statfs":{"total":96636764160,"available":96599314432,"internally_reserved":0,"allocated":1789952,"data_stored":945177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":35651584},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 15:46:20 2023","interfaces":[{"interface":"back","average":{"1min":0.90000000000000002,"5min":0.86699999999999999,"15min":0.89900000000000002},"min":{"1min":0.499,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":1.4039999999999999,"5min":1.486,"15min":1.486},"last":0.81899999999999995},{"interface":"front","average":{"1min":0.89200000000000002,"5min":0.94599999999999995,"15min":0.91100000000000003},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.36299999999999999},"max":{"1min":1.2729999999999999,"5min":1.6240000000000001,"15min":1.6240000000000001},"last":1.054}]},{"osd":1,"last update":"Mon May 22 15:46:20 2023","interfaces":[{"interface":"back","average":{"1min":0.93200000000000005,"5min":0.89100000000000001,"15min":0.91200000000000003},"min":{"1min":0.59799999999999998,"5min":0.504,"15min":0.48299999999999998},"max":{"1min":1.8799999999999999,"5min":1.8799999999999999,"15min":1.8799999999999999},"last":1.03},{"interface":"front","average":{"1min":0.95899999999999996,"5min":0.92400000000000004,"15min":0.93999999999999995},"min":{"1min":0.70299999999999996,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":1.1950000000000001,"5min":1.3320000000000001,"15min":1.554},"last":0.77300000000000002}]},{"osd":2,"last update":"Mon May 22 15:46:20 2023","interfaces":[{"interface":"back","average":{"1min":1.05,"5min":0.97399999999999998,"15min":0.98999999999999999},"min":{"1min":0.55400000000000005,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":3.2429999999999999,"5min":3.2429999999999999,"15min":3.2429999999999999},"last":1.099},{"interface":"front","average":{"1min":1.032,"5min":1.0269999999999999,"15min":1.0429999999999999},"min":{"1min":0.55300000000000005,"5min":0.5,"15min":0.442},"max":{"1min":3.3969999999999998,"5min":3.3969999999999998,"15min":3.3969999999999998},"last":1.1220000000000001}]},{"osd":3,"last update":"Mon May 22 15:46:20 2023","interfaces":[{"interface":"back","average":{"1min":0.88900000000000001,"5min":0.94599999999999995,"15min":0.97899999999999998},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":1.2470000000000001,"5min":1.5780000000000001,"15min":1.679},"last":1.155},{"interface":"front","average":{"1min":0.93500000000000005,"5min":0.95099999999999996,"15min":0.96299999999999997},"min":{"1min":0.69099999999999995,"5min":0.53200000000000003,"15min":0.50800000000000001},"max":{"1min":1.2430000000000001,"5min":1.383,"15min":1.4570000000000001},"last":1.006}]},{"osd":4,"last update":"Mon May 22 15:46:20 2023","interfaces":[{"interface":"back","average":{"1min":0.91300000000000003,"5min":0.93200000000000005,"15min":0.92500000000000004},"min":{"1min":0.61699999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":1.3169999999999999,"5min":1.581,"15min":1.6859999999999999},"last":0.91700000000000004},{"interface":"front","average":{"1min":0.93200000000000005,"5min":1.0209999999999999,"15min":1.0169999999999999},"min":{"1min":0.65500000000000003,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":1.242,"5min":1.452,"15min":1.8300000000000001},"last":0.88500000000000001}]},{"osd":5,"last update":"Mon May 22 15:46:20 2023","interfaces":[{"interface":"back","average":{"1min":0.98799999999999999,"5min":1.0169999999999999,"15min":0.98899999999999999},"min":{"1min":0.67600000000000005,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":1.3200000000000001,"5min":1.4530000000000001,"15min":1.6499999999999999},"last":0.85399999999999998},{"interface":"front","average":{"1min":1.1459999999999999,"5min":1.141,"15min":1.1120000000000001},"min":{"1min":0.81999999999999995,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.9990000000000001,"5min":1.9990000000000001,"15min":1.9990000000000001},"last":0.97799999999999998}]},{"osd":7,"last update":"Mon May 22 15:46:20 2023","interfaces":[{"interface":"back","average":{"1min":1.0860000000000001,"5min":1.131,"15min":1.125},"min":{"1min":0.85499999999999998,"5min":0.37,"15min":0.37},"max":{"1min":1.5529999999999999,"5min":1.853,"15min":1.9179999999999999},"last":1.23},{"interface":"front","average":{"1min":1.131,"5min":1.133,"15min":1.135},"min":{"1min":0.80300000000000005,"5min":0.442,"15min":0.442},"max":{"1min":1.579,"5min":1.579,"15min":1.8520000000000001},"last":0.95199999999999996}]}]},{"osd":5,"up_from":13,"seq":55834575085,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":26012,"kb_used_data":1748,"kb_used_omap":0,"kb_used_meta":24256,"kb_avail":94345828,"statfs":{"total":96636764160,"available":96610127872,"internally_reserved":0,"allocated":1789952,"data_stored":945177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":24838144},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 15:46:10 2023","interfaces":[{"interface":"back","average":{"1min":0.90900000000000003,"5min":0.85599999999999998,"15min":0.88300000000000001},"min":{"1min":0.61399999999999999,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.2889999999999999,"5min":1.3140000000000001,"15min":1.3440000000000001},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.91800000000000004,"5min":0.88600000000000001,"15min":0.88500000000000001},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":1.2709999999999999,"5min":1.3899999999999999,"15min":2.1699999999999999},"last":0.72599999999999998}]},{"osd":1,"last update":"Mon May 22 15:46:10 2023","interfaces":[{"interface":"back","average":{"1min":0.94999999999999996,"5min":0.96099999999999997,"15min":0.995},"min":{"1min":0.64800000000000002,"5min":0.52400000000000002,"15min":0.51000000000000001},"max":{"1min":1.3109999999999999,"5min":1.337,"15min":1.9370000000000001},"last":0.78000000000000003},{"interface":"front","average":{"1min":1.073,"5min":1.008,"15min":1.026},"min":{"1min":0.501,"5min":0.501,"15min":0.501},"max":{"1min":1.3089999999999999,"5min":1.6970000000000001,"15min":1.6970000000000001},"last":0.96899999999999997}]},{"osd":2,"last update":"Mon May 22 15:46:10 2023","interfaces":[{"interface":"back","average":{"1min":0.89100000000000001,"5min":0.91600000000000004,"15min":0.93400000000000005},"min":{"1min":0.505,"5min":0.505,"15min":0.504},"max":{"1min":1.343,"5min":1.5549999999999999,"15min":1.9830000000000001},"last":1.0549999999999999},{"interface":"front","average":{"1min":0.91800000000000004,"5min":0.94999999999999996,"15min":0.95199999999999996},"min":{"1min":0.56100000000000005,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.202,"5min":1.321,"15min":2.1120000000000001},"last":1.0169999999999999}]},{"osd":3,"last update":"Mon May 22 15:46:10 2023","interfaces":[{"interface":"back","average":{"1min":0.92900000000000005,"5min":0.92000000000000004,"15min":0.94899999999999995},"min":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.48199999999999998},"max":{"1min":1.206,"5min":1.2809999999999999,"15min":1.4419999999999999},"last":0.88300000000000001},{"interface":"front","average":{"1min":1.028,"5min":1.008,"15min":1.036},"min":{"1min":0.626,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":1.2490000000000001,"5min":1.371,"15min":2.52},"last":1.161}]},{"osd":4,"last update":"Mon May 22 15:46:10 2023","interfaces":[{"interface":"back","average":{"1min":0.96999999999999997,"5min":0.95699999999999996,"15min":0.95199999999999996},"min":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":1.405,"5min":1.405,"15min":2.226},"last":0.93500000000000005},{"interface":"front","average":{"1min":1.0629999999999999,"5min":1.024,"15min":1.0309999999999999},"min":{"1min":0.76000000000000001,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.3740000000000001,"5min":1.863,"15min":1.863},"last":1.1120000000000001}]},{"osd":6,"last update":"Mon May 22 15:46:10 2023","interfaces":[{"interface":"back","average":{"1min":1.008,"5min":1.024,"15min":1.0349999999999999},"min":{"1min":0.51900000000000002,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.397,"5min":1.4330000000000001,"15min":2.831},"last":1.246},{"interface":"front","average":{"1min":1.0369999999999999,"5min":1.0589999999999999,"15min":1.071},"min":{"1min":0.67000000000000004,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":1.4239999999999999,"5min":1.4239999999999999,"15min":2.3690000000000002},"last":1.296}]},{"osd":7,"last update":"Mon May 22 15:46:10 2023","interfaces":[{"interface":"back","average":{"1min":1.081,"5min":1.0900000000000001,"15min":1.1120000000000001},"min":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.42499999999999999},"max":{"1min":1.4359999999999999,"5min":1.47,"15min":1.849},"last":1.206},{"interface":"front","average":{"1min":1.1599999999999999,"5min":1.149,"15min":1.1659999999999999},"min":{"1min":0.86899999999999999,"5min":0.64000000000000001,"15min":0.64000000000000001},"max":{"1min":1.4339999999999999,"5min":1.575,"15min":2.7429999999999999},"last":1.3360000000000001}]}]},{"osd":4,"up_from":13,"seq":55834575083,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31592,"kb_used_data":1760,"kb_used_omap":0,"kb_used_meta":29824,"kb_avail":94340248,"statfs":{"total":96636764160,"available":96604413952,"internally_reserved":0,"allocated":1802240,"data_stored":952153,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":30539776},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 15:46:19 2023","interfaces":[{"interface":"back","average":{"1min":0.93200000000000005,"5min":0.86799999999999999,"15min":0.90500000000000003},"min":{"1min":0.63400000000000001,"5min":0.48999999999999999,"15min":0.34100000000000003},"max":{"1min":1.2629999999999999,"5min":1.4119999999999999,"15min":1.9910000000000001},"last":0.66600000000000004},{"interface":"front","average":{"1min":0.91300000000000003,"5min":0.90000000000000002,"15min":0.92200000000000004},"min":{"1min":0.51100000000000001,"5min":0.499,"15min":0.373},"max":{"1min":1.284,"5min":1.369,"15min":1.3779999999999999},"last":0.81000000000000005}]},{"osd":1,"last update":"Mon May 22 15:46:19 2023","interfaces":[{"interface":"back","average":{"1min":0.94799999999999995,"5min":0.91800000000000004,"15min":0.96399999999999997},"min":{"1min":0.60499999999999998,"5min":0.436,"15min":0.436},"max":{"1min":1.1970000000000001,"5min":1.3899999999999999,"15min":1.3899999999999999},"last":0.72099999999999997},{"interface":"front","average":{"1min":0.91700000000000004,"5min":0.84499999999999997,"15min":0.90100000000000002},"min":{"1min":0.56699999999999995,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.157,"5min":1.2889999999999999,"15min":1.3640000000000001},"last":0.97799999999999998}]},{"osd":2,"last update":"Mon May 22 15:46:19 2023","interfaces":[{"interface":"back","average":{"1min":0.90100000000000002,"5min":0.91600000000000004,"15min":0.94199999999999995},"min":{"1min":0.56299999999999994,"5min":0.50800000000000001,"15min":0.495},"max":{"1min":1.244,"5min":1.96,"15min":2.1019999999999999},"last":0.76500000000000001},{"interface":"front","average":{"1min":0.97799999999999998,"5min":0.98399999999999999,"15min":0.99199999999999999},"min":{"1min":0.58099999999999996,"5min":0.48799999999999999,"15min":0.48399999999999999},"max":{"1min":1.329,"5min":1.4399999999999999,"15min":1.4399999999999999},"last":0.60299999999999998}]},{"osd":3,"last update":"Mon May 22 15:46:19 2023","interfaces":[{"interface":"back","average":{"1min":0.91100000000000003,"5min":0.90200000000000002,"15min":0.93200000000000005},"min":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":1.224,"5min":1.28,"15min":1.3169999999999999},"last":1.0580000000000001},{"interface":"front","average":{"1min":1.0469999999999999,"5min":1.014,"15min":1.0209999999999999},"min":{"1min":0.72199999999999998,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":1.3420000000000001,"5min":1.4179999999999999,"15min":2.1720000000000002},"last":0.85599999999999998}]},{"osd":5,"last update":"Mon May 22 15:46:19 2023","interfaces":[{"interface":"back","average":{"1min":1.0209999999999999,"5min":0.97199999999999998,"15min":0.97699999999999998},"min":{"1min":0.55000000000000004,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.357,"5min":1.373,"15min":1.472},"last":1.0369999999999999},{"interface":"front","average":{"1min":1.0389999999999999,"5min":0.98999999999999999,"15min":0.97499999999999998},"min":{"1min":0.59999999999999998,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.262,"5min":1.365,"15min":1.4139999999999999},"last":1.155}]},{"osd":6,"last update":"Mon May 22 15:46:19 2023","interfaces":[{"interface":"back","average":{"1min":1.0649999999999999,"5min":0.99299999999999999,"15min":0.98499999999999999},"min":{"1min":0.72099999999999997,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":1.456,"5min":1.456,"15min":2.2349999999999999},"last":0.93500000000000005},{"interface":"front","average":{"1min":1.131,"5min":1.103,"15min":1.1100000000000001},"min":{"1min":0.66500000000000004,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":1.425,"5min":1.486,"15min":1.8220000000000001},"last":0.89800000000000002}]},{"osd":7,"last update":"Mon May 22 15:46:19 2023","interfaces":[{"interface":"back","average":{"1min":1.137,"5min":1.0629999999999999,"15min":1.052},"min":{"1min":0.85699999999999998,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":1.389,"5min":1.46,"15min":1.5800000000000001},"last":1.109},{"interface":"front","average":{"1min":1.1930000000000001,"5min":1.167,"15min":1.143},"min":{"1min":0.78900000000000003,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":1.5620000000000001,"5min":1.6559999999999999,"15min":2.306},"last":1.018}]}]},{"osd":3,"up_from":13,"seq":55834575083,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36956,"kb_used_data":1748,"kb_used_omap":0,"kb_used_meta":35200,"kb_avail":94334884,"statfs":{"total":96636764160,"available":96598921216,"internally_reserved":0,"allocated":1789952,"data_stored":945177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":36044800},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 15:46:03 2023","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.61699999999999999,"15min":0.63600000000000001},"min":{"1min":0.50700000000000001,"5min":0.252,"15min":0.19},"max":{"1min":1.169,"5min":1.169,"15min":1.194},"last":0.64600000000000002},{"interface":"front","average":{"1min":0.746,"5min":0.68500000000000005,"15min":0.70899999999999996},"min":{"1min":0.48899999999999999,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":1.085,"5min":1.296,"15min":1.296},"last":0.83599999999999997}]},{"osd":1,"last update":"Mon May 22 15:46:03 2023","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.64200000000000002,"15min":0.66200000000000003},"min":{"1min":0.52100000000000002,"5min":0.314,"15min":0.29499999999999998},"max":{"1min":0.90600000000000003,"5min":1.2609999999999999,"15min":1.2609999999999999},"last":0.93000000000000005},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.72899999999999998,"15min":0.76800000000000002},"min":{"1min":0.47499999999999998,"5min":0.34399999999999997,"15min":0.313},"max":{"1min":1.081,"5min":1.1359999999999999,"15min":1.222},"last":0.60099999999999998}]},{"osd":2,"last update":"Mon May 22 15:46:03 2023","interfaces":[{"interface":"back","average":{"1min":0.84199999999999997,"5min":0.78700000000000003,"15min":0.81699999999999995},"min":{"1min":0.45600000000000002,"5min":0.40699999999999997,"15min":0.36599999999999999},"max":{"1min":1.0529999999999999,"5min":1.506,"15min":1.506},"last":0.75800000000000001},{"interface":"front","average":{"1min":0.81899999999999995,"5min":0.75600000000000001,"15min":0.80700000000000005},"min":{"1min":0.65300000000000002,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":1.032,"5min":1.147,"15min":1.359},"last":0.89400000000000002}]},{"osd":4,"last update":"Mon May 22 15:46:03 2023","interfaces":[{"interface":"back","average":{"1min":0.97499999999999998,"5min":0.92500000000000004,"15min":0.95599999999999996},"min":{"1min":0.71999999999999997,"5min":0.54500000000000004,"15min":0.48999999999999999},"max":{"1min":1.238,"5min":1.413,"15min":1.4470000000000001},"last":1.0569999999999999},{"interface":"front","average":{"1min":1.0529999999999999,"5min":0.97499999999999998,"15min":1.0029999999999999},"min":{"1min":0.83299999999999996,"5min":0.54400000000000004,"15min":0.49199999999999999},"max":{"1min":1.407,"5min":1.407,"15min":1.4470000000000001},"last":1.0289999999999999}]},{"osd":5,"last update":"Mon May 22 15:46:03 2023","interfaces":[{"interface":"back","average":{"1min":1.0509999999999999,"5min":0.98399999999999999,"15min":1.016},"min":{"1min":0.80900000000000005,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.3149999999999999,"5min":1.5089999999999999,"15min":1.5089999999999999},"last":0.95899999999999996},{"interface":"front","average":{"1min":1.0349999999999999,"5min":1.006,"15min":1.038},"min":{"1min":0.90200000000000002,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.2130000000000001,"5min":1.4299999999999999,"15min":1.4359999999999999},"last":1.347}]},{"osd":6,"last update":"Mon May 22 15:46:03 2023","interfaces":[{"interface":"back","average":{"1min":1.0580000000000001,"5min":1.0129999999999999,"15min":1.042},"min":{"1min":0.873,"5min":0.47699999999999998,"15min":0.46999999999999997},"max":{"1min":1.2989999999999999,"5min":1.4570000000000001,"15min":1.49},"last":1.101},{"interface":"front","average":{"1min":1.083,"5min":1.0269999999999999,"15min":1.0700000000000001},"min":{"1min":0.79800000000000004,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":1.2949999999999999,"5min":1.4330000000000001,"15min":1.5009999999999999},"last":1.403}]},{"osd":7,"last update":"Mon May 22 15:46:03 2023","interfaces":[{"interface":"back","average":{"1min":1.113,"5min":1.0780000000000001,"15min":1.1040000000000001},"min":{"1min":0.83599999999999997,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.3779999999999999,"5min":1.6719999999999999,"15min":1.6719999999999999},"last":1.454},{"interface":"front","average":{"1min":1.117,"5min":1.071,"15min":1.085},"min":{"1min":0.90200000000000002,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":1.361,"5min":1.8700000000000001,"15min":1.8700000000000001},"last":1.0029999999999999}]}]},{"osd":2,"up_from":13,"seq":55834575084,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":18780,"kb_used_data":1748,"kb_used_omap":0,"kb_used_meta":17024,"kb_avail":94353060,"statfs":{"total":96636764160,"available":96617533440,"internally_reserved":0,"allocated":1789952,"data_stored":945177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":17432576},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 15:46:03 2023","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.64400000000000002,"15min":0.65600000000000003},"min":{"1min":0.28699999999999998,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":1.097,"5min":1.1539999999999999,"15min":1.8240000000000001},"last":0.70299999999999996},{"interface":"front","average":{"1min":0.69899999999999995,"5min":0.70499999999999996,"15min":0.746},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":1.079,"5min":1.079,"15min":1.8160000000000001},"last":0.46300000000000002}]},{"osd":1,"last update":"Mon May 22 15:46:03 2023","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.72999999999999998,"15min":0.74399999999999999},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.27000000000000002},"max":{"1min":1.196,"5min":1.196,"15min":1.2},"last":0.92000000000000004},{"interface":"front","average":{"1min":0.754,"5min":0.75900000000000001,"15min":0.80800000000000005},"min":{"1min":0.32400000000000001,"5min":0.317,"15min":0.317},"max":{"1min":1.0309999999999999,"5min":1.246,"15min":2.5089999999999999},"last":0.52500000000000002}]},{"osd":3,"last update":"Mon May 22 15:46:03 2023","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":0.69399999999999995,"15min":0.72599999999999998},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.96199999999999997,"5min":1.077,"15min":2.7629999999999999},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.82699999999999996,"5min":0.84599999999999997,"15min":0.88300000000000001},"min":{"1min":0.39600000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":1.1699999999999999,"5min":1.256,"15min":2.6840000000000002},"last":0.85699999999999998}]},{"osd":4,"last update":"Mon May 22 15:46:03 2023","interfaces":[{"interface":"back","average":{"1min":0.97499999999999998,"5min":0.98499999999999999,"15min":1.018},"min":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":1.274,"5min":1.536,"15min":2.8039999999999998},"last":1.1459999999999999},{"interface":"front","average":{"1min":1.0069999999999999,"5min":1.0009999999999999,"15min":1.028},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.2509999999999999,"5min":1.4299999999999999,"15min":2.0819999999999999},"last":0.81799999999999995}]},{"osd":5,"last update":"Mon May 22 15:46:03 2023","interfaces":[{"interface":"back","average":{"1min":1.0569999999999999,"5min":1.056,"15min":1.052},"min":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.3959999999999999,"5min":1.5049999999999999,"15min":2.3860000000000001},"last":1.175},{"interface":"front","average":{"1min":0.99099999999999999,"5min":0.97399999999999998,"15min":0.998},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.331,"5min":1.331,"15min":3.0299999999999998},"last":0.78100000000000003}]},{"osd":6,"last update":"Mon May 22 15:46:03 2023","interfaces":[{"interface":"back","average":{"1min":1.038,"5min":1.012,"15min":1.036},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":1.3320000000000001,"5min":1.49,"15min":2.883},"last":1.1020000000000001},{"interface":"front","average":{"1min":1.129,"5min":1.081,"15min":1.1040000000000001},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":1.4450000000000001,"5min":1.5660000000000001,"15min":3.1459999999999999},"last":1.0209999999999999}]},{"osd":7,"last update":"Mon May 22 15:46:03 2023","interfaces":[{"interface":"back","average":{"1min":1.1639999999999999,"5min":1.097,"15min":1.1279999999999999},"min":{"1min":0.55000000000000004,"5min":0.499,"15min":0.499},"max":{"1min":1.429,"5min":1.55,"15min":2.5990000000000002},"last":1.052},{"interface":"front","average":{"1min":1.1240000000000001,"5min":1.0640000000000001,"15min":1.1060000000000001},"min":{"1min":0.5,"5min":0.5,"15min":0.5},"max":{"1min":1.4650000000000001,"5min":1.4650000000000001,"15min":3.8799999999999999},"last":0.97699999999999998}]}]},{"osd":1,"up_from":13,"seq":55834575084,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29800,"kb_used_data":1760,"kb_used_omap":0,"kb_used_meta":28032,"kb_avail":94342040,"statfs":{"total":96636764160,"available":96606248960,"internally_reserved":0,"allocated":1802240,"data_stored":952153,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":28704768},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 15:46:17 2023","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.63200000000000001,"15min":0.63400000000000001},"min":{"1min":0.40600000000000003,"5min":0.26300000000000001,"15min":0.22700000000000001},"max":{"1min":1.1200000000000001,"5min":1.2270000000000001,"15min":1.2270000000000001},"last":0.98699999999999999},{"interface":"front","average":{"1min":0.77100000000000002,"5min":0.70699999999999996,"15min":0.76400000000000001},"min":{"1min":0.35999999999999999,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":1.1839999999999999,"5min":1.1839999999999999,"15min":1.2529999999999999},"last":0.58699999999999997}]},{"osd":2,"last update":"Mon May 22 15:46:17 2023","interfaces":[{"interface":"back","average":{"1min":0.76400000000000001,"5min":0.71399999999999997,"15min":0.73799999999999999},"min":{"1min":0.44800000000000001,"5min":0.317,"15min":0.28999999999999998},"max":{"1min":1.0660000000000001,"5min":1.391,"15min":1.391},"last":0.73899999999999999},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.72999999999999998,"15min":0.76300000000000001},"min":{"1min":0.49199999999999999,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.98699999999999999,"5min":1.2270000000000001,"15min":1.276},"last":0.77100000000000002}]},{"osd":3,"last update":"Mon May 22 15:46:17 2023","interfaces":[{"interface":"back","average":{"1min":0.80100000000000005,"5min":0.79900000000000004,"15min":0.82799999999999996},"min":{"1min":0.51200000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.153,"5min":1.657,"15min":1.657},"last":0.93200000000000005},{"interface":"front","average":{"1min":0.81100000000000005,"5min":0.80800000000000005,"15min":0.81499999999999995},"min":{"1min":0.33700000000000002,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":1.095,"5min":2.0569999999999999,"15min":2.0569999999999999},"last":0.876}]},{"osd":4,"last update":"Mon May 22 15:46:17 2023","interfaces":[{"interface":"back","average":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.96499999999999997},"min":{"1min":0.64500000000000002,"5min":0.55200000000000005,"15min":0.34000000000000002},"max":{"1min":1.502,"5min":2.9529999999999998,"15min":2.9529999999999998},"last":1.2450000000000001},{"interface":"front","average":{"1min":0.97499999999999998,"5min":0.999,"15min":1.002},"min":{"1min":0.61299999999999999,"5min":0.504,"15min":0.49299999999999999},"max":{"1min":1.292,"5min":2.7759999999999998,"15min":2.7759999999999998},"last":1.0469999999999999}]},{"osd":5,"last update":"Mon May 22 15:46:17 2023","interfaces":[{"interface":"back","average":{"1min":1.0349999999999999,"5min":1.006,"15min":1.034},"min":{"1min":0.627,"5min":0.58399999999999996,"15min":0.443},"max":{"1min":1.556,"5min":1.9570000000000001,"15min":1.9570000000000001},"last":1.0269999999999999},{"interface":"front","average":{"1min":1.1060000000000001,"5min":1.034,"15min":1.0449999999999999},"min":{"1min":0.59399999999999997,"5min":0.53400000000000003,"15min":0.47299999999999998},"max":{"1min":1.577,"5min":2.1200000000000001,"15min":2.1200000000000001},"last":1.177}]},{"osd":6,"last update":"Mon May 22 15:46:17 2023","interfaces":[{"interface":"back","average":{"1min":1.02,"5min":1.0369999999999999,"15min":1.036},"min":{"1min":0.65900000000000003,"5min":0.54000000000000004,"15min":0.45300000000000001},"max":{"1min":1.3180000000000001,"5min":2.831,"15min":2.831},"last":1.1399999999999999},{"interface":"front","average":{"1min":1.1499999999999999,"5min":1.085,"15min":1.0940000000000001},"min":{"1min":0.55800000000000005,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":1.6299999999999999,"5min":1.7629999999999999,"15min":1.7629999999999999},"last":1.0920000000000001}]},{"osd":7,"last update":"Mon May 22 15:46:17 2023","interfaces":[{"interface":"back","average":{"1min":1.1759999999999999,"5min":1.135,"15min":1.1180000000000001},"min":{"1min":0.61299999999999999,"5min":0.60299999999999998,"15min":0.56799999999999995},"max":{"1min":1.679,"5min":2.4199999999999999,"15min":2.4199999999999999},"last":1.1299999999999999},{"interface":"front","average":{"1min":1.1819999999999999,"5min":1.1259999999999999,"15min":1.1299999999999999},"min":{"1min":0.76300000000000001,"5min":0.47299999999999998,"15min":0.41599999999999998},"max":{"1min":1.619,"5min":3.8799999999999999,"15min":3.8799999999999999},"last":1.337}]}]},{"osd":0,"up_from":13,"seq":55834575085,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":25376,"kb_used_data":2328,"kb_used_omap":0,"kb_used_meta":23040,"kb_avail":94346464,"statfs":{"total":96636764160,"available":96610779136,"internally_reserved":0,"allocated":2383872,"data_stored":1535545,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":23592960},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon May 22 15:46:21 2023","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.65900000000000003,"15min":0.68100000000000005},"min":{"1min":0.307,"5min":0.248,"15min":0.223},"max":{"1min":1.2190000000000001,"5min":1.466,"15min":1.466},"last":0.78400000000000003},{"interface":"front","average":{"1min":0.69899999999999995,"5min":0.70599999999999996,"15min":0.72899999999999998},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.24099999999999999},"max":{"1min":1.2230000000000001,"5min":1.71,"15min":1.71},"last":0.55200000000000005}]},{"osd":2,"last update":"Mon May 22 15:46:21 2023","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":0.70699999999999996,"15min":0.74099999999999999},"min":{"1min":0.39700000000000002,"5min":0.318,"15min":0.26500000000000001},"max":{"1min":0.96799999999999997,"5min":1.165,"15min":1.3220000000000001},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.82799999999999996,"5min":0.75900000000000001,"15min":0.78800000000000003},"min":{"1min":0.375,"5min":0.34999999999999998,"15min":0.32900000000000001},"max":{"1min":1.1699999999999999,"5min":1.3129999999999999,"15min":1.3129999999999999},"last":0.747}]},{"osd":3,"last update":"Mon May 22 15:46:21 2023","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":0.74099999999999999,"15min":0.76000000000000001},"min":{"1min":0.35499999999999998,"5min":0.31,"15min":0.31},"max":{"1min":0.96199999999999997,"5min":1.9299999999999999,"15min":1.9299999999999999},"last":0.88600000000000001},{"interface":"front","average":{"1min":0.83499999999999996,"5min":0.83299999999999996,"15min":0.871},"min":{"1min":0.35599999999999998,"5min":0.35199999999999998,"15min":0.34399999999999997},"max":{"1min":1.1299999999999999,"5min":1.758,"15min":1.758},"last":0.85999999999999999}]},{"osd":4,"last update":"Mon May 22 15:46:21 2023","interfaces":[{"interface":"back","average":{"1min":0.92800000000000005,"5min":0.95099999999999996,"15min":0.95099999999999996},"min":{"1min":0.57599999999999996,"5min":0.57399999999999995,"15min":0.48099999999999998},"max":{"1min":1.149,"5min":2.923,"15min":2.923},"last":1.091},{"interface":"front","average":{"1min":0.98999999999999999,"5min":1.006,"15min":1.026},"min":{"1min":0.624,"5min":0.624,"15min":0.624},"max":{"1min":1.3839999999999999,"5min":2.0129999999999999,"15min":2.0129999999999999},"last":0.93600000000000005}]},{"osd":5,"last update":"Mon May 22 15:46:21 2023","interfaces":[{"interface":"back","average":{"1min":1,"5min":1.004,"15min":1.0409999999999999},"min":{"1min":0.59099999999999997,"5min":0.56000000000000005,"15min":0.46000000000000002},"max":{"1min":1.238,"5min":1.97,"15min":1.97},"last":1.006},{"interface":"front","average":{"1min":1.02,"5min":1.016,"15min":1.008},"min":{"1min":0.67600000000000005,"5min":0.56599999999999995,"15min":0.55300000000000005},"max":{"1min":1.5349999999999999,"5min":5.0519999999999996,"15min":5.0519999999999996},"last":0.98199999999999998}]},{"osd":6,"last update":"Mon May 22 15:46:21 2023","interfaces":[{"interface":"back","average":{"1min":1.044,"5min":1.0589999999999999,"15min":1.089},"min":{"1min":0.58899999999999997,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":1.306,"5min":2.117,"15min":2.117},"last":1.1439999999999999},{"interface":"front","average":{"1min":1.085,"5min":1.1299999999999999,"15min":1.119},"min":{"1min":0.69199999999999995,"5min":0.65500000000000003,"15min":0.499},"max":{"1min":1.3100000000000001,"5min":5.2640000000000002,"15min":5.2640000000000002},"last":1.1910000000000001}]},{"osd":7,"last update":"Mon May 22 15:46:21 2023","interfaces":[{"interface":"back","average":{"1min":1.075,"5min":1.1040000000000001,"15min":1.129},"min":{"1min":0.63800000000000001,"5min":0.61299999999999999,"15min":0.61299999999999999},"max":{"1min":1.3640000000000001,"5min":2.0550000000000002,"15min":2.0550000000000002},"last":1.0680000000000001},{"interface":"front","average":{"1min":1.089,"5min":1.0900000000000001,"15min":1.0960000000000001},"min":{"1min":0.73099999999999998,"5min":0.63300000000000001,"15min":0.60299999999999998},"max":{"1min":1.6240000000000001,"5min":3.9060000000000001,"15min":3.9060000000000001},"last":1.0409999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-05-22T15:46:27.956 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-05-22T15:46:28.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.107+0000 7f72f9d08640 1 Processor -- start 2023-05-22T15:46:28.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.107+0000 7f72f9d08640 1 -- start start 2023-05-22T15:46:28.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.111+0000 7f72f9d08640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f72f40ca4e0 0x7f72f415e460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:28.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.111+0000 7f72f9d08640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f72f40c9b40 0x7f72f40afb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:28.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.111+0000 7f72f9d08640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72f40b0650 con 0x7f72f4070df0 2023-05-22T15:46:28.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.111+0000 7f72f9d08640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72f40b0370 con 0x7f72f40c9b40 2023-05-22T15:46:28.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.111+0000 7f72f9d08640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72f415d980 con 0x7f72f40ca4e0 2023-05-22T15:46:28.118 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.111+0000 7f72f3fff640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f72f4070df0 0x7f72f4071210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:38884/0 (socket says 172.21.15.16:38884) 2023-05-22T15:46:28.118 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.111+0000 7f72f3fff640 1 -- 172.21.15.16:0/1105783215 learned_addr learned my addr 172.21.15.16:0/1105783215 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:46:28.118 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.111+0000 7f72f2ffd640 1 --2- 172.21.15.16:0/1105783215 >> v2:172.21.15.16:3300/0 conn(0x7f72f40c9b40 0x7f72f40afb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:28.118 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.111+0000 7f72f37fe640 1 --2- 172.21.15.16:0/1105783215 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f72f40ca4e0 0x7f72f415e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:28.119 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.111+0000 7f72f2ffd640 1 -- 172.21.15.16:0/1105783215 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f72f40ca4e0 msgr2=0x7f72f415e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:46:28.119 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.111+0000 7f72f2ffd640 1 --2- 172.21.15.16:0/1105783215 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f72f40ca4e0 0x7f72f415e460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:28.119 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.111+0000 7f72f2ffd640 1 -- 172.21.15.16:0/1105783215 >> v1:172.21.15.16:6789/0 conn(0x7f72f4070df0 legacy=0x7f72f4071210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:28.119 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.111+0000 7f72f2ffd640 1 -- 172.21.15.16:0/1105783215 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72f415dc60 con 0x7f72f40c9b40 2023-05-22T15:46:28.121 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.115+0000 7f72f2ffd640 1 --2- 172.21.15.16:0/1105783215 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f72f40c9b40 0x7f72f40afb40 secure :-1 s=READY pgs=1846 cs=0 l=1 rev1=1 crypto rx=0x7f72dc0097c0 tx=0x7f72dc01c3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=baa5fa9ee33c9b9b server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:28.121 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.115+0000 7f72f27fc640 1 -- 172.21.15.16:0/1105783215 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72dc0ac600 con 0x7f72f40c9b40 2023-05-22T15:46:28.121 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.115+0000 7f72f27fc640 1 -- 172.21.15.16:0/1105783215 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72dc0a8050 con 0x7f72f40c9b40 2023-05-22T15:46:28.121 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.115+0000 7f72f27fc640 1 -- 172.21.15.16:0/1105783215 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72dc0ac980 con 0x7f72f40c9b40 2023-05-22T15:46:28.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.115+0000 7f72f9d08640 1 -- 172.21.15.16:0/1105783215 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f72f40c9b40 msgr2=0x7f72f40afb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:28.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.115+0000 7f72f9d08640 1 --2- 172.21.15.16:0/1105783215 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f72f40c9b40 0x7f72f40afb40 secure :-1 s=READY pgs=1846 cs=0 l=1 rev1=1 crypto rx=0x7f72dc0097c0 tx=0x7f72dc01c3c0 comp rx=0 tx=0).stop 2023-05-22T15:46:28.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.115+0000 7f72f9d08640 1 -- 172.21.15.16:0/1105783215 shutdown_connections 2023-05-22T15:46:28.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.115+0000 7f72f9d08640 1 --2- 172.21.15.16:0/1105783215 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f72f40c9b40 0x7f72f40afb40 unknown :-1 s=CLOSED pgs=1846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:28.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.115+0000 7f72f9d08640 1 --2- 172.21.15.16:0/1105783215 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f72f40ca4e0 0x7f72f415e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:28.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.115+0000 7f72f9d08640 1 -- 172.21.15.16:0/1105783215 >> 172.21.15.16:0/1105783215 conn(0x7f72f4069720 msgr2=0x7f72f4069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:46:28.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.115+0000 7f72f9d08640 1 -- 172.21.15.16:0/1105783215 shutdown_connections 2023-05-22T15:46:28.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.115+0000 7f72f9d08640 1 -- 172.21.15.16:0/1105783215 wait complete. 2023-05-22T15:46:28.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.115+0000 7f72f9d08640 1 Processor -- start 2023-05-22T15:46:28.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.115+0000 7f72f9d08640 1 -- start start 2023-05-22T15:46:28.127 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.119+0000 7f72f9d08640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f72f4070df0 0x7f72f40c5400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:28.128 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.119+0000 7f72f9d08640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f72f40c9b40 0x7f72f414c070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:28.128 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.119+0000 7f72f9d08640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f72f40ca4e0 0x7f72f414c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:28.128 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.119+0000 7f72f9d08640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72f414caf0 con 0x7f72f40c9b40 2023-05-22T15:46:28.128 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.119+0000 7f72f9d08640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72f415a2b0 con 0x7f72f40ca4e0 2023-05-22T15:46:28.128 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.119+0000 7f72f9d08640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72f415ae30 con 0x7f72f4070df0 2023-05-22T15:46:28.128 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.119+0000 7f72f2ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f72f40c9b40 0x7f72f414c070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:28.129 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.119+0000 7f72f2ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f72f40c9b40 0x7f72f414c070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34638/0 (socket says 172.21.15.16:34638) 2023-05-22T15:46:28.129 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.119+0000 7f72f2ffd640 1 -- 172.21.15.16:0/4110933893 learned_addr learned my addr 172.21.15.16:0/4110933893 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:46:28.129 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.119+0000 7f72f37fe640 1 --2- 172.21.15.16:0/4110933893 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f72f4070df0 0x7f72f40c5400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:28.129 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.119+0000 7f72f3fff640 1 --2- 172.21.15.16:0/4110933893 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f72f40ca4e0 0x7f72f414c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:28.129 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.119+0000 7f72f2ffd640 1 -- 172.21.15.16:0/4110933893 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f72f4070df0 msgr2=0x7f72f40c5400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:28.129 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.119+0000 7f72f2ffd640 1 --2- 172.21.15.16:0/4110933893 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f72f4070df0 0x7f72f40c5400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:28.130 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.119+0000 7f72f2ffd640 1 -- 172.21.15.16:0/4110933893 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f72f40ca4e0 msgr2=0x7f72f414c5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:28.130 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.119+0000 7f72f2ffd640 1 --2- 172.21.15.16:0/4110933893 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f72f40ca4e0 0x7f72f414c5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:28.130 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.119+0000 7f72f2ffd640 1 -- 172.21.15.16:0/4110933893 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72dc009420 con 0x7f72f40c9b40 2023-05-22T15:46:28.130 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.119+0000 7f72f2ffd640 1 --2- 172.21.15.16:0/4110933893 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f72f40c9b40 0x7f72f414c070 secure :-1 s=READY pgs=1847 cs=0 l=1 rev1=1 crypto rx=0x7f72dc01aa70 tx=0x7f72dc019160 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:28.130 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.119+0000 7f72ebfff640 1 -- 172.21.15.16:0/4110933893 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72dc0b6690 con 0x7f72f40c9b40 2023-05-22T15:46:28.130 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.119+0000 7f72ebfff640 1 -- 172.21.15.16:0/4110933893 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72dc0b67f0 con 0x7f72f40c9b40 2023-05-22T15:46:28.130 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.119+0000 7f72ebfff640 1 -- 172.21.15.16:0/4110933893 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72dc0b6b80 con 0x7f72f40c9b40 2023-05-22T15:46:28.131 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.119+0000 7f72f9d08640 1 -- 172.21.15.16:0/4110933893 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72f40c8e80 con 0x7f72f40c9b40 2023-05-22T15:46:28.131 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.119+0000 7f72f9d08640 1 -- 172.21.15.16:0/4110933893 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72f40ca8b0 con 0x7f72f40c9b40 2023-05-22T15:46:28.131 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.119+0000 7f72f9d08640 1 -- 172.21.15.16:0/4110933893 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72f40c9010 con 0x7f72f40c9b40 2023-05-22T15:46:28.131 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.119+0000 7f72ebfff640 1 -- 172.21.15.16:0/4110933893 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 60) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f72dc0b6ce0 con 0x7f72f40c9b40 2023-05-22T15:46:28.131 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.119+0000 7f72ebfff640 1 --2- 172.21.15.16:0/4110933893 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f72d40654a0 0x7f72d4067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:28.131 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.119+0000 7f72ebfff640 1 -- 172.21.15.16:0/4110933893 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(147..147 src has 1..147) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f72dc120b60 con 0x7f72f40c9b40 2023-05-22T15:46:28.131 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.119+0000 7f72f37fe640 1 --2- 172.21.15.16:0/4110933893 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f72d40654a0 0x7f72d4067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:28.132 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.123+0000 7f72f37fe640 1 --2- 172.21.15.16:0/4110933893 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f72d40654a0 0x7f72d4067950 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f72e400b930 tx=0x7f72e4008040 comp rx=0 tx=0).ready entity=mgr.8835 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:28.132 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.123+0000 7f72ebfff640 1 -- 172.21.15.16:0/4110933893 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f72dc0ee600 con 0x7f72f40c9b40 2023-05-22T15:46:28.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.275+0000 7f72f9d08640 1 -- 172.21.15.16:0/4110933893 --> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f72f415afa0 con 0x7f72d40654a0 2023-05-22T15:46:28.302 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.295+0000 7f72ebfff640 1 -- 172.21.15.16:0/4110933893 <== mgr.8835 v2:172.21.15.16:6832/36921 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+307001 (secure 0 0 0) 0x7f72f415afa0 con 0x7f72d40654a0 2023-05-22T15:46:28.303 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:46:28.306 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2023-05-22T15:46:28.307 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.303+0000 7f72f9d08640 1 -- 172.21.15.16:0/4110933893 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f72d40654a0 msgr2=0x7f72d4067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:28.307 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.303+0000 7f72f9d08640 1 --2- 172.21.15.16:0/4110933893 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f72d40654a0 0x7f72d4067950 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f72e400b930 tx=0x7f72e4008040 comp rx=0 tx=0).stop 2023-05-22T15:46:28.307 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.303+0000 7f72f9d08640 1 -- 172.21.15.16:0/4110933893 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f72f40c9b40 msgr2=0x7f72f414c070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:28.308 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.303+0000 7f72f9d08640 1 --2- 172.21.15.16:0/4110933893 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f72f40c9b40 0x7f72f414c070 secure :-1 s=READY pgs=1847 cs=0 l=1 rev1=1 crypto rx=0x7f72dc01aa70 tx=0x7f72dc019160 comp rx=0 tx=0).stop 2023-05-22T15:46:28.308 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.303+0000 7f72f9d08640 1 -- 172.21.15.16:0/4110933893 shutdown_connections 2023-05-22T15:46:28.308 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.303+0000 7f72f9d08640 1 --2- 172.21.15.16:0/4110933893 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f72f40ca4e0 0x7f72f414c5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:28.308 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.303+0000 7f72f9d08640 1 --2- 172.21.15.16:0/4110933893 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f72f40c9b40 0x7f72f414c070 unknown :-1 s=CLOSED pgs=1847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:28.308 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.303+0000 7f72f9d08640 1 --2- 172.21.15.16:0/4110933893 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f72d40654a0 0x7f72d4067950 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:28.309 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.303+0000 7f72f9d08640 1 --2- 172.21.15.16:0/4110933893 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f72f4070df0 0x7f72f40c5400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:28.309 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.303+0000 7f72f9d08640 1 -- 172.21.15.16:0/4110933893 >> 172.21.15.16:0/4110933893 conn(0x7f72f4069720 msgr2=0x7f72f40712a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:46:28.309 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.303+0000 7f72f9d08640 1 -- 172.21.15.16:0/4110933893 shutdown_connections 2023-05-22T15:46:28.310 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.303+0000 7f72f9d08640 1 -- 172.21.15.16:0/4110933893 wait complete. 2023-05-22T15:46:28.327 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":26,"stamp":"2023-05-22T15:46:27.369508+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1268,"num_read_kb":2572,"num_write":399,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":249,"ondisk_log_size":249,"up":210,"acting":210,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":33,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":224768,"kb_used_data":15168,"kb_used_omap":0,"kb_used_meta":209536,"kb_avail":754749952,"statfs":{"total":773094113280,"available":772863950848,"internally_reserved":0,"allocated":15532032,"data_stored":8756104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":214564864},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":62,"apply_latency_ms":62,"commit_latency_ns":62000000,"apply_latency_ns":62000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.321201"},"pg_stats":[{"pgid":"19.3f","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.079122+0000","last_change":"2023-05-22T15:46:23.079122+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.079122+0000","last_undegraded":"2023-05-22T15:46:23.079122+0000","last_fullsized":"2023-05-22T15:46:23.079122+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:57:19.852836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.3e","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.076282+0000","last_change":"2023-05-22T15:46:23.076282+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.076282+0000","last_undegraded":"2023-05-22T15:46:23.076282+0000","last_fullsized":"2023-05-22T15:46:23.076282+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T03:20:39.433281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.3d","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.072443+0000","last_change":"2023-05-22T15:46:23.072443+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.072443+0000","last_undegraded":"2023-05-22T15:46:23.072443+0000","last_fullsized":"2023-05-22T15:46:23.072443+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:46:28.227459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.3c","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.077931+0000","last_change":"2023-05-22T15:46:23.077931+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.077931+0000","last_undegraded":"2023-05-22T15:46:23.077931+0000","last_fullsized":"2023-05-22T15:46:23.077931+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:42:45.119566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.3b","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.090321+0000","last_change":"2023-05-22T15:46:23.090321+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.090321+0000","last_undegraded":"2023-05-22T15:46:23.090321+0000","last_fullsized":"2023-05-22T15:46:23.090321+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:59:24.451915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.3a","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.072441+0000","last_change":"2023-05-22T15:46:23.072441+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.072441+0000","last_undegraded":"2023-05-22T15:46:23.072441+0000","last_fullsized":"2023-05-22T15:46:23.072441+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:25:55.107784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.39","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.076980+0000","last_change":"2023-05-22T15:46:23.076980+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.076980+0000","last_undegraded":"2023-05-22T15:46:23.076980+0000","last_fullsized":"2023-05-22T15:46:23.076980+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:29:20.717884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.38","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.078358+0000","last_change":"2023-05-22T15:46:23.078358+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.078358+0000","last_undegraded":"2023-05-22T15:46:23.078358+0000","last_fullsized":"2023-05-22T15:46:23.078358+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:51:08.858142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.37","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.074307+0000","last_change":"2023-05-22T15:46:23.074307+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.074307+0000","last_undegraded":"2023-05-22T15:46:23.074307+0000","last_fullsized":"2023-05-22T15:46:23.074307+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:28:23.397061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.36","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.075645+0000","last_change":"2023-05-22T15:46:23.075645+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.075645+0000","last_undegraded":"2023-05-22T15:46:23.075645+0000","last_fullsized":"2023-05-22T15:46:23.075645+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:46:28.227459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.35","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.069852+0000","last_change":"2023-05-22T15:46:23.069852+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.069852+0000","last_undegraded":"2023-05-22T15:46:23.069852+0000","last_fullsized":"2023-05-22T15:46:23.069852+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T15:53:20.317234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.34","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.073019+0000","last_change":"2023-05-22T15:46:23.073019+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.073019+0000","last_undegraded":"2023-05-22T15:46:23.073019+0000","last_fullsized":"2023-05-22T15:46:23.073019+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T18:52:29.352539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.33","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.078530+0000","last_change":"2023-05-22T15:46:23.078530+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.078530+0000","last_undegraded":"2023-05-22T15:46:23.078530+0000","last_fullsized":"2023-05-22T15:46:23.078530+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:22:06.668010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.32","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.072337+0000","last_change":"2023-05-22T15:46:23.072337+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.072337+0000","last_undegraded":"2023-05-22T15:46:23.072337+0000","last_fullsized":"2023-05-22T15:46:23.072337+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T18:05:16.988218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.31","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.074359+0000","last_change":"2023-05-22T15:46:23.074359+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.074359+0000","last_undegraded":"2023-05-22T15:46:23.074359+0000","last_fullsized":"2023-05-22T15:46:23.074359+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T18:05:16.988218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.30","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.063646+0000","last_change":"2023-05-22T15:46:23.063646+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.063646+0000","last_undegraded":"2023-05-22T15:46:23.063646+0000","last_fullsized":"2023-05-22T15:46:23.063646+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:30:04.941241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2f","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.075803+0000","last_change":"2023-05-22T15:46:23.075803+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.075803+0000","last_undegraded":"2023-05-22T15:46:23.075803+0000","last_fullsized":"2023-05-22T15:46:23.075803+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:23:42.844820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.2e","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.064448+0000","last_change":"2023-05-22T15:46:23.064448+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.064448+0000","last_undegraded":"2023-05-22T15:46:23.064448+0000","last_fullsized":"2023-05-22T15:46:23.064448+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:23:58.635276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2d","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.068994+0000","last_change":"2023-05-22T15:46:23.068994+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.068994+0000","last_undegraded":"2023-05-22T15:46:23.068994+0000","last_fullsized":"2023-05-22T15:46:23.068994+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:38:41.516579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.2c","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.062047+0000","last_change":"2023-05-22T15:46:23.062047+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.062047+0000","last_undegraded":"2023-05-22T15:46:23.062047+0000","last_fullsized":"2023-05-22T15:46:23.062047+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:52:08.264737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2b","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.093524+0000","last_change":"2023-05-22T15:46:23.093524+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.093524+0000","last_undegraded":"2023-05-22T15:46:23.093524+0000","last_fullsized":"2023-05-22T15:46:23.093524+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:55:41.010860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.2a","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.069379+0000","last_change":"2023-05-22T15:46:23.069379+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.069379+0000","last_undegraded":"2023-05-22T15:46:23.069379+0000","last_fullsized":"2023-05-22T15:46:23.069379+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:36:49.326554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.29","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.081591+0000","last_change":"2023-05-22T15:46:23.081591+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.081591+0000","last_undegraded":"2023-05-22T15:46:23.081591+0000","last_fullsized":"2023-05-22T15:46:23.081591+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:55:51.486171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.28","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.070163+0000","last_change":"2023-05-22T15:46:23.070163+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.070163+0000","last_undegraded":"2023-05-22T15:46:23.070163+0000","last_fullsized":"2023-05-22T15:46:23.070163+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:21:39.187868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.27","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.071496+0000","last_change":"2023-05-22T15:46:23.071496+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.071496+0000","last_undegraded":"2023-05-22T15:46:23.071496+0000","last_fullsized":"2023-05-22T15:46:23.071496+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:37:26.183234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.26","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.063189+0000","last_change":"2023-05-22T15:46:23.063189+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.063189+0000","last_undegraded":"2023-05-22T15:46:23.063189+0000","last_fullsized":"2023-05-22T15:46:23.063189+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:43:11.401376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.25","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.074484+0000","last_change":"2023-05-22T15:46:23.074484+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.074484+0000","last_undegraded":"2023-05-22T15:46:23.074484+0000","last_fullsized":"2023-05-22T15:46:23.074484+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T03:43:40.124163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.24","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.064681+0000","last_change":"2023-05-22T15:46:23.064681+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.064681+0000","last_undegraded":"2023-05-22T15:46:23.064681+0000","last_fullsized":"2023-05-22T15:46:23.064681+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:20:29.148898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.23","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.077284+0000","last_change":"2023-05-22T15:46:23.077284+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.077284+0000","last_undegraded":"2023-05-22T15:46:23.077284+0000","last_fullsized":"2023-05-22T15:46:23.077284+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:34:43.855049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.22","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.073667+0000","last_change":"2023-05-22T15:46:23.073667+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.073667+0000","last_undegraded":"2023-05-22T15:46:23.073667+0000","last_fullsized":"2023-05-22T15:46:23.073667+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T16:10:54.516325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.21","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.086294+0000","last_change":"2023-05-22T15:46:23.086294+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.086294+0000","last_undegraded":"2023-05-22T15:46:23.086294+0000","last_fullsized":"2023-05-22T15:46:23.086294+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:20:12.734869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.20","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.074629+0000","last_change":"2023-05-22T15:46:23.074629+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.074629+0000","last_undegraded":"2023-05-22T15:46:23.074629+0000","last_fullsized":"2023-05-22T15:46:23.074629+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:52:03.083872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.1e","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.061067+0000","last_change":"2023-05-22T15:46:23.061067+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.061067+0000","last_undegraded":"2023-05-22T15:46:23.061067+0000","last_fullsized":"2023-05-22T15:46:23.061067+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:29:20.717884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1f","version":"0'0","reported_seq":9,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.120601+0000","last_change":"2023-05-22T15:46:23.112442+0000","last_active":"2023-05-22T15:46:23.120601+0000","last_peered":"2023-05-22T15:46:23.120601+0000","last_clean":"2023-05-22T15:46:23.120601+0000","last_became_active":"2023-05-22T15:46:23.111561+0000","last_became_peered":"2023-05-22T15:46:23.111561+0000","last_unstale":"2023-05-22T15:46:23.120601+0000","last_undegraded":"2023-05-22T15:46:23.120601+0000","last_fullsized":"2023-05-22T15:46:23.120601+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_clean_scrub_stamp":"2023-05-22T15:46:22.034488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:28:22.384974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.1f","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.071619+0000","last_change":"2023-05-22T15:46:23.071619+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.071619+0000","last_undegraded":"2023-05-22T15:46:23.071619+0000","last_fullsized":"2023-05-22T15:46:23.071619+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:22:11.366110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.1e","version":"0'0","reported_seq":9,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.092179+0000","last_change":"2023-05-22T15:46:23.078407+0000","last_active":"2023-05-22T15:46:23.092179+0000","last_peered":"2023-05-22T15:46:23.092179+0000","last_clean":"2023-05-22T15:46:23.092179+0000","last_became_active":"2023-05-22T15:46:23.077477+0000","last_became_peered":"2023-05-22T15:46:23.077477+0000","last_unstale":"2023-05-22T15:46:23.092179+0000","last_undegraded":"2023-05-22T15:46:23.092179+0000","last_fullsized":"2023-05-22T15:46:23.092179+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_clean_scrub_stamp":"2023-05-22T15:46:22.034488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:20:11.712441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.1c","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.089330+0000","last_change":"2023-05-22T15:46:23.089330+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.089330+0000","last_undegraded":"2023-05-22T15:46:23.089330+0000","last_fullsized":"2023-05-22T15:46:23.089330+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:47:45.260369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1d","version":"0'0","reported_seq":9,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.090669+0000","last_change":"2023-05-22T15:46:23.083700+0000","last_active":"2023-05-22T15:46:23.090669+0000","last_peered":"2023-05-22T15:46:23.090669+0000","last_clean":"2023-05-22T15:46:23.090669+0000","last_became_active":"2023-05-22T15:46:23.083350+0000","last_became_peered":"2023-05-22T15:46:23.083350+0000","last_unstale":"2023-05-22T15:46:23.090669+0000","last_undegraded":"2023-05-22T15:46:23.090669+0000","last_fullsized":"2023-05-22T15:46:23.090669+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_clean_scrub_stamp":"2023-05-22T15:46:22.034488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:03:14.130053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.1d","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.090978+0000","last_change":"2023-05-22T15:46:23.090978+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.090978+0000","last_undegraded":"2023-05-22T15:46:23.090978+0000","last_fullsized":"2023-05-22T15:46:23.090978+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:07:01.166140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1c","version":"0'0","reported_seq":9,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.094786+0000","last_change":"2023-05-22T15:46:23.092200+0000","last_active":"2023-05-22T15:46:23.094786+0000","last_peered":"2023-05-22T15:46:23.094786+0000","last_clean":"2023-05-22T15:46:23.094786+0000","last_became_active":"2023-05-22T15:46:23.089522+0000","last_became_peered":"2023-05-22T15:46:23.089522+0000","last_unstale":"2023-05-22T15:46:23.094786+0000","last_undegraded":"2023-05-22T15:46:23.094786+0000","last_fullsized":"2023-05-22T15:46:23.094786+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_clean_scrub_stamp":"2023-05-22T15:46:22.034488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:16:51.789402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.1a","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.061711+0000","last_change":"2023-05-22T15:46:23.061711+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.061711+0000","last_undegraded":"2023-05-22T15:46:23.061711+0000","last_fullsized":"2023-05-22T15:46:23.061711+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:23:42.844820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1b","version":"0'0","reported_seq":9,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.094709+0000","last_change":"2023-05-22T15:46:23.088945+0000","last_active":"2023-05-22T15:46:23.094709+0000","last_peered":"2023-05-22T15:46:23.094709+0000","last_clean":"2023-05-22T15:46:23.094709+0000","last_became_active":"2023-05-22T15:46:23.088536+0000","last_became_peered":"2023-05-22T15:46:23.088536+0000","last_unstale":"2023-05-22T15:46:23.094709+0000","last_undegraded":"2023-05-22T15:46:23.094709+0000","last_fullsized":"2023-05-22T15:46:23.094709+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_clean_scrub_stamp":"2023-05-22T15:46:22.034488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:12:41.914428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.1b","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.073120+0000","last_change":"2023-05-22T15:46:23.073120+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.073120+0000","last_undegraded":"2023-05-22T15:46:23.073120+0000","last_fullsized":"2023-05-22T15:46:23.073120+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:59:24.451915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.1a","version":"0'0","reported_seq":9,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.090491+0000","last_change":"2023-05-22T15:46:23.089182+0000","last_active":"2023-05-22T15:46:23.090491+0000","last_peered":"2023-05-22T15:46:23.090491+0000","last_clean":"2023-05-22T15:46:23.090491+0000","last_became_active":"2023-05-22T15:46:23.088748+0000","last_became_peered":"2023-05-22T15:46:23.088748+0000","last_unstale":"2023-05-22T15:46:23.090491+0000","last_undegraded":"2023-05-22T15:46:23.090491+0000","last_fullsized":"2023-05-22T15:46:23.090491+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_clean_scrub_stamp":"2023-05-22T15:46:22.034488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:49:33.716090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.18","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.077114+0000","last_change":"2023-05-22T15:46:23.077114+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.077114+0000","last_undegraded":"2023-05-22T15:46:23.077114+0000","last_fullsized":"2023-05-22T15:46:23.077114+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T03:20:39.103248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.19","version":"0'0","reported_seq":9,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.117596+0000","last_change":"2023-05-22T15:46:23.105440+0000","last_active":"2023-05-22T15:46:23.117596+0000","last_peered":"2023-05-22T15:46:23.117596+0000","last_clean":"2023-05-22T15:46:23.117596+0000","last_became_active":"2023-05-22T15:46:23.104687+0000","last_became_peered":"2023-05-22T15:46:23.104687+0000","last_unstale":"2023-05-22T15:46:23.117596+0000","last_undegraded":"2023-05-22T15:46:23.117596+0000","last_fullsized":"2023-05-22T15:46:23.117596+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_clean_scrub_stamp":"2023-05-22T15:46:22.034488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:51:07.846056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.19","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.075186+0000","last_change":"2023-05-22T15:46:23.075186+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.075186+0000","last_undegraded":"2023-05-22T15:46:23.075186+0000","last_fullsized":"2023-05-22T15:46:23.075186+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:22:20.631045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.18","version":"0'0","reported_seq":9,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.108549+0000","last_change":"2023-05-22T15:46:23.106544+0000","last_active":"2023-05-22T15:46:23.108549+0000","last_peered":"2023-05-22T15:46:23.108549+0000","last_clean":"2023-05-22T15:46:23.108549+0000","last_became_active":"2023-05-22T15:46:23.106098+0000","last_became_peered":"2023-05-22T15:46:23.106098+0000","last_unstale":"2023-05-22T15:46:23.108549+0000","last_undegraded":"2023-05-22T15:46:23.108549+0000","last_fullsized":"2023-05-22T15:46:23.108549+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_clean_scrub_stamp":"2023-05-22T15:46:22.034488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:42:03.486863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.e","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.083614+0000","last_change":"2023-05-22T15:46:23.083614+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.083614+0000","last_undegraded":"2023-05-22T15:46:23.083614+0000","last_fullsized":"2023-05-22T15:46:23.083614+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:20:55.836997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.f","version":"0'0","reported_seq":9,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.116428+0000","last_change":"2023-05-22T15:46:23.112873+0000","last_active":"2023-05-22T15:46:23.116428+0000","last_peered":"2023-05-22T15:46:23.116428+0000","last_clean":"2023-05-22T15:46:23.116428+0000","last_became_active":"2023-05-22T15:46:23.112082+0000","last_became_peered":"2023-05-22T15:46:23.112082+0000","last_unstale":"2023-05-22T15:46:23.116428+0000","last_undegraded":"2023-05-22T15:46:23.116428+0000","last_fullsized":"2023-05-22T15:46:23.116428+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_clean_scrub_stamp":"2023-05-22T15:46:22.034488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:24:32.309552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.f","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.063425+0000","last_change":"2023-05-22T15:46:23.063425+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.063425+0000","last_undegraded":"2023-05-22T15:46:23.063425+0000","last_fullsized":"2023-05-22T15:46:23.063425+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T16:10:54.516325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.e","version":"0'0","reported_seq":9,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.109486+0000","last_change":"2023-05-22T15:46:23.107165+0000","last_active":"2023-05-22T15:46:23.109486+0000","last_peered":"2023-05-22T15:46:23.109486+0000","last_clean":"2023-05-22T15:46:23.109486+0000","last_became_active":"2023-05-22T15:46:23.106681+0000","last_became_peered":"2023-05-22T15:46:23.106681+0000","last_unstale":"2023-05-22T15:46:23.109486+0000","last_undegraded":"2023-05-22T15:46:23.109486+0000","last_fullsized":"2023-05-22T15:46:23.109486+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_clean_scrub_stamp":"2023-05-22T15:46:22.034488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T15:59:17.155128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.c","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.060697+0000","last_change":"2023-05-22T15:46:23.060697+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.060697+0000","last_undegraded":"2023-05-22T15:46:23.060697+0000","last_fullsized":"2023-05-22T15:46:23.060697+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:33:34.943631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.d","version":"0'0","reported_seq":9,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.111448+0000","last_change":"2023-05-22T15:46:23.103043+0000","last_active":"2023-05-22T15:46:23.111448+0000","last_peered":"2023-05-22T15:46:23.111448+0000","last_clean":"2023-05-22T15:46:23.111448+0000","last_became_active":"2023-05-22T15:46:23.102220+0000","last_became_peered":"2023-05-22T15:46:23.102220+0000","last_unstale":"2023-05-22T15:46:23.111448+0000","last_undegraded":"2023-05-22T15:46:23.111448+0000","last_fullsized":"2023-05-22T15:46:23.111448+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_clean_scrub_stamp":"2023-05-22T15:46:22.034488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:20:54.824911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.d","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.073232+0000","last_change":"2023-05-22T15:46:23.073232+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.073232+0000","last_undegraded":"2023-05-22T15:46:23.073232+0000","last_fullsized":"2023-05-22T15:46:23.073232+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:49:48.656654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.c","version":"0'0","reported_seq":9,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.082235+0000","last_change":"2023-05-22T15:46:23.078278+0000","last_active":"2023-05-22T15:46:23.082235+0000","last_peered":"2023-05-22T15:46:23.082235+0000","last_clean":"2023-05-22T15:46:23.082235+0000","last_became_active":"2023-05-22T15:46:23.077343+0000","last_became_peered":"2023-05-22T15:46:23.077343+0000","last_unstale":"2023-05-22T15:46:23.082235+0000","last_undegraded":"2023-05-22T15:46:23.082235+0000","last_fullsized":"2023-05-22T15:46:23.082235+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_clean_scrub_stamp":"2023-05-22T15:46:22.034488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:17:43.848098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.a","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.072991+0000","last_change":"2023-05-22T15:46:23.072991+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.072991+0000","last_undegraded":"2023-05-22T15:46:23.072991+0000","last_fullsized":"2023-05-22T15:46:23.072991+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:24:33.321639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.b","version":"0'0","reported_seq":9,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.092758+0000","last_change":"2023-05-22T15:46:23.086857+0000","last_active":"2023-05-22T15:46:23.092758+0000","last_peered":"2023-05-22T15:46:23.092758+0000","last_clean":"2023-05-22T15:46:23.092758+0000","last_became_active":"2023-05-22T15:46:23.086098+0000","last_became_peered":"2023-05-22T15:46:23.086098+0000","last_unstale":"2023-05-22T15:46:23.092758+0000","last_undegraded":"2023-05-22T15:46:23.092758+0000","last_fullsized":"2023-05-22T15:46:23.092758+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_clean_scrub_stamp":"2023-05-22T15:46:22.034488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:45:50.595714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.b","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.071529+0000","last_change":"2023-05-22T15:46:23.071529+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.071529+0000","last_undegraded":"2023-05-22T15:46:23.071529+0000","last_fullsized":"2023-05-22T15:46:23.071529+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:20:12.734869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.a","version":"0'0","reported_seq":9,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.107500+0000","last_change":"2023-05-22T15:46:23.106615+0000","last_active":"2023-05-22T15:46:23.107500+0000","last_peered":"2023-05-22T15:46:23.107500+0000","last_clean":"2023-05-22T15:46:23.107500+0000","last_became_active":"2023-05-22T15:46:23.106155+0000","last_became_peered":"2023-05-22T15:46:23.106155+0000","last_unstale":"2023-05-22T15:46:23.107500+0000","last_undegraded":"2023-05-22T15:46:23.107500+0000","last_fullsized":"2023-05-22T15:46:23.107500+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_clean_scrub_stamp":"2023-05-22T15:46:22.034488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:55:28.593183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.8","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.071955+0000","last_change":"2023-05-22T15:46:23.071955+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.071955+0000","last_undegraded":"2023-05-22T15:46:23.071955+0000","last_fullsized":"2023-05-22T15:46:23.071955+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T16:27:47.997551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.9","version":"0'0","reported_seq":9,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.082561+0000","last_change":"2023-05-22T15:46:23.080736+0000","last_active":"2023-05-22T15:46:23.082561+0000","last_peered":"2023-05-22T15:46:23.082561+0000","last_clean":"2023-05-22T15:46:23.082561+0000","last_became_active":"2023-05-22T15:46:23.080251+0000","last_became_peered":"2023-05-22T15:46:23.080251+0000","last_unstale":"2023-05-22T15:46:23.082561+0000","last_undegraded":"2023-05-22T15:46:23.082561+0000","last_fullsized":"2023-05-22T15:46:23.082561+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_clean_scrub_stamp":"2023-05-22T15:46:22.034488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T16:52:22.575852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.9","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.089066+0000","last_change":"2023-05-22T15:46:23.089066+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.089066+0000","last_undegraded":"2023-05-22T15:46:23.089066+0000","last_fullsized":"2023-05-22T15:46:23.089066+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:31:09.541110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.8","version":"0'0","reported_seq":9,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.109492+0000","last_change":"2023-05-22T15:46:23.107191+0000","last_active":"2023-05-22T15:46:23.109492+0000","last_peered":"2023-05-22T15:46:23.109492+0000","last_clean":"2023-05-22T15:46:23.109492+0000","last_became_active":"2023-05-22T15:46:23.106653+0000","last_became_peered":"2023-05-22T15:46:23.106653+0000","last_unstale":"2023-05-22T15:46:23.109492+0000","last_undegraded":"2023-05-22T15:46:23.109492+0000","last_fullsized":"2023-05-22T15:46:23.109492+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_clean_scrub_stamp":"2023-05-22T15:46:22.034488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:00:00.382995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.6","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.076413+0000","last_change":"2023-05-22T15:46:23.076413+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.076413+0000","last_undegraded":"2023-05-22T15:46:23.076413+0000","last_fullsized":"2023-05-22T15:46:23.076413+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:55:41.010860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.7","version":"0'0","reported_seq":9,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.092881+0000","last_change":"2023-05-22T15:46:23.091817+0000","last_active":"2023-05-22T15:46:23.092881+0000","last_peered":"2023-05-22T15:46:23.092881+0000","last_clean":"2023-05-22T15:46:23.092881+0000","last_became_active":"2023-05-22T15:46:23.091434+0000","last_became_peered":"2023-05-22T15:46:23.091434+0000","last_unstale":"2023-05-22T15:46:23.092881+0000","last_undegraded":"2023-05-22T15:46:23.092881+0000","last_fullsized":"2023-05-22T15:46:23.092881+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_clean_scrub_stamp":"2023-05-22T15:46:22.034488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T16:37:19.990243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.7","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.079702+0000","last_change":"2023-05-22T15:46:23.079702+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.079702+0000","last_undegraded":"2023-05-22T15:46:23.079702+0000","last_fullsized":"2023-05-22T15:46:23.079702+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:20:29.148898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.6","version":"0'0","reported_seq":9,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.120724+0000","last_change":"2023-05-22T15:46:23.114945+0000","last_active":"2023-05-22T15:46:23.120724+0000","last_peered":"2023-05-22T15:46:23.120724+0000","last_clean":"2023-05-22T15:46:23.120724+0000","last_became_active":"2023-05-22T15:46:23.114027+0000","last_became_peered":"2023-05-22T15:46:23.114027+0000","last_unstale":"2023-05-22T15:46:23.120724+0000","last_undegraded":"2023-05-22T15:46:23.120724+0000","last_fullsized":"2023-05-22T15:46:23.120724+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_clean_scrub_stamp":"2023-05-22T15:46:22.034488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T18:52:28.340453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.4","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.075841+0000","last_change":"2023-05-22T15:46:23.075841+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.075841+0000","last_undegraded":"2023-05-22T15:46:23.075841+0000","last_fullsized":"2023-05-22T15:46:23.075841+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:46:38.837010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.5","version":"0'0","reported_seq":9,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.109205+0000","last_change":"2023-05-22T15:46:23.108380+0000","last_active":"2023-05-22T15:46:23.109205+0000","last_peered":"2023-05-22T15:46:23.109205+0000","last_clean":"2023-05-22T15:46:23.109205+0000","last_became_active":"2023-05-22T15:46:23.108014+0000","last_became_peered":"2023-05-22T15:46:23.108014+0000","last_unstale":"2023-05-22T15:46:23.109205+0000","last_undegraded":"2023-05-22T15:46:23.109205+0000","last_fullsized":"2023-05-22T15:46:23.109205+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_clean_scrub_stamp":"2023-05-22T15:46:22.034488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T03:37:19.248335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.5","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.073829+0000","last_change":"2023-05-22T15:46:23.073829+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.073829+0000","last_undegraded":"2023-05-22T15:46:23.073829+0000","last_fullsized":"2023-05-22T15:46:23.073829+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:47:45.260369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.4","version":"0'0","reported_seq":9,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.094601+0000","last_change":"2023-05-22T15:46:23.093730+0000","last_active":"2023-05-22T15:46:23.094601+0000","last_peered":"2023-05-22T15:46:23.094601+0000","last_clean":"2023-05-22T15:46:23.094601+0000","last_became_active":"2023-05-22T15:46:23.093418+0000","last_became_peered":"2023-05-22T15:46:23.093418+0000","last_unstale":"2023-05-22T15:46:23.094601+0000","last_undegraded":"2023-05-22T15:46:23.094601+0000","last_fullsized":"2023-05-22T15:46:23.094601+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_clean_scrub_stamp":"2023-05-22T15:46:22.034488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T16:38:30.323718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.2","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.084350+0000","last_change":"2023-05-22T15:46:23.084350+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.084350+0000","last_undegraded":"2023-05-22T15:46:23.084350+0000","last_fullsized":"2023-05-22T15:46:23.084350+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:25:55.107784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.3","version":"0'0","reported_seq":9,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.111036+0000","last_change":"2023-05-22T15:46:23.110053+0000","last_active":"2023-05-22T15:46:23.111036+0000","last_peered":"2023-05-22T15:46:23.111036+0000","last_clean":"2023-05-22T15:46:23.111036+0000","last_became_active":"2023-05-22T15:46:23.109753+0000","last_became_peered":"2023-05-22T15:46:23.109753+0000","last_unstale":"2023-05-22T15:46:23.111036+0000","last_undegraded":"2023-05-22T15:46:23.111036+0000","last_fullsized":"2023-05-22T15:46:23.111036+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_clean_scrub_stamp":"2023-05-22T15:46:22.034488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T18:29:44.257708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.3","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.062480+0000","last_change":"2023-05-22T15:46:23.062480+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.062480+0000","last_undegraded":"2023-05-22T15:46:23.062480+0000","last_fullsized":"2023-05-22T15:46:23.062480+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T18:05:16.988218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.2","version":"0'0","reported_seq":9,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.120623+0000","last_change":"2023-05-22T15:46:23.112868+0000","last_active":"2023-05-22T15:46:23.120623+0000","last_peered":"2023-05-22T15:46:23.120623+0000","last_clean":"2023-05-22T15:46:23.120623+0000","last_became_active":"2023-05-22T15:46:23.112288+0000","last_became_peered":"2023-05-22T15:46:23.112288+0000","last_unstale":"2023-05-22T15:46:23.120623+0000","last_undegraded":"2023-05-22T15:46:23.120623+0000","last_fullsized":"2023-05-22T15:46:23.120623+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_clean_scrub_stamp":"2023-05-22T15:46:22.034488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:16:19.873549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.0","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.062429+0000","last_change":"2023-05-22T15:46:23.062429+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.062429+0000","last_undegraded":"2023-05-22T15:46:23.062429+0000","last_fullsized":"2023-05-22T15:46:23.062429+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:03:37.104570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1","version":"0'0","reported_seq":9,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.092294+0000","last_change":"2023-05-22T15:46:23.091423+0000","last_active":"2023-05-22T15:46:23.092294+0000","last_peered":"2023-05-22T15:46:23.092294+0000","last_clean":"2023-05-22T15:46:23.092294+0000","last_became_active":"2023-05-22T15:46:23.091088+0000","last_became_peered":"2023-05-22T15:46:23.091088+0000","last_unstale":"2023-05-22T15:46:23.092294+0000","last_undegraded":"2023-05-22T15:46:23.092294+0000","last_fullsized":"2023-05-22T15:46:23.092294+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_clean_scrub_stamp":"2023-05-22T15:46:22.034488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:52:16.638521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.1","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.072920+0000","last_change":"2023-05-22T15:46:23.072920+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.072920+0000","last_undegraded":"2023-05-22T15:46:23.072920+0000","last_fullsized":"2023-05-22T15:46:23.072920+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:16:20.885635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.0","version":"0'0","reported_seq":9,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.107395+0000","last_change":"2023-05-22T15:46:23.105214+0000","last_active":"2023-05-22T15:46:23.107395+0000","last_peered":"2023-05-22T15:46:23.107395+0000","last_clean":"2023-05-22T15:46:23.107395+0000","last_became_active":"2023-05-22T15:46:23.104909+0000","last_became_peered":"2023-05-22T15:46:23.104909+0000","last_unstale":"2023-05-22T15:46:23.107395+0000","last_undegraded":"2023-05-22T15:46:23.107395+0000","last_fullsized":"2023-05-22T15:46:23.107395+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_clean_scrub_stamp":"2023-05-22T15:46:22.034488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:37:03.818368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.12","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.085873+0000","last_change":"2023-05-22T15:46:23.085873+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.085873+0000","last_undegraded":"2023-05-22T15:46:23.085873+0000","last_fullsized":"2023-05-22T15:46:23.085873+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:20:55.836997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.13","version":"0'0","reported_seq":9,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.120574+0000","last_change":"2023-05-22T15:46:23.112061+0000","last_active":"2023-05-22T15:46:23.120574+0000","last_peered":"2023-05-22T15:46:23.120574+0000","last_clean":"2023-05-22T15:46:23.120574+0000","last_became_active":"2023-05-22T15:46:23.111315+0000","last_became_peered":"2023-05-22T15:46:23.111315+0000","last_unstale":"2023-05-22T15:46:23.120574+0000","last_undegraded":"2023-05-22T15:46:23.120574+0000","last_fullsized":"2023-05-22T15:46:23.120574+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_clean_scrub_stamp":"2023-05-22T15:46:22.034488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:37:25.171147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"134'249","reported_seq":753,"reported_epoch":144,"state":"active+clean","last_fresh":"2023-05-22T15:46:22.069225+0000","last_change":"2023-05-22T15:26:39.318204+0000","last_active":"2023-05-22T15:46:22.069225+0000","last_peered":"2023-05-22T15:46:22.069225+0000","last_clean":"2023-05-22T15:46:22.069225+0000","last_became_active":"2023-05-22T15:26:39.315370+0000","last_became_peered":"2023-05-22T15:26:39.315370+0000","last_unstale":"2023-05-22T15:46:22.069225+0000","last_undegraded":"2023-05-22T15:46:22.069225+0000","last_fullsized":"2023-05-22T15:46:22.069225+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:38.020076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:38.020076+0000","last_clean_scrub_stamp":"2023-05-22T15:26:38.020076+0000","objects_scrubbed":0,"log_size":249,"log_dups_size":0,"ondisk_log_size":249,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:06:39.207543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1268,"num_read_kb":2572,"num_write":399,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":139,"reported_epoch":144,"state":"active+clean","last_fresh":"2023-05-22T15:46:22.059466+0000","last_change":"2023-05-22T15:26:44.971951+0000","last_active":"2023-05-22T15:46:22.059466+0000","last_peered":"2023-05-22T15:46:22.059466+0000","last_clean":"2023-05-22T15:46:22.059466+0000","last_became_active":"2023-05-22T15:26:44.970872+0000","last_became_peered":"2023-05-22T15:26:44.970872+0000","last_unstale":"2023-05-22T15:46:22.059466+0000","last_undegraded":"2023-05-22T15:46:22.059466+0000","last_fullsized":"2023-05-22T15:46:22.059466+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:55:34.752949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.11","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.082774+0000","last_change":"2023-05-22T15:46:23.082774+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.082774+0000","last_undegraded":"2023-05-22T15:46:23.082774+0000","last_fullsized":"2023-05-22T15:46:23.082774+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:37:33.613777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.10","version":"0'0","reported_seq":9,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.107838+0000","last_change":"2023-05-22T15:46:23.091282+0000","last_active":"2023-05-22T15:46:23.107838+0000","last_peered":"2023-05-22T15:46:23.107838+0000","last_clean":"2023-05-22T15:46:23.107838+0000","last_became_active":"2023-05-22T15:46:23.086640+0000","last_became_peered":"2023-05-22T15:46:23.086640+0000","last_unstale":"2023-05-22T15:46:23.107838+0000","last_undegraded":"2023-05-22T15:46:23.107838+0000","last_fullsized":"2023-05-22T15:46:23.107838+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_clean_scrub_stamp":"2023-05-22T15:46:22.034488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T16:27:46.985465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":132,"reported_epoch":144,"state":"active+clean","last_fresh":"2023-05-22T15:46:22.068980+0000","last_change":"2023-05-22T15:26:44.977198+0000","last_active":"2023-05-22T15:46:22.068980+0000","last_peered":"2023-05-22T15:46:22.068980+0000","last_clean":"2023-05-22T15:46:22.068980+0000","last_became_active":"2023-05-22T15:26:44.976445+0000","last_became_peered":"2023-05-22T15:26:44.976445+0000","last_unstale":"2023-05-22T15:46:22.068980+0000","last_undegraded":"2023-05-22T15:46:22.068980+0000","last_fullsized":"2023-05-22T15:46:22.068980+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:12:41.464054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.10","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.073353+0000","last_change":"2023-05-22T15:46:23.073353+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.073353+0000","last_undegraded":"2023-05-22T15:46:23.073353+0000","last_fullsized":"2023-05-22T15:46:23.073353+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:42:45.119566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.11","version":"0'0","reported_seq":9,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.103893+0000","last_change":"2023-05-22T15:46:23.102815+0000","last_active":"2023-05-22T15:46:23.103893+0000","last_peered":"2023-05-22T15:46:23.103893+0000","last_clean":"2023-05-22T15:46:23.103893+0000","last_became_active":"2023-05-22T15:46:23.102405+0000","last_became_peered":"2023-05-22T15:46:23.102405+0000","last_unstale":"2023-05-22T15:46:23.103893+0000","last_undegraded":"2023-05-22T15:46:23.103893+0000","last_fullsized":"2023-05-22T15:46:23.103893+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_clean_scrub_stamp":"2023-05-22T15:46:22.034488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:45:50.595714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":132,"reported_epoch":144,"state":"active+clean","last_fresh":"2023-05-22T15:46:22.051914+0000","last_change":"2023-05-22T15:26:44.969469+0000","last_active":"2023-05-22T15:46:22.051914+0000","last_peered":"2023-05-22T15:46:22.051914+0000","last_clean":"2023-05-22T15:46:22.051914+0000","last_became_active":"2023-05-22T15:26:44.969083+0000","last_became_peered":"2023-05-22T15:26:44.969083+0000","last_unstale":"2023-05-22T15:46:22.051914+0000","last_undegraded":"2023-05-22T15:46:22.051914+0000","last_fullsized":"2023-05-22T15:46:22.051914+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:29:23.670067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.13","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.076508+0000","last_change":"2023-05-22T15:46:23.076508+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.076508+0000","last_undegraded":"2023-05-22T15:46:23.076508+0000","last_fullsized":"2023-05-22T15:46:23.076508+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T16:27:47.997551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.12","version":"0'0","reported_seq":9,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.107682+0000","last_change":"2023-05-22T15:46:23.105368+0000","last_active":"2023-05-22T15:46:23.107682+0000","last_peered":"2023-05-22T15:46:23.107682+0000","last_clean":"2023-05-22T15:46:23.107682+0000","last_became_active":"2023-05-22T15:46:23.104771+0000","last_became_peered":"2023-05-22T15:46:23.104771+0000","last_unstale":"2023-05-22T15:46:23.107682+0000","last_undegraded":"2023-05-22T15:46:23.107682+0000","last_fullsized":"2023-05-22T15:46:23.107682+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_clean_scrub_stamp":"2023-05-22T15:46:22.034488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:42:03.486863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":139,"reported_epoch":144,"state":"active+clean","last_fresh":"2023-05-22T15:46:22.059265+0000","last_change":"2023-05-22T15:26:44.976632+0000","last_active":"2023-05-22T15:46:22.059265+0000","last_peered":"2023-05-22T15:46:22.059265+0000","last_clean":"2023-05-22T15:46:22.059265+0000","last_became_active":"2023-05-22T15:26:44.975546+0000","last_became_peered":"2023-05-22T15:26:44.975546+0000","last_unstale":"2023-05-22T15:46:22.059265+0000","last_undegraded":"2023-05-22T15:46:22.059265+0000","last_fullsized":"2023-05-22T15:46:22.059265+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:42:57.973030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.15","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.084525+0000","last_change":"2023-05-22T15:46:23.084525+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.084525+0000","last_undegraded":"2023-05-22T15:46:23.084525+0000","last_fullsized":"2023-05-22T15:46:23.084525+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T03:25:18.349611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.14","version":"0'0","reported_seq":9,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.109240+0000","last_change":"2023-05-22T15:46:23.104870+0000","last_active":"2023-05-22T15:46:23.109240+0000","last_peered":"2023-05-22T15:46:23.109240+0000","last_clean":"2023-05-22T15:46:23.109240+0000","last_became_active":"2023-05-22T15:46:23.104126+0000","last_became_peered":"2023-05-22T15:46:23.104126+0000","last_unstale":"2023-05-22T15:46:23.109240+0000","last_undegraded":"2023-05-22T15:46:23.109240+0000","last_fullsized":"2023-05-22T15:46:23.109240+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_clean_scrub_stamp":"2023-05-22T15:46:22.034488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:20:28.136812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":123,"reported_epoch":144,"state":"active+clean","last_fresh":"2023-05-22T15:46:22.065910+0000","last_change":"2023-05-22T15:26:45.091475+0000","last_active":"2023-05-22T15:46:22.065910+0000","last_peered":"2023-05-22T15:46:22.065910+0000","last_clean":"2023-05-22T15:46:22.065910+0000","last_became_active":"2023-05-22T15:26:45.091143+0000","last_became_peered":"2023-05-22T15:26:45.091143+0000","last_unstale":"2023-05-22T15:46:22.065910+0000","last_undegraded":"2023-05-22T15:46:22.065910+0000","last_fullsized":"2023-05-22T15:46:22.065910+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:12:41.464054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.14","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.088461+0000","last_change":"2023-05-22T15:46:23.088461+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.088461+0000","last_undegraded":"2023-05-22T15:46:23.088461+0000","last_fullsized":"2023-05-22T15:46:23.088461+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:49:34.728176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.15","version":"0'0","reported_seq":9,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.092614+0000","last_change":"2023-05-22T15:46:23.083261+0000","last_active":"2023-05-22T15:46:23.092614+0000","last_peered":"2023-05-22T15:46:23.092614+0000","last_clean":"2023-05-22T15:46:23.092614+0000","last_became_active":"2023-05-22T15:46:23.082317+0000","last_became_peered":"2023-05-22T15:46:23.082317+0000","last_unstale":"2023-05-22T15:46:23.092614+0000","last_undegraded":"2023-05-22T15:46:23.092614+0000","last_fullsized":"2023-05-22T15:46:23.092614+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_clean_scrub_stamp":"2023-05-22T15:46:22.034488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T18:29:44.257708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":132,"reported_epoch":144,"state":"active+clean","last_fresh":"2023-05-22T15:46:22.069477+0000","last_change":"2023-05-22T15:26:45.092367+0000","last_active":"2023-05-22T15:46:22.069477+0000","last_peered":"2023-05-22T15:46:22.069477+0000","last_clean":"2023-05-22T15:46:22.069477+0000","last_became_active":"2023-05-22T15:26:45.091951+0000","last_became_peered":"2023-05-22T15:26:45.091951+0000","last_unstale":"2023-05-22T15:46:22.069477+0000","last_undegraded":"2023-05-22T15:46:22.069477+0000","last_fullsized":"2023-05-22T15:46:22.069477+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:42:57.973030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.17","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.065336+0000","last_change":"2023-05-22T15:46:23.065336+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.065336+0000","last_undegraded":"2023-05-22T15:46:23.065336+0000","last_fullsized":"2023-05-22T15:46:23.065336+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T16:27:47.997551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.16","version":"0'0","reported_seq":9,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.107509+0000","last_change":"2023-05-22T15:46:23.106558+0000","last_active":"2023-05-22T15:46:23.107509+0000","last_peered":"2023-05-22T15:46:23.107509+0000","last_clean":"2023-05-22T15:46:23.107509+0000","last_became_active":"2023-05-22T15:46:23.106098+0000","last_became_peered":"2023-05-22T15:46:23.106098+0000","last_unstale":"2023-05-22T15:46:23.107509+0000","last_undegraded":"2023-05-22T15:46:23.107509+0000","last_fullsized":"2023-05-22T15:46:23.107509+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_clean_scrub_stamp":"2023-05-22T15:46:22.034488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:30:32.850917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":123,"reported_epoch":144,"state":"active+clean","last_fresh":"2023-05-22T15:46:22.065875+0000","last_change":"2023-05-22T15:26:44.978858+0000","last_active":"2023-05-22T15:46:22.065875+0000","last_peered":"2023-05-22T15:46:22.065875+0000","last_clean":"2023-05-22T15:46:22.065875+0000","last_became_active":"2023-05-22T15:26:44.978443+0000","last_became_peered":"2023-05-22T15:26:44.978443+0000","last_unstale":"2023-05-22T15:46:22.065875+0000","last_undegraded":"2023-05-22T15:46:22.065875+0000","last_fullsized":"2023-05-22T15:46:22.065875+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:46:29.290243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.16","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.071629+0000","last_change":"2023-05-22T15:46:23.071629+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.071629+0000","last_undegraded":"2023-05-22T15:46:23.071629+0000","last_fullsized":"2023-05-22T15:46:23.071629+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T18:29:45.269794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.17","version":"0'0","reported_seq":9,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.092889+0000","last_change":"2023-05-22T15:46:23.084909+0000","last_active":"2023-05-22T15:46:23.092889+0000","last_peered":"2023-05-22T15:46:23.092889+0000","last_clean":"2023-05-22T15:46:23.092889+0000","last_became_active":"2023-05-22T15:46:23.084184+0000","last_became_peered":"2023-05-22T15:46:23.084184+0000","last_unstale":"2023-05-22T15:46:23.092889+0000","last_undegraded":"2023-05-22T15:46:23.092889+0000","last_fullsized":"2023-05-22T15:46:23.092889+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_clean_scrub_stamp":"2023-05-22T15:46:22.034488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:47:36.764285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":125,"reported_epoch":144,"state":"active+clean","last_fresh":"2023-05-22T15:46:22.063472+0000","last_change":"2023-05-22T15:26:44.982421+0000","last_active":"2023-05-22T15:46:22.063472+0000","last_peered":"2023-05-22T15:46:22.063472+0000","last_clean":"2023-05-22T15:46:22.063472+0000","last_became_active":"2023-05-22T15:26:44.981586+0000","last_became_peered":"2023-05-22T15:26:44.981586+0000","last_unstale":"2023-05-22T15:46:22.063472+0000","last_undegraded":"2023-05-22T15:46:22.063472+0000","last_fullsized":"2023-05-22T15:46:22.063472+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:14:11.156544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":19,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":18,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1268,"num_read_kb":2572,"num_write":399,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":249,"ondisk_log_size":249,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575084,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":19680,"kb_used_data":2328,"kb_used_omap":0,"kb_used_meta":17344,"kb_avail":94352160,"statfs":{"total":96636764160,"available":96616611840,"internally_reserved":0,"allocated":2383872,"data_stored":1535545,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":17760256},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 15:46:11 2023","interfaces":[{"interface":"back","average":{"1min":0.86699999999999999,"5min":0.89200000000000002,"15min":0.876},"min":{"1min":0.53600000000000003,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":1.294,"5min":2.637,"15min":2.637},"last":0.76600000000000001},{"interface":"front","average":{"1min":0.88,"5min":0.88700000000000001,"15min":0.89200000000000002},"min":{"1min":0.60799999999999998,"5min":0.502,"15min":0.35799999999999998},"max":{"1min":1.1519999999999999,"5min":3.0449999999999999,"15min":3.0449999999999999},"last":0.86199999999999999}]},{"osd":1,"last update":"Mon May 22 15:46:11 2023","interfaces":[{"interface":"back","average":{"1min":0.96799999999999997,"5min":0.95399999999999996,"15min":0.97899999999999998},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.1899999999999999,"5min":1.385,"15min":1.389},"last":0.73799999999999999},{"interface":"front","average":{"1min":0.96799999999999997,"5min":1.006,"15min":1.022},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.54900000000000004},"max":{"1min":1.226,"5min":2.7269999999999999,"15min":2.7269999999999999},"last":0.99299999999999999}]},{"osd":2,"last update":"Mon May 22 15:46:11 2023","interfaces":[{"interface":"back","average":{"1min":0.86199999999999999,"5min":0.85099999999999998,"15min":0.871},"min":{"1min":0.55600000000000005,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.109,"5min":1.252,"15min":1.252},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.88100000000000001,"5min":0.93999999999999995,"15min":0.95699999999999996},"min":{"1min":0.504,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.143,"5min":3.141,"15min":3.141},"last":0.68500000000000005}]},{"osd":3,"last update":"Mon May 22 15:46:11 2023","interfaces":[{"interface":"back","average":{"1min":0.98199999999999998,"5min":0.98899999999999999,"15min":0.96299999999999997},"min":{"1min":0.67600000000000005,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":1.3839999999999999,"5min":2.8149999999999999,"15min":2.8149999999999999},"last":1.109},{"interface":"front","average":{"1min":0.94399999999999995,"5min":0.92100000000000004,"15min":0.96199999999999997},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.2709999999999999,"5min":1.389,"15min":1.4710000000000001},"last":0.82899999999999996}]},{"osd":4,"last update":"Mon May 22 15:46:11 2023","interfaces":[{"interface":"back","average":{"1min":0.87,"5min":0.94399999999999995,"15min":0.90400000000000003},"min":{"1min":0.60299999999999998,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.5089999999999999,"5min":3.472,"15min":3.472},"last":0.94699999999999995},{"interface":"front","average":{"1min":0.997,"5min":1.1850000000000001,"15min":1.05},"min":{"1min":0.59699999999999998,"5min":0.56000000000000005,"15min":0.44600000000000001},"max":{"1min":1.2949999999999999,"5min":15.627000000000001,"15min":15.627000000000001},"last":1.163}]},{"osd":5,"last update":"Mon May 22 15:46:11 2023","interfaces":[{"interface":"back","average":{"1min":1.012,"5min":1.071,"15min":1.02},"min":{"1min":0.60899999999999999,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":1.4750000000000001,"5min":4.5599999999999996,"15min":4.5599999999999996},"last":0.90900000000000003},{"interface":"front","average":{"1min":1.115,"5min":1.161,"15min":1.139},"min":{"1min":0.82999999999999996,"5min":0.63800000000000001,"15min":0.58799999999999997},"max":{"1min":1.302,"5min":4.3799999999999999,"15min":4.3799999999999999},"last":1.1919999999999999}]},{"osd":6,"last update":"Mon May 22 15:46:11 2023","interfaces":[{"interface":"back","average":{"1min":1.073,"5min":1.1399999999999999,"15min":1.0840000000000001},"min":{"1min":0.80300000000000005,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.415,"5min":6.5069999999999997,"15min":6.5069999999999997},"last":1.1339999999999999},{"interface":"front","average":{"1min":1.1200000000000001,"5min":1.1659999999999999,"15min":1.1419999999999999},"min":{"1min":0.70299999999999996,"5min":0.62,"15min":0.62},"max":{"1min":1.458,"5min":3.3500000000000001,"15min":3.3500000000000001},"last":1.214}]}]},{"osd":6,"up_from":13,"seq":55834575085,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36572,"kb_used_data":1748,"kb_used_omap":0,"kb_used_meta":34816,"kb_avail":94335268,"statfs":{"total":96636764160,"available":96599314432,"internally_reserved":0,"allocated":1789952,"data_stored":945177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":35651584},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 15:46:20 2023","interfaces":[{"interface":"back","average":{"1min":0.90000000000000002,"5min":0.86699999999999999,"15min":0.89900000000000002},"min":{"1min":0.499,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":1.4039999999999999,"5min":1.486,"15min":1.486},"last":0.81899999999999995},{"interface":"front","average":{"1min":0.89200000000000002,"5min":0.94599999999999995,"15min":0.91100000000000003},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.36299999999999999},"max":{"1min":1.2729999999999999,"5min":1.6240000000000001,"15min":1.6240000000000001},"last":1.054}]},{"osd":1,"last update":"Mon May 22 15:46:20 2023","interfaces":[{"interface":"back","average":{"1min":0.93200000000000005,"5min":0.89100000000000001,"15min":0.91200000000000003},"min":{"1min":0.59799999999999998,"5min":0.504,"15min":0.48299999999999998},"max":{"1min":1.8799999999999999,"5min":1.8799999999999999,"15min":1.8799999999999999},"last":1.03},{"interface":"front","average":{"1min":0.95899999999999996,"5min":0.92400000000000004,"15min":0.93999999999999995},"min":{"1min":0.70299999999999996,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":1.1950000000000001,"5min":1.3320000000000001,"15min":1.554},"last":0.77300000000000002}]},{"osd":2,"last update":"Mon May 22 15:46:20 2023","interfaces":[{"interface":"back","average":{"1min":1.05,"5min":0.97399999999999998,"15min":0.98999999999999999},"min":{"1min":0.55400000000000005,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":3.2429999999999999,"5min":3.2429999999999999,"15min":3.2429999999999999},"last":1.099},{"interface":"front","average":{"1min":1.032,"5min":1.0269999999999999,"15min":1.0429999999999999},"min":{"1min":0.55300000000000005,"5min":0.5,"15min":0.442},"max":{"1min":3.3969999999999998,"5min":3.3969999999999998,"15min":3.3969999999999998},"last":1.1220000000000001}]},{"osd":3,"last update":"Mon May 22 15:46:20 2023","interfaces":[{"interface":"back","average":{"1min":0.88900000000000001,"5min":0.94599999999999995,"15min":0.97899999999999998},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":1.2470000000000001,"5min":1.5780000000000001,"15min":1.679},"last":1.155},{"interface":"front","average":{"1min":0.93500000000000005,"5min":0.95099999999999996,"15min":0.96299999999999997},"min":{"1min":0.69099999999999995,"5min":0.53200000000000003,"15min":0.50800000000000001},"max":{"1min":1.2430000000000001,"5min":1.383,"15min":1.4570000000000001},"last":1.006}]},{"osd":4,"last update":"Mon May 22 15:46:20 2023","interfaces":[{"interface":"back","average":{"1min":0.91300000000000003,"5min":0.93200000000000005,"15min":0.92500000000000004},"min":{"1min":0.61699999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":1.3169999999999999,"5min":1.581,"15min":1.6859999999999999},"last":0.91700000000000004},{"interface":"front","average":{"1min":0.93200000000000005,"5min":1.0209999999999999,"15min":1.0169999999999999},"min":{"1min":0.65500000000000003,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":1.242,"5min":1.452,"15min":1.8300000000000001},"last":0.88500000000000001}]},{"osd":5,"last update":"Mon May 22 15:46:20 2023","interfaces":[{"interface":"back","average":{"1min":0.98799999999999999,"5min":1.0169999999999999,"15min":0.98899999999999999},"min":{"1min":0.67600000000000005,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":1.3200000000000001,"5min":1.4530000000000001,"15min":1.6499999999999999},"last":0.85399999999999998},{"interface":"front","average":{"1min":1.1459999999999999,"5min":1.141,"15min":1.1120000000000001},"min":{"1min":0.81999999999999995,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.9990000000000001,"5min":1.9990000000000001,"15min":1.9990000000000001},"last":0.97799999999999998}]},{"osd":7,"last update":"Mon May 22 15:46:20 2023","interfaces":[{"interface":"back","average":{"1min":1.0860000000000001,"5min":1.131,"15min":1.125},"min":{"1min":0.85499999999999998,"5min":0.37,"15min":0.37},"max":{"1min":1.5529999999999999,"5min":1.853,"15min":1.9179999999999999},"last":1.23},{"interface":"front","average":{"1min":1.131,"5min":1.133,"15min":1.135},"min":{"1min":0.80300000000000005,"5min":0.442,"15min":0.442},"max":{"1min":1.579,"5min":1.579,"15min":1.8520000000000001},"last":0.95199999999999996}]}]},{"osd":5,"up_from":13,"seq":55834575085,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":26012,"kb_used_data":1748,"kb_used_omap":0,"kb_used_meta":24256,"kb_avail":94345828,"statfs":{"total":96636764160,"available":96610127872,"internally_reserved":0,"allocated":1789952,"data_stored":945177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":24838144},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 15:46:10 2023","interfaces":[{"interface":"back","average":{"1min":0.90900000000000003,"5min":0.85599999999999998,"15min":0.88300000000000001},"min":{"1min":0.61399999999999999,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.2889999999999999,"5min":1.3140000000000001,"15min":1.3440000000000001},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.91800000000000004,"5min":0.88600000000000001,"15min":0.88500000000000001},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":1.2709999999999999,"5min":1.3899999999999999,"15min":2.1699999999999999},"last":0.72599999999999998}]},{"osd":1,"last update":"Mon May 22 15:46:10 2023","interfaces":[{"interface":"back","average":{"1min":0.94999999999999996,"5min":0.96099999999999997,"15min":0.995},"min":{"1min":0.64800000000000002,"5min":0.52400000000000002,"15min":0.51000000000000001},"max":{"1min":1.3109999999999999,"5min":1.337,"15min":1.9370000000000001},"last":0.78000000000000003},{"interface":"front","average":{"1min":1.073,"5min":1.008,"15min":1.026},"min":{"1min":0.501,"5min":0.501,"15min":0.501},"max":{"1min":1.3089999999999999,"5min":1.6970000000000001,"15min":1.6970000000000001},"last":0.96899999999999997}]},{"osd":2,"last update":"Mon May 22 15:46:10 2023","interfaces":[{"interface":"back","average":{"1min":0.89100000000000001,"5min":0.91600000000000004,"15min":0.93400000000000005},"min":{"1min":0.505,"5min":0.505,"15min":0.504},"max":{"1min":1.343,"5min":1.5549999999999999,"15min":1.9830000000000001},"last":1.0549999999999999},{"interface":"front","average":{"1min":0.91800000000000004,"5min":0.94999999999999996,"15min":0.95199999999999996},"min":{"1min":0.56100000000000005,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.202,"5min":1.321,"15min":2.1120000000000001},"last":1.0169999999999999}]},{"osd":3,"last update":"Mon May 22 15:46:10 2023","interfaces":[{"interface":"back","average":{"1min":0.92900000000000005,"5min":0.92000000000000004,"15min":0.94899999999999995},"min":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.48199999999999998},"max":{"1min":1.206,"5min":1.2809999999999999,"15min":1.4419999999999999},"last":0.88300000000000001},{"interface":"front","average":{"1min":1.028,"5min":1.008,"15min":1.036},"min":{"1min":0.626,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":1.2490000000000001,"5min":1.371,"15min":2.52},"last":1.161}]},{"osd":4,"last update":"Mon May 22 15:46:10 2023","interfaces":[{"interface":"back","average":{"1min":0.96999999999999997,"5min":0.95699999999999996,"15min":0.95199999999999996},"min":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":1.405,"5min":1.405,"15min":2.226},"last":0.93500000000000005},{"interface":"front","average":{"1min":1.0629999999999999,"5min":1.024,"15min":1.0309999999999999},"min":{"1min":0.76000000000000001,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.3740000000000001,"5min":1.863,"15min":1.863},"last":1.1120000000000001}]},{"osd":6,"last update":"Mon May 22 15:46:10 2023","interfaces":[{"interface":"back","average":{"1min":1.008,"5min":1.024,"15min":1.0349999999999999},"min":{"1min":0.51900000000000002,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.397,"5min":1.4330000000000001,"15min":2.831},"last":1.246},{"interface":"front","average":{"1min":1.0369999999999999,"5min":1.0589999999999999,"15min":1.071},"min":{"1min":0.67000000000000004,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":1.4239999999999999,"5min":1.4239999999999999,"15min":2.3690000000000002},"last":1.296}]},{"osd":7,"last update":"Mon May 22 15:46:10 2023","interfaces":[{"interface":"back","average":{"1min":1.081,"5min":1.0900000000000001,"15min":1.1120000000000001},"min":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.42499999999999999},"max":{"1min":1.4359999999999999,"5min":1.47,"15min":1.849},"last":1.206},{"interface":"front","average":{"1min":1.1599999999999999,"5min":1.149,"15min":1.1659999999999999},"min":{"1min":0.86899999999999999,"5min":0.64000000000000001,"15min":0.64000000000000001},"max":{"1min":1.4339999999999999,"5min":1.575,"15min":2.7429999999999999},"last":1.3360000000000001}]}]},{"osd":4,"up_from":13,"seq":55834575083,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31592,"kb_used_data":1760,"kb_used_omap":0,"kb_used_meta":29824,"kb_avail":94340248,"statfs":{"total":96636764160,"available":96604413952,"internally_reserved":0,"allocated":1802240,"data_stored":952153,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":30539776},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 15:46:19 2023","interfaces":[{"interface":"back","average":{"1min":0.93200000000000005,"5min":0.86799999999999999,"15min":0.90500000000000003},"min":{"1min":0.63400000000000001,"5min":0.48999999999999999,"15min":0.34100000000000003},"max":{"1min":1.2629999999999999,"5min":1.4119999999999999,"15min":1.9910000000000001},"last":0.66600000000000004},{"interface":"front","average":{"1min":0.91300000000000003,"5min":0.90000000000000002,"15min":0.92200000000000004},"min":{"1min":0.51100000000000001,"5min":0.499,"15min":0.373},"max":{"1min":1.284,"5min":1.369,"15min":1.3779999999999999},"last":0.81000000000000005}]},{"osd":1,"last update":"Mon May 22 15:46:19 2023","interfaces":[{"interface":"back","average":{"1min":0.94799999999999995,"5min":0.91800000000000004,"15min":0.96399999999999997},"min":{"1min":0.60499999999999998,"5min":0.436,"15min":0.436},"max":{"1min":1.1970000000000001,"5min":1.3899999999999999,"15min":1.3899999999999999},"last":0.72099999999999997},{"interface":"front","average":{"1min":0.91700000000000004,"5min":0.84499999999999997,"15min":0.90100000000000002},"min":{"1min":0.56699999999999995,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.157,"5min":1.2889999999999999,"15min":1.3640000000000001},"last":0.97799999999999998}]},{"osd":2,"last update":"Mon May 22 15:46:19 2023","interfaces":[{"interface":"back","average":{"1min":0.90100000000000002,"5min":0.91600000000000004,"15min":0.94199999999999995},"min":{"1min":0.56299999999999994,"5min":0.50800000000000001,"15min":0.495},"max":{"1min":1.244,"5min":1.96,"15min":2.1019999999999999},"last":0.76500000000000001},{"interface":"front","average":{"1min":0.97799999999999998,"5min":0.98399999999999999,"15min":0.99199999999999999},"min":{"1min":0.58099999999999996,"5min":0.48799999999999999,"15min":0.48399999999999999},"max":{"1min":1.329,"5min":1.4399999999999999,"15min":1.4399999999999999},"last":0.60299999999999998}]},{"osd":3,"last update":"Mon May 22 15:46:19 2023","interfaces":[{"interface":"back","average":{"1min":0.91100000000000003,"5min":0.90200000000000002,"15min":0.93200000000000005},"min":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":1.224,"5min":1.28,"15min":1.3169999999999999},"last":1.0580000000000001},{"interface":"front","average":{"1min":1.0469999999999999,"5min":1.014,"15min":1.0209999999999999},"min":{"1min":0.72199999999999998,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":1.3420000000000001,"5min":1.4179999999999999,"15min":2.1720000000000002},"last":0.85599999999999998}]},{"osd":5,"last update":"Mon May 22 15:46:19 2023","interfaces":[{"interface":"back","average":{"1min":1.0209999999999999,"5min":0.97199999999999998,"15min":0.97699999999999998},"min":{"1min":0.55000000000000004,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.357,"5min":1.373,"15min":1.472},"last":1.0369999999999999},{"interface":"front","average":{"1min":1.0389999999999999,"5min":0.98999999999999999,"15min":0.97499999999999998},"min":{"1min":0.59999999999999998,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.262,"5min":1.365,"15min":1.4139999999999999},"last":1.155}]},{"osd":6,"last update":"Mon May 22 15:46:19 2023","interfaces":[{"interface":"back","average":{"1min":1.0649999999999999,"5min":0.99299999999999999,"15min":0.98499999999999999},"min":{"1min":0.72099999999999997,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":1.456,"5min":1.456,"15min":2.2349999999999999},"last":0.93500000000000005},{"interface":"front","average":{"1min":1.131,"5min":1.103,"15min":1.1100000000000001},"min":{"1min":0.66500000000000004,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":1.425,"5min":1.486,"15min":1.8220000000000001},"last":0.89800000000000002}]},{"osd":7,"last update":"Mon May 22 15:46:19 2023","interfaces":[{"interface":"back","average":{"1min":1.137,"5min":1.0629999999999999,"15min":1.052},"min":{"1min":0.85699999999999998,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":1.389,"5min":1.46,"15min":1.5800000000000001},"last":1.109},{"interface":"front","average":{"1min":1.1930000000000001,"5min":1.167,"15min":1.143},"min":{"1min":0.78900000000000003,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":1.5620000000000001,"5min":1.6559999999999999,"15min":2.306},"last":1.018}]}]},{"osd":3,"up_from":13,"seq":55834575083,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36956,"kb_used_data":1748,"kb_used_omap":0,"kb_used_meta":35200,"kb_avail":94334884,"statfs":{"total":96636764160,"available":96598921216,"internally_reserved":0,"allocated":1789952,"data_stored":945177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":36044800},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 15:46:03 2023","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.61699999999999999,"15min":0.63600000000000001},"min":{"1min":0.50700000000000001,"5min":0.252,"15min":0.19},"max":{"1min":1.169,"5min":1.169,"15min":1.194},"last":0.64600000000000002},{"interface":"front","average":{"1min":0.746,"5min":0.68500000000000005,"15min":0.70899999999999996},"min":{"1min":0.48899999999999999,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":1.085,"5min":1.296,"15min":1.296},"last":0.83599999999999997}]},{"osd":1,"last update":"Mon May 22 15:46:03 2023","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.64200000000000002,"15min":0.66200000000000003},"min":{"1min":0.52100000000000002,"5min":0.314,"15min":0.29499999999999998},"max":{"1min":0.90600000000000003,"5min":1.2609999999999999,"15min":1.2609999999999999},"last":0.93000000000000005},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.72899999999999998,"15min":0.76800000000000002},"min":{"1min":0.47499999999999998,"5min":0.34399999999999997,"15min":0.313},"max":{"1min":1.081,"5min":1.1359999999999999,"15min":1.222},"last":0.60099999999999998}]},{"osd":2,"last update":"Mon May 22 15:46:03 2023","interfaces":[{"interface":"back","average":{"1min":0.84199999999999997,"5min":0.78700000000000003,"15min":0.81699999999999995},"min":{"1min":0.45600000000000002,"5min":0.40699999999999997,"15min":0.36599999999999999},"max":{"1min":1.0529999999999999,"5min":1.506,"15min":1.506},"last":0.75800000000000001},{"interface":"front","average":{"1min":0.81899999999999995,"5min":0.75600000000000001,"15min":0.80700000000000005},"min":{"1min":0.65300000000000002,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":1.032,"5min":1.147,"15min":1.359},"last":0.89400000000000002}]},{"osd":4,"last update":"Mon May 22 15:46:03 2023","interfaces":[{"interface":"back","average":{"1min":0.97499999999999998,"5min":0.92500000000000004,"15min":0.95599999999999996},"min":{"1min":0.71999999999999997,"5min":0.54500000000000004,"15min":0.48999999999999999},"max":{"1min":1.238,"5min":1.413,"15min":1.4470000000000001},"last":1.0569999999999999},{"interface":"front","average":{"1min":1.0529999999999999,"5min":0.97499999999999998,"15min":1.0029999999999999},"min":{"1min":0.83299999999999996,"5min":0.54400000000000004,"15min":0.49199999999999999},"max":{"1min":1.407,"5min":1.407,"15min":1.4470000000000001},"last":1.0289999999999999}]},{"osd":5,"last update":"Mon May 22 15:46:03 2023","interfaces":[{"interface":"back","average":{"1min":1.0509999999999999,"5min":0.98399999999999999,"15min":1.016},"min":{"1min":0.80900000000000005,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.3149999999999999,"5min":1.5089999999999999,"15min":1.5089999999999999},"last":0.95899999999999996},{"interface":"front","average":{"1min":1.0349999999999999,"5min":1.006,"15min":1.038},"min":{"1min":0.90200000000000002,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.2130000000000001,"5min":1.4299999999999999,"15min":1.4359999999999999},"last":1.347}]},{"osd":6,"last update":"Mon May 22 15:46:03 2023","interfaces":[{"interface":"back","average":{"1min":1.0580000000000001,"5min":1.0129999999999999,"15min":1.042},"min":{"1min":0.873,"5min":0.47699999999999998,"15min":0.46999999999999997},"max":{"1min":1.2989999999999999,"5min":1.4570000000000001,"15min":1.49},"last":1.101},{"interface":"front","average":{"1min":1.083,"5min":1.0269999999999999,"15min":1.0700000000000001},"min":{"1min":0.79800000000000004,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":1.2949999999999999,"5min":1.4330000000000001,"15min":1.5009999999999999},"last":1.403}]},{"osd":7,"last update":"Mon May 22 15:46:03 2023","interfaces":[{"interface":"back","average":{"1min":1.113,"5min":1.0780000000000001,"15min":1.1040000000000001},"min":{"1min":0.83599999999999997,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.3779999999999999,"5min":1.6719999999999999,"15min":1.6719999999999999},"last":1.454},{"interface":"front","average":{"1min":1.117,"5min":1.071,"15min":1.085},"min":{"1min":0.90200000000000002,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":1.361,"5min":1.8700000000000001,"15min":1.8700000000000001},"last":1.0029999999999999}]}]},{"osd":2,"up_from":13,"seq":55834575084,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":18780,"kb_used_data":1748,"kb_used_omap":0,"kb_used_meta":17024,"kb_avail":94353060,"statfs":{"total":96636764160,"available":96617533440,"internally_reserved":0,"allocated":1789952,"data_stored":945177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":17432576},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 15:46:03 2023","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.64400000000000002,"15min":0.65600000000000003},"min":{"1min":0.28699999999999998,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":1.097,"5min":1.1539999999999999,"15min":1.8240000000000001},"last":0.70299999999999996},{"interface":"front","average":{"1min":0.69899999999999995,"5min":0.70499999999999996,"15min":0.746},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":1.079,"5min":1.079,"15min":1.8160000000000001},"last":0.46300000000000002}]},{"osd":1,"last update":"Mon May 22 15:46:03 2023","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.72999999999999998,"15min":0.74399999999999999},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.27000000000000002},"max":{"1min":1.196,"5min":1.196,"15min":1.2},"last":0.92000000000000004},{"interface":"front","average":{"1min":0.754,"5min":0.75900000000000001,"15min":0.80800000000000005},"min":{"1min":0.32400000000000001,"5min":0.317,"15min":0.317},"max":{"1min":1.0309999999999999,"5min":1.246,"15min":2.5089999999999999},"last":0.52500000000000002}]},{"osd":3,"last update":"Mon May 22 15:46:03 2023","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":0.69399999999999995,"15min":0.72599999999999998},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.96199999999999997,"5min":1.077,"15min":2.7629999999999999},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.82699999999999996,"5min":0.84599999999999997,"15min":0.88300000000000001},"min":{"1min":0.39600000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":1.1699999999999999,"5min":1.256,"15min":2.6840000000000002},"last":0.85699999999999998}]},{"osd":4,"last update":"Mon May 22 15:46:03 2023","interfaces":[{"interface":"back","average":{"1min":0.97499999999999998,"5min":0.98499999999999999,"15min":1.018},"min":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":1.274,"5min":1.536,"15min":2.8039999999999998},"last":1.1459999999999999},{"interface":"front","average":{"1min":1.0069999999999999,"5min":1.0009999999999999,"15min":1.028},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.2509999999999999,"5min":1.4299999999999999,"15min":2.0819999999999999},"last":0.81799999999999995}]},{"osd":5,"last update":"Mon May 22 15:46:03 2023","interfaces":[{"interface":"back","average":{"1min":1.0569999999999999,"5min":1.056,"15min":1.052},"min":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.3959999999999999,"5min":1.5049999999999999,"15min":2.3860000000000001},"last":1.175},{"interface":"front","average":{"1min":0.99099999999999999,"5min":0.97399999999999998,"15min":0.998},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.331,"5min":1.331,"15min":3.0299999999999998},"last":0.78100000000000003}]},{"osd":6,"last update":"Mon May 22 15:46:03 2023","interfaces":[{"interface":"back","average":{"1min":1.038,"5min":1.012,"15min":1.036},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":1.3320000000000001,"5min":1.49,"15min":2.883},"last":1.1020000000000001},{"interface":"front","average":{"1min":1.129,"5min":1.081,"15min":1.1040000000000001},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":1.4450000000000001,"5min":1.5660000000000001,"15min":3.1459999999999999},"last":1.0209999999999999}]},{"osd":7,"last update":"Mon May 22 15:46:03 2023","interfaces":[{"interface":"back","average":{"1min":1.1639999999999999,"5min":1.097,"15min":1.1279999999999999},"min":{"1min":0.55000000000000004,"5min":0.499,"15min":0.499},"max":{"1min":1.429,"5min":1.55,"15min":2.5990000000000002},"last":1.052},{"interface":"front","average":{"1min":1.1240000000000001,"5min":1.0640000000000001,"15min":1.1060000000000001},"min":{"1min":0.5,"5min":0.5,"15min":0.5},"max":{"1min":1.4650000000000001,"5min":1.4650000000000001,"15min":3.8799999999999999},"last":0.97699999999999998}]}]},{"osd":1,"up_from":13,"seq":55834575084,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29800,"kb_used_data":1760,"kb_used_omap":0,"kb_used_meta":28032,"kb_avail":94342040,"statfs":{"total":96636764160,"available":96606248960,"internally_reserved":0,"allocated":1802240,"data_stored":952153,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":28704768},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 15:46:17 2023","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.63200000000000001,"15min":0.63400000000000001},"min":{"1min":0.40600000000000003,"5min":0.26300000000000001,"15min":0.22700000000000001},"max":{"1min":1.1200000000000001,"5min":1.2270000000000001,"15min":1.2270000000000001},"last":0.98699999999999999},{"interface":"front","average":{"1min":0.77100000000000002,"5min":0.70699999999999996,"15min":0.76400000000000001},"min":{"1min":0.35999999999999999,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":1.1839999999999999,"5min":1.1839999999999999,"15min":1.2529999999999999},"last":0.58699999999999997}]},{"osd":2,"last update":"Mon May 22 15:46:17 2023","interfaces":[{"interface":"back","average":{"1min":0.76400000000000001,"5min":0.71399999999999997,"15min":0.73799999999999999},"min":{"1min":0.44800000000000001,"5min":0.317,"15min":0.28999999999999998},"max":{"1min":1.0660000000000001,"5min":1.391,"15min":1.391},"last":0.73899999999999999},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.72999999999999998,"15min":0.76300000000000001},"min":{"1min":0.49199999999999999,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.98699999999999999,"5min":1.2270000000000001,"15min":1.276},"last":0.77100000000000002}]},{"osd":3,"last update":"Mon May 22 15:46:17 2023","interfaces":[{"interface":"back","average":{"1min":0.80100000000000005,"5min":0.79900000000000004,"15min":0.82799999999999996},"min":{"1min":0.51200000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.153,"5min":1.657,"15min":1.657},"last":0.93200000000000005},{"interface":"front","average":{"1min":0.81100000000000005,"5min":0.80800000000000005,"15min":0.81499999999999995},"min":{"1min":0.33700000000000002,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":1.095,"5min":2.0569999999999999,"15min":2.0569999999999999},"last":0.876}]},{"osd":4,"last update":"Mon May 22 15:46:17 2023","interfaces":[{"interface":"back","average":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.96499999999999997},"min":{"1min":0.64500000000000002,"5min":0.55200000000000005,"15min":0.34000000000000002},"max":{"1min":1.502,"5min":2.9529999999999998,"15min":2.9529999999999998},"last":1.2450000000000001},{"interface":"front","average":{"1min":0.97499999999999998,"5min":0.999,"15min":1.002},"min":{"1min":0.61299999999999999,"5min":0.504,"15min":0.49299999999999999},"max":{"1min":1.292,"5min":2.7759999999999998,"15min":2.7759999999999998},"last":1.0469999999999999}]},{"osd":5,"last update":"Mon May 22 15:46:17 2023","interfaces":[{"interface":"back","average":{"1min":1.0349999999999999,"5min":1.006,"15min":1.034},"min":{"1min":0.627,"5min":0.58399999999999996,"15min":0.443},"max":{"1min":1.556,"5min":1.9570000000000001,"15min":1.9570000000000001},"last":1.0269999999999999},{"interface":"front","average":{"1min":1.1060000000000001,"5min":1.034,"15min":1.0449999999999999},"min":{"1min":0.59399999999999997,"5min":0.53400000000000003,"15min":0.47299999999999998},"max":{"1min":1.577,"5min":2.1200000000000001,"15min":2.1200000000000001},"last":1.177}]},{"osd":6,"last update":"Mon May 22 15:46:17 2023","interfaces":[{"interface":"back","average":{"1min":1.02,"5min":1.0369999999999999,"15min":1.036},"min":{"1min":0.65900000000000003,"5min":0.54000000000000004,"15min":0.45300000000000001},"max":{"1min":1.3180000000000001,"5min":2.831,"15min":2.831},"last":1.1399999999999999},{"interface":"front","average":{"1min":1.1499999999999999,"5min":1.085,"15min":1.0940000000000001},"min":{"1min":0.55800000000000005,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":1.6299999999999999,"5min":1.7629999999999999,"15min":1.7629999999999999},"last":1.0920000000000001}]},{"osd":7,"last update":"Mon May 22 15:46:17 2023","interfaces":[{"interface":"back","average":{"1min":1.1759999999999999,"5min":1.135,"15min":1.1180000000000001},"min":{"1min":0.61299999999999999,"5min":0.60299999999999998,"15min":0.56799999999999995},"max":{"1min":1.679,"5min":2.4199999999999999,"15min":2.4199999999999999},"last":1.1299999999999999},{"interface":"front","average":{"1min":1.1819999999999999,"5min":1.1259999999999999,"15min":1.1299999999999999},"min":{"1min":0.76300000000000001,"5min":0.47299999999999998,"15min":0.41599999999999998},"max":{"1min":1.619,"5min":3.8799999999999999,"15min":3.8799999999999999},"last":1.337}]}]},{"osd":0,"up_from":13,"seq":55834575085,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":25376,"kb_used_data":2328,"kb_used_omap":0,"kb_used_meta":23040,"kb_avail":94346464,"statfs":{"total":96636764160,"available":96610779136,"internally_reserved":0,"allocated":2383872,"data_stored":1535545,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":23592960},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon May 22 15:46:21 2023","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.65900000000000003,"15min":0.68100000000000005},"min":{"1min":0.307,"5min":0.248,"15min":0.223},"max":{"1min":1.2190000000000001,"5min":1.466,"15min":1.466},"last":0.78400000000000003},{"interface":"front","average":{"1min":0.69899999999999995,"5min":0.70599999999999996,"15min":0.72899999999999998},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.24099999999999999},"max":{"1min":1.2230000000000001,"5min":1.71,"15min":1.71},"last":0.55200000000000005}]},{"osd":2,"last update":"Mon May 22 15:46:21 2023","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":0.70699999999999996,"15min":0.74099999999999999},"min":{"1min":0.39700000000000002,"5min":0.318,"15min":0.26500000000000001},"max":{"1min":0.96799999999999997,"5min":1.165,"15min":1.3220000000000001},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.82799999999999996,"5min":0.75900000000000001,"15min":0.78800000000000003},"min":{"1min":0.375,"5min":0.34999999999999998,"15min":0.32900000000000001},"max":{"1min":1.1699999999999999,"5min":1.3129999999999999,"15min":1.3129999999999999},"last":0.747}]},{"osd":3,"last update":"Mon May 22 15:46:21 2023","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":0.74099999999999999,"15min":0.76000000000000001},"min":{"1min":0.35499999999999998,"5min":0.31,"15min":0.31},"max":{"1min":0.96199999999999997,"5min":1.9299999999999999,"15min":1.9299999999999999},"last":0.88600000000000001},{"interface":"front","average":{"1min":0.83499999999999996,"5min":0.83299999999999996,"15min":0.871},"min":{"1min":0.35599999999999998,"5min":0.35199999999999998,"15min":0.34399999999999997},"max":{"1min":1.1299999999999999,"5min":1.758,"15min":1.758},"last":0.85999999999999999}]},{"osd":4,"last update":"Mon May 22 15:46:21 2023","interfaces":[{"interface":"back","average":{"1min":0.92800000000000005,"5min":0.95099999999999996,"15min":0.95099999999999996},"min":{"1min":0.57599999999999996,"5min":0.57399999999999995,"15min":0.48099999999999998},"max":{"1min":1.149,"5min":2.923,"15min":2.923},"last":1.091},{"interface":"front","average":{"1min":0.98999999999999999,"5min":1.006,"15min":1.026},"min":{"1min":0.624,"5min":0.624,"15min":0.624},"max":{"1min":1.3839999999999999,"5min":2.0129999999999999,"15min":2.0129999999999999},"last":0.93600000000000005}]},{"osd":5,"last update":"Mon May 22 15:46:21 2023","interfaces":[{"interface":"back","average":{"1min":1,"5min":1.004,"15min":1.0409999999999999},"min":{"1min":0.59099999999999997,"5min":0.56000000000000005,"15min":0.46000000000000002},"max":{"1min":1.238,"5min":1.97,"15min":1.97},"last":1.006},{"interface":"front","average":{"1min":1.02,"5min":1.016,"15min":1.008},"min":{"1min":0.67600000000000005,"5min":0.56599999999999995,"15min":0.55300000000000005},"max":{"1min":1.5349999999999999,"5min":5.0519999999999996,"15min":5.0519999999999996},"last":0.98199999999999998}]},{"osd":6,"last update":"Mon May 22 15:46:21 2023","interfaces":[{"interface":"back","average":{"1min":1.044,"5min":1.0589999999999999,"15min":1.089},"min":{"1min":0.58899999999999997,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":1.306,"5min":2.117,"15min":2.117},"last":1.1439999999999999},{"interface":"front","average":{"1min":1.085,"5min":1.1299999999999999,"15min":1.119},"min":{"1min":0.69199999999999995,"5min":0.65500000000000003,"15min":0.499},"max":{"1min":1.3100000000000001,"5min":5.2640000000000002,"15min":5.2640000000000002},"last":1.1910000000000001}]},{"osd":7,"last update":"Mon May 22 15:46:21 2023","interfaces":[{"interface":"back","average":{"1min":1.075,"5min":1.1040000000000001,"15min":1.129},"min":{"1min":0.63800000000000001,"5min":0.61299999999999999,"15min":0.61299999999999999},"max":{"1min":1.3640000000000001,"5min":2.0550000000000002,"15min":2.0550000000000002},"last":1.0680000000000001},{"interface":"front","average":{"1min":1.089,"5min":1.0900000000000001,"15min":1.0960000000000001},"min":{"1min":0.73099999999999998,"5min":0.63300000000000001,"15min":0.60299999999999998},"max":{"1min":1.6240000000000001,"5min":3.9060000000000001,"15min":3.9060000000000001},"last":1.0409999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-05-22T15:46:28.334 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.3f is not active+clean 2023-05-22T15:46:28.335 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.3f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 145, 'state': 'creating+peering', 'last_fresh': '2023-05-22T15:46:23.079122+0000', 'last_change': '2023-05-22T15:46:23.079122+0000', 'last_active': '2023-05-22T15:46:23.046574+0000', 'last_peered': '2023-05-22T15:46:23.046574+0000', 'last_clean': '2023-05-22T15:46:23.046574+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-05-22T15:46:23.079122+0000', 'last_undegraded': '2023-05-22T15:46:23.079122+0000', 'last_fullsized': '2023-05-22T15:46:23.079122+0000', 'mapping_epoch': 145, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 145, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-05-22T15:46:23.046574+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-05-22T15:46:23.046574+0000', 'last_clean_scrub_stamp': '2023-05-22T15:46:23.046574+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-05-23T23:57:19.852836+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 4], 'acting': [2, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-05-22T15:46:28.335 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.3e is not active+clean 2023-05-22T15:46:28.335 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.3e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 145, 'state': 'creating+peering', 'last_fresh': '2023-05-22T15:46:23.076282+0000', 'last_change': '2023-05-22T15:46:23.076282+0000', 'last_active': '2023-05-22T15:46:23.046574+0000', 'last_peered': '2023-05-22T15:46:23.046574+0000', 'last_clean': '2023-05-22T15:46:23.046574+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-05-22T15:46:23.076282+0000', 'last_undegraded': '2023-05-22T15:46:23.076282+0000', 'last_fullsized': '2023-05-22T15:46:23.076282+0000', 'mapping_epoch': 145, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 145, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-05-22T15:46:23.046574+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-05-22T15:46:23.046574+0000', 'last_clean_scrub_stamp': '2023-05-22T15:46:23.046574+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-05-24T03:20:39.433281+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 4], 'acting': [1, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-05-22T15:46:28.336 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.3d is not active+clean 2023-05-22T15:46:28.336 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.3d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 145, 'state': 'creating+peering', 'last_fresh': '2023-05-22T15:46:23.072443+0000', 'last_change': '2023-05-22T15:46:23.072443+0000', 'last_active': '2023-05-22T15:46:23.046574+0000', 'last_peered': '2023-05-22T15:46:23.046574+0000', 'last_clean': '2023-05-22T15:46:23.046574+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-05-22T15:46:23.072443+0000', 'last_undegraded': '2023-05-22T15:46:23.072443+0000', 'last_fullsized': '2023-05-22T15:46:23.072443+0000', 'mapping_epoch': 145, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 145, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-05-22T15:46:23.046574+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-05-22T15:46:23.046574+0000', 'last_clean_scrub_stamp': '2023-05-22T15:46:23.046574+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-05-24T01:46:28.227459+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 0], 'acting': [6, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2023-05-22T15:46:28.336 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.3c is not active+clean 2023-05-22T15:46:28.341 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.3c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 145, 'state': 'creating+peering', 'last_fresh': '2023-05-22T15:46:23.077931+0000', 'last_change': '2023-05-22T15:46:23.077931+0000', 'last_active': '2023-05-22T15:46:23.046574+0000', 'last_peered': '2023-05-22T15:46:23.046574+0000', 'last_clean': '2023-05-22T15:46:23.046574+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-05-22T15:46:23.077931+0000', 'last_undegraded': '2023-05-22T15:46:23.077931+0000', 'last_fullsized': '2023-05-22T15:46:23.077931+0000', 'mapping_epoch': 145, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 145, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-05-22T15:46:23.046574+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-05-22T15:46:23.046574+0000', 'last_clean_scrub_stamp': '2023-05-22T15:46:23.046574+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-05-23T19:42:45.119566+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 7], 'acting': [2, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-05-22T15:46:28.341 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.3b is not active+clean 2023-05-22T15:46:28.342 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.3b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 145, 'state': 'creating+peering', 'last_fresh': '2023-05-22T15:46:23.090321+0000', 'last_change': '2023-05-22T15:46:23.090321+0000', 'last_active': '2023-05-22T15:46:23.046574+0000', 'last_peered': '2023-05-22T15:46:23.046574+0000', 'last_clean': '2023-05-22T15:46:23.046574+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-05-22T15:46:23.090321+0000', 'last_undegraded': '2023-05-22T15:46:23.090321+0000', 'last_fullsized': '2023-05-22T15:46:23.090321+0000', 'mapping_epoch': 145, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 145, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-05-22T15:46:23.046574+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-05-22T15:46:23.046574+0000', 'last_clean_scrub_stamp': '2023-05-22T15:46:23.046574+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-05-24T01:59:24.451915+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 5], 'acting': [7, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2023-05-22T15:46:28.342 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.3a is not active+clean 2023-05-22T15:46:28.342 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.3a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 145, 'state': 'creating+peering', 'last_fresh': '2023-05-22T15:46:23.072441+0000', 'last_change': '2023-05-22T15:46:23.072441+0000', 'last_active': '2023-05-22T15:46:23.046574+0000', 'last_peered': '2023-05-22T15:46:23.046574+0000', 'last_clean': '2023-05-22T15:46:23.046574+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-05-22T15:46:23.072441+0000', 'last_undegraded': '2023-05-22T15:46:23.072441+0000', 'last_fullsized': '2023-05-22T15:46:23.072441+0000', 'mapping_epoch': 145, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 145, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-05-22T15:46:23.046574+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-05-22T15:46:23.046574+0000', 'last_clean_scrub_stamp': '2023-05-22T15:46:23.046574+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-05-23T19:25:55.107784+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 2], 'acting': [4, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2023-05-22T15:46:28.343 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.39 is not active+clean 2023-05-22T15:46:28.343 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.39', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 145, 'state': 'creating+peering', 'last_fresh': '2023-05-22T15:46:23.076980+0000', 'last_change': '2023-05-22T15:46:23.076980+0000', 'last_active': '2023-05-22T15:46:23.046574+0000', 'last_peered': '2023-05-22T15:46:23.046574+0000', 'last_clean': '2023-05-22T15:46:23.046574+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-05-22T15:46:23.076980+0000', 'last_undegraded': '2023-05-22T15:46:23.076980+0000', 'last_fullsized': '2023-05-22T15:46:23.076980+0000', 'mapping_epoch': 145, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 145, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-05-22T15:46:23.046574+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-05-22T15:46:23.046574+0000', 'last_clean_scrub_stamp': '2023-05-22T15:46:23.046574+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-05-24T01:29:20.717884+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 6], 'acting': [4, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2023-05-22T15:46:28.344 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.38 is not active+clean 2023-05-22T15:46:28.344 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.38', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 145, 'state': 'creating+peering', 'last_fresh': '2023-05-22T15:46:23.078358+0000', 'last_change': '2023-05-22T15:46:23.078358+0000', 'last_active': '2023-05-22T15:46:23.046574+0000', 'last_peered': '2023-05-22T15:46:23.046574+0000', 'last_clean': '2023-05-22T15:46:23.046574+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-05-22T15:46:23.078358+0000', 'last_undegraded': '2023-05-22T15:46:23.078358+0000', 'last_fullsized': '2023-05-22T15:46:23.078358+0000', 'mapping_epoch': 145, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 145, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-05-22T15:46:23.046574+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-05-22T15:46:23.046574+0000', 'last_clean_scrub_stamp': '2023-05-22T15:46:23.046574+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-05-23T20:51:08.858142+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 3], 'acting': [6, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2023-05-22T15:46:28.345 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.37 is not active+clean 2023-05-22T15:46:28.345 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.37', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 145, 'state': 'creating+peering', 'last_fresh': '2023-05-22T15:46:23.074307+0000', 'last_change': '2023-05-22T15:46:23.074307+0000', 'last_active': '2023-05-22T15:46:23.046574+0000', 'last_peered': '2023-05-22T15:46:23.046574+0000', 'last_clean': '2023-05-22T15:46:23.046574+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-05-22T15:46:23.074307+0000', 'last_undegraded': '2023-05-22T15:46:23.074307+0000', 'last_fullsized': '2023-05-22T15:46:23.074307+0000', 'mapping_epoch': 145, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 145, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-05-22T15:46:23.046574+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-05-22T15:46:23.046574+0000', 'last_clean_scrub_stamp': '2023-05-22T15:46:23.046574+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-05-24T02:28:23.397061+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 5], 'acting': [6, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2023-05-22T15:46:28.346 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.36 is not active+clean 2023-05-22T15:46:28.347 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.36', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 145, 'state': 'creating+peering', 'last_fresh': '2023-05-22T15:46:23.075645+0000', 'last_change': '2023-05-22T15:46:23.075645+0000', 'last_active': '2023-05-22T15:46:23.046574+0000', 'last_peered': '2023-05-22T15:46:23.046574+0000', 'last_clean': '2023-05-22T15:46:23.046574+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-05-22T15:46:23.075645+0000', 'last_undegraded': '2023-05-22T15:46:23.075645+0000', 'last_fullsized': '2023-05-22T15:46:23.075645+0000', 'mapping_epoch': 145, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 145, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-05-22T15:46:23.046574+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-05-22T15:46:23.046574+0000', 'last_clean_scrub_stamp': '2023-05-22T15:46:23.046574+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-05-24T01:46:28.227459+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 4], 'acting': [1, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-05-22T15:46:28.348 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.35 is not active+clean 2023-05-22T15:46:28.348 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.35', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 145, 'state': 'creating+peering', 'last_fresh': '2023-05-22T15:46:23.069852+0000', 'last_change': '2023-05-22T15:46:23.069852+0000', 'last_active': '2023-05-22T15:46:23.046574+0000', 'last_peered': '2023-05-22T15:46:23.046574+0000', 'last_clean': '2023-05-22T15:46:23.046574+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-05-22T15:46:23.069852+0000', 'last_undegraded': '2023-05-22T15:46:23.069852+0000', 'last_fullsized': '2023-05-22T15:46:23.069852+0000', 'mapping_epoch': 145, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 145, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-05-22T15:46:23.046574+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-05-22T15:46:23.046574+0000', 'last_clean_scrub_stamp': '2023-05-22T15:46:23.046574+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-05-23T15:53:20.317234+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2023-05-22T15:46:28.348 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.34 is not active+clean 2023-05-22T15:46:28.349 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.34', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 145, 'state': 'creating+peering', 'last_fresh': '2023-05-22T15:46:23.073019+0000', 'last_change': '2023-05-22T15:46:23.073019+0000', 'last_active': '2023-05-22T15:46:23.046574+0000', 'last_peered': '2023-05-22T15:46:23.046574+0000', 'last_clean': '2023-05-22T15:46:23.046574+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-05-22T15:46:23.073019+0000', 'last_undegraded': '2023-05-22T15:46:23.073019+0000', 'last_fullsized': '2023-05-22T15:46:23.073019+0000', 'mapping_epoch': 145, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 145, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-05-22T15:46:23.046574+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-05-22T15:46:23.046574+0000', 'last_clean_scrub_stamp': '2023-05-22T15:46:23.046574+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-05-23T18:52:29.352539+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 5], 'acting': [6, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2023-05-22T15:46:28.350 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.33 is not active+clean 2023-05-22T15:46:28.350 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.33', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 145, 'state': 'creating+peering', 'last_fresh': '2023-05-22T15:46:23.078530+0000', 'last_change': '2023-05-22T15:46:23.078530+0000', 'last_active': '2023-05-22T15:46:23.046574+0000', 'last_peered': '2023-05-22T15:46:23.046574+0000', 'last_clean': '2023-05-22T15:46:23.046574+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-05-22T15:46:23.078530+0000', 'last_undegraded': '2023-05-22T15:46:23.078530+0000', 'last_fullsized': '2023-05-22T15:46:23.078530+0000', 'mapping_epoch': 145, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 145, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-05-22T15:46:23.046574+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-05-22T15:46:23.046574+0000', 'last_clean_scrub_stamp': '2023-05-22T15:46:23.046574+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-05-23T22:22:06.668010+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 4], 'acting': [2, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-05-22T15:46:28.351 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.32 is not active+clean 2023-05-22T15:46:28.351 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.32', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 145, 'state': 'creating+peering', 'last_fresh': '2023-05-22T15:46:23.072337+0000', 'last_change': '2023-05-22T15:46:23.072337+0000', 'last_active': '2023-05-22T15:46:23.046574+0000', 'last_peered': '2023-05-22T15:46:23.046574+0000', 'last_clean': '2023-05-22T15:46:23.046574+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-05-22T15:46:23.072337+0000', 'last_undegraded': '2023-05-22T15:46:23.072337+0000', 'last_fullsized': '2023-05-22T15:46:23.072337+0000', 'mapping_epoch': 145, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 145, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-05-22T15:46:23.046574+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-05-22T15:46:23.046574+0000', 'last_clean_scrub_stamp': '2023-05-22T15:46:23.046574+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-05-23T18:05:16.988218+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [1, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-05-22T15:46:28.352 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.31 is not active+clean 2023-05-22T15:46:28.353 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.31', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 145, 'state': 'creating+peering', 'last_fresh': '2023-05-22T15:46:23.074359+0000', 'last_change': '2023-05-22T15:46:23.074359+0000', 'last_active': '2023-05-22T15:46:23.046574+0000', 'last_peered': '2023-05-22T15:46:23.046574+0000', 'last_clean': '2023-05-22T15:46:23.046574+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-05-22T15:46:23.074359+0000', 'last_undegraded': '2023-05-22T15:46:23.074359+0000', 'last_fullsized': '2023-05-22T15:46:23.074359+0000', 'mapping_epoch': 145, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 145, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-05-22T15:46:23.046574+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-05-22T15:46:23.046574+0000', 'last_clean_scrub_stamp': '2023-05-22T15:46:23.046574+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-05-23T18:05:16.988218+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 6], 'acting': [2, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-05-22T15:46:28.354 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.30 is not active+clean 2023-05-22T15:46:28.355 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.30', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 145, 'state': 'creating+peering', 'last_fresh': '2023-05-22T15:46:23.063646+0000', 'last_change': '2023-05-22T15:46:23.063646+0000', 'last_active': '2023-05-22T15:46:23.046574+0000', 'last_peered': '2023-05-22T15:46:23.046574+0000', 'last_clean': '2023-05-22T15:46:23.046574+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-05-22T15:46:23.063646+0000', 'last_undegraded': '2023-05-22T15:46:23.063646+0000', 'last_fullsized': '2023-05-22T15:46:23.063646+0000', 'mapping_epoch': 145, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 145, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-05-22T15:46:23.046574+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-05-22T15:46:23.046574+0000', 'last_clean_scrub_stamp': '2023-05-22T15:46:23.046574+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-05-24T01:30:04.941241+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2023-05-22T15:46:28.355 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.2f is not active+clean 2023-05-22T15:46:28.356 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.2f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 145, 'state': 'creating+peering', 'last_fresh': '2023-05-22T15:46:23.075803+0000', 'last_change': '2023-05-22T15:46:23.075803+0000', 'last_active': '2023-05-22T15:46:23.046574+0000', 'last_peered': '2023-05-22T15:46:23.046574+0000', 'last_clean': '2023-05-22T15:46:23.046574+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-05-22T15:46:23.075803+0000', 'last_undegraded': '2023-05-22T15:46:23.075803+0000', 'last_fullsized': '2023-05-22T15:46:23.075803+0000', 'mapping_epoch': 145, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 145, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-05-22T15:46:23.046574+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-05-22T15:46:23.046574+0000', 'last_clean_scrub_stamp': '2023-05-22T15:46:23.046574+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-05-23T20:23:42.844820+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 0], 'acting': [4, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2023-05-22T15:46:28.356 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.2e is not active+clean 2023-05-22T15:46:28.356 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.2e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 145, 'state': 'creating+peering', 'last_fresh': '2023-05-22T15:46:23.064448+0000', 'last_change': '2023-05-22T15:46:23.064448+0000', 'last_active': '2023-05-22T15:46:23.046574+0000', 'last_peered': '2023-05-22T15:46:23.046574+0000', 'last_clean': '2023-05-22T15:46:23.046574+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-05-22T15:46:23.064448+0000', 'last_undegraded': '2023-05-22T15:46:23.064448+0000', 'last_fullsized': '2023-05-22T15:46:23.064448+0000', 'mapping_epoch': 145, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 145, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-05-22T15:46:23.046574+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-05-22T15:46:23.046574+0000', 'last_clean_scrub_stamp': '2023-05-22T15:46:23.046574+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-05-23T22:23:58.635276+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 4], 'acting': [3, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2023-05-22T15:46:28.357 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.2d is not active+clean 2023-05-22T15:46:28.357 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.2d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 145, 'state': 'creating+peering', 'last_fresh': '2023-05-22T15:46:23.068994+0000', 'last_change': '2023-05-22T15:46:23.068994+0000', 'last_active': '2023-05-22T15:46:23.046574+0000', 'last_peered': '2023-05-22T15:46:23.046574+0000', 'last_clean': '2023-05-22T15:46:23.046574+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-05-22T15:46:23.068994+0000', 'last_undegraded': '2023-05-22T15:46:23.068994+0000', 'last_fullsized': '2023-05-22T15:46:23.068994+0000', 'mapping_epoch': 145, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 145, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-05-22T15:46:23.046574+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-05-22T15:46:23.046574+0000', 'last_clean_scrub_stamp': '2023-05-22T15:46:23.046574+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-05-23T21:38:41.516579+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 4], 'acting': [0, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2023-05-22T15:46:28.357 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.2c is not active+clean 2023-05-22T15:46:28.358 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.2c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 145, 'state': 'creating+peering', 'last_fresh': '2023-05-22T15:46:23.062047+0000', 'last_change': '2023-05-22T15:46:23.062047+0000', 'last_active': '2023-05-22T15:46:23.046574+0000', 'last_peered': '2023-05-22T15:46:23.046574+0000', 'last_clean': '2023-05-22T15:46:23.046574+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-05-22T15:46:23.062047+0000', 'last_undegraded': '2023-05-22T15:46:23.062047+0000', 'last_fullsized': '2023-05-22T15:46:23.062047+0000', 'mapping_epoch': 145, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 145, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-05-22T15:46:23.046574+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-05-22T15:46:23.046574+0000', 'last_clean_scrub_stamp': '2023-05-22T15:46:23.046574+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-05-23T22:52:08.264737+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1], 'acting': [3, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2023-05-22T15:46:28.358 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.2b is not active+clean 2023-05-22T15:46:28.358 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.2b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 145, 'state': 'creating+peering', 'last_fresh': '2023-05-22T15:46:23.093524+0000', 'last_change': '2023-05-22T15:46:23.093524+0000', 'last_active': '2023-05-22T15:46:23.046574+0000', 'last_peered': '2023-05-22T15:46:23.046574+0000', 'last_clean': '2023-05-22T15:46:23.046574+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-05-22T15:46:23.093524+0000', 'last_undegraded': '2023-05-22T15:46:23.093524+0000', 'last_fullsized': '2023-05-22T15:46:23.093524+0000', 'mapping_epoch': 145, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 145, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-05-22T15:46:23.046574+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-05-22T15:46:23.046574+0000', 'last_clean_scrub_stamp': '2023-05-22T15:46:23.046574+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-05-23T20:55:41.010860+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 5], 'acting': [7, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2023-05-22T15:46:28.359 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.2a is not active+clean 2023-05-22T15:46:28.359 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.2a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 145, 'state': 'creating+peering', 'last_fresh': '2023-05-22T15:46:23.069379+0000', 'last_change': '2023-05-22T15:46:23.069379+0000', 'last_active': '2023-05-22T15:46:23.046574+0000', 'last_peered': '2023-05-22T15:46:23.046574+0000', 'last_clean': '2023-05-22T15:46:23.046574+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-05-22T15:46:23.069379+0000', 'last_undegraded': '2023-05-22T15:46:23.069379+0000', 'last_fullsized': '2023-05-22T15:46:23.069379+0000', 'mapping_epoch': 145, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 145, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-05-22T15:46:23.046574+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-05-22T15:46:23.046574+0000', 'last_clean_scrub_stamp': '2023-05-22T15:46:23.046574+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-05-23T20:36:49.326554+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2023-05-22T15:46:28.359 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.29 is not active+clean 2023-05-22T15:46:28.360 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.29', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 145, 'state': 'creating+peering', 'last_fresh': '2023-05-22T15:46:23.081591+0000', 'last_change': '2023-05-22T15:46:23.081591+0000', 'last_active': '2023-05-22T15:46:23.046574+0000', 'last_peered': '2023-05-22T15:46:23.046574+0000', 'last_clean': '2023-05-22T15:46:23.046574+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-05-22T15:46:23.081591+0000', 'last_undegraded': '2023-05-22T15:46:23.081591+0000', 'last_fullsized': '2023-05-22T15:46:23.081591+0000', 'mapping_epoch': 145, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 145, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-05-22T15:46:23.046574+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-05-22T15:46:23.046574+0000', 'last_clean_scrub_stamp': '2023-05-22T15:46:23.046574+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-05-24T02:55:51.486171+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 5], 'acting': [6, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2023-05-22T15:46:28.360 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.28 is not active+clean 2023-05-22T15:46:28.361 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.28', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 145, 'state': 'creating+peering', 'last_fresh': '2023-05-22T15:46:23.070163+0000', 'last_change': '2023-05-22T15:46:23.070163+0000', 'last_active': '2023-05-22T15:46:23.046574+0000', 'last_peered': '2023-05-22T15:46:23.046574+0000', 'last_clean': '2023-05-22T15:46:23.046574+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-05-22T15:46:23.070163+0000', 'last_undegraded': '2023-05-22T15:46:23.070163+0000', 'last_fullsized': '2023-05-22T15:46:23.070163+0000', 'mapping_epoch': 145, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 145, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-05-22T15:46:23.046574+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-05-22T15:46:23.046574+0000', 'last_clean_scrub_stamp': '2023-05-22T15:46:23.046574+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-05-23T19:21:39.187868+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 5], 'acting': [0, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2023-05-22T15:46:28.361 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.27 is not active+clean 2023-05-22T15:46:28.361 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.27', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 145, 'state': 'creating+peering', 'last_fresh': '2023-05-22T15:46:23.071496+0000', 'last_change': '2023-05-22T15:46:23.071496+0000', 'last_active': '2023-05-22T15:46:23.046574+0000', 'last_peered': '2023-05-22T15:46:23.046574+0000', 'last_clean': '2023-05-22T15:46:23.046574+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-05-22T15:46:23.071496+0000', 'last_undegraded': '2023-05-22T15:46:23.071496+0000', 'last_fullsized': '2023-05-22T15:46:23.071496+0000', 'mapping_epoch': 145, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 145, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-05-22T15:46:23.046574+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-05-22T15:46:23.046574+0000', 'last_clean_scrub_stamp': '2023-05-22T15:46:23.046574+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-05-24T02:37:26.183234+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 7], 'acting': [1, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-05-22T15:46:28.362 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.26 is not active+clean 2023-05-22T15:46:28.362 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.26', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 145, 'state': 'creating+peering', 'last_fresh': '2023-05-22T15:46:23.063189+0000', 'last_change': '2023-05-22T15:46:23.063189+0000', 'last_active': '2023-05-22T15:46:23.046574+0000', 'last_peered': '2023-05-22T15:46:23.046574+0000', 'last_clean': '2023-05-22T15:46:23.046574+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-05-22T15:46:23.063189+0000', 'last_undegraded': '2023-05-22T15:46:23.063189+0000', 'last_fullsized': '2023-05-22T15:46:23.063189+0000', 'mapping_epoch': 145, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 145, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-05-22T15:46:23.046574+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-05-22T15:46:23.046574+0000', 'last_clean_scrub_stamp': '2023-05-22T15:46:23.046574+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-05-23T21:43:11.401376+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2023-05-22T15:46:28.363 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.25 is not active+clean 2023-05-22T15:46:28.363 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.25', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 145, 'state': 'creating+peering', 'last_fresh': '2023-05-22T15:46:23.074484+0000', 'last_change': '2023-05-22T15:46:23.074484+0000', 'last_active': '2023-05-22T15:46:23.046574+0000', 'last_peered': '2023-05-22T15:46:23.046574+0000', 'last_clean': '2023-05-22T15:46:23.046574+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-05-22T15:46:23.074484+0000', 'last_undegraded': '2023-05-22T15:46:23.074484+0000', 'last_fullsized': '2023-05-22T15:46:23.074484+0000', 'mapping_epoch': 145, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 145, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-05-22T15:46:23.046574+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-05-22T15:46:23.046574+0000', 'last_clean_scrub_stamp': '2023-05-22T15:46:23.046574+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-05-24T03:43:40.124163+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 4], 'acting': [2, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-05-22T15:46:28.363 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.24 is not active+clean 2023-05-22T15:46:28.364 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.24', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 145, 'state': 'creating+peering', 'last_fresh': '2023-05-22T15:46:23.064681+0000', 'last_change': '2023-05-22T15:46:23.064681+0000', 'last_active': '2023-05-22T15:46:23.046574+0000', 'last_peered': '2023-05-22T15:46:23.046574+0000', 'last_clean': '2023-05-22T15:46:23.046574+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-05-22T15:46:23.064681+0000', 'last_undegraded': '2023-05-22T15:46:23.064681+0000', 'last_fullsized': '2023-05-22T15:46:23.064681+0000', 'mapping_epoch': 145, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 145, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-05-22T15:46:23.046574+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-05-22T15:46:23.046574+0000', 'last_clean_scrub_stamp': '2023-05-22T15:46:23.046574+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-05-24T02:20:29.148898+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 3], 'acting': [5, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2023-05-22T15:46:28.364 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.23 is not active+clean 2023-05-22T15:46:28.364 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.23', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 145, 'state': 'creating+peering', 'last_fresh': '2023-05-22T15:46:23.077284+0000', 'last_change': '2023-05-22T15:46:23.077284+0000', 'last_active': '2023-05-22T15:46:23.046574+0000', 'last_peered': '2023-05-22T15:46:23.046574+0000', 'last_clean': '2023-05-22T15:46:23.046574+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-05-22T15:46:23.077284+0000', 'last_undegraded': '2023-05-22T15:46:23.077284+0000', 'last_fullsized': '2023-05-22T15:46:23.077284+0000', 'mapping_epoch': 145, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 145, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-05-22T15:46:23.046574+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-05-22T15:46:23.046574+0000', 'last_clean_scrub_stamp': '2023-05-22T15:46:23.046574+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-05-24T00:34:43.855049+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 6], 'acting': [2, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-05-22T15:46:28.365 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.22 is not active+clean 2023-05-22T15:46:28.365 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.22', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 145, 'state': 'creating+peering', 'last_fresh': '2023-05-22T15:46:23.073667+0000', 'last_change': '2023-05-22T15:46:23.073667+0000', 'last_active': '2023-05-22T15:46:23.046574+0000', 'last_peered': '2023-05-22T15:46:23.046574+0000', 'last_clean': '2023-05-22T15:46:23.046574+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-05-22T15:46:23.073667+0000', 'last_undegraded': '2023-05-22T15:46:23.073667+0000', 'last_fullsized': '2023-05-22T15:46:23.073667+0000', 'mapping_epoch': 145, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 145, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-05-22T15:46:23.046574+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-05-22T15:46:23.046574+0000', 'last_clean_scrub_stamp': '2023-05-22T15:46:23.046574+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-05-23T16:10:54.516325+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 6], 'acting': [2, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-05-22T15:46:28.365 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.21 is not active+clean 2023-05-22T15:46:28.366 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.21', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 145, 'state': 'creating+peering', 'last_fresh': '2023-05-22T15:46:23.086294+0000', 'last_change': '2023-05-22T15:46:23.086294+0000', 'last_active': '2023-05-22T15:46:23.046574+0000', 'last_peered': '2023-05-22T15:46:23.046574+0000', 'last_clean': '2023-05-22T15:46:23.046574+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-05-22T15:46:23.086294+0000', 'last_undegraded': '2023-05-22T15:46:23.086294+0000', 'last_fullsized': '2023-05-22T15:46:23.086294+0000', 'mapping_epoch': 145, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 145, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-05-22T15:46:23.046574+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-05-22T15:46:23.046574+0000', 'last_clean_scrub_stamp': '2023-05-22T15:46:23.046574+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-05-24T00:20:12.734869+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 2], 'acting': [7, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2023-05-22T15:46:28.366 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.20 is not active+clean 2023-05-22T15:46:28.367 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.20', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 145, 'state': 'creating+peering', 'last_fresh': '2023-05-22T15:46:23.074629+0000', 'last_change': '2023-05-22T15:46:23.074629+0000', 'last_active': '2023-05-22T15:46:23.046574+0000', 'last_peered': '2023-05-22T15:46:23.046574+0000', 'last_clean': '2023-05-22T15:46:23.046574+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-05-22T15:46:23.074629+0000', 'last_undegraded': '2023-05-22T15:46:23.074629+0000', 'last_fullsized': '2023-05-22T15:46:23.074629+0000', 'mapping_epoch': 145, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 145, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-05-22T15:46:23.046574+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-05-22T15:46:23.046574+0000', 'last_clean_scrub_stamp': '2023-05-22T15:46:23.046574+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-05-24T00:52:03.083872+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 3], 'acting': [4, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2023-05-22T15:46:28.367 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.1e is not active+clean 2023-05-22T15:46:28.368 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.1e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 145, 'state': 'creating+peering', 'last_fresh': '2023-05-22T15:46:23.061067+0000', 'last_change': '2023-05-22T15:46:23.061067+0000', 'last_active': '2023-05-22T15:46:23.046574+0000', 'last_peered': '2023-05-22T15:46:23.046574+0000', 'last_clean': '2023-05-22T15:46:23.046574+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-05-22T15:46:23.061067+0000', 'last_undegraded': '2023-05-22T15:46:23.061067+0000', 'last_fullsized': '2023-05-22T15:46:23.061067+0000', 'mapping_epoch': 145, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 145, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-05-22T15:46:23.046574+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-05-22T15:46:23.046574+0000', 'last_clean_scrub_stamp': '2023-05-22T15:46:23.046574+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-05-24T01:29:20.717884+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 5], 'acting': [3, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2023-05-22T15:46:28.368 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.1f is not active+clean 2023-05-22T15:46:28.368 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.1f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 145, 'state': 'creating+peering', 'last_fresh': '2023-05-22T15:46:23.071619+0000', 'last_change': '2023-05-22T15:46:23.071619+0000', 'last_active': '2023-05-22T15:46:23.046574+0000', 'last_peered': '2023-05-22T15:46:23.046574+0000', 'last_clean': '2023-05-22T15:46:23.046574+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-05-22T15:46:23.071619+0000', 'last_undegraded': '2023-05-22T15:46:23.071619+0000', 'last_fullsized': '2023-05-22T15:46:23.071619+0000', 'mapping_epoch': 145, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 145, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-05-22T15:46:23.046574+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-05-22T15:46:23.046574+0000', 'last_clean_scrub_stamp': '2023-05-22T15:46:23.046574+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-05-24T00:22:11.366110+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 2], 'acting': [6, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2023-05-22T15:46:28.369 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.1c is not active+clean 2023-05-22T15:46:28.369 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.1c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 145, 'state': 'creating+peering', 'last_fresh': '2023-05-22T15:46:23.089330+0000', 'last_change': '2023-05-22T15:46:23.089330+0000', 'last_active': '2023-05-22T15:46:23.046574+0000', 'last_peered': '2023-05-22T15:46:23.046574+0000', 'last_clean': '2023-05-22T15:46:23.046574+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-05-22T15:46:23.089330+0000', 'last_undegraded': '2023-05-22T15:46:23.089330+0000', 'last_fullsized': '2023-05-22T15:46:23.089330+0000', 'mapping_epoch': 145, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 145, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-05-22T15:46:23.046574+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-05-22T15:46:23.046574+0000', 'last_clean_scrub_stamp': '2023-05-22T15:46:23.046574+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-05-23T22:47:45.260369+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 4], 'acting': [7, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2023-05-22T15:46:28.369 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.1d is not active+clean 2023-05-22T15:46:28.370 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.1d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 145, 'state': 'creating+peering', 'last_fresh': '2023-05-22T15:46:23.090978+0000', 'last_change': '2023-05-22T15:46:23.090978+0000', 'last_active': '2023-05-22T15:46:23.046574+0000', 'last_peered': '2023-05-22T15:46:23.046574+0000', 'last_clean': '2023-05-22T15:46:23.046574+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-05-22T15:46:23.090978+0000', 'last_undegraded': '2023-05-22T15:46:23.090978+0000', 'last_fullsized': '2023-05-22T15:46:23.090978+0000', 'mapping_epoch': 145, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 145, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-05-22T15:46:23.046574+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-05-22T15:46:23.046574+0000', 'last_clean_scrub_stamp': '2023-05-22T15:46:23.046574+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-05-23T17:07:01.166140+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 1], 'acting': [7, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2023-05-22T15:46:28.370 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.1a is not active+clean 2023-05-22T15:46:28.370 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.1a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 145, 'state': 'creating+peering', 'last_fresh': '2023-05-22T15:46:23.061711+0000', 'last_change': '2023-05-22T15:46:23.061711+0000', 'last_active': '2023-05-22T15:46:23.046574+0000', 'last_peered': '2023-05-22T15:46:23.046574+0000', 'last_clean': '2023-05-22T15:46:23.046574+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-05-22T15:46:23.061711+0000', 'last_undegraded': '2023-05-22T15:46:23.061711+0000', 'last_fullsized': '2023-05-22T15:46:23.061711+0000', 'mapping_epoch': 145, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 145, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-05-22T15:46:23.046574+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-05-22T15:46:23.046574+0000', 'last_clean_scrub_stamp': '2023-05-22T15:46:23.046574+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-05-23T20:23:42.844820+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 4], 'acting': [3, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2023-05-22T15:46:28.371 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.1b is not active+clean 2023-05-22T15:46:28.371 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.1b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 145, 'state': 'creating+peering', 'last_fresh': '2023-05-22T15:46:23.073120+0000', 'last_change': '2023-05-22T15:46:23.073120+0000', 'last_active': '2023-05-22T15:46:23.046574+0000', 'last_peered': '2023-05-22T15:46:23.046574+0000', 'last_clean': '2023-05-22T15:46:23.046574+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-05-22T15:46:23.073120+0000', 'last_undegraded': '2023-05-22T15:46:23.073120+0000', 'last_fullsized': '2023-05-22T15:46:23.073120+0000', 'mapping_epoch': 145, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 145, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-05-22T15:46:23.046574+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-05-22T15:46:23.046574+0000', 'last_clean_scrub_stamp': '2023-05-22T15:46:23.046574+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-05-24T01:59:24.451915+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 0], 'acting': [4, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2023-05-22T15:46:28.371 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.18 is not active+clean 2023-05-22T15:46:28.371 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.18', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 145, 'state': 'creating+peering', 'last_fresh': '2023-05-22T15:46:23.077114+0000', 'last_change': '2023-05-22T15:46:23.077114+0000', 'last_active': '2023-05-22T15:46:23.046574+0000', 'last_peered': '2023-05-22T15:46:23.046574+0000', 'last_clean': '2023-05-22T15:46:23.046574+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-05-22T15:46:23.077114+0000', 'last_undegraded': '2023-05-22T15:46:23.077114+0000', 'last_fullsized': '2023-05-22T15:46:23.077114+0000', 'mapping_epoch': 145, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 145, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-05-22T15:46:23.046574+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-05-22T15:46:23.046574+0000', 'last_clean_scrub_stamp': '2023-05-22T15:46:23.046574+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-05-24T03:20:39.103248+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 3], 'acting': [6, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2023-05-22T15:46:28.372 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.19 is not active+clean 2023-05-22T15:46:28.372 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.19', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 145, 'state': 'creating+peering', 'last_fresh': '2023-05-22T15:46:23.075186+0000', 'last_change': '2023-05-22T15:46:23.075186+0000', 'last_active': '2023-05-22T15:46:23.046574+0000', 'last_peered': '2023-05-22T15:46:23.046574+0000', 'last_clean': '2023-05-22T15:46:23.046574+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-05-22T15:46:23.075186+0000', 'last_undegraded': '2023-05-22T15:46:23.075186+0000', 'last_fullsized': '2023-05-22T15:46:23.075186+0000', 'mapping_epoch': 145, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 145, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-05-22T15:46:23.046574+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-05-22T15:46:23.046574+0000', 'last_clean_scrub_stamp': '2023-05-22T15:46:23.046574+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-05-24T01:22:20.631045+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3], 'acting': [1, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-05-22T15:46:28.372 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.e is not active+clean 2023-05-22T15:46:28.373 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 145, 'state': 'creating+peering', 'last_fresh': '2023-05-22T15:46:23.083614+0000', 'last_change': '2023-05-22T15:46:23.083614+0000', 'last_active': '2023-05-22T15:46:23.046574+0000', 'last_peered': '2023-05-22T15:46:23.046574+0000', 'last_clean': '2023-05-22T15:46:23.046574+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-05-22T15:46:23.083614+0000', 'last_undegraded': '2023-05-22T15:46:23.083614+0000', 'last_fullsized': '2023-05-22T15:46:23.083614+0000', 'mapping_epoch': 145, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 145, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-05-22T15:46:23.046574+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-05-22T15:46:23.046574+0000', 'last_clean_scrub_stamp': '2023-05-22T15:46:23.046574+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-05-24T01:20:55.836997+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 2], 'acting': [6, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2023-05-22T15:46:28.373 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.f is not active+clean 2023-05-22T15:46:28.373 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 145, 'state': 'creating+peering', 'last_fresh': '2023-05-22T15:46:23.063425+0000', 'last_change': '2023-05-22T15:46:23.063425+0000', 'last_active': '2023-05-22T15:46:23.046574+0000', 'last_peered': '2023-05-22T15:46:23.046574+0000', 'last_clean': '2023-05-22T15:46:23.046574+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-05-22T15:46:23.063425+0000', 'last_undegraded': '2023-05-22T15:46:23.063425+0000', 'last_fullsized': '2023-05-22T15:46:23.063425+0000', 'mapping_epoch': 145, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 145, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-05-22T15:46:23.046574+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-05-22T15:46:23.046574+0000', 'last_clean_scrub_stamp': '2023-05-22T15:46:23.046574+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-05-23T16:10:54.516325+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 0], 'acting': [5, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2023-05-22T15:46:28.374 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.c is not active+clean 2023-05-22T15:46:28.374 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 145, 'state': 'creating+peering', 'last_fresh': '2023-05-22T15:46:23.060697+0000', 'last_change': '2023-05-22T15:46:23.060697+0000', 'last_active': '2023-05-22T15:46:23.046574+0000', 'last_peered': '2023-05-22T15:46:23.046574+0000', 'last_clean': '2023-05-22T15:46:23.046574+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-05-22T15:46:23.060697+0000', 'last_undegraded': '2023-05-22T15:46:23.060697+0000', 'last_fullsized': '2023-05-22T15:46:23.060697+0000', 'mapping_epoch': 145, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 145, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-05-22T15:46:23.046574+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-05-22T15:46:23.046574+0000', 'last_clean_scrub_stamp': '2023-05-22T15:46:23.046574+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-05-24T02:33:34.943631+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2023-05-22T15:46:28.374 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.d is not active+clean 2023-05-22T15:46:28.375 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 145, 'state': 'creating+peering', 'last_fresh': '2023-05-22T15:46:23.073232+0000', 'last_change': '2023-05-22T15:46:23.073232+0000', 'last_active': '2023-05-22T15:46:23.046574+0000', 'last_peered': '2023-05-22T15:46:23.046574+0000', 'last_clean': '2023-05-22T15:46:23.046574+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-05-22T15:46:23.073232+0000', 'last_undegraded': '2023-05-22T15:46:23.073232+0000', 'last_fullsized': '2023-05-22T15:46:23.073232+0000', 'mapping_epoch': 145, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 145, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-05-22T15:46:23.046574+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-05-22T15:46:23.046574+0000', 'last_clean_scrub_stamp': '2023-05-22T15:46:23.046574+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-05-23T21:49:48.656654+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3], 'acting': [1, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-05-22T15:46:28.375 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.a is not active+clean 2023-05-22T15:46:28.375 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 145, 'state': 'creating+peering', 'last_fresh': '2023-05-22T15:46:23.072991+0000', 'last_change': '2023-05-22T15:46:23.072991+0000', 'last_active': '2023-05-22T15:46:23.046574+0000', 'last_peered': '2023-05-22T15:46:23.046574+0000', 'last_clean': '2023-05-22T15:46:23.046574+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-05-22T15:46:23.072991+0000', 'last_undegraded': '2023-05-22T15:46:23.072991+0000', 'last_fullsized': '2023-05-22T15:46:23.072991+0000', 'mapping_epoch': 145, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 145, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-05-22T15:46:23.046574+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-05-22T15:46:23.046574+0000', 'last_clean_scrub_stamp': '2023-05-22T15:46:23.046574+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-05-23T23:24:33.321639+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 4], 'acting': [2, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-05-22T15:46:28.376 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.b is not active+clean 2023-05-22T15:46:28.376 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 145, 'state': 'creating+peering', 'last_fresh': '2023-05-22T15:46:23.071529+0000', 'last_change': '2023-05-22T15:46:23.071529+0000', 'last_active': '2023-05-22T15:46:23.046574+0000', 'last_peered': '2023-05-22T15:46:23.046574+0000', 'last_clean': '2023-05-22T15:46:23.046574+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-05-22T15:46:23.071529+0000', 'last_undegraded': '2023-05-22T15:46:23.071529+0000', 'last_fullsized': '2023-05-22T15:46:23.071529+0000', 'mapping_epoch': 145, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 145, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-05-22T15:46:23.046574+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-05-22T15:46:23.046574+0000', 'last_clean_scrub_stamp': '2023-05-22T15:46:23.046574+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-05-24T00:20:12.734869+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 6], 'acting': [4, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2023-05-22T15:46:28.376 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.8 is not active+clean 2023-05-22T15:46:28.377 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.8', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 145, 'state': 'creating+peering', 'last_fresh': '2023-05-22T15:46:23.071955+0000', 'last_change': '2023-05-22T15:46:23.071955+0000', 'last_active': '2023-05-22T15:46:23.046574+0000', 'last_peered': '2023-05-22T15:46:23.046574+0000', 'last_clean': '2023-05-22T15:46:23.046574+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-05-22T15:46:23.071955+0000', 'last_undegraded': '2023-05-22T15:46:23.071955+0000', 'last_fullsized': '2023-05-22T15:46:23.071955+0000', 'mapping_epoch': 145, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 145, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-05-22T15:46:23.046574+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-05-22T15:46:23.046574+0000', 'last_clean_scrub_stamp': '2023-05-22T15:46:23.046574+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-05-23T16:27:47.997551+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [1, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-05-22T15:46:28.377 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.9 is not active+clean 2023-05-22T15:46:28.377 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.9', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 145, 'state': 'creating+peering', 'last_fresh': '2023-05-22T15:46:23.089066+0000', 'last_change': '2023-05-22T15:46:23.089066+0000', 'last_active': '2023-05-22T15:46:23.046574+0000', 'last_peered': '2023-05-22T15:46:23.046574+0000', 'last_clean': '2023-05-22T15:46:23.046574+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-05-22T15:46:23.089066+0000', 'last_undegraded': '2023-05-22T15:46:23.089066+0000', 'last_fullsized': '2023-05-22T15:46:23.089066+0000', 'mapping_epoch': 145, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 145, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-05-22T15:46:23.046574+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-05-22T15:46:23.046574+0000', 'last_clean_scrub_stamp': '2023-05-22T15:46:23.046574+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-05-23T20:31:09.541110+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 2], 'acting': [7, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2023-05-22T15:46:28.378 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.6 is not active+clean 2023-05-22T15:46:28.378 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.6', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 145, 'state': 'creating+peering', 'last_fresh': '2023-05-22T15:46:23.076413+0000', 'last_change': '2023-05-22T15:46:23.076413+0000', 'last_active': '2023-05-22T15:46:23.046574+0000', 'last_peered': '2023-05-22T15:46:23.046574+0000', 'last_clean': '2023-05-22T15:46:23.046574+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-05-22T15:46:23.076413+0000', 'last_undegraded': '2023-05-22T15:46:23.076413+0000', 'last_fullsized': '2023-05-22T15:46:23.076413+0000', 'mapping_epoch': 145, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 145, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-05-22T15:46:23.046574+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-05-22T15:46:23.046574+0000', 'last_clean_scrub_stamp': '2023-05-22T15:46:23.046574+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-05-23T20:55:41.010860+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 5], 'acting': [4, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2023-05-22T15:46:28.378 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.7 is not active+clean 2023-05-22T15:46:28.378 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.7', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 145, 'state': 'creating+peering', 'last_fresh': '2023-05-22T15:46:23.079702+0000', 'last_change': '2023-05-22T15:46:23.079702+0000', 'last_active': '2023-05-22T15:46:23.046574+0000', 'last_peered': '2023-05-22T15:46:23.046574+0000', 'last_clean': '2023-05-22T15:46:23.046574+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-05-22T15:46:23.079702+0000', 'last_undegraded': '2023-05-22T15:46:23.079702+0000', 'last_fullsized': '2023-05-22T15:46:23.079702+0000', 'mapping_epoch': 145, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 145, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-05-22T15:46:23.046574+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-05-22T15:46:23.046574+0000', 'last_clean_scrub_stamp': '2023-05-22T15:46:23.046574+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-05-24T02:20:29.148898+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 6], 'acting': [2, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-05-22T15:46:28.379 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.4 is not active+clean 2023-05-22T15:46:28.379 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.4', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 145, 'state': 'creating+peering', 'last_fresh': '2023-05-22T15:46:23.075841+0000', 'last_change': '2023-05-22T15:46:23.075841+0000', 'last_active': '2023-05-22T15:46:23.046574+0000', 'last_peered': '2023-05-22T15:46:23.046574+0000', 'last_clean': '2023-05-22T15:46:23.046574+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-05-22T15:46:23.075841+0000', 'last_undegraded': '2023-05-22T15:46:23.075841+0000', 'last_fullsized': '2023-05-22T15:46:23.075841+0000', 'mapping_epoch': 145, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 145, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-05-22T15:46:23.046574+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-05-22T15:46:23.046574+0000', 'last_clean_scrub_stamp': '2023-05-22T15:46:23.046574+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-05-23T17:46:38.837010+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 7], 'acting': [1, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-05-22T15:46:28.379 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.5 is not active+clean 2023-05-22T15:46:28.380 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.5', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 145, 'state': 'creating+peering', 'last_fresh': '2023-05-22T15:46:23.073829+0000', 'last_change': '2023-05-22T15:46:23.073829+0000', 'last_active': '2023-05-22T15:46:23.046574+0000', 'last_peered': '2023-05-22T15:46:23.046574+0000', 'last_clean': '2023-05-22T15:46:23.046574+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-05-22T15:46:23.073829+0000', 'last_undegraded': '2023-05-22T15:46:23.073829+0000', 'last_fullsized': '2023-05-22T15:46:23.073829+0000', 'mapping_epoch': 145, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 145, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-05-22T15:46:23.046574+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-05-22T15:46:23.046574+0000', 'last_clean_scrub_stamp': '2023-05-22T15:46:23.046574+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-05-23T22:47:45.260369+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 2], 'acting': [4, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2023-05-22T15:46:28.380 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.2 is not active+clean 2023-05-22T15:46:28.380 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.2', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 145, 'state': 'creating+peering', 'last_fresh': '2023-05-22T15:46:23.084350+0000', 'last_change': '2023-05-22T15:46:23.084350+0000', 'last_active': '2023-05-22T15:46:23.046574+0000', 'last_peered': '2023-05-22T15:46:23.046574+0000', 'last_clean': '2023-05-22T15:46:23.046574+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-05-22T15:46:23.084350+0000', 'last_undegraded': '2023-05-22T15:46:23.084350+0000', 'last_fullsized': '2023-05-22T15:46:23.084350+0000', 'mapping_epoch': 145, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 145, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-05-22T15:46:23.046574+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-05-22T15:46:23.046574+0000', 'last_clean_scrub_stamp': '2023-05-22T15:46:23.046574+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-05-23T19:25:55.107784+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 4], 'acting': [7, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2023-05-22T15:46:28.381 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.3 is not active+clean 2023-05-22T15:46:28.381 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.3', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 145, 'state': 'creating+peering', 'last_fresh': '2023-05-22T15:46:23.062480+0000', 'last_change': '2023-05-22T15:46:23.062480+0000', 'last_active': '2023-05-22T15:46:23.046574+0000', 'last_peered': '2023-05-22T15:46:23.046574+0000', 'last_clean': '2023-05-22T15:46:23.046574+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-05-22T15:46:23.062480+0000', 'last_undegraded': '2023-05-22T15:46:23.062480+0000', 'last_fullsized': '2023-05-22T15:46:23.062480+0000', 'mapping_epoch': 145, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 145, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-05-22T15:46:23.046574+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-05-22T15:46:23.046574+0000', 'last_clean_scrub_stamp': '2023-05-22T15:46:23.046574+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-05-23T18:05:16.988218+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 2], 'acting': [5, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2023-05-22T15:46:28.381 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.0 is not active+clean 2023-05-22T15:46:28.382 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.0', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 145, 'state': 'creating+peering', 'last_fresh': '2023-05-22T15:46:23.062429+0000', 'last_change': '2023-05-22T15:46:23.062429+0000', 'last_active': '2023-05-22T15:46:23.046574+0000', 'last_peered': '2023-05-22T15:46:23.046574+0000', 'last_clean': '2023-05-22T15:46:23.046574+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-05-22T15:46:23.062429+0000', 'last_undegraded': '2023-05-22T15:46:23.062429+0000', 'last_fullsized': '2023-05-22T15:46:23.062429+0000', 'mapping_epoch': 145, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 145, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-05-22T15:46:23.046574+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-05-22T15:46:23.046574+0000', 'last_clean_scrub_stamp': '2023-05-22T15:46:23.046574+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-05-24T01:03:37.104570+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 6], 'acting': [3, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2023-05-22T15:46:28.382 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.1 is not active+clean 2023-05-22T15:46:28.382 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.1', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 145, 'state': 'creating+peering', 'last_fresh': '2023-05-22T15:46:23.072920+0000', 'last_change': '2023-05-22T15:46:23.072920+0000', 'last_active': '2023-05-22T15:46:23.046574+0000', 'last_peered': '2023-05-22T15:46:23.046574+0000', 'last_clean': '2023-05-22T15:46:23.046574+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-05-22T15:46:23.072920+0000', 'last_undegraded': '2023-05-22T15:46:23.072920+0000', 'last_fullsized': '2023-05-22T15:46:23.072920+0000', 'mapping_epoch': 145, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 145, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-05-22T15:46:23.046574+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-05-22T15:46:23.046574+0000', 'last_clean_scrub_stamp': '2023-05-22T15:46:23.046574+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-05-23T17:16:20.885635+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 7], 'acting': [1, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-05-22T15:46:28.383 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.12 is not active+clean 2023-05-22T15:46:28.383 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.12', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 145, 'state': 'creating+peering', 'last_fresh': '2023-05-22T15:46:23.085873+0000', 'last_change': '2023-05-22T15:46:23.085873+0000', 'last_active': '2023-05-22T15:46:23.046574+0000', 'last_peered': '2023-05-22T15:46:23.046574+0000', 'last_clean': '2023-05-22T15:46:23.046574+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-05-22T15:46:23.085873+0000', 'last_undegraded': '2023-05-22T15:46:23.085873+0000', 'last_fullsized': '2023-05-22T15:46:23.085873+0000', 'mapping_epoch': 145, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 145, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-05-22T15:46:23.046574+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-05-22T15:46:23.046574+0000', 'last_clean_scrub_stamp': '2023-05-22T15:46:23.046574+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-05-24T01:20:55.836997+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 2], 'acting': [7, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2023-05-22T15:46:28.383 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.11 is not active+clean 2023-05-22T15:46:28.384 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.11', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 145, 'state': 'creating+peering', 'last_fresh': '2023-05-22T15:46:23.082774+0000', 'last_change': '2023-05-22T15:46:23.082774+0000', 'last_active': '2023-05-22T15:46:23.046574+0000', 'last_peered': '2023-05-22T15:46:23.046574+0000', 'last_clean': '2023-05-22T15:46:23.046574+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-05-22T15:46:23.082774+0000', 'last_undegraded': '2023-05-22T15:46:23.082774+0000', 'last_fullsized': '2023-05-22T15:46:23.082774+0000', 'mapping_epoch': 145, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 145, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-05-22T15:46:23.046574+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-05-22T15:46:23.046574+0000', 'last_clean_scrub_stamp': '2023-05-22T15:46:23.046574+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-05-23T22:37:33.613777+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 5], 'acting': [6, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2023-05-22T15:46:28.384 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.10 is not active+clean 2023-05-22T15:46:28.384 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.10', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 145, 'state': 'creating+peering', 'last_fresh': '2023-05-22T15:46:23.073353+0000', 'last_change': '2023-05-22T15:46:23.073353+0000', 'last_active': '2023-05-22T15:46:23.046574+0000', 'last_peered': '2023-05-22T15:46:23.046574+0000', 'last_clean': '2023-05-22T15:46:23.046574+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-05-22T15:46:23.073353+0000', 'last_undegraded': '2023-05-22T15:46:23.073353+0000', 'last_fullsized': '2023-05-22T15:46:23.073353+0000', 'mapping_epoch': 145, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 145, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-05-22T15:46:23.046574+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-05-22T15:46:23.046574+0000', 'last_clean_scrub_stamp': '2023-05-22T15:46:23.046574+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-05-23T19:42:45.119566+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 6], 'acting': [1, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-05-22T15:46:28.385 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.13 is not active+clean 2023-05-22T15:46:28.385 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.13', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 145, 'state': 'creating+peering', 'last_fresh': '2023-05-22T15:46:23.076508+0000', 'last_change': '2023-05-22T15:46:23.076508+0000', 'last_active': '2023-05-22T15:46:23.046574+0000', 'last_peered': '2023-05-22T15:46:23.046574+0000', 'last_clean': '2023-05-22T15:46:23.046574+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-05-22T15:46:23.076508+0000', 'last_undegraded': '2023-05-22T15:46:23.076508+0000', 'last_fullsized': '2023-05-22T15:46:23.076508+0000', 'mapping_epoch': 145, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 145, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-05-22T15:46:23.046574+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-05-22T15:46:23.046574+0000', 'last_clean_scrub_stamp': '2023-05-22T15:46:23.046574+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-05-23T16:27:47.997551+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0], 'acting': [2, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-05-22T15:46:28.385 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.15 is not active+clean 2023-05-22T15:46:28.386 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.15', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 145, 'state': 'creating+peering', 'last_fresh': '2023-05-22T15:46:23.084525+0000', 'last_change': '2023-05-22T15:46:23.084525+0000', 'last_active': '2023-05-22T15:46:23.046574+0000', 'last_peered': '2023-05-22T15:46:23.046574+0000', 'last_clean': '2023-05-22T15:46:23.046574+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-05-22T15:46:23.084525+0000', 'last_undegraded': '2023-05-22T15:46:23.084525+0000', 'last_fullsized': '2023-05-22T15:46:23.084525+0000', 'mapping_epoch': 145, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 145, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-05-22T15:46:23.046574+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-05-22T15:46:23.046574+0000', 'last_clean_scrub_stamp': '2023-05-22T15:46:23.046574+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-05-24T03:25:18.349611+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2023-05-22T15:46:28.386 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.14 is not active+clean 2023-05-22T15:46:28.386 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.14', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 145, 'state': 'creating+peering', 'last_fresh': '2023-05-22T15:46:23.088461+0000', 'last_change': '2023-05-22T15:46:23.088461+0000', 'last_active': '2023-05-22T15:46:23.046574+0000', 'last_peered': '2023-05-22T15:46:23.046574+0000', 'last_clean': '2023-05-22T15:46:23.046574+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-05-22T15:46:23.088461+0000', 'last_undegraded': '2023-05-22T15:46:23.088461+0000', 'last_fullsized': '2023-05-22T15:46:23.088461+0000', 'mapping_epoch': 145, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 145, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-05-22T15:46:23.046574+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-05-22T15:46:23.046574+0000', 'last_clean_scrub_stamp': '2023-05-22T15:46:23.046574+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-05-23T20:49:34.728176+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 3], 'acting': [7, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2023-05-22T15:46:28.387 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.17 is not active+clean 2023-05-22T15:46:28.387 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.17', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 145, 'state': 'creating+peering', 'last_fresh': '2023-05-22T15:46:23.065336+0000', 'last_change': '2023-05-22T15:46:23.065336+0000', 'last_active': '2023-05-22T15:46:23.046574+0000', 'last_peered': '2023-05-22T15:46:23.046574+0000', 'last_clean': '2023-05-22T15:46:23.046574+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-05-22T15:46:23.065336+0000', 'last_undegraded': '2023-05-22T15:46:23.065336+0000', 'last_fullsized': '2023-05-22T15:46:23.065336+0000', 'mapping_epoch': 145, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 145, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-05-22T15:46:23.046574+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-05-22T15:46:23.046574+0000', 'last_clean_scrub_stamp': '2023-05-22T15:46:23.046574+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-05-23T16:27:47.997551+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 3], 'acting': [5, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2023-05-22T15:46:28.387 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.16 is not active+clean 2023-05-22T15:46:28.387 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.16', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 145, 'state': 'creating+peering', 'last_fresh': '2023-05-22T15:46:23.071629+0000', 'last_change': '2023-05-22T15:46:23.071629+0000', 'last_active': '2023-05-22T15:46:23.046574+0000', 'last_peered': '2023-05-22T15:46:23.046574+0000', 'last_clean': '2023-05-22T15:46:23.046574+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-05-22T15:46:23.071629+0000', 'last_undegraded': '2023-05-22T15:46:23.071629+0000', 'last_fullsized': '2023-05-22T15:46:23.071629+0000', 'mapping_epoch': 145, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 145, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-05-22T15:46:23.046574+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-05-22T15:46:23.046574+0000', 'last_clean_scrub_stamp': '2023-05-22T15:46:23.046574+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-05-23T18:29:45.269794+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0], 'acting': [2, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-05-22T15:46:28.388 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2023-05-22T15:46:28.517 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.511+0000 7ff6f8e8d640 1 Processor -- start 2023-05-22T15:46:28.518 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.511+0000 7ff6f8e8d640 1 -- start start 2023-05-22T15:46:28.519 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.511+0000 7ff6f8e8d640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7ff6f40ca4e0 0x7ff6f415e460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:28.519 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.515+0000 7ff6f8e8d640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff6f4070df0 0x7ff6f4071210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:28.519 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.515+0000 7ff6f8e8d640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6f40b0650 con 0x7ff6f40c9b40 2023-05-22T15:46:28.519 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.515+0000 7ff6f8e8d640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6f40b0370 con 0x7ff6f40ca4e0 2023-05-22T15:46:28.519 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.515+0000 7ff6f8e8d640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6f415d980 con 0x7ff6f4070df0 2023-05-22T15:46:28.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.515+0000 7ff6f2d9d640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff6f4070df0 0x7ff6f4071210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:28.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.515+0000 7ff6f259c640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7ff6f40ca4e0 0x7ff6f415e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:28.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.515+0000 7ff6ebfff640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7ff6f40c9b40 0x7ff6f40afb40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.16:38008/0 (socket says 172.21.15.16:38008) 2023-05-22T15:46:28.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.515+0000 7ff6ebfff640 1 -- 172.21.15.16:0/2073318512 learned_addr learned my addr 172.21.15.16:0/2073318512 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:46:28.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.515+0000 7ff6f2d9d640 1 -- 172.21.15.16:0/2073318512 >> v2:172.21.15.62:3300/0 conn(0x7ff6f40ca4e0 msgr2=0x7ff6f415e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:46:28.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.515+0000 7ff6f2d9d640 1 --2- 172.21.15.16:0/2073318512 >> v2:172.21.15.62:3300/0 conn(0x7ff6f40ca4e0 0x7ff6f415e460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:28.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.515+0000 7ff6f2d9d640 1 -- 172.21.15.16:0/2073318512 >> v1:172.21.15.62:6789/0 conn(0x7ff6f40c9b40 legacy=0x7ff6f40afb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:28.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.515+0000 7ff6f2d9d640 1 -- 172.21.15.16:0/2073318512 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6f415dc60 con 0x7ff6f4070df0 2023-05-22T15:46:28.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.519+0000 7ff6f2d9d640 1 --2- 172.21.15.16:0/2073318512 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff6f4070df0 0x7ff6f4071210 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7ff6ec009d90 tx=0x7ff6ec01c920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fb1f1209fd3a289d server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:28.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.519+0000 7ff6f1d9b640 1 -- 172.21.15.16:0/2073318512 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6ec0a5940 con 0x7ff6f4070df0 2023-05-22T15:46:28.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.519+0000 7ff6f1d9b640 1 -- 172.21.15.16:0/2073318512 <== mon.2 v2:172.21.15.62:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6ec0a5aa0 con 0x7ff6f4070df0 2023-05-22T15:46:28.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.519+0000 7ff6f1d9b640 1 -- 172.21.15.16:0/2073318512 <== mon.2 v2:172.21.15.62:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6ec0a5e30 con 0x7ff6f4070df0 2023-05-22T15:46:28.527 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.519+0000 7ff6f8e8d640 1 -- 172.21.15.16:0/2073318512 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff6f4070df0 msgr2=0x7ff6f4071210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:28.527 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.519+0000 7ff6f8e8d640 1 --2- 172.21.15.16:0/2073318512 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff6f4070df0 0x7ff6f4071210 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7ff6ec009d90 tx=0x7ff6ec01c920 comp rx=0 tx=0).stop 2023-05-22T15:46:28.527 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.519+0000 7ff6f8e8d640 1 -- 172.21.15.16:0/2073318512 shutdown_connections 2023-05-22T15:46:28.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.519+0000 7ff6f8e8d640 1 --2- 172.21.15.16:0/2073318512 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff6f4070df0 0x7ff6f4071210 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:28.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.519+0000 7ff6f8e8d640 1 --2- 172.21.15.16:0/2073318512 >> v2:172.21.15.62:3300/0 conn(0x7ff6f40ca4e0 0x7ff6f415e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:28.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.519+0000 7ff6f8e8d640 1 -- 172.21.15.16:0/2073318512 >> 172.21.15.16:0/2073318512 conn(0x7ff6f4069720 msgr2=0x7ff6f4069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:46:28.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.519+0000 7ff6f8e8d640 1 -- 172.21.15.16:0/2073318512 shutdown_connections 2023-05-22T15:46:28.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.519+0000 7ff6f8e8d640 1 -- 172.21.15.16:0/2073318512 wait complete. 2023-05-22T15:46:28.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.519+0000 7ff6f8e8d640 1 Processor -- start 2023-05-22T15:46:28.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.523+0000 7ff6f8e8d640 1 -- start start 2023-05-22T15:46:28.531 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.523+0000 7ff6f8e8d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff6f4070df0 0x7ff6f414c420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:28.532 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.523+0000 7ff6f8e8d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff6f40c9b40 0x7ff6f4158170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:28.532 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.523+0000 7ff6f8e8d640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff6f40ca4e0 0x7ff6f40af020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:28.532 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.523+0000 7ff6f8e8d640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6f4158c50 con 0x7ff6f40c9b40 2023-05-22T15:46:28.532 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.523+0000 7ff6f8e8d640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6f4158f30 con 0x7ff6f4070df0 2023-05-22T15:46:28.532 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.523+0000 7ff6f8e8d640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6f4159210 con 0x7ff6f40ca4e0 2023-05-22T15:46:28.532 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.523+0000 7ff6ebfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff6f40c9b40 0x7ff6f4158170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:28.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.523+0000 7ff6ebfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff6f40c9b40 0x7ff6f4158170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34646/0 (socket says 172.21.15.16:34646) 2023-05-22T15:46:28.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.523+0000 7ff6ebfff640 1 -- 172.21.15.16:0/677418798 learned_addr learned my addr 172.21.15.16:0/677418798 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:46:28.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.523+0000 7ff6f259c640 1 --2- 172.21.15.16:0/677418798 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff6f4070df0 0x7ff6f414c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:28.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.523+0000 7ff6f2d9d640 1 --2- 172.21.15.16:0/677418798 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff6f40ca4e0 0x7ff6f40af020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:28.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.523+0000 7ff6ebfff640 1 -- 172.21.15.16:0/677418798 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff6f40ca4e0 msgr2=0x7ff6f40af020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:28.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.523+0000 7ff6ebfff640 1 --2- 172.21.15.16:0/677418798 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff6f40ca4e0 0x7ff6f40af020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:28.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.523+0000 7ff6ebfff640 1 -- 172.21.15.16:0/677418798 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff6f4070df0 msgr2=0x7ff6f414c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:28.534 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.523+0000 7ff6ebfff640 1 --2- 172.21.15.16:0/677418798 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff6f4070df0 0x7ff6f414c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:28.534 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.523+0000 7ff6ebfff640 1 -- 172.21.15.16:0/677418798 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6ec009a70 con 0x7ff6f40c9b40 2023-05-22T15:46:28.534 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.523+0000 7ff6f2d9d640 1 --2- 172.21.15.16:0/677418798 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff6f40ca4e0 0x7ff6f40af020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T15:46:28.534 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.523+0000 7ff6ebfff640 1 --2- 172.21.15.16:0/677418798 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff6f40c9b40 0x7ff6f4158170 secure :-1 s=READY pgs=1848 cs=0 l=1 rev1=1 crypto rx=0x7ff6dc005890 tx=0x7ff6dc00c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:28.534 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.523+0000 7ff6eaffd640 1 -- 172.21.15.16:0/677418798 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6dc005cf0 con 0x7ff6f40c9b40 2023-05-22T15:46:28.534 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.523+0000 7ff6eaffd640 1 -- 172.21.15.16:0/677418798 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6dc005e50 con 0x7ff6f40c9b40 2023-05-22T15:46:28.534 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.523+0000 7ff6f8e8d640 1 -- 172.21.15.16:0/677418798 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6f40c8e80 con 0x7ff6f40c9b40 2023-05-22T15:46:28.534 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.523+0000 7ff6eaffd640 1 -- 172.21.15.16:0/677418798 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6dc0072b0 con 0x7ff6f40c9b40 2023-05-22T15:46:28.535 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.523+0000 7ff6f8e8d640 1 -- 172.21.15.16:0/677418798 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6f40ca8b0 con 0x7ff6f40c9b40 2023-05-22T15:46:28.535 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.523+0000 7ff6f8e8d640 1 -- 172.21.15.16:0/677418798 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6f40c9010 con 0x7ff6f40c9b40 2023-05-22T15:46:28.535 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.523+0000 7ff6eaffd640 1 -- 172.21.15.16:0/677418798 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 60) v1 ==== 84332+0+0 (secure 0 0 0) 0x7ff6dc007410 con 0x7ff6f40c9b40 2023-05-22T15:46:28.535 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.523+0000 7ff6eaffd640 1 --2- 172.21.15.16:0/677418798 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ff6d40654a0 0x7ff6d4067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:28.535 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.527+0000 7ff6eaffd640 1 -- 172.21.15.16:0/677418798 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(147..147 src has 1..147) v4 ==== 6410+0+0 (secure 0 0 0) 0x7ff6dc081f50 con 0x7ff6f40c9b40 2023-05-22T15:46:28.535 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.527+0000 7ff6f259c640 1 --2- 172.21.15.16:0/677418798 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ff6d40654a0 0x7ff6d4067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:28.535 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.527+0000 7ff6f259c640 1 --2- 172.21.15.16:0/677418798 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ff6d40654a0 0x7ff6d4067950 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7ff6f415d570 tx=0x7ff6e4009190 comp rx=0 tx=0).ready entity=mgr.8835 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:28.536 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.531+0000 7ff6eaffd640 1 -- 172.21.15.16:0/677418798 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7ff6dc04faa0 con 0x7ff6f40c9b40 2023-05-22T15:46:28.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.739+0000 7ff6f8e8d640 1 -- 172.21.15.16:0/677418798 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7ff6f404dbe0 con 0x7ff6f40c9b40 2023-05-22T15:46:28.746 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.739+0000 7ff6eaffd640 1 -- 172.21.15.16:0/677418798 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+3232 (secure 0 0 0) 0x7ff6dc017cb0 con 0x7ff6f40c9b40 2023-05-22T15:46:28.747 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:46:28.747 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"b2719834-7847-4d98-90a6-f65d57f85c49","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1203,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":147,"num_osds":8,"num_up_osds":8,"osd_up_since":1684769196,"num_in_osds":8,"osd_in_since":1684769187,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"creating+peering","count":64},{"state_name":"active+clean","count":41}],"num_pgs":105,"num_pools":4,"num_objects":2,"data_bytes":590368,"bytes_used":230162432,"bytes_avail":772863950848,"bytes_total":773094113280,"inactive_pgs_ratio":0.60952383279800415},"fsmap":{"epoch":58,"id":8,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":8,"rank":0,"name":"a","status":"up:active","gid":9075}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":15,"modified":"2023-05-22T15:46:11.364997+0000","services":{"mds":{"daemons":{"summary":"","b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"d":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}},"mgr":{"daemons":{"summary":"","x":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"y":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}},"mon":{"daemons":{"summary":"","a":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}},"osd":{"daemons":{"summary":"","0":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"1":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"2":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"3":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"4":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"5":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"6":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"7":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2023-05-22T15:46:28.748 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.743+0000 7ff6f8e8d640 1 -- 172.21.15.16:0/677418798 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ff6d40654a0 msgr2=0x7ff6d4067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:28.748 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.743+0000 7ff6f8e8d640 1 --2- 172.21.15.16:0/677418798 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ff6d40654a0 0x7ff6d4067950 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7ff6f415d570 tx=0x7ff6e4009190 comp rx=0 tx=0).stop 2023-05-22T15:46:28.749 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.743+0000 7ff6f8e8d640 1 -- 172.21.15.16:0/677418798 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff6f40c9b40 msgr2=0x7ff6f4158170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:28.749 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.743+0000 7ff6f8e8d640 1 --2- 172.21.15.16:0/677418798 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff6f40c9b40 0x7ff6f4158170 secure :-1 s=READY pgs=1848 cs=0 l=1 rev1=1 crypto rx=0x7ff6dc005890 tx=0x7ff6dc00c960 comp rx=0 tx=0).stop 2023-05-22T15:46:28.749 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.743+0000 7ff6f8e8d640 1 -- 172.21.15.16:0/677418798 shutdown_connections 2023-05-22T15:46:28.750 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.743+0000 7ff6f8e8d640 1 --2- 172.21.15.16:0/677418798 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ff6d40654a0 0x7ff6d4067950 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:28.750 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.743+0000 7ff6f8e8d640 1 --2- 172.21.15.16:0/677418798 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff6f40ca4e0 0x7ff6f40af020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:28.750 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.743+0000 7ff6f8e8d640 1 --2- 172.21.15.16:0/677418798 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff6f40c9b40 0x7ff6f4158170 unknown :-1 s=CLOSED pgs=1848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:28.750 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.743+0000 7ff6f8e8d640 1 --2- 172.21.15.16:0/677418798 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff6f4070df0 0x7ff6f414c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:28.750 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.743+0000 7ff6f8e8d640 1 -- 172.21.15.16:0/677418798 >> 172.21.15.16:0/677418798 conn(0x7ff6f4069720 msgr2=0x7ff6f4155e30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:46:28.751 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.743+0000 7ff6f8e8d640 1 -- 172.21.15.16:0/677418798 shutdown_connections 2023-05-22T15:46:28.751 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.743+0000 7ff6f8e8d640 1 -- 172.21.15.16:0/677418798 wait complete. 2023-05-22T15:46:28.766 INFO:tasks.cephfs.filesystem.ceph_manager:no progress seen, keeping timeout for now 2023-05-22T15:46:28.767 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-05-22T15:46:28.895 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.891+0000 7f12102d7640 1 Processor -- start 2023-05-22T15:46:28.895 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.891+0000 7f12102d7640 1 -- start start 2023-05-22T15:46:28.897 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.891+0000 7f12102d7640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f12080ccab0 0x7f120815fe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:28.897 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.891+0000 7f12102d7640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f12080d0410 0x7f12080d0810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:28.898 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.891+0000 7f12102d7640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12080ccef0 con 0x7f12080b7dd0 2023-05-22T15:46:28.898 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.891+0000 7f12102d7640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12080d3360 con 0x7f12080d0410 2023-05-22T15:46:28.898 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.891+0000 7f12102d7640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12080cd170 con 0x7f12080ccab0 2023-05-22T15:46:28.898 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.891+0000 7f120e874640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f12080d0410 0x7f12080d0810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:28.899 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.891+0000 7f120e073640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f12080b7dd0 0x7f120815f700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:38900/0 (socket says 172.21.15.16:38900) 2023-05-22T15:46:28.899 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.891+0000 7f120e073640 1 -- 172.21.15.16:0/3674169708 learned_addr learned my addr 172.21.15.16:0/3674169708 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:46:28.899 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.891+0000 7f120d872640 1 --2- 172.21.15.16:0/3674169708 >> v2:172.21.15.62:3300/0 conn(0x7f12080ccab0 0x7f120815fe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:28.900 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.891+0000 7f120e874640 1 -- 172.21.15.16:0/3674169708 >> v2:172.21.15.62:3300/0 conn(0x7f12080ccab0 msgr2=0x7f120815fe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:46:28.900 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.891+0000 7f120e874640 1 --2- 172.21.15.16:0/3674169708 >> v2:172.21.15.62:3300/0 conn(0x7f12080ccab0 0x7f120815fe10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:28.900 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.891+0000 7f120e874640 1 -- 172.21.15.16:0/3674169708 >> v1:172.21.15.16:6789/0 conn(0x7f12080b7dd0 legacy=0x7f120815f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:28.900 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.891+0000 7f120d071640 1 -- 172.21.15.16:0/3674169708 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1759202434 0 0) 0x7f12080ccef0 con 0x7f12080b7dd0 2023-05-22T15:46:28.901 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.891+0000 7f120e874640 1 -- 172.21.15.16:0/3674169708 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12080d2ac0 con 0x7f12080d0410 2023-05-22T15:46:28.906 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.895+0000 7f120e874640 1 --2- 172.21.15.16:0/3674169708 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f12080d0410 0x7f12080d0810 secure :-1 s=READY pgs=1850 cs=0 l=1 rev1=1 crypto rx=0x7f12080b1320 tx=0x7f120001c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75f74c3b257ffb4d server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:28.906 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.895+0000 7f120d071640 1 -- 172.21.15.16:0/3674169708 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12000a59a0 con 0x7f12080d0410 2023-05-22T15:46:28.906 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.895+0000 7f120d071640 1 -- 172.21.15.16:0/3674169708 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12000a5b00 con 0x7f12080d0410 2023-05-22T15:46:28.906 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.895+0000 7f120d071640 1 -- 172.21.15.16:0/3674169708 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12000a5e90 con 0x7f12080d0410 2023-05-22T15:46:28.907 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.895+0000 7f12102d7640 1 -- 172.21.15.16:0/3674169708 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f12080d0410 msgr2=0x7f12080d0810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:28.907 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.895+0000 7f12102d7640 1 --2- 172.21.15.16:0/3674169708 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f12080d0410 0x7f12080d0810 secure :-1 s=READY pgs=1850 cs=0 l=1 rev1=1 crypto rx=0x7f12080b1320 tx=0x7f120001c8e0 comp rx=0 tx=0).stop 2023-05-22T15:46:28.907 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.895+0000 7f12102d7640 1 -- 172.21.15.16:0/3674169708 shutdown_connections 2023-05-22T15:46:28.908 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.895+0000 7f12102d7640 1 --2- 172.21.15.16:0/3674169708 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f12080d0410 0x7f12080d0810 unknown :-1 s=CLOSED pgs=1850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:28.908 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.895+0000 7f12102d7640 1 --2- 172.21.15.16:0/3674169708 >> v2:172.21.15.62:3300/0 conn(0x7f12080ccab0 0x7f120815fe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:28.908 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.895+0000 7f12102d7640 1 -- 172.21.15.16:0/3674169708 >> 172.21.15.16:0/3674169708 conn(0x7f12080655c0 msgr2=0x7f1208070e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:46:28.908 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.895+0000 7f12102d7640 1 -- 172.21.15.16:0/3674169708 shutdown_connections 2023-05-22T15:46:28.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.895+0000 7f12102d7640 1 -- 172.21.15.16:0/3674169708 wait complete. 2023-05-22T15:46:28.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.895+0000 7f12102d7640 1 Processor -- start 2023-05-22T15:46:28.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.895+0000 7f12102d7640 1 -- start start 2023-05-22T15:46:28.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.895+0000 7f12102d7640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f12080b7dd0 0x7f1208159880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:28.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.895+0000 7f12102d7640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f12080ccab0 0x7f120814d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:28.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.895+0000 7f12102d7640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f12080d0410 0x7f1208156170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:28.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.895+0000 7f12102d7640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12080cb3f0 con 0x7f12080ccab0 2023-05-22T15:46:28.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.895+0000 7f12102d7640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12080cb0c0 con 0x7f12080d0410 2023-05-22T15:46:28.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.895+0000 7f12102d7640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12080c4ac0 con 0x7f12080b7dd0 2023-05-22T15:46:28.911 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.895+0000 7f120d872640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f12080ccab0 0x7f120814d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:28.911 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.895+0000 7f120d872640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f12080ccab0 0x7f120814d570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34664/0 (socket says 172.21.15.16:34664) 2023-05-22T15:46:28.911 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.895+0000 7f120d872640 1 -- 172.21.15.16:0/2693367226 learned_addr learned my addr 172.21.15.16:0/2693367226 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:46:28.912 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.895+0000 7f120e874640 1 --2- 172.21.15.16:0/2693367226 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f12080d0410 0x7f1208156170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:28.912 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.895+0000 7f120e073640 1 --2- 172.21.15.16:0/2693367226 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f12080b7dd0 0x7f1208159880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:28.912 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.895+0000 7f120d872640 1 -- 172.21.15.16:0/2693367226 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f12080b7dd0 msgr2=0x7f1208159880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:28.912 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.895+0000 7f120d872640 1 --2- 172.21.15.16:0/2693367226 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f12080b7dd0 0x7f1208159880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:28.912 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.895+0000 7f120d872640 1 -- 172.21.15.16:0/2693367226 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f12080d0410 msgr2=0x7f1208156170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:28.912 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.895+0000 7f120d872640 1 --2- 172.21.15.16:0/2693367226 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f12080d0410 0x7f1208156170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:28.913 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.895+0000 7f120d872640 1 -- 172.21.15.16:0/2693367226 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1200009a70 con 0x7f12080ccab0 2023-05-22T15:46:28.913 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.895+0000 7f120e073640 1 --2- 172.21.15.16:0/2693367226 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f12080b7dd0 0x7f1208159880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T15:46:28.913 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.899+0000 7f120d872640 1 --2- 172.21.15.16:0/2693367226 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f12080ccab0 0x7f120814d570 secure :-1 s=READY pgs=1851 cs=0 l=1 rev1=1 crypto rx=0x7f120400e960 tx=0x7f120400ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:28.913 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.899+0000 7f11feffd640 1 -- 172.21.15.16:0/2693367226 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1204009690 con 0x7f12080ccab0 2023-05-22T15:46:28.913 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.899+0000 7f11feffd640 1 -- 172.21.15.16:0/2693367226 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f120400ca00 con 0x7f12080ccab0 2023-05-22T15:46:28.914 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.899+0000 7f11feffd640 1 -- 172.21.15.16:0/2693367226 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f120400cd70 con 0x7f12080ccab0 2023-05-22T15:46:28.914 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.899+0000 7f12102d7640 1 -- 172.21.15.16:0/2693367226 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1208160350 con 0x7f12080ccab0 2023-05-22T15:46:28.914 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.899+0000 7f12102d7640 1 -- 172.21.15.16:0/2693367226 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12080cf730 con 0x7f12080ccab0 2023-05-22T15:46:28.914 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.899+0000 7f12102d7640 1 -- 172.21.15.16:0/2693367226 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12081604e0 con 0x7f12080ccab0 2023-05-22T15:46:28.915 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.899+0000 7f11feffd640 1 -- 172.21.15.16:0/2693367226 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 60) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f1204013470 con 0x7f12080ccab0 2023-05-22T15:46:28.915 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.899+0000 7f11feffd640 1 --2- 172.21.15.16:0/2693367226 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f11ec0654a0 0x7f11ec067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:28.915 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.899+0000 7f11feffd640 1 -- 172.21.15.16:0/2693367226 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(147..147 src has 1..147) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f1204083430 con 0x7f12080ccab0 2023-05-22T15:46:28.916 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.899+0000 7f120e073640 1 --2- 172.21.15.16:0/2693367226 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f11ec0654a0 0x7f11ec067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:28.917 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.899+0000 7f120e073640 1 --2- 172.21.15.16:0/2693367226 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f11ec0654a0 0x7f11ec067950 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f11f8007fd0 tx=0x7f11f8007c20 comp rx=0 tx=0).ready entity=mgr.8835 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:28.917 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:28.903+0000 7f11feffd640 1 -- 172.21.15.16:0/2693367226 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f1204050f50 con 0x7f12080ccab0 2023-05-22T15:46:28.917 INFO:tasks.ceph.mds.a.smithi016.stderr:2023-05-22T15:46:28.911+0000 7fbb12d11640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-05-22T15:46:29.063 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:29.055+0000 7f12102d7640 1 -- 172.21.15.16:0/2693367226 --> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f12080cb530 con 0x7f11ec0654a0 2023-05-22T15:46:29.077 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:29.071+0000 7f11feffd640 1 -- 172.21.15.16:0/2693367226 <== mgr.8835 v2:172.21.15.16:6832/36921 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+307001 (secure 0 0 0) 0x7f12080cb530 con 0x7f11ec0654a0 2023-05-22T15:46:29.080 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:46:29.081 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2023-05-22T15:46:29.082 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:29.075+0000 7f12102d7640 1 -- 172.21.15.16:0/2693367226 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f11ec0654a0 msgr2=0x7f11ec067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:29.083 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:29.075+0000 7f12102d7640 1 --2- 172.21.15.16:0/2693367226 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f11ec0654a0 0x7f11ec067950 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f11f8007fd0 tx=0x7f11f8007c20 comp rx=0 tx=0).stop 2023-05-22T15:46:29.083 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:29.075+0000 7f12102d7640 1 -- 172.21.15.16:0/2693367226 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f12080ccab0 msgr2=0x7f120814d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:29.083 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:29.075+0000 7f12102d7640 1 --2- 172.21.15.16:0/2693367226 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f12080ccab0 0x7f120814d570 secure :-1 s=READY pgs=1851 cs=0 l=1 rev1=1 crypto rx=0x7f120400e960 tx=0x7f120400ee20 comp rx=0 tx=0).stop 2023-05-22T15:46:29.083 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:29.075+0000 7f12102d7640 1 -- 172.21.15.16:0/2693367226 shutdown_connections 2023-05-22T15:46:29.083 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:29.075+0000 7f12102d7640 1 --2- 172.21.15.16:0/2693367226 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f12080d0410 0x7f1208156170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:29.083 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:29.075+0000 7f12102d7640 1 --2- 172.21.15.16:0/2693367226 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f12080ccab0 0x7f120814d570 unknown :-1 s=CLOSED pgs=1851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:29.083 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:29.075+0000 7f12102d7640 1 --2- 172.21.15.16:0/2693367226 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f11ec0654a0 0x7f11ec067950 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:29.084 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:29.075+0000 7f12102d7640 1 --2- 172.21.15.16:0/2693367226 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f12080b7dd0 0x7f1208159880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:29.084 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:29.075+0000 7f12102d7640 1 -- 172.21.15.16:0/2693367226 >> 172.21.15.16:0/2693367226 conn(0x7f12080655c0 msgr2=0x7f120806ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:46:29.084 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:29.075+0000 7f12102d7640 1 -- 172.21.15.16:0/2693367226 shutdown_connections 2023-05-22T15:46:29.084 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:29.075+0000 7f12102d7640 1 -- 172.21.15.16:0/2693367226 wait complete. 2023-05-22T15:46:29.099 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":26,"stamp":"2023-05-22T15:46:27.369508+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1268,"num_read_kb":2572,"num_write":399,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":249,"ondisk_log_size":249,"up":210,"acting":210,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":33,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":224768,"kb_used_data":15168,"kb_used_omap":0,"kb_used_meta":209536,"kb_avail":754749952,"statfs":{"total":773094113280,"available":772863950848,"internally_reserved":0,"allocated":15532032,"data_stored":8756104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":214564864},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":62,"apply_latency_ms":62,"commit_latency_ns":62000000,"apply_latency_ns":62000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.321201"},"pg_stats":[{"pgid":"19.3f","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.079122+0000","last_change":"2023-05-22T15:46:23.079122+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.079122+0000","last_undegraded":"2023-05-22T15:46:23.079122+0000","last_fullsized":"2023-05-22T15:46:23.079122+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:57:19.852836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.3e","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.076282+0000","last_change":"2023-05-22T15:46:23.076282+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.076282+0000","last_undegraded":"2023-05-22T15:46:23.076282+0000","last_fullsized":"2023-05-22T15:46:23.076282+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T03:20:39.433281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.3d","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.072443+0000","last_change":"2023-05-22T15:46:23.072443+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.072443+0000","last_undegraded":"2023-05-22T15:46:23.072443+0000","last_fullsized":"2023-05-22T15:46:23.072443+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:46:28.227459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.3c","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.077931+0000","last_change":"2023-05-22T15:46:23.077931+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.077931+0000","last_undegraded":"2023-05-22T15:46:23.077931+0000","last_fullsized":"2023-05-22T15:46:23.077931+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:42:45.119566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.3b","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.090321+0000","last_change":"2023-05-22T15:46:23.090321+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.090321+0000","last_undegraded":"2023-05-22T15:46:23.090321+0000","last_fullsized":"2023-05-22T15:46:23.090321+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:59:24.451915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.3a","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.072441+0000","last_change":"2023-05-22T15:46:23.072441+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.072441+0000","last_undegraded":"2023-05-22T15:46:23.072441+0000","last_fullsized":"2023-05-22T15:46:23.072441+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:25:55.107784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.39","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.076980+0000","last_change":"2023-05-22T15:46:23.076980+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.076980+0000","last_undegraded":"2023-05-22T15:46:23.076980+0000","last_fullsized":"2023-05-22T15:46:23.076980+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:29:20.717884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.38","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.078358+0000","last_change":"2023-05-22T15:46:23.078358+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.078358+0000","last_undegraded":"2023-05-22T15:46:23.078358+0000","last_fullsized":"2023-05-22T15:46:23.078358+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:51:08.858142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.37","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.074307+0000","last_change":"2023-05-22T15:46:23.074307+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.074307+0000","last_undegraded":"2023-05-22T15:46:23.074307+0000","last_fullsized":"2023-05-22T15:46:23.074307+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:28:23.397061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.36","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.075645+0000","last_change":"2023-05-22T15:46:23.075645+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.075645+0000","last_undegraded":"2023-05-22T15:46:23.075645+0000","last_fullsized":"2023-05-22T15:46:23.075645+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:46:28.227459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.35","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.069852+0000","last_change":"2023-05-22T15:46:23.069852+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.069852+0000","last_undegraded":"2023-05-22T15:46:23.069852+0000","last_fullsized":"2023-05-22T15:46:23.069852+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T15:53:20.317234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.34","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.073019+0000","last_change":"2023-05-22T15:46:23.073019+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.073019+0000","last_undegraded":"2023-05-22T15:46:23.073019+0000","last_fullsized":"2023-05-22T15:46:23.073019+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T18:52:29.352539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.33","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.078530+0000","last_change":"2023-05-22T15:46:23.078530+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.078530+0000","last_undegraded":"2023-05-22T15:46:23.078530+0000","last_fullsized":"2023-05-22T15:46:23.078530+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:22:06.668010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.32","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.072337+0000","last_change":"2023-05-22T15:46:23.072337+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.072337+0000","last_undegraded":"2023-05-22T15:46:23.072337+0000","last_fullsized":"2023-05-22T15:46:23.072337+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T18:05:16.988218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.31","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.074359+0000","last_change":"2023-05-22T15:46:23.074359+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.074359+0000","last_undegraded":"2023-05-22T15:46:23.074359+0000","last_fullsized":"2023-05-22T15:46:23.074359+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T18:05:16.988218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.30","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.063646+0000","last_change":"2023-05-22T15:46:23.063646+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.063646+0000","last_undegraded":"2023-05-22T15:46:23.063646+0000","last_fullsized":"2023-05-22T15:46:23.063646+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:30:04.941241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2f","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.075803+0000","last_change":"2023-05-22T15:46:23.075803+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.075803+0000","last_undegraded":"2023-05-22T15:46:23.075803+0000","last_fullsized":"2023-05-22T15:46:23.075803+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:23:42.844820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.2e","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.064448+0000","last_change":"2023-05-22T15:46:23.064448+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.064448+0000","last_undegraded":"2023-05-22T15:46:23.064448+0000","last_fullsized":"2023-05-22T15:46:23.064448+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:23:58.635276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2d","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.068994+0000","last_change":"2023-05-22T15:46:23.068994+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.068994+0000","last_undegraded":"2023-05-22T15:46:23.068994+0000","last_fullsized":"2023-05-22T15:46:23.068994+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:38:41.516579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.2c","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.062047+0000","last_change":"2023-05-22T15:46:23.062047+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.062047+0000","last_undegraded":"2023-05-22T15:46:23.062047+0000","last_fullsized":"2023-05-22T15:46:23.062047+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:52:08.264737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2b","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.093524+0000","last_change":"2023-05-22T15:46:23.093524+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.093524+0000","last_undegraded":"2023-05-22T15:46:23.093524+0000","last_fullsized":"2023-05-22T15:46:23.093524+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:55:41.010860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.2a","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.069379+0000","last_change":"2023-05-22T15:46:23.069379+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.069379+0000","last_undegraded":"2023-05-22T15:46:23.069379+0000","last_fullsized":"2023-05-22T15:46:23.069379+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:36:49.326554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.29","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.081591+0000","last_change":"2023-05-22T15:46:23.081591+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.081591+0000","last_undegraded":"2023-05-22T15:46:23.081591+0000","last_fullsized":"2023-05-22T15:46:23.081591+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:55:51.486171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.28","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.070163+0000","last_change":"2023-05-22T15:46:23.070163+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.070163+0000","last_undegraded":"2023-05-22T15:46:23.070163+0000","last_fullsized":"2023-05-22T15:46:23.070163+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:21:39.187868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.27","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.071496+0000","last_change":"2023-05-22T15:46:23.071496+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.071496+0000","last_undegraded":"2023-05-22T15:46:23.071496+0000","last_fullsized":"2023-05-22T15:46:23.071496+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:37:26.183234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.26","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.063189+0000","last_change":"2023-05-22T15:46:23.063189+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.063189+0000","last_undegraded":"2023-05-22T15:46:23.063189+0000","last_fullsized":"2023-05-22T15:46:23.063189+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:43:11.401376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.25","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.074484+0000","last_change":"2023-05-22T15:46:23.074484+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.074484+0000","last_undegraded":"2023-05-22T15:46:23.074484+0000","last_fullsized":"2023-05-22T15:46:23.074484+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T03:43:40.124163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.24","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.064681+0000","last_change":"2023-05-22T15:46:23.064681+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.064681+0000","last_undegraded":"2023-05-22T15:46:23.064681+0000","last_fullsized":"2023-05-22T15:46:23.064681+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:20:29.148898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.23","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.077284+0000","last_change":"2023-05-22T15:46:23.077284+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.077284+0000","last_undegraded":"2023-05-22T15:46:23.077284+0000","last_fullsized":"2023-05-22T15:46:23.077284+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:34:43.855049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.22","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.073667+0000","last_change":"2023-05-22T15:46:23.073667+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.073667+0000","last_undegraded":"2023-05-22T15:46:23.073667+0000","last_fullsized":"2023-05-22T15:46:23.073667+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T16:10:54.516325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.21","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.086294+0000","last_change":"2023-05-22T15:46:23.086294+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.086294+0000","last_undegraded":"2023-05-22T15:46:23.086294+0000","last_fullsized":"2023-05-22T15:46:23.086294+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:20:12.734869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.20","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.074629+0000","last_change":"2023-05-22T15:46:23.074629+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.074629+0000","last_undegraded":"2023-05-22T15:46:23.074629+0000","last_fullsized":"2023-05-22T15:46:23.074629+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:52:03.083872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.1e","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.061067+0000","last_change":"2023-05-22T15:46:23.061067+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.061067+0000","last_undegraded":"2023-05-22T15:46:23.061067+0000","last_fullsized":"2023-05-22T15:46:23.061067+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:29:20.717884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1f","version":"0'0","reported_seq":9,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.120601+0000","last_change":"2023-05-22T15:46:23.112442+0000","last_active":"2023-05-22T15:46:23.120601+0000","last_peered":"2023-05-22T15:46:23.120601+0000","last_clean":"2023-05-22T15:46:23.120601+0000","last_became_active":"2023-05-22T15:46:23.111561+0000","last_became_peered":"2023-05-22T15:46:23.111561+0000","last_unstale":"2023-05-22T15:46:23.120601+0000","last_undegraded":"2023-05-22T15:46:23.120601+0000","last_fullsized":"2023-05-22T15:46:23.120601+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_clean_scrub_stamp":"2023-05-22T15:46:22.034488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:28:22.384974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.1f","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.071619+0000","last_change":"2023-05-22T15:46:23.071619+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.071619+0000","last_undegraded":"2023-05-22T15:46:23.071619+0000","last_fullsized":"2023-05-22T15:46:23.071619+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:22:11.366110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.1e","version":"0'0","reported_seq":9,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.092179+0000","last_change":"2023-05-22T15:46:23.078407+0000","last_active":"2023-05-22T15:46:23.092179+0000","last_peered":"2023-05-22T15:46:23.092179+0000","last_clean":"2023-05-22T15:46:23.092179+0000","last_became_active":"2023-05-22T15:46:23.077477+0000","last_became_peered":"2023-05-22T15:46:23.077477+0000","last_unstale":"2023-05-22T15:46:23.092179+0000","last_undegraded":"2023-05-22T15:46:23.092179+0000","last_fullsized":"2023-05-22T15:46:23.092179+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_clean_scrub_stamp":"2023-05-22T15:46:22.034488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:20:11.712441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.1c","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.089330+0000","last_change":"2023-05-22T15:46:23.089330+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.089330+0000","last_undegraded":"2023-05-22T15:46:23.089330+0000","last_fullsized":"2023-05-22T15:46:23.089330+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:47:45.260369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1d","version":"0'0","reported_seq":9,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.090669+0000","last_change":"2023-05-22T15:46:23.083700+0000","last_active":"2023-05-22T15:46:23.090669+0000","last_peered":"2023-05-22T15:46:23.090669+0000","last_clean":"2023-05-22T15:46:23.090669+0000","last_became_active":"2023-05-22T15:46:23.083350+0000","last_became_peered":"2023-05-22T15:46:23.083350+0000","last_unstale":"2023-05-22T15:46:23.090669+0000","last_undegraded":"2023-05-22T15:46:23.090669+0000","last_fullsized":"2023-05-22T15:46:23.090669+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_clean_scrub_stamp":"2023-05-22T15:46:22.034488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:03:14.130053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.1d","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.090978+0000","last_change":"2023-05-22T15:46:23.090978+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.090978+0000","last_undegraded":"2023-05-22T15:46:23.090978+0000","last_fullsized":"2023-05-22T15:46:23.090978+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:07:01.166140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1c","version":"0'0","reported_seq":9,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.094786+0000","last_change":"2023-05-22T15:46:23.092200+0000","last_active":"2023-05-22T15:46:23.094786+0000","last_peered":"2023-05-22T15:46:23.094786+0000","last_clean":"2023-05-22T15:46:23.094786+0000","last_became_active":"2023-05-22T15:46:23.089522+0000","last_became_peered":"2023-05-22T15:46:23.089522+0000","last_unstale":"2023-05-22T15:46:23.094786+0000","last_undegraded":"2023-05-22T15:46:23.094786+0000","last_fullsized":"2023-05-22T15:46:23.094786+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_clean_scrub_stamp":"2023-05-22T15:46:22.034488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:16:51.789402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.1a","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.061711+0000","last_change":"2023-05-22T15:46:23.061711+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.061711+0000","last_undegraded":"2023-05-22T15:46:23.061711+0000","last_fullsized":"2023-05-22T15:46:23.061711+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:23:42.844820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1b","version":"0'0","reported_seq":9,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.094709+0000","last_change":"2023-05-22T15:46:23.088945+0000","last_active":"2023-05-22T15:46:23.094709+0000","last_peered":"2023-05-22T15:46:23.094709+0000","last_clean":"2023-05-22T15:46:23.094709+0000","last_became_active":"2023-05-22T15:46:23.088536+0000","last_became_peered":"2023-05-22T15:46:23.088536+0000","last_unstale":"2023-05-22T15:46:23.094709+0000","last_undegraded":"2023-05-22T15:46:23.094709+0000","last_fullsized":"2023-05-22T15:46:23.094709+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_clean_scrub_stamp":"2023-05-22T15:46:22.034488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:12:41.914428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.1b","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.073120+0000","last_change":"2023-05-22T15:46:23.073120+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.073120+0000","last_undegraded":"2023-05-22T15:46:23.073120+0000","last_fullsized":"2023-05-22T15:46:23.073120+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:59:24.451915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.1a","version":"0'0","reported_seq":9,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.090491+0000","last_change":"2023-05-22T15:46:23.089182+0000","last_active":"2023-05-22T15:46:23.090491+0000","last_peered":"2023-05-22T15:46:23.090491+0000","last_clean":"2023-05-22T15:46:23.090491+0000","last_became_active":"2023-05-22T15:46:23.088748+0000","last_became_peered":"2023-05-22T15:46:23.088748+0000","last_unstale":"2023-05-22T15:46:23.090491+0000","last_undegraded":"2023-05-22T15:46:23.090491+0000","last_fullsized":"2023-05-22T15:46:23.090491+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_clean_scrub_stamp":"2023-05-22T15:46:22.034488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:49:33.716090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.18","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.077114+0000","last_change":"2023-05-22T15:46:23.077114+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.077114+0000","last_undegraded":"2023-05-22T15:46:23.077114+0000","last_fullsized":"2023-05-22T15:46:23.077114+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T03:20:39.103248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.19","version":"0'0","reported_seq":9,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.117596+0000","last_change":"2023-05-22T15:46:23.105440+0000","last_active":"2023-05-22T15:46:23.117596+0000","last_peered":"2023-05-22T15:46:23.117596+0000","last_clean":"2023-05-22T15:46:23.117596+0000","last_became_active":"2023-05-22T15:46:23.104687+0000","last_became_peered":"2023-05-22T15:46:23.104687+0000","last_unstale":"2023-05-22T15:46:23.117596+0000","last_undegraded":"2023-05-22T15:46:23.117596+0000","last_fullsized":"2023-05-22T15:46:23.117596+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_clean_scrub_stamp":"2023-05-22T15:46:22.034488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:51:07.846056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.19","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.075186+0000","last_change":"2023-05-22T15:46:23.075186+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.075186+0000","last_undegraded":"2023-05-22T15:46:23.075186+0000","last_fullsized":"2023-05-22T15:46:23.075186+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:22:20.631045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.18","version":"0'0","reported_seq":9,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.108549+0000","last_change":"2023-05-22T15:46:23.106544+0000","last_active":"2023-05-22T15:46:23.108549+0000","last_peered":"2023-05-22T15:46:23.108549+0000","last_clean":"2023-05-22T15:46:23.108549+0000","last_became_active":"2023-05-22T15:46:23.106098+0000","last_became_peered":"2023-05-22T15:46:23.106098+0000","last_unstale":"2023-05-22T15:46:23.108549+0000","last_undegraded":"2023-05-22T15:46:23.108549+0000","last_fullsized":"2023-05-22T15:46:23.108549+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_clean_scrub_stamp":"2023-05-22T15:46:22.034488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:42:03.486863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.e","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.083614+0000","last_change":"2023-05-22T15:46:23.083614+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.083614+0000","last_undegraded":"2023-05-22T15:46:23.083614+0000","last_fullsized":"2023-05-22T15:46:23.083614+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:20:55.836997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.f","version":"0'0","reported_seq":9,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.116428+0000","last_change":"2023-05-22T15:46:23.112873+0000","last_active":"2023-05-22T15:46:23.116428+0000","last_peered":"2023-05-22T15:46:23.116428+0000","last_clean":"2023-05-22T15:46:23.116428+0000","last_became_active":"2023-05-22T15:46:23.112082+0000","last_became_peered":"2023-05-22T15:46:23.112082+0000","last_unstale":"2023-05-22T15:46:23.116428+0000","last_undegraded":"2023-05-22T15:46:23.116428+0000","last_fullsized":"2023-05-22T15:46:23.116428+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_clean_scrub_stamp":"2023-05-22T15:46:22.034488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:24:32.309552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.f","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.063425+0000","last_change":"2023-05-22T15:46:23.063425+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.063425+0000","last_undegraded":"2023-05-22T15:46:23.063425+0000","last_fullsized":"2023-05-22T15:46:23.063425+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T16:10:54.516325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.e","version":"0'0","reported_seq":9,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.109486+0000","last_change":"2023-05-22T15:46:23.107165+0000","last_active":"2023-05-22T15:46:23.109486+0000","last_peered":"2023-05-22T15:46:23.109486+0000","last_clean":"2023-05-22T15:46:23.109486+0000","last_became_active":"2023-05-22T15:46:23.106681+0000","last_became_peered":"2023-05-22T15:46:23.106681+0000","last_unstale":"2023-05-22T15:46:23.109486+0000","last_undegraded":"2023-05-22T15:46:23.109486+0000","last_fullsized":"2023-05-22T15:46:23.109486+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_clean_scrub_stamp":"2023-05-22T15:46:22.034488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T15:59:17.155128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.c","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.060697+0000","last_change":"2023-05-22T15:46:23.060697+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.060697+0000","last_undegraded":"2023-05-22T15:46:23.060697+0000","last_fullsized":"2023-05-22T15:46:23.060697+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:33:34.943631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.d","version":"0'0","reported_seq":9,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.111448+0000","last_change":"2023-05-22T15:46:23.103043+0000","last_active":"2023-05-22T15:46:23.111448+0000","last_peered":"2023-05-22T15:46:23.111448+0000","last_clean":"2023-05-22T15:46:23.111448+0000","last_became_active":"2023-05-22T15:46:23.102220+0000","last_became_peered":"2023-05-22T15:46:23.102220+0000","last_unstale":"2023-05-22T15:46:23.111448+0000","last_undegraded":"2023-05-22T15:46:23.111448+0000","last_fullsized":"2023-05-22T15:46:23.111448+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_clean_scrub_stamp":"2023-05-22T15:46:22.034488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:20:54.824911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.d","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.073232+0000","last_change":"2023-05-22T15:46:23.073232+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.073232+0000","last_undegraded":"2023-05-22T15:46:23.073232+0000","last_fullsized":"2023-05-22T15:46:23.073232+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:49:48.656654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.c","version":"0'0","reported_seq":9,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.082235+0000","last_change":"2023-05-22T15:46:23.078278+0000","last_active":"2023-05-22T15:46:23.082235+0000","last_peered":"2023-05-22T15:46:23.082235+0000","last_clean":"2023-05-22T15:46:23.082235+0000","last_became_active":"2023-05-22T15:46:23.077343+0000","last_became_peered":"2023-05-22T15:46:23.077343+0000","last_unstale":"2023-05-22T15:46:23.082235+0000","last_undegraded":"2023-05-22T15:46:23.082235+0000","last_fullsized":"2023-05-22T15:46:23.082235+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_clean_scrub_stamp":"2023-05-22T15:46:22.034488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:17:43.848098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.a","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.072991+0000","last_change":"2023-05-22T15:46:23.072991+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.072991+0000","last_undegraded":"2023-05-22T15:46:23.072991+0000","last_fullsized":"2023-05-22T15:46:23.072991+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:24:33.321639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.b","version":"0'0","reported_seq":9,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.092758+0000","last_change":"2023-05-22T15:46:23.086857+0000","last_active":"2023-05-22T15:46:23.092758+0000","last_peered":"2023-05-22T15:46:23.092758+0000","last_clean":"2023-05-22T15:46:23.092758+0000","last_became_active":"2023-05-22T15:46:23.086098+0000","last_became_peered":"2023-05-22T15:46:23.086098+0000","last_unstale":"2023-05-22T15:46:23.092758+0000","last_undegraded":"2023-05-22T15:46:23.092758+0000","last_fullsized":"2023-05-22T15:46:23.092758+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_clean_scrub_stamp":"2023-05-22T15:46:22.034488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:45:50.595714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.b","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.071529+0000","last_change":"2023-05-22T15:46:23.071529+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.071529+0000","last_undegraded":"2023-05-22T15:46:23.071529+0000","last_fullsized":"2023-05-22T15:46:23.071529+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:20:12.734869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.a","version":"0'0","reported_seq":9,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.107500+0000","last_change":"2023-05-22T15:46:23.106615+0000","last_active":"2023-05-22T15:46:23.107500+0000","last_peered":"2023-05-22T15:46:23.107500+0000","last_clean":"2023-05-22T15:46:23.107500+0000","last_became_active":"2023-05-22T15:46:23.106155+0000","last_became_peered":"2023-05-22T15:46:23.106155+0000","last_unstale":"2023-05-22T15:46:23.107500+0000","last_undegraded":"2023-05-22T15:46:23.107500+0000","last_fullsized":"2023-05-22T15:46:23.107500+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_clean_scrub_stamp":"2023-05-22T15:46:22.034488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:55:28.593183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.8","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.071955+0000","last_change":"2023-05-22T15:46:23.071955+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.071955+0000","last_undegraded":"2023-05-22T15:46:23.071955+0000","last_fullsized":"2023-05-22T15:46:23.071955+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T16:27:47.997551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.9","version":"0'0","reported_seq":9,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.082561+0000","last_change":"2023-05-22T15:46:23.080736+0000","last_active":"2023-05-22T15:46:23.082561+0000","last_peered":"2023-05-22T15:46:23.082561+0000","last_clean":"2023-05-22T15:46:23.082561+0000","last_became_active":"2023-05-22T15:46:23.080251+0000","last_became_peered":"2023-05-22T15:46:23.080251+0000","last_unstale":"2023-05-22T15:46:23.082561+0000","last_undegraded":"2023-05-22T15:46:23.082561+0000","last_fullsized":"2023-05-22T15:46:23.082561+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_clean_scrub_stamp":"2023-05-22T15:46:22.034488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T16:52:22.575852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.9","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.089066+0000","last_change":"2023-05-22T15:46:23.089066+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.089066+0000","last_undegraded":"2023-05-22T15:46:23.089066+0000","last_fullsized":"2023-05-22T15:46:23.089066+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:31:09.541110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.8","version":"0'0","reported_seq":9,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.109492+0000","last_change":"2023-05-22T15:46:23.107191+0000","last_active":"2023-05-22T15:46:23.109492+0000","last_peered":"2023-05-22T15:46:23.109492+0000","last_clean":"2023-05-22T15:46:23.109492+0000","last_became_active":"2023-05-22T15:46:23.106653+0000","last_became_peered":"2023-05-22T15:46:23.106653+0000","last_unstale":"2023-05-22T15:46:23.109492+0000","last_undegraded":"2023-05-22T15:46:23.109492+0000","last_fullsized":"2023-05-22T15:46:23.109492+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_clean_scrub_stamp":"2023-05-22T15:46:22.034488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:00:00.382995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.6","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.076413+0000","last_change":"2023-05-22T15:46:23.076413+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.076413+0000","last_undegraded":"2023-05-22T15:46:23.076413+0000","last_fullsized":"2023-05-22T15:46:23.076413+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:55:41.010860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.7","version":"0'0","reported_seq":9,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.092881+0000","last_change":"2023-05-22T15:46:23.091817+0000","last_active":"2023-05-22T15:46:23.092881+0000","last_peered":"2023-05-22T15:46:23.092881+0000","last_clean":"2023-05-22T15:46:23.092881+0000","last_became_active":"2023-05-22T15:46:23.091434+0000","last_became_peered":"2023-05-22T15:46:23.091434+0000","last_unstale":"2023-05-22T15:46:23.092881+0000","last_undegraded":"2023-05-22T15:46:23.092881+0000","last_fullsized":"2023-05-22T15:46:23.092881+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_clean_scrub_stamp":"2023-05-22T15:46:22.034488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T16:37:19.990243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.7","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.079702+0000","last_change":"2023-05-22T15:46:23.079702+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.079702+0000","last_undegraded":"2023-05-22T15:46:23.079702+0000","last_fullsized":"2023-05-22T15:46:23.079702+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:20:29.148898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.6","version":"0'0","reported_seq":9,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.120724+0000","last_change":"2023-05-22T15:46:23.114945+0000","last_active":"2023-05-22T15:46:23.120724+0000","last_peered":"2023-05-22T15:46:23.120724+0000","last_clean":"2023-05-22T15:46:23.120724+0000","last_became_active":"2023-05-22T15:46:23.114027+0000","last_became_peered":"2023-05-22T15:46:23.114027+0000","last_unstale":"2023-05-22T15:46:23.120724+0000","last_undegraded":"2023-05-22T15:46:23.120724+0000","last_fullsized":"2023-05-22T15:46:23.120724+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_clean_scrub_stamp":"2023-05-22T15:46:22.034488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T18:52:28.340453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.4","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.075841+0000","last_change":"2023-05-22T15:46:23.075841+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.075841+0000","last_undegraded":"2023-05-22T15:46:23.075841+0000","last_fullsized":"2023-05-22T15:46:23.075841+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:46:38.837010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.5","version":"0'0","reported_seq":9,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.109205+0000","last_change":"2023-05-22T15:46:23.108380+0000","last_active":"2023-05-22T15:46:23.109205+0000","last_peered":"2023-05-22T15:46:23.109205+0000","last_clean":"2023-05-22T15:46:23.109205+0000","last_became_active":"2023-05-22T15:46:23.108014+0000","last_became_peered":"2023-05-22T15:46:23.108014+0000","last_unstale":"2023-05-22T15:46:23.109205+0000","last_undegraded":"2023-05-22T15:46:23.109205+0000","last_fullsized":"2023-05-22T15:46:23.109205+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_clean_scrub_stamp":"2023-05-22T15:46:22.034488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T03:37:19.248335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.5","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.073829+0000","last_change":"2023-05-22T15:46:23.073829+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.073829+0000","last_undegraded":"2023-05-22T15:46:23.073829+0000","last_fullsized":"2023-05-22T15:46:23.073829+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:47:45.260369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.4","version":"0'0","reported_seq":9,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.094601+0000","last_change":"2023-05-22T15:46:23.093730+0000","last_active":"2023-05-22T15:46:23.094601+0000","last_peered":"2023-05-22T15:46:23.094601+0000","last_clean":"2023-05-22T15:46:23.094601+0000","last_became_active":"2023-05-22T15:46:23.093418+0000","last_became_peered":"2023-05-22T15:46:23.093418+0000","last_unstale":"2023-05-22T15:46:23.094601+0000","last_undegraded":"2023-05-22T15:46:23.094601+0000","last_fullsized":"2023-05-22T15:46:23.094601+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_clean_scrub_stamp":"2023-05-22T15:46:22.034488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T16:38:30.323718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.2","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.084350+0000","last_change":"2023-05-22T15:46:23.084350+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.084350+0000","last_undegraded":"2023-05-22T15:46:23.084350+0000","last_fullsized":"2023-05-22T15:46:23.084350+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:25:55.107784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.3","version":"0'0","reported_seq":9,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.111036+0000","last_change":"2023-05-22T15:46:23.110053+0000","last_active":"2023-05-22T15:46:23.111036+0000","last_peered":"2023-05-22T15:46:23.111036+0000","last_clean":"2023-05-22T15:46:23.111036+0000","last_became_active":"2023-05-22T15:46:23.109753+0000","last_became_peered":"2023-05-22T15:46:23.109753+0000","last_unstale":"2023-05-22T15:46:23.111036+0000","last_undegraded":"2023-05-22T15:46:23.111036+0000","last_fullsized":"2023-05-22T15:46:23.111036+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_clean_scrub_stamp":"2023-05-22T15:46:22.034488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T18:29:44.257708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.3","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.062480+0000","last_change":"2023-05-22T15:46:23.062480+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.062480+0000","last_undegraded":"2023-05-22T15:46:23.062480+0000","last_fullsized":"2023-05-22T15:46:23.062480+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T18:05:16.988218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.2","version":"0'0","reported_seq":9,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.120623+0000","last_change":"2023-05-22T15:46:23.112868+0000","last_active":"2023-05-22T15:46:23.120623+0000","last_peered":"2023-05-22T15:46:23.120623+0000","last_clean":"2023-05-22T15:46:23.120623+0000","last_became_active":"2023-05-22T15:46:23.112288+0000","last_became_peered":"2023-05-22T15:46:23.112288+0000","last_unstale":"2023-05-22T15:46:23.120623+0000","last_undegraded":"2023-05-22T15:46:23.120623+0000","last_fullsized":"2023-05-22T15:46:23.120623+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_clean_scrub_stamp":"2023-05-22T15:46:22.034488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:16:19.873549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.0","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.062429+0000","last_change":"2023-05-22T15:46:23.062429+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.062429+0000","last_undegraded":"2023-05-22T15:46:23.062429+0000","last_fullsized":"2023-05-22T15:46:23.062429+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:03:37.104570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1","version":"0'0","reported_seq":9,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.092294+0000","last_change":"2023-05-22T15:46:23.091423+0000","last_active":"2023-05-22T15:46:23.092294+0000","last_peered":"2023-05-22T15:46:23.092294+0000","last_clean":"2023-05-22T15:46:23.092294+0000","last_became_active":"2023-05-22T15:46:23.091088+0000","last_became_peered":"2023-05-22T15:46:23.091088+0000","last_unstale":"2023-05-22T15:46:23.092294+0000","last_undegraded":"2023-05-22T15:46:23.092294+0000","last_fullsized":"2023-05-22T15:46:23.092294+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_clean_scrub_stamp":"2023-05-22T15:46:22.034488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:52:16.638521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.1","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.072920+0000","last_change":"2023-05-22T15:46:23.072920+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.072920+0000","last_undegraded":"2023-05-22T15:46:23.072920+0000","last_fullsized":"2023-05-22T15:46:23.072920+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:16:20.885635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.0","version":"0'0","reported_seq":9,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.107395+0000","last_change":"2023-05-22T15:46:23.105214+0000","last_active":"2023-05-22T15:46:23.107395+0000","last_peered":"2023-05-22T15:46:23.107395+0000","last_clean":"2023-05-22T15:46:23.107395+0000","last_became_active":"2023-05-22T15:46:23.104909+0000","last_became_peered":"2023-05-22T15:46:23.104909+0000","last_unstale":"2023-05-22T15:46:23.107395+0000","last_undegraded":"2023-05-22T15:46:23.107395+0000","last_fullsized":"2023-05-22T15:46:23.107395+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_clean_scrub_stamp":"2023-05-22T15:46:22.034488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:37:03.818368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.12","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.085873+0000","last_change":"2023-05-22T15:46:23.085873+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.085873+0000","last_undegraded":"2023-05-22T15:46:23.085873+0000","last_fullsized":"2023-05-22T15:46:23.085873+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:20:55.836997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.13","version":"0'0","reported_seq":9,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.120574+0000","last_change":"2023-05-22T15:46:23.112061+0000","last_active":"2023-05-22T15:46:23.120574+0000","last_peered":"2023-05-22T15:46:23.120574+0000","last_clean":"2023-05-22T15:46:23.120574+0000","last_became_active":"2023-05-22T15:46:23.111315+0000","last_became_peered":"2023-05-22T15:46:23.111315+0000","last_unstale":"2023-05-22T15:46:23.120574+0000","last_undegraded":"2023-05-22T15:46:23.120574+0000","last_fullsized":"2023-05-22T15:46:23.120574+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_clean_scrub_stamp":"2023-05-22T15:46:22.034488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:37:25.171147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"134'249","reported_seq":753,"reported_epoch":144,"state":"active+clean","last_fresh":"2023-05-22T15:46:22.069225+0000","last_change":"2023-05-22T15:26:39.318204+0000","last_active":"2023-05-22T15:46:22.069225+0000","last_peered":"2023-05-22T15:46:22.069225+0000","last_clean":"2023-05-22T15:46:22.069225+0000","last_became_active":"2023-05-22T15:26:39.315370+0000","last_became_peered":"2023-05-22T15:26:39.315370+0000","last_unstale":"2023-05-22T15:46:22.069225+0000","last_undegraded":"2023-05-22T15:46:22.069225+0000","last_fullsized":"2023-05-22T15:46:22.069225+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:38.020076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:38.020076+0000","last_clean_scrub_stamp":"2023-05-22T15:26:38.020076+0000","objects_scrubbed":0,"log_size":249,"log_dups_size":0,"ondisk_log_size":249,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:06:39.207543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1268,"num_read_kb":2572,"num_write":399,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":139,"reported_epoch":144,"state":"active+clean","last_fresh":"2023-05-22T15:46:22.059466+0000","last_change":"2023-05-22T15:26:44.971951+0000","last_active":"2023-05-22T15:46:22.059466+0000","last_peered":"2023-05-22T15:46:22.059466+0000","last_clean":"2023-05-22T15:46:22.059466+0000","last_became_active":"2023-05-22T15:26:44.970872+0000","last_became_peered":"2023-05-22T15:26:44.970872+0000","last_unstale":"2023-05-22T15:46:22.059466+0000","last_undegraded":"2023-05-22T15:46:22.059466+0000","last_fullsized":"2023-05-22T15:46:22.059466+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:55:34.752949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.11","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.082774+0000","last_change":"2023-05-22T15:46:23.082774+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.082774+0000","last_undegraded":"2023-05-22T15:46:23.082774+0000","last_fullsized":"2023-05-22T15:46:23.082774+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:37:33.613777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.10","version":"0'0","reported_seq":9,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.107838+0000","last_change":"2023-05-22T15:46:23.091282+0000","last_active":"2023-05-22T15:46:23.107838+0000","last_peered":"2023-05-22T15:46:23.107838+0000","last_clean":"2023-05-22T15:46:23.107838+0000","last_became_active":"2023-05-22T15:46:23.086640+0000","last_became_peered":"2023-05-22T15:46:23.086640+0000","last_unstale":"2023-05-22T15:46:23.107838+0000","last_undegraded":"2023-05-22T15:46:23.107838+0000","last_fullsized":"2023-05-22T15:46:23.107838+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_clean_scrub_stamp":"2023-05-22T15:46:22.034488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T16:27:46.985465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":132,"reported_epoch":144,"state":"active+clean","last_fresh":"2023-05-22T15:46:22.068980+0000","last_change":"2023-05-22T15:26:44.977198+0000","last_active":"2023-05-22T15:46:22.068980+0000","last_peered":"2023-05-22T15:46:22.068980+0000","last_clean":"2023-05-22T15:46:22.068980+0000","last_became_active":"2023-05-22T15:26:44.976445+0000","last_became_peered":"2023-05-22T15:26:44.976445+0000","last_unstale":"2023-05-22T15:46:22.068980+0000","last_undegraded":"2023-05-22T15:46:22.068980+0000","last_fullsized":"2023-05-22T15:46:22.068980+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:12:41.464054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.10","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.073353+0000","last_change":"2023-05-22T15:46:23.073353+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.073353+0000","last_undegraded":"2023-05-22T15:46:23.073353+0000","last_fullsized":"2023-05-22T15:46:23.073353+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:42:45.119566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.11","version":"0'0","reported_seq":9,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.103893+0000","last_change":"2023-05-22T15:46:23.102815+0000","last_active":"2023-05-22T15:46:23.103893+0000","last_peered":"2023-05-22T15:46:23.103893+0000","last_clean":"2023-05-22T15:46:23.103893+0000","last_became_active":"2023-05-22T15:46:23.102405+0000","last_became_peered":"2023-05-22T15:46:23.102405+0000","last_unstale":"2023-05-22T15:46:23.103893+0000","last_undegraded":"2023-05-22T15:46:23.103893+0000","last_fullsized":"2023-05-22T15:46:23.103893+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_clean_scrub_stamp":"2023-05-22T15:46:22.034488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:45:50.595714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":132,"reported_epoch":144,"state":"active+clean","last_fresh":"2023-05-22T15:46:22.051914+0000","last_change":"2023-05-22T15:26:44.969469+0000","last_active":"2023-05-22T15:46:22.051914+0000","last_peered":"2023-05-22T15:46:22.051914+0000","last_clean":"2023-05-22T15:46:22.051914+0000","last_became_active":"2023-05-22T15:26:44.969083+0000","last_became_peered":"2023-05-22T15:26:44.969083+0000","last_unstale":"2023-05-22T15:46:22.051914+0000","last_undegraded":"2023-05-22T15:46:22.051914+0000","last_fullsized":"2023-05-22T15:46:22.051914+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:29:23.670067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.13","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.076508+0000","last_change":"2023-05-22T15:46:23.076508+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.076508+0000","last_undegraded":"2023-05-22T15:46:23.076508+0000","last_fullsized":"2023-05-22T15:46:23.076508+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T16:27:47.997551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.12","version":"0'0","reported_seq":9,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.107682+0000","last_change":"2023-05-22T15:46:23.105368+0000","last_active":"2023-05-22T15:46:23.107682+0000","last_peered":"2023-05-22T15:46:23.107682+0000","last_clean":"2023-05-22T15:46:23.107682+0000","last_became_active":"2023-05-22T15:46:23.104771+0000","last_became_peered":"2023-05-22T15:46:23.104771+0000","last_unstale":"2023-05-22T15:46:23.107682+0000","last_undegraded":"2023-05-22T15:46:23.107682+0000","last_fullsized":"2023-05-22T15:46:23.107682+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_clean_scrub_stamp":"2023-05-22T15:46:22.034488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:42:03.486863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":139,"reported_epoch":144,"state":"active+clean","last_fresh":"2023-05-22T15:46:22.059265+0000","last_change":"2023-05-22T15:26:44.976632+0000","last_active":"2023-05-22T15:46:22.059265+0000","last_peered":"2023-05-22T15:46:22.059265+0000","last_clean":"2023-05-22T15:46:22.059265+0000","last_became_active":"2023-05-22T15:26:44.975546+0000","last_became_peered":"2023-05-22T15:26:44.975546+0000","last_unstale":"2023-05-22T15:46:22.059265+0000","last_undegraded":"2023-05-22T15:46:22.059265+0000","last_fullsized":"2023-05-22T15:46:22.059265+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:42:57.973030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.15","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.084525+0000","last_change":"2023-05-22T15:46:23.084525+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.084525+0000","last_undegraded":"2023-05-22T15:46:23.084525+0000","last_fullsized":"2023-05-22T15:46:23.084525+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T03:25:18.349611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.14","version":"0'0","reported_seq":9,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.109240+0000","last_change":"2023-05-22T15:46:23.104870+0000","last_active":"2023-05-22T15:46:23.109240+0000","last_peered":"2023-05-22T15:46:23.109240+0000","last_clean":"2023-05-22T15:46:23.109240+0000","last_became_active":"2023-05-22T15:46:23.104126+0000","last_became_peered":"2023-05-22T15:46:23.104126+0000","last_unstale":"2023-05-22T15:46:23.109240+0000","last_undegraded":"2023-05-22T15:46:23.109240+0000","last_fullsized":"2023-05-22T15:46:23.109240+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_clean_scrub_stamp":"2023-05-22T15:46:22.034488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:20:28.136812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":123,"reported_epoch":144,"state":"active+clean","last_fresh":"2023-05-22T15:46:22.065910+0000","last_change":"2023-05-22T15:26:45.091475+0000","last_active":"2023-05-22T15:46:22.065910+0000","last_peered":"2023-05-22T15:46:22.065910+0000","last_clean":"2023-05-22T15:46:22.065910+0000","last_became_active":"2023-05-22T15:26:45.091143+0000","last_became_peered":"2023-05-22T15:26:45.091143+0000","last_unstale":"2023-05-22T15:46:22.065910+0000","last_undegraded":"2023-05-22T15:46:22.065910+0000","last_fullsized":"2023-05-22T15:46:22.065910+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:12:41.464054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.14","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.088461+0000","last_change":"2023-05-22T15:46:23.088461+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.088461+0000","last_undegraded":"2023-05-22T15:46:23.088461+0000","last_fullsized":"2023-05-22T15:46:23.088461+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:49:34.728176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.15","version":"0'0","reported_seq":9,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.092614+0000","last_change":"2023-05-22T15:46:23.083261+0000","last_active":"2023-05-22T15:46:23.092614+0000","last_peered":"2023-05-22T15:46:23.092614+0000","last_clean":"2023-05-22T15:46:23.092614+0000","last_became_active":"2023-05-22T15:46:23.082317+0000","last_became_peered":"2023-05-22T15:46:23.082317+0000","last_unstale":"2023-05-22T15:46:23.092614+0000","last_undegraded":"2023-05-22T15:46:23.092614+0000","last_fullsized":"2023-05-22T15:46:23.092614+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_clean_scrub_stamp":"2023-05-22T15:46:22.034488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T18:29:44.257708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":132,"reported_epoch":144,"state":"active+clean","last_fresh":"2023-05-22T15:46:22.069477+0000","last_change":"2023-05-22T15:26:45.092367+0000","last_active":"2023-05-22T15:46:22.069477+0000","last_peered":"2023-05-22T15:46:22.069477+0000","last_clean":"2023-05-22T15:46:22.069477+0000","last_became_active":"2023-05-22T15:26:45.091951+0000","last_became_peered":"2023-05-22T15:26:45.091951+0000","last_unstale":"2023-05-22T15:46:22.069477+0000","last_undegraded":"2023-05-22T15:46:22.069477+0000","last_fullsized":"2023-05-22T15:46:22.069477+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:42:57.973030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.17","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.065336+0000","last_change":"2023-05-22T15:46:23.065336+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.065336+0000","last_undegraded":"2023-05-22T15:46:23.065336+0000","last_fullsized":"2023-05-22T15:46:23.065336+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T16:27:47.997551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.16","version":"0'0","reported_seq":9,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.107509+0000","last_change":"2023-05-22T15:46:23.106558+0000","last_active":"2023-05-22T15:46:23.107509+0000","last_peered":"2023-05-22T15:46:23.107509+0000","last_clean":"2023-05-22T15:46:23.107509+0000","last_became_active":"2023-05-22T15:46:23.106098+0000","last_became_peered":"2023-05-22T15:46:23.106098+0000","last_unstale":"2023-05-22T15:46:23.107509+0000","last_undegraded":"2023-05-22T15:46:23.107509+0000","last_fullsized":"2023-05-22T15:46:23.107509+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_clean_scrub_stamp":"2023-05-22T15:46:22.034488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:30:32.850917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":123,"reported_epoch":144,"state":"active+clean","last_fresh":"2023-05-22T15:46:22.065875+0000","last_change":"2023-05-22T15:26:44.978858+0000","last_active":"2023-05-22T15:46:22.065875+0000","last_peered":"2023-05-22T15:46:22.065875+0000","last_clean":"2023-05-22T15:46:22.065875+0000","last_became_active":"2023-05-22T15:26:44.978443+0000","last_became_peered":"2023-05-22T15:26:44.978443+0000","last_unstale":"2023-05-22T15:46:22.065875+0000","last_undegraded":"2023-05-22T15:46:22.065875+0000","last_fullsized":"2023-05-22T15:46:22.065875+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:46:29.290243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.16","version":"0'0","reported_seq":1,"reported_epoch":145,"state":"creating+peering","last_fresh":"2023-05-22T15:46:23.071629+0000","last_change":"2023-05-22T15:46:23.071629+0000","last_active":"2023-05-22T15:46:23.046574+0000","last_peered":"2023-05-22T15:46:23.046574+0000","last_clean":"2023-05-22T15:46:23.046574+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-05-22T15:46:23.071629+0000","last_undegraded":"2023-05-22T15:46:23.071629+0000","last_fullsized":"2023-05-22T15:46:23.071629+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T18:29:45.269794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.17","version":"0'0","reported_seq":9,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.092889+0000","last_change":"2023-05-22T15:46:23.084909+0000","last_active":"2023-05-22T15:46:23.092889+0000","last_peered":"2023-05-22T15:46:23.092889+0000","last_clean":"2023-05-22T15:46:23.092889+0000","last_became_active":"2023-05-22T15:46:23.084184+0000","last_became_peered":"2023-05-22T15:46:23.084184+0000","last_unstale":"2023-05-22T15:46:23.092889+0000","last_undegraded":"2023-05-22T15:46:23.092889+0000","last_fullsized":"2023-05-22T15:46:23.092889+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_clean_scrub_stamp":"2023-05-22T15:46:22.034488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:47:36.764285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":125,"reported_epoch":144,"state":"active+clean","last_fresh":"2023-05-22T15:46:22.063472+0000","last_change":"2023-05-22T15:26:44.982421+0000","last_active":"2023-05-22T15:46:22.063472+0000","last_peered":"2023-05-22T15:46:22.063472+0000","last_clean":"2023-05-22T15:46:22.063472+0000","last_became_active":"2023-05-22T15:26:44.981586+0000","last_became_peered":"2023-05-22T15:26:44.981586+0000","last_unstale":"2023-05-22T15:46:22.063472+0000","last_undegraded":"2023-05-22T15:46:22.063472+0000","last_fullsized":"2023-05-22T15:46:22.063472+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:14:11.156544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":19,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":18,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1268,"num_read_kb":2572,"num_write":399,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":249,"ondisk_log_size":249,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575084,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":19680,"kb_used_data":2328,"kb_used_omap":0,"kb_used_meta":17344,"kb_avail":94352160,"statfs":{"total":96636764160,"available":96616611840,"internally_reserved":0,"allocated":2383872,"data_stored":1535545,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":17760256},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 15:46:11 2023","interfaces":[{"interface":"back","average":{"1min":0.86699999999999999,"5min":0.89200000000000002,"15min":0.876},"min":{"1min":0.53600000000000003,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":1.294,"5min":2.637,"15min":2.637},"last":0.76600000000000001},{"interface":"front","average":{"1min":0.88,"5min":0.88700000000000001,"15min":0.89200000000000002},"min":{"1min":0.60799999999999998,"5min":0.502,"15min":0.35799999999999998},"max":{"1min":1.1519999999999999,"5min":3.0449999999999999,"15min":3.0449999999999999},"last":0.86199999999999999}]},{"osd":1,"last update":"Mon May 22 15:46:11 2023","interfaces":[{"interface":"back","average":{"1min":0.96799999999999997,"5min":0.95399999999999996,"15min":0.97899999999999998},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.1899999999999999,"5min":1.385,"15min":1.389},"last":0.73799999999999999},{"interface":"front","average":{"1min":0.96799999999999997,"5min":1.006,"15min":1.022},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.54900000000000004},"max":{"1min":1.226,"5min":2.7269999999999999,"15min":2.7269999999999999},"last":0.99299999999999999}]},{"osd":2,"last update":"Mon May 22 15:46:11 2023","interfaces":[{"interface":"back","average":{"1min":0.86199999999999999,"5min":0.85099999999999998,"15min":0.871},"min":{"1min":0.55600000000000005,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.109,"5min":1.252,"15min":1.252},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.88100000000000001,"5min":0.93999999999999995,"15min":0.95699999999999996},"min":{"1min":0.504,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.143,"5min":3.141,"15min":3.141},"last":0.68500000000000005}]},{"osd":3,"last update":"Mon May 22 15:46:11 2023","interfaces":[{"interface":"back","average":{"1min":0.98199999999999998,"5min":0.98899999999999999,"15min":0.96299999999999997},"min":{"1min":0.67600000000000005,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":1.3839999999999999,"5min":2.8149999999999999,"15min":2.8149999999999999},"last":1.109},{"interface":"front","average":{"1min":0.94399999999999995,"5min":0.92100000000000004,"15min":0.96199999999999997},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.2709999999999999,"5min":1.389,"15min":1.4710000000000001},"last":0.82899999999999996}]},{"osd":4,"last update":"Mon May 22 15:46:11 2023","interfaces":[{"interface":"back","average":{"1min":0.87,"5min":0.94399999999999995,"15min":0.90400000000000003},"min":{"1min":0.60299999999999998,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.5089999999999999,"5min":3.472,"15min":3.472},"last":0.94699999999999995},{"interface":"front","average":{"1min":0.997,"5min":1.1850000000000001,"15min":1.05},"min":{"1min":0.59699999999999998,"5min":0.56000000000000005,"15min":0.44600000000000001},"max":{"1min":1.2949999999999999,"5min":15.627000000000001,"15min":15.627000000000001},"last":1.163}]},{"osd":5,"last update":"Mon May 22 15:46:11 2023","interfaces":[{"interface":"back","average":{"1min":1.012,"5min":1.071,"15min":1.02},"min":{"1min":0.60899999999999999,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":1.4750000000000001,"5min":4.5599999999999996,"15min":4.5599999999999996},"last":0.90900000000000003},{"interface":"front","average":{"1min":1.115,"5min":1.161,"15min":1.139},"min":{"1min":0.82999999999999996,"5min":0.63800000000000001,"15min":0.58799999999999997},"max":{"1min":1.302,"5min":4.3799999999999999,"15min":4.3799999999999999},"last":1.1919999999999999}]},{"osd":6,"last update":"Mon May 22 15:46:11 2023","interfaces":[{"interface":"back","average":{"1min":1.073,"5min":1.1399999999999999,"15min":1.0840000000000001},"min":{"1min":0.80300000000000005,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.415,"5min":6.5069999999999997,"15min":6.5069999999999997},"last":1.1339999999999999},{"interface":"front","average":{"1min":1.1200000000000001,"5min":1.1659999999999999,"15min":1.1419999999999999},"min":{"1min":0.70299999999999996,"5min":0.62,"15min":0.62},"max":{"1min":1.458,"5min":3.3500000000000001,"15min":3.3500000000000001},"last":1.214}]}]},{"osd":6,"up_from":13,"seq":55834575085,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36572,"kb_used_data":1748,"kb_used_omap":0,"kb_used_meta":34816,"kb_avail":94335268,"statfs":{"total":96636764160,"available":96599314432,"internally_reserved":0,"allocated":1789952,"data_stored":945177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":35651584},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 15:46:20 2023","interfaces":[{"interface":"back","average":{"1min":0.90000000000000002,"5min":0.86699999999999999,"15min":0.89900000000000002},"min":{"1min":0.499,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":1.4039999999999999,"5min":1.486,"15min":1.486},"last":0.81899999999999995},{"interface":"front","average":{"1min":0.89200000000000002,"5min":0.94599999999999995,"15min":0.91100000000000003},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.36299999999999999},"max":{"1min":1.2729999999999999,"5min":1.6240000000000001,"15min":1.6240000000000001},"last":1.054}]},{"osd":1,"last update":"Mon May 22 15:46:20 2023","interfaces":[{"interface":"back","average":{"1min":0.93200000000000005,"5min":0.89100000000000001,"15min":0.91200000000000003},"min":{"1min":0.59799999999999998,"5min":0.504,"15min":0.48299999999999998},"max":{"1min":1.8799999999999999,"5min":1.8799999999999999,"15min":1.8799999999999999},"last":1.03},{"interface":"front","average":{"1min":0.95899999999999996,"5min":0.92400000000000004,"15min":0.93999999999999995},"min":{"1min":0.70299999999999996,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":1.1950000000000001,"5min":1.3320000000000001,"15min":1.554},"last":0.77300000000000002}]},{"osd":2,"last update":"Mon May 22 15:46:20 2023","interfaces":[{"interface":"back","average":{"1min":1.05,"5min":0.97399999999999998,"15min":0.98999999999999999},"min":{"1min":0.55400000000000005,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":3.2429999999999999,"5min":3.2429999999999999,"15min":3.2429999999999999},"last":1.099},{"interface":"front","average":{"1min":1.032,"5min":1.0269999999999999,"15min":1.0429999999999999},"min":{"1min":0.55300000000000005,"5min":0.5,"15min":0.442},"max":{"1min":3.3969999999999998,"5min":3.3969999999999998,"15min":3.3969999999999998},"last":1.1220000000000001}]},{"osd":3,"last update":"Mon May 22 15:46:20 2023","interfaces":[{"interface":"back","average":{"1min":0.88900000000000001,"5min":0.94599999999999995,"15min":0.97899999999999998},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":1.2470000000000001,"5min":1.5780000000000001,"15min":1.679},"last":1.155},{"interface":"front","average":{"1min":0.93500000000000005,"5min":0.95099999999999996,"15min":0.96299999999999997},"min":{"1min":0.69099999999999995,"5min":0.53200000000000003,"15min":0.50800000000000001},"max":{"1min":1.2430000000000001,"5min":1.383,"15min":1.4570000000000001},"last":1.006}]},{"osd":4,"last update":"Mon May 22 15:46:20 2023","interfaces":[{"interface":"back","average":{"1min":0.91300000000000003,"5min":0.93200000000000005,"15min":0.92500000000000004},"min":{"1min":0.61699999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":1.3169999999999999,"5min":1.581,"15min":1.6859999999999999},"last":0.91700000000000004},{"interface":"front","average":{"1min":0.93200000000000005,"5min":1.0209999999999999,"15min":1.0169999999999999},"min":{"1min":0.65500000000000003,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":1.242,"5min":1.452,"15min":1.8300000000000001},"last":0.88500000000000001}]},{"osd":5,"last update":"Mon May 22 15:46:20 2023","interfaces":[{"interface":"back","average":{"1min":0.98799999999999999,"5min":1.0169999999999999,"15min":0.98899999999999999},"min":{"1min":0.67600000000000005,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":1.3200000000000001,"5min":1.4530000000000001,"15min":1.6499999999999999},"last":0.85399999999999998},{"interface":"front","average":{"1min":1.1459999999999999,"5min":1.141,"15min":1.1120000000000001},"min":{"1min":0.81999999999999995,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.9990000000000001,"5min":1.9990000000000001,"15min":1.9990000000000001},"last":0.97799999999999998}]},{"osd":7,"last update":"Mon May 22 15:46:20 2023","interfaces":[{"interface":"back","average":{"1min":1.0860000000000001,"5min":1.131,"15min":1.125},"min":{"1min":0.85499999999999998,"5min":0.37,"15min":0.37},"max":{"1min":1.5529999999999999,"5min":1.853,"15min":1.9179999999999999},"last":1.23},{"interface":"front","average":{"1min":1.131,"5min":1.133,"15min":1.135},"min":{"1min":0.80300000000000005,"5min":0.442,"15min":0.442},"max":{"1min":1.579,"5min":1.579,"15min":1.8520000000000001},"last":0.95199999999999996}]}]},{"osd":5,"up_from":13,"seq":55834575085,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":26012,"kb_used_data":1748,"kb_used_omap":0,"kb_used_meta":24256,"kb_avail":94345828,"statfs":{"total":96636764160,"available":96610127872,"internally_reserved":0,"allocated":1789952,"data_stored":945177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":24838144},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 15:46:10 2023","interfaces":[{"interface":"back","average":{"1min":0.90900000000000003,"5min":0.85599999999999998,"15min":0.88300000000000001},"min":{"1min":0.61399999999999999,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.2889999999999999,"5min":1.3140000000000001,"15min":1.3440000000000001},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.91800000000000004,"5min":0.88600000000000001,"15min":0.88500000000000001},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":1.2709999999999999,"5min":1.3899999999999999,"15min":2.1699999999999999},"last":0.72599999999999998}]},{"osd":1,"last update":"Mon May 22 15:46:10 2023","interfaces":[{"interface":"back","average":{"1min":0.94999999999999996,"5min":0.96099999999999997,"15min":0.995},"min":{"1min":0.64800000000000002,"5min":0.52400000000000002,"15min":0.51000000000000001},"max":{"1min":1.3109999999999999,"5min":1.337,"15min":1.9370000000000001},"last":0.78000000000000003},{"interface":"front","average":{"1min":1.073,"5min":1.008,"15min":1.026},"min":{"1min":0.501,"5min":0.501,"15min":0.501},"max":{"1min":1.3089999999999999,"5min":1.6970000000000001,"15min":1.6970000000000001},"last":0.96899999999999997}]},{"osd":2,"last update":"Mon May 22 15:46:10 2023","interfaces":[{"interface":"back","average":{"1min":0.89100000000000001,"5min":0.91600000000000004,"15min":0.93400000000000005},"min":{"1min":0.505,"5min":0.505,"15min":0.504},"max":{"1min":1.343,"5min":1.5549999999999999,"15min":1.9830000000000001},"last":1.0549999999999999},{"interface":"front","average":{"1min":0.91800000000000004,"5min":0.94999999999999996,"15min":0.95199999999999996},"min":{"1min":0.56100000000000005,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.202,"5min":1.321,"15min":2.1120000000000001},"last":1.0169999999999999}]},{"osd":3,"last update":"Mon May 22 15:46:10 2023","interfaces":[{"interface":"back","average":{"1min":0.92900000000000005,"5min":0.92000000000000004,"15min":0.94899999999999995},"min":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.48199999999999998},"max":{"1min":1.206,"5min":1.2809999999999999,"15min":1.4419999999999999},"last":0.88300000000000001},{"interface":"front","average":{"1min":1.028,"5min":1.008,"15min":1.036},"min":{"1min":0.626,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":1.2490000000000001,"5min":1.371,"15min":2.52},"last":1.161}]},{"osd":4,"last update":"Mon May 22 15:46:10 2023","interfaces":[{"interface":"back","average":{"1min":0.96999999999999997,"5min":0.95699999999999996,"15min":0.95199999999999996},"min":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":1.405,"5min":1.405,"15min":2.226},"last":0.93500000000000005},{"interface":"front","average":{"1min":1.0629999999999999,"5min":1.024,"15min":1.0309999999999999},"min":{"1min":0.76000000000000001,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.3740000000000001,"5min":1.863,"15min":1.863},"last":1.1120000000000001}]},{"osd":6,"last update":"Mon May 22 15:46:10 2023","interfaces":[{"interface":"back","average":{"1min":1.008,"5min":1.024,"15min":1.0349999999999999},"min":{"1min":0.51900000000000002,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.397,"5min":1.4330000000000001,"15min":2.831},"last":1.246},{"interface":"front","average":{"1min":1.0369999999999999,"5min":1.0589999999999999,"15min":1.071},"min":{"1min":0.67000000000000004,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":1.4239999999999999,"5min":1.4239999999999999,"15min":2.3690000000000002},"last":1.296}]},{"osd":7,"last update":"Mon May 22 15:46:10 2023","interfaces":[{"interface":"back","average":{"1min":1.081,"5min":1.0900000000000001,"15min":1.1120000000000001},"min":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.42499999999999999},"max":{"1min":1.4359999999999999,"5min":1.47,"15min":1.849},"last":1.206},{"interface":"front","average":{"1min":1.1599999999999999,"5min":1.149,"15min":1.1659999999999999},"min":{"1min":0.86899999999999999,"5min":0.64000000000000001,"15min":0.64000000000000001},"max":{"1min":1.4339999999999999,"5min":1.575,"15min":2.7429999999999999},"last":1.3360000000000001}]}]},{"osd":4,"up_from":13,"seq":55834575083,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31592,"kb_used_data":1760,"kb_used_omap":0,"kb_used_meta":29824,"kb_avail":94340248,"statfs":{"total":96636764160,"available":96604413952,"internally_reserved":0,"allocated":1802240,"data_stored":952153,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":30539776},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 15:46:19 2023","interfaces":[{"interface":"back","average":{"1min":0.93200000000000005,"5min":0.86799999999999999,"15min":0.90500000000000003},"min":{"1min":0.63400000000000001,"5min":0.48999999999999999,"15min":0.34100000000000003},"max":{"1min":1.2629999999999999,"5min":1.4119999999999999,"15min":1.9910000000000001},"last":0.66600000000000004},{"interface":"front","average":{"1min":0.91300000000000003,"5min":0.90000000000000002,"15min":0.92200000000000004},"min":{"1min":0.51100000000000001,"5min":0.499,"15min":0.373},"max":{"1min":1.284,"5min":1.369,"15min":1.3779999999999999},"last":0.81000000000000005}]},{"osd":1,"last update":"Mon May 22 15:46:19 2023","interfaces":[{"interface":"back","average":{"1min":0.94799999999999995,"5min":0.91800000000000004,"15min":0.96399999999999997},"min":{"1min":0.60499999999999998,"5min":0.436,"15min":0.436},"max":{"1min":1.1970000000000001,"5min":1.3899999999999999,"15min":1.3899999999999999},"last":0.72099999999999997},{"interface":"front","average":{"1min":0.91700000000000004,"5min":0.84499999999999997,"15min":0.90100000000000002},"min":{"1min":0.56699999999999995,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.157,"5min":1.2889999999999999,"15min":1.3640000000000001},"last":0.97799999999999998}]},{"osd":2,"last update":"Mon May 22 15:46:19 2023","interfaces":[{"interface":"back","average":{"1min":0.90100000000000002,"5min":0.91600000000000004,"15min":0.94199999999999995},"min":{"1min":0.56299999999999994,"5min":0.50800000000000001,"15min":0.495},"max":{"1min":1.244,"5min":1.96,"15min":2.1019999999999999},"last":0.76500000000000001},{"interface":"front","average":{"1min":0.97799999999999998,"5min":0.98399999999999999,"15min":0.99199999999999999},"min":{"1min":0.58099999999999996,"5min":0.48799999999999999,"15min":0.48399999999999999},"max":{"1min":1.329,"5min":1.4399999999999999,"15min":1.4399999999999999},"last":0.60299999999999998}]},{"osd":3,"last update":"Mon May 22 15:46:19 2023","interfaces":[{"interface":"back","average":{"1min":0.91100000000000003,"5min":0.90200000000000002,"15min":0.93200000000000005},"min":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":1.224,"5min":1.28,"15min":1.3169999999999999},"last":1.0580000000000001},{"interface":"front","average":{"1min":1.0469999999999999,"5min":1.014,"15min":1.0209999999999999},"min":{"1min":0.72199999999999998,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":1.3420000000000001,"5min":1.4179999999999999,"15min":2.1720000000000002},"last":0.85599999999999998}]},{"osd":5,"last update":"Mon May 22 15:46:19 2023","interfaces":[{"interface":"back","average":{"1min":1.0209999999999999,"5min":0.97199999999999998,"15min":0.97699999999999998},"min":{"1min":0.55000000000000004,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.357,"5min":1.373,"15min":1.472},"last":1.0369999999999999},{"interface":"front","average":{"1min":1.0389999999999999,"5min":0.98999999999999999,"15min":0.97499999999999998},"min":{"1min":0.59999999999999998,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.262,"5min":1.365,"15min":1.4139999999999999},"last":1.155}]},{"osd":6,"last update":"Mon May 22 15:46:19 2023","interfaces":[{"interface":"back","average":{"1min":1.0649999999999999,"5min":0.99299999999999999,"15min":0.98499999999999999},"min":{"1min":0.72099999999999997,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":1.456,"5min":1.456,"15min":2.2349999999999999},"last":0.93500000000000005},{"interface":"front","average":{"1min":1.131,"5min":1.103,"15min":1.1100000000000001},"min":{"1min":0.66500000000000004,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":1.425,"5min":1.486,"15min":1.8220000000000001},"last":0.89800000000000002}]},{"osd":7,"last update":"Mon May 22 15:46:19 2023","interfaces":[{"interface":"back","average":{"1min":1.137,"5min":1.0629999999999999,"15min":1.052},"min":{"1min":0.85699999999999998,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":1.389,"5min":1.46,"15min":1.5800000000000001},"last":1.109},{"interface":"front","average":{"1min":1.1930000000000001,"5min":1.167,"15min":1.143},"min":{"1min":0.78900000000000003,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":1.5620000000000001,"5min":1.6559999999999999,"15min":2.306},"last":1.018}]}]},{"osd":3,"up_from":13,"seq":55834575083,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36956,"kb_used_data":1748,"kb_used_omap":0,"kb_used_meta":35200,"kb_avail":94334884,"statfs":{"total":96636764160,"available":96598921216,"internally_reserved":0,"allocated":1789952,"data_stored":945177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":36044800},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 15:46:03 2023","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.61699999999999999,"15min":0.63600000000000001},"min":{"1min":0.50700000000000001,"5min":0.252,"15min":0.19},"max":{"1min":1.169,"5min":1.169,"15min":1.194},"last":0.64600000000000002},{"interface":"front","average":{"1min":0.746,"5min":0.68500000000000005,"15min":0.70899999999999996},"min":{"1min":0.48899999999999999,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":1.085,"5min":1.296,"15min":1.296},"last":0.83599999999999997}]},{"osd":1,"last update":"Mon May 22 15:46:03 2023","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.64200000000000002,"15min":0.66200000000000003},"min":{"1min":0.52100000000000002,"5min":0.314,"15min":0.29499999999999998},"max":{"1min":0.90600000000000003,"5min":1.2609999999999999,"15min":1.2609999999999999},"last":0.93000000000000005},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.72899999999999998,"15min":0.76800000000000002},"min":{"1min":0.47499999999999998,"5min":0.34399999999999997,"15min":0.313},"max":{"1min":1.081,"5min":1.1359999999999999,"15min":1.222},"last":0.60099999999999998}]},{"osd":2,"last update":"Mon May 22 15:46:03 2023","interfaces":[{"interface":"back","average":{"1min":0.84199999999999997,"5min":0.78700000000000003,"15min":0.81699999999999995},"min":{"1min":0.45600000000000002,"5min":0.40699999999999997,"15min":0.36599999999999999},"max":{"1min":1.0529999999999999,"5min":1.506,"15min":1.506},"last":0.75800000000000001},{"interface":"front","average":{"1min":0.81899999999999995,"5min":0.75600000000000001,"15min":0.80700000000000005},"min":{"1min":0.65300000000000002,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":1.032,"5min":1.147,"15min":1.359},"last":0.89400000000000002}]},{"osd":4,"last update":"Mon May 22 15:46:03 2023","interfaces":[{"interface":"back","average":{"1min":0.97499999999999998,"5min":0.92500000000000004,"15min":0.95599999999999996},"min":{"1min":0.71999999999999997,"5min":0.54500000000000004,"15min":0.48999999999999999},"max":{"1min":1.238,"5min":1.413,"15min":1.4470000000000001},"last":1.0569999999999999},{"interface":"front","average":{"1min":1.0529999999999999,"5min":0.97499999999999998,"15min":1.0029999999999999},"min":{"1min":0.83299999999999996,"5min":0.54400000000000004,"15min":0.49199999999999999},"max":{"1min":1.407,"5min":1.407,"15min":1.4470000000000001},"last":1.0289999999999999}]},{"osd":5,"last update":"Mon May 22 15:46:03 2023","interfaces":[{"interface":"back","average":{"1min":1.0509999999999999,"5min":0.98399999999999999,"15min":1.016},"min":{"1min":0.80900000000000005,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.3149999999999999,"5min":1.5089999999999999,"15min":1.5089999999999999},"last":0.95899999999999996},{"interface":"front","average":{"1min":1.0349999999999999,"5min":1.006,"15min":1.038},"min":{"1min":0.90200000000000002,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.2130000000000001,"5min":1.4299999999999999,"15min":1.4359999999999999},"last":1.347}]},{"osd":6,"last update":"Mon May 22 15:46:03 2023","interfaces":[{"interface":"back","average":{"1min":1.0580000000000001,"5min":1.0129999999999999,"15min":1.042},"min":{"1min":0.873,"5min":0.47699999999999998,"15min":0.46999999999999997},"max":{"1min":1.2989999999999999,"5min":1.4570000000000001,"15min":1.49},"last":1.101},{"interface":"front","average":{"1min":1.083,"5min":1.0269999999999999,"15min":1.0700000000000001},"min":{"1min":0.79800000000000004,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":1.2949999999999999,"5min":1.4330000000000001,"15min":1.5009999999999999},"last":1.403}]},{"osd":7,"last update":"Mon May 22 15:46:03 2023","interfaces":[{"interface":"back","average":{"1min":1.113,"5min":1.0780000000000001,"15min":1.1040000000000001},"min":{"1min":0.83599999999999997,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.3779999999999999,"5min":1.6719999999999999,"15min":1.6719999999999999},"last":1.454},{"interface":"front","average":{"1min":1.117,"5min":1.071,"15min":1.085},"min":{"1min":0.90200000000000002,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":1.361,"5min":1.8700000000000001,"15min":1.8700000000000001},"last":1.0029999999999999}]}]},{"osd":2,"up_from":13,"seq":55834575084,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":18780,"kb_used_data":1748,"kb_used_omap":0,"kb_used_meta":17024,"kb_avail":94353060,"statfs":{"total":96636764160,"available":96617533440,"internally_reserved":0,"allocated":1789952,"data_stored":945177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":17432576},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 15:46:03 2023","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.64400000000000002,"15min":0.65600000000000003},"min":{"1min":0.28699999999999998,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":1.097,"5min":1.1539999999999999,"15min":1.8240000000000001},"last":0.70299999999999996},{"interface":"front","average":{"1min":0.69899999999999995,"5min":0.70499999999999996,"15min":0.746},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":1.079,"5min":1.079,"15min":1.8160000000000001},"last":0.46300000000000002}]},{"osd":1,"last update":"Mon May 22 15:46:03 2023","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.72999999999999998,"15min":0.74399999999999999},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.27000000000000002},"max":{"1min":1.196,"5min":1.196,"15min":1.2},"last":0.92000000000000004},{"interface":"front","average":{"1min":0.754,"5min":0.75900000000000001,"15min":0.80800000000000005},"min":{"1min":0.32400000000000001,"5min":0.317,"15min":0.317},"max":{"1min":1.0309999999999999,"5min":1.246,"15min":2.5089999999999999},"last":0.52500000000000002}]},{"osd":3,"last update":"Mon May 22 15:46:03 2023","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":0.69399999999999995,"15min":0.72599999999999998},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.96199999999999997,"5min":1.077,"15min":2.7629999999999999},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.82699999999999996,"5min":0.84599999999999997,"15min":0.88300000000000001},"min":{"1min":0.39600000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":1.1699999999999999,"5min":1.256,"15min":2.6840000000000002},"last":0.85699999999999998}]},{"osd":4,"last update":"Mon May 22 15:46:03 2023","interfaces":[{"interface":"back","average":{"1min":0.97499999999999998,"5min":0.98499999999999999,"15min":1.018},"min":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":1.274,"5min":1.536,"15min":2.8039999999999998},"last":1.1459999999999999},{"interface":"front","average":{"1min":1.0069999999999999,"5min":1.0009999999999999,"15min":1.028},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.2509999999999999,"5min":1.4299999999999999,"15min":2.0819999999999999},"last":0.81799999999999995}]},{"osd":5,"last update":"Mon May 22 15:46:03 2023","interfaces":[{"interface":"back","average":{"1min":1.0569999999999999,"5min":1.056,"15min":1.052},"min":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.3959999999999999,"5min":1.5049999999999999,"15min":2.3860000000000001},"last":1.175},{"interface":"front","average":{"1min":0.99099999999999999,"5min":0.97399999999999998,"15min":0.998},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.331,"5min":1.331,"15min":3.0299999999999998},"last":0.78100000000000003}]},{"osd":6,"last update":"Mon May 22 15:46:03 2023","interfaces":[{"interface":"back","average":{"1min":1.038,"5min":1.012,"15min":1.036},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":1.3320000000000001,"5min":1.49,"15min":2.883},"last":1.1020000000000001},{"interface":"front","average":{"1min":1.129,"5min":1.081,"15min":1.1040000000000001},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":1.4450000000000001,"5min":1.5660000000000001,"15min":3.1459999999999999},"last":1.0209999999999999}]},{"osd":7,"last update":"Mon May 22 15:46:03 2023","interfaces":[{"interface":"back","average":{"1min":1.1639999999999999,"5min":1.097,"15min":1.1279999999999999},"min":{"1min":0.55000000000000004,"5min":0.499,"15min":0.499},"max":{"1min":1.429,"5min":1.55,"15min":2.5990000000000002},"last":1.052},{"interface":"front","average":{"1min":1.1240000000000001,"5min":1.0640000000000001,"15min":1.1060000000000001},"min":{"1min":0.5,"5min":0.5,"15min":0.5},"max":{"1min":1.4650000000000001,"5min":1.4650000000000001,"15min":3.8799999999999999},"last":0.97699999999999998}]}]},{"osd":1,"up_from":13,"seq":55834575084,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29800,"kb_used_data":1760,"kb_used_omap":0,"kb_used_meta":28032,"kb_avail":94342040,"statfs":{"total":96636764160,"available":96606248960,"internally_reserved":0,"allocated":1802240,"data_stored":952153,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":28704768},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 15:46:17 2023","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.63200000000000001,"15min":0.63400000000000001},"min":{"1min":0.40600000000000003,"5min":0.26300000000000001,"15min":0.22700000000000001},"max":{"1min":1.1200000000000001,"5min":1.2270000000000001,"15min":1.2270000000000001},"last":0.98699999999999999},{"interface":"front","average":{"1min":0.77100000000000002,"5min":0.70699999999999996,"15min":0.76400000000000001},"min":{"1min":0.35999999999999999,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":1.1839999999999999,"5min":1.1839999999999999,"15min":1.2529999999999999},"last":0.58699999999999997}]},{"osd":2,"last update":"Mon May 22 15:46:17 2023","interfaces":[{"interface":"back","average":{"1min":0.76400000000000001,"5min":0.71399999999999997,"15min":0.73799999999999999},"min":{"1min":0.44800000000000001,"5min":0.317,"15min":0.28999999999999998},"max":{"1min":1.0660000000000001,"5min":1.391,"15min":1.391},"last":0.73899999999999999},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.72999999999999998,"15min":0.76300000000000001},"min":{"1min":0.49199999999999999,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.98699999999999999,"5min":1.2270000000000001,"15min":1.276},"last":0.77100000000000002}]},{"osd":3,"last update":"Mon May 22 15:46:17 2023","interfaces":[{"interface":"back","average":{"1min":0.80100000000000005,"5min":0.79900000000000004,"15min":0.82799999999999996},"min":{"1min":0.51200000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.153,"5min":1.657,"15min":1.657},"last":0.93200000000000005},{"interface":"front","average":{"1min":0.81100000000000005,"5min":0.80800000000000005,"15min":0.81499999999999995},"min":{"1min":0.33700000000000002,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":1.095,"5min":2.0569999999999999,"15min":2.0569999999999999},"last":0.876}]},{"osd":4,"last update":"Mon May 22 15:46:17 2023","interfaces":[{"interface":"back","average":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.96499999999999997},"min":{"1min":0.64500000000000002,"5min":0.55200000000000005,"15min":0.34000000000000002},"max":{"1min":1.502,"5min":2.9529999999999998,"15min":2.9529999999999998},"last":1.2450000000000001},{"interface":"front","average":{"1min":0.97499999999999998,"5min":0.999,"15min":1.002},"min":{"1min":0.61299999999999999,"5min":0.504,"15min":0.49299999999999999},"max":{"1min":1.292,"5min":2.7759999999999998,"15min":2.7759999999999998},"last":1.0469999999999999}]},{"osd":5,"last update":"Mon May 22 15:46:17 2023","interfaces":[{"interface":"back","average":{"1min":1.0349999999999999,"5min":1.006,"15min":1.034},"min":{"1min":0.627,"5min":0.58399999999999996,"15min":0.443},"max":{"1min":1.556,"5min":1.9570000000000001,"15min":1.9570000000000001},"last":1.0269999999999999},{"interface":"front","average":{"1min":1.1060000000000001,"5min":1.034,"15min":1.0449999999999999},"min":{"1min":0.59399999999999997,"5min":0.53400000000000003,"15min":0.47299999999999998},"max":{"1min":1.577,"5min":2.1200000000000001,"15min":2.1200000000000001},"last":1.177}]},{"osd":6,"last update":"Mon May 22 15:46:17 2023","interfaces":[{"interface":"back","average":{"1min":1.02,"5min":1.0369999999999999,"15min":1.036},"min":{"1min":0.65900000000000003,"5min":0.54000000000000004,"15min":0.45300000000000001},"max":{"1min":1.3180000000000001,"5min":2.831,"15min":2.831},"last":1.1399999999999999},{"interface":"front","average":{"1min":1.1499999999999999,"5min":1.085,"15min":1.0940000000000001},"min":{"1min":0.55800000000000005,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":1.6299999999999999,"5min":1.7629999999999999,"15min":1.7629999999999999},"last":1.0920000000000001}]},{"osd":7,"last update":"Mon May 22 15:46:17 2023","interfaces":[{"interface":"back","average":{"1min":1.1759999999999999,"5min":1.135,"15min":1.1180000000000001},"min":{"1min":0.61299999999999999,"5min":0.60299999999999998,"15min":0.56799999999999995},"max":{"1min":1.679,"5min":2.4199999999999999,"15min":2.4199999999999999},"last":1.1299999999999999},{"interface":"front","average":{"1min":1.1819999999999999,"5min":1.1259999999999999,"15min":1.1299999999999999},"min":{"1min":0.76300000000000001,"5min":0.47299999999999998,"15min":0.41599999999999998},"max":{"1min":1.619,"5min":3.8799999999999999,"15min":3.8799999999999999},"last":1.337}]}]},{"osd":0,"up_from":13,"seq":55834575085,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":25376,"kb_used_data":2328,"kb_used_omap":0,"kb_used_meta":23040,"kb_avail":94346464,"statfs":{"total":96636764160,"available":96610779136,"internally_reserved":0,"allocated":2383872,"data_stored":1535545,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":23592960},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon May 22 15:46:21 2023","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.65900000000000003,"15min":0.68100000000000005},"min":{"1min":0.307,"5min":0.248,"15min":0.223},"max":{"1min":1.2190000000000001,"5min":1.466,"15min":1.466},"last":0.78400000000000003},{"interface":"front","average":{"1min":0.69899999999999995,"5min":0.70599999999999996,"15min":0.72899999999999998},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.24099999999999999},"max":{"1min":1.2230000000000001,"5min":1.71,"15min":1.71},"last":0.55200000000000005}]},{"osd":2,"last update":"Mon May 22 15:46:21 2023","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":0.70699999999999996,"15min":0.74099999999999999},"min":{"1min":0.39700000000000002,"5min":0.318,"15min":0.26500000000000001},"max":{"1min":0.96799999999999997,"5min":1.165,"15min":1.3220000000000001},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.82799999999999996,"5min":0.75900000000000001,"15min":0.78800000000000003},"min":{"1min":0.375,"5min":0.34999999999999998,"15min":0.32900000000000001},"max":{"1min":1.1699999999999999,"5min":1.3129999999999999,"15min":1.3129999999999999},"last":0.747}]},{"osd":3,"last update":"Mon May 22 15:46:21 2023","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":0.74099999999999999,"15min":0.76000000000000001},"min":{"1min":0.35499999999999998,"5min":0.31,"15min":0.31},"max":{"1min":0.96199999999999997,"5min":1.9299999999999999,"15min":1.9299999999999999},"last":0.88600000000000001},{"interface":"front","average":{"1min":0.83499999999999996,"5min":0.83299999999999996,"15min":0.871},"min":{"1min":0.35599999999999998,"5min":0.35199999999999998,"15min":0.34399999999999997},"max":{"1min":1.1299999999999999,"5min":1.758,"15min":1.758},"last":0.85999999999999999}]},{"osd":4,"last update":"Mon May 22 15:46:21 2023","interfaces":[{"interface":"back","average":{"1min":0.92800000000000005,"5min":0.95099999999999996,"15min":0.95099999999999996},"min":{"1min":0.57599999999999996,"5min":0.57399999999999995,"15min":0.48099999999999998},"max":{"1min":1.149,"5min":2.923,"15min":2.923},"last":1.091},{"interface":"front","average":{"1min":0.98999999999999999,"5min":1.006,"15min":1.026},"min":{"1min":0.624,"5min":0.624,"15min":0.624},"max":{"1min":1.3839999999999999,"5min":2.0129999999999999,"15min":2.0129999999999999},"last":0.93600000000000005}]},{"osd":5,"last update":"Mon May 22 15:46:21 2023","interfaces":[{"interface":"back","average":{"1min":1,"5min":1.004,"15min":1.0409999999999999},"min":{"1min":0.59099999999999997,"5min":0.56000000000000005,"15min":0.46000000000000002},"max":{"1min":1.238,"5min":1.97,"15min":1.97},"last":1.006},{"interface":"front","average":{"1min":1.02,"5min":1.016,"15min":1.008},"min":{"1min":0.67600000000000005,"5min":0.56599999999999995,"15min":0.55300000000000005},"max":{"1min":1.5349999999999999,"5min":5.0519999999999996,"15min":5.0519999999999996},"last":0.98199999999999998}]},{"osd":6,"last update":"Mon May 22 15:46:21 2023","interfaces":[{"interface":"back","average":{"1min":1.044,"5min":1.0589999999999999,"15min":1.089},"min":{"1min":0.58899999999999997,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":1.306,"5min":2.117,"15min":2.117},"last":1.1439999999999999},{"interface":"front","average":{"1min":1.085,"5min":1.1299999999999999,"15min":1.119},"min":{"1min":0.69199999999999995,"5min":0.65500000000000003,"15min":0.499},"max":{"1min":1.3100000000000001,"5min":5.2640000000000002,"15min":5.2640000000000002},"last":1.1910000000000001}]},{"osd":7,"last update":"Mon May 22 15:46:21 2023","interfaces":[{"interface":"back","average":{"1min":1.075,"5min":1.1040000000000001,"15min":1.129},"min":{"1min":0.63800000000000001,"5min":0.61299999999999999,"15min":0.61299999999999999},"max":{"1min":1.3640000000000001,"5min":2.0550000000000002,"15min":2.0550000000000002},"last":1.0680000000000001},{"interface":"front","average":{"1min":1.089,"5min":1.0900000000000001,"15min":1.0960000000000001},"min":{"1min":0.73099999999999998,"5min":0.63300000000000001,"15min":0.60299999999999998},"max":{"1min":1.6240000000000001,"5min":3.9060000000000001,"15min":3.9060000000000001},"last":1.0409999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-05-22T15:46:32.106 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-05-22T15:46:32.239 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:32.235+0000 7f5d7c0c8640 1 Processor -- start 2023-05-22T15:46:32.239 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:32.235+0000 7f5d7c0c8640 1 -- start start 2023-05-22T15:46:32.240 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:32.235+0000 7f5d7c0c8640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f5d740b0250 0x7f5d74159330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:32.240 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:32.235+0000 7f5d7c0c8640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f5d74159870 0x7f5d740c8ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:32.240 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:32.235+0000 7f5d7c0c8640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d7415a0a0 con 0x7f5d741531b0 2023-05-22T15:46:32.240 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:32.235+0000 7f5d7c0c8640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d74153580 con 0x7f5d74159870 2023-05-22T15:46:32.241 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:32.235+0000 7f5d7c0c8640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d7415d5e0 con 0x7f5d740b0250 2023-05-22T15:46:32.241 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:32.235+0000 7f5d7a665640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f5d74159870 0x7f5d740c8ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:32.241 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:32.235+0000 7f5d7a665640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f5d74159870 0x7f5d740c8ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56506/0 (socket says 172.21.15.16:56506) 2023-05-22T15:46:32.241 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:32.235+0000 7f5d7a665640 1 -- 172.21.15.16:0/3301200465 learned_addr learned my addr 172.21.15.16:0/3301200465 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:46:32.242 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:32.235+0000 7f5d79663640 1 --2- 172.21.15.16:0/3301200465 >> v2:172.21.15.62:3300/0 conn(0x7f5d740b0250 0x7f5d74159330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:32.242 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:32.235+0000 7f5d7a665640 1 -- 172.21.15.16:0/3301200465 >> v2:172.21.15.62:3300/0 conn(0x7f5d740b0250 msgr2=0x7f5d74159330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:46:32.242 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:32.235+0000 7f5d7a665640 1 --2- 172.21.15.16:0/3301200465 >> v2:172.21.15.62:3300/0 conn(0x7f5d740b0250 0x7f5d74159330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:32.242 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:32.235+0000 7f5d7a665640 1 -- 172.21.15.16:0/3301200465 >> v1:172.21.15.16:6789/0 conn(0x7f5d741531b0 legacy=0x7f5d740afb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:32.243 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:32.235+0000 7f5d7a665640 1 -- 172.21.15.16:0/3301200465 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d7415c820 con 0x7f5d74159870 2023-05-22T15:46:32.243 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:32.239+0000 7f5d7a665640 1 --2- 172.21.15.16:0/3301200465 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5d74159870 0x7f5d740c8ba0 secure :-1 s=READY pgs=1853 cs=0 l=1 rev1=1 crypto rx=0x7f5d7406d880 tx=0x7f5d6c01c9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c07dc5ac008b21f server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:32.244 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:32.239+0000 7f5d78e62640 1 -- 172.21.15.16:0/3301200465 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d6c0a5860 con 0x7f5d74159870 2023-05-22T15:46:32.244 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:32.239+0000 7f5d78e62640 1 -- 172.21.15.16:0/3301200465 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d6c0a59c0 con 0x7f5d74159870 2023-05-22T15:46:32.244 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:32.239+0000 7f5d78e62640 1 -- 172.21.15.16:0/3301200465 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d6c0a5d50 con 0x7f5d74159870 2023-05-22T15:46:32.244 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:32.239+0000 7f5d7c0c8640 1 -- 172.21.15.16:0/3301200465 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5d74159870 msgr2=0x7f5d740c8ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:32.244 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:32.239+0000 7f5d7c0c8640 1 --2- 172.21.15.16:0/3301200465 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5d74159870 0x7f5d740c8ba0 secure :-1 s=READY pgs=1853 cs=0 l=1 rev1=1 crypto rx=0x7f5d7406d880 tx=0x7f5d6c01c9a0 comp rx=0 tx=0).stop 2023-05-22T15:46:32.245 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:32.239+0000 7f5d7c0c8640 1 -- 172.21.15.16:0/3301200465 shutdown_connections 2023-05-22T15:46:32.245 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:32.239+0000 7f5d7c0c8640 1 --2- 172.21.15.16:0/3301200465 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5d74159870 0x7f5d740c8ba0 unknown :-1 s=CLOSED pgs=1853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:32.245 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:32.239+0000 7f5d7c0c8640 1 --2- 172.21.15.16:0/3301200465 >> v2:172.21.15.62:3300/0 conn(0x7f5d740b0250 0x7f5d74159330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:32.245 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:32.239+0000 7f5d7c0c8640 1 -- 172.21.15.16:0/3301200465 >> 172.21.15.16:0/3301200465 conn(0x7f5d74069720 msgr2=0x7f5d74069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:46:32.245 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:32.239+0000 7f5d7c0c8640 1 -- 172.21.15.16:0/3301200465 shutdown_connections 2023-05-22T15:46:32.245 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:32.239+0000 7f5d7c0c8640 1 -- 172.21.15.16:0/3301200465 wait complete. 2023-05-22T15:46:32.247 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:32.239+0000 7f5d7c0c8640 1 Processor -- start 2023-05-22T15:46:32.247 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:32.239+0000 7f5d7c0c8640 1 -- start start 2023-05-22T15:46:32.247 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:32.239+0000 7f5d7c0c8640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5d740b0250 0x7f5d74142510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:32.247 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:32.239+0000 7f5d7c0c8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5d741531b0 0x7f5d7414dd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:32.248 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:32.239+0000 7f5d7c0c8640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5d74159870 0x7f5d7414e2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:32.248 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:32.239+0000 7f5d7c0c8640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d74142a50 con 0x7f5d741531b0 2023-05-22T15:46:32.248 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:32.239+0000 7f5d7c0c8640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d740c7fa0 con 0x7f5d740b0250 2023-05-22T15:46:32.248 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:32.239+0000 7f5d7c0c8640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d740c7b10 con 0x7f5d74159870 2023-05-22T15:46:32.248 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:32.239+0000 7f5d79663640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5d741531b0 0x7f5d7414dd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:32.248 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:32.239+0000 7f5d79663640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5d741531b0 0x7f5d7414dd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56516/0 (socket says 172.21.15.16:56516) 2023-05-22T15:46:32.248 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:32.239+0000 7f5d79663640 1 -- 172.21.15.16:0/3549014550 learned_addr learned my addr 172.21.15.16:0/3549014550 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:46:32.248 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:32.239+0000 7f5d79e64640 1 --2- 172.21.15.16:0/3549014550 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5d740b0250 0x7f5d74142510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:32.249 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:32.239+0000 7f5d7a665640 1 --2- 172.21.15.16:0/3549014550 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5d74159870 0x7f5d7414e2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:32.249 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:32.239+0000 7f5d79663640 1 -- 172.21.15.16:0/3549014550 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5d74159870 msgr2=0x7f5d7414e2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:32.249 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:32.239+0000 7f5d79663640 1 --2- 172.21.15.16:0/3549014550 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5d74159870 0x7f5d7414e2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:32.249 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:32.239+0000 7f5d79663640 1 -- 172.21.15.16:0/3549014550 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5d740b0250 msgr2=0x7f5d74142510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:32.249 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:32.239+0000 7f5d79663640 1 --2- 172.21.15.16:0/3549014550 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5d740b0250 0x7f5d74142510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:32.249 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:32.239+0000 7f5d79663640 1 -- 172.21.15.16:0/3549014550 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d6c009b30 con 0x7f5d741531b0 2023-05-22T15:46:32.249 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:32.239+0000 7f5d7a665640 1 --2- 172.21.15.16:0/3549014550 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5d74159870 0x7f5d7414e2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:46:32.250 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:32.239+0000 7f5d79663640 1 --2- 172.21.15.16:0/3549014550 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5d741531b0 0x7f5d7414dd80 secure :-1 s=READY pgs=1854 cs=0 l=1 rev1=1 crypto rx=0x7f5d6800d600 tx=0x7f5d6800dac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:32.250 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:32.239+0000 7f5d667fc640 1 -- 172.21.15.16:0/3549014550 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d680125f0 con 0x7f5d741531b0 2023-05-22T15:46:32.250 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:32.239+0000 7f5d667fc640 1 -- 172.21.15.16:0/3549014550 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d68012750 con 0x7f5d741531b0 2023-05-22T15:46:32.250 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:32.239+0000 7f5d7c0c8640 1 -- 172.21.15.16:0/3549014550 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d74159cc0 con 0x7f5d741531b0 2023-05-22T15:46:32.250 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:32.239+0000 7f5d667fc640 1 -- 172.21.15.16:0/3549014550 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d68012b00 con 0x7f5d741531b0 2023-05-22T15:46:32.250 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:32.243+0000 7f5d7c0c8640 1 -- 172.21.15.16:0/3549014550 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d7415d0c0 con 0x7f5d741531b0 2023-05-22T15:46:32.251 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:32.243+0000 7f5d7c0c8640 1 -- 172.21.15.16:0/3549014550 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d74159e50 con 0x7f5d741531b0 2023-05-22T15:46:32.251 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:32.243+0000 7f5d667fc640 1 -- 172.21.15.16:0/3549014550 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 60) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f5d68012c60 con 0x7f5d741531b0 2023-05-22T15:46:32.251 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:32.243+0000 7f5d667fc640 1 --2- 172.21.15.16:0/3549014550 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f5d580654a0 0x7f5d58067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:32.251 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:32.243+0000 7f5d667fc640 1 -- 172.21.15.16:0/3549014550 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(147..147 src has 1..147) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f5d68083050 con 0x7f5d741531b0 2023-05-22T15:46:32.252 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:32.243+0000 7f5d79e64640 1 --2- 172.21.15.16:0/3549014550 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f5d580654a0 0x7f5d58067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:32.252 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:32.243+0000 7f5d79e64640 1 --2- 172.21.15.16:0/3549014550 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f5d580654a0 0x7f5d58067950 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f5d70000f70 tx=0x7f5d70009040 comp rx=0 tx=0).ready entity=mgr.8835 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:32.253 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:32.247+0000 7f5d667fc640 1 -- 172.21.15.16:0/3549014550 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f5d68050a60 con 0x7f5d741531b0 2023-05-22T15:46:32.403 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:32.399+0000 7f5d7c0c8640 1 -- 172.21.15.16:0/3549014550 --> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f5d740ca8c0 con 0x7f5d580654a0 2023-05-22T15:46:32.426 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:32.419+0000 7f5d667fc640 1 -- 172.21.15.16:0/3549014550 <== mgr.8835 v2:172.21.15.16:6832/36921 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+310036 (secure 0 0 0) 0x7f5d740ca8c0 con 0x7f5d580654a0 2023-05-22T15:46:32.429 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:46:32.430 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2023-05-22T15:46:32.432 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:32.423+0000 7f5d7c0c8640 1 -- 172.21.15.16:0/3549014550 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f5d580654a0 msgr2=0x7f5d58067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:32.432 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:32.423+0000 7f5d7c0c8640 1 --2- 172.21.15.16:0/3549014550 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f5d580654a0 0x7f5d58067950 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f5d70000f70 tx=0x7f5d70009040 comp rx=0 tx=0).stop 2023-05-22T15:46:32.432 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:32.423+0000 7f5d7c0c8640 1 -- 172.21.15.16:0/3549014550 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5d741531b0 msgr2=0x7f5d7414dd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:32.432 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:32.423+0000 7f5d7c0c8640 1 --2- 172.21.15.16:0/3549014550 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5d741531b0 0x7f5d7414dd80 secure :-1 s=READY pgs=1854 cs=0 l=1 rev1=1 crypto rx=0x7f5d6800d600 tx=0x7f5d6800dac0 comp rx=0 tx=0).stop 2023-05-22T15:46:32.433 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:32.423+0000 7f5d7c0c8640 1 -- 172.21.15.16:0/3549014550 shutdown_connections 2023-05-22T15:46:32.433 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:32.423+0000 7f5d7c0c8640 1 --2- 172.21.15.16:0/3549014550 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f5d580654a0 0x7f5d58067950 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:32.434 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:32.423+0000 7f5d7c0c8640 1 --2- 172.21.15.16:0/3549014550 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5d74159870 0x7f5d7414e2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:32.434 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:32.423+0000 7f5d7c0c8640 1 --2- 172.21.15.16:0/3549014550 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5d741531b0 0x7f5d7414dd80 unknown :-1 s=CLOSED pgs=1854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:32.434 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:32.423+0000 7f5d7c0c8640 1 --2- 172.21.15.16:0/3549014550 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5d740b0250 0x7f5d74142510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:32.434 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:32.423+0000 7f5d7c0c8640 1 -- 172.21.15.16:0/3549014550 >> 172.21.15.16:0/3549014550 conn(0x7f5d74069720 msgr2=0x7f5d74069af0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:46:32.434 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:32.423+0000 7f5d7c0c8640 1 -- 172.21.15.16:0/3549014550 shutdown_connections 2023-05-22T15:46:32.434 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:32.423+0000 7f5d7c0c8640 1 -- 172.21.15.16:0/3549014550 wait complete. 2023-05-22T15:46:32.448 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":28,"stamp":"2023-05-22T15:46:31.371261+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592698,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":1268,"num_read_kb":2572,"num_write":444,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":288,"ondisk_log_size":288,"up":210,"acting":210,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":210,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":230152,"kb_used_data":15496,"kb_used_omap":0,"kb_used_meta":214592,"kb_avail":754744568,"statfs":{"total":773094113280,"available":772858437632,"internally_reserved":0,"allocated":15867904,"data_stored":8918436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":219742208},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":70,"apply_latency_ms":70,"commit_latency_ns":70000000,"apply_latency_ns":70000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2330,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.003013"},"pg_stats":[{"pgid":"19.3f","version":"0'0","reported_seq":10,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.491321+0000","last_change":"2023-05-22T15:46:23.490128+0000","last_active":"2023-05-22T15:46:23.491321+0000","last_peered":"2023-05-22T15:46:23.491321+0000","last_clean":"2023-05-22T15:46:23.491321+0000","last_became_active":"2023-05-22T15:46:23.489591+0000","last_became_peered":"2023-05-22T15:46:23.489591+0000","last_unstale":"2023-05-22T15:46:23.491321+0000","last_undegraded":"2023-05-22T15:46:23.491321+0000","last_fullsized":"2023-05-22T15:46:23.491321+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:57:19.852836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.3e","version":"0'0","reported_seq":10,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.494054+0000","last_change":"2023-05-22T15:46:23.492988+0000","last_active":"2023-05-22T15:46:23.494054+0000","last_peered":"2023-05-22T15:46:23.494054+0000","last_clean":"2023-05-22T15:46:23.494054+0000","last_became_active":"2023-05-22T15:46:23.492714+0000","last_became_peered":"2023-05-22T15:46:23.492714+0000","last_unstale":"2023-05-22T15:46:23.494054+0000","last_undegraded":"2023-05-22T15:46:23.494054+0000","last_fullsized":"2023-05-22T15:46:23.494054+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T03:20:39.433281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.3d","version":"0'0","reported_seq":10,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.523342+0000","last_change":"2023-05-22T15:46:23.499122+0000","last_active":"2023-05-22T15:46:23.523342+0000","last_peered":"2023-05-22T15:46:23.523342+0000","last_clean":"2023-05-22T15:46:23.523342+0000","last_became_active":"2023-05-22T15:46:23.498373+0000","last_became_peered":"2023-05-22T15:46:23.498373+0000","last_unstale":"2023-05-22T15:46:23.523342+0000","last_undegraded":"2023-05-22T15:46:23.523342+0000","last_fullsized":"2023-05-22T15:46:23.523342+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:46:28.227459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.3c","version":"0'0","reported_seq":10,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.525831+0000","last_change":"2023-05-22T15:46:23.524912+0000","last_active":"2023-05-22T15:46:23.525831+0000","last_peered":"2023-05-22T15:46:23.525831+0000","last_clean":"2023-05-22T15:46:23.525831+0000","last_became_active":"2023-05-22T15:46:23.524590+0000","last_became_peered":"2023-05-22T15:46:23.524590+0000","last_unstale":"2023-05-22T15:46:23.525831+0000","last_undegraded":"2023-05-22T15:46:23.525831+0000","last_fullsized":"2023-05-22T15:46:23.525831+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:42:45.119566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.3b","version":"0'0","reported_seq":10,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.538876+0000","last_change":"2023-05-22T15:46:23.530787+0000","last_active":"2023-05-22T15:46:23.538876+0000","last_peered":"2023-05-22T15:46:23.538876+0000","last_clean":"2023-05-22T15:46:23.538876+0000","last_became_active":"2023-05-22T15:46:23.527311+0000","last_became_peered":"2023-05-22T15:46:23.527311+0000","last_unstale":"2023-05-22T15:46:23.538876+0000","last_undegraded":"2023-05-22T15:46:23.538876+0000","last_fullsized":"2023-05-22T15:46:23.538876+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:59:24.451915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.3a","version":"0'0","reported_seq":10,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.493879+0000","last_change":"2023-05-22T15:46:23.483211+0000","last_active":"2023-05-22T15:46:23.493879+0000","last_peered":"2023-05-22T15:46:23.493879+0000","last_clean":"2023-05-22T15:46:23.493879+0000","last_became_active":"2023-05-22T15:46:23.481670+0000","last_became_peered":"2023-05-22T15:46:23.481670+0000","last_unstale":"2023-05-22T15:46:23.493879+0000","last_undegraded":"2023-05-22T15:46:23.493879+0000","last_fullsized":"2023-05-22T15:46:23.493879+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:25:55.107784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.39","version":"0'0","reported_seq":10,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.525857+0000","last_change":"2023-05-22T15:46:23.519480+0000","last_active":"2023-05-22T15:46:23.525857+0000","last_peered":"2023-05-22T15:46:23.525857+0000","last_clean":"2023-05-22T15:46:23.525857+0000","last_became_active":"2023-05-22T15:46:23.518831+0000","last_became_peered":"2023-05-22T15:46:23.518831+0000","last_unstale":"2023-05-22T15:46:23.525857+0000","last_undegraded":"2023-05-22T15:46:23.525857+0000","last_fullsized":"2023-05-22T15:46:23.525857+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:29:20.717884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.38","version":"0'0","reported_seq":10,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.529388+0000","last_change":"2023-05-22T15:46:23.516370+0000","last_active":"2023-05-22T15:46:23.529388+0000","last_peered":"2023-05-22T15:46:23.529388+0000","last_clean":"2023-05-22T15:46:23.529388+0000","last_became_active":"2023-05-22T15:46:23.509035+0000","last_became_peered":"2023-05-22T15:46:23.509035+0000","last_unstale":"2023-05-22T15:46:23.529388+0000","last_undegraded":"2023-05-22T15:46:23.529388+0000","last_fullsized":"2023-05-22T15:46:23.529388+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:51:08.858142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.37","version":"0'0","reported_seq":10,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.523773+0000","last_change":"2023-05-22T15:46:23.513304+0000","last_active":"2023-05-22T15:46:23.523773+0000","last_peered":"2023-05-22T15:46:23.523773+0000","last_clean":"2023-05-22T15:46:23.523773+0000","last_became_active":"2023-05-22T15:46:23.512697+0000","last_became_peered":"2023-05-22T15:46:23.512697+0000","last_unstale":"2023-05-22T15:46:23.523773+0000","last_undegraded":"2023-05-22T15:46:23.523773+0000","last_fullsized":"2023-05-22T15:46:23.523773+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:28:23.397061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.36","version":"0'0","reported_seq":10,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.494043+0000","last_change":"2023-05-22T15:46:23.492891+0000","last_active":"2023-05-22T15:46:23.494043+0000","last_peered":"2023-05-22T15:46:23.494043+0000","last_clean":"2023-05-22T15:46:23.494043+0000","last_became_active":"2023-05-22T15:46:23.492674+0000","last_became_peered":"2023-05-22T15:46:23.492674+0000","last_unstale":"2023-05-22T15:46:23.494043+0000","last_undegraded":"2023-05-22T15:46:23.494043+0000","last_fullsized":"2023-05-22T15:46:23.494043+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:46:28.227459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.35","version":"0'0","reported_seq":10,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.472006+0000","last_change":"2023-05-22T15:46:23.469481+0000","last_active":"2023-05-22T15:46:23.472006+0000","last_peered":"2023-05-22T15:46:23.472006+0000","last_clean":"2023-05-22T15:46:23.472006+0000","last_became_active":"2023-05-22T15:46:23.469132+0000","last_became_peered":"2023-05-22T15:46:23.469132+0000","last_unstale":"2023-05-22T15:46:23.472006+0000","last_undegraded":"2023-05-22T15:46:23.472006+0000","last_fullsized":"2023-05-22T15:46:23.472006+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T15:53:20.317234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.34","version":"0'0","reported_seq":10,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.529508+0000","last_change":"2023-05-22T15:46:23.517812+0000","last_active":"2023-05-22T15:46:23.529508+0000","last_peered":"2023-05-22T15:46:23.529508+0000","last_clean":"2023-05-22T15:46:23.529508+0000","last_became_active":"2023-05-22T15:46:23.517218+0000","last_became_peered":"2023-05-22T15:46:23.517218+0000","last_unstale":"2023-05-22T15:46:23.529508+0000","last_undegraded":"2023-05-22T15:46:23.529508+0000","last_fullsized":"2023-05-22T15:46:23.529508+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T18:52:29.352539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.33","version":"0'0","reported_seq":10,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.491087+0000","last_change":"2023-05-22T15:46:23.489913+0000","last_active":"2023-05-22T15:46:23.491087+0000","last_peered":"2023-05-22T15:46:23.491087+0000","last_clean":"2023-05-22T15:46:23.491087+0000","last_became_active":"2023-05-22T15:46:23.489602+0000","last_became_peered":"2023-05-22T15:46:23.489602+0000","last_unstale":"2023-05-22T15:46:23.491087+0000","last_undegraded":"2023-05-22T15:46:23.491087+0000","last_fullsized":"2023-05-22T15:46:23.491087+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:22:06.668010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.32","version":"0'0","reported_seq":10,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.470514+0000","last_change":"2023-05-22T15:46:23.467596+0000","last_active":"2023-05-22T15:46:23.470514+0000","last_peered":"2023-05-22T15:46:23.470514+0000","last_clean":"2023-05-22T15:46:23.470514+0000","last_became_active":"2023-05-22T15:46:23.466879+0000","last_became_peered":"2023-05-22T15:46:23.466879+0000","last_unstale":"2023-05-22T15:46:23.470514+0000","last_undegraded":"2023-05-22T15:46:23.470514+0000","last_fullsized":"2023-05-22T15:46:23.470514+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T18:05:16.988218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.31","version":"0'0","reported_seq":10,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.515763+0000","last_change":"2023-05-22T15:46:23.514767+0000","last_active":"2023-05-22T15:46:23.515763+0000","last_peered":"2023-05-22T15:46:23.515763+0000","last_clean":"2023-05-22T15:46:23.515763+0000","last_became_active":"2023-05-22T15:46:23.514383+0000","last_became_peered":"2023-05-22T15:46:23.514383+0000","last_unstale":"2023-05-22T15:46:23.515763+0000","last_undegraded":"2023-05-22T15:46:23.515763+0000","last_fullsized":"2023-05-22T15:46:23.515763+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T18:05:16.988218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.30","version":"0'0","reported_seq":10,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.473166+0000","last_change":"2023-05-22T15:46:23.471453+0000","last_active":"2023-05-22T15:46:23.473166+0000","last_peered":"2023-05-22T15:46:23.473166+0000","last_clean":"2023-05-22T15:46:23.473166+0000","last_became_active":"2023-05-22T15:46:23.468721+0000","last_became_peered":"2023-05-22T15:46:23.468721+0000","last_unstale":"2023-05-22T15:46:23.473166+0000","last_undegraded":"2023-05-22T15:46:23.473166+0000","last_fullsized":"2023-05-22T15:46:23.473166+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:30:04.941241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2f","version":"0'0","reported_seq":10,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.509649+0000","last_change":"2023-05-22T15:46:23.490087+0000","last_active":"2023-05-22T15:46:23.509649+0000","last_peered":"2023-05-22T15:46:23.509649+0000","last_clean":"2023-05-22T15:46:23.509649+0000","last_became_active":"2023-05-22T15:46:23.489117+0000","last_became_peered":"2023-05-22T15:46:23.489117+0000","last_unstale":"2023-05-22T15:46:23.509649+0000","last_undegraded":"2023-05-22T15:46:23.509649+0000","last_fullsized":"2023-05-22T15:46:23.509649+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:23:42.844820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.2e","version":"0'0","reported_seq":10,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.487535+0000","last_change":"2023-05-22T15:46:23.486976+0000","last_active":"2023-05-22T15:46:23.487535+0000","last_peered":"2023-05-22T15:46:23.487535+0000","last_clean":"2023-05-22T15:46:23.487535+0000","last_became_active":"2023-05-22T15:46:23.486651+0000","last_became_peered":"2023-05-22T15:46:23.486651+0000","last_unstale":"2023-05-22T15:46:23.487535+0000","last_undegraded":"2023-05-22T15:46:23.487535+0000","last_fullsized":"2023-05-22T15:46:23.487535+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:23:58.635276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2d","version":"0'0","reported_seq":10,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.486410+0000","last_change":"2023-05-22T15:46:23.485552+0000","last_active":"2023-05-22T15:46:23.486410+0000","last_peered":"2023-05-22T15:46:23.486410+0000","last_clean":"2023-05-22T15:46:23.486410+0000","last_became_active":"2023-05-22T15:46:23.485268+0000","last_became_peered":"2023-05-22T15:46:23.485268+0000","last_unstale":"2023-05-22T15:46:23.486410+0000","last_undegraded":"2023-05-22T15:46:23.486410+0000","last_fullsized":"2023-05-22T15:46:23.486410+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:38:41.516579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.2c","version":"0'0","reported_seq":10,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.472164+0000","last_change":"2023-05-22T15:46:23.471314+0000","last_active":"2023-05-22T15:46:23.472164+0000","last_peered":"2023-05-22T15:46:23.472164+0000","last_clean":"2023-05-22T15:46:23.472164+0000","last_became_active":"2023-05-22T15:46:23.470996+0000","last_became_peered":"2023-05-22T15:46:23.470996+0000","last_unstale":"2023-05-22T15:46:23.472164+0000","last_undegraded":"2023-05-22T15:46:23.472164+0000","last_fullsized":"2023-05-22T15:46:23.472164+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:52:08.264737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2b","version":"0'0","reported_seq":10,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.538778+0000","last_change":"2023-05-22T15:46:23.529997+0000","last_active":"2023-05-22T15:46:23.538778+0000","last_peered":"2023-05-22T15:46:23.538778+0000","last_clean":"2023-05-22T15:46:23.538778+0000","last_became_active":"2023-05-22T15:46:23.526093+0000","last_became_peered":"2023-05-22T15:46:23.526093+0000","last_unstale":"2023-05-22T15:46:23.538778+0000","last_undegraded":"2023-05-22T15:46:23.538778+0000","last_fullsized":"2023-05-22T15:46:23.538778+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:55:41.010860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.2a","version":"0'0","reported_seq":10,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.463716+0000","last_change":"2023-05-22T15:46:23.462825+0000","last_active":"2023-05-22T15:46:23.463716+0000","last_peered":"2023-05-22T15:46:23.463716+0000","last_clean":"2023-05-22T15:46:23.463716+0000","last_became_active":"2023-05-22T15:46:23.462543+0000","last_became_peered":"2023-05-22T15:46:23.462543+0000","last_unstale":"2023-05-22T15:46:23.463716+0000","last_undegraded":"2023-05-22T15:46:23.463716+0000","last_fullsized":"2023-05-22T15:46:23.463716+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:36:49.326554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.29","version":"0'0","reported_seq":10,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.523833+0000","last_change":"2023-05-22T15:46:23.509313+0000","last_active":"2023-05-22T15:46:23.523833+0000","last_peered":"2023-05-22T15:46:23.523833+0000","last_clean":"2023-05-22T15:46:23.523833+0000","last_became_active":"2023-05-22T15:46:23.508609+0000","last_became_peered":"2023-05-22T15:46:23.508609+0000","last_unstale":"2023-05-22T15:46:23.523833+0000","last_undegraded":"2023-05-22T15:46:23.523833+0000","last_fullsized":"2023-05-22T15:46:23.523833+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:55:51.486171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.28","version":"0'0","reported_seq":10,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.476190+0000","last_change":"2023-05-22T15:46:23.474889+0000","last_active":"2023-05-22T15:46:23.476190+0000","last_peered":"2023-05-22T15:46:23.476190+0000","last_clean":"2023-05-22T15:46:23.476190+0000","last_became_active":"2023-05-22T15:46:23.474644+0000","last_became_peered":"2023-05-22T15:46:23.474644+0000","last_unstale":"2023-05-22T15:46:23.476190+0000","last_undegraded":"2023-05-22T15:46:23.476190+0000","last_fullsized":"2023-05-22T15:46:23.476190+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:21:39.187868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.27","version":"0'0","reported_seq":10,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.531079+0000","last_change":"2023-05-22T15:46:23.529844+0000","last_active":"2023-05-22T15:46:23.531079+0000","last_peered":"2023-05-22T15:46:23.531079+0000","last_clean":"2023-05-22T15:46:23.531079+0000","last_became_active":"2023-05-22T15:46:23.529604+0000","last_became_peered":"2023-05-22T15:46:23.529604+0000","last_unstale":"2023-05-22T15:46:23.531079+0000","last_undegraded":"2023-05-22T15:46:23.531079+0000","last_fullsized":"2023-05-22T15:46:23.531079+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:37:26.183234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.26","version":"0'0","reported_seq":10,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.468366+0000","last_change":"2023-05-22T15:46:23.466715+0000","last_active":"2023-05-22T15:46:23.468366+0000","last_peered":"2023-05-22T15:46:23.468366+0000","last_clean":"2023-05-22T15:46:23.468366+0000","last_became_active":"2023-05-22T15:46:23.466399+0000","last_became_peered":"2023-05-22T15:46:23.466399+0000","last_unstale":"2023-05-22T15:46:23.468366+0000","last_undegraded":"2023-05-22T15:46:23.468366+0000","last_fullsized":"2023-05-22T15:46:23.468366+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:43:11.401376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.25","version":"0'0","reported_seq":10,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.491059+0000","last_change":"2023-05-22T15:46:23.489793+0000","last_active":"2023-05-22T15:46:23.491059+0000","last_peered":"2023-05-22T15:46:23.491059+0000","last_clean":"2023-05-22T15:46:23.491059+0000","last_became_active":"2023-05-22T15:46:23.489493+0000","last_became_peered":"2023-05-22T15:46:23.489493+0000","last_unstale":"2023-05-22T15:46:23.491059+0000","last_undegraded":"2023-05-22T15:46:23.491059+0000","last_fullsized":"2023-05-22T15:46:23.491059+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T03:43:40.124163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.24","version":"0'0","reported_seq":10,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.515965+0000","last_change":"2023-05-22T15:46:23.498961+0000","last_active":"2023-05-22T15:46:23.515965+0000","last_peered":"2023-05-22T15:46:23.515965+0000","last_clean":"2023-05-22T15:46:23.515965+0000","last_became_active":"2023-05-22T15:46:23.493137+0000","last_became_peered":"2023-05-22T15:46:23.493137+0000","last_unstale":"2023-05-22T15:46:23.515965+0000","last_undegraded":"2023-05-22T15:46:23.515965+0000","last_fullsized":"2023-05-22T15:46:23.515965+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:20:29.148898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.23","version":"0'0","reported_seq":10,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.497608+0000","last_change":"2023-05-22T15:46:23.496462+0000","last_active":"2023-05-22T15:46:23.497608+0000","last_peered":"2023-05-22T15:46:23.497608+0000","last_clean":"2023-05-22T15:46:23.497608+0000","last_became_active":"2023-05-22T15:46:23.496128+0000","last_became_peered":"2023-05-22T15:46:23.496128+0000","last_unstale":"2023-05-22T15:46:23.497608+0000","last_undegraded":"2023-05-22T15:46:23.497608+0000","last_fullsized":"2023-05-22T15:46:23.497608+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:34:43.855049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.22","version":"0'0","reported_seq":10,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.516124+0000","last_change":"2023-05-22T15:46:23.515000+0000","last_active":"2023-05-22T15:46:23.516124+0000","last_peered":"2023-05-22T15:46:23.516124+0000","last_clean":"2023-05-22T15:46:23.516124+0000","last_became_active":"2023-05-22T15:46:23.514395+0000","last_became_peered":"2023-05-22T15:46:23.514395+0000","last_unstale":"2023-05-22T15:46:23.516124+0000","last_undegraded":"2023-05-22T15:46:23.516124+0000","last_fullsized":"2023-05-22T15:46:23.516124+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T16:10:54.516325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.21","version":"0'0","reported_seq":10,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.539728+0000","last_change":"2023-05-22T15:46:23.534070+0000","last_active":"2023-05-22T15:46:23.539728+0000","last_peered":"2023-05-22T15:46:23.539728+0000","last_clean":"2023-05-22T15:46:23.539728+0000","last_became_active":"2023-05-22T15:46:23.526647+0000","last_became_peered":"2023-05-22T15:46:23.526647+0000","last_unstale":"2023-05-22T15:46:23.539728+0000","last_undegraded":"2023-05-22T15:46:23.539728+0000","last_fullsized":"2023-05-22T15:46:23.539728+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:20:12.734869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.20","version":"0'0","reported_seq":10,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.506116+0000","last_change":"2023-05-22T15:46:23.488133+0000","last_active":"2023-05-22T15:46:23.506116+0000","last_peered":"2023-05-22T15:46:23.506116+0000","last_clean":"2023-05-22T15:46:23.506116+0000","last_became_active":"2023-05-22T15:46:23.487236+0000","last_became_peered":"2023-05-22T15:46:23.487236+0000","last_unstale":"2023-05-22T15:46:23.506116+0000","last_undegraded":"2023-05-22T15:46:23.506116+0000","last_fullsized":"2023-05-22T15:46:23.506116+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:52:03.083872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.1e","version":"0'0","reported_seq":10,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.490877+0000","last_change":"2023-05-22T15:46:23.490183+0000","last_active":"2023-05-22T15:46:23.490877+0000","last_peered":"2023-05-22T15:46:23.490877+0000","last_clean":"2023-05-22T15:46:23.490877+0000","last_became_active":"2023-05-22T15:46:23.489920+0000","last_became_peered":"2023-05-22T15:46:23.489920+0000","last_unstale":"2023-05-22T15:46:23.490877+0000","last_undegraded":"2023-05-22T15:46:23.490877+0000","last_fullsized":"2023-05-22T15:46:23.490877+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:29:20.717884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1f","version":"146'4","reported_seq":15,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.534429+0000","last_change":"2023-05-22T15:46:23.112442+0000","last_active":"2023-05-22T15:46:23.534429+0000","last_peered":"2023-05-22T15:46:23.534429+0000","last_clean":"2023-05-22T15:46:23.534429+0000","last_became_active":"2023-05-22T15:46:23.111561+0000","last_became_peered":"2023-05-22T15:46:23.111561+0000","last_unstale":"2023-05-22T15:46:23.534429+0000","last_undegraded":"2023-05-22T15:46:23.534429+0000","last_fullsized":"2023-05-22T15:46:23.534429+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_clean_scrub_stamp":"2023-05-22T15:46:22.034488+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:28:22.384974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":566,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.1f","version":"0'0","reported_seq":10,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.523198+0000","last_change":"2023-05-22T15:46:23.497784+0000","last_active":"2023-05-22T15:46:23.523198+0000","last_peered":"2023-05-22T15:46:23.523198+0000","last_clean":"2023-05-22T15:46:23.523198+0000","last_became_active":"2023-05-22T15:46:23.496931+0000","last_became_peered":"2023-05-22T15:46:23.496931+0000","last_unstale":"2023-05-22T15:46:23.523198+0000","last_undegraded":"2023-05-22T15:46:23.523198+0000","last_fullsized":"2023-05-22T15:46:23.523198+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:22:11.366110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.1e","version":"146'1","reported_seq":12,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.515639+0000","last_change":"2023-05-22T15:46:23.078407+0000","last_active":"2023-05-22T15:46:23.515639+0000","last_peered":"2023-05-22T15:46:23.515639+0000","last_clean":"2023-05-22T15:46:23.515639+0000","last_became_active":"2023-05-22T15:46:23.077477+0000","last_became_peered":"2023-05-22T15:46:23.077477+0000","last_unstale":"2023-05-22T15:46:23.515639+0000","last_undegraded":"2023-05-22T15:46:23.515639+0000","last_fullsized":"2023-05-22T15:46:23.515639+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_clean_scrub_stamp":"2023-05-22T15:46:22.034488+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:20:11.712441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.1c","version":"0'0","reported_seq":10,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.538712+0000","last_change":"2023-05-22T15:46:23.527061+0000","last_active":"2023-05-22T15:46:23.538712+0000","last_peered":"2023-05-22T15:46:23.538712+0000","last_clean":"2023-05-22T15:46:23.538712+0000","last_became_active":"2023-05-22T15:46:23.526302+0000","last_became_peered":"2023-05-22T15:46:23.526302+0000","last_unstale":"2023-05-22T15:46:23.538712+0000","last_undegraded":"2023-05-22T15:46:23.538712+0000","last_fullsized":"2023-05-22T15:46:23.538712+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:47:45.260369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1d","version":"146'2","reported_seq":13,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.483624+0000","last_change":"2023-05-22T15:46:23.083700+0000","last_active":"2023-05-22T15:46:23.483624+0000","last_peered":"2023-05-22T15:46:23.483624+0000","last_clean":"2023-05-22T15:46:23.483624+0000","last_became_active":"2023-05-22T15:46:23.083350+0000","last_became_peered":"2023-05-22T15:46:23.083350+0000","last_unstale":"2023-05-22T15:46:23.483624+0000","last_undegraded":"2023-05-22T15:46:23.483624+0000","last_fullsized":"2023-05-22T15:46:23.483624+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_clean_scrub_stamp":"2023-05-22T15:46:22.034488+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:03:14.130053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.1d","version":"0'0","reported_seq":10,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.539042+0000","last_change":"2023-05-22T15:46:23.532562+0000","last_active":"2023-05-22T15:46:23.539042+0000","last_peered":"2023-05-22T15:46:23.539042+0000","last_clean":"2023-05-22T15:46:23.539042+0000","last_became_active":"2023-05-22T15:46:23.532017+0000","last_became_peered":"2023-05-22T15:46:23.532017+0000","last_unstale":"2023-05-22T15:46:23.539042+0000","last_undegraded":"2023-05-22T15:46:23.539042+0000","last_fullsized":"2023-05-22T15:46:23.539042+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:07:01.166140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1c","version":"146'2","reported_seq":13,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.482763+0000","last_change":"2023-05-22T15:46:23.092200+0000","last_active":"2023-05-22T15:46:23.482763+0000","last_peered":"2023-05-22T15:46:23.482763+0000","last_clean":"2023-05-22T15:46:23.482763+0000","last_became_active":"2023-05-22T15:46:23.089522+0000","last_became_peered":"2023-05-22T15:46:23.089522+0000","last_unstale":"2023-05-22T15:46:23.482763+0000","last_undegraded":"2023-05-22T15:46:23.482763+0000","last_fullsized":"2023-05-22T15:46:23.482763+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_clean_scrub_stamp":"2023-05-22T15:46:22.034488+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:16:51.789402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.1a","version":"0'0","reported_seq":10,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.483360+0000","last_change":"2023-05-22T15:46:23.479396+0000","last_active":"2023-05-22T15:46:23.483360+0000","last_peered":"2023-05-22T15:46:23.483360+0000","last_clean":"2023-05-22T15:46:23.483360+0000","last_became_active":"2023-05-22T15:46:23.479116+0000","last_became_peered":"2023-05-22T15:46:23.479116+0000","last_unstale":"2023-05-22T15:46:23.483360+0000","last_undegraded":"2023-05-22T15:46:23.483360+0000","last_fullsized":"2023-05-22T15:46:23.483360+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:23:42.844820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1b","version":"146'1","reported_seq":12,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.483548+0000","last_change":"2023-05-22T15:46:23.088945+0000","last_active":"2023-05-22T15:46:23.483548+0000","last_peered":"2023-05-22T15:46:23.483548+0000","last_clean":"2023-05-22T15:46:23.483548+0000","last_became_active":"2023-05-22T15:46:23.088536+0000","last_became_peered":"2023-05-22T15:46:23.088536+0000","last_unstale":"2023-05-22T15:46:23.483548+0000","last_undegraded":"2023-05-22T15:46:23.483548+0000","last_fullsized":"2023-05-22T15:46:23.483548+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_clean_scrub_stamp":"2023-05-22T15:46:22.034488+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:12:41.914428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.1b","version":"0'0","reported_seq":10,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.493853+0000","last_change":"2023-05-22T15:46:23.478077+0000","last_active":"2023-05-22T15:46:23.493853+0000","last_peered":"2023-05-22T15:46:23.493853+0000","last_clean":"2023-05-22T15:46:23.493853+0000","last_became_active":"2023-05-22T15:46:23.477016+0000","last_became_peered":"2023-05-22T15:46:23.477016+0000","last_unstale":"2023-05-22T15:46:23.493853+0000","last_undegraded":"2023-05-22T15:46:23.493853+0000","last_fullsized":"2023-05-22T15:46:23.493853+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:59:24.451915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.1a","version":"146'1","reported_seq":12,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.483549+0000","last_change":"2023-05-22T15:46:23.089182+0000","last_active":"2023-05-22T15:46:23.483549+0000","last_peered":"2023-05-22T15:46:23.483549+0000","last_clean":"2023-05-22T15:46:23.483549+0000","last_became_active":"2023-05-22T15:46:23.088748+0000","last_became_peered":"2023-05-22T15:46:23.088748+0000","last_unstale":"2023-05-22T15:46:23.483549+0000","last_undegraded":"2023-05-22T15:46:23.483549+0000","last_fullsized":"2023-05-22T15:46:23.483549+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_clean_scrub_stamp":"2023-05-22T15:46:22.034488+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:49:33.716090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.18","version":"0'0","reported_seq":10,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.529623+0000","last_change":"2023-05-22T15:46:23.518361+0000","last_active":"2023-05-22T15:46:23.529623+0000","last_peered":"2023-05-22T15:46:23.529623+0000","last_clean":"2023-05-22T15:46:23.529623+0000","last_became_active":"2023-05-22T15:46:23.508242+0000","last_became_peered":"2023-05-22T15:46:23.508242+0000","last_unstale":"2023-05-22T15:46:23.529623+0000","last_undegraded":"2023-05-22T15:46:23.529623+0000","last_fullsized":"2023-05-22T15:46:23.529623+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T03:20:39.103248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.19","version":"0'0","reported_seq":11,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.466067+0000","last_change":"2023-05-22T15:46:23.105440+0000","last_active":"2023-05-22T15:46:23.466067+0000","last_peered":"2023-05-22T15:46:23.466067+0000","last_clean":"2023-05-22T15:46:23.466067+0000","last_became_active":"2023-05-22T15:46:23.104687+0000","last_became_peered":"2023-05-22T15:46:23.104687+0000","last_unstale":"2023-05-22T15:46:23.466067+0000","last_undegraded":"2023-05-22T15:46:23.466067+0000","last_fullsized":"2023-05-22T15:46:23.466067+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_clean_scrub_stamp":"2023-05-22T15:46:22.034488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:51:07.846056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.19","version":"0'0","reported_seq":10,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.472300+0000","last_change":"2023-05-22T15:46:23.471057+0000","last_active":"2023-05-22T15:46:23.472300+0000","last_peered":"2023-05-22T15:46:23.472300+0000","last_clean":"2023-05-22T15:46:23.472300+0000","last_became_active":"2023-05-22T15:46:23.470799+0000","last_became_peered":"2023-05-22T15:46:23.470799+0000","last_unstale":"2023-05-22T15:46:23.472300+0000","last_undegraded":"2023-05-22T15:46:23.472300+0000","last_fullsized":"2023-05-22T15:46:23.472300+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:22:20.631045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.18","version":"0'0","reported_seq":11,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.462056+0000","last_change":"2023-05-22T15:46:23.106544+0000","last_active":"2023-05-22T15:46:23.462056+0000","last_peered":"2023-05-22T15:46:23.462056+0000","last_clean":"2023-05-22T15:46:23.462056+0000","last_became_active":"2023-05-22T15:46:23.106098+0000","last_became_peered":"2023-05-22T15:46:23.106098+0000","last_unstale":"2023-05-22T15:46:23.462056+0000","last_undegraded":"2023-05-22T15:46:23.462056+0000","last_fullsized":"2023-05-22T15:46:23.462056+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_clean_scrub_stamp":"2023-05-22T15:46:22.034488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:42:03.486863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.e","version":"0'0","reported_seq":10,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.523377+0000","last_change":"2023-05-22T15:46:23.498332+0000","last_active":"2023-05-22T15:46:23.523377+0000","last_peered":"2023-05-22T15:46:23.523377+0000","last_clean":"2023-05-22T15:46:23.523377+0000","last_became_active":"2023-05-22T15:46:23.497552+0000","last_became_peered":"2023-05-22T15:46:23.497552+0000","last_unstale":"2023-05-22T15:46:23.523377+0000","last_undegraded":"2023-05-22T15:46:23.523377+0000","last_fullsized":"2023-05-22T15:46:23.523377+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:20:55.836997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.f","version":"146'1","reported_seq":12,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.530902+0000","last_change":"2023-05-22T15:46:23.112873+0000","last_active":"2023-05-22T15:46:23.530902+0000","last_peered":"2023-05-22T15:46:23.530902+0000","last_clean":"2023-05-22T15:46:23.530902+0000","last_became_active":"2023-05-22T15:46:23.112082+0000","last_became_peered":"2023-05-22T15:46:23.112082+0000","last_unstale":"2023-05-22T15:46:23.530902+0000","last_undegraded":"2023-05-22T15:46:23.530902+0000","last_fullsized":"2023-05-22T15:46:23.530902+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_clean_scrub_stamp":"2023-05-22T15:46:22.034488+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:24:32.309552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.f","version":"0'0","reported_seq":10,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.511843+0000","last_change":"2023-05-22T15:46:23.480437+0000","last_active":"2023-05-22T15:46:23.511843+0000","last_peered":"2023-05-22T15:46:23.511843+0000","last_clean":"2023-05-22T15:46:23.511843+0000","last_became_active":"2023-05-22T15:46:23.479562+0000","last_became_peered":"2023-05-22T15:46:23.479562+0000","last_unstale":"2023-05-22T15:46:23.511843+0000","last_undegraded":"2023-05-22T15:46:23.511843+0000","last_fullsized":"2023-05-22T15:46:23.511843+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T16:10:54.516325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.e","version":"146'2","reported_seq":13,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.514743+0000","last_change":"2023-05-22T15:46:23.107165+0000","last_active":"2023-05-22T15:46:23.514743+0000","last_peered":"2023-05-22T15:46:23.514743+0000","last_clean":"2023-05-22T15:46:23.514743+0000","last_became_active":"2023-05-22T15:46:23.106681+0000","last_became_peered":"2023-05-22T15:46:23.106681+0000","last_unstale":"2023-05-22T15:46:23.514743+0000","last_undegraded":"2023-05-22T15:46:23.514743+0000","last_fullsized":"2023-05-22T15:46:23.514743+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_clean_scrub_stamp":"2023-05-22T15:46:22.034488+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T15:59:17.155128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.c","version":"0'0","reported_seq":10,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.468284+0000","last_change":"2023-05-22T15:46:23.466723+0000","last_active":"2023-05-22T15:46:23.468284+0000","last_peered":"2023-05-22T15:46:23.468284+0000","last_clean":"2023-05-22T15:46:23.468284+0000","last_became_active":"2023-05-22T15:46:23.466454+0000","last_became_peered":"2023-05-22T15:46:23.466454+0000","last_unstale":"2023-05-22T15:46:23.468284+0000","last_undegraded":"2023-05-22T15:46:23.468284+0000","last_fullsized":"2023-05-22T15:46:23.468284+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:33:34.943631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.d","version":"146'1","reported_seq":12,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.494461+0000","last_change":"2023-05-22T15:46:23.103043+0000","last_active":"2023-05-22T15:46:23.494461+0000","last_peered":"2023-05-22T15:46:23.494461+0000","last_clean":"2023-05-22T15:46:23.494461+0000","last_became_active":"2023-05-22T15:46:23.102220+0000","last_became_peered":"2023-05-22T15:46:23.102220+0000","last_unstale":"2023-05-22T15:46:23.494461+0000","last_undegraded":"2023-05-22T15:46:23.494461+0000","last_fullsized":"2023-05-22T15:46:23.494461+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_clean_scrub_stamp":"2023-05-22T15:46:22.034488+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:20:54.824911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.d","version":"0'0","reported_seq":10,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.468333+0000","last_change":"2023-05-22T15:46:23.464411+0000","last_active":"2023-05-22T15:46:23.468333+0000","last_peered":"2023-05-22T15:46:23.468333+0000","last_clean":"2023-05-22T15:46:23.468333+0000","last_became_active":"2023-05-22T15:46:23.464177+0000","last_became_peered":"2023-05-22T15:46:23.464177+0000","last_unstale":"2023-05-22T15:46:23.468333+0000","last_undegraded":"2023-05-22T15:46:23.468333+0000","last_fullsized":"2023-05-22T15:46:23.468333+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:49:48.656654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.c","version":"0'0","reported_seq":11,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.488813+0000","last_change":"2023-05-22T15:46:23.078278+0000","last_active":"2023-05-22T15:46:23.488813+0000","last_peered":"2023-05-22T15:46:23.488813+0000","last_clean":"2023-05-22T15:46:23.488813+0000","last_became_active":"2023-05-22T15:46:23.077343+0000","last_became_peered":"2023-05-22T15:46:23.077343+0000","last_unstale":"2023-05-22T15:46:23.488813+0000","last_undegraded":"2023-05-22T15:46:23.488813+0000","last_fullsized":"2023-05-22T15:46:23.488813+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_clean_scrub_stamp":"2023-05-22T15:46:22.034488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:17:43.848098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.a","version":"0'0","reported_seq":10,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.491065+0000","last_change":"2023-05-22T15:46:23.489797+0000","last_active":"2023-05-22T15:46:23.491065+0000","last_peered":"2023-05-22T15:46:23.491065+0000","last_clean":"2023-05-22T15:46:23.491065+0000","last_became_active":"2023-05-22T15:46:23.489477+0000","last_became_peered":"2023-05-22T15:46:23.489477+0000","last_unstale":"2023-05-22T15:46:23.491065+0000","last_undegraded":"2023-05-22T15:46:23.491065+0000","last_fullsized":"2023-05-22T15:46:23.491065+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:24:33.321639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.b","version":"146'2","reported_seq":13,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.515630+0000","last_change":"2023-05-22T15:46:23.086857+0000","last_active":"2023-05-22T15:46:23.515630+0000","last_peered":"2023-05-22T15:46:23.515630+0000","last_clean":"2023-05-22T15:46:23.515630+0000","last_became_active":"2023-05-22T15:46:23.086098+0000","last_became_peered":"2023-05-22T15:46:23.086098+0000","last_unstale":"2023-05-22T15:46:23.515630+0000","last_undegraded":"2023-05-22T15:46:23.515630+0000","last_fullsized":"2023-05-22T15:46:23.515630+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_clean_scrub_stamp":"2023-05-22T15:46:22.034488+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:45:50.595714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.b","version":"0'0","reported_seq":10,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.525810+0000","last_change":"2023-05-22T15:46:23.519402+0000","last_active":"2023-05-22T15:46:23.525810+0000","last_peered":"2023-05-22T15:46:23.525810+0000","last_clean":"2023-05-22T15:46:23.525810+0000","last_became_active":"2023-05-22T15:46:23.518706+0000","last_became_peered":"2023-05-22T15:46:23.518706+0000","last_unstale":"2023-05-22T15:46:23.525810+0000","last_undegraded":"2023-05-22T15:46:23.525810+0000","last_fullsized":"2023-05-22T15:46:23.525810+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:20:12.734869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.a","version":"0'0","reported_seq":11,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.454428+0000","last_change":"2023-05-22T15:46:23.106615+0000","last_active":"2023-05-22T15:46:23.454428+0000","last_peered":"2023-05-22T15:46:23.454428+0000","last_clean":"2023-05-22T15:46:23.454428+0000","last_became_active":"2023-05-22T15:46:23.106155+0000","last_became_peered":"2023-05-22T15:46:23.106155+0000","last_unstale":"2023-05-22T15:46:23.454428+0000","last_undegraded":"2023-05-22T15:46:23.454428+0000","last_fullsized":"2023-05-22T15:46:23.454428+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_clean_scrub_stamp":"2023-05-22T15:46:22.034488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:55:28.593183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.8","version":"0'0","reported_seq":10,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.469768+0000","last_change":"2023-05-22T15:46:23.463358+0000","last_active":"2023-05-22T15:46:23.469768+0000","last_peered":"2023-05-22T15:46:23.469768+0000","last_clean":"2023-05-22T15:46:23.469768+0000","last_became_active":"2023-05-22T15:46:23.463074+0000","last_became_peered":"2023-05-22T15:46:23.463074+0000","last_unstale":"2023-05-22T15:46:23.469768+0000","last_undegraded":"2023-05-22T15:46:23.469768+0000","last_fullsized":"2023-05-22T15:46:23.469768+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T16:27:47.997551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.9","version":"0'0","reported_seq":11,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.453856+0000","last_change":"2023-05-22T15:46:23.080736+0000","last_active":"2023-05-22T15:46:23.453856+0000","last_peered":"2023-05-22T15:46:23.453856+0000","last_clean":"2023-05-22T15:46:23.453856+0000","last_became_active":"2023-05-22T15:46:23.080251+0000","last_became_peered":"2023-05-22T15:46:23.080251+0000","last_unstale":"2023-05-22T15:46:23.453856+0000","last_undegraded":"2023-05-22T15:46:23.453856+0000","last_fullsized":"2023-05-22T15:46:23.453856+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_clean_scrub_stamp":"2023-05-22T15:46:22.034488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T16:52:22.575852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.9","version":"0'0","reported_seq":10,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.539122+0000","last_change":"2023-05-22T15:46:23.533174+0000","last_active":"2023-05-22T15:46:23.539122+0000","last_peered":"2023-05-22T15:46:23.539122+0000","last_clean":"2023-05-22T15:46:23.539122+0000","last_became_active":"2023-05-22T15:46:23.530070+0000","last_became_peered":"2023-05-22T15:46:23.530070+0000","last_unstale":"2023-05-22T15:46:23.539122+0000","last_undegraded":"2023-05-22T15:46:23.539122+0000","last_fullsized":"2023-05-22T15:46:23.539122+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:31:09.541110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.8","version":"0'0","reported_seq":11,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.451869+0000","last_change":"2023-05-22T15:46:23.107191+0000","last_active":"2023-05-22T15:46:23.451869+0000","last_peered":"2023-05-22T15:46:23.451869+0000","last_clean":"2023-05-22T15:46:23.451869+0000","last_became_active":"2023-05-22T15:46:23.106653+0000","last_became_peered":"2023-05-22T15:46:23.106653+0000","last_unstale":"2023-05-22T15:46:23.451869+0000","last_undegraded":"2023-05-22T15:46:23.451869+0000","last_fullsized":"2023-05-22T15:46:23.451869+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_clean_scrub_stamp":"2023-05-22T15:46:22.034488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:00:00.382995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.6","version":"0'0","reported_seq":10,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.524393+0000","last_change":"2023-05-22T15:46:23.509230+0000","last_active":"2023-05-22T15:46:23.524393+0000","last_peered":"2023-05-22T15:46:23.524393+0000","last_clean":"2023-05-22T15:46:23.524393+0000","last_became_active":"2023-05-22T15:46:23.508470+0000","last_became_peered":"2023-05-22T15:46:23.508470+0000","last_unstale":"2023-05-22T15:46:23.524393+0000","last_undegraded":"2023-05-22T15:46:23.524393+0000","last_fullsized":"2023-05-22T15:46:23.524393+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:55:41.010860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.7","version":"146'2","reported_seq":13,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.520215+0000","last_change":"2023-05-22T15:46:23.091817+0000","last_active":"2023-05-22T15:46:23.520215+0000","last_peered":"2023-05-22T15:46:23.520215+0000","last_clean":"2023-05-22T15:46:23.520215+0000","last_became_active":"2023-05-22T15:46:23.091434+0000","last_became_peered":"2023-05-22T15:46:23.091434+0000","last_unstale":"2023-05-22T15:46:23.520215+0000","last_undegraded":"2023-05-22T15:46:23.520215+0000","last_fullsized":"2023-05-22T15:46:23.520215+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_clean_scrub_stamp":"2023-05-22T15:46:22.034488+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T16:37:19.990243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.7","version":"0'0","reported_seq":10,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.515763+0000","last_change":"2023-05-22T15:46:23.514712+0000","last_active":"2023-05-22T15:46:23.515763+0000","last_peered":"2023-05-22T15:46:23.515763+0000","last_clean":"2023-05-22T15:46:23.515763+0000","last_became_active":"2023-05-22T15:46:23.514413+0000","last_became_peered":"2023-05-22T15:46:23.514413+0000","last_unstale":"2023-05-22T15:46:23.515763+0000","last_undegraded":"2023-05-22T15:46:23.515763+0000","last_fullsized":"2023-05-22T15:46:23.515763+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:20:29.148898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.6","version":"147'1","reported_seq":12,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-05-22T15:46:24.474556+0000","last_change":"2023-05-22T15:46:23.114945+0000","last_active":"2023-05-22T15:46:24.474556+0000","last_peered":"2023-05-22T15:46:24.474556+0000","last_clean":"2023-05-22T15:46:24.474556+0000","last_became_active":"2023-05-22T15:46:23.114027+0000","last_became_peered":"2023-05-22T15:46:23.114027+0000","last_unstale":"2023-05-22T15:46:24.474556+0000","last_undegraded":"2023-05-22T15:46:24.474556+0000","last_fullsized":"2023-05-22T15:46:24.474556+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_clean_scrub_stamp":"2023-05-22T15:46:22.034488+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T18:52:28.340453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.4","version":"0'0","reported_seq":10,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.526363+0000","last_change":"2023-05-22T15:46:23.524799+0000","last_active":"2023-05-22T15:46:23.526363+0000","last_peered":"2023-05-22T15:46:23.526363+0000","last_clean":"2023-05-22T15:46:23.526363+0000","last_became_active":"2023-05-22T15:46:23.524486+0000","last_became_peered":"2023-05-22T15:46:23.524486+0000","last_unstale":"2023-05-22T15:46:23.526363+0000","last_undegraded":"2023-05-22T15:46:23.526363+0000","last_fullsized":"2023-05-22T15:46:23.526363+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:46:38.837010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.5","version":"0'0","reported_seq":11,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.463933+0000","last_change":"2023-05-22T15:46:23.108380+0000","last_active":"2023-05-22T15:46:23.463933+0000","last_peered":"2023-05-22T15:46:23.463933+0000","last_clean":"2023-05-22T15:46:23.463933+0000","last_became_active":"2023-05-22T15:46:23.108014+0000","last_became_peered":"2023-05-22T15:46:23.108014+0000","last_unstale":"2023-05-22T15:46:23.463933+0000","last_undegraded":"2023-05-22T15:46:23.463933+0000","last_fullsized":"2023-05-22T15:46:23.463933+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_clean_scrub_stamp":"2023-05-22T15:46:22.034488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T03:37:19.248335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.5","version":"0'0","reported_seq":10,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.494282+0000","last_change":"2023-05-22T15:46:23.481179+0000","last_active":"2023-05-22T15:46:23.494282+0000","last_peered":"2023-05-22T15:46:23.494282+0000","last_clean":"2023-05-22T15:46:23.494282+0000","last_became_active":"2023-05-22T15:46:23.480122+0000","last_became_peered":"2023-05-22T15:46:23.480122+0000","last_unstale":"2023-05-22T15:46:23.494282+0000","last_undegraded":"2023-05-22T15:46:23.494282+0000","last_fullsized":"2023-05-22T15:46:23.494282+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:47:45.260369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.4","version":"0'0","reported_seq":11,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.457884+0000","last_change":"2023-05-22T15:46:23.093730+0000","last_active":"2023-05-22T15:46:23.457884+0000","last_peered":"2023-05-22T15:46:23.457884+0000","last_clean":"2023-05-22T15:46:23.457884+0000","last_became_active":"2023-05-22T15:46:23.093418+0000","last_became_peered":"2023-05-22T15:46:23.093418+0000","last_unstale":"2023-05-22T15:46:23.457884+0000","last_undegraded":"2023-05-22T15:46:23.457884+0000","last_fullsized":"2023-05-22T15:46:23.457884+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_clean_scrub_stamp":"2023-05-22T15:46:22.034488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T16:38:30.323718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.2","version":"0'0","reported_seq":10,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.539030+0000","last_change":"2023-05-22T15:46:23.532094+0000","last_active":"2023-05-22T15:46:23.539030+0000","last_peered":"2023-05-22T15:46:23.539030+0000","last_clean":"2023-05-22T15:46:23.539030+0000","last_became_active":"2023-05-22T15:46:23.531149+0000","last_became_peered":"2023-05-22T15:46:23.531149+0000","last_unstale":"2023-05-22T15:46:23.539030+0000","last_undegraded":"2023-05-22T15:46:23.539030+0000","last_fullsized":"2023-05-22T15:46:23.539030+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:25:55.107784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.3","version":"0'0","reported_seq":11,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.461807+0000","last_change":"2023-05-22T15:46:23.110053+0000","last_active":"2023-05-22T15:46:23.461807+0000","last_peered":"2023-05-22T15:46:23.461807+0000","last_clean":"2023-05-22T15:46:23.461807+0000","last_became_active":"2023-05-22T15:46:23.109753+0000","last_became_peered":"2023-05-22T15:46:23.109753+0000","last_unstale":"2023-05-22T15:46:23.461807+0000","last_undegraded":"2023-05-22T15:46:23.461807+0000","last_fullsized":"2023-05-22T15:46:23.461807+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_clean_scrub_stamp":"2023-05-22T15:46:22.034488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T18:29:44.257708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.3","version":"0'0","reported_seq":10,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.507381+0000","last_change":"2023-05-22T15:46:23.479044+0000","last_active":"2023-05-22T15:46:23.507381+0000","last_peered":"2023-05-22T15:46:23.507381+0000","last_clean":"2023-05-22T15:46:23.507381+0000","last_became_active":"2023-05-22T15:46:23.478147+0000","last_became_peered":"2023-05-22T15:46:23.478147+0000","last_unstale":"2023-05-22T15:46:23.507381+0000","last_undegraded":"2023-05-22T15:46:23.507381+0000","last_fullsized":"2023-05-22T15:46:23.507381+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T18:05:16.988218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.2","version":"0'0","reported_seq":11,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.502265+0000","last_change":"2023-05-22T15:46:23.112868+0000","last_active":"2023-05-22T15:46:23.502265+0000","last_peered":"2023-05-22T15:46:23.502265+0000","last_clean":"2023-05-22T15:46:23.502265+0000","last_became_active":"2023-05-22T15:46:23.112288+0000","last_became_peered":"2023-05-22T15:46:23.112288+0000","last_unstale":"2023-05-22T15:46:23.502265+0000","last_undegraded":"2023-05-22T15:46:23.502265+0000","last_fullsized":"2023-05-22T15:46:23.502265+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_clean_scrub_stamp":"2023-05-22T15:46:22.034488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:16:19.873549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.0","version":"0'0","reported_seq":10,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.511807+0000","last_change":"2023-05-22T15:46:23.510888+0000","last_active":"2023-05-22T15:46:23.511807+0000","last_peered":"2023-05-22T15:46:23.511807+0000","last_clean":"2023-05-22T15:46:23.511807+0000","last_became_active":"2023-05-22T15:46:23.510590+0000","last_became_peered":"2023-05-22T15:46:23.510590+0000","last_unstale":"2023-05-22T15:46:23.511807+0000","last_undegraded":"2023-05-22T15:46:23.511807+0000","last_fullsized":"2023-05-22T15:46:23.511807+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:03:37.104570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1","version":"0'0","reported_seq":11,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.454122+0000","last_change":"2023-05-22T15:46:23.091423+0000","last_active":"2023-05-22T15:46:23.454122+0000","last_peered":"2023-05-22T15:46:23.454122+0000","last_clean":"2023-05-22T15:46:23.454122+0000","last_became_active":"2023-05-22T15:46:23.091088+0000","last_became_peered":"2023-05-22T15:46:23.091088+0000","last_unstale":"2023-05-22T15:46:23.454122+0000","last_undegraded":"2023-05-22T15:46:23.454122+0000","last_fullsized":"2023-05-22T15:46:23.454122+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_clean_scrub_stamp":"2023-05-22T15:46:22.034488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:52:16.638521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.1","version":"0'0","reported_seq":10,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.526388+0000","last_change":"2023-05-22T15:46:23.525029+0000","last_active":"2023-05-22T15:46:23.526388+0000","last_peered":"2023-05-22T15:46:23.526388+0000","last_clean":"2023-05-22T15:46:23.526388+0000","last_became_active":"2023-05-22T15:46:23.524588+0000","last_became_peered":"2023-05-22T15:46:23.524588+0000","last_unstale":"2023-05-22T15:46:23.526388+0000","last_undegraded":"2023-05-22T15:46:23.526388+0000","last_fullsized":"2023-05-22T15:46:23.526388+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:16:20.885635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.0","version":"0'0","reported_seq":11,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.451974+0000","last_change":"2023-05-22T15:46:23.105214+0000","last_active":"2023-05-22T15:46:23.451974+0000","last_peered":"2023-05-22T15:46:23.451974+0000","last_clean":"2023-05-22T15:46:23.451974+0000","last_became_active":"2023-05-22T15:46:23.104909+0000","last_became_peered":"2023-05-22T15:46:23.104909+0000","last_unstale":"2023-05-22T15:46:23.451974+0000","last_undegraded":"2023-05-22T15:46:23.451974+0000","last_fullsized":"2023-05-22T15:46:23.451974+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_clean_scrub_stamp":"2023-05-22T15:46:22.034488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:37:03.818368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.12","version":"0'0","reported_seq":10,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.538759+0000","last_change":"2023-05-22T15:46:23.527087+0000","last_active":"2023-05-22T15:46:23.538759+0000","last_peered":"2023-05-22T15:46:23.538759+0000","last_clean":"2023-05-22T15:46:23.538759+0000","last_became_active":"2023-05-22T15:46:23.526057+0000","last_became_peered":"2023-05-22T15:46:23.526057+0000","last_unstale":"2023-05-22T15:46:23.538759+0000","last_undegraded":"2023-05-22T15:46:23.538759+0000","last_fullsized":"2023-05-22T15:46:23.538759+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:20:55.836997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.13","version":"146'2","reported_seq":13,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.526686+0000","last_change":"2023-05-22T15:46:23.112061+0000","last_active":"2023-05-22T15:46:23.526686+0000","last_peered":"2023-05-22T15:46:23.526686+0000","last_clean":"2023-05-22T15:46:23.526686+0000","last_became_active":"2023-05-22T15:46:23.111315+0000","last_became_peered":"2023-05-22T15:46:23.111315+0000","last_unstale":"2023-05-22T15:46:23.526686+0000","last_undegraded":"2023-05-22T15:46:23.526686+0000","last_fullsized":"2023-05-22T15:46:23.526686+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_clean_scrub_stamp":"2023-05-22T15:46:22.034488+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:37:25.171147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"134'249","reported_seq":755,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.495538+0000","last_change":"2023-05-22T15:26:39.318204+0000","last_active":"2023-05-22T15:46:23.495538+0000","last_peered":"2023-05-22T15:46:23.495538+0000","last_clean":"2023-05-22T15:46:23.495538+0000","last_became_active":"2023-05-22T15:26:39.315370+0000","last_became_peered":"2023-05-22T15:26:39.315370+0000","last_unstale":"2023-05-22T15:46:23.495538+0000","last_undegraded":"2023-05-22T15:46:23.495538+0000","last_fullsized":"2023-05-22T15:46:23.495538+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:38.020076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:38.020076+0000","last_clean_scrub_stamp":"2023-05-22T15:26:38.020076+0000","objects_scrubbed":0,"log_size":249,"log_dups_size":0,"ondisk_log_size":249,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:06:39.207543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1268,"num_read_kb":2572,"num_write":399,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":141,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.489039+0000","last_change":"2023-05-22T15:26:44.971951+0000","last_active":"2023-05-22T15:46:23.489039+0000","last_peered":"2023-05-22T15:46:23.489039+0000","last_clean":"2023-05-22T15:46:23.489039+0000","last_became_active":"2023-05-22T15:26:44.970872+0000","last_became_peered":"2023-05-22T15:26:44.970872+0000","last_unstale":"2023-05-22T15:46:23.489039+0000","last_undegraded":"2023-05-22T15:46:23.489039+0000","last_fullsized":"2023-05-22T15:46:23.489039+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:55:34.752949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.11","version":"0'0","reported_seq":10,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.523874+0000","last_change":"2023-05-22T15:46:23.510106+0000","last_active":"2023-05-22T15:46:23.523874+0000","last_peered":"2023-05-22T15:46:23.523874+0000","last_clean":"2023-05-22T15:46:23.523874+0000","last_became_active":"2023-05-22T15:46:23.509405+0000","last_became_peered":"2023-05-22T15:46:23.509405+0000","last_unstale":"2023-05-22T15:46:23.523874+0000","last_undegraded":"2023-05-22T15:46:23.523874+0000","last_fullsized":"2023-05-22T15:46:23.523874+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:37:33.613777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.10","version":"146'8","reported_seq":19,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.531254+0000","last_change":"2023-05-22T15:46:23.091282+0000","last_active":"2023-05-22T15:46:23.531254+0000","last_peered":"2023-05-22T15:46:23.531254+0000","last_clean":"2023-05-22T15:46:23.531254+0000","last_became_active":"2023-05-22T15:46:23.086640+0000","last_became_peered":"2023-05-22T15:46:23.086640+0000","last_unstale":"2023-05-22T15:46:23.531254+0000","last_undegraded":"2023-05-22T15:46:23.531254+0000","last_fullsized":"2023-05-22T15:46:23.531254+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_clean_scrub_stamp":"2023-05-22T15:46:22.034488+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T16:27:46.985465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":134,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.495304+0000","last_change":"2023-05-22T15:26:44.977198+0000","last_active":"2023-05-22T15:46:23.495304+0000","last_peered":"2023-05-22T15:46:23.495304+0000","last_clean":"2023-05-22T15:46:23.495304+0000","last_became_active":"2023-05-22T15:26:44.976445+0000","last_became_peered":"2023-05-22T15:26:44.976445+0000","last_unstale":"2023-05-22T15:46:23.495304+0000","last_undegraded":"2023-05-22T15:46:23.495304+0000","last_fullsized":"2023-05-22T15:46:23.495304+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:12:41.464054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.10","version":"0'0","reported_seq":10,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.511947+0000","last_change":"2023-05-22T15:46:23.510954+0000","last_active":"2023-05-22T15:46:23.511947+0000","last_peered":"2023-05-22T15:46:23.511947+0000","last_clean":"2023-05-22T15:46:23.511947+0000","last_became_active":"2023-05-22T15:46:23.510663+0000","last_became_peered":"2023-05-22T15:46:23.510663+0000","last_unstale":"2023-05-22T15:46:23.511947+0000","last_undegraded":"2023-05-22T15:46:23.511947+0000","last_fullsized":"2023-05-22T15:46:23.511947+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:42:45.119566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.11","version":"0'0","reported_seq":11,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.462935+0000","last_change":"2023-05-22T15:46:23.102815+0000","last_active":"2023-05-22T15:46:23.462935+0000","last_peered":"2023-05-22T15:46:23.462935+0000","last_clean":"2023-05-22T15:46:23.462935+0000","last_became_active":"2023-05-22T15:46:23.102405+0000","last_became_peered":"2023-05-22T15:46:23.102405+0000","last_unstale":"2023-05-22T15:46:23.462935+0000","last_undegraded":"2023-05-22T15:46:23.462935+0000","last_fullsized":"2023-05-22T15:46:23.462935+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_clean_scrub_stamp":"2023-05-22T15:46:22.034488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:45:50.595714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":134,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.464003+0000","last_change":"2023-05-22T15:26:44.969469+0000","last_active":"2023-05-22T15:46:23.464003+0000","last_peered":"2023-05-22T15:46:23.464003+0000","last_clean":"2023-05-22T15:46:23.464003+0000","last_became_active":"2023-05-22T15:26:44.969083+0000","last_became_peered":"2023-05-22T15:26:44.969083+0000","last_unstale":"2023-05-22T15:46:23.464003+0000","last_undegraded":"2023-05-22T15:46:23.464003+0000","last_fullsized":"2023-05-22T15:46:23.464003+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:29:23.670067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.13","version":"0'0","reported_seq":10,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.471830+0000","last_change":"2023-05-22T15:46:23.467587+0000","last_active":"2023-05-22T15:46:23.471830+0000","last_peered":"2023-05-22T15:46:23.471830+0000","last_clean":"2023-05-22T15:46:23.471830+0000","last_became_active":"2023-05-22T15:46:23.467328+0000","last_became_peered":"2023-05-22T15:46:23.467328+0000","last_unstale":"2023-05-22T15:46:23.471830+0000","last_undegraded":"2023-05-22T15:46:23.471830+0000","last_fullsized":"2023-05-22T15:46:23.471830+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T16:27:47.997551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.12","version":"0'0","reported_seq":11,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.464875+0000","last_change":"2023-05-22T15:46:23.105368+0000","last_active":"2023-05-22T15:46:23.464875+0000","last_peered":"2023-05-22T15:46:23.464875+0000","last_clean":"2023-05-22T15:46:23.464875+0000","last_became_active":"2023-05-22T15:46:23.104771+0000","last_became_peered":"2023-05-22T15:46:23.104771+0000","last_unstale":"2023-05-22T15:46:23.464875+0000","last_undegraded":"2023-05-22T15:46:23.464875+0000","last_fullsized":"2023-05-22T15:46:23.464875+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_clean_scrub_stamp":"2023-05-22T15:46:22.034488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:42:03.486863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":141,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.480976+0000","last_change":"2023-05-22T15:26:44.976632+0000","last_active":"2023-05-22T15:46:23.480976+0000","last_peered":"2023-05-22T15:46:23.480976+0000","last_clean":"2023-05-22T15:46:23.480976+0000","last_became_active":"2023-05-22T15:26:44.975546+0000","last_became_peered":"2023-05-22T15:26:44.975546+0000","last_unstale":"2023-05-22T15:46:23.480976+0000","last_undegraded":"2023-05-22T15:46:23.480976+0000","last_fullsized":"2023-05-22T15:46:23.480976+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:42:57.973030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.15","version":"0'0","reported_seq":10,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.539077+0000","last_change":"2023-05-22T15:46:23.532990+0000","last_active":"2023-05-22T15:46:23.539077+0000","last_peered":"2023-05-22T15:46:23.539077+0000","last_clean":"2023-05-22T15:46:23.539077+0000","last_became_active":"2023-05-22T15:46:23.531267+0000","last_became_peered":"2023-05-22T15:46:23.531267+0000","last_unstale":"2023-05-22T15:46:23.539077+0000","last_undegraded":"2023-05-22T15:46:23.539077+0000","last_fullsized":"2023-05-22T15:46:23.539077+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T03:25:18.349611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.14","version":"146'6","reported_seq":17,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.539982+0000","last_change":"2023-05-22T15:46:23.104870+0000","last_active":"2023-05-22T15:46:23.539982+0000","last_peered":"2023-05-22T15:46:23.539982+0000","last_clean":"2023-05-22T15:46:23.539982+0000","last_became_active":"2023-05-22T15:46:23.104126+0000","last_became_peered":"2023-05-22T15:46:23.104126+0000","last_unstale":"2023-05-22T15:46:23.539982+0000","last_undegraded":"2023-05-22T15:46:23.539982+0000","last_fullsized":"2023-05-22T15:46:23.539982+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_clean_scrub_stamp":"2023-05-22T15:46:22.034488+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:20:28.136812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1548,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":125,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.466004+0000","last_change":"2023-05-22T15:26:45.091475+0000","last_active":"2023-05-22T15:46:23.466004+0000","last_peered":"2023-05-22T15:46:23.466004+0000","last_clean":"2023-05-22T15:46:23.466004+0000","last_became_active":"2023-05-22T15:26:45.091143+0000","last_became_peered":"2023-05-22T15:26:45.091143+0000","last_unstale":"2023-05-22T15:46:23.466004+0000","last_undegraded":"2023-05-22T15:46:23.466004+0000","last_fullsized":"2023-05-22T15:46:23.466004+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:12:41.464054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.14","version":"0'0","reported_seq":10,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.538891+0000","last_change":"2023-05-22T15:46:23.530892+0000","last_active":"2023-05-22T15:46:23.538891+0000","last_peered":"2023-05-22T15:46:23.538891+0000","last_clean":"2023-05-22T15:46:23.538891+0000","last_became_active":"2023-05-22T15:46:23.527258+0000","last_became_peered":"2023-05-22T15:46:23.527258+0000","last_unstale":"2023-05-22T15:46:23.538891+0000","last_undegraded":"2023-05-22T15:46:23.538891+0000","last_fullsized":"2023-05-22T15:46:23.538891+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:49:34.728176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.15","version":"146'3","reported_seq":14,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.516631+0000","last_change":"2023-05-22T15:46:23.083261+0000","last_active":"2023-05-22T15:46:23.516631+0000","last_peered":"2023-05-22T15:46:23.516631+0000","last_clean":"2023-05-22T15:46:23.516631+0000","last_became_active":"2023-05-22T15:46:23.082317+0000","last_became_peered":"2023-05-22T15:46:23.082317+0000","last_unstale":"2023-05-22T15:46:23.516631+0000","last_undegraded":"2023-05-22T15:46:23.516631+0000","last_fullsized":"2023-05-22T15:46:23.516631+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_clean_scrub_stamp":"2023-05-22T15:46:22.034488+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T18:29:44.257708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":134,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.504247+0000","last_change":"2023-05-22T15:26:45.092367+0000","last_active":"2023-05-22T15:46:23.504247+0000","last_peered":"2023-05-22T15:46:23.504247+0000","last_clean":"2023-05-22T15:46:23.504247+0000","last_became_active":"2023-05-22T15:26:45.091951+0000","last_became_peered":"2023-05-22T15:26:45.091951+0000","last_unstale":"2023-05-22T15:46:23.504247+0000","last_undegraded":"2023-05-22T15:46:23.504247+0000","last_fullsized":"2023-05-22T15:46:23.504247+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:42:57.973030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.17","version":"0'0","reported_seq":10,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.522486+0000","last_change":"2023-05-22T15:46:23.509773+0000","last_active":"2023-05-22T15:46:23.522486+0000","last_peered":"2023-05-22T15:46:23.522486+0000","last_clean":"2023-05-22T15:46:23.522486+0000","last_became_active":"2023-05-22T15:46:23.492812+0000","last_became_peered":"2023-05-22T15:46:23.492812+0000","last_unstale":"2023-05-22T15:46:23.522486+0000","last_undegraded":"2023-05-22T15:46:23.522486+0000","last_fullsized":"2023-05-22T15:46:23.522486+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T16:27:47.997551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.16","version":"0'0","reported_seq":11,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.454576+0000","last_change":"2023-05-22T15:46:23.106558+0000","last_active":"2023-05-22T15:46:23.454576+0000","last_peered":"2023-05-22T15:46:23.454576+0000","last_clean":"2023-05-22T15:46:23.454576+0000","last_became_active":"2023-05-22T15:46:23.106098+0000","last_became_peered":"2023-05-22T15:46:23.106098+0000","last_unstale":"2023-05-22T15:46:23.454576+0000","last_undegraded":"2023-05-22T15:46:23.454576+0000","last_fullsized":"2023-05-22T15:46:23.454576+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_clean_scrub_stamp":"2023-05-22T15:46:22.034488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:30:32.850917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":125,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.463749+0000","last_change":"2023-05-22T15:26:44.978858+0000","last_active":"2023-05-22T15:46:23.463749+0000","last_peered":"2023-05-22T15:46:23.463749+0000","last_clean":"2023-05-22T15:46:23.463749+0000","last_became_active":"2023-05-22T15:26:44.978443+0000","last_became_peered":"2023-05-22T15:26:44.978443+0000","last_unstale":"2023-05-22T15:46:23.463749+0000","last_undegraded":"2023-05-22T15:46:23.463749+0000","last_fullsized":"2023-05-22T15:46:23.463749+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:46:29.290243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.16","version":"0'0","reported_seq":10,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.475192+0000","last_change":"2023-05-22T15:46:23.470516+0000","last_active":"2023-05-22T15:46:23.475192+0000","last_peered":"2023-05-22T15:46:23.475192+0000","last_clean":"2023-05-22T15:46:23.475192+0000","last_became_active":"2023-05-22T15:46:23.470257+0000","last_became_peered":"2023-05-22T15:46:23.470257+0000","last_unstale":"2023-05-22T15:46:23.475192+0000","last_undegraded":"2023-05-22T15:46:23.475192+0000","last_fullsized":"2023-05-22T15:46:23.475192+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:23.046574+0000","last_clean_scrub_stamp":"2023-05-22T15:46:23.046574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T18:29:45.269794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.17","version":"0'0","reported_seq":11,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.510290+0000","last_change":"2023-05-22T15:46:23.084909+0000","last_active":"2023-05-22T15:46:23.510290+0000","last_peered":"2023-05-22T15:46:23.510290+0000","last_clean":"2023-05-22T15:46:23.510290+0000","last_became_active":"2023-05-22T15:46:23.084184+0000","last_became_peered":"2023-05-22T15:46:23.084184+0000","last_unstale":"2023-05-22T15:46:23.510290+0000","last_undegraded":"2023-05-22T15:46:23.510290+0000","last_fullsized":"2023-05-22T15:46:23.510290+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:46:22.034488+0000","last_clean_scrub_stamp":"2023-05-22T15:46:22.034488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:47:36.764285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":127,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-05-22T15:46:23.485100+0000","last_change":"2023-05-22T15:26:44.982421+0000","last_active":"2023-05-22T15:46:23.485100+0000","last_peered":"2023-05-22T15:46:23.485100+0000","last_clean":"2023-05-22T15:46:23.485100+0000","last_became_active":"2023-05-22T15:26:44.981586+0000","last_became_peered":"2023-05-22T15:26:44.981586+0000","last_unstale":"2023-05-22T15:46:23.485100+0000","last_undegraded":"2023-05-22T15:46:23.485100+0000","last_fullsized":"2023-05-22T15:46:23.485100+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:14:11.156544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":19,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":18,"num_pg":32,"stat_sum":{"num_bytes":2330,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1268,"num_read_kb":2572,"num_write":399,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":249,"ondisk_log_size":249,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575085,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":19868,"kb_used_data":2388,"kb_used_omap":0,"kb_used_meta":17472,"kb_avail":94351972,"statfs":{"total":96636764160,"available":96616419328,"internally_reserved":0,"allocated":2445312,"data_stored":1559168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":17891328},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 15:46:11 2023","interfaces":[{"interface":"back","average":{"1min":0.86699999999999999,"5min":0.89200000000000002,"15min":0.876},"min":{"1min":0.53600000000000003,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":1.294,"5min":2.637,"15min":2.637},"last":0.75700000000000001},{"interface":"front","average":{"1min":0.88,"5min":0.88700000000000001,"15min":0.89200000000000002},"min":{"1min":0.60799999999999998,"5min":0.502,"15min":0.35799999999999998},"max":{"1min":1.1519999999999999,"5min":3.0449999999999999,"15min":3.0449999999999999},"last":1.0549999999999999}]},{"osd":1,"last update":"Mon May 22 15:46:11 2023","interfaces":[{"interface":"back","average":{"1min":0.96799999999999997,"5min":0.95399999999999996,"15min":0.97899999999999998},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.1899999999999999,"5min":1.385,"15min":1.389},"last":0.85699999999999998},{"interface":"front","average":{"1min":0.96799999999999997,"5min":1.006,"15min":1.022},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.54900000000000004},"max":{"1min":1.226,"5min":2.7269999999999999,"15min":2.7269999999999999},"last":1.157}]},{"osd":2,"last update":"Mon May 22 15:46:11 2023","interfaces":[{"interface":"back","average":{"1min":0.86199999999999999,"5min":0.85099999999999998,"15min":0.871},"min":{"1min":0.55600000000000005,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.109,"5min":1.252,"15min":1.252},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.88100000000000001,"5min":0.93999999999999995,"15min":0.95699999999999996},"min":{"1min":0.504,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.143,"5min":3.141,"15min":3.141},"last":1.266}]},{"osd":3,"last update":"Mon May 22 15:46:11 2023","interfaces":[{"interface":"back","average":{"1min":0.98199999999999998,"5min":0.98899999999999999,"15min":0.96299999999999997},"min":{"1min":0.67600000000000005,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":1.3839999999999999,"5min":2.8149999999999999,"15min":2.8149999999999999},"last":0.90100000000000002},{"interface":"front","average":{"1min":0.94399999999999995,"5min":0.92100000000000004,"15min":0.96199999999999997},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.2709999999999999,"5min":1.389,"15min":1.4710000000000001},"last":0.93500000000000005}]},{"osd":4,"last update":"Mon May 22 15:46:11 2023","interfaces":[{"interface":"back","average":{"1min":0.87,"5min":0.94399999999999995,"15min":0.90400000000000003},"min":{"1min":0.60299999999999998,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.5089999999999999,"5min":3.472,"15min":3.472},"last":1.1830000000000001},{"interface":"front","average":{"1min":0.997,"5min":1.1850000000000001,"15min":1.05},"min":{"1min":0.59699999999999998,"5min":0.56000000000000005,"15min":0.44600000000000001},"max":{"1min":1.2949999999999999,"5min":15.627000000000001,"15min":15.627000000000001},"last":1.0089999999999999}]},{"osd":5,"last update":"Mon May 22 15:46:11 2023","interfaces":[{"interface":"back","average":{"1min":1.012,"5min":1.071,"15min":1.02},"min":{"1min":0.60899999999999999,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":1.4750000000000001,"5min":4.5599999999999996,"15min":4.5599999999999996},"last":1.119},{"interface":"front","average":{"1min":1.115,"5min":1.161,"15min":1.139},"min":{"1min":0.82999999999999996,"5min":0.63800000000000001,"15min":0.58799999999999997},"max":{"1min":1.302,"5min":4.3799999999999999,"15min":4.3799999999999999},"last":1.429}]},{"osd":6,"last update":"Mon May 22 15:46:11 2023","interfaces":[{"interface":"back","average":{"1min":1.073,"5min":1.1399999999999999,"15min":1.0840000000000001},"min":{"1min":0.80300000000000005,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.415,"5min":6.5069999999999997,"15min":6.5069999999999997},"last":1.3520000000000001},{"interface":"front","average":{"1min":1.1200000000000001,"5min":1.1659999999999999,"15min":1.1419999999999999},"min":{"1min":0.70299999999999996,"5min":0.62,"15min":0.62},"max":{"1min":1.458,"5min":3.3500000000000001,"15min":3.3500000000000001},"last":1.218}]}]},{"osd":6,"up_from":13,"seq":55834575086,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36748,"kb_used_data":1796,"kb_used_omap":0,"kb_used_meta":34944,"kb_avail":94335092,"statfs":{"total":96636764160,"available":96599134208,"internally_reserved":0,"allocated":1839104,"data_stored":968178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":35782656},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 15:46:20 2023","interfaces":[{"interface":"back","average":{"1min":0.90000000000000002,"5min":0.86699999999999999,"15min":0.89900000000000002},"min":{"1min":0.499,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":1.4039999999999999,"5min":1.486,"15min":1.486},"last":1.234},{"interface":"front","average":{"1min":0.89200000000000002,"5min":0.94599999999999995,"15min":0.91100000000000003},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.36299999999999999},"max":{"1min":1.2729999999999999,"5min":1.6240000000000001,"15min":1.6240000000000001},"last":0.96699999999999997}]},{"osd":1,"last update":"Mon May 22 15:46:20 2023","interfaces":[{"interface":"back","average":{"1min":0.93200000000000005,"5min":0.89100000000000001,"15min":0.91200000000000003},"min":{"1min":0.59799999999999998,"5min":0.504,"15min":0.48299999999999998},"max":{"1min":1.8799999999999999,"5min":1.8799999999999999,"15min":1.8799999999999999},"last":1.141},{"interface":"front","average":{"1min":0.95899999999999996,"5min":0.92400000000000004,"15min":0.93999999999999995},"min":{"1min":0.70299999999999996,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":1.1950000000000001,"5min":1.3320000000000001,"15min":1.554},"last":0.86599999999999999}]},{"osd":2,"last update":"Mon May 22 15:46:20 2023","interfaces":[{"interface":"back","average":{"1min":1.05,"5min":0.97399999999999998,"15min":0.98999999999999999},"min":{"1min":0.55400000000000005,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":3.2429999999999999,"5min":3.2429999999999999,"15min":3.2429999999999999},"last":1.198},{"interface":"front","average":{"1min":1.032,"5min":1.0269999999999999,"15min":1.0429999999999999},"min":{"1min":0.55300000000000005,"5min":0.5,"15min":0.442},"max":{"1min":3.3969999999999998,"5min":3.3969999999999998,"15min":3.3969999999999998},"last":0.91300000000000003}]},{"osd":3,"last update":"Mon May 22 15:46:20 2023","interfaces":[{"interface":"back","average":{"1min":0.88900000000000001,"5min":0.94599999999999995,"15min":0.97899999999999998},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":1.2470000000000001,"5min":1.5780000000000001,"15min":1.679},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.93500000000000005,"5min":0.95099999999999996,"15min":0.96299999999999997},"min":{"1min":0.69099999999999995,"5min":0.53200000000000003,"15min":0.50800000000000001},"max":{"1min":1.2430000000000001,"5min":1.383,"15min":1.4570000000000001},"last":0.81200000000000006}]},{"osd":4,"last update":"Mon May 22 15:46:20 2023","interfaces":[{"interface":"back","average":{"1min":0.91300000000000003,"5min":0.93200000000000005,"15min":0.92500000000000004},"min":{"1min":0.61699999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":1.3169999999999999,"5min":1.581,"15min":1.6859999999999999},"last":0.995},{"interface":"front","average":{"1min":0.93200000000000005,"5min":1.0209999999999999,"15min":1.0169999999999999},"min":{"1min":0.65500000000000003,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":1.242,"5min":1.452,"15min":1.8300000000000001},"last":1.101}]},{"osd":5,"last update":"Mon May 22 15:46:20 2023","interfaces":[{"interface":"back","average":{"1min":0.98799999999999999,"5min":1.0169999999999999,"15min":0.98899999999999999},"min":{"1min":0.67600000000000005,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":1.3200000000000001,"5min":1.4530000000000001,"15min":1.6499999999999999},"last":1.0309999999999999},{"interface":"front","average":{"1min":1.1459999999999999,"5min":1.141,"15min":1.1120000000000001},"min":{"1min":0.81999999999999995,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.9990000000000001,"5min":1.9990000000000001,"15min":1.9990000000000001},"last":1.29}]},{"osd":7,"last update":"Mon May 22 15:46:20 2023","interfaces":[{"interface":"back","average":{"1min":1.0860000000000001,"5min":1.131,"15min":1.125},"min":{"1min":0.85499999999999998,"5min":0.37,"15min":0.37},"max":{"1min":1.5529999999999999,"5min":1.853,"15min":1.9179999999999999},"last":1.179},{"interface":"front","average":{"1min":1.131,"5min":1.133,"15min":1.135},"min":{"1min":0.80300000000000005,"5min":0.442,"15min":0.442},"max":{"1min":1.579,"5min":1.579,"15min":1.8520000000000001},"last":1.268}]}]},{"osd":5,"up_from":13,"seq":55834575086,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":26184,"kb_used_data":1792,"kb_used_omap":0,"kb_used_meta":24384,"kb_avail":94345656,"statfs":{"total":96636764160,"available":96609951744,"internally_reserved":0,"allocated":1835008,"data_stored":966742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":24969216},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 15:46:10 2023","interfaces":[{"interface":"back","average":{"1min":0.90900000000000003,"5min":0.85599999999999998,"15min":0.88300000000000001},"min":{"1min":0.61399999999999999,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.2889999999999999,"5min":1.3140000000000001,"15min":1.3440000000000001},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.91800000000000004,"5min":0.88600000000000001,"15min":0.88500000000000001},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":1.2709999999999999,"5min":1.3899999999999999,"15min":2.1699999999999999},"last":1.274}]},{"osd":1,"last update":"Mon May 22 15:46:10 2023","interfaces":[{"interface":"back","average":{"1min":0.94999999999999996,"5min":0.96099999999999997,"15min":0.995},"min":{"1min":0.64800000000000002,"5min":0.52400000000000002,"15min":0.51000000000000001},"max":{"1min":1.3109999999999999,"5min":1.337,"15min":1.9370000000000001},"last":0.98399999999999999},{"interface":"front","average":{"1min":1.073,"5min":1.008,"15min":1.026},"min":{"1min":0.501,"5min":0.501,"15min":0.501},"max":{"1min":1.3089999999999999,"5min":1.6970000000000001,"15min":1.6970000000000001},"last":0.89200000000000002}]},{"osd":2,"last update":"Mon May 22 15:46:10 2023","interfaces":[{"interface":"back","average":{"1min":0.89100000000000001,"5min":0.91600000000000004,"15min":0.93400000000000005},"min":{"1min":0.505,"5min":0.505,"15min":0.504},"max":{"1min":1.343,"5min":1.5549999999999999,"15min":1.9830000000000001},"last":1.0980000000000001},{"interface":"front","average":{"1min":0.91800000000000004,"5min":0.94999999999999996,"15min":0.95199999999999996},"min":{"1min":0.56100000000000005,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.202,"5min":1.321,"15min":2.1120000000000001},"last":0.73899999999999999}]},{"osd":3,"last update":"Mon May 22 15:46:10 2023","interfaces":[{"interface":"back","average":{"1min":0.92900000000000005,"5min":0.92000000000000004,"15min":0.94899999999999995},"min":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.48199999999999998},"max":{"1min":1.206,"5min":1.2809999999999999,"15min":1.4419999999999999},"last":0.80500000000000005},{"interface":"front","average":{"1min":1.028,"5min":1.008,"15min":1.036},"min":{"1min":0.626,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":1.2490000000000001,"5min":1.371,"15min":2.52},"last":1.2}]},{"osd":4,"last update":"Mon May 22 15:46:10 2023","interfaces":[{"interface":"back","average":{"1min":0.96999999999999997,"5min":0.95699999999999996,"15min":0.95199999999999996},"min":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":1.405,"5min":1.405,"15min":2.226},"last":0.85699999999999998},{"interface":"front","average":{"1min":1.0629999999999999,"5min":1.024,"15min":1.0309999999999999},"min":{"1min":0.76000000000000001,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.3740000000000001,"5min":1.863,"15min":1.863},"last":1.04}]},{"osd":6,"last update":"Mon May 22 15:46:10 2023","interfaces":[{"interface":"back","average":{"1min":1.008,"5min":1.024,"15min":1.0349999999999999},"min":{"1min":0.51900000000000002,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.397,"5min":1.4330000000000001,"15min":2.831},"last":1.3360000000000001},{"interface":"front","average":{"1min":1.0369999999999999,"5min":1.0589999999999999,"15min":1.071},"min":{"1min":0.67000000000000004,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":1.4239999999999999,"5min":1.4239999999999999,"15min":2.3690000000000002},"last":0.93500000000000005}]},{"osd":7,"last update":"Mon May 22 15:46:10 2023","interfaces":[{"interface":"back","average":{"1min":1.081,"5min":1.0900000000000001,"15min":1.1120000000000001},"min":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.42499999999999999},"max":{"1min":1.4359999999999999,"5min":1.47,"15min":1.849},"last":0.96199999999999997},{"interface":"front","average":{"1min":1.1599999999999999,"5min":1.149,"15min":1.1659999999999999},"min":{"1min":0.86899999999999999,"5min":0.64000000000000001,"15min":0.64000000000000001},"max":{"1min":1.4339999999999999,"5min":1.575,"15min":2.7429999999999999},"last":1.4019999999999999}]}]},{"osd":4,"up_from":13,"seq":55834575084,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31684,"kb_used_data":1788,"kb_used_omap":0,"kb_used_meta":29888,"kb_avail":94340156,"statfs":{"total":96636764160,"available":96604319744,"internally_reserved":0,"allocated":1830912,"data_stored":966700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":30605312},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 15:46:19 2023","interfaces":[{"interface":"back","average":{"1min":0.93200000000000005,"5min":0.86799999999999999,"15min":0.90500000000000003},"min":{"1min":0.63400000000000001,"5min":0.48999999999999999,"15min":0.34100000000000003},"max":{"1min":1.2629999999999999,"5min":1.4119999999999999,"15min":1.9910000000000001},"last":0.78800000000000003},{"interface":"front","average":{"1min":0.91300000000000003,"5min":0.90000000000000002,"15min":0.92200000000000004},"min":{"1min":0.51100000000000001,"5min":0.499,"15min":0.373},"max":{"1min":1.284,"5min":1.369,"15min":1.3779999999999999},"last":1.0149999999999999}]},{"osd":1,"last update":"Mon May 22 15:46:19 2023","interfaces":[{"interface":"back","average":{"1min":0.94799999999999995,"5min":0.91800000000000004,"15min":0.96399999999999997},"min":{"1min":0.60499999999999998,"5min":0.436,"15min":0.436},"max":{"1min":1.1970000000000001,"5min":1.3899999999999999,"15min":1.3899999999999999},"last":1.137},{"interface":"front","average":{"1min":0.91700000000000004,"5min":0.84499999999999997,"15min":0.90100000000000002},"min":{"1min":0.56699999999999995,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.157,"5min":1.2889999999999999,"15min":1.3640000000000001},"last":0.86099999999999999}]},{"osd":2,"last update":"Mon May 22 15:46:19 2023","interfaces":[{"interface":"back","average":{"1min":0.90100000000000002,"5min":0.91600000000000004,"15min":0.94199999999999995},"min":{"1min":0.56299999999999994,"5min":0.50800000000000001,"15min":0.495},"max":{"1min":1.244,"5min":1.96,"15min":2.1019999999999999},"last":0.94999999999999996},{"interface":"front","average":{"1min":0.97799999999999998,"5min":0.98399999999999999,"15min":0.99199999999999999},"min":{"1min":0.58099999999999996,"5min":0.48799999999999999,"15min":0.48399999999999999},"max":{"1min":1.329,"5min":1.4399999999999999,"15min":1.4399999999999999},"last":0.91500000000000004}]},{"osd":3,"last update":"Mon May 22 15:46:19 2023","interfaces":[{"interface":"back","average":{"1min":0.91100000000000003,"5min":0.90200000000000002,"15min":0.93200000000000005},"min":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":1.224,"5min":1.28,"15min":1.3169999999999999},"last":0.53700000000000003},{"interface":"front","average":{"1min":1.0469999999999999,"5min":1.014,"15min":1.0209999999999999},"min":{"1min":0.72199999999999998,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":1.3420000000000001,"5min":1.4179999999999999,"15min":2.1720000000000002},"last":1.1140000000000001}]},{"osd":5,"last update":"Mon May 22 15:46:19 2023","interfaces":[{"interface":"back","average":{"1min":1.0209999999999999,"5min":0.97199999999999998,"15min":0.97699999999999998},"min":{"1min":0.55000000000000004,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.357,"5min":1.373,"15min":1.472},"last":0.67700000000000005},{"interface":"front","average":{"1min":1.0389999999999999,"5min":0.98999999999999999,"15min":0.97499999999999998},"min":{"1min":0.59999999999999998,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.262,"5min":1.365,"15min":1.4139999999999999},"last":1.0860000000000001}]},{"osd":6,"last update":"Mon May 22 15:46:19 2023","interfaces":[{"interface":"back","average":{"1min":1.0649999999999999,"5min":0.99299999999999999,"15min":0.98499999999999999},"min":{"1min":0.72099999999999997,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":1.456,"5min":1.456,"15min":2.2349999999999999},"last":1.0600000000000001},{"interface":"front","average":{"1min":1.131,"5min":1.103,"15min":1.1100000000000001},"min":{"1min":0.66500000000000004,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":1.425,"5min":1.486,"15min":1.8220000000000001},"last":1.1970000000000001}]},{"osd":7,"last update":"Mon May 22 15:46:19 2023","interfaces":[{"interface":"back","average":{"1min":1.137,"5min":1.0629999999999999,"15min":1.052},"min":{"1min":0.85699999999999998,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":1.389,"5min":1.46,"15min":1.5800000000000001},"last":0.98999999999999999},{"interface":"front","average":{"1min":1.1930000000000001,"5min":1.167,"15min":1.143},"min":{"1min":0.78900000000000003,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":1.5620000000000001,"5min":1.6559999999999999,"15min":2.306},"last":1.1599999999999999}]}]},{"osd":3,"up_from":13,"seq":55834575084,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37128,"kb_used_data":1792,"kb_used_omap":0,"kb_used_meta":35328,"kb_avail":94334712,"statfs":{"total":96636764160,"available":96598745088,"internally_reserved":0,"allocated":1835008,"data_stored":966790,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":36175872},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 15:46:03 2023","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.61699999999999999,"15min":0.63600000000000001},"min":{"1min":0.50700000000000001,"5min":0.252,"15min":0.19},"max":{"1min":1.169,"5min":1.169,"15min":1.194},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.746,"5min":0.68500000000000005,"15min":0.70899999999999996},"min":{"1min":0.48899999999999999,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":1.085,"5min":1.296,"15min":1.296},"last":0.40500000000000003}]},{"osd":1,"last update":"Mon May 22 15:46:03 2023","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.64200000000000002,"15min":0.66200000000000003},"min":{"1min":0.52100000000000002,"5min":0.314,"15min":0.29499999999999998},"max":{"1min":0.90600000000000003,"5min":1.2609999999999999,"15min":1.2609999999999999},"last":0.434},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.72899999999999998,"15min":0.76800000000000002},"min":{"1min":0.47499999999999998,"5min":0.34399999999999997,"15min":0.313},"max":{"1min":1.081,"5min":1.1359999999999999,"15min":1.222},"last":0.501}]},{"osd":2,"last update":"Mon May 22 15:46:03 2023","interfaces":[{"interface":"back","average":{"1min":0.84199999999999997,"5min":0.78700000000000003,"15min":0.81699999999999995},"min":{"1min":0.45600000000000002,"5min":0.40699999999999997,"15min":0.36599999999999999},"max":{"1min":1.0529999999999999,"5min":1.506,"15min":1.506},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.81899999999999995,"5min":0.75600000000000001,"15min":0.80700000000000005},"min":{"1min":0.65300000000000002,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":1.032,"5min":1.147,"15min":1.359},"last":0.47499999999999998}]},{"osd":4,"last update":"Mon May 22 15:46:03 2023","interfaces":[{"interface":"back","average":{"1min":0.97499999999999998,"5min":0.92500000000000004,"15min":0.95599999999999996},"min":{"1min":0.71999999999999997,"5min":0.54500000000000004,"15min":0.48999999999999999},"max":{"1min":1.238,"5min":1.413,"15min":1.4470000000000001},"last":0.60799999999999998},{"interface":"front","average":{"1min":1.0529999999999999,"5min":0.97499999999999998,"15min":1.0029999999999999},"min":{"1min":0.83299999999999996,"5min":0.54400000000000004,"15min":0.49199999999999999},"max":{"1min":1.407,"5min":1.407,"15min":1.4470000000000001},"last":0.70899999999999996}]},{"osd":5,"last update":"Mon May 22 15:46:03 2023","interfaces":[{"interface":"back","average":{"1min":1.0509999999999999,"5min":0.98399999999999999,"15min":1.016},"min":{"1min":0.80900000000000005,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.3149999999999999,"5min":1.5089999999999999,"15min":1.5089999999999999},"last":0.83299999999999996},{"interface":"front","average":{"1min":1.0349999999999999,"5min":1.006,"15min":1.038},"min":{"1min":0.90200000000000002,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.2130000000000001,"5min":1.4299999999999999,"15min":1.4359999999999999},"last":0.66600000000000004}]},{"osd":6,"last update":"Mon May 22 15:46:03 2023","interfaces":[{"interface":"back","average":{"1min":1.0580000000000001,"5min":1.0129999999999999,"15min":1.042},"min":{"1min":0.873,"5min":0.47699999999999998,"15min":0.46999999999999997},"max":{"1min":1.2989999999999999,"5min":1.4570000000000001,"15min":1.49},"last":0.79800000000000004},{"interface":"front","average":{"1min":1.083,"5min":1.0269999999999999,"15min":1.0700000000000001},"min":{"1min":0.79800000000000004,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":1.2949999999999999,"5min":1.4330000000000001,"15min":1.5009999999999999},"last":0.90000000000000002}]},{"osd":7,"last update":"Mon May 22 15:46:03 2023","interfaces":[{"interface":"back","average":{"1min":1.113,"5min":1.0780000000000001,"15min":1.1040000000000001},"min":{"1min":0.83599999999999997,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.3779999999999999,"5min":1.6719999999999999,"15min":1.6719999999999999},"last":0.72199999999999998},{"interface":"front","average":{"1min":1.117,"5min":1.071,"15min":1.085},"min":{"1min":0.90200000000000002,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":1.361,"5min":1.8700000000000001,"15min":1.8700000000000001},"last":0.81599999999999995}]}]},{"osd":2,"up_from":13,"seq":55834575085,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":19016,"kb_used_data":1792,"kb_used_omap":0,"kb_used_meta":17216,"kb_avail":94352824,"statfs":{"total":96636764160,"available":96617291776,"internally_reserved":0,"allocated":1835008,"data_stored":967230,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":17629184},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 15:46:03 2023","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.64400000000000002,"15min":0.65600000000000003},"min":{"1min":0.28699999999999998,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":1.097,"5min":1.1539999999999999,"15min":1.8240000000000001},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.69899999999999995,"5min":0.70499999999999996,"15min":0.746},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":1.079,"5min":1.079,"15min":1.8160000000000001},"last":0.629}]},{"osd":1,"last update":"Mon May 22 15:46:03 2023","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.72999999999999998,"15min":0.74399999999999999},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.27000000000000002},"max":{"1min":1.196,"5min":1.196,"15min":1.2},"last":0.78900000000000003},{"interface":"front","average":{"1min":0.754,"5min":0.75900000000000001,"15min":0.80800000000000005},"min":{"1min":0.32400000000000001,"5min":0.317,"15min":0.317},"max":{"1min":1.0309999999999999,"5min":1.246,"15min":2.5089999999999999},"last":0.82099999999999995}]},{"osd":3,"last update":"Mon May 22 15:46:03 2023","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":0.69399999999999995,"15min":0.72599999999999998},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.96199999999999997,"5min":1.077,"15min":2.7629999999999999},"last":0.94099999999999995},{"interface":"front","average":{"1min":0.82699999999999996,"5min":0.84599999999999997,"15min":0.88300000000000001},"min":{"1min":0.39600000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":1.1699999999999999,"5min":1.256,"15min":2.6840000000000002},"last":0.878}]},{"osd":4,"last update":"Mon May 22 15:46:03 2023","interfaces":[{"interface":"back","average":{"1min":0.97499999999999998,"5min":0.98499999999999999,"15min":1.018},"min":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":1.274,"5min":1.536,"15min":2.8039999999999998},"last":0.998},{"interface":"front","average":{"1min":1.0069999999999999,"5min":1.0009999999999999,"15min":1.028},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.2509999999999999,"5min":1.4299999999999999,"15min":2.0819999999999999},"last":1.04}]},{"osd":5,"last update":"Mon May 22 15:46:03 2023","interfaces":[{"interface":"back","average":{"1min":1.0569999999999999,"5min":1.056,"15min":1.052},"min":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.3959999999999999,"5min":1.5049999999999999,"15min":2.3860000000000001},"last":0.91400000000000003},{"interface":"front","average":{"1min":0.99099999999999999,"5min":0.97399999999999998,"15min":0.998},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.331,"5min":1.331,"15min":3.0299999999999998},"last":1.1830000000000001}]},{"osd":6,"last update":"Mon May 22 15:46:03 2023","interfaces":[{"interface":"back","average":{"1min":1.038,"5min":1.012,"15min":1.036},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":1.3320000000000001,"5min":1.49,"15min":2.883},"last":1.0800000000000001},{"interface":"front","average":{"1min":1.129,"5min":1.081,"15min":1.1040000000000001},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":1.4450000000000001,"5min":1.5660000000000001,"15min":3.1459999999999999},"last":1.157}]},{"osd":7,"last update":"Mon May 22 15:46:03 2023","interfaces":[{"interface":"back","average":{"1min":1.1639999999999999,"5min":1.097,"15min":1.1279999999999999},"min":{"1min":0.55000000000000004,"5min":0.499,"15min":0.499},"max":{"1min":1.429,"5min":1.55,"15min":2.5990000000000002},"last":1.137},{"interface":"front","average":{"1min":1.1240000000000001,"5min":1.0640000000000001,"15min":1.1060000000000001},"min":{"1min":0.5,"5min":0.5,"15min":0.5},"max":{"1min":1.4650000000000001,"5min":1.4650000000000001,"15min":3.8799999999999999},"last":1.2450000000000001}]}]},{"osd":1,"up_from":13,"seq":55834575085,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29888,"kb_used_data":1784,"kb_used_omap":0,"kb_used_meta":28096,"kb_avail":94341952,"statfs":{"total":96636764160,"available":96606158848,"internally_reserved":0,"allocated":1826816,"data_stored":966630,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":28770304},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 15:46:17 2023","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.63200000000000001,"15min":0.63400000000000001},"min":{"1min":0.40600000000000003,"5min":0.26300000000000001,"15min":0.22700000000000001},"max":{"1min":1.1200000000000001,"5min":1.2270000000000001,"15min":1.2270000000000001},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.77100000000000002,"5min":0.70699999999999996,"15min":0.76400000000000001},"min":{"1min":0.35999999999999999,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":1.1839999999999999,"5min":1.1839999999999999,"15min":1.2529999999999999},"last":0.434}]},{"osd":2,"last update":"Mon May 22 15:46:17 2023","interfaces":[{"interface":"back","average":{"1min":0.76400000000000001,"5min":0.71399999999999997,"15min":0.73799999999999999},"min":{"1min":0.44800000000000001,"5min":0.317,"15min":0.28999999999999998},"max":{"1min":1.0660000000000001,"5min":1.391,"15min":1.391},"last":0.70399999999999996},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.72999999999999998,"15min":0.76300000000000001},"min":{"1min":0.49199999999999999,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.98699999999999999,"5min":1.2270000000000001,"15min":1.276},"last":0.41799999999999998}]},{"osd":3,"last update":"Mon May 22 15:46:17 2023","interfaces":[{"interface":"back","average":{"1min":0.80100000000000005,"5min":0.79900000000000004,"15min":0.82799999999999996},"min":{"1min":0.51200000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.153,"5min":1.657,"15min":1.657},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.81100000000000005,"5min":0.80800000000000005,"15min":0.81499999999999995},"min":{"1min":0.33700000000000002,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":1.095,"5min":2.0569999999999999,"15min":2.0569999999999999},"last":0.52200000000000002}]},{"osd":4,"last update":"Mon May 22 15:46:17 2023","interfaces":[{"interface":"back","average":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.96499999999999997},"min":{"1min":0.64500000000000002,"5min":0.55200000000000005,"15min":0.34000000000000002},"max":{"1min":1.502,"5min":2.9529999999999998,"15min":2.9529999999999998},"last":0.82599999999999996},{"interface":"front","average":{"1min":0.97499999999999998,"5min":0.999,"15min":1.002},"min":{"1min":0.61299999999999999,"5min":0.504,"15min":0.49299999999999999},"max":{"1min":1.292,"5min":2.7759999999999998,"15min":2.7759999999999998},"last":0.64500000000000002}]},{"osd":5,"last update":"Mon May 22 15:46:17 2023","interfaces":[{"interface":"back","average":{"1min":1.0349999999999999,"5min":1.006,"15min":1.034},"min":{"1min":0.627,"5min":0.58399999999999996,"15min":0.443},"max":{"1min":1.556,"5min":1.9570000000000001,"15min":1.9570000000000001},"last":0.73799999999999999},{"interface":"front","average":{"1min":1.1060000000000001,"5min":1.034,"15min":1.0449999999999999},"min":{"1min":0.59399999999999997,"5min":0.53400000000000003,"15min":0.47299999999999998},"max":{"1min":1.577,"5min":2.1200000000000001,"15min":2.1200000000000001},"last":0.98899999999999999}]},{"osd":6,"last update":"Mon May 22 15:46:17 2023","interfaces":[{"interface":"back","average":{"1min":1.02,"5min":1.0369999999999999,"15min":1.036},"min":{"1min":0.65900000000000003,"5min":0.54000000000000004,"15min":0.45300000000000001},"max":{"1min":1.3180000000000001,"5min":2.831,"15min":2.831},"last":0.80700000000000005},{"interface":"front","average":{"1min":1.1499999999999999,"5min":1.085,"15min":1.0940000000000001},"min":{"1min":0.55800000000000005,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":1.6299999999999999,"5min":1.7629999999999999,"15min":1.7629999999999999},"last":0.76800000000000002}]},{"osd":7,"last update":"Mon May 22 15:46:17 2023","interfaces":[{"interface":"back","average":{"1min":1.1759999999999999,"5min":1.135,"15min":1.1180000000000001},"min":{"1min":0.61299999999999999,"5min":0.60299999999999998,"15min":0.56799999999999995},"max":{"1min":1.679,"5min":2.4199999999999999,"15min":2.4199999999999999},"last":0.66000000000000003},{"interface":"front","average":{"1min":1.1819999999999999,"5min":1.1259999999999999,"15min":1.1299999999999999},"min":{"1min":0.76300000000000001,"5min":0.47299999999999998,"15min":0.41599999999999998},"max":{"1min":1.619,"5min":3.8799999999999999,"15min":3.8799999999999999},"last":0.92400000000000004}]}]},{"osd":0,"up_from":13,"seq":55834575086,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29636,"kb_used_data":2364,"kb_used_omap":0,"kb_used_meta":27264,"kb_avail":94342204,"statfs":{"total":96636764160,"available":96606416896,"internally_reserved":0,"allocated":2420736,"data_stored":1556998,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27918336},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon May 22 15:46:21 2023","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.65900000000000003,"15min":0.68100000000000005},"min":{"1min":0.307,"5min":0.248,"15min":0.223},"max":{"1min":1.2190000000000001,"5min":1.466,"15min":1.466},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.69899999999999995,"5min":0.70599999999999996,"15min":0.72899999999999998},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.24099999999999999},"max":{"1min":1.2230000000000001,"5min":1.71,"15min":1.71},"last":0.497}]},{"osd":2,"last update":"Mon May 22 15:46:21 2023","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":0.70699999999999996,"15min":0.74099999999999999},"min":{"1min":0.39700000000000002,"5min":0.318,"15min":0.26500000000000001},"max":{"1min":0.96799999999999997,"5min":1.165,"15min":1.3220000000000001},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.82799999999999996,"5min":0.75900000000000001,"15min":0.78800000000000003},"min":{"1min":0.375,"5min":0.34999999999999998,"15min":0.32900000000000001},"max":{"1min":1.1699999999999999,"5min":1.3129999999999999,"15min":1.3129999999999999},"last":0.54500000000000004}]},{"osd":3,"last update":"Mon May 22 15:46:21 2023","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":0.74099999999999999,"15min":0.76000000000000001},"min":{"1min":0.35499999999999998,"5min":0.31,"15min":0.31},"max":{"1min":0.96199999999999997,"5min":1.9299999999999999,"15min":1.9299999999999999},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.83499999999999996,"5min":0.83299999999999996,"15min":0.871},"min":{"1min":0.35599999999999998,"5min":0.35199999999999998,"15min":0.34399999999999997},"max":{"1min":1.1299999999999999,"5min":1.758,"15min":1.758},"last":0.56799999999999995}]},{"osd":4,"last update":"Mon May 22 15:46:21 2023","interfaces":[{"interface":"back","average":{"1min":0.92800000000000005,"5min":0.95099999999999996,"15min":0.95099999999999996},"min":{"1min":0.57599999999999996,"5min":0.57399999999999995,"15min":0.48099999999999998},"max":{"1min":1.149,"5min":2.923,"15min":2.923},"last":0.72799999999999998},{"interface":"front","average":{"1min":0.98999999999999999,"5min":1.006,"15min":1.026},"min":{"1min":0.624,"5min":0.624,"15min":0.624},"max":{"1min":1.3839999999999999,"5min":2.0129999999999999,"15min":2.0129999999999999},"last":0.879}]},{"osd":5,"last update":"Mon May 22 15:46:21 2023","interfaces":[{"interface":"back","average":{"1min":1,"5min":1.004,"15min":1.0409999999999999},"min":{"1min":0.59099999999999997,"5min":0.56000000000000005,"15min":0.46000000000000002},"max":{"1min":1.238,"5min":1.97,"15min":1.97},"last":0.71199999999999997},{"interface":"front","average":{"1min":1.02,"5min":1.016,"15min":1.008},"min":{"1min":0.67600000000000005,"5min":0.56599999999999995,"15min":0.55300000000000005},"max":{"1min":1.5349999999999999,"5min":5.0519999999999996,"15min":5.0519999999999996},"last":0.90100000000000002}]},{"osd":6,"last update":"Mon May 22 15:46:21 2023","interfaces":[{"interface":"back","average":{"1min":1.044,"5min":1.0589999999999999,"15min":1.089},"min":{"1min":0.58899999999999997,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":1.306,"5min":2.117,"15min":2.117},"last":0.74099999999999999},{"interface":"front","average":{"1min":1.085,"5min":1.1299999999999999,"15min":1.119},"min":{"1min":0.69199999999999995,"5min":0.65500000000000003,"15min":0.499},"max":{"1min":1.3100000000000001,"5min":5.2640000000000002,"15min":5.2640000000000002},"last":0.79700000000000004}]},{"osd":7,"last update":"Mon May 22 15:46:21 2023","interfaces":[{"interface":"back","average":{"1min":1.075,"5min":1.1040000000000001,"15min":1.129},"min":{"1min":0.63800000000000001,"5min":0.61299999999999999,"15min":0.61299999999999999},"max":{"1min":1.3640000000000001,"5min":2.0550000000000002,"15min":2.0550000000000002},"last":0.77300000000000002},{"interface":"front","average":{"1min":1.089,"5min":1.0900000000000001,"15min":1.0960000000000001},"min":{"1min":0.73099999999999998,"5min":0.63300000000000001,"15min":0.60299999999999998},"max":{"1min":1.6240000000000001,"5min":3.9060000000000001,"15min":3.9060000000000001},"last":0.68100000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2170,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-05-22T15:46:32.455 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-05-22T15:46:32.456 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2023-05-22T15:46:32.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:32.627+0000 7fbd34439640 1 Processor -- start 2023-05-22T15:46:32.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:32.627+0000 7fbd34439640 1 -- start start 2023-05-22T15:46:32.633 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:32.627+0000 7fbd34439640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbd2c0d2480 0x7fbd2c0d28d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:32.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:32.627+0000 7fbd34439640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fbd2c0b5340 0x7fbd2c0b57c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:32.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:32.627+0000 7fbd34439640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd2c0d2e10 con 0x7fbd2c15bca0 2023-05-22T15:46:32.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:32.627+0000 7fbd34439640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd2c0cd660 con 0x7fbd2c0b5340 2023-05-22T15:46:32.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:32.627+0000 7fbd34439640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd2c0ce190 con 0x7fbd2c0d2480 2023-05-22T15:46:32.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:32.627+0000 7fbd319d4640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbd2c0d2480 0x7fbd2c0d28d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:32.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:32.627+0000 7fbd329d6640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fbd2c0b5340 0x7fbd2c0b57c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:32.635 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:32.627+0000 7fbd319d4640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbd2c0d2480 0x7fbd2c0d28d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3301/0 says I am v2:172.21.15.16:46168/0 (socket says 172.21.15.16:46168) 2023-05-22T15:46:32.635 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:32.627+0000 7fbd319d4640 1 -- 172.21.15.16:0/3694693149 learned_addr learned my addr 172.21.15.16:0/3694693149 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:46:32.635 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:32.627+0000 7fbd319d4640 1 -- 172.21.15.16:0/3694693149 >> v2:172.21.15.62:3300/0 conn(0x7fbd2c0b5340 msgr2=0x7fbd2c0b57c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:46:32.635 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:32.627+0000 7fbd319d4640 1 --2- 172.21.15.16:0/3694693149 >> v2:172.21.15.62:3300/0 conn(0x7fbd2c0b5340 0x7fbd2c0b57c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:32.635 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:32.627+0000 7fbd319d4640 1 -- 172.21.15.16:0/3694693149 >> v1:172.21.15.62:6789/0 conn(0x7fbd2c15bca0 legacy=0x7fbd2c0cda80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:32.635 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:32.627+0000 7fbd319d4640 1 -- 172.21.15.16:0/3694693149 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd2c0cccc0 con 0x7fbd2c0d2480 2023-05-22T15:46:32.637 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:32.631+0000 7fbd319d4640 1 --2- 172.21.15.16:0/3694693149 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbd2c0d2480 0x7fbd2c0d28d0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fbd1c009870 tx=0x7fbd1c01c510 comp rx=0 tx=0).ready entity=mon.2 client_cookie=914b545f0bf5610a server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:32.638 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:32.631+0000 7fbd311d3640 1 -- 172.21.15.16:0/3694693149 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd1c0aa670 con 0x7fbd2c0d2480 2023-05-22T15:46:32.638 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:32.631+0000 7fbd311d3640 1 -- 172.21.15.16:0/3694693149 <== mon.2 v2:172.21.15.62:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd1c0aa7d0 con 0x7fbd2c0d2480 2023-05-22T15:46:32.639 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:32.631+0000 7fbd311d3640 1 -- 172.21.15.16:0/3694693149 <== mon.2 v2:172.21.15.62:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd1c0aab60 con 0x7fbd2c0d2480 2023-05-22T15:46:32.639 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:32.631+0000 7fbd34439640 1 -- 172.21.15.16:0/3694693149 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbd2c0d2480 msgr2=0x7fbd2c0d28d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:32.640 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:32.631+0000 7fbd34439640 1 --2- 172.21.15.16:0/3694693149 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbd2c0d2480 0x7fbd2c0d28d0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fbd1c009870 tx=0x7fbd1c01c510 comp rx=0 tx=0).stop 2023-05-22T15:46:32.640 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:32.631+0000 7fbd34439640 1 -- 172.21.15.16:0/3694693149 shutdown_connections 2023-05-22T15:46:32.640 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:32.631+0000 7fbd34439640 1 --2- 172.21.15.16:0/3694693149 >> v2:172.21.15.62:3300/0 conn(0x7fbd2c0b5340 0x7fbd2c0b57c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:32.641 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:32.631+0000 7fbd34439640 1 --2- 172.21.15.16:0/3694693149 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbd2c0d2480 0x7fbd2c0d28d0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:32.641 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:32.631+0000 7fbd34439640 1 -- 172.21.15.16:0/3694693149 >> 172.21.15.16:0/3694693149 conn(0x7fbd2c0655c0 msgr2=0x7fbd2c070e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:46:32.641 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:32.631+0000 7fbd34439640 1 -- 172.21.15.16:0/3694693149 shutdown_connections 2023-05-22T15:46:32.642 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:32.631+0000 7fbd34439640 1 -- 172.21.15.16:0/3694693149 wait complete. 2023-05-22T15:46:32.644 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:32.631+0000 7fbd34439640 1 Processor -- start 2023-05-22T15:46:32.644 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:32.631+0000 7fbd34439640 1 -- start start 2023-05-22T15:46:32.644 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:32.635+0000 7fbd34439640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbd2c0b5340 0x7fbd2c153bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:32.644 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:32.635+0000 7fbd34439640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbd2c0d2480 0x7fbd2c0c71a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:32.645 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:32.635+0000 7fbd34439640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbd2c15bca0 0x7fbd2c0b77b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:32.645 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:32.635+0000 7fbd34439640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd2c14b0c0 con 0x7fbd2c15bca0 2023-05-22T15:46:32.645 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:32.635+0000 7fbd34439640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd2c151d50 con 0x7fbd2c0d2480 2023-05-22T15:46:32.645 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:32.635+0000 7fbd34439640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd2c0c76e0 con 0x7fbd2c0b5340 2023-05-22T15:46:32.645 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:32.635+0000 7fbd329d6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbd2c15bca0 0x7fbd2c0b77b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:32.645 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:32.635+0000 7fbd329d6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbd2c15bca0 0x7fbd2c0b77b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56522/0 (socket says 172.21.15.16:56522) 2023-05-22T15:46:32.646 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:32.635+0000 7fbd329d6640 1 -- 172.21.15.16:0/2136984485 learned_addr learned my addr 172.21.15.16:0/2136984485 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:46:32.646 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:32.635+0000 7fbd319d4640 1 --2- 172.21.15.16:0/2136984485 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbd2c0d2480 0x7fbd2c0c71a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:32.646 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:32.635+0000 7fbd321d5640 1 --2- 172.21.15.16:0/2136984485 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbd2c0b5340 0x7fbd2c153bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:32.646 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:32.635+0000 7fbd329d6640 1 -- 172.21.15.16:0/2136984485 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbd2c0b5340 msgr2=0x7fbd2c153bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:32.646 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:32.635+0000 7fbd329d6640 1 --2- 172.21.15.16:0/2136984485 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbd2c0b5340 0x7fbd2c153bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:32.646 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:32.635+0000 7fbd329d6640 1 -- 172.21.15.16:0/2136984485 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbd2c0d2480 msgr2=0x7fbd2c0c71a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:32.646 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:32.635+0000 7fbd329d6640 1 --2- 172.21.15.16:0/2136984485 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbd2c0d2480 0x7fbd2c0c71a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:32.647 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:32.635+0000 7fbd329d6640 1 -- 172.21.15.16:0/2136984485 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd1c0094d0 con 0x7fbd2c15bca0 2023-05-22T15:46:32.647 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:32.635+0000 7fbd329d6640 1 --2- 172.21.15.16:0/2136984485 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbd2c15bca0 0x7fbd2c0b77b0 secure :-1 s=READY pgs=1855 cs=0 l=1 rev1=1 crypto rx=0x7fbd2400efc0 tx=0x7fbd2400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:32.647 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:32.635+0000 7fbd22ffd640 1 -- 172.21.15.16:0/2136984485 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd24014480 con 0x7fbd2c15bca0 2023-05-22T15:46:32.648 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:32.635+0000 7fbd22ffd640 1 -- 172.21.15.16:0/2136984485 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd2400f040 con 0x7fbd2c15bca0 2023-05-22T15:46:32.648 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:32.635+0000 7fbd34439640 1 -- 172.21.15.16:0/2136984485 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd2c0b5d00 con 0x7fbd2c15bca0 2023-05-22T15:46:32.648 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:32.635+0000 7fbd22ffd640 1 -- 172.21.15.16:0/2136984485 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd24014820 con 0x7fbd2c15bca0 2023-05-22T15:46:32.649 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:32.635+0000 7fbd34439640 1 -- 172.21.15.16:0/2136984485 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd2c15c070 con 0x7fbd2c15bca0 2023-05-22T15:46:32.649 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:32.635+0000 7fbd34439640 1 -- 172.21.15.16:0/2136984485 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd2c160360 con 0x7fbd2c15bca0 2023-05-22T15:46:32.649 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:32.635+0000 7fbd22ffd640 1 -- 172.21.15.16:0/2136984485 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 60) v1 ==== 84332+0+0 (secure 0 0 0) 0x7fbd24014980 con 0x7fbd2c15bca0 2023-05-22T15:46:32.649 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:32.639+0000 7fbd22ffd640 1 --2- 172.21.15.16:0/2136984485 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fbd100654a0 0x7fbd10067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:32.650 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:32.639+0000 7fbd22ffd640 1 -- 172.21.15.16:0/2136984485 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(147..147 src has 1..147) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fbd24083ea0 con 0x7fbd2c15bca0 2023-05-22T15:46:32.650 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:32.639+0000 7fbd321d5640 1 --2- 172.21.15.16:0/2136984485 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fbd100654a0 0x7fbd10067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:32.650 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:32.639+0000 7fbd321d5640 1 --2- 172.21.15.16:0/2136984485 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fbd100654a0 0x7fbd10067950 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fbd28007fd0 tx=0x7fbd28009040 comp rx=0 tx=0).ready entity=mgr.8835 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:32.651 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:32.639+0000 7fbd22ffd640 1 -- 172.21.15.16:0/2136984485 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fbd24051840 con 0x7fbd2c15bca0 2023-05-22T15:46:32.859 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:32.851+0000 7fbd34439640 1 -- 172.21.15.16:0/2136984485 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7fbd2c151f10 con 0x7fbd2c15bca0 2023-05-22T15:46:32.866 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:32.859+0000 7fbd22ffd640 1 -- 172.21.15.16:0/2136984485 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v25) v1 ==== 190+0+0 (secure 0 0 0) 0x7fbd24018020 con 0x7fbd2c15bca0 2023-05-22T15:46:32.866 INFO:teuthology.orchestra.run.smithi016.stderr:updated caps for client.0 2023-05-22T15:46:32.868 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:32.863+0000 7fbd34439640 1 -- 172.21.15.16:0/2136984485 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fbd100654a0 msgr2=0x7fbd10067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:32.868 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:32.863+0000 7fbd34439640 1 --2- 172.21.15.16:0/2136984485 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fbd100654a0 0x7fbd10067950 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fbd28007fd0 tx=0x7fbd28009040 comp rx=0 tx=0).stop 2023-05-22T15:46:32.869 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:32.863+0000 7fbd34439640 1 -- 172.21.15.16:0/2136984485 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbd2c15bca0 msgr2=0x7fbd2c0b77b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:32.869 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:32.863+0000 7fbd34439640 1 --2- 172.21.15.16:0/2136984485 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbd2c15bca0 0x7fbd2c0b77b0 secure :-1 s=READY pgs=1855 cs=0 l=1 rev1=1 crypto rx=0x7fbd2400efc0 tx=0x7fbd2400c490 comp rx=0 tx=0).stop 2023-05-22T15:46:32.869 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:32.863+0000 7fbd34439640 1 -- 172.21.15.16:0/2136984485 shutdown_connections 2023-05-22T15:46:32.869 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:32.863+0000 7fbd34439640 1 --2- 172.21.15.16:0/2136984485 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbd2c15bca0 0x7fbd2c0b77b0 unknown :-1 s=CLOSED pgs=1855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:32.869 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:32.863+0000 7fbd34439640 1 --2- 172.21.15.16:0/2136984485 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbd2c0d2480 0x7fbd2c0c71a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:32.870 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:32.863+0000 7fbd34439640 1 --2- 172.21.15.16:0/2136984485 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fbd100654a0 0x7fbd10067950 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:32.870 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:32.863+0000 7fbd34439640 1 --2- 172.21.15.16:0/2136984485 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbd2c0b5340 0x7fbd2c153bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:32.870 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:32.863+0000 7fbd34439640 1 -- 172.21.15.16:0/2136984485 >> 172.21.15.16:0/2136984485 conn(0x7fbd2c0655c0 msgr2=0x7fbd2c0cda80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:46:32.870 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:32.863+0000 7fbd34439640 1 -- 172.21.15.16:0/2136984485 shutdown_connections 2023-05-22T15:46:32.870 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:32.863+0000 7fbd34439640 1 -- 172.21.15.16:0/2136984485 wait complete. 2023-05-22T15:46:32.886 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-05-22T15:46:33.003 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:32.999+0000 7f88b929a640 1 Processor -- start 2023-05-22T15:46:33.003 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:32.999+0000 7f88b929a640 1 -- start start 2023-05-22T15:46:33.004 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:32.999+0000 7f88b929a640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f88b4070f10 0x7f88b40712e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:33.005 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:32.999+0000 7f88b929a640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88b40cedd0 con 0x7f88b415e470 2023-05-22T15:46:33.005 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:32.999+0000 7f88b929a640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88b4071820 con 0x7f88b40ca080 2023-05-22T15:46:33.005 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:32.999+0000 7f88b929a640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88b415a070 con 0x7f88b4070f10 2023-05-22T15:46:33.005 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:32.999+0000 7f88b37fe640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f88b415e470 0x7f88b40ce6c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:54900/0 (socket says 172.21.15.16:54900) 2023-05-22T15:46:33.005 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:32.999+0000 7f88b37fe640 1 -- 172.21.15.16:0/3159374953 learned_addr learned my addr 172.21.15.16:0/3159374953 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:46:33.005 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:32.999+0000 7f88b2ffd640 1 --2- 172.21.15.16:0/3159374953 >> v2:172.21.15.62:3300/0 conn(0x7f88b4070f10 0x7f88b40712e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:33.007 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:32.999+0000 7f88b27fc640 1 -- 172.21.15.16:0/3159374953 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2582903533 0 0) 0x7f88b40cedd0 con 0x7f88b415e470 2023-05-22T15:46:33.008 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:32.999+0000 7f88b27fc640 1 -- 172.21.15.16:0/3159374953 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f88a0003530 con 0x7f88b415e470 2023-05-22T15:46:33.008 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:32.999+0000 7f88b27fc640 1 -- 172.21.15.16:0/3159374953 <== mon.1 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1842812550 0 0) 0x7f88b4071820 con 0x7f88b40ca080 2023-05-22T15:46:33.008 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:32.999+0000 7f88b27fc640 1 -- 172.21.15.16:0/3159374953 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f88b40cedd0 con 0x7f88b40ca080 2023-05-22T15:46:33.008 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:32.999+0000 7f88b27fc640 1 -- 172.21.15.16:0/3159374953 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3948945714 0 0) 0x7f88a0003530 con 0x7f88b415e470 2023-05-22T15:46:33.008 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:32.999+0000 7f88b27fc640 1 -- 172.21.15.16:0/3159374953 >> v2:172.21.15.62:3300/0 conn(0x7f88b4070f10 msgr2=0x7f88b40712e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:46:33.008 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:32.999+0000 7f88b27fc640 1 --2- 172.21.15.16:0/3159374953 >> v2:172.21.15.62:3300/0 conn(0x7f88b4070f10 0x7f88b40712e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:33.008 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:32.999+0000 7f88b27fc640 1 -- 172.21.15.16:0/3159374953 >> v1:172.21.15.62:6789/0 conn(0x7f88b40ca080 legacy=0x7f88b40cdfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:33.009 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:32.999+0000 7f88b27fc640 1 -- 172.21.15.16:0/3159374953 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88b40ca580 con 0x7f88b415e470 2023-05-22T15:46:33.009 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:32.999+0000 7f88b27fc640 1 -- 172.21.15.16:0/3159374953 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7f88ac001980 con 0x7f88b415e470 2023-05-22T15:46:33.009 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:32.999+0000 7f88b27fc640 1 -- 172.21.15.16:0/3159374953 >> v1:172.21.15.16:6789/0 conn(0x7f88b415e470 legacy=0x7f88b40ce6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:33.009 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:32.999+0000 7f88b27fc640 1 --2- 172.21.15.16:0/3159374953 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f88a0003b60 0x7f88a0003f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:33.009 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:32.999+0000 7f88b27fc640 1 --2- 172.21.15.16:0/3159374953 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f88a00047d0 0x7f88a0004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:33.010 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:32.999+0000 7f88b27fc640 1 --2- 172.21.15.16:0/3159374953 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f88a0005180 0x7f88a000b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:33.010 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:32.999+0000 7f88b27fc640 1 -- 172.21.15.16:0/3159374953 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88b415a070 con 0x7f88a00047d0 2023-05-22T15:46:33.010 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:32.999+0000 7f88b27fc640 1 -- 172.21.15.16:0/3159374953 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88a000c060 con 0x7f88a0003b60 2023-05-22T15:46:33.010 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:32.999+0000 7f88b27fc640 1 -- 172.21.15.16:0/3159374953 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88a000c1f0 con 0x7f88a0005180 2023-05-22T15:46:33.010 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:32.999+0000 7f88abfff640 1 --2- 172.21.15.16:0/3159374953 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f88a00047d0 0x7f88a0004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:33.010 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:32.999+0000 7f88b2ffd640 1 --2- 172.21.15.16:0/3159374953 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f88a0003b60 0x7f88a0003f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:33.010 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:32.999+0000 7f88b37fe640 1 --2- 172.21.15.16:0/3159374953 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f88a0005180 0x7f88a000b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:33.011 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:32.999+0000 7f88abfff640 1 -- 172.21.15.16:0/3159374953 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f88a0005180 msgr2=0x7f88a000b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:46:33.011 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:32.999+0000 7f88abfff640 1 --2- 172.21.15.16:0/3159374953 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f88a0005180 0x7f88a000b990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:33.011 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:32.999+0000 7f88abfff640 1 -- 172.21.15.16:0/3159374953 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f88a0003b60 msgr2=0x7f88a0003f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:46:33.011 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:32.999+0000 7f88abfff640 1 --2- 172.21.15.16:0/3159374953 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f88a0003b60 0x7f88a0003f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:33.011 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:32.999+0000 7f88abfff640 1 -- 172.21.15.16:0/3159374953 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88a0003530 con 0x7f88a00047d0 2023-05-22T15:46:33.011 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:32.999+0000 7f88b37fe640 1 -- 172.21.15.16:0/3159374953 reap_dead start 2023-05-22T15:46:33.014 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.003+0000 7f88abfff640 1 --2- 172.21.15.16:0/3159374953 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f88a00047d0 0x7f88a0004c40 secure :-1 s=READY pgs=1857 cs=0 l=1 rev1=1 crypto rx=0x7f889c007fd0 tx=0x7f889c01ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2e8badb4ee82140 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:33.014 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.003+0000 7f88b27fc640 1 -- 172.21.15.16:0/3159374953 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f889c0052d0 con 0x7f88a00047d0 2023-05-22T15:46:33.014 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.003+0000 7f88b27fc640 1 -- 172.21.15.16:0/3159374953 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f889c005450 con 0x7f88a00047d0 2023-05-22T15:46:33.015 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.003+0000 7f88b27fc640 1 -- 172.21.15.16:0/3159374953 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f889c0a88d0 con 0x7f88a00047d0 2023-05-22T15:46:33.015 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.003+0000 7f88b929a640 1 -- 172.21.15.16:0/3159374953 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f88a00047d0 msgr2=0x7f88a0004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:33.015 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.003+0000 7f88b929a640 1 --2- 172.21.15.16:0/3159374953 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f88a00047d0 0x7f88a0004c40 secure :-1 s=READY pgs=1857 cs=0 l=1 rev1=1 crypto rx=0x7f889c007fd0 tx=0x7f889c01ec70 comp rx=0 tx=0).stop 2023-05-22T15:46:33.015 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.003+0000 7f88b929a640 1 -- 172.21.15.16:0/3159374953 shutdown_connections 2023-05-22T15:46:33.015 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.003+0000 7f88b929a640 1 --2- 172.21.15.16:0/3159374953 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f88a00047d0 0x7f88a0004c40 unknown :-1 s=CLOSED pgs=1857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:33.015 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.003+0000 7f88b929a640 1 -- 172.21.15.16:0/3159374953 >> 172.21.15.16:0/3159374953 conn(0x7f88b4069720 msgr2=0x7f88b4069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:46:33.015 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.003+0000 7f88b929a640 1 -- 172.21.15.16:0/3159374953 shutdown_connections 2023-05-22T15:46:33.016 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.003+0000 7f88b929a640 1 -- 172.21.15.16:0/3159374953 wait complete. 2023-05-22T15:46:33.016 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.003+0000 7f88b929a640 1 Processor -- start 2023-05-22T15:46:33.016 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.003+0000 7f88b929a640 1 -- start start 2023-05-22T15:46:33.016 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.007+0000 7f88b929a640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f88a00047d0 0x7f88b40b1d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:33.016 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.007+0000 7f88b929a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f88b414e420 0x7f88b4153600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:33.016 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.007+0000 7f88b929a640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f88b414e870 0x7f88b4153b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:33.018 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.007+0000 7f88b929a640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88b40b1880 con 0x7f88b414e420 2023-05-22T15:46:33.018 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.007+0000 7f88b929a640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88b40b0f50 con 0x7f88a00047d0 2023-05-22T15:46:33.018 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.007+0000 7f88b929a640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88b40b1230 con 0x7f88b414e870 2023-05-22T15:46:33.018 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.007+0000 7f88abfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f88b414e420 0x7f88b4153600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:33.018 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.007+0000 7f88abfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f88b414e420 0x7f88b4153600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56542/0 (socket says 172.21.15.16:56542) 2023-05-22T15:46:33.018 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.007+0000 7f88abfff640 1 -- 172.21.15.16:0/63709382 learned_addr learned my addr 172.21.15.16:0/63709382 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:46:33.019 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.007+0000 7f88b2ffd640 1 --2- 172.21.15.16:0/63709382 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f88a00047d0 0x7f88b40b1d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:33.019 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.007+0000 7f88b37fe640 1 --2- 172.21.15.16:0/63709382 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f88b414e870 0x7f88b4153b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:33.019 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.007+0000 7f88abfff640 1 -- 172.21.15.16:0/63709382 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f88b414e870 msgr2=0x7f88b4153b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:33.019 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.007+0000 7f88abfff640 1 --2- 172.21.15.16:0/63709382 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f88b414e870 0x7f88b4153b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:33.019 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.007+0000 7f88abfff640 1 -- 172.21.15.16:0/63709382 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f88a00047d0 msgr2=0x7f88b40b1d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:33.019 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.007+0000 7f88abfff640 1 --2- 172.21.15.16:0/63709382 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f88a00047d0 0x7f88b40b1d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:33.019 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.007+0000 7f88abfff640 1 -- 172.21.15.16:0/63709382 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88b415dc70 con 0x7f88b414e420 2023-05-22T15:46:33.019 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.007+0000 7f88abfff640 1 --2- 172.21.15.16:0/63709382 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f88b414e420 0x7f88b4153600 secure :-1 s=READY pgs=1858 cs=0 l=1 rev1=1 crypto rx=0x7f889c000c10 tx=0x7f88a0005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:33.020 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.007+0000 7f88ab7fe640 1 -- 172.21.15.16:0/63709382 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f889c0a8d10 con 0x7f88b414e420 2023-05-22T15:46:33.020 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.007+0000 7f88ab7fe640 1 -- 172.21.15.16:0/63709382 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f889c0a8e70 con 0x7f88b414e420 2023-05-22T15:46:33.020 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.007+0000 7f88b929a640 1 -- 172.21.15.16:0/63709382 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88b415d990 con 0x7f88b414e420 2023-05-22T15:46:33.020 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.007+0000 7f88ab7fe640 1 -- 172.21.15.16:0/63709382 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f889c0052f0 con 0x7f88b414e420 2023-05-22T15:46:33.020 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.007+0000 7f88b929a640 1 -- 172.21.15.16:0/63709382 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88b415df50 con 0x7f88b414e420 2023-05-22T15:46:33.020 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.007+0000 7f88b929a640 1 -- 172.21.15.16:0/63709382 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88b40b1370 con 0x7f88b414e420 2023-05-22T15:46:33.021 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.007+0000 7f88ab7fe640 1 -- 172.21.15.16:0/63709382 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 60) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f889c0067e0 con 0x7f88b414e420 2023-05-22T15:46:33.021 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.007+0000 7f88ab7fe640 1 --2- 172.21.15.16:0/63709382 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f88940654a0 0x7f8894067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:33.021 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.007+0000 7f88b2ffd640 1 --2- 172.21.15.16:0/63709382 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f88940654a0 0x7f8894067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:33.022 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.007+0000 7f88ab7fe640 1 -- 172.21.15.16:0/63709382 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(147..147 src has 1..147) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f889c11fc40 con 0x7f88b414e420 2023-05-22T15:46:33.022 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.007+0000 7f88b2ffd640 1 --2- 172.21.15.16:0/63709382 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f88940654a0 0x7f8894067950 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f88a40099d0 tx=0x7f88a4008040 comp rx=0 tx=0).ready entity=mgr.8835 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:33.022 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.011+0000 7f88ab7fe640 1 -- 172.21.15.16:0/63709382 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f88b40b1370 con 0x7f88b414e420 2023-05-22T15:46:33.199 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.191+0000 7f88b929a640 1 -- 172.21.15.16:0/63709382 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f88b40b1370 con 0x7f88b414e420 2023-05-22T15:46:33.200 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.195+0000 7f88ab7fe640 1 -- 172.21.15.16:0/63709382 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 58 v58) v1 ==== 94+0+4650 (secure 0 0 0) 0x7f88b40b1370 con 0x7f88b414e420 2023-05-22T15:46:33.201 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:46:33.201 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":58,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7753,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6834/3192074681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":3192074681},{"type":"v1","addr":"172.21.15.62:6834","nonce":3192074681}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":53},{"gid":7756,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/1344243469","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6833","nonce":1344243469},{"type":"v1","addr":"172.21.15.62:6835","nonce":1344243469}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":53},{"gid":9069,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/3636102434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":3636102434},{"type":"v1","addr":"172.21.15.16:6835","nonce":3636102434}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":53}],"filesystems":[{"mdsmap":{"epoch":58,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-05-22T15:46:23.443662+0000","modified":"2023-05-22T15:46:26.557591+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9075},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9075":{"gid":9075,"name":"a","rank":0,"incarnation":54,"state":"up:active","state_seq":3,"addr":"172.21.15.16:6837/2858899325","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":2858899325},{"type":"v1","addr":"172.21.15.16:6837","nonce":2858899325}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2023-05-22T15:46:33.202 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 58 2023-05-22T15:46:33.203 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.195+0000 7f88b929a640 1 -- 172.21.15.16:0/63709382 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f88940654a0 msgr2=0x7f8894067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:33.203 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.195+0000 7f88b929a640 1 --2- 172.21.15.16:0/63709382 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f88940654a0 0x7f8894067950 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f88a40099d0 tx=0x7f88a4008040 comp rx=0 tx=0).stop 2023-05-22T15:46:33.203 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.195+0000 7f88b929a640 1 -- 172.21.15.16:0/63709382 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f88b414e420 msgr2=0x7f88b4153600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:33.204 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.195+0000 7f88b929a640 1 --2- 172.21.15.16:0/63709382 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f88b414e420 0x7f88b4153600 secure :-1 s=READY pgs=1858 cs=0 l=1 rev1=1 crypto rx=0x7f889c000c10 tx=0x7f88a0005800 comp rx=0 tx=0).stop 2023-05-22T15:46:33.204 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.195+0000 7f88b929a640 1 -- 172.21.15.16:0/63709382 shutdown_connections 2023-05-22T15:46:33.204 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.195+0000 7f88b929a640 1 --2- 172.21.15.16:0/63709382 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f88940654a0 0x7f8894067950 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:33.205 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.195+0000 7f88b929a640 1 --2- 172.21.15.16:0/63709382 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f88b414e870 0x7f88b4153b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:33.205 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.195+0000 7f88b929a640 1 --2- 172.21.15.16:0/63709382 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f88b414e420 0x7f88b4153600 unknown :-1 s=CLOSED pgs=1858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:33.205 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.195+0000 7f88b929a640 1 --2- 172.21.15.16:0/63709382 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f88a00047d0 0x7f88b40b1d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:33.205 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.195+0000 7f88b929a640 1 -- 172.21.15.16:0/63709382 >> 172.21.15.16:0/63709382 conn(0x7f88b4069720 msgr2=0x7f88b40c50f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:46:33.206 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.195+0000 7f88b929a640 1 -- 172.21.15.16:0/63709382 shutdown_connections 2023-05-22T15:46:33.206 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.195+0000 7f88b929a640 1 -- 172.21.15.16:0/63709382 wait complete. 2023-05-22T15:46:33.620 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 58, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-05-22T15:46:23.443662+0000', 'modified': '2023-05-22T15:46:26.557591+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 9075}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9075': {'gid': 9075, 'name': 'a', 'rank': 0, 'incarnation': 54, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.16:6837/2858899325', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6836', 'nonce': 2858899325}, {'type': 'v1', 'addr': '172.21.15.16:6837', 'nonce': 2858899325}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-05-22T15:46:33.621 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-05-22T15:46:33.621 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2023-05-22T15:46:33.756 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.751+0000 7f4badc7a640 1 Processor -- start 2023-05-22T15:46:33.756 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.751+0000 7f4badc7a640 1 -- start start 2023-05-22T15:46:33.758 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.751+0000 7f4badc7a640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f4ba806cdf0 0x7f4ba8070be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:33.759 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.751+0000 7f4badc7a640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ba806eb90 con 0x7f4ba8158a10 2023-05-22T15:46:33.759 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.751+0000 7f4badc7a640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ba8158de0 con 0x7f4ba8070760 2023-05-22T15:46:33.759 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.751+0000 7f4badc7a640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ba806e060 con 0x7f4ba806cdf0 2023-05-22T15:46:33.759 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.751+0000 7f4ba77fe640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f4ba8158a10 0x7f4ba806e480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:54910/0 (socket says 172.21.15.16:54910) 2023-05-22T15:46:33.759 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.751+0000 7f4ba77fe640 1 -- 172.21.15.16:0/2315109473 learned_addr learned my addr 172.21.15.16:0/2315109473 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:46:33.759 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.751+0000 7f4ba7fff640 1 --2- 172.21.15.16:0/2315109473 >> v2:172.21.15.62:3300/0 conn(0x7f4ba806cdf0 0x7f4ba8070be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:33.759 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.751+0000 7f4ba67fc640 1 -- 172.21.15.16:0/2315109473 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2978680778 0 0) 0x7f4ba806eb90 con 0x7f4ba8158a10 2023-05-22T15:46:33.760 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.751+0000 7f4ba67fc640 1 -- 172.21.15.16:0/2315109473 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b94003530 con 0x7f4ba8158a10 2023-05-22T15:46:33.760 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.751+0000 7f4ba67fc640 1 -- 172.21.15.16:0/2315109473 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 863847824 0 0) 0x7f4b94003530 con 0x7f4ba8158a10 2023-05-22T15:46:33.760 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.751+0000 7f4ba67fc640 1 -- 172.21.15.16:0/2315109473 >> v2:172.21.15.62:3300/0 conn(0x7f4ba806cdf0 msgr2=0x7f4ba8070be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:46:33.760 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.751+0000 7f4ba67fc640 1 --2- 172.21.15.16:0/2315109473 >> v2:172.21.15.62:3300/0 conn(0x7f4ba806cdf0 0x7f4ba8070be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:33.760 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.751+0000 7f4ba67fc640 1 -- 172.21.15.16:0/2315109473 >> v1:172.21.15.62:6789/0 conn(0x7f4ba8070760 legacy=0x7f4ba806c6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:33.760 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.751+0000 7f4ba67fc640 1 -- 172.21.15.16:0/2315109473 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ba80711e0 con 0x7f4ba8158a10 2023-05-22T15:46:33.760 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.751+0000 7f4ba67fc640 1 -- 172.21.15.16:0/2315109473 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7f4ba0001680 con 0x7f4ba8158a10 2023-05-22T15:46:33.761 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.751+0000 7f4ba67fc640 1 -- 172.21.15.16:0/2315109473 >> v1:172.21.15.16:6789/0 conn(0x7f4ba8158a10 legacy=0x7f4ba806e480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:33.761 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.751+0000 7f4ba67fc640 1 --2- 172.21.15.16:0/2315109473 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4b94003980 0x7f4b94003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:33.761 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.751+0000 7f4ba67fc640 1 --2- 172.21.15.16:0/2315109473 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b940047d0 0x7f4b94004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:33.761 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.751+0000 7f4ba67fc640 1 --2- 172.21.15.16:0/2315109473 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4b94005180 0x7f4b9400b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:33.761 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.751+0000 7f4ba67fc640 1 -- 172.21.15.16:0/2315109473 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ba806e060 con 0x7f4b940047d0 2023-05-22T15:46:33.761 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.751+0000 7f4ba67fc640 1 -- 172.21.15.16:0/2315109473 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b9400c090 con 0x7f4b94003980 2023-05-22T15:46:33.761 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.751+0000 7f4ba67fc640 1 -- 172.21.15.16:0/2315109473 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b9400c250 con 0x7f4b94005180 2023-05-22T15:46:33.761 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.751+0000 7f4ba6ffd640 1 --2- 172.21.15.16:0/2315109473 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b940047d0 0x7f4b94004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:33.762 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.751+0000 7f4ba77fe640 1 --2- 172.21.15.16:0/2315109473 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4b94003980 0x7f4b94003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:33.762 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.751+0000 7f4ba7fff640 1 --2- 172.21.15.16:0/2315109473 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4b94005180 0x7f4b9400b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:33.763 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.755+0000 7f4ba6ffd640 1 -- 172.21.15.16:0/2315109473 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4b94005180 msgr2=0x7f4b9400b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:46:33.763 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.755+0000 7f4ba6ffd640 1 --2- 172.21.15.16:0/2315109473 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4b94005180 0x7f4b9400b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:33.763 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.755+0000 7f4ba6ffd640 1 -- 172.21.15.16:0/2315109473 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4b94003980 msgr2=0x7f4b94003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:46:33.764 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.755+0000 7f4ba6ffd640 1 --2- 172.21.15.16:0/2315109473 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4b94003980 0x7f4b94003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:33.764 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.755+0000 7f4ba6ffd640 1 -- 172.21.15.16:0/2315109473 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b94003530 con 0x7f4b940047d0 2023-05-22T15:46:33.764 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.755+0000 7f4ba7fff640 1 --2- 172.21.15.16:0/2315109473 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4b94005180 0x7f4b9400b990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T15:46:33.764 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.755+0000 7f4ba7fff640 1 -- 172.21.15.16:0/2315109473 reap_dead start 2023-05-22T15:46:33.764 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.755+0000 7f4ba6ffd640 1 --2- 172.21.15.16:0/2315109473 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b940047d0 0x7f4b94004c40 secure :-1 s=READY pgs=1860 cs=0 l=1 rev1=1 crypto rx=0x7f4b90001780 tx=0x7f4b9001c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ce826caa6f7842c server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:33.764 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.755+0000 7f4ba67fc640 1 -- 172.21.15.16:0/2315109473 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b900a4ea0 con 0x7f4b940047d0 2023-05-22T15:46:33.764 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.755+0000 7f4ba67fc640 1 -- 172.21.15.16:0/2315109473 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b900a7050 con 0x7f4b940047d0 2023-05-22T15:46:33.765 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.755+0000 7f4ba67fc640 1 -- 172.21.15.16:0/2315109473 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b90003600 con 0x7f4b940047d0 2023-05-22T15:46:33.765 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.755+0000 7f4badc7a640 1 -- 172.21.15.16:0/2315109473 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b940047d0 msgr2=0x7f4b94004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:33.765 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.755+0000 7f4badc7a640 1 --2- 172.21.15.16:0/2315109473 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b940047d0 0x7f4b94004c40 secure :-1 s=READY pgs=1860 cs=0 l=1 rev1=1 crypto rx=0x7f4b90001780 tx=0x7f4b9001c3e0 comp rx=0 tx=0).stop 2023-05-22T15:46:33.765 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.755+0000 7f4badc7a640 1 -- 172.21.15.16:0/2315109473 shutdown_connections 2023-05-22T15:46:33.765 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.755+0000 7f4badc7a640 1 --2- 172.21.15.16:0/2315109473 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b940047d0 0x7f4b94004c40 unknown :-1 s=CLOSED pgs=1860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:33.765 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.755+0000 7f4badc7a640 1 -- 172.21.15.16:0/2315109473 >> 172.21.15.16:0/2315109473 conn(0x7f4ba80cc870 msgr2=0x7f4ba80cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:46:33.765 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.755+0000 7f4badc7a640 1 -- 172.21.15.16:0/2315109473 shutdown_connections 2023-05-22T15:46:33.766 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.755+0000 7f4badc7a640 1 -- 172.21.15.16:0/2315109473 wait complete. 2023-05-22T15:46:33.766 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.759+0000 7f4badc7a640 1 Processor -- start 2023-05-22T15:46:33.766 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.759+0000 7f4badc7a640 1 -- start start 2023-05-22T15:46:33.767 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.759+0000 7f4badc7a640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4b940047d0 0x7f4ba80c6c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:33.768 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.759+0000 7f4badc7a640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4ba81513f0 0x7f4ba81e45e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:33.768 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.759+0000 7f4badc7a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ba80b2d10 0x7f4ba80c8630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:33.768 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.759+0000 7f4badc7a640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ba80c8230 con 0x7f4ba80b2d10 2023-05-22T15:46:33.768 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.759+0000 7f4badc7a640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ba80c7620 con 0x7f4ba81513f0 2023-05-22T15:46:33.768 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.759+0000 7f4badc7a640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ba80c7890 con 0x7f4b940047d0 2023-05-22T15:46:33.768 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.759+0000 7f4ba7fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ba80b2d10 0x7f4ba80c8630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:33.768 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.759+0000 7f4ba7fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ba80b2d10 0x7f4ba80c8630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56566/0 (socket says 172.21.15.16:56566) 2023-05-22T15:46:33.769 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.759+0000 7f4ba7fff640 1 -- 172.21.15.16:0/3665508396 learned_addr learned my addr 172.21.15.16:0/3665508396 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:46:33.769 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.759+0000 7f4ba6ffd640 1 --2- 172.21.15.16:0/3665508396 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4ba81513f0 0x7f4ba81e45e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:33.769 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.759+0000 7f4ba77fe640 1 --2- 172.21.15.16:0/3665508396 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4b940047d0 0x7f4ba80c6c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:33.769 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.759+0000 7f4ba7fff640 1 -- 172.21.15.16:0/3665508396 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4b940047d0 msgr2=0x7f4ba80c6c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:33.769 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.759+0000 7f4ba7fff640 1 --2- 172.21.15.16:0/3665508396 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4b940047d0 0x7f4ba80c6c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:33.769 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.759+0000 7f4ba7fff640 1 -- 172.21.15.16:0/3665508396 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4ba81513f0 msgr2=0x7f4ba81e45e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:33.769 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.759+0000 7f4ba7fff640 1 --2- 172.21.15.16:0/3665508396 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4ba81513f0 0x7f4ba81e45e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:33.770 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.759+0000 7f4ba7fff640 1 -- 172.21.15.16:0/3665508396 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ba806caf0 con 0x7f4ba80b2d10 2023-05-22T15:46:33.770 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.759+0000 7f4ba7fff640 1 --2- 172.21.15.16:0/3665508396 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ba80b2d10 0x7f4ba80c8630 secure :-1 s=READY pgs=1861 cs=0 l=1 rev1=1 crypto rx=0x7f4ba80b50b0 tx=0x7f4b98007e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:33.770 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.759+0000 7f4b9ffff640 1 -- 172.21.15.16:0/3665508396 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b9801b070 con 0x7f4ba80b2d10 2023-05-22T15:46:33.770 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.759+0000 7f4b9ffff640 1 -- 172.21.15.16:0/3665508396 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b9800f040 con 0x7f4ba80b2d10 2023-05-22T15:46:33.770 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.759+0000 7f4badc7a640 1 -- 172.21.15.16:0/3665508396 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ba80cfe40 con 0x7f4ba80b2d10 2023-05-22T15:46:33.770 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.759+0000 7f4b9ffff640 1 -- 172.21.15.16:0/3665508396 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b9800e050 con 0x7f4ba80b2d10 2023-05-22T15:46:33.770 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.759+0000 7f4badc7a640 1 -- 172.21.15.16:0/3665508396 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ba80ca240 con 0x7f4ba80b2d10 2023-05-22T15:46:33.770 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.759+0000 7f4b9ffff640 1 -- 172.21.15.16:0/3665508396 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 60) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f4b98015710 con 0x7f4ba80b2d10 2023-05-22T15:46:33.771 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.759+0000 7f4b9ffff640 1 --2- 172.21.15.16:0/3665508396 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f4b880654a0 0x7f4b88067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:33.771 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.759+0000 7f4b9ffff640 1 -- 172.21.15.16:0/3665508396 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(147..147 src has 1..147) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f4b98082ba0 con 0x7f4ba80b2d10 2023-05-22T15:46:33.771 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.763+0000 7f4ba77fe640 1 --2- 172.21.15.16:0/3665508396 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f4b880654a0 0x7f4b88067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:33.771 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.763+0000 7f4ba77fe640 1 --2- 172.21.15.16:0/3665508396 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f4b880654a0 0x7f4b88067950 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f4ba0000fc0 tx=0x7f4ba0009040 comp rx=0 tx=0).ready entity=mgr.8835 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:33.771 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.763+0000 7f4bb0d9c000 1 Processor -- start 2023-05-22T15:46:33.771 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.763+0000 7f4bb0d9c000 1 -- start start 2023-05-22T15:46:33.775 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.763+0000 7f4bb0d9c000 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55708fc5afd0 0x55708fecdc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:33.775 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.763+0000 7f4bb0d9c000 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x55708fece170 0x55708fece540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:33.775 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.763+0000 7f4bb0d9c000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55708fe81a10 0x55708fe81de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:33.776 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.763+0000 7f4bb0d9c000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55708fd4bbd0 con 0x55708fe81a10 2023-05-22T15:46:33.776 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.763+0000 7f4bb0d9c000 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55708fc68dd0 con 0x55708fc5afd0 2023-05-22T15:46:33.776 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.763+0000 7f4bb0d9c000 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x55708fcd9640 con 0x55708fece170 2023-05-22T15:46:33.776 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.763+0000 7f4ba7fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55708fe81a10 0x55708fe81de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:33.776 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.763+0000 7f4ba7fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55708fe81a10 0x55708fe81de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56580/0 (socket says 172.21.15.16:56580) 2023-05-22T15:46:33.776 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.763+0000 7f4ba7fff640 1 -- 172.21.15.16:0/2620274667 learned_addr learned my addr 172.21.15.16:0/2620274667 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:46:33.776 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.763+0000 7f4ba6ffd640 1 --2- 172.21.15.16:0/2620274667 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x55708fece170 0x55708fece540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:33.777 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.763+0000 7f4ba77fe640 1 --2- 172.21.15.16:0/2620274667 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55708fc5afd0 0x55708fecdc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:33.777 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.763+0000 7f4ba7fff640 1 -- 172.21.15.16:0/2620274667 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x55708fece170 msgr2=0x55708fece540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:46:33.777 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.763+0000 7f4ba7fff640 1 --2- 172.21.15.16:0/2620274667 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x55708fece170 0x55708fece540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:33.777 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.763+0000 7f4ba7fff640 1 -- 172.21.15.16:0/2620274667 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55708fc5afd0 msgr2=0x55708fecdc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:46:33.777 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.763+0000 7f4ba7fff640 1 --2- 172.21.15.16:0/2620274667 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55708fc5afd0 0x55708fecdc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:33.777 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.763+0000 7f4ba7fff640 1 -- 172.21.15.16:0/2620274667 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55708fc980b0 con 0x55708fe81a10 2023-05-22T15:46:33.777 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.763+0000 7f4ba7fff640 1 --2- 172.21.15.16:0/2620274667 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55708fe81a10 0x55708fe81de0 secure :-1 s=READY pgs=1862 cs=0 l=1 rev1=1 crypto rx=0x7f4b980102d0 tx=0x7f4b98052250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ebad1ba9b5357f60 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:33.777 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.763+0000 7f4b9d7fa640 1 -- 172.21.15.16:0/2620274667 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b98053600 con 0x55708fe81a10 2023-05-22T15:46:33.778 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.763+0000 7f4b9d7fa640 1 -- 172.21.15.16:0/2620274667 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b98053760 con 0x55708fe81a10 2023-05-22T15:46:33.778 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.763+0000 7f4b9d7fa640 1 -- 172.21.15.16:0/2620274667 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b980538d0 con 0x55708fe81a10 2023-05-22T15:46:33.778 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.763+0000 7f4bb0d9c000 1 -- 172.21.15.16:0/2620274667 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55708fe81a10 msgr2=0x55708fe81de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:33.778 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.763+0000 7f4bb0d9c000 1 --2- 172.21.15.16:0/2620274667 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55708fe81a10 0x55708fe81de0 secure :-1 s=READY pgs=1862 cs=0 l=1 rev1=1 crypto rx=0x7f4b980102d0 tx=0x7f4b98052250 comp rx=0 tx=0).stop 2023-05-22T15:46:33.778 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.763+0000 7f4bb0d9c000 1 -- 172.21.15.16:0/2620274667 shutdown_connections 2023-05-22T15:46:33.778 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.763+0000 7f4bb0d9c000 1 --2- 172.21.15.16:0/2620274667 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55708fe81a10 0x55708fe81de0 unknown :-1 s=CLOSED pgs=1862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:33.778 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.763+0000 7f4bb0d9c000 1 --2- 172.21.15.16:0/2620274667 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x55708fece170 0x55708fece540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:33.779 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.763+0000 7f4bb0d9c000 1 --2- 172.21.15.16:0/2620274667 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55708fc5afd0 0x55708fecdc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:33.779 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.763+0000 7f4bb0d9c000 1 -- 172.21.15.16:0/2620274667 >> 172.21.15.16:0/2620274667 conn(0x55708fd2a270 msgr2=0x55708feb1220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:46:33.779 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.763+0000 7f4bb0d9c000 1 -- 172.21.15.16:0/2620274667 shutdown_connections 2023-05-22T15:46:33.779 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.763+0000 7f4bb0d9c000 1 -- 172.21.15.16:0/2620274667 wait complete. 2023-05-22T15:46:33.779 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.763+0000 7f4bb0d9c000 1 -- start start 2023-05-22T15:46:33.779 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.763+0000 7f4bb0d9c000 1 Processor -- start 2023-05-22T15:46:33.779 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.767+0000 7f4bb0d9c000 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x55708fc5afd0 0x55708fed4ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:33.779 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.767+0000 7f4bb0d9c000 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55708fe81a10 0x55708fed50e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:33.780 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.767+0000 7f4bb0d9c000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55708fece170 0x55708feb2450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:33.780 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.767+0000 7f4bb0d9c000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55708fcd2780 con 0x55708fece170 2023-05-22T15:46:33.780 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.767+0000 7f4bb0d9c000 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55708fc90250 con 0x55708fe81a10 2023-05-22T15:46:33.780 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.767+0000 7f4bb0d9c000 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x55708fc8fef0 con 0x55708fc5afd0 2023-05-22T15:46:33.780 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.767+0000 7f4bb0d9c000 10 client.0 _ll_register_callbacks cb 0x55708fe6fb50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-05-22T15:46:33.780 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.767+0000 7f4ba7fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55708fece170 0x55708feb2450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:33.780 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.767+0000 7f4ba7fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55708fece170 0x55708feb2450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56586/0 (socket says 172.21.15.16:56586) 2023-05-22T15:46:33.781 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.767+0000 7f4ba7fff640 1 -- 172.21.15.16:0/3717728928 learned_addr learned my addr 172.21.15.16:0/3717728928 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:46:33.781 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.767+0000 7f4ba6ffd640 1 --2- 172.21.15.16:0/3717728928 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55708fe81a10 0x55708fed50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:33.781 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.767+0000 7f4ba77fe640 1 --2- 172.21.15.16:0/3717728928 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x55708fc5afd0 0x55708fed4ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:33.781 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.767+0000 7f4ba7fff640 1 -- 172.21.15.16:0/3717728928 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x55708fc5afd0 msgr2=0x55708fed4ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:46:33.781 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.767+0000 7f4ba7fff640 1 --2- 172.21.15.16:0/3717728928 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x55708fc5afd0 0x55708fed4ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:33.781 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.767+0000 7f4ba7fff640 1 -- 172.21.15.16:0/3717728928 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55708fe81a10 msgr2=0x55708fed50e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:46:33.781 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.767+0000 7f4ba7fff640 1 --2- 172.21.15.16:0/3717728928 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55708fe81a10 0x55708fed50e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:33.781 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.767+0000 7f4ba7fff640 1 -- 172.21.15.16:0/3717728928 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ba80ca240 con 0x55708fece170 2023-05-22T15:46:33.782 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.767+0000 7f4ba7fff640 1 --2- 172.21.15.16:0/3717728928 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55708fece170 0x55708feb2450 secure :-1 s=READY pgs=1863 cs=0 l=1 rev1=1 crypto rx=0x7f4b980540a0 tx=0x7f4b980098f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c68523b083c2fea4 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:33.782 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.767+0000 7f4b867fc640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2023-05-22T15:46:33.782 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.767+0000 7f4b867fc640 1 -- 172.21.15.16:0/3717728928 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b9804ff00 con 0x55708fece170 2023-05-22T15:46:33.782 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.767+0000 7f4b867fc640 1 -- 172.21.15.16:0/3717728928 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(147..147 src has 1..147) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f4b98050060 con 0x55708fece170 2023-05-22T15:46:33.782 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.767+0000 7f4bb0d9c000 1 -- 172.21.15.16:0/3717728928 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55708fc96c60 con 0x55708fece170 2023-05-22T15:46:33.782 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.767+0000 7f4b867fc640 1 -- 172.21.15.16:0/3717728928 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=58) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4b98050240 con 0x55708fece170 2023-05-22T15:46:33.782 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.767+0000 7f4bb0d9c000 10 client.9258 fetch_fsmap learned FSMap version 58 2023-05-22T15:46:33.782 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.767+0000 7f4bb0d9c000 1 -- 172.21.15.16:0/3717728928 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=58,monmap=2+}) v3 -- 0x55708f9cf890 con 0x55708fece170 2023-05-22T15:46:33.783 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.767+0000 7f4b867fc640 1 -- 172.21.15.16:0/3717728928 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55708fc96c60 con 0x55708fece170 2023-05-22T15:46:33.783 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.767+0000 7f4b867fc640 1 -- 172.21.15.16:0/3717728928 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 58) v1 ==== 3106+0+0 (secure 0 0 0) 0x7f4b98005d20 con 0x55708fece170 2023-05-22T15:46:33.783 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.767+0000 7f4bb0d9c000 10 client.9258 fetch_fsmap finished waiting for FSMap version 58 2023-05-22T15:46:33.783 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.767+0000 7f4bb0d9c000 10 client.9258 resolve_mds: resolved name 'a' to daemon mds.a 2023-05-22T15:46:33.783 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.767+0000 7f4bb0d9c000 20 client.9258 populate_metadata read hostname 'smithi016' 2023-05-22T15:46:33.783 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.767+0000 7f4bb0d9c000 1 --2- 172.21.15.16:0/3717728928 >> [v2:172.21.15.16:6836/2858899325,v1:172.21.15.16:6837/2858899325] conn(0x55708feb31f0 0x55708feb35c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:33.783 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.767+0000 7f4bb0d9c000 4 client.9258 mds_command: new command op to 9075 tid=0[{"prefix": "get_command_descriptions"}] 2023-05-22T15:46:33.784 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.767+0000 7f4bb0d9c000 1 -- 172.21.15.16:0/3717728928 --> [v2:172.21.15.16:6836/2858899325,v1:172.21.15.16:6837/2858899325] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55708fcd10d0 con 0x55708feb31f0 2023-05-22T15:46:33.784 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.767+0000 7f4ba77fe640 1 --2- 172.21.15.16:0/3717728928 >> [v2:172.21.15.16:6836/2858899325,v1:172.21.15.16:6837/2858899325] conn(0x55708feb31f0 0x55708feb35c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:33.784 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.767+0000 7f4ba77fe640 1 --2- 172.21.15.16:0/3717728928 >> [v2:172.21.15.16:6836/2858899325,v1:172.21.15.16:6837/2858899325] conn(0x55708feb31f0 0x55708feb35c0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3c26098959e0df3e server_cookie=7350a9069bd0ede8 in_seq=0 out_seq=0 2023-05-22T15:46:33.784 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.767+0000 7f4b867fc640 10 client.9258 ms_handle_connect on v2:172.21.15.16:6836/2858899325 2023-05-22T15:46:33.784 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.767+0000 7f4b867fc640 1 -- 172.21.15.16:0/3717728928 <== mds.0 v2:172.21.15.16:6836/2858899325 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x55708fcd10d0 con 0x55708feb31f0 2023-05-22T15:46:33.784 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.767+0000 7f4b867fc640 10 client.9258 handle_command_reply: tid=0 2023-05-22T15:46:33.784 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.767+0000 7f4bb0d9c000 1 client.9258 shutdown 2023-05-22T15:46:33.784 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.767+0000 7f4bb0d9c000 1 -- 172.21.15.16:0/3717728928 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55708fece170 msgr2=0x55708feb2450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:33.785 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.767+0000 7f4bb0d9c000 1 --2- 172.21.15.16:0/3717728928 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55708fece170 0x55708feb2450 secure :-1 s=READY pgs=1863 cs=0 l=1 rev1=1 crypto rx=0x7f4b980540a0 tx=0x7f4b980098f0 comp rx=0 tx=0).stop 2023-05-22T15:46:33.785 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.767+0000 7f4bb0d9c000 1 -- 172.21.15.16:0/3717728928 shutdown_connections 2023-05-22T15:46:33.785 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.767+0000 7f4bb0d9c000 1 --2- 172.21.15.16:0/3717728928 >> [v2:172.21.15.16:6836/2858899325,v1:172.21.15.16:6837/2858899325] conn(0x55708feb31f0 0x55708feb35c0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:33.785 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.767+0000 7f4bb0d9c000 1 --2- 172.21.15.16:0/3717728928 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55708fece170 0x55708feb2450 unknown :-1 s=CLOSED pgs=1863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:33.785 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.767+0000 7f4bb0d9c000 1 --2- 172.21.15.16:0/3717728928 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55708fe81a10 0x55708fed50e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:33.785 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.767+0000 7f4b867fc640 0 client.9258 ms_handle_reset on v2:172.21.15.16:6836/2858899325 2023-05-22T15:46:33.785 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.767+0000 7f4bb0d9c000 1 --2- 172.21.15.16:0/3717728928 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x55708fc5afd0 0x55708fed4ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:33.785 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.767+0000 7f4bb0d9c000 1 -- 172.21.15.16:0/3717728928 >> 172.21.15.16:0/3717728928 conn(0x55708fd2a270 msgr2=0x55708fe81100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:46:33.786 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.767+0000 7f4bb0d9c000 1 -- 172.21.15.16:0/3717728928 shutdown_connections 2023-05-22T15:46:33.786 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.767+0000 7f4bb0d9c000 1 -- 172.21.15.16:0/3717728928 wait complete. 2023-05-22T15:46:33.786 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.767+0000 7f4bb0d9c000 20 client.9258 trim_cache size 0 max 16384 2023-05-22T15:46:33.789 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.783+0000 7f4bb0d9c000 1 Processor -- start 2023-05-22T15:46:33.790 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.783+0000 7f4bb0d9c000 1 -- start start 2023-05-22T15:46:33.792 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.783+0000 7f4bb0d9c000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55708fc5afd0 0x55708fe84e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:33.792 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.783+0000 7f4bb0d9c000 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55708fe81a10 0x55708fe85340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:33.792 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.783+0000 7f4bb0d9c000 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x55708fece170 0x55708feaaee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:33.792 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.783+0000 7f4bb0d9c000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55708f990940 con 0x55708fc5afd0 2023-05-22T15:46:33.792 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.783+0000 7f4bb0d9c000 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55708fc8fef0 con 0x55708fe81a10 2023-05-22T15:46:33.793 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.783+0000 7f4bb0d9c000 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x55708fca2a10 con 0x55708fece170 2023-05-22T15:46:33.793 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.783+0000 7f4ba77fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55708fc5afd0 0x55708fe84e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:33.793 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.783+0000 7f4ba77fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55708fc5afd0 0x55708fe84e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56592/0 (socket says 172.21.15.16:56592) 2023-05-22T15:46:33.793 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.783+0000 7f4ba77fe640 1 -- 172.21.15.16:0/827963045 learned_addr learned my addr 172.21.15.16:0/827963045 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:46:33.793 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.783+0000 7f4ba6ffd640 1 --2- 172.21.15.16:0/827963045 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55708fe81a10 0x55708fe85340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:33.793 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.783+0000 7f4ba7fff640 1 --2- 172.21.15.16:0/827963045 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x55708fece170 0x55708feaaee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:33.793 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.783+0000 7f4ba77fe640 1 -- 172.21.15.16:0/827963045 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x55708fece170 msgr2=0x55708feaaee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:46:33.794 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.783+0000 7f4ba77fe640 1 --2- 172.21.15.16:0/827963045 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x55708fece170 0x55708feaaee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:33.794 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.783+0000 7f4ba77fe640 1 -- 172.21.15.16:0/827963045 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55708fe81a10 msgr2=0x55708fe85340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:46:33.794 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.783+0000 7f4ba77fe640 1 --2- 172.21.15.16:0/827963045 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55708fe81a10 0x55708fe85340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:33.794 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.783+0000 7f4ba77fe640 1 -- 172.21.15.16:0/827963045 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55708fc98360 con 0x55708fc5afd0 2023-05-22T15:46:33.794 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.783+0000 7f4ba7fff640 1 --2- 172.21.15.16:0/827963045 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x55708fece170 0x55708feaaee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:46:33.794 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.787+0000 7f4ba77fe640 1 --2- 172.21.15.16:0/827963045 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55708fc5afd0 0x55708fe84e00 secure :-1 s=READY pgs=1864 cs=0 l=1 rev1=1 crypto rx=0x7f4ba80ae5e0 tx=0x7f4ba0014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ccd0511fcdc6ede server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:33.794 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.787+0000 7f4b9dffb640 1 -- 172.21.15.16:0/827963045 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ba0029070 con 0x55708fc5afd0 2023-05-22T15:46:33.795 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.787+0000 7f4b9dffb640 1 -- 172.21.15.16:0/827963045 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ba0015a00 con 0x55708fc5afd0 2023-05-22T15:46:33.795 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.787+0000 7f4b9dffb640 1 -- 172.21.15.16:0/827963045 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ba0015bb0 con 0x55708fc5afd0 2023-05-22T15:46:33.795 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.787+0000 7f4bb0d9c000 1 -- 172.21.15.16:0/827963045 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55708fc5afd0 msgr2=0x55708fe84e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:33.795 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.787+0000 7f4bb0d9c000 1 --2- 172.21.15.16:0/827963045 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55708fc5afd0 0x55708fe84e00 secure :-1 s=READY pgs=1864 cs=0 l=1 rev1=1 crypto rx=0x7f4ba80ae5e0 tx=0x7f4ba0014000 comp rx=0 tx=0).stop 2023-05-22T15:46:33.795 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.787+0000 7f4bb0d9c000 1 -- 172.21.15.16:0/827963045 shutdown_connections 2023-05-22T15:46:33.795 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.787+0000 7f4bb0d9c000 1 --2- 172.21.15.16:0/827963045 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x55708fece170 0x55708feaaee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:33.795 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.787+0000 7f4bb0d9c000 1 --2- 172.21.15.16:0/827963045 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55708fe81a10 0x55708fe85340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:33.796 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.787+0000 7f4bb0d9c000 1 --2- 172.21.15.16:0/827963045 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55708fc5afd0 0x55708fe84e00 unknown :-1 s=CLOSED pgs=1864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:33.796 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.787+0000 7f4bb0d9c000 1 -- 172.21.15.16:0/827963045 >> 172.21.15.16:0/827963045 conn(0x55708fd2a270 msgr2=0x55708fe81080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:46:33.796 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.787+0000 7f4bb0d9c000 1 -- 172.21.15.16:0/827963045 shutdown_connections 2023-05-22T15:46:33.796 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.787+0000 7f4bb0d9c000 1 -- 172.21.15.16:0/827963045 wait complete. 2023-05-22T15:46:33.799 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.787+0000 7f4bb0d9c000 1 -- start start 2023-05-22T15:46:33.799 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.787+0000 7f4bb0d9c000 1 Processor -- start 2023-05-22T15:46:33.799 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.787+0000 7f4bb0d9c000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55708fc5afd0 0x55708ff291a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:33.800 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.787+0000 7f4bb0d9c000 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55708fe81a10 0x55708feae0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:33.800 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.787+0000 7f4bb0d9c000 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x55708fece170 0x55708feae5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:33.800 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.787+0000 7f4bb0d9c000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55708fedb620 con 0x55708fc5afd0 2023-05-22T15:46:33.800 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.787+0000 7f4bb0d9c000 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55708fccfb80 con 0x55708fe81a10 2023-05-22T15:46:33.800 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.787+0000 7f4bb0d9c000 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x55708fcf2860 con 0x55708fece170 2023-05-22T15:46:33.800 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.787+0000 7f4bb0d9c000 10 client.0 _ll_register_callbacks cb 0x55708fd22eb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-05-22T15:46:33.800 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.787+0000 7f4ba77fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55708fc5afd0 0x55708ff291a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:33.801 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.787+0000 7f4ba77fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55708fc5afd0 0x55708ff291a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56596/0 (socket says 172.21.15.16:56596) 2023-05-22T15:46:33.801 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.787+0000 7f4ba77fe640 1 -- 172.21.15.16:0/3987454061 learned_addr learned my addr 172.21.15.16:0/3987454061 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:46:33.801 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.787+0000 7f4ba6ffd640 1 --2- 172.21.15.16:0/3987454061 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55708fe81a10 0x55708feae0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:33.801 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.787+0000 7f4ba7fff640 1 --2- 172.21.15.16:0/3987454061 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x55708fece170 0x55708feae5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:33.801 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.787+0000 7f4ba77fe640 1 -- 172.21.15.16:0/3987454061 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x55708fece170 msgr2=0x55708feae5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:46:33.801 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.787+0000 7f4ba77fe640 1 --2- 172.21.15.16:0/3987454061 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x55708fece170 0x55708feae5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:33.802 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.787+0000 7f4ba77fe640 1 -- 172.21.15.16:0/3987454061 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55708fe81a10 msgr2=0x55708feae0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:46:33.802 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.787+0000 7f4ba77fe640 1 --2- 172.21.15.16:0/3987454061 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55708fe81a10 0x55708feae0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:33.802 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.787+0000 7f4ba77fe640 1 -- 172.21.15.16:0/3987454061 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ba80711e0 con 0x55708fc5afd0 2023-05-22T15:46:33.802 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.787+0000 7f4ba7fff640 1 --2- 172.21.15.16:0/3987454061 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x55708fece170 0x55708feae5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:46:33.802 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.787+0000 7f4ba77fe640 1 --2- 172.21.15.16:0/3987454061 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55708fc5afd0 0x55708ff291a0 secure :-1 s=READY pgs=1865 cs=0 l=1 rev1=1 crypto rx=0x7f4ba0024040 tx=0x7f4ba0000ff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56403f79cf0f2656 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:33.802 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.787+0000 7f4b867fc640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2023-05-22T15:46:33.802 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.787+0000 7f4b867fc640 1 -- 172.21.15.16:0/3987454061 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ba0027040 con 0x55708fc5afd0 2023-05-22T15:46:33.803 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.787+0000 7f4b867fc640 1 -- 172.21.15.16:0/3987454061 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(147..147 src has 1..147) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f4ba001a040 con 0x55708fc5afd0 2023-05-22T15:46:33.803 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.787+0000 7f4bb0d9c000 1 -- 172.21.15.16:0/3987454061 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55708fe881c0 con 0x55708fc5afd0 2023-05-22T15:46:33.803 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.787+0000 7f4b867fc640 1 -- 172.21.15.16:0/3987454061 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=58) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4ba002f8b0 con 0x55708fc5afd0 2023-05-22T15:46:33.803 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.787+0000 7f4bb0d9c000 10 client.9264 fetch_fsmap learned FSMap version 58 2023-05-22T15:46:33.803 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.787+0000 7f4bb0d9c000 1 -- 172.21.15.16:0/3987454061 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=58,monmap=2+}) v3 -- 0x55708fc7dbe0 con 0x55708fc5afd0 2023-05-22T15:46:33.803 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.787+0000 7f4b867fc640 1 -- 172.21.15.16:0/3987454061 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55708fe881c0 con 0x55708fc5afd0 2023-05-22T15:46:33.803 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.787+0000 7f4b867fc640 1 -- 172.21.15.16:0/3987454061 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 58) v1 ==== 3106+0+0 (secure 0 0 0) 0x7f4ba002fa10 con 0x55708fc5afd0 2023-05-22T15:46:33.804 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.787+0000 7f4bb0d9c000 10 client.9264 fetch_fsmap finished waiting for FSMap version 58 2023-05-22T15:46:33.804 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.787+0000 7f4bb0d9c000 10 client.9264 resolve_mds: resolved name 'a' to daemon mds.a 2023-05-22T15:46:33.804 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.787+0000 7f4bb0d9c000 20 client.9264 populate_metadata read hostname 'smithi016' 2023-05-22T15:46:33.804 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.787+0000 7f4bb0d9c000 1 --2- 172.21.15.16:0/3987454061 >> [v2:172.21.15.16:6836/2858899325,v1:172.21.15.16:6837/2858899325] conn(0x55708feaeb30 0x55708feaef00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:33.804 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.787+0000 7f4bb0d9c000 4 client.9264 mds_command: new command op to 9075 tid=0[{"prefix": "status"}] 2023-05-22T15:46:33.804 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.787+0000 7f4bb0d9c000 1 -- 172.21.15.16:0/3987454061 --> [v2:172.21.15.16:6836/2858899325,v1:172.21.15.16:6837/2858899325] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55708fc794d0 con 0x55708feaeb30 2023-05-22T15:46:33.804 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.791+0000 7f4ba6ffd640 1 --2- 172.21.15.16:0/3987454061 >> [v2:172.21.15.16:6836/2858899325,v1:172.21.15.16:6837/2858899325] conn(0x55708feaeb30 0x55708feaef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:33.805 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.791+0000 7f4ba6ffd640 1 --2- 172.21.15.16:0/3987454061 >> [v2:172.21.15.16:6836/2858899325,v1:172.21.15.16:6837/2858899325] conn(0x55708feaeb30 0x55708feaef00 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bcc018c7dc38509d server_cookie=3d824c7074c86c69 in_seq=0 out_seq=0 2023-05-22T15:46:33.805 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.791+0000 7f4b867fc640 10 client.9264 ms_handle_connect on v2:172.21.15.16:6836/2858899325 2023-05-22T15:46:33.805 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.791+0000 7f4b867fc640 1 -- 172.21.15.16:0/3987454061 <== mds.0 v2:172.21.15.16:6836/2858899325 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+321 (crc 0 0 0) 0x55708fc794d0 con 0x55708feaeb30 2023-05-22T15:46:33.805 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.791+0000 7f4b867fc640 10 client.9264 handle_command_reply: tid=0 2023-05-22T15:46:33.805 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.791+0000 7f4bb0d9c000 1 client.9264 shutdown 2023-05-22T15:46:33.805 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.791+0000 7f4bb0d9c000 1 -- 172.21.15.16:0/3987454061 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55708fc5afd0 msgr2=0x55708ff291a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:33.805 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.791+0000 7f4bb0d9c000 1 --2- 172.21.15.16:0/3987454061 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55708fc5afd0 0x55708ff291a0 secure :-1 s=READY pgs=1865 cs=0 l=1 rev1=1 crypto rx=0x7f4ba0024040 tx=0x7f4ba0000ff0 comp rx=0 tx=0).stop 2023-05-22T15:46:33.806 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.791+0000 7f4bb0d9c000 1 -- 172.21.15.16:0/3987454061 shutdown_connections 2023-05-22T15:46:33.806 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.791+0000 7f4bb0d9c000 1 --2- 172.21.15.16:0/3987454061 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x55708fece170 0x55708feae5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:33.806 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.791+0000 7f4bb0d9c000 1 --2- 172.21.15.16:0/3987454061 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55708fe81a10 0x55708feae0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:33.806 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.791+0000 7f4bb0d9c000 1 --2- 172.21.15.16:0/3987454061 >> [v2:172.21.15.16:6836/2858899325,v1:172.21.15.16:6837/2858899325] conn(0x55708feaeb30 0x55708feaef00 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:33.806 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.791+0000 7f4bb0d9c000 1 --2- 172.21.15.16:0/3987454061 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55708fc5afd0 0x55708ff291a0 unknown :-1 s=CLOSED pgs=1865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:33.806 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.791+0000 7f4b867fc640 0 client.9264 ms_handle_reset on v2:172.21.15.16:6836/2858899325 2023-05-22T15:46:33.806 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.791+0000 7f4bb0d9c000 1 -- 172.21.15.16:0/3987454061 >> 172.21.15.16:0/3987454061 conn(0x55708fd2a270 msgr2=0x55708fe83df0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:46:33.807 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.791+0000 7f4bb0d9c000 1 -- 172.21.15.16:0/3987454061 shutdown_connections 2023-05-22T15:46:33.807 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.791+0000 7f4bb0d9c000 1 -- 172.21.15.16:0/3987454061 wait complete. 2023-05-22T15:46:33.807 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.791+0000 7f4bb0d9c000 20 client.9264 trim_cache size 0 max 16384 2023-05-22T15:46:33.807 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-05-22T15:46:33.807 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_fsid": "b2719834-7847-4d98-90a6-f65d57f85c49", 2023-05-22T15:46:33.808 INFO:teuthology.orchestra.run.smithi016.stdout: "whoami": 0, 2023-05-22T15:46:33.808 INFO:teuthology.orchestra.run.smithi016.stdout: "id": 9075, 2023-05-22T15:46:33.808 INFO:teuthology.orchestra.run.smithi016.stdout: "want_state": "up:active", 2023-05-22T15:46:33.808 INFO:teuthology.orchestra.run.smithi016.stdout: "state": "up:active", 2023-05-22T15:46:33.808 INFO:teuthology.orchestra.run.smithi016.stdout: "fs_name": "cephfs", 2023-05-22T15:46:33.808 INFO:teuthology.orchestra.run.smithi016.stdout: "rank_uptime": 10.346061421, 2023-05-22T15:46:33.808 INFO:teuthology.orchestra.run.smithi016.stdout: "mdsmap_epoch": 58, 2023-05-22T15:46:33.809 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch": 147, 2023-05-22T15:46:33.809 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch_barrier": 147, 2023-05-22T15:46:33.809 INFO:teuthology.orchestra.run.smithi016.stdout: "uptime": 15.835037062 2023-05-22T15:46:33.809 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-05-22T15:46:33.810 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.791+0000 7f4badc7a640 1 -- 172.21.15.16:0/3665508396 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f4b880654a0 msgr2=0x7f4b88067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:33.810 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.791+0000 7f4badc7a640 1 --2- 172.21.15.16:0/3665508396 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f4b880654a0 0x7f4b88067950 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f4ba0000fc0 tx=0x7f4ba0009040 comp rx=0 tx=0).stop 2023-05-22T15:46:33.810 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.791+0000 7f4badc7a640 1 -- 172.21.15.16:0/3665508396 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ba80b2d10 msgr2=0x7f4ba80c8630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:33.810 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.791+0000 7f4badc7a640 1 --2- 172.21.15.16:0/3665508396 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ba80b2d10 0x7f4ba80c8630 secure :-1 s=READY pgs=1861 cs=0 l=1 rev1=1 crypto rx=0x7f4ba80b50b0 tx=0x7f4b98007e40 comp rx=0 tx=0).stop 2023-05-22T15:46:33.810 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.791+0000 7f4badc7a640 1 -- 172.21.15.16:0/3665508396 shutdown_connections 2023-05-22T15:46:33.811 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.791+0000 7f4badc7a640 1 --2- 172.21.15.16:0/3665508396 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ba80b2d10 0x7f4ba80c8630 unknown :-1 s=CLOSED pgs=1861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:33.811 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.791+0000 7f4badc7a640 1 --2- 172.21.15.16:0/3665508396 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4ba81513f0 0x7f4ba81e45e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:33.811 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.791+0000 7f4badc7a640 1 --2- 172.21.15.16:0/3665508396 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f4b880654a0 0x7f4b88067950 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:33.811 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.791+0000 7f4badc7a640 1 --2- 172.21.15.16:0/3665508396 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4b940047d0 0x7f4ba80c6c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:33.811 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.791+0000 7f4badc7a640 1 -- 172.21.15.16:0/3665508396 >> 172.21.15.16:0/3665508396 conn(0x7f4ba80cc870 msgr2=0x7f4ba80cf380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:46:33.811 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.791+0000 7f4badc7a640 1 -- 172.21.15.16:0/3665508396 shutdown_connections 2023-05-22T15:46:33.811 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.791+0000 7f4badc7a640 1 -- 172.21.15.16:0/3665508396 wait complete. 2023-05-22T15:46:33.812 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-05-22T15:46:33.923 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.919+0000 7f874a0e5640 1 Processor -- start 2023-05-22T15:46:33.924 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.919+0000 7f874a0e5640 1 -- start start 2023-05-22T15:46:33.925 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.919+0000 7f874a0e5640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f87440ca4e0 0x7f874415e460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:33.926 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.919+0000 7f874a0e5640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f87440c9b40 0x7f87440afb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:33.926 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.919+0000 7f874a0e5640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87440b0650 con 0x7f8744070df0 2023-05-22T15:46:33.926 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.919+0000 7f874a0e5640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87440b0370 con 0x7f87440c9b40 2023-05-22T15:46:33.927 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.919+0000 7f874a0e5640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f874415d980 con 0x7f87440ca4e0 2023-05-22T15:46:33.927 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.919+0000 7f8742ffd640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f87440c9b40 0x7f87440afb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:33.927 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.919+0000 7f8742ffd640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f87440c9b40 0x7f87440afb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56602/0 (socket says 172.21.15.16:56602) 2023-05-22T15:46:33.928 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.919+0000 7f8742ffd640 1 -- 172.21.15.16:0/2872568366 learned_addr learned my addr 172.21.15.16:0/2872568366 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:46:33.928 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.919+0000 7f87437fe640 1 --2- 172.21.15.16:0/2872568366 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f87440ca4e0 0x7f874415e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:33.929 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.919+0000 7f8742ffd640 1 -- 172.21.15.16:0/2872568366 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f87440ca4e0 msgr2=0x7f874415e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:46:33.929 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.919+0000 7f8742ffd640 1 --2- 172.21.15.16:0/2872568366 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f87440ca4e0 0x7f874415e460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:33.929 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.919+0000 7f8742ffd640 1 -- 172.21.15.16:0/2872568366 >> v1:172.21.15.62:6789/0 conn(0x7f8744070df0 legacy=0x7f8744071210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:46:33.929 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.919+0000 7f8742ffd640 1 -- 172.21.15.16:0/2872568366 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f874415dc60 con 0x7f87440c9b40 2023-05-22T15:46:33.932 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.923+0000 7f8742ffd640 1 --2- 172.21.15.16:0/2872568366 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f87440c9b40 0x7f87440afb40 secure :-1 s=READY pgs=1866 cs=0 l=1 rev1=1 crypto rx=0x7f8738009870 tx=0x7f873801c440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35386bd2c17cdde2 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:33.932 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.923+0000 7f87427fc640 1 -- 172.21.15.16:0/2872568366 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87380ac640 con 0x7f87440c9b40 2023-05-22T15:46:33.933 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.923+0000 7f87427fc640 1 -- 172.21.15.16:0/2872568366 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87380ac7a0 con 0x7f87440c9b40 2023-05-22T15:46:33.933 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.923+0000 7f87427fc640 1 -- 172.21.15.16:0/2872568366 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87380acb30 con 0x7f87440c9b40 2023-05-22T15:46:33.933 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.923+0000 7f874a0e5640 1 -- 172.21.15.16:0/2872568366 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f87440c9b40 msgr2=0x7f87440afb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:33.933 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.923+0000 7f874a0e5640 1 --2- 172.21.15.16:0/2872568366 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f87440c9b40 0x7f87440afb40 secure :-1 s=READY pgs=1866 cs=0 l=1 rev1=1 crypto rx=0x7f8738009870 tx=0x7f873801c440 comp rx=0 tx=0).stop 2023-05-22T15:46:33.933 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.923+0000 7f874a0e5640 1 -- 172.21.15.16:0/2872568366 shutdown_connections 2023-05-22T15:46:33.933 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.923+0000 7f874a0e5640 1 --2- 172.21.15.16:0/2872568366 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f87440c9b40 0x7f87440afb40 unknown :-1 s=CLOSED pgs=1866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:33.933 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.923+0000 7f874a0e5640 1 --2- 172.21.15.16:0/2872568366 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f87440ca4e0 0x7f874415e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:33.933 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.923+0000 7f874a0e5640 1 -- 172.21.15.16:0/2872568366 >> 172.21.15.16:0/2872568366 conn(0x7f8744069720 msgr2=0x7f8744069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:46:33.934 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.923+0000 7f874a0e5640 1 -- 172.21.15.16:0/2872568366 shutdown_connections 2023-05-22T15:46:33.934 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.923+0000 7f874a0e5640 1 -- 172.21.15.16:0/2872568366 wait complete. 2023-05-22T15:46:33.934 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.923+0000 7f874a0e5640 1 Processor -- start 2023-05-22T15:46:33.934 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.923+0000 7f874a0e5640 1 -- start start 2023-05-22T15:46:33.934 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.923+0000 7f874a0e5640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8744070df0 0x7f8744069d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:33.934 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.923+0000 7f874a0e5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f87440c9b40 0x7f8744151150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:33.934 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.923+0000 7f874a0e5640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f87440ca4e0 0x7f8744151690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:33.935 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.923+0000 7f874a0e5640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87440cbf00 con 0x7f87440c9b40 2023-05-22T15:46:33.935 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.923+0000 7f874a0e5640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87440cc1c0 con 0x7f8744070df0 2023-05-22T15:46:33.935 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.923+0000 7f874a0e5640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87440cc4a0 con 0x7f87440ca4e0 2023-05-22T15:46:33.935 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.923+0000 7f8742ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f87440c9b40 0x7f8744151150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:33.935 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.923+0000 7f8742ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f87440c9b40 0x7f8744151150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56608/0 (socket says 172.21.15.16:56608) 2023-05-22T15:46:33.935 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.923+0000 7f8742ffd640 1 -- 172.21.15.16:0/445558818 learned_addr learned my addr 172.21.15.16:0/445558818 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:46:33.935 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.923+0000 7f87437fe640 1 --2- 172.21.15.16:0/445558818 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8744070df0 0x7f8744069d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:33.935 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.923+0000 7f8743fff640 1 --2- 172.21.15.16:0/445558818 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f87440ca4e0 0x7f8744151690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:33.936 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.923+0000 7f8742ffd640 1 -- 172.21.15.16:0/445558818 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f87440ca4e0 msgr2=0x7f8744151690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:33.936 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.923+0000 7f8742ffd640 1 --2- 172.21.15.16:0/445558818 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f87440ca4e0 0x7f8744151690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:33.936 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.923+0000 7f8742ffd640 1 -- 172.21.15.16:0/445558818 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8744070df0 msgr2=0x7f8744069d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:33.936 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.923+0000 7f8742ffd640 1 --2- 172.21.15.16:0/445558818 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8744070df0 0x7f8744069d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:33.936 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.923+0000 7f8742ffd640 1 -- 172.21.15.16:0/445558818 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87380094d0 con 0x7f87440c9b40 2023-05-22T15:46:33.936 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.927+0000 7f8742ffd640 1 --2- 172.21.15.16:0/445558818 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f87440c9b40 0x7f8744151150 secure :-1 s=READY pgs=1867 cs=0 l=1 rev1=1 crypto rx=0x7f87380af040 tx=0x7f87380191e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:33.936 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.927+0000 7f872bfff640 1 -- 172.21.15.16:0/445558818 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87380ba070 con 0x7f87440c9b40 2023-05-22T15:46:33.937 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.927+0000 7f872bfff640 1 -- 172.21.15.16:0/445558818 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87380b5440 con 0x7f87440c9b40 2023-05-22T15:46:33.937 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.927+0000 7f874a0e5640 1 -- 172.21.15.16:0/445558818 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87440c8e80 con 0x7f87440c9b40 2023-05-22T15:46:33.937 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.927+0000 7f872bfff640 1 -- 172.21.15.16:0/445558818 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87380b57f0 con 0x7f87440c9b40 2023-05-22T15:46:33.937 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.927+0000 7f874a0e5640 1 -- 172.21.15.16:0/445558818 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87440ca8b0 con 0x7f87440c9b40 2023-05-22T15:46:33.937 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.927+0000 7f874a0e5640 1 -- 172.21.15.16:0/445558818 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87440c9010 con 0x7f87440c9b40 2023-05-22T15:46:33.937 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.927+0000 7f872bfff640 1 -- 172.21.15.16:0/445558818 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 60) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f87380b3070 con 0x7f87440c9b40 2023-05-22T15:46:33.938 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.927+0000 7f872bfff640 1 --2- 172.21.15.16:0/445558818 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f87300654a0 0x7f8730067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:33.938 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.927+0000 7f872bfff640 1 -- 172.21.15.16:0/445558818 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(147..147 src has 1..147) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f873811c950 con 0x7f87440c9b40 2023-05-22T15:46:33.938 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.927+0000 7f87437fe640 1 --2- 172.21.15.16:0/445558818 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f87300654a0 0x7f8730067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:33.938 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.927+0000 7f87437fe640 1 --2- 172.21.15.16:0/445558818 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f87300654a0 0x7f8730067950 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f873400b930 tx=0x7f8734008040 comp rx=0 tx=0).ready entity=mgr.8835 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:33.939 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:33.931+0000 7f872bfff640 1 -- 172.21.15.16:0/445558818 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f87380ea360 con 0x7f87440c9b40 2023-05-22T15:46:34.122 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:34.115+0000 7f874a0e5640 1 -- 172.21.15.16:0/445558818 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f874404dbe0 con 0x7f87440c9b40 2023-05-22T15:46:34.122 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:34.115+0000 7f872bfff640 1 -- 172.21.15.16:0/445558818 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v58) v1 ==== 53+0+73 (secure 0 0 0) 0x7f87380ed980 con 0x7f87440c9b40 2023-05-22T15:46:34.123 INFO:teuthology.orchestra.run.smithi016.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-05-22T15:46:34.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:34.119+0000 7f874a0e5640 1 -- 172.21.15.16:0/445558818 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f87300654a0 msgr2=0x7f8730067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:34.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:34.119+0000 7f874a0e5640 1 --2- 172.21.15.16:0/445558818 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f87300654a0 0x7f8730067950 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f873400b930 tx=0x7f8734008040 comp rx=0 tx=0).stop 2023-05-22T15:46:34.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:34.119+0000 7f874a0e5640 1 -- 172.21.15.16:0/445558818 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f87440c9b40 msgr2=0x7f8744151150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:34.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:34.119+0000 7f874a0e5640 1 --2- 172.21.15.16:0/445558818 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f87440c9b40 0x7f8744151150 secure :-1 s=READY pgs=1867 cs=0 l=1 rev1=1 crypto rx=0x7f87380af040 tx=0x7f87380191e0 comp rx=0 tx=0).stop 2023-05-22T15:46:34.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:34.119+0000 7f874a0e5640 1 -- 172.21.15.16:0/445558818 shutdown_connections 2023-05-22T15:46:34.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:34.119+0000 7f874a0e5640 1 --2- 172.21.15.16:0/445558818 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f87300654a0 0x7f8730067950 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:34.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:34.119+0000 7f874a0e5640 1 --2- 172.21.15.16:0/445558818 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f87440ca4e0 0x7f8744151690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:34.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:34.119+0000 7f874a0e5640 1 --2- 172.21.15.16:0/445558818 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f87440c9b40 0x7f8744151150 unknown :-1 s=CLOSED pgs=1867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:34.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:34.119+0000 7f874a0e5640 1 --2- 172.21.15.16:0/445558818 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8744070df0 0x7f8744069d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:34.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:34.119+0000 7f874a0e5640 1 -- 172.21.15.16:0/445558818 >> 172.21.15.16:0/445558818 conn(0x7f8744069720 msgr2=0x7f87440b20f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:46:34.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:34.119+0000 7f874a0e5640 1 -- 172.21.15.16:0/445558818 shutdown_connections 2023-05-22T15:46:34.126 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:34.119+0000 7f874a0e5640 1 -- 172.21.15.16:0/445558818 wait complete. 2023-05-22T15:46:34.138 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-05-22T15:46:34.139 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-05-22T15:46:34.139 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi016.front.sepia.ceph.com 2023-05-22T15:46:34.140 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-05-22T15:46:34.140 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-05-22T15:46:34.140 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-05-22T15:46:34.141 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-05-22T15:46:34.141 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-05-22T15:46:34.141 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-05-22T15:46:34.142 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:46:34.142 DEBUG:teuthology.orchestra.run.smithi016:> ip addr 2023-05-22T15:46:34.147 INFO:teuthology.orchestra.run.smithi016.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-05-22T15:46:34.147 INFO:teuthology.orchestra.run.smithi016.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-05-22T15:46:34.148 INFO:teuthology.orchestra.run.smithi016.stdout: inet 127.0.0.1/8 scope host lo 2023-05-22T15:46:34.148 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-05-22T15:46:34.148 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 ::1/128 scope host 2023-05-22T15:46:34.148 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-05-22T15:46:34.148 INFO:teuthology.orchestra.run.smithi016.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-05-22T15:46:34.148 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:6b:fd:e0 brd ff:ff:ff:ff:ff:ff 2023-05-22T15:46:34.149 INFO:teuthology.orchestra.run.smithi016.stdout: altname enp7s0f0 2023-05-22T15:46:34.149 INFO:teuthology.orchestra.run.smithi016.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-05-22T15:46:34.149 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:6b:fd:e1 brd ff:ff:ff:ff:ff:ff 2023-05-22T15:46:34.150 INFO:teuthology.orchestra.run.smithi016.stdout: altname enp7s0f1 2023-05-22T15:46:34.150 INFO:teuthology.orchestra.run.smithi016.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2023-05-22T15:46:34.150 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:bd:15:64 brd ff:ff:ff:ff:ff:ff 2023-05-22T15:46:34.150 INFO:teuthology.orchestra.run.smithi016.stdout: altname enp1s0f0 2023-05-22T15:46:34.150 INFO:teuthology.orchestra.run.smithi016.stdout: inet 172.21.15.16/20 brd 172.21.15.255 scope global dynamic ens1f0 2023-05-22T15:46:34.150 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft 41322sec preferred_lft 41322sec 2023-05-22T15:46:34.150 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::ec4:7aff:febd:1564/64 scope link 2023-05-22T15:46:34.151 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-05-22T15:46:34.151 INFO:teuthology.orchestra.run.smithi016.stdout:5: ens1f1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-05-22T15:46:34.151 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:bd:15:65 brd ff:ff:ff:ff:ff:ff 2023-05-22T15:46:34.151 INFO:teuthology.orchestra.run.smithi016.stdout: altname enp1s0f1 2023-05-22T15:46:34.152 INFO:teuthology.orchestra.run.smithi016.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2023-05-22T15:46:34.152 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 02:42:1f:b8:98:0a brd ff:ff:ff:ff:ff:ff 2023-05-22T15:46:34.152 INFO:teuthology.orchestra.run.smithi016.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2023-05-22T15:46:34.152 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-05-22T15:46:34.152 INFO:teuthology.orchestra.run.smithi016.stdout:9: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-05-22T15:46:34.152 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 5e:1f:84:6d:79:ae brd ff:ff:ff:ff:ff:ff 2023-05-22T15:46:34.152 INFO:teuthology.orchestra.run.smithi016.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-05-22T15:46:34.153 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-05-22T15:46:34.153 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::5c1f:84ff:fe6d:79ae/64 scope link 2023-05-22T15:46:34.153 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-05-22T15:46:34.153 INFO:teuthology.orchestra.run.smithi016.stdout:10: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-05-22T15:46:34.153 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0e:5c:f6:1f:82:2f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-05-22T15:46:34.153 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::c5c:f6ff:fe1f:822f/64 scope link 2023-05-22T15:46:34.153 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-05-22T15:46:34.153 INFO:teuthology.orchestra.run.smithi016.stdout:11: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-05-22T15:46:34.154 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether c6:a6:3d:69:d0:db brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-05-22T15:46:34.154 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::c4a6:3dff:fe69:d0db/64 scope link 2023-05-22T15:46:34.154 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-05-22T15:46:34.154 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:46:34.155 DEBUG:teuthology.orchestra.run.smithi016:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-05-22T15:46:34.200 INFO:teuthology.orchestra.run.smithi016.stdout:1 2023-05-22T15:46:34.201 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:46:34.201 DEBUG:teuthology.orchestra.run.smithi016:> route 2023-05-22T15:46:34.249 INFO:teuthology.orchestra.run.smithi016.stdout:Kernel IP routing table 2023-05-22T15:46:34.249 INFO:teuthology.orchestra.run.smithi016.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-05-22T15:46:34.249 INFO:teuthology.orchestra.run.smithi016.stdout:default _gateway 0.0.0.0 UG 0 0 0 ens1f0 2023-05-22T15:46:34.249 INFO:teuthology.orchestra.run.smithi016.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2023-05-22T15:46:34.249 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 0 0 0 ens1f0 2023-05-22T15:46:34.250 INFO:teuthology.orchestra.run.smithi016.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-05-22T15:46:34.250 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:46:34.250 DEBUG:teuthology.orchestra.run.smithi016:> (cd / && exec bash -c ' 2023-05-22T15:46:34.251 DEBUG:teuthology.orchestra.run.smithi016:> set -e 2023-05-22T15:46:34.251 DEBUG:teuthology.orchestra.run.smithi016:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2023-05-22T15:46:34.252 DEBUG:teuthology.orchestra.run.smithi016:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2023-05-22T15:46:34.252 DEBUG:teuthology.orchestra.run.smithi016:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2023-05-22T15:46:34.252 DEBUG:teuthology.orchestra.run.smithi016:> ') 2023-05-22T15:46:34.322 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:46:34.322 DEBUG:teuthology.orchestra.run.smithi016:> ip netns list 2023-05-22T15:46:34.371 INFO:teuthology.orchestra.run.smithi016.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-05-22T15:46:34.371 INFO:teuthology.orchestra.run.smithi016.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-05-22T15:46:34.371 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-05-22T15:46:34.372 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:46:34.372 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip link set brx.0 up 2023-05-22T15:46:34.426 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2023-05-22T15:46:34.427 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-05-22T15:46:34.427 DEBUG:teuthology.orchestra.run.smithi016:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-05-22T15:46:34.475 INFO:teuthology.orchestra.run.smithi016.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-05-22T15:46:34.476 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-05-22T15:46:34.476 DEBUG:teuthology.orchestra.run.smithi016:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-05-22T15:46:34.527 DEBUG:teuthology.orchestra.run.smithi016:> sudo modprobe fuse 2023-05-22T15:46:34.585 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-05-22T15:46:34.586 DEBUG:teuthology.orchestra.run.smithi016:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-05-22T15:46:34.638 INFO:teuthology.orchestra.run.smithi016.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-05-22T15:46:34.642 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-05-22T15:46:34.643 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:46:34.644 DEBUG:teuthology.orchestra.run.smithi016:> ls /sys/fs/fuse/connections 2023-05-22T15:46:34.692 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2023-05-22T15:46:34.692 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2023-05-22T15:46:34.738 DEBUG:teuthology.orchestra.run.smithi016:> sudo modprobe fuse 2023-05-22T15:46:34.753 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-05-22T15:46:34.754 DEBUG:teuthology.orchestra.run.smithi016:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-05-22T15:46:34.811 INFO:teuthology.orchestra.run.smithi016.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-05-22T15:46:34.813 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-05-22T15:46:34.814 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:46:34.814 DEBUG:teuthology.orchestra.run.smithi016:> ls /sys/fs/fuse/connections 2023-05-22T15:46:34.822 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi016.stderr:2023-05-22T15:46:34.815+0000 7fe5b9bf9580 -1 init, newargv = 0x5564fa907dc0 newargc=15 2023-05-22T15:46:34.823 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi016.stderr:ceph-fuse[63363]: starting ceph client 2023-05-22T15:46:34.828 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi016.stderr:ceph-fuse[63363]: starting fuse 2023-05-22T15:46:35.928 DEBUG:teuthology.orchestra.run.smithi016:> sudo modprobe fuse 2023-05-22T15:46:35.945 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-05-22T15:46:35.945 DEBUG:teuthology.orchestra.run.smithi016:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-05-22T15:46:35.999 INFO:teuthology.orchestra.run.smithi016.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-05-22T15:46:36.001 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-05-22T15:46:36.002 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:46:36.002 DEBUG:teuthology.orchestra.run.smithi016:> ls /sys/fs/fuse/connections 2023-05-22T15:46:36.052 INFO:teuthology.orchestra.run.smithi016.stdout:45 2023-05-22T15:46:36.053 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [45] 2023-05-22T15:46:36.053 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2023-05-22T15:46:36.054 DEBUG:teuthology.orchestra.run.smithi016:> import glob 2023-05-22T15:46:36.054 DEBUG:teuthology.orchestra.run.smithi016:> import re 2023-05-22T15:46:36.054 DEBUG:teuthology.orchestra.run.smithi016:> import os 2023-05-22T15:46:36.054 DEBUG:teuthology.orchestra.run.smithi016:> import subprocess 2023-05-22T15:46:36.055 DEBUG:teuthology.orchestra.run.smithi016:> 2023-05-22T15:46:36.055 DEBUG:teuthology.orchestra.run.smithi016:> def _find_admin_socket(client_name): 2023-05-22T15:46:36.055 DEBUG:teuthology.orchestra.run.smithi016:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2023-05-22T15:46:36.056 DEBUG:teuthology.orchestra.run.smithi016:> files = glob.glob(asok_path) 2023-05-22T15:46:36.056 DEBUG:teuthology.orchestra.run.smithi016:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2023-05-22T15:46:36.056 DEBUG:teuthology.orchestra.run.smithi016:> 2023-05-22T15:46:36.057 DEBUG:teuthology.orchestra.run.smithi016:> # Given a non-glob path, it better be there 2023-05-22T15:46:36.057 DEBUG:teuthology.orchestra.run.smithi016:> if "*" not in asok_path: 2023-05-22T15:46:36.057 DEBUG:teuthology.orchestra.run.smithi016:> assert(len(files) == 1) 2023-05-22T15:46:36.057 DEBUG:teuthology.orchestra.run.smithi016:> return files[0] 2023-05-22T15:46:36.058 DEBUG:teuthology.orchestra.run.smithi016:> 2023-05-22T15:46:36.058 DEBUG:teuthology.orchestra.run.smithi016:> for f in files: 2023-05-22T15:46:36.058 DEBUG:teuthology.orchestra.run.smithi016:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-05-22T15:46:36.059 DEBUG:teuthology.orchestra.run.smithi016:> if os.path.exists("/proc/{0}".format(pid)): 2023-05-22T15:46:36.059 DEBUG:teuthology.orchestra.run.smithi016:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-05-22T15:46:36.059 DEBUG:teuthology.orchestra.run.smithi016:> contents = proc_f.read() 2023-05-22T15:46:36.060 DEBUG:teuthology.orchestra.run.smithi016:> if mountpoint in contents: 2023-05-22T15:46:36.060 DEBUG:teuthology.orchestra.run.smithi016:> return f 2023-05-22T15:46:36.060 DEBUG:teuthology.orchestra.run.smithi016:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-05-22T15:46:36.060 DEBUG:teuthology.orchestra.run.smithi016:> 2023-05-22T15:46:36.061 DEBUG:teuthology.orchestra.run.smithi016:> print(_find_admin_socket("client.0")) 2023-05-22T15:46:36.061 DEBUG:teuthology.orchestra.run.smithi016:> ' 2023-05-22T15:46:36.175 INFO:teuthology.orchestra.run.smithi016.stdout:/var/run/ceph/ceph-client.0.63363.asok 2023-05-22T15:46:36.358 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.63363.asok 2023-05-22T15:46:36.358 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:46:36.359 DEBUG:teuthology.orchestra.run.smithi016:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.63363.asok status 2023-05-22T15:46:36.519 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-05-22T15:46:36.519 INFO:teuthology.orchestra.run.smithi016.stdout: "metadata": { 2023-05-22T15:46:36.520 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph_sha1": "9c9ba4df22024d9d8d6cf9f98a4b96751fc78d0c", 2023-05-22T15:46:36.520 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph_version": "ceph version 18.0.0-3192-g9c9ba4df (9c9ba4df22024d9d8d6cf9f98a4b96751fc78d0c) reef (dev)", 2023-05-22T15:46:36.520 INFO:teuthology.orchestra.run.smithi016.stdout: "entity_id": "0", 2023-05-22T15:46:36.520 INFO:teuthology.orchestra.run.smithi016.stdout: "hostname": "smithi016", 2023-05-22T15:46:36.520 INFO:teuthology.orchestra.run.smithi016.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2023-05-22T15:46:36.520 INFO:teuthology.orchestra.run.smithi016.stdout: "pid": "63363", 2023-05-22T15:46:36.521 INFO:teuthology.orchestra.run.smithi016.stdout: "root": "/" 2023-05-22T15:46:36.521 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:46:36.522 INFO:teuthology.orchestra.run.smithi016.stdout: "dentry_count": 0, 2023-05-22T15:46:36.522 INFO:teuthology.orchestra.run.smithi016.stdout: "dentry_pinned_count": 0, 2023-05-22T15:46:36.522 INFO:teuthology.orchestra.run.smithi016.stdout: "id": 9279, 2023-05-22T15:46:36.522 INFO:teuthology.orchestra.run.smithi016.stdout: "inst": { 2023-05-22T15:46:36.522 INFO:teuthology.orchestra.run.smithi016.stdout: "name": { 2023-05-22T15:46:36.522 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "client", 2023-05-22T15:46:36.522 INFO:teuthology.orchestra.run.smithi016.stdout: "num": 9279 2023-05-22T15:46:36.523 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:46:36.523 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": { 2023-05-22T15:46:36.523 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-05-22T15:46:36.523 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "192.168.0.1:0", 2023-05-22T15:46:36.523 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 519735738 2023-05-22T15:46:36.523 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T15:46:36.523 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:46:36.523 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": { 2023-05-22T15:46:36.524 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-05-22T15:46:36.524 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "192.168.0.1:0", 2023-05-22T15:46:36.524 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 519735738 2023-05-22T15:46:36.524 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:46:36.524 INFO:teuthology.orchestra.run.smithi016.stdout: "inst_str": "client.9279 192.168.0.1:0/519735738", 2023-05-22T15:46:36.524 INFO:teuthology.orchestra.run.smithi016.stdout: "addr_str": "192.168.0.1:0/519735738", 2023-05-22T15:46:36.524 INFO:teuthology.orchestra.run.smithi016.stdout: "inode_count": 1, 2023-05-22T15:46:36.525 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_epoch": 58, 2023-05-22T15:46:36.525 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_epoch": 147, 2023-05-22T15:46:36.525 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_epoch_barrier": 0, 2023-05-22T15:46:36.525 INFO:teuthology.orchestra.run.smithi016.stdout: "blocklisted": false, 2023-05-22T15:46:36.526 INFO:teuthology.orchestra.run.smithi016.stdout: "fs_name": "cephfs" 2023-05-22T15:46:36.526 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-05-22T15:46:36.533 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:46:36.533 DEBUG:teuthology.orchestra.run.smithi016:> stat --file-system '--printf=%T 2023-05-22T15:46:36.534 DEBUG:teuthology.orchestra.run.smithi016:> ' -- /home/ubuntu/cephtest/mnt.0 2023-05-22T15:46:36.539 INFO:teuthology.orchestra.run.smithi016.stdout:fuseblk 2023-05-22T15:46:36.540 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2023-05-22T15:46:36.540 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:46:36.541 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-05-22T15:46:36.597 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2023-05-22T15:46:36.773 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:36.767+0000 7f8cda981640 1 Processor -- start 2023-05-22T15:46:36.773 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:36.767+0000 7f8cda981640 1 -- start start 2023-05-22T15:46:36.775 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:36.767+0000 7f8cda981640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f8cd40ca4e0 0x7f8cd415e460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:36.775 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:36.767+0000 7f8cda981640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8cd40c9b40 0x7f8cd40afb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:36.775 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:36.767+0000 7f8cda981640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cd40b0650 con 0x7f8cd4070df0 2023-05-22T15:46:36.776 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:36.767+0000 7f8cda981640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8cd40b0370 con 0x7f8cd40ca4e0 2023-05-22T15:46:36.776 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:36.767+0000 7f8cda981640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8cd415d980 con 0x7f8cd40c9b40 2023-05-22T15:46:36.776 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:36.767+0000 7f8cd3fff640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f8cd40ca4e0 0x7f8cd415e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:36.776 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:36.767+0000 7f8cd3fff640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f8cd40ca4e0 0x7f8cd415e460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56612/0 (socket says 172.21.15.16:56612) 2023-05-22T15:46:36.777 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:36.767+0000 7f8cd3fff640 1 -- 172.21.15.16:0/3634505 learned_addr learned my addr 172.21.15.16:0/3634505 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:46:36.777 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:36.767+0000 7f8cd37fe640 1 --2- 172.21.15.16:0/3634505 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8cd40c9b40 0x7f8cd40afb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:36.777 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:36.767+0000 7f8cd3fff640 1 -- 172.21.15.16:0/3634505 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8cd40c9b40 msgr2=0x7f8cd40afb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:46:36.778 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:36.767+0000 7f8cd3fff640 1 --2- 172.21.15.16:0/3634505 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8cd40c9b40 0x7f8cd40afb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:36.778 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:36.767+0000 7f8cd3fff640 1 -- 172.21.15.16:0/3634505 >> v1:172.21.15.16:6789/0 conn(0x7f8cd4070df0 legacy=0x7f8cd4071210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:36.778 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:36.767+0000 7f8cd3fff640 1 -- 172.21.15.16:0/3634505 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cd415dc60 con 0x7f8cd40ca4e0 2023-05-22T15:46:36.780 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:36.771+0000 7f8cd3fff640 1 --2- 172.21.15.16:0/3634505 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8cd40ca4e0 0x7f8cd415e460 secure :-1 s=READY pgs=1872 cs=0 l=1 rev1=1 crypto rx=0x7f8ccc009870 tx=0x7f8ccc01c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be1340424b756ae2 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:36.781 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:36.771+0000 7f8cd2ffd640 1 -- 172.21.15.16:0/3634505 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ccc0ab660 con 0x7f8cd40ca4e0 2023-05-22T15:46:36.781 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:36.771+0000 7f8cd2ffd640 1 -- 172.21.15.16:0/3634505 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ccc0ab7c0 con 0x7f8cd40ca4e0 2023-05-22T15:46:36.781 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:36.771+0000 7f8cd2ffd640 1 -- 172.21.15.16:0/3634505 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ccc0abb50 con 0x7f8cd40ca4e0 2023-05-22T15:46:36.782 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:36.771+0000 7f8cda981640 1 -- 172.21.15.16:0/3634505 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8cd40ca4e0 msgr2=0x7f8cd415e460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:36.782 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:36.771+0000 7f8cda981640 1 --2- 172.21.15.16:0/3634505 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8cd40ca4e0 0x7f8cd415e460 secure :-1 s=READY pgs=1872 cs=0 l=1 rev1=1 crypto rx=0x7f8ccc009870 tx=0x7f8ccc01c480 comp rx=0 tx=0).stop 2023-05-22T15:46:36.783 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:36.771+0000 7f8cda981640 1 -- 172.21.15.16:0/3634505 shutdown_connections 2023-05-22T15:46:36.783 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:36.771+0000 7f8cda981640 1 --2- 172.21.15.16:0/3634505 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8cd40c9b40 0x7f8cd40afb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:36.783 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:36.771+0000 7f8cda981640 1 --2- 172.21.15.16:0/3634505 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8cd40ca4e0 0x7f8cd415e460 unknown :-1 s=CLOSED pgs=1872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:36.784 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:36.771+0000 7f8cda981640 1 -- 172.21.15.16:0/3634505 >> 172.21.15.16:0/3634505 conn(0x7f8cd4069720 msgr2=0x7f8cd4069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:46:36.784 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:36.771+0000 7f8cda981640 1 -- 172.21.15.16:0/3634505 shutdown_connections 2023-05-22T15:46:36.784 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:36.771+0000 7f8cda981640 1 -- 172.21.15.16:0/3634505 wait complete. 2023-05-22T15:46:36.784 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:36.775+0000 7f8cda981640 1 Processor -- start 2023-05-22T15:46:36.785 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:36.775+0000 7f8cda981640 1 -- start start 2023-05-22T15:46:36.787 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:36.775+0000 7f8cda981640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8cd4070df0 0x7f8cd4069d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:36.787 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:36.775+0000 7f8cda981640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8cd40c9b40 0x7f8cd414b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:36.787 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:36.775+0000 7f8cda981640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8cd40ca4e0 0x7f8cd414be20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:36.788 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:36.775+0000 7f8cda981640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cd414c360 con 0x7f8cd40c9b40 2023-05-22T15:46:36.788 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:36.775+0000 7f8cda981640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cd414ab20 con 0x7f8cd40ca4e0 2023-05-22T15:46:36.788 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:36.775+0000 7f8cda981640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8cd414ae00 con 0x7f8cd4070df0 2023-05-22T15:46:36.788 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:36.775+0000 7f8cd37fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8cd40c9b40 0x7f8cd414b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:36.789 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:36.775+0000 7f8cd37fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8cd40c9b40 0x7f8cd414b8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56626/0 (socket says 172.21.15.16:56626) 2023-05-22T15:46:36.789 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:36.775+0000 7f8cd37fe640 1 -- 172.21.15.16:0/1847992020 learned_addr learned my addr 172.21.15.16:0/1847992020 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:46:36.789 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:36.775+0000 7f8cd3fff640 1 --2- 172.21.15.16:0/1847992020 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8cd4070df0 0x7f8cd4069d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:36.789 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:36.775+0000 7f8cd8f1e640 1 --2- 172.21.15.16:0/1847992020 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8cd40ca4e0 0x7f8cd414be20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:36.789 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:36.775+0000 7f8cd37fe640 1 -- 172.21.15.16:0/1847992020 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8cd4070df0 msgr2=0x7f8cd4069d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:36.789 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:36.775+0000 7f8cd37fe640 1 --2- 172.21.15.16:0/1847992020 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8cd4070df0 0x7f8cd4069d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:36.790 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:36.775+0000 7f8cd37fe640 1 -- 172.21.15.16:0/1847992020 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8cd40ca4e0 msgr2=0x7f8cd414be20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:36.790 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:36.775+0000 7f8cd37fe640 1 --2- 172.21.15.16:0/1847992020 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8cd40ca4e0 0x7f8cd414be20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:36.790 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:36.775+0000 7f8cd37fe640 1 -- 172.21.15.16:0/1847992020 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ccc0094d0 con 0x7f8cd40c9b40 2023-05-22T15:46:36.790 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:36.775+0000 7f8cd37fe640 1 --2- 172.21.15.16:0/1847992020 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8cd40c9b40 0x7f8cd414b8e0 secure :-1 s=READY pgs=1873 cs=0 l=1 rev1=1 crypto rx=0x7f8cbc00d800 tx=0x7f8cbc00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:36.790 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:36.775+0000 7f8cd0ff9640 1 -- 172.21.15.16:0/1847992020 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cbc00b6c0 con 0x7f8cd40c9b40 2023-05-22T15:46:36.790 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:36.775+0000 7f8cd0ff9640 1 -- 172.21.15.16:0/1847992020 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8cbc00b820 con 0x7f8cd40c9b40 2023-05-22T15:46:36.790 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:36.775+0000 7f8cda981640 1 -- 172.21.15.16:0/1847992020 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8cd40c8e80 con 0x7f8cd40c9b40 2023-05-22T15:46:36.791 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:36.775+0000 7f8cd0ff9640 1 -- 172.21.15.16:0/1847992020 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cbc00bb90 con 0x7f8cd40c9b40 2023-05-22T15:46:36.791 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:36.775+0000 7f8cda981640 1 -- 172.21.15.16:0/1847992020 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8cd40ca8b0 con 0x7f8cd40c9b40 2023-05-22T15:46:36.791 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:36.775+0000 7f8cda981640 1 -- 172.21.15.16:0/1847992020 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8cd40c9010 con 0x7f8cd40c9b40 2023-05-22T15:46:36.791 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:36.779+0000 7f8cd0ff9640 1 -- 172.21.15.16:0/1847992020 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 60) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f8cbc00bcf0 con 0x7f8cd40c9b40 2023-05-22T15:46:36.791 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:36.779+0000 7f8cd0ff9640 1 --2- 172.21.15.16:0/1847992020 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f8cb40654a0 0x7f8cb4067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:36.792 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:36.779+0000 7f8cd0ff9640 1 -- 172.21.15.16:0/1847992020 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(147..147 src has 1..147) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f8cbc083130 con 0x7f8cd40c9b40 2023-05-22T15:46:36.792 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:36.779+0000 7f8cd3fff640 1 --2- 172.21.15.16:0/1847992020 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f8cb40654a0 0x7f8cb4067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:36.792 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:36.779+0000 7f8cd3fff640 1 --2- 172.21.15.16:0/1847992020 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f8cb40654a0 0x7f8cb4067950 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f8ccc0af000 tx=0x7f8ccc0a7000 comp rx=0 tx=0).ready entity=mgr.8835 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:36.793 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:36.783+0000 7f8cd0ff9640 1 -- 172.21.15.16:0/1847992020 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f8cbc0544d0 con 0x7f8cd40c9b40 2023-05-22T15:46:36.961 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:36.955+0000 7f8cda981640 1 -- 172.21.15.16:0/1847992020 --> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f8cd40c8330 con 0x7f8cb40654a0 2023-05-22T15:46:36.963 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:36.959+0000 7f8cd0ff9640 1 -- 172.21.15.16:0/1847992020 <== mgr.8835 v2:172.21.15.16:6832/36921 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f8cd40c8330 con 0x7f8cb40654a0 2023-05-22T15:46:36.964 INFO:teuthology.orchestra.run.smithi016.stdout:[ 2023-05-22T15:46:36.964 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T15:46:36.964 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "cephfs" 2023-05-22T15:46:36.964 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T15:46:36.965 INFO:teuthology.orchestra.run.smithi016.stdout:] 2023-05-22T15:46:36.965 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:36.959+0000 7f8cda981640 1 -- 172.21.15.16:0/1847992020 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f8cb40654a0 msgr2=0x7f8cb4067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:36.966 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:36.959+0000 7f8cda981640 1 --2- 172.21.15.16:0/1847992020 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f8cb40654a0 0x7f8cb4067950 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f8ccc0af000 tx=0x7f8ccc0a7000 comp rx=0 tx=0).stop 2023-05-22T15:46:36.966 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:36.959+0000 7f8cda981640 1 -- 172.21.15.16:0/1847992020 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8cd40c9b40 msgr2=0x7f8cd414b8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:36.966 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:36.959+0000 7f8cda981640 1 --2- 172.21.15.16:0/1847992020 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8cd40c9b40 0x7f8cd414b8e0 secure :-1 s=READY pgs=1873 cs=0 l=1 rev1=1 crypto rx=0x7f8cbc00d800 tx=0x7f8cbc00dcc0 comp rx=0 tx=0).stop 2023-05-22T15:46:36.966 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:36.959+0000 7f8cda981640 1 -- 172.21.15.16:0/1847992020 shutdown_connections 2023-05-22T15:46:36.966 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:36.959+0000 7f8cda981640 1 --2- 172.21.15.16:0/1847992020 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8cd40ca4e0 0x7f8cd414be20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:36.966 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:36.959+0000 7f8cda981640 1 --2- 172.21.15.16:0/1847992020 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8cd40c9b40 0x7f8cd414b8e0 unknown :-1 s=CLOSED pgs=1873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:36.966 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:36.959+0000 7f8cda981640 1 --2- 172.21.15.16:0/1847992020 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f8cb40654a0 0x7f8cb4067950 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:36.967 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:36.959+0000 7f8cda981640 1 --2- 172.21.15.16:0/1847992020 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8cd4070df0 0x7f8cd4069d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:36.967 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:36.959+0000 7f8cda981640 1 -- 172.21.15.16:0/1847992020 >> 172.21.15.16:0/1847992020 conn(0x7f8cd4069720 msgr2=0x7f8cd4152820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:46:36.967 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:36.959+0000 7f8cda981640 1 -- 172.21.15.16:0/1847992020 shutdown_connections 2023-05-22T15:46:36.967 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:36.959+0000 7f8cda981640 1 -- 172.21.15.16:0/1847992020 wait complete. 2023-05-22T15:46:36.982 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2023-05-22T15:46:37.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:37.095+0000 7f2880a5b640 1 Processor -- start 2023-05-22T15:46:37.103 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:37.095+0000 7f2880a5b640 1 -- start start 2023-05-22T15:46:37.105 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:37.099+0000 7f2880a5b640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f287c0b5a50 0x7f287c0d2900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:37.105 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:37.099+0000 7f2880a5b640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f287c0cccc0 con 0x7f287c15bca0 2023-05-22T15:46:37.105 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:37.099+0000 7f2880a5b640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f287c0ccfa0 con 0x7f287c0d2480 2023-05-22T15:46:37.105 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:37.099+0000 7f2880a5b640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f287c0cd660 con 0x7f287c0b5a50 2023-05-22T15:46:37.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:37.099+0000 7f287a59c640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f287c15bca0 0x7f287c0cda80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:54932/0 (socket says 172.21.15.16:54932) 2023-05-22T15:46:37.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:37.099+0000 7f287a59c640 1 -- 172.21.15.16:0/3002175767 learned_addr learned my addr 172.21.15.16:0/3002175767 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:46:37.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:37.099+0000 7f287ad9d640 1 --2- 172.21.15.16:0/3002175767 >> v2:172.21.15.62:3300/0 conn(0x7f287c0b5a50 0x7f287c0d2900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:37.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:37.099+0000 7f287959a640 1 -- 172.21.15.16:0/3002175767 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2969705045 0 0) 0x7f287c0cccc0 con 0x7f287c15bca0 2023-05-22T15:46:37.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:37.099+0000 7f287959a640 1 -- 172.21.15.16:0/3002175767 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2868003530 con 0x7f287c15bca0 2023-05-22T15:46:37.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:37.099+0000 7f287959a640 1 -- 172.21.15.16:0/3002175767 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2041142756 0 0) 0x7f2868003530 con 0x7f287c15bca0 2023-05-22T15:46:37.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:37.099+0000 7f287959a640 1 -- 172.21.15.16:0/3002175767 >> v2:172.21.15.62:3300/0 conn(0x7f287c0b5a50 msgr2=0x7f287c0d2900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:46:37.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:37.099+0000 7f287959a640 1 --2- 172.21.15.16:0/3002175767 >> v2:172.21.15.62:3300/0 conn(0x7f287c0b5a50 0x7f287c0d2900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:37.107 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:37.099+0000 7f287959a640 1 -- 172.21.15.16:0/3002175767 >> v1:172.21.15.62:6789/0 conn(0x7f287c0d2480 legacy=0x7f287c0b5340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:37.107 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:37.099+0000 7f287959a640 1 -- 172.21.15.16:0/3002175767 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f287c0cd280 con 0x7f287c15bca0 2023-05-22T15:46:37.107 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:37.099+0000 7f287959a640 1 -- 172.21.15.16:0/3002175767 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7f2874001a00 con 0x7f287c15bca0 2023-05-22T15:46:37.107 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:37.099+0000 7f287959a640 1 -- 172.21.15.16:0/3002175767 >> v1:172.21.15.16:6789/0 conn(0x7f287c15bca0 legacy=0x7f287c0cda80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:37.107 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:37.099+0000 7f287959a640 1 --2- 172.21.15.16:0/3002175767 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2868003980 0x7f2868003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:37.107 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:37.099+0000 7f287959a640 1 --2- 172.21.15.16:0/3002175767 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f28680047d0 0x7f2868004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:37.107 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:37.099+0000 7f287959a640 1 --2- 172.21.15.16:0/3002175767 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2868005180 0x7f286800b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:37.107 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:37.099+0000 7f287959a640 1 -- 172.21.15.16:0/3002175767 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f287c0cd660 con 0x7f2868005180 2023-05-22T15:46:37.108 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:37.099+0000 7f287959a640 1 -- 172.21.15.16:0/3002175767 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f286800c000 con 0x7f28680047d0 2023-05-22T15:46:37.108 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:37.099+0000 7f287959a640 1 -- 172.21.15.16:0/3002175767 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f286800c1c0 con 0x7f2868003980 2023-05-22T15:46:37.108 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:37.099+0000 7f287ad9d640 1 --2- 172.21.15.16:0/3002175767 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2868005180 0x7f286800b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:37.108 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:37.099+0000 7f287a59c640 1 --2- 172.21.15.16:0/3002175767 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2868003980 0x7f2868003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:37.108 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:37.099+0000 7f287ad9d640 1 -- 172.21.15.16:0/3002175767 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2868003980 msgr2=0x7f2868003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:46:37.108 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:37.099+0000 7f287ad9d640 1 --2- 172.21.15.16:0/3002175767 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2868003980 0x7f2868003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:37.108 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:37.099+0000 7f287ad9d640 1 -- 172.21.15.16:0/3002175767 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f28680047d0 msgr2=0x7f2868004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:46:37.109 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:37.099+0000 7f287ad9d640 1 --2- 172.21.15.16:0/3002175767 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f28680047d0 0x7f2868004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:37.109 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:37.099+0000 7f287ad9d640 1 -- 172.21.15.16:0/3002175767 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2868003530 con 0x7f2868005180 2023-05-22T15:46:37.111 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:37.103+0000 7f287ad9d640 1 -- 172.21.15.16:0/3002175767 reap_dead start 2023-05-22T15:46:37.111 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:37.103+0000 7f287ad9d640 1 --2- 172.21.15.16:0/3002175767 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2868005180 0x7f286800b990 secure :-1 s=READY pgs=1875 cs=0 l=1 rev1=1 crypto rx=0x7f286c009ae0 tx=0x7f286c01ca90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26cd4bb24514a28d server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:37.111 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:37.103+0000 7f287959a640 1 -- 172.21.15.16:0/3002175767 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f286c004020 con 0x7f2868005180 2023-05-22T15:46:37.111 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:37.103+0000 7f287959a640 1 -- 172.21.15.16:0/3002175767 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f286c0041c0 con 0x7f2868005180 2023-05-22T15:46:37.111 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:37.103+0000 7f287959a640 1 -- 172.21.15.16:0/3002175767 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f286c0a5680 con 0x7f2868005180 2023-05-22T15:46:37.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:37.103+0000 7f2880a5b640 1 -- 172.21.15.16:0/3002175767 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2868005180 msgr2=0x7f286800b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:37.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:37.103+0000 7f2880a5b640 1 --2- 172.21.15.16:0/3002175767 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2868005180 0x7f286800b990 secure :-1 s=READY pgs=1875 cs=0 l=1 rev1=1 crypto rx=0x7f286c009ae0 tx=0x7f286c01ca90 comp rx=0 tx=0).stop 2023-05-22T15:46:37.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:37.103+0000 7f2880a5b640 1 -- 172.21.15.16:0/3002175767 shutdown_connections 2023-05-22T15:46:37.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:37.103+0000 7f2880a5b640 1 --2- 172.21.15.16:0/3002175767 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2868005180 0x7f286800b990 unknown :-1 s=CLOSED pgs=1875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:37.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:37.103+0000 7f2880a5b640 1 -- 172.21.15.16:0/3002175767 >> 172.21.15.16:0/3002175767 conn(0x7f287c0655c0 msgr2=0x7f287c070e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:46:37.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:37.103+0000 7f2880a5b640 1 -- 172.21.15.16:0/3002175767 shutdown_connections 2023-05-22T15:46:37.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:37.103+0000 7f2880a5b640 1 -- 172.21.15.16:0/3002175767 wait complete. 2023-05-22T15:46:37.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:37.103+0000 7f2880a5b640 1 Processor -- start 2023-05-22T15:46:37.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:37.103+0000 7f2880a5b640 1 -- start start 2023-05-22T15:46:37.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:37.103+0000 7f2880a5b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2868005180 0x7f287c0b2e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:37.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:37.103+0000 7f2880a5b640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f287c1557a0 0x7f287c155bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:37.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:37.103+0000 7f2880a5b640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f287c151450 0x7f287c1518c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:37.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:37.103+0000 7f2880a5b640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f287c0ce190 con 0x7f2868005180 2023-05-22T15:46:37.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:37.103+0000 7f2880a5b640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f287c0cab30 con 0x7f287c151450 2023-05-22T15:46:37.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:37.103+0000 7f2880a5b640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f287c0cae10 con 0x7f287c1557a0 2023-05-22T15:46:37.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:37.103+0000 7f287a59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2868005180 0x7f287c0b2e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:37.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:37.103+0000 7f287a59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2868005180 0x7f287c0b2e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56648/0 (socket says 172.21.15.16:56648) 2023-05-22T15:46:37.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:37.103+0000 7f287a59c640 1 -- 172.21.15.16:0/952065490 learned_addr learned my addr 172.21.15.16:0/952065490 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:46:37.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:37.103+0000 7f287ad9d640 1 --2- 172.21.15.16:0/952065490 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f287c151450 0x7f287c1518c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:37.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:37.103+0000 7f2879d9b640 1 --2- 172.21.15.16:0/952065490 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f287c1557a0 0x7f287c155bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:37.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:37.103+0000 7f287a59c640 1 -- 172.21.15.16:0/952065490 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f287c1557a0 msgr2=0x7f287c155bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:37.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:37.103+0000 7f287a59c640 1 --2- 172.21.15.16:0/952065490 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f287c1557a0 0x7f287c155bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:37.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:37.103+0000 7f287a59c640 1 -- 172.21.15.16:0/952065490 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f287c151450 msgr2=0x7f287c1518c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:37.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:37.103+0000 7f287a59c640 1 --2- 172.21.15.16:0/952065490 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f287c151450 0x7f287c1518c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:37.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:37.103+0000 7f287a59c640 1 -- 172.21.15.16:0/952065490 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f287c0d2900 con 0x7f2868005180 2023-05-22T15:46:37.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:37.103+0000 7f287ad9d640 1 --2- 172.21.15.16:0/952065490 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f287c151450 0x7f287c1518c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:46:37.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:37.103+0000 7f287a59c640 1 --2- 172.21.15.16:0/952065490 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2868005180 0x7f287c0b2e70 secure :-1 s=READY pgs=1876 cs=0 l=1 rev1=1 crypto rx=0x7f2874007a10 tx=0x7f287400e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:37.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:37.107+0000 7f2872ffd640 1 -- 172.21.15.16:0/952065490 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f287400ee50 con 0x7f2868005180 2023-05-22T15:46:37.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:37.107+0000 7f2872ffd640 1 -- 172.21.15.16:0/952065490 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2874013420 con 0x7f2868005180 2023-05-22T15:46:37.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:37.107+0000 7f2880a5b640 1 -- 172.21.15.16:0/952065490 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f287c15c070 con 0x7f2868005180 2023-05-22T15:46:37.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:37.107+0000 7f2872ffd640 1 -- 172.21.15.16:0/952065490 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28740137d0 con 0x7f2868005180 2023-05-22T15:46:37.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:37.107+0000 7f2880a5b640 1 -- 172.21.15.16:0/952065490 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f287c0d2a90 con 0x7f2868005180 2023-05-22T15:46:37.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:37.107+0000 7f2880a5b640 1 -- 172.21.15.16:0/952065490 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f287c04dbe0 con 0x7f2868005180 2023-05-22T15:46:37.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:37.107+0000 7f2872ffd640 1 -- 172.21.15.16:0/952065490 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 60) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f2874013930 con 0x7f2868005180 2023-05-22T15:46:37.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:37.107+0000 7f2872ffd640 1 --2- 172.21.15.16:0/952065490 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f285c0654a0 0x7f285c067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:37.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:37.107+0000 7f2879d9b640 1 --2- 172.21.15.16:0/952065490 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f285c0654a0 0x7f285c067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:37.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:37.107+0000 7f2872ffd640 1 -- 172.21.15.16:0/952065490 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(147..147 src has 1..147) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f2874083ef0 con 0x7f2868005180 2023-05-22T15:46:37.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:37.107+0000 7f2879d9b640 1 --2- 172.21.15.16:0/952065490 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f285c0654a0 0x7f285c067950 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f287c157f00 tx=0x7f2864009040 comp rx=0 tx=0).ready entity=mgr.8835 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:37.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:37.111+0000 7f2872ffd640 1 -- 172.21.15.16:0/952065490 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f2874051850 con 0x7f2868005180 2023-05-22T15:46:37.312 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:37.307+0000 7f2880a5b640 1 -- 172.21.15.16:0/952065490 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7f287c0b4a70 con 0x7f2868005180 2023-05-22T15:46:37.619 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:37.615+0000 7f2872ffd640 1 -- 172.21.15.16:0/952065490 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mgrmap(e 61) v1 ==== 84349+0+0 (secure 0 0 0) 0x7f2874051c20 con 0x7f2868005180 2023-05-22T15:46:37.621 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:37.615+0000 7f2872ffd640 1 -- 172.21.15.16:0/952065490 <== mon.0 v2:172.21.15.16:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v61) v1 ==== 92+0+0 (secure 0 0 0) 0x7f2874061dd0 con 0x7f2868005180 2023-05-22T15:46:37.625 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:37.619+0000 7f2880a5b640 1 -- 172.21.15.16:0/952065490 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f285c0654a0 msgr2=0x7f285c067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:37.625 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:37.619+0000 7f2880a5b640 1 --2- 172.21.15.16:0/952065490 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f285c0654a0 0x7f285c067950 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f287c157f00 tx=0x7f2864009040 comp rx=0 tx=0).stop 2023-05-22T15:46:37.625 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:37.619+0000 7f2880a5b640 1 -- 172.21.15.16:0/952065490 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2868005180 msgr2=0x7f287c0b2e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:37.625 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:37.619+0000 7f2880a5b640 1 --2- 172.21.15.16:0/952065490 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2868005180 0x7f287c0b2e70 secure :-1 s=READY pgs=1876 cs=0 l=1 rev1=1 crypto rx=0x7f2874007a10 tx=0x7f287400e530 comp rx=0 tx=0).stop 2023-05-22T15:46:37.626 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:37.619+0000 7f2880a5b640 1 -- 172.21.15.16:0/952065490 shutdown_connections 2023-05-22T15:46:37.626 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:37.619+0000 7f2880a5b640 1 --2- 172.21.15.16:0/952065490 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f287c151450 0x7f287c1518c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:37.626 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:37.619+0000 7f2880a5b640 1 --2- 172.21.15.16:0/952065490 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f285c0654a0 0x7f285c067950 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:37.626 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:37.619+0000 7f2880a5b640 1 --2- 172.21.15.16:0/952065490 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f287c1557a0 0x7f287c155bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:37.626 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:37.619+0000 7f2880a5b640 1 --2- 172.21.15.16:0/952065490 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2868005180 0x7f287c0b2e70 unknown :-1 s=CLOSED pgs=1876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:37.626 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:37.619+0000 7f2880a5b640 1 -- 172.21.15.16:0/952065490 >> 172.21.15.16:0/952065490 conn(0x7f287c0655c0 msgr2=0x7f287c070e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:46:37.626 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:37.619+0000 7f2880a5b640 1 -- 172.21.15.16:0/952065490 shutdown_connections 2023-05-22T15:46:37.627 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:37.619+0000 7f2880a5b640 1 -- 172.21.15.16:0/952065490 wait complete. 2023-05-22T15:46:37.644 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2023-05-22T15:46:37.745 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:46:37.744+0000 7f12b413c180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-05-22T15:46:37.752 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:46:37.747+0000 7fe213ff5180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-05-22T15:46:37.766 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:37.759+0000 7f30822cf640 1 Processor -- start 2023-05-22T15:46:37.767 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:37.759+0000 7f30822cf640 1 -- start start 2023-05-22T15:46:37.768 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:37.763+0000 7f30822cf640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f307c0c67d0 0x7f307c0cf730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:37.769 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:37.763+0000 7f30822cf640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f307c0cc6f0 0x7f307c0c3db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:37.771 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:37.763+0000 7f30822cf640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f307c0cfc70 con 0x7f307c0b2880 2023-05-22T15:46:37.771 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:37.763+0000 7f30822cf640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f307c15a300 con 0x7f307c0c67d0 2023-05-22T15:46:37.772 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:37.763+0000 7f30822cf640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f307c0c6ba0 con 0x7f307c0cc6f0 2023-05-22T15:46:37.772 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:37.763+0000 7f30737fe640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f307c0b2880 0x7f307c0cbfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:54950/0 (socket says 172.21.15.16:54950) 2023-05-22T15:46:37.772 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:37.763+0000 7f30737fe640 1 -- 172.21.15.16:0/2325107284 learned_addr learned my addr 172.21.15.16:0/2325107284 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:46:37.772 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:37.763+0000 7f308086c640 1 --2- 172.21.15.16:0/2325107284 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f307c0cc6f0 0x7f307c0c3db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:37.773 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:37.763+0000 7f307bfff640 1 --2- 172.21.15.16:0/2325107284 >> v2:172.21.15.62:3300/0 conn(0x7f307c0c67d0 0x7f307c0cf730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:37.773 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:37.763+0000 7f307b7fe640 1 -- 172.21.15.16:0/2325107284 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4003808265 0 0) 0x7f307c0cfc70 con 0x7f307c0b2880 2023-05-22T15:46:37.773 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:37.763+0000 7f307b7fe640 1 -- 172.21.15.16:0/2325107284 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3068003530 con 0x7f307c0b2880 2023-05-22T15:46:37.773 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:37.763+0000 7f307b7fe640 1 -- 172.21.15.16:0/2325107284 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 262799603 0 0) 0x7f3068003530 con 0x7f307c0b2880 2023-05-22T15:46:37.774 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:37.763+0000 7f307b7fe640 1 -- 172.21.15.16:0/2325107284 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f307c0cc6f0 msgr2=0x7f307c0c3db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:46:37.774 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:37.763+0000 7f307b7fe640 1 --2- 172.21.15.16:0/2325107284 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f307c0cc6f0 0x7f307c0c3db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:37.774 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:37.763+0000 7f307b7fe640 1 -- 172.21.15.16:0/2325107284 >> v2:172.21.15.62:3300/0 conn(0x7f307c0c67d0 msgr2=0x7f307c0cf730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:46:37.774 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:37.763+0000 7f307b7fe640 1 --2- 172.21.15.16:0/2325107284 >> v2:172.21.15.62:3300/0 conn(0x7f307c0c67d0 0x7f307c0cf730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:37.775 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:37.763+0000 7f307b7fe640 1 -- 172.21.15.16:0/2325107284 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f307c0cfef0 con 0x7f307c0b2880 2023-05-22T15:46:37.775 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:37.763+0000 7f307b7fe640 1 -- 172.21.15.16:0/2325107284 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7f30640016a0 con 0x7f307c0b2880 2023-05-22T15:46:37.775 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:37.763+0000 7f307b7fe640 1 -- 172.21.15.16:0/2325107284 >> v1:172.21.15.16:6789/0 conn(0x7f307c0b2880 legacy=0x7f307c0cbfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:37.775 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:37.763+0000 7f307b7fe640 1 --2- 172.21.15.16:0/2325107284 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f307c0cc6f0 0x7f3068003980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:37.776 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:37.763+0000 7f307b7fe640 1 --2- 172.21.15.16:0/2325107284 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3068003f00 0x7f30680047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:37.776 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:37.763+0000 7f307b7fe640 1 --2- 172.21.15.16:0/2325107284 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3068004d10 0x7f3068005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:37.776 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:37.763+0000 7f307b7fe640 1 -- 172.21.15.16:0/2325107284 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f307c15a300 con 0x7f3068004d10 2023-05-22T15:46:37.776 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:37.763+0000 7f307b7fe640 1 -- 172.21.15.16:0/2325107284 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f307c0c6ba0 con 0x7f3068003f00 2023-05-22T15:46:37.777 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:37.763+0000 7f307b7fe640 1 -- 172.21.15.16:0/2325107284 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f306800b9c0 con 0x7f307c0cc6f0 2023-05-22T15:46:37.777 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:37.763+0000 7f308086c640 1 --2- 172.21.15.16:0/2325107284 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3068004d10 0x7f3068005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:37.777 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:37.763+0000 7f30737fe640 1 --2- 172.21.15.16:0/2325107284 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3068003f00 0x7f30680047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:37.778 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:37.763+0000 7f307bfff640 1 --2- 172.21.15.16:0/2325107284 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f307c0cc6f0 0x7f3068003980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:37.778 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:37.763+0000 7f308086c640 1 -- 172.21.15.16:0/2325107284 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f307c0cc6f0 msgr2=0x7f3068003980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:46:37.778 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:37.763+0000 7f308086c640 1 --2- 172.21.15.16:0/2325107284 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f307c0cc6f0 0x7f3068003980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:37.778 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:37.763+0000 7f308086c640 1 -- 172.21.15.16:0/2325107284 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3068003f00 msgr2=0x7f30680047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:46:37.779 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:37.763+0000 7f308086c640 1 --2- 172.21.15.16:0/2325107284 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3068003f00 0x7f30680047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:37.779 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:37.763+0000 7f308086c640 1 -- 172.21.15.16:0/2325107284 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3068003530 con 0x7f3068004d10 2023-05-22T15:46:37.784 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:37.763+0000 7f308086c640 1 --2- 172.21.15.16:0/2325107284 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3068004d10 0x7f3068005180 secure :-1 s=READY pgs=1882 cs=0 l=1 rev1=1 crypto rx=0x7f3074009fc0 tx=0x7f307401cc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a74b9eb08e12e950 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:37.784 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:37.767+0000 7f307b7fe640 1 -- 172.21.15.16:0/2325107284 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30740a5b60 con 0x7f3068004d10 2023-05-22T15:46:37.784 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:37.767+0000 7f307b7fe640 1 -- 172.21.15.16:0/2325107284 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f307400f040 con 0x7f3068004d10 2023-05-22T15:46:37.784 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:37.767+0000 7f307b7fe640 1 -- 172.21.15.16:0/2325107284 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30740ad440 con 0x7f3068004d10 2023-05-22T15:46:37.784 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:37.767+0000 7f30822cf640 1 -- 172.21.15.16:0/2325107284 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3068004d10 msgr2=0x7f3068005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:37.785 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:37.767+0000 7f30822cf640 1 --2- 172.21.15.16:0/2325107284 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3068004d10 0x7f3068005180 secure :-1 s=READY pgs=1882 cs=0 l=1 rev1=1 crypto rx=0x7f3074009fc0 tx=0x7f307401cc00 comp rx=0 tx=0).stop 2023-05-22T15:46:37.785 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:37.767+0000 7f308086c640 1 -- 172.21.15.16:0/2325107284 reap_dead start 2023-05-22T15:46:37.785 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:37.767+0000 7f30822cf640 1 -- 172.21.15.16:0/2325107284 shutdown_connections 2023-05-22T15:46:37.785 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:37.767+0000 7f30822cf640 1 -- 172.21.15.16:0/2325107284 >> 172.21.15.16:0/2325107284 conn(0x7f307c06f5a0 msgr2=0x7f307c06f9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:46:37.785 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:37.767+0000 7f30822cf640 1 -- 172.21.15.16:0/2325107284 shutdown_connections 2023-05-22T15:46:37.785 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:37.767+0000 7f30822cf640 1 -- 172.21.15.16:0/2325107284 wait complete. 2023-05-22T15:46:37.785 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:37.767+0000 7f30822cf640 1 Processor -- start 2023-05-22T15:46:37.786 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:37.767+0000 7f30822cf640 1 -- start start 2023-05-22T15:46:37.786 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:37.771+0000 7f30822cf640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f307c14e7b0 0x7f307c15a440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:37.786 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:37.771+0000 7f30822cf640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f307c15b770 0x7f307c1e35b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:37.787 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:37.771+0000 7f30822cf640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f307c1e3af0 0x7f307c0b3650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:37.787 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:37.771+0000 7f30822cf640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f307c14eb80 con 0x7f307c15b770 2023-05-22T15:46:37.787 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:37.771+0000 7f30822cf640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f307c1e3f60 con 0x7f307c1e3af0 2023-05-22T15:46:37.789 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:37.771+0000 7f30822cf640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f307c15b370 con 0x7f307c14e7b0 2023-05-22T15:46:37.789 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:37.771+0000 7f30737fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f307c15b770 0x7f307c1e35b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:37.790 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:37.771+0000 7f30737fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f307c15b770 0x7f307c1e35b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56702/0 (socket says 172.21.15.16:56702) 2023-05-22T15:46:37.790 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:37.771+0000 7f30737fe640 1 -- 172.21.15.16:0/2476435377 learned_addr learned my addr 172.21.15.16:0/2476435377 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:46:37.790 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:37.771+0000 7f307bfff640 1 --2- 172.21.15.16:0/2476435377 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f307c14e7b0 0x7f307c15a440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:37.790 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:37.771+0000 7f308086c640 1 --2- 172.21.15.16:0/2476435377 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f307c1e3af0 0x7f307c0b3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:37.790 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:37.771+0000 7f30737fe640 1 -- 172.21.15.16:0/2476435377 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f307c14e7b0 msgr2=0x7f307c15a440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:37.790 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:37.771+0000 7f30737fe640 1 --2- 172.21.15.16:0/2476435377 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f307c14e7b0 0x7f307c15a440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:37.790 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:37.771+0000 7f30737fe640 1 -- 172.21.15.16:0/2476435377 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f307c1e3af0 msgr2=0x7f307c0b3650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:37.791 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:37.771+0000 7f30737fe640 1 --2- 172.21.15.16:0/2476435377 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f307c1e3af0 0x7f307c0b3650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:37.791 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:37.771+0000 7f30737fe640 1 -- 172.21.15.16:0/2476435377 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f307c0d04b0 con 0x7f307c15b770 2023-05-22T15:46:37.791 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:37.771+0000 7f30737fe640 1 --2- 172.21.15.16:0/2476435377 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f307c15b770 0x7f307c1e35b0 secure :-1 s=READY pgs=1883 cs=0 l=1 rev1=1 crypto rx=0x7f30640119b0 tx=0x7f3064011e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:37.791 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:37.771+0000 7f30797fa640 1 -- 172.21.15.16:0/2476435377 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f306400fbc0 con 0x7f307c15b770 2023-05-22T15:46:37.791 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:37.771+0000 7f30797fa640 1 -- 172.21.15.16:0/2476435377 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f306400fd20 con 0x7f307c15b770 2023-05-22T15:46:37.791 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:37.771+0000 7f30822cf640 1 -- 172.21.15.16:0/2476435377 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f307c0d01d0 con 0x7f307c15b770 2023-05-22T15:46:37.791 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:37.771+0000 7f30797fa640 1 -- 172.21.15.16:0/2476435377 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3064015650 con 0x7f307c15b770 2023-05-22T15:46:37.792 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:37.771+0000 7f30822cf640 1 -- 172.21.15.16:0/2476435377 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f307c0d0790 con 0x7f307c15b770 2023-05-22T15:46:37.792 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:37.771+0000 7f30822cf640 1 -- 172.21.15.16:0/2476435377 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f307c04dbe0 con 0x7f307c15b770 2023-05-22T15:46:37.792 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:37.771+0000 7f30797fa640 1 -- 172.21.15.16:0/2476435377 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 61) v1 ==== 84349+0+0 (secure 0 0 0) 0x7f30640157b0 con 0x7f307c15b770 2023-05-22T15:46:37.792 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:37.771+0000 7f30797fa640 1 --2- 172.21.15.16:0/2476435377 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f305c0654f0 0x7f305c0679a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:37.792 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:37.771+0000 7f307bfff640 1 -- 172.21.15.16:0/2476435377 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f305c0654f0 msgr2=0x7f305c0679a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/36921 2023-05-22T15:46:37.792 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:37.771+0000 7f30797fa640 1 -- 172.21.15.16:0/2476435377 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(147..147 src has 1..147) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f3064085eb0 con 0x7f307c15b770 2023-05-22T15:46:37.793 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:37.771+0000 7f307bfff640 1 --2- 172.21.15.16:0/2476435377 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f305c0654f0 0x7f305c0679a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-05-22T15:46:37.793 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:37.775+0000 7f30797fa640 1 -- 172.21.15.16:0/2476435377 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f30640538c0 con 0x7f307c15b770 2023-05-22T15:46:37.936 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:37.931+0000 7f30822cf640 1 -- 172.21.15.16:0/2476435377 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7f307c14ecc0 con 0x7f307c15b770 2023-05-22T15:46:37.941 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:37.935+0000 7f30797fa640 1 -- 172.21.15.16:0/2476435377 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v47)=0 v47) v1 ==== 138+0+0 (secure 0 0 0) 0x7f3064056ee0 con 0x7f307c15b770 2023-05-22T15:46:37.943 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:37.939+0000 7f30822cf640 1 -- 172.21.15.16:0/2476435377 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f305c0654f0 msgr2=0x7f305c0679a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-05-22T15:46:37.944 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:37.939+0000 7f30822cf640 1 --2- 172.21.15.16:0/2476435377 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f305c0654f0 0x7f305c0679a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:37.944 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:37.939+0000 7f30822cf640 1 -- 172.21.15.16:0/2476435377 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f307c15b770 msgr2=0x7f307c1e35b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:37.944 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:37.939+0000 7f30822cf640 1 --2- 172.21.15.16:0/2476435377 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f307c15b770 0x7f307c1e35b0 secure :-1 s=READY pgs=1883 cs=0 l=1 rev1=1 crypto rx=0x7f30640119b0 tx=0x7f3064011e70 comp rx=0 tx=0).stop 2023-05-22T15:46:37.944 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:37.939+0000 7f30822cf640 1 -- 172.21.15.16:0/2476435377 shutdown_connections 2023-05-22T15:46:37.944 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:37.939+0000 7f30822cf640 1 --2- 172.21.15.16:0/2476435377 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f307c1e3af0 0x7f307c0b3650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:37.944 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:37.939+0000 7f30822cf640 1 --2- 172.21.15.16:0/2476435377 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f307c15b770 0x7f307c1e35b0 unknown :-1 s=CLOSED pgs=1883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:37.945 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:37.939+0000 7f30822cf640 1 --2- 172.21.15.16:0/2476435377 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f305c0654f0 0x7f305c0679a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:37.945 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:37.939+0000 7f30822cf640 1 --2- 172.21.15.16:0/2476435377 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f307c14e7b0 0x7f307c15a440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:37.945 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:37.939+0000 7f30822cf640 1 -- 172.21.15.16:0/2476435377 >> 172.21.15.16:0/2476435377 conn(0x7f307c06f5a0 msgr2=0x7f307c0cd050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:46:37.945 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:37.939+0000 7f30822cf640 1 -- 172.21.15.16:0/2476435377 shutdown_connections 2023-05-22T15:46:37.945 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:37.939+0000 7f30822cf640 1 -- 172.21.15.16:0/2476435377 wait complete. 2023-05-22T15:46:37.964 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2023-05-22T15:46:38.083 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:38.079+0000 7fd3c087f640 1 Processor -- start 2023-05-22T15:46:38.083 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:38.079+0000 7fd3c087f640 1 -- start start 2023-05-22T15:46:38.084 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:38.079+0000 7fd3c087f640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd3bc0ce480 0x7fd3bc071640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:38.084 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:38.079+0000 7fd3c087f640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3bc15cde0 con 0x7fd3bc15c270 2023-05-22T15:46:38.085 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:38.079+0000 7fd3c087f640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3bc069ae0 con 0x7fd3bc0ce850 2023-05-22T15:46:38.085 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:38.079+0000 7fd3c087f640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3bc15f990 con 0x7fd3bc0ce480 2023-05-22T15:46:38.085 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:38.079+0000 7fd3ba59c640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd3bc0ce480 0x7fd3bc071640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:38.085 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:38.079+0000 7fd3ba59c640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd3bc0ce480 0x7fd3bc071640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56708/0 (socket says 172.21.15.16:56708) 2023-05-22T15:46:38.085 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:38.079+0000 7fd3ba59c640 1 -- 172.21.15.16:0/2772110500 learned_addr learned my addr 172.21.15.16:0/2772110500 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:46:38.085 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:38.079+0000 7fd3ba59c640 1 -- 172.21.15.16:0/2772110500 >> v1:172.21.15.62:6789/0 conn(0x7fd3bc0ce850 legacy=0x7fd3bc160090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:38.085 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:38.079+0000 7fd3ba59c640 1 -- 172.21.15.16:0/2772110500 >> v1:172.21.15.16:6789/0 conn(0x7fd3bc15c270 legacy=0x7fd3bc15c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:38.086 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:38.079+0000 7fd3ba59c640 1 -- 172.21.15.16:0/2772110500 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3bc069dc0 con 0x7fd3bc0ce480 2023-05-22T15:46:38.086 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:38.079+0000 7fd3ba59c640 1 --2- 172.21.15.16:0/2772110500 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd3bc0ce480 0x7fd3bc071640 secure :-1 s=READY pgs=1885 cs=0 l=1 rev1=1 crypto rx=0x7fd3ac009870 tx=0x7fd3ac01c510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b65a6737f8f6466d server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:38.087 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:38.083+0000 7fd3b9d9b640 1 -- 172.21.15.16:0/2772110500 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3ac0aa670 con 0x7fd3bc0ce480 2023-05-22T15:46:38.087 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:38.083+0000 7fd3b9d9b640 1 -- 172.21.15.16:0/2772110500 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3ac0aa7d0 con 0x7fd3bc0ce480 2023-05-22T15:46:38.087 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:38.083+0000 7fd3b9d9b640 1 -- 172.21.15.16:0/2772110500 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3ac0aab60 con 0x7fd3bc0ce480 2023-05-22T15:46:38.088 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:38.083+0000 7fd3c087f640 1 -- 172.21.15.16:0/2772110500 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd3bc0ce480 msgr2=0x7fd3bc071640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:38.088 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:38.083+0000 7fd3c087f640 1 --2- 172.21.15.16:0/2772110500 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd3bc0ce480 0x7fd3bc071640 secure :-1 s=READY pgs=1885 cs=0 l=1 rev1=1 crypto rx=0x7fd3ac009870 tx=0x7fd3ac01c510 comp rx=0 tx=0).stop 2023-05-22T15:46:38.088 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:38.083+0000 7fd3c087f640 1 -- 172.21.15.16:0/2772110500 shutdown_connections 2023-05-22T15:46:38.088 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:38.083+0000 7fd3c087f640 1 --2- 172.21.15.16:0/2772110500 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd3bc0ce480 0x7fd3bc071640 unknown :-1 s=CLOSED pgs=1885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:38.089 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:38.083+0000 7fd3c087f640 1 -- 172.21.15.16:0/2772110500 >> 172.21.15.16:0/2772110500 conn(0x7fd3bc0cc870 msgr2=0x7fd3bc0cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:46:38.089 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:38.083+0000 7fd3c087f640 1 -- 172.21.15.16:0/2772110500 shutdown_connections 2023-05-22T15:46:38.089 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:38.083+0000 7fd3c087f640 1 -- 172.21.15.16:0/2772110500 wait complete. 2023-05-22T15:46:38.090 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:38.083+0000 7fd3c087f640 1 Processor -- start 2023-05-22T15:46:38.091 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:38.083+0000 7fd3c087f640 1 -- start start 2023-05-22T15:46:38.091 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:38.083+0000 7fd3c087f640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd3bc0ce480 0x7fd3bc159110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:38.091 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:38.083+0000 7fd3c087f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd3bc0ce850 0x7fd3bc0aee20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:38.091 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:38.083+0000 7fd3c087f640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd3bc15c270 0x7fd3bc0b4440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:38.091 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:38.083+0000 7fd3c087f640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3bc156420 con 0x7fd3bc0ce850 2023-05-22T15:46:38.091 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:38.083+0000 7fd3c087f640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3bc0b4020 con 0x7fd3bc0ce480 2023-05-22T15:46:38.091 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:38.083+0000 7fd3c087f640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3bc0af890 con 0x7fd3bc15c270 2023-05-22T15:46:38.092 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:38.083+0000 7fd3b3fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd3bc0ce850 0x7fd3bc0aee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:38.092 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:38.083+0000 7fd3b3fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd3bc0ce850 0x7fd3bc0aee20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56714/0 (socket says 172.21.15.16:56714) 2023-05-22T15:46:38.092 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:38.083+0000 7fd3b3fff640 1 -- 172.21.15.16:0/1052939941 learned_addr learned my addr 172.21.15.16:0/1052939941 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:46:38.092 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:38.083+0000 7fd3ba59c640 1 --2- 172.21.15.16:0/1052939941 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd3bc0ce480 0x7fd3bc159110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:38.092 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:38.083+0000 7fd3bad9d640 1 --2- 172.21.15.16:0/1052939941 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd3bc15c270 0x7fd3bc0b4440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:38.092 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:38.083+0000 7fd3b3fff640 1 -- 172.21.15.16:0/1052939941 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd3bc15c270 msgr2=0x7fd3bc0b4440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:38.092 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:38.083+0000 7fd3b3fff640 1 --2- 172.21.15.16:0/1052939941 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd3bc15c270 0x7fd3bc0b4440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:38.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:38.083+0000 7fd3b3fff640 1 -- 172.21.15.16:0/1052939941 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd3bc0ce480 msgr2=0x7fd3bc159110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:38.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:38.083+0000 7fd3b3fff640 1 --2- 172.21.15.16:0/1052939941 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd3bc0ce480 0x7fd3bc159110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:38.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:38.083+0000 7fd3b3fff640 1 -- 172.21.15.16:0/1052939941 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3ac0094d0 con 0x7fd3bc0ce850 2023-05-22T15:46:38.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:38.083+0000 7fd3b3fff640 1 --2- 172.21.15.16:0/1052939941 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd3bc0ce850 0x7fd3bc0aee20 secure :-1 s=READY pgs=1886 cs=0 l=1 rev1=1 crypto rx=0x7fd3a400c530 tx=0x7fd3a400c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:38.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:38.083+0000 7fd3b2ffd640 1 -- 172.21.15.16:0/1052939941 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3a4005220 con 0x7fd3bc0ce850 2023-05-22T15:46:38.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:38.083+0000 7fd3b2ffd640 1 -- 172.21.15.16:0/1052939941 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3a4005380 con 0x7fd3bc0ce850 2023-05-22T15:46:38.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:38.083+0000 7fd3c087f640 1 -- 172.21.15.16:0/1052939941 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3bc069720 con 0x7fd3bc0ce850 2023-05-22T15:46:38.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:38.083+0000 7fd3b2ffd640 1 -- 172.21.15.16:0/1052939941 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3a4007270 con 0x7fd3bc0ce850 2023-05-22T15:46:38.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:38.083+0000 7fd3c087f640 1 -- 172.21.15.16:0/1052939941 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3bc1607a0 con 0x7fd3bc0ce850 2023-05-22T15:46:38.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:38.087+0000 7fd3c087f640 1 -- 172.21.15.16:0/1052939941 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3bc0698b0 con 0x7fd3bc0ce850 2023-05-22T15:46:38.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:38.087+0000 7fd3b2ffd640 1 -- 172.21.15.16:0/1052939941 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 61) v1 ==== 84349+0+0 (secure 0 0 0) 0x7fd3a40073d0 con 0x7fd3bc0ce850 2023-05-22T15:46:38.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:38.087+0000 7fd3b2ffd640 1 --2- 172.21.15.16:0/1052939941 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd39c0654f0 0x7fd39c0679a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:38.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:38.087+0000 7fd3ba59c640 1 -- 172.21.15.16:0/1052939941 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd39c0654f0 msgr2=0x7fd39c0679a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/36921 2023-05-22T15:46:38.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:38.087+0000 7fd3b2ffd640 1 -- 172.21.15.16:0/1052939941 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(147..147 src has 1..147) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fd3a40831a0 con 0x7fd3bc0ce850 2023-05-22T15:46:38.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:38.087+0000 7fd3ba59c640 1 --2- 172.21.15.16:0/1052939941 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd39c0654f0 0x7fd39c0679a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-05-22T15:46:38.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:38.095+0000 7fd3b2ffd640 1 -- 172.21.15.16:0/1052939941 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fd3a4050d20 con 0x7fd3bc0ce850 2023-05-22T15:46:38.131 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:46:38.127+0000 7f12b413c180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-05-22T15:46:38.155 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:46:38.147+0000 7fe213ff5180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-05-22T15:46:38.258 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:38.251+0000 7fd3c087f640 1 -- 172.21.15.16:0/1052939941 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7fd3bc0b4160 con 0x7fd3bc0ce850 2023-05-22T15:46:38.263 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:38.259+0000 7fd3b2ffd640 1 -- 172.21.15.16:0/1052939941 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v48)=0 v48) v1 ==== 133+0+0 (secure 0 0 0) 0x7fd3bc0b4160 con 0x7fd3bc0ce850 2023-05-22T15:46:38.266 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:38.259+0000 7fd3c087f640 1 -- 172.21.15.16:0/1052939941 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd39c0654f0 msgr2=0x7fd39c0679a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-05-22T15:46:38.266 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:38.259+0000 7fd3c087f640 1 --2- 172.21.15.16:0/1052939941 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd39c0654f0 0x7fd39c0679a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:38.266 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:38.259+0000 7fd3c087f640 1 -- 172.21.15.16:0/1052939941 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd3bc0ce850 msgr2=0x7fd3bc0aee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:38.266 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:38.259+0000 7fd3c087f640 1 --2- 172.21.15.16:0/1052939941 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd3bc0ce850 0x7fd3bc0aee20 secure :-1 s=READY pgs=1886 cs=0 l=1 rev1=1 crypto rx=0x7fd3a400c530 tx=0x7fd3a400c9f0 comp rx=0 tx=0).stop 2023-05-22T15:46:38.267 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:38.259+0000 7fd3c087f640 1 -- 172.21.15.16:0/1052939941 shutdown_connections 2023-05-22T15:46:38.267 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:38.259+0000 7fd3c087f640 1 --2- 172.21.15.16:0/1052939941 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd39c0654f0 0x7fd39c0679a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:38.267 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:38.259+0000 7fd3c087f640 1 --2- 172.21.15.16:0/1052939941 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd3bc15c270 0x7fd3bc0b4440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:38.267 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:38.259+0000 7fd3c087f640 1 --2- 172.21.15.16:0/1052939941 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd3bc0ce850 0x7fd3bc0aee20 unknown :-1 s=CLOSED pgs=1886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:38.267 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:38.259+0000 7fd3c087f640 1 --2- 172.21.15.16:0/1052939941 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd3bc0ce480 0x7fd3bc159110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:38.267 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:38.259+0000 7fd3c087f640 1 -- 172.21.15.16:0/1052939941 >> 172.21.15.16:0/1052939941 conn(0x7fd3bc0cc870 msgr2=0x7fd3bc0cd100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:46:38.267 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:38.259+0000 7fd3c087f640 1 -- 172.21.15.16:0/1052939941 shutdown_connections 2023-05-22T15:46:38.267 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:38.259+0000 7fd3c087f640 1 -- 172.21.15.16:0/1052939941 wait complete. 2023-05-22T15:46:38.268 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:46:38.263+0000 7f12b413c180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-05-22T15:46:38.282 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:46:38.283 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1) 2023-05-22T15:46:38.294 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:46:38.287+0000 7fe213ff5180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-05-22T15:46:38.295 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add --fs cephfs snap_test_dir1 1h 2023-05-22T15:46:38.454 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:38.447+0000 7fc19676f640 1 Processor -- start 2023-05-22T15:46:38.454 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:38.447+0000 7fc19676f640 1 -- start start 2023-05-22T15:46:38.455 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:38.451+0000 7fc19676f640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fc19015b9e0 0x7fc1900adca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:38.455 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:38.451+0000 7fc19676f640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc190070600 0x7fc1900cad80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:38.455 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:38.451+0000 7fc19676f640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1900ce740 con 0x7fc19015bdb0 2023-05-22T15:46:38.456 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:38.451+0000 7fc19676f640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1900ce460 con 0x7fc19015b9e0 2023-05-22T15:46:38.456 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:38.451+0000 7fc19676f640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1900caae0 con 0x7fc190070600 2023-05-22T15:46:38.456 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:38.451+0000 7fc18f7fe640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fc19015bdb0 0x7fc19006fef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:54970/0 (socket says 172.21.15.16:54970) 2023-05-22T15:46:38.456 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:38.451+0000 7fc18f7fe640 1 -- 172.21.15.16:0/1054515775 learned_addr learned my addr 172.21.15.16:0/1054515775 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:46:38.462 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:38.451+0000 7fc194d0c640 1 --2- 172.21.15.16:0/1054515775 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc190070600 0x7fc1900cad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:38.462 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:38.451+0000 7fc18ffff640 1 --2- 172.21.15.16:0/1054515775 >> v2:172.21.15.62:3300/0 conn(0x7fc19015b9e0 0x7fc1900adca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:38.462 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:38.451+0000 7fc18effd640 1 -- 172.21.15.16:0/1054515775 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1045384513 0 0) 0x7fc1900ce740 con 0x7fc19015bdb0 2023-05-22T15:46:38.462 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:38.451+0000 7fc18effd640 1 -- 172.21.15.16:0/1054515775 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc178003530 con 0x7fc19015bdb0 2023-05-22T15:46:38.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:38.451+0000 7fc18effd640 1 -- 172.21.15.16:0/1054515775 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 435858239 0 0) 0x7fc178003530 con 0x7fc19015bdb0 2023-05-22T15:46:38.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:38.451+0000 7fc18effd640 1 -- 172.21.15.16:0/1054515775 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc190070600 msgr2=0x7fc1900cad80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:46:38.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:38.451+0000 7fc18effd640 1 --2- 172.21.15.16:0/1054515775 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc190070600 0x7fc1900cad80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:38.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:38.451+0000 7fc18effd640 1 -- 172.21.15.16:0/1054515775 >> v2:172.21.15.62:3300/0 conn(0x7fc19015b9e0 msgr2=0x7fc1900adca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:46:38.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:38.451+0000 7fc18effd640 1 --2- 172.21.15.16:0/1054515775 >> v2:172.21.15.62:3300/0 conn(0x7fc19015b9e0 0x7fc1900adca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:38.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:38.451+0000 7fc18effd640 1 -- 172.21.15.16:0/1054515775 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1900cde10 con 0x7fc19015bdb0 2023-05-22T15:46:38.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:38.451+0000 7fc18effd640 1 -- 172.21.15.16:0/1054515775 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7fc184001680 con 0x7fc19015bdb0 2023-05-22T15:46:38.464 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:38.451+0000 7fc18effd640 1 -- 172.21.15.16:0/1054515775 >> v1:172.21.15.16:6789/0 conn(0x7fc19015bdb0 legacy=0x7fc19006fef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:38.464 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:38.451+0000 7fc18effd640 1 --2- 172.21.15.16:0/1054515775 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc178003980 0x7fc178003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:38.464 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:38.451+0000 7fc18effd640 1 --2- 172.21.15.16:0/1054515775 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc190070600 0x7fc1780047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:38.464 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:38.451+0000 7fc18effd640 1 --2- 172.21.15.16:0/1054515775 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc178004d10 0x7fc178005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:38.464 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:38.451+0000 7fc18effd640 1 -- 172.21.15.16:0/1054515775 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1900ce460 con 0x7fc178004d10 2023-05-22T15:46:38.464 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:38.451+0000 7fc18effd640 1 -- 172.21.15.16:0/1054515775 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1900caae0 con 0x7fc178003980 2023-05-22T15:46:38.464 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:38.451+0000 7fc18effd640 1 -- 172.21.15.16:0/1054515775 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc17800b990 con 0x7fc190070600 2023-05-22T15:46:38.465 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:38.451+0000 7fc194d0c640 1 --2- 172.21.15.16:0/1054515775 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc178004d10 0x7fc178005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:38.465 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:38.451+0000 7fc18ffff640 1 --2- 172.21.15.16:0/1054515775 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc178003980 0x7fc178003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:38.465 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:38.451+0000 7fc18f7fe640 1 --2- 172.21.15.16:0/1054515775 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc190070600 0x7fc1780047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:38.465 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:38.451+0000 7fc194d0c640 1 -- 172.21.15.16:0/1054515775 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc190070600 msgr2=0x7fc1780047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:46:38.465 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:38.451+0000 7fc194d0c640 1 --2- 172.21.15.16:0/1054515775 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc190070600 0x7fc1780047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:38.465 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:38.451+0000 7fc194d0c640 1 -- 172.21.15.16:0/1054515775 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc178003980 msgr2=0x7fc178003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:46:38.465 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:38.451+0000 7fc194d0c640 1 --2- 172.21.15.16:0/1054515775 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc178003980 0x7fc178003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:38.465 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:38.451+0000 7fc194d0c640 1 -- 172.21.15.16:0/1054515775 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc178003530 con 0x7fc178004d10 2023-05-22T15:46:38.466 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:38.451+0000 7fc18f7fe640 1 --2- 172.21.15.16:0/1054515775 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc190070600 0x7fc1780047d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:46:38.466 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:38.451+0000 7fc18ffff640 1 --2- 172.21.15.16:0/1054515775 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc178003980 0x7fc178003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T15:46:38.466 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:38.455+0000 7fc194d0c640 1 --2- 172.21.15.16:0/1054515775 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc178004d10 0x7fc178005180 secure :-1 s=READY pgs=1888 cs=0 l=1 rev1=1 crypto rx=0x7fc188009fc0 tx=0x7fc18801cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4cfe8a390250750d server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:38.466 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:38.455+0000 7fc18effd640 1 -- 172.21.15.16:0/1054515775 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1880a5b60 con 0x7fc178004d10 2023-05-22T15:46:38.466 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:38.455+0000 7fc18effd640 1 -- 172.21.15.16:0/1054515775 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc18800f040 con 0x7fc178004d10 2023-05-22T15:46:38.466 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:38.455+0000 7fc18effd640 1 -- 172.21.15.16:0/1054515775 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1880ad440 con 0x7fc178004d10 2023-05-22T15:46:38.466 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:38.455+0000 7fc19676f640 1 -- 172.21.15.16:0/1054515775 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc178004d10 msgr2=0x7fc178005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:38.466 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:38.455+0000 7fc19676f640 1 --2- 172.21.15.16:0/1054515775 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc178004d10 0x7fc178005180 secure :-1 s=READY pgs=1888 cs=0 l=1 rev1=1 crypto rx=0x7fc188009fc0 tx=0x7fc18801cc60 comp rx=0 tx=0).stop 2023-05-22T15:46:38.467 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:38.455+0000 7fc194d0c640 1 -- 172.21.15.16:0/1054515775 reap_dead start 2023-05-22T15:46:38.467 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:38.455+0000 7fc19676f640 1 -- 172.21.15.16:0/1054515775 shutdown_connections 2023-05-22T15:46:38.467 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:38.455+0000 7fc19676f640 1 -- 172.21.15.16:0/1054515775 >> 172.21.15.16:0/1054515775 conn(0x7fc1900ce8d0 msgr2=0x7fc1900cecd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:46:38.467 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:38.455+0000 7fc19676f640 1 -- 172.21.15.16:0/1054515775 shutdown_connections 2023-05-22T15:46:38.467 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:38.455+0000 7fc19676f640 1 -- 172.21.15.16:0/1054515775 wait complete. 2023-05-22T15:46:38.472 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:38.455+0000 7fc19676f640 1 Processor -- start 2023-05-22T15:46:38.472 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:38.455+0000 7fc19676f640 1 -- start start 2023-05-22T15:46:38.472 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:38.455+0000 7fc19676f640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc178004d10 0x7fc19015d100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:38.472 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:38.455+0000 7fc19676f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc190150550 0x7fc190150b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:38.473 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:38.455+0000 7fc19676f640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc190151050 0x7fc190158240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:38.473 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:38.455+0000 7fc19676f640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1900af8d0 con 0x7fc190150550 2023-05-22T15:46:38.473 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:38.455+0000 7fc19676f640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1901514c0 con 0x7fc190151050 2023-05-22T15:46:38.473 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:38.455+0000 7fc19676f640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1900af5a0 con 0x7fc178004d10 2023-05-22T15:46:38.473 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:38.455+0000 7fc18f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc190150550 0x7fc190150b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:38.474 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:38.459+0000 7fc18f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc190150550 0x7fc190150b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56730/0 (socket says 172.21.15.16:56730) 2023-05-22T15:46:38.474 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:38.459+0000 7fc18ffff640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc178004d10 0x7fc19015d100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:38.474 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:38.459+0000 7fc18f7fe640 1 -- 172.21.15.16:0/2995714422 learned_addr learned my addr 172.21.15.16:0/2995714422 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:46:38.474 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:38.459+0000 7fc194d0c640 1 --2- 172.21.15.16:0/2995714422 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc190151050 0x7fc190158240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:38.474 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:38.459+0000 7fc18f7fe640 1 -- 172.21.15.16:0/2995714422 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc178004d10 msgr2=0x7fc19015d100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:38.474 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:38.459+0000 7fc18f7fe640 1 --2- 172.21.15.16:0/2995714422 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc178004d10 0x7fc19015d100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:38.475 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:38.459+0000 7fc18f7fe640 1 -- 172.21.15.16:0/2995714422 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc190151050 msgr2=0x7fc190158240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:38.475 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:38.459+0000 7fc18f7fe640 1 --2- 172.21.15.16:0/2995714422 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc190151050 0x7fc190158240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:38.475 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:38.459+0000 7fc18f7fe640 1 -- 172.21.15.16:0/2995714422 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1900cdb00 con 0x7fc190150550 2023-05-22T15:46:38.475 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:38.459+0000 7fc194d0c640 1 --2- 172.21.15.16:0/2995714422 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc190151050 0x7fc190158240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T15:46:38.475 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:38.459+0000 7fc18f7fe640 1 --2- 172.21.15.16:0/2995714422 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc190150550 0x7fc190150b10 secure :-1 s=READY pgs=1889 cs=0 l=1 rev1=1 crypto rx=0x7fc184001210 tx=0x7fc18400e4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:38.475 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:38.459+0000 7fc18cff9640 1 -- 172.21.15.16:0/2995714422 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc184010e30 con 0x7fc190150550 2023-05-22T15:46:38.475 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:38.459+0000 7fc18cff9640 1 -- 172.21.15.16:0/2995714422 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1840143a0 con 0x7fc190150550 2023-05-22T15:46:38.475 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:38.459+0000 7fc19676f640 1 -- 172.21.15.16:0/2995714422 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1900ce0f0 con 0x7fc190150550 2023-05-22T15:46:38.476 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:38.459+0000 7fc18cff9640 1 -- 172.21.15.16:0/2995714422 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc184014710 con 0x7fc190150550 2023-05-22T15:46:38.476 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:38.459+0000 7fc19676f640 1 -- 172.21.15.16:0/2995714422 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1900cad80 con 0x7fc190150550 2023-05-22T15:46:38.476 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:38.459+0000 7fc19676f640 1 -- 172.21.15.16:0/2995714422 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc19015c560 con 0x7fc190150550 2023-05-22T15:46:38.476 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:38.459+0000 7fc18cff9640 1 -- 172.21.15.16:0/2995714422 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 61) v1 ==== 84349+0+0 (secure 0 0 0) 0x7fc184014870 con 0x7fc190150550 2023-05-22T15:46:38.476 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:38.459+0000 7fc18cff9640 1 --2- 172.21.15.16:0/2995714422 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fc17c0654f0 0x7fc17c0679a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:38.476 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:38.459+0000 7fc18ffff640 1 -- 172.21.15.16:0/2995714422 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fc17c0654f0 msgr2=0x7fc17c0679a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/36921 2023-05-22T15:46:38.476 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:38.459+0000 7fc18cff9640 1 -- 172.21.15.16:0/2995714422 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(147..147 src has 1..147) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fc184085710 con 0x7fc190150550 2023-05-22T15:46:38.477 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:38.459+0000 7fc18ffff640 1 --2- 172.21.15.16:0/2995714422 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fc17c0654f0 0x7fc17c0679a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-05-22T15:46:38.477 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:38.463+0000 7fc18cff9640 1 -- 172.21.15.16:0/2995714422 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fc184056c90 con 0x7fc190150550 2023-05-22T15:46:38.637 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:38.631+0000 7fc19676f640 1 -- 172.21.15.16:0/2995714422 --> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "snap_test_dir1", "snap_schedule": "1h", "target": ["mon-mgr", ""]}) v1 -- 0x7fc1900afa10 con 0x7fc17c0654f0 2023-05-22T15:46:38.667 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:38.663+0000 7fc18ffff640 1 -- 172.21.15.16:0/2995714422 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fc17c0654f0 msgr2=0x7fc17c0679a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/36921 2023-05-22T15:46:38.667 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:38.663+0000 7fc18ffff640 1 --2- 172.21.15.16:0/2995714422 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fc17c0654f0 0x7fc17c0679a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-05-22T15:46:38.711 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:46:38.707+0000 7f12b413c180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-05-22T15:46:38.744 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:46:38.739+0000 7fe213ff5180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-05-22T15:46:38.764 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:46:38.759+0000 7f12b413c180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-05-22T15:46:38.797 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:46:38.791+0000 7fe213ff5180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-05-22T15:46:38.814 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:46:38.811+0000 7f12b413c180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-05-22T15:46:38.848 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:46:38.843+0000 7fe213ff5180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-05-22T15:46:38.892 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:46:38.891+0000 7f12b413c180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-05-22T15:46:38.926 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:46:38.919+0000 7fe213ff5180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-05-22T15:46:38.944 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:46:38.939+0000 7f12b413c180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-05-22T15:46:38.978 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:46:38.971+0000 7fe213ff5180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-05-22T15:46:39.021 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:46:39.015+0000 7f12b413c180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-05-22T15:46:39.058 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:46:39.051+0000 7fe213ff5180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-05-22T15:46:39.071 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:39.067+0000 7fc18ffff640 1 -- 172.21.15.16:0/2995714422 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fc17c0654f0 msgr2=0x7fc17c0679a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/36921 2023-05-22T15:46:39.071 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:39.067+0000 7fc18ffff640 1 --2- 172.21.15.16:0/2995714422 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fc17c0654f0 0x7fc17c0679a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-05-22T15:46:39.073 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:46:39.071+0000 7f12b413c180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-05-22T15:46:39.109 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:46:39.103+0000 7fe213ff5180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-05-22T15:46:39.207 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:46:39.203+0000 7f12b413c180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-05-22T15:46:39.241 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:46:39.235+0000 7fe213ff5180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-05-22T15:46:39.260 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:46:39.255+0000 7f12b413c180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-05-22T15:46:39.294 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:46:39.287+0000 7fe213ff5180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-05-22T15:46:39.485 INFO:tasks.ceph.mgr.x.smithi062.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2023-05-22T15:46:39.485 INFO:tasks.ceph.mgr.x.smithi062.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2023-05-22T15:46:39.485 INFO:tasks.ceph.mgr.x.smithi062.stderr: from numpy import show_config as show_numpy_config 2023-05-22T15:46:39.493 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:46:39.491+0000 7f12b413c180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-05-22T15:46:39.521 INFO:tasks.ceph.mgr.y.smithi016.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2023-05-22T15:46:39.521 INFO:tasks.ceph.mgr.y.smithi016.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2023-05-22T15:46:39.521 INFO:tasks.ceph.mgr.y.smithi016.stderr: from numpy import show_config as show_numpy_config 2023-05-22T15:46:39.529 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:46:39.523+0000 7fe213ff5180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-05-22T15:46:39.626 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:46:39.615+0000 7f12b413c180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-05-22T15:46:39.656 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:46:39.651+0000 7fe213ff5180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-05-22T15:46:39.734 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:46:39.731+0000 7f12b413c180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-05-22T15:46:39.769 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:46:39.763+0000 7fe213ff5180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-05-22T15:46:39.796 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:46:39.791+0000 7f12b413c180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-05-22T15:46:39.832 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:46:39.827+0000 7fe213ff5180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-05-22T15:46:39.875 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:39.871+0000 7fc18ffff640 1 -- 172.21.15.16:0/2995714422 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fc17c0654f0 msgr2=0x7fc17c0679a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/36921 2023-05-22T15:46:39.875 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:39.871+0000 7fc18ffff640 1 --2- 172.21.15.16:0/2995714422 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fc17c0654f0 0x7fc17c0679a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-05-22T15:46:39.921 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:46:39.919+0000 7f12b413c180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-05-22T15:46:39.958 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:46:39.951+0000 7fe213ff5180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-05-22T15:46:40.416 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:46:40.415+0000 7f12b413c180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-05-22T15:46:40.453 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:46:40.447+0000 7fe213ff5180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-05-22T15:46:40.492 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:46:40.491+0000 7f12b413c180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-05-22T15:46:40.529 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:46:40.523+0000 7fe213ff5180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-05-22T15:46:40.630 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:46:40.627+0000 7f12b413c180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-05-22T15:46:40.670 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:46:40.663+0000 7fe213ff5180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-05-22T15:46:40.708 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:46:40.707+0000 7f12b413c180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-05-22T15:46:40.744 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:46:40.739+0000 7fe213ff5180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-05-22T15:46:40.956 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:46:40.951+0000 7f12b413c180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-05-22T15:46:40.992 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:46:40.987+0000 7fe213ff5180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-05-22T15:46:41.010 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:46:41.007+0000 7f12b413c180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-05-22T15:46:41.048 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:46:41.043+0000 7fe213ff5180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-05-22T15:46:41.088 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:41.083+0000 7fc18cff9640 1 -- 172.21.15.16:0/2995714422 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mgrmap(e 62) v1 ==== 83958+0+0 (secure 0 0 0) 0x7fc184053630 con 0x7fc190150550 2023-05-22T15:46:41.089 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:41.083+0000 7fc18cff9640 1 -- 172.21.15.16:0/2995714422 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fc17c0654f0 msgr2=0x7fc17c0679a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-05-22T15:46:41.089 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:41.083+0000 7fc18cff9640 1 --2- 172.21.15.16:0/2995714422 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fc17c0654f0 0x7fc17c0679a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:41.272 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:46:41.267+0000 7fe196271640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T15:46:41.273 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:46:41.267+0000 7fe196271640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T15:46:41.273 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:46:41.267+0000 7fe196271640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T15:46:41.273 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:46:41.267+0000 7fe196271640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T15:46:41.273 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:46:41.267+0000 7fe196271640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T15:46:41.273 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:46:41.267+0000 7fe19326b640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T15:46:41.273 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:46:41.267+0000 7fe19326b640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T15:46:41.273 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:46:41.267+0000 7fe19326b640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T15:46:41.274 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:46:41.267+0000 7fe19326b640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T15:46:41.274 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:46:41.267+0000 7fe19326b640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T15:46:42.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:42.087+0000 7fc18cff9640 1 -- 172.21.15.16:0/2995714422 <== mon.0 v2:172.21.15.16:3300/0 8 ==== mgrmap(e 63) v1 ==== 84085+0+0 (secure 0 0 0) 0x7fc184083940 con 0x7fc190150550 2023-05-22T15:46:42.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:42.087+0000 7fc18cff9640 1 --2- 172.21.15.16:0/2995714422 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fc17c0654f0 0x7fc17c067920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:42.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:42.087+0000 7fc18cff9640 1 -- 172.21.15.16:0/2995714422 --> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "snap_test_dir1", "snap_schedule": "1h", "target": ["mon-mgr", ""]}) v1 -- 0x7fc1900afa10 con 0x7fc17c0654f0 2023-05-22T15:46:42.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:42.091+0000 7fc18ffff640 1 --2- 172.21.15.16:0/2995714422 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fc17c0654f0 0x7fc17c067920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:42.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:42.095+0000 7fc18ffff640 1 --2- 172.21.15.16:0/2995714422 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fc17c0654f0 0x7fc17c067920 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fc180006f90 tx=0x7fc180008040 comp rx=0 tx=0).ready entity=mgr.9309 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:42.249 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:42.243+0000 7fc18cff9640 1 -- 172.21.15.16:0/2995714422 <== mgr.9309 v2:172.21.15.16:6832/36921 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fc1900afa10 con 0x7fc17c0654f0 2023-05-22T15:46:42.250 INFO:teuthology.orchestra.run.smithi016.stdout:Schedule set for path snap_test_dir1 2023-05-22T15:46:42.254 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:42.247+0000 7fc19676f640 1 -- 172.21.15.16:0/2995714422 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fc17c0654f0 msgr2=0x7fc17c067920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:42.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:42.247+0000 7fc19676f640 1 --2- 172.21.15.16:0/2995714422 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fc17c0654f0 0x7fc17c067920 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fc180006f90 tx=0x7fc180008040 comp rx=0 tx=0).stop 2023-05-22T15:46:42.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:42.247+0000 7fc19676f640 1 -- 172.21.15.16:0/2995714422 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc190150550 msgr2=0x7fc190150b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:42.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:42.247+0000 7fc19676f640 1 --2- 172.21.15.16:0/2995714422 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc190150550 0x7fc190150b10 secure :-1 s=READY pgs=1889 cs=0 l=1 rev1=1 crypto rx=0x7fc184001210 tx=0x7fc18400e4d0 comp rx=0 tx=0).stop 2023-05-22T15:46:42.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:42.247+0000 7fc19676f640 1 -- 172.21.15.16:0/2995714422 shutdown_connections 2023-05-22T15:46:42.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:42.247+0000 7fc19676f640 1 --2- 172.21.15.16:0/2995714422 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc190151050 0x7fc190158240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:42.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:42.247+0000 7fc19676f640 1 --2- 172.21.15.16:0/2995714422 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc190150550 0x7fc190150b10 unknown :-1 s=CLOSED pgs=1889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:42.256 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:42.247+0000 7fc19676f640 1 --2- 172.21.15.16:0/2995714422 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fc17c0654f0 0x7fc17c067920 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:42.256 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:42.247+0000 7fc19676f640 1 --2- 172.21.15.16:0/2995714422 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc178004d10 0x7fc19015d100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:42.256 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:42.247+0000 7fc19676f640 1 -- 172.21.15.16:0/2995714422 >> 172.21.15.16:0/2995714422 conn(0x7fc1900ce8d0 msgr2=0x7fc1900c51b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:46:42.256 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:42.247+0000 7fc19676f640 1 -- 172.21.15.16:0/2995714422 shutdown_connections 2023-05-22T15:46:42.256 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:42.247+0000 7fc19676f640 1 -- 172.21.15.16:0/2995714422 wait complete. 2023-05-22T15:46:42.275 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1 2023-05-22T15:46:42.275 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove --fs cephfs snap_test_dir1 2023-05-22T15:46:42.405 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:42.399+0000 7f5d1aae1640 1 Processor -- start 2023-05-22T15:46:42.405 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:42.399+0000 7f5d1aae1640 1 -- start start 2023-05-22T15:46:42.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:42.399+0000 7f5d1aae1640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f5d140ce900 0x7f5d1415c540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:42.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:42.399+0000 7f5d1aae1640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5d140cbe80 0x7f5d140cc680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:42.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:42.399+0000 7f5d1aae1640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d140c3930 con 0x7f5d140ce530 2023-05-22T15:46:42.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:42.399+0000 7f5d1aae1640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d14158f00 con 0x7f5d140ce900 2023-05-22T15:46:42.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:42.399+0000 7f5d1aae1640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d14159d90 con 0x7f5d140cbe80 2023-05-22T15:46:42.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:42.399+0000 7f5d13fff640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f5d140ce900 0x7f5d1415c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:42.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:42.399+0000 7f5d13fff640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f5d140ce900 0x7f5d1415c540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41420/0 (socket says 172.21.15.16:41420) 2023-05-22T15:46:42.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:42.399+0000 7f5d1887d640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f5d140ce530 0x7f5d14159680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:57334/0 (socket says 172.21.15.16:57334) 2023-05-22T15:46:42.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:42.399+0000 7f5d13fff640 1 -- 172.21.15.16:0/3819920366 learned_addr learned my addr 172.21.15.16:0/3819920366 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:46:42.408 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:42.399+0000 7f5d1907e640 1 --2- 172.21.15.16:0/3819920366 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5d140cbe80 0x7f5d140cc680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:42.408 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:42.403+0000 7f5d13fff640 1 -- 172.21.15.16:0/3819920366 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5d140cbe80 msgr2=0x7f5d140cc680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:46:42.408 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:42.403+0000 7f5d13fff640 1 --2- 172.21.15.16:0/3819920366 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5d140cbe80 0x7f5d140cc680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:42.408 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:42.403+0000 7f5d13fff640 1 -- 172.21.15.16:0/3819920366 >> v1:172.21.15.16:6789/0 conn(0x7f5d140ce530 legacy=0x7f5d14159680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:42.408 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:42.403+0000 7f5d137fe640 1 -- 172.21.15.16:0/3819920366 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1727461302 0 0) 0x7f5d08001120 con 0x7f5d140ce530 2023-05-22T15:46:42.409 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:42.403+0000 7f5d13fff640 1 -- 172.21.15.16:0/3819920366 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d1415cc40 con 0x7f5d140ce900 2023-05-22T15:46:42.409 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:42.403+0000 7f5d13fff640 1 --2- 172.21.15.16:0/3819920366 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5d140ce900 0x7f5d1415c540 secure :-1 s=READY pgs=1906 cs=0 l=1 rev1=1 crypto rx=0x7f5d00009870 tx=0x7f5d0001c540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c90442bcc9651eab server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:42.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:42.403+0000 7f5d137fe640 1 -- 172.21.15.16:0/3819920366 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d000ab600 con 0x7f5d140ce900 2023-05-22T15:46:42.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:42.403+0000 7f5d137fe640 1 -- 172.21.15.16:0/3819920366 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d000a7050 con 0x7f5d140ce900 2023-05-22T15:46:42.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:42.403+0000 7f5d137fe640 1 -- 172.21.15.16:0/3819920366 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d000ab980 con 0x7f5d140ce900 2023-05-22T15:46:42.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:42.403+0000 7f5d1aae1640 1 -- 172.21.15.16:0/3819920366 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5d140ce900 msgr2=0x7f5d1415c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:42.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:42.403+0000 7f5d1aae1640 1 --2- 172.21.15.16:0/3819920366 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5d140ce900 0x7f5d1415c540 secure :-1 s=READY pgs=1906 cs=0 l=1 rev1=1 crypto rx=0x7f5d00009870 tx=0x7f5d0001c540 comp rx=0 tx=0).stop 2023-05-22T15:46:42.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:42.403+0000 7f5d1aae1640 1 -- 172.21.15.16:0/3819920366 shutdown_connections 2023-05-22T15:46:42.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:42.403+0000 7f5d1aae1640 1 --2- 172.21.15.16:0/3819920366 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5d140cbe80 0x7f5d140cc680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:42.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:42.403+0000 7f5d1aae1640 1 --2- 172.21.15.16:0/3819920366 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5d140ce900 0x7f5d1415c540 unknown :-1 s=CLOSED pgs=1906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:42.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:42.403+0000 7f5d1aae1640 1 -- 172.21.15.16:0/3819920366 >> 172.21.15.16:0/3819920366 conn(0x7f5d14069720 msgr2=0x7f5d14069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:46:42.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:42.403+0000 7f5d1aae1640 1 -- 172.21.15.16:0/3819920366 shutdown_connections 2023-05-22T15:46:42.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:42.403+0000 7f5d1aae1640 1 -- 172.21.15.16:0/3819920366 wait complete. 2023-05-22T15:46:42.413 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:42.407+0000 7f5d1aae1640 1 Processor -- start 2023-05-22T15:46:42.413 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:42.407+0000 7f5d1aae1640 1 -- start start 2023-05-22T15:46:42.413 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:42.407+0000 7f5d1aae1640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5d140cbe80 0x7f5d140b0fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:42.413 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:42.407+0000 7f5d1aae1640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5d140ce530 0x7f5d1414afa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:42.413 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:42.407+0000 7f5d1aae1640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5d140ce900 0x7f5d14155710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:42.413 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:42.407+0000 7f5d1aae1640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d1414b4e0 con 0x7f5d140ce530 2023-05-22T15:46:42.413 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:42.407+0000 7f5d1aae1640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d14155c50 con 0x7f5d140cbe80 2023-05-22T15:46:42.414 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:42.407+0000 7f5d1aae1640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d140b1590 con 0x7f5d140ce900 2023-05-22T15:46:42.414 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:42.407+0000 7f5d13fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5d140ce530 0x7f5d1414afa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:42.414 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:42.407+0000 7f5d13fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5d140ce530 0x7f5d1414afa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41422/0 (socket says 172.21.15.16:41422) 2023-05-22T15:46:42.414 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:42.407+0000 7f5d13fff640 1 -- 172.21.15.16:0/3385979171 learned_addr learned my addr 172.21.15.16:0/3385979171 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:46:42.414 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:42.407+0000 7f5d1887d640 1 --2- 172.21.15.16:0/3385979171 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5d140cbe80 0x7f5d140b0fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:42.414 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:42.407+0000 7f5d13fff640 1 -- 172.21.15.16:0/3385979171 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5d140ce900 msgr2=0x7f5d14155710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:42.414 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:42.407+0000 7f5d1907e640 1 --2- 172.21.15.16:0/3385979171 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5d140ce900 0x7f5d14155710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:42.415 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:42.407+0000 7f5d13fff640 1 --2- 172.21.15.16:0/3385979171 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5d140ce900 0x7f5d14155710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:42.415 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:42.407+0000 7f5d13fff640 1 -- 172.21.15.16:0/3385979171 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5d140cbe80 msgr2=0x7f5d140b0fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:42.415 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:42.407+0000 7f5d13fff640 1 --2- 172.21.15.16:0/3385979171 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5d140cbe80 0x7f5d140b0fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:42.415 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:42.407+0000 7f5d13fff640 1 -- 172.21.15.16:0/3385979171 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d000094d0 con 0x7f5d140ce530 2023-05-22T15:46:42.415 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:42.407+0000 7f5d1887d640 1 --2- 172.21.15.16:0/3385979171 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5d140cbe80 0x7f5d140b0fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:46:42.415 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:42.407+0000 7f5d13fff640 1 --2- 172.21.15.16:0/3385979171 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5d140ce530 0x7f5d1414afa0 secure :-1 s=READY pgs=1907 cs=0 l=1 rev1=1 crypto rx=0x7f5d0001abf0 tx=0x7f5d000192e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:42.415 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:42.407+0000 7f5d117fa640 1 -- 172.21.15.16:0/3385979171 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d00004030 con 0x7f5d140ce530 2023-05-22T15:46:42.415 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:42.407+0000 7f5d117fa640 1 -- 172.21.15.16:0/3385979171 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d00004190 con 0x7f5d140ce530 2023-05-22T15:46:42.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:42.407+0000 7f5d117fa640 1 -- 172.21.15.16:0/3385979171 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d00004500 con 0x7f5d140ce530 2023-05-22T15:46:42.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:42.407+0000 7f5d1aae1640 1 -- 172.21.15.16:0/3385979171 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d14159160 con 0x7f5d140ce530 2023-05-22T15:46:42.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:42.407+0000 7f5d1aae1640 1 -- 172.21.15.16:0/3385979171 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d140ccfd0 con 0x7f5d140ce530 2023-05-22T15:46:42.417 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:42.407+0000 7f5d1aae1640 1 -- 172.21.15.16:0/3385979171 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d1414f2f0 con 0x7f5d140ce530 2023-05-22T15:46:42.417 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:42.407+0000 7f5d117fa640 1 -- 172.21.15.16:0/3385979171 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 63) v1 ==== 84085+0+0 (secure 0 0 0) 0x7f5d00003620 con 0x7f5d140ce530 2023-05-22T15:46:42.417 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:42.411+0000 7f5d117fa640 1 --2- 172.21.15.16:0/3385979171 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f5cf8065220 0x7f5cf80676d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:42.417 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:42.411+0000 7f5d117fa640 1 -- 172.21.15.16:0/3385979171 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 6754+0+0 (secure 0 0 0) 0x7f5d0011cea0 con 0x7f5d140ce530 2023-05-22T15:46:42.417 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:42.411+0000 7f5d1887d640 1 --2- 172.21.15.16:0/3385979171 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f5cf8065220 0x7f5cf80676d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:42.417 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:42.411+0000 7f5d1887d640 1 --2- 172.21.15.16:0/3385979171 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f5cf8065220 0x7f5cf80676d0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f5d08007fd0 tx=0x7f5d08009040 comp rx=0 tx=0).ready entity=mgr.9309 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:42.419 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:42.415+0000 7f5d117fa640 1 -- 172.21.15.16:0/3385979171 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f5d000eab40 con 0x7f5d140ce530 2023-05-22T15:46:42.591 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:42.587+0000 7f5d1aae1640 1 -- 172.21.15.16:0/3385979171 --> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "fs": "cephfs", "path": "snap_test_dir1", "target": ["mon-mgr", ""]}) v1 -- 0x7f5d1404dbe0 con 0x7f5cf8065220 2023-05-22T15:46:42.736 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:42.731+0000 7f5d117fa640 1 -- 172.21.15.16:0/3385979171 <== mgr.9309 v2:172.21.15.16:6832/36921 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f5d1404dbe0 con 0x7f5cf8065220 2023-05-22T15:46:42.737 INFO:teuthology.orchestra.run.smithi016.stdout:Schedule removed for path snap_test_dir1 2023-05-22T15:46:42.739 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:42.731+0000 7f5d1aae1640 1 -- 172.21.15.16:0/3385979171 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f5cf8065220 msgr2=0x7f5cf80676d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:42.739 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:42.731+0000 7f5d1aae1640 1 --2- 172.21.15.16:0/3385979171 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f5cf8065220 0x7f5cf80676d0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f5d08007fd0 tx=0x7f5d08009040 comp rx=0 tx=0).stop 2023-05-22T15:46:42.739 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:42.731+0000 7f5d1aae1640 1 -- 172.21.15.16:0/3385979171 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5d140ce530 msgr2=0x7f5d1414afa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:42.740 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:42.731+0000 7f5d1aae1640 1 --2- 172.21.15.16:0/3385979171 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5d140ce530 0x7f5d1414afa0 secure :-1 s=READY pgs=1907 cs=0 l=1 rev1=1 crypto rx=0x7f5d0001abf0 tx=0x7f5d000192e0 comp rx=0 tx=0).stop 2023-05-22T15:46:42.740 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:42.731+0000 7f5d1aae1640 1 -- 172.21.15.16:0/3385979171 shutdown_connections 2023-05-22T15:46:42.741 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:42.731+0000 7f5d1aae1640 1 --2- 172.21.15.16:0/3385979171 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f5cf8065220 0x7f5cf80676d0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:42.741 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:42.731+0000 7f5d1aae1640 1 --2- 172.21.15.16:0/3385979171 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5d140ce900 0x7f5d14155710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:42.741 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:42.735+0000 7f5d1aae1640 1 --2- 172.21.15.16:0/3385979171 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5d140ce530 0x7f5d1414afa0 unknown :-1 s=CLOSED pgs=1907 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:42.741 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:42.735+0000 7f5d1aae1640 1 --2- 172.21.15.16:0/3385979171 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5d140cbe80 0x7f5d140b0fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:42.741 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:42.735+0000 7f5d1aae1640 1 -- 172.21.15.16:0/3385979171 >> 172.21.15.16:0/3385979171 conn(0x7f5d14069720 msgr2=0x7f5d14159680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:46:42.742 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:42.735+0000 7f5d1aae1640 1 -- 172.21.15.16:0/3385979171 shutdown_connections 2023-05-22T15:46:42.742 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:42.735+0000 7f5d1aae1640 1 -- 172.21.15.16:0/3385979171 wait complete. 2023-05-22T15:46:42.759 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path snap_test_dir1 2023-05-22T15:46:42.760 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule list --fs cephfs snap_test_dir1 2023-05-22T15:46:42.896 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:42.891+0000 7fdde2e26640 1 Processor -- start 2023-05-22T15:46:42.896 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:42.891+0000 7fdde2e26640 1 -- start start 2023-05-22T15:46:42.898 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:42.891+0000 7fdde2e26640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fdddc159a30 0x7fdddc06e380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:42.898 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:42.891+0000 7fdde2e26640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fdddc0cf570 0x7fdddc0cf9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:42.899 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:42.891+0000 7fdde2e26640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdddc0cde70 con 0x7fdddc15d610 2023-05-22T15:46:42.899 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:42.891+0000 7fdde2e26640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdddc0cc520 con 0x7fdddc0cf570 2023-05-22T15:46:42.899 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:42.891+0000 7fdde2e26640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdddc0cff30 con 0x7fdddc159a30 2023-05-22T15:46:42.899 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:42.891+0000 7fdddbfff640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fdddc15d610 0x7fdddc0cbe10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:57348/0 (socket says 172.21.15.16:57348) 2023-05-22T15:46:42.900 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:42.891+0000 7fdddbfff640 1 -- 172.21.15.16:0/878224698 learned_addr learned my addr 172.21.15.16:0/878224698 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:46:42.900 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:42.891+0000 7fdde13c3640 1 --2- 172.21.15.16:0/878224698 >> v2:172.21.15.62:3300/0 conn(0x7fdddc0cf570 0x7fdddc0cf9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:42.900 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:42.891+0000 7fdde0bc2640 1 --2- 172.21.15.16:0/878224698 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fdddc159a30 0x7fdddc06e380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:42.901 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:42.891+0000 7fdddb7fe640 1 -- 172.21.15.16:0/878224698 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1494789372 0 0) 0x7fdddc0cde70 con 0x7fdddc15d610 2023-05-22T15:46:42.901 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:42.891+0000 7fdddb7fe640 1 -- 172.21.15.16:0/878224698 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fddc4003530 con 0x7fdddc15d610 2023-05-22T15:46:42.901 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:42.891+0000 7fdde13c3640 1 -- 172.21.15.16:0/878224698 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fdddc159a30 msgr2=0x7fdddc06e380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:46:42.901 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:42.891+0000 7fdde13c3640 1 --2- 172.21.15.16:0/878224698 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fdddc159a30 0x7fdddc06e380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:42.902 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:42.891+0000 7fdde13c3640 1 -- 172.21.15.16:0/878224698 >> v1:172.21.15.16:6789/0 conn(0x7fdddc15d610 legacy=0x7fdddc0cbe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:42.902 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:42.891+0000 7fdde13c3640 1 -- 172.21.15.16:0/878224698 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdddc15db20 con 0x7fdddc0cf570 2023-05-22T15:46:42.906 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:42.895+0000 7fdde13c3640 1 --2- 172.21.15.16:0/878224698 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdddc0cf570 0x7fdddc0cf9f0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7fddd4009d90 tx=0x7fddd401c930 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ca05ad2e4d45f902 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:42.907 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:42.895+0000 7fdddb7fe640 1 -- 172.21.15.16:0/878224698 <== mon.1 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddd40a59f0 con 0x7fdddc0cf570 2023-05-22T15:46:42.907 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:42.895+0000 7fdddb7fe640 1 -- 172.21.15.16:0/878224698 <== mon.1 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddd40a5b50 con 0x7fdddc0cf570 2023-05-22T15:46:42.907 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:42.895+0000 7fdddb7fe640 1 -- 172.21.15.16:0/878224698 <== mon.1 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddd40ad440 con 0x7fdddc0cf570 2023-05-22T15:46:42.908 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:42.895+0000 7fdde2e26640 1 -- 172.21.15.16:0/878224698 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdddc0cf570 msgr2=0x7fdddc0cf9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:42.908 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:42.895+0000 7fdde2e26640 1 --2- 172.21.15.16:0/878224698 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdddc0cf570 0x7fdddc0cf9f0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7fddd4009d90 tx=0x7fddd401c930 comp rx=0 tx=0).stop 2023-05-22T15:46:42.908 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:42.895+0000 7fdde2e26640 1 -- 172.21.15.16:0/878224698 shutdown_connections 2023-05-22T15:46:42.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:42.895+0000 7fdde2e26640 1 --2- 172.21.15.16:0/878224698 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdddc0cf570 0x7fdddc0cf9f0 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:42.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:42.895+0000 7fdde2e26640 1 --2- 172.21.15.16:0/878224698 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fdddc159a30 0x7fdddc06e380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:42.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:42.895+0000 7fdde2e26640 1 -- 172.21.15.16:0/878224698 >> 172.21.15.16:0/878224698 conn(0x7fdddc0700b0 msgr2=0x7fdddc06d9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:46:42.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:42.895+0000 7fdde2e26640 1 -- 172.21.15.16:0/878224698 shutdown_connections 2023-05-22T15:46:42.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:42.895+0000 7fdde2e26640 1 -- 172.21.15.16:0/878224698 wait complete. 2023-05-22T15:46:42.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:42.895+0000 7fdde2e26640 1 Processor -- start 2023-05-22T15:46:42.911 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:42.895+0000 7fdde2e26640 1 -- start start 2023-05-22T15:46:42.911 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:42.899+0000 7fdde2e26640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fdddc0cf570 0x7fdddc0c9630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:42.911 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:42.899+0000 7fdde2e26640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdddc159a30 0x7fdddc1f20c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:42.911 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:42.899+0000 7fdde2e26640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdddc15d610 0x7fdddc0b4da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:42.912 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:42.899+0000 7fdde2e26640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdddc0c8a90 con 0x7fdddc15d610 2023-05-22T15:46:42.912 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:42.899+0000 7fdde2e26640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdddc0b4910 con 0x7fdddc159a30 2023-05-22T15:46:42.912 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:42.899+0000 7fdde2e26640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdddc0c9b70 con 0x7fdddc0cf570 2023-05-22T15:46:42.913 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:42.899+0000 7fdde13c3640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdddc15d610 0x7fdddc0b4da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:42.913 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:42.899+0000 7fdde13c3640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdddc15d610 0x7fdddc0b4da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41428/0 (socket says 172.21.15.16:41428) 2023-05-22T15:46:42.913 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:42.899+0000 7fdde13c3640 1 -- 172.21.15.16:0/2820615075 learned_addr learned my addr 172.21.15.16:0/2820615075 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:46:42.913 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:42.899+0000 7fdddbfff640 1 --2- 172.21.15.16:0/2820615075 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdddc159a30 0x7fdddc1f20c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:42.914 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:42.899+0000 7fdde0bc2640 1 --2- 172.21.15.16:0/2820615075 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fdddc0cf570 0x7fdddc0c9630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:42.914 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:42.899+0000 7fdde13c3640 1 -- 172.21.15.16:0/2820615075 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fdddc0cf570 msgr2=0x7fdddc0c9630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:42.914 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:42.899+0000 7fdde13c3640 1 --2- 172.21.15.16:0/2820615075 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fdddc0cf570 0x7fdddc0c9630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:42.914 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:42.899+0000 7fdde13c3640 1 -- 172.21.15.16:0/2820615075 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdddc159a30 msgr2=0x7fdddc1f20c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:42.915 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:42.899+0000 7fdde13c3640 1 --2- 172.21.15.16:0/2820615075 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdddc159a30 0x7fdddc1f20c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:42.915 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:42.899+0000 7fdde13c3640 1 -- 172.21.15.16:0/2820615075 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddd4009a70 con 0x7fdddc15d610 2023-05-22T15:46:42.915 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:42.899+0000 7fdddbfff640 1 --2- 172.21.15.16:0/2820615075 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdddc159a30 0x7fdddc1f20c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T15:46:42.916 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:42.899+0000 7fdde13c3640 1 --2- 172.21.15.16:0/2820615075 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdddc15d610 0x7fdddc0b4da0 secure :-1 s=READY pgs=1909 cs=0 l=1 rev1=1 crypto rx=0x7fddd401afe0 tx=0x7fddd40196d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:42.916 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:42.899+0000 7fddd97fa640 1 -- 172.21.15.16:0/2820615075 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddd4004240 con 0x7fdddc15d610 2023-05-22T15:46:42.916 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:42.899+0000 7fddd97fa640 1 -- 172.21.15.16:0/2820615075 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddd40043a0 con 0x7fdddc15d610 2023-05-22T15:46:42.916 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:42.899+0000 7fdde2e26640 1 -- 172.21.15.16:0/2820615075 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdddc15e3a0 con 0x7fdddc15d610 2023-05-22T15:46:42.916 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:42.899+0000 7fddd97fa640 1 -- 172.21.15.16:0/2820615075 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddd40047b0 con 0x7fdddc15d610 2023-05-22T15:46:42.917 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:42.899+0000 7fdde2e26640 1 -- 172.21.15.16:0/2820615075 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdddc15e0c0 con 0x7fdddc15d610 2023-05-22T15:46:42.917 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:42.899+0000 7fdde2e26640 1 -- 172.21.15.16:0/2820615075 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdddc159e00 con 0x7fdddc15d610 2023-05-22T15:46:42.917 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:42.899+0000 7fddd97fa640 1 -- 172.21.15.16:0/2820615075 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 63) v1 ==== 84085+0+0 (secure 0 0 0) 0x7fddd40ad440 con 0x7fdddc15d610 2023-05-22T15:46:42.917 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:42.899+0000 7fddd97fa640 1 --2- 172.21.15.16:0/2820615075 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fddc8065220 0x7fddc80676d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:42.917 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:42.899+0000 7fddd97fa640 1 -- 172.21.15.16:0/2820615075 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 6754+0+0 (secure 0 0 0) 0x7fddd411e610 con 0x7fdddc15d610 2023-05-22T15:46:42.917 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:42.899+0000 7fdde0bc2640 1 --2- 172.21.15.16:0/2820615075 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fddc8065220 0x7fddc80676d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:42.918 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:42.903+0000 7fdde0bc2640 1 --2- 172.21.15.16:0/2820615075 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fddc8065220 0x7fddc80676d0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fdddc0c6950 tx=0x7fddcc008040 comp rx=0 tx=0).ready entity=mgr.9309 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:42.918 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:42.903+0000 7fddd97fa640 1 -- 172.21.15.16:0/2820615075 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fddd40ec2b0 con 0x7fdddc15d610 2023-05-22T15:46:43.083 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:43.079+0000 7fdde2e26640 1 -- 172.21.15.16:0/2820615075 --> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "snap_test_dir1", "target": ["mon-mgr", ""]}) v1 -- 0x7fdddc066910 con 0x7fddc8065220 2023-05-22T15:46:43.097 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:46:43.091+0000 7fe19d27f640 -1 mgr.server reply reply (2) No such file or directory SnapSchedule for snap_test_dir1 not found 2023-05-22T15:46:43.097 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:43.091+0000 7fddd97fa640 1 -- 172.21.15.16:0/2820615075 <== mgr.9309 v2:172.21.15.16:6832/36921 1 ==== mgr_command_reply(tid 0: -2 SnapSchedule for snap_test_dir1 not found) v1 ==== 49+0+0 (secure 0 0 0) 0x7fdddc066910 con 0x7fddc8065220 2023-05-22T15:46:43.098 INFO:teuthology.orchestra.run.smithi016.stderr:Error ENOENT: SnapSchedule for snap_test_dir1 not found 2023-05-22T15:46:43.102 DEBUG:teuthology.orchestra.run:got remote process result: 2 2023-05-22T15:46:43.104 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:46:43.105 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1) 2023-05-22T15:46:43.119 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2023-05-22T15:46:43.298 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:43.291+0000 7fc294745640 1 Processor -- start 2023-05-22T15:46:43.298 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:43.291+0000 7fc294745640 1 -- start start 2023-05-22T15:46:43.300 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:43.291+0000 7fc294745640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc28c0cb4c0 0x7fc28c158fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:43.301 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:43.291+0000 7fc294745640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fc28c159510 0x7fc28c06fba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:43.301 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:43.295+0000 7fc294745640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc28c159f70 con 0x7fc28c06b6c0 2023-05-22T15:46:43.301 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:43.295+0000 7fc294745640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc28c071760 con 0x7fc28c159510 2023-05-22T15:46:43.301 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:43.295+0000 7fc294745640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc28c0cb910 con 0x7fc28c0cb4c0 2023-05-22T15:46:43.301 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:43.295+0000 7fc2924e1640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fc28c06b6c0 0x7fc28c06ba90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:57356/0 (socket says 172.21.15.16:57356) 2023-05-22T15:46:43.301 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:43.295+0000 7fc2924e1640 1 -- 172.21.15.16:0/2755231399 learned_addr learned my addr 172.21.15.16:0/2755231399 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:46:43.302 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:43.295+0000 7fc291ce0640 1 --2- 172.21.15.16:0/2755231399 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc28c0cb4c0 0x7fc28c158fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:43.302 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:43.295+0000 7fc292ce2640 1 --2- 172.21.15.16:0/2755231399 >> v2:172.21.15.62:3300/0 conn(0x7fc28c159510 0x7fc28c06fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:43.302 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:43.295+0000 7fc2914df640 1 -- 172.21.15.16:0/2755231399 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3872772207 0 0) 0x7fc28c159f70 con 0x7fc28c06b6c0 2023-05-22T15:46:43.302 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:43.295+0000 7fc2914df640 1 -- 172.21.15.16:0/2755231399 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc278003530 con 0x7fc28c06b6c0 2023-05-22T15:46:43.302 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:43.295+0000 7fc2914df640 1 -- 172.21.15.16:0/2755231399 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2888296968 0 0) 0x7fc278003530 con 0x7fc28c06b6c0 2023-05-22T15:46:43.302 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:43.295+0000 7fc2914df640 1 -- 172.21.15.16:0/2755231399 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc28c0cb4c0 msgr2=0x7fc28c158fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:46:43.302 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:43.295+0000 7fc2914df640 1 --2- 172.21.15.16:0/2755231399 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc28c0cb4c0 0x7fc28c158fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:43.303 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:43.295+0000 7fc2914df640 1 -- 172.21.15.16:0/2755231399 >> v2:172.21.15.62:3300/0 conn(0x7fc28c159510 msgr2=0x7fc28c06fba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:46:43.303 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:43.295+0000 7fc2914df640 1 --2- 172.21.15.16:0/2755231399 >> v2:172.21.15.62:3300/0 conn(0x7fc28c159510 0x7fc28c06fba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:43.303 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:43.295+0000 7fc2914df640 1 -- 172.21.15.16:0/2755231399 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc28c0cb030 con 0x7fc28c06b6c0 2023-05-22T15:46:43.303 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:43.295+0000 7fc2914df640 1 -- 172.21.15.16:0/2755231399 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7fc288001680 con 0x7fc28c06b6c0 2023-05-22T15:46:43.303 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:43.295+0000 7fc2914df640 1 -- 172.21.15.16:0/2755231399 >> v1:172.21.15.16:6789/0 conn(0x7fc28c06b6c0 legacy=0x7fc28c06ba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:43.303 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:43.295+0000 7fc2914df640 1 --2- 172.21.15.16:0/2755231399 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc278003980 0x7fc278003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:43.304 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:43.295+0000 7fc2914df640 1 --2- 172.21.15.16:0/2755231399 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc2780043b0 0x7fc278004870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:43.304 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:43.295+0000 7fc2914df640 1 --2- 172.21.15.16:0/2755231399 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc28c0cb4c0 0x7fc278004f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:43.304 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:43.295+0000 7fc2914df640 1 -- 172.21.15.16:0/2755231399 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc28c071760 con 0x7fc278003980 2023-05-22T15:46:43.304 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:43.295+0000 7fc2914df640 1 -- 172.21.15.16:0/2755231399 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc28c0cb910 con 0x7fc2780043b0 2023-05-22T15:46:43.304 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:43.295+0000 7fc2914df640 1 -- 172.21.15.16:0/2755231399 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc27800b990 con 0x7fc28c0cb4c0 2023-05-22T15:46:43.304 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:43.295+0000 7fc2924e1640 1 --2- 172.21.15.16:0/2755231399 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc278003980 0x7fc278003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:43.304 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:43.295+0000 7fc291ce0640 1 --2- 172.21.15.16:0/2755231399 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc2780043b0 0x7fc278004870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:43.305 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:43.295+0000 7fc292ce2640 1 --2- 172.21.15.16:0/2755231399 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc28c0cb4c0 0x7fc278004f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:43.305 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:43.295+0000 7fc2924e1640 1 -- 172.21.15.16:0/2755231399 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc28c0cb4c0 msgr2=0x7fc278004f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:46:43.305 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:43.295+0000 7fc2924e1640 1 --2- 172.21.15.16:0/2755231399 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc28c0cb4c0 0x7fc278004f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:43.305 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:43.295+0000 7fc2924e1640 1 -- 172.21.15.16:0/2755231399 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc2780043b0 msgr2=0x7fc278004870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:46:43.305 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:43.295+0000 7fc2924e1640 1 --2- 172.21.15.16:0/2755231399 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc2780043b0 0x7fc278004870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:43.305 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:43.295+0000 7fc2924e1640 1 -- 172.21.15.16:0/2755231399 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc278003530 con 0x7fc278003980 2023-05-22T15:46:43.306 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:43.295+0000 7fc291ce0640 1 --2- 172.21.15.16:0/2755231399 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc2780043b0 0x7fc278004870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T15:46:43.308 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:43.295+0000 7fc2924e1640 1 --2- 172.21.15.16:0/2755231399 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc278003980 0x7fc278003d90 secure :-1 s=READY pgs=1911 cs=0 l=1 rev1=1 crypto rx=0x7fc28800c210 tx=0x7fc28801cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31f4f29348f04fae server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:43.308 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:43.295+0000 7fc2914df640 1 -- 172.21.15.16:0/2755231399 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2880058e0 con 0x7fc278003980 2023-05-22T15:46:43.308 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:43.295+0000 7fc2914df640 1 -- 172.21.15.16:0/2755231399 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc288005a40 con 0x7fc278003980 2023-05-22T15:46:43.309 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:43.295+0000 7fc2914df640 1 -- 172.21.15.16:0/2755231399 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc288005df0 con 0x7fc278003980 2023-05-22T15:46:43.309 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:43.299+0000 7fc294745640 1 -- 172.21.15.16:0/2755231399 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc278003980 msgr2=0x7fc278003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:43.309 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:43.299+0000 7fc294745640 1 --2- 172.21.15.16:0/2755231399 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc278003980 0x7fc278003d90 secure :-1 s=READY pgs=1911 cs=0 l=1 rev1=1 crypto rx=0x7fc28800c210 tx=0x7fc28801cc40 comp rx=0 tx=0).stop 2023-05-22T15:46:43.309 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:43.299+0000 7fc292ce2640 1 -- 172.21.15.16:0/2755231399 reap_dead start 2023-05-22T15:46:43.309 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:43.299+0000 7fc294745640 1 -- 172.21.15.16:0/2755231399 shutdown_connections 2023-05-22T15:46:43.309 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:43.299+0000 7fc294745640 1 -- 172.21.15.16:0/2755231399 >> 172.21.15.16:0/2755231399 conn(0x7fc28c069720 msgr2=0x7fc28c069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:46:43.309 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:43.299+0000 7fc294745640 1 -- 172.21.15.16:0/2755231399 shutdown_connections 2023-05-22T15:46:43.310 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:43.299+0000 7fc294745640 1 -- 172.21.15.16:0/2755231399 wait complete. 2023-05-22T15:46:43.310 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:43.299+0000 7fc294745640 1 Processor -- start 2023-05-22T15:46:43.310 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:43.299+0000 7fc294745640 1 -- start start 2023-05-22T15:46:43.310 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:43.299+0000 7fc294745640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc28c14e370 0x7fc28c0b1050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:43.310 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:43.299+0000 7fc294745640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc28c14ea90 0x7fc28c0b1590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:43.310 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:43.299+0000 7fc294745640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc28c0c3170 0x7fc28c0c35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:43.310 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:43.299+0000 7fc294745640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc28c14f280 con 0x7fc28c14ea90 2023-05-22T15:46:43.311 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:43.299+0000 7fc294745640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc28c150f50 con 0x7fc28c14e370 2023-05-22T15:46:43.311 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:43.299+0000 7fc294745640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc28c151230 con 0x7fc28c0c3170 2023-05-22T15:46:43.311 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:43.299+0000 7fc291ce0640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc28c14ea90 0x7fc28c0b1590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:43.311 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:43.299+0000 7fc291ce0640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc28c14ea90 0x7fc28c0b1590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41438/0 (socket says 172.21.15.16:41438) 2023-05-22T15:46:43.311 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:43.299+0000 7fc291ce0640 1 -- 172.21.15.16:0/1202329354 learned_addr learned my addr 172.21.15.16:0/1202329354 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:46:43.311 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:43.299+0000 7fc2924e1640 1 --2- 172.21.15.16:0/1202329354 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc28c14e370 0x7fc28c0b1050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:43.311 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:43.299+0000 7fc291ce0640 1 -- 172.21.15.16:0/1202329354 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc28c0c3170 msgr2=0x7fc28c0c35e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:43.312 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:43.299+0000 7fc291ce0640 1 --2- 172.21.15.16:0/1202329354 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc28c0c3170 0x7fc28c0c35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:43.312 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:43.299+0000 7fc291ce0640 1 -- 172.21.15.16:0/1202329354 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc28c14e370 msgr2=0x7fc28c0b1050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:43.312 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:43.299+0000 7fc291ce0640 1 --2- 172.21.15.16:0/1202329354 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc28c14e370 0x7fc28c0b1050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:43.312 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:43.299+0000 7fc291ce0640 1 -- 172.21.15.16:0/1202329354 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc28c159910 con 0x7fc28c14ea90 2023-05-22T15:46:43.312 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:43.299+0000 7fc2924e1640 1 --2- 172.21.15.16:0/1202329354 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc28c14e370 0x7fc28c0b1050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:46:43.312 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:43.299+0000 7fc291ce0640 1 --2- 172.21.15.16:0/1202329354 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc28c14ea90 0x7fc28c0b1590 secure :-1 s=READY pgs=1912 cs=0 l=1 rev1=1 crypto rx=0x7fc27c009160 tx=0x7fc27c011a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:43.312 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:43.299+0000 7fc282ffd640 1 -- 172.21.15.16:0/1202329354 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc27c004220 con 0x7fc28c14ea90 2023-05-22T15:46:43.313 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:43.299+0000 7fc282ffd640 1 -- 172.21.15.16:0/1202329354 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc27c004380 con 0x7fc28c14ea90 2023-05-22T15:46:43.313 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:43.299+0000 7fc294745640 1 -- 172.21.15.16:0/1202329354 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc28c0cacc0 con 0x7fc28c14ea90 2023-05-22T15:46:43.313 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:43.299+0000 7fc282ffd640 1 -- 172.21.15.16:0/1202329354 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc27c015890 con 0x7fc28c14ea90 2023-05-22T15:46:43.313 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:43.299+0000 7fc294745640 1 -- 172.21.15.16:0/1202329354 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc28c156840 con 0x7fc28c14ea90 2023-05-22T15:46:43.313 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:43.303+0000 7fc294745640 1 -- 172.21.15.16:0/1202329354 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc28c04dbe0 con 0x7fc28c14ea90 2023-05-22T15:46:43.314 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:43.303+0000 7fc282ffd640 1 -- 172.21.15.16:0/1202329354 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 63) v1 ==== 84085+0+0 (secure 0 0 0) 0x7fc27c0159f0 con 0x7fc28c14ea90 2023-05-22T15:46:43.316 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:43.303+0000 7fc282ffd640 1 --2- 172.21.15.16:0/1202329354 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fc270065250 0x7fc270067700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:43.316 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:43.303+0000 7fc282ffd640 1 -- 172.21.15.16:0/1202329354 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 6754+0+0 (secure 0 0 0) 0x7fc27c082c10 con 0x7fc28c14ea90 2023-05-22T15:46:43.317 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:43.303+0000 7fc2924e1640 1 --2- 172.21.15.16:0/1202329354 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fc270065250 0x7fc270067700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:43.317 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:43.303+0000 7fc2924e1640 1 --2- 172.21.15.16:0/1202329354 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fc270065250 0x7fc270067700 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fc2880aa000 tx=0x7fc288000ff0 comp rx=0 tx=0).ready entity=mgr.9309 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:43.318 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:43.307+0000 7fc282ffd640 1 -- 172.21.15.16:0/1202329354 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fc27c050910 con 0x7fc28c14ea90 2023-05-22T15:46:43.507 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:43.499+0000 7fc294745640 1 -- 172.21.15.16:0/1202329354 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7fc28c0ca1d0 con 0x7fc28c14ea90 2023-05-22T15:46:44.104 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:44.099+0000 7fc282ffd640 1 -- 172.21.15.16:0/1202329354 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mgrmap(e 64) v1 ==== 84389+0+0 (secure 0 0 0) 0x7fc27c050ce0 con 0x7fc28c14ea90 2023-05-22T15:46:44.105 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:44.099+0000 7fc282ffd640 1 -- 172.21.15.16:0/1202329354 <== mon.0 v2:172.21.15.16:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v64) v1 ==== 93+0+0 (secure 0 0 0) 0x7fc27c052970 con 0x7fc28c14ea90 2023-05-22T15:46:44.108 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:44.103+0000 7fc294745640 1 -- 172.21.15.16:0/1202329354 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fc270065250 msgr2=0x7fc270067700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:44.109 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:44.103+0000 7fc294745640 1 --2- 172.21.15.16:0/1202329354 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fc270065250 0x7fc270067700 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fc2880aa000 tx=0x7fc288000ff0 comp rx=0 tx=0).stop 2023-05-22T15:46:44.109 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:44.103+0000 7fc294745640 1 -- 172.21.15.16:0/1202329354 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc28c14ea90 msgr2=0x7fc28c0b1590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:44.109 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:44.103+0000 7fc294745640 1 --2- 172.21.15.16:0/1202329354 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc28c14ea90 0x7fc28c0b1590 secure :-1 s=READY pgs=1912 cs=0 l=1 rev1=1 crypto rx=0x7fc27c009160 tx=0x7fc27c011a60 comp rx=0 tx=0).stop 2023-05-22T15:46:44.109 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:44.103+0000 7fc294745640 1 -- 172.21.15.16:0/1202329354 shutdown_connections 2023-05-22T15:46:44.109 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:44.103+0000 7fc294745640 1 --2- 172.21.15.16:0/1202329354 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fc270065250 0x7fc270067700 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:44.109 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:44.103+0000 7fc294745640 1 --2- 172.21.15.16:0/1202329354 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc28c0c3170 0x7fc28c0c35e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:44.110 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:44.103+0000 7fc294745640 1 --2- 172.21.15.16:0/1202329354 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc28c14ea90 0x7fc28c0b1590 unknown :-1 s=CLOSED pgs=1912 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:44.110 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:44.103+0000 7fc294745640 1 --2- 172.21.15.16:0/1202329354 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc28c14e370 0x7fc28c0b1050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:44.110 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:44.103+0000 7fc294745640 1 -- 172.21.15.16:0/1202329354 >> 172.21.15.16:0/1202329354 conn(0x7fc28c069720 msgr2=0x7fc28c0c94f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:46:44.110 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:44.103+0000 7fc294745640 1 -- 172.21.15.16:0/1202329354 shutdown_connections 2023-05-22T15:46:44.110 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:44.103+0000 7fc294745640 1 -- 172.21.15.16:0/1202329354 wait complete. 2023-05-22T15:46:44.131 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-05-22T15:46:44.131 DEBUG:teuthology.orchestra.run.smithi016:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-05-22T15:46:44.204 DEBUG:teuthology.orchestra.run.smithi062:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-05-22T15:46:44.234 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:46:44.227+0000 7f5d4045c180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-05-22T15:46:44.242 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-05-22T15:46:44.242 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-05-22T15:46:44.243 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2023-05-22T15:46:44.248 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-05-22T15:46:44.259 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:46:44.255+0000 7fd05b983180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-05-22T15:46:44.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:44.407+0000 7f2e77204640 1 Processor -- start 2023-05-22T15:46:44.412 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:44.407+0000 7f2e77204640 1 -- start start 2023-05-22T15:46:44.413 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:44.407+0000 7f2e77204640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2e70153fe0 0x7f2e7006d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:44.413 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:44.407+0000 7f2e77204640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f2e700cae20 0x7f2e7006fef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:44.413 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:44.407+0000 7f2e77204640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e70069720 con 0x7f2e70070430 2023-05-22T15:46:44.413 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:44.407+0000 7f2e77204640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e700708b0 con 0x7f2e70153fe0 2023-05-22T15:46:44.413 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:44.407+0000 7f2e77204640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e701543b0 con 0x7f2e700cae20 2023-05-22T15:46:44.413 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:44.407+0000 7f2e757a1640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f2e70070430 0x7f2e700ad410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:57372/0 (socket says 172.21.15.16:57372) 2023-05-22T15:46:44.414 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:44.407+0000 7f2e74fa0640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2e70153fe0 0x7f2e7006d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:44.414 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:44.407+0000 7f2e757a1640 1 -- 172.21.15.16:0/456932500 learned_addr learned my addr 172.21.15.16:0/456932500 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:46:44.414 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:44.407+0000 7f2e6ffff640 1 --2- 172.21.15.16:0/456932500 >> v2:172.21.15.62:3300/0 conn(0x7f2e700cae20 0x7f2e7006fef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:44.414 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:44.407+0000 7f2e74fa0640 1 -- 172.21.15.16:0/456932500 >> v2:172.21.15.62:3300/0 conn(0x7f2e700cae20 msgr2=0x7f2e7006fef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:46:44.414 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:44.407+0000 7f2e74fa0640 1 --2- 172.21.15.16:0/456932500 >> v2:172.21.15.62:3300/0 conn(0x7f2e700cae20 0x7f2e7006fef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:44.414 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:44.407+0000 7f2e74fa0640 1 -- 172.21.15.16:0/456932500 >> v1:172.21.15.16:6789/0 conn(0x7f2e70070430 legacy=0x7f2e700ad410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:44.414 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:44.407+0000 7f2e74fa0640 1 -- 172.21.15.16:0/456932500 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e7015e360 con 0x7f2e70153fe0 2023-05-22T15:46:44.415 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:44.411+0000 7f2e74fa0640 1 --2- 172.21.15.16:0/456932500 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e70153fe0 0x7f2e7006d3f0 secure :-1 s=READY pgs=1916 cs=0 l=1 rev1=1 crypto rx=0x7f2e700b46f0 tx=0x7f2e6001c430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c492b92067283d5f server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:44.415 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:44.411+0000 7f2e6f7fe640 1 -- 172.21.15.16:0/456932500 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e600a7070 con 0x7f2e70153fe0 2023-05-22T15:46:44.415 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:44.411+0000 7f2e6f7fe640 1 -- 172.21.15.16:0/456932500 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e600ab430 con 0x7f2e70153fe0 2023-05-22T15:46:44.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:44.411+0000 7f2e6f7fe640 1 -- 172.21.15.16:0/456932500 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e600ab7c0 con 0x7f2e70153fe0 2023-05-22T15:46:44.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:44.411+0000 7f2e77204640 1 -- 172.21.15.16:0/456932500 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e70153fe0 msgr2=0x7f2e7006d3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:44.417 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:44.411+0000 7f2e77204640 1 --2- 172.21.15.16:0/456932500 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e70153fe0 0x7f2e7006d3f0 secure :-1 s=READY pgs=1916 cs=0 l=1 rev1=1 crypto rx=0x7f2e700b46f0 tx=0x7f2e6001c430 comp rx=0 tx=0).stop 2023-05-22T15:46:44.417 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:44.411+0000 7f2e77204640 1 -- 172.21.15.16:0/456932500 shutdown_connections 2023-05-22T15:46:44.417 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:44.411+0000 7f2e77204640 1 --2- 172.21.15.16:0/456932500 >> v2:172.21.15.62:3300/0 conn(0x7f2e700cae20 0x7f2e7006fef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:44.417 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:44.411+0000 7f2e77204640 1 --2- 172.21.15.16:0/456932500 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e70153fe0 0x7f2e7006d3f0 unknown :-1 s=CLOSED pgs=1916 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:44.417 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:44.411+0000 7f2e77204640 1 -- 172.21.15.16:0/456932500 >> 172.21.15.16:0/456932500 conn(0x7f2e700b1110 msgr2=0x7f2e700ce830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:46:44.417 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:44.411+0000 7f2e77204640 1 -- 172.21.15.16:0/456932500 shutdown_connections 2023-05-22T15:46:44.417 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:44.411+0000 7f2e77204640 1 -- 172.21.15.16:0/456932500 wait complete. 2023-05-22T15:46:44.420 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:44.411+0000 7f2e77204640 1 Processor -- start 2023-05-22T15:46:44.420 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:44.411+0000 7f2e77204640 1 -- start start 2023-05-22T15:46:44.420 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:44.411+0000 7f2e77204640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e70070430 0x7f2e70146e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:44.420 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:44.411+0000 7f2e77204640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2e700cae20 0x7f2e70150110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:44.420 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:44.411+0000 7f2e77204640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2e70153fe0 0x7f2e700ce760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:44.420 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:44.411+0000 7f2e77204640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e7014bc20 con 0x7f2e70070430 2023-05-22T15:46:44.421 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:44.411+0000 7f2e77204640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e7014b0a0 con 0x7f2e700cae20 2023-05-22T15:46:44.421 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:44.411+0000 7f2e77204640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e7014b2f0 con 0x7f2e70153fe0 2023-05-22T15:46:44.421 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:44.411+0000 7f2e74fa0640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e70070430 0x7f2e70146e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:44.421 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:44.411+0000 7f2e74fa0640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e70070430 0x7f2e70146e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41456/0 (socket says 172.21.15.16:41456) 2023-05-22T15:46:44.421 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:44.411+0000 7f2e74fa0640 1 -- 172.21.15.16:0/704006997 learned_addr learned my addr 172.21.15.16:0/704006997 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:46:44.421 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:44.411+0000 7f2e6ffff640 1 --2- 172.21.15.16:0/704006997 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2e700cae20 0x7f2e70150110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:44.421 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:44.411+0000 7f2e757a1640 1 --2- 172.21.15.16:0/704006997 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2e70153fe0 0x7f2e700ce760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:44.421 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:44.411+0000 7f2e74fa0640 1 -- 172.21.15.16:0/704006997 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2e70153fe0 msgr2=0x7f2e700ce760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:44.422 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:44.411+0000 7f2e74fa0640 1 --2- 172.21.15.16:0/704006997 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2e70153fe0 0x7f2e700ce760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:44.422 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:44.411+0000 7f2e74fa0640 1 -- 172.21.15.16:0/704006997 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2e700cae20 msgr2=0x7f2e70150110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:44.422 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:44.411+0000 7f2e74fa0640 1 --2- 172.21.15.16:0/704006997 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2e700cae20 0x7f2e70150110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:44.422 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:44.411+0000 7f2e74fa0640 1 -- 172.21.15.16:0/704006997 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e60009420 con 0x7f2e70070430 2023-05-22T15:46:44.422 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:44.415+0000 7f2e74fa0640 1 --2- 172.21.15.16:0/704006997 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e70070430 0x7f2e70146e80 secure :-1 s=READY pgs=1917 cs=0 l=1 rev1=1 crypto rx=0x7f2e600ad040 tx=0x7f2e6001ac70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:44.422 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:44.415+0000 7f2e6d7fa640 1 -- 172.21.15.16:0/704006997 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e600b5810 con 0x7f2e70070430 2023-05-22T15:46:44.422 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:44.415+0000 7f2e6d7fa640 1 -- 172.21.15.16:0/704006997 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e600b5990 con 0x7f2e70070430 2023-05-22T15:46:44.422 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:44.415+0000 7f2e77204640 1 -- 172.21.15.16:0/704006997 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e7015de40 con 0x7f2e70070430 2023-05-22T15:46:44.423 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:44.415+0000 7f2e6d7fa640 1 -- 172.21.15.16:0/704006997 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e600b5d40 con 0x7f2e70070430 2023-05-22T15:46:44.423 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:44.415+0000 7f2e77204640 1 -- 172.21.15.16:0/704006997 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e7015db60 con 0x7f2e70070430 2023-05-22T15:46:44.423 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:44.415+0000 7f2e77204640 1 -- 172.21.15.16:0/704006997 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e7006dd20 con 0x7f2e70070430 2023-05-22T15:46:44.423 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:44.415+0000 7f2e6d7fa640 1 -- 172.21.15.16:0/704006997 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 64) v1 ==== 84389+0+0 (secure 0 0 0) 0x7f2e600bf410 con 0x7f2e70070430 2023-05-22T15:46:44.424 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:44.415+0000 7f2e6d7fa640 1 --2- 172.21.15.16:0/704006997 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f2e50065530 0x7f2e500679e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:44.424 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:44.415+0000 7f2e6ffff640 1 -- 172.21.15.16:0/704006997 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f2e50065530 msgr2=0x7f2e500679e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/36921 2023-05-22T15:46:44.424 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:44.415+0000 7f2e6d7fa640 1 -- 172.21.15.16:0/704006997 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 6754+0+0 (secure 0 0 0) 0x7f2e6011d730 con 0x7f2e70070430 2023-05-22T15:46:44.424 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:44.415+0000 7f2e6ffff640 1 --2- 172.21.15.16:0/704006997 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f2e50065530 0x7f2e500679e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-05-22T15:46:44.425 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:44.419+0000 7f2e6d7fa640 1 -- 172.21.15.16:0/704006997 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f2e600eae40 con 0x7f2e70070430 2023-05-22T15:46:44.579 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:44.575+0000 7f2e77204640 1 -- 172.21.15.16:0/704006997 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2e7014b430 con 0x7f2e70070430 2023-05-22T15:46:44.581 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:44.575+0000 7f2e6d7fa640 1 -- 172.21.15.16:0/704006997 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v148) v1 ==== 74+0+18585 (secure 0 0 0) 0x7f2e7014b430 con 0x7f2e70070430 2023-05-22T15:46:44.581 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:46:44.582 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":148,"fsid":"b2719834-7847-4d98-90a6-f65d57f85c49","created":"2023-05-22T15:26:25.259314+0000","modified":"2023-05-22T15:46:41.068637+0000","last_up_change":"2023-05-22T15:26:36.448524+0000","last_in_change":"2023-05-22T15:26:27.675668+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-05-22T15:26:38.019781+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-05-22T15:26:43.570514+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2023-05-22T15:46:21.084615+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"146","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_data","create_time":"2023-05-22T15:46:22.440577+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"147","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9e7ff854-b25f-4cbb-a5c2-1799d9c53d2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":37021},{"type":"v1","addr":"172.21.15.16:6825","nonce":37021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":37021},{"type":"v1","addr":"172.21.15.16:6827","nonce":37021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":37021},{"type":"v1","addr":"172.21.15.16:6831","nonce":37021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":37021},{"type":"v1","addr":"172.21.15.16:6829","nonce":37021}]},"public_addr":"172.21.15.16:6825/37021","cluster_addr":"172.21.15.16:6827/37021","heartbeat_back_addr":"172.21.15.16:6831/37021","heartbeat_front_addr":"172.21.15.16:6829/37021","state":["exists","up"]},{"osd":1,"uuid":"4702fbc9-a927-4565-aca5-41dc69cc8264","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":37008},{"type":"v1","addr":"172.21.15.16:6801","nonce":37008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":37008},{"type":"v1","addr":"172.21.15.16:6803","nonce":37008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":37008},{"type":"v1","addr":"172.21.15.16:6807","nonce":37008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":37008},{"type":"v1","addr":"172.21.15.16:6805","nonce":37008}]},"public_addr":"172.21.15.16:6801/37008","cluster_addr":"172.21.15.16:6803/37008","heartbeat_back_addr":"172.21.15.16:6807/37008","heartbeat_front_addr":"172.21.15.16:6805/37008","state":["exists","up"]},{"osd":2,"uuid":"1e7bba46-4eba-47ef-bb81-5e2d371638ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":37009},{"type":"v1","addr":"172.21.15.16:6809","nonce":37009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":37009},{"type":"v1","addr":"172.21.15.16:6811","nonce":37009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":37009},{"type":"v1","addr":"172.21.15.16:6815","nonce":37009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":37009},{"type":"v1","addr":"172.21.15.16:6813","nonce":37009}]},"public_addr":"172.21.15.16:6809/37009","cluster_addr":"172.21.15.16:6811/37009","heartbeat_back_addr":"172.21.15.16:6815/37009","heartbeat_front_addr":"172.21.15.16:6813/37009","state":["exists","up"]},{"osd":3,"uuid":"383cacc0-5a88-4904-990e-498ca59ef48a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":37020},{"type":"v1","addr":"172.21.15.16:6817","nonce":37020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":37020},{"type":"v1","addr":"172.21.15.16:6819","nonce":37020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":37020},{"type":"v1","addr":"172.21.15.16:6823","nonce":37020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":37020},{"type":"v1","addr":"172.21.15.16:6821","nonce":37020}]},"public_addr":"172.21.15.16:6817/37020","cluster_addr":"172.21.15.16:6819/37020","heartbeat_back_addr":"172.21.15.16:6823/37020","heartbeat_front_addr":"172.21.15.16:6821/37020","state":["exists","up"]},{"osd":4,"uuid":"970500ef-95b7-42c1-b080-eed4fae7013d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":37353},{"type":"v1","addr":"172.21.15.62:6817","nonce":37353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6819","nonce":37353},{"type":"v1","addr":"172.21.15.62:6822","nonce":37353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37353},{"type":"v1","addr":"172.21.15.62:6831","nonce":37353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37353},{"type":"v1","addr":"172.21.15.62:6826","nonce":37353}]},"public_addr":"172.21.15.62:6817/37353","cluster_addr":"172.21.15.62:6822/37353","heartbeat_back_addr":"172.21.15.62:6831/37353","heartbeat_front_addr":"172.21.15.62:6826/37353","state":["exists","up"]},{"osd":5,"uuid":"022d67db-d4d8-445b-b1b9-8b446a00f5c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":37352},{"type":"v1","addr":"172.21.15.62:6801","nonce":37352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":37352},{"type":"v1","addr":"172.21.15.62:6803","nonce":37352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":37352},{"type":"v1","addr":"172.21.15.62:6807","nonce":37352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":37352},{"type":"v1","addr":"172.21.15.62:6805","nonce":37352}]},"public_addr":"172.21.15.62:6801/37352","cluster_addr":"172.21.15.62:6803/37352","heartbeat_back_addr":"172.21.15.62:6807/37352","heartbeat_front_addr":"172.21.15.62:6805/37352","state":["exists","up"]},{"osd":6,"uuid":"42bd372b-6688-4a1b-8407-e16e8c266e33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":37354},{"type":"v1","addr":"172.21.15.62:6820","nonce":37354}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":37354},{"type":"v1","addr":"172.21.15.62:6823","nonce":37354}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6829","nonce":37354},{"type":"v1","addr":"172.21.15.62:6830","nonce":37354}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6825","nonce":37354},{"type":"v1","addr":"172.21.15.62:6827","nonce":37354}]},"public_addr":"172.21.15.62:6820/37354","cluster_addr":"172.21.15.62:6823/37354","heartbeat_back_addr":"172.21.15.62:6830/37354","heartbeat_front_addr":"172.21.15.62:6827/37354","state":["exists","up"]},{"osd":7,"uuid":"ac70b2a3-f41b-4bf2-b227-7f2cbb55426d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37355},{"type":"v1","addr":"172.21.15.62:6809","nonce":37355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37355},{"type":"v1","addr":"172.21.15.62:6811","nonce":37355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":37355},{"type":"v1","addr":"172.21.15.62:6815","nonce":37355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":37355},{"type":"v1","addr":"172.21.15.62:6813","nonce":37355}]},"public_addr":"172.21.15.62:6809/37355","cluster_addr":"172.21.15.62:6811/37355","heartbeat_back_addr":"172.21.15.62:6815/37355","heartbeat_front_addr":"172.21.15.62:6813/37355","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.368896+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.107273+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.276888+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.287502+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.124791+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.251937+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.255679+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.232797+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:0/3429069808":"2023-05-23T15:46:41.068549+0000","172.21.15.16:0/4084148691":"2023-05-23T15:46:41.068549+0000","172.21.15.16:0/3605381301":"2023-05-23T15:46:41.068549+0000","172.21.15.16:0/2954477712":"2023-05-23T15:46:41.068549+0000","172.21.15.16:0/3271028793":"2023-05-23T15:46:41.068549+0000","172.21.15.16:6833/36921":"2023-05-23T15:46:41.068549+0000","172.21.15.16:6832/36921":"2023-05-23T15:46:41.068549+0000","172.21.15.16:0/1944100655":"2023-05-23T15:46:41.068549+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":17,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-05-22T15:46:44.584 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:44.579+0000 7f2e77204640 1 -- 172.21.15.16:0/704006997 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f2e50065530 msgr2=0x7f2e500679e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-05-22T15:46:44.584 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:44.579+0000 7f2e77204640 1 --2- 172.21.15.16:0/704006997 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f2e50065530 0x7f2e500679e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:44.584 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:44.579+0000 7f2e77204640 1 -- 172.21.15.16:0/704006997 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e70070430 msgr2=0x7f2e70146e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:44.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:44.579+0000 7f2e77204640 1 --2- 172.21.15.16:0/704006997 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e70070430 0x7f2e70146e80 secure :-1 s=READY pgs=1917 cs=0 l=1 rev1=1 crypto rx=0x7f2e600ad040 tx=0x7f2e6001ac70 comp rx=0 tx=0).stop 2023-05-22T15:46:44.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:44.579+0000 7f2e77204640 1 -- 172.21.15.16:0/704006997 shutdown_connections 2023-05-22T15:46:44.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:44.579+0000 7f2e77204640 1 --2- 172.21.15.16:0/704006997 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f2e50065530 0x7f2e500679e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:44.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:44.579+0000 7f2e77204640 1 --2- 172.21.15.16:0/704006997 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2e70153fe0 0x7f2e700ce760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:44.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:44.579+0000 7f2e77204640 1 --2- 172.21.15.16:0/704006997 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2e700cae20 0x7f2e70150110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:44.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:44.579+0000 7f2e77204640 1 --2- 172.21.15.16:0/704006997 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e70070430 0x7f2e70146e80 unknown :-1 s=CLOSED pgs=1917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:44.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:44.579+0000 7f2e77204640 1 -- 172.21.15.16:0/704006997 >> 172.21.15.16:0/704006997 conn(0x7f2e700b1110 msgr2=0x7f2e700698b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:46:44.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:44.579+0000 7f2e77204640 1 -- 172.21.15.16:0/704006997 shutdown_connections 2023-05-22T15:46:44.587 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:44.579+0000 7f2e77204640 1 -- 172.21.15.16:0/704006997 wait complete. 2023-05-22T15:46:44.600 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-05-22T15:26:38.019781+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-05-22T15:26:43.570514+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 18, 'pool_name': 'cephfs_metadata', 'create_time': '2023-05-22T15:46:21.084615+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '146', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 19, 'pool_name': 'cephfs_data', 'create_time': '2023-05-22T15:46:22.440577+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '147', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-05-22T15:46:44.601 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-05-22T15:46:44.625 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:46:44.619+0000 7f5d4045c180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-05-22T15:46:44.654 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:46:44.651+0000 7fd05b983180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-05-22T15:46:44.719 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:44.711+0000 7fab3cc69640 1 Processor -- start 2023-05-22T15:46:44.719 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:44.711+0000 7fab3cc69640 1 -- start start 2023-05-22T15:46:44.720 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:44.715+0000 7fab3cc69640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fab38071640 0x7fab38160090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:44.720 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:44.715+0000 7fab3cc69640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab38069ae0 con 0x7fab380ce480 2023-05-22T15:46:44.720 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:44.715+0000 7fab3cc69640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab38069dc0 con 0x7fab381605d0 2023-05-22T15:46:44.721 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:44.715+0000 7fab3cc69640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab3815f990 con 0x7fab38071640 2023-05-22T15:46:44.721 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:44.715+0000 7fab35d9b640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fab38071640 0x7fab38160090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:44.721 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:44.715+0000 7fab3659c640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fab380ce480 0x7fab380ce850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:57386/0 (socket says 172.21.15.16:57386) 2023-05-22T15:46:44.721 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:44.715+0000 7fab3659c640 1 -- 172.21.15.16:0/2645488310 learned_addr learned my addr 172.21.15.16:0/2645488310 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:46:44.721 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:44.715+0000 7fab35d9b640 1 -- 172.21.15.16:0/2645488310 >> v1:172.21.15.62:6789/0 conn(0x7fab381605d0 legacy=0x7fab3815c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:46:44.722 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:44.715+0000 7fab35d9b640 1 -- 172.21.15.16:0/2645488310 >> v1:172.21.15.16:6789/0 conn(0x7fab380ce480 legacy=0x7fab380ce850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:44.722 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:44.715+0000 7fab35d9b640 1 -- 172.21.15.16:0/2645488310 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab3806a0a0 con 0x7fab38071640 2023-05-22T15:46:44.722 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:44.715+0000 7fab3559a640 1 -- 172.21.15.16:0/2645488310 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1990641164 0 0) 0x7fab38069ae0 con 0x7fab380ce480 2023-05-22T15:46:44.723 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:44.715+0000 7fab35d9b640 1 --2- 172.21.15.16:0/2645488310 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab38071640 0x7fab38160090 secure :-1 s=READY pgs=1919 cs=0 l=1 rev1=1 crypto rx=0x7fab2c0097c0 tx=0x7fab2c01c410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74a4feb426097bec server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:44.723 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:44.715+0000 7fab3559a640 1 -- 172.21.15.16:0/2645488310 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab2c0ab600 con 0x7fab38071640 2023-05-22T15:46:44.723 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:44.715+0000 7fab3559a640 1 -- 172.21.15.16:0/2645488310 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab2c0a7050 con 0x7fab38071640 2023-05-22T15:46:44.723 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:44.715+0000 7fab3559a640 1 -- 172.21.15.16:0/2645488310 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab2c0ab980 con 0x7fab38071640 2023-05-22T15:46:44.724 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:44.719+0000 7fab3cc69640 1 -- 172.21.15.16:0/2645488310 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab38071640 msgr2=0x7fab38160090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:44.724 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:44.719+0000 7fab3cc69640 1 --2- 172.21.15.16:0/2645488310 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab38071640 0x7fab38160090 secure :-1 s=READY pgs=1919 cs=0 l=1 rev1=1 crypto rx=0x7fab2c0097c0 tx=0x7fab2c01c410 comp rx=0 tx=0).stop 2023-05-22T15:46:44.724 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:44.719+0000 7fab3cc69640 1 -- 172.21.15.16:0/2645488310 shutdown_connections 2023-05-22T15:46:44.724 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:44.719+0000 7fab3cc69640 1 --2- 172.21.15.16:0/2645488310 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab38071640 0x7fab38160090 unknown :-1 s=CLOSED pgs=1919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:44.725 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:44.719+0000 7fab3cc69640 1 -- 172.21.15.16:0/2645488310 >> 172.21.15.16:0/2645488310 conn(0x7fab380cc870 msgr2=0x7fab380cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:46:44.725 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:44.719+0000 7fab3cc69640 1 -- 172.21.15.16:0/2645488310 shutdown_connections 2023-05-22T15:46:44.725 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:44.719+0000 7fab3cc69640 1 -- 172.21.15.16:0/2645488310 wait complete. 2023-05-22T15:46:44.726 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:44.719+0000 7fab3cc69640 1 Processor -- start 2023-05-22T15:46:44.727 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:44.719+0000 7fab3cc69640 1 -- start start 2023-05-22T15:46:44.727 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:44.719+0000 7fab3cc69640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab38071640 0x7fab380c5610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:44.727 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:44.719+0000 7fab3cc69640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fab380ce480 0x7fab380b1250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:44.727 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:44.719+0000 7fab3cc69640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fab381605d0 0x7fab380b1790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:44.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:44.719+0000 7fab3cc69640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab38071ac0 con 0x7fab38071640 2023-05-22T15:46:44.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:44.719+0000 7fab3cc69640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab380c79d0 con 0x7fab381605d0 2023-05-22T15:46:44.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:44.719+0000 7fab3cc69640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab380c6dc0 con 0x7fab380ce480 2023-05-22T15:46:44.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:44.719+0000 7fab3659c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab38071640 0x7fab380c5610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:44.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:44.719+0000 7fab3659c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab38071640 0x7fab380c5610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41466/0 (socket says 172.21.15.16:41466) 2023-05-22T15:46:44.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:44.719+0000 7fab3659c640 1 -- 172.21.15.16:0/1757334532 learned_addr learned my addr 172.21.15.16:0/1757334532 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:46:44.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:44.719+0000 7fab36d9d640 1 --2- 172.21.15.16:0/1757334532 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fab381605d0 0x7fab380b1790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:44.729 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:44.719+0000 7fab35d9b640 1 --2- 172.21.15.16:0/1757334532 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fab380ce480 0x7fab380b1250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:44.729 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:44.719+0000 7fab3659c640 1 -- 172.21.15.16:0/1757334532 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fab380ce480 msgr2=0x7fab380b1250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:44.729 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:44.719+0000 7fab3659c640 1 --2- 172.21.15.16:0/1757334532 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fab380ce480 0x7fab380b1250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:44.729 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:44.719+0000 7fab3659c640 1 -- 172.21.15.16:0/1757334532 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fab381605d0 msgr2=0x7fab380b1790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:44.729 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:44.719+0000 7fab3659c640 1 --2- 172.21.15.16:0/1757334532 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fab381605d0 0x7fab380b1790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:44.729 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:44.719+0000 7fab3659c640 1 -- 172.21.15.16:0/1757334532 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab2c009420 con 0x7fab38071640 2023-05-22T15:46:44.730 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:44.719+0000 7fab3659c640 1 --2- 172.21.15.16:0/1757334532 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab38071640 0x7fab380c5610 secure :-1 s=READY pgs=1920 cs=0 l=1 rev1=1 crypto rx=0x7fab280011f0 tx=0x7fab2800c9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:44.730 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:44.719+0000 7fab26ffd640 1 -- 172.21.15.16:0/1757334532 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab280074e0 con 0x7fab38071640 2023-05-22T15:46:44.730 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:44.719+0000 7fab26ffd640 1 -- 172.21.15.16:0/1757334532 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab28007640 con 0x7fab38071640 2023-05-22T15:46:44.730 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:44.719+0000 7fab3cc69640 1 -- 172.21.15.16:0/1757334532 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab380c6000 con 0x7fab38071640 2023-05-22T15:46:44.730 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:44.719+0000 7fab26ffd640 1 -- 172.21.15.16:0/1757334532 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab28004a10 con 0x7fab38071640 2023-05-22T15:46:44.730 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:44.719+0000 7fab3cc69640 1 -- 172.21.15.16:0/1757334532 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab380c5e70 con 0x7fab38071640 2023-05-22T15:46:44.731 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:44.723+0000 7fab3cc69640 1 -- 172.21.15.16:0/1757334532 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab3816a4f0 con 0x7fab38071640 2023-05-22T15:46:44.732 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:44.723+0000 7fab26ffd640 1 -- 172.21.15.16:0/1757334532 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 64) v1 ==== 84389+0+0 (secure 0 0 0) 0x7fab28004b70 con 0x7fab38071640 2023-05-22T15:46:44.732 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:44.723+0000 7fab26ffd640 1 --2- 172.21.15.16:0/1757334532 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fab18065530 0x7fab180679e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:44.732 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:44.723+0000 7fab35d9b640 1 -- 172.21.15.16:0/1757334532 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fab18065530 msgr2=0x7fab180679e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/36921 2023-05-22T15:46:44.732 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:44.723+0000 7fab26ffd640 1 -- 172.21.15.16:0/1757334532 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 6754+0+0 (secure 0 0 0) 0x7fab280822c0 con 0x7fab38071640 2023-05-22T15:46:44.733 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:44.723+0000 7fab35d9b640 1 --2- 172.21.15.16:0/1757334532 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fab18065530 0x7fab180679e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-05-22T15:46:44.735 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:44.731+0000 7fab26ffd640 1 -- 172.21.15.16:0/1757334532 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fab2804fb40 con 0x7fab38071640 2023-05-22T15:46:44.764 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:46:44.759+0000 7f5d4045c180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-05-22T15:46:44.791 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:46:44.787+0000 7fd05b983180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-05-22T15:46:44.889 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:44.883+0000 7fab3cc69640 1 -- 172.21.15.16:0/1757334532 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fab3804dbe0 con 0x7fab38071640 2023-05-22T15:46:44.890 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:44.883+0000 7fab26ffd640 1 -- 172.21.15.16:0/1757334532 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v148) v1 ==== 93+0+10 (secure 0 0 0) 0x7fab28053160 con 0x7fab38071640 2023-05-22T15:46:44.890 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2023-05-22T15:46:44.892 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:44.887+0000 7fab3cc69640 1 -- 172.21.15.16:0/1757334532 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fab18065530 msgr2=0x7fab180679e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-05-22T15:46:44.892 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:44.887+0000 7fab3cc69640 1 --2- 172.21.15.16:0/1757334532 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fab18065530 0x7fab180679e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:44.892 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:44.887+0000 7fab3cc69640 1 -- 172.21.15.16:0/1757334532 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab38071640 msgr2=0x7fab380c5610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:44.892 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:44.887+0000 7fab3cc69640 1 --2- 172.21.15.16:0/1757334532 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab38071640 0x7fab380c5610 secure :-1 s=READY pgs=1920 cs=0 l=1 rev1=1 crypto rx=0x7fab280011f0 tx=0x7fab2800c9b0 comp rx=0 tx=0).stop 2023-05-22T15:46:44.892 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:44.887+0000 7fab3cc69640 1 -- 172.21.15.16:0/1757334532 shutdown_connections 2023-05-22T15:46:44.893 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:44.887+0000 7fab3cc69640 1 --2- 172.21.15.16:0/1757334532 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fab381605d0 0x7fab380b1790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:44.893 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:44.887+0000 7fab3cc69640 1 --2- 172.21.15.16:0/1757334532 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fab18065530 0x7fab180679e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:44.893 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:44.887+0000 7fab3cc69640 1 --2- 172.21.15.16:0/1757334532 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fab380ce480 0x7fab380b1250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:44.893 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:44.887+0000 7fab3cc69640 1 --2- 172.21.15.16:0/1757334532 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab38071640 0x7fab380c5610 unknown :-1 s=CLOSED pgs=1920 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:44.893 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:44.887+0000 7fab3cc69640 1 -- 172.21.15.16:0/1757334532 >> 172.21.15.16:0/1757334532 conn(0x7fab380cc870 msgr2=0x7fab38157aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:46:44.893 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:44.887+0000 7fab3cc69640 1 -- 172.21.15.16:0/1757334532 shutdown_connections 2023-05-22T15:46:44.893 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:44.887+0000 7fab3cc69640 1 -- 172.21.15.16:0/1757334532 wait complete. 2023-05-22T15:46:44.908 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-05-22T15:46:45.075 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.071+0000 7f7a892c8640 1 Processor -- start 2023-05-22T15:46:45.076 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.071+0000 7f7a892c8640 1 -- start start 2023-05-22T15:46:45.077 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.071+0000 7f7a892c8640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7a840ce480 0x7f7a84071640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:45.077 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.071+0000 7f7a892c8640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f7a8415c270 0x7f7a8415c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:45.077 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.071+0000 7f7a892c8640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a84069ae0 con 0x7f7a840ce850 2023-05-22T15:46:45.077 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.071+0000 7f7a892c8640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a8415f990 con 0x7f7a8415c270 2023-05-22T15:46:45.077 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.071+0000 7f7a892c8640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a84069dc0 con 0x7f7a840ce480 2023-05-22T15:46:45.077 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.071+0000 7f7a82ffd640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7a840ce480 0x7f7a84071640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:45.077 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.071+0000 7f7a837fe640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f7a8415c270 0x7f7a8415c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:45.078 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.071+0000 7f7a82ffd640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7a840ce480 0x7f7a84071640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3301/0 says I am v2:172.21.15.16:37918/0 (socket says 172.21.15.16:37918) 2023-05-22T15:46:45.078 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.071+0000 7f7a82ffd640 1 -- 172.21.15.16:0/4071929286 learned_addr learned my addr 172.21.15.16:0/4071929286 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:46:45.078 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.071+0000 7f7a837fe640 1 -- 172.21.15.16:0/4071929286 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7a840ce480 msgr2=0x7f7a84071640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:46:45.078 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.071+0000 7f7a837fe640 1 --2- 172.21.15.16:0/4071929286 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7a840ce480 0x7f7a84071640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:45.079 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.071+0000 7f7a837fe640 1 -- 172.21.15.16:0/4071929286 >> v1:172.21.15.62:6789/0 conn(0x7f7a840ce850 legacy=0x7f7a84160090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:45.079 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.071+0000 7f7a837fe640 1 -- 172.21.15.16:0/4071929286 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a8406a0a0 con 0x7f7a8415c270 2023-05-22T15:46:45.079 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.071+0000 7f7a82ffd640 1 --2- 172.21.15.16:0/4071929286 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7a840ce480 0x7f7a84071640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-05-22T15:46:45.079 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.075+0000 7f7a837fe640 1 --2- 172.21.15.16:0/4071929286 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7a8415c270 0x7f7a8415c6f0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f7a740097c0 tx=0x7f7a7401c450 comp rx=0 tx=0).ready entity=mon.1 client_cookie=556fa71854c934d1 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:45.080 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.075+0000 7f7a81ffb640 1 -- 172.21.15.16:0/4071929286 <== mon.1 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a740ab670 con 0x7f7a8415c270 2023-05-22T15:46:45.080 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.075+0000 7f7a81ffb640 1 -- 172.21.15.16:0/4071929286 <== mon.1 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a740ab7d0 con 0x7f7a8415c270 2023-05-22T15:46:45.080 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.075+0000 7f7a81ffb640 1 -- 172.21.15.16:0/4071929286 <== mon.1 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a740abb60 con 0x7f7a8415c270 2023-05-22T15:46:45.081 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.075+0000 7f7a892c8640 1 -- 172.21.15.16:0/4071929286 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7a8415c270 msgr2=0x7f7a8415c6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:45.081 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.075+0000 7f7a892c8640 1 --2- 172.21.15.16:0/4071929286 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7a8415c270 0x7f7a8415c6f0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f7a740097c0 tx=0x7f7a7401c450 comp rx=0 tx=0).stop 2023-05-22T15:46:45.081 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.075+0000 7f7a892c8640 1 -- 172.21.15.16:0/4071929286 shutdown_connections 2023-05-22T15:46:45.081 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.075+0000 7f7a892c8640 1 --2- 172.21.15.16:0/4071929286 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7a8415c270 0x7f7a8415c6f0 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:45.081 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.075+0000 7f7a892c8640 1 --2- 172.21.15.16:0/4071929286 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7a840ce480 0x7f7a84071640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:45.082 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.075+0000 7f7a892c8640 1 -- 172.21.15.16:0/4071929286 >> 172.21.15.16:0/4071929286 conn(0x7f7a840cc870 msgr2=0x7f7a840cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:46:45.082 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.075+0000 7f7a892c8640 1 -- 172.21.15.16:0/4071929286 shutdown_connections 2023-05-22T15:46:45.082 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.075+0000 7f7a892c8640 1 -- 172.21.15.16:0/4071929286 wait complete. 2023-05-22T15:46:45.084 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.075+0000 7f7a892c8640 1 Processor -- start 2023-05-22T15:46:45.084 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.075+0000 7f7a892c8640 1 -- start start 2023-05-22T15:46:45.084 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.075+0000 7f7a892c8640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7a840ce480 0x7f7a8415a470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:45.084 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.075+0000 7f7a892c8640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7a840ce850 0x7f7a841576a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:45.084 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.075+0000 7f7a892c8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7a8415c270 0x7f7a84157be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:45.085 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.075+0000 7f7a892c8640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a8415aa50 con 0x7f7a8415c270 2023-05-22T15:46:45.085 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.075+0000 7f7a892c8640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a8415ad30 con 0x7f7a840ce480 2023-05-22T15:46:45.085 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.075+0000 7f7a892c8640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a8415b010 con 0x7f7a840ce850 2023-05-22T15:46:45.085 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.075+0000 7f7a837fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7a8415c270 0x7f7a84157be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:45.085 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.075+0000 7f7a837fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7a8415c270 0x7f7a84157be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41480/0 (socket says 172.21.15.16:41480) 2023-05-22T15:46:45.085 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.075+0000 7f7a837fe640 1 -- 172.21.15.16:0/2934045071 learned_addr learned my addr 172.21.15.16:0/2934045071 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:46:45.085 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.075+0000 7f7a82ffd640 1 --2- 172.21.15.16:0/2934045071 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7a840ce480 0x7f7a8415a470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:45.085 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.075+0000 7f7a827fc640 1 --2- 172.21.15.16:0/2934045071 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7a840ce850 0x7f7a841576a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:45.086 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.079+0000 7f7a837fe640 1 -- 172.21.15.16:0/2934045071 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7a840ce850 msgr2=0x7f7a841576a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:45.086 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.079+0000 7f7a837fe640 1 --2- 172.21.15.16:0/2934045071 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7a840ce850 0x7f7a841576a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:45.086 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.079+0000 7f7a837fe640 1 -- 172.21.15.16:0/2934045071 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7a840ce480 msgr2=0x7f7a8415a470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:45.086 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.079+0000 7f7a837fe640 1 --2- 172.21.15.16:0/2934045071 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7a840ce480 0x7f7a8415a470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:45.086 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.079+0000 7f7a837fe640 1 -- 172.21.15.16:0/2934045071 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a7c009b30 con 0x7f7a8415c270 2023-05-22T15:46:45.086 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.079+0000 7f7a82ffd640 1 --2- 172.21.15.16:0/2934045071 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7a840ce480 0x7f7a8415a470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T15:46:45.086 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.079+0000 7f7a837fe640 1 --2- 172.21.15.16:0/2934045071 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7a8415c270 0x7f7a84157be0 secure :-1 s=READY pgs=1921 cs=0 l=1 rev1=1 crypto rx=0x7f7a7401a080 tx=0x7f7a7401ac90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:45.087 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.079+0000 7f7a7b7fe640 1 -- 172.21.15.16:0/2934045071 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a740bf610 con 0x7f7a8415c270 2023-05-22T15:46:45.087 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.079+0000 7f7a7b7fe640 1 -- 172.21.15.16:0/2934045071 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a740bf770 con 0x7f7a8415c270 2023-05-22T15:46:45.087 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.079+0000 7f7a892c8640 1 -- 172.21.15.16:0/2934045071 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a74009420 con 0x7f7a8415c270 2023-05-22T15:46:45.087 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.079+0000 7f7a7b7fe640 1 -- 172.21.15.16:0/2934045071 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a740bfb00 con 0x7f7a8415c270 2023-05-22T15:46:45.087 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.079+0000 7f7a892c8640 1 -- 172.21.15.16:0/2934045071 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a840698b0 con 0x7f7a8415c270 2023-05-22T15:46:45.087 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.079+0000 7f7a892c8640 1 -- 172.21.15.16:0/2934045071 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a84069720 con 0x7f7a8415c270 2023-05-22T15:46:45.088 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.079+0000 7f7a7b7fe640 1 -- 172.21.15.16:0/2934045071 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 64) v1 ==== 84389+0+0 (secure 0 0 0) 0x7f7a740b5400 con 0x7f7a8415c270 2023-05-22T15:46:45.088 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.079+0000 7f7a7b7fe640 1 --2- 172.21.15.16:0/2934045071 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f7a64065530 0x7f7a640679e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:45.088 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.079+0000 7f7a82ffd640 1 -- 172.21.15.16:0/2934045071 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f7a64065530 msgr2=0x7f7a640679e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/36921 2023-05-22T15:46:45.088 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.079+0000 7f7a7b7fe640 1 -- 172.21.15.16:0/2934045071 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 6754+0+0 (secure 0 0 0) 0x7f7a7411dce0 con 0x7f7a8415c270 2023-05-22T15:46:45.088 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.079+0000 7f7a82ffd640 1 --2- 172.21.15.16:0/2934045071 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f7a64065530 0x7f7a640679e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-05-22T15:46:45.089 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.083+0000 7f7a7b7fe640 1 -- 172.21.15.16:0/2934045071 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f7a740eb220 con 0x7f7a8415c270 2023-05-22T15:46:45.215 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:46:45.211+0000 7f5d4045c180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-05-22T15:46:45.235 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:46:45.231+0000 7fd05b983180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-05-22T15:46:45.240 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.235+0000 7f7a892c8640 1 -- 172.21.15.16:0/2934045071 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f7a8415cc30 con 0x7f7a8415c270 2023-05-22T15:46:45.242 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.235+0000 7f7a7b7fe640 1 -- 172.21.15.16:0/2934045071 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v148) v1 ==== 92+0+10 (secure 0 0 0) 0x7f7a740ee840 con 0x7f7a8415c270 2023-05-22T15:46:45.242 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 8 2023-05-22T15:46:45.245 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.235+0000 7f7a892c8640 1 -- 172.21.15.16:0/2934045071 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f7a64065530 msgr2=0x7f7a640679e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-05-22T15:46:45.245 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.235+0000 7f7a892c8640 1 --2- 172.21.15.16:0/2934045071 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f7a64065530 0x7f7a640679e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:45.245 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.235+0000 7f7a892c8640 1 -- 172.21.15.16:0/2934045071 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7a8415c270 msgr2=0x7f7a84157be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:45.245 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.235+0000 7f7a892c8640 1 --2- 172.21.15.16:0/2934045071 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7a8415c270 0x7f7a84157be0 secure :-1 s=READY pgs=1921 cs=0 l=1 rev1=1 crypto rx=0x7f7a7401a080 tx=0x7f7a7401ac90 comp rx=0 tx=0).stop 2023-05-22T15:46:45.245 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.235+0000 7f7a892c8640 1 -- 172.21.15.16:0/2934045071 shutdown_connections 2023-05-22T15:46:45.246 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.235+0000 7f7a892c8640 1 --2- 172.21.15.16:0/2934045071 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7a8415c270 0x7f7a84157be0 unknown :-1 s=CLOSED pgs=1921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:45.246 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.235+0000 7f7a892c8640 1 --2- 172.21.15.16:0/2934045071 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f7a64065530 0x7f7a640679e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:45.246 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.235+0000 7f7a892c8640 1 --2- 172.21.15.16:0/2934045071 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7a840ce850 0x7f7a841576a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:45.246 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.235+0000 7f7a892c8640 1 --2- 172.21.15.16:0/2934045071 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7a840ce480 0x7f7a8415a470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:45.246 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.235+0000 7f7a892c8640 1 -- 172.21.15.16:0/2934045071 >> 172.21.15.16:0/2934045071 conn(0x7f7a840cc870 msgr2=0x7f7a840cec20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:46:45.246 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.235+0000 7f7a892c8640 1 -- 172.21.15.16:0/2934045071 shutdown_connections 2023-05-22T15:46:45.246 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.235+0000 7f7a892c8640 1 -- 172.21.15.16:0/2934045071 wait complete. 2023-05-22T15:46:45.259 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-05-22T15:46:45.270 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:46:45.263+0000 7f5d4045c180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-05-22T15:46:45.288 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:46:45.283+0000 7fd05b983180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-05-22T15:46:45.323 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:46:45.315+0000 7f5d4045c180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-05-22T15:46:45.339 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:46:45.335+0000 7fd05b983180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-05-22T15:46:45.374 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.367+0000 7f492aa8e640 1 Processor -- start 2023-05-22T15:46:45.375 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.367+0000 7f492aa8e640 1 -- start start 2023-05-22T15:46:45.377 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.371+0000 7f492aa8e640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4924158a10 0x7f492406e480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:45.377 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.371+0000 7f492aa8e640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f4924070760 0x7f4924070bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:45.377 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.371+0000 7f492aa8e640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4924158de0 con 0x7f492406c6e0 2023-05-22T15:46:45.377 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.371+0000 7f492aa8e640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f492406e060 con 0x7f4924070760 2023-05-22T15:46:45.377 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.371+0000 7f492aa8e640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49240711e0 con 0x7f4924158a10 2023-05-22T15:46:45.377 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.371+0000 7f4923fff640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f4924070760 0x7f4924070bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:45.378 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.371+0000 7f4923fff640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f4924070760 0x7f4924070bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41488/0 (socket says 172.21.15.16:41488) 2023-05-22T15:46:45.378 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.371+0000 7f4923fff640 1 -- 172.21.15.16:0/184822492 learned_addr learned my addr 172.21.15.16:0/184822492 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:46:45.378 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.371+0000 7f492902b640 1 --1- 172.21.15.16:0/184822492 >> v1:172.21.15.16:6789/0 conn(0x7f492406c6e0 0x7f49240c9a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:57390/0 (socket says 172.21.15.16:57390) 2023-05-22T15:46:45.378 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.371+0000 7f492882a640 1 --2- 172.21.15.16:0/184822492 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4924158a10 0x7f492406e480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:45.378 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.371+0000 7f4923fff640 1 -- 172.21.15.16:0/184822492 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4924158a10 msgr2=0x7f492406e480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:46:45.378 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.371+0000 7f4923fff640 1 --2- 172.21.15.16:0/184822492 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4924158a10 0x7f492406e480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:45.378 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.371+0000 7f4923fff640 1 -- 172.21.15.16:0/184822492 >> v1:172.21.15.16:6789/0 conn(0x7f492406c6e0 legacy=0x7f49240c9a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:45.379 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.371+0000 7f4923fff640 1 -- 172.21.15.16:0/184822492 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49240714c0 con 0x7f4924070760 2023-05-22T15:46:45.379 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.371+0000 7f49237fe640 1 -- 172.21.15.16:0/184822492 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1730948520 0 0) 0x7f4924158de0 con 0x7f492406c6e0 2023-05-22T15:46:45.380 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.371+0000 7f4923fff640 1 --2- 172.21.15.16:0/184822492 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4924070760 0x7f4924070bb0 secure :-1 s=READY pgs=1923 cs=0 l=1 rev1=1 crypto rx=0x7f4918009870 tx=0x7f491801c4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cfa3f17d69fa6c41 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:45.380 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.371+0000 7f49237fe640 1 -- 172.21.15.16:0/184822492 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49180ac620 con 0x7f4924070760 2023-05-22T15:46:45.380 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.371+0000 7f49237fe640 1 -- 172.21.15.16:0/184822492 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49180ac780 con 0x7f4924070760 2023-05-22T15:46:45.381 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.371+0000 7f49237fe640 1 -- 172.21.15.16:0/184822492 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49180acb10 con 0x7f4924070760 2023-05-22T15:46:45.381 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.375+0000 7f492aa8e640 1 -- 172.21.15.16:0/184822492 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4924070760 msgr2=0x7f4924070bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:45.381 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.375+0000 7f492aa8e640 1 --2- 172.21.15.16:0/184822492 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4924070760 0x7f4924070bb0 secure :-1 s=READY pgs=1923 cs=0 l=1 rev1=1 crypto rx=0x7f4918009870 tx=0x7f491801c4b0 comp rx=0 tx=0).stop 2023-05-22T15:46:45.381 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.375+0000 7f492aa8e640 1 -- 172.21.15.16:0/184822492 shutdown_connections 2023-05-22T15:46:45.381 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.375+0000 7f492aa8e640 1 --2- 172.21.15.16:0/184822492 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4924070760 0x7f4924070bb0 unknown :-1 s=CLOSED pgs=1923 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:45.382 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.375+0000 7f492aa8e640 1 --2- 172.21.15.16:0/184822492 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4924158a10 0x7f492406e480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:45.382 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.375+0000 7f492aa8e640 1 -- 172.21.15.16:0/184822492 >> 172.21.15.16:0/184822492 conn(0x7f49240cc870 msgr2=0x7f49240cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:46:45.382 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.375+0000 7f492aa8e640 1 -- 172.21.15.16:0/184822492 shutdown_connections 2023-05-22T15:46:45.382 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.375+0000 7f492aa8e640 1 -- 172.21.15.16:0/184822492 wait complete. 2023-05-22T15:46:45.382 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.375+0000 7f492aa8e640 1 Processor -- start 2023-05-22T15:46:45.382 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.375+0000 7f492aa8e640 1 -- start start 2023-05-22T15:46:45.384 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.375+0000 7f492aa8e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f492406c6e0 0x7f49240ca900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:45.384 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.375+0000 7f492aa8e640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4924070760 0x7f49241561e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:45.384 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.375+0000 7f492aa8e640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4924158a10 0x7f49241493c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:45.385 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.375+0000 7f492aa8e640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49241541f0 con 0x7f492406c6e0 2023-05-22T15:46:45.385 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.375+0000 7f492aa8e640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4924152220 con 0x7f4924070760 2023-05-22T15:46:45.385 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.375+0000 7f492aa8e640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4924152500 con 0x7f4924158a10 2023-05-22T15:46:45.385 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.375+0000 7f492882a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f492406c6e0 0x7f49240ca900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:45.385 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.375+0000 7f492882a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f492406c6e0 0x7f49240ca900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41504/0 (socket says 172.21.15.16:41504) 2023-05-22T15:46:45.385 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.375+0000 7f492882a640 1 -- 172.21.15.16:0/1979286804 learned_addr learned my addr 172.21.15.16:0/1979286804 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:46:45.385 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.375+0000 7f4923fff640 1 --2- 172.21.15.16:0/1979286804 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4924070760 0x7f49241561e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:45.386 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.375+0000 7f492902b640 1 --2- 172.21.15.16:0/1979286804 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4924158a10 0x7f49241493c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:45.386 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.375+0000 7f492882a640 1 -- 172.21.15.16:0/1979286804 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4924158a10 msgr2=0x7f49241493c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:45.386 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.375+0000 7f492882a640 1 --2- 172.21.15.16:0/1979286804 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4924158a10 0x7f49241493c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:45.386 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.375+0000 7f492882a640 1 -- 172.21.15.16:0/1979286804 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4924070760 msgr2=0x7f49241561e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:45.386 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.375+0000 7f492882a640 1 --2- 172.21.15.16:0/1979286804 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4924070760 0x7f49241561e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:45.386 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.375+0000 7f492882a640 1 -- 172.21.15.16:0/1979286804 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49180094d0 con 0x7f492406c6e0 2023-05-22T15:46:45.386 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.375+0000 7f492902b640 1 --2- 172.21.15.16:0/1979286804 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4924158a10 0x7f49241493c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:46:45.386 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.375+0000 7f492882a640 1 --2- 172.21.15.16:0/1979286804 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f492406c6e0 0x7f49240ca900 secure :-1 s=READY pgs=1924 cs=0 l=1 rev1=1 crypto rx=0x7f491400d800 tx=0x7f491400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:45.387 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.375+0000 7f49217fa640 1 -- 172.21.15.16:0/1979286804 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f491400b6c0 con 0x7f492406c6e0 2023-05-22T15:46:45.387 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.375+0000 7f49217fa640 1 -- 172.21.15.16:0/1979286804 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f491400b820 con 0x7f492406c6e0 2023-05-22T15:46:45.387 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.375+0000 7f492aa8e640 1 -- 172.21.15.16:0/1979286804 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f492406e9c0 con 0x7f492406c6e0 2023-05-22T15:46:45.387 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.375+0000 7f49217fa640 1 -- 172.21.15.16:0/1979286804 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f491400bb90 con 0x7f492406c6e0 2023-05-22T15:46:45.387 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.375+0000 7f492aa8e640 1 -- 172.21.15.16:0/1979286804 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f492406cf30 con 0x7f492406c6e0 2023-05-22T15:46:45.388 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.379+0000 7f492aa8e640 1 -- 172.21.15.16:0/1979286804 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f492406eb50 con 0x7f492406c6e0 2023-05-22T15:46:45.388 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.379+0000 7f49217fa640 1 -- 172.21.15.16:0/1979286804 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 64) v1 ==== 84389+0+0 (secure 0 0 0) 0x7f491400bcf0 con 0x7f492406c6e0 2023-05-22T15:46:45.388 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.379+0000 7f49217fa640 1 --2- 172.21.15.16:0/1979286804 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f4910065530 0x7f49100679e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:45.388 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.379+0000 7f4923fff640 1 -- 172.21.15.16:0/1979286804 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f4910065530 msgr2=0x7f49100679e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/36921 2023-05-22T15:46:45.388 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.379+0000 7f49217fa640 1 -- 172.21.15.16:0/1979286804 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 6754+0+0 (secure 0 0 0) 0x7f4914083310 con 0x7f492406c6e0 2023-05-22T15:46:45.388 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.379+0000 7f4923fff640 1 --2- 172.21.15.16:0/1979286804 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f4910065530 0x7f49100679e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-05-22T15:46:45.391 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.387+0000 7f49217fa640 1 -- 172.21.15.16:0/1979286804 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f4914050ae0 con 0x7f492406c6e0 2023-05-22T15:46:45.408 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:46:45.399+0000 7f5d4045c180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-05-22T15:46:45.418 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:46:45.415+0000 7fd05b983180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-05-22T15:46:45.458 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:46:45.451+0000 7f5d4045c180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-05-22T15:46:45.468 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:46:45.467+0000 7fd05b983180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-05-22T15:46:45.535 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:46:45.531+0000 7f5d4045c180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-05-22T15:46:45.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.539+0000 7f492aa8e640 1 -- 172.21.15.16:0/1979286804 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f492404dbe0 con 0x7f492406c6e0 2023-05-22T15:46:45.544 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.539+0000 7f49217fa640 1 -- 172.21.15.16:0/1979286804 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v148) v1 ==== 104+0+11 (secure 0 0 0) 0x7f4914054100 con 0x7f492406c6e0 2023-05-22T15:46:45.544 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2023-05-22T15:46:45.545 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:46:45.543+0000 7fd05b983180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-05-22T15:46:45.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.539+0000 7f492aa8e640 1 -- 172.21.15.16:0/1979286804 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f4910065530 msgr2=0x7f49100679e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-05-22T15:46:45.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.539+0000 7f492aa8e640 1 --2- 172.21.15.16:0/1979286804 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f4910065530 0x7f49100679e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:45.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.539+0000 7f492aa8e640 1 -- 172.21.15.16:0/1979286804 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f492406c6e0 msgr2=0x7f49240ca900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:45.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.539+0000 7f492aa8e640 1 --2- 172.21.15.16:0/1979286804 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f492406c6e0 0x7f49240ca900 secure :-1 s=READY pgs=1924 cs=0 l=1 rev1=1 crypto rx=0x7f491400d800 tx=0x7f491400dcc0 comp rx=0 tx=0).stop 2023-05-22T15:46:45.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.539+0000 7f492aa8e640 1 -- 172.21.15.16:0/1979286804 shutdown_connections 2023-05-22T15:46:45.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.539+0000 7f492aa8e640 1 --2- 172.21.15.16:0/1979286804 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f4910065530 0x7f49100679e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:45.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.539+0000 7f492aa8e640 1 --2- 172.21.15.16:0/1979286804 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4924158a10 0x7f49241493c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:45.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.539+0000 7f492aa8e640 1 --2- 172.21.15.16:0/1979286804 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4924070760 0x7f49241561e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:45.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.539+0000 7f492aa8e640 1 --2- 172.21.15.16:0/1979286804 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f492406c6e0 0x7f49240ca900 unknown :-1 s=CLOSED pgs=1924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:45.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.539+0000 7f492aa8e640 1 -- 172.21.15.16:0/1979286804 >> 172.21.15.16:0/1979286804 conn(0x7f49240cc870 msgr2=0x7f4924157d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:46:45.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.539+0000 7f492aa8e640 1 -- 172.21.15.16:0/1979286804 shutdown_connections 2023-05-22T15:46:45.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.539+0000 7f492aa8e640 1 -- 172.21.15.16:0/1979286804 wait complete. 2023-05-22T15:46:45.563 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-05-22T15:46:45.594 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:46:45.583+0000 7f5d4045c180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-05-22T15:46:45.596 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:46:45.595+0000 7fd05b983180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-05-22T15:46:45.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.711+0000 7fe486f8c640 1 Processor -- start 2023-05-22T15:46:45.719 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.711+0000 7fe486f8c640 1 -- start start 2023-05-22T15:46:45.720 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.715+0000 7fe486f8c640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fe480159a90 0x7fe4800cbb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:45.720 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.715+0000 7fe486f8c640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe480159e60 con 0x7fe4800cdd90 2023-05-22T15:46:45.720 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.715+0000 7fe486f8c640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4800cc1d0 con 0x7fe4800cd770 2023-05-22T15:46:45.720 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.715+0000 7fe486f8c640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe48015d4f0 con 0x7fe480159a90 2023-05-22T15:46:45.720 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.715+0000 7fe485529640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fe4800cdd90 0x7fe4800cf470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:57404/0 (socket says 172.21.15.16:57404) 2023-05-22T15:46:45.720 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.715+0000 7fe485529640 1 -- 172.21.15.16:0/684741967 learned_addr learned my addr 172.21.15.16:0/684741967 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:46:45.721 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.715+0000 7fe484d28640 1 --2- 172.21.15.16:0/684741967 >> v2:172.21.15.62:3300/0 conn(0x7fe480159a90 0x7fe4800cbb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:45.721 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.715+0000 7fe47f7fe640 1 -- 172.21.15.16:0/684741967 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1567120028 0 0) 0x7fe480159e60 con 0x7fe4800cdd90 2023-05-22T15:46:45.721 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.715+0000 7fe47f7fe640 1 -- 172.21.15.16:0/684741967 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe460003530 con 0x7fe4800cdd90 2023-05-22T15:46:45.722 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.715+0000 7fe47f7fe640 1 -- 172.21.15.16:0/684741967 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 41103800 0 0) 0x7fe460003530 con 0x7fe4800cdd90 2023-05-22T15:46:45.722 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.715+0000 7fe47f7fe640 1 -- 172.21.15.16:0/684741967 >> v2:172.21.15.62:3300/0 conn(0x7fe480159a90 msgr2=0x7fe4800cbb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:46:45.722 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.715+0000 7fe47f7fe640 1 --2- 172.21.15.16:0/684741967 >> v2:172.21.15.62:3300/0 conn(0x7fe480159a90 0x7fe4800cbb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:45.723 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.715+0000 7fe47f7fe640 1 -- 172.21.15.16:0/684741967 >> v1:172.21.15.62:6789/0 conn(0x7fe4800cd770 legacy=0x7fe48015dbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:45.723 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.715+0000 7fe47f7fe640 1 -- 172.21.15.16:0/684741967 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4800cc4b0 con 0x7fe4800cdd90 2023-05-22T15:46:45.723 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.715+0000 7fe47f7fe640 1 -- 172.21.15.16:0/684741967 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7fe470001680 con 0x7fe4800cdd90 2023-05-22T15:46:45.723 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.715+0000 7fe47f7fe640 1 -- 172.21.15.16:0/684741967 >> v1:172.21.15.16:6789/0 conn(0x7fe4800cdd90 legacy=0x7fe4800cf470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:45.723 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.715+0000 7fe47f7fe640 1 --2- 172.21.15.16:0/684741967 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe460003980 0x7fe460003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:45.723 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.715+0000 7fe47f7fe640 1 --2- 172.21.15.16:0/684741967 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe4600047d0 0x7fe460004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:45.723 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.715+0000 7fe47f7fe640 1 --2- 172.21.15.16:0/684741967 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe460005180 0x7fe46000b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:45.724 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.715+0000 7fe47f7fe640 1 -- 172.21.15.16:0/684741967 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe48015d4f0 con 0x7fe460003980 2023-05-22T15:46:45.724 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.715+0000 7fe47f7fe640 1 -- 172.21.15.16:0/684741967 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe46000c090 con 0x7fe460005180 2023-05-22T15:46:45.724 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.715+0000 7fe47f7fe640 1 -- 172.21.15.16:0/684741967 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe46000c250 con 0x7fe4600047d0 2023-05-22T15:46:45.724 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.715+0000 7fe484d28640 1 --2- 172.21.15.16:0/684741967 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe460003980 0x7fe460003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:45.724 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.715+0000 7fe47ffff640 1 --2- 172.21.15.16:0/684741967 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe4600047d0 0x7fe460004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:45.724 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.715+0000 7fe484d28640 1 -- 172.21.15.16:0/684741967 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe4600047d0 msgr2=0x7fe460004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:46:45.724 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.715+0000 7fe484d28640 1 --2- 172.21.15.16:0/684741967 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe4600047d0 0x7fe460004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:45.725 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.715+0000 7fe484d28640 1 -- 172.21.15.16:0/684741967 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe460005180 msgr2=0x7fe46000b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:46:45.725 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.715+0000 7fe484d28640 1 --2- 172.21.15.16:0/684741967 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe460005180 0x7fe46000b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:45.725 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.715+0000 7fe484d28640 1 -- 172.21.15.16:0/684741967 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe460003530 con 0x7fe460003980 2023-05-22T15:46:45.726 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.715+0000 7fe485529640 1 -- 172.21.15.16:0/684741967 reap_dead start 2023-05-22T15:46:45.729 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.719+0000 7fe484d28640 1 --2- 172.21.15.16:0/684741967 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe460003980 0x7fe460003d90 secure :-1 s=READY pgs=1926 cs=0 l=1 rev1=1 crypto rx=0x7fe478009fc0 tx=0x7fe47801cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7e4d49140eeb061 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:45.729 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.719+0000 7fe47f7fe640 1 -- 172.21.15.16:0/684741967 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4780a5b70 con 0x7fe460003980 2023-05-22T15:46:45.729 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.719+0000 7fe47f7fe640 1 -- 172.21.15.16:0/684741967 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe47800f040 con 0x7fe460003980 2023-05-22T15:46:45.729 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.719+0000 7fe47f7fe640 1 -- 172.21.15.16:0/684741967 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4780ad440 con 0x7fe460003980 2023-05-22T15:46:45.729 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.719+0000 7fe486f8c640 1 -- 172.21.15.16:0/684741967 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe460003980 msgr2=0x7fe460003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:45.730 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.719+0000 7fe486f8c640 1 --2- 172.21.15.16:0/684741967 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe460003980 0x7fe460003d90 secure :-1 s=READY pgs=1926 cs=0 l=1 rev1=1 crypto rx=0x7fe478009fc0 tx=0x7fe47801cc70 comp rx=0 tx=0).stop 2023-05-22T15:46:45.730 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.719+0000 7fe486f8c640 1 -- 172.21.15.16:0/684741967 shutdown_connections 2023-05-22T15:46:45.730 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.719+0000 7fe486f8c640 1 --2- 172.21.15.16:0/684741967 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe460003980 0x7fe460003d90 unknown :-1 s=CLOSED pgs=1926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:45.730 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.719+0000 7fe486f8c640 1 -- 172.21.15.16:0/684741967 >> 172.21.15.16:0/684741967 conn(0x7fe48006f4f0 msgr2=0x7fe48006f8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:46:45.730 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.719+0000 7fe486f8c640 1 -- 172.21.15.16:0/684741967 shutdown_connections 2023-05-22T15:46:45.730 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.719+0000 7fe486f8c640 1 -- 172.21.15.16:0/684741967 wait complete. 2023-05-22T15:46:45.730 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.719+0000 7fe486f8c640 1 Processor -- start 2023-05-22T15:46:45.731 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.719+0000 7fe486f8c640 1 -- start start 2023-05-22T15:46:45.731 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.723+0000 7fe486f8c640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe460003980 0x7fe4800b4530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:45.731 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.723+0000 7fe486f8c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe4800b1b00 0x7fe480152440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:45.731 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.723+0000 7fe486f8c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe48015aeb0 0x7fe4800c8370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:45.731 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.723+0000 7fe486f8c640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe48015b820 con 0x7fe48015aeb0 2023-05-22T15:46:45.731 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.723+0000 7fe486f8c640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe48015bb00 con 0x7fe4800b1b00 2023-05-22T15:46:45.731 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.723+0000 7fe486f8c640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe48015bde0 con 0x7fe460003980 2023-05-22T15:46:45.731 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.723+0000 7fe485529640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe48015aeb0 0x7fe4800c8370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:45.732 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.723+0000 7fe485529640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe48015aeb0 0x7fe4800c8370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41518/0 (socket says 172.21.15.16:41518) 2023-05-22T15:46:45.732 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.723+0000 7fe485529640 1 -- 172.21.15.16:0/2232185962 learned_addr learned my addr 172.21.15.16:0/2232185962 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:46:45.732 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.723+0000 7fe47ffff640 1 --2- 172.21.15.16:0/2232185962 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe4800b1b00 0x7fe480152440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:45.732 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.723+0000 7fe484d28640 1 --2- 172.21.15.16:0/2232185962 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe460003980 0x7fe4800b4530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:45.732 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.723+0000 7fe485529640 1 -- 172.21.15.16:0/2232185962 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe460003980 msgr2=0x7fe4800b4530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:45.732 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.723+0000 7fe485529640 1 --2- 172.21.15.16:0/2232185962 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe460003980 0x7fe4800b4530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:45.732 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.723+0000 7fe485529640 1 -- 172.21.15.16:0/2232185962 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe4800b1b00 msgr2=0x7fe480152440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:45.733 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.723+0000 7fe485529640 1 --2- 172.21.15.16:0/2232185962 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe4800b1b00 0x7fe480152440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:45.733 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.723+0000 7fe485529640 1 -- 172.21.15.16:0/2232185962 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4800cd2e0 con 0x7fe48015aeb0 2023-05-22T15:46:45.733 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.723+0000 7fe484d28640 1 --2- 172.21.15.16:0/2232185962 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe460003980 0x7fe4800b4530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:46:45.733 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.723+0000 7fe47ffff640 1 --2- 172.21.15.16:0/2232185962 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe4800b1b00 0x7fe480152440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:46:45.733 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:46:45.723+0000 7f5d4045c180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-05-22T15:46:45.734 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:46:45.727+0000 7fd05b983180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-05-22T15:46:45.735 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.723+0000 7fe485529640 1 --2- 172.21.15.16:0/2232185962 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe48015aeb0 0x7fe4800c8370 secure :-1 s=READY pgs=1927 cs=0 l=1 rev1=1 crypto rx=0x7fe470000f70 tx=0x7fe47000c740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:45.735 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.723+0000 7fe47d7fa640 1 -- 172.21.15.16:0/2232185962 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe47000ce60 con 0x7fe48015aeb0 2023-05-22T15:46:45.735 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.723+0000 7fe47d7fa640 1 -- 172.21.15.16:0/2232185962 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe470007030 con 0x7fe48015aeb0 2023-05-22T15:46:45.736 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.723+0000 7fe486f8c640 1 -- 172.21.15.16:0/2232185962 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe48015e400 con 0x7fe48015aeb0 2023-05-22T15:46:45.736 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.723+0000 7fe47d7fa640 1 -- 172.21.15.16:0/2232185962 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4700073e0 con 0x7fe48015aeb0 2023-05-22T15:46:45.736 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.723+0000 7fe486f8c640 1 -- 172.21.15.16:0/2232185962 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4800ce570 con 0x7fe48015aeb0 2023-05-22T15:46:45.736 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.723+0000 7fe486f8c640 1 -- 172.21.15.16:0/2232185962 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4800b0000 con 0x7fe48015aeb0 2023-05-22T15:46:45.736 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.723+0000 7fe47d7fa640 1 -- 172.21.15.16:0/2232185962 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 64) v1 ==== 84389+0+0 (secure 0 0 0) 0x7fe470007540 con 0x7fe48015aeb0 2023-05-22T15:46:45.736 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.723+0000 7fe47d7fa640 1 --2- 172.21.15.16:0/2232185962 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fe464065530 0x7fe4640679e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:45.736 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.723+0000 7fe484d28640 1 -- 172.21.15.16:0/2232185962 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fe464065530 msgr2=0x7fe4640679e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/36921 2023-05-22T15:46:45.737 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.723+0000 7fe47d7fa640 1 -- 172.21.15.16:0/2232185962 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 6754+0+0 (secure 0 0 0) 0x7fe470081920 con 0x7fe48015aeb0 2023-05-22T15:46:45.737 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.723+0000 7fe484d28640 1 --2- 172.21.15.16:0/2232185962 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fe464065530 0x7fe4640679e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-05-22T15:46:45.737 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.727+0000 7fe47d7fa640 1 -- 172.21.15.16:0/2232185962 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fe47004f070 con 0x7fe48015aeb0 2023-05-22T15:46:45.782 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:46:45.775+0000 7f5d4045c180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-05-22T15:46:45.782 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:46:45.779+0000 7fd05b983180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-05-22T15:46:45.883 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.879+0000 7fe486f8c640 1 -- 172.21.15.16:0/2232185962 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fe48015c000 con 0x7fe48015aeb0 2023-05-22T15:46:45.884 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.879+0000 7fe47d7fa640 1 -- 172.21.15.16:0/2232185962 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v148) v1 ==== 100+0+11 (secure 0 0 0) 0x7fe470052690 con 0x7fe48015aeb0 2023-05-22T15:46:45.885 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 64 2023-05-22T15:46:45.886 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.879+0000 7fe486f8c640 1 -- 172.21.15.16:0/2232185962 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fe464065530 msgr2=0x7fe4640679e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-05-22T15:46:45.886 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.879+0000 7fe486f8c640 1 --2- 172.21.15.16:0/2232185962 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fe464065530 0x7fe4640679e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:45.887 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.879+0000 7fe486f8c640 1 -- 172.21.15.16:0/2232185962 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe48015aeb0 msgr2=0x7fe4800c8370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:45.887 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.879+0000 7fe486f8c640 1 --2- 172.21.15.16:0/2232185962 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe48015aeb0 0x7fe4800c8370 secure :-1 s=READY pgs=1927 cs=0 l=1 rev1=1 crypto rx=0x7fe470000f70 tx=0x7fe47000c740 comp rx=0 tx=0).stop 2023-05-22T15:46:45.887 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.879+0000 7fe486f8c640 1 -- 172.21.15.16:0/2232185962 shutdown_connections 2023-05-22T15:46:45.887 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.879+0000 7fe486f8c640 1 --2- 172.21.15.16:0/2232185962 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe48015aeb0 0x7fe4800c8370 unknown :-1 s=CLOSED pgs=1927 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:45.887 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.879+0000 7fe486f8c640 1 --2- 172.21.15.16:0/2232185962 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe4800b1b00 0x7fe480152440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:45.887 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.879+0000 7fe486f8c640 1 --2- 172.21.15.16:0/2232185962 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fe464065530 0x7fe4640679e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:45.888 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.879+0000 7fe486f8c640 1 --2- 172.21.15.16:0/2232185962 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe460003980 0x7fe4800b4530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:45.888 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.879+0000 7fe486f8c640 1 -- 172.21.15.16:0/2232185962 >> 172.21.15.16:0/2232185962 conn(0x7fe48006f4f0 msgr2=0x7fe4800c64e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:46:45.888 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.879+0000 7fe486f8c640 1 -- 172.21.15.16:0/2232185962 shutdown_connections 2023-05-22T15:46:45.888 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:45.879+0000 7fe486f8c640 1 -- 172.21.15.16:0/2232185962 wait complete. 2023-05-22T15:46:45.902 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-05-22T15:46:46.008 INFO:tasks.ceph.mgr.x.smithi062.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2023-05-22T15:46:46.008 INFO:tasks.ceph.mgr.x.smithi062.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2023-05-22T15:46:46.008 INFO:tasks.ceph.mgr.x.smithi062.stderr: from numpy import show_config as show_numpy_config 2023-05-22T15:46:46.011 INFO:tasks.ceph.mgr.y.smithi016.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2023-05-22T15:46:46.011 INFO:tasks.ceph.mgr.y.smithi016.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2023-05-22T15:46:46.012 INFO:tasks.ceph.mgr.y.smithi016.stderr: from numpy import show_config as show_numpy_config 2023-05-22T15:46:46.016 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:46:46.011+0000 7fd05b983180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-05-22T15:46:46.019 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:46:46.015+0000 7f5d4045c180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-05-22T15:46:46.050 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.043+0000 7f9916737640 1 Processor -- start 2023-05-22T15:46:46.051 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.043+0000 7f9916737640 1 -- start start 2023-05-22T15:46:46.052 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.047+0000 7f9916737640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f99100ce480 0x7f9910071640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:46.052 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.047+0000 7f9916737640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f991015c270 0x7f991015c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:46.052 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.047+0000 7f9916737640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9910069ae0 con 0x7f99100ce850 2023-05-22T15:46:46.052 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.047+0000 7f9916737640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f991015f990 con 0x7f99100ce480 2023-05-22T15:46:46.052 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.047+0000 7f9916737640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9910069dc0 con 0x7f991015c270 2023-05-22T15:46:46.052 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.047+0000 7f990ffff640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f99100ce480 0x7f9910071640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:46.053 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.047+0000 7f990f7fe640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f99100ce850 0x7f9910160090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:57418/0 (socket says 172.21.15.16:57418) 2023-05-22T15:46:46.053 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.047+0000 7f990f7fe640 1 -- 172.21.15.16:0/3851954682 learned_addr learned my addr 172.21.15.16:0/3851954682 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:46:46.053 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.047+0000 7f9914cd4640 1 --2- 172.21.15.16:0/3851954682 >> v2:172.21.15.62:3300/0 conn(0x7f991015c270 0x7f991015c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:46.053 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.047+0000 7f990ffff640 1 -- 172.21.15.16:0/3851954682 >> v2:172.21.15.62:3300/0 conn(0x7f991015c270 msgr2=0x7f991015c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:46:46.053 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.047+0000 7f990ffff640 1 --2- 172.21.15.16:0/3851954682 >> v2:172.21.15.62:3300/0 conn(0x7f991015c270 0x7f991015c6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:46.053 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.047+0000 7f990ffff640 1 -- 172.21.15.16:0/3851954682 >> v1:172.21.15.16:6789/0 conn(0x7f99100ce850 legacy=0x7f9910160090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:46.053 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.047+0000 7f990ffff640 1 -- 172.21.15.16:0/3851954682 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f991006a0a0 con 0x7f99100ce480 2023-05-22T15:46:46.053 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.047+0000 7f990effd640 1 -- 172.21.15.16:0/3851954682 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2816719872 0 0) 0x7f9910069ae0 con 0x7f99100ce850 2023-05-22T15:46:46.054 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.047+0000 7f9914cd4640 1 --2- 172.21.15.16:0/3851954682 >> v2:172.21.15.62:3300/0 conn(0x7f991015c270 0x7f991015c6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:46:46.055 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.047+0000 7f990ffff640 1 --2- 172.21.15.16:0/3851954682 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f99100ce480 0x7f9910071640 secure :-1 s=READY pgs=1929 cs=0 l=1 rev1=1 crypto rx=0x7f99100aeb30 tx=0x7f990401c430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5943584443a3f376 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:46.055 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.047+0000 7f990effd640 1 -- 172.21.15.16:0/3851954682 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99040ab650 con 0x7f99100ce480 2023-05-22T15:46:46.055 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.047+0000 7f990effd640 1 -- 172.21.15.16:0/3851954682 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99040ab7b0 con 0x7f99100ce480 2023-05-22T15:46:46.055 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.047+0000 7f990effd640 1 -- 172.21.15.16:0/3851954682 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99040abb40 con 0x7f99100ce480 2023-05-22T15:46:46.055 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.047+0000 7f9916737640 1 -- 172.21.15.16:0/3851954682 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f99100ce480 msgr2=0x7f9910071640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:46.055 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.047+0000 7f9916737640 1 --2- 172.21.15.16:0/3851954682 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f99100ce480 0x7f9910071640 secure :-1 s=READY pgs=1929 cs=0 l=1 rev1=1 crypto rx=0x7f99100aeb30 tx=0x7f990401c430 comp rx=0 tx=0).stop 2023-05-22T15:46:46.055 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.047+0000 7f9916737640 1 -- 172.21.15.16:0/3851954682 shutdown_connections 2023-05-22T15:46:46.056 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.047+0000 7f9916737640 1 --2- 172.21.15.16:0/3851954682 >> v2:172.21.15.62:3300/0 conn(0x7f991015c270 0x7f991015c6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:46.056 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.047+0000 7f9916737640 1 --2- 172.21.15.16:0/3851954682 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f99100ce480 0x7f9910071640 unknown :-1 s=CLOSED pgs=1929 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:46.056 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.047+0000 7f9916737640 1 -- 172.21.15.16:0/3851954682 >> 172.21.15.16:0/3851954682 conn(0x7f99100cc870 msgr2=0x7f99100cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:46:46.056 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.047+0000 7f9916737640 1 -- 172.21.15.16:0/3851954682 shutdown_connections 2023-05-22T15:46:46.056 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.047+0000 7f9916737640 1 -- 172.21.15.16:0/3851954682 wait complete. 2023-05-22T15:46:46.058 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.051+0000 7f9916737640 1 Processor -- start 2023-05-22T15:46:46.058 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.051+0000 7f9916737640 1 -- start start 2023-05-22T15:46:46.058 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.051+0000 7f9916737640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f99100ce480 0x7f99101551f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:46.059 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.051+0000 7f9916737640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f99100ce850 0x7f99100ab7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:46.059 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.051+0000 7f9916737640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f991015c270 0x7f991015c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:46.059 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.051+0000 7f9916737640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99100ae700 con 0x7f99100ce850 2023-05-22T15:46:46.059 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.051+0000 7f9916737640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99100ae270 con 0x7f991015c270 2023-05-22T15:46:46.059 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.051+0000 7f9916737640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99100ad660 con 0x7f99100ce480 2023-05-22T15:46:46.059 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.051+0000 7f990f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f99100ce850 0x7f99100ab7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:46.060 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.051+0000 7f990f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f99100ce850 0x7f99100ab7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41526/0 (socket says 172.21.15.16:41526) 2023-05-22T15:46:46.060 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.051+0000 7f990f7fe640 1 -- 172.21.15.16:0/3185988707 learned_addr learned my addr 172.21.15.16:0/3185988707 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:46:46.060 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.051+0000 7f990ffff640 1 --2- 172.21.15.16:0/3185988707 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f99100ce480 0x7f99101551f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:46.060 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.051+0000 7f9914cd4640 1 --2- 172.21.15.16:0/3185988707 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f991015c270 0x7f991015c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:46.060 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.051+0000 7f990f7fe640 1 -- 172.21.15.16:0/3185988707 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f99100ce480 msgr2=0x7f99101551f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:46.060 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.051+0000 7f990f7fe640 1 --2- 172.21.15.16:0/3185988707 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f99100ce480 0x7f99101551f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:46.060 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.051+0000 7f990f7fe640 1 -- 172.21.15.16:0/3185988707 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f991015c270 msgr2=0x7f991015c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:46.061 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.051+0000 7f990f7fe640 1 --2- 172.21.15.16:0/3185988707 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f991015c270 0x7f991015c6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:46.061 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.051+0000 7f990f7fe640 1 -- 172.21.15.16:0/3185988707 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9904009420 con 0x7f99100ce850 2023-05-22T15:46:46.061 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.051+0000 7f990ffff640 1 --2- 172.21.15.16:0/3185988707 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f99100ce480 0x7f99101551f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T15:46:46.061 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.051+0000 7f9914cd4640 1 --2- 172.21.15.16:0/3185988707 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f991015c270 0x7f991015c6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T15:46:46.061 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.051+0000 7f990f7fe640 1 --2- 172.21.15.16:0/3185988707 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f99100ce850 0x7f99100ab7d0 secure :-1 s=READY pgs=1930 cs=0 l=1 rev1=1 crypto rx=0x7f990000c480 tx=0x7f990000c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:46.061 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.051+0000 7f990cff9640 1 -- 172.21.15.16:0/3185988707 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99000074e0 con 0x7f99100ce850 2023-05-22T15:46:46.061 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.051+0000 7f990cff9640 1 -- 172.21.15.16:0/3185988707 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9900007660 con 0x7f99100ce850 2023-05-22T15:46:46.061 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.051+0000 7f9916737640 1 -- 172.21.15.16:0/3185988707 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99100698b0 con 0x7f99100ce850 2023-05-22T15:46:46.062 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.051+0000 7f990cff9640 1 -- 172.21.15.16:0/3185988707 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9900004a30 con 0x7f99100ce850 2023-05-22T15:46:46.062 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.051+0000 7f9916737640 1 -- 172.21.15.16:0/3185988707 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9910069720 con 0x7f99100ce850 2023-05-22T15:46:46.062 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.051+0000 7f9916737640 1 -- 172.21.15.16:0/3185988707 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99100cec90 con 0x7f99100ce850 2023-05-22T15:46:46.063 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.055+0000 7f990cff9640 1 -- 172.21.15.16:0/3185988707 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 64) v1 ==== 84389+0+0 (secure 0 0 0) 0x7f9900004b90 con 0x7f99100ce850 2023-05-22T15:46:46.063 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.055+0000 7f990cff9640 1 --2- 172.21.15.16:0/3185988707 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f98fc065530 0x7f98fc0679e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:46.063 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.055+0000 7f990ffff640 1 -- 172.21.15.16:0/3185988707 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f98fc065530 msgr2=0x7f98fc0679e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/36921 2023-05-22T15:46:46.063 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.055+0000 7f990cff9640 1 -- 172.21.15.16:0/3185988707 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 6754+0+0 (secure 0 0 0) 0x7f9900081f70 con 0x7f99100ce850 2023-05-22T15:46:46.063 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.055+0000 7f990ffff640 1 --2- 172.21.15.16:0/3185988707 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f98fc065530 0x7f98fc0679e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-05-22T15:46:46.064 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.059+0000 7f990cff9640 1 -- 172.21.15.16:0/3185988707 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f990004f740 con 0x7f99100ce850 2023-05-22T15:46:46.142 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:46:46.139+0000 7fd05b983180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-05-22T15:46:46.148 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:46:46.143+0000 7f5d4045c180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-05-22T15:46:46.232 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.227+0000 7f9916737640 1 -- 172.21.15.16:0/3185988707 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f99100ae3e0 con 0x7f99100ce850 2023-05-22T15:46:46.233 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.227+0000 7f990cff9640 1 -- 172.21.15.16:0/3185988707 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 8 entries v148) v1 ==== 80+0+460 (secure 0 0 0) 0x7f9900018020 con 0x7f99100ce850 2023-05-22T15:46:46.233 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/3429069808 2023-05-23T15:46:41.068549+0000 2023-05-22T15:46:46.233 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/4084148691 2023-05-23T15:46:41.068549+0000 2023-05-22T15:46:46.234 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/3605381301 2023-05-23T15:46:41.068549+0000 2023-05-22T15:46:46.234 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/2954477712 2023-05-23T15:46:41.068549+0000 2023-05-22T15:46:46.234 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/3271028793 2023-05-23T15:46:41.068549+0000 2023-05-22T15:46:46.234 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:6833/36921 2023-05-23T15:46:41.068549+0000 2023-05-22T15:46:46.234 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:6832/36921 2023-05-23T15:46:41.068549+0000 2023-05-22T15:46:46.234 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/1944100655 2023-05-23T15:46:41.068549+0000 2023-05-22T15:46:46.236 INFO:teuthology.orchestra.run.smithi016.stderr:listed 8 entries 2023-05-22T15:46:46.236 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.227+0000 7f9916737640 1 -- 172.21.15.16:0/3185988707 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f98fc065530 msgr2=0x7f98fc0679e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-05-22T15:46:46.236 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.227+0000 7f9916737640 1 --2- 172.21.15.16:0/3185988707 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f98fc065530 0x7f98fc0679e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:46.236 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.231+0000 7f9916737640 1 -- 172.21.15.16:0/3185988707 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f99100ce850 msgr2=0x7f99100ab7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:46.237 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.231+0000 7f9916737640 1 --2- 172.21.15.16:0/3185988707 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f99100ce850 0x7f99100ab7d0 secure :-1 s=READY pgs=1930 cs=0 l=1 rev1=1 crypto rx=0x7f990000c480 tx=0x7f990000c940 comp rx=0 tx=0).stop 2023-05-22T15:46:46.237 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.231+0000 7f9916737640 1 -- 172.21.15.16:0/3185988707 shutdown_connections 2023-05-22T15:46:46.237 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.231+0000 7f9916737640 1 --2- 172.21.15.16:0/3185988707 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f991015c270 0x7f991015c6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:46.237 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.231+0000 7f9916737640 1 --2- 172.21.15.16:0/3185988707 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f99100ce850 0x7f99100ab7d0 unknown :-1 s=CLOSED pgs=1930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:46.237 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.231+0000 7f9916737640 1 --2- 172.21.15.16:0/3185988707 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f98fc065530 0x7f98fc0679e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:46.237 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.231+0000 7f9916737640 1 --2- 172.21.15.16:0/3185988707 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f99100ce480 0x7f99101551f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:46.238 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.231+0000 7f9916737640 1 -- 172.21.15.16:0/3185988707 >> 172.21.15.16:0/3185988707 conn(0x7f99100cc870 msgr2=0x7f991015d580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:46:46.238 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.231+0000 7f9916737640 1 -- 172.21.15.16:0/3185988707 shutdown_connections 2023-05-22T15:46:46.238 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.231+0000 7f9916737640 1 -- 172.21.15.16:0/3185988707 wait complete. 2023-05-22T15:46:46.251 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi016.front.sepia.ceph.com... 2023-05-22T15:46:46.251 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:46:46.252 DEBUG:teuthology.orchestra.run.smithi016:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2023-05-22T15:46:46.256 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:46:46.251+0000 7fd05b983180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-05-22T15:46:46.262 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi016.stderr:ceph-fuse[63363]: fuse finished with error 0 and tester_r 0 2023-05-22T15:46:46.263 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi016.front.sepia.ceph.com 2023-05-22T15:46:46.264 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:46:46.264 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-05-22T15:46:46.265 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:46:46.259+0000 7f5d4045c180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-05-22T15:46:46.321 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:46:46.319+0000 7fd05b983180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-05-22T15:46:46.329 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:46:46.323+0000 7f5d4045c180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-05-22T15:46:46.387 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2023-05-22T15:46:46.388 DEBUG:teuthology.orchestra.run.smithi062:> dd if=/proc/self/mounts of=/dev/stdout 2023-05-22T15:46:46.393 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16342116k,nr_inodes=4085529,mode=755,inode64 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275932k,mode=755,inode64 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev,inode64 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=13250 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275928k,nr_inodes=818982,mode=700,uid=1000,gid=1271,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275932k,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-05-22T15:46:46.394 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi062.front.sepia.ceph.com 2023-05-22T15:46:46.395 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:46:46.395 DEBUG:teuthology.orchestra.run.smithi062:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2023-05-22T15:46:46.441 INFO:teuthology.orchestra.run.smithi062.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2023-05-22T15:46:46.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-05-22T15:46:46.442 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-05-22T15:46:46.443 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2023-05-22T15:46:46.446 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:46:46.443+0000 7fd05b983180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-05-22T15:46:46.448 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16345192k,nr_inodes=4086298,mode=755,inode64 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3276544k,mode=755,inode64 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev,inode64 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=15732 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3276540k,nr_inodes=819135,mode=700,uid=1000,gid=1271,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3276544k,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-05-22T15:46:46.449 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi016.front.sepia.ceph.com 2023-05-22T15:46:46.449 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:46:46.449 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2023-05-22T15:46:46.455 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:46:46.451+0000 7f5d4045c180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-05-22T15:46:46.460 INFO:teuthology.orchestra.run.smithi016.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2023-05-22T15:46:46.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-05-22T15:46:46.461 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2023-05-22T15:46:46.461 DEBUG:teuthology.orchestra.run.smithi062:> dd if=/proc/self/mounts of=/dev/stdout 2023-05-22T15:46:46.467 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16342116k,nr_inodes=4085529,mode=755,inode64 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275932k,mode=755,inode64 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev,inode64 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=13250 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275928k,nr_inodes=818982,mode=700,uid=1000,gid=1271,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275932k,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-05-22T15:46:46.468 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi062.front.sepia.ceph.com 2023-05-22T15:46:46.469 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:46:46.469 DEBUG:teuthology.orchestra.run.smithi062:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2023-05-22T15:46:46.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-05-22T15:46:46.513 INFO:teuthology.orchestra.run.smithi062.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2023-05-22T15:46:46.514 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-05-22T15:46:46.636 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.631+0000 7f5cf1bdd640 1 Processor -- start 2023-05-22T15:46:46.636 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.631+0000 7f5cf1bdd640 1 -- start start 2023-05-22T15:46:46.638 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.631+0000 7f5cf1bdd640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f5cec0ce480 0x7f5cec071640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:46.638 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.631+0000 7f5cf1bdd640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f5cec15c270 0x7f5cec15c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:46.638 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.631+0000 7f5cf1bdd640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5cec069ae0 con 0x7f5cec0ce850 2023-05-22T15:46:46.639 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.631+0000 7f5cf1bdd640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5cec15f990 con 0x7f5cec0ce480 2023-05-22T15:46:46.639 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.631+0000 7f5cf1bdd640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5cec069dc0 con 0x7f5cec15c270 2023-05-22T15:46:46.639 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.631+0000 7f5ceb7fe640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f5cec0ce480 0x7f5cec071640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:46.639 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.631+0000 7f5ceaffd640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f5cec0ce850 0x7f5cec160090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:57434/0 (socket says 172.21.15.16:57434) 2023-05-22T15:46:46.639 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.631+0000 7f5ceaffd640 1 -- 172.21.15.16:0/2671502191 learned_addr learned my addr 172.21.15.16:0/2671502191 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:46:46.639 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.631+0000 7f5cebfff640 1 --2- 172.21.15.16:0/2671502191 >> v2:172.21.15.62:3300/0 conn(0x7f5cec15c270 0x7f5cec15c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:46.640 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.631+0000 7f5ceb7fe640 1 -- 172.21.15.16:0/2671502191 >> v2:172.21.15.62:3300/0 conn(0x7f5cec15c270 msgr2=0x7f5cec15c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:46:46.640 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.631+0000 7f5ceb7fe640 1 --2- 172.21.15.16:0/2671502191 >> v2:172.21.15.62:3300/0 conn(0x7f5cec15c270 0x7f5cec15c6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:46.640 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.631+0000 7f5ceb7fe640 1 -- 172.21.15.16:0/2671502191 >> v1:172.21.15.16:6789/0 conn(0x7f5cec0ce850 legacy=0x7f5cec160090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:46.640 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.631+0000 7f5cea7fc640 1 -- 172.21.15.16:0/2671502191 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3594370921 0 0) 0x7f5cec069ae0 con 0x7f5cec0ce850 2023-05-22T15:46:46.640 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.631+0000 7f5ceb7fe640 1 -- 172.21.15.16:0/2671502191 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cec06a0a0 con 0x7f5cec0ce480 2023-05-22T15:46:46.641 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.635+0000 7f5ceb7fe640 1 --2- 172.21.15.16:0/2671502191 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5cec0ce480 0x7f5cec071640 secure :-1 s=READY pgs=1932 cs=0 l=1 rev1=1 crypto rx=0x7f5cec0aeb30 tx=0x7f5cdc01c430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b6dd02b1f44ccdd server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:46.642 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.635+0000 7f5cea7fc640 1 -- 172.21.15.16:0/2671502191 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cdc0ab650 con 0x7f5cec0ce480 2023-05-22T15:46:46.642 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.635+0000 7f5cea7fc640 1 -- 172.21.15.16:0/2671502191 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5cdc0ab7b0 con 0x7f5cec0ce480 2023-05-22T15:46:46.642 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.635+0000 7f5cea7fc640 1 -- 172.21.15.16:0/2671502191 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cdc0abb40 con 0x7f5cec0ce480 2023-05-22T15:46:46.642 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.635+0000 7f5cf1bdd640 1 -- 172.21.15.16:0/2671502191 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5cec0ce480 msgr2=0x7f5cec071640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:46.642 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.635+0000 7f5cf1bdd640 1 --2- 172.21.15.16:0/2671502191 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5cec0ce480 0x7f5cec071640 secure :-1 s=READY pgs=1932 cs=0 l=1 rev1=1 crypto rx=0x7f5cec0aeb30 tx=0x7f5cdc01c430 comp rx=0 tx=0).stop 2023-05-22T15:46:46.642 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.635+0000 7f5cf1bdd640 1 -- 172.21.15.16:0/2671502191 shutdown_connections 2023-05-22T15:46:46.643 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.635+0000 7f5cf1bdd640 1 --2- 172.21.15.16:0/2671502191 >> v2:172.21.15.62:3300/0 conn(0x7f5cec15c270 0x7f5cec15c6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:46.643 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.635+0000 7f5cf1bdd640 1 --2- 172.21.15.16:0/2671502191 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5cec0ce480 0x7f5cec071640 unknown :-1 s=CLOSED pgs=1932 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:46.643 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.635+0000 7f5cf1bdd640 1 -- 172.21.15.16:0/2671502191 >> 172.21.15.16:0/2671502191 conn(0x7f5cec0cc870 msgr2=0x7f5cec0cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:46:46.643 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.635+0000 7f5cf1bdd640 1 -- 172.21.15.16:0/2671502191 shutdown_connections 2023-05-22T15:46:46.643 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.635+0000 7f5cf1bdd640 1 -- 172.21.15.16:0/2671502191 wait complete. 2023-05-22T15:46:46.643 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.635+0000 7f5cf1bdd640 1 Processor -- start 2023-05-22T15:46:46.643 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.635+0000 7f5cf1bdd640 1 -- start start 2023-05-22T15:46:46.645 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.635+0000 7f5cf1bdd640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5cec0ce480 0x7f5cec0addd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:46.645 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.635+0000 7f5cf1bdd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5cec0ce850 0x7f5cec0ab920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:46.646 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.635+0000 7f5cf1bdd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5cec15c270 0x7f5cec0c4860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:46.646 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.635+0000 7f5cf1bdd640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cec14fe90 con 0x7f5cec0ce850 2023-05-22T15:46:46.646 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.635+0000 7f5cf1bdd640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cec0abe60 con 0x7f5cec15c270 2023-05-22T15:46:46.646 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.635+0000 7f5cf1bdd640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5cec0c4da0 con 0x7f5cec0ce480 2023-05-22T15:46:46.646 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.635+0000 7f5ceaffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5cec0ce850 0x7f5cec0ab920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:46.647 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.635+0000 7f5ceaffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5cec0ce850 0x7f5cec0ab920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41550/0 (socket says 172.21.15.16:41550) 2023-05-22T15:46:46.647 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.635+0000 7f5ceaffd640 1 -- 172.21.15.16:0/3619813563 learned_addr learned my addr 172.21.15.16:0/3619813563 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:46:46.648 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.635+0000 7f5ceb7fe640 1 --2- 172.21.15.16:0/3619813563 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5cec0ce480 0x7f5cec0addd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:46.648 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.635+0000 7f5cebfff640 1 --2- 172.21.15.16:0/3619813563 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5cec15c270 0x7f5cec0c4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:46.648 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.635+0000 7f5ceaffd640 1 -- 172.21.15.16:0/3619813563 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5cec0ce480 msgr2=0x7f5cec0addd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:46.649 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.635+0000 7f5ceaffd640 1 --2- 172.21.15.16:0/3619813563 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5cec0ce480 0x7f5cec0addd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:46.649 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.635+0000 7f5ceaffd640 1 -- 172.21.15.16:0/3619813563 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5cec15c270 msgr2=0x7f5cec0c4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:46.649 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.635+0000 7f5ceaffd640 1 --2- 172.21.15.16:0/3619813563 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5cec15c270 0x7f5cec0c4860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:46.649 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.635+0000 7f5ceaffd640 1 -- 172.21.15.16:0/3619813563 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cdc009420 con 0x7f5cec0ce850 2023-05-22T15:46:46.649 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.635+0000 7f5ceb7fe640 1 --2- 172.21.15.16:0/3619813563 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5cec0ce480 0x7f5cec0addd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T15:46:46.649 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.639+0000 7f5ceaffd640 1 --2- 172.21.15.16:0/3619813563 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5cec0ce850 0x7f5cec0ab920 secure :-1 s=READY pgs=1933 cs=0 l=1 rev1=1 crypto rx=0x7f5cd400c480 tx=0x7f5cd400c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:46.650 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.639+0000 7f5ce3fff640 1 -- 172.21.15.16:0/3619813563 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cd40074e0 con 0x7f5cec0ce850 2023-05-22T15:46:46.650 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.639+0000 7f5ce3fff640 1 -- 172.21.15.16:0/3619813563 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5cd4007660 con 0x7f5cec0ce850 2023-05-22T15:46:46.650 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.639+0000 7f5cf1bdd640 1 -- 172.21.15.16:0/3619813563 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5cec0698b0 con 0x7f5cec0ce850 2023-05-22T15:46:46.650 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.639+0000 7f5ce3fff640 1 -- 172.21.15.16:0/3619813563 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cd4004a30 con 0x7f5cec0ce850 2023-05-22T15:46:46.651 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.639+0000 7f5cf1bdd640 1 -- 172.21.15.16:0/3619813563 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5cec069720 con 0x7f5cec0ce850 2023-05-22T15:46:46.651 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.639+0000 7f5cf1bdd640 1 -- 172.21.15.16:0/3619813563 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5cec0cec90 con 0x7f5cec0ce850 2023-05-22T15:46:46.651 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.639+0000 7f5ce3fff640 1 -- 172.21.15.16:0/3619813563 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 64) v1 ==== 84389+0+0 (secure 0 0 0) 0x7f5cd401c3b0 con 0x7f5cec0ce850 2023-05-22T15:46:46.651 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.639+0000 7f5ce3fff640 1 --2- 172.21.15.16:0/3619813563 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f5ccc065560 0x7f5ccc067a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:46.652 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.639+0000 7f5ceb7fe640 1 -- 172.21.15.16:0/3619813563 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f5ccc065560 msgr2=0x7f5ccc067a10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/36921 2023-05-22T15:46:46.652 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.639+0000 7f5ce3fff640 1 -- 172.21.15.16:0/3619813563 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 6754+0+0 (secure 0 0 0) 0x7f5cd40825c0 con 0x7f5cec0ce850 2023-05-22T15:46:46.652 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.639+0000 7f5ceb7fe640 1 --2- 172.21.15.16:0/3619813563 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f5ccc065560 0x7f5ccc067a10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-05-22T15:46:46.653 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.643+0000 7f5ce3fff640 1 -- 172.21.15.16:0/3619813563 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f5cd404fe10 con 0x7f5cec0ce850 2023-05-22T15:46:46.832 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.827+0000 7f5cf1bdd640 1 -- 172.21.15.16:0/3619813563 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5cec04dbe0 con 0x7f5cec0ce850 2023-05-22T15:46:46.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.827+0000 7f5ce3fff640 1 -- 172.21.15.16:0/3619813563 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 58 v58) v1 ==== 94+0+4650 (secure 0 0 0) 0x7f5cd4018020 con 0x7f5cec0ce850 2023-05-22T15:46:46.834 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:46:46.834 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":58,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7753,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6834/3192074681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":3192074681},{"type":"v1","addr":"172.21.15.62:6834","nonce":3192074681}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":53},{"gid":7756,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/1344243469","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6833","nonce":1344243469},{"type":"v1","addr":"172.21.15.62:6835","nonce":1344243469}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":53},{"gid":9069,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/3636102434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":3636102434},{"type":"v1","addr":"172.21.15.16:6835","nonce":3636102434}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":53}],"filesystems":[{"mdsmap":{"epoch":58,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-05-22T15:46:23.443662+0000","modified":"2023-05-22T15:46:26.557591+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9075},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9075":{"gid":9075,"name":"a","rank":0,"incarnation":54,"state":"up:active","state_seq":3,"addr":"172.21.15.16:6837/2858899325","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":2858899325},{"type":"v1","addr":"172.21.15.16:6837","nonce":2858899325}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2023-05-22T15:46:46.834 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 58 2023-05-22T15:46:46.835 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.831+0000 7f5cf1bdd640 1 -- 172.21.15.16:0/3619813563 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f5ccc065560 msgr2=0x7f5ccc067a10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-05-22T15:46:46.836 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.831+0000 7f5cf1bdd640 1 --2- 172.21.15.16:0/3619813563 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f5ccc065560 0x7f5ccc067a10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:46.836 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.831+0000 7f5cf1bdd640 1 -- 172.21.15.16:0/3619813563 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5cec0ce850 msgr2=0x7f5cec0ab920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:46.836 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.831+0000 7f5cf1bdd640 1 --2- 172.21.15.16:0/3619813563 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5cec0ce850 0x7f5cec0ab920 secure :-1 s=READY pgs=1933 cs=0 l=1 rev1=1 crypto rx=0x7f5cd400c480 tx=0x7f5cd400c940 comp rx=0 tx=0).stop 2023-05-22T15:46:46.836 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.831+0000 7f5cf1bdd640 1 -- 172.21.15.16:0/3619813563 shutdown_connections 2023-05-22T15:46:46.836 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.831+0000 7f5cf1bdd640 1 --2- 172.21.15.16:0/3619813563 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5cec15c270 0x7f5cec0c4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:46.837 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.831+0000 7f5cf1bdd640 1 --2- 172.21.15.16:0/3619813563 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5cec0ce850 0x7f5cec0ab920 unknown :-1 s=CLOSED pgs=1933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:46.837 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.831+0000 7f5cf1bdd640 1 --2- 172.21.15.16:0/3619813563 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f5ccc065560 0x7f5ccc067a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:46.837 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.831+0000 7f5cf1bdd640 1 --2- 172.21.15.16:0/3619813563 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5cec0ce480 0x7f5cec0addd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:46.837 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.831+0000 7f5cf1bdd640 1 -- 172.21.15.16:0/3619813563 >> 172.21.15.16:0/3619813563 conn(0x7f5cec0cc870 msgr2=0x7f5cec0ca130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:46:46.837 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.831+0000 7f5cf1bdd640 1 -- 172.21.15.16:0/3619813563 shutdown_connections 2023-05-22T15:46:46.837 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.831+0000 7f5cf1bdd640 1 -- 172.21.15.16:0/3619813563 wait complete. 2023-05-22T15:46:46.853 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-05-22T15:46:46.945 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:46:46.943+0000 7fd05b983180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-05-22T15:46:46.961 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:46:46.955+0000 7f5d4045c180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-05-22T15:46:46.982 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.975+0000 7fb836212640 1 Processor -- start 2023-05-22T15:46:46.983 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.975+0000 7fb836212640 1 -- start start 2023-05-22T15:46:46.984 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.979+0000 7fb836212640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb8300b28b0 0x7fb8300b2d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:46.984 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.979+0000 7fb836212640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb83006a7d0 con 0x7fb830158760 2023-05-22T15:46:46.984 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.979+0000 7fb836212640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8300b3260 con 0x7fb830158b30 2023-05-22T15:46:46.984 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.979+0000 7fb836212640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb830159df0 con 0x7fb8300b28b0 2023-05-22T15:46:46.984 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.979+0000 7fb82f7fe640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fb830158760 0x7fb83006a0c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:57448/0 (socket says 172.21.15.16:57448) 2023-05-22T15:46:46.984 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.979+0000 7fb82f7fe640 1 -- 172.21.15.16:0/1661071733 learned_addr learned my addr 172.21.15.16:0/1661071733 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:46:46.986 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.979+0000 7fb82effd640 1 --2- 172.21.15.16:0/1661071733 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb8300b28b0 0x7fb8300b2d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:46.986 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.979+0000 7fb82e7fc640 1 -- 172.21.15.16:0/1661071733 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3435553073 0 0) 0x7fb8200014a0 con 0x7fb830158760 2023-05-22T15:46:46.986 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.979+0000 7fb82e7fc640 1 -- 172.21.15.16:0/1661071733 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb818003530 con 0x7fb830158760 2023-05-22T15:46:46.987 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.979+0000 7fb82e7fc640 1 -- 172.21.15.16:0/1661071733 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2903701575 0 0) 0x7fb818003530 con 0x7fb830158760 2023-05-22T15:46:46.987 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.979+0000 7fb82e7fc640 1 -- 172.21.15.16:0/1661071733 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb8300b28b0 msgr2=0x7fb8300b2d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:46:46.987 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.979+0000 7fb82e7fc640 1 --2- 172.21.15.16:0/1661071733 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb8300b28b0 0x7fb8300b2d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:46.987 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.979+0000 7fb82e7fc640 1 -- 172.21.15.16:0/1661071733 >> v1:172.21.15.62:6789/0 conn(0x7fb830158b30 legacy=0x7fb830158fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:46.987 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.979+0000 7fb82e7fc640 1 -- 172.21.15.16:0/1661071733 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb830159830 con 0x7fb830158760 2023-05-22T15:46:46.987 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.979+0000 7fb82e7fc640 1 -- 172.21.15.16:0/1661071733 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7fb820001b90 con 0x7fb830158760 2023-05-22T15:46:46.987 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.979+0000 7fb82e7fc640 1 -- 172.21.15.16:0/1661071733 >> v1:172.21.15.16:6789/0 conn(0x7fb830158760 legacy=0x7fb83006a0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:46.988 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.979+0000 7fb82e7fc640 1 --2- 172.21.15.16:0/1661071733 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb8300b28b0 0x7fb8180039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:46.988 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.979+0000 7fb82e7fc640 1 --2- 172.21.15.16:0/1661071733 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb818003f30 0x7fb8180047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:46.988 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.979+0000 7fb82e7fc640 1 --2- 172.21.15.16:0/1661071733 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb818004d10 0x7fb818005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:46.988 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.979+0000 7fb82e7fc640 1 -- 172.21.15.16:0/1661071733 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb830159df0 con 0x7fb818003f30 2023-05-22T15:46:46.988 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.979+0000 7fb82e7fc640 1 -- 172.21.15.16:0/1661071733 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb81800ba70 con 0x7fb818004d10 2023-05-22T15:46:46.989 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.979+0000 7fb82e7fc640 1 -- 172.21.15.16:0/1661071733 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb81800bc30 con 0x7fb8300b28b0 2023-05-22T15:46:46.989 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.979+0000 7fb82effd640 1 --2- 172.21.15.16:0/1661071733 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb818003f30 0x7fb8180047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:46.989 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.979+0000 7fb82f7fe640 1 --2- 172.21.15.16:0/1661071733 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb8300b28b0 0x7fb8180039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:46.990 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.979+0000 7fb82ffff640 1 --2- 172.21.15.16:0/1661071733 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb818004d10 0x7fb818005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:46.990 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.979+0000 7fb82effd640 1 -- 172.21.15.16:0/1661071733 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb8300b28b0 msgr2=0x7fb8180039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:46:46.990 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.979+0000 7fb82effd640 1 --2- 172.21.15.16:0/1661071733 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb8300b28b0 0x7fb8180039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:46.990 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.979+0000 7fb82effd640 1 -- 172.21.15.16:0/1661071733 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb818004d10 msgr2=0x7fb818005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:46:46.991 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.979+0000 7fb82effd640 1 --2- 172.21.15.16:0/1661071733 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb818004d10 0x7fb818005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:46.991 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.979+0000 7fb82effd640 1 -- 172.21.15.16:0/1661071733 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb818003530 con 0x7fb818003f30 2023-05-22T15:46:46.991 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.979+0000 7fb82f7fe640 1 --2- 172.21.15.16:0/1661071733 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb8300b28b0 0x7fb8180039b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T15:46:46.993 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.983+0000 7fb82effd640 1 --2- 172.21.15.16:0/1661071733 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb818003f30 0x7fb8180047d0 secure :-1 s=READY pgs=1935 cs=0 l=1 rev1=1 crypto rx=0x7fb824009ab0 tx=0x7fb82401ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=716d380f2d439ded server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:46.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.983+0000 7fb82e7fc640 1 -- 172.21.15.16:0/1661071733 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8240a5510 con 0x7fb818003f30 2023-05-22T15:46:46.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.983+0000 7fb82e7fc640 1 -- 172.21.15.16:0/1661071733 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8240a5670 con 0x7fb818003f30 2023-05-22T15:46:46.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.983+0000 7fb82e7fc640 1 -- 172.21.15.16:0/1661071733 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8240a5a20 con 0x7fb818003f30 2023-05-22T15:46:46.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.983+0000 7fb836212640 1 -- 172.21.15.16:0/1661071733 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb818003f30 msgr2=0x7fb8180047d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:46.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.983+0000 7fb836212640 1 --2- 172.21.15.16:0/1661071733 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb818003f30 0x7fb8180047d0 secure :-1 s=READY pgs=1935 cs=0 l=1 rev1=1 crypto rx=0x7fb824009ab0 tx=0x7fb82401ca30 comp rx=0 tx=0).stop 2023-05-22T15:46:46.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.983+0000 7fb82ffff640 1 -- 172.21.15.16:0/1661071733 reap_dead start 2023-05-22T15:46:46.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.983+0000 7fb836212640 1 -- 172.21.15.16:0/1661071733 shutdown_connections 2023-05-22T15:46:46.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.983+0000 7fb836212640 1 -- 172.21.15.16:0/1661071733 >> 172.21.15.16:0/1661071733 conn(0x7fb83006e3c0 msgr2=0x7fb8300b1850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:46:46.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.983+0000 7fb836212640 1 -- 172.21.15.16:0/1661071733 shutdown_connections 2023-05-22T15:46:46.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.983+0000 7fb836212640 1 -- 172.21.15.16:0/1661071733 wait complete. 2023-05-22T15:46:46.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.983+0000 7fb836212640 1 Processor -- start 2023-05-22T15:46:46.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.983+0000 7fb836212640 1 -- start start 2023-05-22T15:46:46.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.983+0000 7fb836212640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb818003f30 0x7fb830150c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:46.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.983+0000 7fb836212640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb83015c4e0 0x7fb83014e380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:46.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.983+0000 7fb836212640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb8301566f0 0x7fb830156b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:46.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.983+0000 7fb836212640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8300c8300 con 0x7fb818003f30 2023-05-22T15:46:46.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.983+0000 7fb836212640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8301570d0 con 0x7fb83015c4e0 2023-05-22T15:46:46.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.983+0000 7fb836212640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb830152410 con 0x7fb8301566f0 2023-05-22T15:46:46.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.983+0000 7fb82f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb818003f30 0x7fb830150c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:46.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.983+0000 7fb82f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb818003f30 0x7fb830150c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41574/0 (socket says 172.21.15.16:41574) 2023-05-22T15:46:46.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.983+0000 7fb82f7fe640 1 -- 172.21.15.16:0/3679262167 learned_addr learned my addr 172.21.15.16:0/3679262167 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:46:46.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.983+0000 7fb82ffff640 1 --2- 172.21.15.16:0/3679262167 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb8301566f0 0x7fb830156b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:46.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.983+0000 7fb82effd640 1 --2- 172.21.15.16:0/3679262167 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb83015c4e0 0x7fb83014e380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:46.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.983+0000 7fb82f7fe640 1 -- 172.21.15.16:0/3679262167 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb8301566f0 msgr2=0x7fb830156b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:46.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.983+0000 7fb82f7fe640 1 --2- 172.21.15.16:0/3679262167 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb8301566f0 0x7fb830156b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:46.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.983+0000 7fb82f7fe640 1 -- 172.21.15.16:0/3679262167 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb83015c4e0 msgr2=0x7fb83014e380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:46.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.983+0000 7fb82f7fe640 1 --2- 172.21.15.16:0/3679262167 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb83015c4e0 0x7fb83014e380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:46.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.983+0000 7fb82f7fe640 1 -- 172.21.15.16:0/3679262167 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8300698c0 con 0x7fb818003f30 2023-05-22T15:46:46.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.987+0000 7fb82f7fe640 1 --2- 172.21.15.16:0/3679262167 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb818003f30 0x7fb830150c70 secure :-1 s=READY pgs=1936 cs=0 l=1 rev1=1 crypto rx=0x7fb8200108d0 tx=0x7fb820010d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:46.998 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.987+0000 7fb817fff640 1 -- 172.21.15.16:0/3679262167 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb820019030 con 0x7fb818003f30 2023-05-22T15:46:46.998 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.987+0000 7fb817fff640 1 -- 172.21.15.16:0/3679262167 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb82000d070 con 0x7fb818003f30 2023-05-22T15:46:46.998 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.987+0000 7fb836212640 1 -- 172.21.15.16:0/3679262167 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb830159b10 con 0x7fb818003f30 2023-05-22T15:46:46.998 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.987+0000 7fb817fff640 1 -- 172.21.15.16:0/3679262167 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb82000e980 con 0x7fb818003f30 2023-05-22T15:46:46.998 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.987+0000 7fb836212640 1 -- 172.21.15.16:0/3679262167 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb830069ba0 con 0x7fb818003f30 2023-05-22T15:46:46.998 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.987+0000 7fb836212640 1 -- 172.21.15.16:0/3679262167 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8300c8440 con 0x7fb818003f30 2023-05-22T15:46:46.998 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.987+0000 7fb817fff640 1 -- 172.21.15.16:0/3679262167 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 64) v1 ==== 84389+0+0 (secure 0 0 0) 0x7fb82001c070 con 0x7fb818003f30 2023-05-22T15:46:46.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.987+0000 7fb817fff640 1 --2- 172.21.15.16:0/3679262167 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fb81c065560 0x7fb81c067a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:46.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.987+0000 7fb82effd640 1 -- 172.21.15.16:0/3679262167 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fb81c065560 msgr2=0x7fb81c067a10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/36921 2023-05-22T15:46:46.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.987+0000 7fb817fff640 1 -- 172.21.15.16:0/3679262167 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 6754+0+0 (secure 0 0 0) 0x7fb82001a070 con 0x7fb818003f30 2023-05-22T15:46:46.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.987+0000 7fb82effd640 1 --2- 172.21.15.16:0/3679262167 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fb81c065560 0x7fb81c067a10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-05-22T15:46:47.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:46.991+0000 7fb817fff640 1 -- 172.21.15.16:0/3679262167 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fb8300c8440 con 0x7fb818003f30 2023-05-22T15:46:47.021 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:46:47.019+0000 7fd05b983180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-05-22T15:46:47.072 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:46:47.067+0000 7f5d4045c180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-05-22T15:46:47.164 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:46:47.159+0000 7fd05b983180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-05-22T15:46:47.195 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:47.191+0000 7fb82effd640 1 -- 172.21.15.16:0/3679262167 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fb81c065560 msgr2=0x7fb81c067a10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/36921 2023-05-22T15:46:47.196 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:47.191+0000 7fb82effd640 1 --2- 172.21.15.16:0/3679262167 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fb81c065560 0x7fb81c067a10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-05-22T15:46:47.239 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:46:47.235+0000 7fd05b983180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-05-22T15:46:47.272 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:47.267+0000 7fb836212640 1 -- 172.21.15.16:0/3679262167 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb83006a7d0 con 0x7fb818003f30 2023-05-22T15:46:47.273 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:47.267+0000 7fb817fff640 1 -- 172.21.15.16:0/3679262167 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v148) v1 ==== 74+0+18585 (secure 0 0 0) 0x7fb83006a7d0 con 0x7fb818003f30 2023-05-22T15:46:47.273 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:46:47.274 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":148,"fsid":"b2719834-7847-4d98-90a6-f65d57f85c49","created":"2023-05-22T15:26:25.259314+0000","modified":"2023-05-22T15:46:41.068637+0000","last_up_change":"2023-05-22T15:26:36.448524+0000","last_in_change":"2023-05-22T15:26:27.675668+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-05-22T15:26:38.019781+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-05-22T15:26:43.570514+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2023-05-22T15:46:21.084615+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"146","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_data","create_time":"2023-05-22T15:46:22.440577+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"147","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9e7ff854-b25f-4cbb-a5c2-1799d9c53d2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":37021},{"type":"v1","addr":"172.21.15.16:6825","nonce":37021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":37021},{"type":"v1","addr":"172.21.15.16:6827","nonce":37021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":37021},{"type":"v1","addr":"172.21.15.16:6831","nonce":37021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":37021},{"type":"v1","addr":"172.21.15.16:6829","nonce":37021}]},"public_addr":"172.21.15.16:6825/37021","cluster_addr":"172.21.15.16:6827/37021","heartbeat_back_addr":"172.21.15.16:6831/37021","heartbeat_front_addr":"172.21.15.16:6829/37021","state":["exists","up"]},{"osd":1,"uuid":"4702fbc9-a927-4565-aca5-41dc69cc8264","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":37008},{"type":"v1","addr":"172.21.15.16:6801","nonce":37008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":37008},{"type":"v1","addr":"172.21.15.16:6803","nonce":37008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":37008},{"type":"v1","addr":"172.21.15.16:6807","nonce":37008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":37008},{"type":"v1","addr":"172.21.15.16:6805","nonce":37008}]},"public_addr":"172.21.15.16:6801/37008","cluster_addr":"172.21.15.16:6803/37008","heartbeat_back_addr":"172.21.15.16:6807/37008","heartbeat_front_addr":"172.21.15.16:6805/37008","state":["exists","up"]},{"osd":2,"uuid":"1e7bba46-4eba-47ef-bb81-5e2d371638ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":37009},{"type":"v1","addr":"172.21.15.16:6809","nonce":37009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":37009},{"type":"v1","addr":"172.21.15.16:6811","nonce":37009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":37009},{"type":"v1","addr":"172.21.15.16:6815","nonce":37009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":37009},{"type":"v1","addr":"172.21.15.16:6813","nonce":37009}]},"public_addr":"172.21.15.16:6809/37009","cluster_addr":"172.21.15.16:6811/37009","heartbeat_back_addr":"172.21.15.16:6815/37009","heartbeat_front_addr":"172.21.15.16:6813/37009","state":["exists","up"]},{"osd":3,"uuid":"383cacc0-5a88-4904-990e-498ca59ef48a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":37020},{"type":"v1","addr":"172.21.15.16:6817","nonce":37020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":37020},{"type":"v1","addr":"172.21.15.16:6819","nonce":37020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":37020},{"type":"v1","addr":"172.21.15.16:6823","nonce":37020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":37020},{"type":"v1","addr":"172.21.15.16:6821","nonce":37020}]},"public_addr":"172.21.15.16:6817/37020","cluster_addr":"172.21.15.16:6819/37020","heartbeat_back_addr":"172.21.15.16:6823/37020","heartbeat_front_addr":"172.21.15.16:6821/37020","state":["exists","up"]},{"osd":4,"uuid":"970500ef-95b7-42c1-b080-eed4fae7013d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":37353},{"type":"v1","addr":"172.21.15.62:6817","nonce":37353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6819","nonce":37353},{"type":"v1","addr":"172.21.15.62:6822","nonce":37353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37353},{"type":"v1","addr":"172.21.15.62:6831","nonce":37353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37353},{"type":"v1","addr":"172.21.15.62:6826","nonce":37353}]},"public_addr":"172.21.15.62:6817/37353","cluster_addr":"172.21.15.62:6822/37353","heartbeat_back_addr":"172.21.15.62:6831/37353","heartbeat_front_addr":"172.21.15.62:6826/37353","state":["exists","up"]},{"osd":5,"uuid":"022d67db-d4d8-445b-b1b9-8b446a00f5c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":37352},{"type":"v1","addr":"172.21.15.62:6801","nonce":37352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":37352},{"type":"v1","addr":"172.21.15.62:6803","nonce":37352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":37352},{"type":"v1","addr":"172.21.15.62:6807","nonce":37352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":37352},{"type":"v1","addr":"172.21.15.62:6805","nonce":37352}]},"public_addr":"172.21.15.62:6801/37352","cluster_addr":"172.21.15.62:6803/37352","heartbeat_back_addr":"172.21.15.62:6807/37352","heartbeat_front_addr":"172.21.15.62:6805/37352","state":["exists","up"]},{"osd":6,"uuid":"42bd372b-6688-4a1b-8407-e16e8c266e33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":37354},{"type":"v1","addr":"172.21.15.62:6820","nonce":37354}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":37354},{"type":"v1","addr":"172.21.15.62:6823","nonce":37354}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6829","nonce":37354},{"type":"v1","addr":"172.21.15.62:6830","nonce":37354}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6825","nonce":37354},{"type":"v1","addr":"172.21.15.62:6827","nonce":37354}]},"public_addr":"172.21.15.62:6820/37354","cluster_addr":"172.21.15.62:6823/37354","heartbeat_back_addr":"172.21.15.62:6830/37354","heartbeat_front_addr":"172.21.15.62:6827/37354","state":["exists","up"]},{"osd":7,"uuid":"ac70b2a3-f41b-4bf2-b227-7f2cbb55426d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37355},{"type":"v1","addr":"172.21.15.62:6809","nonce":37355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37355},{"type":"v1","addr":"172.21.15.62:6811","nonce":37355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":37355},{"type":"v1","addr":"172.21.15.62:6815","nonce":37355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":37355},{"type":"v1","addr":"172.21.15.62:6813","nonce":37355}]},"public_addr":"172.21.15.62:6809/37355","cluster_addr":"172.21.15.62:6811/37355","heartbeat_back_addr":"172.21.15.62:6815/37355","heartbeat_front_addr":"172.21.15.62:6813/37355","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.368896+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.107273+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.276888+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.287502+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.124791+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.251937+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.255679+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.232797+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:0/3429069808":"2023-05-23T15:46:41.068549+0000","172.21.15.16:0/4084148691":"2023-05-23T15:46:41.068549+0000","172.21.15.16:0/3605381301":"2023-05-23T15:46:41.068549+0000","172.21.15.16:0/2954477712":"2023-05-23T15:46:41.068549+0000","172.21.15.16:0/3271028793":"2023-05-23T15:46:41.068549+0000","172.21.15.16:6833/36921":"2023-05-23T15:46:41.068549+0000","172.21.15.16:6832/36921":"2023-05-23T15:46:41.068549+0000","172.21.15.16:0/1944100655":"2023-05-23T15:46:41.068549+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":17,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-05-22T15:46:47.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:47.271+0000 7fb836212640 1 -- 172.21.15.16:0/3679262167 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fb81c065560 msgr2=0x7fb81c067a10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-05-22T15:46:47.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:47.271+0000 7fb836212640 1 --2- 172.21.15.16:0/3679262167 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fb81c065560 0x7fb81c067a10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:47.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:47.271+0000 7fb836212640 1 -- 172.21.15.16:0/3679262167 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb818003f30 msgr2=0x7fb830150c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:47.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:47.271+0000 7fb836212640 1 --2- 172.21.15.16:0/3679262167 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb818003f30 0x7fb830150c70 secure :-1 s=READY pgs=1936 cs=0 l=1 rev1=1 crypto rx=0x7fb8200108d0 tx=0x7fb820010d90 comp rx=0 tx=0).stop 2023-05-22T15:46:47.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:47.271+0000 7fb836212640 1 -- 172.21.15.16:0/3679262167 shutdown_connections 2023-05-22T15:46:47.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:47.271+0000 7fb836212640 1 --2- 172.21.15.16:0/3679262167 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fb81c065560 0x7fb81c067a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:47.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:47.271+0000 7fb836212640 1 --2- 172.21.15.16:0/3679262167 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb8301566f0 0x7fb830156b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:47.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:47.271+0000 7fb836212640 1 --2- 172.21.15.16:0/3679262167 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb83015c4e0 0x7fb83014e380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:47.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:47.271+0000 7fb836212640 1 --2- 172.21.15.16:0/3679262167 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb818003f30 0x7fb830150c70 unknown :-1 s=CLOSED pgs=1936 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:47.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:47.271+0000 7fb836212640 1 -- 172.21.15.16:0/3679262167 >> 172.21.15.16:0/3679262167 conn(0x7fb83006e3c0 msgr2=0x7fb830153970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:46:47.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:47.271+0000 7fb836212640 1 -- 172.21.15.16:0/3679262167 shutdown_connections 2023-05-22T15:46:47.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:47.271+0000 7fb836212640 1 -- 172.21.15.16:0/3679262167 wait complete. 2023-05-22T15:46:47.277 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:46:47.271+0000 7f5d4045c180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-05-22T15:46:47.291 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-05-22T15:26:38.019781+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-05-22T15:26:43.570514+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 18, 'pool_name': 'cephfs_metadata', 'create_time': '2023-05-22T15:46:21.084615+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '146', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 19, 'pool_name': 'cephfs_data', 'create_time': '2023-05-22T15:46:22.440577+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '147', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-05-22T15:46:47.292 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-05-22T15:46:47.355 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:46:47.347+0000 7f5d4045c180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-05-22T15:46:47.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:47.399+0000 7fcf1a1b8640 1 Processor -- start 2023-05-22T15:46:47.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:47.399+0000 7fcf1a1b8640 1 -- start start 2023-05-22T15:46:47.409 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:47.403+0000 7fcf1a1b8640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fcf1406dab0 0x7fcf1406fef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:47.409 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:47.403+0000 7fcf1a1b8640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fcf14070430 0x7fcf1415bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:47.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:47.403+0000 7fcf1a1b8640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fcf1415c120 0x7fcf14069720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:47.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:47.403+0000 7fcf1a1b8640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf1406de80 con 0x7fcf14070430 2023-05-22T15:46:47.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:47.403+0000 7fcf1a1b8640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf14070880 con 0x7fcf1415c120 2023-05-22T15:46:47.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:47.403+0000 7fcf1a1b8640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf14158eb0 con 0x7fcf1406dab0 2023-05-22T15:46:47.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:47.403+0000 7fcf12ffd640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fcf14070430 0x7fcf1415bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:47.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:47.403+0000 7fcf12ffd640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fcf14070430 0x7fcf1415bbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41584/0 (socket says 172.21.15.16:41584) 2023-05-22T15:46:47.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:47.403+0000 7fcf12ffd640 1 -- 172.21.15.16:0/2067913478 learned_addr learned my addr 172.21.15.16:0/2067913478 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:46:47.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:47.403+0000 7fcf137fe640 1 --2- 172.21.15.16:0/2067913478 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fcf1406dab0 0x7fcf1406fef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:47.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:47.403+0000 7fcf13fff640 1 --2- 172.21.15.16:0/2067913478 >> v2:172.21.15.62:3300/0 conn(0x7fcf1415c120 0x7fcf14069720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:47.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:47.403+0000 7fcf12ffd640 1 -- 172.21.15.16:0/2067913478 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fcf1406dab0 msgr2=0x7fcf1406fef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:46:47.412 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:47.403+0000 7fcf12ffd640 1 --2- 172.21.15.16:0/2067913478 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fcf1406dab0 0x7fcf1406fef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:47.412 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:47.403+0000 7fcf12ffd640 1 -- 172.21.15.16:0/2067913478 >> v2:172.21.15.62:3300/0 conn(0x7fcf1415c120 msgr2=0x7fcf14069720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:46:47.412 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:47.403+0000 7fcf12ffd640 1 --2- 172.21.15.16:0/2067913478 >> v2:172.21.15.62:3300/0 conn(0x7fcf1415c120 0x7fcf14069720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:47.412 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:47.403+0000 7fcf12ffd640 1 -- 172.21.15.16:0/2067913478 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf1415c950 con 0x7fcf14070430 2023-05-22T15:46:47.418 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:47.403+0000 7fcf12ffd640 1 --2- 172.21.15.16:0/2067913478 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcf14070430 0x7fcf1415bbe0 secure :-1 s=READY pgs=1937 cs=0 l=1 rev1=1 crypto rx=0x7fcf140cb260 tx=0x7fcf0401c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26583bafbc0cb9dc server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:47.418 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:47.403+0000 7fcf127fc640 1 -- 172.21.15.16:0/2067913478 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf040ab610 con 0x7fcf14070430 2023-05-22T15:46:47.419 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:47.403+0000 7fcf127fc640 1 -- 172.21.15.16:0/2067913478 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf040ab770 con 0x7fcf14070430 2023-05-22T15:46:47.419 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:47.403+0000 7fcf127fc640 1 -- 172.21.15.16:0/2067913478 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf040abb00 con 0x7fcf14070430 2023-05-22T15:46:47.419 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:47.407+0000 7fcf1a1b8640 1 -- 172.21.15.16:0/2067913478 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcf14070430 msgr2=0x7fcf1415bbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:47.420 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:47.407+0000 7fcf1a1b8640 1 --2- 172.21.15.16:0/2067913478 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcf14070430 0x7fcf1415bbe0 secure :-1 s=READY pgs=1937 cs=0 l=1 rev1=1 crypto rx=0x7fcf140cb260 tx=0x7fcf0401c4d0 comp rx=0 tx=0).stop 2023-05-22T15:46:47.420 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:47.407+0000 7fcf1a1b8640 1 -- 172.21.15.16:0/2067913478 shutdown_connections 2023-05-22T15:46:47.420 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:47.407+0000 7fcf1a1b8640 1 --2- 172.21.15.16:0/2067913478 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcf14070430 0x7fcf1415bbe0 unknown :-1 s=CLOSED pgs=1937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:47.420 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:47.407+0000 7fcf1a1b8640 1 --2- 172.21.15.16:0/2067913478 >> v2:172.21.15.62:3300/0 conn(0x7fcf1415c120 0x7fcf14069720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:47.421 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:47.407+0000 7fcf1a1b8640 1 --2- 172.21.15.16:0/2067913478 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fcf1406dab0 0x7fcf1406fef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:47.421 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:47.407+0000 7fcf1a1b8640 1 -- 172.21.15.16:0/2067913478 >> 172.21.15.16:0/2067913478 conn(0x7fcf140b1110 msgr2=0x7fcf140ce830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:46:47.421 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:47.407+0000 7fcf1a1b8640 1 -- 172.21.15.16:0/2067913478 shutdown_connections 2023-05-22T15:46:47.421 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:47.407+0000 7fcf1a1b8640 1 -- 172.21.15.16:0/2067913478 wait complete. 2023-05-22T15:46:47.422 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:47.407+0000 7fcf1a1b8640 1 Processor -- start 2023-05-22T15:46:47.422 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:47.407+0000 7fcf1a1b8640 1 -- start start 2023-05-22T15:46:47.422 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:47.407+0000 7fcf1a1b8640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fcf1406dab0 0x7fcf140b3f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:47.422 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:47.407+0000 7fcf1a1b8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcf14070430 0x7fcf140b1a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:47.423 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:47.407+0000 7fcf1a1b8640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcf1415c120 0x7fcf1413b050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:47.423 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:47.407+0000 7fcf1a1b8640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf140b4900 con 0x7fcf14070430 2023-05-22T15:46:47.423 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:47.407+0000 7fcf1a1b8640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf14159b10 con 0x7fcf1415c120 2023-05-22T15:46:47.424 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:47.407+0000 7fcf1a1b8640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf140b4620 con 0x7fcf1406dab0 2023-05-22T15:46:47.424 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:47.407+0000 7fcf12ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcf14070430 0x7fcf140b1a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:47.424 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:47.407+0000 7fcf12ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcf14070430 0x7fcf140b1a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41586/0 (socket says 172.21.15.16:41586) 2023-05-22T15:46:47.424 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:47.407+0000 7fcf12ffd640 1 -- 172.21.15.16:0/3229779365 learned_addr learned my addr 172.21.15.16:0/3229779365 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:46:47.425 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:47.407+0000 7fcf137fe640 1 --2- 172.21.15.16:0/3229779365 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fcf1406dab0 0x7fcf140b3f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:47.425 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:47.407+0000 7fcf13fff640 1 --2- 172.21.15.16:0/3229779365 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcf1415c120 0x7fcf1413b050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:47.425 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:47.407+0000 7fcf12ffd640 1 -- 172.21.15.16:0/3229779365 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fcf1406dab0 msgr2=0x7fcf140b3f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:47.425 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:47.407+0000 7fcf12ffd640 1 --2- 172.21.15.16:0/3229779365 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fcf1406dab0 0x7fcf140b3f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:47.425 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:47.407+0000 7fcf12ffd640 1 -- 172.21.15.16:0/3229779365 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcf1415c120 msgr2=0x7fcf1413b050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:47.425 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:47.407+0000 7fcf12ffd640 1 --2- 172.21.15.16:0/3229779365 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcf1415c120 0x7fcf1413b050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:47.425 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:47.407+0000 7fcf12ffd640 1 -- 172.21.15.16:0/3229779365 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf040094d0 con 0x7fcf14070430 2023-05-22T15:46:47.426 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:47.407+0000 7fcf12ffd640 1 --2- 172.21.15.16:0/3229779365 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcf14070430 0x7fcf140b1a60 secure :-1 s=READY pgs=1938 cs=0 l=1 rev1=1 crypto rx=0x7fcf0401ab80 tx=0x7fcf0401ad10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:47.426 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:47.407+0000 7fcf0bfff640 1 -- 172.21.15.16:0/3229779365 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf040ab610 con 0x7fcf14070430 2023-05-22T15:46:47.426 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:47.407+0000 7fcf0bfff640 1 -- 172.21.15.16:0/3229779365 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf040b4800 con 0x7fcf14070430 2023-05-22T15:46:47.426 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:47.407+0000 7fcf1a1b8640 1 -- 172.21.15.16:0/3229779365 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf1415c520 con 0x7fcf14070430 2023-05-22T15:46:47.426 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:47.407+0000 7fcf0bfff640 1 -- 172.21.15.16:0/3229779365 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf040b4bb0 con 0x7fcf14070430 2023-05-22T15:46:47.426 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:47.407+0000 7fcf1a1b8640 1 -- 172.21.15.16:0/3229779365 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf14158510 con 0x7fcf14070430 2023-05-22T15:46:47.427 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:47.411+0000 7fcf1a1b8640 1 -- 172.21.15.16:0/3229779365 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf1415c6b0 con 0x7fcf14070430 2023-05-22T15:46:47.427 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:47.411+0000 7fcf0bfff640 1 -- 172.21.15.16:0/3229779365 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 64) v1 ==== 84389+0+0 (secure 0 0 0) 0x7fcf040b4d10 con 0x7fcf14070430 2023-05-22T15:46:47.427 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:47.411+0000 7fcf0bfff640 1 --2- 172.21.15.16:0/3229779365 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fcef4065530 0x7fcef40679e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:47.427 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:47.411+0000 7fcf137fe640 1 -- 172.21.15.16:0/3229779365 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fcef4065530 msgr2=0x7fcef40679e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/36921 2023-05-22T15:46:47.427 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:47.411+0000 7fcf0bfff640 1 -- 172.21.15.16:0/3229779365 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 6754+0+0 (secure 0 0 0) 0x7fcf0411d630 con 0x7fcf14070430 2023-05-22T15:46:47.427 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:47.411+0000 7fcf137fe640 1 --2- 172.21.15.16:0/3229779365 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fcef4065530 0x7fcef40679e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-05-22T15:46:47.428 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:47.415+0000 7fcf0bfff640 1 -- 172.21.15.16:0/3229779365 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fcf040ead10 con 0x7fcf14070430 2023-05-22T15:46:47.489 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:46:47.487+0000 7fd05b983180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-05-22T15:46:47.546 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:46:47.543+0000 7fd05b983180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-05-22T15:46:47.578 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:47.571+0000 7fcf1a1b8640 1 -- 172.21.15.16:0/3229779365 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fcf14159320 con 0x7fcf14070430 2023-05-22T15:46:47.578 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:47.571+0000 7fcf0bfff640 1 -- 172.21.15.16:0/3229779365 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v148) v1 ==== 93+0+10 (secure 0 0 0) 0x7fcf14159320 con 0x7fcf14070430 2023-05-22T15:46:47.579 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2023-05-22T15:46:47.580 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:47.575+0000 7fcf1a1b8640 1 -- 172.21.15.16:0/3229779365 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fcef4065530 msgr2=0x7fcef40679e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-05-22T15:46:47.581 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:47.575+0000 7fcf1a1b8640 1 --2- 172.21.15.16:0/3229779365 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fcef4065530 0x7fcef40679e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:47.581 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:47.575+0000 7fcf1a1b8640 1 -- 172.21.15.16:0/3229779365 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcf14070430 msgr2=0x7fcf140b1a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:47.581 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:47.575+0000 7fcf1a1b8640 1 --2- 172.21.15.16:0/3229779365 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcf14070430 0x7fcf140b1a60 secure :-1 s=READY pgs=1938 cs=0 l=1 rev1=1 crypto rx=0x7fcf0401ab80 tx=0x7fcf0401ad10 comp rx=0 tx=0).stop 2023-05-22T15:46:47.581 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:47.575+0000 7fcf1a1b8640 1 -- 172.21.15.16:0/3229779365 shutdown_connections 2023-05-22T15:46:47.582 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:47.575+0000 7fcf1a1b8640 1 --2- 172.21.15.16:0/3229779365 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcf1415c120 0x7fcf1413b050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:47.582 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:47.575+0000 7fcf1a1b8640 1 --2- 172.21.15.16:0/3229779365 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcf14070430 0x7fcf140b1a60 unknown :-1 s=CLOSED pgs=1938 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:47.582 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:47.575+0000 7fcf1a1b8640 1 --2- 172.21.15.16:0/3229779365 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fcef4065530 0x7fcef40679e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:47.582 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:47.575+0000 7fcf1a1b8640 1 --2- 172.21.15.16:0/3229779365 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fcf1406dab0 0x7fcf140b3f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:47.582 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:47.575+0000 7fcf1a1b8640 1 -- 172.21.15.16:0/3229779365 >> 172.21.15.16:0/3229779365 conn(0x7fcf140b1110 msgr2=0x7fcf1413eef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:46:47.582 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:47.575+0000 7fcf1a1b8640 1 -- 172.21.15.16:0/3229779365 shutdown_connections 2023-05-22T15:46:47.582 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:47.575+0000 7fcf1a1b8640 1 -- 172.21.15.16:0/3229779365 wait complete. 2023-05-22T15:46:47.597 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-05-22T15:46:47.615 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:46:47.607+0000 7f5d4045c180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-05-22T15:46:47.668 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:46:47.663+0000 7f5d4045c180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-05-22T15:46:47.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:47.711+0000 7fcec15bb640 1 Processor -- start 2023-05-22T15:46:47.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:47.711+0000 7fcec15bb640 1 -- start start 2023-05-22T15:46:47.720 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:47.711+0000 7fcec15bb640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fcebc15c8b0 0x7fcebc0cee30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:47.721 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:47.711+0000 7fcec15bb640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fcebc06db80 0x7fcebc06dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:47.721 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:47.711+0000 7fcec15bb640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcebc06e520 con 0x7fcebc06bf60 2023-05-22T15:46:47.721 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:47.711+0000 7fcec15bb640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcebc15a0c0 con 0x7fcebc15c8b0 2023-05-22T15:46:47.722 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:47.711+0000 7fcec15bb640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcebc0ce160 con 0x7fcebc06db80 2023-05-22T15:46:47.722 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:47.711+0000 7fceb3fff640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fcebc06bf60 0x7fcebc15cc80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:57460/0 (socket says 172.21.15.16:57460) 2023-05-22T15:46:47.722 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:47.711+0000 7fceb3fff640 1 -- 172.21.15.16:0/66833801 learned_addr learned my addr 172.21.15.16:0/66833801 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:46:47.722 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:47.711+0000 7fcebb7fe640 1 --2- 172.21.15.16:0/66833801 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fcebc06db80 0x7fcebc06dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:47.723 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:47.711+0000 7fcebaffd640 1 --2- 172.21.15.16:0/66833801 >> v2:172.21.15.62:3300/0 conn(0x7fcebc15c8b0 0x7fcebc0cee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:47.723 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:47.711+0000 7fceba7fc640 1 -- 172.21.15.16:0/66833801 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 838337386 0 0) 0x7fcebc06e520 con 0x7fcebc06bf60 2023-05-22T15:46:47.724 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:47.711+0000 7fceba7fc640 1 -- 172.21.15.16:0/66833801 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcea8003530 con 0x7fcebc06bf60 2023-05-22T15:46:47.724 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:47.711+0000 7fceba7fc640 1 -- 172.21.15.16:0/66833801 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3162638890 0 0) 0x7fcea8003530 con 0x7fcebc06bf60 2023-05-22T15:46:47.724 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:47.711+0000 7fceba7fc640 1 -- 172.21.15.16:0/66833801 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fcebc06db80 msgr2=0x7fcebc06dfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:46:47.725 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:47.711+0000 7fceba7fc640 1 --2- 172.21.15.16:0/66833801 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fcebc06db80 0x7fcebc06dfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:47.725 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:47.711+0000 7fceba7fc640 1 -- 172.21.15.16:0/66833801 >> v2:172.21.15.62:3300/0 conn(0x7fcebc15c8b0 msgr2=0x7fcebc0cee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:46:47.725 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:47.711+0000 7fceba7fc640 1 --2- 172.21.15.16:0/66833801 >> v2:172.21.15.62:3300/0 conn(0x7fcebc15c8b0 0x7fcebc0cee30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:47.725 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:47.711+0000 7fceba7fc640 1 -- 172.21.15.16:0/66833801 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcebc06c3e0 con 0x7fcebc06bf60 2023-05-22T15:46:47.725 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:47.711+0000 7fceba7fc640 1 -- 172.21.15.16:0/66833801 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7fcea4001680 con 0x7fcebc06bf60 2023-05-22T15:46:47.726 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:47.711+0000 7fceba7fc640 1 -- 172.21.15.16:0/66833801 >> v1:172.21.15.16:6789/0 conn(0x7fcebc06bf60 legacy=0x7fcebc15cc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:47.726 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:47.715+0000 7fceba7fc640 1 --2- 172.21.15.16:0/66833801 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcea8003980 0x7fcea8003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:47.726 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:47.715+0000 7fceba7fc640 1 --2- 172.21.15.16:0/66833801 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fcebc06db80 0x7fcea80047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:47.726 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:47.715+0000 7fceba7fc640 1 --2- 172.21.15.16:0/66833801 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcea8004d10 0x7fcea8005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:47.726 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:47.715+0000 7fceba7fc640 1 -- 172.21.15.16:0/66833801 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcebc15a0c0 con 0x7fcea8004d10 2023-05-22T15:46:47.726 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:47.715+0000 7fceba7fc640 1 -- 172.21.15.16:0/66833801 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcebc0ce160 con 0x7fcea8003980 2023-05-22T15:46:47.726 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:47.715+0000 7fceba7fc640 1 -- 172.21.15.16:0/66833801 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcea800b990 con 0x7fcebc06db80 2023-05-22T15:46:47.727 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:47.715+0000 7fcebb7fe640 1 --2- 172.21.15.16:0/66833801 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcea8004d10 0x7fcea8005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:47.727 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:47.715+0000 7fceb3fff640 1 --2- 172.21.15.16:0/66833801 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fcebc06db80 0x7fcea80047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:47.727 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:47.715+0000 7fcebaffd640 1 --2- 172.21.15.16:0/66833801 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcea8003980 0x7fcea8003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:47.727 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:47.715+0000 7fcebb7fe640 1 -- 172.21.15.16:0/66833801 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fcebc06db80 msgr2=0x7fcea80047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:46:47.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:47.715+0000 7fcebb7fe640 1 --2- 172.21.15.16:0/66833801 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fcebc06db80 0x7fcea80047d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:47.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:47.715+0000 7fcebb7fe640 1 -- 172.21.15.16:0/66833801 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcea8003980 msgr2=0x7fcea8003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:46:47.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:47.715+0000 7fcebb7fe640 1 --2- 172.21.15.16:0/66833801 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcea8003980 0x7fcea8003d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:47.729 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:47.715+0000 7fcebb7fe640 1 -- 172.21.15.16:0/66833801 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcea8003530 con 0x7fcea8004d10 2023-05-22T15:46:47.734 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:47.715+0000 7fcebb7fe640 1 --2- 172.21.15.16:0/66833801 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcea8004d10 0x7fcea8005180 secure :-1 s=READY pgs=1943 cs=0 l=1 rev1=1 crypto rx=0x7fceb4009fc0 tx=0x7fceb401cc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=902cca72188d3a62 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:47.734 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:47.715+0000 7fceba7fc640 1 -- 172.21.15.16:0/66833801 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fceb40a5c70 con 0x7fcea8004d10 2023-05-22T15:46:47.734 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:47.715+0000 7fceba7fc640 1 -- 172.21.15.16:0/66833801 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fceb400f040 con 0x7fcea8004d10 2023-05-22T15:46:47.734 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:47.715+0000 7fceba7fc640 1 -- 172.21.15.16:0/66833801 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fceb40ad440 con 0x7fcea8004d10 2023-05-22T15:46:47.734 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:47.715+0000 7fcec15bb640 1 -- 172.21.15.16:0/66833801 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcea8004d10 msgr2=0x7fcea8005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:47.735 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:47.715+0000 7fcec15bb640 1 --2- 172.21.15.16:0/66833801 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcea8004d10 0x7fcea8005180 secure :-1 s=READY pgs=1943 cs=0 l=1 rev1=1 crypto rx=0x7fceb4009fc0 tx=0x7fceb401cc00 comp rx=0 tx=0).stop 2023-05-22T15:46:47.735 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:47.715+0000 7fcebb7fe640 1 -- 172.21.15.16:0/66833801 reap_dead start 2023-05-22T15:46:47.735 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:47.715+0000 7fcec15bb640 1 -- 172.21.15.16:0/66833801 shutdown_connections 2023-05-22T15:46:47.735 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:47.715+0000 7fcec15bb640 1 -- 172.21.15.16:0/66833801 >> 172.21.15.16:0/66833801 conn(0x7fcebc06f5a0 msgr2=0x7fcebc06f9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:46:47.735 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:47.715+0000 7fcec15bb640 1 -- 172.21.15.16:0/66833801 shutdown_connections 2023-05-22T15:46:47.735 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:47.715+0000 7fcec15bb640 1 -- 172.21.15.16:0/66833801 wait complete. 2023-05-22T15:46:47.735 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:47.719+0000 7fcec15bb640 1 Processor -- start 2023-05-22T15:46:47.735 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:47.719+0000 7fcec15bb640 1 -- start start 2023-05-22T15:46:47.736 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:47.719+0000 7fcec15bb640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcea8004d10 0x7fcebc0b5b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:47.736 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:47.719+0000 7fcec15bb640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fcebc1549c0 0x7fcebc0b4ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:47.736 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:47.719+0000 7fcec15bb640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcebc0b50e0 0x7fcebc15f9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:47.736 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:47.719+0000 7fcec15bb640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcebc15e5e0 con 0x7fcea8004d10 2023-05-22T15:46:47.736 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:47.719+0000 7fcec15bb640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcebc0b6090 con 0x7fcebc0b50e0 2023-05-22T15:46:47.736 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:47.719+0000 7fcec15bb640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcebc0b5550 con 0x7fcebc1549c0 2023-05-22T15:46:47.736 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:47.719+0000 7fcebaffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcea8004d10 0x7fcebc0b5b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:47.737 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:47.719+0000 7fcebaffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcea8004d10 0x7fcebc0b5b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41642/0 (socket says 172.21.15.16:41642) 2023-05-22T15:46:47.737 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:47.719+0000 7fcebaffd640 1 -- 172.21.15.16:0/2586626459 learned_addr learned my addr 172.21.15.16:0/2586626459 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:46:47.737 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:47.719+0000 7fceb3fff640 1 --2- 172.21.15.16:0/2586626459 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fcebc1549c0 0x7fcebc0b4ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:47.737 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:47.719+0000 7fcebb7fe640 1 --2- 172.21.15.16:0/2586626459 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcebc0b50e0 0x7fcebc15f9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:47.738 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:47.719+0000 7fcebaffd640 1 -- 172.21.15.16:0/2586626459 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fcebc1549c0 msgr2=0x7fcebc0b4ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:47.738 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:47.719+0000 7fcebaffd640 1 --2- 172.21.15.16:0/2586626459 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fcebc1549c0 0x7fcebc0b4ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:47.738 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:47.719+0000 7fcebaffd640 1 -- 172.21.15.16:0/2586626459 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcebc0b50e0 msgr2=0x7fcebc15f9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:47.738 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:47.719+0000 7fcebaffd640 1 --2- 172.21.15.16:0/2586626459 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcebc0b50e0 0x7fcebc15f9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:47.738 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:47.719+0000 7fcebaffd640 1 -- 172.21.15.16:0/2586626459 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcebc0ce370 con 0x7fcea8004d10 2023-05-22T15:46:47.738 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:47.719+0000 7fcebaffd640 1 --2- 172.21.15.16:0/2586626459 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcea8004d10 0x7fcebc0b5b50 secure :-1 s=READY pgs=1944 cs=0 l=1 rev1=1 crypto rx=0x7fceac0108f0 tx=0x7fceac010db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:47.738 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:47.719+0000 7fceb37fe640 1 -- 172.21.15.16:0/2586626459 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fceac00eb00 con 0x7fcea8004d10 2023-05-22T15:46:47.738 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:47.719+0000 7fceb37fe640 1 -- 172.21.15.16:0/2586626459 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fceac00ec60 con 0x7fcea8004d10 2023-05-22T15:46:47.739 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:47.719+0000 7fcec15bb640 1 -- 172.21.15.16:0/2586626459 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcebc06c640 con 0x7fcea8004d10 2023-05-22T15:46:47.739 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:47.719+0000 7fceb37fe640 1 -- 172.21.15.16:0/2586626459 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fceac01c400 con 0x7fcea8004d10 2023-05-22T15:46:47.739 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:47.719+0000 7fcec15bb640 1 -- 172.21.15.16:0/2586626459 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcebc0ce650 con 0x7fcea8004d10 2023-05-22T15:46:47.739 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:47.719+0000 7fcec15bb640 1 -- 172.21.15.16:0/2586626459 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcebc04dbe0 con 0x7fcea8004d10 2023-05-22T15:46:47.739 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:47.719+0000 7fceb37fe640 1 -- 172.21.15.16:0/2586626459 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 65) v1 ==== 83941+0+0 (secure 0 0 0) 0x7fceac01c560 con 0x7fcea8004d10 2023-05-22T15:46:47.739 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:47.723+0000 7fceb37fe640 1 -- 172.21.15.16:0/2586626459 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 7055+0+0 (secure 0 0 0) 0x7fceac082b10 con 0x7fcea8004d10 2023-05-22T15:46:47.739 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:47.723+0000 7fceb37fe640 1 -- 172.21.15.16:0/2586626459 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fceac050680 con 0x7fcea8004d10 2023-05-22T15:46:47.776 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:46:47.771+0000 7f5cc95e7640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T15:46:47.776 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:46:47.771+0000 7f5cc95e7640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T15:46:47.777 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:46:47.771+0000 7f5cc95e7640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T15:46:47.777 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:46:47.771+0000 7f5cc95e7640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T15:46:47.777 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:46:47.771+0000 7f5cc95e7640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T15:46:47.777 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:46:47.771+0000 7f5cccdee640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T15:46:47.778 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:46:47.771+0000 7f5cccdee640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T15:46:47.778 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:46:47.771+0000 7f5cccdee640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T15:46:47.778 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:46:47.771+0000 7f5cccdee640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T15:46:47.778 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:46:47.771+0000 7f5cccdee640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T15:46:47.898 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:47.891+0000 7fcec15bb640 1 -- 172.21.15.16:0/2586626459 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fcebc0b61e0 con 0x7fcea8004d10 2023-05-22T15:46:47.899 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:47.895+0000 7fceb37fe640 1 -- 172.21.15.16:0/2586626459 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v149) v1 ==== 92+0+10 (secure 0 0 0) 0x7fceac053e70 con 0x7fcea8004d10 2023-05-22T15:46:47.900 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 8 2023-05-22T15:46:47.903 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:47.895+0000 7fcec15bb640 1 -- 172.21.15.16:0/2586626459 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcea8004d10 msgr2=0x7fcebc0b5b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:47.903 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:47.895+0000 7fcec15bb640 1 --2- 172.21.15.16:0/2586626459 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcea8004d10 0x7fcebc0b5b50 secure :-1 s=READY pgs=1944 cs=0 l=1 rev1=1 crypto rx=0x7fceac0108f0 tx=0x7fceac010db0 comp rx=0 tx=0).stop 2023-05-22T15:46:47.903 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:47.895+0000 7fcec15bb640 1 -- 172.21.15.16:0/2586626459 shutdown_connections 2023-05-22T15:46:47.903 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:47.895+0000 7fcec15bb640 1 --2- 172.21.15.16:0/2586626459 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcebc0b50e0 0x7fcebc15f9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:47.903 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:47.895+0000 7fcec15bb640 1 --2- 172.21.15.16:0/2586626459 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fcebc1549c0 0x7fcebc0b4ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:47.903 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:47.895+0000 7fcec15bb640 1 --2- 172.21.15.16:0/2586626459 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcea8004d10 0x7fcebc0b5b50 unknown :-1 s=CLOSED pgs=1944 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:47.903 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:47.895+0000 7fcec15bb640 1 -- 172.21.15.16:0/2586626459 >> 172.21.15.16:0/2586626459 conn(0x7fcebc06f5a0 msgr2=0x7fcebc0b3090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:46:47.904 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:47.895+0000 7fcec15bb640 1 -- 172.21.15.16:0/2586626459 shutdown_connections 2023-05-22T15:46:47.904 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:47.895+0000 7fcec15bb640 1 -- 172.21.15.16:0/2586626459 wait complete. 2023-05-22T15:46:47.919 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-05-22T15:46:48.091 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.083+0000 7fddf8d52640 1 Processor -- start 2023-05-22T15:46:48.091 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.087+0000 7fddf8d52640 1 -- start start 2023-05-22T15:46:48.092 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.087+0000 7fddf8d52640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fddf4159970 0x7fddf4159d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:48.092 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.087+0000 7fddf8d52640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fddf415d680 0x7fddf40cb260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:48.092 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.087+0000 7fddf8d52640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fddf415dad0 0x7fddf406b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:48.092 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.087+0000 7fddf8d52640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fddf4070c30 con 0x7fddf415d680 2023-05-22T15:46:48.092 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.087+0000 7fddf8d52640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fddf4070e60 con 0x7fddf415dad0 2023-05-22T15:46:48.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.087+0000 7fddf8d52640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fddf4071140 con 0x7fddf4159970 2023-05-22T15:46:48.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.087+0000 7fddf1d9b640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fddf415d680 0x7fddf40cb260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:48.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.087+0000 7fddf1d9b640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fddf415d680 0x7fddf40cb260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41748/0 (socket says 172.21.15.16:41748) 2023-05-22T15:46:48.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.087+0000 7fddf1d9b640 1 -- 172.21.15.16:0/178498005 learned_addr learned my addr 172.21.15.16:0/178498005 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:46:48.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.087+0000 7fddf259c640 1 --2- 172.21.15.16:0/178498005 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fddf4159970 0x7fddf4159d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:48.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.087+0000 7fddf2d9d640 1 --2- 172.21.15.16:0/178498005 >> v2:172.21.15.62:3300/0 conn(0x7fddf415dad0 0x7fddf406b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:48.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.087+0000 7fddf1d9b640 1 -- 172.21.15.16:0/178498005 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fddf4159970 msgr2=0x7fddf4159d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:46:48.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.087+0000 7fddf1d9b640 1 --2- 172.21.15.16:0/178498005 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fddf4159970 0x7fddf4159d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:48.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.087+0000 7fddf1d9b640 1 -- 172.21.15.16:0/178498005 >> v2:172.21.15.62:3300/0 conn(0x7fddf415dad0 msgr2=0x7fddf406b6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:46:48.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.087+0000 7fddf1d9b640 1 --2- 172.21.15.16:0/178498005 >> v2:172.21.15.62:3300/0 conn(0x7fddf415dad0 0x7fddf406b6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:48.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.087+0000 7fddf1d9b640 1 -- 172.21.15.16:0/178498005 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddf4071420 con 0x7fddf415d680 2023-05-22T15:46:48.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.087+0000 7fddf259c640 1 --2- 172.21.15.16:0/178498005 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fddf4159970 0x7fddf4159d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:46:48.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.087+0000 7fddf2d9d640 1 --2- 172.21.15.16:0/178498005 >> v2:172.21.15.62:3300/0 conn(0x7fddf415dad0 0x7fddf406b6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:46:48.096 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.087+0000 7fddf1d9b640 1 --2- 172.21.15.16:0/178498005 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fddf415d680 0x7fddf40cb260 secure :-1 s=READY pgs=1954 cs=0 l=1 rev1=1 crypto rx=0x7fddf40b40e0 tx=0x7fdde801c440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b05009cf6c2ac686 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:48.096 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.087+0000 7fddf159a640 1 -- 172.21.15.16:0/178498005 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdde80ac640 con 0x7fddf415d680 2023-05-22T15:46:48.096 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.091+0000 7fddf159a640 1 -- 172.21.15.16:0/178498005 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdde80ac7a0 con 0x7fddf415d680 2023-05-22T15:46:48.096 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.091+0000 7fddf159a640 1 -- 172.21.15.16:0/178498005 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdde80acb30 con 0x7fddf415d680 2023-05-22T15:46:48.096 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.091+0000 7fddf8d52640 1 -- 172.21.15.16:0/178498005 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fddf415d680 msgr2=0x7fddf40cb260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:48.097 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.091+0000 7fddf8d52640 1 --2- 172.21.15.16:0/178498005 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fddf415d680 0x7fddf40cb260 secure :-1 s=READY pgs=1954 cs=0 l=1 rev1=1 crypto rx=0x7fddf40b40e0 tx=0x7fdde801c440 comp rx=0 tx=0).stop 2023-05-22T15:46:48.097 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.091+0000 7fddf8d52640 1 -- 172.21.15.16:0/178498005 shutdown_connections 2023-05-22T15:46:48.097 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.091+0000 7fddf8d52640 1 --2- 172.21.15.16:0/178498005 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fddf415d680 0x7fddf40cb260 unknown :-1 s=CLOSED pgs=1954 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:48.097 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.091+0000 7fddf8d52640 1 --2- 172.21.15.16:0/178498005 >> v2:172.21.15.62:3300/0 conn(0x7fddf415dad0 0x7fddf406b6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:48.097 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.091+0000 7fddf8d52640 1 --2- 172.21.15.16:0/178498005 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fddf4159970 0x7fddf4159d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:48.097 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.091+0000 7fddf8d52640 1 -- 172.21.15.16:0/178498005 >> 172.21.15.16:0/178498005 conn(0x7fddf4069720 msgr2=0x7fddf4069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:46:48.097 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.091+0000 7fddf8d52640 1 -- 172.21.15.16:0/178498005 shutdown_connections 2023-05-22T15:46:48.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.091+0000 7fddf8d52640 1 -- 172.21.15.16:0/178498005 wait complete. 2023-05-22T15:46:48.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.091+0000 7fddf8d52640 1 Processor -- start 2023-05-22T15:46:48.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.091+0000 7fddf8d52640 1 -- start start 2023-05-22T15:46:48.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.091+0000 7fddf8d52640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fddf4159970 0x7fddf414cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:48.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.091+0000 7fddf8d52640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fddf415d680 0x7fddf40c1b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:48.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.091+0000 7fddf8d52640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fddf415dad0 0x7fddf40b3370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:48.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.091+0000 7fddf8d52640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddf40b3cc0 con 0x7fddf4159970 2023-05-22T15:46:48.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.091+0000 7fddf8d52640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddf40cc670 con 0x7fddf415d680 2023-05-22T15:46:48.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.091+0000 7fddf8d52640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fddf40cbd80 con 0x7fddf415dad0 2023-05-22T15:46:48.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.091+0000 7fddf259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fddf4159970 0x7fddf414cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:48.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.091+0000 7fddf259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fddf4159970 0x7fddf414cd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41758/0 (socket says 172.21.15.16:41758) 2023-05-22T15:46:48.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.091+0000 7fddf259c640 1 -- 172.21.15.16:0/1624198726 learned_addr learned my addr 172.21.15.16:0/1624198726 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:46:48.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.091+0000 7fddf2d9d640 1 --2- 172.21.15.16:0/1624198726 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fddf415dad0 0x7fddf40b3370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:48.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.091+0000 7fddf1d9b640 1 --2- 172.21.15.16:0/1624198726 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fddf415d680 0x7fddf40c1b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:48.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.091+0000 7fddf259c640 1 -- 172.21.15.16:0/1624198726 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fddf415dad0 msgr2=0x7fddf40b3370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:48.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.091+0000 7fddf259c640 1 --2- 172.21.15.16:0/1624198726 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fddf415dad0 0x7fddf40b3370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:48.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.091+0000 7fddf259c640 1 -- 172.21.15.16:0/1624198726 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fddf415d680 msgr2=0x7fddf40c1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:48.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.091+0000 7fddf259c640 1 --2- 172.21.15.16:0/1624198726 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fddf415d680 0x7fddf40c1b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:48.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.091+0000 7fddf259c640 1 -- 172.21.15.16:0/1624198726 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdde80094d0 con 0x7fddf4159970 2023-05-22T15:46:48.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.091+0000 7fddf259c640 1 --2- 172.21.15.16:0/1624198726 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fddf4159970 0x7fddf414cd60 secure :-1 s=READY pgs=1955 cs=0 l=1 rev1=1 crypto rx=0x7fdde400c7f0 tx=0x7fdde400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:48.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.091+0000 7fdde2ffd640 1 -- 172.21.15.16:0/1624198726 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdde400cea0 con 0x7fddf4159970 2023-05-22T15:46:48.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.091+0000 7fdde2ffd640 1 -- 172.21.15.16:0/1624198726 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdde4007a30 con 0x7fddf4159970 2023-05-22T15:46:48.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.091+0000 7fddf8d52640 1 -- 172.21.15.16:0/1624198726 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fddf40cae20 con 0x7fddf4159970 2023-05-22T15:46:48.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.091+0000 7fdde2ffd640 1 -- 172.21.15.16:0/1624198726 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdde4007dc0 con 0x7fddf4159970 2023-05-22T15:46:48.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.091+0000 7fddf8d52640 1 -- 172.21.15.16:0/1624198726 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fddf40cac90 con 0x7fddf4159970 2023-05-22T15:46:48.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.095+0000 7fddf8d52640 1 -- 172.21.15.16:0/1624198726 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fddf415adc0 con 0x7fddf4159970 2023-05-22T15:46:48.103 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.095+0000 7fdde2ffd640 1 -- 172.21.15.16:0/1624198726 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 65) v1 ==== 83941+0+0 (secure 0 0 0) 0x7fdde40113f0 con 0x7fddf4159970 2023-05-22T15:46:48.103 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.095+0000 7fdde2ffd640 1 -- 172.21.15.16:0/1624198726 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 7055+0+0 (secure 0 0 0) 0x7fdde4082860 con 0x7fddf4159970 2023-05-22T15:46:48.104 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.099+0000 7fdde2ffd640 1 -- 172.21.15.16:0/1624198726 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fdde4050460 con 0x7fddf4159970 2023-05-22T15:46:48.257 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.251+0000 7fddf8d52640 1 -- 172.21.15.16:0/1624198726 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fddf404dbe0 con 0x7fddf4159970 2023-05-22T15:46:48.258 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.251+0000 7fdde2ffd640 1 -- 172.21.15.16:0/1624198726 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v149) v1 ==== 104+0+11 (secure 0 0 0) 0x7fdde401f020 con 0x7fddf4159970 2023-05-22T15:46:48.259 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2023-05-22T15:46:48.261 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.255+0000 7fddf8d52640 1 -- 172.21.15.16:0/1624198726 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fddf4159970 msgr2=0x7fddf414cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:48.261 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.255+0000 7fddf8d52640 1 --2- 172.21.15.16:0/1624198726 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fddf4159970 0x7fddf414cd60 secure :-1 s=READY pgs=1955 cs=0 l=1 rev1=1 crypto rx=0x7fdde400c7f0 tx=0x7fdde400ccb0 comp rx=0 tx=0).stop 2023-05-22T15:46:48.261 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.255+0000 7fddf8d52640 1 -- 172.21.15.16:0/1624198726 shutdown_connections 2023-05-22T15:46:48.261 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.255+0000 7fddf8d52640 1 --2- 172.21.15.16:0/1624198726 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fddf415dad0 0x7fddf40b3370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:48.262 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.255+0000 7fddf8d52640 1 --2- 172.21.15.16:0/1624198726 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fddf415d680 0x7fddf40c1b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:48.262 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.255+0000 7fddf8d52640 1 --2- 172.21.15.16:0/1624198726 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fddf4159970 0x7fddf414cd60 unknown :-1 s=CLOSED pgs=1955 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:48.262 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.255+0000 7fddf8d52640 1 -- 172.21.15.16:0/1624198726 >> 172.21.15.16:0/1624198726 conn(0x7fddf4069720 msgr2=0x7fddf4157230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:46:48.262 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.255+0000 7fddf8d52640 1 -- 172.21.15.16:0/1624198726 shutdown_connections 2023-05-22T15:46:48.262 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.255+0000 7fddf8d52640 1 -- 172.21.15.16:0/1624198726 wait complete. 2023-05-22T15:46:48.277 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-05-22T15:46:48.404 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.399+0000 7f8d5c0e4640 1 Processor -- start 2023-05-22T15:46:48.404 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.399+0000 7f8d5c0e4640 1 -- start start 2023-05-22T15:46:48.405 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.399+0000 7f8d5c0e4640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f8d54159970 0x7f8d54159d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:48.405 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.399+0000 7f8d5c0e4640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8d5415db00 0x7f8d5406b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:48.405 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.399+0000 7f8d5c0e4640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d54070c30 con 0x7f8d5415d680 2023-05-22T15:46:48.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.399+0000 7f8d5c0e4640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d540cb970 con 0x7f8d54159970 2023-05-22T15:46:48.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.399+0000 7f8d5c0e4640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d54070e60 con 0x7f8d5415db00 2023-05-22T15:46:48.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.399+0000 7f8d59e80640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f8d54159970 0x7f8d54159d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:48.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.399+0000 7f8d59e80640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f8d54159970 0x7f8d54159d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41772/0 (socket says 172.21.15.16:41772) 2023-05-22T15:46:48.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.399+0000 7f8d59e80640 1 -- 172.21.15.16:0/2106417408 learned_addr learned my addr 172.21.15.16:0/2106417408 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:46:48.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.399+0000 7f8d5967f640 1 --1- 172.21.15.16:0/2106417408 >> v1:172.21.15.16:6789/0 conn(0x7f8d5415d680 0x7f8d540cb260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:57464/0 (socket says 172.21.15.16:57464) 2023-05-22T15:46:48.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.399+0000 7f8d5a681640 1 --2- 172.21.15.16:0/2106417408 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8d5415db00 0x7f8d5406b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:48.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.399+0000 7f8d59e80640 1 -- 172.21.15.16:0/2106417408 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8d5415db00 msgr2=0x7f8d5406b6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:46:48.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.399+0000 7f8d59e80640 1 --2- 172.21.15.16:0/2106417408 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8d5415db00 0x7f8d5406b6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:48.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.399+0000 7f8d59e80640 1 -- 172.21.15.16:0/2106417408 >> v1:172.21.15.16:6789/0 conn(0x7f8d5415d680 legacy=0x7f8d540cb260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:48.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.399+0000 7f8d59e80640 1 -- 172.21.15.16:0/2106417408 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d54071140 con 0x7f8d54159970 2023-05-22T15:46:48.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.399+0000 7f8d5a681640 1 --2- 172.21.15.16:0/2106417408 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8d5415db00 0x7f8d5406b6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T15:46:48.408 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.403+0000 7f8d59e80640 1 --2- 172.21.15.16:0/2106417408 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8d54159970 0x7f8d54159d40 secure :-1 s=READY pgs=1957 cs=0 l=1 rev1=1 crypto rx=0x7f8d50009870 tx=0x7f8d5001c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf12e1a684f0d286 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:48.408 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.403+0000 7f8d58e7e640 1 -- 172.21.15.16:0/2106417408 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d500ad640 con 0x7f8d54159970 2023-05-22T15:46:48.409 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.403+0000 7f8d58e7e640 1 -- 172.21.15.16:0/2106417408 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d500ad7a0 con 0x7f8d54159970 2023-05-22T15:46:48.409 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.403+0000 7f8d58e7e640 1 -- 172.21.15.16:0/2106417408 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d500adb30 con 0x7f8d54159970 2023-05-22T15:46:48.409 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.403+0000 7f8d5c0e4640 1 -- 172.21.15.16:0/2106417408 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8d54159970 msgr2=0x7f8d54159d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:48.409 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.403+0000 7f8d5c0e4640 1 --2- 172.21.15.16:0/2106417408 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8d54159970 0x7f8d54159d40 secure :-1 s=READY pgs=1957 cs=0 l=1 rev1=1 crypto rx=0x7f8d50009870 tx=0x7f8d5001c480 comp rx=0 tx=0).stop 2023-05-22T15:46:48.409 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.403+0000 7f8d5c0e4640 1 -- 172.21.15.16:0/2106417408 shutdown_connections 2023-05-22T15:46:48.409 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.403+0000 7f8d5c0e4640 1 --2- 172.21.15.16:0/2106417408 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8d5415db00 0x7f8d5406b6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:48.409 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.403+0000 7f8d5c0e4640 1 --2- 172.21.15.16:0/2106417408 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8d54159970 0x7f8d54159d40 unknown :-1 s=CLOSED pgs=1957 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:48.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.403+0000 7f8d5c0e4640 1 -- 172.21.15.16:0/2106417408 >> 172.21.15.16:0/2106417408 conn(0x7f8d54069720 msgr2=0x7f8d54069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:46:48.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.403+0000 7f8d5c0e4640 1 -- 172.21.15.16:0/2106417408 shutdown_connections 2023-05-22T15:46:48.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.403+0000 7f8d5c0e4640 1 -- 172.21.15.16:0/2106417408 wait complete. 2023-05-22T15:46:48.413 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.403+0000 7f8d5c0e4640 1 Processor -- start 2023-05-22T15:46:48.414 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.403+0000 7f8d5c0e4640 1 -- start start 2023-05-22T15:46:48.414 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.403+0000 7f8d5c0e4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8d54159970 0x7f8d5414c170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:48.414 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.403+0000 7f8d5c0e4640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8d5415d680 0x7f8d54158b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:48.415 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.403+0000 7f8d5c0e4640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8d5415db00 0x7f8d54151810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:48.415 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.403+0000 7f8d5c0e4640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d540b3a90 con 0x7f8d54159970 2023-05-22T15:46:48.415 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.403+0000 7f8d5c0e4640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d540b1b70 con 0x7f8d5415d680 2023-05-22T15:46:48.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.403+0000 7f8d5c0e4640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d540b09c0 con 0x7f8d5415db00 2023-05-22T15:46:48.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.403+0000 7f8d59e80640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8d54159970 0x7f8d5414c170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:48.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.403+0000 7f8d59e80640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8d54159970 0x7f8d5414c170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41780/0 (socket says 172.21.15.16:41780) 2023-05-22T15:46:48.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.403+0000 7f8d59e80640 1 -- 172.21.15.16:0/1964919471 learned_addr learned my addr 172.21.15.16:0/1964919471 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:46:48.417 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.403+0000 7f8d5a681640 1 --2- 172.21.15.16:0/1964919471 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8d5415db00 0x7f8d54151810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:48.417 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.407+0000 7f8d5967f640 1 --2- 172.21.15.16:0/1964919471 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8d5415d680 0x7f8d54158b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:48.417 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.407+0000 7f8d59e80640 1 -- 172.21.15.16:0/1964919471 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8d5415db00 msgr2=0x7f8d54151810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:48.417 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.407+0000 7f8d59e80640 1 --2- 172.21.15.16:0/1964919471 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8d5415db00 0x7f8d54151810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:48.417 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.407+0000 7f8d59e80640 1 -- 172.21.15.16:0/1964919471 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8d5415d680 msgr2=0x7f8d54158b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:48.417 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.407+0000 7f8d59e80640 1 --2- 172.21.15.16:0/1964919471 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8d5415d680 0x7f8d54158b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:48.418 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.407+0000 7f8d59e80640 1 -- 172.21.15.16:0/1964919471 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d500094d0 con 0x7f8d54159970 2023-05-22T15:46:48.418 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.407+0000 7f8d5967f640 1 --2- 172.21.15.16:0/1964919471 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8d5415d680 0x7f8d54158b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:46:48.418 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.407+0000 7f8d59e80640 1 --2- 172.21.15.16:0/1964919471 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8d54159970 0x7f8d5414c170 secure :-1 s=READY pgs=1958 cs=0 l=1 rev1=1 crypto rx=0x7f8d500b0040 tx=0x7f8d50019220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:48.418 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.407+0000 7f8d4a7fc640 1 -- 172.21.15.16:0/1964919471 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d500ad640 con 0x7f8d54159970 2023-05-22T15:46:48.418 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.407+0000 7f8d4a7fc640 1 -- 172.21.15.16:0/1964919471 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d500a6040 con 0x7f8d54159970 2023-05-22T15:46:48.418 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.407+0000 7f8d5c0e4640 1 -- 172.21.15.16:0/1964919471 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d540cac90 con 0x7f8d54159970 2023-05-22T15:46:48.419 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.407+0000 7f8d4a7fc640 1 -- 172.21.15.16:0/1964919471 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d500b7a30 con 0x7f8d54159970 2023-05-22T15:46:48.419 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.407+0000 7f8d5c0e4640 1 -- 172.21.15.16:0/1964919471 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d540cdbf0 con 0x7f8d54159970 2023-05-22T15:46:48.419 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.407+0000 7f8d5c0e4640 1 -- 172.21.15.16:0/1964919471 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d540cae20 con 0x7f8d54159970 2023-05-22T15:46:48.419 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.407+0000 7f8d4a7fc640 1 -- 172.21.15.16:0/1964919471 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 65) v1 ==== 83941+0+0 (secure 0 0 0) 0x7f8d500b7b90 con 0x7f8d54159970 2023-05-22T15:46:48.419 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.407+0000 7f8d4a7fc640 1 -- 172.21.15.16:0/1964919471 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 7055+0+0 (secure 0 0 0) 0x7f8d5011d310 con 0x7f8d54159970 2023-05-22T15:46:48.420 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.411+0000 7f8d4a7fc640 1 -- 172.21.15.16:0/1964919471 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f8d500eb000 con 0x7f8d54159970 2023-05-22T15:46:48.571 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.567+0000 7f8d5c0e4640 1 -- 172.21.15.16:0/1964919471 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f8d5404dbe0 con 0x7f8d54159970 2023-05-22T15:46:48.572 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.567+0000 7f8d4a7fc640 1 -- 172.21.15.16:0/1964919471 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v149) v1 ==== 100+0+11 (secure 0 0 0) 0x7f8d500b20a0 con 0x7f8d54159970 2023-05-22T15:46:48.572 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 64 2023-05-22T15:46:48.574 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.567+0000 7f8d5c0e4640 1 -- 172.21.15.16:0/1964919471 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8d54159970 msgr2=0x7f8d5414c170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:48.574 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.567+0000 7f8d5c0e4640 1 --2- 172.21.15.16:0/1964919471 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8d54159970 0x7f8d5414c170 secure :-1 s=READY pgs=1958 cs=0 l=1 rev1=1 crypto rx=0x7f8d500b0040 tx=0x7f8d50019220 comp rx=0 tx=0).stop 2023-05-22T15:46:48.574 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.567+0000 7f8d5c0e4640 1 -- 172.21.15.16:0/1964919471 shutdown_connections 2023-05-22T15:46:48.574 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.567+0000 7f8d5c0e4640 1 --2- 172.21.15.16:0/1964919471 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8d5415db00 0x7f8d54151810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:48.574 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.567+0000 7f8d5c0e4640 1 --2- 172.21.15.16:0/1964919471 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8d5415d680 0x7f8d54158b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:48.575 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.567+0000 7f8d5c0e4640 1 --2- 172.21.15.16:0/1964919471 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8d54159970 0x7f8d5414c170 unknown :-1 s=CLOSED pgs=1958 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:48.575 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.567+0000 7f8d5c0e4640 1 -- 172.21.15.16:0/1964919471 >> 172.21.15.16:0/1964919471 conn(0x7f8d54069720 msgr2=0x7f8d54069af0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:46:48.575 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.567+0000 7f8d5c0e4640 1 -- 172.21.15.16:0/1964919471 shutdown_connections 2023-05-22T15:46:48.575 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.567+0000 7f8d5c0e4640 1 -- 172.21.15.16:0/1964919471 wait complete. 2023-05-22T15:46:48.591 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-05-22T15:46:48.723 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.719+0000 7f412f3b5640 1 Processor -- start 2023-05-22T15:46:48.724 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.719+0000 7f412f3b5640 1 -- start start 2023-05-22T15:46:48.726 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.719+0000 7f412f3b5640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f41280cae20 0x7f412806fef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:48.726 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.719+0000 7f412f3b5640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4128069720 con 0x7f4128153fe0 2023-05-22T15:46:48.727 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.719+0000 7f412f3b5640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41281543b0 con 0x7f4128070430 2023-05-22T15:46:48.727 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.719+0000 7f412f3b5640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41280708b0 con 0x7f41280cae20 2023-05-22T15:46:48.727 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.719+0000 7f412d151640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f4128153fe0 0x7f412806d3f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:57474/0 (socket says 172.21.15.16:57474) 2023-05-22T15:46:48.727 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.719+0000 7f412d151640 1 -- 172.21.15.16:0/4049599330 learned_addr learned my addr 172.21.15.16:0/4049599330 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:46:48.727 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.719+0000 7f412c950640 1 --2- 172.21.15.16:0/4049599330 >> v2:172.21.15.62:3300/0 conn(0x7f41280cae20 0x7f412806fef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:48.727 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.719+0000 7f411ffff640 1 -- 172.21.15.16:0/4049599330 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2096590462 0 0) 0x7f4128069720 con 0x7f4128153fe0 2023-05-22T15:46:48.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.719+0000 7f411ffff640 1 -- 172.21.15.16:0/4049599330 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4114003530 con 0x7f4128153fe0 2023-05-22T15:46:48.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.719+0000 7f411ffff640 1 -- 172.21.15.16:0/4049599330 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 819295164 0 0) 0x7f4114003530 con 0x7f4128153fe0 2023-05-22T15:46:48.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.719+0000 7f411ffff640 1 -- 172.21.15.16:0/4049599330 >> v2:172.21.15.62:3300/0 conn(0x7f41280cae20 msgr2=0x7f412806fef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:46:48.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.719+0000 7f411ffff640 1 --2- 172.21.15.16:0/4049599330 >> v2:172.21.15.62:3300/0 conn(0x7f41280cae20 0x7f412806fef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:48.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.719+0000 7f411ffff640 1 -- 172.21.15.16:0/4049599330 >> v1:172.21.15.62:6789/0 conn(0x7f4128070430 legacy=0x7f41280ad410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:48.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.719+0000 7f411ffff640 1 -- 172.21.15.16:0/4049599330 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f412815e360 con 0x7f4128153fe0 2023-05-22T15:46:48.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.719+0000 7f411ffff640 1 -- 172.21.15.16:0/4049599330 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7f4124001680 con 0x7f4128153fe0 2023-05-22T15:46:48.729 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.719+0000 7f411ffff640 1 -- 172.21.15.16:0/4049599330 >> v1:172.21.15.16:6789/0 conn(0x7f4128153fe0 legacy=0x7f412806d3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:48.729 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.719+0000 7f411ffff640 1 --2- 172.21.15.16:0/4049599330 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4114003980 0x7f4114003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:48.729 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.719+0000 7f411ffff640 1 --2- 172.21.15.16:0/4049599330 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f41140047d0 0x7f4114004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:48.729 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.719+0000 7f411ffff640 1 --2- 172.21.15.16:0/4049599330 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4114005180 0x7f411400b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:48.730 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.719+0000 7f411ffff640 1 -- 172.21.15.16:0/4049599330 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41280708b0 con 0x7f4114003980 2023-05-22T15:46:48.730 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.719+0000 7f411ffff640 1 -- 172.21.15.16:0/4049599330 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f411400c000 con 0x7f41140047d0 2023-05-22T15:46:48.730 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.719+0000 7f411ffff640 1 -- 172.21.15.16:0/4049599330 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f411400c1c0 con 0x7f4114005180 2023-05-22T15:46:48.730 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.719+0000 7f412d151640 1 --2- 172.21.15.16:0/4049599330 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4114003980 0x7f4114003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:48.731 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.719+0000 7f412d952640 1 --2- 172.21.15.16:0/4049599330 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4114005180 0x7f411400b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:48.731 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.719+0000 7f412d151640 1 -- 172.21.15.16:0/4049599330 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4114005180 msgr2=0x7f411400b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:46:48.731 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.719+0000 7f412c950640 1 --2- 172.21.15.16:0/4049599330 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f41140047d0 0x7f4114004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:48.731 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.719+0000 7f412d151640 1 --2- 172.21.15.16:0/4049599330 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4114005180 0x7f411400b990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:48.732 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.719+0000 7f412d151640 1 -- 172.21.15.16:0/4049599330 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f41140047d0 msgr2=0x7f4114004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:46:48.732 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.719+0000 7f412d151640 1 --2- 172.21.15.16:0/4049599330 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f41140047d0 0x7f4114004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:48.732 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.719+0000 7f412d151640 1 -- 172.21.15.16:0/4049599330 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4114003530 con 0x7f4114003980 2023-05-22T15:46:48.732 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.719+0000 7f412d952640 1 -- 172.21.15.16:0/4049599330 reap_dead start 2023-05-22T15:46:48.738 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.723+0000 7f412d151640 1 --2- 172.21.15.16:0/4049599330 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4114003980 0x7f4114003d90 secure :-1 s=READY pgs=1960 cs=0 l=1 rev1=1 crypto rx=0x7f412400c240 tx=0x7f412401cc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec7c1a055d341acd server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:48.738 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.723+0000 7f411ffff640 1 -- 172.21.15.16:0/4049599330 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4124005ae0 con 0x7f4114003980 2023-05-22T15:46:48.739 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.723+0000 7f411ffff640 1 -- 172.21.15.16:0/4049599330 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4124005c40 con 0x7f4114003980 2023-05-22T15:46:48.739 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.723+0000 7f411ffff640 1 -- 172.21.15.16:0/4049599330 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41240072b0 con 0x7f4114003980 2023-05-22T15:46:48.739 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.723+0000 7f412f3b5640 1 -- 172.21.15.16:0/4049599330 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4114003980 msgr2=0x7f4114003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:48.740 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.723+0000 7f412f3b5640 1 --2- 172.21.15.16:0/4049599330 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4114003980 0x7f4114003d90 secure :-1 s=READY pgs=1960 cs=0 l=1 rev1=1 crypto rx=0x7f412400c240 tx=0x7f412401cc10 comp rx=0 tx=0).stop 2023-05-22T15:46:48.741 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.723+0000 7f412f3b5640 1 -- 172.21.15.16:0/4049599330 shutdown_connections 2023-05-22T15:46:48.741 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.723+0000 7f412f3b5640 1 --2- 172.21.15.16:0/4049599330 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4114003980 0x7f4114003d90 unknown :-1 s=CLOSED pgs=1960 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:48.741 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.723+0000 7f412f3b5640 1 -- 172.21.15.16:0/4049599330 >> 172.21.15.16:0/4049599330 conn(0x7f41280b1110 msgr2=0x7f41280ce830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:46:48.741 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.723+0000 7f412f3b5640 1 -- 172.21.15.16:0/4049599330 shutdown_connections 2023-05-22T15:46:48.742 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.723+0000 7f412f3b5640 1 -- 172.21.15.16:0/4049599330 wait complete. 2023-05-22T15:46:48.742 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.723+0000 7f412f3b5640 1 Processor -- start 2023-05-22T15:46:48.742 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.723+0000 7f412f3b5640 1 -- start start 2023-05-22T15:46:48.742 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.723+0000 7f412f3b5640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4114003980 0x7f41281f1950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:48.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.723+0000 7f412f3b5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f41281545d0 0x7f41280cd110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:48.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.723+0000 7f412f3b5640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4128150c90 0x7f4128151100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:48.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.723+0000 7f412f3b5640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4128151640 con 0x7f41281545d0 2023-05-22T15:46:48.744 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.723+0000 7f412f3b5640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4128154b20 con 0x7f4114003980 2023-05-22T15:46:48.744 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.723+0000 7f412f3b5640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4128154e00 con 0x7f4128150c90 2023-05-22T15:46:48.744 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.723+0000 7f412c950640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f41281545d0 0x7f41280cd110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:48.744 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.727+0000 7f412c950640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f41281545d0 0x7f41280cd110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41796/0 (socket says 172.21.15.16:41796) 2023-05-22T15:46:48.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.727+0000 7f412c950640 1 -- 172.21.15.16:0/3040311862 learned_addr learned my addr 172.21.15.16:0/3040311862 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:46:48.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.727+0000 7f412d151640 1 --2- 172.21.15.16:0/3040311862 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4114003980 0x7f41281f1950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:48.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.727+0000 7f412d952640 1 --2- 172.21.15.16:0/3040311862 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4128150c90 0x7f4128151100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:48.746 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.727+0000 7f412c950640 1 -- 172.21.15.16:0/3040311862 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4128150c90 msgr2=0x7f4128151100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:48.746 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.727+0000 7f412c950640 1 --2- 172.21.15.16:0/3040311862 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4128150c90 0x7f4128151100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:48.747 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.727+0000 7f412c950640 1 -- 172.21.15.16:0/3040311862 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4114003980 msgr2=0x7f41281f1950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:48.747 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.727+0000 7f412c950640 1 --2- 172.21.15.16:0/3040311862 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4114003980 0x7f41281f1950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:48.747 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.727+0000 7f412c950640 1 -- 172.21.15.16:0/3040311862 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f412815de40 con 0x7f41281545d0 2023-05-22T15:46:48.747 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.727+0000 7f412d952640 1 --2- 172.21.15.16:0/3040311862 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4128150c90 0x7f4128151100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:46:48.748 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.727+0000 7f412c950640 1 --2- 172.21.15.16:0/3040311862 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f41281545d0 0x7f41280cd110 secure :-1 s=READY pgs=1961 cs=0 l=1 rev1=1 crypto rx=0x7f4118010900 tx=0x7f4118010dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:48.748 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.727+0000 7f411dffb640 1 -- 172.21.15.16:0/3040311862 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4118009650 con 0x7f41281545d0 2023-05-22T15:46:48.748 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.727+0000 7f411dffb640 1 -- 172.21.15.16:0/3040311862 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f411800ebc0 con 0x7f41281545d0 2023-05-22T15:46:48.748 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.727+0000 7f412f3b5640 1 -- 172.21.15.16:0/3040311862 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f412815db60 con 0x7f41281545d0 2023-05-22T15:46:48.749 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.727+0000 7f411dffb640 1 -- 172.21.15.16:0/3040311862 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f411801c400 con 0x7f41281545d0 2023-05-22T15:46:48.749 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.727+0000 7f412f3b5640 1 -- 172.21.15.16:0/3040311862 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f412806dd20 con 0x7f41281545d0 2023-05-22T15:46:48.749 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.727+0000 7f412f3b5640 1 -- 172.21.15.16:0/3040311862 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f412804dbe0 con 0x7f41281545d0 2023-05-22T15:46:48.750 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.727+0000 7f411dffb640 1 -- 172.21.15.16:0/3040311862 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 66) v1 ==== 84068+0+0 (secure 0 0 0) 0x7f411801c560 con 0x7f41281545d0 2023-05-22T15:46:48.750 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.727+0000 7f411dffb640 1 --2- 172.21.15.16:0/3040311862 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f410c0651d0 0x7f410c067680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:48.750 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.727+0000 7f411dffb640 1 -- 172.21.15.16:0/3040311862 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 7055+0+0 (secure 0 0 0) 0x7f4118082dc0 con 0x7f41281545d0 2023-05-22T15:46:48.751 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.727+0000 7f412d151640 1 --2- 172.21.15.16:0/3040311862 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f410c0651d0 0x7f410c067680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:48.751 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.727+0000 7f412d151640 1 --2- 172.21.15.16:0/3040311862 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f410c0651d0 0x7f410c067680 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f41240ab000 tx=0x7f41240b1040 comp rx=0 tx=0).ready entity=mgr.9396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:48.755 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.731+0000 7f411dffb640 1 -- 172.21.15.16:0/3040311862 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f4118050940 con 0x7f41281545d0 2023-05-22T15:46:48.924 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.919+0000 7f412f3b5640 1 -- 172.21.15.16:0/3040311862 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f412806c440 con 0x7f41281545d0 2023-05-22T15:46:48.925 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.919+0000 7f411dffb640 1 -- 172.21.15.16:0/3040311862 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 58 v58) v1 ==== 94+0+4650 (secure 0 0 0) 0x7f4118053f60 con 0x7f41281545d0 2023-05-22T15:46:48.926 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:46:48.926 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":58,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7753,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6834/3192074681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":3192074681},{"type":"v1","addr":"172.21.15.62:6834","nonce":3192074681}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":53},{"gid":7756,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/1344243469","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6833","nonce":1344243469},{"type":"v1","addr":"172.21.15.62:6835","nonce":1344243469}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":53},{"gid":9069,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/3636102434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":3636102434},{"type":"v1","addr":"172.21.15.16:6835","nonce":3636102434}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":53}],"filesystems":[{"mdsmap":{"epoch":58,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-05-22T15:46:23.443662+0000","modified":"2023-05-22T15:46:26.557591+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9075},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9075":{"gid":9075,"name":"a","rank":0,"incarnation":54,"state":"up:active","state_seq":3,"addr":"172.21.15.16:6837/2858899325","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":2858899325},{"type":"v1","addr":"172.21.15.16:6837","nonce":2858899325}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2023-05-22T15:46:48.926 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 58 2023-05-22T15:46:48.928 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.923+0000 7f412f3b5640 1 -- 172.21.15.16:0/3040311862 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f410c0651d0 msgr2=0x7f410c067680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:48.928 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.923+0000 7f412f3b5640 1 --2- 172.21.15.16:0/3040311862 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f410c0651d0 0x7f410c067680 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f41240ab000 tx=0x7f41240b1040 comp rx=0 tx=0).stop 2023-05-22T15:46:48.928 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.923+0000 7f412f3b5640 1 -- 172.21.15.16:0/3040311862 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f41281545d0 msgr2=0x7f41280cd110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:48.928 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.923+0000 7f412f3b5640 1 --2- 172.21.15.16:0/3040311862 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f41281545d0 0x7f41280cd110 secure :-1 s=READY pgs=1961 cs=0 l=1 rev1=1 crypto rx=0x7f4118010900 tx=0x7f4118010dc0 comp rx=0 tx=0).stop 2023-05-22T15:46:48.929 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.923+0000 7f412f3b5640 1 -- 172.21.15.16:0/3040311862 shutdown_connections 2023-05-22T15:46:48.929 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.923+0000 7f412f3b5640 1 --2- 172.21.15.16:0/3040311862 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f410c0651d0 0x7f410c067680 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:48.929 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.923+0000 7f412f3b5640 1 --2- 172.21.15.16:0/3040311862 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4128150c90 0x7f4128151100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:48.930 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.923+0000 7f412f3b5640 1 --2- 172.21.15.16:0/3040311862 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f41281545d0 0x7f41280cd110 unknown :-1 s=CLOSED pgs=1961 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:48.930 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.923+0000 7f412f3b5640 1 --2- 172.21.15.16:0/3040311862 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4114003980 0x7f41281f1950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:48.930 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.923+0000 7f412f3b5640 1 -- 172.21.15.16:0/3040311862 >> 172.21.15.16:0/3040311862 conn(0x7f41280b1110 msgr2=0x7f4128152300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:46:48.930 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.923+0000 7f412f3b5640 1 -- 172.21.15.16:0/3040311862 shutdown_connections 2023-05-22T15:46:48.931 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:48.923+0000 7f412f3b5640 1 -- 172.21.15.16:0/3040311862 wait complete. 2023-05-22T15:46:48.943 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-05-22T15:46:49.054 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.047+0000 7f04c9943640 1 Processor -- start 2023-05-22T15:46:49.054 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.047+0000 7f04c9943640 1 -- start start 2023-05-22T15:46:49.055 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.051+0000 7f04c9943640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f04c40ca4e0 0x7f04c415e460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:49.055 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.051+0000 7f04c9943640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f04c40c9b40 0x7f04c40afb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:49.056 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.051+0000 7f04c9943640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04c40b0650 con 0x7f04c4070df0 2023-05-22T15:46:49.056 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.051+0000 7f04c9943640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04c40b0370 con 0x7f04c40c9b40 2023-05-22T15:46:49.056 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.051+0000 7f04c9943640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04c415d980 con 0x7f04c40ca4e0 2023-05-22T15:46:49.056 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.051+0000 7f04ba7fc640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f04c40c9b40 0x7f04c40afb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:49.056 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.051+0000 7f04ba7fc640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f04c40c9b40 0x7f04c40afb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41802/0 (socket says 172.21.15.16:41802) 2023-05-22T15:46:49.056 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.051+0000 7f04ba7fc640 1 -- 172.21.15.16:0/1132810625 learned_addr learned my addr 172.21.15.16:0/1132810625 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:46:49.056 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.051+0000 7f04c37fe640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f04c4070df0 0x7f04c4071210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:57482/0 (socket says 172.21.15.16:57482) 2023-05-22T15:46:49.057 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.051+0000 7f04c2ffd640 1 --2- 172.21.15.16:0/1132810625 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f04c40ca4e0 0x7f04c415e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:49.057 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.051+0000 7f04ba7fc640 1 -- 172.21.15.16:0/1132810625 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f04c40ca4e0 msgr2=0x7f04c415e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:46:49.057 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.051+0000 7f04ba7fc640 1 --2- 172.21.15.16:0/1132810625 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f04c40ca4e0 0x7f04c415e460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:49.058 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.051+0000 7f04ba7fc640 1 -- 172.21.15.16:0/1132810625 >> v1:172.21.15.16:6789/0 conn(0x7f04c4070df0 legacy=0x7f04c4071210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:49.058 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.051+0000 7f04ba7fc640 1 -- 172.21.15.16:0/1132810625 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04c415dc60 con 0x7f04c40c9b40 2023-05-22T15:46:49.058 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.051+0000 7f04c27fc640 1 -- 172.21.15.16:0/1132810625 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2731033719 0 0) 0x7f04c40b0650 con 0x7f04c4070df0 2023-05-22T15:46:49.058 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.051+0000 7f04c2ffd640 1 --2- 172.21.15.16:0/1132810625 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f04c40ca4e0 0x7f04c415e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T15:46:49.059 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.051+0000 7f04ba7fc640 1 --2- 172.21.15.16:0/1132810625 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f04c40c9b40 0x7f04c40afb40 secure :-1 s=READY pgs=1963 cs=0 l=1 rev1=1 crypto rx=0x7f04b4009870 tx=0x7f04b401c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87bdf70e72582c79 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:49.059 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.051+0000 7f04c27fc640 1 -- 172.21.15.16:0/1132810625 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04b40ab660 con 0x7f04c40c9b40 2023-05-22T15:46:49.059 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.051+0000 7f04c27fc640 1 -- 172.21.15.16:0/1132810625 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04b40ab7c0 con 0x7f04c40c9b40 2023-05-22T15:46:49.060 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.051+0000 7f04c27fc640 1 -- 172.21.15.16:0/1132810625 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04b40abb50 con 0x7f04c40c9b40 2023-05-22T15:46:49.060 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.051+0000 7f04c9943640 1 -- 172.21.15.16:0/1132810625 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f04c40c9b40 msgr2=0x7f04c40afb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:49.060 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.051+0000 7f04c9943640 1 --2- 172.21.15.16:0/1132810625 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f04c40c9b40 0x7f04c40afb40 secure :-1 s=READY pgs=1963 cs=0 l=1 rev1=1 crypto rx=0x7f04b4009870 tx=0x7f04b401c480 comp rx=0 tx=0).stop 2023-05-22T15:46:49.060 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.051+0000 7f04c9943640 1 -- 172.21.15.16:0/1132810625 shutdown_connections 2023-05-22T15:46:49.060 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.051+0000 7f04c9943640 1 --2- 172.21.15.16:0/1132810625 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f04c40c9b40 0x7f04c40afb40 unknown :-1 s=CLOSED pgs=1963 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:49.060 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.051+0000 7f04c9943640 1 --2- 172.21.15.16:0/1132810625 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f04c40ca4e0 0x7f04c415e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:49.060 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.051+0000 7f04c9943640 1 -- 172.21.15.16:0/1132810625 >> 172.21.15.16:0/1132810625 conn(0x7f04c4069720 msgr2=0x7f04c4069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:46:49.061 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.051+0000 7f04c9943640 1 -- 172.21.15.16:0/1132810625 shutdown_connections 2023-05-22T15:46:49.061 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.051+0000 7f04c9943640 1 -- 172.21.15.16:0/1132810625 wait complete. 2023-05-22T15:46:49.062 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.055+0000 7f04c9943640 1 Processor -- start 2023-05-22T15:46:49.062 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.055+0000 7f04c9943640 1 -- start start 2023-05-22T15:46:49.062 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.055+0000 7f04c9943640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f04c4070df0 0x7f04c414f270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:49.062 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.055+0000 7f04c9943640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f04c40c9b40 0x7f04c414fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:49.062 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.055+0000 7f04c9943640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f04c40ca4e0 0x7f04c41502a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:49.063 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.055+0000 7f04c9943640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04c4151dc0 con 0x7f04c40ca4e0 2023-05-22T15:46:49.063 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.055+0000 7f04c9943640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04c41520a0 con 0x7f04c40c9b40 2023-05-22T15:46:49.063 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.055+0000 7f04c9943640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04c4152380 con 0x7f04c4070df0 2023-05-22T15:46:49.063 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.055+0000 7f04c37fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f04c40ca4e0 0x7f04c41502a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:49.063 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.055+0000 7f04c37fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f04c40ca4e0 0x7f04c41502a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41816/0 (socket says 172.21.15.16:41816) 2023-05-22T15:46:49.063 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.055+0000 7f04c37fe640 1 -- 172.21.15.16:0/3196817309 learned_addr learned my addr 172.21.15.16:0/3196817309 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:46:49.063 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.055+0000 7f04c37fe640 1 -- 172.21.15.16:0/3196817309 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f04c4070df0 msgr2=0x7f04c414f270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:49.064 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.055+0000 7f04c37fe640 1 --2- 172.21.15.16:0/3196817309 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f04c4070df0 0x7f04c414f270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:49.064 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.055+0000 7f04c37fe640 1 -- 172.21.15.16:0/3196817309 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f04c40c9b40 msgr2=0x7f04c414fd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:49.064 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.055+0000 7f04c37fe640 1 --2- 172.21.15.16:0/3196817309 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f04c40c9b40 0x7f04c414fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:49.064 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.055+0000 7f04c37fe640 1 -- 172.21.15.16:0/3196817309 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04b40094d0 con 0x7f04c40ca4e0 2023-05-22T15:46:49.064 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.055+0000 7f04c37fe640 1 --2- 172.21.15.16:0/3196817309 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f04c40ca4e0 0x7f04c41502a0 secure :-1 s=READY pgs=1964 cs=0 l=1 rev1=1 crypto rx=0x7f04bc001570 tx=0x7f04bc00cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:49.064 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.055+0000 7f04c0ff9640 1 -- 172.21.15.16:0/3196817309 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04bc00a640 con 0x7f04c40ca4e0 2023-05-22T15:46:49.064 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.055+0000 7f04c0ff9640 1 -- 172.21.15.16:0/3196817309 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04bc00a7c0 con 0x7f04c40ca4e0 2023-05-22T15:46:49.065 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.055+0000 7f04c9943640 1 -- 172.21.15.16:0/3196817309 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04c40c8e80 con 0x7f04c40ca4e0 2023-05-22T15:46:49.065 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.055+0000 7f04c0ff9640 1 -- 172.21.15.16:0/3196817309 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04bc00ab70 con 0x7f04c40ca4e0 2023-05-22T15:46:49.065 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.055+0000 7f04c9943640 1 -- 172.21.15.16:0/3196817309 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04c40ca8b0 con 0x7f04c40ca4e0 2023-05-22T15:46:49.065 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.055+0000 7f04c9943640 1 -- 172.21.15.16:0/3196817309 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04c40c9010 con 0x7f04c40ca4e0 2023-05-22T15:46:49.066 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.059+0000 7f04c0ff9640 1 -- 172.21.15.16:0/3196817309 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 66) v1 ==== 84068+0+0 (secure 0 0 0) 0x7f04bc00acd0 con 0x7f04c40ca4e0 2023-05-22T15:46:49.066 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.059+0000 7f04c0ff9640 1 --2- 172.21.15.16:0/3196817309 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f04a80651d0 0x7f04a8067680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:49.066 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.059+0000 7f04c0ff9640 1 -- 172.21.15.16:0/3196817309 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 7055+0+0 (secure 0 0 0) 0x7f04bc081e30 con 0x7f04c40ca4e0 2023-05-22T15:46:49.066 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.059+0000 7f04c2ffd640 1 --2- 172.21.15.16:0/3196817309 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f04a80651d0 0x7f04a8067680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:49.066 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.059+0000 7f04c2ffd640 1 --2- 172.21.15.16:0/3196817309 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f04a80651d0 0x7f04a8067680 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f04ac006f10 tx=0x7f04ac008040 comp rx=0 tx=0).ready entity=mgr.9396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:49.067 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.063+0000 7f04c0ff9640 1 -- 172.21.15.16:0/3196817309 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f04bc04fa10 con 0x7f04c40ca4e0 2023-05-22T15:46:49.214 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.207+0000 7f04c9943640 1 -- 172.21.15.16:0/3196817309 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f04c415b010 con 0x7f04c40ca4e0 2023-05-22T15:46:49.217 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.211+0000 7f04c0ff9640 1 -- 172.21.15.16:0/3196817309 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v149) v1 ==== 74+0+19019 (secure 0 0 0) 0x7f04bc018020 con 0x7f04c40ca4e0 2023-05-22T15:46:49.218 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:46:49.218 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":149,"fsid":"b2719834-7847-4d98-90a6-f65d57f85c49","created":"2023-05-22T15:26:25.259314+0000","modified":"2023-05-22T15:46:47.687666+0000","last_up_change":"2023-05-22T15:26:36.448524+0000","last_in_change":"2023-05-22T15:26:27.675668+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-05-22T15:26:38.019781+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-05-22T15:26:43.570514+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2023-05-22T15:46:21.084615+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"146","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_data","create_time":"2023-05-22T15:46:22.440577+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"147","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9e7ff854-b25f-4cbb-a5c2-1799d9c53d2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":37021},{"type":"v1","addr":"172.21.15.16:6825","nonce":37021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":37021},{"type":"v1","addr":"172.21.15.16:6827","nonce":37021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":37021},{"type":"v1","addr":"172.21.15.16:6831","nonce":37021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":37021},{"type":"v1","addr":"172.21.15.16:6829","nonce":37021}]},"public_addr":"172.21.15.16:6825/37021","cluster_addr":"172.21.15.16:6827/37021","heartbeat_back_addr":"172.21.15.16:6831/37021","heartbeat_front_addr":"172.21.15.16:6829/37021","state":["exists","up"]},{"osd":1,"uuid":"4702fbc9-a927-4565-aca5-41dc69cc8264","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":37008},{"type":"v1","addr":"172.21.15.16:6801","nonce":37008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":37008},{"type":"v1","addr":"172.21.15.16:6803","nonce":37008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":37008},{"type":"v1","addr":"172.21.15.16:6807","nonce":37008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":37008},{"type":"v1","addr":"172.21.15.16:6805","nonce":37008}]},"public_addr":"172.21.15.16:6801/37008","cluster_addr":"172.21.15.16:6803/37008","heartbeat_back_addr":"172.21.15.16:6807/37008","heartbeat_front_addr":"172.21.15.16:6805/37008","state":["exists","up"]},{"osd":2,"uuid":"1e7bba46-4eba-47ef-bb81-5e2d371638ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":37009},{"type":"v1","addr":"172.21.15.16:6809","nonce":37009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":37009},{"type":"v1","addr":"172.21.15.16:6811","nonce":37009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":37009},{"type":"v1","addr":"172.21.15.16:6815","nonce":37009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":37009},{"type":"v1","addr":"172.21.15.16:6813","nonce":37009}]},"public_addr":"172.21.15.16:6809/37009","cluster_addr":"172.21.15.16:6811/37009","heartbeat_back_addr":"172.21.15.16:6815/37009","heartbeat_front_addr":"172.21.15.16:6813/37009","state":["exists","up"]},{"osd":3,"uuid":"383cacc0-5a88-4904-990e-498ca59ef48a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":37020},{"type":"v1","addr":"172.21.15.16:6817","nonce":37020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":37020},{"type":"v1","addr":"172.21.15.16:6819","nonce":37020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":37020},{"type":"v1","addr":"172.21.15.16:6823","nonce":37020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":37020},{"type":"v1","addr":"172.21.15.16:6821","nonce":37020}]},"public_addr":"172.21.15.16:6817/37020","cluster_addr":"172.21.15.16:6819/37020","heartbeat_back_addr":"172.21.15.16:6823/37020","heartbeat_front_addr":"172.21.15.16:6821/37020","state":["exists","up"]},{"osd":4,"uuid":"970500ef-95b7-42c1-b080-eed4fae7013d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":37353},{"type":"v1","addr":"172.21.15.62:6817","nonce":37353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6819","nonce":37353},{"type":"v1","addr":"172.21.15.62:6822","nonce":37353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37353},{"type":"v1","addr":"172.21.15.62:6831","nonce":37353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37353},{"type":"v1","addr":"172.21.15.62:6826","nonce":37353}]},"public_addr":"172.21.15.62:6817/37353","cluster_addr":"172.21.15.62:6822/37353","heartbeat_back_addr":"172.21.15.62:6831/37353","heartbeat_front_addr":"172.21.15.62:6826/37353","state":["exists","up"]},{"osd":5,"uuid":"022d67db-d4d8-445b-b1b9-8b446a00f5c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":37352},{"type":"v1","addr":"172.21.15.62:6801","nonce":37352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":37352},{"type":"v1","addr":"172.21.15.62:6803","nonce":37352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":37352},{"type":"v1","addr":"172.21.15.62:6807","nonce":37352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":37352},{"type":"v1","addr":"172.21.15.62:6805","nonce":37352}]},"public_addr":"172.21.15.62:6801/37352","cluster_addr":"172.21.15.62:6803/37352","heartbeat_back_addr":"172.21.15.62:6807/37352","heartbeat_front_addr":"172.21.15.62:6805/37352","state":["exists","up"]},{"osd":6,"uuid":"42bd372b-6688-4a1b-8407-e16e8c266e33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":37354},{"type":"v1","addr":"172.21.15.62:6820","nonce":37354}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":37354},{"type":"v1","addr":"172.21.15.62:6823","nonce":37354}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6829","nonce":37354},{"type":"v1","addr":"172.21.15.62:6830","nonce":37354}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6825","nonce":37354},{"type":"v1","addr":"172.21.15.62:6827","nonce":37354}]},"public_addr":"172.21.15.62:6820/37354","cluster_addr":"172.21.15.62:6823/37354","heartbeat_back_addr":"172.21.15.62:6830/37354","heartbeat_front_addr":"172.21.15.62:6827/37354","state":["exists","up"]},{"osd":7,"uuid":"ac70b2a3-f41b-4bf2-b227-7f2cbb55426d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37355},{"type":"v1","addr":"172.21.15.62:6809","nonce":37355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37355},{"type":"v1","addr":"172.21.15.62:6811","nonce":37355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":37355},{"type":"v1","addr":"172.21.15.62:6815","nonce":37355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":37355},{"type":"v1","addr":"172.21.15.62:6813","nonce":37355}]},"public_addr":"172.21.15.62:6809/37355","cluster_addr":"172.21.15.62:6811/37355","heartbeat_back_addr":"172.21.15.62:6815/37355","heartbeat_front_addr":"172.21.15.62:6813/37355","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.368896+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.107273+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.276888+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.287502+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.124791+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.251937+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.255679+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.232797+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:0/2660771812":"2023-05-23T15:46:47.687617+0000","172.21.15.16:0/2077740732":"2023-05-23T15:46:47.687617+0000","172.21.15.16:0/2030430759":"2023-05-23T15:46:47.687617+0000","172.21.15.16:0/4220632598":"2023-05-23T15:46:47.687617+0000","172.21.15.16:0/2231972099":"2023-05-23T15:46:47.687617+0000","172.21.15.16:0/3429069808":"2023-05-23T15:46:41.068549+0000","172.21.15.16:0/2806812328":"2023-05-23T15:46:47.687617+0000","172.21.15.16:0/4084148691":"2023-05-23T15:46:41.068549+0000","172.21.15.16:0/3718143920":"2023-05-23T15:46:47.687617+0000","172.21.15.16:0/3605381301":"2023-05-23T15:46:41.068549+0000","172.21.15.16:0/2954477712":"2023-05-23T15:46:41.068549+0000","172.21.15.16:0/3271028793":"2023-05-23T15:46:41.068549+0000","172.21.15.16:6833/36921":"2023-05-23T15:46:41.068549+0000","172.21.15.16:6832/36921":"2023-05-23T15:46:41.068549+0000","172.21.15.16:0/1944100655":"2023-05-23T15:46:41.068549+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":17,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-05-22T15:46:49.219 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.215+0000 7f04c9943640 1 -- 172.21.15.16:0/3196817309 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f04a80651d0 msgr2=0x7f04a8067680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:49.219 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.215+0000 7f04c9943640 1 --2- 172.21.15.16:0/3196817309 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f04a80651d0 0x7f04a8067680 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f04ac006f10 tx=0x7f04ac008040 comp rx=0 tx=0).stop 2023-05-22T15:46:49.220 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.215+0000 7f04c9943640 1 -- 172.21.15.16:0/3196817309 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f04c40ca4e0 msgr2=0x7f04c41502a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:49.220 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.215+0000 7f04c9943640 1 --2- 172.21.15.16:0/3196817309 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f04c40ca4e0 0x7f04c41502a0 secure :-1 s=READY pgs=1964 cs=0 l=1 rev1=1 crypto rx=0x7f04bc001570 tx=0x7f04bc00cca0 comp rx=0 tx=0).stop 2023-05-22T15:46:49.220 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.215+0000 7f04c9943640 1 -- 172.21.15.16:0/3196817309 shutdown_connections 2023-05-22T15:46:49.220 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.215+0000 7f04c9943640 1 --2- 172.21.15.16:0/3196817309 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f04c40ca4e0 0x7f04c41502a0 unknown :-1 s=CLOSED pgs=1964 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:49.220 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.215+0000 7f04c9943640 1 --2- 172.21.15.16:0/3196817309 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f04c40c9b40 0x7f04c414fd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:49.220 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.215+0000 7f04c9943640 1 --2- 172.21.15.16:0/3196817309 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f04a80651d0 0x7f04a8067680 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:49.220 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.215+0000 7f04c9943640 1 --2- 172.21.15.16:0/3196817309 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f04c4070df0 0x7f04c414f270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:49.221 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.215+0000 7f04c9943640 1 -- 172.21.15.16:0/3196817309 >> 172.21.15.16:0/3196817309 conn(0x7f04c4069720 msgr2=0x7f04c40b0eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:46:49.221 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.215+0000 7f04c9943640 1 -- 172.21.15.16:0/3196817309 shutdown_connections 2023-05-22T15:46:49.221 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.215+0000 7f04c9943640 1 -- 172.21.15.16:0/3196817309 wait complete. 2023-05-22T15:46:49.235 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-05-22T15:46:49.236 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-05-22T15:46:49.359 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.351+0000 7f4a74f6c640 1 Processor -- start 2023-05-22T15:46:49.359 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.351+0000 7f4a74f6c640 1 -- start start 2023-05-22T15:46:49.360 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.355+0000 7f4a74f6c640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f4a70158290 0x7f4a700c8880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:49.360 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.355+0000 7f4a74f6c640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a700c92b0 con 0x7f4a700ce6c0 2023-05-22T15:46:49.360 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.355+0000 7f4a74f6c640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a700cc060 con 0x7f4a70158660 2023-05-22T15:46:49.361 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.355+0000 7f4a74f6c640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a700cc340 con 0x7f4a70158290 2023-05-22T15:46:49.361 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.355+0000 7f4a6e59c640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f4a70158290 0x7f4a700c8880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:49.361 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.355+0000 7f4a6ed9d640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f4a700ce6c0 0x7f4a7006f630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:55540/0 (socket says 172.21.15.16:55540) 2023-05-22T15:46:49.361 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.355+0000 7f4a6ed9d640 1 -- 172.21.15.16:0/634997467 learned_addr learned my addr 172.21.15.16:0/634997467 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:46:49.361 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.355+0000 7f4a6e59c640 1 -- 172.21.15.16:0/634997467 >> v1:172.21.15.62:6789/0 conn(0x7f4a70158660 legacy=0x7f4a700cdfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:46:49.361 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.355+0000 7f4a6d59a640 1 -- 172.21.15.16:0/634997467 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 283513173 0 0) 0x7f4a700c92b0 con 0x7f4a700ce6c0 2023-05-22T15:46:49.361 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.355+0000 7f4a6e59c640 1 -- 172.21.15.16:0/634997467 >> v1:172.21.15.16:6789/0 conn(0x7f4a700ce6c0 legacy=0x7f4a7006f630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:49.362 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.355+0000 7f4a6e59c640 1 -- 172.21.15.16:0/634997467 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a700cc620 con 0x7f4a70158290 2023-05-22T15:46:49.362 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.355+0000 7f4a6e59c640 1 --2- 172.21.15.16:0/634997467 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4a70158290 0x7f4a700c8880 secure :-1 s=READY pgs=1966 cs=0 l=1 rev1=1 crypto rx=0x7f4a600097c0 tx=0x7f4a6001c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f10593a5062de60f server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:49.362 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.355+0000 7f4a6d59a640 1 -- 172.21.15.16:0/634997467 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a600ab670 con 0x7f4a70158290 2023-05-22T15:46:49.362 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.355+0000 7f4a6d59a640 1 -- 172.21.15.16:0/634997467 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a600ab7d0 con 0x7f4a70158290 2023-05-22T15:46:49.363 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.355+0000 7f4a6d59a640 1 -- 172.21.15.16:0/634997467 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a600abb60 con 0x7f4a70158290 2023-05-22T15:46:49.363 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.359+0000 7f4a74f6c640 1 -- 172.21.15.16:0/634997467 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4a70158290 msgr2=0x7f4a700c8880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:49.363 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.359+0000 7f4a74f6c640 1 --2- 172.21.15.16:0/634997467 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4a70158290 0x7f4a700c8880 secure :-1 s=READY pgs=1966 cs=0 l=1 rev1=1 crypto rx=0x7f4a600097c0 tx=0x7f4a6001c450 comp rx=0 tx=0).stop 2023-05-22T15:46:49.364 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.359+0000 7f4a74f6c640 1 -- 172.21.15.16:0/634997467 shutdown_connections 2023-05-22T15:46:49.364 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.359+0000 7f4a74f6c640 1 --2- 172.21.15.16:0/634997467 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4a70158290 0x7f4a700c8880 unknown :-1 s=CLOSED pgs=1966 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:49.364 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.359+0000 7f4a74f6c640 1 -- 172.21.15.16:0/634997467 >> 172.21.15.16:0/634997467 conn(0x7f4a70069720 msgr2=0x7f4a70069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:46:49.364 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.359+0000 7f4a74f6c640 1 -- 172.21.15.16:0/634997467 shutdown_connections 2023-05-22T15:46:49.364 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.359+0000 7f4a74f6c640 1 -- 172.21.15.16:0/634997467 wait complete. 2023-05-22T15:46:49.365 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.359+0000 7f4a74f6c640 1 Processor -- start 2023-05-22T15:46:49.365 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.359+0000 7f4a74f6c640 1 -- start start 2023-05-22T15:46:49.365 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.359+0000 7f4a74f6c640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4a700ce6c0 0x7f4a7014a930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:49.366 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.359+0000 7f4a74f6c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4a70158290 0x7f4a700c1b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:49.366 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.359+0000 7f4a74f6c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4a70158660 0x7f4a7015ae90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:49.366 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.359+0000 7f4a74f6c640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a700c3140 con 0x7f4a70158290 2023-05-22T15:46:49.366 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.359+0000 7f4a74f6c640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a700c2530 con 0x7f4a70158660 2023-05-22T15:46:49.366 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.359+0000 7f4a74f6c640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a700c2810 con 0x7f4a700ce6c0 2023-05-22T15:46:49.367 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.359+0000 7f4a6dd9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4a70158290 0x7f4a700c1b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:49.368 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.359+0000 7f4a6dd9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4a70158290 0x7f4a700c1b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43364/0 (socket says 172.21.15.16:43364) 2023-05-22T15:46:49.368 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.359+0000 7f4a6dd9b640 1 -- 172.21.15.16:0/2291559334 learned_addr learned my addr 172.21.15.16:0/2291559334 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:46:49.368 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.359+0000 7f4a6e59c640 1 --2- 172.21.15.16:0/2291559334 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4a700ce6c0 0x7f4a7014a930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:49.369 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.359+0000 7f4a6ed9d640 1 --2- 172.21.15.16:0/2291559334 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4a70158660 0x7f4a7015ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:49.369 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.359+0000 7f4a6dd9b640 1 -- 172.21.15.16:0/2291559334 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4a700ce6c0 msgr2=0x7f4a7014a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:49.369 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.359+0000 7f4a6dd9b640 1 --2- 172.21.15.16:0/2291559334 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4a700ce6c0 0x7f4a7014a930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:49.369 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.359+0000 7f4a6dd9b640 1 -- 172.21.15.16:0/2291559334 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4a70158660 msgr2=0x7f4a7015ae90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:49.369 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.359+0000 7f4a6dd9b640 1 --2- 172.21.15.16:0/2291559334 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4a70158660 0x7f4a7015ae90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:49.369 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.359+0000 7f4a6dd9b640 1 -- 172.21.15.16:0/2291559334 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a60009420 con 0x7f4a70158290 2023-05-22T15:46:49.369 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.359+0000 7f4a6dd9b640 1 --2- 172.21.15.16:0/2291559334 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4a70158290 0x7f4a700c1b50 secure :-1 s=READY pgs=1967 cs=0 l=1 rev1=1 crypto rx=0x7f4a6400e690 tx=0x7f4a6400eb50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:49.370 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.359+0000 7f4a5effd640 1 -- 172.21.15.16:0/2291559334 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a640052d0 con 0x7f4a70158290 2023-05-22T15:46:49.370 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.359+0000 7f4a5effd640 1 -- 172.21.15.16:0/2291559334 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a64005430 con 0x7f4a70158290 2023-05-22T15:46:49.370 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.359+0000 7f4a74f6c640 1 -- 172.21.15.16:0/2291559334 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a700ccfb0 con 0x7f4a70158290 2023-05-22T15:46:49.370 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.359+0000 7f4a5effd640 1 -- 172.21.15.16:0/2291559334 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a64001950 con 0x7f4a70158290 2023-05-22T15:46:49.370 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.359+0000 7f4a74f6c640 1 -- 172.21.15.16:0/2291559334 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a700cce20 con 0x7f4a70158290 2023-05-22T15:46:49.370 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.363+0000 7f4a74f6c640 1 -- 172.21.15.16:0/2291559334 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a700c8dc0 con 0x7f4a70158290 2023-05-22T15:46:49.371 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.363+0000 7f4a5effd640 1 -- 172.21.15.16:0/2291559334 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 66) v1 ==== 84068+0+0 (secure 0 0 0) 0x7f4a64001ab0 con 0x7f4a70158290 2023-05-22T15:46:49.371 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.363+0000 7f4a5effd640 1 --2- 172.21.15.16:0/2291559334 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f4a500651d0 0x7f4a50067680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:49.371 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.363+0000 7f4a5effd640 1 -- 172.21.15.16:0/2291559334 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 7055+0+0 (secure 0 0 0) 0x7f4a64084e30 con 0x7f4a70158290 2023-05-22T15:46:49.372 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.363+0000 7f4a6e59c640 1 --2- 172.21.15.16:0/2291559334 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f4a500651d0 0x7f4a50067680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:49.372 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.363+0000 7f4a6e59c640 1 --2- 172.21.15.16:0/2291559334 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f4a500651d0 0x7f4a50067680 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f4a600af000 tx=0x7f4a600a7000 comp rx=0 tx=0).ready entity=mgr.9396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:49.373 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.367+0000 7f4a5effd640 1 -- 172.21.15.16:0/2291559334 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f4a64052970 con 0x7f4a70158290 2023-05-22T15:46:49.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.555+0000 7f4a74f6c640 1 -- 172.21.15.16:0/2291559334 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4a7004dbe0 con 0x7f4a70158290 2023-05-22T15:46:49.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.559+0000 7f4a5effd640 1 -- 172.21.15.16:0/2291559334 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 58 v58) v1 ==== 94+0+4650 (secure 0 0 0) 0x7f4a6401f090 con 0x7f4a70158290 2023-05-22T15:46:49.563 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:46:49.564 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":58,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7753,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6834/3192074681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":3192074681},{"type":"v1","addr":"172.21.15.62:6834","nonce":3192074681}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":53},{"gid":7756,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/1344243469","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6833","nonce":1344243469},{"type":"v1","addr":"172.21.15.62:6835","nonce":1344243469}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":53},{"gid":9069,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/3636102434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":3636102434},{"type":"v1","addr":"172.21.15.16:6835","nonce":3636102434}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":53}],"filesystems":[{"mdsmap":{"epoch":58,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-05-22T15:46:23.443662+0000","modified":"2023-05-22T15:46:26.557591+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9075},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9075":{"gid":9075,"name":"a","rank":0,"incarnation":54,"state":"up:active","state_seq":3,"addr":"172.21.15.16:6837/2858899325","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":2858899325},{"type":"v1","addr":"172.21.15.16:6837","nonce":2858899325}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2023-05-22T15:46:49.564 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 58 2023-05-22T15:46:49.565 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.559+0000 7f4a74f6c640 1 -- 172.21.15.16:0/2291559334 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f4a500651d0 msgr2=0x7f4a50067680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:49.565 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.559+0000 7f4a74f6c640 1 --2- 172.21.15.16:0/2291559334 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f4a500651d0 0x7f4a50067680 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f4a600af000 tx=0x7f4a600a7000 comp rx=0 tx=0).stop 2023-05-22T15:46:49.566 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.559+0000 7f4a74f6c640 1 -- 172.21.15.16:0/2291559334 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4a70158290 msgr2=0x7f4a700c1b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:49.566 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.559+0000 7f4a74f6c640 1 --2- 172.21.15.16:0/2291559334 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4a70158290 0x7f4a700c1b50 secure :-1 s=READY pgs=1967 cs=0 l=1 rev1=1 crypto rx=0x7f4a6400e690 tx=0x7f4a6400eb50 comp rx=0 tx=0).stop 2023-05-22T15:46:49.566 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.559+0000 7f4a74f6c640 1 -- 172.21.15.16:0/2291559334 shutdown_connections 2023-05-22T15:46:49.566 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.559+0000 7f4a74f6c640 1 --2- 172.21.15.16:0/2291559334 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4a70158660 0x7f4a7015ae90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:49.566 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.559+0000 7f4a74f6c640 1 --2- 172.21.15.16:0/2291559334 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4a70158290 0x7f4a700c1b50 unknown :-1 s=CLOSED pgs=1967 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:49.566 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.559+0000 7f4a74f6c640 1 --2- 172.21.15.16:0/2291559334 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f4a500651d0 0x7f4a50067680 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:49.566 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.559+0000 7f4a74f6c640 1 --2- 172.21.15.16:0/2291559334 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4a700ce6c0 0x7f4a7014a930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:49.567 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.559+0000 7f4a74f6c640 1 -- 172.21.15.16:0/2291559334 >> 172.21.15.16:0/2291559334 conn(0x7f4a70069720 msgr2=0x7f4a70069af0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:46:49.567 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.559+0000 7f4a74f6c640 1 -- 172.21.15.16:0/2291559334 shutdown_connections 2023-05-22T15:46:49.567 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.559+0000 7f4a74f6c640 1 -- 172.21.15.16:0/2291559334 wait complete. 2023-05-22T15:46:49.584 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-05-22T15:46:49.703 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.699+0000 7f554cc6f640 1 Processor -- start 2023-05-22T15:46:49.704 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.699+0000 7f554cc6f640 1 -- start start 2023-05-22T15:46:49.705 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.699+0000 7f554cc6f640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f55481563c0 0x7f55480c7b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:49.705 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.699+0000 7f554cc6f640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f55480cc6c0 0x7f55480c8080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:49.706 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.699+0000 7f554cc6f640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f554806c220 0x7f554806c6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:49.706 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.699+0000 7f554cc6f640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f554815c210 con 0x7f55480cc6c0 2023-05-22T15:46:49.706 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.699+0000 7f554cc6f640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f554815c480 con 0x7f554806c220 2023-05-22T15:46:49.706 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.699+0000 7f554cc6f640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f554815c760 con 0x7f55481563c0 2023-05-22T15:46:49.706 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.699+0000 7f5545d9b640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f55480cc6c0 0x7f55480c8080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:49.706 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.699+0000 7f5545d9b640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f55480cc6c0 0x7f55480c8080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43376/0 (socket says 172.21.15.16:43376) 2023-05-22T15:46:49.707 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.699+0000 7f5545d9b640 1 -- 172.21.15.16:0/3188813220 learned_addr learned my addr 172.21.15.16:0/3188813220 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:46:49.707 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.699+0000 7f554659c640 1 --2- 172.21.15.16:0/3188813220 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f55481563c0 0x7f55480c7b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:49.707 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.699+0000 7f5546d9d640 1 --2- 172.21.15.16:0/3188813220 >> v2:172.21.15.62:3300/0 conn(0x7f554806c220 0x7f554806c6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:49.707 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.699+0000 7f5545d9b640 1 -- 172.21.15.16:0/3188813220 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f55481563c0 msgr2=0x7f55480c7b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:46:49.707 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.699+0000 7f5545d9b640 1 --2- 172.21.15.16:0/3188813220 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f55481563c0 0x7f55480c7b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:49.707 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.699+0000 7f5545d9b640 1 -- 172.21.15.16:0/3188813220 >> v2:172.21.15.62:3300/0 conn(0x7f554806c220 msgr2=0x7f554806c6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:46:49.708 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.699+0000 7f5545d9b640 1 --2- 172.21.15.16:0/3188813220 >> v2:172.21.15.62:3300/0 conn(0x7f554806c220 0x7f554806c6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:49.708 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.699+0000 7f5545d9b640 1 -- 172.21.15.16:0/3188813220 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f554815ca40 con 0x7f55480cc6c0 2023-05-22T15:46:49.713 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.703+0000 7f5545d9b640 1 --2- 172.21.15.16:0/3188813220 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f55480cc6c0 0x7f55480c8080 secure :-1 s=READY pgs=1968 cs=0 l=1 rev1=1 crypto rx=0x7f5548159cd0 tx=0x7f553c01c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ab6f840d9ef7008 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:49.714 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.703+0000 7f554559a640 1 -- 172.21.15.16:0/3188813220 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f553c0aa610 con 0x7f55480cc6c0 2023-05-22T15:46:49.714 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.703+0000 7f554559a640 1 -- 172.21.15.16:0/3188813220 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f553c0a6040 con 0x7f55480cc6c0 2023-05-22T15:46:49.714 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.703+0000 7f554559a640 1 -- 172.21.15.16:0/3188813220 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f553c0aa990 con 0x7f55480cc6c0 2023-05-22T15:46:49.714 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.703+0000 7f554cc6f640 1 -- 172.21.15.16:0/3188813220 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f55480cc6c0 msgr2=0x7f55480c8080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:49.715 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.703+0000 7f554cc6f640 1 --2- 172.21.15.16:0/3188813220 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f55480cc6c0 0x7f55480c8080 secure :-1 s=READY pgs=1968 cs=0 l=1 rev1=1 crypto rx=0x7f5548159cd0 tx=0x7f553c01c4d0 comp rx=0 tx=0).stop 2023-05-22T15:46:49.715 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.703+0000 7f554cc6f640 1 -- 172.21.15.16:0/3188813220 shutdown_connections 2023-05-22T15:46:49.715 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.703+0000 7f554cc6f640 1 --2- 172.21.15.16:0/3188813220 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f55480cc6c0 0x7f55480c8080 unknown :-1 s=CLOSED pgs=1968 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:49.716 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.703+0000 7f554cc6f640 1 --2- 172.21.15.16:0/3188813220 >> v2:172.21.15.62:3300/0 conn(0x7f554806c220 0x7f554806c6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:49.716 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.703+0000 7f554cc6f640 1 --2- 172.21.15.16:0/3188813220 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f55481563c0 0x7f55480c7b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:49.716 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.703+0000 7f554cc6f640 1 -- 172.21.15.16:0/3188813220 >> 172.21.15.16:0/3188813220 conn(0x7f55480b1110 msgr2=0x7f55480ce830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:46:49.716 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.703+0000 7f554cc6f640 1 -- 172.21.15.16:0/3188813220 shutdown_connections 2023-05-22T15:46:49.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.703+0000 7f554cc6f640 1 -- 172.21.15.16:0/3188813220 wait complete. 2023-05-22T15:46:49.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.703+0000 7f554cc6f640 1 Processor -- start 2023-05-22T15:46:49.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.703+0000 7f554cc6f640 1 -- start start 2023-05-22T15:46:49.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.707+0000 7f554cc6f640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f554806c220 0x7f554815e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:49.719 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.707+0000 7f554cc6f640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f55480cc6c0 0x7f55481380b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:49.720 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.707+0000 7f554cc6f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f55481534e0 0x7f5548069820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:49.720 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.707+0000 7f554cc6f640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f554815eb40 con 0x7f55481534e0 2023-05-22T15:46:49.720 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.707+0000 7f554cc6f640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f554815ee20 con 0x7f554806c220 2023-05-22T15:46:49.720 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.707+0000 7f554cc6f640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f554815f100 con 0x7f55480cc6c0 2023-05-22T15:46:49.720 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.707+0000 7f5546d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f55481534e0 0x7f5548069820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:49.720 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.707+0000 7f5546d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f55481534e0 0x7f5548069820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43388/0 (socket says 172.21.15.16:43388) 2023-05-22T15:46:49.721 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.707+0000 7f5546d9d640 1 -- 172.21.15.16:0/1015843083 learned_addr learned my addr 172.21.15.16:0/1015843083 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:46:49.721 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.707+0000 7f5545d9b640 1 --2- 172.21.15.16:0/1015843083 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f55480cc6c0 0x7f55481380b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:49.721 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.707+0000 7f554659c640 1 --2- 172.21.15.16:0/1015843083 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f554806c220 0x7f554815e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:49.721 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.707+0000 7f5546d9d640 1 -- 172.21.15.16:0/1015843083 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f55480cc6c0 msgr2=0x7f55481380b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:49.721 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.707+0000 7f5546d9d640 1 --2- 172.21.15.16:0/1015843083 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f55480cc6c0 0x7f55481380b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:49.721 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.707+0000 7f5546d9d640 1 -- 172.21.15.16:0/1015843083 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f554806c220 msgr2=0x7f554815e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:49.721 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.707+0000 7f5546d9d640 1 --2- 172.21.15.16:0/1015843083 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f554806c220 0x7f554815e2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:49.722 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.707+0000 7f5546d9d640 1 -- 172.21.15.16:0/1015843083 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f553c0094d0 con 0x7f55481534e0 2023-05-22T15:46:49.722 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.707+0000 7f5545d9b640 1 --2- 172.21.15.16:0/1015843083 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f55480cc6c0 0x7f55481380b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T15:46:49.722 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.707+0000 7f5546d9d640 1 --2- 172.21.15.16:0/1015843083 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f55481534e0 0x7f5548069820 secure :-1 s=READY pgs=1969 cs=0 l=1 rev1=1 crypto rx=0x7f5540007e40 tx=0x7f554000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:49.722 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.707+0000 7f5536ffd640 1 -- 172.21.15.16:0/1015843083 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5540014620 con 0x7f55481534e0 2023-05-22T15:46:49.722 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.707+0000 7f5536ffd640 1 -- 172.21.15.16:0/1015843083 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f554000ee90 con 0x7f55481534e0 2023-05-22T15:46:49.722 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.707+0000 7f554cc6f640 1 -- 172.21.15.16:0/1015843083 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5548070890 con 0x7f55481534e0 2023-05-22T15:46:49.722 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.707+0000 7f5536ffd640 1 -- 172.21.15.16:0/1015843083 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55400149e0 con 0x7f55481534e0 2023-05-22T15:46:49.722 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.707+0000 7f554cc6f640 1 -- 172.21.15.16:0/1015843083 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5548070700 con 0x7f55481534e0 2023-05-22T15:46:49.723 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.707+0000 7f554cc6f640 1 -- 172.21.15.16:0/1015843083 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5548070a20 con 0x7f55481534e0 2023-05-22T15:46:49.723 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.707+0000 7f5536ffd640 1 -- 172.21.15.16:0/1015843083 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 66) v1 ==== 84068+0+0 (secure 0 0 0) 0x7f5540014b40 con 0x7f55481534e0 2023-05-22T15:46:49.724 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.707+0000 7f5536ffd640 1 --2- 172.21.15.16:0/1015843083 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f55280651d0 0x7f5528067680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:49.724 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.707+0000 7f554659c640 1 --2- 172.21.15.16:0/1015843083 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f55280651d0 0x7f5528067680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:49.724 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.707+0000 7f5536ffd640 1 -- 172.21.15.16:0/1015843083 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 7055+0+0 (secure 0 0 0) 0x7f5540083260 con 0x7f55481534e0 2023-05-22T15:46:49.724 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.707+0000 7f554659c640 1 --2- 172.21.15.16:0/1015843083 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f55280651d0 0x7f5528067680 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f5548159cd0 tx=0x7f5538008040 comp rx=0 tx=0).ready entity=mgr.9396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:49.724 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.711+0000 7f5536ffd640 1 -- 172.21.15.16:0/1015843083 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f5540050e40 con 0x7f55481534e0 2023-05-22T15:46:49.903 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.899+0000 7f554cc6f640 1 -- 172.21.15.16:0/1015843083 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f554804dbe0 con 0x7f55481534e0 2023-05-22T15:46:49.904 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.899+0000 7f5536ffd640 1 -- 172.21.15.16:0/1015843083 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 58 v58) v1 ==== 94+0+4650 (secure 0 0 0) 0x7f5540054460 con 0x7f55481534e0 2023-05-22T15:46:49.904 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:46:49.905 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":58,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7753,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6834/3192074681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":3192074681},{"type":"v1","addr":"172.21.15.62:6834","nonce":3192074681}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":53},{"gid":7756,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/1344243469","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6833","nonce":1344243469},{"type":"v1","addr":"172.21.15.62:6835","nonce":1344243469}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":53},{"gid":9069,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/3636102434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":3636102434},{"type":"v1","addr":"172.21.15.16:6835","nonce":3636102434}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":53}],"filesystems":[{"mdsmap":{"epoch":58,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-05-22T15:46:23.443662+0000","modified":"2023-05-22T15:46:26.557591+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9075},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9075":{"gid":9075,"name":"a","rank":0,"incarnation":54,"state":"up:active","state_seq":3,"addr":"172.21.15.16:6837/2858899325","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":2858899325},{"type":"v1","addr":"172.21.15.16:6837","nonce":2858899325}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2023-05-22T15:46:49.905 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 58 2023-05-22T15:46:49.906 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.899+0000 7f554cc6f640 1 -- 172.21.15.16:0/1015843083 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f55280651d0 msgr2=0x7f5528067680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:49.907 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.899+0000 7f554cc6f640 1 --2- 172.21.15.16:0/1015843083 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f55280651d0 0x7f5528067680 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f5548159cd0 tx=0x7f5538008040 comp rx=0 tx=0).stop 2023-05-22T15:46:49.907 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.899+0000 7f554cc6f640 1 -- 172.21.15.16:0/1015843083 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f55481534e0 msgr2=0x7f5548069820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:49.907 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.899+0000 7f554cc6f640 1 --2- 172.21.15.16:0/1015843083 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f55481534e0 0x7f5548069820 secure :-1 s=READY pgs=1969 cs=0 l=1 rev1=1 crypto rx=0x7f5540007e40 tx=0x7f554000e4b0 comp rx=0 tx=0).stop 2023-05-22T15:46:49.907 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.899+0000 7f554cc6f640 1 -- 172.21.15.16:0/1015843083 shutdown_connections 2023-05-22T15:46:49.907 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.899+0000 7f554cc6f640 1 --2- 172.21.15.16:0/1015843083 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f55481534e0 0x7f5548069820 unknown :-1 s=CLOSED pgs=1969 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:49.907 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.899+0000 7f554cc6f640 1 --2- 172.21.15.16:0/1015843083 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f55280651d0 0x7f5528067680 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:49.907 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.899+0000 7f554cc6f640 1 --2- 172.21.15.16:0/1015843083 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f55480cc6c0 0x7f55481380b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:49.908 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.899+0000 7f554cc6f640 1 --2- 172.21.15.16:0/1015843083 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f554806c220 0x7f554815e2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:49.908 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.899+0000 7f554cc6f640 1 -- 172.21.15.16:0/1015843083 >> 172.21.15.16:0/1015843083 conn(0x7f55480b1110 msgr2=0x7f554814ee10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:46:49.908 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.899+0000 7f554cc6f640 1 -- 172.21.15.16:0/1015843083 shutdown_connections 2023-05-22T15:46:49.908 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:49.899+0000 7f554cc6f640 1 -- 172.21.15.16:0/1015843083 wait complete. 2023-05-22T15:46:49.922 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-05-22T15:46:50.037 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:50.031+0000 7f0658f47640 1 Processor -- start 2023-05-22T15:46:50.038 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:50.031+0000 7f0658f47640 1 -- start start 2023-05-22T15:46:50.038 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:50.031+0000 7f0658f47640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f0654158a10 0x7f065406e480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:50.038 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:50.031+0000 7f0658f47640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0654158de0 con 0x7f0654070760 2023-05-22T15:46:50.039 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:50.031+0000 7f0658f47640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06540711e0 con 0x7f065406cdf0 2023-05-22T15:46:50.039 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:50.031+0000 7f0658f47640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f065406e060 con 0x7f0654158a10 2023-05-22T15:46:50.039 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:50.031+0000 7f0651d9b640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f0654070760 0x7f065406c6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:55542/0 (socket says 172.21.15.16:55542) 2023-05-22T15:46:50.039 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:50.031+0000 7f0651d9b640 1 -- 172.21.15.16:0/3615588091 learned_addr learned my addr 172.21.15.16:0/3615588091 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:46:50.039 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:50.031+0000 7f065259c640 1 --2- 172.21.15.16:0/3615588091 >> v2:172.21.15.62:3300/0 conn(0x7f0654158a10 0x7f065406e480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:50.040 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:50.035+0000 7f065159a640 1 -- 172.21.15.16:0/3615588091 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3373594448 0 0) 0x7f0654158de0 con 0x7f0654070760 2023-05-22T15:46:50.040 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:50.035+0000 7f065159a640 1 -- 172.21.15.16:0/3615588091 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f063c003530 con 0x7f0654070760 2023-05-22T15:46:50.040 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:50.035+0000 7f065159a640 1 -- 172.21.15.16:0/3615588091 <== mon.1 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 302852986 0 0) 0x7f06540711e0 con 0x7f065406cdf0 2023-05-22T15:46:50.041 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:50.035+0000 7f065159a640 1 -- 172.21.15.16:0/3615588091 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0654158de0 con 0x7f065406cdf0 2023-05-22T15:46:50.041 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:50.035+0000 7f065259c640 1 -- 172.21.15.16:0/3615588091 >> v1:172.21.15.62:6789/0 conn(0x7f065406cdf0 legacy=0x7f06540c9a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:50.041 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:50.035+0000 7f065259c640 1 -- 172.21.15.16:0/3615588091 >> v1:172.21.15.16:6789/0 conn(0x7f0654070760 legacy=0x7f065406c6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:50.041 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:50.035+0000 7f065259c640 1 -- 172.21.15.16:0/3615588091 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06540714c0 con 0x7f0654158a10 2023-05-22T15:46:50.042 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:50.035+0000 7f065259c640 1 --2- 172.21.15.16:0/3615588091 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0654158a10 0x7f065406e480 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f06480097c0 tx=0x7f064801c3f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e19f129f9e86ba2c server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:50.042 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:50.035+0000 7f065159a640 1 -- 172.21.15.16:0/3615588091 <== mon.1 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06480ab600 con 0x7f0654158a10 2023-05-22T15:46:50.042 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:50.035+0000 7f065159a640 1 -- 172.21.15.16:0/3615588091 <== mon.1 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06480a7050 con 0x7f0654158a10 2023-05-22T15:46:50.042 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:50.035+0000 7f065159a640 1 -- 172.21.15.16:0/3615588091 <== mon.1 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06480ab960 con 0x7f0654158a10 2023-05-22T15:46:50.043 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:50.035+0000 7f0658f47640 1 -- 172.21.15.16:0/3615588091 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0654158a10 msgr2=0x7f065406e480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:50.043 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:50.035+0000 7f0658f47640 1 --2- 172.21.15.16:0/3615588091 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0654158a10 0x7f065406e480 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f06480097c0 tx=0x7f064801c3f0 comp rx=0 tx=0).stop 2023-05-22T15:46:50.043 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:50.035+0000 7f0658f47640 1 -- 172.21.15.16:0/3615588091 shutdown_connections 2023-05-22T15:46:50.043 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:50.035+0000 7f0658f47640 1 --2- 172.21.15.16:0/3615588091 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0654158a10 0x7f065406e480 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:50.043 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:50.035+0000 7f0658f47640 1 -- 172.21.15.16:0/3615588091 >> 172.21.15.16:0/3615588091 conn(0x7f06540cc870 msgr2=0x7f06540cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:46:50.044 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:50.035+0000 7f0658f47640 1 -- 172.21.15.16:0/3615588091 shutdown_connections 2023-05-22T15:46:50.044 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:50.035+0000 7f0658f47640 1 -- 172.21.15.16:0/3615588091 wait complete. 2023-05-22T15:46:50.045 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:50.039+0000 7f0658f47640 1 Processor -- start 2023-05-22T15:46:50.045 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:50.039+0000 7f0658f47640 1 -- start start 2023-05-22T15:46:50.046 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:50.039+0000 7f0658f47640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f065406cdf0 0x7f06540b42e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:50.046 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:50.039+0000 7f0658f47640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0654070760 0x7f065415a050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:50.046 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:50.039+0000 7f0658f47640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0654158a10 0x7f065415a590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:50.046 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:50.039+0000 7f0658f47640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06541534f0 con 0x7f0654158a10 2023-05-22T15:46:50.046 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:50.039+0000 7f0658f47640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06540c89f0 con 0x7f0654070760 2023-05-22T15:46:50.046 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:50.039+0000 7f0658f47640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0654153210 con 0x7f065406cdf0 2023-05-22T15:46:50.047 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:50.039+0000 7f0652d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0654158a10 0x7f065415a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:50.047 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:50.039+0000 7f0652d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0654158a10 0x7f065415a590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43396/0 (socket says 172.21.15.16:43396) 2023-05-22T15:46:50.047 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:50.039+0000 7f0652d9d640 1 -- 172.21.15.16:0/285697005 learned_addr learned my addr 172.21.15.16:0/285697005 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:46:50.047 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:50.039+0000 7f0651d9b640 1 --2- 172.21.15.16:0/285697005 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0654070760 0x7f065415a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:50.047 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:50.039+0000 7f065259c640 1 --2- 172.21.15.16:0/285697005 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f065406cdf0 0x7f06540b42e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:50.047 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:50.039+0000 7f0652d9d640 1 -- 172.21.15.16:0/285697005 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f065406cdf0 msgr2=0x7f06540b42e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:50.047 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:50.039+0000 7f0652d9d640 1 --2- 172.21.15.16:0/285697005 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f065406cdf0 0x7f06540b42e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:50.048 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:50.039+0000 7f0652d9d640 1 -- 172.21.15.16:0/285697005 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0654070760 msgr2=0x7f065415a050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:50.048 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:50.039+0000 7f0652d9d640 1 --2- 172.21.15.16:0/285697005 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0654070760 0x7f065415a050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:50.048 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:50.039+0000 7f0652d9d640 1 -- 172.21.15.16:0/285697005 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0648009420 con 0x7f0654158a10 2023-05-22T15:46:50.048 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:50.039+0000 7f0652d9d640 1 --2- 172.21.15.16:0/285697005 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0654158a10 0x7f065415a590 secure :-1 s=READY pgs=1971 cs=0 l=1 rev1=1 crypto rx=0x7f064c00c730 tx=0x7f064c00cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:50.049 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:50.039+0000 7f0642ffd640 1 -- 172.21.15.16:0/285697005 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f064c003b80 con 0x7f0654158a10 2023-05-22T15:46:50.049 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:50.039+0000 7f0642ffd640 1 -- 172.21.15.16:0/285697005 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f064c003ce0 con 0x7f0654158a10 2023-05-22T15:46:50.049 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:50.039+0000 7f0658f47640 1 -- 172.21.15.16:0/285697005 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f065406e9c0 con 0x7f0654158a10 2023-05-22T15:46:50.049 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:50.039+0000 7f0642ffd640 1 -- 172.21.15.16:0/285697005 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f064c005310 con 0x7f0654158a10 2023-05-22T15:46:50.049 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:50.039+0000 7f0658f47640 1 -- 172.21.15.16:0/285697005 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06540ca590 con 0x7f0654158a10 2023-05-22T15:46:50.050 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:50.039+0000 7f0658f47640 1 -- 172.21.15.16:0/285697005 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f065406eb50 con 0x7f0654158a10 2023-05-22T15:46:50.050 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:50.039+0000 7f0642ffd640 1 -- 172.21.15.16:0/285697005 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 66) v1 ==== 84068+0+0 (secure 0 0 0) 0x7f064c005680 con 0x7f0654158a10 2023-05-22T15:46:50.050 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:50.043+0000 7f0642ffd640 1 --2- 172.21.15.16:0/285697005 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f06340651d0 0x7f0634067680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:50.051 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:50.043+0000 7f065259c640 1 --2- 172.21.15.16:0/285697005 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f06340651d0 0x7f0634067680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:50.051 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:50.043+0000 7f0642ffd640 1 -- 172.21.15.16:0/285697005 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 7055+0+0 (secure 0 0 0) 0x7f064c082850 con 0x7f0654158a10 2023-05-22T15:46:50.051 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:50.043+0000 7f065259c640 1 --2- 172.21.15.16:0/285697005 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f06340651d0 0x7f0634067680 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f06480ad000 tx=0x7f06480a4f90 comp rx=0 tx=0).ready entity=mgr.9396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:50.051 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:50.047+0000 7f0642ffd640 1 -- 172.21.15.16:0/285697005 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f064c050430 con 0x7f0654158a10 2023-05-22T15:46:50.205 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:50.199+0000 7f0658f47640 1 -- 172.21.15.16:0/285697005 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f06540b4000 con 0x7f0654158a10 2023-05-22T15:46:50.207 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:50.203+0000 7f0642ffd640 1 -- 172.21.15.16:0/285697005 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v149) v1 ==== 74+0+19019 (secure 0 0 0) 0x7f06540b4000 con 0x7f0654158a10 2023-05-22T15:46:50.208 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:46:50.208 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":149,"fsid":"b2719834-7847-4d98-90a6-f65d57f85c49","created":"2023-05-22T15:26:25.259314+0000","modified":"2023-05-22T15:46:47.687666+0000","last_up_change":"2023-05-22T15:26:36.448524+0000","last_in_change":"2023-05-22T15:26:27.675668+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-05-22T15:26:38.019781+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-05-22T15:26:43.570514+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2023-05-22T15:46:21.084615+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"146","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_data","create_time":"2023-05-22T15:46:22.440577+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"147","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9e7ff854-b25f-4cbb-a5c2-1799d9c53d2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":37021},{"type":"v1","addr":"172.21.15.16:6825","nonce":37021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":37021},{"type":"v1","addr":"172.21.15.16:6827","nonce":37021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":37021},{"type":"v1","addr":"172.21.15.16:6831","nonce":37021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":37021},{"type":"v1","addr":"172.21.15.16:6829","nonce":37021}]},"public_addr":"172.21.15.16:6825/37021","cluster_addr":"172.21.15.16:6827/37021","heartbeat_back_addr":"172.21.15.16:6831/37021","heartbeat_front_addr":"172.21.15.16:6829/37021","state":["exists","up"]},{"osd":1,"uuid":"4702fbc9-a927-4565-aca5-41dc69cc8264","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":37008},{"type":"v1","addr":"172.21.15.16:6801","nonce":37008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":37008},{"type":"v1","addr":"172.21.15.16:6803","nonce":37008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":37008},{"type":"v1","addr":"172.21.15.16:6807","nonce":37008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":37008},{"type":"v1","addr":"172.21.15.16:6805","nonce":37008}]},"public_addr":"172.21.15.16:6801/37008","cluster_addr":"172.21.15.16:6803/37008","heartbeat_back_addr":"172.21.15.16:6807/37008","heartbeat_front_addr":"172.21.15.16:6805/37008","state":["exists","up"]},{"osd":2,"uuid":"1e7bba46-4eba-47ef-bb81-5e2d371638ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":37009},{"type":"v1","addr":"172.21.15.16:6809","nonce":37009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":37009},{"type":"v1","addr":"172.21.15.16:6811","nonce":37009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":37009},{"type":"v1","addr":"172.21.15.16:6815","nonce":37009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":37009},{"type":"v1","addr":"172.21.15.16:6813","nonce":37009}]},"public_addr":"172.21.15.16:6809/37009","cluster_addr":"172.21.15.16:6811/37009","heartbeat_back_addr":"172.21.15.16:6815/37009","heartbeat_front_addr":"172.21.15.16:6813/37009","state":["exists","up"]},{"osd":3,"uuid":"383cacc0-5a88-4904-990e-498ca59ef48a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":37020},{"type":"v1","addr":"172.21.15.16:6817","nonce":37020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":37020},{"type":"v1","addr":"172.21.15.16:6819","nonce":37020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":37020},{"type":"v1","addr":"172.21.15.16:6823","nonce":37020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":37020},{"type":"v1","addr":"172.21.15.16:6821","nonce":37020}]},"public_addr":"172.21.15.16:6817/37020","cluster_addr":"172.21.15.16:6819/37020","heartbeat_back_addr":"172.21.15.16:6823/37020","heartbeat_front_addr":"172.21.15.16:6821/37020","state":["exists","up"]},{"osd":4,"uuid":"970500ef-95b7-42c1-b080-eed4fae7013d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":37353},{"type":"v1","addr":"172.21.15.62:6817","nonce":37353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6819","nonce":37353},{"type":"v1","addr":"172.21.15.62:6822","nonce":37353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37353},{"type":"v1","addr":"172.21.15.62:6831","nonce":37353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37353},{"type":"v1","addr":"172.21.15.62:6826","nonce":37353}]},"public_addr":"172.21.15.62:6817/37353","cluster_addr":"172.21.15.62:6822/37353","heartbeat_back_addr":"172.21.15.62:6831/37353","heartbeat_front_addr":"172.21.15.62:6826/37353","state":["exists","up"]},{"osd":5,"uuid":"022d67db-d4d8-445b-b1b9-8b446a00f5c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":37352},{"type":"v1","addr":"172.21.15.62:6801","nonce":37352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":37352},{"type":"v1","addr":"172.21.15.62:6803","nonce":37352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":37352},{"type":"v1","addr":"172.21.15.62:6807","nonce":37352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":37352},{"type":"v1","addr":"172.21.15.62:6805","nonce":37352}]},"public_addr":"172.21.15.62:6801/37352","cluster_addr":"172.21.15.62:6803/37352","heartbeat_back_addr":"172.21.15.62:6807/37352","heartbeat_front_addr":"172.21.15.62:6805/37352","state":["exists","up"]},{"osd":6,"uuid":"42bd372b-6688-4a1b-8407-e16e8c266e33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":37354},{"type":"v1","addr":"172.21.15.62:6820","nonce":37354}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":37354},{"type":"v1","addr":"172.21.15.62:6823","nonce":37354}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6829","nonce":37354},{"type":"v1","addr":"172.21.15.62:6830","nonce":37354}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6825","nonce":37354},{"type":"v1","addr":"172.21.15.62:6827","nonce":37354}]},"public_addr":"172.21.15.62:6820/37354","cluster_addr":"172.21.15.62:6823/37354","heartbeat_back_addr":"172.21.15.62:6830/37354","heartbeat_front_addr":"172.21.15.62:6827/37354","state":["exists","up"]},{"osd":7,"uuid":"ac70b2a3-f41b-4bf2-b227-7f2cbb55426d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37355},{"type":"v1","addr":"172.21.15.62:6809","nonce":37355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37355},{"type":"v1","addr":"172.21.15.62:6811","nonce":37355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":37355},{"type":"v1","addr":"172.21.15.62:6815","nonce":37355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":37355},{"type":"v1","addr":"172.21.15.62:6813","nonce":37355}]},"public_addr":"172.21.15.62:6809/37355","cluster_addr":"172.21.15.62:6811/37355","heartbeat_back_addr":"172.21.15.62:6815/37355","heartbeat_front_addr":"172.21.15.62:6813/37355","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.368896+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.107273+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.276888+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.287502+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.124791+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.251937+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.255679+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.232797+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:0/2660771812":"2023-05-23T15:46:47.687617+0000","172.21.15.16:0/2077740732":"2023-05-23T15:46:47.687617+0000","172.21.15.16:0/2030430759":"2023-05-23T15:46:47.687617+0000","172.21.15.16:0/4220632598":"2023-05-23T15:46:47.687617+0000","172.21.15.16:0/2231972099":"2023-05-23T15:46:47.687617+0000","172.21.15.16:0/3429069808":"2023-05-23T15:46:41.068549+0000","172.21.15.16:0/2806812328":"2023-05-23T15:46:47.687617+0000","172.21.15.16:0/4084148691":"2023-05-23T15:46:41.068549+0000","172.21.15.16:0/3718143920":"2023-05-23T15:46:47.687617+0000","172.21.15.16:0/3605381301":"2023-05-23T15:46:41.068549+0000","172.21.15.16:0/2954477712":"2023-05-23T15:46:41.068549+0000","172.21.15.16:0/3271028793":"2023-05-23T15:46:41.068549+0000","172.21.15.16:6833/36921":"2023-05-23T15:46:41.068549+0000","172.21.15.16:6832/36921":"2023-05-23T15:46:41.068549+0000","172.21.15.16:0/1944100655":"2023-05-23T15:46:41.068549+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":17,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-05-22T15:46:50.209 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:50.203+0000 7f0658f47640 1 -- 172.21.15.16:0/285697005 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f06340651d0 msgr2=0x7f0634067680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:50.210 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:50.203+0000 7f0658f47640 1 --2- 172.21.15.16:0/285697005 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f06340651d0 0x7f0634067680 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f06480ad000 tx=0x7f06480a4f90 comp rx=0 tx=0).stop 2023-05-22T15:46:50.210 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:50.203+0000 7f0658f47640 1 -- 172.21.15.16:0/285697005 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0654158a10 msgr2=0x7f065415a590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:50.210 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:50.203+0000 7f0658f47640 1 --2- 172.21.15.16:0/285697005 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0654158a10 0x7f065415a590 secure :-1 s=READY pgs=1971 cs=0 l=1 rev1=1 crypto rx=0x7f064c00c730 tx=0x7f064c00cbf0 comp rx=0 tx=0).stop 2023-05-22T15:46:50.210 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:50.203+0000 7f0658f47640 1 -- 172.21.15.16:0/285697005 shutdown_connections 2023-05-22T15:46:50.210 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:50.203+0000 7f0658f47640 1 --2- 172.21.15.16:0/285697005 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0654158a10 0x7f065415a590 unknown :-1 s=CLOSED pgs=1971 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:50.210 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:50.203+0000 7f0658f47640 1 --2- 172.21.15.16:0/285697005 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0654070760 0x7f065415a050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:50.210 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:50.203+0000 7f0658f47640 1 --2- 172.21.15.16:0/285697005 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f06340651d0 0x7f0634067680 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:50.211 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:50.203+0000 7f0658f47640 1 --2- 172.21.15.16:0/285697005 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f065406cdf0 0x7f06540b42e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:50.211 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:50.203+0000 7f0658f47640 1 -- 172.21.15.16:0/285697005 >> 172.21.15.16:0/285697005 conn(0x7f06540cc870 msgr2=0x7f06540cf140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:46:50.211 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:50.203+0000 7f0658f47640 1 -- 172.21.15.16:0/285697005 shutdown_connections 2023-05-22T15:46:50.211 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:50.203+0000 7f0658f47640 1 -- 172.21.15.16:0/285697005 wait complete. 2023-05-22T15:46:50.226 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-05-22T15:46:50.352 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:50.347+0000 7f07312a9640 1 Processor -- start 2023-05-22T15:46:50.352 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:50.347+0000 7f07312a9640 1 -- start start 2023-05-22T15:46:50.353 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:50.347+0000 7f07312a9640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f072c158a10 0x7f072c06e480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:50.353 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:50.347+0000 7f07312a9640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f072c06cdf0 0x7f072c070be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:50.353 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:50.347+0000 7f07312a9640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f072c158de0 con 0x7f072c070760 2023-05-22T15:46:50.353 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:50.347+0000 7f07312a9640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f072c06e060 con 0x7f072c158a10 2023-05-22T15:46:50.353 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:50.347+0000 7f07312a9640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f072c0711e0 con 0x7f072c06cdf0 2023-05-22T15:46:50.354 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:50.347+0000 7f072affd640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f072c158a10 0x7f072c06e480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:50.354 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:50.347+0000 7f072affd640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f072c158a10 0x7f072c06e480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43400/0 (socket says 172.21.15.16:43400) 2023-05-22T15:46:50.354 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:50.347+0000 7f072affd640 1 -- 172.21.15.16:0/1928014753 learned_addr learned my addr 172.21.15.16:0/1928014753 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:46:50.355 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:50.347+0000 7f072b7fe640 1 --2- 172.21.15.16:0/1928014753 >> v2:172.21.15.62:3300/0 conn(0x7f072c06cdf0 0x7f072c070be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:50.355 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:50.347+0000 7f072affd640 1 -- 172.21.15.16:0/1928014753 >> v2:172.21.15.62:3300/0 conn(0x7f072c06cdf0 msgr2=0x7f072c070be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:46:50.355 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:50.347+0000 7f072affd640 1 --2- 172.21.15.16:0/1928014753 >> v2:172.21.15.62:3300/0 conn(0x7f072c06cdf0 0x7f072c070be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:50.355 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:50.347+0000 7f072affd640 1 -- 172.21.15.16:0/1928014753 >> v1:172.21.15.62:6789/0 conn(0x7f072c070760 legacy=0x7f072c06c6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:46:50.355 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:50.347+0000 7f072affd640 1 -- 172.21.15.16:0/1928014753 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f072c0714c0 con 0x7f072c158a10 2023-05-22T15:46:50.355 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:50.347+0000 7f072b7fe640 1 --2- 172.21.15.16:0/1928014753 >> v2:172.21.15.62:3300/0 conn(0x7f072c06cdf0 0x7f072c070be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:46:50.356 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:50.351+0000 7f072affd640 1 --2- 172.21.15.16:0/1928014753 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f072c158a10 0x7f072c06e480 secure :-1 s=READY pgs=1972 cs=0 l=1 rev1=1 crypto rx=0x7f072c06d1c0 tx=0x7f072401c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d065c3aa50b46ab server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:50.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:50.351+0000 7f0729ffb640 1 -- 172.21.15.16:0/1928014753 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07240a57e0 con 0x7f072c158a10 2023-05-22T15:46:50.358 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:50.351+0000 7f0729ffb640 1 -- 172.21.15.16:0/1928014753 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07240a5940 con 0x7f072c158a10 2023-05-22T15:46:50.358 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:50.351+0000 7f0729ffb640 1 -- 172.21.15.16:0/1928014753 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07240a5cd0 con 0x7f072c158a10 2023-05-22T15:46:50.358 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:50.351+0000 7f07312a9640 1 -- 172.21.15.16:0/1928014753 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f072c158a10 msgr2=0x7f072c06e480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:50.359 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:50.351+0000 7f07312a9640 1 --2- 172.21.15.16:0/1928014753 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f072c158a10 0x7f072c06e480 secure :-1 s=READY pgs=1972 cs=0 l=1 rev1=1 crypto rx=0x7f072c06d1c0 tx=0x7f072401c900 comp rx=0 tx=0).stop 2023-05-22T15:46:50.359 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:50.351+0000 7f07312a9640 1 -- 172.21.15.16:0/1928014753 shutdown_connections 2023-05-22T15:46:50.359 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:50.351+0000 7f07312a9640 1 --2- 172.21.15.16:0/1928014753 >> v2:172.21.15.62:3300/0 conn(0x7f072c06cdf0 0x7f072c070be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:50.360 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:50.351+0000 7f07312a9640 1 --2- 172.21.15.16:0/1928014753 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f072c158a10 0x7f072c06e480 unknown :-1 s=CLOSED pgs=1972 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:50.360 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:50.351+0000 7f07312a9640 1 -- 172.21.15.16:0/1928014753 >> 172.21.15.16:0/1928014753 conn(0x7f072c0cc870 msgr2=0x7f072c0cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:46:50.360 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:50.351+0000 7f07312a9640 1 -- 172.21.15.16:0/1928014753 shutdown_connections 2023-05-22T15:46:50.360 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:50.351+0000 7f07312a9640 1 -- 172.21.15.16:0/1928014753 wait complete. 2023-05-22T15:46:50.364 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:50.351+0000 7f07312a9640 1 Processor -- start 2023-05-22T15:46:50.364 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:50.351+0000 7f07312a9640 1 -- start start 2023-05-22T15:46:50.364 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:50.355+0000 7f07312a9640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f072c06cdf0 0x7f072c14da40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:50.365 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:50.355+0000 7f07312a9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f072c070760 0x7f072c14f890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:50.365 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:50.355+0000 7f07312a9640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f072c158a10 0x7f072c1465f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:50.365 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:50.355+0000 7f07312a9640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f072c14e020 con 0x7f072c070760 2023-05-22T15:46:50.365 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:50.355+0000 7f07312a9640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f072c14e300 con 0x7f072c06cdf0 2023-05-22T15:46:50.365 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:50.355+0000 7f07312a9640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f072c14e5e0 con 0x7f072c158a10 2023-05-22T15:46:50.365 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:50.355+0000 7f072a7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f072c070760 0x7f072c14f890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:50.365 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:50.355+0000 7f072a7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f072c070760 0x7f072c14f890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43406/0 (socket says 172.21.15.16:43406) 2023-05-22T15:46:50.365 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:50.355+0000 7f072a7fc640 1 -- 172.21.15.16:0/1967836203 learned_addr learned my addr 172.21.15.16:0/1967836203 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:46:50.366 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:50.355+0000 7f072affd640 1 --2- 172.21.15.16:0/1967836203 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f072c06cdf0 0x7f072c14da40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:50.366 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:50.355+0000 7f072b7fe640 1 --2- 172.21.15.16:0/1967836203 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f072c158a10 0x7f072c1465f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:50.366 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:50.355+0000 7f072a7fc640 1 -- 172.21.15.16:0/1967836203 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f072c158a10 msgr2=0x7f072c1465f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:50.366 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:50.355+0000 7f072a7fc640 1 --2- 172.21.15.16:0/1967836203 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f072c158a10 0x7f072c1465f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:50.366 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:50.355+0000 7f072a7fc640 1 -- 172.21.15.16:0/1967836203 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f072c06cdf0 msgr2=0x7f072c14da40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:50.366 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:50.355+0000 7f072a7fc640 1 --2- 172.21.15.16:0/1967836203 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f072c06cdf0 0x7f072c14da40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:50.367 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:50.355+0000 7f072a7fc640 1 -- 172.21.15.16:0/1967836203 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0724009b30 con 0x7f072c070760 2023-05-22T15:46:50.367 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:50.355+0000 7f072affd640 1 --2- 172.21.15.16:0/1967836203 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f072c06cdf0 0x7f072c14da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:46:50.367 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:50.355+0000 7f072b7fe640 1 --2- 172.21.15.16:0/1967836203 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f072c158a10 0x7f072c1465f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:46:50.367 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:50.355+0000 7f072a7fc640 1 --2- 172.21.15.16:0/1967836203 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f072c070760 0x7f072c14f890 secure :-1 s=READY pgs=1973 cs=0 l=1 rev1=1 crypto rx=0x7f071c00c450 tx=0x7f071c00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:50.367 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:50.355+0000 7f07237fe640 1 -- 172.21.15.16:0/1967836203 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f071c012830 con 0x7f072c070760 2023-05-22T15:46:50.367 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:50.355+0000 7f07237fe640 1 -- 172.21.15.16:0/1967836203 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f071c0129b0 con 0x7f072c070760 2023-05-22T15:46:50.367 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:50.355+0000 7f07237fe640 1 -- 172.21.15.16:0/1967836203 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f071c012d60 con 0x7f072c070760 2023-05-22T15:46:50.368 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:50.355+0000 7f07312a9640 1 -- 172.21.15.16:0/1967836203 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f072c06eb50 con 0x7f072c070760 2023-05-22T15:46:50.368 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:50.355+0000 7f07312a9640 1 -- 172.21.15.16:0/1967836203 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f072c06e9c0 con 0x7f072c070760 2023-05-22T15:46:50.368 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:50.355+0000 7f07312a9640 1 -- 172.21.15.16:0/1967836203 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f072c0ca280 con 0x7f072c070760 2023-05-22T15:46:50.368 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:50.355+0000 7f07237fe640 1 -- 172.21.15.16:0/1967836203 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 66) v1 ==== 84068+0+0 (secure 0 0 0) 0x7f071c01c460 con 0x7f072c070760 2023-05-22T15:46:50.368 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:50.355+0000 7f07237fe640 1 --2- 172.21.15.16:0/1967836203 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f070c0651d0 0x7f070c067680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:50.368 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:50.355+0000 7f07237fe640 1 -- 172.21.15.16:0/1967836203 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 7055+0+0 (secure 0 0 0) 0x7f071c082780 con 0x7f072c070760 2023-05-22T15:46:50.369 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:50.355+0000 7f072affd640 1 --2- 172.21.15.16:0/1967836203 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f070c0651d0 0x7f070c067680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:50.369 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:50.359+0000 7f072affd640 1 --2- 172.21.15.16:0/1967836203 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f070c0651d0 0x7f070c067680 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f07240ae000 tx=0x7f07240a7070 comp rx=0 tx=0).ready entity=mgr.9396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:50.369 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:50.359+0000 7f07237fe640 1 -- 172.21.15.16:0/1967836203 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f071c050230 con 0x7f072c070760 2023-05-22T15:46:50.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:50.539+0000 7f07312a9640 1 -- 172.21.15.16:0/1967836203 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f072c04dbe0 con 0x7f072c070760 2023-05-22T15:46:50.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:50.723+0000 7f07237fe640 1 -- 172.21.15.16:0/1967836203 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mgrmap(e 67) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f071c050600 con 0x7f072c070760 2023-05-22T15:46:51.127 INFO:tasks.ceph.mon.a.smithi016.stderr:2023-05-22T15:46:51.123+0000 7f95dd48a640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-05-22T15:46:51.145 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:51.139+0000 7f07237fe640 1 -- 172.21.15.16:0/1967836203 <== mon.0 v2:172.21.15.16:3300/0 8 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v59) v1 ==== 161+0+0 (secure 0 0 0) 0x7f071c050230 con 0x7f072c070760 2023-05-22T15:46:51.146 INFO:teuthology.orchestra.run.smithi016.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-05-22T15:46:51.149 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:51.143+0000 7f07312a9640 1 -- 172.21.15.16:0/1967836203 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f070c0651d0 msgr2=0x7f070c067680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:51.149 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:51.143+0000 7f07312a9640 1 --2- 172.21.15.16:0/1967836203 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f070c0651d0 0x7f070c067680 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f07240ae000 tx=0x7f07240a7070 comp rx=0 tx=0).stop 2023-05-22T15:46:51.149 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:51.143+0000 7f07312a9640 1 -- 172.21.15.16:0/1967836203 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f072c070760 msgr2=0x7f072c14f890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:51.150 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:51.143+0000 7f07312a9640 1 --2- 172.21.15.16:0/1967836203 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f072c070760 0x7f072c14f890 secure :-1 s=READY pgs=1973 cs=0 l=1 rev1=1 crypto rx=0x7f071c00c450 tx=0x7f071c00c910 comp rx=0 tx=0).stop 2023-05-22T15:46:51.150 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:51.143+0000 7f07312a9640 1 -- 172.21.15.16:0/1967836203 shutdown_connections 2023-05-22T15:46:51.150 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:51.143+0000 7f07312a9640 1 --2- 172.21.15.16:0/1967836203 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f070c0651d0 0x7f070c067680 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:51.151 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:51.143+0000 7f07312a9640 1 --2- 172.21.15.16:0/1967836203 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f072c158a10 0x7f072c1465f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:51.151 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:51.143+0000 7f07312a9640 1 --2- 172.21.15.16:0/1967836203 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f072c070760 0x7f072c14f890 unknown :-1 s=CLOSED pgs=1973 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:51.151 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:51.143+0000 7f07312a9640 1 --2- 172.21.15.16:0/1967836203 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f072c06cdf0 0x7f072c14da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:51.152 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:51.143+0000 7f07312a9640 1 -- 172.21.15.16:0/1967836203 >> 172.21.15.16:0/1967836203 conn(0x7f072c0cc870 msgr2=0x7f072c0cd840 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:46:51.152 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:51.143+0000 7f07312a9640 1 -- 172.21.15.16:0/1967836203 shutdown_connections 2023-05-22T15:46:51.152 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:51.143+0000 7f07312a9640 1 -- 172.21.15.16:0/1967836203 wait complete. 2023-05-22T15:46:51.153 INFO:tasks.ceph.mds.a.smithi016.stderr: -2810> 2023-05-22T15:46:28.911+0000 7fbb12d11640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-05-22T15:46:51.169 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-05-22T15:46:51.195 INFO:tasks.ceph.mds.a.smithi016.stdout:starting mds.a at 2023-05-22T15:46:51.293 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:51.287+0000 7f5c59159640 1 Processor -- start 2023-05-22T15:46:51.294 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:51.287+0000 7f5c59159640 1 -- start start 2023-05-22T15:46:51.295 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:51.287+0000 7f5c59159640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f5c54071530 0x7f5c5415c580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:51.295 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:51.287+0000 7f5c59159640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f5c5406b6c0 0x7f5c5406bac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:51.295 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:51.287+0000 7f5c59159640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c5415d520 con 0x7f5c54070c70 2023-05-22T15:46:51.295 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:51.287+0000 7f5c59159640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c54071980 con 0x7f5c54071530 2023-05-22T15:46:51.295 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:51.287+0000 7f5c59159640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c5406c000 con 0x7f5c5406b6c0 2023-05-22T15:46:51.295 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:51.287+0000 7f5c5259c640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f5c54071530 0x7f5c5415c580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:51.296 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:51.287+0000 7f5c5259c640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f5c54071530 0x7f5c5415c580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43440/0 (socket says 172.21.15.16:43440) 2023-05-22T15:46:51.296 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:51.287+0000 7f5c5259c640 1 -- 172.21.15.16:0/2533519757 learned_addr learned my addr 172.21.15.16:0/2533519757 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:46:51.296 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:51.287+0000 7f5c5359e640 1 --2- 172.21.15.16:0/2533519757 >> v2:172.21.15.62:3300/0 conn(0x7f5c5406b6c0 0x7f5c5406bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:51.296 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:51.287+0000 7f5c5259c640 1 -- 172.21.15.16:0/2533519757 >> v2:172.21.15.62:3300/0 conn(0x7f5c5406b6c0 msgr2=0x7f5c5406bac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:46:51.296 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:51.287+0000 7f5c5259c640 1 --2- 172.21.15.16:0/2533519757 >> v2:172.21.15.62:3300/0 conn(0x7f5c5406b6c0 0x7f5c5406bac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:51.296 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:51.287+0000 7f5c5259c640 1 -- 172.21.15.16:0/2533519757 >> v1:172.21.15.16:6789/0 conn(0x7f5c54070c70 legacy=0x7f5c5406fba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:51.296 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:51.287+0000 7f5c5259c640 1 -- 172.21.15.16:0/2533519757 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c540c8b10 con 0x7f5c54071530 2023-05-22T15:46:51.297 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:51.291+0000 7f5c5259c640 1 --2- 172.21.15.16:0/2533519757 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5c54071530 0x7f5c5415c580 secure :-1 s=READY pgs=1976 cs=0 l=1 rev1=1 crypto rx=0x7f5c540702b0 tx=0x7f5c4801c470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7a0171160181da3 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:51.298 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:51.291+0000 7f5c51d9b640 1 -- 172.21.15.16:0/2533519757 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c480ab670 con 0x7f5c54071530 2023-05-22T15:46:51.298 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:51.291+0000 7f5c51d9b640 1 -- 172.21.15.16:0/2533519757 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c480ab7d0 con 0x7f5c54071530 2023-05-22T15:46:51.298 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:51.291+0000 7f5c51d9b640 1 -- 172.21.15.16:0/2533519757 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c480abb60 con 0x7f5c54071530 2023-05-22T15:46:51.298 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:51.291+0000 7f5c59159640 1 -- 172.21.15.16:0/2533519757 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5c54071530 msgr2=0x7f5c5415c580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:51.298 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:51.291+0000 7f5c59159640 1 --2- 172.21.15.16:0/2533519757 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5c54071530 0x7f5c5415c580 secure :-1 s=READY pgs=1976 cs=0 l=1 rev1=1 crypto rx=0x7f5c540702b0 tx=0x7f5c4801c470 comp rx=0 tx=0).stop 2023-05-22T15:46:51.299 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:51.291+0000 7f5c59159640 1 -- 172.21.15.16:0/2533519757 shutdown_connections 2023-05-22T15:46:51.299 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:51.291+0000 7f5c59159640 1 --2- 172.21.15.16:0/2533519757 >> v2:172.21.15.62:3300/0 conn(0x7f5c5406b6c0 0x7f5c5406bac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:51.299 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:51.291+0000 7f5c59159640 1 --2- 172.21.15.16:0/2533519757 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5c54071530 0x7f5c5415c580 unknown :-1 s=CLOSED pgs=1976 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:51.299 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:51.291+0000 7f5c59159640 1 -- 172.21.15.16:0/2533519757 >> 172.21.15.16:0/2533519757 conn(0x7f5c54069720 msgr2=0x7f5c54069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:46:51.299 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:51.291+0000 7f5c59159640 1 -- 172.21.15.16:0/2533519757 shutdown_connections 2023-05-22T15:46:51.299 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:51.291+0000 7f5c59159640 1 -- 172.21.15.16:0/2533519757 wait complete. 2023-05-22T15:46:51.303 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:51.295+0000 7f5c59159640 1 Processor -- start 2023-05-22T15:46:51.304 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:51.295+0000 7f5c59159640 1 -- start start 2023-05-22T15:46:51.304 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:51.295+0000 7f5c59159640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5c5406b6c0 0x7f5c5414ce40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:51.304 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:51.295+0000 7f5c59159640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5c54070c70 0x7f5c54153010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:51.304 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:51.295+0000 7f5c59159640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5c54071530 0x7f5c541572c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:51.304 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:51.295+0000 7f5c59159640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c54154b80 con 0x7f5c54071530 2023-05-22T15:46:51.305 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:51.295+0000 7f5c59159640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c54154e60 con 0x7f5c54070c70 2023-05-22T15:46:51.305 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:51.295+0000 7f5c59159640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c54155140 con 0x7f5c5406b6c0 2023-05-22T15:46:51.305 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:51.295+0000 7f5c5359e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5c54071530 0x7f5c541572c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:51.305 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:51.295+0000 7f5c5359e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5c54071530 0x7f5c541572c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43454/0 (socket says 172.21.15.16:43454) 2023-05-22T15:46:51.305 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:51.295+0000 7f5c5359e640 1 -- 172.21.15.16:0/2546447752 learned_addr learned my addr 172.21.15.16:0/2546447752 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:46:51.305 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:51.295+0000 7f5c5259c640 1 --2- 172.21.15.16:0/2546447752 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5c54070c70 0x7f5c54153010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:51.305 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:51.295+0000 7f5c52d9d640 1 --2- 172.21.15.16:0/2546447752 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5c5406b6c0 0x7f5c5414ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:51.305 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:51.295+0000 7f5c5359e640 1 -- 172.21.15.16:0/2546447752 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5c5406b6c0 msgr2=0x7f5c5414ce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:51.306 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:51.295+0000 7f5c5359e640 1 --2- 172.21.15.16:0/2546447752 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5c5406b6c0 0x7f5c5414ce40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:51.306 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:51.295+0000 7f5c5359e640 1 -- 172.21.15.16:0/2546447752 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5c54070c70 msgr2=0x7f5c54153010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:51.306 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:51.295+0000 7f5c5359e640 1 --2- 172.21.15.16:0/2546447752 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5c54070c70 0x7f5c54153010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:51.306 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:51.295+0000 7f5c5359e640 1 -- 172.21.15.16:0/2546447752 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c480094d0 con 0x7f5c54071530 2023-05-22T15:46:51.306 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:51.295+0000 7f5c5259c640 1 --2- 172.21.15.16:0/2546447752 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5c54070c70 0x7f5c54153010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:46:51.306 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:51.295+0000 7f5c5359e640 1 --2- 172.21.15.16:0/2546447752 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5c54071530 0x7f5c541572c0 secure :-1 s=READY pgs=1977 cs=0 l=1 rev1=1 crypto rx=0x7f5c4c007e40 tx=0x7f5c4c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:51.306 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:51.295+0000 7f5c3b7fe640 1 -- 172.21.15.16:0/2546447752 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c4c014620 con 0x7f5c54071530 2023-05-22T15:46:51.307 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:51.295+0000 7f5c3b7fe640 1 -- 172.21.15.16:0/2546447752 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c4c00ee90 con 0x7f5c54071530 2023-05-22T15:46:51.307 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:51.295+0000 7f5c3b7fe640 1 -- 172.21.15.16:0/2546447752 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c4c0149e0 con 0x7f5c54071530 2023-05-22T15:46:51.307 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:51.295+0000 7f5c59159640 1 -- 172.21.15.16:0/2546447752 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c540cc510 con 0x7f5c54071530 2023-05-22T15:46:51.307 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:51.295+0000 7f5c59159640 1 -- 172.21.15.16:0/2546447752 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c5415d240 con 0x7f5c54071530 2023-05-22T15:46:51.307 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:51.295+0000 7f5c59159640 1 -- 172.21.15.16:0/2546447752 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c54071040 con 0x7f5c54071530 2023-05-22T15:46:51.307 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:51.299+0000 7f5c3b7fe640 1 -- 172.21.15.16:0/2546447752 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 67) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f5c4c014b40 con 0x7f5c54071530 2023-05-22T15:46:51.307 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:51.299+0000 7f5c3b7fe640 1 --2- 172.21.15.16:0/2546447752 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f5c400654d0 0x7f5c40067980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:51.307 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:51.299+0000 7f5c3b7fe640 1 -- 172.21.15.16:0/2546447752 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(150..150 src has 1..150) v4 ==== 7141+0+0 (secure 0 0 0) 0x7f5c4c0837d0 con 0x7f5c54071530 2023-05-22T15:46:51.308 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:51.299+0000 7f5c52d9d640 1 --2- 172.21.15.16:0/2546447752 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f5c400654d0 0x7f5c40067980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:51.308 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:51.299+0000 7f5c52d9d640 1 --2- 172.21.15.16:0/2546447752 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f5c400654d0 0x7f5c40067980 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f5c44000f70 tx=0x7f5c44009040 comp rx=0 tx=0).ready entity=mgr.9396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:51.308 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:51.303+0000 7f5c3b7fe640 1 -- 172.21.15.16:0/2546447752 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f5c4c050ef0 con 0x7f5c54071530 2023-05-22T15:46:51.495 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:51.491+0000 7f5c59159640 1 -- 172.21.15.16:0/2546447752 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f5c54154cc0 con 0x7f5c54071530 2023-05-22T15:46:52.148 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:52.143+0000 7f5c3b7fe640 1 -- 172.21.15.16:0/2546447752 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v60) v1 ==== 104+0+0 (secure 0 0 0) 0x7f5c54154cc0 con 0x7f5c54071530 2023-05-22T15:46:52.152 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:52.147+0000 7f5c59159640 1 -- 172.21.15.16:0/2546447752 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f5c400654d0 msgr2=0x7f5c40067980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:52.153 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:52.147+0000 7f5c59159640 1 --2- 172.21.15.16:0/2546447752 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f5c400654d0 0x7f5c40067980 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f5c44000f70 tx=0x7f5c44009040 comp rx=0 tx=0).stop 2023-05-22T15:46:52.153 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:52.147+0000 7f5c59159640 1 -- 172.21.15.16:0/2546447752 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5c54071530 msgr2=0x7f5c541572c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:52.153 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:52.147+0000 7f5c59159640 1 --2- 172.21.15.16:0/2546447752 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5c54071530 0x7f5c541572c0 secure :-1 s=READY pgs=1977 cs=0 l=1 rev1=1 crypto rx=0x7f5c4c007e40 tx=0x7f5c4c00e4b0 comp rx=0 tx=0).stop 2023-05-22T15:46:52.153 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:52.147+0000 7f5c59159640 1 -- 172.21.15.16:0/2546447752 shutdown_connections 2023-05-22T15:46:52.154 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:52.147+0000 7f5c59159640 1 --2- 172.21.15.16:0/2546447752 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5c54071530 0x7f5c541572c0 unknown :-1 s=CLOSED pgs=1977 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:52.154 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:52.147+0000 7f5c59159640 1 --2- 172.21.15.16:0/2546447752 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5c54070c70 0x7f5c54153010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:52.154 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:52.147+0000 7f5c59159640 1 --2- 172.21.15.16:0/2546447752 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f5c400654d0 0x7f5c40067980 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:52.154 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:52.147+0000 7f5c59159640 1 --2- 172.21.15.16:0/2546447752 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5c5406b6c0 0x7f5c5414ce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:52.154 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:52.147+0000 7f5c59159640 1 -- 172.21.15.16:0/2546447752 >> 172.21.15.16:0/2546447752 conn(0x7f5c54069720 msgr2=0x7f5c5415b4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:46:52.154 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:52.147+0000 7f5c59159640 1 -- 172.21.15.16:0/2546447752 shutdown_connections 2023-05-22T15:46:52.154 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:52.147+0000 7f5c59159640 1 -- 172.21.15.16:0/2546447752 wait complete. 2023-05-22T15:46:52.175 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-05-22T15:46:52.349 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:52.343+0000 7efc123ef640 1 Processor -- start 2023-05-22T15:46:52.350 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:52.343+0000 7efc123ef640 1 -- start start 2023-05-22T15:46:52.351 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:52.343+0000 7efc123ef640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7efc0c15c8e0 0x7efc0c0ccc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:52.351 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:52.343+0000 7efc123ef640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7efc0c0cef10 0x7efc0c0cf350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:52.351 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:52.343+0000 7efc123ef640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc0c0d2a20 con 0x7efc0c0d2eb0 2023-05-22T15:46:52.351 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:52.343+0000 7efc123ef640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc0c0cd230 con 0x7efc0c15c8e0 2023-05-22T15:46:52.352 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:52.343+0000 7efc123ef640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc0c0d23d0 con 0x7efc0c0cef10 2023-05-22T15:46:52.352 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:52.343+0000 7efc0bfff640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7efc0c15c8e0 0x7efc0c0ccc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:52.352 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:52.343+0000 7efc1098c640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7efc0c0cef10 0x7efc0c0cf350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:52.352 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:52.343+0000 7efc1098c640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7efc0c0cef10 0x7efc0c0cf350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3301/0 says I am v2:172.21.15.16:56086/0 (socket says 172.21.15.16:56086) 2023-05-22T15:46:52.352 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:52.343+0000 7efc0b7fe640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7efc0c0d2eb0 0x7efc0c15ccb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.16:59038/0 (socket says 172.21.15.16:59038) 2023-05-22T15:46:52.352 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:52.343+0000 7efc1098c640 1 -- 172.21.15.16:0/32941558 learned_addr learned my addr 172.21.15.16:0/32941558 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:46:52.353 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:52.347+0000 7efc1098c640 1 -- 172.21.15.16:0/32941558 >> v2:172.21.15.62:3300/0 conn(0x7efc0c15c8e0 msgr2=0x7efc0c0ccc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:46:52.353 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:52.347+0000 7efc1098c640 1 --2- 172.21.15.16:0/32941558 >> v2:172.21.15.62:3300/0 conn(0x7efc0c15c8e0 0x7efc0c0ccc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:52.353 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:52.347+0000 7efc1098c640 1 -- 172.21.15.16:0/32941558 >> v1:172.21.15.62:6789/0 conn(0x7efc0c0d2eb0 legacy=0x7efc0c15ccb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:52.353 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:52.347+0000 7efc1098c640 1 -- 172.21.15.16:0/32941558 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc0c0d26b0 con 0x7efc0c0cef10 2023-05-22T15:46:52.354 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:52.347+0000 7efc1098c640 1 --2- 172.21.15.16:0/32941558 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7efc0c0cef10 0x7efc0c0cf350 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7efc04009e50 tx=0x7efc0401c940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=38bb9655e428345a server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:52.354 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:52.347+0000 7efc0affd640 1 -- 172.21.15.16:0/32941558 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc040a5860 con 0x7efc0c0cef10 2023-05-22T15:46:52.354 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:52.347+0000 7efc0affd640 1 -- 172.21.15.16:0/32941558 <== mon.2 v2:172.21.15.62:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc040a59c0 con 0x7efc0c0cef10 2023-05-22T15:46:52.355 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:52.347+0000 7efc0affd640 1 -- 172.21.15.16:0/32941558 <== mon.2 v2:172.21.15.62:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc040a5d50 con 0x7efc0c0cef10 2023-05-22T15:46:52.355 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:52.351+0000 7efc123ef640 1 -- 172.21.15.16:0/32941558 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7efc0c0cef10 msgr2=0x7efc0c0cf350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:52.355 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:52.351+0000 7efc123ef640 1 --2- 172.21.15.16:0/32941558 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7efc0c0cef10 0x7efc0c0cf350 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7efc04009e50 tx=0x7efc0401c940 comp rx=0 tx=0).stop 2023-05-22T15:46:52.356 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:52.351+0000 7efc123ef640 1 -- 172.21.15.16:0/32941558 shutdown_connections 2023-05-22T15:46:52.356 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:52.351+0000 7efc123ef640 1 --2- 172.21.15.16:0/32941558 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7efc0c0cef10 0x7efc0c0cf350 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:52.356 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:52.351+0000 7efc123ef640 1 --2- 172.21.15.16:0/32941558 >> v2:172.21.15.62:3300/0 conn(0x7efc0c15c8e0 0x7efc0c0ccc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:52.356 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:52.351+0000 7efc123ef640 1 -- 172.21.15.16:0/32941558 >> 172.21.15.16:0/32941558 conn(0x7efc0c070e50 msgr2=0x7efc0c071250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:46:52.356 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:52.351+0000 7efc123ef640 1 -- 172.21.15.16:0/32941558 shutdown_connections 2023-05-22T15:46:52.356 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:52.351+0000 7efc123ef640 1 -- 172.21.15.16:0/32941558 wait complete. 2023-05-22T15:46:52.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:52.351+0000 7efc123ef640 1 Processor -- start 2023-05-22T15:46:52.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:52.351+0000 7efc123ef640 1 -- start start 2023-05-22T15:46:52.358 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:52.351+0000 7efc123ef640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efc0c0cef10 0x7efc0c14b520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:52.358 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:52.351+0000 7efc123ef640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7efc0c0d2eb0 0x7efc0c151490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:52.358 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:52.351+0000 7efc123ef640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efc0c15c8e0 0x7efc0c1520a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:52.358 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:52.351+0000 7efc123ef640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc0c1501a0 con 0x7efc0c15c8e0 2023-05-22T15:46:52.358 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:52.351+0000 7efc123ef640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc0c1503f0 con 0x7efc0c0cef10 2023-05-22T15:46:52.358 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:52.351+0000 7efc123ef640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc0c1506d0 con 0x7efc0c0d2eb0 2023-05-22T15:46:52.358 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:52.351+0000 7efc1098c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efc0c15c8e0 0x7efc0c1520a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:52.359 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:52.351+0000 7efc1098c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efc0c15c8e0 0x7efc0c1520a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43458/0 (socket says 172.21.15.16:43458) 2023-05-22T15:46:52.359 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:52.351+0000 7efc1098c640 1 -- 172.21.15.16:0/3748966304 learned_addr learned my addr 172.21.15.16:0/3748966304 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:46:52.359 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:52.351+0000 7efc0bfff640 1 --2- 172.21.15.16:0/3748966304 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efc0c0cef10 0x7efc0c14b520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:52.359 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:52.351+0000 7efc0b7fe640 1 --2- 172.21.15.16:0/3748966304 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7efc0c0d2eb0 0x7efc0c151490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:52.359 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:52.351+0000 7efc1098c640 1 -- 172.21.15.16:0/3748966304 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7efc0c0d2eb0 msgr2=0x7efc0c151490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:52.359 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:52.351+0000 7efc1098c640 1 --2- 172.21.15.16:0/3748966304 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7efc0c0d2eb0 0x7efc0c151490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:52.359 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:52.351+0000 7efc1098c640 1 -- 172.21.15.16:0/3748966304 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efc0c0cef10 msgr2=0x7efc0c14b520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:52.360 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:52.351+0000 7efc1098c640 1 --2- 172.21.15.16:0/3748966304 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efc0c0cef10 0x7efc0c14b520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:52.360 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:52.351+0000 7efc1098c640 1 -- 172.21.15.16:0/3748966304 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc04009b30 con 0x7efc0c15c8e0 2023-05-22T15:46:52.362 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:52.351+0000 7efc1098c640 1 --2- 172.21.15.16:0/3748966304 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efc0c15c8e0 0x7efc0c1520a0 secure :-1 s=READY pgs=1978 cs=0 l=1 rev1=1 crypto rx=0x7efc040af040 tx=0x7efc040196e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:52.362 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:52.351+0000 7efc08ff9640 1 -- 172.21.15.16:0/3748966304 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc040bd070 con 0x7efc0c15c8e0 2023-05-22T15:46:52.362 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:52.351+0000 7efc08ff9640 1 -- 172.21.15.16:0/3748966304 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc040b8470 con 0x7efc0c15c8e0 2023-05-22T15:46:52.362 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:52.351+0000 7efc08ff9640 1 -- 172.21.15.16:0/3748966304 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc040b8840 con 0x7efc0c15c8e0 2023-05-22T15:46:52.362 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:52.351+0000 7efc123ef640 1 -- 172.21.15.16:0/3748966304 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc0c0d3410 con 0x7efc0c15c8e0 2023-05-22T15:46:52.362 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:52.351+0000 7efc123ef640 1 -- 172.21.15.16:0/3748966304 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc0c0d3280 con 0x7efc0c15c8e0 2023-05-22T15:46:52.362 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:52.355+0000 7efc123ef640 1 -- 172.21.15.16:0/3748966304 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc0c0d0bf0 con 0x7efc0c15c8e0 2023-05-22T15:46:52.363 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:52.355+0000 7efc08ff9640 1 -- 172.21.15.16:0/3748966304 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 67) v1 ==== 84332+0+0 (secure 0 0 0) 0x7efc040b89a0 con 0x7efc0c15c8e0 2023-05-22T15:46:52.363 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:52.355+0000 7efc08ff9640 1 --2- 172.21.15.16:0/3748966304 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7efbec0654a0 0x7efbec067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:52.363 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:52.355+0000 7efc08ff9640 1 -- 172.21.15.16:0/3748966304 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(150..150 src has 1..150) v4 ==== 7141+0+0 (secure 0 0 0) 0x7efc0411dca0 con 0x7efc0c15c8e0 2023-05-22T15:46:52.363 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:52.355+0000 7efc0bfff640 1 --2- 172.21.15.16:0/3748966304 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7efbec0654a0 0x7efbec067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:52.363 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:52.355+0000 7efc0bfff640 1 --2- 172.21.15.16:0/3748966304 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7efbec0654a0 0x7efbec067950 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7efbfc009ac0 tx=0x7efbfc009190 comp rx=0 tx=0).ready entity=mgr.9396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:52.364 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:52.359+0000 7efc08ff9640 1 -- 172.21.15.16:0/3748966304 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7efc040eb390 con 0x7efc0c15c8e0 2023-05-22T15:46:52.514 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-05-22T15:46:52.518 DEBUG:teuthology.orchestra.run.smithi062:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-05-22T15:46:52.518 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:52.511+0000 7efc123ef640 1 -- 172.21.15.16:0/3748966304 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7efc0c04dbe0 con 0x7efc0c15c8e0 2023-05-22T15:46:53.242 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:53.235+0000 7efc08ff9640 1 -- 172.21.15.16:0/3748966304 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v151) v1 ==== 181+0+0 (secure 0 0 0) 0x7efc040ee9b0 con 0x7efc0c15c8e0 2023-05-22T15:46:53.243 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'cephfs_metadata' removed 2023-05-22T15:46:53.249 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:53.243+0000 7efc123ef640 1 -- 172.21.15.16:0/3748966304 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7efbec0654a0 msgr2=0x7efbec067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:53.249 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:53.243+0000 7efc123ef640 1 --2- 172.21.15.16:0/3748966304 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7efbec0654a0 0x7efbec067950 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7efbfc009ac0 tx=0x7efbfc009190 comp rx=0 tx=0).stop 2023-05-22T15:46:53.249 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:53.243+0000 7efc123ef640 1 -- 172.21.15.16:0/3748966304 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efc0c15c8e0 msgr2=0x7efc0c1520a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:53.249 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:53.243+0000 7efc123ef640 1 --2- 172.21.15.16:0/3748966304 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efc0c15c8e0 0x7efc0c1520a0 secure :-1 s=READY pgs=1978 cs=0 l=1 rev1=1 crypto rx=0x7efc040af040 tx=0x7efc040196e0 comp rx=0 tx=0).stop 2023-05-22T15:46:53.250 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:53.243+0000 7efc123ef640 1 -- 172.21.15.16:0/3748966304 shutdown_connections 2023-05-22T15:46:53.250 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:53.243+0000 7efc123ef640 1 --2- 172.21.15.16:0/3748966304 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efc0c15c8e0 0x7efc0c1520a0 unknown :-1 s=CLOSED pgs=1978 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:53.251 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:53.243+0000 7efc123ef640 1 --2- 172.21.15.16:0/3748966304 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7efbec0654a0 0x7efbec067950 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:53.251 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:53.243+0000 7efc123ef640 1 --2- 172.21.15.16:0/3748966304 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7efc0c0d2eb0 0x7efc0c151490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:53.252 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:53.243+0000 7efc123ef640 1 --2- 172.21.15.16:0/3748966304 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efc0c0cef10 0x7efc0c14b520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:53.252 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:53.243+0000 7efc123ef640 1 -- 172.21.15.16:0/3748966304 >> 172.21.15.16:0/3748966304 conn(0x7efc0c070e50 msgr2=0x7efc0c15d9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:46:53.252 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:53.243+0000 7efc123ef640 1 -- 172.21.15.16:0/3748966304 shutdown_connections 2023-05-22T15:46:53.252 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:53.243+0000 7efc123ef640 1 -- 172.21.15.16:0/3748966304 wait complete. 2023-05-22T15:46:53.273 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-05-22T15:46:53.401 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:53.395+0000 7ff732aa7640 1 Processor -- start 2023-05-22T15:46:53.402 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:53.395+0000 7ff732aa7640 1 -- start start 2023-05-22T15:46:53.402 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:53.395+0000 7ff732aa7640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff72c15c8e0 0x7ff72c0ccc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:53.403 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:53.395+0000 7ff732aa7640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff72c0d2eb0 0x7ff72c15ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:53.403 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:53.395+0000 7ff732aa7640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff72c0d2a20 con 0x7ff72c0cef10 2023-05-22T15:46:53.403 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:53.395+0000 7ff732aa7640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff72c0cd230 con 0x7ff72c15c8e0 2023-05-22T15:46:53.403 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:53.395+0000 7ff732aa7640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff72c0d23d0 con 0x7ff72c0d2eb0 2023-05-22T15:46:53.403 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:53.395+0000 7ff730843640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff72c15c8e0 0x7ff72c0ccc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:53.403 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:53.395+0000 7ff730843640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff72c15c8e0 0x7ff72c0ccc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43464/0 (socket says 172.21.15.16:43464) 2023-05-22T15:46:53.404 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:53.395+0000 7ff730843640 1 -- 172.21.15.16:0/3407223222 learned_addr learned my addr 172.21.15.16:0/3407223222 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:46:53.404 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:53.395+0000 7ff730843640 1 -- 172.21.15.16:0/3407223222 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff72c0d2eb0 msgr2=0x7ff72c15ccb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:46:53.404 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:53.395+0000 7ff72bfff640 1 --2- 172.21.15.16:0/3407223222 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff72c0d2eb0 0x7ff72c15ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:53.404 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:53.395+0000 7ff730843640 1 --2- 172.21.15.16:0/3407223222 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff72c0d2eb0 0x7ff72c15ccb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:53.404 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:53.395+0000 7ff730843640 1 -- 172.21.15.16:0/3407223222 >> v1:172.21.15.16:6789/0 conn(0x7ff72c0cef10 legacy=0x7ff72c0cf370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:53.404 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:53.395+0000 7ff730843640 1 -- 172.21.15.16:0/3407223222 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff72c0d26b0 con 0x7ff72c15c8e0 2023-05-22T15:46:53.404 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:53.395+0000 7ff72b7fe640 1 -- 172.21.15.16:0/3407223222 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 808367580 0 0) 0x7ff72c0d2a20 con 0x7ff72c0cef10 2023-05-22T15:46:53.404 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:53.395+0000 7ff72bfff640 1 --2- 172.21.15.16:0/3407223222 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff72c0d2eb0 0x7ff72c15ccb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:46:53.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:53.399+0000 7ff730843640 1 --2- 172.21.15.16:0/3407223222 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff72c15c8e0 0x7ff72c0ccc50 secure :-1 s=READY pgs=1980 cs=0 l=1 rev1=1 crypto rx=0x7ff71c009e50 tx=0x7ff71c01c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f258546140ba1368 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:53.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:53.399+0000 7ff72b7fe640 1 -- 172.21.15.16:0/3407223222 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff71c0a57e0 con 0x7ff72c15c8e0 2023-05-22T15:46:53.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:53.399+0000 7ff72b7fe640 1 -- 172.21.15.16:0/3407223222 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff71c0a5940 con 0x7ff72c15c8e0 2023-05-22T15:46:53.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:53.399+0000 7ff72b7fe640 1 -- 172.21.15.16:0/3407223222 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff71c0a5cd0 con 0x7ff72c15c8e0 2023-05-22T15:46:53.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:53.399+0000 7ff732aa7640 1 -- 172.21.15.16:0/3407223222 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff72c15c8e0 msgr2=0x7ff72c0ccc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:53.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:53.399+0000 7ff732aa7640 1 --2- 172.21.15.16:0/3407223222 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff72c15c8e0 0x7ff72c0ccc50 secure :-1 s=READY pgs=1980 cs=0 l=1 rev1=1 crypto rx=0x7ff71c009e50 tx=0x7ff71c01c940 comp rx=0 tx=0).stop 2023-05-22T15:46:53.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:53.399+0000 7ff732aa7640 1 -- 172.21.15.16:0/3407223222 shutdown_connections 2023-05-22T15:46:53.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:53.399+0000 7ff732aa7640 1 --2- 172.21.15.16:0/3407223222 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff72c0d2eb0 0x7ff72c15ccb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:53.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:53.399+0000 7ff732aa7640 1 --2- 172.21.15.16:0/3407223222 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff72c15c8e0 0x7ff72c0ccc50 unknown :-1 s=CLOSED pgs=1980 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:53.408 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:53.399+0000 7ff732aa7640 1 -- 172.21.15.16:0/3407223222 >> 172.21.15.16:0/3407223222 conn(0x7ff72c070e50 msgr2=0x7ff72c071250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:46:53.408 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:53.399+0000 7ff732aa7640 1 -- 172.21.15.16:0/3407223222 shutdown_connections 2023-05-22T15:46:53.408 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:53.399+0000 7ff732aa7640 1 -- 172.21.15.16:0/3407223222 wait complete. 2023-05-22T15:46:53.408 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:53.403+0000 7ff732aa7640 1 Processor -- start 2023-05-22T15:46:53.408 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:53.403+0000 7ff732aa7640 1 -- start start 2023-05-22T15:46:53.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:53.403+0000 7ff732aa7640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff72c0cef10 0x7ff72c0cc1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:53.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:53.403+0000 7ff732aa7640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff72c0d2eb0 0x7ff72c0c96d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:53.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:53.403+0000 7ff732aa7640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff72c15c8e0 0x7ff72c0c83f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:53.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:53.403+0000 7ff732aa7640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff72c0c9eb0 con 0x7ff72c15c8e0 2023-05-22T15:46:53.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:53.403+0000 7ff732aa7640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff72c0b5560 con 0x7ff72c0cef10 2023-05-22T15:46:53.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:53.403+0000 7ff732aa7640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff72c0c9c10 con 0x7ff72c0d2eb0 2023-05-22T15:46:53.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:53.403+0000 7ff731044640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff72c15c8e0 0x7ff72c0c83f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:53.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:53.403+0000 7ff731044640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff72c15c8e0 0x7ff72c0c83f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43468/0 (socket says 172.21.15.16:43468) 2023-05-22T15:46:53.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:53.403+0000 7ff731044640 1 -- 172.21.15.16:0/296461589 learned_addr learned my addr 172.21.15.16:0/296461589 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:46:53.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:53.403+0000 7ff72bfff640 1 --2- 172.21.15.16:0/296461589 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff72c0d2eb0 0x7ff72c0c96d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:53.412 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:53.403+0000 7ff730843640 1 --2- 172.21.15.16:0/296461589 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff72c0cef10 0x7ff72c0cc1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:53.412 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:53.403+0000 7ff731044640 1 -- 172.21.15.16:0/296461589 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff72c0d2eb0 msgr2=0x7ff72c0c96d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:53.412 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:53.403+0000 7ff731044640 1 --2- 172.21.15.16:0/296461589 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff72c0d2eb0 0x7ff72c0c96d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:53.412 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:53.403+0000 7ff731044640 1 -- 172.21.15.16:0/296461589 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff72c0cef10 msgr2=0x7ff72c0cc1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:53.412 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:53.403+0000 7ff731044640 1 --2- 172.21.15.16:0/296461589 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff72c0cef10 0x7ff72c0cc1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:53.412 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:53.403+0000 7ff731044640 1 -- 172.21.15.16:0/296461589 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff71c009b30 con 0x7ff72c15c8e0 2023-05-22T15:46:53.412 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:53.403+0000 7ff730843640 1 --2- 172.21.15.16:0/296461589 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff72c0cef10 0x7ff72c0cc1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T15:46:53.413 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:53.403+0000 7ff731044640 1 --2- 172.21.15.16:0/296461589 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff72c15c8e0 0x7ff72c0c83f0 secure :-1 s=READY pgs=1981 cs=0 l=1 rev1=1 crypto rx=0x7ff72400c450 tx=0x7ff72400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:53.413 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:53.403+0000 7ff7297fa640 1 -- 172.21.15.16:0/296461589 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff724005480 con 0x7ff72c15c8e0 2023-05-22T15:46:53.413 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:53.403+0000 7ff7297fa640 1 -- 172.21.15.16:0/296461589 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff724003600 con 0x7ff72c15c8e0 2023-05-22T15:46:53.413 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:53.403+0000 7ff732aa7640 1 -- 172.21.15.16:0/296461589 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff72c0d3410 con 0x7ff72c15c8e0 2023-05-22T15:46:53.413 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:53.403+0000 7ff7297fa640 1 -- 172.21.15.16:0/296461589 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7240039b0 con 0x7ff72c15c8e0 2023-05-22T15:46:53.413 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:53.403+0000 7ff732aa7640 1 -- 172.21.15.16:0/296461589 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff72c0d3280 con 0x7ff72c15c8e0 2023-05-22T15:46:53.414 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:53.407+0000 7ff732aa7640 1 -- 172.21.15.16:0/296461589 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff72c0d0bf0 con 0x7ff72c15c8e0 2023-05-22T15:46:53.414 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:53.407+0000 7ff7297fa640 1 -- 172.21.15.16:0/296461589 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 67) v1 ==== 84332+0+0 (secure 0 0 0) 0x7ff724003b10 con 0x7ff72c15c8e0 2023-05-22T15:46:53.414 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:53.407+0000 7ff7297fa640 1 --2- 172.21.15.16:0/296461589 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ff7180654a0 0x7ff718067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:53.414 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:53.407+0000 7ff7297fa640 1 -- 172.21.15.16:0/296461589 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(151..151 src has 1..151) v4 ==== 6703+0+0 (secure 0 0 0) 0x7ff724082170 con 0x7ff72c15c8e0 2023-05-22T15:46:53.415 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:53.407+0000 7ff730843640 1 --2- 172.21.15.16:0/296461589 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ff7180654a0 0x7ff718067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:53.415 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:53.407+0000 7ff730843640 1 --2- 172.21.15.16:0/296461589 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ff7180654a0 0x7ff718067950 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7ff71c0ae000 tx=0x7ff71c0a7070 comp rx=0 tx=0).ready entity=mgr.9396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:53.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:53.411+0000 7ff7297fa640 1 -- 172.21.15.16:0/296461589 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7ff7240535f0 con 0x7ff72c15c8e0 2023-05-22T15:46:53.572 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:53.567+0000 7ff732aa7640 1 -- 172.21.15.16:0/296461589 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7ff72c0b56d0 con 0x7ff72c15c8e0 2023-05-22T15:46:54.259 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:54.247+0000 7ff7297fa640 1 -- 172.21.15.16:0/296461589 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v152) v1 ==== 169+0+0 (secure 0 0 0) 0x7ff7240531e0 con 0x7ff72c15c8e0 2023-05-22T15:46:54.271 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'cephfs_data' removed 2023-05-22T15:46:54.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:54.271+0000 7ff732aa7640 1 -- 172.21.15.16:0/296461589 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ff7180654a0 msgr2=0x7ff718067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:54.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:54.271+0000 7ff732aa7640 1 --2- 172.21.15.16:0/296461589 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ff7180654a0 0x7ff718067950 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7ff71c0ae000 tx=0x7ff71c0a7070 comp rx=0 tx=0).stop 2023-05-22T15:46:54.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:54.271+0000 7ff732aa7640 1 -- 172.21.15.16:0/296461589 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff72c15c8e0 msgr2=0x7ff72c0c83f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:54.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:54.271+0000 7ff732aa7640 1 --2- 172.21.15.16:0/296461589 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff72c15c8e0 0x7ff72c0c83f0 secure :-1 s=READY pgs=1981 cs=0 l=1 rev1=1 crypto rx=0x7ff72400c450 tx=0x7ff72400c910 comp rx=0 tx=0).stop 2023-05-22T15:46:54.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:54.271+0000 7ff732aa7640 1 -- 172.21.15.16:0/296461589 shutdown_connections 2023-05-22T15:46:54.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:54.271+0000 7ff732aa7640 1 --2- 172.21.15.16:0/296461589 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff72c15c8e0 0x7ff72c0c83f0 unknown :-1 s=CLOSED pgs=1981 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:54.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:54.271+0000 7ff732aa7640 1 --2- 172.21.15.16:0/296461589 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7ff7180654a0 0x7ff718067950 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:54.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:54.271+0000 7ff732aa7640 1 --2- 172.21.15.16:0/296461589 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff72c0d2eb0 0x7ff72c0c96d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:54.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:54.271+0000 7ff732aa7640 1 --2- 172.21.15.16:0/296461589 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff72c0cef10 0x7ff72c0cc1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:54.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:54.271+0000 7ff732aa7640 1 -- 172.21.15.16:0/296461589 >> 172.21.15.16:0/296461589 conn(0x7ff72c070e50 msgr2=0x7ff72c15e6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:46:54.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:54.275+0000 7ff732aa7640 1 -- 172.21.15.16:0/296461589 shutdown_connections 2023-05-22T15:46:54.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:54.275+0000 7ff732aa7640 1 -- 172.21.15.16:0/296461589 wait complete. 2023-05-22T15:46:54.303 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2023-05-22T15:46:54.426 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:54.419+0000 7f72ae3b8640 1 Processor -- start 2023-05-22T15:46:54.427 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:54.419+0000 7f72ae3b8640 1 -- start start 2023-05-22T15:46:54.428 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:54.423+0000 7f72ae3b8640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f72a815bca0 0x7f72a80cda80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:54.428 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:54.423+0000 7f72ae3b8640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f72a80b5a50 0x7f72a80d2900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:54.428 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:54.423+0000 7f72ae3b8640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72a80cccc0 con 0x7f72a80d2480 2023-05-22T15:46:54.429 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:54.423+0000 7f72ae3b8640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72a80cd660 con 0x7f72a815bca0 2023-05-22T15:46:54.429 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:54.423+0000 7f72ae3b8640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72a80d2e40 con 0x7f72a80b5a50 2023-05-22T15:46:54.429 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:54.423+0000 7f72a7fff640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f72a815bca0 0x7f72a80cda80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:54.429 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:54.423+0000 7f72a7fff640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f72a815bca0 0x7f72a80cda80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43478/0 (socket says 172.21.15.16:43478) 2023-05-22T15:46:54.429 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:54.423+0000 7f72a7fff640 1 -- 172.21.15.16:0/4099622810 learned_addr learned my addr 172.21.15.16:0/4099622810 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:46:54.429 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:54.423+0000 7f72a77fe640 1 --1- 172.21.15.16:0/4099622810 >> v1:172.21.15.16:6789/0 conn(0x7f72a80d2480 0x7f72a80b5340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:55570/0 (socket says 172.21.15.16:55570) 2023-05-22T15:46:54.429 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:54.423+0000 7f72ac955640 1 --2- 172.21.15.16:0/4099622810 >> v2:172.21.15.62:3300/0 conn(0x7f72a80b5a50 0x7f72a80d2900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:54.430 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:54.423+0000 7f72a7fff640 1 -- 172.21.15.16:0/4099622810 >> v2:172.21.15.62:3300/0 conn(0x7f72a80b5a50 msgr2=0x7f72a80d2900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:46:54.430 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:54.423+0000 7f72a7fff640 1 --2- 172.21.15.16:0/4099622810 >> v2:172.21.15.62:3300/0 conn(0x7f72a80b5a50 0x7f72a80d2900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:54.430 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:54.423+0000 7f72a7fff640 1 -- 172.21.15.16:0/4099622810 >> v1:172.21.15.16:6789/0 conn(0x7f72a80d2480 legacy=0x7f72a80b5340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:54.430 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:54.423+0000 7f72a7fff640 1 -- 172.21.15.16:0/4099622810 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72a80ccfa0 con 0x7f72a815bca0 2023-05-22T15:46:54.430 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:54.423+0000 7f72a6ffd640 1 -- 172.21.15.16:0/4099622810 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 97556285 0 0) 0x7f72a80cccc0 con 0x7f72a80d2480 2023-05-22T15:46:54.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:54.423+0000 7f72a7fff640 1 --2- 172.21.15.16:0/4099622810 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f72a815bca0 0x7f72a80cda80 secure :-1 s=READY pgs=1983 cs=0 l=1 rev1=1 crypto rx=0x7f72a80c5fc0 tx=0x7f729801c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1f7784274153c67 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:54.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:54.423+0000 7f72a6ffd640 1 -- 172.21.15.16:0/4099622810 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72980aa680 con 0x7f72a815bca0 2023-05-22T15:46:54.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:54.423+0000 7f72a6ffd640 1 -- 172.21.15.16:0/4099622810 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72980aa7e0 con 0x7f72a815bca0 2023-05-22T15:46:54.432 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:54.423+0000 7f72a6ffd640 1 -- 172.21.15.16:0/4099622810 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72980aab70 con 0x7f72a815bca0 2023-05-22T15:46:54.432 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:54.427+0000 7f72ae3b8640 1 -- 172.21.15.16:0/4099622810 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f72a815bca0 msgr2=0x7f72a80cda80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:54.432 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:54.427+0000 7f72ae3b8640 1 --2- 172.21.15.16:0/4099622810 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f72a815bca0 0x7f72a80cda80 secure :-1 s=READY pgs=1983 cs=0 l=1 rev1=1 crypto rx=0x7f72a80c5fc0 tx=0x7f729801c500 comp rx=0 tx=0).stop 2023-05-22T15:46:54.432 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:54.427+0000 7f72ae3b8640 1 -- 172.21.15.16:0/4099622810 shutdown_connections 2023-05-22T15:46:54.432 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:54.427+0000 7f72ae3b8640 1 --2- 172.21.15.16:0/4099622810 >> v2:172.21.15.62:3300/0 conn(0x7f72a80b5a50 0x7f72a80d2900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:54.432 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:54.427+0000 7f72ae3b8640 1 --2- 172.21.15.16:0/4099622810 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f72a815bca0 0x7f72a80cda80 unknown :-1 s=CLOSED pgs=1983 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:54.432 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:54.427+0000 7f72ae3b8640 1 -- 172.21.15.16:0/4099622810 >> 172.21.15.16:0/4099622810 conn(0x7f72a80655c0 msgr2=0x7f72a8070e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:46:54.433 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:54.427+0000 7f72ae3b8640 1 -- 172.21.15.16:0/4099622810 shutdown_connections 2023-05-22T15:46:54.433 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:54.427+0000 7f72ae3b8640 1 -- 172.21.15.16:0/4099622810 wait complete. 2023-05-22T15:46:54.434 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:54.427+0000 7f72ae3b8640 1 Processor -- start 2023-05-22T15:46:54.434 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:54.427+0000 7f72ae3b8640 1 -- start start 2023-05-22T15:46:54.435 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:54.427+0000 7f72ae3b8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f72a80b5a50 0x7f72a81f7a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:54.435 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:54.427+0000 7f72ae3b8640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f72a80d2480 0x7f72a80ca3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:54.435 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:54.427+0000 7f72ae3b8640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f72a815bca0 0x7f72a8156980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:54.435 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:54.427+0000 7f72ae3b8640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72a80c9f20 con 0x7f72a80b5a50 2023-05-22T15:46:54.435 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:54.427+0000 7f72ae3b8640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72a80c9c20 con 0x7f72a815bca0 2023-05-22T15:46:54.435 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:54.427+0000 7f72ae3b8640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72a81f8280 con 0x7f72a80d2480 2023-05-22T15:46:54.435 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:54.427+0000 7f72a7fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f72a80b5a50 0x7f72a81f7a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:54.436 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:54.427+0000 7f72a7fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f72a80b5a50 0x7f72a81f7a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43498/0 (socket says 172.21.15.16:43498) 2023-05-22T15:46:54.436 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:54.427+0000 7f72a7fff640 1 -- 172.21.15.16:0/2016315857 learned_addr learned my addr 172.21.15.16:0/2016315857 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:46:54.436 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:54.427+0000 7f72ac955640 1 --2- 172.21.15.16:0/2016315857 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f72a815bca0 0x7f72a8156980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:54.436 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:54.427+0000 7f72a77fe640 1 --2- 172.21.15.16:0/2016315857 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f72a80d2480 0x7f72a80ca3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:54.436 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:54.427+0000 7f72a7fff640 1 -- 172.21.15.16:0/2016315857 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f72a80d2480 msgr2=0x7f72a80ca3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:54.436 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:54.427+0000 7f72a7fff640 1 --2- 172.21.15.16:0/2016315857 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f72a80d2480 0x7f72a80ca3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:54.436 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:54.427+0000 7f72a7fff640 1 -- 172.21.15.16:0/2016315857 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f72a815bca0 msgr2=0x7f72a8156980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:54.436 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:54.427+0000 7f72a7fff640 1 --2- 172.21.15.16:0/2016315857 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f72a815bca0 0x7f72a8156980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:54.437 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:54.427+0000 7f72a7fff640 1 -- 172.21.15.16:0/2016315857 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72980094d0 con 0x7f72a80b5a50 2023-05-22T15:46:54.437 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:54.427+0000 7f72ac955640 1 --2- 172.21.15.16:0/2016315857 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f72a815bca0 0x7f72a8156980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:46:54.437 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:54.427+0000 7f72a7fff640 1 --2- 172.21.15.16:0/2016315857 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f72a80b5a50 0x7f72a81f7a30 secure :-1 s=READY pgs=1984 cs=0 l=1 rev1=1 crypto rx=0x7f7298019110 tx=0x7f729801ad40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:54.437 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:54.427+0000 7f72a4ff9640 1 -- 172.21.15.16:0/2016315857 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72980aa680 con 0x7f72a80b5a50 2023-05-22T15:46:54.437 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:54.427+0000 7f72a4ff9640 1 -- 172.21.15.16:0/2016315857 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72980b3800 con 0x7f72a80b5a50 2023-05-22T15:46:54.437 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:54.427+0000 7f72ae3b8640 1 -- 172.21.15.16:0/2016315857 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72a8160360 con 0x7f72a80b5a50 2023-05-22T15:46:54.437 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:54.427+0000 7f72a4ff9640 1 -- 172.21.15.16:0/2016315857 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72980b3bb0 con 0x7f72a80b5a50 2023-05-22T15:46:54.438 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:54.427+0000 7f72ae3b8640 1 -- 172.21.15.16:0/2016315857 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72a80cdfc0 con 0x7f72a80b5a50 2023-05-22T15:46:54.438 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:54.431+0000 7f72ae3b8640 1 -- 172.21.15.16:0/2016315857 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72a81604f0 con 0x7f72a80b5a50 2023-05-22T15:46:54.438 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:54.431+0000 7f72a4ff9640 1 -- 172.21.15.16:0/2016315857 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 67) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f72980b3d10 con 0x7f72a80b5a50 2023-05-22T15:46:54.439 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:54.431+0000 7f72a4ff9640 1 --2- 172.21.15.16:0/2016315857 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f72940654a0 0x7f7294067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:54.439 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:54.431+0000 7f72a4ff9640 1 -- 172.21.15.16:0/2016315857 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(152..152 src has 1..152) v4 ==== 6289+0+0 (secure 0 0 0) 0x7f729811d560 con 0x7f72a80b5a50 2023-05-22T15:46:54.439 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:54.431+0000 7f72a77fe640 1 --2- 172.21.15.16:0/2016315857 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f72940654a0 0x7f7294067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:54.439 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:54.431+0000 7f72a77fe640 1 --2- 172.21.15.16:0/2016315857 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f72940654a0 0x7f7294067950 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f72a806b8c0 tx=0x7f729c009040 comp rx=0 tx=0).ready entity=mgr.9396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:54.440 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:54.435+0000 7f72a4ff9640 1 -- 172.21.15.16:0/2016315857 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f72980eaf20 con 0x7f72a80b5a50 2023-05-22T15:46:54.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:54.591+0000 7f72ae3b8640 1 -- 172.21.15.16:0/2016315857 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7f72a804dbe0 con 0x7f72a80b5a50 2023-05-22T15:46:54.603 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:54.595+0000 7f72a4ff9640 1 -- 172.21.15.16:0/2016315857 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v49) v1 ==== 120+0+0 (secure 0 0 0) 0x7f72980ee540 con 0x7f72a80b5a50 2023-05-22T15:46:54.606 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:54.599+0000 7f72ae3b8640 1 -- 172.21.15.16:0/2016315857 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f72940654a0 msgr2=0x7f7294067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:54.606 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:54.599+0000 7f72ae3b8640 1 --2- 172.21.15.16:0/2016315857 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f72940654a0 0x7f7294067950 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f72a806b8c0 tx=0x7f729c009040 comp rx=0 tx=0).stop 2023-05-22T15:46:54.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:54.599+0000 7f72ae3b8640 1 -- 172.21.15.16:0/2016315857 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f72a80b5a50 msgr2=0x7f72a81f7a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:54.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:54.599+0000 7f72ae3b8640 1 --2- 172.21.15.16:0/2016315857 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f72a80b5a50 0x7f72a81f7a30 secure :-1 s=READY pgs=1984 cs=0 l=1 rev1=1 crypto rx=0x7f7298019110 tx=0x7f729801ad40 comp rx=0 tx=0).stop 2023-05-22T15:46:54.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:54.599+0000 7f72ae3b8640 1 -- 172.21.15.16:0/2016315857 shutdown_connections 2023-05-22T15:46:54.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:54.599+0000 7f72ae3b8640 1 --2- 172.21.15.16:0/2016315857 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f72a815bca0 0x7f72a8156980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:54.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:54.599+0000 7f72ae3b8640 1 --2- 172.21.15.16:0/2016315857 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f72940654a0 0x7f7294067950 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:54.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:54.599+0000 7f72ae3b8640 1 --2- 172.21.15.16:0/2016315857 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f72a80d2480 0x7f72a80ca3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:54.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:54.599+0000 7f72ae3b8640 1 --2- 172.21.15.16:0/2016315857 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f72a80b5a50 0x7f72a81f7a30 unknown :-1 s=CLOSED pgs=1984 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:54.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:54.599+0000 7f72ae3b8640 1 -- 172.21.15.16:0/2016315857 >> 172.21.15.16:0/2016315857 conn(0x7f72a80655c0 msgr2=0x7f72a81577b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:46:54.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:54.599+0000 7f72ae3b8640 1 -- 172.21.15.16:0/2016315857 shutdown_connections 2023-05-22T15:46:54.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:54.599+0000 7f72ae3b8640 1 -- 172.21.15.16:0/2016315857 wait complete. 2023-05-22T15:46:54.627 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2023-05-22T15:46:54.760 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:54.751+0000 7f88ddb73640 1 Processor -- start 2023-05-22T15:46:54.760 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:54.751+0000 7f88ddb73640 1 -- start start 2023-05-22T15:46:54.761 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:54.755+0000 7f88ddb73640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f88d8159a90 0x7f88d80cbb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:54.761 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:54.755+0000 7f88ddb73640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f88d80cd770 0x7f88d815dbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:54.761 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:54.755+0000 7f88ddb73640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f88d815e130 0x7f88d80cdd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:54.761 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:54.755+0000 7f88ddb73640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88d815d4f0 con 0x7f88d80cd770 2023-05-22T15:46:54.761 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:54.755+0000 7f88ddb73640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88d8159e60 con 0x7f88d8159a90 2023-05-22T15:46:54.761 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:54.755+0000 7f88ddb73640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88d80cc1d0 con 0x7f88d815e130 2023-05-22T15:46:54.762 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:54.755+0000 7f88d6ffd640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f88d80cd770 0x7f88d815dbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:54.762 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:54.755+0000 7f88d6ffd640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f88d80cd770 0x7f88d815dbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43500/0 (socket says 172.21.15.16:43500) 2023-05-22T15:46:54.762 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:54.755+0000 7f88d6ffd640 1 -- 172.21.15.16:0/486339624 learned_addr learned my addr 172.21.15.16:0/486339624 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:46:54.762 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:54.755+0000 7f88d77fe640 1 --2- 172.21.15.16:0/486339624 >> v2:172.21.15.62:3300/0 conn(0x7f88d8159a90 0x7f88d80cbb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:54.762 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:54.755+0000 7f88d7fff640 1 --2- 172.21.15.16:0/486339624 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f88d815e130 0x7f88d80cdd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:54.763 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:54.755+0000 7f88d6ffd640 1 -- 172.21.15.16:0/486339624 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f88d815e130 msgr2=0x7f88d80cdd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:46:54.763 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:54.755+0000 7f88d6ffd640 1 --2- 172.21.15.16:0/486339624 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f88d815e130 0x7f88d80cdd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:54.763 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:54.755+0000 7f88d6ffd640 1 -- 172.21.15.16:0/486339624 >> v2:172.21.15.62:3300/0 conn(0x7f88d8159a90 msgr2=0x7f88d80cbb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:46:54.763 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:54.755+0000 7f88d6ffd640 1 --2- 172.21.15.16:0/486339624 >> v2:172.21.15.62:3300/0 conn(0x7f88d8159a90 0x7f88d80cbb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:54.763 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:54.755+0000 7f88d6ffd640 1 -- 172.21.15.16:0/486339624 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88d80cc4b0 con 0x7f88d80cd770 2023-05-22T15:46:54.765 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:54.755+0000 7f88d6ffd640 1 --2- 172.21.15.16:0/486339624 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f88d80cd770 0x7f88d815dbf0 secure :-1 s=READY pgs=1985 cs=0 l=1 rev1=1 crypto rx=0x7f88d80ccba0 tx=0x7f88cc01c3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32e0d9e1c47e58d1 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:54.765 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:54.755+0000 7f88d67fc640 1 -- 172.21.15.16:0/486339624 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88cc0ab600 con 0x7f88d80cd770 2023-05-22T15:46:54.765 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:54.755+0000 7f88d67fc640 1 -- 172.21.15.16:0/486339624 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88cc0a7050 con 0x7f88d80cd770 2023-05-22T15:46:54.765 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:54.755+0000 7f88d67fc640 1 -- 172.21.15.16:0/486339624 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88cc0ab980 con 0x7f88d80cd770 2023-05-22T15:46:54.765 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:54.759+0000 7f88ddb73640 1 -- 172.21.15.16:0/486339624 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f88d80cd770 msgr2=0x7f88d815dbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:54.765 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:54.759+0000 7f88ddb73640 1 --2- 172.21.15.16:0/486339624 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f88d80cd770 0x7f88d815dbf0 secure :-1 s=READY pgs=1985 cs=0 l=1 rev1=1 crypto rx=0x7f88d80ccba0 tx=0x7f88cc01c3b0 comp rx=0 tx=0).stop 2023-05-22T15:46:54.765 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:54.759+0000 7f88ddb73640 1 -- 172.21.15.16:0/486339624 shutdown_connections 2023-05-22T15:46:54.766 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:54.759+0000 7f88ddb73640 1 --2- 172.21.15.16:0/486339624 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f88d80cd770 0x7f88d815dbf0 unknown :-1 s=CLOSED pgs=1985 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:54.766 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:54.759+0000 7f88ddb73640 1 --2- 172.21.15.16:0/486339624 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f88d815e130 0x7f88d80cdd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:54.766 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:54.759+0000 7f88ddb73640 1 --2- 172.21.15.16:0/486339624 >> v2:172.21.15.62:3300/0 conn(0x7f88d8159a90 0x7f88d80cbb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:54.766 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:54.759+0000 7f88ddb73640 1 -- 172.21.15.16:0/486339624 >> 172.21.15.16:0/486339624 conn(0x7f88d806f4f0 msgr2=0x7f88d806f8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:46:54.766 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:54.759+0000 7f88ddb73640 1 -- 172.21.15.16:0/486339624 shutdown_connections 2023-05-22T15:46:54.766 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:54.759+0000 7f88ddb73640 1 -- 172.21.15.16:0/486339624 wait complete. 2023-05-22T15:46:54.766 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:54.759+0000 7f88ddb73640 1 Processor -- start 2023-05-22T15:46:54.767 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:54.759+0000 7f88ddb73640 1 -- start start 2023-05-22T15:46:54.768 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:54.759+0000 7f88ddb73640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f88d80cd770 0x7f88d81631f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:54.769 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:54.759+0000 7f88ddb73640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f88d8159a90 0x7f88d80b0fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:54.769 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:54.759+0000 7f88ddb73640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f88d815e130 0x7f88d80b1500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:54.769 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:54.759+0000 7f88ddb73640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88d814fdc0 con 0x7f88d8159a90 2023-05-22T15:46:54.769 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:54.759+0000 7f88ddb73640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88d814bd00 con 0x7f88d80cd770 2023-05-22T15:46:54.769 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:54.759+0000 7f88ddb73640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88d81525c0 con 0x7f88d815e130 2023-05-22T15:46:54.769 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:54.759+0000 7f88d6ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f88d8159a90 0x7f88d80b0fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:54.769 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:54.759+0000 7f88d6ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f88d8159a90 0x7f88d80b0fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43510/0 (socket says 172.21.15.16:43510) 2023-05-22T15:46:54.770 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:54.759+0000 7f88d6ffd640 1 -- 172.21.15.16:0/1573414229 learned_addr learned my addr 172.21.15.16:0/1573414229 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:46:54.770 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:54.759+0000 7f88d77fe640 1 --2- 172.21.15.16:0/1573414229 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f88d80cd770 0x7f88d81631f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:54.770 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:54.759+0000 7f88d7fff640 1 --2- 172.21.15.16:0/1573414229 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f88d815e130 0x7f88d80b1500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:54.770 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:54.759+0000 7f88d6ffd640 1 -- 172.21.15.16:0/1573414229 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f88d815e130 msgr2=0x7f88d80b1500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:54.770 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:54.759+0000 7f88d6ffd640 1 --2- 172.21.15.16:0/1573414229 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f88d815e130 0x7f88d80b1500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:54.770 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:54.759+0000 7f88d6ffd640 1 -- 172.21.15.16:0/1573414229 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f88d80cd770 msgr2=0x7f88d81631f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:54.770 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:54.759+0000 7f88d6ffd640 1 --2- 172.21.15.16:0/1573414229 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f88d80cd770 0x7f88d81631f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:54.771 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:54.759+0000 7f88d6ffd640 1 -- 172.21.15.16:0/1573414229 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88cc0094d0 con 0x7f88d8159a90 2023-05-22T15:46:54.771 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:54.759+0000 7f88d77fe640 1 --2- 172.21.15.16:0/1573414229 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f88d80cd770 0x7f88d81631f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T15:46:54.771 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:54.759+0000 7f88d6ffd640 1 --2- 172.21.15.16:0/1573414229 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f88d8159a90 0x7f88d80b0fc0 secure :-1 s=READY pgs=1986 cs=0 l=1 rev1=1 crypto rx=0x7f88cc01aa60 tx=0x7f88cc019150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:54.771 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:54.759+0000 7f88bffff640 1 -- 172.21.15.16:0/1573414229 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88cc0b37d0 con 0x7f88d8159a90 2023-05-22T15:46:54.771 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:54.759+0000 7f88bffff640 1 -- 172.21.15.16:0/1573414229 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88cc0b3930 con 0x7f88d8159a90 2023-05-22T15:46:54.771 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:54.759+0000 7f88ddb73640 1 -- 172.21.15.16:0/1573414229 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88d80b1c60 con 0x7f88d8159a90 2023-05-22T15:46:54.771 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:54.759+0000 7f88bffff640 1 -- 172.21.15.16:0/1573414229 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88cc0b3ca0 con 0x7f88d8159a90 2023-05-22T15:46:54.771 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:54.759+0000 7f88ddb73640 1 -- 172.21.15.16:0/1573414229 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88d80cd2e0 con 0x7f88d8159a90 2023-05-22T15:46:54.772 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:54.763+0000 7f88ddb73640 1 -- 172.21.15.16:0/1573414229 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88d80b1df0 con 0x7f88d8159a90 2023-05-22T15:46:54.772 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:54.763+0000 7f88bffff640 1 -- 172.21.15.16:0/1573414229 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 67) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f88cc0b2070 con 0x7f88d8159a90 2023-05-22T15:46:54.772 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:54.763+0000 7f88bffff640 1 --2- 172.21.15.16:0/1573414229 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f88c40654a0 0x7f88c4067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:54.772 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:54.763+0000 7f88bffff640 1 -- 172.21.15.16:0/1573414229 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(152..152 src has 1..152) v4 ==== 6289+0+0 (secure 0 0 0) 0x7f88cc11d0c0 con 0x7f88d8159a90 2023-05-22T15:46:54.772 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:54.763+0000 7f88d77fe640 1 --2- 172.21.15.16:0/1573414229 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f88c40654a0 0x7f88c4067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:54.773 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:54.763+0000 7f88d77fe640 1 --2- 172.21.15.16:0/1573414229 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f88c40654a0 0x7f88c4067950 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f88c8004660 tx=0x7f88c8008040 comp rx=0 tx=0).ready entity=mgr.9396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:54.773 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:54.767+0000 7f88bffff640 1 -- 172.21.15.16:0/1573414229 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f88cc0eaa30 con 0x7f88d8159a90 2023-05-22T15:46:54.930 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:54.923+0000 7f88ddb73640 1 -- 172.21.15.16:0/1573414229 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f88d804dbe0 con 0x7f88d8159a90 2023-05-22T15:46:54.935 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:54.931+0000 7f88bffff640 1 -- 172.21.15.16:0/1573414229 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v50) v1 ==== 104+0+0 (secure 0 0 0) 0x7f88cc0ee050 con 0x7f88d8159a90 2023-05-22T15:46:54.938 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:54.931+0000 7f88ddb73640 1 -- 172.21.15.16:0/1573414229 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f88c40654a0 msgr2=0x7f88c4067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:54.939 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:54.931+0000 7f88ddb73640 1 --2- 172.21.15.16:0/1573414229 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f88c40654a0 0x7f88c4067950 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f88c8004660 tx=0x7f88c8008040 comp rx=0 tx=0).stop 2023-05-22T15:46:54.939 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:54.931+0000 7f88ddb73640 1 -- 172.21.15.16:0/1573414229 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f88d8159a90 msgr2=0x7f88d80b0fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:54.939 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:54.931+0000 7f88ddb73640 1 --2- 172.21.15.16:0/1573414229 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f88d8159a90 0x7f88d80b0fc0 secure :-1 s=READY pgs=1986 cs=0 l=1 rev1=1 crypto rx=0x7f88cc01aa60 tx=0x7f88cc019150 comp rx=0 tx=0).stop 2023-05-22T15:46:54.939 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:54.931+0000 7f88ddb73640 1 -- 172.21.15.16:0/1573414229 shutdown_connections 2023-05-22T15:46:54.939 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:54.931+0000 7f88ddb73640 1 --2- 172.21.15.16:0/1573414229 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f88c40654a0 0x7f88c4067950 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:54.939 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:54.931+0000 7f88ddb73640 1 --2- 172.21.15.16:0/1573414229 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f88d815e130 0x7f88d80b1500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:54.940 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:54.931+0000 7f88ddb73640 1 --2- 172.21.15.16:0/1573414229 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f88d8159a90 0x7f88d80b0fc0 unknown :-1 s=CLOSED pgs=1986 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:54.940 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:54.931+0000 7f88ddb73640 1 --2- 172.21.15.16:0/1573414229 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f88d80cd770 0x7f88d81631f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:54.940 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:54.931+0000 7f88ddb73640 1 -- 172.21.15.16:0/1573414229 >> 172.21.15.16:0/1573414229 conn(0x7f88d806f4f0 msgr2=0x7f88d80b2480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:46:54.940 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:54.931+0000 7f88ddb73640 1 -- 172.21.15.16:0/1573414229 shutdown_connections 2023-05-22T15:46:54.940 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:54.931+0000 7f88ddb73640 1 -- 172.21.15.16:0/1573414229 wait complete. 2023-05-22T15:46:54.957 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2023-05-22T15:46:55.086 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:55.079+0000 7f75232de640 1 Processor -- start 2023-05-22T15:46:55.087 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:55.079+0000 7f75232de640 1 -- start start 2023-05-22T15:46:55.089 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:55.083+0000 7f75232de640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f751c159d40 0x7f751c0cb260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:55.089 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:55.083+0000 7f75232de640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f751c06b6c0 0x7f751c06bb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:55.089 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:55.083+0000 7f75232de640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f751c070c30 con 0x7f751c159970 2023-05-22T15:46:55.089 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:55.083+0000 7f75232de640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f751c15dd90 con 0x7f751c06b6c0 2023-05-22T15:46:55.089 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:55.083+0000 7f75232de640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f751c06c040 con 0x7f751c159d40 2023-05-22T15:46:55.090 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:55.083+0000 7f752187b640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f751c06b6c0 0x7f751c06bb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:55.090 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:55.083+0000 7f752187b640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f751c06b6c0 0x7f751c06bb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43516/0 (socket says 172.21.15.16:43516) 2023-05-22T15:46:55.090 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:55.083+0000 7f752187b640 1 -- 172.21.15.16:0/3044237087 learned_addr learned my addr 172.21.15.16:0/3044237087 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:46:55.090 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:55.083+0000 7f7520879640 1 --2- 172.21.15.16:0/3044237087 >> v2:172.21.15.62:3300/0 conn(0x7f751c159d40 0x7f751c0cb260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:55.090 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:55.083+0000 7f752187b640 1 -- 172.21.15.16:0/3044237087 >> v2:172.21.15.62:3300/0 conn(0x7f751c159d40 msgr2=0x7f751c0cb260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:46:55.090 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:55.083+0000 7f752187b640 1 --2- 172.21.15.16:0/3044237087 >> v2:172.21.15.62:3300/0 conn(0x7f751c159d40 0x7f751c0cb260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:55.090 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:55.083+0000 7f752187b640 1 -- 172.21.15.16:0/3044237087 >> v1:172.21.15.62:6789/0 conn(0x7f751c159970 legacy=0x7f751c15d680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:46:55.091 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:55.083+0000 7f752187b640 1 -- 172.21.15.16:0/3044237087 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f751c070e60 con 0x7f751c06b6c0 2023-05-22T15:46:55.091 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:55.083+0000 7f7520879640 1 --2- 172.21.15.16:0/3044237087 >> v2:172.21.15.62:3300/0 conn(0x7f751c159d40 0x7f751c0cb260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:46:55.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:55.083+0000 7f752187b640 1 --2- 172.21.15.16:0/3044237087 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f751c06b6c0 0x7f751c06bb00 secure :-1 s=READY pgs=1987 cs=0 l=1 rev1=1 crypto rx=0x7f751c15e760 tx=0x7f751401c430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b6d5a21c2bd8cbe server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:55.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:55.083+0000 7f751bfff640 1 -- 172.21.15.16:0/3044237087 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75140ab600 con 0x7f751c06b6c0 2023-05-22T15:46:55.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:55.083+0000 7f751bfff640 1 -- 172.21.15.16:0/3044237087 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75140a7050 con 0x7f751c06b6c0 2023-05-22T15:46:55.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:55.083+0000 7f751bfff640 1 -- 172.21.15.16:0/3044237087 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75140ab980 con 0x7f751c06b6c0 2023-05-22T15:46:55.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:55.083+0000 7f75232de640 1 -- 172.21.15.16:0/3044237087 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f751c06b6c0 msgr2=0x7f751c06bb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:55.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:55.083+0000 7f75232de640 1 --2- 172.21.15.16:0/3044237087 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f751c06b6c0 0x7f751c06bb00 secure :-1 s=READY pgs=1987 cs=0 l=1 rev1=1 crypto rx=0x7f751c15e760 tx=0x7f751401c430 comp rx=0 tx=0).stop 2023-05-22T15:46:55.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:55.083+0000 7f75232de640 1 -- 172.21.15.16:0/3044237087 shutdown_connections 2023-05-22T15:46:55.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:55.083+0000 7f75232de640 1 --2- 172.21.15.16:0/3044237087 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f751c06b6c0 0x7f751c06bb00 unknown :-1 s=CLOSED pgs=1987 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:55.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:55.083+0000 7f75232de640 1 --2- 172.21.15.16:0/3044237087 >> v2:172.21.15.62:3300/0 conn(0x7f751c159d40 0x7f751c0cb260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:55.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:55.083+0000 7f75232de640 1 -- 172.21.15.16:0/3044237087 >> 172.21.15.16:0/3044237087 conn(0x7f751c069720 msgr2=0x7f751c069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:46:55.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:55.083+0000 7f75232de640 1 -- 172.21.15.16:0/3044237087 shutdown_connections 2023-05-22T15:46:55.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:55.083+0000 7f75232de640 1 -- 172.21.15.16:0/3044237087 wait complete. 2023-05-22T15:46:55.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:55.087+0000 7f75232de640 1 Processor -- start 2023-05-22T15:46:55.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:55.087+0000 7f75232de640 1 -- start start 2023-05-22T15:46:55.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:55.087+0000 7f75232de640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f751c06b6c0 0x7f751c14be00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:55.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:55.087+0000 7f75232de640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f751c159970 0x7f751c0c3470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:55.096 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:55.087+0000 7f75232de640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f751c159d40 0x7f751c0c39b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:55.096 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:55.087+0000 7f75232de640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f751c152bc0 con 0x7f751c159d40 2023-05-22T15:46:55.096 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:55.087+0000 7f75232de640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f751c14c340 con 0x7f751c06b6c0 2023-05-22T15:46:55.096 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:55.087+0000 7f75232de640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f751c0b3880 con 0x7f751c159970 2023-05-22T15:46:55.096 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:55.087+0000 7f752187b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f751c159d40 0x7f751c0c39b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:55.097 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:55.087+0000 7f752187b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f751c159d40 0x7f751c0c39b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43528/0 (socket says 172.21.15.16:43528) 2023-05-22T15:46:55.097 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:55.087+0000 7f752187b640 1 -- 172.21.15.16:0/917131140 learned_addr learned my addr 172.21.15.16:0/917131140 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:46:55.097 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:55.087+0000 7f7520879640 1 --2- 172.21.15.16:0/917131140 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f751c159970 0x7f751c0c3470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:55.097 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:55.087+0000 7f752107a640 1 --2- 172.21.15.16:0/917131140 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f751c06b6c0 0x7f751c14be00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:55.097 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:55.087+0000 7f752187b640 1 -- 172.21.15.16:0/917131140 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f751c159970 msgr2=0x7f751c0c3470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:55.097 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:55.087+0000 7f752187b640 1 --2- 172.21.15.16:0/917131140 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f751c159970 0x7f751c0c3470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:55.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:55.087+0000 7f752187b640 1 -- 172.21.15.16:0/917131140 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f751c06b6c0 msgr2=0x7f751c14be00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:55.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:55.087+0000 7f752187b640 1 --2- 172.21.15.16:0/917131140 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f751c06b6c0 0x7f751c14be00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:55.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:55.087+0000 7f752187b640 1 -- 172.21.15.16:0/917131140 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75140094d0 con 0x7f751c159d40 2023-05-22T15:46:55.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:55.087+0000 7f752187b640 1 --2- 172.21.15.16:0/917131140 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f751c159d40 0x7f751c0c39b0 secure :-1 s=READY pgs=1988 cs=0 l=1 rev1=1 crypto rx=0x7f75140ae040 tx=0x7f75140191d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:55.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:55.087+0000 7f7519ffb640 1 -- 172.21.15.16:0/917131140 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75140ba070 con 0x7f751c159d40 2023-05-22T15:46:55.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:55.087+0000 7f7519ffb640 1 -- 172.21.15.16:0/917131140 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75140b6470 con 0x7f751c159d40 2023-05-22T15:46:55.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:55.087+0000 7f75232de640 1 -- 172.21.15.16:0/917131140 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f751c0cb7a0 con 0x7f751c159d40 2023-05-22T15:46:55.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:55.087+0000 7f7519ffb640 1 -- 172.21.15.16:0/917131140 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75140b67e0 con 0x7f751c159d40 2023-05-22T15:46:55.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:55.087+0000 7f75232de640 1 -- 172.21.15.16:0/917131140 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f751c071700 con 0x7f751c159d40 2023-05-22T15:46:55.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:55.087+0000 7f75232de640 1 -- 172.21.15.16:0/917131140 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f751c0cb930 con 0x7f751c159d40 2023-05-22T15:46:55.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:55.091+0000 7f7519ffb640 1 -- 172.21.15.16:0/917131140 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 67) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f75140b2070 con 0x7f751c159d40 2023-05-22T15:46:55.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:55.091+0000 7f7519ffb640 1 --2- 172.21.15.16:0/917131140 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f75080654a0 0x7f7508067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:55.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:55.091+0000 7f7519ffb640 1 -- 172.21.15.16:0/917131140 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(152..152 src has 1..152) v4 ==== 6289+0+0 (secure 0 0 0) 0x7f751411cb70 con 0x7f751c159d40 2023-05-22T15:46:55.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:55.091+0000 7f752107a640 1 --2- 172.21.15.16:0/917131140 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f75080654a0 0x7f7508067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:55.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:55.091+0000 7f752107a640 1 --2- 172.21.15.16:0/917131140 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f75080654a0 0x7f7508067950 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f750c005530 tx=0x7f750c009040 comp rx=0 tx=0).ready entity=mgr.9396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:55.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:55.095+0000 7f7519ffb640 1 -- 172.21.15.16:0/917131140 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f75140ea570 con 0x7f751c159d40 2023-05-22T15:46:55.254 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:55.247+0000 7f75232de640 1 -- 172.21.15.16:0/917131140 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7f751c04dbe0 con 0x7f751c159d40 2023-05-22T15:46:55.292 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:55.259+0000 7f7519ffb640 1 -- 172.21.15.16:0/917131140 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v51) v1 ==== 115+0+0 (secure 0 0 0) 0x7f75140b0020 con 0x7f751c159d40 2023-05-22T15:46:55.293 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:55.259+0000 7f75232de640 1 -- 172.21.15.16:0/917131140 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f75080654a0 msgr2=0x7f7508067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:55.294 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:55.259+0000 7f75232de640 1 --2- 172.21.15.16:0/917131140 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f75080654a0 0x7f7508067950 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f750c005530 tx=0x7f750c009040 comp rx=0 tx=0).stop 2023-05-22T15:46:55.294 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:55.259+0000 7f75232de640 1 -- 172.21.15.16:0/917131140 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f751c159d40 msgr2=0x7f751c0c39b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:55.294 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:55.259+0000 7f75232de640 1 --2- 172.21.15.16:0/917131140 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f751c159d40 0x7f751c0c39b0 secure :-1 s=READY pgs=1988 cs=0 l=1 rev1=1 crypto rx=0x7f75140ae040 tx=0x7f75140191d0 comp rx=0 tx=0).stop 2023-05-22T15:46:55.295 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:55.259+0000 7f75232de640 1 -- 172.21.15.16:0/917131140 shutdown_connections 2023-05-22T15:46:55.295 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:55.259+0000 7f75232de640 1 --2- 172.21.15.16:0/917131140 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f751c159d40 0x7f751c0c39b0 unknown :-1 s=CLOSED pgs=1988 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:55.295 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:55.259+0000 7f75232de640 1 --2- 172.21.15.16:0/917131140 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f75080654a0 0x7f7508067950 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:55.296 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:55.259+0000 7f75232de640 1 --2- 172.21.15.16:0/917131140 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f751c159970 0x7f751c0c3470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:55.296 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:55.259+0000 7f75232de640 1 --2- 172.21.15.16:0/917131140 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f751c06b6c0 0x7f751c14be00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:55.296 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:55.259+0000 7f75232de640 1 -- 172.21.15.16:0/917131140 >> 172.21.15.16:0/917131140 conn(0x7f751c069720 msgr2=0x7f751c0ce490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:46:55.296 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:55.259+0000 7f75232de640 1 -- 172.21.15.16:0/917131140 shutdown_connections 2023-05-22T15:46:55.297 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:55.259+0000 7f75232de640 1 -- 172.21.15.16:0/917131140 wait complete. 2023-05-22T15:46:55.297 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_list_post_schedule_remove' 2023-05-22T15:46:55.419 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:55.415+0000 7f74cda05640 1 Processor -- start 2023-05-22T15:46:55.419 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:55.415+0000 7f74cda05640 1 -- start start 2023-05-22T15:46:55.420 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:55.415+0000 7f74cda05640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f74c80ca4e0 0x7f74c815e460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:55.420 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:55.415+0000 7f74cda05640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74c80b0650 con 0x7f74c80c9b40 2023-05-22T15:46:55.420 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:55.415+0000 7f74cda05640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74c80b0370 con 0x7f74c8070df0 2023-05-22T15:46:55.421 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:55.415+0000 7f74cda05640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74c815d980 con 0x7f74c80ca4e0 2023-05-22T15:46:55.422 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:55.415+0000 7f74c6ffd640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f74c80ca4e0 0x7f74c815e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:55.423 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:55.415+0000 7f74c67fc640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f74c80c9b40 0x7f74c80afb40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:55586/0 (socket says 172.21.15.16:55586) 2023-05-22T15:46:55.423 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:55.415+0000 7f74c6ffd640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f74c80ca4e0 0x7f74c815e460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43536/0 (socket says 172.21.15.16:43536) 2023-05-22T15:46:55.423 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:55.415+0000 7f74c67fc640 1 -- 172.21.15.16:0/1474294275 learned_addr learned my addr 172.21.15.16:0/1474294275 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:46:55.423 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:55.415+0000 7f74c6ffd640 1 -- 172.21.15.16:0/1474294275 >> v1:172.21.15.62:6789/0 conn(0x7f74c8070df0 legacy=0x7f74c8071210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:55.423 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:55.415+0000 7f74c6ffd640 1 -- 172.21.15.16:0/1474294275 >> v1:172.21.15.16:6789/0 conn(0x7f74c80c9b40 legacy=0x7f74c80afb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:55.423 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:55.415+0000 7f74c5ffb640 1 -- 172.21.15.16:0/1474294275 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3293067698 0 0) 0x7f74c80b0650 con 0x7f74c80c9b40 2023-05-22T15:46:55.423 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:55.415+0000 7f74c6ffd640 1 -- 172.21.15.16:0/1474294275 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74c815dc60 con 0x7f74c80ca4e0 2023-05-22T15:46:55.425 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:55.419+0000 7f74c6ffd640 1 --2- 172.21.15.16:0/1474294275 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f74c80ca4e0 0x7f74c815e460 secure :-1 s=READY pgs=1990 cs=0 l=1 rev1=1 crypto rx=0x7f74b80097c0 tx=0x7f74b801c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9268b2ede5d93a0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:55.425 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:55.419+0000 7f74c5ffb640 1 -- 172.21.15.16:0/1474294275 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74b80ab600 con 0x7f74c80ca4e0 2023-05-22T15:46:55.425 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:55.419+0000 7f74c5ffb640 1 -- 172.21.15.16:0/1474294275 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74b80a7050 con 0x7f74c80ca4e0 2023-05-22T15:46:55.425 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:55.419+0000 7f74c5ffb640 1 -- 172.21.15.16:0/1474294275 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74b80ab980 con 0x7f74c80ca4e0 2023-05-22T15:46:55.425 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:55.419+0000 7f74cda05640 1 -- 172.21.15.16:0/1474294275 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f74c80ca4e0 msgr2=0x7f74c815e460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:55.425 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:55.419+0000 7f74cda05640 1 --2- 172.21.15.16:0/1474294275 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f74c80ca4e0 0x7f74c815e460 secure :-1 s=READY pgs=1990 cs=0 l=1 rev1=1 crypto rx=0x7f74b80097c0 tx=0x7f74b801c450 comp rx=0 tx=0).stop 2023-05-22T15:46:55.425 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:55.419+0000 7f74cda05640 1 -- 172.21.15.16:0/1474294275 shutdown_connections 2023-05-22T15:46:55.426 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:55.419+0000 7f74cda05640 1 --2- 172.21.15.16:0/1474294275 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f74c80ca4e0 0x7f74c815e460 unknown :-1 s=CLOSED pgs=1990 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:55.426 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:55.419+0000 7f74cda05640 1 -- 172.21.15.16:0/1474294275 >> 172.21.15.16:0/1474294275 conn(0x7f74c8069720 msgr2=0x7f74c8069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:46:55.426 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:55.419+0000 7f74cda05640 1 -- 172.21.15.16:0/1474294275 shutdown_connections 2023-05-22T15:46:55.426 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:55.419+0000 7f74cda05640 1 -- 172.21.15.16:0/1474294275 wait complete. 2023-05-22T15:46:55.426 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:55.419+0000 7f74cda05640 1 Processor -- start 2023-05-22T15:46:55.426 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:55.419+0000 7f74cda05640 1 -- start start 2023-05-22T15:46:55.429 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:55.419+0000 7f74cda05640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f74c8070df0 0x7f74c81558f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:55.429 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:55.419+0000 7f74cda05640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f74c80c9b40 0x7f74c81595e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:55.429 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:55.419+0000 7f74cda05640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f74c80ca4e0 0x7f74c8148290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:55.429 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:55.419+0000 7f74cda05640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74c814b3e0 con 0x7f74c80ca4e0 2023-05-22T15:46:55.430 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:55.419+0000 7f74cda05640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74c81532e0 con 0x7f74c8070df0 2023-05-22T15:46:55.430 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:55.419+0000 7f74cda05640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74c8153580 con 0x7f74c80c9b40 2023-05-22T15:46:55.430 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:55.419+0000 7f74c77fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f74c80ca4e0 0x7f74c8148290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:55.430 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:55.419+0000 7f74c77fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f74c80ca4e0 0x7f74c8148290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43548/0 (socket says 172.21.15.16:43548) 2023-05-22T15:46:55.430 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:55.419+0000 7f74c77fe640 1 -- 172.21.15.16:0/821263613 learned_addr learned my addr 172.21.15.16:0/821263613 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:46:55.430 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:55.419+0000 7f74c6ffd640 1 --2- 172.21.15.16:0/821263613 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f74c8070df0 0x7f74c81558f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:55.430 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:55.419+0000 7f74c67fc640 1 --2- 172.21.15.16:0/821263613 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f74c80c9b40 0x7f74c81595e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:55.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:55.419+0000 7f74c77fe640 1 -- 172.21.15.16:0/821263613 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f74c80c9b40 msgr2=0x7f74c81595e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:55.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:55.419+0000 7f74c77fe640 1 --2- 172.21.15.16:0/821263613 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f74c80c9b40 0x7f74c81595e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:55.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:55.419+0000 7f74c77fe640 1 -- 172.21.15.16:0/821263613 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f74c8070df0 msgr2=0x7f74c81558f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:55.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:55.419+0000 7f74c77fe640 1 --2- 172.21.15.16:0/821263613 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f74c8070df0 0x7f74c81558f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:55.432 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:55.419+0000 7f74c77fe640 1 -- 172.21.15.16:0/821263613 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74b8009420 con 0x7f74c80ca4e0 2023-05-22T15:46:55.432 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:55.419+0000 7f74c77fe640 1 --2- 172.21.15.16:0/821263613 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f74c80ca4e0 0x7f74c8148290 secure :-1 s=READY pgs=1991 cs=0 l=1 rev1=1 crypto rx=0x7f74c000c770 tx=0x7f74c000cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:55.433 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:55.423+0000 7f74affff640 1 -- 172.21.15.16:0/821263613 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74c0003b80 con 0x7f74c80ca4e0 2023-05-22T15:46:55.433 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:55.423+0000 7f74affff640 1 -- 172.21.15.16:0/821263613 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74c0003ce0 con 0x7f74c80ca4e0 2023-05-22T15:46:55.433 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:55.423+0000 7f74cda05640 1 -- 172.21.15.16:0/821263613 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74c80c9010 con 0x7f74c80ca4e0 2023-05-22T15:46:55.433 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:55.423+0000 7f74affff640 1 -- 172.21.15.16:0/821263613 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74c000aa90 con 0x7f74c80ca4e0 2023-05-22T15:46:55.433 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:55.423+0000 7f74cda05640 1 -- 172.21.15.16:0/821263613 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74c80c8e80 con 0x7f74c80ca4e0 2023-05-22T15:46:55.434 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:55.423+0000 7f74cda05640 1 -- 172.21.15.16:0/821263613 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74c80c91a0 con 0x7f74c80ca4e0 2023-05-22T15:46:55.434 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:55.423+0000 7f74affff640 1 -- 172.21.15.16:0/821263613 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 67) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f74c000abf0 con 0x7f74c80ca4e0 2023-05-22T15:46:55.435 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:55.423+0000 7f74affff640 1 --2- 172.21.15.16:0/821263613 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f74b40654a0 0x7f74b4067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:55.435 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:55.423+0000 7f74affff640 1 -- 172.21.15.16:0/821263613 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(152..152 src has 1..152) v4 ==== 6289+0+0 (secure 0 0 0) 0x7f74c0082950 con 0x7f74c80ca4e0 2023-05-22T15:46:55.435 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:55.423+0000 7f74c6ffd640 1 --2- 172.21.15.16:0/821263613 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f74b40654a0 0x7f74b4067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:55.435 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:55.423+0000 7f74c6ffd640 1 --2- 172.21.15.16:0/821263613 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f74b40654a0 0x7f74b4067950 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f74b80ad000 tx=0x7f74b80a6000 comp rx=0 tx=0).ready entity=mgr.9396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:55.435 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:55.427+0000 7f74affff640 1 -- 172.21.15.16:0/821263613 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f74c00503d0 con 0x7f74c80ca4e0 2023-05-22T15:46:55.643 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:55.639+0000 7f74cda05640 1 -- 172.21.15.16:0/821263613 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_list_post_schedule_remove"]} v 0) v1 -- 0x7f74c804dbe0 con 0x7f74c80ca4e0 2023-05-22T15:46:56.258 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:56.251+0000 7f74affff640 1 -- 172.21.15.16:0/821263613 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_list_post_schedule_remove"]}]=0 v775) v1 ==== 174+0+0 (secure 0 0 0) 0x7f74c00539f0 con 0x7f74c80ca4e0 2023-05-22T15:46:56.262 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:56.255+0000 7f74cda05640 1 -- 172.21.15.16:0/821263613 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f74b40654a0 msgr2=0x7f74b4067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:56.263 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:56.255+0000 7f74cda05640 1 --2- 172.21.15.16:0/821263613 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f74b40654a0 0x7f74b4067950 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f74b80ad000 tx=0x7f74b80a6000 comp rx=0 tx=0).stop 2023-05-22T15:46:56.263 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:56.255+0000 7f74cda05640 1 -- 172.21.15.16:0/821263613 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f74c80ca4e0 msgr2=0x7f74c8148290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:56.263 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:56.255+0000 7f74cda05640 1 --2- 172.21.15.16:0/821263613 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f74c80ca4e0 0x7f74c8148290 secure :-1 s=READY pgs=1991 cs=0 l=1 rev1=1 crypto rx=0x7f74c000c770 tx=0x7f74c000cc30 comp rx=0 tx=0).stop 2023-05-22T15:46:56.264 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:56.259+0000 7f74cda05640 1 -- 172.21.15.16:0/821263613 shutdown_connections 2023-05-22T15:46:56.264 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:56.259+0000 7f74cda05640 1 --2- 172.21.15.16:0/821263613 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f74c80ca4e0 0x7f74c8148290 unknown :-1 s=CLOSED pgs=1991 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:56.264 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:56.259+0000 7f74cda05640 1 --2- 172.21.15.16:0/821263613 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f74b40654a0 0x7f74b4067950 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:56.264 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:56.259+0000 7f74cda05640 1 --2- 172.21.15.16:0/821263613 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f74c80c9b40 0x7f74c81595e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:56.264 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:56.259+0000 7f74cda05640 1 --2- 172.21.15.16:0/821263613 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f74c8070df0 0x7f74c81558f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:56.264 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:56.259+0000 7f74cda05640 1 -- 172.21.15.16:0/821263613 >> 172.21.15.16:0/821263613 conn(0x7f74c8069720 msgr2=0x7f74c8152140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:46:56.265 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:56.259+0000 7f74cda05640 1 -- 172.21.15.16:0/821263613 shutdown_connections 2023-05-22T15:46:56.265 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:56.259+0000 7f74cda05640 1 -- 172.21.15.16:0/821263613 wait complete. 2023-05-22T15:46:56.286 INFO:tasks.cephfs_test_runner:Test listing snap schedules post removal of a schedule ... ok 2023-05-22T15:46:56.287 INFO:tasks.cephfs_test_runner:Starting test: test_snap_schedule_with_mgr_restart (tasks.cephfs.test_snap_schedules.TestSnapSchedules) Test that snap schedule is resumed after mgr restart 2023-05-22T15:46:56.287 INFO:tasks.cephfs_test_runner:test_snap_schedule_with_mgr_restart (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2023-05-22T15:46:56.287 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_mgr_restart' 2023-05-22T15:46:56.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:56.411+0000 7efe9487b640 1 Processor -- start 2023-05-22T15:46:56.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:56.411+0000 7efe9487b640 1 -- start start 2023-05-22T15:46:56.417 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:56.411+0000 7efe9487b640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7efe900b7dd0 0x7efe900ccab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:56.417 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:56.411+0000 7efe9487b640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe900cd170 con 0x7efe90160220 2023-05-22T15:46:56.417 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:56.411+0000 7efe9487b640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe900d2ac0 con 0x7efe9015f700 2023-05-22T15:46:56.417 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:56.411+0000 7efe9487b640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe900d3360 con 0x7efe900b7dd0 2023-05-22T15:46:56.418 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:56.411+0000 7efe8ed9d640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7efe90160220 0x7efe900d0410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:55592/0 (socket says 172.21.15.16:55592) 2023-05-22T15:46:56.418 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:56.411+0000 7efe8ed9d640 1 -- 172.21.15.16:0/2397432817 learned_addr learned my addr 172.21.15.16:0/2397432817 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:46:56.420 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:56.411+0000 7efe8e59c640 1 --2- 172.21.15.16:0/2397432817 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7efe900b7dd0 0x7efe900ccab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:56.420 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:56.411+0000 7efe8d59a640 1 -- 172.21.15.16:0/2397432817 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4092224615 0 0) 0x7efe900cd170 con 0x7efe90160220 2023-05-22T15:46:56.420 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:56.411+0000 7efe8d59a640 1 -- 172.21.15.16:0/2397432817 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe7c003530 con 0x7efe90160220 2023-05-22T15:46:56.420 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:56.411+0000 7efe8d59a640 1 -- 172.21.15.16:0/2397432817 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2326641496 0 0) 0x7efe7c003530 con 0x7efe90160220 2023-05-22T15:46:56.421 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:56.411+0000 7efe8d59a640 1 -- 172.21.15.16:0/2397432817 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7efe900b7dd0 msgr2=0x7efe900ccab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:46:56.421 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:56.411+0000 7efe8d59a640 1 --2- 172.21.15.16:0/2397432817 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7efe900b7dd0 0x7efe900ccab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:56.421 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:56.411+0000 7efe8d59a640 1 -- 172.21.15.16:0/2397432817 >> v1:172.21.15.62:6789/0 conn(0x7efe9015f700 legacy=0x7efe9015fb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:56.421 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:56.411+0000 7efe8d59a640 1 -- 172.21.15.16:0/2397432817 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe900d2da0 con 0x7efe90160220 2023-05-22T15:46:56.421 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:56.411+0000 7efe8d59a640 1 -- 172.21.15.16:0/2397432817 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7efe88001680 con 0x7efe90160220 2023-05-22T15:46:56.421 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:56.411+0000 7efe8d59a640 1 -- 172.21.15.16:0/2397432817 >> v1:172.21.15.16:6789/0 conn(0x7efe90160220 legacy=0x7efe900d0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:56.422 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:56.411+0000 7efe8d59a640 1 --2- 172.21.15.16:0/2397432817 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7efe900b7dd0 0x7efe7c0039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:56.422 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:56.411+0000 7efe8d59a640 1 --2- 172.21.15.16:0/2397432817 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efe7c003f30 0x7efe7c0047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:56.422 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:56.411+0000 7efe8d59a640 1 --2- 172.21.15.16:0/2397432817 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efe7c004d10 0x7efe7c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:56.422 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:56.411+0000 7efe8d59a640 1 -- 172.21.15.16:0/2397432817 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe900d3360 con 0x7efe7c003f30 2023-05-22T15:46:56.422 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:56.411+0000 7efe8d59a640 1 -- 172.21.15.16:0/2397432817 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe7c00bb20 con 0x7efe7c004d10 2023-05-22T15:46:56.422 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:56.411+0000 7efe8d59a640 1 -- 172.21.15.16:0/2397432817 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe7c00bce0 con 0x7efe900b7dd0 2023-05-22T15:46:56.422 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:56.411+0000 7efe8dd9b640 1 --2- 172.21.15.16:0/2397432817 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efe7c003f30 0x7efe7c0047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:56.423 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:56.411+0000 7efe8ed9d640 1 --2- 172.21.15.16:0/2397432817 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efe7c004d10 0x7efe7c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:56.423 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:56.411+0000 7efe8dd9b640 1 -- 172.21.15.16:0/2397432817 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7efe900b7dd0 msgr2=0x7efe7c0039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:46:56.423 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:56.411+0000 7efe8e59c640 1 --2- 172.21.15.16:0/2397432817 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7efe900b7dd0 0x7efe7c0039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:56.423 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:56.411+0000 7efe8dd9b640 1 --2- 172.21.15.16:0/2397432817 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7efe900b7dd0 0x7efe7c0039b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:56.423 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:56.411+0000 7efe8dd9b640 1 -- 172.21.15.16:0/2397432817 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efe7c004d10 msgr2=0x7efe7c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:46:56.423 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:56.411+0000 7efe8dd9b640 1 --2- 172.21.15.16:0/2397432817 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efe7c004d10 0x7efe7c005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:56.423 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:56.411+0000 7efe8dd9b640 1 -- 172.21.15.16:0/2397432817 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe7c003530 con 0x7efe7c003f30 2023-05-22T15:46:56.424 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:56.411+0000 7efe8ed9d640 1 --2- 172.21.15.16:0/2397432817 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efe7c004d10 0x7efe7c005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:46:56.428 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:56.415+0000 7efe8dd9b640 1 --2- 172.21.15.16:0/2397432817 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efe7c003f30 0x7efe7c0047d0 secure :-1 s=READY pgs=1993 cs=0 l=1 rev1=1 crypto rx=0x7efe78001e40 tx=0x7efe7801c710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22a40e1bf6a698e server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:56.428 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:56.415+0000 7efe8d59a640 1 -- 172.21.15.16:0/2397432817 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe78003b80 con 0x7efe7c003f30 2023-05-22T15:46:56.428 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:56.415+0000 7efe8d59a640 1 -- 172.21.15.16:0/2397432817 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe78003ce0 con 0x7efe7c003f30 2023-05-22T15:46:56.428 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:56.415+0000 7efe8d59a640 1 -- 172.21.15.16:0/2397432817 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe780052c0 con 0x7efe7c003f30 2023-05-22T15:46:56.428 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:56.415+0000 7efe9487b640 1 -- 172.21.15.16:0/2397432817 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efe7c003f30 msgr2=0x7efe7c0047d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:56.428 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:56.415+0000 7efe9487b640 1 --2- 172.21.15.16:0/2397432817 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efe7c003f30 0x7efe7c0047d0 secure :-1 s=READY pgs=1993 cs=0 l=1 rev1=1 crypto rx=0x7efe78001e40 tx=0x7efe7801c710 comp rx=0 tx=0).stop 2023-05-22T15:46:56.429 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:56.415+0000 7efe8ed9d640 1 -- 172.21.15.16:0/2397432817 reap_dead start 2023-05-22T15:46:56.429 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:56.415+0000 7efe9487b640 1 -- 172.21.15.16:0/2397432817 shutdown_connections 2023-05-22T15:46:56.429 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:56.415+0000 7efe9487b640 1 -- 172.21.15.16:0/2397432817 >> 172.21.15.16:0/2397432817 conn(0x7efe900655c0 msgr2=0x7efe90070e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:46:56.429 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:56.415+0000 7efe9487b640 1 -- 172.21.15.16:0/2397432817 shutdown_connections 2023-05-22T15:46:56.429 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:56.415+0000 7efe9487b640 1 -- 172.21.15.16:0/2397432817 wait complete. 2023-05-22T15:46:56.430 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:56.415+0000 7efe9487b640 1 Processor -- start 2023-05-22T15:46:56.430 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:56.415+0000 7efe9487b640 1 -- start start 2023-05-22T15:46:56.430 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:56.419+0000 7efe9487b640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7efe901f3dd0 0x7efe900c7f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:56.430 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:56.419+0000 7efe9487b640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efe901f41a0 0x7efe900c84b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:56.430 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:56.419+0000 7efe9487b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efe9015c9b0 0x7efe9015ce50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:56.430 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:56.419+0000 7efe9487b640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe9015e330 con 0x7efe9015c9b0 2023-05-22T15:46:56.430 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:56.419+0000 7efe9487b640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe9015d720 con 0x7efe901f41a0 2023-05-22T15:46:56.430 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:56.419+0000 7efe9487b640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe9015da00 con 0x7efe901f3dd0 2023-05-22T15:46:56.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:56.419+0000 7efe8ed9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efe9015c9b0 0x7efe9015ce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:56.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:56.419+0000 7efe8ed9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efe9015c9b0 0x7efe9015ce50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43576/0 (socket says 172.21.15.16:43576) 2023-05-22T15:46:56.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:56.419+0000 7efe8ed9d640 1 -- 172.21.15.16:0/3857205553 learned_addr learned my addr 172.21.15.16:0/3857205553 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:46:56.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:56.419+0000 7efe8e59c640 1 --2- 172.21.15.16:0/3857205553 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7efe901f3dd0 0x7efe900c7f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:56.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:56.419+0000 7efe8dd9b640 1 --2- 172.21.15.16:0/3857205553 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efe901f41a0 0x7efe900c84b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:56.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:56.419+0000 7efe8ed9d640 1 -- 172.21.15.16:0/3857205553 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7efe901f3dd0 msgr2=0x7efe900c7f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:56.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:56.419+0000 7efe8ed9d640 1 --2- 172.21.15.16:0/3857205553 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7efe901f3dd0 0x7efe900c7f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:56.432 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:56.419+0000 7efe8ed9d640 1 -- 172.21.15.16:0/3857205553 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efe901f41a0 msgr2=0x7efe900c84b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:56.432 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:56.419+0000 7efe8ed9d640 1 --2- 172.21.15.16:0/3857205553 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efe901f41a0 0x7efe900c84b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:56.432 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:56.419+0000 7efe8ed9d640 1 -- 172.21.15.16:0/3857205553 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe900d1330 con 0x7efe9015c9b0 2023-05-22T15:46:56.432 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:56.419+0000 7efe8e59c640 1 --2- 172.21.15.16:0/3857205553 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7efe901f3dd0 0x7efe900c7f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:46:56.432 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:56.419+0000 7efe8dd9b640 1 --2- 172.21.15.16:0/3857205553 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efe901f41a0 0x7efe900c84b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:46:56.432 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:56.419+0000 7efe8ed9d640 1 --2- 172.21.15.16:0/3857205553 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efe9015c9b0 0x7efe9015ce50 secure :-1 s=READY pgs=1994 cs=0 l=1 rev1=1 crypto rx=0x7efe9015f640 tx=0x7efe8800d770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:56.433 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:56.419+0000 7efe86ffd640 1 -- 172.21.15.16:0/3857205553 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe880048b0 con 0x7efe9015c9b0 2023-05-22T15:46:56.433 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:56.419+0000 7efe86ffd640 1 -- 172.21.15.16:0/3857205553 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe88004a30 con 0x7efe9015c9b0 2023-05-22T15:46:56.433 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:56.419+0000 7efe86ffd640 1 -- 172.21.15.16:0/3857205553 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe88004de0 con 0x7efe9015c9b0 2023-05-22T15:46:56.433 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:56.419+0000 7efe9487b640 1 -- 172.21.15.16:0/3857205553 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe900cf730 con 0x7efe9015c9b0 2023-05-22T15:46:56.433 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:56.419+0000 7efe9487b640 1 -- 172.21.15.16:0/3857205553 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe900d14c0 con 0x7efe9015c9b0 2023-05-22T15:46:56.433 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:56.419+0000 7efe9487b640 1 -- 172.21.15.16:0/3857205553 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe9004dbe0 con 0x7efe9015c9b0 2023-05-22T15:46:56.433 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:56.419+0000 7efe86ffd640 1 -- 172.21.15.16:0/3857205553 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 67) v1 ==== 84332+0+0 (secure 0 0 0) 0x7efe88007070 con 0x7efe9015c9b0 2023-05-22T15:46:56.433 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:56.423+0000 7efe86ffd640 1 --2- 172.21.15.16:0/3857205553 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7efe700654d0 0x7efe70067980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:56.434 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:56.423+0000 7efe86ffd640 1 -- 172.21.15.16:0/3857205553 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(152..152 src has 1..152) v4 ==== 6289+0+0 (secure 0 0 0) 0x7efe88081d00 con 0x7efe9015c9b0 2023-05-22T15:46:56.434 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:56.423+0000 7efe8e59c640 1 --2- 172.21.15.16:0/3857205553 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7efe700654d0 0x7efe70067980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:56.434 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:56.423+0000 7efe8e59c640 1 --2- 172.21.15.16:0/3857205553 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7efe700654d0 0x7efe70067980 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7efe80009a00 tx=0x7efe80008040 comp rx=0 tx=0).ready entity=mgr.9396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:56.434 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:56.427+0000 7efe86ffd640 1 -- 172.21.15.16:0/3857205553 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7efe8804f570 con 0x7efe9015c9b0 2023-05-22T15:46:56.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:56.627+0000 7efe9487b640 1 -- 172.21.15.16:0/3857205553 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_mgr_restart"]} v 0) v1 -- 0x7efe90071060 con 0x7efe9015c9b0 2023-05-22T15:46:57.257 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:57.251+0000 7efe86ffd640 1 -- 172.21.15.16:0/3857205553 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_mgr_restart"]}]=0 v776) v1 ==== 168+0+0 (secure 0 0 0) 0x7efe8801a020 con 0x7efe9015c9b0 2023-05-22T15:46:57.261 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:57.255+0000 7efe9487b640 1 -- 172.21.15.16:0/3857205553 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7efe700654d0 msgr2=0x7efe70067980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:57.261 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:57.255+0000 7efe9487b640 1 --2- 172.21.15.16:0/3857205553 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7efe700654d0 0x7efe70067980 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7efe80009a00 tx=0x7efe80008040 comp rx=0 tx=0).stop 2023-05-22T15:46:57.261 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:57.255+0000 7efe9487b640 1 -- 172.21.15.16:0/3857205553 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efe9015c9b0 msgr2=0x7efe9015ce50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:57.261 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:57.255+0000 7efe9487b640 1 --2- 172.21.15.16:0/3857205553 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efe9015c9b0 0x7efe9015ce50 secure :-1 s=READY pgs=1994 cs=0 l=1 rev1=1 crypto rx=0x7efe9015f640 tx=0x7efe8800d770 comp rx=0 tx=0).stop 2023-05-22T15:46:57.261 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:57.255+0000 7efe9487b640 1 -- 172.21.15.16:0/3857205553 shutdown_connections 2023-05-22T15:46:57.262 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:57.255+0000 7efe9487b640 1 --2- 172.21.15.16:0/3857205553 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efe9015c9b0 0x7efe9015ce50 unknown :-1 s=CLOSED pgs=1994 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:57.262 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:57.255+0000 7efe9487b640 1 --2- 172.21.15.16:0/3857205553 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efe901f41a0 0x7efe900c84b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:57.262 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:57.255+0000 7efe9487b640 1 --2- 172.21.15.16:0/3857205553 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7efe700654d0 0x7efe70067980 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:57.262 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:57.255+0000 7efe9487b640 1 --2- 172.21.15.16:0/3857205553 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7efe901f3dd0 0x7efe900c7f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:57.262 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:57.255+0000 7efe9487b640 1 -- 172.21.15.16:0/3857205553 >> 172.21.15.16:0/3857205553 conn(0x7efe900655c0 msgr2=0x7efe900cb540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:46:57.262 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:57.255+0000 7efe9487b640 1 -- 172.21.15.16:0/3857205553 shutdown_connections 2023-05-22T15:46:57.262 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:57.255+0000 7efe9487b640 1 -- 172.21.15.16:0/3857205553 wait complete. 2023-05-22T15:46:57.283 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-05-22T15:46:57.415 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:57.411+0000 7fc77d7e5640 1 Processor -- start 2023-05-22T15:46:57.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:57.411+0000 7fc77d7e5640 1 -- start start 2023-05-22T15:46:57.417 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:57.411+0000 7fc77d7e5640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fc778159970 0x7fc778159d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:57.417 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:57.411+0000 7fc77d7e5640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc778070c30 con 0x7fc77815db00 2023-05-22T15:46:57.417 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:57.411+0000 7fc77d7e5640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc778070e60 con 0x7fc77815d680 2023-05-22T15:46:57.417 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:57.411+0000 7fc77d7e5640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7780cb970 con 0x7fc778159970 2023-05-22T15:46:57.417 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:57.411+0000 7fc7777fe640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fc77815db00 0x7fc77806b6c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:55604/0 (socket says 172.21.15.16:55604) 2023-05-22T15:46:57.418 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:57.411+0000 7fc7777fe640 1 -- 172.21.15.16:0/3123948897 learned_addr learned my addr 172.21.15.16:0/3123948897 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:46:57.418 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:57.411+0000 7fc776ffd640 1 --2- 172.21.15.16:0/3123948897 >> v2:172.21.15.62:3300/0 conn(0x7fc778159970 0x7fc778159d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:57.418 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:57.411+0000 7fc775ffb640 1 -- 172.21.15.16:0/3123948897 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2020910043 0 0) 0x7fc778070c30 con 0x7fc77815db00 2023-05-22T15:46:57.418 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:57.411+0000 7fc775ffb640 1 -- 172.21.15.16:0/3123948897 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc760003530 con 0x7fc77815db00 2023-05-22T15:46:57.420 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:57.411+0000 7fc775ffb640 1 -- 172.21.15.16:0/3123948897 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2550920624 0 0) 0x7fc760003530 con 0x7fc77815db00 2023-05-22T15:46:57.420 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:57.411+0000 7fc775ffb640 1 -- 172.21.15.16:0/3123948897 >> v2:172.21.15.62:3300/0 conn(0x7fc778159970 msgr2=0x7fc778159d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:46:57.420 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:57.411+0000 7fc775ffb640 1 --2- 172.21.15.16:0/3123948897 >> v2:172.21.15.62:3300/0 conn(0x7fc778159970 0x7fc778159d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:57.420 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:57.411+0000 7fc775ffb640 1 -- 172.21.15.16:0/3123948897 >> v1:172.21.15.62:6789/0 conn(0x7fc77815d680 legacy=0x7fc7780cb260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:57.421 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:57.411+0000 7fc775ffb640 1 -- 172.21.15.16:0/3123948897 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc778071140 con 0x7fc77815db00 2023-05-22T15:46:57.421 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:57.411+0000 7fc775ffb640 1 -- 172.21.15.16:0/3123948897 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7fc770001a00 con 0x7fc77815db00 2023-05-22T15:46:57.421 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:57.411+0000 7fc775ffb640 1 -- 172.21.15.16:0/3123948897 >> v1:172.21.15.16:6789/0 conn(0x7fc77815db00 legacy=0x7fc77806b6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:57.421 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:57.411+0000 7fc775ffb640 1 --2- 172.21.15.16:0/3123948897 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc760003980 0x7fc760003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:57.421 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:57.411+0000 7fc775ffb640 1 --2- 172.21.15.16:0/3123948897 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc7600047d0 0x7fc760004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:57.421 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:57.411+0000 7fc775ffb640 1 --2- 172.21.15.16:0/3123948897 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc760005180 0x7fc76000b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:57.421 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:57.411+0000 7fc775ffb640 1 -- 172.21.15.16:0/3123948897 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7780cb970 con 0x7fc760003980 2023-05-22T15:46:57.422 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:57.411+0000 7fc775ffb640 1 -- 172.21.15.16:0/3123948897 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc76000c090 con 0x7fc760005180 2023-05-22T15:46:57.422 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:57.411+0000 7fc775ffb640 1 -- 172.21.15.16:0/3123948897 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc76000c250 con 0x7fc7600047d0 2023-05-22T15:46:57.422 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:57.411+0000 7fc776ffd640 1 --2- 172.21.15.16:0/3123948897 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc760003980 0x7fc760003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:57.422 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:57.411+0000 7fc7777fe640 1 --2- 172.21.15.16:0/3123948897 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc760005180 0x7fc76000b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:57.422 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:57.411+0000 7fc776ffd640 1 -- 172.21.15.16:0/3123948897 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc7600047d0 msgr2=0x7fc760004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:46:57.422 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:57.411+0000 7fc7767fc640 1 --2- 172.21.15.16:0/3123948897 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc7600047d0 0x7fc760004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:57.423 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:57.411+0000 7fc776ffd640 1 --2- 172.21.15.16:0/3123948897 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc7600047d0 0x7fc760004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:57.423 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:57.411+0000 7fc776ffd640 1 -- 172.21.15.16:0/3123948897 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc760005180 msgr2=0x7fc76000b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:46:57.423 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:57.411+0000 7fc776ffd640 1 --2- 172.21.15.16:0/3123948897 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc760005180 0x7fc76000b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:57.423 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:57.411+0000 7fc776ffd640 1 -- 172.21.15.16:0/3123948897 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc760003530 con 0x7fc760003980 2023-05-22T15:46:57.423 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:57.411+0000 7fc7777fe640 1 --2- 172.21.15.16:0/3123948897 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc760005180 0x7fc76000b990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:46:57.423 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:57.411+0000 7fc7777fe640 1 -- 172.21.15.16:0/3123948897 reap_dead start 2023-05-22T15:46:57.425 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:57.415+0000 7fc776ffd640 1 --2- 172.21.15.16:0/3123948897 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc760003980 0x7fc760003d90 secure :-1 s=READY pgs=1996 cs=0 l=1 rev1=1 crypto rx=0x7fc768009ae0 tx=0x7fc76801ca20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=217dcfc78fb1cbd2 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:57.426 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:57.415+0000 7fc775ffb640 1 -- 172.21.15.16:0/3123948897 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7680a6510 con 0x7fc760003980 2023-05-22T15:46:57.426 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:57.415+0000 7fc775ffb640 1 -- 172.21.15.16:0/3123948897 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7680a6670 con 0x7fc760003980 2023-05-22T15:46:57.426 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:57.415+0000 7fc775ffb640 1 -- 172.21.15.16:0/3123948897 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7680a6a20 con 0x7fc760003980 2023-05-22T15:46:57.426 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:57.415+0000 7fc77d7e5640 1 -- 172.21.15.16:0/3123948897 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc760003980 msgr2=0x7fc760003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:57.426 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:57.415+0000 7fc77d7e5640 1 --2- 172.21.15.16:0/3123948897 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc760003980 0x7fc760003d90 secure :-1 s=READY pgs=1996 cs=0 l=1 rev1=1 crypto rx=0x7fc768009ae0 tx=0x7fc76801ca20 comp rx=0 tx=0).stop 2023-05-22T15:46:57.427 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:57.415+0000 7fc77d7e5640 1 -- 172.21.15.16:0/3123948897 shutdown_connections 2023-05-22T15:46:57.427 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:57.415+0000 7fc77d7e5640 1 --2- 172.21.15.16:0/3123948897 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc760003980 0x7fc760003d90 unknown :-1 s=CLOSED pgs=1996 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:57.427 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:57.415+0000 7fc77d7e5640 1 -- 172.21.15.16:0/3123948897 >> 172.21.15.16:0/3123948897 conn(0x7fc778069720 msgr2=0x7fc778069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:46:57.427 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:57.415+0000 7fc77d7e5640 1 -- 172.21.15.16:0/3123948897 shutdown_connections 2023-05-22T15:46:57.427 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:57.415+0000 7fc77d7e5640 1 -- 172.21.15.16:0/3123948897 wait complete. 2023-05-22T15:46:57.427 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:57.415+0000 7fc77d7e5640 1 Processor -- start 2023-05-22T15:46:57.427 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:57.415+0000 7fc77d7e5640 1 -- start start 2023-05-22T15:46:57.428 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:57.419+0000 7fc77d7e5640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc760003980 0x7fc778156e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:57.428 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:57.419+0000 7fc77d7e5640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc778157550 0x7fc77814b570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:57.428 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:57.419+0000 7fc77d7e5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc7780afc80 0x7fc7780b00f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:57.428 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:57.419+0000 7fc77d7e5640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7780b0630 con 0x7fc7780afc80 2023-05-22T15:46:57.428 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:57.419+0000 7fc77d7e5640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc77814bab0 con 0x7fc760003980 2023-05-22T15:46:57.428 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:57.419+0000 7fc77d7e5640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc778158660 con 0x7fc778157550 2023-05-22T15:46:57.428 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:57.419+0000 7fc7777fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc7780afc80 0x7fc7780b00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:57.429 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:57.419+0000 7fc7777fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc7780afc80 0x7fc7780b00f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43594/0 (socket says 172.21.15.16:43594) 2023-05-22T15:46:57.429 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:57.419+0000 7fc7777fe640 1 -- 172.21.15.16:0/1227690005 learned_addr learned my addr 172.21.15.16:0/1227690005 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:46:57.429 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:57.419+0000 7fc7767fc640 1 --2- 172.21.15.16:0/1227690005 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc778157550 0x7fc77814b570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:57.429 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:57.419+0000 7fc776ffd640 1 --2- 172.21.15.16:0/1227690005 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc760003980 0x7fc778156e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:57.429 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:57.419+0000 7fc7777fe640 1 -- 172.21.15.16:0/1227690005 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc778157550 msgr2=0x7fc77814b570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:57.429 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:57.419+0000 7fc7777fe640 1 --2- 172.21.15.16:0/1227690005 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc778157550 0x7fc77814b570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:57.430 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:57.419+0000 7fc7777fe640 1 -- 172.21.15.16:0/1227690005 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc760003980 msgr2=0x7fc778156e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:57.430 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:57.419+0000 7fc7777fe640 1 --2- 172.21.15.16:0/1227690005 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc760003980 0x7fc778156e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:57.430 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:57.419+0000 7fc7777fe640 1 -- 172.21.15.16:0/1227690005 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc77806bed0 con 0x7fc7780afc80 2023-05-22T15:46:57.430 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:57.419+0000 7fc7777fe640 1 --2- 172.21.15.16:0/1227690005 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc7780afc80 0x7fc7780b00f0 secure :-1 s=READY pgs=1997 cs=0 l=1 rev1=1 crypto rx=0x7fc7780707a0 tx=0x7fc77000dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:57.430 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:57.419+0000 7fc75f7fe640 1 -- 172.21.15.16:0/1227690005 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc770007070 con 0x7fc7780afc80 2023-05-22T15:46:57.430 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:57.419+0000 7fc75f7fe640 1 -- 172.21.15.16:0/1227690005 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7700071d0 con 0x7fc7780afc80 2023-05-22T15:46:57.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:57.419+0000 7fc75f7fe640 1 -- 172.21.15.16:0/1227690005 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc770007580 con 0x7fc7780afc80 2023-05-22T15:46:57.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:57.419+0000 7fc77d7e5640 1 -- 172.21.15.16:0/1227690005 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7780cb670 con 0x7fc7780afc80 2023-05-22T15:46:57.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:57.419+0000 7fc77d7e5640 1 -- 172.21.15.16:0/1227690005 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc77806c060 con 0x7fc7780afc80 2023-05-22T15:46:57.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:57.419+0000 7fc77d7e5640 1 -- 172.21.15.16:0/1227690005 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7781587d0 con 0x7fc7780afc80 2023-05-22T15:46:57.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:57.419+0000 7fc75f7fe640 1 -- 172.21.15.16:0/1227690005 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 67) v1 ==== 84332+0+0 (secure 0 0 0) 0x7fc7700076e0 con 0x7fc7780afc80 2023-05-22T15:46:57.432 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:57.419+0000 7fc75f7fe640 1 --2- 172.21.15.16:0/1227690005 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fc7640654a0 0x7fc764067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:57.432 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:57.419+0000 7fc75f7fe640 1 -- 172.21.15.16:0/1227690005 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(152..152 src has 1..152) v4 ==== 6289+0+0 (secure 0 0 0) 0x7fc7700821c0 con 0x7fc7780afc80 2023-05-22T15:46:57.432 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:57.419+0000 7fc776ffd640 1 --2- 172.21.15.16:0/1227690005 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fc7640654a0 0x7fc764067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:57.432 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:57.419+0000 7fc776ffd640 1 --2- 172.21.15.16:0/1227690005 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fc7640654a0 0x7fc764067950 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fc7680af000 tx=0x7fc7680046d0 comp rx=0 tx=0).ready entity=mgr.9396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:57.433 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:57.423+0000 7fc75f7fe640 1 -- 172.21.15.16:0/1227690005 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fc77004fbc0 con 0x7fc7780afc80 2023-05-22T15:46:57.592 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:57.583+0000 7fc77d7e5640 1 -- 172.21.15.16:0/1227690005 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fc77804dbe0 con 0x7fc7780afc80 2023-05-22T15:46:57.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:57.591+0000 7fc75f7fe640 1 -- 172.21.15.16:0/1227690005 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v52)=0 v52) v1 ==== 122+0+0 (secure 0 0 0) 0x7fc7700531e0 con 0x7fc7780afc80 2023-05-22T15:46:57.601 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:57.595+0000 7fc77d7e5640 1 -- 172.21.15.16:0/1227690005 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fc7640654a0 msgr2=0x7fc764067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:57.601 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:57.595+0000 7fc77d7e5640 1 --2- 172.21.15.16:0/1227690005 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fc7640654a0 0x7fc764067950 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fc7680af000 tx=0x7fc7680046d0 comp rx=0 tx=0).stop 2023-05-22T15:46:57.602 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:57.595+0000 7fc77d7e5640 1 -- 172.21.15.16:0/1227690005 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc7780afc80 msgr2=0x7fc7780b00f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:57.602 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:57.595+0000 7fc77d7e5640 1 --2- 172.21.15.16:0/1227690005 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc7780afc80 0x7fc7780b00f0 secure :-1 s=READY pgs=1997 cs=0 l=1 rev1=1 crypto rx=0x7fc7780707a0 tx=0x7fc77000dc60 comp rx=0 tx=0).stop 2023-05-22T15:46:57.602 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:57.595+0000 7fc77d7e5640 1 -- 172.21.15.16:0/1227690005 shutdown_connections 2023-05-22T15:46:57.602 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:57.595+0000 7fc77d7e5640 1 --2- 172.21.15.16:0/1227690005 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc7780afc80 0x7fc7780b00f0 unknown :-1 s=CLOSED pgs=1997 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:57.603 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:57.595+0000 7fc77d7e5640 1 --2- 172.21.15.16:0/1227690005 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fc7640654a0 0x7fc764067950 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:57.603 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:57.595+0000 7fc77d7e5640 1 --2- 172.21.15.16:0/1227690005 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc778157550 0x7fc77814b570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:57.603 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:57.595+0000 7fc77d7e5640 1 --2- 172.21.15.16:0/1227690005 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc760003980 0x7fc778156e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:57.603 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:57.595+0000 7fc77d7e5640 1 -- 172.21.15.16:0/1227690005 >> 172.21.15.16:0/1227690005 conn(0x7fc778069720 msgr2=0x7fc7780ca050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:46:57.603 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:57.595+0000 7fc77d7e5640 1 -- 172.21.15.16:0/1227690005 shutdown_connections 2023-05-22T15:46:57.603 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:57.595+0000 7fc77d7e5640 1 -- 172.21.15.16:0/1227690005 wait complete. 2023-05-22T15:46:57.618 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-05-22T15:46:57.619 DEBUG:teuthology.orchestra.run.smithi016:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-05-22T15:46:57.669 DEBUG:teuthology.orchestra.run.smithi062:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-05-22T15:46:57.723 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-05-22T15:46:57.724 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-05-22T15:46:57.724 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2023-05-22T15:46:57.731 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16345192k,nr_inodes=4086298,mode=755,inode64 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3276544k,mode=755,inode64 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev,inode64 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=15732 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3276540k,nr_inodes=819135,mode=700,uid=1000,gid=1271,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3276544k,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-05-22T15:46:57.731 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2023-05-22T15:46:57.732 DEBUG:teuthology.orchestra.run.smithi062:> dd if=/proc/self/mounts of=/dev/stdout 2023-05-22T15:46:57.739 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16342116k,nr_inodes=4085529,mode=755,inode64 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275932k,mode=755,inode64 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev,inode64 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=13250 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275928k,nr_inodes=818982,mode=700,uid=1000,gid=1271,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275932k,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-05-22T15:46:57.741 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-05-22T15:46:57.741 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2023-05-22T15:46:57.781 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16345192k,nr_inodes=4086298,mode=755,inode64 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3276544k,mode=755,inode64 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev,inode64 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=15732 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3276540k,nr_inodes=819135,mode=700,uid=1000,gid=1271,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3276544k,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-05-22T15:46:57.781 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2023-05-22T15:46:57.782 DEBUG:teuthology.orchestra.run.smithi062:> dd if=/proc/self/mounts of=/dev/stdout 2023-05-22T15:46:57.789 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16342116k,nr_inodes=4085529,mode=755,inode64 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275932k,mode=755,inode64 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev,inode64 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=13250 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275928k,nr_inodes=818982,mode=700,uid=1000,gid=1271,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275932k,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-05-22T15:46:57.789 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2023-05-22T15:46:57.790 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-05-22T15:46:57.955 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:57.947+0000 7fce8ddb2640 1 Processor -- start 2023-05-22T15:46:57.955 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:57.947+0000 7fce8ddb2640 1 -- start start 2023-05-22T15:46:57.960 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:57.951+0000 7fce8ddb2640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fce880c9b40 0x7fce8815e460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:57.960 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:57.951+0000 7fce8ddb2640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fce88070df0 0x7fce88071210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:57.960 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:57.951+0000 7fce8ddb2640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce880b0650 con 0x7fce880ca4e0 2023-05-22T15:46:57.961 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:57.951+0000 7fce8ddb2640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce880b0370 con 0x7fce880c9b40 2023-05-22T15:46:57.961 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:57.951+0000 7fce8ddb2640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce8815d980 con 0x7fce88070df0 2023-05-22T15:46:57.961 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:57.951+0000 7fce86ffd640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fce880c9b40 0x7fce8815e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:57.961 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:57.951+0000 7fce86ffd640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fce880c9b40 0x7fce8815e460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43610/0 (socket says 172.21.15.16:43610) 2023-05-22T15:46:57.961 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:57.951+0000 7fce86ffd640 1 -- 172.21.15.16:0/1932280864 learned_addr learned my addr 172.21.15.16:0/1932280864 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:46:57.961 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:57.951+0000 7fce87fff640 1 --2- 172.21.15.16:0/1932280864 >> v2:172.21.15.62:3300/0 conn(0x7fce88070df0 0x7fce88071210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:57.961 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:57.951+0000 7fce86ffd640 1 -- 172.21.15.16:0/1932280864 >> v2:172.21.15.62:3300/0 conn(0x7fce88070df0 msgr2=0x7fce88071210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:46:57.962 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:57.951+0000 7fce86ffd640 1 --2- 172.21.15.16:0/1932280864 >> v2:172.21.15.62:3300/0 conn(0x7fce88070df0 0x7fce88071210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:57.962 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:57.951+0000 7fce86ffd640 1 -- 172.21.15.16:0/1932280864 >> v1:172.21.15.62:6789/0 conn(0x7fce880ca4e0 legacy=0x7fce880afb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:57.962 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:57.951+0000 7fce86ffd640 1 -- 172.21.15.16:0/1932280864 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce8815dc60 con 0x7fce880c9b40 2023-05-22T15:46:57.962 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:57.951+0000 7fce86ffd640 1 --2- 172.21.15.16:0/1932280864 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fce880c9b40 0x7fce8815e460 secure :-1 s=READY pgs=1998 cs=0 l=1 rev1=1 crypto rx=0x7fce880cad10 tx=0x7fce7801c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=231d0e9604dda0ad server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:57.962 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:57.951+0000 7fce867fc640 1 -- 172.21.15.16:0/1932280864 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce780a7070 con 0x7fce880c9b40 2023-05-22T15:46:57.962 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:57.951+0000 7fce867fc640 1 -- 172.21.15.16:0/1932280864 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce780ab430 con 0x7fce880c9b40 2023-05-22T15:46:57.962 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:57.951+0000 7fce867fc640 1 -- 172.21.15.16:0/1932280864 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce780ab7c0 con 0x7fce880c9b40 2023-05-22T15:46:57.962 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:57.955+0000 7fce8ddb2640 1 -- 172.21.15.16:0/1932280864 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fce880c9b40 msgr2=0x7fce8815e460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:57.963 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:57.955+0000 7fce8ddb2640 1 --2- 172.21.15.16:0/1932280864 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fce880c9b40 0x7fce8815e460 secure :-1 s=READY pgs=1998 cs=0 l=1 rev1=1 crypto rx=0x7fce880cad10 tx=0x7fce7801c4d0 comp rx=0 tx=0).stop 2023-05-22T15:46:57.963 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:57.955+0000 7fce8ddb2640 1 -- 172.21.15.16:0/1932280864 shutdown_connections 2023-05-22T15:46:57.963 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:57.955+0000 7fce8ddb2640 1 --2- 172.21.15.16:0/1932280864 >> v2:172.21.15.62:3300/0 conn(0x7fce88070df0 0x7fce88071210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:57.963 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:57.955+0000 7fce8ddb2640 1 --2- 172.21.15.16:0/1932280864 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fce880c9b40 0x7fce8815e460 unknown :-1 s=CLOSED pgs=1998 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:57.963 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:57.955+0000 7fce8ddb2640 1 -- 172.21.15.16:0/1932280864 >> 172.21.15.16:0/1932280864 conn(0x7fce88069720 msgr2=0x7fce88069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:46:57.963 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:57.955+0000 7fce8ddb2640 1 -- 172.21.15.16:0/1932280864 shutdown_connections 2023-05-22T15:46:57.963 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:57.955+0000 7fce8ddb2640 1 -- 172.21.15.16:0/1932280864 wait complete. 2023-05-22T15:46:57.965 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:57.955+0000 7fce8ddb2640 1 Processor -- start 2023-05-22T15:46:57.967 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:57.955+0000 7fce8ddb2640 1 -- start start 2023-05-22T15:46:57.967 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:57.955+0000 7fce8ddb2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fce88070df0 0x7fce8814ba00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:57.968 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:57.955+0000 7fce8ddb2640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fce880c9b40 0x7fce88159500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:57.968 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:57.955+0000 7fce8ddb2640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fce880ca4e0 0x7fce881570f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:57.968 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:57.955+0000 7fce8ddb2640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce8814b570 con 0x7fce88070df0 2023-05-22T15:46:57.968 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:57.955+0000 7fce8ddb2640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce8814ac80 con 0x7fce880ca4e0 2023-05-22T15:46:57.968 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:57.955+0000 7fce8ddb2640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce88157be0 con 0x7fce880c9b40 2023-05-22T15:46:57.968 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:57.955+0000 7fce877fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fce88070df0 0x7fce8814ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:57.968 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:57.955+0000 7fce877fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fce88070df0 0x7fce8814ba00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43614/0 (socket says 172.21.15.16:43614) 2023-05-22T15:46:57.969 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:57.955+0000 7fce877fe640 1 -- 172.21.15.16:0/3178141255 learned_addr learned my addr 172.21.15.16:0/3178141255 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:46:57.969 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:57.955+0000 7fce86ffd640 1 --2- 172.21.15.16:0/3178141255 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fce880c9b40 0x7fce88159500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:57.969 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:57.955+0000 7fce87fff640 1 --2- 172.21.15.16:0/3178141255 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fce880ca4e0 0x7fce881570f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:57.969 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:57.955+0000 7fce877fe640 1 -- 172.21.15.16:0/3178141255 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fce880c9b40 msgr2=0x7fce88159500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:57.969 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:57.955+0000 7fce877fe640 1 --2- 172.21.15.16:0/3178141255 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fce880c9b40 0x7fce88159500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:57.969 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:57.955+0000 7fce877fe640 1 -- 172.21.15.16:0/3178141255 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fce880ca4e0 msgr2=0x7fce881570f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:57.969 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:57.955+0000 7fce877fe640 1 --2- 172.21.15.16:0/3178141255 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fce880ca4e0 0x7fce881570f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:57.970 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:57.955+0000 7fce877fe640 1 -- 172.21.15.16:0/3178141255 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce780094d0 con 0x7fce88070df0 2023-05-22T15:46:57.970 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:57.955+0000 7fce877fe640 1 --2- 172.21.15.16:0/3178141255 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fce88070df0 0x7fce8814ba00 secure :-1 s=READY pgs=1999 cs=0 l=1 rev1=1 crypto rx=0x7fce7c00c370 tx=0x7fce7c00c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:57.970 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:57.955+0000 7fce6ffff640 1 -- 172.21.15.16:0/3178141255 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce7c0052d0 con 0x7fce88070df0 2023-05-22T15:46:57.970 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:57.955+0000 7fce6ffff640 1 -- 172.21.15.16:0/3178141255 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce7c005430 con 0x7fce88070df0 2023-05-22T15:46:57.970 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:57.955+0000 7fce8ddb2640 1 -- 172.21.15.16:0/3178141255 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce880c8e80 con 0x7fce88070df0 2023-05-22T15:46:57.970 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:57.955+0000 7fce6ffff640 1 -- 172.21.15.16:0/3178141255 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce7c007260 con 0x7fce88070df0 2023-05-22T15:46:57.970 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:57.955+0000 7fce8ddb2640 1 -- 172.21.15.16:0/3178141255 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce880ca8b0 con 0x7fce88070df0 2023-05-22T15:46:57.970 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:57.959+0000 7fce8ddb2640 1 -- 172.21.15.16:0/3178141255 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce880c9010 con 0x7fce88070df0 2023-05-22T15:46:57.971 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:57.959+0000 7fce6ffff640 1 -- 172.21.15.16:0/3178141255 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 67) v1 ==== 84332+0+0 (secure 0 0 0) 0x7fce7c0073c0 con 0x7fce88070df0 2023-05-22T15:46:57.971 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:57.959+0000 7fce6ffff640 1 --2- 172.21.15.16:0/3178141255 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fce740654a0 0x7fce74067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:57.971 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:57.959+0000 7fce6ffff640 1 -- 172.21.15.16:0/3178141255 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(152..152 src has 1..152) v4 ==== 6289+0+0 (secure 0 0 0) 0x7fce7c081db0 con 0x7fce88070df0 2023-05-22T15:46:57.971 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:57.959+0000 7fce86ffd640 1 --2- 172.21.15.16:0/3178141255 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fce740654a0 0x7fce74067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:57.971 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:57.959+0000 7fce86ffd640 1 --2- 172.21.15.16:0/3178141255 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fce740654a0 0x7fce74067950 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fce780af000 tx=0x7fce780a7000 comp rx=0 tx=0).ready entity=mgr.9396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:57.972 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:57.963+0000 7fce6ffff640 1 -- 172.21.15.16:0/3178141255 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fce7c049620 con 0x7fce88070df0 2023-05-22T15:46:58.153 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:58.147+0000 7fce8ddb2640 1 -- 172.21.15.16:0/3178141255 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fce8804dbe0 con 0x7fce88070df0 2023-05-22T15:46:58.154 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:58.147+0000 7fce6ffff640 1 -- 172.21.15.16:0/3178141255 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 60 v60) v1 ==== 94+0+3529 (secure 0 0 0) 0x7fce7c04f8c0 con 0x7fce88070df0 2023-05-22T15:46:58.155 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:46:58.155 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":60,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7753,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6834/3192074681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":3192074681},{"type":"v1","addr":"172.21.15.62:6834","nonce":3192074681}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":53},{"gid":7756,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/1344243469","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6833","nonce":1344243469},{"type":"v1","addr":"172.21.15.62:6835","nonce":1344243469}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":53},{"gid":9069,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/3636102434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":3636102434},{"type":"v1","addr":"172.21.15.16:6835","nonce":3636102434}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":53},{"gid":9570,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/1345726873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":1345726873},{"type":"v1","addr":"172.21.15.16:6837","nonce":1345726873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":60}],"filesystems":[]} 2023-05-22T15:46:58.155 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 60 2023-05-22T15:46:58.157 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:58.151+0000 7fce8ddb2640 1 -- 172.21.15.16:0/3178141255 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fce740654a0 msgr2=0x7fce74067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:58.157 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:58.151+0000 7fce8ddb2640 1 --2- 172.21.15.16:0/3178141255 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fce740654a0 0x7fce74067950 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fce780af000 tx=0x7fce780a7000 comp rx=0 tx=0).stop 2023-05-22T15:46:58.157 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:58.151+0000 7fce8ddb2640 1 -- 172.21.15.16:0/3178141255 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fce88070df0 msgr2=0x7fce8814ba00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:58.157 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:58.151+0000 7fce8ddb2640 1 --2- 172.21.15.16:0/3178141255 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fce88070df0 0x7fce8814ba00 secure :-1 s=READY pgs=1999 cs=0 l=1 rev1=1 crypto rx=0x7fce7c00c370 tx=0x7fce7c00c830 comp rx=0 tx=0).stop 2023-05-22T15:46:58.157 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:58.151+0000 7fce8ddb2640 1 -- 172.21.15.16:0/3178141255 shutdown_connections 2023-05-22T15:46:58.158 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:58.151+0000 7fce8ddb2640 1 --2- 172.21.15.16:0/3178141255 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fce880ca4e0 0x7fce881570f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:58.158 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:58.151+0000 7fce8ddb2640 1 --2- 172.21.15.16:0/3178141255 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fce740654a0 0x7fce74067950 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:58.158 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:58.151+0000 7fce8ddb2640 1 --2- 172.21.15.16:0/3178141255 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fce880c9b40 0x7fce88159500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:58.158 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:58.151+0000 7fce8ddb2640 1 --2- 172.21.15.16:0/3178141255 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fce88070df0 0x7fce8814ba00 unknown :-1 s=CLOSED pgs=1999 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:58.158 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:58.151+0000 7fce8ddb2640 1 -- 172.21.15.16:0/3178141255 >> 172.21.15.16:0/3178141255 conn(0x7fce88069720 msgr2=0x7fce880b29c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:46:58.158 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:58.151+0000 7fce8ddb2640 1 -- 172.21.15.16:0/3178141255 shutdown_connections 2023-05-22T15:46:58.158 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:58.151+0000 7fce8ddb2640 1 -- 172.21.15.16:0/3178141255 wait complete. 2023-05-22T15:46:58.175 INFO:tasks.ceph.mds.a:Restarting daemon 2023-05-22T15:46:58.175 INFO:tasks.ceph.mds.a:Stopping old one... 2023-05-22T15:46:58.175 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-05-22T15:46:58.176 INFO:teuthology.orchestra.run:waiting for 300 2023-05-22T15:46:58.176 INFO:tasks.ceph.mds.c:Restarting daemon 2023-05-22T15:46:58.176 INFO:tasks.ceph.mds.c:Stopping old one... 2023-05-22T15:46:58.176 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-05-22T15:46:58.177 INFO:teuthology.orchestra.run:waiting for 300 2023-05-22T15:46:58.177 INFO:tasks.ceph.mds.b:Restarting daemon 2023-05-22T15:46:58.177 INFO:tasks.ceph.mds.b:Stopping old one... 2023-05-22T15:46:58.177 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-05-22T15:46:58.178 INFO:teuthology.orchestra.run:waiting for 300 2023-05-22T15:46:58.178 INFO:tasks.ceph.mds.d:Restarting daemon 2023-05-22T15:46:58.178 INFO:tasks.ceph.mds.d:Stopping old one... 2023-05-22T15:46:58.178 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-05-22T15:46:58.178 INFO:teuthology.orchestra.run:waiting for 300 2023-05-22T15:46:58.185 INFO:tasks.ceph.mds.a:Stopped 2023-05-22T15:46:58.185 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-05-22T15:46:58.187 INFO:tasks.ceph.mds.c:Stopped 2023-05-22T15:46:58.187 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-05-22T15:46:58.191 INFO:tasks.ceph.mds.d:Stopped 2023-05-22T15:46:58.192 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-05-22T15:46:58.196 INFO:tasks.ceph.mds.b:Stopped 2023-05-22T15:46:58.197 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-05-22T15:46:58.231 INFO:tasks.ceph.mds.a:Started 2023-05-22T15:46:58.232 DEBUG:teuthology.parallel:result is None 2023-05-22T15:46:58.233 INFO:tasks.ceph.mds.c:Started 2023-05-22T15:46:58.233 DEBUG:teuthology.parallel:result is None 2023-05-22T15:46:58.241 INFO:tasks.ceph.mds.d:Started 2023-05-22T15:46:58.242 DEBUG:teuthology.parallel:result is None 2023-05-22T15:46:58.243 INFO:tasks.ceph.mds.b:Started 2023-05-22T15:46:58.243 DEBUG:teuthology.parallel:result is None 2023-05-22T15:46:58.244 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2023-05-22T15:46:58.296 INFO:tasks.ceph.mds.a.smithi016.stdout:starting mds.a at 2023-05-22T15:46:58.303 INFO:tasks.ceph.mds.c.smithi016.stdout:starting mds.c at 2023-05-22T15:46:58.315 INFO:tasks.ceph.mds.b.smithi062.stdout:starting mds.b at 2023-05-22T15:46:58.317 INFO:tasks.ceph.mds.d.smithi062.stdout:starting mds.d at 2023-05-22T15:46:58.365 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:58.359+0000 7f04edefd640 1 Processor -- start 2023-05-22T15:46:58.365 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:58.359+0000 7f04edefd640 1 -- start start 2023-05-22T15:46:58.367 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:58.359+0000 7f04edefd640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f04e815c270 0x7f04e815c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:58.367 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:58.359+0000 7f04edefd640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04e8069ae0 con 0x7f04e8071640 2023-05-22T15:46:58.367 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:58.359+0000 7f04edefd640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04e8069dc0 con 0x7f04e80ce480 2023-05-22T15:46:58.368 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:58.359+0000 7f04edefd640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04e815f990 con 0x7f04e815c270 2023-05-22T15:46:58.368 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:58.359+0000 7f04e6ffd640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f04e8071640 0x7f04e8160090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:55610/0 (socket says 172.21.15.16:55610) 2023-05-22T15:46:58.368 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:58.359+0000 7f04e6ffd640 1 -- 172.21.15.16:0/3340057681 learned_addr learned my addr 172.21.15.16:0/3340057681 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:46:58.368 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:58.359+0000 7f04e7fff640 1 --2- 172.21.15.16:0/3340057681 >> v2:172.21.15.62:3300/0 conn(0x7f04e815c270 0x7f04e815c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:58.368 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:58.359+0000 7f04e67fc640 1 -- 172.21.15.16:0/3340057681 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 954613223 0 0) 0x7f04e8069ae0 con 0x7f04e8071640 2023-05-22T15:46:58.368 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:58.359+0000 7f04e67fc640 1 -- 172.21.15.16:0/3340057681 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04d0003530 con 0x7f04e8071640 2023-05-22T15:46:58.368 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:58.359+0000 7f04e67fc640 1 -- 172.21.15.16:0/3340057681 <== mon.1 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2898897204 0 0) 0x7f04e8069dc0 con 0x7f04e80ce480 2023-05-22T15:46:58.369 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:58.359+0000 7f04e67fc640 1 -- 172.21.15.16:0/3340057681 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04e8069ae0 con 0x7f04e80ce480 2023-05-22T15:46:58.369 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:58.359+0000 7f04e67fc640 1 -- 172.21.15.16:0/3340057681 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3952609896 0 0) 0x7f04d0003530 con 0x7f04e8071640 2023-05-22T15:46:58.369 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:58.359+0000 7f04e67fc640 1 -- 172.21.15.16:0/3340057681 >> v2:172.21.15.62:3300/0 conn(0x7f04e815c270 msgr2=0x7f04e815c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:46:58.369 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:58.359+0000 7f04e67fc640 1 --2- 172.21.15.16:0/3340057681 >> v2:172.21.15.62:3300/0 conn(0x7f04e815c270 0x7f04e815c6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:58.369 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:58.359+0000 7f04e67fc640 1 -- 172.21.15.16:0/3340057681 >> v1:172.21.15.62:6789/0 conn(0x7f04e80ce480 legacy=0x7f04e80ce850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:58.369 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:58.359+0000 7f04e67fc640 1 -- 172.21.15.16:0/3340057681 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04e806a0a0 con 0x7f04e8071640 2023-05-22T15:46:58.369 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:58.359+0000 7f04e67fc640 1 -- 172.21.15.16:0/3340057681 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7f04d8001680 con 0x7f04e8071640 2023-05-22T15:46:58.369 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:58.359+0000 7f04e67fc640 1 -- 172.21.15.16:0/3340057681 >> v1:172.21.15.16:6789/0 conn(0x7f04e8071640 legacy=0x7f04e8160090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:58.370 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:58.359+0000 7f04e67fc640 1 --2- 172.21.15.16:0/3340057681 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f04d0003b60 0x7f04d0003f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:58.370 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:58.359+0000 7f04e67fc640 1 --2- 172.21.15.16:0/3340057681 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f04d00047d0 0x7f04d0004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:58.370 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:58.359+0000 7f04e67fc640 1 --2- 172.21.15.16:0/3340057681 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f04d0005180 0x7f04d000b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:58.370 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:58.359+0000 7f04e67fc640 1 -- 172.21.15.16:0/3340057681 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04e815f990 con 0x7f04d0003b60 2023-05-22T15:46:58.370 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:58.359+0000 7f04e67fc640 1 -- 172.21.15.16:0/3340057681 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04d000c060 con 0x7f04d00047d0 2023-05-22T15:46:58.370 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:58.359+0000 7f04e67fc640 1 -- 172.21.15.16:0/3340057681 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04d000c1f0 con 0x7f04d0005180 2023-05-22T15:46:58.371 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:58.359+0000 7f04e77fe640 1 --2- 172.21.15.16:0/3340057681 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f04d0003b60 0x7f04d0003f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:58.371 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:58.359+0000 7f04e6ffd640 1 --2- 172.21.15.16:0/3340057681 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f04d00047d0 0x7f04d0004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:58.371 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:58.359+0000 7f04e7fff640 1 --2- 172.21.15.16:0/3340057681 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f04d0005180 0x7f04d000b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:58.372 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:58.359+0000 7f04e77fe640 1 -- 172.21.15.16:0/3340057681 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f04d0005180 msgr2=0x7f04d000b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:46:58.372 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:58.359+0000 7f04e77fe640 1 --2- 172.21.15.16:0/3340057681 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f04d0005180 0x7f04d000b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:58.372 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:58.359+0000 7f04e77fe640 1 -- 172.21.15.16:0/3340057681 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f04d00047d0 msgr2=0x7f04d0004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:46:58.372 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:58.359+0000 7f04e77fe640 1 --2- 172.21.15.16:0/3340057681 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f04d00047d0 0x7f04d0004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:58.372 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:58.359+0000 7f04e77fe640 1 -- 172.21.15.16:0/3340057681 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04d0003530 con 0x7f04d0003b60 2023-05-22T15:46:58.372 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:58.359+0000 7f04e7fff640 1 -- 172.21.15.16:0/3340057681 reap_dead start 2023-05-22T15:46:58.375 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:58.363+0000 7f04e77fe640 1 --2- 172.21.15.16:0/3340057681 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f04d0003b60 0x7f04d0003f50 secure :-1 s=READY pgs=2008 cs=0 l=1 rev1=1 crypto rx=0x7f04dc007fd0 tx=0x7f04dc01c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ddd02f33a66dbc4 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:58.375 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:58.363+0000 7f04e67fc640 1 -- 172.21.15.16:0/3340057681 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04dc0052d0 con 0x7f04d0003b60 2023-05-22T15:46:58.375 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:58.363+0000 7f04e67fc640 1 -- 172.21.15.16:0/3340057681 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04dc0a7050 con 0x7f04d0003b60 2023-05-22T15:46:58.375 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:58.363+0000 7f04e67fc640 1 -- 172.21.15.16:0/3340057681 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04dc003600 con 0x7f04d0003b60 2023-05-22T15:46:58.375 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:58.363+0000 7f04edefd640 1 -- 172.21.15.16:0/3340057681 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f04d0003b60 msgr2=0x7f04d0003f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:58.375 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:58.363+0000 7f04edefd640 1 --2- 172.21.15.16:0/3340057681 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f04d0003b60 0x7f04d0003f50 secure :-1 s=READY pgs=2008 cs=0 l=1 rev1=1 crypto rx=0x7f04dc007fd0 tx=0x7f04dc01c490 comp rx=0 tx=0).stop 2023-05-22T15:46:58.376 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:58.363+0000 7f04edefd640 1 -- 172.21.15.16:0/3340057681 shutdown_connections 2023-05-22T15:46:58.376 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:58.363+0000 7f04edefd640 1 --2- 172.21.15.16:0/3340057681 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f04d0003b60 0x7f04d0003f50 unknown :-1 s=CLOSED pgs=2008 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:58.376 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:58.363+0000 7f04edefd640 1 -- 172.21.15.16:0/3340057681 >> 172.21.15.16:0/3340057681 conn(0x7f04e80cc870 msgr2=0x7f04e80cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:46:58.376 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:58.363+0000 7f04edefd640 1 -- 172.21.15.16:0/3340057681 shutdown_connections 2023-05-22T15:46:58.376 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:58.363+0000 7f04edefd640 1 -- 172.21.15.16:0/3340057681 wait complete. 2023-05-22T15:46:58.376 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:58.363+0000 7f04edefd640 1 Processor -- start 2023-05-22T15:46:58.376 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:58.363+0000 7f04edefd640 1 -- start start 2023-05-22T15:46:58.376 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:58.367+0000 7f04edefd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f04d0003b60 0x7f04e80aebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:58.377 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:58.367+0000 7f04edefd640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f04e8150240 0x7f04e80c36a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:58.377 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:58.367+0000 7f04edefd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f04e80c3be0 0x7f04e80ac820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:58.377 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:58.367+0000 7f04edefd640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04e8071ac0 con 0x7f04e80c3be0 2023-05-22T15:46:58.377 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:58.367+0000 7f04edefd640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04e80adeb0 con 0x7f04d0003b60 2023-05-22T15:46:58.377 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:58.367+0000 7f04edefd640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04e80ad8d0 con 0x7f04e8150240 2023-05-22T15:46:58.377 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:58.367+0000 7f04e7fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f04e80c3be0 0x7f04e80ac820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:58.377 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:58.367+0000 7f04e7fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f04e80c3be0 0x7f04e80ac820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43684/0 (socket says 172.21.15.16:43684) 2023-05-22T15:46:58.378 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:58.367+0000 7f04e7fff640 1 -- 172.21.15.16:0/385643455 learned_addr learned my addr 172.21.15.16:0/385643455 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:46:58.378 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:58.367+0000 7f04e6ffd640 1 --2- 172.21.15.16:0/385643455 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f04e8150240 0x7f04e80c36a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:58.378 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:58.367+0000 7f04e77fe640 1 --2- 172.21.15.16:0/385643455 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f04d0003b60 0x7f04e80aebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:58.378 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:58.367+0000 7f04e7fff640 1 -- 172.21.15.16:0/385643455 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f04e8150240 msgr2=0x7f04e80c36a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:58.378 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:58.367+0000 7f04e7fff640 1 --2- 172.21.15.16:0/385643455 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f04e8150240 0x7f04e80c36a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:58.378 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:58.367+0000 7f04e7fff640 1 -- 172.21.15.16:0/385643455 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f04d0003b60 msgr2=0x7f04e80aebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:58.378 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:58.367+0000 7f04e7fff640 1 --2- 172.21.15.16:0/385643455 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f04d0003b60 0x7f04e80aebc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:58.378 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:58.367+0000 7f04e7fff640 1 -- 172.21.15.16:0/385643455 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04e80c6000 con 0x7f04e80c3be0 2023-05-22T15:46:58.379 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:58.367+0000 7f04e7fff640 1 --2- 172.21.15.16:0/385643455 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f04e80c3be0 0x7f04e80ac820 secure :-1 s=READY pgs=2009 cs=0 l=1 rev1=1 crypto rx=0x7f04e80cd850 tx=0x7f04e0011cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:58.379 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:58.367+0000 7f04cffff640 1 -- 172.21.15.16:0/385643455 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04e000e040 con 0x7f04e80c3be0 2023-05-22T15:46:58.379 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:58.367+0000 7f04cffff640 1 -- 172.21.15.16:0/385643455 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04e000f040 con 0x7f04e80c3be0 2023-05-22T15:46:58.379 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:58.367+0000 7f04edefd640 1 -- 172.21.15.16:0/385643455 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04e80c5e70 con 0x7f04e80c3be0 2023-05-22T15:46:58.379 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:58.367+0000 7f04cffff640 1 -- 172.21.15.16:0/385643455 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04e0015a80 con 0x7f04e80c3be0 2023-05-22T15:46:58.379 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:58.367+0000 7f04edefd640 1 -- 172.21.15.16:0/385643455 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04e80cec60 con 0x7f04e80c3be0 2023-05-22T15:46:58.379 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:58.367+0000 7f04edefd640 1 -- 172.21.15.16:0/385643455 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04e804dbe0 con 0x7f04e80c3be0 2023-05-22T15:46:58.379 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:58.367+0000 7f04cffff640 1 -- 172.21.15.16:0/385643455 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 67) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f04e0015be0 con 0x7f04e80c3be0 2023-05-22T15:46:58.380 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:58.367+0000 7f04cffff640 1 --2- 172.21.15.16:0/385643455 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f04d40654d0 0x7f04d4067980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:58.380 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:58.367+0000 7f04cffff640 1 -- 172.21.15.16:0/385643455 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(156..156 src has 1..156) v4 ==== 6289+0+0 (secure 0 0 0) 0x7f04e0082c40 con 0x7f04e80c3be0 2023-05-22T15:46:58.380 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:58.367+0000 7f04e77fe640 1 --2- 172.21.15.16:0/385643455 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f04d40654d0 0x7f04d4067980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:58.380 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:58.371+0000 7f04e77fe640 1 --2- 172.21.15.16:0/385643455 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f04d40654d0 0x7f04d4067980 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f04dc0ac000 tx=0x7f04dc0a6000 comp rx=0 tx=0).ready entity=mgr.9396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:58.381 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:58.371+0000 7f04cffff640 1 -- 172.21.15.16:0/385643455 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f04e00505b0 con 0x7f04e80c3be0 2023-05-22T15:46:58.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:58.539+0000 7f04edefd640 1 -- 172.21.15.16:0/385643455 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f04e80adff0 con 0x7f04e80c3be0 2023-05-22T15:46:59.367 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:59.363+0000 7f04cffff640 1 -- 172.21.15.16:0/385643455 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v157) v1 ==== 97+0+0 (secure 0 0 0) 0x7f04e0053bd0 con 0x7f04e80c3be0 2023-05-22T15:46:59.368 INFO:teuthology.orchestra.run.smithi016.stderr: removed all blocklist entries 2023-05-22T15:46:59.371 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:59.363+0000 7f04edefd640 1 -- 172.21.15.16:0/385643455 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f04d40654d0 msgr2=0x7f04d4067980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:59.372 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:59.363+0000 7f04edefd640 1 --2- 172.21.15.16:0/385643455 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f04d40654d0 0x7f04d4067980 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f04dc0ac000 tx=0x7f04dc0a6000 comp rx=0 tx=0).stop 2023-05-22T15:46:59.372 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:59.363+0000 7f04edefd640 1 -- 172.21.15.16:0/385643455 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f04e80c3be0 msgr2=0x7f04e80ac820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:59.372 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:59.363+0000 7f04edefd640 1 --2- 172.21.15.16:0/385643455 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f04e80c3be0 0x7f04e80ac820 secure :-1 s=READY pgs=2009 cs=0 l=1 rev1=1 crypto rx=0x7f04e80cd850 tx=0x7f04e0011cb0 comp rx=0 tx=0).stop 2023-05-22T15:46:59.372 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:59.363+0000 7f04edefd640 1 -- 172.21.15.16:0/385643455 shutdown_connections 2023-05-22T15:46:59.372 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:59.363+0000 7f04edefd640 1 --2- 172.21.15.16:0/385643455 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f04e80c3be0 0x7f04e80ac820 unknown :-1 s=CLOSED pgs=2009 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:59.372 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:59.363+0000 7f04edefd640 1 --2- 172.21.15.16:0/385643455 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f04d40654d0 0x7f04d4067980 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:59.372 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:59.363+0000 7f04edefd640 1 --2- 172.21.15.16:0/385643455 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f04e8150240 0x7f04e80c36a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:59.373 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:59.363+0000 7f04edefd640 1 --2- 172.21.15.16:0/385643455 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f04d0003b60 0x7f04e80aebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:59.373 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:59.363+0000 7f04edefd640 1 -- 172.21.15.16:0/385643455 >> 172.21.15.16:0/385643455 conn(0x7f04e80cc870 msgr2=0x7f04e815deb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:46:59.373 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:59.363+0000 7f04edefd640 1 -- 172.21.15.16:0/385643455 shutdown_connections 2023-05-22T15:46:59.373 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:59.363+0000 7f04edefd640 1 -- 172.21.15.16:0/385643455 wait complete. 2023-05-22T15:46:59.389 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2023-05-22T15:46:59.519 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:59.511+0000 7f409bd13640 1 Processor -- start 2023-05-22T15:46:59.519 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:59.511+0000 7f409bd13640 1 -- start start 2023-05-22T15:46:59.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:59.515+0000 7f409bd13640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f40940ca4e0 0x7f409415e460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:59.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:59.515+0000 7f409bd13640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40940b0650 con 0x7f40940c9b40 2023-05-22T15:46:59.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:59.515+0000 7f409bd13640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40940b0370 con 0x7f4094070df0 2023-05-22T15:46:59.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:59.515+0000 7f409bd13640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f409415d980 con 0x7f40940ca4e0 2023-05-22T15:46:59.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:59.515+0000 7f40992ae640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f40940c9b40 0x7f40940afb40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:44824/0 (socket says 172.21.15.16:44824) 2023-05-22T15:46:59.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:59.515+0000 7f40992ae640 1 -- 172.21.15.16:0/3807319500 learned_addr learned my addr 172.21.15.16:0/3807319500 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:46:59.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:59.515+0000 7f4099aaf640 1 --2- 172.21.15.16:0/3807319500 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f40940ca4e0 0x7f409415e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:59.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:59.515+0000 7f4098aad640 1 -- 172.21.15.16:0/3807319500 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1115246788 0 0) 0x7f40940b0650 con 0x7f40940c9b40 2023-05-22T15:46:59.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:59.515+0000 7f4098aad640 1 -- 172.21.15.16:0/3807319500 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f407c003530 con 0x7f40940c9b40 2023-05-22T15:46:59.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:59.515+0000 7f4098aad640 1 -- 172.21.15.16:0/3807319500 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 601400269 0 0) 0x7f407c003530 con 0x7f40940c9b40 2023-05-22T15:46:59.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:59.515+0000 7f4098aad640 1 -- 172.21.15.16:0/3807319500 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f40940ca4e0 msgr2=0x7f409415e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:46:59.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:59.515+0000 7f4098aad640 1 --2- 172.21.15.16:0/3807319500 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f40940ca4e0 0x7f409415e460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:59.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:59.515+0000 7f4098aad640 1 -- 172.21.15.16:0/3807319500 >> v1:172.21.15.62:6789/0 conn(0x7f4094070df0 legacy=0x7f4094071210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:59.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:59.515+0000 7f4098aad640 1 -- 172.21.15.16:0/3807319500 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f409415dc60 con 0x7f40940c9b40 2023-05-22T15:46:59.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:59.515+0000 7f4098aad640 1 -- 172.21.15.16:0/3807319500 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7f4088001a00 con 0x7f40940c9b40 2023-05-22T15:46:59.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:59.515+0000 7f4098aad640 1 -- 172.21.15.16:0/3807319500 >> v1:172.21.15.16:6789/0 conn(0x7f40940c9b40 legacy=0x7f40940afb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:59.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:59.515+0000 7f4098aad640 1 --2- 172.21.15.16:0/3807319500 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f407c0039b0 0x7f407c003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:59.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:59.515+0000 7f4098aad640 1 --2- 172.21.15.16:0/3807319500 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f407c0047d0 0x7f407c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:59.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:59.515+0000 7f4098aad640 1 --2- 172.21.15.16:0/3807319500 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f40940ca4e0 0x7f407c0051a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:59.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:59.515+0000 7f4098aad640 1 -- 172.21.15.16:0/3807319500 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f409415d980 con 0x7f407c0039b0 2023-05-22T15:46:59.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:59.515+0000 7f4098aad640 1 -- 172.21.15.16:0/3807319500 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f407c00ba90 con 0x7f407c0047d0 2023-05-22T15:46:59.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:59.515+0000 7f4098aad640 1 -- 172.21.15.16:0/3807319500 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f407c00bc50 con 0x7f40940ca4e0 2023-05-22T15:46:59.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:59.515+0000 7f4099aaf640 1 --2- 172.21.15.16:0/3807319500 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f407c0039b0 0x7f407c003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:59.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:59.515+0000 7f40992ae640 1 --2- 172.21.15.16:0/3807319500 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f407c0047d0 0x7f407c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:59.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:59.515+0000 7f4099aaf640 1 -- 172.21.15.16:0/3807319500 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f40940ca4e0 msgr2=0x7f407c0051a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:46:59.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:59.515+0000 7f4099aaf640 1 --2- 172.21.15.16:0/3807319500 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f40940ca4e0 0x7f407c0051a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:59.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:59.515+0000 7f4099aaf640 1 -- 172.21.15.16:0/3807319500 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f407c0047d0 msgr2=0x7f407c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:46:59.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:59.515+0000 7f4099aaf640 1 --2- 172.21.15.16:0/3807319500 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f407c0047d0 0x7f407c004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:59.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:59.515+0000 7f4099aaf640 1 -- 172.21.15.16:0/3807319500 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f407c003530 con 0x7f407c0039b0 2023-05-22T15:46:59.527 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:59.519+0000 7f4099aaf640 1 --2- 172.21.15.16:0/3807319500 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f407c0039b0 0x7f407c003dc0 secure :-1 s=READY pgs=2011 cs=0 l=1 rev1=1 crypto rx=0x7f4084009ae0 tx=0x7f408401ca60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e78edeb70b47c58e server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:59.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:59.519+0000 7f4098aad640 1 -- 172.21.15.16:0/3807319500 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40840a5540 con 0x7f407c0039b0 2023-05-22T15:46:59.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:59.519+0000 7f4098aad640 1 -- 172.21.15.16:0/3807319500 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40840a56a0 con 0x7f407c0039b0 2023-05-22T15:46:59.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:59.519+0000 7f4098aad640 1 -- 172.21.15.16:0/3807319500 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40840a5a50 con 0x7f407c0039b0 2023-05-22T15:46:59.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:59.519+0000 7f409bd13640 1 -- 172.21.15.16:0/3807319500 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f407c0039b0 msgr2=0x7f407c003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:59.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:59.519+0000 7f409bd13640 1 --2- 172.21.15.16:0/3807319500 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f407c0039b0 0x7f407c003dc0 secure :-1 s=READY pgs=2011 cs=0 l=1 rev1=1 crypto rx=0x7f4084009ae0 tx=0x7f408401ca60 comp rx=0 tx=0).stop 2023-05-22T15:46:59.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:59.519+0000 7f409a2b0640 1 -- 172.21.15.16:0/3807319500 reap_dead start 2023-05-22T15:46:59.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:59.519+0000 7f409bd13640 1 -- 172.21.15.16:0/3807319500 shutdown_connections 2023-05-22T15:46:59.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:59.519+0000 7f409bd13640 1 -- 172.21.15.16:0/3807319500 >> 172.21.15.16:0/3807319500 conn(0x7f4094069720 msgr2=0x7f4094069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:46:59.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:59.519+0000 7f409bd13640 1 -- 172.21.15.16:0/3807319500 shutdown_connections 2023-05-22T15:46:59.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:59.519+0000 7f409bd13640 1 -- 172.21.15.16:0/3807319500 wait complete. 2023-05-22T15:46:59.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:59.519+0000 7f409bd13640 1 Processor -- start 2023-05-22T15:46:59.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:59.519+0000 7f409bd13640 1 -- start start 2023-05-22T15:46:59.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:59.519+0000 7f409bd13640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f409414f390 0x7f40941f3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:59.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:59.519+0000 7f409bd13640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f409414f760 0x7f4094151ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:59.530 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:59.519+0000 7f409bd13640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f409415cc80 0x7f4094152010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:59.530 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:59.519+0000 7f409bd13640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f409414fd80 con 0x7f409414f760 2023-05-22T15:46:59.530 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:59.519+0000 7f409bd13640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f409414ffd0 con 0x7f409414f390 2023-05-22T15:46:59.530 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:59.519+0000 7f409bd13640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40941502b0 con 0x7f409415cc80 2023-05-22T15:46:59.530 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:59.519+0000 7f40992ae640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f409414f760 0x7f4094151ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:59.530 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:59.519+0000 7f40992ae640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f409414f760 0x7f4094151ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46652/0 (socket says 172.21.15.16:46652) 2023-05-22T15:46:59.530 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:59.519+0000 7f40992ae640 1 -- 172.21.15.16:0/2068672889 learned_addr learned my addr 172.21.15.16:0/2068672889 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:46:59.531 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:59.519+0000 7f409a2b0640 1 --2- 172.21.15.16:0/2068672889 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f409415cc80 0x7f4094152010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:59.531 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:59.519+0000 7f4099aaf640 1 --2- 172.21.15.16:0/2068672889 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f409414f390 0x7f40941f3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:59.531 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:59.519+0000 7f40992ae640 1 -- 172.21.15.16:0/2068672889 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f409415cc80 msgr2=0x7f4094152010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:59.531 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:59.519+0000 7f40992ae640 1 --2- 172.21.15.16:0/2068672889 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f409415cc80 0x7f4094152010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:59.531 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:59.519+0000 7f40992ae640 1 -- 172.21.15.16:0/2068672889 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f409414f390 msgr2=0x7f40941f3690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:59.531 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:59.519+0000 7f40992ae640 1 --2- 172.21.15.16:0/2068672889 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f409414f390 0x7f40941f3690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:59.531 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:59.519+0000 7f40992ae640 1 -- 172.21.15.16:0/2068672889 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40940c9010 con 0x7f409414f760 2023-05-22T15:46:59.532 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:59.519+0000 7f4099aaf640 1 --2- 172.21.15.16:0/2068672889 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f409414f390 0x7f40941f3690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:46:59.532 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:59.519+0000 7f409a2b0640 1 --2- 172.21.15.16:0/2068672889 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f409415cc80 0x7f4094152010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:46:59.532 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:59.523+0000 7f40992ae640 1 --2- 172.21.15.16:0/2068672889 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f409414f760 0x7f4094151ad0 secure :-1 s=READY pgs=2012 cs=0 l=1 rev1=1 crypto rx=0x7f408800e410 tx=0x7f408800e8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:59.532 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:59.523+0000 7f40927fc640 1 -- 172.21.15.16:0/2068672889 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4088012590 con 0x7f409414f760 2023-05-22T15:46:59.532 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:59.523+0000 7f40927fc640 1 -- 172.21.15.16:0/2068672889 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4088012710 con 0x7f409414f760 2023-05-22T15:46:59.532 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:59.523+0000 7f409bd13640 1 -- 172.21.15.16:0/2068672889 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40940c8e80 con 0x7f409414f760 2023-05-22T15:46:59.532 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:59.523+0000 7f40927fc640 1 -- 172.21.15.16:0/2068672889 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4088012ac0 con 0x7f409414f760 2023-05-22T15:46:59.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:59.523+0000 7f409bd13640 1 -- 172.21.15.16:0/2068672889 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40940c91a0 con 0x7f409414f760 2023-05-22T15:46:59.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:59.523+0000 7f409bd13640 1 -- 172.21.15.16:0/2068672889 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f409404dbe0 con 0x7f409414f760 2023-05-22T15:46:59.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:59.523+0000 7f40927fc640 1 -- 172.21.15.16:0/2068672889 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 67) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f4088012c20 con 0x7f409414f760 2023-05-22T15:46:59.534 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:59.523+0000 7f40927fc640 1 --2- 172.21.15.16:0/2068672889 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f40800654d0 0x7f4080067980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:59.534 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:59.523+0000 7f40927fc640 1 -- 172.21.15.16:0/2068672889 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(157..157 src has 1..157) v4 ==== 5558+0+0 (secure 0 0 0) 0x7f4088083470 con 0x7f409414f760 2023-05-22T15:46:59.534 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:59.523+0000 7f4099aaf640 1 --2- 172.21.15.16:0/2068672889 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f40800654d0 0x7f4080067980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:59.534 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:59.523+0000 7f4099aaf640 1 --2- 172.21.15.16:0/2068672889 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f40800654d0 0x7f4080067980 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f40840af000 tx=0x7f4084008040 comp rx=0 tx=0).ready entity=mgr.9396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:59.534 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:59.527+0000 7f40927fc640 1 -- 172.21.15.16:0/2068672889 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f4088051150 con 0x7f409414f760 2023-05-22T15:46:59.740 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:59.735+0000 7f409bd13640 1 -- 172.21.15.16:0/2068672889 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f40940cb010 con 0x7f409414f760 2023-05-22T15:46:59.744 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:59.735+0000 7f40927fc640 1 -- 172.21.15.16:0/2068672889 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v25) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f4088054770 con 0x7f409414f760 2023-05-22T15:46:59.745 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:46:59.745 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-05-22T15:46:59.746 INFO:teuthology.orchestra.run.smithi016.stdout: "auth_dump": [ 2023-05-22T15:46:59.746 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T15:46:59.746 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "mds.a", 2023-05-22T15:46:59.746 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQCAiWtkw/K0GRAA9hvTMKMmcbgKV68GeoC4ag==", 2023-05-22T15:46:59.746 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-05-22T15:46:59.746 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2023-05-22T15:46:59.746 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow *", 2023-05-22T15:46:59.747 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow *", 2023-05-22T15:46:59.747 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-05-22T15:46:59.747 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T15:46:59.747 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:46:59.747 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T15:46:59.747 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "mds.b", 2023-05-22T15:46:59.748 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQCAiWtkoFzfJRAAxCiB0MGM5apMVA81WDq+dA==", 2023-05-22T15:46:59.748 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-05-22T15:46:59.748 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2023-05-22T15:46:59.748 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow *", 2023-05-22T15:46:59.748 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow *", 2023-05-22T15:46:59.748 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-05-22T15:46:59.748 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T15:46:59.749 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:46:59.749 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T15:46:59.749 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "mds.c", 2023-05-22T15:46:59.749 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQCAiWtkvgOKHxAAMuASPtMqwyoqpZ5I+MRS7A==", 2023-05-22T15:46:59.749 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-05-22T15:46:59.749 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2023-05-22T15:46:59.750 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow *", 2023-05-22T15:46:59.750 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow *", 2023-05-22T15:46:59.750 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-05-22T15:46:59.750 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T15:46:59.750 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:46:59.750 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T15:46:59.750 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "mds.d", 2023-05-22T15:46:59.751 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQCAiWtk6hPQKxAAlCtfC36vja5ftz6uOqUhHg==", 2023-05-22T15:46:59.751 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-05-22T15:46:59.751 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2023-05-22T15:46:59.751 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow *", 2023-05-22T15:46:59.751 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow *", 2023-05-22T15:46:59.751 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-05-22T15:46:59.751 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T15:46:59.751 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:46:59.752 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T15:46:59.752 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.0", 2023-05-22T15:46:59.752 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQCCiWtk6ZLoCRAAKdCQitZuPnWcIpkiXhJeww==", 2023-05-22T15:46:59.752 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-05-22T15:46:59.752 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2023-05-22T15:46:59.752 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2023-05-22T15:46:59.752 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-05-22T15:46:59.753 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T15:46:59.753 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:46:59.753 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T15:46:59.753 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.1", 2023-05-22T15:46:59.753 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQCFiWtkXtVeERAAkgPeDKuFmWXrDiCyqWOYwQ==", 2023-05-22T15:46:59.753 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-05-22T15:46:59.753 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2023-05-22T15:46:59.754 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2023-05-22T15:46:59.754 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-05-22T15:46:59.754 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T15:46:59.754 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:46:59.754 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T15:46:59.754 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.2", 2023-05-22T15:46:59.755 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQCIiWtkdGaRHRAAx3POIp3mnz7p7XcCVMXm5Q==", 2023-05-22T15:46:59.755 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-05-22T15:46:59.755 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2023-05-22T15:46:59.755 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2023-05-22T15:46:59.755 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-05-22T15:46:59.755 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T15:46:59.755 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:46:59.755 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T15:46:59.756 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.3", 2023-05-22T15:46:59.756 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQCLiWtkSBydJhAAsgc19Xaf/GOyy9nm9MLdJQ==", 2023-05-22T15:46:59.756 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-05-22T15:46:59.756 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2023-05-22T15:46:59.756 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2023-05-22T15:46:59.756 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-05-22T15:46:59.756 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T15:46:59.757 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:46:59.757 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T15:46:59.757 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.4", 2023-05-22T15:46:59.757 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQCRiWtkueYvJRAARXbHlKHrTbgTqBArRJfvBA==", 2023-05-22T15:46:59.757 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-05-22T15:46:59.757 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2023-05-22T15:46:59.757 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2023-05-22T15:46:59.757 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-05-22T15:46:59.758 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T15:46:59.758 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:46:59.758 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T15:46:59.758 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.5", 2023-05-22T15:46:59.758 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQCUiWtkH3hwLRAAUekP6y+wkJVPh+OPDiZbzQ==", 2023-05-22T15:46:59.758 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-05-22T15:46:59.758 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2023-05-22T15:46:59.759 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2023-05-22T15:46:59.759 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-05-22T15:46:59.759 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T15:46:59.759 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:46:59.759 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T15:46:59.759 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.6", 2023-05-22T15:46:59.759 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQCXiWtkLsnBNhAAQ8ilX8ZU4ncJ2JJU0I+U4w==", 2023-05-22T15:46:59.760 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-05-22T15:46:59.760 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2023-05-22T15:46:59.760 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2023-05-22T15:46:59.760 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-05-22T15:46:59.760 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T15:46:59.760 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:46:59.760 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T15:46:59.761 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.7", 2023-05-22T15:46:59.761 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQCbiWtkUvXjBBAApMWw7wdfxWq0WEQDL2qprA==", 2023-05-22T15:46:59.761 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-05-22T15:46:59.761 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2023-05-22T15:46:59.761 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2023-05-22T15:46:59.761 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-05-22T15:46:59.761 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T15:46:59.762 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:46:59.762 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T15:46:59.762 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "client.0", 2023-05-22T15:46:59.762 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQCAiWtkZwuUMRAAnYqSFiaT/QRsajdjNgTetA==", 2023-05-22T15:46:59.762 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-05-22T15:46:59.762 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2023-05-22T15:46:59.762 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow r", 2023-05-22T15:46:59.762 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow rw tag cephfs data=cephfs" 2023-05-22T15:46:59.763 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T15:46:59.763 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:46:59.763 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T15:46:59.763 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "client.1", 2023-05-22T15:46:59.763 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQCAiWtkPimmORAABzNL2rANz5TgCl3p++GVng==", 2023-05-22T15:46:59.763 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-05-22T15:46:59.763 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2023-05-22T15:46:59.763 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow r", 2023-05-22T15:46:59.764 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow rw", 2023-05-22T15:46:59.764 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow rwx" 2023-05-22T15:46:59.764 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T15:46:59.764 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:46:59.764 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T15:46:59.764 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "client.2", 2023-05-22T15:46:59.764 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQCAiWtkqkPMNhAAqugrnD4Mh/4At8QAFmzqcg==", 2023-05-22T15:46:59.764 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-05-22T15:46:59.765 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2023-05-22T15:46:59.765 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow r", 2023-05-22T15:46:59.765 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow rw", 2023-05-22T15:46:59.765 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow rwx" 2023-05-22T15:46:59.765 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T15:46:59.765 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:46:59.765 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T15:46:59.766 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "client.3", 2023-05-22T15:46:59.766 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQCBiWtkpvlWAxAAPuedivXBGZGHIO6hSnz8PQ==", 2023-05-22T15:46:59.766 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-05-22T15:46:59.766 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2023-05-22T15:46:59.766 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow r", 2023-05-22T15:46:59.766 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow rw", 2023-05-22T15:46:59.766 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow rwx" 2023-05-22T15:46:59.766 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T15:46:59.767 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:46:59.767 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T15:46:59.767 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "client.admin", 2023-05-22T15:46:59.767 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQCAiWtkxG9hAhAAfnUnRrBRUFoCEtsb2riNOQ==", 2023-05-22T15:46:59.767 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-05-22T15:46:59.767 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow *", 2023-05-22T15:46:59.767 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow *", 2023-05-22T15:46:59.768 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow *", 2023-05-22T15:46:59.768 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-05-22T15:46:59.768 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T15:46:59.768 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:46:59.768 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T15:46:59.768 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "mgr.x", 2023-05-22T15:46:59.768 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQCAiWtkkaIqFhAAb/4BI6FYWOy3qOSaylXLSw==", 2023-05-22T15:46:59.768 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-05-22T15:46:59.769 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow *", 2023-05-22T15:46:59.769 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile mgr", 2023-05-22T15:46:59.769 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-05-22T15:46:59.769 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T15:46:59.769 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:46:59.769 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T15:46:59.769 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "mgr.y", 2023-05-22T15:46:59.770 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQCAiWtkhgVWExAAjDoh7cexFVfsigsYfGkmVg==", 2023-05-22T15:46:59.770 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-05-22T15:46:59.770 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow *", 2023-05-22T15:46:59.770 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile mgr", 2023-05-22T15:46:59.770 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-05-22T15:46:59.770 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T15:46:59.770 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T15:46:59.770 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-05-22T15:46:59.771 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-05-22T15:46:59.772 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:59.739+0000 7f409bd13640 1 -- 172.21.15.16:0/2068672889 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f40800654d0 msgr2=0x7f4080067980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:59.772 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:59.739+0000 7f409bd13640 1 --2- 172.21.15.16:0/2068672889 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f40800654d0 0x7f4080067980 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f40840af000 tx=0x7f4084008040 comp rx=0 tx=0).stop 2023-05-22T15:46:59.772 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:59.739+0000 7f409bd13640 1 -- 172.21.15.16:0/2068672889 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f409414f760 msgr2=0x7f4094151ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:59.772 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:59.739+0000 7f409bd13640 1 --2- 172.21.15.16:0/2068672889 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f409414f760 0x7f4094151ad0 secure :-1 s=READY pgs=2012 cs=0 l=1 rev1=1 crypto rx=0x7f408800e410 tx=0x7f408800e8d0 comp rx=0 tx=0).stop 2023-05-22T15:46:59.773 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:59.739+0000 7f409bd13640 1 -- 172.21.15.16:0/2068672889 shutdown_connections 2023-05-22T15:46:59.773 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:59.739+0000 7f409bd13640 1 --2- 172.21.15.16:0/2068672889 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f40800654d0 0x7f4080067980 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:59.773 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:59.739+0000 7f409bd13640 1 --2- 172.21.15.16:0/2068672889 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f409415cc80 0x7f4094152010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:59.773 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:59.739+0000 7f409bd13640 1 --2- 172.21.15.16:0/2068672889 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f409414f760 0x7f4094151ad0 unknown :-1 s=CLOSED pgs=2012 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:59.773 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:59.739+0000 7f409bd13640 1 --2- 172.21.15.16:0/2068672889 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f409414f390 0x7f40941f3690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:59.773 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:59.739+0000 7f409bd13640 1 -- 172.21.15.16:0/2068672889 >> 172.21.15.16:0/2068672889 conn(0x7f4094069720 msgr2=0x7f40940b2270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:46:59.773 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:59.739+0000 7f409bd13640 1 -- 172.21.15.16:0/2068672889 shutdown_connections 2023-05-22T15:46:59.773 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:59.739+0000 7f409bd13640 1 -- 172.21.15.16:0/2068672889 wait complete. 2023-05-22T15:46:59.774 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-05-22T15:46:59.900 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:59.895+0000 7fd6d5196640 1 Processor -- start 2023-05-22T15:46:59.901 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:59.895+0000 7fd6d5196640 1 -- start start 2023-05-22T15:46:59.901 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:59.895+0000 7fd6d5196640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fd6d0153fe0 0x7fd6d006d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:59.901 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:59.895+0000 7fd6d5196640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd6d0070600 0x7fd6d00ad410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:59.901 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:59.895+0000 7fd6d5196640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6d0069720 con 0x7fd6d00cae20 2023-05-22T15:46:59.901 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:59.895+0000 7fd6d5196640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6d01543b0 con 0x7fd6d0153fe0 2023-05-22T15:46:59.902 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:59.895+0000 7fd6d5196640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6d015e360 con 0x7fd6d0070600 2023-05-22T15:46:59.902 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:59.895+0000 7fd6ced9d640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fd6d0153fe0 0x7fd6d006d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:59.903 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:59.895+0000 7fd6cf59e640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd6d0070600 0x7fd6d00ad410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:59.903 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:59.895+0000 7fd6ced9d640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fd6d0153fe0 0x7fd6d006d3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.16:54766/0 (socket says 172.21.15.16:54766) 2023-05-22T15:46:59.903 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:59.895+0000 7fd6ce59c640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7fd6d00cae20 0x7fd6d006fef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.16:52976/0 (socket says 172.21.15.16:52976) 2023-05-22T15:46:59.903 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:59.895+0000 7fd6ced9d640 1 -- 172.21.15.16:0/1213521961 learned_addr learned my addr 172.21.15.16:0/1213521961 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:46:59.903 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:59.895+0000 7fd6ced9d640 1 -- 172.21.15.16:0/1213521961 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd6d0070600 msgr2=0x7fd6d00ad410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:46:59.903 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:59.895+0000 7fd6ced9d640 1 --2- 172.21.15.16:0/1213521961 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd6d0070600 0x7fd6d00ad410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:59.904 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:59.895+0000 7fd6ced9d640 1 -- 172.21.15.16:0/1213521961 >> v1:172.21.15.62:6789/0 conn(0x7fd6d00cae20 legacy=0x7fd6d006fef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:59.904 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:59.895+0000 7fd6ced9d640 1 -- 172.21.15.16:0/1213521961 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6d015d5a0 con 0x7fd6d0153fe0 2023-05-22T15:46:59.904 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:59.899+0000 7fd6ced9d640 1 --2- 172.21.15.16:0/1213521961 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd6d0153fe0 0x7fd6d006d3f0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7fd6c0009870 tx=0x7fd6c001c510 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1505db891308e7d1 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:59.906 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:59.899+0000 7fd6cdd9b640 1 -- 172.21.15.16:0/1213521961 <== mon.1 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6c00aa670 con 0x7fd6d0153fe0 2023-05-22T15:46:59.906 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:59.899+0000 7fd6cdd9b640 1 -- 172.21.15.16:0/1213521961 <== mon.1 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6c00aa7d0 con 0x7fd6d0153fe0 2023-05-22T15:46:59.906 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:59.899+0000 7fd6cdd9b640 1 -- 172.21.15.16:0/1213521961 <== mon.1 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6c00aab60 con 0x7fd6d0153fe0 2023-05-22T15:46:59.906 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:59.899+0000 7fd6d5196640 1 -- 172.21.15.16:0/1213521961 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd6d0153fe0 msgr2=0x7fd6d006d3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:59.906 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:59.899+0000 7fd6d5196640 1 --2- 172.21.15.16:0/1213521961 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd6d0153fe0 0x7fd6d006d3f0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7fd6c0009870 tx=0x7fd6c001c510 comp rx=0 tx=0).stop 2023-05-22T15:46:59.907 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:59.899+0000 7fd6d5196640 1 -- 172.21.15.16:0/1213521961 shutdown_connections 2023-05-22T15:46:59.907 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:59.899+0000 7fd6d5196640 1 --2- 172.21.15.16:0/1213521961 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd6d0070600 0x7fd6d00ad410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:59.907 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:59.899+0000 7fd6d5196640 1 --2- 172.21.15.16:0/1213521961 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd6d0153fe0 0x7fd6d006d3f0 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:59.907 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:59.899+0000 7fd6d5196640 1 -- 172.21.15.16:0/1213521961 >> 172.21.15.16:0/1213521961 conn(0x7fd6d00b1110 msgr2=0x7fd6d00ce830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:46:59.907 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:59.899+0000 7fd6d5196640 1 -- 172.21.15.16:0/1213521961 shutdown_connections 2023-05-22T15:46:59.907 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:59.899+0000 7fd6d5196640 1 -- 172.21.15.16:0/1213521961 wait complete. 2023-05-22T15:46:59.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:59.903+0000 7fd6d5196640 1 Processor -- start 2023-05-22T15:46:59.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:59.903+0000 7fd6d5196640 1 -- start start 2023-05-22T15:46:59.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:59.903+0000 7fd6d5196640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd6d0070600 0x7fd6d01558f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:59.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:59.903+0000 7fd6d5196640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd6d00cae20 0x7fd6d0147f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:59.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:59.903+0000 7fd6d5196640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd6d0153fe0 0x7fd6d014c0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:59.911 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:59.903+0000 7fd6d5196640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6d0157380 con 0x7fd6d0070600 2023-05-22T15:46:59.911 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:59.903+0000 7fd6d5196640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6d0156320 con 0x7fd6d0153fe0 2023-05-22T15:46:59.911 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:59.903+0000 7fd6d5196640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6d01565c0 con 0x7fd6d00cae20 2023-05-22T15:46:59.911 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:59.903+0000 7fd6ced9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd6d0070600 0x7fd6d01558f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:59.911 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:59.903+0000 7fd6ced9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd6d0070600 0x7fd6d01558f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46662/0 (socket says 172.21.15.16:46662) 2023-05-22T15:46:59.911 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:59.903+0000 7fd6ced9d640 1 -- 172.21.15.16:0/1666662770 learned_addr learned my addr 172.21.15.16:0/1666662770 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:46:59.911 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:59.903+0000 7fd6ce59c640 1 --2- 172.21.15.16:0/1666662770 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd6d00cae20 0x7fd6d0147f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:59.912 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:59.903+0000 7fd6cf59e640 1 --2- 172.21.15.16:0/1666662770 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd6d0153fe0 0x7fd6d014c0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:59.912 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:59.903+0000 7fd6ced9d640 1 -- 172.21.15.16:0/1666662770 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd6d00cae20 msgr2=0x7fd6d0147f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:59.912 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:59.903+0000 7fd6ced9d640 1 --2- 172.21.15.16:0/1666662770 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd6d00cae20 0x7fd6d0147f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:59.912 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:59.903+0000 7fd6ced9d640 1 -- 172.21.15.16:0/1666662770 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd6d0153fe0 msgr2=0x7fd6d014c0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:46:59.912 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:59.903+0000 7fd6ced9d640 1 --2- 172.21.15.16:0/1666662770 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd6d0153fe0 0x7fd6d014c0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:46:59.912 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:59.903+0000 7fd6ced9d640 1 -- 172.21.15.16:0/1666662770 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6c00094d0 con 0x7fd6d0070600 2023-05-22T15:46:59.913 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:59.903+0000 7fd6ced9d640 1 --2- 172.21.15.16:0/1666662770 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd6d0070600 0x7fd6d01558f0 secure :-1 s=READY pgs=2013 cs=0 l=1 rev1=1 crypto rx=0x7fd6c00ac040 tx=0x7fd6c001ad50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:59.913 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:59.903+0000 7fd6b77fe640 1 -- 172.21.15.16:0/1666662770 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6c00b3800 con 0x7fd6d0070600 2023-05-22T15:46:59.913 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:59.903+0000 7fd6b77fe640 1 -- 172.21.15.16:0/1666662770 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6c00b3960 con 0x7fd6d0070600 2023-05-22T15:46:59.913 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:59.903+0000 7fd6d5196640 1 -- 172.21.15.16:0/1666662770 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6d006dd20 con 0x7fd6d0070600 2023-05-22T15:46:59.913 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:59.903+0000 7fd6b77fe640 1 -- 172.21.15.16:0/1666662770 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6c00b3d10 con 0x7fd6d0070600 2023-05-22T15:46:59.913 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:59.903+0000 7fd6d5196640 1 -- 172.21.15.16:0/1666662770 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6d015de40 con 0x7fd6d0070600 2023-05-22T15:46:59.913 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:59.903+0000 7fd6d5196640 1 -- 172.21.15.16:0/1666662770 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6d006d930 con 0x7fd6d0070600 2023-05-22T15:46:59.913 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:59.903+0000 7fd6b77fe640 1 -- 172.21.15.16:0/1666662770 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 67) v1 ==== 84332+0+0 (secure 0 0 0) 0x7fd6c00c6440 con 0x7fd6d0070600 2023-05-22T15:46:59.914 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:59.903+0000 7fd6b77fe640 1 --2- 172.21.15.16:0/1666662770 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd6bc0654a0 0x7fd6bc067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:46:59.914 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:59.907+0000 7fd6b77fe640 1 -- 172.21.15.16:0/1666662770 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(157..157 src has 1..157) v4 ==== 5558+0+0 (secure 0 0 0) 0x7fd6c011d7c0 con 0x7fd6d0070600 2023-05-22T15:46:59.914 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:59.907+0000 7fd6ce59c640 1 --2- 172.21.15.16:0/1666662770 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd6bc0654a0 0x7fd6bc067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:46:59.915 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:59.907+0000 7fd6ce59c640 1 --2- 172.21.15.16:0/1666662770 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd6bc0654a0 0x7fd6bc067950 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fd6d0069a60 tx=0x7fd6c4007cd0 comp rx=0 tx=0).ready entity=mgr.9396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:46:59.915 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:46:59.907+0000 7fd6b77fe640 1 -- 172.21.15.16:0/1666662770 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fd6c00eb3c0 con 0x7fd6d0070600 2023-05-22T15:47:00.062 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.055+0000 7fd6d5196640 1 -- 172.21.15.16:0/1666662770 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd6d004dbe0 con 0x7fd6d0070600 2023-05-22T15:47:00.064 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.059+0000 7fd6b77fe640 1 -- 172.21.15.16:0/1666662770 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v157) v1 ==== 74+0+14382 (secure 0 0 0) 0x7fd6c00ee9e0 con 0x7fd6d0070600 2023-05-22T15:47:00.064 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:47:00.065 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":157,"fsid":"b2719834-7847-4d98-90a6-f65d57f85c49","created":"2023-05-22T15:26:25.259314+0000","modified":"2023-05-22T15:46:59.356002+0000","last_up_change":"2023-05-22T15:26:36.448524+0000","last_in_change":"2023-05-22T15:26:27.675668+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-05-22T15:26:38.019781+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-05-22T15:26:43.570514+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9e7ff854-b25f-4cbb-a5c2-1799d9c53d2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":37021},{"type":"v1","addr":"172.21.15.16:6825","nonce":37021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":37021},{"type":"v1","addr":"172.21.15.16:6827","nonce":37021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":37021},{"type":"v1","addr":"172.21.15.16:6831","nonce":37021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":37021},{"type":"v1","addr":"172.21.15.16:6829","nonce":37021}]},"public_addr":"172.21.15.16:6825/37021","cluster_addr":"172.21.15.16:6827/37021","heartbeat_back_addr":"172.21.15.16:6831/37021","heartbeat_front_addr":"172.21.15.16:6829/37021","state":["exists","up"]},{"osd":1,"uuid":"4702fbc9-a927-4565-aca5-41dc69cc8264","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":37008},{"type":"v1","addr":"172.21.15.16:6801","nonce":37008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":37008},{"type":"v1","addr":"172.21.15.16:6803","nonce":37008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":37008},{"type":"v1","addr":"172.21.15.16:6807","nonce":37008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":37008},{"type":"v1","addr":"172.21.15.16:6805","nonce":37008}]},"public_addr":"172.21.15.16:6801/37008","cluster_addr":"172.21.15.16:6803/37008","heartbeat_back_addr":"172.21.15.16:6807/37008","heartbeat_front_addr":"172.21.15.16:6805/37008","state":["exists","up"]},{"osd":2,"uuid":"1e7bba46-4eba-47ef-bb81-5e2d371638ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":37009},{"type":"v1","addr":"172.21.15.16:6809","nonce":37009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":37009},{"type":"v1","addr":"172.21.15.16:6811","nonce":37009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":37009},{"type":"v1","addr":"172.21.15.16:6815","nonce":37009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":37009},{"type":"v1","addr":"172.21.15.16:6813","nonce":37009}]},"public_addr":"172.21.15.16:6809/37009","cluster_addr":"172.21.15.16:6811/37009","heartbeat_back_addr":"172.21.15.16:6815/37009","heartbeat_front_addr":"172.21.15.16:6813/37009","state":["exists","up"]},{"osd":3,"uuid":"383cacc0-5a88-4904-990e-498ca59ef48a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":37020},{"type":"v1","addr":"172.21.15.16:6817","nonce":37020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":37020},{"type":"v1","addr":"172.21.15.16:6819","nonce":37020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":37020},{"type":"v1","addr":"172.21.15.16:6823","nonce":37020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":37020},{"type":"v1","addr":"172.21.15.16:6821","nonce":37020}]},"public_addr":"172.21.15.16:6817/37020","cluster_addr":"172.21.15.16:6819/37020","heartbeat_back_addr":"172.21.15.16:6823/37020","heartbeat_front_addr":"172.21.15.16:6821/37020","state":["exists","up"]},{"osd":4,"uuid":"970500ef-95b7-42c1-b080-eed4fae7013d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":37353},{"type":"v1","addr":"172.21.15.62:6817","nonce":37353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6819","nonce":37353},{"type":"v1","addr":"172.21.15.62:6822","nonce":37353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37353},{"type":"v1","addr":"172.21.15.62:6831","nonce":37353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37353},{"type":"v1","addr":"172.21.15.62:6826","nonce":37353}]},"public_addr":"172.21.15.62:6817/37353","cluster_addr":"172.21.15.62:6822/37353","heartbeat_back_addr":"172.21.15.62:6831/37353","heartbeat_front_addr":"172.21.15.62:6826/37353","state":["exists","up"]},{"osd":5,"uuid":"022d67db-d4d8-445b-b1b9-8b446a00f5c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":37352},{"type":"v1","addr":"172.21.15.62:6801","nonce":37352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":37352},{"type":"v1","addr":"172.21.15.62:6803","nonce":37352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":37352},{"type":"v1","addr":"172.21.15.62:6807","nonce":37352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":37352},{"type":"v1","addr":"172.21.15.62:6805","nonce":37352}]},"public_addr":"172.21.15.62:6801/37352","cluster_addr":"172.21.15.62:6803/37352","heartbeat_back_addr":"172.21.15.62:6807/37352","heartbeat_front_addr":"172.21.15.62:6805/37352","state":["exists","up"]},{"osd":6,"uuid":"42bd372b-6688-4a1b-8407-e16e8c266e33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":37354},{"type":"v1","addr":"172.21.15.62:6820","nonce":37354}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":37354},{"type":"v1","addr":"172.21.15.62:6823","nonce":37354}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6829","nonce":37354},{"type":"v1","addr":"172.21.15.62:6830","nonce":37354}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6825","nonce":37354},{"type":"v1","addr":"172.21.15.62:6827","nonce":37354}]},"public_addr":"172.21.15.62:6820/37354","cluster_addr":"172.21.15.62:6823/37354","heartbeat_back_addr":"172.21.15.62:6830/37354","heartbeat_front_addr":"172.21.15.62:6827/37354","state":["exists","up"]},{"osd":7,"uuid":"ac70b2a3-f41b-4bf2-b227-7f2cbb55426d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37355},{"type":"v1","addr":"172.21.15.62:6809","nonce":37355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37355},{"type":"v1","addr":"172.21.15.62:6811","nonce":37355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":37355},{"type":"v1","addr":"172.21.15.62:6815","nonce":37355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":37355},{"type":"v1","addr":"172.21.15.62:6813","nonce":37355}]},"public_addr":"172.21.15.62:6809/37355","cluster_addr":"172.21.15.62:6811/37355","heartbeat_back_addr":"172.21.15.62:6815/37355","heartbeat_front_addr":"172.21.15.62:6813/37355","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.368896+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.107273+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.276888+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.287502+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.124791+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.251937+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.255679+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.232797+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":17,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-05-22T15:47:00.066 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.059+0000 7fd6d5196640 1 -- 172.21.15.16:0/1666662770 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd6bc0654a0 msgr2=0x7fd6bc067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:47:00.066 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.059+0000 7fd6d5196640 1 --2- 172.21.15.16:0/1666662770 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd6bc0654a0 0x7fd6bc067950 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fd6d0069a60 tx=0x7fd6c4007cd0 comp rx=0 tx=0).stop 2023-05-22T15:47:00.066 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.059+0000 7fd6d5196640 1 -- 172.21.15.16:0/1666662770 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd6d0070600 msgr2=0x7fd6d01558f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:47:00.066 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.059+0000 7fd6d5196640 1 --2- 172.21.15.16:0/1666662770 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd6d0070600 0x7fd6d01558f0 secure :-1 s=READY pgs=2013 cs=0 l=1 rev1=1 crypto rx=0x7fd6c00ac040 tx=0x7fd6c001ad50 comp rx=0 tx=0).stop 2023-05-22T15:47:00.066 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.059+0000 7fd6d5196640 1 -- 172.21.15.16:0/1666662770 shutdown_connections 2023-05-22T15:47:00.067 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.059+0000 7fd6d5196640 1 --2- 172.21.15.16:0/1666662770 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd6d0153fe0 0x7fd6d014c0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:00.067 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.059+0000 7fd6d5196640 1 --2- 172.21.15.16:0/1666662770 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd6bc0654a0 0x7fd6bc067950 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:00.067 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.059+0000 7fd6d5196640 1 --2- 172.21.15.16:0/1666662770 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd6d00cae20 0x7fd6d0147f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:00.067 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.059+0000 7fd6d5196640 1 --2- 172.21.15.16:0/1666662770 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd6d0070600 0x7fd6d01558f0 unknown :-1 s=CLOSED pgs=2013 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:00.067 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.059+0000 7fd6d5196640 1 -- 172.21.15.16:0/1666662770 >> 172.21.15.16:0/1666662770 conn(0x7fd6d00b1110 msgr2=0x7fd6d00afbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:47:00.067 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.059+0000 7fd6d5196640 1 -- 172.21.15.16:0/1666662770 shutdown_connections 2023-05-22T15:47:00.067 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.059+0000 7fd6d5196640 1 -- 172.21.15.16:0/1666662770 wait complete. 2023-05-22T15:47:00.082 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-05-22T15:26:38.019781+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-05-22T15:26:43.570514+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-05-22T15:47:00.083 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-05-22T15:47:00.211 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.203+0000 7f82a4146640 1 Processor -- start 2023-05-22T15:47:00.211 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.203+0000 7f82a4146640 1 -- start start 2023-05-22T15:47:00.212 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.207+0000 7f82a4146640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f829c159970 0x7f829c159d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:47:00.212 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.207+0000 7f82a4146640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f829c15d680 0x7f829c0cb260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:47:00.212 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.207+0000 7f82a4146640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f829c070c30 con 0x7f829c15dad0 2023-05-22T15:47:00.212 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.207+0000 7f82a4146640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f829c070e60 con 0x7f829c15d680 2023-05-22T15:47:00.213 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.207+0000 7f82a4146640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f829c071140 con 0x7f829c159970 2023-05-22T15:47:00.213 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.207+0000 7f82a1ee2640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f829c159970 0x7f829c159d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:47:00.213 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.207+0000 7f82a16e1640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f829c15d680 0x7f829c0cb260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:47:00.213 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.207+0000 7f82a26e3640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f829c15dad0 0x7f829c06b6c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.16:52992/0 (socket says 172.21.15.16:52992) 2023-05-22T15:47:00.213 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.207+0000 7f82a26e3640 1 -- 172.21.15.16:0/3238449718 learned_addr learned my addr 172.21.15.16:0/3238449718 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:47:00.214 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.207+0000 7f82a1ee2640 1 -- 172.21.15.16:0/3238449718 >> v2:172.21.15.62:3300/0 conn(0x7f829c15d680 msgr2=0x7f829c0cb260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:47:00.214 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.207+0000 7f82a1ee2640 1 --2- 172.21.15.16:0/3238449718 >> v2:172.21.15.62:3300/0 conn(0x7f829c15d680 0x7f829c0cb260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:00.214 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.207+0000 7f82a1ee2640 1 -- 172.21.15.16:0/3238449718 >> v1:172.21.15.62:6789/0 conn(0x7f829c15dad0 legacy=0x7f829c06b6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:47:00.214 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.207+0000 7f82a1ee2640 1 -- 172.21.15.16:0/3238449718 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f829c071420 con 0x7f829c159970 2023-05-22T15:47:00.215 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.211+0000 7f82a1ee2640 1 --2- 172.21.15.16:0/3238449718 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f829c159970 0x7f829c159d40 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f82980097c0 tx=0x7f829801c3c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=268b0421c74d0104 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:47:00.215 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.211+0000 7f82a0ee0640 1 -- 172.21.15.16:0/3238449718 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82980ac640 con 0x7f829c159970 2023-05-22T15:47:00.215 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.211+0000 7f82a0ee0640 1 -- 172.21.15.16:0/3238449718 <== mon.2 v2:172.21.15.62:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82980ac7a0 con 0x7f829c159970 2023-05-22T15:47:00.215 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.211+0000 7f82a0ee0640 1 -- 172.21.15.16:0/3238449718 <== mon.2 v2:172.21.15.62:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82980acb30 con 0x7f829c159970 2023-05-22T15:47:00.216 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.211+0000 7f82a4146640 1 -- 172.21.15.16:0/3238449718 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f829c159970 msgr2=0x7f829c159d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:47:00.216 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.211+0000 7f82a4146640 1 --2- 172.21.15.16:0/3238449718 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f829c159970 0x7f829c159d40 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f82980097c0 tx=0x7f829801c3c0 comp rx=0 tx=0).stop 2023-05-22T15:47:00.217 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.211+0000 7f82a4146640 1 -- 172.21.15.16:0/3238449718 shutdown_connections 2023-05-22T15:47:00.217 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.211+0000 7f82a4146640 1 --2- 172.21.15.16:0/3238449718 >> v2:172.21.15.62:3300/0 conn(0x7f829c15d680 0x7f829c0cb260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:00.217 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.211+0000 7f82a4146640 1 --2- 172.21.15.16:0/3238449718 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f829c159970 0x7f829c159d40 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:00.217 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.211+0000 7f82a4146640 1 -- 172.21.15.16:0/3238449718 >> 172.21.15.16:0/3238449718 conn(0x7f829c069720 msgr2=0x7f829c069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:47:00.217 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.211+0000 7f82a4146640 1 -- 172.21.15.16:0/3238449718 shutdown_connections 2023-05-22T15:47:00.217 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.211+0000 7f82a4146640 1 -- 172.21.15.16:0/3238449718 wait complete. 2023-05-22T15:47:00.218 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.211+0000 7f82a4146640 1 Processor -- start 2023-05-22T15:47:00.218 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.211+0000 7f82a4146640 1 -- start start 2023-05-22T15:47:00.224 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.211+0000 7f82a4146640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f829c159970 0x7f829c153e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:47:00.224 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.211+0000 7f82a4146640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f829c15d680 0x7f829c14c420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:47:00.224 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.211+0000 7f82a4146640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f829c15dad0 0x7f829c14c960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:47:00.224 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.211+0000 7f82a4146640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f829c14d6e0 con 0x7f829c15d680 2023-05-22T15:47:00.225 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.211+0000 7f82a4146640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f829c0b17c0 con 0x7f829c15dad0 2023-05-22T15:47:00.225 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.211+0000 7f82a4146640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f829c0b1a10 con 0x7f829c159970 2023-05-22T15:47:00.225 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.215+0000 7f82a16e1640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f829c15d680 0x7f829c14c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:47:00.225 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.215+0000 7f82a16e1640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f829c15d680 0x7f829c14c420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46668/0 (socket says 172.21.15.16:46668) 2023-05-22T15:47:00.225 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.215+0000 7f82a16e1640 1 -- 172.21.15.16:0/3204301894 learned_addr learned my addr 172.21.15.16:0/3204301894 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:47:00.225 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.215+0000 7f82a1ee2640 1 --2- 172.21.15.16:0/3204301894 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f829c159970 0x7f829c153e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:47:00.226 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.215+0000 7f82a26e3640 1 --2- 172.21.15.16:0/3204301894 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f829c15dad0 0x7f829c14c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:47:00.226 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.215+0000 7f82a16e1640 1 -- 172.21.15.16:0/3204301894 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f829c159970 msgr2=0x7f829c153e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:47:00.226 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.215+0000 7f82a16e1640 1 --2- 172.21.15.16:0/3204301894 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f829c159970 0x7f829c153e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:00.226 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.215+0000 7f82a16e1640 1 -- 172.21.15.16:0/3204301894 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f829c15dad0 msgr2=0x7f829c14c960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:47:00.226 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.215+0000 7f82a16e1640 1 --2- 172.21.15.16:0/3204301894 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f829c15dad0 0x7f829c14c960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:00.227 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.215+0000 7f82a16e1640 1 -- 172.21.15.16:0/3204301894 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8298009420 con 0x7f829c15d680 2023-05-22T15:47:00.227 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.215+0000 7f82a16e1640 1 --2- 172.21.15.16:0/3204301894 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f829c15d680 0x7f829c14c420 secure :-1 s=READY pgs=2014 cs=0 l=1 rev1=1 crypto rx=0x7f828c00e960 tx=0x7f828c00ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:47:00.227 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.215+0000 7f82927fc640 1 -- 172.21.15.16:0/3204301894 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f828c009690 con 0x7f829c15d680 2023-05-22T15:47:00.227 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.215+0000 7f82927fc640 1 -- 172.21.15.16:0/3204301894 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f828c00c9a0 con 0x7f829c15d680 2023-05-22T15:47:00.227 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.215+0000 7f82a4146640 1 -- 172.21.15.16:0/3204301894 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f829c0cae20 con 0x7f829c15d680 2023-05-22T15:47:00.227 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.215+0000 7f82927fc640 1 -- 172.21.15.16:0/3204301894 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f828c00cd10 con 0x7f829c15d680 2023-05-22T15:47:00.227 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.215+0000 7f82a4146640 1 -- 172.21.15.16:0/3204301894 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f829c0cac90 con 0x7f829c15d680 2023-05-22T15:47:00.228 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.215+0000 7f82a4146640 1 -- 172.21.15.16:0/3204301894 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f829c06bdd0 con 0x7f829c15d680 2023-05-22T15:47:00.228 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.215+0000 7f82927fc640 1 -- 172.21.15.16:0/3204301894 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 67) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f828c013470 con 0x7f829c15d680 2023-05-22T15:47:00.228 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.215+0000 7f82927fc640 1 --2- 172.21.15.16:0/3204301894 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f82800654a0 0x7f8280067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:47:00.228 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.215+0000 7f82927fc640 1 -- 172.21.15.16:0/3204301894 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(157..157 src has 1..157) v4 ==== 5558+0+0 (secure 0 0 0) 0x7f828c0828a0 con 0x7f829c15d680 2023-05-22T15:47:00.228 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.215+0000 7f82a1ee2640 1 --2- 172.21.15.16:0/3204301894 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f82800654a0 0x7f8280067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:47:00.228 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.215+0000 7f82a1ee2640 1 --2- 172.21.15.16:0/3204301894 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f82800654a0 0x7f8280067950 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f82980ae000 tx=0x7f82980a7000 comp rx=0 tx=0).ready entity=mgr.9396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:47:00.229 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.219+0000 7f82927fc640 1 -- 172.21.15.16:0/3204301894 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f828c0506e0 con 0x7f829c15d680 2023-05-22T15:47:00.382 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.375+0000 7f82a4146640 1 -- 172.21.15.16:0/3204301894 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f829c04dbe0 con 0x7f829c15d680 2023-05-22T15:47:00.383 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.379+0000 7f82927fc640 1 -- 172.21.15.16:0/3204301894 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v157) v1 ==== 93+0+10 (secure 0 0 0) 0x7f828c053d00 con 0x7f829c15d680 2023-05-22T15:47:00.383 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2023-05-22T15:47:00.386 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.379+0000 7f82a4146640 1 -- 172.21.15.16:0/3204301894 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f82800654a0 msgr2=0x7f8280067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:47:00.386 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.379+0000 7f82a4146640 1 --2- 172.21.15.16:0/3204301894 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f82800654a0 0x7f8280067950 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f82980ae000 tx=0x7f82980a7000 comp rx=0 tx=0).stop 2023-05-22T15:47:00.386 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.379+0000 7f82a4146640 1 -- 172.21.15.16:0/3204301894 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f829c15d680 msgr2=0x7f829c14c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:47:00.387 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.379+0000 7f82a4146640 1 --2- 172.21.15.16:0/3204301894 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f829c15d680 0x7f829c14c420 secure :-1 s=READY pgs=2014 cs=0 l=1 rev1=1 crypto rx=0x7f828c00e960 tx=0x7f828c00ee20 comp rx=0 tx=0).stop 2023-05-22T15:47:00.387 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.379+0000 7f82a4146640 1 -- 172.21.15.16:0/3204301894 shutdown_connections 2023-05-22T15:47:00.387 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.379+0000 7f82a4146640 1 --2- 172.21.15.16:0/3204301894 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f829c15dad0 0x7f829c14c960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:00.388 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.379+0000 7f82a4146640 1 --2- 172.21.15.16:0/3204301894 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f829c15d680 0x7f829c14c420 unknown :-1 s=CLOSED pgs=2014 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:00.388 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.379+0000 7f82a4146640 1 --2- 172.21.15.16:0/3204301894 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f82800654a0 0x7f8280067950 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:00.388 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.379+0000 7f82a4146640 1 --2- 172.21.15.16:0/3204301894 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f829c159970 0x7f829c153e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:00.389 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.379+0000 7f82a4146640 1 -- 172.21.15.16:0/3204301894 >> 172.21.15.16:0/3204301894 conn(0x7f829c069720 msgr2=0x7f829c163de0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:47:00.389 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.379+0000 7f82a4146640 1 -- 172.21.15.16:0/3204301894 shutdown_connections 2023-05-22T15:47:00.389 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.379+0000 7f82a4146640 1 -- 172.21.15.16:0/3204301894 wait complete. 2023-05-22T15:47:00.401 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-05-22T15:47:00.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.519+0000 7fa130775640 1 Processor -- start 2023-05-22T15:47:00.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.519+0000 7fa130775640 1 -- start start 2023-05-22T15:47:00.527 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.519+0000 7fa130775640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa1280b5a50 0x7fa1280d2900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:47:00.527 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.519+0000 7fa130775640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1280cccc0 con 0x7fa1280d2480 2023-05-22T15:47:00.527 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.519+0000 7fa130775640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1280ccfa0 con 0x7fa12815bca0 2023-05-22T15:47:00.527 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.519+0000 7fa130775640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1280cd660 con 0x7fa1280b5a50 2023-05-22T15:47:00.527 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.519+0000 7fa12dd10640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fa1280d2480 0x7fa1280b5340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:44826/0 (socket says 172.21.15.16:44826) 2023-05-22T15:47:00.527 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.519+0000 7fa12dd10640 1 -- 172.21.15.16:0/3459306455 learned_addr learned my addr 172.21.15.16:0/3459306455 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:47:00.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.519+0000 7fa12ed12640 1 --2- 172.21.15.16:0/3459306455 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa1280b5a50 0x7fa1280d2900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:47:00.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.519+0000 7fa12d50f640 1 -- 172.21.15.16:0/3459306455 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3747207281 0 0) 0x7fa1280cccc0 con 0x7fa1280d2480 2023-05-22T15:47:00.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.519+0000 7fa12d50f640 1 -- 172.21.15.16:0/3459306455 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa114003530 con 0x7fa1280d2480 2023-05-22T15:47:00.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.519+0000 7fa12d50f640 1 -- 172.21.15.16:0/3459306455 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3158888779 0 0) 0x7fa114003530 con 0x7fa1280d2480 2023-05-22T15:47:00.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.519+0000 7fa12d50f640 1 -- 172.21.15.16:0/3459306455 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa1280b5a50 msgr2=0x7fa1280d2900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:47:00.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.519+0000 7fa12d50f640 1 --2- 172.21.15.16:0/3459306455 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa1280b5a50 0x7fa1280d2900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:00.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.519+0000 7fa12d50f640 1 -- 172.21.15.16:0/3459306455 >> v1:172.21.15.62:6789/0 conn(0x7fa12815bca0 legacy=0x7fa1280cda80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:47:00.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.519+0000 7fa12d50f640 1 -- 172.21.15.16:0/3459306455 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1280cd280 con 0x7fa1280d2480 2023-05-22T15:47:00.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.519+0000 7fa12d50f640 1 -- 172.21.15.16:0/3459306455 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7fa11c0016a0 con 0x7fa1280d2480 2023-05-22T15:47:00.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.519+0000 7fa12d50f640 1 -- 172.21.15.16:0/3459306455 >> v1:172.21.15.16:6789/0 conn(0x7fa1280d2480 legacy=0x7fa1280b5340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:47:00.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.519+0000 7fa12d50f640 1 --2- 172.21.15.16:0/3459306455 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa1280b5a50 0x7fa1140039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:47:00.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.519+0000 7fa12d50f640 1 --2- 172.21.15.16:0/3459306455 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa114003f30 0x7fa1140047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:47:00.530 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.519+0000 7fa12d50f640 1 --2- 172.21.15.16:0/3459306455 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa114004d10 0x7fa114005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:47:00.531 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.519+0000 7fa12d50f640 1 -- 172.21.15.16:0/3459306455 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1280cd660 con 0x7fa114003f30 2023-05-22T15:47:00.531 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.519+0000 7fa12d50f640 1 -- 172.21.15.16:0/3459306455 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa11400ba70 con 0x7fa114004d10 2023-05-22T15:47:00.531 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.519+0000 7fa12d50f640 1 -- 172.21.15.16:0/3459306455 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa11400bc30 con 0x7fa1280b5a50 2023-05-22T15:47:00.531 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.519+0000 7fa12dd10640 1 --2- 172.21.15.16:0/3459306455 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa114003f30 0x7fa1140047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:47:00.532 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.519+0000 7fa12ed12640 1 --2- 172.21.15.16:0/3459306455 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa114004d10 0x7fa114005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:47:00.532 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.519+0000 7fa12dd10640 1 -- 172.21.15.16:0/3459306455 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa1280b5a50 msgr2=0x7fa1140039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:47:00.532 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.519+0000 7fa12dd10640 1 --2- 172.21.15.16:0/3459306455 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa1280b5a50 0x7fa1140039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:00.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.519+0000 7fa12dd10640 1 -- 172.21.15.16:0/3459306455 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa114004d10 msgr2=0x7fa114005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:47:00.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.519+0000 7fa12dd10640 1 --2- 172.21.15.16:0/3459306455 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa114004d10 0x7fa114005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:00.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.519+0000 7fa12dd10640 1 -- 172.21.15.16:0/3459306455 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa114003530 con 0x7fa114003f30 2023-05-22T15:47:00.537 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.523+0000 7fa12dd10640 1 --2- 172.21.15.16:0/3459306455 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa114003f30 0x7fa1140047d0 secure :-1 s=READY pgs=2016 cs=0 l=1 rev1=1 crypto rx=0x7fa11c001ec0 tx=0x7fa11c01f350 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97d6d91055309790 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:47:00.538 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.523+0000 7fa12d50f640 1 -- 172.21.15.16:0/3459306455 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa11c005ae0 con 0x7fa114003f30 2023-05-22T15:47:00.538 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.523+0000 7fa12d50f640 1 -- 172.21.15.16:0/3459306455 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa11c0ab070 con 0x7fa114003f30 2023-05-22T15:47:00.538 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.523+0000 7fa12d50f640 1 -- 172.21.15.16:0/3459306455 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa11c005e80 con 0x7fa114003f30 2023-05-22T15:47:00.538 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.523+0000 7fa130775640 1 -- 172.21.15.16:0/3459306455 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa114003f30 msgr2=0x7fa1140047d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:47:00.538 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.523+0000 7fa130775640 1 --2- 172.21.15.16:0/3459306455 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa114003f30 0x7fa1140047d0 secure :-1 s=READY pgs=2016 cs=0 l=1 rev1=1 crypto rx=0x7fa11c001ec0 tx=0x7fa11c01f350 comp rx=0 tx=0).stop 2023-05-22T15:47:00.538 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.523+0000 7fa12ed12640 1 -- 172.21.15.16:0/3459306455 reap_dead start 2023-05-22T15:47:00.539 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.523+0000 7fa130775640 1 -- 172.21.15.16:0/3459306455 shutdown_connections 2023-05-22T15:47:00.539 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.523+0000 7fa130775640 1 -- 172.21.15.16:0/3459306455 >> 172.21.15.16:0/3459306455 conn(0x7fa1280655c0 msgr2=0x7fa128070e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:47:00.539 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.523+0000 7fa130775640 1 -- 172.21.15.16:0/3459306455 shutdown_connections 2023-05-22T15:47:00.539 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.523+0000 7fa130775640 1 -- 172.21.15.16:0/3459306455 wait complete. 2023-05-22T15:47:00.539 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.527+0000 7fa130775640 1 Processor -- start 2023-05-22T15:47:00.539 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.527+0000 7fa130775640 1 -- start start 2023-05-22T15:47:00.539 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.527+0000 7fa130775640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa114003f30 0x7fa12814b9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:47:00.540 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.527+0000 7fa130775640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa1280c6b10 0x7fa1280c6f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:47:00.540 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.527+0000 7fa130775640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa12815c200 0x7fa12815c670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:47:00.540 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.527+0000 7fa130775640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1280ce190 con 0x7fa12815c200 2023-05-22T15:47:00.540 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.527+0000 7fa130775640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1280c6380 con 0x7fa1280c6b10 2023-05-22T15:47:00.540 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.527+0000 7fa130775640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1280c74c0 con 0x7fa114003f30 2023-05-22T15:47:00.541 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.527+0000 7fa12ed12640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa12815c200 0x7fa12815c670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:47:00.541 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.527+0000 7fa12ed12640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa12815c200 0x7fa12815c670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46696/0 (socket says 172.21.15.16:46696) 2023-05-22T15:47:00.541 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.527+0000 7fa12ed12640 1 -- 172.21.15.16:0/359818154 learned_addr learned my addr 172.21.15.16:0/359818154 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:47:00.541 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.527+0000 7fa12e511640 1 --2- 172.21.15.16:0/359818154 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa114003f30 0x7fa12814b9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:47:00.541 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.527+0000 7fa12dd10640 1 --2- 172.21.15.16:0/359818154 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa1280c6b10 0x7fa1280c6f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:47:00.541 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.527+0000 7fa12ed12640 1 -- 172.21.15.16:0/359818154 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa114003f30 msgr2=0x7fa12814b9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:47:00.541 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.527+0000 7fa12ed12640 1 --2- 172.21.15.16:0/359818154 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa114003f30 0x7fa12814b9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:00.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.527+0000 7fa12ed12640 1 -- 172.21.15.16:0/359818154 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa1280c6b10 msgr2=0x7fa1280c6f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:47:00.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.527+0000 7fa12ed12640 1 --2- 172.21.15.16:0/359818154 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa1280c6b10 0x7fa1280c6f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:00.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.527+0000 7fa12ed12640 1 -- 172.21.15.16:0/359818154 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1280d2900 con 0x7fa12815c200 2023-05-22T15:47:00.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.527+0000 7fa12e511640 1 --2- 172.21.15.16:0/359818154 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa114003f30 0x7fa12814b9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:47:00.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.527+0000 7fa12ed12640 1 --2- 172.21.15.16:0/359818154 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa12815c200 0x7fa12815c670 secure :-1 s=READY pgs=2017 cs=0 l=1 rev1=1 crypto rx=0x7fa120009f30 tx=0x7fa120007f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:47:00.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.527+0000 7fa11affd640 1 -- 172.21.15.16:0/359818154 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa120003ba0 con 0x7fa12815c200 2023-05-22T15:47:00.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.527+0000 7fa11affd640 1 -- 172.21.15.16:0/359818154 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa120011040 con 0x7fa12815c200 2023-05-22T15:47:00.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.527+0000 7fa11affd640 1 -- 172.21.15.16:0/359818154 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa120004020 con 0x7fa12815c200 2023-05-22T15:47:00.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.527+0000 7fa130775640 1 -- 172.21.15.16:0/359818154 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa12815c070 con 0x7fa12815c200 2023-05-22T15:47:00.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.527+0000 7fa130775640 1 -- 172.21.15.16:0/359818154 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1280d2a90 con 0x7fa12815c200 2023-05-22T15:47:00.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.527+0000 7fa130775640 1 -- 172.21.15.16:0/359818154 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa12804dbe0 con 0x7fa12815c200 2023-05-22T15:47:00.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.531+0000 7fa11affd640 1 -- 172.21.15.16:0/359818154 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 67) v1 ==== 84332+0+0 (secure 0 0 0) 0x7fa120004180 con 0x7fa12815c200 2023-05-22T15:47:00.544 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.531+0000 7fa11affd640 1 --2- 172.21.15.16:0/359818154 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fa10c0654a0 0x7fa10c067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:47:00.544 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.531+0000 7fa11affd640 1 -- 172.21.15.16:0/359818154 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(157..157 src has 1..157) v4 ==== 5558+0+0 (secure 0 0 0) 0x7fa120082570 con 0x7fa12815c200 2023-05-22T15:47:00.544 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.531+0000 7fa12e511640 1 --2- 172.21.15.16:0/359818154 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fa10c0654a0 0x7fa10c067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:47:00.544 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.531+0000 7fa12e511640 1 --2- 172.21.15.16:0/359818154 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fa10c0654a0 0x7fa10c067950 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fa124007fd0 tx=0x7fa124009040 comp rx=0 tx=0).ready entity=mgr.9396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:47:00.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.535+0000 7fa11affd640 1 -- 172.21.15.16:0/359818154 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fa1200501c0 con 0x7fa12815c200 2023-05-22T15:47:00.692 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.687+0000 7fa130775640 1 -- 172.21.15.16:0/359818154 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fa1280b4a70 con 0x7fa12815c200 2023-05-22T15:47:00.692 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.687+0000 7fa11affd640 1 -- 172.21.15.16:0/359818154 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v157) v1 ==== 92+0+10 (secure 0 0 0) 0x7fa1200537e0 con 0x7fa12815c200 2023-05-22T15:47:00.693 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 8 2023-05-22T15:47:00.694 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.687+0000 7fa130775640 1 -- 172.21.15.16:0/359818154 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fa10c0654a0 msgr2=0x7fa10c067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:47:00.694 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.687+0000 7fa130775640 1 --2- 172.21.15.16:0/359818154 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fa10c0654a0 0x7fa10c067950 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fa124007fd0 tx=0x7fa124009040 comp rx=0 tx=0).stop 2023-05-22T15:47:00.695 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.687+0000 7fa130775640 1 -- 172.21.15.16:0/359818154 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa12815c200 msgr2=0x7fa12815c670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:47:00.695 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.687+0000 7fa130775640 1 --2- 172.21.15.16:0/359818154 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa12815c200 0x7fa12815c670 secure :-1 s=READY pgs=2017 cs=0 l=1 rev1=1 crypto rx=0x7fa120009f30 tx=0x7fa120007f50 comp rx=0 tx=0).stop 2023-05-22T15:47:00.695 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.687+0000 7fa130775640 1 -- 172.21.15.16:0/359818154 shutdown_connections 2023-05-22T15:47:00.695 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.687+0000 7fa130775640 1 --2- 172.21.15.16:0/359818154 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa12815c200 0x7fa12815c670 unknown :-1 s=CLOSED pgs=2017 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:00.695 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.687+0000 7fa130775640 1 --2- 172.21.15.16:0/359818154 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa1280c6b10 0x7fa1280c6f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:00.695 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.687+0000 7fa130775640 1 --2- 172.21.15.16:0/359818154 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fa10c0654a0 0x7fa10c067950 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:00.695 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.687+0000 7fa130775640 1 --2- 172.21.15.16:0/359818154 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa114003f30 0x7fa12814b9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:00.696 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.687+0000 7fa130775640 1 -- 172.21.15.16:0/359818154 >> 172.21.15.16:0/359818154 conn(0x7fa1280655c0 msgr2=0x7fa12815af80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:47:00.696 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.687+0000 7fa130775640 1 -- 172.21.15.16:0/359818154 shutdown_connections 2023-05-22T15:47:00.696 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.687+0000 7fa130775640 1 -- 172.21.15.16:0/359818154 wait complete. 2023-05-22T15:47:00.712 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-05-22T15:47:00.835 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.831+0000 7fc31cffb640 1 Processor -- start 2023-05-22T15:47:00.835 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.831+0000 7fc31cffb640 1 -- start start 2023-05-22T15:47:00.836 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.831+0000 7fc31cffb640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fc3180cbf80 0x7fc31806b210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:47:00.836 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.831+0000 7fc31cffb640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc31806c5d0 0x7fc3180cf8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:47:00.836 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.831+0000 7fc31cffb640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc31806db40 con 0x7fc3180b22e0 2023-05-22T15:47:00.837 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.831+0000 7fc31cffb640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3180cf3f0 con 0x7fc3180cbf80 2023-05-22T15:47:00.837 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.831+0000 7fc31cffb640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc31815c080 con 0x7fc31806c5d0 2023-05-22T15:47:00.838 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.831+0000 7fc316d9d640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc31806c5d0 0x7fc3180cf8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:47:00.838 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.831+0000 7fc31659c640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fc3180cbf80 0x7fc31806b210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:47:00.839 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.831+0000 7fc30ffff640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7fc3180b22e0 0x7fc31806bec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.16:53008/0 (socket says 172.21.15.16:53008) 2023-05-22T15:47:00.839 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.831+0000 7fc30ffff640 1 -- 172.21.15.16:0/2496305924 learned_addr learned my addr 172.21.15.16:0/2496305924 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:47:00.839 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.831+0000 7fc31659c640 1 -- 172.21.15.16:0/2496305924 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc31806c5d0 msgr2=0x7fc3180cf8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:47:00.840 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.831+0000 7fc31659c640 1 --2- 172.21.15.16:0/2496305924 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc31806c5d0 0x7fc3180cf8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:00.840 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.831+0000 7fc31659c640 1 -- 172.21.15.16:0/2496305924 >> v1:172.21.15.62:6789/0 conn(0x7fc3180b22e0 legacy=0x7fc31806bec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:47:00.840 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.831+0000 7fc31659c640 1 -- 172.21.15.16:0/2496305924 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc31806ddc0 con 0x7fc3180cbf80 2023-05-22T15:47:00.841 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.835+0000 7fc31659c640 1 --2- 172.21.15.16:0/2496305924 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc3180cbf80 0x7fc31806b210 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7fc3080097c0 tx=0x7fc30801c3c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fcd8cd3b07b4598c server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:47:00.841 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.835+0000 7fc315d9b640 1 -- 172.21.15.16:0/2496305924 <== mon.1 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3080ac620 con 0x7fc3180cbf80 2023-05-22T15:47:00.841 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.835+0000 7fc315d9b640 1 -- 172.21.15.16:0/2496305924 <== mon.1 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3080ac780 con 0x7fc3180cbf80 2023-05-22T15:47:00.841 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.835+0000 7fc315d9b640 1 -- 172.21.15.16:0/2496305924 <== mon.1 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3080acb10 con 0x7fc3180cbf80 2023-05-22T15:47:00.842 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.835+0000 7fc31cffb640 1 -- 172.21.15.16:0/2496305924 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc3180cbf80 msgr2=0x7fc31806b210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:47:00.842 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.835+0000 7fc31cffb640 1 --2- 172.21.15.16:0/2496305924 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc3180cbf80 0x7fc31806b210 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7fc3080097c0 tx=0x7fc30801c3c0 comp rx=0 tx=0).stop 2023-05-22T15:47:00.842 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.835+0000 7fc31cffb640 1 -- 172.21.15.16:0/2496305924 shutdown_connections 2023-05-22T15:47:00.842 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.835+0000 7fc31cffb640 1 --2- 172.21.15.16:0/2496305924 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc31806c5d0 0x7fc3180cf8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:00.842 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.835+0000 7fc31cffb640 1 --2- 172.21.15.16:0/2496305924 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc3180cbf80 0x7fc31806b210 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:00.842 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.835+0000 7fc31cffb640 1 -- 172.21.15.16:0/2496305924 >> 172.21.15.16:0/2496305924 conn(0x7fc31806f560 msgr2=0x7fc31806f960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:47:00.843 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.835+0000 7fc31cffb640 1 -- 172.21.15.16:0/2496305924 shutdown_connections 2023-05-22T15:47:00.843 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.835+0000 7fc31cffb640 1 -- 172.21.15.16:0/2496305924 wait complete. 2023-05-22T15:47:00.843 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.839+0000 7fc31cffb640 1 Processor -- start 2023-05-22T15:47:00.843 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.839+0000 7fc31cffb640 1 -- start start 2023-05-22T15:47:00.844 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.839+0000 7fc31cffb640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc31806c5d0 0x7fc318157390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:47:00.844 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.839+0000 7fc31cffb640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc3180b22e0 0x7fc3180c9c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:47:00.844 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.839+0000 7fc31cffb640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc3180cbf80 0x7fc3180b0f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:47:00.844 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.839+0000 7fc31cffb640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3181571b0 con 0x7fc31806c5d0 2023-05-22T15:47:00.844 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.839+0000 7fc31cffb640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3180ca1d0 con 0x7fc3180b22e0 2023-05-22T15:47:00.844 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.839+0000 7fc31cffb640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc31815fa00 con 0x7fc3180cbf80 2023-05-22T15:47:00.846 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.839+0000 7fc31659c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc31806c5d0 0x7fc318157390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:47:00.846 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.839+0000 7fc31659c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc31806c5d0 0x7fc318157390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46700/0 (socket says 172.21.15.16:46700) 2023-05-22T15:47:00.846 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.839+0000 7fc31659c640 1 -- 172.21.15.16:0/3200136079 learned_addr learned my addr 172.21.15.16:0/3200136079 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:47:00.846 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.839+0000 7fc30ffff640 1 --2- 172.21.15.16:0/3200136079 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc3180b22e0 0x7fc3180c9c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:47:00.846 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.839+0000 7fc316d9d640 1 --2- 172.21.15.16:0/3200136079 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc3180cbf80 0x7fc3180b0f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:47:00.846 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.839+0000 7fc31659c640 1 -- 172.21.15.16:0/3200136079 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc3180cbf80 msgr2=0x7fc3180b0f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:47:00.847 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.839+0000 7fc31659c640 1 --2- 172.21.15.16:0/3200136079 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc3180cbf80 0x7fc3180b0f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:00.847 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.839+0000 7fc31659c640 1 -- 172.21.15.16:0/3200136079 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc3180b22e0 msgr2=0x7fc3180c9c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:47:00.847 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.839+0000 7fc31659c640 1 --2- 172.21.15.16:0/3200136079 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc3180b22e0 0x7fc3180c9c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:00.847 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.839+0000 7fc31659c640 1 -- 172.21.15.16:0/3200136079 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc308009420 con 0x7fc31806c5d0 2023-05-22T15:47:00.847 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.839+0000 7fc31659c640 1 --2- 172.21.15.16:0/3200136079 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc31806c5d0 0x7fc318157390 secure :-1 s=READY pgs=2018 cs=0 l=1 rev1=1 crypto rx=0x7fc308019ff0 tx=0x7fc308019160 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:47:00.847 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.839+0000 7fc30effd640 1 -- 172.21.15.16:0/3200136079 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3080ac620 con 0x7fc31806c5d0 2023-05-22T15:47:00.848 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.839+0000 7fc30effd640 1 -- 172.21.15.16:0/3200136079 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3080b5400 con 0x7fc31806c5d0 2023-05-22T15:47:00.848 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.839+0000 7fc31cffb640 1 -- 172.21.15.16:0/3200136079 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3180cc350 con 0x7fc31806c5d0 2023-05-22T15:47:00.848 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.839+0000 7fc30effd640 1 -- 172.21.15.16:0/3200136079 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3080b57b0 con 0x7fc31806c5d0 2023-05-22T15:47:00.849 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.839+0000 7fc31cffb640 1 -- 172.21.15.16:0/3200136079 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc31806b750 con 0x7fc31806c5d0 2023-05-22T15:47:00.849 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.839+0000 7fc31cffb640 1 -- 172.21.15.16:0/3200136079 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3180d0620 con 0x7fc31806c5d0 2023-05-22T15:47:00.849 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.839+0000 7fc30effd640 1 -- 172.21.15.16:0/3200136079 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 67) v1 ==== 84332+0+0 (secure 0 0 0) 0x7fc3080b3040 con 0x7fc31806c5d0 2023-05-22T15:47:00.850 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.843+0000 7fc30effd640 1 --2- 172.21.15.16:0/3200136079 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fc2f80654a0 0x7fc2f8067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:47:00.850 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.843+0000 7fc30effd640 1 -- 172.21.15.16:0/3200136079 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(157..157 src has 1..157) v4 ==== 5558+0+0 (secure 0 0 0) 0x7fc30811c840 con 0x7fc31806c5d0 2023-05-22T15:47:00.850 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.843+0000 7fc30ffff640 1 --2- 172.21.15.16:0/3200136079 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fc2f80654a0 0x7fc2f8067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:47:00.851 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.843+0000 7fc30ffff640 1 --2- 172.21.15.16:0/3200136079 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fc2f80654a0 0x7fc2f8067950 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fc3180c8f90 tx=0x7fc300009040 comp rx=0 tx=0).ready entity=mgr.9396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:47:00.851 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.847+0000 7fc30effd640 1 -- 172.21.15.16:0/3200136079 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fc3080ea5d0 con 0x7fc31806c5d0 2023-05-22T15:47:01.006 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.999+0000 7fc31cffb640 1 -- 172.21.15.16:0/3200136079 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fc31804dbe0 con 0x7fc31806c5d0 2023-05-22T15:47:01.007 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:00.999+0000 7fc30effd640 1 -- 172.21.15.16:0/3200136079 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v157) v1 ==== 84+0+126 (secure 0 0 0) 0x7fc3080b1090 con 0x7fc31806c5d0 2023-05-22T15:47:01.007 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:47:01.008 INFO:teuthology.orchestra.run.smithi016.stdout:[ 2023-05-22T15:47:01.008 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T15:47:01.008 INFO:teuthology.orchestra.run.smithi016.stdout: "poolnum": 1, 2023-05-22T15:47:01.008 INFO:teuthology.orchestra.run.smithi016.stdout: "poolname": ".mgr" 2023-05-22T15:47:01.008 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:47:01.009 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T15:47:01.009 INFO:teuthology.orchestra.run.smithi016.stdout: "poolnum": 2, 2023-05-22T15:47:01.009 INFO:teuthology.orchestra.run.smithi016.stdout: "poolname": "rbd" 2023-05-22T15:47:01.009 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T15:47:01.009 INFO:teuthology.orchestra.run.smithi016.stdout:] 2023-05-22T15:47:01.010 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:01.003+0000 7fc31cffb640 1 -- 172.21.15.16:0/3200136079 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fc2f80654a0 msgr2=0x7fc2f8067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:47:01.010 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:01.003+0000 7fc31cffb640 1 --2- 172.21.15.16:0/3200136079 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fc2f80654a0 0x7fc2f8067950 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fc3180c8f90 tx=0x7fc300009040 comp rx=0 tx=0).stop 2023-05-22T15:47:01.011 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:01.003+0000 7fc31cffb640 1 -- 172.21.15.16:0/3200136079 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc31806c5d0 msgr2=0x7fc318157390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:47:01.011 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:01.003+0000 7fc31cffb640 1 --2- 172.21.15.16:0/3200136079 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc31806c5d0 0x7fc318157390 secure :-1 s=READY pgs=2018 cs=0 l=1 rev1=1 crypto rx=0x7fc308019ff0 tx=0x7fc308019160 comp rx=0 tx=0).stop 2023-05-22T15:47:01.011 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:01.003+0000 7fc31cffb640 1 -- 172.21.15.16:0/3200136079 shutdown_connections 2023-05-22T15:47:01.011 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:01.003+0000 7fc31cffb640 1 --2- 172.21.15.16:0/3200136079 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fc2f80654a0 0x7fc2f8067950 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:01.011 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:01.003+0000 7fc31cffb640 1 --2- 172.21.15.16:0/3200136079 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc3180cbf80 0x7fc3180b0f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:01.011 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:01.003+0000 7fc31cffb640 1 --2- 172.21.15.16:0/3200136079 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc3180b22e0 0x7fc3180c9c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:01.011 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:01.003+0000 7fc31cffb640 1 --2- 172.21.15.16:0/3200136079 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc31806c5d0 0x7fc318157390 unknown :-1 s=CLOSED pgs=2018 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:01.011 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:01.003+0000 7fc31cffb640 1 -- 172.21.15.16:0/3200136079 >> 172.21.15.16:0/3200136079 conn(0x7fc31806f560 msgr2=0x7fc31806f930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:47:01.012 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:01.003+0000 7fc31cffb640 1 -- 172.21.15.16:0/3200136079 shutdown_connections 2023-05-22T15:47:01.012 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:01.003+0000 7fc31cffb640 1 -- 172.21.15.16:0/3200136079 wait complete. 2023-05-22T15:47:01.027 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-05-22T15:47:01.028 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-05-22T15:47:01.144 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:01.139+0000 7f051c6d8640 1 Processor -- start 2023-05-22T15:47:01.144 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:01.139+0000 7f051c6d8640 1 -- start start 2023-05-22T15:47:01.145 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:01.139+0000 7f051c6d8640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f0514159a30 0x7f051406e380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:47:01.145 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:01.139+0000 7f051c6d8640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05140cde70 con 0x7f051415d610 2023-05-22T15:47:01.145 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:01.139+0000 7f051c6d8640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f051415db20 con 0x7f05140cf570 2023-05-22T15:47:01.145 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:01.139+0000 7f051c6d8640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05140cc520 con 0x7f0514159a30 2023-05-22T15:47:01.145 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:01.139+0000 7f051a474640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f0514159a30 0x7f051406e380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:47:01.145 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:01.139+0000 7f0519c73640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f051415d610 0x7f05140cbe10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:44828/0 (socket says 172.21.15.16:44828) 2023-05-22T15:47:01.146 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:01.139+0000 7f0519c73640 1 -- 172.21.15.16:0/1360762473 learned_addr learned my addr 172.21.15.16:0/1360762473 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:47:01.146 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:01.139+0000 7f0519472640 1 -- 172.21.15.16:0/1360762473 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1002932648 0 0) 0x7f05140cde70 con 0x7f051415d610 2023-05-22T15:47:01.146 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:01.139+0000 7f0519472640 1 -- 172.21.15.16:0/1360762473 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0500003530 con 0x7f051415d610 2023-05-22T15:47:01.147 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:01.139+0000 7f051a474640 1 -- 172.21.15.16:0/1360762473 >> v1:172.21.15.62:6789/0 conn(0x7f05140cf570 legacy=0x7f05140cf990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:47:01.147 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:01.139+0000 7f051a474640 1 -- 172.21.15.16:0/1360762473 >> v1:172.21.15.16:6789/0 conn(0x7f051415d610 legacy=0x7f05140cbe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:47:01.147 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:01.139+0000 7f051a474640 1 -- 172.21.15.16:0/1360762473 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f051406e8c0 con 0x7f0514159a30 2023-05-22T15:47:01.147 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:01.143+0000 7f051a474640 1 --2- 172.21.15.16:0/1360762473 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0514159a30 0x7f051406e380 secure :-1 s=READY pgs=2020 cs=0 l=1 rev1=1 crypto rx=0x7f05100097f0 tx=0x7f051001c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d83b246944bdf149 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:47:01.148 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:01.143+0000 7f0519472640 1 -- 172.21.15.16:0/1360762473 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05100ab650 con 0x7f0514159a30 2023-05-22T15:47:01.148 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:01.143+0000 7f0519472640 1 -- 172.21.15.16:0/1360762473 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05100ab7b0 con 0x7f0514159a30 2023-05-22T15:47:01.148 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:01.143+0000 7f0519472640 1 -- 172.21.15.16:0/1360762473 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05100abb20 con 0x7f0514159a30 2023-05-22T15:47:01.149 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:01.143+0000 7f051c6d8640 1 -- 172.21.15.16:0/1360762473 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0514159a30 msgr2=0x7f051406e380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:47:01.149 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:01.143+0000 7f051c6d8640 1 --2- 172.21.15.16:0/1360762473 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0514159a30 0x7f051406e380 secure :-1 s=READY pgs=2020 cs=0 l=1 rev1=1 crypto rx=0x7f05100097f0 tx=0x7f051001c450 comp rx=0 tx=0).stop 2023-05-22T15:47:01.149 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:01.143+0000 7f051c6d8640 1 -- 172.21.15.16:0/1360762473 shutdown_connections 2023-05-22T15:47:01.149 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:01.143+0000 7f051c6d8640 1 --2- 172.21.15.16:0/1360762473 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0514159a30 0x7f051406e380 unknown :-1 s=CLOSED pgs=2020 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:01.149 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:01.143+0000 7f051c6d8640 1 -- 172.21.15.16:0/1360762473 >> 172.21.15.16:0/1360762473 conn(0x7f05140700b0 msgr2=0x7f051406d9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:47:01.149 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:01.143+0000 7f051c6d8640 1 -- 172.21.15.16:0/1360762473 shutdown_connections 2023-05-22T15:47:01.150 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:01.143+0000 7f051c6d8640 1 -- 172.21.15.16:0/1360762473 wait complete. 2023-05-22T15:47:01.151 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:01.143+0000 7f051c6d8640 1 Processor -- start 2023-05-22T15:47:01.151 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:01.143+0000 7f051c6d8640 1 -- start start 2023-05-22T15:47:01.151 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:01.143+0000 7f051c6d8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f05140cf570 0x7f051414dec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:47:01.151 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:01.143+0000 7f051c6d8640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0514159a30 0x7f0514148560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:47:01.152 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:01.143+0000 7f051c6d8640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f051415d610 0x7f0514149ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:47:01.152 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:01.143+0000 7f051c6d8640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0514157dd0 con 0x7f05140cf570 2023-05-22T15:47:01.152 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:01.143+0000 7f051c6d8640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f051406c3c0 con 0x7f0514159a30 2023-05-22T15:47:01.152 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:01.143+0000 7f051c6d8640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0514158260 con 0x7f051415d610 2023-05-22T15:47:01.153 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:01.143+0000 7f051a474640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f05140cf570 0x7f051414dec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:47:01.153 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:01.143+0000 7f051a474640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f05140cf570 0x7f051414dec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46722/0 (socket says 172.21.15.16:46722) 2023-05-22T15:47:01.153 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:01.143+0000 7f051a474640 1 -- 172.21.15.16:0/119443452 learned_addr learned my addr 172.21.15.16:0/119443452 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:47:01.153 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:01.143+0000 7f051a474640 1 -- 172.21.15.16:0/119443452 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f051415d610 msgr2=0x7f0514149ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:47:01.153 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:01.143+0000 7f051a474640 1 --2- 172.21.15.16:0/119443452 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f051415d610 0x7f0514149ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:01.153 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:01.143+0000 7f051a474640 1 -- 172.21.15.16:0/119443452 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0514159a30 msgr2=0x7f0514148560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:47:01.153 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:01.143+0000 7f051a474640 1 --2- 172.21.15.16:0/119443452 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0514159a30 0x7f0514148560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:01.154 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:01.143+0000 7f051a474640 1 -- 172.21.15.16:0/119443452 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0510009420 con 0x7f05140cf570 2023-05-22T15:47:01.154 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:01.143+0000 7f051a474640 1 --2- 172.21.15.16:0/119443452 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f05140cf570 0x7f051414dec0 secure :-1 s=READY pgs=2021 cs=0 l=1 rev1=1 crypto rx=0x7f05100097f0 tx=0x7f051001ac90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:47:01.154 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:01.147+0000 7f050affd640 1 -- 172.21.15.16:0/119443452 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0510004220 con 0x7f05140cf570 2023-05-22T15:47:01.154 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:01.147+0000 7f050affd640 1 -- 172.21.15.16:0/119443452 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05100b9070 con 0x7f05140cf570 2023-05-22T15:47:01.154 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:01.147+0000 7f051c6d8640 1 -- 172.21.15.16:0/119443452 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0514159e00 con 0x7f05140cf570 2023-05-22T15:47:01.154 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:01.147+0000 7f050affd640 1 -- 172.21.15.16:0/119443452 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0510003620 con 0x7f05140cf570 2023-05-22T15:47:01.154 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:01.147+0000 7f051c6d8640 1 -- 172.21.15.16:0/119443452 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f051415e3a0 con 0x7f05140cf570 2023-05-22T15:47:01.155 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:01.147+0000 7f051c6d8640 1 -- 172.21.15.16:0/119443452 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05140ceb10 con 0x7f05140cf570 2023-05-22T15:47:01.155 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:01.147+0000 7f050affd640 1 -- 172.21.15.16:0/119443452 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 67) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f0510003780 con 0x7f05140cf570 2023-05-22T15:47:01.156 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:01.147+0000 7f050affd640 1 --2- 172.21.15.16:0/119443452 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f04f80654a0 0x7f04f8067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:47:01.156 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:01.147+0000 7f0519c73640 1 --2- 172.21.15.16:0/119443452 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f04f80654a0 0x7f04f8067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:47:01.156 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:01.147+0000 7f050affd640 1 -- 172.21.15.16:0/119443452 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(157..157 src has 1..157) v4 ==== 5558+0+0 (secure 0 0 0) 0x7f05100bb070 con 0x7f05140cf570 2023-05-22T15:47:01.156 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:01.147+0000 7f0519c73640 1 --2- 172.21.15.16:0/119443452 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f04f80654a0 0x7f04f8067950 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f0504007fd0 tx=0x7f0504007940 comp rx=0 tx=0).ready entity=mgr.9396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:47:01.157 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:01.151+0000 7f050affd640 1 -- 172.21.15.16:0/119443452 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f05100eb1c0 con 0x7f05140cf570 2023-05-22T15:47:01.305 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:01.299+0000 7f051c6d8640 1 -- 172.21.15.16:0/119443452 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f0514066910 con 0x7f05140cf570 2023-05-22T15:47:01.615 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:01.607+0000 7f050affd640 1 -- 172.21.15.16:0/119443452 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v158) v1 ==== 138+0+0 (secure 0 0 0) 0x7f05100ee7e0 con 0x7f05140cf570 2023-05-22T15:47:01.615 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'cephfs_metadata' created 2023-05-22T15:47:01.618 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:01.611+0000 7f051c6d8640 1 -- 172.21.15.16:0/119443452 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f04f80654a0 msgr2=0x7f04f8067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:47:01.618 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:01.611+0000 7f051c6d8640 1 --2- 172.21.15.16:0/119443452 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f04f80654a0 0x7f04f8067950 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f0504007fd0 tx=0x7f0504007940 comp rx=0 tx=0).stop 2023-05-22T15:47:01.619 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:01.611+0000 7f051c6d8640 1 -- 172.21.15.16:0/119443452 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f05140cf570 msgr2=0x7f051414dec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:47:01.619 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:01.611+0000 7f051c6d8640 1 --2- 172.21.15.16:0/119443452 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f05140cf570 0x7f051414dec0 secure :-1 s=READY pgs=2021 cs=0 l=1 rev1=1 crypto rx=0x7f05100097f0 tx=0x7f051001ac90 comp rx=0 tx=0).stop 2023-05-22T15:47:01.619 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:01.611+0000 7f051c6d8640 1 -- 172.21.15.16:0/119443452 shutdown_connections 2023-05-22T15:47:01.619 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:01.611+0000 7f051c6d8640 1 --2- 172.21.15.16:0/119443452 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f04f80654a0 0x7f04f8067950 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:01.619 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:01.611+0000 7f051c6d8640 1 --2- 172.21.15.16:0/119443452 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f051415d610 0x7f0514149ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:01.619 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:01.611+0000 7f051c6d8640 1 --2- 172.21.15.16:0/119443452 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0514159a30 0x7f0514148560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:01.620 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:01.611+0000 7f051c6d8640 1 --2- 172.21.15.16:0/119443452 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f05140cf570 0x7f051414dec0 unknown :-1 s=CLOSED pgs=2021 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:01.620 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:01.611+0000 7f051c6d8640 1 -- 172.21.15.16:0/119443452 >> 172.21.15.16:0/119443452 conn(0x7f05140700b0 msgr2=0x7f05140cf990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:47:01.620 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:01.611+0000 7f051c6d8640 1 -- 172.21.15.16:0/119443452 shutdown_connections 2023-05-22T15:47:01.620 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:01.611+0000 7f051c6d8640 1 -- 172.21.15.16:0/119443452 wait complete. 2023-05-22T15:47:01.642 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-05-22T15:47:01.768 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:01.763+0000 7f5a875c9640 1 Processor -- start 2023-05-22T15:47:01.768 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:01.763+0000 7f5a875c9640 1 -- start start 2023-05-22T15:47:01.769 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:01.763+0000 7f5a875c9640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f5a800710e0 0x7f5a800714b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:47:01.770 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:01.763+0000 7f5a875c9640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f5a800ce3b0 0x7f5a800ce850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:47:01.770 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:01.763+0000 7f5a875c9640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a80160040 con 0x7f5a801604d0 2023-05-22T15:47:01.770 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:01.763+0000 7f5a875c9640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a8006a230 con 0x7f5a800ce3b0 2023-05-22T15:47:01.770 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:01.763+0000 7f5a875c9640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a800719f0 con 0x7f5a800710e0 2023-05-22T15:47:01.770 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:01.763+0000 7f5a84b64640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f5a801604d0 0x7f5a80069720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:44836/0 (socket says 172.21.15.16:44836) 2023-05-22T15:47:01.770 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:01.763+0000 7f5a84b64640 1 -- 172.21.15.16:0/1925940952 learned_addr learned my addr 172.21.15.16:0/1925940952 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:47:01.770 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:01.763+0000 7f5a85b66640 1 --2- 172.21.15.16:0/1925940952 >> v2:172.21.15.16:3300/0 conn(0x7f5a800ce3b0 0x7f5a800ce850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:47:01.771 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:01.763+0000 7f5a85365640 1 --2- 172.21.15.16:0/1925940952 >> v2:172.21.15.62:3300/0 conn(0x7f5a800710e0 0x7f5a800714b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:47:01.771 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:01.763+0000 7f5a73fff640 1 -- 172.21.15.16:0/1925940952 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3166948468 0 0) 0x7f5a80160040 con 0x7f5a801604d0 2023-05-22T15:47:01.771 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:01.763+0000 7f5a73fff640 1 -- 172.21.15.16:0/1925940952 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a6c003530 con 0x7f5a801604d0 2023-05-22T15:47:01.771 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:01.763+0000 7f5a85b66640 1 -- 172.21.15.16:0/1925940952 >> v2:172.21.15.62:3300/0 conn(0x7f5a800710e0 msgr2=0x7f5a800714b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:47:01.772 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:01.763+0000 7f5a85b66640 1 --2- 172.21.15.16:0/1925940952 >> v2:172.21.15.62:3300/0 conn(0x7f5a800710e0 0x7f5a800714b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:01.772 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:01.763+0000 7f5a85b66640 1 -- 172.21.15.16:0/1925940952 >> v1:172.21.15.16:6789/0 conn(0x7f5a801604d0 legacy=0x7f5a80069720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:47:01.772 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:01.763+0000 7f5a85b66640 1 -- 172.21.15.16:0/1925940952 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a8015c160 con 0x7f5a800ce3b0 2023-05-22T15:47:01.774 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:01.767+0000 7f5a85b66640 1 --2- 172.21.15.16:0/1925940952 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5a800ce3b0 0x7f5a800ce850 secure :-1 s=READY pgs=2023 cs=0 l=1 rev1=1 crypto rx=0x7f5a8006a7e0 tx=0x7f5a7801c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=409c7212bb9b855 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:47:01.774 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:01.767+0000 7f5a73fff640 1 -- 172.21.15.16:0/1925940952 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a780a59d0 con 0x7f5a800ce3b0 2023-05-22T15:47:01.774 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:01.767+0000 7f5a73fff640 1 -- 172.21.15.16:0/1925940952 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a780a5b30 con 0x7f5a800ce3b0 2023-05-22T15:47:01.774 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:01.767+0000 7f5a73fff640 1 -- 172.21.15.16:0/1925940952 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a780a5ea0 con 0x7f5a800ce3b0 2023-05-22T15:47:01.774 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:01.767+0000 7f5a875c9640 1 -- 172.21.15.16:0/1925940952 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5a800ce3b0 msgr2=0x7f5a800ce850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:47:01.774 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:01.767+0000 7f5a875c9640 1 --2- 172.21.15.16:0/1925940952 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5a800ce3b0 0x7f5a800ce850 secure :-1 s=READY pgs=2023 cs=0 l=1 rev1=1 crypto rx=0x7f5a8006a7e0 tx=0x7f5a7801c910 comp rx=0 tx=0).stop 2023-05-22T15:47:01.775 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:01.767+0000 7f5a875c9640 1 -- 172.21.15.16:0/1925940952 shutdown_connections 2023-05-22T15:47:01.775 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:01.767+0000 7f5a875c9640 1 --2- 172.21.15.16:0/1925940952 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5a800ce3b0 0x7f5a800ce850 unknown :-1 s=CLOSED pgs=2023 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:01.775 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:01.767+0000 7f5a875c9640 1 --2- 172.21.15.16:0/1925940952 >> v2:172.21.15.62:3300/0 conn(0x7f5a800710e0 0x7f5a800714b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:01.775 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:01.767+0000 7f5a875c9640 1 -- 172.21.15.16:0/1925940952 >> 172.21.15.16:0/1925940952 conn(0x7f5a800b18e0 msgr2=0x7f5a800cf120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:47:01.775 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:01.767+0000 7f5a875c9640 1 -- 172.21.15.16:0/1925940952 shutdown_connections 2023-05-22T15:47:01.775 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:01.767+0000 7f5a875c9640 1 -- 172.21.15.16:0/1925940952 wait complete. 2023-05-22T15:47:01.777 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:01.767+0000 7f5a875c9640 1 Processor -- start 2023-05-22T15:47:01.777 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:01.767+0000 7f5a875c9640 1 -- start start 2023-05-22T15:47:01.778 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:01.771+0000 7f5a875c9640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5a800710e0 0x7f5a8015a230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:47:01.778 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:01.771+0000 7f5a875c9640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5a800ce3b0 0x7f5a800ade10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:47:01.778 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:01.771+0000 7f5a875c9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5a801604d0 0x7f5a801547c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:47:01.778 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:01.771+0000 7f5a875c9640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a800c5510 con 0x7f5a801604d0 2023-05-22T15:47:01.778 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:01.771+0000 7f5a875c9640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a80155720 con 0x7f5a800ce3b0 2023-05-22T15:47:01.778 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:01.771+0000 7f5a875c9640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a800c4ec0 con 0x7f5a800710e0 2023-05-22T15:47:01.778 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:01.771+0000 7f5a85b66640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5a801604d0 0x7f5a801547c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:47:01.778 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:01.771+0000 7f5a85b66640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5a801604d0 0x7f5a801547c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46744/0 (socket says 172.21.15.16:46744) 2023-05-22T15:47:01.779 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:01.771+0000 7f5a85b66640 1 -- 172.21.15.16:0/2213920544 learned_addr learned my addr 172.21.15.16:0/2213920544 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:47:01.779 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:01.771+0000 7f5a84b64640 1 --2- 172.21.15.16:0/2213920544 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5a800ce3b0 0x7f5a800ade10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:47:01.779 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:01.771+0000 7f5a85365640 1 --2- 172.21.15.16:0/2213920544 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5a800710e0 0x7f5a8015a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:47:01.779 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:01.771+0000 7f5a85b66640 1 -- 172.21.15.16:0/2213920544 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5a800710e0 msgr2=0x7f5a8015a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:47:01.779 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:01.771+0000 7f5a85b66640 1 --2- 172.21.15.16:0/2213920544 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5a800710e0 0x7f5a8015a230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:01.779 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:01.771+0000 7f5a85b66640 1 -- 172.21.15.16:0/2213920544 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5a800ce3b0 msgr2=0x7f5a800ade10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:47:01.779 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:01.771+0000 7f5a85b66640 1 --2- 172.21.15.16:0/2213920544 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5a800ce3b0 0x7f5a800ade10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:01.780 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:01.771+0000 7f5a85b66640 1 -- 172.21.15.16:0/2213920544 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a78009a70 con 0x7f5a801604d0 2023-05-22T15:47:01.780 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:01.771+0000 7f5a84b64640 1 --2- 172.21.15.16:0/2213920544 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5a800ce3b0 0x7f5a800ade10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:47:01.780 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:01.771+0000 7f5a85b66640 1 --2- 172.21.15.16:0/2213920544 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5a801604d0 0x7f5a801547c0 secure :-1 s=READY pgs=2024 cs=0 l=1 rev1=1 crypto rx=0x7f5a7801a540 tx=0x7f5a780196b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:47:01.780 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:01.771+0000 7f5a71ffb640 1 -- 172.21.15.16:0/2213920544 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a78004240 con 0x7f5a801604d0 2023-05-22T15:47:01.780 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:01.771+0000 7f5a71ffb640 1 -- 172.21.15.16:0/2213920544 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a780043a0 con 0x7f5a801604d0 2023-05-22T15:47:01.780 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:01.771+0000 7f5a875c9640 1 -- 172.21.15.16:0/2213920544 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a800ced90 con 0x7f5a801604d0 2023-05-22T15:47:01.780 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:01.771+0000 7f5a71ffb640 1 -- 172.21.15.16:0/2213920544 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a78004790 con 0x7f5a801604d0 2023-05-22T15:47:01.781 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:01.771+0000 7f5a875c9640 1 -- 172.21.15.16:0/2213920544 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a8015fcd0 con 0x7f5a801604d0 2023-05-22T15:47:01.781 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:01.771+0000 7f5a875c9640 1 -- 172.21.15.16:0/2213920544 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a8006e310 con 0x7f5a801604d0 2023-05-22T15:47:01.781 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:01.771+0000 7f5a71ffb640 1 -- 172.21.15.16:0/2213920544 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 67) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f5a780b4030 con 0x7f5a801604d0 2023-05-22T15:47:01.782 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:01.771+0000 7f5a71ffb640 1 --2- 172.21.15.16:0/2213920544 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f5a640654a0 0x7f5a64067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:47:01.782 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:01.771+0000 7f5a85365640 1 --2- 172.21.15.16:0/2213920544 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f5a640654a0 0x7f5a64067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:47:01.782 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:01.771+0000 7f5a71ffb640 1 -- 172.21.15.16:0/2213920544 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(158..158 src has 1..158) v4 ==== 5928+0+0 (secure 0 0 0) 0x7f5a7811e440 con 0x7f5a801604d0 2023-05-22T15:47:01.782 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:01.771+0000 7f5a85365640 1 --2- 172.21.15.16:0/2213920544 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f5a640654a0 0x7f5a64067950 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f5a7c009790 tx=0x7f5a7c008040 comp rx=0 tx=0).ready entity=mgr.9396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:47:01.782 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:01.775+0000 7f5a71ffb640 1 -- 172.21.15.16:0/2213920544 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f5a780ec1e0 con 0x7f5a801604d0 2023-05-22T15:47:01.929 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:01.923+0000 7f5a875c9640 1 -- 172.21.15.16:0/2213920544 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f5a8004dbe0 con 0x7f5a801604d0 2023-05-22T15:47:02.650 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:02.631+0000 7f5a71ffb640 1 -- 172.21.15.16:0/2213920544 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v159) v1 ==== 170+0+0 (secure 0 0 0) 0x7f5a780ef7c0 con 0x7f5a801604d0 2023-05-22T15:47:02.651 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'cephfs_data' created 2023-05-22T15:47:02.654 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:02.647+0000 7f5a875c9640 1 -- 172.21.15.16:0/2213920544 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f5a640654a0 msgr2=0x7f5a64067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:47:02.655 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:02.647+0000 7f5a875c9640 1 --2- 172.21.15.16:0/2213920544 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f5a640654a0 0x7f5a64067950 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f5a7c009790 tx=0x7f5a7c008040 comp rx=0 tx=0).stop 2023-05-22T15:47:02.655 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:02.647+0000 7f5a875c9640 1 -- 172.21.15.16:0/2213920544 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5a801604d0 msgr2=0x7f5a801547c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:47:02.655 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:02.647+0000 7f5a875c9640 1 --2- 172.21.15.16:0/2213920544 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5a801604d0 0x7f5a801547c0 secure :-1 s=READY pgs=2024 cs=0 l=1 rev1=1 crypto rx=0x7f5a7801a540 tx=0x7f5a780196b0 comp rx=0 tx=0).stop 2023-05-22T15:47:02.655 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:02.647+0000 7f5a875c9640 1 -- 172.21.15.16:0/2213920544 shutdown_connections 2023-05-22T15:47:02.655 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:02.647+0000 7f5a875c9640 1 --2- 172.21.15.16:0/2213920544 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5a801604d0 0x7f5a801547c0 unknown :-1 s=CLOSED pgs=2024 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:02.655 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:02.647+0000 7f5a875c9640 1 --2- 172.21.15.16:0/2213920544 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5a800ce3b0 0x7f5a800ade10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:02.655 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:02.647+0000 7f5a875c9640 1 --2- 172.21.15.16:0/2213920544 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f5a640654a0 0x7f5a64067950 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:02.656 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:02.647+0000 7f5a875c9640 1 --2- 172.21.15.16:0/2213920544 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5a800710e0 0x7f5a8015a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:02.656 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:02.647+0000 7f5a875c9640 1 -- 172.21.15.16:0/2213920544 >> 172.21.15.16:0/2213920544 conn(0x7f5a800b18e0 msgr2=0x7f5a800b0050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:47:02.656 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:02.651+0000 7f5a875c9640 1 -- 172.21.15.16:0/2213920544 shutdown_connections 2023-05-22T15:47:02.656 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:02.651+0000 7f5a875c9640 1 -- 172.21.15.16:0/2213920544 wait complete. 2023-05-22T15:47:02.681 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-05-22T15:47:02.804 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:02.799+0000 7f3683a07640 1 Processor -- start 2023-05-22T15:47:02.804 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:02.799+0000 7f3683a07640 1 -- start start 2023-05-22T15:47:02.806 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:02.799+0000 7f3683a07640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f367c0cee30 0x7f367c06bf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:47:02.807 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:02.799+0000 7f3683a07640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f367c06db80 0x7f367c06dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:47:02.807 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:02.799+0000 7f3683a07640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f367c06e520 con 0x7f367c15c8b0 2023-05-22T15:47:02.807 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:02.799+0000 7f3683a07640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f367c15a0c0 con 0x7f367c0cee30 2023-05-22T15:47:02.807 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:02.799+0000 7f3683a07640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f367c0ce160 con 0x7f367c06db80 2023-05-22T15:47:02.807 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:02.799+0000 7f3681fa4640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f367c06db80 0x7f367c06dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:47:02.807 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:02.799+0000 7f3680fa2640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f367c0cee30 0x7f367c06bf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:47:02.808 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:02.799+0000 7f3681fa4640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f367c06db80 0x7f367c06dfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3301/0 says I am v2:172.21.15.16:45926/0 (socket says 172.21.15.16:45926) 2023-05-22T15:47:02.808 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:02.799+0000 7f3681fa4640 1 -- 172.21.15.16:0/1734348453 learned_addr learned my addr 172.21.15.16:0/1734348453 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:47:02.808 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:02.799+0000 7f36817a3640 1 --1- 172.21.15.16:0/1734348453 >> v1:172.21.15.62:6789/0 conn(0x7f367c15c8b0 0x7f367c15cc80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.16:53036/0 (socket says 172.21.15.16:53036) 2023-05-22T15:47:02.808 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:02.799+0000 7f3680fa2640 1 -- 172.21.15.16:0/1734348453 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f367c06db80 msgr2=0x7f367c06dfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:47:02.808 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:02.799+0000 7f3680fa2640 1 --2- 172.21.15.16:0/1734348453 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f367c06db80 0x7f367c06dfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:02.808 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:02.799+0000 7f3680fa2640 1 -- 172.21.15.16:0/1734348453 >> v1:172.21.15.62:6789/0 conn(0x7f367c15c8b0 legacy=0x7f367c15cc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:47:02.809 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:02.799+0000 7f3680fa2640 1 -- 172.21.15.16:0/1734348453 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f367c06c640 con 0x7f367c0cee30 2023-05-22T15:47:02.810 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:02.803+0000 7f3680fa2640 1 --2- 172.21.15.16:0/1734348453 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f367c0cee30 0x7f367c06bf60 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f366c0097c0 tx=0x7f366c01c450 comp rx=0 tx=0).ready entity=mon.1 client_cookie=45ed3e319d689d22 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:47:02.810 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:02.803+0000 7f3673fff640 1 -- 172.21.15.16:0/1734348453 <== mon.1 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f366c0ab650 con 0x7f367c0cee30 2023-05-22T15:47:02.810 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:02.803+0000 7f3673fff640 1 -- 172.21.15.16:0/1734348453 <== mon.1 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f366c0ab7b0 con 0x7f367c0cee30 2023-05-22T15:47:02.810 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:02.803+0000 7f3673fff640 1 -- 172.21.15.16:0/1734348453 <== mon.1 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f366c0abb40 con 0x7f367c0cee30 2023-05-22T15:47:02.810 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:02.803+0000 7f3683a07640 1 -- 172.21.15.16:0/1734348453 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f367c0cee30 msgr2=0x7f367c06bf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:47:02.811 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:02.803+0000 7f3683a07640 1 --2- 172.21.15.16:0/1734348453 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f367c0cee30 0x7f367c06bf60 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f366c0097c0 tx=0x7f366c01c450 comp rx=0 tx=0).stop 2023-05-22T15:47:02.811 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:02.803+0000 7f3683a07640 1 -- 172.21.15.16:0/1734348453 shutdown_connections 2023-05-22T15:47:02.811 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:02.803+0000 7f3683a07640 1 --2- 172.21.15.16:0/1734348453 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f367c06db80 0x7f367c06dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:02.811 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:02.803+0000 7f3683a07640 1 --2- 172.21.15.16:0/1734348453 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f367c0cee30 0x7f367c06bf60 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:02.811 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:02.803+0000 7f3683a07640 1 -- 172.21.15.16:0/1734348453 >> 172.21.15.16:0/1734348453 conn(0x7f367c06f5a0 msgr2=0x7f367c06f9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:47:02.811 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:02.803+0000 7f3683a07640 1 -- 172.21.15.16:0/1734348453 shutdown_connections 2023-05-22T15:47:02.811 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:02.803+0000 7f3683a07640 1 -- 172.21.15.16:0/1734348453 wait complete. 2023-05-22T15:47:02.814 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:02.807+0000 7f3683a07640 1 Processor -- start 2023-05-22T15:47:02.814 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:02.807+0000 7f3683a07640 1 -- start start 2023-05-22T15:47:02.814 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:02.807+0000 7f3683a07640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f367c06db80 0x7f367c15efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:47:02.814 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:02.807+0000 7f3683a07640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f367c0cee30 0x7f367c15d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:47:02.814 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:02.807+0000 7f3683a07640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f367c15c8b0 0x7f367c0ccdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:47:02.814 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:02.807+0000 7f3683a07640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f367c15dcc0 con 0x7f367c0cee30 2023-05-22T15:47:02.815 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:02.807+0000 7f3683a07640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f367c15df60 con 0x7f367c06db80 2023-05-22T15:47:02.815 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:02.807+0000 7f3683a07640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f367c15e240 con 0x7f367c15c8b0 2023-05-22T15:47:02.815 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:02.807+0000 7f3680fa2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f367c0cee30 0x7f367c15d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:47:02.815 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:02.807+0000 7f3680fa2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f367c0cee30 0x7f367c15d780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46756/0 (socket says 172.21.15.16:46756) 2023-05-22T15:47:02.815 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:02.807+0000 7f3680fa2640 1 -- 172.21.15.16:0/2300610043 learned_addr learned my addr 172.21.15.16:0/2300610043 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:47:02.815 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:02.807+0000 7f3681fa4640 1 --2- 172.21.15.16:0/2300610043 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f367c15c8b0 0x7f367c0ccdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:47:02.815 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:02.807+0000 7f36817a3640 1 --2- 172.21.15.16:0/2300610043 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f367c06db80 0x7f367c15efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:47:02.816 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:02.807+0000 7f3680fa2640 1 -- 172.21.15.16:0/2300610043 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f367c15c8b0 msgr2=0x7f367c0ccdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:47:02.816 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:02.807+0000 7f3680fa2640 1 --2- 172.21.15.16:0/2300610043 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f367c15c8b0 0x7f367c0ccdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:02.816 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:02.807+0000 7f3680fa2640 1 -- 172.21.15.16:0/2300610043 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f367c06db80 msgr2=0x7f367c15efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:47:02.816 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:02.807+0000 7f3680fa2640 1 --2- 172.21.15.16:0/2300610043 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f367c06db80 0x7f367c15efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:02.816 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:02.807+0000 7f3680fa2640 1 -- 172.21.15.16:0/2300610043 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f366c009420 con 0x7f367c0cee30 2023-05-22T15:47:02.816 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:02.807+0000 7f3680fa2640 1 --2- 172.21.15.16:0/2300610043 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f367c0cee30 0x7f367c15d780 secure :-1 s=READY pgs=2025 cs=0 l=1 rev1=1 crypto rx=0x7f366c0ad040 tx=0x7f366c01ac90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:47:02.816 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:02.807+0000 7f3671ffb640 1 -- 172.21.15.16:0/2300610043 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f366c0bf610 con 0x7f367c0cee30 2023-05-22T15:47:02.817 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:02.807+0000 7f3671ffb640 1 -- 172.21.15.16:0/2300610043 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f366c0bf790 con 0x7f367c0cee30 2023-05-22T15:47:02.817 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:02.807+0000 7f3671ffb640 1 -- 172.21.15.16:0/2300610043 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f366c0bfb40 con 0x7f367c0cee30 2023-05-22T15:47:02.817 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:02.807+0000 7f3683a07640 1 -- 172.21.15.16:0/2300610043 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f367c15d390 con 0x7f367c0cee30 2023-05-22T15:47:02.817 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:02.807+0000 7f3683a07640 1 -- 172.21.15.16:0/2300610043 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f367c0ce930 con 0x7f367c0cee30 2023-05-22T15:47:02.817 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:02.807+0000 7f3683a07640 1 -- 172.21.15.16:0/2300610043 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f367c15d520 con 0x7f367c0cee30 2023-05-22T15:47:02.818 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:02.807+0000 7f3671ffb640 1 -- 172.21.15.16:0/2300610043 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 67) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f366c0b5400 con 0x7f367c0cee30 2023-05-22T15:47:02.818 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:02.811+0000 7f3671ffb640 1 --2- 172.21.15.16:0/2300610043 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f36600654a0 0x7f3660067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:47:02.819 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:02.811+0000 7f3671ffb640 1 -- 172.21.15.16:0/2300610043 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(159..159 src has 1..159) v4 ==== 6310+0+0 (secure 0 0 0) 0x7f366c11e100 con 0x7f367c0cee30 2023-05-22T15:47:02.819 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:02.811+0000 7f36817a3640 1 --2- 172.21.15.16:0/2300610043 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f36600654a0 0x7f3660067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:47:02.819 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:02.811+0000 7f36817a3640 1 --2- 172.21.15.16:0/2300610043 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f36600654a0 0x7f3660067950 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f3678007fd0 tx=0x7f3678005b80 comp rx=0 tx=0).ready entity=mgr.9396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:47:02.819 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:02.815+0000 7f3671ffb640 1 -- 172.21.15.16:0/2300610043 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f366c0ebb60 con 0x7f367c0cee30 2023-05-22T15:47:03.008 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.003+0000 7f3683a07640 1 -- 172.21.15.16:0/2300610043 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f367c04dbe0 con 0x7f367c0cee30 2023-05-22T15:47:03.022 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.015+0000 7f3671ffb640 1 -- 172.21.15.16:0/2300610043 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 20 and data pool 21 v62) v1 ==== 174+0+0 (secure 0 0 0) 0x7f366c0ef180 con 0x7f367c0cee30 2023-05-22T15:47:03.023 INFO:teuthology.orchestra.run.smithi016.stderr:new fs with metadata pool 20 and data pool 21 2023-05-22T15:47:03.027 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.019+0000 7f3683a07640 1 -- 172.21.15.16:0/2300610043 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f36600654a0 msgr2=0x7f3660067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:47:03.027 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.019+0000 7f3683a07640 1 --2- 172.21.15.16:0/2300610043 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f36600654a0 0x7f3660067950 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f3678007fd0 tx=0x7f3678005b80 comp rx=0 tx=0).stop 2023-05-22T15:47:03.029 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.019+0000 7f3683a07640 1 -- 172.21.15.16:0/2300610043 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f367c0cee30 msgr2=0x7f367c15d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:47:03.029 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.023+0000 7f3683a07640 1 --2- 172.21.15.16:0/2300610043 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f367c0cee30 0x7f367c15d780 secure :-1 s=READY pgs=2025 cs=0 l=1 rev1=1 crypto rx=0x7f366c0ad040 tx=0x7f366c01ac90 comp rx=0 tx=0).stop 2023-05-22T15:47:03.029 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.023+0000 7f3683a07640 1 -- 172.21.15.16:0/2300610043 shutdown_connections 2023-05-22T15:47:03.029 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.023+0000 7f3683a07640 1 --2- 172.21.15.16:0/2300610043 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f36600654a0 0x7f3660067950 secure :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f3678007fd0 tx=0x7f3678005b80 comp rx=0 tx=0).stop 2023-05-22T15:47:03.030 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.023+0000 7f3683a07640 1 --2- 172.21.15.16:0/2300610043 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f367c15c8b0 0x7f367c0ccdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:03.030 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.023+0000 7f3683a07640 1 --2- 172.21.15.16:0/2300610043 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f367c0cee30 0x7f367c15d780 unknown :-1 s=CLOSED pgs=2025 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:03.030 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.023+0000 7f3683a07640 1 --2- 172.21.15.16:0/2300610043 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f367c06db80 0x7f367c15efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:03.030 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.023+0000 7f3683a07640 1 -- 172.21.15.16:0/2300610043 >> 172.21.15.16:0/2300610043 conn(0x7f367c06f5a0 msgr2=0x7f367c15bc50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:47:03.034 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.027+0000 7f3683a07640 1 -- 172.21.15.16:0/2300610043 shutdown_connections 2023-05-22T15:47:03.037 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.027+0000 7f3683a07640 1 -- 172.21.15.16:0/2300610043 wait complete. 2023-05-22T15:47:03.064 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-05-22T15:47:03.179 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.171+0000 7f2a0c7e9640 1 Processor -- start 2023-05-22T15:47:03.179 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.171+0000 7f2a0c7e9640 1 -- start start 2023-05-22T15:47:03.180 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.175+0000 7f2a0c7e9640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2a04071500 0x7f2a0415e460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:47:03.180 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.175+0000 7f2a0c7e9640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a040b0650 con 0x7f2a040c9b40 2023-05-22T15:47:03.180 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.175+0000 7f2a0c7e9640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a040b0370 con 0x7f2a040ca4e0 2023-05-22T15:47:03.180 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.175+0000 7f2a0c7e9640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a0415d980 con 0x7f2a04071500 2023-05-22T15:47:03.181 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.175+0000 7f2a09d84640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f2a040c9b40 0x7f2a04070df0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:44840/0 (socket says 172.21.15.16:44840) 2023-05-22T15:47:03.181 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.175+0000 7f2a09d84640 1 -- 172.21.15.16:0/1056911690 learned_addr learned my addr 172.21.15.16:0/1056911690 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:47:03.181 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.175+0000 7f2a0ad86640 1 --2- 172.21.15.16:0/1056911690 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2a04071500 0x7f2a0415e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:47:03.181 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.175+0000 7f2a09583640 1 -- 172.21.15.16:0/1056911690 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1518507085 0 0) 0x7f2a040b0650 con 0x7f2a040c9b40 2023-05-22T15:47:03.181 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.175+0000 7f2a09583640 1 -- 172.21.15.16:0/1056911690 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f29f0003530 con 0x7f2a040c9b40 2023-05-22T15:47:03.183 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.175+0000 7f2a09583640 1 -- 172.21.15.16:0/1056911690 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2859100731 0 0) 0x7f29f0003530 con 0x7f2a040c9b40 2023-05-22T15:47:03.183 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.175+0000 7f2a09583640 1 -- 172.21.15.16:0/1056911690 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2a04071500 msgr2=0x7f2a0415e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:47:03.184 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.175+0000 7f2a09583640 1 --2- 172.21.15.16:0/1056911690 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2a04071500 0x7f2a0415e460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:03.184 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.175+0000 7f2a09583640 1 -- 172.21.15.16:0/1056911690 >> v1:172.21.15.62:6789/0 conn(0x7f2a040ca4e0 legacy=0x7f2a040afb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:47:03.184 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.175+0000 7f2a09583640 1 -- 172.21.15.16:0/1056911690 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a0415dc60 con 0x7f2a040c9b40 2023-05-22T15:47:03.185 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.175+0000 7f2a09583640 1 -- 172.21.15.16:0/1056911690 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7f29f40016a0 con 0x7f2a040c9b40 2023-05-22T15:47:03.185 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.175+0000 7f2a09583640 1 -- 172.21.15.16:0/1056911690 >> v1:172.21.15.16:6789/0 conn(0x7f2a040c9b40 legacy=0x7f2a04070df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:47:03.185 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.175+0000 7f2a09583640 1 --2- 172.21.15.16:0/1056911690 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2a04071500 0x7f29f00039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:47:03.185 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.175+0000 7f2a09583640 1 --2- 172.21.15.16:0/1056911690 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f29f0003f80 0x7f29f00047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:47:03.185 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.175+0000 7f2a09583640 1 --2- 172.21.15.16:0/1056911690 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f29f0004d10 0x7f29f0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:47:03.185 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.175+0000 7f2a09583640 1 -- 172.21.15.16:0/1056911690 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a0415d980 con 0x7f29f0003f80 2023-05-22T15:47:03.186 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.175+0000 7f2a09583640 1 -- 172.21.15.16:0/1056911690 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29f000baa0 con 0x7f29f0004d10 2023-05-22T15:47:03.186 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.175+0000 7f2a09583640 1 -- 172.21.15.16:0/1056911690 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29f000bc60 con 0x7f2a04071500 2023-05-22T15:47:03.186 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.175+0000 7f2a09d84640 1 --2- 172.21.15.16:0/1056911690 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f29f0003f80 0x7f29f00047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:47:03.186 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.175+0000 7f2a0a585640 1 --2- 172.21.15.16:0/1056911690 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2a04071500 0x7f29f00039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:47:03.186 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.175+0000 7f2a0ad86640 1 --2- 172.21.15.16:0/1056911690 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f29f0004d10 0x7f29f0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:47:03.186 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.175+0000 7f2a09d84640 1 -- 172.21.15.16:0/1056911690 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2a04071500 msgr2=0x7f29f00039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:47:03.187 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.175+0000 7f2a09d84640 1 --2- 172.21.15.16:0/1056911690 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2a04071500 0x7f29f00039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:03.187 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.175+0000 7f2a09d84640 1 -- 172.21.15.16:0/1056911690 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f29f0004d10 msgr2=0x7f29f0005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:47:03.187 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.175+0000 7f2a09d84640 1 --2- 172.21.15.16:0/1056911690 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f29f0004d10 0x7f29f0005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:03.187 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.175+0000 7f2a09d84640 1 -- 172.21.15.16:0/1056911690 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29f0003530 con 0x7f29f0003f80 2023-05-22T15:47:03.187 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.179+0000 7f2a09d84640 1 --2- 172.21.15.16:0/1056911690 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f29f0003f80 0x7f29f00047d0 secure :-1 s=READY pgs=2027 cs=0 l=1 rev1=1 crypto rx=0x7f29f400c160 tx=0x7f29f401f330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29edb0e9403d0ca8 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:47:03.187 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.179+0000 7f2a09583640 1 -- 172.21.15.16:0/1056911690 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29f40058e0 con 0x7f29f0003f80 2023-05-22T15:47:03.187 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.179+0000 7f2a09583640 1 -- 172.21.15.16:0/1056911690 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29f4005a40 con 0x7f29f0003f80 2023-05-22T15:47:03.187 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.179+0000 7f2a09583640 1 -- 172.21.15.16:0/1056911690 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29f4005df0 con 0x7f29f0003f80 2023-05-22T15:47:03.190 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.179+0000 7f2a0c7e9640 1 -- 172.21.15.16:0/1056911690 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f29f0003f80 msgr2=0x7f29f00047d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:47:03.190 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.179+0000 7f2a0c7e9640 1 --2- 172.21.15.16:0/1056911690 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f29f0003f80 0x7f29f00047d0 secure :-1 s=READY pgs=2027 cs=0 l=1 rev1=1 crypto rx=0x7f29f400c160 tx=0x7f29f401f330 comp rx=0 tx=0).stop 2023-05-22T15:47:03.191 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.179+0000 7f2a0ad86640 1 -- 172.21.15.16:0/1056911690 reap_dead start 2023-05-22T15:47:03.191 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.179+0000 7f2a0c7e9640 1 -- 172.21.15.16:0/1056911690 shutdown_connections 2023-05-22T15:47:03.191 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.179+0000 7f2a0c7e9640 1 -- 172.21.15.16:0/1056911690 >> 172.21.15.16:0/1056911690 conn(0x7f2a04069720 msgr2=0x7f2a04069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:47:03.191 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.179+0000 7f2a0c7e9640 1 -- 172.21.15.16:0/1056911690 shutdown_connections 2023-05-22T15:47:03.191 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.179+0000 7f2a0c7e9640 1 -- 172.21.15.16:0/1056911690 wait complete. 2023-05-22T15:47:03.191 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.179+0000 7f2a0c7e9640 1 Processor -- start 2023-05-22T15:47:03.192 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.179+0000 7f2a0c7e9640 1 -- start start 2023-05-22T15:47:03.192 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.179+0000 7f2a0c7e9640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f29f0003f80 0x7f2a041eef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:47:03.192 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.179+0000 7f2a0c7e9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2a040c3a30 0x7f2a040c3ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:47:03.192 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.179+0000 7f2a0c7e9640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2a040c1720 0x7f2a040c1b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:47:03.192 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.179+0000 7f2a0c7e9640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a040c43e0 con 0x7f2a040c3a30 2023-05-22T15:47:03.192 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.179+0000 7f2a0c7e9640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a040c20d0 con 0x7f29f0003f80 2023-05-22T15:47:03.193 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.179+0000 7f2a0c7e9640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a040c2f10 con 0x7f2a040c1720 2023-05-22T15:47:03.193 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.179+0000 7f2a09d84640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2a040c3a30 0x7f2a040c3ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:47:03.193 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.179+0000 7f2a09d84640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2a040c3a30 0x7f2a040c3ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46762/0 (socket says 172.21.15.16:46762) 2023-05-22T15:47:03.193 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.179+0000 7f2a09d84640 1 -- 172.21.15.16:0/1594481987 learned_addr learned my addr 172.21.15.16:0/1594481987 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:47:03.193 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.179+0000 7f2a0a585640 1 --2- 172.21.15.16:0/1594481987 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f29f0003f80 0x7f2a041eef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:47:03.193 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.179+0000 7f2a0ad86640 1 --2- 172.21.15.16:0/1594481987 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2a040c1720 0x7f2a040c1b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:47:03.194 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.179+0000 7f2a09d84640 1 -- 172.21.15.16:0/1594481987 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2a040c1720 msgr2=0x7f2a040c1b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:47:03.194 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.179+0000 7f2a09d84640 1 --2- 172.21.15.16:0/1594481987 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2a040c1720 0x7f2a040c1b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:03.194 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.179+0000 7f2a09d84640 1 -- 172.21.15.16:0/1594481987 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f29f0003f80 msgr2=0x7f2a041eef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:47:03.194 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.179+0000 7f2a09d84640 1 --2- 172.21.15.16:0/1594481987 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f29f0003f80 0x7f2a041eef70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:03.194 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.179+0000 7f2a09d84640 1 -- 172.21.15.16:0/1594481987 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a040ca8b0 con 0x7f2a040c3a30 2023-05-22T15:47:03.195 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.179+0000 7f2a09d84640 1 --2- 172.21.15.16:0/1594481987 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2a040c3a30 0x7f2a040c3ea0 secure :-1 s=READY pgs=2028 cs=0 l=1 rev1=1 crypto rx=0x7f29f401f160 tx=0x7f29f0005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:47:03.195 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.183+0000 7f29faffd640 1 -- 172.21.15.16:0/1594481987 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29f40b7070 con 0x7f2a040c3a30 2023-05-22T15:47:03.195 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.183+0000 7f29faffd640 1 -- 172.21.15.16:0/1594481987 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29f40013e0 con 0x7f2a040c3a30 2023-05-22T15:47:03.195 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.183+0000 7f2a0c7e9640 1 -- 172.21.15.16:0/1594481987 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a0415df40 con 0x7f2a040c3a30 2023-05-22T15:47:03.195 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.183+0000 7f29faffd640 1 -- 172.21.15.16:0/1594481987 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29f4007050 con 0x7f2a040c3a30 2023-05-22T15:47:03.195 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.183+0000 7f2a0c7e9640 1 -- 172.21.15.16:0/1594481987 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a0415e460 con 0x7f2a040c3a30 2023-05-22T15:47:03.195 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.183+0000 7f2a0c7e9640 1 -- 172.21.15.16:0/1594481987 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a041ef4b0 con 0x7f2a040c3a30 2023-05-22T15:47:03.196 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.183+0000 7f29faffd640 1 -- 172.21.15.16:0/1594481987 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 67) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f29f40071b0 con 0x7f2a040c3a30 2023-05-22T15:47:03.196 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.183+0000 7f29faffd640 1 --2- 172.21.15.16:0/1594481987 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f29e80654d0 0x7f29e8067980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:47:03.196 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.183+0000 7f29faffd640 1 -- 172.21.15.16:0/1594481987 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(160..160 src has 1..160) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f29f411fd20 con 0x7f2a040c3a30 2023-05-22T15:47:03.196 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.183+0000 7f2a0a585640 1 --2- 172.21.15.16:0/1594481987 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f29e80654d0 0x7f29e8067980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:47:03.196 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.183+0000 7f2a0a585640 1 --2- 172.21.15.16:0/1594481987 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f29e80654d0 0x7f29e8067980 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f2a00000c10 tx=0x7f2a00009040 comp rx=0 tx=0).ready entity=mgr.9396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:47:03.197 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.187+0000 7f29faffd640 1 -- 172.21.15.16:0/1594481987 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f29f40ed7c0 con 0x7f2a040c3a30 2023-05-22T15:47:03.351 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.343+0000 7f2a0c7e9640 1 -- 172.21.15.16:0/1594481987 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2a040c3050 con 0x7f2a040c3a30 2023-05-22T15:47:03.352 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.347+0000 7f29faffd640 1 -- 172.21.15.16:0/1594481987 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v160) v1 ==== 74+0+18107 (secure 0 0 0) 0x7f29f40f0de0 con 0x7f2a040c3a30 2023-05-22T15:47:03.352 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:47:03.353 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":160,"fsid":"b2719834-7847-4d98-90a6-f65d57f85c49","created":"2023-05-22T15:26:25.259314+0000","modified":"2023-05-22T15:47:03.008267+0000","last_up_change":"2023-05-22T15:26:36.448524+0000","last_in_change":"2023-05-22T15:26:27.675668+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-05-22T15:26:38.019781+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-05-22T15:26:43.570514+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_metadata","create_time":"2023-05-22T15:47:01.347542+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"160","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_data","create_time":"2023-05-22T15:47:01.970646+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"160","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9e7ff854-b25f-4cbb-a5c2-1799d9c53d2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":37021},{"type":"v1","addr":"172.21.15.16:6825","nonce":37021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":37021},{"type":"v1","addr":"172.21.15.16:6827","nonce":37021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":37021},{"type":"v1","addr":"172.21.15.16:6831","nonce":37021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":37021},{"type":"v1","addr":"172.21.15.16:6829","nonce":37021}]},"public_addr":"172.21.15.16:6825/37021","cluster_addr":"172.21.15.16:6827/37021","heartbeat_back_addr":"172.21.15.16:6831/37021","heartbeat_front_addr":"172.21.15.16:6829/37021","state":["exists","up"]},{"osd":1,"uuid":"4702fbc9-a927-4565-aca5-41dc69cc8264","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":37008},{"type":"v1","addr":"172.21.15.16:6801","nonce":37008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":37008},{"type":"v1","addr":"172.21.15.16:6803","nonce":37008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":37008},{"type":"v1","addr":"172.21.15.16:6807","nonce":37008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":37008},{"type":"v1","addr":"172.21.15.16:6805","nonce":37008}]},"public_addr":"172.21.15.16:6801/37008","cluster_addr":"172.21.15.16:6803/37008","heartbeat_back_addr":"172.21.15.16:6807/37008","heartbeat_front_addr":"172.21.15.16:6805/37008","state":["exists","up"]},{"osd":2,"uuid":"1e7bba46-4eba-47ef-bb81-5e2d371638ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":37009},{"type":"v1","addr":"172.21.15.16:6809","nonce":37009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":37009},{"type":"v1","addr":"172.21.15.16:6811","nonce":37009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":37009},{"type":"v1","addr":"172.21.15.16:6815","nonce":37009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":37009},{"type":"v1","addr":"172.21.15.16:6813","nonce":37009}]},"public_addr":"172.21.15.16:6809/37009","cluster_addr":"172.21.15.16:6811/37009","heartbeat_back_addr":"172.21.15.16:6815/37009","heartbeat_front_addr":"172.21.15.16:6813/37009","state":["exists","up"]},{"osd":3,"uuid":"383cacc0-5a88-4904-990e-498ca59ef48a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":37020},{"type":"v1","addr":"172.21.15.16:6817","nonce":37020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":37020},{"type":"v1","addr":"172.21.15.16:6819","nonce":37020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":37020},{"type":"v1","addr":"172.21.15.16:6823","nonce":37020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":37020},{"type":"v1","addr":"172.21.15.16:6821","nonce":37020}]},"public_addr":"172.21.15.16:6817/37020","cluster_addr":"172.21.15.16:6819/37020","heartbeat_back_addr":"172.21.15.16:6823/37020","heartbeat_front_addr":"172.21.15.16:6821/37020","state":["exists","up"]},{"osd":4,"uuid":"970500ef-95b7-42c1-b080-eed4fae7013d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":37353},{"type":"v1","addr":"172.21.15.62:6817","nonce":37353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6819","nonce":37353},{"type":"v1","addr":"172.21.15.62:6822","nonce":37353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37353},{"type":"v1","addr":"172.21.15.62:6831","nonce":37353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37353},{"type":"v1","addr":"172.21.15.62:6826","nonce":37353}]},"public_addr":"172.21.15.62:6817/37353","cluster_addr":"172.21.15.62:6822/37353","heartbeat_back_addr":"172.21.15.62:6831/37353","heartbeat_front_addr":"172.21.15.62:6826/37353","state":["exists","up"]},{"osd":5,"uuid":"022d67db-d4d8-445b-b1b9-8b446a00f5c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":37352},{"type":"v1","addr":"172.21.15.62:6801","nonce":37352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":37352},{"type":"v1","addr":"172.21.15.62:6803","nonce":37352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":37352},{"type":"v1","addr":"172.21.15.62:6807","nonce":37352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":37352},{"type":"v1","addr":"172.21.15.62:6805","nonce":37352}]},"public_addr":"172.21.15.62:6801/37352","cluster_addr":"172.21.15.62:6803/37352","heartbeat_back_addr":"172.21.15.62:6807/37352","heartbeat_front_addr":"172.21.15.62:6805/37352","state":["exists","up"]},{"osd":6,"uuid":"42bd372b-6688-4a1b-8407-e16e8c266e33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":37354},{"type":"v1","addr":"172.21.15.62:6820","nonce":37354}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":37354},{"type":"v1","addr":"172.21.15.62:6823","nonce":37354}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6829","nonce":37354},{"type":"v1","addr":"172.21.15.62:6830","nonce":37354}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6825","nonce":37354},{"type":"v1","addr":"172.21.15.62:6827","nonce":37354}]},"public_addr":"172.21.15.62:6820/37354","cluster_addr":"172.21.15.62:6823/37354","heartbeat_back_addr":"172.21.15.62:6830/37354","heartbeat_front_addr":"172.21.15.62:6827/37354","state":["exists","up"]},{"osd":7,"uuid":"ac70b2a3-f41b-4bf2-b227-7f2cbb55426d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37355},{"type":"v1","addr":"172.21.15.62:6809","nonce":37355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37355},{"type":"v1","addr":"172.21.15.62:6811","nonce":37355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":37355},{"type":"v1","addr":"172.21.15.62:6815","nonce":37355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":37355},{"type":"v1","addr":"172.21.15.62:6813","nonce":37355}]},"public_addr":"172.21.15.62:6809/37355","cluster_addr":"172.21.15.62:6811/37355","heartbeat_back_addr":"172.21.15.62:6815/37355","heartbeat_front_addr":"172.21.15.62:6813/37355","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.368896+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.107273+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.276888+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.287502+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.124791+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.251937+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.255679+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.232797+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":17,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-05-22T15:47:03.354 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.347+0000 7f2a0c7e9640 1 -- 172.21.15.16:0/1594481987 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f29e80654d0 msgr2=0x7f29e8067980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:47:03.354 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.347+0000 7f2a0c7e9640 1 --2- 172.21.15.16:0/1594481987 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f29e80654d0 0x7f29e8067980 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f2a00000c10 tx=0x7f2a00009040 comp rx=0 tx=0).stop 2023-05-22T15:47:03.354 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.347+0000 7f2a0c7e9640 1 -- 172.21.15.16:0/1594481987 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2a040c3a30 msgr2=0x7f2a040c3ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:47:03.354 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.347+0000 7f2a0c7e9640 1 --2- 172.21.15.16:0/1594481987 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2a040c3a30 0x7f2a040c3ea0 secure :-1 s=READY pgs=2028 cs=0 l=1 rev1=1 crypto rx=0x7f29f401f160 tx=0x7f29f0005800 comp rx=0 tx=0).stop 2023-05-22T15:47:03.355 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.347+0000 7f2a0c7e9640 1 -- 172.21.15.16:0/1594481987 shutdown_connections 2023-05-22T15:47:03.355 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.347+0000 7f2a0c7e9640 1 --2- 172.21.15.16:0/1594481987 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f29e80654d0 0x7f29e8067980 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:03.355 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.347+0000 7f2a0c7e9640 1 --2- 172.21.15.16:0/1594481987 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2a040c1720 0x7f2a040c1b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:03.355 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.347+0000 7f2a0c7e9640 1 --2- 172.21.15.16:0/1594481987 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2a040c3a30 0x7f2a040c3ea0 unknown :-1 s=CLOSED pgs=2028 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:03.356 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.347+0000 7f2a0c7e9640 1 --2- 172.21.15.16:0/1594481987 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f29f0003f80 0x7f2a041eef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:03.356 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.347+0000 7f2a0c7e9640 1 -- 172.21.15.16:0/1594481987 >> 172.21.15.16:0/1594481987 conn(0x7f2a04069720 msgr2=0x7f2a04159590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:47:03.356 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.347+0000 7f2a0c7e9640 1 -- 172.21.15.16:0/1594481987 shutdown_connections 2023-05-22T15:47:03.356 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.347+0000 7f2a0c7e9640 1 -- 172.21.15.16:0/1594481987 wait complete. 2023-05-22T15:47:03.373 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-05-22T15:47:03.499 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.491+0000 7f41d814b640 1 Processor -- start 2023-05-22T15:47:03.499 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.495+0000 7f41d814b640 1 -- start start 2023-05-22T15:47:03.500 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.495+0000 7f41d814b640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f41d00b0250 0x7f41d00acba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:47:03.500 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.495+0000 7f41d814b640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41d00cc300 con 0x7f41d01531d0 2023-05-22T15:47:03.500 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.495+0000 7f41d814b640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41d00cdb60 con 0x7f41d00cd730 2023-05-22T15:47:03.501 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.495+0000 7f41d814b640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41d015d330 con 0x7f41d00b0250 2023-05-22T15:47:03.501 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.495+0000 7f41d5ee7640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f41d01531d0 0x7f41d00cbbf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:44856/0 (socket says 172.21.15.16:44856) 2023-05-22T15:47:03.501 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.495+0000 7f41d5ee7640 1 -- 172.21.15.16:0/2959310757 learned_addr learned my addr 172.21.15.16:0/2959310757 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:47:03.504 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.495+0000 7f41d66e8640 1 --2- 172.21.15.16:0/2959310757 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f41d00b0250 0x7f41d00acba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:47:03.504 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.495+0000 7f41d4ee5640 1 -- 172.21.15.16:0/2959310757 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1589935522 0 0) 0x7f41d00cc300 con 0x7f41d01531d0 2023-05-22T15:47:03.504 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.495+0000 7f41d4ee5640 1 -- 172.21.15.16:0/2959310757 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f41bc003530 con 0x7f41d01531d0 2023-05-22T15:47:03.505 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.495+0000 7f41d4ee5640 1 -- 172.21.15.16:0/2959310757 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2243072245 0 0) 0x7f41bc003530 con 0x7f41d01531d0 2023-05-22T15:47:03.506 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.495+0000 7f41d4ee5640 1 -- 172.21.15.16:0/2959310757 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f41d00b0250 msgr2=0x7f41d00acba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:47:03.506 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.495+0000 7f41d4ee5640 1 --2- 172.21.15.16:0/2959310757 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f41d00b0250 0x7f41d00acba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:03.506 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.495+0000 7f41d4ee5640 1 -- 172.21.15.16:0/2959310757 >> v1:172.21.15.62:6789/0 conn(0x7f41d00cd730 legacy=0x7f41d00afb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:47:03.506 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.495+0000 7f41d4ee5640 1 -- 172.21.15.16:0/2959310757 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41d015c6b0 con 0x7f41d01531d0 2023-05-22T15:47:03.506 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.495+0000 7f41d4ee5640 1 -- 172.21.15.16:0/2959310757 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7f41cc001680 con 0x7f41d01531d0 2023-05-22T15:47:03.506 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.495+0000 7f41d4ee5640 1 -- 172.21.15.16:0/2959310757 >> v1:172.21.15.16:6789/0 conn(0x7f41d01531d0 legacy=0x7f41d00cbbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:47:03.507 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.495+0000 7f41d4ee5640 1 --2- 172.21.15.16:0/2959310757 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f41bc0039b0 0x7f41bc003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:47:03.507 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.495+0000 7f41d4ee5640 1 --2- 172.21.15.16:0/2959310757 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f41d00b0250 0x7f41bc0047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:47:03.507 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.495+0000 7f41d4ee5640 1 --2- 172.21.15.16:0/2959310757 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f41bc004d10 0x7f41bc005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:47:03.507 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.495+0000 7f41d4ee5640 1 -- 172.21.15.16:0/2959310757 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41d015d330 con 0x7f41bc0039b0 2023-05-22T15:47:03.507 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.495+0000 7f41d4ee5640 1 -- 172.21.15.16:0/2959310757 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41bc00ba40 con 0x7f41bc004d10 2023-05-22T15:47:03.507 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.495+0000 7f41d4ee5640 1 -- 172.21.15.16:0/2959310757 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41bc00bbd0 con 0x7f41d00b0250 2023-05-22T15:47:03.507 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.495+0000 7f41d5ee7640 1 --2- 172.21.15.16:0/2959310757 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f41bc0039b0 0x7f41bc003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:47:03.507 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.495+0000 7f41d56e6640 1 --2- 172.21.15.16:0/2959310757 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f41d00b0250 0x7f41bc0047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:47:03.508 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.495+0000 7f41d5ee7640 1 -- 172.21.15.16:0/2959310757 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f41d00b0250 msgr2=0x7f41bc0047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:47:03.508 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.495+0000 7f41d5ee7640 1 --2- 172.21.15.16:0/2959310757 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f41d00b0250 0x7f41bc0047d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:03.508 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.495+0000 7f41d5ee7640 1 -- 172.21.15.16:0/2959310757 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f41bc004d10 msgr2=0x7f41bc005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:47:03.508 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.495+0000 7f41d66e8640 1 --2- 172.21.15.16:0/2959310757 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f41bc004d10 0x7f41bc005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:47:03.508 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.495+0000 7f41d5ee7640 1 --2- 172.21.15.16:0/2959310757 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f41bc004d10 0x7f41bc005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:03.509 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.495+0000 7f41d5ee7640 1 -- 172.21.15.16:0/2959310757 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41bc003530 con 0x7f41bc0039b0 2023-05-22T15:47:03.509 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.499+0000 7f41d5ee7640 1 --2- 172.21.15.16:0/2959310757 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f41bc0039b0 0x7f41bc003dc0 secure :-1 s=READY pgs=2030 cs=0 l=1 rev1=1 crypto rx=0x7f41cc00c240 tx=0x7f41cc01cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70c9b740d316f7ba server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:47:03.509 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.499+0000 7f41d4ee5640 1 -- 172.21.15.16:0/2959310757 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41cc005ae0 con 0x7f41bc0039b0 2023-05-22T15:47:03.509 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.499+0000 7f41d4ee5640 1 -- 172.21.15.16:0/2959310757 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41cc005c40 con 0x7f41bc0039b0 2023-05-22T15:47:03.509 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.499+0000 7f41d4ee5640 1 -- 172.21.15.16:0/2959310757 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41cc0a5810 con 0x7f41bc0039b0 2023-05-22T15:47:03.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.499+0000 7f41d814b640 1 -- 172.21.15.16:0/2959310757 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f41bc0039b0 msgr2=0x7f41bc003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:47:03.513 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.499+0000 7f41d814b640 1 --2- 172.21.15.16:0/2959310757 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f41bc0039b0 0x7f41bc003dc0 secure :-1 s=READY pgs=2030 cs=0 l=1 rev1=1 crypto rx=0x7f41cc00c240 tx=0x7f41cc01cc40 comp rx=0 tx=0).stop 2023-05-22T15:47:03.513 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.499+0000 7f41d66e8640 1 -- 172.21.15.16:0/2959310757 reap_dead start 2023-05-22T15:47:03.513 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.499+0000 7f41d814b640 1 -- 172.21.15.16:0/2959310757 shutdown_connections 2023-05-22T15:47:03.513 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.499+0000 7f41d814b640 1 -- 172.21.15.16:0/2959310757 >> 172.21.15.16:0/2959310757 conn(0x7f41d0069720 msgr2=0x7f41d0069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:47:03.513 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.499+0000 7f41d814b640 1 -- 172.21.15.16:0/2959310757 shutdown_connections 2023-05-22T15:47:03.513 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.499+0000 7f41d814b640 1 -- 172.21.15.16:0/2959310757 wait complete. 2023-05-22T15:47:03.514 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.499+0000 7f41d814b640 1 Processor -- start 2023-05-22T15:47:03.514 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.499+0000 7f41d814b640 1 -- start start 2023-05-22T15:47:03.514 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.499+0000 7f41d814b640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f41bc0039b0 0x7f41d00c50f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:47:03.514 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.499+0000 7f41d814b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f41d0150840 0x7f41d0150060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:47:03.514 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.499+0000 7f41d814b640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f41d00c5dc0 0x7f41d00c6230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:47:03.514 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.499+0000 7f41d814b640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41d00c6770 con 0x7f41d0150840 2023-05-22T15:47:03.514 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.499+0000 7f41d814b640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41d00c5630 con 0x7f41bc0039b0 2023-05-22T15:47:03.514 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.499+0000 7f41d814b640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41d00c9b60 con 0x7f41d00c5dc0 2023-05-22T15:47:03.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.499+0000 7f41d56e6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f41d0150840 0x7f41d0150060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:47:03.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.499+0000 7f41d56e6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f41d0150840 0x7f41d0150060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46768/0 (socket says 172.21.15.16:46768) 2023-05-22T15:47:03.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.499+0000 7f41d56e6640 1 -- 172.21.15.16:0/3616647767 learned_addr learned my addr 172.21.15.16:0/3616647767 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:47:03.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.499+0000 7f41d66e8640 1 --2- 172.21.15.16:0/3616647767 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f41d00c5dc0 0x7f41d00c6230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:47:03.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.499+0000 7f41d5ee7640 1 --2- 172.21.15.16:0/3616647767 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f41bc0039b0 0x7f41d00c50f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:47:03.518 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.499+0000 7f41d56e6640 1 -- 172.21.15.16:0/3616647767 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f41d00c5dc0 msgr2=0x7f41d00c6230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:47:03.518 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.499+0000 7f41d56e6640 1 --2- 172.21.15.16:0/3616647767 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f41d00c5dc0 0x7f41d00c6230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:03.518 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.499+0000 7f41d56e6640 1 -- 172.21.15.16:0/3616647767 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f41bc0039b0 msgr2=0x7f41d00c50f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:47:03.519 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.499+0000 7f41d56e6640 1 --2- 172.21.15.16:0/3616647767 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f41bc0039b0 0x7f41d00c50f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:03.519 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.503+0000 7f41d56e6640 1 -- 172.21.15.16:0/3616647767 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41d0160740 con 0x7f41d0150840 2023-05-22T15:47:03.519 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.503+0000 7f41d5ee7640 1 --2- 172.21.15.16:0/3616647767 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f41bc0039b0 0x7f41d00c50f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:47:03.519 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.503+0000 7f41d56e6640 1 --2- 172.21.15.16:0/3616647767 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f41d0150840 0x7f41d0150060 secure :-1 s=READY pgs=2031 cs=0 l=1 rev1=1 crypto rx=0x7f41c000d7a0 tx=0x7f41c000dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:47:03.519 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.503+0000 7f41c67fc640 1 -- 172.21.15.16:0/3616647767 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41c0005270 con 0x7f41d0150840 2023-05-22T15:47:03.519 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.503+0000 7f41c67fc640 1 -- 172.21.15.16:0/3616647767 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41c00053d0 con 0x7f41d0150840 2023-05-22T15:47:03.519 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.503+0000 7f41d814b640 1 -- 172.21.15.16:0/3616647767 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41d015cf50 con 0x7f41d0150840 2023-05-22T15:47:03.519 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.503+0000 7f41c67fc640 1 -- 172.21.15.16:0/3616647767 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41c000b830 con 0x7f41d0150840 2023-05-22T15:47:03.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.503+0000 7f41d814b640 1 -- 172.21.15.16:0/3616647767 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41d01608d0 con 0x7f41d0150840 2023-05-22T15:47:03.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.503+0000 7f41d814b640 1 -- 172.21.15.16:0/3616647767 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41d004dbe0 con 0x7f41d0150840 2023-05-22T15:47:03.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.503+0000 7f41c67fc640 1 -- 172.21.15.16:0/3616647767 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 67) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f41c000b990 con 0x7f41d0150840 2023-05-22T15:47:03.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.503+0000 7f41c67fc640 1 --2- 172.21.15.16:0/3616647767 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f41b40654a0 0x7f41b4067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:47:03.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.503+0000 7f41c67fc640 1 -- 172.21.15.16:0/3616647767 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(160..160 src has 1..160) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f41c0082b30 con 0x7f41d0150840 2023-05-22T15:47:03.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.503+0000 7f41d5ee7640 1 --2- 172.21.15.16:0/3616647767 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f41b40654a0 0x7f41b4067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:47:03.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.503+0000 7f41d5ee7640 1 --2- 172.21.15.16:0/3616647767 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f41b40654a0 0x7f41b4067950 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f41cc0ab000 tx=0x7f41cc0a7070 comp rx=0 tx=0).ready entity=mgr.9396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:47:03.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.507+0000 7f41c67fc640 1 -- 172.21.15.16:0/3616647767 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f41c0050390 con 0x7f41d0150840 2023-05-22T15:47:03.668 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.663+0000 7f41d814b640 1 -- 172.21.15.16:0/3616647767 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f41d00cc5d0 con 0x7f41d0150840 2023-05-22T15:47:03.671 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.667+0000 7f41c67fc640 1 -- 172.21.15.16:0/3616647767 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v160) v1 ==== 74+0+18107 (secure 0 0 0) 0x7f41c00539b0 con 0x7f41d0150840 2023-05-22T15:47:03.672 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:47:03.672 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":160,"fsid":"b2719834-7847-4d98-90a6-f65d57f85c49","created":"2023-05-22T15:26:25.259314+0000","modified":"2023-05-22T15:47:03.008267+0000","last_up_change":"2023-05-22T15:26:36.448524+0000","last_in_change":"2023-05-22T15:26:27.675668+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-05-22T15:26:38.019781+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-05-22T15:26:43.570514+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_metadata","create_time":"2023-05-22T15:47:01.347542+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"160","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_data","create_time":"2023-05-22T15:47:01.970646+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"160","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9e7ff854-b25f-4cbb-a5c2-1799d9c53d2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":37021},{"type":"v1","addr":"172.21.15.16:6825","nonce":37021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":37021},{"type":"v1","addr":"172.21.15.16:6827","nonce":37021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":37021},{"type":"v1","addr":"172.21.15.16:6831","nonce":37021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":37021},{"type":"v1","addr":"172.21.15.16:6829","nonce":37021}]},"public_addr":"172.21.15.16:6825/37021","cluster_addr":"172.21.15.16:6827/37021","heartbeat_back_addr":"172.21.15.16:6831/37021","heartbeat_front_addr":"172.21.15.16:6829/37021","state":["exists","up"]},{"osd":1,"uuid":"4702fbc9-a927-4565-aca5-41dc69cc8264","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":37008},{"type":"v1","addr":"172.21.15.16:6801","nonce":37008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":37008},{"type":"v1","addr":"172.21.15.16:6803","nonce":37008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":37008},{"type":"v1","addr":"172.21.15.16:6807","nonce":37008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":37008},{"type":"v1","addr":"172.21.15.16:6805","nonce":37008}]},"public_addr":"172.21.15.16:6801/37008","cluster_addr":"172.21.15.16:6803/37008","heartbeat_back_addr":"172.21.15.16:6807/37008","heartbeat_front_addr":"172.21.15.16:6805/37008","state":["exists","up"]},{"osd":2,"uuid":"1e7bba46-4eba-47ef-bb81-5e2d371638ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":37009},{"type":"v1","addr":"172.21.15.16:6809","nonce":37009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":37009},{"type":"v1","addr":"172.21.15.16:6811","nonce":37009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":37009},{"type":"v1","addr":"172.21.15.16:6815","nonce":37009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":37009},{"type":"v1","addr":"172.21.15.16:6813","nonce":37009}]},"public_addr":"172.21.15.16:6809/37009","cluster_addr":"172.21.15.16:6811/37009","heartbeat_back_addr":"172.21.15.16:6815/37009","heartbeat_front_addr":"172.21.15.16:6813/37009","state":["exists","up"]},{"osd":3,"uuid":"383cacc0-5a88-4904-990e-498ca59ef48a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":37020},{"type":"v1","addr":"172.21.15.16:6817","nonce":37020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":37020},{"type":"v1","addr":"172.21.15.16:6819","nonce":37020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":37020},{"type":"v1","addr":"172.21.15.16:6823","nonce":37020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":37020},{"type":"v1","addr":"172.21.15.16:6821","nonce":37020}]},"public_addr":"172.21.15.16:6817/37020","cluster_addr":"172.21.15.16:6819/37020","heartbeat_back_addr":"172.21.15.16:6823/37020","heartbeat_front_addr":"172.21.15.16:6821/37020","state":["exists","up"]},{"osd":4,"uuid":"970500ef-95b7-42c1-b080-eed4fae7013d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":37353},{"type":"v1","addr":"172.21.15.62:6817","nonce":37353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6819","nonce":37353},{"type":"v1","addr":"172.21.15.62:6822","nonce":37353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37353},{"type":"v1","addr":"172.21.15.62:6831","nonce":37353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37353},{"type":"v1","addr":"172.21.15.62:6826","nonce":37353}]},"public_addr":"172.21.15.62:6817/37353","cluster_addr":"172.21.15.62:6822/37353","heartbeat_back_addr":"172.21.15.62:6831/37353","heartbeat_front_addr":"172.21.15.62:6826/37353","state":["exists","up"]},{"osd":5,"uuid":"022d67db-d4d8-445b-b1b9-8b446a00f5c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":37352},{"type":"v1","addr":"172.21.15.62:6801","nonce":37352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":37352},{"type":"v1","addr":"172.21.15.62:6803","nonce":37352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":37352},{"type":"v1","addr":"172.21.15.62:6807","nonce":37352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":37352},{"type":"v1","addr":"172.21.15.62:6805","nonce":37352}]},"public_addr":"172.21.15.62:6801/37352","cluster_addr":"172.21.15.62:6803/37352","heartbeat_back_addr":"172.21.15.62:6807/37352","heartbeat_front_addr":"172.21.15.62:6805/37352","state":["exists","up"]},{"osd":6,"uuid":"42bd372b-6688-4a1b-8407-e16e8c266e33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":37354},{"type":"v1","addr":"172.21.15.62:6820","nonce":37354}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":37354},{"type":"v1","addr":"172.21.15.62:6823","nonce":37354}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6829","nonce":37354},{"type":"v1","addr":"172.21.15.62:6830","nonce":37354}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6825","nonce":37354},{"type":"v1","addr":"172.21.15.62:6827","nonce":37354}]},"public_addr":"172.21.15.62:6820/37354","cluster_addr":"172.21.15.62:6823/37354","heartbeat_back_addr":"172.21.15.62:6830/37354","heartbeat_front_addr":"172.21.15.62:6827/37354","state":["exists","up"]},{"osd":7,"uuid":"ac70b2a3-f41b-4bf2-b227-7f2cbb55426d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37355},{"type":"v1","addr":"172.21.15.62:6809","nonce":37355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37355},{"type":"v1","addr":"172.21.15.62:6811","nonce":37355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":37355},{"type":"v1","addr":"172.21.15.62:6815","nonce":37355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":37355},{"type":"v1","addr":"172.21.15.62:6813","nonce":37355}]},"public_addr":"172.21.15.62:6809/37355","cluster_addr":"172.21.15.62:6811/37355","heartbeat_back_addr":"172.21.15.62:6815/37355","heartbeat_front_addr":"172.21.15.62:6813/37355","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.368896+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.107273+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.276888+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.287502+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.124791+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.251937+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.255679+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.232797+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":17,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-05-22T15:47:03.674 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.667+0000 7f41d814b640 1 -- 172.21.15.16:0/3616647767 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f41b40654a0 msgr2=0x7f41b4067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:47:03.675 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.667+0000 7f41d814b640 1 --2- 172.21.15.16:0/3616647767 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f41b40654a0 0x7f41b4067950 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f41cc0ab000 tx=0x7f41cc0a7070 comp rx=0 tx=0).stop 2023-05-22T15:47:03.675 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.667+0000 7f41d814b640 1 -- 172.21.15.16:0/3616647767 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f41d0150840 msgr2=0x7f41d0150060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:47:03.675 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.667+0000 7f41d814b640 1 --2- 172.21.15.16:0/3616647767 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f41d0150840 0x7f41d0150060 secure :-1 s=READY pgs=2031 cs=0 l=1 rev1=1 crypto rx=0x7f41c000d7a0 tx=0x7f41c000dc60 comp rx=0 tx=0).stop 2023-05-22T15:47:03.675 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.667+0000 7f41d814b640 1 -- 172.21.15.16:0/3616647767 shutdown_connections 2023-05-22T15:47:03.675 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.667+0000 7f41d814b640 1 --2- 172.21.15.16:0/3616647767 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f41b40654a0 0x7f41b4067950 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:03.675 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.667+0000 7f41d814b640 1 --2- 172.21.15.16:0/3616647767 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f41d00c5dc0 0x7f41d00c6230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:03.676 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.667+0000 7f41d814b640 1 --2- 172.21.15.16:0/3616647767 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f41d0150840 0x7f41d0150060 unknown :-1 s=CLOSED pgs=2031 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:03.676 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.667+0000 7f41d814b640 1 --2- 172.21.15.16:0/3616647767 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f41bc0039b0 0x7f41d00c50f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:03.676 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.667+0000 7f41d814b640 1 -- 172.21.15.16:0/3616647767 >> 172.21.15.16:0/3616647767 conn(0x7f41d0069720 msgr2=0x7f41d0069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:47:03.676 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.667+0000 7f41d814b640 1 -- 172.21.15.16:0/3616647767 shutdown_connections 2023-05-22T15:47:03.676 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.667+0000 7f41d814b640 1 -- 172.21.15.16:0/3616647767 wait complete. 2023-05-22T15:47:03.691 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-05-22T15:47:03.823 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.819+0000 7f83b415c640 1 Processor -- start 2023-05-22T15:47:03.824 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.819+0000 7f83b415c640 1 -- start start 2023-05-22T15:47:03.825 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.819+0000 7f83b415c640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f83ac070c70 0x7f83ac071530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:47:03.825 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.819+0000 7f83b415c640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f83ac15c580 0x7f83ac06fba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:47:03.825 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.819+0000 7f83b415c640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83ac15d520 con 0x7f83ac06b6c0 2023-05-22T15:47:03.825 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.819+0000 7f83b415c640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83ac0c8b10 con 0x7f83ac15c580 2023-05-22T15:47:03.825 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.819+0000 7f83b415c640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83ac15c9d0 con 0x7f83ac070c70 2023-05-22T15:47:03.826 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.819+0000 7f83b16f7640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f83ac15c580 0x7f83ac06fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:47:03.826 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.819+0000 7f83b16f7640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f83ac15c580 0x7f83ac06fba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46776/0 (socket says 172.21.15.16:46776) 2023-05-22T15:47:03.826 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.819+0000 7f83b16f7640 1 -- 172.21.15.16:0/65238822 learned_addr learned my addr 172.21.15.16:0/65238822 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:47:03.826 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.819+0000 7f83b1ef8640 1 --2- 172.21.15.16:0/65238822 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f83ac070c70 0x7f83ac071530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:47:03.826 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.819+0000 7f83b16f7640 1 -- 172.21.15.16:0/65238822 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f83ac070c70 msgr2=0x7f83ac071530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:47:03.826 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.819+0000 7f83b16f7640 1 --2- 172.21.15.16:0/65238822 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f83ac070c70 0x7f83ac071530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:03.826 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.819+0000 7f83b16f7640 1 -- 172.21.15.16:0/65238822 >> v1:172.21.15.62:6789/0 conn(0x7f83ac06b6c0 legacy=0x7f83ac0cca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:47:03.827 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.819+0000 7f83b16f7640 1 -- 172.21.15.16:0/65238822 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83ac0700e0 con 0x7f83ac15c580 2023-05-22T15:47:03.827 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.823+0000 7f83b16f7640 1 --2- 172.21.15.16:0/65238822 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83ac15c580 0x7f83ac06fba0 secure :-1 s=READY pgs=2032 cs=0 l=1 rev1=1 crypto rx=0x7f839c009870 tx=0x7f839c01c440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11cb7c89458cb027 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:47:03.827 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.823+0000 7f83b0ef6640 1 -- 172.21.15.16:0/65238822 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f839c0ab600 con 0x7f83ac15c580 2023-05-22T15:47:03.827 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.823+0000 7f83b0ef6640 1 -- 172.21.15.16:0/65238822 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f839c0ab760 con 0x7f83ac15c580 2023-05-22T15:47:03.828 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.823+0000 7f83b0ef6640 1 -- 172.21.15.16:0/65238822 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f839c0abaf0 con 0x7f83ac15c580 2023-05-22T15:47:03.828 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.823+0000 7f83b415c640 1 -- 172.21.15.16:0/65238822 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83ac15c580 msgr2=0x7f83ac06fba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:47:03.829 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.823+0000 7f83b415c640 1 --2- 172.21.15.16:0/65238822 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83ac15c580 0x7f83ac06fba0 secure :-1 s=READY pgs=2032 cs=0 l=1 rev1=1 crypto rx=0x7f839c009870 tx=0x7f839c01c440 comp rx=0 tx=0).stop 2023-05-22T15:47:03.829 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.823+0000 7f83b415c640 1 -- 172.21.15.16:0/65238822 shutdown_connections 2023-05-22T15:47:03.829 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.823+0000 7f83b415c640 1 --2- 172.21.15.16:0/65238822 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83ac15c580 0x7f83ac06fba0 unknown :-1 s=CLOSED pgs=2032 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:03.829 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.823+0000 7f83b415c640 1 --2- 172.21.15.16:0/65238822 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f83ac070c70 0x7f83ac071530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:03.829 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.823+0000 7f83b415c640 1 -- 172.21.15.16:0/65238822 >> 172.21.15.16:0/65238822 conn(0x7f83ac069720 msgr2=0x7f83ac069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:47:03.829 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.823+0000 7f83b415c640 1 -- 172.21.15.16:0/65238822 shutdown_connections 2023-05-22T15:47:03.829 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.823+0000 7f83b415c640 1 -- 172.21.15.16:0/65238822 wait complete. 2023-05-22T15:47:03.830 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.823+0000 7f83b415c640 1 Processor -- start 2023-05-22T15:47:03.830 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.823+0000 7f83b415c640 1 -- start start 2023-05-22T15:47:03.831 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.823+0000 7f83b415c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83ac06b6c0 0x7f83ac154920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:47:03.831 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.823+0000 7f83b415c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f83ac070c70 0x7f83ac15b930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:47:03.831 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.823+0000 7f83b415c640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f83ac15c580 0x7f83ac159810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:47:03.831 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.823+0000 7f83b415c640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83ac15ac00 con 0x7f83ac06b6c0 2023-05-22T15:47:03.831 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.823+0000 7f83b415c640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83ac159d50 con 0x7f83ac070c70 2023-05-22T15:47:03.831 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.823+0000 7f83b415c640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83ac159ff0 con 0x7f83ac15c580 2023-05-22T15:47:03.831 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.823+0000 7f83b1ef8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83ac06b6c0 0x7f83ac154920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:47:03.832 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.823+0000 7f83b1ef8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83ac06b6c0 0x7f83ac154920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46788/0 (socket says 172.21.15.16:46788) 2023-05-22T15:47:03.832 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.823+0000 7f83b1ef8640 1 -- 172.21.15.16:0/679649637 learned_addr learned my addr 172.21.15.16:0/679649637 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:47:03.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.827+0000 7f83b26f9640 1 --2- 172.21.15.16:0/679649637 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f83ac15c580 0x7f83ac159810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:47:03.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.827+0000 7f83b16f7640 1 --2- 172.21.15.16:0/679649637 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f83ac070c70 0x7f83ac15b930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:47:03.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.827+0000 7f83b1ef8640 1 -- 172.21.15.16:0/679649637 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f83ac15c580 msgr2=0x7f83ac159810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:47:03.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.827+0000 7f83b1ef8640 1 --2- 172.21.15.16:0/679649637 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f83ac15c580 0x7f83ac159810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:03.834 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.827+0000 7f83b1ef8640 1 -- 172.21.15.16:0/679649637 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f83ac070c70 msgr2=0x7f83ac15b930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:47:03.834 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.827+0000 7f83b1ef8640 1 --2- 172.21.15.16:0/679649637 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f83ac070c70 0x7f83ac15b930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:03.834 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.827+0000 7f83b1ef8640 1 -- 172.21.15.16:0/679649637 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f839c0094d0 con 0x7f83ac06b6c0 2023-05-22T15:47:03.834 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.827+0000 7f83b16f7640 1 --2- 172.21.15.16:0/679649637 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f83ac070c70 0x7f83ac15b930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:47:03.834 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.827+0000 7f83b1ef8640 1 --2- 172.21.15.16:0/679649637 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83ac06b6c0 0x7f83ac154920 secure :-1 s=READY pgs=2033 cs=0 l=1 rev1=1 crypto rx=0x7f83a800d800 tx=0x7f83a800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:47:03.834 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.827+0000 7f83a27fc640 1 -- 172.21.15.16:0/679649637 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83a800b6c0 con 0x7f83ac06b6c0 2023-05-22T15:47:03.834 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.827+0000 7f83a27fc640 1 -- 172.21.15.16:0/679649637 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83a800b820 con 0x7f83ac06b6c0 2023-05-22T15:47:03.835 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.827+0000 7f83b415c640 1 -- 172.21.15.16:0/679649637 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83ac0cc510 con 0x7f83ac06b6c0 2023-05-22T15:47:03.835 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.827+0000 7f83a27fc640 1 -- 172.21.15.16:0/679649637 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83a800bb90 con 0x7f83ac06b6c0 2023-05-22T15:47:03.835 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.827+0000 7f83b415c640 1 -- 172.21.15.16:0/679649637 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83ac15d240 con 0x7f83ac06b6c0 2023-05-22T15:47:03.835 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.827+0000 7f83b415c640 1 -- 172.21.15.16:0/679649637 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83ac071040 con 0x7f83ac06b6c0 2023-05-22T15:47:03.835 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.827+0000 7f83a27fc640 1 -- 172.21.15.16:0/679649637 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 67) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f83a800bcf0 con 0x7f83ac06b6c0 2023-05-22T15:47:03.836 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.827+0000 7f83a27fc640 1 --2- 172.21.15.16:0/679649637 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f83900654a0 0x7f8390067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:47:03.836 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.827+0000 7f83a27fc640 1 -- 172.21.15.16:0/679649637 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(160..160 src has 1..160) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f83a8083130 con 0x7f83ac06b6c0 2023-05-22T15:47:03.836 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.827+0000 7f83b16f7640 1 --2- 172.21.15.16:0/679649637 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f83900654a0 0x7f8390067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:47:03.837 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.827+0000 7f83b16f7640 1 --2- 172.21.15.16:0/679649637 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f83900654a0 0x7f8390067950 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f839c0af000 tx=0x7f839c008040 comp rx=0 tx=0).ready entity=mgr.9396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:47:03.837 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:03.831+0000 7f83a27fc640 1 -- 172.21.15.16:0/679649637 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f83a80544d0 con 0x7f83ac06b6c0 2023-05-22T15:47:04.025 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:04.019+0000 7f83b415c640 1 -- 172.21.15.16:0/679649637 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f83ac04dbe0 con 0x7f83ac06b6c0 2023-05-22T15:47:04.281 INFO:tasks.ceph.mds.c.smithi016.stderr:2023-05-22T15:47:04.279+0000 7fabb3455640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-05-22T15:47:05.036 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:05.031+0000 7f83a27fc640 1 -- 172.21.15.16:0/679649637 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v65) v1 ==== 118+0+0 (secure 0 0 0) 0x7f83a80540c0 con 0x7f83ac06b6c0 2023-05-22T15:47:05.040 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:05.035+0000 7f83b415c640 1 -- 172.21.15.16:0/679649637 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f83900654a0 msgr2=0x7f8390067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:47:05.040 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:05.035+0000 7f83b415c640 1 --2- 172.21.15.16:0/679649637 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f83900654a0 0x7f8390067950 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f839c0af000 tx=0x7f839c008040 comp rx=0 tx=0).stop 2023-05-22T15:47:05.040 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:05.035+0000 7f83b415c640 1 -- 172.21.15.16:0/679649637 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83ac06b6c0 msgr2=0x7f83ac154920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:47:05.041 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:05.035+0000 7f83b415c640 1 --2- 172.21.15.16:0/679649637 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83ac06b6c0 0x7f83ac154920 secure :-1 s=READY pgs=2033 cs=0 l=1 rev1=1 crypto rx=0x7f83a800d800 tx=0x7f83a800dcc0 comp rx=0 tx=0).stop 2023-05-22T15:47:05.041 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:05.035+0000 7f83b415c640 1 -- 172.21.15.16:0/679649637 shutdown_connections 2023-05-22T15:47:05.041 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:05.035+0000 7f83b415c640 1 --2- 172.21.15.16:0/679649637 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f83900654a0 0x7f8390067950 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:05.041 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:05.035+0000 7f83b415c640 1 --2- 172.21.15.16:0/679649637 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f83ac15c580 0x7f83ac159810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:05.042 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:05.035+0000 7f83b415c640 1 --2- 172.21.15.16:0/679649637 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f83ac070c70 0x7f83ac15b930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:05.042 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:05.035+0000 7f83b415c640 1 --2- 172.21.15.16:0/679649637 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83ac06b6c0 0x7f83ac154920 unknown :-1 s=CLOSED pgs=2033 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:05.042 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:05.035+0000 7f83b415c640 1 -- 172.21.15.16:0/679649637 >> 172.21.15.16:0/679649637 conn(0x7f83ac069720 msgr2=0x7f83ac067680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:47:05.043 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:05.035+0000 7f83b415c640 1 -- 172.21.15.16:0/679649637 shutdown_connections 2023-05-22T15:47:05.043 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:05.035+0000 7f83b415c640 1 -- 172.21.15.16:0/679649637 wait complete. 2023-05-22T15:47:05.061 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2023-05-22T15:47:05.062 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-05-22T15:47:05.195 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:05.191+0000 7f238598b640 1 Processor -- start 2023-05-22T15:47:05.196 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:05.191+0000 7f238598b640 1 -- start start 2023-05-22T15:47:05.197 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:05.191+0000 7f238598b640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f238006b6c0 0x7f23800cb4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:47:05.197 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:05.191+0000 7f238598b640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f238006ba90 0x7f2380158fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:47:05.197 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:05.191+0000 7f238598b640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2380159510 0x7f23800cc740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:47:05.197 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:05.191+0000 7f238598b640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2380071760 con 0x7f2380159510 2023-05-22T15:47:05.198 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:05.191+0000 7f238598b640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23800ca7d0 con 0x7f238006ba90 2023-05-22T15:47:05.198 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:05.191+0000 7f238598b640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2380159f70 con 0x7f238006b6c0 2023-05-22T15:47:05.198 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:05.191+0000 7f237f7fe640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2380159510 0x7f23800cc740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:47:05.198 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:05.191+0000 7f237f7fe640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2380159510 0x7f23800cc740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46794/0 (socket says 172.21.15.16:46794) 2023-05-22T15:47:05.198 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:05.191+0000 7f237f7fe640 1 -- 172.21.15.16:0/97450717 learned_addr learned my addr 172.21.15.16:0/97450717 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:47:05.198 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:05.191+0000 7f237effd640 1 --2- 172.21.15.16:0/97450717 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f238006b6c0 0x7f23800cb4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:47:05.198 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:05.191+0000 7f237e7fc640 1 --2- 172.21.15.16:0/97450717 >> v2:172.21.15.62:3300/0 conn(0x7f238006ba90 0x7f2380158fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:47:05.199 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:05.191+0000 7f237f7fe640 1 -- 172.21.15.16:0/97450717 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f238006b6c0 msgr2=0x7f23800cb4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:47:05.199 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:05.191+0000 7f237f7fe640 1 --2- 172.21.15.16:0/97450717 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f238006b6c0 0x7f23800cb4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:05.200 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:05.191+0000 7f237f7fe640 1 -- 172.21.15.16:0/97450717 >> v2:172.21.15.62:3300/0 conn(0x7f238006ba90 msgr2=0x7f2380158fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:47:05.200 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:05.191+0000 7f237f7fe640 1 --2- 172.21.15.16:0/97450717 >> v2:172.21.15.62:3300/0 conn(0x7f238006ba90 0x7f2380158fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:05.200 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:05.191+0000 7f237f7fe640 1 -- 172.21.15.16:0/97450717 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23800cb030 con 0x7f2380159510 2023-05-22T15:47:05.202 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:05.195+0000 7f237f7fe640 1 --2- 172.21.15.16:0/97450717 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2380159510 0x7f23800cc740 secure :-1 s=READY pgs=2034 cs=0 l=1 rev1=1 crypto rx=0x7f23800cbb60 tx=0x7f237801c6e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb6f677c302b6cf1 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:47:05.202 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:05.195+0000 7f237dffb640 1 -- 172.21.15.16:0/97450717 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23780a5560 con 0x7f2380159510 2023-05-22T15:47:05.203 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:05.195+0000 7f237dffb640 1 -- 172.21.15.16:0/97450717 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23780a56c0 con 0x7f2380159510 2023-05-22T15:47:05.203 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:05.195+0000 7f237dffb640 1 -- 172.21.15.16:0/97450717 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23780a5a50 con 0x7f2380159510 2023-05-22T15:47:05.203 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:05.195+0000 7f238598b640 1 -- 172.21.15.16:0/97450717 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2380159510 msgr2=0x7f23800cc740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:47:05.203 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:05.195+0000 7f238598b640 1 --2- 172.21.15.16:0/97450717 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2380159510 0x7f23800cc740 secure :-1 s=READY pgs=2034 cs=0 l=1 rev1=1 crypto rx=0x7f23800cbb60 tx=0x7f237801c6e0 comp rx=0 tx=0).stop 2023-05-22T15:47:05.203 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:05.195+0000 7f238598b640 1 -- 172.21.15.16:0/97450717 shutdown_connections 2023-05-22T15:47:05.203 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:05.195+0000 7f238598b640 1 --2- 172.21.15.16:0/97450717 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2380159510 0x7f23800cc740 unknown :-1 s=CLOSED pgs=2034 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:05.204 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:05.195+0000 7f238598b640 1 --2- 172.21.15.16:0/97450717 >> v2:172.21.15.62:3300/0 conn(0x7f238006ba90 0x7f2380158fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:05.204 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:05.195+0000 7f238598b640 1 --2- 172.21.15.16:0/97450717 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f238006b6c0 0x7f23800cb4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:05.204 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:05.195+0000 7f238598b640 1 -- 172.21.15.16:0/97450717 >> 172.21.15.16:0/97450717 conn(0x7f2380069720 msgr2=0x7f2380069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:47:05.204 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:05.195+0000 7f238598b640 1 -- 172.21.15.16:0/97450717 shutdown_connections 2023-05-22T15:47:05.204 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:05.195+0000 7f238598b640 1 -- 172.21.15.16:0/97450717 wait complete. 2023-05-22T15:47:05.204 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:05.195+0000 7f238598b640 1 Processor -- start 2023-05-22T15:47:05.204 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:05.195+0000 7f238598b640 1 -- start start 2023-05-22T15:47:05.207 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:05.195+0000 7f238598b640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f238006b6c0 0x7f23800b3830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:47:05.207 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:05.199+0000 7f238598b640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f238006ba90 0x7f23801539d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:47:05.207 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:05.199+0000 7f238598b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2380159510 0x7f238014bb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:47:05.207 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:05.199+0000 7f238598b640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f238014c7b0 con 0x7f2380159510 2023-05-22T15:47:05.208 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:05.199+0000 7f238598b640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f238014e090 con 0x7f238006ba90 2023-05-22T15:47:05.208 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:05.199+0000 7f238598b640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f238014e330 con 0x7f238006b6c0 2023-05-22T15:47:05.208 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:05.199+0000 7f237f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2380159510 0x7f238014bb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:47:05.208 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:05.199+0000 7f237f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2380159510 0x7f238014bb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46796/0 (socket says 172.21.15.16:46796) 2023-05-22T15:47:05.208 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:05.199+0000 7f237f7fe640 1 -- 172.21.15.16:0/2326415625 learned_addr learned my addr 172.21.15.16:0/2326415625 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:47:05.208 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:05.199+0000 7f237effd640 1 --2- 172.21.15.16:0/2326415625 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f238006b6c0 0x7f23800b3830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:47:05.208 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:05.199+0000 7f237e7fc640 1 --2- 172.21.15.16:0/2326415625 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f238006ba90 0x7f23801539d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:47:05.209 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:05.199+0000 7f237f7fe640 1 -- 172.21.15.16:0/2326415625 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f238006b6c0 msgr2=0x7f23800b3830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:47:05.209 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:05.199+0000 7f237f7fe640 1 --2- 172.21.15.16:0/2326415625 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f238006b6c0 0x7f23800b3830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:05.209 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:05.199+0000 7f237f7fe640 1 -- 172.21.15.16:0/2326415625 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f238006ba90 msgr2=0x7f23801539d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:47:05.209 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:05.199+0000 7f237f7fe640 1 --2- 172.21.15.16:0/2326415625 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f238006ba90 0x7f23801539d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:05.210 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:05.199+0000 7f237f7fe640 1 -- 172.21.15.16:0/2326415625 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2378009b30 con 0x7f2380159510 2023-05-22T15:47:05.210 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:05.199+0000 7f237e7fc640 1 --2- 172.21.15.16:0/2326415625 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f238006ba90 0x7f23801539d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T15:47:05.210 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:05.199+0000 7f237effd640 1 --2- 172.21.15.16:0/2326415625 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f238006b6c0 0x7f23800b3830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T15:47:05.210 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:05.199+0000 7f237f7fe640 1 --2- 172.21.15.16:0/2326415625 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2380159510 0x7f238014bb50 secure :-1 s=READY pgs=2035 cs=0 l=1 rev1=1 crypto rx=0x7f23780af040 tx=0x7f2378019480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:47:05.211 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:05.199+0000 7f2367fff640 1 -- 172.21.15.16:0/2326415625 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23780a5560 con 0x7f2380159510 2023-05-22T15:47:05.211 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:05.199+0000 7f2367fff640 1 -- 172.21.15.16:0/2326415625 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23780b8640 con 0x7f2380159510 2023-05-22T15:47:05.211 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:05.199+0000 7f2367fff640 1 -- 172.21.15.16:0/2326415625 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23780b89b0 con 0x7f2380159510 2023-05-22T15:47:05.212 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:05.199+0000 7f238598b640 1 -- 172.21.15.16:0/2326415625 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2380159990 con 0x7f2380159510 2023-05-22T15:47:05.212 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:05.199+0000 7f238598b640 1 -- 172.21.15.16:0/2326415625 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23800cacc0 con 0x7f2380159510 2023-05-22T15:47:05.213 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:05.199+0000 7f238598b640 1 -- 172.21.15.16:0/2326415625 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f238006be60 con 0x7f2380159510 2023-05-22T15:47:05.213 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:05.199+0000 7f2367fff640 1 -- 172.21.15.16:0/2326415625 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 67) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f23780b8b10 con 0x7f2380159510 2023-05-22T15:47:05.213 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:05.199+0000 7f2367fff640 1 --2- 172.21.15.16:0/2326415625 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f236c0654a0 0x7f236c067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:47:05.213 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:05.199+0000 7f2367fff640 1 -- 172.21.15.16:0/2326415625 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(161..161 src has 1..161) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f237811dca0 con 0x7f2380159510 2023-05-22T15:47:05.213 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:05.199+0000 7f237effd640 1 --2- 172.21.15.16:0/2326415625 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f236c0654a0 0x7f236c067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:47:05.213 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:05.203+0000 7f237effd640 1 --2- 172.21.15.16:0/2326415625 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f236c0654a0 0x7f236c067950 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f2370004660 tx=0x7f2370008040 comp rx=0 tx=0).ready entity=mgr.9396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:47:05.215 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:05.203+0000 7f2367fff640 1 -- 172.21.15.16:0/2326415625 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f23780eb6a0 con 0x7f2380159510 2023-05-22T15:47:05.398 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:05.391+0000 7f238598b640 1 -- 172.21.15.16:0/2326415625 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f238014e850 con 0x7f2380159510 2023-05-22T15:47:06.040 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.035+0000 7f2367fff640 1 -- 172.21.15.16:0/2326415625 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v66) v1 ==== 122+0+0 (secure 0 0 0) 0x7f238014e850 con 0x7f2380159510 2023-05-22T15:47:06.044 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.039+0000 7f238598b640 1 -- 172.21.15.16:0/2326415625 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f236c0654a0 msgr2=0x7f236c067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:47:06.044 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.039+0000 7f238598b640 1 --2- 172.21.15.16:0/2326415625 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f236c0654a0 0x7f236c067950 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f2370004660 tx=0x7f2370008040 comp rx=0 tx=0).stop 2023-05-22T15:47:06.044 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.039+0000 7f238598b640 1 -- 172.21.15.16:0/2326415625 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2380159510 msgr2=0x7f238014bb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:47:06.044 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.039+0000 7f238598b640 1 --2- 172.21.15.16:0/2326415625 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2380159510 0x7f238014bb50 secure :-1 s=READY pgs=2035 cs=0 l=1 rev1=1 crypto rx=0x7f23780af040 tx=0x7f2378019480 comp rx=0 tx=0).stop 2023-05-22T15:47:06.044 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.039+0000 7f238598b640 1 -- 172.21.15.16:0/2326415625 shutdown_connections 2023-05-22T15:47:06.044 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.039+0000 7f238598b640 1 --2- 172.21.15.16:0/2326415625 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2380159510 0x7f238014bb50 unknown :-1 s=CLOSED pgs=2035 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:06.045 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.039+0000 7f238598b640 1 --2- 172.21.15.16:0/2326415625 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f238006ba90 0x7f23801539d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:06.045 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.039+0000 7f238598b640 1 --2- 172.21.15.16:0/2326415625 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f236c0654a0 0x7f236c067950 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:06.045 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.039+0000 7f238598b640 1 --2- 172.21.15.16:0/2326415625 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f238006b6c0 0x7f23800b3830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:06.045 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.039+0000 7f238598b640 1 -- 172.21.15.16:0/2326415625 >> 172.21.15.16:0/2326415625 conn(0x7f2380069720 msgr2=0x7f238006fb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:47:06.045 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.039+0000 7f238598b640 1 -- 172.21.15.16:0/2326415625 shutdown_connections 2023-05-22T15:47:06.045 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.039+0000 7f238598b640 1 -- 172.21.15.16:0/2326415625 wait complete. 2023-05-22T15:47:06.067 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2023-05-22T15:47:06.067 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-05-22T15:47:06.191 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.183+0000 7f07c35cb640 1 Processor -- start 2023-05-22T15:47:06.192 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.183+0000 7f07c35cb640 1 -- start start 2023-05-22T15:47:06.193 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.187+0000 7f07c35cb640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f07bc0ccab0 0x7f07bc15fe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:47:06.194 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.187+0000 7f07c35cb640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f07bc0d0410 0x7f07bc0d0830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:47:06.195 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.187+0000 7f07c35cb640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07bc0ccef0 con 0x7f07bc0b7dd0 2023-05-22T15:47:06.195 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.187+0000 7f07c35cb640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07bc0d3360 con 0x7f07bc0ccab0 2023-05-22T15:47:06.195 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.187+0000 7f07c35cb640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07bc0cd170 con 0x7f07bc0d0410 2023-05-22T15:47:06.195 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.187+0000 7f07c0b66640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f07bc0ccab0 0x7f07bc15fe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:47:06.195 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.187+0000 7f07c1367640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f07bc0b7dd0 0x7f07bc15f700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:44858/0 (socket says 172.21.15.16:44858) 2023-05-22T15:47:06.196 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.187+0000 7f07c1367640 1 -- 172.21.15.16:0/2739853325 learned_addr learned my addr 172.21.15.16:0/2739853325 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:47:06.196 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.187+0000 7f07c0b66640 1 -- 172.21.15.16:0/2739853325 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f07bc0d0410 msgr2=0x7f07bc0d0830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:47:06.196 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.187+0000 7f07c0b66640 1 --2- 172.21.15.16:0/2739853325 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f07bc0d0410 0x7f07bc0d0830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:06.196 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.187+0000 7f07c0b66640 1 -- 172.21.15.16:0/2739853325 >> v1:172.21.15.16:6789/0 conn(0x7f07bc0b7dd0 legacy=0x7f07bc15f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:47:06.196 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.187+0000 7f07c0b66640 1 -- 172.21.15.16:0/2739853325 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07bc0d2ac0 con 0x7f07bc0ccab0 2023-05-22T15:47:06.196 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.187+0000 7f07affff640 1 -- 172.21.15.16:0/2739853325 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2729535613 0 0) 0x7f07bc0ccef0 con 0x7f07bc0b7dd0 2023-05-22T15:47:06.199 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.187+0000 7f07c0b66640 1 --2- 172.21.15.16:0/2739853325 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f07bc0ccab0 0x7f07bc15fe10 secure :-1 s=READY pgs=2037 cs=0 l=1 rev1=1 crypto rx=0x7f07b80097c0 tx=0x7f07b801c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db5c6450300da245 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:47:06.199 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.187+0000 7f07affff640 1 -- 172.21.15.16:0/2739853325 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07b80ab650 con 0x7f07bc0ccab0 2023-05-22T15:47:06.199 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.187+0000 7f07affff640 1 -- 172.21.15.16:0/2739853325 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07b80ab7b0 con 0x7f07bc0ccab0 2023-05-22T15:47:06.199 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.187+0000 7f07affff640 1 -- 172.21.15.16:0/2739853325 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07b80abb40 con 0x7f07bc0ccab0 2023-05-22T15:47:06.199 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.191+0000 7f07c35cb640 1 -- 172.21.15.16:0/2739853325 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f07bc0ccab0 msgr2=0x7f07bc15fe10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:47:06.200 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.191+0000 7f07c35cb640 1 --2- 172.21.15.16:0/2739853325 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f07bc0ccab0 0x7f07bc15fe10 secure :-1 s=READY pgs=2037 cs=0 l=1 rev1=1 crypto rx=0x7f07b80097c0 tx=0x7f07b801c450 comp rx=0 tx=0).stop 2023-05-22T15:47:06.200 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.191+0000 7f07c35cb640 1 -- 172.21.15.16:0/2739853325 shutdown_connections 2023-05-22T15:47:06.200 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.191+0000 7f07c35cb640 1 --2- 172.21.15.16:0/2739853325 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f07bc0d0410 0x7f07bc0d0830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:06.200 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.191+0000 7f07c35cb640 1 --2- 172.21.15.16:0/2739853325 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f07bc0ccab0 0x7f07bc15fe10 unknown :-1 s=CLOSED pgs=2037 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:06.200 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.191+0000 7f07c35cb640 1 -- 172.21.15.16:0/2739853325 >> 172.21.15.16:0/2739853325 conn(0x7f07bc0655c0 msgr2=0x7f07bc070e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:47:06.200 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.191+0000 7f07c35cb640 1 -- 172.21.15.16:0/2739853325 shutdown_connections 2023-05-22T15:47:06.200 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.191+0000 7f07c35cb640 1 -- 172.21.15.16:0/2739853325 wait complete. 2023-05-22T15:47:06.201 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.191+0000 7f07c35cb640 1 Processor -- start 2023-05-22T15:47:06.201 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.191+0000 7f07c35cb640 1 -- start start 2023-05-22T15:47:06.201 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.191+0000 7f07c35cb640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f07bc0b7dd0 0x7f07bc0c5f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:47:06.201 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.191+0000 7f07c35cb640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f07bc0ccab0 0x7f07bc0b2f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:47:06.201 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.191+0000 7f07c35cb640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f07bc0d0410 0x7f07bc0b34d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:47:06.201 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.191+0000 7f07c35cb640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07bc0c8980 con 0x7f07bc0ccab0 2023-05-22T15:47:06.201 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.191+0000 7f07c35cb640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07bc151940 con 0x7f07bc0d0410 2023-05-22T15:47:06.202 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.191+0000 7f07c35cb640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07bc0c68e0 con 0x7f07bc0b7dd0 2023-05-22T15:47:06.202 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.191+0000 7f07c0b66640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f07bc0ccab0 0x7f07bc0b2f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:47:06.202 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.191+0000 7f07c0b66640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f07bc0ccab0 0x7f07bc0b2f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46804/0 (socket says 172.21.15.16:46804) 2023-05-22T15:47:06.202 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.191+0000 7f07c0b66640 1 -- 172.21.15.16:0/2393415265 learned_addr learned my addr 172.21.15.16:0/2393415265 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:47:06.202 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.191+0000 7f07c1367640 1 --2- 172.21.15.16:0/2393415265 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f07bc0b7dd0 0x7f07bc0c5f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:47:06.202 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.191+0000 7f07c1b68640 1 --2- 172.21.15.16:0/2393415265 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f07bc0d0410 0x7f07bc0b34d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:47:06.202 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.191+0000 7f07c0b66640 1 -- 172.21.15.16:0/2393415265 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f07bc0b7dd0 msgr2=0x7f07bc0c5f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:47:06.202 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.191+0000 7f07c0b66640 1 --2- 172.21.15.16:0/2393415265 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f07bc0b7dd0 0x7f07bc0c5f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:06.203 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.191+0000 7f07c0b66640 1 -- 172.21.15.16:0/2393415265 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f07bc0d0410 msgr2=0x7f07bc0b34d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:47:06.203 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.191+0000 7f07c0b66640 1 --2- 172.21.15.16:0/2393415265 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f07bc0d0410 0x7f07bc0b34d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:06.203 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.191+0000 7f07c0b66640 1 -- 172.21.15.16:0/2393415265 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07b8009420 con 0x7f07bc0ccab0 2023-05-22T15:47:06.203 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.191+0000 7f07c1367640 1 --2- 172.21.15.16:0/2393415265 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f07bc0b7dd0 0x7f07bc0c5f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:47:06.203 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.191+0000 7f07c1b68640 1 --2- 172.21.15.16:0/2393415265 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f07bc0d0410 0x7f07bc0b34d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:47:06.203 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.191+0000 7f07c0b66640 1 --2- 172.21.15.16:0/2393415265 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f07bc0ccab0 0x7f07bc0b2f90 secure :-1 s=READY pgs=2038 cs=0 l=1 rev1=1 crypto rx=0x7f07b80ad040 tx=0x7f07b801ac90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:47:06.203 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.191+0000 7f07adffb640 1 -- 172.21.15.16:0/2393415265 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07b8004230 con 0x7f07bc0ccab0 2023-05-22T15:47:06.203 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.191+0000 7f07adffb640 1 -- 172.21.15.16:0/2393415265 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07b80b9070 con 0x7f07bc0ccab0 2023-05-22T15:47:06.204 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.191+0000 7f07c35cb640 1 -- 172.21.15.16:0/2393415265 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07bc160350 con 0x7f07bc0ccab0 2023-05-22T15:47:06.204 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.191+0000 7f07adffb640 1 -- 172.21.15.16:0/2393415265 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07b80ab610 con 0x7f07bc0ccab0 2023-05-22T15:47:06.204 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.191+0000 7f07c35cb640 1 -- 172.21.15.16:0/2393415265 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07bc0cf730 con 0x7f07bc0ccab0 2023-05-22T15:47:06.204 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.195+0000 7f07c35cb640 1 -- 172.21.15.16:0/2393415265 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07bc1604e0 con 0x7f07bc0ccab0 2023-05-22T15:47:06.205 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.195+0000 7f07adffb640 1 -- 172.21.15.16:0/2393415265 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 67) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f07b8003620 con 0x7f07bc0ccab0 2023-05-22T15:47:06.205 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.195+0000 7f07adffb640 1 --2- 172.21.15.16:0/2393415265 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f07a00654a0 0x7f07a0067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:47:06.205 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.195+0000 7f07adffb640 1 -- 172.21.15.16:0/2393415265 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(161..161 src has 1..161) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f07b80bb070 con 0x7f07bc0ccab0 2023-05-22T15:47:06.205 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.195+0000 7f07c1367640 1 --2- 172.21.15.16:0/2393415265 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f07a00654a0 0x7f07a0067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:47:06.206 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.195+0000 7f07c1367640 1 --2- 172.21.15.16:0/2393415265 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f07a00654a0 0x7f07a0067950 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f07b0000f70 tx=0x7f07b0009040 comp rx=0 tx=0).ready entity=mgr.9396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:47:06.206 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.199+0000 7f07adffb640 1 -- 172.21.15.16:0/2393415265 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f07b80eb250 con 0x7f07bc0ccab0 2023-05-22T15:47:06.391 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.383+0000 7f07c35cb640 1 -- 172.21.15.16:0/2393415265 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f07bc0c8ac0 con 0x7f07bc0ccab0 2023-05-22T15:47:06.391 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.387+0000 7f07adffb640 1 -- 172.21.15.16:0/2393415265 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) v1 ==== 94+0+4644 (secure 0 0 0) 0x7f07b80ee870 con 0x7f07bc0ccab0 2023-05-22T15:47:06.392 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:47:06.392 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":66,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8170,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6833/78835131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":78835131},{"type":"v1","addr":"172.21.15.62:6833","nonce":78835131}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":61},{"gid":8173,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/1188098562","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":1188098562},{"type":"v1","addr":"172.21.15.62:6835","nonce":1188098562}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":61},{"gid":9645,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/4204206994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":4204206994},{"type":"v1","addr":"172.21.15.16:6835","nonce":4204206994}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":61}],"filesystems":[{"mdsmap":{"epoch":66,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-05-22T15:47:03.009254+0000","modified":"2023-05-22T15:47:06.030064+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9654},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9654":{"gid":9654,"name":"c","rank":0,"incarnation":62,"state":"up:active","state_seq":3,"addr":"172.21.15.16:6837/1284822222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":1284822222},{"type":"v1","addr":"172.21.15.16:6837","nonce":1284822222}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2023-05-22T15:47:06.393 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 66 2023-05-22T15:47:06.393 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.387+0000 7f07c35cb640 1 -- 172.21.15.16:0/2393415265 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f07a00654a0 msgr2=0x7f07a0067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:47:06.394 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.387+0000 7f07c35cb640 1 --2- 172.21.15.16:0/2393415265 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f07a00654a0 0x7f07a0067950 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f07b0000f70 tx=0x7f07b0009040 comp rx=0 tx=0).stop 2023-05-22T15:47:06.394 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.387+0000 7f07c35cb640 1 -- 172.21.15.16:0/2393415265 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f07bc0ccab0 msgr2=0x7f07bc0b2f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:47:06.394 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.387+0000 7f07c35cb640 1 --2- 172.21.15.16:0/2393415265 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f07bc0ccab0 0x7f07bc0b2f90 secure :-1 s=READY pgs=2038 cs=0 l=1 rev1=1 crypto rx=0x7f07b80ad040 tx=0x7f07b801ac90 comp rx=0 tx=0).stop 2023-05-22T15:47:06.394 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.387+0000 7f07c35cb640 1 -- 172.21.15.16:0/2393415265 shutdown_connections 2023-05-22T15:47:06.394 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.387+0000 7f07c35cb640 1 --2- 172.21.15.16:0/2393415265 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f07bc0d0410 0x7f07bc0b34d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:06.394 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.387+0000 7f07c35cb640 1 --2- 172.21.15.16:0/2393415265 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f07bc0ccab0 0x7f07bc0b2f90 unknown :-1 s=CLOSED pgs=2038 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:06.394 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.387+0000 7f07c35cb640 1 --2- 172.21.15.16:0/2393415265 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f07a00654a0 0x7f07a0067950 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:06.395 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.387+0000 7f07c35cb640 1 --2- 172.21.15.16:0/2393415265 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f07bc0b7dd0 0x7f07bc0c5f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:06.395 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.387+0000 7f07c35cb640 1 -- 172.21.15.16:0/2393415265 >> 172.21.15.16:0/2393415265 conn(0x7f07bc0655c0 msgr2=0x7f07bc1660f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:47:06.395 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.387+0000 7f07c35cb640 1 -- 172.21.15.16:0/2393415265 shutdown_connections 2023-05-22T15:47:06.395 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.387+0000 7f07c35cb640 1 -- 172.21.15.16:0/2393415265 wait complete. 2023-05-22T15:47:06.410 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-05-22T15:47:06.535 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.531+0000 7f0cec961640 1 Processor -- start 2023-05-22T15:47:06.536 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.531+0000 7f0cec961640 1 -- start start 2023-05-22T15:47:06.538 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.531+0000 7f0cec961640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f0ce8070f10 0x7f0ce80afb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:47:06.538 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.531+0000 7f0cec961640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0ce815ad40 0x7f0ce80c8670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:47:06.538 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.531+0000 7f0cec961640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ce815a8b0 con 0x7f0ce815e2f0 2023-05-22T15:47:06.539 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.531+0000 7f0cec961640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ce80b0200 con 0x7f0ce8070f10 2023-05-22T15:47:06.539 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.531+0000 7f0cec961640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ce80b04e0 con 0x7f0ce815ad40 2023-05-22T15:47:06.539 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.531+0000 7f0ce659c640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f0ce8070f10 0x7f0ce80afb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:47:06.539 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.531+0000 7f0ce659c640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f0ce8070f10 0x7f0ce80afb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46816/0 (socket says 172.21.15.16:46816) 2023-05-22T15:47:06.540 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.531+0000 7f0ce659c640 1 -- 172.21.15.16:0/1593444906 learned_addr learned my addr 172.21.15.16:0/1593444906 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:47:06.540 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.531+0000 7f0ce6d9d640 1 --2- 172.21.15.16:0/1593444906 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0ce815ad40 0x7f0ce80c8670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:47:06.540 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.531+0000 7f0ce659c640 1 -- 172.21.15.16:0/1593444906 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0ce815ad40 msgr2=0x7f0ce80c8670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:47:06.540 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.531+0000 7f0ce659c640 1 --2- 172.21.15.16:0/1593444906 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0ce815ad40 0x7f0ce80c8670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:06.541 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.531+0000 7f0ce659c640 1 -- 172.21.15.16:0/1593444906 >> v1:172.21.15.62:6789/0 conn(0x7f0ce815e2f0 legacy=0x7f0ce80712e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:47:06.541 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.531+0000 7f0ce659c640 1 -- 172.21.15.16:0/1593444906 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ce815a540 con 0x7f0ce8070f10 2023-05-22T15:47:06.544 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.535+0000 7f0ce659c640 1 --2- 172.21.15.16:0/1593444906 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ce8070f10 0x7f0ce80afb40 secure :-1 s=READY pgs=2039 cs=0 l=1 rev1=1 crypto rx=0x7f0cd8009870 tx=0x7f0cd801c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9517c3d44216bd67 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:47:06.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.535+0000 7f0ce559a640 1 -- 172.21.15.16:0/1593444906 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cd80ab640 con 0x7f0ce8070f10 2023-05-22T15:47:06.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.535+0000 7f0ce559a640 1 -- 172.21.15.16:0/1593444906 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0cd80a7070 con 0x7f0ce8070f10 2023-05-22T15:47:06.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.535+0000 7f0ce559a640 1 -- 172.21.15.16:0/1593444906 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cd80ab9c0 con 0x7f0ce8070f10 2023-05-22T15:47:06.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.535+0000 7f0cec961640 1 -- 172.21.15.16:0/1593444906 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ce8070f10 msgr2=0x7f0ce80afb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:47:06.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.535+0000 7f0cec961640 1 --2- 172.21.15.16:0/1593444906 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ce8070f10 0x7f0ce80afb40 secure :-1 s=READY pgs=2039 cs=0 l=1 rev1=1 crypto rx=0x7f0cd8009870 tx=0x7f0cd801c480 comp rx=0 tx=0).stop 2023-05-22T15:47:06.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.535+0000 7f0cec961640 1 -- 172.21.15.16:0/1593444906 shutdown_connections 2023-05-22T15:47:06.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.535+0000 7f0cec961640 1 --2- 172.21.15.16:0/1593444906 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0ce815ad40 0x7f0ce80c8670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:06.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.535+0000 7f0cec961640 1 --2- 172.21.15.16:0/1593444906 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ce8070f10 0x7f0ce80afb40 unknown :-1 s=CLOSED pgs=2039 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:06.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.535+0000 7f0cec961640 1 -- 172.21.15.16:0/1593444906 >> 172.21.15.16:0/1593444906 conn(0x7f0ce8069720 msgr2=0x7f0ce8069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:47:06.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.535+0000 7f0cec961640 1 -- 172.21.15.16:0/1593444906 shutdown_connections 2023-05-22T15:47:06.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.535+0000 7f0cec961640 1 -- 172.21.15.16:0/1593444906 wait complete. 2023-05-22T15:47:06.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.535+0000 7f0cec961640 1 Processor -- start 2023-05-22T15:47:06.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.535+0000 7f0cec961640 1 -- start start 2023-05-22T15:47:06.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.535+0000 7f0cec961640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0ce8070f10 0x7f0ce80afba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:47:06.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.535+0000 7f0cec961640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ce815ad40 0x7f0ce8146b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:47:06.549 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.535+0000 7f0cec961640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0ce815e2f0 0x7f0ce8144220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:47:06.549 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.535+0000 7f0cec961640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ce80b26a0 con 0x7f0ce815ad40 2023-05-22T15:47:06.549 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.535+0000 7f0cec961640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ce8145bc0 con 0x7f0ce815e2f0 2023-05-22T15:47:06.549 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.535+0000 7f0cec961640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ce8145ea0 con 0x7f0ce8070f10 2023-05-22T15:47:06.550 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.535+0000 7f0ce5d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ce815ad40 0x7f0ce8146b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:47:06.550 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.535+0000 7f0ce5d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ce815ad40 0x7f0ce8146b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46822/0 (socket says 172.21.15.16:46822) 2023-05-22T15:47:06.550 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.535+0000 7f0ce5d9b640 1 -- 172.21.15.16:0/42283794 learned_addr learned my addr 172.21.15.16:0/42283794 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:47:06.550 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.535+0000 7f0ce659c640 1 --2- 172.21.15.16:0/42283794 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0ce8070f10 0x7f0ce80afba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:47:06.551 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.535+0000 7f0ce6d9d640 1 --2- 172.21.15.16:0/42283794 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0ce815e2f0 0x7f0ce8144220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:47:06.551 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.535+0000 7f0ce5d9b640 1 -- 172.21.15.16:0/42283794 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0ce8070f10 msgr2=0x7f0ce80afba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:47:06.551 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.535+0000 7f0ce5d9b640 1 --2- 172.21.15.16:0/42283794 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0ce8070f10 0x7f0ce80afba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:06.551 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.535+0000 7f0ce5d9b640 1 -- 172.21.15.16:0/42283794 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0ce815e2f0 msgr2=0x7f0ce8144220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:47:06.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.535+0000 7f0ce5d9b640 1 --2- 172.21.15.16:0/42283794 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0ce815e2f0 0x7f0ce8144220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:06.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.535+0000 7f0ce5d9b640 1 -- 172.21.15.16:0/42283794 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cd80094d0 con 0x7f0ce815ad40 2023-05-22T15:47:06.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.535+0000 7f0ce659c640 1 --2- 172.21.15.16:0/42283794 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0ce8070f10 0x7f0ce80afba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T15:47:06.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.535+0000 7f0ce6d9d640 1 --2- 172.21.15.16:0/42283794 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0ce815e2f0 0x7f0ce8144220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T15:47:06.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.539+0000 7f0ce5d9b640 1 --2- 172.21.15.16:0/42283794 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ce815ad40 0x7f0ce8146b70 secure :-1 s=READY pgs=2040 cs=0 l=1 rev1=1 crypto rx=0x7f0cdc00c3c0 tx=0x7f0cdc00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:47:06.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.539+0000 7f0cd6ffd640 1 -- 172.21.15.16:0/42283794 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cdc0052d0 con 0x7f0ce815ad40 2023-05-22T15:47:06.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.539+0000 7f0cd6ffd640 1 -- 172.21.15.16:0/42283794 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0cdc005430 con 0x7f0ce815ad40 2023-05-22T15:47:06.554 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.539+0000 7f0cd6ffd640 1 -- 172.21.15.16:0/42283794 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cdc003840 con 0x7f0ce815ad40 2023-05-22T15:47:06.554 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.539+0000 7f0cec961640 1 -- 172.21.15.16:0/42283794 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ce8157bb0 con 0x7f0ce815ad40 2023-05-22T15:47:06.554 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.539+0000 7f0cec961640 1 -- 172.21.15.16:0/42283794 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ce815b140 con 0x7f0ce815ad40 2023-05-22T15:47:06.554 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.539+0000 7f0cec961640 1 -- 172.21.15.16:0/42283794 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ce8166c10 con 0x7f0ce815ad40 2023-05-22T15:47:06.555 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.539+0000 7f0cd6ffd640 1 -- 172.21.15.16:0/42283794 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 67) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f0cdc0039a0 con 0x7f0ce815ad40 2023-05-22T15:47:06.556 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.539+0000 7f0cd6ffd640 1 --2- 172.21.15.16:0/42283794 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f0cc80654a0 0x7f0cc8067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:47:06.556 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.539+0000 7f0cd6ffd640 1 -- 172.21.15.16:0/42283794 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(161..161 src has 1..161) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f0cdc082210 con 0x7f0ce815ad40 2023-05-22T15:47:06.556 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.539+0000 7f0ce659c640 1 --2- 172.21.15.16:0/42283794 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f0cc80654a0 0x7f0cc8067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:47:06.557 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.539+0000 7f0ce659c640 1 --2- 172.21.15.16:0/42283794 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f0cc80654a0 0x7f0cc8067950 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f0cd80af000 tx=0x7f0cd80a7000 comp rx=0 tx=0).ready entity=mgr.9396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:47:06.557 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.543+0000 7f0cd6ffd640 1 -- 172.21.15.16:0/42283794 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f0cdc04fca0 con 0x7f0ce815ad40 2023-05-22T15:47:06.703 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.699+0000 7f0cec961640 1 -- 172.21.15.16:0/42283794 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0ce804dbe0 con 0x7f0ce815ad40 2023-05-22T15:47:06.707 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.699+0000 7f0cd6ffd640 1 -- 172.21.15.16:0/42283794 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v161) v1 ==== 74+0+18094 (secure 0 0 0) 0x7f0cdc0532c0 con 0x7f0ce815ad40 2023-05-22T15:47:06.708 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:47:06.708 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":161,"fsid":"b2719834-7847-4d98-90a6-f65d57f85c49","created":"2023-05-22T15:26:25.259314+0000","modified":"2023-05-22T15:47:04.013850+0000","last_up_change":"2023-05-22T15:26:36.448524+0000","last_in_change":"2023-05-22T15:26:27.675668+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-05-22T15:26:38.019781+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-05-22T15:26:43.570514+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_metadata","create_time":"2023-05-22T15:47:01.347542+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"160","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_data","create_time":"2023-05-22T15:47:01.970646+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"161","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9e7ff854-b25f-4cbb-a5c2-1799d9c53d2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":37021},{"type":"v1","addr":"172.21.15.16:6825","nonce":37021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":37021},{"type":"v1","addr":"172.21.15.16:6827","nonce":37021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":37021},{"type":"v1","addr":"172.21.15.16:6831","nonce":37021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":37021},{"type":"v1","addr":"172.21.15.16:6829","nonce":37021}]},"public_addr":"172.21.15.16:6825/37021","cluster_addr":"172.21.15.16:6827/37021","heartbeat_back_addr":"172.21.15.16:6831/37021","heartbeat_front_addr":"172.21.15.16:6829/37021","state":["exists","up"]},{"osd":1,"uuid":"4702fbc9-a927-4565-aca5-41dc69cc8264","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":37008},{"type":"v1","addr":"172.21.15.16:6801","nonce":37008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":37008},{"type":"v1","addr":"172.21.15.16:6803","nonce":37008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":37008},{"type":"v1","addr":"172.21.15.16:6807","nonce":37008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":37008},{"type":"v1","addr":"172.21.15.16:6805","nonce":37008}]},"public_addr":"172.21.15.16:6801/37008","cluster_addr":"172.21.15.16:6803/37008","heartbeat_back_addr":"172.21.15.16:6807/37008","heartbeat_front_addr":"172.21.15.16:6805/37008","state":["exists","up"]},{"osd":2,"uuid":"1e7bba46-4eba-47ef-bb81-5e2d371638ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":37009},{"type":"v1","addr":"172.21.15.16:6809","nonce":37009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":37009},{"type":"v1","addr":"172.21.15.16:6811","nonce":37009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":37009},{"type":"v1","addr":"172.21.15.16:6815","nonce":37009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":37009},{"type":"v1","addr":"172.21.15.16:6813","nonce":37009}]},"public_addr":"172.21.15.16:6809/37009","cluster_addr":"172.21.15.16:6811/37009","heartbeat_back_addr":"172.21.15.16:6815/37009","heartbeat_front_addr":"172.21.15.16:6813/37009","state":["exists","up"]},{"osd":3,"uuid":"383cacc0-5a88-4904-990e-498ca59ef48a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":37020},{"type":"v1","addr":"172.21.15.16:6817","nonce":37020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":37020},{"type":"v1","addr":"172.21.15.16:6819","nonce":37020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":37020},{"type":"v1","addr":"172.21.15.16:6823","nonce":37020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":37020},{"type":"v1","addr":"172.21.15.16:6821","nonce":37020}]},"public_addr":"172.21.15.16:6817/37020","cluster_addr":"172.21.15.16:6819/37020","heartbeat_back_addr":"172.21.15.16:6823/37020","heartbeat_front_addr":"172.21.15.16:6821/37020","state":["exists","up"]},{"osd":4,"uuid":"970500ef-95b7-42c1-b080-eed4fae7013d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":37353},{"type":"v1","addr":"172.21.15.62:6817","nonce":37353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6819","nonce":37353},{"type":"v1","addr":"172.21.15.62:6822","nonce":37353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37353},{"type":"v1","addr":"172.21.15.62:6831","nonce":37353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37353},{"type":"v1","addr":"172.21.15.62:6826","nonce":37353}]},"public_addr":"172.21.15.62:6817/37353","cluster_addr":"172.21.15.62:6822/37353","heartbeat_back_addr":"172.21.15.62:6831/37353","heartbeat_front_addr":"172.21.15.62:6826/37353","state":["exists","up"]},{"osd":5,"uuid":"022d67db-d4d8-445b-b1b9-8b446a00f5c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":37352},{"type":"v1","addr":"172.21.15.62:6801","nonce":37352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":37352},{"type":"v1","addr":"172.21.15.62:6803","nonce":37352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":37352},{"type":"v1","addr":"172.21.15.62:6807","nonce":37352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":37352},{"type":"v1","addr":"172.21.15.62:6805","nonce":37352}]},"public_addr":"172.21.15.62:6801/37352","cluster_addr":"172.21.15.62:6803/37352","heartbeat_back_addr":"172.21.15.62:6807/37352","heartbeat_front_addr":"172.21.15.62:6805/37352","state":["exists","up"]},{"osd":6,"uuid":"42bd372b-6688-4a1b-8407-e16e8c266e33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":37354},{"type":"v1","addr":"172.21.15.62:6820","nonce":37354}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":37354},{"type":"v1","addr":"172.21.15.62:6823","nonce":37354}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6829","nonce":37354},{"type":"v1","addr":"172.21.15.62:6830","nonce":37354}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6825","nonce":37354},{"type":"v1","addr":"172.21.15.62:6827","nonce":37354}]},"public_addr":"172.21.15.62:6820/37354","cluster_addr":"172.21.15.62:6823/37354","heartbeat_back_addr":"172.21.15.62:6830/37354","heartbeat_front_addr":"172.21.15.62:6827/37354","state":["exists","up"]},{"osd":7,"uuid":"ac70b2a3-f41b-4bf2-b227-7f2cbb55426d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37355},{"type":"v1","addr":"172.21.15.62:6809","nonce":37355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37355},{"type":"v1","addr":"172.21.15.62:6811","nonce":37355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":37355},{"type":"v1","addr":"172.21.15.62:6815","nonce":37355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":37355},{"type":"v1","addr":"172.21.15.62:6813","nonce":37355}]},"public_addr":"172.21.15.62:6809/37355","cluster_addr":"172.21.15.62:6811/37355","heartbeat_back_addr":"172.21.15.62:6815/37355","heartbeat_front_addr":"172.21.15.62:6813/37355","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.368896+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.107273+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.276888+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.287502+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.124791+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.251937+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.255679+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.232797+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":17,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-05-22T15:47:06.709 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.703+0000 7f0cec961640 1 -- 172.21.15.16:0/42283794 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f0cc80654a0 msgr2=0x7f0cc8067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:47:06.709 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.703+0000 7f0cec961640 1 --2- 172.21.15.16:0/42283794 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f0cc80654a0 0x7f0cc8067950 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f0cd80af000 tx=0x7f0cd80a7000 comp rx=0 tx=0).stop 2023-05-22T15:47:06.710 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.703+0000 7f0cec961640 1 -- 172.21.15.16:0/42283794 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ce815ad40 msgr2=0x7f0ce8146b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:47:06.710 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.703+0000 7f0cec961640 1 --2- 172.21.15.16:0/42283794 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ce815ad40 0x7f0ce8146b70 secure :-1 s=READY pgs=2040 cs=0 l=1 rev1=1 crypto rx=0x7f0cdc00c3c0 tx=0x7f0cdc00c880 comp rx=0 tx=0).stop 2023-05-22T15:47:06.710 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.703+0000 7f0cec961640 1 -- 172.21.15.16:0/42283794 shutdown_connections 2023-05-22T15:47:06.710 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.703+0000 7f0cec961640 1 --2- 172.21.15.16:0/42283794 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0ce815e2f0 0x7f0ce8144220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:06.710 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.703+0000 7f0cec961640 1 --2- 172.21.15.16:0/42283794 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ce815ad40 0x7f0ce8146b70 unknown :-1 s=CLOSED pgs=2040 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:06.710 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.703+0000 7f0cec961640 1 --2- 172.21.15.16:0/42283794 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f0cc80654a0 0x7f0cc8067950 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:06.710 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.703+0000 7f0cec961640 1 --2- 172.21.15.16:0/42283794 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0ce8070f10 0x7f0ce80afba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:06.711 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.703+0000 7f0cec961640 1 -- 172.21.15.16:0/42283794 >> 172.21.15.16:0/42283794 conn(0x7f0ce8069720 msgr2=0x7f0ce8150860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:47:06.711 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.703+0000 7f0cec961640 1 -- 172.21.15.16:0/42283794 shutdown_connections 2023-05-22T15:47:06.711 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.703+0000 7f0cec961640 1 -- 172.21.15.16:0/42283794 wait complete. 2023-05-22T15:47:06.727 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-05-22T15:47:06.727 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-05-22T15:47:06.846 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.839+0000 7fe6c4919640 1 Processor -- start 2023-05-22T15:47:06.847 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.839+0000 7fe6c4919640 1 -- start start 2023-05-22T15:47:06.849 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.843+0000 7fe6c4919640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fe6c00b7dd0 0x7fe6c00ccab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:47:06.849 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.843+0000 7fe6c4919640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6c00cd170 con 0x7fe6c015f700 2023-05-22T15:47:06.849 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.843+0000 7fe6c4919640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6c00d2ac0 con 0x7fe6c0160250 2023-05-22T15:47:06.849 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.843+0000 7fe6c4919640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6c00d3360 con 0x7fe6c00b7dd0 2023-05-22T15:47:06.850 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.843+0000 7fe6bdd9b640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fe6c015f700 0x7fe6c015fb40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:44874/0 (socket says 172.21.15.16:44874) 2023-05-22T15:47:06.850 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.843+0000 7fe6bdd9b640 1 -- 172.21.15.16:0/2438888340 learned_addr learned my addr 172.21.15.16:0/2438888340 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:47:06.850 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.843+0000 7fe6be59c640 1 --2- 172.21.15.16:0/2438888340 >> v2:172.21.15.62:3300/0 conn(0x7fe6c00b7dd0 0x7fe6c00ccab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:47:06.850 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.843+0000 7fe6bd59a640 1 -- 172.21.15.16:0/2438888340 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1333871403 0 0) 0x7fe6c00cd170 con 0x7fe6c015f700 2023-05-22T15:47:06.850 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.843+0000 7fe6bd59a640 1 -- 172.21.15.16:0/2438888340 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe6a8003530 con 0x7fe6c015f700 2023-05-22T15:47:06.851 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.843+0000 7fe6bd59a640 1 -- 172.21.15.16:0/2438888340 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4204111673 0 0) 0x7fe6a8003530 con 0x7fe6c015f700 2023-05-22T15:47:06.851 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.843+0000 7fe6bd59a640 1 -- 172.21.15.16:0/2438888340 >> v2:172.21.15.62:3300/0 conn(0x7fe6c00b7dd0 msgr2=0x7fe6c00ccab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:47:06.851 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.843+0000 7fe6bd59a640 1 --2- 172.21.15.16:0/2438888340 >> v2:172.21.15.62:3300/0 conn(0x7fe6c00b7dd0 0x7fe6c00ccab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:06.851 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.843+0000 7fe6bd59a640 1 -- 172.21.15.16:0/2438888340 >> v1:172.21.15.62:6789/0 conn(0x7fe6c0160250 legacy=0x7fe6c00d0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:47:06.851 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.843+0000 7fe6bd59a640 1 -- 172.21.15.16:0/2438888340 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6c00d2da0 con 0x7fe6c015f700 2023-05-22T15:47:06.851 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.843+0000 7fe6bd59a640 1 -- 172.21.15.16:0/2438888340 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7fe6b4001680 con 0x7fe6c015f700 2023-05-22T15:47:06.851 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.843+0000 7fe6bd59a640 1 -- 172.21.15.16:0/2438888340 >> v1:172.21.15.16:6789/0 conn(0x7fe6c015f700 legacy=0x7fe6c015fb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:47:06.851 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.843+0000 7fe6bd59a640 1 --2- 172.21.15.16:0/2438888340 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe6a80039d0 0x7fe6a8003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:47:06.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.843+0000 7fe6bd59a640 1 --2- 172.21.15.16:0/2438888340 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe6a80047d0 0x7fe6a8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:47:06.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.843+0000 7fe6bd59a640 1 --2- 172.21.15.16:0/2438888340 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe6a8005180 0x7fe6a800b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:47:06.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.843+0000 7fe6bd59a640 1 -- 172.21.15.16:0/2438888340 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6c00d3360 con 0x7fe6a8005180 2023-05-22T15:47:06.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.843+0000 7fe6bd59a640 1 -- 172.21.15.16:0/2438888340 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6a800c060 con 0x7fe6a80047d0 2023-05-22T15:47:06.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.843+0000 7fe6bd59a640 1 -- 172.21.15.16:0/2438888340 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6a800c220 con 0x7fe6a80039d0 2023-05-22T15:47:06.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.843+0000 7fe6bed9d640 1 --2- 172.21.15.16:0/2438888340 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe6a8005180 0x7fe6a800b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:47:06.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.843+0000 7fe6be59c640 1 --2- 172.21.15.16:0/2438888340 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe6a80039d0 0x7fe6a8003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:47:06.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.843+0000 7fe6bdd9b640 1 --2- 172.21.15.16:0/2438888340 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe6a80047d0 0x7fe6a8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:47:06.853 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.843+0000 7fe6bed9d640 1 -- 172.21.15.16:0/2438888340 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe6a80039d0 msgr2=0x7fe6a8003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:47:06.853 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.843+0000 7fe6bed9d640 1 --2- 172.21.15.16:0/2438888340 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe6a80039d0 0x7fe6a8003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:06.853 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.843+0000 7fe6bed9d640 1 -- 172.21.15.16:0/2438888340 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe6a80047d0 msgr2=0x7fe6a8004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:47:06.853 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.843+0000 7fe6bed9d640 1 --2- 172.21.15.16:0/2438888340 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe6a80047d0 0x7fe6a8004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:06.853 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.843+0000 7fe6bed9d640 1 -- 172.21.15.16:0/2438888340 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6a8003530 con 0x7fe6a8005180 2023-05-22T15:47:06.853 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.843+0000 7fe6bdd9b640 1 --2- 172.21.15.16:0/2438888340 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe6a80047d0 0x7fe6a8004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:47:06.855 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.847+0000 7fe6bed9d640 1 -- 172.21.15.16:0/2438888340 reap_dead start 2023-05-22T15:47:06.856 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.847+0000 7fe6bed9d640 1 --2- 172.21.15.16:0/2438888340 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe6a8005180 0x7fe6a800b990 secure :-1 s=READY pgs=2042 cs=0 l=1 rev1=1 crypto rx=0x7fe6b8001e40 tx=0x7fe6b801c740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e666c7e5320caf6 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:47:06.856 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.847+0000 7fe6bd59a640 1 -- 172.21.15.16:0/2438888340 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6b8003b80 con 0x7fe6a8005180 2023-05-22T15:47:06.856 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.847+0000 7fe6bd59a640 1 -- 172.21.15.16:0/2438888340 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6b8003ce0 con 0x7fe6a8005180 2023-05-22T15:47:06.856 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.847+0000 7fe6bd59a640 1 -- 172.21.15.16:0/2438888340 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6b80052f0 con 0x7fe6a8005180 2023-05-22T15:47:06.856 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.847+0000 7fe6c4919640 1 -- 172.21.15.16:0/2438888340 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe6a8005180 msgr2=0x7fe6a800b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:47:06.856 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.847+0000 7fe6c4919640 1 --2- 172.21.15.16:0/2438888340 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe6a8005180 0x7fe6a800b990 secure :-1 s=READY pgs=2042 cs=0 l=1 rev1=1 crypto rx=0x7fe6b8001e40 tx=0x7fe6b801c740 comp rx=0 tx=0).stop 2023-05-22T15:47:06.857 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.847+0000 7fe6c4919640 1 -- 172.21.15.16:0/2438888340 shutdown_connections 2023-05-22T15:47:06.857 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.847+0000 7fe6c4919640 1 --2- 172.21.15.16:0/2438888340 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe6a8005180 0x7fe6a800b990 unknown :-1 s=CLOSED pgs=2042 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:06.857 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.847+0000 7fe6c4919640 1 -- 172.21.15.16:0/2438888340 >> 172.21.15.16:0/2438888340 conn(0x7fe6c00655c0 msgr2=0x7fe6c0070e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:47:06.857 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.847+0000 7fe6c4919640 1 -- 172.21.15.16:0/2438888340 shutdown_connections 2023-05-22T15:47:06.857 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.847+0000 7fe6c4919640 1 -- 172.21.15.16:0/2438888340 wait complete. 2023-05-22T15:47:06.857 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.847+0000 7fe6c4919640 1 Processor -- start 2023-05-22T15:47:06.857 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.847+0000 7fe6c4919640 1 -- start start 2023-05-22T15:47:06.857 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.847+0000 7fe6c4919640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe6a8005180 0x7fe6c01f0f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:47:06.858 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.847+0000 7fe6c4919640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe6c015c200 0x7fe6c00d0a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:47:06.858 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.847+0000 7fe6c4919640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe6c01582a0 0x7fe6c0158740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:47:06.858 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.847+0000 7fe6c4919640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6c01f54a0 con 0x7fe6c01582a0 2023-05-22T15:47:06.858 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.847+0000 7fe6c4919640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6c0158c80 con 0x7fe6a8005180 2023-05-22T15:47:06.858 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.847+0000 7fe6c4919640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6c01f4540 con 0x7fe6c015c200 2023-05-22T15:47:06.858 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.847+0000 7fe6bed9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe6c01582a0 0x7fe6c0158740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:47:06.858 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.847+0000 7fe6bed9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe6c01582a0 0x7fe6c0158740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46844/0 (socket says 172.21.15.16:46844) 2023-05-22T15:47:06.859 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.847+0000 7fe6bed9d640 1 -- 172.21.15.16:0/1019708967 learned_addr learned my addr 172.21.15.16:0/1019708967 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:47:06.859 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.847+0000 7fe6be59c640 1 --2- 172.21.15.16:0/1019708967 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe6a8005180 0x7fe6c01f0f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:47:06.859 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.847+0000 7fe6bdd9b640 1 --2- 172.21.15.16:0/1019708967 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe6c015c200 0x7fe6c00d0a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:47:06.859 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.847+0000 7fe6bed9d640 1 -- 172.21.15.16:0/1019708967 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe6c015c200 msgr2=0x7fe6c00d0a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:47:06.859 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.847+0000 7fe6bed9d640 1 --2- 172.21.15.16:0/1019708967 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe6c015c200 0x7fe6c00d0a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:06.859 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.847+0000 7fe6bed9d640 1 -- 172.21.15.16:0/1019708967 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe6a8005180 msgr2=0x7fe6c01f0f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:47:06.859 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.847+0000 7fe6bed9d640 1 --2- 172.21.15.16:0/1019708967 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe6a8005180 0x7fe6c01f0f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:06.859 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.847+0000 7fe6bed9d640 1 -- 172.21.15.16:0/1019708967 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6c00ccab0 con 0x7fe6c01582a0 2023-05-22T15:47:06.860 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.847+0000 7fe6be59c640 1 --2- 172.21.15.16:0/1019708967 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe6a8005180 0x7fe6c01f0f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T15:47:06.860 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.851+0000 7fe6bed9d640 1 --2- 172.21.15.16:0/1019708967 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe6c01582a0 0x7fe6c0158740 secure :-1 s=READY pgs=2043 cs=0 l=1 rev1=1 crypto rx=0x7fe6b801c780 tx=0x7fe6a8005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:47:06.860 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.851+0000 7fe6aeffd640 1 -- 172.21.15.16:0/1019708967 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6b80b4070 con 0x7fe6c01582a0 2023-05-22T15:47:06.860 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.851+0000 7fe6aeffd640 1 -- 172.21.15.16:0/1019708967 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6b8006b60 con 0x7fe6c01582a0 2023-05-22T15:47:06.860 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.851+0000 7fe6c4919640 1 -- 172.21.15.16:0/1019708967 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6c00d3080 con 0x7fe6c01582a0 2023-05-22T15:47:06.860 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.851+0000 7fe6aeffd640 1 -- 172.21.15.16:0/1019708967 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6b8003b80 con 0x7fe6c01582a0 2023-05-22T15:47:06.860 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.851+0000 7fe6c4919640 1 -- 172.21.15.16:0/1019708967 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6c00ccc40 con 0x7fe6c01582a0 2023-05-22T15:47:06.861 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.851+0000 7fe6c4919640 1 -- 172.21.15.16:0/1019708967 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6c004dbe0 con 0x7fe6c01582a0 2023-05-22T15:47:06.861 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.851+0000 7fe6aeffd640 1 -- 172.21.15.16:0/1019708967 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 67) v1 ==== 84332+0+0 (secure 0 0 0) 0x7fe6b80052e0 con 0x7fe6c01582a0 2023-05-22T15:47:06.862 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.851+0000 7fe6aeffd640 1 --2- 172.21.15.16:0/1019708967 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fe6a00654a0 0x7fe6a0067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:47:06.862 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.851+0000 7fe6aeffd640 1 -- 172.21.15.16:0/1019708967 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(161..161 src has 1..161) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fe6b80ad070 con 0x7fe6c01582a0 2023-05-22T15:47:06.862 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.851+0000 7fe6be59c640 1 --2- 172.21.15.16:0/1019708967 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fe6a00654a0 0x7fe6a0067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:47:06.862 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.851+0000 7fe6be59c640 1 --2- 172.21.15.16:0/1019708967 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fe6a00654a0 0x7fe6a0067950 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fe6c00cf480 tx=0x7fe6b0008040 comp rx=0 tx=0).ready entity=mgr.9396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:47:06.862 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.855+0000 7fe6aeffd640 1 -- 172.21.15.16:0/1019708967 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fe6b80ed190 con 0x7fe6c01582a0 2023-05-22T15:47:07.006 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:06.999+0000 7fe6c4919640 1 -- 172.21.15.16:0/1019708967 --> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fe6c00cf4b0 con 0x7fe6a00654a0 2023-05-22T15:47:07.031 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.027+0000 7fe6aeffd640 1 -- 172.21.15.16:0/1019708967 <== mgr.9396 v2:172.21.15.16:6832/36921 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+309955 (secure 0 0 0) 0x7fe6c00cf4b0 con 0x7fe6a00654a0 2023-05-22T15:47:07.035 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:47:07.037 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2023-05-22T15:47:07.040 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.031+0000 7fe6c4919640 1 -- 172.21.15.16:0/1019708967 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fe6a00654a0 msgr2=0x7fe6a0067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:47:07.040 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.031+0000 7fe6c4919640 1 --2- 172.21.15.16:0/1019708967 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fe6a00654a0 0x7fe6a0067950 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fe6c00cf480 tx=0x7fe6b0008040 comp rx=0 tx=0).stop 2023-05-22T15:47:07.040 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.031+0000 7fe6c4919640 1 -- 172.21.15.16:0/1019708967 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe6c01582a0 msgr2=0x7fe6c0158740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:47:07.041 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.031+0000 7fe6c4919640 1 --2- 172.21.15.16:0/1019708967 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe6c01582a0 0x7fe6c0158740 secure :-1 s=READY pgs=2043 cs=0 l=1 rev1=1 crypto rx=0x7fe6b801c780 tx=0x7fe6a8005800 comp rx=0 tx=0).stop 2023-05-22T15:47:07.041 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.031+0000 7fe6c4919640 1 -- 172.21.15.16:0/1019708967 shutdown_connections 2023-05-22T15:47:07.041 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.031+0000 7fe6c4919640 1 --2- 172.21.15.16:0/1019708967 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe6c01582a0 0x7fe6c0158740 unknown :-1 s=CLOSED pgs=2043 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:07.041 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.031+0000 7fe6c4919640 1 --2- 172.21.15.16:0/1019708967 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fe6a00654a0 0x7fe6a0067950 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:07.041 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.031+0000 7fe6c4919640 1 --2- 172.21.15.16:0/1019708967 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe6c015c200 0x7fe6c00d0a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:07.041 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.031+0000 7fe6c4919640 1 --2- 172.21.15.16:0/1019708967 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe6a8005180 0x7fe6c01f0f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:07.041 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.031+0000 7fe6c4919640 1 -- 172.21.15.16:0/1019708967 >> 172.21.15.16:0/1019708967 conn(0x7fe6c00655c0 msgr2=0x7fe6c00d2010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:47:07.042 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.031+0000 7fe6c4919640 1 -- 172.21.15.16:0/1019708967 shutdown_connections 2023-05-22T15:47:07.042 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.031+0000 7fe6c4919640 1 -- 172.21.15.16:0/1019708967 wait complete. 2023-05-22T15:47:07.052 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2023-05-22T15:47:05.713182+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592698,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1460,"num_read_kb":2984,"num_write":483,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":309,"ondisk_log_size":309,"up":210,"acting":210,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":59,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":242840,"kb_used_data":16792,"kb_used_omap":0,"kb_used_meta":225984,"kb_avail":754731880,"statfs":{"total":773094113280,"available":772845445120,"internally_reserved":0,"allocated":17195008,"data_stored":9630118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":231407616},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":52,"apply_latency_ms":52,"commit_latency_ns":52000000,"apply_latency_ns":52000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2330,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.105499"},"pg_stats":[{"pgid":"21.3f","version":"0'0","reported_seq":9,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-05-22T15:47:03.075301+0000","last_change":"2023-05-22T15:47:03.074499+0000","last_active":"2023-05-22T15:47:03.075301+0000","last_peered":"2023-05-22T15:47:03.075301+0000","last_clean":"2023-05-22T15:47:03.075301+0000","last_became_active":"2023-05-22T15:47:03.074171+0000","last_became_peered":"2023-05-22T15:47:03.074171+0000","last_unstale":"2023-05-22T15:47:03.075301+0000","last_undegraded":"2023-05-22T15:47:03.075301+0000","last_fullsized":"2023-05-22T15:47:03.075301+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_clean_scrub_stamp":"2023-05-22T15:47:02.606868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:07:06.587412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.3e","version":"0'0","reported_seq":9,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-05-22T15:47:03.044416+0000","last_change":"2023-05-22T15:47:03.043270+0000","last_active":"2023-05-22T15:47:03.044416+0000","last_peered":"2023-05-22T15:47:03.044416+0000","last_clean":"2023-05-22T15:47:03.044416+0000","last_became_active":"2023-05-22T15:47:03.042914+0000","last_became_peered":"2023-05-22T15:47:03.042914+0000","last_unstale":"2023-05-22T15:47:03.044416+0000","last_undegraded":"2023-05-22T15:47:03.044416+0000","last_fullsized":"2023-05-22T15:47:03.044416+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_clean_scrub_stamp":"2023-05-22T15:47:02.606868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:59:07.114320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.3d","version":"0'0","reported_seq":9,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-05-22T15:47:03.098373+0000","last_change":"2023-05-22T15:47:03.091606+0000","last_active":"2023-05-22T15:47:03.098373+0000","last_peered":"2023-05-22T15:47:03.098373+0000","last_clean":"2023-05-22T15:47:03.098373+0000","last_became_active":"2023-05-22T15:47:03.087886+0000","last_became_peered":"2023-05-22T15:47:03.087886+0000","last_unstale":"2023-05-22T15:47:03.098373+0000","last_undegraded":"2023-05-22T15:47:03.098373+0000","last_fullsized":"2023-05-22T15:47:03.098373+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_clean_scrub_stamp":"2023-05-22T15:47:02.606868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T03:11:13.112290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.3c","version":"0'0","reported_seq":9,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-05-22T15:47:03.097078+0000","last_change":"2023-05-22T15:47:03.095261+0000","last_active":"2023-05-22T15:47:03.097078+0000","last_peered":"2023-05-22T15:47:03.097078+0000","last_clean":"2023-05-22T15:47:03.097078+0000","last_became_active":"2023-05-22T15:47:03.094750+0000","last_became_peered":"2023-05-22T15:47:03.094750+0000","last_unstale":"2023-05-22T15:47:03.097078+0000","last_undegraded":"2023-05-22T15:47:03.097078+0000","last_fullsized":"2023-05-22T15:47:03.097078+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_clean_scrub_stamp":"2023-05-22T15:47:02.606868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:34:17.773670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.3b","version":"0'0","reported_seq":9,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-05-22T15:47:03.082841+0000","last_change":"2023-05-22T15:47:03.077042+0000","last_active":"2023-05-22T15:47:03.082841+0000","last_peered":"2023-05-22T15:47:03.082841+0000","last_clean":"2023-05-22T15:47:03.082841+0000","last_became_active":"2023-05-22T15:47:03.076461+0000","last_became_peered":"2023-05-22T15:47:03.076461+0000","last_unstale":"2023-05-22T15:47:03.082841+0000","last_undegraded":"2023-05-22T15:47:03.082841+0000","last_fullsized":"2023-05-22T15:47:03.082841+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_clean_scrub_stamp":"2023-05-22T15:47:02.606868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T03:08:55.264476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.3a","version":"0'0","reported_seq":9,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-05-22T15:47:03.094946+0000","last_change":"2023-05-22T15:47:03.094209+0000","last_active":"2023-05-22T15:47:03.094946+0000","last_peered":"2023-05-22T15:47:03.094946+0000","last_clean":"2023-05-22T15:47:03.094946+0000","last_became_active":"2023-05-22T15:47:03.093954+0000","last_became_peered":"2023-05-22T15:47:03.093954+0000","last_unstale":"2023-05-22T15:47:03.094946+0000","last_undegraded":"2023-05-22T15:47:03.094946+0000","last_fullsized":"2023-05-22T15:47:03.094946+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_clean_scrub_stamp":"2023-05-22T15:47:02.606868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:33:18.873310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.39","version":"0'0","reported_seq":9,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-05-22T15:47:03.047868+0000","last_change":"2023-05-22T15:47:03.044426+0000","last_active":"2023-05-22T15:47:03.047868+0000","last_peered":"2023-05-22T15:47:03.047868+0000","last_clean":"2023-05-22T15:47:03.047868+0000","last_became_active":"2023-05-22T15:47:03.044112+0000","last_became_peered":"2023-05-22T15:47:03.044112+0000","last_unstale":"2023-05-22T15:47:03.047868+0000","last_undegraded":"2023-05-22T15:47:03.047868+0000","last_fullsized":"2023-05-22T15:47:03.047868+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_clean_scrub_stamp":"2023-05-22T15:47:02.606868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:11:03.531615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.38","version":"0'0","reported_seq":9,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-05-22T15:47:03.053029+0000","last_change":"2023-05-22T15:47:03.051971+0000","last_active":"2023-05-22T15:47:03.053029+0000","last_peered":"2023-05-22T15:47:03.053029+0000","last_clean":"2023-05-22T15:47:03.053029+0000","last_became_active":"2023-05-22T15:47:03.051613+0000","last_became_peered":"2023-05-22T15:47:03.051613+0000","last_unstale":"2023-05-22T15:47:03.053029+0000","last_undegraded":"2023-05-22T15:47:03.053029+0000","last_fullsized":"2023-05-22T15:47:03.053029+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_clean_scrub_stamp":"2023-05-22T15:47:02.606868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:53:22.500441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.37","version":"0'0","reported_seq":9,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-05-22T15:47:03.043208+0000","last_change":"2023-05-22T15:47:03.041188+0000","last_active":"2023-05-22T15:47:03.043208+0000","last_peered":"2023-05-22T15:47:03.043208+0000","last_clean":"2023-05-22T15:47:03.043208+0000","last_became_active":"2023-05-22T15:47:03.040929+0000","last_became_peered":"2023-05-22T15:47:03.040929+0000","last_unstale":"2023-05-22T15:47:03.043208+0000","last_undegraded":"2023-05-22T15:47:03.043208+0000","last_fullsized":"2023-05-22T15:47:03.043208+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_clean_scrub_stamp":"2023-05-22T15:47:02.606868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:38:13.016335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.36","version":"0'0","reported_seq":9,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-05-22T15:47:03.090999+0000","last_change":"2023-05-22T15:47:03.082621+0000","last_active":"2023-05-22T15:47:03.090999+0000","last_peered":"2023-05-22T15:47:03.090999+0000","last_clean":"2023-05-22T15:47:03.090999+0000","last_became_active":"2023-05-22T15:47:03.082040+0000","last_became_peered":"2023-05-22T15:47:03.082040+0000","last_unstale":"2023-05-22T15:47:03.090999+0000","last_undegraded":"2023-05-22T15:47:03.090999+0000","last_fullsized":"2023-05-22T15:47:03.090999+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_clean_scrub_stamp":"2023-05-22T15:47:02.606868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:01:11.576865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.35","version":"0'0","reported_seq":9,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-05-22T15:47:03.053009+0000","last_change":"2023-05-22T15:47:03.051978+0000","last_active":"2023-05-22T15:47:03.053009+0000","last_peered":"2023-05-22T15:47:03.053009+0000","last_clean":"2023-05-22T15:47:03.053009+0000","last_became_active":"2023-05-22T15:47:03.051666+0000","last_became_peered":"2023-05-22T15:47:03.051666+0000","last_unstale":"2023-05-22T15:47:03.053009+0000","last_undegraded":"2023-05-22T15:47:03.053009+0000","last_fullsized":"2023-05-22T15:47:03.053009+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_clean_scrub_stamp":"2023-05-22T15:47:02.606868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T16:03:39.232845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.34","version":"0'0","reported_seq":9,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-05-22T15:47:03.085407+0000","last_change":"2023-05-22T15:47:03.071738+0000","last_active":"2023-05-22T15:47:03.085407+0000","last_peered":"2023-05-22T15:47:03.085407+0000","last_clean":"2023-05-22T15:47:03.085407+0000","last_became_active":"2023-05-22T15:47:03.071046+0000","last_became_peered":"2023-05-22T15:47:03.071046+0000","last_unstale":"2023-05-22T15:47:03.085407+0000","last_undegraded":"2023-05-22T15:47:03.085407+0000","last_fullsized":"2023-05-22T15:47:03.085407+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_clean_scrub_stamp":"2023-05-22T15:47:02.606868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:44:06.543666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.33","version":"0'0","reported_seq":9,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-05-22T15:47:03.077377+0000","last_change":"2023-05-22T15:47:03.073487+0000","last_active":"2023-05-22T15:47:03.077377+0000","last_peered":"2023-05-22T15:47:03.077377+0000","last_clean":"2023-05-22T15:47:03.077377+0000","last_became_active":"2023-05-22T15:47:03.072913+0000","last_became_peered":"2023-05-22T15:47:03.072913+0000","last_unstale":"2023-05-22T15:47:03.077377+0000","last_undegraded":"2023-05-22T15:47:03.077377+0000","last_fullsized":"2023-05-22T15:47:03.077377+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_clean_scrub_stamp":"2023-05-22T15:47:02.606868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T16:39:32.806138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.32","version":"0'0","reported_seq":9,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-05-22T15:47:03.083712+0000","last_change":"2023-05-22T15:47:03.081205+0000","last_active":"2023-05-22T15:47:03.083712+0000","last_peered":"2023-05-22T15:47:03.083712+0000","last_clean":"2023-05-22T15:47:03.083712+0000","last_became_active":"2023-05-22T15:47:03.080579+0000","last_became_peered":"2023-05-22T15:47:03.080579+0000","last_unstale":"2023-05-22T15:47:03.083712+0000","last_undegraded":"2023-05-22T15:47:03.083712+0000","last_fullsized":"2023-05-22T15:47:03.083712+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_clean_scrub_stamp":"2023-05-22T15:47:02.606868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:04:53.425125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.31","version":"0'0","reported_seq":9,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-05-22T15:47:03.085348+0000","last_change":"2023-05-22T15:47:03.071738+0000","last_active":"2023-05-22T15:47:03.085348+0000","last_peered":"2023-05-22T15:47:03.085348+0000","last_clean":"2023-05-22T15:47:03.085348+0000","last_became_active":"2023-05-22T15:47:03.071137+0000","last_became_peered":"2023-05-22T15:47:03.071137+0000","last_unstale":"2023-05-22T15:47:03.085348+0000","last_undegraded":"2023-05-22T15:47:03.085348+0000","last_fullsized":"2023-05-22T15:47:03.085348+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_clean_scrub_stamp":"2023-05-22T15:47:02.606868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:04:53.425125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.30","version":"0'0","reported_seq":9,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-05-22T15:47:03.039838+0000","last_change":"2023-05-22T15:47:03.038261+0000","last_active":"2023-05-22T15:47:03.039838+0000","last_peered":"2023-05-22T15:47:03.039838+0000","last_clean":"2023-05-22T15:47:03.039838+0000","last_became_active":"2023-05-22T15:47:03.037968+0000","last_became_peered":"2023-05-22T15:47:03.037968+0000","last_unstale":"2023-05-22T15:47:03.039838+0000","last_undegraded":"2023-05-22T15:47:03.039838+0000","last_fullsized":"2023-05-22T15:47:03.039838+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_clean_scrub_stamp":"2023-05-22T15:47:02.606868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:52:37.159122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.2f","version":"0'0","reported_seq":9,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-05-22T15:47:03.090787+0000","last_change":"2023-05-22T15:47:03.077823+0000","last_active":"2023-05-22T15:47:03.090787+0000","last_peered":"2023-05-22T15:47:03.090787+0000","last_clean":"2023-05-22T15:47:03.090787+0000","last_became_active":"2023-05-22T15:47:03.074360+0000","last_became_peered":"2023-05-22T15:47:03.074360+0000","last_unstale":"2023-05-22T15:47:03.090787+0000","last_undegraded":"2023-05-22T15:47:03.090787+0000","last_fullsized":"2023-05-22T15:47:03.090787+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_clean_scrub_stamp":"2023-05-22T15:47:02.606868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T15:57:50.277224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.2e","version":"0'0","reported_seq":9,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-05-22T15:47:03.098310+0000","last_change":"2023-05-22T15:47:03.088461+0000","last_active":"2023-05-22T15:47:03.098310+0000","last_peered":"2023-05-22T15:47:03.098310+0000","last_clean":"2023-05-22T15:47:03.098310+0000","last_became_active":"2023-05-22T15:47:03.086415+0000","last_became_peered":"2023-05-22T15:47:03.086415+0000","last_unstale":"2023-05-22T15:47:03.098310+0000","last_undegraded":"2023-05-22T15:47:03.098310+0000","last_fullsized":"2023-05-22T15:47:03.098310+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_clean_scrub_stamp":"2023-05-22T15:47:02.606868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:07:06.587412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.2d","version":"0'0","reported_seq":9,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-05-22T15:47:03.040380+0000","last_change":"2023-05-22T15:47:03.037825+0000","last_active":"2023-05-22T15:47:03.040380+0000","last_peered":"2023-05-22T15:47:03.040380+0000","last_clean":"2023-05-22T15:47:03.040380+0000","last_became_active":"2023-05-22T15:47:03.037567+0000","last_became_peered":"2023-05-22T15:47:03.037567+0000","last_unstale":"2023-05-22T15:47:03.040380+0000","last_undegraded":"2023-05-22T15:47:03.040380+0000","last_fullsized":"2023-05-22T15:47:03.040380+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_clean_scrub_stamp":"2023-05-22T15:47:02.606868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T16:14:58.298182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.2c","version":"0'0","reported_seq":9,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-05-22T15:47:03.041610+0000","last_change":"2023-05-22T15:47:03.041008+0000","last_active":"2023-05-22T15:47:03.041610+0000","last_peered":"2023-05-22T15:47:03.041610+0000","last_clean":"2023-05-22T15:47:03.041610+0000","last_became_active":"2023-05-22T15:47:03.040700+0000","last_became_peered":"2023-05-22T15:47:03.040700+0000","last_unstale":"2023-05-22T15:47:03.041610+0000","last_undegraded":"2023-05-22T15:47:03.041610+0000","last_fullsized":"2023-05-22T15:47:03.041610+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_clean_scrub_stamp":"2023-05-22T15:47:02.606868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:48:50.437568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.2b","version":"0'0","reported_seq":9,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-05-22T15:47:03.090888+0000","last_change":"2023-05-22T15:47:03.081786+0000","last_active":"2023-05-22T15:47:03.090888+0000","last_peered":"2023-05-22T15:47:03.090888+0000","last_clean":"2023-05-22T15:47:03.090888+0000","last_became_active":"2023-05-22T15:47:03.081105+0000","last_became_peered":"2023-05-22T15:47:03.081105+0000","last_unstale":"2023-05-22T15:47:03.090888+0000","last_undegraded":"2023-05-22T15:47:03.090888+0000","last_fullsized":"2023-05-22T15:47:03.090888+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_clean_scrub_stamp":"2023-05-22T15:47:02.606868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:07:48.107370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.2a","version":"0'0","reported_seq":9,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-05-22T15:47:03.072036+0000","last_change":"2023-05-22T15:47:03.047875+0000","last_active":"2023-05-22T15:47:03.072036+0000","last_peered":"2023-05-22T15:47:03.072036+0000","last_clean":"2023-05-22T15:47:03.072036+0000","last_became_active":"2023-05-22T15:47:03.046928+0000","last_became_peered":"2023-05-22T15:47:03.046928+0000","last_unstale":"2023-05-22T15:47:03.072036+0000","last_undegraded":"2023-05-22T15:47:03.072036+0000","last_fullsized":"2023-05-22T15:47:03.072036+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_clean_scrub_stamp":"2023-05-22T15:47:02.606868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:56:02.336059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.29","version":"0'0","reported_seq":9,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-05-22T15:47:03.046843+0000","last_change":"2023-05-22T15:47:03.038559+0000","last_active":"2023-05-22T15:47:03.046843+0000","last_peered":"2023-05-22T15:47:03.046843+0000","last_clean":"2023-05-22T15:47:03.046843+0000","last_became_active":"2023-05-22T15:47:03.037536+0000","last_became_peered":"2023-05-22T15:47:03.037536+0000","last_unstale":"2023-05-22T15:47:03.046843+0000","last_undegraded":"2023-05-22T15:47:03.046843+0000","last_fullsized":"2023-05-22T15:47:03.046843+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_clean_scrub_stamp":"2023-05-22T15:47:02.606868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:44:41.653299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.28","version":"0'0","reported_seq":9,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-05-22T15:47:03.091006+0000","last_change":"2023-05-22T15:47:03.082635+0000","last_active":"2023-05-22T15:47:03.091006+0000","last_peered":"2023-05-22T15:47:03.091006+0000","last_clean":"2023-05-22T15:47:03.091006+0000","last_became_active":"2023-05-22T15:47:03.081929+0000","last_became_peered":"2023-05-22T15:47:03.081929+0000","last_unstale":"2023-05-22T15:47:03.091006+0000","last_undegraded":"2023-05-22T15:47:03.091006+0000","last_fullsized":"2023-05-22T15:47:03.091006+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_clean_scrub_stamp":"2023-05-22T15:47:02.606868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:47:06.786177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.27","version":"0'0","reported_seq":9,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-05-22T15:47:03.043145+0000","last_change":"2023-05-22T15:47:03.041937+0000","last_active":"2023-05-22T15:47:03.043145+0000","last_peered":"2023-05-22T15:47:03.043145+0000","last_clean":"2023-05-22T15:47:03.043145+0000","last_became_active":"2023-05-22T15:47:03.041609+0000","last_became_peered":"2023-05-22T15:47:03.041609+0000","last_unstale":"2023-05-22T15:47:03.043145+0000","last_undegraded":"2023-05-22T15:47:03.043145+0000","last_fullsized":"2023-05-22T15:47:03.043145+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_clean_scrub_stamp":"2023-05-22T15:47:02.606868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:55:53.695955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.26","version":"0'0","reported_seq":9,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-05-22T15:47:03.045868+0000","last_change":"2023-05-22T15:47:03.041186+0000","last_active":"2023-05-22T15:47:03.045868+0000","last_peered":"2023-05-22T15:47:03.045868+0000","last_clean":"2023-05-22T15:47:03.045868+0000","last_became_active":"2023-05-22T15:47:03.040921+0000","last_became_peered":"2023-05-22T15:47:03.040921+0000","last_unstale":"2023-05-22T15:47:03.045868+0000","last_undegraded":"2023-05-22T15:47:03.045868+0000","last_fullsized":"2023-05-22T15:47:03.045868+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_clean_scrub_stamp":"2023-05-22T15:47:02.606868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T16:02:03.177522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.25","version":"0'0","reported_seq":9,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-05-22T15:47:03.091528+0000","last_change":"2023-05-22T15:47:03.090733+0000","last_active":"2023-05-22T15:47:03.091528+0000","last_peered":"2023-05-22T15:47:03.091528+0000","last_clean":"2023-05-22T15:47:03.091528+0000","last_became_active":"2023-05-22T15:47:03.090411+0000","last_became_peered":"2023-05-22T15:47:03.090411+0000","last_unstale":"2023-05-22T15:47:03.091528+0000","last_undegraded":"2023-05-22T15:47:03.091528+0000","last_fullsized":"2023-05-22T15:47:03.091528+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_clean_scrub_stamp":"2023-05-22T15:47:02.606868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:10:35.110422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.24","version":"0'0","reported_seq":9,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-05-22T15:47:03.088241+0000","last_change":"2023-05-22T15:47:03.087558+0000","last_active":"2023-05-22T15:47:03.088241+0000","last_peered":"2023-05-22T15:47:03.088241+0000","last_clean":"2023-05-22T15:47:03.088241+0000","last_became_active":"2023-05-22T15:47:03.087209+0000","last_became_peered":"2023-05-22T15:47:03.087209+0000","last_unstale":"2023-05-22T15:47:03.088241+0000","last_undegraded":"2023-05-22T15:47:03.088241+0000","last_fullsized":"2023-05-22T15:47:03.088241+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_clean_scrub_stamp":"2023-05-22T15:47:02.606868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:06:44.653800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.23","version":"0'0","reported_seq":9,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-05-22T15:47:03.092043+0000","last_change":"2023-05-22T15:47:03.091438+0000","last_active":"2023-05-22T15:47:03.092043+0000","last_peered":"2023-05-22T15:47:03.092043+0000","last_clean":"2023-05-22T15:47:03.092043+0000","last_became_active":"2023-05-22T15:47:03.091107+0000","last_became_peered":"2023-05-22T15:47:03.091107+0000","last_unstale":"2023-05-22T15:47:03.092043+0000","last_undegraded":"2023-05-22T15:47:03.092043+0000","last_fullsized":"2023-05-22T15:47:03.092043+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_clean_scrub_stamp":"2023-05-22T15:47:02.606868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:03:49.167367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.22","version":"0'0","reported_seq":9,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-05-22T15:47:03.067544+0000","last_change":"2023-05-22T15:47:03.046229+0000","last_active":"2023-05-22T15:47:03.067544+0000","last_peered":"2023-05-22T15:47:03.067544+0000","last_clean":"2023-05-22T15:47:03.067544+0000","last_became_active":"2023-05-22T15:47:03.045295+0000","last_became_peered":"2023-05-22T15:47:03.045295+0000","last_unstale":"2023-05-22T15:47:03.067544+0000","last_undegraded":"2023-05-22T15:47:03.067544+0000","last_fullsized":"2023-05-22T15:47:03.067544+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_clean_scrub_stamp":"2023-05-22T15:47:02.606868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:42:53.822619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.21","version":"0'0","reported_seq":9,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-05-22T15:47:03.087850+0000","last_change":"2023-05-22T15:47:03.086358+0000","last_active":"2023-05-22T15:47:03.087850+0000","last_peered":"2023-05-22T15:47:03.087850+0000","last_clean":"2023-05-22T15:47:03.087850+0000","last_became_active":"2023-05-22T15:47:03.086013+0000","last_became_peered":"2023-05-22T15:47:03.086013+0000","last_unstale":"2023-05-22T15:47:03.087850+0000","last_undegraded":"2023-05-22T15:47:03.087850+0000","last_fullsized":"2023-05-22T15:47:03.087850+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_clean_scrub_stamp":"2023-05-22T15:47:02.606868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:27:02.805634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.20","version":"0'0","reported_seq":9,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-05-22T15:47:03.032633+0000","last_change":"2023-05-22T15:47:03.028936+0000","last_active":"2023-05-22T15:47:03.032633+0000","last_peered":"2023-05-22T15:47:03.032633+0000","last_clean":"2023-05-22T15:47:03.032633+0000","last_became_active":"2023-05-22T15:47:03.028654+0000","last_became_peered":"2023-05-22T15:47:03.028654+0000","last_unstale":"2023-05-22T15:47:03.032633+0000","last_undegraded":"2023-05-22T15:47:03.032633+0000","last_fullsized":"2023-05-22T15:47:03.032633+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_clean_scrub_stamp":"2023-05-22T15:47:02.606868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:23:25.039826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.1e","version":"0'0","reported_seq":9,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-05-22T15:47:03.085690+0000","last_change":"2023-05-22T15:47:03.073955+0000","last_active":"2023-05-22T15:47:03.085690+0000","last_peered":"2023-05-22T15:47:03.085690+0000","last_clean":"2023-05-22T15:47:03.085690+0000","last_became_active":"2023-05-22T15:47:03.072778+0000","last_became_peered":"2023-05-22T15:47:03.072778+0000","last_unstale":"2023-05-22T15:47:03.085690+0000","last_undegraded":"2023-05-22T15:47:03.085690+0000","last_fullsized":"2023-05-22T15:47:03.085690+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_clean_scrub_stamp":"2023-05-22T15:47:02.606868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:38:13.016335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1f","version":"160'4","reported_seq":14,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-05-22T15:47:03.078675+0000","last_change":"2023-05-22T15:47:02.688458+0000","last_active":"2023-05-22T15:47:03.078675+0000","last_peered":"2023-05-22T15:47:03.078675+0000","last_clean":"2023-05-22T15:47:03.078675+0000","last_became_active":"2023-05-22T15:47:02.686134+0000","last_became_peered":"2023-05-22T15:47:02.686134+0000","last_unstale":"2023-05-22T15:47:03.078675+0000","last_undegraded":"2023-05-22T15:47:03.078675+0000","last_fullsized":"2023-05-22T15:47:03.078675+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:47:01.602136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:47:01.602136+0000","last_clean_scrub_stamp":"2023-05-22T15:47:01.602136+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:22:32.490190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":566,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.1f","version":"0'0","reported_seq":9,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-05-22T15:47:03.043226+0000","last_change":"2023-05-22T15:47:03.041771+0000","last_active":"2023-05-22T15:47:03.043226+0000","last_peered":"2023-05-22T15:47:03.043226+0000","last_clean":"2023-05-22T15:47:03.043226+0000","last_became_active":"2023-05-22T15:47:03.041491+0000","last_became_peered":"2023-05-22T15:47:03.041491+0000","last_unstale":"2023-05-22T15:47:03.043226+0000","last_undegraded":"2023-05-22T15:47:03.043226+0000","last_fullsized":"2023-05-22T15:47:03.043226+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_clean_scrub_stamp":"2023-05-22T15:47:02.606868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:07:48.107370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.1e","version":"160'1","reported_seq":11,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-05-22T15:47:03.051041+0000","last_change":"2023-05-22T15:47:02.684398+0000","last_active":"2023-05-22T15:47:03.051041+0000","last_peered":"2023-05-22T15:47:03.051041+0000","last_clean":"2023-05-22T15:47:03.051041+0000","last_became_active":"2023-05-22T15:47:02.684028+0000","last_became_peered":"2023-05-22T15:47:02.684028+0000","last_unstale":"2023-05-22T15:47:03.051041+0000","last_undegraded":"2023-05-22T15:47:03.051041+0000","last_fullsized":"2023-05-22T15:47:03.051041+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:47:01.602136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:47:01.602136+0000","last_clean_scrub_stamp":"2023-05-22T15:47:01.602136+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:11:04.259677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.1c","version":"0'0","reported_seq":9,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-05-22T15:47:03.067453+0000","last_change":"2023-05-22T15:47:03.066707+0000","last_active":"2023-05-22T15:47:03.067453+0000","last_peered":"2023-05-22T15:47:03.067453+0000","last_clean":"2023-05-22T15:47:03.067453+0000","last_became_active":"2023-05-22T15:47:03.066422+0000","last_became_peered":"2023-05-22T15:47:03.066422+0000","last_unstale":"2023-05-22T15:47:03.067453+0000","last_undegraded":"2023-05-22T15:47:03.067453+0000","last_fullsized":"2023-05-22T15:47:03.067453+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_clean_scrub_stamp":"2023-05-22T15:47:02.606868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T03:11:13.112290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.1d","version":"160'2","reported_seq":12,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-05-22T15:47:03.066676+0000","last_change":"2023-05-22T15:47:02.666690+0000","last_active":"2023-05-22T15:47:03.066676+0000","last_peered":"2023-05-22T15:47:03.066676+0000","last_clean":"2023-05-22T15:47:03.066676+0000","last_became_active":"2023-05-22T15:47:02.665970+0000","last_became_peered":"2023-05-22T15:47:02.665970+0000","last_unstale":"2023-05-22T15:47:03.066676+0000","last_undegraded":"2023-05-22T15:47:03.066676+0000","last_fullsized":"2023-05-22T15:47:03.066676+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:47:01.602136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:47:01.602136+0000","last_clean_scrub_stamp":"2023-05-22T15:47:01.602136+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:15:35.219530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.1d","version":"0'0","reported_seq":9,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-05-22T15:47:03.088705+0000","last_change":"2023-05-22T15:47:03.075131+0000","last_active":"2023-05-22T15:47:03.088705+0000","last_peered":"2023-05-22T15:47:03.088705+0000","last_clean":"2023-05-22T15:47:03.088705+0000","last_became_active":"2023-05-22T15:47:03.074431+0000","last_became_peered":"2023-05-22T15:47:03.074431+0000","last_unstale":"2023-05-22T15:47:03.088705+0000","last_undegraded":"2023-05-22T15:47:03.088705+0000","last_fullsized":"2023-05-22T15:47:03.088705+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_clean_scrub_stamp":"2023-05-22T15:47:02.606868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:19:09.278066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1c","version":"160'2","reported_seq":12,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-05-22T15:47:03.067543+0000","last_change":"2023-05-22T15:47:02.663527+0000","last_active":"2023-05-22T15:47:03.067543+0000","last_peered":"2023-05-22T15:47:03.067543+0000","last_clean":"2023-05-22T15:47:03.067543+0000","last_became_active":"2023-05-22T15:47:02.662738+0000","last_became_peered":"2023-05-22T15:47:02.662738+0000","last_unstale":"2023-05-22T15:47:03.067543+0000","last_undegraded":"2023-05-22T15:47:03.067543+0000","last_fullsized":"2023-05-22T15:47:03.067543+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:47:01.602136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:47:01.602136+0000","last_clean_scrub_stamp":"2023-05-22T15:47:01.602136+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:46:40.385953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.1a","version":"0'0","reported_seq":9,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-05-22T15:47:03.087365+0000","last_change":"2023-05-22T15:47:03.086342+0000","last_active":"2023-05-22T15:47:03.087365+0000","last_peered":"2023-05-22T15:47:03.087365+0000","last_clean":"2023-05-22T15:47:03.087365+0000","last_became_active":"2023-05-22T15:47:03.086023+0000","last_became_peered":"2023-05-22T15:47:03.086023+0000","last_unstale":"2023-05-22T15:47:03.087365+0000","last_undegraded":"2023-05-22T15:47:03.087365+0000","last_fullsized":"2023-05-22T15:47:03.087365+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_clean_scrub_stamp":"2023-05-22T15:47:02.606868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:47:06.786177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.1b","version":"160'1","reported_seq":11,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-05-22T15:47:03.058406+0000","last_change":"2023-05-22T15:47:02.684478+0000","last_active":"2023-05-22T15:47:03.058406+0000","last_peered":"2023-05-22T15:47:03.058406+0000","last_clean":"2023-05-22T15:47:03.058406+0000","last_became_active":"2023-05-22T15:47:02.684125+0000","last_became_peered":"2023-05-22T15:47:02.684125+0000","last_unstale":"2023-05-22T15:47:03.058406+0000","last_undegraded":"2023-05-22T15:47:03.058406+0000","last_fullsized":"2023-05-22T15:47:03.058406+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:47:01.602136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:47:01.602136+0000","last_clean_scrub_stamp":"2023-05-22T15:47:01.602136+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T16:39:09.891365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.1b","version":"0'0","reported_seq":9,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-05-22T15:47:03.097134+0000","last_change":"2023-05-22T15:47:03.095177+0000","last_active":"2023-05-22T15:47:03.097134+0000","last_peered":"2023-05-22T15:47:03.097134+0000","last_clean":"2023-05-22T15:47:03.097134+0000","last_became_active":"2023-05-22T15:47:03.094713+0000","last_became_peered":"2023-05-22T15:47:03.094713+0000","last_unstale":"2023-05-22T15:47:03.097134+0000","last_undegraded":"2023-05-22T15:47:03.097134+0000","last_fullsized":"2023-05-22T15:47:03.097134+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_clean_scrub_stamp":"2023-05-22T15:47:02.606868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:44:06.543666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.1a","version":"160'1","reported_seq":11,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-05-22T15:47:03.062830+0000","last_change":"2023-05-22T15:47:02.671236+0000","last_active":"2023-05-22T15:47:03.062830+0000","last_peered":"2023-05-22T15:47:03.062830+0000","last_clean":"2023-05-22T15:47:03.062830+0000","last_became_active":"2023-05-22T15:47:02.670542+0000","last_became_peered":"2023-05-22T15:47:02.670542+0000","last_unstale":"2023-05-22T15:47:03.062830+0000","last_undegraded":"2023-05-22T15:47:03.062830+0000","last_fullsized":"2023-05-22T15:47:03.062830+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:47:01.602136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:47:01.602136+0000","last_clean_scrub_stamp":"2023-05-22T15:47:01.602136+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:03:53.697701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.18","version":"0'0","reported_seq":9,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-05-22T15:47:03.044665+0000","last_change":"2023-05-22T15:47:03.043930+0000","last_active":"2023-05-22T15:47:03.044665+0000","last_peered":"2023-05-22T15:47:03.044665+0000","last_clean":"2023-05-22T15:47:03.044665+0000","last_became_active":"2023-05-22T15:47:03.043647+0000","last_became_peered":"2023-05-22T15:47:03.043647+0000","last_unstale":"2023-05-22T15:47:03.044665+0000","last_undegraded":"2023-05-22T15:47:03.044665+0000","last_fullsized":"2023-05-22T15:47:03.044665+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_clean_scrub_stamp":"2023-05-22T15:47:02.606868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:54:58.772664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.19","version":"0'0","reported_seq":10,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-05-22T15:47:03.018820+0000","last_change":"2023-05-22T15:47:02.656840+0000","last_active":"2023-05-22T15:47:03.018820+0000","last_peered":"2023-05-22T15:47:03.018820+0000","last_clean":"2023-05-22T15:47:03.018820+0000","last_became_active":"2023-05-22T15:47:02.656449+0000","last_became_peered":"2023-05-22T15:47:02.656449+0000","last_unstale":"2023-05-22T15:47:03.018820+0000","last_undegraded":"2023-05-22T15:47:03.018820+0000","last_fullsized":"2023-05-22T15:47:03.018820+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:47:01.602136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:47:01.602136+0000","last_clean_scrub_stamp":"2023-05-22T15:47:01.602136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:59:48.010931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.19","version":"0'0","reported_seq":9,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-05-22T15:47:03.052557+0000","last_change":"2023-05-22T15:47:03.041807+0000","last_active":"2023-05-22T15:47:03.052557+0000","last_peered":"2023-05-22T15:47:03.052557+0000","last_clean":"2023-05-22T15:47:03.052557+0000","last_became_active":"2023-05-22T15:47:03.041060+0000","last_became_peered":"2023-05-22T15:47:03.041060+0000","last_unstale":"2023-05-22T15:47:03.052557+0000","last_undegraded":"2023-05-22T15:47:03.052557+0000","last_fullsized":"2023-05-22T15:47:03.052557+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_clean_scrub_stamp":"2023-05-22T15:47:02.606868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:31:19.675774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.18","version":"0'0","reported_seq":10,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-05-22T15:47:03.033095+0000","last_change":"2023-05-22T15:47:02.681905+0000","last_active":"2023-05-22T15:47:03.033095+0000","last_peered":"2023-05-22T15:47:03.033095+0000","last_clean":"2023-05-22T15:47:03.033095+0000","last_became_active":"2023-05-22T15:47:02.681167+0000","last_became_peered":"2023-05-22T15:47:02.681167+0000","last_unstale":"2023-05-22T15:47:03.033095+0000","last_undegraded":"2023-05-22T15:47:03.033095+0000","last_fullsized":"2023-05-22T15:47:03.033095+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:47:01.602136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:47:01.602136+0000","last_clean_scrub_stamp":"2023-05-22T15:47:01.602136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:31:26.628654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.e","version":"0'0","reported_seq":9,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-05-22T15:47:03.085727+0000","last_change":"2023-05-22T15:47:03.074417+0000","last_active":"2023-05-22T15:47:03.085727+0000","last_peered":"2023-05-22T15:47:03.085727+0000","last_clean":"2023-05-22T15:47:03.085727+0000","last_became_active":"2023-05-22T15:47:03.073379+0000","last_became_peered":"2023-05-22T15:47:03.073379+0000","last_unstale":"2023-05-22T15:47:03.085727+0000","last_undegraded":"2023-05-22T15:47:03.085727+0000","last_fullsized":"2023-05-22T15:47:03.085727+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_clean_scrub_stamp":"2023-05-22T15:47:02.606868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:54:58.772664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.f","version":"160'1","reported_seq":11,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-05-22T15:47:03.088606+0000","last_change":"2023-05-22T15:47:02.658045+0000","last_active":"2023-05-22T15:47:03.088606+0000","last_peered":"2023-05-22T15:47:03.088606+0000","last_clean":"2023-05-22T15:47:03.088606+0000","last_became_active":"2023-05-22T15:47:02.657283+0000","last_became_peered":"2023-05-22T15:47:02.657283+0000","last_unstale":"2023-05-22T15:47:03.088606+0000","last_undegraded":"2023-05-22T15:47:03.088606+0000","last_fullsized":"2023-05-22T15:47:03.088606+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:47:01.602136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:47:01.602136+0000","last_clean_scrub_stamp":"2023-05-22T15:47:01.602136+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T15:59:56.722776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.f","version":"0'0","reported_seq":9,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-05-22T15:47:03.071873+0000","last_change":"2023-05-22T15:47:03.071253+0000","last_active":"2023-05-22T15:47:03.071873+0000","last_peered":"2023-05-22T15:47:03.071873+0000","last_clean":"2023-05-22T15:47:03.071873+0000","last_became_active":"2023-05-22T15:47:03.071044+0000","last_became_peered":"2023-05-22T15:47:03.071044+0000","last_unstale":"2023-05-22T15:47:03.071873+0000","last_undegraded":"2023-05-22T15:47:03.071873+0000","last_fullsized":"2023-05-22T15:47:03.071873+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_clean_scrub_stamp":"2023-05-22T15:47:02.606868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:33:18.873310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.e","version":"160'2","reported_seq":12,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-05-22T15:47:03.094140+0000","last_change":"2023-05-22T15:47:02.656792+0000","last_active":"2023-05-22T15:47:03.094140+0000","last_peered":"2023-05-22T15:47:03.094140+0000","last_clean":"2023-05-22T15:47:03.094140+0000","last_became_active":"2023-05-22T15:47:02.655920+0000","last_became_peered":"2023-05-22T15:47:02.655920+0000","last_unstale":"2023-05-22T15:47:03.094140+0000","last_undegraded":"2023-05-22T15:47:03.094140+0000","last_fullsized":"2023-05-22T15:47:03.094140+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:47:01.602136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:47:01.602136+0000","last_clean_scrub_stamp":"2023-05-22T15:47:01.602136+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T18:18:04.406735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.c","version":"0'0","reported_seq":9,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-05-22T15:47:03.098195+0000","last_change":"2023-05-22T15:47:03.087108+0000","last_active":"2023-05-22T15:47:03.098195+0000","last_peered":"2023-05-22T15:47:03.098195+0000","last_clean":"2023-05-22T15:47:03.098195+0000","last_became_active":"2023-05-22T15:47:03.086489+0000","last_became_peered":"2023-05-22T15:47:03.086489+0000","last_unstale":"2023-05-22T15:47:03.098195+0000","last_undegraded":"2023-05-22T15:47:03.098195+0000","last_fullsized":"2023-05-22T15:47:03.098195+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_clean_scrub_stamp":"2023-05-22T15:47:02.606868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T03:10:01.167398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.d","version":"160'1","reported_seq":11,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-05-22T15:47:03.058401+0000","last_change":"2023-05-22T15:47:02.679126+0000","last_active":"2023-05-22T15:47:03.058401+0000","last_peered":"2023-05-22T15:47:03.058401+0000","last_clean":"2023-05-22T15:47:03.058401+0000","last_became_active":"2023-05-22T15:47:02.678819+0000","last_became_peered":"2023-05-22T15:47:02.678819+0000","last_unstale":"2023-05-22T15:47:03.058401+0000","last_undegraded":"2023-05-22T15:47:03.058401+0000","last_fullsized":"2023-05-22T15:47:03.058401+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:47:01.602136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:47:01.602136+0000","last_clean_scrub_stamp":"2023-05-22T15:47:01.602136+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:52:36.659001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.d","version":"0'0","reported_seq":9,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-05-22T15:47:03.095380+0000","last_change":"2023-05-22T15:47:03.094345+0000","last_active":"2023-05-22T15:47:03.095380+0000","last_peered":"2023-05-22T15:47:03.095380+0000","last_clean":"2023-05-22T15:47:03.095380+0000","last_became_active":"2023-05-22T15:47:03.094059+0000","last_became_peered":"2023-05-22T15:47:03.094059+0000","last_unstale":"2023-05-22T15:47:03.095380+0000","last_undegraded":"2023-05-22T15:47:03.095380+0000","last_fullsized":"2023-05-22T15:47:03.095380+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_clean_scrub_stamp":"2023-05-22T15:47:02.606868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:20:31.328410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.c","version":"0'0","reported_seq":10,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-05-22T15:47:03.030699+0000","last_change":"2023-05-22T15:47:02.656940+0000","last_active":"2023-05-22T15:47:03.030699+0000","last_peered":"2023-05-22T15:47:03.030699+0000","last_clean":"2023-05-22T15:47:03.030699+0000","last_became_active":"2023-05-22T15:47:02.656439+0000","last_became_peered":"2023-05-22T15:47:02.656439+0000","last_unstale":"2023-05-22T15:47:03.030699+0000","last_undegraded":"2023-05-22T15:47:03.030699+0000","last_fullsized":"2023-05-22T15:47:03.030699+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:47:01.602136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:47:01.602136+0000","last_clean_scrub_stamp":"2023-05-22T15:47:01.602136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T16:47:26.602773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.a","version":"0'0","reported_seq":9,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-05-22T15:47:03.067489+0000","last_change":"2023-05-22T15:47:03.045975+0000","last_active":"2023-05-22T15:47:03.067489+0000","last_peered":"2023-05-22T15:47:03.067489+0000","last_clean":"2023-05-22T15:47:03.067489+0000","last_became_active":"2023-05-22T15:47:03.045079+0000","last_became_peered":"2023-05-22T15:47:03.045079+0000","last_unstale":"2023-05-22T15:47:03.067489+0000","last_undegraded":"2023-05-22T15:47:03.067489+0000","last_fullsized":"2023-05-22T15:47:03.067489+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_clean_scrub_stamp":"2023-05-22T15:47:02.606868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:19:39.903101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.b","version":"160'2","reported_seq":12,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-05-22T15:47:03.091202+0000","last_change":"2023-05-22T15:47:02.647704+0000","last_active":"2023-05-22T15:47:03.091202+0000","last_peered":"2023-05-22T15:47:03.091202+0000","last_clean":"2023-05-22T15:47:03.091202+0000","last_became_active":"2023-05-22T15:47:02.647016+0000","last_became_peered":"2023-05-22T15:47:02.647016+0000","last_unstale":"2023-05-22T15:47:03.091202+0000","last_undegraded":"2023-05-22T15:47:03.091202+0000","last_fullsized":"2023-05-22T15:47:03.091202+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:47:01.602136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:47:01.602136+0000","last_clean_scrub_stamp":"2023-05-22T15:47:01.602136+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T18:19:56.170763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.b","version":"0'0","reported_seq":9,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-05-22T15:47:03.048875+0000","last_change":"2023-05-22T15:47:03.046456+0000","last_active":"2023-05-22T15:47:03.048875+0000","last_peered":"2023-05-22T15:47:03.048875+0000","last_clean":"2023-05-22T15:47:03.048875+0000","last_became_active":"2023-05-22T15:47:03.046105+0000","last_became_peered":"2023-05-22T15:47:03.046105+0000","last_unstale":"2023-05-22T15:47:03.048875+0000","last_undegraded":"2023-05-22T15:47:03.048875+0000","last_fullsized":"2023-05-22T15:47:03.048875+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_clean_scrub_stamp":"2023-05-22T15:47:02.606868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:33:09.856999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.a","version":"0'0","reported_seq":10,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-05-22T15:47:03.062300+0000","last_change":"2023-05-22T15:47:02.686740+0000","last_active":"2023-05-22T15:47:03.062300+0000","last_peered":"2023-05-22T15:47:03.062300+0000","last_clean":"2023-05-22T15:47:03.062300+0000","last_became_active":"2023-05-22T15:47:02.685976+0000","last_became_peered":"2023-05-22T15:47:02.685976+0000","last_unstale":"2023-05-22T15:47:03.062300+0000","last_undegraded":"2023-05-22T15:47:03.062300+0000","last_fullsized":"2023-05-22T15:47:03.062300+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:47:01.602136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:47:01.602136+0000","last_clean_scrub_stamp":"2023-05-22T15:47:01.602136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:31:26.628654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.8","version":"0'0","reported_seq":9,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-05-22T15:47:03.070563+0000","last_change":"2023-05-22T15:47:03.048016+0000","last_active":"2023-05-22T15:47:03.070563+0000","last_peered":"2023-05-22T15:47:03.070563+0000","last_clean":"2023-05-22T15:47:03.070563+0000","last_became_active":"2023-05-22T15:47:03.047176+0000","last_became_peered":"2023-05-22T15:47:03.047176+0000","last_unstale":"2023-05-22T15:47:03.070563+0000","last_undegraded":"2023-05-22T15:47:03.070563+0000","last_fullsized":"2023-05-22T15:47:03.070563+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_clean_scrub_stamp":"2023-05-22T15:47:02.606868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T15:57:50.277224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.9","version":"0'0","reported_seq":10,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-05-22T15:47:03.050651+0000","last_change":"2023-05-22T15:47:02.687780+0000","last_active":"2023-05-22T15:47:03.050651+0000","last_peered":"2023-05-22T15:47:03.050651+0000","last_clean":"2023-05-22T15:47:03.050651+0000","last_became_active":"2023-05-22T15:47:02.686009+0000","last_became_peered":"2023-05-22T15:47:02.686009+0000","last_unstale":"2023-05-22T15:47:03.050651+0000","last_undegraded":"2023-05-22T15:47:03.050651+0000","last_fullsized":"2023-05-22T15:47:03.050651+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:47:01.602136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:47:01.602136+0000","last_clean_scrub_stamp":"2023-05-22T15:47:01.602136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:46:40.385953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.9","version":"0'0","reported_seq":9,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-05-22T15:47:03.047880+0000","last_change":"2023-05-22T15:47:03.044434+0000","last_active":"2023-05-22T15:47:03.047880+0000","last_peered":"2023-05-22T15:47:03.047880+0000","last_clean":"2023-05-22T15:47:03.047880+0000","last_became_active":"2023-05-22T15:47:03.043979+0000","last_became_peered":"2023-05-22T15:47:03.043979+0000","last_unstale":"2023-05-22T15:47:03.047880+0000","last_undegraded":"2023-05-22T15:47:03.047880+0000","last_fullsized":"2023-05-22T15:47:03.047880+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_clean_scrub_stamp":"2023-05-22T15:47:02.606868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:09:07.199478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.8","version":"0'0","reported_seq":10,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-05-22T15:47:03.074899+0000","last_change":"2023-05-22T15:47:02.657585+0000","last_active":"2023-05-22T15:47:03.074899+0000","last_peered":"2023-05-22T15:47:03.074899+0000","last_clean":"2023-05-22T15:47:03.074899+0000","last_became_active":"2023-05-22T15:47:02.656859+0000","last_became_peered":"2023-05-22T15:47:02.656859+0000","last_unstale":"2023-05-22T15:47:03.074899+0000","last_undegraded":"2023-05-22T15:47:03.074899+0000","last_fullsized":"2023-05-22T15:47:03.074899+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:47:01.602136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:47:01.602136+0000","last_clean_scrub_stamp":"2023-05-22T15:47:01.602136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:48:49.432836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.6","version":"0'0","reported_seq":9,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-05-22T15:47:03.080252+0000","last_change":"2023-05-22T15:47:03.073859+0000","last_active":"2023-05-22T15:47:03.080252+0000","last_peered":"2023-05-22T15:47:03.080252+0000","last_clean":"2023-05-22T15:47:03.080252+0000","last_became_active":"2023-05-22T15:47:03.073287+0000","last_became_peered":"2023-05-22T15:47:03.073287+0000","last_unstale":"2023-05-22T15:47:03.080252+0000","last_undegraded":"2023-05-22T15:47:03.080252+0000","last_fullsized":"2023-05-22T15:47:03.080252+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_clean_scrub_stamp":"2023-05-22T15:47:02.606868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T03:26:44.703194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.7","version":"160'2","reported_seq":12,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-05-22T15:47:03.094502+0000","last_change":"2023-05-22T15:47:02.688423+0000","last_active":"2023-05-22T15:47:03.094502+0000","last_peered":"2023-05-22T15:47:03.094502+0000","last_clean":"2023-05-22T15:47:03.094502+0000","last_became_active":"2023-05-22T15:47:02.686410+0000","last_became_peered":"2023-05-22T15:47:02.686410+0000","last_unstale":"2023-05-22T15:47:03.094502+0000","last_undegraded":"2023-05-22T15:47:03.094502+0000","last_fullsized":"2023-05-22T15:47:03.094502+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:47:01.602136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:47:01.602136+0000","last_clean_scrub_stamp":"2023-05-22T15:47:01.602136+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:12:10.461479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.7","version":"0'0","reported_seq":9,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-05-22T15:47:03.098292+0000","last_change":"2023-05-22T15:47:03.088246+0000","last_active":"2023-05-22T15:47:03.098292+0000","last_peered":"2023-05-22T15:47:03.098292+0000","last_clean":"2023-05-22T15:47:03.098292+0000","last_became_active":"2023-05-22T15:47:03.086815+0000","last_became_peered":"2023-05-22T15:47:03.086815+0000","last_unstale":"2023-05-22T15:47:03.098292+0000","last_undegraded":"2023-05-22T15:47:03.098292+0000","last_fullsized":"2023-05-22T15:47:03.098292+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_clean_scrub_stamp":"2023-05-22T15:47:02.606868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:19:52.996207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.6","version":"0'0","reported_seq":10,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-05-22T15:47:03.037761+0000","last_change":"2023-05-22T15:47:02.678858+0000","last_active":"2023-05-22T15:47:03.037761+0000","last_peered":"2023-05-22T15:47:03.037761+0000","last_clean":"2023-05-22T15:47:03.037761+0000","last_became_active":"2023-05-22T15:47:02.678514+0000","last_became_peered":"2023-05-22T15:47:02.678514+0000","last_unstale":"2023-05-22T15:47:03.037761+0000","last_undegraded":"2023-05-22T15:47:03.037761+0000","last_fullsized":"2023-05-22T15:47:03.037761+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:47:01.602136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:47:01.602136+0000","last_clean_scrub_stamp":"2023-05-22T15:47:01.602136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:31:18.671042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.4","version":"0'0","reported_seq":9,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-05-22T15:47:03.048540+0000","last_change":"2023-05-22T15:47:03.047949+0000","last_active":"2023-05-22T15:47:03.048540+0000","last_peered":"2023-05-22T15:47:03.048540+0000","last_clean":"2023-05-22T15:47:03.048540+0000","last_became_active":"2023-05-22T15:47:03.047587+0000","last_became_peered":"2023-05-22T15:47:03.047587+0000","last_unstale":"2023-05-22T15:47:03.048540+0000","last_undegraded":"2023-05-22T15:47:03.048540+0000","last_fullsized":"2023-05-22T15:47:03.048540+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_clean_scrub_stamp":"2023-05-22T15:47:02.606868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:19:09.278066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.5","version":"0'0","reported_seq":10,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-05-22T15:47:03.034791+0000","last_change":"2023-05-22T15:47:02.659857+0000","last_active":"2023-05-22T15:47:03.034791+0000","last_peered":"2023-05-22T15:47:03.034791+0000","last_clean":"2023-05-22T15:47:03.034791+0000","last_became_active":"2023-05-22T15:47:02.658823+0000","last_became_peered":"2023-05-22T15:47:02.658823+0000","last_unstale":"2023-05-22T15:47:03.034791+0000","last_undegraded":"2023-05-22T15:47:03.034791+0000","last_fullsized":"2023-05-22T15:47:03.034791+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:47:01.602136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:47:01.602136+0000","last_clean_scrub_stamp":"2023-05-22T15:47:01.602136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:22:17.743430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.5","version":"0'0","reported_seq":9,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-05-22T15:47:03.067155+0000","last_change":"2023-05-22T15:47:03.066466+0000","last_active":"2023-05-22T15:47:03.067155+0000","last_peered":"2023-05-22T15:47:03.067155+0000","last_clean":"2023-05-22T15:47:03.067155+0000","last_became_active":"2023-05-22T15:47:03.066191+0000","last_became_peered":"2023-05-22T15:47:03.066191+0000","last_unstale":"2023-05-22T15:47:03.067155+0000","last_undegraded":"2023-05-22T15:47:03.067155+0000","last_fullsized":"2023-05-22T15:47:03.067155+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_clean_scrub_stamp":"2023-05-22T15:47:02.606868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:17:27.515854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.4","version":"0'0","reported_seq":10,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-05-22T15:47:03.037870+0000","last_change":"2023-05-22T15:47:02.658286+0000","last_active":"2023-05-22T15:47:03.037870+0000","last_peered":"2023-05-22T15:47:03.037870+0000","last_clean":"2023-05-22T15:47:03.037870+0000","last_became_active":"2023-05-22T15:47:02.657956+0000","last_became_peered":"2023-05-22T15:47:02.657956+0000","last_unstale":"2023-05-22T15:47:03.037870+0000","last_undegraded":"2023-05-22T15:47:03.037870+0000","last_fullsized":"2023-05-22T15:47:03.037870+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:47:01.602136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:47:01.602136+0000","last_clean_scrub_stamp":"2023-05-22T15:47:01.602136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:37:27.882115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.2","version":"0'0","reported_seq":9,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-05-22T15:47:03.047838+0000","last_change":"2023-05-22T15:47:03.045741+0000","last_active":"2023-05-22T15:47:03.047838+0000","last_peered":"2023-05-22T15:47:03.047838+0000","last_clean":"2023-05-22T15:47:03.047838+0000","last_became_active":"2023-05-22T15:47:03.045482+0000","last_became_peered":"2023-05-22T15:47:03.045482+0000","last_unstale":"2023-05-22T15:47:03.047838+0000","last_undegraded":"2023-05-22T15:47:03.047838+0000","last_fullsized":"2023-05-22T15:47:03.047838+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_clean_scrub_stamp":"2023-05-22T15:47:02.606868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:18:51.165098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.3","version":"0'0","reported_seq":10,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-05-22T15:47:03.027107+0000","last_change":"2023-05-22T15:47:02.666096+0000","last_active":"2023-05-22T15:47:03.027107+0000","last_peered":"2023-05-22T15:47:03.027107+0000","last_clean":"2023-05-22T15:47:03.027107+0000","last_became_active":"2023-05-22T15:47:02.665849+0000","last_became_peered":"2023-05-22T15:47:02.665849+0000","last_unstale":"2023-05-22T15:47:03.027107+0000","last_undegraded":"2023-05-22T15:47:03.027107+0000","last_fullsized":"2023-05-22T15:47:03.027107+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:47:01.602136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:47:01.602136+0000","last_clean_scrub_stamp":"2023-05-22T15:47:01.602136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:29:44.189527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.3","version":"0'0","reported_seq":9,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-05-22T15:47:03.096207+0000","last_change":"2023-05-22T15:47:03.093524+0000","last_active":"2023-05-22T15:47:03.096207+0000","last_peered":"2023-05-22T15:47:03.096207+0000","last_clean":"2023-05-22T15:47:03.096207+0000","last_became_active":"2023-05-22T15:47:03.092890+0000","last_became_peered":"2023-05-22T15:47:03.092890+0000","last_unstale":"2023-05-22T15:47:03.096207+0000","last_undegraded":"2023-05-22T15:47:03.096207+0000","last_fullsized":"2023-05-22T15:47:03.096207+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_clean_scrub_stamp":"2023-05-22T15:47:02.606868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:37:28.886847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.2","version":"0'0","reported_seq":10,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-05-22T15:47:03.062458+0000","last_change":"2023-05-22T15:47:02.687873+0000","last_active":"2023-05-22T15:47:03.062458+0000","last_peered":"2023-05-22T15:47:03.062458+0000","last_clean":"2023-05-22T15:47:03.062458+0000","last_became_active":"2023-05-22T15:47:02.687243+0000","last_became_peered":"2023-05-22T15:47:02.687243+0000","last_unstale":"2023-05-22T15:47:03.062458+0000","last_undegraded":"2023-05-22T15:47:03.062458+0000","last_fullsized":"2023-05-22T15:47:03.062458+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:47:01.602136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:47:01.602136+0000","last_clean_scrub_stamp":"2023-05-22T15:47:01.602136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:15:35.219530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.0","version":"0'0","reported_seq":9,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-05-22T15:47:03.085643+0000","last_change":"2023-05-22T15:47:03.073766+0000","last_active":"2023-05-22T15:47:03.085643+0000","last_peered":"2023-05-22T15:47:03.085643+0000","last_clean":"2023-05-22T15:47:03.085643+0000","last_became_active":"2023-05-22T15:47:03.072937+0000","last_became_peered":"2023-05-22T15:47:03.072937+0000","last_unstale":"2023-05-22T15:47:03.085643+0000","last_undegraded":"2023-05-22T15:47:03.085643+0000","last_fullsized":"2023-05-22T15:47:03.085643+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_clean_scrub_stamp":"2023-05-22T15:47:02.606868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:27:02.805634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1","version":"0'0","reported_seq":10,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-05-22T15:47:03.075193+0000","last_change":"2023-05-22T15:47:02.651613+0000","last_active":"2023-05-22T15:47:03.075193+0000","last_peered":"2023-05-22T15:47:03.075193+0000","last_clean":"2023-05-22T15:47:03.075193+0000","last_became_active":"2023-05-22T15:47:02.650779+0000","last_became_peered":"2023-05-22T15:47:02.650779+0000","last_unstale":"2023-05-22T15:47:03.075193+0000","last_undegraded":"2023-05-22T15:47:03.075193+0000","last_fullsized":"2023-05-22T15:47:03.075193+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:47:01.602136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:47:01.602136+0000","last_clean_scrub_stamp":"2023-05-22T15:47:01.602136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:23:24.035094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.1","version":"0'0","reported_seq":9,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-05-22T15:47:03.044679+0000","last_change":"2023-05-22T15:47:03.038439+0000","last_active":"2023-05-22T15:47:03.044679+0000","last_peered":"2023-05-22T15:47:03.044679+0000","last_clean":"2023-05-22T15:47:03.044679+0000","last_became_active":"2023-05-22T15:47:03.037663+0000","last_became_peered":"2023-05-22T15:47:03.037663+0000","last_unstale":"2023-05-22T15:47:03.044679+0000","last_undegraded":"2023-05-22T15:47:03.044679+0000","last_fullsized":"2023-05-22T15:47:03.044679+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_clean_scrub_stamp":"2023-05-22T15:47:02.606868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T03:43:14.105010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.0","version":"0'0","reported_seq":10,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-05-22T15:47:03.030082+0000","last_change":"2023-05-22T15:47:02.678937+0000","last_active":"2023-05-22T15:47:03.030082+0000","last_peered":"2023-05-22T15:47:03.030082+0000","last_clean":"2023-05-22T15:47:03.030082+0000","last_became_active":"2023-05-22T15:47:02.678508+0000","last_became_peered":"2023-05-22T15:47:02.678508+0000","last_unstale":"2023-05-22T15:47:03.030082+0000","last_undegraded":"2023-05-22T15:47:03.030082+0000","last_fullsized":"2023-05-22T15:47:03.030082+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:47:01.602136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:47:01.602136+0000","last_clean_scrub_stamp":"2023-05-22T15:47:01.602136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:39:56.802043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.14","version":"0'0","reported_seq":9,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-05-22T15:47:03.048892+0000","last_change":"2023-05-22T15:47:03.046491+0000","last_active":"2023-05-22T15:47:03.048892+0000","last_peered":"2023-05-22T15:47:03.048892+0000","last_clean":"2023-05-22T15:47:03.048892+0000","last_became_active":"2023-05-22T15:47:03.046190+0000","last_became_peered":"2023-05-22T15:47:03.046190+0000","last_unstale":"2023-05-22T15:47:03.048892+0000","last_undegraded":"2023-05-22T15:47:03.048892+0000","last_fullsized":"2023-05-22T15:47:03.048892+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_clean_scrub_stamp":"2023-05-22T15:47:02.606868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T18:38:26.235383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.15","version":"160'3","reported_seq":13,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-05-22T15:47:03.101109+0000","last_change":"2023-05-22T15:47:02.658544+0000","last_active":"2023-05-22T15:47:03.101109+0000","last_peered":"2023-05-22T15:47:03.101109+0000","last_clean":"2023-05-22T15:47:03.101109+0000","last_became_active":"2023-05-22T15:47:02.657802+0000","last_became_peered":"2023-05-22T15:47:02.657802+0000","last_unstale":"2023-05-22T15:47:03.101109+0000","last_undegraded":"2023-05-22T15:47:03.101109+0000","last_fullsized":"2023-05-22T15:47:03.101109+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:47:01.602136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:47:01.602136+0000","last_clean_scrub_stamp":"2023-05-22T15:47:01.602136+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:18:50.160366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"149'271","reported_seq":845,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-05-22T15:47:02.675418+0000","last_change":"2023-05-22T15:26:39.318204+0000","last_active":"2023-05-22T15:47:02.675418+0000","last_peered":"2023-05-22T15:47:02.675418+0000","last_clean":"2023-05-22T15:47:02.675418+0000","last_became_active":"2023-05-22T15:26:39.315370+0000","last_became_peered":"2023-05-22T15:26:39.315370+0000","last_unstale":"2023-05-22T15:47:02.675418+0000","last_undegraded":"2023-05-22T15:47:02.675418+0000","last_fullsized":"2023-05-22T15:47:02.675418+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:38.020076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:38.020076+0000","last_clean_scrub_stamp":"2023-05-22T15:26:38.020076+0000","objects_scrubbed":0,"log_size":271,"log_dups_size":0,"ondisk_log_size":271,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:06:39.207543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1460,"num_read_kb":2984,"num_write":439,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":154,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-05-22T15:47:02.640185+0000","last_change":"2023-05-22T15:26:44.971951+0000","last_active":"2023-05-22T15:47:02.640185+0000","last_peered":"2023-05-22T15:47:02.640185+0000","last_clean":"2023-05-22T15:47:02.640185+0000","last_became_active":"2023-05-22T15:26:44.970872+0000","last_became_peered":"2023-05-22T15:26:44.970872+0000","last_unstale":"2023-05-22T15:47:02.640185+0000","last_undegraded":"2023-05-22T15:47:02.640185+0000","last_fullsized":"2023-05-22T15:47:02.640185+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:55:34.752949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.17","version":"0'0","reported_seq":9,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-05-22T15:47:03.088843+0000","last_change":"2023-05-22T15:47:03.076930+0000","last_active":"2023-05-22T15:47:03.088843+0000","last_peered":"2023-05-22T15:47:03.088843+0000","last_clean":"2023-05-22T15:47:03.088843+0000","last_became_active":"2023-05-22T15:47:03.074333+0000","last_became_peered":"2023-05-22T15:47:03.074333+0000","last_unstale":"2023-05-22T15:47:03.088843+0000","last_undegraded":"2023-05-22T15:47:03.088843+0000","last_fullsized":"2023-05-22T15:47:03.088843+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_clean_scrub_stamp":"2023-05-22T15:47:02.606868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:44:41.653299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.16","version":"0'0","reported_seq":10,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-05-22T15:47:03.059455+0000","last_change":"2023-05-22T15:47:02.686878+0000","last_active":"2023-05-22T15:47:03.059455+0000","last_peered":"2023-05-22T15:47:03.059455+0000","last_clean":"2023-05-22T15:47:03.059455+0000","last_became_active":"2023-05-22T15:47:02.685784+0000","last_became_peered":"2023-05-22T15:47:02.685784+0000","last_unstale":"2023-05-22T15:47:03.059455+0000","last_undegraded":"2023-05-22T15:47:03.059455+0000","last_fullsized":"2023-05-22T15:47:03.059455+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:47:01.602136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:47:01.602136+0000","last_clean_scrub_stamp":"2023-05-22T15:47:01.602136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:19:51.991474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":146,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-05-22T15:47:02.675162+0000","last_change":"2023-05-22T15:26:44.977198+0000","last_active":"2023-05-22T15:47:02.675162+0000","last_peered":"2023-05-22T15:47:02.675162+0000","last_clean":"2023-05-22T15:47:02.675162+0000","last_became_active":"2023-05-22T15:26:44.976445+0000","last_became_peered":"2023-05-22T15:26:44.976445+0000","last_unstale":"2023-05-22T15:47:02.675162+0000","last_undegraded":"2023-05-22T15:47:02.675162+0000","last_fullsized":"2023-05-22T15:47:02.675162+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:12:41.464054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.16","version":"0'0","reported_seq":9,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-05-22T15:47:03.052564+0000","last_change":"2023-05-22T15:47:03.040278+0000","last_active":"2023-05-22T15:47:03.052564+0000","last_peered":"2023-05-22T15:47:03.052564+0000","last_clean":"2023-05-22T15:47:03.052564+0000","last_became_active":"2023-05-22T15:47:03.039500+0000","last_became_peered":"2023-05-22T15:47:03.039500+0000","last_unstale":"2023-05-22T15:47:03.052564+0000","last_undegraded":"2023-05-22T15:47:03.052564+0000","last_fullsized":"2023-05-22T15:47:03.052564+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_clean_scrub_stamp":"2023-05-22T15:47:02.606868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:22:18.748162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.17","version":"0'0","reported_seq":10,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-05-22T15:47:03.036866+0000","last_change":"2023-05-22T15:47:02.653661+0000","last_active":"2023-05-22T15:47:03.036866+0000","last_peered":"2023-05-22T15:47:03.036866+0000","last_clean":"2023-05-22T15:47:03.036866+0000","last_became_active":"2023-05-22T15:47:02.653389+0000","last_became_peered":"2023-05-22T15:47:02.653389+0000","last_unstale":"2023-05-22T15:47:03.036866+0000","last_undegraded":"2023-05-22T15:47:03.036866+0000","last_fullsized":"2023-05-22T15:47:03.036866+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:47:01.602136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:47:01.602136+0000","last_clean_scrub_stamp":"2023-05-22T15:47:01.602136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:28:42.216703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":146,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-05-22T15:47:02.648441+0000","last_change":"2023-05-22T15:26:44.969469+0000","last_active":"2023-05-22T15:47:02.648441+0000","last_peered":"2023-05-22T15:47:02.648441+0000","last_clean":"2023-05-22T15:47:02.648441+0000","last_became_active":"2023-05-22T15:26:44.969083+0000","last_became_peered":"2023-05-22T15:26:44.969083+0000","last_unstale":"2023-05-22T15:47:02.648441+0000","last_undegraded":"2023-05-22T15:47:02.648441+0000","last_fullsized":"2023-05-22T15:47:02.648441+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:29:23.670067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.15","version":"0'0","reported_seq":9,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-05-22T15:47:03.030240+0000","last_change":"2023-05-22T15:47:03.026296+0000","last_active":"2023-05-22T15:47:03.030240+0000","last_peered":"2023-05-22T15:47:03.030240+0000","last_clean":"2023-05-22T15:47:03.030240+0000","last_became_active":"2023-05-22T15:47:03.025995+0000","last_became_peered":"2023-05-22T15:47:03.025995+0000","last_unstale":"2023-05-22T15:47:03.030240+0000","last_undegraded":"2023-05-22T15:47:03.030240+0000","last_fullsized":"2023-05-22T15:47:03.030240+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_clean_scrub_stamp":"2023-05-22T15:47:02.606868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T03:26:44.703194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.14","version":"160'6","reported_seq":16,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-05-22T15:47:03.060992+0000","last_change":"2023-05-22T15:47:02.652400+0000","last_active":"2023-05-22T15:47:03.060992+0000","last_peered":"2023-05-22T15:47:03.060992+0000","last_clean":"2023-05-22T15:47:03.060992+0000","last_became_active":"2023-05-22T15:47:02.652113+0000","last_became_peered":"2023-05-22T15:47:02.652113+0000","last_unstale":"2023-05-22T15:47:03.060992+0000","last_undegraded":"2023-05-22T15:47:03.060992+0000","last_fullsized":"2023-05-22T15:47:03.060992+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:47:01.602136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:47:01.602136+0000","last_clean_scrub_stamp":"2023-05-22T15:47:01.602136+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:18:47.057150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1548,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":154,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-05-22T15:47:02.636750+0000","last_change":"2023-05-22T15:26:44.976632+0000","last_active":"2023-05-22T15:47:02.636750+0000","last_peered":"2023-05-22T15:47:02.636750+0000","last_clean":"2023-05-22T15:47:02.636750+0000","last_became_active":"2023-05-22T15:26:44.975546+0000","last_became_peered":"2023-05-22T15:26:44.975546+0000","last_unstale":"2023-05-22T15:47:02.636750+0000","last_undegraded":"2023-05-22T15:47:02.636750+0000","last_fullsized":"2023-05-22T15:47:02.636750+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:42:57.973030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.13","version":"0'0","reported_seq":9,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-05-22T15:47:03.043174+0000","last_change":"2023-05-22T15:47:03.041807+0000","last_active":"2023-05-22T15:47:03.043174+0000","last_peered":"2023-05-22T15:47:03.043174+0000","last_clean":"2023-05-22T15:47:03.043174+0000","last_became_active":"2023-05-22T15:47:03.041516+0000","last_became_peered":"2023-05-22T15:47:03.041516+0000","last_unstale":"2023-05-22T15:47:03.043174+0000","last_undegraded":"2023-05-22T15:47:03.043174+0000","last_fullsized":"2023-05-22T15:47:03.043174+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_clean_scrub_stamp":"2023-05-22T15:47:02.606868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T15:51:01.546872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.12","version":"0'0","reported_seq":10,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-05-22T15:47:03.075255+0000","last_change":"2023-05-22T15:47:02.660522+0000","last_active":"2023-05-22T15:47:03.075255+0000","last_peered":"2023-05-22T15:47:03.075255+0000","last_clean":"2023-05-22T15:47:03.075255+0000","last_became_active":"2023-05-22T15:47:02.659535+0000","last_became_peered":"2023-05-22T15:47:02.659535+0000","last_unstale":"2023-05-22T15:47:03.075255+0000","last_undegraded":"2023-05-22T15:47:03.075255+0000","last_fullsized":"2023-05-22T15:47:03.075255+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:47:01.602136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:47:01.602136+0000","last_clean_scrub_stamp":"2023-05-22T15:47:01.602136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:55:52.691222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":135,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-05-22T15:47:02.645371+0000","last_change":"2023-05-22T15:26:45.091475+0000","last_active":"2023-05-22T15:47:02.645371+0000","last_peered":"2023-05-22T15:47:02.645371+0000","last_clean":"2023-05-22T15:47:02.645371+0000","last_became_active":"2023-05-22T15:26:45.091143+0000","last_became_peered":"2023-05-22T15:26:45.091143+0000","last_unstale":"2023-05-22T15:47:02.645371+0000","last_undegraded":"2023-05-22T15:47:02.645371+0000","last_fullsized":"2023-05-22T15:47:02.645371+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:12:41.464054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.12","version":"0'0","reported_seq":9,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-05-22T15:47:03.098274+0000","last_change":"2023-05-22T15:47:03.087767+0000","last_active":"2023-05-22T15:47:03.098274+0000","last_peered":"2023-05-22T15:47:03.098274+0000","last_clean":"2023-05-22T15:47:03.098274+0000","last_became_active":"2023-05-22T15:47:03.086853+0000","last_became_peered":"2023-05-22T15:47:03.086853+0000","last_unstale":"2023-05-22T15:47:03.098274+0000","last_undegraded":"2023-05-22T15:47:03.098274+0000","last_fullsized":"2023-05-22T15:47:03.098274+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_clean_scrub_stamp":"2023-05-22T15:47:02.606868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:46:41.390685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.13","version":"160'2","reported_seq":12,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-05-22T15:47:03.098510+0000","last_change":"2023-05-22T15:47:02.651919+0000","last_active":"2023-05-22T15:47:03.098510+0000","last_peered":"2023-05-22T15:47:03.098510+0000","last_clean":"2023-05-22T15:47:03.098510+0000","last_became_active":"2023-05-22T15:47:02.651121+0000","last_became_peered":"2023-05-22T15:47:02.651121+0000","last_unstale":"2023-05-22T15:47:03.098510+0000","last_undegraded":"2023-05-22T15:47:03.098510+0000","last_fullsized":"2023-05-22T15:47:03.098510+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:47:01.602136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:47:01.602136+0000","last_clean_scrub_stamp":"2023-05-22T15:47:01.602136+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:10:34.105690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":145,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-05-22T15:47:02.675258+0000","last_change":"2023-05-22T15:26:45.092367+0000","last_active":"2023-05-22T15:47:02.675258+0000","last_peered":"2023-05-22T15:47:02.675258+0000","last_clean":"2023-05-22T15:47:02.675258+0000","last_became_active":"2023-05-22T15:26:45.091951+0000","last_became_peered":"2023-05-22T15:26:45.091951+0000","last_unstale":"2023-05-22T15:47:02.675258+0000","last_undegraded":"2023-05-22T15:47:02.675258+0000","last_fullsized":"2023-05-22T15:47:02.675258+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:42:57.973030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.11","version":"0'0","reported_seq":9,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-05-22T15:47:03.098177+0000","last_change":"2023-05-22T15:47:03.086458+0000","last_active":"2023-05-22T15:47:03.098177+0000","last_peered":"2023-05-22T15:47:03.098177+0000","last_clean":"2023-05-22T15:47:03.098177+0000","last_became_active":"2023-05-22T15:47:03.080919+0000","last_became_peered":"2023-05-22T15:47:03.080919+0000","last_unstale":"2023-05-22T15:47:03.098177+0000","last_undegraded":"2023-05-22T15:47:03.098177+0000","last_fullsized":"2023-05-22T15:47:03.098177+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_clean_scrub_stamp":"2023-05-22T15:47:02.606868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:17:27.515854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.10","version":"160'8","reported_seq":18,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-05-22T15:47:03.094344+0000","last_change":"2023-05-22T15:47:02.650899+0000","last_active":"2023-05-22T15:47:03.094344+0000","last_peered":"2023-05-22T15:47:03.094344+0000","last_clean":"2023-05-22T15:47:03.094344+0000","last_became_active":"2023-05-22T15:47:02.648924+0000","last_became_peered":"2023-05-22T15:47:02.648924+0000","last_unstale":"2023-05-22T15:47:03.094344+0000","last_undegraded":"2023-05-22T15:47:03.094344+0000","last_fullsized":"2023-05-22T15:47:03.094344+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:47:01.602136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:47:01.602136+0000","last_clean_scrub_stamp":"2023-05-22T15:47:01.602136+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T18:22:29.880531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":135,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-05-22T15:47:02.645219+0000","last_change":"2023-05-22T15:26:44.978858+0000","last_active":"2023-05-22T15:47:02.645219+0000","last_peered":"2023-05-22T15:47:02.645219+0000","last_clean":"2023-05-22T15:47:02.645219+0000","last_became_active":"2023-05-22T15:26:44.978443+0000","last_became_peered":"2023-05-22T15:26:44.978443+0000","last_unstale":"2023-05-22T15:47:02.645219+0000","last_undegraded":"2023-05-22T15:47:02.645219+0000","last_fullsized":"2023-05-22T15:47:02.645219+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:46:29.290243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.10","version":"0'0","reported_seq":9,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-05-22T15:47:03.083577+0000","last_change":"2023-05-22T15:47:03.078634+0000","last_active":"2023-05-22T15:47:03.083577+0000","last_peered":"2023-05-22T15:47:03.083577+0000","last_clean":"2023-05-22T15:47:03.083577+0000","last_became_active":"2023-05-22T15:47:03.078040+0000","last_became_peered":"2023-05-22T15:47:03.078040+0000","last_unstale":"2023-05-22T15:47:03.083577+0000","last_undegraded":"2023-05-22T15:47:03.083577+0000","last_fullsized":"2023-05-22T15:47:03.083577+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_clean_scrub_stamp":"2023-05-22T15:47:02.606868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:40:02.459811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.11","version":"0'0","reported_seq":10,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-05-22T15:47:03.035082+0000","last_change":"2023-05-22T15:47:02.651915+0000","last_active":"2023-05-22T15:47:03.035082+0000","last_peered":"2023-05-22T15:47:03.035082+0000","last_clean":"2023-05-22T15:47:03.035082+0000","last_became_active":"2023-05-22T15:47:02.651560+0000","last_became_peered":"2023-05-22T15:47:02.651560+0000","last_unstale":"2023-05-22T15:47:03.035082+0000","last_undegraded":"2023-05-22T15:47:03.035082+0000","last_fullsized":"2023-05-22T15:47:03.035082+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:47:01.602136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:47:01.602136+0000","last_clean_scrub_stamp":"2023-05-22T15:47:01.602136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:31:47.844837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":136,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-05-22T15:47:02.649430+0000","last_change":"2023-05-22T15:26:44.982421+0000","last_active":"2023-05-22T15:47:02.649430+0000","last_peered":"2023-05-22T15:47:02.649430+0000","last_clean":"2023-05-22T15:47:02.649430+0000","last_became_active":"2023-05-22T15:26:44.981586+0000","last_became_peered":"2023-05-22T15:26:44.981586+0000","last_unstale":"2023-05-22T15:47:02.649430+0000","last_undegraded":"2023-05-22T15:47:02.649430+0000","last_fullsized":"2023-05-22T15:47:02.649430+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:14:11.156544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":21,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":20,"num_pg":32,"stat_sum":{"num_bytes":2330,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1460,"num_read_kb":2984,"num_write":439,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":271,"ondisk_log_size":271,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575091,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":20528,"kb_used_data":2536,"kb_used_omap":0,"kb_used_meta":17984,"kb_avail":94351312,"statfs":{"total":96636764160,"available":96615743488,"internally_reserved":0,"allocated":2596864,"data_stored":1647540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":18415616},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 15:46:11 2023","interfaces":[{"interface":"back","average":{"1min":0.86699999999999999,"5min":0.89200000000000002,"15min":0.876},"min":{"1min":0.53600000000000003,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":1.294,"5min":2.637,"15min":2.637},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.88,"5min":0.88700000000000001,"15min":0.89200000000000002},"min":{"1min":0.60799999999999998,"5min":0.502,"15min":0.35799999999999998},"max":{"1min":1.1519999999999999,"5min":3.0449999999999999,"15min":3.0449999999999999},"last":1.028}]},{"osd":1,"last update":"Mon May 22 15:46:11 2023","interfaces":[{"interface":"back","average":{"1min":0.96799999999999997,"5min":0.95399999999999996,"15min":0.97899999999999998},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.1899999999999999,"5min":1.385,"15min":1.389},"last":0.83999999999999997},{"interface":"front","average":{"1min":0.96799999999999997,"5min":1.006,"15min":1.022},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.54900000000000004},"max":{"1min":1.226,"5min":2.7269999999999999,"15min":2.7269999999999999},"last":0.98899999999999999}]},{"osd":2,"last update":"Mon May 22 15:46:11 2023","interfaces":[{"interface":"back","average":{"1min":0.86199999999999999,"5min":0.85099999999999998,"15min":0.871},"min":{"1min":0.55600000000000005,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.109,"5min":1.252,"15min":1.252},"last":0.95099999999999996},{"interface":"front","average":{"1min":0.88100000000000001,"5min":0.93999999999999995,"15min":0.95699999999999996},"min":{"1min":0.504,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.143,"5min":3.141,"15min":3.141},"last":0.72299999999999998}]},{"osd":3,"last update":"Mon May 22 15:46:11 2023","interfaces":[{"interface":"back","average":{"1min":0.98199999999999998,"5min":0.98899999999999999,"15min":0.96299999999999997},"min":{"1min":0.67600000000000005,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":1.3839999999999999,"5min":2.8149999999999999,"15min":2.8149999999999999},"last":0.79000000000000004},{"interface":"front","average":{"1min":0.94399999999999995,"5min":0.92100000000000004,"15min":0.96199999999999997},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.2709999999999999,"5min":1.389,"15min":1.4710000000000001},"last":0.747}]},{"osd":4,"last update":"Mon May 22 15:46:11 2023","interfaces":[{"interface":"back","average":{"1min":0.87,"5min":0.94399999999999995,"15min":0.90400000000000003},"min":{"1min":0.60299999999999998,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.5089999999999999,"5min":3.472,"15min":3.472},"last":0.62},{"interface":"front","average":{"1min":0.997,"5min":1.1850000000000001,"15min":1.05},"min":{"1min":0.59699999999999998,"5min":0.56000000000000005,"15min":0.44600000000000001},"max":{"1min":1.2949999999999999,"5min":15.627000000000001,"15min":15.627000000000001},"last":0.91200000000000003}]},{"osd":5,"last update":"Mon May 22 15:46:11 2023","interfaces":[{"interface":"back","average":{"1min":1.012,"5min":1.071,"15min":1.02},"min":{"1min":0.60899999999999999,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":1.4750000000000001,"5min":4.5599999999999996,"15min":4.5599999999999996},"last":1.071},{"interface":"front","average":{"1min":1.115,"5min":1.161,"15min":1.139},"min":{"1min":0.82999999999999996,"5min":0.63800000000000001,"15min":0.58799999999999997},"max":{"1min":1.302,"5min":4.3799999999999999,"15min":4.3799999999999999},"last":1.1839999999999999}]},{"osd":6,"last update":"Mon May 22 15:46:11 2023","interfaces":[{"interface":"back","average":{"1min":1.073,"5min":1.1399999999999999,"15min":1.0840000000000001},"min":{"1min":0.80300000000000005,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.415,"5min":6.5069999999999997,"15min":6.5069999999999997},"last":1.1299999999999999},{"interface":"front","average":{"1min":1.1200000000000001,"5min":1.1659999999999999,"15min":1.1419999999999999},"min":{"1min":0.70299999999999996,"5min":0.62,"15min":0.62},"max":{"1min":1.458,"5min":3.3500000000000001,"15min":3.3500000000000001},"last":1.105}]}]},{"osd":6,"up_from":13,"seq":55834575092,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37296,"kb_used_data":1960,"kb_used_omap":0,"kb_used_meta":35328,"kb_avail":94334544,"statfs":{"total":96636764160,"available":96598573056,"internally_reserved":0,"allocated":2007040,"data_stored":1056682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":36175872},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 15:46:20 2023","interfaces":[{"interface":"back","average":{"1min":0.90000000000000002,"5min":0.86699999999999999,"15min":0.89900000000000002},"min":{"1min":0.499,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":1.4039999999999999,"5min":1.486,"15min":1.486},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.89200000000000002,"5min":0.94599999999999995,"15min":0.91100000000000003},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.36299999999999999},"max":{"1min":1.2729999999999999,"5min":1.6240000000000001,"15min":1.6240000000000001},"last":0.82799999999999996}]},{"osd":1,"last update":"Mon May 22 15:46:20 2023","interfaces":[{"interface":"back","average":{"1min":0.93200000000000005,"5min":0.89100000000000001,"15min":0.91200000000000003},"min":{"1min":0.59799999999999998,"5min":0.504,"15min":0.48299999999999998},"max":{"1min":1.8799999999999999,"5min":1.8799999999999999,"15min":1.8799999999999999},"last":0.72399999999999998},{"interface":"front","average":{"1min":0.95899999999999996,"5min":0.92400000000000004,"15min":0.93999999999999995},"min":{"1min":0.70299999999999996,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":1.1950000000000001,"5min":1.3320000000000001,"15min":1.554},"last":1.056}]},{"osd":2,"last update":"Mon May 22 15:46:20 2023","interfaces":[{"interface":"back","average":{"1min":1.05,"5min":0.97399999999999998,"15min":0.98999999999999999},"min":{"1min":0.55400000000000005,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":3.2429999999999999,"5min":3.2429999999999999,"15min":3.2429999999999999},"last":0.76400000000000001},{"interface":"front","average":{"1min":1.032,"5min":1.0269999999999999,"15min":1.0429999999999999},"min":{"1min":0.55300000000000005,"5min":0.5,"15min":0.442},"max":{"1min":3.3969999999999998,"5min":3.3969999999999998,"15min":3.3969999999999998},"last":0.64000000000000001}]},{"osd":3,"last update":"Mon May 22 15:46:20 2023","interfaces":[{"interface":"back","average":{"1min":0.88900000000000001,"5min":0.94599999999999995,"15min":0.97899999999999998},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":1.2470000000000001,"5min":1.5780000000000001,"15min":1.679},"last":1.1379999999999999},{"interface":"front","average":{"1min":0.93500000000000005,"5min":0.95099999999999996,"15min":0.96299999999999997},"min":{"1min":0.69099999999999995,"5min":0.53200000000000003,"15min":0.50800000000000001},"max":{"1min":1.2430000000000001,"5min":1.383,"15min":1.4570000000000001},"last":0.60699999999999998}]},{"osd":4,"last update":"Mon May 22 15:46:20 2023","interfaces":[{"interface":"back","average":{"1min":0.91300000000000003,"5min":0.93200000000000005,"15min":0.92500000000000004},"min":{"1min":0.61699999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":1.3169999999999999,"5min":1.581,"15min":1.6859999999999999},"last":0.874},{"interface":"front","average":{"1min":0.93200000000000005,"5min":1.0209999999999999,"15min":1.0169999999999999},"min":{"1min":0.65500000000000003,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":1.242,"5min":1.452,"15min":1.8300000000000001},"last":1.194}]},{"osd":5,"last update":"Mon May 22 15:46:20 2023","interfaces":[{"interface":"back","average":{"1min":0.98799999999999999,"5min":1.0169999999999999,"15min":0.98899999999999999},"min":{"1min":0.67600000000000005,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":1.3200000000000001,"5min":1.4530000000000001,"15min":1.6499999999999999},"last":0.92200000000000004},{"interface":"front","average":{"1min":1.1459999999999999,"5min":1.141,"15min":1.1120000000000001},"min":{"1min":0.81999999999999995,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.9990000000000001,"5min":1.9990000000000001,"15min":1.9990000000000001},"last":0.97399999999999998}]},{"osd":7,"last update":"Mon May 22 15:46:20 2023","interfaces":[{"interface":"back","average":{"1min":1.0860000000000001,"5min":1.131,"15min":1.125},"min":{"1min":0.85499999999999998,"5min":0.37,"15min":0.37},"max":{"1min":1.5529999999999999,"5min":1.853,"15min":1.9179999999999999},"last":1.2470000000000001},{"interface":"front","average":{"1min":1.131,"5min":1.133,"15min":1.135},"min":{"1min":0.80300000000000005,"5min":0.442,"15min":0.442},"max":{"1min":1.579,"5min":1.579,"15min":1.8520000000000001},"last":1.022}]}]},{"osd":5,"up_from":13,"seq":55834575092,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30696,"kb_used_data":1952,"kb_used_omap":0,"kb_used_meta":28736,"kb_avail":94341144,"statfs":{"total":96636764160,"available":96605331456,"internally_reserved":0,"allocated":1998848,"data_stored":1056558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":29425664},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 15:46:10 2023","interfaces":[{"interface":"back","average":{"1min":0.90900000000000003,"5min":0.85599999999999998,"15min":0.88300000000000001},"min":{"1min":0.61399999999999999,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.2889999999999999,"5min":1.3140000000000001,"15min":1.3440000000000001},"last":1.2789999999999999},{"interface":"front","average":{"1min":0.91800000000000004,"5min":0.88600000000000001,"15min":0.88500000000000001},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":1.2709999999999999,"5min":1.3899999999999999,"15min":2.1699999999999999},"last":0.66800000000000004}]},{"osd":1,"last update":"Mon May 22 15:46:10 2023","interfaces":[{"interface":"back","average":{"1min":0.94999999999999996,"5min":0.96099999999999997,"15min":0.995},"min":{"1min":0.64800000000000002,"5min":0.52400000000000002,"15min":0.51000000000000001},"max":{"1min":1.3109999999999999,"5min":1.337,"15min":1.9370000000000001},"last":1.0229999999999999},{"interface":"front","average":{"1min":1.073,"5min":1.008,"15min":1.026},"min":{"1min":0.501,"5min":0.501,"15min":0.501},"max":{"1min":1.3089999999999999,"5min":1.6970000000000001,"15min":1.6970000000000001},"last":1.23}]},{"osd":2,"last update":"Mon May 22 15:46:10 2023","interfaces":[{"interface":"back","average":{"1min":0.89100000000000001,"5min":0.91600000000000004,"15min":0.93400000000000005},"min":{"1min":0.505,"5min":0.505,"15min":0.504},"max":{"1min":1.343,"5min":1.5549999999999999,"15min":1.9830000000000001},"last":0.85599999999999998},{"interface":"front","average":{"1min":0.91800000000000004,"5min":0.94999999999999996,"15min":0.95199999999999996},"min":{"1min":0.56100000000000005,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.202,"5min":1.321,"15min":2.1120000000000001},"last":0.91400000000000003}]},{"osd":3,"last update":"Mon May 22 15:46:10 2023","interfaces":[{"interface":"back","average":{"1min":0.92900000000000005,"5min":0.92000000000000004,"15min":0.94899999999999995},"min":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.48199999999999998},"max":{"1min":1.206,"5min":1.2809999999999999,"15min":1.4419999999999999},"last":1.0589999999999999},{"interface":"front","average":{"1min":1.028,"5min":1.008,"15min":1.036},"min":{"1min":0.626,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":1.2490000000000001,"5min":1.371,"15min":2.52},"last":0.97799999999999998}]},{"osd":4,"last update":"Mon May 22 15:46:10 2023","interfaces":[{"interface":"back","average":{"1min":0.96999999999999997,"5min":0.95699999999999996,"15min":0.95199999999999996},"min":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":1.405,"5min":1.405,"15min":2.226},"last":0.79400000000000004},{"interface":"front","average":{"1min":1.0629999999999999,"5min":1.024,"15min":1.0309999999999999},"min":{"1min":0.76000000000000001,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.3740000000000001,"5min":1.863,"15min":1.863},"last":1.177}]},{"osd":6,"last update":"Mon May 22 15:46:10 2023","interfaces":[{"interface":"back","average":{"1min":1.008,"5min":1.024,"15min":1.0349999999999999},"min":{"1min":0.51900000000000002,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.397,"5min":1.4330000000000001,"15min":2.831},"last":1.135},{"interface":"front","average":{"1min":1.0369999999999999,"5min":1.0589999999999999,"15min":1.071},"min":{"1min":0.67000000000000004,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":1.4239999999999999,"5min":1.4239999999999999,"15min":2.3690000000000002},"last":1.103}]},{"osd":7,"last update":"Mon May 22 15:46:10 2023","interfaces":[{"interface":"back","average":{"1min":1.081,"5min":1.0900000000000001,"15min":1.1120000000000001},"min":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.42499999999999999},"max":{"1min":1.4359999999999999,"5min":1.47,"15min":1.849},"last":1.3220000000000001},{"interface":"front","average":{"1min":1.1599999999999999,"5min":1.149,"15min":1.1659999999999999},"min":{"1min":0.86899999999999999,"5min":0.64000000000000001,"15min":0.64000000000000001},"max":{"1min":1.4339999999999999,"5min":1.575,"15min":2.7429999999999999},"last":1.2170000000000001}]}]},{"osd":4,"up_from":13,"seq":55834575090,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32548,"kb_used_data":1948,"kb_used_omap":0,"kb_used_meta":30592,"kb_avail":94339292,"statfs":{"total":96636764160,"available":96603435008,"internally_reserved":0,"allocated":1994752,"data_stored":1056536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31326208},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 15:46:19 2023","interfaces":[{"interface":"back","average":{"1min":0.93200000000000005,"5min":0.86799999999999999,"15min":0.90500000000000003},"min":{"1min":0.63400000000000001,"5min":0.48999999999999999,"15min":0.34100000000000003},"max":{"1min":1.2629999999999999,"5min":1.4119999999999999,"15min":1.9910000000000001},"last":1.056},{"interface":"front","average":{"1min":0.91300000000000003,"5min":0.90000000000000002,"15min":0.92200000000000004},"min":{"1min":0.51100000000000001,"5min":0.499,"15min":0.373},"max":{"1min":1.284,"5min":1.369,"15min":1.3779999999999999},"last":0.67800000000000005}]},{"osd":1,"last update":"Mon May 22 15:46:19 2023","interfaces":[{"interface":"back","average":{"1min":0.94799999999999995,"5min":0.91800000000000004,"15min":0.96399999999999997},"min":{"1min":0.60499999999999998,"5min":0.436,"15min":0.436},"max":{"1min":1.1970000000000001,"5min":1.3899999999999999,"15min":1.3899999999999999},"last":0.83599999999999997},{"interface":"front","average":{"1min":0.91700000000000004,"5min":0.84499999999999997,"15min":0.90100000000000002},"min":{"1min":0.56699999999999995,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.157,"5min":1.2889999999999999,"15min":1.3640000000000001},"last":1.1890000000000001}]},{"osd":2,"last update":"Mon May 22 15:46:19 2023","interfaces":[{"interface":"back","average":{"1min":0.90100000000000002,"5min":0.91600000000000004,"15min":0.94199999999999995},"min":{"1min":0.56299999999999994,"5min":0.50800000000000001,"15min":0.495},"max":{"1min":1.244,"5min":1.96,"15min":2.1019999999999999},"last":0.78600000000000003},{"interface":"front","average":{"1min":0.97799999999999998,"5min":0.98399999999999999,"15min":0.99199999999999999},"min":{"1min":0.58099999999999996,"5min":0.48799999999999999,"15min":0.48399999999999999},"max":{"1min":1.329,"5min":1.4399999999999999,"15min":1.4399999999999999},"last":0.99399999999999999}]},{"osd":3,"last update":"Mon May 22 15:46:19 2023","interfaces":[{"interface":"back","average":{"1min":0.91100000000000003,"5min":0.90200000000000002,"15min":0.93200000000000005},"min":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":1.224,"5min":1.28,"15min":1.3169999999999999},"last":0.74299999999999999},{"interface":"front","average":{"1min":1.0469999999999999,"5min":1.014,"15min":1.0209999999999999},"min":{"1min":0.72199999999999998,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":1.3420000000000001,"5min":1.4179999999999999,"15min":2.1720000000000002},"last":1.159}]},{"osd":5,"last update":"Mon May 22 15:46:19 2023","interfaces":[{"interface":"back","average":{"1min":1.0209999999999999,"5min":0.97199999999999998,"15min":0.97699999999999998},"min":{"1min":0.55000000000000004,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.357,"5min":1.373,"15min":1.472},"last":1.1200000000000001},{"interface":"front","average":{"1min":1.0389999999999999,"5min":0.98999999999999999,"15min":0.97499999999999998},"min":{"1min":0.59999999999999998,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.262,"5min":1.365,"15min":1.4139999999999999},"last":0.94699999999999995}]},{"osd":6,"last update":"Mon May 22 15:46:19 2023","interfaces":[{"interface":"back","average":{"1min":1.0649999999999999,"5min":0.99299999999999999,"15min":0.98499999999999999},"min":{"1min":0.72099999999999997,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":1.456,"5min":1.456,"15min":2.2349999999999999},"last":1.274},{"interface":"front","average":{"1min":1.131,"5min":1.103,"15min":1.1100000000000001},"min":{"1min":0.66500000000000004,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":1.425,"5min":1.486,"15min":1.8220000000000001},"last":1.2490000000000001}]},{"osd":7,"last update":"Mon May 22 15:46:19 2023","interfaces":[{"interface":"back","average":{"1min":1.137,"5min":1.0629999999999999,"15min":1.052},"min":{"1min":0.85699999999999998,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":1.389,"5min":1.46,"15min":1.5800000000000001},"last":1.3300000000000001},{"interface":"front","average":{"1min":1.1930000000000001,"5min":1.167,"15min":1.143},"min":{"1min":0.78900000000000003,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":1.5620000000000001,"5min":1.6559999999999999,"15min":2.306},"last":1.361}]}]},{"osd":3,"up_from":13,"seq":55834575090,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37720,"kb_used_data":1936,"kb_used_omap":0,"kb_used_meta":35776,"kb_avail":94334120,"statfs":{"total":96636764160,"available":96598138880,"internally_reserved":0,"allocated":1982464,"data_stored":1049060,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":36634624},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 15:46:03 2023","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.61699999999999999,"15min":0.63600000000000001},"min":{"1min":0.50700000000000001,"5min":0.252,"15min":0.19},"max":{"1min":1.169,"5min":1.169,"15min":1.194},"last":0.73499999999999999},{"interface":"front","average":{"1min":0.746,"5min":0.68500000000000005,"15min":0.70899999999999996},"min":{"1min":0.48899999999999999,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":1.085,"5min":1.296,"15min":1.296},"last":0.61699999999999999}]},{"osd":1,"last update":"Mon May 22 15:46:03 2023","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.64200000000000002,"15min":0.66200000000000003},"min":{"1min":0.52100000000000002,"5min":0.314,"15min":0.29499999999999998},"max":{"1min":0.90600000000000003,"5min":1.2609999999999999,"15min":1.2609999999999999},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.72899999999999998,"15min":0.76800000000000002},"min":{"1min":0.47499999999999998,"5min":0.34399999999999997,"15min":0.313},"max":{"1min":1.081,"5min":1.1359999999999999,"15min":1.222},"last":0.82499999999999996}]},{"osd":2,"last update":"Mon May 22 15:46:03 2023","interfaces":[{"interface":"back","average":{"1min":0.84199999999999997,"5min":0.78700000000000003,"15min":0.81699999999999995},"min":{"1min":0.45600000000000002,"5min":0.40699999999999997,"15min":0.36599999999999999},"max":{"1min":1.0529999999999999,"5min":1.506,"15min":1.506},"last":0.89600000000000002},{"interface":"front","average":{"1min":0.81899999999999995,"5min":0.75600000000000001,"15min":0.80700000000000005},"min":{"1min":0.65300000000000002,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":1.032,"5min":1.147,"15min":1.359},"last":0.86899999999999999}]},{"osd":4,"last update":"Mon May 22 15:46:03 2023","interfaces":[{"interface":"back","average":{"1min":0.97499999999999998,"5min":0.92500000000000004,"15min":0.95599999999999996},"min":{"1min":0.71999999999999997,"5min":0.54500000000000004,"15min":0.48999999999999999},"max":{"1min":1.238,"5min":1.413,"15min":1.4470000000000001},"last":0.98199999999999998},{"interface":"front","average":{"1min":1.0529999999999999,"5min":0.97499999999999998,"15min":1.0029999999999999},"min":{"1min":0.83299999999999996,"5min":0.54400000000000004,"15min":0.49199999999999999},"max":{"1min":1.407,"5min":1.407,"15min":1.4470000000000001},"last":0.94199999999999995}]},{"osd":5,"last update":"Mon May 22 15:46:03 2023","interfaces":[{"interface":"back","average":{"1min":1.0509999999999999,"5min":0.98399999999999999,"15min":1.016},"min":{"1min":0.80900000000000005,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.3149999999999999,"5min":1.5089999999999999,"15min":1.5089999999999999},"last":1.0029999999999999},{"interface":"front","average":{"1min":1.0349999999999999,"5min":1.006,"15min":1.038},"min":{"1min":0.90200000000000002,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.2130000000000001,"5min":1.4299999999999999,"15min":1.4359999999999999},"last":1.4039999999999999}]},{"osd":6,"last update":"Mon May 22 15:46:03 2023","interfaces":[{"interface":"back","average":{"1min":1.0580000000000001,"5min":1.0129999999999999,"15min":1.042},"min":{"1min":0.873,"5min":0.47699999999999998,"15min":0.46999999999999997},"max":{"1min":1.2989999999999999,"5min":1.4570000000000001,"15min":1.49},"last":1.0640000000000001},{"interface":"front","average":{"1min":1.083,"5min":1.0269999999999999,"15min":1.0700000000000001},"min":{"1min":0.79800000000000004,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":1.2949999999999999,"5min":1.4330000000000001,"15min":1.5009999999999999},"last":1.3600000000000001}]},{"osd":7,"last update":"Mon May 22 15:46:03 2023","interfaces":[{"interface":"back","average":{"1min":1.113,"5min":1.0780000000000001,"15min":1.1040000000000001},"min":{"1min":0.83599999999999997,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.3779999999999999,"5min":1.6719999999999999,"15min":1.6719999999999999},"last":1.26},{"interface":"front","average":{"1min":1.117,"5min":1.071,"15min":1.085},"min":{"1min":0.90200000000000002,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":1.361,"5min":1.8700000000000001,"15min":1.8700000000000001},"last":1.1160000000000001}]}]},{"osd":2,"up_from":13,"seq":55834575091,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":23540,"kb_used_data":1964,"kb_used_omap":0,"kb_used_meta":21568,"kb_avail":94348300,"statfs":{"total":96636764160,"available":96612659200,"internally_reserved":0,"allocated":2011136,"data_stored":1058154,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":22085632},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 15:46:03 2023","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.64400000000000002,"15min":0.65600000000000003},"min":{"1min":0.28699999999999998,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":1.097,"5min":1.1539999999999999,"15min":1.8240000000000001},"last":0.495},{"interface":"front","average":{"1min":0.69899999999999995,"5min":0.70499999999999996,"15min":0.746},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":1.079,"5min":1.079,"15min":1.8160000000000001},"last":0.68300000000000005}]},{"osd":1,"last update":"Mon May 22 15:46:03 2023","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.72999999999999998,"15min":0.74399999999999999},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.27000000000000002},"max":{"1min":1.196,"5min":1.196,"15min":1.2},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.754,"5min":0.75900000000000001,"15min":0.80800000000000005},"min":{"1min":0.32400000000000001,"5min":0.317,"15min":0.317},"max":{"1min":1.0309999999999999,"5min":1.246,"15min":2.5089999999999999},"last":0.752}]},{"osd":3,"last update":"Mon May 22 15:46:03 2023","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":0.69399999999999995,"15min":0.72599999999999998},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.96199999999999997,"5min":1.077,"15min":2.7629999999999999},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.82699999999999996,"5min":0.84599999999999997,"15min":0.88300000000000001},"min":{"1min":0.39600000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":1.1699999999999999,"5min":1.256,"15min":2.6840000000000002},"last":0.437}]},{"osd":4,"last update":"Mon May 22 15:46:03 2023","interfaces":[{"interface":"back","average":{"1min":0.97499999999999998,"5min":0.98499999999999999,"15min":1.018},"min":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":1.274,"5min":1.536,"15min":2.8039999999999998},"last":0.89600000000000002},{"interface":"front","average":{"1min":1.0069999999999999,"5min":1.0009999999999999,"15min":1.028},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.2509999999999999,"5min":1.4299999999999999,"15min":2.0819999999999999},"last":0.80800000000000005}]},{"osd":5,"last update":"Mon May 22 15:46:03 2023","interfaces":[{"interface":"back","average":{"1min":1.0569999999999999,"5min":1.056,"15min":1.052},"min":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.3959999999999999,"5min":1.5049999999999999,"15min":2.3860000000000001},"last":0.91000000000000003},{"interface":"front","average":{"1min":0.99099999999999999,"5min":0.97399999999999998,"15min":0.998},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.331,"5min":1.331,"15min":3.0299999999999998},"last":0.86799999999999999}]},{"osd":6,"last update":"Mon May 22 15:46:03 2023","interfaces":[{"interface":"back","average":{"1min":1.038,"5min":1.012,"15min":1.036},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":1.3320000000000001,"5min":1.49,"15min":2.883},"last":0.96299999999999997},{"interface":"front","average":{"1min":1.129,"5min":1.081,"15min":1.1040000000000001},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":1.4450000000000001,"5min":1.5660000000000001,"15min":3.1459999999999999},"last":0.84199999999999997}]},{"osd":7,"last update":"Mon May 22 15:46:03 2023","interfaces":[{"interface":"back","average":{"1min":1.1639999999999999,"5min":1.097,"15min":1.1279999999999999},"min":{"1min":0.55000000000000004,"5min":0.499,"15min":0.499},"max":{"1min":1.429,"5min":1.55,"15min":2.5990000000000002},"last":0.99299999999999999},{"interface":"front","average":{"1min":1.1240000000000001,"5min":1.0640000000000001,"15min":1.1060000000000001},"min":{"1min":0.5,"5min":0.5,"15min":0.5},"max":{"1min":1.4650000000000001,"5min":1.4650000000000001,"15min":3.8799999999999999},"last":1.0449999999999999}]}]},{"osd":1,"up_from":13,"seq":55834575091,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30376,"kb_used_data":1952,"kb_used_omap":0,"kb_used_meta":28416,"kb_avail":94341464,"statfs":{"total":96636764160,"available":96605659136,"internally_reserved":0,"allocated":1998848,"data_stored":1057102,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":29097984},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 15:46:17 2023","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.63200000000000001,"15min":0.63400000000000001},"min":{"1min":0.40600000000000003,"5min":0.26300000000000001,"15min":0.22700000000000001},"max":{"1min":1.1200000000000001,"5min":1.2270000000000001,"15min":1.2270000000000001},"last":0.74199999999999999},{"interface":"front","average":{"1min":0.77100000000000002,"5min":0.70699999999999996,"15min":0.76400000000000001},"min":{"1min":0.35999999999999999,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":1.1839999999999999,"5min":1.1839999999999999,"15min":1.2529999999999999},"last":1.218}]},{"osd":2,"last update":"Mon May 22 15:46:17 2023","interfaces":[{"interface":"back","average":{"1min":0.76400000000000001,"5min":0.71399999999999997,"15min":0.73799999999999999},"min":{"1min":0.44800000000000001,"5min":0.317,"15min":0.28999999999999998},"max":{"1min":1.0660000000000001,"5min":1.391,"15min":1.391},"last":1.012},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.72999999999999998,"15min":0.76300000000000001},"min":{"1min":0.49199999999999999,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.98699999999999999,"5min":1.2270000000000001,"15min":1.276},"last":0.58199999999999996}]},{"osd":3,"last update":"Mon May 22 15:46:17 2023","interfaces":[{"interface":"back","average":{"1min":0.80100000000000005,"5min":0.79900000000000004,"15min":0.82799999999999996},"min":{"1min":0.51200000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.153,"5min":1.657,"15min":1.657},"last":1.056},{"interface":"front","average":{"1min":0.81100000000000005,"5min":0.80800000000000005,"15min":0.81499999999999995},"min":{"1min":0.33700000000000002,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":1.095,"5min":2.0569999999999999,"15min":2.0569999999999999},"last":0.90800000000000003}]},{"osd":4,"last update":"Mon May 22 15:46:17 2023","interfaces":[{"interface":"back","average":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.96499999999999997},"min":{"1min":0.64500000000000002,"5min":0.55200000000000005,"15min":0.34000000000000002},"max":{"1min":1.502,"5min":2.9529999999999998,"15min":2.9529999999999998},"last":1.173},{"interface":"front","average":{"1min":0.97499999999999998,"5min":0.999,"15min":1.002},"min":{"1min":0.61299999999999999,"5min":0.504,"15min":0.49299999999999999},"max":{"1min":1.292,"5min":2.7759999999999998,"15min":2.7759999999999998},"last":0.96299999999999997}]},{"osd":5,"last update":"Mon May 22 15:46:17 2023","interfaces":[{"interface":"back","average":{"1min":1.0349999999999999,"5min":1.006,"15min":1.034},"min":{"1min":0.627,"5min":0.58399999999999996,"15min":0.443},"max":{"1min":1.556,"5min":1.9570000000000001,"15min":1.9570000000000001},"last":1.089},{"interface":"front","average":{"1min":1.1060000000000001,"5min":1.034,"15min":1.0449999999999999},"min":{"1min":0.59399999999999997,"5min":0.53400000000000003,"15min":0.47299999999999998},"max":{"1min":1.577,"5min":2.1200000000000001,"15min":2.1200000000000001},"last":1.419}]},{"osd":6,"last update":"Mon May 22 15:46:17 2023","interfaces":[{"interface":"back","average":{"1min":1.02,"5min":1.0369999999999999,"15min":1.036},"min":{"1min":0.65900000000000003,"5min":0.54000000000000004,"15min":0.45300000000000001},"max":{"1min":1.3180000000000001,"5min":2.831,"15min":2.831},"last":1.1359999999999999},{"interface":"front","average":{"1min":1.1499999999999999,"5min":1.085,"15min":1.0940000000000001},"min":{"1min":0.55800000000000005,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":1.6299999999999999,"5min":1.7629999999999999,"15min":1.7629999999999999},"last":1.335}]},{"osd":7,"last update":"Mon May 22 15:46:17 2023","interfaces":[{"interface":"back","average":{"1min":1.1759999999999999,"5min":1.135,"15min":1.1180000000000001},"min":{"1min":0.61299999999999999,"5min":0.60299999999999998,"15min":0.56799999999999995},"max":{"1min":1.679,"5min":2.4199999999999999,"15min":2.4199999999999999},"last":1.472},{"interface":"front","average":{"1min":1.1819999999999999,"5min":1.1259999999999999,"15min":1.1299999999999999},"min":{"1min":0.76300000000000001,"5min":0.47299999999999998,"15min":0.41599999999999998},"max":{"1min":1.619,"5min":3.8799999999999999,"15min":3.8799999999999999},"last":1.5069999999999999}]}]},{"osd":0,"up_from":13,"seq":55834575092,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30136,"kb_used_data":2544,"kb_used_omap":0,"kb_used_meta":27584,"kb_avail":94341704,"statfs":{"total":96636764160,"available":96605904896,"internally_reserved":0,"allocated":2605056,"data_stored":1648486,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":28246016},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon May 22 15:46:21 2023","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.65900000000000003,"15min":0.68100000000000005},"min":{"1min":0.307,"5min":0.248,"15min":0.223},"max":{"1min":1.2190000000000001,"5min":1.466,"15min":1.466},"last":0.27000000000000002},{"interface":"front","average":{"1min":0.69899999999999995,"5min":0.70599999999999996,"15min":0.72899999999999998},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.24099999999999999},"max":{"1min":1.2230000000000001,"5min":1.71,"15min":1.71},"last":0.36899999999999999}]},{"osd":2,"last update":"Mon May 22 15:46:21 2023","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":0.70699999999999996,"15min":0.74099999999999999},"min":{"1min":0.39700000000000002,"5min":0.318,"15min":0.26500000000000001},"max":{"1min":0.96799999999999997,"5min":1.165,"15min":1.3220000000000001},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.82799999999999996,"5min":0.75900000000000001,"15min":0.78800000000000003},"min":{"1min":0.375,"5min":0.34999999999999998,"15min":0.32900000000000001},"max":{"1min":1.1699999999999999,"5min":1.3129999999999999,"15min":1.3129999999999999},"last":0.42499999999999999}]},{"osd":3,"last update":"Mon May 22 15:46:21 2023","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":0.74099999999999999,"15min":0.76000000000000001},"min":{"1min":0.35499999999999998,"5min":0.31,"15min":0.31},"max":{"1min":0.96199999999999997,"5min":1.9299999999999999,"15min":1.9299999999999999},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.83499999999999996,"5min":0.83299999999999996,"15min":0.871},"min":{"1min":0.35599999999999998,"5min":0.35199999999999998,"15min":0.34399999999999997},"max":{"1min":1.1299999999999999,"5min":1.758,"15min":1.758},"last":0.45800000000000002}]},{"osd":4,"last update":"Mon May 22 15:46:21 2023","interfaces":[{"interface":"back","average":{"1min":0.92800000000000005,"5min":0.95099999999999996,"15min":0.95099999999999996},"min":{"1min":0.57599999999999996,"5min":0.57399999999999995,"15min":0.48099999999999998},"max":{"1min":1.149,"5min":2.923,"15min":2.923},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.98999999999999999,"5min":1.006,"15min":1.026},"min":{"1min":0.624,"5min":0.624,"15min":0.624},"max":{"1min":1.3839999999999999,"5min":2.0129999999999999,"15min":2.0129999999999999},"last":0.84099999999999997}]},{"osd":5,"last update":"Mon May 22 15:46:21 2023","interfaces":[{"interface":"back","average":{"1min":1,"5min":1.004,"15min":1.0409999999999999},"min":{"1min":0.59099999999999997,"5min":0.56000000000000005,"15min":0.46000000000000002},"max":{"1min":1.238,"5min":1.97,"15min":1.97},"last":0.77300000000000002},{"interface":"front","average":{"1min":1.02,"5min":1.016,"15min":1.008},"min":{"1min":0.67600000000000005,"5min":0.56599999999999995,"15min":0.55300000000000005},"max":{"1min":1.5349999999999999,"5min":5.0519999999999996,"15min":5.0519999999999996},"last":0.58399999999999996}]},{"osd":6,"last update":"Mon May 22 15:46:21 2023","interfaces":[{"interface":"back","average":{"1min":1.044,"5min":1.0589999999999999,"15min":1.089},"min":{"1min":0.58899999999999997,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":1.306,"5min":2.117,"15min":2.117},"last":0.74199999999999999},{"interface":"front","average":{"1min":1.085,"5min":1.1299999999999999,"15min":1.119},"min":{"1min":0.69199999999999995,"5min":0.65500000000000003,"15min":0.499},"max":{"1min":1.3100000000000001,"5min":5.2640000000000002,"15min":5.2640000000000002},"last":0.66500000000000004}]},{"osd":7,"last update":"Mon May 22 15:46:21 2023","interfaces":[{"interface":"back","average":{"1min":1.075,"5min":1.1040000000000001,"15min":1.129},"min":{"1min":0.63800000000000001,"5min":0.61299999999999999,"15min":0.61299999999999999},"max":{"1min":1.3640000000000001,"5min":2.0550000000000002,"15min":2.0550000000000002},"last":0.60499999999999998},{"interface":"front","average":{"1min":1.089,"5min":1.0900000000000001,"15min":1.0960000000000001},"min":{"1min":0.73099999999999998,"5min":0.63300000000000001,"15min":0.60299999999999998},"max":{"1min":1.6240000000000001,"5min":3.9060000000000001,"15min":3.9060000000000001},"last":0.81899999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-05-22T15:47:07.060 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-05-22T15:47:07.213 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.207+0000 7fa681184640 1 Processor -- start 2023-05-22T15:47:07.214 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.207+0000 7fa681184640 1 -- start start 2023-05-22T15:47:07.215 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.207+0000 7fa681184640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fa67c0c9b40 0x7fa67c15e460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:47:07.215 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.207+0000 7fa681184640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fa67c070df0 0x7fa67c071210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:47:07.215 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.207+0000 7fa681184640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa67c0b0650 con 0x7fa67c0ca4e0 2023-05-22T15:47:07.216 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.207+0000 7fa681184640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa67c0b0370 con 0x7fa67c0c9b40 2023-05-22T15:47:07.216 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.207+0000 7fa681184640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa67c15d980 con 0x7fa67c070df0 2023-05-22T15:47:07.216 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.207+0000 7fa67a59c640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fa67c0c9b40 0x7fa67c15e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:47:07.216 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.207+0000 7fa67a59c640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fa67c0c9b40 0x7fa67c15e460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46852/0 (socket says 172.21.15.16:46852) 2023-05-22T15:47:07.216 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.207+0000 7fa67a59c640 1 -- 172.21.15.16:0/408061628 learned_addr learned my addr 172.21.15.16:0/408061628 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:47:07.216 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.211+0000 7fa67b59e640 1 --2- 172.21.15.16:0/408061628 >> v2:172.21.15.62:3300/0 conn(0x7fa67c070df0 0x7fa67c071210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:47:07.216 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.211+0000 7fa67a59c640 1 -- 172.21.15.16:0/408061628 >> v2:172.21.15.62:3300/0 conn(0x7fa67c070df0 msgr2=0x7fa67c071210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:47:07.217 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.211+0000 7fa67a59c640 1 --2- 172.21.15.16:0/408061628 >> v2:172.21.15.62:3300/0 conn(0x7fa67c070df0 0x7fa67c071210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:07.217 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.211+0000 7fa67a59c640 1 -- 172.21.15.16:0/408061628 >> v1:172.21.15.62:6789/0 conn(0x7fa67c0ca4e0 legacy=0x7fa67c0afb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:47:07.217 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.211+0000 7fa67a59c640 1 -- 172.21.15.16:0/408061628 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa67c15dc60 con 0x7fa67c0c9b40 2023-05-22T15:47:07.217 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.211+0000 7fa67a59c640 1 --2- 172.21.15.16:0/408061628 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa67c0c9b40 0x7fa67c15e460 secure :-1 s=READY pgs=2044 cs=0 l=1 rev1=1 crypto rx=0x7fa67c0cad10 tx=0x7fa67001c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=440d82ac95c757c4 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:47:07.218 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.211+0000 7fa679d9b640 1 -- 172.21.15.16:0/408061628 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6700a7070 con 0x7fa67c0c9b40 2023-05-22T15:47:07.218 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.211+0000 7fa679d9b640 1 -- 172.21.15.16:0/408061628 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6700ab430 con 0x7fa67c0c9b40 2023-05-22T15:47:07.218 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.211+0000 7fa679d9b640 1 -- 172.21.15.16:0/408061628 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6700ab7c0 con 0x7fa67c0c9b40 2023-05-22T15:47:07.219 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.211+0000 7fa681184640 1 -- 172.21.15.16:0/408061628 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa67c0c9b40 msgr2=0x7fa67c15e460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:47:07.219 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.211+0000 7fa681184640 1 --2- 172.21.15.16:0/408061628 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa67c0c9b40 0x7fa67c15e460 secure :-1 s=READY pgs=2044 cs=0 l=1 rev1=1 crypto rx=0x7fa67c0cad10 tx=0x7fa67001c4d0 comp rx=0 tx=0).stop 2023-05-22T15:47:07.219 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.211+0000 7fa681184640 1 -- 172.21.15.16:0/408061628 shutdown_connections 2023-05-22T15:47:07.219 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.211+0000 7fa681184640 1 --2- 172.21.15.16:0/408061628 >> v2:172.21.15.62:3300/0 conn(0x7fa67c070df0 0x7fa67c071210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:07.219 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.211+0000 7fa681184640 1 --2- 172.21.15.16:0/408061628 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa67c0c9b40 0x7fa67c15e460 unknown :-1 s=CLOSED pgs=2044 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:07.219 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.211+0000 7fa681184640 1 -- 172.21.15.16:0/408061628 >> 172.21.15.16:0/408061628 conn(0x7fa67c069720 msgr2=0x7fa67c069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:47:07.219 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.211+0000 7fa681184640 1 -- 172.21.15.16:0/408061628 shutdown_connections 2023-05-22T15:47:07.220 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.211+0000 7fa681184640 1 -- 172.21.15.16:0/408061628 wait complete. 2023-05-22T15:47:07.222 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.215+0000 7fa681184640 1 Processor -- start 2023-05-22T15:47:07.222 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.215+0000 7fa681184640 1 -- start start 2023-05-22T15:47:07.222 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.215+0000 7fa681184640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa67c070df0 0x7fa67c1594d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:47:07.222 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.215+0000 7fa681184640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa67c0c9b40 0x7fa67c14b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:47:07.222 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.215+0000 7fa681184640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa67c0ca4e0 0x7fa67c15a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:47:07.222 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.215+0000 7fa681184640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa67c158d70 con 0x7fa67c070df0 2023-05-22T15:47:07.222 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.215+0000 7fa681184640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa67c159ab0 con 0x7fa67c0ca4e0 2023-05-22T15:47:07.223 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.215+0000 7fa681184640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa67c157fb0 con 0x7fa67c0c9b40 2023-05-22T15:47:07.223 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.215+0000 7fa67ad9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa67c070df0 0x7fa67c1594d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:47:07.223 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.215+0000 7fa67ad9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa67c070df0 0x7fa67c1594d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46868/0 (socket says 172.21.15.16:46868) 2023-05-22T15:47:07.223 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.215+0000 7fa67ad9d640 1 -- 172.21.15.16:0/2838748502 learned_addr learned my addr 172.21.15.16:0/2838748502 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:47:07.223 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.215+0000 7fa67b59e640 1 --2- 172.21.15.16:0/2838748502 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa67c0ca4e0 0x7fa67c15a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:47:07.223 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.215+0000 7fa67a59c640 1 --2- 172.21.15.16:0/2838748502 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa67c0c9b40 0x7fa67c14b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:47:07.223 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.215+0000 7fa67ad9d640 1 -- 172.21.15.16:0/2838748502 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa67c0c9b40 msgr2=0x7fa67c14b7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:47:07.224 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.215+0000 7fa67ad9d640 1 --2- 172.21.15.16:0/2838748502 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa67c0c9b40 0x7fa67c14b7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:07.224 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.215+0000 7fa67ad9d640 1 -- 172.21.15.16:0/2838748502 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa67c0ca4e0 msgr2=0x7fa67c15a2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:47:07.224 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.215+0000 7fa67ad9d640 1 --2- 172.21.15.16:0/2838748502 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa67c0ca4e0 0x7fa67c15a2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:07.224 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.215+0000 7fa67ad9d640 1 -- 172.21.15.16:0/2838748502 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6700094d0 con 0x7fa67c070df0 2023-05-22T15:47:07.224 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.215+0000 7fa67b59e640 1 --2- 172.21.15.16:0/2838748502 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa67c0ca4e0 0x7fa67c15a2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:47:07.225 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.215+0000 7fa67ad9d640 1 --2- 172.21.15.16:0/2838748502 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa67c070df0 0x7fa67c1594d0 secure :-1 s=READY pgs=2045 cs=0 l=1 rev1=1 crypto rx=0x7fa66c00c3c0 tx=0x7fa66c00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:47:07.225 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.215+0000 7fa6637fe640 1 -- 172.21.15.16:0/2838748502 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa66c0052d0 con 0x7fa67c070df0 2023-05-22T15:47:07.225 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.215+0000 7fa6637fe640 1 -- 172.21.15.16:0/2838748502 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa66c005430 con 0x7fa67c070df0 2023-05-22T15:47:07.225 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.215+0000 7fa681184640 1 -- 172.21.15.16:0/2838748502 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa67c0c8e80 con 0x7fa67c070df0 2023-05-22T15:47:07.225 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.215+0000 7fa6637fe640 1 -- 172.21.15.16:0/2838748502 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa66c003840 con 0x7fa67c070df0 2023-05-22T15:47:07.226 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.215+0000 7fa681184640 1 -- 172.21.15.16:0/2838748502 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa67c0ca8b0 con 0x7fa67c070df0 2023-05-22T15:47:07.226 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.215+0000 7fa681184640 1 -- 172.21.15.16:0/2838748502 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa67c0c9010 con 0x7fa67c070df0 2023-05-22T15:47:07.226 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.215+0000 7fa6637fe640 1 -- 172.21.15.16:0/2838748502 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 67) v1 ==== 84332+0+0 (secure 0 0 0) 0x7fa66c0039a0 con 0x7fa67c070df0 2023-05-22T15:47:07.227 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.215+0000 7fa6637fe640 1 --2- 172.21.15.16:0/2838748502 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fa6680654a0 0x7fa668067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:47:07.227 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.215+0000 7fa6637fe640 1 -- 172.21.15.16:0/2838748502 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(161..161 src has 1..161) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fa66c082070 con 0x7fa67c070df0 2023-05-22T15:47:07.227 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.219+0000 7fa67a59c640 1 --2- 172.21.15.16:0/2838748502 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fa6680654a0 0x7fa668067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:47:07.227 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.219+0000 7fa67a59c640 1 --2- 172.21.15.16:0/2838748502 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fa6680654a0 0x7fa668067950 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fa6700af000 tx=0x7fa670008040 comp rx=0 tx=0).ready entity=mgr.9396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:47:07.227 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.223+0000 7fa6637fe640 1 -- 172.21.15.16:0/2838748502 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fa66c04fa20 con 0x7fa67c070df0 2023-05-22T15:47:07.370 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.363+0000 7fa681184640 1 -- 172.21.15.16:0/2838748502 --> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fa67c04dc50 con 0x7fa6680654a0 2023-05-22T15:47:07.379 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.371+0000 7fa6637fe640 1 -- 172.21.15.16:0/2838748502 <== mgr.9396 v2:172.21.15.16:6832/36921 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+309955 (secure 0 0 0) 0x7fa67c04dc50 con 0x7fa6680654a0 2023-05-22T15:47:07.381 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:47:07.383 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2023-05-22T15:47:07.384 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.379+0000 7fa681184640 1 -- 172.21.15.16:0/2838748502 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fa6680654a0 msgr2=0x7fa668067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:47:07.384 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.379+0000 7fa681184640 1 --2- 172.21.15.16:0/2838748502 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fa6680654a0 0x7fa668067950 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fa6700af000 tx=0x7fa670008040 comp rx=0 tx=0).stop 2023-05-22T15:47:07.384 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.379+0000 7fa681184640 1 -- 172.21.15.16:0/2838748502 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa67c070df0 msgr2=0x7fa67c1594d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:47:07.384 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.379+0000 7fa681184640 1 --2- 172.21.15.16:0/2838748502 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa67c070df0 0x7fa67c1594d0 secure :-1 s=READY pgs=2045 cs=0 l=1 rev1=1 crypto rx=0x7fa66c00c3c0 tx=0x7fa66c00c880 comp rx=0 tx=0).stop 2023-05-22T15:47:07.384 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.379+0000 7fa681184640 1 -- 172.21.15.16:0/2838748502 shutdown_connections 2023-05-22T15:47:07.385 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.379+0000 7fa681184640 1 --2- 172.21.15.16:0/2838748502 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa67c0ca4e0 0x7fa67c15a2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:07.385 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.379+0000 7fa681184640 1 --2- 172.21.15.16:0/2838748502 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fa6680654a0 0x7fa668067950 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:07.385 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.379+0000 7fa681184640 1 --2- 172.21.15.16:0/2838748502 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa67c0c9b40 0x7fa67c14b7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:07.385 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.379+0000 7fa681184640 1 --2- 172.21.15.16:0/2838748502 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa67c070df0 0x7fa67c1594d0 unknown :-1 s=CLOSED pgs=2045 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:07.385 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.379+0000 7fa681184640 1 -- 172.21.15.16:0/2838748502 >> 172.21.15.16:0/2838748502 conn(0x7fa67c069720 msgr2=0x7fa67c06b730 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:47:07.385 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.379+0000 7fa681184640 1 -- 172.21.15.16:0/2838748502 shutdown_connections 2023-05-22T15:47:07.385 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.379+0000 7fa681184640 1 -- 172.21.15.16:0/2838748502 wait complete. 2023-05-22T15:47:07.400 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2023-05-22T15:47:05.713182+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592698,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1460,"num_read_kb":2984,"num_write":483,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":309,"ondisk_log_size":309,"up":210,"acting":210,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":59,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":242840,"kb_used_data":16792,"kb_used_omap":0,"kb_used_meta":225984,"kb_avail":754731880,"statfs":{"total":773094113280,"available":772845445120,"internally_reserved":0,"allocated":17195008,"data_stored":9630118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":231407616},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":52,"apply_latency_ms":52,"commit_latency_ns":52000000,"apply_latency_ns":52000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2330,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.105499"},"pg_stats":[{"pgid":"21.3f","version":"0'0","reported_seq":9,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-05-22T15:47:03.075301+0000","last_change":"2023-05-22T15:47:03.074499+0000","last_active":"2023-05-22T15:47:03.075301+0000","last_peered":"2023-05-22T15:47:03.075301+0000","last_clean":"2023-05-22T15:47:03.075301+0000","last_became_active":"2023-05-22T15:47:03.074171+0000","last_became_peered":"2023-05-22T15:47:03.074171+0000","last_unstale":"2023-05-22T15:47:03.075301+0000","last_undegraded":"2023-05-22T15:47:03.075301+0000","last_fullsized":"2023-05-22T15:47:03.075301+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_clean_scrub_stamp":"2023-05-22T15:47:02.606868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:07:06.587412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.3e","version":"0'0","reported_seq":9,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-05-22T15:47:03.044416+0000","last_change":"2023-05-22T15:47:03.043270+0000","last_active":"2023-05-22T15:47:03.044416+0000","last_peered":"2023-05-22T15:47:03.044416+0000","last_clean":"2023-05-22T15:47:03.044416+0000","last_became_active":"2023-05-22T15:47:03.042914+0000","last_became_peered":"2023-05-22T15:47:03.042914+0000","last_unstale":"2023-05-22T15:47:03.044416+0000","last_undegraded":"2023-05-22T15:47:03.044416+0000","last_fullsized":"2023-05-22T15:47:03.044416+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_clean_scrub_stamp":"2023-05-22T15:47:02.606868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:59:07.114320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.3d","version":"0'0","reported_seq":9,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-05-22T15:47:03.098373+0000","last_change":"2023-05-22T15:47:03.091606+0000","last_active":"2023-05-22T15:47:03.098373+0000","last_peered":"2023-05-22T15:47:03.098373+0000","last_clean":"2023-05-22T15:47:03.098373+0000","last_became_active":"2023-05-22T15:47:03.087886+0000","last_became_peered":"2023-05-22T15:47:03.087886+0000","last_unstale":"2023-05-22T15:47:03.098373+0000","last_undegraded":"2023-05-22T15:47:03.098373+0000","last_fullsized":"2023-05-22T15:47:03.098373+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_clean_scrub_stamp":"2023-05-22T15:47:02.606868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T03:11:13.112290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.3c","version":"0'0","reported_seq":9,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-05-22T15:47:03.097078+0000","last_change":"2023-05-22T15:47:03.095261+0000","last_active":"2023-05-22T15:47:03.097078+0000","last_peered":"2023-05-22T15:47:03.097078+0000","last_clean":"2023-05-22T15:47:03.097078+0000","last_became_active":"2023-05-22T15:47:03.094750+0000","last_became_peered":"2023-05-22T15:47:03.094750+0000","last_unstale":"2023-05-22T15:47:03.097078+0000","last_undegraded":"2023-05-22T15:47:03.097078+0000","last_fullsized":"2023-05-22T15:47:03.097078+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_clean_scrub_stamp":"2023-05-22T15:47:02.606868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:34:17.773670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.3b","version":"0'0","reported_seq":9,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-05-22T15:47:03.082841+0000","last_change":"2023-05-22T15:47:03.077042+0000","last_active":"2023-05-22T15:47:03.082841+0000","last_peered":"2023-05-22T15:47:03.082841+0000","last_clean":"2023-05-22T15:47:03.082841+0000","last_became_active":"2023-05-22T15:47:03.076461+0000","last_became_peered":"2023-05-22T15:47:03.076461+0000","last_unstale":"2023-05-22T15:47:03.082841+0000","last_undegraded":"2023-05-22T15:47:03.082841+0000","last_fullsized":"2023-05-22T15:47:03.082841+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_clean_scrub_stamp":"2023-05-22T15:47:02.606868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T03:08:55.264476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.3a","version":"0'0","reported_seq":9,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-05-22T15:47:03.094946+0000","last_change":"2023-05-22T15:47:03.094209+0000","last_active":"2023-05-22T15:47:03.094946+0000","last_peered":"2023-05-22T15:47:03.094946+0000","last_clean":"2023-05-22T15:47:03.094946+0000","last_became_active":"2023-05-22T15:47:03.093954+0000","last_became_peered":"2023-05-22T15:47:03.093954+0000","last_unstale":"2023-05-22T15:47:03.094946+0000","last_undegraded":"2023-05-22T15:47:03.094946+0000","last_fullsized":"2023-05-22T15:47:03.094946+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_clean_scrub_stamp":"2023-05-22T15:47:02.606868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:33:18.873310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.39","version":"0'0","reported_seq":9,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-05-22T15:47:03.047868+0000","last_change":"2023-05-22T15:47:03.044426+0000","last_active":"2023-05-22T15:47:03.047868+0000","last_peered":"2023-05-22T15:47:03.047868+0000","last_clean":"2023-05-22T15:47:03.047868+0000","last_became_active":"2023-05-22T15:47:03.044112+0000","last_became_peered":"2023-05-22T15:47:03.044112+0000","last_unstale":"2023-05-22T15:47:03.047868+0000","last_undegraded":"2023-05-22T15:47:03.047868+0000","last_fullsized":"2023-05-22T15:47:03.047868+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_clean_scrub_stamp":"2023-05-22T15:47:02.606868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:11:03.531615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.38","version":"0'0","reported_seq":9,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-05-22T15:47:03.053029+0000","last_change":"2023-05-22T15:47:03.051971+0000","last_active":"2023-05-22T15:47:03.053029+0000","last_peered":"2023-05-22T15:47:03.053029+0000","last_clean":"2023-05-22T15:47:03.053029+0000","last_became_active":"2023-05-22T15:47:03.051613+0000","last_became_peered":"2023-05-22T15:47:03.051613+0000","last_unstale":"2023-05-22T15:47:03.053029+0000","last_undegraded":"2023-05-22T15:47:03.053029+0000","last_fullsized":"2023-05-22T15:47:03.053029+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_clean_scrub_stamp":"2023-05-22T15:47:02.606868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:53:22.500441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.37","version":"0'0","reported_seq":9,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-05-22T15:47:03.043208+0000","last_change":"2023-05-22T15:47:03.041188+0000","last_active":"2023-05-22T15:47:03.043208+0000","last_peered":"2023-05-22T15:47:03.043208+0000","last_clean":"2023-05-22T15:47:03.043208+0000","last_became_active":"2023-05-22T15:47:03.040929+0000","last_became_peered":"2023-05-22T15:47:03.040929+0000","last_unstale":"2023-05-22T15:47:03.043208+0000","last_undegraded":"2023-05-22T15:47:03.043208+0000","last_fullsized":"2023-05-22T15:47:03.043208+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_clean_scrub_stamp":"2023-05-22T15:47:02.606868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:38:13.016335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.36","version":"0'0","reported_seq":9,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-05-22T15:47:03.090999+0000","last_change":"2023-05-22T15:47:03.082621+0000","last_active":"2023-05-22T15:47:03.090999+0000","last_peered":"2023-05-22T15:47:03.090999+0000","last_clean":"2023-05-22T15:47:03.090999+0000","last_became_active":"2023-05-22T15:47:03.082040+0000","last_became_peered":"2023-05-22T15:47:03.082040+0000","last_unstale":"2023-05-22T15:47:03.090999+0000","last_undegraded":"2023-05-22T15:47:03.090999+0000","last_fullsized":"2023-05-22T15:47:03.090999+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_clean_scrub_stamp":"2023-05-22T15:47:02.606868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:01:11.576865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.35","version":"0'0","reported_seq":9,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-05-22T15:47:03.053009+0000","last_change":"2023-05-22T15:47:03.051978+0000","last_active":"2023-05-22T15:47:03.053009+0000","last_peered":"2023-05-22T15:47:03.053009+0000","last_clean":"2023-05-22T15:47:03.053009+0000","last_became_active":"2023-05-22T15:47:03.051666+0000","last_became_peered":"2023-05-22T15:47:03.051666+0000","last_unstale":"2023-05-22T15:47:03.053009+0000","last_undegraded":"2023-05-22T15:47:03.053009+0000","last_fullsized":"2023-05-22T15:47:03.053009+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_clean_scrub_stamp":"2023-05-22T15:47:02.606868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T16:03:39.232845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.34","version":"0'0","reported_seq":9,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-05-22T15:47:03.085407+0000","last_change":"2023-05-22T15:47:03.071738+0000","last_active":"2023-05-22T15:47:03.085407+0000","last_peered":"2023-05-22T15:47:03.085407+0000","last_clean":"2023-05-22T15:47:03.085407+0000","last_became_active":"2023-05-22T15:47:03.071046+0000","last_became_peered":"2023-05-22T15:47:03.071046+0000","last_unstale":"2023-05-22T15:47:03.085407+0000","last_undegraded":"2023-05-22T15:47:03.085407+0000","last_fullsized":"2023-05-22T15:47:03.085407+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_clean_scrub_stamp":"2023-05-22T15:47:02.606868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:44:06.543666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.33","version":"0'0","reported_seq":9,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-05-22T15:47:03.077377+0000","last_change":"2023-05-22T15:47:03.073487+0000","last_active":"2023-05-22T15:47:03.077377+0000","last_peered":"2023-05-22T15:47:03.077377+0000","last_clean":"2023-05-22T15:47:03.077377+0000","last_became_active":"2023-05-22T15:47:03.072913+0000","last_became_peered":"2023-05-22T15:47:03.072913+0000","last_unstale":"2023-05-22T15:47:03.077377+0000","last_undegraded":"2023-05-22T15:47:03.077377+0000","last_fullsized":"2023-05-22T15:47:03.077377+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_clean_scrub_stamp":"2023-05-22T15:47:02.606868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T16:39:32.806138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.32","version":"0'0","reported_seq":9,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-05-22T15:47:03.083712+0000","last_change":"2023-05-22T15:47:03.081205+0000","last_active":"2023-05-22T15:47:03.083712+0000","last_peered":"2023-05-22T15:47:03.083712+0000","last_clean":"2023-05-22T15:47:03.083712+0000","last_became_active":"2023-05-22T15:47:03.080579+0000","last_became_peered":"2023-05-22T15:47:03.080579+0000","last_unstale":"2023-05-22T15:47:03.083712+0000","last_undegraded":"2023-05-22T15:47:03.083712+0000","last_fullsized":"2023-05-22T15:47:03.083712+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_clean_scrub_stamp":"2023-05-22T15:47:02.606868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:04:53.425125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.31","version":"0'0","reported_seq":9,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-05-22T15:47:03.085348+0000","last_change":"2023-05-22T15:47:03.071738+0000","last_active":"2023-05-22T15:47:03.085348+0000","last_peered":"2023-05-22T15:47:03.085348+0000","last_clean":"2023-05-22T15:47:03.085348+0000","last_became_active":"2023-05-22T15:47:03.071137+0000","last_became_peered":"2023-05-22T15:47:03.071137+0000","last_unstale":"2023-05-22T15:47:03.085348+0000","last_undegraded":"2023-05-22T15:47:03.085348+0000","last_fullsized":"2023-05-22T15:47:03.085348+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_clean_scrub_stamp":"2023-05-22T15:47:02.606868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:04:53.425125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.30","version":"0'0","reported_seq":9,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-05-22T15:47:03.039838+0000","last_change":"2023-05-22T15:47:03.038261+0000","last_active":"2023-05-22T15:47:03.039838+0000","last_peered":"2023-05-22T15:47:03.039838+0000","last_clean":"2023-05-22T15:47:03.039838+0000","last_became_active":"2023-05-22T15:47:03.037968+0000","last_became_peered":"2023-05-22T15:47:03.037968+0000","last_unstale":"2023-05-22T15:47:03.039838+0000","last_undegraded":"2023-05-22T15:47:03.039838+0000","last_fullsized":"2023-05-22T15:47:03.039838+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_clean_scrub_stamp":"2023-05-22T15:47:02.606868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:52:37.159122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.2f","version":"0'0","reported_seq":9,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-05-22T15:47:03.090787+0000","last_change":"2023-05-22T15:47:03.077823+0000","last_active":"2023-05-22T15:47:03.090787+0000","last_peered":"2023-05-22T15:47:03.090787+0000","last_clean":"2023-05-22T15:47:03.090787+0000","last_became_active":"2023-05-22T15:47:03.074360+0000","last_became_peered":"2023-05-22T15:47:03.074360+0000","last_unstale":"2023-05-22T15:47:03.090787+0000","last_undegraded":"2023-05-22T15:47:03.090787+0000","last_fullsized":"2023-05-22T15:47:03.090787+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_clean_scrub_stamp":"2023-05-22T15:47:02.606868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T15:57:50.277224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.2e","version":"0'0","reported_seq":9,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-05-22T15:47:03.098310+0000","last_change":"2023-05-22T15:47:03.088461+0000","last_active":"2023-05-22T15:47:03.098310+0000","last_peered":"2023-05-22T15:47:03.098310+0000","last_clean":"2023-05-22T15:47:03.098310+0000","last_became_active":"2023-05-22T15:47:03.086415+0000","last_became_peered":"2023-05-22T15:47:03.086415+0000","last_unstale":"2023-05-22T15:47:03.098310+0000","last_undegraded":"2023-05-22T15:47:03.098310+0000","last_fullsized":"2023-05-22T15:47:03.098310+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_clean_scrub_stamp":"2023-05-22T15:47:02.606868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:07:06.587412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.2d","version":"0'0","reported_seq":9,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-05-22T15:47:03.040380+0000","last_change":"2023-05-22T15:47:03.037825+0000","last_active":"2023-05-22T15:47:03.040380+0000","last_peered":"2023-05-22T15:47:03.040380+0000","last_clean":"2023-05-22T15:47:03.040380+0000","last_became_active":"2023-05-22T15:47:03.037567+0000","last_became_peered":"2023-05-22T15:47:03.037567+0000","last_unstale":"2023-05-22T15:47:03.040380+0000","last_undegraded":"2023-05-22T15:47:03.040380+0000","last_fullsized":"2023-05-22T15:47:03.040380+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_clean_scrub_stamp":"2023-05-22T15:47:02.606868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T16:14:58.298182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.2c","version":"0'0","reported_seq":9,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-05-22T15:47:03.041610+0000","last_change":"2023-05-22T15:47:03.041008+0000","last_active":"2023-05-22T15:47:03.041610+0000","last_peered":"2023-05-22T15:47:03.041610+0000","last_clean":"2023-05-22T15:47:03.041610+0000","last_became_active":"2023-05-22T15:47:03.040700+0000","last_became_peered":"2023-05-22T15:47:03.040700+0000","last_unstale":"2023-05-22T15:47:03.041610+0000","last_undegraded":"2023-05-22T15:47:03.041610+0000","last_fullsized":"2023-05-22T15:47:03.041610+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_clean_scrub_stamp":"2023-05-22T15:47:02.606868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:48:50.437568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.2b","version":"0'0","reported_seq":9,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-05-22T15:47:03.090888+0000","last_change":"2023-05-22T15:47:03.081786+0000","last_active":"2023-05-22T15:47:03.090888+0000","last_peered":"2023-05-22T15:47:03.090888+0000","last_clean":"2023-05-22T15:47:03.090888+0000","last_became_active":"2023-05-22T15:47:03.081105+0000","last_became_peered":"2023-05-22T15:47:03.081105+0000","last_unstale":"2023-05-22T15:47:03.090888+0000","last_undegraded":"2023-05-22T15:47:03.090888+0000","last_fullsized":"2023-05-22T15:47:03.090888+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_clean_scrub_stamp":"2023-05-22T15:47:02.606868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:07:48.107370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.2a","version":"0'0","reported_seq":9,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-05-22T15:47:03.072036+0000","last_change":"2023-05-22T15:47:03.047875+0000","last_active":"2023-05-22T15:47:03.072036+0000","last_peered":"2023-05-22T15:47:03.072036+0000","last_clean":"2023-05-22T15:47:03.072036+0000","last_became_active":"2023-05-22T15:47:03.046928+0000","last_became_peered":"2023-05-22T15:47:03.046928+0000","last_unstale":"2023-05-22T15:47:03.072036+0000","last_undegraded":"2023-05-22T15:47:03.072036+0000","last_fullsized":"2023-05-22T15:47:03.072036+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_clean_scrub_stamp":"2023-05-22T15:47:02.606868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:56:02.336059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.29","version":"0'0","reported_seq":9,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-05-22T15:47:03.046843+0000","last_change":"2023-05-22T15:47:03.038559+0000","last_active":"2023-05-22T15:47:03.046843+0000","last_peered":"2023-05-22T15:47:03.046843+0000","last_clean":"2023-05-22T15:47:03.046843+0000","last_became_active":"2023-05-22T15:47:03.037536+0000","last_became_peered":"2023-05-22T15:47:03.037536+0000","last_unstale":"2023-05-22T15:47:03.046843+0000","last_undegraded":"2023-05-22T15:47:03.046843+0000","last_fullsized":"2023-05-22T15:47:03.046843+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_clean_scrub_stamp":"2023-05-22T15:47:02.606868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:44:41.653299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.28","version":"0'0","reported_seq":9,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-05-22T15:47:03.091006+0000","last_change":"2023-05-22T15:47:03.082635+0000","last_active":"2023-05-22T15:47:03.091006+0000","last_peered":"2023-05-22T15:47:03.091006+0000","last_clean":"2023-05-22T15:47:03.091006+0000","last_became_active":"2023-05-22T15:47:03.081929+0000","last_became_peered":"2023-05-22T15:47:03.081929+0000","last_unstale":"2023-05-22T15:47:03.091006+0000","last_undegraded":"2023-05-22T15:47:03.091006+0000","last_fullsized":"2023-05-22T15:47:03.091006+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_clean_scrub_stamp":"2023-05-22T15:47:02.606868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:47:06.786177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.27","version":"0'0","reported_seq":9,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-05-22T15:47:03.043145+0000","last_change":"2023-05-22T15:47:03.041937+0000","last_active":"2023-05-22T15:47:03.043145+0000","last_peered":"2023-05-22T15:47:03.043145+0000","last_clean":"2023-05-22T15:47:03.043145+0000","last_became_active":"2023-05-22T15:47:03.041609+0000","last_became_peered":"2023-05-22T15:47:03.041609+0000","last_unstale":"2023-05-22T15:47:03.043145+0000","last_undegraded":"2023-05-22T15:47:03.043145+0000","last_fullsized":"2023-05-22T15:47:03.043145+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_clean_scrub_stamp":"2023-05-22T15:47:02.606868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:55:53.695955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.26","version":"0'0","reported_seq":9,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-05-22T15:47:03.045868+0000","last_change":"2023-05-22T15:47:03.041186+0000","last_active":"2023-05-22T15:47:03.045868+0000","last_peered":"2023-05-22T15:47:03.045868+0000","last_clean":"2023-05-22T15:47:03.045868+0000","last_became_active":"2023-05-22T15:47:03.040921+0000","last_became_peered":"2023-05-22T15:47:03.040921+0000","last_unstale":"2023-05-22T15:47:03.045868+0000","last_undegraded":"2023-05-22T15:47:03.045868+0000","last_fullsized":"2023-05-22T15:47:03.045868+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_clean_scrub_stamp":"2023-05-22T15:47:02.606868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T16:02:03.177522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.25","version":"0'0","reported_seq":9,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-05-22T15:47:03.091528+0000","last_change":"2023-05-22T15:47:03.090733+0000","last_active":"2023-05-22T15:47:03.091528+0000","last_peered":"2023-05-22T15:47:03.091528+0000","last_clean":"2023-05-22T15:47:03.091528+0000","last_became_active":"2023-05-22T15:47:03.090411+0000","last_became_peered":"2023-05-22T15:47:03.090411+0000","last_unstale":"2023-05-22T15:47:03.091528+0000","last_undegraded":"2023-05-22T15:47:03.091528+0000","last_fullsized":"2023-05-22T15:47:03.091528+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_clean_scrub_stamp":"2023-05-22T15:47:02.606868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:10:35.110422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.24","version":"0'0","reported_seq":9,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-05-22T15:47:03.088241+0000","last_change":"2023-05-22T15:47:03.087558+0000","last_active":"2023-05-22T15:47:03.088241+0000","last_peered":"2023-05-22T15:47:03.088241+0000","last_clean":"2023-05-22T15:47:03.088241+0000","last_became_active":"2023-05-22T15:47:03.087209+0000","last_became_peered":"2023-05-22T15:47:03.087209+0000","last_unstale":"2023-05-22T15:47:03.088241+0000","last_undegraded":"2023-05-22T15:47:03.088241+0000","last_fullsized":"2023-05-22T15:47:03.088241+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_clean_scrub_stamp":"2023-05-22T15:47:02.606868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:06:44.653800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.23","version":"0'0","reported_seq":9,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-05-22T15:47:03.092043+0000","last_change":"2023-05-22T15:47:03.091438+0000","last_active":"2023-05-22T15:47:03.092043+0000","last_peered":"2023-05-22T15:47:03.092043+0000","last_clean":"2023-05-22T15:47:03.092043+0000","last_became_active":"2023-05-22T15:47:03.091107+0000","last_became_peered":"2023-05-22T15:47:03.091107+0000","last_unstale":"2023-05-22T15:47:03.092043+0000","last_undegraded":"2023-05-22T15:47:03.092043+0000","last_fullsized":"2023-05-22T15:47:03.092043+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_clean_scrub_stamp":"2023-05-22T15:47:02.606868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:03:49.167367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.22","version":"0'0","reported_seq":9,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-05-22T15:47:03.067544+0000","last_change":"2023-05-22T15:47:03.046229+0000","last_active":"2023-05-22T15:47:03.067544+0000","last_peered":"2023-05-22T15:47:03.067544+0000","last_clean":"2023-05-22T15:47:03.067544+0000","last_became_active":"2023-05-22T15:47:03.045295+0000","last_became_peered":"2023-05-22T15:47:03.045295+0000","last_unstale":"2023-05-22T15:47:03.067544+0000","last_undegraded":"2023-05-22T15:47:03.067544+0000","last_fullsized":"2023-05-22T15:47:03.067544+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_clean_scrub_stamp":"2023-05-22T15:47:02.606868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:42:53.822619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.21","version":"0'0","reported_seq":9,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-05-22T15:47:03.087850+0000","last_change":"2023-05-22T15:47:03.086358+0000","last_active":"2023-05-22T15:47:03.087850+0000","last_peered":"2023-05-22T15:47:03.087850+0000","last_clean":"2023-05-22T15:47:03.087850+0000","last_became_active":"2023-05-22T15:47:03.086013+0000","last_became_peered":"2023-05-22T15:47:03.086013+0000","last_unstale":"2023-05-22T15:47:03.087850+0000","last_undegraded":"2023-05-22T15:47:03.087850+0000","last_fullsized":"2023-05-22T15:47:03.087850+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_clean_scrub_stamp":"2023-05-22T15:47:02.606868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:27:02.805634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.20","version":"0'0","reported_seq":9,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-05-22T15:47:03.032633+0000","last_change":"2023-05-22T15:47:03.028936+0000","last_active":"2023-05-22T15:47:03.032633+0000","last_peered":"2023-05-22T15:47:03.032633+0000","last_clean":"2023-05-22T15:47:03.032633+0000","last_became_active":"2023-05-22T15:47:03.028654+0000","last_became_peered":"2023-05-22T15:47:03.028654+0000","last_unstale":"2023-05-22T15:47:03.032633+0000","last_undegraded":"2023-05-22T15:47:03.032633+0000","last_fullsized":"2023-05-22T15:47:03.032633+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_clean_scrub_stamp":"2023-05-22T15:47:02.606868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:23:25.039826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.1e","version":"0'0","reported_seq":9,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-05-22T15:47:03.085690+0000","last_change":"2023-05-22T15:47:03.073955+0000","last_active":"2023-05-22T15:47:03.085690+0000","last_peered":"2023-05-22T15:47:03.085690+0000","last_clean":"2023-05-22T15:47:03.085690+0000","last_became_active":"2023-05-22T15:47:03.072778+0000","last_became_peered":"2023-05-22T15:47:03.072778+0000","last_unstale":"2023-05-22T15:47:03.085690+0000","last_undegraded":"2023-05-22T15:47:03.085690+0000","last_fullsized":"2023-05-22T15:47:03.085690+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_clean_scrub_stamp":"2023-05-22T15:47:02.606868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:38:13.016335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1f","version":"160'4","reported_seq":14,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-05-22T15:47:03.078675+0000","last_change":"2023-05-22T15:47:02.688458+0000","last_active":"2023-05-22T15:47:03.078675+0000","last_peered":"2023-05-22T15:47:03.078675+0000","last_clean":"2023-05-22T15:47:03.078675+0000","last_became_active":"2023-05-22T15:47:02.686134+0000","last_became_peered":"2023-05-22T15:47:02.686134+0000","last_unstale":"2023-05-22T15:47:03.078675+0000","last_undegraded":"2023-05-22T15:47:03.078675+0000","last_fullsized":"2023-05-22T15:47:03.078675+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:47:01.602136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:47:01.602136+0000","last_clean_scrub_stamp":"2023-05-22T15:47:01.602136+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:22:32.490190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":566,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.1f","version":"0'0","reported_seq":9,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-05-22T15:47:03.043226+0000","last_change":"2023-05-22T15:47:03.041771+0000","last_active":"2023-05-22T15:47:03.043226+0000","last_peered":"2023-05-22T15:47:03.043226+0000","last_clean":"2023-05-22T15:47:03.043226+0000","last_became_active":"2023-05-22T15:47:03.041491+0000","last_became_peered":"2023-05-22T15:47:03.041491+0000","last_unstale":"2023-05-22T15:47:03.043226+0000","last_undegraded":"2023-05-22T15:47:03.043226+0000","last_fullsized":"2023-05-22T15:47:03.043226+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_clean_scrub_stamp":"2023-05-22T15:47:02.606868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:07:48.107370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.1e","version":"160'1","reported_seq":11,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-05-22T15:47:03.051041+0000","last_change":"2023-05-22T15:47:02.684398+0000","last_active":"2023-05-22T15:47:03.051041+0000","last_peered":"2023-05-22T15:47:03.051041+0000","last_clean":"2023-05-22T15:47:03.051041+0000","last_became_active":"2023-05-22T15:47:02.684028+0000","last_became_peered":"2023-05-22T15:47:02.684028+0000","last_unstale":"2023-05-22T15:47:03.051041+0000","last_undegraded":"2023-05-22T15:47:03.051041+0000","last_fullsized":"2023-05-22T15:47:03.051041+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:47:01.602136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:47:01.602136+0000","last_clean_scrub_stamp":"2023-05-22T15:47:01.602136+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:11:04.259677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.1c","version":"0'0","reported_seq":9,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-05-22T15:47:03.067453+0000","last_change":"2023-05-22T15:47:03.066707+0000","last_active":"2023-05-22T15:47:03.067453+0000","last_peered":"2023-05-22T15:47:03.067453+0000","last_clean":"2023-05-22T15:47:03.067453+0000","last_became_active":"2023-05-22T15:47:03.066422+0000","last_became_peered":"2023-05-22T15:47:03.066422+0000","last_unstale":"2023-05-22T15:47:03.067453+0000","last_undegraded":"2023-05-22T15:47:03.067453+0000","last_fullsized":"2023-05-22T15:47:03.067453+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_clean_scrub_stamp":"2023-05-22T15:47:02.606868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T03:11:13.112290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.1d","version":"160'2","reported_seq":12,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-05-22T15:47:03.066676+0000","last_change":"2023-05-22T15:47:02.666690+0000","last_active":"2023-05-22T15:47:03.066676+0000","last_peered":"2023-05-22T15:47:03.066676+0000","last_clean":"2023-05-22T15:47:03.066676+0000","last_became_active":"2023-05-22T15:47:02.665970+0000","last_became_peered":"2023-05-22T15:47:02.665970+0000","last_unstale":"2023-05-22T15:47:03.066676+0000","last_undegraded":"2023-05-22T15:47:03.066676+0000","last_fullsized":"2023-05-22T15:47:03.066676+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:47:01.602136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:47:01.602136+0000","last_clean_scrub_stamp":"2023-05-22T15:47:01.602136+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:15:35.219530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.1d","version":"0'0","reported_seq":9,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-05-22T15:47:03.088705+0000","last_change":"2023-05-22T15:47:03.075131+0000","last_active":"2023-05-22T15:47:03.088705+0000","last_peered":"2023-05-22T15:47:03.088705+0000","last_clean":"2023-05-22T15:47:03.088705+0000","last_became_active":"2023-05-22T15:47:03.074431+0000","last_became_peered":"2023-05-22T15:47:03.074431+0000","last_unstale":"2023-05-22T15:47:03.088705+0000","last_undegraded":"2023-05-22T15:47:03.088705+0000","last_fullsized":"2023-05-22T15:47:03.088705+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_clean_scrub_stamp":"2023-05-22T15:47:02.606868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:19:09.278066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1c","version":"160'2","reported_seq":12,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-05-22T15:47:03.067543+0000","last_change":"2023-05-22T15:47:02.663527+0000","last_active":"2023-05-22T15:47:03.067543+0000","last_peered":"2023-05-22T15:47:03.067543+0000","last_clean":"2023-05-22T15:47:03.067543+0000","last_became_active":"2023-05-22T15:47:02.662738+0000","last_became_peered":"2023-05-22T15:47:02.662738+0000","last_unstale":"2023-05-22T15:47:03.067543+0000","last_undegraded":"2023-05-22T15:47:03.067543+0000","last_fullsized":"2023-05-22T15:47:03.067543+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:47:01.602136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:47:01.602136+0000","last_clean_scrub_stamp":"2023-05-22T15:47:01.602136+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:46:40.385953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.1a","version":"0'0","reported_seq":9,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-05-22T15:47:03.087365+0000","last_change":"2023-05-22T15:47:03.086342+0000","last_active":"2023-05-22T15:47:03.087365+0000","last_peered":"2023-05-22T15:47:03.087365+0000","last_clean":"2023-05-22T15:47:03.087365+0000","last_became_active":"2023-05-22T15:47:03.086023+0000","last_became_peered":"2023-05-22T15:47:03.086023+0000","last_unstale":"2023-05-22T15:47:03.087365+0000","last_undegraded":"2023-05-22T15:47:03.087365+0000","last_fullsized":"2023-05-22T15:47:03.087365+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_clean_scrub_stamp":"2023-05-22T15:47:02.606868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:47:06.786177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.1b","version":"160'1","reported_seq":11,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-05-22T15:47:03.058406+0000","last_change":"2023-05-22T15:47:02.684478+0000","last_active":"2023-05-22T15:47:03.058406+0000","last_peered":"2023-05-22T15:47:03.058406+0000","last_clean":"2023-05-22T15:47:03.058406+0000","last_became_active":"2023-05-22T15:47:02.684125+0000","last_became_peered":"2023-05-22T15:47:02.684125+0000","last_unstale":"2023-05-22T15:47:03.058406+0000","last_undegraded":"2023-05-22T15:47:03.058406+0000","last_fullsized":"2023-05-22T15:47:03.058406+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:47:01.602136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:47:01.602136+0000","last_clean_scrub_stamp":"2023-05-22T15:47:01.602136+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T16:39:09.891365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.1b","version":"0'0","reported_seq":9,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-05-22T15:47:03.097134+0000","last_change":"2023-05-22T15:47:03.095177+0000","last_active":"2023-05-22T15:47:03.097134+0000","last_peered":"2023-05-22T15:47:03.097134+0000","last_clean":"2023-05-22T15:47:03.097134+0000","last_became_active":"2023-05-22T15:47:03.094713+0000","last_became_peered":"2023-05-22T15:47:03.094713+0000","last_unstale":"2023-05-22T15:47:03.097134+0000","last_undegraded":"2023-05-22T15:47:03.097134+0000","last_fullsized":"2023-05-22T15:47:03.097134+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_clean_scrub_stamp":"2023-05-22T15:47:02.606868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:44:06.543666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.1a","version":"160'1","reported_seq":11,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-05-22T15:47:03.062830+0000","last_change":"2023-05-22T15:47:02.671236+0000","last_active":"2023-05-22T15:47:03.062830+0000","last_peered":"2023-05-22T15:47:03.062830+0000","last_clean":"2023-05-22T15:47:03.062830+0000","last_became_active":"2023-05-22T15:47:02.670542+0000","last_became_peered":"2023-05-22T15:47:02.670542+0000","last_unstale":"2023-05-22T15:47:03.062830+0000","last_undegraded":"2023-05-22T15:47:03.062830+0000","last_fullsized":"2023-05-22T15:47:03.062830+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:47:01.602136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:47:01.602136+0000","last_clean_scrub_stamp":"2023-05-22T15:47:01.602136+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:03:53.697701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.18","version":"0'0","reported_seq":9,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-05-22T15:47:03.044665+0000","last_change":"2023-05-22T15:47:03.043930+0000","last_active":"2023-05-22T15:47:03.044665+0000","last_peered":"2023-05-22T15:47:03.044665+0000","last_clean":"2023-05-22T15:47:03.044665+0000","last_became_active":"2023-05-22T15:47:03.043647+0000","last_became_peered":"2023-05-22T15:47:03.043647+0000","last_unstale":"2023-05-22T15:47:03.044665+0000","last_undegraded":"2023-05-22T15:47:03.044665+0000","last_fullsized":"2023-05-22T15:47:03.044665+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_clean_scrub_stamp":"2023-05-22T15:47:02.606868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:54:58.772664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.19","version":"0'0","reported_seq":10,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-05-22T15:47:03.018820+0000","last_change":"2023-05-22T15:47:02.656840+0000","last_active":"2023-05-22T15:47:03.018820+0000","last_peered":"2023-05-22T15:47:03.018820+0000","last_clean":"2023-05-22T15:47:03.018820+0000","last_became_active":"2023-05-22T15:47:02.656449+0000","last_became_peered":"2023-05-22T15:47:02.656449+0000","last_unstale":"2023-05-22T15:47:03.018820+0000","last_undegraded":"2023-05-22T15:47:03.018820+0000","last_fullsized":"2023-05-22T15:47:03.018820+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:47:01.602136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:47:01.602136+0000","last_clean_scrub_stamp":"2023-05-22T15:47:01.602136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:59:48.010931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.19","version":"0'0","reported_seq":9,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-05-22T15:47:03.052557+0000","last_change":"2023-05-22T15:47:03.041807+0000","last_active":"2023-05-22T15:47:03.052557+0000","last_peered":"2023-05-22T15:47:03.052557+0000","last_clean":"2023-05-22T15:47:03.052557+0000","last_became_active":"2023-05-22T15:47:03.041060+0000","last_became_peered":"2023-05-22T15:47:03.041060+0000","last_unstale":"2023-05-22T15:47:03.052557+0000","last_undegraded":"2023-05-22T15:47:03.052557+0000","last_fullsized":"2023-05-22T15:47:03.052557+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_clean_scrub_stamp":"2023-05-22T15:47:02.606868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:31:19.675774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.18","version":"0'0","reported_seq":10,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-05-22T15:47:03.033095+0000","last_change":"2023-05-22T15:47:02.681905+0000","last_active":"2023-05-22T15:47:03.033095+0000","last_peered":"2023-05-22T15:47:03.033095+0000","last_clean":"2023-05-22T15:47:03.033095+0000","last_became_active":"2023-05-22T15:47:02.681167+0000","last_became_peered":"2023-05-22T15:47:02.681167+0000","last_unstale":"2023-05-22T15:47:03.033095+0000","last_undegraded":"2023-05-22T15:47:03.033095+0000","last_fullsized":"2023-05-22T15:47:03.033095+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:47:01.602136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:47:01.602136+0000","last_clean_scrub_stamp":"2023-05-22T15:47:01.602136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:31:26.628654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.e","version":"0'0","reported_seq":9,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-05-22T15:47:03.085727+0000","last_change":"2023-05-22T15:47:03.074417+0000","last_active":"2023-05-22T15:47:03.085727+0000","last_peered":"2023-05-22T15:47:03.085727+0000","last_clean":"2023-05-22T15:47:03.085727+0000","last_became_active":"2023-05-22T15:47:03.073379+0000","last_became_peered":"2023-05-22T15:47:03.073379+0000","last_unstale":"2023-05-22T15:47:03.085727+0000","last_undegraded":"2023-05-22T15:47:03.085727+0000","last_fullsized":"2023-05-22T15:47:03.085727+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_clean_scrub_stamp":"2023-05-22T15:47:02.606868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:54:58.772664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.f","version":"160'1","reported_seq":11,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-05-22T15:47:03.088606+0000","last_change":"2023-05-22T15:47:02.658045+0000","last_active":"2023-05-22T15:47:03.088606+0000","last_peered":"2023-05-22T15:47:03.088606+0000","last_clean":"2023-05-22T15:47:03.088606+0000","last_became_active":"2023-05-22T15:47:02.657283+0000","last_became_peered":"2023-05-22T15:47:02.657283+0000","last_unstale":"2023-05-22T15:47:03.088606+0000","last_undegraded":"2023-05-22T15:47:03.088606+0000","last_fullsized":"2023-05-22T15:47:03.088606+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:47:01.602136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:47:01.602136+0000","last_clean_scrub_stamp":"2023-05-22T15:47:01.602136+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T15:59:56.722776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.f","version":"0'0","reported_seq":9,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-05-22T15:47:03.071873+0000","last_change":"2023-05-22T15:47:03.071253+0000","last_active":"2023-05-22T15:47:03.071873+0000","last_peered":"2023-05-22T15:47:03.071873+0000","last_clean":"2023-05-22T15:47:03.071873+0000","last_became_active":"2023-05-22T15:47:03.071044+0000","last_became_peered":"2023-05-22T15:47:03.071044+0000","last_unstale":"2023-05-22T15:47:03.071873+0000","last_undegraded":"2023-05-22T15:47:03.071873+0000","last_fullsized":"2023-05-22T15:47:03.071873+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_clean_scrub_stamp":"2023-05-22T15:47:02.606868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:33:18.873310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.e","version":"160'2","reported_seq":12,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-05-22T15:47:03.094140+0000","last_change":"2023-05-22T15:47:02.656792+0000","last_active":"2023-05-22T15:47:03.094140+0000","last_peered":"2023-05-22T15:47:03.094140+0000","last_clean":"2023-05-22T15:47:03.094140+0000","last_became_active":"2023-05-22T15:47:02.655920+0000","last_became_peered":"2023-05-22T15:47:02.655920+0000","last_unstale":"2023-05-22T15:47:03.094140+0000","last_undegraded":"2023-05-22T15:47:03.094140+0000","last_fullsized":"2023-05-22T15:47:03.094140+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:47:01.602136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:47:01.602136+0000","last_clean_scrub_stamp":"2023-05-22T15:47:01.602136+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T18:18:04.406735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.c","version":"0'0","reported_seq":9,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-05-22T15:47:03.098195+0000","last_change":"2023-05-22T15:47:03.087108+0000","last_active":"2023-05-22T15:47:03.098195+0000","last_peered":"2023-05-22T15:47:03.098195+0000","last_clean":"2023-05-22T15:47:03.098195+0000","last_became_active":"2023-05-22T15:47:03.086489+0000","last_became_peered":"2023-05-22T15:47:03.086489+0000","last_unstale":"2023-05-22T15:47:03.098195+0000","last_undegraded":"2023-05-22T15:47:03.098195+0000","last_fullsized":"2023-05-22T15:47:03.098195+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_clean_scrub_stamp":"2023-05-22T15:47:02.606868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T03:10:01.167398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.d","version":"160'1","reported_seq":11,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-05-22T15:47:03.058401+0000","last_change":"2023-05-22T15:47:02.679126+0000","last_active":"2023-05-22T15:47:03.058401+0000","last_peered":"2023-05-22T15:47:03.058401+0000","last_clean":"2023-05-22T15:47:03.058401+0000","last_became_active":"2023-05-22T15:47:02.678819+0000","last_became_peered":"2023-05-22T15:47:02.678819+0000","last_unstale":"2023-05-22T15:47:03.058401+0000","last_undegraded":"2023-05-22T15:47:03.058401+0000","last_fullsized":"2023-05-22T15:47:03.058401+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:47:01.602136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:47:01.602136+0000","last_clean_scrub_stamp":"2023-05-22T15:47:01.602136+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:52:36.659001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.d","version":"0'0","reported_seq":9,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-05-22T15:47:03.095380+0000","last_change":"2023-05-22T15:47:03.094345+0000","last_active":"2023-05-22T15:47:03.095380+0000","last_peered":"2023-05-22T15:47:03.095380+0000","last_clean":"2023-05-22T15:47:03.095380+0000","last_became_active":"2023-05-22T15:47:03.094059+0000","last_became_peered":"2023-05-22T15:47:03.094059+0000","last_unstale":"2023-05-22T15:47:03.095380+0000","last_undegraded":"2023-05-22T15:47:03.095380+0000","last_fullsized":"2023-05-22T15:47:03.095380+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_clean_scrub_stamp":"2023-05-22T15:47:02.606868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:20:31.328410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.c","version":"0'0","reported_seq":10,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-05-22T15:47:03.030699+0000","last_change":"2023-05-22T15:47:02.656940+0000","last_active":"2023-05-22T15:47:03.030699+0000","last_peered":"2023-05-22T15:47:03.030699+0000","last_clean":"2023-05-22T15:47:03.030699+0000","last_became_active":"2023-05-22T15:47:02.656439+0000","last_became_peered":"2023-05-22T15:47:02.656439+0000","last_unstale":"2023-05-22T15:47:03.030699+0000","last_undegraded":"2023-05-22T15:47:03.030699+0000","last_fullsized":"2023-05-22T15:47:03.030699+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:47:01.602136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:47:01.602136+0000","last_clean_scrub_stamp":"2023-05-22T15:47:01.602136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T16:47:26.602773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.a","version":"0'0","reported_seq":9,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-05-22T15:47:03.067489+0000","last_change":"2023-05-22T15:47:03.045975+0000","last_active":"2023-05-22T15:47:03.067489+0000","last_peered":"2023-05-22T15:47:03.067489+0000","last_clean":"2023-05-22T15:47:03.067489+0000","last_became_active":"2023-05-22T15:47:03.045079+0000","last_became_peered":"2023-05-22T15:47:03.045079+0000","last_unstale":"2023-05-22T15:47:03.067489+0000","last_undegraded":"2023-05-22T15:47:03.067489+0000","last_fullsized":"2023-05-22T15:47:03.067489+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_clean_scrub_stamp":"2023-05-22T15:47:02.606868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:19:39.903101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.b","version":"160'2","reported_seq":12,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-05-22T15:47:03.091202+0000","last_change":"2023-05-22T15:47:02.647704+0000","last_active":"2023-05-22T15:47:03.091202+0000","last_peered":"2023-05-22T15:47:03.091202+0000","last_clean":"2023-05-22T15:47:03.091202+0000","last_became_active":"2023-05-22T15:47:02.647016+0000","last_became_peered":"2023-05-22T15:47:02.647016+0000","last_unstale":"2023-05-22T15:47:03.091202+0000","last_undegraded":"2023-05-22T15:47:03.091202+0000","last_fullsized":"2023-05-22T15:47:03.091202+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:47:01.602136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:47:01.602136+0000","last_clean_scrub_stamp":"2023-05-22T15:47:01.602136+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T18:19:56.170763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.b","version":"0'0","reported_seq":9,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-05-22T15:47:03.048875+0000","last_change":"2023-05-22T15:47:03.046456+0000","last_active":"2023-05-22T15:47:03.048875+0000","last_peered":"2023-05-22T15:47:03.048875+0000","last_clean":"2023-05-22T15:47:03.048875+0000","last_became_active":"2023-05-22T15:47:03.046105+0000","last_became_peered":"2023-05-22T15:47:03.046105+0000","last_unstale":"2023-05-22T15:47:03.048875+0000","last_undegraded":"2023-05-22T15:47:03.048875+0000","last_fullsized":"2023-05-22T15:47:03.048875+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_clean_scrub_stamp":"2023-05-22T15:47:02.606868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:33:09.856999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.a","version":"0'0","reported_seq":10,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-05-22T15:47:03.062300+0000","last_change":"2023-05-22T15:47:02.686740+0000","last_active":"2023-05-22T15:47:03.062300+0000","last_peered":"2023-05-22T15:47:03.062300+0000","last_clean":"2023-05-22T15:47:03.062300+0000","last_became_active":"2023-05-22T15:47:02.685976+0000","last_became_peered":"2023-05-22T15:47:02.685976+0000","last_unstale":"2023-05-22T15:47:03.062300+0000","last_undegraded":"2023-05-22T15:47:03.062300+0000","last_fullsized":"2023-05-22T15:47:03.062300+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:47:01.602136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:47:01.602136+0000","last_clean_scrub_stamp":"2023-05-22T15:47:01.602136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:31:26.628654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.8","version":"0'0","reported_seq":9,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-05-22T15:47:03.070563+0000","last_change":"2023-05-22T15:47:03.048016+0000","last_active":"2023-05-22T15:47:03.070563+0000","last_peered":"2023-05-22T15:47:03.070563+0000","last_clean":"2023-05-22T15:47:03.070563+0000","last_became_active":"2023-05-22T15:47:03.047176+0000","last_became_peered":"2023-05-22T15:47:03.047176+0000","last_unstale":"2023-05-22T15:47:03.070563+0000","last_undegraded":"2023-05-22T15:47:03.070563+0000","last_fullsized":"2023-05-22T15:47:03.070563+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_clean_scrub_stamp":"2023-05-22T15:47:02.606868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T15:57:50.277224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.9","version":"0'0","reported_seq":10,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-05-22T15:47:03.050651+0000","last_change":"2023-05-22T15:47:02.687780+0000","last_active":"2023-05-22T15:47:03.050651+0000","last_peered":"2023-05-22T15:47:03.050651+0000","last_clean":"2023-05-22T15:47:03.050651+0000","last_became_active":"2023-05-22T15:47:02.686009+0000","last_became_peered":"2023-05-22T15:47:02.686009+0000","last_unstale":"2023-05-22T15:47:03.050651+0000","last_undegraded":"2023-05-22T15:47:03.050651+0000","last_fullsized":"2023-05-22T15:47:03.050651+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:47:01.602136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:47:01.602136+0000","last_clean_scrub_stamp":"2023-05-22T15:47:01.602136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:46:40.385953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.9","version":"0'0","reported_seq":9,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-05-22T15:47:03.047880+0000","last_change":"2023-05-22T15:47:03.044434+0000","last_active":"2023-05-22T15:47:03.047880+0000","last_peered":"2023-05-22T15:47:03.047880+0000","last_clean":"2023-05-22T15:47:03.047880+0000","last_became_active":"2023-05-22T15:47:03.043979+0000","last_became_peered":"2023-05-22T15:47:03.043979+0000","last_unstale":"2023-05-22T15:47:03.047880+0000","last_undegraded":"2023-05-22T15:47:03.047880+0000","last_fullsized":"2023-05-22T15:47:03.047880+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_clean_scrub_stamp":"2023-05-22T15:47:02.606868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:09:07.199478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.8","version":"0'0","reported_seq":10,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-05-22T15:47:03.074899+0000","last_change":"2023-05-22T15:47:02.657585+0000","last_active":"2023-05-22T15:47:03.074899+0000","last_peered":"2023-05-22T15:47:03.074899+0000","last_clean":"2023-05-22T15:47:03.074899+0000","last_became_active":"2023-05-22T15:47:02.656859+0000","last_became_peered":"2023-05-22T15:47:02.656859+0000","last_unstale":"2023-05-22T15:47:03.074899+0000","last_undegraded":"2023-05-22T15:47:03.074899+0000","last_fullsized":"2023-05-22T15:47:03.074899+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:47:01.602136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:47:01.602136+0000","last_clean_scrub_stamp":"2023-05-22T15:47:01.602136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:48:49.432836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.6","version":"0'0","reported_seq":9,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-05-22T15:47:03.080252+0000","last_change":"2023-05-22T15:47:03.073859+0000","last_active":"2023-05-22T15:47:03.080252+0000","last_peered":"2023-05-22T15:47:03.080252+0000","last_clean":"2023-05-22T15:47:03.080252+0000","last_became_active":"2023-05-22T15:47:03.073287+0000","last_became_peered":"2023-05-22T15:47:03.073287+0000","last_unstale":"2023-05-22T15:47:03.080252+0000","last_undegraded":"2023-05-22T15:47:03.080252+0000","last_fullsized":"2023-05-22T15:47:03.080252+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_clean_scrub_stamp":"2023-05-22T15:47:02.606868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T03:26:44.703194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.7","version":"160'2","reported_seq":12,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-05-22T15:47:03.094502+0000","last_change":"2023-05-22T15:47:02.688423+0000","last_active":"2023-05-22T15:47:03.094502+0000","last_peered":"2023-05-22T15:47:03.094502+0000","last_clean":"2023-05-22T15:47:03.094502+0000","last_became_active":"2023-05-22T15:47:02.686410+0000","last_became_peered":"2023-05-22T15:47:02.686410+0000","last_unstale":"2023-05-22T15:47:03.094502+0000","last_undegraded":"2023-05-22T15:47:03.094502+0000","last_fullsized":"2023-05-22T15:47:03.094502+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:47:01.602136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:47:01.602136+0000","last_clean_scrub_stamp":"2023-05-22T15:47:01.602136+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:12:10.461479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.7","version":"0'0","reported_seq":9,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-05-22T15:47:03.098292+0000","last_change":"2023-05-22T15:47:03.088246+0000","last_active":"2023-05-22T15:47:03.098292+0000","last_peered":"2023-05-22T15:47:03.098292+0000","last_clean":"2023-05-22T15:47:03.098292+0000","last_became_active":"2023-05-22T15:47:03.086815+0000","last_became_peered":"2023-05-22T15:47:03.086815+0000","last_unstale":"2023-05-22T15:47:03.098292+0000","last_undegraded":"2023-05-22T15:47:03.098292+0000","last_fullsized":"2023-05-22T15:47:03.098292+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_clean_scrub_stamp":"2023-05-22T15:47:02.606868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:19:52.996207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.6","version":"0'0","reported_seq":10,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-05-22T15:47:03.037761+0000","last_change":"2023-05-22T15:47:02.678858+0000","last_active":"2023-05-22T15:47:03.037761+0000","last_peered":"2023-05-22T15:47:03.037761+0000","last_clean":"2023-05-22T15:47:03.037761+0000","last_became_active":"2023-05-22T15:47:02.678514+0000","last_became_peered":"2023-05-22T15:47:02.678514+0000","last_unstale":"2023-05-22T15:47:03.037761+0000","last_undegraded":"2023-05-22T15:47:03.037761+0000","last_fullsized":"2023-05-22T15:47:03.037761+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:47:01.602136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:47:01.602136+0000","last_clean_scrub_stamp":"2023-05-22T15:47:01.602136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:31:18.671042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.4","version":"0'0","reported_seq":9,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-05-22T15:47:03.048540+0000","last_change":"2023-05-22T15:47:03.047949+0000","last_active":"2023-05-22T15:47:03.048540+0000","last_peered":"2023-05-22T15:47:03.048540+0000","last_clean":"2023-05-22T15:47:03.048540+0000","last_became_active":"2023-05-22T15:47:03.047587+0000","last_became_peered":"2023-05-22T15:47:03.047587+0000","last_unstale":"2023-05-22T15:47:03.048540+0000","last_undegraded":"2023-05-22T15:47:03.048540+0000","last_fullsized":"2023-05-22T15:47:03.048540+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_clean_scrub_stamp":"2023-05-22T15:47:02.606868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:19:09.278066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.5","version":"0'0","reported_seq":10,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-05-22T15:47:03.034791+0000","last_change":"2023-05-22T15:47:02.659857+0000","last_active":"2023-05-22T15:47:03.034791+0000","last_peered":"2023-05-22T15:47:03.034791+0000","last_clean":"2023-05-22T15:47:03.034791+0000","last_became_active":"2023-05-22T15:47:02.658823+0000","last_became_peered":"2023-05-22T15:47:02.658823+0000","last_unstale":"2023-05-22T15:47:03.034791+0000","last_undegraded":"2023-05-22T15:47:03.034791+0000","last_fullsized":"2023-05-22T15:47:03.034791+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:47:01.602136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:47:01.602136+0000","last_clean_scrub_stamp":"2023-05-22T15:47:01.602136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:22:17.743430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.5","version":"0'0","reported_seq":9,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-05-22T15:47:03.067155+0000","last_change":"2023-05-22T15:47:03.066466+0000","last_active":"2023-05-22T15:47:03.067155+0000","last_peered":"2023-05-22T15:47:03.067155+0000","last_clean":"2023-05-22T15:47:03.067155+0000","last_became_active":"2023-05-22T15:47:03.066191+0000","last_became_peered":"2023-05-22T15:47:03.066191+0000","last_unstale":"2023-05-22T15:47:03.067155+0000","last_undegraded":"2023-05-22T15:47:03.067155+0000","last_fullsized":"2023-05-22T15:47:03.067155+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_clean_scrub_stamp":"2023-05-22T15:47:02.606868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:17:27.515854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.4","version":"0'0","reported_seq":10,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-05-22T15:47:03.037870+0000","last_change":"2023-05-22T15:47:02.658286+0000","last_active":"2023-05-22T15:47:03.037870+0000","last_peered":"2023-05-22T15:47:03.037870+0000","last_clean":"2023-05-22T15:47:03.037870+0000","last_became_active":"2023-05-22T15:47:02.657956+0000","last_became_peered":"2023-05-22T15:47:02.657956+0000","last_unstale":"2023-05-22T15:47:03.037870+0000","last_undegraded":"2023-05-22T15:47:03.037870+0000","last_fullsized":"2023-05-22T15:47:03.037870+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:47:01.602136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:47:01.602136+0000","last_clean_scrub_stamp":"2023-05-22T15:47:01.602136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:37:27.882115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.2","version":"0'0","reported_seq":9,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-05-22T15:47:03.047838+0000","last_change":"2023-05-22T15:47:03.045741+0000","last_active":"2023-05-22T15:47:03.047838+0000","last_peered":"2023-05-22T15:47:03.047838+0000","last_clean":"2023-05-22T15:47:03.047838+0000","last_became_active":"2023-05-22T15:47:03.045482+0000","last_became_peered":"2023-05-22T15:47:03.045482+0000","last_unstale":"2023-05-22T15:47:03.047838+0000","last_undegraded":"2023-05-22T15:47:03.047838+0000","last_fullsized":"2023-05-22T15:47:03.047838+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_clean_scrub_stamp":"2023-05-22T15:47:02.606868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:18:51.165098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.3","version":"0'0","reported_seq":10,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-05-22T15:47:03.027107+0000","last_change":"2023-05-22T15:47:02.666096+0000","last_active":"2023-05-22T15:47:03.027107+0000","last_peered":"2023-05-22T15:47:03.027107+0000","last_clean":"2023-05-22T15:47:03.027107+0000","last_became_active":"2023-05-22T15:47:02.665849+0000","last_became_peered":"2023-05-22T15:47:02.665849+0000","last_unstale":"2023-05-22T15:47:03.027107+0000","last_undegraded":"2023-05-22T15:47:03.027107+0000","last_fullsized":"2023-05-22T15:47:03.027107+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:47:01.602136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:47:01.602136+0000","last_clean_scrub_stamp":"2023-05-22T15:47:01.602136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:29:44.189527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.3","version":"0'0","reported_seq":9,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-05-22T15:47:03.096207+0000","last_change":"2023-05-22T15:47:03.093524+0000","last_active":"2023-05-22T15:47:03.096207+0000","last_peered":"2023-05-22T15:47:03.096207+0000","last_clean":"2023-05-22T15:47:03.096207+0000","last_became_active":"2023-05-22T15:47:03.092890+0000","last_became_peered":"2023-05-22T15:47:03.092890+0000","last_unstale":"2023-05-22T15:47:03.096207+0000","last_undegraded":"2023-05-22T15:47:03.096207+0000","last_fullsized":"2023-05-22T15:47:03.096207+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_clean_scrub_stamp":"2023-05-22T15:47:02.606868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:37:28.886847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.2","version":"0'0","reported_seq":10,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-05-22T15:47:03.062458+0000","last_change":"2023-05-22T15:47:02.687873+0000","last_active":"2023-05-22T15:47:03.062458+0000","last_peered":"2023-05-22T15:47:03.062458+0000","last_clean":"2023-05-22T15:47:03.062458+0000","last_became_active":"2023-05-22T15:47:02.687243+0000","last_became_peered":"2023-05-22T15:47:02.687243+0000","last_unstale":"2023-05-22T15:47:03.062458+0000","last_undegraded":"2023-05-22T15:47:03.062458+0000","last_fullsized":"2023-05-22T15:47:03.062458+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:47:01.602136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:47:01.602136+0000","last_clean_scrub_stamp":"2023-05-22T15:47:01.602136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:15:35.219530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.0","version":"0'0","reported_seq":9,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-05-22T15:47:03.085643+0000","last_change":"2023-05-22T15:47:03.073766+0000","last_active":"2023-05-22T15:47:03.085643+0000","last_peered":"2023-05-22T15:47:03.085643+0000","last_clean":"2023-05-22T15:47:03.085643+0000","last_became_active":"2023-05-22T15:47:03.072937+0000","last_became_peered":"2023-05-22T15:47:03.072937+0000","last_unstale":"2023-05-22T15:47:03.085643+0000","last_undegraded":"2023-05-22T15:47:03.085643+0000","last_fullsized":"2023-05-22T15:47:03.085643+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_clean_scrub_stamp":"2023-05-22T15:47:02.606868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:27:02.805634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1","version":"0'0","reported_seq":10,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-05-22T15:47:03.075193+0000","last_change":"2023-05-22T15:47:02.651613+0000","last_active":"2023-05-22T15:47:03.075193+0000","last_peered":"2023-05-22T15:47:03.075193+0000","last_clean":"2023-05-22T15:47:03.075193+0000","last_became_active":"2023-05-22T15:47:02.650779+0000","last_became_peered":"2023-05-22T15:47:02.650779+0000","last_unstale":"2023-05-22T15:47:03.075193+0000","last_undegraded":"2023-05-22T15:47:03.075193+0000","last_fullsized":"2023-05-22T15:47:03.075193+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:47:01.602136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:47:01.602136+0000","last_clean_scrub_stamp":"2023-05-22T15:47:01.602136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:23:24.035094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.1","version":"0'0","reported_seq":9,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-05-22T15:47:03.044679+0000","last_change":"2023-05-22T15:47:03.038439+0000","last_active":"2023-05-22T15:47:03.044679+0000","last_peered":"2023-05-22T15:47:03.044679+0000","last_clean":"2023-05-22T15:47:03.044679+0000","last_became_active":"2023-05-22T15:47:03.037663+0000","last_became_peered":"2023-05-22T15:47:03.037663+0000","last_unstale":"2023-05-22T15:47:03.044679+0000","last_undegraded":"2023-05-22T15:47:03.044679+0000","last_fullsized":"2023-05-22T15:47:03.044679+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_clean_scrub_stamp":"2023-05-22T15:47:02.606868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T03:43:14.105010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.0","version":"0'0","reported_seq":10,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-05-22T15:47:03.030082+0000","last_change":"2023-05-22T15:47:02.678937+0000","last_active":"2023-05-22T15:47:03.030082+0000","last_peered":"2023-05-22T15:47:03.030082+0000","last_clean":"2023-05-22T15:47:03.030082+0000","last_became_active":"2023-05-22T15:47:02.678508+0000","last_became_peered":"2023-05-22T15:47:02.678508+0000","last_unstale":"2023-05-22T15:47:03.030082+0000","last_undegraded":"2023-05-22T15:47:03.030082+0000","last_fullsized":"2023-05-22T15:47:03.030082+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:47:01.602136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:47:01.602136+0000","last_clean_scrub_stamp":"2023-05-22T15:47:01.602136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:39:56.802043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.14","version":"0'0","reported_seq":9,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-05-22T15:47:03.048892+0000","last_change":"2023-05-22T15:47:03.046491+0000","last_active":"2023-05-22T15:47:03.048892+0000","last_peered":"2023-05-22T15:47:03.048892+0000","last_clean":"2023-05-22T15:47:03.048892+0000","last_became_active":"2023-05-22T15:47:03.046190+0000","last_became_peered":"2023-05-22T15:47:03.046190+0000","last_unstale":"2023-05-22T15:47:03.048892+0000","last_undegraded":"2023-05-22T15:47:03.048892+0000","last_fullsized":"2023-05-22T15:47:03.048892+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_clean_scrub_stamp":"2023-05-22T15:47:02.606868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T18:38:26.235383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.15","version":"160'3","reported_seq":13,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-05-22T15:47:03.101109+0000","last_change":"2023-05-22T15:47:02.658544+0000","last_active":"2023-05-22T15:47:03.101109+0000","last_peered":"2023-05-22T15:47:03.101109+0000","last_clean":"2023-05-22T15:47:03.101109+0000","last_became_active":"2023-05-22T15:47:02.657802+0000","last_became_peered":"2023-05-22T15:47:02.657802+0000","last_unstale":"2023-05-22T15:47:03.101109+0000","last_undegraded":"2023-05-22T15:47:03.101109+0000","last_fullsized":"2023-05-22T15:47:03.101109+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:47:01.602136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:47:01.602136+0000","last_clean_scrub_stamp":"2023-05-22T15:47:01.602136+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:18:50.160366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"149'271","reported_seq":845,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-05-22T15:47:02.675418+0000","last_change":"2023-05-22T15:26:39.318204+0000","last_active":"2023-05-22T15:47:02.675418+0000","last_peered":"2023-05-22T15:47:02.675418+0000","last_clean":"2023-05-22T15:47:02.675418+0000","last_became_active":"2023-05-22T15:26:39.315370+0000","last_became_peered":"2023-05-22T15:26:39.315370+0000","last_unstale":"2023-05-22T15:47:02.675418+0000","last_undegraded":"2023-05-22T15:47:02.675418+0000","last_fullsized":"2023-05-22T15:47:02.675418+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:38.020076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:38.020076+0000","last_clean_scrub_stamp":"2023-05-22T15:26:38.020076+0000","objects_scrubbed":0,"log_size":271,"log_dups_size":0,"ondisk_log_size":271,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:06:39.207543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1460,"num_read_kb":2984,"num_write":439,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":154,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-05-22T15:47:02.640185+0000","last_change":"2023-05-22T15:26:44.971951+0000","last_active":"2023-05-22T15:47:02.640185+0000","last_peered":"2023-05-22T15:47:02.640185+0000","last_clean":"2023-05-22T15:47:02.640185+0000","last_became_active":"2023-05-22T15:26:44.970872+0000","last_became_peered":"2023-05-22T15:26:44.970872+0000","last_unstale":"2023-05-22T15:47:02.640185+0000","last_undegraded":"2023-05-22T15:47:02.640185+0000","last_fullsized":"2023-05-22T15:47:02.640185+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:55:34.752949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.17","version":"0'0","reported_seq":9,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-05-22T15:47:03.088843+0000","last_change":"2023-05-22T15:47:03.076930+0000","last_active":"2023-05-22T15:47:03.088843+0000","last_peered":"2023-05-22T15:47:03.088843+0000","last_clean":"2023-05-22T15:47:03.088843+0000","last_became_active":"2023-05-22T15:47:03.074333+0000","last_became_peered":"2023-05-22T15:47:03.074333+0000","last_unstale":"2023-05-22T15:47:03.088843+0000","last_undegraded":"2023-05-22T15:47:03.088843+0000","last_fullsized":"2023-05-22T15:47:03.088843+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_clean_scrub_stamp":"2023-05-22T15:47:02.606868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:44:41.653299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.16","version":"0'0","reported_seq":10,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-05-22T15:47:03.059455+0000","last_change":"2023-05-22T15:47:02.686878+0000","last_active":"2023-05-22T15:47:03.059455+0000","last_peered":"2023-05-22T15:47:03.059455+0000","last_clean":"2023-05-22T15:47:03.059455+0000","last_became_active":"2023-05-22T15:47:02.685784+0000","last_became_peered":"2023-05-22T15:47:02.685784+0000","last_unstale":"2023-05-22T15:47:03.059455+0000","last_undegraded":"2023-05-22T15:47:03.059455+0000","last_fullsized":"2023-05-22T15:47:03.059455+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:47:01.602136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:47:01.602136+0000","last_clean_scrub_stamp":"2023-05-22T15:47:01.602136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:19:51.991474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":146,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-05-22T15:47:02.675162+0000","last_change":"2023-05-22T15:26:44.977198+0000","last_active":"2023-05-22T15:47:02.675162+0000","last_peered":"2023-05-22T15:47:02.675162+0000","last_clean":"2023-05-22T15:47:02.675162+0000","last_became_active":"2023-05-22T15:26:44.976445+0000","last_became_peered":"2023-05-22T15:26:44.976445+0000","last_unstale":"2023-05-22T15:47:02.675162+0000","last_undegraded":"2023-05-22T15:47:02.675162+0000","last_fullsized":"2023-05-22T15:47:02.675162+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:12:41.464054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.16","version":"0'0","reported_seq":9,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-05-22T15:47:03.052564+0000","last_change":"2023-05-22T15:47:03.040278+0000","last_active":"2023-05-22T15:47:03.052564+0000","last_peered":"2023-05-22T15:47:03.052564+0000","last_clean":"2023-05-22T15:47:03.052564+0000","last_became_active":"2023-05-22T15:47:03.039500+0000","last_became_peered":"2023-05-22T15:47:03.039500+0000","last_unstale":"2023-05-22T15:47:03.052564+0000","last_undegraded":"2023-05-22T15:47:03.052564+0000","last_fullsized":"2023-05-22T15:47:03.052564+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_clean_scrub_stamp":"2023-05-22T15:47:02.606868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:22:18.748162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.17","version":"0'0","reported_seq":10,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-05-22T15:47:03.036866+0000","last_change":"2023-05-22T15:47:02.653661+0000","last_active":"2023-05-22T15:47:03.036866+0000","last_peered":"2023-05-22T15:47:03.036866+0000","last_clean":"2023-05-22T15:47:03.036866+0000","last_became_active":"2023-05-22T15:47:02.653389+0000","last_became_peered":"2023-05-22T15:47:02.653389+0000","last_unstale":"2023-05-22T15:47:03.036866+0000","last_undegraded":"2023-05-22T15:47:03.036866+0000","last_fullsized":"2023-05-22T15:47:03.036866+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:47:01.602136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:47:01.602136+0000","last_clean_scrub_stamp":"2023-05-22T15:47:01.602136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:28:42.216703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":146,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-05-22T15:47:02.648441+0000","last_change":"2023-05-22T15:26:44.969469+0000","last_active":"2023-05-22T15:47:02.648441+0000","last_peered":"2023-05-22T15:47:02.648441+0000","last_clean":"2023-05-22T15:47:02.648441+0000","last_became_active":"2023-05-22T15:26:44.969083+0000","last_became_peered":"2023-05-22T15:26:44.969083+0000","last_unstale":"2023-05-22T15:47:02.648441+0000","last_undegraded":"2023-05-22T15:47:02.648441+0000","last_fullsized":"2023-05-22T15:47:02.648441+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:29:23.670067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.15","version":"0'0","reported_seq":9,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-05-22T15:47:03.030240+0000","last_change":"2023-05-22T15:47:03.026296+0000","last_active":"2023-05-22T15:47:03.030240+0000","last_peered":"2023-05-22T15:47:03.030240+0000","last_clean":"2023-05-22T15:47:03.030240+0000","last_became_active":"2023-05-22T15:47:03.025995+0000","last_became_peered":"2023-05-22T15:47:03.025995+0000","last_unstale":"2023-05-22T15:47:03.030240+0000","last_undegraded":"2023-05-22T15:47:03.030240+0000","last_fullsized":"2023-05-22T15:47:03.030240+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_clean_scrub_stamp":"2023-05-22T15:47:02.606868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T03:26:44.703194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.14","version":"160'6","reported_seq":16,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-05-22T15:47:03.060992+0000","last_change":"2023-05-22T15:47:02.652400+0000","last_active":"2023-05-22T15:47:03.060992+0000","last_peered":"2023-05-22T15:47:03.060992+0000","last_clean":"2023-05-22T15:47:03.060992+0000","last_became_active":"2023-05-22T15:47:02.652113+0000","last_became_peered":"2023-05-22T15:47:02.652113+0000","last_unstale":"2023-05-22T15:47:03.060992+0000","last_undegraded":"2023-05-22T15:47:03.060992+0000","last_fullsized":"2023-05-22T15:47:03.060992+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:47:01.602136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:47:01.602136+0000","last_clean_scrub_stamp":"2023-05-22T15:47:01.602136+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:18:47.057150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1548,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":154,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-05-22T15:47:02.636750+0000","last_change":"2023-05-22T15:26:44.976632+0000","last_active":"2023-05-22T15:47:02.636750+0000","last_peered":"2023-05-22T15:47:02.636750+0000","last_clean":"2023-05-22T15:47:02.636750+0000","last_became_active":"2023-05-22T15:26:44.975546+0000","last_became_peered":"2023-05-22T15:26:44.975546+0000","last_unstale":"2023-05-22T15:47:02.636750+0000","last_undegraded":"2023-05-22T15:47:02.636750+0000","last_fullsized":"2023-05-22T15:47:02.636750+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:42:57.973030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.13","version":"0'0","reported_seq":9,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-05-22T15:47:03.043174+0000","last_change":"2023-05-22T15:47:03.041807+0000","last_active":"2023-05-22T15:47:03.043174+0000","last_peered":"2023-05-22T15:47:03.043174+0000","last_clean":"2023-05-22T15:47:03.043174+0000","last_became_active":"2023-05-22T15:47:03.041516+0000","last_became_peered":"2023-05-22T15:47:03.041516+0000","last_unstale":"2023-05-22T15:47:03.043174+0000","last_undegraded":"2023-05-22T15:47:03.043174+0000","last_fullsized":"2023-05-22T15:47:03.043174+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_clean_scrub_stamp":"2023-05-22T15:47:02.606868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T15:51:01.546872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.12","version":"0'0","reported_seq":10,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-05-22T15:47:03.075255+0000","last_change":"2023-05-22T15:47:02.660522+0000","last_active":"2023-05-22T15:47:03.075255+0000","last_peered":"2023-05-22T15:47:03.075255+0000","last_clean":"2023-05-22T15:47:03.075255+0000","last_became_active":"2023-05-22T15:47:02.659535+0000","last_became_peered":"2023-05-22T15:47:02.659535+0000","last_unstale":"2023-05-22T15:47:03.075255+0000","last_undegraded":"2023-05-22T15:47:03.075255+0000","last_fullsized":"2023-05-22T15:47:03.075255+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:47:01.602136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:47:01.602136+0000","last_clean_scrub_stamp":"2023-05-22T15:47:01.602136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:55:52.691222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":135,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-05-22T15:47:02.645371+0000","last_change":"2023-05-22T15:26:45.091475+0000","last_active":"2023-05-22T15:47:02.645371+0000","last_peered":"2023-05-22T15:47:02.645371+0000","last_clean":"2023-05-22T15:47:02.645371+0000","last_became_active":"2023-05-22T15:26:45.091143+0000","last_became_peered":"2023-05-22T15:26:45.091143+0000","last_unstale":"2023-05-22T15:47:02.645371+0000","last_undegraded":"2023-05-22T15:47:02.645371+0000","last_fullsized":"2023-05-22T15:47:02.645371+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:12:41.464054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.12","version":"0'0","reported_seq":9,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-05-22T15:47:03.098274+0000","last_change":"2023-05-22T15:47:03.087767+0000","last_active":"2023-05-22T15:47:03.098274+0000","last_peered":"2023-05-22T15:47:03.098274+0000","last_clean":"2023-05-22T15:47:03.098274+0000","last_became_active":"2023-05-22T15:47:03.086853+0000","last_became_peered":"2023-05-22T15:47:03.086853+0000","last_unstale":"2023-05-22T15:47:03.098274+0000","last_undegraded":"2023-05-22T15:47:03.098274+0000","last_fullsized":"2023-05-22T15:47:03.098274+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_clean_scrub_stamp":"2023-05-22T15:47:02.606868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:46:41.390685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.13","version":"160'2","reported_seq":12,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-05-22T15:47:03.098510+0000","last_change":"2023-05-22T15:47:02.651919+0000","last_active":"2023-05-22T15:47:03.098510+0000","last_peered":"2023-05-22T15:47:03.098510+0000","last_clean":"2023-05-22T15:47:03.098510+0000","last_became_active":"2023-05-22T15:47:02.651121+0000","last_became_peered":"2023-05-22T15:47:02.651121+0000","last_unstale":"2023-05-22T15:47:03.098510+0000","last_undegraded":"2023-05-22T15:47:03.098510+0000","last_fullsized":"2023-05-22T15:47:03.098510+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:47:01.602136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:47:01.602136+0000","last_clean_scrub_stamp":"2023-05-22T15:47:01.602136+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:10:34.105690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":145,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-05-22T15:47:02.675258+0000","last_change":"2023-05-22T15:26:45.092367+0000","last_active":"2023-05-22T15:47:02.675258+0000","last_peered":"2023-05-22T15:47:02.675258+0000","last_clean":"2023-05-22T15:47:02.675258+0000","last_became_active":"2023-05-22T15:26:45.091951+0000","last_became_peered":"2023-05-22T15:26:45.091951+0000","last_unstale":"2023-05-22T15:47:02.675258+0000","last_undegraded":"2023-05-22T15:47:02.675258+0000","last_fullsized":"2023-05-22T15:47:02.675258+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:42:57.973030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.11","version":"0'0","reported_seq":9,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-05-22T15:47:03.098177+0000","last_change":"2023-05-22T15:47:03.086458+0000","last_active":"2023-05-22T15:47:03.098177+0000","last_peered":"2023-05-22T15:47:03.098177+0000","last_clean":"2023-05-22T15:47:03.098177+0000","last_became_active":"2023-05-22T15:47:03.080919+0000","last_became_peered":"2023-05-22T15:47:03.080919+0000","last_unstale":"2023-05-22T15:47:03.098177+0000","last_undegraded":"2023-05-22T15:47:03.098177+0000","last_fullsized":"2023-05-22T15:47:03.098177+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_clean_scrub_stamp":"2023-05-22T15:47:02.606868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:17:27.515854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.10","version":"160'8","reported_seq":18,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-05-22T15:47:03.094344+0000","last_change":"2023-05-22T15:47:02.650899+0000","last_active":"2023-05-22T15:47:03.094344+0000","last_peered":"2023-05-22T15:47:03.094344+0000","last_clean":"2023-05-22T15:47:03.094344+0000","last_became_active":"2023-05-22T15:47:02.648924+0000","last_became_peered":"2023-05-22T15:47:02.648924+0000","last_unstale":"2023-05-22T15:47:03.094344+0000","last_undegraded":"2023-05-22T15:47:03.094344+0000","last_fullsized":"2023-05-22T15:47:03.094344+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:47:01.602136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:47:01.602136+0000","last_clean_scrub_stamp":"2023-05-22T15:47:01.602136+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T18:22:29.880531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":135,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-05-22T15:47:02.645219+0000","last_change":"2023-05-22T15:26:44.978858+0000","last_active":"2023-05-22T15:47:02.645219+0000","last_peered":"2023-05-22T15:47:02.645219+0000","last_clean":"2023-05-22T15:47:02.645219+0000","last_became_active":"2023-05-22T15:26:44.978443+0000","last_became_peered":"2023-05-22T15:26:44.978443+0000","last_unstale":"2023-05-22T15:47:02.645219+0000","last_undegraded":"2023-05-22T15:47:02.645219+0000","last_fullsized":"2023-05-22T15:47:02.645219+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:46:29.290243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.10","version":"0'0","reported_seq":9,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-05-22T15:47:03.083577+0000","last_change":"2023-05-22T15:47:03.078634+0000","last_active":"2023-05-22T15:47:03.083577+0000","last_peered":"2023-05-22T15:47:03.083577+0000","last_clean":"2023-05-22T15:47:03.083577+0000","last_became_active":"2023-05-22T15:47:03.078040+0000","last_became_peered":"2023-05-22T15:47:03.078040+0000","last_unstale":"2023-05-22T15:47:03.083577+0000","last_undegraded":"2023-05-22T15:47:03.083577+0000","last_fullsized":"2023-05-22T15:47:03.083577+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:47:02.606868+0000","last_clean_scrub_stamp":"2023-05-22T15:47:02.606868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:40:02.459811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.11","version":"0'0","reported_seq":10,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-05-22T15:47:03.035082+0000","last_change":"2023-05-22T15:47:02.651915+0000","last_active":"2023-05-22T15:47:03.035082+0000","last_peered":"2023-05-22T15:47:03.035082+0000","last_clean":"2023-05-22T15:47:03.035082+0000","last_became_active":"2023-05-22T15:47:02.651560+0000","last_became_peered":"2023-05-22T15:47:02.651560+0000","last_unstale":"2023-05-22T15:47:03.035082+0000","last_undegraded":"2023-05-22T15:47:03.035082+0000","last_fullsized":"2023-05-22T15:47:03.035082+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:47:01.602136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:47:01.602136+0000","last_clean_scrub_stamp":"2023-05-22T15:47:01.602136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:31:47.844837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":136,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-05-22T15:47:02.649430+0000","last_change":"2023-05-22T15:26:44.982421+0000","last_active":"2023-05-22T15:47:02.649430+0000","last_peered":"2023-05-22T15:47:02.649430+0000","last_clean":"2023-05-22T15:47:02.649430+0000","last_became_active":"2023-05-22T15:26:44.981586+0000","last_became_peered":"2023-05-22T15:26:44.981586+0000","last_unstale":"2023-05-22T15:47:02.649430+0000","last_undegraded":"2023-05-22T15:47:02.649430+0000","last_fullsized":"2023-05-22T15:47:02.649430+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:14:11.156544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":21,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":20,"num_pg":32,"stat_sum":{"num_bytes":2330,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1460,"num_read_kb":2984,"num_write":439,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":271,"ondisk_log_size":271,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575091,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":20528,"kb_used_data":2536,"kb_used_omap":0,"kb_used_meta":17984,"kb_avail":94351312,"statfs":{"total":96636764160,"available":96615743488,"internally_reserved":0,"allocated":2596864,"data_stored":1647540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":18415616},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 15:46:11 2023","interfaces":[{"interface":"back","average":{"1min":0.86699999999999999,"5min":0.89200000000000002,"15min":0.876},"min":{"1min":0.53600000000000003,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":1.294,"5min":2.637,"15min":2.637},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.88,"5min":0.88700000000000001,"15min":0.89200000000000002},"min":{"1min":0.60799999999999998,"5min":0.502,"15min":0.35799999999999998},"max":{"1min":1.1519999999999999,"5min":3.0449999999999999,"15min":3.0449999999999999},"last":1.028}]},{"osd":1,"last update":"Mon May 22 15:46:11 2023","interfaces":[{"interface":"back","average":{"1min":0.96799999999999997,"5min":0.95399999999999996,"15min":0.97899999999999998},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.1899999999999999,"5min":1.385,"15min":1.389},"last":0.83999999999999997},{"interface":"front","average":{"1min":0.96799999999999997,"5min":1.006,"15min":1.022},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.54900000000000004},"max":{"1min":1.226,"5min":2.7269999999999999,"15min":2.7269999999999999},"last":0.98899999999999999}]},{"osd":2,"last update":"Mon May 22 15:46:11 2023","interfaces":[{"interface":"back","average":{"1min":0.86199999999999999,"5min":0.85099999999999998,"15min":0.871},"min":{"1min":0.55600000000000005,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.109,"5min":1.252,"15min":1.252},"last":0.95099999999999996},{"interface":"front","average":{"1min":0.88100000000000001,"5min":0.93999999999999995,"15min":0.95699999999999996},"min":{"1min":0.504,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.143,"5min":3.141,"15min":3.141},"last":0.72299999999999998}]},{"osd":3,"last update":"Mon May 22 15:46:11 2023","interfaces":[{"interface":"back","average":{"1min":0.98199999999999998,"5min":0.98899999999999999,"15min":0.96299999999999997},"min":{"1min":0.67600000000000005,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":1.3839999999999999,"5min":2.8149999999999999,"15min":2.8149999999999999},"last":0.79000000000000004},{"interface":"front","average":{"1min":0.94399999999999995,"5min":0.92100000000000004,"15min":0.96199999999999997},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.2709999999999999,"5min":1.389,"15min":1.4710000000000001},"last":0.747}]},{"osd":4,"last update":"Mon May 22 15:46:11 2023","interfaces":[{"interface":"back","average":{"1min":0.87,"5min":0.94399999999999995,"15min":0.90400000000000003},"min":{"1min":0.60299999999999998,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.5089999999999999,"5min":3.472,"15min":3.472},"last":0.62},{"interface":"front","average":{"1min":0.997,"5min":1.1850000000000001,"15min":1.05},"min":{"1min":0.59699999999999998,"5min":0.56000000000000005,"15min":0.44600000000000001},"max":{"1min":1.2949999999999999,"5min":15.627000000000001,"15min":15.627000000000001},"last":0.91200000000000003}]},{"osd":5,"last update":"Mon May 22 15:46:11 2023","interfaces":[{"interface":"back","average":{"1min":1.012,"5min":1.071,"15min":1.02},"min":{"1min":0.60899999999999999,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":1.4750000000000001,"5min":4.5599999999999996,"15min":4.5599999999999996},"last":1.071},{"interface":"front","average":{"1min":1.115,"5min":1.161,"15min":1.139},"min":{"1min":0.82999999999999996,"5min":0.63800000000000001,"15min":0.58799999999999997},"max":{"1min":1.302,"5min":4.3799999999999999,"15min":4.3799999999999999},"last":1.1839999999999999}]},{"osd":6,"last update":"Mon May 22 15:46:11 2023","interfaces":[{"interface":"back","average":{"1min":1.073,"5min":1.1399999999999999,"15min":1.0840000000000001},"min":{"1min":0.80300000000000005,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.415,"5min":6.5069999999999997,"15min":6.5069999999999997},"last":1.1299999999999999},{"interface":"front","average":{"1min":1.1200000000000001,"5min":1.1659999999999999,"15min":1.1419999999999999},"min":{"1min":0.70299999999999996,"5min":0.62,"15min":0.62},"max":{"1min":1.458,"5min":3.3500000000000001,"15min":3.3500000000000001},"last":1.105}]}]},{"osd":6,"up_from":13,"seq":55834575092,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37296,"kb_used_data":1960,"kb_used_omap":0,"kb_used_meta":35328,"kb_avail":94334544,"statfs":{"total":96636764160,"available":96598573056,"internally_reserved":0,"allocated":2007040,"data_stored":1056682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":36175872},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 15:46:20 2023","interfaces":[{"interface":"back","average":{"1min":0.90000000000000002,"5min":0.86699999999999999,"15min":0.89900000000000002},"min":{"1min":0.499,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":1.4039999999999999,"5min":1.486,"15min":1.486},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.89200000000000002,"5min":0.94599999999999995,"15min":0.91100000000000003},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.36299999999999999},"max":{"1min":1.2729999999999999,"5min":1.6240000000000001,"15min":1.6240000000000001},"last":0.82799999999999996}]},{"osd":1,"last update":"Mon May 22 15:46:20 2023","interfaces":[{"interface":"back","average":{"1min":0.93200000000000005,"5min":0.89100000000000001,"15min":0.91200000000000003},"min":{"1min":0.59799999999999998,"5min":0.504,"15min":0.48299999999999998},"max":{"1min":1.8799999999999999,"5min":1.8799999999999999,"15min":1.8799999999999999},"last":0.72399999999999998},{"interface":"front","average":{"1min":0.95899999999999996,"5min":0.92400000000000004,"15min":0.93999999999999995},"min":{"1min":0.70299999999999996,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":1.1950000000000001,"5min":1.3320000000000001,"15min":1.554},"last":1.056}]},{"osd":2,"last update":"Mon May 22 15:46:20 2023","interfaces":[{"interface":"back","average":{"1min":1.05,"5min":0.97399999999999998,"15min":0.98999999999999999},"min":{"1min":0.55400000000000005,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":3.2429999999999999,"5min":3.2429999999999999,"15min":3.2429999999999999},"last":0.76400000000000001},{"interface":"front","average":{"1min":1.032,"5min":1.0269999999999999,"15min":1.0429999999999999},"min":{"1min":0.55300000000000005,"5min":0.5,"15min":0.442},"max":{"1min":3.3969999999999998,"5min":3.3969999999999998,"15min":3.3969999999999998},"last":0.64000000000000001}]},{"osd":3,"last update":"Mon May 22 15:46:20 2023","interfaces":[{"interface":"back","average":{"1min":0.88900000000000001,"5min":0.94599999999999995,"15min":0.97899999999999998},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":1.2470000000000001,"5min":1.5780000000000001,"15min":1.679},"last":1.1379999999999999},{"interface":"front","average":{"1min":0.93500000000000005,"5min":0.95099999999999996,"15min":0.96299999999999997},"min":{"1min":0.69099999999999995,"5min":0.53200000000000003,"15min":0.50800000000000001},"max":{"1min":1.2430000000000001,"5min":1.383,"15min":1.4570000000000001},"last":0.60699999999999998}]},{"osd":4,"last update":"Mon May 22 15:46:20 2023","interfaces":[{"interface":"back","average":{"1min":0.91300000000000003,"5min":0.93200000000000005,"15min":0.92500000000000004},"min":{"1min":0.61699999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":1.3169999999999999,"5min":1.581,"15min":1.6859999999999999},"last":0.874},{"interface":"front","average":{"1min":0.93200000000000005,"5min":1.0209999999999999,"15min":1.0169999999999999},"min":{"1min":0.65500000000000003,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":1.242,"5min":1.452,"15min":1.8300000000000001},"last":1.194}]},{"osd":5,"last update":"Mon May 22 15:46:20 2023","interfaces":[{"interface":"back","average":{"1min":0.98799999999999999,"5min":1.0169999999999999,"15min":0.98899999999999999},"min":{"1min":0.67600000000000005,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":1.3200000000000001,"5min":1.4530000000000001,"15min":1.6499999999999999},"last":0.92200000000000004},{"interface":"front","average":{"1min":1.1459999999999999,"5min":1.141,"15min":1.1120000000000001},"min":{"1min":0.81999999999999995,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.9990000000000001,"5min":1.9990000000000001,"15min":1.9990000000000001},"last":0.97399999999999998}]},{"osd":7,"last update":"Mon May 22 15:46:20 2023","interfaces":[{"interface":"back","average":{"1min":1.0860000000000001,"5min":1.131,"15min":1.125},"min":{"1min":0.85499999999999998,"5min":0.37,"15min":0.37},"max":{"1min":1.5529999999999999,"5min":1.853,"15min":1.9179999999999999},"last":1.2470000000000001},{"interface":"front","average":{"1min":1.131,"5min":1.133,"15min":1.135},"min":{"1min":0.80300000000000005,"5min":0.442,"15min":0.442},"max":{"1min":1.579,"5min":1.579,"15min":1.8520000000000001},"last":1.022}]}]},{"osd":5,"up_from":13,"seq":55834575092,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30696,"kb_used_data":1952,"kb_used_omap":0,"kb_used_meta":28736,"kb_avail":94341144,"statfs":{"total":96636764160,"available":96605331456,"internally_reserved":0,"allocated":1998848,"data_stored":1056558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":29425664},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 15:46:10 2023","interfaces":[{"interface":"back","average":{"1min":0.90900000000000003,"5min":0.85599999999999998,"15min":0.88300000000000001},"min":{"1min":0.61399999999999999,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.2889999999999999,"5min":1.3140000000000001,"15min":1.3440000000000001},"last":1.2789999999999999},{"interface":"front","average":{"1min":0.91800000000000004,"5min":0.88600000000000001,"15min":0.88500000000000001},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":1.2709999999999999,"5min":1.3899999999999999,"15min":2.1699999999999999},"last":0.66800000000000004}]},{"osd":1,"last update":"Mon May 22 15:46:10 2023","interfaces":[{"interface":"back","average":{"1min":0.94999999999999996,"5min":0.96099999999999997,"15min":0.995},"min":{"1min":0.64800000000000002,"5min":0.52400000000000002,"15min":0.51000000000000001},"max":{"1min":1.3109999999999999,"5min":1.337,"15min":1.9370000000000001},"last":1.0229999999999999},{"interface":"front","average":{"1min":1.073,"5min":1.008,"15min":1.026},"min":{"1min":0.501,"5min":0.501,"15min":0.501},"max":{"1min":1.3089999999999999,"5min":1.6970000000000001,"15min":1.6970000000000001},"last":1.23}]},{"osd":2,"last update":"Mon May 22 15:46:10 2023","interfaces":[{"interface":"back","average":{"1min":0.89100000000000001,"5min":0.91600000000000004,"15min":0.93400000000000005},"min":{"1min":0.505,"5min":0.505,"15min":0.504},"max":{"1min":1.343,"5min":1.5549999999999999,"15min":1.9830000000000001},"last":0.85599999999999998},{"interface":"front","average":{"1min":0.91800000000000004,"5min":0.94999999999999996,"15min":0.95199999999999996},"min":{"1min":0.56100000000000005,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.202,"5min":1.321,"15min":2.1120000000000001},"last":0.91400000000000003}]},{"osd":3,"last update":"Mon May 22 15:46:10 2023","interfaces":[{"interface":"back","average":{"1min":0.92900000000000005,"5min":0.92000000000000004,"15min":0.94899999999999995},"min":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.48199999999999998},"max":{"1min":1.206,"5min":1.2809999999999999,"15min":1.4419999999999999},"last":1.0589999999999999},{"interface":"front","average":{"1min":1.028,"5min":1.008,"15min":1.036},"min":{"1min":0.626,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":1.2490000000000001,"5min":1.371,"15min":2.52},"last":0.97799999999999998}]},{"osd":4,"last update":"Mon May 22 15:46:10 2023","interfaces":[{"interface":"back","average":{"1min":0.96999999999999997,"5min":0.95699999999999996,"15min":0.95199999999999996},"min":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":1.405,"5min":1.405,"15min":2.226},"last":0.79400000000000004},{"interface":"front","average":{"1min":1.0629999999999999,"5min":1.024,"15min":1.0309999999999999},"min":{"1min":0.76000000000000001,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.3740000000000001,"5min":1.863,"15min":1.863},"last":1.177}]},{"osd":6,"last update":"Mon May 22 15:46:10 2023","interfaces":[{"interface":"back","average":{"1min":1.008,"5min":1.024,"15min":1.0349999999999999},"min":{"1min":0.51900000000000002,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.397,"5min":1.4330000000000001,"15min":2.831},"last":1.135},{"interface":"front","average":{"1min":1.0369999999999999,"5min":1.0589999999999999,"15min":1.071},"min":{"1min":0.67000000000000004,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":1.4239999999999999,"5min":1.4239999999999999,"15min":2.3690000000000002},"last":1.103}]},{"osd":7,"last update":"Mon May 22 15:46:10 2023","interfaces":[{"interface":"back","average":{"1min":1.081,"5min":1.0900000000000001,"15min":1.1120000000000001},"min":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.42499999999999999},"max":{"1min":1.4359999999999999,"5min":1.47,"15min":1.849},"last":1.3220000000000001},{"interface":"front","average":{"1min":1.1599999999999999,"5min":1.149,"15min":1.1659999999999999},"min":{"1min":0.86899999999999999,"5min":0.64000000000000001,"15min":0.64000000000000001},"max":{"1min":1.4339999999999999,"5min":1.575,"15min":2.7429999999999999},"last":1.2170000000000001}]}]},{"osd":4,"up_from":13,"seq":55834575090,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32548,"kb_used_data":1948,"kb_used_omap":0,"kb_used_meta":30592,"kb_avail":94339292,"statfs":{"total":96636764160,"available":96603435008,"internally_reserved":0,"allocated":1994752,"data_stored":1056536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31326208},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 15:46:19 2023","interfaces":[{"interface":"back","average":{"1min":0.93200000000000005,"5min":0.86799999999999999,"15min":0.90500000000000003},"min":{"1min":0.63400000000000001,"5min":0.48999999999999999,"15min":0.34100000000000003},"max":{"1min":1.2629999999999999,"5min":1.4119999999999999,"15min":1.9910000000000001},"last":1.056},{"interface":"front","average":{"1min":0.91300000000000003,"5min":0.90000000000000002,"15min":0.92200000000000004},"min":{"1min":0.51100000000000001,"5min":0.499,"15min":0.373},"max":{"1min":1.284,"5min":1.369,"15min":1.3779999999999999},"last":0.67800000000000005}]},{"osd":1,"last update":"Mon May 22 15:46:19 2023","interfaces":[{"interface":"back","average":{"1min":0.94799999999999995,"5min":0.91800000000000004,"15min":0.96399999999999997},"min":{"1min":0.60499999999999998,"5min":0.436,"15min":0.436},"max":{"1min":1.1970000000000001,"5min":1.3899999999999999,"15min":1.3899999999999999},"last":0.83599999999999997},{"interface":"front","average":{"1min":0.91700000000000004,"5min":0.84499999999999997,"15min":0.90100000000000002},"min":{"1min":0.56699999999999995,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.157,"5min":1.2889999999999999,"15min":1.3640000000000001},"last":1.1890000000000001}]},{"osd":2,"last update":"Mon May 22 15:46:19 2023","interfaces":[{"interface":"back","average":{"1min":0.90100000000000002,"5min":0.91600000000000004,"15min":0.94199999999999995},"min":{"1min":0.56299999999999994,"5min":0.50800000000000001,"15min":0.495},"max":{"1min":1.244,"5min":1.96,"15min":2.1019999999999999},"last":0.78600000000000003},{"interface":"front","average":{"1min":0.97799999999999998,"5min":0.98399999999999999,"15min":0.99199999999999999},"min":{"1min":0.58099999999999996,"5min":0.48799999999999999,"15min":0.48399999999999999},"max":{"1min":1.329,"5min":1.4399999999999999,"15min":1.4399999999999999},"last":0.99399999999999999}]},{"osd":3,"last update":"Mon May 22 15:46:19 2023","interfaces":[{"interface":"back","average":{"1min":0.91100000000000003,"5min":0.90200000000000002,"15min":0.93200000000000005},"min":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":1.224,"5min":1.28,"15min":1.3169999999999999},"last":0.74299999999999999},{"interface":"front","average":{"1min":1.0469999999999999,"5min":1.014,"15min":1.0209999999999999},"min":{"1min":0.72199999999999998,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":1.3420000000000001,"5min":1.4179999999999999,"15min":2.1720000000000002},"last":1.159}]},{"osd":5,"last update":"Mon May 22 15:46:19 2023","interfaces":[{"interface":"back","average":{"1min":1.0209999999999999,"5min":0.97199999999999998,"15min":0.97699999999999998},"min":{"1min":0.55000000000000004,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.357,"5min":1.373,"15min":1.472},"last":1.1200000000000001},{"interface":"front","average":{"1min":1.0389999999999999,"5min":0.98999999999999999,"15min":0.97499999999999998},"min":{"1min":0.59999999999999998,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.262,"5min":1.365,"15min":1.4139999999999999},"last":0.94699999999999995}]},{"osd":6,"last update":"Mon May 22 15:46:19 2023","interfaces":[{"interface":"back","average":{"1min":1.0649999999999999,"5min":0.99299999999999999,"15min":0.98499999999999999},"min":{"1min":0.72099999999999997,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":1.456,"5min":1.456,"15min":2.2349999999999999},"last":1.274},{"interface":"front","average":{"1min":1.131,"5min":1.103,"15min":1.1100000000000001},"min":{"1min":0.66500000000000004,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":1.425,"5min":1.486,"15min":1.8220000000000001},"last":1.2490000000000001}]},{"osd":7,"last update":"Mon May 22 15:46:19 2023","interfaces":[{"interface":"back","average":{"1min":1.137,"5min":1.0629999999999999,"15min":1.052},"min":{"1min":0.85699999999999998,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":1.389,"5min":1.46,"15min":1.5800000000000001},"last":1.3300000000000001},{"interface":"front","average":{"1min":1.1930000000000001,"5min":1.167,"15min":1.143},"min":{"1min":0.78900000000000003,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":1.5620000000000001,"5min":1.6559999999999999,"15min":2.306},"last":1.361}]}]},{"osd":3,"up_from":13,"seq":55834575090,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37720,"kb_used_data":1936,"kb_used_omap":0,"kb_used_meta":35776,"kb_avail":94334120,"statfs":{"total":96636764160,"available":96598138880,"internally_reserved":0,"allocated":1982464,"data_stored":1049060,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":36634624},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 15:46:03 2023","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.61699999999999999,"15min":0.63600000000000001},"min":{"1min":0.50700000000000001,"5min":0.252,"15min":0.19},"max":{"1min":1.169,"5min":1.169,"15min":1.194},"last":0.73499999999999999},{"interface":"front","average":{"1min":0.746,"5min":0.68500000000000005,"15min":0.70899999999999996},"min":{"1min":0.48899999999999999,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":1.085,"5min":1.296,"15min":1.296},"last":0.61699999999999999}]},{"osd":1,"last update":"Mon May 22 15:46:03 2023","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.64200000000000002,"15min":0.66200000000000003},"min":{"1min":0.52100000000000002,"5min":0.314,"15min":0.29499999999999998},"max":{"1min":0.90600000000000003,"5min":1.2609999999999999,"15min":1.2609999999999999},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.72899999999999998,"15min":0.76800000000000002},"min":{"1min":0.47499999999999998,"5min":0.34399999999999997,"15min":0.313},"max":{"1min":1.081,"5min":1.1359999999999999,"15min":1.222},"last":0.82499999999999996}]},{"osd":2,"last update":"Mon May 22 15:46:03 2023","interfaces":[{"interface":"back","average":{"1min":0.84199999999999997,"5min":0.78700000000000003,"15min":0.81699999999999995},"min":{"1min":0.45600000000000002,"5min":0.40699999999999997,"15min":0.36599999999999999},"max":{"1min":1.0529999999999999,"5min":1.506,"15min":1.506},"last":0.89600000000000002},{"interface":"front","average":{"1min":0.81899999999999995,"5min":0.75600000000000001,"15min":0.80700000000000005},"min":{"1min":0.65300000000000002,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":1.032,"5min":1.147,"15min":1.359},"last":0.86899999999999999}]},{"osd":4,"last update":"Mon May 22 15:46:03 2023","interfaces":[{"interface":"back","average":{"1min":0.97499999999999998,"5min":0.92500000000000004,"15min":0.95599999999999996},"min":{"1min":0.71999999999999997,"5min":0.54500000000000004,"15min":0.48999999999999999},"max":{"1min":1.238,"5min":1.413,"15min":1.4470000000000001},"last":0.98199999999999998},{"interface":"front","average":{"1min":1.0529999999999999,"5min":0.97499999999999998,"15min":1.0029999999999999},"min":{"1min":0.83299999999999996,"5min":0.54400000000000004,"15min":0.49199999999999999},"max":{"1min":1.407,"5min":1.407,"15min":1.4470000000000001},"last":0.94199999999999995}]},{"osd":5,"last update":"Mon May 22 15:46:03 2023","interfaces":[{"interface":"back","average":{"1min":1.0509999999999999,"5min":0.98399999999999999,"15min":1.016},"min":{"1min":0.80900000000000005,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.3149999999999999,"5min":1.5089999999999999,"15min":1.5089999999999999},"last":1.0029999999999999},{"interface":"front","average":{"1min":1.0349999999999999,"5min":1.006,"15min":1.038},"min":{"1min":0.90200000000000002,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.2130000000000001,"5min":1.4299999999999999,"15min":1.4359999999999999},"last":1.4039999999999999}]},{"osd":6,"last update":"Mon May 22 15:46:03 2023","interfaces":[{"interface":"back","average":{"1min":1.0580000000000001,"5min":1.0129999999999999,"15min":1.042},"min":{"1min":0.873,"5min":0.47699999999999998,"15min":0.46999999999999997},"max":{"1min":1.2989999999999999,"5min":1.4570000000000001,"15min":1.49},"last":1.0640000000000001},{"interface":"front","average":{"1min":1.083,"5min":1.0269999999999999,"15min":1.0700000000000001},"min":{"1min":0.79800000000000004,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":1.2949999999999999,"5min":1.4330000000000001,"15min":1.5009999999999999},"last":1.3600000000000001}]},{"osd":7,"last update":"Mon May 22 15:46:03 2023","interfaces":[{"interface":"back","average":{"1min":1.113,"5min":1.0780000000000001,"15min":1.1040000000000001},"min":{"1min":0.83599999999999997,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.3779999999999999,"5min":1.6719999999999999,"15min":1.6719999999999999},"last":1.26},{"interface":"front","average":{"1min":1.117,"5min":1.071,"15min":1.085},"min":{"1min":0.90200000000000002,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":1.361,"5min":1.8700000000000001,"15min":1.8700000000000001},"last":1.1160000000000001}]}]},{"osd":2,"up_from":13,"seq":55834575091,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":23540,"kb_used_data":1964,"kb_used_omap":0,"kb_used_meta":21568,"kb_avail":94348300,"statfs":{"total":96636764160,"available":96612659200,"internally_reserved":0,"allocated":2011136,"data_stored":1058154,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":22085632},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 15:46:03 2023","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.64400000000000002,"15min":0.65600000000000003},"min":{"1min":0.28699999999999998,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":1.097,"5min":1.1539999999999999,"15min":1.8240000000000001},"last":0.495},{"interface":"front","average":{"1min":0.69899999999999995,"5min":0.70499999999999996,"15min":0.746},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":1.079,"5min":1.079,"15min":1.8160000000000001},"last":0.68300000000000005}]},{"osd":1,"last update":"Mon May 22 15:46:03 2023","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.72999999999999998,"15min":0.74399999999999999},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.27000000000000002},"max":{"1min":1.196,"5min":1.196,"15min":1.2},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.754,"5min":0.75900000000000001,"15min":0.80800000000000005},"min":{"1min":0.32400000000000001,"5min":0.317,"15min":0.317},"max":{"1min":1.0309999999999999,"5min":1.246,"15min":2.5089999999999999},"last":0.752}]},{"osd":3,"last update":"Mon May 22 15:46:03 2023","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":0.69399999999999995,"15min":0.72599999999999998},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.96199999999999997,"5min":1.077,"15min":2.7629999999999999},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.82699999999999996,"5min":0.84599999999999997,"15min":0.88300000000000001},"min":{"1min":0.39600000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":1.1699999999999999,"5min":1.256,"15min":2.6840000000000002},"last":0.437}]},{"osd":4,"last update":"Mon May 22 15:46:03 2023","interfaces":[{"interface":"back","average":{"1min":0.97499999999999998,"5min":0.98499999999999999,"15min":1.018},"min":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":1.274,"5min":1.536,"15min":2.8039999999999998},"last":0.89600000000000002},{"interface":"front","average":{"1min":1.0069999999999999,"5min":1.0009999999999999,"15min":1.028},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.2509999999999999,"5min":1.4299999999999999,"15min":2.0819999999999999},"last":0.80800000000000005}]},{"osd":5,"last update":"Mon May 22 15:46:03 2023","interfaces":[{"interface":"back","average":{"1min":1.0569999999999999,"5min":1.056,"15min":1.052},"min":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.3959999999999999,"5min":1.5049999999999999,"15min":2.3860000000000001},"last":0.91000000000000003},{"interface":"front","average":{"1min":0.99099999999999999,"5min":0.97399999999999998,"15min":0.998},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.331,"5min":1.331,"15min":3.0299999999999998},"last":0.86799999999999999}]},{"osd":6,"last update":"Mon May 22 15:46:03 2023","interfaces":[{"interface":"back","average":{"1min":1.038,"5min":1.012,"15min":1.036},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":1.3320000000000001,"5min":1.49,"15min":2.883},"last":0.96299999999999997},{"interface":"front","average":{"1min":1.129,"5min":1.081,"15min":1.1040000000000001},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":1.4450000000000001,"5min":1.5660000000000001,"15min":3.1459999999999999},"last":0.84199999999999997}]},{"osd":7,"last update":"Mon May 22 15:46:03 2023","interfaces":[{"interface":"back","average":{"1min":1.1639999999999999,"5min":1.097,"15min":1.1279999999999999},"min":{"1min":0.55000000000000004,"5min":0.499,"15min":0.499},"max":{"1min":1.429,"5min":1.55,"15min":2.5990000000000002},"last":0.99299999999999999},{"interface":"front","average":{"1min":1.1240000000000001,"5min":1.0640000000000001,"15min":1.1060000000000001},"min":{"1min":0.5,"5min":0.5,"15min":0.5},"max":{"1min":1.4650000000000001,"5min":1.4650000000000001,"15min":3.8799999999999999},"last":1.0449999999999999}]}]},{"osd":1,"up_from":13,"seq":55834575091,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30376,"kb_used_data":1952,"kb_used_omap":0,"kb_used_meta":28416,"kb_avail":94341464,"statfs":{"total":96636764160,"available":96605659136,"internally_reserved":0,"allocated":1998848,"data_stored":1057102,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":29097984},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 15:46:17 2023","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.63200000000000001,"15min":0.63400000000000001},"min":{"1min":0.40600000000000003,"5min":0.26300000000000001,"15min":0.22700000000000001},"max":{"1min":1.1200000000000001,"5min":1.2270000000000001,"15min":1.2270000000000001},"last":0.74199999999999999},{"interface":"front","average":{"1min":0.77100000000000002,"5min":0.70699999999999996,"15min":0.76400000000000001},"min":{"1min":0.35999999999999999,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":1.1839999999999999,"5min":1.1839999999999999,"15min":1.2529999999999999},"last":1.218}]},{"osd":2,"last update":"Mon May 22 15:46:17 2023","interfaces":[{"interface":"back","average":{"1min":0.76400000000000001,"5min":0.71399999999999997,"15min":0.73799999999999999},"min":{"1min":0.44800000000000001,"5min":0.317,"15min":0.28999999999999998},"max":{"1min":1.0660000000000001,"5min":1.391,"15min":1.391},"last":1.012},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.72999999999999998,"15min":0.76300000000000001},"min":{"1min":0.49199999999999999,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.98699999999999999,"5min":1.2270000000000001,"15min":1.276},"last":0.58199999999999996}]},{"osd":3,"last update":"Mon May 22 15:46:17 2023","interfaces":[{"interface":"back","average":{"1min":0.80100000000000005,"5min":0.79900000000000004,"15min":0.82799999999999996},"min":{"1min":0.51200000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.153,"5min":1.657,"15min":1.657},"last":1.056},{"interface":"front","average":{"1min":0.81100000000000005,"5min":0.80800000000000005,"15min":0.81499999999999995},"min":{"1min":0.33700000000000002,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":1.095,"5min":2.0569999999999999,"15min":2.0569999999999999},"last":0.90800000000000003}]},{"osd":4,"last update":"Mon May 22 15:46:17 2023","interfaces":[{"interface":"back","average":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.96499999999999997},"min":{"1min":0.64500000000000002,"5min":0.55200000000000005,"15min":0.34000000000000002},"max":{"1min":1.502,"5min":2.9529999999999998,"15min":2.9529999999999998},"last":1.173},{"interface":"front","average":{"1min":0.97499999999999998,"5min":0.999,"15min":1.002},"min":{"1min":0.61299999999999999,"5min":0.504,"15min":0.49299999999999999},"max":{"1min":1.292,"5min":2.7759999999999998,"15min":2.7759999999999998},"last":0.96299999999999997}]},{"osd":5,"last update":"Mon May 22 15:46:17 2023","interfaces":[{"interface":"back","average":{"1min":1.0349999999999999,"5min":1.006,"15min":1.034},"min":{"1min":0.627,"5min":0.58399999999999996,"15min":0.443},"max":{"1min":1.556,"5min":1.9570000000000001,"15min":1.9570000000000001},"last":1.089},{"interface":"front","average":{"1min":1.1060000000000001,"5min":1.034,"15min":1.0449999999999999},"min":{"1min":0.59399999999999997,"5min":0.53400000000000003,"15min":0.47299999999999998},"max":{"1min":1.577,"5min":2.1200000000000001,"15min":2.1200000000000001},"last":1.419}]},{"osd":6,"last update":"Mon May 22 15:46:17 2023","interfaces":[{"interface":"back","average":{"1min":1.02,"5min":1.0369999999999999,"15min":1.036},"min":{"1min":0.65900000000000003,"5min":0.54000000000000004,"15min":0.45300000000000001},"max":{"1min":1.3180000000000001,"5min":2.831,"15min":2.831},"last":1.1359999999999999},{"interface":"front","average":{"1min":1.1499999999999999,"5min":1.085,"15min":1.0940000000000001},"min":{"1min":0.55800000000000005,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":1.6299999999999999,"5min":1.7629999999999999,"15min":1.7629999999999999},"last":1.335}]},{"osd":7,"last update":"Mon May 22 15:46:17 2023","interfaces":[{"interface":"back","average":{"1min":1.1759999999999999,"5min":1.135,"15min":1.1180000000000001},"min":{"1min":0.61299999999999999,"5min":0.60299999999999998,"15min":0.56799999999999995},"max":{"1min":1.679,"5min":2.4199999999999999,"15min":2.4199999999999999},"last":1.472},{"interface":"front","average":{"1min":1.1819999999999999,"5min":1.1259999999999999,"15min":1.1299999999999999},"min":{"1min":0.76300000000000001,"5min":0.47299999999999998,"15min":0.41599999999999998},"max":{"1min":1.619,"5min":3.8799999999999999,"15min":3.8799999999999999},"last":1.5069999999999999}]}]},{"osd":0,"up_from":13,"seq":55834575092,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30136,"kb_used_data":2544,"kb_used_omap":0,"kb_used_meta":27584,"kb_avail":94341704,"statfs":{"total":96636764160,"available":96605904896,"internally_reserved":0,"allocated":2605056,"data_stored":1648486,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":28246016},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon May 22 15:46:21 2023","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.65900000000000003,"15min":0.68100000000000005},"min":{"1min":0.307,"5min":0.248,"15min":0.223},"max":{"1min":1.2190000000000001,"5min":1.466,"15min":1.466},"last":0.27000000000000002},{"interface":"front","average":{"1min":0.69899999999999995,"5min":0.70599999999999996,"15min":0.72899999999999998},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.24099999999999999},"max":{"1min":1.2230000000000001,"5min":1.71,"15min":1.71},"last":0.36899999999999999}]},{"osd":2,"last update":"Mon May 22 15:46:21 2023","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":0.70699999999999996,"15min":0.74099999999999999},"min":{"1min":0.39700000000000002,"5min":0.318,"15min":0.26500000000000001},"max":{"1min":0.96799999999999997,"5min":1.165,"15min":1.3220000000000001},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.82799999999999996,"5min":0.75900000000000001,"15min":0.78800000000000003},"min":{"1min":0.375,"5min":0.34999999999999998,"15min":0.32900000000000001},"max":{"1min":1.1699999999999999,"5min":1.3129999999999999,"15min":1.3129999999999999},"last":0.42499999999999999}]},{"osd":3,"last update":"Mon May 22 15:46:21 2023","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":0.74099999999999999,"15min":0.76000000000000001},"min":{"1min":0.35499999999999998,"5min":0.31,"15min":0.31},"max":{"1min":0.96199999999999997,"5min":1.9299999999999999,"15min":1.9299999999999999},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.83499999999999996,"5min":0.83299999999999996,"15min":0.871},"min":{"1min":0.35599999999999998,"5min":0.35199999999999998,"15min":0.34399999999999997},"max":{"1min":1.1299999999999999,"5min":1.758,"15min":1.758},"last":0.45800000000000002}]},{"osd":4,"last update":"Mon May 22 15:46:21 2023","interfaces":[{"interface":"back","average":{"1min":0.92800000000000005,"5min":0.95099999999999996,"15min":0.95099999999999996},"min":{"1min":0.57599999999999996,"5min":0.57399999999999995,"15min":0.48099999999999998},"max":{"1min":1.149,"5min":2.923,"15min":2.923},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.98999999999999999,"5min":1.006,"15min":1.026},"min":{"1min":0.624,"5min":0.624,"15min":0.624},"max":{"1min":1.3839999999999999,"5min":2.0129999999999999,"15min":2.0129999999999999},"last":0.84099999999999997}]},{"osd":5,"last update":"Mon May 22 15:46:21 2023","interfaces":[{"interface":"back","average":{"1min":1,"5min":1.004,"15min":1.0409999999999999},"min":{"1min":0.59099999999999997,"5min":0.56000000000000005,"15min":0.46000000000000002},"max":{"1min":1.238,"5min":1.97,"15min":1.97},"last":0.77300000000000002},{"interface":"front","average":{"1min":1.02,"5min":1.016,"15min":1.008},"min":{"1min":0.67600000000000005,"5min":0.56599999999999995,"15min":0.55300000000000005},"max":{"1min":1.5349999999999999,"5min":5.0519999999999996,"15min":5.0519999999999996},"last":0.58399999999999996}]},{"osd":6,"last update":"Mon May 22 15:46:21 2023","interfaces":[{"interface":"back","average":{"1min":1.044,"5min":1.0589999999999999,"15min":1.089},"min":{"1min":0.58899999999999997,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":1.306,"5min":2.117,"15min":2.117},"last":0.74199999999999999},{"interface":"front","average":{"1min":1.085,"5min":1.1299999999999999,"15min":1.119},"min":{"1min":0.69199999999999995,"5min":0.65500000000000003,"15min":0.499},"max":{"1min":1.3100000000000001,"5min":5.2640000000000002,"15min":5.2640000000000002},"last":0.66500000000000004}]},{"osd":7,"last update":"Mon May 22 15:46:21 2023","interfaces":[{"interface":"back","average":{"1min":1.075,"5min":1.1040000000000001,"15min":1.129},"min":{"1min":0.63800000000000001,"5min":0.61299999999999999,"15min":0.61299999999999999},"max":{"1min":1.3640000000000001,"5min":2.0550000000000002,"15min":2.0550000000000002},"last":0.60499999999999998},{"interface":"front","average":{"1min":1.089,"5min":1.0900000000000001,"15min":1.0960000000000001},"min":{"1min":0.73099999999999998,"5min":0.63300000000000001,"15min":0.60299999999999998},"max":{"1min":1.6240000000000001,"5min":3.9060000000000001,"15min":3.9060000000000001},"last":0.81899999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-05-22T15:47:07.409 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-05-22T15:47:07.409 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2023-05-22T15:47:07.578 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.571+0000 7fbe4a5c2640 1 Processor -- start 2023-05-22T15:47:07.578 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.571+0000 7fbe4a5c2640 1 -- start start 2023-05-22T15:47:07.579 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.575+0000 7fbe4a5c2640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbe4415c350 0x7fbe4415c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:47:07.579 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.575+0000 7fbe4a5c2640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fbe44069720 0x7fbe44069b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:47:07.579 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.575+0000 7fbe4a5c2640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe440c95f0 con 0x7fbe440ade80 2023-05-22T15:47:07.580 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.575+0000 7fbe4a5c2640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe440ca270 con 0x7fbe44069720 2023-05-22T15:47:07.580 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.575+0000 7fbe4a5c2640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe440ca690 con 0x7fbe4415c350 2023-05-22T15:47:07.581 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.575+0000 7fbe437fe640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbe4415c350 0x7fbe4415c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:47:07.581 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.575+0000 7fbe48b5f640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fbe44069720 0x7fbe44069b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:47:07.581 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.575+0000 7fbe437fe640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbe4415c350 0x7fbe4415c7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3301/0 says I am v2:172.21.15.16:46104/0 (socket says 172.21.15.16:46104) 2023-05-22T15:47:07.581 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.575+0000 7fbe437fe640 1 -- 172.21.15.16:0/1482704936 learned_addr learned my addr 172.21.15.16:0/1482704936 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:47:07.581 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.575+0000 7fbe43fff640 1 --1- 172.21.15.16:0/1482704936 >> v1:172.21.15.62:6789/0 conn(0x7fbe440ade80 0x7fbe440cc400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.16:53098/0 (socket says 172.21.15.16:53098) 2023-05-22T15:47:07.581 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.575+0000 7fbe437fe640 1 -- 172.21.15.16:0/1482704936 >> v2:172.21.15.62:3300/0 conn(0x7fbe44069720 msgr2=0x7fbe44069b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:47:07.581 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.575+0000 7fbe437fe640 1 --2- 172.21.15.16:0/1482704936 >> v2:172.21.15.62:3300/0 conn(0x7fbe44069720 0x7fbe44069b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:07.582 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.575+0000 7fbe437fe640 1 -- 172.21.15.16:0/1482704936 >> v1:172.21.15.62:6789/0 conn(0x7fbe440ade80 legacy=0x7fbe440cc400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:47:07.582 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.575+0000 7fbe437fe640 1 -- 172.21.15.16:0/1482704936 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe440c98d0 con 0x7fbe4415c350 2023-05-22T15:47:07.582 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.575+0000 7fbe48b5f640 1 --2- 172.21.15.16:0/1482704936 >> v2:172.21.15.62:3300/0 conn(0x7fbe44069720 0x7fbe44069b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-05-22T15:47:07.582 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.575+0000 7fbe437fe640 1 --2- 172.21.15.16:0/1482704936 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbe4415c350 0x7fbe4415c7c0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fbe38009870 tx=0x7fbe3801c4b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dccdbc249ac2c4ad server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:47:07.583 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.575+0000 7fbe42ffd640 1 -- 172.21.15.16:0/1482704936 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe380ab600 con 0x7fbe4415c350 2023-05-22T15:47:07.583 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.579+0000 7fbe42ffd640 1 -- 172.21.15.16:0/1482704936 <== mon.2 v2:172.21.15.62:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe380a7050 con 0x7fbe4415c350 2023-05-22T15:47:07.583 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.579+0000 7fbe42ffd640 1 -- 172.21.15.16:0/1482704936 <== mon.2 v2:172.21.15.62:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe380ab980 con 0x7fbe4415c350 2023-05-22T15:47:07.584 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.579+0000 7fbe4a5c2640 1 -- 172.21.15.16:0/1482704936 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbe4415c350 msgr2=0x7fbe4415c7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:47:07.584 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.579+0000 7fbe4a5c2640 1 --2- 172.21.15.16:0/1482704936 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbe4415c350 0x7fbe4415c7c0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fbe38009870 tx=0x7fbe3801c4b0 comp rx=0 tx=0).stop 2023-05-22T15:47:07.584 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.579+0000 7fbe4a5c2640 1 -- 172.21.15.16:0/1482704936 shutdown_connections 2023-05-22T15:47:07.584 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.579+0000 7fbe4a5c2640 1 --2- 172.21.15.16:0/1482704936 >> v2:172.21.15.62:3300/0 conn(0x7fbe44069720 0x7fbe44069b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:07.584 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.579+0000 7fbe4a5c2640 1 --2- 172.21.15.16:0/1482704936 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbe4415c350 0x7fbe4415c7c0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:07.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.579+0000 7fbe4a5c2640 1 -- 172.21.15.16:0/1482704936 >> 172.21.15.16:0/1482704936 conn(0x7fbe440b1110 msgr2=0x7fbe440ce830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:47:07.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.579+0000 7fbe4a5c2640 1 -- 172.21.15.16:0/1482704936 shutdown_connections 2023-05-22T15:47:07.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.579+0000 7fbe4a5c2640 1 -- 172.21.15.16:0/1482704936 wait complete. 2023-05-22T15:47:07.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.579+0000 7fbe4a5c2640 1 Processor -- start 2023-05-22T15:47:07.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.579+0000 7fbe4a5c2640 1 -- start start 2023-05-22T15:47:07.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.579+0000 7fbe4a5c2640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbe44069720 0x7fbe44156c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:47:07.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.579+0000 7fbe4a5c2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbe440ade80 0x7fbe44070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:47:07.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.579+0000 7fbe4a5c2640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbe4415c350 0x7fbe44153420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:47:07.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.579+0000 7fbe4a5c2640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe44152cb0 con 0x7fbe440ade80 2023-05-22T15:47:07.587 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.579+0000 7fbe4a5c2640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe441482f0 con 0x7fbe44069720 2023-05-22T15:47:07.587 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.579+0000 7fbe4a5c2640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe44154350 con 0x7fbe4415c350 2023-05-22T15:47:07.587 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.579+0000 7fbe437fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbe440ade80 0x7fbe44070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:47:07.587 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.579+0000 7fbe437fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbe440ade80 0x7fbe44070150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46878/0 (socket says 172.21.15.16:46878) 2023-05-22T15:47:07.587 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.579+0000 7fbe437fe640 1 -- 172.21.15.16:0/3381830584 learned_addr learned my addr 172.21.15.16:0/3381830584 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:47:07.587 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.579+0000 7fbe48b5f640 1 --2- 172.21.15.16:0/3381830584 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbe4415c350 0x7fbe44153420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:47:07.587 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.579+0000 7fbe43fff640 1 --2- 172.21.15.16:0/3381830584 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbe44069720 0x7fbe44156c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:47:07.588 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.579+0000 7fbe437fe640 1 -- 172.21.15.16:0/3381830584 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbe4415c350 msgr2=0x7fbe44153420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:47:07.589 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.579+0000 7fbe437fe640 1 --2- 172.21.15.16:0/3381830584 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbe4415c350 0x7fbe44153420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:07.589 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.579+0000 7fbe437fe640 1 -- 172.21.15.16:0/3381830584 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbe44069720 msgr2=0x7fbe44156c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:47:07.589 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.579+0000 7fbe437fe640 1 --2- 172.21.15.16:0/3381830584 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbe44069720 0x7fbe44156c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:07.589 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.579+0000 7fbe437fe640 1 -- 172.21.15.16:0/3381830584 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe3c009a70 con 0x7fbe440ade80 2023-05-22T15:47:07.589 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.579+0000 7fbe437fe640 1 --2- 172.21.15.16:0/3381830584 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbe440ade80 0x7fbe44070150 secure :-1 s=READY pgs=2046 cs=0 l=1 rev1=1 crypto rx=0x7fbe3801a0e0 tx=0x7fbe38019250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:47:07.589 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.579+0000 7fbe40ff9640 1 -- 172.21.15.16:0/3381830584 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe380b6420 con 0x7fbe440ade80 2023-05-22T15:47:07.589 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.579+0000 7fbe40ff9640 1 -- 172.21.15.16:0/3381830584 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe380b65a0 con 0x7fbe440ade80 2023-05-22T15:47:07.590 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.579+0000 7fbe4a5c2640 1 -- 172.21.15.16:0/3381830584 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe380094d0 con 0x7fbe440ade80 2023-05-22T15:47:07.590 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.579+0000 7fbe40ff9640 1 -- 172.21.15.16:0/3381830584 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe380b6950 con 0x7fbe440ade80 2023-05-22T15:47:07.590 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.583+0000 7fbe4a5c2640 1 -- 172.21.15.16:0/3381830584 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe440c9210 con 0x7fbe440ade80 2023-05-22T15:47:07.590 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.583+0000 7fbe4a5c2640 1 -- 172.21.15.16:0/3381830584 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe4415d090 con 0x7fbe440ade80 2023-05-22T15:47:07.590 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.583+0000 7fbe40ff9640 1 -- 172.21.15.16:0/3381830584 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 67) v1 ==== 84332+0+0 (secure 0 0 0) 0x7fbe380b2040 con 0x7fbe440ade80 2023-05-22T15:47:07.590 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.583+0000 7fbe40ff9640 1 --2- 172.21.15.16:0/3381830584 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fbe300654d0 0x7fbe30067980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:47:07.591 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.583+0000 7fbe40ff9640 1 -- 172.21.15.16:0/3381830584 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(161..161 src has 1..161) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fbe4406a8b0 con 0x7fbe440ade80 2023-05-22T15:47:07.591 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.583+0000 7fbe43fff640 1 --2- 172.21.15.16:0/3381830584 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fbe300654d0 0x7fbe30067980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:47:07.591 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.583+0000 7fbe43fff640 1 --2- 172.21.15.16:0/3381830584 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fbe300654d0 0x7fbe30067980 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fbe34000f70 tx=0x7fbe34007e50 comp rx=0 tx=0).ready entity=mgr.9396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:47:07.593 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.587+0000 7fbe40ff9640 1 -- 172.21.15.16:0/3381830584 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fbe380ea7b0 con 0x7fbe440ade80 2023-05-22T15:47:07.806 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.799+0000 7fbe4a5c2640 1 -- 172.21.15.16:0/3381830584 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7fbe440c93a0 con 0x7fbe440ade80 2023-05-22T15:47:07.813 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.807+0000 7fbe40ff9640 1 -- 172.21.15.16:0/3381830584 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v26) v1 ==== 190+0+0 (secure 0 0 0) 0x7fbe380eddd0 con 0x7fbe440ade80 2023-05-22T15:47:07.813 INFO:teuthology.orchestra.run.smithi016.stderr:updated caps for client.0 2023-05-22T15:47:07.815 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.811+0000 7fbe4a5c2640 1 -- 172.21.15.16:0/3381830584 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fbe300654d0 msgr2=0x7fbe30067980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:47:07.816 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.811+0000 7fbe4a5c2640 1 --2- 172.21.15.16:0/3381830584 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fbe300654d0 0x7fbe30067980 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fbe34000f70 tx=0x7fbe34007e50 comp rx=0 tx=0).stop 2023-05-22T15:47:07.816 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.811+0000 7fbe4a5c2640 1 -- 172.21.15.16:0/3381830584 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbe440ade80 msgr2=0x7fbe44070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:47:07.816 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.811+0000 7fbe4a5c2640 1 --2- 172.21.15.16:0/3381830584 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbe440ade80 0x7fbe44070150 secure :-1 s=READY pgs=2046 cs=0 l=1 rev1=1 crypto rx=0x7fbe3801a0e0 tx=0x7fbe38019250 comp rx=0 tx=0).stop 2023-05-22T15:47:07.816 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.811+0000 7fbe4a5c2640 1 -- 172.21.15.16:0/3381830584 shutdown_connections 2023-05-22T15:47:07.816 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.811+0000 7fbe4a5c2640 1 --2- 172.21.15.16:0/3381830584 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fbe300654d0 0x7fbe30067980 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:07.816 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.811+0000 7fbe4a5c2640 1 --2- 172.21.15.16:0/3381830584 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbe4415c350 0x7fbe44153420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:07.816 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.811+0000 7fbe4a5c2640 1 --2- 172.21.15.16:0/3381830584 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbe440ade80 0x7fbe44070150 unknown :-1 s=CLOSED pgs=2046 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:07.817 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.811+0000 7fbe4a5c2640 1 --2- 172.21.15.16:0/3381830584 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbe44069720 0x7fbe44156c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:07.817 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.811+0000 7fbe4a5c2640 1 -- 172.21.15.16:0/3381830584 >> 172.21.15.16:0/3381830584 conn(0x7fbe440b1110 msgr2=0x7fbe4415b120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:47:07.817 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.811+0000 7fbe4a5c2640 1 -- 172.21.15.16:0/3381830584 shutdown_connections 2023-05-22T15:47:07.817 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.811+0000 7fbe4a5c2640 1 -- 172.21.15.16:0/3381830584 wait complete. 2023-05-22T15:47:07.833 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-05-22T15:47:07.950 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.943+0000 7f7cb21af640 1 Processor -- start 2023-05-22T15:47:07.951 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.943+0000 7f7cb21af640 1 -- start start 2023-05-22T15:47:07.951 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.947+0000 7f7cb21af640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7cac0ca4e0 0x7f7cac15e460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:47:07.952 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.947+0000 7f7cb21af640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7cac0b0650 con 0x7f7cac070df0 2023-05-22T15:47:07.952 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.947+0000 7f7cb21af640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7cac0b0370 con 0x7f7cac0c9b40 2023-05-22T15:47:07.952 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.947+0000 7f7cb21af640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7cac15d980 con 0x7f7cac0ca4e0 2023-05-22T15:47:07.952 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.947+0000 7f7cabfff640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f7cac070df0 0x7f7cac071210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:44880/0 (socket says 172.21.15.16:44880) 2023-05-22T15:47:07.952 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.947+0000 7f7cabfff640 1 -- 172.21.15.16:0/282439327 learned_addr learned my addr 172.21.15.16:0/282439327 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:47:07.953 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.947+0000 7f7cab7fe640 1 --2- 172.21.15.16:0/282439327 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7cac0ca4e0 0x7f7cac15e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:47:07.954 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.947+0000 7f7caa7fc640 1 -- 172.21.15.16:0/282439327 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1226105553 0 0) 0x7f7cac0b0650 con 0x7f7cac070df0 2023-05-22T15:47:07.954 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.947+0000 7f7caa7fc640 1 -- 172.21.15.16:0/282439327 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7c98003530 con 0x7f7cac070df0 2023-05-22T15:47:07.954 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.947+0000 7f7caa7fc640 1 -- 172.21.15.16:0/282439327 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1500340195 0 0) 0x7f7c98003530 con 0x7f7cac070df0 2023-05-22T15:47:07.954 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.947+0000 7f7caa7fc640 1 -- 172.21.15.16:0/282439327 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7cac0ca4e0 msgr2=0x7f7cac15e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:47:07.954 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.947+0000 7f7caa7fc640 1 --2- 172.21.15.16:0/282439327 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7cac0ca4e0 0x7f7cac15e460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:07.954 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.947+0000 7f7caa7fc640 1 -- 172.21.15.16:0/282439327 >> v1:172.21.15.62:6789/0 conn(0x7f7cac0c9b40 legacy=0x7f7cac0afb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:47:07.954 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.947+0000 7f7caa7fc640 1 -- 172.21.15.16:0/282439327 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cac15dc60 con 0x7f7cac070df0 2023-05-22T15:47:07.955 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.947+0000 7f7caa7fc640 1 -- 172.21.15.16:0/282439327 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7f7ca4001a00 con 0x7f7cac070df0 2023-05-22T15:47:07.955 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.947+0000 7f7caa7fc640 1 -- 172.21.15.16:0/282439327 >> v1:172.21.15.16:6789/0 conn(0x7f7cac070df0 legacy=0x7f7cac071210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:47:07.955 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.947+0000 7f7caa7fc640 1 --2- 172.21.15.16:0/282439327 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7cac0ca4e0 0x7f7c980039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:47:07.955 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.947+0000 7f7caa7fc640 1 --2- 172.21.15.16:0/282439327 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7c98003f30 0x7f7c980047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:47:07.955 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.947+0000 7f7caa7fc640 1 --2- 172.21.15.16:0/282439327 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7c98004d10 0x7f7c98005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:47:07.955 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.947+0000 7f7caa7fc640 1 -- 172.21.15.16:0/282439327 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cac15d980 con 0x7f7c98003f30 2023-05-22T15:47:07.955 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.947+0000 7f7caa7fc640 1 -- 172.21.15.16:0/282439327 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c9800bb20 con 0x7f7c98004d10 2023-05-22T15:47:07.956 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.947+0000 7f7caa7fc640 1 -- 172.21.15.16:0/282439327 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c9800bce0 con 0x7f7cac0ca4e0 2023-05-22T15:47:07.956 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.947+0000 7f7caaffd640 1 --2- 172.21.15.16:0/282439327 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7c98003f30 0x7f7c980047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:47:07.956 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.947+0000 7f7cabfff640 1 --2- 172.21.15.16:0/282439327 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7c98004d10 0x7f7c98005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:47:07.956 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.947+0000 7f7cab7fe640 1 --2- 172.21.15.16:0/282439327 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7cac0ca4e0 0x7f7c980039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:47:07.956 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.947+0000 7f7caaffd640 1 -- 172.21.15.16:0/282439327 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7cac0ca4e0 msgr2=0x7f7c980039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:47:07.956 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.947+0000 7f7caaffd640 1 --2- 172.21.15.16:0/282439327 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7cac0ca4e0 0x7f7c980039b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:07.956 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.947+0000 7f7caaffd640 1 -- 172.21.15.16:0/282439327 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7c98004d10 msgr2=0x7f7c98005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:47:07.956 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.947+0000 7f7caaffd640 1 --2- 172.21.15.16:0/282439327 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7c98004d10 0x7f7c98005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:07.957 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.947+0000 7f7caaffd640 1 -- 172.21.15.16:0/282439327 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c98003530 con 0x7f7c98003f30 2023-05-22T15:47:07.959 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.951+0000 7f7caaffd640 1 --2- 172.21.15.16:0/282439327 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7c98003f30 0x7f7c980047d0 secure :-1 s=READY pgs=2048 cs=0 l=1 rev1=1 crypto rx=0x7f7c94001750 tx=0x7f7c9401c390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a2fcbdc0723e4b3 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:47:07.959 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.951+0000 7f7caa7fc640 1 -- 172.21.15.16:0/282439327 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c940052d0 con 0x7f7c98003f30 2023-05-22T15:47:07.959 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.951+0000 7f7caa7fc640 1 -- 172.21.15.16:0/282439327 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c940a7050 con 0x7f7c98003f30 2023-05-22T15:47:07.959 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.951+0000 7f7caa7fc640 1 -- 172.21.15.16:0/282439327 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c94003600 con 0x7f7c98003f30 2023-05-22T15:47:07.959 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.951+0000 7f7cb21af640 1 -- 172.21.15.16:0/282439327 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7c98003f30 msgr2=0x7f7c980047d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:47:07.959 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.951+0000 7f7cb21af640 1 --2- 172.21.15.16:0/282439327 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7c98003f30 0x7f7c980047d0 secure :-1 s=READY pgs=2048 cs=0 l=1 rev1=1 crypto rx=0x7f7c94001750 tx=0x7f7c9401c390 comp rx=0 tx=0).stop 2023-05-22T15:47:07.960 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.951+0000 7f7cabfff640 1 -- 172.21.15.16:0/282439327 reap_dead start 2023-05-22T15:47:07.960 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.951+0000 7f7cb21af640 1 -- 172.21.15.16:0/282439327 shutdown_connections 2023-05-22T15:47:07.960 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.951+0000 7f7cb21af640 1 -- 172.21.15.16:0/282439327 >> 172.21.15.16:0/282439327 conn(0x7f7cac069720 msgr2=0x7f7cac069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:47:07.960 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.951+0000 7f7cb21af640 1 -- 172.21.15.16:0/282439327 shutdown_connections 2023-05-22T15:47:07.960 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.951+0000 7f7cb21af640 1 -- 172.21.15.16:0/282439327 wait complete. 2023-05-22T15:47:07.960 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.951+0000 7f7cb21af640 1 Processor -- start 2023-05-22T15:47:07.960 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.951+0000 7f7cb21af640 1 -- start start 2023-05-22T15:47:07.961 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.951+0000 7f7cb21af640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7c98003f30 0x7f7cac14fd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:47:07.961 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.951+0000 7f7cb21af640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7cac1516c0 0x7f7cac1f8510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:47:07.961 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.951+0000 7f7cb21af640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7cac1f8a50 0x7f7cac1f4ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:47:07.961 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.951+0000 7f7cb21af640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cac152b20 con 0x7f7cac1516c0 2023-05-22T15:47:07.961 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.951+0000 7f7cb21af640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cac1f8ec0 con 0x7f7cac1f8a50 2023-05-22T15:47:07.961 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.951+0000 7f7cb21af640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7cac151ea0 con 0x7f7c98003f30 2023-05-22T15:47:07.961 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.951+0000 7f7caaffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7cac1516c0 0x7f7cac1f8510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:47:07.961 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.951+0000 7f7caaffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7cac1516c0 0x7f7cac1f8510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46884/0 (socket says 172.21.15.16:46884) 2023-05-22T15:47:07.962 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.951+0000 7f7caaffd640 1 -- 172.21.15.16:0/262701953 learned_addr learned my addr 172.21.15.16:0/262701953 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:47:07.962 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.951+0000 7f7cab7fe640 1 --2- 172.21.15.16:0/262701953 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7c98003f30 0x7f7cac14fd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:47:07.962 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.951+0000 7f7cabfff640 1 --2- 172.21.15.16:0/262701953 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7cac1f8a50 0x7f7cac1f4ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:47:07.962 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.951+0000 7f7caaffd640 1 -- 172.21.15.16:0/262701953 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7c98003f30 msgr2=0x7f7cac14fd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:47:07.962 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.951+0000 7f7caaffd640 1 --2- 172.21.15.16:0/262701953 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7c98003f30 0x7f7cac14fd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:07.962 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.951+0000 7f7caaffd640 1 -- 172.21.15.16:0/262701953 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7cac1f8a50 msgr2=0x7f7cac1f4ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:47:07.962 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.951+0000 7f7caaffd640 1 --2- 172.21.15.16:0/262701953 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7cac1f8a50 0x7f7cac1f4ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:07.962 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.951+0000 7f7caaffd640 1 -- 172.21.15.16:0/262701953 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cac0c9010 con 0x7f7cac1516c0 2023-05-22T15:47:07.963 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.951+0000 7f7cab7fe640 1 --2- 172.21.15.16:0/262701953 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7c98003f30 0x7f7cac14fd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T15:47:07.963 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.951+0000 7f7caaffd640 1 --2- 172.21.15.16:0/262701953 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7cac1516c0 0x7f7cac1f8510 secure :-1 s=READY pgs=2049 cs=0 l=1 rev1=1 crypto rx=0x7f7c94001750 tx=0x7f7c98005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:47:07.963 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.955+0000 7f7ca3fff640 1 -- 172.21.15.16:0/262701953 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c94007040 con 0x7f7cac1516c0 2023-05-22T15:47:07.963 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.955+0000 7f7ca3fff640 1 -- 172.21.15.16:0/262701953 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c940071a0 con 0x7f7cac1516c0 2023-05-22T15:47:07.963 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.955+0000 7f7ca3fff640 1 -- 172.21.15.16:0/262701953 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c94007510 con 0x7f7cac1516c0 2023-05-22T15:47:07.963 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.955+0000 7f7cb21af640 1 -- 172.21.15.16:0/262701953 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7cac0c8e80 con 0x7f7cac1516c0 2023-05-22T15:47:07.963 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.955+0000 7f7cb21af640 1 -- 172.21.15.16:0/262701953 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7cac0c91a0 con 0x7f7cac1516c0 2023-05-22T15:47:07.964 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.955+0000 7f7cb21af640 1 -- 172.21.15.16:0/262701953 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7cac04dbe0 con 0x7f7cac1516c0 2023-05-22T15:47:07.964 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.955+0000 7f7ca3fff640 1 -- 172.21.15.16:0/262701953 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 67) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f7c940067e0 con 0x7f7cac1516c0 2023-05-22T15:47:07.964 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.955+0000 7f7ca3fff640 1 --2- 172.21.15.16:0/262701953 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f7c8c0654d0 0x7f7c8c067980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:47:07.965 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.955+0000 7f7ca3fff640 1 -- 172.21.15.16:0/262701953 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(161..161 src has 1..161) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f7c9411cdc0 con 0x7f7cac1516c0 2023-05-22T15:47:07.965 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.955+0000 7f7cab7fe640 1 --2- 172.21.15.16:0/262701953 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f7c8c0654d0 0x7f7c8c067980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:47:07.965 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.955+0000 7f7cab7fe640 1 --2- 172.21.15.16:0/262701953 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f7c8c0654d0 0x7f7c8c067980 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f7c9c009af0 tx=0x7f7c9c008040 comp rx=0 tx=0).ready entity=mgr.9396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:47:07.965 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:07.959+0000 7f7ca3fff640 1 -- 172.21.15.16:0/262701953 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f7c940ee180 con 0x7f7cac1516c0 2023-05-22T15:47:08.152 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.147+0000 7f7cb21af640 1 -- 172.21.15.16:0/262701953 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7cac0af5d0 con 0x7f7cac1516c0 2023-05-22T15:47:08.153 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.147+0000 7f7ca3fff640 1 -- 172.21.15.16:0/262701953 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) v1 ==== 94+0+4644 (secure 0 0 0) 0x7f7c940edd70 con 0x7f7cac1516c0 2023-05-22T15:47:08.154 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:47:08.154 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":66,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8170,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6833/78835131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":78835131},{"type":"v1","addr":"172.21.15.62:6833","nonce":78835131}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":61},{"gid":8173,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/1188098562","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":1188098562},{"type":"v1","addr":"172.21.15.62:6835","nonce":1188098562}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":61},{"gid":9645,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/4204206994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":4204206994},{"type":"v1","addr":"172.21.15.16:6835","nonce":4204206994}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":61}],"filesystems":[{"mdsmap":{"epoch":66,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-05-22T15:47:03.009254+0000","modified":"2023-05-22T15:47:06.030064+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9654},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9654":{"gid":9654,"name":"c","rank":0,"incarnation":62,"state":"up:active","state_seq":3,"addr":"172.21.15.16:6837/1284822222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":1284822222},{"type":"v1","addr":"172.21.15.16:6837","nonce":1284822222}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2023-05-22T15:47:08.154 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 66 2023-05-22T15:47:08.155 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.151+0000 7f7cb21af640 1 -- 172.21.15.16:0/262701953 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f7c8c0654d0 msgr2=0x7f7c8c067980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:47:08.156 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.151+0000 7f7cb21af640 1 --2- 172.21.15.16:0/262701953 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f7c8c0654d0 0x7f7c8c067980 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f7c9c009af0 tx=0x7f7c9c008040 comp rx=0 tx=0).stop 2023-05-22T15:47:08.156 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.151+0000 7f7cb21af640 1 -- 172.21.15.16:0/262701953 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7cac1516c0 msgr2=0x7f7cac1f8510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:47:08.156 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.151+0000 7f7cb21af640 1 --2- 172.21.15.16:0/262701953 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7cac1516c0 0x7f7cac1f8510 secure :-1 s=READY pgs=2049 cs=0 l=1 rev1=1 crypto rx=0x7f7c94001750 tx=0x7f7c98005800 comp rx=0 tx=0).stop 2023-05-22T15:47:08.156 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.151+0000 7f7cb21af640 1 -- 172.21.15.16:0/262701953 shutdown_connections 2023-05-22T15:47:08.156 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.151+0000 7f7cb21af640 1 --2- 172.21.15.16:0/262701953 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7cac1f8a50 0x7f7cac1f4ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:08.156 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.151+0000 7f7cb21af640 1 --2- 172.21.15.16:0/262701953 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7cac1516c0 0x7f7cac1f8510 unknown :-1 s=CLOSED pgs=2049 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:08.156 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.151+0000 7f7cb21af640 1 --2- 172.21.15.16:0/262701953 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f7c8c0654d0 0x7f7c8c067980 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:08.157 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.151+0000 7f7cb21af640 1 --2- 172.21.15.16:0/262701953 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7c98003f30 0x7f7cac14fd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:08.157 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.151+0000 7f7cb21af640 1 -- 172.21.15.16:0/262701953 >> 172.21.15.16:0/262701953 conn(0x7f7cac069720 msgr2=0x7f7cac0ad5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:47:08.157 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.151+0000 7f7cb21af640 1 -- 172.21.15.16:0/262701953 shutdown_connections 2023-05-22T15:47:08.157 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.151+0000 7f7cb21af640 1 -- 172.21.15.16:0/262701953 wait complete. 2023-05-22T15:47:08.574 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 66, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-05-22T15:47:03.009254+0000', 'modified': '2023-05-22T15:47:06.030064+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 9654}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9654': {'gid': 9654, 'name': 'c', 'rank': 0, 'incarnation': 62, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.16:6837/1284822222', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6836', 'nonce': 1284822222}, {'type': 'v1', 'addr': '172.21.15.16:6837', 'nonce': 1284822222}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-05-22T15:47:08.575 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-05-22T15:47:08.575 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2023-05-22T15:47:08.712 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.707+0000 7f810fc94640 1 Processor -- start 2023-05-22T15:47:08.712 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.707+0000 7f810fc94640 1 -- start start 2023-05-22T15:47:08.714 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.707+0000 7f810fc94640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f8108159cd0 0x7f810815a0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:47:08.714 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.707+0000 7f810fc94640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f81080cc6f0 0x7f81080cfe30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:47:08.714 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.707+0000 7f810fc94640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81080cf9a0 con 0x7f81080b2880 2023-05-22T15:47:08.714 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.707+0000 7f810fc94640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f810815aba0 con 0x7f81080cc6f0 2023-05-22T15:47:08.714 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.707+0000 7f810fc94640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f810815ae80 con 0x7f8108159cd0 2023-05-22T15:47:08.715 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.707+0000 7f810e231640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f81080cc6f0 0x7f81080cfe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:47:08.715 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.707+0000 7f810e231640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f81080cc6f0 0x7f81080cfe30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46892/0 (socket says 172.21.15.16:46892) 2023-05-22T15:47:08.715 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.707+0000 7f810e231640 1 -- 172.21.15.16:0/1893534466 learned_addr learned my addr 172.21.15.16:0/1893534466 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:47:08.715 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.707+0000 7f810da30640 1 --2- 172.21.15.16:0/1893534466 >> v2:172.21.15.62:3300/0 conn(0x7f8108159cd0 0x7f810815a0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:47:08.715 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.707+0000 7f810e231640 1 -- 172.21.15.16:0/1893534466 >> v2:172.21.15.62:3300/0 conn(0x7f8108159cd0 msgr2=0x7f810815a0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:47:08.715 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.707+0000 7f810e231640 1 --2- 172.21.15.16:0/1893534466 >> v2:172.21.15.62:3300/0 conn(0x7f8108159cd0 0x7f810815a0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:08.715 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.707+0000 7f810e231640 1 -- 172.21.15.16:0/1893534466 >> v1:172.21.15.62:6789/0 conn(0x7f81080b2880 legacy=0x7f81080cbfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:47:08.716 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.707+0000 7f810e231640 1 -- 172.21.15.16:0/1893534466 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f810815b160 con 0x7f81080cc6f0 2023-05-22T15:47:08.716 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.711+0000 7f810e231640 1 --2- 172.21.15.16:0/1893534466 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f81080cc6f0 0x7f81080cfe30 secure :-1 s=READY pgs=2050 cs=0 l=1 rev1=1 crypto rx=0x7f81080d1330 tx=0x7f80f801c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e95d8f66557066fe server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:47:08.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.711+0000 7f810ca2e640 1 -- 172.21.15.16:0/1893534466 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80f80ac640 con 0x7f81080cc6f0 2023-05-22T15:47:08.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.711+0000 7f810ca2e640 1 -- 172.21.15.16:0/1893534466 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80f80ac7a0 con 0x7f81080cc6f0 2023-05-22T15:47:08.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.711+0000 7f810ca2e640 1 -- 172.21.15.16:0/1893534466 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80f80acb30 con 0x7f81080cc6f0 2023-05-22T15:47:08.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.711+0000 7f810fc94640 1 -- 172.21.15.16:0/1893534466 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f81080cc6f0 msgr2=0x7f81080cfe30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:47:08.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.711+0000 7f810fc94640 1 --2- 172.21.15.16:0/1893534466 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f81080cc6f0 0x7f81080cfe30 secure :-1 s=READY pgs=2050 cs=0 l=1 rev1=1 crypto rx=0x7f81080d1330 tx=0x7f80f801c4d0 comp rx=0 tx=0).stop 2023-05-22T15:47:08.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.711+0000 7f810fc94640 1 -- 172.21.15.16:0/1893534466 shutdown_connections 2023-05-22T15:47:08.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.711+0000 7f810fc94640 1 --2- 172.21.15.16:0/1893534466 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f81080cc6f0 0x7f81080cfe30 unknown :-1 s=CLOSED pgs=2050 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:08.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.711+0000 7f810fc94640 1 --2- 172.21.15.16:0/1893534466 >> v2:172.21.15.62:3300/0 conn(0x7f8108159cd0 0x7f810815a0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:08.719 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.711+0000 7f810fc94640 1 -- 172.21.15.16:0/1893534466 >> 172.21.15.16:0/1893534466 conn(0x7f810806f5c0 msgr2=0x7f810806f9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:47:08.719 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.711+0000 7f810fc94640 1 -- 172.21.15.16:0/1893534466 shutdown_connections 2023-05-22T15:47:08.720 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.711+0000 7f810fc94640 1 -- 172.21.15.16:0/1893534466 wait complete. 2023-05-22T15:47:08.724 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.711+0000 7f810fc94640 1 Processor -- start 2023-05-22T15:47:08.724 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.711+0000 7f810fc94640 1 -- start start 2023-05-22T15:47:08.725 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.715+0000 7f810fc94640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f81080b2880 0x7f81081f8130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:47:08.725 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.715+0000 7f810fc94640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f81080cc6f0 0x7f81080c5e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:47:08.725 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.715+0000 7f810fc94640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8108159cd0 0x7f8108156d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:47:08.726 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.715+0000 7f810fc94640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81080c6c90 con 0x7f81080cc6f0 2023-05-22T15:47:08.726 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.715+0000 7f810fc94640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8108154320 con 0x7f81080b2880 2023-05-22T15:47:08.726 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.715+0000 7f810fc94640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81081f7ca0 con 0x7f8108159cd0 2023-05-22T15:47:08.726 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.715+0000 7f810d22f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f81080cc6f0 0x7f81080c5e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:47:08.727 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.715+0000 7f810d22f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f81080cc6f0 0x7f81080c5e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46908/0 (socket says 172.21.15.16:46908) 2023-05-22T15:47:08.727 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.715+0000 7f810d22f640 1 -- 172.21.15.16:0/2056584205 learned_addr learned my addr 172.21.15.16:0/2056584205 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:47:08.727 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.715+0000 7f810da30640 1 --2- 172.21.15.16:0/2056584205 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f81080b2880 0x7f81081f8130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:47:08.727 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.715+0000 7f810e231640 1 --2- 172.21.15.16:0/2056584205 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8108159cd0 0x7f8108156d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:47:08.727 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.715+0000 7f810d22f640 1 -- 172.21.15.16:0/2056584205 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8108159cd0 msgr2=0x7f8108156d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:47:08.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.715+0000 7f810d22f640 1 --2- 172.21.15.16:0/2056584205 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8108159cd0 0x7f8108156d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:08.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.715+0000 7f810d22f640 1 -- 172.21.15.16:0/2056584205 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f81080b2880 msgr2=0x7f81081f8130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:47:08.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.715+0000 7f810d22f640 1 --2- 172.21.15.16:0/2056584205 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f81080b2880 0x7f81081f8130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:08.729 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.715+0000 7f810d22f640 1 -- 172.21.15.16:0/2056584205 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80f80094d0 con 0x7f81080cc6f0 2023-05-22T15:47:08.729 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.715+0000 7f810d22f640 1 --2- 172.21.15.16:0/2056584205 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f81080cc6f0 0x7f81080c5e60 secure :-1 s=READY pgs=2051 cs=0 l=1 rev1=1 crypto rx=0x7f80fc00c3c0 tx=0x7f80fc00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:47:08.729 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.715+0000 7f81067fc640 1 -- 172.21.15.16:0/2056584205 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80fc0052d0 con 0x7f81080cc6f0 2023-05-22T15:47:08.730 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.715+0000 7f81067fc640 1 -- 172.21.15.16:0/2056584205 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80fc005430 con 0x7f81080cc6f0 2023-05-22T15:47:08.730 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.715+0000 7f81067fc640 1 -- 172.21.15.16:0/2056584205 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80fc003840 con 0x7f81080cc6f0 2023-05-22T15:47:08.730 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.715+0000 7f810fc94640 1 -- 172.21.15.16:0/2056584205 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81080d0d10 con 0x7f81080cc6f0 2023-05-22T15:47:08.730 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.715+0000 7f810fc94640 1 -- 172.21.15.16:0/2056584205 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81080d0b80 con 0x7f81080cc6f0 2023-05-22T15:47:08.730 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.715+0000 7f81067fc640 1 -- 172.21.15.16:0/2056584205 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 67) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f80fc0039a0 con 0x7f81080cc6f0 2023-05-22T15:47:08.730 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.715+0000 7f81067fc640 1 --2- 172.21.15.16:0/2056584205 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f80f40654a0 0x7f80f4067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:47:08.731 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.719+0000 7f81067fc640 1 -- 172.21.15.16:0/2056584205 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(161..161 src has 1..161) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f80fc082110 con 0x7f81080cc6f0 2023-05-22T15:47:08.731 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.719+0000 7f810da30640 1 --2- 172.21.15.16:0/2056584205 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f80f40654a0 0x7f80f4067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:47:08.731 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.719+0000 7f810da30640 1 --2- 172.21.15.16:0/2056584205 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f80f40654a0 0x7f80f4067950 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f8100003fd0 tx=0x7f81000039a0 comp rx=0 tx=0).ready entity=mgr.9396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:47:08.731 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.719+0000 7f8112db6000 1 Processor -- start 2023-05-22T15:47:08.731 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.719+0000 7f8112db6000 1 -- start start 2023-05-22T15:47:08.735 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.719+0000 7f8112db6000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ea8576cfd0 0x55ea859dfc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:47:08.735 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.719+0000 7f8112db6000 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x55ea859e0170 0x55ea859e0540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:47:08.735 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.719+0000 7f8112db6000 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ea85993a10 0x55ea85993de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:47:08.735 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.719+0000 7f8112db6000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ea8585dbd0 con 0x55ea8576cfd0 2023-05-22T15:47:08.736 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.719+0000 7f8112db6000 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ea8577add0 con 0x55ea85993a10 2023-05-22T15:47:08.736 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.719+0000 7f8112db6000 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ea857eb640 con 0x55ea859e0170 2023-05-22T15:47:08.736 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.719+0000 7f810da30640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ea8576cfd0 0x55ea859dfc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:47:08.736 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.719+0000 7f810da30640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ea8576cfd0 0x55ea859dfc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46912/0 (socket says 172.21.15.16:46912) 2023-05-22T15:47:08.736 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.719+0000 7f810da30640 1 -- 172.21.15.16:0/3813933190 learned_addr learned my addr 172.21.15.16:0/3813933190 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:47:08.736 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.719+0000 7f810d22f640 1 --2- 172.21.15.16:0/3813933190 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x55ea859e0170 0x55ea859e0540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:47:08.736 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.719+0000 7f810e231640 1 --2- 172.21.15.16:0/3813933190 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ea85993a10 0x55ea85993de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:47:08.737 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.719+0000 7f810da30640 1 -- 172.21.15.16:0/3813933190 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x55ea859e0170 msgr2=0x55ea859e0540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:47:08.737 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.719+0000 7f810da30640 1 --2- 172.21.15.16:0/3813933190 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x55ea859e0170 0x55ea859e0540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:08.737 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.719+0000 7f810da30640 1 -- 172.21.15.16:0/3813933190 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ea85993a10 msgr2=0x55ea85993de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:47:08.737 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.719+0000 7f810da30640 1 --2- 172.21.15.16:0/3813933190 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ea85993a10 0x55ea85993de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:08.737 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.719+0000 7f810da30640 1 -- 172.21.15.16:0/3813933190 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ea857aa0b0 con 0x55ea8576cfd0 2023-05-22T15:47:08.737 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.719+0000 7f810e231640 1 --2- 172.21.15.16:0/3813933190 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ea85993a10 0x55ea85993de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:47:08.738 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.719+0000 7f810da30640 1 --2- 172.21.15.16:0/3813933190 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ea8576cfd0 0x55ea859dfc30 secure :-1 s=READY pgs=2052 cs=0 l=1 rev1=1 crypto rx=0x7f81000049e0 tx=0x7f8100012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df55fdb48fb26ae6 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:47:08.738 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.719+0000 7f80ef7fe640 1 -- 172.21.15.16:0/3813933190 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81000147e0 con 0x55ea8576cfd0 2023-05-22T15:47:08.738 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.719+0000 7f80ef7fe640 1 -- 172.21.15.16:0/3813933190 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8100014940 con 0x55ea8576cfd0 2023-05-22T15:47:08.738 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.719+0000 7f80ef7fe640 1 -- 172.21.15.16:0/3813933190 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8100014cb0 con 0x55ea8576cfd0 2023-05-22T15:47:08.739 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.719+0000 7f8112db6000 1 -- 172.21.15.16:0/3813933190 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ea8576cfd0 msgr2=0x55ea859dfc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:47:08.739 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.719+0000 7f8112db6000 1 --2- 172.21.15.16:0/3813933190 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ea8576cfd0 0x55ea859dfc30 secure :-1 s=READY pgs=2052 cs=0 l=1 rev1=1 crypto rx=0x7f81000049e0 tx=0x7f8100012000 comp rx=0 tx=0).stop 2023-05-22T15:47:08.739 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.719+0000 7f8112db6000 1 -- 172.21.15.16:0/3813933190 shutdown_connections 2023-05-22T15:47:08.739 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.719+0000 7f8112db6000 1 --2- 172.21.15.16:0/3813933190 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ea85993a10 0x55ea85993de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:08.740 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.719+0000 7f8112db6000 1 --2- 172.21.15.16:0/3813933190 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x55ea859e0170 0x55ea859e0540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:08.740 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.719+0000 7f8112db6000 1 --2- 172.21.15.16:0/3813933190 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ea8576cfd0 0x55ea859dfc30 unknown :-1 s=CLOSED pgs=2052 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:08.740 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.719+0000 7f8112db6000 1 -- 172.21.15.16:0/3813933190 >> 172.21.15.16:0/3813933190 conn(0x55ea8583c270 msgr2=0x55ea859c3220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:47:08.740 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.719+0000 7f8112db6000 1 -- 172.21.15.16:0/3813933190 shutdown_connections 2023-05-22T15:47:08.740 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.719+0000 7f8112db6000 1 -- 172.21.15.16:0/3813933190 wait complete. 2023-05-22T15:47:08.740 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.719+0000 7f8112db6000 1 -- start start 2023-05-22T15:47:08.741 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.719+0000 7f8112db6000 1 Processor -- start 2023-05-22T15:47:08.741 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.723+0000 7f8112db6000 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ea8576cfd0 0x55ea859e6ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:47:08.741 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.723+0000 7f8112db6000 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x55ea85993a10 0x55ea859e70e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:47:08.741 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.723+0000 7f8112db6000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ea859e0170 0x55ea859c4450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:47:08.741 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.723+0000 7f8112db6000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ea857e4780 con 0x55ea859e0170 2023-05-22T15:47:08.741 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.723+0000 7f8112db6000 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ea857a2250 con 0x55ea8576cfd0 2023-05-22T15:47:08.741 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.723+0000 7f8112db6000 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ea857a1ef0 con 0x55ea85993a10 2023-05-22T15:47:08.742 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.723+0000 7f8112db6000 10 client.0 _ll_register_callbacks cb 0x55ea85981b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-05-22T15:47:08.742 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.723+0000 7f810e231640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ea859e0170 0x55ea859c4450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:47:08.742 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.723+0000 7f810e231640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ea859e0170 0x55ea859c4450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46914/0 (socket says 172.21.15.16:46914) 2023-05-22T15:47:08.742 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.723+0000 7f810e231640 1 -- 172.21.15.16:0/967317455 learned_addr learned my addr 172.21.15.16:0/967317455 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:47:08.742 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.723+0000 7f810da30640 1 --2- 172.21.15.16:0/967317455 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ea8576cfd0 0x55ea859e6ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:47:08.742 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.723+0000 7f810d22f640 1 --2- 172.21.15.16:0/967317455 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x55ea85993a10 0x55ea859e70e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:47:08.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.723+0000 7f810e231640 1 -- 172.21.15.16:0/967317455 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x55ea85993a10 msgr2=0x55ea859e70e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:47:08.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.723+0000 7f810e231640 1 --2- 172.21.15.16:0/967317455 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x55ea85993a10 0x55ea859e70e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:08.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.723+0000 7f810e231640 1 -- 172.21.15.16:0/967317455 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ea8576cfd0 msgr2=0x55ea859e6ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:47:08.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.723+0000 7f810e231640 1 --2- 172.21.15.16:0/967317455 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ea8576cfd0 0x55ea859e6ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:08.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.723+0000 7f810e231640 1 -- 172.21.15.16:0/967317455 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81000047a0 con 0x55ea859e0170 2023-05-22T15:47:08.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.723+0000 7f810e231640 1 --2- 172.21.15.16:0/967317455 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ea859e0170 0x55ea859c4450 secure :-1 s=READY pgs=2053 cs=0 l=1 rev1=1 crypto rx=0x7f80f80190e0 tx=0x7f80f801ad10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ed78d9a17c71a94 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:47:08.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.723+0000 7f80ecff9640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2023-05-22T15:47:08.744 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.723+0000 7f80ecff9640 1 -- 172.21.15.16:0/967317455 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80f80b6470 con 0x55ea859e0170 2023-05-22T15:47:08.744 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.723+0000 7f80ecff9640 1 -- 172.21.15.16:0/967317455 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(161..161 src has 1..161) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f80f80b65d0 con 0x55ea859e0170 2023-05-22T15:47:08.744 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.723+0000 7f8112db6000 1 -- 172.21.15.16:0/967317455 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55ea857a8c60 con 0x55ea859e0170 2023-05-22T15:47:08.744 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.723+0000 7f80ecff9640 1 -- 172.21.15.16:0/967317455 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=66) v2 ==== 24+0+0 (secure 0 0 0) 0x7f80f80b6790 con 0x55ea859e0170 2023-05-22T15:47:08.744 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.723+0000 7f8112db6000 10 client.9786 fetch_fsmap learned FSMap version 66 2023-05-22T15:47:08.744 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.723+0000 7f8112db6000 1 -- 172.21.15.16:0/967317455 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=66,monmap=2+}) v3 -- 0x55ea854e1890 con 0x55ea859e0170 2023-05-22T15:47:08.744 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.723+0000 7f80ecff9640 1 -- 172.21.15.16:0/967317455 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55ea857a8c60 con 0x55ea859e0170 2023-05-22T15:47:08.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.723+0000 7f80ecff9640 1 -- 172.21.15.16:0/967317455 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 66) v1 ==== 3106+0+0 (secure 0 0 0) 0x7f80f80b68f0 con 0x55ea859e0170 2023-05-22T15:47:08.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.723+0000 7f8112db6000 10 client.9786 fetch_fsmap finished waiting for FSMap version 66 2023-05-22T15:47:08.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.723+0000 7f8112db6000 10 client.9786 resolve_mds: resolved name 'c' to daemon mds.c 2023-05-22T15:47:08.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.723+0000 7f8112db6000 20 client.9786 populate_metadata read hostname 'smithi016' 2023-05-22T15:47:08.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.723+0000 7f8112db6000 1 --2- 172.21.15.16:0/967317455 >> [v2:172.21.15.16:6836/1284822222,v1:172.21.15.16:6837/1284822222] conn(0x55ea859c51f0 0x55ea859c55c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:47:08.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.723+0000 7f8112db6000 4 client.9786 mds_command: new command op to 9654 tid=0[{"prefix": "get_command_descriptions"}] 2023-05-22T15:47:08.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.723+0000 7f8112db6000 1 -- 172.21.15.16:0/967317455 --> [v2:172.21.15.16:6836/1284822222,v1:172.21.15.16:6837/1284822222] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ea857e30d0 con 0x55ea859c51f0 2023-05-22T15:47:08.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.723+0000 7f810da30640 1 --2- 172.21.15.16:0/967317455 >> [v2:172.21.15.16:6836/1284822222,v1:172.21.15.16:6837/1284822222] conn(0x55ea859c51f0 0x55ea859c55c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:47:08.746 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.723+0000 7f810da30640 1 --2- 172.21.15.16:0/967317455 >> [v2:172.21.15.16:6836/1284822222,v1:172.21.15.16:6837/1284822222] conn(0x55ea859c51f0 0x55ea859c55c0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5f14037ab9925aae server_cookie=a25310b86d93b0e1 in_seq=0 out_seq=0 2023-05-22T15:47:08.746 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.723+0000 7f80ecff9640 10 client.9786 ms_handle_connect on v2:172.21.15.16:6836/1284822222 2023-05-22T15:47:08.746 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.723+0000 7f80ecff9640 1 -- 172.21.15.16:0/967317455 <== mds.0 v2:172.21.15.16:6836/1284822222 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x55ea857e30d0 con 0x55ea859c51f0 2023-05-22T15:47:08.746 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.723+0000 7f80ecff9640 10 client.9786 handle_command_reply: tid=0 2023-05-22T15:47:08.746 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.723+0000 7f8112db6000 1 client.9786 shutdown 2023-05-22T15:47:08.746 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.723+0000 7f8112db6000 1 -- 172.21.15.16:0/967317455 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ea859e0170 msgr2=0x55ea859c4450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:47:08.747 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.723+0000 7f8112db6000 1 --2- 172.21.15.16:0/967317455 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ea859e0170 0x55ea859c4450 secure :-1 s=READY pgs=2053 cs=0 l=1 rev1=1 crypto rx=0x7f80f80190e0 tx=0x7f80f801ad10 comp rx=0 tx=0).stop 2023-05-22T15:47:08.747 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.723+0000 7f8112db6000 1 -- 172.21.15.16:0/967317455 shutdown_connections 2023-05-22T15:47:08.747 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.723+0000 7f8112db6000 1 --2- 172.21.15.16:0/967317455 >> [v2:172.21.15.16:6836/1284822222,v1:172.21.15.16:6837/1284822222] conn(0x55ea859c51f0 0x55ea859c55c0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:08.748 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.723+0000 7f8112db6000 1 --2- 172.21.15.16:0/967317455 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ea859e0170 0x55ea859c4450 unknown :-1 s=CLOSED pgs=2053 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:08.748 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.723+0000 7f80ecff9640 0 client.9786 ms_handle_reset on v2:172.21.15.16:6836/1284822222 2023-05-22T15:47:08.748 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.723+0000 7f8112db6000 1 --2- 172.21.15.16:0/967317455 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x55ea85993a10 0x55ea859e70e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:08.748 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.723+0000 7f8112db6000 1 --2- 172.21.15.16:0/967317455 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ea8576cfd0 0x55ea859e6ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:08.748 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.723+0000 7f8112db6000 1 -- 172.21.15.16:0/967317455 >> 172.21.15.16:0/967317455 conn(0x55ea8583c270 msgr2=0x55ea85993100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:47:08.749 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.723+0000 7f8112db6000 1 -- 172.21.15.16:0/967317455 shutdown_connections 2023-05-22T15:47:08.749 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.723+0000 7f8112db6000 1 -- 172.21.15.16:0/967317455 wait complete. 2023-05-22T15:47:08.749 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.723+0000 7f8112db6000 20 client.9786 trim_cache size 0 max 16384 2023-05-22T15:47:08.753 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.739+0000 7f8112db6000 1 Processor -- start 2023-05-22T15:47:08.753 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.739+0000 7f8112db6000 1 -- start start 2023-05-22T15:47:08.753 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.739+0000 7f8112db6000 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ea8576cfd0 0x55ea85996e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:47:08.753 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.739+0000 7f8112db6000 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x55ea85993a10 0x55ea85997340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:47:08.754 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.739+0000 7f8112db6000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ea859e0170 0x55ea859bcee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:47:08.754 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.739+0000 7f8112db6000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ea854a2940 con 0x55ea859e0170 2023-05-22T15:47:08.754 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.739+0000 7f8112db6000 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ea859ecc20 con 0x55ea8576cfd0 2023-05-22T15:47:08.754 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.739+0000 7f8112db6000 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ea857e1f90 con 0x55ea85993a10 2023-05-22T15:47:08.754 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.739+0000 7f810e231640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ea859e0170 0x55ea859bcee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:47:08.754 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.739+0000 7f810e231640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ea859e0170 0x55ea859bcee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46930/0 (socket says 172.21.15.16:46930) 2023-05-22T15:47:08.755 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.739+0000 7f810e231640 1 -- 172.21.15.16:0/3686204274 learned_addr learned my addr 172.21.15.16:0/3686204274 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:47:08.755 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.739+0000 7f810da30640 1 --2- 172.21.15.16:0/3686204274 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ea8576cfd0 0x55ea85996e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:47:08.755 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.739+0000 7f810e231640 1 -- 172.21.15.16:0/3686204274 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x55ea85993a10 msgr2=0x55ea85997340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:47:08.755 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.739+0000 7f810d22f640 1 --2- 172.21.15.16:0/3686204274 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x55ea85993a10 0x55ea85997340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:47:08.755 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.739+0000 7f810e231640 1 --2- 172.21.15.16:0/3686204274 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x55ea85993a10 0x55ea85997340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:08.755 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.739+0000 7f810e231640 1 -- 172.21.15.16:0/3686204274 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ea8576cfd0 msgr2=0x55ea85996e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:47:08.756 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.739+0000 7f810e231640 1 --2- 172.21.15.16:0/3686204274 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ea8576cfd0 0x55ea85996e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:08.756 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.739+0000 7f810e231640 1 -- 172.21.15.16:0/3686204274 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ea859dfcb0 con 0x55ea859e0170 2023-05-22T15:47:08.756 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.739+0000 7f810e231640 1 --2- 172.21.15.16:0/3686204274 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ea859e0170 0x55ea859bcee0 secure :-1 s=READY pgs=2054 cs=0 l=1 rev1=1 crypto rx=0x7f80f801ad50 tx=0x7f80f80b6000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91f68f42dc9713a2 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:47:08.756 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.739+0000 7f80effff640 1 -- 172.21.15.16:0/3686204274 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80f8004040 con 0x55ea859e0170 2023-05-22T15:47:08.756 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.739+0000 7f80effff640 1 -- 172.21.15.16:0/3686204274 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80f80041a0 con 0x55ea859e0170 2023-05-22T15:47:08.756 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.739+0000 7f80effff640 1 -- 172.21.15.16:0/3686204274 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80f8004530 con 0x55ea859e0170 2023-05-22T15:47:08.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.743+0000 7f8112db6000 1 -- 172.21.15.16:0/3686204274 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ea859e0170 msgr2=0x55ea859bcee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:47:08.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.743+0000 7f8112db6000 1 --2- 172.21.15.16:0/3686204274 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ea859e0170 0x55ea859bcee0 secure :-1 s=READY pgs=2054 cs=0 l=1 rev1=1 crypto rx=0x7f80f801ad50 tx=0x7f80f80b6000 comp rx=0 tx=0).stop 2023-05-22T15:47:08.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.743+0000 7f8112db6000 1 -- 172.21.15.16:0/3686204274 shutdown_connections 2023-05-22T15:47:08.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.743+0000 7f8112db6000 1 --2- 172.21.15.16:0/3686204274 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ea859e0170 0x55ea859bcee0 unknown :-1 s=CLOSED pgs=2054 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:08.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.743+0000 7f8112db6000 1 --2- 172.21.15.16:0/3686204274 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x55ea85993a10 0x55ea85997340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:08.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.743+0000 7f8112db6000 1 --2- 172.21.15.16:0/3686204274 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ea8576cfd0 0x55ea85996e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:08.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.743+0000 7f8112db6000 1 -- 172.21.15.16:0/3686204274 >> 172.21.15.16:0/3686204274 conn(0x55ea8583c270 msgr2=0x55ea85993080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:47:08.758 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.743+0000 7f8112db6000 1 -- 172.21.15.16:0/3686204274 shutdown_connections 2023-05-22T15:47:08.758 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.743+0000 7f8112db6000 1 -- 172.21.15.16:0/3686204274 wait complete. 2023-05-22T15:47:08.758 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.743+0000 7f8112db6000 1 -- start start 2023-05-22T15:47:08.758 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.743+0000 7f8112db6000 1 Processor -- start 2023-05-22T15:47:08.758 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.743+0000 7f8112db6000 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ea8576cfd0 0x55ea85a3c530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:47:08.759 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.743+0000 7f8112db6000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ea85993a10 0x55ea85a3ca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:47:08.759 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.743+0000 7f8112db6000 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x55ea859e0170 0x55ea85a38ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:47:08.759 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.743+0000 7f8112db6000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ea857e72e0 con 0x55ea85993a10 2023-05-22T15:47:08.759 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.743+0000 7f8112db6000 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ea85999600 con 0x55ea8576cfd0 2023-05-22T15:47:08.760 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.743+0000 7f8112db6000 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ea857ec8d0 con 0x55ea859e0170 2023-05-22T15:47:08.760 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.743+0000 7f8112db6000 10 client.0 _ll_register_callbacks cb 0x55ea85834eb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-05-22T15:47:08.760 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.743+0000 7f810d22f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ea85993a10 0x55ea85a3ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:47:08.761 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.743+0000 7f810d22f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ea85993a10 0x55ea85a3ca70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46936/0 (socket says 172.21.15.16:46936) 2023-05-22T15:47:08.761 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.743+0000 7f810d22f640 1 -- 172.21.15.16:0/3337198258 learned_addr learned my addr 172.21.15.16:0/3337198258 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:47:08.761 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.743+0000 7f810da30640 1 --2- 172.21.15.16:0/3337198258 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ea8576cfd0 0x55ea85a3c530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:47:08.761 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.743+0000 7f810e231640 1 --2- 172.21.15.16:0/3337198258 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x55ea859e0170 0x55ea85a38ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:47:08.762 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.743+0000 7f810d22f640 1 -- 172.21.15.16:0/3337198258 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x55ea859e0170 msgr2=0x55ea85a38ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:47:08.762 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.743+0000 7f810d22f640 1 --2- 172.21.15.16:0/3337198258 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x55ea859e0170 0x55ea85a38ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:08.762 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.743+0000 7f810d22f640 1 -- 172.21.15.16:0/3337198258 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ea8576cfd0 msgr2=0x55ea85a3c530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:47:08.763 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.743+0000 7f810d22f640 1 --2- 172.21.15.16:0/3337198258 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ea8576cfd0 0x55ea85a3c530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:08.763 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.743+0000 7f810d22f640 1 -- 172.21.15.16:0/3337198258 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ea854e1890 con 0x55ea85993a10 2023-05-22T15:47:08.763 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.743+0000 7f810e231640 1 --2- 172.21.15.16:0/3337198258 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x55ea859e0170 0x55ea85a38ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:47:08.764 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.743+0000 7f810da30640 1 --2- 172.21.15.16:0/3337198258 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ea8576cfd0 0x55ea85a3c530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:47:08.764 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.743+0000 7f810d22f640 1 --2- 172.21.15.16:0/3337198258 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ea85993a10 0x55ea85a3ca70 secure :-1 s=READY pgs=2055 cs=0 l=1 rev1=1 crypto rx=0x7f80fc05b2a0 tx=0x7f80fc053130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7476c2cdf0057941 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:47:08.764 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.743+0000 7f80ecff9640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2023-05-22T15:47:08.764 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.743+0000 7f80ecff9640 1 -- 172.21.15.16:0/3337198258 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80fc05b540 con 0x55ea85993a10 2023-05-22T15:47:08.765 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.743+0000 7f80ecff9640 1 -- 172.21.15.16:0/3337198258 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(161..161 src has 1..161) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f80fc01ddb0 con 0x55ea85993a10 2023-05-22T15:47:08.765 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.743+0000 7f8112db6000 1 -- 172.21.15.16:0/3337198258 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55ea8599a160 con 0x55ea85993a10 2023-05-22T15:47:08.765 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.743+0000 7f80ecff9640 1 -- 172.21.15.16:0/3337198258 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=66) v2 ==== 24+0+0 (secure 0 0 0) 0x7f80fc057150 con 0x55ea85993a10 2023-05-22T15:47:08.766 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.743+0000 7f8112db6000 10 client.9792 fetch_fsmap learned FSMap version 66 2023-05-22T15:47:08.766 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.743+0000 7f8112db6000 1 -- 172.21.15.16:0/3337198258 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=66,monmap=2+}) v3 -- 0x55ea85790280 con 0x55ea85993a10 2023-05-22T15:47:08.766 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.743+0000 7f80ecff9640 1 -- 172.21.15.16:0/3337198258 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80fc07e070 con 0x55ea85993a10 2023-05-22T15:47:08.766 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.743+0000 7f80ecff9640 1 -- 172.21.15.16:0/3337198258 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 66) v1 ==== 3106+0+0 (secure 0 0 0) 0x7f80fc0572b0 con 0x55ea85993a10 2023-05-22T15:47:08.767 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.743+0000 7f8112db6000 10 client.9792 fetch_fsmap finished waiting for FSMap version 66 2023-05-22T15:47:08.767 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.743+0000 7f8112db6000 10 client.9792 resolve_mds: resolved name 'c' to daemon mds.c 2023-05-22T15:47:08.767 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.743+0000 7f8112db6000 20 client.9792 populate_metadata read hostname 'smithi016' 2023-05-22T15:47:08.767 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.743+0000 7f8112db6000 1 --2- 172.21.15.16:0/3337198258 >> [v2:172.21.15.16:6836/1284822222,v1:172.21.15.16:6837/1284822222] conn(0x55ea85a43400 0x55ea85a457e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:47:08.767 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.743+0000 7f8112db6000 4 client.9792 mds_command: new command op to 9654 tid=0[{"prefix": "status"}] 2023-05-22T15:47:08.768 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.743+0000 7f8112db6000 1 -- 172.21.15.16:0/3337198258 --> [v2:172.21.15.16:6836/1284822222,v1:172.21.15.16:6837/1284822222] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55ea8578b4d0 con 0x55ea85a43400 2023-05-22T15:47:08.768 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.743+0000 7f810da30640 1 --2- 172.21.15.16:0/3337198258 >> [v2:172.21.15.16:6836/1284822222,v1:172.21.15.16:6837/1284822222] conn(0x55ea85a43400 0x55ea85a457e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:47:08.768 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.743+0000 7f80ecff9640 10 client.9792 ms_handle_connect on v2:172.21.15.16:6836/1284822222 2023-05-22T15:47:08.769 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.743+0000 7f810da30640 1 --2- 172.21.15.16:0/3337198258 >> [v2:172.21.15.16:6836/1284822222,v1:172.21.15.16:6837/1284822222] conn(0x55ea85a43400 0x55ea85a457e0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=708162913ad86c8f server_cookie=8cad57c8de0996f2 in_seq=0 out_seq=0 2023-05-22T15:47:08.769 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.747+0000 7f80ecff9640 1 -- 172.21.15.16:0/3337198258 <== mds.0 v2:172.21.15.16:6836/1284822222 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+333 (crc 0 0 0) 0x55ea8578b4d0 con 0x55ea85a43400 2023-05-22T15:47:08.769 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.747+0000 7f80ecff9640 10 client.9792 handle_command_reply: tid=0 2023-05-22T15:47:08.769 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.747+0000 7f8112db6000 1 client.9792 shutdown 2023-05-22T15:47:08.770 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.747+0000 7f8112db6000 1 -- 172.21.15.16:0/3337198258 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ea85993a10 msgr2=0x55ea85a3ca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:47:08.770 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.747+0000 7f8112db6000 1 --2- 172.21.15.16:0/3337198258 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ea85993a10 0x55ea85a3ca70 secure :-1 s=READY pgs=2055 cs=0 l=1 rev1=1 crypto rx=0x7f80fc05b2a0 tx=0x7f80fc053130 comp rx=0 tx=0).stop 2023-05-22T15:47:08.770 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.747+0000 7f8112db6000 1 -- 172.21.15.16:0/3337198258 shutdown_connections 2023-05-22T15:47:08.770 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.747+0000 7f8112db6000 1 --2- 172.21.15.16:0/3337198258 >> [v2:172.21.15.16:6836/1284822222,v1:172.21.15.16:6837/1284822222] conn(0x55ea85a43400 0x55ea85a457e0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:08.770 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.747+0000 7f8112db6000 1 --2- 172.21.15.16:0/3337198258 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x55ea859e0170 0x55ea85a38ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:08.770 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.747+0000 7f8112db6000 1 --2- 172.21.15.16:0/3337198258 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ea85993a10 0x55ea85a3ca70 unknown :-1 s=CLOSED pgs=2055 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:08.770 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.747+0000 7f8112db6000 1 --2- 172.21.15.16:0/3337198258 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ea8576cfd0 0x55ea85a3c530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:08.771 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.747+0000 7f80ecff9640 0 client.9792 ms_handle_reset on v2:172.21.15.16:6836/1284822222 2023-05-22T15:47:08.771 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.747+0000 7f8112db6000 1 -- 172.21.15.16:0/3337198258 >> 172.21.15.16:0/3337198258 conn(0x55ea8583c270 msgr2=0x55ea85995df0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:47:08.771 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.747+0000 7f8112db6000 1 -- 172.21.15.16:0/3337198258 shutdown_connections 2023-05-22T15:47:08.771 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.747+0000 7f8112db6000 1 -- 172.21.15.16:0/3337198258 wait complete. 2023-05-22T15:47:08.771 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.747+0000 7f8112db6000 20 client.9792 trim_cache size 0 max 16384 2023-05-22T15:47:08.771 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-05-22T15:47:08.772 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_fsid": "b2719834-7847-4d98-90a6-f65d57f85c49", 2023-05-22T15:47:08.772 INFO:teuthology.orchestra.run.smithi016.stdout: "whoami": 0, 2023-05-22T15:47:08.772 INFO:teuthology.orchestra.run.smithi016.stdout: "id": 9654, 2023-05-22T15:47:08.772 INFO:teuthology.orchestra.run.smithi016.stdout: "want_state": "up:active", 2023-05-22T15:47:08.772 INFO:teuthology.orchestra.run.smithi016.stdout: "state": "up:active", 2023-05-22T15:47:08.772 INFO:teuthology.orchestra.run.smithi016.stdout: "fs_name": "cephfs", 2023-05-22T15:47:08.773 INFO:teuthology.orchestra.run.smithi016.stdout: "rank_uptime": 5.7360846780000001, 2023-05-22T15:47:08.773 INFO:teuthology.orchestra.run.smithi016.stdout: "mdsmap_epoch": 66, 2023-05-22T15:47:08.773 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch": 161, 2023-05-22T15:47:08.773 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch_barrier": 161, 2023-05-22T15:47:08.773 INFO:teuthology.orchestra.run.smithi016.stdout: "uptime": 10.446462584000001 2023-05-22T15:47:08.773 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-05-22T15:47:08.774 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.747+0000 7f810fc94640 1 -- 172.21.15.16:0/2056584205 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f80f40654a0 msgr2=0x7f80f4067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:47:08.774 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.747+0000 7f810fc94640 1 --2- 172.21.15.16:0/2056584205 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f80f40654a0 0x7f80f4067950 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f8100003fd0 tx=0x7f81000039a0 comp rx=0 tx=0).stop 2023-05-22T15:47:08.775 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.747+0000 7f810fc94640 1 -- 172.21.15.16:0/2056584205 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f81080cc6f0 msgr2=0x7f81080c5e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:47:08.775 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.747+0000 7f810fc94640 1 --2- 172.21.15.16:0/2056584205 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f81080cc6f0 0x7f81080c5e60 secure :-1 s=READY pgs=2051 cs=0 l=1 rev1=1 crypto rx=0x7f80fc00c3c0 tx=0x7f80fc00c880 comp rx=0 tx=0).stop 2023-05-22T15:47:08.775 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.747+0000 7f810fc94640 1 -- 172.21.15.16:0/2056584205 shutdown_connections 2023-05-22T15:47:08.775 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.747+0000 7f810fc94640 1 --2- 172.21.15.16:0/2056584205 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f80f40654a0 0x7f80f4067950 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:08.775 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.747+0000 7f810fc94640 1 --2- 172.21.15.16:0/2056584205 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8108159cd0 0x7f8108156d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:08.775 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.747+0000 7f810fc94640 1 --2- 172.21.15.16:0/2056584205 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f81080cc6f0 0x7f81080c5e60 unknown :-1 s=CLOSED pgs=2051 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:08.775 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.747+0000 7f810fc94640 1 --2- 172.21.15.16:0/2056584205 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f81080b2880 0x7f81081f8130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:08.776 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.747+0000 7f810fc94640 1 -- 172.21.15.16:0/2056584205 >> 172.21.15.16:0/2056584205 conn(0x7f810806f5c0 msgr2=0x7f81080c9340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:47:08.776 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.747+0000 7f810fc94640 1 -- 172.21.15.16:0/2056584205 shutdown_connections 2023-05-22T15:47:08.776 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.747+0000 7f810fc94640 1 -- 172.21.15.16:0/2056584205 wait complete. 2023-05-22T15:47:08.776 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-05-22T15:47:08.891 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.883+0000 7f630ec5a640 1 Processor -- start 2023-05-22T15:47:08.891 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.883+0000 7f630ec5a640 1 -- start start 2023-05-22T15:47:08.892 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.887+0000 7f630ec5a640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f630815fc50 0x7f6308160020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:47:08.893 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.887+0000 7f630ec5a640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f630815c920 0x7f63080d0030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:47:08.893 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.887+0000 7f630ec5a640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f630815c490 con 0x7f630815cd70 2023-05-22T15:47:08.893 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.887+0000 7f630ec5a640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63080cf7a0 con 0x7f630815c920 2023-05-22T15:47:08.893 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.887+0000 7f630ec5a640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f630815c140 con 0x7f630815fc50 2023-05-22T15:47:08.893 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.887+0000 7f6307fff640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f630815c920 0x7f63080d0030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:47:08.893 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.887+0000 7f6307fff640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f630815c920 0x7f63080d0030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46952/0 (socket says 172.21.15.16:46952) 2023-05-22T15:47:08.893 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.887+0000 7f6307fff640 1 -- 172.21.15.16:0/775486251 learned_addr learned my addr 172.21.15.16:0/775486251 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:47:08.894 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.887+0000 7f630c9f6640 1 --2- 172.21.15.16:0/775486251 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f630815fc50 0x7f6308160020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:47:08.894 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.887+0000 7f6307fff640 1 -- 172.21.15.16:0/775486251 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f630815fc50 msgr2=0x7f6308160020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:47:08.894 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.887+0000 7f6307fff640 1 --2- 172.21.15.16:0/775486251 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f630815fc50 0x7f6308160020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:08.894 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.887+0000 7f6307fff640 1 -- 172.21.15.16:0/775486251 >> v1:172.21.15.62:6789/0 conn(0x7f630815cd70 legacy=0x7f63080b3ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:47:08.894 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.887+0000 7f6307fff640 1 -- 172.21.15.16:0/775486251 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63080ccb50 con 0x7f630815c920 2023-05-22T15:47:08.895 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.887+0000 7f630c9f6640 1 --2- 172.21.15.16:0/775486251 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f630815fc50 0x7f6308160020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:47:08.896 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.887+0000 7f6307fff640 1 --2- 172.21.15.16:0/775486251 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f630815c920 0x7f63080d0030 secure :-1 s=READY pgs=2056 cs=0 l=1 rev1=1 crypto rx=0x7f62f4009870 tx=0x7f62f401c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82e6ba5be81102c3 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:47:08.897 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.887+0000 7f63077fe640 1 -- 172.21.15.16:0/775486251 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62f40aa5f0 con 0x7f630815c920 2023-05-22T15:47:08.897 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.887+0000 7f63077fe640 1 -- 172.21.15.16:0/775486251 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62f40aa750 con 0x7f630815c920 2023-05-22T15:47:08.898 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.887+0000 7f63077fe640 1 -- 172.21.15.16:0/775486251 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62f40aaae0 con 0x7f630815c920 2023-05-22T15:47:08.898 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.891+0000 7f630ec5a640 1 -- 172.21.15.16:0/775486251 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f630815c920 msgr2=0x7f63080d0030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:47:08.898 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.891+0000 7f630ec5a640 1 --2- 172.21.15.16:0/775486251 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f630815c920 0x7f63080d0030 secure :-1 s=READY pgs=2056 cs=0 l=1 rev1=1 crypto rx=0x7f62f4009870 tx=0x7f62f401c4d0 comp rx=0 tx=0).stop 2023-05-22T15:47:08.899 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.891+0000 7f630ec5a640 1 -- 172.21.15.16:0/775486251 shutdown_connections 2023-05-22T15:47:08.899 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.891+0000 7f630ec5a640 1 --2- 172.21.15.16:0/775486251 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f630815c920 0x7f63080d0030 unknown :-1 s=CLOSED pgs=2056 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:08.899 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.891+0000 7f630ec5a640 1 --2- 172.21.15.16:0/775486251 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f630815fc50 0x7f6308160020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:08.899 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.891+0000 7f630ec5a640 1 -- 172.21.15.16:0/775486251 >> 172.21.15.16:0/775486251 conn(0x7f63080657f0 msgr2=0x7f6308070e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:47:08.900 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.891+0000 7f630ec5a640 1 -- 172.21.15.16:0/775486251 shutdown_connections 2023-05-22T15:47:08.900 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.891+0000 7f630ec5a640 1 -- 172.21.15.16:0/775486251 wait complete. 2023-05-22T15:47:08.900 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.891+0000 7f630ec5a640 1 Processor -- start 2023-05-22T15:47:08.901 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.891+0000 7f630ec5a640 1 -- start start 2023-05-22T15:47:08.905 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.891+0000 7f630ec5a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f630815c920 0x7f63080c56a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:47:08.906 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.891+0000 7f630ec5a640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f630815cd70 0x7f63080b5290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:47:08.906 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.891+0000 7f630ec5a640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f630815fc50 0x7f63080c4660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:47:08.906 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.891+0000 7f630ec5a640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f630814c990 con 0x7f630815c920 2023-05-22T15:47:08.906 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.891+0000 7f630ec5a640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63080b5900 con 0x7f630815cd70 2023-05-22T15:47:08.907 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.891+0000 7f630ec5a640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63080b7cf0 con 0x7f630815fc50 2023-05-22T15:47:08.907 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.891+0000 7f630c9f6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f630815c920 0x7f63080c56a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:47:08.907 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.891+0000 7f630c9f6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f630815c920 0x7f63080c56a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46966/0 (socket says 172.21.15.16:46966) 2023-05-22T15:47:08.907 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.891+0000 7f630c9f6640 1 -- 172.21.15.16:0/2123045686 learned_addr learned my addr 172.21.15.16:0/2123045686 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:47:08.908 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.891+0000 7f6307fff640 1 --2- 172.21.15.16:0/2123045686 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f630815cd70 0x7f63080b5290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:47:08.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.891+0000 7f630d1f7640 1 --2- 172.21.15.16:0/2123045686 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f630815fc50 0x7f63080c4660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:47:08.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.891+0000 7f630c9f6640 1 -- 172.21.15.16:0/2123045686 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f630815fc50 msgr2=0x7f63080c4660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:47:08.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.891+0000 7f630c9f6640 1 --2- 172.21.15.16:0/2123045686 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f630815fc50 0x7f63080c4660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:08.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.891+0000 7f630c9f6640 1 -- 172.21.15.16:0/2123045686 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f630815cd70 msgr2=0x7f63080b5290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:47:08.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.891+0000 7f630c9f6640 1 --2- 172.21.15.16:0/2123045686 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f630815cd70 0x7f63080b5290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:08.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.891+0000 7f630c9f6640 1 -- 172.21.15.16:0/2123045686 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62f40094d0 con 0x7f630815c920 2023-05-22T15:47:08.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.891+0000 7f630d1f7640 1 --2- 172.21.15.16:0/2123045686 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f630815fc50 0x7f63080c4660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:47:08.911 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.891+0000 7f630c9f6640 1 --2- 172.21.15.16:0/2123045686 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f630815c920 0x7f63080c56a0 secure :-1 s=READY pgs=2057 cs=0 l=1 rev1=1 crypto rx=0x7f62fc00c7f0 tx=0x7f62fc00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:47:08.911 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.891+0000 7f63057fa640 1 -- 172.21.15.16:0/2123045686 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62fc007820 con 0x7f630815c920 2023-05-22T15:47:08.911 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.891+0000 7f63057fa640 1 -- 172.21.15.16:0/2123045686 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62fc007980 con 0x7f630815c920 2023-05-22T15:47:08.911 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.891+0000 7f63057fa640 1 -- 172.21.15.16:0/2123045686 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62fc007d30 con 0x7f630815c920 2023-05-22T15:47:08.911 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.891+0000 7f630ec5a640 1 -- 172.21.15.16:0/2123045686 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6308160560 con 0x7f630815c920 2023-05-22T15:47:08.912 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.891+0000 7f630ec5a640 1 -- 172.21.15.16:0/2123045686 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63080d0980 con 0x7f630815c920 2023-05-22T15:47:08.912 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.895+0000 7f630ec5a640 1 -- 172.21.15.16:0/2123045686 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6308168ad0 con 0x7f630815c920 2023-05-22T15:47:08.912 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.895+0000 7f63057fa640 1 -- 172.21.15.16:0/2123045686 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 67) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f62fc024480 con 0x7f630815c920 2023-05-22T15:47:08.912 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.895+0000 7f63057fa640 1 --2- 172.21.15.16:0/2123045686 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f62ec0654a0 0x7f62ec067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:47:08.913 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.895+0000 7f63057fa640 1 -- 172.21.15.16:0/2123045686 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(161..161 src has 1..161) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f62fc083330 con 0x7f630815c920 2023-05-22T15:47:08.913 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.895+0000 7f6307fff640 1 --2- 172.21.15.16:0/2123045686 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f62ec0654a0 0x7f62ec067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:47:08.913 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.895+0000 7f6307fff640 1 --2- 172.21.15.16:0/2123045686 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f62ec0654a0 0x7f62ec067950 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f62f40ae000 tx=0x7f62f40a7000 comp rx=0 tx=0).ready entity=mgr.9396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:47:08.913 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:08.899+0000 7f63057fa640 1 -- 172.21.15.16:0/2123045686 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f62fc050d40 con 0x7f630815c920 2023-05-22T15:47:09.091 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:09.087+0000 7f630ec5a640 1 -- 172.21.15.16:0/2123045686 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f63080b8210 con 0x7f630815c920 2023-05-22T15:47:09.092 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:09.087+0000 7f63057fa640 1 -- 172.21.15.16:0/2123045686 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v66) v1 ==== 53+0+73 (secure 0 0 0) 0x7f62fc054360 con 0x7f630815c920 2023-05-22T15:47:09.092 INFO:teuthology.orchestra.run.smithi016.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-05-22T15:47:09.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:09.087+0000 7f630ec5a640 1 -- 172.21.15.16:0/2123045686 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f62ec0654a0 msgr2=0x7f62ec067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:47:09.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:09.087+0000 7f630ec5a640 1 --2- 172.21.15.16:0/2123045686 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f62ec0654a0 0x7f62ec067950 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f62f40ae000 tx=0x7f62f40a7000 comp rx=0 tx=0).stop 2023-05-22T15:47:09.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:09.087+0000 7f630ec5a640 1 -- 172.21.15.16:0/2123045686 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f630815c920 msgr2=0x7f63080c56a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:47:09.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:09.087+0000 7f630ec5a640 1 --2- 172.21.15.16:0/2123045686 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f630815c920 0x7f63080c56a0 secure :-1 s=READY pgs=2057 cs=0 l=1 rev1=1 crypto rx=0x7f62fc00c7f0 tx=0x7f62fc00ccb0 comp rx=0 tx=0).stop 2023-05-22T15:47:09.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:09.087+0000 7f630ec5a640 1 -- 172.21.15.16:0/2123045686 shutdown_connections 2023-05-22T15:47:09.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:09.087+0000 7f630ec5a640 1 --2- 172.21.15.16:0/2123045686 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f62ec0654a0 0x7f62ec067950 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:09.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:09.087+0000 7f630ec5a640 1 --2- 172.21.15.16:0/2123045686 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f630815fc50 0x7f63080c4660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:09.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:09.087+0000 7f630ec5a640 1 --2- 172.21.15.16:0/2123045686 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f630815cd70 0x7f63080b5290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:09.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:09.087+0000 7f630ec5a640 1 --2- 172.21.15.16:0/2123045686 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f630815c920 0x7f63080c56a0 unknown :-1 s=CLOSED pgs=2057 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:09.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:09.087+0000 7f630ec5a640 1 -- 172.21.15.16:0/2123045686 >> 172.21.15.16:0/2123045686 conn(0x7f63080657f0 msgr2=0x7f63080ceee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:47:09.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:09.087+0000 7f630ec5a640 1 -- 172.21.15.16:0/2123045686 shutdown_connections 2023-05-22T15:47:09.096 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:09.087+0000 7f630ec5a640 1 -- 172.21.15.16:0/2123045686 wait complete. 2023-05-22T15:47:09.110 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-05-22T15:47:09.110 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-05-22T15:47:09.111 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi016.front.sepia.ceph.com 2023-05-22T15:47:09.111 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-05-22T15:47:09.111 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-05-22T15:47:09.111 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-05-22T15:47:09.112 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-05-22T15:47:09.112 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-05-22T15:47:09.112 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-05-22T15:47:09.113 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:47:09.113 DEBUG:teuthology.orchestra.run.smithi016:> ip addr 2023-05-22T15:47:09.119 INFO:teuthology.orchestra.run.smithi016.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-05-22T15:47:09.120 INFO:teuthology.orchestra.run.smithi016.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-05-22T15:47:09.120 INFO:teuthology.orchestra.run.smithi016.stdout: inet 127.0.0.1/8 scope host lo 2023-05-22T15:47:09.120 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-05-22T15:47:09.120 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 ::1/128 scope host 2023-05-22T15:47:09.120 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-05-22T15:47:09.120 INFO:teuthology.orchestra.run.smithi016.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-05-22T15:47:09.121 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:6b:fd:e0 brd ff:ff:ff:ff:ff:ff 2023-05-22T15:47:09.121 INFO:teuthology.orchestra.run.smithi016.stdout: altname enp7s0f0 2023-05-22T15:47:09.121 INFO:teuthology.orchestra.run.smithi016.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-05-22T15:47:09.121 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:6b:fd:e1 brd ff:ff:ff:ff:ff:ff 2023-05-22T15:47:09.121 INFO:teuthology.orchestra.run.smithi016.stdout: altname enp7s0f1 2023-05-22T15:47:09.121 INFO:teuthology.orchestra.run.smithi016.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2023-05-22T15:47:09.121 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:bd:15:64 brd ff:ff:ff:ff:ff:ff 2023-05-22T15:47:09.121 INFO:teuthology.orchestra.run.smithi016.stdout: altname enp1s0f0 2023-05-22T15:47:09.122 INFO:teuthology.orchestra.run.smithi016.stdout: inet 172.21.15.16/20 brd 172.21.15.255 scope global dynamic ens1f0 2023-05-22T15:47:09.122 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft 41287sec preferred_lft 41287sec 2023-05-22T15:47:09.122 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::ec4:7aff:febd:1564/64 scope link 2023-05-22T15:47:09.122 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-05-22T15:47:09.122 INFO:teuthology.orchestra.run.smithi016.stdout:5: ens1f1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-05-22T15:47:09.122 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:bd:15:65 brd ff:ff:ff:ff:ff:ff 2023-05-22T15:47:09.122 INFO:teuthology.orchestra.run.smithi016.stdout: altname enp1s0f1 2023-05-22T15:47:09.122 INFO:teuthology.orchestra.run.smithi016.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2023-05-22T15:47:09.123 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 02:42:1f:b8:98:0a brd ff:ff:ff:ff:ff:ff 2023-05-22T15:47:09.123 INFO:teuthology.orchestra.run.smithi016.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2023-05-22T15:47:09.123 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-05-22T15:47:09.123 INFO:teuthology.orchestra.run.smithi016.stdout:9: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-05-22T15:47:09.123 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 5e:1f:84:6d:79:ae brd ff:ff:ff:ff:ff:ff 2023-05-22T15:47:09.123 INFO:teuthology.orchestra.run.smithi016.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-05-22T15:47:09.123 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-05-22T15:47:09.123 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::5c1f:84ff:fe6d:79ae/64 scope link 2023-05-22T15:47:09.124 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-05-22T15:47:09.124 INFO:teuthology.orchestra.run.smithi016.stdout:10: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-05-22T15:47:09.124 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0e:5c:f6:1f:82:2f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-05-22T15:47:09.124 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::c5c:f6ff:fe1f:822f/64 scope link 2023-05-22T15:47:09.124 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-05-22T15:47:09.124 INFO:teuthology.orchestra.run.smithi016.stdout:11: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-05-22T15:47:09.124 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether c6:a6:3d:69:d0:db brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-05-22T15:47:09.125 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::c4a6:3dff:fe69:d0db/64 scope link 2023-05-22T15:47:09.125 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-05-22T15:47:09.125 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:47:09.125 DEBUG:teuthology.orchestra.run.smithi016:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-05-22T15:47:09.173 INFO:teuthology.orchestra.run.smithi016.stdout:1 2023-05-22T15:47:09.174 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:47:09.175 DEBUG:teuthology.orchestra.run.smithi016:> route 2023-05-22T15:47:09.225 INFO:teuthology.orchestra.run.smithi016.stdout:Kernel IP routing table 2023-05-22T15:47:09.225 INFO:teuthology.orchestra.run.smithi016.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-05-22T15:47:09.225 INFO:teuthology.orchestra.run.smithi016.stdout:default _gateway 0.0.0.0 UG 0 0 0 ens1f0 2023-05-22T15:47:09.225 INFO:teuthology.orchestra.run.smithi016.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2023-05-22T15:47:09.225 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 0 0 0 ens1f0 2023-05-22T15:47:09.226 INFO:teuthology.orchestra.run.smithi016.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-05-22T15:47:09.226 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:47:09.226 DEBUG:teuthology.orchestra.run.smithi016:> (cd / && exec bash -c ' 2023-05-22T15:47:09.227 DEBUG:teuthology.orchestra.run.smithi016:> set -e 2023-05-22T15:47:09.227 DEBUG:teuthology.orchestra.run.smithi016:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2023-05-22T15:47:09.227 DEBUG:teuthology.orchestra.run.smithi016:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2023-05-22T15:47:09.227 DEBUG:teuthology.orchestra.run.smithi016:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2023-05-22T15:47:09.228 DEBUG:teuthology.orchestra.run.smithi016:> ') 2023-05-22T15:47:09.304 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:47:09.305 DEBUG:teuthology.orchestra.run.smithi016:> ip netns list 2023-05-22T15:47:09.356 INFO:teuthology.orchestra.run.smithi016.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-05-22T15:47:09.356 INFO:teuthology.orchestra.run.smithi016.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-05-22T15:47:09.356 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-05-22T15:47:09.357 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:47:09.357 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip link set brx.0 up 2023-05-22T15:47:09.413 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2023-05-22T15:47:09.414 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-05-22T15:47:09.414 DEBUG:teuthology.orchestra.run.smithi016:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-05-22T15:47:09.464 INFO:teuthology.orchestra.run.smithi016.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-05-22T15:47:09.465 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-05-22T15:47:09.466 DEBUG:teuthology.orchestra.run.smithi016:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-05-22T15:47:09.517 DEBUG:teuthology.orchestra.run.smithi016:> sudo modprobe fuse 2023-05-22T15:47:09.573 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-05-22T15:47:09.573 DEBUG:teuthology.orchestra.run.smithi016:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-05-22T15:47:09.627 INFO:teuthology.orchestra.run.smithi016.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-05-22T15:47:09.629 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-05-22T15:47:09.630 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:47:09.630 DEBUG:teuthology.orchestra.run.smithi016:> ls /sys/fs/fuse/connections 2023-05-22T15:47:09.680 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2023-05-22T15:47:09.681 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2023-05-22T15:47:09.729 DEBUG:teuthology.orchestra.run.smithi016:> sudo modprobe fuse 2023-05-22T15:47:09.745 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-05-22T15:47:09.746 DEBUG:teuthology.orchestra.run.smithi016:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-05-22T15:47:09.800 INFO:teuthology.orchestra.run.smithi016.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-05-22T15:47:09.801 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-05-22T15:47:09.802 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:47:09.802 DEBUG:teuthology.orchestra.run.smithi016:> ls /sys/fs/fuse/connections 2023-05-22T15:47:09.812 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi016.stderr:2023-05-22T15:47:09.807+0000 7f8f2b7c5580 -1 init, newargv = 0x563358df23a0 newargc=15 2023-05-22T15:47:09.813 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi016.stderr:ceph-fuse[65370]: starting ceph client 2023-05-22T15:47:09.817 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi016.stderr:ceph-fuse[65370]: starting fuse 2023-05-22T15:47:10.918 DEBUG:teuthology.orchestra.run.smithi016:> sudo modprobe fuse 2023-05-22T15:47:10.938 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-05-22T15:47:10.938 DEBUG:teuthology.orchestra.run.smithi016:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-05-22T15:47:10.992 INFO:teuthology.orchestra.run.smithi016.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-05-22T15:47:10.993 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-05-22T15:47:10.993 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:47:10.994 DEBUG:teuthology.orchestra.run.smithi016:> ls /sys/fs/fuse/connections 2023-05-22T15:47:11.040 INFO:teuthology.orchestra.run.smithi016.stdout:45 2023-05-22T15:47:11.041 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [45] 2023-05-22T15:47:11.041 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2023-05-22T15:47:11.041 DEBUG:teuthology.orchestra.run.smithi016:> import glob 2023-05-22T15:47:11.042 DEBUG:teuthology.orchestra.run.smithi016:> import re 2023-05-22T15:47:11.042 DEBUG:teuthology.orchestra.run.smithi016:> import os 2023-05-22T15:47:11.042 DEBUG:teuthology.orchestra.run.smithi016:> import subprocess 2023-05-22T15:47:11.043 DEBUG:teuthology.orchestra.run.smithi016:> 2023-05-22T15:47:11.043 DEBUG:teuthology.orchestra.run.smithi016:> def _find_admin_socket(client_name): 2023-05-22T15:47:11.043 DEBUG:teuthology.orchestra.run.smithi016:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2023-05-22T15:47:11.044 DEBUG:teuthology.orchestra.run.smithi016:> files = glob.glob(asok_path) 2023-05-22T15:47:11.044 DEBUG:teuthology.orchestra.run.smithi016:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2023-05-22T15:47:11.044 DEBUG:teuthology.orchestra.run.smithi016:> 2023-05-22T15:47:11.044 DEBUG:teuthology.orchestra.run.smithi016:> # Given a non-glob path, it better be there 2023-05-22T15:47:11.045 DEBUG:teuthology.orchestra.run.smithi016:> if "*" not in asok_path: 2023-05-22T15:47:11.045 DEBUG:teuthology.orchestra.run.smithi016:> assert(len(files) == 1) 2023-05-22T15:47:11.045 DEBUG:teuthology.orchestra.run.smithi016:> return files[0] 2023-05-22T15:47:11.046 DEBUG:teuthology.orchestra.run.smithi016:> 2023-05-22T15:47:11.046 DEBUG:teuthology.orchestra.run.smithi016:> for f in files: 2023-05-22T15:47:11.047 DEBUG:teuthology.orchestra.run.smithi016:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-05-22T15:47:11.047 DEBUG:teuthology.orchestra.run.smithi016:> if os.path.exists("/proc/{0}".format(pid)): 2023-05-22T15:47:11.047 DEBUG:teuthology.orchestra.run.smithi016:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-05-22T15:47:11.048 DEBUG:teuthology.orchestra.run.smithi016:> contents = proc_f.read() 2023-05-22T15:47:11.048 DEBUG:teuthology.orchestra.run.smithi016:> if mountpoint in contents: 2023-05-22T15:47:11.048 DEBUG:teuthology.orchestra.run.smithi016:> return f 2023-05-22T15:47:11.049 DEBUG:teuthology.orchestra.run.smithi016:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-05-22T15:47:11.049 DEBUG:teuthology.orchestra.run.smithi016:> 2023-05-22T15:47:11.049 DEBUG:teuthology.orchestra.run.smithi016:> print(_find_admin_socket("client.0")) 2023-05-22T15:47:11.049 DEBUG:teuthology.orchestra.run.smithi016:> ' 2023-05-22T15:47:11.168 INFO:teuthology.orchestra.run.smithi016.stdout:/var/run/ceph/ceph-client.0.65370.asok 2023-05-22T15:47:11.350 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.65370.asok 2023-05-22T15:47:11.351 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:47:11.351 DEBUG:teuthology.orchestra.run.smithi016:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.65370.asok status 2023-05-22T15:47:11.513 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-05-22T15:47:11.513 INFO:teuthology.orchestra.run.smithi016.stdout: "metadata": { 2023-05-22T15:47:11.513 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph_sha1": "9c9ba4df22024d9d8d6cf9f98a4b96751fc78d0c", 2023-05-22T15:47:11.513 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph_version": "ceph version 18.0.0-3192-g9c9ba4df (9c9ba4df22024d9d8d6cf9f98a4b96751fc78d0c) reef (dev)", 2023-05-22T15:47:11.513 INFO:teuthology.orchestra.run.smithi016.stdout: "entity_id": "0", 2023-05-22T15:47:11.513 INFO:teuthology.orchestra.run.smithi016.stdout: "hostname": "smithi016", 2023-05-22T15:47:11.514 INFO:teuthology.orchestra.run.smithi016.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2023-05-22T15:47:11.514 INFO:teuthology.orchestra.run.smithi016.stdout: "pid": "65370", 2023-05-22T15:47:11.514 INFO:teuthology.orchestra.run.smithi016.stdout: "root": "/" 2023-05-22T15:47:11.514 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:47:11.514 INFO:teuthology.orchestra.run.smithi016.stdout: "dentry_count": 0, 2023-05-22T15:47:11.514 INFO:teuthology.orchestra.run.smithi016.stdout: "dentry_pinned_count": 0, 2023-05-22T15:47:11.514 INFO:teuthology.orchestra.run.smithi016.stdout: "id": 9804, 2023-05-22T15:47:11.514 INFO:teuthology.orchestra.run.smithi016.stdout: "inst": { 2023-05-22T15:47:11.515 INFO:teuthology.orchestra.run.smithi016.stdout: "name": { 2023-05-22T15:47:11.515 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "client", 2023-05-22T15:47:11.515 INFO:teuthology.orchestra.run.smithi016.stdout: "num": 9804 2023-05-22T15:47:11.515 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:47:11.515 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": { 2023-05-22T15:47:11.515 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-05-22T15:47:11.515 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "192.168.0.1:0", 2023-05-22T15:47:11.516 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 116950703 2023-05-22T15:47:11.516 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T15:47:11.516 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:47:11.516 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": { 2023-05-22T15:47:11.516 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-05-22T15:47:11.516 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "192.168.0.1:0", 2023-05-22T15:47:11.516 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 116950703 2023-05-22T15:47:11.516 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T15:47:11.517 INFO:teuthology.orchestra.run.smithi016.stdout: "inst_str": "client.9804 192.168.0.1:0/116950703", 2023-05-22T15:47:11.517 INFO:teuthology.orchestra.run.smithi016.stdout: "addr_str": "192.168.0.1:0/116950703", 2023-05-22T15:47:11.517 INFO:teuthology.orchestra.run.smithi016.stdout: "inode_count": 1, 2023-05-22T15:47:11.517 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_epoch": 66, 2023-05-22T15:47:11.517 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_epoch": 161, 2023-05-22T15:47:11.517 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_epoch_barrier": 0, 2023-05-22T15:47:11.517 INFO:teuthology.orchestra.run.smithi016.stdout: "blocklisted": false, 2023-05-22T15:47:11.518 INFO:teuthology.orchestra.run.smithi016.stdout: "fs_name": "cephfs" 2023-05-22T15:47:11.518 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-05-22T15:47:11.526 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:47:11.526 DEBUG:teuthology.orchestra.run.smithi016:> stat --file-system '--printf=%T 2023-05-22T15:47:11.527 DEBUG:teuthology.orchestra.run.smithi016:> ' -- /home/ubuntu/cephtest/mnt.0 2023-05-22T15:47:11.532 INFO:teuthology.orchestra.run.smithi016.stdout:fuseblk 2023-05-22T15:47:11.533 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2023-05-22T15:47:11.533 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:47:11.533 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-05-22T15:47:11.587 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2023-05-22T15:47:11.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:11.751+0000 7f38d0351640 1 Processor -- start 2023-05-22T15:47:11.758 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:11.751+0000 7f38d0351640 1 -- start start 2023-05-22T15:47:11.759 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:11.751+0000 7f38d0351640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f38c8159b10 0x7f38c80cc780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:47:11.759 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:11.751+0000 7f38d0351640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f38c80b5e80 0x7f38c806db40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:47:11.759 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:11.751+0000 7f38d0351640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38c806e500 con 0x7f38c806e080 2023-05-22T15:47:11.759 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:11.751+0000 7f38d0351640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38c8159ee0 con 0x7f38c80b5e80 2023-05-22T15:47:11.759 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:11.751+0000 7f38d0351640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38c80b30b0 con 0x7f38c8159b10 2023-05-22T15:47:11.759 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:11.751+0000 7f38ce8ee640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f38c806e080 0x7f38c815d590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:43368/0 (socket says 172.21.15.16:43368) 2023-05-22T15:47:11.759 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:11.751+0000 7f38ce8ee640 1 -- 172.21.15.16:0/2416419415 learned_addr learned my addr 172.21.15.16:0/2416419415 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:47:11.760 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:11.751+0000 7f38cd8ec640 1 --2- 172.21.15.16:0/2416419415 >> v2:172.21.15.16:3300/0 conn(0x7f38c80b5e80 0x7f38c806db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:47:11.760 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:11.751+0000 7f38ce0ed640 1 --2- 172.21.15.16:0/2416419415 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f38c8159b10 0x7f38c80cc780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:47:11.760 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:11.751+0000 7f38cd0eb640 1 -- 172.21.15.16:0/2416419415 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 271263149 0 0) 0x7f38c00014a0 con 0x7f38c806e080 2023-05-22T15:47:11.760 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:11.751+0000 7f38cd0eb640 1 -- 172.21.15.16:0/2416419415 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f38b4003530 con 0x7f38c806e080 2023-05-22T15:47:11.760 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:11.751+0000 7f38cd8ec640 1 -- 172.21.15.16:0/2416419415 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f38c8159b10 msgr2=0x7f38c80cc780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:47:11.760 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:11.751+0000 7f38cd8ec640 1 --2- 172.21.15.16:0/2416419415 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f38c8159b10 0x7f38c80cc780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:11.760 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:11.751+0000 7f38cd8ec640 1 -- 172.21.15.16:0/2416419415 >> v1:172.21.15.16:6789/0 conn(0x7f38c806e080 legacy=0x7f38c815d590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:47:11.761 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:11.751+0000 7f38cd8ec640 1 -- 172.21.15.16:0/2416419415 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38c80b2810 con 0x7f38c80b5e80 2023-05-22T15:47:11.762 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:11.755+0000 7f38cd8ec640 1 --2- 172.21.15.16:0/2416419415 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38c80b5e80 0x7f38c806db40 secure :-1 s=READY pgs=2062 cs=0 l=1 rev1=1 crypto rx=0x7f38b80097c0 tx=0x7f38b801c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f53264c5aa40276 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:47:11.762 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:11.755+0000 7f38cd0eb640 1 -- 172.21.15.16:0/2416419415 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38b80ab650 con 0x7f38c80b5e80 2023-05-22T15:47:11.762 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:11.755+0000 7f38cd0eb640 1 -- 172.21.15.16:0/2416419415 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38b80ab7b0 con 0x7f38c80b5e80 2023-05-22T15:47:11.762 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:11.755+0000 7f38cd0eb640 1 -- 172.21.15.16:0/2416419415 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38b80abb40 con 0x7f38c80b5e80 2023-05-22T15:47:11.762 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:11.755+0000 7f38d0351640 1 -- 172.21.15.16:0/2416419415 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38c80b5e80 msgr2=0x7f38c806db40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:47:11.762 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:11.755+0000 7f38d0351640 1 --2- 172.21.15.16:0/2416419415 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38c80b5e80 0x7f38c806db40 secure :-1 s=READY pgs=2062 cs=0 l=1 rev1=1 crypto rx=0x7f38b80097c0 tx=0x7f38b801c450 comp rx=0 tx=0).stop 2023-05-22T15:47:11.762 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:11.755+0000 7f38d0351640 1 -- 172.21.15.16:0/2416419415 shutdown_connections 2023-05-22T15:47:11.763 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:11.755+0000 7f38d0351640 1 --2- 172.21.15.16:0/2416419415 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38c80b5e80 0x7f38c806db40 unknown :-1 s=CLOSED pgs=2062 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:11.763 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:11.755+0000 7f38d0351640 1 --2- 172.21.15.16:0/2416419415 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f38c8159b10 0x7f38c80cc780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:11.763 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:11.755+0000 7f38d0351640 1 -- 172.21.15.16:0/2416419415 >> 172.21.15.16:0/2416419415 conn(0x7f38c806f560 msgr2=0x7f38c806f960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:47:11.763 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:11.755+0000 7f38d0351640 1 -- 172.21.15.16:0/2416419415 shutdown_connections 2023-05-22T15:47:11.763 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:11.755+0000 7f38d0351640 1 -- 172.21.15.16:0/2416419415 wait complete. 2023-05-22T15:47:11.765 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:11.759+0000 7f38d0351640 1 Processor -- start 2023-05-22T15:47:11.765 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:11.759+0000 7f38d0351640 1 -- start start 2023-05-22T15:47:11.765 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:11.759+0000 7f38d0351640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f38c806e080 0x7f38c815cd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:47:11.765 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:11.759+0000 7f38d0351640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f38c80b5e80 0x7f38c8154500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:47:11.765 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:11.759+0000 7f38d0351640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38c8159b10 0x7f38c81571a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:47:11.765 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:11.759+0000 7f38d0351640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38c8155100 con 0x7f38c8159b10 2023-05-22T15:47:11.766 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:11.759+0000 7f38d0351640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38c81553e0 con 0x7f38c806e080 2023-05-22T15:47:11.766 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:11.759+0000 7f38d0351640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38c81556c0 con 0x7f38c80b5e80 2023-05-22T15:47:11.766 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:11.759+0000 7f38ce8ee640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38c8159b10 0x7f38c81571a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:47:11.766 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:11.759+0000 7f38ce8ee640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38c8159b10 0x7f38c81571a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53858/0 (socket says 172.21.15.16:53858) 2023-05-22T15:47:11.766 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:11.759+0000 7f38ce8ee640 1 -- 172.21.15.16:0/1784575431 learned_addr learned my addr 172.21.15.16:0/1784575431 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:47:11.766 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:11.759+0000 7f38cd8ec640 1 --2- 172.21.15.16:0/1784575431 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f38c80b5e80 0x7f38c8154500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:47:11.766 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:11.759+0000 7f38ce0ed640 1 --2- 172.21.15.16:0/1784575431 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f38c806e080 0x7f38c815cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:47:11.767 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:11.759+0000 7f38ce8ee640 1 -- 172.21.15.16:0/1784575431 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f38c80b5e80 msgr2=0x7f38c8154500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:47:11.767 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:11.759+0000 7f38ce8ee640 1 --2- 172.21.15.16:0/1784575431 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f38c80b5e80 0x7f38c8154500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:11.767 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:11.759+0000 7f38ce8ee640 1 -- 172.21.15.16:0/1784575431 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f38c806e080 msgr2=0x7f38c815cd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:47:11.767 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:11.759+0000 7f38ce8ee640 1 --2- 172.21.15.16:0/1784575431 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f38c806e080 0x7f38c815cd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:11.767 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:11.759+0000 7f38ce8ee640 1 -- 172.21.15.16:0/1784575431 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38b8009420 con 0x7f38c8159b10 2023-05-22T15:47:11.767 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:11.759+0000 7f38cd8ec640 1 --2- 172.21.15.16:0/1784575431 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f38c80b5e80 0x7f38c8154500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T15:47:11.767 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:11.759+0000 7f38ce8ee640 1 --2- 172.21.15.16:0/1784575431 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38c8159b10 0x7f38c81571a0 secure :-1 s=READY pgs=2063 cs=0 l=1 rev1=1 crypto rx=0x7f38c000cae0 tx=0x7f38c000a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:47:11.767 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:11.759+0000 7f38beffd640 1 -- 172.21.15.16:0/1784575431 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38c000ac40 con 0x7f38c8159b10 2023-05-22T15:47:11.768 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:11.759+0000 7f38beffd640 1 -- 172.21.15.16:0/1784575431 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38c000ada0 con 0x7f38c8159b10 2023-05-22T15:47:11.768 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:11.759+0000 7f38d0351640 1 -- 172.21.15.16:0/1784575431 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38c80b3910 con 0x7f38c8159b10 2023-05-22T15:47:11.768 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:11.759+0000 7f38beffd640 1 -- 172.21.15.16:0/1784575431 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38c0012620 con 0x7f38c8159b10 2023-05-22T15:47:11.768 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:11.759+0000 7f38d0351640 1 -- 172.21.15.16:0/1784575431 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38c80b3630 con 0x7f38c8159b10 2023-05-22T15:47:11.769 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:11.759+0000 7f38d0351640 1 -- 172.21.15.16:0/1784575431 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38c8166720 con 0x7f38c8159b10 2023-05-22T15:47:11.769 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:11.759+0000 7f38beffd640 1 -- 172.21.15.16:0/1784575431 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 67) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f38c0007070 con 0x7f38c8159b10 2023-05-22T15:47:11.769 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:11.759+0000 7f38beffd640 1 --2- 172.21.15.16:0/1784575431 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f38ac0654a0 0x7f38ac067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:47:11.769 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:11.759+0000 7f38beffd640 1 -- 172.21.15.16:0/1784575431 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(161..161 src has 1..161) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f38c0082980 con 0x7f38c8159b10 2023-05-22T15:47:11.769 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:11.759+0000 7f38ce0ed640 1 --2- 172.21.15.16:0/1784575431 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f38ac0654a0 0x7f38ac067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:47:11.769 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:11.763+0000 7f38ce0ed640 1 --2- 172.21.15.16:0/1784575431 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f38ac0654a0 0x7f38ac067950 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f38c80c62f0 tx=0x7f38c4008040 comp rx=0 tx=0).ready entity=mgr.9396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:47:11.770 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:11.767+0000 7f38beffd640 1 -- 172.21.15.16:0/1784575431 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f38c0053dc0 con 0x7f38c8159b10 2023-05-22T15:47:11.947 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:11.943+0000 7f38d0351640 1 -- 172.21.15.16:0/1784575431 --> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f38c806e500 con 0x7f38ac0654a0 2023-05-22T15:47:11.950 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:11.943+0000 7f38beffd640 1 -- 172.21.15.16:0/1784575431 <== mgr.9396 v2:172.21.15.16:6832/36921 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f38c806e500 con 0x7f38ac0654a0 2023-05-22T15:47:11.950 INFO:teuthology.orchestra.run.smithi016.stdout:[ 2023-05-22T15:47:11.951 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T15:47:11.951 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "cephfs" 2023-05-22T15:47:11.951 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T15:47:11.951 INFO:teuthology.orchestra.run.smithi016.stdout:] 2023-05-22T15:47:11.952 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:11.947+0000 7f38d0351640 1 -- 172.21.15.16:0/1784575431 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f38ac0654a0 msgr2=0x7f38ac067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:47:11.952 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:11.947+0000 7f38d0351640 1 --2- 172.21.15.16:0/1784575431 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f38ac0654a0 0x7f38ac067950 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f38c80c62f0 tx=0x7f38c4008040 comp rx=0 tx=0).stop 2023-05-22T15:47:11.953 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:11.947+0000 7f38d0351640 1 -- 172.21.15.16:0/1784575431 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38c8159b10 msgr2=0x7f38c81571a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:47:11.953 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:11.947+0000 7f38d0351640 1 --2- 172.21.15.16:0/1784575431 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38c8159b10 0x7f38c81571a0 secure :-1 s=READY pgs=2063 cs=0 l=1 rev1=1 crypto rx=0x7f38c000cae0 tx=0x7f38c000a060 comp rx=0 tx=0).stop 2023-05-22T15:47:11.953 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:11.947+0000 7f38d0351640 1 -- 172.21.15.16:0/1784575431 shutdown_connections 2023-05-22T15:47:11.953 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:11.947+0000 7f38d0351640 1 --2- 172.21.15.16:0/1784575431 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38c8159b10 0x7f38c81571a0 unknown :-1 s=CLOSED pgs=2063 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:11.953 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:11.947+0000 7f38d0351640 1 --2- 172.21.15.16:0/1784575431 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f38ac0654a0 0x7f38ac067950 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:11.953 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:11.947+0000 7f38d0351640 1 --2- 172.21.15.16:0/1784575431 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f38c80b5e80 0x7f38c8154500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:11.953 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:11.947+0000 7f38d0351640 1 --2- 172.21.15.16:0/1784575431 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f38c806e080 0x7f38c815cd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:11.954 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:11.947+0000 7f38d0351640 1 -- 172.21.15.16:0/1784575431 >> 172.21.15.16:0/1784575431 conn(0x7f38c806f560 msgr2=0x7f38c80c3d70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:47:11.954 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:11.947+0000 7f38d0351640 1 -- 172.21.15.16:0/1784575431 shutdown_connections 2023-05-22T15:47:11.954 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:11.947+0000 7f38d0351640 1 -- 172.21.15.16:0/1784575431 wait complete. 2023-05-22T15:47:11.969 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2023-05-22T15:47:12.091 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:12.083+0000 7fd6e5848640 1 Processor -- start 2023-05-22T15:47:12.091 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:12.083+0000 7fd6e5848640 1 -- start start 2023-05-22T15:47:12.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:12.087+0000 7fd6e5848640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd6e015bdf0 0x7fd6e00cda20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:47:12.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:12.087+0000 7fd6e5848640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6e00ce6c0 con 0x7fd6e006fec0 2023-05-22T15:47:12.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:12.087+0000 7fd6e5848640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6e00ce100 con 0x7fd6e006ac00 2023-05-22T15:47:12.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:12.087+0000 7fd6e5848640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6e015c1c0 con 0x7fd6e015bdf0 2023-05-22T15:47:12.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:12.087+0000 7fd6df7fe640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fd6e006fec0 0x7fd6e0069720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:43380/0 (socket says 172.21.15.16:43380) 2023-05-22T15:47:12.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:12.087+0000 7fd6df7fe640 1 -- 172.21.15.16:0/3025430337 learned_addr learned my addr 172.21.15.16:0/3025430337 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:47:12.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:12.087+0000 7fd6deffd640 1 --2- 172.21.15.16:0/3025430337 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd6e015bdf0 0x7fd6e00cda20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:47:12.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:12.087+0000 7fd6de7fc640 1 -- 172.21.15.16:0/3025430337 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1045517565 0 0) 0x7fd6e00ce6c0 con 0x7fd6e006fec0 2023-05-22T15:47:12.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:12.087+0000 7fd6de7fc640 1 -- 172.21.15.16:0/3025430337 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd6cc003530 con 0x7fd6e006fec0 2023-05-22T15:47:12.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:12.087+0000 7fd6de7fc640 1 -- 172.21.15.16:0/3025430337 <== mon.1 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 69899006 0 0) 0x7fd6e00ce100 con 0x7fd6e006ac00 2023-05-22T15:47:12.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:12.087+0000 7fd6de7fc640 1 -- 172.21.15.16:0/3025430337 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd6e00ce6c0 con 0x7fd6e006ac00 2023-05-22T15:47:12.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:12.087+0000 7fd6de7fc640 1 -- 172.21.15.16:0/3025430337 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2459271081 0 0) 0x7fd6cc003530 con 0x7fd6e006fec0 2023-05-22T15:47:12.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:12.087+0000 7fd6deffd640 1 -- 172.21.15.16:0/3025430337 >> v1:172.21.15.62:6789/0 conn(0x7fd6e006ac00 legacy=0x7fd6e006c080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:47:12.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:12.087+0000 7fd6deffd640 1 -- 172.21.15.16:0/3025430337 >> v1:172.21.15.16:6789/0 conn(0x7fd6e006fec0 legacy=0x7fd6e0069720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:47:12.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:12.087+0000 7fd6deffd640 1 -- 172.21.15.16:0/3025430337 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6e00ce3e0 con 0x7fd6e015bdf0 2023-05-22T15:47:12.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:12.091+0000 7fd6deffd640 1 --2- 172.21.15.16:0/3025430337 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd6e015bdf0 0x7fd6e00cda20 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fd6d00097f0 tx=0x7fd6d001c250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1978e538ba3ec942 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:47:12.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:12.091+0000 7fd6de7fc640 1 -- 172.21.15.16:0/3025430337 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6d8001980 con 0x7fd6e015bdf0 2023-05-22T15:47:12.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:12.091+0000 7fd6de7fc640 1 -- 172.21.15.16:0/3025430337 <== mon.2 v2:172.21.15.62:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6d00a7050 con 0x7fd6e015bdf0 2023-05-22T15:47:12.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:12.091+0000 7fd6de7fc640 1 -- 172.21.15.16:0/3025430337 <== mon.2 v2:172.21.15.62:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6d00ab600 con 0x7fd6e015bdf0 2023-05-22T15:47:12.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:12.091+0000 7fd6e5848640 1 -- 172.21.15.16:0/3025430337 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd6e015bdf0 msgr2=0x7fd6e00cda20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:47:12.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:12.091+0000 7fd6e5848640 1 --2- 172.21.15.16:0/3025430337 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd6e015bdf0 0x7fd6e00cda20 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fd6d00097f0 tx=0x7fd6d001c250 comp rx=0 tx=0).stop 2023-05-22T15:47:12.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:12.091+0000 7fd6e5848640 1 -- 172.21.15.16:0/3025430337 shutdown_connections 2023-05-22T15:47:12.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:12.091+0000 7fd6e5848640 1 --2- 172.21.15.16:0/3025430337 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd6e015bdf0 0x7fd6e00cda20 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:12.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:12.091+0000 7fd6e5848640 1 -- 172.21.15.16:0/3025430337 >> 172.21.15.16:0/3025430337 conn(0x7fd6e00ce890 msgr2=0x7fd6e00cec90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:47:12.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:12.091+0000 7fd6e5848640 1 -- 172.21.15.16:0/3025430337 shutdown_connections 2023-05-22T15:47:12.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:12.091+0000 7fd6e5848640 1 -- 172.21.15.16:0/3025430337 wait complete. 2023-05-22T15:47:12.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:12.091+0000 7fd6e5848640 1 Processor -- start 2023-05-22T15:47:12.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:12.091+0000 7fd6e5848640 1 -- start start 2023-05-22T15:47:12.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:12.091+0000 7fd6e5848640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd6e006ac00 0x7fd6e0156240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:47:12.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:12.091+0000 7fd6e5848640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd6e006fec0 0x7fd6e006ced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:47:12.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:12.091+0000 7fd6e5848640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd6e015bdf0 0x7fd6e00b2b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:47:12.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:12.091+0000 7fd6e5848640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6e00ca0b0 con 0x7fd6e006fec0 2023-05-22T15:47:12.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:12.091+0000 7fd6e5848640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6e00ca390 con 0x7fd6e015bdf0 2023-05-22T15:47:12.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:12.091+0000 7fd6e5848640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6e00ca670 con 0x7fd6e006ac00 2023-05-22T15:47:12.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:12.091+0000 7fd6d7fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd6e006fec0 0x7fd6e006ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:47:12.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:12.091+0000 7fd6d7fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd6e006fec0 0x7fd6e006ced0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53868/0 (socket says 172.21.15.16:53868) 2023-05-22T15:47:12.103 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:12.091+0000 7fd6deffd640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd6e006ac00 0x7fd6e0156240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:47:12.103 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:12.091+0000 7fd6d7fff640 1 -- 172.21.15.16:0/219434898 learned_addr learned my addr 172.21.15.16:0/219434898 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:47:12.103 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:12.091+0000 7fd6df7fe640 1 --2- 172.21.15.16:0/219434898 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd6e015bdf0 0x7fd6e00b2b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:47:12.103 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:12.095+0000 7fd6d7fff640 1 -- 172.21.15.16:0/219434898 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd6e006ac00 msgr2=0x7fd6e0156240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:47:12.103 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:12.095+0000 7fd6d7fff640 1 --2- 172.21.15.16:0/219434898 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd6e006ac00 0x7fd6e0156240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:12.103 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:12.095+0000 7fd6d7fff640 1 -- 172.21.15.16:0/219434898 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd6e015bdf0 msgr2=0x7fd6e00b2b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:47:12.104 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:12.095+0000 7fd6d7fff640 1 --2- 172.21.15.16:0/219434898 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd6e015bdf0 0x7fd6e00b2b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:12.104 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:12.095+0000 7fd6d7fff640 1 -- 172.21.15.16:0/219434898 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6d0009420 con 0x7fd6e006fec0 2023-05-22T15:47:12.104 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:12.095+0000 7fd6d7fff640 1 --2- 172.21.15.16:0/219434898 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd6e006fec0 0x7fd6e006ced0 secure :-1 s=READY pgs=2065 cs=0 l=1 rev1=1 crypto rx=0x7fd6c800c460 tx=0x7fd6c800c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:47:12.104 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:12.095+0000 7fd6dcff9640 1 -- 172.21.15.16:0/219434898 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6c8005480 con 0x7fd6e006fec0 2023-05-22T15:47:12.104 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:12.095+0000 7fd6dcff9640 1 -- 172.21.15.16:0/219434898 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6c8003600 con 0x7fd6e006fec0 2023-05-22T15:47:12.104 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:12.095+0000 7fd6e5848640 1 -- 172.21.15.16:0/219434898 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6e0070750 con 0x7fd6e006fec0 2023-05-22T15:47:12.104 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:12.095+0000 7fd6dcff9640 1 -- 172.21.15.16:0/219434898 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6c80039f0 con 0x7fd6e006fec0 2023-05-22T15:47:12.105 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:12.095+0000 7fd6e5848640 1 -- 172.21.15.16:0/219434898 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6e006bb60 con 0x7fd6e006fec0 2023-05-22T15:47:12.105 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:12.095+0000 7fd6e5848640 1 -- 172.21.15.16:0/219434898 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6e00c8be0 con 0x7fd6e006fec0 2023-05-22T15:47:12.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:12.095+0000 7fd6dcff9640 1 -- 172.21.15.16:0/219434898 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 67) v1 ==== 84332+0+0 (secure 0 0 0) 0x7fd6c8003b50 con 0x7fd6e006fec0 2023-05-22T15:47:12.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:12.095+0000 7fd6dcff9640 1 --2- 172.21.15.16:0/219434898 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd6c00654d0 0x7fd6c0067980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:47:12.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:12.095+0000 7fd6deffd640 1 --2- 172.21.15.16:0/219434898 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd6c00654d0 0x7fd6c0067980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:47:12.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:12.095+0000 7fd6dcff9640 1 -- 172.21.15.16:0/219434898 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(161..161 src has 1..161) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fd6c8082540 con 0x7fd6e006fec0 2023-05-22T15:47:12.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:12.095+0000 7fd6deffd640 1 --2- 172.21.15.16:0/219434898 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd6c00654d0 0x7fd6c0067980 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fd6d00ad000 tx=0x7fd6d0008040 comp rx=0 tx=0).ready entity=mgr.9396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:47:12.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:12.099+0000 7fd6dcff9640 1 -- 172.21.15.16:0/219434898 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fd6c804ff60 con 0x7fd6e006fec0 2023-05-22T15:47:12.301 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:12.295+0000 7fd6e5848640 1 -- 172.21.15.16:0/219434898 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7fd6e00cd0f0 con 0x7fd6e006fec0 2023-05-22T15:47:13.096 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.091+0000 7fd6dcff9640 1 -- 172.21.15.16:0/219434898 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mgrmap(e 68) v1 ==== 84349+0+0 (secure 0 0 0) 0x7fd6c8050330 con 0x7fd6e006fec0 2023-05-22T15:47:13.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.091+0000 7fd6dcff9640 1 -- 172.21.15.16:0/219434898 <== mon.0 v2:172.21.15.16:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v68) v1 ==== 92+0+0 (secure 0 0 0) 0x7fd6c804ff60 con 0x7fd6e006fec0 2023-05-22T15:47:13.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.095+0000 7fd6e5848640 1 -- 172.21.15.16:0/219434898 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd6c00654d0 msgr2=0x7fd6c0067980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:47:13.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.095+0000 7fd6e5848640 1 --2- 172.21.15.16:0/219434898 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd6c00654d0 0x7fd6c0067980 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fd6d00ad000 tx=0x7fd6d0008040 comp rx=0 tx=0).stop 2023-05-22T15:47:13.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.095+0000 7fd6e5848640 1 -- 172.21.15.16:0/219434898 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd6e006fec0 msgr2=0x7fd6e006ced0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:47:13.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.095+0000 7fd6e5848640 1 --2- 172.21.15.16:0/219434898 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd6e006fec0 0x7fd6e006ced0 secure :-1 s=READY pgs=2065 cs=0 l=1 rev1=1 crypto rx=0x7fd6c800c460 tx=0x7fd6c800c920 comp rx=0 tx=0).stop 2023-05-22T15:47:13.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.095+0000 7fd6e5848640 1 -- 172.21.15.16:0/219434898 shutdown_connections 2023-05-22T15:47:13.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.095+0000 7fd6e5848640 1 --2- 172.21.15.16:0/219434898 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd6e015bdf0 0x7fd6e00b2b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:13.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.095+0000 7fd6e5848640 1 --2- 172.21.15.16:0/219434898 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd6e006fec0 0x7fd6e006ced0 unknown :-1 s=CLOSED pgs=2065 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:13.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.095+0000 7fd6e5848640 1 --2- 172.21.15.16:0/219434898 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd6c00654d0 0x7fd6c0067980 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:13.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.095+0000 7fd6e5848640 1 --2- 172.21.15.16:0/219434898 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd6e006ac00 0x7fd6e0156240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:13.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.095+0000 7fd6e5848640 1 -- 172.21.15.16:0/219434898 >> 172.21.15.16:0/219434898 conn(0x7fd6e00ce890 msgr2=0x7fd6e0153b10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:47:13.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.095+0000 7fd6e5848640 1 -- 172.21.15.16:0/219434898 shutdown_connections 2023-05-22T15:47:13.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.095+0000 7fd6e5848640 1 -- 172.21.15.16:0/219434898 wait complete. 2023-05-22T15:47:13.121 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2023-05-22T15:47:13.220 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:47:13.219+0000 7f73270fd180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-05-22T15:47:13.238 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:47:13.231+0000 7f79c4e90180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-05-22T15:47:13.251 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.243+0000 7f8680c31640 1 Processor -- start 2023-05-22T15:47:13.252 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.243+0000 7f8680c31640 1 -- start start 2023-05-22T15:47:13.253 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.247+0000 7f8680c31640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f867c0cf870 0x7f867c06db00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:47:13.254 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.247+0000 7f8680c31640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f867c06e040 con 0x7f867c15cfa0 2023-05-22T15:47:13.254 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.247+0000 7f8680c31640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f867c06e490 con 0x7f867c06e740 2023-05-22T15:47:13.254 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.247+0000 7f8680c31640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f867c0cb9f0 con 0x7f867c0cf870 2023-05-22T15:47:13.254 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.247+0000 7f8679d9b640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f867c0cf870 0x7f867c06db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:47:13.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.247+0000 7f8679d9b640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f867c0cf870 0x7f867c06db00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53882/0 (socket says 172.21.15.16:53882) 2023-05-22T15:47:13.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.247+0000 7f8679d9b640 1 -- 172.21.15.16:0/1908812237 learned_addr learned my addr 172.21.15.16:0/1908812237 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:47:13.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.247+0000 7f8679d9b640 1 -- 172.21.15.16:0/1908812237 >> v1:172.21.15.62:6789/0 conn(0x7f867c06e740 legacy=0x7f867c070de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:47:13.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.247+0000 7f8679d9b640 1 -- 172.21.15.16:0/1908812237 >> v1:172.21.15.16:6789/0 conn(0x7f867c15cfa0 legacy=0x7f867c0b1fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:47:13.256 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.247+0000 7f8679d9b640 1 -- 172.21.15.16:0/1908812237 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f867c0cad70 con 0x7f867c0cf870 2023-05-22T15:47:13.259 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.247+0000 7f8679d9b640 1 --2- 172.21.15.16:0/1908812237 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f867c0cf870 0x7f867c06db00 secure :-1 s=READY pgs=2070 cs=0 l=1 rev1=1 crypto rx=0x7f8670009870 tx=0x7f867001c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a1b167e8858f126 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:47:13.260 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.247+0000 7f867959a640 1 -- 172.21.15.16:0/1908812237 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86700aa430 con 0x7f867c0cf870 2023-05-22T15:47:13.260 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.247+0000 7f867959a640 1 -- 172.21.15.16:0/1908812237 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86700aa590 con 0x7f867c0cf870 2023-05-22T15:47:13.260 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.247+0000 7f867959a640 1 -- 172.21.15.16:0/1908812237 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86700aa920 con 0x7f867c0cf870 2023-05-22T15:47:13.261 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.251+0000 7f8680c31640 1 -- 172.21.15.16:0/1908812237 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f867c0cf870 msgr2=0x7f867c06db00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:47:13.261 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.251+0000 7f8680c31640 1 --2- 172.21.15.16:0/1908812237 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f867c0cf870 0x7f867c06db00 secure :-1 s=READY pgs=2070 cs=0 l=1 rev1=1 crypto rx=0x7f8670009870 tx=0x7f867001c4d0 comp rx=0 tx=0).stop 2023-05-22T15:47:13.262 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.251+0000 7f8680c31640 1 -- 172.21.15.16:0/1908812237 shutdown_connections 2023-05-22T15:47:13.262 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.251+0000 7f8680c31640 1 --2- 172.21.15.16:0/1908812237 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f867c0cf870 0x7f867c06db00 unknown :-1 s=CLOSED pgs=2070 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:13.262 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.251+0000 7f8680c31640 1 -- 172.21.15.16:0/1908812237 >> 172.21.15.16:0/1908812237 conn(0x7f867c0cc870 msgr2=0x7f867c0cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:47:13.262 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.251+0000 7f8680c31640 1 -- 172.21.15.16:0/1908812237 shutdown_connections 2023-05-22T15:47:13.263 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.251+0000 7f8680c31640 1 -- 172.21.15.16:0/1908812237 wait complete. 2023-05-22T15:47:13.263 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.251+0000 7f8680c31640 1 Processor -- start 2023-05-22T15:47:13.263 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.251+0000 7f8680c31640 1 -- start start 2023-05-22T15:47:13.263 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.251+0000 7f8680c31640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f867c06e740 0x7f867c15aca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:47:13.264 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.251+0000 7f8680c31640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f867c0cf870 0x7f867c0ab7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:47:13.264 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.251+0000 7f8680c31640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f867c15cfa0 0x7f867c0aeb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:47:13.264 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.251+0000 7f8680c31640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f867c0c88d0 con 0x7f867c06e740 2023-05-22T15:47:13.264 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.251+0000 7f8680c31640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f867c15b1e0 con 0x7f867c15cfa0 2023-05-22T15:47:13.265 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.251+0000 7f8680c31640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f867c159100 con 0x7f867c0cf870 2023-05-22T15:47:13.265 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.251+0000 7f867a59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f867c06e740 0x7f867c15aca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:47:13.265 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.251+0000 7f867a59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f867c06e740 0x7f867c15aca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53890/0 (socket says 172.21.15.16:53890) 2023-05-22T15:47:13.265 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.251+0000 7f867ad9d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f867c15cfa0 0x7f867c0aeb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:47:13.266 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.251+0000 7f867a59c640 1 -- 172.21.15.16:0/3944919905 learned_addr learned my addr 172.21.15.16:0/3944919905 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:47:13.266 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.251+0000 7f8679d9b640 1 --2- 172.21.15.16:0/3944919905 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f867c0cf870 0x7f867c0ab7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:47:13.266 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.251+0000 7f867a59c640 1 -- 172.21.15.16:0/3944919905 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f867c0cf870 msgr2=0x7f867c0ab7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:47:13.266 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.251+0000 7f867a59c640 1 --2- 172.21.15.16:0/3944919905 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f867c0cf870 0x7f867c0ab7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:13.267 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.251+0000 7f867a59c640 1 -- 172.21.15.16:0/3944919905 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f867c15cfa0 msgr2=0x7f867c0aeb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:47:13.267 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.251+0000 7f867a59c640 1 --2- 172.21.15.16:0/3944919905 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f867c15cfa0 0x7f867c0aeb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:13.267 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.251+0000 7f867a59c640 1 -- 172.21.15.16:0/3944919905 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86700094d0 con 0x7f867c06e740 2023-05-22T15:47:13.267 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.251+0000 7f867a59c640 1 --2- 172.21.15.16:0/3944919905 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f867c06e740 0x7f867c15aca0 secure :-1 s=READY pgs=2071 cs=0 l=1 rev1=1 crypto rx=0x7f866c00c3c0 tx=0x7f866c00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:47:13.268 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.251+0000 7f866affd640 1 -- 172.21.15.16:0/3944919905 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f866c0052d0 con 0x7f867c06e740 2023-05-22T15:47:13.268 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.251+0000 7f866affd640 1 -- 172.21.15.16:0/3944919905 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f866c005430 con 0x7f867c06e740 2023-05-22T15:47:13.268 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.251+0000 7f8680c31640 1 -- 172.21.15.16:0/3944919905 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f867c06efd0 con 0x7f867c06e740 2023-05-22T15:47:13.269 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.251+0000 7f866affd640 1 -- 172.21.15.16:0/3944919905 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f866c003840 con 0x7f867c06e740 2023-05-22T15:47:13.269 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.251+0000 7f8680c31640 1 -- 172.21.15.16:0/3944919905 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f867c0cb610 con 0x7f867c06e740 2023-05-22T15:47:13.269 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.255+0000 7f8680c31640 1 -- 172.21.15.16:0/3944919905 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f867c071900 con 0x7f867c06e740 2023-05-22T15:47:13.271 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.255+0000 7f866affd640 1 -- 172.21.15.16:0/3944919905 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 68) v1 ==== 84349+0+0 (secure 0 0 0) 0x7f866c0039a0 con 0x7f867c06e740 2023-05-22T15:47:13.271 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.255+0000 7f866affd640 1 --2- 172.21.15.16:0/3944919905 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f865c0654f0 0x7f865c0679a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:47:13.272 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.255+0000 7f8679d9b640 1 -- 172.21.15.16:0/3944919905 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f865c0654f0 msgr2=0x7f865c0679a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/36921 2023-05-22T15:47:13.272 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.255+0000 7f866affd640 1 -- 172.21.15.16:0/3944919905 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(161..161 src has 1..161) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f866c082030 con 0x7f867c06e740 2023-05-22T15:47:13.272 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.255+0000 7f8679d9b640 1 --2- 172.21.15.16:0/3944919905 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f865c0654f0 0x7f865c0679a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-05-22T15:47:13.272 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.263+0000 7f866affd640 1 -- 172.21.15.16:0/3944919905 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f866c04fb70 con 0x7f867c06e740 2023-05-22T15:47:13.420 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.415+0000 7f8680c31640 1 -- 172.21.15.16:0/3944919905 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7f867c06e040 con 0x7f867c06e740 2023-05-22T15:47:13.425 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.419+0000 7f866affd640 1 -- 172.21.15.16:0/3944919905 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v53)=0 v53) v1 ==== 138+0+0 (secure 0 0 0) 0x7f867c06e040 con 0x7f867c06e740 2023-05-22T15:47:13.427 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.419+0000 7f8680c31640 1 -- 172.21.15.16:0/3944919905 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f865c0654f0 msgr2=0x7f865c0679a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-05-22T15:47:13.427 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.419+0000 7f8680c31640 1 --2- 172.21.15.16:0/3944919905 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f865c0654f0 0x7f865c0679a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:13.427 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.423+0000 7f8680c31640 1 -- 172.21.15.16:0/3944919905 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f867c06e740 msgr2=0x7f867c15aca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:47:13.428 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.423+0000 7f8680c31640 1 --2- 172.21.15.16:0/3944919905 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f867c06e740 0x7f867c15aca0 secure :-1 s=READY pgs=2071 cs=0 l=1 rev1=1 crypto rx=0x7f866c00c3c0 tx=0x7f866c00c880 comp rx=0 tx=0).stop 2023-05-22T15:47:13.428 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.423+0000 7f8680c31640 1 -- 172.21.15.16:0/3944919905 shutdown_connections 2023-05-22T15:47:13.428 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.423+0000 7f8680c31640 1 --2- 172.21.15.16:0/3944919905 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f867c15cfa0 0x7f867c0aeb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:13.428 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.423+0000 7f8680c31640 1 --2- 172.21.15.16:0/3944919905 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f865c0654f0 0x7f865c0679a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:13.428 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.423+0000 7f8680c31640 1 --2- 172.21.15.16:0/3944919905 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f867c0cf870 0x7f867c0ab7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:13.428 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.423+0000 7f8680c31640 1 --2- 172.21.15.16:0/3944919905 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f867c06e740 0x7f867c15aca0 unknown :-1 s=CLOSED pgs=2071 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:13.429 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.423+0000 7f8680c31640 1 -- 172.21.15.16:0/3944919905 >> 172.21.15.16:0/3944919905 conn(0x7f867c0cc870 msgr2=0x7f867c0ce830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:47:13.429 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.423+0000 7f8680c31640 1 -- 172.21.15.16:0/3944919905 shutdown_connections 2023-05-22T15:47:13.429 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.423+0000 7f8680c31640 1 -- 172.21.15.16:0/3944919905 wait complete. 2023-05-22T15:47:13.444 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2023-05-22T15:47:13.560 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.555+0000 7f0abbdd2640 1 Processor -- start 2023-05-22T15:47:13.560 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.555+0000 7f0abbdd2640 1 -- start start 2023-05-22T15:47:13.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.555+0000 7f0abbdd2640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0ab40ce900 0x7f0ab40ced50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:47:13.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.555+0000 7f0abbdd2640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ab40d0400 con 0x7f0ab40cf290 2023-05-22T15:47:13.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.555+0000 7f0abbdd2640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ab40d00d0 con 0x7f0ab415bca0 2023-05-22T15:47:13.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.555+0000 7f0abbdd2640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ab415c070 con 0x7f0ab40ce900 2023-05-22T15:47:13.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.555+0000 7f0aba36f640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f0ab40cf290 0x7f0ab40d0890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:43398/0 (socket says 172.21.15.16:43398) 2023-05-22T15:47:13.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.555+0000 7f0aba36f640 1 -- 172.21.15.16:0/4151918955 learned_addr learned my addr 172.21.15.16:0/4151918955 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:47:13.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.555+0000 7f0ab936d640 1 --2- 172.21.15.16:0/4151918955 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0ab40ce900 0x7f0ab40ced50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:47:13.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.555+0000 7f0ab8b6c640 1 -- 172.21.15.16:0/4151918955 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1484068962 0 0) 0x7f0ab40d0400 con 0x7f0ab40cf290 2023-05-22T15:47:13.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.555+0000 7f0ab8b6c640 1 -- 172.21.15.16:0/4151918955 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0aa0003530 con 0x7f0ab40cf290 2023-05-22T15:47:13.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.555+0000 7f0ab8b6c640 1 -- 172.21.15.16:0/4151918955 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 428061668 0 0) 0x7f0aa0003530 con 0x7f0ab40cf290 2023-05-22T15:47:13.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.555+0000 7f0ab8b6c640 1 -- 172.21.15.16:0/4151918955 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0ab40ce900 msgr2=0x7f0ab40ced50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:47:13.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.555+0000 7f0ab8b6c640 1 --2- 172.21.15.16:0/4151918955 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0ab40ce900 0x7f0ab40ced50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:13.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.555+0000 7f0ab8b6c640 1 -- 172.21.15.16:0/4151918955 >> v1:172.21.15.62:6789/0 conn(0x7f0ab415bca0 legacy=0x7f0ab4160160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:47:13.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.555+0000 7f0ab8b6c640 1 -- 172.21.15.16:0/4151918955 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ab40ccbe0 con 0x7f0ab40cf290 2023-05-22T15:47:13.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.555+0000 7f0ab8b6c640 1 -- 172.21.15.16:0/4151918955 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7f0aac001a00 con 0x7f0ab40cf290 2023-05-22T15:47:13.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.555+0000 7f0ab8b6c640 1 -- 172.21.15.16:0/4151918955 >> v1:172.21.15.16:6789/0 conn(0x7f0ab40cf290 legacy=0x7f0ab40d0890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:47:13.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.555+0000 7f0ab8b6c640 1 --2- 172.21.15.16:0/4151918955 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0aa00039b0 0x7f0aa0003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:47:13.565 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.555+0000 7f0ab8b6c640 1 --2- 172.21.15.16:0/4151918955 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0ab40ce900 0x7f0aa00047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:47:13.565 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.555+0000 7f0ab8b6c640 1 --2- 172.21.15.16:0/4151918955 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0aa0004d10 0x7f0aa0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:47:13.565 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.555+0000 7f0ab8b6c640 1 -- 172.21.15.16:0/4151918955 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ab415c070 con 0x7f0aa0004d10 2023-05-22T15:47:13.565 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.555+0000 7f0ab8b6c640 1 -- 172.21.15.16:0/4151918955 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0aa000bad0 con 0x7f0aa00039b0 2023-05-22T15:47:13.565 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.555+0000 7f0ab8b6c640 1 -- 172.21.15.16:0/4151918955 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0aa000bc90 con 0x7f0ab40ce900 2023-05-22T15:47:13.565 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.555+0000 7f0aba36f640 1 --2- 172.21.15.16:0/4151918955 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0aa0004d10 0x7f0aa0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:47:13.565 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.555+0000 7f0ab9b6e640 1 --2- 172.21.15.16:0/4151918955 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0aa00039b0 0x7f0aa0003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:47:13.565 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.555+0000 7f0ab936d640 1 --2- 172.21.15.16:0/4151918955 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0ab40ce900 0x7f0aa00047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:47:13.566 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.555+0000 7f0aba36f640 1 -- 172.21.15.16:0/4151918955 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0ab40ce900 msgr2=0x7f0aa00047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:47:13.566 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.555+0000 7f0aba36f640 1 --2- 172.21.15.16:0/4151918955 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0ab40ce900 0x7f0aa00047d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:13.566 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.555+0000 7f0aba36f640 1 -- 172.21.15.16:0/4151918955 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0aa00039b0 msgr2=0x7f0aa0003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:47:13.566 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.555+0000 7f0aba36f640 1 --2- 172.21.15.16:0/4151918955 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0aa00039b0 0x7f0aa0003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:13.566 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.555+0000 7f0aba36f640 1 -- 172.21.15.16:0/4151918955 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0aa0003530 con 0x7f0aa0004d10 2023-05-22T15:47:13.566 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.555+0000 7f0ab9b6e640 1 --2- 172.21.15.16:0/4151918955 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0aa00039b0 0x7f0aa0003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T15:47:13.568 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.559+0000 7f0aba36f640 1 --2- 172.21.15.16:0/4151918955 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0aa0004d10 0x7f0aa0005180 secure :-1 s=READY pgs=2073 cs=0 l=1 rev1=1 crypto rx=0x7f0aac00a830 tx=0x7f0aac01faa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb3ba3c9a4118a3b server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:47:13.569 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.559+0000 7f0ab8b6c640 1 -- 172.21.15.16:0/4151918955 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0aac0aa6d0 con 0x7f0aa0004d10 2023-05-22T15:47:13.569 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.559+0000 7f0ab8b6c640 1 -- 172.21.15.16:0/4151918955 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0aac0aa830 con 0x7f0aa0004d10 2023-05-22T15:47:13.569 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.559+0000 7f0ab8b6c640 1 -- 172.21.15.16:0/4151918955 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0aac0aac20 con 0x7f0aa0004d10 2023-05-22T15:47:13.569 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.559+0000 7f0abbdd2640 1 -- 172.21.15.16:0/4151918955 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0aa0004d10 msgr2=0x7f0aa0005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:47:13.569 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.559+0000 7f0abbdd2640 1 --2- 172.21.15.16:0/4151918955 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0aa0004d10 0x7f0aa0005180 secure :-1 s=READY pgs=2073 cs=0 l=1 rev1=1 crypto rx=0x7f0aac00a830 tx=0x7f0aac01faa0 comp rx=0 tx=0).stop 2023-05-22T15:47:13.569 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.559+0000 7f0aba36f640 1 -- 172.21.15.16:0/4151918955 reap_dead start 2023-05-22T15:47:13.569 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.559+0000 7f0abbdd2640 1 -- 172.21.15.16:0/4151918955 shutdown_connections 2023-05-22T15:47:13.570 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.559+0000 7f0abbdd2640 1 -- 172.21.15.16:0/4151918955 >> 172.21.15.16:0/4151918955 conn(0x7f0ab40657f0 msgr2=0x7f0ab4070e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:47:13.570 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.559+0000 7f0abbdd2640 1 -- 172.21.15.16:0/4151918955 shutdown_connections 2023-05-22T15:47:13.570 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.559+0000 7f0abbdd2640 1 -- 172.21.15.16:0/4151918955 wait complete. 2023-05-22T15:47:13.570 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.559+0000 7f0abbdd2640 1 Processor -- start 2023-05-22T15:47:13.570 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.559+0000 7f0abbdd2640 1 -- start start 2023-05-22T15:47:13.570 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.559+0000 7f0abbdd2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0aa0004d10 0x7f0ab40b3200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:47:13.570 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.559+0000 7f0abbdd2640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0ab41f1b90 0x7f0ab40b3740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:47:13.571 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.559+0000 7f0abbdd2640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0ab414a300 0x7f0ab414a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:47:13.571 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.559+0000 7f0abbdd2640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ab40b3c80 con 0x7f0aa0004d10 2023-05-22T15:47:13.571 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.559+0000 7f0abbdd2640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ab40b3f30 con 0x7f0ab41f1b90 2023-05-22T15:47:13.571 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.559+0000 7f0abbdd2640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ab40b4210 con 0x7f0ab414a300 2023-05-22T15:47:13.571 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.559+0000 7f0ab9b6e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0aa0004d10 0x7f0ab40b3200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:47:13.571 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.559+0000 7f0ab9b6e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0aa0004d10 0x7f0ab40b3200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53912/0 (socket says 172.21.15.16:53912) 2023-05-22T15:47:13.571 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.559+0000 7f0ab9b6e640 1 -- 172.21.15.16:0/829703573 learned_addr learned my addr 172.21.15.16:0/829703573 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:47:13.571 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.563+0000 7f0ab936d640 1 --2- 172.21.15.16:0/829703573 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0ab41f1b90 0x7f0ab40b3740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:47:13.572 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.563+0000 7f0aba36f640 1 --2- 172.21.15.16:0/829703573 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0ab414a300 0x7f0ab414a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:47:13.572 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.563+0000 7f0ab9b6e640 1 -- 172.21.15.16:0/829703573 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0ab414a300 msgr2=0x7f0ab414a7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:47:13.572 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.563+0000 7f0ab9b6e640 1 --2- 172.21.15.16:0/829703573 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0ab414a300 0x7f0ab414a7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:13.572 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.563+0000 7f0ab9b6e640 1 -- 172.21.15.16:0/829703573 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0ab41f1b90 msgr2=0x7f0ab40b3740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:47:13.572 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.563+0000 7f0ab9b6e640 1 --2- 172.21.15.16:0/829703573 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0ab41f1b90 0x7f0ab40b3740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:13.572 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.563+0000 7f0ab9b6e640 1 -- 172.21.15.16:0/829703573 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ab40cf6d0 con 0x7f0aa0004d10 2023-05-22T15:47:13.572 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.563+0000 7f0ab9b6e640 1 --2- 172.21.15.16:0/829703573 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0aa0004d10 0x7f0ab40b3200 secure :-1 s=READY pgs=2074 cs=0 l=1 rev1=1 crypto rx=0x7f0ab000e550 tx=0x7f0ab000ea10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:47:13.573 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.563+0000 7f0aaa7fc640 1 -- 172.21.15.16:0/829703573 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ab00052e0 con 0x7f0aa0004d10 2023-05-22T15:47:13.573 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.563+0000 7f0aaa7fc640 1 -- 172.21.15.16:0/829703573 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ab0005440 con 0x7f0aa0004d10 2023-05-22T15:47:13.573 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.563+0000 7f0aaa7fc640 1 -- 172.21.15.16:0/829703573 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ab0003840 con 0x7f0aa0004d10 2023-05-22T15:47:13.573 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.563+0000 7f0abbdd2640 1 -- 172.21.15.16:0/829703573 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ab40ccec0 con 0x7f0aa0004d10 2023-05-22T15:47:13.573 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.563+0000 7f0abbdd2640 1 -- 172.21.15.16:0/829703573 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ab40cd1a0 con 0x7f0aa0004d10 2023-05-22T15:47:13.574 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.563+0000 7f0abbdd2640 1 -- 172.21.15.16:0/829703573 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ab4148300 con 0x7f0aa0004d10 2023-05-22T15:47:13.574 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.563+0000 7f0aaa7fc640 1 -- 172.21.15.16:0/829703573 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 68) v1 ==== 84349+0+0 (secure 0 0 0) 0x7f0ab00039a0 con 0x7f0aa0004d10 2023-05-22T15:47:13.575 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.563+0000 7f0aaa7fc640 1 --2- 172.21.15.16:0/829703573 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f0a980654f0 0x7f0a980679a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:47:13.575 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.563+0000 7f0ab936d640 1 -- 172.21.15.16:0/829703573 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f0a980654f0 msgr2=0x7f0a980679a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/36921 2023-05-22T15:47:13.575 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.563+0000 7f0aaa7fc640 1 -- 172.21.15.16:0/829703573 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(161..161 src has 1..161) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f0ab0082980 con 0x7f0aa0004d10 2023-05-22T15:47:13.575 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.563+0000 7f0ab936d640 1 --2- 172.21.15.16:0/829703573 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f0a980654f0 0x7f0a980679a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-05-22T15:47:13.575 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.567+0000 7f0aaa7fc640 1 -- 172.21.15.16:0/829703573 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f0ab0050290 con 0x7f0aa0004d10 2023-05-22T15:47:13.608 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:47:13.603+0000 7f73270fd180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-05-22T15:47:13.634 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:47:13.627+0000 7f79c4e90180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-05-22T15:47:13.726 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.719+0000 7f0abbdd2640 1 -- 172.21.15.16:0/829703573 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7f0ab404dbe0 con 0x7f0aa0004d10 2023-05-22T15:47:13.733 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.727+0000 7f0aaa7fc640 1 -- 172.21.15.16:0/829703573 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v54)=0 v54) v1 ==== 133+0+0 (secure 0 0 0) 0x7f0ab00538b0 con 0x7f0aa0004d10 2023-05-22T15:47:13.735 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.731+0000 7f0abbdd2640 1 -- 172.21.15.16:0/829703573 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f0a980654f0 msgr2=0x7f0a980679a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-05-22T15:47:13.736 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.731+0000 7f0abbdd2640 1 --2- 172.21.15.16:0/829703573 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f0a980654f0 0x7f0a980679a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:13.736 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.731+0000 7f0abbdd2640 1 -- 172.21.15.16:0/829703573 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0aa0004d10 msgr2=0x7f0ab40b3200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:47:13.736 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.731+0000 7f0abbdd2640 1 --2- 172.21.15.16:0/829703573 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0aa0004d10 0x7f0ab40b3200 secure :-1 s=READY pgs=2074 cs=0 l=1 rev1=1 crypto rx=0x7f0ab000e550 tx=0x7f0ab000ea10 comp rx=0 tx=0).stop 2023-05-22T15:47:13.736 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.731+0000 7f0abbdd2640 1 -- 172.21.15.16:0/829703573 shutdown_connections 2023-05-22T15:47:13.737 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.731+0000 7f0abbdd2640 1 --2- 172.21.15.16:0/829703573 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f0a980654f0 0x7f0a980679a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:13.737 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.731+0000 7f0abbdd2640 1 --2- 172.21.15.16:0/829703573 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0ab414a300 0x7f0ab414a7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:13.737 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.731+0000 7f0abbdd2640 1 --2- 172.21.15.16:0/829703573 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0ab41f1b90 0x7f0ab40b3740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:13.737 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.731+0000 7f0abbdd2640 1 --2- 172.21.15.16:0/829703573 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0aa0004d10 0x7f0ab40b3200 unknown :-1 s=CLOSED pgs=2074 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:13.738 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.731+0000 7f0abbdd2640 1 -- 172.21.15.16:0/829703573 >> 172.21.15.16:0/829703573 conn(0x7f0ab40657f0 msgr2=0x7f0ab40b1dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:47:13.738 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.731+0000 7f0abbdd2640 1 -- 172.21.15.16:0/829703573 shutdown_connections 2023-05-22T15:47:13.738 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.731+0000 7f0abbdd2640 1 -- 172.21.15.16:0/829703573 wait complete. 2023-05-22T15:47:13.746 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:47:13.743+0000 7f73270fd180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-05-22T15:47:13.752 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:47:13.753 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1) 2023-05-22T15:47:13.763 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:47:13.764 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/test_restart) 2023-05-22T15:47:13.774 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:47:13.767+0000 7f79c4e90180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-05-22T15:47:13.784 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add --fs cephfs /snap_test_dir1/test_restart 1M 2023-05-22T15:47:13.945 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.938+0000 7f8551910640 1 Processor -- start 2023-05-22T15:47:13.946 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.938+0000 7f8551910640 1 -- start start 2023-05-22T15:47:13.947 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.938+0000 7f8551910640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f854c0caf40 0x7f854c15c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:47:13.947 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.938+0000 7f8551910640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f854c0cc060 0x7f854c0c9d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:47:13.947 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.938+0000 7f8551910640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f854c0ca290 con 0x7f854c0cd510 2023-05-22T15:47:13.947 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.938+0000 7f8551910640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f854c159150 con 0x7f854c0cc060 2023-05-22T15:47:13.948 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.938+0000 7f8551910640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f854c15ce30 con 0x7f854c0caf40 2023-05-22T15:47:13.948 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.938+0000 7f854a7fc640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f854c0cc060 0x7f854c0c9d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:47:13.948 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.938+0000 7f854a7fc640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f854c0cc060 0x7f854c0c9d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53916/0 (socket says 172.21.15.16:53916) 2023-05-22T15:47:13.948 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.938+0000 7f854a7fc640 1 -- 172.21.15.16:0/1591363559 learned_addr learned my addr 172.21.15.16:0/1591363559 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:47:13.948 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.942+0000 7f854affd640 1 --2- 172.21.15.16:0/1591363559 >> v2:172.21.15.62:3300/0 conn(0x7f854c0caf40 0x7f854c15c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:47:13.948 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.942+0000 7f854a7fc640 1 -- 172.21.15.16:0/1591363559 >> v2:172.21.15.62:3300/0 conn(0x7f854c0caf40 msgr2=0x7f854c15c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:47:13.948 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.942+0000 7f854a7fc640 1 --2- 172.21.15.16:0/1591363559 >> v2:172.21.15.62:3300/0 conn(0x7f854c0caf40 0x7f854c15c7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:13.949 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.942+0000 7f854a7fc640 1 -- 172.21.15.16:0/1591363559 >> v1:172.21.15.16:6789/0 conn(0x7f854c0cd510 legacy=0x7f854c0cd920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:47:13.949 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.942+0000 7f854a7fc640 1 -- 172.21.15.16:0/1591363559 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f854c15d110 con 0x7f854c0cc060 2023-05-22T15:47:13.950 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.942+0000 7f854a7fc640 1 --2- 172.21.15.16:0/1591363559 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f854c0cc060 0x7f854c0c9d50 secure :-1 s=READY pgs=2076 cs=0 l=1 rev1=1 crypto rx=0x7f854c0c9860 tx=0x7f854001eb40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ca9f4c3616282b8 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:47:13.950 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.942+0000 7f8549ffb640 1 -- 172.21.15.16:0/1591363559 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85400a7a10 con 0x7f854c0cc060 2023-05-22T15:47:13.950 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.942+0000 7f8549ffb640 1 -- 172.21.15.16:0/1591363559 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85400a7b70 con 0x7f854c0cc060 2023-05-22T15:47:13.950 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.942+0000 7f8549ffb640 1 -- 172.21.15.16:0/1591363559 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85400af440 con 0x7f854c0cc060 2023-05-22T15:47:13.951 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.942+0000 7f8551910640 1 -- 172.21.15.16:0/1591363559 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f854c0cc060 msgr2=0x7f854c0c9d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:47:13.951 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.942+0000 7f8551910640 1 --2- 172.21.15.16:0/1591363559 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f854c0cc060 0x7f854c0c9d50 secure :-1 s=READY pgs=2076 cs=0 l=1 rev1=1 crypto rx=0x7f854c0c9860 tx=0x7f854001eb40 comp rx=0 tx=0).stop 2023-05-22T15:47:13.951 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.942+0000 7f8551910640 1 -- 172.21.15.16:0/1591363559 shutdown_connections 2023-05-22T15:47:13.951 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.942+0000 7f8551910640 1 --2- 172.21.15.16:0/1591363559 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f854c0cc060 0x7f854c0c9d50 unknown :-1 s=CLOSED pgs=2076 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:13.951 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.942+0000 7f8551910640 1 --2- 172.21.15.16:0/1591363559 >> v2:172.21.15.62:3300/0 conn(0x7f854c0caf40 0x7f854c15c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:13.952 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.942+0000 7f8551910640 1 -- 172.21.15.16:0/1591363559 >> 172.21.15.16:0/1591363559 conn(0x7f854c0b0d30 msgr2=0x7f854c0b1130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:47:13.952 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.942+0000 7f8551910640 1 -- 172.21.15.16:0/1591363559 shutdown_connections 2023-05-22T15:47:13.952 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.942+0000 7f8551910640 1 -- 172.21.15.16:0/1591363559 wait complete. 2023-05-22T15:47:13.954 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.946+0000 7f8551910640 1 Processor -- start 2023-05-22T15:47:13.954 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.946+0000 7f8551910640 1 -- start start 2023-05-22T15:47:13.955 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.946+0000 7f8551910640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f854c0caf40 0x7f854c156280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:47:13.955 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.946+0000 7f8551910640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f854c0cc060 0x7f854c0b42d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:47:13.955 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.946+0000 7f8551910640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f854c0cd510 0x7f854c0b4810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:47:13.955 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.946+0000 7f8551910640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f854c158270 con 0x7f854c0cc060 2023-05-22T15:47:13.955 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.946+0000 7f8551910640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f854c0c7c20 con 0x7f854c0cd510 2023-05-22T15:47:13.955 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.946+0000 7f8551910640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f854c1f8960 con 0x7f854c0caf40 2023-05-22T15:47:13.955 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.946+0000 7f854a7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f854c0cc060 0x7f854c0b42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:47:13.956 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.946+0000 7f854a7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f854c0cc060 0x7f854c0b42d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53926/0 (socket says 172.21.15.16:53926) 2023-05-22T15:47:13.956 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.946+0000 7f854a7fc640 1 -- 172.21.15.16:0/2612240180 learned_addr learned my addr 172.21.15.16:0/2612240180 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:47:13.956 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.946+0000 7f854b7fe640 1 --2- 172.21.15.16:0/2612240180 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f854c0cd510 0x7f854c0b4810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:47:13.956 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.946+0000 7f854affd640 1 --2- 172.21.15.16:0/2612240180 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f854c0caf40 0x7f854c156280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:47:13.956 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.946+0000 7f854a7fc640 1 -- 172.21.15.16:0/2612240180 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f854c0caf40 msgr2=0x7f854c156280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:47:13.956 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.946+0000 7f854a7fc640 1 --2- 172.21.15.16:0/2612240180 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f854c0caf40 0x7f854c156280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:13.956 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.946+0000 7f854a7fc640 1 -- 172.21.15.16:0/2612240180 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f854c0cd510 msgr2=0x7f854c0b4810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:47:13.957 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.946+0000 7f854a7fc640 1 --2- 172.21.15.16:0/2612240180 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f854c0cd510 0x7f854c0b4810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:13.957 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.946+0000 7f854a7fc640 1 -- 172.21.15.16:0/2612240180 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f854000b770 con 0x7f854c0cc060 2023-05-22T15:47:13.957 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.946+0000 7f854b7fe640 1 --2- 172.21.15.16:0/2612240180 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f854c0cd510 0x7f854c0b4810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T15:47:13.957 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.946+0000 7f854a7fc640 1 --2- 172.21.15.16:0/2612240180 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f854c0cc060 0x7f854c0b42d0 secure :-1 s=READY pgs=2077 cs=0 l=1 rev1=1 crypto rx=0x7f854001b750 tx=0x7f854001b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:47:13.957 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.946+0000 7f8533fff640 1 -- 172.21.15.16:0/2612240180 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85400ba660 con 0x7f854c0cc060 2023-05-22T15:47:13.957 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.946+0000 7f8533fff640 1 -- 172.21.15.16:0/2612240180 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85400ba7c0 con 0x7f854c0cc060 2023-05-22T15:47:13.957 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.946+0000 7f8551910640 1 -- 172.21.15.16:0/2612240180 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f854c0cc4b0 con 0x7f854c0cc060 2023-05-22T15:47:13.957 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.946+0000 7f8533fff640 1 -- 172.21.15.16:0/2612240180 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85400bab70 con 0x7f854c0cc060 2023-05-22T15:47:13.958 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.946+0000 7f8551910640 1 -- 172.21.15.16:0/2612240180 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f854c070800 con 0x7f854c0cc060 2023-05-22T15:47:13.958 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.946+0000 7f8551910640 1 -- 172.21.15.16:0/2612240180 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f854c071530 con 0x7f854c0cc060 2023-05-22T15:47:13.958 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.946+0000 7f8533fff640 1 -- 172.21.15.16:0/2612240180 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 68) v1 ==== 84349+0+0 (secure 0 0 0) 0x7f85400c33b0 con 0x7f854c0cc060 2023-05-22T15:47:13.958 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.946+0000 7f8533fff640 1 --2- 172.21.15.16:0/2612240180 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f8538065520 0x7f85380679d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:47:13.958 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.946+0000 7f854affd640 1 -- 172.21.15.16:0/2612240180 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f8538065520 msgr2=0x7f85380679d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/36921 2023-05-22T15:47:13.958 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.946+0000 7f8533fff640 1 -- 172.21.15.16:0/2612240180 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(161..161 src has 1..161) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f854011deb0 con 0x7f854c0cc060 2023-05-22T15:47:13.959 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.946+0000 7f854affd640 1 --2- 172.21.15.16:0/2612240180 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f8538065520 0x7f85380679d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-05-22T15:47:13.959 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:13.950+0000 7f8533fff640 1 -- 172.21.15.16:0/2612240180 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f85400eb860 con 0x7f854c0cc060 2023-05-22T15:47:14.127 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:14.122+0000 7f8551910640 1 -- 172.21.15.16:0/2612240180 --> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/snap_test_dir1/test_restart", "snap_schedule": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f854c0c6060 con 0x7f8538065520 2023-05-22T15:47:14.155 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:14.150+0000 7f854affd640 1 -- 172.21.15.16:0/2612240180 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f8538065520 msgr2=0x7f85380679d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/36921 2023-05-22T15:47:14.155 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:14.150+0000 7f854affd640 1 --2- 172.21.15.16:0/2612240180 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f8538065520 0x7f85380679d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-05-22T15:47:14.192 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:47:14.191+0000 7f73270fd180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-05-22T15:47:14.222 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:47:14.214+0000 7f79c4e90180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-05-22T15:47:14.246 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:47:14.243+0000 7f73270fd180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-05-22T15:47:14.275 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:47:14.270+0000 7f79c4e90180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-05-22T15:47:14.296 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:47:14.291+0000 7f73270fd180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-05-22T15:47:14.326 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:47:14.318+0000 7f79c4e90180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-05-22T15:47:14.375 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:47:14.371+0000 7f73270fd180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-05-22T15:47:14.403 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:47:14.398+0000 7f79c4e90180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-05-22T15:47:14.426 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:47:14.423+0000 7f73270fd180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-05-22T15:47:14.454 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:47:14.446+0000 7f79c4e90180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-05-22T15:47:14.503 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:47:14.499+0000 7f73270fd180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-05-22T15:47:14.530 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:47:14.522+0000 7f79c4e90180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-05-22T15:47:14.557 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:47:14.555+0000 7f73270fd180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-05-22T15:47:14.559 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:14.554+0000 7f854affd640 1 -- 172.21.15.16:0/2612240180 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f8538065520 msgr2=0x7f85380679d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/36921 2023-05-22T15:47:14.559 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:14.554+0000 7f854affd640 1 --2- 172.21.15.16:0/2612240180 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f8538065520 0x7f85380679d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-05-22T15:47:14.584 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:47:14.578+0000 7f79c4e90180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-05-22T15:47:14.689 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:47:14.687+0000 7f73270fd180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-05-22T15:47:14.716 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:47:14.710+0000 7f79c4e90180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-05-22T15:47:14.742 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:47:14.739+0000 7f73270fd180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-05-22T15:47:14.769 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:47:14.762+0000 7f79c4e90180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-05-22T15:47:14.967 INFO:tasks.ceph.mgr.x.smithi062.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2023-05-22T15:47:14.968 INFO:tasks.ceph.mgr.x.smithi062.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2023-05-22T15:47:14.968 INFO:tasks.ceph.mgr.x.smithi062.stderr: from numpy import show_config as show_numpy_config 2023-05-22T15:47:14.975 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:47:14.971+0000 7f73270fd180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-05-22T15:47:14.993 INFO:tasks.ceph.mgr.y.smithi016.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2023-05-22T15:47:14.993 INFO:tasks.ceph.mgr.y.smithi016.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2023-05-22T15:47:14.993 INFO:tasks.ceph.mgr.y.smithi016.stderr: from numpy import show_config as show_numpy_config 2023-05-22T15:47:15.001 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:47:14.994+0000 7f79c4e90180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-05-22T15:47:15.101 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:47:15.099+0000 7f73270fd180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-05-22T15:47:15.127 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:47:15.118+0000 7f79c4e90180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-05-22T15:47:15.214 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:47:15.211+0000 7f73270fd180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-05-22T15:47:15.240 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:47:15.234+0000 7f79c4e90180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-05-22T15:47:15.277 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:47:15.275+0000 7f73270fd180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-05-22T15:47:15.304 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:47:15.298+0000 7f79c4e90180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-05-22T15:47:15.363 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:15.358+0000 7f854affd640 1 -- 172.21.15.16:0/2612240180 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f8538065520 msgr2=0x7f85380679d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/36921 2023-05-22T15:47:15.363 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:15.358+0000 7f854affd640 1 --2- 172.21.15.16:0/2612240180 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f8538065520 0x7f85380679d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-05-22T15:47:15.403 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:47:15.399+0000 7f73270fd180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-05-22T15:47:15.431 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:47:15.426+0000 7f79c4e90180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-05-22T15:47:15.900 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:47:15.895+0000 7f73270fd180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-05-22T15:47:15.925 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:47:15.918+0000 7f79c4e90180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-05-22T15:47:15.976 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:47:15.971+0000 7f73270fd180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-05-22T15:47:16.001 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:47:15.994+0000 7f79c4e90180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-05-22T15:47:16.113 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:47:16.111+0000 7f73270fd180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-05-22T15:47:16.140 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:47:16.134+0000 7f79c4e90180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-05-22T15:47:16.190 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:47:16.187+0000 7f73270fd180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-05-22T15:47:16.216 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:47:16.210+0000 7f79c4e90180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-05-22T15:47:16.441 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:47:16.439+0000 7f73270fd180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-05-22T15:47:16.463 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:47:16.458+0000 7f79c4e90180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-05-22T15:47:16.495 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T15:47:16.491+0000 7f73270fd180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-05-22T15:47:16.518 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:47:16.510+0000 7f79c4e90180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-05-22T15:47:16.555 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:16.550+0000 7f8533fff640 1 -- 172.21.15.16:0/2612240180 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mgrmap(e 69) v1 ==== 83958+0+0 (secure 0 0 0) 0x7f854011c110 con 0x7f854c0cc060 2023-05-22T15:47:16.556 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:16.550+0000 7f8533fff640 1 -- 172.21.15.16:0/2612240180 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f8538065520 msgr2=0x7f85380679d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-05-22T15:47:16.556 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:16.550+0000 7f8533fff640 1 --2- 172.21.15.16:0/2612240180 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f8538065520 0x7f85380679d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:16.723 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:47:16.718+0000 7f794680d640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T15:47:16.724 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:47:16.718+0000 7f794680d640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T15:47:16.724 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:47:16.718+0000 7f794680d640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T15:47:16.724 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:47:16.718+0000 7f794680d640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T15:47:16.724 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:47:16.718+0000 7f794680d640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T15:47:16.724 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:47:16.718+0000 7f7943006640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T15:47:16.725 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:47:16.718+0000 7f7943006640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T15:47:16.725 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:47:16.718+0000 7f7943006640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T15:47:16.725 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:47:16.718+0000 7f7943006640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T15:47:16.725 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:47:16.718+0000 7f7943006640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T15:47:17.561 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:17.554+0000 7f8533fff640 1 -- 172.21.15.16:0/2612240180 <== mon.0 v2:172.21.15.16:3300/0 8 ==== mgrmap(e 70) v1 ==== 84085+0+0 (secure 0 0 0) 0x7f85400ebc30 con 0x7f854c0cc060 2023-05-22T15:47:17.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:17.554+0000 7f8533fff640 1 --2- 172.21.15.16:0/2612240180 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f8538065520 0x7f8538067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:47:17.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:17.554+0000 7f8533fff640 1 -- 172.21.15.16:0/2612240180 --> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/snap_test_dir1/test_restart", "snap_schedule": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f854c0c6060 con 0x7f8538065520 2023-05-22T15:47:17.568 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:17.562+0000 7f854affd640 1 --2- 172.21.15.16:0/2612240180 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f8538065520 0x7f8538067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:47:17.571 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:17.562+0000 7f854affd640 1 --2- 172.21.15.16:0/2612240180 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f8538065520 0x7f8538067950 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f853c007170 tx=0x7f853c008040 comp rx=0 tx=0).ready entity=mgr.9831 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:47:17.701 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:17.694+0000 7f8533fff640 1 -- 172.21.15.16:0/2612240180 <== mgr.9831 v2:172.21.15.16:6832/36921 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f854c0c6060 con 0x7f8538065520 2023-05-22T15:47:17.701 INFO:teuthology.orchestra.run.smithi016.stdout:Schedule set for path /snap_test_dir1/test_restart 2023-05-22T15:47:17.704 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:17.698+0000 7f8551910640 1 -- 172.21.15.16:0/2612240180 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f8538065520 msgr2=0x7f8538067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:47:17.705 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:17.698+0000 7f8551910640 1 --2- 172.21.15.16:0/2612240180 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f8538065520 0x7f8538067950 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f853c007170 tx=0x7f853c008040 comp rx=0 tx=0).stop 2023-05-22T15:47:17.705 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:17.698+0000 7f8551910640 1 -- 172.21.15.16:0/2612240180 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f854c0cc060 msgr2=0x7f854c0b42d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:47:17.705 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:17.698+0000 7f8551910640 1 --2- 172.21.15.16:0/2612240180 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f854c0cc060 0x7f854c0b42d0 secure :-1 s=READY pgs=2077 cs=0 l=1 rev1=1 crypto rx=0x7f854001b750 tx=0x7f854001b8e0 comp rx=0 tx=0).stop 2023-05-22T15:47:17.705 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:17.698+0000 7f8551910640 1 -- 172.21.15.16:0/2612240180 shutdown_connections 2023-05-22T15:47:17.705 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:17.698+0000 7f8551910640 1 --2- 172.21.15.16:0/2612240180 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f854c0cd510 0x7f854c0b4810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:17.705 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:17.698+0000 7f8551910640 1 --2- 172.21.15.16:0/2612240180 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f854c0cc060 0x7f854c0b42d0 unknown :-1 s=CLOSED pgs=2077 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:17.706 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:17.698+0000 7f8551910640 1 --2- 172.21.15.16:0/2612240180 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f8538065520 0x7f8538067950 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:17.706 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:17.698+0000 7f8551910640 1 --2- 172.21.15.16:0/2612240180 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f854c0caf40 0x7f854c156280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:17.706 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:17.698+0000 7f8551910640 1 -- 172.21.15.16:0/2612240180 >> 172.21.15.16:0/2612240180 conn(0x7f854c0b0d30 msgr2=0x7f854c0cf970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:47:17.706 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:17.698+0000 7f8551910640 1 -- 172.21.15.16:0/2612240180 shutdown_connections 2023-05-22T15:47:17.706 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:17.698+0000 7f8551910640 1 -- 172.21.15.16:0/2612240180 wait complete. 2023-05-22T15:47:17.726 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/test_restart 2023-05-22T15:47:17.727 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate --fs cephfs /snap_test_dir1/test_restart 1M 2023-05-22T15:47:17.861 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:17.854+0000 7fa5d4ae6640 1 Processor -- start 2023-05-22T15:47:17.861 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:17.854+0000 7fa5d4ae6640 1 -- start start 2023-05-22T15:47:17.862 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:17.854+0000 7fa5d4ae6640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fa5d0070c10 0x7fa5d00cc4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:47:17.862 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:17.854+0000 7fa5d4ae6640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa5d00c9e40 0x7fa5d006bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:47:17.862 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:17.854+0000 7fa5d4ae6640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5d00ccbb0 con 0x7fa5d015d650 2023-05-22T15:47:17.862 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:17.854+0000 7fa5d4ae6640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5d00cd7c0 con 0x7fa5d0070c10 2023-05-22T15:47:17.863 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:17.854+0000 7fa5d4ae6640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5d00cdaa0 con 0x7fa5d00c9e40 2023-05-22T15:47:17.863 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:17.854+0000 7fa5ce59c640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fa5d0070c10 0x7fa5d00cc4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:47:17.864 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:17.854+0000 7fa5ced9d640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fa5d015d650 0x7fa5d00ce2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:43422/0 (socket says 172.21.15.16:43422) 2023-05-22T15:47:17.864 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:17.854+0000 7fa5ced9d640 1 -- 172.21.15.16:0/3646496473 learned_addr learned my addr 172.21.15.16:0/3646496473 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:47:17.864 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:17.854+0000 7fa5cdd9b640 1 --2- 172.21.15.16:0/3646496473 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa5d00c9e40 0x7fa5d006bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:47:17.864 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:17.858+0000 7fa5ce59c640 1 -- 172.21.15.16:0/3646496473 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa5d00c9e40 msgr2=0x7fa5d006bfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:47:17.864 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:17.858+0000 7fa5ce59c640 1 --2- 172.21.15.16:0/3646496473 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa5d00c9e40 0x7fa5d006bfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:17.864 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:17.858+0000 7fa5ce59c640 1 -- 172.21.15.16:0/3646496473 >> v1:172.21.15.16:6789/0 conn(0x7fa5d015d650 legacy=0x7fa5d00ce2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:47:17.864 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:17.858+0000 7fa5ce59c640 1 -- 172.21.15.16:0/3646496473 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5d00cdd80 con 0x7fa5d0070c10 2023-05-22T15:47:17.865 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:17.858+0000 7fa5cd59a640 1 -- 172.21.15.16:0/3646496473 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1124550832 0 0) 0x7fa5d00ccbb0 con 0x7fa5d015d650 2023-05-22T15:47:17.866 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:17.858+0000 7fa5ce59c640 1 --2- 172.21.15.16:0/3646496473 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa5d0070c10 0x7fa5d00cc4e0 secure :-1 s=READY pgs=2094 cs=0 l=1 rev1=1 crypto rx=0x7fa5c00097c0 tx=0x7fa5c001c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b88eba13e27cd048 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:47:17.866 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:17.858+0000 7fa5cd59a640 1 -- 172.21.15.16:0/3646496473 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5c00a6040 con 0x7fa5d0070c10 2023-05-22T15:47:17.866 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:17.858+0000 7fa5cd59a640 1 -- 172.21.15.16:0/3646496473 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5c00aa600 con 0x7fa5d0070c10 2023-05-22T15:47:17.867 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:17.858+0000 7fa5cd59a640 1 -- 172.21.15.16:0/3646496473 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5c00aa990 con 0x7fa5d0070c10 2023-05-22T15:47:17.867 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:17.858+0000 7fa5d4ae6640 1 -- 172.21.15.16:0/3646496473 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa5d0070c10 msgr2=0x7fa5d00cc4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:47:17.867 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:17.858+0000 7fa5d4ae6640 1 --2- 172.21.15.16:0/3646496473 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa5d0070c10 0x7fa5d00cc4e0 secure :-1 s=READY pgs=2094 cs=0 l=1 rev1=1 crypto rx=0x7fa5c00097c0 tx=0x7fa5c001c480 comp rx=0 tx=0).stop 2023-05-22T15:47:17.867 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:17.858+0000 7fa5d4ae6640 1 -- 172.21.15.16:0/3646496473 shutdown_connections 2023-05-22T15:47:17.867 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:17.858+0000 7fa5d4ae6640 1 --2- 172.21.15.16:0/3646496473 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa5d00c9e40 0x7fa5d006bfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:17.867 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:17.858+0000 7fa5d4ae6640 1 --2- 172.21.15.16:0/3646496473 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa5d0070c10 0x7fa5d00cc4e0 unknown :-1 s=CLOSED pgs=2094 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:17.867 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:17.858+0000 7fa5d4ae6640 1 -- 172.21.15.16:0/3646496473 >> 172.21.15.16:0/3646496473 conn(0x7fa5d0069720 msgr2=0x7fa5d0069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:47:17.867 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:17.858+0000 7fa5d4ae6640 1 -- 172.21.15.16:0/3646496473 shutdown_connections 2023-05-22T15:47:17.868 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:17.858+0000 7fa5d4ae6640 1 -- 172.21.15.16:0/3646496473 wait complete. 2023-05-22T15:47:17.869 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:17.862+0000 7fa5d4ae6640 1 Processor -- start 2023-05-22T15:47:17.869 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:17.862+0000 7fa5d4ae6640 1 -- start start 2023-05-22T15:47:17.869 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:17.862+0000 7fa5d4ae6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa5d0070c10 0x7fa5d01468a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:47:17.870 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:17.862+0000 7fa5d4ae6640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa5d00c9e40 0x7fa5d015c850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:47:17.870 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:17.862+0000 7fa5d4ae6640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa5d015d650 0x7fa5d0150b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:47:17.870 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:17.862+0000 7fa5d4ae6640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5d014fa90 con 0x7fa5d0070c10 2023-05-22T15:47:17.870 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:17.862+0000 7fa5d4ae6640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5d014f510 con 0x7fa5d00c9e40 2023-05-22T15:47:17.870 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:17.862+0000 7fa5d4ae6640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5d015bdb0 con 0x7fa5d015d650 2023-05-22T15:47:17.870 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:17.862+0000 7fa5ce59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa5d0070c10 0x7fa5d01468a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:47:17.871 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:17.862+0000 7fa5ce59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa5d0070c10 0x7fa5d01468a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54076/0 (socket says 172.21.15.16:54076) 2023-05-22T15:47:17.871 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:17.862+0000 7fa5ce59c640 1 -- 172.21.15.16:0/2308502029 learned_addr learned my addr 172.21.15.16:0/2308502029 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:47:17.871 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:17.862+0000 7fa5ced9d640 1 --2- 172.21.15.16:0/2308502029 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa5d015d650 0x7fa5d0150b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:47:17.871 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:17.862+0000 7fa5cdd9b640 1 --2- 172.21.15.16:0/2308502029 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa5d00c9e40 0x7fa5d015c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:47:17.871 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:17.862+0000 7fa5ce59c640 1 -- 172.21.15.16:0/2308502029 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa5d015d650 msgr2=0x7fa5d0150b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:47:17.871 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:17.862+0000 7fa5ce59c640 1 --2- 172.21.15.16:0/2308502029 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa5d015d650 0x7fa5d0150b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:17.871 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:17.862+0000 7fa5ce59c640 1 -- 172.21.15.16:0/2308502029 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa5d00c9e40 msgr2=0x7fa5d015c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:47:17.872 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:17.862+0000 7fa5ce59c640 1 --2- 172.21.15.16:0/2308502029 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa5d00c9e40 0x7fa5d015c850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:17.872 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:17.862+0000 7fa5ce59c640 1 -- 172.21.15.16:0/2308502029 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5c0009420 con 0x7fa5d0070c10 2023-05-22T15:47:17.872 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:17.862+0000 7fa5cdd9b640 1 --2- 172.21.15.16:0/2308502029 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa5d00c9e40 0x7fa5d015c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:47:17.872 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:17.862+0000 7fa5ce59c640 1 --2- 172.21.15.16:0/2308502029 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa5d0070c10 0x7fa5d01468a0 secure :-1 s=READY pgs=2095 cs=0 l=1 rev1=1 crypto rx=0x7fa5c00ac040 tx=0x7fa5c001acc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:47:17.872 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:17.862+0000 7fa5beffd640 1 -- 172.21.15.16:0/2308502029 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5c00aac40 con 0x7fa5d0070c10 2023-05-22T15:47:17.872 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:17.862+0000 7fa5beffd640 1 -- 172.21.15.16:0/2308502029 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5c00b7070 con 0x7fa5d0070c10 2023-05-22T15:47:17.872 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:17.862+0000 7fa5beffd640 1 -- 172.21.15.16:0/2308502029 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5c0004020 con 0x7fa5d0070c10 2023-05-22T15:47:17.873 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:17.862+0000 7fa5d4ae6640 1 -- 172.21.15.16:0/2308502029 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5d015dea0 con 0x7fa5d0070c10 2023-05-22T15:47:17.873 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:17.862+0000 7fa5d4ae6640 1 -- 172.21.15.16:0/2308502029 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5d00cce90 con 0x7fa5d0070c10 2023-05-22T15:47:17.873 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:17.862+0000 7fa5d4ae6640 1 -- 172.21.15.16:0/2308502029 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5d00cedc0 con 0x7fa5d0070c10 2023-05-22T15:47:17.874 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:17.862+0000 7fa5beffd640 1 -- 172.21.15.16:0/2308502029 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 70) v1 ==== 84085+0+0 (secure 0 0 0) 0x7fa5c0004180 con 0x7fa5d0070c10 2023-05-22T15:47:17.874 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:17.862+0000 7fa5beffd640 1 --2- 172.21.15.16:0/2308502029 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fa5b0065220 0x7fa5b00676d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:47:17.874 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:17.866+0000 7fa5beffd640 1 -- 172.21.15.16:0/2308502029 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(162..162 src has 1..162) v4 ==== 6754+0+0 (secure 0 0 0) 0x7fa5c011cad0 con 0x7fa5d0070c10 2023-05-22T15:47:17.874 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:17.866+0000 7fa5cdd9b640 1 --2- 172.21.15.16:0/2308502029 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fa5b0065220 0x7fa5b00676d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:47:17.874 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:17.866+0000 7fa5cdd9b640 1 --2- 172.21.15.16:0/2308502029 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fa5b0065220 0x7fa5b00676d0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fa5d0165980 tx=0x7fa5c4008040 comp rx=0 tx=0).ready entity=mgr.9831 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:47:17.875 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:17.870+0000 7fa5beffd640 1 -- 172.21.15.16:0/2308502029 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fa5c00ea7d0 con 0x7fa5d0070c10 2023-05-22T15:47:18.048 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:18.042+0000 7fa5d4ae6640 1 -- 172.21.15.16:0/2308502029 --> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "fs": "cephfs", "path": "/snap_test_dir1/test_restart", "repeat": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7fa5d004dbe0 con 0x7fa5b0065220 2023-05-22T15:47:18.146 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:18.138+0000 7fa5beffd640 1 -- 172.21.15.16:0/2308502029 <== mgr.9831 v2:172.21.15.16:6832/36921 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+57 (secure 0 0 0) 0x7fa5d004dbe0 con 0x7fa5b0065220 2023-05-22T15:47:18.147 INFO:teuthology.orchestra.run.smithi016.stdout:Schedule activated for path /snap_test_dir1/test_restart 2023-05-22T15:47:18.149 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:18.142+0000 7fa5d4ae6640 1 -- 172.21.15.16:0/2308502029 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fa5b0065220 msgr2=0x7fa5b00676d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:47:18.150 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:18.142+0000 7fa5d4ae6640 1 --2- 172.21.15.16:0/2308502029 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fa5b0065220 0x7fa5b00676d0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fa5d0165980 tx=0x7fa5c4008040 comp rx=0 tx=0).stop 2023-05-22T15:47:18.150 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:18.142+0000 7fa5d4ae6640 1 -- 172.21.15.16:0/2308502029 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa5d0070c10 msgr2=0x7fa5d01468a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:47:18.150 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:18.142+0000 7fa5d4ae6640 1 --2- 172.21.15.16:0/2308502029 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa5d0070c10 0x7fa5d01468a0 secure :-1 s=READY pgs=2095 cs=0 l=1 rev1=1 crypto rx=0x7fa5c00ac040 tx=0x7fa5c001acc0 comp rx=0 tx=0).stop 2023-05-22T15:47:18.151 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:18.142+0000 7fa5d4ae6640 1 -- 172.21.15.16:0/2308502029 shutdown_connections 2023-05-22T15:47:18.151 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:18.142+0000 7fa5d4ae6640 1 --2- 172.21.15.16:0/2308502029 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fa5b0065220 0x7fa5b00676d0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:18.151 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:18.142+0000 7fa5d4ae6640 1 --2- 172.21.15.16:0/2308502029 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa5d015d650 0x7fa5d0150b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:18.151 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:18.142+0000 7fa5d4ae6640 1 --2- 172.21.15.16:0/2308502029 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa5d00c9e40 0x7fa5d015c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:18.151 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:18.142+0000 7fa5d4ae6640 1 --2- 172.21.15.16:0/2308502029 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa5d0070c10 0x7fa5d01468a0 unknown :-1 s=CLOSED pgs=2095 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:47:18.151 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:18.142+0000 7fa5d4ae6640 1 -- 172.21.15.16:0/2308502029 >> 172.21.15.16:0/2308502029 conn(0x7fa5d0069720 msgr2=0x7fa5d0157c60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:47:18.152 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:18.142+0000 7fa5d4ae6640 1 -- 172.21.15.16:0/2308502029 shutdown_connections 2023-05-22T15:47:18.152 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:47:18.142+0000 7fa5d4ae6640 1 -- 172.21.15.16:0/2308502029 wait complete. 2023-05-22T15:47:18.172 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/test_restart 2023-05-22T15:47:22.576 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-05-22T15:47:22.580 DEBUG:teuthology.orchestra.run.smithi062:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-05-22T15:47:52.633 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-05-22T15:47:52.637 DEBUG:teuthology.orchestra.run.smithi062:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-05-22T15:48:00.166 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:48:00.162+0000 7f7935fec640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T15:48:00.166 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:48:00.162+0000 7f7935fec640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T15:48:00.166 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:48:00.162+0000 7f7935fec640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T15:48:00.167 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:48:00.162+0000 7f7935fec640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T15:48:00.167 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:48:00.162+0000 7f7935fec640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T15:48:22.680 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-05-22T15:48:22.683 DEBUG:teuthology.orchestra.run.smithi062:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-05-22T15:48:52.729 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-05-22T15:48:52.732 DEBUG:teuthology.orchestra.run.smithi062:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-05-22T15:49:22.835 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-05-22T15:49:22.840 DEBUG:teuthology.orchestra.run.smithi062:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-05-22T15:49:53.000 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-05-22T15:49:53.002 DEBUG:teuthology.orchestra.run.smithi062:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-05-22T15:50:23.055 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-05-22T15:50:23.058 DEBUG:teuthology.orchestra.run.smithi062:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-05-22T15:50:53.103 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-05-22T15:50:53.106 DEBUG:teuthology.orchestra.run.smithi062:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-05-22T15:51:23.178 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-05-22T15:51:23.181 DEBUG:teuthology.orchestra.run.smithi062:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-05-22T15:51:53.226 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-05-22T15:51:53.229 DEBUG:teuthology.orchestra.run.smithi062:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-05-22T15:52:23.274 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-05-22T15:52:23.277 DEBUG:teuthology.orchestra.run.smithi062:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-05-22T15:52:53.321 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-05-22T15:52:53.325 DEBUG:teuthology.orchestra.run.smithi062:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-05-22T15:53:23.369 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-05-22T15:53:23.372 DEBUG:teuthology.orchestra.run.smithi062:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-05-22T15:53:53.417 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-05-22T15:53:53.420 DEBUG:teuthology.orchestra.run.smithi062:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-05-22T15:54:23.466 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-05-22T15:54:23.469 DEBUG:teuthology.orchestra.run.smithi062:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-05-22T15:54:53.513 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-05-22T15:54:53.516 DEBUG:teuthology.orchestra.run.smithi062:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-05-22T15:55:23.561 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-05-22T15:55:23.565 DEBUG:teuthology.orchestra.run.smithi062:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-05-22T15:55:53.610 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-05-22T15:55:53.613 DEBUG:teuthology.orchestra.run.smithi062:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-05-22T15:56:23.792 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-05-22T15:56:23.795 DEBUG:teuthology.orchestra.run.smithi062:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-05-22T15:56:53.849 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-05-22T15:56:53.858 DEBUG:teuthology.orchestra.run.smithi062:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-05-22T15:57:23.912 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-05-22T15:57:23.916 DEBUG:teuthology.orchestra.run.smithi062:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-05-22T15:57:53.960 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-05-22T15:57:53.963 DEBUG:teuthology.orchestra.run.smithi062:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-05-22T15:58:16.173 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T15:58:16.174 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/test_restart/.snap) 2023-05-22T15:58:16.191 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_48_00_UTC 2023-05-22T15:58:16.192 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_49_00_UTC 2023-05-22T15:58:16.192 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_50_00_UTC 2023-05-22T15:58:16.192 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_51_00_UTC 2023-05-22T15:58:16.192 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_52_00_UTC 2023-05-22T15:58:16.193 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_53_00_UTC 2023-05-22T15:58:16.193 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_54_00_UTC 2023-05-22T15:58:16.193 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_55_00_UTC 2023-05-22T15:58:16.193 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_56_00_UTC 2023-05-22T15:58:16.193 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_57_00_UTC 2023-05-22T15:58:16.193 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_58_00_UTC 2023-05-22T15:58:16.193 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2023-05-22-15_48_00_UTC', 'scheduled-2023-05-22-15_49_00_UTC', 'scheduled-2023-05-22-15_50_00_UTC', 'scheduled-2023-05-22-15_51_00_UTC', 'scheduled-2023-05-22-15_52_00_UTC', 'scheduled-2023-05-22-15_53_00_UTC', 'scheduled-2023-05-22-15_54_00_UTC', 'scheduled-2023-05-22-15_55_00_UTC', 'scheduled-2023-05-22-15_56_00_UTC', 'scheduled-2023-05-22-15_57_00_UTC', 'scheduled-2023-05-22-15_58_00_UTC'] 2023-05-22T15:58:16.194 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --fs cephfs --format json /snap_test_dir1/test_restart 2023-05-22T15:58:16.370 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:58:16.366+0000 7f7631af1640 1 Processor -- start 2023-05-22T15:58:16.370 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:58:16.366+0000 7f7631af1640 1 -- start start 2023-05-22T15:58:16.371 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:58:16.366+0000 7f7631af1640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f762c15bcf0 0x7f762c0c33f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:58:16.372 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:58:16.366+0000 7f7631af1640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f762c157f70 0x7f762c0ce0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:58:16.372 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:58:16.366+0000 7f7631af1640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f762c0c30e0 con 0x7f762c158380 2023-05-22T15:58:16.372 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:58:16.366+0000 7f7631af1640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f762c06f770 con 0x7f762c157f70 2023-05-22T15:58:16.372 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:58:16.366+0000 7f7631af1640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f762c0c3a40 con 0x7f762c15bcf0 2023-05-22T15:58:16.373 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:58:16.366+0000 7f762affd640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f762c157f70 0x7f762c0ce0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:58:16.373 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:58:16.366+0000 7f762bfff640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f762c158380 0x7f762c0cbc70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:35760/0 (socket says 172.21.15.16:35760) 2023-05-22T15:58:16.373 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:58:16.366+0000 7f762bfff640 1 -- 172.21.15.16:0/2650448458 learned_addr learned my addr 172.21.15.16:0/2650448458 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T15:58:16.373 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:58:16.366+0000 7f762b7fe640 1 --2- 172.21.15.16:0/2650448458 >> v2:172.21.15.62:3300/0 conn(0x7f762c15bcf0 0x7f762c0c33f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:58:16.374 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:58:16.366+0000 7f762affd640 1 -- 172.21.15.16:0/2650448458 >> v2:172.21.15.62:3300/0 conn(0x7f762c15bcf0 msgr2=0x7f762c0c33f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:58:16.374 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:58:16.366+0000 7f762affd640 1 --2- 172.21.15.16:0/2650448458 >> v2:172.21.15.62:3300/0 conn(0x7f762c15bcf0 0x7f762c0c33f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:58:16.374 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:58:16.366+0000 7f762affd640 1 -- 172.21.15.16:0/2650448458 >> v1:172.21.15.16:6789/0 conn(0x7f762c158380 legacy=0x7f762c0cbc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:58:16.374 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:58:16.366+0000 7f762a7fc640 1 -- 172.21.15.16:0/2650448458 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4226345610 0 0) 0x7f762c0c30e0 con 0x7f762c158380 2023-05-22T15:58:16.374 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:58:16.366+0000 7f762affd640 1 -- 172.21.15.16:0/2650448458 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f762c0c3d20 con 0x7f762c157f70 2023-05-22T15:58:16.375 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:58:16.370+0000 7f762affd640 1 --2- 172.21.15.16:0/2650448458 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f762c157f70 0x7f762c0ce0a0 secure :-1 s=READY pgs=2098 cs=0 l=1 rev1=1 crypto rx=0x7f76140097c0 tx=0x7f761401c440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bdc0c4f97e876c73 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:58:16.376 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:58:16.370+0000 7f762a7fc640 1 -- 172.21.15.16:0/2650448458 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76140ab600 con 0x7f762c157f70 2023-05-22T15:58:16.376 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:58:16.370+0000 7f762a7fc640 1 -- 172.21.15.16:0/2650448458 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76140a7050 con 0x7f762c157f70 2023-05-22T15:58:16.376 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:58:16.370+0000 7f762a7fc640 1 -- 172.21.15.16:0/2650448458 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76140ab980 con 0x7f762c157f70 2023-05-22T15:58:16.376 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:58:16.370+0000 7f7631af1640 1 -- 172.21.15.16:0/2650448458 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f762c157f70 msgr2=0x7f762c0ce0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:58:16.376 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:58:16.370+0000 7f7631af1640 1 --2- 172.21.15.16:0/2650448458 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f762c157f70 0x7f762c0ce0a0 secure :-1 s=READY pgs=2098 cs=0 l=1 rev1=1 crypto rx=0x7f76140097c0 tx=0x7f761401c440 comp rx=0 tx=0).stop 2023-05-22T15:58:16.377 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:58:16.370+0000 7f7631af1640 1 -- 172.21.15.16:0/2650448458 shutdown_connections 2023-05-22T15:58:16.378 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:58:16.370+0000 7f7631af1640 1 --2- 172.21.15.16:0/2650448458 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f762c157f70 0x7f762c0ce0a0 unknown :-1 s=CLOSED pgs=2098 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:58:16.378 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:58:16.370+0000 7f7631af1640 1 --2- 172.21.15.16:0/2650448458 >> v2:172.21.15.62:3300/0 conn(0x7f762c15bcf0 0x7f762c0c33f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:58:16.378 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:58:16.370+0000 7f7631af1640 1 -- 172.21.15.16:0/2650448458 >> 172.21.15.16:0/2650448458 conn(0x7f762c069720 msgr2=0x7f762c069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:58:16.378 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:58:16.370+0000 7f7631af1640 1 -- 172.21.15.16:0/2650448458 shutdown_connections 2023-05-22T15:58:16.378 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:58:16.370+0000 7f7631af1640 1 -- 172.21.15.16:0/2650448458 wait complete. 2023-05-22T15:58:16.379 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:58:16.374+0000 7f7631af1640 1 Processor -- start 2023-05-22T15:58:16.379 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:58:16.374+0000 7f7631af1640 1 -- start start 2023-05-22T15:58:16.380 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:58:16.374+0000 7f7631af1640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f762c157f70 0x7f762c0c5dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:58:16.380 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:58:16.374+0000 7f7631af1640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f762c158380 0x7f762c141a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:58:16.381 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:58:16.374+0000 7f7631af1640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f762c15bcf0 0x7f762c153f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:58:16.381 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:58:16.374+0000 7f7631af1640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f762c0cb570 con 0x7f762c157f70 2023-05-22T15:58:16.381 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:58:16.374+0000 7f7631af1640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f762c153380 con 0x7f762c158380 2023-05-22T15:58:16.381 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:58:16.374+0000 7f7631af1640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f762c0c6310 con 0x7f762c15bcf0 2023-05-22T15:58:16.381 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:58:16.374+0000 7f762b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f762c157f70 0x7f762c0c5dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:58:16.381 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:58:16.374+0000 7f762b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f762c157f70 0x7f762c0c5dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42342/0 (socket says 172.21.15.16:42342) 2023-05-22T15:58:16.382 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:58:16.374+0000 7f762b7fe640 1 -- 172.21.15.16:0/3597894962 learned_addr learned my addr 172.21.15.16:0/3597894962 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:58:16.382 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:58:16.374+0000 7f762bfff640 1 --2- 172.21.15.16:0/3597894962 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f762c15bcf0 0x7f762c153f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:58:16.382 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:58:16.374+0000 7f762affd640 1 --2- 172.21.15.16:0/3597894962 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f762c158380 0x7f762c141a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:58:16.382 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:58:16.374+0000 7f762b7fe640 1 -- 172.21.15.16:0/3597894962 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f762c15bcf0 msgr2=0x7f762c153f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:58:16.383 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:58:16.374+0000 7f762b7fe640 1 --2- 172.21.15.16:0/3597894962 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f762c15bcf0 0x7f762c153f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:58:16.383 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:58:16.374+0000 7f762b7fe640 1 -- 172.21.15.16:0/3597894962 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f762c158380 msgr2=0x7f762c141a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:58:16.383 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:58:16.374+0000 7f762b7fe640 1 --2- 172.21.15.16:0/3597894962 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f762c158380 0x7f762c141a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:58:16.383 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:58:16.374+0000 7f762b7fe640 1 -- 172.21.15.16:0/3597894962 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7614009420 con 0x7f762c157f70 2023-05-22T15:58:16.383 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:58:16.374+0000 7f762affd640 1 --2- 172.21.15.16:0/3597894962 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f762c158380 0x7f762c141a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:58:16.383 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:58:16.374+0000 7f762b7fe640 1 --2- 172.21.15.16:0/3597894962 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f762c157f70 0x7f762c0c5dd0 secure :-1 s=READY pgs=2099 cs=0 l=1 rev1=1 crypto rx=0x7f761c007e40 tx=0x7f761c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:58:16.383 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:58:16.374+0000 7f7623fff640 1 -- 172.21.15.16:0/3597894962 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f761c014620 con 0x7f762c157f70 2023-05-22T15:58:16.383 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:58:16.374+0000 7f7623fff640 1 -- 172.21.15.16:0/3597894962 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f761c00ee90 con 0x7f762c157f70 2023-05-22T15:58:16.384 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:58:16.374+0000 7f7631af1640 1 -- 172.21.15.16:0/3597894962 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f762c0ce770 con 0x7f762c157f70 2023-05-22T15:58:16.384 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:58:16.374+0000 7f7623fff640 1 -- 172.21.15.16:0/3597894962 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f761c0149c0 con 0x7f762c157f70 2023-05-22T15:58:16.384 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:58:16.374+0000 7f7631af1640 1 -- 172.21.15.16:0/3597894962 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f762c0ce5e0 con 0x7f762c157f70 2023-05-22T15:58:16.384 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:58:16.374+0000 7f7631af1640 1 -- 172.21.15.16:0/3597894962 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f762c0c5150 con 0x7f762c157f70 2023-05-22T15:58:16.385 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:58:16.374+0000 7f7623fff640 1 -- 172.21.15.16:0/3597894962 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 73) v1 ==== 84361+0+0 (secure 0 0 0) 0x7f761c014b20 con 0x7f762c157f70 2023-05-22T15:58:16.385 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:58:16.378+0000 7f7623fff640 1 --2- 172.21.15.16:0/3597894962 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f760c0654f0 0x7f760c0679a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:58:16.385 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:58:16.378+0000 7f7623fff640 1 -- 172.21.15.16:0/3597894962 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(164..164 src has 1..164) v4 ==== 6783+0+0 (secure 0 0 0) 0x7f761c083740 con 0x7f762c157f70 2023-05-22T15:58:16.385 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:58:16.378+0000 7f762affd640 1 --2- 172.21.15.16:0/3597894962 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f760c0654f0 0x7f760c0679a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:58:16.386 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:58:16.378+0000 7f762affd640 1 --2- 172.21.15.16:0/3597894962 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f760c0654f0 0x7f760c0679a0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f76140ad000 tx=0x7f7614008040 comp rx=0 tx=0).ready entity=mgr.9831 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:58:16.386 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:58:16.382+0000 7f7623fff640 1 -- 172.21.15.16:0/3597894962 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f762c0c5150 con 0x7f762c157f70 2023-05-22T15:58:16.558 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:58:16.554+0000 7f7631af1640 1 -- 172.21.15.16:0/3597894962 --> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/snap_test_dir1/test_restart", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f762c0c5150 con 0x7f760c0654f0 2023-05-22T15:58:16.572 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:58:16.566+0000 7f7623fff640 1 -- 172.21.15.16:0/3597894962 <== mgr.9831 v2:172.21.15.16:6832/36921 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+360 (secure 0 0 0) 0x7f762c0c5150 con 0x7f760c0654f0 2023-05-22T15:58:16.573 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:58:16.573 INFO:teuthology.orchestra.run.smithi016.stdout:[{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/test_restart", "rel_path": "/snap_test_dir1/test_restart", "schedule": "1M", "retention": {}, "start": "2023-05-22T00:00:00", "created": "2023-05-22T15:47:17", "first": "2023-05-22T15:48:00", "last": "2023-05-22T15:58:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": true}] 2023-05-22T15:58:16.575 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:58:16.570+0000 7f7631af1640 1 -- 172.21.15.16:0/3597894962 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f760c0654f0 msgr2=0x7f760c0679a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:58:16.576 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:58:16.570+0000 7f7631af1640 1 --2- 172.21.15.16:0/3597894962 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f760c0654f0 0x7f760c0679a0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f76140ad000 tx=0x7f7614008040 comp rx=0 tx=0).stop 2023-05-22T15:58:16.576 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:58:16.570+0000 7f7631af1640 1 -- 172.21.15.16:0/3597894962 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f762c157f70 msgr2=0x7f762c0c5dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:58:16.576 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:58:16.570+0000 7f7631af1640 1 --2- 172.21.15.16:0/3597894962 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f762c157f70 0x7f762c0c5dd0 secure :-1 s=READY pgs=2099 cs=0 l=1 rev1=1 crypto rx=0x7f761c007e40 tx=0x7f761c00e4b0 comp rx=0 tx=0).stop 2023-05-22T15:58:16.576 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:58:16.570+0000 7f7631af1640 1 -- 172.21.15.16:0/3597894962 shutdown_connections 2023-05-22T15:58:16.576 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:58:16.570+0000 7f7631af1640 1 --2- 172.21.15.16:0/3597894962 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7f760c0654f0 0x7f760c0679a0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:58:16.576 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:58:16.570+0000 7f7631af1640 1 --2- 172.21.15.16:0/3597894962 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f762c15bcf0 0x7f762c153f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:58:16.577 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:58:16.570+0000 7f7631af1640 1 --2- 172.21.15.16:0/3597894962 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f762c158380 0x7f762c141a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:58:16.577 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:58:16.570+0000 7f7631af1640 1 --2- 172.21.15.16:0/3597894962 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f762c157f70 0x7f762c0c5dd0 unknown :-1 s=CLOSED pgs=2099 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:58:16.577 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:58:16.570+0000 7f7631af1640 1 -- 172.21.15.16:0/3597894962 >> 172.21.15.16:0/3597894962 conn(0x7f762c069720 msgr2=0x7f762c0b1fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:58:16.577 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:58:16.570+0000 7f7631af1640 1 -- 172.21.15.16:0/3597894962 shutdown_connections 2023-05-22T15:58:16.578 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:58:16.570+0000 7f7631af1640 1 -- 172.21.15.16:0/3597894962 wait complete. 2023-05-22T15:58:16.592 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/test_restart", "rel_path": "/snap_test_dir1/test_restart", "schedule": "1M", "retention": {}, "start": "2023-05-22T00:00:00", "created": "2023-05-22T15:47:17", "first": "2023-05-22T15:48:00", "last": "2023-05-22T15:58:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": true}] 2023-05-22T15:58:16.593 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'path': '/snap_test_dir1/test_restart', 'rel_path': '/snap_test_dir1/test_restart', 'schedule': '1M', 'retention': {}, 'start': '2023-05-22T00:00:00', 'created': '2023-05-22T15:47:17', 'first': '2023-05-22T15:48:00', 'last': '2023-05-22T15:58:00', 'last_pruned': None, 'created_count': 11, 'pruned_count': 0, 'active': True} 2023-05-22T15:58:16.593 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2023-05-22T15:58:16.712 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:58:16.706+0000 7fd055528640 1 Processor -- start 2023-05-22T15:58:16.712 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:58:16.706+0000 7fd055528640 1 -- start start 2023-05-22T15:58:16.714 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:58:16.706+0000 7fd055528640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd0501531b0 0x7fd0500afb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:58:16.714 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:58:16.706+0000 7fd055528640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd0500b0080 0x7fd050159330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:58:16.715 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:58:16.706+0000 7fd055528640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd05015a0a0 con 0x7fd050159870 2023-05-22T15:58:16.715 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:58:16.706+0000 7fd055528640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0500b04f0 con 0x7fd0500b0080 2023-05-22T15:58:16.715 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:58:16.706+0000 7fd055528640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd050153580 con 0x7fd0501531b0 2023-05-22T15:58:16.715 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:58:16.706+0000 7fd04e7fc640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd0500b0080 0x7fd050159330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:58:16.715 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:58:16.706+0000 7fd04e7fc640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd0500b0080 0x7fd050159330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42358/0 (socket says 172.21.15.16:42358) 2023-05-22T15:58:16.715 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:58:16.706+0000 7fd04e7fc640 1 -- 172.21.15.16:0/3256464310 learned_addr learned my addr 172.21.15.16:0/3256464310 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:58:16.716 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:58:16.706+0000 7fd04f7fe640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fd050159870 0x7fd0500c8ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:35770/0 (socket says 172.21.15.16:35770) 2023-05-22T15:58:16.716 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:58:16.710+0000 7fd04effd640 1 --2- 172.21.15.16:0/3256464310 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd0501531b0 0x7fd0500afb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:58:16.716 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:58:16.710+0000 7fd04e7fc640 1 -- 172.21.15.16:0/3256464310 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd0501531b0 msgr2=0x7fd0500afb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T15:58:16.716 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:58:16.710+0000 7fd04e7fc640 1 --2- 172.21.15.16:0/3256464310 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd0501531b0 0x7fd0500afb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:58:16.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:58:16.710+0000 7fd04e7fc640 1 -- 172.21.15.16:0/3256464310 >> v1:172.21.15.16:6789/0 conn(0x7fd050159870 legacy=0x7fd0500c8ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:58:16.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:58:16.710+0000 7fd04e7fc640 1 -- 172.21.15.16:0/3256464310 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd05015d5e0 con 0x7fd0500b0080 2023-05-22T15:58:16.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:58:16.710+0000 7fd04dffb640 1 -- 172.21.15.16:0/3256464310 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2196184459 0 0) 0x7fd05015a0a0 con 0x7fd050159870 2023-05-22T15:58:16.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:58:16.710+0000 7fd04effd640 1 --2- 172.21.15.16:0/3256464310 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd0501531b0 0x7fd0500afb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T15:58:16.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:58:16.710+0000 7fd04e7fc640 1 --2- 172.21.15.16:0/3256464310 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd0500b0080 0x7fd050159330 secure :-1 s=READY pgs=2101 cs=0 l=1 rev1=1 crypto rx=0x7fd044009870 tx=0x7fd04401c510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7b5502a82724920 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:58:16.719 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:58:16.710+0000 7fd04dffb640 1 -- 172.21.15.16:0/3256464310 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0440aa670 con 0x7fd0500b0080 2023-05-22T15:58:16.719 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:58:16.710+0000 7fd04dffb640 1 -- 172.21.15.16:0/3256464310 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0440aa7d0 con 0x7fd0500b0080 2023-05-22T15:58:16.719 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:58:16.710+0000 7fd04dffb640 1 -- 172.21.15.16:0/3256464310 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0440aab60 con 0x7fd0500b0080 2023-05-22T15:58:16.719 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:58:16.710+0000 7fd055528640 1 -- 172.21.15.16:0/3256464310 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd0500b0080 msgr2=0x7fd050159330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:58:16.720 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:58:16.710+0000 7fd055528640 1 --2- 172.21.15.16:0/3256464310 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd0500b0080 0x7fd050159330 secure :-1 s=READY pgs=2101 cs=0 l=1 rev1=1 crypto rx=0x7fd044009870 tx=0x7fd04401c510 comp rx=0 tx=0).stop 2023-05-22T15:58:16.720 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:58:16.710+0000 7fd055528640 1 -- 172.21.15.16:0/3256464310 shutdown_connections 2023-05-22T15:58:16.720 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:58:16.710+0000 7fd055528640 1 --2- 172.21.15.16:0/3256464310 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd0500b0080 0x7fd050159330 unknown :-1 s=CLOSED pgs=2101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:58:16.720 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:58:16.710+0000 7fd055528640 1 --2- 172.21.15.16:0/3256464310 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd0501531b0 0x7fd0500afb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:58:16.721 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:58:16.710+0000 7fd055528640 1 -- 172.21.15.16:0/3256464310 >> 172.21.15.16:0/3256464310 conn(0x7fd050069720 msgr2=0x7fd050069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:58:16.721 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:58:16.710+0000 7fd055528640 1 -- 172.21.15.16:0/3256464310 shutdown_connections 2023-05-22T15:58:16.722 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:58:16.710+0000 7fd055528640 1 -- 172.21.15.16:0/3256464310 wait complete. 2023-05-22T15:58:16.722 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:58:16.714+0000 7fd055528640 1 Processor -- start 2023-05-22T15:58:16.722 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:58:16.714+0000 7fd055528640 1 -- start start 2023-05-22T15:58:16.725 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:58:16.714+0000 7fd055528640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd0500b0080 0x7fd0500af240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:58:16.725 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:58:16.714+0000 7fd055528640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd0501531b0 0x7fd0500701b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:58:16.725 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:58:16.714+0000 7fd055528640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd050159870 0x7fd050143070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:58:16.725 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:58:16.714+0000 7fd055528640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd050149590 con 0x7fd0500b0080 2023-05-22T15:58:16.726 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:58:16.714+0000 7fd055528640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd050149b50 con 0x7fd0501531b0 2023-05-22T15:58:16.726 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:58:16.714+0000 7fd055528640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd050144af0 con 0x7fd050159870 2023-05-22T15:58:16.726 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:58:16.714+0000 7fd04effd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd0500b0080 0x7fd0500af240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:58:16.726 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:58:16.714+0000 7fd04effd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd0500b0080 0x7fd0500af240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42366/0 (socket says 172.21.15.16:42366) 2023-05-22T15:58:16.726 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:58:16.714+0000 7fd04effd640 1 -- 172.21.15.16:0/3093783422 learned_addr learned my addr 172.21.15.16:0/3093783422 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T15:58:16.726 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:58:16.714+0000 7fd04e7fc640 1 --2- 172.21.15.16:0/3093783422 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd0501531b0 0x7fd0500701b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:58:16.726 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:58:16.714+0000 7fd04f7fe640 1 --2- 172.21.15.16:0/3093783422 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd050159870 0x7fd050143070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:58:16.727 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:58:16.714+0000 7fd04effd640 1 -- 172.21.15.16:0/3093783422 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd050159870 msgr2=0x7fd050143070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:58:16.727 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:58:16.714+0000 7fd04effd640 1 --2- 172.21.15.16:0/3093783422 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd050159870 0x7fd050143070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:58:16.727 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:58:16.714+0000 7fd04effd640 1 -- 172.21.15.16:0/3093783422 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd0501531b0 msgr2=0x7fd0500701b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:58:16.727 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:58:16.714+0000 7fd04effd640 1 --2- 172.21.15.16:0/3093783422 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd0501531b0 0x7fd0500701b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:58:16.727 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:58:16.714+0000 7fd04effd640 1 -- 172.21.15.16:0/3093783422 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0440094d0 con 0x7fd0500b0080 2023-05-22T15:58:16.727 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:58:16.714+0000 7fd04f7fe640 1 --2- 172.21.15.16:0/3093783422 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd050159870 0x7fd050143070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T15:58:16.727 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:58:16.714+0000 7fd04effd640 1 --2- 172.21.15.16:0/3093783422 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd0500b0080 0x7fd0500af240 secure :-1 s=READY pgs=2102 cs=0 l=1 rev1=1 crypto rx=0x7fd04000d830 tx=0x7fd04000dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:58:16.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:58:16.714+0000 7fd0377fe640 1 -- 172.21.15.16:0/3093783422 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd04000b6c0 con 0x7fd0500b0080 2023-05-22T15:58:16.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:58:16.714+0000 7fd0377fe640 1 -- 172.21.15.16:0/3093783422 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd04000b820 con 0x7fd0500b0080 2023-05-22T15:58:16.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:58:16.714+0000 7fd055528640 1 -- 172.21.15.16:0/3093783422 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd05015d0c0 con 0x7fd0500b0080 2023-05-22T15:58:16.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:58:16.714+0000 7fd0377fe640 1 -- 172.21.15.16:0/3093783422 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd04000bbd0 con 0x7fd0500b0080 2023-05-22T15:58:16.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:58:16.714+0000 7fd055528640 1 -- 172.21.15.16:0/3093783422 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd05015cde0 con 0x7fd0500b0080 2023-05-22T15:58:16.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:58:16.718+0000 7fd055528640 1 -- 172.21.15.16:0/3093783422 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd050159c70 con 0x7fd0500b0080 2023-05-22T15:58:16.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:58:16.718+0000 7fd0377fe640 1 -- 172.21.15.16:0/3093783422 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 73) v1 ==== 84361+0+0 (secure 0 0 0) 0x7fd04001e460 con 0x7fd0500b0080 2023-05-22T15:58:16.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:58:16.718+0000 7fd0377fe640 1 --2- 172.21.15.16:0/3093783422 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd03c0654f0 0x7fd03c0679a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T15:58:16.729 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:58:16.718+0000 7fd0377fe640 1 -- 172.21.15.16:0/3093783422 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(164..164 src has 1..164) v4 ==== 6783+0+0 (secure 0 0 0) 0x7fd0400833c0 con 0x7fd0500b0080 2023-05-22T15:58:16.729 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:58:16.718+0000 7fd04e7fc640 1 --2- 172.21.15.16:0/3093783422 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd03c0654f0 0x7fd03c0679a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T15:58:16.729 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:58:16.718+0000 7fd04e7fc640 1 --2- 172.21.15.16:0/3093783422 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd03c0654f0 0x7fd03c0679a0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fd0440ae000 tx=0x7fd044008040 comp rx=0 tx=0).ready entity=mgr.9831 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T15:58:16.729 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:58:16.722+0000 7fd0377fe640 1 -- 172.21.15.16:0/3093783422 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fd040050c90 con 0x7fd0500b0080 2023-05-22T15:58:16.917 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:58:16.910+0000 7fd055528640 1 -- 172.21.15.16:0/3093783422 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7fd05004dbe0 con 0x7fd0500b0080 2023-05-22T15:58:16.924 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:58:16.918+0000 7fd0377fe640 1 -- 172.21.15.16:0/3093783422 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v73) v1 ==== 74+0+162348 (secure 0 0 0) 0x7fd0400542b0 con 0x7fd0500b0080 2023-05-22T15:58:16.926 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T15:58:16.927 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:58:16.922+0000 7fd055528640 1 -- 172.21.15.16:0/3093783422 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd03c0654f0 msgr2=0x7fd03c0679a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:58:16.927 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:58:16.922+0000 7fd055528640 1 --2- 172.21.15.16:0/3093783422 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd03c0654f0 0x7fd03c0679a0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fd0440ae000 tx=0x7fd044008040 comp rx=0 tx=0).stop 2023-05-22T15:58:16.928 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:58:16.922+0000 7fd055528640 1 -- 172.21.15.16:0/3093783422 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd0500b0080 msgr2=0x7fd0500af240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T15:58:16.928 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:58:16.922+0000 7fd055528640 1 --2- 172.21.15.16:0/3093783422 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd0500b0080 0x7fd0500af240 secure :-1 s=READY pgs=2102 cs=0 l=1 rev1=1 crypto rx=0x7fd04000d830 tx=0x7fd04000dcf0 comp rx=0 tx=0).stop 2023-05-22T15:58:16.928 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:58:16.922+0000 7fd055528640 1 -- 172.21.15.16:0/3093783422 shutdown_connections 2023-05-22T15:58:16.928 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:58:16.922+0000 7fd055528640 1 --2- 172.21.15.16:0/3093783422 >> [v2:172.21.15.16:6832/36921,v1:172.21.15.16:6833/36921] conn(0x7fd03c0654f0 0x7fd03c0679a0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:58:16.928 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:58:16.922+0000 7fd055528640 1 --2- 172.21.15.16:0/3093783422 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd050159870 0x7fd050143070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:58:16.928 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:58:16.922+0000 7fd055528640 1 --2- 172.21.15.16:0/3093783422 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd0501531b0 0x7fd0500701b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:58:16.929 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:58:16.922+0000 7fd055528640 1 --2- 172.21.15.16:0/3093783422 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd0500b0080 0x7fd0500af240 unknown :-1 s=CLOSED pgs=2102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T15:58:16.929 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:58:16.922+0000 7fd055528640 1 -- 172.21.15.16:0/3093783422 >> 172.21.15.16:0/3093783422 conn(0x7fd050069720 msgr2=0x7fd050069af0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T15:58:16.929 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:58:16.922+0000 7fd055528640 1 -- 172.21.15.16:0/3093783422 shutdown_connections 2023-05-22T15:58:16.929 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T15:58:16.922+0000 7fd055528640 1 -- 172.21.15.16:0/3093783422 wait complete. 2023-05-22T15:58:16.942 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":73,"active_gid":9831,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":36921},{"type":"v1","addr":"172.21.15.16:6833","nonce":36921}]},"active_addr":"172.21.15.16:6833/36921","active_change":"2023-05-22T15:47:16.536582+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[{"gid":7400,"name":"x","mgr_features":4540138322906710015,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"alertmanager_web_password":{"name":"alertmanager_web_password","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Alertmanager web password","long_desc":"","tags":[],"see_also":[]},"alertmanager_web_user":{"name":"alertmanager_web_user","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Alertmanager web user","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"prometheus_web_password":{"name":"prometheus_web_password","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Prometheus web password","long_desc":"","tags":[],"see_also":[]},"prometheus_web_user":{"name":"prometheus_web_user","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Prometheus web user","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful","snap_schedule"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"alertmanager_web_password":{"name":"alertmanager_web_password","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Alertmanager web password","long_desc":"","tags":[],"see_also":[]},"alertmanager_web_user":{"name":"alertmanager_web_user","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Alertmanager web user","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"prometheus_web_password":{"name":"prometheus_web_password","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Prometheus web password","long_desc":"","tags":[],"see_also":[]},"prometheus_web_user":{"name":"prometheus_web_user","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Prometheus web user","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":162,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.16:0","nonce":3082249167}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.16:0","nonce":2019135485}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.16:0","nonce":3417178745}]},{"name":"snap_schedule","addrvec":[{"type":"v2","addr":"172.21.15.16:0","nonce":4113149961}]},{"name":"snap_schedule","addrvec":[{"type":"v2","addr":"172.21.15.16:0","nonce":2592954608}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.16:0","nonce":2582627335}]}]} 2023-05-22T15:58:16.994 DEBUG:tasks.cephfs.test_snap_schedules:restarting active mgr: y 2023-05-22T15:58:16.994 INFO:tasks.ceph.mgr.y:Restarting daemon 2023-05-22T15:58:16.994 INFO:tasks.ceph.mgr.y:Stopping old one... 2023-05-22T15:58:16.995 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2023-05-22T15:58:16.995 INFO:teuthology.orchestra.run:waiting for 300 2023-05-22T15:58:17.025 INFO:tasks.ceph.mgr.y:Stopped 2023-05-22T15:58:17.025 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2023-05-22T15:58:17.071 INFO:tasks.ceph.mgr.y:Started 2023-05-22T15:58:17.206 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:58:17.202+0000 7f50a8628180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-05-22T15:58:17.586 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:58:17.582+0000 7f50a8628180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-05-22T15:58:17.721 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:58:17.718+0000 7f50a8628180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-05-22T15:58:18.162 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:58:18.158+0000 7f50a8628180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-05-22T15:58:18.215 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:58:18.210+0000 7f50a8628180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-05-22T15:58:18.265 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:58:18.262+0000 7f50a8628180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-05-22T15:58:18.344 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:58:18.338+0000 7f50a8628180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-05-22T15:58:18.395 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:58:18.390+0000 7f50a8628180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-05-22T15:58:18.473 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:58:18.470+0000 7f50a8628180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-05-22T15:58:18.525 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:58:18.518+0000 7f50a8628180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-05-22T15:58:18.657 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:58:18.654+0000 7f50a8628180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-05-22T15:58:18.710 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:58:18.706+0000 7f50a8628180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-05-22T15:58:18.935 INFO:tasks.ceph.mgr.y.smithi016.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2023-05-22T15:58:18.935 INFO:tasks.ceph.mgr.y.smithi016.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2023-05-22T15:58:18.935 INFO:tasks.ceph.mgr.y.smithi016.stderr: from numpy import show_config as show_numpy_config 2023-05-22T15:58:18.943 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:58:18.938+0000 7f50a8628180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-05-22T15:58:19.068 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:58:19.062+0000 7f50a8628180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-05-22T15:58:19.182 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:58:19.178+0000 7f50a8628180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-05-22T15:58:19.245 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:58:19.238+0000 7f50a8628180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-05-22T15:58:19.370 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:58:19.366+0000 7f50a8628180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-05-22T15:58:19.869 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:58:19.862+0000 7f50a8628180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-05-22T15:58:19.942 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:58:19.938+0000 7f50a8628180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-05-22T15:58:20.079 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:58:20.074+0000 7f50a8628180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-05-22T15:58:20.155 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:58:20.150+0000 7f50a8628180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-05-22T15:58:20.404 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:58:20.398+0000 7f50a8628180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-05-22T15:58:20.460 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:58:20.454+0000 7f50a8628180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-05-22T15:58:20.682 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:58:20.678+0000 7f5026f9f640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T15:58:20.682 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:58:20.678+0000 7f5026f9f640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T15:58:20.682 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:58:20.678+0000 7f5026f9f640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T15:58:20.683 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:58:20.678+0000 7f5026f9f640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T15:58:20.683 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:58:20.678+0000 7f5026f9f640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T15:58:20.684 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:58:20.678+0000 7f50297a4640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T15:58:20.684 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:58:20.678+0000 7f50297a4640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T15:58:20.684 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:58:20.678+0000 7f50297a4640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T15:58:20.685 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:58:20.678+0000 7f50297a4640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T15:58:20.685 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:58:20.678+0000 7f50297a4640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T15:58:24.020 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-05-22T15:58:24.023 DEBUG:teuthology.orchestra.run.smithi062:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-05-22T15:58:54.073 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-05-22T15:58:54.075 DEBUG:teuthology.orchestra.run.smithi062:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-05-22T15:59:00.691 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:59:00.689+0000 7f5031fb5640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T15:59:00.691 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:59:00.689+0000 7f5031fb5640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T15:59:00.691 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:59:00.689+0000 7f5031fb5640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T15:59:00.691 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:59:00.689+0000 7f5031fb5640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T15:59:00.692 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T15:59:00.689+0000 7f5031fb5640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T15:59:24.119 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-05-22T15:59:24.123 DEBUG:teuthology.orchestra.run.smithi062:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-05-22T15:59:54.169 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-05-22T15:59:54.172 DEBUG:teuthology.orchestra.run.smithi062:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-05-22T16:00:24.225 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-05-22T16:00:24.228 DEBUG:teuthology.orchestra.run.smithi062:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-05-22T16:00:54.279 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-05-22T16:00:54.282 DEBUG:teuthology.orchestra.run.smithi062:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-05-22T16:01:24.353 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-05-22T16:01:24.356 DEBUG:teuthology.orchestra.run.smithi062:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-05-22T16:01:54.408 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-05-22T16:01:54.412 DEBUG:teuthology.orchestra.run.smithi062:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-05-22T16:02:24.466 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-05-22T16:02:24.469 DEBUG:teuthology.orchestra.run.smithi062:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-05-22T16:02:54.520 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-05-22T16:02:54.523 DEBUG:teuthology.orchestra.run.smithi062:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-05-22T16:03:17.072 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate --fs cephfs /snap_test_dir1/test_restart 1M 2023-05-22T16:03:17.215 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:17.210+0000 7f940e5ee640 1 Processor -- start 2023-05-22T16:03:17.215 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:17.210+0000 7f940e5ee640 1 -- start start 2023-05-22T16:03:17.217 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:17.214+0000 7f940e5ee640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f94080ce530 0x7f940815c540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:17.217 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:17.214+0000 7f940e5ee640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f94080cbe80 0x7f94080cc680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:17.217 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:17.214+0000 7f940e5ee640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94080c3930 con 0x7f94080ce900 2023-05-22T16:03:17.217 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:17.214+0000 7f940e5ee640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9408158f00 con 0x7f94080cbe80 2023-05-22T16:03:17.217 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:17.214+0000 7f940e5ee640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9408159d90 con 0x7f94080ce530 2023-05-22T16:03:17.218 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:17.214+0000 7f940cb8b640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f94080cbe80 0x7f94080cc680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:17.218 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:17.214+0000 7f940cb8b640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f94080cbe80 0x7f94080cc680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57602/0 (socket says 172.21.15.16:57602) 2023-05-22T16:03:17.218 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:17.214+0000 7f940cb8b640 1 -- 172.21.15.16:0/529905567 learned_addr learned my addr 172.21.15.16:0/529905567 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:03:17.218 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:17.214+0000 7f94077fe640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f94080ce900 0x7f9408159680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:57834/0 (socket says 172.21.15.16:57834) 2023-05-22T16:03:17.218 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:17.214+0000 7f9407fff640 1 --2- 172.21.15.16:0/529905567 >> v2:172.21.15.62:3300/0 conn(0x7f94080ce530 0x7f940815c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:17.218 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:17.214+0000 7f940cb8b640 1 -- 172.21.15.16:0/529905567 >> v2:172.21.15.62:3300/0 conn(0x7f94080ce530 msgr2=0x7f940815c540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:03:17.219 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:17.214+0000 7f940cb8b640 1 --2- 172.21.15.16:0/529905567 >> v2:172.21.15.62:3300/0 conn(0x7f94080ce530 0x7f940815c540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:17.219 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:17.214+0000 7f940cb8b640 1 -- 172.21.15.16:0/529905567 >> v1:172.21.15.16:6789/0 conn(0x7f94080ce900 legacy=0x7f9408159680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:17.219 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:17.214+0000 7f9406ffd640 1 -- 172.21.15.16:0/529905567 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3781052965 0 0) 0x7f93fc001120 con 0x7f94080ce900 2023-05-22T16:03:17.219 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:17.214+0000 7f940cb8b640 1 -- 172.21.15.16:0/529905567 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f940815cc40 con 0x7f94080cbe80 2023-05-22T16:03:17.220 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:17.214+0000 7f940cb8b640 1 --2- 172.21.15.16:0/529905567 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f94080cbe80 0x7f94080cc680 secure :-1 s=READY pgs=2123 cs=0 l=1 rev1=1 crypto rx=0x7f94080cf160 tx=0x7f940001c9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d79fd80b9fc5c37d server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:03:17.220 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:17.214+0000 7f9406ffd640 1 -- 172.21.15.16:0/529905567 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94000a8440 con 0x7f94080cbe80 2023-05-22T16:03:17.220 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:17.214+0000 7f9406ffd640 1 -- 172.21.15.16:0/529905567 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94000a85a0 con 0x7f94080cbe80 2023-05-22T16:03:17.221 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:17.214+0000 7f9406ffd640 1 -- 172.21.15.16:0/529905567 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94000a8930 con 0x7f94080cbe80 2023-05-22T16:03:17.221 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:17.214+0000 7f940e5ee640 1 -- 172.21.15.16:0/529905567 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f94080cbe80 msgr2=0x7f94080cc680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:17.221 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:17.214+0000 7f940e5ee640 1 --2- 172.21.15.16:0/529905567 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f94080cbe80 0x7f94080cc680 secure :-1 s=READY pgs=2123 cs=0 l=1 rev1=1 crypto rx=0x7f94080cf160 tx=0x7f940001c9d0 comp rx=0 tx=0).stop 2023-05-22T16:03:17.221 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:17.214+0000 7f940e5ee640 1 -- 172.21.15.16:0/529905567 shutdown_connections 2023-05-22T16:03:17.221 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:17.214+0000 7f940e5ee640 1 --2- 172.21.15.16:0/529905567 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f94080cbe80 0x7f94080cc680 unknown :-1 s=CLOSED pgs=2123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:17.221 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:17.214+0000 7f940e5ee640 1 --2- 172.21.15.16:0/529905567 >> v2:172.21.15.62:3300/0 conn(0x7f94080ce530 0x7f940815c540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:17.221 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:17.214+0000 7f940e5ee640 1 -- 172.21.15.16:0/529905567 >> 172.21.15.16:0/529905567 conn(0x7f9408069720 msgr2=0x7f9408069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:03:17.222 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:17.214+0000 7f940e5ee640 1 -- 172.21.15.16:0/529905567 shutdown_connections 2023-05-22T16:03:17.222 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:17.214+0000 7f940e5ee640 1 -- 172.21.15.16:0/529905567 wait complete. 2023-05-22T16:03:17.223 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:17.218+0000 7f940e5ee640 1 Processor -- start 2023-05-22T16:03:17.223 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:17.218+0000 7f940e5ee640 1 -- start start 2023-05-22T16:03:17.223 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:17.218+0000 7f940e5ee640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f94080cbe80 0x7f94080afa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:17.224 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:17.218+0000 7f940e5ee640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f94080ce530 0x7f94080b2e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:17.224 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:17.218+0000 7f940e5ee640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f94080ce900 0x7f9408152a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:17.224 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:17.218+0000 7f940e5ee640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94081487d0 con 0x7f94080cbe80 2023-05-22T16:03:17.224 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:17.218+0000 7f940e5ee640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9408148a40 con 0x7f94080ce530 2023-05-22T16:03:17.224 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:17.218+0000 7f940e5ee640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9408158310 con 0x7f94080ce900 2023-05-22T16:03:17.224 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:17.218+0000 7f9407fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f94080cbe80 0x7f94080afa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:17.224 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:17.218+0000 7f9407fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f94080cbe80 0x7f94080afa30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57614/0 (socket says 172.21.15.16:57614) 2023-05-22T16:03:17.225 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:17.218+0000 7f9407fff640 1 -- 172.21.15.16:0/1046150765 learned_addr learned my addr 172.21.15.16:0/1046150765 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:03:17.225 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:17.218+0000 7f94077fe640 1 --2- 172.21.15.16:0/1046150765 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f94080ce530 0x7f94080b2e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:17.225 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:17.218+0000 7f940cb8b640 1 --2- 172.21.15.16:0/1046150765 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f94080ce900 0x7f9408152a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:17.225 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:17.218+0000 7f9407fff640 1 -- 172.21.15.16:0/1046150765 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f94080ce900 msgr2=0x7f9408152a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:17.225 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:17.218+0000 7f9407fff640 1 --2- 172.21.15.16:0/1046150765 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f94080ce900 0x7f9408152a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:17.225 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:17.218+0000 7f9407fff640 1 -- 172.21.15.16:0/1046150765 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f94080ce530 msgr2=0x7f94080b2e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:17.225 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:17.218+0000 7f9407fff640 1 --2- 172.21.15.16:0/1046150765 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f94080ce530 0x7f94080b2e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:17.225 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:17.218+0000 7f9407fff640 1 -- 172.21.15.16:0/1046150765 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9400009b30 con 0x7f94080cbe80 2023-05-22T16:03:17.226 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:17.218+0000 7f9407fff640 1 --2- 172.21.15.16:0/1046150765 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f94080cbe80 0x7f94080afa30 secure :-1 s=READY pgs=2124 cs=0 l=1 rev1=1 crypto rx=0x7f93f800d800 tx=0x7f93f800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:03:17.226 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:17.218+0000 7f9404ff9640 1 -- 172.21.15.16:0/1046150765 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93f800b6c0 con 0x7f94080cbe80 2023-05-22T16:03:17.226 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:17.218+0000 7f9404ff9640 1 -- 172.21.15.16:0/1046150765 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93f800b860 con 0x7f94080cbe80 2023-05-22T16:03:17.226 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:17.218+0000 7f940e5ee640 1 -- 172.21.15.16:0/1046150765 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f940814f2f0 con 0x7f94080cbe80 2023-05-22T16:03:17.226 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:17.218+0000 7f9404ff9640 1 -- 172.21.15.16:0/1046150765 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93f800bc10 con 0x7f94080cbe80 2023-05-22T16:03:17.226 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:17.218+0000 7f940e5ee640 1 -- 172.21.15.16:0/1046150765 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9408159160 con 0x7f94080cbe80 2023-05-22T16:03:17.227 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:17.218+0000 7f940e5ee640 1 -- 172.21.15.16:0/1046150765 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f940814f480 con 0x7f94080cbe80 2023-05-22T16:03:17.227 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:17.222+0000 7f9404ff9640 1 -- 172.21.15.16:0/1046150765 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 78) v1 ==== 84361+0+0 (secure 0 0 0) 0x7f93f801e460 con 0x7f94080cbe80 2023-05-22T16:03:17.227 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:17.222+0000 7f9404ff9640 1 --2- 172.21.15.16:0/1046150765 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f93f40654f0 0x7f93f40679a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:17.227 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:17.222+0000 7f9404ff9640 1 -- 172.21.15.16:0/1046150765 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(165..165 src has 1..165) v4 ==== 7041+0+0 (secure 0 0 0) 0x7f93f8083420 con 0x7f94080cbe80 2023-05-22T16:03:17.228 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:17.222+0000 7f94077fe640 1 --2- 172.21.15.16:0/1046150765 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f93f40654f0 0x7f93f40679a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:17.228 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:17.222+0000 7f94077fe640 1 --2- 172.21.15.16:0/1046150765 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f93f40654f0 0x7f93f40679a0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f94080b3fb0 tx=0x7f93fc007c10 comp rx=0 tx=0).ready entity=mgr.9933 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:03:17.229 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:17.226+0000 7f9404ff9640 1 -- 172.21.15.16:0/1046150765 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f93f8050b50 con 0x7f94080cbe80 2023-05-22T16:03:17.404 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:17.398+0000 7f940e5ee640 1 -- 172.21.15.16:0/1046150765 --> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "fs": "cephfs", "path": "/snap_test_dir1/test_restart", "repeat": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f940804dbe0 con 0x7f93f40654f0 2023-05-22T16:03:17.537 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:17.534+0000 7f9404ff9640 1 -- 172.21.15.16:0/1046150765 <== mgr.9933 v2:172.21.15.16:6832/65890 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+59 (secure 0 0 0) 0x7f940804dbe0 con 0x7f93f40654f0 2023-05-22T16:03:17.537 INFO:teuthology.orchestra.run.smithi016.stdout:Schedule deactivated for path /snap_test_dir1/test_restart 2023-05-22T16:03:17.541 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:17.538+0000 7f940e5ee640 1 -- 172.21.15.16:0/1046150765 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f93f40654f0 msgr2=0x7f93f40679a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:17.541 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:17.538+0000 7f940e5ee640 1 --2- 172.21.15.16:0/1046150765 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f93f40654f0 0x7f93f40679a0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f94080b3fb0 tx=0x7f93fc007c10 comp rx=0 tx=0).stop 2023-05-22T16:03:17.541 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:17.538+0000 7f940e5ee640 1 -- 172.21.15.16:0/1046150765 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f94080cbe80 msgr2=0x7f94080afa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:17.541 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:17.538+0000 7f940e5ee640 1 --2- 172.21.15.16:0/1046150765 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f94080cbe80 0x7f94080afa30 secure :-1 s=READY pgs=2124 cs=0 l=1 rev1=1 crypto rx=0x7f93f800d800 tx=0x7f93f800dcc0 comp rx=0 tx=0).stop 2023-05-22T16:03:17.541 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:17.538+0000 7f940e5ee640 1 -- 172.21.15.16:0/1046150765 shutdown_connections 2023-05-22T16:03:17.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:17.538+0000 7f940e5ee640 1 --2- 172.21.15.16:0/1046150765 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f94080ce900 0x7f9408152a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:17.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:17.538+0000 7f940e5ee640 1 --2- 172.21.15.16:0/1046150765 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f94080ce530 0x7f94080b2e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:17.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:17.538+0000 7f940e5ee640 1 --2- 172.21.15.16:0/1046150765 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f93f40654f0 0x7f93f40679a0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:17.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:17.538+0000 7f940e5ee640 1 --2- 172.21.15.16:0/1046150765 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f94080cbe80 0x7f94080afa30 unknown :-1 s=CLOSED pgs=2124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:17.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:17.538+0000 7f940e5ee640 1 -- 172.21.15.16:0/1046150765 >> 172.21.15.16:0/1046150765 conn(0x7f9408069720 msgr2=0x7f94080705e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:03:17.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:17.538+0000 7f940e5ee640 1 -- 172.21.15.16:0/1046150765 shutdown_connections 2023-05-22T16:03:17.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:17.538+0000 7f940e5ee640 1 -- 172.21.15.16:0/1046150765 wait complete. 2023-05-22T16:03:17.565 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/test_restart 2023-05-22T16:03:17.566 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T16:03:17.566 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/test_restart/.snap) 2023-05-22T16:03:17.578 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_48_00_UTC 2023-05-22T16:03:17.579 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_49_00_UTC 2023-05-22T16:03:17.579 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_50_00_UTC 2023-05-22T16:03:17.579 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_51_00_UTC 2023-05-22T16:03:17.579 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_52_00_UTC 2023-05-22T16:03:17.579 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_53_00_UTC 2023-05-22T16:03:17.580 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_54_00_UTC 2023-05-22T16:03:17.580 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_55_00_UTC 2023-05-22T16:03:17.580 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_56_00_UTC 2023-05-22T16:03:17.580 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_57_00_UTC 2023-05-22T16:03:17.580 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_58_00_UTC 2023-05-22T16:03:17.580 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_59_00_UTC 2023-05-22T16:03:17.580 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-16_00_00_UTC 2023-05-22T16:03:17.580 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-16_01_00_UTC 2023-05-22T16:03:17.581 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-16_02_00_UTC 2023-05-22T16:03:17.581 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-16_03_00_UTC 2023-05-22T16:03:17.582 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2023-05-22-15_48_00_UTC', 'scheduled-2023-05-22-15_49_00_UTC', 'scheduled-2023-05-22-15_50_00_UTC', 'scheduled-2023-05-22-15_51_00_UTC', 'scheduled-2023-05-22-15_52_00_UTC', 'scheduled-2023-05-22-15_53_00_UTC', 'scheduled-2023-05-22-15_54_00_UTC', 'scheduled-2023-05-22-15_55_00_UTC', 'scheduled-2023-05-22-15_56_00_UTC', 'scheduled-2023-05-22-15_57_00_UTC', 'scheduled-2023-05-22-15_58_00_UTC', 'scheduled-2023-05-22-15_59_00_UTC', 'scheduled-2023-05-22-16_00_00_UTC', 'scheduled-2023-05-22-16_01_00_UTC', 'scheduled-2023-05-22-16_02_00_UTC', 'scheduled-2023-05-22-16_03_00_UTC'] 2023-05-22T16:03:17.582 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --fs cephfs --format json /snap_test_dir1/test_restart 2023-05-22T16:03:17.762 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:17.758+0000 7f5e880d3640 1 Processor -- start 2023-05-22T16:03:17.763 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:17.758+0000 7f5e880d3640 1 -- start start 2023-05-22T16:03:17.763 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:17.758+0000 7f5e880d3640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5e800cc4e0 0x7f5e8006bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:17.764 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:17.758+0000 7f5e880d3640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f5e8015d650 0x7f5e8015dad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:17.764 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:17.758+0000 7f5e880d3640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e800cc930 con 0x7f5e80070c10 2023-05-22T16:03:17.764 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:17.758+0000 7f5e880d3640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e8015e010 con 0x7f5e8015d650 2023-05-22T16:03:17.764 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:17.758+0000 7f5e880d3640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e800ccbb0 con 0x7f5e800cc4e0 2023-05-22T16:03:17.764 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:17.758+0000 7f5e86670640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f5e8015d650 0x7f5e8015dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:17.764 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:17.758+0000 7f5e85e6f640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f5e80070c10 0x7f5e800c9e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:57850/0 (socket says 172.21.15.16:57850) 2023-05-22T16:03:17.765 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:17.758+0000 7f5e85e6f640 1 -- 172.21.15.16:0/3231185023 learned_addr learned my addr 172.21.15.16:0/3231185023 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T16:03:17.765 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:17.758+0000 7f5e86670640 1 -- 172.21.15.16:0/3231185023 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5e800cc4e0 msgr2=0x7f5e8006bfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:03:17.765 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:17.758+0000 7f5e8566e640 1 --2- 172.21.15.16:0/3231185023 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5e800cc4e0 0x7f5e8006bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:17.766 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:17.758+0000 7f5e86670640 1 --2- 172.21.15.16:0/3231185023 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5e800cc4e0 0x7f5e8006bfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:17.766 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:17.762+0000 7f5e86670640 1 -- 172.21.15.16:0/3231185023 >> v1:172.21.15.16:6789/0 conn(0x7f5e80070c10 legacy=0x7f5e800c9e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:17.766 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:17.758+0000 7f5e84e6d640 1 -- 172.21.15.16:0/3231185023 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2078812687 0 0) 0x7f5e800cc930 con 0x7f5e80070c10 2023-05-22T16:03:17.766 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:17.762+0000 7f5e86670640 1 -- 172.21.15.16:0/3231185023 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e800cd7c0 con 0x7f5e8015d650 2023-05-22T16:03:17.766 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:17.762+0000 7f5e8566e640 1 --2- 172.21.15.16:0/3231185023 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5e800cc4e0 0x7f5e8006bfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T16:03:17.767 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:17.762+0000 7f5e86670640 1 --2- 172.21.15.16:0/3231185023 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5e8015d650 0x7f5e8015dad0 secure :-1 s=READY pgs=2126 cs=0 l=1 rev1=1 crypto rx=0x7f5e78009d90 tx=0x7f5e7801c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94bd35615104b6cc server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:03:17.767 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:17.762+0000 7f5e84e6d640 1 -- 172.21.15.16:0/3231185023 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e780a59e0 con 0x7f5e8015d650 2023-05-22T16:03:17.767 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:17.762+0000 7f5e84e6d640 1 -- 172.21.15.16:0/3231185023 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e780a5b40 con 0x7f5e8015d650 2023-05-22T16:03:17.768 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:17.762+0000 7f5e84e6d640 1 -- 172.21.15.16:0/3231185023 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e780ad440 con 0x7f5e8015d650 2023-05-22T16:03:17.768 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:17.762+0000 7f5e880d3640 1 -- 172.21.15.16:0/3231185023 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5e8015d650 msgr2=0x7f5e8015dad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:17.768 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:17.762+0000 7f5e880d3640 1 --2- 172.21.15.16:0/3231185023 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5e8015d650 0x7f5e8015dad0 secure :-1 s=READY pgs=2126 cs=0 l=1 rev1=1 crypto rx=0x7f5e78009d90 tx=0x7f5e7801c920 comp rx=0 tx=0).stop 2023-05-22T16:03:17.768 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:17.762+0000 7f5e880d3640 1 -- 172.21.15.16:0/3231185023 shutdown_connections 2023-05-22T16:03:17.768 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:17.762+0000 7f5e880d3640 1 --2- 172.21.15.16:0/3231185023 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5e8015d650 0x7f5e8015dad0 unknown :-1 s=CLOSED pgs=2126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:17.768 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:17.762+0000 7f5e880d3640 1 --2- 172.21.15.16:0/3231185023 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5e800cc4e0 0x7f5e8006bfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:17.768 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:17.762+0000 7f5e880d3640 1 -- 172.21.15.16:0/3231185023 >> 172.21.15.16:0/3231185023 conn(0x7f5e80069720 msgr2=0x7f5e80069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:03:17.769 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:17.762+0000 7f5e880d3640 1 -- 172.21.15.16:0/3231185023 shutdown_connections 2023-05-22T16:03:17.769 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:17.762+0000 7f5e880d3640 1 -- 172.21.15.16:0/3231185023 wait complete. 2023-05-22T16:03:17.770 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:17.766+0000 7f5e880d3640 1 Processor -- start 2023-05-22T16:03:17.770 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:17.766+0000 7f5e880d3640 1 -- start start 2023-05-22T16:03:17.770 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:17.766+0000 7f5e880d3640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5e80070c10 0x7f5e80156110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:17.770 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:17.766+0000 7f5e880d3640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5e800cc4e0 0x7f5e8014b7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:17.770 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:17.766+0000 7f5e880d3640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5e8015d650 0x7f5e800c64d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:17.771 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:17.766+0000 7f5e880d3640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e800c5e90 con 0x7f5e8015d650 2023-05-22T16:03:17.771 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:17.766+0000 7f5e880d3640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e8014fd70 con 0x7f5e800cc4e0 2023-05-22T16:03:17.771 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:17.766+0000 7f5e880d3640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e800c7350 con 0x7f5e80070c10 2023-05-22T16:03:17.771 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:17.766+0000 7f5e86670640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5e8015d650 0x7f5e800c64d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:17.771 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:17.766+0000 7f5e86670640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5e8015d650 0x7f5e800c64d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57630/0 (socket says 172.21.15.16:57630) 2023-05-22T16:03:17.771 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:17.766+0000 7f5e86670640 1 -- 172.21.15.16:0/1808286286 learned_addr learned my addr 172.21.15.16:0/1808286286 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:03:17.771 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:17.766+0000 7f5e85e6f640 1 --2- 172.21.15.16:0/1808286286 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5e80070c10 0x7f5e80156110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:17.771 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:17.766+0000 7f5e8566e640 1 --2- 172.21.15.16:0/1808286286 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5e800cc4e0 0x7f5e8014b7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:17.772 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:17.766+0000 7f5e86670640 1 -- 172.21.15.16:0/1808286286 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5e80070c10 msgr2=0x7f5e80156110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:17.772 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:17.766+0000 7f5e86670640 1 --2- 172.21.15.16:0/1808286286 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5e80070c10 0x7f5e80156110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:17.772 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:17.766+0000 7f5e86670640 1 -- 172.21.15.16:0/1808286286 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5e800cc4e0 msgr2=0x7f5e8014b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:17.772 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:17.766+0000 7f5e86670640 1 --2- 172.21.15.16:0/1808286286 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5e800cc4e0 0x7f5e8014b7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:17.772 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:17.766+0000 7f5e86670640 1 -- 172.21.15.16:0/1808286286 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e78009a70 con 0x7f5e8015d650 2023-05-22T16:03:17.772 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:17.766+0000 7f5e86670640 1 --2- 172.21.15.16:0/1808286286 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5e8015d650 0x7f5e800c64d0 secure :-1 s=READY pgs=2127 cs=0 l=1 rev1=1 crypto rx=0x7f5e7801afd0 tx=0x7f5e780196c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:03:17.773 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:17.766+0000 7f5e767fc640 1 -- 172.21.15.16:0/1808286286 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e780ade40 con 0x7f5e8015d650 2023-05-22T16:03:17.774 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:17.766+0000 7f5e767fc640 1 -- 172.21.15.16:0/1808286286 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e780b7410 con 0x7f5e8015d650 2023-05-22T16:03:17.774 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:17.766+0000 7f5e880d3640 1 -- 172.21.15.16:0/1808286286 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e8006c4e0 con 0x7f5e8015d650 2023-05-22T16:03:17.774 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:17.766+0000 7f5e767fc640 1 -- 172.21.15.16:0/1808286286 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e780b77a0 con 0x7f5e8015d650 2023-05-22T16:03:17.774 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:17.766+0000 7f5e880d3640 1 -- 172.21.15.16:0/1808286286 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e80070fe0 con 0x7f5e8015d650 2023-05-22T16:03:17.774 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:17.766+0000 7f5e880d3640 1 -- 172.21.15.16:0/1808286286 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e800cce90 con 0x7f5e8015d650 2023-05-22T16:03:17.774 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:17.770+0000 7f5e767fc640 1 -- 172.21.15.16:0/1808286286 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 78) v1 ==== 84361+0+0 (secure 0 0 0) 0x7f5e780b4050 con 0x7f5e8015d650 2023-05-22T16:03:17.774 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:17.770+0000 7f5e767fc640 1 --2- 172.21.15.16:0/1808286286 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f5e640654f0 0x7f5e640679a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:17.774 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:17.770+0000 7f5e767fc640 1 -- 172.21.15.16:0/1808286286 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(165..165 src has 1..165) v4 ==== 7041+0+0 (secure 0 0 0) 0x7f5e7811e0e0 con 0x7f5e8015d650 2023-05-22T16:03:17.775 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:17.770+0000 7f5e85e6f640 1 --2- 172.21.15.16:0/1808286286 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f5e640654f0 0x7f5e640679a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:17.775 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:17.770+0000 7f5e85e6f640 1 --2- 172.21.15.16:0/1808286286 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f5e640654f0 0x7f5e640679a0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f5e7c000f70 tx=0x7f5e7c007c10 comp rx=0 tx=0).ready entity=mgr.9933 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:03:17.776 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:17.774+0000 7f5e767fc640 1 -- 172.21.15.16:0/1808286286 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f5e780eb930 con 0x7f5e8015d650 2023-05-22T16:03:17.950 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:17.946+0000 7f5e880d3640 1 -- 172.21.15.16:0/1808286286 --> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/snap_test_dir1/test_restart", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f5e8014ff20 con 0x7f5e640654f0 2023-05-22T16:03:17.963 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:17.958+0000 7f5e767fc640 1 -- 172.21.15.16:0/1808286286 <== mgr.9933 v2:172.21.15.16:6832/65890 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+361 (secure 0 0 0) 0x7f5e8014ff20 con 0x7f5e640654f0 2023-05-22T16:03:17.964 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T16:03:17.964 INFO:teuthology.orchestra.run.smithi016.stdout:[{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/test_restart", "rel_path": "/snap_test_dir1/test_restart", "schedule": "1M", "retention": {}, "start": "2023-05-22T00:00:00", "created": "2023-05-22T15:47:17", "first": "2023-05-22T15:48:00", "last": "2023-05-22T16:03:00", "last_pruned": null, "created_count": 16, "pruned_count": 0, "active": false}] 2023-05-22T16:03:17.966 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:17.962+0000 7f5e880d3640 1 -- 172.21.15.16:0/1808286286 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f5e640654f0 msgr2=0x7f5e640679a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:17.966 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:17.962+0000 7f5e880d3640 1 --2- 172.21.15.16:0/1808286286 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f5e640654f0 0x7f5e640679a0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f5e7c000f70 tx=0x7f5e7c007c10 comp rx=0 tx=0).stop 2023-05-22T16:03:17.966 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:17.962+0000 7f5e880d3640 1 -- 172.21.15.16:0/1808286286 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5e8015d650 msgr2=0x7f5e800c64d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:17.967 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:17.962+0000 7f5e880d3640 1 --2- 172.21.15.16:0/1808286286 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5e8015d650 0x7f5e800c64d0 secure :-1 s=READY pgs=2127 cs=0 l=1 rev1=1 crypto rx=0x7f5e7801afd0 tx=0x7f5e780196c0 comp rx=0 tx=0).stop 2023-05-22T16:03:17.967 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:17.962+0000 7f5e880d3640 1 -- 172.21.15.16:0/1808286286 shutdown_connections 2023-05-22T16:03:17.967 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:17.962+0000 7f5e880d3640 1 --2- 172.21.15.16:0/1808286286 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f5e640654f0 0x7f5e640679a0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:17.968 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:17.962+0000 7f5e880d3640 1 --2- 172.21.15.16:0/1808286286 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5e8015d650 0x7f5e800c64d0 unknown :-1 s=CLOSED pgs=2127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:17.968 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:17.962+0000 7f5e880d3640 1 --2- 172.21.15.16:0/1808286286 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5e800cc4e0 0x7f5e8014b7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:17.968 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:17.962+0000 7f5e880d3640 1 --2- 172.21.15.16:0/1808286286 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5e80070c10 0x7f5e80156110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:17.968 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:17.962+0000 7f5e880d3640 1 -- 172.21.15.16:0/1808286286 >> 172.21.15.16:0/1808286286 conn(0x7f5e80069720 msgr2=0x7f5e80069af0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:03:17.969 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:17.962+0000 7f5e880d3640 1 -- 172.21.15.16:0/1808286286 shutdown_connections 2023-05-22T16:03:17.969 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:17.962+0000 7f5e880d3640 1 -- 172.21.15.16:0/1808286286 wait complete. 2023-05-22T16:03:17.982 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/test_restart", "rel_path": "/snap_test_dir1/test_restart", "schedule": "1M", "retention": {}, "start": "2023-05-22T00:00:00", "created": "2023-05-22T15:47:17", "first": "2023-05-22T15:48:00", "last": "2023-05-22T16:03:00", "last_pruned": null, "created_count": 16, "pruned_count": 0, "active": false}] 2023-05-22T16:03:17.983 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'path': '/snap_test_dir1/test_restart', 'rel_path': '/snap_test_dir1/test_restart', 'schedule': '1M', 'retention': {}, 'start': '2023-05-22T00:00:00', 'created': '2023-05-22T15:47:17', 'first': '2023-05-22T15:48:00', 'last': '2023-05-22T16:03:00', 'last_pruned': None, 'created_count': 16, 'pruned_count': 0, 'active': False} 2023-05-22T16:03:17.983 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove --fs cephfs /snap_test_dir1/test_restart 1M 2023-05-22T16:03:18.109 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:18.106+0000 7f2d40c2f640 1 Processor -- start 2023-05-22T16:03:18.109 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:18.106+0000 7f2d40c2f640 1 -- start start 2023-05-22T16:03:18.110 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:18.106+0000 7f2d40c2f640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2d3c15c8e0 0x7f2d3c0ccc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:18.111 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:18.106+0000 7f2d40c2f640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2d3c0d2eb0 0x7f2d3c15ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:18.111 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:18.106+0000 7f2d40c2f640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d3c0d2a20 con 0x7f2d3c0cef10 2023-05-22T16:03:18.111 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:18.106+0000 7f2d40c2f640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d3c0cd230 con 0x7f2d3c0d2eb0 2023-05-22T16:03:18.111 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:18.106+0000 7f2d40c2f640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d3c0d23d0 con 0x7f2d3c15c8e0 2023-05-22T16:03:18.111 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:18.106+0000 7f2d39d9b640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2d3c0d2eb0 0x7f2d3c15ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:18.111 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:18.106+0000 7f2d3ad9d640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f2d3c0cef10 0x7f2d3c0cf370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:57862/0 (socket says 172.21.15.16:57862) 2023-05-22T16:03:18.111 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:18.106+0000 7f2d3ad9d640 1 -- 172.21.15.16:0/4236450732 learned_addr learned my addr 172.21.15.16:0/4236450732 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T16:03:18.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:18.106+0000 7f2d3a59c640 1 --2- 172.21.15.16:0/4236450732 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2d3c15c8e0 0x7f2d3c0ccc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:18.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:18.106+0000 7f2d3959a640 1 -- 172.21.15.16:0/4236450732 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2662607266 0 0) 0x7f2d3c0d2a20 con 0x7f2d3c0cef10 2023-05-22T16:03:18.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:18.106+0000 7f2d3959a640 1 -- 172.21.15.16:0/4236450732 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d24003530 con 0x7f2d3c0cef10 2023-05-22T16:03:18.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:18.106+0000 7f2d39d9b640 1 -- 172.21.15.16:0/4236450732 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2d3c15c8e0 msgr2=0x7f2d3c0ccc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:03:18.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:18.106+0000 7f2d39d9b640 1 --2- 172.21.15.16:0/4236450732 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2d3c15c8e0 0x7f2d3c0ccc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:18.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:18.106+0000 7f2d39d9b640 1 -- 172.21.15.16:0/4236450732 >> v1:172.21.15.16:6789/0 conn(0x7f2d3c0cef10 legacy=0x7f2d3c0cf370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:18.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:18.106+0000 7f2d39d9b640 1 -- 172.21.15.16:0/4236450732 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d3c0d26b0 con 0x7f2d3c0d2eb0 2023-05-22T16:03:18.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:18.110+0000 7f2d39d9b640 1 --2- 172.21.15.16:0/4236450732 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2d3c0d2eb0 0x7f2d3c15ccb0 secure :-1 s=READY pgs=2129 cs=0 l=1 rev1=1 crypto rx=0x7f2d300097c0 tx=0x7f2d3001c3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d913fb33f955a830 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:03:18.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:18.110+0000 7f2d3959a640 1 -- 172.21.15.16:0/4236450732 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d300ad640 con 0x7f2d3c0d2eb0 2023-05-22T16:03:18.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:18.110+0000 7f2d3959a640 1 -- 172.21.15.16:0/4236450732 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d300ad7a0 con 0x7f2d3c0d2eb0 2023-05-22T16:03:18.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:18.110+0000 7f2d3959a640 1 -- 172.21.15.16:0/4236450732 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d300adb30 con 0x7f2d3c0d2eb0 2023-05-22T16:03:18.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:18.110+0000 7f2d40c2f640 1 -- 172.21.15.16:0/4236450732 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2d3c0d2eb0 msgr2=0x7f2d3c15ccb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:18.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:18.110+0000 7f2d40c2f640 1 --2- 172.21.15.16:0/4236450732 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2d3c0d2eb0 0x7f2d3c15ccb0 secure :-1 s=READY pgs=2129 cs=0 l=1 rev1=1 crypto rx=0x7f2d300097c0 tx=0x7f2d3001c3b0 comp rx=0 tx=0).stop 2023-05-22T16:03:18.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:18.110+0000 7f2d40c2f640 1 -- 172.21.15.16:0/4236450732 shutdown_connections 2023-05-22T16:03:18.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:18.110+0000 7f2d40c2f640 1 --2- 172.21.15.16:0/4236450732 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2d3c0d2eb0 0x7f2d3c15ccb0 unknown :-1 s=CLOSED pgs=2129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:18.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:18.110+0000 7f2d40c2f640 1 --2- 172.21.15.16:0/4236450732 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2d3c15c8e0 0x7f2d3c0ccc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:18.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:18.110+0000 7f2d40c2f640 1 -- 172.21.15.16:0/4236450732 >> 172.21.15.16:0/4236450732 conn(0x7f2d3c070e50 msgr2=0x7f2d3c071250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:03:18.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:18.110+0000 7f2d40c2f640 1 -- 172.21.15.16:0/4236450732 shutdown_connections 2023-05-22T16:03:18.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:18.110+0000 7f2d40c2f640 1 -- 172.21.15.16:0/4236450732 wait complete. 2023-05-22T16:03:18.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:18.110+0000 7f2d40c2f640 1 Processor -- start 2023-05-22T16:03:18.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:18.110+0000 7f2d40c2f640 1 -- start start 2023-05-22T16:03:18.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:18.114+0000 7f2d40c2f640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2d3c0cef10 0x7f2d3c0cc550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:18.118 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:18.114+0000 7f2d40c2f640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2d3c0d2eb0 0x7f2d3c15acd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:18.118 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:18.114+0000 7f2d40c2f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2d3c15c8e0 0x7f2d3c15b210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:18.118 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:18.114+0000 7f2d40c2f640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d3c153e10 con 0x7f2d3c15c8e0 2023-05-22T16:03:18.118 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:18.114+0000 7f2d40c2f640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d3c153500 con 0x7f2d3c0d2eb0 2023-05-22T16:03:18.118 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:18.114+0000 7f2d40c2f640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d3c1537c0 con 0x7f2d3c0cef10 2023-05-22T16:03:18.118 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:18.114+0000 7f2d3ad9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2d3c15c8e0 0x7f2d3c15b210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:18.118 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:18.114+0000 7f2d3ad9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2d3c15c8e0 0x7f2d3c15b210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57652/0 (socket says 172.21.15.16:57652) 2023-05-22T16:03:18.119 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:18.114+0000 7f2d3ad9d640 1 -- 172.21.15.16:0/534681553 learned_addr learned my addr 172.21.15.16:0/534681553 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:03:18.119 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:18.114+0000 7f2d39d9b640 1 --2- 172.21.15.16:0/534681553 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2d3c0d2eb0 0x7f2d3c15acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:18.119 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:18.114+0000 7f2d3a59c640 1 --2- 172.21.15.16:0/534681553 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2d3c0cef10 0x7f2d3c0cc550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:18.119 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:18.114+0000 7f2d3ad9d640 1 -- 172.21.15.16:0/534681553 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2d3c0cef10 msgr2=0x7f2d3c0cc550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:18.119 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:18.114+0000 7f2d3ad9d640 1 --2- 172.21.15.16:0/534681553 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2d3c0cef10 0x7f2d3c0cc550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:18.119 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:18.114+0000 7f2d3ad9d640 1 -- 172.21.15.16:0/534681553 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2d3c0d2eb0 msgr2=0x7f2d3c15acd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:18.119 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:18.114+0000 7f2d3ad9d640 1 --2- 172.21.15.16:0/534681553 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2d3c0d2eb0 0x7f2d3c15acd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:18.120 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:18.114+0000 7f2d3ad9d640 1 -- 172.21.15.16:0/534681553 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d30009420 con 0x7f2d3c15c8e0 2023-05-22T16:03:18.120 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:18.114+0000 7f2d3a59c640 1 --2- 172.21.15.16:0/534681553 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2d3c0cef10 0x7f2d3c0cc550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T16:03:18.120 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:18.114+0000 7f2d3ad9d640 1 --2- 172.21.15.16:0/534681553 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2d3c15c8e0 0x7f2d3c15b210 secure :-1 s=READY pgs=2130 cs=0 l=1 rev1=1 crypto rx=0x7f2d3400cb40 tx=0x7f2d3400a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:03:18.120 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:18.114+0000 7f2d2affd640 1 -- 172.21.15.16:0/534681553 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d3400aca0 con 0x7f2d3c15c8e0 2023-05-22T16:03:18.120 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:18.114+0000 7f2d2affd640 1 -- 172.21.15.16:0/534681553 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d3400ae00 con 0x7f2d3c15c8e0 2023-05-22T16:03:18.121 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:18.114+0000 7f2d40c2f640 1 -- 172.21.15.16:0/534681553 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d3c0d0bf0 con 0x7f2d3c15c8e0 2023-05-22T16:03:18.121 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:18.114+0000 7f2d2affd640 1 -- 172.21.15.16:0/534681553 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d34012620 con 0x7f2d3c15c8e0 2023-05-22T16:03:18.121 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:18.114+0000 7f2d40c2f640 1 -- 172.21.15.16:0/534681553 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d3c0d3300 con 0x7f2d3c15c8e0 2023-05-22T16:03:18.122 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:18.114+0000 7f2d40c2f640 1 -- 172.21.15.16:0/534681553 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d3c0d0d80 con 0x7f2d3c15c8e0 2023-05-22T16:03:18.122 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:18.114+0000 7f2d2affd640 1 -- 172.21.15.16:0/534681553 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 78) v1 ==== 84361+0+0 (secure 0 0 0) 0x7f2d34007070 con 0x7f2d3c15c8e0 2023-05-22T16:03:18.122 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:18.114+0000 7f2d2affd640 1 --2- 172.21.15.16:0/534681553 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f2d1c065520 0x7f2d1c0679d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:18.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:18.114+0000 7f2d2affd640 1 -- 172.21.15.16:0/534681553 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(165..165 src has 1..165) v4 ==== 7041+0+0 (secure 0 0 0) 0x7f2d34082e80 con 0x7f2d3c15c8e0 2023-05-22T16:03:18.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:18.114+0000 7f2d3a59c640 1 --2- 172.21.15.16:0/534681553 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f2d1c065520 0x7f2d1c0679d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:18.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:18.114+0000 7f2d3a59c640 1 --2- 172.21.15.16:0/534681553 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f2d1c065520 0x7f2d1c0679d0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f2d3c0cb830 tx=0x7f2d2c008040 comp rx=0 tx=0).ready entity=mgr.9933 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:03:18.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:18.118+0000 7f2d2affd640 1 -- 172.21.15.16:0/534681553 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f2d340505b0 con 0x7f2d3c15c8e0 2023-05-22T16:03:18.296 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:18.290+0000 7f2d40c2f640 1 -- 172.21.15.16:0/534681553 --> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "fs": "cephfs", "path": "/snap_test_dir1/test_restart", "repeat": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f2d3c04dbe0 con 0x7f2d1c065520 2023-05-22T16:03:18.433 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:18.430+0000 7f2d2affd640 1 -- 172.21.15.16:0/534681553 <== mgr.9933 v2:172.21.15.16:6832/65890 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+55 (secure 0 0 0) 0x7f2d3c04dbe0 con 0x7f2d1c065520 2023-05-22T16:03:18.434 INFO:teuthology.orchestra.run.smithi016.stdout:Schedule removed for path /snap_test_dir1/test_restart 2023-05-22T16:03:18.435 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:18.430+0000 7f2d40c2f640 1 -- 172.21.15.16:0/534681553 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f2d1c065520 msgr2=0x7f2d1c0679d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:18.435 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:18.430+0000 7f2d40c2f640 1 --2- 172.21.15.16:0/534681553 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f2d1c065520 0x7f2d1c0679d0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f2d3c0cb830 tx=0x7f2d2c008040 comp rx=0 tx=0).stop 2023-05-22T16:03:18.435 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:18.430+0000 7f2d40c2f640 1 -- 172.21.15.16:0/534681553 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2d3c15c8e0 msgr2=0x7f2d3c15b210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:18.436 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:18.430+0000 7f2d40c2f640 1 --2- 172.21.15.16:0/534681553 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2d3c15c8e0 0x7f2d3c15b210 secure :-1 s=READY pgs=2130 cs=0 l=1 rev1=1 crypto rx=0x7f2d3400cb40 tx=0x7f2d3400a060 comp rx=0 tx=0).stop 2023-05-22T16:03:18.436 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:18.430+0000 7f2d40c2f640 1 -- 172.21.15.16:0/534681553 shutdown_connections 2023-05-22T16:03:18.436 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:18.430+0000 7f2d40c2f640 1 --2- 172.21.15.16:0/534681553 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f2d1c065520 0x7f2d1c0679d0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:18.436 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:18.430+0000 7f2d40c2f640 1 --2- 172.21.15.16:0/534681553 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2d3c15c8e0 0x7f2d3c15b210 unknown :-1 s=CLOSED pgs=2130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:18.436 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:18.430+0000 7f2d40c2f640 1 --2- 172.21.15.16:0/534681553 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2d3c0d2eb0 0x7f2d3c15acd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:18.436 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:18.430+0000 7f2d40c2f640 1 --2- 172.21.15.16:0/534681553 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2d3c0cef10 0x7f2d3c0cc550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:18.436 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:18.430+0000 7f2d40c2f640 1 -- 172.21.15.16:0/534681553 >> 172.21.15.16:0/534681553 conn(0x7f2d3c070e50 msgr2=0x7f2d3c15e6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:03:18.437 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:18.430+0000 7f2d40c2f640 1 -- 172.21.15.16:0/534681553 shutdown_connections 2023-05-22T16:03:18.437 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:18.430+0000 7f2d40c2f640 1 -- 172.21.15.16:0/534681553 wait complete. 2023-05-22T16:03:18.456 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/test_restart 2023-05-22T16:03:18.457 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T16:03:18.457 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/test_restart/.snap) 2023-05-22T16:03:18.468 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_48_00_UTC 2023-05-22T16:03:18.469 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_49_00_UTC 2023-05-22T16:03:18.469 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_50_00_UTC 2023-05-22T16:03:18.469 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_51_00_UTC 2023-05-22T16:03:18.469 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_52_00_UTC 2023-05-22T16:03:18.469 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_53_00_UTC 2023-05-22T16:03:18.469 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_54_00_UTC 2023-05-22T16:03:18.469 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_55_00_UTC 2023-05-22T16:03:18.470 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_56_00_UTC 2023-05-22T16:03:18.470 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_57_00_UTC 2023-05-22T16:03:18.470 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_58_00_UTC 2023-05-22T16:03:18.470 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-15_59_00_UTC 2023-05-22T16:03:18.470 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-16_00_00_UTC 2023-05-22T16:03:18.470 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-16_01_00_UTC 2023-05-22T16:03:18.470 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-16_02_00_UTC 2023-05-22T16:03:18.471 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-16_03_00_UTC 2023-05-22T16:03:18.471 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2023-05-22-15_48_00_UTC 2023-05-22T16:03:18.472 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T16:03:18.472 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2023-05-22-15_48_00_UTC) 2023-05-22T16:03:18.533 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2023-05-22-15_49_00_UTC 2023-05-22T16:03:18.534 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T16:03:18.534 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2023-05-22-15_49_00_UTC) 2023-05-22T16:03:18.598 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2023-05-22-15_50_00_UTC 2023-05-22T16:03:18.598 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T16:03:18.599 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2023-05-22-15_50_00_UTC) 2023-05-22T16:03:18.660 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2023-05-22-15_51_00_UTC 2023-05-22T16:03:18.661 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T16:03:18.662 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2023-05-22-15_51_00_UTC) 2023-05-22T16:03:18.724 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2023-05-22-15_52_00_UTC 2023-05-22T16:03:18.725 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T16:03:18.725 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2023-05-22-15_52_00_UTC) 2023-05-22T16:03:18.788 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2023-05-22-15_53_00_UTC 2023-05-22T16:03:18.788 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T16:03:18.789 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2023-05-22-15_53_00_UTC) 2023-05-22T16:03:18.854 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2023-05-22-15_54_00_UTC 2023-05-22T16:03:18.854 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T16:03:18.854 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2023-05-22-15_54_00_UTC) 2023-05-22T16:03:18.917 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2023-05-22-15_55_00_UTC 2023-05-22T16:03:18.918 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T16:03:18.919 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2023-05-22-15_55_00_UTC) 2023-05-22T16:03:18.979 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2023-05-22-15_56_00_UTC 2023-05-22T16:03:18.979 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T16:03:18.980 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2023-05-22-15_56_00_UTC) 2023-05-22T16:03:19.041 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2023-05-22-15_57_00_UTC 2023-05-22T16:03:19.041 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T16:03:19.042 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2023-05-22-15_57_00_UTC) 2023-05-22T16:03:19.100 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2023-05-22-15_58_00_UTC 2023-05-22T16:03:19.101 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T16:03:19.101 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2023-05-22-15_58_00_UTC) 2023-05-22T16:03:19.161 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2023-05-22-15_59_00_UTC 2023-05-22T16:03:19.161 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T16:03:19.162 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2023-05-22-15_59_00_UTC) 2023-05-22T16:03:19.222 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2023-05-22-16_00_00_UTC 2023-05-22T16:03:19.223 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T16:03:19.223 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2023-05-22-16_00_00_UTC) 2023-05-22T16:03:19.281 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2023-05-22-16_01_00_UTC 2023-05-22T16:03:19.281 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T16:03:19.282 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2023-05-22-16_01_00_UTC) 2023-05-22T16:03:19.343 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2023-05-22-16_02_00_UTC 2023-05-22T16:03:19.344 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T16:03:19.344 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2023-05-22-16_02_00_UTC) 2023-05-22T16:03:19.400 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2023-05-22-16_03_00_UTC 2023-05-22T16:03:19.400 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T16:03:19.401 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2023-05-22-16_03_00_UTC) 2023-05-22T16:03:19.464 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T16:03:19.464 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart) 2023-05-22T16:03:19.518 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2023-05-22T16:03:19.695 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:19.690+0000 7f0dcec6d640 1 Processor -- start 2023-05-22T16:03:19.695 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:19.690+0000 7f0dcec6d640 1 -- start start 2023-05-22T16:03:19.696 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:19.690+0000 7f0dcec6d640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0dc8159d40 0x7f0dc80cb260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:19.696 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:19.690+0000 7f0dcec6d640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f0dc806b6c0 0x7f0dc806bb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:19.696 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:19.690+0000 7f0dcec6d640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0dc8070c30 con 0x7f0dc8159970 2023-05-22T16:03:19.697 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:19.690+0000 7f0dcec6d640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0dc815dd90 con 0x7f0dc806b6c0 2023-05-22T16:03:19.697 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:19.690+0000 7f0dcec6d640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0dc815a190 con 0x7f0dc8159d40 2023-05-22T16:03:19.697 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:19.690+0000 7f0dcd20a640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f0dc806b6c0 0x7f0dc806bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:19.697 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:19.690+0000 7f0dcca09640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f0dc8159970 0x7f0dc815d680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:41930/0 (socket says 172.21.15.16:41930) 2023-05-22T16:03:19.698 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:19.690+0000 7f0dcca09640 1 -- 172.21.15.16:0/297070177 learned_addr learned my addr 172.21.15.16:0/297070177 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T16:03:19.700 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:19.694+0000 7f0dc7fff640 1 --2- 172.21.15.16:0/297070177 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0dc8159d40 0x7f0dc80cb260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:19.700 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:19.694+0000 7f0dcd20a640 1 -- 172.21.15.16:0/297070177 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0dc8159d40 msgr2=0x7f0dc80cb260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:03:19.700 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:19.694+0000 7f0dcd20a640 1 --2- 172.21.15.16:0/297070177 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0dc8159d40 0x7f0dc80cb260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:19.701 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:19.694+0000 7f0dcd20a640 1 -- 172.21.15.16:0/297070177 >> v1:172.21.15.16:6789/0 conn(0x7f0dc8159970 legacy=0x7f0dc815d680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:19.701 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:19.694+0000 7f0dcd20a640 1 -- 172.21.15.16:0/297070177 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0dc806c060 con 0x7f0dc806b6c0 2023-05-22T16:03:19.701 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:19.694+0000 7f0dc77fe640 1 -- 172.21.15.16:0/297070177 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2018502872 0 0) 0x7f0dc8070c30 con 0x7f0dc8159970 2023-05-22T16:03:19.702 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:19.694+0000 7f0dcd20a640 1 --2- 172.21.15.16:0/297070177 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0dc806b6c0 0x7f0dc806bb20 secure :-1 s=READY pgs=2132 cs=0 l=1 rev1=1 crypto rx=0x7f0dc0009d90 tx=0x7f0dc001c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58edbc105bf63016 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:03:19.702 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:19.694+0000 7f0dc77fe640 1 -- 172.21.15.16:0/297070177 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dc00a5950 con 0x7f0dc806b6c0 2023-05-22T16:03:19.702 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:19.694+0000 7f0dc77fe640 1 -- 172.21.15.16:0/297070177 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0dc00a5ab0 con 0x7f0dc806b6c0 2023-05-22T16:03:19.702 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:19.694+0000 7f0dc77fe640 1 -- 172.21.15.16:0/297070177 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dc00a5e40 con 0x7f0dc806b6c0 2023-05-22T16:03:19.702 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:19.694+0000 7f0dcec6d640 1 -- 172.21.15.16:0/297070177 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0dc806b6c0 msgr2=0x7f0dc806bb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:19.702 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:19.694+0000 7f0dcec6d640 1 --2- 172.21.15.16:0/297070177 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0dc806b6c0 0x7f0dc806bb20 secure :-1 s=READY pgs=2132 cs=0 l=1 rev1=1 crypto rx=0x7f0dc0009d90 tx=0x7f0dc001c890 comp rx=0 tx=0).stop 2023-05-22T16:03:19.702 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:19.694+0000 7f0dcec6d640 1 -- 172.21.15.16:0/297070177 shutdown_connections 2023-05-22T16:03:19.703 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:19.694+0000 7f0dcec6d640 1 --2- 172.21.15.16:0/297070177 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0dc806b6c0 0x7f0dc806bb20 unknown :-1 s=CLOSED pgs=2132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:19.703 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:19.694+0000 7f0dcec6d640 1 --2- 172.21.15.16:0/297070177 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0dc8159d40 0x7f0dc80cb260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:19.703 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:19.694+0000 7f0dcec6d640 1 -- 172.21.15.16:0/297070177 >> 172.21.15.16:0/297070177 conn(0x7f0dc8069720 msgr2=0x7f0dc8069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:03:19.703 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:19.694+0000 7f0dcec6d640 1 -- 172.21.15.16:0/297070177 shutdown_connections 2023-05-22T16:03:19.703 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:19.694+0000 7f0dcec6d640 1 -- 172.21.15.16:0/297070177 wait complete. 2023-05-22T16:03:19.705 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:19.698+0000 7f0dcec6d640 1 Processor -- start 2023-05-22T16:03:19.706 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:19.698+0000 7f0dcec6d640 1 -- start start 2023-05-22T16:03:19.706 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:19.698+0000 7f0dcec6d640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0dc806b6c0 0x7f0dc80cabc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:19.706 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:19.698+0000 7f0dcec6d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0dc8159970 0x7f0dc814cf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:19.706 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:19.698+0000 7f0dcec6d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0dc8159d40 0x7f0dc81472d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:19.707 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:19.698+0000 7f0dcec6d640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0dc814cb00 con 0x7f0dc8159d40 2023-05-22T16:03:19.707 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:19.698+0000 7f0dcec6d640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0dc814e150 con 0x7f0dc8159970 2023-05-22T16:03:19.707 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:19.698+0000 7f0dcec6d640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0dc80cbbe0 con 0x7f0dc806b6c0 2023-05-22T16:03:19.708 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:19.698+0000 7f0dcd20a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0dc8159d40 0x7f0dc81472d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:19.708 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:19.698+0000 7f0dcd20a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0dc8159d40 0x7f0dc81472d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54996/0 (socket says 172.21.15.16:54996) 2023-05-22T16:03:19.708 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:19.698+0000 7f0dcd20a640 1 -- 172.21.15.16:0/2160739325 learned_addr learned my addr 172.21.15.16:0/2160739325 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:03:19.708 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:19.698+0000 7f0dcca09640 1 --2- 172.21.15.16:0/2160739325 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0dc806b6c0 0x7f0dc80cabc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:19.709 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:19.698+0000 7f0dc7fff640 1 --2- 172.21.15.16:0/2160739325 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0dc8159970 0x7f0dc814cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:19.709 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:19.698+0000 7f0dcd20a640 1 -- 172.21.15.16:0/2160739325 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0dc806b6c0 msgr2=0x7f0dc80cabc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:19.709 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:19.698+0000 7f0dcd20a640 1 --2- 172.21.15.16:0/2160739325 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0dc806b6c0 0x7f0dc80cabc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:19.709 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:19.698+0000 7f0dcd20a640 1 -- 172.21.15.16:0/2160739325 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0dc8159970 msgr2=0x7f0dc814cf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:19.710 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:19.698+0000 7f0dcd20a640 1 --2- 172.21.15.16:0/2160739325 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0dc8159970 0x7f0dc814cf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:19.710 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:19.698+0000 7f0dcd20a640 1 -- 172.21.15.16:0/2160739325 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0dc0009a70 con 0x7f0dc8159d40 2023-05-22T16:03:19.710 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:19.698+0000 7f0dcd20a640 1 --2- 172.21.15.16:0/2160739325 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0dc8159d40 0x7f0dc81472d0 secure :-1 s=READY pgs=2133 cs=0 l=1 rev1=1 crypto rx=0x7f0dc00b0040 tx=0x7f0dc0019630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:03:19.711 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:19.698+0000 7f0dc57fa640 1 -- 172.21.15.16:0/2160739325 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dc00ade40 con 0x7f0dc8159d40 2023-05-22T16:03:19.711 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:19.698+0000 7f0dc57fa640 1 -- 172.21.15.16:0/2160739325 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0dc0009450 con 0x7f0dc8159d40 2023-05-22T16:03:19.711 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:19.698+0000 7f0dc57fa640 1 -- 172.21.15.16:0/2160739325 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dc00b7630 con 0x7f0dc8159d40 2023-05-22T16:03:19.711 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:19.698+0000 7f0dcec6d640 1 -- 172.21.15.16:0/2160739325 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0dc8071700 con 0x7f0dc8159d40 2023-05-22T16:03:19.711 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:19.698+0000 7f0dcec6d640 1 -- 172.21.15.16:0/2160739325 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0dc8071420 con 0x7f0dc8159d40 2023-05-22T16:03:19.711 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:19.698+0000 7f0dcec6d640 1 -- 172.21.15.16:0/2160739325 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0dc80cb7a0 con 0x7f0dc8159d40 2023-05-22T16:03:19.712 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:19.702+0000 7f0dc57fa640 1 -- 172.21.15.16:0/2160739325 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 78) v1 ==== 84361+0+0 (secure 0 0 0) 0x7f0dc00b4050 con 0x7f0dc8159d40 2023-05-22T16:03:19.712 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:19.702+0000 7f0dc57fa640 1 --2- 172.21.15.16:0/2160739325 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f0db40654f0 0x7f0db40679a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:19.712 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:19.702+0000 7f0dc57fa640 1 -- 172.21.15.16:0/2160739325 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(165..165 src has 1..165) v4 ==== 7041+0+0 (secure 0 0 0) 0x7f0dc011e210 con 0x7f0dc8159d40 2023-05-22T16:03:19.712 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:19.702+0000 7f0dcca09640 1 --2- 172.21.15.16:0/2160739325 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f0db40654f0 0x7f0db40679a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:19.712 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:19.702+0000 7f0dcca09640 1 --2- 172.21.15.16:0/2160739325 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f0db40654f0 0x7f0db40679a0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f0db8000f70 tx=0x7f0db8007b60 comp rx=0 tx=0).ready entity=mgr.9933 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:03:19.713 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:19.706+0000 7f0dc57fa640 1 -- 172.21.15.16:0/2160739325 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f0dc00eba30 con 0x7f0dc8159d40 2023-05-22T16:03:19.900 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:19.898+0000 7f0dcec6d640 1 -- 172.21.15.16:0/2160739325 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7f0dc814cc40 con 0x7f0dc8159d40 2023-05-22T16:03:20.174 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:20.170+0000 7f0dc57fa640 1 -- 172.21.15.16:0/2160739325 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mgrmap(e 79) v1 ==== 84344+0+0 (secure 0 0 0) 0x7f0dc00ebe00 con 0x7f0dc8159d40 2023-05-22T16:03:20.175 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:20.170+0000 7f0dc57fa640 1 -- 172.21.15.16:0/2160739325 <== mon.0 v2:172.21.15.16:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v79) v1 ==== 93+0+0 (secure 0 0 0) 0x7f0dc00eba30 con 0x7f0dc8159d40 2023-05-22T16:03:20.177 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:20.174+0000 7f0dcec6d640 1 -- 172.21.15.16:0/2160739325 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f0db40654f0 msgr2=0x7f0db40679a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:20.177 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:20.174+0000 7f0dcec6d640 1 --2- 172.21.15.16:0/2160739325 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f0db40654f0 0x7f0db40679a0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f0db8000f70 tx=0x7f0db8007b60 comp rx=0 tx=0).stop 2023-05-22T16:03:20.178 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:20.174+0000 7f0dcec6d640 1 -- 172.21.15.16:0/2160739325 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0dc8159d40 msgr2=0x7f0dc81472d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:20.178 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:20.174+0000 7f0dcec6d640 1 --2- 172.21.15.16:0/2160739325 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0dc8159d40 0x7f0dc81472d0 secure :-1 s=READY pgs=2133 cs=0 l=1 rev1=1 crypto rx=0x7f0dc00b0040 tx=0x7f0dc0019630 comp rx=0 tx=0).stop 2023-05-22T16:03:20.178 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:20.174+0000 7f0dcec6d640 1 -- 172.21.15.16:0/2160739325 shutdown_connections 2023-05-22T16:03:20.178 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:20.174+0000 7f0dcec6d640 1 --2- 172.21.15.16:0/2160739325 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f0db40654f0 0x7f0db40679a0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:20.178 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:20.174+0000 7f0dcec6d640 1 --2- 172.21.15.16:0/2160739325 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0dc8159d40 0x7f0dc81472d0 unknown :-1 s=CLOSED pgs=2133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:20.178 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:20.174+0000 7f0dcec6d640 1 --2- 172.21.15.16:0/2160739325 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0dc8159970 0x7f0dc814cf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:20.178 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:20.174+0000 7f0dcec6d640 1 --2- 172.21.15.16:0/2160739325 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0dc806b6c0 0x7f0dc80cabc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:20.179 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:20.174+0000 7f0dcec6d640 1 -- 172.21.15.16:0/2160739325 >> 172.21.15.16:0/2160739325 conn(0x7f0dc8069720 msgr2=0x7f0dc815adc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:03:20.179 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:20.174+0000 7f0dcec6d640 1 -- 172.21.15.16:0/2160739325 shutdown_connections 2023-05-22T16:03:20.179 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:20.174+0000 7f0dcec6d640 1 -- 172.21.15.16:0/2160739325 wait complete. 2023-05-22T16:03:20.197 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-05-22T16:03:20.197 DEBUG:teuthology.orchestra.run.smithi016:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-05-22T16:03:20.246 DEBUG:teuthology.orchestra.run.smithi062:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-05-22T16:03:20.288 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-05-22T16:03:20.289 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-05-22T16:03:20.289 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2023-05-22T16:03:20.294 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-05-22T16:03:20.300 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T16:03:20.293+0000 7f46151e9180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-05-22T16:03:20.303 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T16:03:20.298+0000 7fd0922ba180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-05-22T16:03:20.420 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:20.414+0000 7f095e8d2640 1 Processor -- start 2023-05-22T16:03:20.420 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:20.414+0000 7f095e8d2640 1 -- start start 2023-05-22T16:03:20.421 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:20.418+0000 7f095e8d2640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f09580ce480 0x7f0958071640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:20.421 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:20.418+0000 7f095e8d2640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f09580ce850 0x7f0958160090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:20.422 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:20.418+0000 7f095e8d2640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f09581605d0 0x7f095815c270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:20.422 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:20.418+0000 7f095e8d2640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f095815f990 con 0x7f09580ce850 2023-05-22T16:03:20.422 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:20.418+0000 7f095e8d2640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0958069ae0 con 0x7f09581605d0 2023-05-22T16:03:20.422 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:20.418+0000 7f095e8d2640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0958069dc0 con 0x7f09580ce480 2023-05-22T16:03:20.422 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:20.418+0000 7f09577fe640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f09580ce850 0x7f0958160090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:20.422 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:20.418+0000 7f09577fe640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f09580ce850 0x7f0958160090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55012/0 (socket says 172.21.15.16:55012) 2023-05-22T16:03:20.422 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:20.418+0000 7f09577fe640 1 -- 172.21.15.16:0/293179139 learned_addr learned my addr 172.21.15.16:0/293179139 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:03:20.423 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:20.418+0000 7f0957fff640 1 --2- 172.21.15.16:0/293179139 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f09580ce480 0x7f0958071640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:20.423 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:20.418+0000 7f095ce6f640 1 --2- 172.21.15.16:0/293179139 >> v2:172.21.15.62:3300/0 conn(0x7f09581605d0 0x7f095815c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:20.423 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:20.418+0000 7f09577fe640 1 -- 172.21.15.16:0/293179139 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f09580ce480 msgr2=0x7f0958071640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:03:20.423 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:20.418+0000 7f09577fe640 1 --2- 172.21.15.16:0/293179139 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f09580ce480 0x7f0958071640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:20.424 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:20.418+0000 7f09577fe640 1 -- 172.21.15.16:0/293179139 >> v2:172.21.15.62:3300/0 conn(0x7f09581605d0 msgr2=0x7f095815c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:03:20.424 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:20.418+0000 7f09577fe640 1 --2- 172.21.15.16:0/293179139 >> v2:172.21.15.62:3300/0 conn(0x7f09581605d0 0x7f095815c270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:20.424 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:20.418+0000 7f09577fe640 1 -- 172.21.15.16:0/293179139 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f095806a0a0 con 0x7f09580ce850 2023-05-22T16:03:20.425 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:20.418+0000 7f09577fe640 1 --2- 172.21.15.16:0/293179139 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f09580ce850 0x7f0958160090 secure :-1 s=READY pgs=2138 cs=0 l=1 rev1=1 crypto rx=0x7f0958070880 tx=0x7f094c01c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc8e46f86f750fa5 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:03:20.425 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:20.422+0000 7f0956ffd640 1 -- 172.21.15.16:0/293179139 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f094c0ac410 con 0x7f09580ce850 2023-05-22T16:03:20.425 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:20.422+0000 7f0956ffd640 1 -- 172.21.15.16:0/293179139 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f094c0ac570 con 0x7f09580ce850 2023-05-22T16:03:20.425 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:20.422+0000 7f0956ffd640 1 -- 172.21.15.16:0/293179139 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f094c0ac900 con 0x7f09580ce850 2023-05-22T16:03:20.425 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:20.422+0000 7f095e8d2640 1 -- 172.21.15.16:0/293179139 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f09580ce850 msgr2=0x7f0958160090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:20.425 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:20.422+0000 7f095e8d2640 1 --2- 172.21.15.16:0/293179139 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f09580ce850 0x7f0958160090 secure :-1 s=READY pgs=2138 cs=0 l=1 rev1=1 crypto rx=0x7f0958070880 tx=0x7f094c01c4d0 comp rx=0 tx=0).stop 2023-05-22T16:03:20.426 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:20.422+0000 7f095e8d2640 1 -- 172.21.15.16:0/293179139 shutdown_connections 2023-05-22T16:03:20.426 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:20.422+0000 7f095e8d2640 1 --2- 172.21.15.16:0/293179139 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f09580ce850 0x7f0958160090 unknown :-1 s=CLOSED pgs=2138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:20.426 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:20.422+0000 7f095e8d2640 1 --2- 172.21.15.16:0/293179139 >> v2:172.21.15.62:3300/0 conn(0x7f09581605d0 0x7f095815c270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:20.426 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:20.422+0000 7f095e8d2640 1 --2- 172.21.15.16:0/293179139 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f09580ce480 0x7f0958071640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:20.426 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:20.422+0000 7f095e8d2640 1 -- 172.21.15.16:0/293179139 >> 172.21.15.16:0/293179139 conn(0x7f09580cc870 msgr2=0x7f09580cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:03:20.427 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:20.422+0000 7f095e8d2640 1 -- 172.21.15.16:0/293179139 shutdown_connections 2023-05-22T16:03:20.427 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:20.422+0000 7f095e8d2640 1 -- 172.21.15.16:0/293179139 wait complete. 2023-05-22T16:03:20.428 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:20.422+0000 7f095e8d2640 1 Processor -- start 2023-05-22T16:03:20.429 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:20.422+0000 7f095e8d2640 1 -- start start 2023-05-22T16:03:20.429 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:20.422+0000 7f095e8d2640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f09580ce480 0x7f095814fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:20.429 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:20.422+0000 7f095e8d2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f09580ce850 0x7f09580ab840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:20.429 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:20.422+0000 7f095e8d2640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f09581605d0 0x7f095814d0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:20.429 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:20.422+0000 7f095e8d2640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f095814f030 con 0x7f09580ce850 2023-05-22T16:03:20.429 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:20.422+0000 7f095e8d2640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09580abd80 con 0x7f09580ce480 2023-05-22T16:03:20.429 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:20.422+0000 7f095e8d2640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f095814eba0 con 0x7f09581605d0 2023-05-22T16:03:20.430 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:20.422+0000 7f09577fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f09580ce850 0x7f09580ab840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:20.430 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:20.422+0000 7f09577fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f09580ce850 0x7f09580ab840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55018/0 (socket says 172.21.15.16:55018) 2023-05-22T16:03:20.430 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:20.422+0000 7f09577fe640 1 -- 172.21.15.16:0/4261701775 learned_addr learned my addr 172.21.15.16:0/4261701775 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:03:20.430 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:20.422+0000 7f095ce6f640 1 --2- 172.21.15.16:0/4261701775 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f09581605d0 0x7f095814d0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:20.430 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:20.422+0000 7f0957fff640 1 --2- 172.21.15.16:0/4261701775 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f09580ce480 0x7f095814fbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:20.430 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:20.422+0000 7f09577fe640 1 -- 172.21.15.16:0/4261701775 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f09581605d0 msgr2=0x7f095814d0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:20.430 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:20.422+0000 7f09577fe640 1 --2- 172.21.15.16:0/4261701775 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f09581605d0 0x7f095814d0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:20.430 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:20.422+0000 7f09577fe640 1 -- 172.21.15.16:0/4261701775 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f09580ce480 msgr2=0x7f095814fbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:20.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:20.422+0000 7f09577fe640 1 --2- 172.21.15.16:0/4261701775 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f09580ce480 0x7f095814fbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:20.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:20.426+0000 7f09577fe640 1 -- 172.21.15.16:0/4261701775 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f094c0094d0 con 0x7f09580ce850 2023-05-22T16:03:20.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:20.426+0000 7f0957fff640 1 --2- 172.21.15.16:0/4261701775 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f09580ce480 0x7f095814fbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T16:03:20.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:20.426+0000 7f09577fe640 1 --2- 172.21.15.16:0/4261701775 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f09580ce850 0x7f09580ab840 secure :-1 s=READY pgs=2139 cs=0 l=1 rev1=1 crypto rx=0x7f094c0af040 tx=0x7f094c019270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:03:20.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:20.426+0000 7f0954ff9640 1 -- 172.21.15.16:0/4261701775 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f094c0ac410 con 0x7f09580ce850 2023-05-22T16:03:20.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:20.426+0000 7f0954ff9640 1 -- 172.21.15.16:0/4261701775 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f094c0b5440 con 0x7f09580ce850 2023-05-22T16:03:20.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:20.426+0000 7f095e8d2640 1 -- 172.21.15.16:0/4261701775 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09580698b0 con 0x7f09580ce850 2023-05-22T16:03:20.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:20.426+0000 7f0954ff9640 1 -- 172.21.15.16:0/4261701775 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f094c0b5810 con 0x7f09580ce850 2023-05-22T16:03:20.432 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:20.426+0000 7f095e8d2640 1 -- 172.21.15.16:0/4261701775 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0958069720 con 0x7f09580ce850 2023-05-22T16:03:20.432 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:20.426+0000 7f095e8d2640 1 -- 172.21.15.16:0/4261701775 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09580c5e70 con 0x7f09580ce850 2023-05-22T16:03:20.433 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:20.426+0000 7f0954ff9640 1 -- 172.21.15.16:0/4261701775 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 79) v1 ==== 84344+0+0 (secure 0 0 0) 0x7f094c0b3070 con 0x7f09580ce850 2023-05-22T16:03:20.433 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:20.426+0000 7f0954ff9640 1 --2- 172.21.15.16:0/4261701775 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f09440654a0 0x7f0944067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:20.433 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:20.426+0000 7f0957fff640 1 -- 172.21.15.16:0/4261701775 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f09440654a0 msgr2=0x7f0944067950 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/65890 2023-05-22T16:03:20.433 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:20.426+0000 7f0954ff9640 1 -- 172.21.15.16:0/4261701775 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(165..165 src has 1..165) v4 ==== 7041+0+0 (secure 0 0 0) 0x7f094c11ce70 con 0x7f09580ce850 2023-05-22T16:03:20.433 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:20.426+0000 7f0957fff640 1 --2- 172.21.15.16:0/4261701775 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f09440654a0 0x7f0944067950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-05-22T16:03:20.435 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:20.430+0000 7f0954ff9640 1 -- 172.21.15.16:0/4261701775 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f094c0ea460 con 0x7f09580ce850 2023-05-22T16:03:20.589 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:20.586+0000 7f095e8d2640 1 -- 172.21.15.16:0/4261701775 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f095814ed30 con 0x7f09580ce850 2023-05-22T16:03:20.590 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:20.586+0000 7f0954ff9640 1 -- 172.21.15.16:0/4261701775 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v165) v1 ==== 74+0+19005 (secure 0 0 0) 0x7f094c0b1090 con 0x7f09580ce850 2023-05-22T16:03:20.591 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T16:03:20.591 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":165,"fsid":"b2719834-7847-4d98-90a6-f65d57f85c49","created":"2023-05-22T15:26:25.259314+0000","modified":"2023-05-22T15:58:20.479654+0000","last_up_change":"2023-05-22T15:26:36.448524+0000","last_in_change":"2023-05-22T15:26:27.675668+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-05-22T15:26:38.019781+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-05-22T15:26:43.570514+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_metadata","create_time":"2023-05-22T15:47:01.347542+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"160","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_data","create_time":"2023-05-22T15:47:01.970646+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"161","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9e7ff854-b25f-4cbb-a5c2-1799d9c53d2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":37021},{"type":"v1","addr":"172.21.15.16:6825","nonce":37021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":37021},{"type":"v1","addr":"172.21.15.16:6827","nonce":37021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":37021},{"type":"v1","addr":"172.21.15.16:6831","nonce":37021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":37021},{"type":"v1","addr":"172.21.15.16:6829","nonce":37021}]},"public_addr":"172.21.15.16:6825/37021","cluster_addr":"172.21.15.16:6827/37021","heartbeat_back_addr":"172.21.15.16:6831/37021","heartbeat_front_addr":"172.21.15.16:6829/37021","state":["exists","up"]},{"osd":1,"uuid":"4702fbc9-a927-4565-aca5-41dc69cc8264","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":37008},{"type":"v1","addr":"172.21.15.16:6801","nonce":37008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":37008},{"type":"v1","addr":"172.21.15.16:6803","nonce":37008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":37008},{"type":"v1","addr":"172.21.15.16:6807","nonce":37008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":37008},{"type":"v1","addr":"172.21.15.16:6805","nonce":37008}]},"public_addr":"172.21.15.16:6801/37008","cluster_addr":"172.21.15.16:6803/37008","heartbeat_back_addr":"172.21.15.16:6807/37008","heartbeat_front_addr":"172.21.15.16:6805/37008","state":["exists","up"]},{"osd":2,"uuid":"1e7bba46-4eba-47ef-bb81-5e2d371638ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":37009},{"type":"v1","addr":"172.21.15.16:6809","nonce":37009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":37009},{"type":"v1","addr":"172.21.15.16:6811","nonce":37009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":37009},{"type":"v1","addr":"172.21.15.16:6815","nonce":37009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":37009},{"type":"v1","addr":"172.21.15.16:6813","nonce":37009}]},"public_addr":"172.21.15.16:6809/37009","cluster_addr":"172.21.15.16:6811/37009","heartbeat_back_addr":"172.21.15.16:6815/37009","heartbeat_front_addr":"172.21.15.16:6813/37009","state":["exists","up"]},{"osd":3,"uuid":"383cacc0-5a88-4904-990e-498ca59ef48a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":37020},{"type":"v1","addr":"172.21.15.16:6817","nonce":37020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":37020},{"type":"v1","addr":"172.21.15.16:6819","nonce":37020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":37020},{"type":"v1","addr":"172.21.15.16:6823","nonce":37020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":37020},{"type":"v1","addr":"172.21.15.16:6821","nonce":37020}]},"public_addr":"172.21.15.16:6817/37020","cluster_addr":"172.21.15.16:6819/37020","heartbeat_back_addr":"172.21.15.16:6823/37020","heartbeat_front_addr":"172.21.15.16:6821/37020","state":["exists","up"]},{"osd":4,"uuid":"970500ef-95b7-42c1-b080-eed4fae7013d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":37353},{"type":"v1","addr":"172.21.15.62:6817","nonce":37353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6819","nonce":37353},{"type":"v1","addr":"172.21.15.62:6822","nonce":37353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37353},{"type":"v1","addr":"172.21.15.62:6831","nonce":37353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37353},{"type":"v1","addr":"172.21.15.62:6826","nonce":37353}]},"public_addr":"172.21.15.62:6817/37353","cluster_addr":"172.21.15.62:6822/37353","heartbeat_back_addr":"172.21.15.62:6831/37353","heartbeat_front_addr":"172.21.15.62:6826/37353","state":["exists","up"]},{"osd":5,"uuid":"022d67db-d4d8-445b-b1b9-8b446a00f5c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":37352},{"type":"v1","addr":"172.21.15.62:6801","nonce":37352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":37352},{"type":"v1","addr":"172.21.15.62:6803","nonce":37352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":37352},{"type":"v1","addr":"172.21.15.62:6807","nonce":37352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":37352},{"type":"v1","addr":"172.21.15.62:6805","nonce":37352}]},"public_addr":"172.21.15.62:6801/37352","cluster_addr":"172.21.15.62:6803/37352","heartbeat_back_addr":"172.21.15.62:6807/37352","heartbeat_front_addr":"172.21.15.62:6805/37352","state":["exists","up"]},{"osd":6,"uuid":"42bd372b-6688-4a1b-8407-e16e8c266e33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":37354},{"type":"v1","addr":"172.21.15.62:6820","nonce":37354}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":37354},{"type":"v1","addr":"172.21.15.62:6823","nonce":37354}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6829","nonce":37354},{"type":"v1","addr":"172.21.15.62:6830","nonce":37354}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6825","nonce":37354},{"type":"v1","addr":"172.21.15.62:6827","nonce":37354}]},"public_addr":"172.21.15.62:6820/37354","cluster_addr":"172.21.15.62:6823/37354","heartbeat_back_addr":"172.21.15.62:6830/37354","heartbeat_front_addr":"172.21.15.62:6827/37354","state":["exists","up"]},{"osd":7,"uuid":"ac70b2a3-f41b-4bf2-b227-7f2cbb55426d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":163,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37355},{"type":"v1","addr":"172.21.15.62:6809","nonce":37355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37355},{"type":"v1","addr":"172.21.15.62:6811","nonce":37355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":37355},{"type":"v1","addr":"172.21.15.62:6815","nonce":37355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":37355},{"type":"v1","addr":"172.21.15.62:6813","nonce":37355}]},"public_addr":"172.21.15.62:6809/37355","cluster_addr":"172.21.15.62:6811/37355","heartbeat_back_addr":"172.21.15.62:6815/37355","heartbeat_front_addr":"172.21.15.62:6813/37355","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.368896+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.107273+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.276888+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.287502+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.124791+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.251937+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.255679+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.232797+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"21.28","mappings":[{"from":4,"to":3}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:0/2019135485":"2023-05-23T15:58:20.479559+0000","172.21.15.16:0/3082249167":"2023-05-23T15:58:20.479559+0000","172.21.15.16:0/4113149961":"2023-05-23T15:58:20.479559+0000","172.21.15.16:0/2582627335":"2023-05-23T15:58:20.479559+0000","172.21.15.16:0/2722168808":"2023-05-23T15:47:16.536062+0000","172.21.15.16:0/1232860864":"2023-05-23T15:47:16.536062+0000","172.21.15.16:0/3766829209":"2023-05-23T15:47:16.536062+0000","172.21.15.16:0/3369591631":"2023-05-23T15:47:16.536062+0000","172.21.15.16:0/3417178745":"2023-05-23T15:58:20.479559+0000","172.21.15.16:6833/36921":"2023-05-23T15:47:16.536062+0000","172.21.15.16:0/2592954608":"2023-05-23T15:58:20.479559+0000","172.21.15.16:0/49958236":"2023-05-23T15:47:16.536062+0000","172.21.15.16:6832/36921":"2023-05-23T15:47:16.536062+0000","172.21.15.16:0/4161645610":"2023-05-23T15:47:16.536062+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":17,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-05-22T16:03:20.593 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:20.590+0000 7f095e8d2640 1 -- 172.21.15.16:0/4261701775 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f09440654a0 msgr2=0x7f0944067950 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-05-22T16:03:20.593 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:20.590+0000 7f095e8d2640 1 --2- 172.21.15.16:0/4261701775 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f09440654a0 0x7f0944067950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:20.593 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:20.590+0000 7f095e8d2640 1 -- 172.21.15.16:0/4261701775 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f09580ce850 msgr2=0x7f09580ab840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:20.593 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:20.590+0000 7f095e8d2640 1 --2- 172.21.15.16:0/4261701775 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f09580ce850 0x7f09580ab840 secure :-1 s=READY pgs=2139 cs=0 l=1 rev1=1 crypto rx=0x7f094c0af040 tx=0x7f094c019270 comp rx=0 tx=0).stop 2023-05-22T16:03:20.593 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:20.590+0000 7f095e8d2640 1 -- 172.21.15.16:0/4261701775 shutdown_connections 2023-05-22T16:03:20.593 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:20.590+0000 7f095e8d2640 1 --2- 172.21.15.16:0/4261701775 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f09581605d0 0x7f095814d0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:20.594 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:20.590+0000 7f095e8d2640 1 --2- 172.21.15.16:0/4261701775 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f09440654a0 0x7f0944067950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:20.594 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:20.590+0000 7f095e8d2640 1 --2- 172.21.15.16:0/4261701775 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f09580ce850 0x7f09580ab840 unknown :-1 s=CLOSED pgs=2139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:20.594 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:20.590+0000 7f095e8d2640 1 --2- 172.21.15.16:0/4261701775 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f09580ce480 0x7f095814fbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:20.594 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:20.590+0000 7f095e8d2640 1 -- 172.21.15.16:0/4261701775 >> 172.21.15.16:0/4261701775 conn(0x7f09580cc870 msgr2=0x7f095806e990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:03:20.594 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:20.590+0000 7f095e8d2640 1 -- 172.21.15.16:0/4261701775 shutdown_connections 2023-05-22T16:03:20.594 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:20.590+0000 7f095e8d2640 1 -- 172.21.15.16:0/4261701775 wait complete. 2023-05-22T16:03:20.609 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-05-22T15:26:38.019781+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-05-22T15:26:43.570514+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 20, 'pool_name': 'cephfs_metadata', 'create_time': '2023-05-22T15:47:01.347542+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '160', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 21, 'pool_name': 'cephfs_data', 'create_time': '2023-05-22T15:47:01.970646+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '161', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-05-22T16:03:20.610 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-05-22T16:03:20.702 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T16:03:20.698+0000 7fd0922ba180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-05-22T16:03:20.702 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T16:03:20.697+0000 7f46151e9180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-05-22T16:03:20.741 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:20.738+0000 7f4de557d640 1 Processor -- start 2023-05-22T16:03:20.741 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:20.738+0000 7f4de557d640 1 -- start start 2023-05-22T16:03:20.744 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:20.738+0000 7f4de557d640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f4de006cdf0 0x7f4de0070be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:20.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:20.738+0000 7f4de557d640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4de006eb90 con 0x7f4de0070760 2023-05-22T16:03:20.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:20.738+0000 7f4de557d640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4de0158de0 con 0x7f4de0158a10 2023-05-22T16:03:20.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:20.738+0000 7f4de557d640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4de006e060 con 0x7f4de006cdf0 2023-05-22T16:03:20.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:20.738+0000 7f4dd7fff640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f4de0070760 0x7f4de006c6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:41942/0 (socket says 172.21.15.16:41942) 2023-05-22T16:03:20.746 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:20.738+0000 7f4dd7fff640 1 -- 172.21.15.16:0/3397407754 learned_addr learned my addr 172.21.15.16:0/3397407754 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T16:03:20.746 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:20.738+0000 7f4ddf7fe640 1 --2- 172.21.15.16:0/3397407754 >> v2:172.21.15.62:3300/0 conn(0x7f4de006cdf0 0x7f4de0070be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:20.746 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:20.738+0000 7f4dde7fc640 1 -- 172.21.15.16:0/3397407754 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1233772796 0 0) 0x7f4de006eb90 con 0x7f4de0070760 2023-05-22T16:03:20.747 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:20.738+0000 7f4dde7fc640 1 -- 172.21.15.16:0/3397407754 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4dcc005740 con 0x7f4de0070760 2023-05-22T16:03:20.747 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:20.738+0000 7f4dde7fc640 1 -- 172.21.15.16:0/3397407754 <== mon.1 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4128634801 0 0) 0x7f4de0158de0 con 0x7f4de0158a10 2023-05-22T16:03:20.747 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:20.738+0000 7f4dde7fc640 1 -- 172.21.15.16:0/3397407754 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4de006eb90 con 0x7f4de0158a10 2023-05-22T16:03:20.747 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:20.738+0000 7f4dde7fc640 1 -- 172.21.15.16:0/3397407754 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4039815357 0 0) 0x7f4dcc005740 con 0x7f4de0070760 2023-05-22T16:03:20.747 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:20.738+0000 7f4dde7fc640 1 -- 172.21.15.16:0/3397407754 >> v2:172.21.15.62:3300/0 conn(0x7f4de006cdf0 msgr2=0x7f4de0070be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:03:20.747 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:20.738+0000 7f4dde7fc640 1 --2- 172.21.15.16:0/3397407754 >> v2:172.21.15.62:3300/0 conn(0x7f4de006cdf0 0x7f4de0070be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:20.748 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:20.738+0000 7f4dde7fc640 1 -- 172.21.15.16:0/3397407754 >> v1:172.21.15.62:6789/0 conn(0x7f4de0158a10 legacy=0x7f4de006e480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:20.748 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:20.738+0000 7f4dde7fc640 1 -- 172.21.15.16:0/3397407754 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4de00711e0 con 0x7f4de0070760 2023-05-22T16:03:20.748 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:20.738+0000 7f4dde7fc640 1 -- 172.21.15.16:0/3397407754 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7f4dc8001680 con 0x7f4de0070760 2023-05-22T16:03:20.748 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:20.738+0000 7f4dde7fc640 1 -- 172.21.15.16:0/3397407754 >> v1:172.21.15.16:6789/0 conn(0x7f4de0070760 legacy=0x7f4de006c6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:20.748 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:20.738+0000 7f4dde7fc640 1 --2- 172.21.15.16:0/3397407754 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4dcc005d70 0x7f4dcc006160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:20.748 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:20.738+0000 7f4dde7fc640 1 --2- 172.21.15.16:0/3397407754 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4dcc0069e0 0x7f4dcc006e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:20.748 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:20.738+0000 7f4dde7fc640 1 --2- 172.21.15.16:0/3397407754 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4dcc0073e0 0x7f4dcc00dba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:20.749 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:20.738+0000 7f4dde7fc640 1 -- 172.21.15.16:0/3397407754 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4de006e060 con 0x7f4dcc0073e0 2023-05-22T16:03:20.749 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:20.738+0000 7f4dde7fc640 1 -- 172.21.15.16:0/3397407754 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dcc00e2f0 con 0x7f4dcc0069e0 2023-05-22T16:03:20.749 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:20.738+0000 7f4dde7fc640 1 -- 172.21.15.16:0/3397407754 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4dcc00e4b0 con 0x7f4dcc005d70 2023-05-22T16:03:20.749 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:20.738+0000 7f4ddf7fe640 1 --2- 172.21.15.16:0/3397407754 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4dcc0073e0 0x7f4dcc00dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:20.749 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:20.738+0000 7f4dd7fff640 1 --2- 172.21.15.16:0/3397407754 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4dcc0069e0 0x7f4dcc006e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:20.749 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:20.738+0000 7f4ddeffd640 1 --2- 172.21.15.16:0/3397407754 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4dcc005d70 0x7f4dcc006160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:20.749 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:20.738+0000 7f4ddf7fe640 1 -- 172.21.15.16:0/3397407754 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4dcc005d70 msgr2=0x7f4dcc006160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:03:20.749 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:20.738+0000 7f4ddf7fe640 1 --2- 172.21.15.16:0/3397407754 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4dcc005d70 0x7f4dcc006160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:20.750 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:20.738+0000 7f4ddf7fe640 1 -- 172.21.15.16:0/3397407754 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4dcc0069e0 msgr2=0x7f4dcc006e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:03:20.750 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:20.738+0000 7f4ddf7fe640 1 --2- 172.21.15.16:0/3397407754 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4dcc0069e0 0x7f4dcc006e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:20.750 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:20.738+0000 7f4ddf7fe640 1 -- 172.21.15.16:0/3397407754 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4dcc005740 con 0x7f4dcc0073e0 2023-05-22T16:03:20.753 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:20.742+0000 7f4ddf7fe640 1 -- 172.21.15.16:0/3397407754 reap_dead start 2023-05-22T16:03:20.753 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:20.742+0000 7f4ddf7fe640 1 --2- 172.21.15.16:0/3397407754 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4dcc0073e0 0x7f4dcc00dba0 secure :-1 s=READY pgs=2141 cs=0 l=1 rev1=1 crypto rx=0x7f4de00b50b0 tx=0x7f4dd801cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4adcd2bc23ac71bb server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:03:20.753 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:20.742+0000 7f4dde7fc640 1 -- 172.21.15.16:0/3397407754 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dd80a5c50 con 0x7f4dcc0073e0 2023-05-22T16:03:20.753 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:20.742+0000 7f4dde7fc640 1 -- 172.21.15.16:0/3397407754 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4dd800f040 con 0x7f4dcc0073e0 2023-05-22T16:03:20.753 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:20.742+0000 7f4dde7fc640 1 -- 172.21.15.16:0/3397407754 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dd80039a0 con 0x7f4dcc0073e0 2023-05-22T16:03:20.754 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:20.742+0000 7f4de557d640 1 -- 172.21.15.16:0/3397407754 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4dcc0073e0 msgr2=0x7f4dcc00dba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:20.754 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:20.742+0000 7f4de557d640 1 --2- 172.21.15.16:0/3397407754 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4dcc0073e0 0x7f4dcc00dba0 secure :-1 s=READY pgs=2141 cs=0 l=1 rev1=1 crypto rx=0x7f4de00b50b0 tx=0x7f4dd801cc40 comp rx=0 tx=0).stop 2023-05-22T16:03:20.754 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:20.742+0000 7f4de557d640 1 -- 172.21.15.16:0/3397407754 shutdown_connections 2023-05-22T16:03:20.754 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:20.742+0000 7f4de557d640 1 --2- 172.21.15.16:0/3397407754 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4dcc0073e0 0x7f4dcc00dba0 unknown :-1 s=CLOSED pgs=2141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:20.754 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:20.742+0000 7f4de557d640 1 -- 172.21.15.16:0/3397407754 >> 172.21.15.16:0/3397407754 conn(0x7f4de00cc870 msgr2=0x7f4de00cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:03:20.754 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:20.742+0000 7f4de557d640 1 -- 172.21.15.16:0/3397407754 shutdown_connections 2023-05-22T16:03:20.754 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:20.742+0000 7f4de557d640 1 -- 172.21.15.16:0/3397407754 wait complete. 2023-05-22T16:03:20.754 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:20.746+0000 7f4de557d640 1 Processor -- start 2023-05-22T16:03:20.755 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:20.746+0000 7f4de557d640 1 -- start start 2023-05-22T16:03:20.755 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:20.746+0000 7f4de557d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4dcc0073e0 0x7f4de00c6cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:20.755 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:20.746+0000 7f4de557d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4de00c7e40 0x7f4de01f0be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:20.755 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:20.746+0000 7f4de557d640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4de015b1a0 0x7f4de015b640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:20.755 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:20.746+0000 7f4de557d640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4de015bb80 con 0x7f4dcc0073e0 2023-05-22T16:03:20.755 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:20.746+0000 7f4de557d640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4de00c8690 con 0x7f4de00c7e40 2023-05-22T16:03:20.756 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:20.746+0000 7f4de557d640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4de00b2130 con 0x7f4de015b1a0 2023-05-22T16:03:20.756 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:20.746+0000 7f4ddeffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4dcc0073e0 0x7f4de00c6cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:20.756 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:20.746+0000 7f4ddeffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4dcc0073e0 0x7f4de00c6cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55034/0 (socket says 172.21.15.16:55034) 2023-05-22T16:03:20.756 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:20.746+0000 7f4ddeffd640 1 -- 172.21.15.16:0/1500150946 learned_addr learned my addr 172.21.15.16:0/1500150946 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:03:20.756 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:20.746+0000 7f4dd7fff640 1 --2- 172.21.15.16:0/1500150946 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4de00c7e40 0x7f4de01f0be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:20.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:20.746+0000 7f4ddf7fe640 1 --2- 172.21.15.16:0/1500150946 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4de015b1a0 0x7f4de015b640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:20.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:20.746+0000 7f4ddeffd640 1 -- 172.21.15.16:0/1500150946 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4de015b1a0 msgr2=0x7f4de015b640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:20.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:20.746+0000 7f4ddeffd640 1 --2- 172.21.15.16:0/1500150946 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4de015b1a0 0x7f4de015b640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:20.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:20.746+0000 7f4ddeffd640 1 -- 172.21.15.16:0/1500150946 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4de00c7e40 msgr2=0x7f4de01f0be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:20.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:20.746+0000 7f4ddeffd640 1 --2- 172.21.15.16:0/1500150946 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4de00c7e40 0x7f4de01f0be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:20.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:20.746+0000 7f4ddeffd640 1 -- 172.21.15.16:0/1500150946 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4de006dc80 con 0x7f4dcc0073e0 2023-05-22T16:03:20.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:20.746+0000 7f4ddeffd640 1 --2- 172.21.15.16:0/1500150946 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4dcc0073e0 0x7f4de00c6cd0 secure :-1 s=READY pgs=2142 cs=0 l=1 rev1=1 crypto rx=0x7f4dd000f870 tx=0x7f4dd000fd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:03:20.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:20.746+0000 7f4dd77fe640 1 -- 172.21.15.16:0/1500150946 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dd000d870 con 0x7f4dcc0073e0 2023-05-22T16:03:20.758 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:20.746+0000 7f4dd77fe640 1 -- 172.21.15.16:0/1500150946 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4dd000d9d0 con 0x7f4dcc0073e0 2023-05-22T16:03:20.758 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:20.746+0000 7f4de557d640 1 -- 172.21.15.16:0/1500150946 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4de00714c0 con 0x7f4dcc0073e0 2023-05-22T16:03:20.758 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:20.746+0000 7f4dd77fe640 1 -- 172.21.15.16:0/1500150946 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dd000dd60 con 0x7f4dcc0073e0 2023-05-22T16:03:20.758 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:20.746+0000 7f4de557d640 1 -- 172.21.15.16:0/1500150946 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4de00cfe40 con 0x7f4dcc0073e0 2023-05-22T16:03:20.758 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:20.746+0000 7f4de557d640 1 -- 172.21.15.16:0/1500150946 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4de00b1f90 con 0x7f4dcc0073e0 2023-05-22T16:03:20.758 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:20.746+0000 7f4dd77fe640 1 -- 172.21.15.16:0/1500150946 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 79) v1 ==== 84344+0+0 (secure 0 0 0) 0x7f4dd0014400 con 0x7f4dcc0073e0 2023-05-22T16:03:20.758 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:20.750+0000 7f4dd77fe640 1 --2- 172.21.15.16:0/1500150946 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f4dc00654d0 0x7f4dc0067980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:20.759 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:20.750+0000 7f4dd7fff640 1 -- 172.21.15.16:0/1500150946 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f4dc00654d0 msgr2=0x7f4dc0067980 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/65890 2023-05-22T16:03:20.759 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:20.750+0000 7f4dd77fe640 1 -- 172.21.15.16:0/1500150946 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(165..165 src has 1..165) v4 ==== 7041+0+0 (secure 0 0 0) 0x7f4dd0085f00 con 0x7f4dcc0073e0 2023-05-22T16:03:20.759 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:20.750+0000 7f4dd7fff640 1 --2- 172.21.15.16:0/1500150946 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f4dc00654d0 0x7f4dc0067980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-05-22T16:03:20.759 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:20.754+0000 7f4dd77fe640 1 -- 172.21.15.16:0/1500150946 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f4de00b1f90 con 0x7f4dcc0073e0 2023-05-22T16:03:20.839 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T16:03:20.833+0000 7f46151e9180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-05-22T16:03:20.844 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T16:03:20.838+0000 7fd0922ba180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-05-22T16:03:20.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:20.906+0000 7f4de557d640 1 -- 172.21.15.16:0/1500150946 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f4de00b1f90 con 0x7f4dcc0073e0 2023-05-22T16:03:20.911 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:20.906+0000 7f4dd77fe640 1 -- 172.21.15.16:0/1500150946 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v165) v1 ==== 93+0+10 (secure 0 0 0) 0x7f4de00b1f90 con 0x7f4dcc0073e0 2023-05-22T16:03:20.911 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2023-05-22T16:03:20.913 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:20.910+0000 7f4de557d640 1 -- 172.21.15.16:0/1500150946 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f4dc00654d0 msgr2=0x7f4dc0067980 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-05-22T16:03:20.913 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:20.910+0000 7f4de557d640 1 --2- 172.21.15.16:0/1500150946 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f4dc00654d0 0x7f4dc0067980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:20.913 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:20.910+0000 7f4de557d640 1 -- 172.21.15.16:0/1500150946 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4dcc0073e0 msgr2=0x7f4de00c6cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:20.913 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:20.910+0000 7f4de557d640 1 --2- 172.21.15.16:0/1500150946 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4dcc0073e0 0x7f4de00c6cd0 secure :-1 s=READY pgs=2142 cs=0 l=1 rev1=1 crypto rx=0x7f4dd000f870 tx=0x7f4dd000fd30 comp rx=0 tx=0).stop 2023-05-22T16:03:20.914 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:20.910+0000 7f4de557d640 1 -- 172.21.15.16:0/1500150946 shutdown_connections 2023-05-22T16:03:20.914 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:20.910+0000 7f4de557d640 1 --2- 172.21.15.16:0/1500150946 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4de015b1a0 0x7f4de015b640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:20.914 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:20.910+0000 7f4de557d640 1 --2- 172.21.15.16:0/1500150946 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4de00c7e40 0x7f4de01f0be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:20.914 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:20.910+0000 7f4de557d640 1 --2- 172.21.15.16:0/1500150946 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f4dc00654d0 0x7f4dc0067980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:20.914 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:20.910+0000 7f4de557d640 1 --2- 172.21.15.16:0/1500150946 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4dcc0073e0 0x7f4de00c6cd0 unknown :-1 s=CLOSED pgs=2142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:20.914 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:20.910+0000 7f4de557d640 1 -- 172.21.15.16:0/1500150946 >> 172.21.15.16:0/1500150946 conn(0x7f4de00cc870 msgr2=0x7f4de00ce3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:03:20.914 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:20.910+0000 7f4de557d640 1 -- 172.21.15.16:0/1500150946 shutdown_connections 2023-05-22T16:03:20.915 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:20.910+0000 7f4de557d640 1 -- 172.21.15.16:0/1500150946 wait complete. 2023-05-22T16:03:20.929 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-05-22T16:03:21.052 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.046+0000 7f30c6611640 1 Processor -- start 2023-05-22T16:03:21.052 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.046+0000 7f30c6611640 1 -- start start 2023-05-22T16:03:21.053 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.050+0000 7f30c6611640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f30c00c67d0 0x7f30c00cf730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:21.053 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.050+0000 7f30c6611640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f30c00b2880 0x7f30c00cbfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:21.053 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.050+0000 7f30c6611640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30c00cfc70 con 0x7f30c00cc520 2023-05-22T16:03:21.053 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.050+0000 7f30c6611640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30c015a300 con 0x7f30c00c67d0 2023-05-22T16:03:21.053 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.050+0000 7f30c6611640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30c00c4dc0 con 0x7f30c00b2880 2023-05-22T16:03:21.053 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.050+0000 7f30bffff640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f30c00c67d0 0x7f30c00cf730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:21.054 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.050+0000 7f30bffff640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f30c00c67d0 0x7f30c00cf730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55038/0 (socket says 172.21.15.16:55038) 2023-05-22T16:03:21.054 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.050+0000 7f30bffff640 1 -- 172.21.15.16:0/3047110281 learned_addr learned my addr 172.21.15.16:0/3047110281 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:03:21.054 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.050+0000 7f30bf7fe640 1 --2- 172.21.15.16:0/3047110281 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f30c00b2880 0x7f30c00cbfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:21.054 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.050+0000 7f30bffff640 1 -- 172.21.15.16:0/3047110281 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f30c00b2880 msgr2=0x7f30c00cbfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:03:21.055 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.050+0000 7f30bffff640 1 --2- 172.21.15.16:0/3047110281 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f30c00b2880 0x7f30c00cbfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:21.055 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.050+0000 7f30bffff640 1 -- 172.21.15.16:0/3047110281 >> v1:172.21.15.62:6789/0 conn(0x7f30c00cc520 legacy=0x7f30c00ce160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:03:21.055 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.050+0000 7f30bffff640 1 -- 172.21.15.16:0/3047110281 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30c00c6ba0 con 0x7f30c00c67d0 2023-05-22T16:03:21.055 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.050+0000 7f30bffff640 1 --2- 172.21.15.16:0/3047110281 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f30c00c67d0 0x7f30c00cf730 secure :-1 s=READY pgs=2143 cs=0 l=1 rev1=1 crypto rx=0x7f30b8009e50 tx=0x7f30b801c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69a0b280c3cd0f3c server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:03:21.056 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.050+0000 7f30beffd640 1 -- 172.21.15.16:0/3047110281 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30b80a56b0 con 0x7f30c00c67d0 2023-05-22T16:03:21.056 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.050+0000 7f30beffd640 1 -- 172.21.15.16:0/3047110281 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30b80a5810 con 0x7f30c00c67d0 2023-05-22T16:03:21.056 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.050+0000 7f30beffd640 1 -- 172.21.15.16:0/3047110281 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30b80a5ba0 con 0x7f30c00c67d0 2023-05-22T16:03:21.057 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.054+0000 7f30c6611640 1 -- 172.21.15.16:0/3047110281 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f30c00c67d0 msgr2=0x7f30c00cf730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:21.057 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.054+0000 7f30c6611640 1 --2- 172.21.15.16:0/3047110281 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f30c00c67d0 0x7f30c00cf730 secure :-1 s=READY pgs=2143 cs=0 l=1 rev1=1 crypto rx=0x7f30b8009e50 tx=0x7f30b801c910 comp rx=0 tx=0).stop 2023-05-22T16:03:21.057 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.054+0000 7f30c6611640 1 -- 172.21.15.16:0/3047110281 shutdown_connections 2023-05-22T16:03:21.057 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.054+0000 7f30c6611640 1 --2- 172.21.15.16:0/3047110281 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f30c00b2880 0x7f30c00cbfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:21.057 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.054+0000 7f30c6611640 1 --2- 172.21.15.16:0/3047110281 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f30c00c67d0 0x7f30c00cf730 unknown :-1 s=CLOSED pgs=2143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:21.057 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.054+0000 7f30c6611640 1 -- 172.21.15.16:0/3047110281 >> 172.21.15.16:0/3047110281 conn(0x7f30c006f5a0 msgr2=0x7f30c006f9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:03:21.058 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.054+0000 7f30c6611640 1 -- 172.21.15.16:0/3047110281 shutdown_connections 2023-05-22T16:03:21.058 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.054+0000 7f30c6611640 1 -- 172.21.15.16:0/3047110281 wait complete. 2023-05-22T16:03:21.059 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.054+0000 7f30c6611640 1 Processor -- start 2023-05-22T16:03:21.059 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.054+0000 7f30c6611640 1 -- start start 2023-05-22T16:03:21.060 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.054+0000 7f30c6611640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f30c00b2880 0x7f30c01522b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:21.060 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.054+0000 7f30c6611640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f30c00c67d0 0x7f30c0159500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:21.060 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.054+0000 7f30c6611640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f30c00cc520 0x7f30c0155320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:21.060 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.054+0000 7f30c6611640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30c0158c10 con 0x7f30c00b2880 2023-05-22T16:03:21.060 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.054+0000 7f30c6611640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30c0159a40 con 0x7f30c00cc520 2023-05-22T16:03:21.060 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.054+0000 7f30c6611640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30c0158000 con 0x7f30c00c67d0 2023-05-22T16:03:21.060 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.054+0000 7f30bffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f30c00b2880 0x7f30c01522b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:21.061 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.054+0000 7f30bffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f30c00b2880 0x7f30c01522b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55054/0 (socket says 172.21.15.16:55054) 2023-05-22T16:03:21.061 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.054+0000 7f30bffff640 1 -- 172.21.15.16:0/2801872171 learned_addr learned my addr 172.21.15.16:0/2801872171 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:03:21.061 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.054+0000 7f30bf7fe640 1 --2- 172.21.15.16:0/2801872171 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f30c00c67d0 0x7f30c0159500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:21.061 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.054+0000 7f30c4bae640 1 --2- 172.21.15.16:0/2801872171 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f30c00cc520 0x7f30c0155320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:21.061 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.054+0000 7f30bffff640 1 -- 172.21.15.16:0/2801872171 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f30c00c67d0 msgr2=0x7f30c0159500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:21.061 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.054+0000 7f30bffff640 1 --2- 172.21.15.16:0/2801872171 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f30c00c67d0 0x7f30c0159500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:21.061 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.054+0000 7f30bffff640 1 -- 172.21.15.16:0/2801872171 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f30c00cc520 msgr2=0x7f30c0155320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:21.061 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.054+0000 7f30bffff640 1 --2- 172.21.15.16:0/2801872171 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f30c00cc520 0x7f30c0155320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:21.062 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.054+0000 7f30bffff640 1 -- 172.21.15.16:0/2801872171 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30b8009b30 con 0x7f30c00b2880 2023-05-22T16:03:21.062 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.054+0000 7f30c4bae640 1 --2- 172.21.15.16:0/2801872171 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f30c00cc520 0x7f30c0155320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T16:03:21.062 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.054+0000 7f30bffff640 1 --2- 172.21.15.16:0/2801872171 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f30c00b2880 0x7f30c01522b0 secure :-1 s=READY pgs=2144 cs=0 l=1 rev1=1 crypto rx=0x7f30b80af040 tx=0x7f30b80196b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:03:21.062 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.054+0000 7f30bcff9640 1 -- 172.21.15.16:0/2801872171 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30b80b8670 con 0x7f30c00b2880 2023-05-22T16:03:21.062 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.054+0000 7f30bcff9640 1 -- 172.21.15.16:0/2801872171 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30b80b87f0 con 0x7f30c00b2880 2023-05-22T16:03:21.062 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.054+0000 7f30c6611640 1 -- 172.21.15.16:0/2801872171 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30c00d0790 con 0x7f30c00b2880 2023-05-22T16:03:21.062 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.054+0000 7f30bcff9640 1 -- 172.21.15.16:0/2801872171 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30b80b8ba0 con 0x7f30c00b2880 2023-05-22T16:03:21.063 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.054+0000 7f30c6611640 1 -- 172.21.15.16:0/2801872171 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30c00d04b0 con 0x7f30c00b2880 2023-05-22T16:03:21.063 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.054+0000 7f30c6611640 1 -- 172.21.15.16:0/2801872171 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30c00d0a70 con 0x7f30c00b2880 2023-05-22T16:03:21.063 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.058+0000 7f30bcff9640 1 -- 172.21.15.16:0/2801872171 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 79) v1 ==== 84344+0+0 (secure 0 0 0) 0x7f30b80b8d00 con 0x7f30c00b2880 2023-05-22T16:03:21.064 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.058+0000 7f30bcff9640 1 --2- 172.21.15.16:0/2801872171 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f30a00654a0 0x7f30a0067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:21.064 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.058+0000 7f30bf7fe640 1 -- 172.21.15.16:0/2801872171 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f30a00654a0 msgr2=0x7f30a0067950 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/65890 2023-05-22T16:03:21.064 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.058+0000 7f30bcff9640 1 -- 172.21.15.16:0/2801872171 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(165..165 src has 1..165) v4 ==== 7041+0+0 (secure 0 0 0) 0x7f30b811e750 con 0x7f30c00b2880 2023-05-22T16:03:21.064 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.058+0000 7f30bf7fe640 1 --2- 172.21.15.16:0/2801872171 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f30a00654a0 0x7f30a0067950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-05-22T16:03:21.068 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.066+0000 7f30bcff9640 1 -- 172.21.15.16:0/2801872171 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f30b80ebec0 con 0x7f30c00b2880 2023-05-22T16:03:21.220 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.214+0000 7f30c6611640 1 -- 172.21.15.16:0/2801872171 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f30c004dbe0 con 0x7f30c00b2880 2023-05-22T16:03:21.220 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.218+0000 7f30bcff9640 1 -- 172.21.15.16:0/2801872171 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v165) v1 ==== 92+0+10 (secure 0 0 0) 0x7f30b80ef4e0 con 0x7f30c00b2880 2023-05-22T16:03:21.221 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 8 2023-05-22T16:03:21.222 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.218+0000 7f30c6611640 1 -- 172.21.15.16:0/2801872171 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f30a00654a0 msgr2=0x7f30a0067950 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-05-22T16:03:21.223 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.218+0000 7f30c6611640 1 --2- 172.21.15.16:0/2801872171 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f30a00654a0 0x7f30a0067950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:21.223 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.218+0000 7f30c6611640 1 -- 172.21.15.16:0/2801872171 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f30c00b2880 msgr2=0x7f30c01522b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:21.223 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.218+0000 7f30c6611640 1 --2- 172.21.15.16:0/2801872171 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f30c00b2880 0x7f30c01522b0 secure :-1 s=READY pgs=2144 cs=0 l=1 rev1=1 crypto rx=0x7f30b80af040 tx=0x7f30b80196b0 comp rx=0 tx=0).stop 2023-05-22T16:03:21.223 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.218+0000 7f30c6611640 1 -- 172.21.15.16:0/2801872171 shutdown_connections 2023-05-22T16:03:21.223 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.218+0000 7f30c6611640 1 --2- 172.21.15.16:0/2801872171 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f30c00cc520 0x7f30c0155320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:21.223 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.218+0000 7f30c6611640 1 --2- 172.21.15.16:0/2801872171 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f30c00c67d0 0x7f30c0159500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:21.223 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.218+0000 7f30c6611640 1 --2- 172.21.15.16:0/2801872171 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f30a00654a0 0x7f30a0067950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:21.223 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.218+0000 7f30c6611640 1 --2- 172.21.15.16:0/2801872171 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f30c00b2880 0x7f30c01522b0 unknown :-1 s=CLOSED pgs=2144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:21.224 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.218+0000 7f30c6611640 1 -- 172.21.15.16:0/2801872171 >> 172.21.15.16:0/2801872171 conn(0x7f30c006f5a0 msgr2=0x7f30c00675a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:03:21.224 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.218+0000 7f30c6611640 1 -- 172.21.15.16:0/2801872171 shutdown_connections 2023-05-22T16:03:21.224 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.218+0000 7f30c6611640 1 -- 172.21.15.16:0/2801872171 wait complete. 2023-05-22T16:03:21.240 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-05-22T16:03:21.287 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T16:03:21.281+0000 7f46151e9180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-05-22T16:03:21.301 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T16:03:21.298+0000 7fd0922ba180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-05-22T16:03:21.340 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T16:03:21.333+0000 7f46151e9180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-05-22T16:03:21.356 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T16:03:21.350+0000 7fd0922ba180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-05-22T16:03:21.362 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.358+0000 7ff0ae09b640 1 Processor -- start 2023-05-22T16:03:21.362 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.358+0000 7ff0ae09b640 1 -- start start 2023-05-22T16:03:21.363 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.358+0000 7ff0ae09b640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7ff0a8071640 0x7ff0a8160090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:21.363 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.358+0000 7ff0ae09b640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff0a81605d0 0x7ff0a815c270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:21.363 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.358+0000 7ff0ae09b640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0a8069ae0 con 0x7ff0a80ce480 2023-05-22T16:03:21.363 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.358+0000 7ff0ae09b640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0a815f990 con 0x7ff0a81605d0 2023-05-22T16:03:21.364 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.358+0000 7ff0ae09b640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0a8071ac0 con 0x7ff0a8071640 2023-05-22T16:03:21.364 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.358+0000 7ff0a7fff640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff0a81605d0 0x7ff0a815c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:21.364 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.358+0000 7ff0a7fff640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff0a81605d0 0x7ff0a815c270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55058/0 (socket says 172.21.15.16:55058) 2023-05-22T16:03:21.364 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.358+0000 7ff0a7fff640 1 -- 172.21.15.16:0/1852614834 learned_addr learned my addr 172.21.15.16:0/1852614834 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:03:21.364 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.358+0000 7ff0a6ffd640 1 --2- 172.21.15.16:0/1852614834 >> v2:172.21.15.62:3300/0 conn(0x7ff0a8071640 0x7ff0a8160090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:21.364 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.358+0000 7ff0a7fff640 1 -- 172.21.15.16:0/1852614834 >> v2:172.21.15.62:3300/0 conn(0x7ff0a8071640 msgr2=0x7ff0a8160090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:03:21.364 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.358+0000 7ff0a7fff640 1 --2- 172.21.15.16:0/1852614834 >> v2:172.21.15.62:3300/0 conn(0x7ff0a8071640 0x7ff0a8160090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:21.365 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.358+0000 7ff0a7fff640 1 -- 172.21.15.16:0/1852614834 >> v1:172.21.15.16:6789/0 conn(0x7ff0a80ce480 legacy=0x7ff0a80ce850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:21.365 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.358+0000 7ff0a7fff640 1 -- 172.21.15.16:0/1852614834 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0a8069dc0 con 0x7ff0a81605d0 2023-05-22T16:03:21.365 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.358+0000 7ff0a6ffd640 1 --2- 172.21.15.16:0/1852614834 >> v2:172.21.15.62:3300/0 conn(0x7ff0a8071640 0x7ff0a8160090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T16:03:21.366 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.362+0000 7ff0a7fff640 1 --2- 172.21.15.16:0/1852614834 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff0a81605d0 0x7ff0a815c270 secure :-1 s=READY pgs=2146 cs=0 l=1 rev1=1 crypto rx=0x7ff0a80b1d50 tx=0x7ff0a001c9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=921d70f04d501871 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:03:21.366 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.362+0000 7ff0a67fc640 1 -- 172.21.15.16:0/1852614834 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0a00a5860 con 0x7ff0a81605d0 2023-05-22T16:03:21.366 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.362+0000 7ff0a67fc640 1 -- 172.21.15.16:0/1852614834 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0a00a59c0 con 0x7ff0a81605d0 2023-05-22T16:03:21.366 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.362+0000 7ff0a67fc640 1 -- 172.21.15.16:0/1852614834 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0a00a5d50 con 0x7ff0a81605d0 2023-05-22T16:03:21.366 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.362+0000 7ff0ae09b640 1 -- 172.21.15.16:0/1852614834 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff0a81605d0 msgr2=0x7ff0a815c270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:21.366 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.362+0000 7ff0ae09b640 1 --2- 172.21.15.16:0/1852614834 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff0a81605d0 0x7ff0a815c270 secure :-1 s=READY pgs=2146 cs=0 l=1 rev1=1 crypto rx=0x7ff0a80b1d50 tx=0x7ff0a001c9a0 comp rx=0 tx=0).stop 2023-05-22T16:03:21.367 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.362+0000 7ff0ae09b640 1 -- 172.21.15.16:0/1852614834 shutdown_connections 2023-05-22T16:03:21.367 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.362+0000 7ff0ae09b640 1 --2- 172.21.15.16:0/1852614834 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff0a81605d0 0x7ff0a815c270 unknown :-1 s=CLOSED pgs=2146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:21.367 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.362+0000 7ff0ae09b640 1 --2- 172.21.15.16:0/1852614834 >> v2:172.21.15.62:3300/0 conn(0x7ff0a8071640 0x7ff0a8160090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:21.367 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.362+0000 7ff0ae09b640 1 -- 172.21.15.16:0/1852614834 >> 172.21.15.16:0/1852614834 conn(0x7ff0a80cc870 msgr2=0x7ff0a80cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:03:21.367 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.362+0000 7ff0ae09b640 1 -- 172.21.15.16:0/1852614834 shutdown_connections 2023-05-22T16:03:21.367 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.362+0000 7ff0ae09b640 1 -- 172.21.15.16:0/1852614834 wait complete. 2023-05-22T16:03:21.369 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.362+0000 7ff0ae09b640 1 Processor -- start 2023-05-22T16:03:21.369 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.362+0000 7ff0ae09b640 1 -- start start 2023-05-22T16:03:21.369 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.362+0000 7ff0ae09b640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff0a8071640 0x7ff0a80aedd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:21.369 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.362+0000 7ff0ae09b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff0a80ce480 0x7ff0a815db20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:21.370 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.362+0000 7ff0ae09b640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff0a81605d0 0x7ff0a8153460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:21.370 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.362+0000 7ff0ae09b640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0a80af6d0 con 0x7ff0a80ce480 2023-05-22T16:03:21.370 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.362+0000 7ff0ae09b640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0a80af9b0 con 0x7ff0a81605d0 2023-05-22T16:03:21.370 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.362+0000 7ff0ae09b640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0a80ac500 con 0x7ff0a8071640 2023-05-22T16:03:21.370 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.362+0000 7ff0a6ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff0a80ce480 0x7ff0a815db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:21.370 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.362+0000 7ff0a6ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff0a80ce480 0x7ff0a815db20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55062/0 (socket says 172.21.15.16:55062) 2023-05-22T16:03:21.370 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.362+0000 7ff0a6ffd640 1 -- 172.21.15.16:0/2178055377 learned_addr learned my addr 172.21.15.16:0/2178055377 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:03:21.371 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.366+0000 7ff0a77fe640 1 --2- 172.21.15.16:0/2178055377 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff0a8071640 0x7ff0a80aedd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:21.371 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.366+0000 7ff0a7fff640 1 --2- 172.21.15.16:0/2178055377 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff0a81605d0 0x7ff0a8153460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:21.371 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.366+0000 7ff0a6ffd640 1 -- 172.21.15.16:0/2178055377 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff0a8071640 msgr2=0x7ff0a80aedd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:21.371 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.366+0000 7ff0a6ffd640 1 --2- 172.21.15.16:0/2178055377 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff0a8071640 0x7ff0a80aedd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:21.371 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.366+0000 7ff0a6ffd640 1 -- 172.21.15.16:0/2178055377 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff0a81605d0 msgr2=0x7ff0a8153460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:21.371 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.366+0000 7ff0a6ffd640 1 --2- 172.21.15.16:0/2178055377 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff0a81605d0 0x7ff0a8153460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:21.371 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.366+0000 7ff0a6ffd640 1 -- 172.21.15.16:0/2178055377 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0a0009b30 con 0x7ff0a80ce480 2023-05-22T16:03:21.371 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.366+0000 7ff0a6ffd640 1 --2- 172.21.15.16:0/2178055377 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff0a80ce480 0x7ff0a815db20 secure :-1 s=READY pgs=2147 cs=0 l=1 rev1=1 crypto rx=0x7ff09c00c7f0 tx=0x7ff09c00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:03:21.372 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.366+0000 7ff08ffff640 1 -- 172.21.15.16:0/2178055377 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff09c007820 con 0x7ff0a80ce480 2023-05-22T16:03:21.372 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.366+0000 7ff08ffff640 1 -- 172.21.15.16:0/2178055377 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff09c007980 con 0x7ff0a80ce480 2023-05-22T16:03:21.372 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.366+0000 7ff0ae09b640 1 -- 172.21.15.16:0/2178055377 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0a80698b0 con 0x7ff0a80ce480 2023-05-22T16:03:21.372 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.366+0000 7ff08ffff640 1 -- 172.21.15.16:0/2178055377 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff09c007d30 con 0x7ff0a80ce480 2023-05-22T16:03:21.372 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.366+0000 7ff0ae09b640 1 -- 172.21.15.16:0/2178055377 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0a8069720 con 0x7ff0a80ce480 2023-05-22T16:03:21.373 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.366+0000 7ff08dffb640 1 -- 172.21.15.16:0/2178055377 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff080005260 con 0x7ff0a80ce480 2023-05-22T16:03:21.373 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.366+0000 7ff08ffff640 1 -- 172.21.15.16:0/2178055377 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 79) v1 ==== 84344+0+0 (secure 0 0 0) 0x7ff09c0113f0 con 0x7ff0a80ce480 2023-05-22T16:03:21.373 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.366+0000 7ff08ffff640 1 --2- 172.21.15.16:0/2178055377 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7ff0940654a0 0x7ff094067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:21.373 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.366+0000 7ff0a77fe640 1 -- 172.21.15.16:0/2178055377 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7ff0940654a0 msgr2=0x7ff094067950 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/65890 2023-05-22T16:03:21.374 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.366+0000 7ff08ffff640 1 -- 172.21.15.16:0/2178055377 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(165..165 src has 1..165) v4 ==== 7041+0+0 (secure 0 0 0) 0x7ff09c083360 con 0x7ff0a80ce480 2023-05-22T16:03:21.374 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.366+0000 7ff0a77fe640 1 --2- 172.21.15.16:0/2178055377 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7ff0940654a0 0x7ff094067950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-05-22T16:03:21.375 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.370+0000 7ff08ffff640 1 -- 172.21.15.16:0/2178055377 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7ff09c050860 con 0x7ff0a80ce480 2023-05-22T16:03:21.391 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T16:03:21.381+0000 7f46151e9180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-05-22T16:03:21.427 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T16:03:21.422+0000 7fd0922ba180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-05-22T16:03:21.468 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T16:03:21.461+0000 7f46151e9180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-05-22T16:03:21.506 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T16:03:21.502+0000 7fd0922ba180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-05-22T16:03:21.519 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T16:03:21.513+0000 7f46151e9180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-05-22T16:03:21.550 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.546+0000 7ff08dffb640 1 -- 172.21.15.16:0/2178055377 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7ff080005a90 con 0x7ff0a80ce480 2023-05-22T16:03:21.551 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.546+0000 7ff08ffff640 1 -- 172.21.15.16:0/2178055377 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v165) v1 ==== 104+0+11 (secure 0 0 0) 0x7ff09c053e80 con 0x7ff0a80ce480 2023-05-22T16:03:21.552 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2023-05-22T16:03:21.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.550+0000 7ff08dffb640 1 -- 172.21.15.16:0/2178055377 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7ff0940654a0 msgr2=0x7ff094067950 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-05-22T16:03:21.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.550+0000 7ff08dffb640 1 --2- 172.21.15.16:0/2178055377 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7ff0940654a0 0x7ff094067950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:21.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.550+0000 7ff08dffb640 1 -- 172.21.15.16:0/2178055377 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff0a80ce480 msgr2=0x7ff0a815db20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:21.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.550+0000 7ff08dffb640 1 --2- 172.21.15.16:0/2178055377 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff0a80ce480 0x7ff0a815db20 secure :-1 s=READY pgs=2147 cs=0 l=1 rev1=1 crypto rx=0x7ff09c00c7f0 tx=0x7ff09c00ccb0 comp rx=0 tx=0).stop 2023-05-22T16:03:21.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.550+0000 7ff08dffb640 1 -- 172.21.15.16:0/2178055377 shutdown_connections 2023-05-22T16:03:21.554 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.550+0000 7ff08dffb640 1 --2- 172.21.15.16:0/2178055377 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff0a81605d0 0x7ff0a8153460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:21.554 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.550+0000 7ff08dffb640 1 --2- 172.21.15.16:0/2178055377 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7ff0940654a0 0x7ff094067950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:21.554 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.550+0000 7ff08dffb640 1 --2- 172.21.15.16:0/2178055377 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff0a80ce480 0x7ff0a815db20 unknown :-1 s=CLOSED pgs=2147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:21.554 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.550+0000 7ff08dffb640 1 --2- 172.21.15.16:0/2178055377 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff0a8071640 0x7ff0a80aedd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:21.554 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.550+0000 7ff08dffb640 1 -- 172.21.15.16:0/2178055377 >> 172.21.15.16:0/2178055377 conn(0x7ff0a80cc870 msgr2=0x7ff0a806db00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:03:21.554 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.550+0000 7ff08dffb640 1 -- 172.21.15.16:0/2178055377 shutdown_connections 2023-05-22T16:03:21.554 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.550+0000 7ff08dffb640 1 -- 172.21.15.16:0/2178055377 wait complete. 2023-05-22T16:03:21.558 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T16:03:21.554+0000 7fd0922ba180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-05-22T16:03:21.569 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-05-22T16:03:21.595 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T16:03:21.589+0000 7f46151e9180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-05-22T16:03:21.639 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T16:03:21.634+0000 7fd0922ba180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-05-22T16:03:21.648 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T16:03:21.641+0000 7f46151e9180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-05-22T16:03:21.686 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.682+0000 7f0766e9f640 1 Processor -- start 2023-05-22T16:03:21.687 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.682+0000 7f0766e9f640 1 -- start start 2023-05-22T16:03:21.688 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.682+0000 7f0766e9f640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f076015cfa0 0x7f076006db00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:21.688 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.682+0000 7f0766e9f640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f07600cf870 0x7f07600b1fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:21.688 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.682+0000 7f0766e9f640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f076006e740 0x7f076006ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:21.689 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.682+0000 7f0766e9f640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07600cb9f0 con 0x7f076006e740 2023-05-22T16:03:21.689 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.682+0000 7f0766e9f640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f076006f100 con 0x7f07600cf870 2023-05-22T16:03:21.689 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.682+0000 7f0766e9f640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f076006e040 con 0x7f076015cfa0 2023-05-22T16:03:21.689 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.682+0000 7f076543c640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f076006e740 0x7f076006ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:21.689 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.682+0000 7f076543c640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f076006e740 0x7f076006ebc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55064/0 (socket says 172.21.15.16:55064) 2023-05-22T16:03:21.689 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.682+0000 7f076543c640 1 -- 172.21.15.16:0/873515478 learned_addr learned my addr 172.21.15.16:0/873515478 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:03:21.689 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.682+0000 7f075ffff640 1 --2- 172.21.15.16:0/873515478 >> v2:172.21.15.62:3300/0 conn(0x7f07600cf870 0x7f07600b1fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:21.690 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.682+0000 7f0764c3b640 1 --2- 172.21.15.16:0/873515478 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f076015cfa0 0x7f076006db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:21.690 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.686+0000 7f076543c640 1 -- 172.21.15.16:0/873515478 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f076015cfa0 msgr2=0x7f076006db00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:03:21.690 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.686+0000 7f076543c640 1 --2- 172.21.15.16:0/873515478 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f076015cfa0 0x7f076006db00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:21.690 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.686+0000 7f076543c640 1 -- 172.21.15.16:0/873515478 >> v2:172.21.15.62:3300/0 conn(0x7f07600cf870 msgr2=0x7f07600b1fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:03:21.690 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.686+0000 7f076543c640 1 --2- 172.21.15.16:0/873515478 >> v2:172.21.15.62:3300/0 conn(0x7f07600cf870 0x7f07600b1fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:21.690 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.686+0000 7f076543c640 1 -- 172.21.15.16:0/873515478 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07600b24f0 con 0x7f076006e740 2023-05-22T16:03:21.691 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.686+0000 7f076543c640 1 --2- 172.21.15.16:0/873515478 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f076006e740 0x7f076006ebc0 secure :-1 s=READY pgs=2148 cs=0 l=1 rev1=1 crypto rx=0x7f076006e1b0 tx=0x7f075801c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6bcbdf41fc2f05e5 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:03:21.691 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.686+0000 7f075f7fe640 1 -- 172.21.15.16:0/873515478 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07580a56b0 con 0x7f076006e740 2023-05-22T16:03:21.691 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.686+0000 7f075f7fe640 1 -- 172.21.15.16:0/873515478 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07580a5810 con 0x7f076006e740 2023-05-22T16:03:21.691 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.686+0000 7f075f7fe640 1 -- 172.21.15.16:0/873515478 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07580a5ba0 con 0x7f076006e740 2023-05-22T16:03:21.692 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.686+0000 7f0766e9f640 1 -- 172.21.15.16:0/873515478 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f076006e740 msgr2=0x7f076006ebc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:21.692 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.686+0000 7f0766e9f640 1 --2- 172.21.15.16:0/873515478 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f076006e740 0x7f076006ebc0 secure :-1 s=READY pgs=2148 cs=0 l=1 rev1=1 crypto rx=0x7f076006e1b0 tx=0x7f075801c940 comp rx=0 tx=0).stop 2023-05-22T16:03:21.692 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.686+0000 7f0766e9f640 1 -- 172.21.15.16:0/873515478 shutdown_connections 2023-05-22T16:03:21.693 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.686+0000 7f0766e9f640 1 --2- 172.21.15.16:0/873515478 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f076006e740 0x7f076006ebc0 unknown :-1 s=CLOSED pgs=2148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:21.693 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.686+0000 7f0766e9f640 1 --2- 172.21.15.16:0/873515478 >> v2:172.21.15.62:3300/0 conn(0x7f07600cf870 0x7f07600b1fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:21.693 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.686+0000 7f0766e9f640 1 --2- 172.21.15.16:0/873515478 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f076015cfa0 0x7f076006db00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:21.693 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.686+0000 7f0766e9f640 1 -- 172.21.15.16:0/873515478 >> 172.21.15.16:0/873515478 conn(0x7f07600cc870 msgr2=0x7f07600cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:03:21.693 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.686+0000 7f0766e9f640 1 -- 172.21.15.16:0/873515478 shutdown_connections 2023-05-22T16:03:21.693 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.686+0000 7f0766e9f640 1 -- 172.21.15.16:0/873515478 wait complete. 2023-05-22T16:03:21.694 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T16:03:21.686+0000 7fd0922ba180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-05-22T16:03:21.695 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.690+0000 7f0766e9f640 1 Processor -- start 2023-05-22T16:03:21.695 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.690+0000 7f0766e9f640 1 -- start start 2023-05-22T16:03:21.695 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.690+0000 7f0766e9f640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f076006e740 0x7f07601431e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:21.695 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.690+0000 7f0766e9f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f07600cf870 0x7f0760146450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:21.696 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.690+0000 7f0766e9f640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f076015cfa0 0x7f0760162ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:21.696 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.690+0000 7f0766e9f640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0760143720 con 0x7f07600cf870 2023-05-22T16:03:21.696 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.690+0000 7f0766e9f640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0760158350 con 0x7f076015cfa0 2023-05-22T16:03:21.696 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.690+0000 7f0766e9f640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0760146990 con 0x7f076006e740 2023-05-22T16:03:21.696 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.690+0000 7f075ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f07600cf870 0x7f0760146450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:21.696 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.690+0000 7f075ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f07600cf870 0x7f0760146450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55070/0 (socket says 172.21.15.16:55070) 2023-05-22T16:03:21.696 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.690+0000 7f075ffff640 1 -- 172.21.15.16:0/3094380354 learned_addr learned my addr 172.21.15.16:0/3094380354 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:03:21.697 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.690+0000 7f076543c640 1 --2- 172.21.15.16:0/3094380354 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f076015cfa0 0x7f0760162ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:21.697 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.690+0000 7f0764c3b640 1 --2- 172.21.15.16:0/3094380354 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f076006e740 0x7f07601431e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:21.697 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.690+0000 7f075ffff640 1 -- 172.21.15.16:0/3094380354 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f076006e740 msgr2=0x7f07601431e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:21.697 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.690+0000 7f075ffff640 1 --2- 172.21.15.16:0/3094380354 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f076006e740 0x7f07601431e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:21.697 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.690+0000 7f075ffff640 1 -- 172.21.15.16:0/3094380354 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f076015cfa0 msgr2=0x7f0760162ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:21.697 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.690+0000 7f075ffff640 1 --2- 172.21.15.16:0/3094380354 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f076015cfa0 0x7f0760162ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:21.697 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.690+0000 7f075ffff640 1 -- 172.21.15.16:0/3094380354 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0758009b30 con 0x7f07600cf870 2023-05-22T16:03:21.698 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.690+0000 7f075ffff640 1 --2- 172.21.15.16:0/3094380354 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f07600cf870 0x7f0760146450 secure :-1 s=READY pgs=2149 cs=0 l=1 rev1=1 crypto rx=0x7f074c00d800 tx=0x7f074c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:03:21.698 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.690+0000 7f075d7fa640 1 -- 172.21.15.16:0/3094380354 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f074c0096a0 con 0x7f07600cf870 2023-05-22T16:03:21.698 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.690+0000 7f075d7fa640 1 -- 172.21.15.16:0/3094380354 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f074c009800 con 0x7f07600cf870 2023-05-22T16:03:21.698 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.690+0000 7f0766e9f640 1 -- 172.21.15.16:0/3094380354 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07600cb610 con 0x7f07600cf870 2023-05-22T16:03:21.698 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.690+0000 7f075d7fa640 1 -- 172.21.15.16:0/3094380354 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f074c009b70 con 0x7f07600cf870 2023-05-22T16:03:21.698 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.690+0000 7f0766e9f640 1 -- 172.21.15.16:0/3094380354 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07600cb330 con 0x7f07600cf870 2023-05-22T16:03:21.699 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.690+0000 7f0766e9f640 1 -- 172.21.15.16:0/3094380354 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f076015d370 con 0x7f07600cf870 2023-05-22T16:03:21.699 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.694+0000 7f075d7fa640 1 -- 172.21.15.16:0/3094380354 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 79) v1 ==== 84344+0+0 (secure 0 0 0) 0x7f074c009cd0 con 0x7f07600cf870 2023-05-22T16:03:21.699 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.694+0000 7f075d7fa640 1 --2- 172.21.15.16:0/3094380354 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f07440654a0 0x7f0744067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:21.699 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.694+0000 7f0764c3b640 1 -- 172.21.15.16:0/3094380354 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f07440654a0 msgr2=0x7f0744067950 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/65890 2023-05-22T16:03:21.700 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.694+0000 7f075d7fa640 1 -- 172.21.15.16:0/3094380354 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(165..165 src has 1..165) v4 ==== 7041+0+0 (secure 0 0 0) 0x7f074c083680 con 0x7f07600cf870 2023-05-22T16:03:21.700 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.694+0000 7f0764c3b640 1 --2- 172.21.15.16:0/3094380354 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f07440654a0 0x7f0744067950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-05-22T16:03:21.701 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.698+0000 7f075d7fa640 1 -- 172.21.15.16:0/3094380354 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f074c050ce0 con 0x7f07600cf870 2023-05-22T16:03:21.782 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T16:03:21.773+0000 7f46151e9180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-05-22T16:03:21.828 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T16:03:21.822+0000 7fd0922ba180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-05-22T16:03:21.835 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T16:03:21.829+0000 7f46151e9180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-05-22T16:03:21.857 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.854+0000 7f0766e9f640 1 -- 172.21.15.16:0/3094380354 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f076006a560 con 0x7f07600cf870 2023-05-22T16:03:21.858 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.854+0000 7f075d7fa640 1 -- 172.21.15.16:0/3094380354 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v165) v1 ==== 100+0+11 (secure 0 0 0) 0x7f074c054300 con 0x7f07600cf870 2023-05-22T16:03:21.858 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 64 2023-05-22T16:03:21.859 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.854+0000 7f0766e9f640 1 -- 172.21.15.16:0/3094380354 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f07440654a0 msgr2=0x7f0744067950 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-05-22T16:03:21.860 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.854+0000 7f0766e9f640 1 --2- 172.21.15.16:0/3094380354 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f07440654a0 0x7f0744067950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:21.860 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.854+0000 7f0766e9f640 1 -- 172.21.15.16:0/3094380354 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f07600cf870 msgr2=0x7f0760146450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:21.860 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.854+0000 7f0766e9f640 1 --2- 172.21.15.16:0/3094380354 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f07600cf870 0x7f0760146450 secure :-1 s=READY pgs=2149 cs=0 l=1 rev1=1 crypto rx=0x7f074c00d800 tx=0x7f074c00dcc0 comp rx=0 tx=0).stop 2023-05-22T16:03:21.860 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.854+0000 7f0766e9f640 1 -- 172.21.15.16:0/3094380354 shutdown_connections 2023-05-22T16:03:21.860 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.854+0000 7f0766e9f640 1 --2- 172.21.15.16:0/3094380354 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f076015cfa0 0x7f0760162ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:21.860 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.854+0000 7f0766e9f640 1 --2- 172.21.15.16:0/3094380354 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f07440654a0 0x7f0744067950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:21.861 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.854+0000 7f0766e9f640 1 --2- 172.21.15.16:0/3094380354 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f07600cf870 0x7f0760146450 unknown :-1 s=CLOSED pgs=2149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:21.861 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.854+0000 7f0766e9f640 1 --2- 172.21.15.16:0/3094380354 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f076006e740 0x7f07601431e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:21.861 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.854+0000 7f0766e9f640 1 -- 172.21.15.16:0/3094380354 >> 172.21.15.16:0/3094380354 conn(0x7f07600cc870 msgr2=0x7f0760163f40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:03:21.861 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.854+0000 7f0766e9f640 1 -- 172.21.15.16:0/3094380354 shutdown_connections 2023-05-22T16:03:21.861 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.854+0000 7f0766e9f640 1 -- 172.21.15.16:0/3094380354 wait complete. 2023-05-22T16:03:21.876 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-05-22T16:03:21.882 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T16:03:21.878+0000 7fd0922ba180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-05-22T16:03:21.986 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.982+0000 7f65b6418640 1 Processor -- start 2023-05-22T16:03:21.986 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.982+0000 7f65b6418640 1 -- start start 2023-05-22T16:03:21.987 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.982+0000 7f65b6418640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f65b00cf870 0x7f65b006db00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:21.987 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.982+0000 7f65b6418640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f65b006e740 0x7f65b006ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:21.987 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.982+0000 7f65b6418640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65b006e040 con 0x7f65b015cfa0 2023-05-22T16:03:21.988 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.982+0000 7f65b6418640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65b00cb9f0 con 0x7f65b00cf870 2023-05-22T16:03:21.988 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.982+0000 7f65b6418640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65b006f100 con 0x7f65b006e740 2023-05-22T16:03:21.988 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.982+0000 7f65af7fe640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f65b00cf870 0x7f65b006db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:21.988 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.982+0000 7f65af7fe640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f65b00cf870 0x7f65b006db00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55074/0 (socket says 172.21.15.16:55074) 2023-05-22T16:03:21.988 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.982+0000 7f65af7fe640 1 -- 172.21.15.16:0/591171181 learned_addr learned my addr 172.21.15.16:0/591171181 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:03:21.988 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.982+0000 7f65b49b5640 1 --2- 172.21.15.16:0/591171181 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f65b006e740 0x7f65b006ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:21.989 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.982+0000 7f65af7fe640 1 -- 172.21.15.16:0/591171181 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f65b006e740 msgr2=0x7f65b006ebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:03:21.989 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.982+0000 7f65af7fe640 1 --2- 172.21.15.16:0/591171181 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f65b006e740 0x7f65b006ebc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:21.989 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.982+0000 7f65af7fe640 1 -- 172.21.15.16:0/591171181 >> v1:172.21.15.16:6789/0 conn(0x7f65b015cfa0 legacy=0x7f65b00b1fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:21.990 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.982+0000 7f65af7fe640 1 -- 172.21.15.16:0/591171181 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65b006e490 con 0x7f65b00cf870 2023-05-22T16:03:21.990 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.982+0000 7f65b49b5640 1 --2- 172.21.15.16:0/591171181 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f65b006e740 0x7f65b006ebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T16:03:21.991 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.986+0000 7f65af7fe640 1 --2- 172.21.15.16:0/591171181 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f65b00cf870 0x7f65b006db00 secure :-1 s=READY pgs=2151 cs=0 l=1 rev1=1 crypto rx=0x7f65a4009870 tx=0x7f65a401c510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d9c1714ceefc147 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:03:21.991 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.986+0000 7f65aeffd640 1 -- 172.21.15.16:0/591171181 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65a40aa670 con 0x7f65b00cf870 2023-05-22T16:03:21.991 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.986+0000 7f65aeffd640 1 -- 172.21.15.16:0/591171181 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65a40aa7d0 con 0x7f65b00cf870 2023-05-22T16:03:21.991 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.986+0000 7f65aeffd640 1 -- 172.21.15.16:0/591171181 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65a40aab60 con 0x7f65b00cf870 2023-05-22T16:03:21.992 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.986+0000 7f65b6418640 1 -- 172.21.15.16:0/591171181 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f65b00cf870 msgr2=0x7f65b006db00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:21.992 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.986+0000 7f65b6418640 1 --2- 172.21.15.16:0/591171181 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f65b00cf870 0x7f65b006db00 secure :-1 s=READY pgs=2151 cs=0 l=1 rev1=1 crypto rx=0x7f65a4009870 tx=0x7f65a401c510 comp rx=0 tx=0).stop 2023-05-22T16:03:21.992 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.986+0000 7f65b6418640 1 -- 172.21.15.16:0/591171181 shutdown_connections 2023-05-22T16:03:21.992 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.986+0000 7f65b6418640 1 --2- 172.21.15.16:0/591171181 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f65b006e740 0x7f65b006ebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:21.992 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.986+0000 7f65b6418640 1 --2- 172.21.15.16:0/591171181 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f65b00cf870 0x7f65b006db00 unknown :-1 s=CLOSED pgs=2151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:21.992 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.986+0000 7f65b6418640 1 -- 172.21.15.16:0/591171181 >> 172.21.15.16:0/591171181 conn(0x7f65b00cc870 msgr2=0x7f65b00cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:03:21.992 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.986+0000 7f65b6418640 1 -- 172.21.15.16:0/591171181 shutdown_connections 2023-05-22T16:03:21.993 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.986+0000 7f65b6418640 1 -- 172.21.15.16:0/591171181 wait complete. 2023-05-22T16:03:21.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.986+0000 7f65b6418640 1 Processor -- start 2023-05-22T16:03:21.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.986+0000 7f65b6418640 1 -- start start 2023-05-22T16:03:21.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.990+0000 7f65b6418640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f65b006e740 0x7f65b01519c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:21.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.990+0000 7f65b6418640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f65b00cf870 0x7f65b0155c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:21.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.990+0000 7f65b6418640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f65b015cfa0 0x7f65b01561d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:21.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.990+0000 7f65b6418640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65b0155590 con 0x7f65b015cfa0 2023-05-22T16:03:21.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.990+0000 7f65b6418640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65b00c9110 con 0x7f65b006e740 2023-05-22T16:03:21.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.990+0000 7f65b6418640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65b00c8070 con 0x7f65b00cf870 2023-05-22T16:03:21.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.990+0000 7f65b49b5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f65b015cfa0 0x7f65b01561d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:21.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.990+0000 7f65b49b5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f65b015cfa0 0x7f65b01561d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55084/0 (socket says 172.21.15.16:55084) 2023-05-22T16:03:21.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.990+0000 7f65b49b5640 1 -- 172.21.15.16:0/388077493 learned_addr learned my addr 172.21.15.16:0/388077493 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:03:21.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.990+0000 7f65af7fe640 1 --2- 172.21.15.16:0/388077493 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f65b00cf870 0x7f65b0155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:21.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.990+0000 7f65affff640 1 --2- 172.21.15.16:0/388077493 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f65b006e740 0x7f65b01519c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:21.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.990+0000 7f65b49b5640 1 -- 172.21.15.16:0/388077493 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f65b00cf870 msgr2=0x7f65b0155c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:21.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.990+0000 7f65b49b5640 1 --2- 172.21.15.16:0/388077493 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f65b00cf870 0x7f65b0155c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:21.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.990+0000 7f65b49b5640 1 -- 172.21.15.16:0/388077493 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f65b006e740 msgr2=0x7f65b01519c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:21.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.990+0000 7f65b49b5640 1 --2- 172.21.15.16:0/388077493 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f65b006e740 0x7f65b01519c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:21.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.990+0000 7f65b49b5640 1 -- 172.21.15.16:0/388077493 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65a40094d0 con 0x7f65b015cfa0 2023-05-22T16:03:21.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.990+0000 7f65b49b5640 1 --2- 172.21.15.16:0/388077493 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f65b015cfa0 0x7f65b01561d0 secure :-1 s=READY pgs=2152 cs=0 l=1 rev1=1 crypto rx=0x7f65a8007f20 tx=0x7f65a800d560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:03:21.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.990+0000 7f65acff9640 1 -- 172.21.15.16:0/388077493 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65a800dee0 con 0x7f65b015cfa0 2023-05-22T16:03:21.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.990+0000 7f65acff9640 1 -- 172.21.15.16:0/388077493 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65a800f040 con 0x7f65b015cfa0 2023-05-22T16:03:21.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.990+0000 7f65b6418640 1 -- 172.21.15.16:0/388077493 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65b00cb610 con 0x7f65b015cfa0 2023-05-22T16:03:21.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.990+0000 7f65acff9640 1 -- 172.21.15.16:0/388077493 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65a8003de0 con 0x7f65b015cfa0 2023-05-22T16:03:21.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.990+0000 7f65b6418640 1 -- 172.21.15.16:0/388077493 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65b00cb330 con 0x7f65b015cfa0 2023-05-22T16:03:21.998 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.990+0000 7f65b6418640 1 -- 172.21.15.16:0/388077493 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65b015d370 con 0x7f65b015cfa0 2023-05-22T16:03:21.998 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.990+0000 7f65acff9640 1 -- 172.21.15.16:0/388077493 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 79) v1 ==== 84344+0+0 (secure 0 0 0) 0x7f65a8004030 con 0x7f65b015cfa0 2023-05-22T16:03:21.998 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.990+0000 7f65acff9640 1 --2- 172.21.15.16:0/388077493 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f659c0654a0 0x7f659c067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:21.998 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.990+0000 7f65affff640 1 -- 172.21.15.16:0/388077493 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f659c0654a0 msgr2=0x7f659c067950 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/65890 2023-05-22T16:03:21.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.990+0000 7f65acff9640 1 -- 172.21.15.16:0/388077493 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(165..165 src has 1..165) v4 ==== 7041+0+0 (secure 0 0 0) 0x7f65a80831f0 con 0x7f65b015cfa0 2023-05-22T16:03:21.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.990+0000 7f65affff640 1 --2- 172.21.15.16:0/388077493 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f659c0654a0 0x7f659c067950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-05-22T16:03:21.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:21.994+0000 7f65acff9640 1 -- 172.21.15.16:0/388077493 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f65a8050920 con 0x7f65b015cfa0 2023-05-22T16:03:22.059 INFO:tasks.ceph.mgr.x.smithi062.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2023-05-22T16:03:22.059 INFO:tasks.ceph.mgr.x.smithi062.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2023-05-22T16:03:22.060 INFO:tasks.ceph.mgr.x.smithi062.stderr: from numpy import show_config as show_numpy_config 2023-05-22T16:03:22.067 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T16:03:22.061+0000 7f46151e9180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-05-22T16:03:22.117 INFO:tasks.ceph.mgr.y.smithi016.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2023-05-22T16:03:22.117 INFO:tasks.ceph.mgr.y.smithi016.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2023-05-22T16:03:22.118 INFO:tasks.ceph.mgr.y.smithi016.stderr: from numpy import show_config as show_numpy_config 2023-05-22T16:03:22.125 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T16:03:22.122+0000 7fd0922ba180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-05-22T16:03:22.194 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T16:03:22.185+0000 7f46151e9180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-05-22T16:03:22.196 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:22.194+0000 7f65affff640 1 -- 172.21.15.16:0/388077493 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f659c0654a0 msgr2=0x7f659c067950 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/65890 2023-05-22T16:03:22.197 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:22.194+0000 7f65affff640 1 --2- 172.21.15.16:0/388077493 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f659c0654a0 0x7f659c067950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-05-22T16:03:22.207 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:22.202+0000 7f65b6418640 1 -- 172.21.15.16:0/388077493 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f65b004dbe0 con 0x7f65b015cfa0 2023-05-22T16:03:22.207 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:22.202+0000 7f65acff9640 1 -- 172.21.15.16:0/388077493 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 14 entries v165) v1 ==== 81+0+806 (secure 0 0 0) 0x7f65a8053f40 con 0x7f65b015cfa0 2023-05-22T16:03:22.208 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/2019135485 2023-05-23T15:58:20.479559+0000 2023-05-22T16:03:22.208 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/3082249167 2023-05-23T15:58:20.479559+0000 2023-05-22T16:03:22.208 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/4113149961 2023-05-23T15:58:20.479559+0000 2023-05-22T16:03:22.209 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/2582627335 2023-05-23T15:58:20.479559+0000 2023-05-22T16:03:22.209 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/2722168808 2023-05-23T15:47:16.536062+0000 2023-05-22T16:03:22.209 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/1232860864 2023-05-23T15:47:16.536062+0000 2023-05-22T16:03:22.209 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/3766829209 2023-05-23T15:47:16.536062+0000 2023-05-22T16:03:22.209 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/3369591631 2023-05-23T15:47:16.536062+0000 2023-05-22T16:03:22.209 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/3417178745 2023-05-23T15:58:20.479559+0000 2023-05-22T16:03:22.209 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:6833/36921 2023-05-23T15:47:16.536062+0000 2023-05-22T16:03:22.210 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/2592954608 2023-05-23T15:58:20.479559+0000 2023-05-22T16:03:22.210 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/49958236 2023-05-23T15:47:16.536062+0000 2023-05-22T16:03:22.210 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:6832/36921 2023-05-23T15:47:16.536062+0000 2023-05-22T16:03:22.210 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/4161645610 2023-05-23T15:47:16.536062+0000 2023-05-22T16:03:22.211 INFO:teuthology.orchestra.run.smithi016.stderr:listed 14 entries 2023-05-22T16:03:22.211 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:22.206+0000 7f65b6418640 1 -- 172.21.15.16:0/388077493 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f659c0654a0 msgr2=0x7f659c067950 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-05-22T16:03:22.211 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:22.206+0000 7f65b6418640 1 --2- 172.21.15.16:0/388077493 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f659c0654a0 0x7f659c067950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:22.211 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:22.206+0000 7f65b6418640 1 -- 172.21.15.16:0/388077493 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f65b015cfa0 msgr2=0x7f65b01561d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:22.211 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:22.206+0000 7f65b6418640 1 --2- 172.21.15.16:0/388077493 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f65b015cfa0 0x7f65b01561d0 secure :-1 s=READY pgs=2152 cs=0 l=1 rev1=1 crypto rx=0x7f65a8007f20 tx=0x7f65a800d560 comp rx=0 tx=0).stop 2023-05-22T16:03:22.212 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:22.206+0000 7f65b6418640 1 -- 172.21.15.16:0/388077493 shutdown_connections 2023-05-22T16:03:22.212 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:22.206+0000 7f65b6418640 1 --2- 172.21.15.16:0/388077493 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f659c0654a0 0x7f659c067950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:22.212 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:22.206+0000 7f65b6418640 1 --2- 172.21.15.16:0/388077493 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f65b015cfa0 0x7f65b01561d0 unknown :-1 s=CLOSED pgs=2152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:22.212 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:22.206+0000 7f65b6418640 1 --2- 172.21.15.16:0/388077493 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f65b00cf870 0x7f65b0155c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:22.212 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:22.206+0000 7f65b6418640 1 --2- 172.21.15.16:0/388077493 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f65b006e740 0x7f65b01519c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:22.212 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:22.206+0000 7f65b6418640 1 -- 172.21.15.16:0/388077493 >> 172.21.15.16:0/388077493 conn(0x7f65b00cc870 msgr2=0x7f65b015bb90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:03:22.212 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:22.206+0000 7f65b6418640 1 -- 172.21.15.16:0/388077493 shutdown_connections 2023-05-22T16:03:22.213 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:22.206+0000 7f65b6418640 1 -- 172.21.15.16:0/388077493 wait complete. 2023-05-22T16:03:22.225 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi016.front.sepia.ceph.com... 2023-05-22T16:03:22.226 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T16:03:22.226 DEBUG:teuthology.orchestra.run.smithi016:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2023-05-22T16:03:22.253 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T16:03:22.250+0000 7fd0922ba180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-05-22T16:03:22.261 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi016.stderr:ceph-fuse[65370]: fuse finished with error 0 and tester_r 0 2023-05-22T16:03:22.263 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi016.front.sepia.ceph.com 2023-05-22T16:03:22.263 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T16:03:22.264 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-05-22T16:03:22.307 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T16:03:22.301+0000 7f46151e9180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-05-22T16:03:22.368 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T16:03:22.362+0000 7fd0922ba180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-05-22T16:03:22.370 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T16:03:22.365+0000 7f46151e9180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-05-22T16:03:22.426 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2023-05-22T16:03:22.427 DEBUG:teuthology.orchestra.run.smithi062:> dd if=/proc/self/mounts of=/dev/stdout 2023-05-22T16:03:22.433 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16342116k,nr_inodes=4085529,mode=755,inode64 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275932k,mode=755,inode64 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev,inode64 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=13250 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275928k,nr_inodes=818982,mode=700,uid=1000,gid=1271,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275932k,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-05-22T16:03:22.434 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi062.front.sepia.ceph.com 2023-05-22T16:03:22.435 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T16:03:22.435 DEBUG:teuthology.orchestra.run.smithi062:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2023-05-22T16:03:22.436 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T16:03:22.430+0000 7fd0922ba180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-05-22T16:03:22.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-05-22T16:03:22.480 INFO:teuthology.orchestra.run.smithi062.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2023-05-22T16:03:22.481 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-05-22T16:03:22.482 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2023-05-22T16:03:22.488 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16345192k,nr_inodes=4086298,mode=755,inode64 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3276544k,mode=755,inode64 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev,inode64 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=15732 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3276540k,nr_inodes=819135,mode=700,uid=1000,gid=1271,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3276544k,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-05-22T16:03:22.489 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi016.front.sepia.ceph.com 2023-05-22T16:03:22.489 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T16:03:22.490 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2023-05-22T16:03:22.498 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T16:03:22.489+0000 7f46151e9180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-05-22T16:03:22.541 INFO:teuthology.orchestra.run.smithi016.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2023-05-22T16:03:22.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-05-22T16:03:22.542 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2023-05-22T16:03:22.543 DEBUG:teuthology.orchestra.run.smithi062:> dd if=/proc/self/mounts of=/dev/stdout 2023-05-22T16:03:22.548 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16342116k,nr_inodes=4085529,mode=755,inode64 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275932k,mode=755,inode64 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev,inode64 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=13250 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275928k,nr_inodes=818982,mode=700,uid=1000,gid=1271,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275932k,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-05-22T16:03:22.549 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi062.front.sepia.ceph.com 2023-05-22T16:03:22.549 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T16:03:22.549 DEBUG:teuthology.orchestra.run.smithi062:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2023-05-22T16:03:22.562 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T16:03:22.558+0000 7fd0922ba180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-05-22T16:03:22.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-05-22T16:03:22.596 INFO:teuthology.orchestra.run.smithi062.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2023-05-22T16:03:22.596 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-05-22T16:03:22.708 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:22.706+0000 7f7214e2d640 1 Processor -- start 2023-05-22T16:03:22.709 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:22.706+0000 7f7214e2d640 1 -- start start 2023-05-22T16:03:22.710 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:22.706+0000 7f7214e2d640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f721015cfa0 0x7f721006db00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:22.710 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:22.706+0000 7f7214e2d640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f72100cf870 0x7f72100b1fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:22.710 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:22.706+0000 7f7214e2d640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f721006e740 0x7f721006ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:22.711 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:22.706+0000 7f7214e2d640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72100cb9f0 con 0x7f721015cfa0 2023-05-22T16:03:22.711 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:22.706+0000 7f7214e2d640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f721006f100 con 0x7f72100cf870 2023-05-22T16:03:22.711 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:22.706+0000 7f7214e2d640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f721006e040 con 0x7f721006e740 2023-05-22T16:03:22.711 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:22.706+0000 7f720e59c640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f721015cfa0 0x7f721006db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:22.711 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:22.706+0000 7f720e59c640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f721015cfa0 0x7f721006db00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55098/0 (socket says 172.21.15.16:55098) 2023-05-22T16:03:22.711 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:22.706+0000 7f720e59c640 1 -- 172.21.15.16:0/3637075391 learned_addr learned my addr 172.21.15.16:0/3637075391 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:03:22.712 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:22.706+0000 7f7207fff640 1 --2- 172.21.15.16:0/3637075391 >> v2:172.21.15.62:3300/0 conn(0x7f72100cf870 0x7f72100b1fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:22.712 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:22.706+0000 7f720e59c640 1 -- 172.21.15.16:0/3637075391 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f721006e740 msgr2=0x7f721006ebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:03:22.712 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:22.706+0000 7f720ed9d640 1 --2- 172.21.15.16:0/3637075391 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f721006e740 0x7f721006ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:22.712 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:22.706+0000 7f720e59c640 1 --2- 172.21.15.16:0/3637075391 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f721006e740 0x7f721006ebc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:22.712 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:22.706+0000 7f720e59c640 1 -- 172.21.15.16:0/3637075391 >> v2:172.21.15.62:3300/0 conn(0x7f72100cf870 msgr2=0x7f72100b1fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:03:22.712 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:22.706+0000 7f720e59c640 1 --2- 172.21.15.16:0/3637075391 >> v2:172.21.15.62:3300/0 conn(0x7f72100cf870 0x7f72100b1fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:22.713 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:22.706+0000 7f720e59c640 1 -- 172.21.15.16:0/3637075391 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72100b24f0 con 0x7f721015cfa0 2023-05-22T16:03:22.713 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:22.706+0000 7f7207fff640 1 --2- 172.21.15.16:0/3637075391 >> v2:172.21.15.62:3300/0 conn(0x7f72100cf870 0x7f72100b1fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T16:03:22.713 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:22.706+0000 7f720ed9d640 1 --2- 172.21.15.16:0/3637075391 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f721006e740 0x7f721006ebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T16:03:22.714 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:22.706+0000 7f720e59c640 1 --2- 172.21.15.16:0/3637075391 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f721015cfa0 0x7f721006db00 secure :-1 s=READY pgs=2153 cs=0 l=1 rev1=1 crypto rx=0x7f72100cef60 tx=0x7f720001c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4eaaea0e332e2956 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:03:22.714 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:22.710+0000 7f720dd9b640 1 -- 172.21.15.16:0/3637075391 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72000ac410 con 0x7f721015cfa0 2023-05-22T16:03:22.714 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:22.710+0000 7f720dd9b640 1 -- 172.21.15.16:0/3637075391 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72000ac570 con 0x7f721015cfa0 2023-05-22T16:03:22.714 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:22.710+0000 7f720dd9b640 1 -- 172.21.15.16:0/3637075391 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72000ac900 con 0x7f721015cfa0 2023-05-22T16:03:22.714 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:22.710+0000 7f7214e2d640 1 -- 172.21.15.16:0/3637075391 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f721015cfa0 msgr2=0x7f721006db00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:22.714 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:22.710+0000 7f7214e2d640 1 --2- 172.21.15.16:0/3637075391 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f721015cfa0 0x7f721006db00 secure :-1 s=READY pgs=2153 cs=0 l=1 rev1=1 crypto rx=0x7f72100cef60 tx=0x7f720001c500 comp rx=0 tx=0).stop 2023-05-22T16:03:22.714 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:22.710+0000 7f7214e2d640 1 -- 172.21.15.16:0/3637075391 shutdown_connections 2023-05-22T16:03:22.715 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:22.710+0000 7f7214e2d640 1 --2- 172.21.15.16:0/3637075391 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f721006e740 0x7f721006ebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:22.715 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:22.710+0000 7f7214e2d640 1 --2- 172.21.15.16:0/3637075391 >> v2:172.21.15.62:3300/0 conn(0x7f72100cf870 0x7f72100b1fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:22.715 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:22.710+0000 7f7214e2d640 1 --2- 172.21.15.16:0/3637075391 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f721015cfa0 0x7f721006db00 unknown :-1 s=CLOSED pgs=2153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:22.715 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:22.710+0000 7f7214e2d640 1 -- 172.21.15.16:0/3637075391 >> 172.21.15.16:0/3637075391 conn(0x7f72100cc870 msgr2=0x7f72100cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:03:22.715 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:22.710+0000 7f7214e2d640 1 -- 172.21.15.16:0/3637075391 shutdown_connections 2023-05-22T16:03:22.715 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:22.710+0000 7f7214e2d640 1 -- 172.21.15.16:0/3637075391 wait complete. 2023-05-22T16:03:22.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:22.710+0000 7f7214e2d640 1 Processor -- start 2023-05-22T16:03:22.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:22.710+0000 7f7214e2d640 1 -- start start 2023-05-22T16:03:22.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:22.714+0000 7f7214e2d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f721006e740 0x7f72100c5300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:22.719 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:22.714+0000 7f7214e2d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f72100cf870 0x7f72100c7770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:22.719 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:22.714+0000 7f7214e2d640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f721015cfa0 0x7f72100c7cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:22.719 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:22.714+0000 7f7214e2d640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72100c8fb0 con 0x7f721006e740 2023-05-22T16:03:22.719 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:22.714+0000 7f7214e2d640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72100c5840 con 0x7f72100cf870 2023-05-22T16:03:22.720 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:22.714+0000 7f7214e2d640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72100c8330 con 0x7f721015cfa0 2023-05-22T16:03:22.720 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:22.714+0000 7f720e59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f721006e740 0x7f72100c5300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:22.720 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:22.714+0000 7f720e59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f721006e740 0x7f72100c5300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55108/0 (socket says 172.21.15.16:55108) 2023-05-22T16:03:22.720 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:22.714+0000 7f720e59c640 1 -- 172.21.15.16:0/2952670183 learned_addr learned my addr 172.21.15.16:0/2952670183 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:03:22.721 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:22.714+0000 7f720ed9d640 1 --2- 172.21.15.16:0/2952670183 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f721015cfa0 0x7f72100c7cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:22.721 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:22.714+0000 7f7207fff640 1 --2- 172.21.15.16:0/2952670183 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f72100cf870 0x7f72100c7770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:22.721 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:22.714+0000 7f720e59c640 1 -- 172.21.15.16:0/2952670183 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f721015cfa0 msgr2=0x7f72100c7cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:22.722 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:22.714+0000 7f720e59c640 1 --2- 172.21.15.16:0/2952670183 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f721015cfa0 0x7f72100c7cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:22.722 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:22.714+0000 7f720e59c640 1 -- 172.21.15.16:0/2952670183 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f72100cf870 msgr2=0x7f72100c7770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:22.722 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:22.714+0000 7f720e59c640 1 --2- 172.21.15.16:0/2952670183 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f72100cf870 0x7f72100c7770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:22.723 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:22.714+0000 7f720e59c640 1 -- 172.21.15.16:0/2952670183 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72000094d0 con 0x7f721006e740 2023-05-22T16:03:22.723 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:22.714+0000 7f720ed9d640 1 --2- 172.21.15.16:0/2952670183 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f721015cfa0 0x7f72100c7cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T16:03:22.723 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:22.714+0000 7f720e59c640 1 --2- 172.21.15.16:0/2952670183 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f721006e740 0x7f72100c5300 secure :-1 s=READY pgs=2154 cs=0 l=1 rev1=1 crypto rx=0x7f7200000c10 tx=0x7f72000192a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:03:22.723 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:22.714+0000 7f7206ffd640 1 -- 172.21.15.16:0/2952670183 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72000ac410 con 0x7f721006e740 2023-05-22T16:03:22.724 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:22.714+0000 7f7206ffd640 1 -- 172.21.15.16:0/2952670183 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72000b5640 con 0x7f721006e740 2023-05-22T16:03:22.724 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:22.714+0000 7f7206ffd640 1 -- 172.21.15.16:0/2952670183 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72000b59f0 con 0x7f721006e740 2023-05-22T16:03:22.724 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:22.714+0000 7f7214e2d640 1 -- 172.21.15.16:0/2952670183 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72100cb610 con 0x7f721006e740 2023-05-22T16:03:22.725 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:22.714+0000 7f7214e2d640 1 -- 172.21.15.16:0/2952670183 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72100cb330 con 0x7f721006e740 2023-05-22T16:03:22.726 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:22.714+0000 7f7214e2d640 1 -- 172.21.15.16:0/2952670183 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f721015d370 con 0x7f721006e740 2023-05-22T16:03:22.726 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:22.714+0000 7f7206ffd640 1 -- 172.21.15.16:0/2952670183 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 79) v1 ==== 84344+0+0 (secure 0 0 0) 0x7f72000b3040 con 0x7f721006e740 2023-05-22T16:03:22.727 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:22.714+0000 7f7206ffd640 1 --2- 172.21.15.16:0/2952670183 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f71f00654a0 0x7f71f0067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:22.727 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:22.714+0000 7f7207fff640 1 -- 172.21.15.16:0/2952670183 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f71f00654a0 msgr2=0x7f71f0067950 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/65890 2023-05-22T16:03:22.727 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:22.714+0000 7f7206ffd640 1 -- 172.21.15.16:0/2952670183 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(165..165 src has 1..165) v4 ==== 7041+0+0 (secure 0 0 0) 0x7f720011cf90 con 0x7f721006e740 2023-05-22T16:03:22.727 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:22.714+0000 7f7207fff640 1 --2- 172.21.15.16:0/2952670183 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f71f00654a0 0x7f71f0067950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-05-22T16:03:22.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:22.718+0000 7f7206ffd640 1 -- 172.21.15.16:0/2952670183 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f72000ea610 con 0x7f721006e740 2023-05-22T16:03:22.906 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:22.902+0000 7f7214e2d640 1 -- 172.21.15.16:0/2952670183 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7210072160 con 0x7f721006e740 2023-05-22T16:03:22.907 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:22.902+0000 7f7206ffd640 1 -- 172.21.15.16:0/2952670183 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) v1 ==== 94+0+4644 (secure 0 0 0) 0x7f72000b1020 con 0x7f721006e740 2023-05-22T16:03:22.907 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T16:03:22.908 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":66,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8170,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6833/78835131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":78835131},{"type":"v1","addr":"172.21.15.62:6833","nonce":78835131}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":61},{"gid":8173,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/1188098562","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":1188098562},{"type":"v1","addr":"172.21.15.62:6835","nonce":1188098562}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":61},{"gid":9645,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/4204206994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":4204206994},{"type":"v1","addr":"172.21.15.16:6835","nonce":4204206994}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":61}],"filesystems":[{"mdsmap":{"epoch":66,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-05-22T15:47:03.009254+0000","modified":"2023-05-22T15:47:06.030064+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9654},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9654":{"gid":9654,"name":"c","rank":0,"incarnation":62,"state":"up:active","state_seq":3,"addr":"172.21.15.16:6837/1284822222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":1284822222},{"type":"v1","addr":"172.21.15.16:6837","nonce":1284822222}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2023-05-22T16:03:22.908 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 66 2023-05-22T16:03:22.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:22.906+0000 7f7214e2d640 1 -- 172.21.15.16:0/2952670183 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f71f00654a0 msgr2=0x7f71f0067950 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-05-22T16:03:22.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:22.906+0000 7f7214e2d640 1 --2- 172.21.15.16:0/2952670183 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f71f00654a0 0x7f71f0067950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:22.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:22.906+0000 7f7214e2d640 1 -- 172.21.15.16:0/2952670183 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f721006e740 msgr2=0x7f72100c5300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:22.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:22.906+0000 7f7214e2d640 1 --2- 172.21.15.16:0/2952670183 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f721006e740 0x7f72100c5300 secure :-1 s=READY pgs=2154 cs=0 l=1 rev1=1 crypto rx=0x7f7200000c10 tx=0x7f72000192a0 comp rx=0 tx=0).stop 2023-05-22T16:03:22.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:22.906+0000 7f7214e2d640 1 -- 172.21.15.16:0/2952670183 shutdown_connections 2023-05-22T16:03:22.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:22.906+0000 7f7214e2d640 1 --2- 172.21.15.16:0/2952670183 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f721015cfa0 0x7f72100c7cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:22.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:22.906+0000 7f7214e2d640 1 --2- 172.21.15.16:0/2952670183 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f72100cf870 0x7f72100c7770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:22.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:22.906+0000 7f7214e2d640 1 --2- 172.21.15.16:0/2952670183 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f71f00654a0 0x7f71f0067950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:22.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:22.906+0000 7f7214e2d640 1 --2- 172.21.15.16:0/2952670183 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f721006e740 0x7f72100c5300 unknown :-1 s=CLOSED pgs=2154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:22.911 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:22.906+0000 7f7214e2d640 1 -- 172.21.15.16:0/2952670183 >> 172.21.15.16:0/2952670183 conn(0x7f72100cc870 msgr2=0x7f72100c5af0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:03:22.911 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:22.906+0000 7f7214e2d640 1 -- 172.21.15.16:0/2952670183 shutdown_connections 2023-05-22T16:03:22.911 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:22.906+0000 7f7214e2d640 1 -- 172.21.15.16:0/2952670183 wait complete. 2023-05-22T16:03:22.927 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-05-22T16:03:22.995 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T16:03:22.989+0000 7f46151e9180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-05-22T16:03:23.039 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.034+0000 7f0fa5d4f640 1 Processor -- start 2023-05-22T16:03:23.040 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.034+0000 7f0fa5d4f640 1 -- start start 2023-05-22T16:03:23.041 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.038+0000 7f0fa5d4f640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0fa01563c0 0x7f0fa00c7b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:23.041 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.038+0000 7f0fa5d4f640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f0fa00c8080 0x7f0fa006c930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:23.041 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.038+0000 7f0fa5d4f640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0fa006ce70 con 0x7f0fa00cc6c0 2023-05-22T16:03:23.041 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.038+0000 7f0fa5d4f640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0fa0160700 con 0x7f0fa00c8080 2023-05-22T16:03:23.041 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.038+0000 7f0fa5d4f640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0fa00c8520 con 0x7f0fa01563c0 2023-05-22T16:03:23.041 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.038+0000 7f0f9ffff640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f0fa00c8080 0x7f0fa006c930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:23.042 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.038+0000 7f0f9ffff640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f0fa00c8080 0x7f0fa006c930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55112/0 (socket says 172.21.15.16:55112) 2023-05-22T16:03:23.042 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.038+0000 7f0f9ffff640 1 -- 172.21.15.16:0/804348341 learned_addr learned my addr 172.21.15.16:0/804348341 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:03:23.042 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.038+0000 7f0f9f7fe640 1 --2- 172.21.15.16:0/804348341 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0fa01563c0 0x7f0fa00c7b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:23.042 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.038+0000 7f0f9ffff640 1 -- 172.21.15.16:0/804348341 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0fa01563c0 msgr2=0x7f0fa00c7b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:03:23.042 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.038+0000 7f0f9ffff640 1 --2- 172.21.15.16:0/804348341 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0fa01563c0 0x7f0fa00c7b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:23.043 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.038+0000 7f0f9ffff640 1 -- 172.21.15.16:0/804348341 >> v1:172.21.15.62:6789/0 conn(0x7f0fa00cc6c0 legacy=0x7f0fa006c220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:23.043 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.038+0000 7f0f9ffff640 1 -- 172.21.15.16:0/804348341 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fa015c210 con 0x7f0fa00c8080 2023-05-22T16:03:23.043 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.038+0000 7f0f9ffff640 1 --2- 172.21.15.16:0/804348341 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0fa00c8080 0x7f0fa006c930 secure :-1 s=READY pgs=2155 cs=0 l=1 rev1=1 crypto rx=0x7f0f98009e50 tx=0x7f0f9801c9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1873520a145401d server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:03:23.043 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.038+0000 7f0f9e7fc640 1 -- 172.21.15.16:0/804348341 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f980a58a0 con 0x7f0fa00c8080 2023-05-22T16:03:23.044 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.038+0000 7f0f9e7fc640 1 -- 172.21.15.16:0/804348341 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f980a5a00 con 0x7f0fa00c8080 2023-05-22T16:03:23.044 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.038+0000 7f0f9e7fc640 1 -- 172.21.15.16:0/804348341 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f980a5d90 con 0x7f0fa00c8080 2023-05-22T16:03:23.044 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.038+0000 7f0fa5d4f640 1 -- 172.21.15.16:0/804348341 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0fa00c8080 msgr2=0x7f0fa006c930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:23.045 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.038+0000 7f0fa5d4f640 1 --2- 172.21.15.16:0/804348341 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0fa00c8080 0x7f0fa006c930 secure :-1 s=READY pgs=2155 cs=0 l=1 rev1=1 crypto rx=0x7f0f98009e50 tx=0x7f0f9801c9e0 comp rx=0 tx=0).stop 2023-05-22T16:03:23.045 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.038+0000 7f0fa5d4f640 1 -- 172.21.15.16:0/804348341 shutdown_connections 2023-05-22T16:03:23.045 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.038+0000 7f0fa5d4f640 1 --2- 172.21.15.16:0/804348341 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0fa00c8080 0x7f0fa006c930 unknown :-1 s=CLOSED pgs=2155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:23.045 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.038+0000 7f0fa5d4f640 1 --2- 172.21.15.16:0/804348341 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0fa01563c0 0x7f0fa00c7b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:23.045 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.038+0000 7f0fa5d4f640 1 -- 172.21.15.16:0/804348341 >> 172.21.15.16:0/804348341 conn(0x7f0fa00b1110 msgr2=0x7f0fa00ce830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:03:23.045 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.042+0000 7f0fa5d4f640 1 -- 172.21.15.16:0/804348341 shutdown_connections 2023-05-22T16:03:23.045 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.042+0000 7f0fa5d4f640 1 -- 172.21.15.16:0/804348341 wait complete. 2023-05-22T16:03:23.047 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.042+0000 7f0fa5d4f640 1 Processor -- start 2023-05-22T16:03:23.047 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.042+0000 7f0fa5d4f640 1 -- start start 2023-05-22T16:03:23.047 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.042+0000 7f0fa5d4f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0fa00c8080 0x7f0fa014bc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:23.048 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.042+0000 7f0fa5d4f640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0fa00cc6c0 0x7f0fa0155180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:23.048 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.042+0000 7f0fa5d4f640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0fa01556c0 0x7f0fa00c9200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:23.048 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.042+0000 7f0fa5d4f640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fa0155b40 con 0x7f0fa00c8080 2023-05-22T16:03:23.048 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.042+0000 7f0fa5d4f640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fa00c9f70 con 0x7f0fa00cc6c0 2023-05-22T16:03:23.048 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.042+0000 7f0fa5d4f640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0fa00ca250 con 0x7f0fa01556c0 2023-05-22T16:03:23.048 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.042+0000 7f0f9f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0fa00c8080 0x7f0fa014bc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:23.048 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.042+0000 7f0f9f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0fa00c8080 0x7f0fa014bc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55124/0 (socket says 172.21.15.16:55124) 2023-05-22T16:03:23.049 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.042+0000 7f0f9f7fe640 1 -- 172.21.15.16:0/2187994228 learned_addr learned my addr 172.21.15.16:0/2187994228 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:03:23.049 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.042+0000 7f0f9effd640 1 --2- 172.21.15.16:0/2187994228 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0fa00cc6c0 0x7f0fa0155180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:23.049 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.042+0000 7f0f9ffff640 1 --2- 172.21.15.16:0/2187994228 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0fa01556c0 0x7f0fa00c9200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:23.049 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.042+0000 7f0f9f7fe640 1 -- 172.21.15.16:0/2187994228 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0fa01556c0 msgr2=0x7f0fa00c9200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:23.049 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.042+0000 7f0f9f7fe640 1 --2- 172.21.15.16:0/2187994228 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0fa01556c0 0x7f0fa00c9200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:23.049 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.042+0000 7f0f9f7fe640 1 -- 172.21.15.16:0/2187994228 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0fa00cc6c0 msgr2=0x7f0fa0155180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:23.049 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.042+0000 7f0f9f7fe640 1 --2- 172.21.15.16:0/2187994228 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0fa00cc6c0 0x7f0fa0155180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:23.049 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.042+0000 7f0f9f7fe640 1 -- 172.21.15.16:0/2187994228 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f98009b30 con 0x7f0fa00c8080 2023-05-22T16:03:23.050 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.042+0000 7f0f9ffff640 1 --2- 172.21.15.16:0/2187994228 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0fa01556c0 0x7f0fa00c9200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T16:03:23.050 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.042+0000 7f0f9f7fe640 1 --2- 172.21.15.16:0/2187994228 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0fa00c8080 0x7f0fa014bc70 secure :-1 s=READY pgs=2156 cs=0 l=1 rev1=1 crypto rx=0x7f0f9000d5f0 tx=0x7f0f9000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:03:23.050 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.042+0000 7f0f97fff640 1 -- 172.21.15.16:0/2187994228 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f900125f0 con 0x7f0fa00c8080 2023-05-22T16:03:23.050 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.042+0000 7f0f97fff640 1 -- 172.21.15.16:0/2187994228 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f90012750 con 0x7f0fa00c8080 2023-05-22T16:03:23.050 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.042+0000 7f0fa5d4f640 1 -- 172.21.15.16:0/2187994228 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0fa015cd20 con 0x7f0fa00c8080 2023-05-22T16:03:23.050 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.042+0000 7f0f97fff640 1 -- 172.21.15.16:0/2187994228 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f90012ae0 con 0x7f0fa00c8080 2023-05-22T16:03:23.050 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.042+0000 7f0fa5d4f640 1 -- 172.21.15.16:0/2187994228 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0fa015ca40 con 0x7f0fa00c8080 2023-05-22T16:03:23.051 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.042+0000 7f0fa5d4f640 1 -- 172.21.15.16:0/2187994228 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0fa015ff00 con 0x7f0fa00c8080 2023-05-22T16:03:23.051 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.046+0000 7f0f97fff640 1 -- 172.21.15.16:0/2187994228 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 79) v1 ==== 84344+0+0 (secure 0 0 0) 0x7f0f90012c40 con 0x7f0fa00c8080 2023-05-22T16:03:23.051 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.046+0000 7f0f97fff640 1 --2- 172.21.15.16:0/2187994228 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f0f800654a0 0x7f0f80067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:23.052 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.046+0000 7f0f9effd640 1 -- 172.21.15.16:0/2187994228 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f0f800654a0 msgr2=0x7f0f80067950 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/65890 2023-05-22T16:03:23.052 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.046+0000 7f0f97fff640 1 -- 172.21.15.16:0/2187994228 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(165..165 src has 1..165) v4 ==== 7041+0+0 (secure 0 0 0) 0x7f0f900831a0 con 0x7f0fa00c8080 2023-05-22T16:03:23.052 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.046+0000 7f0f9effd640 1 --2- 172.21.15.16:0/2187994228 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f0f800654a0 0x7f0f80067950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-05-22T16:03:23.053 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.050+0000 7f0f97fff640 1 -- 172.21.15.16:0/2187994228 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f0f90050960 con 0x7f0fa00c8080 2023-05-22T16:03:23.069 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T16:03:23.066+0000 7fd0922ba180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-05-22T16:03:23.072 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T16:03:23.065+0000 7f46151e9180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-05-22T16:03:23.147 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T16:03:23.142+0000 7fd0922ba180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-05-22T16:03:23.202 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.198+0000 7f0fa5d4f640 1 -- 172.21.15.16:0/2187994228 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0fa004dbe0 con 0x7f0fa00c8080 2023-05-22T16:03:23.205 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.202+0000 7f0f97fff640 1 -- 172.21.15.16:0/2187994228 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v165) v1 ==== 74+0+19005 (secure 0 0 0) 0x7f0f90053f80 con 0x7f0fa00c8080 2023-05-22T16:03:23.205 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T16:03:23.206 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":165,"fsid":"b2719834-7847-4d98-90a6-f65d57f85c49","created":"2023-05-22T15:26:25.259314+0000","modified":"2023-05-22T15:58:20.479654+0000","last_up_change":"2023-05-22T15:26:36.448524+0000","last_in_change":"2023-05-22T15:26:27.675668+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-05-22T15:26:38.019781+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-05-22T15:26:43.570514+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_metadata","create_time":"2023-05-22T15:47:01.347542+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"160","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_data","create_time":"2023-05-22T15:47:01.970646+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"161","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9e7ff854-b25f-4cbb-a5c2-1799d9c53d2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":37021},{"type":"v1","addr":"172.21.15.16:6825","nonce":37021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":37021},{"type":"v1","addr":"172.21.15.16:6827","nonce":37021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":37021},{"type":"v1","addr":"172.21.15.16:6831","nonce":37021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":37021},{"type":"v1","addr":"172.21.15.16:6829","nonce":37021}]},"public_addr":"172.21.15.16:6825/37021","cluster_addr":"172.21.15.16:6827/37021","heartbeat_back_addr":"172.21.15.16:6831/37021","heartbeat_front_addr":"172.21.15.16:6829/37021","state":["exists","up"]},{"osd":1,"uuid":"4702fbc9-a927-4565-aca5-41dc69cc8264","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":37008},{"type":"v1","addr":"172.21.15.16:6801","nonce":37008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":37008},{"type":"v1","addr":"172.21.15.16:6803","nonce":37008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":37008},{"type":"v1","addr":"172.21.15.16:6807","nonce":37008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":37008},{"type":"v1","addr":"172.21.15.16:6805","nonce":37008}]},"public_addr":"172.21.15.16:6801/37008","cluster_addr":"172.21.15.16:6803/37008","heartbeat_back_addr":"172.21.15.16:6807/37008","heartbeat_front_addr":"172.21.15.16:6805/37008","state":["exists","up"]},{"osd":2,"uuid":"1e7bba46-4eba-47ef-bb81-5e2d371638ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":37009},{"type":"v1","addr":"172.21.15.16:6809","nonce":37009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":37009},{"type":"v1","addr":"172.21.15.16:6811","nonce":37009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":37009},{"type":"v1","addr":"172.21.15.16:6815","nonce":37009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":37009},{"type":"v1","addr":"172.21.15.16:6813","nonce":37009}]},"public_addr":"172.21.15.16:6809/37009","cluster_addr":"172.21.15.16:6811/37009","heartbeat_back_addr":"172.21.15.16:6815/37009","heartbeat_front_addr":"172.21.15.16:6813/37009","state":["exists","up"]},{"osd":3,"uuid":"383cacc0-5a88-4904-990e-498ca59ef48a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":37020},{"type":"v1","addr":"172.21.15.16:6817","nonce":37020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":37020},{"type":"v1","addr":"172.21.15.16:6819","nonce":37020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":37020},{"type":"v1","addr":"172.21.15.16:6823","nonce":37020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":37020},{"type":"v1","addr":"172.21.15.16:6821","nonce":37020}]},"public_addr":"172.21.15.16:6817/37020","cluster_addr":"172.21.15.16:6819/37020","heartbeat_back_addr":"172.21.15.16:6823/37020","heartbeat_front_addr":"172.21.15.16:6821/37020","state":["exists","up"]},{"osd":4,"uuid":"970500ef-95b7-42c1-b080-eed4fae7013d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":37353},{"type":"v1","addr":"172.21.15.62:6817","nonce":37353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6819","nonce":37353},{"type":"v1","addr":"172.21.15.62:6822","nonce":37353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37353},{"type":"v1","addr":"172.21.15.62:6831","nonce":37353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37353},{"type":"v1","addr":"172.21.15.62:6826","nonce":37353}]},"public_addr":"172.21.15.62:6817/37353","cluster_addr":"172.21.15.62:6822/37353","heartbeat_back_addr":"172.21.15.62:6831/37353","heartbeat_front_addr":"172.21.15.62:6826/37353","state":["exists","up"]},{"osd":5,"uuid":"022d67db-d4d8-445b-b1b9-8b446a00f5c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":37352},{"type":"v1","addr":"172.21.15.62:6801","nonce":37352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":37352},{"type":"v1","addr":"172.21.15.62:6803","nonce":37352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":37352},{"type":"v1","addr":"172.21.15.62:6807","nonce":37352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":37352},{"type":"v1","addr":"172.21.15.62:6805","nonce":37352}]},"public_addr":"172.21.15.62:6801/37352","cluster_addr":"172.21.15.62:6803/37352","heartbeat_back_addr":"172.21.15.62:6807/37352","heartbeat_front_addr":"172.21.15.62:6805/37352","state":["exists","up"]},{"osd":6,"uuid":"42bd372b-6688-4a1b-8407-e16e8c266e33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":37354},{"type":"v1","addr":"172.21.15.62:6820","nonce":37354}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":37354},{"type":"v1","addr":"172.21.15.62:6823","nonce":37354}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6829","nonce":37354},{"type":"v1","addr":"172.21.15.62:6830","nonce":37354}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6825","nonce":37354},{"type":"v1","addr":"172.21.15.62:6827","nonce":37354}]},"public_addr":"172.21.15.62:6820/37354","cluster_addr":"172.21.15.62:6823/37354","heartbeat_back_addr":"172.21.15.62:6830/37354","heartbeat_front_addr":"172.21.15.62:6827/37354","state":["exists","up"]},{"osd":7,"uuid":"ac70b2a3-f41b-4bf2-b227-7f2cbb55426d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":163,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37355},{"type":"v1","addr":"172.21.15.62:6809","nonce":37355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37355},{"type":"v1","addr":"172.21.15.62:6811","nonce":37355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":37355},{"type":"v1","addr":"172.21.15.62:6815","nonce":37355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":37355},{"type":"v1","addr":"172.21.15.62:6813","nonce":37355}]},"public_addr":"172.21.15.62:6809/37355","cluster_addr":"172.21.15.62:6811/37355","heartbeat_back_addr":"172.21.15.62:6815/37355","heartbeat_front_addr":"172.21.15.62:6813/37355","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.368896+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.107273+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.276888+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.287502+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.124791+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.251937+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.255679+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.232797+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"21.28","mappings":[{"from":4,"to":3}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:0/2019135485":"2023-05-23T15:58:20.479559+0000","172.21.15.16:0/3082249167":"2023-05-23T15:58:20.479559+0000","172.21.15.16:0/4113149961":"2023-05-23T15:58:20.479559+0000","172.21.15.16:0/2582627335":"2023-05-23T15:58:20.479559+0000","172.21.15.16:0/2722168808":"2023-05-23T15:47:16.536062+0000","172.21.15.16:0/1232860864":"2023-05-23T15:47:16.536062+0000","172.21.15.16:0/3766829209":"2023-05-23T15:47:16.536062+0000","172.21.15.16:0/3369591631":"2023-05-23T15:47:16.536062+0000","172.21.15.16:0/3417178745":"2023-05-23T15:58:20.479559+0000","172.21.15.16:6833/36921":"2023-05-23T15:47:16.536062+0000","172.21.15.16:0/2592954608":"2023-05-23T15:58:20.479559+0000","172.21.15.16:0/49958236":"2023-05-23T15:47:16.536062+0000","172.21.15.16:6832/36921":"2023-05-23T15:47:16.536062+0000","172.21.15.16:0/4161645610":"2023-05-23T15:47:16.536062+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":17,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-05-22T16:03:23.207 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.202+0000 7f0fa5d4f640 1 -- 172.21.15.16:0/2187994228 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f0f800654a0 msgr2=0x7f0f80067950 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-05-22T16:03:23.207 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.202+0000 7f0fa5d4f640 1 --2- 172.21.15.16:0/2187994228 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f0f800654a0 0x7f0f80067950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:23.208 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.202+0000 7f0fa5d4f640 1 -- 172.21.15.16:0/2187994228 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0fa00c8080 msgr2=0x7f0fa014bc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:23.208 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.202+0000 7f0fa5d4f640 1 --2- 172.21.15.16:0/2187994228 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0fa00c8080 0x7f0fa014bc70 secure :-1 s=READY pgs=2156 cs=0 l=1 rev1=1 crypto rx=0x7f0f9000d5f0 tx=0x7f0f9000dab0 comp rx=0 tx=0).stop 2023-05-22T16:03:23.208 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.202+0000 7f0fa5d4f640 1 -- 172.21.15.16:0/2187994228 shutdown_connections 2023-05-22T16:03:23.208 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.202+0000 7f0fa5d4f640 1 --2- 172.21.15.16:0/2187994228 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0fa01556c0 0x7f0fa00c9200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:23.208 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.202+0000 7f0fa5d4f640 1 --2- 172.21.15.16:0/2187994228 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0fa00cc6c0 0x7f0fa0155180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:23.208 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.202+0000 7f0fa5d4f640 1 --2- 172.21.15.16:0/2187994228 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f0f800654a0 0x7f0f80067950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:23.209 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.202+0000 7f0fa5d4f640 1 --2- 172.21.15.16:0/2187994228 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0fa00c8080 0x7f0fa014bc70 unknown :-1 s=CLOSED pgs=2156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:23.209 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.202+0000 7f0fa5d4f640 1 -- 172.21.15.16:0/2187994228 >> 172.21.15.16:0/2187994228 conn(0x7f0fa00b1110 msgr2=0x7f0fa0152290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:03:23.209 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.202+0000 7f0fa5d4f640 1 -- 172.21.15.16:0/2187994228 shutdown_connections 2023-05-22T16:03:23.209 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.202+0000 7f0fa5d4f640 1 -- 172.21.15.16:0/2187994228 wait complete. 2023-05-22T16:03:23.212 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T16:03:23.205+0000 7f46151e9180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-05-22T16:03:23.225 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-05-22T15:26:38.019781+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-05-22T15:26:43.570514+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 20, 'pool_name': 'cephfs_metadata', 'create_time': '2023-05-22T15:47:01.347542+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '160', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 21, 'pool_name': 'cephfs_data', 'create_time': '2023-05-22T15:47:01.970646+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '161', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-05-22T16:03:23.227 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-05-22T16:03:23.289 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T16:03:23.281+0000 7f46151e9180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-05-22T16:03:23.290 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T16:03:23.286+0000 7fd0922ba180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-05-22T16:03:23.356 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.350+0000 7f05c08e4640 1 Processor -- start 2023-05-22T16:03:23.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.354+0000 7f05c08e4640 1 -- start start 2023-05-22T16:03:23.359 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.354+0000 7f05c08e4640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f05bc15cfa0 0x7f05bc06db00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:23.359 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.354+0000 7f05c08e4640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f05bc0cf870 0x7f05bc0b1fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:23.359 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.354+0000 7f05c08e4640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05bc06e040 con 0x7f05bc06e740 2023-05-22T16:03:23.359 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.354+0000 7f05c08e4640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05bc0cb9f0 con 0x7f05bc15cfa0 2023-05-22T16:03:23.360 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.354+0000 7f05c08e4640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05bc0b24f0 con 0x7f05bc0cf870 2023-05-22T16:03:23.360 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.354+0000 7f05bad9d640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f05bc06e740 0x7f05bc070de0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.16:49704/0 (socket says 172.21.15.16:49704) 2023-05-22T16:03:23.360 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.354+0000 7f05bad9d640 1 -- 172.21.15.16:0/319580337 learned_addr learned my addr 172.21.15.16:0/319580337 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T16:03:23.361 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.354+0000 7f05b9d9b640 1 --2- 172.21.15.16:0/319580337 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f05bc0cf870 0x7f05bc0b1fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:23.361 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.354+0000 7f05ba59c640 1 --2- 172.21.15.16:0/319580337 >> v2:172.21.15.62:3300/0 conn(0x7f05bc15cfa0 0x7f05bc06db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:23.361 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.354+0000 7f05b959a640 1 -- 172.21.15.16:0/319580337 <== mon.1 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1747088211 0 0) 0x7f05ac001120 con 0x7f05bc06e740 2023-05-22T16:03:23.361 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.354+0000 7f05b959a640 1 -- 172.21.15.16:0/319580337 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f05a4003530 con 0x7f05bc06e740 2023-05-22T16:03:23.362 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.354+0000 7f05b9d9b640 1 -- 172.21.15.16:0/319580337 >> v2:172.21.15.62:3300/0 conn(0x7f05bc15cfa0 msgr2=0x7f05bc06db00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:03:23.362 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.354+0000 7f05b9d9b640 1 --2- 172.21.15.16:0/319580337 >> v2:172.21.15.62:3300/0 conn(0x7f05bc15cfa0 0x7f05bc06db00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:23.362 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.354+0000 7f05b9d9b640 1 -- 172.21.15.16:0/319580337 >> v1:172.21.15.62:6789/0 conn(0x7f05bc06e740 legacy=0x7f05bc070de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:23.362 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.354+0000 7f05b9d9b640 1 -- 172.21.15.16:0/319580337 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05bc06e490 con 0x7f05bc0cf870 2023-05-22T16:03:23.362 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.354+0000 7f05ba59c640 1 --2- 172.21.15.16:0/319580337 >> v2:172.21.15.62:3300/0 conn(0x7f05bc15cfa0 0x7f05bc06db00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-05-22T16:03:23.365 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.358+0000 7f05b9d9b640 1 --2- 172.21.15.16:0/319580337 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f05bc0cf870 0x7f05bc0b1fb0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f05b00097c0 tx=0x7f05b001c3c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eabab5b964fa3708 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:03:23.365 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.358+0000 7f05b959a640 1 -- 172.21.15.16:0/319580337 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05b00ac620 con 0x7f05bc0cf870 2023-05-22T16:03:23.365 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.358+0000 7f05b959a640 1 -- 172.21.15.16:0/319580337 <== mon.2 v2:172.21.15.62:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05b00ac780 con 0x7f05bc0cf870 2023-05-22T16:03:23.365 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.358+0000 7f05b959a640 1 -- 172.21.15.16:0/319580337 <== mon.2 v2:172.21.15.62:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05b00acb10 con 0x7f05bc0cf870 2023-05-22T16:03:23.365 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.358+0000 7f05c08e4640 1 -- 172.21.15.16:0/319580337 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f05bc0cf870 msgr2=0x7f05bc0b1fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:23.365 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.358+0000 7f05c08e4640 1 --2- 172.21.15.16:0/319580337 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f05bc0cf870 0x7f05bc0b1fb0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f05b00097c0 tx=0x7f05b001c3c0 comp rx=0 tx=0).stop 2023-05-22T16:03:23.366 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.358+0000 7f05c08e4640 1 -- 172.21.15.16:0/319580337 shutdown_connections 2023-05-22T16:03:23.366 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.358+0000 7f05c08e4640 1 --2- 172.21.15.16:0/319580337 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f05bc0cf870 0x7f05bc0b1fb0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:23.366 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.358+0000 7f05c08e4640 1 --2- 172.21.15.16:0/319580337 >> v2:172.21.15.62:3300/0 conn(0x7f05bc15cfa0 0x7f05bc06db00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:23.366 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.358+0000 7f05c08e4640 1 -- 172.21.15.16:0/319580337 >> 172.21.15.16:0/319580337 conn(0x7f05bc0cc870 msgr2=0x7f05bc0cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:03:23.366 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.358+0000 7f05c08e4640 1 -- 172.21.15.16:0/319580337 shutdown_connections 2023-05-22T16:03:23.366 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.358+0000 7f05c08e4640 1 -- 172.21.15.16:0/319580337 wait complete. 2023-05-22T16:03:23.366 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.358+0000 7f05c08e4640 1 Processor -- start 2023-05-22T16:03:23.367 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.358+0000 7f05c08e4640 1 -- start start 2023-05-22T16:03:23.367 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.358+0000 7f05c08e4640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f05bc06e740 0x7f05bc1521d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:23.367 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.358+0000 7f05c08e4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f05bc0cf870 0x7f05bc162150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:23.367 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.358+0000 7f05c08e4640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f05bc15cfa0 0x7f05bc0ccd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:23.367 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.358+0000 7f05c08e4640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05bc161780 con 0x7f05bc0cf870 2023-05-22T16:03:23.368 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.358+0000 7f05c08e4640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05bc161a60 con 0x7f05bc15cfa0 2023-05-22T16:03:23.368 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.358+0000 7f05c08e4640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05bc161d40 con 0x7f05bc06e740 2023-05-22T16:03:23.368 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.358+0000 7f05b9d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f05bc0cf870 0x7f05bc162150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:23.368 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.358+0000 7f05b9d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f05bc0cf870 0x7f05bc162150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55136/0 (socket says 172.21.15.16:55136) 2023-05-22T16:03:23.368 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.358+0000 7f05b9d9b640 1 -- 172.21.15.16:0/3759120688 learned_addr learned my addr 172.21.15.16:0/3759120688 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:03:23.368 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.358+0000 7f05ba59c640 1 --2- 172.21.15.16:0/3759120688 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f05bc06e740 0x7f05bc1521d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:23.369 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.358+0000 7f05bad9d640 1 --2- 172.21.15.16:0/3759120688 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f05bc15cfa0 0x7f05bc0ccd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:23.369 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.358+0000 7f05b9d9b640 1 -- 172.21.15.16:0/3759120688 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f05bc06e740 msgr2=0x7f05bc1521d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:23.369 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.358+0000 7f05b9d9b640 1 --2- 172.21.15.16:0/3759120688 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f05bc06e740 0x7f05bc1521d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:23.369 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.358+0000 7f05b9d9b640 1 -- 172.21.15.16:0/3759120688 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f05bc15cfa0 msgr2=0x7f05bc0ccd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:23.369 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.358+0000 7f05b9d9b640 1 --2- 172.21.15.16:0/3759120688 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f05bc15cfa0 0x7f05bc0ccd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:23.369 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.358+0000 7f05b9d9b640 1 -- 172.21.15.16:0/3759120688 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05b4009a70 con 0x7f05bc0cf870 2023-05-22T16:03:23.369 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.362+0000 7f05b9d9b640 1 --2- 172.21.15.16:0/3759120688 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f05bc0cf870 0x7f05bc162150 secure :-1 s=READY pgs=2157 cs=0 l=1 rev1=1 crypto rx=0x7f05b0019ff0 tx=0x7f05b0019160 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:03:23.370 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.362+0000 7f05aaffd640 1 -- 172.21.15.16:0/3759120688 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05b00bb070 con 0x7f05bc0cf870 2023-05-22T16:03:23.370 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.362+0000 7f05aaffd640 1 -- 172.21.15.16:0/3759120688 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05b00ac420 con 0x7f05bc0cf870 2023-05-22T16:03:23.370 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.362+0000 7f05c08e4640 1 -- 172.21.15.16:0/3759120688 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05b0009420 con 0x7f05bc0cf870 2023-05-22T16:03:23.370 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.362+0000 7f05aaffd640 1 -- 172.21.15.16:0/3759120688 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05b00b5660 con 0x7f05bc0cf870 2023-05-22T16:03:23.370 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.362+0000 7f05c08e4640 1 -- 172.21.15.16:0/3759120688 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05bc0cb610 con 0x7f05bc0cf870 2023-05-22T16:03:23.371 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.362+0000 7f05c08e4640 1 -- 172.21.15.16:0/3759120688 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05bc0cb330 con 0x7f05bc0cf870 2023-05-22T16:03:23.371 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.362+0000 7f05aaffd640 1 -- 172.21.15.16:0/3759120688 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 79) v1 ==== 84344+0+0 (secure 0 0 0) 0x7f05b00b3040 con 0x7f05bc0cf870 2023-05-22T16:03:23.371 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.362+0000 7f05aaffd640 1 --2- 172.21.15.16:0/3759120688 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f059c0654a0 0x7f059c067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:23.371 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.362+0000 7f05ba59c640 1 -- 172.21.15.16:0/3759120688 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f059c0654a0 msgr2=0x7f059c067950 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/65890 2023-05-22T16:03:23.371 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.362+0000 7f05aaffd640 1 -- 172.21.15.16:0/3759120688 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(165..165 src has 1..165) v4 ==== 7041+0+0 (secure 0 0 0) 0x7f05b011c4f0 con 0x7f05bc0cf870 2023-05-22T16:03:23.372 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.362+0000 7f05ba59c640 1 --2- 172.21.15.16:0/3759120688 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f059c0654a0 0x7f059c067950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-05-22T16:03:23.372 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T16:03:23.366+0000 7fd0922ba180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-05-22T16:03:23.373 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.366+0000 7f05aaffd640 1 -- 172.21.15.16:0/3759120688 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f05b00e9b00 con 0x7f05bc0cf870 2023-05-22T16:03:23.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.518+0000 7f05c08e4640 1 -- 172.21.15.16:0/3759120688 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f05bc071900 con 0x7f05bc0cf870 2023-05-22T16:03:23.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.518+0000 7f05aaffd640 1 -- 172.21.15.16:0/3759120688 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v165) v1 ==== 93+0+10 (secure 0 0 0) 0x7f05b00ed120 con 0x7f05bc0cf870 2023-05-22T16:03:23.523 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2023-05-22T16:03:23.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.522+0000 7f05c08e4640 1 -- 172.21.15.16:0/3759120688 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f059c0654a0 msgr2=0x7f059c067950 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-05-22T16:03:23.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.522+0000 7f05c08e4640 1 --2- 172.21.15.16:0/3759120688 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f059c0654a0 0x7f059c067950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:23.526 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.522+0000 7f05c08e4640 1 -- 172.21.15.16:0/3759120688 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f05bc0cf870 msgr2=0x7f05bc162150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:23.526 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.522+0000 7f05c08e4640 1 --2- 172.21.15.16:0/3759120688 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f05bc0cf870 0x7f05bc162150 secure :-1 s=READY pgs=2157 cs=0 l=1 rev1=1 crypto rx=0x7f05b0019ff0 tx=0x7f05b0019160 comp rx=0 tx=0).stop 2023-05-22T16:03:23.526 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.522+0000 7f05c08e4640 1 -- 172.21.15.16:0/3759120688 shutdown_connections 2023-05-22T16:03:23.526 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.522+0000 7f05c08e4640 1 --2- 172.21.15.16:0/3759120688 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f05bc15cfa0 0x7f05bc0ccd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:23.526 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.522+0000 7f05c08e4640 1 --2- 172.21.15.16:0/3759120688 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f059c0654a0 0x7f059c067950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:23.527 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.522+0000 7f05c08e4640 1 --2- 172.21.15.16:0/3759120688 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f05bc0cf870 0x7f05bc162150 unknown :-1 s=CLOSED pgs=2157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:23.527 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.522+0000 7f05c08e4640 1 --2- 172.21.15.16:0/3759120688 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f05bc06e740 0x7f05bc1521d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:23.527 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.522+0000 7f05c08e4640 1 -- 172.21.15.16:0/3759120688 >> 172.21.15.16:0/3759120688 conn(0x7f05bc0cc870 msgr2=0x7f05bc0c58b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:03:23.527 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.522+0000 7f05c08e4640 1 -- 172.21.15.16:0/3759120688 shutdown_connections 2023-05-22T16:03:23.527 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.522+0000 7f05c08e4640 1 -- 172.21.15.16:0/3759120688 wait complete. 2023-05-22T16:03:23.537 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T16:03:23.529+0000 7f46151e9180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-05-22T16:03:23.541 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-05-22T16:03:23.592 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T16:03:23.585+0000 7f46151e9180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-05-22T16:03:23.623 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T16:03:23.618+0000 7fd0922ba180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-05-22T16:03:23.657 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.654+0000 7f35d4c61640 1 Processor -- start 2023-05-22T16:03:23.657 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.654+0000 7f35d4c61640 1 -- start start 2023-05-22T16:03:23.659 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.654+0000 7f35d4c61640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f35d00c67d0 0x7f35d00cf730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:23.659 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.654+0000 7f35d4c61640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f35d00cc6f0 0x7f35d00c3db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:23.659 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.654+0000 7f35d4c61640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35d00cfc70 con 0x7f35d00b2880 2023-05-22T16:03:23.659 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.654+0000 7f35d4c61640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35d015a300 con 0x7f35d00cc6f0 2023-05-22T16:03:23.659 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.654+0000 7f35d4c61640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35d00c6ba0 con 0x7f35d00c67d0 2023-05-22T16:03:23.660 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.654+0000 7f35ced9d640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f35d00cc6f0 0x7f35d00c3db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:23.660 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.654+0000 7f35ced9d640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f35d00cc6f0 0x7f35d00c3db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55150/0 (socket says 172.21.15.16:55150) 2023-05-22T16:03:23.660 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.654+0000 7f35ced9d640 1 -- 172.21.15.16:0/2303477314 learned_addr learned my addr 172.21.15.16:0/2303477314 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:03:23.660 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.654+0000 7f35ce59c640 1 --2- 172.21.15.16:0/2303477314 >> v2:172.21.15.62:3300/0 conn(0x7f35d00c67d0 0x7f35d00cf730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:23.660 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.654+0000 7f35ced9d640 1 -- 172.21.15.16:0/2303477314 >> v2:172.21.15.62:3300/0 conn(0x7f35d00c67d0 msgr2=0x7f35d00cf730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:03:23.660 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.654+0000 7f35ced9d640 1 --2- 172.21.15.16:0/2303477314 >> v2:172.21.15.62:3300/0 conn(0x7f35d00c67d0 0x7f35d00cf730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:23.661 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.654+0000 7f35ced9d640 1 -- 172.21.15.16:0/2303477314 >> v1:172.21.15.62:6789/0 conn(0x7f35d00b2880 legacy=0x7f35d00cbfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:23.661 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.654+0000 7f35ced9d640 1 -- 172.21.15.16:0/2303477314 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35d00cfef0 con 0x7f35d00cc6f0 2023-05-22T16:03:23.662 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.658+0000 7f35ced9d640 1 --2- 172.21.15.16:0/2303477314 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f35d00cc6f0 0x7f35d00c3db0 secure :-1 s=READY pgs=2158 cs=0 l=1 rev1=1 crypto rx=0x7f35d006eac0 tx=0x7f35c801c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54434f46cd079c8e server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:03:23.662 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.658+0000 7f35cd59a640 1 -- 172.21.15.16:0/2303477314 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35c80a57d0 con 0x7f35d00cc6f0 2023-05-22T16:03:23.662 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.658+0000 7f35cd59a640 1 -- 172.21.15.16:0/2303477314 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35c80a5930 con 0x7f35d00cc6f0 2023-05-22T16:03:23.662 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.658+0000 7f35cd59a640 1 -- 172.21.15.16:0/2303477314 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35c80a5cc0 con 0x7f35d00cc6f0 2023-05-22T16:03:23.662 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.658+0000 7f35d4c61640 1 -- 172.21.15.16:0/2303477314 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f35d00cc6f0 msgr2=0x7f35d00c3db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:23.663 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.658+0000 7f35d4c61640 1 --2- 172.21.15.16:0/2303477314 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f35d00cc6f0 0x7f35d00c3db0 secure :-1 s=READY pgs=2158 cs=0 l=1 rev1=1 crypto rx=0x7f35d006eac0 tx=0x7f35c801c910 comp rx=0 tx=0).stop 2023-05-22T16:03:23.663 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.658+0000 7f35d4c61640 1 -- 172.21.15.16:0/2303477314 shutdown_connections 2023-05-22T16:03:23.663 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.658+0000 7f35d4c61640 1 --2- 172.21.15.16:0/2303477314 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f35d00cc6f0 0x7f35d00c3db0 unknown :-1 s=CLOSED pgs=2158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:23.663 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.658+0000 7f35d4c61640 1 --2- 172.21.15.16:0/2303477314 >> v2:172.21.15.62:3300/0 conn(0x7f35d00c67d0 0x7f35d00cf730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:23.663 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.658+0000 7f35d4c61640 1 -- 172.21.15.16:0/2303477314 >> 172.21.15.16:0/2303477314 conn(0x7f35d006f5a0 msgr2=0x7f35d006f9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:03:23.663 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.658+0000 7f35d4c61640 1 -- 172.21.15.16:0/2303477314 shutdown_connections 2023-05-22T16:03:23.664 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.658+0000 7f35d4c61640 1 -- 172.21.15.16:0/2303477314 wait complete. 2023-05-22T16:03:23.666 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.658+0000 7f35d4c61640 1 Processor -- start 2023-05-22T16:03:23.666 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.658+0000 7f35d4c61640 1 -- start start 2023-05-22T16:03:23.666 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.658+0000 7f35d4c61640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f35d00b2880 0x7f35d014fb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:23.666 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.658+0000 7f35d4c61640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f35d00c67d0 0x7f35d006a8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:23.666 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.658+0000 7f35d4c61640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f35d00cc6f0 0x7f35d00b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:23.667 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.658+0000 7f35d4c61640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35d0150040 con 0x7f35d00c67d0 2023-05-22T16:03:23.667 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.658+0000 7f35d4c61640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35d0159c90 con 0x7f35d00cc6f0 2023-05-22T16:03:23.667 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.658+0000 7f35d4c61640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35d0154bf0 con 0x7f35d00b2880 2023-05-22T16:03:23.667 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.658+0000 7f35cdd9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f35d00c67d0 0x7f35d006a8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:23.667 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.658+0000 7f35cdd9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f35d00c67d0 0x7f35d006a8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55164/0 (socket says 172.21.15.16:55164) 2023-05-22T16:03:23.667 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.658+0000 7f35cdd9b640 1 -- 172.21.15.16:0/1604393101 learned_addr learned my addr 172.21.15.16:0/1604393101 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:03:23.667 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.662+0000 7f35ced9d640 1 --2- 172.21.15.16:0/1604393101 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f35d00cc6f0 0x7f35d00b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:23.668 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.662+0000 7f35ce59c640 1 --2- 172.21.15.16:0/1604393101 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f35d00b2880 0x7f35d014fb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:23.668 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.662+0000 7f35cdd9b640 1 -- 172.21.15.16:0/1604393101 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f35d00b2880 msgr2=0x7f35d014fb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:23.668 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.662+0000 7f35cdd9b640 1 --2- 172.21.15.16:0/1604393101 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f35d00b2880 0x7f35d014fb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:23.668 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.662+0000 7f35cdd9b640 1 -- 172.21.15.16:0/1604393101 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f35d00cc6f0 msgr2=0x7f35d00b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:23.668 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.662+0000 7f35cdd9b640 1 --2- 172.21.15.16:0/1604393101 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f35d00cc6f0 0x7f35d00b04c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:23.668 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.662+0000 7f35cdd9b640 1 -- 172.21.15.16:0/1604393101 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35c8009b30 con 0x7f35d00c67d0 2023-05-22T16:03:23.668 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.662+0000 7f35cdd9b640 1 --2- 172.21.15.16:0/1604393101 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f35d00c67d0 0x7f35d006a8e0 secure :-1 s=READY pgs=2159 cs=0 l=1 rev1=1 crypto rx=0x7f35c400c3c0 tx=0x7f35c400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:03:23.669 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.662+0000 7f35beffd640 1 -- 172.21.15.16:0/1604393101 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35c40052d0 con 0x7f35d00c67d0 2023-05-22T16:03:23.669 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.662+0000 7f35beffd640 1 -- 172.21.15.16:0/1604393101 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35c4005430 con 0x7f35d00c67d0 2023-05-22T16:03:23.669 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.662+0000 7f35beffd640 1 -- 172.21.15.16:0/1604393101 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35c4003840 con 0x7f35d00c67d0 2023-05-22T16:03:23.669 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.662+0000 7f35d4c61640 1 -- 172.21.15.16:0/1604393101 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35d00d0a70 con 0x7f35d00c67d0 2023-05-22T16:03:23.669 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.662+0000 7f35d4c61640 1 -- 172.21.15.16:0/1604393101 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35d00d0790 con 0x7f35d00c67d0 2023-05-22T16:03:23.669 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.662+0000 7f35d4c61640 1 -- 172.21.15.16:0/1604393101 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35d00d0f90 con 0x7f35d00c67d0 2023-05-22T16:03:23.670 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.662+0000 7f35beffd640 1 -- 172.21.15.16:0/1604393101 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 79) v1 ==== 84344+0+0 (secure 0 0 0) 0x7f35c40039a0 con 0x7f35d00c67d0 2023-05-22T16:03:23.670 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.662+0000 7f35beffd640 1 --2- 172.21.15.16:0/1604393101 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f35b00654d0 0x7f35b0067980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:23.670 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.662+0000 7f35beffd640 1 -- 172.21.15.16:0/1604393101 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(165..165 src has 1..165) v4 ==== 7041+0+0 (secure 0 0 0) 0x7f35c40822e0 con 0x7f35d00c67d0 2023-05-22T16:03:23.670 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.662+0000 7f35ce59c640 1 -- 172.21.15.16:0/1604393101 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f35b00654d0 msgr2=0x7f35b0067980 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/65890 2023-05-22T16:03:23.671 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.662+0000 7f35ce59c640 1 --2- 172.21.15.16:0/1604393101 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f35b00654d0 0x7f35b0067980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-05-22T16:03:23.671 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.666+0000 7f35beffd640 1 -- 172.21.15.16:0/1604393101 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f35c404fa10 con 0x7f35d00c67d0 2023-05-22T16:03:23.681 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T16:03:23.678+0000 7fd0922ba180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-05-22T16:03:23.719 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.714+0000 7f35beffd640 1 -- 172.21.15.16:0/1604393101 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mgrmap(e 80) v1 ==== 83941+0+0 (secure 0 0 0) 0x7f35c40802c0 con 0x7f35d00c67d0 2023-05-22T16:03:23.720 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.714+0000 7f35beffd640 1 -- 172.21.15.16:0/1604393101 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f35b00654d0 msgr2=0x7f35b0067980 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-05-22T16:03:23.720 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.714+0000 7f35beffd640 1 --2- 172.21.15.16:0/1604393101 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f35b00654d0 0x7f35b0067980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:23.806 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T16:03:23.802+0000 7fd01ac43640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T16:03:23.807 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T16:03:23.802+0000 7fd01ac43640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T16:03:23.807 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T16:03:23.802+0000 7fd01ac43640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T16:03:23.807 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T16:03:23.802+0000 7fd01ac43640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T16:03:23.807 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T16:03:23.802+0000 7fd01ac43640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T16:03:23.807 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T16:03:23.802+0000 7fd01e44a640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T16:03:23.807 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T16:03:23.802+0000 7fd01e44a640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T16:03:23.808 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T16:03:23.802+0000 7fd01e44a640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T16:03:23.808 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T16:03:23.802+0000 7fd01e44a640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T16:03:23.808 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T16:03:23.802+0000 7fd01e44a640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T16:03:23.907 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.902+0000 7f35d4c61640 1 -- 172.21.15.16:0/1604393101 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f35d004dbe0 con 0x7f35d00c67d0 2023-05-22T16:03:23.907 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.902+0000 7f35beffd640 1 -- 172.21.15.16:0/1604393101 <== mon.0 v2:172.21.15.16:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v166) v1 ==== 92+0+10 (secure 0 0 0) 0x7f35c407c600 con 0x7f35d00c67d0 2023-05-22T16:03:23.908 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 8 2023-05-22T16:03:23.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.906+0000 7f35d4c61640 1 -- 172.21.15.16:0/1604393101 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f35d00c67d0 msgr2=0x7f35d006a8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:23.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.906+0000 7f35d4c61640 1 --2- 172.21.15.16:0/1604393101 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f35d00c67d0 0x7f35d006a8e0 secure :-1 s=READY pgs=2159 cs=0 l=1 rev1=1 crypto rx=0x7f35c400c3c0 tx=0x7f35c400c880 comp rx=0 tx=0).stop 2023-05-22T16:03:23.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.906+0000 7f35d4c61640 1 -- 172.21.15.16:0/1604393101 shutdown_connections 2023-05-22T16:03:23.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.906+0000 7f35d4c61640 1 --2- 172.21.15.16:0/1604393101 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f35d00cc6f0 0x7f35d00b04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:23.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.906+0000 7f35d4c61640 1 --2- 172.21.15.16:0/1604393101 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f35b00654d0 0x7f35b0067980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:23.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.906+0000 7f35d4c61640 1 --2- 172.21.15.16:0/1604393101 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f35d00c67d0 0x7f35d006a8e0 unknown :-1 s=CLOSED pgs=2159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:23.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.906+0000 7f35d4c61640 1 --2- 172.21.15.16:0/1604393101 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f35d00b2880 0x7f35d014fb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:23.911 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.906+0000 7f35d4c61640 1 -- 172.21.15.16:0/1604393101 >> 172.21.15.16:0/1604393101 conn(0x7f35d006f5a0 msgr2=0x7f35d00ce160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:03:23.911 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.906+0000 7f35d4c61640 1 -- 172.21.15.16:0/1604393101 shutdown_connections 2023-05-22T16:03:23.911 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:23.906+0000 7f35d4c61640 1 -- 172.21.15.16:0/1604393101 wait complete. 2023-05-22T16:03:23.932 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-05-22T16:03:24.065 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.062+0000 7f05da525640 1 Processor -- start 2023-05-22T16:03:24.066 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.062+0000 7f05da525640 1 -- start start 2023-05-22T16:03:24.066 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.062+0000 7f05da525640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f05d406cdf0 0x7f05d4070be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:24.067 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.062+0000 7f05da525640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05d406eb90 con 0x7f05d4070760 2023-05-22T16:03:24.067 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.062+0000 7f05da525640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05d4158de0 con 0x7f05d4158a10 2023-05-22T16:03:24.067 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.062+0000 7f05da525640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05d406e060 con 0x7f05d406cdf0 2023-05-22T16:03:24.067 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.062+0000 7f05d37fe640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f05d4070760 0x7f05d406c6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:41962/0 (socket says 172.21.15.16:41962) 2023-05-22T16:03:24.067 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.062+0000 7f05d37fe640 1 -- 172.21.15.16:0/2800604746 learned_addr learned my addr 172.21.15.16:0/2800604746 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T16:03:24.067 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.062+0000 7f05d8ac2640 1 --2- 172.21.15.16:0/2800604746 >> v2:172.21.15.62:3300/0 conn(0x7f05d406cdf0 0x7f05d4070be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:24.068 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.062+0000 7f05d2ffd640 1 -- 172.21.15.16:0/2800604746 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4222151667 0 0) 0x7f05d406eb90 con 0x7f05d4070760 2023-05-22T16:03:24.068 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.062+0000 7f05d2ffd640 1 -- 172.21.15.16:0/2800604746 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f05bc005740 con 0x7f05d4070760 2023-05-22T16:03:24.068 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.062+0000 7f05d8ac2640 1 -- 172.21.15.16:0/2800604746 >> v1:172.21.15.62:6789/0 conn(0x7f05d4158a10 legacy=0x7f05d406e480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:24.068 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.062+0000 7f05d8ac2640 1 -- 172.21.15.16:0/2800604746 >> v1:172.21.15.16:6789/0 conn(0x7f05d4070760 legacy=0x7f05d406c6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:24.069 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.062+0000 7f05d8ac2640 1 -- 172.21.15.16:0/2800604746 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05d40711e0 con 0x7f05d406cdf0 2023-05-22T16:03:24.069 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.066+0000 7f05d8ac2640 1 --2- 172.21.15.16:0/2800604746 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f05d406cdf0 0x7f05d4070be0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f05cc009d90 tx=0x7f05cc01c8e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=68d17138797c5798 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:03:24.069 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.066+0000 7f05d2ffd640 1 -- 172.21.15.16:0/2800604746 <== mon.1 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05cc0a5980 con 0x7f05d406cdf0 2023-05-22T16:03:24.070 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.066+0000 7f05d2ffd640 1 -- 172.21.15.16:0/2800604746 <== mon.1 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05cc0a5ae0 con 0x7f05d406cdf0 2023-05-22T16:03:24.070 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.066+0000 7f05d2ffd640 1 -- 172.21.15.16:0/2800604746 <== mon.1 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05cc0a5e70 con 0x7f05d406cdf0 2023-05-22T16:03:24.071 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.066+0000 7f05da525640 1 -- 172.21.15.16:0/2800604746 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f05d406cdf0 msgr2=0x7f05d4070be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:24.071 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.066+0000 7f05da525640 1 --2- 172.21.15.16:0/2800604746 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f05d406cdf0 0x7f05d4070be0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f05cc009d90 tx=0x7f05cc01c8e0 comp rx=0 tx=0).stop 2023-05-22T16:03:24.071 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.066+0000 7f05da525640 1 -- 172.21.15.16:0/2800604746 shutdown_connections 2023-05-22T16:03:24.071 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.066+0000 7f05da525640 1 --2- 172.21.15.16:0/2800604746 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f05d406cdf0 0x7f05d4070be0 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:24.071 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.066+0000 7f05da525640 1 -- 172.21.15.16:0/2800604746 >> 172.21.15.16:0/2800604746 conn(0x7f05d40cc870 msgr2=0x7f05d40cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:03:24.071 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.066+0000 7f05da525640 1 -- 172.21.15.16:0/2800604746 shutdown_connections 2023-05-22T16:03:24.071 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.066+0000 7f05da525640 1 -- 172.21.15.16:0/2800604746 wait complete. 2023-05-22T16:03:24.072 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.066+0000 7f05da525640 1 Processor -- start 2023-05-22T16:03:24.072 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.066+0000 7f05da525640 1 -- start start 2023-05-22T16:03:24.074 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.070+0000 7f05da525640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f05d406cdf0 0x7f05d40b0e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:24.074 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.070+0000 7f05da525640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f05d4070760 0x7f05d4151260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:24.074 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.070+0000 7f05da525640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f05d4158a10 0x7f05d4159000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:24.074 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.070+0000 7f05da525640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05d40ab9e0 con 0x7f05d406cdf0 2023-05-22T16:03:24.075 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.070+0000 7f05da525640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05d4151c00 con 0x7f05d4158a10 2023-05-22T16:03:24.075 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.070+0000 7f05da525640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05d4151ee0 con 0x7f05d4070760 2023-05-22T16:03:24.075 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.070+0000 7f05d3fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f05d406cdf0 0x7f05d40b0e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:24.075 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.070+0000 7f05d3fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f05d406cdf0 0x7f05d40b0e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55280/0 (socket says 172.21.15.16:55280) 2023-05-22T16:03:24.075 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.070+0000 7f05d3fff640 1 -- 172.21.15.16:0/1999433015 learned_addr learned my addr 172.21.15.16:0/1999433015 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:03:24.075 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.070+0000 7f05d37fe640 1 --2- 172.21.15.16:0/1999433015 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f05d4070760 0x7f05d4151260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:24.075 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.070+0000 7f05d8ac2640 1 --2- 172.21.15.16:0/1999433015 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f05d4158a10 0x7f05d4159000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:24.076 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.070+0000 7f05d3fff640 1 -- 172.21.15.16:0/1999433015 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f05d4070760 msgr2=0x7f05d4151260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:24.076 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.070+0000 7f05d3fff640 1 --2- 172.21.15.16:0/1999433015 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f05d4070760 0x7f05d4151260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:24.076 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.070+0000 7f05d3fff640 1 -- 172.21.15.16:0/1999433015 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f05d4158a10 msgr2=0x7f05d4159000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:24.076 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.070+0000 7f05d3fff640 1 --2- 172.21.15.16:0/1999433015 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f05d4158a10 0x7f05d4159000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:24.076 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.070+0000 7f05d3fff640 1 -- 172.21.15.16:0/1999433015 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05cc009a70 con 0x7f05d406cdf0 2023-05-22T16:03:24.076 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.070+0000 7f05d3fff640 1 --2- 172.21.15.16:0/1999433015 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f05d406cdf0 0x7f05d40b0e90 secure :-1 s=READY pgs=2173 cs=0 l=1 rev1=1 crypto rx=0x7f05c800c3c0 tx=0x7f05c800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:03:24.076 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.070+0000 7f05d0ff9640 1 -- 172.21.15.16:0/1999433015 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05c80052d0 con 0x7f05d406cdf0 2023-05-22T16:03:24.077 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.070+0000 7f05d0ff9640 1 -- 172.21.15.16:0/1999433015 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05c8005430 con 0x7f05d406cdf0 2023-05-22T16:03:24.077 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.070+0000 7f05da525640 1 -- 172.21.15.16:0/1999433015 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05d40ca280 con 0x7f05d406cdf0 2023-05-22T16:03:24.077 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.070+0000 7f05d0ff9640 1 -- 172.21.15.16:0/1999433015 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05c8003840 con 0x7f05d406cdf0 2023-05-22T16:03:24.077 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.070+0000 7f05da525640 1 -- 172.21.15.16:0/1999433015 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05d40cfe40 con 0x7f05d406cdf0 2023-05-22T16:03:24.077 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.070+0000 7f05da525640 1 -- 172.21.15.16:0/1999433015 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05d40ca410 con 0x7f05d406cdf0 2023-05-22T16:03:24.078 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.070+0000 7f05d0ff9640 1 -- 172.21.15.16:0/1999433015 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 80) v1 ==== 83941+0+0 (secure 0 0 0) 0x7f05c80039a0 con 0x7f05d406cdf0 2023-05-22T16:03:24.078 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.070+0000 7f05d0ff9640 1 -- 172.21.15.16:0/1999433015 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(166..166 src has 1..166) v4 ==== 7441+0+0 (secure 0 0 0) 0x7f05c8081cf0 con 0x7f05d406cdf0 2023-05-22T16:03:24.081 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.078+0000 7f05d0ff9640 1 -- 172.21.15.16:0/1999433015 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f05c804f8a0 con 0x7f05d406cdf0 2023-05-22T16:03:24.263 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.258+0000 7f05da525640 1 -- 172.21.15.16:0/1999433015 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f05d404dbe0 con 0x7f05d406cdf0 2023-05-22T16:03:24.264 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.258+0000 7f05d0ff9640 1 -- 172.21.15.16:0/1999433015 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v166) v1 ==== 104+0+11 (secure 0 0 0) 0x7f05c8052ec0 con 0x7f05d406cdf0 2023-05-22T16:03:24.265 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2023-05-22T16:03:24.266 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.262+0000 7f05da525640 1 -- 172.21.15.16:0/1999433015 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f05d406cdf0 msgr2=0x7f05d40b0e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:24.266 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.262+0000 7f05da525640 1 --2- 172.21.15.16:0/1999433015 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f05d406cdf0 0x7f05d40b0e90 secure :-1 s=READY pgs=2173 cs=0 l=1 rev1=1 crypto rx=0x7f05c800c3c0 tx=0x7f05c800c880 comp rx=0 tx=0).stop 2023-05-22T16:03:24.266 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.262+0000 7f05da525640 1 -- 172.21.15.16:0/1999433015 shutdown_connections 2023-05-22T16:03:24.266 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.262+0000 7f05da525640 1 --2- 172.21.15.16:0/1999433015 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f05d4158a10 0x7f05d4159000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:24.267 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.262+0000 7f05da525640 1 --2- 172.21.15.16:0/1999433015 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f05d4070760 0x7f05d4151260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:24.267 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.262+0000 7f05da525640 1 --2- 172.21.15.16:0/1999433015 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f05d406cdf0 0x7f05d40b0e90 secure :-1 s=CLOSED pgs=2173 cs=0 l=1 rev1=1 crypto rx=0x7f05c800c3c0 tx=0x7f05c800c880 comp rx=0 tx=0).stop 2023-05-22T16:03:24.267 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.262+0000 7f05da525640 1 -- 172.21.15.16:0/1999433015 >> 172.21.15.16:0/1999433015 conn(0x7f05d40cc870 msgr2=0x7f05d40cf140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:03:24.267 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.262+0000 7f05da525640 1 -- 172.21.15.16:0/1999433015 shutdown_connections 2023-05-22T16:03:24.267 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.262+0000 7f05da525640 1 -- 172.21.15.16:0/1999433015 wait complete. 2023-05-22T16:03:24.284 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-05-22T16:03:24.403 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.398+0000 7fce61451640 1 Processor -- start 2023-05-22T16:03:24.404 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.398+0000 7fce61451640 1 -- start start 2023-05-22T16:03:24.405 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.398+0000 7fce61451640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fce5c0ce900 0x7fce5c0ced50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:24.405 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.398+0000 7fce61451640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fce5c0cf290 0x7fce5c0d0890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:24.405 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.398+0000 7fce61451640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce5c0d0400 con 0x7fce5c15bca0 2023-05-22T16:03:24.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.398+0000 7fce61451640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce5c15c070 con 0x7fce5c0cf290 2023-05-22T16:03:24.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.398+0000 7fce61451640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce5c0d00d0 con 0x7fce5c0ce900 2023-05-22T16:03:24.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.398+0000 7fce5b7fe640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fce5c0cf290 0x7fce5c0d0890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:24.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.402+0000 7fce5b7fe640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fce5c0cf290 0x7fce5c0d0890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55288/0 (socket says 172.21.15.16:55288) 2023-05-22T16:03:24.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.402+0000 7fce5b7fe640 1 -- 172.21.15.16:0/1223364914 learned_addr learned my addr 172.21.15.16:0/1223364914 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:03:24.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.402+0000 7fce5a7fc640 1 --2- 172.21.15.16:0/1223364914 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fce5c0ce900 0x7fce5c0ced50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:24.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.402+0000 7fce5b7fe640 1 -- 172.21.15.16:0/1223364914 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fce5c0ce900 msgr2=0x7fce5c0ced50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:03:24.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.402+0000 7fce5b7fe640 1 --2- 172.21.15.16:0/1223364914 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fce5c0ce900 0x7fce5c0ced50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:24.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.402+0000 7fce5b7fe640 1 -- 172.21.15.16:0/1223364914 >> v1:172.21.15.62:6789/0 conn(0x7fce5c15bca0 legacy=0x7fce5c160160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:24.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.402+0000 7fce5b7fe640 1 -- 172.21.15.16:0/1223364914 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce5c0ccbe0 con 0x7fce5c0cf290 2023-05-22T16:03:24.408 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.402+0000 7fce5b7fe640 1 --2- 172.21.15.16:0/1223364914 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fce5c0cf290 0x7fce5c0d0890 secure :-1 s=READY pgs=2174 cs=0 l=1 rev1=1 crypto rx=0x7fce50009870 tx=0x7fce5001c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=723f5b76f22012d5 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:03:24.408 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.402+0000 7fce59ffb640 1 -- 172.21.15.16:0/1223364914 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce500ac640 con 0x7fce5c0cf290 2023-05-22T16:03:24.408 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.402+0000 7fce59ffb640 1 -- 172.21.15.16:0/1223364914 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce500ac7a0 con 0x7fce5c0cf290 2023-05-22T16:03:24.408 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.402+0000 7fce59ffb640 1 -- 172.21.15.16:0/1223364914 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce500acb30 con 0x7fce5c0cf290 2023-05-22T16:03:24.408 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.402+0000 7fce61451640 1 -- 172.21.15.16:0/1223364914 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fce5c0cf290 msgr2=0x7fce5c0d0890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:24.409 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.402+0000 7fce61451640 1 --2- 172.21.15.16:0/1223364914 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fce5c0cf290 0x7fce5c0d0890 secure :-1 s=READY pgs=2174 cs=0 l=1 rev1=1 crypto rx=0x7fce50009870 tx=0x7fce5001c480 comp rx=0 tx=0).stop 2023-05-22T16:03:24.409 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.402+0000 7fce61451640 1 -- 172.21.15.16:0/1223364914 shutdown_connections 2023-05-22T16:03:24.409 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.402+0000 7fce61451640 1 --2- 172.21.15.16:0/1223364914 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fce5c0cf290 0x7fce5c0d0890 unknown :-1 s=CLOSED pgs=2174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:24.409 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.402+0000 7fce61451640 1 --2- 172.21.15.16:0/1223364914 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fce5c0ce900 0x7fce5c0ced50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:24.409 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.402+0000 7fce61451640 1 -- 172.21.15.16:0/1223364914 >> 172.21.15.16:0/1223364914 conn(0x7fce5c0657f0 msgr2=0x7fce5c070e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:03:24.409 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.402+0000 7fce61451640 1 -- 172.21.15.16:0/1223364914 shutdown_connections 2023-05-22T16:03:24.409 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.402+0000 7fce61451640 1 -- 172.21.15.16:0/1223364914 wait complete. 2023-05-22T16:03:24.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.406+0000 7fce61451640 1 Processor -- start 2023-05-22T16:03:24.412 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.406+0000 7fce61451640 1 -- start start 2023-05-22T16:03:24.412 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.406+0000 7fce61451640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fce5c0ce900 0x7fce5c14ec30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:24.412 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.406+0000 7fce61451640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fce5c0cf290 0x7fce5c1490c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:24.412 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.406+0000 7fce61451640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fce5c15bca0 0x7fce5c0cb910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:24.412 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.406+0000 7fce61451640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce5c149600 con 0x7fce5c0cf290 2023-05-22T16:03:24.412 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.406+0000 7fce61451640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce5c14deb0 con 0x7fce5c0ce900 2023-05-22T16:03:24.413 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.406+0000 7fce61451640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce5c0ca470 con 0x7fce5c15bca0 2023-05-22T16:03:24.413 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.406+0000 7fce5a7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fce5c0cf290 0x7fce5c1490c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:24.413 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.406+0000 7fce5a7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fce5c0cf290 0x7fce5c1490c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55300/0 (socket says 172.21.15.16:55300) 2023-05-22T16:03:24.413 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.406+0000 7fce5a7fc640 1 -- 172.21.15.16:0/3588422173 learned_addr learned my addr 172.21.15.16:0/3588422173 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:03:24.413 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.406+0000 7fce5affd640 1 --2- 172.21.15.16:0/3588422173 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fce5c0ce900 0x7fce5c14ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:24.413 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.406+0000 7fce5b7fe640 1 --2- 172.21.15.16:0/3588422173 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fce5c15bca0 0x7fce5c0cb910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:24.413 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.406+0000 7fce5a7fc640 1 -- 172.21.15.16:0/3588422173 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fce5c15bca0 msgr2=0x7fce5c0cb910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:24.414 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.406+0000 7fce5a7fc640 1 --2- 172.21.15.16:0/3588422173 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fce5c15bca0 0x7fce5c0cb910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:24.414 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.406+0000 7fce5a7fc640 1 -- 172.21.15.16:0/3588422173 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fce5c0ce900 msgr2=0x7fce5c14ec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:24.414 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.406+0000 7fce5a7fc640 1 --2- 172.21.15.16:0/3588422173 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fce5c0ce900 0x7fce5c14ec30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:24.414 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.406+0000 7fce5a7fc640 1 -- 172.21.15.16:0/3588422173 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce500094d0 con 0x7fce5c0cf290 2023-05-22T16:03:24.414 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.406+0000 7fce5a7fc640 1 --2- 172.21.15.16:0/3588422173 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fce5c0cf290 0x7fce5c1490c0 secure :-1 s=READY pgs=2175 cs=0 l=1 rev1=1 crypto rx=0x7fce4c00d800 tx=0x7fce4c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:03:24.414 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.406+0000 7fce437fe640 1 -- 172.21.15.16:0/3588422173 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce4c00b6c0 con 0x7fce5c0cf290 2023-05-22T16:03:24.414 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.406+0000 7fce61451640 1 -- 172.21.15.16:0/3588422173 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce5c0d15e0 con 0x7fce5c0cf290 2023-05-22T16:03:24.414 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.406+0000 7fce437fe640 1 -- 172.21.15.16:0/3588422173 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce4c00b820 con 0x7fce5c0cf290 2023-05-22T16:03:24.415 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.406+0000 7fce437fe640 1 -- 172.21.15.16:0/3588422173 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce4c00bb90 con 0x7fce5c0cf290 2023-05-22T16:03:24.415 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.406+0000 7fce61451640 1 -- 172.21.15.16:0/3588422173 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce5c0cd1a0 con 0x7fce5c0cf290 2023-05-22T16:03:24.415 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.406+0000 7fce61451640 1 -- 172.21.15.16:0/3588422173 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce5c0d1770 con 0x7fce5c0cf290 2023-05-22T16:03:24.415 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.406+0000 7fce437fe640 1 -- 172.21.15.16:0/3588422173 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 80) v1 ==== 83941+0+0 (secure 0 0 0) 0x7fce4c00bcf0 con 0x7fce5c0cf290 2023-05-22T16:03:24.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.410+0000 7fce437fe640 1 -- 172.21.15.16:0/3588422173 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(166..166 src has 1..166) v4 ==== 7441+0+0 (secure 0 0 0) 0x7fce4c082db0 con 0x7fce5c0cf290 2023-05-22T16:03:24.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.414+0000 7fce437fe640 1 -- 172.21.15.16:0/3588422173 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fce4c0506c0 con 0x7fce5c0cf290 2023-05-22T16:03:24.573 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-05-22T16:03:24.577 DEBUG:teuthology.orchestra.run.smithi062:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-05-22T16:03:24.579 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.574+0000 7fce61451640 1 -- 172.21.15.16:0/3588422173 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fce5c0c5d60 con 0x7fce5c0cf290 2023-05-22T16:03:24.580 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.574+0000 7fce437fe640 1 -- 172.21.15.16:0/3588422173 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v166) v1 ==== 100+0+11 (secure 0 0 0) 0x7fce4c053eb0 con 0x7fce5c0cf290 2023-05-22T16:03:24.580 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 64 2023-05-22T16:03:24.581 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.578+0000 7fce61451640 1 -- 172.21.15.16:0/3588422173 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fce5c0cf290 msgr2=0x7fce5c1490c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:24.581 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.578+0000 7fce61451640 1 --2- 172.21.15.16:0/3588422173 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fce5c0cf290 0x7fce5c1490c0 secure :-1 s=READY pgs=2175 cs=0 l=1 rev1=1 crypto rx=0x7fce4c00d800 tx=0x7fce4c00dcc0 comp rx=0 tx=0).stop 2023-05-22T16:03:24.581 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.578+0000 7fce61451640 1 -- 172.21.15.16:0/3588422173 shutdown_connections 2023-05-22T16:03:24.581 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.578+0000 7fce61451640 1 --2- 172.21.15.16:0/3588422173 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fce5c15bca0 0x7fce5c0cb910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:24.582 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.578+0000 7fce61451640 1 --2- 172.21.15.16:0/3588422173 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fce5c0cf290 0x7fce5c1490c0 unknown :-1 s=CLOSED pgs=2175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:24.582 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.578+0000 7fce61451640 1 --2- 172.21.15.16:0/3588422173 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fce5c0ce900 0x7fce5c14ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:24.582 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.578+0000 7fce61451640 1 -- 172.21.15.16:0/3588422173 >> 172.21.15.16:0/3588422173 conn(0x7fce5c0657f0 msgr2=0x7fce5c154b40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:03:24.582 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.578+0000 7fce61451640 1 -- 172.21.15.16:0/3588422173 shutdown_connections 2023-05-22T16:03:24.582 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.578+0000 7fce61451640 1 -- 172.21.15.16:0/3588422173 wait complete. 2023-05-22T16:03:24.598 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-05-22T16:03:24.787 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.782+0000 7f8b4a4cd640 1 Processor -- start 2023-05-22T16:03:24.788 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.782+0000 7f8b4a4cd640 1 -- start start 2023-05-22T16:03:24.789 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.782+0000 7f8b4a4cd640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8b440ca4e0 0x7f8b4415e460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:24.789 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.782+0000 7f8b4a4cd640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f8b44070df0 0x7f8b44071210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:24.789 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.782+0000 7f8b4a4cd640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b440b0650 con 0x7f8b440c9b40 2023-05-22T16:03:24.789 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.782+0000 7f8b4a4cd640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b440b0370 con 0x7f8b44070df0 2023-05-22T16:03:24.790 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.782+0000 7f8b4a4cd640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b4415d980 con 0x7f8b440ca4e0 2023-05-22T16:03:24.790 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.786+0000 7f8b48a6a640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f8b44070df0 0x7f8b44071210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:24.790 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.786+0000 7f8b48a6a640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f8b44070df0 0x7f8b44071210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55308/0 (socket says 172.21.15.16:55308) 2023-05-22T16:03:24.790 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.786+0000 7f8b48a6a640 1 -- 172.21.15.16:0/1948392532 learned_addr learned my addr 172.21.15.16:0/1948392532 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:03:24.790 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.786+0000 7f8b43fff640 1 --2- 172.21.15.16:0/1948392532 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8b440ca4e0 0x7f8b4415e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:24.790 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.786+0000 7f8b48a6a640 1 -- 172.21.15.16:0/1948392532 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8b440ca4e0 msgr2=0x7f8b4415e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:03:24.791 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.786+0000 7f8b48a6a640 1 --2- 172.21.15.16:0/1948392532 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8b440ca4e0 0x7f8b4415e460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:24.791 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.786+0000 7f8b48a6a640 1 -- 172.21.15.16:0/1948392532 >> v1:172.21.15.62:6789/0 conn(0x7f8b440c9b40 legacy=0x7f8b440afb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:03:24.791 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.786+0000 7f8b48a6a640 1 -- 172.21.15.16:0/1948392532 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b4415dc60 con 0x7f8b44070df0 2023-05-22T16:03:24.792 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.786+0000 7f8b48a6a640 1 --2- 172.21.15.16:0/1948392532 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8b44070df0 0x7f8b44071210 secure :-1 s=READY pgs=2176 cs=0 l=1 rev1=1 crypto rx=0x7f8b3c009870 tx=0x7f8b3c01c440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=608a19c6931db226 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:03:24.792 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.786+0000 7f8b42ffd640 1 -- 172.21.15.16:0/1948392532 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b3c0ad660 con 0x7f8b44070df0 2023-05-22T16:03:24.792 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.786+0000 7f8b42ffd640 1 -- 172.21.15.16:0/1948392532 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b3c0ad7c0 con 0x7f8b44070df0 2023-05-22T16:03:24.792 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.786+0000 7f8b42ffd640 1 -- 172.21.15.16:0/1948392532 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b3c0adb50 con 0x7f8b44070df0 2023-05-22T16:03:24.793 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.786+0000 7f8b4a4cd640 1 -- 172.21.15.16:0/1948392532 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8b44070df0 msgr2=0x7f8b44071210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:24.793 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.786+0000 7f8b4a4cd640 1 --2- 172.21.15.16:0/1948392532 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8b44070df0 0x7f8b44071210 secure :-1 s=READY pgs=2176 cs=0 l=1 rev1=1 crypto rx=0x7f8b3c009870 tx=0x7f8b3c01c440 comp rx=0 tx=0).stop 2023-05-22T16:03:24.793 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.786+0000 7f8b4a4cd640 1 -- 172.21.15.16:0/1948392532 shutdown_connections 2023-05-22T16:03:24.793 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.786+0000 7f8b4a4cd640 1 --2- 172.21.15.16:0/1948392532 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8b44070df0 0x7f8b44071210 unknown :-1 s=CLOSED pgs=2176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:24.793 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.786+0000 7f8b4a4cd640 1 --2- 172.21.15.16:0/1948392532 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8b440ca4e0 0x7f8b4415e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:24.793 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.786+0000 7f8b4a4cd640 1 -- 172.21.15.16:0/1948392532 >> 172.21.15.16:0/1948392532 conn(0x7f8b44069720 msgr2=0x7f8b44069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:03:24.793 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.786+0000 7f8b4a4cd640 1 -- 172.21.15.16:0/1948392532 shutdown_connections 2023-05-22T16:03:24.794 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.786+0000 7f8b4a4cd640 1 -- 172.21.15.16:0/1948392532 wait complete. 2023-05-22T16:03:24.795 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.790+0000 7f8b4a4cd640 1 Processor -- start 2023-05-22T16:03:24.796 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.790+0000 7f8b4a4cd640 1 -- start start 2023-05-22T16:03:24.796 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.790+0000 7f8b4a4cd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8b44070df0 0x7f8b441522c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:24.796 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.790+0000 7f8b4a4cd640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8b440c9b40 0x7f8b4414a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:24.797 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.790+0000 7f8b4a4cd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8b440ca4e0 0x7f8b4414b950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:24.797 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.790+0000 7f8b4a4cd640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b4414a940 con 0x7f8b44070df0 2023-05-22T16:03:24.797 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.790+0000 7f8b4a4cd640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b4414be90 con 0x7f8b440ca4e0 2023-05-22T16:03:24.797 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.790+0000 7f8b4a4cd640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b44159a60 con 0x7f8b440c9b40 2023-05-22T16:03:24.797 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.790+0000 7f8b43fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8b44070df0 0x7f8b441522c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:24.797 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.790+0000 7f8b43fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8b44070df0 0x7f8b441522c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55310/0 (socket says 172.21.15.16:55310) 2023-05-22T16:03:24.798 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.790+0000 7f8b43fff640 1 -- 172.21.15.16:0/747761038 learned_addr learned my addr 172.21.15.16:0/747761038 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:03:24.798 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.790+0000 7f8b437fe640 1 --2- 172.21.15.16:0/747761038 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8b440c9b40 0x7f8b4414a400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:24.798 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.790+0000 7f8b48a6a640 1 --2- 172.21.15.16:0/747761038 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8b440ca4e0 0x7f8b4414b950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:24.798 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.790+0000 7f8b43fff640 1 -- 172.21.15.16:0/747761038 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8b440c9b40 msgr2=0x7f8b4414a400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:24.798 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.790+0000 7f8b43fff640 1 --2- 172.21.15.16:0/747761038 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8b440c9b40 0x7f8b4414a400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:24.798 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.790+0000 7f8b43fff640 1 -- 172.21.15.16:0/747761038 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8b440ca4e0 msgr2=0x7f8b4414b950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:24.798 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.790+0000 7f8b43fff640 1 --2- 172.21.15.16:0/747761038 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8b440ca4e0 0x7f8b4414b950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:24.798 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.790+0000 7f8b43fff640 1 -- 172.21.15.16:0/747761038 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b3c0094d0 con 0x7f8b44070df0 2023-05-22T16:03:24.799 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.790+0000 7f8b437fe640 1 --2- 172.21.15.16:0/747761038 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8b440c9b40 0x7f8b4414a400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T16:03:24.799 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.790+0000 7f8b48a6a640 1 --2- 172.21.15.16:0/747761038 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8b440ca4e0 0x7f8b4414b950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T16:03:24.799 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.790+0000 7f8b43fff640 1 --2- 172.21.15.16:0/747761038 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8b44070df0 0x7f8b441522c0 secure :-1 s=READY pgs=2177 cs=0 l=1 rev1=1 crypto rx=0x7f8b2c00d5f0 tx=0x7f8b2c00dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:03:24.799 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.790+0000 7f8b40ff9640 1 -- 172.21.15.16:0/747761038 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b2c0125f0 con 0x7f8b44070df0 2023-05-22T16:03:24.799 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.790+0000 7f8b40ff9640 1 -- 172.21.15.16:0/747761038 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b2c012750 con 0x7f8b44070df0 2023-05-22T16:03:24.799 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.790+0000 7f8b4a4cd640 1 -- 172.21.15.16:0/747761038 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b440c8e80 con 0x7f8b44070df0 2023-05-22T16:03:24.799 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.790+0000 7f8b40ff9640 1 -- 172.21.15.16:0/747761038 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b2c012ae0 con 0x7f8b44070df0 2023-05-22T16:03:24.800 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.790+0000 7f8b4a4cd640 1 -- 172.21.15.16:0/747761038 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b440ca8b0 con 0x7f8b44070df0 2023-05-22T16:03:24.800 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.790+0000 7f8b4a4cd640 1 -- 172.21.15.16:0/747761038 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b440c9010 con 0x7f8b44070df0 2023-05-22T16:03:24.800 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.790+0000 7f8b40ff9640 1 -- 172.21.15.16:0/747761038 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 81) v1 ==== 84068+0+0 (secure 0 0 0) 0x7f8b2c012c40 con 0x7f8b44070df0 2023-05-22T16:03:24.801 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.790+0000 7f8b40ff9640 1 --2- 172.21.15.16:0/747761038 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f8b28065200 0x7f8b280676b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:24.801 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.794+0000 7f8b40ff9640 1 -- 172.21.15.16:0/747761038 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(166..166 src has 1..166) v4 ==== 7441+0+0 (secure 0 0 0) 0x7f8b2c082ec0 con 0x7f8b44070df0 2023-05-22T16:03:24.801 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.794+0000 7f8b437fe640 1 --2- 172.21.15.16:0/747761038 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f8b28065200 0x7f8b280676b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:24.801 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.794+0000 7f8b437fe640 1 --2- 172.21.15.16:0/747761038 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f8b28065200 0x7f8b280676b0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f8b440c4b90 tx=0x7f8b34009040 comp rx=0 tx=0).ready entity=mgr.10032 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:03:24.801 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.798+0000 7f8b40ff9640 1 -- 172.21.15.16:0/747761038 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f8b2c050910 con 0x7f8b44070df0 2023-05-22T16:03:24.987 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.982+0000 7f8b4a4cd640 1 -- 172.21.15.16:0/747761038 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8b4404dbe0 con 0x7f8b44070df0 2023-05-22T16:03:24.987 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.982+0000 7f8b40ff9640 1 -- 172.21.15.16:0/747761038 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) v1 ==== 94+0+4644 (secure 0 0 0) 0x7f8b2c007380 con 0x7f8b44070df0 2023-05-22T16:03:24.988 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T16:03:24.988 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":66,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8170,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6833/78835131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":78835131},{"type":"v1","addr":"172.21.15.62:6833","nonce":78835131}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":61},{"gid":8173,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/1188098562","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":1188098562},{"type":"v1","addr":"172.21.15.62:6835","nonce":1188098562}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":61},{"gid":9645,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/4204206994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":4204206994},{"type":"v1","addr":"172.21.15.16:6835","nonce":4204206994}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":61}],"filesystems":[{"mdsmap":{"epoch":66,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-05-22T15:47:03.009254+0000","modified":"2023-05-22T15:47:06.030064+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9654},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9654":{"gid":9654,"name":"c","rank":0,"incarnation":62,"state":"up:active","state_seq":3,"addr":"172.21.15.16:6837/1284822222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":1284822222},{"type":"v1","addr":"172.21.15.16:6837","nonce":1284822222}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2023-05-22T16:03:24.988 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 66 2023-05-22T16:03:24.990 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.986+0000 7f8b4a4cd640 1 -- 172.21.15.16:0/747761038 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f8b28065200 msgr2=0x7f8b280676b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:24.990 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.986+0000 7f8b4a4cd640 1 --2- 172.21.15.16:0/747761038 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f8b28065200 0x7f8b280676b0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f8b440c4b90 tx=0x7f8b34009040 comp rx=0 tx=0).stop 2023-05-22T16:03:24.990 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.986+0000 7f8b4a4cd640 1 -- 172.21.15.16:0/747761038 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8b44070df0 msgr2=0x7f8b441522c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:24.990 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.986+0000 7f8b4a4cd640 1 --2- 172.21.15.16:0/747761038 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8b44070df0 0x7f8b441522c0 secure :-1 s=READY pgs=2177 cs=0 l=1 rev1=1 crypto rx=0x7f8b2c00d5f0 tx=0x7f8b2c00dab0 comp rx=0 tx=0).stop 2023-05-22T16:03:24.991 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.986+0000 7f8b4a4cd640 1 -- 172.21.15.16:0/747761038 shutdown_connections 2023-05-22T16:03:24.991 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.986+0000 7f8b4a4cd640 1 --2- 172.21.15.16:0/747761038 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8b440ca4e0 0x7f8b4414b950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:24.991 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.986+0000 7f8b4a4cd640 1 --2- 172.21.15.16:0/747761038 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8b440c9b40 0x7f8b4414a400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:24.991 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.986+0000 7f8b4a4cd640 1 --2- 172.21.15.16:0/747761038 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f8b28065200 0x7f8b280676b0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:24.991 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.986+0000 7f8b4a4cd640 1 --2- 172.21.15.16:0/747761038 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8b44070df0 0x7f8b441522c0 unknown :-1 s=CLOSED pgs=2177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:24.991 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.986+0000 7f8b4a4cd640 1 -- 172.21.15.16:0/747761038 >> 172.21.15.16:0/747761038 conn(0x7f8b44069720 msgr2=0x7f8b44153250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:03:24.991 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.986+0000 7f8b4a4cd640 1 -- 172.21.15.16:0/747761038 shutdown_connections 2023-05-22T16:03:24.992 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:24.986+0000 7f8b4a4cd640 1 -- 172.21.15.16:0/747761038 wait complete. 2023-05-22T16:03:25.006 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-05-22T16:03:25.118 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.114+0000 7f41aec91640 1 Processor -- start 2023-05-22T16:03:25.119 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.114+0000 7f41aec91640 1 -- start start 2023-05-22T16:03:25.120 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.114+0000 7f41aec91640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f41a80b7dd0 0x7f41a80ccab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:25.120 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.114+0000 7f41aec91640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41a80cd170 con 0x7f41a8160250 2023-05-22T16:03:25.120 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.114+0000 7f41aec91640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41a80d2ac0 con 0x7f41a815f700 2023-05-22T16:03:25.120 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.114+0000 7f41aec91640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41a80d3360 con 0x7f41a80b7dd0 2023-05-22T16:03:25.120 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.114+0000 7f41ad22e640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f41a8160250 0x7f41a80d0410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:41972/0 (socket says 172.21.15.16:41972) 2023-05-22T16:03:25.121 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.114+0000 7f41ad22e640 1 -- 172.21.15.16:0/2855854263 learned_addr learned my addr 172.21.15.16:0/2855854263 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T16:03:25.122 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.118+0000 7f41aca2d640 1 --2- 172.21.15.16:0/2855854263 >> v2:172.21.15.62:3300/0 conn(0x7f41a80b7dd0 0x7f41a80ccab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:25.122 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.118+0000 7f41a77fe640 1 -- 172.21.15.16:0/2855854263 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3457185087 0 0) 0x7f41a80cd170 con 0x7f41a8160250 2023-05-22T16:03:25.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.118+0000 7f41a77fe640 1 -- 172.21.15.16:0/2855854263 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4198003530 con 0x7f41a8160250 2023-05-22T16:03:25.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.118+0000 7f41a77fe640 1 -- 172.21.15.16:0/2855854263 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 258799193 0 0) 0x7f4198003530 con 0x7f41a8160250 2023-05-22T16:03:25.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.118+0000 7f41a77fe640 1 -- 172.21.15.16:0/2855854263 >> v2:172.21.15.62:3300/0 conn(0x7f41a80b7dd0 msgr2=0x7f41a80ccab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:03:25.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.118+0000 7f41a77fe640 1 --2- 172.21.15.16:0/2855854263 >> v2:172.21.15.62:3300/0 conn(0x7f41a80b7dd0 0x7f41a80ccab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:25.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.118+0000 7f41a77fe640 1 -- 172.21.15.16:0/2855854263 >> v1:172.21.15.62:6789/0 conn(0x7f41a815f700 legacy=0x7f41a815fb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:25.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.118+0000 7f41a77fe640 1 -- 172.21.15.16:0/2855854263 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41a80d2da0 con 0x7f41a8160250 2023-05-22T16:03:25.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.118+0000 7f41a77fe640 1 -- 172.21.15.16:0/2855854263 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7f41a0001a00 con 0x7f41a8160250 2023-05-22T16:03:25.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.118+0000 7f41a77fe640 1 -- 172.21.15.16:0/2855854263 >> v1:172.21.15.16:6789/0 conn(0x7f41a8160250 legacy=0x7f41a80d0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:25.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.118+0000 7f41a77fe640 1 --2- 172.21.15.16:0/2855854263 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4198003980 0x7f4198003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:25.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.118+0000 7f41a77fe640 1 --2- 172.21.15.16:0/2855854263 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f41980047d0 0x7f4198004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:25.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.118+0000 7f41a77fe640 1 --2- 172.21.15.16:0/2855854263 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4198005180 0x7f419800b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:25.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.118+0000 7f41a77fe640 1 -- 172.21.15.16:0/2855854263 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41a80d3360 con 0x7f41980047d0 2023-05-22T16:03:25.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.118+0000 7f41a77fe640 1 -- 172.21.15.16:0/2855854263 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f419800c000 con 0x7f4198005180 2023-05-22T16:03:25.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.118+0000 7f41a77fe640 1 -- 172.21.15.16:0/2855854263 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f419800c1c0 con 0x7f4198003980 2023-05-22T16:03:25.126 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.118+0000 7f41a7fff640 1 --2- 172.21.15.16:0/2855854263 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f41980047d0 0x7f4198004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:25.127 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.118+0000 7f41aca2d640 1 --2- 172.21.15.16:0/2855854263 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4198003980 0x7f4198003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:25.127 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.118+0000 7f41ad22e640 1 --2- 172.21.15.16:0/2855854263 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4198005180 0x7f419800b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:25.127 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.118+0000 7f41a7fff640 1 -- 172.21.15.16:0/2855854263 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4198003980 msgr2=0x7f4198003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:03:25.128 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.118+0000 7f41a7fff640 1 --2- 172.21.15.16:0/2855854263 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4198003980 0x7f4198003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:25.128 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.118+0000 7f41a7fff640 1 -- 172.21.15.16:0/2855854263 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4198005180 msgr2=0x7f419800b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:03:25.128 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.118+0000 7f41a7fff640 1 --2- 172.21.15.16:0/2855854263 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4198005180 0x7f419800b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:25.128 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.118+0000 7f41a7fff640 1 -- 172.21.15.16:0/2855854263 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4198003530 con 0x7f41980047d0 2023-05-22T16:03:25.128 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.118+0000 7f41ad22e640 1 -- 172.21.15.16:0/2855854263 reap_dead start 2023-05-22T16:03:25.131 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.118+0000 7f41a7fff640 1 --2- 172.21.15.16:0/2855854263 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f41980047d0 0x7f4198004c40 secure :-1 s=READY pgs=2179 cs=0 l=1 rev1=1 crypto rx=0x7f4194001780 tx=0x7f419401c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3537762bc381669b server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:03:25.131 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.118+0000 7f41a77fe640 1 -- 172.21.15.16:0/2855854263 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41940052d0 con 0x7f41980047d0 2023-05-22T16:03:25.131 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.118+0000 7f41a77fe640 1 -- 172.21.15.16:0/2855854263 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41940a7050 con 0x7f41980047d0 2023-05-22T16:03:25.131 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.118+0000 7f41a77fe640 1 -- 172.21.15.16:0/2855854263 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4194003600 con 0x7f41980047d0 2023-05-22T16:03:25.132 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.118+0000 7f41aec91640 1 -- 172.21.15.16:0/2855854263 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f41980047d0 msgr2=0x7f4198004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:25.132 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.118+0000 7f41aec91640 1 --2- 172.21.15.16:0/2855854263 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f41980047d0 0x7f4198004c40 secure :-1 s=READY pgs=2179 cs=0 l=1 rev1=1 crypto rx=0x7f4194001780 tx=0x7f419401c3e0 comp rx=0 tx=0).stop 2023-05-22T16:03:25.132 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.118+0000 7f41aec91640 1 -- 172.21.15.16:0/2855854263 shutdown_connections 2023-05-22T16:03:25.132 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.118+0000 7f41aec91640 1 --2- 172.21.15.16:0/2855854263 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f41980047d0 0x7f4198004c40 unknown :-1 s=CLOSED pgs=2179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:25.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.118+0000 7f41aec91640 1 -- 172.21.15.16:0/2855854263 >> 172.21.15.16:0/2855854263 conn(0x7f41a80655c0 msgr2=0x7f41a8070e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:03:25.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.122+0000 7f41aec91640 1 -- 172.21.15.16:0/2855854263 shutdown_connections 2023-05-22T16:03:25.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.122+0000 7f41aec91640 1 -- 172.21.15.16:0/2855854263 wait complete. 2023-05-22T16:03:25.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.122+0000 7f41aec91640 1 Processor -- start 2023-05-22T16:03:25.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.122+0000 7f41aec91640 1 -- start start 2023-05-22T16:03:25.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.122+0000 7f41aec91640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f41980047d0 0x7f41a80b6200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:25.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.122+0000 7f41aec91640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f41a80c6920 0x7f41a80c6d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:25.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.122+0000 7f41aec91640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f41a815adf0 0x7f41a815b270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:25.134 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.122+0000 7f41aec91640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41a80c72d0 con 0x7f41a815adf0 2023-05-22T16:03:25.134 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.122+0000 7f41aec91640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41a815b7b0 con 0x7f41a80c6920 2023-05-22T16:03:25.134 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.122+0000 7f41aec91640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41a80c74d0 con 0x7f41980047d0 2023-05-22T16:03:25.134 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.122+0000 7f41ad22e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f41a815adf0 0x7f41a815b270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:25.134 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.122+0000 7f41ad22e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f41a815adf0 0x7f41a815b270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55326/0 (socket says 172.21.15.16:55326) 2023-05-22T16:03:25.134 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.122+0000 7f41ad22e640 1 -- 172.21.15.16:0/530292972 learned_addr learned my addr 172.21.15.16:0/530292972 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:03:25.134 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.122+0000 7f41aca2d640 1 --2- 172.21.15.16:0/530292972 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f41980047d0 0x7f41a80b6200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:25.135 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.122+0000 7f41a7fff640 1 --2- 172.21.15.16:0/530292972 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f41a80c6920 0x7f41a80c6d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:25.135 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.122+0000 7f41ad22e640 1 -- 172.21.15.16:0/530292972 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f41980047d0 msgr2=0x7f41a80b6200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:25.136 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.122+0000 7f41ad22e640 1 --2- 172.21.15.16:0/530292972 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f41980047d0 0x7f41a80b6200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:25.136 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.122+0000 7f41ad22e640 1 -- 172.21.15.16:0/530292972 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f41a80c6920 msgr2=0x7f41a80c6d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:25.136 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.122+0000 7f41ad22e640 1 --2- 172.21.15.16:0/530292972 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f41a80c6920 0x7f41a80c6d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:25.136 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.122+0000 7f41ad22e640 1 -- 172.21.15.16:0/530292972 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41a80ccab0 con 0x7f41a815adf0 2023-05-22T16:03:25.136 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.122+0000 7f41ad22e640 1 --2- 172.21.15.16:0/530292972 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f41a815adf0 0x7f41a815b270 secure :-1 s=READY pgs=2180 cs=0 l=1 rev1=1 crypto rx=0x7f41a81607a0 tx=0x7f41a0011850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:03:25.136 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.122+0000 7f41a57fa640 1 -- 172.21.15.16:0/530292972 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41a0014370 con 0x7f41a815adf0 2023-05-22T16:03:25.137 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.122+0000 7f41a57fa640 1 -- 172.21.15.16:0/530292972 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41a00144d0 con 0x7f41a815adf0 2023-05-22T16:03:25.137 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.122+0000 7f41aec91640 1 -- 172.21.15.16:0/530292972 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41a80d3080 con 0x7f41a815adf0 2023-05-22T16:03:25.137 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.122+0000 7f41a57fa640 1 -- 172.21.15.16:0/530292972 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41a0014880 con 0x7f41a815adf0 2023-05-22T16:03:25.137 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.122+0000 7f41aec91640 1 -- 172.21.15.16:0/530292972 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41a80ccc40 con 0x7f41a815adf0 2023-05-22T16:03:25.137 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.122+0000 7f41aec91640 1 -- 172.21.15.16:0/530292972 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41a804dbe0 con 0x7f41a815adf0 2023-05-22T16:03:25.137 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.126+0000 7f41a57fa640 1 -- 172.21.15.16:0/530292972 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 81) v1 ==== 84068+0+0 (secure 0 0 0) 0x7f41a00149e0 con 0x7f41a815adf0 2023-05-22T16:03:25.137 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.126+0000 7f41a57fa640 1 --2- 172.21.15.16:0/530292972 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f418c0651d0 0x7f418c067680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:25.137 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.126+0000 7f41a57fa640 1 -- 172.21.15.16:0/530292972 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(166..166 src has 1..166) v4 ==== 7441+0+0 (secure 0 0 0) 0x7f41a0084e80 con 0x7f41a815adf0 2023-05-22T16:03:25.138 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.126+0000 7f41aca2d640 1 --2- 172.21.15.16:0/530292972 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f418c0651d0 0x7f418c067680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:25.138 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.126+0000 7f41aca2d640 1 --2- 172.21.15.16:0/530292972 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f418c0651d0 0x7f418c067680 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f419c009ab0 tx=0x7f419c008040 comp rx=0 tx=0).ready entity=mgr.10032 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:03:25.138 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.130+0000 7f41a57fa640 1 -- 172.21.15.16:0/530292972 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f41a0052800 con 0x7f41a815adf0 2023-05-22T16:03:25.286 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.282+0000 7f41aec91640 1 -- 172.21.15.16:0/530292972 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f41a80b5610 con 0x7f41a815adf0 2023-05-22T16:03:25.288 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.282+0000 7f41a57fa640 1 -- 172.21.15.16:0/530292972 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v166) v1 ==== 74+0+19609 (secure 0 0 0) 0x7f41a001c090 con 0x7f41a815adf0 2023-05-22T16:03:25.289 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T16:03:25.289 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":166,"fsid":"b2719834-7847-4d98-90a6-f65d57f85c49","created":"2023-05-22T15:26:25.259314+0000","modified":"2023-05-22T16:03:23.703347+0000","last_up_change":"2023-05-22T15:26:36.448524+0000","last_in_change":"2023-05-22T15:26:27.675668+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-05-22T15:26:38.019781+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-05-22T15:26:43.570514+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_metadata","create_time":"2023-05-22T15:47:01.347542+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"160","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_data","create_time":"2023-05-22T15:47:01.970646+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"166","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":33,"snap_epoch":166,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9e7ff854-b25f-4cbb-a5c2-1799d9c53d2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":37021},{"type":"v1","addr":"172.21.15.16:6825","nonce":37021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":37021},{"type":"v1","addr":"172.21.15.16:6827","nonce":37021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":37021},{"type":"v1","addr":"172.21.15.16:6831","nonce":37021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":37021},{"type":"v1","addr":"172.21.15.16:6829","nonce":37021}]},"public_addr":"172.21.15.16:6825/37021","cluster_addr":"172.21.15.16:6827/37021","heartbeat_back_addr":"172.21.15.16:6831/37021","heartbeat_front_addr":"172.21.15.16:6829/37021","state":["exists","up"]},{"osd":1,"uuid":"4702fbc9-a927-4565-aca5-41dc69cc8264","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":37008},{"type":"v1","addr":"172.21.15.16:6801","nonce":37008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":37008},{"type":"v1","addr":"172.21.15.16:6803","nonce":37008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":37008},{"type":"v1","addr":"172.21.15.16:6807","nonce":37008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":37008},{"type":"v1","addr":"172.21.15.16:6805","nonce":37008}]},"public_addr":"172.21.15.16:6801/37008","cluster_addr":"172.21.15.16:6803/37008","heartbeat_back_addr":"172.21.15.16:6807/37008","heartbeat_front_addr":"172.21.15.16:6805/37008","state":["exists","up"]},{"osd":2,"uuid":"1e7bba46-4eba-47ef-bb81-5e2d371638ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":37009},{"type":"v1","addr":"172.21.15.16:6809","nonce":37009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":37009},{"type":"v1","addr":"172.21.15.16:6811","nonce":37009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":37009},{"type":"v1","addr":"172.21.15.16:6815","nonce":37009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":37009},{"type":"v1","addr":"172.21.15.16:6813","nonce":37009}]},"public_addr":"172.21.15.16:6809/37009","cluster_addr":"172.21.15.16:6811/37009","heartbeat_back_addr":"172.21.15.16:6815/37009","heartbeat_front_addr":"172.21.15.16:6813/37009","state":["exists","up"]},{"osd":3,"uuid":"383cacc0-5a88-4904-990e-498ca59ef48a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":37020},{"type":"v1","addr":"172.21.15.16:6817","nonce":37020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":37020},{"type":"v1","addr":"172.21.15.16:6819","nonce":37020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":37020},{"type":"v1","addr":"172.21.15.16:6823","nonce":37020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":37020},{"type":"v1","addr":"172.21.15.16:6821","nonce":37020}]},"public_addr":"172.21.15.16:6817/37020","cluster_addr":"172.21.15.16:6819/37020","heartbeat_back_addr":"172.21.15.16:6823/37020","heartbeat_front_addr":"172.21.15.16:6821/37020","state":["exists","up"]},{"osd":4,"uuid":"970500ef-95b7-42c1-b080-eed4fae7013d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":37353},{"type":"v1","addr":"172.21.15.62:6817","nonce":37353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6819","nonce":37353},{"type":"v1","addr":"172.21.15.62:6822","nonce":37353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37353},{"type":"v1","addr":"172.21.15.62:6831","nonce":37353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37353},{"type":"v1","addr":"172.21.15.62:6826","nonce":37353}]},"public_addr":"172.21.15.62:6817/37353","cluster_addr":"172.21.15.62:6822/37353","heartbeat_back_addr":"172.21.15.62:6831/37353","heartbeat_front_addr":"172.21.15.62:6826/37353","state":["exists","up"]},{"osd":5,"uuid":"022d67db-d4d8-445b-b1b9-8b446a00f5c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":37352},{"type":"v1","addr":"172.21.15.62:6801","nonce":37352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":37352},{"type":"v1","addr":"172.21.15.62:6803","nonce":37352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":37352},{"type":"v1","addr":"172.21.15.62:6807","nonce":37352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":37352},{"type":"v1","addr":"172.21.15.62:6805","nonce":37352}]},"public_addr":"172.21.15.62:6801/37352","cluster_addr":"172.21.15.62:6803/37352","heartbeat_back_addr":"172.21.15.62:6807/37352","heartbeat_front_addr":"172.21.15.62:6805/37352","state":["exists","up"]},{"osd":6,"uuid":"42bd372b-6688-4a1b-8407-e16e8c266e33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":37354},{"type":"v1","addr":"172.21.15.62:6820","nonce":37354}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":37354},{"type":"v1","addr":"172.21.15.62:6823","nonce":37354}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6829","nonce":37354},{"type":"v1","addr":"172.21.15.62:6830","nonce":37354}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6825","nonce":37354},{"type":"v1","addr":"172.21.15.62:6827","nonce":37354}]},"public_addr":"172.21.15.62:6820/37354","cluster_addr":"172.21.15.62:6823/37354","heartbeat_back_addr":"172.21.15.62:6830/37354","heartbeat_front_addr":"172.21.15.62:6827/37354","state":["exists","up"]},{"osd":7,"uuid":"ac70b2a3-f41b-4bf2-b227-7f2cbb55426d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":163,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37355},{"type":"v1","addr":"172.21.15.62:6809","nonce":37355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37355},{"type":"v1","addr":"172.21.15.62:6811","nonce":37355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":37355},{"type":"v1","addr":"172.21.15.62:6815","nonce":37355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":37355},{"type":"v1","addr":"172.21.15.62:6813","nonce":37355}]},"public_addr":"172.21.15.62:6809/37355","cluster_addr":"172.21.15.62:6811/37355","heartbeat_back_addr":"172.21.15.62:6815/37355","heartbeat_front_addr":"172.21.15.62:6813/37355","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.368896+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.107273+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.276888+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.287502+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.124791+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.251937+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.255679+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.232797+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"21.28","mappings":[{"from":4,"to":3}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:0/313468131":"2023-05-23T16:03:23.703312+0000","172.21.15.16:0/2645427422":"2023-05-23T16:03:23.703312+0000","172.21.15.16:6833/65890":"2023-05-23T16:03:23.703312+0000","172.21.15.16:6832/65890":"2023-05-23T16:03:23.703312+0000","172.21.15.16:0/1654177733":"2023-05-23T16:03:23.703312+0000","172.21.15.16:0/2019135485":"2023-05-23T15:58:20.479559+0000","172.21.15.16:0/3082249167":"2023-05-23T15:58:20.479559+0000","172.21.15.16:0/4113149961":"2023-05-23T15:58:20.479559+0000","172.21.15.16:0/631429095":"2023-05-23T16:03:23.703312+0000","172.21.15.16:0/2582627335":"2023-05-23T15:58:20.479559+0000","172.21.15.16:0/2722168808":"2023-05-23T15:47:16.536062+0000","172.21.15.16:0/1232860864":"2023-05-23T15:47:16.536062+0000","172.21.15.16:0/3766829209":"2023-05-23T15:47:16.536062+0000","172.21.15.16:0/3369591631":"2023-05-23T15:47:16.536062+0000","172.21.15.16:0/3417178745":"2023-05-23T15:58:20.479559+0000","172.21.15.16:6833/36921":"2023-05-23T15:47:16.536062+0000","172.21.15.16:0/392276665":"2023-05-23T16:03:23.703312+0000","172.21.15.16:0/2592954608":"2023-05-23T15:58:20.479559+0000","172.21.15.16:0/49958236":"2023-05-23T15:47:16.536062+0000","172.21.15.16:6832/36921":"2023-05-23T15:47:16.536062+0000","172.21.15.16:0/1747179873":"2023-05-23T16:03:23.703312+0000","172.21.15.16:0/4161645610":"2023-05-23T15:47:16.536062+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":17,"snaps":[{"begin":2,"length":2}]},{"pool":21,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[{"pool":21,"snaps":[{"begin":2,"length":32}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-05-22T16:03:25.290 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.286+0000 7f41aec91640 1 -- 172.21.15.16:0/530292972 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f418c0651d0 msgr2=0x7f418c067680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:25.290 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.286+0000 7f41aec91640 1 --2- 172.21.15.16:0/530292972 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f418c0651d0 0x7f418c067680 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f419c009ab0 tx=0x7f419c008040 comp rx=0 tx=0).stop 2023-05-22T16:03:25.291 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.286+0000 7f41aec91640 1 -- 172.21.15.16:0/530292972 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f41a815adf0 msgr2=0x7f41a815b270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:25.291 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.286+0000 7f41aec91640 1 --2- 172.21.15.16:0/530292972 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f41a815adf0 0x7f41a815b270 secure :-1 s=READY pgs=2180 cs=0 l=1 rev1=1 crypto rx=0x7f41a81607a0 tx=0x7f41a0011850 comp rx=0 tx=0).stop 2023-05-22T16:03:25.291 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.286+0000 7f41aec91640 1 -- 172.21.15.16:0/530292972 shutdown_connections 2023-05-22T16:03:25.291 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.286+0000 7f41aec91640 1 --2- 172.21.15.16:0/530292972 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f418c0651d0 0x7f418c067680 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:25.291 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.286+0000 7f41aec91640 1 --2- 172.21.15.16:0/530292972 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f41a815adf0 0x7f41a815b270 unknown :-1 s=CLOSED pgs=2180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:25.291 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.286+0000 7f41aec91640 1 --2- 172.21.15.16:0/530292972 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f41a80c6920 0x7f41a80c6d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:25.292 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.286+0000 7f41aec91640 1 --2- 172.21.15.16:0/530292972 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f41980047d0 0x7f41a80b6200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:25.292 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.286+0000 7f41aec91640 1 -- 172.21.15.16:0/530292972 >> 172.21.15.16:0/530292972 conn(0x7f41a80655c0 msgr2=0x7f41a80b15b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:03:25.292 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.286+0000 7f41aec91640 1 -- 172.21.15.16:0/530292972 shutdown_connections 2023-05-22T16:03:25.293 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.286+0000 7f41aec91640 1 -- 172.21.15.16:0/530292972 wait complete. 2023-05-22T16:03:25.307 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-05-22T16:03:25.308 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-05-22T16:03:25.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.426+0000 7fc11d6ed640 1 Processor -- start 2023-05-22T16:03:25.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.426+0000 7fc11d6ed640 1 -- start start 2023-05-22T16:03:25.432 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.426+0000 7fc11d6ed640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc1180b7dd0 0x7fc1180ccab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:25.432 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.426+0000 7fc11d6ed640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc11815f700 0x7fc11815fb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:25.433 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.426+0000 7fc11d6ed640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc118160500 con 0x7fc118160080 2023-05-22T16:03:25.433 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.426+0000 7fc11d6ed640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1180d3360 con 0x7fc1180b7dd0 2023-05-22T16:03:25.433 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.426+0000 7fc11d6ed640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1180cd170 con 0x7fc11815f700 2023-05-22T16:03:25.433 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.426+0000 7fc116ffd640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc1180b7dd0 0x7fc1180ccab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:25.433 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.426+0000 7fc116ffd640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc1180b7dd0 0x7fc1180ccab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55336/0 (socket says 172.21.15.16:55336) 2023-05-22T16:03:25.433 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.426+0000 7fc116ffd640 1 -- 172.21.15.16:0/4241289445 learned_addr learned my addr 172.21.15.16:0/4241289445 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:03:25.433 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.430+0000 7fc10ffff640 1 --2- 172.21.15.16:0/4241289445 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc11815f700 0x7fc11815fb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:25.434 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.430+0000 7fc116ffd640 1 -- 172.21.15.16:0/4241289445 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc11815f700 msgr2=0x7fc11815fb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:03:25.434 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.430+0000 7fc116ffd640 1 --2- 172.21.15.16:0/4241289445 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc11815f700 0x7fc11815fb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:25.434 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.430+0000 7fc116ffd640 1 -- 172.21.15.16:0/4241289445 >> v1:172.21.15.62:6789/0 conn(0x7fc118160080 legacy=0x7fc1180ce720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:03:25.434 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.430+0000 7fc116ffd640 1 -- 172.21.15.16:0/4241289445 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1180d2ac0 con 0x7fc1180b7dd0 2023-05-22T16:03:25.435 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.430+0000 7fc116ffd640 1 --2- 172.21.15.16:0/4241289445 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc1180b7dd0 0x7fc1180ccab0 secure :-1 s=READY pgs=2181 cs=0 l=1 rev1=1 crypto rx=0x7fc108009870 tx=0x7fc10801c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a245130440bb666 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:03:25.435 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.430+0000 7fc1167fc640 1 -- 172.21.15.16:0/4241289445 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1080aa5f0 con 0x7fc1180b7dd0 2023-05-22T16:03:25.435 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.430+0000 7fc1167fc640 1 -- 172.21.15.16:0/4241289445 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1080aa750 con 0x7fc1180b7dd0 2023-05-22T16:03:25.436 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.430+0000 7fc1167fc640 1 -- 172.21.15.16:0/4241289445 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1080aaae0 con 0x7fc1180b7dd0 2023-05-22T16:03:25.436 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.430+0000 7fc11d6ed640 1 -- 172.21.15.16:0/4241289445 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc1180b7dd0 msgr2=0x7fc1180ccab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:25.436 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.430+0000 7fc11d6ed640 1 --2- 172.21.15.16:0/4241289445 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc1180b7dd0 0x7fc1180ccab0 secure :-1 s=READY pgs=2181 cs=0 l=1 rev1=1 crypto rx=0x7fc108009870 tx=0x7fc10801c4d0 comp rx=0 tx=0).stop 2023-05-22T16:03:25.437 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.430+0000 7fc11d6ed640 1 -- 172.21.15.16:0/4241289445 shutdown_connections 2023-05-22T16:03:25.437 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.430+0000 7fc11d6ed640 1 --2- 172.21.15.16:0/4241289445 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc11815f700 0x7fc11815fb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:25.437 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.430+0000 7fc11d6ed640 1 --2- 172.21.15.16:0/4241289445 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc1180b7dd0 0x7fc1180ccab0 unknown :-1 s=CLOSED pgs=2181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:25.437 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.430+0000 7fc11d6ed640 1 -- 172.21.15.16:0/4241289445 >> 172.21.15.16:0/4241289445 conn(0x7fc1180655c0 msgr2=0x7fc118070e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:03:25.437 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.430+0000 7fc11d6ed640 1 -- 172.21.15.16:0/4241289445 shutdown_connections 2023-05-22T16:03:25.437 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.430+0000 7fc11d6ed640 1 -- 172.21.15.16:0/4241289445 wait complete. 2023-05-22T16:03:25.439 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.434+0000 7fc11d6ed640 1 Processor -- start 2023-05-22T16:03:25.439 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.434+0000 7fc11d6ed640 1 -- start start 2023-05-22T16:03:25.439 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.434+0000 7fc11d6ed640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc1180b7dd0 0x7fc11815c200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:25.439 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.434+0000 7fc11d6ed640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc11815f700 0x7fc118154a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:25.439 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.434+0000 7fc11d6ed640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc118160080 0x7fc1181588f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:25.440 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.434+0000 7fc11d6ed640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1180cacf0 con 0x7fc118160080 2023-05-22T16:03:25.440 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.434+0000 7fc11d6ed640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1180ca950 con 0x7fc1180b7dd0 2023-05-22T16:03:25.440 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.434+0000 7fc11d6ed640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc11815ca50 con 0x7fc11815f700 2023-05-22T16:03:25.440 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.434+0000 7fc1177fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc118160080 0x7fc1181588f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:25.440 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.434+0000 7fc1177fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc118160080 0x7fc1181588f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55340/0 (socket says 172.21.15.16:55340) 2023-05-22T16:03:25.440 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.434+0000 7fc1177fe640 1 -- 172.21.15.16:0/2318725549 learned_addr learned my addr 172.21.15.16:0/2318725549 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:03:25.440 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.434+0000 7fc116ffd640 1 --2- 172.21.15.16:0/2318725549 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc1180b7dd0 0x7fc11815c200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:25.440 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.434+0000 7fc10ffff640 1 --2- 172.21.15.16:0/2318725549 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc11815f700 0x7fc118154a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:25.441 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.434+0000 7fc1177fe640 1 -- 172.21.15.16:0/2318725549 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc11815f700 msgr2=0x7fc118154a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:25.441 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.434+0000 7fc1177fe640 1 --2- 172.21.15.16:0/2318725549 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc11815f700 0x7fc118154a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:25.441 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.434+0000 7fc1177fe640 1 -- 172.21.15.16:0/2318725549 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc1180b7dd0 msgr2=0x7fc11815c200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:25.441 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.434+0000 7fc1177fe640 1 --2- 172.21.15.16:0/2318725549 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc1180b7dd0 0x7fc11815c200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:25.441 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.434+0000 7fc1177fe640 1 -- 172.21.15.16:0/2318725549 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1080094d0 con 0x7fc118160080 2023-05-22T16:03:25.441 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.434+0000 7fc1177fe640 1 --2- 172.21.15.16:0/2318725549 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc118160080 0x7fc1181588f0 secure :-1 s=READY pgs=2182 cs=0 l=1 rev1=1 crypto rx=0x7fc11000b970 tx=0x7fc11000be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:03:25.441 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.434+0000 7fc10f7fe640 1 -- 172.21.15.16:0/2318725549 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc11000cb40 con 0x7fc118160080 2023-05-22T16:03:25.442 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.434+0000 7fc10f7fe640 1 -- 172.21.15.16:0/2318725549 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc11000cca0 con 0x7fc118160080 2023-05-22T16:03:25.442 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.434+0000 7fc11d6ed640 1 -- 172.21.15.16:0/2318725549 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc118168a20 con 0x7fc118160080 2023-05-22T16:03:25.442 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.434+0000 7fc10f7fe640 1 -- 172.21.15.16:0/2318725549 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc110014680 con 0x7fc118160080 2023-05-22T16:03:25.442 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.434+0000 7fc11d6ed640 1 -- 172.21.15.16:0/2318725549 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc118168890 con 0x7fc118160080 2023-05-22T16:03:25.443 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.434+0000 7fc11d6ed640 1 -- 172.21.15.16:0/2318725549 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc118168bb0 con 0x7fc118160080 2023-05-22T16:03:25.443 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.434+0000 7fc10f7fe640 1 -- 172.21.15.16:0/2318725549 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 81) v1 ==== 84068+0+0 (secure 0 0 0) 0x7fc11001d3b0 con 0x7fc118160080 2023-05-22T16:03:25.443 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.438+0000 7fc10f7fe640 1 --2- 172.21.15.16:0/2318725549 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fc0f80651d0 0x7fc0f8067680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:25.443 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.438+0000 7fc10f7fe640 1 -- 172.21.15.16:0/2318725549 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(166..166 src has 1..166) v4 ==== 7441+0+0 (secure 0 0 0) 0x7fc1100827f0 con 0x7fc118160080 2023-05-22T16:03:25.443 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.438+0000 7fc116ffd640 1 --2- 172.21.15.16:0/2318725549 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fc0f80651d0 0x7fc0f8067680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:25.443 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.438+0000 7fc116ffd640 1 --2- 172.21.15.16:0/2318725549 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fc0f80651d0 0x7fc0f8067680 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fc1080ae000 tx=0x7fc1080a7000 comp rx=0 tx=0).ready entity=mgr.10032 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:03:25.444 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.442+0000 7fc10f7fe640 1 -- 172.21.15.16:0/2318725549 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fc110053c70 con 0x7fc118160080 2023-05-22T16:03:25.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.626+0000 7fc11d6ed640 1 -- 172.21.15.16:0/2318725549 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc11804dbe0 con 0x7fc118160080 2023-05-22T16:03:25.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.626+0000 7fc10f7fe640 1 -- 172.21.15.16:0/2318725549 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) v1 ==== 94+0+4644 (secure 0 0 0) 0x7fc110053800 con 0x7fc118160080 2023-05-22T16:03:25.631 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T16:03:25.631 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":66,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8170,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6833/78835131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":78835131},{"type":"v1","addr":"172.21.15.62:6833","nonce":78835131}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":61},{"gid":8173,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/1188098562","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":1188098562},{"type":"v1","addr":"172.21.15.62:6835","nonce":1188098562}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":61},{"gid":9645,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/4204206994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":4204206994},{"type":"v1","addr":"172.21.15.16:6835","nonce":4204206994}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":61}],"filesystems":[{"mdsmap":{"epoch":66,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-05-22T15:47:03.009254+0000","modified":"2023-05-22T15:47:06.030064+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9654},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9654":{"gid":9654,"name":"c","rank":0,"incarnation":62,"state":"up:active","state_seq":3,"addr":"172.21.15.16:6837/1284822222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":1284822222},{"type":"v1","addr":"172.21.15.16:6837","nonce":1284822222}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2023-05-22T16:03:25.631 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 66 2023-05-22T16:03:25.633 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.630+0000 7fc11d6ed640 1 -- 172.21.15.16:0/2318725549 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fc0f80651d0 msgr2=0x7fc0f8067680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:25.633 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.630+0000 7fc11d6ed640 1 --2- 172.21.15.16:0/2318725549 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fc0f80651d0 0x7fc0f8067680 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fc1080ae000 tx=0x7fc1080a7000 comp rx=0 tx=0).stop 2023-05-22T16:03:25.633 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.630+0000 7fc11d6ed640 1 -- 172.21.15.16:0/2318725549 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc118160080 msgr2=0x7fc1181588f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:25.633 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.630+0000 7fc11d6ed640 1 --2- 172.21.15.16:0/2318725549 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc118160080 0x7fc1181588f0 secure :-1 s=READY pgs=2182 cs=0 l=1 rev1=1 crypto rx=0x7fc11000b970 tx=0x7fc11000be30 comp rx=0 tx=0).stop 2023-05-22T16:03:25.633 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.630+0000 7fc11d6ed640 1 -- 172.21.15.16:0/2318725549 shutdown_connections 2023-05-22T16:03:25.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.630+0000 7fc11d6ed640 1 --2- 172.21.15.16:0/2318725549 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fc0f80651d0 0x7fc0f8067680 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:25.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.630+0000 7fc11d6ed640 1 --2- 172.21.15.16:0/2318725549 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc118160080 0x7fc1181588f0 unknown :-1 s=CLOSED pgs=2182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:25.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.630+0000 7fc11d6ed640 1 --2- 172.21.15.16:0/2318725549 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc11815f700 0x7fc118154a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:25.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.630+0000 7fc11d6ed640 1 --2- 172.21.15.16:0/2318725549 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc1180b7dd0 0x7fc11815c200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:25.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.630+0000 7fc11d6ed640 1 -- 172.21.15.16:0/2318725549 >> 172.21.15.16:0/2318725549 conn(0x7fc1180655c0 msgr2=0x7fc1180b38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:03:25.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.630+0000 7fc11d6ed640 1 -- 172.21.15.16:0/2318725549 shutdown_connections 2023-05-22T16:03:25.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.630+0000 7fc11d6ed640 1 -- 172.21.15.16:0/2318725549 wait complete. 2023-05-22T16:03:25.650 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-05-22T16:03:25.766 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.762+0000 7f0998b1c640 1 Processor -- start 2023-05-22T16:03:25.767 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.762+0000 7f0998b1c640 1 -- start start 2023-05-22T16:03:25.768 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.762+0000 7f0998b1c640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f0994159a40 0x7f09940c8ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:25.768 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.762+0000 7f0998b1c640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f099415a0a0 con 0x7f09940b0250 2023-05-22T16:03:25.768 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.762+0000 7f0998b1c640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0994153580 con 0x7f09941531b0 2023-05-22T16:03:25.768 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.762+0000 7f0998b1c640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0994159e90 con 0x7f0994159a40 2023-05-22T16:03:25.769 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.762+0000 7f0991d9b640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f09940b0250 0x7f0994159330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:41984/0 (socket says 172.21.15.16:41984) 2023-05-22T16:03:25.769 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.762+0000 7f0991d9b640 1 -- 172.21.15.16:0/1420799613 learned_addr learned my addr 172.21.15.16:0/1420799613 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T16:03:25.769 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.762+0000 7f0992d9d640 1 --2- 172.21.15.16:0/1420799613 >> v2:172.21.15.62:3300/0 conn(0x7f0994159a40 0x7f09940c8ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:25.769 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.762+0000 7f099159a640 1 -- 172.21.15.16:0/1420799613 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4204632681 0 0) 0x7f099415a0a0 con 0x7f09940b0250 2023-05-22T16:03:25.769 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.762+0000 7f099159a640 1 -- 172.21.15.16:0/1420799613 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f097c003530 con 0x7f09940b0250 2023-05-22T16:03:25.770 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.766+0000 7f099159a640 1 -- 172.21.15.16:0/1420799613 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1732141506 0 0) 0x7f097c003530 con 0x7f09940b0250 2023-05-22T16:03:25.771 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.766+0000 7f099159a640 1 -- 172.21.15.16:0/1420799613 >> v2:172.21.15.62:3300/0 conn(0x7f0994159a40 msgr2=0x7f09940c8ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:03:25.771 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.766+0000 7f099159a640 1 --2- 172.21.15.16:0/1420799613 >> v2:172.21.15.62:3300/0 conn(0x7f0994159a40 0x7f09940c8ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:25.771 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.766+0000 7f099159a640 1 -- 172.21.15.16:0/1420799613 >> v1:172.21.15.62:6789/0 conn(0x7f09941531b0 legacy=0x7f09940afb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:25.771 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.766+0000 7f099159a640 1 -- 172.21.15.16:0/1420799613 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f099415d5e0 con 0x7f09940b0250 2023-05-22T16:03:25.771 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.766+0000 7f099159a640 1 -- 172.21.15.16:0/1420799613 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7f0988001680 con 0x7f09940b0250 2023-05-22T16:03:25.771 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.766+0000 7f099159a640 1 -- 172.21.15.16:0/1420799613 >> v1:172.21.15.16:6789/0 conn(0x7f09940b0250 legacy=0x7f0994159330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:25.771 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.766+0000 7f099159a640 1 --2- 172.21.15.16:0/1420799613 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f097c003980 0x7f097c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:25.772 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.766+0000 7f099159a640 1 --2- 172.21.15.16:0/1420799613 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f097c0047d0 0x7f097c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:25.772 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.766+0000 7f099159a640 1 --2- 172.21.15.16:0/1420799613 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f097c005180 0x7f097c00b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:25.772 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.766+0000 7f099159a640 1 -- 172.21.15.16:0/1420799613 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0994159e90 con 0x7f097c0047d0 2023-05-22T16:03:25.772 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.766+0000 7f099159a640 1 -- 172.21.15.16:0/1420799613 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f097c00c000 con 0x7f097c005180 2023-05-22T16:03:25.772 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.766+0000 7f099159a640 1 -- 172.21.15.16:0/1420799613 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f097c00c1c0 con 0x7f097c003980 2023-05-22T16:03:25.772 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.766+0000 7f0991d9b640 1 --2- 172.21.15.16:0/1420799613 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f097c0047d0 0x7f097c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:25.772 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.766+0000 7f0992d9d640 1 --2- 172.21.15.16:0/1420799613 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f097c005180 0x7f097c00b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:25.773 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.766+0000 7f099259c640 1 --2- 172.21.15.16:0/1420799613 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f097c003980 0x7f097c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:25.773 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.766+0000 7f0991d9b640 1 -- 172.21.15.16:0/1420799613 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f097c003980 msgr2=0x7f097c003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:03:25.773 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.766+0000 7f0991d9b640 1 --2- 172.21.15.16:0/1420799613 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f097c003980 0x7f097c003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:25.773 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.766+0000 7f0991d9b640 1 -- 172.21.15.16:0/1420799613 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f097c005180 msgr2=0x7f097c00b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:03:25.773 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.766+0000 7f0991d9b640 1 --2- 172.21.15.16:0/1420799613 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f097c005180 0x7f097c00b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:25.774 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.766+0000 7f0992d9d640 1 -- 172.21.15.16:0/1420799613 reap_dead start 2023-05-22T16:03:25.774 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.766+0000 7f0991d9b640 1 -- 172.21.15.16:0/1420799613 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f097c003530 con 0x7f097c0047d0 2023-05-22T16:03:25.774 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.766+0000 7f099259c640 1 --2- 172.21.15.16:0/1420799613 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f097c003980 0x7f097c003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T16:03:25.777 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.766+0000 7f0991d9b640 1 --2- 172.21.15.16:0/1420799613 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f097c0047d0 0x7f097c004c40 secure :-1 s=READY pgs=2184 cs=0 l=1 rev1=1 crypto rx=0x7f099406f400 tx=0x7f098801f310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9b34ab0ed0ef5d6 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:03:25.777 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.766+0000 7f099159a640 1 -- 172.21.15.16:0/1420799613 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09880a8e30 con 0x7f097c0047d0 2023-05-22T16:03:25.777 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.766+0000 7f099159a640 1 -- 172.21.15.16:0/1420799613 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09880058e0 con 0x7f097c0047d0 2023-05-22T16:03:25.777 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.766+0000 7f099159a640 1 -- 172.21.15.16:0/1420799613 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0988005c90 con 0x7f097c0047d0 2023-05-22T16:03:25.778 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.766+0000 7f0998b1c640 1 -- 172.21.15.16:0/1420799613 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f097c0047d0 msgr2=0x7f097c004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:25.778 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.766+0000 7f0998b1c640 1 --2- 172.21.15.16:0/1420799613 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f097c0047d0 0x7f097c004c40 secure :-1 s=READY pgs=2184 cs=0 l=1 rev1=1 crypto rx=0x7f099406f400 tx=0x7f098801f310 comp rx=0 tx=0).stop 2023-05-22T16:03:25.778 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.766+0000 7f0998b1c640 1 -- 172.21.15.16:0/1420799613 shutdown_connections 2023-05-22T16:03:25.778 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.766+0000 7f0998b1c640 1 --2- 172.21.15.16:0/1420799613 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f097c0047d0 0x7f097c004c40 unknown :-1 s=CLOSED pgs=2184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:25.778 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.766+0000 7f0998b1c640 1 -- 172.21.15.16:0/1420799613 >> 172.21.15.16:0/1420799613 conn(0x7f0994069720 msgr2=0x7f0994069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:03:25.779 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.766+0000 7f0998b1c640 1 -- 172.21.15.16:0/1420799613 shutdown_connections 2023-05-22T16:03:25.779 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.766+0000 7f0998b1c640 1 -- 172.21.15.16:0/1420799613 wait complete. 2023-05-22T16:03:25.779 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.770+0000 7f0998b1c640 1 Processor -- start 2023-05-22T16:03:25.779 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.770+0000 7f0998b1c640 1 -- start start 2023-05-22T16:03:25.779 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.770+0000 7f0998b1c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f097c0047d0 0x7f09941498a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:25.779 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.770+0000 7f0998b1c640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0994166a40 0x7f09940ce260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:25.779 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.770+0000 7f0998b1c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f09940caa30 0x7f09940caea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:25.779 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.770+0000 7f0998b1c640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0994071690 con 0x7f09940caa30 2023-05-22T16:03:25.780 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.770+0000 7f0998b1c640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0994070960 con 0x7f097c0047d0 2023-05-22T16:03:25.780 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.770+0000 7f0998b1c640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0994071200 con 0x7f0994166a40 2023-05-22T16:03:25.780 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.770+0000 7f0992d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f09940caa30 0x7f09940caea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:25.780 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.770+0000 7f0992d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f09940caa30 0x7f09940caea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55362/0 (socket says 172.21.15.16:55362) 2023-05-22T16:03:25.780 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.770+0000 7f0992d9d640 1 -- 172.21.15.16:0/1668528247 learned_addr learned my addr 172.21.15.16:0/1668528247 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:03:25.780 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.770+0000 7f0991d9b640 1 --2- 172.21.15.16:0/1668528247 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0994166a40 0x7f09940ce260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:25.780 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.770+0000 7f099259c640 1 --2- 172.21.15.16:0/1668528247 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f097c0047d0 0x7f09941498a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:25.781 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.770+0000 7f0992d9d640 1 -- 172.21.15.16:0/1668528247 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0994166a40 msgr2=0x7f09940ce260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:25.781 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.770+0000 7f0992d9d640 1 --2- 172.21.15.16:0/1668528247 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0994166a40 0x7f09940ce260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:25.781 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.770+0000 7f0992d9d640 1 -- 172.21.15.16:0/1668528247 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f097c0047d0 msgr2=0x7f09941498a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:25.781 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.770+0000 7f0992d9d640 1 --2- 172.21.15.16:0/1668528247 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f097c0047d0 0x7f09941498a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:25.781 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.770+0000 7f0992d9d640 1 -- 172.21.15.16:0/1668528247 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f099415cb00 con 0x7f09940caa30 2023-05-22T16:03:25.781 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.770+0000 7f0992d9d640 1 --2- 172.21.15.16:0/1668528247 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f09940caa30 0x7f09940caea0 secure :-1 s=READY pgs=2185 cs=0 l=1 rev1=1 crypto rx=0x7f099406d850 tx=0x7f098c007d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:03:25.781 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.770+0000 7f0982ffd640 1 -- 172.21.15.16:0/1668528247 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f098c00e070 con 0x7f09940caa30 2023-05-22T16:03:25.781 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.770+0000 7f0982ffd640 1 -- 172.21.15.16:0/1668528247 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f098c003c70 con 0x7f09940caa30 2023-05-22T16:03:25.782 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.770+0000 7f0998b1c640 1 -- 172.21.15.16:0/1668528247 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f099415c820 con 0x7f09940caa30 2023-05-22T16:03:25.782 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.770+0000 7f0982ffd640 1 -- 172.21.15.16:0/1668528247 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f098c003e00 con 0x7f09940caa30 2023-05-22T16:03:25.782 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.770+0000 7f0998b1c640 1 -- 172.21.15.16:0/1668528247 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f099415cde0 con 0x7f09940caa30 2023-05-22T16:03:25.782 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.774+0000 7f0998b1c640 1 -- 172.21.15.16:0/1668528247 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09940721a0 con 0x7f09940caa30 2023-05-22T16:03:25.782 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.774+0000 7f0982ffd640 1 -- 172.21.15.16:0/1668528247 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 81) v1 ==== 84068+0+0 (secure 0 0 0) 0x7f098c004030 con 0x7f09940caa30 2023-05-22T16:03:25.782 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.774+0000 7f0982ffd640 1 --2- 172.21.15.16:0/1668528247 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f0974065200 0x7f09740676b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:25.782 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.774+0000 7f0982ffd640 1 -- 172.21.15.16:0/1668528247 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(166..166 src has 1..166) v4 ==== 7441+0+0 (secure 0 0 0) 0x7f098c082a50 con 0x7f09940caa30 2023-05-22T16:03:25.783 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.774+0000 7f099259c640 1 --2- 172.21.15.16:0/1668528247 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f0974065200 0x7f09740676b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:25.783 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.774+0000 7f099259c640 1 --2- 172.21.15.16:0/1668528247 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f0974065200 0x7f09740676b0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f0984000c10 tx=0x7f0984009040 comp rx=0 tx=0).ready entity=mgr.10032 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:03:25.783 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.778+0000 7f0982ffd640 1 -- 172.21.15.16:0/1668528247 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f09940721a0 con 0x7f09940caa30 2023-05-22T16:03:25.967 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.962+0000 7f0998b1c640 1 -- 172.21.15.16:0/1668528247 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f09940721a0 con 0x7f09940caa30 2023-05-22T16:03:25.968 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.962+0000 7f0982ffd640 1 -- 172.21.15.16:0/1668528247 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) v1 ==== 94+0+4644 (secure 0 0 0) 0x7f09940721a0 con 0x7f09940caa30 2023-05-22T16:03:25.968 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T16:03:25.969 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":66,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8170,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6833/78835131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":78835131},{"type":"v1","addr":"172.21.15.62:6833","nonce":78835131}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":61},{"gid":8173,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/1188098562","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":1188098562},{"type":"v1","addr":"172.21.15.62:6835","nonce":1188098562}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":61},{"gid":9645,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/4204206994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":4204206994},{"type":"v1","addr":"172.21.15.16:6835","nonce":4204206994}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":61}],"filesystems":[{"mdsmap":{"epoch":66,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-05-22T15:47:03.009254+0000","modified":"2023-05-22T15:47:06.030064+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9654},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9654":{"gid":9654,"name":"c","rank":0,"incarnation":62,"state":"up:active","state_seq":3,"addr":"172.21.15.16:6837/1284822222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":1284822222},{"type":"v1","addr":"172.21.15.16:6837","nonce":1284822222}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2023-05-22T16:03:25.969 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 66 2023-05-22T16:03:25.970 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.966+0000 7f0998b1c640 1 -- 172.21.15.16:0/1668528247 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f0974065200 msgr2=0x7f09740676b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:25.970 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.966+0000 7f0998b1c640 1 --2- 172.21.15.16:0/1668528247 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f0974065200 0x7f09740676b0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f0984000c10 tx=0x7f0984009040 comp rx=0 tx=0).stop 2023-05-22T16:03:25.971 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.966+0000 7f0998b1c640 1 -- 172.21.15.16:0/1668528247 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f09940caa30 msgr2=0x7f09940caea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:25.971 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.966+0000 7f0998b1c640 1 --2- 172.21.15.16:0/1668528247 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f09940caa30 0x7f09940caea0 secure :-1 s=READY pgs=2185 cs=0 l=1 rev1=1 crypto rx=0x7f099406d850 tx=0x7f098c007d90 comp rx=0 tx=0).stop 2023-05-22T16:03:25.971 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.966+0000 7f0998b1c640 1 -- 172.21.15.16:0/1668528247 shutdown_connections 2023-05-22T16:03:25.971 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.966+0000 7f0998b1c640 1 --2- 172.21.15.16:0/1668528247 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f0974065200 0x7f09740676b0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:25.971 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.966+0000 7f0998b1c640 1 --2- 172.21.15.16:0/1668528247 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f09940caa30 0x7f09940caea0 unknown :-1 s=CLOSED pgs=2185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:25.972 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.966+0000 7f0998b1c640 1 --2- 172.21.15.16:0/1668528247 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0994166a40 0x7f09940ce260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:25.972 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.966+0000 7f0998b1c640 1 --2- 172.21.15.16:0/1668528247 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f097c0047d0 0x7f09941498a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:25.972 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.966+0000 7f0998b1c640 1 -- 172.21.15.16:0/1668528247 >> 172.21.15.16:0/1668528247 conn(0x7f0994069720 msgr2=0x7f099414a600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:03:25.972 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.966+0000 7f0998b1c640 1 -- 172.21.15.16:0/1668528247 shutdown_connections 2023-05-22T16:03:25.972 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:25.966+0000 7f0998b1c640 1 -- 172.21.15.16:0/1668528247 wait complete. 2023-05-22T16:03:25.987 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-05-22T16:03:26.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:26.106+0000 7f9d1f552640 1 Processor -- start 2023-05-22T16:03:26.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:26.106+0000 7f9d1f552640 1 -- start start 2023-05-22T16:03:26.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:26.110+0000 7f9d1f552640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f9d180ca4e0 0x7f9d1815e460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:26.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:26.110+0000 7f9d1f552640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f9d180c9b40 0x7f9d180afb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:26.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:26.110+0000 7f9d1f552640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9d18070df0 0x7f9d18071210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:26.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:26.110+0000 7f9d1f552640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d180b0650 con 0x7f9d180ca4e0 2023-05-22T16:03:26.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:26.110+0000 7f9d1f552640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d180b0370 con 0x7f9d180c9b40 2023-05-22T16:03:26.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:26.110+0000 7f9d1f552640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d1815d980 con 0x7f9d18070df0 2023-05-22T16:03:26.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:26.110+0000 7f9d1d2ee640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f9d180ca4e0 0x7f9d1815e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:26.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:26.110+0000 7f9d1d2ee640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f9d180ca4e0 0x7f9d1815e460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55374/0 (socket says 172.21.15.16:55374) 2023-05-22T16:03:26.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:26.110+0000 7f9d1d2ee640 1 -- 172.21.15.16:0/820683268 learned_addr learned my addr 172.21.15.16:0/820683268 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:03:26.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:26.110+0000 7f9d1daef640 1 --2- 172.21.15.16:0/820683268 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9d18070df0 0x7f9d18071210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:26.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:26.110+0000 7f9d1caed640 1 --2- 172.21.15.16:0/820683268 >> v2:172.21.15.62:3300/0 conn(0x7f9d180c9b40 0x7f9d180afb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:26.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:26.110+0000 7f9d1d2ee640 1 -- 172.21.15.16:0/820683268 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9d18070df0 msgr2=0x7f9d18071210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:03:26.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:26.110+0000 7f9d1d2ee640 1 --2- 172.21.15.16:0/820683268 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9d18070df0 0x7f9d18071210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:26.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:26.110+0000 7f9d1d2ee640 1 -- 172.21.15.16:0/820683268 >> v2:172.21.15.62:3300/0 conn(0x7f9d180c9b40 msgr2=0x7f9d180afb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:03:26.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:26.110+0000 7f9d1d2ee640 1 --2- 172.21.15.16:0/820683268 >> v2:172.21.15.62:3300/0 conn(0x7f9d180c9b40 0x7f9d180afb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:26.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:26.110+0000 7f9d1d2ee640 1 -- 172.21.15.16:0/820683268 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d1815dc60 con 0x7f9d180ca4e0 2023-05-22T16:03:26.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:26.110+0000 7f9d1caed640 1 --2- 172.21.15.16:0/820683268 >> v2:172.21.15.62:3300/0 conn(0x7f9d180c9b40 0x7f9d180afb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T16:03:26.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:26.110+0000 7f9d1d2ee640 1 --2- 172.21.15.16:0/820683268 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9d180ca4e0 0x7f9d1815e460 secure :-1 s=READY pgs=2186 cs=0 l=1 rev1=1 crypto rx=0x7f9d1806f0e0 tx=0x7f9d1401c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ab5411ef8b80d02 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:03:26.118 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:26.114+0000 7f9d0ffff640 1 -- 172.21.15.16:0/820683268 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d140ab670 con 0x7f9d180ca4e0 2023-05-22T16:03:26.118 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:26.114+0000 7f9d0ffff640 1 -- 172.21.15.16:0/820683268 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d140ab7d0 con 0x7f9d180ca4e0 2023-05-22T16:03:26.118 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:26.114+0000 7f9d0ffff640 1 -- 172.21.15.16:0/820683268 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d140abb60 con 0x7f9d180ca4e0 2023-05-22T16:03:26.118 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:26.114+0000 7f9d1f552640 1 -- 172.21.15.16:0/820683268 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9d180ca4e0 msgr2=0x7f9d1815e460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:26.118 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:26.114+0000 7f9d1f552640 1 --2- 172.21.15.16:0/820683268 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9d180ca4e0 0x7f9d1815e460 secure :-1 s=READY pgs=2186 cs=0 l=1 rev1=1 crypto rx=0x7f9d1806f0e0 tx=0x7f9d1401c4d0 comp rx=0 tx=0).stop 2023-05-22T16:03:26.118 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:26.114+0000 7f9d1f552640 1 -- 172.21.15.16:0/820683268 shutdown_connections 2023-05-22T16:03:26.119 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:26.114+0000 7f9d1f552640 1 --2- 172.21.15.16:0/820683268 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9d18070df0 0x7f9d18071210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:26.119 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:26.114+0000 7f9d1f552640 1 --2- 172.21.15.16:0/820683268 >> v2:172.21.15.62:3300/0 conn(0x7f9d180c9b40 0x7f9d180afb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:26.119 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:26.114+0000 7f9d1f552640 1 --2- 172.21.15.16:0/820683268 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9d180ca4e0 0x7f9d1815e460 unknown :-1 s=CLOSED pgs=2186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:26.119 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:26.114+0000 7f9d1f552640 1 -- 172.21.15.16:0/820683268 >> 172.21.15.16:0/820683268 conn(0x7f9d18069720 msgr2=0x7f9d18069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:03:26.119 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:26.114+0000 7f9d1f552640 1 -- 172.21.15.16:0/820683268 shutdown_connections 2023-05-22T16:03:26.119 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:26.114+0000 7f9d1f552640 1 -- 172.21.15.16:0/820683268 wait complete. 2023-05-22T16:03:26.122 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:26.114+0000 7f9d1f552640 1 Processor -- start 2023-05-22T16:03:26.122 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:26.114+0000 7f9d1f552640 1 -- start start 2023-05-22T16:03:26.122 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:26.114+0000 7f9d1f552640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9d18070df0 0x7f9d181501f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:26.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:26.114+0000 7f9d1f552640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9d180c9b40 0x7f9d18156cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:26.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:26.114+0000 7f9d1f552640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9d180ca4e0 0x7f9d18157210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:26.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:26.114+0000 7f9d1f552640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d1814fd60 con 0x7f9d180ca4e0 2023-05-22T16:03:26.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:26.114+0000 7f9d1f552640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d1814f750 con 0x7f9d180c9b40 2023-05-22T16:03:26.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:26.114+0000 7f9d1f552640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d181550f0 con 0x7f9d18070df0 2023-05-22T16:03:26.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:26.114+0000 7f9d1daef640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9d180ca4e0 0x7f9d18157210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:26.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:26.114+0000 7f9d1daef640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9d180ca4e0 0x7f9d18157210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55382/0 (socket says 172.21.15.16:55382) 2023-05-22T16:03:26.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:26.114+0000 7f9d1daef640 1 -- 172.21.15.16:0/1344329403 learned_addr learned my addr 172.21.15.16:0/1344329403 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:03:26.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:26.114+0000 7f9d1caed640 1 --2- 172.21.15.16:0/1344329403 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9d180c9b40 0x7f9d18156cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:26.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:26.114+0000 7f9d1d2ee640 1 --2- 172.21.15.16:0/1344329403 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9d18070df0 0x7f9d181501f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:26.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:26.114+0000 7f9d1daef640 1 -- 172.21.15.16:0/1344329403 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9d18070df0 msgr2=0x7f9d181501f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:26.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:26.114+0000 7f9d1daef640 1 --2- 172.21.15.16:0/1344329403 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9d18070df0 0x7f9d181501f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:26.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:26.114+0000 7f9d1daef640 1 -- 172.21.15.16:0/1344329403 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9d180c9b40 msgr2=0x7f9d18156cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:26.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:26.114+0000 7f9d1daef640 1 --2- 172.21.15.16:0/1344329403 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9d180c9b40 0x7f9d18156cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:26.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:26.114+0000 7f9d1daef640 1 -- 172.21.15.16:0/1344329403 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d140094d0 con 0x7f9d180ca4e0 2023-05-22T16:03:26.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:26.114+0000 7f9d1daef640 1 --2- 172.21.15.16:0/1344329403 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9d180ca4e0 0x7f9d18157210 secure :-1 s=READY pgs=2187 cs=0 l=1 rev1=1 crypto rx=0x7f9d10007e60 tx=0x7f9d1000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:03:26.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:26.118+0000 7f9d0dffb640 1 -- 172.21.15.16:0/1344329403 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d10014620 con 0x7f9d180ca4e0 2023-05-22T16:03:26.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:26.118+0000 7f9d0dffb640 1 -- 172.21.15.16:0/1344329403 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d1000ee90 con 0x7f9d180ca4e0 2023-05-22T16:03:26.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:26.118+0000 7f9d1f552640 1 -- 172.21.15.16:0/1344329403 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d180ca8b0 con 0x7f9d180ca4e0 2023-05-22T16:03:26.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:26.118+0000 7f9d0dffb640 1 -- 172.21.15.16:0/1344329403 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d100149e0 con 0x7f9d180ca4e0 2023-05-22T16:03:26.126 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:26.118+0000 7f9d1f552640 1 -- 172.21.15.16:0/1344329403 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d18071750 con 0x7f9d180ca4e0 2023-05-22T16:03:26.126 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:26.118+0000 7f9d1f552640 1 -- 172.21.15.16:0/1344329403 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d180c8e80 con 0x7f9d180ca4e0 2023-05-22T16:03:26.127 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:26.118+0000 7f9d0dffb640 1 -- 172.21.15.16:0/1344329403 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 81) v1 ==== 84068+0+0 (secure 0 0 0) 0x7f9d10014b40 con 0x7f9d180ca4e0 2023-05-22T16:03:26.127 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:26.118+0000 7f9d0dffb640 1 --2- 172.21.15.16:0/1344329403 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f9cfc0651d0 0x7f9cfc067680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:26.127 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:26.118+0000 7f9d1d2ee640 1 --2- 172.21.15.16:0/1344329403 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f9cfc0651d0 0x7f9cfc067680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:26.127 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:26.122+0000 7f9d1d2ee640 1 --2- 172.21.15.16:0/1344329403 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f9cfc0651d0 0x7f9cfc067680 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f9d140af000 tx=0x7f9d140a7000 comp rx=0 tx=0).ready entity=mgr.10032 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:03:26.127 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:26.122+0000 7f9d0dffb640 1 -- 172.21.15.16:0/1344329403 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(167..167 src has 1..167) v4 ==== 7441+0+0 (secure 0 0 0) 0x7f9d1005ca70 con 0x7f9d180ca4e0 2023-05-22T16:03:26.131 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:26.126+0000 7f9d0dffb640 1 -- 172.21.15.16:0/1344329403 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f9d10050d20 con 0x7f9d180ca4e0 2023-05-22T16:03:26.286 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:26.282+0000 7f9d1f552640 1 -- 172.21.15.16:0/1344329403 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9d1804dbe0 con 0x7f9d180ca4e0 2023-05-22T16:03:26.289 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:26.286+0000 7f9d0dffb640 1 -- 172.21.15.16:0/1344329403 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v167) v1 ==== 74+0+19609 (secure 0 0 0) 0x7f9d10054340 con 0x7f9d180ca4e0 2023-05-22T16:03:26.290 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T16:03:26.290 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":167,"fsid":"b2719834-7847-4d98-90a6-f65d57f85c49","created":"2023-05-22T15:26:25.259314+0000","modified":"2023-05-22T16:03:26.122573+0000","last_up_change":"2023-05-22T15:26:36.448524+0000","last_in_change":"2023-05-22T15:26:27.675668+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-05-22T15:26:38.019781+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-05-22T15:26:43.570514+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_metadata","create_time":"2023-05-22T15:47:01.347542+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"160","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_data","create_time":"2023-05-22T15:47:01.970646+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"166","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":33,"snap_epoch":166,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9e7ff854-b25f-4cbb-a5c2-1799d9c53d2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":37021},{"type":"v1","addr":"172.21.15.16:6825","nonce":37021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":37021},{"type":"v1","addr":"172.21.15.16:6827","nonce":37021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":37021},{"type":"v1","addr":"172.21.15.16:6831","nonce":37021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":37021},{"type":"v1","addr":"172.21.15.16:6829","nonce":37021}]},"public_addr":"172.21.15.16:6825/37021","cluster_addr":"172.21.15.16:6827/37021","heartbeat_back_addr":"172.21.15.16:6831/37021","heartbeat_front_addr":"172.21.15.16:6829/37021","state":["exists","up"]},{"osd":1,"uuid":"4702fbc9-a927-4565-aca5-41dc69cc8264","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":37008},{"type":"v1","addr":"172.21.15.16:6801","nonce":37008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":37008},{"type":"v1","addr":"172.21.15.16:6803","nonce":37008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":37008},{"type":"v1","addr":"172.21.15.16:6807","nonce":37008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":37008},{"type":"v1","addr":"172.21.15.16:6805","nonce":37008}]},"public_addr":"172.21.15.16:6801/37008","cluster_addr":"172.21.15.16:6803/37008","heartbeat_back_addr":"172.21.15.16:6807/37008","heartbeat_front_addr":"172.21.15.16:6805/37008","state":["exists","up"]},{"osd":2,"uuid":"1e7bba46-4eba-47ef-bb81-5e2d371638ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":37009},{"type":"v1","addr":"172.21.15.16:6809","nonce":37009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":37009},{"type":"v1","addr":"172.21.15.16:6811","nonce":37009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":37009},{"type":"v1","addr":"172.21.15.16:6815","nonce":37009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":37009},{"type":"v1","addr":"172.21.15.16:6813","nonce":37009}]},"public_addr":"172.21.15.16:6809/37009","cluster_addr":"172.21.15.16:6811/37009","heartbeat_back_addr":"172.21.15.16:6815/37009","heartbeat_front_addr":"172.21.15.16:6813/37009","state":["exists","up"]},{"osd":3,"uuid":"383cacc0-5a88-4904-990e-498ca59ef48a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":37020},{"type":"v1","addr":"172.21.15.16:6817","nonce":37020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":37020},{"type":"v1","addr":"172.21.15.16:6819","nonce":37020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":37020},{"type":"v1","addr":"172.21.15.16:6823","nonce":37020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":37020},{"type":"v1","addr":"172.21.15.16:6821","nonce":37020}]},"public_addr":"172.21.15.16:6817/37020","cluster_addr":"172.21.15.16:6819/37020","heartbeat_back_addr":"172.21.15.16:6823/37020","heartbeat_front_addr":"172.21.15.16:6821/37020","state":["exists","up"]},{"osd":4,"uuid":"970500ef-95b7-42c1-b080-eed4fae7013d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":37353},{"type":"v1","addr":"172.21.15.62:6817","nonce":37353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6819","nonce":37353},{"type":"v1","addr":"172.21.15.62:6822","nonce":37353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37353},{"type":"v1","addr":"172.21.15.62:6831","nonce":37353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37353},{"type":"v1","addr":"172.21.15.62:6826","nonce":37353}]},"public_addr":"172.21.15.62:6817/37353","cluster_addr":"172.21.15.62:6822/37353","heartbeat_back_addr":"172.21.15.62:6831/37353","heartbeat_front_addr":"172.21.15.62:6826/37353","state":["exists","up"]},{"osd":5,"uuid":"022d67db-d4d8-445b-b1b9-8b446a00f5c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":37352},{"type":"v1","addr":"172.21.15.62:6801","nonce":37352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":37352},{"type":"v1","addr":"172.21.15.62:6803","nonce":37352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":37352},{"type":"v1","addr":"172.21.15.62:6807","nonce":37352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":37352},{"type":"v1","addr":"172.21.15.62:6805","nonce":37352}]},"public_addr":"172.21.15.62:6801/37352","cluster_addr":"172.21.15.62:6803/37352","heartbeat_back_addr":"172.21.15.62:6807/37352","heartbeat_front_addr":"172.21.15.62:6805/37352","state":["exists","up"]},{"osd":6,"uuid":"42bd372b-6688-4a1b-8407-e16e8c266e33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":37354},{"type":"v1","addr":"172.21.15.62:6820","nonce":37354}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":37354},{"type":"v1","addr":"172.21.15.62:6823","nonce":37354}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6829","nonce":37354},{"type":"v1","addr":"172.21.15.62:6830","nonce":37354}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6825","nonce":37354},{"type":"v1","addr":"172.21.15.62:6827","nonce":37354}]},"public_addr":"172.21.15.62:6820/37354","cluster_addr":"172.21.15.62:6823/37354","heartbeat_back_addr":"172.21.15.62:6830/37354","heartbeat_front_addr":"172.21.15.62:6827/37354","state":["exists","up"]},{"osd":7,"uuid":"ac70b2a3-f41b-4bf2-b227-7f2cbb55426d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":163,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37355},{"type":"v1","addr":"172.21.15.62:6809","nonce":37355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37355},{"type":"v1","addr":"172.21.15.62:6811","nonce":37355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":37355},{"type":"v1","addr":"172.21.15.62:6815","nonce":37355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":37355},{"type":"v1","addr":"172.21.15.62:6813","nonce":37355}]},"public_addr":"172.21.15.62:6809/37355","cluster_addr":"172.21.15.62:6811/37355","heartbeat_back_addr":"172.21.15.62:6815/37355","heartbeat_front_addr":"172.21.15.62:6813/37355","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.368896+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.107273+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.276888+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.287502+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.124791+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.251937+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.255679+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.232797+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"21.28","mappings":[{"from":4,"to":3}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:0/313468131":"2023-05-23T16:03:23.703312+0000","172.21.15.16:0/2645427422":"2023-05-23T16:03:23.703312+0000","172.21.15.16:6833/65890":"2023-05-23T16:03:23.703312+0000","172.21.15.16:6832/65890":"2023-05-23T16:03:23.703312+0000","172.21.15.16:0/1654177733":"2023-05-23T16:03:23.703312+0000","172.21.15.16:0/2019135485":"2023-05-23T15:58:20.479559+0000","172.21.15.16:0/3082249167":"2023-05-23T15:58:20.479559+0000","172.21.15.16:0/4113149961":"2023-05-23T15:58:20.479559+0000","172.21.15.16:0/631429095":"2023-05-23T16:03:23.703312+0000","172.21.15.16:0/2582627335":"2023-05-23T15:58:20.479559+0000","172.21.15.16:0/2722168808":"2023-05-23T15:47:16.536062+0000","172.21.15.16:0/1232860864":"2023-05-23T15:47:16.536062+0000","172.21.15.16:0/3766829209":"2023-05-23T15:47:16.536062+0000","172.21.15.16:0/3369591631":"2023-05-23T15:47:16.536062+0000","172.21.15.16:0/3417178745":"2023-05-23T15:58:20.479559+0000","172.21.15.16:6833/36921":"2023-05-23T15:47:16.536062+0000","172.21.15.16:0/392276665":"2023-05-23T16:03:23.703312+0000","172.21.15.16:0/2592954608":"2023-05-23T15:58:20.479559+0000","172.21.15.16:0/49958236":"2023-05-23T15:47:16.536062+0000","172.21.15.16:6832/36921":"2023-05-23T15:47:16.536062+0000","172.21.15.16:0/1747179873":"2023-05-23T16:03:23.703312+0000","172.21.15.16:0/4161645610":"2023-05-23T15:47:16.536062+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":17,"snaps":[{"begin":2,"length":2}]},{"pool":21,"snaps":[{"begin":25,"length":9}]}],"new_removed_snaps":[],"new_purged_snaps":[{"pool":21,"snaps":[{"begin":2,"length":23}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-05-22T16:03:26.291 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:26.286+0000 7f9d1f552640 1 -- 172.21.15.16:0/1344329403 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f9cfc0651d0 msgr2=0x7f9cfc067680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:26.292 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:26.286+0000 7f9d1f552640 1 --2- 172.21.15.16:0/1344329403 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f9cfc0651d0 0x7f9cfc067680 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f9d140af000 tx=0x7f9d140a7000 comp rx=0 tx=0).stop 2023-05-22T16:03:26.292 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:26.286+0000 7f9d1f552640 1 -- 172.21.15.16:0/1344329403 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9d180ca4e0 msgr2=0x7f9d18157210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:26.292 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:26.286+0000 7f9d1f552640 1 --2- 172.21.15.16:0/1344329403 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9d180ca4e0 0x7f9d18157210 secure :-1 s=READY pgs=2187 cs=0 l=1 rev1=1 crypto rx=0x7f9d10007e60 tx=0x7f9d1000e4b0 comp rx=0 tx=0).stop 2023-05-22T16:03:26.292 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:26.286+0000 7f9d1f552640 1 -- 172.21.15.16:0/1344329403 shutdown_connections 2023-05-22T16:03:26.292 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:26.286+0000 7f9d1f552640 1 --2- 172.21.15.16:0/1344329403 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f9cfc0651d0 0x7f9cfc067680 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:26.293 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:26.286+0000 7f9d1f552640 1 --2- 172.21.15.16:0/1344329403 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9d180ca4e0 0x7f9d18157210 unknown :-1 s=CLOSED pgs=2187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:26.293 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:26.286+0000 7f9d1f552640 1 --2- 172.21.15.16:0/1344329403 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9d180c9b40 0x7f9d18156cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:26.293 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:26.286+0000 7f9d1f552640 1 --2- 172.21.15.16:0/1344329403 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9d18070df0 0x7f9d181501f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:26.293 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:26.286+0000 7f9d1f552640 1 -- 172.21.15.16:0/1344329403 >> 172.21.15.16:0/1344329403 conn(0x7f9d18069720 msgr2=0x7f9d180b0eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:03:26.293 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:26.286+0000 7f9d1f552640 1 -- 172.21.15.16:0/1344329403 shutdown_connections 2023-05-22T16:03:26.293 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:26.286+0000 7f9d1f552640 1 -- 172.21.15.16:0/1344329403 wait complete. 2023-05-22T16:03:26.309 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-05-22T16:03:26.426 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:26.422+0000 7fba30559640 1 Processor -- start 2023-05-22T16:03:26.426 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:26.422+0000 7fba30559640 1 -- start start 2023-05-22T16:03:26.429 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:26.422+0000 7fba30559640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fba280ca4e0 0x7fba2815e460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:26.429 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:26.422+0000 7fba30559640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fba28070df0 0x7fba28071210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:26.430 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:26.422+0000 7fba30559640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba280b0650 con 0x7fba280c9b40 2023-05-22T16:03:26.430 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:26.422+0000 7fba30559640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba280b0370 con 0x7fba28070df0 2023-05-22T16:03:26.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:26.422+0000 7fba30559640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba2815d980 con 0x7fba280ca4e0 2023-05-22T16:03:26.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:26.422+0000 7fba2e2f5640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fba280ca4e0 0x7fba2815e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:26.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:26.422+0000 7fba2eaf6640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fba28070df0 0x7fba28071210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:26.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:26.422+0000 7fba2daf4640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7fba280c9b40 0x7fba280afb40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.16:49782/0 (socket says 172.21.15.16:49782) 2023-05-22T16:03:26.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:26.422+0000 7fba2daf4640 1 -- 172.21.15.16:0/1418935678 learned_addr learned my addr 172.21.15.16:0/1418935678 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T16:03:26.432 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:26.422+0000 7fba2e2f5640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fba280ca4e0 0x7fba2815e460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3301/0 says I am v2:172.21.15.16:46820/0 (socket says 172.21.15.16:46820) 2023-05-22T16:03:26.432 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:26.422+0000 7fba2d2f3640 1 -- 172.21.15.16:0/1418935678 <== mon.1 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3055053483 0 0) 0x7fba280b0650 con 0x7fba280c9b40 2023-05-22T16:03:26.432 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:26.422+0000 7fba2d2f3640 1 -- 172.21.15.16:0/1418935678 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba14003530 con 0x7fba280c9b40 2023-05-22T16:03:26.432 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:26.422+0000 7fba2e2f5640 1 -- 172.21.15.16:0/1418935678 >> v2:172.21.15.62:3300/0 conn(0x7fba28070df0 msgr2=0x7fba28071210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:03:26.432 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:26.422+0000 7fba2e2f5640 1 --2- 172.21.15.16:0/1418935678 >> v2:172.21.15.62:3300/0 conn(0x7fba28070df0 0x7fba28071210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:26.432 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:26.422+0000 7fba2e2f5640 1 -- 172.21.15.16:0/1418935678 >> v1:172.21.15.62:6789/0 conn(0x7fba280c9b40 legacy=0x7fba280afb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:26.432 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:26.422+0000 7fba2e2f5640 1 -- 172.21.15.16:0/1418935678 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba2815dc60 con 0x7fba280ca4e0 2023-05-22T16:03:26.433 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:26.422+0000 7fba2eaf6640 1 --2- 172.21.15.16:0/1418935678 >> v2:172.21.15.62:3300/0 conn(0x7fba28070df0 0x7fba28071210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-05-22T16:03:26.436 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:26.426+0000 7fba2e2f5640 1 --2- 172.21.15.16:0/1418935678 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fba280ca4e0 0x7fba2815e460 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fba240097c0 tx=0x7fba2401c3c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eb33be51f75cbd19 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:03:26.436 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:26.426+0000 7fba2d2f3640 1 -- 172.21.15.16:0/1418935678 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba240ab600 con 0x7fba280ca4e0 2023-05-22T16:03:26.436 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:26.426+0000 7fba2d2f3640 1 -- 172.21.15.16:0/1418935678 <== mon.2 v2:172.21.15.62:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba240a7050 con 0x7fba280ca4e0 2023-05-22T16:03:26.437 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:26.426+0000 7fba2d2f3640 1 -- 172.21.15.16:0/1418935678 <== mon.2 v2:172.21.15.62:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba240ab960 con 0x7fba280ca4e0 2023-05-22T16:03:26.437 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:26.426+0000 7fba30559640 1 -- 172.21.15.16:0/1418935678 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fba280ca4e0 msgr2=0x7fba2815e460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:26.437 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:26.426+0000 7fba30559640 1 --2- 172.21.15.16:0/1418935678 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fba280ca4e0 0x7fba2815e460 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fba240097c0 tx=0x7fba2401c3c0 comp rx=0 tx=0).stop 2023-05-22T16:03:26.437 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:26.426+0000 7fba30559640 1 -- 172.21.15.16:0/1418935678 shutdown_connections 2023-05-22T16:03:26.437 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:26.426+0000 7fba30559640 1 --2- 172.21.15.16:0/1418935678 >> v2:172.21.15.62:3300/0 conn(0x7fba28070df0 0x7fba28071210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:26.437 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:26.426+0000 7fba30559640 1 --2- 172.21.15.16:0/1418935678 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fba280ca4e0 0x7fba2815e460 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:26.437 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:26.426+0000 7fba30559640 1 -- 172.21.15.16:0/1418935678 >> 172.21.15.16:0/1418935678 conn(0x7fba28069720 msgr2=0x7fba28069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:03:26.438 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:26.426+0000 7fba30559640 1 -- 172.21.15.16:0/1418935678 shutdown_connections 2023-05-22T16:03:26.438 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:26.426+0000 7fba30559640 1 -- 172.21.15.16:0/1418935678 wait complete. 2023-05-22T16:03:26.438 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:26.430+0000 7fba30559640 1 Processor -- start 2023-05-22T16:03:26.438 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:26.430+0000 7fba30559640 1 -- start start 2023-05-22T16:03:26.438 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:26.430+0000 7fba30559640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fba28070df0 0x7fba281512d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:26.438 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:26.430+0000 7fba30559640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fba280c9b40 0x7fba281f7d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:26.439 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:26.430+0000 7fba30559640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fba280ca4e0 0x7fba281f82d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:26.439 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:26.430+0000 7fba30559640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba280cd850 con 0x7fba280c9b40 2023-05-22T16:03:26.439 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:26.430+0000 7fba30559640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba280cc7b0 con 0x7fba28070df0 2023-05-22T16:03:26.439 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:26.430+0000 7fba30559640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba280cca90 con 0x7fba280ca4e0 2023-05-22T16:03:26.440 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:26.430+0000 7fba2daf4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fba280c9b40 0x7fba281f7d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:26.440 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:26.430+0000 7fba2daf4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fba280c9b40 0x7fba281f7d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55394/0 (socket says 172.21.15.16:55394) 2023-05-22T16:03:26.440 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:26.430+0000 7fba2daf4640 1 -- 172.21.15.16:0/2368584312 learned_addr learned my addr 172.21.15.16:0/2368584312 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:03:26.440 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:26.430+0000 7fba2eaf6640 1 --2- 172.21.15.16:0/2368584312 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fba280ca4e0 0x7fba281f82d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:26.440 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:26.430+0000 7fba2e2f5640 1 --2- 172.21.15.16:0/2368584312 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fba28070df0 0x7fba281512d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:26.441 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:26.430+0000 7fba2daf4640 1 -- 172.21.15.16:0/2368584312 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fba280ca4e0 msgr2=0x7fba281f82d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:26.441 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:26.430+0000 7fba2daf4640 1 --2- 172.21.15.16:0/2368584312 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fba280ca4e0 0x7fba281f82d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:26.441 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:26.430+0000 7fba2daf4640 1 -- 172.21.15.16:0/2368584312 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fba28070df0 msgr2=0x7fba281512d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:26.441 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:26.430+0000 7fba2daf4640 1 --2- 172.21.15.16:0/2368584312 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fba28070df0 0x7fba281512d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:26.441 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:26.430+0000 7fba2daf4640 1 -- 172.21.15.16:0/2368584312 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba20009a70 con 0x7fba280c9b40 2023-05-22T16:03:26.441 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:26.430+0000 7fba2eaf6640 1 --2- 172.21.15.16:0/2368584312 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fba280ca4e0 0x7fba281f82d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T16:03:26.441 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:26.430+0000 7fba2daf4640 1 --2- 172.21.15.16:0/2368584312 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fba280c9b40 0x7fba281f7d90 secure :-1 s=READY pgs=2188 cs=0 l=1 rev1=1 crypto rx=0x7fba1800c450 tx=0x7fba1800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:03:26.441 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:26.430+0000 7fba1effd640 1 -- 172.21.15.16:0/2368584312 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba18005cf0 con 0x7fba280c9b40 2023-05-22T16:03:26.442 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:26.430+0000 7fba1effd640 1 -- 172.21.15.16:0/2368584312 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba18005e50 con 0x7fba280c9b40 2023-05-22T16:03:26.442 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:26.430+0000 7fba1effd640 1 -- 172.21.15.16:0/2368584312 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba18007290 con 0x7fba280c9b40 2023-05-22T16:03:26.442 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:26.430+0000 7fba30559640 1 -- 172.21.15.16:0/2368584312 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba24009420 con 0x7fba280c9b40 2023-05-22T16:03:26.442 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:26.430+0000 7fba30559640 1 -- 172.21.15.16:0/2368584312 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba280c8e80 con 0x7fba280c9b40 2023-05-22T16:03:26.442 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:26.430+0000 7fba30559640 1 -- 172.21.15.16:0/2368584312 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba280ca8b0 con 0x7fba280c9b40 2023-05-22T16:03:26.442 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:26.430+0000 7fba1effd640 1 -- 172.21.15.16:0/2368584312 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 81) v1 ==== 84068+0+0 (secure 0 0 0) 0x7fba180073f0 con 0x7fba280c9b40 2023-05-22T16:03:26.442 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:26.434+0000 7fba1effd640 1 --2- 172.21.15.16:0/2368584312 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fba0c0651d0 0x7fba0c067680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:26.443 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:26.434+0000 7fba1effd640 1 -- 172.21.15.16:0/2368584312 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(167..167 src has 1..167) v4 ==== 7441+0+0 (secure 0 0 0) 0x7fba18081fe0 con 0x7fba280c9b40 2023-05-22T16:03:26.443 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:26.434+0000 7fba2e2f5640 1 --2- 172.21.15.16:0/2368584312 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fba0c0651d0 0x7fba0c067680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:26.443 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:26.434+0000 7fba2e2f5640 1 --2- 172.21.15.16:0/2368584312 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fba0c0651d0 0x7fba0c067680 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fba240ad000 tx=0x7fba240a6000 comp rx=0 tx=0).ready entity=mgr.10032 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:03:26.444 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:26.438+0000 7fba1effd640 1 -- 172.21.15.16:0/2368584312 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fba1804fa10 con 0x7fba280c9b40 2023-05-22T16:03:26.627 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:26.622+0000 7fba30559640 1 -- 172.21.15.16:0/2368584312 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fba28072160 con 0x7fba280c9b40 2023-05-22T16:03:26.741 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:26.738+0000 7fba1effd640 1 -- 172.21.15.16:0/2368584312 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mgrmap(e 82) v1 ==== 84332+0+0 (secure 0 0 0) 0x7fba1804fde0 con 0x7fba280c9b40 2023-05-22T16:03:27.177 INFO:tasks.ceph.mon.a.smithi016.stderr:2023-05-22T16:03:27.174+0000 7f95dd48a640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-05-22T16:03:27.187 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:27.182+0000 7fba1effd640 1 -- 172.21.15.16:0/2368584312 <== mon.0 v2:172.21.15.16:3300/0 8 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v67) v1 ==== 161+0+0 (secure 0 0 0) 0x7fba1804fa10 con 0x7fba280c9b40 2023-05-22T16:03:27.187 INFO:teuthology.orchestra.run.smithi016.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-05-22T16:03:27.190 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:27.186+0000 7fba30559640 1 -- 172.21.15.16:0/2368584312 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fba0c0651d0 msgr2=0x7fba0c067680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:27.190 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:27.186+0000 7fba30559640 1 --2- 172.21.15.16:0/2368584312 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fba0c0651d0 0x7fba0c067680 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fba240ad000 tx=0x7fba240a6000 comp rx=0 tx=0).stop 2023-05-22T16:03:27.190 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:27.186+0000 7fba30559640 1 -- 172.21.15.16:0/2368584312 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fba280c9b40 msgr2=0x7fba281f7d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:27.190 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:27.186+0000 7fba30559640 1 --2- 172.21.15.16:0/2368584312 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fba280c9b40 0x7fba281f7d90 secure :-1 s=READY pgs=2188 cs=0 l=1 rev1=1 crypto rx=0x7fba1800c450 tx=0x7fba1800c910 comp rx=0 tx=0).stop 2023-05-22T16:03:27.190 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:27.186+0000 7fba30559640 1 -- 172.21.15.16:0/2368584312 shutdown_connections 2023-05-22T16:03:27.190 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:27.186+0000 7fba30559640 1 --2- 172.21.15.16:0/2368584312 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fba280ca4e0 0x7fba281f82d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:27.190 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:27.186+0000 7fba30559640 1 --2- 172.21.15.16:0/2368584312 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fba0c0651d0 0x7fba0c067680 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:27.191 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:27.186+0000 7fba30559640 1 --2- 172.21.15.16:0/2368584312 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fba280c9b40 0x7fba281f7d90 unknown :-1 s=CLOSED pgs=2188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:27.191 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:27.186+0000 7fba30559640 1 --2- 172.21.15.16:0/2368584312 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fba28070df0 0x7fba281512d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:27.191 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:27.186+0000 7fba30559640 1 -- 172.21.15.16:0/2368584312 >> 172.21.15.16:0/2368584312 conn(0x7fba28069720 msgr2=0x7fba280b1830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:03:27.191 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:27.186+0000 7fba30559640 1 -- 172.21.15.16:0/2368584312 shutdown_connections 2023-05-22T16:03:27.191 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:27.186+0000 7fba30559640 1 -- 172.21.15.16:0/2368584312 wait complete. 2023-05-22T16:03:27.210 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-05-22T16:03:27.246 INFO:tasks.ceph.mds.c.smithi016.stdout:starting mds.c at 2023-05-22T16:03:27.350 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:27.346+0000 7fe8e713b640 1 Processor -- start 2023-05-22T16:03:27.351 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:27.346+0000 7fe8e713b640 1 -- start start 2023-05-22T16:03:27.352 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:27.346+0000 7fe8e713b640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe8e0159970 0x7fe8e0159d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:27.353 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:27.346+0000 7fe8e713b640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8e0070c30 con 0x7fe8e015d680 2023-05-22T16:03:27.353 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:27.346+0000 7fe8e713b640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8e0070e60 con 0x7fe8e015db00 2023-05-22T16:03:27.353 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:27.346+0000 7fe8e713b640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8e00cb970 con 0x7fe8e0159970 2023-05-22T16:03:27.354 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:27.346+0000 7fe8e4ed7640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe8e0159970 0x7fe8e0159d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:27.354 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:27.346+0000 7fe8dffff640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fe8e015d680 0x7fe8e00cb260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:41988/0 (socket says 172.21.15.16:41988) 2023-05-22T16:03:27.354 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:27.346+0000 7fe8dffff640 1 -- 172.21.15.16:0/1015585914 learned_addr learned my addr 172.21.15.16:0/1015585914 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T16:03:27.355 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:27.346+0000 7fe8e4ed7640 1 -- 172.21.15.16:0/1015585914 >> v1:172.21.15.62:6789/0 conn(0x7fe8e015db00 legacy=0x7fe8e006b6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:27.355 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:27.346+0000 7fe8e4ed7640 1 -- 172.21.15.16:0/1015585914 >> v1:172.21.15.16:6789/0 conn(0x7fe8e015d680 legacy=0x7fe8e00cb260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:27.355 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:27.346+0000 7fe8df7fe640 1 -- 172.21.15.16:0/1015585914 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3969786699 0 0) 0x7fe8e0070c30 con 0x7fe8e015d680 2023-05-22T16:03:27.355 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:27.346+0000 7fe8e4ed7640 1 -- 172.21.15.16:0/1015585914 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8e0071140 con 0x7fe8e0159970 2023-05-22T16:03:27.359 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:27.350+0000 7fe8e4ed7640 1 --2- 172.21.15.16:0/1015585914 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe8e0159970 0x7fe8e0159d40 secure :-1 s=READY pgs=2192 cs=0 l=1 rev1=1 crypto rx=0x7fe8d00097c0 tx=0x7fe8d001c3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5d2f6f2a35cd501 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:03:27.359 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:27.350+0000 7fe8df7fe640 1 -- 172.21.15.16:0/1015585914 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8d00ac620 con 0x7fe8e0159970 2023-05-22T16:03:27.359 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:27.350+0000 7fe8df7fe640 1 -- 172.21.15.16:0/1015585914 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8d00ac780 con 0x7fe8e0159970 2023-05-22T16:03:27.359 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:27.350+0000 7fe8df7fe640 1 -- 172.21.15.16:0/1015585914 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8d00acb10 con 0x7fe8e0159970 2023-05-22T16:03:27.360 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:27.350+0000 7fe8e713b640 1 -- 172.21.15.16:0/1015585914 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe8e0159970 msgr2=0x7fe8e0159d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:27.360 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:27.350+0000 7fe8e713b640 1 --2- 172.21.15.16:0/1015585914 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe8e0159970 0x7fe8e0159d40 secure :-1 s=READY pgs=2192 cs=0 l=1 rev1=1 crypto rx=0x7fe8d00097c0 tx=0x7fe8d001c3c0 comp rx=0 tx=0).stop 2023-05-22T16:03:27.360 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:27.350+0000 7fe8e713b640 1 -- 172.21.15.16:0/1015585914 shutdown_connections 2023-05-22T16:03:27.360 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:27.350+0000 7fe8e713b640 1 --2- 172.21.15.16:0/1015585914 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe8e0159970 0x7fe8e0159d40 unknown :-1 s=CLOSED pgs=2192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:27.360 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:27.350+0000 7fe8e713b640 1 -- 172.21.15.16:0/1015585914 >> 172.21.15.16:0/1015585914 conn(0x7fe8e0069720 msgr2=0x7fe8e0069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:03:27.361 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:27.350+0000 7fe8e713b640 1 -- 172.21.15.16:0/1015585914 shutdown_connections 2023-05-22T16:03:27.361 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:27.350+0000 7fe8e713b640 1 -- 172.21.15.16:0/1015585914 wait complete. 2023-05-22T16:03:27.361 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:27.354+0000 7fe8e713b640 1 Processor -- start 2023-05-22T16:03:27.361 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:27.354+0000 7fe8e713b640 1 -- start start 2023-05-22T16:03:27.361 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:27.354+0000 7fe8e713b640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe8e0159970 0x7fe8e015e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:27.361 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:27.354+0000 7fe8e713b640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe8e015d680 0x7fe8e00b0050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:27.362 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:27.354+0000 7fe8e713b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe8e015db00 0x7fe8e00b0590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:27.362 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:27.354+0000 7fe8e713b640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8e00b0ad0 con 0x7fe8e015db00 2023-05-22T16:03:27.362 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:27.354+0000 7fe8e713b640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8e00c6750 con 0x7fe8e0159970 2023-05-22T16:03:27.362 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:27.354+0000 7fe8e713b640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8e00c6190 con 0x7fe8e015d680 2023-05-22T16:03:27.362 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:27.354+0000 7fe8e56d8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe8e015db00 0x7fe8e00b0590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:27.362 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:27.354+0000 7fe8e56d8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe8e015db00 0x7fe8e00b0590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55430/0 (socket says 172.21.15.16:55430) 2023-05-22T16:03:27.363 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:27.354+0000 7fe8e56d8640 1 -- 172.21.15.16:0/2691863598 learned_addr learned my addr 172.21.15.16:0/2691863598 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:03:27.363 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:27.354+0000 7fe8dffff640 1 --2- 172.21.15.16:0/2691863598 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe8e015d680 0x7fe8e00b0050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:27.363 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:27.354+0000 7fe8e4ed7640 1 --2- 172.21.15.16:0/2691863598 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe8e0159970 0x7fe8e015e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:27.363 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:27.354+0000 7fe8e56d8640 1 -- 172.21.15.16:0/2691863598 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe8e015d680 msgr2=0x7fe8e00b0050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:27.363 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:27.354+0000 7fe8e56d8640 1 --2- 172.21.15.16:0/2691863598 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe8e015d680 0x7fe8e00b0050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:27.364 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:27.354+0000 7fe8e56d8640 1 -- 172.21.15.16:0/2691863598 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe8e0159970 msgr2=0x7fe8e015e7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:27.364 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:27.354+0000 7fe8e56d8640 1 --2- 172.21.15.16:0/2691863598 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe8e0159970 0x7fe8e015e7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:27.364 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:27.354+0000 7fe8e56d8640 1 -- 172.21.15.16:0/2691863598 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8d0009420 con 0x7fe8e015db00 2023-05-22T16:03:27.364 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:27.354+0000 7fe8e4ed7640 1 --2- 172.21.15.16:0/2691863598 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe8e0159970 0x7fe8e015e7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T16:03:27.364 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:27.354+0000 7fe8e56d8640 1 --2- 172.21.15.16:0/2691863598 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe8e015db00 0x7fe8e00b0590 secure :-1 s=READY pgs=2193 cs=0 l=1 rev1=1 crypto rx=0x7fe8d800c730 tx=0x7fe8d800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:03:27.365 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:27.354+0000 7fe8dd7fa640 1 -- 172.21.15.16:0/2691863598 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8d8003b80 con 0x7fe8e015db00 2023-05-22T16:03:27.365 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:27.354+0000 7fe8dd7fa640 1 -- 172.21.15.16:0/2691863598 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8d8003ce0 con 0x7fe8e015db00 2023-05-22T16:03:27.365 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:27.354+0000 7fe8dd7fa640 1 -- 172.21.15.16:0/2691863598 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8d800aa30 con 0x7fe8e015db00 2023-05-22T16:03:27.365 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:27.354+0000 7fe8e713b640 1 -- 172.21.15.16:0/2691863598 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8e00cae20 con 0x7fe8e015db00 2023-05-22T16:03:27.365 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:27.354+0000 7fe8e713b640 1 -- 172.21.15.16:0/2691863598 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8e00cac90 con 0x7fe8e015db00 2023-05-22T16:03:27.367 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:27.354+0000 7fe8e713b640 1 -- 172.21.15.16:0/2691863598 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8e006bdd0 con 0x7fe8e015db00 2023-05-22T16:03:27.367 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:27.354+0000 7fe8dd7fa640 1 -- 172.21.15.16:0/2691863598 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 82) v1 ==== 84332+0+0 (secure 0 0 0) 0x7fe8d800ab90 con 0x7fe8e015db00 2023-05-22T16:03:27.367 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:27.354+0000 7fe8dd7fa640 1 --2- 172.21.15.16:0/2691863598 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fe8cc0654a0 0x7fe8cc067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:27.367 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:27.358+0000 7fe8dd7fa640 1 -- 172.21.15.16:0/2691863598 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(168..168 src has 1..168) v4 ==== 7499+0+0 (secure 0 0 0) 0x7fe8d8082c90 con 0x7fe8e015db00 2023-05-22T16:03:27.367 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:27.358+0000 7fe8e4ed7640 1 --2- 172.21.15.16:0/2691863598 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fe8cc0654a0 0x7fe8cc067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:27.367 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:27.358+0000 7fe8e4ed7640 1 --2- 172.21.15.16:0/2691863598 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fe8cc0654a0 0x7fe8cc067950 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fe8d00ae000 tx=0x7fe8d00a7000 comp rx=0 tx=0).ready entity=mgr.10032 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:03:27.368 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:27.362+0000 7fe8dd7fa640 1 -- 172.21.15.16:0/2691863598 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fe8d80502f0 con 0x7fe8e015db00 2023-05-22T16:03:27.550 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:27.546+0000 7fe8e713b640 1 -- 172.21.15.16:0/2691863598 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fe8e006b810 con 0x7fe8e015db00 2023-05-22T16:03:28.199 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:28.194+0000 7fe8dd7fa640 1 -- 172.21.15.16:0/2691863598 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v68) v1 ==== 104+0+0 (secure 0 0 0) 0x7fe8d8053910 con 0x7fe8e015db00 2023-05-22T16:03:28.205 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:28.198+0000 7fe8e713b640 1 -- 172.21.15.16:0/2691863598 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fe8cc0654a0 msgr2=0x7fe8cc067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:28.205 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:28.198+0000 7fe8e713b640 1 --2- 172.21.15.16:0/2691863598 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fe8cc0654a0 0x7fe8cc067950 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fe8d00ae000 tx=0x7fe8d00a7000 comp rx=0 tx=0).stop 2023-05-22T16:03:28.205 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:28.198+0000 7fe8e713b640 1 -- 172.21.15.16:0/2691863598 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe8e015db00 msgr2=0x7fe8e00b0590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:28.205 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:28.198+0000 7fe8e713b640 1 --2- 172.21.15.16:0/2691863598 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe8e015db00 0x7fe8e00b0590 secure :-1 s=READY pgs=2193 cs=0 l=1 rev1=1 crypto rx=0x7fe8d800c730 tx=0x7fe8d800cbf0 comp rx=0 tx=0).stop 2023-05-22T16:03:28.206 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:28.198+0000 7fe8e713b640 1 -- 172.21.15.16:0/2691863598 shutdown_connections 2023-05-22T16:03:28.206 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:28.198+0000 7fe8e713b640 1 --2- 172.21.15.16:0/2691863598 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fe8cc0654a0 0x7fe8cc067950 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:28.206 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:28.198+0000 7fe8e713b640 1 --2- 172.21.15.16:0/2691863598 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe8e015db00 0x7fe8e00b0590 unknown :-1 s=CLOSED pgs=2193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:28.206 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:28.198+0000 7fe8e713b640 1 --2- 172.21.15.16:0/2691863598 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe8e015d680 0x7fe8e00b0050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:28.206 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:28.198+0000 7fe8e713b640 1 --2- 172.21.15.16:0/2691863598 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe8e0159970 0x7fe8e015e7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:28.206 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:28.198+0000 7fe8e713b640 1 -- 172.21.15.16:0/2691863598 >> 172.21.15.16:0/2691863598 conn(0x7fe8e0069720 msgr2=0x7fe8e0069af0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:03:28.206 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:28.198+0000 7fe8e713b640 1 -- 172.21.15.16:0/2691863598 shutdown_connections 2023-05-22T16:03:28.207 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:28.198+0000 7fe8e713b640 1 -- 172.21.15.16:0/2691863598 wait complete. 2023-05-22T16:03:28.224 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-05-22T16:03:28.397 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:28.394+0000 7f949db74640 1 Processor -- start 2023-05-22T16:03:28.397 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:28.394+0000 7f949db74640 1 -- start start 2023-05-22T16:03:28.398 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:28.394+0000 7f949db74640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f94980ce530 0x7f949815c540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:28.398 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:28.394+0000 7f949db74640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f94980ce900 0x7f9498159680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:28.399 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:28.394+0000 7f949db74640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f9498159bc0 0x7f94980cc680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:28.399 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:28.394+0000 7f949db74640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94980c3930 con 0x7f94980ce900 2023-05-22T16:03:28.399 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:28.394+0000 7f949db74640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9498158f00 con 0x7f9498159bc0 2023-05-22T16:03:28.399 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:28.394+0000 7f949db74640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f949815cc40 con 0x7f94980ce530 2023-05-22T16:03:28.399 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:28.394+0000 7f9496ffd640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f94980ce900 0x7f9498159680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:28.399 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:28.394+0000 7f9496ffd640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f94980ce900 0x7f9498159680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55432/0 (socket says 172.21.15.16:55432) 2023-05-22T16:03:28.399 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:28.394+0000 7f9496ffd640 1 -- 172.21.15.16:0/2872584550 learned_addr learned my addr 172.21.15.16:0/2872584550 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:03:28.400 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:28.394+0000 7f94977fe640 1 --2- 172.21.15.16:0/2872584550 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f94980ce530 0x7f949815c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:28.400 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:28.394+0000 7f9497fff640 1 --2- 172.21.15.16:0/2872584550 >> v2:172.21.15.62:3300/0 conn(0x7f9498159bc0 0x7f94980cc680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:28.400 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:28.394+0000 7f9496ffd640 1 -- 172.21.15.16:0/2872584550 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f94980ce530 msgr2=0x7f949815c540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:03:28.400 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:28.394+0000 7f9496ffd640 1 --2- 172.21.15.16:0/2872584550 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f94980ce530 0x7f949815c540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:28.400 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:28.394+0000 7f9496ffd640 1 -- 172.21.15.16:0/2872584550 >> v2:172.21.15.62:3300/0 conn(0x7f9498159bc0 msgr2=0x7f94980cc680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:03:28.401 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:28.394+0000 7f9496ffd640 1 --2- 172.21.15.16:0/2872584550 >> v2:172.21.15.62:3300/0 conn(0x7f9498159bc0 0x7f94980cc680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:28.401 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:28.394+0000 7f9496ffd640 1 -- 172.21.15.16:0/2872584550 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f949815cf20 con 0x7f94980ce900 2023-05-22T16:03:28.401 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:28.398+0000 7f9496ffd640 1 --2- 172.21.15.16:0/2872584550 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f94980ce900 0x7f9498159680 secure :-1 s=READY pgs=2194 cs=0 l=1 rev1=1 crypto rx=0x7f94980caad0 tx=0x7f948c01c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51f883825d2325e7 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:03:28.402 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:28.398+0000 7f94967fc640 1 -- 172.21.15.16:0/2872584550 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f948c0aa620 con 0x7f94980ce900 2023-05-22T16:03:28.402 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:28.398+0000 7f94967fc640 1 -- 172.21.15.16:0/2872584550 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f948c0aa780 con 0x7f94980ce900 2023-05-22T16:03:28.402 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:28.398+0000 7f94967fc640 1 -- 172.21.15.16:0/2872584550 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f948c0aab10 con 0x7f94980ce900 2023-05-22T16:03:28.402 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:28.398+0000 7f949db74640 1 -- 172.21.15.16:0/2872584550 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f94980ce900 msgr2=0x7f9498159680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:28.403 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:28.398+0000 7f949db74640 1 --2- 172.21.15.16:0/2872584550 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f94980ce900 0x7f9498159680 secure :-1 s=READY pgs=2194 cs=0 l=1 rev1=1 crypto rx=0x7f94980caad0 tx=0x7f948c01c500 comp rx=0 tx=0).stop 2023-05-22T16:03:28.403 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:28.398+0000 7f949db74640 1 -- 172.21.15.16:0/2872584550 shutdown_connections 2023-05-22T16:03:28.403 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:28.398+0000 7f949db74640 1 --2- 172.21.15.16:0/2872584550 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f94980ce900 0x7f9498159680 unknown :-1 s=CLOSED pgs=2194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:28.403 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:28.398+0000 7f949db74640 1 --2- 172.21.15.16:0/2872584550 >> v2:172.21.15.62:3300/0 conn(0x7f9498159bc0 0x7f94980cc680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:28.403 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:28.398+0000 7f949db74640 1 --2- 172.21.15.16:0/2872584550 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f94980ce530 0x7f949815c540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:28.403 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:28.398+0000 7f949db74640 1 -- 172.21.15.16:0/2872584550 >> 172.21.15.16:0/2872584550 conn(0x7f9498069720 msgr2=0x7f9498069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:03:28.404 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:28.398+0000 7f949db74640 1 -- 172.21.15.16:0/2872584550 shutdown_connections 2023-05-22T16:03:28.404 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:28.398+0000 7f949db74640 1 -- 172.21.15.16:0/2872584550 wait complete. 2023-05-22T16:03:28.405 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:28.398+0000 7f949db74640 1 Processor -- start 2023-05-22T16:03:28.405 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:28.398+0000 7f949db74640 1 -- start start 2023-05-22T16:03:28.405 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:28.402+0000 7f949db74640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f94980ce530 0x7f94981f4bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:28.405 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:28.402+0000 7f949db74640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f94980ce900 0x7f9498158130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:28.405 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:28.402+0000 7f949db74640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9498159bc0 0x7f949815aa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:28.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:28.402+0000 7f949db74640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9498158670 con 0x7f94980ce900 2023-05-22T16:03:28.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:28.402+0000 7f949db74640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f949815af80 con 0x7f94980ce530 2023-05-22T16:03:28.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:28.402+0000 7f949db74640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f949815b190 con 0x7f9498159bc0 2023-05-22T16:03:28.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:28.402+0000 7f9496ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f94980ce900 0x7f9498158130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:28.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:28.402+0000 7f9496ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f94980ce900 0x7f9498158130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55444/0 (socket says 172.21.15.16:55444) 2023-05-22T16:03:28.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:28.402+0000 7f9496ffd640 1 -- 172.21.15.16:0/2364690248 learned_addr learned my addr 172.21.15.16:0/2364690248 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:03:28.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:28.402+0000 7f94977fe640 1 --2- 172.21.15.16:0/2364690248 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f94980ce530 0x7f94981f4bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:28.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:28.402+0000 7f9497fff640 1 --2- 172.21.15.16:0/2364690248 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9498159bc0 0x7f949815aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:28.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:28.402+0000 7f9496ffd640 1 -- 172.21.15.16:0/2364690248 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9498159bc0 msgr2=0x7f949815aa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:28.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:28.402+0000 7f9496ffd640 1 --2- 172.21.15.16:0/2364690248 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9498159bc0 0x7f949815aa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:28.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:28.402+0000 7f9496ffd640 1 -- 172.21.15.16:0/2364690248 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f94980ce530 msgr2=0x7f94981f4bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:28.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:28.402+0000 7f9496ffd640 1 --2- 172.21.15.16:0/2364690248 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f94980ce530 0x7f94981f4bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:28.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:28.402+0000 7f9496ffd640 1 -- 172.21.15.16:0/2364690248 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f948c0094d0 con 0x7f94980ce900 2023-05-22T16:03:28.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:28.402+0000 7f9496ffd640 1 --2- 172.21.15.16:0/2364690248 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f94980ce900 0x7f9498158130 secure :-1 s=READY pgs=2195 cs=0 l=1 rev1=1 crypto rx=0x7f948c01abb0 tx=0x7f948c01ad40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:03:28.408 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:28.402+0000 7f947ffff640 1 -- 172.21.15.16:0/2364690248 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f948c0bd3f0 con 0x7f94980ce900 2023-05-22T16:03:28.409 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:28.402+0000 7f947ffff640 1 -- 172.21.15.16:0/2364690248 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f948c0bd550 con 0x7f94980ce900 2023-05-22T16:03:28.409 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:28.402+0000 7f949db74640 1 -- 172.21.15.16:0/2364690248 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94980c9570 con 0x7f94980ce900 2023-05-22T16:03:28.409 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:28.402+0000 7f947ffff640 1 -- 172.21.15.16:0/2364690248 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f948c0bd900 con 0x7f94980ce900 2023-05-22T16:03:28.409 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:28.402+0000 7f949db74640 1 -- 172.21.15.16:0/2364690248 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94980ccfd0 con 0x7f94980ce900 2023-05-22T16:03:28.409 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:28.402+0000 7f949db74640 1 -- 172.21.15.16:0/2364690248 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94980c9700 con 0x7f94980ce900 2023-05-22T16:03:28.409 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:28.402+0000 7f947ffff640 1 -- 172.21.15.16:0/2364690248 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 82) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f948c0c6440 con 0x7f94980ce900 2023-05-22T16:03:28.409 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:28.402+0000 7f947ffff640 1 --2- 172.21.15.16:0/2364690248 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f94840654a0 0x7f9484067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:28.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:28.402+0000 7f947ffff640 1 -- 172.21.15.16:0/2364690248 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(168..168 src has 1..168) v4 ==== 7499+0+0 (secure 0 0 0) 0x7f948c11dcb0 con 0x7f94980ce900 2023-05-22T16:03:28.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:28.402+0000 7f94977fe640 1 --2- 172.21.15.16:0/2364690248 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f94840654a0 0x7f9484067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:28.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:28.406+0000 7f94977fe640 1 --2- 172.21.15.16:0/2364690248 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f94840654a0 0x7f9484067950 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f948800b930 tx=0x7f9488008040 comp rx=0 tx=0).ready entity=mgr.10032 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:03:28.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:28.406+0000 7f947ffff640 1 -- 172.21.15.16:0/2364690248 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f948c0eb180 con 0x7f94980ce900 2023-05-22T16:03:28.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:28.558+0000 7f949db74640 1 -- 172.21.15.16:0/2364690248 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f94980cc460 con 0x7f94980ce900 2023-05-22T16:03:29.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:29.278+0000 7f947ffff640 1 -- 172.21.15.16:0/2364690248 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v169) v1 ==== 181+0+0 (secure 0 0 0) 0x7f948c0ee7a0 con 0x7f94980ce900 2023-05-22T16:03:29.288 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'cephfs_metadata' removed 2023-05-22T16:03:29.292 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:29.286+0000 7f949db74640 1 -- 172.21.15.16:0/2364690248 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f94840654a0 msgr2=0x7f9484067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:29.293 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:29.286+0000 7f949db74640 1 --2- 172.21.15.16:0/2364690248 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f94840654a0 0x7f9484067950 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f948800b930 tx=0x7f9488008040 comp rx=0 tx=0).stop 2023-05-22T16:03:29.293 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:29.286+0000 7f949db74640 1 -- 172.21.15.16:0/2364690248 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f94980ce900 msgr2=0x7f9498158130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:29.293 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:29.286+0000 7f949db74640 1 --2- 172.21.15.16:0/2364690248 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f94980ce900 0x7f9498158130 secure :-1 s=READY pgs=2195 cs=0 l=1 rev1=1 crypto rx=0x7f948c01abb0 tx=0x7f948c01ad40 comp rx=0 tx=0).stop 2023-05-22T16:03:29.294 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:29.290+0000 7f949db74640 1 -- 172.21.15.16:0/2364690248 shutdown_connections 2023-05-22T16:03:29.294 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:29.290+0000 7f949db74640 1 --2- 172.21.15.16:0/2364690248 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9498159bc0 0x7f949815aa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:29.294 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:29.290+0000 7f949db74640 1 --2- 172.21.15.16:0/2364690248 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f94840654a0 0x7f9484067950 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:29.294 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:29.290+0000 7f949db74640 1 --2- 172.21.15.16:0/2364690248 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f94980ce900 0x7f9498158130 unknown :-1 s=CLOSED pgs=2195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:29.294 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:29.290+0000 7f949db74640 1 --2- 172.21.15.16:0/2364690248 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f94980ce530 0x7f94981f4bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:29.294 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:29.290+0000 7f949db74640 1 -- 172.21.15.16:0/2364690248 >> 172.21.15.16:0/2364690248 conn(0x7f9498069720 msgr2=0x7f94980cab20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:03:29.295 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:29.290+0000 7f949db74640 1 -- 172.21.15.16:0/2364690248 shutdown_connections 2023-05-22T16:03:29.295 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:29.290+0000 7f949db74640 1 -- 172.21.15.16:0/2364690248 wait complete. 2023-05-22T16:03:29.318 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-05-22T16:03:29.442 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:29.438+0000 7faefdba9640 1 Processor -- start 2023-05-22T16:03:29.442 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:29.438+0000 7faefdba9640 1 -- start start 2023-05-22T16:03:29.443 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:29.438+0000 7faefdba9640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7faef806fef0 0x7faef8159080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:29.443 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:29.438+0000 7faefdba9640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7faef80702c0 0x7faef80c8f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:29.443 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:29.438+0000 7faefdba9640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faef81595c0 con 0x7faef80c9460 2023-05-22T16:03:29.444 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:29.438+0000 7faefdba9640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faef8069a90 con 0x7faef80702c0 2023-05-22T16:03:29.444 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:29.438+0000 7faefdba9640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faef80cc7f0 con 0x7faef806fef0 2023-05-22T16:03:29.444 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:29.438+0000 7faef77fe640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7faef806fef0 0x7faef8159080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:29.444 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:29.438+0000 7faef6ffd640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7faef80702c0 0x7faef80c8f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:29.444 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:29.438+0000 7faef6ffd640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7faef80702c0 0x7faef80c8f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.16:44854/0 (socket says 172.21.15.16:44854) 2023-05-22T16:03:29.444 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:29.438+0000 7faef7fff640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7faef80c9460 0x7faef8069f20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.16:48954/0 (socket says 172.21.15.16:48954) 2023-05-22T16:03:29.444 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:29.438+0000 7faef6ffd640 1 -- 172.21.15.16:0/3830270210 learned_addr learned my addr 172.21.15.16:0/3830270210 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:03:29.445 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:29.438+0000 7faef77fe640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7faef806fef0 0x7faef8159080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3301/0 says I am v2:172.21.15.16:55026/0 (socket says 172.21.15.16:55026) 2023-05-22T16:03:29.445 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:29.438+0000 7faef6ffd640 1 -- 172.21.15.16:0/3830270210 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7faef806fef0 msgr2=0x7faef8159080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:03:29.445 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:29.438+0000 7faef6ffd640 1 --2- 172.21.15.16:0/3830270210 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7faef806fef0 0x7faef8159080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:29.445 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:29.438+0000 7faef6ffd640 1 -- 172.21.15.16:0/3830270210 >> v1:172.21.15.62:6789/0 conn(0x7faef80c9460 legacy=0x7faef8069f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:29.446 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:29.438+0000 7faef6ffd640 1 -- 172.21.15.16:0/3830270210 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faef80ccad0 con 0x7faef80702c0 2023-05-22T16:03:29.447 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:29.442+0000 7faef6ffd640 1 --2- 172.21.15.16:0/3830270210 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faef80702c0 0x7faef80c8f20 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7faee000bb10 tx=0x7faee001ebe0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3486d2002c6f6d2d server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:03:29.447 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:29.442+0000 7faef67fc640 1 -- 172.21.15.16:0/3830270210 <== mon.1 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faee00a7b40 con 0x7faef80702c0 2023-05-22T16:03:29.447 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:29.442+0000 7faef67fc640 1 -- 172.21.15.16:0/3830270210 <== mon.1 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faee00a7ca0 con 0x7faef80702c0 2023-05-22T16:03:29.448 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:29.442+0000 7faef67fc640 1 -- 172.21.15.16:0/3830270210 <== mon.1 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faee00af660 con 0x7faef80702c0 2023-05-22T16:03:29.449 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:29.442+0000 7faefdba9640 1 -- 172.21.15.16:0/3830270210 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faef80702c0 msgr2=0x7faef80c8f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:29.449 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:29.442+0000 7faefdba9640 1 --2- 172.21.15.16:0/3830270210 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faef80702c0 0x7faef80c8f20 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7faee000bb10 tx=0x7faee001ebe0 comp rx=0 tx=0).stop 2023-05-22T16:03:29.449 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:29.442+0000 7faefdba9640 1 -- 172.21.15.16:0/3830270210 shutdown_connections 2023-05-22T16:03:29.449 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:29.442+0000 7faefdba9640 1 --2- 172.21.15.16:0/3830270210 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faef80702c0 0x7faef80c8f20 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:29.449 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:29.442+0000 7faefdba9640 1 --2- 172.21.15.16:0/3830270210 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7faef806fef0 0x7faef8159080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:29.449 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:29.446+0000 7faefdba9640 1 -- 172.21.15.16:0/3830270210 >> 172.21.15.16:0/3830270210 conn(0x7faef80ce8d0 msgr2=0x7faef80cecd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:03:29.450 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:29.446+0000 7faefdba9640 1 -- 172.21.15.16:0/3830270210 shutdown_connections 2023-05-22T16:03:29.450 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:29.446+0000 7faefdba9640 1 -- 172.21.15.16:0/3830270210 wait complete. 2023-05-22T16:03:29.451 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:29.446+0000 7faefdba9640 1 Processor -- start 2023-05-22T16:03:29.452 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:29.446+0000 7faefdba9640 1 -- start start 2023-05-22T16:03:29.452 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:29.446+0000 7faefdba9640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faef806fef0 0x7faef80c7090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:29.452 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:29.446+0000 7faefdba9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faef80702c0 0x7faef80af740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:29.452 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:29.446+0000 7faefdba9640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7faef80c9460 0x7faef80c7df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:29.452 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:29.446+0000 7faefdba9640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faef80c75d0 con 0x7faef80702c0 2023-05-22T16:03:29.453 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:29.446+0000 7faefdba9640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faef80afc80 con 0x7faef806fef0 2023-05-22T16:03:29.453 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:29.446+0000 7faefdba9640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faef80afea0 con 0x7faef80c9460 2023-05-22T16:03:29.453 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:29.446+0000 7faef6ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faef80702c0 0x7faef80af740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:29.453 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:29.446+0000 7faef6ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faef80702c0 0x7faef80af740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47918/0 (socket says 172.21.15.16:47918) 2023-05-22T16:03:29.453 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:29.446+0000 7faef6ffd640 1 -- 172.21.15.16:0/3885719268 learned_addr learned my addr 172.21.15.16:0/3885719268 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:03:29.453 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:29.446+0000 7faef6ffd640 1 -- 172.21.15.16:0/3885719268 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7faef80c9460 msgr2=0x7faef80c7df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:29.453 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:29.446+0000 7faef6ffd640 1 --2- 172.21.15.16:0/3885719268 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7faef80c9460 0x7faef80c7df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:29.453 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:29.446+0000 7faef6ffd640 1 -- 172.21.15.16:0/3885719268 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faef806fef0 msgr2=0x7faef80c7090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:29.454 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:29.446+0000 7faef6ffd640 1 --2- 172.21.15.16:0/3885719268 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faef806fef0 0x7faef80c7090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:29.454 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:29.446+0000 7faef6ffd640 1 -- 172.21.15.16:0/3885719268 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faee000b770 con 0x7faef80702c0 2023-05-22T16:03:29.454 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:29.446+0000 7faef6ffd640 1 --2- 172.21.15.16:0/3885719268 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faef80702c0 0x7faef80af740 secure :-1 s=READY pgs=2196 cs=0 l=1 rev1=1 crypto rx=0x7faee001c810 tx=0x7faee001b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:03:29.454 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:29.446+0000 7faeeffff640 1 -- 172.21.15.16:0/3885719268 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faee0006ee0 con 0x7faef80702c0 2023-05-22T16:03:29.454 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:29.446+0000 7faeeffff640 1 -- 172.21.15.16:0/3885719268 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faee00afe30 con 0x7faef80702c0 2023-05-22T16:03:29.454 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:29.446+0000 7faefdba9640 1 -- 172.21.15.16:0/3885719268 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faef8070710 con 0x7faef80702c0 2023-05-22T16:03:29.454 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:29.446+0000 7faeeffff640 1 -- 172.21.15.16:0/3885719268 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faee00b8640 con 0x7faef80702c0 2023-05-22T16:03:29.455 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:29.446+0000 7faefdba9640 1 -- 172.21.15.16:0/3885719268 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faef80cc4a0 con 0x7faef80702c0 2023-05-22T16:03:29.455 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:29.450+0000 7faefdba9640 1 -- 172.21.15.16:0/3885719268 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faef8158340 con 0x7faef80702c0 2023-05-22T16:03:29.455 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:29.450+0000 7faeeffff640 1 -- 172.21.15.16:0/3885719268 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 82) v1 ==== 84332+0+0 (secure 0 0 0) 0x7faee00b6040 con 0x7faef80702c0 2023-05-22T16:03:29.456 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:29.450+0000 7faeeffff640 1 --2- 172.21.15.16:0/3885719268 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7faed80654a0 0x7faed8067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:29.456 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:29.450+0000 7faeeffff640 1 -- 172.21.15.16:0/3885719268 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(169..169 src has 1..169) v4 ==== 7061+0+0 (secure 0 0 0) 0x7faee011fbb0 con 0x7faef80702c0 2023-05-22T16:03:29.456 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:29.450+0000 7faef77fe640 1 --2- 172.21.15.16:0/3885719268 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7faed80654a0 0x7faed8067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:29.456 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:29.450+0000 7faef77fe640 1 --2- 172.21.15.16:0/3885719268 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7faed80654a0 0x7faed8067950 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7faee8006aa0 tx=0x7faee8006a30 comp rx=0 tx=0).ready entity=mgr.10032 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:03:29.457 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:29.454+0000 7faeeffff640 1 -- 172.21.15.16:0/3885719268 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7faee00ed3c0 con 0x7faef80702c0 2023-05-22T16:03:29.613 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:29.610+0000 7faefdba9640 1 -- 172.21.15.16:0/3885719268 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7faef804dbe0 con 0x7faef80702c0 2023-05-22T16:03:30.296 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:30.290+0000 7faeeffff640 1 -- 172.21.15.16:0/3885719268 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v170) v1 ==== 169+0+0 (secure 0 0 0) 0x7faee00bc0d0 con 0x7faef80702c0 2023-05-22T16:03:30.299 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'cephfs_data' removed 2023-05-22T16:03:30.304 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:30.298+0000 7faefdba9640 1 -- 172.21.15.16:0/3885719268 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7faed80654a0 msgr2=0x7faed8067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:30.304 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:30.298+0000 7faefdba9640 1 --2- 172.21.15.16:0/3885719268 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7faed80654a0 0x7faed8067950 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7faee8006aa0 tx=0x7faee8006a30 comp rx=0 tx=0).stop 2023-05-22T16:03:30.304 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:30.298+0000 7faefdba9640 1 -- 172.21.15.16:0/3885719268 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faef80702c0 msgr2=0x7faef80af740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:30.305 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:30.298+0000 7faefdba9640 1 --2- 172.21.15.16:0/3885719268 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faef80702c0 0x7faef80af740 secure :-1 s=READY pgs=2196 cs=0 l=1 rev1=1 crypto rx=0x7faee001c810 tx=0x7faee001b980 comp rx=0 tx=0).stop 2023-05-22T16:03:30.306 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:30.302+0000 7faefdba9640 1 -- 172.21.15.16:0/3885719268 shutdown_connections 2023-05-22T16:03:30.306 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:30.302+0000 7faefdba9640 1 --2- 172.21.15.16:0/3885719268 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7faef80c9460 0x7faef80c7df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:30.306 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:30.302+0000 7faefdba9640 1 --2- 172.21.15.16:0/3885719268 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7faed80654a0 0x7faed8067950 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:30.307 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:30.302+0000 7faefdba9640 1 --2- 172.21.15.16:0/3885719268 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faef80702c0 0x7faef80af740 unknown :-1 s=CLOSED pgs=2196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:30.307 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:30.302+0000 7faefdba9640 1 --2- 172.21.15.16:0/3885719268 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faef806fef0 0x7faef80c7090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:30.307 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:30.302+0000 7faefdba9640 1 -- 172.21.15.16:0/3885719268 >> 172.21.15.16:0/3885719268 conn(0x7faef80ce8d0 msgr2=0x7faef80b1ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:03:30.308 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:30.302+0000 7faefdba9640 1 -- 172.21.15.16:0/3885719268 shutdown_connections 2023-05-22T16:03:30.308 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:30.302+0000 7faefdba9640 1 -- 172.21.15.16:0/3885719268 wait complete. 2023-05-22T16:03:30.328 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2023-05-22T16:03:30.455 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:30.450+0000 7fc4e5358640 1 Processor -- start 2023-05-22T16:03:30.455 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:30.450+0000 7fc4e5358640 1 -- start start 2023-05-22T16:03:30.458 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:30.450+0000 7fc4e5358640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fc4e00b2880 0x7fc4e00cbfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:30.458 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:30.450+0000 7fc4e5358640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4e00c6ba0 con 0x7fc4e00cc520 2023-05-22T16:03:30.458 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:30.450+0000 7fc4e5358640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4e00cfef0 con 0x7fc4e00c67d0 2023-05-22T16:03:30.458 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:30.450+0000 7fc4e5358640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4e015a300 con 0x7fc4e00b2880 2023-05-22T16:03:30.458 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:30.450+0000 7fc4df7fe640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fc4e00cc520 0x7fc4e00c3db0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:48098/0 (socket says 172.21.15.16:48098) 2023-05-22T16:03:30.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:30.454+0000 7fc4df7fe640 1 -- 172.21.15.16:0/4056905594 learned_addr learned my addr 172.21.15.16:0/4056905594 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T16:03:30.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:30.454+0000 7fc4d7fff640 1 --2- 172.21.15.16:0/4056905594 >> v2:172.21.15.62:3300/0 conn(0x7fc4e00b2880 0x7fc4e00cbfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:30.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:30.454+0000 7fc4de7fc640 1 -- 172.21.15.16:0/4056905594 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 41998374 0 0) 0x7fc4e00c6ba0 con 0x7fc4e00cc520 2023-05-22T16:03:30.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:30.454+0000 7fc4de7fc640 1 -- 172.21.15.16:0/4056905594 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc4cc003530 con 0x7fc4e00cc520 2023-05-22T16:03:30.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:30.454+0000 7fc4de7fc640 1 -- 172.21.15.16:0/4056905594 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4073542952 0 0) 0x7fc4cc003530 con 0x7fc4e00cc520 2023-05-22T16:03:30.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:30.454+0000 7fc4de7fc640 1 -- 172.21.15.16:0/4056905594 >> v2:172.21.15.62:3300/0 conn(0x7fc4e00b2880 msgr2=0x7fc4e00cbfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:03:30.460 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:30.454+0000 7fc4de7fc640 1 --2- 172.21.15.16:0/4056905594 >> v2:172.21.15.62:3300/0 conn(0x7fc4e00b2880 0x7fc4e00cbfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:30.460 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:30.454+0000 7fc4de7fc640 1 -- 172.21.15.16:0/4056905594 >> v1:172.21.15.62:6789/0 conn(0x7fc4e00c67d0 legacy=0x7fc4e00cf730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:30.460 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:30.454+0000 7fc4de7fc640 1 -- 172.21.15.16:0/4056905594 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4e00d01d0 con 0x7fc4e00cc520 2023-05-22T16:03:30.460 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:30.454+0000 7fc4de7fc640 1 -- 172.21.15.16:0/4056905594 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7fc4d8001a00 con 0x7fc4e00cc520 2023-05-22T16:03:30.460 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:30.454+0000 7fc4de7fc640 1 -- 172.21.15.16:0/4056905594 >> v1:172.21.15.16:6789/0 conn(0x7fc4e00cc520 legacy=0x7fc4e00c3db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:30.460 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:30.454+0000 7fc4de7fc640 1 --2- 172.21.15.16:0/4056905594 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4cc003980 0x7fc4cc003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:30.461 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:30.454+0000 7fc4de7fc640 1 --2- 172.21.15.16:0/4056905594 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc4cc0047d0 0x7fc4cc004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:30.461 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:30.454+0000 7fc4de7fc640 1 --2- 172.21.15.16:0/4056905594 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc4cc005180 0x7fc4cc00b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:30.461 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:30.454+0000 7fc4de7fc640 1 -- 172.21.15.16:0/4056905594 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4e015a300 con 0x7fc4cc003980 2023-05-22T16:03:30.461 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:30.454+0000 7fc4de7fc640 1 -- 172.21.15.16:0/4056905594 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4cc00c090 con 0x7fc4cc0047d0 2023-05-22T16:03:30.461 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:30.454+0000 7fc4de7fc640 1 -- 172.21.15.16:0/4056905594 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4cc00c250 con 0x7fc4cc005180 2023-05-22T16:03:30.461 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:30.454+0000 7fc4deffd640 1 --2- 172.21.15.16:0/4056905594 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4cc003980 0x7fc4cc003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:30.461 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:30.454+0000 7fc4d7fff640 1 --2- 172.21.15.16:0/4056905594 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc4cc0047d0 0x7fc4cc004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:30.462 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:30.454+0000 7fc4df7fe640 1 --2- 172.21.15.16:0/4056905594 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc4cc005180 0x7fc4cc00b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:30.462 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:30.454+0000 7fc4deffd640 1 -- 172.21.15.16:0/4056905594 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc4cc005180 msgr2=0x7fc4cc00b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:03:30.462 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:30.454+0000 7fc4deffd640 1 --2- 172.21.15.16:0/4056905594 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc4cc005180 0x7fc4cc00b990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:30.462 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:30.454+0000 7fc4deffd640 1 -- 172.21.15.16:0/4056905594 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc4cc0047d0 msgr2=0x7fc4cc004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:03:30.462 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:30.454+0000 7fc4deffd640 1 --2- 172.21.15.16:0/4056905594 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc4cc0047d0 0x7fc4cc004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:30.462 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:30.454+0000 7fc4deffd640 1 -- 172.21.15.16:0/4056905594 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4cc003530 con 0x7fc4cc003980 2023-05-22T16:03:30.462 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:30.454+0000 7fc4df7fe640 1 -- 172.21.15.16:0/4056905594 reap_dead start 2023-05-22T16:03:30.464 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:30.454+0000 7fc4deffd640 1 --2- 172.21.15.16:0/4056905594 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4cc003980 0x7fc4cc003d90 secure :-1 s=READY pgs=2198 cs=0 l=1 rev1=1 crypto rx=0x7fc4c8001780 tx=0x7fc4c801c3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65a4a33d21211723 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:03:30.465 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:30.454+0000 7fc4de7fc640 1 -- 172.21.15.16:0/4056905594 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4c80052d0 con 0x7fc4cc003980 2023-05-22T16:03:30.465 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:30.454+0000 7fc4de7fc640 1 -- 172.21.15.16:0/4056905594 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4c80a7050 con 0x7fc4cc003980 2023-05-22T16:03:30.465 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:30.454+0000 7fc4de7fc640 1 -- 172.21.15.16:0/4056905594 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4c8003600 con 0x7fc4cc003980 2023-05-22T16:03:30.465 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:30.458+0000 7fc4e5358640 1 -- 172.21.15.16:0/4056905594 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4cc003980 msgr2=0x7fc4cc003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:30.465 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:30.458+0000 7fc4e5358640 1 --2- 172.21.15.16:0/4056905594 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4cc003980 0x7fc4cc003d90 secure :-1 s=READY pgs=2198 cs=0 l=1 rev1=1 crypto rx=0x7fc4c8001780 tx=0x7fc4c801c3b0 comp rx=0 tx=0).stop 2023-05-22T16:03:30.465 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:30.458+0000 7fc4e5358640 1 -- 172.21.15.16:0/4056905594 shutdown_connections 2023-05-22T16:03:30.465 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:30.458+0000 7fc4e5358640 1 --2- 172.21.15.16:0/4056905594 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4cc003980 0x7fc4cc003d90 unknown :-1 s=CLOSED pgs=2198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:30.466 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:30.458+0000 7fc4e5358640 1 -- 172.21.15.16:0/4056905594 >> 172.21.15.16:0/4056905594 conn(0x7fc4e006f5a0 msgr2=0x7fc4e006f9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:03:30.466 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:30.458+0000 7fc4e5358640 1 -- 172.21.15.16:0/4056905594 shutdown_connections 2023-05-22T16:03:30.466 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:30.458+0000 7fc4e5358640 1 -- 172.21.15.16:0/4056905594 wait complete. 2023-05-22T16:03:30.466 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:30.458+0000 7fc4e5358640 1 Processor -- start 2023-05-22T16:03:30.466 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:30.458+0000 7fc4e5358640 1 -- start start 2023-05-22T16:03:30.466 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:30.458+0000 7fc4e5358640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4cc003980 0x7fc4e014f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:30.466 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:30.458+0000 7fc4e5358640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc4e00c8380 0x7fc4e00b3a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:30.467 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:30.458+0000 7fc4e5358640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc4e00c87f0 0x7fc4e00b4ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:30.467 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:30.458+0000 7fc4e5358640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4e014f8f0 con 0x7fc4cc003980 2023-05-22T16:03:30.467 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:30.458+0000 7fc4e5358640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4e015c340 con 0x7fc4e00c87f0 2023-05-22T16:03:30.467 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:30.458+0000 7fc4e5358640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4e015ba10 con 0x7fc4e00c8380 2023-05-22T16:03:30.467 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:30.458+0000 7fc4deffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4cc003980 0x7fc4e014f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:30.467 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:30.458+0000 7fc4deffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4cc003980 0x7fc4e014f3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47928/0 (socket says 172.21.15.16:47928) 2023-05-22T16:03:30.467 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:30.458+0000 7fc4deffd640 1 -- 172.21.15.16:0/434622112 learned_addr learned my addr 172.21.15.16:0/434622112 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:03:30.467 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:30.458+0000 7fc4df7fe640 1 --2- 172.21.15.16:0/434622112 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc4e00c87f0 0x7fc4e00b4ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:30.468 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:30.458+0000 7fc4d7fff640 1 --2- 172.21.15.16:0/434622112 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc4e00c8380 0x7fc4e00b3a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:30.468 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:30.458+0000 7fc4deffd640 1 -- 172.21.15.16:0/434622112 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc4e00c8380 msgr2=0x7fc4e00b3a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:30.468 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:30.458+0000 7fc4deffd640 1 --2- 172.21.15.16:0/434622112 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc4e00c8380 0x7fc4e00b3a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:30.468 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:30.458+0000 7fc4deffd640 1 -- 172.21.15.16:0/434622112 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc4e00c87f0 msgr2=0x7fc4e00b4ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:30.468 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:30.458+0000 7fc4deffd640 1 --2- 172.21.15.16:0/434622112 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc4e00c87f0 0x7fc4e00b4ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:30.468 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:30.458+0000 7fc4deffd640 1 -- 172.21.15.16:0/434622112 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4e00d0f90 con 0x7fc4cc003980 2023-05-22T16:03:30.468 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:30.458+0000 7fc4deffd640 1 --2- 172.21.15.16:0/434622112 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4cc003980 0x7fc4e014f3b0 secure :-1 s=READY pgs=2199 cs=0 l=1 rev1=1 crypto rx=0x7fc4c8005040 tx=0x7fc4cc005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:03:30.469 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:30.458+0000 7fc4d77fe640 1 -- 172.21.15.16:0/434622112 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4c80072f0 con 0x7fc4cc003980 2023-05-22T16:03:30.469 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:30.458+0000 7fc4d77fe640 1 -- 172.21.15.16:0/434622112 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4c8007450 con 0x7fc4cc003980 2023-05-22T16:03:30.469 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:30.458+0000 7fc4e5358640 1 -- 172.21.15.16:0/434622112 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4e00d0a70 con 0x7fc4cc003980 2023-05-22T16:03:30.469 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:30.458+0000 7fc4d77fe640 1 -- 172.21.15.16:0/434622112 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4c80067e0 con 0x7fc4cc003980 2023-05-22T16:03:30.469 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:30.458+0000 7fc4e5358640 1 -- 172.21.15.16:0/434622112 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4e00cc910 con 0x7fc4cc003980 2023-05-22T16:03:30.469 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:30.458+0000 7fc4e5358640 1 -- 172.21.15.16:0/434622112 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4e004dbe0 con 0x7fc4cc003980 2023-05-22T16:03:30.469 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:30.462+0000 7fc4d77fe640 1 -- 172.21.15.16:0/434622112 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 82) v1 ==== 84332+0+0 (secure 0 0 0) 0x7fc4c8006940 con 0x7fc4cc003980 2023-05-22T16:03:30.470 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:30.462+0000 7fc4d77fe640 1 --2- 172.21.15.16:0/434622112 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fc4c00654d0 0x7fc4c0067980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:30.470 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:30.462+0000 7fc4d77fe640 1 -- 172.21.15.16:0/434622112 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(170..170 src has 1..170) v4 ==== 6618+0+0 (secure 0 0 0) 0x7fc4c811cdb0 con 0x7fc4cc003980 2023-05-22T16:03:30.471 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:30.462+0000 7fc4d7fff640 1 --2- 172.21.15.16:0/434622112 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fc4c00654d0 0x7fc4c0067980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:30.471 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:30.462+0000 7fc4d7fff640 1 --2- 172.21.15.16:0/434622112 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fc4c00654d0 0x7fc4c0067980 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fc4e0159670 tx=0x7fc4d0008040 comp rx=0 tx=0).ready entity=mgr.10032 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:03:30.471 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:30.466+0000 7fc4d77fe640 1 -- 172.21.15.16:0/434622112 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fc4c80ee000 con 0x7fc4cc003980 2023-05-22T16:03:30.629 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:30.626+0000 7fc4e5358640 1 -- 172.21.15.16:0/434622112 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7fc4e004dbe0 con 0x7fc4cc003980 2023-05-22T16:03:30.640 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:30.634+0000 7fc4d77fe640 1 -- 172.21.15.16:0/434622112 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v55) v1 ==== 120+0+0 (secure 0 0 0) 0x7fc4c80edbf0 con 0x7fc4cc003980 2023-05-22T16:03:30.642 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:30.638+0000 7fc4e5358640 1 -- 172.21.15.16:0/434622112 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fc4c00654d0 msgr2=0x7fc4c0067980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:30.642 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:30.638+0000 7fc4e5358640 1 --2- 172.21.15.16:0/434622112 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fc4c00654d0 0x7fc4c0067980 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fc4e0159670 tx=0x7fc4d0008040 comp rx=0 tx=0).stop 2023-05-22T16:03:30.643 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:30.638+0000 7fc4e5358640 1 -- 172.21.15.16:0/434622112 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4cc003980 msgr2=0x7fc4e014f3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:30.643 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:30.638+0000 7fc4e5358640 1 --2- 172.21.15.16:0/434622112 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4cc003980 0x7fc4e014f3b0 secure :-1 s=READY pgs=2199 cs=0 l=1 rev1=1 crypto rx=0x7fc4c8005040 tx=0x7fc4cc005800 comp rx=0 tx=0).stop 2023-05-22T16:03:30.643 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:30.638+0000 7fc4e5358640 1 -- 172.21.15.16:0/434622112 shutdown_connections 2023-05-22T16:03:30.643 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:30.638+0000 7fc4e5358640 1 --2- 172.21.15.16:0/434622112 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc4e00c87f0 0x7fc4e00b4ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:30.643 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:30.638+0000 7fc4e5358640 1 --2- 172.21.15.16:0/434622112 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc4e00c8380 0x7fc4e00b3a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:30.643 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:30.638+0000 7fc4e5358640 1 --2- 172.21.15.16:0/434622112 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fc4c00654d0 0x7fc4c0067980 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:30.643 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:30.638+0000 7fc4e5358640 1 --2- 172.21.15.16:0/434622112 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4cc003980 0x7fc4e014f3b0 unknown :-1 s=CLOSED pgs=2199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:30.644 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:30.638+0000 7fc4e5358640 1 -- 172.21.15.16:0/434622112 >> 172.21.15.16:0/434622112 conn(0x7fc4e006f5a0 msgr2=0x7fc4e015d750 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:03:30.644 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:30.638+0000 7fc4e5358640 1 -- 172.21.15.16:0/434622112 shutdown_connections 2023-05-22T16:03:30.644 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:30.638+0000 7fc4e5358640 1 -- 172.21.15.16:0/434622112 wait complete. 2023-05-22T16:03:30.662 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2023-05-22T16:03:30.793 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:30.790+0000 7f97ebf18640 1 Processor -- start 2023-05-22T16:03:30.794 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:30.790+0000 7f97ebf18640 1 -- start start 2023-05-22T16:03:30.796 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:30.790+0000 7f97ebf18640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f97e40cf870 0x7f97e406db00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:30.796 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:30.790+0000 7f97ebf18640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f97e406e740 0x7f97e406ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:30.796 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:30.790+0000 7f97ebf18640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97e406e040 con 0x7f97e415cfa0 2023-05-22T16:03:30.796 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:30.790+0000 7f97ebf18640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97e40cb9f0 con 0x7f97e406e740 2023-05-22T16:03:30.796 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:30.790+0000 7f97ebf18640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97e406f100 con 0x7f97e40cf870 2023-05-22T16:03:30.797 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:30.790+0000 7f97ea4b5640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f97e406e740 0x7f97e406ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:30.797 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:30.790+0000 7f97ea4b5640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f97e406e740 0x7f97e406ebc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47936/0 (socket says 172.21.15.16:47936) 2023-05-22T16:03:30.797 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:30.790+0000 7f97ea4b5640 1 -- 172.21.15.16:0/358377954 learned_addr learned my addr 172.21.15.16:0/358377954 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:03:30.797 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:30.790+0000 7f97e94b3640 1 --2- 172.21.15.16:0/358377954 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f97e40cf870 0x7f97e406db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:30.797 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:30.790+0000 7f97ea4b5640 1 -- 172.21.15.16:0/358377954 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f97e40cf870 msgr2=0x7f97e406db00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:03:30.797 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:30.790+0000 7f97ea4b5640 1 --2- 172.21.15.16:0/358377954 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f97e40cf870 0x7f97e406db00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:30.797 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:30.790+0000 7f97ea4b5640 1 -- 172.21.15.16:0/358377954 >> v1:172.21.15.62:6789/0 conn(0x7f97e415cfa0 legacy=0x7f97e40b1fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:30.797 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:30.790+0000 7f97ea4b5640 1 -- 172.21.15.16:0/358377954 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97e406e490 con 0x7f97e406e740 2023-05-22T16:03:30.798 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:30.794+0000 7f97ea4b5640 1 --2- 172.21.15.16:0/358377954 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f97e406e740 0x7f97e406ebc0 secure :-1 s=READY pgs=2200 cs=0 l=1 rev1=1 crypto rx=0x7f97dc009e50 tx=0x7f97dc01c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=419dd098395d5dd7 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:03:30.799 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:30.794+0000 7f97e8cb2640 1 -- 172.21.15.16:0/358377954 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97dc0a5810 con 0x7f97e406e740 2023-05-22T16:03:30.799 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:30.794+0000 7f97e8cb2640 1 -- 172.21.15.16:0/358377954 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97dc0a5970 con 0x7f97e406e740 2023-05-22T16:03:30.799 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:30.794+0000 7f97e8cb2640 1 -- 172.21.15.16:0/358377954 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97dc0a5d00 con 0x7f97e406e740 2023-05-22T16:03:30.799 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:30.794+0000 7f97ebf18640 1 -- 172.21.15.16:0/358377954 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f97e406e740 msgr2=0x7f97e406ebc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:30.799 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:30.794+0000 7f97ebf18640 1 --2- 172.21.15.16:0/358377954 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f97e406e740 0x7f97e406ebc0 secure :-1 s=READY pgs=2200 cs=0 l=1 rev1=1 crypto rx=0x7f97dc009e50 tx=0x7f97dc01c950 comp rx=0 tx=0).stop 2023-05-22T16:03:30.799 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:30.794+0000 7f97ebf18640 1 -- 172.21.15.16:0/358377954 shutdown_connections 2023-05-22T16:03:30.800 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:30.794+0000 7f97ebf18640 1 --2- 172.21.15.16:0/358377954 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f97e406e740 0x7f97e406ebc0 unknown :-1 s=CLOSED pgs=2200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:30.800 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:30.794+0000 7f97ebf18640 1 --2- 172.21.15.16:0/358377954 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f97e40cf870 0x7f97e406db00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:30.800 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:30.794+0000 7f97ebf18640 1 -- 172.21.15.16:0/358377954 >> 172.21.15.16:0/358377954 conn(0x7f97e40cc870 msgr2=0x7f97e40cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:03:30.800 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:30.794+0000 7f97ebf18640 1 -- 172.21.15.16:0/358377954 shutdown_connections 2023-05-22T16:03:30.800 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:30.794+0000 7f97ebf18640 1 -- 172.21.15.16:0/358377954 wait complete. 2023-05-22T16:03:30.801 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:30.794+0000 7f97ebf18640 1 Processor -- start 2023-05-22T16:03:30.802 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:30.794+0000 7f97ebf18640 1 -- start start 2023-05-22T16:03:30.802 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:30.798+0000 7f97ebf18640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f97e406e740 0x7f97e40ac490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:30.802 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:30.798+0000 7f97ebf18640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f97e40cf870 0x7f97e41578e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:30.802 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:30.798+0000 7f97ebf18640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f97e415cfa0 0x7f97e40ce5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:30.802 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:30.798+0000 7f97ebf18640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97e40ccd30 con 0x7f97e406e740 2023-05-22T16:03:30.802 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:30.798+0000 7f97ebf18640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97e4157e20 con 0x7f97e40cf870 2023-05-22T16:03:30.803 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:30.798+0000 7f97ebf18640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97e40b2420 con 0x7f97e415cfa0 2023-05-22T16:03:30.803 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:30.798+0000 7f97e9cb4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f97e406e740 0x7f97e40ac490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:30.803 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:30.798+0000 7f97e9cb4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f97e406e740 0x7f97e40ac490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47942/0 (socket says 172.21.15.16:47942) 2023-05-22T16:03:30.803 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:30.798+0000 7f97e9cb4640 1 -- 172.21.15.16:0/189211033 learned_addr learned my addr 172.21.15.16:0/189211033 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:03:30.803 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:30.798+0000 7f97e94b3640 1 --2- 172.21.15.16:0/189211033 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f97e40cf870 0x7f97e41578e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:30.803 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:30.798+0000 7f97ea4b5640 1 --2- 172.21.15.16:0/189211033 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f97e415cfa0 0x7f97e40ce5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:30.803 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:30.798+0000 7f97e9cb4640 1 -- 172.21.15.16:0/189211033 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f97e415cfa0 msgr2=0x7f97e40ce5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:30.803 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:30.798+0000 7f97e9cb4640 1 --2- 172.21.15.16:0/189211033 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f97e415cfa0 0x7f97e40ce5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:30.804 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:30.798+0000 7f97e9cb4640 1 -- 172.21.15.16:0/189211033 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f97e40cf870 msgr2=0x7f97e41578e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:30.804 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:30.798+0000 7f97e9cb4640 1 --2- 172.21.15.16:0/189211033 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f97e40cf870 0x7f97e41578e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:30.804 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:30.798+0000 7f97e9cb4640 1 -- 172.21.15.16:0/189211033 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97dc009b30 con 0x7f97e406e740 2023-05-22T16:03:30.804 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:30.798+0000 7f97e94b3640 1 --2- 172.21.15.16:0/189211033 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f97e40cf870 0x7f97e41578e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T16:03:30.804 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:30.798+0000 7f97e9cb4640 1 --2- 172.21.15.16:0/189211033 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f97e406e740 0x7f97e40ac490 secure :-1 s=READY pgs=2201 cs=0 l=1 rev1=1 crypto rx=0x7f97e000c500 tx=0x7f97e000c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:03:30.804 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:30.798+0000 7f97da7fc640 1 -- 172.21.15.16:0/189211033 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97e0005220 con 0x7f97e406e740 2023-05-22T16:03:30.804 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:30.798+0000 7f97da7fc640 1 -- 172.21.15.16:0/189211033 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97e0005380 con 0x7f97e406e740 2023-05-22T16:03:30.805 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:30.798+0000 7f97ebf18640 1 -- 172.21.15.16:0/189211033 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97e40cb610 con 0x7f97e406e740 2023-05-22T16:03:30.805 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:30.798+0000 7f97da7fc640 1 -- 172.21.15.16:0/189211033 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97e0007270 con 0x7f97e406e740 2023-05-22T16:03:30.805 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:30.798+0000 7f97ebf18640 1 -- 172.21.15.16:0/189211033 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97e40cb330 con 0x7f97e406e740 2023-05-22T16:03:30.806 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:30.798+0000 7f97ebf18640 1 -- 172.21.15.16:0/189211033 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97e415d370 con 0x7f97e406e740 2023-05-22T16:03:30.806 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:30.798+0000 7f97da7fc640 1 -- 172.21.15.16:0/189211033 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 82) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f97e00073d0 con 0x7f97e406e740 2023-05-22T16:03:30.806 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:30.798+0000 7f97da7fc640 1 --2- 172.21.15.16:0/189211033 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f97c80654a0 0x7f97c8067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:30.807 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:30.798+0000 7f97da7fc640 1 -- 172.21.15.16:0/189211033 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(170..170 src has 1..170) v4 ==== 6618+0+0 (secure 0 0 0) 0x7f97e0082360 con 0x7f97e406e740 2023-05-22T16:03:30.807 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:30.798+0000 7f97e94b3640 1 --2- 172.21.15.16:0/189211033 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f97c80654a0 0x7f97c8067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:30.807 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:30.798+0000 7f97e94b3640 1 --2- 172.21.15.16:0/189211033 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f97c80654a0 0x7f97c8067950 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f97e4153df0 tx=0x7f97d4008040 comp rx=0 tx=0).ready entity=mgr.10032 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:03:30.808 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:30.802+0000 7f97da7fc640 1 -- 172.21.15.16:0/189211033 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f97e004fd60 con 0x7f97e406e740 2023-05-22T16:03:30.964 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:30.958+0000 7f97ebf18640 1 -- 172.21.15.16:0/189211033 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f97e406e040 con 0x7f97e406e740 2023-05-22T16:03:30.970 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:30.966+0000 7f97da7fc640 1 -- 172.21.15.16:0/189211033 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v56) v1 ==== 104+0+0 (secure 0 0 0) 0x7f97e406e040 con 0x7f97e406e740 2023-05-22T16:03:30.973 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:30.966+0000 7f97ebf18640 1 -- 172.21.15.16:0/189211033 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f97c80654a0 msgr2=0x7f97c8067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:30.973 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:30.966+0000 7f97ebf18640 1 --2- 172.21.15.16:0/189211033 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f97c80654a0 0x7f97c8067950 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f97e4153df0 tx=0x7f97d4008040 comp rx=0 tx=0).stop 2023-05-22T16:03:30.973 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:30.966+0000 7f97ebf18640 1 -- 172.21.15.16:0/189211033 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f97e406e740 msgr2=0x7f97e40ac490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:30.974 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:30.966+0000 7f97ebf18640 1 --2- 172.21.15.16:0/189211033 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f97e406e740 0x7f97e40ac490 secure :-1 s=READY pgs=2201 cs=0 l=1 rev1=1 crypto rx=0x7f97e000c500 tx=0x7f97e000c9c0 comp rx=0 tx=0).stop 2023-05-22T16:03:30.974 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:30.966+0000 7f97ebf18640 1 -- 172.21.15.16:0/189211033 shutdown_connections 2023-05-22T16:03:30.974 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:30.966+0000 7f97ebf18640 1 --2- 172.21.15.16:0/189211033 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f97e415cfa0 0x7f97e40ce5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:30.974 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:30.966+0000 7f97ebf18640 1 --2- 172.21.15.16:0/189211033 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f97e40cf870 0x7f97e41578e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:30.975 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:30.966+0000 7f97ebf18640 1 --2- 172.21.15.16:0/189211033 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f97c80654a0 0x7f97c8067950 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:30.975 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:30.966+0000 7f97ebf18640 1 --2- 172.21.15.16:0/189211033 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f97e406e740 0x7f97e40ac490 unknown :-1 s=CLOSED pgs=2201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:30.975 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:30.966+0000 7f97ebf18640 1 -- 172.21.15.16:0/189211033 >> 172.21.15.16:0/189211033 conn(0x7f97e40cc870 msgr2=0x7f97e4154170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:03:30.975 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:30.966+0000 7f97ebf18640 1 -- 172.21.15.16:0/189211033 shutdown_connections 2023-05-22T16:03:30.976 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:30.970+0000 7f97ebf18640 1 -- 172.21.15.16:0/189211033 wait complete. 2023-05-22T16:03:30.990 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2023-05-22T16:03:31.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:31.114+0000 7f4cb262a640 1 Processor -- start 2023-05-22T16:03:31.118 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:31.114+0000 7f4cb262a640 1 -- start start 2023-05-22T16:03:31.119 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:31.114+0000 7f4cb262a640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4cac164b10 0x7f4cac06db80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:31.119 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:31.114+0000 7f4cb262a640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4cac06e520 con 0x7f4cac06e0c0 2023-05-22T16:03:31.119 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:31.114+0000 7f4cb262a640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4cac06c670 con 0x7f4cac1603a0 2023-05-22T16:03:31.119 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:31.114+0000 7f4cb262a640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4cac164fb0 con 0x7f4cac164b10 2023-05-22T16:03:31.120 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:31.114+0000 7f4cb0bc7640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f4cac06e0c0 0x7f4cac0c2e60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:48114/0 (socket says 172.21.15.16:48114) 2023-05-22T16:03:31.120 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:31.114+0000 7f4cb0bc7640 1 -- 172.21.15.16:0/2193614037 learned_addr learned my addr 172.21.15.16:0/2193614037 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T16:03:31.120 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:31.114+0000 7f4cab7fe640 1 --2- 172.21.15.16:0/2193614037 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4cac164b10 0x7f4cac06db80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:31.121 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:31.114+0000 7f4caaffd640 1 -- 172.21.15.16:0/2193614037 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1016055106 0 0) 0x7f4cac06e520 con 0x7f4cac06e0c0 2023-05-22T16:03:31.121 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:31.114+0000 7f4caaffd640 1 -- 172.21.15.16:0/2193614037 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c94003530 con 0x7f4cac06e0c0 2023-05-22T16:03:31.121 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:31.114+0000 7f4caaffd640 1 -- 172.21.15.16:0/2193614037 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3816562509 0 0) 0x7f4c94003530 con 0x7f4cac06e0c0 2023-05-22T16:03:31.122 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:31.114+0000 7f4caaffd640 1 -- 172.21.15.16:0/2193614037 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4cac164b10 msgr2=0x7f4cac06db80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:03:31.122 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:31.114+0000 7f4caaffd640 1 --2- 172.21.15.16:0/2193614037 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4cac164b10 0x7f4cac06db80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:31.122 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:31.114+0000 7f4caaffd640 1 -- 172.21.15.16:0/2193614037 >> v1:172.21.15.62:6789/0 conn(0x7f4cac1603a0 legacy=0x7f4cac06bf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:31.122 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:31.114+0000 7f4caaffd640 1 -- 172.21.15.16:0/2193614037 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cac0ce160 con 0x7f4cac06e0c0 2023-05-22T16:03:31.122 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:31.114+0000 7f4caaffd640 1 -- 172.21.15.16:0/2193614037 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7f4c9c001680 con 0x7f4cac06e0c0 2023-05-22T16:03:31.122 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:31.114+0000 7f4caaffd640 1 -- 172.21.15.16:0/2193614037 >> v1:172.21.15.16:6789/0 conn(0x7f4cac06e0c0 legacy=0x7f4cac0c2e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:31.122 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:31.114+0000 7f4caaffd640 1 --2- 172.21.15.16:0/2193614037 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4c940039b0 0x7f4c94003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:31.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:31.114+0000 7f4caaffd640 1 --2- 172.21.15.16:0/2193614037 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4c940047d0 0x7f4c94004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:31.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:31.114+0000 7f4caaffd640 1 --2- 172.21.15.16:0/2193614037 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4cac164b10 0x7f4c94005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:31.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:31.114+0000 7f4caaffd640 1 -- 172.21.15.16:0/2193614037 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cac164fb0 con 0x7f4c940047d0 2023-05-22T16:03:31.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:31.114+0000 7f4caaffd640 1 -- 172.21.15.16:0/2193614037 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c9400ba40 con 0x7f4c940039b0 2023-05-22T16:03:31.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:31.114+0000 7f4caaffd640 1 -- 172.21.15.16:0/2193614037 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c9400bc00 con 0x7f4cac164b10 2023-05-22T16:03:31.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:31.114+0000 7f4cab7fe640 1 --2- 172.21.15.16:0/2193614037 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4c940047d0 0x7f4c94004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:31.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:31.114+0000 7f4cabfff640 1 --2- 172.21.15.16:0/2193614037 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4c940039b0 0x7f4c94003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:31.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:31.114+0000 7f4cb0bc7640 1 --2- 172.21.15.16:0/2193614037 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4cac164b10 0x7f4c94005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:31.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:31.114+0000 7f4cab7fe640 1 -- 172.21.15.16:0/2193614037 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4cac164b10 msgr2=0x7f4c94005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:03:31.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:31.114+0000 7f4cab7fe640 1 --2- 172.21.15.16:0/2193614037 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4cac164b10 0x7f4c94005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:31.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:31.114+0000 7f4cab7fe640 1 -- 172.21.15.16:0/2193614037 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4c940039b0 msgr2=0x7f4c94003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:03:31.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:31.114+0000 7f4cab7fe640 1 --2- 172.21.15.16:0/2193614037 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4c940039b0 0x7f4c94003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:31.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:31.114+0000 7f4cab7fe640 1 -- 172.21.15.16:0/2193614037 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c94003530 con 0x7f4c940047d0 2023-05-22T16:03:31.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:31.118+0000 7f4cab7fe640 1 --2- 172.21.15.16:0/2193614037 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4c940047d0 0x7f4c94004c40 secure :-1 s=READY pgs=2203 cs=0 l=1 rev1=1 crypto rx=0x7f4ca0009ae0 tx=0x7f4ca001ca50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4bfb9327b3071474 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:03:31.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:31.118+0000 7f4caaffd640 1 -- 172.21.15.16:0/2193614037 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ca00a5530 con 0x7f4c940047d0 2023-05-22T16:03:31.126 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:31.118+0000 7f4caaffd640 1 -- 172.21.15.16:0/2193614037 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ca00a5690 con 0x7f4c940047d0 2023-05-22T16:03:31.126 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:31.118+0000 7f4caaffd640 1 -- 172.21.15.16:0/2193614037 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ca00a5a40 con 0x7f4c940047d0 2023-05-22T16:03:31.126 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:31.118+0000 7f4cb262a640 1 -- 172.21.15.16:0/2193614037 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4c940047d0 msgr2=0x7f4c94004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:31.126 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:31.118+0000 7f4cb262a640 1 --2- 172.21.15.16:0/2193614037 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4c940047d0 0x7f4c94004c40 secure :-1 s=READY pgs=2203 cs=0 l=1 rev1=1 crypto rx=0x7f4ca0009ae0 tx=0x7f4ca001ca50 comp rx=0 tx=0).stop 2023-05-22T16:03:31.126 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:31.118+0000 7f4cb0bc7640 1 -- 172.21.15.16:0/2193614037 reap_dead start 2023-05-22T16:03:31.126 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:31.118+0000 7f4cb262a640 1 -- 172.21.15.16:0/2193614037 shutdown_connections 2023-05-22T16:03:31.126 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:31.118+0000 7f4cb262a640 1 -- 172.21.15.16:0/2193614037 >> 172.21.15.16:0/2193614037 conn(0x7f4cac06f5a0 msgr2=0x7f4cac06f9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:03:31.126 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:31.118+0000 7f4cb262a640 1 -- 172.21.15.16:0/2193614037 shutdown_connections 2023-05-22T16:03:31.127 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:31.118+0000 7f4cb262a640 1 -- 172.21.15.16:0/2193614037 wait complete. 2023-05-22T16:03:31.127 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:31.122+0000 7f4cb262a640 1 Processor -- start 2023-05-22T16:03:31.127 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:31.122+0000 7f4cb262a640 1 -- start start 2023-05-22T16:03:31.128 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:31.122+0000 7f4cb262a640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4c940047d0 0x7f4cac161bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:31.128 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:31.122+0000 7f4cb262a640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4cac163360 0x7f4cac14b020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:31.129 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:31.122+0000 7f4cb262a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4cac14b7c0 0x7f4cac0b07b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:31.129 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:31.122+0000 7f4cb262a640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cac162f40 con 0x7f4cac14b7c0 2023-05-22T16:03:31.129 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:31.122+0000 7f4cb262a640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cac0c5730 con 0x7f4cac163360 2023-05-22T16:03:31.129 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:31.122+0000 7f4cb262a640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4cac0b15e0 con 0x7f4c940047d0 2023-05-22T16:03:31.129 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:31.122+0000 7f4cb0bc7640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4cac14b7c0 0x7f4cac0b07b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:31.129 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:31.122+0000 7f4cb0bc7640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4cac14b7c0 0x7f4cac0b07b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47962/0 (socket says 172.21.15.16:47962) 2023-05-22T16:03:31.130 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:31.122+0000 7f4cb0bc7640 1 -- 172.21.15.16:0/1857257054 learned_addr learned my addr 172.21.15.16:0/1857257054 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:03:31.130 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:31.122+0000 7f4cabfff640 1 --2- 172.21.15.16:0/1857257054 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4c940047d0 0x7f4cac161bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:31.130 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:31.122+0000 7f4cab7fe640 1 --2- 172.21.15.16:0/1857257054 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4cac163360 0x7f4cac14b020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:31.130 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:31.122+0000 7f4cb0bc7640 1 -- 172.21.15.16:0/1857257054 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4c940047d0 msgr2=0x7f4cac161bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:31.130 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:31.122+0000 7f4cb0bc7640 1 --2- 172.21.15.16:0/1857257054 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4c940047d0 0x7f4cac161bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:31.131 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:31.122+0000 7f4cb0bc7640 1 -- 172.21.15.16:0/1857257054 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4cac163360 msgr2=0x7f4cac14b020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:31.131 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:31.122+0000 7f4cb0bc7640 1 --2- 172.21.15.16:0/1857257054 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4cac163360 0x7f4cac14b020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:31.131 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:31.122+0000 7f4cb0bc7640 1 -- 172.21.15.16:0/1857257054 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cac0ce720 con 0x7f4cac14b7c0 2023-05-22T16:03:31.131 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:31.122+0000 7f4cb0bc7640 1 --2- 172.21.15.16:0/1857257054 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4cac14b7c0 0x7f4cac0b07b0 secure :-1 s=READY pgs=2204 cs=0 l=1 rev1=1 crypto rx=0x7f4cac164ee0 tx=0x7f4c9c011850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:03:31.131 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:31.122+0000 7f4ca8ff9640 1 -- 172.21.15.16:0/1857257054 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c9c014370 con 0x7f4cac14b7c0 2023-05-22T16:03:31.131 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:31.122+0000 7f4ca8ff9640 1 -- 172.21.15.16:0/1857257054 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c9c0144f0 con 0x7f4cac14b7c0 2023-05-22T16:03:31.131 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:31.122+0000 7f4ca8ff9640 1 -- 172.21.15.16:0/1857257054 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c9c0148a0 con 0x7f4cac14b7c0 2023-05-22T16:03:31.131 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:31.122+0000 7f4cb262a640 1 -- 172.21.15.16:0/1857257054 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4cac0ce440 con 0x7f4cac14b7c0 2023-05-22T16:03:31.132 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:31.122+0000 7f4cb262a640 1 -- 172.21.15.16:0/1857257054 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4cac0cea00 con 0x7f4cac14b7c0 2023-05-22T16:03:31.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:31.126+0000 7f4cb262a640 1 -- 172.21.15.16:0/1857257054 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4cac163080 con 0x7f4cac14b7c0 2023-05-22T16:03:31.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:31.126+0000 7f4ca8ff9640 1 -- 172.21.15.16:0/1857257054 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 82) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f4c9c014a00 con 0x7f4cac14b7c0 2023-05-22T16:03:31.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:31.126+0000 7f4ca8ff9640 1 --2- 172.21.15.16:0/1857257054 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f4c980654d0 0x7f4c98067980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:31.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:31.126+0000 7f4cabfff640 1 --2- 172.21.15.16:0/1857257054 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f4c980654d0 0x7f4c98067980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:31.134 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:31.126+0000 7f4ca8ff9640 1 -- 172.21.15.16:0/1857257054 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(170..170 src has 1..170) v4 ==== 6618+0+0 (secure 0 0 0) 0x7f4c9c084c60 con 0x7f4cac14b7c0 2023-05-22T16:03:31.134 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:31.126+0000 7f4cabfff640 1 --2- 172.21.15.16:0/1857257054 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f4c980654d0 0x7f4c98067980 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f4ca4005210 tx=0x7f4ca4009040 comp rx=0 tx=0).ready entity=mgr.10032 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:03:31.134 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:31.130+0000 7f4ca8ff9640 1 -- 172.21.15.16:0/1857257054 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f4cac163080 con 0x7f4cac14b7c0 2023-05-22T16:03:31.292 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:31.286+0000 7f4cb262a640 1 -- 172.21.15.16:0/1857257054 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7f4cac163080 con 0x7f4cac14b7c0 2023-05-22T16:03:31.304 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:31.298+0000 7f4ca8ff9640 1 -- 172.21.15.16:0/1857257054 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v57) v1 ==== 115+0+0 (secure 0 0 0) 0x7f4cac163080 con 0x7f4cac14b7c0 2023-05-22T16:03:31.306 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:31.302+0000 7f4cb262a640 1 -- 172.21.15.16:0/1857257054 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f4c980654d0 msgr2=0x7f4c98067980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:31.307 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:31.302+0000 7f4cb262a640 1 --2- 172.21.15.16:0/1857257054 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f4c980654d0 0x7f4c98067980 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f4ca4005210 tx=0x7f4ca4009040 comp rx=0 tx=0).stop 2023-05-22T16:03:31.307 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:31.302+0000 7f4cb262a640 1 -- 172.21.15.16:0/1857257054 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4cac14b7c0 msgr2=0x7f4cac0b07b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:31.307 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:31.302+0000 7f4cb262a640 1 --2- 172.21.15.16:0/1857257054 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4cac14b7c0 0x7f4cac0b07b0 secure :-1 s=READY pgs=2204 cs=0 l=1 rev1=1 crypto rx=0x7f4cac164ee0 tx=0x7f4c9c011850 comp rx=0 tx=0).stop 2023-05-22T16:03:31.307 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:31.302+0000 7f4cb262a640 1 -- 172.21.15.16:0/1857257054 shutdown_connections 2023-05-22T16:03:31.307 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:31.302+0000 7f4cb262a640 1 --2- 172.21.15.16:0/1857257054 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f4c980654d0 0x7f4c98067980 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:31.307 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:31.302+0000 7f4cb262a640 1 --2- 172.21.15.16:0/1857257054 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4cac14b7c0 0x7f4cac0b07b0 unknown :-1 s=CLOSED pgs=2204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:31.307 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:31.302+0000 7f4cb262a640 1 --2- 172.21.15.16:0/1857257054 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4cac163360 0x7f4cac14b020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:31.308 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:31.302+0000 7f4cb262a640 1 --2- 172.21.15.16:0/1857257054 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4c940047d0 0x7f4cac161bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:31.308 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:31.302+0000 7f4cb262a640 1 -- 172.21.15.16:0/1857257054 >> 172.21.15.16:0/1857257054 conn(0x7f4cac06f5a0 msgr2=0x7f4cac0cf050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:03:31.308 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:31.302+0000 7f4cb262a640 1 -- 172.21.15.16:0/1857257054 shutdown_connections 2023-05-22T16:03:31.308 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:31.302+0000 7f4cb262a640 1 -- 172.21.15.16:0/1857257054 wait complete. 2023-05-22T16:03:31.324 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_mgr_restart' 2023-05-22T16:03:31.451 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:31.446+0000 7faa13b72640 1 Processor -- start 2023-05-22T16:03:31.452 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:31.446+0000 7faa13b72640 1 -- start start 2023-05-22T16:03:31.454 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:31.450+0000 7faa13b72640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7faa0c15c2f0 0x7faa0c162e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:31.455 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:31.450+0000 7faa13b72640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa0c158eb0 con 0x7faa0c070600 2023-05-22T16:03:31.455 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:31.450+0000 7faa13b72640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa0c06cf20 con 0x7faa0c06dab0 2023-05-22T16:03:31.455 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:31.450+0000 7faa13b72640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa0c06de80 con 0x7faa0c15c2f0 2023-05-22T16:03:31.456 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:31.450+0000 7faa1110d640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7faa0c070600 0x7faa0c15bbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:48116/0 (socket says 172.21.15.16:48116) 2023-05-22T16:03:31.456 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:31.450+0000 7faa1110d640 1 -- 172.21.15.16:0/2599571382 learned_addr learned my addr 172.21.15.16:0/2599571382 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T16:03:31.456 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:31.450+0000 7faa1090c640 1 -- 172.21.15.16:0/2599571382 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3073844900 0 0) 0x7faa0c158eb0 con 0x7faa0c070600 2023-05-22T16:03:31.456 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:31.450+0000 7faa1090c640 1 -- 172.21.15.16:0/2599571382 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9f4003530 con 0x7faa0c070600 2023-05-22T16:03:31.456 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:31.450+0000 7faa1210f640 1 --2- 172.21.15.16:0/2599571382 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7faa0c15c2f0 0x7faa0c162e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:31.457 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:31.450+0000 7faa1090c640 1 -- 172.21.15.16:0/2599571382 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3094272284 0 0) 0x7fa9f4003530 con 0x7faa0c070600 2023-05-22T16:03:31.457 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:31.450+0000 7faa1090c640 1 -- 172.21.15.16:0/2599571382 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7faa0c15c2f0 msgr2=0x7faa0c162e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:03:31.457 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:31.450+0000 7faa1090c640 1 --2- 172.21.15.16:0/2599571382 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7faa0c15c2f0 0x7faa0c162e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:31.457 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:31.450+0000 7faa1090c640 1 -- 172.21.15.16:0/2599571382 >> v1:172.21.15.62:6789/0 conn(0x7faa0c06dab0 legacy=0x7faa0c06fef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:03:31.457 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:31.450+0000 7faa1090c640 1 -- 172.21.15.16:0/2599571382 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa0c15c950 con 0x7faa0c070600 2023-05-22T16:03:31.458 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:31.450+0000 7faa1090c640 1 -- 172.21.15.16:0/2599571382 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7faa00001680 con 0x7faa0c070600 2023-05-22T16:03:31.458 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:31.450+0000 7faa1090c640 1 -- 172.21.15.16:0/2599571382 >> v1:172.21.15.16:6789/0 conn(0x7faa0c070600 legacy=0x7faa0c15bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:31.458 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:31.450+0000 7faa1090c640 1 --2- 172.21.15.16:0/2599571382 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa9f40039b0 0x7fa9f4003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:31.458 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:31.450+0000 7faa1090c640 1 --2- 172.21.15.16:0/2599571382 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7faa0c15c2f0 0x7fa9f40047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:31.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:31.450+0000 7faa1090c640 1 --2- 172.21.15.16:0/2599571382 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa9f4004d10 0x7fa9f4005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:31.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:31.450+0000 7faa1090c640 1 -- 172.21.15.16:0/2599571382 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa0c06de80 con 0x7fa9f40039b0 2023-05-22T16:03:31.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:31.450+0000 7faa1090c640 1 -- 172.21.15.16:0/2599571382 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9f400baa0 con 0x7fa9f4004d10 2023-05-22T16:03:31.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:31.450+0000 7faa1090c640 1 -- 172.21.15.16:0/2599571382 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9f400bc60 con 0x7faa0c15c2f0 2023-05-22T16:03:31.460 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:31.450+0000 7faa1190e640 1 --2- 172.21.15.16:0/2599571382 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa9f40039b0 0x7fa9f4003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:31.460 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:31.450+0000 7faa1110d640 1 --2- 172.21.15.16:0/2599571382 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7faa0c15c2f0 0x7fa9f40047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:31.460 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:31.450+0000 7faa1190e640 1 -- 172.21.15.16:0/2599571382 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7faa0c15c2f0 msgr2=0x7fa9f40047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:03:31.461 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:31.450+0000 7faa1190e640 1 --2- 172.21.15.16:0/2599571382 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7faa0c15c2f0 0x7fa9f40047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:31.461 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:31.450+0000 7faa1190e640 1 -- 172.21.15.16:0/2599571382 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa9f4004d10 msgr2=0x7fa9f4005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:03:31.461 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:31.450+0000 7faa1190e640 1 --2- 172.21.15.16:0/2599571382 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa9f4004d10 0x7fa9f4005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:31.462 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:31.450+0000 7faa1190e640 1 -- 172.21.15.16:0/2599571382 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9f4003530 con 0x7fa9f40039b0 2023-05-22T16:03:31.462 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:31.450+0000 7faa1110d640 1 --2- 172.21.15.16:0/2599571382 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7faa0c15c2f0 0x7fa9f40047d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T16:03:31.466 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:31.450+0000 7faa1190e640 1 --2- 172.21.15.16:0/2599571382 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa9f40039b0 0x7fa9f4003da0 secure :-1 s=READY pgs=2206 cs=0 l=1 rev1=1 crypto rx=0x7faa04001d50 tx=0x7faa0401c630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cff622b590ec4ec5 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:03:31.467 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:31.454+0000 7faa1090c640 1 -- 172.21.15.16:0/2599571382 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa040039a0 con 0x7fa9f40039b0 2023-05-22T16:03:31.467 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:31.454+0000 7faa1090c640 1 -- 172.21.15.16:0/2599571382 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa04003b00 con 0x7fa9f40039b0 2023-05-22T16:03:31.467 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:31.454+0000 7faa1090c640 1 -- 172.21.15.16:0/2599571382 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa04005070 con 0x7fa9f40039b0 2023-05-22T16:03:31.467 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:31.454+0000 7faa13b72640 1 -- 172.21.15.16:0/2599571382 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa9f40039b0 msgr2=0x7fa9f4003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:31.468 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:31.454+0000 7faa13b72640 1 --2- 172.21.15.16:0/2599571382 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa9f40039b0 0x7fa9f4003da0 secure :-1 s=READY pgs=2206 cs=0 l=1 rev1=1 crypto rx=0x7faa04001d50 tx=0x7faa0401c630 comp rx=0 tx=0).stop 2023-05-22T16:03:31.468 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:31.454+0000 7faa1210f640 1 -- 172.21.15.16:0/2599571382 reap_dead start 2023-05-22T16:03:31.468 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:31.454+0000 7faa13b72640 1 -- 172.21.15.16:0/2599571382 shutdown_connections 2023-05-22T16:03:31.468 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:31.454+0000 7faa13b72640 1 -- 172.21.15.16:0/2599571382 >> 172.21.15.16:0/2599571382 conn(0x7faa0c0b1110 msgr2=0x7faa0c0ce830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:03:31.469 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:31.454+0000 7faa13b72640 1 -- 172.21.15.16:0/2599571382 shutdown_connections 2023-05-22T16:03:31.469 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:31.454+0000 7faa13b72640 1 -- 172.21.15.16:0/2599571382 wait complete. 2023-05-22T16:03:31.469 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:31.454+0000 7faa13b72640 1 Processor -- start 2023-05-22T16:03:31.469 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:31.454+0000 7faa13b72640 1 -- start start 2023-05-22T16:03:31.469 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:31.454+0000 7faa13b72640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faa0c156ef0 0x7faa0c1572c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:31.470 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:31.454+0000 7faa13b72640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faa0c0b17c0 0x7faa0c0b3f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:31.470 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:31.454+0000 7faa13b72640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7faa0c0b4440 0x7faa0c13b1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:31.470 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:31.454+0000 7faa13b72640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa0c15c770 con 0x7faa0c156ef0 2023-05-22T16:03:31.470 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:31.454+0000 7faa13b72640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa0c157a50 con 0x7faa0c0b17c0 2023-05-22T16:03:31.470 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:31.454+0000 7faa13b72640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa0c0b3a70 con 0x7faa0c0b4440 2023-05-22T16:03:31.470 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:31.454+0000 7faa1190e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faa0c156ef0 0x7faa0c1572c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:31.470 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:31.454+0000 7faa1190e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faa0c156ef0 0x7faa0c1572c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47992/0 (socket says 172.21.15.16:47992) 2023-05-22T16:03:31.471 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:31.454+0000 7faa1190e640 1 -- 172.21.15.16:0/1283838602 learned_addr learned my addr 172.21.15.16:0/1283838602 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:03:31.471 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:31.454+0000 7faa1210f640 1 --2- 172.21.15.16:0/1283838602 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7faa0c0b4440 0x7faa0c13b1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:31.471 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:31.454+0000 7faa1190e640 1 -- 172.21.15.16:0/1283838602 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7faa0c0b4440 msgr2=0x7faa0c13b1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:31.471 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:31.454+0000 7faa1190e640 1 --2- 172.21.15.16:0/1283838602 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7faa0c0b4440 0x7faa0c13b1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:31.471 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:31.454+0000 7faa1190e640 1 -- 172.21.15.16:0/1283838602 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faa0c0b17c0 msgr2=0x7faa0c0b3f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:31.471 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:31.454+0000 7faa1190e640 1 --2- 172.21.15.16:0/1283838602 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faa0c0b17c0 0x7faa0c0b3f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:31.471 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:31.454+0000 7faa1190e640 1 -- 172.21.15.16:0/1283838602 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa0c159340 con 0x7faa0c156ef0 2023-05-22T16:03:31.472 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:31.454+0000 7faa1210f640 1 --2- 172.21.15.16:0/1283838602 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7faa0c0b4440 0x7faa0c13b1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T16:03:31.472 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:31.454+0000 7faa1190e640 1 --2- 172.21.15.16:0/1283838602 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faa0c156ef0 0x7faa0c1572c0 secure :-1 s=READY pgs=2207 cs=0 l=1 rev1=1 crypto rx=0x7faa04001d50 tx=0x7fa9f4005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:03:31.472 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:31.454+0000 7faa0a7fc640 1 -- 172.21.15.16:0/1283838602 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa04005070 con 0x7faa0c156ef0 2023-05-22T16:03:31.472 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:31.454+0000 7faa0a7fc640 1 -- 172.21.15.16:0/1283838602 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa040051d0 con 0x7faa0c156ef0 2023-05-22T16:03:31.472 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:31.454+0000 7faa13b72640 1 -- 172.21.15.16:0/1283838602 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa0c158510 con 0x7faa0c156ef0 2023-05-22T16:03:31.472 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:31.454+0000 7faa0a7fc640 1 -- 172.21.15.16:0/1283838602 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa04005580 con 0x7faa0c156ef0 2023-05-22T16:03:31.473 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:31.454+0000 7faa13b72640 1 -- 172.21.15.16:0/1283838602 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa0c1594d0 con 0x7faa0c156ef0 2023-05-22T16:03:31.473 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:31.458+0000 7faa13b72640 1 -- 172.21.15.16:0/1283838602 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa0c06b0d0 con 0x7faa0c156ef0 2023-05-22T16:03:31.473 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:31.458+0000 7faa0a7fc640 1 -- 172.21.15.16:0/1283838602 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 82) v1 ==== 84332+0+0 (secure 0 0 0) 0x7faa040056e0 con 0x7faa0c156ef0 2023-05-22T16:03:31.473 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:31.458+0000 7faa0a7fc640 1 --2- 172.21.15.16:0/1283838602 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fa9f80654a0 0x7fa9f8067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:31.473 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:31.458+0000 7faa0a7fc640 1 -- 172.21.15.16:0/1283838602 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(170..170 src has 1..170) v4 ==== 6618+0+0 (secure 0 0 0) 0x7faa0411d440 con 0x7faa0c156ef0 2023-05-22T16:03:31.473 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:31.458+0000 7faa1110d640 1 --2- 172.21.15.16:0/1283838602 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fa9f80654a0 0x7fa9f8067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:31.473 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:31.458+0000 7faa1110d640 1 --2- 172.21.15.16:0/1283838602 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fa9f80654a0 0x7fa9f8067950 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7faa00000fe0 tx=0x7faa00009040 comp rx=0 tx=0).ready entity=mgr.10032 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:03:31.474 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:31.462+0000 7faa0a7fc640 1 -- 172.21.15.16:0/1283838602 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7faa040eac70 con 0x7faa0c156ef0 2023-05-22T16:03:31.679 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:31.674+0000 7faa13b72640 1 -- 172.21.15.16:0/1283838602 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_mgr_restart"]} v 0) v1 -- 0x7faa0c04dbe0 con 0x7faa0c156ef0 2023-05-22T16:03:32.299 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:32.294+0000 7faa0a7fc640 1 -- 172.21.15.16:0/1283838602 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_mgr_restart"]}]=0 v1295) v1 ==== 165+0+0 (secure 0 0 0) 0x7faa040ee290 con 0x7faa0c156ef0 2023-05-22T16:03:32.302 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:32.298+0000 7faa13b72640 1 -- 172.21.15.16:0/1283838602 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fa9f80654a0 msgr2=0x7fa9f8067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:32.303 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:32.298+0000 7faa13b72640 1 --2- 172.21.15.16:0/1283838602 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fa9f80654a0 0x7fa9f8067950 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7faa00000fe0 tx=0x7faa00009040 comp rx=0 tx=0).stop 2023-05-22T16:03:32.303 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:32.298+0000 7faa13b72640 1 -- 172.21.15.16:0/1283838602 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faa0c156ef0 msgr2=0x7faa0c1572c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:32.303 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:32.298+0000 7faa13b72640 1 --2- 172.21.15.16:0/1283838602 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faa0c156ef0 0x7faa0c1572c0 secure :-1 s=READY pgs=2207 cs=0 l=1 rev1=1 crypto rx=0x7faa04001d50 tx=0x7fa9f4005800 comp rx=0 tx=0).stop 2023-05-22T16:03:32.303 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:32.298+0000 7faa13b72640 1 -- 172.21.15.16:0/1283838602 shutdown_connections 2023-05-22T16:03:32.303 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:32.298+0000 7faa13b72640 1 --2- 172.21.15.16:0/1283838602 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7faa0c0b4440 0x7faa0c13b1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:32.303 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:32.298+0000 7faa13b72640 1 --2- 172.21.15.16:0/1283838602 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faa0c0b17c0 0x7faa0c0b3f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:32.304 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:32.298+0000 7faa13b72640 1 --2- 172.21.15.16:0/1283838602 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fa9f80654a0 0x7fa9f8067950 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:32.304 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:32.298+0000 7faa13b72640 1 --2- 172.21.15.16:0/1283838602 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faa0c156ef0 0x7faa0c1572c0 unknown :-1 s=CLOSED pgs=2207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:32.304 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:32.298+0000 7faa13b72640 1 -- 172.21.15.16:0/1283838602 >> 172.21.15.16:0/1283838602 conn(0x7faa0c0b1110 msgr2=0x7faa0c069720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:03:32.304 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:32.298+0000 7faa13b72640 1 -- 172.21.15.16:0/1283838602 shutdown_connections 2023-05-22T16:03:32.304 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:32.298+0000 7faa13b72640 1 -- 172.21.15.16:0/1283838602 wait complete. 2023-05-22T16:03:32.322 INFO:tasks.cephfs_test_runner:Test that snap schedule is resumed after mgr restart ... ok 2023-05-22T16:03:32.322 INFO:tasks.cephfs_test_runner:Starting test: test_snap_schedule_with_retention (tasks.cephfs.test_snap_schedules.TestSnapSchedules) Test scheduled snapshots along with rentention policy 2023-05-22T16:03:32.323 INFO:tasks.cephfs_test_runner:test_snap_schedule_with_retention (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2023-05-22T16:03:32.323 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_retention' 2023-05-22T16:03:32.452 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:32.446+0000 7f0068a33640 1 Processor -- start 2023-05-22T16:03:32.453 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:32.450+0000 7f0068a33640 1 -- start start 2023-05-22T16:03:32.453 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:32.450+0000 7f0068a33640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f00640b0250 0x7f0064159330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:32.453 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:32.450+0000 7f0068a33640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f0064159870 0x7f00640c8ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:32.454 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:32.450+0000 7f0068a33640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f006415a0a0 con 0x7f00641531b0 2023-05-22T16:03:32.454 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:32.450+0000 7f0068a33640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0064153580 con 0x7f0064159870 2023-05-22T16:03:32.454 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:32.450+0000 7f0068a33640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f006415d5e0 con 0x7f00640b0250 2023-05-22T16:03:32.455 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:32.450+0000 7f0061d9b640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f00640b0250 0x7f0064159330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:32.455 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:32.450+0000 7f006259c640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f00641531b0 0x7f00640afb40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.16:48980/0 (socket says 172.21.15.16:48980) 2023-05-22T16:03:32.455 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:32.450+0000 7f006259c640 1 -- 172.21.15.16:0/753750777 learned_addr learned my addr 172.21.15.16:0/753750777 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T16:03:32.455 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:32.450+0000 7f0062d9d640 1 --2- 172.21.15.16:0/753750777 >> v2:172.21.15.62:3300/0 conn(0x7f0064159870 0x7f00640c8ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:32.455 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:32.450+0000 7f006159a640 1 -- 172.21.15.16:0/753750777 <== mon.1 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2630751822 0 0) 0x7f006415a0a0 con 0x7f00641531b0 2023-05-22T16:03:32.455 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:32.450+0000 7f006159a640 1 -- 172.21.15.16:0/753750777 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0050003530 con 0x7f00641531b0 2023-05-22T16:03:32.456 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:32.450+0000 7f0061d9b640 1 -- 172.21.15.16:0/753750777 >> v2:172.21.15.62:3300/0 conn(0x7f0064159870 msgr2=0x7f00640c8ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:03:32.456 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:32.450+0000 7f0061d9b640 1 --2- 172.21.15.16:0/753750777 >> v2:172.21.15.62:3300/0 conn(0x7f0064159870 0x7f00640c8ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:32.456 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:32.450+0000 7f0061d9b640 1 -- 172.21.15.16:0/753750777 >> v1:172.21.15.62:6789/0 conn(0x7f00641531b0 legacy=0x7f00640afb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:32.456 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:32.450+0000 7f0061d9b640 1 -- 172.21.15.16:0/753750777 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f006415c820 con 0x7f00640b0250 2023-05-22T16:03:32.457 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:32.454+0000 7f0061d9b640 1 --2- 172.21.15.16:0/753750777 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f00640b0250 0x7f0064159330 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f004c0097c0 tx=0x7f004c01c450 comp rx=0 tx=0).ready entity=mon.2 client_cookie=baf20431e21d9967 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:03:32.457 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:32.454+0000 7f006159a640 1 -- 172.21.15.16:0/753750777 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f004c0ab650 con 0x7f00640b0250 2023-05-22T16:03:32.457 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:32.454+0000 7f006159a640 1 -- 172.21.15.16:0/753750777 <== mon.2 v2:172.21.15.62:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f004c0ab7b0 con 0x7f00640b0250 2023-05-22T16:03:32.457 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:32.454+0000 7f006159a640 1 -- 172.21.15.16:0/753750777 <== mon.2 v2:172.21.15.62:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f004c0abb20 con 0x7f00640b0250 2023-05-22T16:03:32.458 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:32.454+0000 7f0068a33640 1 -- 172.21.15.16:0/753750777 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f00640b0250 msgr2=0x7f0064159330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:32.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:32.454+0000 7f0068a33640 1 --2- 172.21.15.16:0/753750777 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f00640b0250 0x7f0064159330 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f004c0097c0 tx=0x7f004c01c450 comp rx=0 tx=0).stop 2023-05-22T16:03:32.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:32.454+0000 7f0068a33640 1 -- 172.21.15.16:0/753750777 shutdown_connections 2023-05-22T16:03:32.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:32.454+0000 7f0068a33640 1 --2- 172.21.15.16:0/753750777 >> v2:172.21.15.62:3300/0 conn(0x7f0064159870 0x7f00640c8ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:32.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:32.454+0000 7f0068a33640 1 --2- 172.21.15.16:0/753750777 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f00640b0250 0x7f0064159330 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:32.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:32.454+0000 7f0068a33640 1 -- 172.21.15.16:0/753750777 >> 172.21.15.16:0/753750777 conn(0x7f0064069720 msgr2=0x7f0064069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:03:32.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:32.454+0000 7f0068a33640 1 -- 172.21.15.16:0/753750777 shutdown_connections 2023-05-22T16:03:32.460 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:32.454+0000 7f0068a33640 1 -- 172.21.15.16:0/753750777 wait complete. 2023-05-22T16:03:32.461 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:32.454+0000 7f0068a33640 1 Processor -- start 2023-05-22T16:03:32.461 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:32.454+0000 7f0068a33640 1 -- start start 2023-05-22T16:03:32.462 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:32.454+0000 7f0068a33640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f00640b0250 0x7f00640c0d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:32.462 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:32.454+0000 7f0068a33640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f00641531b0 0x7f00640cc4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:32.462 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:32.454+0000 7f0068a33640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0064159870 0x7f00640cca30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:32.462 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:32.454+0000 7f0068a33640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0064151400 con 0x7f0064159870 2023-05-22T16:03:32.462 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:32.454+0000 7f0068a33640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00641516e0 con 0x7f00640b0250 2023-05-22T16:03:32.462 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:32.454+0000 7f0068a33640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00641519c0 con 0x7f00641531b0 2023-05-22T16:03:32.462 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:32.458+0000 7f0062d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0064159870 0x7f00640cca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:32.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:32.458+0000 7f0062d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0064159870 0x7f00640cca30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48006/0 (socket says 172.21.15.16:48006) 2023-05-22T16:03:32.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:32.458+0000 7f0062d9d640 1 -- 172.21.15.16:0/378407471 learned_addr learned my addr 172.21.15.16:0/378407471 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:03:32.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:32.458+0000 7f006259c640 1 --2- 172.21.15.16:0/378407471 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f00640b0250 0x7f00640c0d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:32.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:32.458+0000 7f0061d9b640 1 --2- 172.21.15.16:0/378407471 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f00641531b0 0x7f00640cc4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:32.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:32.458+0000 7f0062d9d640 1 -- 172.21.15.16:0/378407471 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f00641531b0 msgr2=0x7f00640cc4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:32.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:32.458+0000 7f0062d9d640 1 --2- 172.21.15.16:0/378407471 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f00641531b0 0x7f00640cc4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:32.464 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:32.458+0000 7f0062d9d640 1 -- 172.21.15.16:0/378407471 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f00640b0250 msgr2=0x7f00640c0d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:32.464 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:32.458+0000 7f0062d9d640 1 --2- 172.21.15.16:0/378407471 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f00640b0250 0x7f00640c0d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:32.464 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:32.458+0000 7f0062d9d640 1 -- 172.21.15.16:0/378407471 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f004c009420 con 0x7f0064159870 2023-05-22T16:03:32.464 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:32.458+0000 7f0062d9d640 1 --2- 172.21.15.16:0/378407471 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0064159870 0x7f00640cca30 secure :-1 s=READY pgs=2208 cs=0 l=1 rev1=1 crypto rx=0x7f005c00efc0 tx=0x7f005c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:03:32.464 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:32.458+0000 7f005affd640 1 -- 172.21.15.16:0/378407471 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f005c014480 con 0x7f0064159870 2023-05-22T16:03:32.464 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:32.458+0000 7f005affd640 1 -- 172.21.15.16:0/378407471 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f005c00f040 con 0x7f0064159870 2023-05-22T16:03:32.465 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:32.458+0000 7f0068a33640 1 -- 172.21.15.16:0/378407471 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0064159c40 con 0x7f0064159870 2023-05-22T16:03:32.465 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:32.458+0000 7f005affd640 1 -- 172.21.15.16:0/378407471 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f005c014820 con 0x7f0064159870 2023-05-22T16:03:32.465 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:32.458+0000 7f0068a33640 1 -- 172.21.15.16:0/378407471 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f006415d0c0 con 0x7f0064159870 2023-05-22T16:03:32.466 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:32.458+0000 7f0068a33640 1 -- 172.21.15.16:0/378407471 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0064159dd0 con 0x7f0064159870 2023-05-22T16:03:32.466 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:32.458+0000 7f005affd640 1 -- 172.21.15.16:0/378407471 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 82) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f005c014980 con 0x7f0064159870 2023-05-22T16:03:32.466 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:32.458+0000 7f005affd640 1 --2- 172.21.15.16:0/378407471 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f00440654a0 0x7f0044067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:32.466 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:32.458+0000 7f006259c640 1 --2- 172.21.15.16:0/378407471 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f00440654a0 0x7f0044067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:32.466 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:32.458+0000 7f005affd640 1 -- 172.21.15.16:0/378407471 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(170..170 src has 1..170) v4 ==== 6618+0+0 (secure 0 0 0) 0x7f005c083740 con 0x7f0064159870 2023-05-22T16:03:32.467 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:32.458+0000 7f006259c640 1 --2- 172.21.15.16:0/378407471 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f00440654a0 0x7f0044067950 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f00540011f0 tx=0x7f0054007d40 comp rx=0 tx=0).ready entity=mgr.10032 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:03:32.467 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:32.462+0000 7f005affd640 1 -- 172.21.15.16:0/378407471 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f005c0510b0 con 0x7f0064159870 2023-05-22T16:03:32.676 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:32.670+0000 7f0068a33640 1 -- 172.21.15.16:0/378407471 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_retention"]} v 0) v1 -- 0x7f006404dbe0 con 0x7f0064159870 2023-05-22T16:03:33.301 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:33.298+0000 7f005affd640 1 -- 172.21.15.16:0/378407471 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_retention"]}]=0 v1296) v1 ==== 166+0+0 (secure 0 0 0) 0x7f005c0546d0 con 0x7f0064159870 2023-05-22T16:03:33.305 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:33.302+0000 7f0068a33640 1 -- 172.21.15.16:0/378407471 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f00440654a0 msgr2=0x7f0044067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:33.306 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:33.302+0000 7f0068a33640 1 --2- 172.21.15.16:0/378407471 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f00440654a0 0x7f0044067950 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f00540011f0 tx=0x7f0054007d40 comp rx=0 tx=0).stop 2023-05-22T16:03:33.306 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:33.302+0000 7f0068a33640 1 -- 172.21.15.16:0/378407471 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0064159870 msgr2=0x7f00640cca30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:33.306 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:33.302+0000 7f0068a33640 1 --2- 172.21.15.16:0/378407471 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0064159870 0x7f00640cca30 secure :-1 s=READY pgs=2208 cs=0 l=1 rev1=1 crypto rx=0x7f005c00efc0 tx=0x7f005c00c490 comp rx=0 tx=0).stop 2023-05-22T16:03:33.306 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:33.302+0000 7f0068a33640 1 -- 172.21.15.16:0/378407471 shutdown_connections 2023-05-22T16:03:33.306 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:33.302+0000 7f0068a33640 1 --2- 172.21.15.16:0/378407471 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f00440654a0 0x7f0044067950 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:33.306 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:33.302+0000 7f0068a33640 1 --2- 172.21.15.16:0/378407471 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0064159870 0x7f00640cca30 unknown :-1 s=CLOSED pgs=2208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:33.306 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:33.302+0000 7f0068a33640 1 --2- 172.21.15.16:0/378407471 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f00641531b0 0x7f00640cc4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:33.307 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:33.302+0000 7f0068a33640 1 --2- 172.21.15.16:0/378407471 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f00640b0250 0x7f00640c0d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:33.307 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:33.302+0000 7f0068a33640 1 -- 172.21.15.16:0/378407471 >> 172.21.15.16:0/378407471 conn(0x7f0064069720 msgr2=0x7f006406fd30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:03:33.307 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:33.302+0000 7f0068a33640 1 -- 172.21.15.16:0/378407471 shutdown_connections 2023-05-22T16:03:33.307 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:33.302+0000 7f0068a33640 1 -- 172.21.15.16:0/378407471 wait complete. 2023-05-22T16:03:33.328 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-05-22T16:03:33.457 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:33.454+0000 7f9ae97b9640 1 Processor -- start 2023-05-22T16:03:33.457 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:33.454+0000 7f9ae97b9640 1 -- start start 2023-05-22T16:03:33.458 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:33.454+0000 7f9ae97b9640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f9ae40c67d0 0x7f9ae40cf730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:33.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:33.454+0000 7f9ae97b9640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ae40cfc70 con 0x7f9ae40b2880 2023-05-22T16:03:33.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:33.454+0000 7f9ae97b9640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ae40c6ba0 con 0x7f9ae40cc6f0 2023-05-22T16:03:33.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:33.454+0000 7f9ae97b9640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ae415a300 con 0x7f9ae40c67d0 2023-05-22T16:03:33.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:33.454+0000 7f9ae27fc640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f9ae40b2880 0x7f9ae40cbfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:48124/0 (socket says 172.21.15.16:48124) 2023-05-22T16:03:33.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:33.454+0000 7f9ae27fc640 1 -- 172.21.15.16:0/4143294134 learned_addr learned my addr 172.21.15.16:0/4143294134 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T16:03:33.461 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:33.454+0000 7f9ae2ffd640 1 --2- 172.21.15.16:0/4143294134 >> v2:172.21.15.62:3300/0 conn(0x7f9ae40c67d0 0x7f9ae40cf730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:33.461 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:33.454+0000 7f9ae1ffb640 1 -- 172.21.15.16:0/4143294134 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1625229052 0 0) 0x7f9ae40cfc70 con 0x7f9ae40b2880 2023-05-22T16:03:33.461 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:33.454+0000 7f9ae1ffb640 1 -- 172.21.15.16:0/4143294134 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9acc003530 con 0x7f9ae40b2880 2023-05-22T16:03:33.461 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:33.454+0000 7f9ae1ffb640 1 -- 172.21.15.16:0/4143294134 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1291556304 0 0) 0x7f9acc003530 con 0x7f9ae40b2880 2023-05-22T16:03:33.461 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:33.454+0000 7f9ae1ffb640 1 -- 172.21.15.16:0/4143294134 >> v2:172.21.15.62:3300/0 conn(0x7f9ae40c67d0 msgr2=0x7f9ae40cf730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:03:33.462 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:33.454+0000 7f9ae1ffb640 1 --2- 172.21.15.16:0/4143294134 >> v2:172.21.15.62:3300/0 conn(0x7f9ae40c67d0 0x7f9ae40cf730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:33.462 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:33.454+0000 7f9ae1ffb640 1 -- 172.21.15.16:0/4143294134 >> v1:172.21.15.62:6789/0 conn(0x7f9ae40cc6f0 legacy=0x7f9ae40c3db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:33.462 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:33.454+0000 7f9ae1ffb640 1 -- 172.21.15.16:0/4143294134 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ae40cfef0 con 0x7f9ae40b2880 2023-05-22T16:03:33.462 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:33.454+0000 7f9ae1ffb640 1 -- 172.21.15.16:0/4143294134 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7f9ad8001680 con 0x7f9ae40b2880 2023-05-22T16:03:33.462 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:33.454+0000 7f9ae1ffb640 1 -- 172.21.15.16:0/4143294134 >> v1:172.21.15.16:6789/0 conn(0x7f9ae40b2880 legacy=0x7f9ae40cbfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:33.462 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:33.454+0000 7f9ae1ffb640 1 --2- 172.21.15.16:0/4143294134 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9acc003980 0x7f9acc003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:33.462 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:33.454+0000 7f9ae1ffb640 1 --2- 172.21.15.16:0/4143294134 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9acc0047d0 0x7f9acc004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:33.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:33.454+0000 7f9ae1ffb640 1 --2- 172.21.15.16:0/4143294134 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9acc005180 0x7f9acc00b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:33.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:33.454+0000 7f9ae1ffb640 1 -- 172.21.15.16:0/4143294134 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ae415a300 con 0x7f9acc005180 2023-05-22T16:03:33.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:33.454+0000 7f9ae1ffb640 1 -- 172.21.15.16:0/4143294134 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9acc00c090 con 0x7f9acc0047d0 2023-05-22T16:03:33.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:33.454+0000 7f9ae1ffb640 1 -- 172.21.15.16:0/4143294134 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9acc00c250 con 0x7f9acc003980 2023-05-22T16:03:33.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:33.454+0000 7f9ae37fe640 1 --2- 172.21.15.16:0/4143294134 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9acc005180 0x7f9acc00b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:33.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:33.454+0000 7f9ae27fc640 1 --2- 172.21.15.16:0/4143294134 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9acc0047d0 0x7f9acc004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:33.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:33.454+0000 7f9ae2ffd640 1 --2- 172.21.15.16:0/4143294134 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9acc003980 0x7f9acc003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:33.464 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:33.454+0000 7f9ae37fe640 1 -- 172.21.15.16:0/4143294134 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9acc003980 msgr2=0x7f9acc003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:03:33.464 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:33.454+0000 7f9ae37fe640 1 --2- 172.21.15.16:0/4143294134 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9acc003980 0x7f9acc003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:33.464 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:33.454+0000 7f9ae37fe640 1 -- 172.21.15.16:0/4143294134 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9acc0047d0 msgr2=0x7f9acc004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:03:33.464 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:33.454+0000 7f9ae37fe640 1 --2- 172.21.15.16:0/4143294134 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9acc0047d0 0x7f9acc004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:33.464 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:33.454+0000 7f9ae37fe640 1 -- 172.21.15.16:0/4143294134 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9acc003530 con 0x7f9acc005180 2023-05-22T16:03:33.466 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:33.458+0000 7f9ae37fe640 1 -- 172.21.15.16:0/4143294134 reap_dead start 2023-05-22T16:03:33.467 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:33.458+0000 7f9ae37fe640 1 --2- 172.21.15.16:0/4143294134 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9acc005180 0x7f9acc00b990 secure :-1 s=READY pgs=2210 cs=0 l=1 rev1=1 crypto rx=0x7f9adc001e70 tx=0x7f9adc01c760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27c63df07100a51e server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:03:33.467 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:33.458+0000 7f9ae1ffb640 1 -- 172.21.15.16:0/4143294134 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9adc003b80 con 0x7f9acc005180 2023-05-22T16:03:33.467 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:33.458+0000 7f9ae1ffb640 1 -- 172.21.15.16:0/4143294134 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9adc003d00 con 0x7f9acc005180 2023-05-22T16:03:33.467 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:33.458+0000 7f9ae1ffb640 1 -- 172.21.15.16:0/4143294134 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9adc005310 con 0x7f9acc005180 2023-05-22T16:03:33.467 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:33.458+0000 7f9ae97b9640 1 -- 172.21.15.16:0/4143294134 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9acc005180 msgr2=0x7f9acc00b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:33.467 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:33.458+0000 7f9ae97b9640 1 --2- 172.21.15.16:0/4143294134 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9acc005180 0x7f9acc00b990 secure :-1 s=READY pgs=2210 cs=0 l=1 rev1=1 crypto rx=0x7f9adc001e70 tx=0x7f9adc01c760 comp rx=0 tx=0).stop 2023-05-22T16:03:33.467 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:33.458+0000 7f9ae97b9640 1 -- 172.21.15.16:0/4143294134 shutdown_connections 2023-05-22T16:03:33.468 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:33.458+0000 7f9ae97b9640 1 --2- 172.21.15.16:0/4143294134 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9acc005180 0x7f9acc00b990 unknown :-1 s=CLOSED pgs=2210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:33.468 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:33.458+0000 7f9ae97b9640 1 -- 172.21.15.16:0/4143294134 >> 172.21.15.16:0/4143294134 conn(0x7f9ae406f5a0 msgr2=0x7f9ae406f9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:03:33.468 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:33.458+0000 7f9ae97b9640 1 -- 172.21.15.16:0/4143294134 shutdown_connections 2023-05-22T16:03:33.468 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:33.458+0000 7f9ae97b9640 1 -- 172.21.15.16:0/4143294134 wait complete. 2023-05-22T16:03:33.468 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:33.458+0000 7f9ae97b9640 1 Processor -- start 2023-05-22T16:03:33.468 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:33.458+0000 7f9ae97b9640 1 -- start start 2023-05-22T16:03:33.468 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:33.462+0000 7f9ae97b9640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9acc005180 0x7f9ae40c8f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:33.469 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:33.462+0000 7f9ae97b9640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9ae40b1d60 0x7f9ae40c8190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:33.469 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:33.462+0000 7f9ae97b9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9ae415b4e0 0x7f9ae41551b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:33.469 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:33.462+0000 7f9ae97b9640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ae40c7d40 con 0x7f9ae415b4e0 2023-05-22T16:03:33.469 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:33.462+0000 7f9ae97b9640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ae40c9510 con 0x7f9ae40b1d60 2023-05-22T16:03:33.469 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:33.462+0000 7f9ae97b9640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ae406bcc0 con 0x7f9acc005180 2023-05-22T16:03:33.469 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:33.462+0000 7f9ae37fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9ae415b4e0 0x7f9ae41551b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:33.469 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:33.462+0000 7f9ae37fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9ae415b4e0 0x7f9ae41551b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48026/0 (socket says 172.21.15.16:48026) 2023-05-22T16:03:33.470 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:33.462+0000 7f9ae37fe640 1 -- 172.21.15.16:0/4252395631 learned_addr learned my addr 172.21.15.16:0/4252395631 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:03:33.470 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:33.462+0000 7f9ae27fc640 1 --2- 172.21.15.16:0/4252395631 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9ae40b1d60 0x7f9ae40c8190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:33.470 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:33.462+0000 7f9ae2ffd640 1 --2- 172.21.15.16:0/4252395631 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9acc005180 0x7f9ae40c8f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:33.470 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:33.462+0000 7f9ae37fe640 1 -- 172.21.15.16:0/4252395631 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9acc005180 msgr2=0x7f9ae40c8f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:33.470 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:33.462+0000 7f9ae37fe640 1 --2- 172.21.15.16:0/4252395631 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9acc005180 0x7f9ae40c8f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:33.470 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:33.462+0000 7f9ae37fe640 1 -- 172.21.15.16:0/4252395631 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9ae40b1d60 msgr2=0x7f9ae40c8190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:33.470 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:33.462+0000 7f9ae37fe640 1 --2- 172.21.15.16:0/4252395631 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9ae40b1d60 0x7f9ae40c8190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:33.470 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:33.462+0000 7f9ae37fe640 1 -- 172.21.15.16:0/4252395631 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ae40d04b0 con 0x7f9ae415b4e0 2023-05-22T16:03:33.471 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:33.462+0000 7f9ae27fc640 1 --2- 172.21.15.16:0/4252395631 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9ae40b1d60 0x7f9ae40c8190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T16:03:33.471 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:33.462+0000 7f9ae37fe640 1 --2- 172.21.15.16:0/4252395631 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9ae415b4e0 0x7f9ae41551b0 secure :-1 s=READY pgs=2211 cs=0 l=1 rev1=1 crypto rx=0x7f9adc01d000 tx=0x7f9acc005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:03:33.471 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:33.462+0000 7f9acb7fe640 1 -- 172.21.15.16:0/4252395631 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9adc006d60 con 0x7f9ae415b4e0 2023-05-22T16:03:33.471 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:33.462+0000 7f9acb7fe640 1 -- 172.21.15.16:0/4252395631 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9adc003b90 con 0x7f9ae415b4e0 2023-05-22T16:03:33.471 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:33.462+0000 7f9acb7fe640 1 -- 172.21.15.16:0/4252395631 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9adc007250 con 0x7f9ae415b4e0 2023-05-22T16:03:33.471 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:33.462+0000 7f9ae97b9640 1 -- 172.21.15.16:0/4252395631 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ae40d01d0 con 0x7f9ae415b4e0 2023-05-22T16:03:33.471 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:33.462+0000 7f9ae97b9640 1 -- 172.21.15.16:0/4252395631 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ae40d0790 con 0x7f9ae415b4e0 2023-05-22T16:03:33.472 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:33.462+0000 7f9ae97b9640 1 -- 172.21.15.16:0/4252395631 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ae40c7eb0 con 0x7f9ae415b4e0 2023-05-22T16:03:33.472 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:33.462+0000 7f9acb7fe640 1 -- 172.21.15.16:0/4252395631 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 82) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f9adc0073b0 con 0x7f9ae415b4e0 2023-05-22T16:03:33.472 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:33.462+0000 7f9acb7fe640 1 --2- 172.21.15.16:0/4252395631 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f9ad00654a0 0x7f9ad0067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:33.473 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:33.462+0000 7f9acb7fe640 1 -- 172.21.15.16:0/4252395631 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(170..170 src has 1..170) v4 ==== 6618+0+0 (secure 0 0 0) 0x7f9adc11c360 con 0x7f9ae415b4e0 2023-05-22T16:03:33.473 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:33.462+0000 7f9ae2ffd640 1 --2- 172.21.15.16:0/4252395631 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f9ad00654a0 0x7f9ad0067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:33.473 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:33.462+0000 7f9ae2ffd640 1 --2- 172.21.15.16:0/4252395631 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f9ad00654a0 0x7f9ad0067950 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f9ae415bfb0 tx=0x7f9ad4008040 comp rx=0 tx=0).ready entity=mgr.10032 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:03:33.473 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:33.466+0000 7f9acb7fe640 1 -- 172.21.15.16:0/4252395631 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f9ae40c7eb0 con 0x7f9ae415b4e0 2023-05-22T16:03:33.626 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:33.622+0000 7f9ae97b9640 1 -- 172.21.15.16:0/4252395631 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f9ae404dbe0 con 0x7f9ae415b4e0 2023-05-22T16:03:33.638 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:33.634+0000 7f9acb7fe640 1 -- 172.21.15.16:0/4252395631 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v58)=0 v58) v1 ==== 122+0+0 (secure 0 0 0) 0x7f9adc0e9b00 con 0x7f9ae415b4e0 2023-05-22T16:03:33.640 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:33.634+0000 7f9ae97b9640 1 -- 172.21.15.16:0/4252395631 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f9ad00654a0 msgr2=0x7f9ad0067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:33.640 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:33.634+0000 7f9ae97b9640 1 --2- 172.21.15.16:0/4252395631 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f9ad00654a0 0x7f9ad0067950 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f9ae415bfb0 tx=0x7f9ad4008040 comp rx=0 tx=0).stop 2023-05-22T16:03:33.640 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:33.634+0000 7f9ae97b9640 1 -- 172.21.15.16:0/4252395631 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9ae415b4e0 msgr2=0x7f9ae41551b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:33.640 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:33.634+0000 7f9ae97b9640 1 --2- 172.21.15.16:0/4252395631 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9ae415b4e0 0x7f9ae41551b0 secure :-1 s=READY pgs=2211 cs=0 l=1 rev1=1 crypto rx=0x7f9adc01d000 tx=0x7f9acc005800 comp rx=0 tx=0).stop 2023-05-22T16:03:33.640 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:33.634+0000 7f9ae97b9640 1 -- 172.21.15.16:0/4252395631 shutdown_connections 2023-05-22T16:03:33.641 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:33.634+0000 7f9ae97b9640 1 --2- 172.21.15.16:0/4252395631 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f9ad00654a0 0x7f9ad0067950 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:33.641 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:33.634+0000 7f9ae97b9640 1 --2- 172.21.15.16:0/4252395631 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9ae415b4e0 0x7f9ae41551b0 unknown :-1 s=CLOSED pgs=2211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:33.641 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:33.634+0000 7f9ae97b9640 1 --2- 172.21.15.16:0/4252395631 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9ae40b1d60 0x7f9ae40c8190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:33.641 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:33.634+0000 7f9ae97b9640 1 --2- 172.21.15.16:0/4252395631 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9acc005180 0x7f9ae40c8f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:33.641 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:33.634+0000 7f9ae97b9640 1 -- 172.21.15.16:0/4252395631 >> 172.21.15.16:0/4252395631 conn(0x7f9ae406f5a0 msgr2=0x7f9ae41e4170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:03:33.642 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:33.634+0000 7f9ae97b9640 1 -- 172.21.15.16:0/4252395631 shutdown_connections 2023-05-22T16:03:33.642 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:33.634+0000 7f9ae97b9640 1 -- 172.21.15.16:0/4252395631 wait complete. 2023-05-22T16:03:33.656 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-05-22T16:03:33.657 DEBUG:teuthology.orchestra.run.smithi016:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-05-22T16:03:33.693 DEBUG:teuthology.orchestra.run.smithi062:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-05-22T16:03:33.741 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-05-22T16:03:33.741 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-05-22T16:03:33.742 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2023-05-22T16:03:33.747 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16345192k,nr_inodes=4086298,mode=755,inode64 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3276544k,mode=755,inode64 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev,inode64 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=15732 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3276540k,nr_inodes=819135,mode=700,uid=1000,gid=1271,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3276544k,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-05-22T16:03:33.747 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2023-05-22T16:03:33.748 DEBUG:teuthology.orchestra.run.smithi062:> dd if=/proc/self/mounts of=/dev/stdout 2023-05-22T16:03:33.753 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16342116k,nr_inodes=4085529,mode=755,inode64 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275932k,mode=755,inode64 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev,inode64 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=13250 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275928k,nr_inodes=818982,mode=700,uid=1000,gid=1271,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275932k,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-05-22T16:03:33.753 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-05-22T16:03:33.754 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2023-05-22T16:03:33.792 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16345192k,nr_inodes=4086298,mode=755,inode64 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3276544k,mode=755,inode64 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev,inode64 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=15732 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3276540k,nr_inodes=819135,mode=700,uid=1000,gid=1271,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3276544k,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-05-22T16:03:33.792 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2023-05-22T16:03:33.793 DEBUG:teuthology.orchestra.run.smithi062:> dd if=/proc/self/mounts of=/dev/stdout 2023-05-22T16:03:33.802 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16342116k,nr_inodes=4085529,mode=755,inode64 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275932k,mode=755,inode64 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev,inode64 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=13250 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275928k,nr_inodes=818982,mode=700,uid=1000,gid=1271,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275932k,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-05-22T16:03:33.802 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2023-05-22T16:03:33.803 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-05-22T16:03:33.956 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:33.950+0000 7f077ea00640 1 Processor -- start 2023-05-22T16:03:33.956 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:33.950+0000 7f077ea00640 1 -- start start 2023-05-22T16:03:33.957 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:33.954+0000 7f077ea00640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f077815fc50 0x7f0778160020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:33.957 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:33.954+0000 7f077ea00640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f077815c920 0x7f07780d0030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:33.957 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:33.954+0000 7f077ea00640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f077815c490 con 0x7f077815cd70 2023-05-22T16:03:33.957 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:33.954+0000 7f077ea00640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07780cf7a0 con 0x7f077815c920 2023-05-22T16:03:33.958 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:33.954+0000 7f077ea00640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f077815c140 con 0x7f077815fc50 2023-05-22T16:03:33.958 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:33.954+0000 7f077cf9d640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f077815cd70 0x7f07780b3ee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:48134/0 (socket says 172.21.15.16:48134) 2023-05-22T16:03:33.959 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:33.954+0000 7f07777fe640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f077815c920 0x7f07780d0030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:33.959 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:33.954+0000 7f077cf9d640 1 -- 172.21.15.16:0/1170711830 learned_addr learned my addr 172.21.15.16:0/1170711830 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T16:03:33.959 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:33.954+0000 7f0777fff640 1 --2- 172.21.15.16:0/1170711830 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f077815fc50 0x7f0778160020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:33.959 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:33.954+0000 7f07777fe640 1 -- 172.21.15.16:0/1170711830 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f077815fc50 msgr2=0x7f0778160020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:03:33.959 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:33.954+0000 7f07777fe640 1 --2- 172.21.15.16:0/1170711830 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f077815fc50 0x7f0778160020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:33.959 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:33.954+0000 7f07777fe640 1 -- 172.21.15.16:0/1170711830 >> v1:172.21.15.16:6789/0 conn(0x7f077815cd70 legacy=0x7f07780b3ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:33.960 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:33.954+0000 7f07777fe640 1 -- 172.21.15.16:0/1170711830 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07780ccb50 con 0x7f077815c920 2023-05-22T16:03:33.960 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:33.954+0000 7f0776ffd640 1 -- 172.21.15.16:0/1170711830 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3603830868 0 0) 0x7f077815c490 con 0x7f077815cd70 2023-05-22T16:03:33.960 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:33.958+0000 7f07777fe640 1 --2- 172.21.15.16:0/1170711830 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f077815c920 0x7f07780d0030 secure :-1 s=READY pgs=2213 cs=0 l=1 rev1=1 crypto rx=0x7f07680097c0 tx=0x7f076801c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c11a4949451ce0aa server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:03:33.962 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:33.958+0000 7f0776ffd640 1 -- 172.21.15.16:0/1170711830 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07680ab650 con 0x7f077815c920 2023-05-22T16:03:33.962 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:33.958+0000 7f0776ffd640 1 -- 172.21.15.16:0/1170711830 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07680ab7b0 con 0x7f077815c920 2023-05-22T16:03:33.963 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:33.958+0000 7f0776ffd640 1 -- 172.21.15.16:0/1170711830 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07680abb40 con 0x7f077815c920 2023-05-22T16:03:33.963 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:33.958+0000 7f077ea00640 1 -- 172.21.15.16:0/1170711830 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f077815c920 msgr2=0x7f07780d0030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:33.963 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:33.958+0000 7f077ea00640 1 --2- 172.21.15.16:0/1170711830 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f077815c920 0x7f07780d0030 secure :-1 s=READY pgs=2213 cs=0 l=1 rev1=1 crypto rx=0x7f07680097c0 tx=0x7f076801c450 comp rx=0 tx=0).stop 2023-05-22T16:03:33.963 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:33.958+0000 7f077ea00640 1 -- 172.21.15.16:0/1170711830 shutdown_connections 2023-05-22T16:03:33.964 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:33.958+0000 7f077ea00640 1 --2- 172.21.15.16:0/1170711830 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f077815c920 0x7f07780d0030 unknown :-1 s=CLOSED pgs=2213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:33.964 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:33.958+0000 7f077ea00640 1 --2- 172.21.15.16:0/1170711830 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f077815fc50 0x7f0778160020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:33.964 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:33.958+0000 7f077ea00640 1 -- 172.21.15.16:0/1170711830 >> 172.21.15.16:0/1170711830 conn(0x7f07780657f0 msgr2=0x7f0778070e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:03:33.964 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:33.958+0000 7f077ea00640 1 -- 172.21.15.16:0/1170711830 shutdown_connections 2023-05-22T16:03:33.964 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:33.958+0000 7f077ea00640 1 -- 172.21.15.16:0/1170711830 wait complete. 2023-05-22T16:03:33.967 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:33.958+0000 7f077ea00640 1 Processor -- start 2023-05-22T16:03:33.967 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:33.958+0000 7f077ea00640 1 -- start start 2023-05-22T16:03:33.967 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:33.962+0000 7f077ea00640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f077815c920 0x7f07780cacc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:33.968 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:33.962+0000 7f077ea00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f077815cd70 0x7f07780c8450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:33.968 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:33.962+0000 7f077ea00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f077815fc50 0x7f07780c8990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:33.968 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:33.962+0000 7f077ea00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07780b3b60 con 0x7f077815cd70 2023-05-22T16:03:33.968 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:33.962+0000 7f077ea00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0778153350 con 0x7f077815fc50 2023-05-22T16:03:33.969 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:33.962+0000 7f077ea00640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07780c7780 con 0x7f077815c920 2023-05-22T16:03:33.969 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:33.962+0000 7f07777fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f077815cd70 0x7f07780c8450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:33.969 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:33.962+0000 7f07777fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f077815cd70 0x7f07780c8450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48036/0 (socket says 172.21.15.16:48036) 2023-05-22T16:03:33.969 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:33.962+0000 7f07777fe640 1 -- 172.21.15.16:0/4280957813 learned_addr learned my addr 172.21.15.16:0/4280957813 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:03:33.969 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:33.962+0000 7f0777fff640 1 --2- 172.21.15.16:0/4280957813 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f077815c920 0x7f07780cacc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:33.970 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:33.962+0000 7f077cf9d640 1 --2- 172.21.15.16:0/4280957813 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f077815fc50 0x7f07780c8990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:33.970 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:33.962+0000 7f07777fe640 1 -- 172.21.15.16:0/4280957813 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f077815c920 msgr2=0x7f07780cacc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:33.970 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:33.962+0000 7f07777fe640 1 --2- 172.21.15.16:0/4280957813 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f077815c920 0x7f07780cacc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:33.970 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:33.962+0000 7f07777fe640 1 -- 172.21.15.16:0/4280957813 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f077815fc50 msgr2=0x7f07780c8990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:33.971 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:33.962+0000 7f07777fe640 1 --2- 172.21.15.16:0/4280957813 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f077815fc50 0x7f07780c8990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:33.971 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:33.962+0000 7f07777fe640 1 -- 172.21.15.16:0/4280957813 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0768009420 con 0x7f077815cd70 2023-05-22T16:03:33.972 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:33.962+0000 7f07777fe640 1 --2- 172.21.15.16:0/4280957813 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f077815cd70 0x7f07780c8450 secure :-1 s=READY pgs=2214 cs=0 l=1 rev1=1 crypto rx=0x7f07680097c0 tx=0x7f076801ac90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:03:33.972 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:33.962+0000 7f0774ff9640 1 -- 172.21.15.16:0/4280957813 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0768004220 con 0x7f077815cd70 2023-05-22T16:03:33.972 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:33.962+0000 7f0774ff9640 1 -- 172.21.15.16:0/4280957813 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07680b9070 con 0x7f077815cd70 2023-05-22T16:03:33.972 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:33.962+0000 7f077ea00640 1 -- 172.21.15.16:0/4280957813 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0778160560 con 0x7f077815cd70 2023-05-22T16:03:33.973 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:33.962+0000 7f0774ff9640 1 -- 172.21.15.16:0/4280957813 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0768003620 con 0x7f077815cd70 2023-05-22T16:03:33.973 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:33.962+0000 7f077ea00640 1 -- 172.21.15.16:0/4280957813 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07780d0980 con 0x7f077815cd70 2023-05-22T16:03:33.973 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:33.962+0000 7f077ea00640 1 -- 172.21.15.16:0/4280957813 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0778168ad0 con 0x7f077815cd70 2023-05-22T16:03:33.974 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:33.962+0000 7f0774ff9640 1 -- 172.21.15.16:0/4280957813 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 82) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f0768003990 con 0x7f077815cd70 2023-05-22T16:03:33.975 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:33.962+0000 7f0774ff9640 1 --2- 172.21.15.16:0/4280957813 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f07580654a0 0x7f0758067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:33.975 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:33.962+0000 7f0774ff9640 1 -- 172.21.15.16:0/4280957813 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(170..170 src has 1..170) v4 ==== 6618+0+0 (secure 0 0 0) 0x7f07680bb070 con 0x7f077815cd70 2023-05-22T16:03:33.975 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:33.962+0000 7f0777fff640 1 --2- 172.21.15.16:0/4280957813 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f07580654a0 0x7f0758067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:33.975 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:33.966+0000 7f0777fff640 1 --2- 172.21.15.16:0/4280957813 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f07580654a0 0x7f0758067950 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f076000b930 tx=0x7f0760008040 comp rx=0 tx=0).ready entity=mgr.10032 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:03:33.976 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:33.966+0000 7f0774ff9640 1 -- 172.21.15.16:0/4280957813 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f07680eb4a0 con 0x7f077815cd70 2023-05-22T16:03:34.162 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:34.158+0000 7f077ea00640 1 -- 172.21.15.16:0/4280957813 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f077806b260 con 0x7f077815cd70 2023-05-22T16:03:34.163 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:34.158+0000 7f0774ff9640 1 -- 172.21.15.16:0/4280957813 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 68 v68) v1 ==== 94+0+3524 (secure 0 0 0) 0x7f07680c4030 con 0x7f077815cd70 2023-05-22T16:03:34.163 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T16:03:34.164 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":68,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8170,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6833/78835131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":78835131},{"type":"v1","addr":"172.21.15.62:6833","nonce":78835131}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":61},{"gid":8173,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/1188098562","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":1188098562},{"type":"v1","addr":"172.21.15.62:6835","nonce":1188098562}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":61},{"gid":9645,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/4204206994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":4204206994},{"type":"v1","addr":"172.21.15.16:6835","nonce":4204206994}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":61},{"gid":10176,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/1110759737","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":1110759737},{"type":"v1","addr":"172.21.15.16:6837","nonce":1110759737}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":68}],"filesystems":[]} 2023-05-22T16:03:34.164 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 68 2023-05-22T16:03:34.165 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:34.162+0000 7f077ea00640 1 -- 172.21.15.16:0/4280957813 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f07580654a0 msgr2=0x7f0758067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:34.165 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:34.162+0000 7f077ea00640 1 --2- 172.21.15.16:0/4280957813 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f07580654a0 0x7f0758067950 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f076000b930 tx=0x7f0760008040 comp rx=0 tx=0).stop 2023-05-22T16:03:34.165 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:34.162+0000 7f077ea00640 1 -- 172.21.15.16:0/4280957813 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f077815cd70 msgr2=0x7f07780c8450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:34.166 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:34.162+0000 7f077ea00640 1 --2- 172.21.15.16:0/4280957813 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f077815cd70 0x7f07780c8450 secure :-1 s=READY pgs=2214 cs=0 l=1 rev1=1 crypto rx=0x7f07680097c0 tx=0x7f076801ac90 comp rx=0 tx=0).stop 2023-05-22T16:03:34.166 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:34.162+0000 7f077ea00640 1 -- 172.21.15.16:0/4280957813 shutdown_connections 2023-05-22T16:03:34.166 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:34.162+0000 7f077ea00640 1 --2- 172.21.15.16:0/4280957813 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f077815fc50 0x7f07780c8990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:34.166 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:34.162+0000 7f077ea00640 1 --2- 172.21.15.16:0/4280957813 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f07580654a0 0x7f0758067950 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:34.166 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:34.162+0000 7f077ea00640 1 --2- 172.21.15.16:0/4280957813 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f077815cd70 0x7f07780c8450 unknown :-1 s=CLOSED pgs=2214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:34.166 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:34.162+0000 7f077ea00640 1 --2- 172.21.15.16:0/4280957813 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f077815c920 0x7f07780cacc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:34.166 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:34.162+0000 7f077ea00640 1 -- 172.21.15.16:0/4280957813 >> 172.21.15.16:0/4280957813 conn(0x7f07780657f0 msgr2=0x7f07780ceea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:03:34.167 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:34.162+0000 7f077ea00640 1 -- 172.21.15.16:0/4280957813 shutdown_connections 2023-05-22T16:03:34.167 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:34.162+0000 7f077ea00640 1 -- 172.21.15.16:0/4280957813 wait complete. 2023-05-22T16:03:34.183 INFO:tasks.ceph.mds.a:Restarting daemon 2023-05-22T16:03:34.183 INFO:tasks.ceph.mds.a:Stopping old one... 2023-05-22T16:03:34.183 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-05-22T16:03:34.183 INFO:teuthology.orchestra.run:waiting for 300 2023-05-22T16:03:34.184 INFO:tasks.ceph.mds.c:Restarting daemon 2023-05-22T16:03:34.184 INFO:tasks.ceph.mds.c:Stopping old one... 2023-05-22T16:03:34.184 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-05-22T16:03:34.184 INFO:teuthology.orchestra.run:waiting for 300 2023-05-22T16:03:34.185 INFO:tasks.ceph.mds.b:Restarting daemon 2023-05-22T16:03:34.185 INFO:tasks.ceph.mds.b:Stopping old one... 2023-05-22T16:03:34.185 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-05-22T16:03:34.185 INFO:teuthology.orchestra.run:waiting for 300 2023-05-22T16:03:34.185 INFO:tasks.ceph.mds.d:Restarting daemon 2023-05-22T16:03:34.186 INFO:tasks.ceph.mds.d:Stopping old one... 2023-05-22T16:03:34.186 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-05-22T16:03:34.186 INFO:teuthology.orchestra.run:waiting for 300 2023-05-22T16:03:34.196 INFO:tasks.ceph.mds.c:Stopped 2023-05-22T16:03:34.196 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-05-22T16:03:34.197 INFO:tasks.ceph.mds.a:Stopped 2023-05-22T16:03:34.197 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-05-22T16:03:34.203 INFO:tasks.ceph.mds.d:Stopped 2023-05-22T16:03:34.203 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-05-22T16:03:34.207 INFO:tasks.ceph.mds.b:Stopped 2023-05-22T16:03:34.207 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-05-22T16:03:34.241 INFO:tasks.ceph.mds.c:Started 2023-05-22T16:03:34.242 DEBUG:teuthology.parallel:result is None 2023-05-22T16:03:34.242 INFO:tasks.ceph.mds.a:Started 2023-05-22T16:03:34.243 DEBUG:teuthology.parallel:result is None 2023-05-22T16:03:34.251 INFO:tasks.ceph.mds.d:Started 2023-05-22T16:03:34.251 DEBUG:teuthology.parallel:result is None 2023-05-22T16:03:34.252 INFO:tasks.ceph.mds.b:Started 2023-05-22T16:03:34.253 DEBUG:teuthology.parallel:result is None 2023-05-22T16:03:34.253 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2023-05-22T16:03:34.303 INFO:tasks.ceph.mds.c.smithi016.stdout:starting mds.c at 2023-05-22T16:03:34.303 INFO:tasks.ceph.mds.a.smithi016.stdout:starting mds.a at 2023-05-22T16:03:34.332 INFO:tasks.ceph.mds.d.smithi062.stdout:starting mds.d at 2023-05-22T16:03:34.334 INFO:tasks.ceph.mds.b.smithi062.stdout:starting mds.b at 2023-05-22T16:03:34.371 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:34.366+0000 7fbc852d0640 1 Processor -- start 2023-05-22T16:03:34.371 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:34.366+0000 7fbc852d0640 1 -- start start 2023-05-22T16:03:34.373 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:34.366+0000 7fbc852d0640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fbc80159cd0 0x7fbc8015a0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:34.373 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:34.366+0000 7fbc852d0640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbc800b2880 0x7fbc800cbfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:34.374 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:34.366+0000 7fbc852d0640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc800cf9a0 con 0x7fbc8015a5e0 2023-05-22T16:03:34.374 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:34.366+0000 7fbc852d0640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc8015aba0 con 0x7fbc80159cd0 2023-05-22T16:03:34.374 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:34.366+0000 7fbc852d0640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc8015ae80 con 0x7fbc800b2880 2023-05-22T16:03:34.374 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:34.366+0000 7fbc7f7fe640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fbc8015a5e0 0x7fbc800ce160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:48158/0 (socket says 172.21.15.16:48158) 2023-05-22T16:03:34.374 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:34.366+0000 7fbc7f7fe640 1 -- 172.21.15.16:0/601029619 learned_addr learned my addr 172.21.15.16:0/601029619 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T16:03:34.374 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:34.366+0000 7fbc7effd640 1 --2- 172.21.15.16:0/601029619 >> v2:172.21.15.62:3300/0 conn(0x7fbc80159cd0 0x7fbc8015a0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:34.374 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:34.366+0000 7fbc7e7fc640 1 --2- 172.21.15.16:0/601029619 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbc800b2880 0x7fbc800cbfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:34.375 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:34.366+0000 7fbc7dffb640 1 -- 172.21.15.16:0/601029619 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1335782846 0 0) 0x7fbc800cf9a0 con 0x7fbc8015a5e0 2023-05-22T16:03:34.375 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:34.366+0000 7fbc7dffb640 1 -- 172.21.15.16:0/601029619 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc68003530 con 0x7fbc8015a5e0 2023-05-22T16:03:34.375 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:34.370+0000 7fbc7dffb640 1 -- 172.21.15.16:0/601029619 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3547748722 0 0) 0x7fbc68003530 con 0x7fbc8015a5e0 2023-05-22T16:03:34.375 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:34.370+0000 7fbc7dffb640 1 -- 172.21.15.16:0/601029619 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbc800b2880 msgr2=0x7fbc800cbfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:03:34.375 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:34.370+0000 7fbc7dffb640 1 --2- 172.21.15.16:0/601029619 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbc800b2880 0x7fbc800cbfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:34.375 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:34.370+0000 7fbc7dffb640 1 -- 172.21.15.16:0/601029619 >> v2:172.21.15.62:3300/0 conn(0x7fbc80159cd0 msgr2=0x7fbc8015a0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:03:34.376 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:34.370+0000 7fbc7dffb640 1 --2- 172.21.15.16:0/601029619 >> v2:172.21.15.62:3300/0 conn(0x7fbc80159cd0 0x7fbc8015a0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:34.376 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:34.370+0000 7fbc7dffb640 1 -- 172.21.15.16:0/601029619 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc8015b160 con 0x7fbc8015a5e0 2023-05-22T16:03:34.376 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:34.370+0000 7fbc7dffb640 1 -- 172.21.15.16:0/601029619 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7fbc78001680 con 0x7fbc8015a5e0 2023-05-22T16:03:34.376 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:34.370+0000 7fbc7dffb640 1 -- 172.21.15.16:0/601029619 >> v1:172.21.15.16:6789/0 conn(0x7fbc8015a5e0 legacy=0x7fbc800ce160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:34.377 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:34.370+0000 7fbc7dffb640 1 --2- 172.21.15.16:0/601029619 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbc68003980 0x7fbc68003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:34.377 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:34.370+0000 7fbc7dffb640 1 --2- 172.21.15.16:0/601029619 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc680043b0 0x7fbc68004870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:34.377 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:34.370+0000 7fbc7dffb640 1 --2- 172.21.15.16:0/601029619 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbc800b2880 0x7fbc68004f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:34.378 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:34.370+0000 7fbc7dffb640 1 -- 172.21.15.16:0/601029619 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc8015aba0 con 0x7fbc680043b0 2023-05-22T16:03:34.378 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:34.370+0000 7fbc7dffb640 1 -- 172.21.15.16:0/601029619 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc8015ae80 con 0x7fbc68003980 2023-05-22T16:03:34.378 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:34.370+0000 7fbc7dffb640 1 -- 172.21.15.16:0/601029619 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc6800b990 con 0x7fbc800b2880 2023-05-22T16:03:34.378 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:34.370+0000 7fbc7e7fc640 1 --2- 172.21.15.16:0/601029619 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc680043b0 0x7fbc68004870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:34.379 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:34.370+0000 7fbc7effd640 1 --2- 172.21.15.16:0/601029619 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbc68003980 0x7fbc68003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:34.379 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:34.370+0000 7fbc7f7fe640 1 --2- 172.21.15.16:0/601029619 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbc800b2880 0x7fbc68004f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:34.379 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:34.370+0000 7fbc7e7fc640 1 -- 172.21.15.16:0/601029619 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbc800b2880 msgr2=0x7fbc68004f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:03:34.379 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:34.370+0000 7fbc7e7fc640 1 --2- 172.21.15.16:0/601029619 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbc800b2880 0x7fbc68004f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:34.380 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:34.370+0000 7fbc7e7fc640 1 -- 172.21.15.16:0/601029619 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbc68003980 msgr2=0x7fbc68003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:03:34.380 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:34.370+0000 7fbc7e7fc640 1 --2- 172.21.15.16:0/601029619 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbc68003980 0x7fbc68003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:34.380 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:34.370+0000 7fbc7e7fc640 1 -- 172.21.15.16:0/601029619 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc68003530 con 0x7fbc680043b0 2023-05-22T16:03:34.380 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:34.370+0000 7fbc7f7fe640 1 --2- 172.21.15.16:0/601029619 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbc800b2880 0x7fbc68004f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T16:03:34.385 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:34.370+0000 7fbc7e7fc640 1 --2- 172.21.15.16:0/601029619 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc680043b0 0x7fbc68004870 secure :-1 s=READY pgs=2223 cs=0 l=1 rev1=1 crypto rx=0x7fbc74009ae0 tx=0x7fbc7401ca50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f35244116eb15a2 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:03:34.385 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:34.370+0000 7fbc7dffb640 1 -- 172.21.15.16:0/601029619 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc740a5640 con 0x7fbc680043b0 2023-05-22T16:03:34.386 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:34.370+0000 7fbc7dffb640 1 -- 172.21.15.16:0/601029619 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc740a57a0 con 0x7fbc680043b0 2023-05-22T16:03:34.386 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:34.370+0000 7fbc7dffb640 1 -- 172.21.15.16:0/601029619 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc740a5b50 con 0x7fbc680043b0 2023-05-22T16:03:34.386 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:34.370+0000 7fbc852d0640 1 -- 172.21.15.16:0/601029619 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc680043b0 msgr2=0x7fbc68004870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:34.387 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:34.370+0000 7fbc852d0640 1 --2- 172.21.15.16:0/601029619 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc680043b0 0x7fbc68004870 secure :-1 s=READY pgs=2223 cs=0 l=1 rev1=1 crypto rx=0x7fbc74009ae0 tx=0x7fbc7401ca50 comp rx=0 tx=0).stop 2023-05-22T16:03:34.387 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:34.370+0000 7fbc7f7fe640 1 -- 172.21.15.16:0/601029619 reap_dead start 2023-05-22T16:03:34.387 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:34.370+0000 7fbc852d0640 1 -- 172.21.15.16:0/601029619 shutdown_connections 2023-05-22T16:03:34.387 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:34.370+0000 7fbc852d0640 1 -- 172.21.15.16:0/601029619 >> 172.21.15.16:0/601029619 conn(0x7fbc8006f5c0 msgr2=0x7fbc8006f9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:03:34.387 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:34.370+0000 7fbc852d0640 1 -- 172.21.15.16:0/601029619 shutdown_connections 2023-05-22T16:03:34.388 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:34.370+0000 7fbc852d0640 1 -- 172.21.15.16:0/601029619 wait complete. 2023-05-22T16:03:34.388 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:34.374+0000 7fbc852d0640 1 Processor -- start 2023-05-22T16:03:34.388 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:34.374+0000 7fbc852d0640 1 -- start start 2023-05-22T16:03:34.388 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:34.374+0000 7fbc852d0640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbc680043b0 0x7fbc80152da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:34.388 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:34.374+0000 7fbc852d0640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc800c4580 0x7fbc800c9760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:34.388 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:34.374+0000 7fbc852d0640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbc8014f210 0x7fbc8014f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:34.388 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:34.374+0000 7fbc852d0640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc800c4cf0 con 0x7fbc800c4580 2023-05-22T16:03:34.389 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:34.374+0000 7fbc852d0640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc801eefb0 con 0x7fbc680043b0 2023-05-22T16:03:34.389 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:34.374+0000 7fbc852d0640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc800c4a10 con 0x7fbc8014f210 2023-05-22T16:03:34.389 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:34.374+0000 7fbc7e7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc800c4580 0x7fbc800c9760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:34.389 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:34.374+0000 7fbc7e7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc800c4580 0x7fbc800c9760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48064/0 (socket says 172.21.15.16:48064) 2023-05-22T16:03:34.389 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:34.374+0000 7fbc7e7fc640 1 -- 172.21.15.16:0/194097896 learned_addr learned my addr 172.21.15.16:0/194097896 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:03:34.389 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:34.374+0000 7fbc7effd640 1 --2- 172.21.15.16:0/194097896 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbc680043b0 0x7fbc80152da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:34.389 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:34.374+0000 7fbc7f7fe640 1 --2- 172.21.15.16:0/194097896 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbc8014f210 0x7fbc8014f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:34.389 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:34.374+0000 7fbc7e7fc640 1 -- 172.21.15.16:0/194097896 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbc8014f210 msgr2=0x7fbc8014f6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:34.390 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:34.374+0000 7fbc7e7fc640 1 --2- 172.21.15.16:0/194097896 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbc8014f210 0x7fbc8014f6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:34.390 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:34.374+0000 7fbc7e7fc640 1 -- 172.21.15.16:0/194097896 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbc680043b0 msgr2=0x7fbc80152da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:34.390 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:34.374+0000 7fbc7e7fc640 1 --2- 172.21.15.16:0/194097896 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbc680043b0 0x7fbc80152da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:34.390 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:34.374+0000 7fbc7e7fc640 1 -- 172.21.15.16:0/194097896 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc800d0e40 con 0x7fbc800c4580 2023-05-22T16:03:34.390 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:34.374+0000 7fbc7e7fc640 1 --2- 172.21.15.16:0/194097896 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc800c4580 0x7fbc800c9760 secure :-1 s=READY pgs=2224 cs=0 l=1 rev1=1 crypto rx=0x7fbc74009ae0 tx=0x7fbc68005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:03:34.390 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:34.374+0000 7fbc677fe640 1 -- 172.21.15.16:0/194097896 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc74004230 con 0x7fbc800c4580 2023-05-22T16:03:34.390 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:34.374+0000 7fbc677fe640 1 -- 172.21.15.16:0/194097896 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc74004390 con 0x7fbc800c4580 2023-05-22T16:03:34.391 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:34.374+0000 7fbc852d0640 1 -- 172.21.15.16:0/194097896 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc800cf630 con 0x7fbc800c4580 2023-05-22T16:03:34.391 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:34.374+0000 7fbc677fe640 1 -- 172.21.15.16:0/194097896 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc740ad850 con 0x7fbc800c4580 2023-05-22T16:03:34.391 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:34.374+0000 7fbc852d0640 1 -- 172.21.15.16:0/194097896 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc800d0fd0 con 0x7fbc800c4580 2023-05-22T16:03:34.391 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:34.374+0000 7fbc852d0640 1 -- 172.21.15.16:0/194097896 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc80156e40 con 0x7fbc800c4580 2023-05-22T16:03:34.391 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:34.378+0000 7fbc677fe640 1 -- 172.21.15.16:0/194097896 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 82) v1 ==== 84332+0+0 (secure 0 0 0) 0x7fbc740b4040 con 0x7fbc800c4580 2023-05-22T16:03:34.391 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:34.378+0000 7fbc677fe640 1 --2- 172.21.15.16:0/194097896 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fbc6c0654a0 0x7fbc6c067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:34.391 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:34.378+0000 7fbc677fe640 1 -- 172.21.15.16:0/194097896 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(174..174 src has 1..174) v4 ==== 6618+0+0 (secure 0 0 0) 0x7fbc7411d290 con 0x7fbc800c4580 2023-05-22T16:03:34.391 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:34.378+0000 7fbc7effd640 1 --2- 172.21.15.16:0/194097896 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fbc6c0654a0 0x7fbc6c067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:34.392 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:34.378+0000 7fbc7effd640 1 --2- 172.21.15.16:0/194097896 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fbc6c0654a0 0x7fbc6c067950 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fbc70014510 tx=0x7fbc7000b040 comp rx=0 tx=0).ready entity=mgr.10032 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:03:34.392 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:34.382+0000 7fbc677fe640 1 -- 172.21.15.16:0/194097896 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fbc740eac80 con 0x7fbc800c4580 2023-05-22T16:03:34.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:34.542+0000 7fbc852d0640 1 -- 172.21.15.16:0/194097896 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7fbc8004dbe0 con 0x7fbc800c4580 2023-05-22T16:03:35.387 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:35.386+0000 7fbc677fe640 1 -- 172.21.15.16:0/194097896 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v175) v1 ==== 97+0+0 (secure 0 0 0) 0x7fbc740ee2a0 con 0x7fbc800c4580 2023-05-22T16:03:35.387 INFO:teuthology.orchestra.run.smithi016.stderr: removed all blocklist entries 2023-05-22T16:03:35.390 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:35.386+0000 7fbc852d0640 1 -- 172.21.15.16:0/194097896 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fbc6c0654a0 msgr2=0x7fbc6c067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:35.390 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:35.386+0000 7fbc852d0640 1 --2- 172.21.15.16:0/194097896 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fbc6c0654a0 0x7fbc6c067950 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fbc70014510 tx=0x7fbc7000b040 comp rx=0 tx=0).stop 2023-05-22T16:03:35.390 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:35.386+0000 7fbc852d0640 1 -- 172.21.15.16:0/194097896 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc800c4580 msgr2=0x7fbc800c9760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:35.391 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:35.386+0000 7fbc852d0640 1 --2- 172.21.15.16:0/194097896 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc800c4580 0x7fbc800c9760 secure :-1 s=READY pgs=2224 cs=0 l=1 rev1=1 crypto rx=0x7fbc74009ae0 tx=0x7fbc68005800 comp rx=0 tx=0).stop 2023-05-22T16:03:35.391 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:35.386+0000 7fbc852d0640 1 -- 172.21.15.16:0/194097896 shutdown_connections 2023-05-22T16:03:35.391 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:35.386+0000 7fbc852d0640 1 --2- 172.21.15.16:0/194097896 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbc8014f210 0x7fbc8014f6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:35.391 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:35.386+0000 7fbc852d0640 1 --2- 172.21.15.16:0/194097896 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fbc6c0654a0 0x7fbc6c067950 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:35.391 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:35.386+0000 7fbc852d0640 1 --2- 172.21.15.16:0/194097896 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc800c4580 0x7fbc800c9760 unknown :-1 s=CLOSED pgs=2224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:35.391 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:35.386+0000 7fbc852d0640 1 --2- 172.21.15.16:0/194097896 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbc680043b0 0x7fbc80152da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:35.391 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:35.386+0000 7fbc852d0640 1 -- 172.21.15.16:0/194097896 >> 172.21.15.16:0/194097896 conn(0x7fbc8006f5c0 msgr2=0x7fbc8006e260 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:03:35.391 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:35.386+0000 7fbc852d0640 1 -- 172.21.15.16:0/194097896 shutdown_connections 2023-05-22T16:03:35.392 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:35.386+0000 7fbc852d0640 1 -- 172.21.15.16:0/194097896 wait complete. 2023-05-22T16:03:35.411 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2023-05-22T16:03:35.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:35.534+0000 7fc22d780640 1 Processor -- start 2023-05-22T16:03:35.534 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:35.534+0000 7fc22d780640 1 -- start start 2023-05-22T16:03:35.534 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:35.534+0000 7fc22d780640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fc2280c9b40 0x7fc22815e460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:35.535 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:35.534+0000 7fc22d780640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc228070df0 0x7fc228071210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:35.535 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:35.534+0000 7fc22d780640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2280b0650 con 0x7fc2280ca4e0 2023-05-22T16:03:35.535 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:35.534+0000 7fc22d780640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2280b0370 con 0x7fc2280c9b40 2023-05-22T16:03:35.535 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:35.534+0000 7fc22d780640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc22815d980 con 0x7fc228070df0 2023-05-22T16:03:35.536 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:35.534+0000 7fc2267fc640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fc2280c9b40 0x7fc22815e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:35.536 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:35.534+0000 7fc2277fe640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc228070df0 0x7fc228071210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:35.536 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:35.534+0000 7fc2277fe640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc228070df0 0x7fc228071210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3301/0 says I am v2:172.21.15.16:55248/0 (socket says 172.21.15.16:55248) 2023-05-22T16:03:35.536 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:35.534+0000 7fc226ffd640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7fc2280ca4e0 0x7fc2280afb40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.16:49008/0 (socket says 172.21.15.16:49008) 2023-05-22T16:03:35.536 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:35.534+0000 7fc2267fc640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fc2280c9b40 0x7fc22815e460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.16:45072/0 (socket says 172.21.15.16:45072) 2023-05-22T16:03:35.536 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:35.534+0000 7fc2277fe640 1 -- 172.21.15.16:0/1880392941 learned_addr learned my addr 172.21.15.16:0/1880392941 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:03:35.537 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:35.534+0000 7fc2267fc640 1 -- 172.21.15.16:0/1880392941 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc228070df0 msgr2=0x7fc228071210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:03:35.537 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:35.534+0000 7fc2267fc640 1 --2- 172.21.15.16:0/1880392941 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc228070df0 0x7fc228071210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:35.537 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:35.534+0000 7fc2267fc640 1 -- 172.21.15.16:0/1880392941 >> v1:172.21.15.62:6789/0 conn(0x7fc2280ca4e0 legacy=0x7fc2280afb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:35.538 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:35.534+0000 7fc2267fc640 1 -- 172.21.15.16:0/1880392941 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc22815dc60 con 0x7fc2280c9b40 2023-05-22T16:03:35.538 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:35.534+0000 7fc2277fe640 1 --2- 172.21.15.16:0/1880392941 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc228070df0 0x7fc228071210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-05-22T16:03:35.538 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:35.534+0000 7fc2267fc640 1 --2- 172.21.15.16:0/1880392941 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc2280c9b40 0x7fc22815e460 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7fc2180097c0 tx=0x7fc21801c450 comp rx=0 tx=0).ready entity=mon.1 client_cookie=68ad4d4370a393a1 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:03:35.539 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:35.534+0000 7fc225ffb640 1 -- 172.21.15.16:0/1880392941 <== mon.1 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2180ac410 con 0x7fc2280c9b40 2023-05-22T16:03:35.539 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:35.534+0000 7fc225ffb640 1 -- 172.21.15.16:0/1880392941 <== mon.1 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2180ac570 con 0x7fc2280c9b40 2023-05-22T16:03:35.539 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:35.534+0000 7fc225ffb640 1 -- 172.21.15.16:0/1880392941 <== mon.1 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2180ac900 con 0x7fc2280c9b40 2023-05-22T16:03:35.540 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:35.534+0000 7fc22d780640 1 -- 172.21.15.16:0/1880392941 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc2280c9b40 msgr2=0x7fc22815e460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:35.540 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:35.534+0000 7fc22d780640 1 --2- 172.21.15.16:0/1880392941 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc2280c9b40 0x7fc22815e460 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7fc2180097c0 tx=0x7fc21801c450 comp rx=0 tx=0).stop 2023-05-22T16:03:35.540 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:35.534+0000 7fc22d780640 1 -- 172.21.15.16:0/1880392941 shutdown_connections 2023-05-22T16:03:35.540 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:35.534+0000 7fc22d780640 1 --2- 172.21.15.16:0/1880392941 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc228070df0 0x7fc228071210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:35.540 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:35.534+0000 7fc22d780640 1 --2- 172.21.15.16:0/1880392941 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc2280c9b40 0x7fc22815e460 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:35.540 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:35.534+0000 7fc22d780640 1 -- 172.21.15.16:0/1880392941 >> 172.21.15.16:0/1880392941 conn(0x7fc228069720 msgr2=0x7fc228069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:03:35.541 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:35.534+0000 7fc22d780640 1 -- 172.21.15.16:0/1880392941 shutdown_connections 2023-05-22T16:03:35.541 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:35.534+0000 7fc22d780640 1 -- 172.21.15.16:0/1880392941 wait complete. 2023-05-22T16:03:35.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:35.534+0000 7fc22d780640 1 Processor -- start 2023-05-22T16:03:35.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:35.534+0000 7fc22d780640 1 -- start start 2023-05-22T16:03:35.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:35.538+0000 7fc22d780640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc228070df0 0x7fc228158bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:35.544 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:35.538+0000 7fc22d780640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc2280c9b40 0x7fc228159370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:35.544 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:35.538+0000 7fc22d780640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc2280ca4e0 0x7fc2280cd700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:35.544 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:35.538+0000 7fc22d780640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2280cd270 con 0x7fc228070df0 2023-05-22T16:03:35.544 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:35.538+0000 7fc22d780640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2280cdc40 con 0x7fc2280ca4e0 2023-05-22T16:03:35.544 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:35.538+0000 7fc22d780640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc228154db0 con 0x7fc2280c9b40 2023-05-22T16:03:35.544 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:35.538+0000 7fc226ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc228070df0 0x7fc228158bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:35.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:35.538+0000 7fc226ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc228070df0 0x7fc228158bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48070/0 (socket says 172.21.15.16:48070) 2023-05-22T16:03:35.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:35.538+0000 7fc226ffd640 1 -- 172.21.15.16:0/3802667719 learned_addr learned my addr 172.21.15.16:0/3802667719 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:03:35.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:35.538+0000 7fc2267fc640 1 --2- 172.21.15.16:0/3802667719 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc2280c9b40 0x7fc228159370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:35.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:35.538+0000 7fc2277fe640 1 --2- 172.21.15.16:0/3802667719 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc2280ca4e0 0x7fc2280cd700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:35.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:35.538+0000 7fc226ffd640 1 -- 172.21.15.16:0/3802667719 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc2280c9b40 msgr2=0x7fc228159370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:35.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:35.538+0000 7fc226ffd640 1 --2- 172.21.15.16:0/3802667719 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc2280c9b40 0x7fc228159370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:35.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:35.538+0000 7fc226ffd640 1 -- 172.21.15.16:0/3802667719 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc2280ca4e0 msgr2=0x7fc2280cd700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:35.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:35.538+0000 7fc226ffd640 1 --2- 172.21.15.16:0/3802667719 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc2280ca4e0 0x7fc2280cd700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:35.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:35.538+0000 7fc226ffd640 1 -- 172.21.15.16:0/3802667719 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc220009b30 con 0x7fc228070df0 2023-05-22T16:03:35.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:35.538+0000 7fc226ffd640 1 --2- 172.21.15.16:0/3802667719 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc228070df0 0x7fc228158bf0 secure :-1 s=READY pgs=2225 cs=0 l=1 rev1=1 crypto rx=0x7fc21c00c370 tx=0x7fc21c00c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:03:35.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:35.538+0000 7fc20f7fe640 1 -- 172.21.15.16:0/3802667719 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc21c0074e0 con 0x7fc228070df0 2023-05-22T16:03:35.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:35.538+0000 7fc20f7fe640 1 -- 172.21.15.16:0/3802667719 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc21c007660 con 0x7fc228070df0 2023-05-22T16:03:35.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:35.538+0000 7fc22d780640 1 -- 172.21.15.16:0/3802667719 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc218009420 con 0x7fc228070df0 2023-05-22T16:03:35.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:35.538+0000 7fc20f7fe640 1 -- 172.21.15.16:0/3802667719 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc21c004a30 con 0x7fc228070df0 2023-05-22T16:03:35.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:35.538+0000 7fc22d780640 1 -- 172.21.15.16:0/3802667719 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2280c8e80 con 0x7fc228070df0 2023-05-22T16:03:35.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:35.538+0000 7fc22d780640 1 -- 172.21.15.16:0/3802667719 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2280ca8b0 con 0x7fc228070df0 2023-05-22T16:03:35.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:35.538+0000 7fc20f7fe640 1 -- 172.21.15.16:0/3802667719 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 82) v1 ==== 84332+0+0 (secure 0 0 0) 0x7fc21c004c50 con 0x7fc228070df0 2023-05-22T16:03:35.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:35.538+0000 7fc20f7fe640 1 --2- 172.21.15.16:0/3802667719 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fc2140654a0 0x7fc214067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:35.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:35.542+0000 7fc20f7fe640 1 -- 172.21.15.16:0/3802667719 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(175..175 src has 1..175) v4 ==== 5586+0+0 (secure 0 0 0) 0x7fc21c0820d0 con 0x7fc228070df0 2023-05-22T16:03:35.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:35.542+0000 7fc2267fc640 1 --2- 172.21.15.16:0/3802667719 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fc2140654a0 0x7fc214067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:35.549 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:35.542+0000 7fc2267fc640 1 --2- 172.21.15.16:0/3802667719 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fc2140654a0 0x7fc214067950 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fc2180ae000 tx=0x7fc2180a6040 comp rx=0 tx=0).ready entity=mgr.10032 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:03:35.550 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:35.550+0000 7fc20f7fe640 1 -- 172.21.15.16:0/3802667719 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fc21c04fea0 con 0x7fc228070df0 2023-05-22T16:03:35.754 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:35.754+0000 7fc22d780640 1 -- 172.21.15.16:0/3802667719 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fc2280c9010 con 0x7fc228070df0 2023-05-22T16:03:35.755 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:35.754+0000 7fc20f7fe640 1 -- 172.21.15.16:0/3802667719 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v32) v1 ==== 80+0+5304 (secure 0 0 0) 0x7fc21c0534c0 con 0x7fc228070df0 2023-05-22T16:03:35.756 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T16:03:35.756 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-05-22T16:03:35.757 INFO:teuthology.orchestra.run.smithi016.stdout: "auth_dump": [ 2023-05-22T16:03:35.757 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T16:03:35.757 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "mds.a", 2023-05-22T16:03:35.757 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQCAiWtkw/K0GRAA9hvTMKMmcbgKV68GeoC4ag==", 2023-05-22T16:03:35.758 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-05-22T16:03:35.758 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2023-05-22T16:03:35.758 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow *", 2023-05-22T16:03:35.759 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow *", 2023-05-22T16:03:35.759 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-05-22T16:03:35.759 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T16:03:35.760 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T16:03:35.760 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T16:03:35.760 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "mds.b", 2023-05-22T16:03:35.760 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQCAiWtkoFzfJRAAxCiB0MGM5apMVA81WDq+dA==", 2023-05-22T16:03:35.761 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-05-22T16:03:35.761 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2023-05-22T16:03:35.761 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow *", 2023-05-22T16:03:35.762 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow *", 2023-05-22T16:03:35.762 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-05-22T16:03:35.762 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T16:03:35.762 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T16:03:35.762 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T16:03:35.763 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "mds.c", 2023-05-22T16:03:35.763 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQCAiWtkvgOKHxAAMuASPtMqwyoqpZ5I+MRS7A==", 2023-05-22T16:03:35.763 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-05-22T16:03:35.763 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2023-05-22T16:03:35.763 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow *", 2023-05-22T16:03:35.764 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow *", 2023-05-22T16:03:35.764 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-05-22T16:03:35.764 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T16:03:35.765 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T16:03:35.765 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T16:03:35.765 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "mds.d", 2023-05-22T16:03:35.765 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQCAiWtk6hPQKxAAlCtfC36vja5ftz6uOqUhHg==", 2023-05-22T16:03:35.766 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-05-22T16:03:35.766 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2023-05-22T16:03:35.766 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow *", 2023-05-22T16:03:35.766 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow *", 2023-05-22T16:03:35.766 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-05-22T16:03:35.766 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T16:03:35.766 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T16:03:35.767 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T16:03:35.767 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.0", 2023-05-22T16:03:35.767 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQCCiWtk6ZLoCRAAKdCQitZuPnWcIpkiXhJeww==", 2023-05-22T16:03:35.767 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-05-22T16:03:35.767 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2023-05-22T16:03:35.768 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2023-05-22T16:03:35.768 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-05-22T16:03:35.768 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T16:03:35.768 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T16:03:35.768 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T16:03:35.769 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.1", 2023-05-22T16:03:35.769 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQCFiWtkXtVeERAAkgPeDKuFmWXrDiCyqWOYwQ==", 2023-05-22T16:03:35.769 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-05-22T16:03:35.769 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2023-05-22T16:03:35.769 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2023-05-22T16:03:35.769 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-05-22T16:03:35.769 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T16:03:35.769 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T16:03:35.770 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T16:03:35.770 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.2", 2023-05-22T16:03:35.770 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQCIiWtkdGaRHRAAx3POIp3mnz7p7XcCVMXm5Q==", 2023-05-22T16:03:35.770 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-05-22T16:03:35.770 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2023-05-22T16:03:35.770 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2023-05-22T16:03:35.770 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-05-22T16:03:35.771 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T16:03:35.771 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T16:03:35.771 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T16:03:35.771 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.3", 2023-05-22T16:03:35.771 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQCLiWtkSBydJhAAsgc19Xaf/GOyy9nm9MLdJQ==", 2023-05-22T16:03:35.771 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-05-22T16:03:35.771 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2023-05-22T16:03:35.772 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2023-05-22T16:03:35.772 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-05-22T16:03:35.772 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T16:03:35.772 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T16:03:35.772 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T16:03:35.773 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.4", 2023-05-22T16:03:35.773 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQCRiWtkueYvJRAARXbHlKHrTbgTqBArRJfvBA==", 2023-05-22T16:03:35.773 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-05-22T16:03:35.773 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2023-05-22T16:03:35.773 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2023-05-22T16:03:35.773 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-05-22T16:03:35.773 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T16:03:35.773 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T16:03:35.774 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T16:03:35.774 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.5", 2023-05-22T16:03:35.774 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQCUiWtkH3hwLRAAUekP6y+wkJVPh+OPDiZbzQ==", 2023-05-22T16:03:35.774 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-05-22T16:03:35.774 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2023-05-22T16:03:35.774 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2023-05-22T16:03:35.774 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-05-22T16:03:35.774 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T16:03:35.775 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T16:03:35.775 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T16:03:35.775 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.6", 2023-05-22T16:03:35.775 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQCXiWtkLsnBNhAAQ8ilX8ZU4ncJ2JJU0I+U4w==", 2023-05-22T16:03:35.775 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-05-22T16:03:35.775 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2023-05-22T16:03:35.775 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2023-05-22T16:03:35.775 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-05-22T16:03:35.776 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T16:03:35.776 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T16:03:35.776 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T16:03:35.776 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.7", 2023-05-22T16:03:35.776 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQCbiWtkUvXjBBAApMWw7wdfxWq0WEQDL2qprA==", 2023-05-22T16:03:35.776 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-05-22T16:03:35.776 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2023-05-22T16:03:35.777 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2023-05-22T16:03:35.777 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-05-22T16:03:35.777 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T16:03:35.777 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T16:03:35.777 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T16:03:35.777 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "client.0", 2023-05-22T16:03:35.777 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQCAiWtkZwuUMRAAnYqSFiaT/QRsajdjNgTetA==", 2023-05-22T16:03:35.778 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-05-22T16:03:35.778 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2023-05-22T16:03:35.778 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow r", 2023-05-22T16:03:35.778 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow rw tag cephfs data=cephfs" 2023-05-22T16:03:35.778 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T16:03:35.778 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T16:03:35.779 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T16:03:35.779 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "client.1", 2023-05-22T16:03:35.779 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQCAiWtkPimmORAABzNL2rANz5TgCl3p++GVng==", 2023-05-22T16:03:35.779 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-05-22T16:03:35.779 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2023-05-22T16:03:35.779 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow r", 2023-05-22T16:03:35.779 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow rw", 2023-05-22T16:03:35.780 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow rwx" 2023-05-22T16:03:35.780 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T16:03:35.780 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T16:03:35.780 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T16:03:35.780 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "client.2", 2023-05-22T16:03:35.780 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQCAiWtkqkPMNhAAqugrnD4Mh/4At8QAFmzqcg==", 2023-05-22T16:03:35.781 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-05-22T16:03:35.781 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2023-05-22T16:03:35.781 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow r", 2023-05-22T16:03:35.781 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow rw", 2023-05-22T16:03:35.781 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow rwx" 2023-05-22T16:03:35.782 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T16:03:35.782 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T16:03:35.782 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T16:03:35.782 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "client.3", 2023-05-22T16:03:35.782 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQCBiWtkpvlWAxAAPuedivXBGZGHIO6hSnz8PQ==", 2023-05-22T16:03:35.782 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-05-22T16:03:35.782 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2023-05-22T16:03:35.783 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow r", 2023-05-22T16:03:35.783 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow rw", 2023-05-22T16:03:35.783 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow rwx" 2023-05-22T16:03:35.783 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T16:03:35.784 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T16:03:35.784 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T16:03:35.784 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "client.admin", 2023-05-22T16:03:35.785 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQCAiWtkxG9hAhAAfnUnRrBRUFoCEtsb2riNOQ==", 2023-05-22T16:03:35.785 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-05-22T16:03:35.785 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow *", 2023-05-22T16:03:35.785 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow *", 2023-05-22T16:03:35.786 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow *", 2023-05-22T16:03:35.786 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-05-22T16:03:35.786 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T16:03:35.786 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T16:03:35.787 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T16:03:35.787 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "mgr.x", 2023-05-22T16:03:35.787 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQCAiWtkkaIqFhAAb/4BI6FYWOy3qOSaylXLSw==", 2023-05-22T16:03:35.787 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-05-22T16:03:35.788 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow *", 2023-05-22T16:03:35.788 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile mgr", 2023-05-22T16:03:35.788 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-05-22T16:03:35.789 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T16:03:35.789 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T16:03:35.789 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T16:03:35.789 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "mgr.y", 2023-05-22T16:03:35.790 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQCAiWtkhgVWExAAjDoh7cexFVfsigsYfGkmVg==", 2023-05-22T16:03:35.790 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-05-22T16:03:35.790 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow *", 2023-05-22T16:03:35.791 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile mgr", 2023-05-22T16:03:35.791 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-05-22T16:03:35.791 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T16:03:35.792 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T16:03:35.792 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-05-22T16:03:35.792 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-05-22T16:03:35.794 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:35.758+0000 7fc22d780640 1 -- 172.21.15.16:0/3802667719 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fc2140654a0 msgr2=0x7fc214067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:35.795 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:35.758+0000 7fc22d780640 1 --2- 172.21.15.16:0/3802667719 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fc2140654a0 0x7fc214067950 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fc2180ae000 tx=0x7fc2180a6040 comp rx=0 tx=0).stop 2023-05-22T16:03:35.795 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:35.758+0000 7fc22d780640 1 -- 172.21.15.16:0/3802667719 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc228070df0 msgr2=0x7fc228158bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:35.795 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:35.758+0000 7fc22d780640 1 --2- 172.21.15.16:0/3802667719 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc228070df0 0x7fc228158bf0 secure :-1 s=READY pgs=2225 cs=0 l=1 rev1=1 crypto rx=0x7fc21c00c370 tx=0x7fc21c00c830 comp rx=0 tx=0).stop 2023-05-22T16:03:35.795 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:35.758+0000 7fc22d780640 1 -- 172.21.15.16:0/3802667719 shutdown_connections 2023-05-22T16:03:35.796 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:35.758+0000 7fc22d780640 1 --2- 172.21.15.16:0/3802667719 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc2280ca4e0 0x7fc2280cd700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:35.796 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:35.758+0000 7fc22d780640 1 --2- 172.21.15.16:0/3802667719 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc2280c9b40 0x7fc228159370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:35.796 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:35.758+0000 7fc22d780640 1 --2- 172.21.15.16:0/3802667719 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fc2140654a0 0x7fc214067950 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:35.797 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:35.758+0000 7fc22d780640 1 --2- 172.21.15.16:0/3802667719 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc228070df0 0x7fc228158bf0 unknown :-1 s=CLOSED pgs=2225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:35.797 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:35.758+0000 7fc22d780640 1 -- 172.21.15.16:0/3802667719 >> 172.21.15.16:0/3802667719 conn(0x7fc228069720 msgr2=0x7fc228153150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:03:35.797 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:35.758+0000 7fc22d780640 1 -- 172.21.15.16:0/3802667719 shutdown_connections 2023-05-22T16:03:35.797 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:35.758+0000 7fc22d780640 1 -- 172.21.15.16:0/3802667719 wait complete. 2023-05-22T16:03:35.798 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-05-22T16:03:35.917 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:35.918+0000 7fbffb32d640 1 Processor -- start 2023-05-22T16:03:35.917 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:35.918+0000 7fbffb32d640 1 -- start start 2023-05-22T16:03:35.918 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:35.918+0000 7fbffb32d640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fbff41531b0 0x7fbff40afb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:35.918 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:35.918+0000 7fbffb32d640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fbff40b0080 0x7fbff4159330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:35.918 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:35.918+0000 7fbffb32d640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbff415a0a0 con 0x7fbff4159870 2023-05-22T16:03:35.918 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:35.918+0000 7fbffb32d640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbff40b0520 con 0x7fbff40b0080 2023-05-22T16:03:35.918 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:35.918+0000 7fbffb32d640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbff4153580 con 0x7fbff41531b0 2023-05-22T16:03:35.918 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:35.918+0000 7fbff88c8640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fbff40b0080 0x7fbff4159330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:35.919 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:35.918+0000 7fbff88c8640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fbff40b0080 0x7fbff4159330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48084/0 (socket says 172.21.15.16:48084) 2023-05-22T16:03:35.919 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:35.918+0000 7fbff88c8640 1 -- 172.21.15.16:0/2202688081 learned_addr learned my addr 172.21.15.16:0/2202688081 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:03:35.919 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:35.918+0000 7fbff98ca640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fbff4159870 0x7fbff40c8ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:48172/0 (socket says 172.21.15.16:48172) 2023-05-22T16:03:35.920 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:35.918+0000 7fbff90c9640 1 --2- 172.21.15.16:0/2202688081 >> v2:172.21.15.62:3300/0 conn(0x7fbff41531b0 0x7fbff40afb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:35.920 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:35.918+0000 7fbff88c8640 1 -- 172.21.15.16:0/2202688081 >> v2:172.21.15.62:3300/0 conn(0x7fbff41531b0 msgr2=0x7fbff40afb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:03:35.920 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:35.918+0000 7fbff88c8640 1 --2- 172.21.15.16:0/2202688081 >> v2:172.21.15.62:3300/0 conn(0x7fbff41531b0 0x7fbff40afb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:35.920 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:35.918+0000 7fbff88c8640 1 -- 172.21.15.16:0/2202688081 >> v1:172.21.15.16:6789/0 conn(0x7fbff4159870 legacy=0x7fbff40c8ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:35.920 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:35.918+0000 7fbfebfff640 1 -- 172.21.15.16:0/2202688081 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3533950312 0 0) 0x7fbff415a0a0 con 0x7fbff4159870 2023-05-22T16:03:35.920 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:35.918+0000 7fbff88c8640 1 -- 172.21.15.16:0/2202688081 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbff415d5e0 con 0x7fbff40b0080 2023-05-22T16:03:35.921 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:35.918+0000 7fbff90c9640 1 --2- 172.21.15.16:0/2202688081 >> v2:172.21.15.62:3300/0 conn(0x7fbff41531b0 0x7fbff40afb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T16:03:35.922 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:35.922+0000 7fbff88c8640 1 --2- 172.21.15.16:0/2202688081 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbff40b0080 0x7fbff4159330 secure :-1 s=READY pgs=2227 cs=0 l=1 rev1=1 crypto rx=0x7fbff40b0790 tx=0x7fbfe401c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=206d25be39ddc77b server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:03:35.922 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:35.922+0000 7fbfebfff640 1 -- 172.21.15.16:0/2202688081 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfe40aa430 con 0x7fbff40b0080 2023-05-22T16:03:35.922 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:35.922+0000 7fbfebfff640 1 -- 172.21.15.16:0/2202688081 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfe40aa590 con 0x7fbff40b0080 2023-05-22T16:03:35.922 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:35.922+0000 7fbfebfff640 1 -- 172.21.15.16:0/2202688081 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfe40aa920 con 0x7fbff40b0080 2023-05-22T16:03:35.923 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:35.922+0000 7fbffb32d640 1 -- 172.21.15.16:0/2202688081 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbff40b0080 msgr2=0x7fbff4159330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:35.923 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:35.922+0000 7fbffb32d640 1 --2- 172.21.15.16:0/2202688081 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbff40b0080 0x7fbff4159330 secure :-1 s=READY pgs=2227 cs=0 l=1 rev1=1 crypto rx=0x7fbff40b0790 tx=0x7fbfe401c4d0 comp rx=0 tx=0).stop 2023-05-22T16:03:35.923 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:35.922+0000 7fbffb32d640 1 -- 172.21.15.16:0/2202688081 shutdown_connections 2023-05-22T16:03:35.923 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:35.922+0000 7fbffb32d640 1 --2- 172.21.15.16:0/2202688081 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbff40b0080 0x7fbff4159330 unknown :-1 s=CLOSED pgs=2227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:35.923 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:35.922+0000 7fbffb32d640 1 --2- 172.21.15.16:0/2202688081 >> v2:172.21.15.62:3300/0 conn(0x7fbff41531b0 0x7fbff40afb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:35.923 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:35.922+0000 7fbffb32d640 1 -- 172.21.15.16:0/2202688081 >> 172.21.15.16:0/2202688081 conn(0x7fbff4069720 msgr2=0x7fbff4069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:03:35.924 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:35.922+0000 7fbffb32d640 1 -- 172.21.15.16:0/2202688081 shutdown_connections 2023-05-22T16:03:35.924 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:35.922+0000 7fbffb32d640 1 -- 172.21.15.16:0/2202688081 wait complete. 2023-05-22T16:03:35.927 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:35.922+0000 7fbffb32d640 1 Processor -- start 2023-05-22T16:03:35.927 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:35.922+0000 7fbffb32d640 1 -- start start 2023-05-22T16:03:35.927 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:35.922+0000 7fbffb32d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbff40b0080 0x7fbff40c7bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:35.927 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:35.922+0000 7fbffb32d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbff41531b0 0x7fbff40b1900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:35.927 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:35.922+0000 7fbffb32d640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbff4159870 0x7fbff40b2280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:35.927 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:35.922+0000 7fbffb32d640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbff40c80f0 con 0x7fbff40b0080 2023-05-22T16:03:35.928 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:35.922+0000 7fbffb32d640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbff40b0eb0 con 0x7fbff41531b0 2023-05-22T16:03:35.928 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:35.922+0000 7fbffb32d640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbff40b1e40 con 0x7fbff4159870 2023-05-22T16:03:35.928 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:35.922+0000 7fbff90c9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbff40b0080 0x7fbff40c7bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:35.928 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:35.922+0000 7fbff90c9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbff40b0080 0x7fbff40c7bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48098/0 (socket says 172.21.15.16:48098) 2023-05-22T16:03:35.928 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:35.922+0000 7fbff90c9640 1 -- 172.21.15.16:0/3876389705 learned_addr learned my addr 172.21.15.16:0/3876389705 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:03:35.928 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:35.922+0000 7fbff88c8640 1 --2- 172.21.15.16:0/3876389705 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbff41531b0 0x7fbff40b1900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:35.928 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:35.922+0000 7fbff98ca640 1 --2- 172.21.15.16:0/3876389705 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbff4159870 0x7fbff40b2280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:35.929 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:35.922+0000 7fbff90c9640 1 -- 172.21.15.16:0/3876389705 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbff4159870 msgr2=0x7fbff40b2280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:35.929 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:35.922+0000 7fbff90c9640 1 --2- 172.21.15.16:0/3876389705 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbff4159870 0x7fbff40b2280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:35.929 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:35.922+0000 7fbff90c9640 1 -- 172.21.15.16:0/3876389705 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbff41531b0 msgr2=0x7fbff40b1900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:35.929 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:35.922+0000 7fbff90c9640 1 --2- 172.21.15.16:0/3876389705 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbff41531b0 0x7fbff40b1900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:35.929 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:35.922+0000 7fbff90c9640 1 -- 172.21.15.16:0/3876389705 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfe40094d0 con 0x7fbff40b0080 2023-05-22T16:03:35.929 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:35.922+0000 7fbff98ca640 1 --2- 172.21.15.16:0/3876389705 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbff4159870 0x7fbff40b2280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T16:03:35.930 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:35.926+0000 7fbff90c9640 1 --2- 172.21.15.16:0/3876389705 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbff40b0080 0x7fbff40c7bb0 secure :-1 s=READY pgs=2228 cs=0 l=1 rev1=1 crypto rx=0x7fbff000d800 tx=0x7fbff000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:03:35.930 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:35.926+0000 7fbfe9ffb640 1 -- 172.21.15.16:0/3876389705 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbff000b6c0 con 0x7fbff40b0080 2023-05-22T16:03:35.930 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:35.926+0000 7fbfe9ffb640 1 -- 172.21.15.16:0/3876389705 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbff000b820 con 0x7fbff40b0080 2023-05-22T16:03:35.930 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:35.926+0000 7fbffb32d640 1 -- 172.21.15.16:0/3876389705 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbff415d0c0 con 0x7fbff40b0080 2023-05-22T16:03:35.930 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:35.926+0000 7fbfe9ffb640 1 -- 172.21.15.16:0/3876389705 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbff000bbd0 con 0x7fbff40b0080 2023-05-22T16:03:35.930 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:35.926+0000 7fbffb32d640 1 -- 172.21.15.16:0/3876389705 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbff415cde0 con 0x7fbff40b0080 2023-05-22T16:03:35.930 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:35.926+0000 7fbffb32d640 1 -- 172.21.15.16:0/3876389705 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbff4159c40 con 0x7fbff40b0080 2023-05-22T16:03:35.930 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:35.926+0000 7fbfe9ffb640 1 -- 172.21.15.16:0/3876389705 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 82) v1 ==== 84332+0+0 (secure 0 0 0) 0x7fbff000bd30 con 0x7fbff40b0080 2023-05-22T16:03:35.931 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:35.926+0000 7fbfe9ffb640 1 --2- 172.21.15.16:0/3876389705 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fbfd80654d0 0x7fbfd8067980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:35.931 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:35.926+0000 7fbff88c8640 1 --2- 172.21.15.16:0/3876389705 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fbfd80654d0 0x7fbfd8067980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:35.931 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:35.926+0000 7fbfe9ffb640 1 -- 172.21.15.16:0/3876389705 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(175..175 src has 1..175) v4 ==== 5586+0+0 (secure 0 0 0) 0x7fbff0082ce0 con 0x7fbff40b0080 2023-05-22T16:03:35.931 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:35.926+0000 7fbff88c8640 1 --2- 172.21.15.16:0/3876389705 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fbfd80654d0 0x7fbfd8067980 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fbfe40ae000 tx=0x7fbfe40a7000 comp rx=0 tx=0).ready entity=mgr.10032 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:03:35.932 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:35.930+0000 7fbfe9ffb640 1 -- 172.21.15.16:0/3876389705 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fbff0050a10 con 0x7fbff40b0080 2023-05-22T16:03:36.084 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.082+0000 7fbffb32d640 1 -- 172.21.15.16:0/3876389705 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbff404dbe0 con 0x7fbff40b0080 2023-05-22T16:03:36.085 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.086+0000 7fbfe9ffb640 1 -- 172.21.15.16:0/3876389705 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v175) v1 ==== 74+0+14428 (secure 0 0 0) 0x7fbff0054030 con 0x7fbff40b0080 2023-05-22T16:03:36.086 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T16:03:36.086 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":175,"fsid":"b2719834-7847-4d98-90a6-f65d57f85c49","created":"2023-05-22T15:26:25.259314+0000","modified":"2023-05-22T16:03:35.378249+0000","last_up_change":"2023-05-22T15:26:36.448524+0000","last_in_change":"2023-05-22T15:26:27.675668+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-05-22T15:26:38.019781+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-05-22T15:26:43.570514+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9e7ff854-b25f-4cbb-a5c2-1799d9c53d2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":37021},{"type":"v1","addr":"172.21.15.16:6825","nonce":37021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":37021},{"type":"v1","addr":"172.21.15.16:6827","nonce":37021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":37021},{"type":"v1","addr":"172.21.15.16:6831","nonce":37021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":37021},{"type":"v1","addr":"172.21.15.16:6829","nonce":37021}]},"public_addr":"172.21.15.16:6825/37021","cluster_addr":"172.21.15.16:6827/37021","heartbeat_back_addr":"172.21.15.16:6831/37021","heartbeat_front_addr":"172.21.15.16:6829/37021","state":["exists","up"]},{"osd":1,"uuid":"4702fbc9-a927-4565-aca5-41dc69cc8264","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":37008},{"type":"v1","addr":"172.21.15.16:6801","nonce":37008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":37008},{"type":"v1","addr":"172.21.15.16:6803","nonce":37008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":37008},{"type":"v1","addr":"172.21.15.16:6807","nonce":37008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":37008},{"type":"v1","addr":"172.21.15.16:6805","nonce":37008}]},"public_addr":"172.21.15.16:6801/37008","cluster_addr":"172.21.15.16:6803/37008","heartbeat_back_addr":"172.21.15.16:6807/37008","heartbeat_front_addr":"172.21.15.16:6805/37008","state":["exists","up"]},{"osd":2,"uuid":"1e7bba46-4eba-47ef-bb81-5e2d371638ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":37009},{"type":"v1","addr":"172.21.15.16:6809","nonce":37009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":37009},{"type":"v1","addr":"172.21.15.16:6811","nonce":37009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":37009},{"type":"v1","addr":"172.21.15.16:6815","nonce":37009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":37009},{"type":"v1","addr":"172.21.15.16:6813","nonce":37009}]},"public_addr":"172.21.15.16:6809/37009","cluster_addr":"172.21.15.16:6811/37009","heartbeat_back_addr":"172.21.15.16:6815/37009","heartbeat_front_addr":"172.21.15.16:6813/37009","state":["exists","up"]},{"osd":3,"uuid":"383cacc0-5a88-4904-990e-498ca59ef48a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":37020},{"type":"v1","addr":"172.21.15.16:6817","nonce":37020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":37020},{"type":"v1","addr":"172.21.15.16:6819","nonce":37020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":37020},{"type":"v1","addr":"172.21.15.16:6823","nonce":37020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":37020},{"type":"v1","addr":"172.21.15.16:6821","nonce":37020}]},"public_addr":"172.21.15.16:6817/37020","cluster_addr":"172.21.15.16:6819/37020","heartbeat_back_addr":"172.21.15.16:6823/37020","heartbeat_front_addr":"172.21.15.16:6821/37020","state":["exists","up"]},{"osd":4,"uuid":"970500ef-95b7-42c1-b080-eed4fae7013d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":37353},{"type":"v1","addr":"172.21.15.62:6817","nonce":37353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6819","nonce":37353},{"type":"v1","addr":"172.21.15.62:6822","nonce":37353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37353},{"type":"v1","addr":"172.21.15.62:6831","nonce":37353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37353},{"type":"v1","addr":"172.21.15.62:6826","nonce":37353}]},"public_addr":"172.21.15.62:6817/37353","cluster_addr":"172.21.15.62:6822/37353","heartbeat_back_addr":"172.21.15.62:6831/37353","heartbeat_front_addr":"172.21.15.62:6826/37353","state":["exists","up"]},{"osd":5,"uuid":"022d67db-d4d8-445b-b1b9-8b446a00f5c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":37352},{"type":"v1","addr":"172.21.15.62:6801","nonce":37352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":37352},{"type":"v1","addr":"172.21.15.62:6803","nonce":37352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":37352},{"type":"v1","addr":"172.21.15.62:6807","nonce":37352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":37352},{"type":"v1","addr":"172.21.15.62:6805","nonce":37352}]},"public_addr":"172.21.15.62:6801/37352","cluster_addr":"172.21.15.62:6803/37352","heartbeat_back_addr":"172.21.15.62:6807/37352","heartbeat_front_addr":"172.21.15.62:6805/37352","state":["exists","up"]},{"osd":6,"uuid":"42bd372b-6688-4a1b-8407-e16e8c266e33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":37354},{"type":"v1","addr":"172.21.15.62:6820","nonce":37354}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":37354},{"type":"v1","addr":"172.21.15.62:6823","nonce":37354}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6829","nonce":37354},{"type":"v1","addr":"172.21.15.62:6830","nonce":37354}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6825","nonce":37354},{"type":"v1","addr":"172.21.15.62:6827","nonce":37354}]},"public_addr":"172.21.15.62:6820/37354","cluster_addr":"172.21.15.62:6823/37354","heartbeat_back_addr":"172.21.15.62:6830/37354","heartbeat_front_addr":"172.21.15.62:6827/37354","state":["exists","up"]},{"osd":7,"uuid":"ac70b2a3-f41b-4bf2-b227-7f2cbb55426d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":163,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37355},{"type":"v1","addr":"172.21.15.62:6809","nonce":37355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37355},{"type":"v1","addr":"172.21.15.62:6811","nonce":37355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":37355},{"type":"v1","addr":"172.21.15.62:6815","nonce":37355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":37355},{"type":"v1","addr":"172.21.15.62:6813","nonce":37355}]},"public_addr":"172.21.15.62:6809/37355","cluster_addr":"172.21.15.62:6811/37355","heartbeat_back_addr":"172.21.15.62:6815/37355","heartbeat_front_addr":"172.21.15.62:6813/37355","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.368896+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.107273+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.276888+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.287502+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.124791+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.251937+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.255679+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.232797+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":17,"snaps":[{"begin":2,"length":2}]},{"pool":21,"snaps":[{"begin":25,"length":9}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-05-22T16:03:36.089 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.086+0000 7fbffb32d640 1 -- 172.21.15.16:0/3876389705 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fbfd80654d0 msgr2=0x7fbfd8067980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:36.089 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.086+0000 7fbffb32d640 1 --2- 172.21.15.16:0/3876389705 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fbfd80654d0 0x7fbfd8067980 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fbfe40ae000 tx=0x7fbfe40a7000 comp rx=0 tx=0).stop 2023-05-22T16:03:36.089 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.086+0000 7fbffb32d640 1 -- 172.21.15.16:0/3876389705 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbff40b0080 msgr2=0x7fbff40c7bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:36.089 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.086+0000 7fbffb32d640 1 --2- 172.21.15.16:0/3876389705 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbff40b0080 0x7fbff40c7bb0 secure :-1 s=READY pgs=2228 cs=0 l=1 rev1=1 crypto rx=0x7fbff000d800 tx=0x7fbff000dcc0 comp rx=0 tx=0).stop 2023-05-22T16:03:36.089 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.086+0000 7fbffb32d640 1 -- 172.21.15.16:0/3876389705 shutdown_connections 2023-05-22T16:03:36.089 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.086+0000 7fbffb32d640 1 --2- 172.21.15.16:0/3876389705 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbff4159870 0x7fbff40b2280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:36.089 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.086+0000 7fbffb32d640 1 --2- 172.21.15.16:0/3876389705 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbff41531b0 0x7fbff40b1900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:36.090 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.086+0000 7fbffb32d640 1 --2- 172.21.15.16:0/3876389705 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fbfd80654d0 0x7fbfd8067980 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:36.090 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.086+0000 7fbffb32d640 1 --2- 172.21.15.16:0/3876389705 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbff40b0080 0x7fbff40c7bb0 unknown :-1 s=CLOSED pgs=2228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:36.090 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.086+0000 7fbffb32d640 1 -- 172.21.15.16:0/3876389705 >> 172.21.15.16:0/3876389705 conn(0x7fbff4069720 msgr2=0x7fbff406b730 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:03:36.090 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.086+0000 7fbffb32d640 1 -- 172.21.15.16:0/3876389705 shutdown_connections 2023-05-22T16:03:36.090 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.086+0000 7fbffb32d640 1 -- 172.21.15.16:0/3876389705 wait complete. 2023-05-22T16:03:36.105 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-05-22T15:26:38.019781+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-05-22T15:26:43.570514+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-05-22T16:03:36.106 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-05-22T16:03:36.230 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.230+0000 7f60f4f61640 1 Processor -- start 2023-05-22T16:03:36.231 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.230+0000 7f60f4f61640 1 -- start start 2023-05-22T16:03:36.232 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.230+0000 7f60f4f61640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f60f0159970 0x7f60f0159d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:36.232 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.230+0000 7f60f4f61640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60f0070c30 con 0x7f60f015d680 2023-05-22T16:03:36.232 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.230+0000 7f60f4f61640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60f0070e60 con 0x7f60f015db00 2023-05-22T16:03:36.232 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.230+0000 7f60f4f61640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60f00cb970 con 0x7f60f0159970 2023-05-22T16:03:36.233 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.230+0000 7f60ee59c640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f60f0159970 0x7f60f0159d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:36.233 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.230+0000 7f60ee59c640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f60f0159970 0x7f60f0159d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48110/0 (socket says 172.21.15.16:48110) 2023-05-22T16:03:36.233 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.230+0000 7f60edd9b640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f60f015d680 0x7f60f00cb260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:48176/0 (socket says 172.21.15.16:48176) 2023-05-22T16:03:36.233 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.230+0000 7f60ee59c640 1 -- 172.21.15.16:0/1706342118 learned_addr learned my addr 172.21.15.16:0/1706342118 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:03:36.233 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.230+0000 7f60ee59c640 1 -- 172.21.15.16:0/1706342118 >> v1:172.21.15.62:6789/0 conn(0x7f60f015db00 legacy=0x7f60f006b6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:03:36.233 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.230+0000 7f60ee59c640 1 -- 172.21.15.16:0/1706342118 >> v1:172.21.15.16:6789/0 conn(0x7f60f015d680 legacy=0x7f60f00cb260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:36.233 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.234+0000 7f60ee59c640 1 -- 172.21.15.16:0/1706342118 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60f0071140 con 0x7f60f0159970 2023-05-22T16:03:36.233 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.234+0000 7f60ed59a640 1 -- 172.21.15.16:0/1706342118 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2852021506 0 0) 0x7f60f0070c30 con 0x7f60f015d680 2023-05-22T16:03:36.235 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.234+0000 7f60ee59c640 1 --2- 172.21.15.16:0/1706342118 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f60f0159970 0x7f60f0159d40 secure :-1 s=READY pgs=2230 cs=0 l=1 rev1=1 crypto rx=0x7f60e0009e50 tx=0x7f60e001c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2fd95f7990b17a00 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:03:36.235 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.234+0000 7f60ed59a640 1 -- 172.21.15.16:0/1706342118 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60e00a57d0 con 0x7f60f0159970 2023-05-22T16:03:36.235 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.234+0000 7f60ed59a640 1 -- 172.21.15.16:0/1706342118 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60e00a5930 con 0x7f60f0159970 2023-05-22T16:03:36.235 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.234+0000 7f60ed59a640 1 -- 172.21.15.16:0/1706342118 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60e00a5cc0 con 0x7f60f0159970 2023-05-22T16:03:36.236 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.234+0000 7f60f4f61640 1 -- 172.21.15.16:0/1706342118 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f60f0159970 msgr2=0x7f60f0159d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:36.236 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.234+0000 7f60f4f61640 1 --2- 172.21.15.16:0/1706342118 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f60f0159970 0x7f60f0159d40 secure :-1 s=READY pgs=2230 cs=0 l=1 rev1=1 crypto rx=0x7f60e0009e50 tx=0x7f60e001c910 comp rx=0 tx=0).stop 2023-05-22T16:03:36.236 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.234+0000 7f60f4f61640 1 -- 172.21.15.16:0/1706342118 shutdown_connections 2023-05-22T16:03:36.236 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.234+0000 7f60f4f61640 1 --2- 172.21.15.16:0/1706342118 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f60f0159970 0x7f60f0159d40 unknown :-1 s=CLOSED pgs=2230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:36.237 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.234+0000 7f60f4f61640 1 -- 172.21.15.16:0/1706342118 >> 172.21.15.16:0/1706342118 conn(0x7f60f0069720 msgr2=0x7f60f0069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:03:36.237 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.234+0000 7f60f4f61640 1 -- 172.21.15.16:0/1706342118 shutdown_connections 2023-05-22T16:03:36.237 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.234+0000 7f60f4f61640 1 -- 172.21.15.16:0/1706342118 wait complete. 2023-05-22T16:03:36.238 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.238+0000 7f60f4f61640 1 Processor -- start 2023-05-22T16:03:36.238 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.238+0000 7f60f4f61640 1 -- start start 2023-05-22T16:03:36.239 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.238+0000 7f60f4f61640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f60f0159970 0x7f60f0155900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:36.239 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.238+0000 7f60f4f61640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f60f015d680 0x7f60f015bf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:36.240 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.238+0000 7f60f4f61640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f60f015db00 0x7f60f015c4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:36.240 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.238+0000 7f60f4f61640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60f0156510 con 0x7f60f0159970 2023-05-22T16:03:36.240 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.238+0000 7f60f4f61640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60f00c8020 con 0x7f60f015db00 2023-05-22T16:03:36.240 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.238+0000 7f60f4f61640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60f00c8300 con 0x7f60f015d680 2023-05-22T16:03:36.240 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.238+0000 7f60ee59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f60f0159970 0x7f60f0155900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:36.240 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.238+0000 7f60ee59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f60f0159970 0x7f60f0155900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48116/0 (socket says 172.21.15.16:48116) 2023-05-22T16:03:36.241 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.238+0000 7f60ee59c640 1 -- 172.21.15.16:0/1148231305 learned_addr learned my addr 172.21.15.16:0/1148231305 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:03:36.241 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.238+0000 7f60edd9b640 1 --2- 172.21.15.16:0/1148231305 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f60f015d680 0x7f60f015bf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:36.241 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.238+0000 7f60eed9d640 1 --2- 172.21.15.16:0/1148231305 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f60f015db00 0x7f60f015c4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:36.241 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.238+0000 7f60ee59c640 1 -- 172.21.15.16:0/1148231305 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f60f015d680 msgr2=0x7f60f015bf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:36.241 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.238+0000 7f60ee59c640 1 --2- 172.21.15.16:0/1148231305 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f60f015d680 0x7f60f015bf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:36.241 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.238+0000 7f60ee59c640 1 -- 172.21.15.16:0/1148231305 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f60f015db00 msgr2=0x7f60f015c4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:36.241 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.238+0000 7f60ee59c640 1 --2- 172.21.15.16:0/1148231305 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f60f015db00 0x7f60f015c4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:36.241 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.238+0000 7f60ee59c640 1 -- 172.21.15.16:0/1148231305 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60e0009b30 con 0x7f60f0159970 2023-05-22T16:03:36.242 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.238+0000 7f60ee59c640 1 --2- 172.21.15.16:0/1148231305 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f60f0159970 0x7f60f0155900 secure :-1 s=READY pgs=2231 cs=0 l=1 rev1=1 crypto rx=0x7f60e0019520 tx=0x7f60e00196b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:03:36.242 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.238+0000 7f60deffd640 1 -- 172.21.15.16:0/1148231305 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60e00b8880 con 0x7f60f0159970 2023-05-22T16:03:36.242 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.238+0000 7f60deffd640 1 -- 172.21.15.16:0/1148231305 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60e00b89e0 con 0x7f60f0159970 2023-05-22T16:03:36.242 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.238+0000 7f60deffd640 1 -- 172.21.15.16:0/1148231305 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60e00b8d90 con 0x7f60f0159970 2023-05-22T16:03:36.242 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.238+0000 7f60f4f61640 1 -- 172.21.15.16:0/1148231305 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60f00cae20 con 0x7f60f0159970 2023-05-22T16:03:36.242 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.238+0000 7f60f4f61640 1 -- 172.21.15.16:0/1148231305 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60f00cac90 con 0x7f60f0159970 2023-05-22T16:03:36.243 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.242+0000 7f60f4f61640 1 -- 172.21.15.16:0/1148231305 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60f006bdd0 con 0x7f60f0159970 2023-05-22T16:03:36.243 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.242+0000 7f60deffd640 1 -- 172.21.15.16:0/1148231305 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 82) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f60e00b6440 con 0x7f60f0159970 2023-05-22T16:03:36.243 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.242+0000 7f60deffd640 1 --2- 172.21.15.16:0/1148231305 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f60d00654a0 0x7f60d0067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:36.243 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.242+0000 7f60edd9b640 1 --2- 172.21.15.16:0/1148231305 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f60d00654a0 0x7f60d0067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:36.244 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.242+0000 7f60deffd640 1 -- 172.21.15.16:0/1148231305 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(175..175 src has 1..175) v4 ==== 5586+0+0 (secure 0 0 0) 0x7f60e011df60 con 0x7f60f0159970 2023-05-22T16:03:36.244 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.242+0000 7f60edd9b640 1 --2- 172.21.15.16:0/1148231305 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f60d00654a0 0x7f60d0067950 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f60f00b3a10 tx=0x7f60e4009040 comp rx=0 tx=0).ready entity=mgr.10032 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:03:36.246 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.246+0000 7f60deffd640 1 -- 172.21.15.16:0/1148231305 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f60e00ebcd0 con 0x7f60f0159970 2023-05-22T16:03:36.404 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.402+0000 7f60f4f61640 1 -- 172.21.15.16:0/1148231305 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f60f004dbe0 con 0x7f60f0159970 2023-05-22T16:03:36.405 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.406+0000 7f60deffd640 1 -- 172.21.15.16:0/1148231305 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v175) v1 ==== 93+0+10 (secure 0 0 0) 0x7f60e00ef2f0 con 0x7f60f0159970 2023-05-22T16:03:36.405 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2023-05-22T16:03:36.408 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.406+0000 7f60f4f61640 1 -- 172.21.15.16:0/1148231305 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f60d00654a0 msgr2=0x7f60d0067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:36.409 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.406+0000 7f60f4f61640 1 --2- 172.21.15.16:0/1148231305 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f60d00654a0 0x7f60d0067950 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f60f00b3a10 tx=0x7f60e4009040 comp rx=0 tx=0).stop 2023-05-22T16:03:36.409 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.406+0000 7f60f4f61640 1 -- 172.21.15.16:0/1148231305 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f60f0159970 msgr2=0x7f60f0155900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:36.409 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.406+0000 7f60f4f61640 1 --2- 172.21.15.16:0/1148231305 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f60f0159970 0x7f60f0155900 secure :-1 s=READY pgs=2231 cs=0 l=1 rev1=1 crypto rx=0x7f60e0019520 tx=0x7f60e00196b0 comp rx=0 tx=0).stop 2023-05-22T16:03:36.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.406+0000 7f60f4f61640 1 -- 172.21.15.16:0/1148231305 shutdown_connections 2023-05-22T16:03:36.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.406+0000 7f60f4f61640 1 --2- 172.21.15.16:0/1148231305 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f60f015db00 0x7f60f015c4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:36.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.406+0000 7f60f4f61640 1 --2- 172.21.15.16:0/1148231305 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f60f015d680 0x7f60f015bf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:36.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.406+0000 7f60f4f61640 1 --2- 172.21.15.16:0/1148231305 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f60d00654a0 0x7f60d0067950 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:36.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.406+0000 7f60f4f61640 1 --2- 172.21.15.16:0/1148231305 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f60f0159970 0x7f60f0155900 unknown :-1 s=CLOSED pgs=2231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:36.412 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.406+0000 7f60f4f61640 1 -- 172.21.15.16:0/1148231305 >> 172.21.15.16:0/1148231305 conn(0x7f60f0069720 msgr2=0x7f60f0069af0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:03:36.412 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.406+0000 7f60f4f61640 1 -- 172.21.15.16:0/1148231305 shutdown_connections 2023-05-22T16:03:36.412 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.406+0000 7f60f4f61640 1 -- 172.21.15.16:0/1148231305 wait complete. 2023-05-22T16:03:36.426 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-05-22T16:03:36.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.550+0000 7f7abad0e640 1 Processor -- start 2023-05-22T16:03:36.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.550+0000 7f7abad0e640 1 -- start start 2023-05-22T16:03:36.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.554+0000 7f7abad0e640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7ab415bca0 0x7f7ab40cda80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:36.554 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.554+0000 7f7abad0e640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f7ab40b5a50 0x7f7ab40d28d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:36.554 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.554+0000 7f7abad0e640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ab40d2e10 con 0x7f7ab40d2480 2023-05-22T16:03:36.554 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.554+0000 7f7abad0e640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ab40cd660 con 0x7f7ab40b5a50 2023-05-22T16:03:36.554 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.554+0000 7f7abad0e640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ab40cccc0 con 0x7f7ab415bca0 2023-05-22T16:03:36.555 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.554+0000 7f7ab92ab640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f7ab40b5a50 0x7f7ab40d28d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:36.555 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.554+0000 7f7ab92ab640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f7ab40b5a50 0x7f7ab40d28d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48122/0 (socket says 172.21.15.16:48122) 2023-05-22T16:03:36.555 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.554+0000 7f7ab92ab640 1 -- 172.21.15.16:0/2009455602 learned_addr learned my addr 172.21.15.16:0/2009455602 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:03:36.555 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.554+0000 7f7ab8aaa640 1 --2- 172.21.15.16:0/2009455602 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7ab415bca0 0x7f7ab40cda80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:36.555 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.554+0000 7f7ab92ab640 1 -- 172.21.15.16:0/2009455602 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7ab415bca0 msgr2=0x7f7ab40cda80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:03:36.555 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.554+0000 7f7ab92ab640 1 --2- 172.21.15.16:0/2009455602 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7ab415bca0 0x7f7ab40cda80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:36.555 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.554+0000 7f7ab92ab640 1 -- 172.21.15.16:0/2009455602 >> v1:172.21.15.62:6789/0 conn(0x7f7ab40d2480 legacy=0x7f7ab40b5340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:03:36.556 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.554+0000 7f7ab92ab640 1 -- 172.21.15.16:0/2009455602 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ab40ccfa0 con 0x7f7ab40b5a50 2023-05-22T16:03:36.557 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.554+0000 7f7ab92ab640 1 --2- 172.21.15.16:0/2009455602 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ab40b5a50 0x7f7ab40d28d0 secure :-1 s=READY pgs=2232 cs=0 l=1 rev1=1 crypto rx=0x7f7aa4009870 tx=0x7f7aa401c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad0cc1e78cec421d server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:03:36.557 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.554+0000 7f7ab37fe640 1 -- 172.21.15.16:0/2009455602 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7aa40aa610 con 0x7f7ab40b5a50 2023-05-22T16:03:36.557 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.554+0000 7f7ab37fe640 1 -- 172.21.15.16:0/2009455602 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7aa40a6040 con 0x7f7ab40b5a50 2023-05-22T16:03:36.557 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.554+0000 7f7ab37fe640 1 -- 172.21.15.16:0/2009455602 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7aa40aa990 con 0x7f7ab40b5a50 2023-05-22T16:03:36.557 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.558+0000 7f7abad0e640 1 -- 172.21.15.16:0/2009455602 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ab40b5a50 msgr2=0x7f7ab40d28d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:36.557 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.558+0000 7f7abad0e640 1 --2- 172.21.15.16:0/2009455602 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ab40b5a50 0x7f7ab40d28d0 secure :-1 s=READY pgs=2232 cs=0 l=1 rev1=1 crypto rx=0x7f7aa4009870 tx=0x7f7aa401c4d0 comp rx=0 tx=0).stop 2023-05-22T16:03:36.558 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.558+0000 7f7abad0e640 1 -- 172.21.15.16:0/2009455602 shutdown_connections 2023-05-22T16:03:36.558 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.558+0000 7f7abad0e640 1 --2- 172.21.15.16:0/2009455602 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ab40b5a50 0x7f7ab40d28d0 unknown :-1 s=CLOSED pgs=2232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:36.558 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.558+0000 7f7abad0e640 1 --2- 172.21.15.16:0/2009455602 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7ab415bca0 0x7f7ab40cda80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:36.558 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.558+0000 7f7abad0e640 1 -- 172.21.15.16:0/2009455602 >> 172.21.15.16:0/2009455602 conn(0x7f7ab40655c0 msgr2=0x7f7ab4070e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:03:36.558 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.558+0000 7f7abad0e640 1 -- 172.21.15.16:0/2009455602 shutdown_connections 2023-05-22T16:03:36.558 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.558+0000 7f7abad0e640 1 -- 172.21.15.16:0/2009455602 wait complete. 2023-05-22T16:03:36.560 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.558+0000 7f7abad0e640 1 Processor -- start 2023-05-22T16:03:36.560 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.558+0000 7f7abad0e640 1 -- start start 2023-05-22T16:03:36.560 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.558+0000 7f7abad0e640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7ab40b5a50 0x7f7ab4156790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:36.560 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.558+0000 7f7abad0e640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7ab40d2480 0x7f7ab40cef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:36.560 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.558+0000 7f7abad0e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ab415bca0 0x7f7ab40cf440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:36.561 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.558+0000 7f7abad0e640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ab4152650 con 0x7f7ab415bca0 2023-05-22T16:03:36.561 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.558+0000 7f7abad0e640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ab41521c0 con 0x7f7ab40d2480 2023-05-22T16:03:36.561 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.558+0000 7f7abad0e640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ab41554b0 con 0x7f7ab40b5a50 2023-05-22T16:03:36.561 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.558+0000 7f7ab92ab640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ab415bca0 0x7f7ab40cf440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:36.561 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.558+0000 7f7ab92ab640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ab415bca0 0x7f7ab40cf440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48134/0 (socket says 172.21.15.16:48134) 2023-05-22T16:03:36.561 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.558+0000 7f7ab92ab640 1 -- 172.21.15.16:0/2736250471 learned_addr learned my addr 172.21.15.16:0/2736250471 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:03:36.561 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.558+0000 7f7ab8aaa640 1 --2- 172.21.15.16:0/2736250471 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7ab40b5a50 0x7f7ab4156790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:36.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.558+0000 7f7ab3fff640 1 --2- 172.21.15.16:0/2736250471 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7ab40d2480 0x7f7ab40cef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:36.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.558+0000 7f7ab92ab640 1 -- 172.21.15.16:0/2736250471 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7ab40b5a50 msgr2=0x7f7ab4156790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:36.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.558+0000 7f7ab92ab640 1 --2- 172.21.15.16:0/2736250471 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7ab40b5a50 0x7f7ab4156790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:36.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.558+0000 7f7ab92ab640 1 -- 172.21.15.16:0/2736250471 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7ab40d2480 msgr2=0x7f7ab40cef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:36.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.558+0000 7f7ab92ab640 1 --2- 172.21.15.16:0/2736250471 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7ab40d2480 0x7f7ab40cef00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:36.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.558+0000 7f7ab92ab640 1 -- 172.21.15.16:0/2736250471 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7aa40094d0 con 0x7f7ab415bca0 2023-05-22T16:03:36.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.558+0000 7f7ab92ab640 1 --2- 172.21.15.16:0/2736250471 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ab415bca0 0x7f7ab40cf440 secure :-1 s=READY pgs=2233 cs=0 l=1 rev1=1 crypto rx=0x7f7aa40ac040 tx=0x7f7aa401ad10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:03:36.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.558+0000 7f7ab17fa640 1 -- 172.21.15.16:0/2736250471 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7aa40aaea0 con 0x7f7ab415bca0 2023-05-22T16:03:36.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.558+0000 7f7ab17fa640 1 -- 172.21.15.16:0/2736250471 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7aa40bd610 con 0x7f7ab415bca0 2023-05-22T16:03:36.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.558+0000 7f7abad0e640 1 -- 172.21.15.16:0/2736250471 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ab4160360 con 0x7f7ab415bca0 2023-05-22T16:03:36.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.558+0000 7f7ab17fa640 1 -- 172.21.15.16:0/2736250471 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7aa40bd9c0 con 0x7f7ab415bca0 2023-05-22T16:03:36.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.558+0000 7f7abad0e640 1 -- 172.21.15.16:0/2736250471 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ab40cdfc0 con 0x7f7ab415bca0 2023-05-22T16:03:36.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.562+0000 7f7abad0e640 1 -- 172.21.15.16:0/2736250471 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ab41604f0 con 0x7f7ab415bca0 2023-05-22T16:03:36.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.562+0000 7f7ab17fa640 1 -- 172.21.15.16:0/2736250471 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 82) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f7aa40b33f0 con 0x7f7ab415bca0 2023-05-22T16:03:36.565 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.562+0000 7f7ab17fa640 1 --2- 172.21.15.16:0/2736250471 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f7a940654a0 0x7f7a94067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:36.565 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.562+0000 7f7ab8aaa640 1 --2- 172.21.15.16:0/2736250471 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f7a940654a0 0x7f7a94067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:36.565 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.562+0000 7f7ab17fa640 1 -- 172.21.15.16:0/2736250471 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(175..175 src has 1..175) v4 ==== 5586+0+0 (secure 0 0 0) 0x7f7aa411d1f0 con 0x7f7ab415bca0 2023-05-22T16:03:36.565 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.562+0000 7f7ab8aaa640 1 --2- 172.21.15.16:0/2736250471 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f7a940654a0 0x7f7a94067950 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f7aac0070b0 tx=0x7f7aac007040 comp rx=0 tx=0).ready entity=mgr.10032 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:03:36.566 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.566+0000 7f7ab17fa640 1 -- 172.21.15.16:0/2736250471 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f7aa40eae50 con 0x7f7ab415bca0 2023-05-22T16:03:36.721 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.722+0000 7f7abad0e640 1 -- 172.21.15.16:0/2736250471 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f7ab40d2e10 con 0x7f7ab415bca0 2023-05-22T16:03:36.722 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.722+0000 7f7ab17fa640 1 -- 172.21.15.16:0/2736250471 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v175) v1 ==== 92+0+10 (secure 0 0 0) 0x7f7ab40d2e10 con 0x7f7ab415bca0 2023-05-22T16:03:36.722 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 8 2023-05-22T16:03:36.723 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.722+0000 7f7abad0e640 1 -- 172.21.15.16:0/2736250471 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f7a940654a0 msgr2=0x7f7a94067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:36.724 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.722+0000 7f7abad0e640 1 --2- 172.21.15.16:0/2736250471 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f7a940654a0 0x7f7a94067950 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f7aac0070b0 tx=0x7f7aac007040 comp rx=0 tx=0).stop 2023-05-22T16:03:36.724 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.722+0000 7f7abad0e640 1 -- 172.21.15.16:0/2736250471 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ab415bca0 msgr2=0x7f7ab40cf440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:36.724 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.722+0000 7f7abad0e640 1 --2- 172.21.15.16:0/2736250471 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ab415bca0 0x7f7ab40cf440 secure :-1 s=READY pgs=2233 cs=0 l=1 rev1=1 crypto rx=0x7f7aa40ac040 tx=0x7f7aa401ad10 comp rx=0 tx=0).stop 2023-05-22T16:03:36.724 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.722+0000 7f7abad0e640 1 -- 172.21.15.16:0/2736250471 shutdown_connections 2023-05-22T16:03:36.724 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.722+0000 7f7abad0e640 1 --2- 172.21.15.16:0/2736250471 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f7a940654a0 0x7f7a94067950 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:36.724 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.722+0000 7f7abad0e640 1 --2- 172.21.15.16:0/2736250471 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ab415bca0 0x7f7ab40cf440 unknown :-1 s=CLOSED pgs=2233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:36.725 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.722+0000 7f7abad0e640 1 --2- 172.21.15.16:0/2736250471 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7ab40d2480 0x7f7ab40cef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:36.725 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.722+0000 7f7abad0e640 1 --2- 172.21.15.16:0/2736250471 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7ab40b5a50 0x7f7ab4156790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:36.725 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.722+0000 7f7abad0e640 1 -- 172.21.15.16:0/2736250471 >> 172.21.15.16:0/2736250471 conn(0x7f7ab40655c0 msgr2=0x7f7ab40b7730 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:03:36.725 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.722+0000 7f7abad0e640 1 -- 172.21.15.16:0/2736250471 shutdown_connections 2023-05-22T16:03:36.725 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.722+0000 7f7abad0e640 1 -- 172.21.15.16:0/2736250471 wait complete. 2023-05-22T16:03:36.741 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-05-22T16:03:36.867 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.866+0000 7f23b4c74640 1 Processor -- start 2023-05-22T16:03:36.867 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.866+0000 7f23b4c74640 1 -- start start 2023-05-22T16:03:36.869 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.866+0000 7f23b4c74640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f23b00ca4e0 0x7f23b015e460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:36.869 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.866+0000 7f23b4c74640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f23b00c9b40 0x7f23b00afb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:36.869 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.866+0000 7f23b4c74640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23b00b0650 con 0x7f23b0070df0 2023-05-22T16:03:36.869 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.866+0000 7f23b4c74640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23b00b0370 con 0x7f23b00c9b40 2023-05-22T16:03:36.870 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.866+0000 7f23b4c74640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23b015d980 con 0x7f23b00ca4e0 2023-05-22T16:03:36.870 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.866+0000 7f23aed9d640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f23b0070df0 0x7f23b0071210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:48192/0 (socket says 172.21.15.16:48192) 2023-05-22T16:03:36.870 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.866+0000 7f23aed9d640 1 -- 172.21.15.16:0/156369708 learned_addr learned my addr 172.21.15.16:0/156369708 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T16:03:36.870 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.866+0000 7f23add9b640 1 --2- 172.21.15.16:0/156369708 >> v2:172.21.15.16:3300/0 conn(0x7f23b00c9b40 0x7f23b00afb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:36.870 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.870+0000 7f23ae59c640 1 --2- 172.21.15.16:0/156369708 >> v2:172.21.15.62:3300/0 conn(0x7f23b00ca4e0 0x7f23b015e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:36.870 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.870+0000 7f23ad59a640 1 -- 172.21.15.16:0/156369708 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3252932818 0 0) 0x7f23b00b0650 con 0x7f23b0070df0 2023-05-22T16:03:36.870 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.870+0000 7f23ad59a640 1 -- 172.21.15.16:0/156369708 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f239c003530 con 0x7f23b0070df0 2023-05-22T16:03:36.870 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.870+0000 7f23add9b640 1 -- 172.21.15.16:0/156369708 >> v2:172.21.15.62:3300/0 conn(0x7f23b00ca4e0 msgr2=0x7f23b015e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:03:36.871 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.870+0000 7f23add9b640 1 --2- 172.21.15.16:0/156369708 >> v2:172.21.15.62:3300/0 conn(0x7f23b00ca4e0 0x7f23b015e460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:36.871 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.870+0000 7f23add9b640 1 -- 172.21.15.16:0/156369708 >> v1:172.21.15.16:6789/0 conn(0x7f23b0070df0 legacy=0x7f23b0071210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:36.871 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.870+0000 7f23add9b640 1 -- 172.21.15.16:0/156369708 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23b015dc60 con 0x7f23b00c9b40 2023-05-22T16:03:36.871 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.870+0000 7f23ae59c640 1 --2- 172.21.15.16:0/156369708 >> v2:172.21.15.62:3300/0 conn(0x7f23b00ca4e0 0x7f23b015e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T16:03:36.872 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.870+0000 7f23add9b640 1 --2- 172.21.15.16:0/156369708 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f23b00c9b40 0x7f23b00afb40 secure :-1 s=READY pgs=2235 cs=0 l=1 rev1=1 crypto rx=0x7f23b00cad40 tx=0x7f239801c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce358e510353011 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:03:36.872 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.870+0000 7f23ad59a640 1 -- 172.21.15.16:0/156369708 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23980a5980 con 0x7f23b00c9b40 2023-05-22T16:03:36.872 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.870+0000 7f23ad59a640 1 -- 172.21.15.16:0/156369708 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23980a5ae0 con 0x7f23b00c9b40 2023-05-22T16:03:36.873 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.870+0000 7f23ad59a640 1 -- 172.21.15.16:0/156369708 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23980a5e70 con 0x7f23b00c9b40 2023-05-22T16:03:36.873 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.870+0000 7f23b4c74640 1 -- 172.21.15.16:0/156369708 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f23b00c9b40 msgr2=0x7f23b00afb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:36.873 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.870+0000 7f23b4c74640 1 --2- 172.21.15.16:0/156369708 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f23b00c9b40 0x7f23b00afb40 secure :-1 s=READY pgs=2235 cs=0 l=1 rev1=1 crypto rx=0x7f23b00cad40 tx=0x7f239801c8e0 comp rx=0 tx=0).stop 2023-05-22T16:03:36.873 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.870+0000 7f23b4c74640 1 -- 172.21.15.16:0/156369708 shutdown_connections 2023-05-22T16:03:36.873 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.870+0000 7f23b4c74640 1 --2- 172.21.15.16:0/156369708 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f23b00c9b40 0x7f23b00afb40 unknown :-1 s=CLOSED pgs=2235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:36.873 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.870+0000 7f23b4c74640 1 --2- 172.21.15.16:0/156369708 >> v2:172.21.15.62:3300/0 conn(0x7f23b00ca4e0 0x7f23b015e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:36.873 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.870+0000 7f23b4c74640 1 -- 172.21.15.16:0/156369708 >> 172.21.15.16:0/156369708 conn(0x7f23b0069720 msgr2=0x7f23b0069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:03:36.874 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.870+0000 7f23b4c74640 1 -- 172.21.15.16:0/156369708 shutdown_connections 2023-05-22T16:03:36.874 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.870+0000 7f23b4c74640 1 -- 172.21.15.16:0/156369708 wait complete. 2023-05-22T16:03:36.876 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.874+0000 7f23b4c74640 1 Processor -- start 2023-05-22T16:03:36.876 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.874+0000 7f23b4c74640 1 -- start start 2023-05-22T16:03:36.876 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.874+0000 7f23b4c74640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f23b0070df0 0x7f23b015cf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:36.876 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.874+0000 7f23b4c74640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f23b00c9b40 0x7f23b00c7680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:36.876 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.874+0000 7f23b4c74640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f23b00ca4e0 0x7f23b014d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:36.876 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.874+0000 7f23b4c74640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23b00c8450 con 0x7f23b00ca4e0 2023-05-22T16:03:36.877 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.874+0000 7f23b4c74640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23b015a9f0 con 0x7f23b0070df0 2023-05-22T16:03:36.877 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.874+0000 7f23b4c74640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23b00c7bc0 con 0x7f23b00c9b40 2023-05-22T16:03:36.877 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.874+0000 7f23aed9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f23b00ca4e0 0x7f23b014d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:36.877 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.874+0000 7f23aed9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f23b00ca4e0 0x7f23b014d300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48162/0 (socket says 172.21.15.16:48162) 2023-05-22T16:03:36.877 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.874+0000 7f23aed9d640 1 -- 172.21.15.16:0/2182721184 learned_addr learned my addr 172.21.15.16:0/2182721184 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:03:36.877 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.874+0000 7f23ae59c640 1 --2- 172.21.15.16:0/2182721184 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f23b0070df0 0x7f23b015cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:36.877 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.874+0000 7f23add9b640 1 --2- 172.21.15.16:0/2182721184 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f23b00c9b40 0x7f23b00c7680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:36.878 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.874+0000 7f23aed9d640 1 -- 172.21.15.16:0/2182721184 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f23b00c9b40 msgr2=0x7f23b00c7680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:36.878 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.874+0000 7f23aed9d640 1 --2- 172.21.15.16:0/2182721184 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f23b00c9b40 0x7f23b00c7680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:36.878 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.874+0000 7f23aed9d640 1 -- 172.21.15.16:0/2182721184 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f23b0070df0 msgr2=0x7f23b015cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:36.878 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.874+0000 7f23aed9d640 1 --2- 172.21.15.16:0/2182721184 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f23b0070df0 0x7f23b015cf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:36.878 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.874+0000 7f23aed9d640 1 -- 172.21.15.16:0/2182721184 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2398009a70 con 0x7f23b00ca4e0 2023-05-22T16:03:36.878 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.874+0000 7f23aed9d640 1 --2- 172.21.15.16:0/2182721184 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f23b00ca4e0 0x7f23b014d300 secure :-1 s=READY pgs=2236 cs=0 l=1 rev1=1 crypto rx=0x7f23a800c450 tx=0x7f23a800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:03:36.878 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.874+0000 7f23a6ffd640 1 -- 172.21.15.16:0/2182721184 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23a8005cf0 con 0x7f23b00ca4e0 2023-05-22T16:03:36.878 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.874+0000 7f23a6ffd640 1 -- 172.21.15.16:0/2182721184 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23a8005e50 con 0x7f23b00ca4e0 2023-05-22T16:03:36.879 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.874+0000 7f23b4c74640 1 -- 172.21.15.16:0/2182721184 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23b00c8e80 con 0x7f23b00ca4e0 2023-05-22T16:03:36.879 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.874+0000 7f23a6ffd640 1 -- 172.21.15.16:0/2182721184 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23a80072b0 con 0x7f23b00ca4e0 2023-05-22T16:03:36.879 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.874+0000 7f23b4c74640 1 -- 172.21.15.16:0/2182721184 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23b00ca8b0 con 0x7f23b00ca4e0 2023-05-22T16:03:36.880 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.878+0000 7f23b4c74640 1 -- 172.21.15.16:0/2182721184 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23b00c9010 con 0x7f23b00ca4e0 2023-05-22T16:03:36.880 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.878+0000 7f23a6ffd640 1 -- 172.21.15.16:0/2182721184 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 82) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f23a8007410 con 0x7f23b00ca4e0 2023-05-22T16:03:36.880 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.878+0000 7f23a6ffd640 1 --2- 172.21.15.16:0/2182721184 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f23900654a0 0x7f2390067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:36.880 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.878+0000 7f23a6ffd640 1 -- 172.21.15.16:0/2182721184 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(175..175 src has 1..175) v4 ==== 5586+0+0 (secure 0 0 0) 0x7f23a8081ce0 con 0x7f23b00ca4e0 2023-05-22T16:03:36.880 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.878+0000 7f23ae59c640 1 --2- 172.21.15.16:0/2182721184 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f23900654a0 0x7f2390067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:36.880 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.878+0000 7f23ae59c640 1 --2- 172.21.15.16:0/2182721184 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f23900654a0 0x7f2390067950 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f23a000ba10 tx=0x7f23a0008040 comp rx=0 tx=0).ready entity=mgr.10032 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:03:36.881 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:36.882+0000 7f23a6ffd640 1 -- 172.21.15.16:0/2182721184 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f23a804fa20 con 0x7f23b00ca4e0 2023-05-22T16:03:37.031 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:37.030+0000 7f23b4c74640 1 -- 172.21.15.16:0/2182721184 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f23b004dbe0 con 0x7f23b00ca4e0 2023-05-22T16:03:37.031 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:37.030+0000 7f23a6ffd640 1 -- 172.21.15.16:0/2182721184 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v175) v1 ==== 84+0+126 (secure 0 0 0) 0x7f23a8017090 con 0x7f23b00ca4e0 2023-05-22T16:03:37.032 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T16:03:37.032 INFO:teuthology.orchestra.run.smithi016.stdout:[ 2023-05-22T16:03:37.032 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T16:03:37.032 INFO:teuthology.orchestra.run.smithi016.stdout: "poolnum": 1, 2023-05-22T16:03:37.032 INFO:teuthology.orchestra.run.smithi016.stdout: "poolname": ".mgr" 2023-05-22T16:03:37.033 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T16:03:37.033 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T16:03:37.033 INFO:teuthology.orchestra.run.smithi016.stdout: "poolnum": 2, 2023-05-22T16:03:37.033 INFO:teuthology.orchestra.run.smithi016.stdout: "poolname": "rbd" 2023-05-22T16:03:37.033 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T16:03:37.033 INFO:teuthology.orchestra.run.smithi016.stdout:] 2023-05-22T16:03:37.034 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:37.034+0000 7f23b4c74640 1 -- 172.21.15.16:0/2182721184 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f23900654a0 msgr2=0x7f2390067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:37.034 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:37.034+0000 7f23b4c74640 1 --2- 172.21.15.16:0/2182721184 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f23900654a0 0x7f2390067950 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f23a000ba10 tx=0x7f23a0008040 comp rx=0 tx=0).stop 2023-05-22T16:03:37.034 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:37.034+0000 7f23b4c74640 1 -- 172.21.15.16:0/2182721184 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f23b00ca4e0 msgr2=0x7f23b014d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:37.034 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:37.034+0000 7f23b4c74640 1 --2- 172.21.15.16:0/2182721184 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f23b00ca4e0 0x7f23b014d300 secure :-1 s=READY pgs=2236 cs=0 l=1 rev1=1 crypto rx=0x7f23a800c450 tx=0x7f23a800c910 comp rx=0 tx=0).stop 2023-05-22T16:03:37.035 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:37.034+0000 7f23b4c74640 1 -- 172.21.15.16:0/2182721184 shutdown_connections 2023-05-22T16:03:37.035 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:37.034+0000 7f23b4c74640 1 --2- 172.21.15.16:0/2182721184 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f23900654a0 0x7f2390067950 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:37.035 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:37.034+0000 7f23b4c74640 1 --2- 172.21.15.16:0/2182721184 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f23b00ca4e0 0x7f23b014d300 unknown :-1 s=CLOSED pgs=2236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:37.035 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:37.034+0000 7f23b4c74640 1 --2- 172.21.15.16:0/2182721184 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f23b00c9b40 0x7f23b00c7680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:37.035 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:37.034+0000 7f23b4c74640 1 --2- 172.21.15.16:0/2182721184 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f23b0070df0 0x7f23b015cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:37.035 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:37.034+0000 7f23b4c74640 1 -- 172.21.15.16:0/2182721184 >> 172.21.15.16:0/2182721184 conn(0x7f23b0069720 msgr2=0x7f23b00af3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:03:37.035 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:37.034+0000 7f23b4c74640 1 -- 172.21.15.16:0/2182721184 shutdown_connections 2023-05-22T16:03:37.036 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:37.034+0000 7f23b4c74640 1 -- 172.21.15.16:0/2182721184 wait complete. 2023-05-22T16:03:37.049 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-05-22T16:03:37.050 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-05-22T16:03:37.166 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:37.166+0000 7f1c55e7d640 1 Processor -- start 2023-05-22T16:03:37.166 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:37.166+0000 7f1c55e7d640 1 -- start start 2023-05-22T16:03:37.168 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:37.166+0000 7f1c55e7d640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f1c50070c10 0x7f1c500cc4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:37.168 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:37.166+0000 7f1c55e7d640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1c5015d650 0x7f1c5015dab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:37.168 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:37.166+0000 7f1c55e7d640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c5015dff0 con 0x7f1c500c9e40 2023-05-22T16:03:37.169 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:37.166+0000 7f1c55e7d640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c500ccbb0 con 0x7f1c50070c10 2023-05-22T16:03:37.169 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:37.166+0000 7f1c55e7d640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c500cd7c0 con 0x7f1c5015d650 2023-05-22T16:03:37.169 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:37.166+0000 7f1c4ffff640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1c5015d650 0x7f1c5015dab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:37.169 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:37.166+0000 7f1c4f7fe640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f1c50070c10 0x7f1c500cc4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:37.169 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:37.166+0000 7f1c46ffd640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f1c500c9e40 0x7f1c5006bfa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.16:49034/0 (socket says 172.21.15.16:49034) 2023-05-22T16:03:37.170 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:37.166+0000 7f1c46ffd640 1 -- 172.21.15.16:0/1288327864 learned_addr learned my addr 172.21.15.16:0/1288327864 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T16:03:37.170 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:37.166+0000 7f1c4f7fe640 1 -- 172.21.15.16:0/1288327864 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1c5015d650 msgr2=0x7f1c5015dab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:03:37.170 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:37.166+0000 7f1c4f7fe640 1 --2- 172.21.15.16:0/1288327864 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1c5015d650 0x7f1c5015dab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:37.170 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:37.166+0000 7f1c4f7fe640 1 -- 172.21.15.16:0/1288327864 >> v1:172.21.15.62:6789/0 conn(0x7f1c500c9e40 legacy=0x7f1c5006bfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:37.171 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:37.166+0000 7f1c4f7fe640 1 -- 172.21.15.16:0/1288327864 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c500cdaa0 con 0x7f1c50070c10 2023-05-22T16:03:37.171 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:37.166+0000 7f1c4ffff640 1 --2- 172.21.15.16:0/1288327864 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1c5015d650 0x7f1c5015dab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-05-22T16:03:37.173 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:37.170+0000 7f1c4f7fe640 1 --2- 172.21.15.16:0/1288327864 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1c50070c10 0x7f1c500cc4e0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f1c400097c0 tx=0x7f1c4001c450 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5d03a45d402ac189 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:03:37.173 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:37.170+0000 7f1c4effd640 1 -- 172.21.15.16:0/1288327864 <== mon.1 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c400ab670 con 0x7f1c50070c10 2023-05-22T16:03:37.173 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:37.170+0000 7f1c4effd640 1 -- 172.21.15.16:0/1288327864 <== mon.1 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c400ab7d0 con 0x7f1c50070c10 2023-05-22T16:03:37.174 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:37.170+0000 7f1c4effd640 1 -- 172.21.15.16:0/1288327864 <== mon.1 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c400abb60 con 0x7f1c50070c10 2023-05-22T16:03:37.174 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:37.170+0000 7f1c55e7d640 1 -- 172.21.15.16:0/1288327864 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1c50070c10 msgr2=0x7f1c500cc4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:37.174 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:37.170+0000 7f1c55e7d640 1 --2- 172.21.15.16:0/1288327864 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1c50070c10 0x7f1c500cc4e0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f1c400097c0 tx=0x7f1c4001c450 comp rx=0 tx=0).stop 2023-05-22T16:03:37.174 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:37.170+0000 7f1c55e7d640 1 -- 172.21.15.16:0/1288327864 shutdown_connections 2023-05-22T16:03:37.175 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:37.170+0000 7f1c55e7d640 1 --2- 172.21.15.16:0/1288327864 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1c5015d650 0x7f1c5015dab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:37.175 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:37.170+0000 7f1c55e7d640 1 --2- 172.21.15.16:0/1288327864 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1c50070c10 0x7f1c500cc4e0 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:37.175 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:37.170+0000 7f1c55e7d640 1 -- 172.21.15.16:0/1288327864 >> 172.21.15.16:0/1288327864 conn(0x7f1c50069720 msgr2=0x7f1c50069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:03:37.175 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:37.170+0000 7f1c55e7d640 1 -- 172.21.15.16:0/1288327864 shutdown_connections 2023-05-22T16:03:37.176 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:37.170+0000 7f1c55e7d640 1 -- 172.21.15.16:0/1288327864 wait complete. 2023-05-22T16:03:37.176 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:37.174+0000 7f1c55e7d640 1 Processor -- start 2023-05-22T16:03:37.176 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:37.174+0000 7f1c55e7d640 1 -- start start 2023-05-22T16:03:37.180 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:37.174+0000 7f1c55e7d640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1c50070c10 0x7f1c501591a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:37.181 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:37.174+0000 7f1c55e7d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1c500c9e40 0x7f1c500c78b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:37.181 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:37.174+0000 7f1c55e7d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c5015d650 0x7f1c5014e870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:37.181 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:37.174+0000 7f1c55e7d640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c501596e0 con 0x7f1c5015d650 2023-05-22T16:03:37.181 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:37.174+0000 7f1c55e7d640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c50159ba0 con 0x7f1c500c9e40 2023-05-22T16:03:37.182 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:37.174+0000 7f1c55e7d640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c50158730 con 0x7f1c50070c10 2023-05-22T16:03:37.182 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:37.174+0000 7f1c4ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c5015d650 0x7f1c5014e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:37.182 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:37.174+0000 7f1c4ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c5015d650 0x7f1c5014e870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48166/0 (socket says 172.21.15.16:48166) 2023-05-22T16:03:37.182 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:37.174+0000 7f1c4ffff640 1 -- 172.21.15.16:0/4279703633 learned_addr learned my addr 172.21.15.16:0/4279703633 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:03:37.182 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:37.174+0000 7f1c4f7fe640 1 --2- 172.21.15.16:0/4279703633 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1c50070c10 0x7f1c501591a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:37.182 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:37.174+0000 7f1c46ffd640 1 --2- 172.21.15.16:0/4279703633 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1c500c9e40 0x7f1c500c78b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:37.182 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:37.174+0000 7f1c4ffff640 1 -- 172.21.15.16:0/4279703633 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1c50070c10 msgr2=0x7f1c501591a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:37.183 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:37.174+0000 7f1c4ffff640 1 --2- 172.21.15.16:0/4279703633 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1c50070c10 0x7f1c501591a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:37.183 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:37.174+0000 7f1c4ffff640 1 -- 172.21.15.16:0/4279703633 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1c500c9e40 msgr2=0x7f1c500c78b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:37.183 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:37.174+0000 7f1c4ffff640 1 --2- 172.21.15.16:0/4279703633 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1c500c9e40 0x7f1c500c78b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:37.183 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:37.174+0000 7f1c4ffff640 1 -- 172.21.15.16:0/4279703633 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c48009a70 con 0x7f1c5015d650 2023-05-22T16:03:37.183 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:37.174+0000 7f1c4f7fe640 1 --2- 172.21.15.16:0/4279703633 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1c50070c10 0x7f1c501591a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T16:03:37.183 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:37.174+0000 7f1c46ffd640 1 --2- 172.21.15.16:0/4279703633 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1c500c9e40 0x7f1c500c78b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T16:03:37.183 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:37.174+0000 7f1c4ffff640 1 --2- 172.21.15.16:0/4279703633 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c5015d650 0x7f1c5014e870 secure :-1 s=READY pgs=2237 cs=0 l=1 rev1=1 crypto rx=0x7f1c48009f00 tx=0x7f1c4800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:03:37.184 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:37.174+0000 7f1c4cff9640 1 -- 172.21.15.16:0/4279703633 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c48009d70 con 0x7f1c5015d650 2023-05-22T16:03:37.184 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:37.174+0000 7f1c4cff9640 1 -- 172.21.15.16:0/4279703633 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c4800f040 con 0x7f1c5015d650 2023-05-22T16:03:37.184 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:37.174+0000 7f1c4cff9640 1 -- 172.21.15.16:0/4279703633 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c48003bc0 con 0x7f1c5015d650 2023-05-22T16:03:37.184 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:37.174+0000 7f1c55e7d640 1 -- 172.21.15.16:0/4279703633 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c40009420 con 0x7f1c5015d650 2023-05-22T16:03:37.184 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:37.174+0000 7f1c55e7d640 1 -- 172.21.15.16:0/4279703633 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c500ca240 con 0x7f1c5015d650 2023-05-22T16:03:37.184 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:37.174+0000 7f1c55e7d640 1 -- 172.21.15.16:0/4279703633 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c500cce90 con 0x7f1c5015d650 2023-05-22T16:03:37.184 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:37.174+0000 7f1c4cff9640 1 -- 172.21.15.16:0/4279703633 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 82) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f1c48014480 con 0x7f1c5015d650 2023-05-22T16:03:37.185 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:37.178+0000 7f1c4cff9640 1 --2- 172.21.15.16:0/4279703633 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f1c300654d0 0x7f1c30067980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:37.185 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:37.178+0000 7f1c4f7fe640 1 --2- 172.21.15.16:0/4279703633 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f1c300654d0 0x7f1c30067980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:37.185 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:37.178+0000 7f1c4cff9640 1 -- 172.21.15.16:0/4279703633 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(175..175 src has 1..175) v4 ==== 5586+0+0 (secure 0 0 0) 0x7f1c480834f0 con 0x7f1c5015d650 2023-05-22T16:03:37.185 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:37.178+0000 7f1c4f7fe640 1 --2- 172.21.15.16:0/4279703633 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f1c300654d0 0x7f1c30067980 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f1c400af000 tx=0x7f1c40008040 comp rx=0 tx=0).ready entity=mgr.10032 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:03:37.185 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:37.182+0000 7f1c4cff9640 1 -- 172.21.15.16:0/4279703633 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f1c48054be0 con 0x7f1c5015d650 2023-05-22T16:03:37.332 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:37.334+0000 7f1c55e7d640 1 -- 172.21.15.16:0/4279703633 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f1c500ca3d0 con 0x7f1c5015d650 2023-05-22T16:03:37.392 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:37.390+0000 7f1c4cff9640 1 -- 172.21.15.16:0/4279703633 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v176) v1 ==== 138+0+0 (secure 0 0 0) 0x7f1c48054770 con 0x7f1c5015d650 2023-05-22T16:03:37.392 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'cephfs_metadata' created 2023-05-22T16:03:37.395 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:37.394+0000 7f1c55e7d640 1 -- 172.21.15.16:0/4279703633 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f1c300654d0 msgr2=0x7f1c30067980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:37.396 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:37.394+0000 7f1c55e7d640 1 --2- 172.21.15.16:0/4279703633 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f1c300654d0 0x7f1c30067980 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f1c400af000 tx=0x7f1c40008040 comp rx=0 tx=0).stop 2023-05-22T16:03:37.396 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:37.394+0000 7f1c55e7d640 1 -- 172.21.15.16:0/4279703633 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c5015d650 msgr2=0x7f1c5014e870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:37.397 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:37.394+0000 7f1c55e7d640 1 --2- 172.21.15.16:0/4279703633 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c5015d650 0x7f1c5014e870 secure :-1 s=READY pgs=2237 cs=0 l=1 rev1=1 crypto rx=0x7f1c48009f00 tx=0x7f1c4800c490 comp rx=0 tx=0).stop 2023-05-22T16:03:37.397 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:37.394+0000 7f1c55e7d640 1 -- 172.21.15.16:0/4279703633 shutdown_connections 2023-05-22T16:03:37.397 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:37.394+0000 7f1c55e7d640 1 --2- 172.21.15.16:0/4279703633 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f1c300654d0 0x7f1c30067980 secure :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f1c400af000 tx=0x7f1c40008040 comp rx=0 tx=0).stop 2023-05-22T16:03:37.397 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:37.394+0000 7f1c55e7d640 1 --2- 172.21.15.16:0/4279703633 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c5015d650 0x7f1c5014e870 unknown :-1 s=CLOSED pgs=2237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:37.398 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:37.394+0000 7f1c55e7d640 1 --2- 172.21.15.16:0/4279703633 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1c500c9e40 0x7f1c500c78b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:37.398 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:37.394+0000 7f1c55e7d640 1 --2- 172.21.15.16:0/4279703633 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1c50070c10 0x7f1c501591a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:37.398 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:37.394+0000 7f1c55e7d640 1 -- 172.21.15.16:0/4279703633 >> 172.21.15.16:0/4279703633 conn(0x7f1c50069720 msgr2=0x7f1c5006bfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:03:37.398 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:37.394+0000 7f1c55e7d640 1 -- 172.21.15.16:0/4279703633 shutdown_connections 2023-05-22T16:03:37.398 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:37.394+0000 7f1c55e7d640 1 -- 172.21.15.16:0/4279703633 wait complete. 2023-05-22T16:03:37.416 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-05-22T16:03:37.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:37.542+0000 7f9a04acc640 1 Processor -- start 2023-05-22T16:03:37.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:37.542+0000 7f9a04acc640 1 -- start start 2023-05-22T16:03:37.544 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:37.542+0000 7f9a04acc640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f9a0006b6c0 0x7f9a000cb4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:37.544 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:37.542+0000 7f9a04acc640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9a0006ba90 0x7f9a00158fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:37.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:37.542+0000 7f9a04acc640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a00159f70 con 0x7f9a00159510 2023-05-22T16:03:37.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:37.542+0000 7f9a04acc640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a00071760 con 0x7f9a0006b6c0 2023-05-22T16:03:37.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:37.542+0000 7f9a04acc640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a000ca7d0 con 0x7f9a0006ba90 2023-05-22T16:03:37.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:37.542+0000 7f99fe59c640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f9a0006b6c0 0x7f9a000cb4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:37.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:37.542+0000 7f99fed9d640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f9a00159510 0x7f9a000cc740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:48198/0 (socket says 172.21.15.16:48198) 2023-05-22T16:03:37.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:37.542+0000 7f99fed9d640 1 -- 172.21.15.16:0/2359071877 learned_addr learned my addr 172.21.15.16:0/2359071877 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T16:03:37.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:37.542+0000 7f99fdd9b640 1 --2- 172.21.15.16:0/2359071877 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9a0006ba90 0x7f9a00158fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:37.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:37.546+0000 7f99fe59c640 1 -- 172.21.15.16:0/2359071877 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9a0006ba90 msgr2=0x7f9a00158fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:03:37.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:37.546+0000 7f99fe59c640 1 --2- 172.21.15.16:0/2359071877 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9a0006ba90 0x7f9a00158fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:37.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:37.546+0000 7f99fe59c640 1 -- 172.21.15.16:0/2359071877 >> v1:172.21.15.16:6789/0 conn(0x7f9a00159510 legacy=0x7f9a000cc740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:37.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:37.546+0000 7f99fe59c640 1 -- 172.21.15.16:0/2359071877 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a000cb030 con 0x7f9a0006b6c0 2023-05-22T16:03:37.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:37.546+0000 7f99fd59a640 1 -- 172.21.15.16:0/2359071877 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 402247177 0 0) 0x7f9a00159f70 con 0x7f9a00159510 2023-05-22T16:03:37.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:37.546+0000 7f99fe59c640 1 --2- 172.21.15.16:0/2359071877 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9a0006b6c0 0x7f9a000cb4c0 secure :-1 s=READY pgs=2239 cs=0 l=1 rev1=1 crypto rx=0x7f99f00097c0 tx=0x7f99f001c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=210d93b6008379d0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:03:37.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:37.546+0000 7f99fd59a640 1 -- 172.21.15.16:0/2359071877 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99f00ab600 con 0x7f9a0006b6c0 2023-05-22T16:03:37.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:37.546+0000 7f99fd59a640 1 -- 172.21.15.16:0/2359071877 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99f00a7050 con 0x7f9a0006b6c0 2023-05-22T16:03:37.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:37.546+0000 7f99fd59a640 1 -- 172.21.15.16:0/2359071877 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99f00ab980 con 0x7f9a0006b6c0 2023-05-22T16:03:37.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:37.546+0000 7f9a04acc640 1 -- 172.21.15.16:0/2359071877 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9a0006b6c0 msgr2=0x7f9a000cb4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:37.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:37.546+0000 7f9a04acc640 1 --2- 172.21.15.16:0/2359071877 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9a0006b6c0 0x7f9a000cb4c0 secure :-1 s=READY pgs=2239 cs=0 l=1 rev1=1 crypto rx=0x7f99f00097c0 tx=0x7f99f001c450 comp rx=0 tx=0).stop 2023-05-22T16:03:37.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:37.546+0000 7f9a04acc640 1 -- 172.21.15.16:0/2359071877 shutdown_connections 2023-05-22T16:03:37.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:37.546+0000 7f9a04acc640 1 --2- 172.21.15.16:0/2359071877 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9a0006ba90 0x7f9a00158fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:37.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:37.546+0000 7f9a04acc640 1 --2- 172.21.15.16:0/2359071877 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9a0006b6c0 0x7f9a000cb4c0 unknown :-1 s=CLOSED pgs=2239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:37.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:37.546+0000 7f9a04acc640 1 -- 172.21.15.16:0/2359071877 >> 172.21.15.16:0/2359071877 conn(0x7f9a00069720 msgr2=0x7f9a00069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:03:37.549 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:37.546+0000 7f9a04acc640 1 -- 172.21.15.16:0/2359071877 shutdown_connections 2023-05-22T16:03:37.549 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:37.546+0000 7f9a04acc640 1 -- 172.21.15.16:0/2359071877 wait complete. 2023-05-22T16:03:37.551 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:37.550+0000 7f9a04acc640 1 Processor -- start 2023-05-22T16:03:37.551 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:37.550+0000 7f9a04acc640 1 -- start start 2023-05-22T16:03:37.551 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:37.550+0000 7f9a04acc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9a0006b6c0 0x7f9a001f5220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:37.551 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:37.550+0000 7f9a04acc640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9a0006ba90 0x7f9a001f5760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:37.551 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:37.550+0000 7f9a04acc640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9a00159510 0x7f9a001514b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:37.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:37.550+0000 7f9a04acc640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a00150390 con 0x7f9a0006b6c0 2023-05-22T16:03:37.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:37.550+0000 7f9a04acc640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a00150670 con 0x7f9a00159510 2023-05-22T16:03:37.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:37.550+0000 7f9a04acc640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a00150950 con 0x7f9a0006ba90 2023-05-22T16:03:37.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:37.550+0000 7f99fe59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9a0006b6c0 0x7f9a001f5220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:37.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:37.550+0000 7f99fe59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9a0006b6c0 0x7f9a001f5220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48186/0 (socket says 172.21.15.16:48186) 2023-05-22T16:03:37.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:37.550+0000 7f99fe59c640 1 -- 172.21.15.16:0/2226210417 learned_addr learned my addr 172.21.15.16:0/2226210417 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:03:37.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:37.550+0000 7f99fdd9b640 1 --2- 172.21.15.16:0/2226210417 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9a0006ba90 0x7f9a001f5760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:37.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:37.550+0000 7f99fed9d640 1 --2- 172.21.15.16:0/2226210417 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9a00159510 0x7f9a001514b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:37.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:37.550+0000 7f99fe59c640 1 -- 172.21.15.16:0/2226210417 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9a0006ba90 msgr2=0x7f9a001f5760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:37.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:37.550+0000 7f99fe59c640 1 --2- 172.21.15.16:0/2226210417 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9a0006ba90 0x7f9a001f5760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:37.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:37.550+0000 7f99fe59c640 1 -- 172.21.15.16:0/2226210417 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9a00159510 msgr2=0x7f9a001514b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:37.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:37.550+0000 7f99fe59c640 1 --2- 172.21.15.16:0/2226210417 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9a00159510 0x7f9a001514b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:37.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:37.550+0000 7f99fe59c640 1 -- 172.21.15.16:0/2226210417 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99f0009420 con 0x7f9a0006b6c0 2023-05-22T16:03:37.554 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:37.550+0000 7f99fed9d640 1 --2- 172.21.15.16:0/2226210417 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9a00159510 0x7f9a001514b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T16:03:37.554 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:37.550+0000 7f99fdd9b640 1 --2- 172.21.15.16:0/2226210417 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9a0006ba90 0x7f9a001f5760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T16:03:37.554 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:37.550+0000 7f99fe59c640 1 --2- 172.21.15.16:0/2226210417 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9a0006b6c0 0x7f9a001f5220 secure :-1 s=READY pgs=2240 cs=0 l=1 rev1=1 crypto rx=0x7f99f0009710 tx=0x7f99f001ac90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:03:37.554 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:37.550+0000 7f99eeffd640 1 -- 172.21.15.16:0/2226210417 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99f0004230 con 0x7f9a0006b6c0 2023-05-22T16:03:37.554 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:37.550+0000 7f99eeffd640 1 -- 172.21.15.16:0/2226210417 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99f0004390 con 0x7f9a0006b6c0 2023-05-22T16:03:37.554 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:37.550+0000 7f9a04acc640 1 -- 172.21.15.16:0/2226210417 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a00159950 con 0x7f9a0006b6c0 2023-05-22T16:03:37.554 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:37.550+0000 7f99eeffd640 1 -- 172.21.15.16:0/2226210417 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99f0003820 con 0x7f9a0006b6c0 2023-05-22T16:03:37.554 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:37.550+0000 7f9a04acc640 1 -- 172.21.15.16:0/2226210417 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a000cacc0 con 0x7f9a0006b6c0 2023-05-22T16:03:37.555 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:37.550+0000 7f9a04acc640 1 -- 172.21.15.16:0/2226210417 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a0006be60 con 0x7f9a0006b6c0 2023-05-22T16:03:37.555 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:37.550+0000 7f99eeffd640 1 -- 172.21.15.16:0/2226210417 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 82) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f99f0003980 con 0x7f9a0006b6c0 2023-05-22T16:03:37.555 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:37.554+0000 7f99eeffd640 1 --2- 172.21.15.16:0/2226210417 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f99e00654a0 0x7f99e0067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:37.556 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:37.554+0000 7f99eeffd640 1 -- 172.21.15.16:0/2226210417 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(176..176 src has 1..176) v4 ==== 5956+0+0 (secure 0 0 0) 0x7f99f011d810 con 0x7f9a0006b6c0 2023-05-22T16:03:37.556 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:37.554+0000 7f99fdd9b640 1 --2- 172.21.15.16:0/2226210417 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f99e00654a0 0x7f99e0067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:37.556 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:37.554+0000 7f99fdd9b640 1 --2- 172.21.15.16:0/2226210417 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f99e00654a0 0x7f99e0067950 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f9a0015b620 tx=0x7f99f4008040 comp rx=0 tx=0).ready entity=mgr.10032 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:03:37.557 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:37.558+0000 7f99eeffd640 1 -- 172.21.15.16:0/2226210417 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f99f00eb2b0 con 0x7f9a0006b6c0 2023-05-22T16:03:37.711 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:37.710+0000 7f9a04acc640 1 -- 172.21.15.16:0/2226210417 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f9a0004dbe0 con 0x7f9a0006b6c0 2023-05-22T16:03:38.400 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:38.398+0000 7f99eeffd640 1 -- 172.21.15.16:0/2226210417 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v177) v1 ==== 170+0+0 (secure 0 0 0) 0x7f99f00ba0d0 con 0x7f9a0006b6c0 2023-05-22T16:03:38.400 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'cephfs_data' created 2023-05-22T16:03:38.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:38.406+0000 7f9a04acc640 1 -- 172.21.15.16:0/2226210417 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f99e00654a0 msgr2=0x7f99e0067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:38.408 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:38.406+0000 7f9a04acc640 1 --2- 172.21.15.16:0/2226210417 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f99e00654a0 0x7f99e0067950 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f9a0015b620 tx=0x7f99f4008040 comp rx=0 tx=0).stop 2023-05-22T16:03:38.408 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:38.406+0000 7f9a04acc640 1 -- 172.21.15.16:0/2226210417 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9a0006b6c0 msgr2=0x7f9a001f5220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:38.408 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:38.406+0000 7f9a04acc640 1 --2- 172.21.15.16:0/2226210417 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9a0006b6c0 0x7f9a001f5220 secure :-1 s=READY pgs=2240 cs=0 l=1 rev1=1 crypto rx=0x7f99f0009710 tx=0x7f99f001ac90 comp rx=0 tx=0).stop 2023-05-22T16:03:38.418 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:38.410+0000 7f9a04acc640 1 -- 172.21.15.16:0/2226210417 shutdown_connections 2023-05-22T16:03:38.418 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:38.410+0000 7f9a04acc640 1 --2- 172.21.15.16:0/2226210417 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9a00159510 0x7f9a001514b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:38.418 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:38.410+0000 7f9a04acc640 1 --2- 172.21.15.16:0/2226210417 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9a0006ba90 0x7f9a001f5760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:38.418 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:38.410+0000 7f9a04acc640 1 --2- 172.21.15.16:0/2226210417 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f99e00654a0 0x7f99e0067950 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:38.418 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:38.410+0000 7f9a04acc640 1 --2- 172.21.15.16:0/2226210417 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9a0006b6c0 0x7f9a001f5220 unknown :-1 s=CLOSED pgs=2240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:38.419 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:38.410+0000 7f9a04acc640 1 -- 172.21.15.16:0/2226210417 >> 172.21.15.16:0/2226210417 conn(0x7f9a00069720 msgr2=0x7f9a0015b930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:03:38.420 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:38.410+0000 7f9a04acc640 1 -- 172.21.15.16:0/2226210417 shutdown_connections 2023-05-22T16:03:38.420 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:38.410+0000 7f9a04acc640 1 -- 172.21.15.16:0/2226210417 wait complete. 2023-05-22T16:03:38.448 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-05-22T16:03:38.573 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:38.574+0000 7f4a5c85c640 1 Processor -- start 2023-05-22T16:03:38.573 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:38.574+0000 7f4a5c85c640 1 -- start start 2023-05-22T16:03:38.574 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:38.574+0000 7f4a5c85c640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4a58159970 0x7f4a58159d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:38.574 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:38.574+0000 7f4a5c85c640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f4a5815d680 0x7f4a580cb260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:38.574 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:38.574+0000 7f4a5c85c640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f4a5815dad0 0x7f4a5806b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:38.575 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:38.574+0000 7f4a5c85c640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a58070c30 con 0x7f4a5815d680 2023-05-22T16:03:38.575 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:38.574+0000 7f4a5c85c640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a58070e60 con 0x7f4a5815dad0 2023-05-22T16:03:38.575 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:38.574+0000 7f4a5c85c640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a58071140 con 0x7f4a58159970 2023-05-22T16:03:38.575 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:38.574+0000 7f4a55d9b640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f4a5815d680 0x7f4a580cb260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:38.575 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:38.574+0000 7f4a55d9b640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f4a5815d680 0x7f4a580cb260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48188/0 (socket says 172.21.15.16:48188) 2023-05-22T16:03:38.575 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:38.574+0000 7f4a55d9b640 1 -- 172.21.15.16:0/3794434492 learned_addr learned my addr 172.21.15.16:0/3794434492 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:03:38.575 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:38.574+0000 7f4a5659c640 1 --2- 172.21.15.16:0/3794434492 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4a58159970 0x7f4a58159d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:38.576 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:38.574+0000 7f4a56d9d640 1 --2- 172.21.15.16:0/3794434492 >> v2:172.21.15.62:3300/0 conn(0x7f4a5815dad0 0x7f4a5806b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:38.576 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:38.574+0000 7f4a55d9b640 1 -- 172.21.15.16:0/3794434492 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4a58159970 msgr2=0x7f4a58159d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:03:38.576 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:38.574+0000 7f4a55d9b640 1 --2- 172.21.15.16:0/3794434492 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4a58159970 0x7f4a58159d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:38.576 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:38.574+0000 7f4a55d9b640 1 -- 172.21.15.16:0/3794434492 >> v2:172.21.15.62:3300/0 conn(0x7f4a5815dad0 msgr2=0x7f4a5806b6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:03:38.577 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:38.574+0000 7f4a55d9b640 1 --2- 172.21.15.16:0/3794434492 >> v2:172.21.15.62:3300/0 conn(0x7f4a5815dad0 0x7f4a5806b6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:38.577 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:38.574+0000 7f4a55d9b640 1 -- 172.21.15.16:0/3794434492 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a58071420 con 0x7f4a5815d680 2023-05-22T16:03:38.578 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:38.578+0000 7f4a55d9b640 1 --2- 172.21.15.16:0/3794434492 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4a5815d680 0x7f4a580cb260 secure :-1 s=READY pgs=2241 cs=0 l=1 rev1=1 crypto rx=0x7f4a580b40e0 tx=0x7f4a4c01c440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd66b980d0a2785a server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:03:38.578 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:38.578+0000 7f4a5559a640 1 -- 172.21.15.16:0/3794434492 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a4c0ac640 con 0x7f4a5815d680 2023-05-22T16:03:38.578 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:38.578+0000 7f4a5559a640 1 -- 172.21.15.16:0/3794434492 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a4c0ac7a0 con 0x7f4a5815d680 2023-05-22T16:03:38.578 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:38.578+0000 7f4a5559a640 1 -- 172.21.15.16:0/3794434492 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a4c0acb30 con 0x7f4a5815d680 2023-05-22T16:03:38.578 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:38.578+0000 7f4a5c85c640 1 -- 172.21.15.16:0/3794434492 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4a5815d680 msgr2=0x7f4a580cb260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:38.579 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:38.578+0000 7f4a5c85c640 1 --2- 172.21.15.16:0/3794434492 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4a5815d680 0x7f4a580cb260 secure :-1 s=READY pgs=2241 cs=0 l=1 rev1=1 crypto rx=0x7f4a580b40e0 tx=0x7f4a4c01c440 comp rx=0 tx=0).stop 2023-05-22T16:03:38.579 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:38.578+0000 7f4a5c85c640 1 -- 172.21.15.16:0/3794434492 shutdown_connections 2023-05-22T16:03:38.579 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:38.578+0000 7f4a5c85c640 1 --2- 172.21.15.16:0/3794434492 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4a5815d680 0x7f4a580cb260 unknown :-1 s=CLOSED pgs=2241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:38.579 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:38.578+0000 7f4a5c85c640 1 --2- 172.21.15.16:0/3794434492 >> v2:172.21.15.62:3300/0 conn(0x7f4a5815dad0 0x7f4a5806b6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:38.579 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:38.578+0000 7f4a5c85c640 1 --2- 172.21.15.16:0/3794434492 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4a58159970 0x7f4a58159d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:38.580 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:38.578+0000 7f4a5c85c640 1 -- 172.21.15.16:0/3794434492 >> 172.21.15.16:0/3794434492 conn(0x7f4a58069720 msgr2=0x7f4a58069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:03:38.580 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:38.578+0000 7f4a5c85c640 1 -- 172.21.15.16:0/3794434492 shutdown_connections 2023-05-22T16:03:38.580 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:38.578+0000 7f4a5c85c640 1 -- 172.21.15.16:0/3794434492 wait complete. 2023-05-22T16:03:38.582 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:38.578+0000 7f4a5c85c640 1 Processor -- start 2023-05-22T16:03:38.582 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:38.578+0000 7f4a5c85c640 1 -- start start 2023-05-22T16:03:38.582 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:38.578+0000 7f4a5c85c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4a58159970 0x7f4a5814dd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:38.582 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:38.578+0000 7f4a5c85c640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4a5815d680 0x7f4a580b14d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:38.582 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:38.578+0000 7f4a5c85c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4a5815dad0 0x7f4a5814f740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:38.583 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:38.578+0000 7f4a5c85c640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a5814e6a0 con 0x7f4a5815dad0 2023-05-22T16:03:38.583 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:38.578+0000 7f4a5c85c640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a5814e980 con 0x7f4a58159970 2023-05-22T16:03:38.583 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:38.578+0000 7f4a5c85c640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a5814ec60 con 0x7f4a5815d680 2023-05-22T16:03:38.583 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:38.578+0000 7f4a56d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4a5815dad0 0x7f4a5814f740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:38.583 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:38.578+0000 7f4a56d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4a5815dad0 0x7f4a5814f740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48200/0 (socket says 172.21.15.16:48200) 2023-05-22T16:03:38.583 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:38.578+0000 7f4a56d9d640 1 -- 172.21.15.16:0/1687754815 learned_addr learned my addr 172.21.15.16:0/1687754815 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:03:38.583 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:38.578+0000 7f4a5659c640 1 --2- 172.21.15.16:0/1687754815 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4a58159970 0x7f4a5814dd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:38.583 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:38.578+0000 7f4a55d9b640 1 --2- 172.21.15.16:0/1687754815 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4a5815d680 0x7f4a580b14d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:38.584 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:38.582+0000 7f4a56d9d640 1 -- 172.21.15.16:0/1687754815 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4a5815d680 msgr2=0x7f4a580b14d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:38.584 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:38.582+0000 7f4a56d9d640 1 --2- 172.21.15.16:0/1687754815 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4a5815d680 0x7f4a580b14d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:38.584 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:38.582+0000 7f4a56d9d640 1 -- 172.21.15.16:0/1687754815 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4a58159970 msgr2=0x7f4a5814dd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:38.584 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:38.582+0000 7f4a56d9d640 1 --2- 172.21.15.16:0/1687754815 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4a58159970 0x7f4a5814dd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:38.584 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:38.582+0000 7f4a56d9d640 1 -- 172.21.15.16:0/1687754815 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a4c0094d0 con 0x7f4a5815dad0 2023-05-22T16:03:38.584 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:38.582+0000 7f4a56d9d640 1 --2- 172.21.15.16:0/1687754815 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4a5815dad0 0x7f4a5814f740 secure :-1 s=READY pgs=2242 cs=0 l=1 rev1=1 crypto rx=0x7f4a50007d40 tx=0x7f4a5000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:03:38.584 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:38.582+0000 7f4a46ffd640 1 -- 172.21.15.16:0/1687754815 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a50014420 con 0x7f4a5815dad0 2023-05-22T16:03:38.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:38.582+0000 7f4a46ffd640 1 -- 172.21.15.16:0/1687754815 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a500145a0 con 0x7f4a5815dad0 2023-05-22T16:03:38.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:38.582+0000 7f4a5c85c640 1 -- 172.21.15.16:0/1687754815 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a580cae20 con 0x7f4a5815dad0 2023-05-22T16:03:38.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:38.582+0000 7f4a46ffd640 1 -- 172.21.15.16:0/1687754815 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a50014950 con 0x7f4a5815dad0 2023-05-22T16:03:38.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:38.582+0000 7f4a5c85c640 1 -- 172.21.15.16:0/1687754815 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a580cac90 con 0x7f4a5815dad0 2023-05-22T16:03:38.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:38.582+0000 7f4a5c85c640 1 -- 172.21.15.16:0/1687754815 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a5815adc0 con 0x7f4a5815dad0 2023-05-22T16:03:38.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:38.582+0000 7f4a46ffd640 1 -- 172.21.15.16:0/1687754815 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 82) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f4a50014ab0 con 0x7f4a5815dad0 2023-05-22T16:03:38.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:38.582+0000 7f4a46ffd640 1 --2- 172.21.15.16:0/1687754815 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f4a380654a0 0x7f4a38067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:38.587 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:38.582+0000 7f4a46ffd640 1 -- 172.21.15.16:0/1687754815 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(177..177 src has 1..177) v4 ==== 6338+0+0 (secure 0 0 0) 0x7f4a50084010 con 0x7f4a5815dad0 2023-05-22T16:03:38.587 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:38.582+0000 7f4a5659c640 1 --2- 172.21.15.16:0/1687754815 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f4a380654a0 0x7f4a38067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:38.587 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:38.582+0000 7f4a5659c640 1 --2- 172.21.15.16:0/1687754815 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f4a380654a0 0x7f4a38067950 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f4a480045f0 tx=0x7f4a48008040 comp rx=0 tx=0).ready entity=mgr.10032 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:03:38.587 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:38.586+0000 7f4a46ffd640 1 -- 172.21.15.16:0/1687754815 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f4a50051c80 con 0x7f4a5815dad0 2023-05-22T16:03:38.776 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:38.774+0000 7f4a5c85c640 1 -- 172.21.15.16:0/1687754815 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f4a5804dbe0 con 0x7f4a5815dad0 2023-05-22T16:03:38.790 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:38.790+0000 7f4a46ffd640 1 -- 172.21.15.16:0/1687754815 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 22 and data pool 23 v70) v1 ==== 174+0+0 (secure 0 0 0) 0x7f4a500552a0 con 0x7f4a5815dad0 2023-05-22T16:03:38.800 INFO:teuthology.orchestra.run.smithi016.stderr:new fs with metadata pool 22 and data pool 23 2023-05-22T16:03:38.808 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:38.806+0000 7f4a5c85c640 1 -- 172.21.15.16:0/1687754815 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f4a380654a0 msgr2=0x7f4a38067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:38.808 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:38.806+0000 7f4a5c85c640 1 --2- 172.21.15.16:0/1687754815 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f4a380654a0 0x7f4a38067950 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f4a480045f0 tx=0x7f4a48008040 comp rx=0 tx=0).stop 2023-05-22T16:03:38.808 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:38.806+0000 7f4a5c85c640 1 -- 172.21.15.16:0/1687754815 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4a5815dad0 msgr2=0x7f4a5814f740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:38.808 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:38.806+0000 7f4a5c85c640 1 --2- 172.21.15.16:0/1687754815 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4a5815dad0 0x7f4a5814f740 secure :-1 s=READY pgs=2242 cs=0 l=1 rev1=1 crypto rx=0x7f4a50007d40 tx=0x7f4a5000e4b0 comp rx=0 tx=0).stop 2023-05-22T16:03:38.814 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:38.814+0000 7f4a5c85c640 1 -- 172.21.15.16:0/1687754815 shutdown_connections 2023-05-22T16:03:38.815 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:38.814+0000 7f4a5c85c640 1 --2- 172.21.15.16:0/1687754815 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f4a380654a0 0x7f4a38067950 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:38.815 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:38.814+0000 7f4a5c85c640 1 --2- 172.21.15.16:0/1687754815 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4a5815dad0 0x7f4a5814f740 unknown :-1 s=CLOSED pgs=2242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:38.815 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:38.814+0000 7f4a5c85c640 1 --2- 172.21.15.16:0/1687754815 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4a5815d680 0x7f4a580b14d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:38.815 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:38.814+0000 7f4a5c85c640 1 --2- 172.21.15.16:0/1687754815 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4a58159970 0x7f4a5814dd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:38.815 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:38.814+0000 7f4a5c85c640 1 -- 172.21.15.16:0/1687754815 >> 172.21.15.16:0/1687754815 conn(0x7f4a58069720 msgr2=0x7f4a58158140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:03:38.817 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:38.818+0000 7f4a5c85c640 1 -- 172.21.15.16:0/1687754815 shutdown_connections 2023-05-22T16:03:38.817 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:38.818+0000 7f4a5c85c640 1 -- 172.21.15.16:0/1687754815 wait complete. 2023-05-22T16:03:38.842 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-05-22T16:03:38.956 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:38.958+0000 7fd79c9a0640 1 Processor -- start 2023-05-22T16:03:38.957 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:38.958+0000 7fd79c9a0640 1 -- start start 2023-05-22T16:03:38.958 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:38.958+0000 7fd79c9a0640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd7980ca4e0 0x7fd79815e460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:38.959 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:38.958+0000 7fd79c9a0640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd7980c9b40 0x7fd7980afb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:38.959 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:38.958+0000 7fd79c9a0640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7980b0650 con 0x7fd798070df0 2023-05-22T16:03:38.959 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:38.958+0000 7fd79c9a0640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7980b0370 con 0x7fd7980c9b40 2023-05-22T16:03:38.959 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:38.958+0000 7fd79c9a0640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd79815d980 con 0x7fd7980ca4e0 2023-05-22T16:03:38.959 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:38.958+0000 7fd795d9b640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd7980c9b40 0x7fd7980afb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:38.960 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:38.958+0000 7fd795d9b640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd7980c9b40 0x7fd7980afb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48212/0 (socket says 172.21.15.16:48212) 2023-05-22T16:03:38.960 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:38.958+0000 7fd795d9b640 1 -- 172.21.15.16:0/2356501299 learned_addr learned my addr 172.21.15.16:0/2356501299 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:03:38.960 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:38.958+0000 7fd79659c640 1 --2- 172.21.15.16:0/2356501299 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd7980ca4e0 0x7fd79815e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:38.960 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:38.958+0000 7fd795d9b640 1 -- 172.21.15.16:0/2356501299 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd7980ca4e0 msgr2=0x7fd79815e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:03:38.960 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:38.958+0000 7fd795d9b640 1 --2- 172.21.15.16:0/2356501299 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd7980ca4e0 0x7fd79815e460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:38.960 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:38.958+0000 7fd795d9b640 1 -- 172.21.15.16:0/2356501299 >> v1:172.21.15.16:6789/0 conn(0x7fd798070df0 legacy=0x7fd798071210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:38.960 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:38.958+0000 7fd795d9b640 1 -- 172.21.15.16:0/2356501299 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd79815dc60 con 0x7fd7980c9b40 2023-05-22T16:03:38.961 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:38.958+0000 7fd79559a640 1 -- 172.21.15.16:0/2356501299 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3108243499 0 0) 0x7fd7980b0650 con 0x7fd798070df0 2023-05-22T16:03:38.962 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:38.962+0000 7fd795d9b640 1 --2- 172.21.15.16:0/2356501299 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd7980c9b40 0x7fd7980afb40 secure :-1 s=READY pgs=2244 cs=0 l=1 rev1=1 crypto rx=0x7fd78c009870 tx=0x7fd78c01c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91b9a0399ca81e24 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:03:38.962 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:38.962+0000 7fd79559a640 1 -- 172.21.15.16:0/2356501299 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd78c0ab660 con 0x7fd7980c9b40 2023-05-22T16:03:38.962 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:38.962+0000 7fd79559a640 1 -- 172.21.15.16:0/2356501299 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd78c0ab7c0 con 0x7fd7980c9b40 2023-05-22T16:03:38.962 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:38.962+0000 7fd79559a640 1 -- 172.21.15.16:0/2356501299 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd78c0abb50 con 0x7fd7980c9b40 2023-05-22T16:03:38.963 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:38.962+0000 7fd79c9a0640 1 -- 172.21.15.16:0/2356501299 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd7980c9b40 msgr2=0x7fd7980afb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:38.963 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:38.962+0000 7fd79c9a0640 1 --2- 172.21.15.16:0/2356501299 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd7980c9b40 0x7fd7980afb40 secure :-1 s=READY pgs=2244 cs=0 l=1 rev1=1 crypto rx=0x7fd78c009870 tx=0x7fd78c01c480 comp rx=0 tx=0).stop 2023-05-22T16:03:38.963 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:38.962+0000 7fd79c9a0640 1 -- 172.21.15.16:0/2356501299 shutdown_connections 2023-05-22T16:03:38.963 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:38.962+0000 7fd79c9a0640 1 --2- 172.21.15.16:0/2356501299 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd7980c9b40 0x7fd7980afb40 unknown :-1 s=CLOSED pgs=2244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:38.963 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:38.962+0000 7fd79c9a0640 1 --2- 172.21.15.16:0/2356501299 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd7980ca4e0 0x7fd79815e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:38.963 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:38.962+0000 7fd79c9a0640 1 -- 172.21.15.16:0/2356501299 >> 172.21.15.16:0/2356501299 conn(0x7fd798069720 msgr2=0x7fd798069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:03:38.963 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:38.962+0000 7fd79c9a0640 1 -- 172.21.15.16:0/2356501299 shutdown_connections 2023-05-22T16:03:38.964 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:38.962+0000 7fd79c9a0640 1 -- 172.21.15.16:0/2356501299 wait complete. 2023-05-22T16:03:38.966 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:38.962+0000 7fd79c9a0640 1 Processor -- start 2023-05-22T16:03:38.966 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:38.962+0000 7fd79c9a0640 1 -- start start 2023-05-22T16:03:38.966 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:38.962+0000 7fd79c9a0640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd798070df0 0x7fd79814e640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:38.966 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:38.962+0000 7fd79c9a0640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd7980c9b40 0x7fd7980c61a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:38.966 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:38.962+0000 7fd79c9a0640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd7980ca4e0 0x7fd79814fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:38.966 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:38.962+0000 7fd79c9a0640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd79814ecc0 con 0x7fd7980c9b40 2023-05-22T16:03:38.967 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:38.962+0000 7fd79c9a0640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd79814ef10 con 0x7fd798070df0 2023-05-22T16:03:38.967 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:38.962+0000 7fd79c9a0640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd79814f1f0 con 0x7fd7980ca4e0 2023-05-22T16:03:38.967 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:38.962+0000 7fd795d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd7980c9b40 0x7fd7980c61a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:38.967 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:38.962+0000 7fd795d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd7980c9b40 0x7fd7980c61a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48216/0 (socket says 172.21.15.16:48216) 2023-05-22T16:03:38.967 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:38.962+0000 7fd795d9b640 1 -- 172.21.15.16:0/74967809 learned_addr learned my addr 172.21.15.16:0/74967809 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:03:38.967 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:38.962+0000 7fd796d9d640 1 --2- 172.21.15.16:0/74967809 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd7980ca4e0 0x7fd79814fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:38.967 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:38.966+0000 7fd79659c640 1 --2- 172.21.15.16:0/74967809 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd798070df0 0x7fd79814e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:38.967 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:38.966+0000 7fd795d9b640 1 -- 172.21.15.16:0/74967809 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd7980ca4e0 msgr2=0x7fd79814fcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:38.968 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:38.966+0000 7fd795d9b640 1 --2- 172.21.15.16:0/74967809 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd7980ca4e0 0x7fd79814fcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:38.968 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:38.966+0000 7fd795d9b640 1 -- 172.21.15.16:0/74967809 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd798070df0 msgr2=0x7fd79814e640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:38.968 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:38.966+0000 7fd795d9b640 1 --2- 172.21.15.16:0/74967809 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd798070df0 0x7fd79814e640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:38.968 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:38.966+0000 7fd795d9b640 1 -- 172.21.15.16:0/74967809 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd78c0094d0 con 0x7fd7980c9b40 2023-05-22T16:03:38.968 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:38.966+0000 7fd795d9b640 1 --2- 172.21.15.16:0/74967809 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd7980c9b40 0x7fd7980c61a0 secure :-1 s=READY pgs=2245 cs=0 l=1 rev1=1 crypto rx=0x7fd78c01a0b0 tx=0x7fd78c01acc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:03:38.968 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:38.966+0000 7fd786ffd640 1 -- 172.21.15.16:0/74967809 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd78c0abe00 con 0x7fd7980c9b40 2023-05-22T16:03:38.968 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:38.966+0000 7fd786ffd640 1 -- 172.21.15.16:0/74967809 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd78c0b8070 con 0x7fd7980c9b40 2023-05-22T16:03:38.969 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:38.966+0000 7fd79c9a0640 1 -- 172.21.15.16:0/74967809 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7980c8e80 con 0x7fd7980c9b40 2023-05-22T16:03:38.969 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:38.966+0000 7fd786ffd640 1 -- 172.21.15.16:0/74967809 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd78c004220 con 0x7fd7980c9b40 2023-05-22T16:03:38.969 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:38.966+0000 7fd79c9a0640 1 -- 172.21.15.16:0/74967809 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7980ca8b0 con 0x7fd7980c9b40 2023-05-22T16:03:38.969 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:38.966+0000 7fd79c9a0640 1 -- 172.21.15.16:0/74967809 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7980c9010 con 0x7fd7980c9b40 2023-05-22T16:03:38.970 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:38.966+0000 7fd786ffd640 1 -- 172.21.15.16:0/74967809 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 82) v1 ==== 84332+0+0 (secure 0 0 0) 0x7fd78c0a9070 con 0x7fd7980c9b40 2023-05-22T16:03:38.970 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:38.966+0000 7fd786ffd640 1 --2- 172.21.15.16:0/74967809 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fd7780654d0 0x7fd778067980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:38.970 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:38.966+0000 7fd786ffd640 1 -- 172.21.15.16:0/74967809 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(178..178 src has 1..178) v4 ==== 6438+0+0 (secure 0 0 0) 0x7fd78c11cd90 con 0x7fd7980c9b40 2023-05-22T16:03:38.970 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:38.966+0000 7fd79659c640 1 --2- 172.21.15.16:0/74967809 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fd7780654d0 0x7fd778067980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:38.970 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:38.966+0000 7fd79659c640 1 --2- 172.21.15.16:0/74967809 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fd7780654d0 0x7fd778067980 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fd798150720 tx=0x7fd788008040 comp rx=0 tx=0).ready entity=mgr.10032 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:03:38.971 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:38.970+0000 7fd786ffd640 1 -- 172.21.15.16:0/74967809 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fd78c0ea810 con 0x7fd7980c9b40 2023-05-22T16:03:39.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:39.122+0000 7fd79c9a0640 1 -- 172.21.15.16:0/74967809 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd79804dbe0 con 0x7fd7980c9b40 2023-05-22T16:03:39.126 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:39.126+0000 7fd786ffd640 1 -- 172.21.15.16:0/74967809 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v178) v1 ==== 74+0+18145 (secure 0 0 0) 0x7fd78c0ba090 con 0x7fd7980c9b40 2023-05-22T16:03:39.127 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T16:03:39.127 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":178,"fsid":"b2719834-7847-4d98-90a6-f65d57f85c49","created":"2023-05-22T15:26:25.259314+0000","modified":"2023-05-22T16:03:38.778799+0000","last_up_change":"2023-05-22T15:26:36.448524+0000","last_in_change":"2023-05-22T15:26:27.675668+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":23,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-05-22T15:26:38.019781+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-05-22T15:26:43.570514+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":22,"pool_name":"cephfs_metadata","create_time":"2023-05-22T16:03:37.371980+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"178","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_data","create_time":"2023-05-22T16:03:37.756887+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"178","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9e7ff854-b25f-4cbb-a5c2-1799d9c53d2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":37021},{"type":"v1","addr":"172.21.15.16:6825","nonce":37021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":37021},{"type":"v1","addr":"172.21.15.16:6827","nonce":37021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":37021},{"type":"v1","addr":"172.21.15.16:6831","nonce":37021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":37021},{"type":"v1","addr":"172.21.15.16:6829","nonce":37021}]},"public_addr":"172.21.15.16:6825/37021","cluster_addr":"172.21.15.16:6827/37021","heartbeat_back_addr":"172.21.15.16:6831/37021","heartbeat_front_addr":"172.21.15.16:6829/37021","state":["exists","up"]},{"osd":1,"uuid":"4702fbc9-a927-4565-aca5-41dc69cc8264","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":37008},{"type":"v1","addr":"172.21.15.16:6801","nonce":37008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":37008},{"type":"v1","addr":"172.21.15.16:6803","nonce":37008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":37008},{"type":"v1","addr":"172.21.15.16:6807","nonce":37008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":37008},{"type":"v1","addr":"172.21.15.16:6805","nonce":37008}]},"public_addr":"172.21.15.16:6801/37008","cluster_addr":"172.21.15.16:6803/37008","heartbeat_back_addr":"172.21.15.16:6807/37008","heartbeat_front_addr":"172.21.15.16:6805/37008","state":["exists","up"]},{"osd":2,"uuid":"1e7bba46-4eba-47ef-bb81-5e2d371638ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":37009},{"type":"v1","addr":"172.21.15.16:6809","nonce":37009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":37009},{"type":"v1","addr":"172.21.15.16:6811","nonce":37009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":37009},{"type":"v1","addr":"172.21.15.16:6815","nonce":37009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":37009},{"type":"v1","addr":"172.21.15.16:6813","nonce":37009}]},"public_addr":"172.21.15.16:6809/37009","cluster_addr":"172.21.15.16:6811/37009","heartbeat_back_addr":"172.21.15.16:6815/37009","heartbeat_front_addr":"172.21.15.16:6813/37009","state":["exists","up"]},{"osd":3,"uuid":"383cacc0-5a88-4904-990e-498ca59ef48a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":37020},{"type":"v1","addr":"172.21.15.16:6817","nonce":37020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":37020},{"type":"v1","addr":"172.21.15.16:6819","nonce":37020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":37020},{"type":"v1","addr":"172.21.15.16:6823","nonce":37020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":37020},{"type":"v1","addr":"172.21.15.16:6821","nonce":37020}]},"public_addr":"172.21.15.16:6817/37020","cluster_addr":"172.21.15.16:6819/37020","heartbeat_back_addr":"172.21.15.16:6823/37020","heartbeat_front_addr":"172.21.15.16:6821/37020","state":["exists","up"]},{"osd":4,"uuid":"970500ef-95b7-42c1-b080-eed4fae7013d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":37353},{"type":"v1","addr":"172.21.15.62:6817","nonce":37353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6819","nonce":37353},{"type":"v1","addr":"172.21.15.62:6822","nonce":37353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37353},{"type":"v1","addr":"172.21.15.62:6831","nonce":37353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37353},{"type":"v1","addr":"172.21.15.62:6826","nonce":37353}]},"public_addr":"172.21.15.62:6817/37353","cluster_addr":"172.21.15.62:6822/37353","heartbeat_back_addr":"172.21.15.62:6831/37353","heartbeat_front_addr":"172.21.15.62:6826/37353","state":["exists","up"]},{"osd":5,"uuid":"022d67db-d4d8-445b-b1b9-8b446a00f5c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":37352},{"type":"v1","addr":"172.21.15.62:6801","nonce":37352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":37352},{"type":"v1","addr":"172.21.15.62:6803","nonce":37352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":37352},{"type":"v1","addr":"172.21.15.62:6807","nonce":37352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":37352},{"type":"v1","addr":"172.21.15.62:6805","nonce":37352}]},"public_addr":"172.21.15.62:6801/37352","cluster_addr":"172.21.15.62:6803/37352","heartbeat_back_addr":"172.21.15.62:6807/37352","heartbeat_front_addr":"172.21.15.62:6805/37352","state":["exists","up"]},{"osd":6,"uuid":"42bd372b-6688-4a1b-8407-e16e8c266e33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":37354},{"type":"v1","addr":"172.21.15.62:6820","nonce":37354}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":37354},{"type":"v1","addr":"172.21.15.62:6823","nonce":37354}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6829","nonce":37354},{"type":"v1","addr":"172.21.15.62:6830","nonce":37354}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6825","nonce":37354},{"type":"v1","addr":"172.21.15.62:6827","nonce":37354}]},"public_addr":"172.21.15.62:6820/37354","cluster_addr":"172.21.15.62:6823/37354","heartbeat_back_addr":"172.21.15.62:6830/37354","heartbeat_front_addr":"172.21.15.62:6827/37354","state":["exists","up"]},{"osd":7,"uuid":"ac70b2a3-f41b-4bf2-b227-7f2cbb55426d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37355},{"type":"v1","addr":"172.21.15.62:6809","nonce":37355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37355},{"type":"v1","addr":"172.21.15.62:6811","nonce":37355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":37355},{"type":"v1","addr":"172.21.15.62:6815","nonce":37355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":37355},{"type":"v1","addr":"172.21.15.62:6813","nonce":37355}]},"public_addr":"172.21.15.62:6809/37355","cluster_addr":"172.21.15.62:6811/37355","heartbeat_back_addr":"172.21.15.62:6815/37355","heartbeat_front_addr":"172.21.15.62:6813/37355","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.368896+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.107273+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.276888+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.287502+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.124791+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.251937+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.255679+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.232797+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":17,"snaps":[{"begin":2,"length":2}]},{"pool":21,"snaps":[{"begin":25,"length":9}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-05-22T16:03:39.129 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:39.130+0000 7fd79c9a0640 1 -- 172.21.15.16:0/74967809 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fd7780654d0 msgr2=0x7fd778067980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:39.129 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:39.130+0000 7fd79c9a0640 1 --2- 172.21.15.16:0/74967809 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fd7780654d0 0x7fd778067980 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fd798150720 tx=0x7fd788008040 comp rx=0 tx=0).stop 2023-05-22T16:03:39.129 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:39.130+0000 7fd79c9a0640 1 -- 172.21.15.16:0/74967809 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd7980c9b40 msgr2=0x7fd7980c61a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:39.129 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:39.130+0000 7fd79c9a0640 1 --2- 172.21.15.16:0/74967809 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd7980c9b40 0x7fd7980c61a0 secure :-1 s=READY pgs=2245 cs=0 l=1 rev1=1 crypto rx=0x7fd78c01a0b0 tx=0x7fd78c01acc0 comp rx=0 tx=0).stop 2023-05-22T16:03:39.129 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:39.130+0000 7fd79c9a0640 1 -- 172.21.15.16:0/74967809 shutdown_connections 2023-05-22T16:03:39.129 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:39.130+0000 7fd79c9a0640 1 --2- 172.21.15.16:0/74967809 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd7980ca4e0 0x7fd79814fcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:39.130 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:39.130+0000 7fd79c9a0640 1 --2- 172.21.15.16:0/74967809 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fd7780654d0 0x7fd778067980 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:39.130 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:39.130+0000 7fd79c9a0640 1 --2- 172.21.15.16:0/74967809 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd7980c9b40 0x7fd7980c61a0 unknown :-1 s=CLOSED pgs=2245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:39.130 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:39.130+0000 7fd79c9a0640 1 --2- 172.21.15.16:0/74967809 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd798070df0 0x7fd79814e640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:39.130 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:39.130+0000 7fd79c9a0640 1 -- 172.21.15.16:0/74967809 >> 172.21.15.16:0/74967809 conn(0x7fd798069720 msgr2=0x7fd798069af0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:03:39.130 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:39.130+0000 7fd79c9a0640 1 -- 172.21.15.16:0/74967809 shutdown_connections 2023-05-22T16:03:39.130 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:39.130+0000 7fd79c9a0640 1 -- 172.21.15.16:0/74967809 wait complete. 2023-05-22T16:03:39.146 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-05-22T16:03:39.269 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:39.270+0000 7f409cc73640 1 Processor -- start 2023-05-22T16:03:39.269 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:39.270+0000 7f409cc73640 1 -- start start 2023-05-22T16:03:39.272 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:39.270+0000 7f409cc73640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f40980ca4e0 0x7f409815e460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:39.272 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:39.270+0000 7f409cc73640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f40980c9b40 0x7f40980afb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:39.272 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:39.270+0000 7f409cc73640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40980b0650 con 0x7f4098070df0 2023-05-22T16:03:39.272 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:39.270+0000 7f409cc73640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40980b0370 con 0x7f40980ca4e0 2023-05-22T16:03:39.272 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:39.270+0000 7f409cc73640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f409815d980 con 0x7f40980c9b40 2023-05-22T16:03:39.272 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:39.270+0000 7f4096d9d640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f4098070df0 0x7f4098071210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:53180/0 (socket says 172.21.15.16:53180) 2023-05-22T16:03:39.273 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:39.270+0000 7f4096d9d640 1 -- 172.21.15.16:0/1277712512 learned_addr learned my addr 172.21.15.16:0/1277712512 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T16:03:39.273 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:39.270+0000 7f4095d9b640 1 --2- 172.21.15.16:0/1277712512 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f40980c9b40 0x7f40980afb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:39.273 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:39.270+0000 7f409659c640 1 --2- 172.21.15.16:0/1277712512 >> v2:172.21.15.62:3300/0 conn(0x7f40980ca4e0 0x7f409815e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:39.273 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:39.270+0000 7f409559a640 1 -- 172.21.15.16:0/1277712512 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 191273652 0 0) 0x7f40980b0650 con 0x7f4098070df0 2023-05-22T16:03:39.273 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:39.270+0000 7f409559a640 1 -- 172.21.15.16:0/1277712512 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4080003530 con 0x7f4098070df0 2023-05-22T16:03:39.273 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:39.270+0000 7f409559a640 1 -- 172.21.15.16:0/1277712512 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2463810230 0 0) 0x7f4080003530 con 0x7f4098070df0 2023-05-22T16:03:39.273 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:39.270+0000 7f409559a640 1 -- 172.21.15.16:0/1277712512 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f40980c9b40 msgr2=0x7f40980afb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:03:39.274 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:39.270+0000 7f409559a640 1 --2- 172.21.15.16:0/1277712512 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f40980c9b40 0x7f40980afb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:39.274 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:39.270+0000 7f409559a640 1 -- 172.21.15.16:0/1277712512 >> v2:172.21.15.62:3300/0 conn(0x7f40980ca4e0 msgr2=0x7f409815e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:03:39.274 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:39.270+0000 7f409559a640 1 --2- 172.21.15.16:0/1277712512 >> v2:172.21.15.62:3300/0 conn(0x7f40980ca4e0 0x7f409815e460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:39.274 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:39.270+0000 7f409559a640 1 -- 172.21.15.16:0/1277712512 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f409815dc60 con 0x7f4098070df0 2023-05-22T16:03:39.274 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:39.270+0000 7f409559a640 1 -- 172.21.15.16:0/1277712512 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7f4090001a00 con 0x7f4098070df0 2023-05-22T16:03:39.274 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:39.270+0000 7f4095d9b640 1 --2- 172.21.15.16:0/1277712512 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f40980c9b40 0x7f40980afb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-05-22T16:03:39.274 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:39.270+0000 7f409559a640 1 -- 172.21.15.16:0/1277712512 >> v1:172.21.15.16:6789/0 conn(0x7f4098070df0 legacy=0x7f4098071210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:39.274 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:39.270+0000 7f409559a640 1 --2- 172.21.15.16:0/1277712512 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f40980c9b40 0x7f4080003980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:39.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:39.270+0000 7f409559a640 1 --2- 172.21.15.16:0/1277712512 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4080003ec0 0x7f40800047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:39.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:39.270+0000 7f409559a640 1 --2- 172.21.15.16:0/1277712512 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4080004d10 0x7f4080005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:39.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:39.270+0000 7f409559a640 1 -- 172.21.15.16:0/1277712512 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40980b0370 con 0x7f4080004d10 2023-05-22T16:03:39.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:39.270+0000 7f409559a640 1 -- 172.21.15.16:0/1277712512 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f409815d980 con 0x7f4080003ec0 2023-05-22T16:03:39.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:39.270+0000 7f409559a640 1 -- 172.21.15.16:0/1277712512 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f408000b990 con 0x7f40980c9b40 2023-05-22T16:03:39.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:39.270+0000 7f4096d9d640 1 --2- 172.21.15.16:0/1277712512 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4080004d10 0x7f4080005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:39.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:39.270+0000 7f409659c640 1 --2- 172.21.15.16:0/1277712512 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f40980c9b40 0x7f4080003980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:39.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:39.270+0000 7f4095d9b640 1 --2- 172.21.15.16:0/1277712512 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4080003ec0 0x7f40800047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:39.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:39.270+0000 7f4096d9d640 1 -- 172.21.15.16:0/1277712512 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f40980c9b40 msgr2=0x7f4080003980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:03:39.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:39.270+0000 7f4096d9d640 1 --2- 172.21.15.16:0/1277712512 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f40980c9b40 0x7f4080003980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:39.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:39.270+0000 7f4096d9d640 1 -- 172.21.15.16:0/1277712512 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4080003ec0 msgr2=0x7f40800047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:03:39.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:39.270+0000 7f4096d9d640 1 --2- 172.21.15.16:0/1277712512 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4080003ec0 0x7f40800047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:39.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:39.270+0000 7f4096d9d640 1 -- 172.21.15.16:0/1277712512 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f408c009420 con 0x7f4080004d10 2023-05-22T16:03:39.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:39.270+0000 7f409659c640 1 --2- 172.21.15.16:0/1277712512 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f40980c9b40 0x7f4080003980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T16:03:39.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:39.270+0000 7f4095d9b640 1 --2- 172.21.15.16:0/1277712512 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4080003ec0 0x7f40800047d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T16:03:39.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:39.274+0000 7f4096d9d640 1 --2- 172.21.15.16:0/1277712512 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4080004d10 0x7f4080005180 secure :-1 s=READY pgs=2247 cs=0 l=1 rev1=1 crypto rx=0x7f409000a860 tx=0x7f409001fac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55062a2d3b5ed777 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:03:39.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:39.274+0000 7f409559a640 1 -- 172.21.15.16:0/1277712512 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40900aa6f0 con 0x7f4080004d10 2023-05-22T16:03:39.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:39.274+0000 7f409559a640 1 -- 172.21.15.16:0/1277712512 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40900aa850 con 0x7f4080004d10 2023-05-22T16:03:39.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:39.274+0000 7f409559a640 1 -- 172.21.15.16:0/1277712512 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40900aac60 con 0x7f4080004d10 2023-05-22T16:03:39.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:39.274+0000 7f409cc73640 1 -- 172.21.15.16:0/1277712512 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4080004d10 msgr2=0x7f4080005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:39.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:39.274+0000 7f409cc73640 1 --2- 172.21.15.16:0/1277712512 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4080004d10 0x7f4080005180 secure :-1 s=READY pgs=2247 cs=0 l=1 rev1=1 crypto rx=0x7f409000a860 tx=0x7f409001fac0 comp rx=0 tx=0).stop 2023-05-22T16:03:39.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:39.274+0000 7f4096d9d640 1 -- 172.21.15.16:0/1277712512 reap_dead start 2023-05-22T16:03:39.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:39.274+0000 7f409cc73640 1 -- 172.21.15.16:0/1277712512 shutdown_connections 2023-05-22T16:03:39.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:39.274+0000 7f409cc73640 1 -- 172.21.15.16:0/1277712512 >> 172.21.15.16:0/1277712512 conn(0x7f4098069720 msgr2=0x7f4098069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:03:39.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:39.274+0000 7f409cc73640 1 -- 172.21.15.16:0/1277712512 shutdown_connections 2023-05-22T16:03:39.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:39.274+0000 7f409cc73640 1 -- 172.21.15.16:0/1277712512 wait complete. 2023-05-22T16:03:39.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:39.274+0000 7f409cc73640 1 Processor -- start 2023-05-22T16:03:39.282 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:39.274+0000 7f409cc73640 1 -- start start 2023-05-22T16:03:39.282 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:39.274+0000 7f409cc73640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4080004d10 0x7f4098147440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:39.282 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:39.274+0000 7f409cc73640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f409814f400 0x7f40980c1720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:39.282 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:39.274+0000 7f409cc73640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4098154250 0x7f40981546d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:39.282 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:39.274+0000 7f409cc73640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40980c1c60 con 0x7f4080004d10 2023-05-22T16:03:39.282 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:39.274+0000 7f409cc73640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40980b1ea0 con 0x7f409814f400 2023-05-22T16:03:39.282 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:39.274+0000 7f409cc73640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4098154c10 con 0x7f4098154250 2023-05-22T16:03:39.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:39.274+0000 7f409659c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4080004d10 0x7f4098147440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:39.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:39.278+0000 7f409659c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4080004d10 0x7f4098147440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48928/0 (socket says 172.21.15.16:48928) 2023-05-22T16:03:39.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:39.278+0000 7f409659c640 1 -- 172.21.15.16:0/3225385920 learned_addr learned my addr 172.21.15.16:0/3225385920 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:03:39.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:39.278+0000 7f4095d9b640 1 --2- 172.21.15.16:0/3225385920 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f409814f400 0x7f40980c1720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:39.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:39.278+0000 7f4096d9d640 1 --2- 172.21.15.16:0/3225385920 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4098154250 0x7f40981546d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:39.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:39.278+0000 7f409659c640 1 -- 172.21.15.16:0/3225385920 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4098154250 msgr2=0x7f40981546d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:39.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:39.278+0000 7f409659c640 1 --2- 172.21.15.16:0/3225385920 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4098154250 0x7f40981546d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:39.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:39.278+0000 7f409659c640 1 -- 172.21.15.16:0/3225385920 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f409814f400 msgr2=0x7f40980c1720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:39.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:39.278+0000 7f409659c640 1 --2- 172.21.15.16:0/3225385920 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f409814f400 0x7f40980c1720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:39.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:39.278+0000 7f409659c640 1 -- 172.21.15.16:0/3225385920 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40980ca8b0 con 0x7f4080004d10 2023-05-22T16:03:39.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:39.278+0000 7f4095d9b640 1 --2- 172.21.15.16:0/3225385920 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f409814f400 0x7f40980c1720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T16:03:39.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:39.278+0000 7f4096d9d640 1 --2- 172.21.15.16:0/3225385920 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4098154250 0x7f40981546d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T16:03:39.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:39.278+0000 7f409659c640 1 --2- 172.21.15.16:0/3225385920 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4080004d10 0x7f4098147440 secure :-1 s=READY pgs=2248 cs=0 l=1 rev1=1 crypto rx=0x7f4088010950 tx=0x7f4088010e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:03:39.285 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:39.278+0000 7f4086ffd640 1 -- 172.21.15.16:0/3225385920 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f408800ec20 con 0x7f4080004d10 2023-05-22T16:03:39.285 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:39.278+0000 7f4086ffd640 1 -- 172.21.15.16:0/3225385920 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f408800ed80 con 0x7f4080004d10 2023-05-22T16:03:39.285 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:39.278+0000 7f409cc73640 1 -- 172.21.15.16:0/3225385920 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f409815df40 con 0x7f4080004d10 2023-05-22T16:03:39.285 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:39.278+0000 7f4086ffd640 1 -- 172.21.15.16:0/3225385920 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4088004220 con 0x7f4080004d10 2023-05-22T16:03:39.285 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:39.278+0000 7f409cc73640 1 -- 172.21.15.16:0/3225385920 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40980afb40 con 0x7f4080004d10 2023-05-22T16:03:39.285 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:39.278+0000 7f409cc73640 1 -- 172.21.15.16:0/3225385920 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f409804dbe0 con 0x7f4080004d10 2023-05-22T16:03:39.286 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:39.278+0000 7f4086ffd640 1 -- 172.21.15.16:0/3225385920 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 82) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f4088004380 con 0x7f4080004d10 2023-05-22T16:03:39.286 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:39.278+0000 7f4086ffd640 1 --2- 172.21.15.16:0/3225385920 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f40780654a0 0x7f4078067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:39.287 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:39.278+0000 7f4095d9b640 1 --2- 172.21.15.16:0/3225385920 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f40780654a0 0x7f4078067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:39.287 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:39.278+0000 7f4086ffd640 1 -- 172.21.15.16:0/3225385920 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(178..178 src has 1..178) v4 ==== 6438+0+0 (secure 0 0 0) 0x7f4088083200 con 0x7f4080004d10 2023-05-22T16:03:39.287 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:39.282+0000 7f4095d9b640 1 --2- 172.21.15.16:0/3225385920 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f40780654a0 0x7f4078067950 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f408c006e90 tx=0x7f408c008040 comp rx=0 tx=0).ready entity=mgr.10032 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:03:39.287 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:39.282+0000 7f4086ffd640 1 -- 172.21.15.16:0/3225385920 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f4088050d60 con 0x7f4080004d10 2023-05-22T16:03:39.436 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:39.434+0000 7f409cc73640 1 -- 172.21.15.16:0/3225385920 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f40980cb010 con 0x7f4080004d10 2023-05-22T16:03:39.439 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:39.438+0000 7f4086ffd640 1 -- 172.21.15.16:0/3225385920 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v178) v1 ==== 74+0+18145 (secure 0 0 0) 0x7f4088054380 con 0x7f4080004d10 2023-05-22T16:03:39.440 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T16:03:39.440 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":178,"fsid":"b2719834-7847-4d98-90a6-f65d57f85c49","created":"2023-05-22T15:26:25.259314+0000","modified":"2023-05-22T16:03:38.778799+0000","last_up_change":"2023-05-22T15:26:36.448524+0000","last_in_change":"2023-05-22T15:26:27.675668+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":23,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-05-22T15:26:38.019781+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-05-22T15:26:43.570514+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":22,"pool_name":"cephfs_metadata","create_time":"2023-05-22T16:03:37.371980+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"178","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_data","create_time":"2023-05-22T16:03:37.756887+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"178","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9e7ff854-b25f-4cbb-a5c2-1799d9c53d2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":37021},{"type":"v1","addr":"172.21.15.16:6825","nonce":37021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":37021},{"type":"v1","addr":"172.21.15.16:6827","nonce":37021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":37021},{"type":"v1","addr":"172.21.15.16:6831","nonce":37021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":37021},{"type":"v1","addr":"172.21.15.16:6829","nonce":37021}]},"public_addr":"172.21.15.16:6825/37021","cluster_addr":"172.21.15.16:6827/37021","heartbeat_back_addr":"172.21.15.16:6831/37021","heartbeat_front_addr":"172.21.15.16:6829/37021","state":["exists","up"]},{"osd":1,"uuid":"4702fbc9-a927-4565-aca5-41dc69cc8264","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":37008},{"type":"v1","addr":"172.21.15.16:6801","nonce":37008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":37008},{"type":"v1","addr":"172.21.15.16:6803","nonce":37008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":37008},{"type":"v1","addr":"172.21.15.16:6807","nonce":37008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":37008},{"type":"v1","addr":"172.21.15.16:6805","nonce":37008}]},"public_addr":"172.21.15.16:6801/37008","cluster_addr":"172.21.15.16:6803/37008","heartbeat_back_addr":"172.21.15.16:6807/37008","heartbeat_front_addr":"172.21.15.16:6805/37008","state":["exists","up"]},{"osd":2,"uuid":"1e7bba46-4eba-47ef-bb81-5e2d371638ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":37009},{"type":"v1","addr":"172.21.15.16:6809","nonce":37009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":37009},{"type":"v1","addr":"172.21.15.16:6811","nonce":37009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":37009},{"type":"v1","addr":"172.21.15.16:6815","nonce":37009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":37009},{"type":"v1","addr":"172.21.15.16:6813","nonce":37009}]},"public_addr":"172.21.15.16:6809/37009","cluster_addr":"172.21.15.16:6811/37009","heartbeat_back_addr":"172.21.15.16:6815/37009","heartbeat_front_addr":"172.21.15.16:6813/37009","state":["exists","up"]},{"osd":3,"uuid":"383cacc0-5a88-4904-990e-498ca59ef48a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":37020},{"type":"v1","addr":"172.21.15.16:6817","nonce":37020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":37020},{"type":"v1","addr":"172.21.15.16:6819","nonce":37020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":37020},{"type":"v1","addr":"172.21.15.16:6823","nonce":37020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":37020},{"type":"v1","addr":"172.21.15.16:6821","nonce":37020}]},"public_addr":"172.21.15.16:6817/37020","cluster_addr":"172.21.15.16:6819/37020","heartbeat_back_addr":"172.21.15.16:6823/37020","heartbeat_front_addr":"172.21.15.16:6821/37020","state":["exists","up"]},{"osd":4,"uuid":"970500ef-95b7-42c1-b080-eed4fae7013d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":37353},{"type":"v1","addr":"172.21.15.62:6817","nonce":37353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6819","nonce":37353},{"type":"v1","addr":"172.21.15.62:6822","nonce":37353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37353},{"type":"v1","addr":"172.21.15.62:6831","nonce":37353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37353},{"type":"v1","addr":"172.21.15.62:6826","nonce":37353}]},"public_addr":"172.21.15.62:6817/37353","cluster_addr":"172.21.15.62:6822/37353","heartbeat_back_addr":"172.21.15.62:6831/37353","heartbeat_front_addr":"172.21.15.62:6826/37353","state":["exists","up"]},{"osd":5,"uuid":"022d67db-d4d8-445b-b1b9-8b446a00f5c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":37352},{"type":"v1","addr":"172.21.15.62:6801","nonce":37352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":37352},{"type":"v1","addr":"172.21.15.62:6803","nonce":37352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":37352},{"type":"v1","addr":"172.21.15.62:6807","nonce":37352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":37352},{"type":"v1","addr":"172.21.15.62:6805","nonce":37352}]},"public_addr":"172.21.15.62:6801/37352","cluster_addr":"172.21.15.62:6803/37352","heartbeat_back_addr":"172.21.15.62:6807/37352","heartbeat_front_addr":"172.21.15.62:6805/37352","state":["exists","up"]},{"osd":6,"uuid":"42bd372b-6688-4a1b-8407-e16e8c266e33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":37354},{"type":"v1","addr":"172.21.15.62:6820","nonce":37354}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":37354},{"type":"v1","addr":"172.21.15.62:6823","nonce":37354}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6829","nonce":37354},{"type":"v1","addr":"172.21.15.62:6830","nonce":37354}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6825","nonce":37354},{"type":"v1","addr":"172.21.15.62:6827","nonce":37354}]},"public_addr":"172.21.15.62:6820/37354","cluster_addr":"172.21.15.62:6823/37354","heartbeat_back_addr":"172.21.15.62:6830/37354","heartbeat_front_addr":"172.21.15.62:6827/37354","state":["exists","up"]},{"osd":7,"uuid":"ac70b2a3-f41b-4bf2-b227-7f2cbb55426d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37355},{"type":"v1","addr":"172.21.15.62:6809","nonce":37355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37355},{"type":"v1","addr":"172.21.15.62:6811","nonce":37355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":37355},{"type":"v1","addr":"172.21.15.62:6815","nonce":37355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":37355},{"type":"v1","addr":"172.21.15.62:6813","nonce":37355}]},"public_addr":"172.21.15.62:6809/37355","cluster_addr":"172.21.15.62:6811/37355","heartbeat_back_addr":"172.21.15.62:6815/37355","heartbeat_front_addr":"172.21.15.62:6813/37355","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.368896+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.107273+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.276888+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.287502+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.124791+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.251937+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.255679+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.232797+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":17,"snaps":[{"begin":2,"length":2}]},{"pool":21,"snaps":[{"begin":25,"length":9}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-05-22T16:03:39.441 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:39.442+0000 7f409cc73640 1 -- 172.21.15.16:0/3225385920 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f40780654a0 msgr2=0x7f4078067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:39.442 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:39.442+0000 7f409cc73640 1 --2- 172.21.15.16:0/3225385920 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f40780654a0 0x7f4078067950 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f408c006e90 tx=0x7f408c008040 comp rx=0 tx=0).stop 2023-05-22T16:03:39.442 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:39.442+0000 7f409cc73640 1 -- 172.21.15.16:0/3225385920 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4080004d10 msgr2=0x7f4098147440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:39.442 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:39.442+0000 7f409cc73640 1 --2- 172.21.15.16:0/3225385920 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4080004d10 0x7f4098147440 secure :-1 s=READY pgs=2248 cs=0 l=1 rev1=1 crypto rx=0x7f4088010950 tx=0x7f4088010e10 comp rx=0 tx=0).stop 2023-05-22T16:03:39.442 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:39.442+0000 7f409cc73640 1 -- 172.21.15.16:0/3225385920 shutdown_connections 2023-05-22T16:03:39.442 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:39.442+0000 7f409cc73640 1 --2- 172.21.15.16:0/3225385920 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4098154250 0x7f40981546d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:39.442 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:39.442+0000 7f409cc73640 1 --2- 172.21.15.16:0/3225385920 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f409814f400 0x7f40980c1720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:39.443 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:39.442+0000 7f409cc73640 1 --2- 172.21.15.16:0/3225385920 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f40780654a0 0x7f4078067950 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:39.443 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:39.442+0000 7f409cc73640 1 --2- 172.21.15.16:0/3225385920 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4080004d10 0x7f4098147440 unknown :-1 s=CLOSED pgs=2248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:39.443 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:39.442+0000 7f409cc73640 1 -- 172.21.15.16:0/3225385920 >> 172.21.15.16:0/3225385920 conn(0x7f4098069720 msgr2=0x7f409815a2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:03:39.443 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:39.442+0000 7f409cc73640 1 -- 172.21.15.16:0/3225385920 shutdown_connections 2023-05-22T16:03:39.443 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:39.442+0000 7f409cc73640 1 -- 172.21.15.16:0/3225385920 wait complete. 2023-05-22T16:03:39.457 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-05-22T16:03:39.589 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:39.586+0000 7f5ef43e8640 1 Processor -- start 2023-05-22T16:03:39.589 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:39.586+0000 7f5ef43e8640 1 -- start start 2023-05-22T16:03:39.590 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:39.586+0000 7f5ef43e8640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5eec159d40 0x7f5eec0cb260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:39.590 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:39.586+0000 7f5ef43e8640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f5eec06b6c0 0x7f5eec06bb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:39.590 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:39.586+0000 7f5ef43e8640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5eec070c30 con 0x7f5eec159970 2023-05-22T16:03:39.591 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:39.586+0000 7f5ef43e8640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5eec15dd90 con 0x7f5eec06b6c0 2023-05-22T16:03:39.591 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:39.586+0000 7f5ef43e8640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5eec15a190 con 0x7f5eec159d40 2023-05-22T16:03:39.591 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:39.590+0000 7f5ef2985640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f5eec06b6c0 0x7f5eec06bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:39.591 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:39.590+0000 7f5ef2985640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f5eec06b6c0 0x7f5eec06bb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48930/0 (socket says 172.21.15.16:48930) 2023-05-22T16:03:39.592 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:39.590+0000 7f5ef2985640 1 -- 172.21.15.16:0/492225421 learned_addr learned my addr 172.21.15.16:0/492225421 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:03:39.592 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:39.590+0000 7f5ef1983640 1 --2- 172.21.15.16:0/492225421 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5eec159d40 0x7f5eec0cb260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:39.592 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:39.590+0000 7f5ef2985640 1 -- 172.21.15.16:0/492225421 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5eec159d40 msgr2=0x7f5eec0cb260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:03:39.593 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:39.590+0000 7f5ef2985640 1 --2- 172.21.15.16:0/492225421 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5eec159d40 0x7f5eec0cb260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:39.593 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:39.590+0000 7f5ef2985640 1 -- 172.21.15.16:0/492225421 >> v1:172.21.15.62:6789/0 conn(0x7f5eec159970 legacy=0x7f5eec15d680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:03:39.593 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:39.590+0000 7f5ef2985640 1 -- 172.21.15.16:0/492225421 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5eec06c060 con 0x7f5eec06b6c0 2023-05-22T16:03:39.593 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:39.590+0000 7f5ef1983640 1 --2- 172.21.15.16:0/492225421 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5eec159d40 0x7f5eec0cb260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T16:03:39.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:39.590+0000 7f5ef2985640 1 --2- 172.21.15.16:0/492225421 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5eec06b6c0 0x7f5eec06bb20 secure :-1 s=READY pgs=2249 cs=0 l=1 rev1=1 crypto rx=0x7f5ee4009e50 tx=0x7f5ee401c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cbd17f5b1ebdc2dc server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:03:39.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:39.590+0000 7f5ef1182640 1 -- 172.21.15.16:0/492225421 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ee40a56a0 con 0x7f5eec06b6c0 2023-05-22T16:03:39.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:39.590+0000 7f5ef1182640 1 -- 172.21.15.16:0/492225421 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ee40a5800 con 0x7f5eec06b6c0 2023-05-22T16:03:39.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:39.590+0000 7f5ef1182640 1 -- 172.21.15.16:0/492225421 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ee40a5b90 con 0x7f5eec06b6c0 2023-05-22T16:03:39.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:39.590+0000 7f5ef43e8640 1 -- 172.21.15.16:0/492225421 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5eec06b6c0 msgr2=0x7f5eec06bb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:39.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:39.590+0000 7f5ef43e8640 1 --2- 172.21.15.16:0/492225421 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5eec06b6c0 0x7f5eec06bb20 secure :-1 s=READY pgs=2249 cs=0 l=1 rev1=1 crypto rx=0x7f5ee4009e50 tx=0x7f5ee401c910 comp rx=0 tx=0).stop 2023-05-22T16:03:39.600 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:39.590+0000 7f5ef43e8640 1 -- 172.21.15.16:0/492225421 shutdown_connections 2023-05-22T16:03:39.600 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:39.590+0000 7f5ef43e8640 1 --2- 172.21.15.16:0/492225421 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5eec06b6c0 0x7f5eec06bb20 unknown :-1 s=CLOSED pgs=2249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:39.600 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:39.590+0000 7f5ef43e8640 1 --2- 172.21.15.16:0/492225421 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5eec159d40 0x7f5eec0cb260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:39.601 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:39.590+0000 7f5ef43e8640 1 -- 172.21.15.16:0/492225421 >> 172.21.15.16:0/492225421 conn(0x7f5eec069720 msgr2=0x7f5eec069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:03:39.601 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:39.590+0000 7f5ef43e8640 1 -- 172.21.15.16:0/492225421 shutdown_connections 2023-05-22T16:03:39.601 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:39.590+0000 7f5ef43e8640 1 -- 172.21.15.16:0/492225421 wait complete. 2023-05-22T16:03:39.601 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:39.594+0000 7f5ef43e8640 1 Processor -- start 2023-05-22T16:03:39.602 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:39.594+0000 7f5ef43e8640 1 -- start start 2023-05-22T16:03:39.602 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:39.594+0000 7f5ef43e8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5eec06b6c0 0x7f5eec147310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:39.602 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:39.594+0000 7f5ef43e8640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5eec159970 0x7f5eec0c1b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:39.603 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:39.594+0000 7f5ef43e8640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5eec159d40 0x7f5eec150c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:39.603 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:39.594+0000 7f5ef43e8640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5eec0b2340 con 0x7f5eec06b6c0 2023-05-22T16:03:39.603 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:39.594+0000 7f5ef43e8640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5eec0b25e0 con 0x7f5eec159970 2023-05-22T16:03:39.603 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:39.594+0000 7f5ef43e8640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5eec0b28c0 con 0x7f5eec159d40 2023-05-22T16:03:39.604 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:39.594+0000 7f5ef2184640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5eec06b6c0 0x7f5eec147310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:39.604 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:39.594+0000 7f5ef2184640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5eec06b6c0 0x7f5eec147310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48944/0 (socket says 172.21.15.16:48944) 2023-05-22T16:03:39.604 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:39.594+0000 7f5ef2184640 1 -- 172.21.15.16:0/9495508 learned_addr learned my addr 172.21.15.16:0/9495508 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:03:39.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:39.594+0000 7f5ef2985640 1 --2- 172.21.15.16:0/9495508 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5eec159d40 0x7f5eec150c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:39.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:39.594+0000 7f5ef2184640 1 -- 172.21.15.16:0/9495508 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5eec159d40 msgr2=0x7f5eec150c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:39.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:39.594+0000 7f5ef1983640 1 --2- 172.21.15.16:0/9495508 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5eec159970 0x7f5eec0c1b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:39.606 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:39.594+0000 7f5ef2184640 1 --2- 172.21.15.16:0/9495508 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5eec159d40 0x7f5eec150c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:39.606 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:39.594+0000 7f5ef2184640 1 -- 172.21.15.16:0/9495508 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5eec159970 msgr2=0x7f5eec0c1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:39.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:39.594+0000 7f5ef2184640 1 --2- 172.21.15.16:0/9495508 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5eec159970 0x7f5eec0c1b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:39.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:39.594+0000 7f5ef2184640 1 -- 172.21.15.16:0/9495508 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ee4009b30 con 0x7f5eec06b6c0 2023-05-22T16:03:39.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:39.594+0000 7f5ef2184640 1 --2- 172.21.15.16:0/9495508 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5eec06b6c0 0x7f5eec147310 secure :-1 s=READY pgs=2250 cs=0 l=1 rev1=1 crypto rx=0x7f5edc00c450 tx=0x7f5edc00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:03:39.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:39.594+0000 7f5ee2ffd640 1 -- 172.21.15.16:0/9495508 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5edc012830 con 0x7f5eec06b6c0 2023-05-22T16:03:39.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:39.594+0000 7f5ee2ffd640 1 -- 172.21.15.16:0/9495508 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5edc0129b0 con 0x7f5eec06b6c0 2023-05-22T16:03:39.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:39.594+0000 7f5ee2ffd640 1 -- 172.21.15.16:0/9495508 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5edc012d60 con 0x7f5eec06b6c0 2023-05-22T16:03:39.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:39.594+0000 7f5ef43e8640 1 -- 172.21.15.16:0/9495508 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5eec071700 con 0x7f5eec06b6c0 2023-05-22T16:03:39.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:39.594+0000 7f5ef43e8640 1 -- 172.21.15.16:0/9495508 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5eec071420 con 0x7f5eec06b6c0 2023-05-22T16:03:39.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:39.594+0000 7f5ef43e8640 1 -- 172.21.15.16:0/9495508 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5eec0cb7a0 con 0x7f5eec06b6c0 2023-05-22T16:03:39.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:39.598+0000 7f5ee2ffd640 1 -- 172.21.15.16:0/9495508 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 82) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f5edc01c460 con 0x7f5eec06b6c0 2023-05-22T16:03:39.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:39.598+0000 7f5ee2ffd640 1 --2- 172.21.15.16:0/9495508 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f5ed00654a0 0x7f5ed0067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:39.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:39.598+0000 7f5ee2ffd640 1 -- 172.21.15.16:0/9495508 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(178..178 src has 1..178) v4 ==== 6438+0+0 (secure 0 0 0) 0x7f5edc0829a0 con 0x7f5eec06b6c0 2023-05-22T16:03:39.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:39.598+0000 7f5ef1983640 1 --2- 172.21.15.16:0/9495508 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f5ed00654a0 0x7f5ed0067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:39.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:39.598+0000 7f5ef1983640 1 --2- 172.21.15.16:0/9495508 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f5ed00654a0 0x7f5ed0067950 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f5ee8006fd0 tx=0x7f5ee8008040 comp rx=0 tx=0).ready entity=mgr.10032 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:03:39.609 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:39.602+0000 7f5ee2ffd640 1 -- 172.21.15.16:0/9495508 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f5edc0503c0 con 0x7f5eec06b6c0 2023-05-22T16:03:39.792 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:39.790+0000 7f5ef43e8640 1 -- 172.21.15.16:0/9495508 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f5eec04dbe0 con 0x7f5eec06b6c0 2023-05-22T16:03:40.326 INFO:tasks.ceph.mds.d.smithi062.stderr:2023-05-22T16:03:40.321+0000 7f65d7ed4640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-05-22T16:03:40.807 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:40.806+0000 7f5ee2ffd640 1 -- 172.21.15.16:0/9495508 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v72) v1 ==== 118+0+0 (secure 0 0 0) 0x7f5edc0539e0 con 0x7f5eec06b6c0 2023-05-22T16:03:40.810 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:40.810+0000 7f5ef43e8640 1 -- 172.21.15.16:0/9495508 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f5ed00654a0 msgr2=0x7f5ed0067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:40.810 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:40.810+0000 7f5ef43e8640 1 --2- 172.21.15.16:0/9495508 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f5ed00654a0 0x7f5ed0067950 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f5ee8006fd0 tx=0x7f5ee8008040 comp rx=0 tx=0).stop 2023-05-22T16:03:40.810 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:40.810+0000 7f5ef43e8640 1 -- 172.21.15.16:0/9495508 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5eec06b6c0 msgr2=0x7f5eec147310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:40.810 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:40.810+0000 7f5ef43e8640 1 --2- 172.21.15.16:0/9495508 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5eec06b6c0 0x7f5eec147310 secure :-1 s=READY pgs=2250 cs=0 l=1 rev1=1 crypto rx=0x7f5edc00c450 tx=0x7f5edc00c910 comp rx=0 tx=0).stop 2023-05-22T16:03:40.811 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:40.810+0000 7f5ef43e8640 1 -- 172.21.15.16:0/9495508 shutdown_connections 2023-05-22T16:03:40.811 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:40.810+0000 7f5ef43e8640 1 --2- 172.21.15.16:0/9495508 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5eec159d40 0x7f5eec150c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:40.811 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:40.810+0000 7f5ef43e8640 1 --2- 172.21.15.16:0/9495508 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5eec159970 0x7f5eec0c1b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:40.811 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:40.810+0000 7f5ef43e8640 1 --2- 172.21.15.16:0/9495508 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f5ed00654a0 0x7f5ed0067950 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:40.811 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:40.810+0000 7f5ef43e8640 1 --2- 172.21.15.16:0/9495508 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5eec06b6c0 0x7f5eec147310 unknown :-1 s=CLOSED pgs=2250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:40.811 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:40.810+0000 7f5ef43e8640 1 -- 172.21.15.16:0/9495508 >> 172.21.15.16:0/9495508 conn(0x7f5eec069720 msgr2=0x7f5eec069af0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:03:40.811 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:40.810+0000 7f5ef43e8640 1 -- 172.21.15.16:0/9495508 shutdown_connections 2023-05-22T16:03:40.811 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:40.810+0000 7f5ef43e8640 1 -- 172.21.15.16:0/9495508 wait complete. 2023-05-22T16:03:40.833 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2023-05-22T16:03:40.834 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-05-22T16:03:40.965 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:40.966+0000 7fad0c582640 1 Processor -- start 2023-05-22T16:03:40.965 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:40.966+0000 7fad0c582640 1 -- start start 2023-05-22T16:03:40.966 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:40.966+0000 7fad0c582640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fad0415bca0 0x7fad040cda80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:40.966 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:40.966+0000 7fad0c582640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fad040d2480 0x7fad040d2900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:40.966 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:40.966+0000 7fad0c582640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fad040b5340 0x7fad040b5790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:40.967 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:40.966+0000 7fad0c582640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad040cd660 con 0x7fad0415bca0 2023-05-22T16:03:40.967 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:40.966+0000 7fad0c582640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad040d2e40 con 0x7fad040d2480 2023-05-22T16:03:40.967 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:40.966+0000 7fad0c582640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad040cccc0 con 0x7fad040b5340 2023-05-22T16:03:40.967 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:40.966+0000 7fad0a31e640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fad0415bca0 0x7fad040cda80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:40.967 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:40.966+0000 7fad0a31e640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fad0415bca0 0x7fad040cda80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48950/0 (socket says 172.21.15.16:48950) 2023-05-22T16:03:40.967 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:40.966+0000 7fad0a31e640 1 -- 172.21.15.16:0/3272762764 learned_addr learned my addr 172.21.15.16:0/3272762764 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:03:40.967 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:40.966+0000 7fad09b1d640 1 --2- 172.21.15.16:0/3272762764 >> v2:172.21.15.62:3300/0 conn(0x7fad040d2480 0x7fad040d2900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:40.968 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:40.966+0000 7fad0ab1f640 1 --2- 172.21.15.16:0/3272762764 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fad040b5340 0x7fad040b5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:40.968 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:40.966+0000 7fad0a31e640 1 -- 172.21.15.16:0/3272762764 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fad040b5340 msgr2=0x7fad040b5790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:03:40.968 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:40.966+0000 7fad0a31e640 1 --2- 172.21.15.16:0/3272762764 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fad040b5340 0x7fad040b5790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:40.968 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:40.966+0000 7fad0a31e640 1 -- 172.21.15.16:0/3272762764 >> v2:172.21.15.62:3300/0 conn(0x7fad040d2480 msgr2=0x7fad040d2900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:03:40.968 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:40.966+0000 7fad0a31e640 1 --2- 172.21.15.16:0/3272762764 >> v2:172.21.15.62:3300/0 conn(0x7fad040d2480 0x7fad040d2900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:40.968 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:40.966+0000 7fad0a31e640 1 -- 172.21.15.16:0/3272762764 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad040ccfa0 con 0x7fad0415bca0 2023-05-22T16:03:40.969 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:40.970+0000 7fad0a31e640 1 --2- 172.21.15.16:0/3272762764 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fad0415bca0 0x7fad040cda80 secure :-1 s=READY pgs=2251 cs=0 l=1 rev1=1 crypto rx=0x7fad04065e80 tx=0x7fad0001c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=338e2f79a9659dcd server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:03:40.969 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:40.970+0000 7fad0931c640 1 -- 172.21.15.16:0/3272762764 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad000aa610 con 0x7fad0415bca0 2023-05-22T16:03:40.970 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:40.970+0000 7fad0931c640 1 -- 172.21.15.16:0/3272762764 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad000a6040 con 0x7fad0415bca0 2023-05-22T16:03:40.970 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:40.970+0000 7fad0931c640 1 -- 172.21.15.16:0/3272762764 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad000aa990 con 0x7fad0415bca0 2023-05-22T16:03:40.970 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:40.970+0000 7fad0c582640 1 -- 172.21.15.16:0/3272762764 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fad0415bca0 msgr2=0x7fad040cda80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:40.970 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:40.970+0000 7fad0c582640 1 --2- 172.21.15.16:0/3272762764 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fad0415bca0 0x7fad040cda80 secure :-1 s=READY pgs=2251 cs=0 l=1 rev1=1 crypto rx=0x7fad04065e80 tx=0x7fad0001c4d0 comp rx=0 tx=0).stop 2023-05-22T16:03:40.970 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:40.970+0000 7fad0c582640 1 -- 172.21.15.16:0/3272762764 shutdown_connections 2023-05-22T16:03:40.970 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:40.970+0000 7fad0c582640 1 --2- 172.21.15.16:0/3272762764 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fad040b5340 0x7fad040b5790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:40.970 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:40.970+0000 7fad0c582640 1 --2- 172.21.15.16:0/3272762764 >> v2:172.21.15.62:3300/0 conn(0x7fad040d2480 0x7fad040d2900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:40.971 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:40.970+0000 7fad0c582640 1 --2- 172.21.15.16:0/3272762764 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fad0415bca0 0x7fad040cda80 unknown :-1 s=CLOSED pgs=2251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:40.971 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:40.970+0000 7fad0c582640 1 -- 172.21.15.16:0/3272762764 >> 172.21.15.16:0/3272762764 conn(0x7fad040655c0 msgr2=0x7fad04070e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:03:40.971 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:40.970+0000 7fad0c582640 1 -- 172.21.15.16:0/3272762764 shutdown_connections 2023-05-22T16:03:40.971 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:40.970+0000 7fad0c582640 1 -- 172.21.15.16:0/3272762764 wait complete. 2023-05-22T16:03:40.973 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:40.970+0000 7fad0c582640 1 Processor -- start 2023-05-22T16:03:40.973 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:40.970+0000 7fad0c582640 1 -- start start 2023-05-22T16:03:40.973 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:40.970+0000 7fad0c582640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fad040b5340 0x7fad0415a960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:40.973 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:40.970+0000 7fad0c582640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fad040d2480 0x7fad0415cd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:40.973 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:40.970+0000 7fad0c582640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fad0415bca0 0x7fad04154360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:40.973 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:40.970+0000 7fad0c582640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad0415aea0 con 0x7fad0415bca0 2023-05-22T16:03:40.973 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:40.970+0000 7fad0c582640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad0415d250 con 0x7fad040d2480 2023-05-22T16:03:40.974 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:40.970+0000 7fad0c582640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad0415d470 con 0x7fad040b5340 2023-05-22T16:03:40.974 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:40.970+0000 7fad0ab1f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fad0415bca0 0x7fad04154360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:40.974 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:40.970+0000 7fad0ab1f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fad0415bca0 0x7fad04154360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48960/0 (socket says 172.21.15.16:48960) 2023-05-22T16:03:40.974 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:40.970+0000 7fad0ab1f640 1 -- 172.21.15.16:0/3206862082 learned_addr learned my addr 172.21.15.16:0/3206862082 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:03:40.974 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:40.970+0000 7fad0a31e640 1 --2- 172.21.15.16:0/3206862082 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fad040b5340 0x7fad0415a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:40.974 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:40.970+0000 7fad09b1d640 1 --2- 172.21.15.16:0/3206862082 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fad040d2480 0x7fad0415cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:40.974 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:40.970+0000 7fad0ab1f640 1 -- 172.21.15.16:0/3206862082 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fad040b5340 msgr2=0x7fad0415a960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:40.975 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:40.970+0000 7fad0ab1f640 1 --2- 172.21.15.16:0/3206862082 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fad040b5340 0x7fad0415a960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:40.975 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:40.970+0000 7fad0ab1f640 1 -- 172.21.15.16:0/3206862082 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fad040d2480 msgr2=0x7fad0415cd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:40.975 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:40.970+0000 7fad0ab1f640 1 --2- 172.21.15.16:0/3206862082 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fad040d2480 0x7fad0415cd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:40.975 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:40.970+0000 7fad0ab1f640 1 -- 172.21.15.16:0/3206862082 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad000094d0 con 0x7fad0415bca0 2023-05-22T16:03:40.975 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:40.970+0000 7fad0a31e640 1 --2- 172.21.15.16:0/3206862082 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fad040b5340 0x7fad0415a960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T16:03:40.975 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:40.974+0000 7fad0ab1f640 1 --2- 172.21.15.16:0/3206862082 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fad0415bca0 0x7fad04154360 secure :-1 s=READY pgs=2252 cs=0 l=1 rev1=1 crypto rx=0x7facfc00ccf0 tx=0x7facfc00a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:03:40.975 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:40.974+0000 7facfaffd640 1 -- 172.21.15.16:0/3206862082 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facfc003ba0 con 0x7fad0415bca0 2023-05-22T16:03:40.975 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:40.974+0000 7facfaffd640 1 -- 172.21.15.16:0/3206862082 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facfc003d00 con 0x7fad0415bca0 2023-05-22T16:03:40.976 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:40.974+0000 7fad0c582640 1 -- 172.21.15.16:0/3206862082 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad040cdfc0 con 0x7fad0415bca0 2023-05-22T16:03:40.976 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:40.974+0000 7facfaffd640 1 -- 172.21.15.16:0/3206862082 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facfc004240 con 0x7fad0415bca0 2023-05-22T16:03:40.976 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:40.974+0000 7fad0c582640 1 -- 172.21.15.16:0/3206862082 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad040b5cd0 con 0x7fad0415bca0 2023-05-22T16:03:40.977 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:40.974+0000 7fad0c582640 1 -- 172.21.15.16:0/3206862082 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad04160360 con 0x7fad0415bca0 2023-05-22T16:03:40.977 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:40.974+0000 7facfaffd640 1 -- 172.21.15.16:0/3206862082 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 82) v1 ==== 84332+0+0 (secure 0 0 0) 0x7facfc0043a0 con 0x7fad0415bca0 2023-05-22T16:03:40.977 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:40.974+0000 7facfaffd640 1 --2- 172.21.15.16:0/3206862082 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7face80654a0 0x7face8067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:40.977 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:40.974+0000 7fad0a31e640 1 --2- 172.21.15.16:0/3206862082 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7face80654a0 0x7face8067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:40.977 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:40.974+0000 7facfaffd640 1 -- 172.21.15.16:0/3206862082 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(179..179 src has 1..179) v4 ==== 6438+0+0 (secure 0 0 0) 0x7facfc012070 con 0x7fad0415bca0 2023-05-22T16:03:40.977 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:40.974+0000 7fad0a31e640 1 --2- 172.21.15.16:0/3206862082 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7face80654a0 0x7face8067950 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fad000ae000 tx=0x7fad000a7000 comp rx=0 tx=0).ready entity=mgr.10032 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:03:40.979 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:40.978+0000 7facfaffd640 1 -- 172.21.15.16:0/3206862082 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7facfc050fb0 con 0x7fad0415bca0 2023-05-22T16:03:41.169 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:41.170+0000 7fad0c582640 1 -- 172.21.15.16:0/3206862082 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fad0404dbe0 con 0x7fad0415bca0 2023-05-22T16:03:42.135 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.134+0000 7facfaffd640 1 -- 172.21.15.16:0/3206862082 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v73) v1 ==== 122+0+0 (secure 0 0 0) 0x7facfc0545d0 con 0x7fad0415bca0 2023-05-22T16:03:42.139 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.138+0000 7fad0c582640 1 -- 172.21.15.16:0/3206862082 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7face80654a0 msgr2=0x7face8067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:42.139 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.138+0000 7fad0c582640 1 --2- 172.21.15.16:0/3206862082 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7face80654a0 0x7face8067950 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fad000ae000 tx=0x7fad000a7000 comp rx=0 tx=0).stop 2023-05-22T16:03:42.139 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.138+0000 7fad0c582640 1 -- 172.21.15.16:0/3206862082 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fad0415bca0 msgr2=0x7fad04154360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:42.139 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.138+0000 7fad0c582640 1 --2- 172.21.15.16:0/3206862082 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fad0415bca0 0x7fad04154360 secure :-1 s=READY pgs=2252 cs=0 l=1 rev1=1 crypto rx=0x7facfc00ccf0 tx=0x7facfc00a660 comp rx=0 tx=0).stop 2023-05-22T16:03:42.139 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.138+0000 7fad0c582640 1 -- 172.21.15.16:0/3206862082 shutdown_connections 2023-05-22T16:03:42.140 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.138+0000 7fad0c582640 1 --2- 172.21.15.16:0/3206862082 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7face80654a0 0x7face8067950 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:42.140 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.138+0000 7fad0c582640 1 --2- 172.21.15.16:0/3206862082 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fad0415bca0 0x7fad04154360 unknown :-1 s=CLOSED pgs=2252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:42.140 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.138+0000 7fad0c582640 1 --2- 172.21.15.16:0/3206862082 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fad040d2480 0x7fad0415cd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:42.140 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.138+0000 7fad0c582640 1 --2- 172.21.15.16:0/3206862082 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fad040b5340 0x7fad0415a960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:42.140 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.138+0000 7fad0c582640 1 -- 172.21.15.16:0/3206862082 >> 172.21.15.16:0/3206862082 conn(0x7fad040655c0 msgr2=0x7fad040d0500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:03:42.140 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.138+0000 7fad0c582640 1 -- 172.21.15.16:0/3206862082 shutdown_connections 2023-05-22T16:03:42.140 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.138+0000 7fad0c582640 1 -- 172.21.15.16:0/3206862082 wait complete. 2023-05-22T16:03:42.162 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2023-05-22T16:03:42.163 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-05-22T16:03:42.292 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.294+0000 7f7fff768640 1 Processor -- start 2023-05-22T16:03:42.293 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.294+0000 7f7fff768640 1 -- start start 2023-05-22T16:03:42.293 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.294+0000 7f7fff768640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f7ff8159cd0 0x7f7ff815a0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:42.294 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.294+0000 7f7fff768640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7ff80cc6f0 0x7f7ff80cfe30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:42.294 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.294+0000 7f7fff768640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ff80cf9a0 con 0x7f7ff80b2880 2023-05-22T16:03:42.294 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.294+0000 7f7fff768640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ff815aba0 con 0x7f7ff8159cd0 2023-05-22T16:03:42.294 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.294+0000 7f7fff768640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ff815ae80 con 0x7f7ff80cc6f0 2023-05-22T16:03:42.294 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.294+0000 7f7ffcd03640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f7ff80b2880 0x7f7ff80cbfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:53190/0 (socket says 172.21.15.16:53190) 2023-05-22T16:03:42.294 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.294+0000 7f7ffcd03640 1 -- 172.21.15.16:0/393204582 learned_addr learned my addr 172.21.15.16:0/393204582 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T16:03:42.294 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.294+0000 7f7ffd504640 1 --2- 172.21.15.16:0/393204582 >> v2:172.21.15.62:3300/0 conn(0x7f7ff8159cd0 0x7f7ff815a0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:42.295 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.294+0000 7f7ffdd05640 1 --2- 172.21.15.16:0/393204582 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7ff80cc6f0 0x7f7ff80cfe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:42.296 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.294+0000 7f7ff7fff640 1 -- 172.21.15.16:0/393204582 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3426291120 0 0) 0x7f7ff80cf9a0 con 0x7f7ff80b2880 2023-05-22T16:03:42.297 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.294+0000 7f7ff7fff640 1 -- 172.21.15.16:0/393204582 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7fe0003530 con 0x7f7ff80b2880 2023-05-22T16:03:42.297 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.294+0000 7f7ff7fff640 1 -- 172.21.15.16:0/393204582 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2187069195 0 0) 0x7f7fe0003530 con 0x7f7ff80b2880 2023-05-22T16:03:42.297 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.294+0000 7f7ff7fff640 1 -- 172.21.15.16:0/393204582 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7ff80cc6f0 msgr2=0x7f7ff80cfe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:03:42.297 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.294+0000 7f7ff7fff640 1 --2- 172.21.15.16:0/393204582 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7ff80cc6f0 0x7f7ff80cfe30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:42.297 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.294+0000 7f7ff7fff640 1 -- 172.21.15.16:0/393204582 >> v2:172.21.15.62:3300/0 conn(0x7f7ff8159cd0 msgr2=0x7f7ff815a0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:03:42.297 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.294+0000 7f7ff7fff640 1 --2- 172.21.15.16:0/393204582 >> v2:172.21.15.62:3300/0 conn(0x7f7ff8159cd0 0x7f7ff815a0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:42.298 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.294+0000 7f7ff7fff640 1 -- 172.21.15.16:0/393204582 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ff815b160 con 0x7f7ff80b2880 2023-05-22T16:03:42.298 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.294+0000 7f7ff7fff640 1 -- 172.21.15.16:0/393204582 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7f7fec001680 con 0x7f7ff80b2880 2023-05-22T16:03:42.298 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.294+0000 7f7ff7fff640 1 -- 172.21.15.16:0/393204582 >> v1:172.21.15.16:6789/0 conn(0x7f7ff80b2880 legacy=0x7f7ff80cbfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:42.298 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.294+0000 7f7ff7fff640 1 --2- 172.21.15.16:0/393204582 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7fe0003980 0x7f7fe0003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:42.298 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.294+0000 7f7ff7fff640 1 --2- 172.21.15.16:0/393204582 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7fe00043b0 0x7f7fe0004870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:42.298 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.294+0000 7f7ff7fff640 1 --2- 172.21.15.16:0/393204582 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7ff80cc6f0 0x7f7fe0004f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:42.298 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.294+0000 7f7ff7fff640 1 -- 172.21.15.16:0/393204582 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ff815aba0 con 0x7f7fe00043b0 2023-05-22T16:03:42.299 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.294+0000 7f7ff7fff640 1 -- 172.21.15.16:0/393204582 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ff815ae80 con 0x7f7fe0003980 2023-05-22T16:03:42.299 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.294+0000 7f7ff7fff640 1 -- 172.21.15.16:0/393204582 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7fe000b990 con 0x7f7ff80cc6f0 2023-05-22T16:03:42.299 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.294+0000 7f7ffcd03640 1 --2- 172.21.15.16:0/393204582 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7fe00043b0 0x7f7fe0004870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:42.299 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.294+0000 7f7ffdd05640 1 --2- 172.21.15.16:0/393204582 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7ff80cc6f0 0x7f7fe0004f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:42.299 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.294+0000 7f7ffcd03640 1 -- 172.21.15.16:0/393204582 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7ff80cc6f0 msgr2=0x7f7fe0004f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:03:42.299 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.294+0000 7f7ffcd03640 1 --2- 172.21.15.16:0/393204582 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7ff80cc6f0 0x7f7fe0004f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:42.299 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.294+0000 7f7ffd504640 1 --2- 172.21.15.16:0/393204582 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7fe0003980 0x7f7fe0003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:42.299 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.294+0000 7f7ffcd03640 1 -- 172.21.15.16:0/393204582 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7fe0003980 msgr2=0x7f7fe0003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:03:42.300 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.294+0000 7f7ffcd03640 1 --2- 172.21.15.16:0/393204582 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7fe0003980 0x7f7fe0003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:42.300 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.294+0000 7f7ffcd03640 1 -- 172.21.15.16:0/393204582 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fe0003530 con 0x7f7fe00043b0 2023-05-22T16:03:42.300 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.294+0000 7f7ffd504640 1 --2- 172.21.15.16:0/393204582 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7fe0003980 0x7f7fe0003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T16:03:42.302 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.298+0000 7f7ffcd03640 1 --2- 172.21.15.16:0/393204582 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7fe00043b0 0x7f7fe0004870 secure :-1 s=READY pgs=2254 cs=0 l=1 rev1=1 crypto rx=0x7f7fec00c290 tx=0x7f7fec01f430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb7f757bebfa3df4 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:03:42.302 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.298+0000 7f7ff7fff640 1 -- 172.21.15.16:0/393204582 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fec005ae0 con 0x7f7fe00043b0 2023-05-22T16:03:42.303 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.298+0000 7f7ff7fff640 1 -- 172.21.15.16:0/393204582 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fec0ab070 con 0x7f7fe00043b0 2023-05-22T16:03:42.303 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.298+0000 7f7ff7fff640 1 -- 172.21.15.16:0/393204582 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fec005e80 con 0x7f7fe00043b0 2023-05-22T16:03:42.303 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.298+0000 7f7fff768640 1 -- 172.21.15.16:0/393204582 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7fe00043b0 msgr2=0x7f7fe0004870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:42.303 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.298+0000 7f7fff768640 1 --2- 172.21.15.16:0/393204582 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7fe00043b0 0x7f7fe0004870 secure :-1 s=READY pgs=2254 cs=0 l=1 rev1=1 crypto rx=0x7f7fec00c290 tx=0x7f7fec01f430 comp rx=0 tx=0).stop 2023-05-22T16:03:42.303 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.298+0000 7f7ffdd05640 1 -- 172.21.15.16:0/393204582 reap_dead start 2023-05-22T16:03:42.303 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.298+0000 7f7fff768640 1 -- 172.21.15.16:0/393204582 shutdown_connections 2023-05-22T16:03:42.303 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.298+0000 7f7fff768640 1 -- 172.21.15.16:0/393204582 >> 172.21.15.16:0/393204582 conn(0x7f7ff806f5c0 msgr2=0x7f7ff806f9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:03:42.303 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.298+0000 7f7fff768640 1 -- 172.21.15.16:0/393204582 shutdown_connections 2023-05-22T16:03:42.304 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.298+0000 7f7fff768640 1 -- 172.21.15.16:0/393204582 wait complete. 2023-05-22T16:03:42.304 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.298+0000 7f7fff768640 1 Processor -- start 2023-05-22T16:03:42.304 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.298+0000 7f7fff768640 1 -- start start 2023-05-22T16:03:42.304 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.302+0000 7f7fff768640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ff8151ce0 0x7f7ff80b3090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:42.304 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.302+0000 7f7fff768640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7ff81520b0 0x7f7ff80c7d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:42.304 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.302+0000 7f7fff768640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7ff80b0590 0x7f7ff80b0a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:42.304 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.302+0000 7f7fff768640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ff80b0f70 con 0x7f7ff8151ce0 2023-05-22T16:03:42.305 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.302+0000 7f7fff768640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ff806a830 con 0x7f7ff80b0590 2023-05-22T16:03:42.305 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.302+0000 7f7fff768640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ff806aa80 con 0x7f7ff81520b0 2023-05-22T16:03:42.305 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.302+0000 7f7ffd504640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ff8151ce0 0x7f7ff80b3090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:42.305 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.302+0000 7f7ffd504640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ff8151ce0 0x7f7ff80b3090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48972/0 (socket says 172.21.15.16:48972) 2023-05-22T16:03:42.305 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.302+0000 7f7ffd504640 1 -- 172.21.15.16:0/102658688 learned_addr learned my addr 172.21.15.16:0/102658688 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:03:42.305 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.302+0000 7f7ffcd03640 1 --2- 172.21.15.16:0/102658688 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7ff81520b0 0x7f7ff80c7d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:42.305 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.302+0000 7f7ffdd05640 1 --2- 172.21.15.16:0/102658688 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7ff80b0590 0x7f7ff80b0a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:42.305 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.302+0000 7f7ffd504640 1 -- 172.21.15.16:0/102658688 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7ff81520b0 msgr2=0x7f7ff80c7d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:42.306 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.302+0000 7f7ffd504640 1 --2- 172.21.15.16:0/102658688 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7ff81520b0 0x7f7ff80c7d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:42.306 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.302+0000 7f7ffd504640 1 -- 172.21.15.16:0/102658688 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7ff80b0590 msgr2=0x7f7ff80b0a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:42.306 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.302+0000 7f7ffd504640 1 --2- 172.21.15.16:0/102658688 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7ff80b0590 0x7f7ff80b0a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:42.306 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.302+0000 7f7ffd504640 1 -- 172.21.15.16:0/102658688 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ff815a5e0 con 0x7f7ff8151ce0 2023-05-22T16:03:42.306 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.302+0000 7f7ffd504640 1 --2- 172.21.15.16:0/102658688 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ff8151ce0 0x7f7ff80b3090 secure :-1 s=READY pgs=2255 cs=0 l=1 rev1=1 crypto rx=0x7f7fe800f3d0 tx=0x7f7fe800f890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:03:42.306 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.302+0000 7f7ff5ffb640 1 -- 172.21.15.16:0/102658688 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fe801a070 con 0x7f7ff8151ce0 2023-05-22T16:03:42.306 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.302+0000 7f7ff5ffb640 1 -- 172.21.15.16:0/102658688 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fe80039a0 con 0x7f7ff8151ce0 2023-05-22T16:03:42.307 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.302+0000 7f7fff768640 1 -- 172.21.15.16:0/102658688 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ff80cf630 con 0x7f7ff8151ce0 2023-05-22T16:03:42.307 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.302+0000 7f7ff5ffb640 1 -- 172.21.15.16:0/102658688 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fe8003d10 con 0x7f7ff8151ce0 2023-05-22T16:03:42.307 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.302+0000 7f7fff768640 1 -- 172.21.15.16:0/102658688 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ff815a770 con 0x7f7ff8151ce0 2023-05-22T16:03:42.307 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.302+0000 7f7fff768640 1 -- 172.21.15.16:0/102658688 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ff804dbe0 con 0x7f7ff8151ce0 2023-05-22T16:03:42.308 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.302+0000 7f7ff5ffb640 1 -- 172.21.15.16:0/102658688 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 82) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f7fe8004020 con 0x7f7ff8151ce0 2023-05-22T16:03:42.308 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.302+0000 7f7ff5ffb640 1 --2- 172.21.15.16:0/102658688 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f7fe40654d0 0x7f7fe4067980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:42.308 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.302+0000 7f7ff5ffb640 1 -- 172.21.15.16:0/102658688 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(179..179 src has 1..179) v4 ==== 6438+0+0 (secure 0 0 0) 0x7f7fe8084690 con 0x7f7ff8151ce0 2023-05-22T16:03:42.308 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.302+0000 7f7ffcd03640 1 --2- 172.21.15.16:0/102658688 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f7fe40654d0 0x7f7fe4067980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:42.308 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.302+0000 7f7ffcd03640 1 --2- 172.21.15.16:0/102658688 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f7fe40654d0 0x7f7fe4067980 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f7fec0ae000 tx=0x7f7fec007040 comp rx=0 tx=0).ready entity=mgr.10032 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:03:42.309 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.306+0000 7f7ff5ffb640 1 -- 172.21.15.16:0/102658688 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f7fe8051fe0 con 0x7f7ff8151ce0 2023-05-22T16:03:42.495 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.494+0000 7f7fff768640 1 -- 172.21.15.16:0/102658688 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7ff80b1800 con 0x7f7ff8151ce0 2023-05-22T16:03:42.496 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.494+0000 7f7ff5ffb640 1 -- 172.21.15.16:0/102658688 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 73 v73) v1 ==== 94+0+4657 (secure 0 0 0) 0x7f7fe8055600 con 0x7f7ff8151ce0 2023-05-22T16:03:42.496 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T16:03:42.496 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":73,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7682,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/2815032143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":2815032143},{"type":"v1","addr":"172.21.15.62:6835","nonce":2815032143}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":69},{"gid":10239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6836/1961058261","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1961058261},{"type":"v1","addr":"172.21.15.16:6836","nonce":1961058261}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":69},{"gid":10242,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/1453687184","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6835","nonce":1453687184},{"type":"v1","addr":"172.21.15.16:6837","nonce":1453687184}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":73,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-05-22T16:03:38.779645+0000","modified":"2023-05-22T16:03:42.129478+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10251":{"gid":10251,"name":"d","rank":0,"incarnation":70,"state":"up:active","state_seq":3,"addr":"172.21.15.62:6833/2486335217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":2486335217},{"type":"v1","addr":"172.21.15.62:6833","nonce":2486335217}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[23],"metadata_pool":22,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2023-05-22T16:03:42.497 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 73 2023-05-22T16:03:42.499 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.498+0000 7f7fff768640 1 -- 172.21.15.16:0/102658688 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f7fe40654d0 msgr2=0x7f7fe4067980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:42.499 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.498+0000 7f7fff768640 1 --2- 172.21.15.16:0/102658688 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f7fe40654d0 0x7f7fe4067980 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f7fec0ae000 tx=0x7f7fec007040 comp rx=0 tx=0).stop 2023-05-22T16:03:42.499 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.498+0000 7f7fff768640 1 -- 172.21.15.16:0/102658688 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ff8151ce0 msgr2=0x7f7ff80b3090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:42.499 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.498+0000 7f7fff768640 1 --2- 172.21.15.16:0/102658688 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ff8151ce0 0x7f7ff80b3090 secure :-1 s=READY pgs=2255 cs=0 l=1 rev1=1 crypto rx=0x7f7fe800f3d0 tx=0x7f7fe800f890 comp rx=0 tx=0).stop 2023-05-22T16:03:42.500 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.498+0000 7f7fff768640 1 -- 172.21.15.16:0/102658688 shutdown_connections 2023-05-22T16:03:42.500 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.498+0000 7f7fff768640 1 --2- 172.21.15.16:0/102658688 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7ff80b0590 0x7f7ff80b0a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:42.500 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.498+0000 7f7fff768640 1 --2- 172.21.15.16:0/102658688 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7ff81520b0 0x7f7ff80c7d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:42.501 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.498+0000 7f7fff768640 1 --2- 172.21.15.16:0/102658688 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f7fe40654d0 0x7f7fe4067980 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:42.501 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.498+0000 7f7fff768640 1 --2- 172.21.15.16:0/102658688 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ff8151ce0 0x7f7ff80b3090 unknown :-1 s=CLOSED pgs=2255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:42.501 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.498+0000 7f7fff768640 1 -- 172.21.15.16:0/102658688 >> 172.21.15.16:0/102658688 conn(0x7f7ff806f5c0 msgr2=0x7f7ff80d0af0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:03:42.501 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.498+0000 7f7fff768640 1 -- 172.21.15.16:0/102658688 shutdown_connections 2023-05-22T16:03:42.502 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.498+0000 7f7fff768640 1 -- 172.21.15.16:0/102658688 wait complete. 2023-05-22T16:03:42.516 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-05-22T16:03:42.645 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.646+0000 7f411b115640 1 Processor -- start 2023-05-22T16:03:42.645 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.646+0000 7f411b115640 1 -- start start 2023-05-22T16:03:42.645 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.646+0000 7f411b115640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f41140ccab0 0x7f411415fe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:42.646 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.646+0000 7f411b115640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f41140d0410 0x7f41140d0830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:42.646 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.646+0000 7f411b115640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41140ccef0 con 0x7f41140b7dd0 2023-05-22T16:03:42.646 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.646+0000 7f411b115640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41140d3360 con 0x7f41140ccab0 2023-05-22T16:03:42.646 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.646+0000 7f411b115640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41140cd170 con 0x7f41140d0410 2023-05-22T16:03:42.647 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.646+0000 7f41196b2640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f41140d0410 0x7f41140d0830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:42.647 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.646+0000 7f410bfff640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f41140ccab0 0x7f411415fe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:42.647 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.646+0000 7f410bfff640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f41140ccab0 0x7f411415fe10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.16:59244/0 (socket says 172.21.15.16:59244) 2023-05-22T16:03:42.647 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.646+0000 7f41196b2640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f41140d0410 0x7f41140d0830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3301/0 says I am v2:172.21.15.16:43114/0 (socket says 172.21.15.16:43114) 2023-05-22T16:03:42.648 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.646+0000 7f410bfff640 1 -- 172.21.15.16:0/1089973563 learned_addr learned my addr 172.21.15.16:0/1089973563 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:03:42.648 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.646+0000 7f41196b2640 1 -- 172.21.15.16:0/1089973563 >> v2:172.21.15.62:3300/0 conn(0x7f41140ccab0 msgr2=0x7f411415fe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:03:42.648 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.646+0000 7f41196b2640 1 --2- 172.21.15.16:0/1089973563 >> v2:172.21.15.62:3300/0 conn(0x7f41140ccab0 0x7f411415fe10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:42.649 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.646+0000 7f41196b2640 1 -- 172.21.15.16:0/1089973563 >> v1:172.21.15.62:6789/0 conn(0x7f41140b7dd0 legacy=0x7f411415f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:42.649 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.646+0000 7f41196b2640 1 -- 172.21.15.16:0/1089973563 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41140d2ac0 con 0x7f41140d0410 2023-05-22T16:03:42.649 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.646+0000 7f410bfff640 1 --2- 172.21.15.16:0/1089973563 >> v2:172.21.15.62:3300/0 conn(0x7f41140ccab0 0x7f411415fe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-05-22T16:03:42.650 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.650+0000 7f41196b2640 1 --2- 172.21.15.16:0/1089973563 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f41140d0410 0x7f41140d0830 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f410c009d90 tx=0x7f410c01c7b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=600fa39bc53f883a server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:03:42.650 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.650+0000 7f410b7fe640 1 -- 172.21.15.16:0/1089973563 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f410c0a57b0 con 0x7f41140d0410 2023-05-22T16:03:42.651 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.650+0000 7f410b7fe640 1 -- 172.21.15.16:0/1089973563 <== mon.2 v2:172.21.15.62:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f410c0a5910 con 0x7f41140d0410 2023-05-22T16:03:42.651 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.650+0000 7f410b7fe640 1 -- 172.21.15.16:0/1089973563 <== mon.2 v2:172.21.15.62:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f410c0a5ca0 con 0x7f41140d0410 2023-05-22T16:03:42.652 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.650+0000 7f411b115640 1 -- 172.21.15.16:0/1089973563 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f41140d0410 msgr2=0x7f41140d0830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:42.652 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.650+0000 7f411b115640 1 --2- 172.21.15.16:0/1089973563 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f41140d0410 0x7f41140d0830 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f410c009d90 tx=0x7f410c01c7b0 comp rx=0 tx=0).stop 2023-05-22T16:03:42.652 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.650+0000 7f411b115640 1 -- 172.21.15.16:0/1089973563 shutdown_connections 2023-05-22T16:03:42.652 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.650+0000 7f411b115640 1 --2- 172.21.15.16:0/1089973563 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f41140d0410 0x7f41140d0830 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:42.653 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.650+0000 7f411b115640 1 --2- 172.21.15.16:0/1089973563 >> v2:172.21.15.62:3300/0 conn(0x7f41140ccab0 0x7f411415fe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:42.653 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.650+0000 7f411b115640 1 -- 172.21.15.16:0/1089973563 >> 172.21.15.16:0/1089973563 conn(0x7f41140655c0 msgr2=0x7f4114070e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:03:42.653 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.650+0000 7f411b115640 1 -- 172.21.15.16:0/1089973563 shutdown_connections 2023-05-22T16:03:42.653 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.650+0000 7f411b115640 1 -- 172.21.15.16:0/1089973563 wait complete. 2023-05-22T16:03:42.656 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.654+0000 7f411b115640 1 Processor -- start 2023-05-22T16:03:42.657 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.654+0000 7f411b115640 1 -- start start 2023-05-22T16:03:42.657 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.654+0000 7f411b115640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f41140b7dd0 0x7f41140c8350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:42.657 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.654+0000 7f411b115640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f41140ccab0 0x7f41141557b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:42.658 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.654+0000 7f411b115640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f41140d0410 0x7f4114155cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:42.658 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.654+0000 7f411b115640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41140c5ff0 con 0x7f41140b7dd0 2023-05-22T16:03:42.658 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.654+0000 7f411b115640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41140c6290 con 0x7f41140ccab0 2023-05-22T16:03:42.659 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.654+0000 7f411b115640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41140c6570 con 0x7f41140d0410 2023-05-22T16:03:42.659 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.654+0000 7f4118eb1640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f41140b7dd0 0x7f41140c8350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:42.660 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.654+0000 7f4118eb1640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f41140b7dd0 0x7f41140c8350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48984/0 (socket says 172.21.15.16:48984) 2023-05-22T16:03:42.660 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.654+0000 7f4118eb1640 1 -- 172.21.15.16:0/1452147420 learned_addr learned my addr 172.21.15.16:0/1452147420 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:03:42.660 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.654+0000 7f410bfff640 1 --2- 172.21.15.16:0/1452147420 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f41140ccab0 0x7f41141557b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:42.660 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.654+0000 7f41196b2640 1 --2- 172.21.15.16:0/1452147420 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f41140d0410 0x7f4114155cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:42.661 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.654+0000 7f4118eb1640 1 -- 172.21.15.16:0/1452147420 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f41140d0410 msgr2=0x7f4114155cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:42.661 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.654+0000 7f4118eb1640 1 --2- 172.21.15.16:0/1452147420 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f41140d0410 0x7f4114155cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:42.662 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.654+0000 7f4118eb1640 1 -- 172.21.15.16:0/1452147420 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f41140ccab0 msgr2=0x7f41141557b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:42.662 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.654+0000 7f4118eb1640 1 --2- 172.21.15.16:0/1452147420 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f41140ccab0 0x7f41141557b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:42.662 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.654+0000 7f4118eb1640 1 -- 172.21.15.16:0/1452147420 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41040094d0 con 0x7f41140b7dd0 2023-05-22T16:03:42.662 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.654+0000 7f4118eb1640 1 --2- 172.21.15.16:0/1452147420 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f41140b7dd0 0x7f41140c8350 secure :-1 s=READY pgs=2256 cs=0 l=1 rev1=1 crypto rx=0x7f411000c800 tx=0x7f411000ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:03:42.662 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.654+0000 7f41097fa640 1 -- 172.21.15.16:0/1452147420 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4110005270 con 0x7f41140b7dd0 2023-05-22T16:03:42.663 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.654+0000 7f41097fa640 1 -- 172.21.15.16:0/1452147420 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41100053f0 con 0x7f41140b7dd0 2023-05-22T16:03:42.663 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.654+0000 7f411b115640 1 -- 172.21.15.16:0/1452147420 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f410c009a70 con 0x7f41140b7dd0 2023-05-22T16:03:42.663 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.654+0000 7f41097fa640 1 -- 172.21.15.16:0/1452147420 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f411000a880 con 0x7f41140b7dd0 2023-05-22T16:03:42.663 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.654+0000 7f411b115640 1 -- 172.21.15.16:0/1452147420 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4114160350 con 0x7f41140b7dd0 2023-05-22T16:03:42.663 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.654+0000 7f411b115640 1 -- 172.21.15.16:0/1452147420 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41140cf730 con 0x7f41140b7dd0 2023-05-22T16:03:42.664 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.658+0000 7f41097fa640 1 -- 172.21.15.16:0/1452147420 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 82) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f411000a9e0 con 0x7f41140b7dd0 2023-05-22T16:03:42.664 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.658+0000 7f41097fa640 1 --2- 172.21.15.16:0/1452147420 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f41000654a0 0x7f4100067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:42.664 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.658+0000 7f41097fa640 1 -- 172.21.15.16:0/1452147420 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(179..179 src has 1..179) v4 ==== 6438+0+0 (secure 0 0 0) 0x7f41100823e0 con 0x7f41140b7dd0 2023-05-22T16:03:42.665 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.658+0000 7f410bfff640 1 --2- 172.21.15.16:0/1452147420 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f41000654a0 0x7f4100067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:42.665 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.658+0000 7f410bfff640 1 --2- 172.21.15.16:0/1452147420 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f41000654a0 0x7f4100067950 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f4114157b80 tx=0x7f4104008040 comp rx=0 tx=0).ready entity=mgr.10032 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:03:42.665 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.662+0000 7f41097fa640 1 -- 172.21.15.16:0/1452147420 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f411004fcb0 con 0x7f41140b7dd0 2023-05-22T16:03:42.818 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.818+0000 7f411b115640 1 -- 172.21.15.16:0/1452147420 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f41141604e0 con 0x7f41140b7dd0 2023-05-22T16:03:42.821 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.822+0000 7f41097fa640 1 -- 172.21.15.16:0/1452147420 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v179) v1 ==== 74+0+18132 (secure 0 0 0) 0x7f41100532d0 con 0x7f41140b7dd0 2023-05-22T16:03:42.822 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T16:03:42.822 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":179,"fsid":"b2719834-7847-4d98-90a6-f65d57f85c49","created":"2023-05-22T15:26:25.259314+0000","modified":"2023-05-22T16:03:39.783017+0000","last_up_change":"2023-05-22T15:26:36.448524+0000","last_in_change":"2023-05-22T15:26:27.675668+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":23,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-05-22T15:26:38.019781+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-05-22T15:26:43.570514+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":22,"pool_name":"cephfs_metadata","create_time":"2023-05-22T16:03:37.371980+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"178","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_data","create_time":"2023-05-22T16:03:37.756887+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"179","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9e7ff854-b25f-4cbb-a5c2-1799d9c53d2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":37021},{"type":"v1","addr":"172.21.15.16:6825","nonce":37021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":37021},{"type":"v1","addr":"172.21.15.16:6827","nonce":37021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":37021},{"type":"v1","addr":"172.21.15.16:6831","nonce":37021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":37021},{"type":"v1","addr":"172.21.15.16:6829","nonce":37021}]},"public_addr":"172.21.15.16:6825/37021","cluster_addr":"172.21.15.16:6827/37021","heartbeat_back_addr":"172.21.15.16:6831/37021","heartbeat_front_addr":"172.21.15.16:6829/37021","state":["exists","up"]},{"osd":1,"uuid":"4702fbc9-a927-4565-aca5-41dc69cc8264","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":37008},{"type":"v1","addr":"172.21.15.16:6801","nonce":37008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":37008},{"type":"v1","addr":"172.21.15.16:6803","nonce":37008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":37008},{"type":"v1","addr":"172.21.15.16:6807","nonce":37008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":37008},{"type":"v1","addr":"172.21.15.16:6805","nonce":37008}]},"public_addr":"172.21.15.16:6801/37008","cluster_addr":"172.21.15.16:6803/37008","heartbeat_back_addr":"172.21.15.16:6807/37008","heartbeat_front_addr":"172.21.15.16:6805/37008","state":["exists","up"]},{"osd":2,"uuid":"1e7bba46-4eba-47ef-bb81-5e2d371638ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":37009},{"type":"v1","addr":"172.21.15.16:6809","nonce":37009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":37009},{"type":"v1","addr":"172.21.15.16:6811","nonce":37009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":37009},{"type":"v1","addr":"172.21.15.16:6815","nonce":37009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":37009},{"type":"v1","addr":"172.21.15.16:6813","nonce":37009}]},"public_addr":"172.21.15.16:6809/37009","cluster_addr":"172.21.15.16:6811/37009","heartbeat_back_addr":"172.21.15.16:6815/37009","heartbeat_front_addr":"172.21.15.16:6813/37009","state":["exists","up"]},{"osd":3,"uuid":"383cacc0-5a88-4904-990e-498ca59ef48a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":37020},{"type":"v1","addr":"172.21.15.16:6817","nonce":37020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":37020},{"type":"v1","addr":"172.21.15.16:6819","nonce":37020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":37020},{"type":"v1","addr":"172.21.15.16:6823","nonce":37020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":37020},{"type":"v1","addr":"172.21.15.16:6821","nonce":37020}]},"public_addr":"172.21.15.16:6817/37020","cluster_addr":"172.21.15.16:6819/37020","heartbeat_back_addr":"172.21.15.16:6823/37020","heartbeat_front_addr":"172.21.15.16:6821/37020","state":["exists","up"]},{"osd":4,"uuid":"970500ef-95b7-42c1-b080-eed4fae7013d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":37353},{"type":"v1","addr":"172.21.15.62:6817","nonce":37353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6819","nonce":37353},{"type":"v1","addr":"172.21.15.62:6822","nonce":37353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37353},{"type":"v1","addr":"172.21.15.62:6831","nonce":37353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37353},{"type":"v1","addr":"172.21.15.62:6826","nonce":37353}]},"public_addr":"172.21.15.62:6817/37353","cluster_addr":"172.21.15.62:6822/37353","heartbeat_back_addr":"172.21.15.62:6831/37353","heartbeat_front_addr":"172.21.15.62:6826/37353","state":["exists","up"]},{"osd":5,"uuid":"022d67db-d4d8-445b-b1b9-8b446a00f5c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":37352},{"type":"v1","addr":"172.21.15.62:6801","nonce":37352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":37352},{"type":"v1","addr":"172.21.15.62:6803","nonce":37352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":37352},{"type":"v1","addr":"172.21.15.62:6807","nonce":37352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":37352},{"type":"v1","addr":"172.21.15.62:6805","nonce":37352}]},"public_addr":"172.21.15.62:6801/37352","cluster_addr":"172.21.15.62:6803/37352","heartbeat_back_addr":"172.21.15.62:6807/37352","heartbeat_front_addr":"172.21.15.62:6805/37352","state":["exists","up"]},{"osd":6,"uuid":"42bd372b-6688-4a1b-8407-e16e8c266e33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":37354},{"type":"v1","addr":"172.21.15.62:6820","nonce":37354}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":37354},{"type":"v1","addr":"172.21.15.62:6823","nonce":37354}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6829","nonce":37354},{"type":"v1","addr":"172.21.15.62:6830","nonce":37354}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6825","nonce":37354},{"type":"v1","addr":"172.21.15.62:6827","nonce":37354}]},"public_addr":"172.21.15.62:6820/37354","cluster_addr":"172.21.15.62:6823/37354","heartbeat_back_addr":"172.21.15.62:6830/37354","heartbeat_front_addr":"172.21.15.62:6827/37354","state":["exists","up"]},{"osd":7,"uuid":"ac70b2a3-f41b-4bf2-b227-7f2cbb55426d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37355},{"type":"v1","addr":"172.21.15.62:6809","nonce":37355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37355},{"type":"v1","addr":"172.21.15.62:6811","nonce":37355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":37355},{"type":"v1","addr":"172.21.15.62:6815","nonce":37355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":37355},{"type":"v1","addr":"172.21.15.62:6813","nonce":37355}]},"public_addr":"172.21.15.62:6809/37355","cluster_addr":"172.21.15.62:6811/37355","heartbeat_back_addr":"172.21.15.62:6815/37355","heartbeat_front_addr":"172.21.15.62:6813/37355","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.368896+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.107273+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.276888+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.287502+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.124791+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.251937+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.255679+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.232797+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":17,"snaps":[{"begin":2,"length":2}]},{"pool":21,"snaps":[{"begin":25,"length":9}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-05-22T16:03:42.824 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.822+0000 7f411b115640 1 -- 172.21.15.16:0/1452147420 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f41000654a0 msgr2=0x7f4100067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:42.824 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.822+0000 7f411b115640 1 --2- 172.21.15.16:0/1452147420 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f41000654a0 0x7f4100067950 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f4114157b80 tx=0x7f4104008040 comp rx=0 tx=0).stop 2023-05-22T16:03:42.824 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.822+0000 7f411b115640 1 -- 172.21.15.16:0/1452147420 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f41140b7dd0 msgr2=0x7f41140c8350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:42.824 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.822+0000 7f411b115640 1 --2- 172.21.15.16:0/1452147420 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f41140b7dd0 0x7f41140c8350 secure :-1 s=READY pgs=2256 cs=0 l=1 rev1=1 crypto rx=0x7f411000c800 tx=0x7f411000ccc0 comp rx=0 tx=0).stop 2023-05-22T16:03:42.825 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.822+0000 7f411b115640 1 -- 172.21.15.16:0/1452147420 shutdown_connections 2023-05-22T16:03:42.825 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.822+0000 7f411b115640 1 --2- 172.21.15.16:0/1452147420 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f41140d0410 0x7f4114155cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:42.825 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.822+0000 7f411b115640 1 --2- 172.21.15.16:0/1452147420 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f41140ccab0 0x7f41141557b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:42.825 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.822+0000 7f411b115640 1 --2- 172.21.15.16:0/1452147420 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f41000654a0 0x7f4100067950 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:42.826 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.822+0000 7f411b115640 1 --2- 172.21.15.16:0/1452147420 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f41140b7dd0 0x7f41140c8350 unknown :-1 s=CLOSED pgs=2256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:42.826 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.822+0000 7f411b115640 1 -- 172.21.15.16:0/1452147420 >> 172.21.15.16:0/1452147420 conn(0x7f41140655c0 msgr2=0x7f411415b790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:03:42.826 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.822+0000 7f411b115640 1 -- 172.21.15.16:0/1452147420 shutdown_connections 2023-05-22T16:03:42.826 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.822+0000 7f411b115640 1 -- 172.21.15.16:0/1452147420 wait complete. 2023-05-22T16:03:42.842 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-05-22T16:03:42.843 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-05-22T16:03:42.964 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.962+0000 7fd233159640 1 Processor -- start 2023-05-22T16:03:42.964 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.962+0000 7fd233159640 1 -- start start 2023-05-22T16:03:42.966 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.966+0000 7fd233159640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fd22c1567c0 0x7fd22c0afb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:42.966 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.966+0000 7fd233159640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd22c0cb6a0 0x7fd22c0b0080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:42.966 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.966+0000 7fd233159640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd22c159340 0x7fd22c1597b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:42.967 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.966+0000 7fd233159640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd22c156b90 con 0x7fd22c0cb6a0 2023-05-22T16:03:42.967 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.966+0000 7fd233159640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd22c15a0b0 con 0x7fd22c1567c0 2023-05-22T16:03:42.967 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.966+0000 7fd233159640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd22c0ca8e0 con 0x7fd22c159340 2023-05-22T16:03:42.968 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.966+0000 7fd22bfff640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd22c0cb6a0 0x7fd22c0b0080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:42.968 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.966+0000 7fd22bfff640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd22c0cb6a0 0x7fd22c0b0080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48986/0 (socket says 172.21.15.16:48986) 2023-05-22T16:03:42.968 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.966+0000 7fd22bfff640 1 -- 172.21.15.16:0/550704584 learned_addr learned my addr 172.21.15.16:0/550704584 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:03:42.968 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.966+0000 7fd2316f6640 1 --2- 172.21.15.16:0/550704584 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd22c159340 0x7fd22c1597b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:42.969 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.966+0000 7fd230ef5640 1 --2- 172.21.15.16:0/550704584 >> v2:172.21.15.62:3300/0 conn(0x7fd22c1567c0 0x7fd22c0afb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:42.969 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.966+0000 7fd22bfff640 1 -- 172.21.15.16:0/550704584 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd22c159340 msgr2=0x7fd22c1597b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:03:42.969 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.966+0000 7fd22bfff640 1 --2- 172.21.15.16:0/550704584 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd22c159340 0x7fd22c1597b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:42.970 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.966+0000 7fd22bfff640 1 -- 172.21.15.16:0/550704584 >> v2:172.21.15.62:3300/0 conn(0x7fd22c1567c0 msgr2=0x7fd22c0afb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:03:42.970 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.966+0000 7fd22bfff640 1 --2- 172.21.15.16:0/550704584 >> v2:172.21.15.62:3300/0 conn(0x7fd22c1567c0 0x7fd22c0afb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:42.971 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.966+0000 7fd22bfff640 1 -- 172.21.15.16:0/550704584 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd22c0cabc0 con 0x7fd22c0cb6a0 2023-05-22T16:03:42.971 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.966+0000 7fd2316f6640 1 --2- 172.21.15.16:0/550704584 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd22c159340 0x7fd22c1597b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T16:03:42.973 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.966+0000 7fd22bfff640 1 --2- 172.21.15.16:0/550704584 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd22c0cb6a0 0x7fd22c0b0080 secure :-1 s=READY pgs=2257 cs=0 l=1 rev1=1 crypto rx=0x7fd22c0c50a0 tx=0x7fd21401c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5711ee2fd812c9c9 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:03:42.973 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.970+0000 7fd22b7fe640 1 -- 172.21.15.16:0/550704584 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2140ac410 con 0x7fd22c0cb6a0 2023-05-22T16:03:42.974 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.970+0000 7fd22b7fe640 1 -- 172.21.15.16:0/550704584 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2140ac570 con 0x7fd22c0cb6a0 2023-05-22T16:03:42.974 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.970+0000 7fd22b7fe640 1 -- 172.21.15.16:0/550704584 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2140ac900 con 0x7fd22c0cb6a0 2023-05-22T16:03:42.974 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.970+0000 7fd233159640 1 -- 172.21.15.16:0/550704584 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd22c0cb6a0 msgr2=0x7fd22c0b0080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:42.974 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.970+0000 7fd233159640 1 --2- 172.21.15.16:0/550704584 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd22c0cb6a0 0x7fd22c0b0080 secure :-1 s=READY pgs=2257 cs=0 l=1 rev1=1 crypto rx=0x7fd22c0c50a0 tx=0x7fd21401c500 comp rx=0 tx=0).stop 2023-05-22T16:03:42.974 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.970+0000 7fd233159640 1 -- 172.21.15.16:0/550704584 shutdown_connections 2023-05-22T16:03:42.974 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.970+0000 7fd233159640 1 --2- 172.21.15.16:0/550704584 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd22c0cb6a0 0x7fd22c0b0080 unknown :-1 s=CLOSED pgs=2257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:42.974 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.970+0000 7fd233159640 1 --2- 172.21.15.16:0/550704584 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd22c159340 0x7fd22c1597b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:42.974 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.970+0000 7fd233159640 1 --2- 172.21.15.16:0/550704584 >> v2:172.21.15.62:3300/0 conn(0x7fd22c1567c0 0x7fd22c0afb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:42.975 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.970+0000 7fd233159640 1 -- 172.21.15.16:0/550704584 >> 172.21.15.16:0/550704584 conn(0x7fd22c069720 msgr2=0x7fd22c069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:03:42.975 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.970+0000 7fd233159640 1 -- 172.21.15.16:0/550704584 shutdown_connections 2023-05-22T16:03:42.975 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.970+0000 7fd233159640 1 -- 172.21.15.16:0/550704584 wait complete. 2023-05-22T16:03:42.975 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.970+0000 7fd233159640 1 Processor -- start 2023-05-22T16:03:42.975 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.970+0000 7fd233159640 1 -- start start 2023-05-22T16:03:42.975 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.970+0000 7fd233159640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd22c0cb6a0 0x7fd22c0c4340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:42.975 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.970+0000 7fd233159640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd22c1567c0 0x7fd22c14e520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:42.976 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.970+0000 7fd233159640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd22c159340 0x7fd22c1f2c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:42.976 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.970+0000 7fd233159640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd22c14ea60 con 0x7fd22c0cb6a0 2023-05-22T16:03:42.976 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.970+0000 7fd233159640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd22c1f31a0 con 0x7fd22c1567c0 2023-05-22T16:03:42.976 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.970+0000 7fd233159640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd22c0c2e80 con 0x7fd22c159340 2023-05-22T16:03:42.976 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.970+0000 7fd230ef5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd22c0cb6a0 0x7fd22c0c4340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:42.976 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.970+0000 7fd230ef5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd22c0cb6a0 0x7fd22c0c4340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48990/0 (socket says 172.21.15.16:48990) 2023-05-22T16:03:42.976 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.970+0000 7fd230ef5640 1 -- 172.21.15.16:0/2397431633 learned_addr learned my addr 172.21.15.16:0/2397431633 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:03:42.977 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.970+0000 7fd22bfff640 1 --2- 172.21.15.16:0/2397431633 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd22c1567c0 0x7fd22c14e520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:42.977 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.970+0000 7fd2316f6640 1 --2- 172.21.15.16:0/2397431633 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd22c159340 0x7fd22c1f2c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:42.977 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.970+0000 7fd230ef5640 1 -- 172.21.15.16:0/2397431633 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd22c159340 msgr2=0x7fd22c1f2c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:42.977 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.970+0000 7fd230ef5640 1 --2- 172.21.15.16:0/2397431633 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd22c159340 0x7fd22c1f2c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:42.977 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.970+0000 7fd230ef5640 1 -- 172.21.15.16:0/2397431633 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd22c1567c0 msgr2=0x7fd22c14e520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:42.977 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.970+0000 7fd230ef5640 1 --2- 172.21.15.16:0/2397431633 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd22c1567c0 0x7fd22c14e520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:42.977 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.970+0000 7fd230ef5640 1 -- 172.21.15.16:0/2397431633 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2140094d0 con 0x7fd22c0cb6a0 2023-05-22T16:03:42.977 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.970+0000 7fd22bfff640 1 --2- 172.21.15.16:0/2397431633 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd22c1567c0 0x7fd22c14e520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T16:03:42.978 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.970+0000 7fd230ef5640 1 --2- 172.21.15.16:0/2397431633 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd22c0cb6a0 0x7fd22c0c4340 secure :-1 s=READY pgs=2258 cs=0 l=1 rev1=1 crypto rx=0x7fd224007ec0 tx=0x7fd22400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:03:42.978 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.970+0000 7fd2297fa640 1 -- 172.21.15.16:0/2397431633 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd224014620 con 0x7fd22c0cb6a0 2023-05-22T16:03:42.978 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.970+0000 7fd2297fa640 1 -- 172.21.15.16:0/2397431633 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2240147a0 con 0x7fd22c0cb6a0 2023-05-22T16:03:42.978 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.970+0000 7fd233159640 1 -- 172.21.15.16:0/2397431633 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd22c159e80 con 0x7fd22c0cb6a0 2023-05-22T16:03:42.978 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.970+0000 7fd2297fa640 1 -- 172.21.15.16:0/2397431633 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd224014b50 con 0x7fd22c0cb6a0 2023-05-22T16:03:42.978 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.970+0000 7fd233159640 1 -- 172.21.15.16:0/2397431633 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd22c159cf0 con 0x7fd22c0cb6a0 2023-05-22T16:03:42.978 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.974+0000 7fd233159640 1 -- 172.21.15.16:0/2397431633 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd22c0ccd70 con 0x7fd22c0cb6a0 2023-05-22T16:03:42.978 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.974+0000 7fd2297fa640 1 -- 172.21.15.16:0/2397431633 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 82) v1 ==== 84332+0+0 (secure 0 0 0) 0x7fd224014cb0 con 0x7fd22c0cb6a0 2023-05-22T16:03:42.979 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.974+0000 7fd2297fa640 1 --2- 172.21.15.16:0/2397431633 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fd20c0654a0 0x7fd20c067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:42.979 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.974+0000 7fd2297fa640 1 -- 172.21.15.16:0/2397431633 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(179..179 src has 1..179) v4 ==== 6438+0+0 (secure 0 0 0) 0x7fd224083500 con 0x7fd22c0cb6a0 2023-05-22T16:03:42.980 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.974+0000 7fd22bfff640 1 --2- 172.21.15.16:0/2397431633 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fd20c0654a0 0x7fd20c067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:42.980 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.974+0000 7fd22bfff640 1 --2- 172.21.15.16:0/2397431633 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fd20c0654a0 0x7fd20c067950 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fd2140ae000 tx=0x7fd2140a6040 comp rx=0 tx=0).ready entity=mgr.10032 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:03:42.980 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:42.978+0000 7fd2297fa640 1 -- 172.21.15.16:0/2397431633 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fd224050f80 con 0x7fd22c0cb6a0 2023-05-22T16:03:43.129 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:43.130+0000 7fd233159640 1 -- 172.21.15.16:0/2397431633 --> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd22c04dc50 con 0x7fd20c0654a0 2023-05-22T16:03:43.154 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:43.154+0000 7fd2297fa640 1 -- 172.21.15.16:0/2397431633 <== mgr.10032 v2:172.21.15.16:6832/65890 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+309917 (secure 0 0 0) 0x7fd22c04dc50 con 0x7fd20c0654a0 2023-05-22T16:03:43.157 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T16:03:43.158 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2023-05-22T16:03:43.159 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:43.158+0000 7fd233159640 1 -- 172.21.15.16:0/2397431633 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fd20c0654a0 msgr2=0x7fd20c067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:43.159 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:43.158+0000 7fd233159640 1 --2- 172.21.15.16:0/2397431633 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fd20c0654a0 0x7fd20c067950 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fd2140ae000 tx=0x7fd2140a6040 comp rx=0 tx=0).stop 2023-05-22T16:03:43.160 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:43.158+0000 7fd233159640 1 -- 172.21.15.16:0/2397431633 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd22c0cb6a0 msgr2=0x7fd22c0c4340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:43.160 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:43.158+0000 7fd233159640 1 --2- 172.21.15.16:0/2397431633 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd22c0cb6a0 0x7fd22c0c4340 secure :-1 s=READY pgs=2258 cs=0 l=1 rev1=1 crypto rx=0x7fd224007ec0 tx=0x7fd22400e4b0 comp rx=0 tx=0).stop 2023-05-22T16:03:43.160 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:43.158+0000 7fd233159640 1 -- 172.21.15.16:0/2397431633 shutdown_connections 2023-05-22T16:03:43.160 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:43.158+0000 7fd233159640 1 --2- 172.21.15.16:0/2397431633 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd22c159340 0x7fd22c1f2c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:43.160 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:43.158+0000 7fd233159640 1 --2- 172.21.15.16:0/2397431633 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd22c1567c0 0x7fd22c14e520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:43.160 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:43.158+0000 7fd233159640 1 --2- 172.21.15.16:0/2397431633 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fd20c0654a0 0x7fd20c067950 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:43.160 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:43.158+0000 7fd233159640 1 --2- 172.21.15.16:0/2397431633 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd22c0cb6a0 0x7fd22c0c4340 unknown :-1 s=CLOSED pgs=2258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:43.161 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:43.158+0000 7fd233159640 1 -- 172.21.15.16:0/2397431633 >> 172.21.15.16:0/2397431633 conn(0x7fd22c069720 msgr2=0x7fd22c153dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:03:43.161 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:43.158+0000 7fd233159640 1 -- 172.21.15.16:0/2397431633 shutdown_connections 2023-05-22T16:03:43.161 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:43.158+0000 7fd233159640 1 -- 172.21.15.16:0/2397431633 wait complete. 2023-05-22T16:03:43.177 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":25,"stamp":"2023-05-22T16:03:41.738188+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592698,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1748,"num_read_kb":3602,"num_write":543,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":342,"ondisk_log_size":342,"up":210,"acting":210,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":97,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":237160,"kb_used_data":18600,"kb_used_omap":0,"kb_used_meta":218496,"kb_avail":754737560,"statfs":{"total":773094113280,"available":772851261440,"internally_reserved":0,"allocated":19046400,"data_stored":10676364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":223739904},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":62,"apply_latency_ms":62,"commit_latency_ns":62000000,"apply_latency_ns":62000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2330,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.349851"},"pg_stats":[{"pgid":"23.3f","version":"0'0","reported_seq":9,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.834936+0000","last_change":"2023-05-22T16:03:38.831697+0000","last_active":"2023-05-22T16:03:38.834936+0000","last_peered":"2023-05-22T16:03:38.834936+0000","last_clean":"2023-05-22T16:03:38.834936+0000","last_became_active":"2023-05-22T16:03:38.831380+0000","last_became_peered":"2023-05-22T16:03:38.831380+0000","last_unstale":"2023-05-22T16:03:38.834936+0000","last_undegraded":"2023-05-22T16:03:38.834936+0000","last_fullsized":"2023-05-22T16:03:38.834936+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_clean_scrub_stamp":"2023-05-22T16:03:38.387135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:53:38.976658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.3e","version":"0'0","reported_seq":9,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.831119+0000","last_change":"2023-05-22T16:03:38.824745+0000","last_active":"2023-05-22T16:03:38.831119+0000","last_peered":"2023-05-22T16:03:38.831119+0000","last_clean":"2023-05-22T16:03:38.831119+0000","last_became_active":"2023-05-22T16:03:38.819756+0000","last_became_peered":"2023-05-22T16:03:38.819756+0000","last_unstale":"2023-05-22T16:03:38.831119+0000","last_undegraded":"2023-05-22T16:03:38.831119+0000","last_fullsized":"2023-05-22T16:03:38.831119+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_clean_scrub_stamp":"2023-05-22T16:03:38.387135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:42:06.604797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.3d","version":"0'0","reported_seq":9,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.837362+0000","last_change":"2023-05-22T16:03:38.828466+0000","last_active":"2023-05-22T16:03:38.837362+0000","last_peered":"2023-05-22T16:03:38.837362+0000","last_clean":"2023-05-22T16:03:38.837362+0000","last_became_active":"2023-05-22T16:03:38.826096+0000","last_became_peered":"2023-05-22T16:03:38.826096+0000","last_unstale":"2023-05-22T16:03:38.837362+0000","last_undegraded":"2023-05-22T16:03:38.837362+0000","last_fullsized":"2023-05-22T16:03:38.837362+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_clean_scrub_stamp":"2023-05-22T16:03:38.387135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:03:18.095743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.3c","version":"0'0","reported_seq":9,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.813436+0000","last_change":"2023-05-22T16:03:38.810194+0000","last_active":"2023-05-22T16:03:38.813436+0000","last_peered":"2023-05-22T16:03:38.813436+0000","last_clean":"2023-05-22T16:03:38.813436+0000","last_became_active":"2023-05-22T16:03:38.809542+0000","last_became_peered":"2023-05-22T16:03:38.809542+0000","last_unstale":"2023-05-22T16:03:38.813436+0000","last_undegraded":"2023-05-22T16:03:38.813436+0000","last_fullsized":"2023-05-22T16:03:38.813436+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_clean_scrub_stamp":"2023-05-22T16:03:38.387135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:01:48.420792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.3b","version":"0'0","reported_seq":9,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.838725+0000","last_change":"2023-05-22T16:03:38.837188+0000","last_active":"2023-05-22T16:03:38.838725+0000","last_peered":"2023-05-22T16:03:38.838725+0000","last_clean":"2023-05-22T16:03:38.838725+0000","last_became_active":"2023-05-22T16:03:38.836750+0000","last_became_peered":"2023-05-22T16:03:38.836750+0000","last_unstale":"2023-05-22T16:03:38.838725+0000","last_undegraded":"2023-05-22T16:03:38.838725+0000","last_fullsized":"2023-05-22T16:03:38.838725+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_clean_scrub_stamp":"2023-05-22T16:03:38.387135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:14:08.133076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.3a","version":"0'0","reported_seq":9,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.817549+0000","last_change":"2023-05-22T16:03:38.803227+0000","last_active":"2023-05-22T16:03:38.817549+0000","last_peered":"2023-05-22T16:03:38.817549+0000","last_clean":"2023-05-22T16:03:38.817549+0000","last_became_active":"2023-05-22T16:03:38.802489+0000","last_became_peered":"2023-05-22T16:03:38.802489+0000","last_unstale":"2023-05-22T16:03:38.817549+0000","last_undegraded":"2023-05-22T16:03:38.817549+0000","last_fullsized":"2023-05-22T16:03:38.817549+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_clean_scrub_stamp":"2023-05-22T16:03:38.387135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:16:16.828826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.39","version":"0'0","reported_seq":9,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.812527+0000","last_change":"2023-05-22T16:03:38.806822+0000","last_active":"2023-05-22T16:03:38.812527+0000","last_peered":"2023-05-22T16:03:38.812527+0000","last_clean":"2023-05-22T16:03:38.812527+0000","last_became_active":"2023-05-22T16:03:38.806056+0000","last_became_peered":"2023-05-22T16:03:38.806056+0000","last_unstale":"2023-05-22T16:03:38.812527+0000","last_undegraded":"2023-05-22T16:03:38.812527+0000","last_fullsized":"2023-05-22T16:03:38.812527+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_clean_scrub_stamp":"2023-05-22T16:03:38.387135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:48:38.105506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.38","version":"0'0","reported_seq":9,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.836771+0000","last_change":"2023-05-22T16:03:38.830698+0000","last_active":"2023-05-22T16:03:38.836771+0000","last_peered":"2023-05-22T16:03:38.836771+0000","last_clean":"2023-05-22T16:03:38.836771+0000","last_became_active":"2023-05-22T16:03:38.830389+0000","last_became_peered":"2023-05-22T16:03:38.830389+0000","last_unstale":"2023-05-22T16:03:38.836771+0000","last_undegraded":"2023-05-22T16:03:38.836771+0000","last_fullsized":"2023-05-22T16:03:38.836771+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_clean_scrub_stamp":"2023-05-22T16:03:38.387135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:40:17.506020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.37","version":"0'0","reported_seq":9,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.831076+0000","last_change":"2023-05-22T16:03:38.818858+0000","last_active":"2023-05-22T16:03:38.831076+0000","last_peered":"2023-05-22T16:03:38.831076+0000","last_clean":"2023-05-22T16:03:38.831076+0000","last_became_active":"2023-05-22T16:03:38.818431+0000","last_became_peered":"2023-05-22T16:03:38.818431+0000","last_unstale":"2023-05-22T16:03:38.831076+0000","last_undegraded":"2023-05-22T16:03:38.831076+0000","last_fullsized":"2023-05-22T16:03:38.831076+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_clean_scrub_stamp":"2023-05-22T16:03:38.387135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:22:36.886924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.36","version":"0'0","reported_seq":9,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.831151+0000","last_change":"2023-05-22T16:03:38.825183+0000","last_active":"2023-05-22T16:03:38.831151+0000","last_peered":"2023-05-22T16:03:38.831151+0000","last_clean":"2023-05-22T16:03:38.831151+0000","last_became_active":"2023-05-22T16:03:38.824914+0000","last_became_peered":"2023-05-22T16:03:38.824914+0000","last_unstale":"2023-05-22T16:03:38.831151+0000","last_undegraded":"2023-05-22T16:03:38.831151+0000","last_fullsized":"2023-05-22T16:03:38.831151+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_clean_scrub_stamp":"2023-05-22T16:03:38.387135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:20:29.890916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.35","version":"0'0","reported_seq":9,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.839055+0000","last_change":"2023-05-22T16:03:38.837665+0000","last_active":"2023-05-22T16:03:38.839055+0000","last_peered":"2023-05-22T16:03:38.839055+0000","last_clean":"2023-05-22T16:03:38.839055+0000","last_became_active":"2023-05-22T16:03:38.837234+0000","last_became_peered":"2023-05-22T16:03:38.837234+0000","last_unstale":"2023-05-22T16:03:38.839055+0000","last_undegraded":"2023-05-22T16:03:38.839055+0000","last_fullsized":"2023-05-22T16:03:38.839055+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_clean_scrub_stamp":"2023-05-22T16:03:38.387135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:44:20.427086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.34","version":"0'0","reported_seq":9,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.831026+0000","last_change":"2023-05-22T16:03:38.827948+0000","last_active":"2023-05-22T16:03:38.831026+0000","last_peered":"2023-05-22T16:03:38.831026+0000","last_clean":"2023-05-22T16:03:38.831026+0000","last_became_active":"2023-05-22T16:03:38.827558+0000","last_became_peered":"2023-05-22T16:03:38.827558+0000","last_unstale":"2023-05-22T16:03:38.831026+0000","last_undegraded":"2023-05-22T16:03:38.831026+0000","last_fullsized":"2023-05-22T16:03:38.831026+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_clean_scrub_stamp":"2023-05-22T16:03:38.387135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T03:42:31.724542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.33","version":"0'0","reported_seq":9,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.808994+0000","last_change":"2023-05-22T16:03:38.805157+0000","last_active":"2023-05-22T16:03:38.808994+0000","last_peered":"2023-05-22T16:03:38.808994+0000","last_clean":"2023-05-22T16:03:38.808994+0000","last_became_active":"2023-05-22T16:03:38.804749+0000","last_became_peered":"2023-05-22T16:03:38.804749+0000","last_unstale":"2023-05-22T16:03:38.808994+0000","last_undegraded":"2023-05-22T16:03:38.808994+0000","last_fullsized":"2023-05-22T16:03:38.808994+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_clean_scrub_stamp":"2023-05-22T16:03:38.387135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:49:45.112747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.32","version":"0'0","reported_seq":9,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.834252+0000","last_change":"2023-05-22T16:03:38.825823+0000","last_active":"2023-05-22T16:03:38.834252+0000","last_peered":"2023-05-22T16:03:38.834252+0000","last_clean":"2023-05-22T16:03:38.834252+0000","last_became_active":"2023-05-22T16:03:38.825281+0000","last_became_peered":"2023-05-22T16:03:38.825281+0000","last_unstale":"2023-05-22T16:03:38.834252+0000","last_undegraded":"2023-05-22T16:03:38.834252+0000","last_fullsized":"2023-05-22T16:03:38.834252+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_clean_scrub_stamp":"2023-05-22T16:03:38.387135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T03:06:41.364328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.31","version":"0'0","reported_seq":9,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.830706+0000","last_change":"2023-05-22T16:03:38.812630+0000","last_active":"2023-05-22T16:03:38.830706+0000","last_peered":"2023-05-22T16:03:38.830706+0000","last_clean":"2023-05-22T16:03:38.830706+0000","last_became_active":"2023-05-22T16:03:38.811901+0000","last_became_peered":"2023-05-22T16:03:38.811901+0000","last_unstale":"2023-05-22T16:03:38.830706+0000","last_undegraded":"2023-05-22T16:03:38.830706+0000","last_fullsized":"2023-05-22T16:03:38.830706+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_clean_scrub_stamp":"2023-05-22T16:03:38.387135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:08:59.616966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.30","version":"0'0","reported_seq":9,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.830817+0000","last_change":"2023-05-22T16:03:38.829912+0000","last_active":"2023-05-22T16:03:38.830817+0000","last_peered":"2023-05-22T16:03:38.830817+0000","last_clean":"2023-05-22T16:03:38.830817+0000","last_became_active":"2023-05-22T16:03:38.829561+0000","last_became_peered":"2023-05-22T16:03:38.829561+0000","last_unstale":"2023-05-22T16:03:38.830817+0000","last_undegraded":"2023-05-22T16:03:38.830817+0000","last_fullsized":"2023-05-22T16:03:38.830817+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_clean_scrub_stamp":"2023-05-22T16:03:38.387135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:20:17.651383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.2f","version":"0'0","reported_seq":9,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.817945+0000","last_change":"2023-05-22T16:03:38.809726+0000","last_active":"2023-05-22T16:03:38.817945+0000","last_peered":"2023-05-22T16:03:38.817945+0000","last_clean":"2023-05-22T16:03:38.817945+0000","last_became_active":"2023-05-22T16:03:38.809472+0000","last_became_peered":"2023-05-22T16:03:38.809472+0000","last_unstale":"2023-05-22T16:03:38.817945+0000","last_undegraded":"2023-05-22T16:03:38.817945+0000","last_fullsized":"2023-05-22T16:03:38.817945+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_clean_scrub_stamp":"2023-05-22T16:03:38.387135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:05:33.932707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.2e","version":"0'0","reported_seq":9,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.806937+0000","last_change":"2023-05-22T16:03:38.800994+0000","last_active":"2023-05-22T16:03:38.806937+0000","last_peered":"2023-05-22T16:03:38.806937+0000","last_clean":"2023-05-22T16:03:38.806937+0000","last_became_active":"2023-05-22T16:03:38.800724+0000","last_became_peered":"2023-05-22T16:03:38.800724+0000","last_unstale":"2023-05-22T16:03:38.806937+0000","last_undegraded":"2023-05-22T16:03:38.806937+0000","last_fullsized":"2023-05-22T16:03:38.806937+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_clean_scrub_stamp":"2023-05-22T16:03:38.387135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T18:39:51.525114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.2d","version":"0'0","reported_seq":9,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.817455+0000","last_change":"2023-05-22T16:03:38.808401+0000","last_active":"2023-05-22T16:03:38.817455+0000","last_peered":"2023-05-22T16:03:38.817455+0000","last_clean":"2023-05-22T16:03:38.817455+0000","last_became_active":"2023-05-22T16:03:38.807633+0000","last_became_peered":"2023-05-22T16:03:38.807633+0000","last_unstale":"2023-05-22T16:03:38.817455+0000","last_undegraded":"2023-05-22T16:03:38.817455+0000","last_fullsized":"2023-05-22T16:03:38.817455+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_clean_scrub_stamp":"2023-05-22T16:03:38.387135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:59:10.609792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.2c","version":"0'0","reported_seq":9,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.834197+0000","last_change":"2023-05-22T16:03:38.825547+0000","last_active":"2023-05-22T16:03:38.834197+0000","last_peered":"2023-05-22T16:03:38.834197+0000","last_clean":"2023-05-22T16:03:38.834197+0000","last_became_active":"2023-05-22T16:03:38.825059+0000","last_became_peered":"2023-05-22T16:03:38.825059+0000","last_unstale":"2023-05-22T16:03:38.834197+0000","last_undegraded":"2023-05-22T16:03:38.834197+0000","last_fullsized":"2023-05-22T16:03:38.834197+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_clean_scrub_stamp":"2023-05-22T16:03:38.387135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:03:34.709150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.2b","version":"0'0","reported_seq":9,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.826073+0000","last_change":"2023-05-22T16:03:38.825090+0000","last_active":"2023-05-22T16:03:38.826073+0000","last_peered":"2023-05-22T16:03:38.826073+0000","last_clean":"2023-05-22T16:03:38.826073+0000","last_became_active":"2023-05-22T16:03:38.824769+0000","last_became_peered":"2023-05-22T16:03:38.824769+0000","last_unstale":"2023-05-22T16:03:38.826073+0000","last_undegraded":"2023-05-22T16:03:38.826073+0000","last_fullsized":"2023-05-22T16:03:38.826073+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_clean_scrub_stamp":"2023-05-22T16:03:38.387135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:32:40.328940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.2a","version":"0'0","reported_seq":9,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.831201+0000","last_change":"2023-05-22T16:03:38.825914+0000","last_active":"2023-05-22T16:03:38.831201+0000","last_peered":"2023-05-22T16:03:38.831201+0000","last_clean":"2023-05-22T16:03:38.831201+0000","last_became_active":"2023-05-22T16:03:38.825706+0000","last_became_peered":"2023-05-22T16:03:38.825706+0000","last_unstale":"2023-05-22T16:03:38.831201+0000","last_undegraded":"2023-05-22T16:03:38.831201+0000","last_fullsized":"2023-05-22T16:03:38.831201+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_clean_scrub_stamp":"2023-05-22T16:03:38.387135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:55:10.166213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.29","version":"0'0","reported_seq":9,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.829415+0000","last_change":"2023-05-22T16:03:38.828535+0000","last_active":"2023-05-22T16:03:38.829415+0000","last_peered":"2023-05-22T16:03:38.829415+0000","last_clean":"2023-05-22T16:03:38.829415+0000","last_became_active":"2023-05-22T16:03:38.828237+0000","last_became_peered":"2023-05-22T16:03:38.828237+0000","last_unstale":"2023-05-22T16:03:38.829415+0000","last_undegraded":"2023-05-22T16:03:38.829415+0000","last_fullsized":"2023-05-22T16:03:38.829415+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_clean_scrub_stamp":"2023-05-22T16:03:38.387135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T03:22:36.354019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.28","version":"0'0","reported_seq":9,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.830921+0000","last_change":"2023-05-22T16:03:38.819073+0000","last_active":"2023-05-22T16:03:38.830921+0000","last_peered":"2023-05-22T16:03:38.830921+0000","last_clean":"2023-05-22T16:03:38.830921+0000","last_became_active":"2023-05-22T16:03:38.817944+0000","last_became_peered":"2023-05-22T16:03:38.817944+0000","last_unstale":"2023-05-22T16:03:38.830921+0000","last_undegraded":"2023-05-22T16:03:38.830921+0000","last_fullsized":"2023-05-22T16:03:38.830921+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_clean_scrub_stamp":"2023-05-22T16:03:38.387135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:17:50.697042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.27","version":"0'0","reported_seq":9,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.838058+0000","last_change":"2023-05-22T16:03:38.836502+0000","last_active":"2023-05-22T16:03:38.838058+0000","last_peered":"2023-05-22T16:03:38.838058+0000","last_clean":"2023-05-22T16:03:38.838058+0000","last_became_active":"2023-05-22T16:03:38.836238+0000","last_became_peered":"2023-05-22T16:03:38.836238+0000","last_unstale":"2023-05-22T16:03:38.838058+0000","last_undegraded":"2023-05-22T16:03:38.838058+0000","last_fullsized":"2023-05-22T16:03:38.838058+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_clean_scrub_stamp":"2023-05-22T16:03:38.387135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:43:01.268208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.26","version":"0'0","reported_seq":9,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.818774+0000","last_change":"2023-05-22T16:03:38.816847+0000","last_active":"2023-05-22T16:03:38.818774+0000","last_peered":"2023-05-22T16:03:38.818774+0000","last_clean":"2023-05-22T16:03:38.818774+0000","last_became_active":"2023-05-22T16:03:38.816335+0000","last_became_peered":"2023-05-22T16:03:38.816335+0000","last_unstale":"2023-05-22T16:03:38.818774+0000","last_undegraded":"2023-05-22T16:03:38.818774+0000","last_fullsized":"2023-05-22T16:03:38.818774+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_clean_scrub_stamp":"2023-05-22T16:03:38.387135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:30:36.909855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.25","version":"0'0","reported_seq":9,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.837336+0000","last_change":"2023-05-22T16:03:38.826726+0000","last_active":"2023-05-22T16:03:38.837336+0000","last_peered":"2023-05-22T16:03:38.837336+0000","last_clean":"2023-05-22T16:03:38.837336+0000","last_became_active":"2023-05-22T16:03:38.822171+0000","last_became_peered":"2023-05-22T16:03:38.822171+0000","last_unstale":"2023-05-22T16:03:38.837336+0000","last_undegraded":"2023-05-22T16:03:38.837336+0000","last_fullsized":"2023-05-22T16:03:38.837336+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_clean_scrub_stamp":"2023-05-22T16:03:38.387135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T16:33:34.861902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.24","version":"0'0","reported_seq":9,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.826521+0000","last_change":"2023-05-22T16:03:38.825353+0000","last_active":"2023-05-22T16:03:38.826521+0000","last_peered":"2023-05-22T16:03:38.826521+0000","last_clean":"2023-05-22T16:03:38.826521+0000","last_became_active":"2023-05-22T16:03:38.824976+0000","last_became_peered":"2023-05-22T16:03:38.824976+0000","last_unstale":"2023-05-22T16:03:38.826521+0000","last_undegraded":"2023-05-22T16:03:38.826521+0000","last_fullsized":"2023-05-22T16:03:38.826521+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_clean_scrub_stamp":"2023-05-22T16:03:38.387135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:14:08.133076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.23","version":"0'0","reported_seq":9,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.834226+0000","last_change":"2023-05-22T16:03:38.824726+0000","last_active":"2023-05-22T16:03:38.834226+0000","last_peered":"2023-05-22T16:03:38.834226+0000","last_clean":"2023-05-22T16:03:38.834226+0000","last_became_active":"2023-05-22T16:03:38.822077+0000","last_became_peered":"2023-05-22T16:03:38.822077+0000","last_unstale":"2023-05-22T16:03:38.834226+0000","last_undegraded":"2023-05-22T16:03:38.834226+0000","last_fullsized":"2023-05-22T16:03:38.834226+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_clean_scrub_stamp":"2023-05-22T16:03:38.387135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:59:31.259784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.22","version":"0'0","reported_seq":9,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.837983+0000","last_change":"2023-05-22T16:03:38.828353+0000","last_active":"2023-05-22T16:03:38.837983+0000","last_peered":"2023-05-22T16:03:38.837983+0000","last_clean":"2023-05-22T16:03:38.837983+0000","last_became_active":"2023-05-22T16:03:38.828046+0000","last_became_peered":"2023-05-22T16:03:38.828046+0000","last_unstale":"2023-05-22T16:03:38.837983+0000","last_undegraded":"2023-05-22T16:03:38.837983+0000","last_fullsized":"2023-05-22T16:03:38.837983+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_clean_scrub_stamp":"2023-05-22T16:03:38.387135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:34:38.930308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.21","version":"0'0","reported_seq":9,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.836140+0000","last_change":"2023-05-22T16:03:38.815565+0000","last_active":"2023-05-22T16:03:38.836140+0000","last_peered":"2023-05-22T16:03:38.836140+0000","last_clean":"2023-05-22T16:03:38.836140+0000","last_became_active":"2023-05-22T16:03:38.814892+0000","last_became_peered":"2023-05-22T16:03:38.814892+0000","last_unstale":"2023-05-22T16:03:38.836140+0000","last_undegraded":"2023-05-22T16:03:38.836140+0000","last_fullsized":"2023-05-22T16:03:38.836140+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_clean_scrub_stamp":"2023-05-22T16:03:38.387135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T03:14:45.639272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.20","version":"0'0","reported_seq":9,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.806931+0000","last_change":"2023-05-22T16:03:38.800427+0000","last_active":"2023-05-22T16:03:38.806931+0000","last_peered":"2023-05-22T16:03:38.806931+0000","last_clean":"2023-05-22T16:03:38.806931+0000","last_became_active":"2023-05-22T16:03:38.800151+0000","last_became_peered":"2023-05-22T16:03:38.800151+0000","last_unstale":"2023-05-22T16:03:38.806931+0000","last_undegraded":"2023-05-22T16:03:38.806931+0000","last_fullsized":"2023-05-22T16:03:38.806931+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_clean_scrub_stamp":"2023-05-22T16:03:38.387135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T03:14:45.639272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.1e","version":"0'0","reported_seq":9,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.818753+0000","last_change":"2023-05-22T16:03:38.816615+0000","last_active":"2023-05-22T16:03:38.818753+0000","last_peered":"2023-05-22T16:03:38.818753+0000","last_clean":"2023-05-22T16:03:38.818753+0000","last_became_active":"2023-05-22T16:03:38.815975+0000","last_became_peered":"2023-05-22T16:03:38.815975+0000","last_unstale":"2023-05-22T16:03:38.818753+0000","last_undegraded":"2023-05-22T16:03:38.818753+0000","last_fullsized":"2023-05-22T16:03:38.818753+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_clean_scrub_stamp":"2023-05-22T16:03:38.387135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:58:45.869044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.1f","version":"178'4","reported_seq":14,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.834972+0000","last_change":"2023-05-22T16:03:38.447497+0000","last_active":"2023-05-22T16:03:38.834972+0000","last_peered":"2023-05-22T16:03:38.834972+0000","last_clean":"2023-05-22T16:03:38.834972+0000","last_became_active":"2023-05-22T16:03:38.446759+0000","last_became_peered":"2023-05-22T16:03:38.446759+0000","last_unstale":"2023-05-22T16:03:38.834972+0000","last_undegraded":"2023-05-22T16:03:38.834972+0000","last_fullsized":"2023-05-22T16:03:38.834972+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:03:37.383765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:03:37.383765+0000","last_clean_scrub_stamp":"2023-05-22T16:03:37.383765+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:55:15.045866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":566,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.1f","version":"0'0","reported_seq":9,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.835759+0000","last_change":"2023-05-22T16:03:38.807854+0000","last_active":"2023-05-22T16:03:38.835759+0000","last_peered":"2023-05-22T16:03:38.835759+0000","last_clean":"2023-05-22T16:03:38.835759+0000","last_became_active":"2023-05-22T16:03:38.807446+0000","last_became_peered":"2023-05-22T16:03:38.807446+0000","last_unstale":"2023-05-22T16:03:38.835759+0000","last_undegraded":"2023-05-22T16:03:38.835759+0000","last_fullsized":"2023-05-22T16:03:38.835759+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_clean_scrub_stamp":"2023-05-22T16:03:38.387135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T18:18:52.766623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.1e","version":"178'1","reported_seq":11,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.835813+0000","last_change":"2023-05-22T16:03:38.450008+0000","last_active":"2023-05-22T16:03:38.835813+0000","last_peered":"2023-05-22T16:03:38.835813+0000","last_clean":"2023-05-22T16:03:38.835813+0000","last_became_active":"2023-05-22T16:03:38.449244+0000","last_became_peered":"2023-05-22T16:03:38.449244+0000","last_unstale":"2023-05-22T16:03:38.835813+0000","last_undegraded":"2023-05-22T16:03:38.835813+0000","last_fullsized":"2023-05-22T16:03:38.835813+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:03:37.383765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:03:37.383765+0000","last_clean_scrub_stamp":"2023-05-22T16:03:37.383765+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:44:19.423716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.1c","version":"0'0","reported_seq":9,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.818860+0000","last_change":"2023-05-22T16:03:38.817798+0000","last_active":"2023-05-22T16:03:38.818860+0000","last_peered":"2023-05-22T16:03:38.818860+0000","last_clean":"2023-05-22T16:03:38.818860+0000","last_became_active":"2023-05-22T16:03:38.817461+0000","last_became_peered":"2023-05-22T16:03:38.817461+0000","last_unstale":"2023-05-22T16:03:38.818860+0000","last_undegraded":"2023-05-22T16:03:38.818860+0000","last_fullsized":"2023-05-22T16:03:38.818860+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_clean_scrub_stamp":"2023-05-22T16:03:38.387135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:28:05.049393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.1d","version":"178'2","reported_seq":12,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.833072+0000","last_change":"2023-05-22T16:03:38.452005+0000","last_active":"2023-05-22T16:03:38.833072+0000","last_peered":"2023-05-22T16:03:38.833072+0000","last_clean":"2023-05-22T16:03:38.833072+0000","last_became_active":"2023-05-22T16:03:38.451740+0000","last_became_peered":"2023-05-22T16:03:38.451740+0000","last_unstale":"2023-05-22T16:03:38.833072+0000","last_undegraded":"2023-05-22T16:03:38.833072+0000","last_fullsized":"2023-05-22T16:03:38.833072+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:03:37.383765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:03:37.383765+0000","last_clean_scrub_stamp":"2023-05-22T16:03:37.383765+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:01:48.284969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.1d","version":"0'0","reported_seq":9,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.833903+0000","last_change":"2023-05-22T16:03:38.821882+0000","last_active":"2023-05-22T16:03:38.833903+0000","last_peered":"2023-05-22T16:03:38.833903+0000","last_clean":"2023-05-22T16:03:38.833903+0000","last_became_active":"2023-05-22T16:03:38.821601+0000","last_became_peered":"2023-05-22T16:03:38.821601+0000","last_unstale":"2023-05-22T16:03:38.833903+0000","last_undegraded":"2023-05-22T16:03:38.833903+0000","last_fullsized":"2023-05-22T16:03:38.833903+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_clean_scrub_stamp":"2023-05-22T16:03:38.387135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T16:32:33.168433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.1c","version":"178'2","reported_seq":12,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.831140+0000","last_change":"2023-05-22T16:03:38.437032+0000","last_active":"2023-05-22T16:03:38.831140+0000","last_peered":"2023-05-22T16:03:38.831140+0000","last_clean":"2023-05-22T16:03:38.831140+0000","last_became_active":"2023-05-22T16:03:38.436774+0000","last_became_peered":"2023-05-22T16:03:38.436774+0000","last_unstale":"2023-05-22T16:03:38.831140+0000","last_undegraded":"2023-05-22T16:03:38.831140+0000","last_fullsized":"2023-05-22T16:03:38.831140+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:03:37.383765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:03:37.383765+0000","last_clean_scrub_stamp":"2023-05-22T16:03:37.383765+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:03:33.705779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.1a","version":"0'0","reported_seq":9,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.813045+0000","last_change":"2023-05-22T16:03:38.802058+0000","last_active":"2023-05-22T16:03:38.813045+0000","last_peered":"2023-05-22T16:03:38.813045+0000","last_clean":"2023-05-22T16:03:38.813045+0000","last_became_active":"2023-05-22T16:03:38.801322+0000","last_became_peered":"2023-05-22T16:03:38.801322+0000","last_unstale":"2023-05-22T16:03:38.813045+0000","last_undegraded":"2023-05-22T16:03:38.813045+0000","last_fullsized":"2023-05-22T16:03:38.813045+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_clean_scrub_stamp":"2023-05-22T16:03:38.387135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T03:55:40.501656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.1b","version":"178'1","reported_seq":11,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.837599+0000","last_change":"2023-05-22T16:03:38.435428+0000","last_active":"2023-05-22T16:03:38.837599+0000","last_peered":"2023-05-22T16:03:38.837599+0000","last_clean":"2023-05-22T16:03:38.837599+0000","last_became_active":"2023-05-22T16:03:38.434696+0000","last_became_peered":"2023-05-22T16:03:38.434696+0000","last_unstale":"2023-05-22T16:03:38.837599+0000","last_undegraded":"2023-05-22T16:03:38.837599+0000","last_fullsized":"2023-05-22T16:03:38.837599+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:03:37.383765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:03:37.383765+0000","last_clean_scrub_stamp":"2023-05-22T16:03:37.383765+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:16:15.825456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.1b","version":"0'0","reported_seq":9,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.821402+0000","last_change":"2023-05-22T16:03:38.817010+0000","last_active":"2023-05-22T16:03:38.821402+0000","last_peered":"2023-05-22T16:03:38.821402+0000","last_clean":"2023-05-22T16:03:38.821402+0000","last_became_active":"2023-05-22T16:03:38.816702+0000","last_became_peered":"2023-05-22T16:03:38.816702+0000","last_unstale":"2023-05-22T16:03:38.821402+0000","last_undegraded":"2023-05-22T16:03:38.821402+0000","last_fullsized":"2023-05-22T16:03:38.821402+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_clean_scrub_stamp":"2023-05-22T16:03:38.387135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:44:38.836588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.1a","version":"178'1","reported_seq":11,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.820144+0000","last_change":"2023-05-22T16:03:38.424525+0000","last_active":"2023-05-22T16:03:38.820144+0000","last_peered":"2023-05-22T16:03:38.820144+0000","last_clean":"2023-05-22T16:03:38.820144+0000","last_became_active":"2023-05-22T16:03:38.424291+0000","last_became_peered":"2023-05-22T16:03:38.424291+0000","last_unstale":"2023-05-22T16:03:38.820144+0000","last_undegraded":"2023-05-22T16:03:38.820144+0000","last_fullsized":"2023-05-22T16:03:38.820144+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:03:37.383765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:03:37.383765+0000","last_clean_scrub_stamp":"2023-05-22T16:03:37.383765+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:22:12.076229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.18","version":"0'0","reported_seq":9,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.832156+0000","last_change":"2023-05-22T16:03:38.831443+0000","last_active":"2023-05-22T16:03:38.832156+0000","last_peered":"2023-05-22T16:03:38.832156+0000","last_clean":"2023-05-22T16:03:38.832156+0000","last_became_active":"2023-05-22T16:03:38.831234+0000","last_became_peered":"2023-05-22T16:03:38.831234+0000","last_unstale":"2023-05-22T16:03:38.832156+0000","last_undegraded":"2023-05-22T16:03:38.832156+0000","last_fullsized":"2023-05-22T16:03:38.832156+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_clean_scrub_stamp":"2023-05-22T16:03:38.387135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T03:42:31.724542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.19","version":"0'0","reported_seq":10,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.804574+0000","last_change":"2023-05-22T16:03:38.442821+0000","last_active":"2023-05-22T16:03:38.804574+0000","last_peered":"2023-05-22T16:03:38.804574+0000","last_clean":"2023-05-22T16:03:38.804574+0000","last_became_active":"2023-05-22T16:03:38.439824+0000","last_became_peered":"2023-05-22T16:03:38.439824+0000","last_unstale":"2023-05-22T16:03:38.804574+0000","last_undegraded":"2023-05-22T16:03:38.804574+0000","last_fullsized":"2023-05-22T16:03:38.804574+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:03:37.383765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:03:37.383765+0000","last_clean_scrub_stamp":"2023-05-22T16:03:37.383765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:43:00.264838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.19","version":"0'0","reported_seq":9,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.817959+0000","last_change":"2023-05-22T16:03:38.809063+0000","last_active":"2023-05-22T16:03:38.817959+0000","last_peered":"2023-05-22T16:03:38.817959+0000","last_clean":"2023-05-22T16:03:38.817959+0000","last_became_active":"2023-05-22T16:03:38.803361+0000","last_became_peered":"2023-05-22T16:03:38.803361+0000","last_unstale":"2023-05-22T16:03:38.817959+0000","last_undegraded":"2023-05-22T16:03:38.817959+0000","last_fullsized":"2023-05-22T16:03:38.817959+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_clean_scrub_stamp":"2023-05-22T16:03:38.387135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:18:36.200155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.18","version":"0'0","reported_seq":10,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.793003+0000","last_change":"2023-05-22T16:03:38.421051+0000","last_active":"2023-05-22T16:03:38.793003+0000","last_peered":"2023-05-22T16:03:38.793003+0000","last_clean":"2023-05-22T16:03:38.793003+0000","last_became_active":"2023-05-22T16:03:38.420654+0000","last_became_peered":"2023-05-22T16:03:38.420654+0000","last_unstale":"2023-05-22T16:03:38.793003+0000","last_undegraded":"2023-05-22T16:03:38.793003+0000","last_fullsized":"2023-05-22T16:03:38.793003+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:03:37.383765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:03:37.383765+0000","last_clean_scrub_stamp":"2023-05-22T16:03:37.383765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:00:41.383707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.e","version":"0'0","reported_seq":9,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.821582+0000","last_change":"2023-05-22T16:03:38.818612+0000","last_active":"2023-05-22T16:03:38.821582+0000","last_peered":"2023-05-22T16:03:38.821582+0000","last_clean":"2023-05-22T16:03:38.821582+0000","last_became_active":"2023-05-22T16:03:38.817701+0000","last_became_peered":"2023-05-22T16:03:38.817701+0000","last_unstale":"2023-05-22T16:03:38.821582+0000","last_undegraded":"2023-05-22T16:03:38.821582+0000","last_fullsized":"2023-05-22T16:03:38.821582+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_clean_scrub_stamp":"2023-05-22T16:03:38.387135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:55:10.166213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.f","version":"178'1","reported_seq":11,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.838466+0000","last_change":"2023-05-22T16:03:38.443237+0000","last_active":"2023-05-22T16:03:38.838466+0000","last_peered":"2023-05-22T16:03:38.838466+0000","last_clean":"2023-05-22T16:03:38.838466+0000","last_became_active":"2023-05-22T16:03:38.442601+0000","last_became_peered":"2023-05-22T16:03:38.442601+0000","last_unstale":"2023-05-22T16:03:38.838466+0000","last_undegraded":"2023-05-22T16:03:38.838466+0000","last_fullsized":"2023-05-22T16:03:38.838466+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:03:37.383765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:03:37.383765+0000","last_clean_scrub_stamp":"2023-05-22T16:03:37.383765+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:18:35.196785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.f","version":"0'0","reported_seq":9,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.825950+0000","last_change":"2023-05-22T16:03:38.825070+0000","last_active":"2023-05-22T16:03:38.825950+0000","last_peered":"2023-05-22T16:03:38.825950+0000","last_clean":"2023-05-22T16:03:38.825950+0000","last_became_active":"2023-05-22T16:03:38.824696+0000","last_became_peered":"2023-05-22T16:03:38.824696+0000","last_unstale":"2023-05-22T16:03:38.825950+0000","last_undegraded":"2023-05-22T16:03:38.825950+0000","last_fullsized":"2023-05-22T16:03:38.825950+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_clean_scrub_stamp":"2023-05-22T16:03:38.387135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:34:38.930308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.e","version":"178'2","reported_seq":12,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.833978+0000","last_change":"2023-05-22T16:03:38.424783+0000","last_active":"2023-05-22T16:03:38.833978+0000","last_peered":"2023-05-22T16:03:38.833978+0000","last_clean":"2023-05-22T16:03:38.833978+0000","last_became_active":"2023-05-22T16:03:38.423990+0000","last_became_peered":"2023-05-22T16:03:38.423990+0000","last_unstale":"2023-05-22T16:03:38.833978+0000","last_undegraded":"2023-05-22T16:03:38.833978+0000","last_fullsized":"2023-05-22T16:03:38.833978+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:03:37.383765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:03:37.383765+0000","last_clean_scrub_stamp":"2023-05-22T16:03:37.383765+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:27:32.880811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.c","version":"0'0","reported_seq":9,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.813462+0000","last_change":"2023-05-22T16:03:38.810867+0000","last_active":"2023-05-22T16:03:38.813462+0000","last_peered":"2023-05-22T16:03:38.813462+0000","last_clean":"2023-05-22T16:03:38.813462+0000","last_became_active":"2023-05-22T16:03:38.810495+0000","last_became_peered":"2023-05-22T16:03:38.810495+0000","last_unstale":"2023-05-22T16:03:38.813462+0000","last_undegraded":"2023-05-22T16:03:38.813462+0000","last_fullsized":"2023-05-22T16:03:38.813462+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_clean_scrub_stamp":"2023-05-22T16:03:38.387135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:29:41.023569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.d","version":"178'1","reported_seq":11,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.836748+0000","last_change":"2023-05-22T16:03:38.439903+0000","last_active":"2023-05-22T16:03:38.836748+0000","last_peered":"2023-05-22T16:03:38.836748+0000","last_clean":"2023-05-22T16:03:38.836748+0000","last_became_active":"2023-05-22T16:03:38.439194+0000","last_became_peered":"2023-05-22T16:03:38.439194+0000","last_unstale":"2023-05-22T16:03:38.836748+0000","last_undegraded":"2023-05-22T16:03:38.836748+0000","last_fullsized":"2023-05-22T16:03:38.836748+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:03:37.383765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:03:37.383765+0000","last_clean_scrub_stamp":"2023-05-22T16:03:37.383765+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T18:18:51.763252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.d","version":"0'0","reported_seq":9,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.825900+0000","last_change":"2023-05-22T16:03:38.824954+0000","last_active":"2023-05-22T16:03:38.825900+0000","last_peered":"2023-05-22T16:03:38.825900+0000","last_clean":"2023-05-22T16:03:38.825900+0000","last_became_active":"2023-05-22T16:03:38.824677+0000","last_became_peered":"2023-05-22T16:03:38.824677+0000","last_unstale":"2023-05-22T16:03:38.825900+0000","last_undegraded":"2023-05-22T16:03:38.825900+0000","last_fullsized":"2023-05-22T16:03:38.825900+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_clean_scrub_stamp":"2023-05-22T16:03:38.387135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:35:24.987929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.c","version":"0'0","reported_seq":10,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.795221+0000","last_change":"2023-05-22T16:03:38.433373+0000","last_active":"2023-05-22T16:03:38.795221+0000","last_peered":"2023-05-22T16:03:38.795221+0000","last_clean":"2023-05-22T16:03:38.795221+0000","last_became_active":"2023-05-22T16:03:38.432357+0000","last_became_peered":"2023-05-22T16:03:38.432357+0000","last_unstale":"2023-05-22T16:03:38.795221+0000","last_undegraded":"2023-05-22T16:03:38.795221+0000","last_fullsized":"2023-05-22T16:03:38.795221+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:03:37.383765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:03:37.383765+0000","last_clean_scrub_stamp":"2023-05-22T16:03:37.383765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:35:31.827140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.a","version":"0'0","reported_seq":9,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.829742+0000","last_change":"2023-05-22T16:03:38.806843+0000","last_active":"2023-05-22T16:03:38.829742+0000","last_peered":"2023-05-22T16:03:38.829742+0000","last_clean":"2023-05-22T16:03:38.829742+0000","last_became_active":"2023-05-22T16:03:38.806337+0000","last_became_peered":"2023-05-22T16:03:38.806337+0000","last_unstale":"2023-05-22T16:03:38.829742+0000","last_undegraded":"2023-05-22T16:03:38.829742+0000","last_fullsized":"2023-05-22T16:03:38.829742+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_clean_scrub_stamp":"2023-05-22T16:03:38.387135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:46:58.323247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.b","version":"178'2","reported_seq":12,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.832800+0000","last_change":"2023-05-22T16:03:38.444272+0000","last_active":"2023-05-22T16:03:38.832800+0000","last_peered":"2023-05-22T16:03:38.832800+0000","last_clean":"2023-05-22T16:03:38.832800+0000","last_became_active":"2023-05-22T16:03:38.444069+0000","last_became_peered":"2023-05-22T16:03:38.444069+0000","last_unstale":"2023-05-22T16:03:38.832800+0000","last_undegraded":"2023-05-22T16:03:38.832800+0000","last_fullsized":"2023-05-22T16:03:38.832800+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:03:37.383765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:03:37.383765+0000","last_clean_scrub_stamp":"2023-05-22T16:03:37.383765+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T03:47:45.896671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.b","version":"0'0","reported_seq":9,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.821783+0000","last_change":"2023-05-22T16:03:38.820971+0000","last_active":"2023-05-22T16:03:38.821783+0000","last_peered":"2023-05-22T16:03:38.821783+0000","last_clean":"2023-05-22T16:03:38.821783+0000","last_became_active":"2023-05-22T16:03:38.820633+0000","last_became_peered":"2023-05-22T16:03:38.820633+0000","last_unstale":"2023-05-22T16:03:38.821783+0000","last_undegraded":"2023-05-22T16:03:38.821783+0000","last_fullsized":"2023-05-22T16:03:38.821783+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_clean_scrub_stamp":"2023-05-22T16:03:38.387135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:45:10.755716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.a","version":"0'0","reported_seq":10,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.794722+0000","last_change":"2023-05-22T16:03:38.431470+0000","last_active":"2023-05-22T16:03:38.794722+0000","last_peered":"2023-05-22T16:03:38.794722+0000","last_clean":"2023-05-22T16:03:38.794722+0000","last_became_active":"2023-05-22T16:03:38.430646+0000","last_became_peered":"2023-05-22T16:03:38.430646+0000","last_unstale":"2023-05-22T16:03:38.794722+0000","last_undegraded":"2023-05-22T16:03:38.794722+0000","last_fullsized":"2023-05-22T16:03:38.794722+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:03:37.383765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:03:37.383765+0000","last_clean_scrub_stamp":"2023-05-22T16:03:37.383765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:30:37.858474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.8","version":"0'0","reported_seq":9,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.834928+0000","last_change":"2023-05-22T16:03:38.831619+0000","last_active":"2023-05-22T16:03:38.834928+0000","last_peered":"2023-05-22T16:03:38.834928+0000","last_clean":"2023-05-22T16:03:38.834928+0000","last_became_active":"2023-05-22T16:03:38.831313+0000","last_became_peered":"2023-05-22T16:03:38.831313+0000","last_unstale":"2023-05-22T16:03:38.834928+0000","last_undegraded":"2023-05-22T16:03:38.834928+0000","last_fullsized":"2023-05-22T16:03:38.834928+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_clean_scrub_stamp":"2023-05-22T16:03:38.387135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T03:37:21.656379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.9","version":"0'0","reported_seq":10,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.793329+0000","last_change":"2023-05-22T16:03:38.454438+0000","last_active":"2023-05-22T16:03:38.793329+0000","last_peered":"2023-05-22T16:03:38.793329+0000","last_clean":"2023-05-22T16:03:38.793329+0000","last_became_active":"2023-05-22T16:03:38.453989+0000","last_became_peered":"2023-05-22T16:03:38.453989+0000","last_unstale":"2023-05-22T16:03:38.793329+0000","last_undegraded":"2023-05-22T16:03:38.793329+0000","last_fullsized":"2023-05-22T16:03:38.793329+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:03:37.383765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:03:37.383765+0000","last_clean_scrub_stamp":"2023-05-22T16:03:37.383765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:57:38.049040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.9","version":"0'0","reported_seq":9,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.829435+0000","last_change":"2023-05-22T16:03:38.828625+0000","last_active":"2023-05-22T16:03:38.829435+0000","last_peered":"2023-05-22T16:03:38.829435+0000","last_clean":"2023-05-22T16:03:38.829435+0000","last_became_active":"2023-05-22T16:03:38.828315+0000","last_became_peered":"2023-05-22T16:03:38.828315+0000","last_unstale":"2023-05-22T16:03:38.829435+0000","last_undegraded":"2023-05-22T16:03:38.829435+0000","last_fullsized":"2023-05-22T16:03:38.829435+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_clean_scrub_stamp":"2023-05-22T16:03:38.387135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:46:56.808362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.8","version":"0'0","reported_seq":10,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.812433+0000","last_change":"2023-05-22T16:03:38.436430+0000","last_active":"2023-05-22T16:03:38.812433+0000","last_peered":"2023-05-22T16:03:38.812433+0000","last_clean":"2023-05-22T16:03:38.812433+0000","last_became_active":"2023-05-22T16:03:38.434194+0000","last_became_peered":"2023-05-22T16:03:38.434194+0000","last_unstale":"2023-05-22T16:03:38.812433+0000","last_undegraded":"2023-05-22T16:03:38.812433+0000","last_fullsized":"2023-05-22T16:03:38.812433+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:03:37.383765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:03:37.383765+0000","last_clean_scrub_stamp":"2023-05-22T16:03:37.383765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:34:37.926938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.6","version":"0'0","reported_seq":9,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.821360+0000","last_change":"2023-05-22T16:03:38.814348+0000","last_active":"2023-05-22T16:03:38.821360+0000","last_peered":"2023-05-22T16:03:38.821360+0000","last_clean":"2023-05-22T16:03:38.821360+0000","last_became_active":"2023-05-22T16:03:38.813914+0000","last_became_peered":"2023-05-22T16:03:38.813914+0000","last_unstale":"2023-05-22T16:03:38.821360+0000","last_undegraded":"2023-05-22T16:03:38.821360+0000","last_fullsized":"2023-05-22T16:03:38.821360+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_clean_scrub_stamp":"2023-05-22T16:03:38.387135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:08:59.616966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.7","version":"178'2","reported_seq":12,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.832107+0000","last_change":"2023-05-22T16:03:38.451344+0000","last_active":"2023-05-22T16:03:38.832107+0000","last_peered":"2023-05-22T16:03:38.832107+0000","last_clean":"2023-05-22T16:03:38.832107+0000","last_became_active":"2023-05-22T16:03:38.451136+0000","last_became_peered":"2023-05-22T16:03:38.451136+0000","last_unstale":"2023-05-22T16:03:38.832107+0000","last_undegraded":"2023-05-22T16:03:38.832107+0000","last_fullsized":"2023-05-22T16:03:38.832107+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:03:37.383765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:03:37.383765+0000","last_clean_scrub_stamp":"2023-05-22T16:03:37.383765+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:20:28.887546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.7","version":"0'0","reported_seq":9,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.823748+0000","last_change":"2023-05-22T16:03:38.822821+0000","last_active":"2023-05-22T16:03:38.823748+0000","last_peered":"2023-05-22T16:03:38.823748+0000","last_clean":"2023-05-22T16:03:38.823748+0000","last_became_active":"2023-05-22T16:03:38.822541+0000","last_became_peered":"2023-05-22T16:03:38.822541+0000","last_unstale":"2023-05-22T16:03:38.823748+0000","last_undegraded":"2023-05-22T16:03:38.823748+0000","last_fullsized":"2023-05-22T16:03:38.823748+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_clean_scrub_stamp":"2023-05-22T16:03:38.387135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:02:50.415413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.6","version":"0'0","reported_seq":10,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.803747+0000","last_change":"2023-05-22T16:03:38.433322+0000","last_active":"2023-05-22T16:03:38.803747+0000","last_peered":"2023-05-22T16:03:38.803747+0000","last_clean":"2023-05-22T16:03:38.803747+0000","last_became_active":"2023-05-22T16:03:38.432663+0000","last_became_peered":"2023-05-22T16:03:38.432663+0000","last_unstale":"2023-05-22T16:03:38.803747+0000","last_undegraded":"2023-05-22T16:03:38.803747+0000","last_fullsized":"2023-05-22T16:03:38.803747+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:03:37.383765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:03:37.383765+0000","last_clean_scrub_stamp":"2023-05-22T16:03:37.383765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:32:39.325570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.4","version":"0'0","reported_seq":9,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.812851+0000","last_change":"2023-05-22T16:03:38.800848+0000","last_active":"2023-05-22T16:03:38.812851+0000","last_peered":"2023-05-22T16:03:38.812851+0000","last_clean":"2023-05-22T16:03:38.812851+0000","last_became_active":"2023-05-22T16:03:38.800037+0000","last_became_peered":"2023-05-22T16:03:38.800037+0000","last_unstale":"2023-05-22T16:03:38.812851+0000","last_undegraded":"2023-05-22T16:03:38.812851+0000","last_fullsized":"2023-05-22T16:03:38.812851+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_clean_scrub_stamp":"2023-05-22T16:03:38.387135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:36:22.242179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.5","version":"0'0","reported_seq":10,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.803738+0000","last_change":"2023-05-22T16:03:38.441974+0000","last_active":"2023-05-22T16:03:38.803738+0000","last_peered":"2023-05-22T16:03:38.803738+0000","last_clean":"2023-05-22T16:03:38.803738+0000","last_became_active":"2023-05-22T16:03:38.441132+0000","last_became_peered":"2023-05-22T16:03:38.441132+0000","last_unstale":"2023-05-22T16:03:38.803738+0000","last_undegraded":"2023-05-22T16:03:38.803738+0000","last_fullsized":"2023-05-22T16:03:38.803738+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:03:37.383765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:03:37.383765+0000","last_clean_scrub_stamp":"2023-05-22T16:03:37.383765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:49:44.109377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.5","version":"0'0","reported_seq":9,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.821456+0000","last_change":"2023-05-22T16:03:38.818770+0000","last_active":"2023-05-22T16:03:38.821456+0000","last_peered":"2023-05-22T16:03:38.821456+0000","last_clean":"2023-05-22T16:03:38.821456+0000","last_became_active":"2023-05-22T16:03:38.818427+0000","last_became_peered":"2023-05-22T16:03:38.818427+0000","last_unstale":"2023-05-22T16:03:38.821456+0000","last_undegraded":"2023-05-22T16:03:38.821456+0000","last_fullsized":"2023-05-22T16:03:38.821456+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_clean_scrub_stamp":"2023-05-22T16:03:38.387135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T03:37:21.656379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.4","version":"0'0","reported_seq":10,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.794878+0000","last_change":"2023-05-22T16:03:38.454342+0000","last_active":"2023-05-22T16:03:38.794878+0000","last_peered":"2023-05-22T16:03:38.794878+0000","last_clean":"2023-05-22T16:03:38.794878+0000","last_became_active":"2023-05-22T16:03:38.453961+0000","last_became_peered":"2023-05-22T16:03:38.453961+0000","last_unstale":"2023-05-22T16:03:38.794878+0000","last_undegraded":"2023-05-22T16:03:38.794878+0000","last_fullsized":"2023-05-22T16:03:38.794878+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:03:37.383765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:03:37.383765+0000","last_clean_scrub_stamp":"2023-05-22T16:03:37.383765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:44:19.423716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.2","version":"0'0","reported_seq":9,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.809003+0000","last_change":"2023-05-22T16:03:38.804374+0000","last_active":"2023-05-22T16:03:38.809003+0000","last_peered":"2023-05-22T16:03:38.809003+0000","last_clean":"2023-05-22T16:03:38.809003+0000","last_became_active":"2023-05-22T16:03:38.803971+0000","last_became_peered":"2023-05-22T16:03:38.803971+0000","last_unstale":"2023-05-22T16:03:38.809003+0000","last_undegraded":"2023-05-22T16:03:38.809003+0000","last_fullsized":"2023-05-22T16:03:38.809003+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_clean_scrub_stamp":"2023-05-22T16:03:38.387135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:53:38.976658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.3","version":"0'0","reported_seq":10,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.793613+0000","last_change":"2023-05-22T16:03:38.432703+0000","last_active":"2023-05-22T16:03:38.793613+0000","last_peered":"2023-05-22T16:03:38.793613+0000","last_clean":"2023-05-22T16:03:38.793613+0000","last_became_active":"2023-05-22T16:03:38.431828+0000","last_became_peered":"2023-05-22T16:03:38.431828+0000","last_unstale":"2023-05-22T16:03:38.793613+0000","last_undegraded":"2023-05-22T16:03:38.793613+0000","last_fullsized":"2023-05-22T16:03:38.793613+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:03:37.383765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:03:37.383765+0000","last_clean_scrub_stamp":"2023-05-22T16:03:37.383765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:22:35.883553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.3","version":"0'0","reported_seq":9,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.836119+0000","last_change":"2023-05-22T16:03:38.818400+0000","last_active":"2023-05-22T16:03:38.836119+0000","last_peered":"2023-05-22T16:03:38.836119+0000","last_clean":"2023-05-22T16:03:38.836119+0000","last_became_active":"2023-05-22T16:03:38.817915+0000","last_became_peered":"2023-05-22T16:03:38.817915+0000","last_unstale":"2023-05-22T16:03:38.836119+0000","last_undegraded":"2023-05-22T16:03:38.836119+0000","last_fullsized":"2023-05-22T16:03:38.836119+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_clean_scrub_stamp":"2023-05-22T16:03:38.387135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T18:39:51.525114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.2","version":"0'0","reported_seq":10,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.793747+0000","last_change":"2023-05-22T16:03:38.442188+0000","last_active":"2023-05-22T16:03:38.793747+0000","last_peered":"2023-05-22T16:03:38.793747+0000","last_clean":"2023-05-22T16:03:38.793747+0000","last_became_active":"2023-05-22T16:03:38.441873+0000","last_became_peered":"2023-05-22T16:03:38.441873+0000","last_unstale":"2023-05-22T16:03:38.793747+0000","last_undegraded":"2023-05-22T16:03:38.793747+0000","last_fullsized":"2023-05-22T16:03:38.793747+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:03:37.383765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:03:37.383765+0000","last_clean_scrub_stamp":"2023-05-22T16:03:37.383765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:57:02.653723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.0","version":"0'0","reported_seq":9,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.820779+0000","last_change":"2023-05-22T16:03:38.819579+0000","last_active":"2023-05-22T16:03:38.820779+0000","last_peered":"2023-05-22T16:03:38.820779+0000","last_clean":"2023-05-22T16:03:38.820779+0000","last_became_active":"2023-05-22T16:03:38.819241+0000","last_became_peered":"2023-05-22T16:03:38.819241+0000","last_unstale":"2023-05-22T16:03:38.820779+0000","last_undegraded":"2023-05-22T16:03:38.820779+0000","last_fullsized":"2023-05-22T16:03:38.820779+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_clean_scrub_stamp":"2023-05-22T16:03:38.387135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:07:09.650623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.1","version":"0'0","reported_seq":10,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.804786+0000","last_change":"2023-05-22T16:03:38.443276+0000","last_active":"2023-05-22T16:03:38.804786+0000","last_peered":"2023-05-22T16:03:38.804786+0000","last_clean":"2023-05-22T16:03:38.804786+0000","last_became_active":"2023-05-22T16:03:38.433576+0000","last_became_peered":"2023-05-22T16:03:38.433576+0000","last_unstale":"2023-05-22T16:03:38.804786+0000","last_undegraded":"2023-05-22T16:03:38.804786+0000","last_fullsized":"2023-05-22T16:03:38.804786+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:03:37.383765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:03:37.383765+0000","last_clean_scrub_stamp":"2023-05-22T16:03:37.383765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:46:57.319877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.1","version":"0'0","reported_seq":9,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.833912+0000","last_change":"2023-05-22T16:03:38.819249+0000","last_active":"2023-05-22T16:03:38.833912+0000","last_peered":"2023-05-22T16:03:38.833912+0000","last_clean":"2023-05-22T16:03:38.833912+0000","last_became_active":"2023-05-22T16:03:38.818956+0000","last_became_peered":"2023-05-22T16:03:38.818956+0000","last_unstale":"2023-05-22T16:03:38.833912+0000","last_undegraded":"2023-05-22T16:03:38.833912+0000","last_fullsized":"2023-05-22T16:03:38.833912+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_clean_scrub_stamp":"2023-05-22T16:03:38.387135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:59:32.502376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.0","version":"0'0","reported_seq":10,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.793886+0000","last_change":"2023-05-22T16:03:38.430698+0000","last_active":"2023-05-22T16:03:38.793886+0000","last_peered":"2023-05-22T16:03:38.793886+0000","last_clean":"2023-05-22T16:03:38.793886+0000","last_became_active":"2023-05-22T16:03:38.429895+0000","last_became_peered":"2023-05-22T16:03:38.429895+0000","last_unstale":"2023-05-22T16:03:38.793886+0000","last_undegraded":"2023-05-22T16:03:38.793886+0000","last_fullsized":"2023-05-22T16:03:38.793886+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:03:37.383765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:03:37.383765+0000","last_clean_scrub_stamp":"2023-05-22T16:03:37.383765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:16:36.548828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.16","version":"0'0","reported_seq":9,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.821384+0000","last_change":"2023-05-22T16:03:38.814445+0000","last_active":"2023-05-22T16:03:38.821384+0000","last_peered":"2023-05-22T16:03:38.821384+0000","last_clean":"2023-05-22T16:03:38.821384+0000","last_became_active":"2023-05-22T16:03:38.813749+0000","last_became_peered":"2023-05-22T16:03:38.813749+0000","last_unstale":"2023-05-22T16:03:38.821384+0000","last_undegraded":"2023-05-22T16:03:38.821384+0000","last_fullsized":"2023-05-22T16:03:38.821384+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_clean_scrub_stamp":"2023-05-22T16:03:38.387135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:45:10.755716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.17","version":"0'0","reported_seq":10,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.793248+0000","last_change":"2023-05-22T16:03:38.431425+0000","last_active":"2023-05-22T16:03:38.793248+0000","last_peered":"2023-05-22T16:03:38.793248+0000","last_clean":"2023-05-22T16:03:38.793248+0000","last_became_active":"2023-05-22T16:03:38.430685+0000","last_became_peered":"2023-05-22T16:03:38.430685+0000","last_unstale":"2023-05-22T16:03:38.793248+0000","last_undegraded":"2023-05-22T16:03:38.793248+0000","last_fullsized":"2023-05-22T16:03:38.793248+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:03:37.383765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:03:37.383765+0000","last_clean_scrub_stamp":"2023-05-22T16:03:37.383765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T03:39:03.580030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"166'304","reported_seq":985,"reported_epoch":177,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.406538+0000","last_change":"2023-05-22T15:26:39.318204+0000","last_active":"2023-05-22T16:03:38.406538+0000","last_peered":"2023-05-22T16:03:38.406538+0000","last_clean":"2023-05-22T16:03:38.406538+0000","last_became_active":"2023-05-22T15:26:39.315370+0000","last_became_peered":"2023-05-22T15:26:39.315370+0000","last_unstale":"2023-05-22T16:03:38.406538+0000","last_undegraded":"2023-05-22T16:03:38.406538+0000","last_fullsized":"2023-05-22T16:03:38.406538+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:38.020076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:38.020076+0000","last_clean_scrub_stamp":"2023-05-22T15:26:38.020076+0000","objects_scrubbed":0,"log_size":304,"log_dups_size":0,"ondisk_log_size":304,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:06:39.207543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1748,"num_read_kb":3602,"num_write":499,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":178,"reported_epoch":177,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.409433+0000","last_change":"2023-05-22T15:26:44.971951+0000","last_active":"2023-05-22T16:03:38.409433+0000","last_peered":"2023-05-22T16:03:38.409433+0000","last_clean":"2023-05-22T16:03:38.409433+0000","last_became_active":"2023-05-22T15:26:44.970872+0000","last_became_peered":"2023-05-22T15:26:44.970872+0000","last_unstale":"2023-05-22T16:03:38.409433+0000","last_undegraded":"2023-05-22T16:03:38.409433+0000","last_fullsized":"2023-05-22T16:03:38.409433+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:55:34.752949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.15","version":"0'0","reported_seq":9,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.837225+0000","last_change":"2023-05-22T16:03:38.825114+0000","last_active":"2023-05-22T16:03:38.837225+0000","last_peered":"2023-05-22T16:03:38.837225+0000","last_clean":"2023-05-22T16:03:38.837225+0000","last_became_active":"2023-05-22T16:03:38.824362+0000","last_became_peered":"2023-05-22T16:03:38.824362+0000","last_unstale":"2023-05-22T16:03:38.837225+0000","last_undegraded":"2023-05-22T16:03:38.837225+0000","last_fullsized":"2023-05-22T16:03:38.837225+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_clean_scrub_stamp":"2023-05-22T16:03:38.387135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:30:18.570472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.14","version":"178'6","reported_seq":16,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.841501+0000","last_change":"2023-05-22T16:03:38.434806+0000","last_active":"2023-05-22T16:03:38.841501+0000","last_peered":"2023-05-22T16:03:38.841501+0000","last_clean":"2023-05-22T16:03:38.841501+0000","last_became_active":"2023-05-22T16:03:38.434146+0000","last_became_peered":"2023-05-22T16:03:38.434146+0000","last_unstale":"2023-05-22T16:03:38.841501+0000","last_undegraded":"2023-05-22T16:03:38.841501+0000","last_fullsized":"2023-05-22T16:03:38.841501+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:03:37.383765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:03:37.383765+0000","last_clean_scrub_stamp":"2023-05-22T16:03:37.383765+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:40:13.538937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1548,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":169,"reported_epoch":177,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.406328+0000","last_change":"2023-05-22T15:26:44.977198+0000","last_active":"2023-05-22T16:03:38.406328+0000","last_peered":"2023-05-22T16:03:38.406328+0000","last_clean":"2023-05-22T16:03:38.406328+0000","last_became_active":"2023-05-22T15:26:44.976445+0000","last_became_peered":"2023-05-22T15:26:44.976445+0000","last_unstale":"2023-05-22T16:03:38.406328+0000","last_undegraded":"2023-05-22T16:03:38.406328+0000","last_fullsized":"2023-05-22T16:03:38.406328+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:12:41.464054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.14","version":"0'0","reported_seq":9,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.821447+0000","last_change":"2023-05-22T16:03:38.814616+0000","last_active":"2023-05-22T16:03:38.821447+0000","last_peered":"2023-05-22T16:03:38.821447+0000","last_clean":"2023-05-22T16:03:38.821447+0000","last_became_active":"2023-05-22T16:03:38.814239+0000","last_became_peered":"2023-05-22T16:03:38.814239+0000","last_unstale":"2023-05-22T16:03:38.821447+0000","last_undegraded":"2023-05-22T16:03:38.821447+0000","last_fullsized":"2023-05-22T16:03:38.821447+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_clean_scrub_stamp":"2023-05-22T16:03:38.387135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:56:21.206906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.15","version":"178'3","reported_seq":13,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.827868+0000","last_change":"2023-05-22T16:03:38.422653+0000","last_active":"2023-05-22T16:03:38.827868+0000","last_peered":"2023-05-22T16:03:38.827868+0000","last_clean":"2023-05-22T16:03:38.827868+0000","last_became_active":"2023-05-22T16:03:38.422328+0000","last_became_peered":"2023-05-22T16:03:38.422328+0000","last_unstale":"2023-05-22T16:03:38.827868+0000","last_undegraded":"2023-05-22T16:03:38.827868+0000","last_fullsized":"2023-05-22T16:03:38.827868+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:03:37.383765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:03:37.383765+0000","last_clean_scrub_stamp":"2023-05-22T16:03:37.383765+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:13:15.091000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":171,"reported_epoch":177,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.397500+0000","last_change":"2023-05-22T15:26:44.969469+0000","last_active":"2023-05-22T16:03:38.397500+0000","last_peered":"2023-05-22T16:03:38.397500+0000","last_clean":"2023-05-22T16:03:38.397500+0000","last_became_active":"2023-05-22T15:26:44.969083+0000","last_became_peered":"2023-05-22T15:26:44.969083+0000","last_unstale":"2023-05-22T16:03:38.397500+0000","last_undegraded":"2023-05-22T16:03:38.397500+0000","last_fullsized":"2023-05-22T16:03:38.397500+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:29:23.670067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.17","version":"0'0","reported_seq":9,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.821348+0000","last_change":"2023-05-22T16:03:38.814628+0000","last_active":"2023-05-22T16:03:38.821348+0000","last_peered":"2023-05-22T16:03:38.821348+0000","last_clean":"2023-05-22T16:03:38.821348+0000","last_became_active":"2023-05-22T16:03:38.813818+0000","last_became_peered":"2023-05-22T16:03:38.813818+0000","last_unstale":"2023-05-22T16:03:38.821348+0000","last_undegraded":"2023-05-22T16:03:38.821348+0000","last_fullsized":"2023-05-22T16:03:38.821348+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_clean_scrub_stamp":"2023-05-22T16:03:38.387135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:30:18.570472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.16","version":"0'0","reported_seq":10,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.787619+0000","last_change":"2023-05-22T16:03:38.433642+0000","last_active":"2023-05-22T16:03:38.787619+0000","last_peered":"2023-05-22T16:03:38.787619+0000","last_clean":"2023-05-22T16:03:38.787619+0000","last_became_active":"2023-05-22T16:03:38.433310+0000","last_became_peered":"2023-05-22T16:03:38.433310+0000","last_unstale":"2023-05-22T16:03:38.787619+0000","last_undegraded":"2023-05-22T16:03:38.787619+0000","last_fullsized":"2023-05-22T16:03:38.787619+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:03:37.383765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:03:37.383765+0000","last_clean_scrub_stamp":"2023-05-22T16:03:37.383765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:19:25.192874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":178,"reported_epoch":177,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.411957+0000","last_change":"2023-05-22T15:26:44.976632+0000","last_active":"2023-05-22T16:03:38.411957+0000","last_peered":"2023-05-22T16:03:38.411957+0000","last_clean":"2023-05-22T16:03:38.411957+0000","last_became_active":"2023-05-22T15:26:44.975546+0000","last_became_peered":"2023-05-22T15:26:44.975546+0000","last_unstale":"2023-05-22T16:03:38.411957+0000","last_undegraded":"2023-05-22T16:03:38.411957+0000","last_fullsized":"2023-05-22T16:03:38.411957+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:42:57.973030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.11","version":"0'0","reported_seq":9,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.837268+0000","last_change":"2023-05-22T16:03:38.827159+0000","last_active":"2023-05-22T16:03:38.837268+0000","last_peered":"2023-05-22T16:03:38.837268+0000","last_clean":"2023-05-22T16:03:38.837268+0000","last_became_active":"2023-05-22T16:03:38.826600+0000","last_became_peered":"2023-05-22T16:03:38.826600+0000","last_unstale":"2023-05-22T16:03:38.837268+0000","last_undegraded":"2023-05-22T16:03:38.837268+0000","last_fullsized":"2023-05-22T16:03:38.837268+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_clean_scrub_stamp":"2023-05-22T16:03:38.387135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T03:47:46.900041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.10","version":"178'8","reported_seq":18,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.838027+0000","last_change":"2023-05-22T16:03:38.432666+0000","last_active":"2023-05-22T16:03:38.838027+0000","last_peered":"2023-05-22T16:03:38.838027+0000","last_clean":"2023-05-22T16:03:38.838027+0000","last_became_active":"2023-05-22T16:03:38.431969+0000","last_became_peered":"2023-05-22T16:03:38.431969+0000","last_unstale":"2023-05-22T16:03:38.838027+0000","last_undegraded":"2023-05-22T16:03:38.838027+0000","last_fullsized":"2023-05-22T16:03:38.838027+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:03:37.383765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:03:37.383765+0000","last_clean_scrub_stamp":"2023-05-22T16:03:37.383765+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:00:45.061713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":159,"reported_epoch":177,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.421536+0000","last_change":"2023-05-22T15:26:45.091475+0000","last_active":"2023-05-22T16:03:38.421536+0000","last_peered":"2023-05-22T16:03:38.421536+0000","last_clean":"2023-05-22T16:03:38.421536+0000","last_became_active":"2023-05-22T15:26:45.091143+0000","last_became_peered":"2023-05-22T15:26:45.091143+0000","last_unstale":"2023-05-22T16:03:38.421536+0000","last_undegraded":"2023-05-22T16:03:38.421536+0000","last_fullsized":"2023-05-22T16:03:38.421536+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:12:41.464054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.10","version":"0'0","reported_seq":9,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.813416+0000","last_change":"2023-05-22T16:03:38.810097+0000","last_active":"2023-05-22T16:03:38.813416+0000","last_peered":"2023-05-22T16:03:38.813416+0000","last_clean":"2023-05-22T16:03:38.813416+0000","last_became_active":"2023-05-22T16:03:38.809662+0000","last_became_peered":"2023-05-22T16:03:38.809662+0000","last_unstale":"2023-05-22T16:03:38.813416+0000","last_undegraded":"2023-05-22T16:03:38.813416+0000","last_fullsized":"2023-05-22T16:03:38.813416+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_clean_scrub_stamp":"2023-05-22T16:03:38.387135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:46:34.382160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.11","version":"0'0","reported_seq":10,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.793236+0000","last_change":"2023-05-22T16:03:38.434216+0000","last_active":"2023-05-22T16:03:38.793236+0000","last_peered":"2023-05-22T16:03:38.793236+0000","last_clean":"2023-05-22T16:03:38.793236+0000","last_became_active":"2023-05-22T16:03:38.433896+0000","last_became_peered":"2023-05-22T16:03:38.433896+0000","last_unstale":"2023-05-22T16:03:38.793236+0000","last_undegraded":"2023-05-22T16:03:38.793236+0000","last_fullsized":"2023-05-22T16:03:38.793236+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:03:37.383765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:03:37.383765+0000","last_clean_scrub_stamp":"2023-05-22T16:03:37.383765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T16:32:32.165063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":168,"reported_epoch":177,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.408860+0000","last_change":"2023-05-22T15:26:45.092367+0000","last_active":"2023-05-22T16:03:38.408860+0000","last_peered":"2023-05-22T16:03:38.408860+0000","last_clean":"2023-05-22T16:03:38.408860+0000","last_became_active":"2023-05-22T15:26:45.091951+0000","last_became_peered":"2023-05-22T15:26:45.091951+0000","last_unstale":"2023-05-22T16:03:38.408860+0000","last_undegraded":"2023-05-22T16:03:38.408860+0000","last_fullsized":"2023-05-22T16:03:38.408860+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:42:57.973030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.13","version":"0'0","reported_seq":9,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.819120+0000","last_change":"2023-05-22T16:03:38.817592+0000","last_active":"2023-05-22T16:03:38.819120+0000","last_peered":"2023-05-22T16:03:38.819120+0000","last_clean":"2023-05-22T16:03:38.819120+0000","last_became_active":"2023-05-22T16:03:38.816466+0000","last_became_peered":"2023-05-22T16:03:38.816466+0000","last_unstale":"2023-05-22T16:03:38.819120+0000","last_undegraded":"2023-05-22T16:03:38.819120+0000","last_fullsized":"2023-05-22T16:03:38.819120+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_clean_scrub_stamp":"2023-05-22T16:03:38.387135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T03:37:21.656379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.12","version":"0'0","reported_seq":10,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.793487+0000","last_change":"2023-05-22T16:03:38.426071+0000","last_active":"2023-05-22T16:03:38.793487+0000","last_peered":"2023-05-22T16:03:38.793487+0000","last_clean":"2023-05-22T16:03:38.793487+0000","last_became_active":"2023-05-22T16:03:38.425620+0000","last_became_peered":"2023-05-22T16:03:38.425620+0000","last_unstale":"2023-05-22T16:03:38.793487+0000","last_undegraded":"2023-05-22T16:03:38.793487+0000","last_fullsized":"2023-05-22T16:03:38.793487+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:03:37.383765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:03:37.383765+0000","last_clean_scrub_stamp":"2023-05-22T16:03:37.383765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:22:35.883553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":159,"reported_epoch":177,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.421501+0000","last_change":"2023-05-22T15:26:44.978858+0000","last_active":"2023-05-22T16:03:38.421501+0000","last_peered":"2023-05-22T16:03:38.421501+0000","last_clean":"2023-05-22T16:03:38.421501+0000","last_became_active":"2023-05-22T15:26:44.978443+0000","last_became_peered":"2023-05-22T15:26:44.978443+0000","last_unstale":"2023-05-22T16:03:38.421501+0000","last_undegraded":"2023-05-22T16:03:38.421501+0000","last_fullsized":"2023-05-22T16:03:38.421501+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:46:29.290243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.12","version":"0'0","reported_seq":9,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.830554+0000","last_change":"2023-05-22T16:03:38.829555+0000","last_active":"2023-05-22T16:03:38.830554+0000","last_peered":"2023-05-22T16:03:38.830554+0000","last_clean":"2023-05-22T16:03:38.830554+0000","last_became_active":"2023-05-22T16:03:38.829218+0000","last_became_peered":"2023-05-22T16:03:38.829218+0000","last_unstale":"2023-05-22T16:03:38.830554+0000","last_undegraded":"2023-05-22T16:03:38.830554+0000","last_fullsized":"2023-05-22T16:03:38.830554+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_clean_scrub_stamp":"2023-05-22T16:03:38.387135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:07:09.650623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.13","version":"178'2","reported_seq":12,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.821817+0000","last_change":"2023-05-22T16:03:38.427651+0000","last_active":"2023-05-22T16:03:38.821817+0000","last_peered":"2023-05-22T16:03:38.821817+0000","last_clean":"2023-05-22T16:03:38.821817+0000","last_became_active":"2023-05-22T16:03:38.427358+0000","last_became_peered":"2023-05-22T16:03:38.427358+0000","last_unstale":"2023-05-22T16:03:38.821817+0000","last_undegraded":"2023-05-22T16:03:38.821817+0000","last_fullsized":"2023-05-22T16:03:38.821817+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:03:37.383765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:03:37.383765+0000","last_clean_scrub_stamp":"2023-05-22T16:03:37.383765+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:59:31.499006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":158,"reported_epoch":177,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.441489+0000","last_change":"2023-05-22T15:26:44.982421+0000","last_active":"2023-05-22T16:03:38.441489+0000","last_peered":"2023-05-22T16:03:38.441489+0000","last_clean":"2023-05-22T16:03:38.441489+0000","last_became_active":"2023-05-22T15:26:44.981586+0000","last_became_peered":"2023-05-22T15:26:44.981586+0000","last_unstale":"2023-05-22T16:03:38.441489+0000","last_undegraded":"2023-05-22T16:03:38.441489+0000","last_fullsized":"2023-05-22T16:03:38.441489+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:14:11.156544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":23,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":22,"num_pg":32,"stat_sum":{"num_bytes":2330,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1748,"num_read_kb":3602,"num_write":499,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":304,"ondisk_log_size":304,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575288,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27936,"kb_used_data":2776,"kb_used_omap":0,"kb_used_meta":25152,"kb_avail":94343904,"statfs":{"total":96636764160,"available":96608157696,"internally_reserved":0,"allocated":2842624,"data_stored":1779013,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":25755648},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 16:02:42 2023","interfaces":[{"interface":"back","average":{"1min":0.82599999999999996,"5min":0.85199999999999998,"15min":0.86599999999999999},"min":{"1min":0.61499999999999999,"5min":0.52800000000000002,"15min":0.51100000000000001},"max":{"1min":1.194,"5min":1.2569999999999999,"15min":1.385},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.91500000000000004,"5min":0.89000000000000001,"15min":0.88800000000000001},"min":{"1min":0.629,"5min":0.501,"15min":0.501},"max":{"1min":1.244,"5min":1.2529999999999999,"15min":1.2529999999999999},"last":0.94699999999999995}]},{"osd":1,"last update":"Mon May 22 16:02:42 2023","interfaces":[{"interface":"back","average":{"1min":0.97499999999999998,"5min":0.998,"15min":0.99299999999999999},"min":{"1min":0.63400000000000001,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.1890000000000001,"5min":1.3260000000000001,"15min":1.361},"last":1.089},{"interface":"front","average":{"1min":1.085,"5min":1.0469999999999999,"15min":1.046},"min":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.626},"max":{"1min":1.3029999999999999,"5min":1.3380000000000001,"15min":1.4019999999999999},"last":1.028}]},{"osd":2,"last update":"Mon May 22 16:02:42 2023","interfaces":[{"interface":"back","average":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.90200000000000002},"min":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"max":{"1min":1.1739999999999999,"5min":1.286,"15min":1.407},"last":0.98899999999999999},{"interface":"front","average":{"1min":0.97499999999999998,"5min":0.99199999999999999,"15min":0.97099999999999997},"min":{"1min":0.68400000000000005,"5min":0.65400000000000003,"15min":0.65400000000000003},"max":{"1min":1.214,"5min":1.2829999999999999,"15min":1.2829999999999999},"last":1.179}]},{"osd":3,"last update":"Mon May 22 16:02:42 2023","interfaces":[{"interface":"back","average":{"1min":0.94599999999999995,"5min":0.96499999999999997,"15min":0.95799999999999996},"min":{"1min":0.73999999999999999,"5min":0.65700000000000003,"15min":0.62},"max":{"1min":1.167,"5min":1.401,"15min":1.401},"last":0.88900000000000001},{"interface":"front","average":{"1min":0.996,"5min":1.002,"15min":0.98999999999999999},"min":{"1min":0.77000000000000002,"5min":0.66300000000000003,"15min":0.66300000000000003},"max":{"1min":1.3460000000000001,"5min":1.411,"15min":1.411},"last":0.83399999999999996}]},{"osd":4,"last update":"Mon May 22 16:02:42 2023","interfaces":[{"interface":"back","average":{"1min":0.84299999999999997,"5min":0.89200000000000002,"15min":0.88700000000000001},"min":{"1min":0.63,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":1.163,"5min":1.3440000000000001,"15min":1.3779999999999999},"last":0.77900000000000003},{"interface":"front","average":{"1min":1.012,"5min":0.95999999999999996,"15min":0.95999999999999996},"min":{"1min":0.67900000000000005,"5min":0.57999999999999996,"15min":0.55400000000000005},"max":{"1min":1.3300000000000001,"5min":1.3300000000000001,"15min":1.4419999999999999},"last":1.125}]},{"osd":5,"last update":"Mon May 22 16:02:42 2023","interfaces":[{"interface":"back","average":{"1min":1.024,"5min":0.98599999999999999,"15min":0.996},"min":{"1min":0.65600000000000003,"5min":0.621,"15min":0.441},"max":{"1min":1.2909999999999999,"5min":1.3759999999999999,"15min":1.3819999999999999},"last":1.1539999999999999},{"interface":"front","average":{"1min":1.1339999999999999,"5min":1.1439999999999999,"15min":1.1379999999999999},"min":{"1min":0.83199999999999996,"5min":0.66600000000000004,"15min":0.59799999999999998},"max":{"1min":1.298,"5min":1.4079999999999999,"15min":1.5429999999999999},"last":1.056}]},{"osd":6,"last update":"Mon May 22 16:02:42 2023","interfaces":[{"interface":"back","average":{"1min":1.052,"5min":1.0369999999999999,"15min":1.0589999999999999},"min":{"1min":0.72299999999999998,"5min":0.65800000000000003,"15min":0.65800000000000003},"max":{"1min":1.3169999999999999,"5min":1.3480000000000001,"15min":1.4470000000000001},"last":1.2490000000000001},{"interface":"front","average":{"1min":1.1140000000000001,"5min":1.089,"15min":1.101},"min":{"1min":0.70199999999999996,"5min":0.67600000000000005,"15min":0.61499999999999999},"max":{"1min":1.391,"5min":1.4550000000000001,"15min":1.7050000000000001},"last":0.69699999999999995}]}]},{"osd":6,"up_from":13,"seq":55834575289,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":22024,"kb_used_data":2176,"kb_used_omap":0,"kb_used_meta":19840,"kb_avail":94349816,"statfs":{"total":96636764160,"available":96614211584,"internally_reserved":0,"allocated":2228224,"data_stored":1186393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":20316160},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 16:02:53 2023","interfaces":[{"interface":"back","average":{"1min":0.88100000000000001,"5min":0.89800000000000002,"15min":0.91000000000000003},"min":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"max":{"1min":1.2250000000000001,"5min":1.2250000000000001,"15min":1.482},"last":0.69999999999999996},{"interface":"front","average":{"1min":0.89900000000000002,"5min":0.89100000000000001,"15min":0.91500000000000004},"min":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.57599999999999996},"max":{"1min":1.397,"5min":1.397,"15min":1.397},"last":0.55900000000000005}]},{"osd":1,"last update":"Mon May 22 16:02:53 2023","interfaces":[{"interface":"back","average":{"1min":0.88500000000000001,"5min":0.91400000000000003,"15min":0.94299999999999995},"min":{"1min":0.66200000000000003,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.1839999999999999,"5min":1.3380000000000001,"15min":1.395},"last":0.875},{"interface":"front","average":{"1min":0.94599999999999995,"5min":0.93600000000000005,"15min":0.94299999999999995},"min":{"1min":0.66400000000000003,"5min":0.65100000000000002,"15min":0.63500000000000001},"max":{"1min":1.2849999999999999,"5min":1.2849999999999999,"15min":1.357},"last":0.66000000000000003}]},{"osd":2,"last update":"Mon May 22 16:02:53 2023","interfaces":[{"interface":"back","average":{"1min":0.99299999999999999,"5min":0.99299999999999999,"15min":1.0029999999999999},"min":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":1.3580000000000001,"5min":1.3580000000000001,"15min":1.3580000000000001},"last":0.99299999999999999},{"interface":"front","average":{"1min":1.0649999999999999,"5min":1.091,"15min":1.085},"min":{"1min":0.77900000000000003,"5min":0.76900000000000002,"15min":0.63200000000000001},"max":{"1min":1.2649999999999999,"5min":1.359,"15min":1.6579999999999999},"last":0.96099999999999997}]},{"osd":3,"last update":"Mon May 22 16:02:53 2023","interfaces":[{"interface":"back","average":{"1min":0.997,"5min":1.01,"15min":0.996},"min":{"1min":0.69599999999999995,"5min":0.65500000000000003,"15min":0.63700000000000001},"max":{"1min":1.169,"5min":1.327,"15min":1.4339999999999999},"last":0.75900000000000001},{"interface":"front","average":{"1min":0.97999999999999998,"5min":0.96699999999999997,"15min":0.97599999999999998},"min":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"max":{"1min":1.1899999999999999,"5min":1.2350000000000001,"15min":1.3779999999999999},"last":0.80500000000000005}]},{"osd":4,"last update":"Mon May 22 16:02:53 2023","interfaces":[{"interface":"back","average":{"1min":0.99299999999999999,"5min":0.92500000000000004,"15min":0.95599999999999996},"min":{"1min":0.62,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":1.333,"5min":1.359,"15min":1.5720000000000001},"last":1.0609999999999999},{"interface":"front","average":{"1min":1.0740000000000001,"5min":1.0469999999999999,"15min":1.0349999999999999},"min":{"1min":0.64700000000000002,"5min":0.623,"15min":0.622},"max":{"1min":1.3049999999999999,"5min":1.4770000000000001,"15min":1.52},"last":0.84699999999999998}]},{"osd":5,"last update":"Mon May 22 16:02:53 2023","interfaces":[{"interface":"back","average":{"1min":1.024,"5min":0.999,"15min":0.98699999999999999},"min":{"1min":0.71399999999999997,"5min":0.66900000000000004,"15min":0.61799999999999999},"max":{"1min":1.3100000000000001,"5min":1.375,"15min":1.504},"last":0.89900000000000002},{"interface":"front","average":{"1min":1.095,"5min":1.1080000000000001,"15min":1.129},"min":{"1min":0.748,"5min":0.69799999999999995,"15min":0.69699999999999995},"max":{"1min":1.325,"5min":1.4430000000000001,"15min":1.74},"last":1.1140000000000001}]},{"osd":7,"last update":"Mon May 22 16:02:53 2023","interfaces":[{"interface":"back","average":{"1min":1.161,"5min":1.1419999999999999,"15min":1.1339999999999999},"min":{"1min":0.92700000000000005,"5min":0.64200000000000002,"15min":0.58599999999999997},"max":{"1min":1.359,"5min":1.546,"15min":1.603},"last":0.93000000000000005},{"interface":"front","average":{"1min":1.1599999999999999,"5min":1.131,"15min":1.139},"min":{"1min":0.81899999999999995,"5min":0.755,"15min":0.746},"max":{"1min":1.4450000000000001,"5min":1.669,"15min":1.669},"last":1.0860000000000001}]}]},{"osd":5,"up_from":13,"seq":55834575289,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31884,"kb_used_data":2180,"kb_used_omap":0,"kb_used_meta":29696,"kb_avail":94339956,"statfs":{"total":96636764160,"available":96604114944,"internally_reserved":0,"allocated":2232320,"data_stored":1186959,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":30408704},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 16:02:47 2023","interfaces":[{"interface":"back","average":{"1min":0.90400000000000003,"5min":0.92000000000000004,"15min":0.89800000000000002},"min":{"1min":0.63500000000000001,"5min":0.61899999999999999,"15min":0.57199999999999995},"max":{"1min":1.1850000000000001,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.99299999999999999},{"interface":"front","average":{"1min":0.93999999999999995,"5min":0.92400000000000004,"15min":0.94499999999999995},"min":{"1min":0.623,"5min":0.502,"15min":0.502},"max":{"1min":1.325,"5min":1.325,"15min":1.329},"last":1.042}]},{"osd":1,"last update":"Mon May 22 16:02:47 2023","interfaces":[{"interface":"back","average":{"1min":1.002,"5min":1.0109999999999999,"15min":1.022},"min":{"1min":0.754,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":1.3280000000000001,"5min":1.365,"15min":1.365},"last":1.0740000000000001},{"interface":"front","average":{"1min":1.1000000000000001,"5min":1.083,"15min":1.081},"min":{"1min":0.85399999999999998,"5min":0.80500000000000005,"15min":0.61599999999999999},"max":{"1min":1.3500000000000001,"5min":1.4630000000000001,"15min":1.4630000000000001},"last":1.2110000000000001}]},{"osd":2,"last update":"Mon May 22 16:02:47 2023","interfaces":[{"interface":"back","average":{"1min":0.96399999999999997,"5min":0.89200000000000002,"15min":0.92700000000000005},"min":{"1min":0.68100000000000005,"5min":0.63700000000000001,"15min":0.63600000000000001},"max":{"1min":1.2,"5min":1.3200000000000001,"15min":1.413},"last":1.1299999999999999},{"interface":"front","average":{"1min":0.98199999999999998,"5min":0.96599999999999997,"15min":0.95199999999999996},"min":{"1min":0.71599999999999997,"5min":0.65800000000000003,"15min":0.65800000000000003},"max":{"1min":1.379,"5min":1.379,"15min":1.379},"last":0.88300000000000001}]},{"osd":3,"last update":"Mon May 22 16:02:47 2023","interfaces":[{"interface":"back","average":{"1min":0.97499999999999998,"5min":0.96799999999999997,"15min":0.97899999999999998},"min":{"1min":0.79100000000000004,"5min":0.71099999999999997,"15min":0.60799999999999998},"max":{"1min":1.296,"5min":1.367,"15min":1.367},"last":1.0960000000000001},{"interface":"front","average":{"1min":1.0129999999999999,"5min":1.056,"15min":1.0600000000000001},"min":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"max":{"1min":1.2410000000000001,"5min":1.371,"15min":1.387},"last":0.93100000000000005}]},{"osd":4,"last update":"Mon May 22 16:02:47 2023","interfaces":[{"interface":"back","average":{"1min":0.94099999999999995,"5min":0.92800000000000005,"15min":0.92100000000000004},"min":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"max":{"1min":1.282,"5min":1.4079999999999999,"15min":1.5109999999999999},"last":1.018},{"interface":"front","average":{"1min":0.99399999999999999,"5min":1.0109999999999999,"15min":1.0249999999999999},"min":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.58399999999999996},"max":{"1min":1.4630000000000001,"5min":1.4630000000000001,"15min":1.502},"last":0.83399999999999996}]},{"osd":6,"last update":"Mon May 22 16:02:47 2023","interfaces":[{"interface":"back","average":{"1min":1.0229999999999999,"5min":1.0109999999999999,"15min":1.0269999999999999},"min":{"1min":0.67100000000000004,"5min":0.64400000000000002,"15min":0.59799999999999998},"max":{"1min":1.389,"5min":1.389,"15min":1.4790000000000001},"last":0.77400000000000002},{"interface":"front","average":{"1min":1.0600000000000001,"5min":1.077,"15min":1.0860000000000001},"min":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"max":{"1min":1.3819999999999999,"5min":1.413,"15min":1.512},"last":0.95799999999999996}]},{"osd":7,"last update":"Mon May 22 16:02:47 2023","interfaces":[{"interface":"back","average":{"1min":1.1220000000000001,"5min":1.1359999999999999,"15min":1.1339999999999999},"min":{"1min":0.73199999999999998,"5min":0.65800000000000003,"15min":0.65800000000000003},"max":{"1min":1.359,"5min":1.5109999999999999,"15min":1.8360000000000001},"last":1.169},{"interface":"front","average":{"1min":1.1799999999999999,"5min":1.1879999999999999,"15min":1.194},"min":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.72799999999999998},"max":{"1min":1.5349999999999999,"5min":1.575,"15min":1.7210000000000001},"last":1.1799999999999999}]}]},{"osd":4,"up_from":13,"seq":55834575287,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33876,"kb_used_data":2188,"kb_used_omap":0,"kb_used_meta":31680,"kb_avail":94337964,"statfs":{"total":96636764160,"available":96602075136,"internally_reserved":0,"allocated":2240512,"data_stored":1187989,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":32440320},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 16:02:56 2023","interfaces":[{"interface":"back","average":{"1min":0.94299999999999995,"5min":0.91700000000000004,"15min":0.90700000000000003},"min":{"1min":0.71299999999999997,"5min":0.53100000000000003,"15min":0.499},"max":{"1min":1.262,"5min":1.7210000000000001,"15min":1.7210000000000001},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.98799999999999999,"5min":0.96399999999999997,"15min":0.95299999999999996},"min":{"1min":0.73699999999999999,"5min":0.621,"15min":0.58999999999999997},"max":{"1min":1.2749999999999999,"5min":1.4730000000000001,"15min":1.4730000000000001},"last":0.93600000000000005}]},{"osd":1,"last update":"Mon May 22 16:02:56 2023","interfaces":[{"interface":"back","average":{"1min":0.94899999999999995,"5min":0.96499999999999997,"15min":0.96099999999999997},"min":{"1min":0.72199999999999998,"5min":0.65400000000000003,"15min":0.63400000000000001},"max":{"1min":1.173,"5min":1.3999999999999999,"15min":1.3999999999999999},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.96299999999999997,"5min":0.95499999999999996,"15min":0.94899999999999995},"min":{"1min":0.70099999999999996,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":1.2210000000000001,"5min":1.306,"15min":1.306},"last":0.58299999999999996}]},{"osd":2,"last update":"Mon May 22 16:02:56 2023","interfaces":[{"interface":"back","average":{"1min":0.93000000000000005,"5min":0.93600000000000005,"15min":0.94699999999999995},"min":{"1min":0.70599999999999996,"5min":0.65900000000000003,"15min":0.59199999999999997},"max":{"1min":1.383,"5min":1.383,"15min":1.383},"last":0.73699999999999999},{"interface":"front","average":{"1min":1.0600000000000001,"5min":1.0489999999999999,"15min":1.0529999999999999},"min":{"1min":0.68700000000000006,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.3899999999999999,"5min":1.492,"15min":1.492},"last":1.0649999999999999}]},{"osd":3,"last update":"Mon May 22 16:02:56 2023","interfaces":[{"interface":"back","average":{"1min":0.92300000000000004,"5min":0.94799999999999995,"15min":0.95299999999999996},"min":{"1min":0.72799999999999998,"5min":0.67300000000000004,"15min":0.65300000000000002},"max":{"1min":1.3740000000000001,"5min":1.4610000000000001,"15min":1.4610000000000001},"last":0.77700000000000002},{"interface":"front","average":{"1min":1.0269999999999999,"5min":1.075,"15min":1.0529999999999999},"min":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.69099999999999995},"max":{"1min":1.2549999999999999,"5min":1.484,"15min":1.484},"last":0.84599999999999997}]},{"osd":5,"last update":"Mon May 22 16:02:56 2023","interfaces":[{"interface":"back","average":{"1min":0.97999999999999998,"5min":1.0189999999999999,"15min":0.999},"min":{"1min":0.67200000000000004,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":1.2949999999999999,"5min":1.417,"15min":1.417},"last":1.139},{"interface":"front","average":{"1min":0.96099999999999997,"5min":0.99399999999999999,"15min":0.99199999999999999},"min":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"max":{"1min":1.2490000000000001,"5min":1.5449999999999999,"15min":1.5449999999999999},"last":0.89700000000000002}]},{"osd":6,"last update":"Mon May 22 16:02:56 2023","interfaces":[{"interface":"back","average":{"1min":0.95999999999999996,"5min":0.98299999999999998,"15min":0.99199999999999999},"min":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":1.224,"5min":1.397,"15min":1.4339999999999999},"last":1.008},{"interface":"front","average":{"1min":1.071,"5min":1.1319999999999999,"15min":1.125},"min":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"max":{"1min":1.456,"5min":1.5600000000000001,"15min":1.5600000000000001},"last":1.1879999999999999}]},{"osd":7,"last update":"Mon May 22 16:02:56 2023","interfaces":[{"interface":"back","average":{"1min":1.0669999999999999,"5min":1.0700000000000001,"15min":1.0669999999999999},"min":{"1min":0.76700000000000002,"5min":0.71399999999999997,"15min":0.71399999999999997},"max":{"1min":1.3280000000000001,"5min":1.5920000000000001,"15min":1.5920000000000001},"last":1.042},{"interface":"front","average":{"1min":1.169,"5min":1.1599999999999999,"15min":1.155},"min":{"1min":0.95999999999999996,"5min":0.68100000000000005,"15min":0.68100000000000005},"max":{"1min":1.3500000000000001,"5min":1.448,"15min":1.5089999999999999},"last":1.1000000000000001}]}]},{"osd":3,"up_from":13,"seq":55834575287,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":26568,"kb_used_data":2176,"kb_used_omap":0,"kb_used_meta":24384,"kb_avail":94345272,"statfs":{"total":96636764160,"available":96609558528,"internally_reserved":0,"allocated":2228224,"data_stored":1186405,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":24969216},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 16:03:38 2023","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":0.63600000000000001,"15min":0.64000000000000001},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":1.0049999999999999,"5min":1.0049999999999999,"15min":1.048},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.76500000000000001,"15min":0.76900000000000002},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":1.1539999999999999,"5min":1.2789999999999999,"15min":1.2789999999999999},"last":0.63300000000000001}]},{"osd":1,"last update":"Mon May 22 16:03:38 2023","interfaces":[{"interface":"back","average":{"1min":0.82499999999999996,"5min":0.74099999999999999,"15min":0.71399999999999997},"min":{"1min":0.55100000000000005,"5min":0.52700000000000002,"15min":0.51600000000000001},"max":{"1min":2.4969999999999999,"5min":2.4969999999999999,"15min":2.4969999999999999},"last":0.89700000000000002},{"interface":"front","average":{"1min":0.89400000000000002,"5min":0.80300000000000005,"15min":0.80600000000000005},"min":{"1min":0.61399999999999999,"5min":0.55800000000000005,"15min":0.54400000000000004},"max":{"1min":2.4769999999999999,"5min":2.4769999999999999,"15min":2.4769999999999999},"last":0.47899999999999998}]},{"osd":2,"last update":"Mon May 22 16:03:38 2023","interfaces":[{"interface":"back","average":{"1min":0.84499999999999997,"5min":0.84399999999999997,"15min":0.85399999999999998},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":1.1759999999999999,"5min":1.1759999999999999,"15min":1.1759999999999999},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.873,"5min":0.81699999999999995,"15min":0.83599999999999997},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":1.1259999999999999,"5min":1.2270000000000001,"15min":1.2270000000000001},"last":0.66100000000000003}]},{"osd":4,"last update":"Mon May 22 16:03:38 2023","interfaces":[{"interface":"back","average":{"1min":0.95299999999999996,"5min":0.96299999999999997,"15min":0.97599999999999998},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":1.343,"5min":1.343,"15min":1.343},"last":0.995},{"interface":"front","average":{"1min":1.002,"5min":1.012,"15min":1.036},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":1.3180000000000001,"5min":1.4119999999999999,"15min":1.49},"last":0.81299999999999994}]},{"osd":5,"last update":"Mon May 22 16:03:38 2023","interfaces":[{"interface":"back","average":{"1min":0.996,"5min":1.056,"15min":1.0580000000000001},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":1.3520000000000001,"5min":1.5149999999999999,"15min":1.5149999999999999},"last":0.68300000000000005},{"interface":"front","average":{"1min":1.0489999999999999,"5min":1.054,"15min":1.0569999999999999},"min":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":1.571,"5min":1.571,"15min":1.571},"last":0.78100000000000003}]},{"osd":6,"last update":"Mon May 22 16:03:38 2023","interfaces":[{"interface":"back","average":{"1min":1.0620000000000001,"5min":1.0900000000000001,"15min":1.099},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":1.446,"5min":1.6739999999999999,"15min":1.6739999999999999},"last":0.749},{"interface":"front","average":{"1min":1.1220000000000001,"5min":1.107,"15min":1.1080000000000001},"min":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":1.754,"5min":1.754,"15min":1.754},"last":0.93899999999999995}]},{"osd":7,"last update":"Mon May 22 16:03:38 2023","interfaces":[{"interface":"back","average":{"1min":1.139,"5min":1.1220000000000001,"15min":1.137},"min":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.673,"5min":1.673,"15min":1.673},"last":0.84399999999999997},{"interface":"front","average":{"1min":1.129,"5min":1.105,"15min":1.1120000000000001},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":1.5369999999999999,"5min":1.5369999999999999,"15min":1.54},"last":1.1140000000000001}]}]},{"osd":2,"up_from":13,"seq":55834575288,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":26504,"kb_used_data":2176,"kb_used_omap":0,"kb_used_meta":24320,"kb_avail":94345336,"statfs":{"total":96636764160,"available":96609624064,"internally_reserved":0,"allocated":2228224,"data_stored":1186461,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":24903680},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 16:03:29 2023","interfaces":[{"interface":"back","average":{"1min":0.69499999999999995,"5min":0.65600000000000003,"15min":0.67800000000000005},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":1.0549999999999999,"5min":1.1180000000000001,"15min":1.1519999999999999},"last":0.438},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.74199999999999999,"15min":0.754},"min":{"1min":0.17999999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":1.038,"5min":1.1319999999999999,"15min":1.2090000000000001},"last":0.51300000000000001}]},{"osd":1,"last update":"Mon May 22 16:03:29 2023","interfaces":[{"interface":"back","average":{"1min":0.77300000000000002,"5min":0.76500000000000001,"15min":0.76400000000000001},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":1.071,"5min":1.0800000000000001,"15min":1.099},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.81499999999999995,"5min":0.83799999999999997,"15min":0.82499999999999996},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":1.073,"5min":1.1739999999999999,"15min":1.2769999999999999},"last":0.39200000000000002}]},{"osd":3,"last update":"Mon May 22 16:03:29 2023","interfaces":[{"interface":"back","average":{"1min":0.77900000000000003,"5min":0.77700000000000002,"15min":0.76200000000000001},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":1.3540000000000001,"5min":1.3540000000000001,"15min":1.3540000000000001},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.85399999999999998,"5min":0.877,"15min":0.90200000000000002},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":1.2,"5min":1.2,"15min":1.2949999999999999},"last":0.61599999999999999}]},{"osd":4,"last update":"Mon May 22 16:03:29 2023","interfaces":[{"interface":"back","average":{"1min":1.018,"5min":1.0329999999999999,"15min":1.0429999999999999},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":1.2709999999999999,"5min":1.6539999999999999,"15min":1.6539999999999999},"last":0.82599999999999996},{"interface":"front","average":{"1min":0.95899999999999996,"5min":1.0289999999999999,"15min":1.0449999999999999},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.216,"5min":1.472,"15min":1.538},"last":1.014}]},{"osd":5,"last update":"Mon May 22 16:03:29 2023","interfaces":[{"interface":"back","average":{"1min":0.995,"5min":1.0529999999999999,"15min":1.0609999999999999},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":1.3200000000000001,"5min":1.5640000000000001,"15min":1.605},"last":0.75700000000000001},{"interface":"front","average":{"1min":1.0449999999999999,"5min":1.016,"15min":1.036},"min":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"max":{"1min":1.4650000000000001,"5min":1.4870000000000001,"15min":1.5109999999999999},"last":0.91100000000000003}]},{"osd":6,"last update":"Mon May 22 16:03:29 2023","interfaces":[{"interface":"back","average":{"1min":1.034,"5min":1.0409999999999999,"15min":1.028},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":1.4019999999999999,"5min":1.429,"15min":1.462},"last":0.83699999999999997},{"interface":"front","average":{"1min":1.1020000000000001,"5min":1.1180000000000001,"15min":1.137},"min":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"max":{"1min":1.389,"5min":1.7110000000000001,"15min":1.7110000000000001},"last":0.79200000000000004}]},{"osd":7,"last update":"Mon May 22 16:03:29 2023","interfaces":[{"interface":"back","average":{"1min":1.0369999999999999,"5min":1.1000000000000001,"15min":1.1279999999999999},"min":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":1.3660000000000001,"5min":1.6399999999999999,"15min":1.6399999999999999},"last":0.80800000000000005},{"interface":"front","average":{"1min":1.071,"5min":1.113,"15min":1.1200000000000001},"min":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":1.4330000000000001,"5min":1.673,"15min":1.673},"last":0.95699999999999996}]}]},{"osd":1,"up_from":13,"seq":55834575288,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36488,"kb_used_data":2176,"kb_used_omap":0,"kb_used_meta":34304,"kb_avail":94335352,"statfs":{"total":96636764160,"available":96599400448,"internally_reserved":0,"allocated":2228224,"data_stored":1186405,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":35127296},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 16:02:50 2023","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.66000000000000003,"15min":0.66300000000000003},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.88800000000000001,"5min":1.151,"15min":1.329},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.71799999999999997,"5min":0.75800000000000001,"15min":0.77600000000000002},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":0.95599999999999996,"5min":1.2350000000000001,"15min":1.268},"last":0.85899999999999999}]},{"osd":2,"last update":"Mon May 22 16:02:50 2023","interfaces":[{"interface":"back","average":{"1min":0.746,"5min":0.76500000000000001,"15min":0.77200000000000002},"min":{"1min":0.57299999999999995,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.98499999999999999,"5min":1.0880000000000001,"15min":1.26},"last":0.73899999999999999},{"interface":"front","average":{"1min":0.80200000000000005,"5min":0.78800000000000003,"15min":0.78600000000000003},"min":{"1min":0.56999999999999995,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.127,"5min":1.175,"15min":1.268},"last":0.77700000000000002}]},{"osd":3,"last update":"Mon May 22 16:02:50 2023","interfaces":[{"interface":"back","average":{"1min":0.88700000000000001,"5min":0.878,"15min":0.89000000000000001},"min":{"1min":0.621,"5min":0.61199999999999999,"15min":0.55900000000000005},"max":{"1min":1.2829999999999999,"5min":1.2829999999999999,"15min":1.373},"last":0.996},{"interface":"front","average":{"1min":0.88400000000000001,"5min":0.83699999999999997,"15min":0.83499999999999996},"min":{"1min":0.63600000000000001,"5min":0.57199999999999995,"15min":0.57199999999999995},"max":{"1min":1.2749999999999999,"5min":1.2749999999999999,"15min":1.2749999999999999},"last":0.63200000000000001}]},{"osd":4,"last update":"Mon May 22 16:02:50 2023","interfaces":[{"interface":"back","average":{"1min":0.98999999999999999,"5min":1.002,"15min":0.99399999999999999},"min":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"max":{"1min":1.2769999999999999,"5min":1.542,"15min":1.542},"last":1.089},{"interface":"front","average":{"1min":1.056,"5min":1.042,"15min":1.0389999999999999},"min":{"1min":0.85699999999999998,"5min":0.79500000000000004,"15min":0.73499999999999999},"max":{"1min":1.327,"5min":1.4350000000000001,"15min":1.4650000000000001},"last":1.111}]},{"osd":5,"last update":"Mon May 22 16:02:50 2023","interfaces":[{"interface":"back","average":{"1min":1.075,"5min":1.0600000000000001,"15min":1.0609999999999999},"min":{"1min":0.88,"5min":0.78100000000000003,"15min":0.76800000000000002},"max":{"1min":1.236,"5min":1.605,"15min":1.605},"last":1.046},{"interface":"front","average":{"1min":1.0309999999999999,"5min":1.095,"15min":1.085},"min":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"max":{"1min":1.365,"5min":1.411,"15min":1.5389999999999999},"last":0.95699999999999996}]},{"osd":6,"last update":"Mon May 22 16:02:50 2023","interfaces":[{"interface":"back","average":{"1min":1.0700000000000001,"5min":1.0720000000000001,"15min":1.075},"min":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.73899999999999999},"max":{"1min":1.371,"5min":1.371,"15min":1.5249999999999999},"last":1.22},{"interface":"front","average":{"1min":1.105,"5min":1.111,"15min":1.123},"min":{"1min":0.85199999999999998,"5min":0.84599999999999997,"15min":0.75700000000000001},"max":{"1min":1.329,"5min":1.4630000000000001,"15min":1.6479999999999999},"last":1.27}]},{"osd":7,"last update":"Mon May 22 16:02:50 2023","interfaces":[{"interface":"back","average":{"1min":1.151,"5min":1.167,"15min":1.171},"min":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"max":{"1min":1.403,"5min":1.7030000000000001,"15min":1.7030000000000001},"last":1.1739999999999999},{"interface":"front","average":{"1min":1.125,"5min":1.155,"15min":1.157},"min":{"1min":0.85199999999999998,"5min":0.81100000000000005,"15min":0.81100000000000005},"max":{"1min":1.45,"5min":1.6439999999999999,"15min":1.6439999999999999},"last":1.1519999999999999}]}]},{"osd":0,"up_from":13,"seq":55834575289,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31880,"kb_used_data":2752,"kb_used_omap":0,"kb_used_meta":29120,"kb_avail":94339960,"statfs":{"total":96636764160,"available":96604119040,"internally_reserved":0,"allocated":2818048,"data_stored":1776739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":29818880},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon May 22 16:02:55 2023","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.71499999999999997,"15min":0.70999999999999996},"min":{"1min":0.53500000000000003,"5min":0.501,"15min":0.47999999999999998},"max":{"1min":1.296,"5min":1.296,"15min":1.296},"last":0.82299999999999995},{"interface":"front","average":{"1min":0.78800000000000003,"5min":0.75600000000000001,"15min":0.747},"min":{"1min":0.54700000000000004,"5min":0.496,"15min":0.47599999999999998},"max":{"1min":1.181,"5min":1.286,"15min":1.3149999999999999},"last":1.0229999999999999}]},{"osd":2,"last update":"Mon May 22 16:02:55 2023","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":0.77200000000000002,"15min":0.78100000000000003},"min":{"1min":0.63500000000000001,"5min":0.56699999999999995,"15min":0.52400000000000002},"max":{"1min":1.1419999999999999,"5min":1.1839999999999999,"15min":1.284},"last":0.97899999999999998},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.84499999999999997,"15min":0.82799999999999996},"min":{"1min":0.61299999999999999,"5min":0.56200000000000006,"15min":0.54200000000000004},"max":{"1min":1.038,"5min":1.2170000000000001,"15min":1.2270000000000001},"last":0.73799999999999999}]},{"osd":3,"last update":"Mon May 22 16:02:55 2023","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":0.79000000000000004,"15min":0.77600000000000002},"min":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.51900000000000002},"max":{"1min":0.995,"5min":1.2350000000000001,"15min":1.2350000000000001},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.91900000000000004,"5min":0.89700000000000002,"15min":0.89200000000000002},"min":{"1min":0.70599999999999996,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":1.2949999999999999,"5min":1.2949999999999999,"15min":1.331},"last":1.198}]},{"osd":4,"last update":"Mon May 22 16:02:55 2023","interfaces":[{"interface":"back","average":{"1min":0.98099999999999998,"5min":0.96899999999999997,"15min":0.95999999999999996},"min":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.67300000000000004},"max":{"1min":1.1719999999999999,"5min":1.4059999999999999,"15min":1.4059999999999999},"last":0.78900000000000003},{"interface":"front","average":{"1min":1.089,"5min":1.0720000000000001,"15min":1.0600000000000001},"min":{"1min":0.89300000000000002,"5min":0.83099999999999996,"15min":0.74299999999999999},"max":{"1min":1.3839999999999999,"5min":1.3839999999999999,"15min":1.482},"last":1.401}]},{"osd":5,"last update":"Mon May 22 16:02:55 2023","interfaces":[{"interface":"back","average":{"1min":1.044,"5min":1.0429999999999999,"15min":1.0489999999999999},"min":{"1min":0.84799999999999998,"5min":0.77200000000000002,"15min":0.65300000000000002},"max":{"1min":1.2729999999999999,"5min":1.361,"15min":1.6419999999999999},"last":1.264},{"interface":"front","average":{"1min":0.995,"5min":1.032,"15min":1.018},"min":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.72299999999999998},"max":{"1min":1.27,"5min":1.458,"15min":1.458},"last":0.93500000000000005}]},{"osd":6,"last update":"Mon May 22 16:02:55 2023","interfaces":[{"interface":"back","average":{"1min":1.119,"5min":1.1180000000000001,"15min":1.1160000000000001},"min":{"1min":0.89700000000000002,"5min":0.80100000000000005,"15min":0.79900000000000004},"max":{"1min":1.3660000000000001,"5min":1.4359999999999999,"15min":1.77},"last":1.3120000000000001},{"interface":"front","average":{"1min":1.105,"5min":1.131,"15min":1.1080000000000001},"min":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.747},"max":{"1min":1.256,"5min":1.4990000000000001,"15min":1.53},"last":1.45}]},{"osd":7,"last update":"Mon May 22 16:02:55 2023","interfaces":[{"interface":"back","average":{"1min":1.1659999999999999,"5min":1.165,"15min":1.1379999999999999},"min":{"1min":0.86399999999999999,"5min":0.86399999999999999,"15min":0.76000000000000001},"max":{"1min":1.355,"5min":1.532,"15min":1.5960000000000001},"last":1.1499999999999999},{"interface":"front","average":{"1min":1.1200000000000001,"5min":1.1259999999999999,"15min":1.111},"min":{"1min":0.94999999999999996,"5min":0.83999999999999997,"15min":0.79000000000000004},"max":{"1min":1.403,"5min":1.4650000000000001,"15min":1.4990000000000001},"last":1.0589999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2274,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-05-22T16:03:43.186 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-05-22T16:03:43.348 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:43.346+0000 7f07950de640 1 Processor -- start 2023-05-22T16:03:43.348 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:43.346+0000 7f07950de640 1 -- start start 2023-05-22T16:03:43.350 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:43.350+0000 7f07950de640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f07900b0250 0x7f07900cd1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:43.350 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:43.350+0000 7f07950de640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f079015fcf0 0x7f0790160190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:43.350 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:43.350+0000 7f07950de640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07900cd8d0 con 0x7f0790070fb0 2023-05-22T16:03:43.350 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:43.350+0000 7f07950de640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0790164de0 con 0x7f079015fcf0 2023-05-22T16:03:43.350 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:43.350+0000 7f07950de640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07900ca400 con 0x7f07900b0250 2023-05-22T16:03:43.350 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:43.350+0000 7f078f59e640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f079015fcf0 0x7f0790160190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:43.351 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:43.350+0000 7f078ed9d640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f0790070fb0 0x7f07900afb40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:53204/0 (socket says 172.21.15.16:53204) 2023-05-22T16:03:43.351 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:43.350+0000 7f078ed9d640 1 -- 172.21.15.16:0/1804777661 learned_addr learned my addr 172.21.15.16:0/1804777661 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T16:03:43.351 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:43.350+0000 7f078e59c640 1 --2- 172.21.15.16:0/1804777661 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f07900b0250 0x7f07900cd1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:43.351 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:43.350+0000 7f078f59e640 1 -- 172.21.15.16:0/1804777661 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f07900b0250 msgr2=0x7f07900cd1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:03:43.351 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:43.350+0000 7f078f59e640 1 --2- 172.21.15.16:0/1804777661 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f07900b0250 0x7f07900cd1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:43.351 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:43.350+0000 7f078f59e640 1 -- 172.21.15.16:0/1804777661 >> v1:172.21.15.16:6789/0 conn(0x7f0790070fb0 legacy=0x7f07900afb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:43.351 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:43.350+0000 7f078dd9b640 1 -- 172.21.15.16:0/1804777661 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3704912227 0 0) 0x7f07900cd8d0 con 0x7f0790070fb0 2023-05-22T16:03:43.351 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:43.350+0000 7f078f59e640 1 -- 172.21.15.16:0/1804777661 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07900ca6e0 con 0x7f079015fcf0 2023-05-22T16:03:43.352 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:43.350+0000 7f078e59c640 1 --2- 172.21.15.16:0/1804777661 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f07900b0250 0x7f07900cd1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T16:03:43.352 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:43.354+0000 7f078f59e640 1 --2- 172.21.15.16:0/1804777661 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f079015fcf0 0x7f0790160190 secure :-1 s=READY pgs=2260 cs=0 l=1 rev1=1 crypto rx=0x7f0788009d90 tx=0x7f078801c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5535a8c116dee8ca server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:03:43.353 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:43.354+0000 7f078dd9b640 1 -- 172.21.15.16:0/1804777661 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07880a5810 con 0x7f079015fcf0 2023-05-22T16:03:43.353 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:43.354+0000 7f078dd9b640 1 -- 172.21.15.16:0/1804777661 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07880a5970 con 0x7f079015fcf0 2023-05-22T16:03:43.353 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:43.354+0000 7f078dd9b640 1 -- 172.21.15.16:0/1804777661 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07880a5d00 con 0x7f079015fcf0 2023-05-22T16:03:43.354 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:43.354+0000 7f07950de640 1 -- 172.21.15.16:0/1804777661 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f079015fcf0 msgr2=0x7f0790160190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:43.354 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:43.354+0000 7f07950de640 1 --2- 172.21.15.16:0/1804777661 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f079015fcf0 0x7f0790160190 secure :-1 s=READY pgs=2260 cs=0 l=1 rev1=1 crypto rx=0x7f0788009d90 tx=0x7f078801c950 comp rx=0 tx=0).stop 2023-05-22T16:03:43.354 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:43.354+0000 7f07950de640 1 -- 172.21.15.16:0/1804777661 shutdown_connections 2023-05-22T16:03:43.354 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:43.354+0000 7f07950de640 1 --2- 172.21.15.16:0/1804777661 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f079015fcf0 0x7f0790160190 unknown :-1 s=CLOSED pgs=2260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:43.354 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:43.354+0000 7f07950de640 1 --2- 172.21.15.16:0/1804777661 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f07900b0250 0x7f07900cd1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:43.354 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:43.354+0000 7f07950de640 1 -- 172.21.15.16:0/1804777661 >> 172.21.15.16:0/1804777661 conn(0x7f0790069720 msgr2=0x7f0790069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:03:43.355 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:43.354+0000 7f07950de640 1 -- 172.21.15.16:0/1804777661 shutdown_connections 2023-05-22T16:03:43.355 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:43.354+0000 7f07950de640 1 -- 172.21.15.16:0/1804777661 wait complete. 2023-05-22T16:03:43.356 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:43.354+0000 7f07950de640 1 Processor -- start 2023-05-22T16:03:43.356 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:43.354+0000 7f07950de640 1 -- start start 2023-05-22T16:03:43.356 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:43.354+0000 7f07950de640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0790070fb0 0x7f0790151c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:43.356 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:43.354+0000 7f07950de640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f07900b0250 0x7f07900b2330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:43.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:43.354+0000 7f07950de640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f079015fcf0 0x7f07900af890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:43.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:43.354+0000 7f07950de640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0790152150 con 0x7f0790070fb0 2023-05-22T16:03:43.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:43.354+0000 7f07950de640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07900b2870 con 0x7f07900b0250 2023-05-22T16:03:43.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:43.354+0000 7f07950de640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07900b1290 con 0x7f079015fcf0 2023-05-22T16:03:43.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:43.354+0000 7f078ed9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0790070fb0 0x7f0790151c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:43.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:43.354+0000 7f078ed9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0790070fb0 0x7f0790151c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49014/0 (socket says 172.21.15.16:49014) 2023-05-22T16:03:43.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:43.354+0000 7f078ed9d640 1 -- 172.21.15.16:0/1591427553 learned_addr learned my addr 172.21.15.16:0/1591427553 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:03:43.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:43.354+0000 7f078e59c640 1 --2- 172.21.15.16:0/1591427553 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f07900b0250 0x7f07900b2330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:43.358 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:43.354+0000 7f078f59e640 1 --2- 172.21.15.16:0/1591427553 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f079015fcf0 0x7f07900af890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:43.358 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:43.354+0000 7f078ed9d640 1 -- 172.21.15.16:0/1591427553 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f079015fcf0 msgr2=0x7f07900af890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:43.358 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:43.354+0000 7f078ed9d640 1 --2- 172.21.15.16:0/1591427553 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f079015fcf0 0x7f07900af890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:43.358 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:43.354+0000 7f078ed9d640 1 -- 172.21.15.16:0/1591427553 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f07900b0250 msgr2=0x7f07900b2330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:43.358 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:43.354+0000 7f078ed9d640 1 --2- 172.21.15.16:0/1591427553 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f07900b0250 0x7f07900b2330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:43.358 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:43.354+0000 7f078ed9d640 1 -- 172.21.15.16:0/1591427553 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0788009a70 con 0x7f0790070fb0 2023-05-22T16:03:43.358 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:43.354+0000 7f078e59c640 1 --2- 172.21.15.16:0/1591427553 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f07900b0250 0x7f07900b2330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T16:03:43.359 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:43.358+0000 7f078ed9d640 1 --2- 172.21.15.16:0/1591427553 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0790070fb0 0x7f0790151c10 secure :-1 s=READY pgs=2261 cs=0 l=1 rev1=1 crypto rx=0x7f07800011f0 tx=0x7f078000c9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:03:43.360 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:43.358+0000 7f07777fe640 1 -- 172.21.15.16:0/1591427553 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07800074e0 con 0x7f0790070fb0 2023-05-22T16:03:43.360 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:43.358+0000 7f07777fe640 1 -- 172.21.15.16:0/1591427553 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0780007640 con 0x7f0790070fb0 2023-05-22T16:03:43.360 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:43.358+0000 7f07950de640 1 -- 172.21.15.16:0/1591427553 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07900c1970 con 0x7f0790070fb0 2023-05-22T16:03:43.360 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:43.358+0000 7f07777fe640 1 -- 172.21.15.16:0/1591427553 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0780004a10 con 0x7f0790070fb0 2023-05-22T16:03:43.360 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:43.358+0000 7f07950de640 1 -- 172.21.15.16:0/1591427553 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07900c17e0 con 0x7f0790070fb0 2023-05-22T16:03:43.361 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:43.358+0000 7f07950de640 1 -- 172.21.15.16:0/1591427553 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0790071380 con 0x7f0790070fb0 2023-05-22T16:03:43.361 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:43.358+0000 7f07777fe640 1 -- 172.21.15.16:0/1591427553 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 82) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f0780004b70 con 0x7f0790070fb0 2023-05-22T16:03:43.361 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:43.358+0000 7f07777fe640 1 --2- 172.21.15.16:0/1591427553 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f077c0654a0 0x7f077c067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:43.361 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:43.358+0000 7f07777fe640 1 -- 172.21.15.16:0/1591427553 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(179..179 src has 1..179) v4 ==== 6438+0+0 (secure 0 0 0) 0x7f0780081fb0 con 0x7f0790070fb0 2023-05-22T16:03:43.361 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:43.358+0000 7f078e59c640 1 --2- 172.21.15.16:0/1591427553 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f077c0654a0 0x7f077c067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:43.362 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:43.358+0000 7f078e59c640 1 --2- 172.21.15.16:0/1591427553 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f077c0654a0 0x7f077c067950 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f07900c2e80 tx=0x7f0784008040 comp rx=0 tx=0).ready entity=mgr.10032 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:03:43.362 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:43.362+0000 7f07777fe640 1 -- 172.21.15.16:0/1591427553 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f078004fa30 con 0x7f0790070fb0 2023-05-22T16:03:43.513 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:43.514+0000 7f07950de640 1 -- 172.21.15.16:0/1591427553 --> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f07900cb210 con 0x7f077c0654a0 2023-05-22T16:03:43.526 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:43.526+0000 7f07777fe640 1 -- 172.21.15.16:0/1591427553 <== mgr.10032 v2:172.21.15.16:6832/65890 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+309917 (secure 0 0 0) 0x7f07900cb210 con 0x7f077c0654a0 2023-05-22T16:03:43.530 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T16:03:43.533 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2023-05-22T16:03:43.535 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:43.530+0000 7f07950de640 1 -- 172.21.15.16:0/1591427553 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f077c0654a0 msgr2=0x7f077c067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:43.536 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:43.530+0000 7f07950de640 1 --2- 172.21.15.16:0/1591427553 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f077c0654a0 0x7f077c067950 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f07900c2e80 tx=0x7f0784008040 comp rx=0 tx=0).stop 2023-05-22T16:03:43.536 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:43.530+0000 7f07950de640 1 -- 172.21.15.16:0/1591427553 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0790070fb0 msgr2=0x7f0790151c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:43.536 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:43.530+0000 7f07950de640 1 --2- 172.21.15.16:0/1591427553 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0790070fb0 0x7f0790151c10 secure :-1 s=READY pgs=2261 cs=0 l=1 rev1=1 crypto rx=0x7f07800011f0 tx=0x7f078000c9b0 comp rx=0 tx=0).stop 2023-05-22T16:03:43.537 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:43.530+0000 7f07950de640 1 -- 172.21.15.16:0/1591427553 shutdown_connections 2023-05-22T16:03:43.537 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:43.530+0000 7f07950de640 1 --2- 172.21.15.16:0/1591427553 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f079015fcf0 0x7f07900af890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:43.537 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:43.530+0000 7f07950de640 1 --2- 172.21.15.16:0/1591427553 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f07900b0250 0x7f07900b2330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:43.538 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:43.530+0000 7f07950de640 1 --2- 172.21.15.16:0/1591427553 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f077c0654a0 0x7f077c067950 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:43.538 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:43.530+0000 7f07950de640 1 --2- 172.21.15.16:0/1591427553 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0790070fb0 0x7f0790151c10 unknown :-1 s=CLOSED pgs=2261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:43.538 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:43.530+0000 7f07950de640 1 -- 172.21.15.16:0/1591427553 >> 172.21.15.16:0/1591427553 conn(0x7f0790069720 msgr2=0x7f079015f5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:03:43.538 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:43.530+0000 7f07950de640 1 -- 172.21.15.16:0/1591427553 shutdown_connections 2023-05-22T16:03:43.539 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:43.530+0000 7f07950de640 1 -- 172.21.15.16:0/1591427553 wait complete. 2023-05-22T16:03:43.550 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":25,"stamp":"2023-05-22T16:03:41.738188+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592698,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1748,"num_read_kb":3602,"num_write":543,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":342,"ondisk_log_size":342,"up":210,"acting":210,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":97,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":237160,"kb_used_data":18600,"kb_used_omap":0,"kb_used_meta":218496,"kb_avail":754737560,"statfs":{"total":773094113280,"available":772851261440,"internally_reserved":0,"allocated":19046400,"data_stored":10676364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":223739904},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":62,"apply_latency_ms":62,"commit_latency_ns":62000000,"apply_latency_ns":62000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2330,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.349851"},"pg_stats":[{"pgid":"23.3f","version":"0'0","reported_seq":9,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.834936+0000","last_change":"2023-05-22T16:03:38.831697+0000","last_active":"2023-05-22T16:03:38.834936+0000","last_peered":"2023-05-22T16:03:38.834936+0000","last_clean":"2023-05-22T16:03:38.834936+0000","last_became_active":"2023-05-22T16:03:38.831380+0000","last_became_peered":"2023-05-22T16:03:38.831380+0000","last_unstale":"2023-05-22T16:03:38.834936+0000","last_undegraded":"2023-05-22T16:03:38.834936+0000","last_fullsized":"2023-05-22T16:03:38.834936+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_clean_scrub_stamp":"2023-05-22T16:03:38.387135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:53:38.976658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.3e","version":"0'0","reported_seq":9,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.831119+0000","last_change":"2023-05-22T16:03:38.824745+0000","last_active":"2023-05-22T16:03:38.831119+0000","last_peered":"2023-05-22T16:03:38.831119+0000","last_clean":"2023-05-22T16:03:38.831119+0000","last_became_active":"2023-05-22T16:03:38.819756+0000","last_became_peered":"2023-05-22T16:03:38.819756+0000","last_unstale":"2023-05-22T16:03:38.831119+0000","last_undegraded":"2023-05-22T16:03:38.831119+0000","last_fullsized":"2023-05-22T16:03:38.831119+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_clean_scrub_stamp":"2023-05-22T16:03:38.387135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:42:06.604797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.3d","version":"0'0","reported_seq":9,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.837362+0000","last_change":"2023-05-22T16:03:38.828466+0000","last_active":"2023-05-22T16:03:38.837362+0000","last_peered":"2023-05-22T16:03:38.837362+0000","last_clean":"2023-05-22T16:03:38.837362+0000","last_became_active":"2023-05-22T16:03:38.826096+0000","last_became_peered":"2023-05-22T16:03:38.826096+0000","last_unstale":"2023-05-22T16:03:38.837362+0000","last_undegraded":"2023-05-22T16:03:38.837362+0000","last_fullsized":"2023-05-22T16:03:38.837362+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_clean_scrub_stamp":"2023-05-22T16:03:38.387135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:03:18.095743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.3c","version":"0'0","reported_seq":9,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.813436+0000","last_change":"2023-05-22T16:03:38.810194+0000","last_active":"2023-05-22T16:03:38.813436+0000","last_peered":"2023-05-22T16:03:38.813436+0000","last_clean":"2023-05-22T16:03:38.813436+0000","last_became_active":"2023-05-22T16:03:38.809542+0000","last_became_peered":"2023-05-22T16:03:38.809542+0000","last_unstale":"2023-05-22T16:03:38.813436+0000","last_undegraded":"2023-05-22T16:03:38.813436+0000","last_fullsized":"2023-05-22T16:03:38.813436+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_clean_scrub_stamp":"2023-05-22T16:03:38.387135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:01:48.420792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.3b","version":"0'0","reported_seq":9,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.838725+0000","last_change":"2023-05-22T16:03:38.837188+0000","last_active":"2023-05-22T16:03:38.838725+0000","last_peered":"2023-05-22T16:03:38.838725+0000","last_clean":"2023-05-22T16:03:38.838725+0000","last_became_active":"2023-05-22T16:03:38.836750+0000","last_became_peered":"2023-05-22T16:03:38.836750+0000","last_unstale":"2023-05-22T16:03:38.838725+0000","last_undegraded":"2023-05-22T16:03:38.838725+0000","last_fullsized":"2023-05-22T16:03:38.838725+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_clean_scrub_stamp":"2023-05-22T16:03:38.387135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:14:08.133076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.3a","version":"0'0","reported_seq":9,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.817549+0000","last_change":"2023-05-22T16:03:38.803227+0000","last_active":"2023-05-22T16:03:38.817549+0000","last_peered":"2023-05-22T16:03:38.817549+0000","last_clean":"2023-05-22T16:03:38.817549+0000","last_became_active":"2023-05-22T16:03:38.802489+0000","last_became_peered":"2023-05-22T16:03:38.802489+0000","last_unstale":"2023-05-22T16:03:38.817549+0000","last_undegraded":"2023-05-22T16:03:38.817549+0000","last_fullsized":"2023-05-22T16:03:38.817549+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_clean_scrub_stamp":"2023-05-22T16:03:38.387135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:16:16.828826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.39","version":"0'0","reported_seq":9,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.812527+0000","last_change":"2023-05-22T16:03:38.806822+0000","last_active":"2023-05-22T16:03:38.812527+0000","last_peered":"2023-05-22T16:03:38.812527+0000","last_clean":"2023-05-22T16:03:38.812527+0000","last_became_active":"2023-05-22T16:03:38.806056+0000","last_became_peered":"2023-05-22T16:03:38.806056+0000","last_unstale":"2023-05-22T16:03:38.812527+0000","last_undegraded":"2023-05-22T16:03:38.812527+0000","last_fullsized":"2023-05-22T16:03:38.812527+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_clean_scrub_stamp":"2023-05-22T16:03:38.387135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:48:38.105506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.38","version":"0'0","reported_seq":9,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.836771+0000","last_change":"2023-05-22T16:03:38.830698+0000","last_active":"2023-05-22T16:03:38.836771+0000","last_peered":"2023-05-22T16:03:38.836771+0000","last_clean":"2023-05-22T16:03:38.836771+0000","last_became_active":"2023-05-22T16:03:38.830389+0000","last_became_peered":"2023-05-22T16:03:38.830389+0000","last_unstale":"2023-05-22T16:03:38.836771+0000","last_undegraded":"2023-05-22T16:03:38.836771+0000","last_fullsized":"2023-05-22T16:03:38.836771+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_clean_scrub_stamp":"2023-05-22T16:03:38.387135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:40:17.506020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.37","version":"0'0","reported_seq":9,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.831076+0000","last_change":"2023-05-22T16:03:38.818858+0000","last_active":"2023-05-22T16:03:38.831076+0000","last_peered":"2023-05-22T16:03:38.831076+0000","last_clean":"2023-05-22T16:03:38.831076+0000","last_became_active":"2023-05-22T16:03:38.818431+0000","last_became_peered":"2023-05-22T16:03:38.818431+0000","last_unstale":"2023-05-22T16:03:38.831076+0000","last_undegraded":"2023-05-22T16:03:38.831076+0000","last_fullsized":"2023-05-22T16:03:38.831076+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_clean_scrub_stamp":"2023-05-22T16:03:38.387135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:22:36.886924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.36","version":"0'0","reported_seq":9,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.831151+0000","last_change":"2023-05-22T16:03:38.825183+0000","last_active":"2023-05-22T16:03:38.831151+0000","last_peered":"2023-05-22T16:03:38.831151+0000","last_clean":"2023-05-22T16:03:38.831151+0000","last_became_active":"2023-05-22T16:03:38.824914+0000","last_became_peered":"2023-05-22T16:03:38.824914+0000","last_unstale":"2023-05-22T16:03:38.831151+0000","last_undegraded":"2023-05-22T16:03:38.831151+0000","last_fullsized":"2023-05-22T16:03:38.831151+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_clean_scrub_stamp":"2023-05-22T16:03:38.387135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:20:29.890916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.35","version":"0'0","reported_seq":9,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.839055+0000","last_change":"2023-05-22T16:03:38.837665+0000","last_active":"2023-05-22T16:03:38.839055+0000","last_peered":"2023-05-22T16:03:38.839055+0000","last_clean":"2023-05-22T16:03:38.839055+0000","last_became_active":"2023-05-22T16:03:38.837234+0000","last_became_peered":"2023-05-22T16:03:38.837234+0000","last_unstale":"2023-05-22T16:03:38.839055+0000","last_undegraded":"2023-05-22T16:03:38.839055+0000","last_fullsized":"2023-05-22T16:03:38.839055+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_clean_scrub_stamp":"2023-05-22T16:03:38.387135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:44:20.427086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.34","version":"0'0","reported_seq":9,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.831026+0000","last_change":"2023-05-22T16:03:38.827948+0000","last_active":"2023-05-22T16:03:38.831026+0000","last_peered":"2023-05-22T16:03:38.831026+0000","last_clean":"2023-05-22T16:03:38.831026+0000","last_became_active":"2023-05-22T16:03:38.827558+0000","last_became_peered":"2023-05-22T16:03:38.827558+0000","last_unstale":"2023-05-22T16:03:38.831026+0000","last_undegraded":"2023-05-22T16:03:38.831026+0000","last_fullsized":"2023-05-22T16:03:38.831026+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_clean_scrub_stamp":"2023-05-22T16:03:38.387135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T03:42:31.724542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.33","version":"0'0","reported_seq":9,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.808994+0000","last_change":"2023-05-22T16:03:38.805157+0000","last_active":"2023-05-22T16:03:38.808994+0000","last_peered":"2023-05-22T16:03:38.808994+0000","last_clean":"2023-05-22T16:03:38.808994+0000","last_became_active":"2023-05-22T16:03:38.804749+0000","last_became_peered":"2023-05-22T16:03:38.804749+0000","last_unstale":"2023-05-22T16:03:38.808994+0000","last_undegraded":"2023-05-22T16:03:38.808994+0000","last_fullsized":"2023-05-22T16:03:38.808994+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_clean_scrub_stamp":"2023-05-22T16:03:38.387135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:49:45.112747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.32","version":"0'0","reported_seq":9,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.834252+0000","last_change":"2023-05-22T16:03:38.825823+0000","last_active":"2023-05-22T16:03:38.834252+0000","last_peered":"2023-05-22T16:03:38.834252+0000","last_clean":"2023-05-22T16:03:38.834252+0000","last_became_active":"2023-05-22T16:03:38.825281+0000","last_became_peered":"2023-05-22T16:03:38.825281+0000","last_unstale":"2023-05-22T16:03:38.834252+0000","last_undegraded":"2023-05-22T16:03:38.834252+0000","last_fullsized":"2023-05-22T16:03:38.834252+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_clean_scrub_stamp":"2023-05-22T16:03:38.387135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T03:06:41.364328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.31","version":"0'0","reported_seq":9,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.830706+0000","last_change":"2023-05-22T16:03:38.812630+0000","last_active":"2023-05-22T16:03:38.830706+0000","last_peered":"2023-05-22T16:03:38.830706+0000","last_clean":"2023-05-22T16:03:38.830706+0000","last_became_active":"2023-05-22T16:03:38.811901+0000","last_became_peered":"2023-05-22T16:03:38.811901+0000","last_unstale":"2023-05-22T16:03:38.830706+0000","last_undegraded":"2023-05-22T16:03:38.830706+0000","last_fullsized":"2023-05-22T16:03:38.830706+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_clean_scrub_stamp":"2023-05-22T16:03:38.387135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:08:59.616966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.30","version":"0'0","reported_seq":9,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.830817+0000","last_change":"2023-05-22T16:03:38.829912+0000","last_active":"2023-05-22T16:03:38.830817+0000","last_peered":"2023-05-22T16:03:38.830817+0000","last_clean":"2023-05-22T16:03:38.830817+0000","last_became_active":"2023-05-22T16:03:38.829561+0000","last_became_peered":"2023-05-22T16:03:38.829561+0000","last_unstale":"2023-05-22T16:03:38.830817+0000","last_undegraded":"2023-05-22T16:03:38.830817+0000","last_fullsized":"2023-05-22T16:03:38.830817+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_clean_scrub_stamp":"2023-05-22T16:03:38.387135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:20:17.651383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.2f","version":"0'0","reported_seq":9,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.817945+0000","last_change":"2023-05-22T16:03:38.809726+0000","last_active":"2023-05-22T16:03:38.817945+0000","last_peered":"2023-05-22T16:03:38.817945+0000","last_clean":"2023-05-22T16:03:38.817945+0000","last_became_active":"2023-05-22T16:03:38.809472+0000","last_became_peered":"2023-05-22T16:03:38.809472+0000","last_unstale":"2023-05-22T16:03:38.817945+0000","last_undegraded":"2023-05-22T16:03:38.817945+0000","last_fullsized":"2023-05-22T16:03:38.817945+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_clean_scrub_stamp":"2023-05-22T16:03:38.387135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:05:33.932707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.2e","version":"0'0","reported_seq":9,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.806937+0000","last_change":"2023-05-22T16:03:38.800994+0000","last_active":"2023-05-22T16:03:38.806937+0000","last_peered":"2023-05-22T16:03:38.806937+0000","last_clean":"2023-05-22T16:03:38.806937+0000","last_became_active":"2023-05-22T16:03:38.800724+0000","last_became_peered":"2023-05-22T16:03:38.800724+0000","last_unstale":"2023-05-22T16:03:38.806937+0000","last_undegraded":"2023-05-22T16:03:38.806937+0000","last_fullsized":"2023-05-22T16:03:38.806937+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_clean_scrub_stamp":"2023-05-22T16:03:38.387135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T18:39:51.525114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.2d","version":"0'0","reported_seq":9,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.817455+0000","last_change":"2023-05-22T16:03:38.808401+0000","last_active":"2023-05-22T16:03:38.817455+0000","last_peered":"2023-05-22T16:03:38.817455+0000","last_clean":"2023-05-22T16:03:38.817455+0000","last_became_active":"2023-05-22T16:03:38.807633+0000","last_became_peered":"2023-05-22T16:03:38.807633+0000","last_unstale":"2023-05-22T16:03:38.817455+0000","last_undegraded":"2023-05-22T16:03:38.817455+0000","last_fullsized":"2023-05-22T16:03:38.817455+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_clean_scrub_stamp":"2023-05-22T16:03:38.387135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:59:10.609792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.2c","version":"0'0","reported_seq":9,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.834197+0000","last_change":"2023-05-22T16:03:38.825547+0000","last_active":"2023-05-22T16:03:38.834197+0000","last_peered":"2023-05-22T16:03:38.834197+0000","last_clean":"2023-05-22T16:03:38.834197+0000","last_became_active":"2023-05-22T16:03:38.825059+0000","last_became_peered":"2023-05-22T16:03:38.825059+0000","last_unstale":"2023-05-22T16:03:38.834197+0000","last_undegraded":"2023-05-22T16:03:38.834197+0000","last_fullsized":"2023-05-22T16:03:38.834197+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_clean_scrub_stamp":"2023-05-22T16:03:38.387135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:03:34.709150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.2b","version":"0'0","reported_seq":9,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.826073+0000","last_change":"2023-05-22T16:03:38.825090+0000","last_active":"2023-05-22T16:03:38.826073+0000","last_peered":"2023-05-22T16:03:38.826073+0000","last_clean":"2023-05-22T16:03:38.826073+0000","last_became_active":"2023-05-22T16:03:38.824769+0000","last_became_peered":"2023-05-22T16:03:38.824769+0000","last_unstale":"2023-05-22T16:03:38.826073+0000","last_undegraded":"2023-05-22T16:03:38.826073+0000","last_fullsized":"2023-05-22T16:03:38.826073+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_clean_scrub_stamp":"2023-05-22T16:03:38.387135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:32:40.328940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.2a","version":"0'0","reported_seq":9,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.831201+0000","last_change":"2023-05-22T16:03:38.825914+0000","last_active":"2023-05-22T16:03:38.831201+0000","last_peered":"2023-05-22T16:03:38.831201+0000","last_clean":"2023-05-22T16:03:38.831201+0000","last_became_active":"2023-05-22T16:03:38.825706+0000","last_became_peered":"2023-05-22T16:03:38.825706+0000","last_unstale":"2023-05-22T16:03:38.831201+0000","last_undegraded":"2023-05-22T16:03:38.831201+0000","last_fullsized":"2023-05-22T16:03:38.831201+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_clean_scrub_stamp":"2023-05-22T16:03:38.387135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:55:10.166213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.29","version":"0'0","reported_seq":9,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.829415+0000","last_change":"2023-05-22T16:03:38.828535+0000","last_active":"2023-05-22T16:03:38.829415+0000","last_peered":"2023-05-22T16:03:38.829415+0000","last_clean":"2023-05-22T16:03:38.829415+0000","last_became_active":"2023-05-22T16:03:38.828237+0000","last_became_peered":"2023-05-22T16:03:38.828237+0000","last_unstale":"2023-05-22T16:03:38.829415+0000","last_undegraded":"2023-05-22T16:03:38.829415+0000","last_fullsized":"2023-05-22T16:03:38.829415+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_clean_scrub_stamp":"2023-05-22T16:03:38.387135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T03:22:36.354019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.28","version":"0'0","reported_seq":9,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.830921+0000","last_change":"2023-05-22T16:03:38.819073+0000","last_active":"2023-05-22T16:03:38.830921+0000","last_peered":"2023-05-22T16:03:38.830921+0000","last_clean":"2023-05-22T16:03:38.830921+0000","last_became_active":"2023-05-22T16:03:38.817944+0000","last_became_peered":"2023-05-22T16:03:38.817944+0000","last_unstale":"2023-05-22T16:03:38.830921+0000","last_undegraded":"2023-05-22T16:03:38.830921+0000","last_fullsized":"2023-05-22T16:03:38.830921+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_clean_scrub_stamp":"2023-05-22T16:03:38.387135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:17:50.697042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.27","version":"0'0","reported_seq":9,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.838058+0000","last_change":"2023-05-22T16:03:38.836502+0000","last_active":"2023-05-22T16:03:38.838058+0000","last_peered":"2023-05-22T16:03:38.838058+0000","last_clean":"2023-05-22T16:03:38.838058+0000","last_became_active":"2023-05-22T16:03:38.836238+0000","last_became_peered":"2023-05-22T16:03:38.836238+0000","last_unstale":"2023-05-22T16:03:38.838058+0000","last_undegraded":"2023-05-22T16:03:38.838058+0000","last_fullsized":"2023-05-22T16:03:38.838058+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_clean_scrub_stamp":"2023-05-22T16:03:38.387135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:43:01.268208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.26","version":"0'0","reported_seq":9,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.818774+0000","last_change":"2023-05-22T16:03:38.816847+0000","last_active":"2023-05-22T16:03:38.818774+0000","last_peered":"2023-05-22T16:03:38.818774+0000","last_clean":"2023-05-22T16:03:38.818774+0000","last_became_active":"2023-05-22T16:03:38.816335+0000","last_became_peered":"2023-05-22T16:03:38.816335+0000","last_unstale":"2023-05-22T16:03:38.818774+0000","last_undegraded":"2023-05-22T16:03:38.818774+0000","last_fullsized":"2023-05-22T16:03:38.818774+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_clean_scrub_stamp":"2023-05-22T16:03:38.387135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:30:36.909855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.25","version":"0'0","reported_seq":9,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.837336+0000","last_change":"2023-05-22T16:03:38.826726+0000","last_active":"2023-05-22T16:03:38.837336+0000","last_peered":"2023-05-22T16:03:38.837336+0000","last_clean":"2023-05-22T16:03:38.837336+0000","last_became_active":"2023-05-22T16:03:38.822171+0000","last_became_peered":"2023-05-22T16:03:38.822171+0000","last_unstale":"2023-05-22T16:03:38.837336+0000","last_undegraded":"2023-05-22T16:03:38.837336+0000","last_fullsized":"2023-05-22T16:03:38.837336+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_clean_scrub_stamp":"2023-05-22T16:03:38.387135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T16:33:34.861902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.24","version":"0'0","reported_seq":9,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.826521+0000","last_change":"2023-05-22T16:03:38.825353+0000","last_active":"2023-05-22T16:03:38.826521+0000","last_peered":"2023-05-22T16:03:38.826521+0000","last_clean":"2023-05-22T16:03:38.826521+0000","last_became_active":"2023-05-22T16:03:38.824976+0000","last_became_peered":"2023-05-22T16:03:38.824976+0000","last_unstale":"2023-05-22T16:03:38.826521+0000","last_undegraded":"2023-05-22T16:03:38.826521+0000","last_fullsized":"2023-05-22T16:03:38.826521+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_clean_scrub_stamp":"2023-05-22T16:03:38.387135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:14:08.133076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.23","version":"0'0","reported_seq":9,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.834226+0000","last_change":"2023-05-22T16:03:38.824726+0000","last_active":"2023-05-22T16:03:38.834226+0000","last_peered":"2023-05-22T16:03:38.834226+0000","last_clean":"2023-05-22T16:03:38.834226+0000","last_became_active":"2023-05-22T16:03:38.822077+0000","last_became_peered":"2023-05-22T16:03:38.822077+0000","last_unstale":"2023-05-22T16:03:38.834226+0000","last_undegraded":"2023-05-22T16:03:38.834226+0000","last_fullsized":"2023-05-22T16:03:38.834226+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_clean_scrub_stamp":"2023-05-22T16:03:38.387135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:59:31.259784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.22","version":"0'0","reported_seq":9,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.837983+0000","last_change":"2023-05-22T16:03:38.828353+0000","last_active":"2023-05-22T16:03:38.837983+0000","last_peered":"2023-05-22T16:03:38.837983+0000","last_clean":"2023-05-22T16:03:38.837983+0000","last_became_active":"2023-05-22T16:03:38.828046+0000","last_became_peered":"2023-05-22T16:03:38.828046+0000","last_unstale":"2023-05-22T16:03:38.837983+0000","last_undegraded":"2023-05-22T16:03:38.837983+0000","last_fullsized":"2023-05-22T16:03:38.837983+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_clean_scrub_stamp":"2023-05-22T16:03:38.387135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:34:38.930308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.21","version":"0'0","reported_seq":9,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.836140+0000","last_change":"2023-05-22T16:03:38.815565+0000","last_active":"2023-05-22T16:03:38.836140+0000","last_peered":"2023-05-22T16:03:38.836140+0000","last_clean":"2023-05-22T16:03:38.836140+0000","last_became_active":"2023-05-22T16:03:38.814892+0000","last_became_peered":"2023-05-22T16:03:38.814892+0000","last_unstale":"2023-05-22T16:03:38.836140+0000","last_undegraded":"2023-05-22T16:03:38.836140+0000","last_fullsized":"2023-05-22T16:03:38.836140+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_clean_scrub_stamp":"2023-05-22T16:03:38.387135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T03:14:45.639272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.20","version":"0'0","reported_seq":9,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.806931+0000","last_change":"2023-05-22T16:03:38.800427+0000","last_active":"2023-05-22T16:03:38.806931+0000","last_peered":"2023-05-22T16:03:38.806931+0000","last_clean":"2023-05-22T16:03:38.806931+0000","last_became_active":"2023-05-22T16:03:38.800151+0000","last_became_peered":"2023-05-22T16:03:38.800151+0000","last_unstale":"2023-05-22T16:03:38.806931+0000","last_undegraded":"2023-05-22T16:03:38.806931+0000","last_fullsized":"2023-05-22T16:03:38.806931+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_clean_scrub_stamp":"2023-05-22T16:03:38.387135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T03:14:45.639272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.1e","version":"0'0","reported_seq":9,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.818753+0000","last_change":"2023-05-22T16:03:38.816615+0000","last_active":"2023-05-22T16:03:38.818753+0000","last_peered":"2023-05-22T16:03:38.818753+0000","last_clean":"2023-05-22T16:03:38.818753+0000","last_became_active":"2023-05-22T16:03:38.815975+0000","last_became_peered":"2023-05-22T16:03:38.815975+0000","last_unstale":"2023-05-22T16:03:38.818753+0000","last_undegraded":"2023-05-22T16:03:38.818753+0000","last_fullsized":"2023-05-22T16:03:38.818753+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_clean_scrub_stamp":"2023-05-22T16:03:38.387135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:58:45.869044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.1f","version":"178'4","reported_seq":14,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.834972+0000","last_change":"2023-05-22T16:03:38.447497+0000","last_active":"2023-05-22T16:03:38.834972+0000","last_peered":"2023-05-22T16:03:38.834972+0000","last_clean":"2023-05-22T16:03:38.834972+0000","last_became_active":"2023-05-22T16:03:38.446759+0000","last_became_peered":"2023-05-22T16:03:38.446759+0000","last_unstale":"2023-05-22T16:03:38.834972+0000","last_undegraded":"2023-05-22T16:03:38.834972+0000","last_fullsized":"2023-05-22T16:03:38.834972+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:03:37.383765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:03:37.383765+0000","last_clean_scrub_stamp":"2023-05-22T16:03:37.383765+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:55:15.045866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":566,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.1f","version":"0'0","reported_seq":9,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.835759+0000","last_change":"2023-05-22T16:03:38.807854+0000","last_active":"2023-05-22T16:03:38.835759+0000","last_peered":"2023-05-22T16:03:38.835759+0000","last_clean":"2023-05-22T16:03:38.835759+0000","last_became_active":"2023-05-22T16:03:38.807446+0000","last_became_peered":"2023-05-22T16:03:38.807446+0000","last_unstale":"2023-05-22T16:03:38.835759+0000","last_undegraded":"2023-05-22T16:03:38.835759+0000","last_fullsized":"2023-05-22T16:03:38.835759+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_clean_scrub_stamp":"2023-05-22T16:03:38.387135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T18:18:52.766623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.1e","version":"178'1","reported_seq":11,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.835813+0000","last_change":"2023-05-22T16:03:38.450008+0000","last_active":"2023-05-22T16:03:38.835813+0000","last_peered":"2023-05-22T16:03:38.835813+0000","last_clean":"2023-05-22T16:03:38.835813+0000","last_became_active":"2023-05-22T16:03:38.449244+0000","last_became_peered":"2023-05-22T16:03:38.449244+0000","last_unstale":"2023-05-22T16:03:38.835813+0000","last_undegraded":"2023-05-22T16:03:38.835813+0000","last_fullsized":"2023-05-22T16:03:38.835813+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:03:37.383765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:03:37.383765+0000","last_clean_scrub_stamp":"2023-05-22T16:03:37.383765+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:44:19.423716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.1c","version":"0'0","reported_seq":9,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.818860+0000","last_change":"2023-05-22T16:03:38.817798+0000","last_active":"2023-05-22T16:03:38.818860+0000","last_peered":"2023-05-22T16:03:38.818860+0000","last_clean":"2023-05-22T16:03:38.818860+0000","last_became_active":"2023-05-22T16:03:38.817461+0000","last_became_peered":"2023-05-22T16:03:38.817461+0000","last_unstale":"2023-05-22T16:03:38.818860+0000","last_undegraded":"2023-05-22T16:03:38.818860+0000","last_fullsized":"2023-05-22T16:03:38.818860+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_clean_scrub_stamp":"2023-05-22T16:03:38.387135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:28:05.049393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.1d","version":"178'2","reported_seq":12,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.833072+0000","last_change":"2023-05-22T16:03:38.452005+0000","last_active":"2023-05-22T16:03:38.833072+0000","last_peered":"2023-05-22T16:03:38.833072+0000","last_clean":"2023-05-22T16:03:38.833072+0000","last_became_active":"2023-05-22T16:03:38.451740+0000","last_became_peered":"2023-05-22T16:03:38.451740+0000","last_unstale":"2023-05-22T16:03:38.833072+0000","last_undegraded":"2023-05-22T16:03:38.833072+0000","last_fullsized":"2023-05-22T16:03:38.833072+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:03:37.383765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:03:37.383765+0000","last_clean_scrub_stamp":"2023-05-22T16:03:37.383765+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:01:48.284969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.1d","version":"0'0","reported_seq":9,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.833903+0000","last_change":"2023-05-22T16:03:38.821882+0000","last_active":"2023-05-22T16:03:38.833903+0000","last_peered":"2023-05-22T16:03:38.833903+0000","last_clean":"2023-05-22T16:03:38.833903+0000","last_became_active":"2023-05-22T16:03:38.821601+0000","last_became_peered":"2023-05-22T16:03:38.821601+0000","last_unstale":"2023-05-22T16:03:38.833903+0000","last_undegraded":"2023-05-22T16:03:38.833903+0000","last_fullsized":"2023-05-22T16:03:38.833903+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_clean_scrub_stamp":"2023-05-22T16:03:38.387135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T16:32:33.168433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.1c","version":"178'2","reported_seq":12,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.831140+0000","last_change":"2023-05-22T16:03:38.437032+0000","last_active":"2023-05-22T16:03:38.831140+0000","last_peered":"2023-05-22T16:03:38.831140+0000","last_clean":"2023-05-22T16:03:38.831140+0000","last_became_active":"2023-05-22T16:03:38.436774+0000","last_became_peered":"2023-05-22T16:03:38.436774+0000","last_unstale":"2023-05-22T16:03:38.831140+0000","last_undegraded":"2023-05-22T16:03:38.831140+0000","last_fullsized":"2023-05-22T16:03:38.831140+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:03:37.383765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:03:37.383765+0000","last_clean_scrub_stamp":"2023-05-22T16:03:37.383765+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:03:33.705779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.1a","version":"0'0","reported_seq":9,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.813045+0000","last_change":"2023-05-22T16:03:38.802058+0000","last_active":"2023-05-22T16:03:38.813045+0000","last_peered":"2023-05-22T16:03:38.813045+0000","last_clean":"2023-05-22T16:03:38.813045+0000","last_became_active":"2023-05-22T16:03:38.801322+0000","last_became_peered":"2023-05-22T16:03:38.801322+0000","last_unstale":"2023-05-22T16:03:38.813045+0000","last_undegraded":"2023-05-22T16:03:38.813045+0000","last_fullsized":"2023-05-22T16:03:38.813045+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_clean_scrub_stamp":"2023-05-22T16:03:38.387135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T03:55:40.501656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.1b","version":"178'1","reported_seq":11,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.837599+0000","last_change":"2023-05-22T16:03:38.435428+0000","last_active":"2023-05-22T16:03:38.837599+0000","last_peered":"2023-05-22T16:03:38.837599+0000","last_clean":"2023-05-22T16:03:38.837599+0000","last_became_active":"2023-05-22T16:03:38.434696+0000","last_became_peered":"2023-05-22T16:03:38.434696+0000","last_unstale":"2023-05-22T16:03:38.837599+0000","last_undegraded":"2023-05-22T16:03:38.837599+0000","last_fullsized":"2023-05-22T16:03:38.837599+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:03:37.383765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:03:37.383765+0000","last_clean_scrub_stamp":"2023-05-22T16:03:37.383765+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:16:15.825456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.1b","version":"0'0","reported_seq":9,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.821402+0000","last_change":"2023-05-22T16:03:38.817010+0000","last_active":"2023-05-22T16:03:38.821402+0000","last_peered":"2023-05-22T16:03:38.821402+0000","last_clean":"2023-05-22T16:03:38.821402+0000","last_became_active":"2023-05-22T16:03:38.816702+0000","last_became_peered":"2023-05-22T16:03:38.816702+0000","last_unstale":"2023-05-22T16:03:38.821402+0000","last_undegraded":"2023-05-22T16:03:38.821402+0000","last_fullsized":"2023-05-22T16:03:38.821402+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_clean_scrub_stamp":"2023-05-22T16:03:38.387135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:44:38.836588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.1a","version":"178'1","reported_seq":11,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.820144+0000","last_change":"2023-05-22T16:03:38.424525+0000","last_active":"2023-05-22T16:03:38.820144+0000","last_peered":"2023-05-22T16:03:38.820144+0000","last_clean":"2023-05-22T16:03:38.820144+0000","last_became_active":"2023-05-22T16:03:38.424291+0000","last_became_peered":"2023-05-22T16:03:38.424291+0000","last_unstale":"2023-05-22T16:03:38.820144+0000","last_undegraded":"2023-05-22T16:03:38.820144+0000","last_fullsized":"2023-05-22T16:03:38.820144+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:03:37.383765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:03:37.383765+0000","last_clean_scrub_stamp":"2023-05-22T16:03:37.383765+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:22:12.076229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.18","version":"0'0","reported_seq":9,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.832156+0000","last_change":"2023-05-22T16:03:38.831443+0000","last_active":"2023-05-22T16:03:38.832156+0000","last_peered":"2023-05-22T16:03:38.832156+0000","last_clean":"2023-05-22T16:03:38.832156+0000","last_became_active":"2023-05-22T16:03:38.831234+0000","last_became_peered":"2023-05-22T16:03:38.831234+0000","last_unstale":"2023-05-22T16:03:38.832156+0000","last_undegraded":"2023-05-22T16:03:38.832156+0000","last_fullsized":"2023-05-22T16:03:38.832156+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_clean_scrub_stamp":"2023-05-22T16:03:38.387135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T03:42:31.724542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.19","version":"0'0","reported_seq":10,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.804574+0000","last_change":"2023-05-22T16:03:38.442821+0000","last_active":"2023-05-22T16:03:38.804574+0000","last_peered":"2023-05-22T16:03:38.804574+0000","last_clean":"2023-05-22T16:03:38.804574+0000","last_became_active":"2023-05-22T16:03:38.439824+0000","last_became_peered":"2023-05-22T16:03:38.439824+0000","last_unstale":"2023-05-22T16:03:38.804574+0000","last_undegraded":"2023-05-22T16:03:38.804574+0000","last_fullsized":"2023-05-22T16:03:38.804574+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:03:37.383765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:03:37.383765+0000","last_clean_scrub_stamp":"2023-05-22T16:03:37.383765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:43:00.264838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.19","version":"0'0","reported_seq":9,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.817959+0000","last_change":"2023-05-22T16:03:38.809063+0000","last_active":"2023-05-22T16:03:38.817959+0000","last_peered":"2023-05-22T16:03:38.817959+0000","last_clean":"2023-05-22T16:03:38.817959+0000","last_became_active":"2023-05-22T16:03:38.803361+0000","last_became_peered":"2023-05-22T16:03:38.803361+0000","last_unstale":"2023-05-22T16:03:38.817959+0000","last_undegraded":"2023-05-22T16:03:38.817959+0000","last_fullsized":"2023-05-22T16:03:38.817959+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_clean_scrub_stamp":"2023-05-22T16:03:38.387135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:18:36.200155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.18","version":"0'0","reported_seq":10,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.793003+0000","last_change":"2023-05-22T16:03:38.421051+0000","last_active":"2023-05-22T16:03:38.793003+0000","last_peered":"2023-05-22T16:03:38.793003+0000","last_clean":"2023-05-22T16:03:38.793003+0000","last_became_active":"2023-05-22T16:03:38.420654+0000","last_became_peered":"2023-05-22T16:03:38.420654+0000","last_unstale":"2023-05-22T16:03:38.793003+0000","last_undegraded":"2023-05-22T16:03:38.793003+0000","last_fullsized":"2023-05-22T16:03:38.793003+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:03:37.383765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:03:37.383765+0000","last_clean_scrub_stamp":"2023-05-22T16:03:37.383765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:00:41.383707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.e","version":"0'0","reported_seq":9,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.821582+0000","last_change":"2023-05-22T16:03:38.818612+0000","last_active":"2023-05-22T16:03:38.821582+0000","last_peered":"2023-05-22T16:03:38.821582+0000","last_clean":"2023-05-22T16:03:38.821582+0000","last_became_active":"2023-05-22T16:03:38.817701+0000","last_became_peered":"2023-05-22T16:03:38.817701+0000","last_unstale":"2023-05-22T16:03:38.821582+0000","last_undegraded":"2023-05-22T16:03:38.821582+0000","last_fullsized":"2023-05-22T16:03:38.821582+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_clean_scrub_stamp":"2023-05-22T16:03:38.387135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:55:10.166213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.f","version":"178'1","reported_seq":11,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.838466+0000","last_change":"2023-05-22T16:03:38.443237+0000","last_active":"2023-05-22T16:03:38.838466+0000","last_peered":"2023-05-22T16:03:38.838466+0000","last_clean":"2023-05-22T16:03:38.838466+0000","last_became_active":"2023-05-22T16:03:38.442601+0000","last_became_peered":"2023-05-22T16:03:38.442601+0000","last_unstale":"2023-05-22T16:03:38.838466+0000","last_undegraded":"2023-05-22T16:03:38.838466+0000","last_fullsized":"2023-05-22T16:03:38.838466+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:03:37.383765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:03:37.383765+0000","last_clean_scrub_stamp":"2023-05-22T16:03:37.383765+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:18:35.196785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.f","version":"0'0","reported_seq":9,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.825950+0000","last_change":"2023-05-22T16:03:38.825070+0000","last_active":"2023-05-22T16:03:38.825950+0000","last_peered":"2023-05-22T16:03:38.825950+0000","last_clean":"2023-05-22T16:03:38.825950+0000","last_became_active":"2023-05-22T16:03:38.824696+0000","last_became_peered":"2023-05-22T16:03:38.824696+0000","last_unstale":"2023-05-22T16:03:38.825950+0000","last_undegraded":"2023-05-22T16:03:38.825950+0000","last_fullsized":"2023-05-22T16:03:38.825950+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_clean_scrub_stamp":"2023-05-22T16:03:38.387135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:34:38.930308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.e","version":"178'2","reported_seq":12,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.833978+0000","last_change":"2023-05-22T16:03:38.424783+0000","last_active":"2023-05-22T16:03:38.833978+0000","last_peered":"2023-05-22T16:03:38.833978+0000","last_clean":"2023-05-22T16:03:38.833978+0000","last_became_active":"2023-05-22T16:03:38.423990+0000","last_became_peered":"2023-05-22T16:03:38.423990+0000","last_unstale":"2023-05-22T16:03:38.833978+0000","last_undegraded":"2023-05-22T16:03:38.833978+0000","last_fullsized":"2023-05-22T16:03:38.833978+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:03:37.383765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:03:37.383765+0000","last_clean_scrub_stamp":"2023-05-22T16:03:37.383765+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:27:32.880811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.c","version":"0'0","reported_seq":9,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.813462+0000","last_change":"2023-05-22T16:03:38.810867+0000","last_active":"2023-05-22T16:03:38.813462+0000","last_peered":"2023-05-22T16:03:38.813462+0000","last_clean":"2023-05-22T16:03:38.813462+0000","last_became_active":"2023-05-22T16:03:38.810495+0000","last_became_peered":"2023-05-22T16:03:38.810495+0000","last_unstale":"2023-05-22T16:03:38.813462+0000","last_undegraded":"2023-05-22T16:03:38.813462+0000","last_fullsized":"2023-05-22T16:03:38.813462+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_clean_scrub_stamp":"2023-05-22T16:03:38.387135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:29:41.023569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.d","version":"178'1","reported_seq":11,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.836748+0000","last_change":"2023-05-22T16:03:38.439903+0000","last_active":"2023-05-22T16:03:38.836748+0000","last_peered":"2023-05-22T16:03:38.836748+0000","last_clean":"2023-05-22T16:03:38.836748+0000","last_became_active":"2023-05-22T16:03:38.439194+0000","last_became_peered":"2023-05-22T16:03:38.439194+0000","last_unstale":"2023-05-22T16:03:38.836748+0000","last_undegraded":"2023-05-22T16:03:38.836748+0000","last_fullsized":"2023-05-22T16:03:38.836748+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:03:37.383765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:03:37.383765+0000","last_clean_scrub_stamp":"2023-05-22T16:03:37.383765+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T18:18:51.763252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.d","version":"0'0","reported_seq":9,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.825900+0000","last_change":"2023-05-22T16:03:38.824954+0000","last_active":"2023-05-22T16:03:38.825900+0000","last_peered":"2023-05-22T16:03:38.825900+0000","last_clean":"2023-05-22T16:03:38.825900+0000","last_became_active":"2023-05-22T16:03:38.824677+0000","last_became_peered":"2023-05-22T16:03:38.824677+0000","last_unstale":"2023-05-22T16:03:38.825900+0000","last_undegraded":"2023-05-22T16:03:38.825900+0000","last_fullsized":"2023-05-22T16:03:38.825900+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_clean_scrub_stamp":"2023-05-22T16:03:38.387135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:35:24.987929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.c","version":"0'0","reported_seq":10,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.795221+0000","last_change":"2023-05-22T16:03:38.433373+0000","last_active":"2023-05-22T16:03:38.795221+0000","last_peered":"2023-05-22T16:03:38.795221+0000","last_clean":"2023-05-22T16:03:38.795221+0000","last_became_active":"2023-05-22T16:03:38.432357+0000","last_became_peered":"2023-05-22T16:03:38.432357+0000","last_unstale":"2023-05-22T16:03:38.795221+0000","last_undegraded":"2023-05-22T16:03:38.795221+0000","last_fullsized":"2023-05-22T16:03:38.795221+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:03:37.383765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:03:37.383765+0000","last_clean_scrub_stamp":"2023-05-22T16:03:37.383765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:35:31.827140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.a","version":"0'0","reported_seq":9,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.829742+0000","last_change":"2023-05-22T16:03:38.806843+0000","last_active":"2023-05-22T16:03:38.829742+0000","last_peered":"2023-05-22T16:03:38.829742+0000","last_clean":"2023-05-22T16:03:38.829742+0000","last_became_active":"2023-05-22T16:03:38.806337+0000","last_became_peered":"2023-05-22T16:03:38.806337+0000","last_unstale":"2023-05-22T16:03:38.829742+0000","last_undegraded":"2023-05-22T16:03:38.829742+0000","last_fullsized":"2023-05-22T16:03:38.829742+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_clean_scrub_stamp":"2023-05-22T16:03:38.387135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:46:58.323247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.b","version":"178'2","reported_seq":12,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.832800+0000","last_change":"2023-05-22T16:03:38.444272+0000","last_active":"2023-05-22T16:03:38.832800+0000","last_peered":"2023-05-22T16:03:38.832800+0000","last_clean":"2023-05-22T16:03:38.832800+0000","last_became_active":"2023-05-22T16:03:38.444069+0000","last_became_peered":"2023-05-22T16:03:38.444069+0000","last_unstale":"2023-05-22T16:03:38.832800+0000","last_undegraded":"2023-05-22T16:03:38.832800+0000","last_fullsized":"2023-05-22T16:03:38.832800+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:03:37.383765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:03:37.383765+0000","last_clean_scrub_stamp":"2023-05-22T16:03:37.383765+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T03:47:45.896671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.b","version":"0'0","reported_seq":9,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.821783+0000","last_change":"2023-05-22T16:03:38.820971+0000","last_active":"2023-05-22T16:03:38.821783+0000","last_peered":"2023-05-22T16:03:38.821783+0000","last_clean":"2023-05-22T16:03:38.821783+0000","last_became_active":"2023-05-22T16:03:38.820633+0000","last_became_peered":"2023-05-22T16:03:38.820633+0000","last_unstale":"2023-05-22T16:03:38.821783+0000","last_undegraded":"2023-05-22T16:03:38.821783+0000","last_fullsized":"2023-05-22T16:03:38.821783+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_clean_scrub_stamp":"2023-05-22T16:03:38.387135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:45:10.755716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.a","version":"0'0","reported_seq":10,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.794722+0000","last_change":"2023-05-22T16:03:38.431470+0000","last_active":"2023-05-22T16:03:38.794722+0000","last_peered":"2023-05-22T16:03:38.794722+0000","last_clean":"2023-05-22T16:03:38.794722+0000","last_became_active":"2023-05-22T16:03:38.430646+0000","last_became_peered":"2023-05-22T16:03:38.430646+0000","last_unstale":"2023-05-22T16:03:38.794722+0000","last_undegraded":"2023-05-22T16:03:38.794722+0000","last_fullsized":"2023-05-22T16:03:38.794722+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:03:37.383765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:03:37.383765+0000","last_clean_scrub_stamp":"2023-05-22T16:03:37.383765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:30:37.858474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.8","version":"0'0","reported_seq":9,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.834928+0000","last_change":"2023-05-22T16:03:38.831619+0000","last_active":"2023-05-22T16:03:38.834928+0000","last_peered":"2023-05-22T16:03:38.834928+0000","last_clean":"2023-05-22T16:03:38.834928+0000","last_became_active":"2023-05-22T16:03:38.831313+0000","last_became_peered":"2023-05-22T16:03:38.831313+0000","last_unstale":"2023-05-22T16:03:38.834928+0000","last_undegraded":"2023-05-22T16:03:38.834928+0000","last_fullsized":"2023-05-22T16:03:38.834928+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_clean_scrub_stamp":"2023-05-22T16:03:38.387135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T03:37:21.656379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.9","version":"0'0","reported_seq":10,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.793329+0000","last_change":"2023-05-22T16:03:38.454438+0000","last_active":"2023-05-22T16:03:38.793329+0000","last_peered":"2023-05-22T16:03:38.793329+0000","last_clean":"2023-05-22T16:03:38.793329+0000","last_became_active":"2023-05-22T16:03:38.453989+0000","last_became_peered":"2023-05-22T16:03:38.453989+0000","last_unstale":"2023-05-22T16:03:38.793329+0000","last_undegraded":"2023-05-22T16:03:38.793329+0000","last_fullsized":"2023-05-22T16:03:38.793329+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:03:37.383765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:03:37.383765+0000","last_clean_scrub_stamp":"2023-05-22T16:03:37.383765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:57:38.049040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.9","version":"0'0","reported_seq":9,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.829435+0000","last_change":"2023-05-22T16:03:38.828625+0000","last_active":"2023-05-22T16:03:38.829435+0000","last_peered":"2023-05-22T16:03:38.829435+0000","last_clean":"2023-05-22T16:03:38.829435+0000","last_became_active":"2023-05-22T16:03:38.828315+0000","last_became_peered":"2023-05-22T16:03:38.828315+0000","last_unstale":"2023-05-22T16:03:38.829435+0000","last_undegraded":"2023-05-22T16:03:38.829435+0000","last_fullsized":"2023-05-22T16:03:38.829435+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_clean_scrub_stamp":"2023-05-22T16:03:38.387135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:46:56.808362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.8","version":"0'0","reported_seq":10,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.812433+0000","last_change":"2023-05-22T16:03:38.436430+0000","last_active":"2023-05-22T16:03:38.812433+0000","last_peered":"2023-05-22T16:03:38.812433+0000","last_clean":"2023-05-22T16:03:38.812433+0000","last_became_active":"2023-05-22T16:03:38.434194+0000","last_became_peered":"2023-05-22T16:03:38.434194+0000","last_unstale":"2023-05-22T16:03:38.812433+0000","last_undegraded":"2023-05-22T16:03:38.812433+0000","last_fullsized":"2023-05-22T16:03:38.812433+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:03:37.383765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:03:37.383765+0000","last_clean_scrub_stamp":"2023-05-22T16:03:37.383765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:34:37.926938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.6","version":"0'0","reported_seq":9,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.821360+0000","last_change":"2023-05-22T16:03:38.814348+0000","last_active":"2023-05-22T16:03:38.821360+0000","last_peered":"2023-05-22T16:03:38.821360+0000","last_clean":"2023-05-22T16:03:38.821360+0000","last_became_active":"2023-05-22T16:03:38.813914+0000","last_became_peered":"2023-05-22T16:03:38.813914+0000","last_unstale":"2023-05-22T16:03:38.821360+0000","last_undegraded":"2023-05-22T16:03:38.821360+0000","last_fullsized":"2023-05-22T16:03:38.821360+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_clean_scrub_stamp":"2023-05-22T16:03:38.387135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:08:59.616966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.7","version":"178'2","reported_seq":12,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.832107+0000","last_change":"2023-05-22T16:03:38.451344+0000","last_active":"2023-05-22T16:03:38.832107+0000","last_peered":"2023-05-22T16:03:38.832107+0000","last_clean":"2023-05-22T16:03:38.832107+0000","last_became_active":"2023-05-22T16:03:38.451136+0000","last_became_peered":"2023-05-22T16:03:38.451136+0000","last_unstale":"2023-05-22T16:03:38.832107+0000","last_undegraded":"2023-05-22T16:03:38.832107+0000","last_fullsized":"2023-05-22T16:03:38.832107+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:03:37.383765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:03:37.383765+0000","last_clean_scrub_stamp":"2023-05-22T16:03:37.383765+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:20:28.887546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.7","version":"0'0","reported_seq":9,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.823748+0000","last_change":"2023-05-22T16:03:38.822821+0000","last_active":"2023-05-22T16:03:38.823748+0000","last_peered":"2023-05-22T16:03:38.823748+0000","last_clean":"2023-05-22T16:03:38.823748+0000","last_became_active":"2023-05-22T16:03:38.822541+0000","last_became_peered":"2023-05-22T16:03:38.822541+0000","last_unstale":"2023-05-22T16:03:38.823748+0000","last_undegraded":"2023-05-22T16:03:38.823748+0000","last_fullsized":"2023-05-22T16:03:38.823748+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_clean_scrub_stamp":"2023-05-22T16:03:38.387135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:02:50.415413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.6","version":"0'0","reported_seq":10,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.803747+0000","last_change":"2023-05-22T16:03:38.433322+0000","last_active":"2023-05-22T16:03:38.803747+0000","last_peered":"2023-05-22T16:03:38.803747+0000","last_clean":"2023-05-22T16:03:38.803747+0000","last_became_active":"2023-05-22T16:03:38.432663+0000","last_became_peered":"2023-05-22T16:03:38.432663+0000","last_unstale":"2023-05-22T16:03:38.803747+0000","last_undegraded":"2023-05-22T16:03:38.803747+0000","last_fullsized":"2023-05-22T16:03:38.803747+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:03:37.383765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:03:37.383765+0000","last_clean_scrub_stamp":"2023-05-22T16:03:37.383765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:32:39.325570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.4","version":"0'0","reported_seq":9,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.812851+0000","last_change":"2023-05-22T16:03:38.800848+0000","last_active":"2023-05-22T16:03:38.812851+0000","last_peered":"2023-05-22T16:03:38.812851+0000","last_clean":"2023-05-22T16:03:38.812851+0000","last_became_active":"2023-05-22T16:03:38.800037+0000","last_became_peered":"2023-05-22T16:03:38.800037+0000","last_unstale":"2023-05-22T16:03:38.812851+0000","last_undegraded":"2023-05-22T16:03:38.812851+0000","last_fullsized":"2023-05-22T16:03:38.812851+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_clean_scrub_stamp":"2023-05-22T16:03:38.387135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:36:22.242179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.5","version":"0'0","reported_seq":10,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.803738+0000","last_change":"2023-05-22T16:03:38.441974+0000","last_active":"2023-05-22T16:03:38.803738+0000","last_peered":"2023-05-22T16:03:38.803738+0000","last_clean":"2023-05-22T16:03:38.803738+0000","last_became_active":"2023-05-22T16:03:38.441132+0000","last_became_peered":"2023-05-22T16:03:38.441132+0000","last_unstale":"2023-05-22T16:03:38.803738+0000","last_undegraded":"2023-05-22T16:03:38.803738+0000","last_fullsized":"2023-05-22T16:03:38.803738+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:03:37.383765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:03:37.383765+0000","last_clean_scrub_stamp":"2023-05-22T16:03:37.383765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:49:44.109377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.5","version":"0'0","reported_seq":9,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.821456+0000","last_change":"2023-05-22T16:03:38.818770+0000","last_active":"2023-05-22T16:03:38.821456+0000","last_peered":"2023-05-22T16:03:38.821456+0000","last_clean":"2023-05-22T16:03:38.821456+0000","last_became_active":"2023-05-22T16:03:38.818427+0000","last_became_peered":"2023-05-22T16:03:38.818427+0000","last_unstale":"2023-05-22T16:03:38.821456+0000","last_undegraded":"2023-05-22T16:03:38.821456+0000","last_fullsized":"2023-05-22T16:03:38.821456+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_clean_scrub_stamp":"2023-05-22T16:03:38.387135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T03:37:21.656379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.4","version":"0'0","reported_seq":10,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.794878+0000","last_change":"2023-05-22T16:03:38.454342+0000","last_active":"2023-05-22T16:03:38.794878+0000","last_peered":"2023-05-22T16:03:38.794878+0000","last_clean":"2023-05-22T16:03:38.794878+0000","last_became_active":"2023-05-22T16:03:38.453961+0000","last_became_peered":"2023-05-22T16:03:38.453961+0000","last_unstale":"2023-05-22T16:03:38.794878+0000","last_undegraded":"2023-05-22T16:03:38.794878+0000","last_fullsized":"2023-05-22T16:03:38.794878+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:03:37.383765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:03:37.383765+0000","last_clean_scrub_stamp":"2023-05-22T16:03:37.383765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:44:19.423716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.2","version":"0'0","reported_seq":9,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.809003+0000","last_change":"2023-05-22T16:03:38.804374+0000","last_active":"2023-05-22T16:03:38.809003+0000","last_peered":"2023-05-22T16:03:38.809003+0000","last_clean":"2023-05-22T16:03:38.809003+0000","last_became_active":"2023-05-22T16:03:38.803971+0000","last_became_peered":"2023-05-22T16:03:38.803971+0000","last_unstale":"2023-05-22T16:03:38.809003+0000","last_undegraded":"2023-05-22T16:03:38.809003+0000","last_fullsized":"2023-05-22T16:03:38.809003+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_clean_scrub_stamp":"2023-05-22T16:03:38.387135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:53:38.976658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.3","version":"0'0","reported_seq":10,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.793613+0000","last_change":"2023-05-22T16:03:38.432703+0000","last_active":"2023-05-22T16:03:38.793613+0000","last_peered":"2023-05-22T16:03:38.793613+0000","last_clean":"2023-05-22T16:03:38.793613+0000","last_became_active":"2023-05-22T16:03:38.431828+0000","last_became_peered":"2023-05-22T16:03:38.431828+0000","last_unstale":"2023-05-22T16:03:38.793613+0000","last_undegraded":"2023-05-22T16:03:38.793613+0000","last_fullsized":"2023-05-22T16:03:38.793613+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:03:37.383765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:03:37.383765+0000","last_clean_scrub_stamp":"2023-05-22T16:03:37.383765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:22:35.883553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.3","version":"0'0","reported_seq":9,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.836119+0000","last_change":"2023-05-22T16:03:38.818400+0000","last_active":"2023-05-22T16:03:38.836119+0000","last_peered":"2023-05-22T16:03:38.836119+0000","last_clean":"2023-05-22T16:03:38.836119+0000","last_became_active":"2023-05-22T16:03:38.817915+0000","last_became_peered":"2023-05-22T16:03:38.817915+0000","last_unstale":"2023-05-22T16:03:38.836119+0000","last_undegraded":"2023-05-22T16:03:38.836119+0000","last_fullsized":"2023-05-22T16:03:38.836119+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_clean_scrub_stamp":"2023-05-22T16:03:38.387135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T18:39:51.525114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.2","version":"0'0","reported_seq":10,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.793747+0000","last_change":"2023-05-22T16:03:38.442188+0000","last_active":"2023-05-22T16:03:38.793747+0000","last_peered":"2023-05-22T16:03:38.793747+0000","last_clean":"2023-05-22T16:03:38.793747+0000","last_became_active":"2023-05-22T16:03:38.441873+0000","last_became_peered":"2023-05-22T16:03:38.441873+0000","last_unstale":"2023-05-22T16:03:38.793747+0000","last_undegraded":"2023-05-22T16:03:38.793747+0000","last_fullsized":"2023-05-22T16:03:38.793747+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:03:37.383765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:03:37.383765+0000","last_clean_scrub_stamp":"2023-05-22T16:03:37.383765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:57:02.653723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.0","version":"0'0","reported_seq":9,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.820779+0000","last_change":"2023-05-22T16:03:38.819579+0000","last_active":"2023-05-22T16:03:38.820779+0000","last_peered":"2023-05-22T16:03:38.820779+0000","last_clean":"2023-05-22T16:03:38.820779+0000","last_became_active":"2023-05-22T16:03:38.819241+0000","last_became_peered":"2023-05-22T16:03:38.819241+0000","last_unstale":"2023-05-22T16:03:38.820779+0000","last_undegraded":"2023-05-22T16:03:38.820779+0000","last_fullsized":"2023-05-22T16:03:38.820779+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_clean_scrub_stamp":"2023-05-22T16:03:38.387135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:07:09.650623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.1","version":"0'0","reported_seq":10,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.804786+0000","last_change":"2023-05-22T16:03:38.443276+0000","last_active":"2023-05-22T16:03:38.804786+0000","last_peered":"2023-05-22T16:03:38.804786+0000","last_clean":"2023-05-22T16:03:38.804786+0000","last_became_active":"2023-05-22T16:03:38.433576+0000","last_became_peered":"2023-05-22T16:03:38.433576+0000","last_unstale":"2023-05-22T16:03:38.804786+0000","last_undegraded":"2023-05-22T16:03:38.804786+0000","last_fullsized":"2023-05-22T16:03:38.804786+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:03:37.383765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:03:37.383765+0000","last_clean_scrub_stamp":"2023-05-22T16:03:37.383765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:46:57.319877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.1","version":"0'0","reported_seq":9,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.833912+0000","last_change":"2023-05-22T16:03:38.819249+0000","last_active":"2023-05-22T16:03:38.833912+0000","last_peered":"2023-05-22T16:03:38.833912+0000","last_clean":"2023-05-22T16:03:38.833912+0000","last_became_active":"2023-05-22T16:03:38.818956+0000","last_became_peered":"2023-05-22T16:03:38.818956+0000","last_unstale":"2023-05-22T16:03:38.833912+0000","last_undegraded":"2023-05-22T16:03:38.833912+0000","last_fullsized":"2023-05-22T16:03:38.833912+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_clean_scrub_stamp":"2023-05-22T16:03:38.387135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:59:32.502376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.0","version":"0'0","reported_seq":10,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.793886+0000","last_change":"2023-05-22T16:03:38.430698+0000","last_active":"2023-05-22T16:03:38.793886+0000","last_peered":"2023-05-22T16:03:38.793886+0000","last_clean":"2023-05-22T16:03:38.793886+0000","last_became_active":"2023-05-22T16:03:38.429895+0000","last_became_peered":"2023-05-22T16:03:38.429895+0000","last_unstale":"2023-05-22T16:03:38.793886+0000","last_undegraded":"2023-05-22T16:03:38.793886+0000","last_fullsized":"2023-05-22T16:03:38.793886+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:03:37.383765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:03:37.383765+0000","last_clean_scrub_stamp":"2023-05-22T16:03:37.383765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:16:36.548828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.16","version":"0'0","reported_seq":9,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.821384+0000","last_change":"2023-05-22T16:03:38.814445+0000","last_active":"2023-05-22T16:03:38.821384+0000","last_peered":"2023-05-22T16:03:38.821384+0000","last_clean":"2023-05-22T16:03:38.821384+0000","last_became_active":"2023-05-22T16:03:38.813749+0000","last_became_peered":"2023-05-22T16:03:38.813749+0000","last_unstale":"2023-05-22T16:03:38.821384+0000","last_undegraded":"2023-05-22T16:03:38.821384+0000","last_fullsized":"2023-05-22T16:03:38.821384+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_clean_scrub_stamp":"2023-05-22T16:03:38.387135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:45:10.755716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.17","version":"0'0","reported_seq":10,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.793248+0000","last_change":"2023-05-22T16:03:38.431425+0000","last_active":"2023-05-22T16:03:38.793248+0000","last_peered":"2023-05-22T16:03:38.793248+0000","last_clean":"2023-05-22T16:03:38.793248+0000","last_became_active":"2023-05-22T16:03:38.430685+0000","last_became_peered":"2023-05-22T16:03:38.430685+0000","last_unstale":"2023-05-22T16:03:38.793248+0000","last_undegraded":"2023-05-22T16:03:38.793248+0000","last_fullsized":"2023-05-22T16:03:38.793248+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:03:37.383765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:03:37.383765+0000","last_clean_scrub_stamp":"2023-05-22T16:03:37.383765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T03:39:03.580030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"166'304","reported_seq":985,"reported_epoch":177,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.406538+0000","last_change":"2023-05-22T15:26:39.318204+0000","last_active":"2023-05-22T16:03:38.406538+0000","last_peered":"2023-05-22T16:03:38.406538+0000","last_clean":"2023-05-22T16:03:38.406538+0000","last_became_active":"2023-05-22T15:26:39.315370+0000","last_became_peered":"2023-05-22T15:26:39.315370+0000","last_unstale":"2023-05-22T16:03:38.406538+0000","last_undegraded":"2023-05-22T16:03:38.406538+0000","last_fullsized":"2023-05-22T16:03:38.406538+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:38.020076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:38.020076+0000","last_clean_scrub_stamp":"2023-05-22T15:26:38.020076+0000","objects_scrubbed":0,"log_size":304,"log_dups_size":0,"ondisk_log_size":304,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:06:39.207543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1748,"num_read_kb":3602,"num_write":499,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":178,"reported_epoch":177,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.409433+0000","last_change":"2023-05-22T15:26:44.971951+0000","last_active":"2023-05-22T16:03:38.409433+0000","last_peered":"2023-05-22T16:03:38.409433+0000","last_clean":"2023-05-22T16:03:38.409433+0000","last_became_active":"2023-05-22T15:26:44.970872+0000","last_became_peered":"2023-05-22T15:26:44.970872+0000","last_unstale":"2023-05-22T16:03:38.409433+0000","last_undegraded":"2023-05-22T16:03:38.409433+0000","last_fullsized":"2023-05-22T16:03:38.409433+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:55:34.752949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.15","version":"0'0","reported_seq":9,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.837225+0000","last_change":"2023-05-22T16:03:38.825114+0000","last_active":"2023-05-22T16:03:38.837225+0000","last_peered":"2023-05-22T16:03:38.837225+0000","last_clean":"2023-05-22T16:03:38.837225+0000","last_became_active":"2023-05-22T16:03:38.824362+0000","last_became_peered":"2023-05-22T16:03:38.824362+0000","last_unstale":"2023-05-22T16:03:38.837225+0000","last_undegraded":"2023-05-22T16:03:38.837225+0000","last_fullsized":"2023-05-22T16:03:38.837225+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_clean_scrub_stamp":"2023-05-22T16:03:38.387135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:30:18.570472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.14","version":"178'6","reported_seq":16,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.841501+0000","last_change":"2023-05-22T16:03:38.434806+0000","last_active":"2023-05-22T16:03:38.841501+0000","last_peered":"2023-05-22T16:03:38.841501+0000","last_clean":"2023-05-22T16:03:38.841501+0000","last_became_active":"2023-05-22T16:03:38.434146+0000","last_became_peered":"2023-05-22T16:03:38.434146+0000","last_unstale":"2023-05-22T16:03:38.841501+0000","last_undegraded":"2023-05-22T16:03:38.841501+0000","last_fullsized":"2023-05-22T16:03:38.841501+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:03:37.383765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:03:37.383765+0000","last_clean_scrub_stamp":"2023-05-22T16:03:37.383765+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:40:13.538937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1548,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":169,"reported_epoch":177,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.406328+0000","last_change":"2023-05-22T15:26:44.977198+0000","last_active":"2023-05-22T16:03:38.406328+0000","last_peered":"2023-05-22T16:03:38.406328+0000","last_clean":"2023-05-22T16:03:38.406328+0000","last_became_active":"2023-05-22T15:26:44.976445+0000","last_became_peered":"2023-05-22T15:26:44.976445+0000","last_unstale":"2023-05-22T16:03:38.406328+0000","last_undegraded":"2023-05-22T16:03:38.406328+0000","last_fullsized":"2023-05-22T16:03:38.406328+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:12:41.464054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.14","version":"0'0","reported_seq":9,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.821447+0000","last_change":"2023-05-22T16:03:38.814616+0000","last_active":"2023-05-22T16:03:38.821447+0000","last_peered":"2023-05-22T16:03:38.821447+0000","last_clean":"2023-05-22T16:03:38.821447+0000","last_became_active":"2023-05-22T16:03:38.814239+0000","last_became_peered":"2023-05-22T16:03:38.814239+0000","last_unstale":"2023-05-22T16:03:38.821447+0000","last_undegraded":"2023-05-22T16:03:38.821447+0000","last_fullsized":"2023-05-22T16:03:38.821447+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_clean_scrub_stamp":"2023-05-22T16:03:38.387135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:56:21.206906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.15","version":"178'3","reported_seq":13,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.827868+0000","last_change":"2023-05-22T16:03:38.422653+0000","last_active":"2023-05-22T16:03:38.827868+0000","last_peered":"2023-05-22T16:03:38.827868+0000","last_clean":"2023-05-22T16:03:38.827868+0000","last_became_active":"2023-05-22T16:03:38.422328+0000","last_became_peered":"2023-05-22T16:03:38.422328+0000","last_unstale":"2023-05-22T16:03:38.827868+0000","last_undegraded":"2023-05-22T16:03:38.827868+0000","last_fullsized":"2023-05-22T16:03:38.827868+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:03:37.383765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:03:37.383765+0000","last_clean_scrub_stamp":"2023-05-22T16:03:37.383765+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:13:15.091000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":171,"reported_epoch":177,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.397500+0000","last_change":"2023-05-22T15:26:44.969469+0000","last_active":"2023-05-22T16:03:38.397500+0000","last_peered":"2023-05-22T16:03:38.397500+0000","last_clean":"2023-05-22T16:03:38.397500+0000","last_became_active":"2023-05-22T15:26:44.969083+0000","last_became_peered":"2023-05-22T15:26:44.969083+0000","last_unstale":"2023-05-22T16:03:38.397500+0000","last_undegraded":"2023-05-22T16:03:38.397500+0000","last_fullsized":"2023-05-22T16:03:38.397500+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:29:23.670067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.17","version":"0'0","reported_seq":9,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.821348+0000","last_change":"2023-05-22T16:03:38.814628+0000","last_active":"2023-05-22T16:03:38.821348+0000","last_peered":"2023-05-22T16:03:38.821348+0000","last_clean":"2023-05-22T16:03:38.821348+0000","last_became_active":"2023-05-22T16:03:38.813818+0000","last_became_peered":"2023-05-22T16:03:38.813818+0000","last_unstale":"2023-05-22T16:03:38.821348+0000","last_undegraded":"2023-05-22T16:03:38.821348+0000","last_fullsized":"2023-05-22T16:03:38.821348+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_clean_scrub_stamp":"2023-05-22T16:03:38.387135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:30:18.570472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.16","version":"0'0","reported_seq":10,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.787619+0000","last_change":"2023-05-22T16:03:38.433642+0000","last_active":"2023-05-22T16:03:38.787619+0000","last_peered":"2023-05-22T16:03:38.787619+0000","last_clean":"2023-05-22T16:03:38.787619+0000","last_became_active":"2023-05-22T16:03:38.433310+0000","last_became_peered":"2023-05-22T16:03:38.433310+0000","last_unstale":"2023-05-22T16:03:38.787619+0000","last_undegraded":"2023-05-22T16:03:38.787619+0000","last_fullsized":"2023-05-22T16:03:38.787619+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:03:37.383765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:03:37.383765+0000","last_clean_scrub_stamp":"2023-05-22T16:03:37.383765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:19:25.192874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":178,"reported_epoch":177,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.411957+0000","last_change":"2023-05-22T15:26:44.976632+0000","last_active":"2023-05-22T16:03:38.411957+0000","last_peered":"2023-05-22T16:03:38.411957+0000","last_clean":"2023-05-22T16:03:38.411957+0000","last_became_active":"2023-05-22T15:26:44.975546+0000","last_became_peered":"2023-05-22T15:26:44.975546+0000","last_unstale":"2023-05-22T16:03:38.411957+0000","last_undegraded":"2023-05-22T16:03:38.411957+0000","last_fullsized":"2023-05-22T16:03:38.411957+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:42:57.973030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.11","version":"0'0","reported_seq":9,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.837268+0000","last_change":"2023-05-22T16:03:38.827159+0000","last_active":"2023-05-22T16:03:38.837268+0000","last_peered":"2023-05-22T16:03:38.837268+0000","last_clean":"2023-05-22T16:03:38.837268+0000","last_became_active":"2023-05-22T16:03:38.826600+0000","last_became_peered":"2023-05-22T16:03:38.826600+0000","last_unstale":"2023-05-22T16:03:38.837268+0000","last_undegraded":"2023-05-22T16:03:38.837268+0000","last_fullsized":"2023-05-22T16:03:38.837268+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_clean_scrub_stamp":"2023-05-22T16:03:38.387135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T03:47:46.900041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.10","version":"178'8","reported_seq":18,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.838027+0000","last_change":"2023-05-22T16:03:38.432666+0000","last_active":"2023-05-22T16:03:38.838027+0000","last_peered":"2023-05-22T16:03:38.838027+0000","last_clean":"2023-05-22T16:03:38.838027+0000","last_became_active":"2023-05-22T16:03:38.431969+0000","last_became_peered":"2023-05-22T16:03:38.431969+0000","last_unstale":"2023-05-22T16:03:38.838027+0000","last_undegraded":"2023-05-22T16:03:38.838027+0000","last_fullsized":"2023-05-22T16:03:38.838027+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:03:37.383765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:03:37.383765+0000","last_clean_scrub_stamp":"2023-05-22T16:03:37.383765+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:00:45.061713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":159,"reported_epoch":177,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.421536+0000","last_change":"2023-05-22T15:26:45.091475+0000","last_active":"2023-05-22T16:03:38.421536+0000","last_peered":"2023-05-22T16:03:38.421536+0000","last_clean":"2023-05-22T16:03:38.421536+0000","last_became_active":"2023-05-22T15:26:45.091143+0000","last_became_peered":"2023-05-22T15:26:45.091143+0000","last_unstale":"2023-05-22T16:03:38.421536+0000","last_undegraded":"2023-05-22T16:03:38.421536+0000","last_fullsized":"2023-05-22T16:03:38.421536+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:12:41.464054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.10","version":"0'0","reported_seq":9,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.813416+0000","last_change":"2023-05-22T16:03:38.810097+0000","last_active":"2023-05-22T16:03:38.813416+0000","last_peered":"2023-05-22T16:03:38.813416+0000","last_clean":"2023-05-22T16:03:38.813416+0000","last_became_active":"2023-05-22T16:03:38.809662+0000","last_became_peered":"2023-05-22T16:03:38.809662+0000","last_unstale":"2023-05-22T16:03:38.813416+0000","last_undegraded":"2023-05-22T16:03:38.813416+0000","last_fullsized":"2023-05-22T16:03:38.813416+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_clean_scrub_stamp":"2023-05-22T16:03:38.387135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:46:34.382160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.11","version":"0'0","reported_seq":10,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.793236+0000","last_change":"2023-05-22T16:03:38.434216+0000","last_active":"2023-05-22T16:03:38.793236+0000","last_peered":"2023-05-22T16:03:38.793236+0000","last_clean":"2023-05-22T16:03:38.793236+0000","last_became_active":"2023-05-22T16:03:38.433896+0000","last_became_peered":"2023-05-22T16:03:38.433896+0000","last_unstale":"2023-05-22T16:03:38.793236+0000","last_undegraded":"2023-05-22T16:03:38.793236+0000","last_fullsized":"2023-05-22T16:03:38.793236+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:03:37.383765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:03:37.383765+0000","last_clean_scrub_stamp":"2023-05-22T16:03:37.383765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T16:32:32.165063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":168,"reported_epoch":177,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.408860+0000","last_change":"2023-05-22T15:26:45.092367+0000","last_active":"2023-05-22T16:03:38.408860+0000","last_peered":"2023-05-22T16:03:38.408860+0000","last_clean":"2023-05-22T16:03:38.408860+0000","last_became_active":"2023-05-22T15:26:45.091951+0000","last_became_peered":"2023-05-22T15:26:45.091951+0000","last_unstale":"2023-05-22T16:03:38.408860+0000","last_undegraded":"2023-05-22T16:03:38.408860+0000","last_fullsized":"2023-05-22T16:03:38.408860+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:42:57.973030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.13","version":"0'0","reported_seq":9,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.819120+0000","last_change":"2023-05-22T16:03:38.817592+0000","last_active":"2023-05-22T16:03:38.819120+0000","last_peered":"2023-05-22T16:03:38.819120+0000","last_clean":"2023-05-22T16:03:38.819120+0000","last_became_active":"2023-05-22T16:03:38.816466+0000","last_became_peered":"2023-05-22T16:03:38.816466+0000","last_unstale":"2023-05-22T16:03:38.819120+0000","last_undegraded":"2023-05-22T16:03:38.819120+0000","last_fullsized":"2023-05-22T16:03:38.819120+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_clean_scrub_stamp":"2023-05-22T16:03:38.387135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T03:37:21.656379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.12","version":"0'0","reported_seq":10,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.793487+0000","last_change":"2023-05-22T16:03:38.426071+0000","last_active":"2023-05-22T16:03:38.793487+0000","last_peered":"2023-05-22T16:03:38.793487+0000","last_clean":"2023-05-22T16:03:38.793487+0000","last_became_active":"2023-05-22T16:03:38.425620+0000","last_became_peered":"2023-05-22T16:03:38.425620+0000","last_unstale":"2023-05-22T16:03:38.793487+0000","last_undegraded":"2023-05-22T16:03:38.793487+0000","last_fullsized":"2023-05-22T16:03:38.793487+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:03:37.383765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:03:37.383765+0000","last_clean_scrub_stamp":"2023-05-22T16:03:37.383765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:22:35.883553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":159,"reported_epoch":177,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.421501+0000","last_change":"2023-05-22T15:26:44.978858+0000","last_active":"2023-05-22T16:03:38.421501+0000","last_peered":"2023-05-22T16:03:38.421501+0000","last_clean":"2023-05-22T16:03:38.421501+0000","last_became_active":"2023-05-22T15:26:44.978443+0000","last_became_peered":"2023-05-22T15:26:44.978443+0000","last_unstale":"2023-05-22T16:03:38.421501+0000","last_undegraded":"2023-05-22T16:03:38.421501+0000","last_fullsized":"2023-05-22T16:03:38.421501+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:46:29.290243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.12","version":"0'0","reported_seq":9,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.830554+0000","last_change":"2023-05-22T16:03:38.829555+0000","last_active":"2023-05-22T16:03:38.830554+0000","last_peered":"2023-05-22T16:03:38.830554+0000","last_clean":"2023-05-22T16:03:38.830554+0000","last_became_active":"2023-05-22T16:03:38.829218+0000","last_became_peered":"2023-05-22T16:03:38.829218+0000","last_unstale":"2023-05-22T16:03:38.830554+0000","last_undegraded":"2023-05-22T16:03:38.830554+0000","last_fullsized":"2023-05-22T16:03:38.830554+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:03:38.387135+0000","last_clean_scrub_stamp":"2023-05-22T16:03:38.387135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:07:09.650623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.13","version":"178'2","reported_seq":12,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.821817+0000","last_change":"2023-05-22T16:03:38.427651+0000","last_active":"2023-05-22T16:03:38.821817+0000","last_peered":"2023-05-22T16:03:38.821817+0000","last_clean":"2023-05-22T16:03:38.821817+0000","last_became_active":"2023-05-22T16:03:38.427358+0000","last_became_peered":"2023-05-22T16:03:38.427358+0000","last_unstale":"2023-05-22T16:03:38.821817+0000","last_undegraded":"2023-05-22T16:03:38.821817+0000","last_fullsized":"2023-05-22T16:03:38.821817+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:03:37.383765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:03:37.383765+0000","last_clean_scrub_stamp":"2023-05-22T16:03:37.383765+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:59:31.499006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":158,"reported_epoch":177,"state":"active+clean","last_fresh":"2023-05-22T16:03:38.441489+0000","last_change":"2023-05-22T15:26:44.982421+0000","last_active":"2023-05-22T16:03:38.441489+0000","last_peered":"2023-05-22T16:03:38.441489+0000","last_clean":"2023-05-22T16:03:38.441489+0000","last_became_active":"2023-05-22T15:26:44.981586+0000","last_became_peered":"2023-05-22T15:26:44.981586+0000","last_unstale":"2023-05-22T16:03:38.441489+0000","last_undegraded":"2023-05-22T16:03:38.441489+0000","last_fullsized":"2023-05-22T16:03:38.441489+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:14:11.156544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":23,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":22,"num_pg":32,"stat_sum":{"num_bytes":2330,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1748,"num_read_kb":3602,"num_write":499,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":304,"ondisk_log_size":304,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575288,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27936,"kb_used_data":2776,"kb_used_omap":0,"kb_used_meta":25152,"kb_avail":94343904,"statfs":{"total":96636764160,"available":96608157696,"internally_reserved":0,"allocated":2842624,"data_stored":1779013,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":25755648},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 16:02:42 2023","interfaces":[{"interface":"back","average":{"1min":0.82599999999999996,"5min":0.85199999999999998,"15min":0.86599999999999999},"min":{"1min":0.61499999999999999,"5min":0.52800000000000002,"15min":0.51100000000000001},"max":{"1min":1.194,"5min":1.2569999999999999,"15min":1.385},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.91500000000000004,"5min":0.89000000000000001,"15min":0.88800000000000001},"min":{"1min":0.629,"5min":0.501,"15min":0.501},"max":{"1min":1.244,"5min":1.2529999999999999,"15min":1.2529999999999999},"last":0.94699999999999995}]},{"osd":1,"last update":"Mon May 22 16:02:42 2023","interfaces":[{"interface":"back","average":{"1min":0.97499999999999998,"5min":0.998,"15min":0.99299999999999999},"min":{"1min":0.63400000000000001,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.1890000000000001,"5min":1.3260000000000001,"15min":1.361},"last":1.089},{"interface":"front","average":{"1min":1.085,"5min":1.0469999999999999,"15min":1.046},"min":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.626},"max":{"1min":1.3029999999999999,"5min":1.3380000000000001,"15min":1.4019999999999999},"last":1.028}]},{"osd":2,"last update":"Mon May 22 16:02:42 2023","interfaces":[{"interface":"back","average":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.90200000000000002},"min":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"max":{"1min":1.1739999999999999,"5min":1.286,"15min":1.407},"last":0.98899999999999999},{"interface":"front","average":{"1min":0.97499999999999998,"5min":0.99199999999999999,"15min":0.97099999999999997},"min":{"1min":0.68400000000000005,"5min":0.65400000000000003,"15min":0.65400000000000003},"max":{"1min":1.214,"5min":1.2829999999999999,"15min":1.2829999999999999},"last":1.179}]},{"osd":3,"last update":"Mon May 22 16:02:42 2023","interfaces":[{"interface":"back","average":{"1min":0.94599999999999995,"5min":0.96499999999999997,"15min":0.95799999999999996},"min":{"1min":0.73999999999999999,"5min":0.65700000000000003,"15min":0.62},"max":{"1min":1.167,"5min":1.401,"15min":1.401},"last":0.88900000000000001},{"interface":"front","average":{"1min":0.996,"5min":1.002,"15min":0.98999999999999999},"min":{"1min":0.77000000000000002,"5min":0.66300000000000003,"15min":0.66300000000000003},"max":{"1min":1.3460000000000001,"5min":1.411,"15min":1.411},"last":0.83399999999999996}]},{"osd":4,"last update":"Mon May 22 16:02:42 2023","interfaces":[{"interface":"back","average":{"1min":0.84299999999999997,"5min":0.89200000000000002,"15min":0.88700000000000001},"min":{"1min":0.63,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":1.163,"5min":1.3440000000000001,"15min":1.3779999999999999},"last":0.77900000000000003},{"interface":"front","average":{"1min":1.012,"5min":0.95999999999999996,"15min":0.95999999999999996},"min":{"1min":0.67900000000000005,"5min":0.57999999999999996,"15min":0.55400000000000005},"max":{"1min":1.3300000000000001,"5min":1.3300000000000001,"15min":1.4419999999999999},"last":1.125}]},{"osd":5,"last update":"Mon May 22 16:02:42 2023","interfaces":[{"interface":"back","average":{"1min":1.024,"5min":0.98599999999999999,"15min":0.996},"min":{"1min":0.65600000000000003,"5min":0.621,"15min":0.441},"max":{"1min":1.2909999999999999,"5min":1.3759999999999999,"15min":1.3819999999999999},"last":1.1539999999999999},{"interface":"front","average":{"1min":1.1339999999999999,"5min":1.1439999999999999,"15min":1.1379999999999999},"min":{"1min":0.83199999999999996,"5min":0.66600000000000004,"15min":0.59799999999999998},"max":{"1min":1.298,"5min":1.4079999999999999,"15min":1.5429999999999999},"last":1.056}]},{"osd":6,"last update":"Mon May 22 16:02:42 2023","interfaces":[{"interface":"back","average":{"1min":1.052,"5min":1.0369999999999999,"15min":1.0589999999999999},"min":{"1min":0.72299999999999998,"5min":0.65800000000000003,"15min":0.65800000000000003},"max":{"1min":1.3169999999999999,"5min":1.3480000000000001,"15min":1.4470000000000001},"last":1.2490000000000001},{"interface":"front","average":{"1min":1.1140000000000001,"5min":1.089,"15min":1.101},"min":{"1min":0.70199999999999996,"5min":0.67600000000000005,"15min":0.61499999999999999},"max":{"1min":1.391,"5min":1.4550000000000001,"15min":1.7050000000000001},"last":0.69699999999999995}]}]},{"osd":6,"up_from":13,"seq":55834575289,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":22024,"kb_used_data":2176,"kb_used_omap":0,"kb_used_meta":19840,"kb_avail":94349816,"statfs":{"total":96636764160,"available":96614211584,"internally_reserved":0,"allocated":2228224,"data_stored":1186393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":20316160},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 16:02:53 2023","interfaces":[{"interface":"back","average":{"1min":0.88100000000000001,"5min":0.89800000000000002,"15min":0.91000000000000003},"min":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"max":{"1min":1.2250000000000001,"5min":1.2250000000000001,"15min":1.482},"last":0.69999999999999996},{"interface":"front","average":{"1min":0.89900000000000002,"5min":0.89100000000000001,"15min":0.91500000000000004},"min":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.57599999999999996},"max":{"1min":1.397,"5min":1.397,"15min":1.397},"last":0.55900000000000005}]},{"osd":1,"last update":"Mon May 22 16:02:53 2023","interfaces":[{"interface":"back","average":{"1min":0.88500000000000001,"5min":0.91400000000000003,"15min":0.94299999999999995},"min":{"1min":0.66200000000000003,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.1839999999999999,"5min":1.3380000000000001,"15min":1.395},"last":0.875},{"interface":"front","average":{"1min":0.94599999999999995,"5min":0.93600000000000005,"15min":0.94299999999999995},"min":{"1min":0.66400000000000003,"5min":0.65100000000000002,"15min":0.63500000000000001},"max":{"1min":1.2849999999999999,"5min":1.2849999999999999,"15min":1.357},"last":0.66000000000000003}]},{"osd":2,"last update":"Mon May 22 16:02:53 2023","interfaces":[{"interface":"back","average":{"1min":0.99299999999999999,"5min":0.99299999999999999,"15min":1.0029999999999999},"min":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":1.3580000000000001,"5min":1.3580000000000001,"15min":1.3580000000000001},"last":0.99299999999999999},{"interface":"front","average":{"1min":1.0649999999999999,"5min":1.091,"15min":1.085},"min":{"1min":0.77900000000000003,"5min":0.76900000000000002,"15min":0.63200000000000001},"max":{"1min":1.2649999999999999,"5min":1.359,"15min":1.6579999999999999},"last":0.96099999999999997}]},{"osd":3,"last update":"Mon May 22 16:02:53 2023","interfaces":[{"interface":"back","average":{"1min":0.997,"5min":1.01,"15min":0.996},"min":{"1min":0.69599999999999995,"5min":0.65500000000000003,"15min":0.63700000000000001},"max":{"1min":1.169,"5min":1.327,"15min":1.4339999999999999},"last":0.75900000000000001},{"interface":"front","average":{"1min":0.97999999999999998,"5min":0.96699999999999997,"15min":0.97599999999999998},"min":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"max":{"1min":1.1899999999999999,"5min":1.2350000000000001,"15min":1.3779999999999999},"last":0.80500000000000005}]},{"osd":4,"last update":"Mon May 22 16:02:53 2023","interfaces":[{"interface":"back","average":{"1min":0.99299999999999999,"5min":0.92500000000000004,"15min":0.95599999999999996},"min":{"1min":0.62,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":1.333,"5min":1.359,"15min":1.5720000000000001},"last":1.0609999999999999},{"interface":"front","average":{"1min":1.0740000000000001,"5min":1.0469999999999999,"15min":1.0349999999999999},"min":{"1min":0.64700000000000002,"5min":0.623,"15min":0.622},"max":{"1min":1.3049999999999999,"5min":1.4770000000000001,"15min":1.52},"last":0.84699999999999998}]},{"osd":5,"last update":"Mon May 22 16:02:53 2023","interfaces":[{"interface":"back","average":{"1min":1.024,"5min":0.999,"15min":0.98699999999999999},"min":{"1min":0.71399999999999997,"5min":0.66900000000000004,"15min":0.61799999999999999},"max":{"1min":1.3100000000000001,"5min":1.375,"15min":1.504},"last":0.89900000000000002},{"interface":"front","average":{"1min":1.095,"5min":1.1080000000000001,"15min":1.129},"min":{"1min":0.748,"5min":0.69799999999999995,"15min":0.69699999999999995},"max":{"1min":1.325,"5min":1.4430000000000001,"15min":1.74},"last":1.1140000000000001}]},{"osd":7,"last update":"Mon May 22 16:02:53 2023","interfaces":[{"interface":"back","average":{"1min":1.161,"5min":1.1419999999999999,"15min":1.1339999999999999},"min":{"1min":0.92700000000000005,"5min":0.64200000000000002,"15min":0.58599999999999997},"max":{"1min":1.359,"5min":1.546,"15min":1.603},"last":0.93000000000000005},{"interface":"front","average":{"1min":1.1599999999999999,"5min":1.131,"15min":1.139},"min":{"1min":0.81899999999999995,"5min":0.755,"15min":0.746},"max":{"1min":1.4450000000000001,"5min":1.669,"15min":1.669},"last":1.0860000000000001}]}]},{"osd":5,"up_from":13,"seq":55834575289,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31884,"kb_used_data":2180,"kb_used_omap":0,"kb_used_meta":29696,"kb_avail":94339956,"statfs":{"total":96636764160,"available":96604114944,"internally_reserved":0,"allocated":2232320,"data_stored":1186959,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":30408704},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 16:02:47 2023","interfaces":[{"interface":"back","average":{"1min":0.90400000000000003,"5min":0.92000000000000004,"15min":0.89800000000000002},"min":{"1min":0.63500000000000001,"5min":0.61899999999999999,"15min":0.57199999999999995},"max":{"1min":1.1850000000000001,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.99299999999999999},{"interface":"front","average":{"1min":0.93999999999999995,"5min":0.92400000000000004,"15min":0.94499999999999995},"min":{"1min":0.623,"5min":0.502,"15min":0.502},"max":{"1min":1.325,"5min":1.325,"15min":1.329},"last":1.042}]},{"osd":1,"last update":"Mon May 22 16:02:47 2023","interfaces":[{"interface":"back","average":{"1min":1.002,"5min":1.0109999999999999,"15min":1.022},"min":{"1min":0.754,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":1.3280000000000001,"5min":1.365,"15min":1.365},"last":1.0740000000000001},{"interface":"front","average":{"1min":1.1000000000000001,"5min":1.083,"15min":1.081},"min":{"1min":0.85399999999999998,"5min":0.80500000000000005,"15min":0.61599999999999999},"max":{"1min":1.3500000000000001,"5min":1.4630000000000001,"15min":1.4630000000000001},"last":1.2110000000000001}]},{"osd":2,"last update":"Mon May 22 16:02:47 2023","interfaces":[{"interface":"back","average":{"1min":0.96399999999999997,"5min":0.89200000000000002,"15min":0.92700000000000005},"min":{"1min":0.68100000000000005,"5min":0.63700000000000001,"15min":0.63600000000000001},"max":{"1min":1.2,"5min":1.3200000000000001,"15min":1.413},"last":1.1299999999999999},{"interface":"front","average":{"1min":0.98199999999999998,"5min":0.96599999999999997,"15min":0.95199999999999996},"min":{"1min":0.71599999999999997,"5min":0.65800000000000003,"15min":0.65800000000000003},"max":{"1min":1.379,"5min":1.379,"15min":1.379},"last":0.88300000000000001}]},{"osd":3,"last update":"Mon May 22 16:02:47 2023","interfaces":[{"interface":"back","average":{"1min":0.97499999999999998,"5min":0.96799999999999997,"15min":0.97899999999999998},"min":{"1min":0.79100000000000004,"5min":0.71099999999999997,"15min":0.60799999999999998},"max":{"1min":1.296,"5min":1.367,"15min":1.367},"last":1.0960000000000001},{"interface":"front","average":{"1min":1.0129999999999999,"5min":1.056,"15min":1.0600000000000001},"min":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"max":{"1min":1.2410000000000001,"5min":1.371,"15min":1.387},"last":0.93100000000000005}]},{"osd":4,"last update":"Mon May 22 16:02:47 2023","interfaces":[{"interface":"back","average":{"1min":0.94099999999999995,"5min":0.92800000000000005,"15min":0.92100000000000004},"min":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"max":{"1min":1.282,"5min":1.4079999999999999,"15min":1.5109999999999999},"last":1.018},{"interface":"front","average":{"1min":0.99399999999999999,"5min":1.0109999999999999,"15min":1.0249999999999999},"min":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.58399999999999996},"max":{"1min":1.4630000000000001,"5min":1.4630000000000001,"15min":1.502},"last":0.83399999999999996}]},{"osd":6,"last update":"Mon May 22 16:02:47 2023","interfaces":[{"interface":"back","average":{"1min":1.0229999999999999,"5min":1.0109999999999999,"15min":1.0269999999999999},"min":{"1min":0.67100000000000004,"5min":0.64400000000000002,"15min":0.59799999999999998},"max":{"1min":1.389,"5min":1.389,"15min":1.4790000000000001},"last":0.77400000000000002},{"interface":"front","average":{"1min":1.0600000000000001,"5min":1.077,"15min":1.0860000000000001},"min":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"max":{"1min":1.3819999999999999,"5min":1.413,"15min":1.512},"last":0.95799999999999996}]},{"osd":7,"last update":"Mon May 22 16:02:47 2023","interfaces":[{"interface":"back","average":{"1min":1.1220000000000001,"5min":1.1359999999999999,"15min":1.1339999999999999},"min":{"1min":0.73199999999999998,"5min":0.65800000000000003,"15min":0.65800000000000003},"max":{"1min":1.359,"5min":1.5109999999999999,"15min":1.8360000000000001},"last":1.169},{"interface":"front","average":{"1min":1.1799999999999999,"5min":1.1879999999999999,"15min":1.194},"min":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.72799999999999998},"max":{"1min":1.5349999999999999,"5min":1.575,"15min":1.7210000000000001},"last":1.1799999999999999}]}]},{"osd":4,"up_from":13,"seq":55834575287,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33876,"kb_used_data":2188,"kb_used_omap":0,"kb_used_meta":31680,"kb_avail":94337964,"statfs":{"total":96636764160,"available":96602075136,"internally_reserved":0,"allocated":2240512,"data_stored":1187989,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":32440320},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 16:02:56 2023","interfaces":[{"interface":"back","average":{"1min":0.94299999999999995,"5min":0.91700000000000004,"15min":0.90700000000000003},"min":{"1min":0.71299999999999997,"5min":0.53100000000000003,"15min":0.499},"max":{"1min":1.262,"5min":1.7210000000000001,"15min":1.7210000000000001},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.98799999999999999,"5min":0.96399999999999997,"15min":0.95299999999999996},"min":{"1min":0.73699999999999999,"5min":0.621,"15min":0.58999999999999997},"max":{"1min":1.2749999999999999,"5min":1.4730000000000001,"15min":1.4730000000000001},"last":0.93600000000000005}]},{"osd":1,"last update":"Mon May 22 16:02:56 2023","interfaces":[{"interface":"back","average":{"1min":0.94899999999999995,"5min":0.96499999999999997,"15min":0.96099999999999997},"min":{"1min":0.72199999999999998,"5min":0.65400000000000003,"15min":0.63400000000000001},"max":{"1min":1.173,"5min":1.3999999999999999,"15min":1.3999999999999999},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.96299999999999997,"5min":0.95499999999999996,"15min":0.94899999999999995},"min":{"1min":0.70099999999999996,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":1.2210000000000001,"5min":1.306,"15min":1.306},"last":0.58299999999999996}]},{"osd":2,"last update":"Mon May 22 16:02:56 2023","interfaces":[{"interface":"back","average":{"1min":0.93000000000000005,"5min":0.93600000000000005,"15min":0.94699999999999995},"min":{"1min":0.70599999999999996,"5min":0.65900000000000003,"15min":0.59199999999999997},"max":{"1min":1.383,"5min":1.383,"15min":1.383},"last":0.73699999999999999},{"interface":"front","average":{"1min":1.0600000000000001,"5min":1.0489999999999999,"15min":1.0529999999999999},"min":{"1min":0.68700000000000006,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.3899999999999999,"5min":1.492,"15min":1.492},"last":1.0649999999999999}]},{"osd":3,"last update":"Mon May 22 16:02:56 2023","interfaces":[{"interface":"back","average":{"1min":0.92300000000000004,"5min":0.94799999999999995,"15min":0.95299999999999996},"min":{"1min":0.72799999999999998,"5min":0.67300000000000004,"15min":0.65300000000000002},"max":{"1min":1.3740000000000001,"5min":1.4610000000000001,"15min":1.4610000000000001},"last":0.77700000000000002},{"interface":"front","average":{"1min":1.0269999999999999,"5min":1.075,"15min":1.0529999999999999},"min":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.69099999999999995},"max":{"1min":1.2549999999999999,"5min":1.484,"15min":1.484},"last":0.84599999999999997}]},{"osd":5,"last update":"Mon May 22 16:02:56 2023","interfaces":[{"interface":"back","average":{"1min":0.97999999999999998,"5min":1.0189999999999999,"15min":0.999},"min":{"1min":0.67200000000000004,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":1.2949999999999999,"5min":1.417,"15min":1.417},"last":1.139},{"interface":"front","average":{"1min":0.96099999999999997,"5min":0.99399999999999999,"15min":0.99199999999999999},"min":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"max":{"1min":1.2490000000000001,"5min":1.5449999999999999,"15min":1.5449999999999999},"last":0.89700000000000002}]},{"osd":6,"last update":"Mon May 22 16:02:56 2023","interfaces":[{"interface":"back","average":{"1min":0.95999999999999996,"5min":0.98299999999999998,"15min":0.99199999999999999},"min":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":1.224,"5min":1.397,"15min":1.4339999999999999},"last":1.008},{"interface":"front","average":{"1min":1.071,"5min":1.1319999999999999,"15min":1.125},"min":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"max":{"1min":1.456,"5min":1.5600000000000001,"15min":1.5600000000000001},"last":1.1879999999999999}]},{"osd":7,"last update":"Mon May 22 16:02:56 2023","interfaces":[{"interface":"back","average":{"1min":1.0669999999999999,"5min":1.0700000000000001,"15min":1.0669999999999999},"min":{"1min":0.76700000000000002,"5min":0.71399999999999997,"15min":0.71399999999999997},"max":{"1min":1.3280000000000001,"5min":1.5920000000000001,"15min":1.5920000000000001},"last":1.042},{"interface":"front","average":{"1min":1.169,"5min":1.1599999999999999,"15min":1.155},"min":{"1min":0.95999999999999996,"5min":0.68100000000000005,"15min":0.68100000000000005},"max":{"1min":1.3500000000000001,"5min":1.448,"15min":1.5089999999999999},"last":1.1000000000000001}]}]},{"osd":3,"up_from":13,"seq":55834575287,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":26568,"kb_used_data":2176,"kb_used_omap":0,"kb_used_meta":24384,"kb_avail":94345272,"statfs":{"total":96636764160,"available":96609558528,"internally_reserved":0,"allocated":2228224,"data_stored":1186405,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":24969216},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 16:03:38 2023","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":0.63600000000000001,"15min":0.64000000000000001},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":1.0049999999999999,"5min":1.0049999999999999,"15min":1.048},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.76500000000000001,"15min":0.76900000000000002},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":1.1539999999999999,"5min":1.2789999999999999,"15min":1.2789999999999999},"last":0.63300000000000001}]},{"osd":1,"last update":"Mon May 22 16:03:38 2023","interfaces":[{"interface":"back","average":{"1min":0.82499999999999996,"5min":0.74099999999999999,"15min":0.71399999999999997},"min":{"1min":0.55100000000000005,"5min":0.52700000000000002,"15min":0.51600000000000001},"max":{"1min":2.4969999999999999,"5min":2.4969999999999999,"15min":2.4969999999999999},"last":0.89700000000000002},{"interface":"front","average":{"1min":0.89400000000000002,"5min":0.80300000000000005,"15min":0.80600000000000005},"min":{"1min":0.61399999999999999,"5min":0.55800000000000005,"15min":0.54400000000000004},"max":{"1min":2.4769999999999999,"5min":2.4769999999999999,"15min":2.4769999999999999},"last":0.47899999999999998}]},{"osd":2,"last update":"Mon May 22 16:03:38 2023","interfaces":[{"interface":"back","average":{"1min":0.84499999999999997,"5min":0.84399999999999997,"15min":0.85399999999999998},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":1.1759999999999999,"5min":1.1759999999999999,"15min":1.1759999999999999},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.873,"5min":0.81699999999999995,"15min":0.83599999999999997},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":1.1259999999999999,"5min":1.2270000000000001,"15min":1.2270000000000001},"last":0.66100000000000003}]},{"osd":4,"last update":"Mon May 22 16:03:38 2023","interfaces":[{"interface":"back","average":{"1min":0.95299999999999996,"5min":0.96299999999999997,"15min":0.97599999999999998},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":1.343,"5min":1.343,"15min":1.343},"last":0.995},{"interface":"front","average":{"1min":1.002,"5min":1.012,"15min":1.036},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":1.3180000000000001,"5min":1.4119999999999999,"15min":1.49},"last":0.81299999999999994}]},{"osd":5,"last update":"Mon May 22 16:03:38 2023","interfaces":[{"interface":"back","average":{"1min":0.996,"5min":1.056,"15min":1.0580000000000001},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":1.3520000000000001,"5min":1.5149999999999999,"15min":1.5149999999999999},"last":0.68300000000000005},{"interface":"front","average":{"1min":1.0489999999999999,"5min":1.054,"15min":1.0569999999999999},"min":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":1.571,"5min":1.571,"15min":1.571},"last":0.78100000000000003}]},{"osd":6,"last update":"Mon May 22 16:03:38 2023","interfaces":[{"interface":"back","average":{"1min":1.0620000000000001,"5min":1.0900000000000001,"15min":1.099},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":1.446,"5min":1.6739999999999999,"15min":1.6739999999999999},"last":0.749},{"interface":"front","average":{"1min":1.1220000000000001,"5min":1.107,"15min":1.1080000000000001},"min":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":1.754,"5min":1.754,"15min":1.754},"last":0.93899999999999995}]},{"osd":7,"last update":"Mon May 22 16:03:38 2023","interfaces":[{"interface":"back","average":{"1min":1.139,"5min":1.1220000000000001,"15min":1.137},"min":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.673,"5min":1.673,"15min":1.673},"last":0.84399999999999997},{"interface":"front","average":{"1min":1.129,"5min":1.105,"15min":1.1120000000000001},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":1.5369999999999999,"5min":1.5369999999999999,"15min":1.54},"last":1.1140000000000001}]}]},{"osd":2,"up_from":13,"seq":55834575288,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":26504,"kb_used_data":2176,"kb_used_omap":0,"kb_used_meta":24320,"kb_avail":94345336,"statfs":{"total":96636764160,"available":96609624064,"internally_reserved":0,"allocated":2228224,"data_stored":1186461,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":24903680},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 16:03:29 2023","interfaces":[{"interface":"back","average":{"1min":0.69499999999999995,"5min":0.65600000000000003,"15min":0.67800000000000005},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":1.0549999999999999,"5min":1.1180000000000001,"15min":1.1519999999999999},"last":0.438},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.74199999999999999,"15min":0.754},"min":{"1min":0.17999999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":1.038,"5min":1.1319999999999999,"15min":1.2090000000000001},"last":0.51300000000000001}]},{"osd":1,"last update":"Mon May 22 16:03:29 2023","interfaces":[{"interface":"back","average":{"1min":0.77300000000000002,"5min":0.76500000000000001,"15min":0.76400000000000001},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":1.071,"5min":1.0800000000000001,"15min":1.099},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.81499999999999995,"5min":0.83799999999999997,"15min":0.82499999999999996},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":1.073,"5min":1.1739999999999999,"15min":1.2769999999999999},"last":0.39200000000000002}]},{"osd":3,"last update":"Mon May 22 16:03:29 2023","interfaces":[{"interface":"back","average":{"1min":0.77900000000000003,"5min":0.77700000000000002,"15min":0.76200000000000001},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":1.3540000000000001,"5min":1.3540000000000001,"15min":1.3540000000000001},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.85399999999999998,"5min":0.877,"15min":0.90200000000000002},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":1.2,"5min":1.2,"15min":1.2949999999999999},"last":0.61599999999999999}]},{"osd":4,"last update":"Mon May 22 16:03:29 2023","interfaces":[{"interface":"back","average":{"1min":1.018,"5min":1.0329999999999999,"15min":1.0429999999999999},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":1.2709999999999999,"5min":1.6539999999999999,"15min":1.6539999999999999},"last":0.82599999999999996},{"interface":"front","average":{"1min":0.95899999999999996,"5min":1.0289999999999999,"15min":1.0449999999999999},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.216,"5min":1.472,"15min":1.538},"last":1.014}]},{"osd":5,"last update":"Mon May 22 16:03:29 2023","interfaces":[{"interface":"back","average":{"1min":0.995,"5min":1.0529999999999999,"15min":1.0609999999999999},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":1.3200000000000001,"5min":1.5640000000000001,"15min":1.605},"last":0.75700000000000001},{"interface":"front","average":{"1min":1.0449999999999999,"5min":1.016,"15min":1.036},"min":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"max":{"1min":1.4650000000000001,"5min":1.4870000000000001,"15min":1.5109999999999999},"last":0.91100000000000003}]},{"osd":6,"last update":"Mon May 22 16:03:29 2023","interfaces":[{"interface":"back","average":{"1min":1.034,"5min":1.0409999999999999,"15min":1.028},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":1.4019999999999999,"5min":1.429,"15min":1.462},"last":0.83699999999999997},{"interface":"front","average":{"1min":1.1020000000000001,"5min":1.1180000000000001,"15min":1.137},"min":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"max":{"1min":1.389,"5min":1.7110000000000001,"15min":1.7110000000000001},"last":0.79200000000000004}]},{"osd":7,"last update":"Mon May 22 16:03:29 2023","interfaces":[{"interface":"back","average":{"1min":1.0369999999999999,"5min":1.1000000000000001,"15min":1.1279999999999999},"min":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":1.3660000000000001,"5min":1.6399999999999999,"15min":1.6399999999999999},"last":0.80800000000000005},{"interface":"front","average":{"1min":1.071,"5min":1.113,"15min":1.1200000000000001},"min":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":1.4330000000000001,"5min":1.673,"15min":1.673},"last":0.95699999999999996}]}]},{"osd":1,"up_from":13,"seq":55834575288,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36488,"kb_used_data":2176,"kb_used_omap":0,"kb_used_meta":34304,"kb_avail":94335352,"statfs":{"total":96636764160,"available":96599400448,"internally_reserved":0,"allocated":2228224,"data_stored":1186405,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":35127296},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 16:02:50 2023","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.66000000000000003,"15min":0.66300000000000003},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.88800000000000001,"5min":1.151,"15min":1.329},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.71799999999999997,"5min":0.75800000000000001,"15min":0.77600000000000002},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":0.95599999999999996,"5min":1.2350000000000001,"15min":1.268},"last":0.85899999999999999}]},{"osd":2,"last update":"Mon May 22 16:02:50 2023","interfaces":[{"interface":"back","average":{"1min":0.746,"5min":0.76500000000000001,"15min":0.77200000000000002},"min":{"1min":0.57299999999999995,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.98499999999999999,"5min":1.0880000000000001,"15min":1.26},"last":0.73899999999999999},{"interface":"front","average":{"1min":0.80200000000000005,"5min":0.78800000000000003,"15min":0.78600000000000003},"min":{"1min":0.56999999999999995,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.127,"5min":1.175,"15min":1.268},"last":0.77700000000000002}]},{"osd":3,"last update":"Mon May 22 16:02:50 2023","interfaces":[{"interface":"back","average":{"1min":0.88700000000000001,"5min":0.878,"15min":0.89000000000000001},"min":{"1min":0.621,"5min":0.61199999999999999,"15min":0.55900000000000005},"max":{"1min":1.2829999999999999,"5min":1.2829999999999999,"15min":1.373},"last":0.996},{"interface":"front","average":{"1min":0.88400000000000001,"5min":0.83699999999999997,"15min":0.83499999999999996},"min":{"1min":0.63600000000000001,"5min":0.57199999999999995,"15min":0.57199999999999995},"max":{"1min":1.2749999999999999,"5min":1.2749999999999999,"15min":1.2749999999999999},"last":0.63200000000000001}]},{"osd":4,"last update":"Mon May 22 16:02:50 2023","interfaces":[{"interface":"back","average":{"1min":0.98999999999999999,"5min":1.002,"15min":0.99399999999999999},"min":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"max":{"1min":1.2769999999999999,"5min":1.542,"15min":1.542},"last":1.089},{"interface":"front","average":{"1min":1.056,"5min":1.042,"15min":1.0389999999999999},"min":{"1min":0.85699999999999998,"5min":0.79500000000000004,"15min":0.73499999999999999},"max":{"1min":1.327,"5min":1.4350000000000001,"15min":1.4650000000000001},"last":1.111}]},{"osd":5,"last update":"Mon May 22 16:02:50 2023","interfaces":[{"interface":"back","average":{"1min":1.075,"5min":1.0600000000000001,"15min":1.0609999999999999},"min":{"1min":0.88,"5min":0.78100000000000003,"15min":0.76800000000000002},"max":{"1min":1.236,"5min":1.605,"15min":1.605},"last":1.046},{"interface":"front","average":{"1min":1.0309999999999999,"5min":1.095,"15min":1.085},"min":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"max":{"1min":1.365,"5min":1.411,"15min":1.5389999999999999},"last":0.95699999999999996}]},{"osd":6,"last update":"Mon May 22 16:02:50 2023","interfaces":[{"interface":"back","average":{"1min":1.0700000000000001,"5min":1.0720000000000001,"15min":1.075},"min":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.73899999999999999},"max":{"1min":1.371,"5min":1.371,"15min":1.5249999999999999},"last":1.22},{"interface":"front","average":{"1min":1.105,"5min":1.111,"15min":1.123},"min":{"1min":0.85199999999999998,"5min":0.84599999999999997,"15min":0.75700000000000001},"max":{"1min":1.329,"5min":1.4630000000000001,"15min":1.6479999999999999},"last":1.27}]},{"osd":7,"last update":"Mon May 22 16:02:50 2023","interfaces":[{"interface":"back","average":{"1min":1.151,"5min":1.167,"15min":1.171},"min":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"max":{"1min":1.403,"5min":1.7030000000000001,"15min":1.7030000000000001},"last":1.1739999999999999},{"interface":"front","average":{"1min":1.125,"5min":1.155,"15min":1.157},"min":{"1min":0.85199999999999998,"5min":0.81100000000000005,"15min":0.81100000000000005},"max":{"1min":1.45,"5min":1.6439999999999999,"15min":1.6439999999999999},"last":1.1519999999999999}]}]},{"osd":0,"up_from":13,"seq":55834575289,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31880,"kb_used_data":2752,"kb_used_omap":0,"kb_used_meta":29120,"kb_avail":94339960,"statfs":{"total":96636764160,"available":96604119040,"internally_reserved":0,"allocated":2818048,"data_stored":1776739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":29818880},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon May 22 16:02:55 2023","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.71499999999999997,"15min":0.70999999999999996},"min":{"1min":0.53500000000000003,"5min":0.501,"15min":0.47999999999999998},"max":{"1min":1.296,"5min":1.296,"15min":1.296},"last":0.82299999999999995},{"interface":"front","average":{"1min":0.78800000000000003,"5min":0.75600000000000001,"15min":0.747},"min":{"1min":0.54700000000000004,"5min":0.496,"15min":0.47599999999999998},"max":{"1min":1.181,"5min":1.286,"15min":1.3149999999999999},"last":1.0229999999999999}]},{"osd":2,"last update":"Mon May 22 16:02:55 2023","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":0.77200000000000002,"15min":0.78100000000000003},"min":{"1min":0.63500000000000001,"5min":0.56699999999999995,"15min":0.52400000000000002},"max":{"1min":1.1419999999999999,"5min":1.1839999999999999,"15min":1.284},"last":0.97899999999999998},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.84499999999999997,"15min":0.82799999999999996},"min":{"1min":0.61299999999999999,"5min":0.56200000000000006,"15min":0.54200000000000004},"max":{"1min":1.038,"5min":1.2170000000000001,"15min":1.2270000000000001},"last":0.73799999999999999}]},{"osd":3,"last update":"Mon May 22 16:02:55 2023","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":0.79000000000000004,"15min":0.77600000000000002},"min":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.51900000000000002},"max":{"1min":0.995,"5min":1.2350000000000001,"15min":1.2350000000000001},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.91900000000000004,"5min":0.89700000000000002,"15min":0.89200000000000002},"min":{"1min":0.70599999999999996,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":1.2949999999999999,"5min":1.2949999999999999,"15min":1.331},"last":1.198}]},{"osd":4,"last update":"Mon May 22 16:02:55 2023","interfaces":[{"interface":"back","average":{"1min":0.98099999999999998,"5min":0.96899999999999997,"15min":0.95999999999999996},"min":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.67300000000000004},"max":{"1min":1.1719999999999999,"5min":1.4059999999999999,"15min":1.4059999999999999},"last":0.78900000000000003},{"interface":"front","average":{"1min":1.089,"5min":1.0720000000000001,"15min":1.0600000000000001},"min":{"1min":0.89300000000000002,"5min":0.83099999999999996,"15min":0.74299999999999999},"max":{"1min":1.3839999999999999,"5min":1.3839999999999999,"15min":1.482},"last":1.401}]},{"osd":5,"last update":"Mon May 22 16:02:55 2023","interfaces":[{"interface":"back","average":{"1min":1.044,"5min":1.0429999999999999,"15min":1.0489999999999999},"min":{"1min":0.84799999999999998,"5min":0.77200000000000002,"15min":0.65300000000000002},"max":{"1min":1.2729999999999999,"5min":1.361,"15min":1.6419999999999999},"last":1.264},{"interface":"front","average":{"1min":0.995,"5min":1.032,"15min":1.018},"min":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.72299999999999998},"max":{"1min":1.27,"5min":1.458,"15min":1.458},"last":0.93500000000000005}]},{"osd":6,"last update":"Mon May 22 16:02:55 2023","interfaces":[{"interface":"back","average":{"1min":1.119,"5min":1.1180000000000001,"15min":1.1160000000000001},"min":{"1min":0.89700000000000002,"5min":0.80100000000000005,"15min":0.79900000000000004},"max":{"1min":1.3660000000000001,"5min":1.4359999999999999,"15min":1.77},"last":1.3120000000000001},{"interface":"front","average":{"1min":1.105,"5min":1.131,"15min":1.1080000000000001},"min":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.747},"max":{"1min":1.256,"5min":1.4990000000000001,"15min":1.53},"last":1.45}]},{"osd":7,"last update":"Mon May 22 16:02:55 2023","interfaces":[{"interface":"back","average":{"1min":1.1659999999999999,"5min":1.165,"15min":1.1379999999999999},"min":{"1min":0.86399999999999999,"5min":0.86399999999999999,"15min":0.76000000000000001},"max":{"1min":1.355,"5min":1.532,"15min":1.5960000000000001},"last":1.1499999999999999},{"interface":"front","average":{"1min":1.1200000000000001,"5min":1.1259999999999999,"15min":1.111},"min":{"1min":0.94999999999999996,"5min":0.83999999999999997,"15min":0.79000000000000004},"max":{"1min":1.403,"5min":1.4650000000000001,"15min":1.4990000000000001},"last":1.0589999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2274,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-05-22T16:03:43.562 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-05-22T16:03:43.564 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2023-05-22T16:03:43.729 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:43.730+0000 7f5d90650640 1 Processor -- start 2023-05-22T16:03:43.730 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:43.730+0000 7f5d90650640 1 -- start start 2023-05-22T16:03:43.731 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:43.730+0000 7f5d90650640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f5d88159970 0x7f5d88159d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:43.731 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:43.730+0000 7f5d90650640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f5d8815db00 0x7f5d8806b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:43.731 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:43.730+0000 7f5d90650640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d88070c30 con 0x7f5d8815d680 2023-05-22T16:03:43.731 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:43.730+0000 7f5d90650640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d880cb970 con 0x7f5d88159970 2023-05-22T16:03:43.731 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:43.730+0000 7f5d90650640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d88070e60 con 0x7f5d8815db00 2023-05-22T16:03:43.732 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:43.730+0000 7f5d8e3ec640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f5d88159970 0x7f5d88159d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:43.732 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:43.730+0000 7f5d8e3ec640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f5d88159970 0x7f5d88159d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49016/0 (socket says 172.21.15.16:49016) 2023-05-22T16:03:43.732 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:43.730+0000 7f5d8e3ec640 1 -- 172.21.15.16:0/3895655622 learned_addr learned my addr 172.21.15.16:0/3895655622 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:03:43.732 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:43.730+0000 7f5d8ebed640 1 --2- 172.21.15.16:0/3895655622 >> v2:172.21.15.62:3300/0 conn(0x7f5d8815db00 0x7f5d8806b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:43.732 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:43.730+0000 7f5d8e3ec640 1 -- 172.21.15.16:0/3895655622 >> v2:172.21.15.62:3300/0 conn(0x7f5d8815db00 msgr2=0x7f5d8806b6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:03:43.733 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:43.730+0000 7f5d8e3ec640 1 --2- 172.21.15.16:0/3895655622 >> v2:172.21.15.62:3300/0 conn(0x7f5d8815db00 0x7f5d8806b6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:43.733 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:43.730+0000 7f5d8e3ec640 1 -- 172.21.15.16:0/3895655622 >> v1:172.21.15.62:6789/0 conn(0x7f5d8815d680 legacy=0x7f5d880cb260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:43.733 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:43.730+0000 7f5d8e3ec640 1 -- 172.21.15.16:0/3895655622 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d88071140 con 0x7f5d88159970 2023-05-22T16:03:43.733 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:43.734+0000 7f5d8e3ec640 1 --2- 172.21.15.16:0/3895655622 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5d88159970 0x7f5d88159d40 secure :-1 s=READY pgs=2262 cs=0 l=1 rev1=1 crypto rx=0x7f5d88070770 tx=0x7f5d7801c440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=854eae5b4bf94b96 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:03:43.734 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:43.734+0000 7f5d8d3ea640 1 -- 172.21.15.16:0/3895655622 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d78004230 con 0x7f5d88159970 2023-05-22T16:03:43.734 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:43.734+0000 7f5d8d3ea640 1 -- 172.21.15.16:0/3895655622 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d780a7050 con 0x7f5d88159970 2023-05-22T16:03:43.734 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:43.734+0000 7f5d8d3ea640 1 -- 172.21.15.16:0/3895655622 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d78003620 con 0x7f5d88159970 2023-05-22T16:03:43.735 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:43.734+0000 7f5d90650640 1 -- 172.21.15.16:0/3895655622 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5d88159970 msgr2=0x7f5d88159d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:43.735 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:43.734+0000 7f5d90650640 1 --2- 172.21.15.16:0/3895655622 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5d88159970 0x7f5d88159d40 secure :-1 s=READY pgs=2262 cs=0 l=1 rev1=1 crypto rx=0x7f5d88070770 tx=0x7f5d7801c440 comp rx=0 tx=0).stop 2023-05-22T16:03:43.735 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:43.734+0000 7f5d90650640 1 -- 172.21.15.16:0/3895655622 shutdown_connections 2023-05-22T16:03:43.735 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:43.734+0000 7f5d90650640 1 --2- 172.21.15.16:0/3895655622 >> v2:172.21.15.62:3300/0 conn(0x7f5d8815db00 0x7f5d8806b6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:43.735 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:43.734+0000 7f5d90650640 1 --2- 172.21.15.16:0/3895655622 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5d88159970 0x7f5d88159d40 unknown :-1 s=CLOSED pgs=2262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:43.735 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:43.734+0000 7f5d90650640 1 -- 172.21.15.16:0/3895655622 >> 172.21.15.16:0/3895655622 conn(0x7f5d88069720 msgr2=0x7f5d88069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:03:43.735 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:43.734+0000 7f5d90650640 1 -- 172.21.15.16:0/3895655622 shutdown_connections 2023-05-22T16:03:43.736 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:43.734+0000 7f5d90650640 1 -- 172.21.15.16:0/3895655622 wait complete. 2023-05-22T16:03:43.737 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:43.734+0000 7f5d90650640 1 Processor -- start 2023-05-22T16:03:43.737 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:43.734+0000 7f5d90650640 1 -- start start 2023-05-22T16:03:43.737 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:43.738+0000 7f5d90650640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5d88159970 0x7f5d88155cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:43.738 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:43.738+0000 7f5d90650640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5d8815d680 0x7f5d880c7880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:43.738 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:43.738+0000 7f5d90650640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5d8815db00 0x7f5d880c7dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:43.738 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:43.738+0000 7f5d90650640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d880c9030 con 0x7f5d88159970 2023-05-22T16:03:43.738 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:43.738+0000 7f5d90650640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d88156230 con 0x7f5d8815d680 2023-05-22T16:03:43.738 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:43.738+0000 7f5d90650640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d880c83b0 con 0x7f5d8815db00 2023-05-22T16:03:43.738 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:43.738+0000 7f5d8e3ec640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5d88159970 0x7f5d88155cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:43.738 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:43.738+0000 7f5d8e3ec640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5d88159970 0x7f5d88155cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49028/0 (socket says 172.21.15.16:49028) 2023-05-22T16:03:43.738 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:43.738+0000 7f5d8e3ec640 1 -- 172.21.15.16:0/762225071 learned_addr learned my addr 172.21.15.16:0/762225071 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:03:43.739 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:43.738+0000 7f5d8dbeb640 1 --2- 172.21.15.16:0/762225071 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5d8815d680 0x7f5d880c7880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:43.739 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:43.738+0000 7f5d8ebed640 1 --2- 172.21.15.16:0/762225071 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5d8815db00 0x7f5d880c7dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:43.739 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:43.738+0000 7f5d8e3ec640 1 -- 172.21.15.16:0/762225071 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5d8815db00 msgr2=0x7f5d880c7dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:43.739 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:43.738+0000 7f5d8e3ec640 1 --2- 172.21.15.16:0/762225071 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5d8815db00 0x7f5d880c7dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:43.739 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:43.738+0000 7f5d8e3ec640 1 -- 172.21.15.16:0/762225071 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5d8815d680 msgr2=0x7f5d880c7880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:43.739 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:43.738+0000 7f5d8e3ec640 1 --2- 172.21.15.16:0/762225071 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5d8815d680 0x7f5d880c7880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:43.739 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:43.738+0000 7f5d8e3ec640 1 -- 172.21.15.16:0/762225071 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d780094d0 con 0x7f5d88159970 2023-05-22T16:03:43.740 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:43.738+0000 7f5d8dbeb640 1 --2- 172.21.15.16:0/762225071 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5d8815d680 0x7f5d880c7880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T16:03:43.740 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:43.738+0000 7f5d8e3ec640 1 --2- 172.21.15.16:0/762225071 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5d88159970 0x7f5d88155cf0 secure :-1 s=READY pgs=2263 cs=0 l=1 rev1=1 crypto rx=0x7f5d780ae040 tx=0x7f5d780191e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:03:43.741 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:43.738+0000 7f5d86ffd640 1 -- 172.21.15.16:0/762225071 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d78003c70 con 0x7f5d88159970 2023-05-22T16:03:43.741 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:43.738+0000 7f5d86ffd640 1 -- 172.21.15.16:0/762225071 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d78003dd0 con 0x7f5d88159970 2023-05-22T16:03:43.741 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:43.738+0000 7f5d90650640 1 -- 172.21.15.16:0/762225071 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d880cae20 con 0x7f5d88159970 2023-05-22T16:03:43.741 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:43.738+0000 7f5d86ffd640 1 -- 172.21.15.16:0/762225071 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d780b6680 con 0x7f5d88159970 2023-05-22T16:03:43.741 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:43.738+0000 7f5d90650640 1 -- 172.21.15.16:0/762225071 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d880cac90 con 0x7f5d88159970 2023-05-22T16:03:43.741 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:43.738+0000 7f5d90650640 1 -- 172.21.15.16:0/762225071 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d8815adc0 con 0x7f5d88159970 2023-05-22T16:03:43.742 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:43.738+0000 7f5d86ffd640 1 -- 172.21.15.16:0/762225071 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 82) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f5d780b2040 con 0x7f5d88159970 2023-05-22T16:03:43.742 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:43.738+0000 7f5d86ffd640 1 --2- 172.21.15.16:0/762225071 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f5d740654a0 0x7f5d74067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:43.742 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:43.738+0000 7f5d8dbeb640 1 --2- 172.21.15.16:0/762225071 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f5d740654a0 0x7f5d74067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:43.742 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:43.738+0000 7f5d86ffd640 1 -- 172.21.15.16:0/762225071 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(179..179 src has 1..179) v4 ==== 6438+0+0 (secure 0 0 0) 0x7f5d7811cad0 con 0x7f5d88159970 2023-05-22T16:03:43.742 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:43.738+0000 7f5d8dbeb640 1 --2- 172.21.15.16:0/762225071 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f5d740654a0 0x7f5d74067950 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f5d880ae9c0 tx=0x7f5d7c009040 comp rx=0 tx=0).ready entity=mgr.10032 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:03:43.744 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:43.742+0000 7f5d86ffd640 1 -- 172.21.15.16:0/762225071 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f5d780ea550 con 0x7f5d88159970 2023-05-22T16:03:43.953 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:43.954+0000 7f5d90650640 1 -- 172.21.15.16:0/762225071 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f5d8804dbe0 con 0x7f5d88159970 2023-05-22T16:03:43.963 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:43.962+0000 7f5d86ffd640 1 -- 172.21.15.16:0/762225071 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v33) v1 ==== 190+0+0 (secure 0 0 0) 0x7f5d780f30e0 con 0x7f5d88159970 2023-05-22T16:03:43.963 INFO:teuthology.orchestra.run.smithi016.stderr:updated caps for client.0 2023-05-22T16:03:43.966 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:43.966+0000 7f5d90650640 1 -- 172.21.15.16:0/762225071 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f5d740654a0 msgr2=0x7f5d74067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:43.966 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:43.966+0000 7f5d90650640 1 --2- 172.21.15.16:0/762225071 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f5d740654a0 0x7f5d74067950 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f5d880ae9c0 tx=0x7f5d7c009040 comp rx=0 tx=0).stop 2023-05-22T16:03:43.966 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:43.966+0000 7f5d90650640 1 -- 172.21.15.16:0/762225071 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5d88159970 msgr2=0x7f5d88155cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:43.967 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:43.966+0000 7f5d90650640 1 --2- 172.21.15.16:0/762225071 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5d88159970 0x7f5d88155cf0 secure :-1 s=READY pgs=2263 cs=0 l=1 rev1=1 crypto rx=0x7f5d780ae040 tx=0x7f5d780191e0 comp rx=0 tx=0).stop 2023-05-22T16:03:43.967 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:43.966+0000 7f5d90650640 1 -- 172.21.15.16:0/762225071 shutdown_connections 2023-05-22T16:03:43.967 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:43.966+0000 7f5d90650640 1 --2- 172.21.15.16:0/762225071 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5d8815db00 0x7f5d880c7dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:43.967 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:43.966+0000 7f5d90650640 1 --2- 172.21.15.16:0/762225071 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5d8815d680 0x7f5d880c7880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:43.968 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:43.966+0000 7f5d90650640 1 --2- 172.21.15.16:0/762225071 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f5d740654a0 0x7f5d74067950 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:43.969 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:43.966+0000 7f5d90650640 1 --2- 172.21.15.16:0/762225071 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5d88159970 0x7f5d88155cf0 unknown :-1 s=CLOSED pgs=2263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:43.969 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:43.966+0000 7f5d90650640 1 -- 172.21.15.16:0/762225071 >> 172.21.15.16:0/762225071 conn(0x7f5d88069720 msgr2=0x7f5d880cb1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:03:43.969 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:43.966+0000 7f5d90650640 1 -- 172.21.15.16:0/762225071 shutdown_connections 2023-05-22T16:03:43.969 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:43.966+0000 7f5d90650640 1 -- 172.21.15.16:0/762225071 wait complete. 2023-05-22T16:03:43.983 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-05-22T16:03:44.109 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.110+0000 7fda97d1c640 1 Processor -- start 2023-05-22T16:03:44.109 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.110+0000 7fda97d1c640 1 -- start start 2023-05-22T16:03:44.110 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.110+0000 7fda97d1c640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fda900b0250 0x7fda90159330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:44.110 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.110+0000 7fda97d1c640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda9015a0a0 con 0x7fda901531b0 2023-05-22T16:03:44.110 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.110+0000 7fda97d1c640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda90153580 con 0x7fda90159870 2023-05-22T16:03:44.110 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.110+0000 7fda97d1c640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda9015d5e0 con 0x7fda900b0250 2023-05-22T16:03:44.110 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.110+0000 7fda952b7640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fda900b0250 0x7fda90159330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:44.110 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.110+0000 7fda95ab8640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fda901531b0 0x7fda900afb40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:53220/0 (socket says 172.21.15.16:53220) 2023-05-22T16:03:44.111 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.110+0000 7fda95ab8640 1 -- 172.21.15.16:0/1181124610 learned_addr learned my addr 172.21.15.16:0/1181124610 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T16:03:44.111 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.110+0000 7fda952b7640 1 -- 172.21.15.16:0/1181124610 >> v1:172.21.15.62:6789/0 conn(0x7fda90159870 legacy=0x7fda900c8ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:03:44.111 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.110+0000 7fda94ab6640 1 -- 172.21.15.16:0/1181124610 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1378017373 0 0) 0x7fda9015a0a0 con 0x7fda901531b0 2023-05-22T16:03:44.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.110+0000 7fda952b7640 1 -- 172.21.15.16:0/1181124610 >> v1:172.21.15.16:6789/0 conn(0x7fda901531b0 legacy=0x7fda900afb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:44.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.110+0000 7fda952b7640 1 -- 172.21.15.16:0/1181124610 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda9015c820 con 0x7fda900b0250 2023-05-22T16:03:44.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.114+0000 7fda952b7640 1 --2- 172.21.15.16:0/1181124610 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fda900b0250 0x7fda90159330 secure :-1 s=READY pgs=2265 cs=0 l=1 rev1=1 crypto rx=0x7fda800097c0 tx=0x7fda8001c410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eea848c299e456a9 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:03:44.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.114+0000 7fda94ab6640 1 -- 172.21.15.16:0/1181124610 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda800ab610 con 0x7fda900b0250 2023-05-22T16:03:44.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.114+0000 7fda94ab6640 1 -- 172.21.15.16:0/1181124610 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda800ab770 con 0x7fda900b0250 2023-05-22T16:03:44.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.114+0000 7fda94ab6640 1 -- 172.21.15.16:0/1181124610 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda800abb00 con 0x7fda900b0250 2023-05-22T16:03:44.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.114+0000 7fda97d1c640 1 -- 172.21.15.16:0/1181124610 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fda900b0250 msgr2=0x7fda90159330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:44.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.114+0000 7fda97d1c640 1 --2- 172.21.15.16:0/1181124610 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fda900b0250 0x7fda90159330 secure :-1 s=READY pgs=2265 cs=0 l=1 rev1=1 crypto rx=0x7fda800097c0 tx=0x7fda8001c410 comp rx=0 tx=0).stop 2023-05-22T16:03:44.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.114+0000 7fda97d1c640 1 -- 172.21.15.16:0/1181124610 shutdown_connections 2023-05-22T16:03:44.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.114+0000 7fda97d1c640 1 --2- 172.21.15.16:0/1181124610 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fda900b0250 0x7fda90159330 unknown :-1 s=CLOSED pgs=2265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:44.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.114+0000 7fda97d1c640 1 -- 172.21.15.16:0/1181124610 >> 172.21.15.16:0/1181124610 conn(0x7fda90069720 msgr2=0x7fda90069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:03:44.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.114+0000 7fda97d1c640 1 -- 172.21.15.16:0/1181124610 shutdown_connections 2023-05-22T16:03:44.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.114+0000 7fda97d1c640 1 -- 172.21.15.16:0/1181124610 wait complete. 2023-05-22T16:03:44.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.114+0000 7fda97d1c640 1 Processor -- start 2023-05-22T16:03:44.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.114+0000 7fda97d1c640 1 -- start start 2023-05-22T16:03:44.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.114+0000 7fda97d1c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fda900b0250 0x7fda900b39f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:44.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.114+0000 7fda97d1c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fda901531b0 0x7fda90150330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:44.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.114+0000 7fda97d1c640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fda90159870 0x7fda900b0eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:44.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.114+0000 7fda97d1c640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda90150870 con 0x7fda900b0250 2023-05-22T16:03:44.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.114+0000 7fda97d1c640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda901f4390 con 0x7fda901531b0 2023-05-22T16:03:44.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.114+0000 7fda97d1c640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda9014f290 con 0x7fda90159870 2023-05-22T16:03:44.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.114+0000 7fda95ab8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fda900b0250 0x7fda900b39f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:44.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.114+0000 7fda95ab8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fda900b0250 0x7fda900b39f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49052/0 (socket says 172.21.15.16:49052) 2023-05-22T16:03:44.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.114+0000 7fda95ab8640 1 -- 172.21.15.16:0/1435944526 learned_addr learned my addr 172.21.15.16:0/1435944526 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:03:44.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.114+0000 7fda952b7640 1 --2- 172.21.15.16:0/1435944526 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fda901531b0 0x7fda90150330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:44.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.114+0000 7fda95ab8640 1 -- 172.21.15.16:0/1435944526 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fda90159870 msgr2=0x7fda900b0eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:44.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.114+0000 7fda95ab8640 1 --2- 172.21.15.16:0/1435944526 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fda90159870 0x7fda900b0eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:44.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.114+0000 7fda95ab8640 1 -- 172.21.15.16:0/1435944526 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fda901531b0 msgr2=0x7fda90150330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:44.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.114+0000 7fda95ab8640 1 --2- 172.21.15.16:0/1435944526 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fda901531b0 0x7fda90150330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:44.118 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.114+0000 7fda95ab8640 1 -- 172.21.15.16:0/1435944526 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda80009420 con 0x7fda900b0250 2023-05-22T16:03:44.119 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.114+0000 7fda95ab8640 1 --2- 172.21.15.16:0/1435944526 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fda900b0250 0x7fda900b39f0 secure :-1 s=READY pgs=2266 cs=0 l=1 rev1=1 crypto rx=0x7fda8c0011f0 tx=0x7fda8c00c9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:03:44.119 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.114+0000 7fda867fc640 1 -- 172.21.15.16:0/1435944526 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda8c0074e0 con 0x7fda900b0250 2023-05-22T16:03:44.119 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.114+0000 7fda867fc640 1 -- 172.21.15.16:0/1435944526 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda8c007640 con 0x7fda900b0250 2023-05-22T16:03:44.119 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.118+0000 7fda97d1c640 1 -- 172.21.15.16:0/1435944526 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda90159cc0 con 0x7fda900b0250 2023-05-22T16:03:44.119 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.118+0000 7fda867fc640 1 -- 172.21.15.16:0/1435944526 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda8c004a10 con 0x7fda900b0250 2023-05-22T16:03:44.119 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.118+0000 7fda97d1c640 1 -- 172.21.15.16:0/1435944526 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda9015d0c0 con 0x7fda900b0250 2023-05-22T16:03:44.120 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.118+0000 7fda97d1c640 1 -- 172.21.15.16:0/1435944526 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda90159e50 con 0x7fda900b0250 2023-05-22T16:03:44.120 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.118+0000 7fda867fc640 1 -- 172.21.15.16:0/1435944526 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 82) v1 ==== 84332+0+0 (secure 0 0 0) 0x7fda8c004b70 con 0x7fda900b0250 2023-05-22T16:03:44.120 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.118+0000 7fda867fc640 1 --2- 172.21.15.16:0/1435944526 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fda740654d0 0x7fda74067980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:44.120 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.118+0000 7fda867fc640 1 -- 172.21.15.16:0/1435944526 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(179..179 src has 1..179) v4 ==== 6438+0+0 (secure 0 0 0) 0x7fda8c081c70 con 0x7fda900b0250 2023-05-22T16:03:44.120 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.118+0000 7fda952b7640 1 --2- 172.21.15.16:0/1435944526 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fda740654d0 0x7fda74067980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:44.120 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.118+0000 7fda952b7640 1 --2- 172.21.15.16:0/1435944526 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fda740654d0 0x7fda74067980 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fda800af000 tx=0x7fda800a7000 comp rx=0 tx=0).ready entity=mgr.10032 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:03:44.122 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.122+0000 7fda867fc640 1 -- 172.21.15.16:0/1435944526 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fda8c04f6f0 con 0x7fda900b0250 2023-05-22T16:03:44.313 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.314+0000 7fda97d1c640 1 -- 172.21.15.16:0/1435944526 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fda9004dbe0 con 0x7fda900b0250 2023-05-22T16:03:44.314 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.314+0000 7fda867fc640 1 -- 172.21.15.16:0/1435944526 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 73 v73) v1 ==== 94+0+4657 (secure 0 0 0) 0x7fda8c018020 con 0x7fda900b0250 2023-05-22T16:03:44.315 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T16:03:44.315 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":73,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7682,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/2815032143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":2815032143},{"type":"v1","addr":"172.21.15.62:6835","nonce":2815032143}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":69},{"gid":10239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6836/1961058261","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1961058261},{"type":"v1","addr":"172.21.15.16:6836","nonce":1961058261}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":69},{"gid":10242,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/1453687184","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6835","nonce":1453687184},{"type":"v1","addr":"172.21.15.16:6837","nonce":1453687184}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":73,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-05-22T16:03:38.779645+0000","modified":"2023-05-22T16:03:42.129478+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10251":{"gid":10251,"name":"d","rank":0,"incarnation":70,"state":"up:active","state_seq":3,"addr":"172.21.15.62:6833/2486335217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":2486335217},{"type":"v1","addr":"172.21.15.62:6833","nonce":2486335217}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[23],"metadata_pool":22,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2023-05-22T16:03:44.315 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 73 2023-05-22T16:03:44.317 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.318+0000 7fda97d1c640 1 -- 172.21.15.16:0/1435944526 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fda740654d0 msgr2=0x7fda74067980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:44.317 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.318+0000 7fda97d1c640 1 --2- 172.21.15.16:0/1435944526 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fda740654d0 0x7fda74067980 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fda800af000 tx=0x7fda800a7000 comp rx=0 tx=0).stop 2023-05-22T16:03:44.317 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.318+0000 7fda97d1c640 1 -- 172.21.15.16:0/1435944526 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fda900b0250 msgr2=0x7fda900b39f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:44.317 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.318+0000 7fda97d1c640 1 --2- 172.21.15.16:0/1435944526 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fda900b0250 0x7fda900b39f0 secure :-1 s=READY pgs=2266 cs=0 l=1 rev1=1 crypto rx=0x7fda8c0011f0 tx=0x7fda8c00c9b0 comp rx=0 tx=0).stop 2023-05-22T16:03:44.317 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.318+0000 7fda97d1c640 1 -- 172.21.15.16:0/1435944526 shutdown_connections 2023-05-22T16:03:44.318 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.318+0000 7fda97d1c640 1 --2- 172.21.15.16:0/1435944526 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fda90159870 0x7fda900b0eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:44.318 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.318+0000 7fda97d1c640 1 --2- 172.21.15.16:0/1435944526 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fda901531b0 0x7fda90150330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:44.318 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.318+0000 7fda97d1c640 1 --2- 172.21.15.16:0/1435944526 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fda740654d0 0x7fda74067980 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:44.318 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.318+0000 7fda97d1c640 1 --2- 172.21.15.16:0/1435944526 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fda900b0250 0x7fda900b39f0 unknown :-1 s=CLOSED pgs=2266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:44.318 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.318+0000 7fda97d1c640 1 -- 172.21.15.16:0/1435944526 >> 172.21.15.16:0/1435944526 conn(0x7fda90069720 msgr2=0x7fda9014dcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:03:44.318 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.318+0000 7fda97d1c640 1 -- 172.21.15.16:0/1435944526 shutdown_connections 2023-05-22T16:03:44.318 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.318+0000 7fda97d1c640 1 -- 172.21.15.16:0/1435944526 wait complete. 2023-05-22T16:03:44.736 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 73, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-05-22T16:03:38.779645+0000', 'modified': '2023-05-22T16:03:42.129478+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 10251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10251': {'gid': 10251, 'name': 'd', 'rank': 0, 'incarnation': 70, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.62:6833/2486335217', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.62:6832', 'nonce': 2486335217}, {'type': 'v1', 'addr': '172.21.15.62:6833', 'nonce': 2486335217}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [23], 'metadata_pool': 22, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-05-22T16:03:44.737 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-05-22T16:03:44.737 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2023-05-22T16:03:44.875 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.874+0000 7f0bf2df2640 1 Processor -- start 2023-05-22T16:03:44.875 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.874+0000 7f0bf2df2640 1 -- start start 2023-05-22T16:03:44.876 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.874+0000 7f0bf2df2640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f0bec0b7dd0 0x7f0bec0ccab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:44.877 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.874+0000 7f0bf2df2640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0bec0cd170 con 0x7f0bec160250 2023-05-22T16:03:44.877 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.874+0000 7f0bf2df2640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0bec0d2ac0 con 0x7f0bec15f700 2023-05-22T16:03:44.877 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.874+0000 7f0bf2df2640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0bec0d3360 con 0x7f0bec0b7dd0 2023-05-22T16:03:44.877 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.878+0000 7f0bf138f640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f0bec160250 0x7f0bec0d0410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:53228/0 (socket says 172.21.15.16:53228) 2023-05-22T16:03:44.877 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.878+0000 7f0bf138f640 1 -- 172.21.15.16:0/2751054818 learned_addr learned my addr 172.21.15.16:0/2751054818 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T16:03:44.877 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.878+0000 7f0bf0b8e640 1 --2- 172.21.15.16:0/2751054818 >> v2:172.21.15.62:3300/0 conn(0x7f0bec0b7dd0 0x7f0bec0ccab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:44.877 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.878+0000 7f0beb7fe640 1 -- 172.21.15.16:0/2751054818 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3427147721 0 0) 0x7f0bec0cd170 con 0x7f0bec160250 2023-05-22T16:03:44.878 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.878+0000 7f0beb7fe640 1 -- 172.21.15.16:0/2751054818 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0bd4003530 con 0x7f0bec160250 2023-05-22T16:03:44.879 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.878+0000 7f0beb7fe640 1 -- 172.21.15.16:0/2751054818 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3523183978 0 0) 0x7f0bd4003530 con 0x7f0bec160250 2023-05-22T16:03:44.879 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.878+0000 7f0beb7fe640 1 -- 172.21.15.16:0/2751054818 >> v2:172.21.15.62:3300/0 conn(0x7f0bec0b7dd0 msgr2=0x7f0bec0ccab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:03:44.879 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.878+0000 7f0beb7fe640 1 --2- 172.21.15.16:0/2751054818 >> v2:172.21.15.62:3300/0 conn(0x7f0bec0b7dd0 0x7f0bec0ccab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:44.879 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.878+0000 7f0beb7fe640 1 -- 172.21.15.16:0/2751054818 >> v1:172.21.15.62:6789/0 conn(0x7f0bec15f700 legacy=0x7f0bec15fb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:44.879 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.878+0000 7f0beb7fe640 1 -- 172.21.15.16:0/2751054818 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bec0d2da0 con 0x7f0bec160250 2023-05-22T16:03:44.879 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.878+0000 7f0beb7fe640 1 -- 172.21.15.16:0/2751054818 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7f0be4001a00 con 0x7f0bec160250 2023-05-22T16:03:44.880 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.878+0000 7f0beb7fe640 1 -- 172.21.15.16:0/2751054818 >> v1:172.21.15.16:6789/0 conn(0x7f0bec160250 legacy=0x7f0bec0d0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:44.880 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.878+0000 7f0beb7fe640 1 --2- 172.21.15.16:0/2751054818 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0bd4003980 0x7f0bd4003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:44.880 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.878+0000 7f0beb7fe640 1 --2- 172.21.15.16:0/2751054818 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0bd40047d0 0x7f0bd4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:44.880 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.878+0000 7f0beb7fe640 1 --2- 172.21.15.16:0/2751054818 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0bd4005180 0x7f0bd400b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:44.881 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.878+0000 7f0beb7fe640 1 -- 172.21.15.16:0/2751054818 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bec0d3360 con 0x7f0bd4005180 2023-05-22T16:03:44.881 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.878+0000 7f0beb7fe640 1 -- 172.21.15.16:0/2751054818 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bd400c000 con 0x7f0bd4003980 2023-05-22T16:03:44.881 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.878+0000 7f0beb7fe640 1 -- 172.21.15.16:0/2751054818 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0bd400c1c0 con 0x7f0bd40047d0 2023-05-22T16:03:44.881 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.878+0000 7f0bf138f640 1 --2- 172.21.15.16:0/2751054818 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0bd4005180 0x7f0bd400b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:44.881 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.878+0000 7f0bf0b8e640 1 --2- 172.21.15.16:0/2751054818 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0bd4003980 0x7f0bd4003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:44.881 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.878+0000 7f0bebfff640 1 --2- 172.21.15.16:0/2751054818 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0bd40047d0 0x7f0bd4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:44.881 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.878+0000 7f0bf138f640 1 -- 172.21.15.16:0/2751054818 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0bd40047d0 msgr2=0x7f0bd4004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:03:44.882 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.878+0000 7f0bf138f640 1 --2- 172.21.15.16:0/2751054818 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0bd40047d0 0x7f0bd4004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:44.882 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.878+0000 7f0bf138f640 1 -- 172.21.15.16:0/2751054818 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0bd4003980 msgr2=0x7f0bd4003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:03:44.882 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.878+0000 7f0bf138f640 1 --2- 172.21.15.16:0/2751054818 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0bd4003980 0x7f0bd4003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:44.882 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.878+0000 7f0bf138f640 1 -- 172.21.15.16:0/2751054818 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bd4003530 con 0x7f0bd4005180 2023-05-22T16:03:44.882 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.878+0000 7f0bf0b8e640 1 --2- 172.21.15.16:0/2751054818 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0bd4003980 0x7f0bd4003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T16:03:44.885 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.878+0000 7f0bf138f640 1 -- 172.21.15.16:0/2751054818 reap_dead start 2023-05-22T16:03:44.885 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.878+0000 7f0bf138f640 1 --2- 172.21.15.16:0/2751054818 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0bd4005180 0x7f0bd400b990 secure :-1 s=READY pgs=2268 cs=0 l=1 rev1=1 crypto rx=0x7f0be400a830 tx=0x7f0be401d340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9214a704cb7237d5 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:03:44.886 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.878+0000 7f0beb7fe640 1 -- 172.21.15.16:0/2751054818 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0be40072d0 con 0x7f0bd4005180 2023-05-22T16:03:44.886 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.878+0000 7f0beb7fe640 1 -- 172.21.15.16:0/2751054818 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0be40a8070 con 0x7f0bd4005180 2023-05-22T16:03:44.886 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.878+0000 7f0beb7fe640 1 -- 172.21.15.16:0/2751054818 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0be40076b0 con 0x7f0bd4005180 2023-05-22T16:03:44.886 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.882+0000 7f0bf2df2640 1 -- 172.21.15.16:0/2751054818 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0bd4005180 msgr2=0x7f0bd400b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:44.886 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.882+0000 7f0bf2df2640 1 --2- 172.21.15.16:0/2751054818 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0bd4005180 0x7f0bd400b990 secure :-1 s=READY pgs=2268 cs=0 l=1 rev1=1 crypto rx=0x7f0be400a830 tx=0x7f0be401d340 comp rx=0 tx=0).stop 2023-05-22T16:03:44.887 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.882+0000 7f0bf2df2640 1 -- 172.21.15.16:0/2751054818 shutdown_connections 2023-05-22T16:03:44.887 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.882+0000 7f0bf2df2640 1 --2- 172.21.15.16:0/2751054818 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0bd4005180 0x7f0bd400b990 unknown :-1 s=CLOSED pgs=2268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:44.887 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.882+0000 7f0bf2df2640 1 -- 172.21.15.16:0/2751054818 >> 172.21.15.16:0/2751054818 conn(0x7f0bec0655c0 msgr2=0x7f0bec070e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:03:44.887 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.882+0000 7f0bf2df2640 1 -- 172.21.15.16:0/2751054818 shutdown_connections 2023-05-22T16:03:44.887 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.882+0000 7f0bf2df2640 1 -- 172.21.15.16:0/2751054818 wait complete. 2023-05-22T16:03:44.887 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.882+0000 7f0bf2df2640 1 Processor -- start 2023-05-22T16:03:44.887 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.882+0000 7f0bf2df2640 1 -- start start 2023-05-22T16:03:44.888 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.882+0000 7f0bf2df2640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0bd4005180 0x7f0bec1f0d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:44.888 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.882+0000 7f0bf2df2640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0bec0caf90 0x7f0bec0b5290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:44.888 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.882+0000 7f0bf2df2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0bec0b57d0 0x7f0bec15c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:44.888 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.882+0000 7f0bf2df2640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bec0b5c40 con 0x7f0bec0b57d0 2023-05-22T16:03:44.888 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.882+0000 7f0bf2df2640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bec152230 con 0x7f0bd4005180 2023-05-22T16:03:44.888 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.882+0000 7f0bf2df2640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0bec0cbc70 con 0x7f0bec0caf90 2023-05-22T16:03:44.888 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.882+0000 7f0bf138f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0bec0b57d0 0x7f0bec15c1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:44.889 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.882+0000 7f0bf138f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0bec0b57d0 0x7f0bec15c1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49064/0 (socket says 172.21.15.16:49064) 2023-05-22T16:03:44.889 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.882+0000 7f0bf138f640 1 -- 172.21.15.16:0/1807861777 learned_addr learned my addr 172.21.15.16:0/1807861777 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:03:44.889 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.882+0000 7f0bebfff640 1 --2- 172.21.15.16:0/1807861777 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0bec0caf90 0x7f0bec0b5290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:44.889 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.882+0000 7f0bf0b8e640 1 --2- 172.21.15.16:0/1807861777 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0bd4005180 0x7f0bec1f0d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:44.889 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.882+0000 7f0bf138f640 1 -- 172.21.15.16:0/1807861777 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0bec0caf90 msgr2=0x7f0bec0b5290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:44.889 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.882+0000 7f0bf138f640 1 --2- 172.21.15.16:0/1807861777 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0bec0caf90 0x7f0bec0b5290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:44.889 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.882+0000 7f0bf138f640 1 -- 172.21.15.16:0/1807861777 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0bd4005180 msgr2=0x7f0bec1f0d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:44.889 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.882+0000 7f0bf138f640 1 --2- 172.21.15.16:0/1807861777 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0bd4005180 0x7f0bec1f0d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:44.890 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.882+0000 7f0bf138f640 1 -- 172.21.15.16:0/1807861777 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bec0ccab0 con 0x7f0bec0b57d0 2023-05-22T16:03:44.890 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.882+0000 7f0bf0b8e640 1 --2- 172.21.15.16:0/1807861777 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0bd4005180 0x7f0bec1f0d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T16:03:44.890 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.882+0000 7f0bf138f640 1 --2- 172.21.15.16:0/1807861777 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0bec0b57d0 0x7f0bec15c1e0 secure :-1 s=READY pgs=2269 cs=0 l=1 rev1=1 crypto rx=0x7f0be4007040 tx=0x7f0bd4005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:03:44.890 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.882+0000 7f0be97fa640 1 -- 172.21.15.16:0/1807861777 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0be4001940 con 0x7f0bec0b57d0 2023-05-22T16:03:44.890 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.882+0000 7f0be97fa640 1 -- 172.21.15.16:0/1807861777 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0be4007450 con 0x7f0bec0b57d0 2023-05-22T16:03:44.890 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.882+0000 7f0bf2df2640 1 -- 172.21.15.16:0/1807861777 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0bec0d3080 con 0x7f0bec0b57d0 2023-05-22T16:03:44.890 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.882+0000 7f0be97fa640 1 -- 172.21.15.16:0/1807861777 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0be4007800 con 0x7f0bec0b57d0 2023-05-22T16:03:44.891 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.882+0000 7f0bf2df2640 1 -- 172.21.15.16:0/1807861777 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0bec0ccc40 con 0x7f0bec0b57d0 2023-05-22T16:03:44.891 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.886+0000 7f0be97fa640 1 -- 172.21.15.16:0/1807861777 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 82) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f0be40aa070 con 0x7f0bec0b57d0 2023-05-22T16:03:44.896 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.886+0000 7f0be97fa640 1 --2- 172.21.15.16:0/1807861777 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f0bd80654a0 0x7f0bd8067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:44.896 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.886+0000 7f0be97fa640 1 -- 172.21.15.16:0/1807861777 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(179..179 src has 1..179) v4 ==== 6438+0+0 (secure 0 0 0) 0x7f0be411c260 con 0x7f0bec0b57d0 2023-05-22T16:03:44.896 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.886+0000 7f0bf0b8e640 1 --2- 172.21.15.16:0/1807861777 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f0bd80654a0 0x7f0bd8067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:44.896 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.886+0000 7f0bf0b8e640 1 --2- 172.21.15.16:0/1807861777 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f0bd80654a0 0x7f0bd8067950 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f0bdc009ac0 tx=0x7f0bdc008040 comp rx=0 tx=0).ready entity=mgr.10032 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:03:44.897 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.886+0000 7f0bf5f14000 1 Processor -- start 2023-05-22T16:03:44.897 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.886+0000 7f0bf5f14000 1 -- start start 2023-05-22T16:03:44.897 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.886+0000 7f0bf5f14000 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x55a4fdbecfd0 0x55a4fde5fc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:44.897 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.886+0000 7f0bf5f14000 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a4fde60170 0x55a4fde60540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:44.897 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.886+0000 7f0bf5f14000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a4fde13a10 0x55a4fde13de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:44.897 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.886+0000 7f0bf5f14000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a4fdcddbd0 con 0x55a4fde13a10 2023-05-22T16:03:44.897 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.886+0000 7f0bf5f14000 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a4fdbfadd0 con 0x55a4fde60170 2023-05-22T16:03:44.897 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.886+0000 7f0bf5f14000 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a4fdc6b640 con 0x55a4fdbecfd0 2023-05-22T16:03:44.898 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.886+0000 7f0bf138f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a4fde13a10 0x55a4fde13de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:44.898 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.886+0000 7f0bf138f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a4fde13a10 0x55a4fde13de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49080/0 (socket says 172.21.15.16:49080) 2023-05-22T16:03:44.898 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.886+0000 7f0bf138f640 1 -- 172.21.15.16:0/1892935122 learned_addr learned my addr 172.21.15.16:0/1892935122 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:03:44.898 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.886+0000 7f0bf138f640 1 -- 172.21.15.16:0/1892935122 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x55a4fdbecfd0 msgr2=0x55a4fde5fc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:03:44.898 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.886+0000 7f0bf0b8e640 1 --2- 172.21.15.16:0/1892935122 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x55a4fdbecfd0 0x55a4fde5fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:44.898 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.886+0000 7f0bebfff640 1 --2- 172.21.15.16:0/1892935122 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a4fde60170 0x55a4fde60540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:44.899 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.886+0000 7f0bf138f640 1 --2- 172.21.15.16:0/1892935122 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x55a4fdbecfd0 0x55a4fde5fc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:44.899 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.886+0000 7f0bf138f640 1 -- 172.21.15.16:0/1892935122 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a4fde60170 msgr2=0x55a4fde60540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:03:44.899 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.886+0000 7f0bf138f640 1 --2- 172.21.15.16:0/1892935122 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a4fde60170 0x55a4fde60540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:44.899 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.886+0000 7f0bf138f640 1 -- 172.21.15.16:0/1892935122 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a4fdc2a0b0 con 0x55a4fde13a10 2023-05-22T16:03:44.899 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.890+0000 7f0bf138f640 1 --2- 172.21.15.16:0/1892935122 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a4fde13a10 0x55a4fde13de0 secure :-1 s=READY pgs=2270 cs=0 l=1 rev1=1 crypto rx=0x7f0be40af040 tx=0x7f0be40ed500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32b0eaaf2f3c1ae1 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:03:44.899 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.890+0000 7f0bd27fc640 1 -- 172.21.15.16:0/1892935122 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0be400a450 con 0x55a4fde13a10 2023-05-22T16:03:44.900 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.890+0000 7f0bd27fc640 1 -- 172.21.15.16:0/1892935122 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0be4007960 con 0x55a4fde13a10 2023-05-22T16:03:44.900 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.890+0000 7f0bd27fc640 1 -- 172.21.15.16:0/1892935122 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0be4007b10 con 0x55a4fde13a10 2023-05-22T16:03:44.900 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.890+0000 7f0bf5f14000 1 -- 172.21.15.16:0/1892935122 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a4fde13a10 msgr2=0x55a4fde13de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:44.900 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.890+0000 7f0bf5f14000 1 --2- 172.21.15.16:0/1892935122 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a4fde13a10 0x55a4fde13de0 secure :-1 s=READY pgs=2270 cs=0 l=1 rev1=1 crypto rx=0x7f0be40af040 tx=0x7f0be40ed500 comp rx=0 tx=0).stop 2023-05-22T16:03:44.900 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.890+0000 7f0bf5f14000 1 -- 172.21.15.16:0/1892935122 shutdown_connections 2023-05-22T16:03:44.900 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.890+0000 7f0bf5f14000 1 --2- 172.21.15.16:0/1892935122 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a4fde13a10 0x55a4fde13de0 unknown :-1 s=CLOSED pgs=2270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:44.900 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.890+0000 7f0bf5f14000 1 --2- 172.21.15.16:0/1892935122 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a4fde60170 0x55a4fde60540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:44.901 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.890+0000 7f0bf5f14000 1 --2- 172.21.15.16:0/1892935122 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x55a4fdbecfd0 0x55a4fde5fc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:44.901 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.890+0000 7f0bf5f14000 1 -- 172.21.15.16:0/1892935122 >> 172.21.15.16:0/1892935122 conn(0x55a4fdcbc270 msgr2=0x55a4fde43220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:03:44.901 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.890+0000 7f0bf5f14000 1 -- 172.21.15.16:0/1892935122 shutdown_connections 2023-05-22T16:03:44.901 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.890+0000 7f0bf5f14000 1 -- 172.21.15.16:0/1892935122 wait complete. 2023-05-22T16:03:44.901 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.890+0000 7f0bf5f14000 1 -- start start 2023-05-22T16:03:44.901 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.890+0000 7f0bf5f14000 1 Processor -- start 2023-05-22T16:03:44.901 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.890+0000 7f0bf5f14000 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a4fdbecfd0 0x55a4fde43c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:44.902 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.890+0000 7f0bf5f14000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a4fde13a10 0x55a4fde1b9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:44.902 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.890+0000 7f0bf5f14000 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x55a4fde60170 0x55a4fde1bf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:44.902 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.890+0000 7f0bf5f14000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a4fdc64780 con 0x55a4fde13a10 2023-05-22T16:03:44.902 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.890+0000 7f0bf5f14000 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a4fdc22250 con 0x55a4fdbecfd0 2023-05-22T16:03:44.902 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.890+0000 7f0bf5f14000 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a4fdc21ef0 con 0x55a4fde60170 2023-05-22T16:03:44.902 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.890+0000 7f0bf5f14000 10 client.0 _ll_register_callbacks cb 0x55a4fde01b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-05-22T16:03:44.903 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.890+0000 7f0bebfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a4fde13a10 0x55a4fde1b9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:44.903 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.890+0000 7f0bebfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a4fde13a10 0x55a4fde1b9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49090/0 (socket says 172.21.15.16:49090) 2023-05-22T16:03:44.903 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.890+0000 7f0bebfff640 1 -- 172.21.15.16:0/4187820317 learned_addr learned my addr 172.21.15.16:0/4187820317 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:03:44.903 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.890+0000 7f0bf138f640 1 --2- 172.21.15.16:0/4187820317 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x55a4fde60170 0x55a4fde1bf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:44.903 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.890+0000 7f0bebfff640 1 -- 172.21.15.16:0/4187820317 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x55a4fde60170 msgr2=0x55a4fde1bf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:03:44.903 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.890+0000 7f0bf0b8e640 1 --2- 172.21.15.16:0/4187820317 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a4fdbecfd0 0x55a4fde43c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:44.904 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.890+0000 7f0bebfff640 1 --2- 172.21.15.16:0/4187820317 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x55a4fde60170 0x55a4fde1bf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:44.904 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.890+0000 7f0bebfff640 1 -- 172.21.15.16:0/4187820317 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a4fdbecfd0 msgr2=0x55a4fde43c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:03:44.904 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.890+0000 7f0bebfff640 1 --2- 172.21.15.16:0/4187820317 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a4fdbecfd0 0x55a4fde43c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:44.904 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.890+0000 7f0bebfff640 1 -- 172.21.15.16:0/4187820317 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0bec0ccc40 con 0x55a4fde13a10 2023-05-22T16:03:44.904 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.890+0000 7f0bebfff640 1 --2- 172.21.15.16:0/4187820317 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a4fde13a10 0x55a4fde1b9f0 secure :-1 s=READY pgs=2271 cs=0 l=1 rev1=1 crypto rx=0x7f0be0001e80 tx=0x7f0be00116e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea08d6832c56cd10 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:03:44.904 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.890+0000 7f0bcb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2023-05-22T16:03:44.905 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.890+0000 7f0bcb7fe640 1 -- 172.21.15.16:0/4187820317 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0be00012c0 con 0x55a4fde13a10 2023-05-22T16:03:44.905 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.890+0000 7f0bcb7fe640 1 -- 172.21.15.16:0/4187820317 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(179..179 src has 1..179) v4 ==== 6438+0+0 (secure 0 0 0) 0x7f0be00052e0 con 0x55a4fde13a10 2023-05-22T16:03:44.905 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.890+0000 7f0bf5f14000 1 -- 172.21.15.16:0/4187820317 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55a4fdc28c60 con 0x55a4fde13a10 2023-05-22T16:03:44.905 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.890+0000 7f0bcb7fe640 1 -- 172.21.15.16:0/4187820317 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=73) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0be00054a0 con 0x55a4fde13a10 2023-05-22T16:03:44.905 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.890+0000 7f0bf5f14000 10 client.10392 fetch_fsmap learned FSMap version 73 2023-05-22T16:03:44.905 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.890+0000 7f0bf5f14000 1 -- 172.21.15.16:0/4187820317 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=73,monmap=2+}) v3 -- 0x55a4fd961890 con 0x55a4fde13a10 2023-05-22T16:03:44.905 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.890+0000 7f0bcb7fe640 1 -- 172.21.15.16:0/4187820317 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55a4fdc28c60 con 0x55a4fde13a10 2023-05-22T16:03:44.905 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.890+0000 7f0bcb7fe640 1 -- 172.21.15.16:0/4187820317 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 73) v1 ==== 3106+0+0 (secure 0 0 0) 0x7f0be001d350 con 0x55a4fde13a10 2023-05-22T16:03:44.906 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.890+0000 7f0bf5f14000 10 client.10392 fetch_fsmap finished waiting for FSMap version 73 2023-05-22T16:03:44.906 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.890+0000 7f0bf5f14000 10 client.10392 resolve_mds: resolved name 'd' to daemon mds.d 2023-05-22T16:03:44.906 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.890+0000 7f0bf5f14000 20 client.10392 populate_metadata read hostname 'smithi016' 2023-05-22T16:03:44.906 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.890+0000 7f0bf5f14000 1 --2- 172.21.15.16:0/4187820317 >> [v2:172.21.15.62:6832/2486335217,v1:172.21.15.62:6833/2486335217] conn(0x55a4fde45ca0 0x55a4fde718f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:44.906 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.890+0000 7f0bf5f14000 4 client.10392 mds_command: new command op to 10251 tid=0[{"prefix": "get_command_descriptions"}] 2023-05-22T16:03:44.906 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.890+0000 7f0bf5f14000 1 -- 172.21.15.16:0/4187820317 --> [v2:172.21.15.62:6832/2486335217,v1:172.21.15.62:6833/2486335217] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a4fdc630d0 con 0x55a4fde45ca0 2023-05-22T16:03:44.906 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.894+0000 7f0bf0b8e640 1 --2- 172.21.15.16:0/4187820317 >> [v2:172.21.15.62:6832/2486335217,v1:172.21.15.62:6833/2486335217] conn(0x55a4fde45ca0 0x55a4fde718f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:44.906 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.894+0000 7f0bf0b8e640 1 --2- 172.21.15.16:0/4187820317 >> [v2:172.21.15.62:6832/2486335217,v1:172.21.15.62:6833/2486335217] conn(0x55a4fde45ca0 0x55a4fde718f0 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c9f0a0cb81d9fcf4 server_cookie=72e0652c88a4c271 in_seq=0 out_seq=0 2023-05-22T16:03:44.907 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.894+0000 7f0bcb7fe640 10 client.10392 ms_handle_connect on v2:172.21.15.62:6832/2486335217 2023-05-22T16:03:44.907 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.894+0000 7f0bcb7fe640 1 -- 172.21.15.16:0/4187820317 <== mds.0 v2:172.21.15.62:6832/2486335217 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x55a4fdc630d0 con 0x55a4fde45ca0 2023-05-22T16:03:44.907 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.894+0000 7f0bcb7fe640 10 client.10392 handle_command_reply: tid=0 2023-05-22T16:03:44.907 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.894+0000 7f0bf5f14000 1 client.10392 shutdown 2023-05-22T16:03:44.907 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.894+0000 7f0bf5f14000 1 -- 172.21.15.16:0/4187820317 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a4fde13a10 msgr2=0x55a4fde1b9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:44.907 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.894+0000 7f0bf5f14000 1 --2- 172.21.15.16:0/4187820317 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a4fde13a10 0x55a4fde1b9f0 secure :-1 s=READY pgs=2271 cs=0 l=1 rev1=1 crypto rx=0x7f0be0001e80 tx=0x7f0be00116e0 comp rx=0 tx=0).stop 2023-05-22T16:03:44.907 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.894+0000 7f0bf5f14000 1 -- 172.21.15.16:0/4187820317 shutdown_connections 2023-05-22T16:03:44.908 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.894+0000 7f0bf5f14000 1 --2- 172.21.15.16:0/4187820317 >> [v2:172.21.15.62:6832/2486335217,v1:172.21.15.62:6833/2486335217] conn(0x55a4fde45ca0 0x55a4fde718f0 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:44.908 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.894+0000 7f0bf5f14000 1 --2- 172.21.15.16:0/4187820317 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x55a4fde60170 0x55a4fde1bf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:44.908 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.894+0000 7f0bf5f14000 1 --2- 172.21.15.16:0/4187820317 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a4fde13a10 0x55a4fde1b9f0 unknown :-1 s=CLOSED pgs=2271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:44.908 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.894+0000 7f0bcb7fe640 0 client.10392 ms_handle_reset on v2:172.21.15.62:6832/2486335217 2023-05-22T16:03:44.908 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.894+0000 7f0bf5f14000 1 --2- 172.21.15.16:0/4187820317 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a4fdbecfd0 0x55a4fde43c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:44.908 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.894+0000 7f0bf5f14000 1 -- 172.21.15.16:0/4187820317 >> 172.21.15.16:0/4187820317 conn(0x55a4fdcbc270 msgr2=0x55a4fde13100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:03:44.908 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.894+0000 7f0bf5f14000 1 -- 172.21.15.16:0/4187820317 shutdown_connections 2023-05-22T16:03:44.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.894+0000 7f0bf5f14000 1 -- 172.21.15.16:0/4187820317 wait complete. 2023-05-22T16:03:44.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.894+0000 7f0bf5f14000 20 client.10392 trim_cache size 0 max 16384 2023-05-22T16:03:44.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.910+0000 7f0bf5f14000 1 Processor -- start 2023-05-22T16:03:44.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.910+0000 7f0bf5f14000 1 -- start start 2023-05-22T16:03:44.911 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.910+0000 7f0bf5f14000 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x55a4fdbecfd0 0x55a4fde42e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:44.911 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.910+0000 7f0bf5f14000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a4fde13a10 0x55a4fde40f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:44.911 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.910+0000 7f0bf5f14000 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a4fde60170 0x55a4fde41440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:44.912 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.910+0000 7f0bf5f14000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a4fd922940 con 0x55a4fde13a10 2023-05-22T16:03:44.912 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.910+0000 7f0bf5f14000 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a4fdc61b80 con 0x55a4fde60170 2023-05-22T16:03:44.912 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.910+0000 7f0bf5f14000 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a4fdcee250 con 0x55a4fdbecfd0 2023-05-22T16:03:44.912 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.910+0000 7f0bebfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a4fde13a10 0x55a4fde40f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:44.912 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.910+0000 7f0bebfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a4fde13a10 0x55a4fde40f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49100/0 (socket says 172.21.15.16:49100) 2023-05-22T16:03:44.913 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.910+0000 7f0bebfff640 1 -- 172.21.15.16:0/442822208 learned_addr learned my addr 172.21.15.16:0/442822208 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:03:44.913 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.910+0000 7f0bf0b8e640 1 --2- 172.21.15.16:0/442822208 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x55a4fdbecfd0 0x55a4fde42e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:44.913 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.910+0000 7f0bf138f640 1 --2- 172.21.15.16:0/442822208 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a4fde60170 0x55a4fde41440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:44.913 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.910+0000 7f0bebfff640 1 -- 172.21.15.16:0/442822208 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x55a4fdbecfd0 msgr2=0x55a4fde42e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:03:44.913 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.910+0000 7f0bebfff640 1 --2- 172.21.15.16:0/442822208 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x55a4fdbecfd0 0x55a4fde42e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:44.913 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.910+0000 7f0bebfff640 1 -- 172.21.15.16:0/442822208 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a4fde60170 msgr2=0x55a4fde41440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:03:44.913 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.910+0000 7f0bebfff640 1 --2- 172.21.15.16:0/442822208 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a4fde60170 0x55a4fde41440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:44.913 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.910+0000 7f0bebfff640 1 -- 172.21.15.16:0/442822208 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a4fdc2a360 con 0x55a4fde13a10 2023-05-22T16:03:44.916 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.910+0000 7f0bebfff640 1 --2- 172.21.15.16:0/442822208 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a4fde13a10 0x55a4fde40f00 secure :-1 s=READY pgs=2272 cs=0 l=1 rev1=1 crypto rx=0x7f0be0008ea0 tx=0x7f0be000b000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=664ade25e1531882 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:03:44.916 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.910+0000 7f0bd2ffd640 1 -- 172.21.15.16:0/442822208 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0be00012c0 con 0x55a4fde13a10 2023-05-22T16:03:44.917 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.910+0000 7f0bd2ffd640 1 -- 172.21.15.16:0/442822208 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0be000d040 con 0x55a4fde13a10 2023-05-22T16:03:44.917 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.910+0000 7f0bd2ffd640 1 -- 172.21.15.16:0/442822208 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0be001d8b0 con 0x55a4fde13a10 2023-05-22T16:03:44.917 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.914+0000 7f0bf5f14000 1 -- 172.21.15.16:0/442822208 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a4fde13a10 msgr2=0x55a4fde40f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:44.917 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.914+0000 7f0bf5f14000 1 --2- 172.21.15.16:0/442822208 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a4fde13a10 0x55a4fde40f00 secure :-1 s=READY pgs=2272 cs=0 l=1 rev1=1 crypto rx=0x7f0be0008ea0 tx=0x7f0be000b000 comp rx=0 tx=0).stop 2023-05-22T16:03:44.917 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.914+0000 7f0bf5f14000 1 -- 172.21.15.16:0/442822208 shutdown_connections 2023-05-22T16:03:44.917 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.914+0000 7f0bf5f14000 1 --2- 172.21.15.16:0/442822208 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a4fde60170 0x55a4fde41440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:44.918 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.914+0000 7f0bf5f14000 1 --2- 172.21.15.16:0/442822208 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a4fde13a10 0x55a4fde40f00 unknown :-1 s=CLOSED pgs=2272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:44.918 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.914+0000 7f0bf5f14000 1 --2- 172.21.15.16:0/442822208 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x55a4fdbecfd0 0x55a4fde42e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:44.918 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.914+0000 7f0bf5f14000 1 -- 172.21.15.16:0/442822208 >> 172.21.15.16:0/442822208 conn(0x55a4fdcbc270 msgr2=0x55a4fde44450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:03:44.918 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.914+0000 7f0bf5f14000 1 -- 172.21.15.16:0/442822208 shutdown_connections 2023-05-22T16:03:44.918 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.914+0000 7f0bf5f14000 1 -- 172.21.15.16:0/442822208 wait complete. 2023-05-22T16:03:44.918 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.914+0000 7f0bf5f14000 1 -- start start 2023-05-22T16:03:44.918 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.914+0000 7f0bf5f14000 1 Processor -- start 2023-05-22T16:03:44.918 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.914+0000 7f0bf5f14000 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x55a4fdbecfd0 0x55a4fdebb120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:44.919 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.914+0000 7f0bf5f14000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a4fde13a10 0x55a4fde14e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:44.919 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.914+0000 7f0bf5f14000 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a4fde60170 0x55a4fde15370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:44.919 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.914+0000 7f0bf5f14000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a4fdc5eb20 con 0x55a4fde13a10 2023-05-22T16:03:44.919 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.914+0000 7f0bf5f14000 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a4fdc6c8d0 con 0x55a4fde60170 2023-05-22T16:03:44.919 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.914+0000 7f0bf5f14000 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a4fdc600f0 con 0x55a4fdbecfd0 2023-05-22T16:03:44.919 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.914+0000 7f0bf5f14000 10 client.0 _ll_register_callbacks cb 0x55a4fdcb4eb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-05-22T16:03:44.919 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.914+0000 7f0bebfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a4fde13a10 0x55a4fde14e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:44.920 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.914+0000 7f0bebfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a4fde13a10 0x55a4fde14e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49112/0 (socket says 172.21.15.16:49112) 2023-05-22T16:03:44.920 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.914+0000 7f0bebfff640 1 -- 172.21.15.16:0/2852044756 learned_addr learned my addr 172.21.15.16:0/2852044756 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:03:44.920 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.914+0000 7f0bf0b8e640 1 --2- 172.21.15.16:0/2852044756 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x55a4fdbecfd0 0x55a4fdebb120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:44.920 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.914+0000 7f0bf138f640 1 --2- 172.21.15.16:0/2852044756 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a4fde60170 0x55a4fde15370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:44.920 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.914+0000 7f0bebfff640 1 -- 172.21.15.16:0/2852044756 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x55a4fdbecfd0 msgr2=0x55a4fdebb120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:03:44.920 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.914+0000 7f0bebfff640 1 --2- 172.21.15.16:0/2852044756 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x55a4fdbecfd0 0x55a4fdebb120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:44.920 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.914+0000 7f0bebfff640 1 -- 172.21.15.16:0/2852044756 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a4fde60170 msgr2=0x55a4fde15370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:03:44.920 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.914+0000 7f0bebfff640 1 --2- 172.21.15.16:0/2852044756 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a4fde60170 0x55a4fde15370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:44.921 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.914+0000 7f0bebfff640 1 -- 172.21.15.16:0/2852044756 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a4fd961890 con 0x55a4fde13a10 2023-05-22T16:03:44.921 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.914+0000 7f0bf138f640 1 --2- 172.21.15.16:0/2852044756 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a4fde60170 0x55a4fde15370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T16:03:44.921 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.914+0000 7f0bf0b8e640 1 --2- 172.21.15.16:0/2852044756 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x55a4fdbecfd0 0x55a4fdebb120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T16:03:44.921 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.914+0000 7f0bebfff640 1 --2- 172.21.15.16:0/2852044756 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a4fde13a10 0x55a4fde14e30 secure :-1 s=READY pgs=2273 cs=0 l=1 rev1=1 crypto rx=0x7f0be0011d60 tx=0x7f0be0001130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57017a2fe7f541b8 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:03:44.921 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.914+0000 7f0bcb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2023-05-22T16:03:44.921 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.914+0000 7f0bcb7fe640 1 -- 172.21.15.16:0/2852044756 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0be00069e0 con 0x55a4fde13a10 2023-05-22T16:03:44.921 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.914+0000 7f0bcb7fe640 1 -- 172.21.15.16:0/2852044756 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(179..179 src has 1..179) v4 ==== 6438+0+0 (secure 0 0 0) 0x7f0be0006b40 con 0x55a4fde13a10 2023-05-22T16:03:44.922 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.914+0000 7f0bf5f14000 1 -- 172.21.15.16:0/2852044756 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55a4fde1ab70 con 0x55a4fde13a10 2023-05-22T16:03:44.922 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.914+0000 7f0bcb7fe640 1 -- 172.21.15.16:0/2852044756 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=73) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0be0006d40 con 0x55a4fde13a10 2023-05-22T16:03:44.922 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.914+0000 7f0bf5f14000 10 client.10398 fetch_fsmap learned FSMap version 73 2023-05-22T16:03:44.922 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.914+0000 7f0bf5f14000 1 -- 172.21.15.16:0/2852044756 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=73,monmap=2+}) v3 -- 0x55a4fdc0fbe0 con 0x55a4fde13a10 2023-05-22T16:03:44.922 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.914+0000 7f0bcb7fe640 1 -- 172.21.15.16:0/2852044756 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55a4fde1ab70 con 0x55a4fde13a10 2023-05-22T16:03:44.922 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.914+0000 7f0bcb7fe640 1 -- 172.21.15.16:0/2852044756 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 73) v1 ==== 3106+0+0 (secure 0 0 0) 0x7f0be0007020 con 0x55a4fde13a10 2023-05-22T16:03:44.922 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.914+0000 7f0bf5f14000 10 client.10398 fetch_fsmap finished waiting for FSMap version 73 2023-05-22T16:03:44.923 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.914+0000 7f0bf5f14000 10 client.10398 resolve_mds: resolved name 'd' to daemon mds.d 2023-05-22T16:03:44.923 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.914+0000 7f0bf5f14000 20 client.10398 populate_metadata read hostname 'smithi016' 2023-05-22T16:03:44.923 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.914+0000 7f0bf5f14000 1 --2- 172.21.15.16:0/2852044756 >> [v2:172.21.15.62:6832/2486335217,v1:172.21.15.62:6833/2486335217] conn(0x55a4fde15cd0 0x55a4fde160a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:44.923 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.914+0000 7f0bf5f14000 4 client.10398 mds_command: new command op to 10251 tid=0[{"prefix": "status"}] 2023-05-22T16:03:44.923 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.914+0000 7f0bf5f14000 1 -- 172.21.15.16:0/2852044756 --> [v2:172.21.15.62:6832/2486335217,v1:172.21.15.62:6833/2486335217] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55a4fdc0b4d0 con 0x55a4fde15cd0 2023-05-22T16:03:44.924 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.914+0000 7f0bf0b8e640 1 --2- 172.21.15.16:0/2852044756 >> [v2:172.21.15.62:6832/2486335217,v1:172.21.15.62:6833/2486335217] conn(0x55a4fde15cd0 0x55a4fde160a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:44.924 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.918+0000 7f0bcb7fe640 10 client.10398 ms_handle_connect on v2:172.21.15.62:6832/2486335217 2023-05-22T16:03:44.924 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.918+0000 7f0bf0b8e640 1 --2- 172.21.15.16:0/2852044756 >> [v2:172.21.15.62:6832/2486335217,v1:172.21.15.62:6833/2486335217] conn(0x55a4fde15cd0 0x55a4fde160a0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d6a6260b11820cd8 server_cookie=6f977ca84695fceb in_seq=0 out_seq=0 2023-05-22T16:03:44.925 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.918+0000 7f0bcb7fe640 1 -- 172.21.15.16:0/2852044756 <== mds.0 v2:172.21.15.62:6832/2486335217 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+321 (crc 0 0 0) 0x55a4fdc0b4d0 con 0x55a4fde15cd0 2023-05-22T16:03:44.925 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.918+0000 7f0bcb7fe640 10 client.10398 handle_command_reply: tid=0 2023-05-22T16:03:44.925 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.918+0000 7f0bf5f14000 1 client.10398 shutdown 2023-05-22T16:03:44.925 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.918+0000 7f0bf5f14000 1 -- 172.21.15.16:0/2852044756 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a4fde13a10 msgr2=0x55a4fde14e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:44.926 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.918+0000 7f0bf5f14000 1 --2- 172.21.15.16:0/2852044756 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a4fde13a10 0x55a4fde14e30 secure :-1 s=READY pgs=2273 cs=0 l=1 rev1=1 crypto rx=0x7f0be0011d60 tx=0x7f0be0001130 comp rx=0 tx=0).stop 2023-05-22T16:03:44.926 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.918+0000 7f0bf5f14000 1 -- 172.21.15.16:0/2852044756 shutdown_connections 2023-05-22T16:03:44.926 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.918+0000 7f0bf5f14000 1 --2- 172.21.15.16:0/2852044756 >> [v2:172.21.15.62:6832/2486335217,v1:172.21.15.62:6833/2486335217] conn(0x55a4fde15cd0 0x55a4fde160a0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:44.926 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.918+0000 7f0bf5f14000 1 --2- 172.21.15.16:0/2852044756 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a4fde60170 0x55a4fde15370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:44.926 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.918+0000 7f0bf5f14000 1 --2- 172.21.15.16:0/2852044756 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a4fde13a10 0x55a4fde14e30 unknown :-1 s=CLOSED pgs=2273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:44.926 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.918+0000 7f0bcb7fe640 0 client.10398 ms_handle_reset on v2:172.21.15.62:6832/2486335217 2023-05-22T16:03:44.926 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.918+0000 7f0bf5f14000 1 --2- 172.21.15.16:0/2852044756 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x55a4fdbecfd0 0x55a4fdebb120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:44.927 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.918+0000 7f0bf5f14000 1 -- 172.21.15.16:0/2852044756 >> 172.21.15.16:0/2852044756 conn(0x55a4fdcbc270 msgr2=0x55a4fde44450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:03:44.927 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.918+0000 7f0bf5f14000 1 -- 172.21.15.16:0/2852044756 shutdown_connections 2023-05-22T16:03:44.927 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.918+0000 7f0bf5f14000 1 -- 172.21.15.16:0/2852044756 wait complete. 2023-05-22T16:03:44.927 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.918+0000 7f0bf5f14000 20 client.10398 trim_cache size 0 max 16384 2023-05-22T16:03:44.927 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-05-22T16:03:44.928 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_fsid": "b2719834-7847-4d98-90a6-f65d57f85c49", 2023-05-22T16:03:44.928 INFO:teuthology.orchestra.run.smithi016.stdout: "whoami": 0, 2023-05-22T16:03:44.928 INFO:teuthology.orchestra.run.smithi016.stdout: "id": 10251, 2023-05-22T16:03:44.928 INFO:teuthology.orchestra.run.smithi016.stdout: "want_state": "up:active", 2023-05-22T16:03:44.928 INFO:teuthology.orchestra.run.smithi016.stdout: "state": "up:active", 2023-05-22T16:03:44.928 INFO:teuthology.orchestra.run.smithi016.stdout: "fs_name": "cephfs", 2023-05-22T16:03:44.928 INFO:teuthology.orchestra.run.smithi016.stdout: "rank_uptime": 6.134689786, 2023-05-22T16:03:44.929 INFO:teuthology.orchestra.run.smithi016.stdout: "mdsmap_epoch": 73, 2023-05-22T16:03:44.929 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch": 179, 2023-05-22T16:03:44.929 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch_barrier": 179, 2023-05-22T16:03:44.929 INFO:teuthology.orchestra.run.smithi016.stdout: "uptime": 10.583251633 2023-05-22T16:03:44.929 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-05-22T16:03:44.930 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.918+0000 7f0bf2df2640 1 -- 172.21.15.16:0/1807861777 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f0bd80654a0 msgr2=0x7f0bd8067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:44.930 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.918+0000 7f0bf2df2640 1 --2- 172.21.15.16:0/1807861777 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f0bd80654a0 0x7f0bd8067950 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f0bdc009ac0 tx=0x7f0bdc008040 comp rx=0 tx=0).stop 2023-05-22T16:03:44.930 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.918+0000 7f0bf2df2640 1 -- 172.21.15.16:0/1807861777 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0bec0b57d0 msgr2=0x7f0bec15c1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:44.930 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.918+0000 7f0bf2df2640 1 --2- 172.21.15.16:0/1807861777 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0bec0b57d0 0x7f0bec15c1e0 secure :-1 s=READY pgs=2269 cs=0 l=1 rev1=1 crypto rx=0x7f0be4007040 tx=0x7f0bd4005800 comp rx=0 tx=0).stop 2023-05-22T16:03:44.930 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.918+0000 7f0bf2df2640 1 -- 172.21.15.16:0/1807861777 shutdown_connections 2023-05-22T16:03:44.931 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.918+0000 7f0bf2df2640 1 --2- 172.21.15.16:0/1807861777 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f0bd80654a0 0x7f0bd8067950 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:44.931 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.918+0000 7f0bf2df2640 1 --2- 172.21.15.16:0/1807861777 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0bec0b57d0 0x7f0bec15c1e0 unknown :-1 s=CLOSED pgs=2269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:44.931 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.918+0000 7f0bf2df2640 1 --2- 172.21.15.16:0/1807861777 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0bec0caf90 0x7f0bec0b5290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:44.931 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.918+0000 7f0bf2df2640 1 --2- 172.21.15.16:0/1807861777 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0bd4005180 0x7f0bec1f0d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:44.931 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.918+0000 7f0bf2df2640 1 -- 172.21.15.16:0/1807861777 >> 172.21.15.16:0/1807861777 conn(0x7f0bec0655c0 msgr2=0x7f0bec15b9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:03:44.931 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.918+0000 7f0bf2df2640 1 -- 172.21.15.16:0/1807861777 shutdown_connections 2023-05-22T16:03:44.931 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:44.918+0000 7f0bf2df2640 1 -- 172.21.15.16:0/1807861777 wait complete. 2023-05-22T16:03:44.935 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-05-22T16:03:45.052 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:45.050+0000 7f77ca5b8640 1 Processor -- start 2023-05-22T16:03:45.053 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:45.050+0000 7f77ca5b8640 1 -- start start 2023-05-22T16:03:45.055 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:45.054+0000 7f77ca5b8640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f77c4158660 0x7f77c40cdfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:45.055 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:45.054+0000 7f77ca5b8640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77c40c92b0 con 0x7f77c40cce20 2023-05-22T16:03:45.055 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:45.054+0000 7f77ca5b8640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77c40cc060 con 0x7f77c4158290 2023-05-22T16:03:45.055 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:45.054+0000 7f77ca5b8640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77c40cc340 con 0x7f77c4158660 2023-05-22T16:03:45.055 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:45.054+0000 7f77c8b55640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f77c40cce20 0x7f77c406f630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:53240/0 (socket says 172.21.15.16:53240) 2023-05-22T16:03:45.055 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:45.054+0000 7f77c8b55640 1 -- 172.21.15.16:0/3529468485 learned_addr learned my addr 172.21.15.16:0/3529468485 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T16:03:45.056 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:45.054+0000 7f77bb7fe640 1 --2- 172.21.15.16:0/3529468485 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f77c4158660 0x7f77c40cdfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:45.056 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:45.054+0000 7f77c37fe640 1 -- 172.21.15.16:0/3529468485 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1909888461 0 0) 0x7f77c40c92b0 con 0x7f77c40cce20 2023-05-22T16:03:45.056 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:45.054+0000 7f77c37fe640 1 -- 172.21.15.16:0/3529468485 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f77b0003530 con 0x7f77c40cce20 2023-05-22T16:03:45.056 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:45.054+0000 7f77c37fe640 1 -- 172.21.15.16:0/3529468485 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2006100089 0 0) 0x7f77b0003530 con 0x7f77c40cce20 2023-05-22T16:03:45.056 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:45.054+0000 7f77c37fe640 1 -- 172.21.15.16:0/3529468485 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f77c4158660 msgr2=0x7f77c40cdfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:03:45.056 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:45.054+0000 7f77c37fe640 1 --2- 172.21.15.16:0/3529468485 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f77c4158660 0x7f77c40cdfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:45.056 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:45.054+0000 7f77c37fe640 1 -- 172.21.15.16:0/3529468485 >> v1:172.21.15.62:6789/0 conn(0x7f77c4158290 legacy=0x7f77c40c8880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:45.057 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:45.054+0000 7f77c37fe640 1 -- 172.21.15.16:0/3529468485 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77c40cc620 con 0x7f77c40cce20 2023-05-22T16:03:45.057 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:45.054+0000 7f77c37fe640 1 -- 172.21.15.16:0/3529468485 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7f77bc001a00 con 0x7f77c40cce20 2023-05-22T16:03:45.057 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:45.054+0000 7f77c37fe640 1 -- 172.21.15.16:0/3529468485 >> v1:172.21.15.16:6789/0 conn(0x7f77c40cce20 legacy=0x7f77c406f630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:45.057 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:45.054+0000 7f77c37fe640 1 --2- 172.21.15.16:0/3529468485 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f77c4158660 0x7f77b00039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:45.057 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:45.054+0000 7f77c37fe640 1 --2- 172.21.15.16:0/3529468485 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f77b0003f30 0x7f77b00047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:45.057 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:45.054+0000 7f77c37fe640 1 --2- 172.21.15.16:0/3529468485 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f77b0004d10 0x7f77b0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:45.057 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:45.054+0000 7f77c37fe640 1 -- 172.21.15.16:0/3529468485 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77c40cc340 con 0x7f77b0003f30 2023-05-22T16:03:45.058 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:45.054+0000 7f77c37fe640 1 -- 172.21.15.16:0/3529468485 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77b000ba70 con 0x7f77b0004d10 2023-05-22T16:03:45.058 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:45.054+0000 7f77c37fe640 1 -- 172.21.15.16:0/3529468485 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77b000bc30 con 0x7f77c4158660 2023-05-22T16:03:45.058 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:45.054+0000 7f77bb7fe640 1 --2- 172.21.15.16:0/3529468485 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f77b0003f30 0x7f77b00047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:45.058 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:45.054+0000 7f77c3fff640 1 --2- 172.21.15.16:0/3529468485 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f77c4158660 0x7f77b00039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:45.058 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:45.054+0000 7f77c8b55640 1 --2- 172.21.15.16:0/3529468485 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f77b0004d10 0x7f77b0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:45.058 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:45.054+0000 7f77bb7fe640 1 -- 172.21.15.16:0/3529468485 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f77c4158660 msgr2=0x7f77b00039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:03:45.058 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:45.054+0000 7f77bb7fe640 1 --2- 172.21.15.16:0/3529468485 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f77c4158660 0x7f77b00039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:45.059 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:45.054+0000 7f77bb7fe640 1 -- 172.21.15.16:0/3529468485 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f77b0004d10 msgr2=0x7f77b0005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:03:45.059 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:45.054+0000 7f77bb7fe640 1 --2- 172.21.15.16:0/3529468485 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f77b0004d10 0x7f77b0005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:45.059 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:45.054+0000 7f77bb7fe640 1 -- 172.21.15.16:0/3529468485 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77b0003530 con 0x7f77b0003f30 2023-05-22T16:03:45.059 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:45.054+0000 7f77c3fff640 1 --2- 172.21.15.16:0/3529468485 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f77c4158660 0x7f77b00039b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T16:03:45.061 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:45.058+0000 7f77bb7fe640 1 --2- 172.21.15.16:0/3529468485 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f77b0003f30 0x7f77b00047d0 secure :-1 s=READY pgs=2275 cs=0 l=1 rev1=1 crypto rx=0x7f77ac00aab0 tx=0x7f77ac01ce60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a25455c11445407d server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:03:45.061 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:45.058+0000 7f77c37fe640 1 -- 172.21.15.16:0/3529468485 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77ac0a5d60 con 0x7f77b0003f30 2023-05-22T16:03:45.062 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:45.058+0000 7f77c37fe640 1 -- 172.21.15.16:0/3529468485 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77ac0ad440 con 0x7f77b0003f30 2023-05-22T16:03:45.062 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:45.058+0000 7f77c37fe640 1 -- 172.21.15.16:0/3529468485 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77ac0ad7f0 con 0x7f77b0003f30 2023-05-22T16:03:45.062 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:45.058+0000 7f77ca5b8640 1 -- 172.21.15.16:0/3529468485 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f77b0003f30 msgr2=0x7f77b00047d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:45.062 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:45.058+0000 7f77ca5b8640 1 --2- 172.21.15.16:0/3529468485 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f77b0003f30 0x7f77b00047d0 secure :-1 s=READY pgs=2275 cs=0 l=1 rev1=1 crypto rx=0x7f77ac00aab0 tx=0x7f77ac01ce60 comp rx=0 tx=0).stop 2023-05-22T16:03:45.062 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:45.058+0000 7f77c8b55640 1 -- 172.21.15.16:0/3529468485 reap_dead start 2023-05-22T16:03:45.062 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:45.058+0000 7f77ca5b8640 1 -- 172.21.15.16:0/3529468485 shutdown_connections 2023-05-22T16:03:45.062 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:45.058+0000 7f77ca5b8640 1 -- 172.21.15.16:0/3529468485 >> 172.21.15.16:0/3529468485 conn(0x7f77c4069720 msgr2=0x7f77c4069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:03:45.063 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:45.058+0000 7f77ca5b8640 1 -- 172.21.15.16:0/3529468485 shutdown_connections 2023-05-22T16:03:45.063 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:45.058+0000 7f77ca5b8640 1 -- 172.21.15.16:0/3529468485 wait complete. 2023-05-22T16:03:45.063 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:45.058+0000 7f77ca5b8640 1 Processor -- start 2023-05-22T16:03:45.063 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:45.058+0000 7f77ca5b8640 1 -- start start 2023-05-22T16:03:45.063 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:45.058+0000 7f77ca5b8640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f77b0003f30 0x7f77c4150080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:45.063 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:45.058+0000 7f77ca5b8640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f77c40c1840 0x7f77c40c7900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:45.063 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:45.058+0000 7f77ca5b8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f77c40c50f0 0x7f77c40c5560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:45.064 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:45.058+0000 7f77ca5b8640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77c40c6470 con 0x7f77c40c50f0 2023-05-22T16:03:45.064 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:45.058+0000 7f77ca5b8640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77c40b26e0 con 0x7f77b0003f30 2023-05-22T16:03:45.064 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:45.058+0000 7f77ca5b8640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77c41505c0 con 0x7f77c40c1840 2023-05-22T16:03:45.064 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:45.058+0000 7f77c8b55640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f77c40c50f0 0x7f77c40c5560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:45.065 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:45.058+0000 7f77c8b55640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f77c40c50f0 0x7f77c40c5560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49130/0 (socket says 172.21.15.16:49130) 2023-05-22T16:03:45.065 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:45.058+0000 7f77c8b55640 1 -- 172.21.15.16:0/1114167853 learned_addr learned my addr 172.21.15.16:0/1114167853 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:03:45.065 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:45.058+0000 7f77bb7fe640 1 --2- 172.21.15.16:0/1114167853 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f77c40c1840 0x7f77c40c7900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:45.065 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:45.058+0000 7f77c3fff640 1 --2- 172.21.15.16:0/1114167853 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f77b0003f30 0x7f77c4150080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:45.065 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:45.062+0000 7f77c8b55640 1 -- 172.21.15.16:0/1114167853 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f77c40c1840 msgr2=0x7f77c40c7900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:45.065 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:45.062+0000 7f77c8b55640 1 --2- 172.21.15.16:0/1114167853 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f77c40c1840 0x7f77c40c7900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:45.065 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:45.062+0000 7f77c8b55640 1 -- 172.21.15.16:0/1114167853 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f77b0003f30 msgr2=0x7f77c4150080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:45.065 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:45.062+0000 7f77c8b55640 1 --2- 172.21.15.16:0/1114167853 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f77b0003f30 0x7f77c4150080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:45.066 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:45.062+0000 7f77c8b55640 1 -- 172.21.15.16:0/1114167853 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77c40c8f90 con 0x7f77c40c50f0 2023-05-22T16:03:45.066 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:45.062+0000 7f77c8b55640 1 --2- 172.21.15.16:0/1114167853 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f77c40c50f0 0x7f77c40c5560 secure :-1 s=READY pgs=2276 cs=0 l=1 rev1=1 crypto rx=0x7f77c40cbe20 tx=0x7f77bc011850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:03:45.066 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:45.062+0000 7f77c17fa640 1 -- 172.21.15.16:0/1114167853 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77bc014370 con 0x7f77c40c50f0 2023-05-22T16:03:45.066 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:45.062+0000 7f77c17fa640 1 -- 172.21.15.16:0/1114167853 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77bc0144d0 con 0x7f77c40c50f0 2023-05-22T16:03:45.066 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:45.062+0000 7f77ca5b8640 1 -- 172.21.15.16:0/1114167853 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77c40cc900 con 0x7f77c40c50f0 2023-05-22T16:03:45.066 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:45.062+0000 7f77c17fa640 1 -- 172.21.15.16:0/1114167853 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77bc014880 con 0x7f77c40c50f0 2023-05-22T16:03:45.066 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:45.062+0000 7f77ca5b8640 1 -- 172.21.15.16:0/1114167853 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77c415cd10 con 0x7f77c40c50f0 2023-05-22T16:03:45.067 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:45.062+0000 7f77ca5b8640 1 -- 172.21.15.16:0/1114167853 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77c404dbe0 con 0x7f77c40c50f0 2023-05-22T16:03:45.067 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:45.062+0000 7f77c17fa640 1 -- 172.21.15.16:0/1114167853 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 82) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f77bc0149e0 con 0x7f77c40c50f0 2023-05-22T16:03:45.067 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:45.062+0000 7f77c17fa640 1 --2- 172.21.15.16:0/1114167853 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f77a40654a0 0x7f77a4067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:45.068 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:45.062+0000 7f77c17fa640 1 -- 172.21.15.16:0/1114167853 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(179..179 src has 1..179) v4 ==== 6438+0+0 (secure 0 0 0) 0x7f77bc085090 con 0x7f77c40c50f0 2023-05-22T16:03:45.068 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:45.062+0000 7f77c3fff640 1 --2- 172.21.15.16:0/1114167853 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f77a40654a0 0x7f77a4067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:45.068 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:45.062+0000 7f77c3fff640 1 --2- 172.21.15.16:0/1114167853 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f77a40654a0 0x7f77a4067950 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f77b4007fd0 tx=0x7f77b4007b70 comp rx=0 tx=0).ready entity=mgr.10032 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:03:45.068 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:45.066+0000 7f77c17fa640 1 -- 172.21.15.16:0/1114167853 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f77bc0564a0 con 0x7f77c40c50f0 2023-05-22T16:03:45.250 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:45.250+0000 7f77ca5b8640 1 -- 172.21.15.16:0/1114167853 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f77c40b2820 con 0x7f77c40c50f0 2023-05-22T16:03:45.250 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:45.250+0000 7f77c17fa640 1 -- 172.21.15.16:0/1114167853 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v73) v1 ==== 53+0+73 (secure 0 0 0) 0x7f77bc056090 con 0x7f77c40c50f0 2023-05-22T16:03:45.251 INFO:teuthology.orchestra.run.smithi016.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-05-22T16:03:45.252 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:45.250+0000 7f77ca5b8640 1 -- 172.21.15.16:0/1114167853 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f77a40654a0 msgr2=0x7f77a4067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:45.253 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:45.250+0000 7f77ca5b8640 1 --2- 172.21.15.16:0/1114167853 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f77a40654a0 0x7f77a4067950 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f77b4007fd0 tx=0x7f77b4007b70 comp rx=0 tx=0).stop 2023-05-22T16:03:45.253 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:45.250+0000 7f77ca5b8640 1 -- 172.21.15.16:0/1114167853 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f77c40c50f0 msgr2=0x7f77c40c5560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:45.253 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:45.250+0000 7f77ca5b8640 1 --2- 172.21.15.16:0/1114167853 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f77c40c50f0 0x7f77c40c5560 secure :-1 s=READY pgs=2276 cs=0 l=1 rev1=1 crypto rx=0x7f77c40cbe20 tx=0x7f77bc011850 comp rx=0 tx=0).stop 2023-05-22T16:03:45.253 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:45.254+0000 7f77ca5b8640 1 -- 172.21.15.16:0/1114167853 shutdown_connections 2023-05-22T16:03:45.253 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:45.254+0000 7f77ca5b8640 1 --2- 172.21.15.16:0/1114167853 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f77a40654a0 0x7f77a4067950 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:45.254 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:45.254+0000 7f77ca5b8640 1 --2- 172.21.15.16:0/1114167853 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f77c40c50f0 0x7f77c40c5560 unknown :-1 s=CLOSED pgs=2276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:45.254 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:45.254+0000 7f77ca5b8640 1 --2- 172.21.15.16:0/1114167853 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f77c40c1840 0x7f77c40c7900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:45.254 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:45.254+0000 7f77ca5b8640 1 --2- 172.21.15.16:0/1114167853 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f77b0003f30 0x7f77c4150080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:45.254 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:45.254+0000 7f77ca5b8640 1 -- 172.21.15.16:0/1114167853 >> 172.21.15.16:0/1114167853 conn(0x7f77c4069720 msgr2=0x7f77c40ae850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:03:45.254 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:45.254+0000 7f77ca5b8640 1 -- 172.21.15.16:0/1114167853 shutdown_connections 2023-05-22T16:03:45.254 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:45.254+0000 7f77ca5b8640 1 -- 172.21.15.16:0/1114167853 wait complete. 2023-05-22T16:03:45.267 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-05-22T16:03:45.268 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-05-22T16:03:45.268 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi016.front.sepia.ceph.com 2023-05-22T16:03:45.269 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-05-22T16:03:45.269 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-05-22T16:03:45.269 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-05-22T16:03:45.269 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-05-22T16:03:45.270 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-05-22T16:03:45.270 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-05-22T16:03:45.270 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T16:03:45.271 DEBUG:teuthology.orchestra.run.smithi016:> ip addr 2023-05-22T16:03:45.278 INFO:teuthology.orchestra.run.smithi016.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-05-22T16:03:45.278 INFO:teuthology.orchestra.run.smithi016.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-05-22T16:03:45.278 INFO:teuthology.orchestra.run.smithi016.stdout: inet 127.0.0.1/8 scope host lo 2023-05-22T16:03:45.278 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-05-22T16:03:45.278 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 ::1/128 scope host 2023-05-22T16:03:45.278 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-05-22T16:03:45.279 INFO:teuthology.orchestra.run.smithi016.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-05-22T16:03:45.279 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:6b:fd:e0 brd ff:ff:ff:ff:ff:ff 2023-05-22T16:03:45.279 INFO:teuthology.orchestra.run.smithi016.stdout: altname enp7s0f0 2023-05-22T16:03:45.279 INFO:teuthology.orchestra.run.smithi016.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-05-22T16:03:45.279 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:6b:fd:e1 brd ff:ff:ff:ff:ff:ff 2023-05-22T16:03:45.279 INFO:teuthology.orchestra.run.smithi016.stdout: altname enp7s0f1 2023-05-22T16:03:45.279 INFO:teuthology.orchestra.run.smithi016.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2023-05-22T16:03:45.279 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:bd:15:64 brd ff:ff:ff:ff:ff:ff 2023-05-22T16:03:45.280 INFO:teuthology.orchestra.run.smithi016.stdout: altname enp1s0f0 2023-05-22T16:03:45.280 INFO:teuthology.orchestra.run.smithi016.stdout: inet 172.21.15.16/20 brd 172.21.15.255 scope global dynamic ens1f0 2023-05-22T16:03:45.280 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft 40291sec preferred_lft 40291sec 2023-05-22T16:03:45.280 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::ec4:7aff:febd:1564/64 scope link 2023-05-22T16:03:45.280 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-05-22T16:03:45.280 INFO:teuthology.orchestra.run.smithi016.stdout:5: ens1f1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-05-22T16:03:45.280 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:bd:15:65 brd ff:ff:ff:ff:ff:ff 2023-05-22T16:03:45.281 INFO:teuthology.orchestra.run.smithi016.stdout: altname enp1s0f1 2023-05-22T16:03:45.281 INFO:teuthology.orchestra.run.smithi016.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2023-05-22T16:03:45.281 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 02:42:1f:b8:98:0a brd ff:ff:ff:ff:ff:ff 2023-05-22T16:03:45.281 INFO:teuthology.orchestra.run.smithi016.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2023-05-22T16:03:45.281 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-05-22T16:03:45.281 INFO:teuthology.orchestra.run.smithi016.stdout:9: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-05-22T16:03:45.281 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 5e:1f:84:6d:79:ae brd ff:ff:ff:ff:ff:ff 2023-05-22T16:03:45.282 INFO:teuthology.orchestra.run.smithi016.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-05-22T16:03:45.282 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-05-22T16:03:45.282 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::5c1f:84ff:fe6d:79ae/64 scope link 2023-05-22T16:03:45.282 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-05-22T16:03:45.282 INFO:teuthology.orchestra.run.smithi016.stdout:10: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-05-22T16:03:45.282 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0e:5c:f6:1f:82:2f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-05-22T16:03:45.282 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::c5c:f6ff:fe1f:822f/64 scope link 2023-05-22T16:03:45.283 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-05-22T16:03:45.283 INFO:teuthology.orchestra.run.smithi016.stdout:11: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-05-22T16:03:45.283 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether c6:a6:3d:69:d0:db brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-05-22T16:03:45.283 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::c4a6:3dff:fe69:d0db/64 scope link 2023-05-22T16:03:45.283 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-05-22T16:03:45.283 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T16:03:45.284 DEBUG:teuthology.orchestra.run.smithi016:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-05-22T16:03:45.331 INFO:teuthology.orchestra.run.smithi016.stdout:1 2023-05-22T16:03:45.333 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T16:03:45.333 DEBUG:teuthology.orchestra.run.smithi016:> route 2023-05-22T16:03:45.383 INFO:teuthology.orchestra.run.smithi016.stdout:Kernel IP routing table 2023-05-22T16:03:45.383 INFO:teuthology.orchestra.run.smithi016.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-05-22T16:03:45.383 INFO:teuthology.orchestra.run.smithi016.stdout:default _gateway 0.0.0.0 UG 0 0 0 ens1f0 2023-05-22T16:03:45.383 INFO:teuthology.orchestra.run.smithi016.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2023-05-22T16:03:45.384 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 0 0 0 ens1f0 2023-05-22T16:03:45.384 INFO:teuthology.orchestra.run.smithi016.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-05-22T16:03:45.384 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T16:03:45.385 DEBUG:teuthology.orchestra.run.smithi016:> (cd / && exec bash -c ' 2023-05-22T16:03:45.385 DEBUG:teuthology.orchestra.run.smithi016:> set -e 2023-05-22T16:03:45.385 DEBUG:teuthology.orchestra.run.smithi016:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2023-05-22T16:03:45.385 DEBUG:teuthology.orchestra.run.smithi016:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2023-05-22T16:03:45.386 DEBUG:teuthology.orchestra.run.smithi016:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2023-05-22T16:03:45.386 DEBUG:teuthology.orchestra.run.smithi016:> ') 2023-05-22T16:03:45.457 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T16:03:45.458 DEBUG:teuthology.orchestra.run.smithi016:> ip netns list 2023-05-22T16:03:45.509 INFO:teuthology.orchestra.run.smithi016.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-05-22T16:03:45.509 INFO:teuthology.orchestra.run.smithi016.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-05-22T16:03:45.510 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-05-22T16:03:45.510 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T16:03:45.510 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip link set brx.0 up 2023-05-22T16:03:45.567 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2023-05-22T16:03:45.567 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-05-22T16:03:45.568 DEBUG:teuthology.orchestra.run.smithi016:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-05-22T16:03:45.613 INFO:teuthology.orchestra.run.smithi016.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-05-22T16:03:45.614 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-05-22T16:03:45.614 DEBUG:teuthology.orchestra.run.smithi016:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-05-22T16:03:45.665 DEBUG:teuthology.orchestra.run.smithi016:> sudo modprobe fuse 2023-05-22T16:03:45.723 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-05-22T16:03:45.723 DEBUG:teuthology.orchestra.run.smithi016:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-05-22T16:03:45.777 INFO:teuthology.orchestra.run.smithi016.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-05-22T16:03:45.778 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-05-22T16:03:45.779 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T16:03:45.779 DEBUG:teuthology.orchestra.run.smithi016:> ls /sys/fs/fuse/connections 2023-05-22T16:03:45.829 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2023-05-22T16:03:45.830 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2023-05-22T16:03:45.875 DEBUG:teuthology.orchestra.run.smithi016:> sudo modprobe fuse 2023-05-22T16:03:45.890 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-05-22T16:03:45.891 DEBUG:teuthology.orchestra.run.smithi016:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-05-22T16:03:45.943 INFO:teuthology.orchestra.run.smithi016.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-05-22T16:03:45.945 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-05-22T16:03:45.946 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T16:03:45.946 DEBUG:teuthology.orchestra.run.smithi016:> ls /sys/fs/fuse/connections 2023-05-22T16:03:45.959 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi016.stderr:2023-05-22T16:03:45.958+0000 7f3722196580 -1 init, newargv = 0x55d3646cde20 newargc=15 2023-05-22T16:03:45.959 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi016.stderr:ceph-fuse[67810]: starting ceph client 2023-05-22T16:03:45.968 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi016.stderr:ceph-fuse[67810]: starting fuse 2023-05-22T16:03:47.064 DEBUG:teuthology.orchestra.run.smithi016:> sudo modprobe fuse 2023-05-22T16:03:47.083 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-05-22T16:03:47.084 DEBUG:teuthology.orchestra.run.smithi016:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-05-22T16:03:47.144 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-05-22T16:03:47.145 INFO:teuthology.orchestra.run.smithi016.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-05-22T16:03:47.145 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T16:03:47.146 DEBUG:teuthology.orchestra.run.smithi016:> ls /sys/fs/fuse/connections 2023-05-22T16:03:47.193 INFO:teuthology.orchestra.run.smithi016.stdout:45 2023-05-22T16:03:47.194 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [45] 2023-05-22T16:03:47.194 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2023-05-22T16:03:47.195 DEBUG:teuthology.orchestra.run.smithi016:> import glob 2023-05-22T16:03:47.195 DEBUG:teuthology.orchestra.run.smithi016:> import re 2023-05-22T16:03:47.195 DEBUG:teuthology.orchestra.run.smithi016:> import os 2023-05-22T16:03:47.196 DEBUG:teuthology.orchestra.run.smithi016:> import subprocess 2023-05-22T16:03:47.196 DEBUG:teuthology.orchestra.run.smithi016:> 2023-05-22T16:03:47.196 DEBUG:teuthology.orchestra.run.smithi016:> def _find_admin_socket(client_name): 2023-05-22T16:03:47.197 DEBUG:teuthology.orchestra.run.smithi016:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2023-05-22T16:03:47.197 DEBUG:teuthology.orchestra.run.smithi016:> files = glob.glob(asok_path) 2023-05-22T16:03:47.197 DEBUG:teuthology.orchestra.run.smithi016:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2023-05-22T16:03:47.197 DEBUG:teuthology.orchestra.run.smithi016:> 2023-05-22T16:03:47.198 DEBUG:teuthology.orchestra.run.smithi016:> # Given a non-glob path, it better be there 2023-05-22T16:03:47.198 DEBUG:teuthology.orchestra.run.smithi016:> if "*" not in asok_path: 2023-05-22T16:03:47.198 DEBUG:teuthology.orchestra.run.smithi016:> assert(len(files) == 1) 2023-05-22T16:03:47.199 DEBUG:teuthology.orchestra.run.smithi016:> return files[0] 2023-05-22T16:03:47.199 DEBUG:teuthology.orchestra.run.smithi016:> 2023-05-22T16:03:47.199 DEBUG:teuthology.orchestra.run.smithi016:> for f in files: 2023-05-22T16:03:47.200 DEBUG:teuthology.orchestra.run.smithi016:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-05-22T16:03:47.200 DEBUG:teuthology.orchestra.run.smithi016:> if os.path.exists("/proc/{0}".format(pid)): 2023-05-22T16:03:47.200 DEBUG:teuthology.orchestra.run.smithi016:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-05-22T16:03:47.200 DEBUG:teuthology.orchestra.run.smithi016:> contents = proc_f.read() 2023-05-22T16:03:47.201 DEBUG:teuthology.orchestra.run.smithi016:> if mountpoint in contents: 2023-05-22T16:03:47.201 DEBUG:teuthology.orchestra.run.smithi016:> return f 2023-05-22T16:03:47.201 DEBUG:teuthology.orchestra.run.smithi016:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-05-22T16:03:47.202 DEBUG:teuthology.orchestra.run.smithi016:> 2023-05-22T16:03:47.202 DEBUG:teuthology.orchestra.run.smithi016:> print(_find_admin_socket("client.0")) 2023-05-22T16:03:47.202 DEBUG:teuthology.orchestra.run.smithi016:> ' 2023-05-22T16:03:47.319 INFO:teuthology.orchestra.run.smithi016.stdout:/var/run/ceph/ceph-client.0.67810.asok 2023-05-22T16:03:47.500 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.67810.asok 2023-05-22T16:03:47.501 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T16:03:47.501 DEBUG:teuthology.orchestra.run.smithi016:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.67810.asok status 2023-05-22T16:03:47.661 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-05-22T16:03:47.661 INFO:teuthology.orchestra.run.smithi016.stdout: "metadata": { 2023-05-22T16:03:47.661 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph_sha1": "9c9ba4df22024d9d8d6cf9f98a4b96751fc78d0c", 2023-05-22T16:03:47.662 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph_version": "ceph version 18.0.0-3192-g9c9ba4df (9c9ba4df22024d9d8d6cf9f98a4b96751fc78d0c) reef (dev)", 2023-05-22T16:03:47.662 INFO:teuthology.orchestra.run.smithi016.stdout: "entity_id": "0", 2023-05-22T16:03:47.662 INFO:teuthology.orchestra.run.smithi016.stdout: "hostname": "smithi016", 2023-05-22T16:03:47.663 INFO:teuthology.orchestra.run.smithi016.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2023-05-22T16:03:47.663 INFO:teuthology.orchestra.run.smithi016.stdout: "pid": "67810", 2023-05-22T16:03:47.663 INFO:teuthology.orchestra.run.smithi016.stdout: "root": "/" 2023-05-22T16:03:47.664 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T16:03:47.664 INFO:teuthology.orchestra.run.smithi016.stdout: "dentry_count": 0, 2023-05-22T16:03:47.664 INFO:teuthology.orchestra.run.smithi016.stdout: "dentry_pinned_count": 0, 2023-05-22T16:03:47.664 INFO:teuthology.orchestra.run.smithi016.stdout: "id": 10407, 2023-05-22T16:03:47.664 INFO:teuthology.orchestra.run.smithi016.stdout: "inst": { 2023-05-22T16:03:47.664 INFO:teuthology.orchestra.run.smithi016.stdout: "name": { 2023-05-22T16:03:47.664 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "client", 2023-05-22T16:03:47.665 INFO:teuthology.orchestra.run.smithi016.stdout: "num": 10407 2023-05-22T16:03:47.665 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T16:03:47.665 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": { 2023-05-22T16:03:47.665 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-05-22T16:03:47.666 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "192.168.0.1:0", 2023-05-22T16:03:47.666 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 3732102373 2023-05-22T16:03:47.666 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T16:03:47.666 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T16:03:47.666 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": { 2023-05-22T16:03:47.666 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-05-22T16:03:47.666 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "192.168.0.1:0", 2023-05-22T16:03:47.667 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 3732102373 2023-05-22T16:03:47.667 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T16:03:47.667 INFO:teuthology.orchestra.run.smithi016.stdout: "inst_str": "client.10407 192.168.0.1:0/3732102373", 2023-05-22T16:03:47.667 INFO:teuthology.orchestra.run.smithi016.stdout: "addr_str": "192.168.0.1:0/3732102373", 2023-05-22T16:03:47.667 INFO:teuthology.orchestra.run.smithi016.stdout: "inode_count": 1, 2023-05-22T16:03:47.667 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_epoch": 73, 2023-05-22T16:03:47.667 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_epoch": 179, 2023-05-22T16:03:47.667 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_epoch_barrier": 0, 2023-05-22T16:03:47.668 INFO:teuthology.orchestra.run.smithi016.stdout: "blocklisted": false, 2023-05-22T16:03:47.668 INFO:teuthology.orchestra.run.smithi016.stdout: "fs_name": "cephfs" 2023-05-22T16:03:47.668 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-05-22T16:03:47.675 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T16:03:47.676 DEBUG:teuthology.orchestra.run.smithi016:> stat --file-system '--printf=%T 2023-05-22T16:03:47.676 DEBUG:teuthology.orchestra.run.smithi016:> ' -- /home/ubuntu/cephtest/mnt.0 2023-05-22T16:03:47.685 INFO:teuthology.orchestra.run.smithi016.stdout:fuseblk 2023-05-22T16:03:47.685 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2023-05-22T16:03:47.685 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T16:03:47.686 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-05-22T16:03:47.742 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2023-05-22T16:03:47.915 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:47.914+0000 7f66817ba640 1 Processor -- start 2023-05-22T16:03:47.915 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:47.914+0000 7f66817ba640 1 -- start start 2023-05-22T16:03:47.917 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:47.914+0000 7f66817ba640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f667c0cb6a0 0x7f667c159340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:47.917 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:47.914+0000 7f66817ba640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f667c156b90 con 0x7f667c0b0250 2023-05-22T16:03:47.918 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:47.914+0000 7f66817ba640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f667c15a0b0 con 0x7f667c1567c0 2023-05-22T16:03:47.918 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:47.914+0000 7f66817ba640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f667c0ca8e0 con 0x7f667c0cb6a0 2023-05-22T16:03:47.918 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:47.914+0000 7f667a7fc640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f667c0cb6a0 0x7f667c159340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:47.919 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:47.914+0000 7f667a7fc640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f667c0cb6a0 0x7f667c159340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49132/0 (socket says 172.21.15.16:49132) 2023-05-22T16:03:47.919 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:47.914+0000 7f667a7fc640 1 -- 172.21.15.16:0/2213071941 learned_addr learned my addr 172.21.15.16:0/2213071941 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:03:47.919 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:47.914+0000 7f667b7fe640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f667c0b0250 0x7f667c159880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:53250/0 (socket says 172.21.15.16:53250) 2023-05-22T16:03:47.919 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:47.914+0000 7f667a7fc640 1 -- 172.21.15.16:0/2213071941 >> v1:172.21.15.62:6789/0 conn(0x7f667c1567c0 legacy=0x7f667c0afb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:03:47.920 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:47.914+0000 7f667a7fc640 1 -- 172.21.15.16:0/2213071941 >> v1:172.21.15.16:6789/0 conn(0x7f667c0b0250 legacy=0x7f667c159880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:47.920 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:47.914+0000 7f667a7fc640 1 -- 172.21.15.16:0/2213071941 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f667c0cabc0 con 0x7f667c0cb6a0 2023-05-22T16:03:47.926 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:47.918+0000 7f667a7fc640 1 --2- 172.21.15.16:0/2213071941 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f667c0cb6a0 0x7f667c159340 secure :-1 s=READY pgs=2279 cs=0 l=1 rev1=1 crypto rx=0x7f6670009870 tx=0x7f667001c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee4f068ed336d8eb server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:03:47.926 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:47.918+0000 7f6679ffb640 1 -- 172.21.15.16:0/2213071941 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66700ab600 con 0x7f667c0cb6a0 2023-05-22T16:03:47.926 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:47.918+0000 7f6679ffb640 1 -- 172.21.15.16:0/2213071941 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66700a7050 con 0x7f667c0cb6a0 2023-05-22T16:03:47.927 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:47.918+0000 7f6679ffb640 1 -- 172.21.15.16:0/2213071941 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66700ab980 con 0x7f667c0cb6a0 2023-05-22T16:03:47.927 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:47.918+0000 7f66817ba640 1 -- 172.21.15.16:0/2213071941 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f667c0cb6a0 msgr2=0x7f667c159340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:47.927 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:47.918+0000 7f66817ba640 1 --2- 172.21.15.16:0/2213071941 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f667c0cb6a0 0x7f667c159340 secure :-1 s=READY pgs=2279 cs=0 l=1 rev1=1 crypto rx=0x7f6670009870 tx=0x7f667001c500 comp rx=0 tx=0).stop 2023-05-22T16:03:47.928 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:47.918+0000 7f66817ba640 1 -- 172.21.15.16:0/2213071941 shutdown_connections 2023-05-22T16:03:47.928 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:47.918+0000 7f66817ba640 1 --2- 172.21.15.16:0/2213071941 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f667c0cb6a0 0x7f667c159340 unknown :-1 s=CLOSED pgs=2279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:47.928 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:47.918+0000 7f66817ba640 1 -- 172.21.15.16:0/2213071941 >> 172.21.15.16:0/2213071941 conn(0x7f667c069720 msgr2=0x7f667c069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:03:47.929 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:47.918+0000 7f66817ba640 1 -- 172.21.15.16:0/2213071941 shutdown_connections 2023-05-22T16:03:47.929 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:47.918+0000 7f66817ba640 1 -- 172.21.15.16:0/2213071941 wait complete. 2023-05-22T16:03:47.929 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:47.922+0000 7f66817ba640 1 Processor -- start 2023-05-22T16:03:47.929 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:47.922+0000 7f66817ba640 1 -- start start 2023-05-22T16:03:47.929 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:47.922+0000 7f66817ba640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f667c0b0250 0x7f667c14a020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:47.929 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:47.922+0000 7f66817ba640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f667c0cb6a0 0x7f667c154860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:47.929 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:47.922+0000 7f66817ba640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f667c1567c0 0x7f667c0adb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:47.930 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:47.922+0000 7f66817ba640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f667c1506c0 con 0x7f667c1567c0 2023-05-22T16:03:47.930 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:47.922+0000 7f66817ba640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f667c14a560 con 0x7f667c0b0250 2023-05-22T16:03:47.930 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:47.922+0000 7f66817ba640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f667c14fab0 con 0x7f667c0cb6a0 2023-05-22T16:03:47.930 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:47.922+0000 7f667b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f667c1567c0 0x7f667c0adb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:47.930 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:47.922+0000 7f667b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f667c1567c0 0x7f667c0adb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49144/0 (socket says 172.21.15.16:49144) 2023-05-22T16:03:47.930 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:47.922+0000 7f667b7fe640 1 -- 172.21.15.16:0/996516913 learned_addr learned my addr 172.21.15.16:0/996516913 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:03:47.930 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:47.922+0000 7f667affd640 1 --2- 172.21.15.16:0/996516913 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f667c0b0250 0x7f667c14a020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:47.931 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:47.922+0000 7f667a7fc640 1 --2- 172.21.15.16:0/996516913 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f667c0cb6a0 0x7f667c154860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:47.931 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:47.922+0000 7f667b7fe640 1 -- 172.21.15.16:0/996516913 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f667c0cb6a0 msgr2=0x7f667c154860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:47.931 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:47.922+0000 7f667b7fe640 1 --2- 172.21.15.16:0/996516913 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f667c0cb6a0 0x7f667c154860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:47.931 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:47.922+0000 7f667b7fe640 1 -- 172.21.15.16:0/996516913 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f667c0b0250 msgr2=0x7f667c14a020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:47.931 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:47.922+0000 7f667b7fe640 1 --2- 172.21.15.16:0/996516913 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f667c0b0250 0x7f667c14a020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:47.931 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:47.922+0000 7f667b7fe640 1 -- 172.21.15.16:0/996516913 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66700094d0 con 0x7f667c1567c0 2023-05-22T16:03:47.931 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:47.922+0000 7f667a7fc640 1 --2- 172.21.15.16:0/996516913 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f667c0cb6a0 0x7f667c154860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T16:03:47.931 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:47.922+0000 7f667b7fe640 1 --2- 172.21.15.16:0/996516913 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f667c1567c0 0x7f667c0adb20 secure :-1 s=READY pgs=2280 cs=0 l=1 rev1=1 crypto rx=0x7f667400c3f0 tx=0x7f667400c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:03:47.932 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:47.922+0000 7f66637fe640 1 -- 172.21.15.16:0/996516913 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66740074e0 con 0x7f667c1567c0 2023-05-22T16:03:47.932 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:47.922+0000 7f66637fe640 1 -- 172.21.15.16:0/996516913 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6674007640 con 0x7f667c1567c0 2023-05-22T16:03:47.932 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:47.922+0000 7f66637fe640 1 -- 172.21.15.16:0/996516913 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66740049f0 con 0x7f667c1567c0 2023-05-22T16:03:47.932 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:47.922+0000 7f66817ba640 1 -- 172.21.15.16:0/996516913 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f667c166850 con 0x7f667c1567c0 2023-05-22T16:03:47.932 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:47.922+0000 7f66817ba640 1 -- 172.21.15.16:0/996516913 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f667c1666c0 con 0x7f667c1567c0 2023-05-22T16:03:47.932 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:47.922+0000 7f66817ba640 1 -- 172.21.15.16:0/996516913 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f667c1669e0 con 0x7f667c1567c0 2023-05-22T16:03:47.932 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:47.926+0000 7f66637fe640 1 -- 172.21.15.16:0/996516913 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 82) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f6674004b50 con 0x7f667c1567c0 2023-05-22T16:03:47.932 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:47.926+0000 7f66637fe640 1 --2- 172.21.15.16:0/996516913 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f66680654a0 0x7f6668067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:47.933 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:47.926+0000 7f66637fe640 1 -- 172.21.15.16:0/996516913 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(179..179 src has 1..179) v4 ==== 6438+0+0 (secure 0 0 0) 0x7f6674082440 con 0x7f667c1567c0 2023-05-22T16:03:47.933 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:47.926+0000 7f667affd640 1 --2- 172.21.15.16:0/996516913 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f66680654a0 0x7f6668067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:47.933 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:47.926+0000 7f667affd640 1 --2- 172.21.15.16:0/996516913 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f66680654a0 0x7f6668067950 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f666c009f20 tx=0x7f666c0094d0 comp rx=0 tx=0).ready entity=mgr.10032 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:03:47.933 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:47.930+0000 7f66637fe640 1 -- 172.21.15.16:0/996516913 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f667404fd90 con 0x7f667c1567c0 2023-05-22T16:03:48.104 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:48.102+0000 7f66817ba640 1 -- 172.21.15.16:0/996516913 --> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f667c156b90 con 0x7f66680654a0 2023-05-22T16:03:48.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:48.106+0000 7f66637fe640 1 -- 172.21.15.16:0/996516913 <== mgr.10032 v2:172.21.15.16:6832/65890 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f667c156b90 con 0x7f66680654a0 2023-05-22T16:03:48.107 INFO:teuthology.orchestra.run.smithi016.stdout:[ 2023-05-22T16:03:48.107 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T16:03:48.107 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "cephfs" 2023-05-22T16:03:48.108 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T16:03:48.108 INFO:teuthology.orchestra.run.smithi016.stdout:] 2023-05-22T16:03:48.109 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:48.110+0000 7f66817ba640 1 -- 172.21.15.16:0/996516913 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f66680654a0 msgr2=0x7f6668067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:48.110 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:48.110+0000 7f66817ba640 1 --2- 172.21.15.16:0/996516913 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f66680654a0 0x7f6668067950 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f666c009f20 tx=0x7f666c0094d0 comp rx=0 tx=0).stop 2023-05-22T16:03:48.110 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:48.110+0000 7f66817ba640 1 -- 172.21.15.16:0/996516913 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f667c1567c0 msgr2=0x7f667c0adb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:48.110 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:48.110+0000 7f66817ba640 1 --2- 172.21.15.16:0/996516913 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f667c1567c0 0x7f667c0adb20 secure :-1 s=READY pgs=2280 cs=0 l=1 rev1=1 crypto rx=0x7f667400c3f0 tx=0x7f667400c8b0 comp rx=0 tx=0).stop 2023-05-22T16:03:48.110 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:48.110+0000 7f66817ba640 1 -- 172.21.15.16:0/996516913 shutdown_connections 2023-05-22T16:03:48.111 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:48.110+0000 7f66817ba640 1 --2- 172.21.15.16:0/996516913 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f66680654a0 0x7f6668067950 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:48.111 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:48.110+0000 7f66817ba640 1 --2- 172.21.15.16:0/996516913 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f667c1567c0 0x7f667c0adb20 unknown :-1 s=CLOSED pgs=2280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:48.111 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:48.110+0000 7f66817ba640 1 --2- 172.21.15.16:0/996516913 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f667c0cb6a0 0x7f667c154860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:48.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:48.110+0000 7f66817ba640 1 --2- 172.21.15.16:0/996516913 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f667c0b0250 0x7f667c14a020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:48.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:48.110+0000 7f66817ba640 1 -- 172.21.15.16:0/996516913 >> 172.21.15.16:0/996516913 conn(0x7f667c069720 msgr2=0x7f667c14d430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:03:48.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:48.110+0000 7f66817ba640 1 -- 172.21.15.16:0/996516913 shutdown_connections 2023-05-22T16:03:48.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:48.110+0000 7f66817ba640 1 -- 172.21.15.16:0/996516913 wait complete. 2023-05-22T16:03:48.125 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2023-05-22T16:03:48.240 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:48.242+0000 7fe27137b640 1 Processor -- start 2023-05-22T16:03:48.241 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:48.242+0000 7fe27137b640 1 -- start start 2023-05-22T16:03:48.242 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:48.242+0000 7fe27137b640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe26c0ce480 0x7fe26c071640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:48.242 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:48.242+0000 7fe27137b640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fe26c15c270 0x7fe26c15c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:48.244 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:48.242+0000 7fe27137b640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe26c069ae0 con 0x7fe26c0ce850 2023-05-22T16:03:48.244 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:48.242+0000 7fe27137b640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe26c15f990 con 0x7fe26c0ce480 2023-05-22T16:03:48.244 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:48.242+0000 7fe27137b640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe26c069dc0 con 0x7fe26c15c270 2023-05-22T16:03:48.244 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:48.242+0000 7fe26affd640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe26c0ce480 0x7fe26c071640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:48.244 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:48.242+0000 7fe26affd640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe26c0ce480 0x7fe26c071640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49156/0 (socket says 172.21.15.16:49156) 2023-05-22T16:03:48.244 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:48.242+0000 7fe26affd640 1 -- 172.21.15.16:0/261174144 learned_addr learned my addr 172.21.15.16:0/261174144 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:03:48.245 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:48.242+0000 7fe26a7fc640 1 --1- 172.21.15.16:0/261174144 >> v1:172.21.15.16:6789/0 conn(0x7fe26c0ce850 0x7fe26c160090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:53264/0 (socket says 172.21.15.16:53264) 2023-05-22T16:03:48.245 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:48.242+0000 7fe26b7fe640 1 --2- 172.21.15.16:0/261174144 >> v2:172.21.15.62:3300/0 conn(0x7fe26c15c270 0x7fe26c15c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:48.245 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:48.242+0000 7fe26affd640 1 -- 172.21.15.16:0/261174144 >> v2:172.21.15.62:3300/0 conn(0x7fe26c15c270 msgr2=0x7fe26c15c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:03:48.245 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:48.242+0000 7fe26affd640 1 --2- 172.21.15.16:0/261174144 >> v2:172.21.15.62:3300/0 conn(0x7fe26c15c270 0x7fe26c15c6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:48.245 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:48.242+0000 7fe26affd640 1 -- 172.21.15.16:0/261174144 >> v1:172.21.15.16:6789/0 conn(0x7fe26c0ce850 legacy=0x7fe26c160090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:48.245 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:48.242+0000 7fe26affd640 1 -- 172.21.15.16:0/261174144 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe26c06a0a0 con 0x7fe26c0ce480 2023-05-22T16:03:48.248 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:48.242+0000 7fe26affd640 1 --2- 172.21.15.16:0/261174144 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe26c0ce480 0x7fe26c071640 secure :-1 s=READY pgs=2282 cs=0 l=1 rev1=1 crypto rx=0x7fe26c0aeb30 tx=0x7fe25c01c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36fee57fd51f92b6 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:03:48.248 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:48.242+0000 7fe269ffb640 1 -- 172.21.15.16:0/261174144 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe25c0aa650 con 0x7fe26c0ce480 2023-05-22T16:03:48.248 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:48.242+0000 7fe269ffb640 1 -- 172.21.15.16:0/261174144 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe25c0aa7b0 con 0x7fe26c0ce480 2023-05-22T16:03:48.248 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:48.242+0000 7fe269ffb640 1 -- 172.21.15.16:0/261174144 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe25c0aab40 con 0x7fe26c0ce480 2023-05-22T16:03:48.248 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:48.246+0000 7fe27137b640 1 -- 172.21.15.16:0/261174144 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe26c0ce480 msgr2=0x7fe26c071640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:48.249 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:48.246+0000 7fe27137b640 1 --2- 172.21.15.16:0/261174144 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe26c0ce480 0x7fe26c071640 secure :-1 s=READY pgs=2282 cs=0 l=1 rev1=1 crypto rx=0x7fe26c0aeb30 tx=0x7fe25c01c4d0 comp rx=0 tx=0).stop 2023-05-22T16:03:48.249 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:48.246+0000 7fe27137b640 1 -- 172.21.15.16:0/261174144 shutdown_connections 2023-05-22T16:03:48.249 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:48.246+0000 7fe27137b640 1 --2- 172.21.15.16:0/261174144 >> v2:172.21.15.62:3300/0 conn(0x7fe26c15c270 0x7fe26c15c6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:48.249 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:48.246+0000 7fe27137b640 1 --2- 172.21.15.16:0/261174144 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe26c0ce480 0x7fe26c071640 unknown :-1 s=CLOSED pgs=2282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:48.249 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:48.246+0000 7fe27137b640 1 -- 172.21.15.16:0/261174144 >> 172.21.15.16:0/261174144 conn(0x7fe26c0cc870 msgr2=0x7fe26c0cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:03:48.249 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:48.246+0000 7fe27137b640 1 -- 172.21.15.16:0/261174144 shutdown_connections 2023-05-22T16:03:48.249 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:48.246+0000 7fe27137b640 1 -- 172.21.15.16:0/261174144 wait complete. 2023-05-22T16:03:48.250 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:48.246+0000 7fe27137b640 1 Processor -- start 2023-05-22T16:03:48.250 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:48.246+0000 7fe27137b640 1 -- start start 2023-05-22T16:03:48.250 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:48.246+0000 7fe27137b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe26c0ce480 0x7fe26c14e800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:48.250 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:48.246+0000 7fe27137b640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe26c0ce850 0x7fe26c1f89b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:48.250 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:48.246+0000 7fe27137b640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe26c15c270 0x7fe26c15d580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:48.250 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:48.246+0000 7fe27137b640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe26c154460 con 0x7fe26c0ce480 2023-05-22T16:03:48.250 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:48.246+0000 7fe27137b640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe26c154880 con 0x7fe26c15c270 2023-05-22T16:03:48.251 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:48.246+0000 7fe27137b640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe26c1f8ef0 con 0x7fe26c0ce850 2023-05-22T16:03:48.251 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:48.246+0000 7fe26affd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe26c0ce480 0x7fe26c14e800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:48.251 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:48.246+0000 7fe26affd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe26c0ce480 0x7fe26c14e800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49164/0 (socket says 172.21.15.16:49164) 2023-05-22T16:03:48.251 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:48.246+0000 7fe26affd640 1 -- 172.21.15.16:0/2979553966 learned_addr learned my addr 172.21.15.16:0/2979553966 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:03:48.251 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:48.246+0000 7fe26b7fe640 1 --2- 172.21.15.16:0/2979553966 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe26c15c270 0x7fe26c15d580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:48.251 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:48.246+0000 7fe26a7fc640 1 --2- 172.21.15.16:0/2979553966 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe26c0ce850 0x7fe26c1f89b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:48.251 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:48.246+0000 7fe26affd640 1 -- 172.21.15.16:0/2979553966 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe26c0ce850 msgr2=0x7fe26c1f89b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:48.252 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:48.246+0000 7fe26affd640 1 --2- 172.21.15.16:0/2979553966 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe26c0ce850 0x7fe26c1f89b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:48.252 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:48.246+0000 7fe26affd640 1 -- 172.21.15.16:0/2979553966 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe26c15c270 msgr2=0x7fe26c15d580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:48.252 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:48.246+0000 7fe26affd640 1 --2- 172.21.15.16:0/2979553966 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe26c15c270 0x7fe26c15d580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:48.252 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:48.246+0000 7fe26affd640 1 -- 172.21.15.16:0/2979553966 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe25c0094d0 con 0x7fe26c0ce480 2023-05-22T16:03:48.252 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:48.246+0000 7fe26affd640 1 --2- 172.21.15.16:0/2979553966 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe26c0ce480 0x7fe26c14e800 secure :-1 s=READY pgs=2283 cs=0 l=1 rev1=1 crypto rx=0x7fe25c0ac040 tx=0x7fe25c01ad10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:03:48.252 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:48.246+0000 7fe2637fe640 1 -- 172.21.15.16:0/2979553966 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe25c0b3800 con 0x7fe26c0ce480 2023-05-22T16:03:48.252 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:48.246+0000 7fe2637fe640 1 -- 172.21.15.16:0/2979553966 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe25c0b3960 con 0x7fe26c0ce480 2023-05-22T16:03:48.253 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:48.246+0000 7fe27137b640 1 -- 172.21.15.16:0/2979553966 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe26c0698b0 con 0x7fe26c0ce480 2023-05-22T16:03:48.253 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:48.246+0000 7fe2637fe640 1 -- 172.21.15.16:0/2979553966 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe25c0b3d10 con 0x7fe26c0ce480 2023-05-22T16:03:48.253 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:48.246+0000 7fe27137b640 1 -- 172.21.15.16:0/2979553966 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe26c069720 con 0x7fe26c0ce480 2023-05-22T16:03:48.254 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:48.250+0000 7fe27137b640 1 -- 172.21.15.16:0/2979553966 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe26c0c5e70 con 0x7fe26c0ce480 2023-05-22T16:03:48.254 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:48.250+0000 7fe2637fe640 1 -- 172.21.15.16:0/2979553966 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 82) v1 ==== 84332+0+0 (secure 0 0 0) 0x7fe25c0c6390 con 0x7fe26c0ce480 2023-05-22T16:03:48.254 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:48.250+0000 7fe2637fe640 1 --2- 172.21.15.16:0/2979553966 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fe24c0654a0 0x7fe24c067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:48.254 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:48.250+0000 7fe2637fe640 1 -- 172.21.15.16:0/2979553966 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(179..179 src has 1..179) v4 ==== 6438+0+0 (secure 0 0 0) 0x7fe25c11d960 con 0x7fe26c0ce480 2023-05-22T16:03:48.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:48.250+0000 7fe26a7fc640 1 --2- 172.21.15.16:0/2979553966 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fe24c0654a0 0x7fe24c067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:48.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:48.250+0000 7fe26a7fc640 1 --2- 172.21.15.16:0/2979553966 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fe24c0654a0 0x7fe24c067950 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fe26c0ad020 tx=0x7fe254007c90 comp rx=0 tx=0).ready entity=mgr.10032 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:03:48.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:48.254+0000 7fe2637fe640 1 -- 172.21.15.16:0/2979553966 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fe25c0eb270 con 0x7fe26c0ce480 2023-05-22T16:03:48.443 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:48.442+0000 7fe27137b640 1 -- 172.21.15.16:0/2979553966 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7fe26c1545a0 con 0x7fe26c0ce480 2023-05-22T16:03:49.191 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:49.190+0000 7fe2637fe640 1 -- 172.21.15.16:0/2979553966 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mgrmap(e 83) v1 ==== 84349+0+0 (secure 0 0 0) 0x7fe25c11bba0 con 0x7fe26c0ce480 2023-05-22T16:03:49.193 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:49.194+0000 7fe2637fe640 1 -- 172.21.15.16:0/2979553966 <== mon.0 v2:172.21.15.16:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v83) v1 ==== 92+0+0 (secure 0 0 0) 0x7fe25c0eb270 con 0x7fe26c0ce480 2023-05-22T16:03:49.196 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:49.198+0000 7fe27137b640 1 -- 172.21.15.16:0/2979553966 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fe24c0654a0 msgr2=0x7fe24c067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:49.197 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:49.198+0000 7fe27137b640 1 --2- 172.21.15.16:0/2979553966 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fe24c0654a0 0x7fe24c067950 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fe26c0ad020 tx=0x7fe254007c90 comp rx=0 tx=0).stop 2023-05-22T16:03:49.197 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:49.198+0000 7fe27137b640 1 -- 172.21.15.16:0/2979553966 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe26c0ce480 msgr2=0x7fe26c14e800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:49.197 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:49.198+0000 7fe27137b640 1 --2- 172.21.15.16:0/2979553966 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe26c0ce480 0x7fe26c14e800 secure :-1 s=READY pgs=2283 cs=0 l=1 rev1=1 crypto rx=0x7fe25c0ac040 tx=0x7fe25c01ad10 comp rx=0 tx=0).stop 2023-05-22T16:03:49.197 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:49.198+0000 7fe27137b640 1 -- 172.21.15.16:0/2979553966 shutdown_connections 2023-05-22T16:03:49.197 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:49.198+0000 7fe27137b640 1 --2- 172.21.15.16:0/2979553966 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe26c15c270 0x7fe26c15d580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:49.197 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:49.198+0000 7fe27137b640 1 --2- 172.21.15.16:0/2979553966 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe26c0ce850 0x7fe26c1f89b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:49.197 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:49.198+0000 7fe27137b640 1 --2- 172.21.15.16:0/2979553966 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fe24c0654a0 0x7fe24c067950 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:49.198 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:49.198+0000 7fe27137b640 1 --2- 172.21.15.16:0/2979553966 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe26c0ce480 0x7fe26c14e800 unknown :-1 s=CLOSED pgs=2283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:49.198 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:49.198+0000 7fe27137b640 1 -- 172.21.15.16:0/2979553966 >> 172.21.15.16:0/2979553966 conn(0x7fe26c0cc870 msgr2=0x7fe26c158800 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:03:49.198 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:49.198+0000 7fe27137b640 1 -- 172.21.15.16:0/2979553966 shutdown_connections 2023-05-22T16:03:49.198 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:49.198+0000 7fe27137b640 1 -- 172.21.15.16:0/2979553966 wait complete. 2023-05-22T16:03:49.220 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2023-05-22T16:03:49.314 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T16:03:49.309+0000 7f9e9009b180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-05-22T16:03:49.323 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T16:03:49.322+0000 7f2207031180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-05-22T16:03:49.344 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:49.342+0000 7f645b5ee640 1 Processor -- start 2023-05-22T16:03:49.344 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:49.346+0000 7f645b5ee640 1 -- start start 2023-05-22T16:03:49.346 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:49.346+0000 7f645b5ee640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6454159b50 0x7f64540c8f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:49.347 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:49.346+0000 7f645b5ee640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64540cb120 con 0x7f64540add30 2023-05-22T16:03:49.347 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:49.346+0000 7f645b5ee640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64540cdc40 con 0x7f64540c9440 2023-05-22T16:03:49.347 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:49.346+0000 7f645b5ee640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64540c9890 con 0x7f6454159b50 2023-05-22T16:03:49.347 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:49.346+0000 7f6459b8b640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f64540add30 0x7f64540699c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:53926/0 (socket says 172.21.15.16:53926) 2023-05-22T16:03:49.347 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:49.346+0000 7f6459b8b640 1 -- 172.21.15.16:0/2267557688 learned_addr learned my addr 172.21.15.16:0/2267557688 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T16:03:49.347 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:49.346+0000 7f645938a640 1 --2- 172.21.15.16:0/2267557688 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6454159b50 0x7f64540c8f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:49.347 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:49.346+0000 7f644bfff640 1 -- 172.21.15.16:0/2267557688 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3908071789 0 0) 0x7f644c0014a0 con 0x7f64540add30 2023-05-22T16:03:49.348 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:49.346+0000 7f644bfff640 1 -- 172.21.15.16:0/2267557688 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6440003530 con 0x7f64540add30 2023-05-22T16:03:49.348 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:49.346+0000 7f645938a640 1 -- 172.21.15.16:0/2267557688 >> v1:172.21.15.62:6789/0 conn(0x7f64540c9440 legacy=0x7f64540ad620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:49.348 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:49.346+0000 7f644bfff640 1 -- 172.21.15.16:0/2267557688 <== mon.1 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 85602748 0 0) 0x7f64540cdc40 con 0x7f64540c9440 2023-05-22T16:03:49.348 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:49.346+0000 7f645938a640 1 -- 172.21.15.16:0/2267557688 >> v1:172.21.15.16:6789/0 conn(0x7f64540add30 legacy=0x7f64540699c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:49.348 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:49.346+0000 7f645938a640 1 -- 172.21.15.16:0/2267557688 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6454159f20 con 0x7f6454159b50 2023-05-22T16:03:49.349 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:49.350+0000 7f645938a640 1 --2- 172.21.15.16:0/2267557688 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6454159b50 0x7f64540c8f00 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f64500097f0 tx=0x7f645001c400 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ee7fb5044a8f6794 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:03:49.350 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:49.350+0000 7f644bfff640 1 -- 172.21.15.16:0/2267557688 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64500ab600 con 0x7f6454159b50 2023-05-22T16:03:49.350 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:49.350+0000 7f644bfff640 1 -- 172.21.15.16:0/2267557688 <== mon.2 v2:172.21.15.62:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64500ab760 con 0x7f6454159b50 2023-05-22T16:03:49.350 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:49.350+0000 7f644bfff640 1 -- 172.21.15.16:0/2267557688 <== mon.2 v2:172.21.15.62:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64500abad0 con 0x7f6454159b50 2023-05-22T16:03:49.351 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:49.350+0000 7f645b5ee640 1 -- 172.21.15.16:0/2267557688 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6454159b50 msgr2=0x7f64540c8f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:49.351 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:49.350+0000 7f645b5ee640 1 --2- 172.21.15.16:0/2267557688 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6454159b50 0x7f64540c8f00 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f64500097f0 tx=0x7f645001c400 comp rx=0 tx=0).stop 2023-05-22T16:03:49.351 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:49.350+0000 7f645b5ee640 1 -- 172.21.15.16:0/2267557688 shutdown_connections 2023-05-22T16:03:49.351 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:49.350+0000 7f645b5ee640 1 --2- 172.21.15.16:0/2267557688 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6454159b50 0x7f64540c8f00 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:49.351 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:49.350+0000 7f645b5ee640 1 -- 172.21.15.16:0/2267557688 >> 172.21.15.16:0/2267557688 conn(0x7f64540cf540 msgr2=0x7f64540cb330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:03:49.351 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:49.350+0000 7f645b5ee640 1 -- 172.21.15.16:0/2267557688 shutdown_connections 2023-05-22T16:03:49.352 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:49.350+0000 7f645b5ee640 1 -- 172.21.15.16:0/2267557688 wait complete. 2023-05-22T16:03:49.352 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:49.350+0000 7f645b5ee640 1 Processor -- start 2023-05-22T16:03:49.352 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:49.354+0000 7f645b5ee640 1 -- start start 2023-05-22T16:03:49.354 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:49.354+0000 7f645b5ee640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f64540add30 0x7f645413fc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:49.355 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:49.354+0000 7f645b5ee640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f64540c9440 0x7f64540acb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:49.355 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:49.354+0000 7f645b5ee640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6454143890 0x7f6454143d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:49.355 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:49.354+0000 7f645b5ee640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64541401a0 con 0x7f64540c9440 2023-05-22T16:03:49.355 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:49.354+0000 7f645b5ee640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6454144250 con 0x7f6454143890 2023-05-22T16:03:49.356 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:49.354+0000 7f645b5ee640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6454144730 con 0x7f64540add30 2023-05-22T16:03:49.356 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:49.354+0000 7f6458b89640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f64540c9440 0x7f64540acb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:49.356 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:49.354+0000 7f6458b89640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f64540c9440 0x7f64540acb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59438/0 (socket says 172.21.15.16:59438) 2023-05-22T16:03:49.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:49.354+0000 7f6458b89640 1 -- 172.21.15.16:0/2398689810 learned_addr learned my addr 172.21.15.16:0/2398689810 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:03:49.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:49.354+0000 7f645938a640 1 --2- 172.21.15.16:0/2398689810 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f64540add30 0x7f645413fc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:49.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:49.354+0000 7f6459b8b640 1 --2- 172.21.15.16:0/2398689810 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6454143890 0x7f6454143d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:49.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:49.354+0000 7f6458b89640 1 -- 172.21.15.16:0/2398689810 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f64540add30 msgr2=0x7f645413fc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:49.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:49.354+0000 7f6458b89640 1 --2- 172.21.15.16:0/2398689810 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f64540add30 0x7f645413fc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:49.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:49.354+0000 7f6458b89640 1 -- 172.21.15.16:0/2398689810 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6454143890 msgr2=0x7f6454143d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:49.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:49.354+0000 7f6458b89640 1 --2- 172.21.15.16:0/2398689810 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6454143890 0x7f6454143d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:49.358 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:49.354+0000 7f6458b89640 1 -- 172.21.15.16:0/2398689810 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6450009420 con 0x7f64540c9440 2023-05-22T16:03:49.358 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:49.354+0000 7f6458b89640 1 --2- 172.21.15.16:0/2398689810 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f64540c9440 0x7f64540acb50 secure :-1 s=READY pgs=2285 cs=0 l=1 rev1=1 crypto rx=0x7f6444001ec0 tx=0x7f644400c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:03:49.358 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:49.354+0000 7f6449ffb640 1 -- 172.21.15.16:0/2398689810 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64440054e0 con 0x7f64540c9440 2023-05-22T16:03:49.358 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:49.354+0000 7f645b5ee640 1 -- 172.21.15.16:0/2398689810 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64540cd8d0 con 0x7f64540c9440 2023-05-22T16:03:49.358 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:49.354+0000 7f6449ffb640 1 -- 172.21.15.16:0/2398689810 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6444003600 con 0x7f64540c9440 2023-05-22T16:03:49.358 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:49.354+0000 7f6449ffb640 1 -- 172.21.15.16:0/2398689810 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64440039b0 con 0x7f64540c9440 2023-05-22T16:03:49.358 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:49.354+0000 7f645b5ee640 1 -- 172.21.15.16:0/2398689810 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64540cd5f0 con 0x7f64540c9440 2023-05-22T16:03:49.360 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:49.354+0000 7f645b5ee640 1 -- 172.21.15.16:0/2398689810 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f645406a8e0 con 0x7f64540c9440 2023-05-22T16:03:49.360 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:49.354+0000 7f6449ffb640 1 -- 172.21.15.16:0/2398689810 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 83) v1 ==== 84349+0+0 (secure 0 0 0) 0x7f6444003b10 con 0x7f64540c9440 2023-05-22T16:03:49.360 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:49.358+0000 7f6449ffb640 1 --2- 172.21.15.16:0/2398689810 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f64380654f0 0x7f64380679a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:49.360 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:49.358+0000 7f645938a640 1 -- 172.21.15.16:0/2398689810 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f64380654f0 msgr2=0x7f64380679a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/65890 2023-05-22T16:03:49.360 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:49.358+0000 7f6449ffb640 1 -- 172.21.15.16:0/2398689810 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(179..179 src has 1..179) v4 ==== 6438+0+0 (secure 0 0 0) 0x7f64440821a0 con 0x7f64540c9440 2023-05-22T16:03:49.361 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:49.358+0000 7f645938a640 1 --2- 172.21.15.16:0/2398689810 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f64380654f0 0x7f64380679a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-05-22T16:03:49.361 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:49.362+0000 7f6449ffb640 1 -- 172.21.15.16:0/2398689810 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f644404fbc0 con 0x7f64540c9440 2023-05-22T16:03:49.519 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:49.518+0000 7f645b5ee640 1 -- 172.21.15.16:0/2398689810 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7f6454073920 con 0x7f64540c9440 2023-05-22T16:03:49.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:49.526+0000 7f6449ffb640 1 -- 172.21.15.16:0/2398689810 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v59)=0 v59) v1 ==== 138+0+0 (secure 0 0 0) 0x7f64440531e0 con 0x7f64540c9440 2023-05-22T16:03:49.526 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:49.526+0000 7f645b5ee640 1 -- 172.21.15.16:0/2398689810 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f64380654f0 msgr2=0x7f64380679a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-05-22T16:03:49.527 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:49.526+0000 7f645b5ee640 1 --2- 172.21.15.16:0/2398689810 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f64380654f0 0x7f64380679a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:49.527 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:49.526+0000 7f645b5ee640 1 -- 172.21.15.16:0/2398689810 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f64540c9440 msgr2=0x7f64540acb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:49.527 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:49.526+0000 7f645b5ee640 1 --2- 172.21.15.16:0/2398689810 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f64540c9440 0x7f64540acb50 secure :-1 s=READY pgs=2285 cs=0 l=1 rev1=1 crypto rx=0x7f6444001ec0 tx=0x7f644400c8e0 comp rx=0 tx=0).stop 2023-05-22T16:03:49.527 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:49.526+0000 7f645b5ee640 1 -- 172.21.15.16:0/2398689810 shutdown_connections 2023-05-22T16:03:49.527 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:49.526+0000 7f645b5ee640 1 --2- 172.21.15.16:0/2398689810 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6454143890 0x7f6454143d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:49.527 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:49.526+0000 7f645b5ee640 1 --2- 172.21.15.16:0/2398689810 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f64380654f0 0x7f64380679a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:49.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:49.526+0000 7f645b5ee640 1 --2- 172.21.15.16:0/2398689810 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f64540c9440 0x7f64540acb50 unknown :-1 s=CLOSED pgs=2285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:49.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:49.526+0000 7f645b5ee640 1 --2- 172.21.15.16:0/2398689810 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f64540add30 0x7f645413fc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:49.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:49.526+0000 7f645b5ee640 1 -- 172.21.15.16:0/2398689810 >> 172.21.15.16:0/2398689810 conn(0x7f64540cf540 msgr2=0x7f645406c060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:03:49.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:49.526+0000 7f645b5ee640 1 -- 172.21.15.16:0/2398689810 shutdown_connections 2023-05-22T16:03:49.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:49.526+0000 7f645b5ee640 1 -- 172.21.15.16:0/2398689810 wait complete. 2023-05-22T16:03:49.548 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2023-05-22T16:03:49.673 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:49.674+0000 7f2848f15640 1 Processor -- start 2023-05-22T16:03:49.673 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:49.674+0000 7f2848f15640 1 -- start start 2023-05-22T16:03:49.674 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:49.674+0000 7f2848f15640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f28440c67d0 0x7f28440cf730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:49.674 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:49.674+0000 7f2848f15640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f28440b2880 0x7f28440cbfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:49.674 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:49.674+0000 7f2848f15640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28440cfc70 con 0x7f28440cc520 2023-05-22T16:03:49.675 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:49.674+0000 7f2848f15640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f284415a300 con 0x7f28440b2880 2023-05-22T16:03:49.675 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:49.674+0000 7f2848f15640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28440c4dc0 con 0x7f28440c67d0 2023-05-22T16:03:49.675 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:49.674+0000 7f2841d9b640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f28440b2880 0x7f28440cbfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:49.675 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:49.674+0000 7f2841d9b640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f28440b2880 0x7f28440cbfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59450/0 (socket says 172.21.15.16:59450) 2023-05-22T16:03:49.675 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:49.674+0000 7f2841d9b640 1 -- 172.21.15.16:0/1398078675 learned_addr learned my addr 172.21.15.16:0/1398078675 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:03:49.675 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:49.674+0000 7f284259c640 1 --2- 172.21.15.16:0/1398078675 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f28440c67d0 0x7f28440cf730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:49.675 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:49.674+0000 7f2841d9b640 1 -- 172.21.15.16:0/1398078675 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f28440c67d0 msgr2=0x7f28440cf730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:03:49.675 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:49.674+0000 7f2841d9b640 1 --2- 172.21.15.16:0/1398078675 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f28440c67d0 0x7f28440cf730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:49.676 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:49.674+0000 7f2841d9b640 1 -- 172.21.15.16:0/1398078675 >> v1:172.21.15.16:6789/0 conn(0x7f28440cc520 legacy=0x7f28440ce160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:49.676 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:49.674+0000 7f2841d9b640 1 -- 172.21.15.16:0/1398078675 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28440c6ba0 con 0x7f28440b2880 2023-05-22T16:03:49.676 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:49.678+0000 7f2841d9b640 1 --2- 172.21.15.16:0/1398078675 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f28440b2880 0x7f28440cbfe0 secure :-1 s=READY pgs=2287 cs=0 l=1 rev1=1 crypto rx=0x7f282c009870 tx=0x7f282c01c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a89d64f1482ac4c server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:03:49.677 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:49.678+0000 7f284159a640 1 -- 172.21.15.16:0/1398078675 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f282c0ac600 con 0x7f28440b2880 2023-05-22T16:03:49.677 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:49.678+0000 7f284159a640 1 -- 172.21.15.16:0/1398078675 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f282c0a8050 con 0x7f28440b2880 2023-05-22T16:03:49.677 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:49.678+0000 7f284159a640 1 -- 172.21.15.16:0/1398078675 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f282c0ac980 con 0x7f28440b2880 2023-05-22T16:03:49.678 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:49.678+0000 7f2848f15640 1 -- 172.21.15.16:0/1398078675 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f28440b2880 msgr2=0x7f28440cbfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:49.678 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:49.678+0000 7f2848f15640 1 --2- 172.21.15.16:0/1398078675 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f28440b2880 0x7f28440cbfe0 secure :-1 s=READY pgs=2287 cs=0 l=1 rev1=1 crypto rx=0x7f282c009870 tx=0x7f282c01c480 comp rx=0 tx=0).stop 2023-05-22T16:03:49.679 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:49.678+0000 7f2848f15640 1 -- 172.21.15.16:0/1398078675 shutdown_connections 2023-05-22T16:03:49.679 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:49.678+0000 7f2848f15640 1 --2- 172.21.15.16:0/1398078675 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f28440b2880 0x7f28440cbfe0 unknown :-1 s=CLOSED pgs=2287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:49.679 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:49.678+0000 7f2848f15640 1 --2- 172.21.15.16:0/1398078675 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f28440c67d0 0x7f28440cf730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:49.679 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:49.678+0000 7f2848f15640 1 -- 172.21.15.16:0/1398078675 >> 172.21.15.16:0/1398078675 conn(0x7f284406f5a0 msgr2=0x7f284406f9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:03:49.679 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:49.678+0000 7f2848f15640 1 -- 172.21.15.16:0/1398078675 shutdown_connections 2023-05-22T16:03:49.679 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:49.678+0000 7f2848f15640 1 -- 172.21.15.16:0/1398078675 wait complete. 2023-05-22T16:03:49.681 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:49.678+0000 7f2848f15640 1 Processor -- start 2023-05-22T16:03:49.681 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:49.678+0000 7f2848f15640 1 -- start start 2023-05-22T16:03:49.681 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:49.678+0000 7f2848f15640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f28440b2880 0x7f28440c7f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:49.681 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:49.678+0000 7f2848f15640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f28440c67d0 0x7f28440b4a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:49.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:49.678+0000 7f2848f15640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f28440cc520 0x7f284415b830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:49.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:49.678+0000 7f2848f15640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28440b4570 con 0x7f28440cc520 2023-05-22T16:03:49.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:49.678+0000 7f2848f15640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28440b4240 con 0x7f28440b2880 2023-05-22T16:03:49.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:49.678+0000 7f2848f15640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28440c8460 con 0x7f28440c67d0 2023-05-22T16:03:49.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:49.678+0000 7f2842d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f28440cc520 0x7f284415b830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:49.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:49.678+0000 7f2842d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f28440cc520 0x7f284415b830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59452/0 (socket says 172.21.15.16:59452) 2023-05-22T16:03:49.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:49.678+0000 7f2842d9d640 1 -- 172.21.15.16:0/260547594 learned_addr learned my addr 172.21.15.16:0/260547594 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:03:49.683 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:49.682+0000 7f2841d9b640 1 --2- 172.21.15.16:0/260547594 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f28440c67d0 0x7f28440b4a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:49.683 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:49.682+0000 7f284259c640 1 --2- 172.21.15.16:0/260547594 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f28440b2880 0x7f28440c7f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:49.683 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:49.682+0000 7f2842d9d640 1 -- 172.21.15.16:0/260547594 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f28440c67d0 msgr2=0x7f28440b4a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:49.683 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:49.682+0000 7f2842d9d640 1 --2- 172.21.15.16:0/260547594 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f28440c67d0 0x7f28440b4a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:49.683 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:49.682+0000 7f2842d9d640 1 -- 172.21.15.16:0/260547594 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f28440b2880 msgr2=0x7f28440c7f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:49.683 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:49.682+0000 7f2842d9d640 1 --2- 172.21.15.16:0/260547594 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f28440b2880 0x7f28440c7f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:49.683 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:49.682+0000 7f2842d9d640 1 -- 172.21.15.16:0/260547594 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f282c0094d0 con 0x7f28440cc520 2023-05-22T16:03:49.683 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:49.682+0000 7f2842d9d640 1 --2- 172.21.15.16:0/260547594 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f28440cc520 0x7f284415b830 secure :-1 s=READY pgs=2288 cs=0 l=1 rev1=1 crypto rx=0x7f283400c770 tx=0x7f283400cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:03:49.684 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:49.682+0000 7f283affd640 1 -- 172.21.15.16:0/260547594 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2834003b80 con 0x7f28440cc520 2023-05-22T16:03:49.684 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:49.682+0000 7f283affd640 1 -- 172.21.15.16:0/260547594 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2834003ce0 con 0x7f28440cc520 2023-05-22T16:03:49.684 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:49.682+0000 7f2848f15640 1 -- 172.21.15.16:0/260547594 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28440d0790 con 0x7f28440cc520 2023-05-22T16:03:49.684 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:49.682+0000 7f283affd640 1 -- 172.21.15.16:0/260547594 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f283400aa90 con 0x7f28440cc520 2023-05-22T16:03:49.684 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:49.682+0000 7f2848f15640 1 -- 172.21.15.16:0/260547594 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28440d04b0 con 0x7f28440cc520 2023-05-22T16:03:49.685 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:49.682+0000 7f2848f15640 1 -- 172.21.15.16:0/260547594 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28440d0a70 con 0x7f28440cc520 2023-05-22T16:03:49.685 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:49.682+0000 7f283affd640 1 -- 172.21.15.16:0/260547594 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 83) v1 ==== 84349+0+0 (secure 0 0 0) 0x7f283400abf0 con 0x7f28440cc520 2023-05-22T16:03:49.685 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:49.682+0000 7f283affd640 1 --2- 172.21.15.16:0/260547594 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f28240654f0 0x7f28240679a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:49.685 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:49.682+0000 7f284259c640 1 -- 172.21.15.16:0/260547594 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f28240654f0 msgr2=0x7f28240679a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/65890 2023-05-22T16:03:49.685 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:49.682+0000 7f283affd640 1 -- 172.21.15.16:0/260547594 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(179..179 src has 1..179) v4 ==== 6438+0+0 (secure 0 0 0) 0x7f2834083410 con 0x7f28440cc520 2023-05-22T16:03:49.686 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:49.682+0000 7f284259c640 1 --2- 172.21.15.16:0/260547594 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f28240654f0 0x7f28240679a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-05-22T16:03:49.687 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:49.686+0000 7f283affd640 1 -- 172.21.15.16:0/260547594 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f2834050e50 con 0x7f28440cc520 2023-05-22T16:03:49.699 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T16:03:49.697+0000 7f9e9009b180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-05-22T16:03:49.731 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T16:03:49.730+0000 7f2207031180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-05-22T16:03:49.833 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T16:03:49.829+0000 7f9e9009b180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-05-22T16:03:49.842 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:49.842+0000 7f2848f15640 1 -- 172.21.15.16:0/260547594 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7f284404dbe0 con 0x7f28440cc520 2023-05-22T16:03:49.848 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:49.850+0000 7f283affd640 1 -- 172.21.15.16:0/260547594 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v60)=0 v60) v1 ==== 133+0+0 (secure 0 0 0) 0x7f2834054470 con 0x7f28440cc520 2023-05-22T16:03:49.850 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:49.850+0000 7f2848f15640 1 -- 172.21.15.16:0/260547594 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f28240654f0 msgr2=0x7f28240679a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-05-22T16:03:49.851 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:49.850+0000 7f2848f15640 1 --2- 172.21.15.16:0/260547594 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f28240654f0 0x7f28240679a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:49.851 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:49.850+0000 7f2848f15640 1 -- 172.21.15.16:0/260547594 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f28440cc520 msgr2=0x7f284415b830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:49.851 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:49.850+0000 7f2848f15640 1 --2- 172.21.15.16:0/260547594 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f28440cc520 0x7f284415b830 secure :-1 s=READY pgs=2288 cs=0 l=1 rev1=1 crypto rx=0x7f283400c770 tx=0x7f283400cc30 comp rx=0 tx=0).stop 2023-05-22T16:03:49.851 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:49.850+0000 7f2848f15640 1 -- 172.21.15.16:0/260547594 shutdown_connections 2023-05-22T16:03:49.851 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:49.850+0000 7f2848f15640 1 --2- 172.21.15.16:0/260547594 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f28240654f0 0x7f28240679a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:49.851 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:49.850+0000 7f2848f15640 1 --2- 172.21.15.16:0/260547594 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f28440cc520 0x7f284415b830 unknown :-1 s=CLOSED pgs=2288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:49.851 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:49.850+0000 7f2848f15640 1 --2- 172.21.15.16:0/260547594 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f28440c67d0 0x7f28440b4a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:49.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:49.850+0000 7f2848f15640 1 --2- 172.21.15.16:0/260547594 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f28440b2880 0x7f28440c7f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:49.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:49.850+0000 7f2848f15640 1 -- 172.21.15.16:0/260547594 >> 172.21.15.16:0/260547594 conn(0x7f284406f5a0 msgr2=0x7f28440ce160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:03:49.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:49.850+0000 7f2848f15640 1 -- 172.21.15.16:0/260547594 shutdown_connections 2023-05-22T16:03:49.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:49.850+0000 7f2848f15640 1 -- 172.21.15.16:0/260547594 wait complete. 2023-05-22T16:03:49.867 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T16:03:49.867 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1) 2023-05-22T16:03:49.871 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T16:03:49.870+0000 7f2207031180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-05-22T16:03:49.880 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add --fs cephfs snap_test_dir1 1M 2023-05-22T16:03:50.049 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:50.050+0000 7f3a62bb0640 1 Processor -- start 2023-05-22T16:03:50.049 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:50.050+0000 7f3a62bb0640 1 -- start start 2023-05-22T16:03:50.050 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:50.050+0000 7f3a62bb0640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f3a5c0cfa40 0x7f3a5c0cd260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:50.051 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:50.050+0000 7f3a62bb0640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3a5c06a830 0x7f3a5c06acb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:50.051 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:50.050+0000 7f3a62bb0640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a5c15c0a0 con 0x7f3a5c0cf670 2023-05-22T16:03:50.051 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:50.050+0000 7f3a62bb0640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a5c0cc850 con 0x7f3a5c0cfa40 2023-05-22T16:03:50.051 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:50.050+0000 7f3a62bb0640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a5c0ccdd0 con 0x7f3a5c06a830 2023-05-22T16:03:50.052 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:50.050+0000 7f3a6094c640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f3a5c0cf670 0x7f3a5c1601d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:53946/0 (socket says 172.21.15.16:53946) 2023-05-22T16:03:50.052 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:50.050+0000 7f3a6094c640 1 -- 172.21.15.16:0/2552382927 learned_addr learned my addr 172.21.15.16:0/2552382927 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T16:03:50.052 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:50.050+0000 7f3a5bfff640 1 --2- 172.21.15.16:0/2552382927 >> v2:172.21.15.62:3300/0 conn(0x7f3a5c0cfa40 0x7f3a5c0cd260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:50.053 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:50.050+0000 7f3a6114d640 1 --2- 172.21.15.16:0/2552382927 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3a5c06a830 0x7f3a5c06acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:50.054 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:50.050+0000 7f3a5b7fe640 1 -- 172.21.15.16:0/2552382927 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2670157315 0 0) 0x7f3a50001120 con 0x7f3a5c0cf670 2023-05-22T16:03:50.054 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:50.050+0000 7f3a5b7fe640 1 -- 172.21.15.16:0/2552382927 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3a4c003530 con 0x7f3a5c0cf670 2023-05-22T16:03:50.055 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:50.050+0000 7f3a5bfff640 1 -- 172.21.15.16:0/2552382927 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3a5c06a830 msgr2=0x7f3a5c06acb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:03:50.055 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:50.050+0000 7f3a5bfff640 1 --2- 172.21.15.16:0/2552382927 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3a5c06a830 0x7f3a5c06acb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:50.055 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:50.050+0000 7f3a5bfff640 1 -- 172.21.15.16:0/2552382927 >> v1:172.21.15.16:6789/0 conn(0x7f3a5c0cf670 legacy=0x7f3a5c1601d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:50.056 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:50.050+0000 7f3a5bfff640 1 -- 172.21.15.16:0/2552382927 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a5c06c3d0 con 0x7f3a5c0cfa40 2023-05-22T16:03:50.060 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:50.054+0000 7f3a5bfff640 1 --2- 172.21.15.16:0/2552382927 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3a5c0cfa40 0x7f3a5c0cd260 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f3a480097c0 tx=0x7f3a4801c450 comp rx=0 tx=0).ready entity=mon.1 client_cookie=25a440366e571d2a server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:03:50.060 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:50.054+0000 7f3a5b7fe640 1 -- 172.21.15.16:0/2552382927 <== mon.1 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a480ab650 con 0x7f3a5c0cfa40 2023-05-22T16:03:50.060 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:50.054+0000 7f3a5b7fe640 1 -- 172.21.15.16:0/2552382927 <== mon.1 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a480ab7b0 con 0x7f3a5c0cfa40 2023-05-22T16:03:50.061 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:50.054+0000 7f3a5b7fe640 1 -- 172.21.15.16:0/2552382927 <== mon.1 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a480abb20 con 0x7f3a5c0cfa40 2023-05-22T16:03:50.061 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:50.054+0000 7f3a62bb0640 1 -- 172.21.15.16:0/2552382927 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3a5c0cfa40 msgr2=0x7f3a5c0cd260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:50.061 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:50.054+0000 7f3a62bb0640 1 --2- 172.21.15.16:0/2552382927 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3a5c0cfa40 0x7f3a5c0cd260 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f3a480097c0 tx=0x7f3a4801c450 comp rx=0 tx=0).stop 2023-05-22T16:03:50.061 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:50.054+0000 7f3a62bb0640 1 -- 172.21.15.16:0/2552382927 shutdown_connections 2023-05-22T16:03:50.061 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:50.054+0000 7f3a62bb0640 1 --2- 172.21.15.16:0/2552382927 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3a5c06a830 0x7f3a5c06acb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:50.061 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:50.054+0000 7f3a62bb0640 1 --2- 172.21.15.16:0/2552382927 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3a5c0cfa40 0x7f3a5c0cd260 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:50.061 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:50.054+0000 7f3a62bb0640 1 -- 172.21.15.16:0/2552382927 >> 172.21.15.16:0/2552382927 conn(0x7f3a5c0700b0 msgr2=0x7f3a5c06d9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:03:50.062 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:50.054+0000 7f3a62bb0640 1 -- 172.21.15.16:0/2552382927 shutdown_connections 2023-05-22T16:03:50.062 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:50.054+0000 7f3a62bb0640 1 -- 172.21.15.16:0/2552382927 wait complete. 2023-05-22T16:03:50.062 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:50.054+0000 7f3a62bb0640 1 Processor -- start 2023-05-22T16:03:50.062 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:50.054+0000 7f3a62bb0640 1 -- start start 2023-05-22T16:03:50.062 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:50.054+0000 7f3a62bb0640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3a5c06a830 0x7f3a5c14bbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:50.062 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:50.054+0000 7f3a62bb0640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3a5c0cf670 0x7f3a5c0c5d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:50.062 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:50.054+0000 7f3a62bb0640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3a5c0cfa40 0x7f3a5c14cd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:50.062 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:50.054+0000 7f3a62bb0640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a5c0c6da0 con 0x7f3a5c0cfa40 2023-05-22T16:03:50.063 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:50.054+0000 7f3a62bb0640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a5c0c67e0 con 0x7f3a5c06a830 2023-05-22T16:03:50.063 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:50.054+0000 7f3a62bb0640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a5c0c6ac0 con 0x7f3a5c0cf670 2023-05-22T16:03:50.063 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:50.054+0000 7f3a6114d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3a5c0cfa40 0x7f3a5c14cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:50.063 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:50.058+0000 7f3a6114d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3a5c0cfa40 0x7f3a5c14cd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59456/0 (socket says 172.21.15.16:59456) 2023-05-22T16:03:50.063 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:50.058+0000 7f3a6114d640 1 -- 172.21.15.16:0/3027364254 learned_addr learned my addr 172.21.15.16:0/3027364254 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:03:50.063 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:50.058+0000 7f3a6094c640 1 --2- 172.21.15.16:0/3027364254 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3a5c06a830 0x7f3a5c14bbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:50.063 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:50.058+0000 7f3a5bfff640 1 --2- 172.21.15.16:0/3027364254 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3a5c0cf670 0x7f3a5c0c5d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:50.064 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:50.058+0000 7f3a6114d640 1 -- 172.21.15.16:0/3027364254 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3a5c0cf670 msgr2=0x7f3a5c0c5d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:50.064 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:50.058+0000 7f3a6114d640 1 --2- 172.21.15.16:0/3027364254 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3a5c0cf670 0x7f3a5c0c5d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:50.064 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:50.058+0000 7f3a6114d640 1 -- 172.21.15.16:0/3027364254 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3a5c06a830 msgr2=0x7f3a5c14bbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:50.064 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:50.058+0000 7f3a6114d640 1 --2- 172.21.15.16:0/3027364254 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3a5c06a830 0x7f3a5c14bbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:50.064 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:50.058+0000 7f3a6114d640 1 -- 172.21.15.16:0/3027364254 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a48009420 con 0x7f3a5c0cfa40 2023-05-22T16:03:50.064 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:50.058+0000 7f3a6114d640 1 --2- 172.21.15.16:0/3027364254 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3a5c0cfa40 0x7f3a5c14cd90 secure :-1 s=READY pgs=2290 cs=0 l=1 rev1=1 crypto rx=0x7f3a5400efc0 tx=0x7f3a5400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:03:50.064 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:50.058+0000 7f3a597fa640 1 -- 172.21.15.16:0/3027364254 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a54014480 con 0x7f3a5c0cfa40 2023-05-22T16:03:50.064 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:50.058+0000 7f3a597fa640 1 -- 172.21.15.16:0/3027364254 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a5400f040 con 0x7f3a5c0cfa40 2023-05-22T16:03:50.065 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:50.058+0000 7f3a62bb0640 1 -- 172.21.15.16:0/3027364254 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a5c0cd7a0 con 0x7f3a5c0cfa40 2023-05-22T16:03:50.065 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:50.058+0000 7f3a597fa640 1 -- 172.21.15.16:0/3027364254 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a54014820 con 0x7f3a5c0cfa40 2023-05-22T16:03:50.065 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:50.058+0000 7f3a62bb0640 1 -- 172.21.15.16:0/3027364254 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a5c06c0e0 con 0x7f3a5c0cfa40 2023-05-22T16:03:50.065 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:50.058+0000 7f3a46ffd640 1 -- 172.21.15.16:0/3027364254 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a34005260 con 0x7f3a5c0cfa40 2023-05-22T16:03:50.065 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:50.058+0000 7f3a597fa640 1 -- 172.21.15.16:0/3027364254 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 83) v1 ==== 84349+0+0 (secure 0 0 0) 0x7f3a54014980 con 0x7f3a5c0cfa40 2023-05-22T16:03:50.065 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:50.058+0000 7f3a597fa640 1 --2- 172.21.15.16:0/3027364254 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f3a400654f0 0x7f3a400679a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:50.065 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:50.058+0000 7f3a6094c640 1 -- 172.21.15.16:0/3027364254 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f3a400654f0 msgr2=0x7f3a400679a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/65890 2023-05-22T16:03:50.066 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:50.058+0000 7f3a597fa640 1 -- 172.21.15.16:0/3027364254 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(179..179 src has 1..179) v4 ==== 6438+0+0 (secure 0 0 0) 0x7f3a540837b0 con 0x7f3a5c0cfa40 2023-05-22T16:03:50.066 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:50.058+0000 7f3a6094c640 1 --2- 172.21.15.16:0/3027364254 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f3a400654f0 0x7f3a400679a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-05-22T16:03:50.068 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:50.070+0000 7f3a597fa640 1 -- 172.21.15.16:0/3027364254 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f3a540510c0 con 0x7f3a5c0cfa40 2023-05-22T16:03:50.237 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:50.238+0000 7f3a46ffd640 1 -- 172.21.15.16:0/3027364254 --> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "snap_test_dir1", "snap_schedule": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f3a34000bd0 con 0x7f3a400654f0 2023-05-22T16:03:50.260 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:50.262+0000 7f3a6094c640 1 -- 172.21.15.16:0/3027364254 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f3a400654f0 msgr2=0x7f3a400679a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/65890 2023-05-22T16:03:50.261 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:50.262+0000 7f3a6094c640 1 --2- 172.21.15.16:0/3027364254 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f3a400654f0 0x7f3a400679a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-05-22T16:03:50.275 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T16:03:50.273+0000 7f9e9009b180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-05-22T16:03:50.323 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T16:03:50.322+0000 7f2207031180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-05-22T16:03:50.328 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T16:03:50.325+0000 7f9e9009b180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-05-22T16:03:50.376 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T16:03:50.374+0000 7f2207031180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-05-22T16:03:50.378 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T16:03:50.373+0000 7f9e9009b180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-05-22T16:03:50.427 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T16:03:50.426+0000 7f2207031180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-05-22T16:03:50.455 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T16:03:50.453+0000 7f9e9009b180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-05-22T16:03:50.505 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T16:03:50.506+0000 7f2207031180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-05-22T16:03:50.506 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T16:03:50.501+0000 7f9e9009b180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-05-22T16:03:50.557 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T16:03:50.558+0000 7f2207031180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-05-22T16:03:50.582 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T16:03:50.577+0000 7f9e9009b180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-05-22T16:03:50.635 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T16:03:50.634+0000 7f2207031180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-05-22T16:03:50.635 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T16:03:50.629+0000 7f9e9009b180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-05-22T16:03:50.664 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:50.666+0000 7f3a6094c640 1 -- 172.21.15.16:0/3027364254 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f3a400654f0 msgr2=0x7f3a400679a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/65890 2023-05-22T16:03:50.665 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:50.666+0000 7f3a6094c640 1 --2- 172.21.15.16:0/3027364254 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f3a400654f0 0x7f3a400679a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-05-22T16:03:50.687 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T16:03:50.686+0000 7f2207031180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-05-22T16:03:50.766 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T16:03:50.761+0000 7f9e9009b180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-05-22T16:03:50.819 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T16:03:50.817+0000 7f9e9009b180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-05-22T16:03:50.821 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T16:03:50.822+0000 7f2207031180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-05-22T16:03:50.874 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T16:03:50.874+0000 7f2207031180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-05-22T16:03:51.042 INFO:tasks.ceph.mgr.x.smithi062.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2023-05-22T16:03:51.043 INFO:tasks.ceph.mgr.x.smithi062.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2023-05-22T16:03:51.043 INFO:tasks.ceph.mgr.x.smithi062.stderr: from numpy import show_config as show_numpy_config 2023-05-22T16:03:51.050 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T16:03:51.045+0000 7f9e9009b180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-05-22T16:03:51.099 INFO:tasks.ceph.mgr.y.smithi016.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2023-05-22T16:03:51.099 INFO:tasks.ceph.mgr.y.smithi016.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2023-05-22T16:03:51.099 INFO:tasks.ceph.mgr.y.smithi016.stderr: from numpy import show_config as show_numpy_config 2023-05-22T16:03:51.107 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T16:03:51.106+0000 7f2207031180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-05-22T16:03:51.175 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T16:03:51.173+0000 7f9e9009b180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-05-22T16:03:51.235 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T16:03:51.234+0000 7f2207031180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-05-22T16:03:51.289 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T16:03:51.285+0000 7f9e9009b180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-05-22T16:03:51.348 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T16:03:51.346+0000 7f2207031180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-05-22T16:03:51.351 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T16:03:51.349+0000 7f9e9009b180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-05-22T16:03:51.412 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T16:03:51.410+0000 7f2207031180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-05-22T16:03:51.468 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:51.470+0000 7f3a6094c640 1 -- 172.21.15.16:0/3027364254 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f3a400654f0 msgr2=0x7f3a400679a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/65890 2023-05-22T16:03:51.469 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:51.470+0000 7f3a6094c640 1 --2- 172.21.15.16:0/3027364254 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f3a400654f0 0x7f3a400679a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-05-22T16:03:51.475 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T16:03:51.473+0000 7f9e9009b180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-05-22T16:03:51.540 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T16:03:51.538+0000 7f2207031180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-05-22T16:03:51.966 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T16:03:51.965+0000 7f9e9009b180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-05-22T16:03:52.038 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T16:03:52.038+0000 7f2207031180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-05-22T16:03:52.042 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T16:03:52.037+0000 7f9e9009b180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-05-22T16:03:52.114 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T16:03:52.114+0000 7f2207031180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-05-22T16:03:52.179 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T16:03:52.177+0000 7f9e9009b180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-05-22T16:03:52.254 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T16:03:52.254+0000 7f2207031180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-05-22T16:03:52.255 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T16:03:52.253+0000 7f9e9009b180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-05-22T16:03:52.330 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T16:03:52.330+0000 7f2207031180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-05-22T16:03:52.501 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T16:03:52.497+0000 7f9e9009b180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-05-22T16:03:52.556 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T16:03:52.553+0000 7f9e9009b180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-05-22T16:03:52.580 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T16:03:52.578+0000 7f2207031180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-05-22T16:03:52.635 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T16:03:52.634+0000 7f2207031180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-05-22T16:03:52.638 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:52.638+0000 7f3a597fa640 1 -- 172.21.15.16:0/3027364254 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mgrmap(e 84) v1 ==== 84349+0+0 (secure 0 0 0) 0x7f3a540819f0 con 0x7f3a5c0cfa40 2023-05-22T16:03:52.670 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:52.670+0000 7f3a597fa640 1 -- 172.21.15.16:0/3027364254 <== mon.0 v2:172.21.15.16:3300/0 8 ==== mgrmap(e 85) v1 ==== 83958+0+0 (secure 0 0 0) 0x7f3a54051490 con 0x7f3a5c0cfa40 2023-05-22T16:03:52.671 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:52.670+0000 7f3a597fa640 1 -- 172.21.15.16:0/3027364254 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f3a400654f0 msgr2=0x7f3a400679a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-05-22T16:03:52.671 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:52.670+0000 7f3a597fa640 1 --2- 172.21.15.16:0/3027364254 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f3a400654f0 0x7f3a400679a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:52.856 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T16:03:52.854+0000 7f21892ae640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T16:03:52.857 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T16:03:52.854+0000 7f21892ae640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T16:03:52.857 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T16:03:52.854+0000 7f21892ae640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T16:03:52.858 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T16:03:52.854+0000 7f21892ae640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T16:03:52.858 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T16:03:52.854+0000 7f21892ae640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T16:03:52.858 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T16:03:52.854+0000 7f2184aa5640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T16:03:52.858 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T16:03:52.854+0000 7f2184aa5640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T16:03:52.859 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T16:03:52.854+0000 7f2184aa5640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T16:03:52.859 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T16:03:52.854+0000 7f2184aa5640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T16:03:52.859 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T16:03:52.854+0000 7f2184aa5640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T16:03:53.678 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:53.677+0000 7f3a597fa640 1 -- 172.21.15.16:0/3027364254 <== mon.0 v2:172.21.15.16:3300/0 9 ==== mgrmap(e 86) v1 ==== 84085+0+0 (secure 0 0 0) 0x7f3a54051d60 con 0x7f3a5c0cfa40 2023-05-22T16:03:53.679 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:53.677+0000 7f3a597fa640 1 --2- 172.21.15.16:0/3027364254 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f3a400654f0 0x7f3a40067920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:53.679 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:53.677+0000 7f3a597fa640 1 -- 172.21.15.16:0/3027364254 --> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "snap_test_dir1", "snap_schedule": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f3a34000bd0 con 0x7f3a400654f0 2023-05-22T16:03:53.683 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:53.681+0000 7f3a6094c640 1 --2- 172.21.15.16:0/3027364254 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f3a400654f0 0x7f3a40067920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:53.686 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:53.685+0000 7f3a6094c640 1 --2- 172.21.15.16:0/3027364254 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f3a400654f0 0x7f3a40067920 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f3a50005890 tx=0x7f3a50007b80 comp rx=0 tx=0).ready entity=mgr.8746 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:03:53.831 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:53.829+0000 7f3a597fa640 1 -- 172.21.15.16:0/3027364254 <== mgr.8746 v2:172.21.15.16:6832/65890 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f3a34000bd0 con 0x7f3a400654f0 2023-05-22T16:03:53.832 INFO:teuthology.orchestra.run.smithi016.stdout:Schedule set for path snap_test_dir1 2023-05-22T16:03:53.835 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:53.833+0000 7f3a46ffd640 1 -- 172.21.15.16:0/3027364254 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f3a400654f0 msgr2=0x7f3a40067920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:53.835 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:53.833+0000 7f3a46ffd640 1 --2- 172.21.15.16:0/3027364254 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f3a400654f0 0x7f3a40067920 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f3a50005890 tx=0x7f3a50007b80 comp rx=0 tx=0).stop 2023-05-22T16:03:53.835 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:53.833+0000 7f3a46ffd640 1 -- 172.21.15.16:0/3027364254 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3a5c0cfa40 msgr2=0x7f3a5c14cd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:53.835 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:53.833+0000 7f3a46ffd640 1 --2- 172.21.15.16:0/3027364254 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3a5c0cfa40 0x7f3a5c14cd90 secure :-1 s=READY pgs=2290 cs=0 l=1 rev1=1 crypto rx=0x7f3a5400efc0 tx=0x7f3a5400c490 comp rx=0 tx=0).stop 2023-05-22T16:03:53.836 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:53.833+0000 7f3a46ffd640 1 -- 172.21.15.16:0/3027364254 shutdown_connections 2023-05-22T16:03:53.836 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:53.833+0000 7f3a46ffd640 1 --2- 172.21.15.16:0/3027364254 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f3a400654f0 0x7f3a40067920 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:53.836 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:53.833+0000 7f3a46ffd640 1 --2- 172.21.15.16:0/3027364254 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3a5c0cfa40 0x7f3a5c14cd90 unknown :-1 s=CLOSED pgs=2290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:53.836 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:53.833+0000 7f3a46ffd640 1 --2- 172.21.15.16:0/3027364254 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3a5c0cf670 0x7f3a5c0c5d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:53.836 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:53.833+0000 7f3a46ffd640 1 --2- 172.21.15.16:0/3027364254 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3a5c06a830 0x7f3a5c14bbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:53.836 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:53.833+0000 7f3a46ffd640 1 -- 172.21.15.16:0/3027364254 >> 172.21.15.16:0/3027364254 conn(0x7f3a5c0700b0 msgr2=0x7f3a5c1578d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:03:53.836 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:53.833+0000 7f3a46ffd640 1 -- 172.21.15.16:0/3027364254 shutdown_connections 2023-05-22T16:03:53.837 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:53.833+0000 7f3a46ffd640 1 -- 172.21.15.16:0/3027364254 wait complete. 2023-05-22T16:03:53.855 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1 2023-05-22T16:03:53.855 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule retention add --fs cephfs snap_test_dir1 1M 2023-05-22T16:03:53.985 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:53.985+0000 7fc6611b4640 1 Processor -- start 2023-05-22T16:03:53.985 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:53.985+0000 7fc6611b4640 1 -- start start 2023-05-22T16:03:53.988 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:53.985+0000 7fc6611b4640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc65c0ce480 0x7fc65c071640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:53.988 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:53.985+0000 7fc6611b4640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fc65c15c270 0x7fc65c15c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:53.988 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:53.985+0000 7fc6611b4640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc65c069ae0 con 0x7fc65c0ce850 2023-05-22T16:03:53.988 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:53.985+0000 7fc6611b4640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc65c15f990 con 0x7fc65c15c270 2023-05-22T16:03:53.989 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:53.985+0000 7fc6611b4640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc65c069dc0 con 0x7fc65c0ce480 2023-05-22T16:03:53.989 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:53.985+0000 7fc65a59c640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fc65c0ce850 0x7fc65c160090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:53950/0 (socket says 172.21.15.16:53950) 2023-05-22T16:03:53.989 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:53.985+0000 7fc65a59c640 1 -- 172.21.15.16:0/255470791 learned_addr learned my addr 172.21.15.16:0/255470791 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T16:03:53.989 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:53.985+0000 7fc65b59e640 1 --2- 172.21.15.16:0/255470791 >> v2:172.21.15.62:3300/0 conn(0x7fc65c15c270 0x7fc65c15c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:53.989 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:53.985+0000 7fc65ad9d640 1 --2- 172.21.15.16:0/255470791 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc65c0ce480 0x7fc65c071640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:53.989 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:53.985+0000 7fc659d9b640 1 -- 172.21.15.16:0/255470791 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 701594406 0 0) 0x7fc65c069ae0 con 0x7fc65c0ce850 2023-05-22T16:03:53.989 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:53.985+0000 7fc659d9b640 1 -- 172.21.15.16:0/255470791 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc644003530 con 0x7fc65c0ce850 2023-05-22T16:03:53.990 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:53.985+0000 7fc659d9b640 1 -- 172.21.15.16:0/255470791 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3425721030 0 0) 0x7fc644003530 con 0x7fc65c0ce850 2023-05-22T16:03:53.990 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:53.985+0000 7fc659d9b640 1 -- 172.21.15.16:0/255470791 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc65c0ce480 msgr2=0x7fc65c071640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:03:53.990 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:53.985+0000 7fc659d9b640 1 --2- 172.21.15.16:0/255470791 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc65c0ce480 0x7fc65c071640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:53.990 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:53.985+0000 7fc659d9b640 1 -- 172.21.15.16:0/255470791 >> v2:172.21.15.62:3300/0 conn(0x7fc65c15c270 msgr2=0x7fc65c15c6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:03:53.990 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:53.985+0000 7fc659d9b640 1 --2- 172.21.15.16:0/255470791 >> v2:172.21.15.62:3300/0 conn(0x7fc65c15c270 0x7fc65c15c6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:53.990 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:53.985+0000 7fc659d9b640 1 -- 172.21.15.16:0/255470791 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc65c06a0a0 con 0x7fc65c0ce850 2023-05-22T16:03:53.990 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:53.985+0000 7fc659d9b640 1 -- 172.21.15.16:0/255470791 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7fc650001680 con 0x7fc65c0ce850 2023-05-22T16:03:53.990 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:53.985+0000 7fc659d9b640 1 -- 172.21.15.16:0/255470791 >> v1:172.21.15.16:6789/0 conn(0x7fc65c0ce850 legacy=0x7fc65c160090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:53.991 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:53.985+0000 7fc659d9b640 1 --2- 172.21.15.16:0/255470791 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc644003980 0x7fc644003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:53.991 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:53.985+0000 7fc659d9b640 1 --2- 172.21.15.16:0/255470791 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc65c0ce480 0x7fc6440047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:53.991 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:53.985+0000 7fc659d9b640 1 --2- 172.21.15.16:0/255470791 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc644004d10 0x7fc644005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:53.991 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:53.985+0000 7fc659d9b640 1 -- 172.21.15.16:0/255470791 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc65c15f990 con 0x7fc644003980 2023-05-22T16:03:53.991 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:53.985+0000 7fc659d9b640 1 -- 172.21.15.16:0/255470791 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc65c069dc0 con 0x7fc644004d10 2023-05-22T16:03:53.991 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:53.985+0000 7fc659d9b640 1 -- 172.21.15.16:0/255470791 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc64400b990 con 0x7fc65c0ce480 2023-05-22T16:03:53.991 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:53.985+0000 7fc65ad9d640 1 --2- 172.21.15.16:0/255470791 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc644003980 0x7fc644003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:53.992 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:53.985+0000 7fc65a59c640 1 --2- 172.21.15.16:0/255470791 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc65c0ce480 0x7fc6440047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:53.992 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:53.985+0000 7fc65b59e640 1 --2- 172.21.15.16:0/255470791 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc644004d10 0x7fc644005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:53.992 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:53.985+0000 7fc65ad9d640 1 -- 172.21.15.16:0/255470791 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc65c0ce480 msgr2=0x7fc6440047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:03:53.992 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:53.985+0000 7fc65ad9d640 1 --2- 172.21.15.16:0/255470791 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc65c0ce480 0x7fc6440047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:53.992 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:53.985+0000 7fc65ad9d640 1 -- 172.21.15.16:0/255470791 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc644004d10 msgr2=0x7fc644005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:03:53.992 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:53.985+0000 7fc65ad9d640 1 --2- 172.21.15.16:0/255470791 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc644004d10 0x7fc644005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:53.992 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:53.985+0000 7fc65ad9d640 1 -- 172.21.15.16:0/255470791 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc644003530 con 0x7fc644003980 2023-05-22T16:03:53.992 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:53.985+0000 7fc65b59e640 1 --2- 172.21.15.16:0/255470791 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc644004d10 0x7fc644005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T16:03:53.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:53.989+0000 7fc65ad9d640 1 --2- 172.21.15.16:0/255470791 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc644003980 0x7fc644003d90 secure :-1 s=READY pgs=2307 cs=0 l=1 rev1=1 crypto rx=0x7fc64c009ae0 tx=0x7fc64c01c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4cab3208c4182532 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:03:53.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:53.989+0000 7fc659d9b640 1 -- 172.21.15.16:0/255470791 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc64c0a6510 con 0x7fc644003980 2023-05-22T16:03:53.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:53.989+0000 7fc659d9b640 1 -- 172.21.15.16:0/255470791 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc64c0a6670 con 0x7fc644003980 2023-05-22T16:03:53.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:53.989+0000 7fc659d9b640 1 -- 172.21.15.16:0/255470791 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc64c0a6a20 con 0x7fc644003980 2023-05-22T16:03:53.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:53.989+0000 7fc6611b4640 1 -- 172.21.15.16:0/255470791 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc644003980 msgr2=0x7fc644003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:53.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:53.989+0000 7fc6611b4640 1 --2- 172.21.15.16:0/255470791 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc644003980 0x7fc644003d90 secure :-1 s=READY pgs=2307 cs=0 l=1 rev1=1 crypto rx=0x7fc64c009ae0 tx=0x7fc64c01c9c0 comp rx=0 tx=0).stop 2023-05-22T16:03:53.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:53.989+0000 7fc65b59e640 1 -- 172.21.15.16:0/255470791 reap_dead start 2023-05-22T16:03:53.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:53.989+0000 7fc6611b4640 1 -- 172.21.15.16:0/255470791 shutdown_connections 2023-05-22T16:03:53.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:53.989+0000 7fc6611b4640 1 -- 172.21.15.16:0/255470791 >> 172.21.15.16:0/255470791 conn(0x7fc65c0cc870 msgr2=0x7fc65c0cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:03:53.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:53.989+0000 7fc6611b4640 1 -- 172.21.15.16:0/255470791 shutdown_connections 2023-05-22T16:03:53.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:53.989+0000 7fc6611b4640 1 -- 172.21.15.16:0/255470791 wait complete. 2023-05-22T16:03:53.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:53.993+0000 7fc6611b4640 1 Processor -- start 2023-05-22T16:03:53.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:53.993+0000 7fc6611b4640 1 -- start start 2023-05-22T16:03:53.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:53.993+0000 7fc6611b4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc644003980 0x7fc65c1f8920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:53.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:53.993+0000 7fc6611b4640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc65c0b4be0 0x7fc65c164f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:53.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:53.993+0000 7fc6611b4640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc65c1575e0 0x7fc65c0ca450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:53.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:53.993+0000 7fc6611b4640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc65c1f8500 con 0x7fc644003980 2023-05-22T16:03:53.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:53.993+0000 7fc6611b4640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc65c165750 con 0x7fc65c0b4be0 2023-05-22T16:03:53.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:53.993+0000 7fc6611b4640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc65c1f7e40 con 0x7fc65c1575e0 2023-05-22T16:03:53.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:53.993+0000 7fc65ad9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc644003980 0x7fc65c1f8920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:53.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:53.993+0000 7fc65ad9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc644003980 0x7fc65c1f8920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59610/0 (socket says 172.21.15.16:59610) 2023-05-22T16:03:53.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:53.993+0000 7fc65ad9d640 1 -- 172.21.15.16:0/1646047261 learned_addr learned my addr 172.21.15.16:0/1646047261 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:03:53.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:53.993+0000 7fc65b59e640 1 --2- 172.21.15.16:0/1646047261 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc65c1575e0 0x7fc65c0ca450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:53.998 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:53.993+0000 7fc65a59c640 1 --2- 172.21.15.16:0/1646047261 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc65c0b4be0 0x7fc65c164f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:53.998 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:53.993+0000 7fc65ad9d640 1 -- 172.21.15.16:0/1646047261 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc65c1575e0 msgr2=0x7fc65c0ca450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:53.998 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:53.993+0000 7fc65ad9d640 1 --2- 172.21.15.16:0/1646047261 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc65c1575e0 0x7fc65c0ca450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:53.998 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:53.993+0000 7fc65ad9d640 1 -- 172.21.15.16:0/1646047261 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc65c0b4be0 msgr2=0x7fc65c164f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:53.998 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:53.993+0000 7fc65ad9d640 1 --2- 172.21.15.16:0/1646047261 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc65c0b4be0 0x7fc65c164f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:53.998 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:53.993+0000 7fc65ad9d640 1 -- 172.21.15.16:0/1646047261 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc65c0698b0 con 0x7fc644003980 2023-05-22T16:03:53.998 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:53.993+0000 7fc65a59c640 1 --2- 172.21.15.16:0/1646047261 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc65c0b4be0 0x7fc65c164f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T16:03:53.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:53.993+0000 7fc65ad9d640 1 --2- 172.21.15.16:0/1646047261 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc644003980 0x7fc65c1f8920 secure :-1 s=READY pgs=2308 cs=0 l=1 rev1=1 crypto rx=0x7fc64c0b0040 tx=0x7fc644005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:03:53.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:53.993+0000 7fc6437fe640 1 -- 172.21.15.16:0/1646047261 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc64c0a67e0 con 0x7fc644003980 2023-05-22T16:03:53.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:53.993+0000 7fc6437fe640 1 -- 172.21.15.16:0/1646047261 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc64c0a6940 con 0x7fc644003980 2023-05-22T16:03:53.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:53.993+0000 7fc6611b4640 1 -- 172.21.15.16:0/1646047261 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc65c069720 con 0x7fc644003980 2023-05-22T16:03:53.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:53.993+0000 7fc6437fe640 1 -- 172.21.15.16:0/1646047261 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc64c0a6cf0 con 0x7fc644003980 2023-05-22T16:03:53.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:53.993+0000 7fc6611b4640 1 -- 172.21.15.16:0/1646047261 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc65c15cc30 con 0x7fc644003980 2023-05-22T16:03:54.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:53.993+0000 7fc6611b4640 1 -- 172.21.15.16:0/1646047261 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc65c1f8770 con 0x7fc644003980 2023-05-22T16:03:54.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:53.997+0000 7fc6437fe640 1 -- 172.21.15.16:0/1646047261 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 86) v1 ==== 84085+0+0 (secure 0 0 0) 0x7fc64c006040 con 0x7fc644003980 2023-05-22T16:03:54.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:53.997+0000 7fc6437fe640 1 --2- 172.21.15.16:0/1646047261 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fc648065250 0x7fc648067700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:54.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:53.997+0000 7fc6437fe640 1 -- 172.21.15.16:0/1646047261 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(180..180 src has 1..180) v4 ==== 6782+0+0 (secure 0 0 0) 0x7fc64c0aa070 con 0x7fc644003980 2023-05-22T16:03:54.001 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:53.997+0000 7fc65a59c640 1 --2- 172.21.15.16:0/1646047261 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fc648065250 0x7fc648067700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:54.001 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:53.997+0000 7fc65a59c640 1 --2- 172.21.15.16:0/1646047261 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fc648065250 0x7fc648067700 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fc650001060 tx=0x7fc650009040 comp rx=0 tx=0).ready entity=mgr.8746 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:03:54.001 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:54.001+0000 7fc6437fe640 1 -- 172.21.15.16:0/1646047261 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fc64c0eefc0 con 0x7fc644003980 2023-05-22T16:03:54.171 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:54.169+0000 7fc6611b4640 1 -- 172.21.15.16:0/1646047261 --> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] -- mgr_command(tid 0: {"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "snap_test_dir1", "retention_spec_or_period": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7fc65c04dbe0 con 0x7fc648065250 2023-05-22T16:03:54.299 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:54.297+0000 7fc6437fe640 1 -- 172.21.15.16:0/1646047261 <== mgr.8746 v2:172.21.15.16:6832/65890 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+39 (secure 0 0 0) 0x7fc65c04dbe0 con 0x7fc648065250 2023-05-22T16:03:54.299 INFO:teuthology.orchestra.run.smithi016.stdout:Retention added to path snap_test_dir1 2023-05-22T16:03:54.301 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:54.301+0000 7fc6611b4640 1 -- 172.21.15.16:0/1646047261 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fc648065250 msgr2=0x7fc648067700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:54.302 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:54.301+0000 7fc6611b4640 1 --2- 172.21.15.16:0/1646047261 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fc648065250 0x7fc648067700 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fc650001060 tx=0x7fc650009040 comp rx=0 tx=0).stop 2023-05-22T16:03:54.302 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:54.301+0000 7fc6611b4640 1 -- 172.21.15.16:0/1646047261 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc644003980 msgr2=0x7fc65c1f8920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:54.302 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:54.301+0000 7fc6611b4640 1 --2- 172.21.15.16:0/1646047261 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc644003980 0x7fc65c1f8920 secure :-1 s=READY pgs=2308 cs=0 l=1 rev1=1 crypto rx=0x7fc64c0b0040 tx=0x7fc644005800 comp rx=0 tx=0).stop 2023-05-22T16:03:54.304 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:54.301+0000 7fc6611b4640 1 -- 172.21.15.16:0/1646047261 shutdown_connections 2023-05-22T16:03:54.304 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:54.301+0000 7fc6611b4640 1 --2- 172.21.15.16:0/1646047261 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc65c1575e0 0x7fc65c0ca450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:54.304 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:54.301+0000 7fc6611b4640 1 --2- 172.21.15.16:0/1646047261 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc65c0b4be0 0x7fc65c164f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:54.304 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:54.301+0000 7fc6611b4640 1 --2- 172.21.15.16:0/1646047261 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fc648065250 0x7fc648067700 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:54.304 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:54.301+0000 7fc6611b4640 1 --2- 172.21.15.16:0/1646047261 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc644003980 0x7fc65c1f8920 unknown :-1 s=CLOSED pgs=2308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:54.304 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:54.301+0000 7fc6611b4640 1 -- 172.21.15.16:0/1646047261 >> 172.21.15.16:0/1646047261 conn(0x7fc65c0cc870 msgr2=0x7fc65c0c99b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:03:54.305 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:54.301+0000 7fc6611b4640 1 -- 172.21.15.16:0/1646047261 shutdown_connections 2023-05-22T16:03:54.305 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:54.301+0000 7fc6611b4640 1 -- 172.21.15.16:0/1646047261 wait complete. 2023-05-22T16:03:54.321 DEBUG:tasks.cephfs.test_snap_schedules:res=Retention added to path snap_test_dir1 2023-05-22T16:03:54.321 DEBUG:tasks.cephfs.test_snap_schedules:expecting snap snap_test_dir1/.snap/scheduled-2023-05-22-16_04_00 in ~6s... 2023-05-22T16:03:54.322 DEBUG:tasks.cephfs.test_snap_schedules:expected_schedule: ['1M'], expected_retention: [{'M': 1}] 2023-05-22T16:03:54.322 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule list --fs cephfs --format json snap_test_dir1 2023-05-22T16:03:54.450 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:54.449+0000 7f3760058640 1 Processor -- start 2023-05-22T16:03:54.450 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:54.449+0000 7f3760058640 1 -- start start 2023-05-22T16:03:54.452 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:54.449+0000 7f3760058640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f375815c8e0 0x7f37580ccc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:54.452 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:54.449+0000 7f3760058640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f37580d2eb0 0x7f375815ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:54.452 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:54.449+0000 7f3760058640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37580d2a20 con 0x7f37580cef10 2023-05-22T16:03:54.452 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:54.449+0000 7f3760058640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37580cd230 con 0x7f375815c8e0 2023-05-22T16:03:54.452 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:54.449+0000 7f3760058640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37580d23d0 con 0x7f37580d2eb0 2023-05-22T16:03:54.452 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:54.449+0000 7f375ddf4640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f375815c8e0 0x7f37580ccc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:54.453 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:54.449+0000 7f375ddf4640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f375815c8e0 0x7f37580ccc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59618/0 (socket says 172.21.15.16:59618) 2023-05-22T16:03:54.453 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:54.449+0000 7f375ddf4640 1 -- 172.21.15.16:0/2768969768 learned_addr learned my addr 172.21.15.16:0/2768969768 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:03:54.453 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:54.449+0000 7f375d5f3640 1 --2- 172.21.15.16:0/2768969768 >> v2:172.21.15.62:3300/0 conn(0x7f37580d2eb0 0x7f375815ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:54.453 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:54.449+0000 7f375ddf4640 1 -- 172.21.15.16:0/2768969768 >> v2:172.21.15.62:3300/0 conn(0x7f37580d2eb0 msgr2=0x7f375815ccb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:03:54.453 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:54.449+0000 7f375ddf4640 1 --2- 172.21.15.16:0/2768969768 >> v2:172.21.15.62:3300/0 conn(0x7f37580d2eb0 0x7f375815ccb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:54.453 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:54.449+0000 7f375ddf4640 1 -- 172.21.15.16:0/2768969768 >> v1:172.21.15.16:6789/0 conn(0x7f37580cef10 legacy=0x7f37580cf310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:54.453 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:54.449+0000 7f375ddf4640 1 -- 172.21.15.16:0/2768969768 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37580d26b0 con 0x7f375815c8e0 2023-05-22T16:03:54.455 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:54.453+0000 7f375ddf4640 1 --2- 172.21.15.16:0/2768969768 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f375815c8e0 0x7f37580ccc50 secure :-1 s=READY pgs=2310 cs=0 l=1 rev1=1 crypto rx=0x7f37580d1ad0 tx=0x7f375401c470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d524b42964d6a19 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:03:54.455 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:54.453+0000 7f375cdf2640 1 -- 172.21.15.16:0/2768969768 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37540ab610 con 0x7f375815c8e0 2023-05-22T16:03:54.455 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:54.453+0000 7f375cdf2640 1 -- 172.21.15.16:0/2768969768 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37540ab770 con 0x7f375815c8e0 2023-05-22T16:03:54.455 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:54.453+0000 7f375cdf2640 1 -- 172.21.15.16:0/2768969768 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37540abb00 con 0x7f375815c8e0 2023-05-22T16:03:54.456 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:54.453+0000 7f3760058640 1 -- 172.21.15.16:0/2768969768 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f375815c8e0 msgr2=0x7f37580ccc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:54.456 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:54.453+0000 7f3760058640 1 --2- 172.21.15.16:0/2768969768 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f375815c8e0 0x7f37580ccc50 secure :-1 s=READY pgs=2310 cs=0 l=1 rev1=1 crypto rx=0x7f37580d1ad0 tx=0x7f375401c470 comp rx=0 tx=0).stop 2023-05-22T16:03:54.456 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:54.453+0000 7f3760058640 1 -- 172.21.15.16:0/2768969768 shutdown_connections 2023-05-22T16:03:54.457 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:54.453+0000 7f3760058640 1 --2- 172.21.15.16:0/2768969768 >> v2:172.21.15.62:3300/0 conn(0x7f37580d2eb0 0x7f375815ccb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:54.457 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:54.453+0000 7f3760058640 1 --2- 172.21.15.16:0/2768969768 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f375815c8e0 0x7f37580ccc50 unknown :-1 s=CLOSED pgs=2310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:54.457 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:54.453+0000 7f3760058640 1 -- 172.21.15.16:0/2768969768 >> 172.21.15.16:0/2768969768 conn(0x7f3758070e50 msgr2=0x7f3758071250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:03:54.457 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:54.453+0000 7f3760058640 1 -- 172.21.15.16:0/2768969768 shutdown_connections 2023-05-22T16:03:54.457 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:54.453+0000 7f3760058640 1 -- 172.21.15.16:0/2768969768 wait complete. 2023-05-22T16:03:54.458 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:54.457+0000 7f3760058640 1 Processor -- start 2023-05-22T16:03:54.458 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:54.457+0000 7f3760058640 1 -- start start 2023-05-22T16:03:54.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:54.457+0000 7f3760058640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f37580cef10 0x7f37581600c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:54.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:54.457+0000 7f3760058640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f37580d2eb0 0x7f375813c380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:54.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:54.457+0000 7f3760058640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f375815c8e0 0x7f37580b6480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:54.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:54.457+0000 7f3760058640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3758153560 con 0x7f375815c8e0 2023-05-22T16:03:54.460 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:54.457+0000 7f3760058640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3758153840 con 0x7f37580d2eb0 2023-05-22T16:03:54.460 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:54.457+0000 7f3760058640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3758153b20 con 0x7f37580cef10 2023-05-22T16:03:54.460 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:54.457+0000 7f375e5f5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f375815c8e0 0x7f37580b6480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:54.460 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:54.457+0000 7f375e5f5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f375815c8e0 0x7f37580b6480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59634/0 (socket says 172.21.15.16:59634) 2023-05-22T16:03:54.460 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:54.457+0000 7f375e5f5640 1 -- 172.21.15.16:0/304713306 learned_addr learned my addr 172.21.15.16:0/304713306 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:03:54.460 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:54.457+0000 7f375ddf4640 1 --2- 172.21.15.16:0/304713306 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f37580cef10 0x7f37581600c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:54.460 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:54.457+0000 7f375d5f3640 1 --2- 172.21.15.16:0/304713306 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f37580d2eb0 0x7f375813c380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:54.461 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:54.457+0000 7f375e5f5640 1 -- 172.21.15.16:0/304713306 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f37580cef10 msgr2=0x7f37581600c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:54.461 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:54.457+0000 7f375e5f5640 1 --2- 172.21.15.16:0/304713306 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f37580cef10 0x7f37581600c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:54.461 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:54.457+0000 7f375e5f5640 1 -- 172.21.15.16:0/304713306 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f37580d2eb0 msgr2=0x7f375813c380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:54.461 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:54.457+0000 7f375e5f5640 1 --2- 172.21.15.16:0/304713306 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f37580d2eb0 0x7f375813c380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:54.461 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:54.457+0000 7f375e5f5640 1 -- 172.21.15.16:0/304713306 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37540094d0 con 0x7f375815c8e0 2023-05-22T16:03:54.461 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:54.457+0000 7f375e5f5640 1 --2- 172.21.15.16:0/304713306 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f375815c8e0 0x7f37580b6480 secure :-1 s=READY pgs=2311 cs=0 l=1 rev1=1 crypto rx=0x7f375000c770 tx=0x7f375000cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:03:54.461 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:54.457+0000 7f374e7fc640 1 -- 172.21.15.16:0/304713306 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3750003b80 con 0x7f375815c8e0 2023-05-22T16:03:54.461 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:54.457+0000 7f374e7fc640 1 -- 172.21.15.16:0/304713306 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3750003ce0 con 0x7f375815c8e0 2023-05-22T16:03:54.462 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:54.457+0000 7f374e7fc640 1 -- 172.21.15.16:0/304713306 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f375000aa90 con 0x7f375815c8e0 2023-05-22T16:03:54.462 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:54.457+0000 7f3760058640 1 -- 172.21.15.16:0/304713306 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37580d3410 con 0x7f375815c8e0 2023-05-22T16:03:54.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:54.457+0000 7f3760058640 1 -- 172.21.15.16:0/304713306 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37580d3280 con 0x7f375815c8e0 2023-05-22T16:03:54.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:54.461+0000 7f3760058640 1 -- 172.21.15.16:0/304713306 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37580d0bf0 con 0x7f375815c8e0 2023-05-22T16:03:54.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:54.461+0000 7f374e7fc640 1 -- 172.21.15.16:0/304713306 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 86) v1 ==== 84085+0+0 (secure 0 0 0) 0x7f375000abf0 con 0x7f375815c8e0 2023-05-22T16:03:54.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:54.461+0000 7f374e7fc640 1 --2- 172.21.15.16:0/304713306 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f373c065220 0x7f373c0676d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:03:54.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:54.461+0000 7f374e7fc640 1 -- 172.21.15.16:0/304713306 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(180..180 src has 1..180) v4 ==== 6782+0+0 (secure 0 0 0) 0x7f37500826b0 con 0x7f375815c8e0 2023-05-22T16:03:54.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:54.461+0000 7f375ddf4640 1 --2- 172.21.15.16:0/304713306 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f373c065220 0x7f373c0676d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:03:54.464 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:54.461+0000 7f375ddf4640 1 --2- 172.21.15.16:0/304713306 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f373c065220 0x7f373c0676d0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f37540af000 tx=0x7f37540a7000 comp rx=0 tx=0).ready entity=mgr.8746 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:03:54.465 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:54.465+0000 7f374e7fc640 1 -- 172.21.15.16:0/304713306 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f37500502d0 con 0x7f375815c8e0 2023-05-22T16:03:54.624 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-05-22T16:03:54.627 DEBUG:teuthology.orchestra.run.smithi062:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-05-22T16:03:54.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:54.629+0000 7f3760058640 1 -- 172.21.15.16:0/304713306 --> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "snap_test_dir1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f375804dbe0 con 0x7f373c065220 2023-05-22T16:03:54.644 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:54.641+0000 7f374e7fc640 1 -- 172.21.15.16:0/304713306 <== mgr.8746 v2:172.21.15.16:6832/65890 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+72 (secure 0 0 0) 0x7f375804dbe0 con 0x7f373c065220 2023-05-22T16:03:54.644 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T16:03:54.644 INFO:teuthology.orchestra.run.smithi016.stdout:{"path": "snap_test_dir1", "schedule": ["1M"], "retention": [{"M": 1}]} 2023-05-22T16:03:54.646 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:54.645+0000 7f3760058640 1 -- 172.21.15.16:0/304713306 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f373c065220 msgr2=0x7f373c0676d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:54.646 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:54.645+0000 7f3760058640 1 --2- 172.21.15.16:0/304713306 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f373c065220 0x7f373c0676d0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f37540af000 tx=0x7f37540a7000 comp rx=0 tx=0).stop 2023-05-22T16:03:54.646 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:54.645+0000 7f3760058640 1 -- 172.21.15.16:0/304713306 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f375815c8e0 msgr2=0x7f37580b6480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:03:54.646 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:54.645+0000 7f3760058640 1 --2- 172.21.15.16:0/304713306 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f375815c8e0 0x7f37580b6480 secure :-1 s=READY pgs=2311 cs=0 l=1 rev1=1 crypto rx=0x7f375000c770 tx=0x7f375000cc30 comp rx=0 tx=0).stop 2023-05-22T16:03:54.647 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:54.645+0000 7f3760058640 1 -- 172.21.15.16:0/304713306 shutdown_connections 2023-05-22T16:03:54.647 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:54.645+0000 7f3760058640 1 --2- 172.21.15.16:0/304713306 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f373c065220 0x7f373c0676d0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:54.647 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:54.645+0000 7f3760058640 1 --2- 172.21.15.16:0/304713306 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f375815c8e0 0x7f37580b6480 unknown :-1 s=CLOSED pgs=2311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:54.647 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:54.645+0000 7f3760058640 1 --2- 172.21.15.16:0/304713306 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f37580d2eb0 0x7f375813c380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:54.647 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:54.645+0000 7f3760058640 1 --2- 172.21.15.16:0/304713306 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f37580cef10 0x7f37581600c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:03:54.647 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:54.645+0000 7f3760058640 1 -- 172.21.15.16:0/304713306 >> 172.21.15.16:0/304713306 conn(0x7f3758070e50 msgr2=0x7f37580cd720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:03:54.647 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:54.645+0000 7f3760058640 1 -- 172.21.15.16:0/304713306 shutdown_connections 2023-05-22T16:03:54.648 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:03:54.645+0000 7f3760058640 1 -- 172.21.15.16:0/304713306 wait complete. 2023-05-22T16:03:54.662 DEBUG:tasks.cephfs.test_snap_schedules:res= {"path": "snap_test_dir1", "schedule": ["1M"], "retention": [{"M": 1}]} 2023-05-22T16:03:54.662 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'path': 'snap_test_dir1', 'schedule': ['1M'], 'retention': [{'M': 1}]} 2023-05-22T16:03:54.662 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T16:03:54.663 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T16:03:54.717 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-05-22T16:03:54.717 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T16:03:54.718 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T16:03:55.719 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T16:03:55.720 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T16:03:55.737 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-05-22T16:03:55.738 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T16:03:55.738 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T16:03:56.739 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T16:03:56.740 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T16:03:56.757 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-05-22T16:03:56.758 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T16:03:56.759 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T16:03:57.760 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T16:03:57.761 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T16:03:57.779 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-05-22T16:03:57.779 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T16:03:57.779 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T16:03:58.780 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T16:03:58.781 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T16:03:58.799 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-05-22T16:03:58.799 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T16:03:58.799 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T16:03:59.800 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T16:03:59.801 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T16:03:59.819 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-05-22T16:03:59.819 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T16:03:59.819 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T16:04:00.317 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T16:04:00.317+0000 7f217828c640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T16:04:00.317 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T16:04:00.317+0000 7f217828c640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T16:04:00.318 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T16:04:00.317+0000 7f217828c640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T16:04:00.318 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T16:04:00.317+0000 7f217828c640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T16:04:00.318 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T16:04:00.317+0000 7f217828c640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T16:04:00.820 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T16:04:00.821 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T16:04:00.836 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-16_04_00_UTC 2023-05-22T16:04:00.837 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-05-22-16_04_00_UTC'} 2023-05-22T16:04:00.837 INFO:tasks.cephfs.test_snap_schedules:added: {'scheduled-2023-05-22-16_04_00_UTC'} 2023-05-22T16:04:00.837 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T16:04:00.837 DEBUG:tasks.cephfs.test_snap_schedules:snapshots added=['scheduled-2023-05-22-16_04_00_UTC'] 2023-05-22T16:04:00.838 DEBUG:tasks.cephfs.test_snap_schedules:exec=1684771434.321878, now = 1684771440.8380957, timo = 6 2023-05-22T16:04:01.838 DEBUG:tasks.cephfs.test_snap_schedules:expecting removal of snap snap_test_dir1/.snap/scheduled-2023-05-22-16_04_00 in ~66s... 2023-05-22T16:04:01.839 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T16:04:01.839 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T16:04:01.855 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-16_04_00_UTC 2023-05-22T16:04:01.856 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-05-22-16_04_00_UTC'} 2023-05-22T16:04:01.856 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T16:04:01.856 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T16:04:02.857 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T16:04:02.858 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T16:04:02.876 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-16_04_00_UTC 2023-05-22T16:04:02.877 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-05-22-16_04_00_UTC'} 2023-05-22T16:04:02.877 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T16:04:02.878 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T16:04:03.878 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T16:04:03.879 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T16:04:03.895 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-16_04_00_UTC 2023-05-22T16:04:03.896 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-05-22-16_04_00_UTC'} 2023-05-22T16:04:03.897 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T16:04:03.897 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T16:04:04.898 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T16:04:04.899 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T16:04:04.915 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-16_04_00_UTC 2023-05-22T16:04:04.916 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-05-22-16_04_00_UTC'} 2023-05-22T16:04:04.917 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T16:04:04.917 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T16:04:05.918 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T16:04:05.919 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T16:04:05.936 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-16_04_00_UTC 2023-05-22T16:04:05.937 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-05-22-16_04_00_UTC'} 2023-05-22T16:04:05.938 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T16:04:05.938 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T16:04:06.939 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T16:04:06.939 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T16:04:06.957 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-16_04_00_UTC 2023-05-22T16:04:06.958 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-05-22-16_04_00_UTC'} 2023-05-22T16:04:06.958 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T16:04:06.958 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T16:04:07.959 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T16:04:07.960 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T16:04:07.976 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-16_04_00_UTC 2023-05-22T16:04:07.977 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-05-22-16_04_00_UTC'} 2023-05-22T16:04:07.978 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T16:04:07.978 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T16:04:08.979 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T16:04:08.979 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T16:04:08.996 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-16_04_00_UTC 2023-05-22T16:04:08.997 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-05-22-16_04_00_UTC'} 2023-05-22T16:04:08.998 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T16:04:08.998 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T16:04:09.999 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T16:04:09.999 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T16:04:10.015 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-16_04_00_UTC 2023-05-22T16:04:10.016 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-05-22-16_04_00_UTC'} 2023-05-22T16:04:10.016 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T16:04:10.017 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T16:04:11.017 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T16:04:11.018 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T16:04:11.035 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-16_04_00_UTC 2023-05-22T16:04:11.036 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-05-22-16_04_00_UTC'} 2023-05-22T16:04:11.036 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T16:04:11.036 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T16:04:12.037 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T16:04:12.037 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T16:04:12.055 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-16_04_00_UTC 2023-05-22T16:04:12.056 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-05-22-16_04_00_UTC'} 2023-05-22T16:04:12.056 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T16:04:12.056 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T16:04:13.057 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T16:04:13.058 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T16:04:13.076 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-16_04_00_UTC 2023-05-22T16:04:13.077 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-05-22-16_04_00_UTC'} 2023-05-22T16:04:13.078 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T16:04:13.078 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T16:04:14.079 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T16:04:14.080 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T16:04:14.097 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-16_04_00_UTC 2023-05-22T16:04:14.098 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-05-22-16_04_00_UTC'} 2023-05-22T16:04:14.099 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T16:04:14.099 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T16:04:15.100 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T16:04:15.101 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T16:04:15.119 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-16_04_00_UTC 2023-05-22T16:04:15.120 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-05-22-16_04_00_UTC'} 2023-05-22T16:04:15.120 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T16:04:15.120 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T16:04:16.121 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T16:04:16.122 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T16:04:16.139 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-16_04_00_UTC 2023-05-22T16:04:16.140 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-05-22-16_04_00_UTC'} 2023-05-22T16:04:16.140 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T16:04:16.140 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T16:04:17.141 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T16:04:17.142 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T16:04:17.160 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-16_04_00_UTC 2023-05-22T16:04:17.161 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-05-22-16_04_00_UTC'} 2023-05-22T16:04:17.161 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T16:04:17.162 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T16:04:18.163 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T16:04:18.164 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T16:04:18.180 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-16_04_00_UTC 2023-05-22T16:04:18.181 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-05-22-16_04_00_UTC'} 2023-05-22T16:04:18.181 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T16:04:18.181 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T16:04:19.182 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T16:04:19.183 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T16:04:19.199 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-16_04_00_UTC 2023-05-22T16:04:19.201 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-05-22-16_04_00_UTC'} 2023-05-22T16:04:19.201 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T16:04:19.202 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T16:04:20.203 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T16:04:20.203 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T16:04:20.220 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-16_04_00_UTC 2023-05-22T16:04:20.221 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-05-22-16_04_00_UTC'} 2023-05-22T16:04:20.221 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T16:04:20.221 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T16:04:21.222 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T16:04:21.223 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T16:04:21.239 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-16_04_00_UTC 2023-05-22T16:04:21.240 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-05-22-16_04_00_UTC'} 2023-05-22T16:04:21.240 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T16:04:21.241 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T16:04:22.242 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T16:04:22.242 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T16:04:22.259 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-16_04_00_UTC 2023-05-22T16:04:22.260 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-05-22-16_04_00_UTC'} 2023-05-22T16:04:22.260 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T16:04:22.260 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T16:04:23.261 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T16:04:23.262 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T16:04:23.279 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-16_04_00_UTC 2023-05-22T16:04:23.280 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-05-22-16_04_00_UTC'} 2023-05-22T16:04:23.280 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T16:04:23.281 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T16:04:24.282 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T16:04:24.282 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T16:04:24.299 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-16_04_00_UTC 2023-05-22T16:04:24.300 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-05-22-16_04_00_UTC'} 2023-05-22T16:04:24.300 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T16:04:24.301 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T16:04:24.672 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-05-22T16:04:24.675 DEBUG:teuthology.orchestra.run.smithi062:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-05-22T16:04:25.302 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T16:04:25.303 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T16:04:25.320 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-16_04_00_UTC 2023-05-22T16:04:25.321 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-05-22-16_04_00_UTC'} 2023-05-22T16:04:25.321 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T16:04:25.321 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T16:04:26.323 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T16:04:26.323 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T16:04:26.339 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-16_04_00_UTC 2023-05-22T16:04:26.340 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-05-22-16_04_00_UTC'} 2023-05-22T16:04:26.340 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T16:04:26.341 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T16:04:27.342 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T16:04:27.342 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T16:04:27.359 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-16_04_00_UTC 2023-05-22T16:04:27.360 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-05-22-16_04_00_UTC'} 2023-05-22T16:04:27.360 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T16:04:27.360 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T16:04:28.361 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T16:04:28.361 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T16:04:28.378 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-16_04_00_UTC 2023-05-22T16:04:28.379 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-05-22-16_04_00_UTC'} 2023-05-22T16:04:28.379 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T16:04:28.380 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T16:04:29.381 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T16:04:29.382 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T16:04:29.398 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-16_04_00_UTC 2023-05-22T16:04:29.399 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-05-22-16_04_00_UTC'} 2023-05-22T16:04:29.400 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T16:04:29.400 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T16:04:30.401 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T16:04:30.402 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T16:04:30.419 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-16_04_00_UTC 2023-05-22T16:04:30.419 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-05-22-16_04_00_UTC'} 2023-05-22T16:04:30.420 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T16:04:30.420 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T16:04:31.421 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T16:04:31.422 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T16:04:31.439 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-16_04_00_UTC 2023-05-22T16:04:31.440 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-05-22-16_04_00_UTC'} 2023-05-22T16:04:31.440 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T16:04:31.440 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T16:04:32.441 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T16:04:32.442 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T16:04:32.457 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-16_04_00_UTC 2023-05-22T16:04:32.458 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-05-22-16_04_00_UTC'} 2023-05-22T16:04:32.459 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T16:04:32.459 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T16:04:33.460 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T16:04:33.461 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T16:04:33.477 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-16_04_00_UTC 2023-05-22T16:04:33.478 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-05-22-16_04_00_UTC'} 2023-05-22T16:04:33.478 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T16:04:33.478 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T16:04:34.479 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T16:04:34.479 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T16:04:34.495 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-16_04_00_UTC 2023-05-22T16:04:34.497 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-05-22-16_04_00_UTC'} 2023-05-22T16:04:34.497 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T16:04:34.498 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T16:04:35.499 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T16:04:35.500 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T16:04:35.515 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-16_04_00_UTC 2023-05-22T16:04:35.516 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-05-22-16_04_00_UTC'} 2023-05-22T16:04:35.517 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T16:04:35.517 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T16:04:36.517 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T16:04:36.518 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T16:04:36.535 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-16_04_00_UTC 2023-05-22T16:04:36.536 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-05-22-16_04_00_UTC'} 2023-05-22T16:04:36.536 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T16:04:36.536 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T16:04:37.538 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T16:04:37.538 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T16:04:37.554 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-16_04_00_UTC 2023-05-22T16:04:37.555 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-05-22-16_04_00_UTC'} 2023-05-22T16:04:37.556 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T16:04:37.556 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T16:04:38.557 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T16:04:38.557 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T16:04:38.573 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-16_04_00_UTC 2023-05-22T16:04:38.574 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-05-22-16_04_00_UTC'} 2023-05-22T16:04:38.574 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T16:04:38.575 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T16:04:39.576 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T16:04:39.577 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T16:04:39.594 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-16_04_00_UTC 2023-05-22T16:04:39.595 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-05-22-16_04_00_UTC'} 2023-05-22T16:04:39.596 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T16:04:39.596 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T16:04:40.597 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T16:04:40.598 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T16:04:40.615 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-16_04_00_UTC 2023-05-22T16:04:40.616 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-05-22-16_04_00_UTC'} 2023-05-22T16:04:40.616 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T16:04:40.616 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T16:04:41.618 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T16:04:41.618 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T16:04:41.636 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-16_04_00_UTC 2023-05-22T16:04:41.637 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-05-22-16_04_00_UTC'} 2023-05-22T16:04:41.637 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T16:04:41.637 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T16:04:42.639 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T16:04:42.639 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T16:04:42.660 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-16_04_00_UTC 2023-05-22T16:04:42.661 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-05-22-16_04_00_UTC'} 2023-05-22T16:04:42.661 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T16:04:42.662 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T16:04:43.663 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T16:04:43.663 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T16:04:43.681 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-16_04_00_UTC 2023-05-22T16:04:43.682 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-05-22-16_04_00_UTC'} 2023-05-22T16:04:43.683 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T16:04:43.683 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T16:04:44.684 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T16:04:44.685 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T16:04:44.702 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-16_04_00_UTC 2023-05-22T16:04:44.703 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-05-22-16_04_00_UTC'} 2023-05-22T16:04:44.703 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T16:04:44.703 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T16:04:45.704 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T16:04:45.705 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T16:04:45.722 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-16_04_00_UTC 2023-05-22T16:04:45.722 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-05-22-16_04_00_UTC'} 2023-05-22T16:04:45.723 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T16:04:45.723 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T16:04:46.724 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T16:04:46.725 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T16:04:46.742 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-16_04_00_UTC 2023-05-22T16:04:46.743 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-05-22-16_04_00_UTC'} 2023-05-22T16:04:46.743 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T16:04:46.743 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T16:04:47.744 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T16:04:47.745 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T16:04:47.762 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-16_04_00_UTC 2023-05-22T16:04:47.763 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-05-22-16_04_00_UTC'} 2023-05-22T16:04:47.763 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T16:04:47.763 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T16:04:48.764 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T16:04:48.765 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T16:04:48.783 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-16_04_00_UTC 2023-05-22T16:04:48.784 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-05-22-16_04_00_UTC'} 2023-05-22T16:04:48.784 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T16:04:48.784 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T16:04:49.785 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T16:04:49.786 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T16:04:49.805 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-16_04_00_UTC 2023-05-22T16:04:49.805 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-05-22-16_04_00_UTC'} 2023-05-22T16:04:49.806 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T16:04:49.806 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T16:04:50.807 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T16:04:50.808 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T16:04:50.825 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-16_04_00_UTC 2023-05-22T16:04:50.825 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-05-22-16_04_00_UTC'} 2023-05-22T16:04:50.825 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T16:04:50.826 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T16:04:51.826 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T16:04:51.827 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T16:04:51.846 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-16_04_00_UTC 2023-05-22T16:04:51.847 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-05-22-16_04_00_UTC'} 2023-05-22T16:04:51.847 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T16:04:51.848 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T16:04:52.848 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T16:04:52.849 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T16:04:52.867 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-16_04_00_UTC 2023-05-22T16:04:52.868 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-05-22-16_04_00_UTC'} 2023-05-22T16:04:52.869 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T16:04:52.869 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T16:04:53.870 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T16:04:53.871 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T16:04:53.888 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-16_04_00_UTC 2023-05-22T16:04:53.889 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-05-22-16_04_00_UTC'} 2023-05-22T16:04:53.890 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T16:04:53.890 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T16:04:54.719 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-05-22T16:04:54.723 DEBUG:teuthology.orchestra.run.smithi062:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-05-22T16:04:54.891 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T16:04:54.892 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T16:04:54.909 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-16_04_00_UTC 2023-05-22T16:04:54.910 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-05-22-16_04_00_UTC'} 2023-05-22T16:04:54.910 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T16:04:54.911 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T16:04:55.912 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T16:04:55.913 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T16:04:55.930 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-16_04_00_UTC 2023-05-22T16:04:55.930 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-05-22-16_04_00_UTC'} 2023-05-22T16:04:55.932 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T16:04:55.933 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T16:04:56.934 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T16:04:56.935 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T16:04:56.951 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-16_04_00_UTC 2023-05-22T16:04:56.952 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-05-22-16_04_00_UTC'} 2023-05-22T16:04:56.953 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T16:04:56.953 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T16:04:57.954 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T16:04:57.955 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T16:04:57.972 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-16_04_00_UTC 2023-05-22T16:04:57.973 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-05-22-16_04_00_UTC'} 2023-05-22T16:04:57.973 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T16:04:57.974 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T16:04:58.974 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T16:04:58.975 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T16:04:58.993 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-16_04_00_UTC 2023-05-22T16:04:58.994 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-05-22-16_04_00_UTC'} 2023-05-22T16:04:58.995 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T16:04:58.995 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T16:04:59.996 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T16:04:59.997 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T16:05:00.012 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-16_04_00_UTC 2023-05-22T16:05:00.013 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-05-22-16_04_00_UTC'} 2023-05-22T16:05:00.013 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-05-22T16:05:00.013 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-05-22T16:05:01.015 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T16:05:01.015 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T16:05:01.033 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-16_05_00_UTC 2023-05-22T16:05:01.034 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-05-22-16_05_00_UTC'} 2023-05-22T16:05:01.035 INFO:tasks.cephfs.test_snap_schedules:added: {'scheduled-2023-05-22-16_05_00_UTC'} 2023-05-22T16:05:01.036 INFO:tasks.cephfs.test_snap_schedules:removed: {'scheduled-2023-05-22-16_04_00_UTC'} 2023-05-22T16:05:01.036 DEBUG:tasks.cephfs.test_snap_schedules:snapshots removed=['scheduled-2023-05-22-16_04_00_UTC'] 2023-05-22T16:05:01.037 DEBUG:tasks.cephfs.test_snap_schedules:exec=1684771434.321878, now = 1684771501.037415, timo = 66 2023-05-22T16:05:02.039 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove --fs cephfs snap_test_dir1 2023-05-22T16:05:02.178 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:02.176+0000 7fc03082c640 1 Processor -- start 2023-05-22T16:05:02.178 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:02.176+0000 7fc03082c640 1 -- start start 2023-05-22T16:05:02.179 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:02.176+0000 7fc03082c640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc02c070c10 0x7fc02c0cc4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:02.179 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:02.176+0000 7fc03082c640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc02c0c9e40 0x7fc02c06bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:02.179 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:02.176+0000 7fc03082c640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc02c0ccbb0 con 0x7fc02c15d650 2023-05-22T16:05:02.180 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:02.176+0000 7fc03082c640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc02c0cd7c0 con 0x7fc02c0c9e40 2023-05-22T16:05:02.180 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:02.176+0000 7fc03082c640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc02c0cdaa0 con 0x7fc02c070c10 2023-05-22T16:05:02.181 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:02.176+0000 7fc029d9b640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc02c0c9e40 0x7fc02c06bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:02.181 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:02.176+0000 7fc02ad9d640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fc02c15d650 0x7fc02c0ce2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:45408/0 (socket says 172.21.15.16:45408) 2023-05-22T16:05:02.181 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:02.176+0000 7fc02ad9d640 1 -- 172.21.15.16:0/3690448046 learned_addr learned my addr 172.21.15.16:0/3690448046 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T16:05:02.181 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:02.176+0000 7fc029d9b640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc02c0c9e40 0x7fc02c06bfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45984/0 (socket says 172.21.15.16:45984) 2023-05-22T16:05:02.183 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:02.176+0000 7fc02a59c640 1 --2- 172.21.15.16:0/3690448046 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc02c070c10 0x7fc02c0cc4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:02.183 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:02.176+0000 7fc029d9b640 1 -- 172.21.15.16:0/3690448046 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc02c070c10 msgr2=0x7fc02c0cc4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:05:02.183 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:02.176+0000 7fc029d9b640 1 --2- 172.21.15.16:0/3690448046 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc02c070c10 0x7fc02c0cc4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:02.183 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:02.176+0000 7fc029d9b640 1 -- 172.21.15.16:0/3690448046 >> v1:172.21.15.16:6789/0 conn(0x7fc02c15d650 legacy=0x7fc02c0ce2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:02.183 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:02.176+0000 7fc02959a640 1 -- 172.21.15.16:0/3690448046 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4128402688 0 0) 0x7fc02c0ccbb0 con 0x7fc02c15d650 2023-05-22T16:05:02.184 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:02.176+0000 7fc029d9b640 1 -- 172.21.15.16:0/3690448046 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc02c0cdd80 con 0x7fc02c0c9e40 2023-05-22T16:05:02.184 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:02.176+0000 7fc02a59c640 1 --2- 172.21.15.16:0/3690448046 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc02c070c10 0x7fc02c0cc4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T16:05:02.184 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:02.180+0000 7fc029d9b640 1 --2- 172.21.15.16:0/3690448046 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc02c0c9e40 0x7fc02c06bfa0 secure :-1 s=READY pgs=2315 cs=0 l=1 rev1=1 crypto rx=0x7fc01c0097c0 tx=0x7fc01c01c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a720fcacb0d807fc server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:02.184 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:02.180+0000 7fc02959a640 1 -- 172.21.15.16:0/3690448046 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc01c0a6040 con 0x7fc02c0c9e40 2023-05-22T16:05:02.184 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:02.180+0000 7fc02959a640 1 -- 172.21.15.16:0/3690448046 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc01c0aa600 con 0x7fc02c0c9e40 2023-05-22T16:05:02.184 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:02.180+0000 7fc02959a640 1 -- 172.21.15.16:0/3690448046 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc01c0aa990 con 0x7fc02c0c9e40 2023-05-22T16:05:02.184 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:02.180+0000 7fc03082c640 1 -- 172.21.15.16:0/3690448046 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc02c0c9e40 msgr2=0x7fc02c06bfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:02.185 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:02.180+0000 7fc03082c640 1 --2- 172.21.15.16:0/3690448046 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc02c0c9e40 0x7fc02c06bfa0 secure :-1 s=READY pgs=2315 cs=0 l=1 rev1=1 crypto rx=0x7fc01c0097c0 tx=0x7fc01c01c480 comp rx=0 tx=0).stop 2023-05-22T16:05:02.185 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:02.180+0000 7fc03082c640 1 -- 172.21.15.16:0/3690448046 shutdown_connections 2023-05-22T16:05:02.185 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:02.180+0000 7fc03082c640 1 --2- 172.21.15.16:0/3690448046 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc02c0c9e40 0x7fc02c06bfa0 unknown :-1 s=CLOSED pgs=2315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:02.185 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:02.180+0000 7fc03082c640 1 --2- 172.21.15.16:0/3690448046 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc02c070c10 0x7fc02c0cc4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:02.185 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:02.180+0000 7fc03082c640 1 -- 172.21.15.16:0/3690448046 >> 172.21.15.16:0/3690448046 conn(0x7fc02c069720 msgr2=0x7fc02c069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:05:02.185 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:02.180+0000 7fc03082c640 1 -- 172.21.15.16:0/3690448046 shutdown_connections 2023-05-22T16:05:02.185 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:02.180+0000 7fc03082c640 1 -- 172.21.15.16:0/3690448046 wait complete. 2023-05-22T16:05:02.187 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:02.180+0000 7fc03082c640 1 Processor -- start 2023-05-22T16:05:02.187 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:02.180+0000 7fc03082c640 1 -- start start 2023-05-22T16:05:02.188 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:02.184+0000 7fc03082c640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc02c070c10 0x7fc02c15c390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:02.188 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:02.184+0000 7fc03082c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc02c0c9e40 0x7fc02c142b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:02.188 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:02.184+0000 7fc03082c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc02c15d650 0x7fc02c15b200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:02.188 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:02.184+0000 7fc03082c640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc02c15c8d0 con 0x7fc02c15d650 2023-05-22T16:05:02.188 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:02.184+0000 7fc03082c640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc02c143cb0 con 0x7fc02c0c9e40 2023-05-22T16:05:02.188 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:02.184+0000 7fc03082c640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc02c14faa0 con 0x7fc02c070c10 2023-05-22T16:05:02.188 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:02.184+0000 7fc02ad9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc02c15d650 0x7fc02c15b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:02.189 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:02.184+0000 7fc02ad9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc02c15d650 0x7fc02c15b200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45998/0 (socket says 172.21.15.16:45998) 2023-05-22T16:05:02.189 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:02.184+0000 7fc02ad9d640 1 -- 172.21.15.16:0/1711583617 learned_addr learned my addr 172.21.15.16:0/1711583617 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:05:02.189 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:02.184+0000 7fc02a59c640 1 --2- 172.21.15.16:0/1711583617 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc02c070c10 0x7fc02c15c390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:02.189 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:02.184+0000 7fc029d9b640 1 --2- 172.21.15.16:0/1711583617 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc02c0c9e40 0x7fc02c142b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:02.189 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:02.184+0000 7fc02ad9d640 1 -- 172.21.15.16:0/1711583617 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc02c070c10 msgr2=0x7fc02c15c390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:02.189 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:02.184+0000 7fc02ad9d640 1 --2- 172.21.15.16:0/1711583617 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc02c070c10 0x7fc02c15c390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:02.189 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:02.184+0000 7fc02ad9d640 1 -- 172.21.15.16:0/1711583617 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc02c0c9e40 msgr2=0x7fc02c142b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:02.190 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:02.184+0000 7fc02ad9d640 1 --2- 172.21.15.16:0/1711583617 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc02c0c9e40 0x7fc02c142b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:02.190 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:02.184+0000 7fc02ad9d640 1 -- 172.21.15.16:0/1711583617 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc01c009420 con 0x7fc02c15d650 2023-05-22T16:05:02.190 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:02.184+0000 7fc02a59c640 1 --2- 172.21.15.16:0/1711583617 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc02c070c10 0x7fc02c15c390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T16:05:02.190 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:02.184+0000 7fc02ad9d640 1 --2- 172.21.15.16:0/1711583617 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc02c15d650 0x7fc02c15b200 secure :-1 s=READY pgs=2316 cs=0 l=1 rev1=1 crypto rx=0x7fc02400c780 tx=0x7fc02400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:02.190 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:02.184+0000 7fc022ffd640 1 -- 172.21.15.16:0/1711583617 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc02400ce90 con 0x7fc02c15d650 2023-05-22T16:05:02.190 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:02.184+0000 7fc022ffd640 1 -- 172.21.15.16:0/1711583617 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc02400a640 con 0x7fc02c15d650 2023-05-22T16:05:02.191 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:02.184+0000 7fc03082c640 1 -- 172.21.15.16:0/1711583617 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc02c15dea0 con 0x7fc02c15d650 2023-05-22T16:05:02.191 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:02.184+0000 7fc022ffd640 1 -- 172.21.15.16:0/1711583617 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc02400a9d0 con 0x7fc02c15d650 2023-05-22T16:05:02.191 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:02.184+0000 7fc03082c640 1 -- 172.21.15.16:0/1711583617 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc02c0cce90 con 0x7fc02c15d650 2023-05-22T16:05:02.191 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:02.184+0000 7fc03082c640 1 -- 172.21.15.16:0/1711583617 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc02c0ca290 con 0x7fc02c15d650 2023-05-22T16:05:02.192 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:02.184+0000 7fc022ffd640 1 -- 172.21.15.16:0/1711583617 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 88) v1 ==== 84463+0+0 (secure 0 0 0) 0x7fc02400ab30 con 0x7fc02c15d650 2023-05-22T16:05:02.192 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:02.184+0000 7fc022ffd640 1 --2- 172.21.15.16:0/1711583617 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fc00c065640 0x7fc00c067af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:02.192 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:02.188+0000 7fc022ffd640 1 -- 172.21.15.16:0/1711583617 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(180..180 src has 1..180) v4 ==== 6782+0+0 (secure 0 0 0) 0x7fc0240826d0 con 0x7fc02c15d650 2023-05-22T16:05:02.193 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:02.188+0000 7fc02a59c640 1 --2- 172.21.15.16:0/1711583617 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fc00c065640 0x7fc00c067af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:02.193 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:02.188+0000 7fc02a59c640 1 --2- 172.21.15.16:0/1711583617 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fc00c065640 0x7fc00c067af0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fc01400a9b0 tx=0x7fc014008040 comp rx=0 tx=0).ready entity=mgr.8746 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:02.193 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:02.192+0000 7fc022ffd640 1 -- 172.21.15.16:0/1711583617 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fc02404fd50 con 0x7fc02c15d650 2023-05-22T16:05:02.366 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:02.364+0000 7fc03082c640 1 -- 172.21.15.16:0/1711583617 --> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "fs": "cephfs", "path": "snap_test_dir1", "target": ["mon-mgr", ""]}) v1 -- 0x7fc02c14fc60 con 0x7fc00c065640 2023-05-22T16:05:02.518 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:02.516+0000 7fc022ffd640 1 -- 172.21.15.16:0/1711583617 <== mgr.8746 v2:172.21.15.16:6832/65890 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7fc02c14fc60 con 0x7fc00c065640 2023-05-22T16:05:02.519 INFO:teuthology.orchestra.run.smithi016.stdout:Schedule removed for path snap_test_dir1 2023-05-22T16:05:02.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:02.520+0000 7fc03082c640 1 -- 172.21.15.16:0/1711583617 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fc00c065640 msgr2=0x7fc00c067af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:02.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:02.520+0000 7fc03082c640 1 --2- 172.21.15.16:0/1711583617 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fc00c065640 0x7fc00c067af0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fc01400a9b0 tx=0x7fc014008040 comp rx=0 tx=0).stop 2023-05-22T16:05:02.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:02.520+0000 7fc03082c640 1 -- 172.21.15.16:0/1711583617 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc02c15d650 msgr2=0x7fc02c15b200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:02.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:02.520+0000 7fc03082c640 1 --2- 172.21.15.16:0/1711583617 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc02c15d650 0x7fc02c15b200 secure :-1 s=READY pgs=2316 cs=0 l=1 rev1=1 crypto rx=0x7fc02400c780 tx=0x7fc02400cc40 comp rx=0 tx=0).stop 2023-05-22T16:05:02.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:02.520+0000 7fc03082c640 1 -- 172.21.15.16:0/1711583617 shutdown_connections 2023-05-22T16:05:02.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:02.520+0000 7fc03082c640 1 --2- 172.21.15.16:0/1711583617 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fc00c065640 0x7fc00c067af0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:02.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:02.520+0000 7fc03082c640 1 --2- 172.21.15.16:0/1711583617 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc02c15d650 0x7fc02c15b200 unknown :-1 s=CLOSED pgs=2316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:02.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:02.520+0000 7fc03082c640 1 --2- 172.21.15.16:0/1711583617 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc02c0c9e40 0x7fc02c142b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:02.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:02.520+0000 7fc03082c640 1 --2- 172.21.15.16:0/1711583617 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc02c070c10 0x7fc02c15c390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:02.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:02.520+0000 7fc03082c640 1 -- 172.21.15.16:0/1711583617 >> 172.21.15.16:0/1711583617 conn(0x7fc02c069720 msgr2=0x7fc02c0712a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:05:02.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:02.520+0000 7fc03082c640 1 -- 172.21.15.16:0/1711583617 shutdown_connections 2023-05-22T16:05:02.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:02.520+0000 7fc03082c640 1 -- 172.21.15.16:0/1711583617 wait complete. 2023-05-22T16:05:02.543 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path snap_test_dir1 2023-05-22T16:05:02.543 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T16:05:02.544 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T16:05:02.557 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-05-22-16_05_00_UTC 2023-05-22T16:05:02.558 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-05-22-16_05_00_UTC 2023-05-22T16:05:02.558 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T16:05:02.559 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-05-22-16_05_00_UTC) 2023-05-22T16:05:02.621 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T16:05:02.621 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1) 2023-05-22T16:05:02.677 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2023-05-22T16:05:02.856 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:02.852+0000 7ff7278b8640 1 Processor -- start 2023-05-22T16:05:02.856 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:02.852+0000 7ff7278b8640 1 -- start start 2023-05-22T16:05:02.857 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:02.856+0000 7ff7278b8640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff720159970 0x7ff720159d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:02.857 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:02.856+0000 7ff7278b8640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff72015d680 0x7ff7200cb260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:02.858 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:02.856+0000 7ff7278b8640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff720070c30 con 0x7ff72015dad0 2023-05-22T16:05:02.858 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:02.856+0000 7ff7278b8640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff720070e60 con 0x7ff720159970 2023-05-22T16:05:02.858 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:02.856+0000 7ff7278b8640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff720071140 con 0x7ff72015d680 2023-05-22T16:05:02.858 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:02.856+0000 7ff725654640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff720159970 0x7ff720159d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:02.858 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:02.856+0000 7ff725654640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff720159970 0x7ff720159d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46012/0 (socket says 172.21.15.16:46012) 2023-05-22T16:05:02.858 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:02.856+0000 7ff725654640 1 -- 172.21.15.16:0/4177361393 learned_addr learned my addr 172.21.15.16:0/4177361393 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:05:02.858 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:02.856+0000 7ff724e53640 1 --2- 172.21.15.16:0/4177361393 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff72015d680 0x7ff7200cb260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:02.859 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:02.856+0000 7ff725654640 1 -- 172.21.15.16:0/4177361393 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff72015d680 msgr2=0x7ff7200cb260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:05:02.859 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:02.856+0000 7ff725654640 1 --2- 172.21.15.16:0/4177361393 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff72015d680 0x7ff7200cb260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:02.859 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:02.856+0000 7ff725654640 1 -- 172.21.15.16:0/4177361393 >> v1:172.21.15.62:6789/0 conn(0x7ff72015dad0 legacy=0x7ff72006b6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:05:02.859 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:02.856+0000 7ff725654640 1 -- 172.21.15.16:0/4177361393 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff720071420 con 0x7ff720159970 2023-05-22T16:05:02.860 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:02.856+0000 7ff725654640 1 --2- 172.21.15.16:0/4177361393 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff720159970 0x7ff720159d40 secure :-1 s=READY pgs=2317 cs=0 l=1 rev1=1 crypto rx=0x7ff718009e50 tx=0x7ff71801c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3bb6dacdd4f7d5 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:02.860 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:02.856+0000 7ff71ffff640 1 -- 172.21.15.16:0/4177361393 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7180a57b0 con 0x7ff720159970 2023-05-22T16:05:02.861 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:02.856+0000 7ff71ffff640 1 -- 172.21.15.16:0/4177361393 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7180a5910 con 0x7ff720159970 2023-05-22T16:05:02.861 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:02.860+0000 7ff71ffff640 1 -- 172.21.15.16:0/4177361393 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7180a5ca0 con 0x7ff720159970 2023-05-22T16:05:02.861 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:02.860+0000 7ff7278b8640 1 -- 172.21.15.16:0/4177361393 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff720159970 msgr2=0x7ff720159d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:02.861 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:02.860+0000 7ff7278b8640 1 --2- 172.21.15.16:0/4177361393 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff720159970 0x7ff720159d40 secure :-1 s=READY pgs=2317 cs=0 l=1 rev1=1 crypto rx=0x7ff718009e50 tx=0x7ff71801c910 comp rx=0 tx=0).stop 2023-05-22T16:05:02.862 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:02.860+0000 7ff7278b8640 1 -- 172.21.15.16:0/4177361393 shutdown_connections 2023-05-22T16:05:02.862 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:02.860+0000 7ff7278b8640 1 --2- 172.21.15.16:0/4177361393 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff72015d680 0x7ff7200cb260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:02.862 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:02.860+0000 7ff7278b8640 1 --2- 172.21.15.16:0/4177361393 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff720159970 0x7ff720159d40 unknown :-1 s=CLOSED pgs=2317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:02.862 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:02.860+0000 7ff7278b8640 1 -- 172.21.15.16:0/4177361393 >> 172.21.15.16:0/4177361393 conn(0x7ff720069720 msgr2=0x7ff720069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:05:02.862 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:02.860+0000 7ff7278b8640 1 -- 172.21.15.16:0/4177361393 shutdown_connections 2023-05-22T16:05:02.862 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:02.860+0000 7ff7278b8640 1 -- 172.21.15.16:0/4177361393 wait complete. 2023-05-22T16:05:02.864 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:02.860+0000 7ff7278b8640 1 Processor -- start 2023-05-22T16:05:02.864 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:02.860+0000 7ff7278b8640 1 -- start start 2023-05-22T16:05:02.864 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:02.860+0000 7ff7278b8640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff720159970 0x7ff7200b3950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:02.864 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:02.860+0000 7ff7278b8640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff72015d680 0x7ff7201f3a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:02.864 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:02.860+0000 7ff7278b8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff72015dad0 0x7ff7201f7730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:02.864 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:02.860+0000 7ff7278b8640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7200b0eb0 con 0x7ff72015dad0 2023-05-22T16:05:02.865 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:02.860+0000 7ff7278b8640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff720152960 con 0x7ff720159970 2023-05-22T16:05:02.865 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:02.860+0000 7ff7278b8640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7200b3e90 con 0x7ff72015d680 2023-05-22T16:05:02.865 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:02.860+0000 7ff725e55640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff72015dad0 0x7ff7201f7730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:02.865 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:02.860+0000 7ff725e55640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff72015dad0 0x7ff7201f7730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46026/0 (socket says 172.21.15.16:46026) 2023-05-22T16:05:02.865 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:02.860+0000 7ff725e55640 1 -- 172.21.15.16:0/2085634577 learned_addr learned my addr 172.21.15.16:0/2085634577 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:05:02.865 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:02.860+0000 7ff724e53640 1 --2- 172.21.15.16:0/2085634577 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff72015d680 0x7ff7201f3a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:02.865 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:02.860+0000 7ff725654640 1 --2- 172.21.15.16:0/2085634577 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff720159970 0x7ff7200b3950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:02.866 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:02.860+0000 7ff725e55640 1 -- 172.21.15.16:0/2085634577 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff72015d680 msgr2=0x7ff7201f3a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:02.866 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:02.860+0000 7ff725e55640 1 --2- 172.21.15.16:0/2085634577 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff72015d680 0x7ff7201f3a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:02.866 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:02.860+0000 7ff725e55640 1 -- 172.21.15.16:0/2085634577 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff720159970 msgr2=0x7ff7200b3950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:02.866 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:02.860+0000 7ff725e55640 1 --2- 172.21.15.16:0/2085634577 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff720159970 0x7ff7200b3950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:02.866 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:02.860+0000 7ff725e55640 1 -- 172.21.15.16:0/2085634577 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff718009b30 con 0x7ff72015dad0 2023-05-22T16:05:02.866 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:02.860+0000 7ff725e55640 1 --2- 172.21.15.16:0/2085634577 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff72015dad0 0x7ff7201f7730 secure :-1 s=READY pgs=2318 cs=0 l=1 rev1=1 crypto rx=0x7ff71000b410 tx=0x7ff71000b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:02.866 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:02.860+0000 7ff71dffb640 1 -- 172.21.15.16:0/2085634577 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff710004020 con 0x7ff72015dad0 2023-05-22T16:05:02.866 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:02.860+0000 7ff71dffb640 1 -- 172.21.15.16:0/2085634577 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff710004180 con 0x7ff72015dad0 2023-05-22T16:05:02.867 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:02.860+0000 7ff7278b8640 1 -- 172.21.15.16:0/2085634577 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7200cae20 con 0x7ff72015dad0 2023-05-22T16:05:02.867 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:02.860+0000 7ff71dffb640 1 -- 172.21.15.16:0/2085634577 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff710011630 con 0x7ff72015dad0 2023-05-22T16:05:02.867 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:02.860+0000 7ff7278b8640 1 -- 172.21.15.16:0/2085634577 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7200cac90 con 0x7ff72015dad0 2023-05-22T16:05:02.868 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:02.864+0000 7ff7278b8640 1 -- 172.21.15.16:0/2085634577 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff72006bdd0 con 0x7ff72015dad0 2023-05-22T16:05:02.868 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:02.864+0000 7ff71dffb640 1 -- 172.21.15.16:0/2085634577 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 88) v1 ==== 84463+0+0 (secure 0 0 0) 0x7ff710011790 con 0x7ff72015dad0 2023-05-22T16:05:02.868 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:02.864+0000 7ff71dffb640 1 --2- 172.21.15.16:0/2085634577 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7ff70c065610 0x7ff70c067ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:02.868 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:02.864+0000 7ff71dffb640 1 -- 172.21.15.16:0/2085634577 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(180..180 src has 1..180) v4 ==== 6782+0+0 (secure 0 0 0) 0x7ff710082cb0 con 0x7ff72015dad0 2023-05-22T16:05:02.869 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:02.864+0000 7ff725654640 1 --2- 172.21.15.16:0/2085634577 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7ff70c065610 0x7ff70c067ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:02.869 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:02.864+0000 7ff725654640 1 --2- 172.21.15.16:0/2085634577 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7ff70c065610 0x7ff70c067ac0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7ff7180ae000 tx=0x7ff7180a7070 comp rx=0 tx=0).ready entity=mgr.8746 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:02.870 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:02.868+0000 7ff71dffb640 1 -- 172.21.15.16:0/2085634577 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7ff710050270 con 0x7ff72015dad0 2023-05-22T16:05:03.062 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:03.060+0000 7ff7278b8640 1 -- 172.21.15.16:0/2085634577 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7ff72004dbe0 con 0x7ff72015dad0 2023-05-22T16:05:03.236 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:03.232+0000 7ff71dffb640 1 -- 172.21.15.16:0/2085634577 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mgrmap(e 89) v1 ==== 84446+0+0 (secure 0 0 0) 0x7ff710050640 con 0x7ff72015dad0 2023-05-22T16:05:03.237 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:03.236+0000 7ff71dffb640 1 -- 172.21.15.16:0/2085634577 <== mon.0 v2:172.21.15.16:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v89) v1 ==== 93+0+0 (secure 0 0 0) 0x7ff710053ca0 con 0x7ff72015dad0 2023-05-22T16:05:03.240 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:03.236+0000 7ff7278b8640 1 -- 172.21.15.16:0/2085634577 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7ff70c065610 msgr2=0x7ff70c067ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:03.240 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:03.236+0000 7ff7278b8640 1 --2- 172.21.15.16:0/2085634577 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7ff70c065610 0x7ff70c067ac0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7ff7180ae000 tx=0x7ff7180a7070 comp rx=0 tx=0).stop 2023-05-22T16:05:03.240 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:03.236+0000 7ff7278b8640 1 -- 172.21.15.16:0/2085634577 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff72015dad0 msgr2=0x7ff7201f7730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:03.240 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:03.236+0000 7ff7278b8640 1 --2- 172.21.15.16:0/2085634577 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff72015dad0 0x7ff7201f7730 secure :-1 s=READY pgs=2318 cs=0 l=1 rev1=1 crypto rx=0x7ff71000b410 tx=0x7ff71000b8d0 comp rx=0 tx=0).stop 2023-05-22T16:05:03.241 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:03.236+0000 7ff7278b8640 1 -- 172.21.15.16:0/2085634577 shutdown_connections 2023-05-22T16:05:03.241 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:03.236+0000 7ff7278b8640 1 --2- 172.21.15.16:0/2085634577 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7ff70c065610 0x7ff70c067ac0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:03.241 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:03.236+0000 7ff7278b8640 1 --2- 172.21.15.16:0/2085634577 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff72015dad0 0x7ff7201f7730 unknown :-1 s=CLOSED pgs=2318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:03.241 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:03.236+0000 7ff7278b8640 1 --2- 172.21.15.16:0/2085634577 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff72015d680 0x7ff7201f3a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:03.241 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:03.236+0000 7ff7278b8640 1 --2- 172.21.15.16:0/2085634577 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff720159970 0x7ff7200b3950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:03.241 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:03.236+0000 7ff7278b8640 1 -- 172.21.15.16:0/2085634577 >> 172.21.15.16:0/2085634577 conn(0x7ff720069720 msgr2=0x7ff720069af0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:05:03.241 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:03.236+0000 7ff7278b8640 1 -- 172.21.15.16:0/2085634577 shutdown_connections 2023-05-22T16:05:03.242 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:03.236+0000 7ff7278b8640 1 -- 172.21.15.16:0/2085634577 wait complete. 2023-05-22T16:05:03.260 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-05-22T16:05:03.260 DEBUG:teuthology.orchestra.run.smithi016:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-05-22T16:05:03.310 DEBUG:teuthology.orchestra.run.smithi062:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-05-22T16:05:03.360 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-05-22T16:05:03.361 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-05-22T16:05:03.361 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2023-05-22T16:05:03.363 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T16:05:03.360+0000 7fdbee8f1180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-05-22T16:05:03.366 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-05-22T16:05:03.367 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T16:05:03.364+0000 7f1f8c5e7180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-05-22T16:05:03.488 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:03.484+0000 7f1ee09db640 1 Processor -- start 2023-05-22T16:05:03.488 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:03.484+0000 7f1ee09db640 1 -- start start 2023-05-22T16:05:03.489 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:03.488+0000 7f1ee09db640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f1edc159cd0 0x7f1edc15a0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:03.489 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:03.488+0000 7f1ee09db640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1edc0cc6f0 0x7f1edc0cfe30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:03.489 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:03.488+0000 7f1ee09db640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1edc0cf9a0 con 0x7f1edc0b2880 2023-05-22T16:05:03.489 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:03.488+0000 7f1ee09db640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1edc15aba0 con 0x7f1edc159cd0 2023-05-22T16:05:03.490 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:03.488+0000 7f1ee09db640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1edc15ae80 con 0x7f1edc0cc6f0 2023-05-22T16:05:03.490 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:03.488+0000 7f1eda59c640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f1edc159cd0 0x7f1edc15a0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:03.491 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:03.488+0000 7f1edad9d640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1edc0cc6f0 0x7f1edc0cfe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:03.491 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:03.488+0000 7f1ed9d9b640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f1edc0b2880 0x7f1edc0cbfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.16:37518/0 (socket says 172.21.15.16:37518) 2023-05-22T16:05:03.491 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:03.488+0000 7f1ed9d9b640 1 -- 172.21.15.16:0/1216669714 learned_addr learned my addr 172.21.15.16:0/1216669714 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T16:05:03.491 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:03.488+0000 7f1ed959a640 1 -- 172.21.15.16:0/1216669714 <== mon.1 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 555618762 0 0) 0x7f1edc0cf9a0 con 0x7f1edc0b2880 2023-05-22T16:05:03.491 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:03.488+0000 7f1ed959a640 1 -- 172.21.15.16:0/1216669714 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ec8003530 con 0x7f1edc0b2880 2023-05-22T16:05:03.491 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:03.488+0000 7f1edad9d640 1 -- 172.21.15.16:0/1216669714 >> v2:172.21.15.62:3300/0 conn(0x7f1edc159cd0 msgr2=0x7f1edc15a0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:05:03.491 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:03.488+0000 7f1edad9d640 1 --2- 172.21.15.16:0/1216669714 >> v2:172.21.15.62:3300/0 conn(0x7f1edc159cd0 0x7f1edc15a0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:03.492 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:03.488+0000 7f1edad9d640 1 -- 172.21.15.16:0/1216669714 >> v1:172.21.15.62:6789/0 conn(0x7f1edc0b2880 legacy=0x7f1edc0cbfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:03.492 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:03.488+0000 7f1edad9d640 1 -- 172.21.15.16:0/1216669714 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1edc15b160 con 0x7f1edc0cc6f0 2023-05-22T16:05:03.492 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:03.488+0000 7f1edad9d640 1 --2- 172.21.15.16:0/1216669714 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1edc0cc6f0 0x7f1edc0cfe30 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f1ecc0097c0 tx=0x7f1ecc01c450 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b681bf6d2679268d server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:03.492 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:03.492+0000 7f1ed959a640 1 -- 172.21.15.16:0/1216669714 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ecc0ad660 con 0x7f1edc0cc6f0 2023-05-22T16:05:03.493 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:03.492+0000 7f1ed959a640 1 -- 172.21.15.16:0/1216669714 <== mon.2 v2:172.21.15.62:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ecc0ad7c0 con 0x7f1edc0cc6f0 2023-05-22T16:05:03.493 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:03.492+0000 7f1ed959a640 1 -- 172.21.15.16:0/1216669714 <== mon.2 v2:172.21.15.62:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ecc0adb30 con 0x7f1edc0cc6f0 2023-05-22T16:05:03.493 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:03.492+0000 7f1ee09db640 1 -- 172.21.15.16:0/1216669714 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1edc0cc6f0 msgr2=0x7f1edc0cfe30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:03.494 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:03.492+0000 7f1ee09db640 1 --2- 172.21.15.16:0/1216669714 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1edc0cc6f0 0x7f1edc0cfe30 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f1ecc0097c0 tx=0x7f1ecc01c450 comp rx=0 tx=0).stop 2023-05-22T16:05:03.494 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:03.492+0000 7f1ee09db640 1 -- 172.21.15.16:0/1216669714 shutdown_connections 2023-05-22T16:05:03.494 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:03.492+0000 7f1ee09db640 1 --2- 172.21.15.16:0/1216669714 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1edc0cc6f0 0x7f1edc0cfe30 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:03.494 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:03.492+0000 7f1ee09db640 1 --2- 172.21.15.16:0/1216669714 >> v2:172.21.15.62:3300/0 conn(0x7f1edc159cd0 0x7f1edc15a0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:03.494 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:03.492+0000 7f1ee09db640 1 -- 172.21.15.16:0/1216669714 >> 172.21.15.16:0/1216669714 conn(0x7f1edc06f5c0 msgr2=0x7f1edc06f9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:05:03.494 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:03.492+0000 7f1ee09db640 1 -- 172.21.15.16:0/1216669714 shutdown_connections 2023-05-22T16:05:03.494 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:03.492+0000 7f1ee09db640 1 -- 172.21.15.16:0/1216669714 wait complete. 2023-05-22T16:05:03.496 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:03.492+0000 7f1ee09db640 1 Processor -- start 2023-05-22T16:05:03.496 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:03.492+0000 7f1ee09db640 1 -- start start 2023-05-22T16:05:03.496 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:03.492+0000 7f1ee09db640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1edc0b2880 0x7f1edc0b3090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:03.496 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:03.492+0000 7f1ee09db640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1edc0cc6f0 0x7f1edc149210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:03.497 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:03.492+0000 7f1ee09db640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1edc159cd0 0x7f1edc0c99d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:03.497 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:03.492+0000 7f1ee09db640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1edc0c76d0 con 0x7f1edc159cd0 2023-05-22T16:05:03.497 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:03.492+0000 7f1ee09db640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1edc0c79b0 con 0x7f1edc0cc6f0 2023-05-22T16:05:03.497 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:03.492+0000 7f1ee09db640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1edc0c7c90 con 0x7f1edc0b2880 2023-05-22T16:05:03.497 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:03.492+0000 7f1edad9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1edc159cd0 0x7f1edc0c99d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:03.497 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:03.492+0000 7f1edad9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1edc159cd0 0x7f1edc0c99d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46046/0 (socket says 172.21.15.16:46046) 2023-05-22T16:05:03.497 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:03.492+0000 7f1edad9d640 1 -- 172.21.15.16:0/3641822413 learned_addr learned my addr 172.21.15.16:0/3641822413 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:05:03.497 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:03.492+0000 7f1eda59c640 1 --2- 172.21.15.16:0/3641822413 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1edc0b2880 0x7f1edc0b3090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:03.498 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:03.492+0000 7f1ed9d9b640 1 --2- 172.21.15.16:0/3641822413 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1edc0cc6f0 0x7f1edc149210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:03.498 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:03.492+0000 7f1edad9d640 1 -- 172.21.15.16:0/3641822413 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1edc0b2880 msgr2=0x7f1edc0b3090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:03.498 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:03.492+0000 7f1edad9d640 1 --2- 172.21.15.16:0/3641822413 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1edc0b2880 0x7f1edc0b3090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:03.498 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:03.492+0000 7f1edad9d640 1 -- 172.21.15.16:0/3641822413 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1edc0cc6f0 msgr2=0x7f1edc149210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:03.498 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:03.492+0000 7f1edad9d640 1 --2- 172.21.15.16:0/3641822413 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1edc0cc6f0 0x7f1edc149210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:03.498 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:03.492+0000 7f1edad9d640 1 -- 172.21.15.16:0/3641822413 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ecc009420 con 0x7f1edc159cd0 2023-05-22T16:05:03.498 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:03.492+0000 7f1edad9d640 1 --2- 172.21.15.16:0/3641822413 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1edc159cd0 0x7f1edc0c99d0 secure :-1 s=READY pgs=2322 cs=0 l=1 rev1=1 crypto rx=0x7f1ecc01a080 tx=0x7f1ecc0191f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:03.499 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:03.492+0000 7f1ed2ffd640 1 -- 172.21.15.16:0/3641822413 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ecc0b6600 con 0x7f1edc159cd0 2023-05-22T16:05:03.499 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:03.492+0000 7f1ed2ffd640 1 -- 172.21.15.16:0/3641822413 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ecc0b6760 con 0x7f1edc159cd0 2023-05-22T16:05:03.499 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:03.492+0000 7f1ed2ffd640 1 -- 172.21.15.16:0/3641822413 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ecc0b6ad0 con 0x7f1edc159cd0 2023-05-22T16:05:03.499 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:03.492+0000 7f1ee09db640 1 -- 172.21.15.16:0/3641822413 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1edc0d0d10 con 0x7f1edc159cd0 2023-05-22T16:05:03.499 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:03.492+0000 7f1ee09db640 1 -- 172.21.15.16:0/3641822413 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1edc0d0b80 con 0x7f1edc159cd0 2023-05-22T16:05:03.500 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:03.496+0000 7f1ee09db640 1 -- 172.21.15.16:0/3641822413 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1edc0d0ea0 con 0x7f1edc159cd0 2023-05-22T16:05:03.500 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:03.496+0000 7f1ed2ffd640 1 -- 172.21.15.16:0/3641822413 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 89) v1 ==== 84446+0+0 (secure 0 0 0) 0x7f1ecc0b4040 con 0x7f1edc159cd0 2023-05-22T16:05:03.500 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:03.496+0000 7f1ed2ffd640 1 --2- 172.21.15.16:0/3641822413 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f1ebc0655c0 0x7f1ebc067a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:03.500 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:03.496+0000 7f1eda59c640 1 -- 172.21.15.16:0/3641822413 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f1ebc0655c0 msgr2=0x7f1ebc067a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/65890 2023-05-22T16:05:03.500 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:03.496+0000 7f1ed2ffd640 1 -- 172.21.15.16:0/3641822413 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(180..180 src has 1..180) v4 ==== 6782+0+0 (secure 0 0 0) 0x7f1ecc11dce0 con 0x7f1edc159cd0 2023-05-22T16:05:03.501 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:03.496+0000 7f1eda59c640 1 --2- 172.21.15.16:0/3641822413 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f1ebc0655c0 0x7f1ebc067a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-05-22T16:05:03.502 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:03.500+0000 7f1ed2ffd640 1 -- 172.21.15.16:0/3641822413 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f1ecc0eb390 con 0x7f1edc159cd0 2023-05-22T16:05:03.655 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:03.652+0000 7f1ee09db640 1 -- 172.21.15.16:0/3641822413 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1edc0c81b0 con 0x7f1edc159cd0 2023-05-22T16:05:03.657 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:03.656+0000 7f1ed2ffd640 1 -- 172.21.15.16:0/3641822413 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v180) v1 ==== 74+0+18623 (secure 0 0 0) 0x7f1edc0c81b0 con 0x7f1edc159cd0 2023-05-22T16:05:03.657 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T16:05:03.658 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":180,"fsid":"b2719834-7847-4d98-90a6-f65d57f85c49","created":"2023-05-22T15:26:25.259314+0000","modified":"2023-05-22T16:03:52.657865+0000","last_up_change":"2023-05-22T15:26:36.448524+0000","last_in_change":"2023-05-22T15:26:27.675668+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":23,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-05-22T15:26:38.019781+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-05-22T15:26:43.570514+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":22,"pool_name":"cephfs_metadata","create_time":"2023-05-22T16:03:37.371980+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"178","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_data","create_time":"2023-05-22T16:03:37.756887+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"179","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9e7ff854-b25f-4cbb-a5c2-1799d9c53d2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":37021},{"type":"v1","addr":"172.21.15.16:6825","nonce":37021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":37021},{"type":"v1","addr":"172.21.15.16:6827","nonce":37021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":37021},{"type":"v1","addr":"172.21.15.16:6831","nonce":37021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":37021},{"type":"v1","addr":"172.21.15.16:6829","nonce":37021}]},"public_addr":"172.21.15.16:6825/37021","cluster_addr":"172.21.15.16:6827/37021","heartbeat_back_addr":"172.21.15.16:6831/37021","heartbeat_front_addr":"172.21.15.16:6829/37021","state":["exists","up"]},{"osd":1,"uuid":"4702fbc9-a927-4565-aca5-41dc69cc8264","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":37008},{"type":"v1","addr":"172.21.15.16:6801","nonce":37008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":37008},{"type":"v1","addr":"172.21.15.16:6803","nonce":37008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":37008},{"type":"v1","addr":"172.21.15.16:6807","nonce":37008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":37008},{"type":"v1","addr":"172.21.15.16:6805","nonce":37008}]},"public_addr":"172.21.15.16:6801/37008","cluster_addr":"172.21.15.16:6803/37008","heartbeat_back_addr":"172.21.15.16:6807/37008","heartbeat_front_addr":"172.21.15.16:6805/37008","state":["exists","up"]},{"osd":2,"uuid":"1e7bba46-4eba-47ef-bb81-5e2d371638ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":37009},{"type":"v1","addr":"172.21.15.16:6809","nonce":37009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":37009},{"type":"v1","addr":"172.21.15.16:6811","nonce":37009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":37009},{"type":"v1","addr":"172.21.15.16:6815","nonce":37009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":37009},{"type":"v1","addr":"172.21.15.16:6813","nonce":37009}]},"public_addr":"172.21.15.16:6809/37009","cluster_addr":"172.21.15.16:6811/37009","heartbeat_back_addr":"172.21.15.16:6815/37009","heartbeat_front_addr":"172.21.15.16:6813/37009","state":["exists","up"]},{"osd":3,"uuid":"383cacc0-5a88-4904-990e-498ca59ef48a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":37020},{"type":"v1","addr":"172.21.15.16:6817","nonce":37020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":37020},{"type":"v1","addr":"172.21.15.16:6819","nonce":37020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":37020},{"type":"v1","addr":"172.21.15.16:6823","nonce":37020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":37020},{"type":"v1","addr":"172.21.15.16:6821","nonce":37020}]},"public_addr":"172.21.15.16:6817/37020","cluster_addr":"172.21.15.16:6819/37020","heartbeat_back_addr":"172.21.15.16:6823/37020","heartbeat_front_addr":"172.21.15.16:6821/37020","state":["exists","up"]},{"osd":4,"uuid":"970500ef-95b7-42c1-b080-eed4fae7013d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":37353},{"type":"v1","addr":"172.21.15.62:6817","nonce":37353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6819","nonce":37353},{"type":"v1","addr":"172.21.15.62:6822","nonce":37353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37353},{"type":"v1","addr":"172.21.15.62:6831","nonce":37353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37353},{"type":"v1","addr":"172.21.15.62:6826","nonce":37353}]},"public_addr":"172.21.15.62:6817/37353","cluster_addr":"172.21.15.62:6822/37353","heartbeat_back_addr":"172.21.15.62:6831/37353","heartbeat_front_addr":"172.21.15.62:6826/37353","state":["exists","up"]},{"osd":5,"uuid":"022d67db-d4d8-445b-b1b9-8b446a00f5c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":37352},{"type":"v1","addr":"172.21.15.62:6801","nonce":37352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":37352},{"type":"v1","addr":"172.21.15.62:6803","nonce":37352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":37352},{"type":"v1","addr":"172.21.15.62:6807","nonce":37352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":37352},{"type":"v1","addr":"172.21.15.62:6805","nonce":37352}]},"public_addr":"172.21.15.62:6801/37352","cluster_addr":"172.21.15.62:6803/37352","heartbeat_back_addr":"172.21.15.62:6807/37352","heartbeat_front_addr":"172.21.15.62:6805/37352","state":["exists","up"]},{"osd":6,"uuid":"42bd372b-6688-4a1b-8407-e16e8c266e33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":37354},{"type":"v1","addr":"172.21.15.62:6820","nonce":37354}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":37354},{"type":"v1","addr":"172.21.15.62:6823","nonce":37354}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6829","nonce":37354},{"type":"v1","addr":"172.21.15.62:6830","nonce":37354}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6825","nonce":37354},{"type":"v1","addr":"172.21.15.62:6827","nonce":37354}]},"public_addr":"172.21.15.62:6820/37354","cluster_addr":"172.21.15.62:6823/37354","heartbeat_back_addr":"172.21.15.62:6830/37354","heartbeat_front_addr":"172.21.15.62:6827/37354","state":["exists","up"]},{"osd":7,"uuid":"ac70b2a3-f41b-4bf2-b227-7f2cbb55426d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37355},{"type":"v1","addr":"172.21.15.62:6809","nonce":37355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37355},{"type":"v1","addr":"172.21.15.62:6811","nonce":37355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":37355},{"type":"v1","addr":"172.21.15.62:6815","nonce":37355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":37355},{"type":"v1","addr":"172.21.15.62:6813","nonce":37355}]},"public_addr":"172.21.15.62:6809/37355","cluster_addr":"172.21.15.62:6811/37355","heartbeat_back_addr":"172.21.15.62:6815/37355","heartbeat_front_addr":"172.21.15.62:6813/37355","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.368896+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.107273+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.276888+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.287502+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.124791+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.251937+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.255679+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.232797+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:0/2495910544":"2023-05-23T16:03:52.657800+0000","172.21.15.16:0/1761665129":"2023-05-23T16:03:52.657800+0000","172.21.15.16:6833/65890":"2023-05-23T16:03:52.657800+0000","172.21.15.16:6832/65890":"2023-05-23T16:03:52.657800+0000","172.21.15.16:0/4138270792":"2023-05-23T16:03:52.657800+0000","172.21.15.16:0/2442191907":"2023-05-23T16:03:52.657800+0000","172.21.15.16:0/3487890192":"2023-05-23T16:03:52.657800+0000","172.21.15.16:0/3247164174":"2023-05-23T16:03:52.657800+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":17,"snaps":[{"begin":2,"length":2}]},{"pool":21,"snaps":[{"begin":25,"length":9}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-05-22T16:05:03.659 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:03.656+0000 7f1ee09db640 1 -- 172.21.15.16:0/3641822413 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f1ebc0655c0 msgr2=0x7f1ebc067a70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-05-22T16:05:03.660 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:03.656+0000 7f1ee09db640 1 --2- 172.21.15.16:0/3641822413 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f1ebc0655c0 0x7f1ebc067a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:03.660 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:03.656+0000 7f1ee09db640 1 -- 172.21.15.16:0/3641822413 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1edc159cd0 msgr2=0x7f1edc0c99d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:03.660 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:03.656+0000 7f1ee09db640 1 --2- 172.21.15.16:0/3641822413 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1edc159cd0 0x7f1edc0c99d0 secure :-1 s=READY pgs=2322 cs=0 l=1 rev1=1 crypto rx=0x7f1ecc01a080 tx=0x7f1ecc0191f0 comp rx=0 tx=0).stop 2023-05-22T16:05:03.660 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:03.656+0000 7f1ee09db640 1 -- 172.21.15.16:0/3641822413 shutdown_connections 2023-05-22T16:05:03.660 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:03.656+0000 7f1ee09db640 1 --2- 172.21.15.16:0/3641822413 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f1ebc0655c0 0x7f1ebc067a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:03.660 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:03.656+0000 7f1ee09db640 1 --2- 172.21.15.16:0/3641822413 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1edc159cd0 0x7f1edc0c99d0 unknown :-1 s=CLOSED pgs=2322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:03.661 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:03.656+0000 7f1ee09db640 1 --2- 172.21.15.16:0/3641822413 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1edc0cc6f0 0x7f1edc149210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:03.661 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:03.656+0000 7f1ee09db640 1 --2- 172.21.15.16:0/3641822413 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1edc0b2880 0x7f1edc0b3090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:03.661 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:03.656+0000 7f1ee09db640 1 -- 172.21.15.16:0/3641822413 >> 172.21.15.16:0/3641822413 conn(0x7f1edc06f5c0 msgr2=0x7f1edc0cce00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:05:03.661 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:03.656+0000 7f1ee09db640 1 -- 172.21.15.16:0/3641822413 shutdown_connections 2023-05-22T16:05:03.661 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:03.656+0000 7f1ee09db640 1 -- 172.21.15.16:0/3641822413 wait complete. 2023-05-22T16:05:03.676 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-05-22T15:26:38.019781+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-05-22T15:26:43.570514+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 22, 'pool_name': 'cephfs_metadata', 'create_time': '2023-05-22T16:03:37.371980+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '178', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 23, 'pool_name': 'cephfs_data', 'create_time': '2023-05-22T16:03:37.756887+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '179', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-05-22T16:05:03.677 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-05-22T16:05:03.754 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T16:05:03.752+0000 7fdbee8f1180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-05-22T16:05:03.758 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T16:05:03.756+0000 7f1f8c5e7180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-05-22T16:05:03.797 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:03.796+0000 7f2ec8d34640 1 Processor -- start 2023-05-22T16:05:03.797 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:03.796+0000 7f2ec8d34640 1 -- start start 2023-05-22T16:05:03.798 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:03.796+0000 7f2ec8d34640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f2ec4071640 0x7f2ec4160090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:03.798 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:03.796+0000 7f2ec8d34640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ec4069ae0 con 0x7f2ec41605d0 2023-05-22T16:05:03.799 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:03.796+0000 7f2ec8d34640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ec4069dc0 con 0x7f2ec40ce480 2023-05-22T16:05:03.799 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:03.796+0000 7f2ec8d34640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ec415f990 con 0x7f2ec4071640 2023-05-22T16:05:03.799 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:03.796+0000 7f2ec2d9d640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f2ec41605d0 0x7f2ec415c270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:45428/0 (socket says 172.21.15.16:45428) 2023-05-22T16:05:03.799 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:03.796+0000 7f2ec2d9d640 1 -- 172.21.15.16:0/3761791950 learned_addr learned my addr 172.21.15.16:0/3761791950 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T16:05:03.799 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:03.796+0000 7f2ec1d9b640 1 --2- 172.21.15.16:0/3761791950 >> v2:172.21.15.62:3300/0 conn(0x7f2ec4071640 0x7f2ec4160090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:03.799 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:03.796+0000 7f2ec159a640 1 -- 172.21.15.16:0/3761791950 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2980754609 0 0) 0x7f2ec4069ae0 con 0x7f2ec41605d0 2023-05-22T16:05:03.799 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:03.796+0000 7f2ec159a640 1 -- 172.21.15.16:0/3761791950 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2eac003530 con 0x7f2ec41605d0 2023-05-22T16:05:03.801 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:03.796+0000 7f2ec159a640 1 -- 172.21.15.16:0/3761791950 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2470324267 0 0) 0x7f2eac003530 con 0x7f2ec41605d0 2023-05-22T16:05:03.801 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:03.796+0000 7f2ec159a640 1 -- 172.21.15.16:0/3761791950 >> v2:172.21.15.62:3300/0 conn(0x7f2ec4071640 msgr2=0x7f2ec4160090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:05:03.801 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:03.796+0000 7f2ec159a640 1 --2- 172.21.15.16:0/3761791950 >> v2:172.21.15.62:3300/0 conn(0x7f2ec4071640 0x7f2ec4160090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:03.802 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:03.796+0000 7f2ec159a640 1 -- 172.21.15.16:0/3761791950 >> v1:172.21.15.62:6789/0 conn(0x7f2ec40ce480 legacy=0x7f2ec40ce850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:03.802 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:03.796+0000 7f2ec159a640 1 -- 172.21.15.16:0/3761791950 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ec406a0a0 con 0x7f2ec41605d0 2023-05-22T16:05:03.802 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:03.796+0000 7f2ec159a640 1 -- 172.21.15.16:0/3761791950 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7f2ebc001a00 con 0x7f2ec41605d0 2023-05-22T16:05:03.802 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:03.796+0000 7f2ec159a640 1 -- 172.21.15.16:0/3761791950 >> v1:172.21.15.16:6789/0 conn(0x7f2ec41605d0 legacy=0x7f2ec415c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:03.802 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:03.796+0000 7f2ec159a640 1 --2- 172.21.15.16:0/3761791950 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2eac003980 0x7f2eac003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:03.802 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:03.796+0000 7f2ec159a640 1 --2- 172.21.15.16:0/3761791950 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2eac0047d0 0x7f2eac004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:03.803 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:03.796+0000 7f2ec159a640 1 --2- 172.21.15.16:0/3761791950 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2eac005180 0x7f2eac00b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:03.803 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:03.796+0000 7f2ec159a640 1 -- 172.21.15.16:0/3761791950 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ec415f990 con 0x7f2eac0047d0 2023-05-22T16:05:03.803 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:03.796+0000 7f2ec159a640 1 -- 172.21.15.16:0/3761791950 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2eac00c000 con 0x7f2eac003980 2023-05-22T16:05:03.803 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:03.796+0000 7f2ec159a640 1 -- 172.21.15.16:0/3761791950 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2eac00c1c0 con 0x7f2eac005180 2023-05-22T16:05:03.803 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:03.796+0000 7f2ec1d9b640 1 --2- 172.21.15.16:0/3761791950 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2eac0047d0 0x7f2eac004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:03.803 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:03.796+0000 7f2ec259c640 1 --2- 172.21.15.16:0/3761791950 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2eac003980 0x7f2eac003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:03.803 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:03.796+0000 7f2ec2d9d640 1 --2- 172.21.15.16:0/3761791950 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2eac005180 0x7f2eac00b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:03.804 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:03.796+0000 7f2ec1d9b640 1 -- 172.21.15.16:0/3761791950 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2eac005180 msgr2=0x7f2eac00b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:05:03.804 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:03.796+0000 7f2ec1d9b640 1 --2- 172.21.15.16:0/3761791950 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2eac005180 0x7f2eac00b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:03.804 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:03.796+0000 7f2ec1d9b640 1 -- 172.21.15.16:0/3761791950 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2eac003980 msgr2=0x7f2eac003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:05:03.804 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:03.796+0000 7f2ec1d9b640 1 --2- 172.21.15.16:0/3761791950 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2eac003980 0x7f2eac003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:03.804 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:03.796+0000 7f2ec1d9b640 1 -- 172.21.15.16:0/3761791950 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2eac003530 con 0x7f2eac0047d0 2023-05-22T16:05:03.804 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:03.796+0000 7f2ec2d9d640 1 -- 172.21.15.16:0/3761791950 reap_dead start 2023-05-22T16:05:03.807 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:03.800+0000 7f2ec1d9b640 1 --2- 172.21.15.16:0/3761791950 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2eac0047d0 0x7f2eac004c40 secure :-1 s=READY pgs=2324 cs=0 l=1 rev1=1 crypto rx=0x7f2eb8009ae0 tx=0x7f2eb801ca50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f95d98ff7a5cca47 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:03.807 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:03.800+0000 7f2ec159a640 1 -- 172.21.15.16:0/3761791950 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2eb80a6460 con 0x7f2eac0047d0 2023-05-22T16:05:03.807 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:03.800+0000 7f2ec159a640 1 -- 172.21.15.16:0/3761791950 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2eb80a65c0 con 0x7f2eac0047d0 2023-05-22T16:05:03.807 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:03.800+0000 7f2ec159a640 1 -- 172.21.15.16:0/3761791950 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2eb80a6970 con 0x7f2eac0047d0 2023-05-22T16:05:03.808 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:03.800+0000 7f2ec8d34640 1 -- 172.21.15.16:0/3761791950 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2eac0047d0 msgr2=0x7f2eac004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:03.808 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:03.800+0000 7f2ec8d34640 1 --2- 172.21.15.16:0/3761791950 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2eac0047d0 0x7f2eac004c40 secure :-1 s=READY pgs=2324 cs=0 l=1 rev1=1 crypto rx=0x7f2eb8009ae0 tx=0x7f2eb801ca50 comp rx=0 tx=0).stop 2023-05-22T16:05:03.808 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:03.800+0000 7f2ec8d34640 1 -- 172.21.15.16:0/3761791950 shutdown_connections 2023-05-22T16:05:03.808 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:03.800+0000 7f2ec8d34640 1 --2- 172.21.15.16:0/3761791950 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2eac0047d0 0x7f2eac004c40 unknown :-1 s=CLOSED pgs=2324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:03.808 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:03.800+0000 7f2ec8d34640 1 -- 172.21.15.16:0/3761791950 >> 172.21.15.16:0/3761791950 conn(0x7f2ec40cc870 msgr2=0x7f2ec40cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:05:03.809 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:03.800+0000 7f2ec8d34640 1 -- 172.21.15.16:0/3761791950 shutdown_connections 2023-05-22T16:05:03.809 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:03.800+0000 7f2ec8d34640 1 -- 172.21.15.16:0/3761791950 wait complete. 2023-05-22T16:05:03.809 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:03.800+0000 7f2ec8d34640 1 Processor -- start 2023-05-22T16:05:03.810 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:03.800+0000 7f2ec8d34640 1 -- start start 2023-05-22T16:05:03.810 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:03.804+0000 7f2ec8d34640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2eac0047d0 0x7f2ec406e0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:03.810 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:03.804+0000 7f2ec8d34640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2ec40af2e0 0x7f2ec40af750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:03.810 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:03.804+0000 7f2ec8d34640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2ec4154220 0x7f2ec4154690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:03.811 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:03.804+0000 7f2ec8d34640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ec4071ac0 con 0x7f2eac0047d0 2023-05-22T16:05:03.811 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:03.804+0000 7f2ec8d34640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ec41f61d0 con 0x7f2ec4154220 2023-05-22T16:05:03.811 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:03.804+0000 7f2ec8d34640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ec41583e0 con 0x7f2ec40af2e0 2023-05-22T16:05:03.812 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:03.804+0000 7f2ec259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2eac0047d0 0x7f2ec406e0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:03.815 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:03.804+0000 7f2ec259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2eac0047d0 0x7f2ec406e0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46066/0 (socket says 172.21.15.16:46066) 2023-05-22T16:05:03.815 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:03.804+0000 7f2ec259c640 1 -- 172.21.15.16:0/833247362 learned_addr learned my addr 172.21.15.16:0/833247362 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:05:03.815 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:03.804+0000 7f2ec1d9b640 1 --2- 172.21.15.16:0/833247362 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2ec40af2e0 0x7f2ec40af750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:03.816 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:03.804+0000 7f2ec2d9d640 1 --2- 172.21.15.16:0/833247362 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2ec4154220 0x7f2ec4154690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:03.816 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:03.804+0000 7f2ec259c640 1 -- 172.21.15.16:0/833247362 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2ec40af2e0 msgr2=0x7f2ec40af750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:03.816 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:03.804+0000 7f2ec259c640 1 --2- 172.21.15.16:0/833247362 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2ec40af2e0 0x7f2ec40af750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:03.816 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:03.804+0000 7f2ec259c640 1 -- 172.21.15.16:0/833247362 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2ec4154220 msgr2=0x7f2ec4154690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:03.816 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:03.804+0000 7f2ec259c640 1 --2- 172.21.15.16:0/833247362 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2ec4154220 0x7f2ec4154690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:03.816 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:03.804+0000 7f2ec259c640 1 -- 172.21.15.16:0/833247362 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ec4069720 con 0x7f2eac0047d0 2023-05-22T16:05:03.816 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:03.804+0000 7f2ec259c640 1 --2- 172.21.15.16:0/833247362 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2eac0047d0 0x7f2ec406e0d0 secure :-1 s=READY pgs=2325 cs=0 l=1 rev1=1 crypto rx=0x7f2eb400e4d0 tx=0x7f2eb400e990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:03.817 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:03.804+0000 7f2eb2ffd640 1 -- 172.21.15.16:0/833247362 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2eb40053a0 con 0x7f2eac0047d0 2023-05-22T16:05:03.817 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:03.804+0000 7f2eb2ffd640 1 -- 172.21.15.16:0/833247362 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2eb4003600 con 0x7f2eac0047d0 2023-05-22T16:05:03.817 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:03.804+0000 7f2ec8d34640 1 -- 172.21.15.16:0/833247362 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ec415fb70 con 0x7f2eac0047d0 2023-05-22T16:05:03.817 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:03.804+0000 7f2eb2ffd640 1 -- 172.21.15.16:0/833247362 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2eb4003990 con 0x7f2eac0047d0 2023-05-22T16:05:03.817 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:03.804+0000 7f2ec8d34640 1 -- 172.21.15.16:0/833247362 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ec40698b0 con 0x7f2eac0047d0 2023-05-22T16:05:03.817 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:03.804+0000 7f2ec8d34640 1 -- 172.21.15.16:0/833247362 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ec404dbe0 con 0x7f2eac0047d0 2023-05-22T16:05:03.818 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:03.804+0000 7f2eb2ffd640 1 -- 172.21.15.16:0/833247362 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 89) v1 ==== 84446+0+0 (secure 0 0 0) 0x7f2eb4003af0 con 0x7f2eac0047d0 2023-05-22T16:05:03.818 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:03.808+0000 7f2eb2ffd640 1 --2- 172.21.15.16:0/833247362 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f2ea40655c0 0x7f2ea4067a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:03.818 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:03.808+0000 7f2eb2ffd640 1 -- 172.21.15.16:0/833247362 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(180..180 src has 1..180) v4 ==== 6782+0+0 (secure 0 0 0) 0x7f2eb4083590 con 0x7f2eac0047d0 2023-05-22T16:05:03.819 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:03.808+0000 7f2ec1d9b640 1 -- 172.21.15.16:0/833247362 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f2ea40655c0 msgr2=0x7f2ea4067a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/65890 2023-05-22T16:05:03.819 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:03.808+0000 7f2ec1d9b640 1 --2- 172.21.15.16:0/833247362 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f2ea40655c0 0x7f2ea4067a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-05-22T16:05:03.819 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:03.808+0000 7f2eb2ffd640 1 -- 172.21.15.16:0/833247362 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f2eb4050d50 con 0x7f2eac0047d0 2023-05-22T16:05:03.890 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T16:05:03.884+0000 7fdbee8f1180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-05-22T16:05:03.898 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T16:05:03.896+0000 7f1f8c5e7180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-05-22T16:05:03.965 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:03.964+0000 7f2ec8d34640 1 -- 172.21.15.16:0/833247362 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f2ec4070a10 con 0x7f2eac0047d0 2023-05-22T16:05:03.966 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:03.964+0000 7f2eb2ffd640 1 -- 172.21.15.16:0/833247362 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v180) v1 ==== 93+0+10 (secure 0 0 0) 0x7f2eb4054370 con 0x7f2eac0047d0 2023-05-22T16:05:03.966 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2023-05-22T16:05:03.968 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:03.964+0000 7f2ec8d34640 1 -- 172.21.15.16:0/833247362 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f2ea40655c0 msgr2=0x7f2ea4067a70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-05-22T16:05:03.969 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:03.964+0000 7f2ec8d34640 1 --2- 172.21.15.16:0/833247362 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f2ea40655c0 0x7f2ea4067a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:03.969 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:03.964+0000 7f2ec8d34640 1 -- 172.21.15.16:0/833247362 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2eac0047d0 msgr2=0x7f2ec406e0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:03.969 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:03.964+0000 7f2ec8d34640 1 --2- 172.21.15.16:0/833247362 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2eac0047d0 0x7f2ec406e0d0 secure :-1 s=READY pgs=2325 cs=0 l=1 rev1=1 crypto rx=0x7f2eb400e4d0 tx=0x7f2eb400e990 comp rx=0 tx=0).stop 2023-05-22T16:05:03.969 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:03.964+0000 7f2ec8d34640 1 -- 172.21.15.16:0/833247362 shutdown_connections 2023-05-22T16:05:03.970 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:03.964+0000 7f2ec8d34640 1 --2- 172.21.15.16:0/833247362 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2ec4154220 0x7f2ec4154690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:03.970 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:03.964+0000 7f2ec8d34640 1 --2- 172.21.15.16:0/833247362 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2ec40af2e0 0x7f2ec40af750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:03.970 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:03.964+0000 7f2ec8d34640 1 --2- 172.21.15.16:0/833247362 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f2ea40655c0 0x7f2ea4067a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:03.970 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:03.964+0000 7f2ec8d34640 1 --2- 172.21.15.16:0/833247362 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2eac0047d0 0x7f2ec406e0d0 unknown :-1 s=CLOSED pgs=2325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:03.970 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:03.964+0000 7f2ec8d34640 1 -- 172.21.15.16:0/833247362 >> 172.21.15.16:0/833247362 conn(0x7f2ec40cc870 msgr2=0x7f2ec40ad6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:05:03.970 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:03.968+0000 7f2ec8d34640 1 -- 172.21.15.16:0/833247362 shutdown_connections 2023-05-22T16:05:03.970 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:03.968+0000 7f2ec8d34640 1 -- 172.21.15.16:0/833247362 wait complete. 2023-05-22T16:05:03.984 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-05-22T16:05:04.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.096+0000 7fd3a4d08640 1 Processor -- start 2023-05-22T16:05:04.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.096+0000 7fd3a4d08640 1 -- start start 2023-05-22T16:05:04.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.100+0000 7fd3a4d08640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fd3a015c8b0 0x7fd3a00cee30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:04.103 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.100+0000 7fd3a4d08640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd3a006bf60 0x7fd3a015cc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:04.103 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.100+0000 7fd3a4d08640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3a006c3e0 con 0x7fd3a015d1c0 2023-05-22T16:05:04.103 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.100+0000 7fd3a4d08640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3a015a0c0 con 0x7fd3a006bf60 2023-05-22T16:05:04.103 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.100+0000 7fd3a4d08640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3a00ce160 con 0x7fd3a015c8b0 2023-05-22T16:05:04.103 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.100+0000 7fd397fff640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd3a006bf60 0x7fd3a015cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:04.104 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.100+0000 7fd397fff640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd3a006bf60 0x7fd3a015cc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46082/0 (socket says 172.21.15.16:46082) 2023-05-22T16:05:04.104 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.100+0000 7fd397fff640 1 -- 172.21.15.16:0/1644045182 learned_addr learned my addr 172.21.15.16:0/1644045182 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:05:04.104 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.100+0000 7fd39e59c640 1 --2- 172.21.15.16:0/1644045182 >> v2:172.21.15.62:3300/0 conn(0x7fd3a015c8b0 0x7fd3a00cee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:04.104 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.100+0000 7fd397fff640 1 -- 172.21.15.16:0/1644045182 >> v2:172.21.15.62:3300/0 conn(0x7fd3a015c8b0 msgr2=0x7fd3a00cee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:05:04.104 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.100+0000 7fd397fff640 1 --2- 172.21.15.16:0/1644045182 >> v2:172.21.15.62:3300/0 conn(0x7fd3a015c8b0 0x7fd3a00cee30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:04.104 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.100+0000 7fd397fff640 1 -- 172.21.15.16:0/1644045182 >> v1:172.21.15.16:6789/0 conn(0x7fd3a015d1c0 legacy=0x7fd3a006db80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:04.105 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.100+0000 7fd397fff640 1 -- 172.21.15.16:0/1644045182 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3a006c640 con 0x7fd3a006bf60 2023-05-22T16:05:04.105 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.100+0000 7fd39e59c640 1 --2- 172.21.15.16:0/1644045182 >> v2:172.21.15.62:3300/0 conn(0x7fd3a015c8b0 0x7fd3a00cee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T16:05:04.108 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.100+0000 7fd397fff640 1 --2- 172.21.15.16:0/1644045182 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd3a006bf60 0x7fd3a015cc80 secure :-1 s=READY pgs=2327 cs=0 l=1 rev1=1 crypto rx=0x7fd390009870 tx=0x7fd39001c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3f8557a798a20e3 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:04.108 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.100+0000 7fd39dd9b640 1 -- 172.21.15.16:0/1644045182 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3900ac640 con 0x7fd3a006bf60 2023-05-22T16:05:04.109 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.100+0000 7fd39dd9b640 1 -- 172.21.15.16:0/1644045182 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3900ac7a0 con 0x7fd3a006bf60 2023-05-22T16:05:04.109 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.100+0000 7fd39dd9b640 1 -- 172.21.15.16:0/1644045182 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3900acb30 con 0x7fd3a006bf60 2023-05-22T16:05:04.109 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.100+0000 7fd3a4d08640 1 -- 172.21.15.16:0/1644045182 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd3a006bf60 msgr2=0x7fd3a015cc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:04.109 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.100+0000 7fd3a4d08640 1 --2- 172.21.15.16:0/1644045182 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd3a006bf60 0x7fd3a015cc80 secure :-1 s=READY pgs=2327 cs=0 l=1 rev1=1 crypto rx=0x7fd390009870 tx=0x7fd39001c4d0 comp rx=0 tx=0).stop 2023-05-22T16:05:04.109 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.100+0000 7fd3a4d08640 1 -- 172.21.15.16:0/1644045182 shutdown_connections 2023-05-22T16:05:04.110 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.100+0000 7fd3a4d08640 1 --2- 172.21.15.16:0/1644045182 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd3a006bf60 0x7fd3a015cc80 unknown :-1 s=CLOSED pgs=2327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:04.110 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.100+0000 7fd3a4d08640 1 --2- 172.21.15.16:0/1644045182 >> v2:172.21.15.62:3300/0 conn(0x7fd3a015c8b0 0x7fd3a00cee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:04.110 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.100+0000 7fd3a4d08640 1 -- 172.21.15.16:0/1644045182 >> 172.21.15.16:0/1644045182 conn(0x7fd3a006f5a0 msgr2=0x7fd3a006f9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:05:04.110 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.100+0000 7fd3a4d08640 1 -- 172.21.15.16:0/1644045182 shutdown_connections 2023-05-22T16:05:04.110 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.100+0000 7fd3a4d08640 1 -- 172.21.15.16:0/1644045182 wait complete. 2023-05-22T16:05:04.111 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.104+0000 7fd3a4d08640 1 Processor -- start 2023-05-22T16:05:04.111 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.104+0000 7fd3a4d08640 1 -- start start 2023-05-22T16:05:04.111 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.104+0000 7fd3a4d08640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd3a006bf60 0x7fd3a015fa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:04.111 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.104+0000 7fd3a4d08640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd3a015c8b0 0x7fd3a01593b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:04.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.104+0000 7fd3a4d08640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd3a015d1c0 0x7fd3a0150c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:04.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.104+0000 7fd3a4d08640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3a00b3230 con 0x7fd3a015d1c0 2023-05-22T16:05:04.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.104+0000 7fd3a4d08640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3a00b3510 con 0x7fd3a006bf60 2023-05-22T16:05:04.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.104+0000 7fd3a4d08640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3a00b37f0 con 0x7fd3a015c8b0 2023-05-22T16:05:04.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.104+0000 7fd39ed9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd3a015d1c0 0x7fd3a0150c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:04.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.104+0000 7fd39ed9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd3a015d1c0 0x7fd3a0150c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46088/0 (socket says 172.21.15.16:46088) 2023-05-22T16:05:04.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.104+0000 7fd39ed9d640 1 -- 172.21.15.16:0/389738391 learned_addr learned my addr 172.21.15.16:0/389738391 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:05:04.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.104+0000 7fd39e59c640 1 --2- 172.21.15.16:0/389738391 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd3a006bf60 0x7fd3a015fa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:04.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.104+0000 7fd397fff640 1 --2- 172.21.15.16:0/389738391 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd3a015c8b0 0x7fd3a01593b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:04.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.104+0000 7fd39ed9d640 1 -- 172.21.15.16:0/389738391 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd3a015c8b0 msgr2=0x7fd3a01593b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:04.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.104+0000 7fd39ed9d640 1 --2- 172.21.15.16:0/389738391 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd3a015c8b0 0x7fd3a01593b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:04.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.104+0000 7fd39ed9d640 1 -- 172.21.15.16:0/389738391 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd3a006bf60 msgr2=0x7fd3a015fa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:04.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.104+0000 7fd39ed9d640 1 --2- 172.21.15.16:0/389738391 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd3a006bf60 0x7fd3a015fa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:04.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.104+0000 7fd39ed9d640 1 -- 172.21.15.16:0/389738391 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3900094d0 con 0x7fd3a015d1c0 2023-05-22T16:05:04.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.104+0000 7fd39ed9d640 1 --2- 172.21.15.16:0/389738391 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd3a015d1c0 0x7fd3a0150c70 secure :-1 s=READY pgs=2328 cs=0 l=1 rev1=1 crypto rx=0x7fd39800c730 tx=0x7fd39800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:04.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.104+0000 7fd396ffd640 1 -- 172.21.15.16:0/389738391 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd398003b80 con 0x7fd3a015d1c0 2023-05-22T16:05:04.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.104+0000 7fd396ffd640 1 -- 172.21.15.16:0/389738391 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd398003ce0 con 0x7fd3a015d1c0 2023-05-22T16:05:04.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.104+0000 7fd3a4d08640 1 -- 172.21.15.16:0/389738391 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3a006e290 con 0x7fd3a015d1c0 2023-05-22T16:05:04.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.104+0000 7fd396ffd640 1 -- 172.21.15.16:0/389738391 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd39800aa30 con 0x7fd3a015d1c0 2023-05-22T16:05:04.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.104+0000 7fd3a4d08640 1 -- 172.21.15.16:0/389738391 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3a00ce930 con 0x7fd3a015d1c0 2023-05-22T16:05:04.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.104+0000 7fd3a4d08640 1 -- 172.21.15.16:0/389738391 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3a006e420 con 0x7fd3a015d1c0 2023-05-22T16:05:04.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.108+0000 7fd396ffd640 1 -- 172.21.15.16:0/389738391 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 89) v1 ==== 84446+0+0 (secure 0 0 0) 0x7fd39800ab90 con 0x7fd3a015d1c0 2023-05-22T16:05:04.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.108+0000 7fd396ffd640 1 --2- 172.21.15.16:0/389738391 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fd3800655c0 0x7fd380067a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:04.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.108+0000 7fd39e59c640 1 -- 172.21.15.16:0/389738391 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fd3800655c0 msgr2=0x7fd380067a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/65890 2023-05-22T16:05:04.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.108+0000 7fd396ffd640 1 -- 172.21.15.16:0/389738391 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(180..180 src has 1..180) v4 ==== 6782+0+0 (secure 0 0 0) 0x7fd398082d10 con 0x7fd3a015d1c0 2023-05-22T16:05:04.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.108+0000 7fd39e59c640 1 --2- 172.21.15.16:0/389738391 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fd3800655c0 0x7fd380067a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-05-22T16:05:04.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.116+0000 7fd396ffd640 1 -- 172.21.15.16:0/389738391 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fd398050360 con 0x7fd3a015d1c0 2023-05-22T16:05:04.269 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.268+0000 7fd3a4d08640 1 -- 172.21.15.16:0/389738391 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fd3a004dbe0 con 0x7fd3a015d1c0 2023-05-22T16:05:04.270 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.268+0000 7fd396ffd640 1 -- 172.21.15.16:0/389738391 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v180) v1 ==== 92+0+10 (secure 0 0 0) 0x7fd398053980 con 0x7fd3a015d1c0 2023-05-22T16:05:04.270 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 8 2023-05-22T16:05:04.272 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.268+0000 7fd3a4d08640 1 -- 172.21.15.16:0/389738391 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fd3800655c0 msgr2=0x7fd380067a70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-05-22T16:05:04.272 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.268+0000 7fd3a4d08640 1 --2- 172.21.15.16:0/389738391 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fd3800655c0 0x7fd380067a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:04.273 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.268+0000 7fd3a4d08640 1 -- 172.21.15.16:0/389738391 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd3a015d1c0 msgr2=0x7fd3a0150c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:04.273 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.268+0000 7fd3a4d08640 1 --2- 172.21.15.16:0/389738391 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd3a015d1c0 0x7fd3a0150c70 secure :-1 s=READY pgs=2328 cs=0 l=1 rev1=1 crypto rx=0x7fd39800c730 tx=0x7fd39800cbf0 comp rx=0 tx=0).stop 2023-05-22T16:05:04.273 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.268+0000 7fd3a4d08640 1 -- 172.21.15.16:0/389738391 shutdown_connections 2023-05-22T16:05:04.273 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.268+0000 7fd3a4d08640 1 --2- 172.21.15.16:0/389738391 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fd3800655c0 0x7fd380067a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:04.273 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.268+0000 7fd3a4d08640 1 --2- 172.21.15.16:0/389738391 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd3a015d1c0 0x7fd3a0150c70 unknown :-1 s=CLOSED pgs=2328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:04.273 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.268+0000 7fd3a4d08640 1 --2- 172.21.15.16:0/389738391 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd3a015c8b0 0x7fd3a01593b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:04.274 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.268+0000 7fd3a4d08640 1 --2- 172.21.15.16:0/389738391 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd3a006bf60 0x7fd3a015fa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:04.274 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.268+0000 7fd3a4d08640 1 -- 172.21.15.16:0/389738391 >> 172.21.15.16:0/389738391 conn(0x7fd3a006f5a0 msgr2=0x7fd3a00ccdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:05:04.274 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.268+0000 7fd3a4d08640 1 -- 172.21.15.16:0/389738391 shutdown_connections 2023-05-22T16:05:04.274 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.268+0000 7fd3a4d08640 1 -- 172.21.15.16:0/389738391 wait complete. 2023-05-22T16:05:04.289 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-05-22T16:05:04.328 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T16:05:04.324+0000 7fdbee8f1180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-05-22T16:05:04.354 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T16:05:04.352+0000 7f1f8c5e7180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-05-22T16:05:04.382 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T16:05:04.376+0000 7fdbee8f1180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-05-22T16:05:04.404 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.404+0000 7f61c20ad640 1 Processor -- start 2023-05-22T16:05:04.405 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.404+0000 7f61c20ad640 1 -- start start 2023-05-22T16:05:04.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.404+0000 7f61c20ad640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f61bc15cfa0 0x7f61bc06db00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:04.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.404+0000 7f61c20ad640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f61bc0cf870 0x7f61bc0b1fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:04.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.404+0000 7f61c20ad640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61bc06e040 con 0x7f61bc06e740 2023-05-22T16:05:04.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.404+0000 7f61c20ad640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61bc0cb9f0 con 0x7f61bc15cfa0 2023-05-22T16:05:04.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.404+0000 7f61c20ad640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61bc0b24f0 con 0x7f61bc0cf870 2023-05-22T16:05:04.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.404+0000 7f61bbfff640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f61bc06e740 0x7f61bc070de0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.16:37532/0 (socket says 172.21.15.16:37532) 2023-05-22T16:05:04.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.404+0000 7f61bbfff640 1 -- 172.21.15.16:0/2232531119 learned_addr learned my addr 172.21.15.16:0/2232531119 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T16:05:04.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.404+0000 7f61bb7fe640 1 --2- 172.21.15.16:0/2232531119 >> v2:172.21.15.62:3300/0 conn(0x7f61bc15cfa0 0x7f61bc06db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:04.408 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.404+0000 7f61baffd640 1 --2- 172.21.15.16:0/2232531119 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f61bc0cf870 0x7f61bc0b1fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:04.408 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.404+0000 7f61bb7fe640 1 -- 172.21.15.16:0/2232531119 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f61bc0cf870 msgr2=0x7f61bc0b1fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:05:04.408 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.404+0000 7f61bb7fe640 1 --2- 172.21.15.16:0/2232531119 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f61bc0cf870 0x7f61bc0b1fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:04.408 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.404+0000 7f61bb7fe640 1 -- 172.21.15.16:0/2232531119 >> v1:172.21.15.62:6789/0 conn(0x7f61bc06e740 legacy=0x7f61bc070de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:04.408 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.404+0000 7f61bb7fe640 1 -- 172.21.15.16:0/2232531119 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61bc06e490 con 0x7f61bc15cfa0 2023-05-22T16:05:04.409 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.408+0000 7f61bb7fe640 1 --2- 172.21.15.16:0/2232531119 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f61bc15cfa0 0x7f61bc06db00 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f61ac0097c0 tx=0x7f61ac01c3c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c517f0e339332546 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:04.409 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.408+0000 7f61ba7fc640 1 -- 172.21.15.16:0/2232531119 <== mon.1 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61ac0ac620 con 0x7f61bc15cfa0 2023-05-22T16:05:04.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.408+0000 7f61ba7fc640 1 -- 172.21.15.16:0/2232531119 <== mon.1 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61ac0ac780 con 0x7f61bc15cfa0 2023-05-22T16:05:04.410 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T16:05:04.408+0000 7f1f8c5e7180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-05-22T16:05:04.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.408+0000 7f61ba7fc640 1 -- 172.21.15.16:0/2232531119 <== mon.1 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61ac0acb10 con 0x7f61bc15cfa0 2023-05-22T16:05:04.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.408+0000 7f61c20ad640 1 -- 172.21.15.16:0/2232531119 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f61bc15cfa0 msgr2=0x7f61bc06db00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:04.412 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.408+0000 7f61c20ad640 1 --2- 172.21.15.16:0/2232531119 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f61bc15cfa0 0x7f61bc06db00 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f61ac0097c0 tx=0x7f61ac01c3c0 comp rx=0 tx=0).stop 2023-05-22T16:05:04.412 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.408+0000 7f61c20ad640 1 -- 172.21.15.16:0/2232531119 shutdown_connections 2023-05-22T16:05:04.412 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.408+0000 7f61c20ad640 1 --2- 172.21.15.16:0/2232531119 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f61bc0cf870 0x7f61bc0b1fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:04.412 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.408+0000 7f61c20ad640 1 --2- 172.21.15.16:0/2232531119 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f61bc15cfa0 0x7f61bc06db00 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:04.412 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.408+0000 7f61c20ad640 1 -- 172.21.15.16:0/2232531119 >> 172.21.15.16:0/2232531119 conn(0x7f61bc0cc870 msgr2=0x7f61bc0cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:05:04.413 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.408+0000 7f61c20ad640 1 -- 172.21.15.16:0/2232531119 shutdown_connections 2023-05-22T16:05:04.413 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.408+0000 7f61c20ad640 1 -- 172.21.15.16:0/2232531119 wait complete. 2023-05-22T16:05:04.413 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.408+0000 7f61c20ad640 1 Processor -- start 2023-05-22T16:05:04.413 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.408+0000 7f61c20ad640 1 -- start start 2023-05-22T16:05:04.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.408+0000 7f61c20ad640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f61bc06e740 0x7f61bc0c7730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:04.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.408+0000 7f61c20ad640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f61bc0cf870 0x7f61bc149500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:04.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.408+0000 7f61c20ad640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f61bc15cfa0 0x7f61bc151cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:04.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.408+0000 7f61c20ad640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61bc0cced0 con 0x7f61bc0cf870 2023-05-22T16:05:04.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.408+0000 7f61c20ad640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61bc0cd1b0 con 0x7f61bc15cfa0 2023-05-22T16:05:04.417 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.408+0000 7f61c20ad640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61bc0cd490 con 0x7f61bc06e740 2023-05-22T16:05:04.417 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.408+0000 7f61baffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f61bc0cf870 0x7f61bc149500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:04.417 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.408+0000 7f61baffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f61bc0cf870 0x7f61bc149500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46090/0 (socket says 172.21.15.16:46090) 2023-05-22T16:05:04.417 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.408+0000 7f61baffd640 1 -- 172.21.15.16:0/3964016893 learned_addr learned my addr 172.21.15.16:0/3964016893 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:05:04.417 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.408+0000 7f61bb7fe640 1 --2- 172.21.15.16:0/3964016893 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f61bc06e740 0x7f61bc0c7730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:04.418 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.408+0000 7f61bbfff640 1 --2- 172.21.15.16:0/3964016893 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f61bc15cfa0 0x7f61bc151cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:04.418 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.408+0000 7f61baffd640 1 -- 172.21.15.16:0/3964016893 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f61bc06e740 msgr2=0x7f61bc0c7730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:04.418 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.408+0000 7f61baffd640 1 --2- 172.21.15.16:0/3964016893 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f61bc06e740 0x7f61bc0c7730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:04.418 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.408+0000 7f61baffd640 1 -- 172.21.15.16:0/3964016893 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f61bc15cfa0 msgr2=0x7f61bc151cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:04.418 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.408+0000 7f61baffd640 1 --2- 172.21.15.16:0/3964016893 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f61bc15cfa0 0x7f61bc151cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:04.418 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.412+0000 7f61baffd640 1 -- 172.21.15.16:0/3964016893 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61ac009420 con 0x7f61bc0cf870 2023-05-22T16:05:04.419 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.412+0000 7f61bb7fe640 1 --2- 172.21.15.16:0/3964016893 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f61bc06e740 0x7f61bc0c7730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T16:05:04.419 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.412+0000 7f61baffd640 1 --2- 172.21.15.16:0/3964016893 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f61bc0cf870 0x7f61bc149500 secure :-1 s=READY pgs=2329 cs=0 l=1 rev1=1 crypto rx=0x7f61b000e930 tx=0x7f61b000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:04.419 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.412+0000 7f61a3fff640 1 -- 172.21.15.16:0/3964016893 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61b0009690 con 0x7f61bc0cf870 2023-05-22T16:05:04.419 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.412+0000 7f61a3fff640 1 -- 172.21.15.16:0/3964016893 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61b000ca00 con 0x7f61bc0cf870 2023-05-22T16:05:04.419 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.412+0000 7f61c20ad640 1 -- 172.21.15.16:0/3964016893 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61bc0cb610 con 0x7f61bc0cf870 2023-05-22T16:05:04.419 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.412+0000 7f61a3fff640 1 -- 172.21.15.16:0/3964016893 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61b000cd70 con 0x7f61bc0cf870 2023-05-22T16:05:04.419 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.412+0000 7f61c20ad640 1 -- 172.21.15.16:0/3964016893 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61bc0cb330 con 0x7f61bc0cf870 2023-05-22T16:05:04.420 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.412+0000 7f61c20ad640 1 -- 172.21.15.16:0/3964016893 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61bc071900 con 0x7f61bc0cf870 2023-05-22T16:05:04.420 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.412+0000 7f61a3fff640 1 -- 172.21.15.16:0/3964016893 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 89) v1 ==== 84446+0+0 (secure 0 0 0) 0x7f61b0013470 con 0x7f61bc0cf870 2023-05-22T16:05:04.420 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.412+0000 7f61a3fff640 1 --2- 172.21.15.16:0/3964016893 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f61a80655f0 0x7f61a8067aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:04.420 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.412+0000 7f61bb7fe640 1 -- 172.21.15.16:0/3964016893 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f61a80655f0 msgr2=0x7f61a8067aa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/65890 2023-05-22T16:05:04.420 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.412+0000 7f61a3fff640 1 -- 172.21.15.16:0/3964016893 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(180..180 src has 1..180) v4 ==== 6782+0+0 (secure 0 0 0) 0x7f61b00837f0 con 0x7f61bc0cf870 2023-05-22T16:05:04.420 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.412+0000 7f61bb7fe640 1 --2- 172.21.15.16:0/3964016893 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f61a80655f0 0x7f61a8067aa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-05-22T16:05:04.422 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.420+0000 7f61a3fff640 1 -- 172.21.15.16:0/3964016893 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f61b0051010 con 0x7f61bc0cf870 2023-05-22T16:05:04.432 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T16:05:04.428+0000 7fdbee8f1180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-05-22T16:05:04.464 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T16:05:04.460+0000 7f1f8c5e7180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-05-22T16:05:04.509 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T16:05:04.504+0000 7fdbee8f1180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-05-22T16:05:04.542 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T16:05:04.540+0000 7f1f8c5e7180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-05-22T16:05:04.559 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T16:05:04.556+0000 7fdbee8f1180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-05-22T16:05:04.577 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.576+0000 7f61c20ad640 1 -- 172.21.15.16:0/3964016893 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f61bc04dbe0 con 0x7f61bc0cf870 2023-05-22T16:05:04.578 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.576+0000 7f61a3fff640 1 -- 172.21.15.16:0/3964016893 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v180) v1 ==== 104+0+11 (secure 0 0 0) 0x7f61b0054630 con 0x7f61bc0cf870 2023-05-22T16:05:04.578 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2023-05-22T16:05:04.580 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.576+0000 7f61c20ad640 1 -- 172.21.15.16:0/3964016893 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f61a80655f0 msgr2=0x7f61a8067aa0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-05-22T16:05:04.581 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.576+0000 7f61c20ad640 1 --2- 172.21.15.16:0/3964016893 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f61a80655f0 0x7f61a8067aa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:04.581 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.576+0000 7f61c20ad640 1 -- 172.21.15.16:0/3964016893 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f61bc0cf870 msgr2=0x7f61bc149500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:04.581 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.576+0000 7f61c20ad640 1 --2- 172.21.15.16:0/3964016893 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f61bc0cf870 0x7f61bc149500 secure :-1 s=READY pgs=2329 cs=0 l=1 rev1=1 crypto rx=0x7f61b000e930 tx=0x7f61b000edf0 comp rx=0 tx=0).stop 2023-05-22T16:05:04.581 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.576+0000 7f61c20ad640 1 -- 172.21.15.16:0/3964016893 shutdown_connections 2023-05-22T16:05:04.581 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.576+0000 7f61c20ad640 1 --2- 172.21.15.16:0/3964016893 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f61bc15cfa0 0x7f61bc151cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:04.581 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.576+0000 7f61c20ad640 1 --2- 172.21.15.16:0/3964016893 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f61a80655f0 0x7f61a8067aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:04.582 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.576+0000 7f61c20ad640 1 --2- 172.21.15.16:0/3964016893 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f61bc0cf870 0x7f61bc149500 unknown :-1 s=CLOSED pgs=2329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:04.582 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.576+0000 7f61c20ad640 1 --2- 172.21.15.16:0/3964016893 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f61bc06e740 0x7f61bc0c7730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:04.582 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.576+0000 7f61c20ad640 1 -- 172.21.15.16:0/3964016893 >> 172.21.15.16:0/3964016893 conn(0x7f61bc0cc870 msgr2=0x7f61bc0b2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:05:04.582 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.576+0000 7f61c20ad640 1 -- 172.21.15.16:0/3964016893 shutdown_connections 2023-05-22T16:05:04.582 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.576+0000 7f61c20ad640 1 -- 172.21.15.16:0/3964016893 wait complete. 2023-05-22T16:05:04.596 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-05-22T16:05:04.597 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T16:05:04.592+0000 7f1f8c5e7180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-05-22T16:05:04.634 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T16:05:04.632+0000 7fdbee8f1180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-05-22T16:05:04.677 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T16:05:04.676+0000 7f1f8c5e7180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-05-22T16:05:04.685 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T16:05:04.680+0000 7fdbee8f1180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-05-22T16:05:04.715 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.712+0000 7fe6723b6640 1 Processor -- start 2023-05-22T16:05:04.716 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.712+0000 7fe6723b6640 1 -- start start 2023-05-22T16:05:04.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.712+0000 7fe6723b6640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fe66c15cfa0 0x7fe66c06db00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:04.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.712+0000 7fe6723b6640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe66c06e740 0x7fe66c06ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:04.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.712+0000 7fe6723b6640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe66c06e040 con 0x7fe66c0cf870 2023-05-22T16:05:04.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.712+0000 7fe6723b6640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe66c0cb9f0 con 0x7fe66c06e740 2023-05-22T16:05:04.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.712+0000 7fe6723b6640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe66c06f100 con 0x7fe66c15cfa0 2023-05-22T16:05:04.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.712+0000 7fe670953640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe66c06e740 0x7fe66c06ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:04.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.712+0000 7fe670953640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe66c06e740 0x7fe66c06ebc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46102/0 (socket says 172.21.15.16:46102) 2023-05-22T16:05:04.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.712+0000 7fe670953640 1 -- 172.21.15.16:0/2500535943 learned_addr learned my addr 172.21.15.16:0/2500535943 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:05:04.719 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.716+0000 7fe66bfff640 1 --2- 172.21.15.16:0/2500535943 >> v2:172.21.15.62:3300/0 conn(0x7fe66c15cfa0 0x7fe66c06db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:04.719 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.716+0000 7fe670953640 1 -- 172.21.15.16:0/2500535943 >> v2:172.21.15.62:3300/0 conn(0x7fe66c15cfa0 msgr2=0x7fe66c06db00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:05:04.719 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.716+0000 7fe670953640 1 --2- 172.21.15.16:0/2500535943 >> v2:172.21.15.62:3300/0 conn(0x7fe66c15cfa0 0x7fe66c06db00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:04.719 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.716+0000 7fe670953640 1 -- 172.21.15.16:0/2500535943 >> v1:172.21.15.62:6789/0 conn(0x7fe66c0cf870 legacy=0x7fe66c0b1fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:04.719 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.716+0000 7fe670953640 1 -- 172.21.15.16:0/2500535943 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe66c06e490 con 0x7fe66c06e740 2023-05-22T16:05:04.720 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.716+0000 7fe670953640 1 --2- 172.21.15.16:0/2500535943 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe66c06e740 0x7fe66c06ebc0 secure :-1 s=READY pgs=2330 cs=0 l=1 rev1=1 crypto rx=0x7fe66c0b1d50 tx=0x7fe66401c9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52414a98b00cf48d server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:04.721 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.716+0000 7fe66affd640 1 -- 172.21.15.16:0/2500535943 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6640a5860 con 0x7fe66c06e740 2023-05-22T16:05:04.721 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.716+0000 7fe66affd640 1 -- 172.21.15.16:0/2500535943 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6640a59c0 con 0x7fe66c06e740 2023-05-22T16:05:04.721 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.716+0000 7fe66affd640 1 -- 172.21.15.16:0/2500535943 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6640a5d50 con 0x7fe66c06e740 2023-05-22T16:05:04.721 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.716+0000 7fe6723b6640 1 -- 172.21.15.16:0/2500535943 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe66c06e740 msgr2=0x7fe66c06ebc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:04.721 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.716+0000 7fe6723b6640 1 --2- 172.21.15.16:0/2500535943 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe66c06e740 0x7fe66c06ebc0 secure :-1 s=READY pgs=2330 cs=0 l=1 rev1=1 crypto rx=0x7fe66c0b1d50 tx=0x7fe66401c9a0 comp rx=0 tx=0).stop 2023-05-22T16:05:04.721 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.716+0000 7fe6723b6640 1 -- 172.21.15.16:0/2500535943 shutdown_connections 2023-05-22T16:05:04.722 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.716+0000 7fe6723b6640 1 --2- 172.21.15.16:0/2500535943 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe66c06e740 0x7fe66c06ebc0 unknown :-1 s=CLOSED pgs=2330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:04.722 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.716+0000 7fe6723b6640 1 --2- 172.21.15.16:0/2500535943 >> v2:172.21.15.62:3300/0 conn(0x7fe66c15cfa0 0x7fe66c06db00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:04.722 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.716+0000 7fe6723b6640 1 -- 172.21.15.16:0/2500535943 >> 172.21.15.16:0/2500535943 conn(0x7fe66c0cc870 msgr2=0x7fe66c0cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:05:04.722 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.716+0000 7fe6723b6640 1 -- 172.21.15.16:0/2500535943 shutdown_connections 2023-05-22T16:05:04.722 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.716+0000 7fe6723b6640 1 -- 172.21.15.16:0/2500535943 wait complete. 2023-05-22T16:05:04.725 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.720+0000 7fe6723b6640 1 Processor -- start 2023-05-22T16:05:04.726 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.720+0000 7fe6723b6640 1 -- start start 2023-05-22T16:05:04.726 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.720+0000 7fe6723b6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe66c06e740 0x7fe66c0c8ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:04.726 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.720+0000 7fe6723b6640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe66c0cf870 0x7fe66c1516f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:04.726 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.720+0000 7fe6723b6640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe66c15cfa0 0x7fe66c15b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:04.726 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.720+0000 7fe6723b6640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe66c151c30 con 0x7fe66c06e740 2023-05-22T16:05:04.726 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.720+0000 7fe6723b6640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe66c153a40 con 0x7fe66c0cf870 2023-05-22T16:05:04.726 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.720+0000 7fe6723b6640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe66c0addb0 con 0x7fe66c15cfa0 2023-05-22T16:05:04.727 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.720+0000 7fe66bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe66c06e740 0x7fe66c0c8ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:04.727 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.720+0000 7fe66bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe66c06e740 0x7fe66c0c8ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46114/0 (socket says 172.21.15.16:46114) 2023-05-22T16:05:04.727 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.720+0000 7fe66bfff640 1 -- 172.21.15.16:0/3348179772 learned_addr learned my addr 172.21.15.16:0/3348179772 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:05:04.727 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.720+0000 7fe66b7fe640 1 --2- 172.21.15.16:0/3348179772 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe66c0cf870 0x7fe66c1516f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:04.727 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.720+0000 7fe670953640 1 --2- 172.21.15.16:0/3348179772 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe66c15cfa0 0x7fe66c15b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:04.727 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.720+0000 7fe66bfff640 1 -- 172.21.15.16:0/3348179772 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe66c15cfa0 msgr2=0x7fe66c15b5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:04.727 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.720+0000 7fe66bfff640 1 --2- 172.21.15.16:0/3348179772 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe66c15cfa0 0x7fe66c15b5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:04.727 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.720+0000 7fe66bfff640 1 -- 172.21.15.16:0/3348179772 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe66c0cf870 msgr2=0x7fe66c1516f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:04.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.720+0000 7fe66bfff640 1 --2- 172.21.15.16:0/3348179772 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe66c0cf870 0x7fe66c1516f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:04.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.720+0000 7fe66bfff640 1 -- 172.21.15.16:0/3348179772 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe664009b30 con 0x7fe66c06e740 2023-05-22T16:05:04.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.720+0000 7fe66b7fe640 1 --2- 172.21.15.16:0/3348179772 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe66c0cf870 0x7fe66c1516f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T16:05:04.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.720+0000 7fe66bfff640 1 --2- 172.21.15.16:0/3348179772 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe66c06e740 0x7fe66c0c8ce0 secure :-1 s=READY pgs=2331 cs=0 l=1 rev1=1 crypto rx=0x7fe65c00d620 tx=0x7fe65c00dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:04.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.720+0000 7fe668ff9640 1 -- 172.21.15.16:0/3348179772 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe65c00b620 con 0x7fe66c06e740 2023-05-22T16:05:04.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.720+0000 7fe668ff9640 1 -- 172.21.15.16:0/3348179772 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe65c00b780 con 0x7fe66c06e740 2023-05-22T16:05:04.729 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.720+0000 7fe6723b6640 1 -- 172.21.15.16:0/3348179772 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe66c0cb610 con 0x7fe66c06e740 2023-05-22T16:05:04.729 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.720+0000 7fe668ff9640 1 -- 172.21.15.16:0/3348179772 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe65c00bb10 con 0x7fe66c06e740 2023-05-22T16:05:04.729 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.720+0000 7fe6723b6640 1 -- 172.21.15.16:0/3348179772 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe66c0cb330 con 0x7fe66c06e740 2023-05-22T16:05:04.729 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.724+0000 7fe6723b6640 1 -- 172.21.15.16:0/3348179772 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe66c15d370 con 0x7fe66c06e740 2023-05-22T16:05:04.729 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.724+0000 7fe668ff9640 1 -- 172.21.15.16:0/3348179772 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 89) v1 ==== 84446+0+0 (secure 0 0 0) 0x7fe65c00bc70 con 0x7fe66c06e740 2023-05-22T16:05:04.729 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.724+0000 7fe668ff9640 1 --2- 172.21.15.16:0/3348179772 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fe64c0655c0 0x7fe64c067a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:04.729 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.724+0000 7fe66b7fe640 1 -- 172.21.15.16:0/3348179772 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fe64c0655c0 msgr2=0x7fe64c067a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/65890 2023-05-22T16:05:04.730 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.724+0000 7fe668ff9640 1 -- 172.21.15.16:0/3348179772 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(180..180 src has 1..180) v4 ==== 6782+0+0 (secure 0 0 0) 0x7fe65c083460 con 0x7fe66c06e740 2023-05-22T16:05:04.730 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.724+0000 7fe66b7fe640 1 --2- 172.21.15.16:0/3348179772 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fe64c0655c0 0x7fe64c067a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-05-22T16:05:04.733 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.732+0000 7fe668ff9640 1 -- 172.21.15.16:0/3348179772 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fe65c050ab0 con 0x7fe66c06e740 2023-05-22T16:05:04.734 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T16:05:04.732+0000 7f1f8c5e7180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-05-22T16:05:04.818 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T16:05:04.812+0000 7fdbee8f1180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-05-22T16:05:04.869 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T16:05:04.868+0000 7f1f8c5e7180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-05-22T16:05:04.870 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T16:05:04.868+0000 7fdbee8f1180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-05-22T16:05:04.882 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.880+0000 7fe6723b6640 1 -- 172.21.15.16:0/3348179772 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fe66c04dbe0 con 0x7fe66c06e740 2023-05-22T16:05:04.882 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.880+0000 7fe668ff9640 1 -- 172.21.15.16:0/3348179772 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v180) v1 ==== 100+0+11 (secure 0 0 0) 0x7fe65c007380 con 0x7fe66c06e740 2023-05-22T16:05:04.883 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 64 2023-05-22T16:05:04.885 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.884+0000 7fe6723b6640 1 -- 172.21.15.16:0/3348179772 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fe64c0655c0 msgr2=0x7fe64c067a70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-05-22T16:05:04.885 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.884+0000 7fe6723b6640 1 --2- 172.21.15.16:0/3348179772 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fe64c0655c0 0x7fe64c067a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:04.885 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.884+0000 7fe6723b6640 1 -- 172.21.15.16:0/3348179772 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe66c06e740 msgr2=0x7fe66c0c8ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:04.886 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.884+0000 7fe6723b6640 1 --2- 172.21.15.16:0/3348179772 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe66c06e740 0x7fe66c0c8ce0 secure :-1 s=READY pgs=2331 cs=0 l=1 rev1=1 crypto rx=0x7fe65c00d620 tx=0x7fe65c00dae0 comp rx=0 tx=0).stop 2023-05-22T16:05:04.886 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.884+0000 7fe6723b6640 1 -- 172.21.15.16:0/3348179772 shutdown_connections 2023-05-22T16:05:04.886 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.884+0000 7fe6723b6640 1 --2- 172.21.15.16:0/3348179772 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe66c15cfa0 0x7fe66c15b5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:04.886 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.884+0000 7fe6723b6640 1 --2- 172.21.15.16:0/3348179772 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe66c0cf870 0x7fe66c1516f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:04.886 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.884+0000 7fe6723b6640 1 --2- 172.21.15.16:0/3348179772 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fe64c0655c0 0x7fe64c067a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:04.887 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.884+0000 7fe6723b6640 1 --2- 172.21.15.16:0/3348179772 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe66c06e740 0x7fe66c0c8ce0 unknown :-1 s=CLOSED pgs=2331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:04.887 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.884+0000 7fe6723b6640 1 -- 172.21.15.16:0/3348179772 >> 172.21.15.16:0/3348179772 conn(0x7fe66c0cc870 msgr2=0x7fe66c157880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:05:04.887 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.884+0000 7fe6723b6640 1 -- 172.21.15.16:0/3348179772 shutdown_connections 2023-05-22T16:05:04.887 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:04.884+0000 7fe6723b6640 1 -- 172.21.15.16:0/3348179772 wait complete. 2023-05-22T16:05:04.902 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-05-22T16:05:04.923 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T16:05:04.920+0000 7f1f8c5e7180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-05-22T16:05:05.017 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:05.016+0000 7f9f14932640 1 Processor -- start 2023-05-22T16:05:05.018 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:05.016+0000 7f9f14932640 1 -- start start 2023-05-22T16:05:05.023 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:05.016+0000 7f9f14932640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f9f10159cd0 0x7f9f1015a0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:05.024 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:05.016+0000 7f9f14932640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9f100cc6f0 0x7f9f100cfe30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:05.024 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:05.016+0000 7f9f14932640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f100cf9a0 con 0x7f9f100b2880 2023-05-22T16:05:05.024 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:05.016+0000 7f9f14932640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f1015aba0 con 0x7f9f10159cd0 2023-05-22T16:05:05.024 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:05.016+0000 7f9f14932640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f1015ae80 con 0x7f9f100cc6f0 2023-05-22T16:05:05.024 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:05.016+0000 7f9f0dd9b640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f9f100b2880 0x7f9f100cbfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:45436/0 (socket says 172.21.15.16:45436) 2023-05-22T16:05:05.024 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:05.016+0000 7f9f0dd9b640 1 -- 172.21.15.16:0/318918459 learned_addr learned my addr 172.21.15.16:0/318918459 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T16:05:05.025 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:05.016+0000 7f9f0ed9d640 1 --2- 172.21.15.16:0/318918459 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9f100cc6f0 0x7f9f100cfe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:05.025 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:05.016+0000 7f9f0e59c640 1 --2- 172.21.15.16:0/318918459 >> v2:172.21.15.62:3300/0 conn(0x7f9f10159cd0 0x7f9f1015a0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:05.025 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:05.016+0000 7f9f0d59a640 1 -- 172.21.15.16:0/318918459 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4168831743 0 0) 0x7f9f100cf9a0 con 0x7f9f100b2880 2023-05-22T16:05:05.025 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:05.016+0000 7f9f0d59a640 1 -- 172.21.15.16:0/318918459 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ef8003530 con 0x7f9f100b2880 2023-05-22T16:05:05.026 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:05.016+0000 7f9f0d59a640 1 -- 172.21.15.16:0/318918459 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 683949866 0 0) 0x7f9ef8003530 con 0x7f9f100b2880 2023-05-22T16:05:05.026 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:05.016+0000 7f9f0d59a640 1 -- 172.21.15.16:0/318918459 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9f100cc6f0 msgr2=0x7f9f100cfe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:05:05.026 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:05.016+0000 7f9f0d59a640 1 --2- 172.21.15.16:0/318918459 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9f100cc6f0 0x7f9f100cfe30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:05.026 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:05.016+0000 7f9f0d59a640 1 -- 172.21.15.16:0/318918459 >> v2:172.21.15.62:3300/0 conn(0x7f9f10159cd0 msgr2=0x7f9f1015a0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:05:05.026 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:05.016+0000 7f9f0d59a640 1 --2- 172.21.15.16:0/318918459 >> v2:172.21.15.62:3300/0 conn(0x7f9f10159cd0 0x7f9f1015a0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:05.026 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:05.016+0000 7f9f0d59a640 1 -- 172.21.15.16:0/318918459 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f1015b160 con 0x7f9f100b2880 2023-05-22T16:05:05.027 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:05.016+0000 7f9f0d59a640 1 -- 172.21.15.16:0/318918459 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7f9f04001680 con 0x7f9f100b2880 2023-05-22T16:05:05.027 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:05.016+0000 7f9f0d59a640 1 -- 172.21.15.16:0/318918459 >> v1:172.21.15.16:6789/0 conn(0x7f9f100b2880 legacy=0x7f9f100cbfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:05.027 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:05.016+0000 7f9f0d59a640 1 --2- 172.21.15.16:0/318918459 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9f100cc6f0 0x7f9ef8003980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:05.027 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:05.016+0000 7f9f0d59a640 1 --2- 172.21.15.16:0/318918459 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9ef8003f00 0x7f9ef80047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:05.027 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:05.016+0000 7f9f0d59a640 1 --2- 172.21.15.16:0/318918459 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9ef8004d10 0x7f9ef8005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:05.027 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:05.016+0000 7f9f0d59a640 1 -- 172.21.15.16:0/318918459 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f1015aba0 con 0x7f9ef8003f00 2023-05-22T16:05:05.028 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:05.016+0000 7f9f0d59a640 1 -- 172.21.15.16:0/318918459 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f1015ae80 con 0x7f9ef8004d10 2023-05-22T16:05:05.028 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:05.016+0000 7f9f0d59a640 1 -- 172.21.15.16:0/318918459 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ef800b990 con 0x7f9f100cc6f0 2023-05-22T16:05:05.028 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:05.016+0000 7f9f0dd9b640 1 --2- 172.21.15.16:0/318918459 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9ef8003f00 0x7f9ef80047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:05.028 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:05.016+0000 7f9f0e59c640 1 --2- 172.21.15.16:0/318918459 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9f100cc6f0 0x7f9ef8003980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:05.028 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:05.016+0000 7f9f0ed9d640 1 --2- 172.21.15.16:0/318918459 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9ef8004d10 0x7f9ef8005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:05.029 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:05.016+0000 7f9f0dd9b640 1 -- 172.21.15.16:0/318918459 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9f100cc6f0 msgr2=0x7f9ef8003980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:05:05.029 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:05.016+0000 7f9f0dd9b640 1 --2- 172.21.15.16:0/318918459 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9f100cc6f0 0x7f9ef8003980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:05.029 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:05.016+0000 7f9f0dd9b640 1 -- 172.21.15.16:0/318918459 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9ef8004d10 msgr2=0x7f9ef8005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:05:05.029 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:05.016+0000 7f9f0dd9b640 1 --2- 172.21.15.16:0/318918459 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9ef8004d10 0x7f9ef8005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:05.029 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:05.016+0000 7f9f0dd9b640 1 -- 172.21.15.16:0/318918459 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ef8003530 con 0x7f9ef8003f00 2023-05-22T16:05:05.029 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:05.016+0000 7f9f0e59c640 1 --2- 172.21.15.16:0/318918459 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9f100cc6f0 0x7f9ef8003980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T16:05:05.029 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:05.020+0000 7f9f0dd9b640 1 --2- 172.21.15.16:0/318918459 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9ef8003f00 0x7f9ef80047d0 secure :-1 s=READY pgs=2333 cs=0 l=1 rev1=1 crypto rx=0x7f9f0400c210 tx=0x7f9f0401f3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=614cdedaf643e6b5 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:05.030 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:05.020+0000 7f9f0d59a640 1 -- 172.21.15.16:0/318918459 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f04005ae0 con 0x7f9ef8003f00 2023-05-22T16:05:05.030 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:05.020+0000 7f9f0d59a640 1 -- 172.21.15.16:0/318918459 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f040ab070 con 0x7f9ef8003f00 2023-05-22T16:05:05.030 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:05.020+0000 7f9f0d59a640 1 -- 172.21.15.16:0/318918459 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f04005ea0 con 0x7f9ef8003f00 2023-05-22T16:05:05.030 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:05.020+0000 7f9f14932640 1 -- 172.21.15.16:0/318918459 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9ef8003f00 msgr2=0x7f9ef80047d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:05.030 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:05.020+0000 7f9f14932640 1 --2- 172.21.15.16:0/318918459 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9ef8003f00 0x7f9ef80047d0 secure :-1 s=READY pgs=2333 cs=0 l=1 rev1=1 crypto rx=0x7f9f0400c210 tx=0x7f9f0401f3f0 comp rx=0 tx=0).stop 2023-05-22T16:05:05.031 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:05.020+0000 7f9f0ed9d640 1 -- 172.21.15.16:0/318918459 reap_dead start 2023-05-22T16:05:05.031 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:05.020+0000 7f9f14932640 1 -- 172.21.15.16:0/318918459 shutdown_connections 2023-05-22T16:05:05.031 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:05.020+0000 7f9f14932640 1 -- 172.21.15.16:0/318918459 >> 172.21.15.16:0/318918459 conn(0x7f9f1006f5c0 msgr2=0x7f9f1006f9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:05:05.031 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:05.020+0000 7f9f14932640 1 -- 172.21.15.16:0/318918459 shutdown_connections 2023-05-22T16:05:05.031 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:05.020+0000 7f9f14932640 1 -- 172.21.15.16:0/318918459 wait complete. 2023-05-22T16:05:05.033 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:05.020+0000 7f9f14932640 1 Processor -- start 2023-05-22T16:05:05.034 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:05.020+0000 7f9f14932640 1 -- start start 2023-05-22T16:05:05.034 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:05.024+0000 7f9f14932640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9f100af0e0 0x7f9f100b4e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:05.034 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:05.024+0000 7f9f14932640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9f101f4940 0x7f9f101f4db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:05.034 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:05.024+0000 7f9f14932640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f100b0660 0x7f9f100b0b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:05.034 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:05.024+0000 7f9f14932640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f100b1040 con 0x7f9f100b0660 2023-05-22T16:05:05.034 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:05.024+0000 7f9f14932640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f100b49b0 con 0x7f9f101f4940 2023-05-22T16:05:05.035 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:05.024+0000 7f9f14932640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f101f53c0 con 0x7f9f100af0e0 2023-05-22T16:05:05.035 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:05.024+0000 7f9f0ed9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f100b0660 0x7f9f100b0b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:05.035 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:05.024+0000 7f9f0ed9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f100b0660 0x7f9f100b0b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46134/0 (socket says 172.21.15.16:46134) 2023-05-22T16:05:05.035 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:05.024+0000 7f9f0ed9d640 1 -- 172.21.15.16:0/405040941 learned_addr learned my addr 172.21.15.16:0/405040941 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:05:05.035 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:05.024+0000 7f9f0dd9b640 1 --2- 172.21.15.16:0/405040941 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9f101f4940 0x7f9f101f4db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:05.035 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:05.024+0000 7f9f0e59c640 1 --2- 172.21.15.16:0/405040941 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9f100af0e0 0x7f9f100b4e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:05.036 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:05.024+0000 7f9f0ed9d640 1 -- 172.21.15.16:0/405040941 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9f100af0e0 msgr2=0x7f9f100b4e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:05.036 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:05.024+0000 7f9f0ed9d640 1 --2- 172.21.15.16:0/405040941 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9f100af0e0 0x7f9f100b4e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:05.036 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:05.024+0000 7f9f0ed9d640 1 -- 172.21.15.16:0/405040941 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9f101f4940 msgr2=0x7f9f101f4db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:05.036 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:05.024+0000 7f9f0ed9d640 1 --2- 172.21.15.16:0/405040941 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9f101f4940 0x7f9f101f4db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:05.036 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:05.024+0000 7f9f0ed9d640 1 -- 172.21.15.16:0/405040941 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f1015a5e0 con 0x7f9f100b0660 2023-05-22T16:05:05.036 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:05.024+0000 7f9f0dd9b640 1 --2- 172.21.15.16:0/405040941 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9f101f4940 0x7f9f101f4db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T16:05:05.037 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:05.024+0000 7f9f0ed9d640 1 --2- 172.21.15.16:0/405040941 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f100b0660 0x7f9f100b0b00 secure :-1 s=READY pgs=2334 cs=0 l=1 rev1=1 crypto rx=0x7f9f1006ddd0 tx=0x7f9f08007dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:05.037 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:05.024+0000 7f9efeffd640 1 -- 172.21.15.16:0/405040941 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f0800e070 con 0x7f9f100b0660 2023-05-22T16:05:05.037 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:05.024+0000 7f9efeffd640 1 -- 172.21.15.16:0/405040941 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f08003c70 con 0x7f9f100b0660 2023-05-22T16:05:05.037 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:05.024+0000 7f9f14932640 1 -- 172.21.15.16:0/405040941 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f100cf630 con 0x7f9f100b0660 2023-05-22T16:05:05.037 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:05.024+0000 7f9efeffd640 1 -- 172.21.15.16:0/405040941 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f08003e60 con 0x7f9f100b0660 2023-05-22T16:05:05.037 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:05.024+0000 7f9f14932640 1 -- 172.21.15.16:0/405040941 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f1015a770 con 0x7f9f100b0660 2023-05-22T16:05:05.037 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:05.024+0000 7f9efeffd640 1 -- 172.21.15.16:0/405040941 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 89) v1 ==== 84446+0+0 (secure 0 0 0) 0x7f9f08004030 con 0x7f9f100b0660 2023-05-22T16:05:05.037 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:05.024+0000 7f9f14932640 1 -- 172.21.15.16:0/405040941 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f1004dbe0 con 0x7f9f100b0660 2023-05-22T16:05:05.038 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:05.024+0000 7f9efeffd640 1 --2- 172.21.15.16:0/405040941 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f9ef00655f0 0x7f9ef0067aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:05.038 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:05.024+0000 7f9f0e59c640 1 -- 172.21.15.16:0/405040941 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f9ef00655f0 msgr2=0x7f9ef0067aa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/65890 2023-05-22T16:05:05.038 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:05.024+0000 7f9efeffd640 1 -- 172.21.15.16:0/405040941 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(180..180 src has 1..180) v4 ==== 6782+0+0 (secure 0 0 0) 0x7f9f08082e80 con 0x7f9f100b0660 2023-05-22T16:05:05.038 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:05.024+0000 7f9f0e59c640 1 --2- 172.21.15.16:0/405040941 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f9ef00655f0 0x7f9ef0067aa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-05-22T16:05:05.038 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:05.028+0000 7f9efeffd640 1 -- 172.21.15.16:0/405040941 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f9f080502c0 con 0x7f9f100b0660 2023-05-22T16:05:05.093 INFO:tasks.ceph.mgr.x.smithi062.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2023-05-22T16:05:05.094 INFO:tasks.ceph.mgr.x.smithi062.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2023-05-22T16:05:05.094 INFO:tasks.ceph.mgr.x.smithi062.stderr: from numpy import show_config as show_numpy_config 2023-05-22T16:05:05.101 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T16:05:05.096+0000 7fdbee8f1180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-05-22T16:05:05.162 INFO:tasks.ceph.mgr.y.smithi016.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2023-05-22T16:05:05.162 INFO:tasks.ceph.mgr.y.smithi016.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2023-05-22T16:05:05.162 INFO:tasks.ceph.mgr.y.smithi016.stderr: from numpy import show_config as show_numpy_config 2023-05-22T16:05:05.170 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T16:05:05.168+0000 7f1f8c5e7180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-05-22T16:05:05.226 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T16:05:05.220+0000 7fdbee8f1180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-05-22T16:05:05.228 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:05.228+0000 7f9f0e59c640 1 -- 172.21.15.16:0/405040941 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f9ef00655f0 msgr2=0x7f9ef0067aa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/65890 2023-05-22T16:05:05.229 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:05.228+0000 7f9f0e59c640 1 --2- 172.21.15.16:0/405040941 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f9ef00655f0 0x7f9ef0067aa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-05-22T16:05:05.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:05.252+0000 7f9f14932640 1 -- 172.21.15.16:0/405040941 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f9f100b4af0 con 0x7f9f100b0660 2023-05-22T16:05:05.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:05.252+0000 7f9efeffd640 1 -- 172.21.15.16:0/405040941 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 8 entries v181) v1 ==== 80+0+460 (secure 0 0 0) 0x7f9f080538e0 con 0x7f9f100b0660 2023-05-22T16:05:05.256 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/2495910544 2023-05-23T16:03:52.657800+0000 2023-05-22T16:05:05.256 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/1761665129 2023-05-23T16:03:52.657800+0000 2023-05-22T16:05:05.256 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:6833/65890 2023-05-23T16:03:52.657800+0000 2023-05-22T16:05:05.256 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:6832/65890 2023-05-23T16:03:52.657800+0000 2023-05-22T16:05:05.256 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/4138270792 2023-05-23T16:03:52.657800+0000 2023-05-22T16:05:05.256 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/2442191907 2023-05-23T16:03:52.657800+0000 2023-05-22T16:05:05.257 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/3487890192 2023-05-23T16:03:52.657800+0000 2023-05-22T16:05:05.257 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/3247164174 2023-05-23T16:03:52.657800+0000 2023-05-22T16:05:05.257 INFO:teuthology.orchestra.run.smithi016.stderr:listed 8 entries 2023-05-22T16:05:05.258 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:05.256+0000 7f9f14932640 1 -- 172.21.15.16:0/405040941 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f9ef00655f0 msgr2=0x7f9ef0067aa0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-05-22T16:05:05.258 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:05.256+0000 7f9f14932640 1 --2- 172.21.15.16:0/405040941 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f9ef00655f0 0x7f9ef0067aa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:05.258 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:05.256+0000 7f9f14932640 1 -- 172.21.15.16:0/405040941 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f100b0660 msgr2=0x7f9f100b0b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:05.259 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:05.256+0000 7f9f14932640 1 --2- 172.21.15.16:0/405040941 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f100b0660 0x7f9f100b0b00 secure :-1 s=READY pgs=2334 cs=0 l=1 rev1=1 crypto rx=0x7f9f1006ddd0 tx=0x7f9f08007dd0 comp rx=0 tx=0).stop 2023-05-22T16:05:05.259 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:05.256+0000 7f9f14932640 1 -- 172.21.15.16:0/405040941 shutdown_connections 2023-05-22T16:05:05.260 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:05.256+0000 7f9f14932640 1 --2- 172.21.15.16:0/405040941 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f9ef00655f0 0x7f9ef0067aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:05.260 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:05.256+0000 7f9f14932640 1 --2- 172.21.15.16:0/405040941 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f100b0660 0x7f9f100b0b00 unknown :-1 s=CLOSED pgs=2334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:05.260 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:05.256+0000 7f9f14932640 1 --2- 172.21.15.16:0/405040941 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9f101f4940 0x7f9f101f4db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:05.261 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:05.256+0000 7f9f14932640 1 --2- 172.21.15.16:0/405040941 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9f100af0e0 0x7f9f100b4e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:05.261 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:05.256+0000 7f9f14932640 1 -- 172.21.15.16:0/405040941 >> 172.21.15.16:0/405040941 conn(0x7f9f1006f5c0 msgr2=0x7f9f100cfd30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:05:05.261 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:05.256+0000 7f9f14932640 1 -- 172.21.15.16:0/405040941 shutdown_connections 2023-05-22T16:05:05.261 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:05.256+0000 7f9f14932640 1 -- 172.21.15.16:0/405040941 wait complete. 2023-05-22T16:05:05.274 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi016.front.sepia.ceph.com... 2023-05-22T16:05:05.275 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T16:05:05.276 DEBUG:teuthology.orchestra.run.smithi016:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2023-05-22T16:05:05.303 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T16:05:05.300+0000 7f1f8c5e7180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-05-22T16:05:05.311 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi016.stderr:ceph-fuse[67810]: fuse finished with error 0 and tester_r 0 2023-05-22T16:05:05.313 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi016.front.sepia.ceph.com 2023-05-22T16:05:05.314 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T16:05:05.314 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-05-22T16:05:05.353 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T16:05:05.348+0000 7fdbee8f1180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-05-22T16:05:05.418 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T16:05:05.412+0000 7fdbee8f1180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-05-22T16:05:05.424 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T16:05:05.420+0000 7f1f8c5e7180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-05-22T16:05:05.458 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2023-05-22T16:05:05.459 DEBUG:teuthology.orchestra.run.smithi062:> dd if=/proc/self/mounts of=/dev/stdout 2023-05-22T16:05:05.466 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16342116k,nr_inodes=4085529,mode=755,inode64 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275932k,mode=755,inode64 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev,inode64 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=13250 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275928k,nr_inodes=818982,mode=700,uid=1000,gid=1271,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275932k,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-05-22T16:05:05.467 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi062.front.sepia.ceph.com 2023-05-22T16:05:05.467 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T16:05:05.468 DEBUG:teuthology.orchestra.run.smithi062:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2023-05-22T16:05:05.488 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T16:05:05.484+0000 7f1f8c5e7180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-05-22T16:05:05.515 INFO:teuthology.orchestra.run.smithi062.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2023-05-22T16:05:05.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-05-22T16:05:05.516 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-05-22T16:05:05.517 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2023-05-22T16:05:05.523 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16345192k,nr_inodes=4086298,mode=755,inode64 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3276544k,mode=755,inode64 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev,inode64 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=15732 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3276540k,nr_inodes=819135,mode=700,uid=1000,gid=1271,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3276544k,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-05-22T16:05:05.523 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi016.front.sepia.ceph.com 2023-05-22T16:05:05.523 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T16:05:05.524 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2023-05-22T16:05:05.542 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T16:05:05.536+0000 7fdbee8f1180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-05-22T16:05:05.569 INFO:teuthology.orchestra.run.smithi016.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2023-05-22T16:05:05.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-05-22T16:05:05.570 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2023-05-22T16:05:05.571 DEBUG:teuthology.orchestra.run.smithi062:> dd if=/proc/self/mounts of=/dev/stdout 2023-05-22T16:05:05.576 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16342116k,nr_inodes=4085529,mode=755,inode64 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275932k,mode=755,inode64 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev,inode64 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=13250 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275928k,nr_inodes=818982,mode=700,uid=1000,gid=1271,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275932k,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-05-22T16:05:05.577 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi062.front.sepia.ceph.com 2023-05-22T16:05:05.577 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T16:05:05.577 DEBUG:teuthology.orchestra.run.smithi062:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2023-05-22T16:05:05.617 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T16:05:05.616+0000 7f1f8c5e7180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-05-22T16:05:05.622 INFO:teuthology.orchestra.run.smithi062.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2023-05-22T16:05:05.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-05-22T16:05:05.624 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-05-22T16:05:05.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:05.744+0000 7fc351133640 1 Processor -- start 2023-05-22T16:05:05.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:05.744+0000 7fc351133640 1 -- start start 2023-05-22T16:05:05.746 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:05.744+0000 7fc351133640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc34c15cfa0 0x7fc34c06db00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:05.746 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:05.744+0000 7fc351133640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fc34c0cf870 0x7fc34c0b1fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:05.747 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:05.744+0000 7fc351133640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc34c06e040 con 0x7fc34c06e740 2023-05-22T16:05:05.747 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:05.744+0000 7fc351133640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc34c0cb9f0 con 0x7fc34c0cf870 2023-05-22T16:05:05.747 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:05.744+0000 7fc351133640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc34c0b24f0 con 0x7fc34c15cfa0 2023-05-22T16:05:05.747 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:05.744+0000 7fc34ad9d640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc34c15cfa0 0x7fc34c06db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:05.747 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:05.744+0000 7fc34ad9d640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc34c15cfa0 0x7fc34c06db00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3301/0 says I am v2:172.21.15.16:50580/0 (socket says 172.21.15.16:50580) 2023-05-22T16:05:05.747 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:05.744+0000 7fc34ad9d640 1 -- 172.21.15.16:0/3631531341 learned_addr learned my addr 172.21.15.16:0/3631531341 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:05:05.747 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:05.744+0000 7fc34a59c640 1 --2- 172.21.15.16:0/3631531341 >> v2:172.21.15.62:3300/0 conn(0x7fc34c0cf870 0x7fc34c0b1fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:05.748 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:05.744+0000 7fc34ad9d640 1 -- 172.21.15.16:0/3631531341 >> v2:172.21.15.62:3300/0 conn(0x7fc34c0cf870 msgr2=0x7fc34c0b1fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:05:05.748 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:05.744+0000 7fc34ad9d640 1 --2- 172.21.15.16:0/3631531341 >> v2:172.21.15.62:3300/0 conn(0x7fc34c0cf870 0x7fc34c0b1fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:05.748 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:05.744+0000 7fc34ad9d640 1 -- 172.21.15.16:0/3631531341 >> v1:172.21.15.62:6789/0 conn(0x7fc34c06e740 legacy=0x7fc34c070de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:05.748 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:05.744+0000 7fc34ad9d640 1 -- 172.21.15.16:0/3631531341 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc34c06e490 con 0x7fc34c15cfa0 2023-05-22T16:05:05.749 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:05.748+0000 7fc34ad9d640 1 --2- 172.21.15.16:0/3631531341 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc34c15cfa0 0x7fc34c06db00 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fc33c009870 tx=0x7fc33c01c480 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3f38c505d33bc9e2 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:05.749 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:05.748+0000 7fc349d9b640 1 -- 172.21.15.16:0/3631531341 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc33c0ab680 con 0x7fc34c15cfa0 2023-05-22T16:05:05.749 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:05.748+0000 7fc349d9b640 1 -- 172.21.15.16:0/3631531341 <== mon.2 v2:172.21.15.62:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc33c0ab7e0 con 0x7fc34c15cfa0 2023-05-22T16:05:05.750 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:05.748+0000 7fc349d9b640 1 -- 172.21.15.16:0/3631531341 <== mon.2 v2:172.21.15.62:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc33c0abb70 con 0x7fc34c15cfa0 2023-05-22T16:05:05.750 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:05.748+0000 7fc351133640 1 -- 172.21.15.16:0/3631531341 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc34c15cfa0 msgr2=0x7fc34c06db00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:05.751 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:05.748+0000 7fc351133640 1 --2- 172.21.15.16:0/3631531341 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc34c15cfa0 0x7fc34c06db00 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fc33c009870 tx=0x7fc33c01c480 comp rx=0 tx=0).stop 2023-05-22T16:05:05.751 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:05.748+0000 7fc351133640 1 -- 172.21.15.16:0/3631531341 shutdown_connections 2023-05-22T16:05:05.751 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:05.748+0000 7fc351133640 1 --2- 172.21.15.16:0/3631531341 >> v2:172.21.15.62:3300/0 conn(0x7fc34c0cf870 0x7fc34c0b1fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:05.751 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:05.748+0000 7fc351133640 1 --2- 172.21.15.16:0/3631531341 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc34c15cfa0 0x7fc34c06db00 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:05.751 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:05.748+0000 7fc351133640 1 -- 172.21.15.16:0/3631531341 >> 172.21.15.16:0/3631531341 conn(0x7fc34c0cc870 msgr2=0x7fc34c0cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:05:05.751 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:05.748+0000 7fc351133640 1 -- 172.21.15.16:0/3631531341 shutdown_connections 2023-05-22T16:05:05.751 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:05.748+0000 7fc351133640 1 -- 172.21.15.16:0/3631531341 wait complete. 2023-05-22T16:05:05.754 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:05.748+0000 7fc351133640 1 Processor -- start 2023-05-22T16:05:05.754 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:05.748+0000 7fc351133640 1 -- start start 2023-05-22T16:05:05.755 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:05.748+0000 7fc351133640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc34c06e740 0x7fc34c163960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:05.755 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:05.748+0000 7fc351133640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc34c0cf870 0x7fc34c159290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:05.755 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:05.748+0000 7fc351133640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc34c15cfa0 0x7fc34c0c85f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:05.755 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:05.748+0000 7fc351133640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc34c163ea0 con 0x7fc34c0cf870 2023-05-22T16:05:05.755 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:05.748+0000 7fc351133640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc34c1597d0 con 0x7fc34c06e740 2023-05-22T16:05:05.755 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:05.748+0000 7fc351133640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc34c0c7550 con 0x7fc34c15cfa0 2023-05-22T16:05:05.756 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:05.748+0000 7fc34a59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc34c0cf870 0x7fc34c159290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:05.756 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:05.748+0000 7fc34a59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc34c0cf870 0x7fc34c159290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46136/0 (socket says 172.21.15.16:46136) 2023-05-22T16:05:05.756 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:05.748+0000 7fc34a59c640 1 -- 172.21.15.16:0/3674022654 learned_addr learned my addr 172.21.15.16:0/3674022654 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:05:05.756 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:05.748+0000 7fc34ad9d640 1 --2- 172.21.15.16:0/3674022654 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc34c06e740 0x7fc34c163960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:05.756 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:05.748+0000 7fc34b59e640 1 --2- 172.21.15.16:0/3674022654 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc34c15cfa0 0x7fc34c0c85f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:05.756 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:05.748+0000 7fc34a59c640 1 -- 172.21.15.16:0/3674022654 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc34c15cfa0 msgr2=0x7fc34c0c85f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:05.756 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:05.748+0000 7fc34a59c640 1 --2- 172.21.15.16:0/3674022654 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc34c15cfa0 0x7fc34c0c85f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:05.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:05.748+0000 7fc34a59c640 1 -- 172.21.15.16:0/3674022654 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc34c06e740 msgr2=0x7fc34c163960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:05.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:05.748+0000 7fc34a59c640 1 --2- 172.21.15.16:0/3674022654 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc34c06e740 0x7fc34c163960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:05.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:05.748+0000 7fc34a59c640 1 -- 172.21.15.16:0/3674022654 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc33c0094d0 con 0x7fc34c0cf870 2023-05-22T16:05:05.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:05.748+0000 7fc34b59e640 1 --2- 172.21.15.16:0/3674022654 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc34c15cfa0 0x7fc34c0c85f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T16:05:05.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:05.752+0000 7fc34a59c640 1 --2- 172.21.15.16:0/3674022654 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc34c0cf870 0x7fc34c159290 secure :-1 s=READY pgs=2335 cs=0 l=1 rev1=1 crypto rx=0x7fc34000e930 tx=0x7fc34000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:05.758 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:05.752+0000 7fc3337fe640 1 -- 172.21.15.16:0/3674022654 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc340009690 con 0x7fc34c0cf870 2023-05-22T16:05:05.758 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:05.752+0000 7fc3337fe640 1 -- 172.21.15.16:0/3674022654 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc34000ca00 con 0x7fc34c0cf870 2023-05-22T16:05:05.758 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:05.752+0000 7fc351133640 1 -- 172.21.15.16:0/3674022654 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc34c0cb610 con 0x7fc34c0cf870 2023-05-22T16:05:05.758 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:05.752+0000 7fc3337fe640 1 -- 172.21.15.16:0/3674022654 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc34000cd70 con 0x7fc34c0cf870 2023-05-22T16:05:05.758 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:05.752+0000 7fc351133640 1 -- 172.21.15.16:0/3674022654 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc34c0cb330 con 0x7fc34c0cf870 2023-05-22T16:05:05.758 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:05.752+0000 7fc351133640 1 -- 172.21.15.16:0/3674022654 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc34c071900 con 0x7fc34c0cf870 2023-05-22T16:05:05.759 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:05.752+0000 7fc3337fe640 1 -- 172.21.15.16:0/3674022654 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 89) v1 ==== 84446+0+0 (secure 0 0 0) 0x7fc340013470 con 0x7fc34c0cf870 2023-05-22T16:05:05.759 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:05.752+0000 7fc3337fe640 1 --2- 172.21.15.16:0/3674022654 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fc3380655c0 0x7fc338067a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:05.759 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:05.752+0000 7fc34ad9d640 1 -- 172.21.15.16:0/3674022654 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fc3380655c0 msgr2=0x7fc338067a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/65890 2023-05-22T16:05:05.760 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:05.752+0000 7fc3337fe640 1 -- 172.21.15.16:0/3674022654 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(181..181 src has 1..181) v4 ==== 6870+0+0 (secure 0 0 0) 0x7fc3400831f0 con 0x7fc34c0cf870 2023-05-22T16:05:05.760 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:05.752+0000 7fc34ad9d640 1 --2- 172.21.15.16:0/3674022654 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fc3380655c0 0x7fc338067a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-05-22T16:05:05.760 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:05.756+0000 7fc3337fe640 1 -- 172.21.15.16:0/3674022654 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fc340050840 con 0x7fc34c0cf870 2023-05-22T16:05:05.941 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:05.936+0000 7fc351133640 1 -- 172.21.15.16:0/3674022654 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc34c04dbe0 con 0x7fc34c0cf870 2023-05-22T16:05:05.942 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:05.940+0000 7fc3337fe640 1 -- 172.21.15.16:0/3674022654 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 73 v73) v1 ==== 94+0+4657 (secure 0 0 0) 0x7fc340053e60 con 0x7fc34c0cf870 2023-05-22T16:05:05.943 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T16:05:05.943 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":73,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7682,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/2815032143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":2815032143},{"type":"v1","addr":"172.21.15.62:6835","nonce":2815032143}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":69},{"gid":10239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6836/1961058261","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1961058261},{"type":"v1","addr":"172.21.15.16:6836","nonce":1961058261}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":69},{"gid":10242,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/1453687184","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6835","nonce":1453687184},{"type":"v1","addr":"172.21.15.16:6837","nonce":1453687184}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":73,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-05-22T16:03:38.779645+0000","modified":"2023-05-22T16:03:42.129478+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10251":{"gid":10251,"name":"d","rank":0,"incarnation":70,"state":"up:active","state_seq":3,"addr":"172.21.15.62:6833/2486335217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":2486335217},{"type":"v1","addr":"172.21.15.62:6833","nonce":2486335217}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[23],"metadata_pool":22,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2023-05-22T16:05:05.944 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 73 2023-05-22T16:05:05.945 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:05.940+0000 7fc351133640 1 -- 172.21.15.16:0/3674022654 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fc3380655c0 msgr2=0x7fc338067a70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-05-22T16:05:05.945 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:05.940+0000 7fc351133640 1 --2- 172.21.15.16:0/3674022654 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fc3380655c0 0x7fc338067a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:05.945 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:05.940+0000 7fc351133640 1 -- 172.21.15.16:0/3674022654 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc34c0cf870 msgr2=0x7fc34c159290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:05.945 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:05.940+0000 7fc351133640 1 --2- 172.21.15.16:0/3674022654 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc34c0cf870 0x7fc34c159290 secure :-1 s=READY pgs=2335 cs=0 l=1 rev1=1 crypto rx=0x7fc34000e930 tx=0x7fc34000edf0 comp rx=0 tx=0).stop 2023-05-22T16:05:05.946 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:05.940+0000 7fc351133640 1 -- 172.21.15.16:0/3674022654 shutdown_connections 2023-05-22T16:05:05.946 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:05.940+0000 7fc351133640 1 --2- 172.21.15.16:0/3674022654 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc34c15cfa0 0x7fc34c0c85f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:05.946 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:05.940+0000 7fc351133640 1 --2- 172.21.15.16:0/3674022654 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fc3380655c0 0x7fc338067a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:05.946 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:05.940+0000 7fc351133640 1 --2- 172.21.15.16:0/3674022654 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc34c0cf870 0x7fc34c159290 unknown :-1 s=CLOSED pgs=2335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:05.947 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:05.940+0000 7fc351133640 1 --2- 172.21.15.16:0/3674022654 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc34c06e740 0x7fc34c163960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:05.947 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:05.940+0000 7fc351133640 1 -- 172.21.15.16:0/3674022654 >> 172.21.15.16:0/3674022654 conn(0x7fc34c0cc870 msgr2=0x7fc34c0b41c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:05:05.947 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:05.940+0000 7fc351133640 1 -- 172.21.15.16:0/3674022654 shutdown_connections 2023-05-22T16:05:05.948 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:05.940+0000 7fc351133640 1 -- 172.21.15.16:0/3674022654 wait complete. 2023-05-22T16:05:05.959 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-05-22T16:05:06.041 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T16:05:06.036+0000 7fdbee8f1180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-05-22T16:05:06.072 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.072+0000 7fac2d348640 1 Processor -- start 2023-05-22T16:05:06.073 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.072+0000 7fac2d348640 1 -- start start 2023-05-22T16:05:06.073 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.072+0000 7fac2d348640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fac280b2a40 0x7fac280cf510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:06.074 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.072+0000 7fac2d348640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fac280cfa50 0x7fac280ca8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:06.074 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.072+0000 7fac2d348640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac280ce160 con 0x7fac2815c130 2023-05-22T16:05:06.074 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.072+0000 7fac2d348640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac2806c4a0 con 0x7fac280b2a40 2023-05-22T16:05:06.074 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.072+0000 7fac2d348640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac280cece0 con 0x7fac280cfa50 2023-05-22T16:05:06.074 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.072+0000 7fac267fc640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fac280b2a40 0x7fac280cf510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:06.074 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.072+0000 7fac267fc640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fac280b2a40 0x7fac280cf510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46138/0 (socket says 172.21.15.16:46138) 2023-05-22T16:05:06.075 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.072+0000 7fac267fc640 1 -- 172.21.15.16:0/4084116090 learned_addr learned my addr 172.21.15.16:0/4084116090 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:05:06.075 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.072+0000 7fac277fe640 1 --2- 172.21.15.16:0/4084116090 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fac280cfa50 0x7fac280ca8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:06.075 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.072+0000 7fac267fc640 1 -- 172.21.15.16:0/4084116090 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fac280cfa50 msgr2=0x7fac280ca8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:05:06.075 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.072+0000 7fac267fc640 1 --2- 172.21.15.16:0/4084116090 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fac280cfa50 0x7fac280ca8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:06.076 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.072+0000 7fac267fc640 1 -- 172.21.15.16:0/4084116090 >> v1:172.21.15.16:6789/0 conn(0x7fac2815c130 legacy=0x7fac280b2330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:06.076 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.072+0000 7fac267fc640 1 -- 172.21.15.16:0/4084116090 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac280cfeb0 con 0x7fac280b2a40 2023-05-22T16:05:06.076 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.072+0000 7fac267fc640 1 --2- 172.21.15.16:0/4084116090 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fac280b2a40 0x7fac280cf510 secure :-1 s=READY pgs=2337 cs=0 l=1 rev1=1 crypto rx=0x7fac1c009870 tx=0x7fac1c01c540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3dd0e3f1c3077ca9 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:06.077 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.072+0000 7fac25ffb640 1 -- 172.21.15.16:0/4084116090 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac1c0a7070 con 0x7fac280b2a40 2023-05-22T16:05:06.077 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.076+0000 7fac25ffb640 1 -- 172.21.15.16:0/4084116090 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac1c0ab430 con 0x7fac280b2a40 2023-05-22T16:05:06.077 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.076+0000 7fac25ffb640 1 -- 172.21.15.16:0/4084116090 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac1c0ab7c0 con 0x7fac280b2a40 2023-05-22T16:05:06.078 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.076+0000 7fac2d348640 1 -- 172.21.15.16:0/4084116090 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fac280b2a40 msgr2=0x7fac280cf510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:06.078 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.076+0000 7fac2d348640 1 --2- 172.21.15.16:0/4084116090 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fac280b2a40 0x7fac280cf510 secure :-1 s=READY pgs=2337 cs=0 l=1 rev1=1 crypto rx=0x7fac1c009870 tx=0x7fac1c01c540 comp rx=0 tx=0).stop 2023-05-22T16:05:06.078 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.076+0000 7fac2d348640 1 -- 172.21.15.16:0/4084116090 shutdown_connections 2023-05-22T16:05:06.078 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.076+0000 7fac2d348640 1 --2- 172.21.15.16:0/4084116090 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fac280cfa50 0x7fac280ca8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:06.078 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.076+0000 7fac2d348640 1 --2- 172.21.15.16:0/4084116090 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fac280b2a40 0x7fac280cf510 unknown :-1 s=CLOSED pgs=2337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:06.078 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.076+0000 7fac2d348640 1 -- 172.21.15.16:0/4084116090 >> 172.21.15.16:0/4084116090 conn(0x7fac2806f5c0 msgr2=0x7fac2806f9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:05:06.079 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.076+0000 7fac2d348640 1 -- 172.21.15.16:0/4084116090 shutdown_connections 2023-05-22T16:05:06.079 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.076+0000 7fac2d348640 1 -- 172.21.15.16:0/4084116090 wait complete. 2023-05-22T16:05:06.080 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.076+0000 7fac2d348640 1 Processor -- start 2023-05-22T16:05:06.080 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.076+0000 7fac2d348640 1 -- start start 2023-05-22T16:05:06.081 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.076+0000 7fac2d348640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fac280b2a40 0x7fac28154930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:06.081 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.076+0000 7fac2d348640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fac280cfa50 0x7fac2815b4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:06.081 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.076+0000 7fac2d348640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fac2815c130 0x7fac280c77b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:06.081 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.076+0000 7fac2d348640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac280b4b00 con 0x7fac280b2a40 2023-05-22T16:05:06.081 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.076+0000 7fac2d348640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac281591e0 con 0x7fac2815c130 2023-05-22T16:05:06.081 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.076+0000 7fac2d348640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac280b3e80 con 0x7fac280cfa50 2023-05-22T16:05:06.081 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.076+0000 7fac26ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fac280b2a40 0x7fac28154930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:06.082 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.076+0000 7fac26ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fac280b2a40 0x7fac28154930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46154/0 (socket says 172.21.15.16:46154) 2023-05-22T16:05:06.082 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.076+0000 7fac26ffd640 1 -- 172.21.15.16:0/1668878417 learned_addr learned my addr 172.21.15.16:0/1668878417 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:05:06.082 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.076+0000 7fac267fc640 1 --2- 172.21.15.16:0/1668878417 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fac280cfa50 0x7fac2815b4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:06.082 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.076+0000 7fac26ffd640 1 -- 172.21.15.16:0/1668878417 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fac280cfa50 msgr2=0x7fac2815b4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:06.082 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.076+0000 7fac26ffd640 1 --2- 172.21.15.16:0/1668878417 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fac280cfa50 0x7fac2815b4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:06.082 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.076+0000 7fac26ffd640 1 -- 172.21.15.16:0/1668878417 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fac2815c130 msgr2=0x7fac280c77b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:06.082 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.076+0000 7fac277fe640 1 --2- 172.21.15.16:0/1668878417 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fac2815c130 0x7fac280c77b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:06.083 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.076+0000 7fac26ffd640 1 --2- 172.21.15.16:0/1668878417 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fac2815c130 0x7fac280c77b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:06.083 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.076+0000 7fac26ffd640 1 -- 172.21.15.16:0/1668878417 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac1c0094d0 con 0x7fac280b2a40 2023-05-22T16:05:06.083 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.076+0000 7fac26ffd640 1 --2- 172.21.15.16:0/1668878417 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fac280b2a40 0x7fac28154930 secure :-1 s=READY pgs=2338 cs=0 l=1 rev1=1 crypto rx=0x7fac1800c3c0 tx=0x7fac1800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:06.083 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.076+0000 7fac0f7fe640 1 -- 172.21.15.16:0/1668878417 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac180052d0 con 0x7fac280b2a40 2023-05-22T16:05:06.083 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.076+0000 7fac0f7fe640 1 -- 172.21.15.16:0/1668878417 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac18005430 con 0x7fac280b2a40 2023-05-22T16:05:06.083 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.076+0000 7fac2d348640 1 -- 172.21.15.16:0/1668878417 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac280cb600 con 0x7fac280b2a40 2023-05-22T16:05:06.083 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.076+0000 7fac0f7fe640 1 -- 172.21.15.16:0/1668878417 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac18003840 con 0x7fac280b2a40 2023-05-22T16:05:06.084 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.076+0000 7fac2d348640 1 -- 172.21.15.16:0/1668878417 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac280ce970 con 0x7fac280b2a40 2023-05-22T16:05:06.084 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.080+0000 7fac2d348640 1 -- 172.21.15.16:0/1668878417 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac280cb790 con 0x7fac280b2a40 2023-05-22T16:05:06.084 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.080+0000 7fac0f7fe640 1 -- 172.21.15.16:0/1668878417 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 89) v1 ==== 84446+0+0 (secure 0 0 0) 0x7fac180039a0 con 0x7fac280b2a40 2023-05-22T16:05:06.085 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.080+0000 7fac0f7fe640 1 --2- 172.21.15.16:0/1668878417 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fac140655c0 0x7fac14067a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:06.085 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.080+0000 7fac267fc640 1 -- 172.21.15.16:0/1668878417 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fac140655c0 msgr2=0x7fac14067a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/65890 2023-05-22T16:05:06.085 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.080+0000 7fac0f7fe640 1 -- 172.21.15.16:0/1668878417 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(181..181 src has 1..181) v4 ==== 6870+0+0 (secure 0 0 0) 0x7fac18082440 con 0x7fac280b2a40 2023-05-22T16:05:06.085 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.080+0000 7fac267fc640 1 --2- 172.21.15.16:0/1668878417 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fac140655c0 0x7fac14067a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-05-22T16:05:06.086 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.084+0000 7fac0f7fe640 1 -- 172.21.15.16:0/1668878417 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fac1804fa90 con 0x7fac280b2a40 2023-05-22T16:05:06.116 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T16:05:06.112+0000 7fdbee8f1180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-05-22T16:05:06.127 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T16:05:06.124+0000 7f1f8c5e7180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-05-22T16:05:06.206 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T16:05:06.204+0000 7f1f8c5e7180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-05-22T16:05:06.239 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.236+0000 7fac2d348640 1 -- 172.21.15.16:0/1668878417 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fac2804dbe0 con 0x7fac280b2a40 2023-05-22T16:05:06.241 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.240+0000 7fac0f7fe640 1 -- 172.21.15.16:0/1668878417 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v181) v1 ==== 74+0+18781 (secure 0 0 0) 0x7fac180530b0 con 0x7fac280b2a40 2023-05-22T16:05:06.242 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T16:05:06.242 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":181,"fsid":"b2719834-7847-4d98-90a6-f65d57f85c49","created":"2023-05-22T15:26:25.259314+0000","modified":"2023-05-22T16:05:05.229755+0000","last_up_change":"2023-05-22T15:26:36.448524+0000","last_in_change":"2023-05-22T15:26:27.675668+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":23,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-05-22T15:26:38.019781+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-05-22T15:26:43.570514+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":22,"pool_name":"cephfs_metadata","create_time":"2023-05-22T16:03:37.371980+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"178","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_data","create_time":"2023-05-22T16:03:37.756887+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"181","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":4,"snap_epoch":181,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9e7ff854-b25f-4cbb-a5c2-1799d9c53d2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":37021},{"type":"v1","addr":"172.21.15.16:6825","nonce":37021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":37021},{"type":"v1","addr":"172.21.15.16:6827","nonce":37021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":37021},{"type":"v1","addr":"172.21.15.16:6831","nonce":37021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":37021},{"type":"v1","addr":"172.21.15.16:6829","nonce":37021}]},"public_addr":"172.21.15.16:6825/37021","cluster_addr":"172.21.15.16:6827/37021","heartbeat_back_addr":"172.21.15.16:6831/37021","heartbeat_front_addr":"172.21.15.16:6829/37021","state":["exists","up"]},{"osd":1,"uuid":"4702fbc9-a927-4565-aca5-41dc69cc8264","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":37008},{"type":"v1","addr":"172.21.15.16:6801","nonce":37008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":37008},{"type":"v1","addr":"172.21.15.16:6803","nonce":37008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":37008},{"type":"v1","addr":"172.21.15.16:6807","nonce":37008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":37008},{"type":"v1","addr":"172.21.15.16:6805","nonce":37008}]},"public_addr":"172.21.15.16:6801/37008","cluster_addr":"172.21.15.16:6803/37008","heartbeat_back_addr":"172.21.15.16:6807/37008","heartbeat_front_addr":"172.21.15.16:6805/37008","state":["exists","up"]},{"osd":2,"uuid":"1e7bba46-4eba-47ef-bb81-5e2d371638ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":37009},{"type":"v1","addr":"172.21.15.16:6809","nonce":37009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":37009},{"type":"v1","addr":"172.21.15.16:6811","nonce":37009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":37009},{"type":"v1","addr":"172.21.15.16:6815","nonce":37009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":37009},{"type":"v1","addr":"172.21.15.16:6813","nonce":37009}]},"public_addr":"172.21.15.16:6809/37009","cluster_addr":"172.21.15.16:6811/37009","heartbeat_back_addr":"172.21.15.16:6815/37009","heartbeat_front_addr":"172.21.15.16:6813/37009","state":["exists","up"]},{"osd":3,"uuid":"383cacc0-5a88-4904-990e-498ca59ef48a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":37020},{"type":"v1","addr":"172.21.15.16:6817","nonce":37020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":37020},{"type":"v1","addr":"172.21.15.16:6819","nonce":37020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":37020},{"type":"v1","addr":"172.21.15.16:6823","nonce":37020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":37020},{"type":"v1","addr":"172.21.15.16:6821","nonce":37020}]},"public_addr":"172.21.15.16:6817/37020","cluster_addr":"172.21.15.16:6819/37020","heartbeat_back_addr":"172.21.15.16:6823/37020","heartbeat_front_addr":"172.21.15.16:6821/37020","state":["exists","up"]},{"osd":4,"uuid":"970500ef-95b7-42c1-b080-eed4fae7013d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":37353},{"type":"v1","addr":"172.21.15.62:6817","nonce":37353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6819","nonce":37353},{"type":"v1","addr":"172.21.15.62:6822","nonce":37353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37353},{"type":"v1","addr":"172.21.15.62:6831","nonce":37353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37353},{"type":"v1","addr":"172.21.15.62:6826","nonce":37353}]},"public_addr":"172.21.15.62:6817/37353","cluster_addr":"172.21.15.62:6822/37353","heartbeat_back_addr":"172.21.15.62:6831/37353","heartbeat_front_addr":"172.21.15.62:6826/37353","state":["exists","up"]},{"osd":5,"uuid":"022d67db-d4d8-445b-b1b9-8b446a00f5c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":37352},{"type":"v1","addr":"172.21.15.62:6801","nonce":37352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":37352},{"type":"v1","addr":"172.21.15.62:6803","nonce":37352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":37352},{"type":"v1","addr":"172.21.15.62:6807","nonce":37352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":37352},{"type":"v1","addr":"172.21.15.62:6805","nonce":37352}]},"public_addr":"172.21.15.62:6801/37352","cluster_addr":"172.21.15.62:6803/37352","heartbeat_back_addr":"172.21.15.62:6807/37352","heartbeat_front_addr":"172.21.15.62:6805/37352","state":["exists","up"]},{"osd":6,"uuid":"42bd372b-6688-4a1b-8407-e16e8c266e33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":37354},{"type":"v1","addr":"172.21.15.62:6820","nonce":37354}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":37354},{"type":"v1","addr":"172.21.15.62:6823","nonce":37354}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6829","nonce":37354},{"type":"v1","addr":"172.21.15.62:6830","nonce":37354}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6825","nonce":37354},{"type":"v1","addr":"172.21.15.62:6827","nonce":37354}]},"public_addr":"172.21.15.62:6820/37354","cluster_addr":"172.21.15.62:6823/37354","heartbeat_back_addr":"172.21.15.62:6830/37354","heartbeat_front_addr":"172.21.15.62:6827/37354","state":["exists","up"]},{"osd":7,"uuid":"ac70b2a3-f41b-4bf2-b227-7f2cbb55426d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37355},{"type":"v1","addr":"172.21.15.62:6809","nonce":37355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37355},{"type":"v1","addr":"172.21.15.62:6811","nonce":37355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":37355},{"type":"v1","addr":"172.21.15.62:6815","nonce":37355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":37355},{"type":"v1","addr":"172.21.15.62:6813","nonce":37355}]},"public_addr":"172.21.15.62:6809/37355","cluster_addr":"172.21.15.62:6811/37355","heartbeat_back_addr":"172.21.15.62:6815/37355","heartbeat_front_addr":"172.21.15.62:6813/37355","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.368896+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.107273+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.276888+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.287502+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.124791+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.251937+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.255679+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.232797+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:0/2495910544":"2023-05-23T16:03:52.657800+0000","172.21.15.16:0/1761665129":"2023-05-23T16:03:52.657800+0000","172.21.15.16:6833/65890":"2023-05-23T16:03:52.657800+0000","172.21.15.16:6832/65890":"2023-05-23T16:03:52.657800+0000","172.21.15.16:0/4138270792":"2023-05-23T16:03:52.657800+0000","172.21.15.16:0/2442191907":"2023-05-23T16:03:52.657800+0000","172.21.15.16:0/3487890192":"2023-05-23T16:03:52.657800+0000","172.21.15.16:0/3247164174":"2023-05-23T16:03:52.657800+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":17,"snaps":[{"begin":2,"length":2}]},{"pool":21,"snaps":[{"begin":25,"length":9}]},{"pool":23,"snaps":[{"begin":2,"length":1},{"begin":4,"length":1}]}],"new_removed_snaps":[{"pool":23,"snaps":[{"begin":2,"length":1},{"begin":4,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-05-22T16:05:06.243 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.240+0000 7fac2d348640 1 -- 172.21.15.16:0/1668878417 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fac140655c0 msgr2=0x7fac14067a70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-05-22T16:05:06.243 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.240+0000 7fac2d348640 1 --2- 172.21.15.16:0/1668878417 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fac140655c0 0x7fac14067a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:06.243 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.240+0000 7fac2d348640 1 -- 172.21.15.16:0/1668878417 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fac280b2a40 msgr2=0x7fac28154930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:06.244 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.240+0000 7fac2d348640 1 --2- 172.21.15.16:0/1668878417 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fac280b2a40 0x7fac28154930 secure :-1 s=READY pgs=2338 cs=0 l=1 rev1=1 crypto rx=0x7fac1800c3c0 tx=0x7fac1800c880 comp rx=0 tx=0).stop 2023-05-22T16:05:06.244 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.240+0000 7fac2d348640 1 -- 172.21.15.16:0/1668878417 shutdown_connections 2023-05-22T16:05:06.244 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.240+0000 7fac2d348640 1 --2- 172.21.15.16:0/1668878417 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fac2815c130 0x7fac280c77b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:06.244 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.240+0000 7fac2d348640 1 --2- 172.21.15.16:0/1668878417 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fac280cfa50 0x7fac2815b4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:06.244 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.240+0000 7fac2d348640 1 --2- 172.21.15.16:0/1668878417 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fac140655c0 0x7fac14067a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:06.245 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.240+0000 7fac2d348640 1 --2- 172.21.15.16:0/1668878417 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fac280b2a40 0x7fac28154930 unknown :-1 s=CLOSED pgs=2338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:06.245 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.240+0000 7fac2d348640 1 -- 172.21.15.16:0/1668878417 >> 172.21.15.16:0/1668878417 conn(0x7fac2806f5c0 msgr2=0x7fac280cbb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:05:06.245 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.240+0000 7fac2d348640 1 -- 172.21.15.16:0/1668878417 shutdown_connections 2023-05-22T16:05:06.245 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.240+0000 7fac2d348640 1 -- 172.21.15.16:0/1668878417 wait complete. 2023-05-22T16:05:06.255 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T16:05:06.252+0000 7fdbee8f1180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-05-22T16:05:06.261 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-05-22T15:26:38.019781+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-05-22T15:26:43.570514+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 22, 'pool_name': 'cephfs_metadata', 'create_time': '2023-05-22T16:03:37.371980+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '178', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 23, 'pool_name': 'cephfs_data', 'create_time': '2023-05-22T16:03:37.756887+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '181', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 4, 'snap_epoch': 181, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-05-22T16:05:06.262 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-05-22T16:05:06.340 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T16:05:06.336+0000 7fdbee8f1180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-05-22T16:05:06.356 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T16:05:06.352+0000 7f1f8c5e7180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-05-22T16:05:06.384 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.380+0000 7fc68ef65640 1 Processor -- start 2023-05-22T16:05:06.385 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.380+0000 7fc68ef65640 1 -- start start 2023-05-22T16:05:06.387 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.384+0000 7fc68ef65640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc688070760 0x7fc688070be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:06.387 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.384+0000 7fc68ef65640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fc68806c6e0 0x7fc68806cb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:06.387 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.384+0000 7fc68ef65640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc68806eb90 con 0x7fc688158a10 2023-05-22T16:05:06.388 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.384+0000 7fc68ef65640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc68806e060 con 0x7fc688070760 2023-05-22T16:05:06.388 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.384+0000 7fc68ef65640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc68806d0c0 con 0x7fc68806c6e0 2023-05-22T16:05:06.389 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.384+0000 7fc68cd01640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fc688158a10 0x7fc68806e480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:45454/0 (socket says 172.21.15.16:45454) 2023-05-22T16:05:06.389 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.384+0000 7fc68cd01640 1 -- 172.21.15.16:0/2154498913 learned_addr learned my addr 172.21.15.16:0/2154498913 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T16:05:06.389 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.384+0000 7fc687fff640 1 --2- 172.21.15.16:0/2154498913 >> v2:172.21.15.16:3300/0 conn(0x7fc688070760 0x7fc688070be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:06.390 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.384+0000 7fc68d502640 1 --2- 172.21.15.16:0/2154498913 >> v2:172.21.15.62:3300/0 conn(0x7fc68806c6e0 0x7fc68806cb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:06.390 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.384+0000 7fc687fff640 1 -- 172.21.15.16:0/2154498913 >> v2:172.21.15.62:3300/0 conn(0x7fc68806c6e0 msgr2=0x7fc68806cb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:05:06.390 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.384+0000 7fc687fff640 1 --2- 172.21.15.16:0/2154498913 >> v2:172.21.15.62:3300/0 conn(0x7fc68806c6e0 0x7fc68806cb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:06.390 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.384+0000 7fc687fff640 1 -- 172.21.15.16:0/2154498913 >> v1:172.21.15.16:6789/0 conn(0x7fc688158a10 legacy=0x7fc68806e480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:06.390 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.384+0000 7fc687fff640 1 -- 172.21.15.16:0/2154498913 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc688158de0 con 0x7fc688070760 2023-05-22T16:05:06.394 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.384+0000 7fc687fff640 1 --2- 172.21.15.16:0/2154498913 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc688070760 0x7fc688070be0 secure :-1 s=READY pgs=2340 cs=0 l=1 rev1=1 crypto rx=0x7fc688071be0 tx=0x7fc67001e5e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3f9ed37eaa4fffd server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:06.394 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.384+0000 7fc6877fe640 1 -- 172.21.15.16:0/2154498913 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6700aa610 con 0x7fc688070760 2023-05-22T16:05:06.395 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.384+0000 7fc6877fe640 1 -- 172.21.15.16:0/2154498913 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6700aa770 con 0x7fc688070760 2023-05-22T16:05:06.395 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.384+0000 7fc6877fe640 1 -- 172.21.15.16:0/2154498913 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6700aab00 con 0x7fc688070760 2023-05-22T16:05:06.395 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.388+0000 7fc68ef65640 1 -- 172.21.15.16:0/2154498913 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc688070760 msgr2=0x7fc688070be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:06.395 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.388+0000 7fc68ef65640 1 --2- 172.21.15.16:0/2154498913 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc688070760 0x7fc688070be0 secure :-1 s=READY pgs=2340 cs=0 l=1 rev1=1 crypto rx=0x7fc688071be0 tx=0x7fc67001e5e0 comp rx=0 tx=0).stop 2023-05-22T16:05:06.395 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.388+0000 7fc68ef65640 1 -- 172.21.15.16:0/2154498913 shutdown_connections 2023-05-22T16:05:06.395 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.388+0000 7fc68ef65640 1 --2- 172.21.15.16:0/2154498913 >> v2:172.21.15.62:3300/0 conn(0x7fc68806c6e0 0x7fc68806cb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:06.395 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.388+0000 7fc68ef65640 1 --2- 172.21.15.16:0/2154498913 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc688070760 0x7fc688070be0 unknown :-1 s=CLOSED pgs=2340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:06.396 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.388+0000 7fc68ef65640 1 -- 172.21.15.16:0/2154498913 >> 172.21.15.16:0/2154498913 conn(0x7fc6880cc870 msgr2=0x7fc6880cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:05:06.396 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.388+0000 7fc68ef65640 1 -- 172.21.15.16:0/2154498913 shutdown_connections 2023-05-22T16:05:06.396 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.388+0000 7fc68ef65640 1 -- 172.21.15.16:0/2154498913 wait complete. 2023-05-22T16:05:06.396 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.388+0000 7fc68ef65640 1 Processor -- start 2023-05-22T16:05:06.396 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.388+0000 7fc68ef65640 1 -- start start 2023-05-22T16:05:06.396 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.388+0000 7fc68ef65640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc68806c6e0 0x7fc6880b4a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:06.397 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.388+0000 7fc68ef65640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc688070760 0x7fc6880ca650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:06.397 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.388+0000 7fc68ef65640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc688158a10 0x7fc6880cab90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:06.397 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.388+0000 7fc68ef65640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6880b4f40 con 0x7fc688070760 2023-05-22T16:05:06.397 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.388+0000 7fc68ef65640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc688159000 con 0x7fc68806c6e0 2023-05-22T16:05:06.397 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.388+0000 7fc68ef65640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6881592e0 con 0x7fc688158a10 2023-05-22T16:05:06.397 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.388+0000 7fc687fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc688070760 0x7fc6880ca650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:06.397 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.388+0000 7fc687fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc688070760 0x7fc6880ca650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46160/0 (socket says 172.21.15.16:46160) 2023-05-22T16:05:06.397 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.388+0000 7fc687fff640 1 -- 172.21.15.16:0/3849540380 learned_addr learned my addr 172.21.15.16:0/3849540380 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:05:06.398 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.388+0000 7fc68cd01640 1 --2- 172.21.15.16:0/3849540380 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc68806c6e0 0x7fc6880b4a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:06.398 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.388+0000 7fc68d502640 1 --2- 172.21.15.16:0/3849540380 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc688158a10 0x7fc6880cab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:06.398 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.388+0000 7fc687fff640 1 -- 172.21.15.16:0/3849540380 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc688158a10 msgr2=0x7fc6880cab90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:06.398 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.388+0000 7fc687fff640 1 --2- 172.21.15.16:0/3849540380 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc688158a10 0x7fc6880cab90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:06.398 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.388+0000 7fc687fff640 1 -- 172.21.15.16:0/3849540380 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc68806c6e0 msgr2=0x7fc6880b4a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:06.398 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.388+0000 7fc687fff640 1 --2- 172.21.15.16:0/3849540380 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc68806c6e0 0x7fc6880b4a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:06.399 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.388+0000 7fc687fff640 1 -- 172.21.15.16:0/3849540380 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc670009420 con 0x7fc688070760 2023-05-22T16:05:06.399 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.388+0000 7fc687fff640 1 --2- 172.21.15.16:0/3849540380 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc688070760 0x7fc6880ca650 secure :-1 s=READY pgs=2341 cs=0 l=1 rev1=1 crypto rx=0x7fc67001b1f0 tx=0x7fc67001b380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:06.399 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.388+0000 7fc6857fa640 1 -- 172.21.15.16:0/3849540380 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6700ba640 con 0x7fc688070760 2023-05-22T16:05:06.399 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.388+0000 7fc6857fa640 1 -- 172.21.15.16:0/3849540380 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6700ba7a0 con 0x7fc688070760 2023-05-22T16:05:06.399 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.388+0000 7fc68ef65640 1 -- 172.21.15.16:0/3849540380 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6880cfe40 con 0x7fc688070760 2023-05-22T16:05:06.399 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.388+0000 7fc6857fa640 1 -- 172.21.15.16:0/3849540380 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6700bab50 con 0x7fc688070760 2023-05-22T16:05:06.399 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.388+0000 7fc68ef65640 1 -- 172.21.15.16:0/3849540380 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc68806dc80 con 0x7fc688070760 2023-05-22T16:05:06.399 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.392+0000 7fc68ef65640 1 -- 172.21.15.16:0/3849540380 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6880ca280 con 0x7fc688070760 2023-05-22T16:05:06.400 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.392+0000 7fc6857fa640 1 -- 172.21.15.16:0/3849540380 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 89) v1 ==== 84446+0+0 (secure 0 0 0) 0x7fc6700a8040 con 0x7fc688070760 2023-05-22T16:05:06.400 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.392+0000 7fc6857fa640 1 --2- 172.21.15.16:0/3849540380 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fc6680655c0 0x7fc668067a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:06.400 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.392+0000 7fc68cd01640 1 -- 172.21.15.16:0/3849540380 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fc6680655c0 msgr2=0x7fc668067a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/65890 2023-05-22T16:05:06.400 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.392+0000 7fc6857fa640 1 -- 172.21.15.16:0/3849540380 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(182..182 src has 1..182) v4 ==== 6854+0+0 (secure 0 0 0) 0x7fc67011db30 con 0x7fc688070760 2023-05-22T16:05:06.400 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.392+0000 7fc68cd01640 1 --2- 172.21.15.16:0/3849540380 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fc6680655c0 0x7fc668067a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-05-22T16:05:06.401 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.396+0000 7fc6857fa640 1 -- 172.21.15.16:0/3849540380 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fc6700eb130 con 0x7fc688070760 2023-05-22T16:05:06.448 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T16:05:06.444+0000 7f1f8c5e7180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-05-22T16:05:06.569 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.568+0000 7fc68ef65640 1 -- 172.21.15.16:0/3849540380 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fc68804dbe0 con 0x7fc688070760 2023-05-22T16:05:06.570 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.568+0000 7fc6857fa640 1 -- 172.21.15.16:0/3849540380 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v182) v1 ==== 93+0+10 (secure 0 0 0) 0x7fc6700ee750 con 0x7fc688070760 2023-05-22T16:05:06.571 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2023-05-22T16:05:06.573 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.568+0000 7fc68ef65640 1 -- 172.21.15.16:0/3849540380 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fc6680655c0 msgr2=0x7fc668067a70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-05-22T16:05:06.573 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.568+0000 7fc68ef65640 1 --2- 172.21.15.16:0/3849540380 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fc6680655c0 0x7fc668067a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:06.573 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.568+0000 7fc68ef65640 1 -- 172.21.15.16:0/3849540380 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc688070760 msgr2=0x7fc6880ca650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:06.573 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.568+0000 7fc68ef65640 1 --2- 172.21.15.16:0/3849540380 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc688070760 0x7fc6880ca650 secure :-1 s=READY pgs=2341 cs=0 l=1 rev1=1 crypto rx=0x7fc67001b1f0 tx=0x7fc67001b380 comp rx=0 tx=0).stop 2023-05-22T16:05:06.574 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.568+0000 7fc68ef65640 1 -- 172.21.15.16:0/3849540380 shutdown_connections 2023-05-22T16:05:06.574 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.568+0000 7fc68ef65640 1 --2- 172.21.15.16:0/3849540380 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc688158a10 0x7fc6880cab90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:06.574 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.568+0000 7fc68ef65640 1 --2- 172.21.15.16:0/3849540380 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fc6680655c0 0x7fc668067a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:06.574 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.568+0000 7fc68ef65640 1 --2- 172.21.15.16:0/3849540380 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc688070760 0x7fc6880ca650 unknown :-1 s=CLOSED pgs=2341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:06.575 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.568+0000 7fc68ef65640 1 --2- 172.21.15.16:0/3849540380 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc68806c6e0 0x7fc6880b4a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:06.575 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.568+0000 7fc68ef65640 1 -- 172.21.15.16:0/3849540380 >> 172.21.15.16:0/3849540380 conn(0x7fc6880cc870 msgr2=0x7fc6880ccec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:05:06.575 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.572+0000 7fc68ef65640 1 -- 172.21.15.16:0/3849540380 shutdown_connections 2023-05-22T16:05:06.575 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.572+0000 7fc68ef65640 1 -- 172.21.15.16:0/3849540380 wait complete. 2023-05-22T16:05:06.588 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-05-22T16:05:06.592 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T16:05:06.588+0000 7fdbee8f1180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-05-22T16:05:06.646 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T16:05:06.640+0000 7fdbee8f1180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-05-22T16:05:06.705 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T16:05:06.704+0000 7f1f8c5e7180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-05-22T16:05:06.721 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.720+0000 7f6935233640 1 Processor -- start 2023-05-22T16:05:06.721 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.720+0000 7f6935233640 1 -- start start 2023-05-22T16:05:06.722 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.720+0000 7f6935233640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f69300c67d0 0x7f69300cf730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:06.722 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.720+0000 7f6935233640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f69300cc6f0 0x7f69300c3db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:06.723 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.720+0000 7f6935233640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69300cfc70 con 0x7f69300b2880 2023-05-22T16:05:06.723 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.720+0000 7f6935233640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f693015a300 con 0x7f69300c67d0 2023-05-22T16:05:06.723 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.720+0000 7f6935233640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69300c6ba0 con 0x7f69300cc6f0 2023-05-22T16:05:06.723 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.720+0000 7f692ed9d640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f69300c67d0 0x7f69300cf730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:06.723 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.720+0000 7f692ed9d640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f69300c67d0 0x7f69300cf730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46168/0 (socket says 172.21.15.16:46168) 2023-05-22T16:05:06.723 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.720+0000 7f692ed9d640 1 -- 172.21.15.16:0/1956725659 learned_addr learned my addr 172.21.15.16:0/1956725659 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:05:06.724 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.720+0000 7f692f59e640 1 --2- 172.21.15.16:0/1956725659 >> v2:172.21.15.62:3300/0 conn(0x7f69300cc6f0 0x7f69300c3db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:06.724 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.720+0000 7f692ed9d640 1 -- 172.21.15.16:0/1956725659 >> v2:172.21.15.62:3300/0 conn(0x7f69300cc6f0 msgr2=0x7f69300c3db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:05:06.724 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.720+0000 7f692ed9d640 1 --2- 172.21.15.16:0/1956725659 >> v2:172.21.15.62:3300/0 conn(0x7f69300cc6f0 0x7f69300c3db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:06.724 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.720+0000 7f692ed9d640 1 -- 172.21.15.16:0/1956725659 >> v1:172.21.15.16:6789/0 conn(0x7f69300b2880 legacy=0x7f69300cbfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:06.725 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.720+0000 7f692ed9d640 1 -- 172.21.15.16:0/1956725659 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69300cfef0 con 0x7f69300c67d0 2023-05-22T16:05:06.726 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.724+0000 7f692ed9d640 1 --2- 172.21.15.16:0/1956725659 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f69300c67d0 0x7f69300cf730 secure :-1 s=READY pgs=2343 cs=0 l=1 rev1=1 crypto rx=0x7f69300b6360 tx=0x7f692801c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39a058283576f39e server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:06.726 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.724+0000 7f692dd9b640 1 -- 172.21.15.16:0/1956725659 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69280a56e0 con 0x7f69300c67d0 2023-05-22T16:05:06.726 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.724+0000 7f692dd9b640 1 -- 172.21.15.16:0/1956725659 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69280a5840 con 0x7f69300c67d0 2023-05-22T16:05:06.726 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.724+0000 7f692dd9b640 1 -- 172.21.15.16:0/1956725659 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69280a5bd0 con 0x7f69300c67d0 2023-05-22T16:05:06.726 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.724+0000 7f6935233640 1 -- 172.21.15.16:0/1956725659 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f69300c67d0 msgr2=0x7f69300cf730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:06.726 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.724+0000 7f6935233640 1 --2- 172.21.15.16:0/1956725659 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f69300c67d0 0x7f69300cf730 secure :-1 s=READY pgs=2343 cs=0 l=1 rev1=1 crypto rx=0x7f69300b6360 tx=0x7f692801c910 comp rx=0 tx=0).stop 2023-05-22T16:05:06.727 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.724+0000 7f6935233640 1 -- 172.21.15.16:0/1956725659 shutdown_connections 2023-05-22T16:05:06.727 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.724+0000 7f6935233640 1 --2- 172.21.15.16:0/1956725659 >> v2:172.21.15.62:3300/0 conn(0x7f69300cc6f0 0x7f69300c3db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:06.727 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.724+0000 7f6935233640 1 --2- 172.21.15.16:0/1956725659 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f69300c67d0 0x7f69300cf730 unknown :-1 s=CLOSED pgs=2343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:06.727 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.724+0000 7f6935233640 1 -- 172.21.15.16:0/1956725659 >> 172.21.15.16:0/1956725659 conn(0x7f693006f5a0 msgr2=0x7f693006f9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:05:06.727 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.724+0000 7f6935233640 1 -- 172.21.15.16:0/1956725659 shutdown_connections 2023-05-22T16:05:06.727 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.724+0000 7f6935233640 1 -- 172.21.15.16:0/1956725659 wait complete. 2023-05-22T16:05:06.729 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.724+0000 7f6935233640 1 Processor -- start 2023-05-22T16:05:06.730 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.724+0000 7f6935233640 1 -- start start 2023-05-22T16:05:06.731 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.724+0000 7f6935233640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f69300b2880 0x7f69300b5ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:06.731 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.724+0000 7f6935233640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f69300c67d0 0x7f693015d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:06.731 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.724+0000 7f6935233640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f69300cc6f0 0x7f6930158290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:06.732 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.724+0000 7f6935233640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69300b61e0 con 0x7f69300c67d0 2023-05-22T16:05:06.732 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.724+0000 7f6935233640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6930154e10 con 0x7f69300cc6f0 2023-05-22T16:05:06.732 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.724+0000 7f6935233640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69301550f0 con 0x7f69300b2880 2023-05-22T16:05:06.733 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.724+0000 7f692e59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f69300c67d0 0x7f693015d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:06.733 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.724+0000 7f692e59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f69300c67d0 0x7f693015d8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46176/0 (socket says 172.21.15.16:46176) 2023-05-22T16:05:06.733 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.724+0000 7f692e59c640 1 -- 172.21.15.16:0/3611861096 learned_addr learned my addr 172.21.15.16:0/3611861096 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:05:06.734 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.724+0000 7f692ed9d640 1 --2- 172.21.15.16:0/3611861096 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f69300b2880 0x7f69300b5ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:06.734 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.724+0000 7f692f59e640 1 --2- 172.21.15.16:0/3611861096 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f69300cc6f0 0x7f6930158290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:06.734 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.724+0000 7f692e59c640 1 -- 172.21.15.16:0/3611861096 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f69300b2880 msgr2=0x7f69300b5ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:06.734 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.724+0000 7f692e59c640 1 --2- 172.21.15.16:0/3611861096 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f69300b2880 0x7f69300b5ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:06.734 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.724+0000 7f692e59c640 1 -- 172.21.15.16:0/3611861096 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f69300cc6f0 msgr2=0x7f6930158290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:06.734 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.724+0000 7f692e59c640 1 --2- 172.21.15.16:0/3611861096 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f69300cc6f0 0x7f6930158290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:06.735 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.724+0000 7f692e59c640 1 -- 172.21.15.16:0/3611861096 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6928009b30 con 0x7f69300c67d0 2023-05-22T16:05:06.735 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.724+0000 7f692ed9d640 1 --2- 172.21.15.16:0/3611861096 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f69300b2880 0x7f69300b5ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T16:05:06.735 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.724+0000 7f692f59e640 1 --2- 172.21.15.16:0/3611861096 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f69300cc6f0 0x7f6930158290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T16:05:06.735 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.724+0000 7f692e59c640 1 --2- 172.21.15.16:0/3611861096 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f69300c67d0 0x7f693015d8e0 secure :-1 s=READY pgs=2344 cs=0 l=1 rev1=1 crypto rx=0x7f691800c370 tx=0x7f691800c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:06.735 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.724+0000 7f69277fe640 1 -- 172.21.15.16:0/3611861096 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69180052d0 con 0x7f69300c67d0 2023-05-22T16:05:06.735 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.724+0000 7f69277fe640 1 -- 172.21.15.16:0/3611861096 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6918005430 con 0x7f69300c67d0 2023-05-22T16:05:06.735 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.724+0000 7f6935233640 1 -- 172.21.15.16:0/3611861096 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69300d0a70 con 0x7f69300c67d0 2023-05-22T16:05:06.736 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.724+0000 7f69277fe640 1 -- 172.21.15.16:0/3611861096 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6918007260 con 0x7f69300c67d0 2023-05-22T16:05:06.736 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.728+0000 7f6935233640 1 -- 172.21.15.16:0/3611861096 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69300d0790 con 0x7f69300c67d0 2023-05-22T16:05:06.736 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.728+0000 7f6935233640 1 -- 172.21.15.16:0/3611861096 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69300d0f90 con 0x7f69300c67d0 2023-05-22T16:05:06.737 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.728+0000 7f69277fe640 1 -- 172.21.15.16:0/3611861096 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 89) v1 ==== 84446+0+0 (secure 0 0 0) 0x7f69180073c0 con 0x7f69300c67d0 2023-05-22T16:05:06.737 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.728+0000 7f69277fe640 1 --2- 172.21.15.16:0/3611861096 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f69100655f0 0x7f6910067aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:06.737 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.728+0000 7f69277fe640 1 -- 172.21.15.16:0/3611861096 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(182..182 src has 1..182) v4 ==== 6854+0+0 (secure 0 0 0) 0x7f6918082230 con 0x7f69300c67d0 2023-05-22T16:05:06.737 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.728+0000 7f692ed9d640 1 -- 172.21.15.16:0/3611861096 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f69100655f0 msgr2=0x7f6910067aa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/65890 2023-05-22T16:05:06.737 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.728+0000 7f692ed9d640 1 --2- 172.21.15.16:0/3611861096 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f69100655f0 0x7f6910067aa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-05-22T16:05:06.738 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.736+0000 7f69277fe640 1 -- 172.21.15.16:0/3611861096 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f6918049620 con 0x7f69300c67d0 2023-05-22T16:05:06.766 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T16:05:06.764+0000 7f1f8c5e7180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-05-22T16:05:06.803 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.800+0000 7f69277fe640 1 -- 172.21.15.16:0/3611861096 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mgrmap(e 90) v1 ==== 83941+0+0 (secure 0 0 0) 0x7f691804fcd0 con 0x7f69300c67d0 2023-05-22T16:05:06.804 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.800+0000 7f69277fe640 1 -- 172.21.15.16:0/3611861096 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f69100655f0 msgr2=0x7f6910067aa0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-05-22T16:05:06.804 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.800+0000 7f69277fe640 1 --2- 172.21.15.16:0/3611861096 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f69100655f0 0x7f6910067aa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:06.880 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T16:05:06.876+0000 7f1f18775640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T16:05:06.880 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T16:05:06.876+0000 7f1f18775640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T16:05:06.880 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T16:05:06.876+0000 7f1f18775640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T16:05:06.881 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T16:05:06.876+0000 7f1f18775640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T16:05:06.881 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T16:05:06.876+0000 7f1f18775640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T16:05:06.881 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T16:05:06.876+0000 7f1f1576f640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T16:05:06.881 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T16:05:06.876+0000 7f1f1576f640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T16:05:06.881 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T16:05:06.876+0000 7f1f1576f640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T16:05:06.881 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T16:05:06.876+0000 7f1f1576f640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T16:05:06.881 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T16:05:06.876+0000 7f1f1576f640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T16:05:06.930 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.928+0000 7f6935233640 1 -- 172.21.15.16:0/3611861096 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f693004dbe0 con 0x7f69300c67d0 2023-05-22T16:05:06.930 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.928+0000 7f69277fe640 1 -- 172.21.15.16:0/3611861096 <== mon.0 v2:172.21.15.16:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v183) v1 ==== 92+0+10 (secure 0 0 0) 0x7f691805c890 con 0x7f69300c67d0 2023-05-22T16:05:06.931 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 8 2023-05-22T16:05:06.932 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.928+0000 7f6935233640 1 -- 172.21.15.16:0/3611861096 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f69300c67d0 msgr2=0x7f693015d8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:06.933 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.928+0000 7f6935233640 1 --2- 172.21.15.16:0/3611861096 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f69300c67d0 0x7f693015d8e0 secure :-1 s=READY pgs=2344 cs=0 l=1 rev1=1 crypto rx=0x7f691800c370 tx=0x7f691800c830 comp rx=0 tx=0).stop 2023-05-22T16:05:06.933 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.928+0000 7f6935233640 1 -- 172.21.15.16:0/3611861096 shutdown_connections 2023-05-22T16:05:06.933 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.928+0000 7f6935233640 1 --2- 172.21.15.16:0/3611861096 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f69300cc6f0 0x7f6930158290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:06.933 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.928+0000 7f6935233640 1 --2- 172.21.15.16:0/3611861096 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f69100655f0 0x7f6910067aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:06.933 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.928+0000 7f6935233640 1 --2- 172.21.15.16:0/3611861096 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f69300c67d0 0x7f693015d8e0 unknown :-1 s=CLOSED pgs=2344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:06.933 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.928+0000 7f6935233640 1 --2- 172.21.15.16:0/3611861096 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f69300b2880 0x7f69300b5ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:06.933 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.928+0000 7f6935233640 1 -- 172.21.15.16:0/3611861096 >> 172.21.15.16:0/3611861096 conn(0x7f693006f5a0 msgr2=0x7f69300ce160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:05:06.934 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.928+0000 7f6935233640 1 -- 172.21.15.16:0/3611861096 shutdown_connections 2023-05-22T16:05:06.934 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:06.928+0000 7f6935233640 1 -- 172.21.15.16:0/3611861096 wait complete. 2023-05-22T16:05:06.947 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-05-22T16:05:07.059 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.056+0000 7f3cbcbfb640 1 Processor -- start 2023-05-22T16:05:07.060 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.056+0000 7f3cbcbfb640 1 -- start start 2023-05-22T16:05:07.061 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.060+0000 7f3cbcbfb640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3cb8159970 0x7f3cb8159d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:07.061 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.060+0000 7f3cbcbfb640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f3cb815d680 0x7f3cb80cb260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:07.061 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.060+0000 7f3cbcbfb640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f3cb815dad0 0x7f3cb806b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:07.061 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.060+0000 7f3cbcbfb640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3cb8070c30 con 0x7f3cb815d680 2023-05-22T16:05:07.061 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.060+0000 7f3cbcbfb640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3cb8070e60 con 0x7f3cb815dad0 2023-05-22T16:05:07.062 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.060+0000 7f3cbcbfb640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3cb8071140 con 0x7f3cb8159970 2023-05-22T16:05:07.062 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.060+0000 7f3cb5d9b640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f3cb815d680 0x7f3cb80cb260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:07.062 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.060+0000 7f3cb5d9b640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f3cb815d680 0x7f3cb80cb260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46318/0 (socket says 172.21.15.16:46318) 2023-05-22T16:05:07.062 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.060+0000 7f3cb5d9b640 1 -- 172.21.15.16:0/822155334 learned_addr learned my addr 172.21.15.16:0/822155334 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:05:07.062 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.060+0000 7f3cb6d9d640 1 --2- 172.21.15.16:0/822155334 >> v2:172.21.15.62:3300/0 conn(0x7f3cb815dad0 0x7f3cb806b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:07.062 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.060+0000 7f3cb659c640 1 --2- 172.21.15.16:0/822155334 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3cb8159970 0x7f3cb8159d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:07.062 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.060+0000 7f3cb5d9b640 1 -- 172.21.15.16:0/822155334 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3cb8159970 msgr2=0x7f3cb8159d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:05:07.063 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.060+0000 7f3cb5d9b640 1 --2- 172.21.15.16:0/822155334 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3cb8159970 0x7f3cb8159d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:07.063 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.060+0000 7f3cb5d9b640 1 -- 172.21.15.16:0/822155334 >> v2:172.21.15.62:3300/0 conn(0x7f3cb815dad0 msgr2=0x7f3cb806b6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:05:07.063 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.060+0000 7f3cb5d9b640 1 --2- 172.21.15.16:0/822155334 >> v2:172.21.15.62:3300/0 conn(0x7f3cb815dad0 0x7f3cb806b6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:07.063 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.060+0000 7f3cb5d9b640 1 -- 172.21.15.16:0/822155334 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cb8071420 con 0x7f3cb815d680 2023-05-22T16:05:07.064 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.060+0000 7f3cb5d9b640 1 --2- 172.21.15.16:0/822155334 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3cb815d680 0x7f3cb80cb260 secure :-1 s=READY pgs=2358 cs=0 l=1 rev1=1 crypto rx=0x7f3cb80b40e0 tx=0x7f3cac01c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee557e6ea46cde0c server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:07.064 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.060+0000 7f3cb559a640 1 -- 172.21.15.16:0/822155334 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cac0a56a0 con 0x7f3cb815d680 2023-05-22T16:05:07.064 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.060+0000 7f3cb559a640 1 -- 172.21.15.16:0/822155334 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3cac0a5800 con 0x7f3cb815d680 2023-05-22T16:05:07.064 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.060+0000 7f3cb559a640 1 -- 172.21.15.16:0/822155334 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cac0a5b90 con 0x7f3cb815d680 2023-05-22T16:05:07.065 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.060+0000 7f3cbcbfb640 1 -- 172.21.15.16:0/822155334 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3cb815d680 msgr2=0x7f3cb80cb260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:07.065 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.060+0000 7f3cbcbfb640 1 --2- 172.21.15.16:0/822155334 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3cb815d680 0x7f3cb80cb260 secure :-1 s=READY pgs=2358 cs=0 l=1 rev1=1 crypto rx=0x7f3cb80b40e0 tx=0x7f3cac01c910 comp rx=0 tx=0).stop 2023-05-22T16:05:07.065 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.060+0000 7f3cbcbfb640 1 -- 172.21.15.16:0/822155334 shutdown_connections 2023-05-22T16:05:07.065 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.060+0000 7f3cbcbfb640 1 --2- 172.21.15.16:0/822155334 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3cb815d680 0x7f3cb80cb260 unknown :-1 s=CLOSED pgs=2358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:07.065 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.060+0000 7f3cbcbfb640 1 --2- 172.21.15.16:0/822155334 >> v2:172.21.15.62:3300/0 conn(0x7f3cb815dad0 0x7f3cb806b6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:07.065 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.060+0000 7f3cbcbfb640 1 --2- 172.21.15.16:0/822155334 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3cb8159970 0x7f3cb8159d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:07.065 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.060+0000 7f3cbcbfb640 1 -- 172.21.15.16:0/822155334 >> 172.21.15.16:0/822155334 conn(0x7f3cb8069720 msgr2=0x7f3cb8069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:05:07.065 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.060+0000 7f3cbcbfb640 1 -- 172.21.15.16:0/822155334 shutdown_connections 2023-05-22T16:05:07.066 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.060+0000 7f3cbcbfb640 1 -- 172.21.15.16:0/822155334 wait complete. 2023-05-22T16:05:07.067 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.064+0000 7f3cbcbfb640 1 Processor -- start 2023-05-22T16:05:07.067 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.064+0000 7f3cbcbfb640 1 -- start start 2023-05-22T16:05:07.067 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.064+0000 7f3cbcbfb640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3cb8159970 0x7f3cb8151f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:07.068 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.064+0000 7f3cbcbfb640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3cb815d680 0x7f3cb8152700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:07.068 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.064+0000 7f3cbcbfb640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3cb815dad0 0x7f3cb81508b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:07.068 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.064+0000 7f3cbcbfb640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cb80c35a0 con 0x7f3cb8159970 2023-05-22T16:05:07.068 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.064+0000 7f3cbcbfb640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cb80c3860 con 0x7f3cb815dad0 2023-05-22T16:05:07.068 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.064+0000 7f3cbcbfb640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3cb80c3b40 con 0x7f3cb815d680 2023-05-22T16:05:07.068 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.064+0000 7f3cb659c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3cb8159970 0x7f3cb8151f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:07.068 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.064+0000 7f3cb659c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3cb8159970 0x7f3cb8151f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46334/0 (socket says 172.21.15.16:46334) 2023-05-22T16:05:07.068 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.064+0000 7f3cb659c640 1 -- 172.21.15.16:0/2047249439 learned_addr learned my addr 172.21.15.16:0/2047249439 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:05:07.069 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.064+0000 7f3cb5d9b640 1 --2- 172.21.15.16:0/2047249439 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3cb815d680 0x7f3cb8152700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:07.069 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.064+0000 7f3cb6d9d640 1 --2- 172.21.15.16:0/2047249439 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3cb815dad0 0x7f3cb81508b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:07.069 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.064+0000 7f3cb659c640 1 -- 172.21.15.16:0/2047249439 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3cb815d680 msgr2=0x7f3cb8152700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:07.069 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.064+0000 7f3cb659c640 1 --2- 172.21.15.16:0/2047249439 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3cb815d680 0x7f3cb8152700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:07.069 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.064+0000 7f3cb659c640 1 -- 172.21.15.16:0/2047249439 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3cb815dad0 msgr2=0x7f3cb81508b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:07.069 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.064+0000 7f3cb659c640 1 --2- 172.21.15.16:0/2047249439 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3cb815dad0 0x7f3cb81508b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:07.069 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.064+0000 7f3cb659c640 1 -- 172.21.15.16:0/2047249439 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cac009b30 con 0x7f3cb8159970 2023-05-22T16:05:07.070 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.064+0000 7f3cb659c640 1 --2- 172.21.15.16:0/2047249439 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3cb8159970 0x7f3cb8151f00 secure :-1 s=READY pgs=2359 cs=0 l=1 rev1=1 crypto rx=0x7f3cb000d800 tx=0x7f3cb000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:07.070 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.064+0000 7f3ca6ffd640 1 -- 172.21.15.16:0/2047249439 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cb00096a0 con 0x7f3cb8159970 2023-05-22T16:05:07.070 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.064+0000 7f3ca6ffd640 1 -- 172.21.15.16:0/2047249439 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3cb0009800 con 0x7f3cb8159970 2023-05-22T16:05:07.070 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.064+0000 7f3cbcbfb640 1 -- 172.21.15.16:0/2047249439 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3cb80cae20 con 0x7f3cb8159970 2023-05-22T16:05:07.070 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.064+0000 7f3ca6ffd640 1 -- 172.21.15.16:0/2047249439 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cb0009b70 con 0x7f3cb8159970 2023-05-22T16:05:07.070 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.064+0000 7f3cbcbfb640 1 -- 172.21.15.16:0/2047249439 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3cb80cac90 con 0x7f3cb8159970 2023-05-22T16:05:07.071 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.064+0000 7f3cbcbfb640 1 -- 172.21.15.16:0/2047249439 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3cb815adc0 con 0x7f3cb8159970 2023-05-22T16:05:07.071 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.064+0000 7f3ca6ffd640 1 -- 172.21.15.16:0/2047249439 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 90) v1 ==== 83941+0+0 (secure 0 0 0) 0x7f3cb0009cd0 con 0x7f3cb8159970 2023-05-22T16:05:07.071 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.068+0000 7f3ca6ffd640 1 -- 172.21.15.16:0/2047249439 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(183..183 src has 1..183) v4 ==== 7154+0+0 (secure 0 0 0) 0x7f3cb0082e80 con 0x7f3cb8159970 2023-05-22T16:05:07.072 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.072+0000 7f3ca6ffd640 1 -- 172.21.15.16:0/2047249439 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f3cb0050b00 con 0x7f3cb8159970 2023-05-22T16:05:07.228 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.224+0000 7f3cbcbfb640 1 -- 172.21.15.16:0/2047249439 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f3cb804dbe0 con 0x7f3cb8159970 2023-05-22T16:05:07.229 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.228+0000 7f3ca6ffd640 1 -- 172.21.15.16:0/2047249439 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v183) v1 ==== 104+0+11 (secure 0 0 0) 0x7f3cb0054120 con 0x7f3cb8159970 2023-05-22T16:05:07.229 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2023-05-22T16:05:07.231 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.228+0000 7f3cbcbfb640 1 -- 172.21.15.16:0/2047249439 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3cb8159970 msgr2=0x7f3cb8151f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:07.231 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.228+0000 7f3cbcbfb640 1 --2- 172.21.15.16:0/2047249439 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3cb8159970 0x7f3cb8151f00 secure :-1 s=READY pgs=2359 cs=0 l=1 rev1=1 crypto rx=0x7f3cb000d800 tx=0x7f3cb000dcc0 comp rx=0 tx=0).stop 2023-05-22T16:05:07.231 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.228+0000 7f3cbcbfb640 1 -- 172.21.15.16:0/2047249439 shutdown_connections 2023-05-22T16:05:07.231 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.228+0000 7f3cbcbfb640 1 --2- 172.21.15.16:0/2047249439 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3cb815dad0 0x7f3cb81508b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:07.231 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.228+0000 7f3cbcbfb640 1 --2- 172.21.15.16:0/2047249439 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3cb815d680 0x7f3cb8152700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:07.232 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.228+0000 7f3cbcbfb640 1 --2- 172.21.15.16:0/2047249439 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3cb8159970 0x7f3cb8151f00 unknown :-1 s=CLOSED pgs=2359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:07.232 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.228+0000 7f3cbcbfb640 1 -- 172.21.15.16:0/2047249439 >> 172.21.15.16:0/2047249439 conn(0x7f3cb8069720 msgr2=0x7f3cb80cd5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:05:07.232 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.228+0000 7f3cbcbfb640 1 -- 172.21.15.16:0/2047249439 shutdown_connections 2023-05-22T16:05:07.232 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.228+0000 7f3cbcbfb640 1 -- 172.21.15.16:0/2047249439 wait complete. 2023-05-22T16:05:07.248 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-05-22T16:05:07.369 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.368+0000 7f8678358640 1 Processor -- start 2023-05-22T16:05:07.369 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.368+0000 7f8678358640 1 -- start start 2023-05-22T16:05:07.370 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.368+0000 7f8678358640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f867015bca0 0x7f8670160160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:07.371 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.368+0000 7f8678358640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f86700b3ee0 0x7f86700b4330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:07.371 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.368+0000 7f8678358640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86700d0400 con 0x7f86700b4870 2023-05-22T16:05:07.371 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.368+0000 7f8678358640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f867015c070 con 0x7f867015bca0 2023-05-22T16:05:07.371 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.368+0000 7f8678358640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86701606a0 con 0x7f86700b3ee0 2023-05-22T16:05:07.371 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.368+0000 7f86760f4640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f867015bca0 0x7f8670160160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:07.371 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.368+0000 7f86760f4640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f867015bca0 0x7f8670160160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46340/0 (socket says 172.21.15.16:46340) 2023-05-22T16:05:07.372 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.368+0000 7f86760f4640 1 -- 172.21.15.16:0/406729259 learned_addr learned my addr 172.21.15.16:0/406729259 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:05:07.372 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.368+0000 7f86760f4640 1 -- 172.21.15.16:0/406729259 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f86700b3ee0 msgr2=0x7f86700b4330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:05:07.372 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.368+0000 7f86758f3640 1 --2- 172.21.15.16:0/406729259 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f86700b3ee0 0x7f86700b4330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:07.372 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.368+0000 7f86760f4640 1 --2- 172.21.15.16:0/406729259 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f86700b3ee0 0x7f86700b4330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:07.372 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.368+0000 7f86760f4640 1 -- 172.21.15.16:0/406729259 >> v1:172.21.15.16:6789/0 conn(0x7f86700b4870 legacy=0x7f86700ce900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:07.372 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.368+0000 7f86760f4640 1 -- 172.21.15.16:0/406729259 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86700d00d0 con 0x7f867015bca0 2023-05-22T16:05:07.373 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.372+0000 7f86760f4640 1 --2- 172.21.15.16:0/406729259 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f867015bca0 0x7f8670160160 secure :-1 s=READY pgs=2361 cs=0 l=1 rev1=1 crypto rx=0x7f8660009870 tx=0x7f866001c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd485615daf9876b server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:07.373 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.372+0000 7f86750f2640 1 -- 172.21.15.16:0/406729259 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86600ab680 con 0x7f867015bca0 2023-05-22T16:05:07.374 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.372+0000 7f86750f2640 1 -- 172.21.15.16:0/406729259 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86600ab7e0 con 0x7f867015bca0 2023-05-22T16:05:07.374 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.372+0000 7f86750f2640 1 -- 172.21.15.16:0/406729259 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86600abb70 con 0x7f867015bca0 2023-05-22T16:05:07.374 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.372+0000 7f8678358640 1 -- 172.21.15.16:0/406729259 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f867015bca0 msgr2=0x7f8670160160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:07.374 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.372+0000 7f8678358640 1 --2- 172.21.15.16:0/406729259 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f867015bca0 0x7f8670160160 secure :-1 s=READY pgs=2361 cs=0 l=1 rev1=1 crypto rx=0x7f8660009870 tx=0x7f866001c480 comp rx=0 tx=0).stop 2023-05-22T16:05:07.374 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.372+0000 7f8678358640 1 -- 172.21.15.16:0/406729259 shutdown_connections 2023-05-22T16:05:07.374 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.372+0000 7f8678358640 1 --2- 172.21.15.16:0/406729259 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f86700b3ee0 0x7f86700b4330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:07.375 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.372+0000 7f8678358640 1 --2- 172.21.15.16:0/406729259 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f867015bca0 0x7f8670160160 unknown :-1 s=CLOSED pgs=2361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:07.375 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.372+0000 7f8678358640 1 -- 172.21.15.16:0/406729259 >> 172.21.15.16:0/406729259 conn(0x7f86700657f0 msgr2=0x7f8670070e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:05:07.375 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.372+0000 7f8678358640 1 -- 172.21.15.16:0/406729259 shutdown_connections 2023-05-22T16:05:07.375 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.372+0000 7f8678358640 1 -- 172.21.15.16:0/406729259 wait complete. 2023-05-22T16:05:07.377 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.372+0000 7f8678358640 1 Processor -- start 2023-05-22T16:05:07.377 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.372+0000 7f8678358640 1 -- start start 2023-05-22T16:05:07.377 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.372+0000 7f8678358640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f86700b3ee0 0x7f86700b24b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:07.377 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.372+0000 7f8678358640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f86700b4870 0x7f86700b2c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:07.377 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.372+0000 7f8678358640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f867015bca0 0x7f86700b3150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:07.378 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.372+0000 7f8678358640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f867015c500 con 0x7f86700b3ee0 2023-05-22T16:05:07.378 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.372+0000 7f8678358640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86700b29f0 con 0x7f867015bca0 2023-05-22T16:05:07.378 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.372+0000 7f8678358640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8670152700 con 0x7f86700b4870 2023-05-22T16:05:07.378 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.372+0000 7f86760f4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f86700b3ee0 0x7f86700b24b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:07.378 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.372+0000 7f86760f4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f86700b3ee0 0x7f86700b24b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46344/0 (socket says 172.21.15.16:46344) 2023-05-22T16:05:07.378 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.372+0000 7f86760f4640 1 -- 172.21.15.16:0/2227449067 learned_addr learned my addr 172.21.15.16:0/2227449067 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:05:07.378 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.372+0000 7f86760f4640 1 -- 172.21.15.16:0/2227449067 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f86700b4870 msgr2=0x7f86700b2c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:07.378 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.372+0000 7f86760f4640 1 --2- 172.21.15.16:0/2227449067 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f86700b4870 0x7f86700b2c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:07.379 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.372+0000 7f86760f4640 1 -- 172.21.15.16:0/2227449067 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f867015bca0 msgr2=0x7f86700b3150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:07.379 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.372+0000 7f86760f4640 1 --2- 172.21.15.16:0/2227449067 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f867015bca0 0x7f86700b3150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:07.379 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.372+0000 7f86760f4640 1 -- 172.21.15.16:0/2227449067 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86600094d0 con 0x7f86700b3ee0 2023-05-22T16:05:07.379 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.372+0000 7f86760f4640 1 --2- 172.21.15.16:0/2227449067 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f86700b3ee0 0x7f86700b24b0 secure :-1 s=READY pgs=2362 cs=0 l=1 rev1=1 crypto rx=0x7f86600ad040 tx=0x7f866001acc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:07.379 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.372+0000 7f8666ffd640 1 -- 172.21.15.16:0/2227449067 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86600b48c0 con 0x7f86700b3ee0 2023-05-22T16:05:07.379 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.372+0000 7f8666ffd640 1 -- 172.21.15.16:0/2227449067 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86600b4a60 con 0x7f86700b3ee0 2023-05-22T16:05:07.379 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.372+0000 7f8678358640 1 -- 172.21.15.16:0/2227449067 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86700cd1a0 con 0x7f86700b3ee0 2023-05-22T16:05:07.380 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.372+0000 7f8666ffd640 1 -- 172.21.15.16:0/2227449067 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86600b4e10 con 0x7f86700b3ee0 2023-05-22T16:05:07.380 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.372+0000 7f8678358640 1 -- 172.21.15.16:0/2227449067 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86700b4c70 con 0x7f86700b3ee0 2023-05-22T16:05:07.380 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.376+0000 7f8678358640 1 -- 172.21.15.16:0/2227449067 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8670168af0 con 0x7f86700b3ee0 2023-05-22T16:05:07.380 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.376+0000 7f8666ffd640 1 -- 172.21.15.16:0/2227449067 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 90) v1 ==== 83941+0+0 (secure 0 0 0) 0x7f86600c7440 con 0x7f86700b3ee0 2023-05-22T16:05:07.381 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.376+0000 7f8666ffd640 1 -- 172.21.15.16:0/2227449067 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(183..183 src has 1..183) v4 ==== 7154+0+0 (secure 0 0 0) 0x7f866011d860 con 0x7f86700b3ee0 2023-05-22T16:05:07.382 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.380+0000 7f8666ffd640 1 -- 172.21.15.16:0/2227449067 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f86600eb480 con 0x7f86700b3ee0 2023-05-22T16:05:07.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.532+0000 7f8678358640 1 -- 172.21.15.16:0/2227449067 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f867004dbe0 con 0x7f86700b3ee0 2023-05-22T16:05:07.534 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.532+0000 7f8666ffd640 1 -- 172.21.15.16:0/2227449067 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v183) v1 ==== 100+0+11 (secure 0 0 0) 0x7f86600eeaa0 con 0x7f86700b3ee0 2023-05-22T16:05:07.534 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 64 2023-05-22T16:05:07.536 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.532+0000 7f8678358640 1 -- 172.21.15.16:0/2227449067 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f86700b3ee0 msgr2=0x7f86700b24b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:07.536 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.532+0000 7f8678358640 1 --2- 172.21.15.16:0/2227449067 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f86700b3ee0 0x7f86700b24b0 secure :-1 s=READY pgs=2362 cs=0 l=1 rev1=1 crypto rx=0x7f86600ad040 tx=0x7f866001acc0 comp rx=0 tx=0).stop 2023-05-22T16:05:07.537 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.532+0000 7f8678358640 1 -- 172.21.15.16:0/2227449067 shutdown_connections 2023-05-22T16:05:07.537 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.532+0000 7f8678358640 1 --2- 172.21.15.16:0/2227449067 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f867015bca0 0x7f86700b3150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:07.537 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.532+0000 7f8678358640 1 --2- 172.21.15.16:0/2227449067 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f86700b4870 0x7f86700b2c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:07.538 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.532+0000 7f8678358640 1 --2- 172.21.15.16:0/2227449067 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f86700b3ee0 0x7f86700b24b0 unknown :-1 s=CLOSED pgs=2362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:07.538 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.532+0000 7f8678358640 1 -- 172.21.15.16:0/2227449067 >> 172.21.15.16:0/2227449067 conn(0x7f86700657f0 msgr2=0x7f86700d23c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:05:07.538 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.532+0000 7f8678358640 1 -- 172.21.15.16:0/2227449067 shutdown_connections 2023-05-22T16:05:07.538 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.532+0000 7f8678358640 1 -- 172.21.15.16:0/2227449067 wait complete. 2023-05-22T16:05:07.553 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-05-22T16:05:07.669 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.668+0000 7f3215dcb640 1 Processor -- start 2023-05-22T16:05:07.669 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.668+0000 7f3215dcb640 1 -- start start 2023-05-22T16:05:07.670 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.668+0000 7f3215dcb640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f321015fc50 0x7f3210160020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:07.670 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.668+0000 7f3215dcb640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f321015c490 con 0x7f321015c920 2023-05-22T16:05:07.671 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.668+0000 7f3215dcb640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f321015c140 con 0x7f32100d0740 2023-05-22T16:05:07.671 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.668+0000 7f3215dcb640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32100ccb50 con 0x7f321015fc50 2023-05-22T16:05:07.671 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.668+0000 7f320effd640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f321015c920 0x7f32100d0030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:45488/0 (socket says 172.21.15.16:45488) 2023-05-22T16:05:07.671 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.668+0000 7f320effd640 1 -- 172.21.15.16:0/3435518121 learned_addr learned my addr 172.21.15.16:0/3435518121 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T16:05:07.672 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.668+0000 7f320f7fe640 1 --2- 172.21.15.16:0/3435518121 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f321015fc50 0x7f3210160020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:07.673 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.668+0000 7f320e7fc640 1 -- 172.21.15.16:0/3435518121 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3467338099 0 0) 0x7f321015c490 con 0x7f321015c920 2023-05-22T16:05:07.673 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.668+0000 7f320e7fc640 1 -- 172.21.15.16:0/3435518121 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f31f8003530 con 0x7f321015c920 2023-05-22T16:05:07.673 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.668+0000 7f320e7fc640 1 -- 172.21.15.16:0/3435518121 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1986366044 0 0) 0x7f31f8003530 con 0x7f321015c920 2023-05-22T16:05:07.673 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.668+0000 7f320e7fc640 1 -- 172.21.15.16:0/3435518121 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f321015fc50 msgr2=0x7f3210160020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:05:07.673 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.668+0000 7f320e7fc640 1 --2- 172.21.15.16:0/3435518121 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f321015fc50 0x7f3210160020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:07.674 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.668+0000 7f320e7fc640 1 -- 172.21.15.16:0/3435518121 >> v1:172.21.15.62:6789/0 conn(0x7f32100d0740 legacy=0x7f32100ce790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:07.674 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.668+0000 7f320e7fc640 1 -- 172.21.15.16:0/3435518121 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32100cce30 con 0x7f321015c920 2023-05-22T16:05:07.674 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.668+0000 7f320e7fc640 1 -- 172.21.15.16:0/3435518121 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7f3204001680 con 0x7f321015c920 2023-05-22T16:05:07.674 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.668+0000 7f320e7fc640 1 -- 172.21.15.16:0/3435518121 >> v1:172.21.15.16:6789/0 conn(0x7f321015c920 legacy=0x7f32100d0030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:07.674 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.668+0000 7f320e7fc640 1 --2- 172.21.15.16:0/3435518121 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f321015fc50 0x7f31f80039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:07.674 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.668+0000 7f320e7fc640 1 --2- 172.21.15.16:0/3435518121 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f31f8003f30 0x7f31f80047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:07.674 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.668+0000 7f320e7fc640 1 --2- 172.21.15.16:0/3435518121 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f31f8004d10 0x7f31f8005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:07.675 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.668+0000 7f320e7fc640 1 -- 172.21.15.16:0/3435518121 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32100ccb50 con 0x7f31f8004d10 2023-05-22T16:05:07.675 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.668+0000 7f320e7fc640 1 -- 172.21.15.16:0/3435518121 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31f800ba70 con 0x7f31f8003f30 2023-05-22T16:05:07.675 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.668+0000 7f320e7fc640 1 -- 172.21.15.16:0/3435518121 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31f800bc30 con 0x7f321015fc50 2023-05-22T16:05:07.675 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.668+0000 7f320ffff640 1 --2- 172.21.15.16:0/3435518121 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f31f8004d10 0x7f31f8005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:07.675 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.668+0000 7f320effd640 1 --2- 172.21.15.16:0/3435518121 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f31f8003f30 0x7f31f80047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:07.675 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.668+0000 7f320f7fe640 1 --2- 172.21.15.16:0/3435518121 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f321015fc50 0x7f31f80039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:07.675 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.668+0000 7f320ffff640 1 -- 172.21.15.16:0/3435518121 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f321015fc50 msgr2=0x7f31f80039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:05:07.675 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.668+0000 7f320ffff640 1 --2- 172.21.15.16:0/3435518121 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f321015fc50 0x7f31f80039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:07.676 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.668+0000 7f320ffff640 1 -- 172.21.15.16:0/3435518121 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f31f8003f30 msgr2=0x7f31f80047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:05:07.676 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.668+0000 7f320ffff640 1 --2- 172.21.15.16:0/3435518121 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f31f8003f30 0x7f31f80047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:07.676 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.668+0000 7f320ffff640 1 -- 172.21.15.16:0/3435518121 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31f8003530 con 0x7f31f8004d10 2023-05-22T16:05:07.676 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.668+0000 7f320effd640 1 --2- 172.21.15.16:0/3435518121 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f31f8003f30 0x7f31f80047d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T16:05:07.679 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.672+0000 7f320ffff640 1 --2- 172.21.15.16:0/3435518121 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f31f8004d10 0x7f31f8005180 secure :-1 s=READY pgs=2364 cs=0 l=1 rev1=1 crypto rx=0x7f32100d0ba0 tx=0x7f320801c700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad197a841311b4e7 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:07.679 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.672+0000 7f320e7fc640 1 -- 172.21.15.16:0/3435518121 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3208003b80 con 0x7f31f8004d10 2023-05-22T16:05:07.679 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.672+0000 7f320e7fc640 1 -- 172.21.15.16:0/3435518121 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3208003ce0 con 0x7f31f8004d10 2023-05-22T16:05:07.679 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.672+0000 7f320e7fc640 1 -- 172.21.15.16:0/3435518121 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3208005310 con 0x7f31f8004d10 2023-05-22T16:05:07.679 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.672+0000 7f3215dcb640 1 -- 172.21.15.16:0/3435518121 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f31f8004d10 msgr2=0x7f31f8005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:07.679 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.672+0000 7f3215dcb640 1 --2- 172.21.15.16:0/3435518121 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f31f8004d10 0x7f31f8005180 secure :-1 s=READY pgs=2364 cs=0 l=1 rev1=1 crypto rx=0x7f32100d0ba0 tx=0x7f320801c700 comp rx=0 tx=0).stop 2023-05-22T16:05:07.680 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.672+0000 7f320ffff640 1 -- 172.21.15.16:0/3435518121 reap_dead start 2023-05-22T16:05:07.680 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.672+0000 7f3215dcb640 1 -- 172.21.15.16:0/3435518121 shutdown_connections 2023-05-22T16:05:07.680 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.672+0000 7f3215dcb640 1 -- 172.21.15.16:0/3435518121 >> 172.21.15.16:0/3435518121 conn(0x7f32100657f0 msgr2=0x7f3210070e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:05:07.680 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.672+0000 7f3215dcb640 1 -- 172.21.15.16:0/3435518121 shutdown_connections 2023-05-22T16:05:07.680 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.672+0000 7f3215dcb640 1 -- 172.21.15.16:0/3435518121 wait complete. 2023-05-22T16:05:07.680 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.672+0000 7f3215dcb640 1 Processor -- start 2023-05-22T16:05:07.681 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.672+0000 7f3215dcb640 1 -- start start 2023-05-22T16:05:07.681 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.676+0000 7f3215dcb640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3210151ea0 0x7f32100b1ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:07.681 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.676+0000 7f3215dcb640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f32100b38d0 0x7f32100b4390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:07.681 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.676+0000 7f3215dcb640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3210152b20 0x7f32100b48d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:07.681 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.676+0000 7f3215dcb640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32100b3440 con 0x7f3210152b20 2023-05-22T16:05:07.681 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.676+0000 7f3215dcb640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32100b2830 con 0x7f32100b38d0 2023-05-22T16:05:07.681 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.676+0000 7f3215dcb640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32100b2b10 con 0x7f3210151ea0 2023-05-22T16:05:07.681 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.676+0000 7f320ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3210152b20 0x7f32100b48d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:07.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.676+0000 7f320ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3210152b20 0x7f32100b48d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46366/0 (socket says 172.21.15.16:46366) 2023-05-22T16:05:07.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.676+0000 7f320ffff640 1 -- 172.21.15.16:0/3131598498 learned_addr learned my addr 172.21.15.16:0/3131598498 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:05:07.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.676+0000 7f320f7fe640 1 --2- 172.21.15.16:0/3131598498 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3210151ea0 0x7f32100b1ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:07.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.676+0000 7f320effd640 1 --2- 172.21.15.16:0/3131598498 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f32100b38d0 0x7f32100b4390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:07.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.676+0000 7f320ffff640 1 -- 172.21.15.16:0/3131598498 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3210151ea0 msgr2=0x7f32100b1ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:07.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.676+0000 7f320ffff640 1 --2- 172.21.15.16:0/3131598498 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3210151ea0 0x7f32100b1ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:07.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.676+0000 7f320ffff640 1 -- 172.21.15.16:0/3131598498 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f32100b38d0 msgr2=0x7f32100b4390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:07.683 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.676+0000 7f320ffff640 1 --2- 172.21.15.16:0/3131598498 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f32100b38d0 0x7f32100b4390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:07.683 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.676+0000 7f320ffff640 1 -- 172.21.15.16:0/3131598498 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f321015cd70 con 0x7f3210152b20 2023-05-22T16:05:07.683 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.676+0000 7f320ffff640 1 --2- 172.21.15.16:0/3131598498 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3210152b20 0x7f32100b48d0 secure :-1 s=READY pgs=2365 cs=0 l=1 rev1=1 crypto rx=0x7f320801c740 tx=0x7f31f8005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:07.683 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.676+0000 7f31f7fff640 1 -- 172.21.15.16:0/3131598498 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3208003b80 con 0x7f3210152b20 2023-05-22T16:05:07.683 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.676+0000 7f31f7fff640 1 -- 172.21.15.16:0/3131598498 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3208007dc0 con 0x7f3210152b20 2023-05-22T16:05:07.684 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.676+0000 7f3215dcb640 1 -- 172.21.15.16:0/3131598498 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32100cd110 con 0x7f3210152b20 2023-05-22T16:05:07.684 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.676+0000 7f31f7fff640 1 -- 172.21.15.16:0/3131598498 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3208006dc0 con 0x7f3210152b20 2023-05-22T16:05:07.684 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.676+0000 7f3215dcb640 1 -- 172.21.15.16:0/3131598498 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f321015cf00 con 0x7f3210152b20 2023-05-22T16:05:07.684 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.676+0000 7f3215dcb640 1 -- 172.21.15.16:0/3131598498 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32100b35b0 con 0x7f3210152b20 2023-05-22T16:05:07.685 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.676+0000 7f31f7fff640 1 -- 172.21.15.16:0/3131598498 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 90) v1 ==== 83941+0+0 (secure 0 0 0) 0x7f3208007030 con 0x7f3210152b20 2023-05-22T16:05:07.685 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.676+0000 7f31f7fff640 1 -- 172.21.15.16:0/3131598498 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(183..183 src has 1..183) v4 ==== 7154+0+0 (secure 0 0 0) 0x7f320811c4c0 con 0x7f3210152b20 2023-05-22T16:05:07.685 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.680+0000 7f31f7fff640 1 -- 172.21.15.16:0/3131598498 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f32080ea050 con 0x7f3210152b20 2023-05-22T16:05:07.810 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.808+0000 7f31f7fff640 1 -- 172.21.15.16:0/3131598498 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mgrmap(e 91) v1 ==== 84068+0+0 (secure 0 0 0) 0x7f32080e9b00 con 0x7f3210152b20 2023-05-22T16:05:07.811 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.808+0000 7f31f7fff640 1 --2- 172.21.15.16:0/3131598498 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f31fc06b860 0x7f31fc06dc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:07.811 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.808+0000 7f320f7fe640 1 --2- 172.21.15.16:0/3131598498 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f31fc06b860 0x7f31fc06dc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:07.815 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.812+0000 7f320f7fe640 1 --2- 172.21.15.16:0/3131598498 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f31fc06b860 0x7f31fc06dc40 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f3200009a30 tx=0x7f3200008040 comp rx=0 tx=0).ready entity=mgr.10542 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:07.878 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.876+0000 7f3215dcb640 1 -- 172.21.15.16:0/3131598498 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f321004dbe0 con 0x7f3210152b20 2023-05-22T16:05:07.879 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.876+0000 7f31f7fff640 1 -- 172.21.15.16:0/3131598498 <== mon.0 v2:172.21.15.16:3300/0 8 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 73 v73) v1 ==== 94+0+4657 (secure 0 0 0) 0x7f3208119450 con 0x7f3210152b20 2023-05-22T16:05:07.880 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T16:05:07.880 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":73,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7682,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/2815032143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":2815032143},{"type":"v1","addr":"172.21.15.62:6835","nonce":2815032143}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":69},{"gid":10239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6836/1961058261","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1961058261},{"type":"v1","addr":"172.21.15.16:6836","nonce":1961058261}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":69},{"gid":10242,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/1453687184","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6835","nonce":1453687184},{"type":"v1","addr":"172.21.15.16:6837","nonce":1453687184}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":73,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-05-22T16:03:38.779645+0000","modified":"2023-05-22T16:03:42.129478+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10251":{"gid":10251,"name":"d","rank":0,"incarnation":70,"state":"up:active","state_seq":3,"addr":"172.21.15.62:6833/2486335217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":2486335217},{"type":"v1","addr":"172.21.15.62:6833","nonce":2486335217}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[23],"metadata_pool":22,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2023-05-22T16:05:07.880 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 73 2023-05-22T16:05:07.881 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.880+0000 7f3215dcb640 1 -- 172.21.15.16:0/3131598498 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f31fc06b860 msgr2=0x7f31fc06dc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:07.882 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.880+0000 7f3215dcb640 1 --2- 172.21.15.16:0/3131598498 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f31fc06b860 0x7f31fc06dc40 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f3200009a30 tx=0x7f3200008040 comp rx=0 tx=0).stop 2023-05-22T16:05:07.882 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.880+0000 7f3215dcb640 1 -- 172.21.15.16:0/3131598498 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3210152b20 msgr2=0x7f32100b48d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:07.882 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.880+0000 7f3215dcb640 1 --2- 172.21.15.16:0/3131598498 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3210152b20 0x7f32100b48d0 secure :-1 s=READY pgs=2365 cs=0 l=1 rev1=1 crypto rx=0x7f320801c740 tx=0x7f31f8005800 comp rx=0 tx=0).stop 2023-05-22T16:05:07.882 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.880+0000 7f3215dcb640 1 -- 172.21.15.16:0/3131598498 shutdown_connections 2023-05-22T16:05:07.882 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.880+0000 7f3215dcb640 1 --2- 172.21.15.16:0/3131598498 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f31fc06b860 0x7f31fc06dc40 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:07.883 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.880+0000 7f3215dcb640 1 --2- 172.21.15.16:0/3131598498 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3210152b20 0x7f32100b48d0 unknown :-1 s=CLOSED pgs=2365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:07.883 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.880+0000 7f3215dcb640 1 --2- 172.21.15.16:0/3131598498 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f32100b38d0 0x7f32100b4390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:07.883 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.880+0000 7f3215dcb640 1 --2- 172.21.15.16:0/3131598498 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3210151ea0 0x7f32100b1ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:07.883 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.880+0000 7f3215dcb640 1 -- 172.21.15.16:0/3131598498 >> 172.21.15.16:0/3131598498 conn(0x7f32100657f0 msgr2=0x7f32100d0b10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:05:07.883 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.880+0000 7f3215dcb640 1 -- 172.21.15.16:0/3131598498 shutdown_connections 2023-05-22T16:05:07.883 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:07.880+0000 7f3215dcb640 1 -- 172.21.15.16:0/3131598498 wait complete. 2023-05-22T16:05:07.900 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-05-22T16:05:08.017 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.016+0000 7fe3fdda8640 1 Processor -- start 2023-05-22T16:05:08.018 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.016+0000 7fe3fdda8640 1 -- start start 2023-05-22T16:05:08.019 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.016+0000 7fe3fdda8640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe3f80c9b40 0x7fe3f815e460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:08.020 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.016+0000 7fe3fdda8640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fe3f8070df0 0x7fe3f8071210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:08.020 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.016+0000 7fe3fdda8640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3f80b0650 con 0x7fe3f80ca4e0 2023-05-22T16:05:08.020 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.016+0000 7fe3fdda8640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3f80b0370 con 0x7fe3f80c9b40 2023-05-22T16:05:08.021 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.016+0000 7fe3fdda8640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3f815d980 con 0x7fe3f8070df0 2023-05-22T16:05:08.021 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.016+0000 7fe3f6ffd640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe3f80c9b40 0x7fe3f815e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:08.021 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.016+0000 7fe3f77fe640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fe3f80ca4e0 0x7fe3f80afb40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:45504/0 (socket says 172.21.15.16:45504) 2023-05-22T16:05:08.021 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.016+0000 7fe3f77fe640 1 -- 172.21.15.16:0/1173663996 learned_addr learned my addr 172.21.15.16:0/1173663996 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T16:05:08.022 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.016+0000 7fe3f7fff640 1 --2- 172.21.15.16:0/1173663996 >> v2:172.21.15.62:3300/0 conn(0x7fe3f8070df0 0x7fe3f8071210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:08.022 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.016+0000 7fe3f6ffd640 1 -- 172.21.15.16:0/1173663996 >> v2:172.21.15.62:3300/0 conn(0x7fe3f8070df0 msgr2=0x7fe3f8071210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:05:08.022 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.016+0000 7fe3f6ffd640 1 --2- 172.21.15.16:0/1173663996 >> v2:172.21.15.62:3300/0 conn(0x7fe3f8070df0 0x7fe3f8071210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:08.022 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.016+0000 7fe3f6ffd640 1 -- 172.21.15.16:0/1173663996 >> v1:172.21.15.16:6789/0 conn(0x7fe3f80ca4e0 legacy=0x7fe3f80afb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:08.023 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.016+0000 7fe3f6ffd640 1 -- 172.21.15.16:0/1173663996 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3f815dc60 con 0x7fe3f80c9b40 2023-05-22T16:05:08.029 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.020+0000 7fe3f6ffd640 1 --2- 172.21.15.16:0/1173663996 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe3f80c9b40 0x7fe3f815e460 secure :-1 s=READY pgs=2367 cs=0 l=1 rev1=1 crypto rx=0x7fe3f80cad10 tx=0x7fe3e801c430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd6c69d85b83c549 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:08.029 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.020+0000 7fe3f67fc640 1 -- 172.21.15.16:0/1173663996 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3e80a8050 con 0x7fe3f80c9b40 2023-05-22T16:05:08.030 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.020+0000 7fe3f67fc640 1 -- 172.21.15.16:0/1173663996 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3e80ac5f0 con 0x7fe3f80c9b40 2023-05-22T16:05:08.030 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.020+0000 7fe3f67fc640 1 -- 172.21.15.16:0/1173663996 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3e80ac980 con 0x7fe3f80c9b40 2023-05-22T16:05:08.030 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.020+0000 7fe3fdda8640 1 -- 172.21.15.16:0/1173663996 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe3f80c9b40 msgr2=0x7fe3f815e460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:08.031 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.020+0000 7fe3fdda8640 1 --2- 172.21.15.16:0/1173663996 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe3f80c9b40 0x7fe3f815e460 secure :-1 s=READY pgs=2367 cs=0 l=1 rev1=1 crypto rx=0x7fe3f80cad10 tx=0x7fe3e801c430 comp rx=0 tx=0).stop 2023-05-22T16:05:08.031 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.020+0000 7fe3fdda8640 1 -- 172.21.15.16:0/1173663996 shutdown_connections 2023-05-22T16:05:08.031 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.020+0000 7fe3fdda8640 1 --2- 172.21.15.16:0/1173663996 >> v2:172.21.15.62:3300/0 conn(0x7fe3f8070df0 0x7fe3f8071210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:08.031 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.020+0000 7fe3fdda8640 1 --2- 172.21.15.16:0/1173663996 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe3f80c9b40 0x7fe3f815e460 unknown :-1 s=CLOSED pgs=2367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:08.031 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.020+0000 7fe3fdda8640 1 -- 172.21.15.16:0/1173663996 >> 172.21.15.16:0/1173663996 conn(0x7fe3f8069720 msgr2=0x7fe3f8069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:05:08.032 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.020+0000 7fe3fdda8640 1 -- 172.21.15.16:0/1173663996 shutdown_connections 2023-05-22T16:05:08.032 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.020+0000 7fe3fdda8640 1 -- 172.21.15.16:0/1173663996 wait complete. 2023-05-22T16:05:08.032 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.020+0000 7fe3fdda8640 1 Processor -- start 2023-05-22T16:05:08.032 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.020+0000 7fe3fdda8640 1 -- start start 2023-05-22T16:05:08.032 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.024+0000 7fe3fdda8640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe3f8070df0 0x7fe3f80ace30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:08.032 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.024+0000 7fe3fdda8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe3f80c9b40 0x7fe3f814bbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:08.033 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.024+0000 7fe3fdda8640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe3f80ca4e0 0x7fe3f80cbbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:08.033 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.024+0000 7fe3fdda8640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3f80ac9a0 con 0x7fe3f80c9b40 2023-05-22T16:05:08.033 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.024+0000 7fe3fdda8640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3f80ad370 con 0x7fe3f80ca4e0 2023-05-22T16:05:08.033 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.024+0000 7fe3fdda8640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3f80cc770 con 0x7fe3f8070df0 2023-05-22T16:05:08.033 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.024+0000 7fe3f6ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe3f80c9b40 0x7fe3f814bbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:08.033 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.024+0000 7fe3f6ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe3f80c9b40 0x7fe3f814bbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46382/0 (socket says 172.21.15.16:46382) 2023-05-22T16:05:08.033 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.024+0000 7fe3f6ffd640 1 -- 172.21.15.16:0/2722613438 learned_addr learned my addr 172.21.15.16:0/2722613438 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:05:08.034 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.024+0000 7fe3f7fff640 1 --2- 172.21.15.16:0/2722613438 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe3f80ca4e0 0x7fe3f80cbbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:08.034 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.024+0000 7fe3f77fe640 1 --2- 172.21.15.16:0/2722613438 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe3f8070df0 0x7fe3f80ace30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:08.034 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.024+0000 7fe3f6ffd640 1 -- 172.21.15.16:0/2722613438 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe3f8070df0 msgr2=0x7fe3f80ace30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:08.034 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.024+0000 7fe3f6ffd640 1 --2- 172.21.15.16:0/2722613438 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe3f8070df0 0x7fe3f80ace30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:08.034 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.024+0000 7fe3f6ffd640 1 -- 172.21.15.16:0/2722613438 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe3f80ca4e0 msgr2=0x7fe3f80cbbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:08.034 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.024+0000 7fe3f6ffd640 1 --2- 172.21.15.16:0/2722613438 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe3f80ca4e0 0x7fe3f80cbbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:08.034 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.024+0000 7fe3f6ffd640 1 -- 172.21.15.16:0/2722613438 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3e8009420 con 0x7fe3f80c9b40 2023-05-22T16:05:08.035 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.024+0000 7fe3f6ffd640 1 --2- 172.21.15.16:0/2722613438 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe3f80c9b40 0x7fe3f814bbb0 secure :-1 s=READY pgs=2368 cs=0 l=1 rev1=1 crypto rx=0x7fe3e801aae0 tx=0x7fe3e80191d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:08.035 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.024+0000 7fe3effff640 1 -- 172.21.15.16:0/2722613438 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3e80a8040 con 0x7fe3f80c9b40 2023-05-22T16:05:08.035 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.024+0000 7fe3effff640 1 -- 172.21.15.16:0/2722613438 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3e80b45c0 con 0x7fe3f80c9b40 2023-05-22T16:05:08.036 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.024+0000 7fe3fdda8640 1 -- 172.21.15.16:0/2722613438 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3f80c8e80 con 0x7fe3f80c9b40 2023-05-22T16:05:08.036 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.024+0000 7fe3effff640 1 -- 172.21.15.16:0/2722613438 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3e80b4950 con 0x7fe3f80c9b40 2023-05-22T16:05:08.036 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.024+0000 7fe3fdda8640 1 -- 172.21.15.16:0/2722613438 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3f80ca8b0 con 0x7fe3f80c9b40 2023-05-22T16:05:08.037 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.024+0000 7fe3fdda8640 1 -- 172.21.15.16:0/2722613438 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3f80c9010 con 0x7fe3f80c9b40 2023-05-22T16:05:08.037 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.024+0000 7fe3effff640 1 -- 172.21.15.16:0/2722613438 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 91) v1 ==== 84068+0+0 (secure 0 0 0) 0x7fe3e80b4ab0 con 0x7fe3f80c9b40 2023-05-22T16:05:08.037 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.024+0000 7fe3effff640 1 --2- 172.21.15.16:0/2722613438 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fe3d8065200 0x7fe3d80676b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:08.038 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.024+0000 7fe3effff640 1 -- 172.21.15.16:0/2722613438 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(183..183 src has 1..183) v4 ==== 7154+0+0 (secure 0 0 0) 0x7fe3e811ccd0 con 0x7fe3f80c9b40 2023-05-22T16:05:08.038 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.024+0000 7fe3f77fe640 1 --2- 172.21.15.16:0/2722613438 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fe3d8065200 0x7fe3d80676b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:08.038 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.024+0000 7fe3f77fe640 1 --2- 172.21.15.16:0/2722613438 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fe3d8065200 0x7fe3d80676b0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fe3e0005890 tx=0x7fe3e0009040 comp rx=0 tx=0).ready entity=mgr.10542 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:08.039 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.028+0000 7fe3effff640 1 -- 172.21.15.16:0/2722613438 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fe3e80ea840 con 0x7fe3f80c9b40 2023-05-22T16:05:08.183 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.180+0000 7fe3fdda8640 1 -- 172.21.15.16:0/2722613438 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe3f804dbe0 con 0x7fe3f80c9b40 2023-05-22T16:05:08.186 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.184+0000 7fe3effff640 1 -- 172.21.15.16:0/2722613438 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v183) v1 ==== 74+0+19182 (secure 0 0 0) 0x7fe3e80b1090 con 0x7fe3f80c9b40 2023-05-22T16:05:08.186 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T16:05:08.187 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":183,"fsid":"b2719834-7847-4d98-90a6-f65d57f85c49","created":"2023-05-22T15:26:25.259314+0000","modified":"2023-05-22T16:05:06.787958+0000","last_up_change":"2023-05-22T15:26:36.448524+0000","last_in_change":"2023-05-22T15:26:27.675668+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":23,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-05-22T15:26:38.019781+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-05-22T15:26:43.570514+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":22,"pool_name":"cephfs_metadata","create_time":"2023-05-22T16:03:37.371980+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"178","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_data","create_time":"2023-05-22T16:03:37.756887+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"182","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":5,"snap_epoch":182,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9e7ff854-b25f-4cbb-a5c2-1799d9c53d2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":37021},{"type":"v1","addr":"172.21.15.16:6825","nonce":37021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":37021},{"type":"v1","addr":"172.21.15.16:6827","nonce":37021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":37021},{"type":"v1","addr":"172.21.15.16:6831","nonce":37021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":37021},{"type":"v1","addr":"172.21.15.16:6829","nonce":37021}]},"public_addr":"172.21.15.16:6825/37021","cluster_addr":"172.21.15.16:6827/37021","heartbeat_back_addr":"172.21.15.16:6831/37021","heartbeat_front_addr":"172.21.15.16:6829/37021","state":["exists","up"]},{"osd":1,"uuid":"4702fbc9-a927-4565-aca5-41dc69cc8264","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":37008},{"type":"v1","addr":"172.21.15.16:6801","nonce":37008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":37008},{"type":"v1","addr":"172.21.15.16:6803","nonce":37008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":37008},{"type":"v1","addr":"172.21.15.16:6807","nonce":37008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":37008},{"type":"v1","addr":"172.21.15.16:6805","nonce":37008}]},"public_addr":"172.21.15.16:6801/37008","cluster_addr":"172.21.15.16:6803/37008","heartbeat_back_addr":"172.21.15.16:6807/37008","heartbeat_front_addr":"172.21.15.16:6805/37008","state":["exists","up"]},{"osd":2,"uuid":"1e7bba46-4eba-47ef-bb81-5e2d371638ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":37009},{"type":"v1","addr":"172.21.15.16:6809","nonce":37009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":37009},{"type":"v1","addr":"172.21.15.16:6811","nonce":37009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":37009},{"type":"v1","addr":"172.21.15.16:6815","nonce":37009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":37009},{"type":"v1","addr":"172.21.15.16:6813","nonce":37009}]},"public_addr":"172.21.15.16:6809/37009","cluster_addr":"172.21.15.16:6811/37009","heartbeat_back_addr":"172.21.15.16:6815/37009","heartbeat_front_addr":"172.21.15.16:6813/37009","state":["exists","up"]},{"osd":3,"uuid":"383cacc0-5a88-4904-990e-498ca59ef48a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":37020},{"type":"v1","addr":"172.21.15.16:6817","nonce":37020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":37020},{"type":"v1","addr":"172.21.15.16:6819","nonce":37020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":37020},{"type":"v1","addr":"172.21.15.16:6823","nonce":37020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":37020},{"type":"v1","addr":"172.21.15.16:6821","nonce":37020}]},"public_addr":"172.21.15.16:6817/37020","cluster_addr":"172.21.15.16:6819/37020","heartbeat_back_addr":"172.21.15.16:6823/37020","heartbeat_front_addr":"172.21.15.16:6821/37020","state":["exists","up"]},{"osd":4,"uuid":"970500ef-95b7-42c1-b080-eed4fae7013d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":37353},{"type":"v1","addr":"172.21.15.62:6817","nonce":37353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6819","nonce":37353},{"type":"v1","addr":"172.21.15.62:6822","nonce":37353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37353},{"type":"v1","addr":"172.21.15.62:6831","nonce":37353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37353},{"type":"v1","addr":"172.21.15.62:6826","nonce":37353}]},"public_addr":"172.21.15.62:6817/37353","cluster_addr":"172.21.15.62:6822/37353","heartbeat_back_addr":"172.21.15.62:6831/37353","heartbeat_front_addr":"172.21.15.62:6826/37353","state":["exists","up"]},{"osd":5,"uuid":"022d67db-d4d8-445b-b1b9-8b446a00f5c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":37352},{"type":"v1","addr":"172.21.15.62:6801","nonce":37352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":37352},{"type":"v1","addr":"172.21.15.62:6803","nonce":37352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":37352},{"type":"v1","addr":"172.21.15.62:6807","nonce":37352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":37352},{"type":"v1","addr":"172.21.15.62:6805","nonce":37352}]},"public_addr":"172.21.15.62:6801/37352","cluster_addr":"172.21.15.62:6803/37352","heartbeat_back_addr":"172.21.15.62:6807/37352","heartbeat_front_addr":"172.21.15.62:6805/37352","state":["exists","up"]},{"osd":6,"uuid":"42bd372b-6688-4a1b-8407-e16e8c266e33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":37354},{"type":"v1","addr":"172.21.15.62:6820","nonce":37354}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":37354},{"type":"v1","addr":"172.21.15.62:6823","nonce":37354}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6829","nonce":37354},{"type":"v1","addr":"172.21.15.62:6830","nonce":37354}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6825","nonce":37354},{"type":"v1","addr":"172.21.15.62:6827","nonce":37354}]},"public_addr":"172.21.15.62:6820/37354","cluster_addr":"172.21.15.62:6823/37354","heartbeat_back_addr":"172.21.15.62:6830/37354","heartbeat_front_addr":"172.21.15.62:6827/37354","state":["exists","up"]},{"osd":7,"uuid":"ac70b2a3-f41b-4bf2-b227-7f2cbb55426d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37355},{"type":"v1","addr":"172.21.15.62:6809","nonce":37355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37355},{"type":"v1","addr":"172.21.15.62:6811","nonce":37355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":37355},{"type":"v1","addr":"172.21.15.62:6815","nonce":37355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":37355},{"type":"v1","addr":"172.21.15.62:6813","nonce":37355}]},"public_addr":"172.21.15.62:6809/37355","cluster_addr":"172.21.15.62:6811/37355","heartbeat_back_addr":"172.21.15.62:6815/37355","heartbeat_front_addr":"172.21.15.62:6813/37355","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.368896+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.107273+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.276888+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.287502+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.124791+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.251937+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.255679+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.232797+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:0/1900219297":"2023-05-23T16:05:06.787906+0000","172.21.15.16:0/4377472":"2023-05-23T16:05:06.787906+0000","172.21.15.16:0/1923190612":"2023-05-23T16:05:06.787906+0000","172.21.15.16:0/2696230409":"2023-05-23T16:05:06.787906+0000","172.21.15.16:0/2495910544":"2023-05-23T16:03:52.657800+0000","172.21.15.16:0/1761665129":"2023-05-23T16:03:52.657800+0000","172.21.15.16:6833/65890":"2023-05-23T16:03:52.657800+0000","172.21.15.16:0/2966441324":"2023-05-23T16:05:06.787906+0000","172.21.15.16:6832/65890":"2023-05-23T16:03:52.657800+0000","172.21.15.16:0/586164935":"2023-05-23T16:05:06.787906+0000","172.21.15.16:0/216273587":"2023-05-23T16:05:06.787906+0000","172.21.15.16:0/3698789191":"2023-05-23T16:05:06.787906+0000","172.21.15.16:0/4138270792":"2023-05-23T16:03:52.657800+0000","172.21.15.16:0/2442191907":"2023-05-23T16:03:52.657800+0000","172.21.15.16:0/3487890192":"2023-05-23T16:03:52.657800+0000","172.21.15.16:0/3247164174":"2023-05-23T16:03:52.657800+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":17,"snaps":[{"begin":2,"length":2}]},{"pool":21,"snaps":[{"begin":25,"length":9}]},{"pool":23,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-05-22T16:05:08.188 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.184+0000 7fe3fdda8640 1 -- 172.21.15.16:0/2722613438 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fe3d8065200 msgr2=0x7fe3d80676b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:08.188 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.184+0000 7fe3fdda8640 1 --2- 172.21.15.16:0/2722613438 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fe3d8065200 0x7fe3d80676b0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fe3e0005890 tx=0x7fe3e0009040 comp rx=0 tx=0).stop 2023-05-22T16:05:08.188 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.184+0000 7fe3fdda8640 1 -- 172.21.15.16:0/2722613438 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe3f80c9b40 msgr2=0x7fe3f814bbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:08.188 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.184+0000 7fe3fdda8640 1 --2- 172.21.15.16:0/2722613438 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe3f80c9b40 0x7fe3f814bbb0 secure :-1 s=READY pgs=2368 cs=0 l=1 rev1=1 crypto rx=0x7fe3e801aae0 tx=0x7fe3e80191d0 comp rx=0 tx=0).stop 2023-05-22T16:05:08.189 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.184+0000 7fe3fdda8640 1 -- 172.21.15.16:0/2722613438 shutdown_connections 2023-05-22T16:05:08.189 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.184+0000 7fe3fdda8640 1 --2- 172.21.15.16:0/2722613438 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe3f80ca4e0 0x7fe3f80cbbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:08.189 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.184+0000 7fe3fdda8640 1 --2- 172.21.15.16:0/2722613438 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fe3d8065200 0x7fe3d80676b0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:08.189 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.184+0000 7fe3fdda8640 1 --2- 172.21.15.16:0/2722613438 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe3f80c9b40 0x7fe3f814bbb0 unknown :-1 s=CLOSED pgs=2368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:08.189 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.184+0000 7fe3fdda8640 1 --2- 172.21.15.16:0/2722613438 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe3f8070df0 0x7fe3f80ace30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:08.189 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.184+0000 7fe3fdda8640 1 -- 172.21.15.16:0/2722613438 >> 172.21.15.16:0/2722613438 conn(0x7fe3f8069720 msgr2=0x7fe3f80b29c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:05:08.190 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.184+0000 7fe3fdda8640 1 -- 172.21.15.16:0/2722613438 shutdown_connections 2023-05-22T16:05:08.190 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.184+0000 7fe3fdda8640 1 -- 172.21.15.16:0/2722613438 wait complete. 2023-05-22T16:05:08.205 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-05-22T16:05:08.205 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-05-22T16:05:08.328 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.324+0000 7eff76eb9640 1 Processor -- start 2023-05-22T16:05:08.329 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.324+0000 7eff76eb9640 1 -- start start 2023-05-22T16:05:08.331 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.328+0000 7eff76eb9640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7eff70070fb0 0x7eff700cd1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:08.331 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.328+0000 7eff76eb9640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7eff700affa0 0x7eff70160400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:08.331 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.328+0000 7eff76eb9640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff70160940 con 0x7eff700afb40 2023-05-22T16:05:08.331 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.328+0000 7eff76eb9640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff700cd8d0 con 0x7eff70070fb0 2023-05-22T16:05:08.332 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.328+0000 7eff76eb9640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff70164de0 con 0x7eff700affa0 2023-05-22T16:05:08.332 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.328+0000 7eff74c55640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7eff70070fb0 0x7eff700cd1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:08.332 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.328+0000 7eff74c55640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7eff70070fb0 0x7eff700cd1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46396/0 (socket says 172.21.15.16:46396) 2023-05-22T16:05:08.333 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.328+0000 7eff74c55640 1 -- 172.21.15.16:0/768670295 learned_addr learned my addr 172.21.15.16:0/768670295 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:05:08.333 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.328+0000 7eff75456640 1 --2- 172.21.15.16:0/768670295 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7eff700affa0 0x7eff70160400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:08.333 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.328+0000 7eff74c55640 1 -- 172.21.15.16:0/768670295 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7eff700affa0 msgr2=0x7eff70160400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:05:08.333 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.328+0000 7eff74c55640 1 --2- 172.21.15.16:0/768670295 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7eff700affa0 0x7eff70160400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:08.333 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.328+0000 7eff74c55640 1 -- 172.21.15.16:0/768670295 >> v1:172.21.15.62:6789/0 conn(0x7eff700afb40 legacy=0x7eff7015fcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:05:08.333 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.328+0000 7eff74c55640 1 -- 172.21.15.16:0/768670295 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff700ca400 con 0x7eff70070fb0 2023-05-22T16:05:08.333 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.328+0000 7eff75456640 1 --2- 172.21.15.16:0/768670295 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7eff700affa0 0x7eff70160400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T16:05:08.338 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.328+0000 7eff74c55640 1 --2- 172.21.15.16:0/768670295 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7eff70070fb0 0x7eff700cd1f0 secure :-1 s=READY pgs=2369 cs=0 l=1 rev1=1 crypto rx=0x7eff600098a0 tx=0x7eff6001c4e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4bee94e2092803d0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:08.338 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.332+0000 7eff6f7fe640 1 -- 172.21.15.16:0/768670295 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff600ab600 con 0x7eff70070fb0 2023-05-22T16:05:08.339 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.332+0000 7eff6f7fe640 1 -- 172.21.15.16:0/768670295 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff600a7050 con 0x7eff70070fb0 2023-05-22T16:05:08.339 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.332+0000 7eff6f7fe640 1 -- 172.21.15.16:0/768670295 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff600ab980 con 0x7eff70070fb0 2023-05-22T16:05:08.339 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.332+0000 7eff76eb9640 1 -- 172.21.15.16:0/768670295 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7eff70070fb0 msgr2=0x7eff700cd1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:08.340 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.332+0000 7eff76eb9640 1 --2- 172.21.15.16:0/768670295 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7eff70070fb0 0x7eff700cd1f0 secure :-1 s=READY pgs=2369 cs=0 l=1 rev1=1 crypto rx=0x7eff600098a0 tx=0x7eff6001c4e0 comp rx=0 tx=0).stop 2023-05-22T16:05:08.340 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.332+0000 7eff76eb9640 1 -- 172.21.15.16:0/768670295 shutdown_connections 2023-05-22T16:05:08.340 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.332+0000 7eff76eb9640 1 --2- 172.21.15.16:0/768670295 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7eff700affa0 0x7eff70160400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:08.341 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.332+0000 7eff76eb9640 1 --2- 172.21.15.16:0/768670295 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7eff70070fb0 0x7eff700cd1f0 unknown :-1 s=CLOSED pgs=2369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:08.341 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.332+0000 7eff76eb9640 1 -- 172.21.15.16:0/768670295 >> 172.21.15.16:0/768670295 conn(0x7eff70069720 msgr2=0x7eff70069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:05:08.341 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.332+0000 7eff76eb9640 1 -- 172.21.15.16:0/768670295 shutdown_connections 2023-05-22T16:05:08.342 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.332+0000 7eff76eb9640 1 -- 172.21.15.16:0/768670295 wait complete. 2023-05-22T16:05:08.342 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.332+0000 7eff76eb9640 1 Processor -- start 2023-05-22T16:05:08.343 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.332+0000 7eff76eb9640 1 -- start start 2023-05-22T16:05:08.343 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.332+0000 7eff76eb9640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7eff70070fb0 0x7eff700cbbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:08.343 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.332+0000 7eff76eb9640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7eff700afb40 0x7eff700cc110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:08.344 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.332+0000 7eff76eb9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7eff700affa0 0x7eff7015b980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:08.344 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.332+0000 7eff76eb9640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff7015f480 con 0x7eff700affa0 2023-05-22T16:05:08.344 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.332+0000 7eff76eb9640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff7015ef90 con 0x7eff700afb40 2023-05-22T16:05:08.344 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.332+0000 7eff76eb9640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff7015f1a0 con 0x7eff70070fb0 2023-05-22T16:05:08.344 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.332+0000 7eff75456640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7eff700affa0 0x7eff7015b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:08.345 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.332+0000 7eff75456640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7eff700affa0 0x7eff7015b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46404/0 (socket says 172.21.15.16:46404) 2023-05-22T16:05:08.345 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.332+0000 7eff75456640 1 -- 172.21.15.16:0/191568066 learned_addr learned my addr 172.21.15.16:0/191568066 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:05:08.346 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.332+0000 7eff74c55640 1 --2- 172.21.15.16:0/191568066 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7eff70070fb0 0x7eff700cbbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:08.346 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.332+0000 7eff6ffff640 1 --2- 172.21.15.16:0/191568066 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7eff700afb40 0x7eff700cc110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:08.346 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.332+0000 7eff75456640 1 -- 172.21.15.16:0/191568066 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7eff70070fb0 msgr2=0x7eff700cbbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:08.347 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.332+0000 7eff75456640 1 --2- 172.21.15.16:0/191568066 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7eff70070fb0 0x7eff700cbbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:08.347 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.332+0000 7eff75456640 1 -- 172.21.15.16:0/191568066 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7eff700afb40 msgr2=0x7eff700cc110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:08.347 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.332+0000 7eff75456640 1 --2- 172.21.15.16:0/191568066 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7eff700afb40 0x7eff700cc110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:08.347 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.332+0000 7eff75456640 1 -- 172.21.15.16:0/191568066 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff600094d0 con 0x7eff700affa0 2023-05-22T16:05:08.347 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.332+0000 7eff74c55640 1 --2- 172.21.15.16:0/191568066 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7eff70070fb0 0x7eff700cbbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T16:05:08.347 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.332+0000 7eff75456640 1 --2- 172.21.15.16:0/191568066 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7eff700affa0 0x7eff7015b980 secure :-1 s=READY pgs=2370 cs=0 l=1 rev1=1 crypto rx=0x7eff5800c7f0 tx=0x7eff5800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:08.348 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.332+0000 7eff6d7fa640 1 -- 172.21.15.16:0/191568066 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff58004230 con 0x7eff700affa0 2023-05-22T16:05:08.348 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.332+0000 7eff6d7fa640 1 -- 172.21.15.16:0/191568066 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff580043b0 con 0x7eff700affa0 2023-05-22T16:05:08.348 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.332+0000 7eff76eb9640 1 -- 172.21.15.16:0/191568066 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff700ca040 con 0x7eff700affa0 2023-05-22T16:05:08.348 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.332+0000 7eff6d7fa640 1 -- 172.21.15.16:0/191568066 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff58003860 con 0x7eff700affa0 2023-05-22T16:05:08.349 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.332+0000 7eff76eb9640 1 -- 172.21.15.16:0/191568066 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff700b03a0 con 0x7eff700affa0 2023-05-22T16:05:08.349 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.336+0000 7eff76eb9640 1 -- 172.21.15.16:0/191568066 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff700ca1d0 con 0x7eff700affa0 2023-05-22T16:05:08.349 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.336+0000 7eff6d7fa640 1 -- 172.21.15.16:0/191568066 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 91) v1 ==== 84068+0+0 (secure 0 0 0) 0x7eff58003bd0 con 0x7eff700affa0 2023-05-22T16:05:08.349 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.336+0000 7eff6d7fa640 1 --2- 172.21.15.16:0/191568066 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7eff540651d0 0x7eff54067680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:08.350 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.336+0000 7eff6d7fa640 1 -- 172.21.15.16:0/191568066 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(183..183 src has 1..183) v4 ==== 7154+0+0 (secure 0 0 0) 0x7eff58082d70 con 0x7eff700affa0 2023-05-22T16:05:08.350 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.336+0000 7eff74c55640 1 --2- 172.21.15.16:0/191568066 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7eff540651d0 0x7eff54067680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:08.351 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.336+0000 7eff74c55640 1 --2- 172.21.15.16:0/191568066 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7eff540651d0 0x7eff54067680 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7eff600ad000 tx=0x7eff60008040 comp rx=0 tx=0).ready entity=mgr.10542 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:08.351 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.340+0000 7eff6d7fa640 1 -- 172.21.15.16:0/191568066 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7eff58050880 con 0x7eff700affa0 2023-05-22T16:05:08.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.528+0000 7eff76eb9640 1 -- 172.21.15.16:0/191568066 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7eff700721a0 con 0x7eff700affa0 2023-05-22T16:05:08.530 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.528+0000 7eff6d7fa640 1 -- 172.21.15.16:0/191568066 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 73 v73) v1 ==== 94+0+4657 (secure 0 0 0) 0x7eff700721a0 con 0x7eff700affa0 2023-05-22T16:05:08.531 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T16:05:08.531 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":73,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7682,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/2815032143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":2815032143},{"type":"v1","addr":"172.21.15.62:6835","nonce":2815032143}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":69},{"gid":10239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6836/1961058261","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1961058261},{"type":"v1","addr":"172.21.15.16:6836","nonce":1961058261}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":69},{"gid":10242,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/1453687184","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6835","nonce":1453687184},{"type":"v1","addr":"172.21.15.16:6837","nonce":1453687184}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":73,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-05-22T16:03:38.779645+0000","modified":"2023-05-22T16:03:42.129478+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10251":{"gid":10251,"name":"d","rank":0,"incarnation":70,"state":"up:active","state_seq":3,"addr":"172.21.15.62:6833/2486335217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":2486335217},{"type":"v1","addr":"172.21.15.62:6833","nonce":2486335217}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[23],"metadata_pool":22,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2023-05-22T16:05:08.533 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 73 2023-05-22T16:05:08.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.528+0000 7eff76eb9640 1 -- 172.21.15.16:0/191568066 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7eff540651d0 msgr2=0x7eff54067680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:08.534 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.528+0000 7eff76eb9640 1 --2- 172.21.15.16:0/191568066 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7eff540651d0 0x7eff54067680 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7eff600ad000 tx=0x7eff60008040 comp rx=0 tx=0).stop 2023-05-22T16:05:08.534 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.528+0000 7eff76eb9640 1 -- 172.21.15.16:0/191568066 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7eff700affa0 msgr2=0x7eff7015b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:08.534 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.528+0000 7eff76eb9640 1 --2- 172.21.15.16:0/191568066 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7eff700affa0 0x7eff7015b980 secure :-1 s=READY pgs=2370 cs=0 l=1 rev1=1 crypto rx=0x7eff5800c7f0 tx=0x7eff5800ccb0 comp rx=0 tx=0).stop 2023-05-22T16:05:08.534 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.528+0000 7eff76eb9640 1 -- 172.21.15.16:0/191568066 shutdown_connections 2023-05-22T16:05:08.535 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.528+0000 7eff76eb9640 1 --2- 172.21.15.16:0/191568066 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7eff540651d0 0x7eff54067680 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:08.535 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.528+0000 7eff76eb9640 1 --2- 172.21.15.16:0/191568066 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7eff700affa0 0x7eff7015b980 unknown :-1 s=CLOSED pgs=2370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:08.535 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.528+0000 7eff76eb9640 1 --2- 172.21.15.16:0/191568066 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7eff700afb40 0x7eff700cc110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:08.535 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.528+0000 7eff76eb9640 1 --2- 172.21.15.16:0/191568066 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7eff70070fb0 0x7eff700cbbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:08.536 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.528+0000 7eff76eb9640 1 -- 172.21.15.16:0/191568066 >> 172.21.15.16:0/191568066 conn(0x7eff70069720 msgr2=0x7eff700ce5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:05:08.536 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.528+0000 7eff76eb9640 1 -- 172.21.15.16:0/191568066 shutdown_connections 2023-05-22T16:05:08.536 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.532+0000 7eff76eb9640 1 -- 172.21.15.16:0/191568066 wait complete. 2023-05-22T16:05:08.549 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-05-22T16:05:08.668 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.664+0000 7fbc6ed5a640 1 Processor -- start 2023-05-22T16:05:08.668 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.664+0000 7fbc6ed5a640 1 -- start start 2023-05-22T16:05:08.669 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.668+0000 7fbc6ed5a640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fbc680ca4e0 0x7fbc6815e460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:08.669 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.668+0000 7fbc6ed5a640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbc680c9b40 0x7fbc680afb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:08.669 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.668+0000 7fbc6ed5a640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc680b0650 con 0x7fbc68070df0 2023-05-22T16:05:08.669 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.668+0000 7fbc6ed5a640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc680b0370 con 0x7fbc680ca4e0 2023-05-22T16:05:08.670 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.668+0000 7fbc6ed5a640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc6815d980 con 0x7fbc680c9b40 2023-05-22T16:05:08.670 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.668+0000 7fbc6caf6640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fbc680ca4e0 0x7fbc6815e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:08.670 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.668+0000 7fbc6d2f7640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fbc68070df0 0x7fbc68071210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:45518/0 (socket says 172.21.15.16:45518) 2023-05-22T16:05:08.670 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.668+0000 7fbc6d2f7640 1 -- 172.21.15.16:0/67086865 learned_addr learned my addr 172.21.15.16:0/67086865 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T16:05:08.671 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.668+0000 7fbc67fff640 1 --2- 172.21.15.16:0/67086865 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbc680c9b40 0x7fbc680afb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:08.672 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.668+0000 7fbc6caf6640 1 -- 172.21.15.16:0/67086865 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbc680c9b40 msgr2=0x7fbc680afb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:05:08.672 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.668+0000 7fbc6caf6640 1 --2- 172.21.15.16:0/67086865 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbc680c9b40 0x7fbc680afb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:08.672 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.668+0000 7fbc6caf6640 1 -- 172.21.15.16:0/67086865 >> v1:172.21.15.16:6789/0 conn(0x7fbc68070df0 legacy=0x7fbc68071210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:08.673 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.668+0000 7fbc677fe640 1 -- 172.21.15.16:0/67086865 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 816333916 0 0) 0x7fbc680b0650 con 0x7fbc68070df0 2023-05-22T16:05:08.673 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.668+0000 7fbc6caf6640 1 -- 172.21.15.16:0/67086865 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc6815dc60 con 0x7fbc680ca4e0 2023-05-22T16:05:08.676 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.672+0000 7fbc6caf6640 1 --2- 172.21.15.16:0/67086865 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc680ca4e0 0x7fbc6815e460 secure :-1 s=READY pgs=2372 cs=0 l=1 rev1=1 crypto rx=0x7fbc500097c0 tx=0x7fbc5001c3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=567d8bc7bed31f21 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:08.677 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.672+0000 7fbc677fe640 1 -- 172.21.15.16:0/67086865 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc500ac600 con 0x7fbc680ca4e0 2023-05-22T16:05:08.677 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.672+0000 7fbc677fe640 1 -- 172.21.15.16:0/67086865 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc500a8050 con 0x7fbc680ca4e0 2023-05-22T16:05:08.677 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.672+0000 7fbc677fe640 1 -- 172.21.15.16:0/67086865 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc500ac980 con 0x7fbc680ca4e0 2023-05-22T16:05:08.677 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.672+0000 7fbc6ed5a640 1 -- 172.21.15.16:0/67086865 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc680ca4e0 msgr2=0x7fbc6815e460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:08.677 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.672+0000 7fbc6ed5a640 1 --2- 172.21.15.16:0/67086865 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc680ca4e0 0x7fbc6815e460 secure :-1 s=READY pgs=2372 cs=0 l=1 rev1=1 crypto rx=0x7fbc500097c0 tx=0x7fbc5001c3c0 comp rx=0 tx=0).stop 2023-05-22T16:05:08.677 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.672+0000 7fbc6ed5a640 1 -- 172.21.15.16:0/67086865 shutdown_connections 2023-05-22T16:05:08.678 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.672+0000 7fbc6ed5a640 1 --2- 172.21.15.16:0/67086865 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbc680c9b40 0x7fbc680afb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:08.678 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.672+0000 7fbc6ed5a640 1 --2- 172.21.15.16:0/67086865 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc680ca4e0 0x7fbc6815e460 unknown :-1 s=CLOSED pgs=2372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:08.678 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.672+0000 7fbc6ed5a640 1 -- 172.21.15.16:0/67086865 >> 172.21.15.16:0/67086865 conn(0x7fbc68069720 msgr2=0x7fbc68069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:05:08.678 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.672+0000 7fbc6ed5a640 1 -- 172.21.15.16:0/67086865 shutdown_connections 2023-05-22T16:05:08.678 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.672+0000 7fbc6ed5a640 1 -- 172.21.15.16:0/67086865 wait complete. 2023-05-22T16:05:08.678 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.672+0000 7fbc6ed5a640 1 Processor -- start 2023-05-22T16:05:08.679 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.672+0000 7fbc6ed5a640 1 -- start start 2023-05-22T16:05:08.679 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.672+0000 7fbc6ed5a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc68070df0 0x7fbc6814c620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:08.679 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.672+0000 7fbc6ed5a640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbc680c9b40 0x7fbc680aee10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:08.679 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.672+0000 7fbc6ed5a640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbc680ca4e0 0x7fbc680af350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:08.679 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.672+0000 7fbc6ed5a640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc681514f0 con 0x7fbc68070df0 2023-05-22T16:05:08.679 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.672+0000 7fbc6ed5a640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc681517d0 con 0x7fbc680c9b40 2023-05-22T16:05:08.679 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.672+0000 7fbc6ed5a640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc68151ab0 con 0x7fbc680ca4e0 2023-05-22T16:05:08.679 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.672+0000 7fbc6caf6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc68070df0 0x7fbc6814c620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:08.680 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.672+0000 7fbc6caf6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc68070df0 0x7fbc6814c620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46420/0 (socket says 172.21.15.16:46420) 2023-05-22T16:05:08.680 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.672+0000 7fbc6caf6640 1 -- 172.21.15.16:0/1432712341 learned_addr learned my addr 172.21.15.16:0/1432712341 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:05:08.680 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.672+0000 7fbc6d2f7640 1 --2- 172.21.15.16:0/1432712341 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbc680ca4e0 0x7fbc680af350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:08.680 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.672+0000 7fbc67fff640 1 --2- 172.21.15.16:0/1432712341 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbc680c9b40 0x7fbc680aee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:08.680 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.672+0000 7fbc6caf6640 1 -- 172.21.15.16:0/1432712341 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbc680ca4e0 msgr2=0x7fbc680af350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:08.680 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.672+0000 7fbc6caf6640 1 --2- 172.21.15.16:0/1432712341 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbc680ca4e0 0x7fbc680af350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:08.681 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.672+0000 7fbc6caf6640 1 -- 172.21.15.16:0/1432712341 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbc680c9b40 msgr2=0x7fbc680aee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:08.681 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.672+0000 7fbc6caf6640 1 --2- 172.21.15.16:0/1432712341 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbc680c9b40 0x7fbc680aee10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:08.681 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.672+0000 7fbc6caf6640 1 -- 172.21.15.16:0/1432712341 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc50009420 con 0x7fbc68070df0 2023-05-22T16:05:08.681 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.672+0000 7fbc6d2f7640 1 --2- 172.21.15.16:0/1432712341 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbc680ca4e0 0x7fbc680af350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T16:05:08.681 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.672+0000 7fbc67fff640 1 --2- 172.21.15.16:0/1432712341 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbc680c9b40 0x7fbc680aee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T16:05:08.681 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.672+0000 7fbc6caf6640 1 --2- 172.21.15.16:0/1432712341 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc68070df0 0x7fbc6814c620 secure :-1 s=READY pgs=2373 cs=0 l=1 rev1=1 crypto rx=0x7fbc50019ff0 tx=0x7fbc5001ac00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:08.681 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.672+0000 7fbc657fa640 1 -- 172.21.15.16:0/1432712341 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc50004220 con 0x7fbc68070df0 2023-05-22T16:05:08.681 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.672+0000 7fbc657fa640 1 -- 172.21.15.16:0/1432712341 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc50004380 con 0x7fbc68070df0 2023-05-22T16:05:08.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.672+0000 7fbc657fa640 1 -- 172.21.15.16:0/1432712341 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc50003820 con 0x7fbc68070df0 2023-05-22T16:05:08.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.672+0000 7fbc6ed5a640 1 -- 172.21.15.16:0/1432712341 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc680c8e80 con 0x7fbc68070df0 2023-05-22T16:05:08.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.676+0000 7fbc6ed5a640 1 -- 172.21.15.16:0/1432712341 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc680ca8b0 con 0x7fbc68070df0 2023-05-22T16:05:08.683 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.676+0000 7fbc6ed5a640 1 -- 172.21.15.16:0/1432712341 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc680c9010 con 0x7fbc68070df0 2023-05-22T16:05:08.683 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.676+0000 7fbc657fa640 1 -- 172.21.15.16:0/1432712341 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 91) v1 ==== 84068+0+0 (secure 0 0 0) 0x7fbc500b3050 con 0x7fbc68070df0 2023-05-22T16:05:08.683 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.676+0000 7fbc657fa640 1 --2- 172.21.15.16:0/1432712341 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fbc48065200 0x7fbc480676b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:08.683 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.676+0000 7fbc657fa640 1 -- 172.21.15.16:0/1432712341 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(183..183 src has 1..183) v4 ==== 7154+0+0 (secure 0 0 0) 0x7fbc5011cb60 con 0x7fbc68070df0 2023-05-22T16:05:08.684 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.676+0000 7fbc67fff640 1 --2- 172.21.15.16:0/1432712341 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fbc48065200 0x7fbc480676b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:08.684 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.676+0000 7fbc67fff640 1 --2- 172.21.15.16:0/1432712341 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fbc48065200 0x7fbc480676b0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fbc68164490 tx=0x7fbc5800b040 comp rx=0 tx=0).ready entity=mgr.10542 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:08.684 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.680+0000 7fbc657fa640 1 -- 172.21.15.16:0/1432712341 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fbc500ea6d0 con 0x7fbc68070df0 2023-05-22T16:05:08.865 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.864+0000 7fbc6ed5a640 1 -- 172.21.15.16:0/1432712341 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbc6804dbe0 con 0x7fbc68070df0 2023-05-22T16:05:08.866 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.864+0000 7fbc657fa640 1 -- 172.21.15.16:0/1432712341 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 73 v73) v1 ==== 94+0+4657 (secure 0 0 0) 0x7fbc500b1090 con 0x7fbc68070df0 2023-05-22T16:05:08.867 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T16:05:08.867 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":73,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7682,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/2815032143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":2815032143},{"type":"v1","addr":"172.21.15.62:6835","nonce":2815032143}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":69},{"gid":10239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6836/1961058261","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1961058261},{"type":"v1","addr":"172.21.15.16:6836","nonce":1961058261}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":69},{"gid":10242,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/1453687184","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6835","nonce":1453687184},{"type":"v1","addr":"172.21.15.16:6837","nonce":1453687184}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":73,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-05-22T16:03:38.779645+0000","modified":"2023-05-22T16:03:42.129478+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10251":{"gid":10251,"name":"d","rank":0,"incarnation":70,"state":"up:active","state_seq":3,"addr":"172.21.15.62:6833/2486335217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":2486335217},{"type":"v1","addr":"172.21.15.62:6833","nonce":2486335217}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[23],"metadata_pool":22,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2023-05-22T16:05:08.867 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 73 2023-05-22T16:05:08.869 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.864+0000 7fbc6ed5a640 1 -- 172.21.15.16:0/1432712341 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fbc48065200 msgr2=0x7fbc480676b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:08.869 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.864+0000 7fbc6ed5a640 1 --2- 172.21.15.16:0/1432712341 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fbc48065200 0x7fbc480676b0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fbc68164490 tx=0x7fbc5800b040 comp rx=0 tx=0).stop 2023-05-22T16:05:08.869 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.864+0000 7fbc6ed5a640 1 -- 172.21.15.16:0/1432712341 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc68070df0 msgr2=0x7fbc6814c620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:08.870 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.864+0000 7fbc6ed5a640 1 --2- 172.21.15.16:0/1432712341 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc68070df0 0x7fbc6814c620 secure :-1 s=READY pgs=2373 cs=0 l=1 rev1=1 crypto rx=0x7fbc50019ff0 tx=0x7fbc5001ac00 comp rx=0 tx=0).stop 2023-05-22T16:05:08.870 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.864+0000 7fbc6ed5a640 1 -- 172.21.15.16:0/1432712341 shutdown_connections 2023-05-22T16:05:08.870 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.864+0000 7fbc6ed5a640 1 --2- 172.21.15.16:0/1432712341 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbc680ca4e0 0x7fbc680af350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:08.871 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.864+0000 7fbc6ed5a640 1 --2- 172.21.15.16:0/1432712341 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbc680c9b40 0x7fbc680aee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:08.871 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.864+0000 7fbc6ed5a640 1 --2- 172.21.15.16:0/1432712341 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fbc48065200 0x7fbc480676b0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:08.871 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.864+0000 7fbc6ed5a640 1 --2- 172.21.15.16:0/1432712341 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc68070df0 0x7fbc6814c620 unknown :-1 s=CLOSED pgs=2373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:08.871 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.864+0000 7fbc6ed5a640 1 -- 172.21.15.16:0/1432712341 >> 172.21.15.16:0/1432712341 conn(0x7fbc68069720 msgr2=0x7fbc68069af0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:05:08.871 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.864+0000 7fbc6ed5a640 1 -- 172.21.15.16:0/1432712341 shutdown_connections 2023-05-22T16:05:08.871 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.868+0000 7fbc6ed5a640 1 -- 172.21.15.16:0/1432712341 wait complete. 2023-05-22T16:05:08.884 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-05-22T16:05:08.992 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.988+0000 7f1e2cea4640 1 Processor -- start 2023-05-22T16:05:08.992 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.988+0000 7f1e2cea4640 1 -- start start 2023-05-22T16:05:08.993 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.992+0000 7f1e2cea4640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f1e280c9b40 0x7f1e2815e460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:08.993 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.992+0000 7f1e2cea4640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e280b0650 con 0x7f1e280ca4e0 2023-05-22T16:05:08.993 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.992+0000 7f1e2cea4640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e280b0370 con 0x7f1e28070df0 2023-05-22T16:05:08.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.992+0000 7f1e2cea4640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e2815d980 con 0x7f1e280c9b40 2023-05-22T16:05:08.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.992+0000 7f1e25d9b640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f1e280c9b40 0x7f1e2815e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:08.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.992+0000 7f1e2659c640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f1e280ca4e0 0x7f1e280afb40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:45534/0 (socket says 172.21.15.16:45534) 2023-05-22T16:05:08.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.992+0000 7f1e2659c640 1 -- 172.21.15.16:0/1897052720 learned_addr learned my addr 172.21.15.16:0/1897052720 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T16:05:08.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.992+0000 7f1e25d9b640 1 -- 172.21.15.16:0/1897052720 >> v1:172.21.15.62:6789/0 conn(0x7f1e28070df0 legacy=0x7f1e28071210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:05:08.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.992+0000 7f1e25d9b640 1 -- 172.21.15.16:0/1897052720 >> v1:172.21.15.16:6789/0 conn(0x7f1e280ca4e0 legacy=0x7f1e280afb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:08.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.992+0000 7f1e25d9b640 1 -- 172.21.15.16:0/1897052720 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e2815dc60 con 0x7f1e280c9b40 2023-05-22T16:05:08.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.992+0000 7f1e25d9b640 1 --2- 172.21.15.16:0/1897052720 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1e280c9b40 0x7f1e2815e460 secure :-1 s=READY pgs=2375 cs=0 l=1 rev1=1 crypto rx=0x7f1e1c0097c0 tx=0x7f1e1c01c410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8dbfae308f8d8876 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:08.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.992+0000 7f1e2559a640 1 -- 172.21.15.16:0/1897052720 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e1c0ab600 con 0x7f1e280c9b40 2023-05-22T16:05:08.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.992+0000 7f1e2559a640 1 -- 172.21.15.16:0/1897052720 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e1c0a7050 con 0x7f1e280c9b40 2023-05-22T16:05:08.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.992+0000 7f1e2559a640 1 -- 172.21.15.16:0/1897052720 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e1c0ab980 con 0x7f1e280c9b40 2023-05-22T16:05:08.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.992+0000 7f1e2cea4640 1 -- 172.21.15.16:0/1897052720 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1e280c9b40 msgr2=0x7f1e2815e460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:09.002 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.992+0000 7f1e2cea4640 1 --2- 172.21.15.16:0/1897052720 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1e280c9b40 0x7f1e2815e460 secure :-1 s=READY pgs=2375 cs=0 l=1 rev1=1 crypto rx=0x7f1e1c0097c0 tx=0x7f1e1c01c410 comp rx=0 tx=0).stop 2023-05-22T16:05:09.002 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.992+0000 7f1e2cea4640 1 -- 172.21.15.16:0/1897052720 shutdown_connections 2023-05-22T16:05:09.002 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.992+0000 7f1e2cea4640 1 --2- 172.21.15.16:0/1897052720 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1e280c9b40 0x7f1e2815e460 unknown :-1 s=CLOSED pgs=2375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:09.002 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.992+0000 7f1e2cea4640 1 -- 172.21.15.16:0/1897052720 >> 172.21.15.16:0/1897052720 conn(0x7f1e28069720 msgr2=0x7f1e28069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:05:09.002 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.992+0000 7f1e2cea4640 1 -- 172.21.15.16:0/1897052720 shutdown_connections 2023-05-22T16:05:09.002 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.992+0000 7f1e2cea4640 1 -- 172.21.15.16:0/1897052720 wait complete. 2023-05-22T16:05:09.004 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.996+0000 7f1e2cea4640 1 Processor -- start 2023-05-22T16:05:09.005 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.996+0000 7f1e2cea4640 1 -- start start 2023-05-22T16:05:09.005 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.996+0000 7f1e2cea4640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1e28070df0 0x7f1e28150e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:09.005 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.996+0000 7f1e2cea4640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1e280c9b40 0x7f1e28154420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:09.005 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.996+0000 7f1e2cea4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1e280ca4e0 0x7f1e28154960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:09.005 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.996+0000 7f1e2cea4640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e280ab7d0 con 0x7f1e280ca4e0 2023-05-22T16:05:09.006 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.996+0000 7f1e2cea4640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e281f8150 con 0x7f1e280c9b40 2023-05-22T16:05:09.006 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.996+0000 7f1e2cea4640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e281513a0 con 0x7f1e28070df0 2023-05-22T16:05:09.006 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.996+0000 7f1e26d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1e280ca4e0 0x7f1e28154960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:09.006 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.996+0000 7f1e26d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1e280ca4e0 0x7f1e28154960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46440/0 (socket says 172.21.15.16:46440) 2023-05-22T16:05:09.007 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.996+0000 7f1e26d9d640 1 -- 172.21.15.16:0/869528178 learned_addr learned my addr 172.21.15.16:0/869528178 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:05:09.007 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.996+0000 7f1e25d9b640 1 --2- 172.21.15.16:0/869528178 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1e280c9b40 0x7f1e28154420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:09.007 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.996+0000 7f1e2659c640 1 --2- 172.21.15.16:0/869528178 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1e28070df0 0x7f1e28150e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:09.007 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.996+0000 7f1e26d9d640 1 -- 172.21.15.16:0/869528178 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1e28070df0 msgr2=0x7f1e28150e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:09.007 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.996+0000 7f1e26d9d640 1 --2- 172.21.15.16:0/869528178 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1e28070df0 0x7f1e28150e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:09.007 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.996+0000 7f1e26d9d640 1 -- 172.21.15.16:0/869528178 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1e280c9b40 msgr2=0x7f1e28154420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:09.008 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.996+0000 7f1e26d9d640 1 --2- 172.21.15.16:0/869528178 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1e280c9b40 0x7f1e28154420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:09.008 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.996+0000 7f1e26d9d640 1 -- 172.21.15.16:0/869528178 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e1c009420 con 0x7f1e280ca4e0 2023-05-22T16:05:09.008 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.996+0000 7f1e25d9b640 1 --2- 172.21.15.16:0/869528178 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1e280c9b40 0x7f1e28154420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T16:05:09.008 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.996+0000 7f1e26d9d640 1 --2- 172.21.15.16:0/869528178 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1e280ca4e0 0x7f1e28154960 secure :-1 s=READY pgs=2376 cs=0 l=1 rev1=1 crypto rx=0x7f1e2000b990 tx=0x7f1e2000be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:09.008 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.996+0000 7f1e16ffd640 1 -- 172.21.15.16:0/869528178 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e2000cba0 con 0x7f1e280ca4e0 2023-05-22T16:05:09.008 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.996+0000 7f1e16ffd640 1 -- 172.21.15.16:0/869528178 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e2000cd00 con 0x7f1e280ca4e0 2023-05-22T16:05:09.008 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.996+0000 7f1e2cea4640 1 -- 172.21.15.16:0/869528178 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e280c9010 con 0x7f1e280ca4e0 2023-05-22T16:05:09.009 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.996+0000 7f1e16ffd640 1 -- 172.21.15.16:0/869528178 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e20014680 con 0x7f1e280ca4e0 2023-05-22T16:05:09.009 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.996+0000 7f1e2cea4640 1 -- 172.21.15.16:0/869528178 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e280c8e80 con 0x7f1e280ca4e0 2023-05-22T16:05:09.009 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:08.996+0000 7f1e2cea4640 1 -- 172.21.15.16:0/869528178 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e280c91a0 con 0x7f1e280ca4e0 2023-05-22T16:05:09.009 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:09.000+0000 7f1e16ffd640 1 -- 172.21.15.16:0/869528178 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 91) v1 ==== 84068+0+0 (secure 0 0 0) 0x7f1e200147e0 con 0x7f1e280ca4e0 2023-05-22T16:05:09.009 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:09.000+0000 7f1e16ffd640 1 --2- 172.21.15.16:0/869528178 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f1e080651d0 0x7f1e08067680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:09.009 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:09.000+0000 7f1e16ffd640 1 -- 172.21.15.16:0/869528178 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(183..183 src has 1..183) v4 ==== 7154+0+0 (secure 0 0 0) 0x7f1e20082850 con 0x7f1e280ca4e0 2023-05-22T16:05:09.009 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:09.000+0000 7f1e2659c640 1 --2- 172.21.15.16:0/869528178 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f1e080651d0 0x7f1e08067680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:09.010 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:09.000+0000 7f1e2659c640 1 --2- 172.21.15.16:0/869528178 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f1e080651d0 0x7f1e08067680 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f1e18005890 tx=0x7f1e18009040 comp rx=0 tx=0).ready entity=mgr.10542 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:09.010 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:09.004+0000 7f1e16ffd640 1 -- 172.21.15.16:0/869528178 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f1e200503c0 con 0x7f1e280ca4e0 2023-05-22T16:05:09.156 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:09.152+0000 7f1e2cea4640 1 -- 172.21.15.16:0/869528178 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1e2804dbe0 con 0x7f1e280ca4e0 2023-05-22T16:05:09.157 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:09.156+0000 7f1e16ffd640 1 -- 172.21.15.16:0/869528178 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v183) v1 ==== 74+0+19182 (secure 0 0 0) 0x7f1e20019020 con 0x7f1e280ca4e0 2023-05-22T16:05:09.158 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T16:05:09.158 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":183,"fsid":"b2719834-7847-4d98-90a6-f65d57f85c49","created":"2023-05-22T15:26:25.259314+0000","modified":"2023-05-22T16:05:06.787958+0000","last_up_change":"2023-05-22T15:26:36.448524+0000","last_in_change":"2023-05-22T15:26:27.675668+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":23,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-05-22T15:26:38.019781+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-05-22T15:26:43.570514+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":22,"pool_name":"cephfs_metadata","create_time":"2023-05-22T16:03:37.371980+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"178","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_data","create_time":"2023-05-22T16:03:37.756887+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"182","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":5,"snap_epoch":182,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9e7ff854-b25f-4cbb-a5c2-1799d9c53d2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":37021},{"type":"v1","addr":"172.21.15.16:6825","nonce":37021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":37021},{"type":"v1","addr":"172.21.15.16:6827","nonce":37021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":37021},{"type":"v1","addr":"172.21.15.16:6831","nonce":37021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":37021},{"type":"v1","addr":"172.21.15.16:6829","nonce":37021}]},"public_addr":"172.21.15.16:6825/37021","cluster_addr":"172.21.15.16:6827/37021","heartbeat_back_addr":"172.21.15.16:6831/37021","heartbeat_front_addr":"172.21.15.16:6829/37021","state":["exists","up"]},{"osd":1,"uuid":"4702fbc9-a927-4565-aca5-41dc69cc8264","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":37008},{"type":"v1","addr":"172.21.15.16:6801","nonce":37008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":37008},{"type":"v1","addr":"172.21.15.16:6803","nonce":37008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":37008},{"type":"v1","addr":"172.21.15.16:6807","nonce":37008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":37008},{"type":"v1","addr":"172.21.15.16:6805","nonce":37008}]},"public_addr":"172.21.15.16:6801/37008","cluster_addr":"172.21.15.16:6803/37008","heartbeat_back_addr":"172.21.15.16:6807/37008","heartbeat_front_addr":"172.21.15.16:6805/37008","state":["exists","up"]},{"osd":2,"uuid":"1e7bba46-4eba-47ef-bb81-5e2d371638ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":37009},{"type":"v1","addr":"172.21.15.16:6809","nonce":37009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":37009},{"type":"v1","addr":"172.21.15.16:6811","nonce":37009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":37009},{"type":"v1","addr":"172.21.15.16:6815","nonce":37009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":37009},{"type":"v1","addr":"172.21.15.16:6813","nonce":37009}]},"public_addr":"172.21.15.16:6809/37009","cluster_addr":"172.21.15.16:6811/37009","heartbeat_back_addr":"172.21.15.16:6815/37009","heartbeat_front_addr":"172.21.15.16:6813/37009","state":["exists","up"]},{"osd":3,"uuid":"383cacc0-5a88-4904-990e-498ca59ef48a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":37020},{"type":"v1","addr":"172.21.15.16:6817","nonce":37020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":37020},{"type":"v1","addr":"172.21.15.16:6819","nonce":37020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":37020},{"type":"v1","addr":"172.21.15.16:6823","nonce":37020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":37020},{"type":"v1","addr":"172.21.15.16:6821","nonce":37020}]},"public_addr":"172.21.15.16:6817/37020","cluster_addr":"172.21.15.16:6819/37020","heartbeat_back_addr":"172.21.15.16:6823/37020","heartbeat_front_addr":"172.21.15.16:6821/37020","state":["exists","up"]},{"osd":4,"uuid":"970500ef-95b7-42c1-b080-eed4fae7013d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":37353},{"type":"v1","addr":"172.21.15.62:6817","nonce":37353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6819","nonce":37353},{"type":"v1","addr":"172.21.15.62:6822","nonce":37353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37353},{"type":"v1","addr":"172.21.15.62:6831","nonce":37353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37353},{"type":"v1","addr":"172.21.15.62:6826","nonce":37353}]},"public_addr":"172.21.15.62:6817/37353","cluster_addr":"172.21.15.62:6822/37353","heartbeat_back_addr":"172.21.15.62:6831/37353","heartbeat_front_addr":"172.21.15.62:6826/37353","state":["exists","up"]},{"osd":5,"uuid":"022d67db-d4d8-445b-b1b9-8b446a00f5c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":37352},{"type":"v1","addr":"172.21.15.62:6801","nonce":37352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":37352},{"type":"v1","addr":"172.21.15.62:6803","nonce":37352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":37352},{"type":"v1","addr":"172.21.15.62:6807","nonce":37352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":37352},{"type":"v1","addr":"172.21.15.62:6805","nonce":37352}]},"public_addr":"172.21.15.62:6801/37352","cluster_addr":"172.21.15.62:6803/37352","heartbeat_back_addr":"172.21.15.62:6807/37352","heartbeat_front_addr":"172.21.15.62:6805/37352","state":["exists","up"]},{"osd":6,"uuid":"42bd372b-6688-4a1b-8407-e16e8c266e33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":37354},{"type":"v1","addr":"172.21.15.62:6820","nonce":37354}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":37354},{"type":"v1","addr":"172.21.15.62:6823","nonce":37354}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6829","nonce":37354},{"type":"v1","addr":"172.21.15.62:6830","nonce":37354}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6825","nonce":37354},{"type":"v1","addr":"172.21.15.62:6827","nonce":37354}]},"public_addr":"172.21.15.62:6820/37354","cluster_addr":"172.21.15.62:6823/37354","heartbeat_back_addr":"172.21.15.62:6830/37354","heartbeat_front_addr":"172.21.15.62:6827/37354","state":["exists","up"]},{"osd":7,"uuid":"ac70b2a3-f41b-4bf2-b227-7f2cbb55426d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37355},{"type":"v1","addr":"172.21.15.62:6809","nonce":37355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37355},{"type":"v1","addr":"172.21.15.62:6811","nonce":37355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":37355},{"type":"v1","addr":"172.21.15.62:6815","nonce":37355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":37355},{"type":"v1","addr":"172.21.15.62:6813","nonce":37355}]},"public_addr":"172.21.15.62:6809/37355","cluster_addr":"172.21.15.62:6811/37355","heartbeat_back_addr":"172.21.15.62:6815/37355","heartbeat_front_addr":"172.21.15.62:6813/37355","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.368896+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.107273+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.276888+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.287502+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.124791+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.251937+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.255679+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.232797+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:0/1900219297":"2023-05-23T16:05:06.787906+0000","172.21.15.16:0/4377472":"2023-05-23T16:05:06.787906+0000","172.21.15.16:0/1923190612":"2023-05-23T16:05:06.787906+0000","172.21.15.16:0/2696230409":"2023-05-23T16:05:06.787906+0000","172.21.15.16:0/2495910544":"2023-05-23T16:03:52.657800+0000","172.21.15.16:0/1761665129":"2023-05-23T16:03:52.657800+0000","172.21.15.16:6833/65890":"2023-05-23T16:03:52.657800+0000","172.21.15.16:0/2966441324":"2023-05-23T16:05:06.787906+0000","172.21.15.16:6832/65890":"2023-05-23T16:03:52.657800+0000","172.21.15.16:0/586164935":"2023-05-23T16:05:06.787906+0000","172.21.15.16:0/216273587":"2023-05-23T16:05:06.787906+0000","172.21.15.16:0/3698789191":"2023-05-23T16:05:06.787906+0000","172.21.15.16:0/4138270792":"2023-05-23T16:03:52.657800+0000","172.21.15.16:0/2442191907":"2023-05-23T16:03:52.657800+0000","172.21.15.16:0/3487890192":"2023-05-23T16:03:52.657800+0000","172.21.15.16:0/3247164174":"2023-05-23T16:03:52.657800+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":17,"snaps":[{"begin":2,"length":2}]},{"pool":21,"snaps":[{"begin":25,"length":9}]},{"pool":23,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-05-22T16:05:09.159 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:09.156+0000 7f1e2cea4640 1 -- 172.21.15.16:0/869528178 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f1e080651d0 msgr2=0x7f1e08067680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:09.159 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:09.156+0000 7f1e2cea4640 1 --2- 172.21.15.16:0/869528178 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f1e080651d0 0x7f1e08067680 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f1e18005890 tx=0x7f1e18009040 comp rx=0 tx=0).stop 2023-05-22T16:05:09.159 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:09.156+0000 7f1e2cea4640 1 -- 172.21.15.16:0/869528178 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1e280ca4e0 msgr2=0x7f1e28154960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:09.160 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:09.156+0000 7f1e2cea4640 1 --2- 172.21.15.16:0/869528178 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1e280ca4e0 0x7f1e28154960 secure :-1 s=READY pgs=2376 cs=0 l=1 rev1=1 crypto rx=0x7f1e2000b990 tx=0x7f1e2000be50 comp rx=0 tx=0).stop 2023-05-22T16:05:09.160 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:09.156+0000 7f1e2cea4640 1 -- 172.21.15.16:0/869528178 shutdown_connections 2023-05-22T16:05:09.160 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:09.156+0000 7f1e2cea4640 1 --2- 172.21.15.16:0/869528178 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f1e080651d0 0x7f1e08067680 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:09.160 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:09.156+0000 7f1e2cea4640 1 --2- 172.21.15.16:0/869528178 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1e280ca4e0 0x7f1e28154960 unknown :-1 s=CLOSED pgs=2376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:09.160 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:09.156+0000 7f1e2cea4640 1 --2- 172.21.15.16:0/869528178 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1e280c9b40 0x7f1e28154420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:09.160 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:09.156+0000 7f1e2cea4640 1 --2- 172.21.15.16:0/869528178 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1e28070df0 0x7f1e28150e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:09.161 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:09.156+0000 7f1e2cea4640 1 -- 172.21.15.16:0/869528178 >> 172.21.15.16:0/869528178 conn(0x7f1e28069720 msgr2=0x7f1e280afb40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:05:09.161 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:09.156+0000 7f1e2cea4640 1 -- 172.21.15.16:0/869528178 shutdown_connections 2023-05-22T16:05:09.161 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:09.156+0000 7f1e2cea4640 1 -- 172.21.15.16:0/869528178 wait complete. 2023-05-22T16:05:09.175 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-05-22T16:05:09.292 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:09.292+0000 7fef6635d640 1 Processor -- start 2023-05-22T16:05:09.293 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:09.292+0000 7fef6635d640 1 -- start start 2023-05-22T16:05:09.293 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:09.292+0000 7fef6635d640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fef60158a10 0x7fef6006e480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:09.294 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:09.292+0000 7fef6635d640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fef6006cdf0 0x7fef60070be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:09.294 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:09.292+0000 7fef6635d640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef60158de0 con 0x7fef60070760 2023-05-22T16:05:09.294 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:09.292+0000 7fef6635d640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef6006e060 con 0x7fef60158a10 2023-05-22T16:05:09.295 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:09.292+0000 7fef6635d640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef600711e0 con 0x7fef6006cdf0 2023-05-22T16:05:09.295 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:09.292+0000 7fef5f7fe640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fef60070760 0x7fef6006c6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:44312/0 (socket says 172.21.15.16:44312) 2023-05-22T16:05:09.295 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:09.292+0000 7fef5f7fe640 1 -- 172.21.15.16:0/704028766 learned_addr learned my addr 172.21.15.16:0/704028766 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T16:05:09.297 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:09.292+0000 7fef5ffff640 1 --2- 172.21.15.16:0/704028766 >> v2:172.21.15.62:3300/0 conn(0x7fef60158a10 0x7fef6006e480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:09.297 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:09.292+0000 7fef648fa640 1 --2- 172.21.15.16:0/704028766 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fef6006cdf0 0x7fef60070be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:09.297 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:09.292+0000 7fef5effd640 1 -- 172.21.15.16:0/704028766 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2711595868 0 0) 0x7fef60158de0 con 0x7fef60070760 2023-05-22T16:05:09.298 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:09.292+0000 7fef5effd640 1 -- 172.21.15.16:0/704028766 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef4c003530 con 0x7fef60070760 2023-05-22T16:05:09.298 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:09.292+0000 7fef5effd640 1 -- 172.21.15.16:0/704028766 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1151835502 0 0) 0x7fef4c003530 con 0x7fef60070760 2023-05-22T16:05:09.298 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:09.292+0000 7fef5effd640 1 -- 172.21.15.16:0/704028766 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fef6006cdf0 msgr2=0x7fef60070be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:05:09.299 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:09.292+0000 7fef5effd640 1 --2- 172.21.15.16:0/704028766 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fef6006cdf0 0x7fef60070be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:09.299 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:09.292+0000 7fef5effd640 1 -- 172.21.15.16:0/704028766 >> v2:172.21.15.62:3300/0 conn(0x7fef60158a10 msgr2=0x7fef6006e480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:05:09.299 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:09.292+0000 7fef5effd640 1 --2- 172.21.15.16:0/704028766 >> v2:172.21.15.62:3300/0 conn(0x7fef60158a10 0x7fef6006e480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:09.299 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:09.292+0000 7fef5effd640 1 -- 172.21.15.16:0/704028766 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef600714c0 con 0x7fef60070760 2023-05-22T16:05:09.299 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:09.292+0000 7fef5effd640 1 -- 172.21.15.16:0/704028766 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7fef48001680 con 0x7fef60070760 2023-05-22T16:05:09.299 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:09.292+0000 7fef5effd640 1 -- 172.21.15.16:0/704028766 >> v1:172.21.15.16:6789/0 conn(0x7fef60070760 legacy=0x7fef6006c6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:09.300 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:09.292+0000 7fef5effd640 1 --2- 172.21.15.16:0/704028766 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fef4c003980 0x7fef4c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:09.300 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:09.292+0000 7fef5effd640 1 --2- 172.21.15.16:0/704028766 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fef6006cdf0 0x7fef4c0047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:09.300 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:09.292+0000 7fef5effd640 1 --2- 172.21.15.16:0/704028766 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fef4c004d10 0x7fef4c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:09.300 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:09.292+0000 7fef5effd640 1 -- 172.21.15.16:0/704028766 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef6006e060 con 0x7fef4c004d10 2023-05-22T16:05:09.300 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:09.292+0000 7fef5effd640 1 -- 172.21.15.16:0/704028766 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef600711e0 con 0x7fef4c003980 2023-05-22T16:05:09.300 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:09.292+0000 7fef5effd640 1 -- 172.21.15.16:0/704028766 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef4c00b990 con 0x7fef6006cdf0 2023-05-22T16:05:09.300 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:09.292+0000 7fef648fa640 1 --2- 172.21.15.16:0/704028766 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fef4c004d10 0x7fef4c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:09.301 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:09.292+0000 7fef5ffff640 1 --2- 172.21.15.16:0/704028766 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fef4c003980 0x7fef4c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:09.301 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:09.292+0000 7fef5f7fe640 1 --2- 172.21.15.16:0/704028766 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fef6006cdf0 0x7fef4c0047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:09.301 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:09.292+0000 7fef648fa640 1 -- 172.21.15.16:0/704028766 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fef6006cdf0 msgr2=0x7fef4c0047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:05:09.301 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:09.292+0000 7fef648fa640 1 --2- 172.21.15.16:0/704028766 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fef6006cdf0 0x7fef4c0047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:09.301 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:09.292+0000 7fef648fa640 1 -- 172.21.15.16:0/704028766 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fef4c003980 msgr2=0x7fef4c003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:05:09.301 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:09.292+0000 7fef648fa640 1 --2- 172.21.15.16:0/704028766 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fef4c003980 0x7fef4c003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:09.301 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:09.292+0000 7fef648fa640 1 -- 172.21.15.16:0/704028766 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef4c003530 con 0x7fef4c004d10 2023-05-22T16:05:09.302 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:09.292+0000 7fef5ffff640 1 --2- 172.21.15.16:0/704028766 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fef4c003980 0x7fef4c003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T16:05:09.302 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:09.296+0000 7fef648fa640 1 --2- 172.21.15.16:0/704028766 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fef4c004d10 0x7fef4c005180 secure :-1 s=READY pgs=2378 cs=0 l=1 rev1=1 crypto rx=0x7fef50009f90 tx=0x7fef5001cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64f473bbc5e4167f server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:09.302 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:09.296+0000 7fef5effd640 1 -- 172.21.15.16:0/704028766 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef500a5b20 con 0x7fef4c004d10 2023-05-22T16:05:09.302 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:09.296+0000 7fef5effd640 1 -- 172.21.15.16:0/704028766 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef5000f040 con 0x7fef4c004d10 2023-05-22T16:05:09.302 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:09.296+0000 7fef5effd640 1 -- 172.21.15.16:0/704028766 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef500ad440 con 0x7fef4c004d10 2023-05-22T16:05:09.306 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:09.296+0000 7fef6635d640 1 -- 172.21.15.16:0/704028766 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fef4c004d10 msgr2=0x7fef4c005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:09.306 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:09.296+0000 7fef6635d640 1 --2- 172.21.15.16:0/704028766 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fef4c004d10 0x7fef4c005180 secure :-1 s=READY pgs=2378 cs=0 l=1 rev1=1 crypto rx=0x7fef50009f90 tx=0x7fef5001cc20 comp rx=0 tx=0).stop 2023-05-22T16:05:09.307 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:09.296+0000 7fef648fa640 1 -- 172.21.15.16:0/704028766 reap_dead start 2023-05-22T16:05:09.307 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:09.296+0000 7fef6635d640 1 -- 172.21.15.16:0/704028766 shutdown_connections 2023-05-22T16:05:09.307 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:09.296+0000 7fef6635d640 1 -- 172.21.15.16:0/704028766 >> 172.21.15.16:0/704028766 conn(0x7fef600cc870 msgr2=0x7fef600cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:05:09.307 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:09.296+0000 7fef6635d640 1 -- 172.21.15.16:0/704028766 shutdown_connections 2023-05-22T16:05:09.307 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:09.296+0000 7fef6635d640 1 -- 172.21.15.16:0/704028766 wait complete. 2023-05-22T16:05:09.307 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:09.296+0000 7fef6635d640 1 Processor -- start 2023-05-22T16:05:09.308 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:09.296+0000 7fef6635d640 1 -- start start 2023-05-22T16:05:09.308 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:09.296+0000 7fef6635d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fef4c004d10 0x7fef600a9240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:09.308 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:09.296+0000 7fef6635d640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fef600c8490 0x7fef600c8900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:09.308 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:09.296+0000 7fef6635d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fef600a7430 0x7fef600a78a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:09.308 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:09.300+0000 7fef6635d640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef600c8e40 con 0x7fef600a7430 2023-05-22T16:05:09.309 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:09.300+0000 7fef6635d640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef600a9780 con 0x7fef4c004d10 2023-05-22T16:05:09.309 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:09.300+0000 7fef6635d640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef600a81a0 con 0x7fef600c8490 2023-05-22T16:05:09.309 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:09.300+0000 7fef648fa640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fef600a7430 0x7fef600a78a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:09.309 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:09.300+0000 7fef648fa640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fef600a7430 0x7fef600a78a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39986/0 (socket says 172.21.15.16:39986) 2023-05-22T16:05:09.309 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:09.300+0000 7fef648fa640 1 -- 172.21.15.16:0/2269441423 learned_addr learned my addr 172.21.15.16:0/2269441423 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:05:09.309 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:09.300+0000 7fef5f7fe640 1 --2- 172.21.15.16:0/2269441423 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fef600c8490 0x7fef600c8900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:09.309 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:09.300+0000 7fef5ffff640 1 --2- 172.21.15.16:0/2269441423 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fef4c004d10 0x7fef600a9240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:09.310 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:09.300+0000 7fef648fa640 1 -- 172.21.15.16:0/2269441423 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fef600c8490 msgr2=0x7fef600c8900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:09.310 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:09.300+0000 7fef648fa640 1 --2- 172.21.15.16:0/2269441423 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fef600c8490 0x7fef600c8900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:09.310 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:09.300+0000 7fef648fa640 1 -- 172.21.15.16:0/2269441423 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fef4c004d10 msgr2=0x7fef600a9240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:09.310 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:09.300+0000 7fef648fa640 1 --2- 172.21.15.16:0/2269441423 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fef4c004d10 0x7fef600a9240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:09.310 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:09.300+0000 7fef648fa640 1 -- 172.21.15.16:0/2269441423 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef600cfe40 con 0x7fef600a7430 2023-05-22T16:05:09.310 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:09.300+0000 7fef5f7fe640 1 --2- 172.21.15.16:0/2269441423 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fef600c8490 0x7fef600c8900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T16:05:09.310 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:09.300+0000 7fef648fa640 1 --2- 172.21.15.16:0/2269441423 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fef600a7430 0x7fef600a78a0 secure :-1 s=READY pgs=2379 cs=0 l=1 rev1=1 crypto rx=0x7fef500b0040 tx=0x7fef50009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:09.310 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:09.300+0000 7fef5cff9640 1 -- 172.21.15.16:0/2269441423 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef500039a0 con 0x7fef600a7430 2023-05-22T16:05:09.311 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:09.300+0000 7fef5cff9640 1 -- 172.21.15.16:0/2269441423 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef50003b00 con 0x7fef600a7430 2023-05-22T16:05:09.311 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:09.300+0000 7fef5cff9640 1 -- 172.21.15.16:0/2269441423 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef500ae890 con 0x7fef600a7430 2023-05-22T16:05:09.311 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:09.300+0000 7fef6635d640 1 -- 172.21.15.16:0/2269441423 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef6006dc80 con 0x7fef600a7430 2023-05-22T16:05:09.311 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:09.300+0000 7fef6635d640 1 -- 172.21.15.16:0/2269441423 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef6006e9c0 con 0x7fef600a7430 2023-05-22T16:05:09.311 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:09.300+0000 7fef6635d640 1 -- 172.21.15.16:0/2269441423 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef6004dbe0 con 0x7fef600a7430 2023-05-22T16:05:09.311 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:09.300+0000 7fef5cff9640 1 -- 172.21.15.16:0/2269441423 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 91) v1 ==== 84068+0+0 (secure 0 0 0) 0x7fef500b4040 con 0x7fef600a7430 2023-05-22T16:05:09.311 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:09.300+0000 7fef5cff9640 1 --2- 172.21.15.16:0/2269441423 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fef40065200 0x7fef400676b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:09.311 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:09.300+0000 7fef5cff9640 1 -- 172.21.15.16:0/2269441423 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(183..183 src has 1..183) v4 ==== 7154+0+0 (secure 0 0 0) 0x7fef5011c7e0 con 0x7fef600a7430 2023-05-22T16:05:09.312 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:09.300+0000 7fef5ffff640 1 --2- 172.21.15.16:0/2269441423 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fef40065200 0x7fef400676b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:09.312 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:09.300+0000 7fef5ffff640 1 --2- 172.21.15.16:0/2269441423 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fef40065200 0x7fef400676b0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fef58009b40 tx=0x7fef58008040 comp rx=0 tx=0).ready entity=mgr.10542 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:09.312 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:09.304+0000 7fef5cff9640 1 -- 172.21.15.16:0/2269441423 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fef500ea400 con 0x7fef600a7430 2023-05-22T16:05:09.494 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:09.492+0000 7fef6635d640 1 -- 172.21.15.16:0/2269441423 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fef600b50e0 con 0x7fef600a7430 2023-05-22T16:05:09.824 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:09.820+0000 7fef5cff9640 1 -- 172.21.15.16:0/2269441423 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mgrmap(e 92) v1 ==== 84332+0+0 (secure 0 0 0) 0x7fef500ea7d0 con 0x7fef600a7430 2023-05-22T16:05:10.251 INFO:tasks.ceph.mon.a.smithi016.stderr:2023-05-22T16:05:10.248+0000 7f95dd48a640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-05-22T16:05:10.268 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:10.264+0000 7fef5cff9640 1 -- 172.21.15.16:0/2269441423 <== mon.0 v2:172.21.15.16:3300/0 8 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v74) v1 ==== 161+0+0 (secure 0 0 0) 0x7fef500ea400 con 0x7fef600a7430 2023-05-22T16:05:10.268 INFO:teuthology.orchestra.run.smithi016.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-05-22T16:05:10.270 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:10.268+0000 7fef6635d640 1 -- 172.21.15.16:0/2269441423 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fef40065200 msgr2=0x7fef400676b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:10.271 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:10.268+0000 7fef6635d640 1 --2- 172.21.15.16:0/2269441423 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fef40065200 0x7fef400676b0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fef58009b40 tx=0x7fef58008040 comp rx=0 tx=0).stop 2023-05-22T16:05:10.271 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:10.268+0000 7fef6635d640 1 -- 172.21.15.16:0/2269441423 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fef600a7430 msgr2=0x7fef600a78a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:10.271 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:10.268+0000 7fef6635d640 1 --2- 172.21.15.16:0/2269441423 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fef600a7430 0x7fef600a78a0 secure :-1 s=READY pgs=2379 cs=0 l=1 rev1=1 crypto rx=0x7fef500b0040 tx=0x7fef50009750 comp rx=0 tx=0).stop 2023-05-22T16:05:10.271 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:10.268+0000 7fef6635d640 1 -- 172.21.15.16:0/2269441423 shutdown_connections 2023-05-22T16:05:10.271 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:10.268+0000 7fef6635d640 1 --2- 172.21.15.16:0/2269441423 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fef40065200 0x7fef400676b0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:10.272 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:10.268+0000 7fef6635d640 1 --2- 172.21.15.16:0/2269441423 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fef600a7430 0x7fef600a78a0 unknown :-1 s=CLOSED pgs=2379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:10.272 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:10.268+0000 7fef6635d640 1 --2- 172.21.15.16:0/2269441423 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fef600c8490 0x7fef600c8900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:10.272 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:10.268+0000 7fef6635d640 1 --2- 172.21.15.16:0/2269441423 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fef4c004d10 0x7fef600a9240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:10.272 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:10.268+0000 7fef6635d640 1 -- 172.21.15.16:0/2269441423 >> 172.21.15.16:0/2269441423 conn(0x7fef600cc870 msgr2=0x7fef600cdc10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:05:10.272 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:10.268+0000 7fef6635d640 1 -- 172.21.15.16:0/2269441423 shutdown_connections 2023-05-22T16:05:10.272 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:10.268+0000 7fef6635d640 1 -- 172.21.15.16:0/2269441423 wait complete. 2023-05-22T16:05:10.294 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-05-22T16:05:10.340 INFO:tasks.ceph.mds.d.smithi062.stdout:starting mds.d at 2023-05-22T16:05:10.424 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:10.420+0000 7fa2f0251640 1 Processor -- start 2023-05-22T16:05:10.424 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:10.420+0000 7fa2f0251640 1 -- start start 2023-05-22T16:05:10.425 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:10.424+0000 7fa2f0251640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fa2e80d2480 0x7fa2e80d2900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:10.425 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:10.424+0000 7fa2f0251640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa2e80b5340 0x7fa2e80b5790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:10.425 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:10.424+0000 7fa2f0251640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2e80cccc0 con 0x7fa2e815bca0 2023-05-22T16:05:10.425 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:10.424+0000 7fa2f0251640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2e80cd660 con 0x7fa2e80d2480 2023-05-22T16:05:10.426 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:10.424+0000 7fa2f0251640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2e80ce190 con 0x7fa2e80b5340 2023-05-22T16:05:10.426 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:10.424+0000 7fa2ee7ee640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa2e80b5340 0x7fa2e80b5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:10.427 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:10.424+0000 7fa2edfed640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7fa2e815bca0 0x7fa2e80cda80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.16:37282/0 (socket says 172.21.15.16:37282) 2023-05-22T16:05:10.427 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:10.424+0000 7fa2ed7ec640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fa2e80d2480 0x7fa2e80d2900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:10.427 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:10.424+0000 7fa2edfed640 1 -- 172.21.15.16:0/1078669400 learned_addr learned my addr 172.21.15.16:0/1078669400 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T16:05:10.427 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:10.424+0000 7fa2ecfeb640 1 -- 172.21.15.16:0/1078669400 <== mon.1 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1191119133 0 0) 0x7fa2e80cccc0 con 0x7fa2e815bca0 2023-05-22T16:05:10.427 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:10.424+0000 7fa2ecfeb640 1 -- 172.21.15.16:0/1078669400 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa2d4003530 con 0x7fa2e815bca0 2023-05-22T16:05:10.428 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:10.424+0000 7fa2ee7ee640 1 -- 172.21.15.16:0/1078669400 >> v2:172.21.15.62:3300/0 conn(0x7fa2e80d2480 msgr2=0x7fa2e80d2900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:05:10.428 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:10.424+0000 7fa2ee7ee640 1 --2- 172.21.15.16:0/1078669400 >> v2:172.21.15.62:3300/0 conn(0x7fa2e80d2480 0x7fa2e80d2900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:10.428 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:10.424+0000 7fa2ee7ee640 1 -- 172.21.15.16:0/1078669400 >> v1:172.21.15.62:6789/0 conn(0x7fa2e815bca0 legacy=0x7fa2e80cda80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:10.428 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:10.424+0000 7fa2ee7ee640 1 -- 172.21.15.16:0/1078669400 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2e80ccfa0 con 0x7fa2e80b5340 2023-05-22T16:05:10.429 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:10.428+0000 7fa2ee7ee640 1 --2- 172.21.15.16:0/1078669400 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa2e80b5340 0x7fa2e80b5790 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fa2e0009d90 tx=0x7fa2e001c950 comp rx=0 tx=0).ready entity=mon.2 client_cookie=815963bda1e31d80 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:10.429 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:10.428+0000 7fa2ecfeb640 1 -- 172.21.15.16:0/1078669400 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2e00a5920 con 0x7fa2e80b5340 2023-05-22T16:05:10.429 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:10.428+0000 7fa2ecfeb640 1 -- 172.21.15.16:0/1078669400 <== mon.2 v2:172.21.15.62:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2e00a5a80 con 0x7fa2e80b5340 2023-05-22T16:05:10.429 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:10.428+0000 7fa2ecfeb640 1 -- 172.21.15.16:0/1078669400 <== mon.2 v2:172.21.15.62:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2e00a5df0 con 0x7fa2e80b5340 2023-05-22T16:05:10.434 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:10.428+0000 7fa2f0251640 1 -- 172.21.15.16:0/1078669400 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa2e80b5340 msgr2=0x7fa2e80b5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:10.434 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:10.428+0000 7fa2f0251640 1 --2- 172.21.15.16:0/1078669400 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa2e80b5340 0x7fa2e80b5790 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fa2e0009d90 tx=0x7fa2e001c950 comp rx=0 tx=0).stop 2023-05-22T16:05:10.434 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:10.428+0000 7fa2f0251640 1 -- 172.21.15.16:0/1078669400 shutdown_connections 2023-05-22T16:05:10.434 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:10.428+0000 7fa2f0251640 1 --2- 172.21.15.16:0/1078669400 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa2e80b5340 0x7fa2e80b5790 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:10.434 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:10.428+0000 7fa2f0251640 1 --2- 172.21.15.16:0/1078669400 >> v2:172.21.15.62:3300/0 conn(0x7fa2e80d2480 0x7fa2e80d2900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:10.434 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:10.428+0000 7fa2f0251640 1 -- 172.21.15.16:0/1078669400 >> 172.21.15.16:0/1078669400 conn(0x7fa2e80655c0 msgr2=0x7fa2e8070e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:05:10.435 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:10.428+0000 7fa2f0251640 1 -- 172.21.15.16:0/1078669400 shutdown_connections 2023-05-22T16:05:10.435 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:10.428+0000 7fa2f0251640 1 -- 172.21.15.16:0/1078669400 wait complete. 2023-05-22T16:05:10.435 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:10.428+0000 7fa2f0251640 1 Processor -- start 2023-05-22T16:05:10.435 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:10.428+0000 7fa2f0251640 1 -- start start 2023-05-22T16:05:10.435 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:10.428+0000 7fa2f0251640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa2e80b5340 0x7fa2e8159620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:10.435 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:10.428+0000 7fa2f0251640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa2e80d2480 0x7fa2e814ea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:10.435 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:10.428+0000 7fa2f0251640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa2e815bca0 0x7fa2e815cb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:10.435 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:10.428+0000 7fa2f0251640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2e8154e70 con 0x7fa2e80d2480 2023-05-22T16:05:10.436 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:10.428+0000 7fa2f0251640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2e814ef80 con 0x7fa2e815bca0 2023-05-22T16:05:10.436 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:10.428+0000 7fa2f0251640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2e815d070 con 0x7fa2e80b5340 2023-05-22T16:05:10.436 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:10.428+0000 7fa2ed7ec640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa2e80d2480 0x7fa2e814ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:10.436 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:10.428+0000 7fa2ed7ec640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa2e80d2480 0x7fa2e814ea40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39988/0 (socket says 172.21.15.16:39988) 2023-05-22T16:05:10.436 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:10.428+0000 7fa2ed7ec640 1 -- 172.21.15.16:0/1407044041 learned_addr learned my addr 172.21.15.16:0/1407044041 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:05:10.437 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:10.428+0000 7fa2edfed640 1 --2- 172.21.15.16:0/1407044041 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa2e80b5340 0x7fa2e8159620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:10.437 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:10.428+0000 7fa2ee7ee640 1 --2- 172.21.15.16:0/1407044041 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa2e815bca0 0x7fa2e815cb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:10.437 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:10.428+0000 7fa2ed7ec640 1 -- 172.21.15.16:0/1407044041 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa2e80b5340 msgr2=0x7fa2e8159620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:10.437 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:10.428+0000 7fa2ed7ec640 1 --2- 172.21.15.16:0/1407044041 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa2e80b5340 0x7fa2e8159620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:10.437 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:10.428+0000 7fa2ed7ec640 1 -- 172.21.15.16:0/1407044041 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa2e815bca0 msgr2=0x7fa2e815cb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:10.437 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:10.428+0000 7fa2ed7ec640 1 --2- 172.21.15.16:0/1407044041 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa2e815bca0 0x7fa2e815cb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:10.437 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:10.428+0000 7fa2ed7ec640 1 -- 172.21.15.16:0/1407044041 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2e0009a70 con 0x7fa2e80d2480 2023-05-22T16:05:10.438 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:10.428+0000 7fa2edfed640 1 --2- 172.21.15.16:0/1407044041 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa2e80b5340 0x7fa2e8159620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T16:05:10.438 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:10.428+0000 7fa2ed7ec640 1 --2- 172.21.15.16:0/1407044041 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa2e80d2480 0x7fa2e814ea40 secure :-1 s=READY pgs=2381 cs=0 l=1 rev1=1 crypto rx=0x7fa2d800e930 tx=0x7fa2d800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:10.438 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:10.432+0000 7fa2de7fc640 1 -- 172.21.15.16:0/1407044041 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2d8009690 con 0x7fa2e80d2480 2023-05-22T16:05:10.438 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:10.432+0000 7fa2de7fc640 1 -- 172.21.15.16:0/1407044041 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2d800ca00 con 0x7fa2e80d2480 2023-05-22T16:05:10.438 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:10.432+0000 7fa2f0251640 1 -- 172.21.15.16:0/1407044041 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2e8160360 con 0x7fa2e80d2480 2023-05-22T16:05:10.438 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:10.432+0000 7fa2de7fc640 1 -- 172.21.15.16:0/1407044041 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2d800cd70 con 0x7fa2e80d2480 2023-05-22T16:05:10.438 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:10.432+0000 7fa2f0251640 1 -- 172.21.15.16:0/1407044041 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2e80b5cd0 con 0x7fa2e80d2480 2023-05-22T16:05:10.438 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:10.432+0000 7fa2f0251640 1 -- 172.21.15.16:0/1407044041 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2e81604f0 con 0x7fa2e80d2480 2023-05-22T16:05:10.439 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:10.432+0000 7fa2de7fc640 1 -- 172.21.15.16:0/1407044041 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 92) v1 ==== 84332+0+0 (secure 0 0 0) 0x7fa2d8013470 con 0x7fa2e80d2480 2023-05-22T16:05:10.439 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:10.432+0000 7fa2de7fc640 1 --2- 172.21.15.16:0/1407044041 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fa2cc0654a0 0x7fa2cc067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:10.440 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:10.432+0000 7fa2de7fc640 1 -- 172.21.15.16:0/1407044041 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(184..184 src has 1..184) v4 ==== 7240+0+0 (secure 0 0 0) 0x7fa2d8083060 con 0x7fa2e80d2480 2023-05-22T16:05:10.440 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:10.432+0000 7fa2edfed640 1 --2- 172.21.15.16:0/1407044041 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fa2cc0654a0 0x7fa2cc067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:10.440 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:10.432+0000 7fa2edfed640 1 --2- 172.21.15.16:0/1407044041 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fa2cc0654a0 0x7fa2cc067950 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fa2e4001210 tx=0x7fa2e4009040 comp rx=0 tx=0).ready entity=mgr.10542 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:10.440 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:10.436+0000 7fa2de7fc640 1 -- 172.21.15.16:0/1407044041 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fa2d80507a0 con 0x7fa2e80d2480 2023-05-22T16:05:10.627 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:10.624+0000 7fa2f0251640 1 -- 172.21.15.16:0/1407044041 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fa2e815e2f0 con 0x7fa2e80d2480 2023-05-22T16:05:11.271 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:11.268+0000 7fa2de7fc640 1 -- 172.21.15.16:0/1407044041 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v75) v1 ==== 104+0+0 (secure 0 0 0) 0x7fa2e815e2f0 con 0x7fa2e80d2480 2023-05-22T16:05:11.274 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:11.272+0000 7fa2f0251640 1 -- 172.21.15.16:0/1407044041 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fa2cc0654a0 msgr2=0x7fa2cc067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:11.274 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:11.272+0000 7fa2f0251640 1 --2- 172.21.15.16:0/1407044041 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fa2cc0654a0 0x7fa2cc067950 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fa2e4001210 tx=0x7fa2e4009040 comp rx=0 tx=0).stop 2023-05-22T16:05:11.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:11.272+0000 7fa2f0251640 1 -- 172.21.15.16:0/1407044041 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa2e80d2480 msgr2=0x7fa2e814ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:11.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:11.272+0000 7fa2f0251640 1 --2- 172.21.15.16:0/1407044041 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa2e80d2480 0x7fa2e814ea40 secure :-1 s=READY pgs=2381 cs=0 l=1 rev1=1 crypto rx=0x7fa2d800e930 tx=0x7fa2d800edf0 comp rx=0 tx=0).stop 2023-05-22T16:05:11.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:11.272+0000 7fa2f0251640 1 -- 172.21.15.16:0/1407044041 shutdown_connections 2023-05-22T16:05:11.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:11.272+0000 7fa2f0251640 1 --2- 172.21.15.16:0/1407044041 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa2e815bca0 0x7fa2e815cb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:11.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:11.272+0000 7fa2f0251640 1 --2- 172.21.15.16:0/1407044041 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fa2cc0654a0 0x7fa2cc067950 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:11.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:11.272+0000 7fa2f0251640 1 --2- 172.21.15.16:0/1407044041 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa2e80d2480 0x7fa2e814ea40 unknown :-1 s=CLOSED pgs=2381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:11.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:11.272+0000 7fa2f0251640 1 --2- 172.21.15.16:0/1407044041 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa2e80b5340 0x7fa2e8159620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:11.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:11.272+0000 7fa2f0251640 1 -- 172.21.15.16:0/1407044041 >> 172.21.15.16:0/1407044041 conn(0x7fa2e80655c0 msgr2=0x7fa2e80cda80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:05:11.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:11.272+0000 7fa2f0251640 1 -- 172.21.15.16:0/1407044041 shutdown_connections 2023-05-22T16:05:11.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:11.272+0000 7fa2f0251640 1 -- 172.21.15.16:0/1407044041 wait complete. 2023-05-22T16:05:11.298 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-05-22T16:05:11.429 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:11.428+0000 7fe9484e7640 1 Processor -- start 2023-05-22T16:05:11.430 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:11.428+0000 7fe9484e7640 1 -- start start 2023-05-22T16:05:11.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:11.428+0000 7fe9484e7640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe94015c8e0 0x7fe9400ccc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:11.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:11.428+0000 7fe9484e7640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fe9400cef10 0x7fe9400cf370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:11.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:11.428+0000 7fe9484e7640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9400d2a20 con 0x7fe9400d2eb0 2023-05-22T16:05:11.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:11.428+0000 7fe9484e7640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9400cd230 con 0x7fe9400cef10 2023-05-22T16:05:11.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:11.428+0000 7fe9484e7640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9400d23d0 con 0x7fe94015c8e0 2023-05-22T16:05:11.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:11.428+0000 7fe946283640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe94015c8e0 0x7fe9400ccc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:11.432 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:11.428+0000 7fe946a84640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fe9400cef10 0x7fe9400cf370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:11.432 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:11.428+0000 7fe946283640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe94015c8e0 0x7fe9400ccc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3301/0 says I am v2:172.21.15.16:43946/0 (socket says 172.21.15.16:43946) 2023-05-22T16:05:11.432 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:11.428+0000 7fe946283640 1 -- 172.21.15.16:0/1973828414 learned_addr learned my addr 172.21.15.16:0/1973828414 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:05:11.432 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:11.428+0000 7fe945a82640 1 --1- 172.21.15.16:0/1973828414 >> v1:172.21.15.62:6789/0 conn(0x7fe9400d2eb0 0x7fe94015ccb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.16:37298/0 (socket says 172.21.15.16:37298) 2023-05-22T16:05:11.433 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:11.428+0000 7fe946283640 1 -- 172.21.15.16:0/1973828414 >> v2:172.21.15.62:3300/0 conn(0x7fe9400cef10 msgr2=0x7fe9400cf370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:05:11.433 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:11.428+0000 7fe946283640 1 --2- 172.21.15.16:0/1973828414 >> v2:172.21.15.62:3300/0 conn(0x7fe9400cef10 0x7fe9400cf370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:11.433 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:11.428+0000 7fe946283640 1 -- 172.21.15.16:0/1973828414 >> v1:172.21.15.62:6789/0 conn(0x7fe9400d2eb0 legacy=0x7fe94015ccb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:11.433 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:11.428+0000 7fe946283640 1 -- 172.21.15.16:0/1973828414 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9400d26b0 con 0x7fe94015c8e0 2023-05-22T16:05:11.434 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:11.432+0000 7fe946283640 1 --2- 172.21.15.16:0/1973828414 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe94015c8e0 0x7fe9400ccc50 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fe93c009870 tx=0x7fe93c01c470 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5ad49f950f569f server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:11.434 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:11.432+0000 7fe945281640 1 -- 172.21.15.16:0/1973828414 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe93c0ab650 con 0x7fe94015c8e0 2023-05-22T16:05:11.434 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:11.432+0000 7fe945281640 1 -- 172.21.15.16:0/1973828414 <== mon.2 v2:172.21.15.62:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe93c0ab7b0 con 0x7fe94015c8e0 2023-05-22T16:05:11.434 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:11.432+0000 7fe945281640 1 -- 172.21.15.16:0/1973828414 <== mon.2 v2:172.21.15.62:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe93c0abb40 con 0x7fe94015c8e0 2023-05-22T16:05:11.435 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:11.432+0000 7fe9484e7640 1 -- 172.21.15.16:0/1973828414 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe94015c8e0 msgr2=0x7fe9400ccc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:11.435 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:11.432+0000 7fe9484e7640 1 --2- 172.21.15.16:0/1973828414 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe94015c8e0 0x7fe9400ccc50 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fe93c009870 tx=0x7fe93c01c470 comp rx=0 tx=0).stop 2023-05-22T16:05:11.435 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:11.432+0000 7fe9484e7640 1 -- 172.21.15.16:0/1973828414 shutdown_connections 2023-05-22T16:05:11.436 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:11.432+0000 7fe9484e7640 1 --2- 172.21.15.16:0/1973828414 >> v2:172.21.15.62:3300/0 conn(0x7fe9400cef10 0x7fe9400cf370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:11.436 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:11.432+0000 7fe9484e7640 1 --2- 172.21.15.16:0/1973828414 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe94015c8e0 0x7fe9400ccc50 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:11.436 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:11.432+0000 7fe9484e7640 1 -- 172.21.15.16:0/1973828414 >> 172.21.15.16:0/1973828414 conn(0x7fe940070e50 msgr2=0x7fe940071250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:05:11.436 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:11.432+0000 7fe9484e7640 1 -- 172.21.15.16:0/1973828414 shutdown_connections 2023-05-22T16:05:11.436 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:11.432+0000 7fe9484e7640 1 -- 172.21.15.16:0/1973828414 wait complete. 2023-05-22T16:05:11.437 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:11.436+0000 7fe9484e7640 1 Processor -- start 2023-05-22T16:05:11.437 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:11.436+0000 7fe9484e7640 1 -- start start 2023-05-22T16:05:11.438 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:11.436+0000 7fe9484e7640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe9400cef10 0x7fe9400cbbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:11.438 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:11.436+0000 7fe9484e7640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe9400d2eb0 0x7fe940150350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:11.438 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:11.436+0000 7fe9484e7640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe94015c8e0 0x7fe940142b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:11.438 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:11.436+0000 7fe9484e7640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9401529c0 con 0x7fe9400cef10 2023-05-22T16:05:11.439 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:11.436+0000 7fe9484e7640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe940151db0 con 0x7fe9400d2eb0 2023-05-22T16:05:11.439 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:11.436+0000 7fe9484e7640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe940152090 con 0x7fe94015c8e0 2023-05-22T16:05:11.439 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:11.436+0000 7fe946283640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe9400cef10 0x7fe9400cbbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:11.439 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:11.436+0000 7fe946283640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe9400cef10 0x7fe9400cbbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39998/0 (socket says 172.21.15.16:39998) 2023-05-22T16:05:11.440 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:11.436+0000 7fe946283640 1 -- 172.21.15.16:0/2495651345 learned_addr learned my addr 172.21.15.16:0/2495651345 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:05:11.440 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:11.436+0000 7fe946a84640 1 --2- 172.21.15.16:0/2495651345 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe94015c8e0 0x7fe940142b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:11.440 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:11.436+0000 7fe945a82640 1 --2- 172.21.15.16:0/2495651345 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe9400d2eb0 0x7fe940150350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:11.440 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:11.436+0000 7fe946283640 1 -- 172.21.15.16:0/2495651345 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe94015c8e0 msgr2=0x7fe940142b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:11.440 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:11.436+0000 7fe946283640 1 --2- 172.21.15.16:0/2495651345 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe94015c8e0 0x7fe940142b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:11.441 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:11.436+0000 7fe946283640 1 -- 172.21.15.16:0/2495651345 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe9400d2eb0 msgr2=0x7fe940150350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:11.441 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:11.436+0000 7fe946283640 1 --2- 172.21.15.16:0/2495651345 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe9400d2eb0 0x7fe940150350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:11.441 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:11.436+0000 7fe946283640 1 -- 172.21.15.16:0/2495651345 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe93c0094d0 con 0x7fe9400cef10 2023-05-22T16:05:11.441 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:11.436+0000 7fe946283640 1 --2- 172.21.15.16:0/2495651345 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe9400cef10 0x7fe9400cbbc0 secure :-1 s=READY pgs=2382 cs=0 l=1 rev1=1 crypto rx=0x7fe93c0ad040 tx=0x7fe93c01acb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:11.442 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:11.436+0000 7fe936ffd640 1 -- 172.21.15.16:0/2495651345 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe93c0b4800 con 0x7fe9400cef10 2023-05-22T16:05:11.442 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:11.436+0000 7fe936ffd640 1 -- 172.21.15.16:0/2495651345 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe93c0b4980 con 0x7fe9400cef10 2023-05-22T16:05:11.442 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:11.436+0000 7fe9484e7640 1 -- 172.21.15.16:0/2495651345 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9400d3410 con 0x7fe9400cef10 2023-05-22T16:05:11.443 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:11.436+0000 7fe936ffd640 1 -- 172.21.15.16:0/2495651345 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe93c0b4d30 con 0x7fe9400cef10 2023-05-22T16:05:11.443 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:11.436+0000 7fe9484e7640 1 -- 172.21.15.16:0/2495651345 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9400d3280 con 0x7fe9400cef10 2023-05-22T16:05:11.443 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:11.436+0000 7fe9484e7640 1 -- 172.21.15.16:0/2495651345 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9400d0bf0 con 0x7fe9400cef10 2023-05-22T16:05:11.443 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:11.436+0000 7fe936ffd640 1 -- 172.21.15.16:0/2495651345 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 92) v1 ==== 84332+0+0 (secure 0 0 0) 0x7fe93c0b3380 con 0x7fe9400cef10 2023-05-22T16:05:11.443 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:11.440+0000 7fe936ffd640 1 --2- 172.21.15.16:0/2495651345 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fe9240654d0 0x7fe924067980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:11.444 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:11.440+0000 7fe936ffd640 1 -- 172.21.15.16:0/2495651345 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(185..185 src has 1..185) v4 ==== 7240+0+0 (secure 0 0 0) 0x7fe93c11d0f0 con 0x7fe9400cef10 2023-05-22T16:05:11.444 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:11.440+0000 7fe945a82640 1 --2- 172.21.15.16:0/2495651345 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fe9240654d0 0x7fe924067980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:11.444 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:11.440+0000 7fe945a82640 1 --2- 172.21.15.16:0/2495651345 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fe9240654d0 0x7fe924067980 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fe9400cc6d0 tx=0x7fe930009040 comp rx=0 tx=0).ready entity=mgr.10542 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:11.445 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:11.444+0000 7fe936ffd640 1 -- 172.21.15.16:0/2495651345 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fe93c0ea760 con 0x7fe9400cef10 2023-05-22T16:05:11.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:11.596+0000 7fe9484e7640 1 -- 172.21.15.16:0/2495651345 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fe94004dbe0 con 0x7fe9400cef10 2023-05-22T16:05:12.382 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:12.380+0000 7fe936ffd640 1 -- 172.21.15.16:0/2495651345 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v186) v1 ==== 181+0+0 (secure 0 0 0) 0x7fe93c0ab610 con 0x7fe9400cef10 2023-05-22T16:05:12.392 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'cephfs_metadata' removed 2023-05-22T16:05:12.392 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:12.388+0000 7fe9484e7640 1 -- 172.21.15.16:0/2495651345 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fe9240654d0 msgr2=0x7fe924067980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:12.392 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:12.388+0000 7fe9484e7640 1 --2- 172.21.15.16:0/2495651345 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fe9240654d0 0x7fe924067980 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fe9400cc6d0 tx=0x7fe930009040 comp rx=0 tx=0).stop 2023-05-22T16:05:12.392 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:12.388+0000 7fe9484e7640 1 -- 172.21.15.16:0/2495651345 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe9400cef10 msgr2=0x7fe9400cbbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:12.393 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:12.388+0000 7fe9484e7640 1 --2- 172.21.15.16:0/2495651345 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe9400cef10 0x7fe9400cbbc0 secure :-1 s=READY pgs=2382 cs=0 l=1 rev1=1 crypto rx=0x7fe93c0ad040 tx=0x7fe93c01acb0 comp rx=0 tx=0).stop 2023-05-22T16:05:12.393 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:12.388+0000 7fe9484e7640 1 -- 172.21.15.16:0/2495651345 shutdown_connections 2023-05-22T16:05:12.393 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:12.388+0000 7fe9484e7640 1 --2- 172.21.15.16:0/2495651345 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe94015c8e0 0x7fe940142b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:12.393 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:12.388+0000 7fe9484e7640 1 --2- 172.21.15.16:0/2495651345 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe9400d2eb0 0x7fe940150350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:12.393 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:12.388+0000 7fe9484e7640 1 --2- 172.21.15.16:0/2495651345 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fe9240654d0 0x7fe924067980 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:12.393 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:12.388+0000 7fe9484e7640 1 --2- 172.21.15.16:0/2495651345 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe9400cef10 0x7fe9400cbbc0 unknown :-1 s=CLOSED pgs=2382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:12.393 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:12.388+0000 7fe9484e7640 1 -- 172.21.15.16:0/2495651345 >> 172.21.15.16:0/2495651345 conn(0x7fe940070e50 msgr2=0x7fe94014bb50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:05:12.394 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:12.388+0000 7fe9484e7640 1 -- 172.21.15.16:0/2495651345 shutdown_connections 2023-05-22T16:05:12.394 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:12.388+0000 7fe9484e7640 1 -- 172.21.15.16:0/2495651345 wait complete. 2023-05-22T16:05:12.411 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-05-22T16:05:12.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:12.544+0000 7f9b9f837640 1 Processor -- start 2023-05-22T16:05:12.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:12.544+0000 7f9b9f837640 1 -- start start 2023-05-22T16:05:12.550 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:12.548+0000 7f9b9f837640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f9b980b7c90 0x7f9b980d2440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:12.550 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:12.548+0000 7f9b9f837640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9b980cf210 0x7f9b980d2980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:12.550 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:12.548+0000 7f9b9f837640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f9b980b46e0 0x7f9b980d2ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:12.551 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:12.548+0000 7f9b9f837640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b980cd2d0 con 0x7f9b980b46e0 2023-05-22T16:05:12.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:12.548+0000 7f9b9f837640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b980b8060 con 0x7f9b980b7c90 2023-05-22T16:05:12.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:12.548+0000 7f9b9f837640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b980cf680 con 0x7f9b980cf210 2023-05-22T16:05:12.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:12.548+0000 7f9b9ddd4640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f9b980b46e0 0x7f9b980d2ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:12.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:12.548+0000 7f9b9ddd4640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f9b980b46e0 0x7f9b980d2ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40012/0 (socket says 172.21.15.16:40012) 2023-05-22T16:05:12.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:12.548+0000 7f9b9ddd4640 1 -- 172.21.15.16:0/4201820556 learned_addr learned my addr 172.21.15.16:0/4201820556 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:05:12.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:12.548+0000 7f9b9d5d3640 1 --2- 172.21.15.16:0/4201820556 >> v2:172.21.15.62:3300/0 conn(0x7f9b980b7c90 0x7f9b980d2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:12.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:12.548+0000 7f9b9cdd2640 1 --2- 172.21.15.16:0/4201820556 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9b980cf210 0x7f9b980d2980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:12.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:12.548+0000 7f9b9ddd4640 1 -- 172.21.15.16:0/4201820556 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9b980cf210 msgr2=0x7f9b980d2980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:05:12.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:12.548+0000 7f9b9ddd4640 1 --2- 172.21.15.16:0/4201820556 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9b980cf210 0x7f9b980d2980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:12.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:12.548+0000 7f9b9ddd4640 1 -- 172.21.15.16:0/4201820556 >> v2:172.21.15.62:3300/0 conn(0x7f9b980b7c90 msgr2=0x7f9b980d2440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:05:12.554 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:12.548+0000 7f9b9ddd4640 1 --2- 172.21.15.16:0/4201820556 >> v2:172.21.15.62:3300/0 conn(0x7f9b980b7c90 0x7f9b980d2440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:12.554 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:12.548+0000 7f9b9ddd4640 1 -- 172.21.15.16:0/4201820556 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b980ccc80 con 0x7f9b980b46e0 2023-05-22T16:05:12.554 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:12.548+0000 7f9b9d5d3640 1 --2- 172.21.15.16:0/4201820556 >> v2:172.21.15.62:3300/0 conn(0x7f9b980b7c90 0x7f9b980d2440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T16:05:12.556 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:12.548+0000 7f9b9ddd4640 1 --2- 172.21.15.16:0/4201820556 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9b980b46e0 0x7f9b980d2ec0 secure :-1 s=READY pgs=2383 cs=0 l=1 rev1=1 crypto rx=0x7f9b98065dc0 tx=0x7f9b9001c9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=520f50fa8edef2ae server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:12.557 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:12.548+0000 7f9b8ffff640 1 -- 172.21.15.16:0/4201820556 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b900a5840 con 0x7f9b980b46e0 2023-05-22T16:05:12.557 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:12.548+0000 7f9b8ffff640 1 -- 172.21.15.16:0/4201820556 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b900a59a0 con 0x7f9b980b46e0 2023-05-22T16:05:12.557 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:12.548+0000 7f9b8ffff640 1 -- 172.21.15.16:0/4201820556 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b900a5d30 con 0x7f9b980b46e0 2023-05-22T16:05:12.557 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:12.552+0000 7f9b9f837640 1 -- 172.21.15.16:0/4201820556 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9b980b46e0 msgr2=0x7f9b980d2ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:12.557 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:12.552+0000 7f9b9f837640 1 --2- 172.21.15.16:0/4201820556 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9b980b46e0 0x7f9b980d2ec0 secure :-1 s=READY pgs=2383 cs=0 l=1 rev1=1 crypto rx=0x7f9b98065dc0 tx=0x7f9b9001c9a0 comp rx=0 tx=0).stop 2023-05-22T16:05:12.557 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:12.552+0000 7f9b9f837640 1 -- 172.21.15.16:0/4201820556 shutdown_connections 2023-05-22T16:05:12.558 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:12.552+0000 7f9b9f837640 1 --2- 172.21.15.16:0/4201820556 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9b980b46e0 0x7f9b980d2ec0 unknown :-1 s=CLOSED pgs=2383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:12.558 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:12.552+0000 7f9b9f837640 1 --2- 172.21.15.16:0/4201820556 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9b980cf210 0x7f9b980d2980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:12.558 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:12.552+0000 7f9b9f837640 1 --2- 172.21.15.16:0/4201820556 >> v2:172.21.15.62:3300/0 conn(0x7f9b980b7c90 0x7f9b980d2440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:12.558 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:12.552+0000 7f9b9f837640 1 -- 172.21.15.16:0/4201820556 >> 172.21.15.16:0/4201820556 conn(0x7f9b980655c0 msgr2=0x7f9b98070e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:05:12.558 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:12.552+0000 7f9b9f837640 1 -- 172.21.15.16:0/4201820556 shutdown_connections 2023-05-22T16:05:12.558 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:12.552+0000 7f9b9f837640 1 -- 172.21.15.16:0/4201820556 wait complete. 2023-05-22T16:05:12.558 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:12.552+0000 7f9b9f837640 1 Processor -- start 2023-05-22T16:05:12.559 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:12.552+0000 7f9b9f837640 1 -- start start 2023-05-22T16:05:12.559 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:12.552+0000 7f9b9f837640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9b980b46e0 0x7f9b98155dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:12.559 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:12.552+0000 7f9b9f837640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9b980b7c90 0x7f9b980c9580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:12.559 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:12.552+0000 7f9b9f837640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9b980cf210 0x7f9b980cbe60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:12.559 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:12.552+0000 7f9b9f837640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b98156300 con 0x7f9b980cf210 2023-05-22T16:05:12.559 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:12.552+0000 7f9b9f837640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b980cac10 con 0x7f9b980b46e0 2023-05-22T16:05:12.559 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:12.552+0000 7f9b9f837640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b980ca000 con 0x7f9b980b7c90 2023-05-22T16:05:12.560 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:12.552+0000 7f9b9ddd4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9b980cf210 0x7f9b980cbe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:12.560 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:12.552+0000 7f9b9ddd4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9b980cf210 0x7f9b980cbe60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40024/0 (socket says 172.21.15.16:40024) 2023-05-22T16:05:12.560 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:12.552+0000 7f9b9ddd4640 1 -- 172.21.15.16:0/3721280796 learned_addr learned my addr 172.21.15.16:0/3721280796 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:05:12.560 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:12.552+0000 7f9b9d5d3640 1 --2- 172.21.15.16:0/3721280796 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9b980b46e0 0x7f9b98155dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:12.560 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:12.552+0000 7f9b9cdd2640 1 --2- 172.21.15.16:0/3721280796 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9b980b7c90 0x7f9b980c9580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:12.560 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:12.552+0000 7f9b9ddd4640 1 -- 172.21.15.16:0/3721280796 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9b980b7c90 msgr2=0x7f9b980c9580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:12.561 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:12.552+0000 7f9b9ddd4640 1 --2- 172.21.15.16:0/3721280796 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9b980b7c90 0x7f9b980c9580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:12.561 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:12.552+0000 7f9b9ddd4640 1 -- 172.21.15.16:0/3721280796 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9b980b46e0 msgr2=0x7f9b98155dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:12.561 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:12.552+0000 7f9b9ddd4640 1 --2- 172.21.15.16:0/3721280796 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9b980b46e0 0x7f9b98155dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:12.561 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:12.552+0000 7f9b9ddd4640 1 -- 172.21.15.16:0/3721280796 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b90009b30 con 0x7f9b980cf210 2023-05-22T16:05:12.561 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:12.552+0000 7f9b9ddd4640 1 --2- 172.21.15.16:0/3721280796 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9b980cf210 0x7f9b980cbe60 secure :-1 s=READY pgs=2384 cs=0 l=1 rev1=1 crypto rx=0x7f9b9001b050 tx=0x7f9b90019740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:12.561 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:12.552+0000 7f9b8dffb640 1 -- 172.21.15.16:0/3721280796 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b900b8880 con 0x7f9b980cf210 2023-05-22T16:05:12.561 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:12.552+0000 7f9b8dffb640 1 -- 172.21.15.16:0/3721280796 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b900b8a00 con 0x7f9b980cf210 2023-05-22T16:05:12.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:12.552+0000 7f9b8dffb640 1 -- 172.21.15.16:0/3721280796 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b900b8db0 con 0x7f9b980cf210 2023-05-22T16:05:12.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:12.552+0000 7f9b9f837640 1 -- 172.21.15.16:0/3721280796 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b981601a0 con 0x7f9b980cf210 2023-05-22T16:05:12.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:12.552+0000 7f9b9f837640 1 -- 172.21.15.16:0/3721280796 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b98160010 con 0x7f9b980cf210 2023-05-22T16:05:12.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:12.556+0000 7f9b9f837640 1 -- 172.21.15.16:0/3721280796 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b98160330 con 0x7f9b980cf210 2023-05-22T16:05:12.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:12.556+0000 7f9b8dffb640 1 -- 172.21.15.16:0/3721280796 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 92) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f9b900b6440 con 0x7f9b980cf210 2023-05-22T16:05:12.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:12.556+0000 7f9b8dffb640 1 --2- 172.21.15.16:0/3721280796 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f9b7c0654a0 0x7f9b7c067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:12.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:12.556+0000 7f9b8dffb640 1 -- 172.21.15.16:0/3721280796 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(186..186 src has 1..186) v4 ==== 6774+0+0 (secure 0 0 0) 0x7f9b9011e6b0 con 0x7f9b980cf210 2023-05-22T16:05:12.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:12.556+0000 7f9b9d5d3640 1 --2- 172.21.15.16:0/3721280796 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f9b7c0654a0 0x7f9b7c067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:12.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:12.556+0000 7f9b9d5d3640 1 --2- 172.21.15.16:0/3721280796 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f9b7c0654a0 0x7f9b7c067950 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f9b94006b20 tx=0x7f9b94008040 comp rx=0 tx=0).ready entity=mgr.10542 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:12.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:12.560+0000 7f9b8dffb640 1 -- 172.21.15.16:0/3721280796 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f9b900ebf00 con 0x7f9b980cf210 2023-05-22T16:05:12.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:12.716+0000 7f9b9f837640 1 -- 172.21.15.16:0/3721280796 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f9b980cada0 con 0x7f9b980cf210 2023-05-22T16:05:13.399 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:13.396+0000 7f9b8dffb640 1 -- 172.21.15.16:0/3721280796 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v187) v1 ==== 169+0+0 (secure 0 0 0) 0x7f9b900ef520 con 0x7f9b980cf210 2023-05-22T16:05:13.401 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'cephfs_data' removed 2023-05-22T16:05:13.404 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:13.400+0000 7f9b9f837640 1 -- 172.21.15.16:0/3721280796 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f9b7c0654a0 msgr2=0x7f9b7c067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:13.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:13.400+0000 7f9b9f837640 1 --2- 172.21.15.16:0/3721280796 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f9b7c0654a0 0x7f9b7c067950 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f9b94006b20 tx=0x7f9b94008040 comp rx=0 tx=0).stop 2023-05-22T16:05:13.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:13.400+0000 7f9b9f837640 1 -- 172.21.15.16:0/3721280796 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9b980cf210 msgr2=0x7f9b980cbe60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:13.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:13.400+0000 7f9b9f837640 1 --2- 172.21.15.16:0/3721280796 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9b980cf210 0x7f9b980cbe60 secure :-1 s=READY pgs=2384 cs=0 l=1 rev1=1 crypto rx=0x7f9b9001b050 tx=0x7f9b90019740 comp rx=0 tx=0).stop 2023-05-22T16:05:13.409 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:13.404+0000 7f9b9f837640 1 -- 172.21.15.16:0/3721280796 shutdown_connections 2023-05-22T16:05:13.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:13.404+0000 7f9b9f837640 1 --2- 172.21.15.16:0/3721280796 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f9b7c0654a0 0x7f9b7c067950 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:13.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:13.404+0000 7f9b9f837640 1 --2- 172.21.15.16:0/3721280796 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9b980cf210 0x7f9b980cbe60 unknown :-1 s=CLOSED pgs=2384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:13.412 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:13.404+0000 7f9b9f837640 1 --2- 172.21.15.16:0/3721280796 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9b980b7c90 0x7f9b980c9580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:13.412 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:13.404+0000 7f9b9f837640 1 --2- 172.21.15.16:0/3721280796 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9b980b46e0 0x7f9b98155dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:13.412 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:13.404+0000 7f9b9f837640 1 -- 172.21.15.16:0/3721280796 >> 172.21.15.16:0/3721280796 conn(0x7f9b980655c0 msgr2=0x7f9b980d0360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:05:13.413 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:13.404+0000 7f9b9f837640 1 -- 172.21.15.16:0/3721280796 shutdown_connections 2023-05-22T16:05:13.413 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:13.404+0000 7f9b9f837640 1 -- 172.21.15.16:0/3721280796 wait complete. 2023-05-22T16:05:13.433 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2023-05-22T16:05:13.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:13.568+0000 7f95b3738640 1 Processor -- start 2023-05-22T16:05:13.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:13.568+0000 7f95b3738640 1 -- start start 2023-05-22T16:05:13.587 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:13.568+0000 7f95b3738640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f95ac159970 0x7f95ac159d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:13.587 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:13.568+0000 7f95b3738640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f95ac15d680 0x7f95ac0cb260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:13.587 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:13.568+0000 7f95b3738640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95ac070c30 con 0x7f95ac15dad0 2023-05-22T16:05:13.587 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:13.568+0000 7f95b3738640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95ac070e60 con 0x7f95ac159970 2023-05-22T16:05:13.587 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:13.568+0000 7f95b3738640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95ac071140 con 0x7f95ac15d680 2023-05-22T16:05:13.587 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:13.568+0000 7f95b1cd5640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f95ac15dad0 0x7f95ac06b6c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:44328/0 (socket says 172.21.15.16:44328) 2023-05-22T16:05:13.587 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:13.568+0000 7f95b1cd5640 1 -- 172.21.15.16:0/3543654787 learned_addr learned my addr 172.21.15.16:0/3543654787 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T16:05:13.588 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:13.568+0000 7f95b0cd3640 1 --2- 172.21.15.16:0/3543654787 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f95ac15d680 0x7f95ac0cb260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:13.588 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:13.568+0000 7f95b14d4640 1 --2- 172.21.15.16:0/3543654787 >> v2:172.21.15.62:3300/0 conn(0x7f95ac159970 0x7f95ac159d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:13.588 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:13.568+0000 7f959ffff640 1 -- 172.21.15.16:0/3543654787 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 935017443 0 0) 0x7f95ac070c30 con 0x7f95ac15dad0 2023-05-22T16:05:13.589 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:13.568+0000 7f959ffff640 1 -- 172.21.15.16:0/3543654787 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9598003530 con 0x7f95ac15dad0 2023-05-22T16:05:13.589 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:13.568+0000 7f959ffff640 1 -- 172.21.15.16:0/3543654787 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4124174614 0 0) 0x7f9598003530 con 0x7f95ac15dad0 2023-05-22T16:05:13.589 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:13.568+0000 7f959ffff640 1 -- 172.21.15.16:0/3543654787 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f95ac15d680 msgr2=0x7f95ac0cb260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:05:13.589 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:13.568+0000 7f959ffff640 1 --2- 172.21.15.16:0/3543654787 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f95ac15d680 0x7f95ac0cb260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:13.589 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:13.568+0000 7f959ffff640 1 -- 172.21.15.16:0/3543654787 >> v2:172.21.15.62:3300/0 conn(0x7f95ac159970 msgr2=0x7f95ac159d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:05:13.590 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:13.572+0000 7f959ffff640 1 --2- 172.21.15.16:0/3543654787 >> v2:172.21.15.62:3300/0 conn(0x7f95ac159970 0x7f95ac159d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:13.590 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:13.572+0000 7f959ffff640 1 -- 172.21.15.16:0/3543654787 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95ac071420 con 0x7f95ac15dad0 2023-05-22T16:05:13.590 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:13.572+0000 7f959ffff640 1 -- 172.21.15.16:0/3543654787 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7f95a4001680 con 0x7f95ac15dad0 2023-05-22T16:05:13.590 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:13.572+0000 7f95b14d4640 1 --2- 172.21.15.16:0/3543654787 >> v2:172.21.15.62:3300/0 conn(0x7f95ac159970 0x7f95ac159d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T16:05:13.590 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:13.572+0000 7f959ffff640 1 -- 172.21.15.16:0/3543654787 >> v1:172.21.15.16:6789/0 conn(0x7f95ac15dad0 legacy=0x7f95ac06b6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:13.590 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:13.572+0000 7f959ffff640 1 --2- 172.21.15.16:0/3543654787 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f95ac15d680 0x7f9598003980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:13.590 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:13.572+0000 7f959ffff640 1 --2- 172.21.15.16:0/3543654787 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9598003f00 0x7f95980047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:13.590 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:13.572+0000 7f959ffff640 1 --2- 172.21.15.16:0/3543654787 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9598004d10 0x7f9598005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:13.591 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:13.572+0000 7f959ffff640 1 -- 172.21.15.16:0/3543654787 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95ac070e60 con 0x7f9598003f00 2023-05-22T16:05:13.591 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:13.572+0000 7f959ffff640 1 -- 172.21.15.16:0/3543654787 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95ac071140 con 0x7f9598004d10 2023-05-22T16:05:13.591 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:13.572+0000 7f959ffff640 1 -- 172.21.15.16:0/3543654787 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f959800ba70 con 0x7f95ac15d680 2023-05-22T16:05:13.591 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:13.572+0000 7f95b0cd3640 1 --2- 172.21.15.16:0/3543654787 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9598003f00 0x7f95980047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:13.591 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:13.572+0000 7f95b14d4640 1 --2- 172.21.15.16:0/3543654787 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f95ac15d680 0x7f9598003980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:13.591 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:13.572+0000 7f95b1cd5640 1 --2- 172.21.15.16:0/3543654787 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9598004d10 0x7f9598005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:13.591 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:13.572+0000 7f95b0cd3640 1 -- 172.21.15.16:0/3543654787 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f95ac15d680 msgr2=0x7f9598003980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:05:13.592 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:13.572+0000 7f95b0cd3640 1 --2- 172.21.15.16:0/3543654787 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f95ac15d680 0x7f9598003980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:13.592 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:13.572+0000 7f95b0cd3640 1 -- 172.21.15.16:0/3543654787 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9598004d10 msgr2=0x7f9598005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:05:13.592 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:13.572+0000 7f95b0cd3640 1 --2- 172.21.15.16:0/3543654787 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9598004d10 0x7f9598005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:13.592 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:13.572+0000 7f95b0cd3640 1 -- 172.21.15.16:0/3543654787 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9598003530 con 0x7f9598003f00 2023-05-22T16:05:13.592 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:13.572+0000 7f95b1cd5640 1 --2- 172.21.15.16:0/3543654787 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9598004d10 0x7f9598005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T16:05:13.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:13.572+0000 7f95b0cd3640 1 --2- 172.21.15.16:0/3543654787 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9598003f00 0x7f95980047d0 secure :-1 s=READY pgs=2386 cs=0 l=1 rev1=1 crypto rx=0x7f95a8009f90 tx=0x7f95a801cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9955077c556b4850 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:13.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:13.572+0000 7f959ffff640 1 -- 172.21.15.16:0/3543654787 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95a80a5c00 con 0x7f9598003f00 2023-05-22T16:05:13.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:13.572+0000 7f959ffff640 1 -- 172.21.15.16:0/3543654787 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95a800f040 con 0x7f9598003f00 2023-05-22T16:05:13.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:13.572+0000 7f959ffff640 1 -- 172.21.15.16:0/3543654787 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95a80ad440 con 0x7f9598003f00 2023-05-22T16:05:13.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:13.572+0000 7f95b3738640 1 -- 172.21.15.16:0/3543654787 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9598003f00 msgr2=0x7f95980047d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:13.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:13.572+0000 7f95b3738640 1 --2- 172.21.15.16:0/3543654787 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9598003f00 0x7f95980047d0 secure :-1 s=READY pgs=2386 cs=0 l=1 rev1=1 crypto rx=0x7f95a8009f90 tx=0x7f95a801cbf0 comp rx=0 tx=0).stop 2023-05-22T16:05:13.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:13.572+0000 7f95b1cd5640 1 -- 172.21.15.16:0/3543654787 reap_dead start 2023-05-22T16:05:13.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:13.572+0000 7f95b3738640 1 -- 172.21.15.16:0/3543654787 shutdown_connections 2023-05-22T16:05:13.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:13.572+0000 7f95b3738640 1 -- 172.21.15.16:0/3543654787 >> 172.21.15.16:0/3543654787 conn(0x7f95ac069720 msgr2=0x7f95ac069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:05:13.600 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:13.572+0000 7f95b3738640 1 -- 172.21.15.16:0/3543654787 shutdown_connections 2023-05-22T16:05:13.600 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:13.572+0000 7f95b3738640 1 -- 172.21.15.16:0/3543654787 wait complete. 2023-05-22T16:05:13.600 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:13.576+0000 7f95b3738640 1 Processor -- start 2023-05-22T16:05:13.600 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:13.576+0000 7f95b3738640 1 -- start start 2023-05-22T16:05:13.600 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:13.576+0000 7f95b3738640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9598003f00 0x7f95ac1587a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:13.601 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:13.576+0000 7f95b3738640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f95ac15c5f0 0x7f95ac151660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:13.601 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:13.576+0000 7f95b3738640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f95ac1552a0 0x7f95ac0ae5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:13.601 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:13.576+0000 7f95b3738640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95ac158ce0 con 0x7f95ac1552a0 2023-05-22T16:05:13.601 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:13.576+0000 7f95b3738640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95ac0b2200 con 0x7f9598003f00 2023-05-22T16:05:13.601 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:13.576+0000 7f95b3738640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95ac0b24e0 con 0x7f95ac15c5f0 2023-05-22T16:05:13.602 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:13.576+0000 7f95b1cd5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f95ac1552a0 0x7f95ac0ae5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:13.602 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:13.576+0000 7f95b1cd5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f95ac1552a0 0x7f95ac0ae5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40042/0 (socket says 172.21.15.16:40042) 2023-05-22T16:05:13.602 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:13.576+0000 7f95b1cd5640 1 -- 172.21.15.16:0/3747416979 learned_addr learned my addr 172.21.15.16:0/3747416979 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:05:13.602 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:13.576+0000 7f95b14d4640 1 --2- 172.21.15.16:0/3747416979 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9598003f00 0x7f95ac1587a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:13.602 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:13.576+0000 7f95b0cd3640 1 --2- 172.21.15.16:0/3747416979 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f95ac15c5f0 0x7f95ac151660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:13.602 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:13.576+0000 7f95b1cd5640 1 -- 172.21.15.16:0/3747416979 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f95ac15c5f0 msgr2=0x7f95ac151660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:13.602 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:13.576+0000 7f95b1cd5640 1 --2- 172.21.15.16:0/3747416979 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f95ac15c5f0 0x7f95ac151660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:13.603 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:13.576+0000 7f95b1cd5640 1 -- 172.21.15.16:0/3747416979 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9598003f00 msgr2=0x7f95ac1587a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:13.603 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:13.576+0000 7f95b1cd5640 1 --2- 172.21.15.16:0/3747416979 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9598003f00 0x7f95ac1587a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:13.603 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:13.576+0000 7f95b1cd5640 1 -- 172.21.15.16:0/3747416979 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95ac06bf60 con 0x7f95ac1552a0 2023-05-22T16:05:13.604 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:13.576+0000 7f95b1cd5640 1 --2- 172.21.15.16:0/3747416979 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f95ac1552a0 0x7f95ac0ae5e0 secure :-1 s=READY pgs=2387 cs=0 l=1 rev1=1 crypto rx=0x7f95a4000fc0 tx=0x7f95a400d580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:13.604 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:13.576+0000 7f959dffb640 1 -- 172.21.15.16:0/3747416979 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95a400ddc0 con 0x7f95ac1552a0 2023-05-22T16:05:13.604 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:13.576+0000 7f959dffb640 1 -- 172.21.15.16:0/3747416979 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95a4007030 con 0x7f95ac1552a0 2023-05-22T16:05:13.604 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:13.576+0000 7f959dffb640 1 -- 172.21.15.16:0/3747416979 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95a40073e0 con 0x7f95ac1552a0 2023-05-22T16:05:13.604 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:13.576+0000 7f95b3738640 1 -- 172.21.15.16:0/3747416979 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95ac06bdd0 con 0x7f95ac1552a0 2023-05-22T16:05:13.604 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:13.576+0000 7f95b3738640 1 -- 172.21.15.16:0/3747416979 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95ac15a5b0 con 0x7f95ac1552a0 2023-05-22T16:05:13.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:13.576+0000 7f95b3738640 1 -- 172.21.15.16:0/3747416979 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95ac04dbe0 con 0x7f95ac1552a0 2023-05-22T16:05:13.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:13.580+0000 7f959dffb640 1 -- 172.21.15.16:0/3747416979 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 92) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f95a4007540 con 0x7f95ac1552a0 2023-05-22T16:05:13.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:13.580+0000 7f959dffb640 1 --2- 172.21.15.16:0/3747416979 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f95900654a0 0x7f9590067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:13.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:13.580+0000 7f959dffb640 1 -- 172.21.15.16:0/3747416979 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(187..187 src has 1..187) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f95a4081cd0 con 0x7f95ac1552a0 2023-05-22T16:05:13.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:13.580+0000 7f95b14d4640 1 --2- 172.21.15.16:0/3747416979 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f95900654a0 0x7f9590067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:13.606 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:13.580+0000 7f95b14d4640 1 --2- 172.21.15.16:0/3747416979 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f95900654a0 0x7f9590067950 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f95a00098b0 tx=0x7f95a0008040 comp rx=0 tx=0).ready entity=mgr.10542 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:13.606 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:13.584+0000 7f959dffb640 1 -- 172.21.15.16:0/3747416979 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f95a404f4d0 con 0x7f95ac1552a0 2023-05-22T16:05:13.741 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:13.740+0000 7f95b3738640 1 -- 172.21.15.16:0/3747416979 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7f95ac06ba00 con 0x7f95ac1552a0 2023-05-22T16:05:13.747 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:13.744+0000 7f959dffb640 1 -- 172.21.15.16:0/3747416979 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v61) v1 ==== 120+0+0 (secure 0 0 0) 0x7f95a401a070 con 0x7f95ac1552a0 2023-05-22T16:05:13.751 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:13.748+0000 7f95b3738640 1 -- 172.21.15.16:0/3747416979 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f95900654a0 msgr2=0x7f9590067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:13.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:13.748+0000 7f95b3738640 1 --2- 172.21.15.16:0/3747416979 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f95900654a0 0x7f9590067950 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f95a00098b0 tx=0x7f95a0008040 comp rx=0 tx=0).stop 2023-05-22T16:05:13.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:13.748+0000 7f95b3738640 1 -- 172.21.15.16:0/3747416979 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f95ac1552a0 msgr2=0x7f95ac0ae5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:13.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:13.748+0000 7f95b3738640 1 --2- 172.21.15.16:0/3747416979 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f95ac1552a0 0x7f95ac0ae5e0 secure :-1 s=READY pgs=2387 cs=0 l=1 rev1=1 crypto rx=0x7f95a4000fc0 tx=0x7f95a400d580 comp rx=0 tx=0).stop 2023-05-22T16:05:13.753 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:13.748+0000 7f95b3738640 1 -- 172.21.15.16:0/3747416979 shutdown_connections 2023-05-22T16:05:13.753 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:13.748+0000 7f95b3738640 1 --2- 172.21.15.16:0/3747416979 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f95900654a0 0x7f9590067950 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:13.753 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:13.748+0000 7f95b3738640 1 --2- 172.21.15.16:0/3747416979 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f95ac1552a0 0x7f95ac0ae5e0 unknown :-1 s=CLOSED pgs=2387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:13.753 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:13.748+0000 7f95b3738640 1 --2- 172.21.15.16:0/3747416979 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f95ac15c5f0 0x7f95ac151660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:13.754 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:13.748+0000 7f95b3738640 1 --2- 172.21.15.16:0/3747416979 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9598003f00 0x7f95ac1587a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:13.754 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:13.748+0000 7f95b3738640 1 -- 172.21.15.16:0/3747416979 >> 172.21.15.16:0/3747416979 conn(0x7f95ac069720 msgr2=0x7f95ac153540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:05:13.754 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:13.748+0000 7f95b3738640 1 -- 172.21.15.16:0/3747416979 shutdown_connections 2023-05-22T16:05:13.754 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:13.748+0000 7f95b3738640 1 -- 172.21.15.16:0/3747416979 wait complete. 2023-05-22T16:05:13.769 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2023-05-22T16:05:13.891 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:13.888+0000 7f789245e640 1 Processor -- start 2023-05-22T16:05:13.892 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:13.888+0000 7f789245e640 1 -- start start 2023-05-22T16:05:13.892 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:13.892+0000 7f789245e640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f788c159970 0x7f788c159d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:13.893 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:13.892+0000 7f789245e640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f788c15d680 0x7f788c0cb260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:13.893 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:13.892+0000 7f789245e640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f788c070c30 con 0x7f788c15dad0 2023-05-22T16:05:13.893 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:13.892+0000 7f789245e640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f788c070e60 con 0x7f788c159970 2023-05-22T16:05:13.893 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:13.892+0000 7f789245e640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f788c071140 con 0x7f788c15d680 2023-05-22T16:05:13.893 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:13.892+0000 7f78909fb640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f788c15dad0 0x7f788c06b6c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:44342/0 (socket says 172.21.15.16:44342) 2023-05-22T16:05:13.893 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:13.892+0000 7f78909fb640 1 -- 172.21.15.16:0/3586573663 learned_addr learned my addr 172.21.15.16:0/3586573663 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T16:05:13.895 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:13.892+0000 7f788b7fe640 1 --2- 172.21.15.16:0/3586573663 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f788c15d680 0x7f788c0cb260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:13.895 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:13.892+0000 7f788bfff640 1 --2- 172.21.15.16:0/3586573663 >> v2:172.21.15.62:3300/0 conn(0x7f788c159970 0x7f788c159d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:13.895 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:13.892+0000 7f788affd640 1 -- 172.21.15.16:0/3586573663 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2071772766 0 0) 0x7f788c070c30 con 0x7f788c15dad0 2023-05-22T16:05:13.895 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:13.892+0000 7f788affd640 1 -- 172.21.15.16:0/3586573663 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7874003530 con 0x7f788c15dad0 2023-05-22T16:05:13.895 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:13.892+0000 7f788affd640 1 -- 172.21.15.16:0/3586573663 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2748436093 0 0) 0x7f7874003530 con 0x7f788c15dad0 2023-05-22T16:05:13.896 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:13.892+0000 7f788affd640 1 -- 172.21.15.16:0/3586573663 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f788c15d680 msgr2=0x7f788c0cb260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:05:13.896 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:13.892+0000 7f788affd640 1 --2- 172.21.15.16:0/3586573663 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f788c15d680 0x7f788c0cb260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:13.896 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:13.892+0000 7f788affd640 1 -- 172.21.15.16:0/3586573663 >> v2:172.21.15.62:3300/0 conn(0x7f788c159970 msgr2=0x7f788c159d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:05:13.896 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:13.892+0000 7f788affd640 1 --2- 172.21.15.16:0/3586573663 >> v2:172.21.15.62:3300/0 conn(0x7f788c159970 0x7f788c159d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:13.896 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:13.892+0000 7f788affd640 1 -- 172.21.15.16:0/3586573663 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f788c071420 con 0x7f788c15dad0 2023-05-22T16:05:13.896 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:13.892+0000 7f788affd640 1 -- 172.21.15.16:0/3586573663 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7f7884001a00 con 0x7f788c15dad0 2023-05-22T16:05:13.896 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:13.892+0000 7f788affd640 1 -- 172.21.15.16:0/3586573663 >> v1:172.21.15.16:6789/0 conn(0x7f788c15dad0 legacy=0x7f788c06b6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:13.897 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:13.892+0000 7f788affd640 1 --2- 172.21.15.16:0/3586573663 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7874003980 0x7f7874003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:13.897 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:13.892+0000 7f788affd640 1 --2- 172.21.15.16:0/3586573663 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f788c15d680 0x7f78740047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:13.897 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:13.892+0000 7f788affd640 1 --2- 172.21.15.16:0/3586573663 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7874004d10 0x7f7874005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:13.897 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:13.892+0000 7f788affd640 1 -- 172.21.15.16:0/3586573663 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f788c070e60 con 0x7f7874004d10 2023-05-22T16:05:13.897 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:13.892+0000 7f788affd640 1 -- 172.21.15.16:0/3586573663 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f788c071140 con 0x7f7874003980 2023-05-22T16:05:13.897 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:13.892+0000 7f788affd640 1 -- 172.21.15.16:0/3586573663 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f787400b990 con 0x7f788c15d680 2023-05-22T16:05:13.897 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:13.892+0000 7f78909fb640 1 --2- 172.21.15.16:0/3586573663 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7874004d10 0x7f7874005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:13.898 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:13.892+0000 7f788bfff640 1 --2- 172.21.15.16:0/3586573663 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7874003980 0x7f7874003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:13.898 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:13.892+0000 7f788b7fe640 1 --2- 172.21.15.16:0/3586573663 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f788c15d680 0x7f78740047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:13.898 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:13.892+0000 7f78909fb640 1 -- 172.21.15.16:0/3586573663 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f788c15d680 msgr2=0x7f78740047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:05:13.898 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:13.892+0000 7f78909fb640 1 --2- 172.21.15.16:0/3586573663 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f788c15d680 0x7f78740047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:13.898 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:13.892+0000 7f78909fb640 1 -- 172.21.15.16:0/3586573663 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7874003980 msgr2=0x7f7874003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:05:13.898 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:13.892+0000 7f78909fb640 1 --2- 172.21.15.16:0/3586573663 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7874003980 0x7f7874003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:13.899 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:13.892+0000 7f78909fb640 1 -- 172.21.15.16:0/3586573663 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7874003530 con 0x7f7874004d10 2023-05-22T16:05:13.901 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:13.896+0000 7f78909fb640 1 --2- 172.21.15.16:0/3586573663 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7874004d10 0x7f7874005180 secure :-1 s=READY pgs=2389 cs=0 l=1 rev1=1 crypto rx=0x7f788c15a2a0 tx=0x7f788401fa60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78d72e5f0aefc8a8 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:13.901 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:13.896+0000 7f788affd640 1 -- 172.21.15.16:0/3586573663 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78840aa690 con 0x7f7874004d10 2023-05-22T16:05:13.902 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:13.896+0000 7f788affd640 1 -- 172.21.15.16:0/3586573663 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78840aa7f0 con 0x7f7874004d10 2023-05-22T16:05:13.902 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:13.896+0000 7f788affd640 1 -- 172.21.15.16:0/3586573663 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78840aabc0 con 0x7f7874004d10 2023-05-22T16:05:13.902 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:13.896+0000 7f789245e640 1 -- 172.21.15.16:0/3586573663 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7874004d10 msgr2=0x7f7874005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:13.902 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:13.896+0000 7f789245e640 1 --2- 172.21.15.16:0/3586573663 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7874004d10 0x7f7874005180 secure :-1 s=READY pgs=2389 cs=0 l=1 rev1=1 crypto rx=0x7f788c15a2a0 tx=0x7f788401fa60 comp rx=0 tx=0).stop 2023-05-22T16:05:13.902 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:13.896+0000 7f78909fb640 1 -- 172.21.15.16:0/3586573663 reap_dead start 2023-05-22T16:05:13.902 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:13.896+0000 7f789245e640 1 -- 172.21.15.16:0/3586573663 shutdown_connections 2023-05-22T16:05:13.903 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:13.896+0000 7f789245e640 1 -- 172.21.15.16:0/3586573663 >> 172.21.15.16:0/3586573663 conn(0x7f788c069720 msgr2=0x7f788c069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:05:13.903 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:13.896+0000 7f789245e640 1 -- 172.21.15.16:0/3586573663 shutdown_connections 2023-05-22T16:05:13.903 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:13.896+0000 7f789245e640 1 -- 172.21.15.16:0/3586573663 wait complete. 2023-05-22T16:05:13.903 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:13.896+0000 7f789245e640 1 Processor -- start 2023-05-22T16:05:13.903 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:13.896+0000 7f789245e640 1 -- start start 2023-05-22T16:05:13.903 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:13.896+0000 7f789245e640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f788c154a50 0x7f788c0c8390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:13.903 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:13.896+0000 7f789245e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f788c1f8b40 0x7f788c1f8fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:13.904 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:13.896+0000 7f789245e640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f788c0c65c0 0x7f788c0c6a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:13.904 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:13.896+0000 7f789245e640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f788c1f94f0 con 0x7f788c1f8b40 2023-05-22T16:05:13.904 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:13.896+0000 7f789245e640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f788c0c72f0 con 0x7f788c154a50 2023-05-22T16:05:13.905 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:13.896+0000 7f789245e640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f788c0c75d0 con 0x7f788c0c65c0 2023-05-22T16:05:13.905 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:13.896+0000 7f788b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f788c1f8b40 0x7f788c1f8fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:13.905 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:13.896+0000 7f788b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f788c1f8b40 0x7f788c1f8fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40046/0 (socket says 172.21.15.16:40046) 2023-05-22T16:05:13.905 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:13.896+0000 7f788b7fe640 1 -- 172.21.15.16:0/1875302327 learned_addr learned my addr 172.21.15.16:0/1875302327 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:05:13.905 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:13.896+0000 7f788bfff640 1 --2- 172.21.15.16:0/1875302327 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f788c154a50 0x7f788c0c8390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:13.905 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:13.896+0000 7f78909fb640 1 --2- 172.21.15.16:0/1875302327 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f788c0c65c0 0x7f788c0c6a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:13.906 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:13.896+0000 7f788b7fe640 1 -- 172.21.15.16:0/1875302327 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f788c0c65c0 msgr2=0x7f788c0c6a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:13.906 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:13.896+0000 7f788b7fe640 1 --2- 172.21.15.16:0/1875302327 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f788c0c65c0 0x7f788c0c6a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:13.906 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:13.896+0000 7f788b7fe640 1 -- 172.21.15.16:0/1875302327 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f788c154a50 msgr2=0x7f788c0c8390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:13.907 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:13.896+0000 7f788b7fe640 1 --2- 172.21.15.16:0/1875302327 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f788c154a50 0x7f788c0c8390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:13.907 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:13.896+0000 7f788b7fe640 1 -- 172.21.15.16:0/1875302327 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f788c0cac90 con 0x7f788c1f8b40 2023-05-22T16:05:13.907 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:13.896+0000 7f788b7fe640 1 --2- 172.21.15.16:0/1875302327 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f788c1f8b40 0x7f788c1f8fb0 secure :-1 s=READY pgs=2390 cs=0 l=1 rev1=1 crypto rx=0x7f78800119d0 tx=0x7f7880011e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:13.907 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:13.900+0000 7f7888ff9640 1 -- 172.21.15.16:0/1875302327 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f788000fb50 con 0x7f788c1f8b40 2023-05-22T16:05:13.907 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:13.900+0000 7f7888ff9640 1 -- 172.21.15.16:0/1875302327 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f788000fcb0 con 0x7f788c1f8b40 2023-05-22T16:05:13.908 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:13.900+0000 7f789245e640 1 -- 172.21.15.16:0/1875302327 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f788c071700 con 0x7f788c1f8b40 2023-05-22T16:05:13.908 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:13.900+0000 7f7888ff9640 1 -- 172.21.15.16:0/1875302327 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7880014440 con 0x7f788c1f8b40 2023-05-22T16:05:13.908 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:13.900+0000 7f789245e640 1 -- 172.21.15.16:0/1875302327 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f788c0cae20 con 0x7f788c1f8b40 2023-05-22T16:05:13.908 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:13.900+0000 7f789245e640 1 -- 172.21.15.16:0/1875302327 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f788c04dbe0 con 0x7f788c1f8b40 2023-05-22T16:05:13.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:13.900+0000 7f7888ff9640 1 -- 172.21.15.16:0/1875302327 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 92) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f78800145a0 con 0x7f788c1f8b40 2023-05-22T16:05:13.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:13.900+0000 7f7888ff9640 1 --2- 172.21.15.16:0/1875302327 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f78780654d0 0x7f7878067980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:13.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:13.900+0000 7f7888ff9640 1 -- 172.21.15.16:0/1875302327 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(187..187 src has 1..187) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f7880083330 con 0x7f788c1f8b40 2023-05-22T16:05:13.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:13.900+0000 7f788bfff640 1 --2- 172.21.15.16:0/1875302327 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f78780654d0 0x7f7878067980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:13.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:13.900+0000 7f788bfff640 1 --2- 172.21.15.16:0/1875302327 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f78780654d0 0x7f7878067980 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f787c009980 tx=0x7f787c008040 comp rx=0 tx=0).ready entity=mgr.10542 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:13.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:13.904+0000 7f7888ff9640 1 -- 172.21.15.16:0/1875302327 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f7880054860 con 0x7f788c1f8b40 2023-05-22T16:05:14.063 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:14.060+0000 7f789245e640 1 -- 172.21.15.16:0/1875302327 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f788c0c9130 con 0x7f788c1f8b40 2023-05-22T16:05:14.070 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:14.068+0000 7f7888ff9640 1 -- 172.21.15.16:0/1875302327 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v62) v1 ==== 104+0+0 (secure 0 0 0) 0x7f78800543f0 con 0x7f788c1f8b40 2023-05-22T16:05:14.073 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:14.068+0000 7f789245e640 1 -- 172.21.15.16:0/1875302327 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f78780654d0 msgr2=0x7f7878067980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:14.073 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:14.068+0000 7f789245e640 1 --2- 172.21.15.16:0/1875302327 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f78780654d0 0x7f7878067980 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f787c009980 tx=0x7f787c008040 comp rx=0 tx=0).stop 2023-05-22T16:05:14.073 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:14.068+0000 7f789245e640 1 -- 172.21.15.16:0/1875302327 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f788c1f8b40 msgr2=0x7f788c1f8fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:14.073 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:14.068+0000 7f789245e640 1 --2- 172.21.15.16:0/1875302327 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f788c1f8b40 0x7f788c1f8fb0 secure :-1 s=READY pgs=2390 cs=0 l=1 rev1=1 crypto rx=0x7f78800119d0 tx=0x7f7880011e90 comp rx=0 tx=0).stop 2023-05-22T16:05:14.073 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:14.068+0000 7f789245e640 1 -- 172.21.15.16:0/1875302327 shutdown_connections 2023-05-22T16:05:14.073 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:14.068+0000 7f789245e640 1 --2- 172.21.15.16:0/1875302327 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f788c0c65c0 0x7f788c0c6a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:14.074 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:14.068+0000 7f789245e640 1 --2- 172.21.15.16:0/1875302327 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f78780654d0 0x7f7878067980 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:14.074 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:14.068+0000 7f789245e640 1 --2- 172.21.15.16:0/1875302327 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f788c1f8b40 0x7f788c1f8fb0 unknown :-1 s=CLOSED pgs=2390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:14.074 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:14.068+0000 7f789245e640 1 --2- 172.21.15.16:0/1875302327 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f788c154a50 0x7f788c0c8390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:14.074 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:14.068+0000 7f789245e640 1 -- 172.21.15.16:0/1875302327 >> 172.21.15.16:0/1875302327 conn(0x7f788c069720 msgr2=0x7f788c0cafe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:05:14.074 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:14.072+0000 7f789245e640 1 -- 172.21.15.16:0/1875302327 shutdown_connections 2023-05-22T16:05:14.074 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:14.072+0000 7f789245e640 1 -- 172.21.15.16:0/1875302327 wait complete. 2023-05-22T16:05:14.090 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2023-05-22T16:05:14.216 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:14.212+0000 7effb6ba2640 1 Processor -- start 2023-05-22T16:05:14.217 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:14.212+0000 7effb6ba2640 1 -- start start 2023-05-22T16:05:14.218 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:14.216+0000 7effb6ba2640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7effb0159d40 0x7effb00cb260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:14.218 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:14.216+0000 7effb6ba2640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7effb006b6c0 0x7effb006bb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:14.218 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:14.216+0000 7effb6ba2640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effb0070c30 con 0x7effb0159970 2023-05-22T16:05:14.218 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:14.216+0000 7effb6ba2640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7effb015dd90 con 0x7effb006b6c0 2023-05-22T16:05:14.219 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:14.216+0000 7effb6ba2640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effb015a190 con 0x7effb0159d40 2023-05-22T16:05:14.219 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:14.216+0000 7effb493e640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7effb0159970 0x7effb015d680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:44350/0 (socket says 172.21.15.16:44350) 2023-05-22T16:05:14.219 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:14.216+0000 7effb493e640 1 -- 172.21.15.16:0/3381307018 learned_addr learned my addr 172.21.15.16:0/3381307018 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T16:05:14.219 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:14.216+0000 7effb513f640 1 --2- 172.21.15.16:0/3381307018 >> v2:172.21.15.16:3300/0 conn(0x7effb006b6c0 0x7effb006bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:14.219 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:14.216+0000 7effaf7fe640 1 -- 172.21.15.16:0/3381307018 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1620035910 0 0) 0x7effb0070c30 con 0x7effb0159970 2023-05-22T16:05:14.219 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:14.216+0000 7effaffff640 1 --2- 172.21.15.16:0/3381307018 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7effb0159d40 0x7effb00cb260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:14.219 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:14.216+0000 7effaf7fe640 1 -- 172.21.15.16:0/3381307018 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7effa0003530 con 0x7effb0159970 2023-05-22T16:05:14.220 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:14.216+0000 7effb513f640 1 -- 172.21.15.16:0/3381307018 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7effb0159d40 msgr2=0x7effb00cb260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:05:14.220 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:14.216+0000 7effb513f640 1 --2- 172.21.15.16:0/3381307018 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7effb0159d40 0x7effb00cb260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:14.220 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:14.216+0000 7effb513f640 1 -- 172.21.15.16:0/3381307018 >> v1:172.21.15.16:6789/0 conn(0x7effb0159970 legacy=0x7effb015d680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:14.220 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:14.216+0000 7effb513f640 1 -- 172.21.15.16:0/3381307018 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effb006c060 con 0x7effb006b6c0 2023-05-22T16:05:14.221 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:14.216+0000 7effb513f640 1 --2- 172.21.15.16:0/3381307018 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7effb006b6c0 0x7effb006bb20 secure :-1 s=READY pgs=2392 cs=0 l=1 rev1=1 crypto rx=0x7effa8009d90 tx=0x7effa801c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c7547e4bdb3633a server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:14.222 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:14.216+0000 7effaf7fe640 1 -- 172.21.15.16:0/3381307018 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effa80a5950 con 0x7effb006b6c0 2023-05-22T16:05:14.222 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:14.216+0000 7effaf7fe640 1 -- 172.21.15.16:0/3381307018 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effa80a5ab0 con 0x7effb006b6c0 2023-05-22T16:05:14.222 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:14.216+0000 7effaf7fe640 1 -- 172.21.15.16:0/3381307018 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effa80a5e20 con 0x7effb006b6c0 2023-05-22T16:05:14.222 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:14.216+0000 7effb6ba2640 1 -- 172.21.15.16:0/3381307018 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7effb006b6c0 msgr2=0x7effb006bb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:14.222 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:14.216+0000 7effb6ba2640 1 --2- 172.21.15.16:0/3381307018 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7effb006b6c0 0x7effb006bb20 secure :-1 s=READY pgs=2392 cs=0 l=1 rev1=1 crypto rx=0x7effa8009d90 tx=0x7effa801c890 comp rx=0 tx=0).stop 2023-05-22T16:05:14.222 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:14.216+0000 7effb6ba2640 1 -- 172.21.15.16:0/3381307018 shutdown_connections 2023-05-22T16:05:14.223 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:14.220+0000 7effb6ba2640 1 --2- 172.21.15.16:0/3381307018 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7effb006b6c0 0x7effb006bb20 unknown :-1 s=CLOSED pgs=2392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:14.223 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:14.220+0000 7effb6ba2640 1 --2- 172.21.15.16:0/3381307018 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7effb0159d40 0x7effb00cb260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:14.223 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:14.220+0000 7effb6ba2640 1 -- 172.21.15.16:0/3381307018 >> 172.21.15.16:0/3381307018 conn(0x7effb0069720 msgr2=0x7effb0069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:05:14.223 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:14.220+0000 7effb6ba2640 1 -- 172.21.15.16:0/3381307018 shutdown_connections 2023-05-22T16:05:14.223 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:14.220+0000 7effb6ba2640 1 -- 172.21.15.16:0/3381307018 wait complete. 2023-05-22T16:05:14.223 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:14.220+0000 7effb6ba2640 1 Processor -- start 2023-05-22T16:05:14.223 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:14.220+0000 7effb6ba2640 1 -- start start 2023-05-22T16:05:14.225 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:14.220+0000 7effb6ba2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7effb006b6c0 0x7effb0149de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:14.225 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:14.220+0000 7effb6ba2640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7effb0159970 0x7effb014d970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:14.225 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:14.220+0000 7effb6ba2640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7effb0159d40 0x7effb01537b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:14.225 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:14.220+0000 7effb6ba2640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effb014deb0 con 0x7effb006b6c0 2023-05-22T16:05:14.225 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:14.220+0000 7effb6ba2640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effb0153cf0 con 0x7effb0159d40 2023-05-22T16:05:14.225 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:14.220+0000 7effb6ba2640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effb014e0a0 con 0x7effb0159970 2023-05-22T16:05:14.226 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:14.220+0000 7effb493e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7effb006b6c0 0x7effb0149de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:14.226 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:14.220+0000 7effb493e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7effb006b6c0 0x7effb0149de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40062/0 (socket says 172.21.15.16:40062) 2023-05-22T16:05:14.226 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:14.220+0000 7effb493e640 1 -- 172.21.15.16:0/1783982070 learned_addr learned my addr 172.21.15.16:0/1783982070 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:05:14.226 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:14.220+0000 7effaffff640 1 --2- 172.21.15.16:0/1783982070 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7effb0159970 0x7effb014d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:14.226 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:14.220+0000 7effb493e640 1 -- 172.21.15.16:0/1783982070 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7effb0159970 msgr2=0x7effb014d970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:14.227 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:14.220+0000 7effb493e640 1 --2- 172.21.15.16:0/1783982070 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7effb0159970 0x7effb014d970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:14.227 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:14.220+0000 7effb513f640 1 --2- 172.21.15.16:0/1783982070 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7effb0159d40 0x7effb01537b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:14.227 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:14.220+0000 7effb493e640 1 -- 172.21.15.16:0/1783982070 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7effb0159d40 msgr2=0x7effb01537b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:14.227 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:14.220+0000 7effb493e640 1 --2- 172.21.15.16:0/1783982070 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7effb0159d40 0x7effb01537b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:14.227 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:14.220+0000 7effb493e640 1 -- 172.21.15.16:0/1783982070 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effa8009a70 con 0x7effb006b6c0 2023-05-22T16:05:14.227 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:14.220+0000 7effb493e640 1 --2- 172.21.15.16:0/1783982070 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7effb006b6c0 0x7effb0149de0 secure :-1 s=READY pgs=2393 cs=0 l=1 rev1=1 crypto rx=0x7effa400c450 tx=0x7effa400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:14.228 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:14.220+0000 7effad7fa640 1 -- 172.21.15.16:0/1783982070 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effa4005cf0 con 0x7effb006b6c0 2023-05-22T16:05:14.228 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:14.220+0000 7effad7fa640 1 -- 172.21.15.16:0/1783982070 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effa4005e50 con 0x7effb006b6c0 2023-05-22T16:05:14.228 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:14.220+0000 7effb6ba2640 1 -- 172.21.15.16:0/1783982070 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7effb0071700 con 0x7effb006b6c0 2023-05-22T16:05:14.228 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:14.220+0000 7effad7fa640 1 -- 172.21.15.16:0/1783982070 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effa40072f0 con 0x7effb006b6c0 2023-05-22T16:05:14.228 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:14.220+0000 7effb6ba2640 1 -- 172.21.15.16:0/1783982070 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7effb0071420 con 0x7effb006b6c0 2023-05-22T16:05:14.228 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:14.220+0000 7effb6ba2640 1 -- 172.21.15.16:0/1783982070 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7effb00cb7a0 con 0x7effb006b6c0 2023-05-22T16:05:14.229 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:14.224+0000 7effad7fa640 1 -- 172.21.15.16:0/1783982070 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 92) v1 ==== 84332+0+0 (secure 0 0 0) 0x7effa4007450 con 0x7effb006b6c0 2023-05-22T16:05:14.229 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:14.224+0000 7effad7fa640 1 --2- 172.21.15.16:0/1783982070 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7eff940654a0 0x7eff94067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:14.229 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:14.224+0000 7effad7fa640 1 -- 172.21.15.16:0/1783982070 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(187..187 src has 1..187) v4 ==== 6360+0+0 (secure 0 0 0) 0x7effa4082140 con 0x7effb006b6c0 2023-05-22T16:05:14.230 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:14.224+0000 7effaffff640 1 --2- 172.21.15.16:0/1783982070 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7eff940654a0 0x7eff94067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:14.230 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:14.224+0000 7effaffff640 1 --2- 172.21.15.16:0/1783982070 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7eff940654a0 0x7eff94067950 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7effb0159390 tx=0x7eff9c008040 comp rx=0 tx=0).ready entity=mgr.10542 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:14.230 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:14.228+0000 7effad7fa640 1 -- 172.21.15.16:0/1783982070 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7effa404fb80 con 0x7effb006b6c0 2023-05-22T16:05:14.384 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:14.380+0000 7effb6ba2640 1 -- 172.21.15.16:0/1783982070 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7effb004dc30 con 0x7effb006b6c0 2023-05-22T16:05:14.391 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:14.388+0000 7effad7fa640 1 -- 172.21.15.16:0/1783982070 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v63) v1 ==== 115+0+0 (secure 0 0 0) 0x7effa40531a0 con 0x7effb006b6c0 2023-05-22T16:05:14.394 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:14.392+0000 7effb6ba2640 1 -- 172.21.15.16:0/1783982070 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7eff940654a0 msgr2=0x7eff94067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:14.394 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:14.392+0000 7effb6ba2640 1 --2- 172.21.15.16:0/1783982070 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7eff940654a0 0x7eff94067950 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7effb0159390 tx=0x7eff9c008040 comp rx=0 tx=0).stop 2023-05-22T16:05:14.395 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:14.392+0000 7effb6ba2640 1 -- 172.21.15.16:0/1783982070 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7effb006b6c0 msgr2=0x7effb0149de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:14.395 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:14.392+0000 7effb6ba2640 1 --2- 172.21.15.16:0/1783982070 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7effb006b6c0 0x7effb0149de0 secure :-1 s=READY pgs=2393 cs=0 l=1 rev1=1 crypto rx=0x7effa400c450 tx=0x7effa400c910 comp rx=0 tx=0).stop 2023-05-22T16:05:14.395 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:14.392+0000 7effb6ba2640 1 -- 172.21.15.16:0/1783982070 shutdown_connections 2023-05-22T16:05:14.395 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:14.392+0000 7effb6ba2640 1 --2- 172.21.15.16:0/1783982070 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7effb0159d40 0x7effb01537b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:14.396 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:14.392+0000 7effb6ba2640 1 --2- 172.21.15.16:0/1783982070 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7effb0159970 0x7effb014d970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:14.396 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:14.392+0000 7effb6ba2640 1 --2- 172.21.15.16:0/1783982070 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7eff940654a0 0x7eff94067950 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:14.396 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:14.392+0000 7effb6ba2640 1 --2- 172.21.15.16:0/1783982070 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7effb006b6c0 0x7effb0149de0 unknown :-1 s=CLOSED pgs=2393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:14.396 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:14.392+0000 7effb6ba2640 1 -- 172.21.15.16:0/1783982070 >> 172.21.15.16:0/1783982070 conn(0x7effb0069720 msgr2=0x7effb0069af0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:05:14.396 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:14.392+0000 7effb6ba2640 1 -- 172.21.15.16:0/1783982070 shutdown_connections 2023-05-22T16:05:14.397 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:14.392+0000 7effb6ba2640 1 -- 172.21.15.16:0/1783982070 wait complete. 2023-05-22T16:05:14.412 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_retention' 2023-05-22T16:05:14.541 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:14.540+0000 7f56d8731640 1 Processor -- start 2023-05-22T16:05:14.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:14.540+0000 7f56d8731640 1 -- start start 2023-05-22T16:05:14.544 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:14.540+0000 7f56d8731640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f56d01452c0 0x7f56d015c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:14.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:14.540+0000 7f56d8731640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f56d015cf10 0x7f56d006fed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:14.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:14.540+0000 7f56d8731640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56d015c1f0 con 0x7f56d0145690 2023-05-22T16:05:14.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:14.540+0000 7f56d8731640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56d00ab860 con 0x7f56d01452c0 2023-05-22T16:05:14.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:14.540+0000 7f56d8731640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56d00abb40 con 0x7f56d015cf10 2023-05-22T16:05:14.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:14.540+0000 7f56d64cd640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f56d01452c0 0x7f56d015c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:14.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:14.540+0000 7f56d5ccc640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f56d0145690 0x7f56d00ac1a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:44358/0 (socket says 172.21.15.16:44358) 2023-05-22T16:05:14.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:14.540+0000 7f56d5ccc640 1 -- 172.21.15.16:0/486474489 learned_addr learned my addr 172.21.15.16:0/486474489 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T16:05:14.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:14.540+0000 7f56d6cce640 1 --2- 172.21.15.16:0/486474489 >> v2:172.21.15.62:3300/0 conn(0x7f56d015cf10 0x7f56d006fed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:14.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:14.540+0000 7f56d64cd640 1 -- 172.21.15.16:0/486474489 >> v2:172.21.15.62:3300/0 conn(0x7f56d015cf10 msgr2=0x7f56d006fed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:05:14.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:14.540+0000 7f56d64cd640 1 --2- 172.21.15.16:0/486474489 >> v2:172.21.15.62:3300/0 conn(0x7f56d015cf10 0x7f56d006fed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:14.549 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:14.540+0000 7f56d64cd640 1 -- 172.21.15.16:0/486474489 >> v1:172.21.15.16:6789/0 conn(0x7f56d0145690 legacy=0x7f56d00ac1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:14.549 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:14.540+0000 7f56d64cd640 1 -- 172.21.15.16:0/486474489 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56d00abe20 con 0x7f56d01452c0 2023-05-22T16:05:14.549 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:14.540+0000 7f56d54cb640 1 -- 172.21.15.16:0/486474489 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3345158317 0 0) 0x7f56d015c1f0 con 0x7f56d0145690 2023-05-22T16:05:14.549 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:14.540+0000 7f56d64cd640 1 --2- 172.21.15.16:0/486474489 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f56d01452c0 0x7f56d015c9d0 secure :-1 s=READY pgs=2395 cs=0 l=1 rev1=1 crypto rx=0x7f56d00ad970 tx=0x7f56cc01c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f872a4f37b9019de server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:14.550 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:14.544+0000 7f56d54cb640 1 -- 172.21.15.16:0/486474489 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56cc0ab600 con 0x7f56d01452c0 2023-05-22T16:05:14.550 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:14.544+0000 7f56d54cb640 1 -- 172.21.15.16:0/486474489 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56cc0a7050 con 0x7f56d01452c0 2023-05-22T16:05:14.550 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:14.544+0000 7f56d54cb640 1 -- 172.21.15.16:0/486474489 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56cc0ab980 con 0x7f56d01452c0 2023-05-22T16:05:14.555 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:14.544+0000 7f56d8731640 1 -- 172.21.15.16:0/486474489 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f56d01452c0 msgr2=0x7f56d015c9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:14.555 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:14.544+0000 7f56d8731640 1 --2- 172.21.15.16:0/486474489 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f56d01452c0 0x7f56d015c9d0 secure :-1 s=READY pgs=2395 cs=0 l=1 rev1=1 crypto rx=0x7f56d00ad970 tx=0x7f56cc01c3a0 comp rx=0 tx=0).stop 2023-05-22T16:05:14.555 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:14.544+0000 7f56d8731640 1 -- 172.21.15.16:0/486474489 shutdown_connections 2023-05-22T16:05:14.555 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:14.544+0000 7f56d8731640 1 --2- 172.21.15.16:0/486474489 >> v2:172.21.15.62:3300/0 conn(0x7f56d015cf10 0x7f56d006fed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:14.556 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:14.544+0000 7f56d8731640 1 --2- 172.21.15.16:0/486474489 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f56d01452c0 0x7f56d015c9d0 unknown :-1 s=CLOSED pgs=2395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:14.556 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:14.544+0000 7f56d8731640 1 -- 172.21.15.16:0/486474489 >> 172.21.15.16:0/486474489 conn(0x7f56d006e3c0 msgr2=0x7f56d00b1850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:05:14.556 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:14.544+0000 7f56d8731640 1 -- 172.21.15.16:0/486474489 shutdown_connections 2023-05-22T16:05:14.557 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:14.544+0000 7f56d8731640 1 -- 172.21.15.16:0/486474489 wait complete. 2023-05-22T16:05:14.557 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:14.544+0000 7f56d8731640 1 Processor -- start 2023-05-22T16:05:14.557 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:14.544+0000 7f56d8731640 1 -- start start 2023-05-22T16:05:14.557 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:14.544+0000 7f56d8731640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f56d01452c0 0x7f56d013b090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:14.558 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:14.544+0000 7f56d8731640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f56d0145690 0x7f56d0142ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:14.558 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:14.544+0000 7f56d8731640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f56d015cf10 0x7f56d0159350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:14.558 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:14.544+0000 7f56d8731640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56d0142d20 con 0x7f56d01452c0 2023-05-22T16:05:14.559 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:14.544+0000 7f56d8731640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56d01449a0 con 0x7f56d015cf10 2023-05-22T16:05:14.559 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:14.544+0000 7f56d8731640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56d00aedd0 con 0x7f56d0145690 2023-05-22T16:05:14.559 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:14.544+0000 7f56d64cd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f56d01452c0 0x7f56d013b090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:14.559 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:14.544+0000 7f56d64cd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f56d01452c0 0x7f56d013b090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40074/0 (socket says 172.21.15.16:40074) 2023-05-22T16:05:14.560 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:14.544+0000 7f56d64cd640 1 -- 172.21.15.16:0/381478912 learned_addr learned my addr 172.21.15.16:0/381478912 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:05:14.560 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:14.544+0000 7f56d6cce640 1 --2- 172.21.15.16:0/381478912 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f56d015cf10 0x7f56d0159350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:14.560 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:14.544+0000 7f56d5ccc640 1 --2- 172.21.15.16:0/381478912 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f56d0145690 0x7f56d0142ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:14.561 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:14.544+0000 7f56d64cd640 1 -- 172.21.15.16:0/381478912 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f56d0145690 msgr2=0x7f56d0142ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:14.561 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:14.544+0000 7f56d64cd640 1 --2- 172.21.15.16:0/381478912 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f56d0145690 0x7f56d0142ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:14.561 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:14.544+0000 7f56d64cd640 1 -- 172.21.15.16:0/381478912 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f56d015cf10 msgr2=0x7f56d0159350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:14.561 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:14.544+0000 7f56d64cd640 1 --2- 172.21.15.16:0/381478912 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f56d015cf10 0x7f56d0159350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:14.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:14.544+0000 7f56d64cd640 1 -- 172.21.15.16:0/381478912 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56cc009420 con 0x7f56d01452c0 2023-05-22T16:05:14.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:14.544+0000 7f56d6cce640 1 --2- 172.21.15.16:0/381478912 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f56d015cf10 0x7f56d0159350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T16:05:14.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:14.544+0000 7f56d64cd640 1 --2- 172.21.15.16:0/381478912 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f56d01452c0 0x7f56d013b090 secure :-1 s=READY pgs=2396 cs=0 l=1 rev1=1 crypto rx=0x7f56cc0ae040 tx=0x7f56cc01abe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:14.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:14.544+0000 7f56c6ffd640 1 -- 172.21.15.16:0/381478912 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56cc0b7450 con 0x7f56d01452c0 2023-05-22T16:05:14.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:14.544+0000 7f56c6ffd640 1 -- 172.21.15.16:0/381478912 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56cc0b75d0 con 0x7f56d01452c0 2023-05-22T16:05:14.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:14.544+0000 7f56d8731640 1 -- 172.21.15.16:0/381478912 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56d0160740 con 0x7f56d01452c0 2023-05-22T16:05:14.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:14.544+0000 7f56c6ffd640 1 -- 172.21.15.16:0/381478912 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56cc0b7980 con 0x7f56d01452c0 2023-05-22T16:05:14.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:14.544+0000 7f56d8731640 1 -- 172.21.15.16:0/381478912 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56d01605b0 con 0x7f56d01452c0 2023-05-22T16:05:14.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:14.548+0000 7f56d8731640 1 -- 172.21.15.16:0/381478912 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56d0168b80 con 0x7f56d01452c0 2023-05-22T16:05:14.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:14.548+0000 7f56c6ffd640 1 -- 172.21.15.16:0/381478912 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 92) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f56cc0b7ae0 con 0x7f56d01452c0 2023-05-22T16:05:14.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:14.548+0000 7f56c6ffd640 1 --2- 172.21.15.16:0/381478912 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f56b40654a0 0x7f56b4067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:14.565 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:14.548+0000 7f56c6ffd640 1 -- 172.21.15.16:0/381478912 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(187..187 src has 1..187) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f56cc11d4d0 con 0x7f56d01452c0 2023-05-22T16:05:14.565 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:14.548+0000 7f56d5ccc640 1 --2- 172.21.15.16:0/381478912 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f56b40654a0 0x7f56b4067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:14.565 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:14.548+0000 7f56d5ccc640 1 --2- 172.21.15.16:0/381478912 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f56b40654a0 0x7f56b4067950 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f56d0144cc0 tx=0x7f56c0007b60 comp rx=0 tx=0).ready entity=mgr.10542 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:14.565 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:14.552+0000 7f56c6ffd640 1 -- 172.21.15.16:0/381478912 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f56cc0eae00 con 0x7f56d01452c0 2023-05-22T16:05:14.762 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:14.760+0000 7f56d8731640 1 -- 172.21.15.16:0/381478912 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_retention"]} v 0) v1 -- 0x7f56d004dbe0 con 0x7f56d01452c0 2023-05-22T16:05:15.423 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:15.420+0000 7f56c6ffd640 1 -- 172.21.15.16:0/381478912 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_retention"]}]=0 v1362) v1 ==== 163+0+0 (secure 0 0 0) 0x7f56cc0ee420 con 0x7f56d01452c0 2023-05-22T16:05:15.426 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:15.424+0000 7f56d8731640 1 -- 172.21.15.16:0/381478912 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f56b40654a0 msgr2=0x7f56b4067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:15.426 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:15.424+0000 7f56d8731640 1 --2- 172.21.15.16:0/381478912 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f56b40654a0 0x7f56b4067950 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f56d0144cc0 tx=0x7f56c0007b60 comp rx=0 tx=0).stop 2023-05-22T16:05:15.427 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:15.424+0000 7f56d8731640 1 -- 172.21.15.16:0/381478912 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f56d01452c0 msgr2=0x7f56d013b090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:15.427 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:15.424+0000 7f56d8731640 1 --2- 172.21.15.16:0/381478912 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f56d01452c0 0x7f56d013b090 secure :-1 s=READY pgs=2396 cs=0 l=1 rev1=1 crypto rx=0x7f56cc0ae040 tx=0x7f56cc01abe0 comp rx=0 tx=0).stop 2023-05-22T16:05:15.427 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:15.424+0000 7f56d8731640 1 -- 172.21.15.16:0/381478912 shutdown_connections 2023-05-22T16:05:15.427 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:15.424+0000 7f56d8731640 1 --2- 172.21.15.16:0/381478912 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f56d015cf10 0x7f56d0159350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:15.427 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:15.424+0000 7f56d8731640 1 --2- 172.21.15.16:0/381478912 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f56d0145690 0x7f56d0142ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:15.427 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:15.424+0000 7f56d8731640 1 --2- 172.21.15.16:0/381478912 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f56b40654a0 0x7f56b4067950 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:15.427 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:15.424+0000 7f56d8731640 1 --2- 172.21.15.16:0/381478912 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f56d01452c0 0x7f56d013b090 unknown :-1 s=CLOSED pgs=2396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:15.428 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:15.424+0000 7f56d8731640 1 -- 172.21.15.16:0/381478912 >> 172.21.15.16:0/381478912 conn(0x7f56d006e3c0 msgr2=0x7f56d00b2860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:05:15.428 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:15.424+0000 7f56d8731640 1 -- 172.21.15.16:0/381478912 shutdown_connections 2023-05-22T16:05:15.428 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:15.424+0000 7f56d8731640 1 -- 172.21.15.16:0/381478912 wait complete. 2023-05-22T16:05:15.448 INFO:tasks.cephfs_test_runner:Test scheduled snapshots along with rentention policy ... ok 2023-05-22T16:05:15.449 INFO:tasks.cephfs_test_runner:Starting test: test_snap_dir_name (tasks.cephfs.test_snap_schedules.TestSnapSchedulesSnapdir) Test the correctness of snap directory name 2023-05-22T16:05:15.449 INFO:tasks.cephfs_test_runner:test_snap_dir_name (tasks.cephfs.test_snap_schedules.TestSnapSchedulesSnapdir) 2023-05-22T16:05:15.450 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedulesSnapdir.test_snap_dir_name' 2023-05-22T16:05:15.580 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:15.576+0000 7f722e1c5640 1 Processor -- start 2023-05-22T16:05:15.580 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:15.576+0000 7f722e1c5640 1 -- start start 2023-05-22T16:05:15.581 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:15.580+0000 7f722e1c5640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f72280b7dd0 0x7f72280ccab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:15.582 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:15.580+0000 7f722e1c5640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f722815f700 0x7f722815fb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:15.582 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:15.580+0000 7f722e1c5640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72280cd170 con 0x7f7228160080 2023-05-22T16:05:15.582 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:15.580+0000 7f722e1c5640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72280d3360 con 0x7f722815f700 2023-05-22T16:05:15.582 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:15.580+0000 7f722e1c5640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72280d2ac0 con 0x7f72280b7dd0 2023-05-22T16:05:15.582 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:15.580+0000 7f7226ffd640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f722815f700 0x7f722815fb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:15.582 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:15.580+0000 7f7227fff640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f7228160080 0x7f72280ce720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:44372/0 (socket says 172.21.15.16:44372) 2023-05-22T16:05:15.582 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:15.580+0000 7f7227fff640 1 -- 172.21.15.16:0/113767961 learned_addr learned my addr 172.21.15.16:0/113767961 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T16:05:15.583 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:15.580+0000 7f72277fe640 1 --2- 172.21.15.16:0/113767961 >> v2:172.21.15.62:3300/0 conn(0x7f72280b7dd0 0x7f72280ccab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:15.583 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:15.580+0000 7f7226ffd640 1 -- 172.21.15.16:0/113767961 >> v2:172.21.15.62:3300/0 conn(0x7f72280b7dd0 msgr2=0x7f72280ccab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:05:15.583 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:15.580+0000 7f7226ffd640 1 --2- 172.21.15.16:0/113767961 >> v2:172.21.15.62:3300/0 conn(0x7f72280b7dd0 0x7f72280ccab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:15.583 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:15.580+0000 7f7226ffd640 1 -- 172.21.15.16:0/113767961 >> v1:172.21.15.16:6789/0 conn(0x7f7228160080 legacy=0x7f72280ce720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:15.583 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:15.580+0000 7f72267fc640 1 -- 172.21.15.16:0/113767961 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 46327782 0 0) 0x7f72280cd170 con 0x7f7228160080 2023-05-22T16:05:15.583 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:15.580+0000 7f7226ffd640 1 -- 172.21.15.16:0/113767961 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72280d2da0 con 0x7f722815f700 2023-05-22T16:05:15.584 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:15.580+0000 7f7226ffd640 1 --2- 172.21.15.16:0/113767961 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f722815f700 0x7f722815fb40 secure :-1 s=READY pgs=2398 cs=0 l=1 rev1=1 crypto rx=0x7f7228071e30 tx=0x7f721001c430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=692c7c44aa74ad6d server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:15.584 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:15.584+0000 7f72267fc640 1 -- 172.21.15.16:0/113767961 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72100ac630 con 0x7f722815f700 2023-05-22T16:05:15.584 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:15.584+0000 7f72267fc640 1 -- 172.21.15.16:0/113767961 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72100ac790 con 0x7f722815f700 2023-05-22T16:05:15.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:15.584+0000 7f72267fc640 1 -- 172.21.15.16:0/113767961 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72100acb20 con 0x7f722815f700 2023-05-22T16:05:15.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:15.584+0000 7f722e1c5640 1 -- 172.21.15.16:0/113767961 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f722815f700 msgr2=0x7f722815fb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:15.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:15.584+0000 7f722e1c5640 1 --2- 172.21.15.16:0/113767961 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f722815f700 0x7f722815fb40 secure :-1 s=READY pgs=2398 cs=0 l=1 rev1=1 crypto rx=0x7f7228071e30 tx=0x7f721001c430 comp rx=0 tx=0).stop 2023-05-22T16:05:15.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:15.584+0000 7f722e1c5640 1 -- 172.21.15.16:0/113767961 shutdown_connections 2023-05-22T16:05:15.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:15.584+0000 7f722e1c5640 1 --2- 172.21.15.16:0/113767961 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f722815f700 0x7f722815fb40 unknown :-1 s=CLOSED pgs=2398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:15.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:15.584+0000 7f722e1c5640 1 --2- 172.21.15.16:0/113767961 >> v2:172.21.15.62:3300/0 conn(0x7f72280b7dd0 0x7f72280ccab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:15.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:15.584+0000 7f722e1c5640 1 -- 172.21.15.16:0/113767961 >> 172.21.15.16:0/113767961 conn(0x7f72280655c0 msgr2=0x7f7228070e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:05:15.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:15.584+0000 7f722e1c5640 1 -- 172.21.15.16:0/113767961 shutdown_connections 2023-05-22T16:05:15.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:15.584+0000 7f722e1c5640 1 -- 172.21.15.16:0/113767961 wait complete. 2023-05-22T16:05:15.588 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:15.584+0000 7f722e1c5640 1 Processor -- start 2023-05-22T16:05:15.588 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:15.584+0000 7f722e1c5640 1 -- start start 2023-05-22T16:05:15.588 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:15.584+0000 7f722e1c5640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f72280b7dd0 0x7f722814d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:15.588 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:15.584+0000 7f722e1c5640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f722815f700 0x7f72280b2d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:15.589 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:15.584+0000 7f722e1c5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7228160080 0x7f72281547f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:15.589 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:15.584+0000 7f722e1c5640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72280b3980 con 0x7f7228160080 2023-05-22T16:05:15.589 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:15.584+0000 7f722e1c5640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f722815aaa0 con 0x7f72280b7dd0 2023-05-22T16:05:15.589 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:15.584+0000 7f722e1c5640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72280b36a0 con 0x7f722815f700 2023-05-22T16:05:15.589 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:15.584+0000 7f7227fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7228160080 0x7f72281547f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:15.589 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:15.584+0000 7f7227fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7228160080 0x7f72281547f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40104/0 (socket says 172.21.15.16:40104) 2023-05-22T16:05:15.589 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:15.584+0000 7f7227fff640 1 -- 172.21.15.16:0/581589590 learned_addr learned my addr 172.21.15.16:0/581589590 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:05:15.590 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:15.584+0000 7f72277fe640 1 --2- 172.21.15.16:0/581589590 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f72280b7dd0 0x7f722814d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:15.590 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:15.584+0000 7f7226ffd640 1 --2- 172.21.15.16:0/581589590 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f722815f700 0x7f72280b2d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:15.590 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:15.584+0000 7f7227fff640 1 -- 172.21.15.16:0/581589590 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f722815f700 msgr2=0x7f72280b2d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:15.590 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:15.584+0000 7f7227fff640 1 --2- 172.21.15.16:0/581589590 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f722815f700 0x7f72280b2d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:15.590 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:15.584+0000 7f7227fff640 1 -- 172.21.15.16:0/581589590 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f72280b7dd0 msgr2=0x7f722814d6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:15.590 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:15.584+0000 7f7227fff640 1 --2- 172.21.15.16:0/581589590 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f72280b7dd0 0x7f722814d6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:15.590 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:15.584+0000 7f7227fff640 1 -- 172.21.15.16:0/581589590 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7210009420 con 0x7f7228160080 2023-05-22T16:05:15.591 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:15.584+0000 7f7227fff640 1 --2- 172.21.15.16:0/581589590 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7228160080 0x7f72281547f0 secure :-1 s=READY pgs=2399 cs=0 l=1 rev1=1 crypto rx=0x7f722000c780 tx=0x7f722000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:15.591 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:15.584+0000 7f721ffff640 1 -- 172.21.15.16:0/581589590 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f722000ce90 con 0x7f7228160080 2023-05-22T16:05:15.591 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:15.584+0000 7f721ffff640 1 -- 172.21.15.16:0/581589590 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f722000a640 con 0x7f7228160080 2023-05-22T16:05:15.591 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:15.584+0000 7f722e1c5640 1 -- 172.21.15.16:0/581589590 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7228168a20 con 0x7f7228160080 2023-05-22T16:05:15.591 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:15.584+0000 7f721ffff640 1 -- 172.21.15.16:0/581589590 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f722000a9d0 con 0x7f7228160080 2023-05-22T16:05:15.591 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:15.584+0000 7f722e1c5640 1 -- 172.21.15.16:0/581589590 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7228168890 con 0x7f7228160080 2023-05-22T16:05:15.592 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:15.588+0000 7f722e1c5640 1 -- 172.21.15.16:0/581589590 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7228168bb0 con 0x7f7228160080 2023-05-22T16:05:15.592 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:15.588+0000 7f721ffff640 1 -- 172.21.15.16:0/581589590 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 92) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f722000ab30 con 0x7f7228160080 2023-05-22T16:05:15.592 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:15.588+0000 7f721ffff640 1 --2- 172.21.15.16:0/581589590 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f72080654a0 0x7f7208067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:15.592 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:15.588+0000 7f721ffff640 1 -- 172.21.15.16:0/581589590 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(187..187 src has 1..187) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f7220081ff0 con 0x7f7228160080 2023-05-22T16:05:15.593 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:15.588+0000 7f72277fe640 1 --2- 172.21.15.16:0/581589590 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f72080654a0 0x7f7208067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:15.593 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:15.588+0000 7f72277fe640 1 --2- 172.21.15.16:0/581589590 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f72080654a0 0x7f7208067950 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f721800b9d0 tx=0x7f7218008040 comp rx=0 tx=0).ready entity=mgr.10542 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:15.594 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:15.592+0000 7f721ffff640 1 -- 172.21.15.16:0/581589590 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f722004fac0 con 0x7f7228160080 2023-05-22T16:05:15.805 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:15.804+0000 7f722e1c5640 1 -- 172.21.15.16:0/581589590 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedulesSnapdir.test_snap_dir_name"]} v 0) v1 -- 0x7f722815a620 con 0x7f7228160080 2023-05-22T16:05:16.426 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:16.424+0000 7f721ffff640 1 -- 172.21.15.16:0/581589590 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedulesSnapdir.test_snap_dir_name"]}]=0 v1363) v1 ==== 158+0+0 (secure 0 0 0) 0x7f722815a620 con 0x7f7228160080 2023-05-22T16:05:16.430 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:16.428+0000 7f722e1c5640 1 -- 172.21.15.16:0/581589590 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f72080654a0 msgr2=0x7f7208067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:16.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:16.428+0000 7f722e1c5640 1 --2- 172.21.15.16:0/581589590 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f72080654a0 0x7f7208067950 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f721800b9d0 tx=0x7f7218008040 comp rx=0 tx=0).stop 2023-05-22T16:05:16.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:16.428+0000 7f722e1c5640 1 -- 172.21.15.16:0/581589590 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7228160080 msgr2=0x7f72281547f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:16.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:16.428+0000 7f722e1c5640 1 --2- 172.21.15.16:0/581589590 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7228160080 0x7f72281547f0 secure :-1 s=READY pgs=2399 cs=0 l=1 rev1=1 crypto rx=0x7f722000c780 tx=0x7f722000cc40 comp rx=0 tx=0).stop 2023-05-22T16:05:16.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:16.428+0000 7f722e1c5640 1 -- 172.21.15.16:0/581589590 shutdown_connections 2023-05-22T16:05:16.432 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:16.428+0000 7f722e1c5640 1 --2- 172.21.15.16:0/581589590 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f72080654a0 0x7f7208067950 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:16.432 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:16.428+0000 7f722e1c5640 1 --2- 172.21.15.16:0/581589590 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7228160080 0x7f72281547f0 unknown :-1 s=CLOSED pgs=2399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:16.432 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:16.428+0000 7f722e1c5640 1 --2- 172.21.15.16:0/581589590 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f722815f700 0x7f72280b2d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:16.432 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:16.428+0000 7f722e1c5640 1 --2- 172.21.15.16:0/581589590 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f72280b7dd0 0x7f722814d6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:16.433 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:16.428+0000 7f722e1c5640 1 -- 172.21.15.16:0/581589590 >> 172.21.15.16:0/581589590 conn(0x7f72280655c0 msgr2=0x7f72280b1e80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:05:16.433 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:16.428+0000 7f722e1c5640 1 -- 172.21.15.16:0/581589590 shutdown_connections 2023-05-22T16:05:16.433 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:16.428+0000 7f722e1c5640 1 -- 172.21.15.16:0/581589590 wait complete. 2023-05-22T16:05:16.452 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-05-22T16:05:16.583 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:16.580+0000 7f6bc07b4640 1 Processor -- start 2023-05-22T16:05:16.583 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:16.580+0000 7f6bc07b4640 1 -- start start 2023-05-22T16:05:16.584 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:16.580+0000 7f6bc07b4640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6bb8159d40 0x7f6bb80cb260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:16.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:16.580+0000 7f6bc07b4640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6bb806b6c0 0x7f6bb806bb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:16.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:16.580+0000 7f6bc07b4640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bb8070c30 con 0x7f6bb8159970 2023-05-22T16:05:16.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:16.580+0000 7f6bc07b4640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6bb815dd90 con 0x7f6bb8159d40 2023-05-22T16:05:16.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:16.580+0000 7f6bc07b4640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6bb806c040 con 0x7f6bb806b6c0 2023-05-22T16:05:16.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:16.580+0000 7f6bbdd4f640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6bb8159d40 0x7f6bb80cb260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:16.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:16.580+0000 7f6bbdd4f640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6bb8159d40 0x7f6bb80cb260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40118/0 (socket says 172.21.15.16:40118) 2023-05-22T16:05:16.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:16.580+0000 7f6bbdd4f640 1 -- 172.21.15.16:0/3096381967 learned_addr learned my addr 172.21.15.16:0/3096381967 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:05:16.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:16.584+0000 7f6bbdd4f640 1 -- 172.21.15.16:0/3096381967 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6bb806b6c0 msgr2=0x7f6bb806bb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:05:16.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:16.584+0000 7f6bbdd4f640 1 --2- 172.21.15.16:0/3096381967 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6bb806b6c0 0x7f6bb806bb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:16.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:16.584+0000 7f6bbdd4f640 1 -- 172.21.15.16:0/3096381967 >> v1:172.21.15.62:6789/0 conn(0x7f6bb8159970 legacy=0x7f6bb815d680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:05:16.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:16.584+0000 7f6bbdd4f640 1 -- 172.21.15.16:0/3096381967 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bb8070e60 con 0x7f6bb8159d40 2023-05-22T16:05:16.587 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:16.584+0000 7f6bbdd4f640 1 --2- 172.21.15.16:0/3096381967 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6bb8159d40 0x7f6bb80cb260 secure :-1 s=READY pgs=2400 cs=0 l=1 rev1=1 crypto rx=0x7f6bac009870 tx=0x7f6bac01c470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db660a9c22f09847 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:16.587 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:16.584+0000 7f6bbd54e640 1 -- 172.21.15.16:0/3096381967 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bac0ab600 con 0x7f6bb8159d40 2023-05-22T16:05:16.587 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:16.584+0000 7f6bbd54e640 1 -- 172.21.15.16:0/3096381967 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bac0a7050 con 0x7f6bb8159d40 2023-05-22T16:05:16.587 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:16.584+0000 7f6bbd54e640 1 -- 172.21.15.16:0/3096381967 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bac0ab980 con 0x7f6bb8159d40 2023-05-22T16:05:16.593 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:16.584+0000 7f6bc07b4640 1 -- 172.21.15.16:0/3096381967 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6bb8159d40 msgr2=0x7f6bb80cb260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:16.593 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:16.584+0000 7f6bc07b4640 1 --2- 172.21.15.16:0/3096381967 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6bb8159d40 0x7f6bb80cb260 secure :-1 s=READY pgs=2400 cs=0 l=1 rev1=1 crypto rx=0x7f6bac009870 tx=0x7f6bac01c470 comp rx=0 tx=0).stop 2023-05-22T16:05:16.593 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:16.584+0000 7f6bc07b4640 1 -- 172.21.15.16:0/3096381967 shutdown_connections 2023-05-22T16:05:16.593 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:16.584+0000 7f6bc07b4640 1 --2- 172.21.15.16:0/3096381967 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6bb806b6c0 0x7f6bb806bb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:16.593 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:16.584+0000 7f6bc07b4640 1 --2- 172.21.15.16:0/3096381967 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6bb8159d40 0x7f6bb80cb260 unknown :-1 s=CLOSED pgs=2400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:16.593 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:16.584+0000 7f6bc07b4640 1 -- 172.21.15.16:0/3096381967 >> 172.21.15.16:0/3096381967 conn(0x7f6bb8069720 msgr2=0x7f6bb8069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:05:16.594 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:16.584+0000 7f6bc07b4640 1 -- 172.21.15.16:0/3096381967 shutdown_connections 2023-05-22T16:05:16.594 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:16.584+0000 7f6bc07b4640 1 -- 172.21.15.16:0/3096381967 wait complete. 2023-05-22T16:05:16.594 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:16.588+0000 7f6bc07b4640 1 Processor -- start 2023-05-22T16:05:16.594 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:16.588+0000 7f6bc07b4640 1 -- start start 2023-05-22T16:05:16.594 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:16.588+0000 7f6bc07b4640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6bb806b6c0 0x7f6bb81f4040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:16.594 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:16.588+0000 7f6bc07b4640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6bb8159970 0x7f6bb81f4b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:16.594 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:16.588+0000 7f6bc07b4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6bb8159d40 0x7f6bb80b2e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:16.595 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:16.588+0000 7f6bc07b4640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bb80b3d00 con 0x7f6bb8159d40 2023-05-22T16:05:16.595 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:16.588+0000 7f6bc07b4640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bb80c42f0 con 0x7f6bb8159970 2023-05-22T16:05:16.595 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:16.588+0000 7f6bc07b4640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6bb81f5070 con 0x7f6bb806b6c0 2023-05-22T16:05:16.595 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:16.588+0000 7f6bbed51640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6bb8159d40 0x7f6bb80b2e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:16.595 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:16.588+0000 7f6bbed51640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6bb8159d40 0x7f6bb80b2e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40124/0 (socket says 172.21.15.16:40124) 2023-05-22T16:05:16.595 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:16.588+0000 7f6bbed51640 1 -- 172.21.15.16:0/505627670 learned_addr learned my addr 172.21.15.16:0/505627670 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:05:16.595 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:16.588+0000 7f6bbdd4f640 1 --2- 172.21.15.16:0/505627670 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6bb8159970 0x7f6bb81f4b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:16.595 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:16.588+0000 7f6bbed51640 1 -- 172.21.15.16:0/505627670 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6bb806b6c0 msgr2=0x7f6bb81f4040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:16.596 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:16.588+0000 7f6bbe550640 1 --2- 172.21.15.16:0/505627670 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6bb806b6c0 0x7f6bb81f4040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:16.596 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:16.588+0000 7f6bbed51640 1 --2- 172.21.15.16:0/505627670 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6bb806b6c0 0x7f6bb81f4040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:16.596 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:16.588+0000 7f6bbed51640 1 -- 172.21.15.16:0/505627670 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6bb8159970 msgr2=0x7f6bb81f4b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:16.596 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:16.588+0000 7f6bbed51640 1 --2- 172.21.15.16:0/505627670 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6bb8159970 0x7f6bb81f4b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:16.596 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:16.588+0000 7f6bbed51640 1 -- 172.21.15.16:0/505627670 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bac0094d0 con 0x7f6bb8159d40 2023-05-22T16:05:16.596 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:16.588+0000 7f6bbdd4f640 1 --2- 172.21.15.16:0/505627670 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6bb8159970 0x7f6bb81f4b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T16:05:16.596 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:16.588+0000 7f6bbed51640 1 --2- 172.21.15.16:0/505627670 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6bb8159d40 0x7f6bb80b2e80 secure :-1 s=READY pgs=2401 cs=0 l=1 rev1=1 crypto rx=0x7f6ba800b430 tx=0x7f6ba800b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:16.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:16.588+0000 7f6bb6ffd640 1 -- 172.21.15.16:0/505627670 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ba8011830 con 0x7f6bb8159d40 2023-05-22T16:05:16.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:16.588+0000 7f6bb6ffd640 1 -- 172.21.15.16:0/505627670 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ba8011990 con 0x7f6bb8159d40 2023-05-22T16:05:16.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:16.588+0000 7f6bc07b4640 1 -- 172.21.15.16:0/505627670 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6bb80cb7a0 con 0x7f6bb8159d40 2023-05-22T16:05:16.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:16.588+0000 7f6bb6ffd640 1 -- 172.21.15.16:0/505627670 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ba8011d40 con 0x7f6bb8159d40 2023-05-22T16:05:16.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:16.588+0000 7f6bc07b4640 1 -- 172.21.15.16:0/505627670 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6bb8071700 con 0x7f6bb8159d40 2023-05-22T16:05:16.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:16.588+0000 7f6bc07b4640 1 -- 172.21.15.16:0/505627670 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6bb80cb930 con 0x7f6bb8159d40 2023-05-22T16:05:16.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:16.588+0000 7f6bb6ffd640 1 -- 172.21.15.16:0/505627670 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 92) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f6ba801a400 con 0x7f6bb8159d40 2023-05-22T16:05:16.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:16.592+0000 7f6bb6ffd640 1 --2- 172.21.15.16:0/505627670 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f6ba40654a0 0x7f6ba4067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:16.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:16.592+0000 7f6bb6ffd640 1 -- 172.21.15.16:0/505627670 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(187..187 src has 1..187) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f6ba8082cc0 con 0x7f6bb8159d40 2023-05-22T16:05:16.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:16.592+0000 7f6bbe550640 1 --2- 172.21.15.16:0/505627670 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f6ba40654a0 0x7f6ba4067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:16.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:16.592+0000 7f6bbe550640 1 --2- 172.21.15.16:0/505627670 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f6ba40654a0 0x7f6ba4067950 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f6bb8155e10 tx=0x7f6bb0009040 comp rx=0 tx=0).ready entity=mgr.10542 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:16.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:16.596+0000 7f6bb6ffd640 1 -- 172.21.15.16:0/505627670 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f6ba80506a0 con 0x7f6bb8159d40 2023-05-22T16:05:16.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:16.748+0000 7f6bc07b4640 1 -- 172.21.15.16:0/505627670 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f6bb815c5d0 con 0x7f6bb8159d40 2023-05-22T16:05:16.760 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:16.756+0000 7f6bb6ffd640 1 -- 172.21.15.16:0/505627670 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v64)=0 v64) v1 ==== 122+0+0 (secure 0 0 0) 0x7f6ba8015020 con 0x7f6bb8159d40 2023-05-22T16:05:16.763 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:16.760+0000 7f6bc07b4640 1 -- 172.21.15.16:0/505627670 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f6ba40654a0 msgr2=0x7f6ba4067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:16.763 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:16.760+0000 7f6bc07b4640 1 --2- 172.21.15.16:0/505627670 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f6ba40654a0 0x7f6ba4067950 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f6bb8155e10 tx=0x7f6bb0009040 comp rx=0 tx=0).stop 2023-05-22T16:05:16.763 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:16.760+0000 7f6bc07b4640 1 -- 172.21.15.16:0/505627670 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6bb8159d40 msgr2=0x7f6bb80b2e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:16.764 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:16.760+0000 7f6bc07b4640 1 --2- 172.21.15.16:0/505627670 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6bb8159d40 0x7f6bb80b2e80 secure :-1 s=READY pgs=2401 cs=0 l=1 rev1=1 crypto rx=0x7f6ba800b430 tx=0x7f6ba800b8f0 comp rx=0 tx=0).stop 2023-05-22T16:05:16.764 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:16.760+0000 7f6bc07b4640 1 -- 172.21.15.16:0/505627670 shutdown_connections 2023-05-22T16:05:16.764 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:16.760+0000 7f6bc07b4640 1 --2- 172.21.15.16:0/505627670 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f6ba40654a0 0x7f6ba4067950 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:16.764 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:16.760+0000 7f6bc07b4640 1 --2- 172.21.15.16:0/505627670 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6bb8159d40 0x7f6bb80b2e80 unknown :-1 s=CLOSED pgs=2401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:16.765 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:16.760+0000 7f6bc07b4640 1 --2- 172.21.15.16:0/505627670 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6bb8159970 0x7f6bb81f4b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:16.765 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:16.760+0000 7f6bc07b4640 1 --2- 172.21.15.16:0/505627670 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6bb806b6c0 0x7f6bb81f4040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:16.765 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:16.760+0000 7f6bc07b4640 1 -- 172.21.15.16:0/505627670 >> 172.21.15.16:0/505627670 conn(0x7f6bb8069720 msgr2=0x7f6bb815bec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:05:16.766 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:16.760+0000 7f6bc07b4640 1 -- 172.21.15.16:0/505627670 shutdown_connections 2023-05-22T16:05:16.766 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:16.760+0000 7f6bc07b4640 1 -- 172.21.15.16:0/505627670 wait complete. 2023-05-22T16:05:16.780 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-05-22T16:05:16.780 DEBUG:teuthology.orchestra.run.smithi016:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-05-22T16:05:16.830 DEBUG:teuthology.orchestra.run.smithi062:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-05-22T16:05:16.880 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-05-22T16:05:16.881 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-05-22T16:05:16.882 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2023-05-22T16:05:16.888 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16345192k,nr_inodes=4086298,mode=755,inode64 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3276544k,mode=755,inode64 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev,inode64 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=15732 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3276540k,nr_inodes=819135,mode=700,uid=1000,gid=1271,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3276544k,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-05-22T16:05:16.889 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2023-05-22T16:05:16.889 DEBUG:teuthology.orchestra.run.smithi062:> dd if=/proc/self/mounts of=/dev/stdout 2023-05-22T16:05:16.895 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16342116k,nr_inodes=4085529,mode=755,inode64 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275932k,mode=755,inode64 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev,inode64 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=13250 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275928k,nr_inodes=818982,mode=700,uid=1000,gid=1271,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275932k,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-05-22T16:05:16.895 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-05-22T16:05:16.895 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2023-05-22T16:05:16.937 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16345192k,nr_inodes=4086298,mode=755,inode64 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3276544k,mode=755,inode64 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev,inode64 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=15732 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3276540k,nr_inodes=819135,mode=700,uid=1000,gid=1271,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3276544k,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-05-22T16:05:16.937 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2023-05-22T16:05:16.938 DEBUG:teuthology.orchestra.run.smithi062:> dd if=/proc/self/mounts of=/dev/stdout 2023-05-22T16:05:16.943 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16342116k,nr_inodes=4085529,mode=755,inode64 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275932k,mode=755,inode64 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev,inode64 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=13250 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275928k,nr_inodes=818982,mode=700,uid=1000,gid=1271,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275932k,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-05-22T16:05:16.943 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2023-05-22T16:05:16.944 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-05-22T16:05:17.104 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:17.100+0000 7effab4f5640 1 Processor -- start 2023-05-22T16:05:17.105 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:17.104+0000 7effab4f5640 1 -- start start 2023-05-22T16:05:17.108 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:17.104+0000 7effab4f5640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7effa40c9b40 0x7effa415e460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:17.109 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:17.104+0000 7effab4f5640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7effa4070df0 0x7effa4071210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:17.109 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:17.104+0000 7effab4f5640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effa40b0650 con 0x7effa40ca4e0 2023-05-22T16:05:17.109 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:17.104+0000 7effab4f5640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7effa40b0370 con 0x7effa4070df0 2023-05-22T16:05:17.110 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:17.104+0000 7effab4f5640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effa415d980 con 0x7effa40c9b40 2023-05-22T16:05:17.110 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:17.104+0000 7effa9291640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7effa40ca4e0 0x7effa40afb40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:44384/0 (socket says 172.21.15.16:44384) 2023-05-22T16:05:17.111 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:17.104+0000 7effa9291640 1 -- 172.21.15.16:0/3516568353 learned_addr learned my addr 172.21.15.16:0/3516568353 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T16:05:17.111 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:17.104+0000 7eff9bfff640 1 -- 172.21.15.16:0/3516568353 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2956833832 0 0) 0x7effa40b0650 con 0x7effa40ca4e0 2023-05-22T16:05:17.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:17.104+0000 7effa8a90640 1 --2- 172.21.15.16:0/3516568353 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7effa40c9b40 0x7effa415e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:17.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:17.104+0000 7effa9a92640 1 --2- 172.21.15.16:0/3516568353 >> v2:172.21.15.62:3300/0 conn(0x7effa4070df0 0x7effa4071210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:17.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:17.104+0000 7eff9bfff640 1 -- 172.21.15.16:0/3516568353 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7eff90003530 con 0x7effa40ca4e0 2023-05-22T16:05:17.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:17.104+0000 7eff9bfff640 1 -- 172.21.15.16:0/3516568353 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3365939446 0 0) 0x7eff90003530 con 0x7effa40ca4e0 2023-05-22T16:05:17.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:17.104+0000 7eff9bfff640 1 -- 172.21.15.16:0/3516568353 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7effa40c9b40 msgr2=0x7effa415e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:05:17.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:17.104+0000 7eff9bfff640 1 --2- 172.21.15.16:0/3516568353 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7effa40c9b40 0x7effa415e460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:17.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:17.104+0000 7eff9bfff640 1 -- 172.21.15.16:0/3516568353 >> v2:172.21.15.62:3300/0 conn(0x7effa4070df0 msgr2=0x7effa4071210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:05:17.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:17.104+0000 7eff9bfff640 1 --2- 172.21.15.16:0/3516568353 >> v2:172.21.15.62:3300/0 conn(0x7effa4070df0 0x7effa4071210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:17.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:17.104+0000 7eff9bfff640 1 -- 172.21.15.16:0/3516568353 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effa415dc60 con 0x7effa40ca4e0 2023-05-22T16:05:17.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:17.104+0000 7eff9bfff640 1 -- 172.21.15.16:0/3516568353 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7effa0001680 con 0x7effa40ca4e0 2023-05-22T16:05:17.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:17.104+0000 7eff9bfff640 1 -- 172.21.15.16:0/3516568353 >> v1:172.21.15.16:6789/0 conn(0x7effa40ca4e0 legacy=0x7effa40afb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:17.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:17.104+0000 7eff9bfff640 1 --2- 172.21.15.16:0/3516568353 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7effa40c9b40 0x7eff90003980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:17.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:17.104+0000 7eff9bfff640 1 --2- 172.21.15.16:0/3516568353 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7eff90003f00 0x7eff900047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:17.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:17.104+0000 7eff9bfff640 1 --2- 172.21.15.16:0/3516568353 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7eff90004d10 0x7eff90005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:17.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:17.104+0000 7eff9bfff640 1 -- 172.21.15.16:0/3516568353 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effa40b0370 con 0x7eff90004d10 2023-05-22T16:05:17.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:17.104+0000 7eff9bfff640 1 -- 172.21.15.16:0/3516568353 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effa415d980 con 0x7eff90003f00 2023-05-22T16:05:17.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:17.104+0000 7eff9bfff640 1 -- 172.21.15.16:0/3516568353 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff9000ba70 con 0x7effa40c9b40 2023-05-22T16:05:17.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:17.104+0000 7effa9a92640 1 --2- 172.21.15.16:0/3516568353 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7eff90004d10 0x7eff90005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:17.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:17.104+0000 7effa8a90640 1 --2- 172.21.15.16:0/3516568353 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7eff90003f00 0x7eff900047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:17.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:17.104+0000 7effa9291640 1 --2- 172.21.15.16:0/3516568353 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7effa40c9b40 0x7eff90003980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:17.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:17.104+0000 7effa9a92640 1 -- 172.21.15.16:0/3516568353 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7effa40c9b40 msgr2=0x7eff90003980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:05:17.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:17.104+0000 7effa9a92640 1 --2- 172.21.15.16:0/3516568353 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7effa40c9b40 0x7eff90003980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:17.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:17.104+0000 7effa9a92640 1 -- 172.21.15.16:0/3516568353 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7eff90003f00 msgr2=0x7eff900047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:05:17.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:17.104+0000 7effa9a92640 1 --2- 172.21.15.16:0/3516568353 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7eff90003f00 0x7eff900047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:17.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:17.104+0000 7effa9a92640 1 -- 172.21.15.16:0/3516568353 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff90003530 con 0x7eff90004d10 2023-05-22T16:05:17.119 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:17.108+0000 7effa9a92640 1 --2- 172.21.15.16:0/3516568353 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7eff90004d10 0x7eff90005180 secure :-1 s=READY pgs=2403 cs=0 l=1 rev1=1 crypto rx=0x7eff9c009f90 tx=0x7eff9c01cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5bb8aa8882a689a3 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:17.119 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:17.108+0000 7eff9bfff640 1 -- 172.21.15.16:0/3516568353 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff9c0a5b60 con 0x7eff90004d10 2023-05-22T16:05:17.119 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:17.108+0000 7eff9bfff640 1 -- 172.21.15.16:0/3516568353 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff9c00f040 con 0x7eff90004d10 2023-05-22T16:05:17.120 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:17.108+0000 7eff9bfff640 1 -- 172.21.15.16:0/3516568353 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff9c0ad440 con 0x7eff90004d10 2023-05-22T16:05:17.120 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:17.108+0000 7effab4f5640 1 -- 172.21.15.16:0/3516568353 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7eff90004d10 msgr2=0x7eff90005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:17.120 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:17.108+0000 7effab4f5640 1 --2- 172.21.15.16:0/3516568353 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7eff90004d10 0x7eff90005180 secure :-1 s=READY pgs=2403 cs=0 l=1 rev1=1 crypto rx=0x7eff9c009f90 tx=0x7eff9c01cc60 comp rx=0 tx=0).stop 2023-05-22T16:05:17.120 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:17.108+0000 7effa9a92640 1 -- 172.21.15.16:0/3516568353 reap_dead start 2023-05-22T16:05:17.120 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:17.108+0000 7effab4f5640 1 -- 172.21.15.16:0/3516568353 shutdown_connections 2023-05-22T16:05:17.120 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:17.108+0000 7effab4f5640 1 -- 172.21.15.16:0/3516568353 >> 172.21.15.16:0/3516568353 conn(0x7effa4069720 msgr2=0x7effa4069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:05:17.120 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:17.108+0000 7effab4f5640 1 -- 172.21.15.16:0/3516568353 shutdown_connections 2023-05-22T16:05:17.121 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:17.108+0000 7effab4f5640 1 -- 172.21.15.16:0/3516568353 wait complete. 2023-05-22T16:05:17.121 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:17.108+0000 7effab4f5640 1 Processor -- start 2023-05-22T16:05:17.121 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:17.108+0000 7effab4f5640 1 -- start start 2023-05-22T16:05:17.121 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:17.108+0000 7effab4f5640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7effa40c1980 0x7effa40c1d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:17.121 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:17.108+0000 7effab4f5640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7effa40c2290 0x7effa41f29e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:17.121 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:17.108+0000 7effab4f5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7effa41f2f20 0x7effa4154f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:17.121 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:17.108+0000 7effab4f5640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effa41f2800 con 0x7effa41f2f20 2023-05-22T16:05:17.122 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:17.108+0000 7effab4f5640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effa41f3390 con 0x7effa40c1980 2023-05-22T16:05:17.122 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:17.108+0000 7effab4f5640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effa41f05b0 con 0x7effa40c2290 2023-05-22T16:05:17.122 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:17.108+0000 7effa9a92640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7effa41f2f20 0x7effa4154f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:17.122 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:17.108+0000 7effa9a92640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7effa41f2f20 0x7effa4154f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40140/0 (socket says 172.21.15.16:40140) 2023-05-22T16:05:17.122 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:17.108+0000 7effa9a92640 1 -- 172.21.15.16:0/2314480131 learned_addr learned my addr 172.21.15.16:0/2314480131 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:05:17.122 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:17.108+0000 7effa8a90640 1 --2- 172.21.15.16:0/2314480131 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7effa40c2290 0x7effa41f29e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:17.122 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:17.108+0000 7effa9291640 1 --2- 172.21.15.16:0/2314480131 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7effa40c1980 0x7effa40c1d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:17.122 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:17.108+0000 7effa9a92640 1 -- 172.21.15.16:0/2314480131 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7effa40c2290 msgr2=0x7effa41f29e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:17.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:17.112+0000 7effa9a92640 1 --2- 172.21.15.16:0/2314480131 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7effa40c2290 0x7effa41f29e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:17.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:17.112+0000 7effa9a92640 1 -- 172.21.15.16:0/2314480131 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7effa40c1980 msgr2=0x7effa40c1d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:17.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:17.112+0000 7effa9a92640 1 --2- 172.21.15.16:0/2314480131 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7effa40c1980 0x7effa40c1d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:17.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:17.112+0000 7effa9a92640 1 -- 172.21.15.16:0/2314480131 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effa4071750 con 0x7effa41f2f20 2023-05-22T16:05:17.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:17.112+0000 7effa9a92640 1 --2- 172.21.15.16:0/2314480131 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7effa41f2f20 0x7effa4154f80 secure :-1 s=READY pgs=2404 cs=0 l=1 rev1=1 crypto rx=0x7eff9c009f90 tx=0x7eff9c009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:17.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:17.112+0000 7eff99ffb640 1 -- 172.21.15.16:0/2314480131 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff9c0039a0 con 0x7effa41f2f20 2023-05-22T16:05:17.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:17.112+0000 7eff99ffb640 1 -- 172.21.15.16:0/2314480131 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff9c003b00 con 0x7effa41f2f20 2023-05-22T16:05:17.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:17.112+0000 7effab4f5640 1 -- 172.21.15.16:0/2314480131 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7effa415df40 con 0x7effa41f2f20 2023-05-22T16:05:17.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:17.112+0000 7eff99ffb640 1 -- 172.21.15.16:0/2314480131 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff9c0ae890 con 0x7effa41f2f20 2023-05-22T16:05:17.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:17.112+0000 7effab4f5640 1 -- 172.21.15.16:0/2314480131 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7effa40ca8b0 con 0x7effa41f2f20 2023-05-22T16:05:17.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:17.112+0000 7effab4f5640 1 -- 172.21.15.16:0/2314480131 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7effa404dbe0 con 0x7effa41f2f20 2023-05-22T16:05:17.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:17.112+0000 7eff99ffb640 1 -- 172.21.15.16:0/2314480131 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 92) v1 ==== 84332+0+0 (secure 0 0 0) 0x7eff9c0b4040 con 0x7effa41f2f20 2023-05-22T16:05:17.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:17.112+0000 7eff99ffb640 1 --2- 172.21.15.16:0/2314480131 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7eff880654a0 0x7eff88067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:17.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:17.112+0000 7eff99ffb640 1 -- 172.21.15.16:0/2314480131 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(187..187 src has 1..187) v4 ==== 6360+0+0 (secure 0 0 0) 0x7eff9c11c7c0 con 0x7effa41f2f20 2023-05-22T16:05:17.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:17.112+0000 7effa9291640 1 --2- 172.21.15.16:0/2314480131 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7eff880654a0 0x7eff88067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:17.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:17.112+0000 7effa9291640 1 --2- 172.21.15.16:0/2314480131 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7eff880654a0 0x7eff88067950 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7effa0000fc0 tx=0x7effa0009040 comp rx=0 tx=0).ready entity=mgr.10542 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:17.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:17.116+0000 7eff99ffb640 1 -- 172.21.15.16:0/2314480131 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7eff9c0ea2a0 con 0x7effa41f2f20 2023-05-22T16:05:17.303 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:17.300+0000 7effab4f5640 1 -- 172.21.15.16:0/2314480131 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7effa40cb010 con 0x7effa41f2f20 2023-05-22T16:05:17.304 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:17.300+0000 7eff99ffb640 1 -- 172.21.15.16:0/2314480131 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 75 v75) v1 ==== 94+0+3531 (secure 0 0 0) 0x7eff9c0bede0 con 0x7effa41f2f20 2023-05-22T16:05:17.304 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T16:05:17.304 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":75,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7682,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/2815032143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":2815032143},{"type":"v1","addr":"172.21.15.62:6835","nonce":2815032143}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":69},{"gid":7925,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6833/2000904888","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":2000904888},{"type":"v1","addr":"172.21.15.62:6833","nonce":2000904888}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":75},{"gid":10239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6836/1961058261","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1961058261},{"type":"v1","addr":"172.21.15.16:6836","nonce":1961058261}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":69},{"gid":10242,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/1453687184","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6835","nonce":1453687184},{"type":"v1","addr":"172.21.15.16:6837","nonce":1453687184}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":69}],"filesystems":[]} 2023-05-22T16:05:17.305 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 75 2023-05-22T16:05:17.306 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:17.304+0000 7effab4f5640 1 -- 172.21.15.16:0/2314480131 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7eff880654a0 msgr2=0x7eff88067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:17.306 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:17.304+0000 7effab4f5640 1 --2- 172.21.15.16:0/2314480131 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7eff880654a0 0x7eff88067950 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7effa0000fc0 tx=0x7effa0009040 comp rx=0 tx=0).stop 2023-05-22T16:05:17.306 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:17.304+0000 7effab4f5640 1 -- 172.21.15.16:0/2314480131 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7effa41f2f20 msgr2=0x7effa4154f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:17.306 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:17.304+0000 7effab4f5640 1 --2- 172.21.15.16:0/2314480131 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7effa41f2f20 0x7effa4154f80 secure :-1 s=READY pgs=2404 cs=0 l=1 rev1=1 crypto rx=0x7eff9c009f90 tx=0x7eff9c009750 comp rx=0 tx=0).stop 2023-05-22T16:05:17.307 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:17.304+0000 7effab4f5640 1 -- 172.21.15.16:0/2314480131 shutdown_connections 2023-05-22T16:05:17.307 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:17.304+0000 7effab4f5640 1 --2- 172.21.15.16:0/2314480131 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7eff880654a0 0x7eff88067950 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:17.307 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:17.304+0000 7effab4f5640 1 --2- 172.21.15.16:0/2314480131 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7effa41f2f20 0x7effa4154f80 unknown :-1 s=CLOSED pgs=2404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:17.307 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:17.304+0000 7effab4f5640 1 --2- 172.21.15.16:0/2314480131 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7effa40c2290 0x7effa41f29e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:17.307 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:17.304+0000 7effab4f5640 1 --2- 172.21.15.16:0/2314480131 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7effa40c1980 0x7effa40c1d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:17.308 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:17.304+0000 7effab4f5640 1 -- 172.21.15.16:0/2314480131 >> 172.21.15.16:0/2314480131 conn(0x7effa4069720 msgr2=0x7effa40b1330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:05:17.308 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:17.304+0000 7effab4f5640 1 -- 172.21.15.16:0/2314480131 shutdown_connections 2023-05-22T16:05:17.308 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:17.304+0000 7effab4f5640 1 -- 172.21.15.16:0/2314480131 wait complete. 2023-05-22T16:05:17.321 INFO:tasks.ceph.mds.a:Restarting daemon 2023-05-22T16:05:17.321 INFO:tasks.ceph.mds.a:Stopping old one... 2023-05-22T16:05:17.322 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-05-22T16:05:17.322 INFO:teuthology.orchestra.run:waiting for 300 2023-05-22T16:05:17.322 INFO:tasks.ceph.mds.c:Restarting daemon 2023-05-22T16:05:17.322 INFO:tasks.ceph.mds.c:Stopping old one... 2023-05-22T16:05:17.323 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-05-22T16:05:17.323 INFO:teuthology.orchestra.run:waiting for 300 2023-05-22T16:05:17.323 INFO:tasks.ceph.mds.b:Restarting daemon 2023-05-22T16:05:17.323 INFO:tasks.ceph.mds.b:Stopping old one... 2023-05-22T16:05:17.324 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-05-22T16:05:17.324 INFO:teuthology.orchestra.run:waiting for 300 2023-05-22T16:05:17.324 INFO:tasks.ceph.mds.d:Restarting daemon 2023-05-22T16:05:17.324 INFO:tasks.ceph.mds.d:Stopping old one... 2023-05-22T16:05:17.324 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-05-22T16:05:17.325 INFO:teuthology.orchestra.run:waiting for 300 2023-05-22T16:05:17.334 INFO:tasks.ceph.mds.a:Stopped 2023-05-22T16:05:17.335 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-05-22T16:05:17.336 INFO:tasks.ceph.mds.c:Stopped 2023-05-22T16:05:17.337 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-05-22T16:05:17.338 INFO:tasks.ceph.mds.b:Stopped 2023-05-22T16:05:17.339 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-05-22T16:05:17.341 INFO:tasks.ceph.mds.d:Stopped 2023-05-22T16:05:17.342 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-05-22T16:05:17.378 INFO:tasks.ceph.mds.a:Started 2023-05-22T16:05:17.379 INFO:tasks.ceph.mds.c:Started 2023-05-22T16:05:17.380 DEBUG:teuthology.parallel:result is None 2023-05-22T16:05:17.381 DEBUG:teuthology.parallel:result is None 2023-05-22T16:05:17.384 INFO:tasks.ceph.mds.b:Started 2023-05-22T16:05:17.384 DEBUG:teuthology.parallel:result is None 2023-05-22T16:05:17.385 INFO:tasks.ceph.mds.d:Started 2023-05-22T16:05:17.385 DEBUG:teuthology.parallel:result is None 2023-05-22T16:05:17.386 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2023-05-22T16:05:17.440 INFO:tasks.ceph.mds.a.smithi016.stdout:starting mds.a at 2023-05-22T16:05:17.449 INFO:tasks.ceph.mds.c.smithi016.stdout:starting mds.c at 2023-05-22T16:05:17.460 INFO:tasks.ceph.mds.d.smithi062.stdout:starting mds.d at 2023-05-22T16:05:17.461 INFO:tasks.ceph.mds.b.smithi062.stdout:starting mds.b at 2023-05-22T16:05:17.501 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:17.500+0000 7f6c96ea3640 1 Processor -- start 2023-05-22T16:05:17.501 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:17.500+0000 7f6c96ea3640 1 -- start start 2023-05-22T16:05:17.502 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:17.500+0000 7f6c96ea3640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f6c90159cd0 0x7f6c9015a0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:17.502 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:17.500+0000 7f6c96ea3640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6c900cc6f0 0x7f6c900cfe30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:17.502 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:17.500+0000 7f6c96ea3640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c900cf9a0 con 0x7f6c900b2880 2023-05-22T16:05:17.502 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:17.500+0000 7f6c96ea3640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c9015aba0 con 0x7f6c900cc6f0 2023-05-22T16:05:17.503 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:17.500+0000 7f6c96ea3640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c9015ae80 con 0x7f6c90159cd0 2023-05-22T16:05:17.503 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:17.500+0000 7f6c95440640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6c900cc6f0 0x7f6c900cfe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:17.503 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:17.500+0000 7f6c8ffff640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f6c900b2880 0x7f6c900cbfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:44414/0 (socket says 172.21.15.16:44414) 2023-05-22T16:05:17.503 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:17.500+0000 7f6c95440640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6c900cc6f0 0x7f6c900cfe30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40206/0 (socket says 172.21.15.16:40206) 2023-05-22T16:05:17.503 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:17.500+0000 7f6c8ffff640 1 -- 172.21.15.16:0/316059908 learned_addr learned my addr 172.21.15.16:0/316059908 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T16:05:17.503 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:17.500+0000 7f6c94c3f640 1 --2- 172.21.15.16:0/316059908 >> v2:172.21.15.62:3300/0 conn(0x7f6c90159cd0 0x7f6c9015a0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:17.503 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:17.500+0000 7f6c95440640 1 -- 172.21.15.16:0/316059908 >> v2:172.21.15.62:3300/0 conn(0x7f6c90159cd0 msgr2=0x7f6c9015a0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:05:17.504 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:17.500+0000 7f6c95440640 1 --2- 172.21.15.16:0/316059908 >> v2:172.21.15.62:3300/0 conn(0x7f6c90159cd0 0x7f6c9015a0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:17.504 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:17.500+0000 7f6c95440640 1 -- 172.21.15.16:0/316059908 >> v1:172.21.15.16:6789/0 conn(0x7f6c900b2880 legacy=0x7f6c900cbfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:17.504 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:17.500+0000 7f6c95440640 1 -- 172.21.15.16:0/316059908 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c9015b160 con 0x7f6c900cc6f0 2023-05-22T16:05:17.504 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:17.500+0000 7f6c94c3f640 1 --2- 172.21.15.16:0/316059908 >> v2:172.21.15.62:3300/0 conn(0x7f6c90159cd0 0x7f6c9015a0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T16:05:17.504 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:17.500+0000 7f6c8f7fe640 1 -- 172.21.15.16:0/316059908 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1398593817 0 0) 0x7f6c900cf9a0 con 0x7f6c900b2880 2023-05-22T16:05:17.505 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:17.500+0000 7f6c95440640 1 --2- 172.21.15.16:0/316059908 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c900cc6f0 0x7f6c900cfe30 secure :-1 s=READY pgs=2411 cs=0 l=1 rev1=1 crypto rx=0x7f6c900d1330 tx=0x7f6c8801c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c14d763e8402604 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:17.505 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:17.500+0000 7f6c8f7fe640 1 -- 172.21.15.16:0/316059908 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c880a58b0 con 0x7f6c900cc6f0 2023-05-22T16:05:17.506 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:17.500+0000 7f6c8f7fe640 1 -- 172.21.15.16:0/316059908 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c880a5a10 con 0x7f6c900cc6f0 2023-05-22T16:05:17.506 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:17.500+0000 7f6c8f7fe640 1 -- 172.21.15.16:0/316059908 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c880a5da0 con 0x7f6c900cc6f0 2023-05-22T16:05:17.506 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:17.504+0000 7f6c96ea3640 1 -- 172.21.15.16:0/316059908 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c900cc6f0 msgr2=0x7f6c900cfe30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:17.506 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:17.504+0000 7f6c96ea3640 1 --2- 172.21.15.16:0/316059908 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c900cc6f0 0x7f6c900cfe30 secure :-1 s=READY pgs=2411 cs=0 l=1 rev1=1 crypto rx=0x7f6c900d1330 tx=0x7f6c8801c8e0 comp rx=0 tx=0).stop 2023-05-22T16:05:17.506 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:17.504+0000 7f6c96ea3640 1 -- 172.21.15.16:0/316059908 shutdown_connections 2023-05-22T16:05:17.506 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:17.504+0000 7f6c96ea3640 1 --2- 172.21.15.16:0/316059908 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c900cc6f0 0x7f6c900cfe30 unknown :-1 s=CLOSED pgs=2411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:17.506 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:17.504+0000 7f6c96ea3640 1 --2- 172.21.15.16:0/316059908 >> v2:172.21.15.62:3300/0 conn(0x7f6c90159cd0 0x7f6c9015a0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:17.507 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:17.504+0000 7f6c96ea3640 1 -- 172.21.15.16:0/316059908 >> 172.21.15.16:0/316059908 conn(0x7f6c9006f5c0 msgr2=0x7f6c9006f9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:05:17.507 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:17.504+0000 7f6c96ea3640 1 -- 172.21.15.16:0/316059908 shutdown_connections 2023-05-22T16:05:17.507 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:17.504+0000 7f6c96ea3640 1 -- 172.21.15.16:0/316059908 wait complete. 2023-05-22T16:05:17.510 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:17.504+0000 7f6c96ea3640 1 Processor -- start 2023-05-22T16:05:17.510 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:17.504+0000 7f6c96ea3640 1 -- start start 2023-05-22T16:05:17.511 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:17.504+0000 7f6c96ea3640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6c900b2880 0x7f6c90158e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:17.511 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:17.504+0000 7f6c96ea3640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6c900cc6f0 0x7f6c900c5540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:17.511 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:17.504+0000 7f6c96ea3640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c90159cd0 0x7f6c9014e660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:17.511 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:17.504+0000 7f6c96ea3640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c901491f0 con 0x7f6c90159cd0 2023-05-22T16:05:17.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:17.504+0000 7f6c96ea3640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c901593d0 con 0x7f6c900cc6f0 2023-05-22T16:05:17.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:17.504+0000 7f6c96ea3640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c9014d230 con 0x7f6c900b2880 2023-05-22T16:05:17.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:17.504+0000 7f6c95440640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c90159cd0 0x7f6c9014e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:17.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:17.504+0000 7f6c95440640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c90159cd0 0x7f6c9014e660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40218/0 (socket says 172.21.15.16:40218) 2023-05-22T16:05:17.513 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:17.504+0000 7f6c95440640 1 -- 172.21.15.16:0/2096738372 learned_addr learned my addr 172.21.15.16:0/2096738372 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:05:17.513 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:17.504+0000 7f6c8ffff640 1 --2- 172.21.15.16:0/2096738372 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6c900cc6f0 0x7f6c900c5540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:17.513 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:17.504+0000 7f6c94c3f640 1 --2- 172.21.15.16:0/2096738372 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6c900b2880 0x7f6c90158e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:17.514 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:17.504+0000 7f6c95440640 1 -- 172.21.15.16:0/2096738372 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6c900b2880 msgr2=0x7f6c90158e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:17.514 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:17.504+0000 7f6c95440640 1 --2- 172.21.15.16:0/2096738372 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6c900b2880 0x7f6c90158e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:17.514 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:17.504+0000 7f6c95440640 1 -- 172.21.15.16:0/2096738372 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6c900cc6f0 msgr2=0x7f6c900c5540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:17.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:17.504+0000 7f6c95440640 1 --2- 172.21.15.16:0/2096738372 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6c900cc6f0 0x7f6c900c5540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:17.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:17.504+0000 7f6c95440640 1 -- 172.21.15.16:0/2096738372 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c88009a70 con 0x7f6c90159cd0 2023-05-22T16:05:17.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:17.504+0000 7f6c8ffff640 1 --2- 172.21.15.16:0/2096738372 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6c900cc6f0 0x7f6c900c5540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T16:05:17.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:17.504+0000 7f6c95440640 1 --2- 172.21.15.16:0/2096738372 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c90159cd0 0x7f6c9014e660 secure :-1 s=READY pgs=2412 cs=0 l=1 rev1=1 crypto rx=0x7f6c880b0040 tx=0x7f6c88019680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:17.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:17.504+0000 7f6c8d7fa640 1 -- 172.21.15.16:0/2096738372 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c880ad9d0 con 0x7f6c90159cd0 2023-05-22T16:05:17.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:17.504+0000 7f6c8d7fa640 1 -- 172.21.15.16:0/2096738372 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c880adb30 con 0x7f6c90159cd0 2023-05-22T16:05:17.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:17.504+0000 7f6c96ea3640 1 -- 172.21.15.16:0/2096738372 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c900d0d10 con 0x7f6c90159cd0 2023-05-22T16:05:17.516 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:17.504+0000 7f6c8d7fa640 1 -- 172.21.15.16:0/2096738372 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c880b9450 con 0x7f6c90159cd0 2023-05-22T16:05:17.516 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:17.504+0000 7f6c96ea3640 1 -- 172.21.15.16:0/2096738372 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c900d0b80 con 0x7f6c90159cd0 2023-05-22T16:05:17.516 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:17.508+0000 7f6c96ea3640 1 -- 172.21.15.16:0/2096738372 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c900d0ea0 con 0x7f6c90159cd0 2023-05-22T16:05:17.516 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:17.508+0000 7f6c8d7fa640 1 -- 172.21.15.16:0/2096738372 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 92) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f6c880b95b0 con 0x7f6c90159cd0 2023-05-22T16:05:17.516 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:17.508+0000 7f6c8d7fa640 1 --2- 172.21.15.16:0/2096738372 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f6c740654d0 0x7f6c74067980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:17.516 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:17.508+0000 7f6c8d7fa640 1 -- 172.21.15.16:0/2096738372 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(191..191 src has 1..191) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f6c8811d9b0 con 0x7f6c90159cd0 2023-05-22T16:05:17.516 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:17.508+0000 7f6c94c3f640 1 --2- 172.21.15.16:0/2096738372 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f6c740654d0 0x7f6c74067980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:17.516 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:17.508+0000 7f6c94c3f640 1 --2- 172.21.15.16:0/2096738372 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f6c740654d0 0x7f6c74067980 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f6c8400d8b0 tx=0x7f6c84008040 comp rx=0 tx=0).ready entity=mgr.10542 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:17.517 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:17.512+0000 7f6c8d7fa640 1 -- 172.21.15.16:0/2096738372 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f6c880eb480 con 0x7f6c90159cd0 2023-05-22T16:05:17.675 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:17.672+0000 7f6c96ea3640 1 -- 172.21.15.16:0/2096738372 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f6c9004dbe0 con 0x7f6c90159cd0 2023-05-22T16:05:18.508 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:18.504+0000 7f6c8d7fa640 1 -- 172.21.15.16:0/2096738372 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v192) v1 ==== 97+0+0 (secure 0 0 0) 0x7f6c880eeaa0 con 0x7f6c90159cd0 2023-05-22T16:05:18.508 INFO:teuthology.orchestra.run.smithi016.stderr: removed all blocklist entries 2023-05-22T16:05:18.511 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:18.508+0000 7f6c96ea3640 1 -- 172.21.15.16:0/2096738372 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f6c740654d0 msgr2=0x7f6c74067980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:18.511 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:18.508+0000 7f6c96ea3640 1 --2- 172.21.15.16:0/2096738372 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f6c740654d0 0x7f6c74067980 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f6c8400d8b0 tx=0x7f6c84008040 comp rx=0 tx=0).stop 2023-05-22T16:05:18.511 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:18.508+0000 7f6c96ea3640 1 -- 172.21.15.16:0/2096738372 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c90159cd0 msgr2=0x7f6c9014e660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:18.511 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:18.508+0000 7f6c96ea3640 1 --2- 172.21.15.16:0/2096738372 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c90159cd0 0x7f6c9014e660 secure :-1 s=READY pgs=2412 cs=0 l=1 rev1=1 crypto rx=0x7f6c880b0040 tx=0x7f6c88019680 comp rx=0 tx=0).stop 2023-05-22T16:05:18.511 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:18.508+0000 7f6c96ea3640 1 -- 172.21.15.16:0/2096738372 shutdown_connections 2023-05-22T16:05:18.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:18.508+0000 7f6c96ea3640 1 --2- 172.21.15.16:0/2096738372 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f6c740654d0 0x7f6c74067980 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:18.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:18.508+0000 7f6c96ea3640 1 --2- 172.21.15.16:0/2096738372 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c90159cd0 0x7f6c9014e660 unknown :-1 s=CLOSED pgs=2412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:18.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:18.508+0000 7f6c96ea3640 1 --2- 172.21.15.16:0/2096738372 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6c900cc6f0 0x7f6c900c5540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:18.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:18.508+0000 7f6c96ea3640 1 --2- 172.21.15.16:0/2096738372 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6c900b2880 0x7f6c90158e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:18.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:18.508+0000 7f6c96ea3640 1 -- 172.21.15.16:0/2096738372 >> 172.21.15.16:0/2096738372 conn(0x7f6c9006f5c0 msgr2=0x7f6c9006de00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:05:18.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:18.508+0000 7f6c96ea3640 1 -- 172.21.15.16:0/2096738372 shutdown_connections 2023-05-22T16:05:18.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:18.508+0000 7f6c96ea3640 1 -- 172.21.15.16:0/2096738372 wait complete. 2023-05-22T16:05:18.537 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2023-05-22T16:05:18.661 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:18.660+0000 7f1c2a5c4640 1 Processor -- start 2023-05-22T16:05:18.661 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:18.660+0000 7f1c2a5c4640 1 -- start start 2023-05-22T16:05:18.666 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:18.660+0000 7f1c2a5c4640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1c24071500 0x7f1c2415e460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:18.666 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:18.660+0000 7f1c2a5c4640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c240b0650 con 0x7f1c240ca4e0 2023-05-22T16:05:18.666 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:18.660+0000 7f1c2a5c4640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c240b0370 con 0x7f1c240c9b40 2023-05-22T16:05:18.666 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:18.660+0000 7f1c2a5c4640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c2415d980 con 0x7f1c24071500 2023-05-22T16:05:18.667 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:18.660+0000 7f1c23fff640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f1c240ca4e0 0x7f1c240afb40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:44420/0 (socket says 172.21.15.16:44420) 2023-05-22T16:05:18.667 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:18.660+0000 7f1c23fff640 1 -- 172.21.15.16:0/3331455177 learned_addr learned my addr 172.21.15.16:0/3331455177 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T16:05:18.667 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:18.660+0000 7f1c28b61640 1 --2- 172.21.15.16:0/3331455177 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1c24071500 0x7f1c2415e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:18.667 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:18.660+0000 7f1c22ffd640 1 -- 172.21.15.16:0/3331455177 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1238685716 0 0) 0x7f1c240b0650 con 0x7f1c240ca4e0 2023-05-22T16:05:18.667 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:18.660+0000 7f1c22ffd640 1 -- 172.21.15.16:0/3331455177 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c0c003530 con 0x7f1c240ca4e0 2023-05-22T16:05:18.667 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:18.660+0000 7f1c22ffd640 1 -- 172.21.15.16:0/3331455177 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2609727383 0 0) 0x7f1c0c003530 con 0x7f1c240ca4e0 2023-05-22T16:05:18.667 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:18.660+0000 7f1c22ffd640 1 -- 172.21.15.16:0/3331455177 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1c24071500 msgr2=0x7f1c2415e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:05:18.668 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:18.660+0000 7f1c22ffd640 1 --2- 172.21.15.16:0/3331455177 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1c24071500 0x7f1c2415e460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:18.668 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:18.660+0000 7f1c22ffd640 1 -- 172.21.15.16:0/3331455177 >> v1:172.21.15.62:6789/0 conn(0x7f1c240c9b40 legacy=0x7f1c24070df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:18.668 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:18.660+0000 7f1c22ffd640 1 -- 172.21.15.16:0/3331455177 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c2415dc60 con 0x7f1c240ca4e0 2023-05-22T16:05:18.668 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:18.660+0000 7f1c22ffd640 1 -- 172.21.15.16:0/3331455177 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7f1c140016a0 con 0x7f1c240ca4e0 2023-05-22T16:05:18.668 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:18.660+0000 7f1c22ffd640 1 -- 172.21.15.16:0/3331455177 >> v1:172.21.15.16:6789/0 conn(0x7f1c240ca4e0 legacy=0x7f1c240afb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:18.668 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:18.660+0000 7f1c22ffd640 1 --2- 172.21.15.16:0/3331455177 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c0c0039b0 0x7f1c0c003d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:18.668 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:18.660+0000 7f1c22ffd640 1 --2- 172.21.15.16:0/3331455177 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1c0c004350 0x7f1c0c004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:18.669 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:18.660+0000 7f1c22ffd640 1 --2- 172.21.15.16:0/3331455177 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1c24071500 0x7f1c0c005070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:18.669 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:18.660+0000 7f1c22ffd640 1 -- 172.21.15.16:0/3331455177 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c2415d980 con 0x7f1c0c0039b0 2023-05-22T16:05:18.669 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:18.660+0000 7f1c22ffd640 1 -- 172.21.15.16:0/3331455177 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c0c00b990 con 0x7f1c0c004350 2023-05-22T16:05:18.669 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:18.660+0000 7f1c22ffd640 1 -- 172.21.15.16:0/3331455177 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c0c00bb50 con 0x7f1c24071500 2023-05-22T16:05:18.669 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:18.660+0000 7f1c23fff640 1 --2- 172.21.15.16:0/3331455177 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c0c0039b0 0x7f1c0c003d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:18.669 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:18.660+0000 7f1c237fe640 1 --2- 172.21.15.16:0/3331455177 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1c0c004350 0x7f1c0c004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:18.669 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:18.660+0000 7f1c28b61640 1 --2- 172.21.15.16:0/3331455177 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1c24071500 0x7f1c0c005070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:18.669 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:18.660+0000 7f1c23fff640 1 -- 172.21.15.16:0/3331455177 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1c24071500 msgr2=0x7f1c0c005070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:05:18.670 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:18.660+0000 7f1c23fff640 1 --2- 172.21.15.16:0/3331455177 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1c24071500 0x7f1c0c005070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:18.670 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:18.660+0000 7f1c23fff640 1 -- 172.21.15.16:0/3331455177 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1c0c004350 msgr2=0x7f1c0c004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:05:18.670 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:18.660+0000 7f1c23fff640 1 --2- 172.21.15.16:0/3331455177 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1c0c004350 0x7f1c0c004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:18.670 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:18.660+0000 7f1c23fff640 1 -- 172.21.15.16:0/3331455177 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c0c003530 con 0x7f1c0c0039b0 2023-05-22T16:05:18.670 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:18.660+0000 7f1c237fe640 1 --2- 172.21.15.16:0/3331455177 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1c0c004350 0x7f1c0c004810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T16:05:18.670 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:18.660+0000 7f1c28b61640 1 --2- 172.21.15.16:0/3331455177 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1c24071500 0x7f1c0c005070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T16:05:18.672 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:18.664+0000 7f1c23fff640 1 --2- 172.21.15.16:0/3331455177 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c0c0039b0 0x7f1c0c003d80 secure :-1 s=READY pgs=2414 cs=0 l=1 rev1=1 crypto rx=0x7f1c2415d840 tx=0x7f1c1401cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=343a4a166cc3a6e0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:18.673 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:18.664+0000 7f1c22ffd640 1 -- 172.21.15.16:0/3331455177 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c140a5af0 con 0x7f1c0c0039b0 2023-05-22T16:05:18.673 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:18.664+0000 7f1c22ffd640 1 -- 172.21.15.16:0/3331455177 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c140a5c70 con 0x7f1c0c0039b0 2023-05-22T16:05:18.673 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:18.664+0000 7f1c22ffd640 1 -- 172.21.15.16:0/3331455177 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c140058e0 con 0x7f1c0c0039b0 2023-05-22T16:05:18.673 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:18.664+0000 7f1c2a5c4640 1 -- 172.21.15.16:0/3331455177 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c0c0039b0 msgr2=0x7f1c0c003d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:18.673 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:18.664+0000 7f1c2a5c4640 1 --2- 172.21.15.16:0/3331455177 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c0c0039b0 0x7f1c0c003d80 secure :-1 s=READY pgs=2414 cs=0 l=1 rev1=1 crypto rx=0x7f1c2415d840 tx=0x7f1c1401cbf0 comp rx=0 tx=0).stop 2023-05-22T16:05:18.673 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:18.664+0000 7f1c28b61640 1 -- 172.21.15.16:0/3331455177 reap_dead start 2023-05-22T16:05:18.674 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:18.664+0000 7f1c2a5c4640 1 -- 172.21.15.16:0/3331455177 shutdown_connections 2023-05-22T16:05:18.674 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:18.664+0000 7f1c2a5c4640 1 -- 172.21.15.16:0/3331455177 >> 172.21.15.16:0/3331455177 conn(0x7f1c24069720 msgr2=0x7f1c24069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:05:18.674 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:18.668+0000 7f1c2a5c4640 1 -- 172.21.15.16:0/3331455177 shutdown_connections 2023-05-22T16:05:18.674 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:18.668+0000 7f1c2a5c4640 1 -- 172.21.15.16:0/3331455177 wait complete. 2023-05-22T16:05:18.674 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:18.668+0000 7f1c2a5c4640 1 Processor -- start 2023-05-22T16:05:18.674 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:18.668+0000 7f1c2a5c4640 1 -- start start 2023-05-22T16:05:18.674 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:18.668+0000 7f1c2a5c4640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1c0c0039b0 0x7f1c2414e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:18.674 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:18.668+0000 7f1c2a5c4640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1c2414f4e0 0x7f1c24162e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:18.675 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:18.668+0000 7f1c2a5c4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c2414f930 0x7f1c24163360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:18.675 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:18.668+0000 7f1c2a5c4640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c241523b0 con 0x7f1c2414f930 2023-05-22T16:05:18.675 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:18.668+0000 7f1c2a5c4640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c24152600 con 0x7f1c0c0039b0 2023-05-22T16:05:18.675 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:18.668+0000 7f1c2a5c4640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c241528e0 con 0x7f1c2414f4e0 2023-05-22T16:05:18.675 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:18.668+0000 7f1c28b61640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c2414f930 0x7f1c24163360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:18.675 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:18.668+0000 7f1c28b61640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c2414f930 0x7f1c24163360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40236/0 (socket says 172.21.15.16:40236) 2023-05-22T16:05:18.675 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:18.668+0000 7f1c28b61640 1 -- 172.21.15.16:0/1754326517 learned_addr learned my addr 172.21.15.16:0/1754326517 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:05:18.676 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:18.668+0000 7f1c237fe640 1 --2- 172.21.15.16:0/1754326517 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1c2414f4e0 0x7f1c24162e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:18.676 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:18.668+0000 7f1c23fff640 1 --2- 172.21.15.16:0/1754326517 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1c0c0039b0 0x7f1c2414e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:18.676 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:18.668+0000 7f1c28b61640 1 -- 172.21.15.16:0/1754326517 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1c2414f4e0 msgr2=0x7f1c24162e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:18.676 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:18.668+0000 7f1c28b61640 1 --2- 172.21.15.16:0/1754326517 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1c2414f4e0 0x7f1c24162e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:18.676 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:18.668+0000 7f1c28b61640 1 -- 172.21.15.16:0/1754326517 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1c0c0039b0 msgr2=0x7f1c2414e5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:18.676 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:18.668+0000 7f1c28b61640 1 --2- 172.21.15.16:0/1754326517 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1c0c0039b0 0x7f1c2414e5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:18.676 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:18.668+0000 7f1c28b61640 1 -- 172.21.15.16:0/1754326517 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c240c8e80 con 0x7f1c2414f930 2023-05-22T16:05:18.677 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:18.668+0000 7f1c28b61640 1 --2- 172.21.15.16:0/1754326517 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c2414f930 0x7f1c24163360 secure :-1 s=READY pgs=2415 cs=0 l=1 rev1=1 crypto rx=0x7f1c240cdef0 tx=0x7f1c1c009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:18.677 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:18.668+0000 7f1c20ff9640 1 -- 172.21.15.16:0/1754326517 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c1c00e050 con 0x7f1c2414f930 2023-05-22T16:05:18.677 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:18.668+0000 7f1c20ff9640 1 -- 172.21.15.16:0/1754326517 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c1c00f040 con 0x7f1c2414f930 2023-05-22T16:05:18.677 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:18.668+0000 7f1c2a5c4640 1 -- 172.21.15.16:0/1754326517 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c240c9010 con 0x7f1c2414f930 2023-05-22T16:05:18.677 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:18.668+0000 7f1c20ff9640 1 -- 172.21.15.16:0/1754326517 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c1c003be0 con 0x7f1c2414f930 2023-05-22T16:05:18.678 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:18.668+0000 7f1c2a5c4640 1 -- 172.21.15.16:0/1754326517 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c240c91a0 con 0x7f1c2414f930 2023-05-22T16:05:18.678 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:18.668+0000 7f1c2a5c4640 1 -- 172.21.15.16:0/1754326517 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c2415e870 con 0x7f1c2414f930 2023-05-22T16:05:18.678 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:18.672+0000 7f1c20ff9640 1 -- 172.21.15.16:0/1754326517 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 92) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f1c1c015650 con 0x7f1c2414f930 2023-05-22T16:05:18.679 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:18.672+0000 7f1c20ff9640 1 --2- 172.21.15.16:0/1754326517 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f1c100654a0 0x7f1c10067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:18.679 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:18.672+0000 7f1c20ff9640 1 -- 172.21.15.16:0/1754326517 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(192..192 src has 1..192) v4 ==== 5586+0+0 (secure 0 0 0) 0x7f1c1c0827a0 con 0x7f1c2414f930 2023-05-22T16:05:18.679 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:18.672+0000 7f1c23fff640 1 --2- 172.21.15.16:0/1754326517 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f1c100654a0 0x7f1c10067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:18.679 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:18.672+0000 7f1c23fff640 1 --2- 172.21.15.16:0/1754326517 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f1c100654a0 0x7f1c10067950 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f1c140ab000 tx=0x7f1c140a7070 comp rx=0 tx=0).ready entity=mgr.10542 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:18.679 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:18.676+0000 7f1c20ff9640 1 -- 172.21.15.16:0/1754326517 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f1c2415e870 con 0x7f1c2414f930 2023-05-22T16:05:18.883 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:18.880+0000 7f1c2a5c4640 1 -- 172.21.15.16:0/1754326517 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f1c2415e870 con 0x7f1c2414f930 2023-05-22T16:05:18.884 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:18.880+0000 7f1c20ff9640 1 -- 172.21.15.16:0/1754326517 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v33) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f1c2415e870 con 0x7f1c2414f930 2023-05-22T16:05:18.884 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T16:05:18.885 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-05-22T16:05:18.885 INFO:teuthology.orchestra.run.smithi016.stdout: "auth_dump": [ 2023-05-22T16:05:18.885 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T16:05:18.885 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "mds.a", 2023-05-22T16:05:18.885 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQCAiWtkw/K0GRAA9hvTMKMmcbgKV68GeoC4ag==", 2023-05-22T16:05:18.885 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-05-22T16:05:18.885 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2023-05-22T16:05:18.886 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow *", 2023-05-22T16:05:18.886 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow *", 2023-05-22T16:05:18.886 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-05-22T16:05:18.886 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T16:05:18.886 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T16:05:18.886 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T16:05:18.886 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "mds.b", 2023-05-22T16:05:18.886 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQCAiWtkoFzfJRAAxCiB0MGM5apMVA81WDq+dA==", 2023-05-22T16:05:18.887 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-05-22T16:05:18.887 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2023-05-22T16:05:18.887 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow *", 2023-05-22T16:05:18.887 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow *", 2023-05-22T16:05:18.887 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-05-22T16:05:18.887 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T16:05:18.887 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T16:05:18.887 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T16:05:18.888 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "mds.c", 2023-05-22T16:05:18.888 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQCAiWtkvgOKHxAAMuASPtMqwyoqpZ5I+MRS7A==", 2023-05-22T16:05:18.888 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-05-22T16:05:18.888 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2023-05-22T16:05:18.888 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow *", 2023-05-22T16:05:18.888 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow *", 2023-05-22T16:05:18.888 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-05-22T16:05:18.889 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T16:05:18.889 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T16:05:18.889 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T16:05:18.889 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "mds.d", 2023-05-22T16:05:18.889 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQCAiWtk6hPQKxAAlCtfC36vja5ftz6uOqUhHg==", 2023-05-22T16:05:18.889 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-05-22T16:05:18.889 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2023-05-22T16:05:18.889 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow *", 2023-05-22T16:05:18.890 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow *", 2023-05-22T16:05:18.890 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-05-22T16:05:18.890 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T16:05:18.890 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T16:05:18.890 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T16:05:18.890 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.0", 2023-05-22T16:05:18.890 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQCCiWtk6ZLoCRAAKdCQitZuPnWcIpkiXhJeww==", 2023-05-22T16:05:18.890 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-05-22T16:05:18.891 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2023-05-22T16:05:18.891 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2023-05-22T16:05:18.891 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-05-22T16:05:18.891 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T16:05:18.891 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T16:05:18.891 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T16:05:18.891 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.1", 2023-05-22T16:05:18.892 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQCFiWtkXtVeERAAkgPeDKuFmWXrDiCyqWOYwQ==", 2023-05-22T16:05:18.892 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-05-22T16:05:18.892 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2023-05-22T16:05:18.892 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2023-05-22T16:05:18.892 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-05-22T16:05:18.892 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T16:05:18.892 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T16:05:18.892 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T16:05:18.893 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.2", 2023-05-22T16:05:18.893 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQCIiWtkdGaRHRAAx3POIp3mnz7p7XcCVMXm5Q==", 2023-05-22T16:05:18.893 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-05-22T16:05:18.893 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2023-05-22T16:05:18.893 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2023-05-22T16:05:18.893 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-05-22T16:05:18.893 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T16:05:18.893 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T16:05:18.894 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T16:05:18.894 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.3", 2023-05-22T16:05:18.894 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQCLiWtkSBydJhAAsgc19Xaf/GOyy9nm9MLdJQ==", 2023-05-22T16:05:18.894 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-05-22T16:05:18.894 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2023-05-22T16:05:18.894 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2023-05-22T16:05:18.894 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-05-22T16:05:18.895 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T16:05:18.895 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T16:05:18.895 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T16:05:18.895 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.4", 2023-05-22T16:05:18.895 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQCRiWtkueYvJRAARXbHlKHrTbgTqBArRJfvBA==", 2023-05-22T16:05:18.895 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-05-22T16:05:18.895 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2023-05-22T16:05:18.895 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2023-05-22T16:05:18.896 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-05-22T16:05:18.896 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T16:05:18.896 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T16:05:18.896 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T16:05:18.896 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.5", 2023-05-22T16:05:18.896 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQCUiWtkH3hwLRAAUekP6y+wkJVPh+OPDiZbzQ==", 2023-05-22T16:05:18.896 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-05-22T16:05:18.896 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2023-05-22T16:05:18.897 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2023-05-22T16:05:18.897 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-05-22T16:05:18.897 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T16:05:18.897 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T16:05:18.897 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T16:05:18.897 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.6", 2023-05-22T16:05:18.897 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQCXiWtkLsnBNhAAQ8ilX8ZU4ncJ2JJU0I+U4w==", 2023-05-22T16:05:18.898 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-05-22T16:05:18.898 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2023-05-22T16:05:18.898 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2023-05-22T16:05:18.898 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-05-22T16:05:18.898 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T16:05:18.898 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T16:05:18.898 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T16:05:18.898 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.7", 2023-05-22T16:05:18.899 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQCbiWtkUvXjBBAApMWw7wdfxWq0WEQDL2qprA==", 2023-05-22T16:05:18.899 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-05-22T16:05:18.899 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2023-05-22T16:05:18.899 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2023-05-22T16:05:18.899 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-05-22T16:05:18.899 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T16:05:18.899 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T16:05:18.899 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T16:05:18.900 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "client.0", 2023-05-22T16:05:18.900 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQCAiWtkZwuUMRAAnYqSFiaT/QRsajdjNgTetA==", 2023-05-22T16:05:18.900 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-05-22T16:05:18.900 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2023-05-22T16:05:18.900 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow r", 2023-05-22T16:05:18.900 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow rw tag cephfs data=cephfs" 2023-05-22T16:05:18.900 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T16:05:18.900 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T16:05:18.901 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T16:05:18.901 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "client.1", 2023-05-22T16:05:18.901 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQCAiWtkPimmORAABzNL2rANz5TgCl3p++GVng==", 2023-05-22T16:05:18.901 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-05-22T16:05:18.901 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2023-05-22T16:05:18.901 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow r", 2023-05-22T16:05:18.901 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow rw", 2023-05-22T16:05:18.902 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow rwx" 2023-05-22T16:05:18.902 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T16:05:18.902 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T16:05:18.902 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T16:05:18.902 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "client.2", 2023-05-22T16:05:18.902 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQCAiWtkqkPMNhAAqugrnD4Mh/4At8QAFmzqcg==", 2023-05-22T16:05:18.902 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-05-22T16:05:18.902 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2023-05-22T16:05:18.903 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow r", 2023-05-22T16:05:18.903 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow rw", 2023-05-22T16:05:18.903 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow rwx" 2023-05-22T16:05:18.903 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T16:05:18.903 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T16:05:18.903 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T16:05:18.903 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "client.3", 2023-05-22T16:05:18.903 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQCBiWtkpvlWAxAAPuedivXBGZGHIO6hSnz8PQ==", 2023-05-22T16:05:18.904 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-05-22T16:05:18.904 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2023-05-22T16:05:18.904 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow r", 2023-05-22T16:05:18.904 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow rw", 2023-05-22T16:05:18.904 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow rwx" 2023-05-22T16:05:18.904 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T16:05:18.904 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T16:05:18.904 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T16:05:18.905 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "client.admin", 2023-05-22T16:05:18.905 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQCAiWtkxG9hAhAAfnUnRrBRUFoCEtsb2riNOQ==", 2023-05-22T16:05:18.905 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-05-22T16:05:18.905 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow *", 2023-05-22T16:05:18.905 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow *", 2023-05-22T16:05:18.905 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow *", 2023-05-22T16:05:18.905 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-05-22T16:05:18.906 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T16:05:18.906 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T16:05:18.906 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T16:05:18.906 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "mgr.x", 2023-05-22T16:05:18.906 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQCAiWtkkaIqFhAAb/4BI6FYWOy3qOSaylXLSw==", 2023-05-22T16:05:18.906 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-05-22T16:05:18.906 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow *", 2023-05-22T16:05:18.906 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile mgr", 2023-05-22T16:05:18.907 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-05-22T16:05:18.907 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T16:05:18.907 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T16:05:18.907 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T16:05:18.907 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "mgr.y", 2023-05-22T16:05:18.907 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQCAiWtkhgVWExAAjDoh7cexFVfsigsYfGkmVg==", 2023-05-22T16:05:18.907 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-05-22T16:05:18.907 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow *", 2023-05-22T16:05:18.908 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile mgr", 2023-05-22T16:05:18.908 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-05-22T16:05:18.908 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T16:05:18.908 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T16:05:18.908 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-05-22T16:05:18.908 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-05-22T16:05:18.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:18.884+0000 7f1c2a5c4640 1 -- 172.21.15.16:0/1754326517 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f1c100654a0 msgr2=0x7f1c10067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:18.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:18.884+0000 7f1c2a5c4640 1 --2- 172.21.15.16:0/1754326517 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f1c100654a0 0x7f1c10067950 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f1c140ab000 tx=0x7f1c140a7070 comp rx=0 tx=0).stop 2023-05-22T16:05:18.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:18.884+0000 7f1c2a5c4640 1 -- 172.21.15.16:0/1754326517 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c2414f930 msgr2=0x7f1c24163360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:18.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:18.884+0000 7f1c2a5c4640 1 --2- 172.21.15.16:0/1754326517 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c2414f930 0x7f1c24163360 secure :-1 s=READY pgs=2415 cs=0 l=1 rev1=1 crypto rx=0x7f1c240cdef0 tx=0x7f1c1c009f90 comp rx=0 tx=0).stop 2023-05-22T16:05:18.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:18.884+0000 7f1c2a5c4640 1 -- 172.21.15.16:0/1754326517 shutdown_connections 2023-05-22T16:05:18.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:18.884+0000 7f1c2a5c4640 1 --2- 172.21.15.16:0/1754326517 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f1c100654a0 0x7f1c10067950 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:18.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:18.884+0000 7f1c2a5c4640 1 --2- 172.21.15.16:0/1754326517 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c2414f930 0x7f1c24163360 unknown :-1 s=CLOSED pgs=2415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:18.911 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:18.884+0000 7f1c2a5c4640 1 --2- 172.21.15.16:0/1754326517 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1c2414f4e0 0x7f1c24162e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:18.911 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:18.884+0000 7f1c2a5c4640 1 --2- 172.21.15.16:0/1754326517 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1c0c0039b0 0x7f1c2414e5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:18.911 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:18.884+0000 7f1c2a5c4640 1 -- 172.21.15.16:0/1754326517 >> 172.21.15.16:0/1754326517 conn(0x7f1c24069720 msgr2=0x7f1c240ad5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:05:18.911 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:18.884+0000 7f1c2a5c4640 1 -- 172.21.15.16:0/1754326517 shutdown_connections 2023-05-22T16:05:18.911 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:18.884+0000 7f1c2a5c4640 1 -- 172.21.15.16:0/1754326517 wait complete. 2023-05-22T16:05:18.912 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-05-22T16:05:19.027 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.024+0000 7ff881607640 1 Processor -- start 2023-05-22T16:05:19.028 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.024+0000 7ff881607640 1 -- start start 2023-05-22T16:05:19.028 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.024+0000 7ff881607640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff87c0cce20 0x7ff87c158290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:19.029 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.024+0000 7ff881607640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7ff87c0afbb0 0x7ff87c0c8880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:19.029 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.024+0000 7ff881607640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff87c0b0000 0x7ff87c0cdfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:19.030 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.024+0000 7ff881607640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff87c0c92b0 con 0x7ff87c0b0000 2023-05-22T16:05:19.030 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.024+0000 7ff881607640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff87c0cc060 con 0x7ff87c0afbb0 2023-05-22T16:05:19.030 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.024+0000 7ff881607640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff87c0cc340 con 0x7ff87c0cce20 2023-05-22T16:05:19.030 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.028+0000 7ff87b7fe640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff87c0b0000 0x7ff87c0cdfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:19.030 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.028+0000 7ff87a7fc640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7ff87c0afbb0 0x7ff87c0c8880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:19.030 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.028+0000 7ff87affd640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff87c0cce20 0x7ff87c158290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:19.030 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.028+0000 7ff87affd640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff87c0cce20 0x7ff87c158290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3301/0 says I am v2:172.21.15.16:44174/0 (socket says 172.21.15.16:44174) 2023-05-22T16:05:19.031 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.028+0000 7ff87affd640 1 -- 172.21.15.16:0/220916841 learned_addr learned my addr 172.21.15.16:0/220916841 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:05:19.031 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.028+0000 7ff87b7fe640 1 --2- 172.21.15.16:0/220916841 >> v2:172.21.15.16:3300/0 conn(0x7ff87c0b0000 0x7ff87c0cdfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40252/0 (socket says 172.21.15.16:40252) 2023-05-22T16:05:19.032 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.028+0000 7ff87b7fe640 1 -- 172.21.15.16:0/220916841 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff87c0cce20 msgr2=0x7ff87c158290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:05:19.032 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.028+0000 7ff87b7fe640 1 --2- 172.21.15.16:0/220916841 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff87c0cce20 0x7ff87c158290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:19.032 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.028+0000 7ff87b7fe640 1 -- 172.21.15.16:0/220916841 >> v2:172.21.15.62:3300/0 conn(0x7ff87c0afbb0 msgr2=0x7ff87c0c8880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:05:19.032 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.028+0000 7ff87b7fe640 1 --2- 172.21.15.16:0/220916841 >> v2:172.21.15.62:3300/0 conn(0x7ff87c0afbb0 0x7ff87c0c8880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:19.032 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.028+0000 7ff87b7fe640 1 -- 172.21.15.16:0/220916841 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff87c0cc620 con 0x7ff87c0b0000 2023-05-22T16:05:19.033 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.028+0000 7ff87b7fe640 1 --2- 172.21.15.16:0/220916841 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff87c0b0000 0x7ff87c0cdfb0 secure :-1 s=READY pgs=2416 cs=0 l=1 rev1=1 crypto rx=0x7ff87c06d720 tx=0x7ff87401c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7be3e9a37071fb16 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:19.033 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.028+0000 7ff879ffb640 1 -- 172.21.15.16:0/220916841 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8740a5980 con 0x7ff87c0b0000 2023-05-22T16:05:19.033 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.028+0000 7ff879ffb640 1 -- 172.21.15.16:0/220916841 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8740a5ae0 con 0x7ff87c0b0000 2023-05-22T16:05:19.033 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.028+0000 7ff879ffb640 1 -- 172.21.15.16:0/220916841 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8740a5e70 con 0x7ff87c0b0000 2023-05-22T16:05:19.033 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.028+0000 7ff881607640 1 -- 172.21.15.16:0/220916841 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff87c0b0000 msgr2=0x7ff87c0cdfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:19.033 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.028+0000 7ff881607640 1 --2- 172.21.15.16:0/220916841 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff87c0b0000 0x7ff87c0cdfb0 secure :-1 s=READY pgs=2416 cs=0 l=1 rev1=1 crypto rx=0x7ff87c06d720 tx=0x7ff87401c8e0 comp rx=0 tx=0).stop 2023-05-22T16:05:19.033 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.028+0000 7ff881607640 1 -- 172.21.15.16:0/220916841 shutdown_connections 2023-05-22T16:05:19.034 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.028+0000 7ff881607640 1 --2- 172.21.15.16:0/220916841 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff87c0b0000 0x7ff87c0cdfb0 unknown :-1 s=CLOSED pgs=2416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:19.034 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.028+0000 7ff881607640 1 --2- 172.21.15.16:0/220916841 >> v2:172.21.15.62:3300/0 conn(0x7ff87c0afbb0 0x7ff87c0c8880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:19.034 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.028+0000 7ff881607640 1 --2- 172.21.15.16:0/220916841 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff87c0cce20 0x7ff87c158290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:19.034 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.028+0000 7ff881607640 1 -- 172.21.15.16:0/220916841 >> 172.21.15.16:0/220916841 conn(0x7ff87c069720 msgr2=0x7ff87c069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:05:19.034 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.028+0000 7ff881607640 1 -- 172.21.15.16:0/220916841 shutdown_connections 2023-05-22T16:05:19.034 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.028+0000 7ff881607640 1 -- 172.21.15.16:0/220916841 wait complete. 2023-05-22T16:05:19.036 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.032+0000 7ff881607640 1 Processor -- start 2023-05-22T16:05:19.036 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.032+0000 7ff881607640 1 -- start start 2023-05-22T16:05:19.036 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.032+0000 7ff881607640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff87c0afbb0 0x7ff87c0c1020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:19.036 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.032+0000 7ff881607640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff87c0b0000 0x7ff87c0711a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:19.036 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.032+0000 7ff881607640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff87c0cce20 0x7ff87c0c1860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:19.037 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.032+0000 7ff881607640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff87c0ae940 con 0x7ff87c0cce20 2023-05-22T16:05:19.037 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.032+0000 7ff881607640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff87c0b28a0 con 0x7ff87c0afbb0 2023-05-22T16:05:19.037 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.032+0000 7ff881607640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff87c0b2b10 con 0x7ff87c0b0000 2023-05-22T16:05:19.037 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.032+0000 7ff87b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff87c0cce20 0x7ff87c0c1860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:19.038 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.032+0000 7ff87b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff87c0cce20 0x7ff87c0c1860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40266/0 (socket says 172.21.15.16:40266) 2023-05-22T16:05:19.038 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.032+0000 7ff87b7fe640 1 -- 172.21.15.16:0/4293345384 learned_addr learned my addr 172.21.15.16:0/4293345384 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:05:19.038 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.032+0000 7ff87a7fc640 1 --2- 172.21.15.16:0/4293345384 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff87c0b0000 0x7ff87c0711a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:19.038 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.032+0000 7ff87affd640 1 --2- 172.21.15.16:0/4293345384 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff87c0afbb0 0x7ff87c0c1020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:19.038 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.032+0000 7ff87b7fe640 1 -- 172.21.15.16:0/4293345384 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff87c0b0000 msgr2=0x7ff87c0711a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:19.038 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.032+0000 7ff87b7fe640 1 --2- 172.21.15.16:0/4293345384 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff87c0b0000 0x7ff87c0711a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:19.038 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.032+0000 7ff87b7fe640 1 -- 172.21.15.16:0/4293345384 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff87c0afbb0 msgr2=0x7ff87c0c1020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:19.039 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.032+0000 7ff87b7fe640 1 --2- 172.21.15.16:0/4293345384 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff87c0afbb0 0x7ff87c0c1020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:19.039 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.032+0000 7ff87b7fe640 1 -- 172.21.15.16:0/4293345384 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff874009a70 con 0x7ff87c0cce20 2023-05-22T16:05:19.039 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.032+0000 7ff87affd640 1 --2- 172.21.15.16:0/4293345384 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff87c0afbb0 0x7ff87c0c1020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T16:05:19.039 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.032+0000 7ff87b7fe640 1 --2- 172.21.15.16:0/4293345384 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff87c0cce20 0x7ff87c0c1860 secure :-1 s=READY pgs=2417 cs=0 l=1 rev1=1 crypto rx=0x7ff87401af90 tx=0x7ff874019680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:19.039 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.032+0000 7ff8637fe640 1 -- 172.21.15.16:0/4293345384 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8740b9860 con 0x7ff87c0cce20 2023-05-22T16:05:19.039 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.032+0000 7ff8637fe640 1 -- 172.21.15.16:0/4293345384 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8740b99c0 con 0x7ff87c0cce20 2023-05-22T16:05:19.039 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.032+0000 7ff881607640 1 -- 172.21.15.16:0/4293345384 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff87c15cd10 con 0x7ff87c0cce20 2023-05-22T16:05:19.039 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.032+0000 7ff8637fe640 1 -- 172.21.15.16:0/4293345384 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8740b9d70 con 0x7ff87c0cce20 2023-05-22T16:05:19.040 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.032+0000 7ff881607640 1 -- 172.21.15.16:0/4293345384 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff87c0ce900 con 0x7ff87c0cce20 2023-05-22T16:05:19.040 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.032+0000 7ff881607640 1 -- 172.21.15.16:0/4293345384 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff87c15cea0 con 0x7ff87c0cce20 2023-05-22T16:05:19.040 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.036+0000 7ff8637fe640 1 -- 172.21.15.16:0/4293345384 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 92) v1 ==== 84332+0+0 (secure 0 0 0) 0x7ff8740c4460 con 0x7ff87c0cce20 2023-05-22T16:05:19.041 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.036+0000 7ff8637fe640 1 --2- 172.21.15.16:0/4293345384 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7ff8680654a0 0x7ff868067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:19.041 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.036+0000 7ff8637fe640 1 -- 172.21.15.16:0/4293345384 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(192..192 src has 1..192) v4 ==== 5586+0+0 (secure 0 0 0) 0x7ff87411dbe0 con 0x7ff87c0cce20 2023-05-22T16:05:19.041 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.036+0000 7ff87affd640 1 --2- 172.21.15.16:0/4293345384 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7ff8680654a0 0x7ff868067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:19.041 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.036+0000 7ff87affd640 1 --2- 172.21.15.16:0/4293345384 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7ff8680654a0 0x7ff868067950 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7ff870009a80 tx=0x7ff870008040 comp rx=0 tx=0).ready entity=mgr.10542 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:19.041 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.040+0000 7ff8637fe640 1 -- 172.21.15.16:0/4293345384 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7ff8740eb820 con 0x7ff87c0cce20 2023-05-22T16:05:19.196 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.192+0000 7ff881607640 1 -- 172.21.15.16:0/4293345384 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff87c04dbe0 con 0x7ff87c0cce20 2023-05-22T16:05:19.198 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.196+0000 7ff8637fe640 1 -- 172.21.15.16:0/4293345384 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v192) v1 ==== 74+0+14428 (secure 0 0 0) 0x7ff8740a5940 con 0x7ff87c0cce20 2023-05-22T16:05:19.198 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T16:05:19.199 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":192,"fsid":"b2719834-7847-4d98-90a6-f65d57f85c49","created":"2023-05-22T15:26:25.259314+0000","modified":"2023-05-22T16:05:18.498827+0000","last_up_change":"2023-05-22T15:26:36.448524+0000","last_in_change":"2023-05-22T15:26:27.675668+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":23,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-05-22T15:26:38.019781+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-05-22T15:26:43.570514+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9e7ff854-b25f-4cbb-a5c2-1799d9c53d2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":37021},{"type":"v1","addr":"172.21.15.16:6825","nonce":37021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":37021},{"type":"v1","addr":"172.21.15.16:6827","nonce":37021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":37021},{"type":"v1","addr":"172.21.15.16:6831","nonce":37021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":37021},{"type":"v1","addr":"172.21.15.16:6829","nonce":37021}]},"public_addr":"172.21.15.16:6825/37021","cluster_addr":"172.21.15.16:6827/37021","heartbeat_back_addr":"172.21.15.16:6831/37021","heartbeat_front_addr":"172.21.15.16:6829/37021","state":["exists","up"]},{"osd":1,"uuid":"4702fbc9-a927-4565-aca5-41dc69cc8264","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":37008},{"type":"v1","addr":"172.21.15.16:6801","nonce":37008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":37008},{"type":"v1","addr":"172.21.15.16:6803","nonce":37008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":37008},{"type":"v1","addr":"172.21.15.16:6807","nonce":37008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":37008},{"type":"v1","addr":"172.21.15.16:6805","nonce":37008}]},"public_addr":"172.21.15.16:6801/37008","cluster_addr":"172.21.15.16:6803/37008","heartbeat_back_addr":"172.21.15.16:6807/37008","heartbeat_front_addr":"172.21.15.16:6805/37008","state":["exists","up"]},{"osd":2,"uuid":"1e7bba46-4eba-47ef-bb81-5e2d371638ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":37009},{"type":"v1","addr":"172.21.15.16:6809","nonce":37009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":37009},{"type":"v1","addr":"172.21.15.16:6811","nonce":37009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":37009},{"type":"v1","addr":"172.21.15.16:6815","nonce":37009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":37009},{"type":"v1","addr":"172.21.15.16:6813","nonce":37009}]},"public_addr":"172.21.15.16:6809/37009","cluster_addr":"172.21.15.16:6811/37009","heartbeat_back_addr":"172.21.15.16:6815/37009","heartbeat_front_addr":"172.21.15.16:6813/37009","state":["exists","up"]},{"osd":3,"uuid":"383cacc0-5a88-4904-990e-498ca59ef48a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":37020},{"type":"v1","addr":"172.21.15.16:6817","nonce":37020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":37020},{"type":"v1","addr":"172.21.15.16:6819","nonce":37020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":37020},{"type":"v1","addr":"172.21.15.16:6823","nonce":37020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":37020},{"type":"v1","addr":"172.21.15.16:6821","nonce":37020}]},"public_addr":"172.21.15.16:6817/37020","cluster_addr":"172.21.15.16:6819/37020","heartbeat_back_addr":"172.21.15.16:6823/37020","heartbeat_front_addr":"172.21.15.16:6821/37020","state":["exists","up"]},{"osd":4,"uuid":"970500ef-95b7-42c1-b080-eed4fae7013d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":37353},{"type":"v1","addr":"172.21.15.62:6817","nonce":37353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6819","nonce":37353},{"type":"v1","addr":"172.21.15.62:6822","nonce":37353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37353},{"type":"v1","addr":"172.21.15.62:6831","nonce":37353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37353},{"type":"v1","addr":"172.21.15.62:6826","nonce":37353}]},"public_addr":"172.21.15.62:6817/37353","cluster_addr":"172.21.15.62:6822/37353","heartbeat_back_addr":"172.21.15.62:6831/37353","heartbeat_front_addr":"172.21.15.62:6826/37353","state":["exists","up"]},{"osd":5,"uuid":"022d67db-d4d8-445b-b1b9-8b446a00f5c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":37352},{"type":"v1","addr":"172.21.15.62:6801","nonce":37352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":37352},{"type":"v1","addr":"172.21.15.62:6803","nonce":37352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":37352},{"type":"v1","addr":"172.21.15.62:6807","nonce":37352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":37352},{"type":"v1","addr":"172.21.15.62:6805","nonce":37352}]},"public_addr":"172.21.15.62:6801/37352","cluster_addr":"172.21.15.62:6803/37352","heartbeat_back_addr":"172.21.15.62:6807/37352","heartbeat_front_addr":"172.21.15.62:6805/37352","state":["exists","up"]},{"osd":6,"uuid":"42bd372b-6688-4a1b-8407-e16e8c266e33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":37354},{"type":"v1","addr":"172.21.15.62:6820","nonce":37354}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":37354},{"type":"v1","addr":"172.21.15.62:6823","nonce":37354}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6829","nonce":37354},{"type":"v1","addr":"172.21.15.62:6830","nonce":37354}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6825","nonce":37354},{"type":"v1","addr":"172.21.15.62:6827","nonce":37354}]},"public_addr":"172.21.15.62:6820/37354","cluster_addr":"172.21.15.62:6823/37354","heartbeat_back_addr":"172.21.15.62:6830/37354","heartbeat_front_addr":"172.21.15.62:6827/37354","state":["exists","up"]},{"osd":7,"uuid":"ac70b2a3-f41b-4bf2-b227-7f2cbb55426d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37355},{"type":"v1","addr":"172.21.15.62:6809","nonce":37355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37355},{"type":"v1","addr":"172.21.15.62:6811","nonce":37355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":37355},{"type":"v1","addr":"172.21.15.62:6815","nonce":37355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":37355},{"type":"v1","addr":"172.21.15.62:6813","nonce":37355}]},"public_addr":"172.21.15.62:6809/37355","cluster_addr":"172.21.15.62:6811/37355","heartbeat_back_addr":"172.21.15.62:6815/37355","heartbeat_front_addr":"172.21.15.62:6813/37355","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.368896+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.107273+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.276888+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.287502+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.124791+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.251937+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.255679+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.232797+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":17,"snaps":[{"begin":2,"length":2}]},{"pool":21,"snaps":[{"begin":25,"length":9}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-05-22T16:05:19.200 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.196+0000 7ff881607640 1 -- 172.21.15.16:0/4293345384 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7ff8680654a0 msgr2=0x7ff868067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:19.200 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.196+0000 7ff881607640 1 --2- 172.21.15.16:0/4293345384 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7ff8680654a0 0x7ff868067950 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7ff870009a80 tx=0x7ff870008040 comp rx=0 tx=0).stop 2023-05-22T16:05:19.201 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.196+0000 7ff881607640 1 -- 172.21.15.16:0/4293345384 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff87c0cce20 msgr2=0x7ff87c0c1860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:19.201 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.196+0000 7ff881607640 1 --2- 172.21.15.16:0/4293345384 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff87c0cce20 0x7ff87c0c1860 secure :-1 s=READY pgs=2417 cs=0 l=1 rev1=1 crypto rx=0x7ff87401af90 tx=0x7ff874019680 comp rx=0 tx=0).stop 2023-05-22T16:05:19.201 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.200+0000 7ff881607640 1 -- 172.21.15.16:0/4293345384 shutdown_connections 2023-05-22T16:05:19.201 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.200+0000 7ff881607640 1 --2- 172.21.15.16:0/4293345384 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7ff8680654a0 0x7ff868067950 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:19.201 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.200+0000 7ff881607640 1 --2- 172.21.15.16:0/4293345384 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff87c0cce20 0x7ff87c0c1860 unknown :-1 s=CLOSED pgs=2417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:19.201 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.200+0000 7ff881607640 1 --2- 172.21.15.16:0/4293345384 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff87c0b0000 0x7ff87c0711a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:19.202 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.200+0000 7ff881607640 1 --2- 172.21.15.16:0/4293345384 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff87c0afbb0 0x7ff87c0c1020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:19.202 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.200+0000 7ff881607640 1 -- 172.21.15.16:0/4293345384 >> 172.21.15.16:0/4293345384 conn(0x7ff87c069720 msgr2=0x7ff87c06f610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:05:19.202 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.200+0000 7ff881607640 1 -- 172.21.15.16:0/4293345384 shutdown_connections 2023-05-22T16:05:19.202 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.200+0000 7ff881607640 1 -- 172.21.15.16:0/4293345384 wait complete. 2023-05-22T16:05:19.220 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-05-22T15:26:38.019781+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-05-22T15:26:43.570514+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-05-22T16:05:19.221 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-05-22T16:05:19.343 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.340+0000 7f3094484640 1 Processor -- start 2023-05-22T16:05:19.343 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.340+0000 7f3094484640 1 -- start start 2023-05-22T16:05:19.344 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.340+0000 7f3094484640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f308c158210 0x7f308c159160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:19.344 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.340+0000 7f3094484640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f308c15bcd0 0x7f308c15c120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:19.345 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.340+0000 7f3094484640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f308c158cd0 con 0x7f308c0c8660 2023-05-22T16:05:19.345 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.340+0000 7f3094484640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f308c1596a0 con 0x7f308c158210 2023-05-22T16:05:19.345 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.340+0000 7f3094484640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f308c15c790 con 0x7f308c15bcd0 2023-05-22T16:05:19.345 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.340+0000 7f3092220640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f308c158210 0x7f308c159160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:19.345 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.340+0000 7f3092220640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f308c158210 0x7f308c159160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41518/0 (socket says 172.21.15.16:41518) 2023-05-22T16:05:19.345 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.340+0000 7f3092220640 1 -- 172.21.15.16:0/573009046 learned_addr learned my addr 172.21.15.16:0/573009046 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:05:19.345 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.340+0000 7f3091a1f640 1 --2- 172.21.15.16:0/573009046 >> v2:172.21.15.62:3300/0 conn(0x7f308c15bcd0 0x7f308c15c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:19.346 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.340+0000 7f3092220640 1 -- 172.21.15.16:0/573009046 >> v2:172.21.15.62:3300/0 conn(0x7f308c15bcd0 msgr2=0x7f308c15c120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:05:19.346 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.340+0000 7f3092220640 1 --2- 172.21.15.16:0/573009046 >> v2:172.21.15.62:3300/0 conn(0x7f308c15bcd0 0x7f308c15c120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:19.346 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.340+0000 7f3092220640 1 -- 172.21.15.16:0/573009046 >> v1:172.21.15.62:6789/0 conn(0x7f308c0c8660 legacy=0x7f308c0ca690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:05:19.346 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.340+0000 7f3092220640 1 -- 172.21.15.16:0/573009046 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f308c15ca70 con 0x7f308c158210 2023-05-22T16:05:19.346 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.340+0000 7f3091a1f640 1 --2- 172.21.15.16:0/573009046 >> v2:172.21.15.62:3300/0 conn(0x7f308c15bcd0 0x7f308c15c120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T16:05:19.351 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.344+0000 7f3092220640 1 --2- 172.21.15.16:0/573009046 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f308c158210 0x7f308c159160 secure :-1 s=READY pgs=2418 cs=0 l=1 rev1=1 crypto rx=0x7f308c0cbb60 tx=0x7f308801e8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27b3469d36f56c67 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:19.351 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.344+0000 7f309121e640 1 -- 172.21.15.16:0/573009046 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3088004230 con 0x7f308c158210 2023-05-22T16:05:19.351 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.344+0000 7f309121e640 1 -- 172.21.15.16:0/573009046 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30880ab070 con 0x7f308c158210 2023-05-22T16:05:19.351 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.344+0000 7f309121e640 1 -- 172.21.15.16:0/573009046 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30880a7710 con 0x7f308c158210 2023-05-22T16:05:19.351 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.344+0000 7f3094484640 1 -- 172.21.15.16:0/573009046 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f308c158210 msgr2=0x7f308c159160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:19.351 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.344+0000 7f3094484640 1 --2- 172.21.15.16:0/573009046 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f308c158210 0x7f308c159160 secure :-1 s=READY pgs=2418 cs=0 l=1 rev1=1 crypto rx=0x7f308c0cbb60 tx=0x7f308801e8b0 comp rx=0 tx=0).stop 2023-05-22T16:05:19.351 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.344+0000 7f3094484640 1 -- 172.21.15.16:0/573009046 shutdown_connections 2023-05-22T16:05:19.352 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.344+0000 7f3094484640 1 --2- 172.21.15.16:0/573009046 >> v2:172.21.15.62:3300/0 conn(0x7f308c15bcd0 0x7f308c15c120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:19.352 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.344+0000 7f3094484640 1 --2- 172.21.15.16:0/573009046 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f308c158210 0x7f308c159160 unknown :-1 s=CLOSED pgs=2418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:19.352 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.344+0000 7f3094484640 1 -- 172.21.15.16:0/573009046 >> 172.21.15.16:0/573009046 conn(0x7f308c069720 msgr2=0x7f308c069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:05:19.352 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.344+0000 7f3094484640 1 -- 172.21.15.16:0/573009046 shutdown_connections 2023-05-22T16:05:19.352 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.344+0000 7f3094484640 1 -- 172.21.15.16:0/573009046 wait complete. 2023-05-22T16:05:19.352 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.348+0000 7f3094484640 1 Processor -- start 2023-05-22T16:05:19.352 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.348+0000 7f3094484640 1 -- start start 2023-05-22T16:05:19.353 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.348+0000 7f3094484640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f308c0c8660 0x7f308c152ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:19.353 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.348+0000 7f3094484640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f308c158210 0x7f308c153160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:19.353 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.348+0000 7f3094484640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f308c15bcd0 0x7f308c0b3300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:19.353 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.348+0000 7f3094484640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f308c153000 con 0x7f308c0c8660 2023-05-22T16:05:19.353 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.348+0000 7f3094484640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f308c0c77c0 con 0x7f308c158210 2023-05-22T16:05:19.353 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.348+0000 7f3094484640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f308c1610c0 con 0x7f308c15bcd0 2023-05-22T16:05:19.353 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.348+0000 7f3092220640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f308c0c8660 0x7f308c152ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:19.353 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.348+0000 7f3092220640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f308c0c8660 0x7f308c152ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41524/0 (socket says 172.21.15.16:41524) 2023-05-22T16:05:19.354 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.348+0000 7f3092220640 1 -- 172.21.15.16:0/2293283387 learned_addr learned my addr 172.21.15.16:0/2293283387 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:05:19.354 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.348+0000 7f3091a1f640 1 --2- 172.21.15.16:0/2293283387 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f308c158210 0x7f308c153160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:19.354 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.348+0000 7f3092a21640 1 --2- 172.21.15.16:0/2293283387 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f308c15bcd0 0x7f308c0b3300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:19.354 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.348+0000 7f3092220640 1 -- 172.21.15.16:0/2293283387 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f308c15bcd0 msgr2=0x7f308c0b3300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:19.354 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.348+0000 7f3092220640 1 --2- 172.21.15.16:0/2293283387 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f308c15bcd0 0x7f308c0b3300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:19.354 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.348+0000 7f3092220640 1 -- 172.21.15.16:0/2293283387 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f308c158210 msgr2=0x7f308c153160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:19.354 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.348+0000 7f3092220640 1 --2- 172.21.15.16:0/2293283387 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f308c158210 0x7f308c153160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:19.355 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.348+0000 7f3092220640 1 -- 172.21.15.16:0/2293283387 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f308800b780 con 0x7f308c0c8660 2023-05-22T16:05:19.355 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.348+0000 7f3091a1f640 1 --2- 172.21.15.16:0/2293283387 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f308c158210 0x7f308c153160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T16:05:19.355 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.348+0000 7f3092220640 1 --2- 172.21.15.16:0/2293283387 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f308c0c8660 0x7f308c152ac0 secure :-1 s=READY pgs=2419 cs=0 l=1 rev1=1 crypto rx=0x7f308801c4e0 tx=0x7f308801b650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:19.355 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.348+0000 7f3082ffd640 1 -- 172.21.15.16:0/2293283387 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30880a7b20 con 0x7f308c0c8660 2023-05-22T16:05:19.355 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.348+0000 7f3082ffd640 1 -- 172.21.15.16:0/2293283387 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30880a7c80 con 0x7f308c0c8660 2023-05-22T16:05:19.355 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.348+0000 7f3094484640 1 -- 172.21.15.16:0/2293283387 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f308c1585e0 con 0x7f308c0c8660 2023-05-22T16:05:19.355 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.348+0000 7f3082ffd640 1 -- 172.21.15.16:0/2293283387 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3088003620 con 0x7f308c0c8660 2023-05-22T16:05:19.355 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.348+0000 7f3094484640 1 -- 172.21.15.16:0/2293283387 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f308c0cd980 con 0x7f308c0c8660 2023-05-22T16:05:19.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.348+0000 7f3094484640 1 -- 172.21.15.16:0/2293283387 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f308c0cb1b0 con 0x7f308c0c8660 2023-05-22T16:05:19.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.348+0000 7f3082ffd640 1 -- 172.21.15.16:0/2293283387 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 92) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f3088003780 con 0x7f308c0c8660 2023-05-22T16:05:19.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.348+0000 7f3082ffd640 1 --2- 172.21.15.16:0/2293283387 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f30700654a0 0x7f3070067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:19.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.348+0000 7f3082ffd640 1 -- 172.21.15.16:0/2293283387 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(192..192 src has 1..192) v4 ==== 5586+0+0 (secure 0 0 0) 0x7f308811d110 con 0x7f308c0c8660 2023-05-22T16:05:19.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.348+0000 7f3091a1f640 1 --2- 172.21.15.16:0/2293283387 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f30700654a0 0x7f3070067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:19.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.352+0000 7f3091a1f640 1 --2- 172.21.15.16:0/2293283387 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f30700654a0 0x7f3070067950 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f308c0ad9d0 tx=0x7f307c008040 comp rx=0 tx=0).ready entity=mgr.10542 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:19.358 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.352+0000 7f3082ffd640 1 -- 172.21.15.16:0/2293283387 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f30880eae50 con 0x7f308c0c8660 2023-05-22T16:05:19.509 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.508+0000 7f3094484640 1 -- 172.21.15.16:0/2293283387 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f308c0cb1b0 con 0x7f308c0c8660 2023-05-22T16:05:19.509 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.508+0000 7f3082ffd640 1 -- 172.21.15.16:0/2293283387 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v192) v1 ==== 93+0+10 (secure 0 0 0) 0x7f30880ee470 con 0x7f308c0c8660 2023-05-22T16:05:19.510 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2023-05-22T16:05:19.511 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.508+0000 7f3094484640 1 -- 172.21.15.16:0/2293283387 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f30700654a0 msgr2=0x7f3070067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:19.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.508+0000 7f3094484640 1 --2- 172.21.15.16:0/2293283387 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f30700654a0 0x7f3070067950 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f308c0ad9d0 tx=0x7f307c008040 comp rx=0 tx=0).stop 2023-05-22T16:05:19.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.508+0000 7f3094484640 1 -- 172.21.15.16:0/2293283387 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f308c0c8660 msgr2=0x7f308c152ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:19.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.508+0000 7f3094484640 1 --2- 172.21.15.16:0/2293283387 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f308c0c8660 0x7f308c152ac0 secure :-1 s=READY pgs=2419 cs=0 l=1 rev1=1 crypto rx=0x7f308801c4e0 tx=0x7f308801b650 comp rx=0 tx=0).stop 2023-05-22T16:05:19.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.508+0000 7f3094484640 1 -- 172.21.15.16:0/2293283387 shutdown_connections 2023-05-22T16:05:19.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.508+0000 7f3094484640 1 --2- 172.21.15.16:0/2293283387 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f308c15bcd0 0x7f308c0b3300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:19.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.508+0000 7f3094484640 1 --2- 172.21.15.16:0/2293283387 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f308c158210 0x7f308c153160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:19.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.508+0000 7f3094484640 1 --2- 172.21.15.16:0/2293283387 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f30700654a0 0x7f3070067950 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:19.513 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.508+0000 7f3094484640 1 --2- 172.21.15.16:0/2293283387 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f308c0c8660 0x7f308c152ac0 unknown :-1 s=CLOSED pgs=2419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:19.513 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.508+0000 7f3094484640 1 -- 172.21.15.16:0/2293283387 >> 172.21.15.16:0/2293283387 conn(0x7f308c069720 msgr2=0x7f308c154730 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:05:19.513 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.508+0000 7f3094484640 1 -- 172.21.15.16:0/2293283387 shutdown_connections 2023-05-22T16:05:19.513 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.508+0000 7f3094484640 1 -- 172.21.15.16:0/2293283387 wait complete. 2023-05-22T16:05:19.528 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-05-22T16:05:19.648 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.648+0000 7f03b4a44640 1 Processor -- start 2023-05-22T16:05:19.649 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.648+0000 7f03b4a44640 1 -- start start 2023-05-22T16:05:19.649 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.648+0000 7f03b4a44640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f03b0070c70 0x7f03b0071530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:19.650 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.648+0000 7f03b4a44640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f03b006b6c0 0x7f03b006bb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:19.650 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.648+0000 7f03b4a44640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03b015d520 con 0x7f03b015c580 2023-05-22T16:05:19.650 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.648+0000 7f03b4a44640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03b00c8b10 con 0x7f03b006b6c0 2023-05-22T16:05:19.650 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.648+0000 7f03b4a44640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03b015c9d0 con 0x7f03b0070c70 2023-05-22T16:05:19.650 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.648+0000 7f03aed9d640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f03b006b6c0 0x7f03b006bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:19.650 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.648+0000 7f03aed9d640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f03b006b6c0 0x7f03b006bb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41528/0 (socket says 172.21.15.16:41528) 2023-05-22T16:05:19.651 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.648+0000 7f03aed9d640 1 -- 172.21.15.16:0/2110675566 learned_addr learned my addr 172.21.15.16:0/2110675566 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:05:19.651 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.648+0000 7f03ae59c640 1 --2- 172.21.15.16:0/2110675566 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f03b0070c70 0x7f03b0071530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:19.651 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.648+0000 7f03aed9d640 1 -- 172.21.15.16:0/2110675566 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f03b0070c70 msgr2=0x7f03b0071530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:05:19.652 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.648+0000 7f03aed9d640 1 --2- 172.21.15.16:0/2110675566 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f03b0070c70 0x7f03b0071530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:19.652 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.648+0000 7f03aed9d640 1 -- 172.21.15.16:0/2110675566 >> v1:172.21.15.62:6789/0 conn(0x7f03b015c580 legacy=0x7f03b006fba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:05:19.652 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.648+0000 7f03aed9d640 1 -- 172.21.15.16:0/2110675566 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03b006c060 con 0x7f03b006b6c0 2023-05-22T16:05:19.653 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.648+0000 7f03aed9d640 1 --2- 172.21.15.16:0/2110675566 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f03b006b6c0 0x7f03b006bb20 secure :-1 s=READY pgs=2420 cs=0 l=1 rev1=1 crypto rx=0x7f03a8009e50 tx=0x7f03a801c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8581304a331027e0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:19.653 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.652+0000 7f03ad59a640 1 -- 172.21.15.16:0/2110675566 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03a80a5790 con 0x7f03b006b6c0 2023-05-22T16:05:19.653 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.652+0000 7f03ad59a640 1 -- 172.21.15.16:0/2110675566 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03a80a58f0 con 0x7f03b006b6c0 2023-05-22T16:05:19.653 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.652+0000 7f03ad59a640 1 -- 172.21.15.16:0/2110675566 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03a80a5c80 con 0x7f03b006b6c0 2023-05-22T16:05:19.654 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.652+0000 7f03b4a44640 1 -- 172.21.15.16:0/2110675566 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f03b006b6c0 msgr2=0x7f03b006bb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:19.654 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.652+0000 7f03b4a44640 1 --2- 172.21.15.16:0/2110675566 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f03b006b6c0 0x7f03b006bb20 secure :-1 s=READY pgs=2420 cs=0 l=1 rev1=1 crypto rx=0x7f03a8009e50 tx=0x7f03a801c910 comp rx=0 tx=0).stop 2023-05-22T16:05:19.654 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.652+0000 7f03b4a44640 1 -- 172.21.15.16:0/2110675566 shutdown_connections 2023-05-22T16:05:19.654 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.652+0000 7f03b4a44640 1 --2- 172.21.15.16:0/2110675566 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f03b006b6c0 0x7f03b006bb20 unknown :-1 s=CLOSED pgs=2420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:19.654 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.652+0000 7f03b4a44640 1 --2- 172.21.15.16:0/2110675566 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f03b0070c70 0x7f03b0071530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:19.654 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.652+0000 7f03b4a44640 1 -- 172.21.15.16:0/2110675566 >> 172.21.15.16:0/2110675566 conn(0x7f03b0069720 msgr2=0x7f03b0069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:05:19.655 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.652+0000 7f03b4a44640 1 -- 172.21.15.16:0/2110675566 shutdown_connections 2023-05-22T16:05:19.655 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.652+0000 7f03b4a44640 1 -- 172.21.15.16:0/2110675566 wait complete. 2023-05-22T16:05:19.657 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.652+0000 7f03b4a44640 1 Processor -- start 2023-05-22T16:05:19.657 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.652+0000 7f03b4a44640 1 -- start start 2023-05-22T16:05:19.657 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.652+0000 7f03b4a44640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f03b006b6c0 0x7f03b0159010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:19.657 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.652+0000 7f03b4a44640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f03b0070c70 0x7f03b014b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:19.657 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.652+0000 7f03b4a44640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f03b015c580 0x7f03b00adfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:19.658 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.652+0000 7f03b4a44640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03b0157e60 con 0x7f03b015c580 2023-05-22T16:05:19.658 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.652+0000 7f03b4a44640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03b015aae0 con 0x7f03b006b6c0 2023-05-22T16:05:19.658 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.652+0000 7f03b4a44640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03b015ad80 con 0x7f03b0070c70 2023-05-22T16:05:19.658 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.652+0000 7f03aed9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f03b015c580 0x7f03b00adfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:19.658 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.652+0000 7f03ae59c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f03b006b6c0 0x7f03b0159010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:19.658 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.652+0000 7f03aed9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f03b015c580 0x7f03b00adfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41544/0 (socket says 172.21.15.16:41544) 2023-05-22T16:05:19.658 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.652+0000 7f03aed9d640 1 -- 172.21.15.16:0/1096820915 learned_addr learned my addr 172.21.15.16:0/1096820915 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:05:19.658 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.652+0000 7f03add9b640 1 --2- 172.21.15.16:0/1096820915 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f03b0070c70 0x7f03b014b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:19.659 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.652+0000 7f03aed9d640 1 -- 172.21.15.16:0/1096820915 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f03b0070c70 msgr2=0x7f03b014b910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:19.659 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.652+0000 7f03aed9d640 1 --2- 172.21.15.16:0/1096820915 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f03b0070c70 0x7f03b014b910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:19.659 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.652+0000 7f03aed9d640 1 -- 172.21.15.16:0/1096820915 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f03b006b6c0 msgr2=0x7f03b0159010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:19.659 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.652+0000 7f03aed9d640 1 --2- 172.21.15.16:0/1096820915 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f03b006b6c0 0x7f03b0159010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:19.659 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.652+0000 7f03aed9d640 1 -- 172.21.15.16:0/1096820915 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03a8009b30 con 0x7f03b015c580 2023-05-22T16:05:19.659 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.652+0000 7f03aed9d640 1 --2- 172.21.15.16:0/1096820915 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f03b015c580 0x7f03b00adfe0 secure :-1 s=READY pgs=2421 cs=0 l=1 rev1=1 crypto rx=0x7f03a801afc0 tx=0x7f03a80196b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:19.659 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.656+0000 7f039effd640 1 -- 172.21.15.16:0/1096820915 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03a80b8670 con 0x7f03b015c580 2023-05-22T16:05:19.660 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.656+0000 7f039effd640 1 -- 172.21.15.16:0/1096820915 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03a80b87d0 con 0x7f03b015c580 2023-05-22T16:05:19.660 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.656+0000 7f03b4a44640 1 -- 172.21.15.16:0/1096820915 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03b00cc510 con 0x7f03b015c580 2023-05-22T16:05:19.660 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.656+0000 7f039effd640 1 -- 172.21.15.16:0/1096820915 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03a80b8b60 con 0x7f03b015c580 2023-05-22T16:05:19.660 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.656+0000 7f03b4a44640 1 -- 172.21.15.16:0/1096820915 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03b015d240 con 0x7f03b015c580 2023-05-22T16:05:19.661 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.656+0000 7f03b4a44640 1 -- 172.21.15.16:0/1096820915 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03b0071040 con 0x7f03b015c580 2023-05-22T16:05:19.661 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.656+0000 7f039effd640 1 -- 172.21.15.16:0/1096820915 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 92) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f03a80b8cc0 con 0x7f03b015c580 2023-05-22T16:05:19.661 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.656+0000 7f039effd640 1 --2- 172.21.15.16:0/1096820915 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f03900654a0 0x7f0390067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:19.661 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.656+0000 7f039effd640 1 -- 172.21.15.16:0/1096820915 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(192..192 src has 1..192) v4 ==== 5586+0+0 (secure 0 0 0) 0x7f03a811dd10 con 0x7f03b015c580 2023-05-22T16:05:19.661 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.656+0000 7f03ae59c640 1 --2- 172.21.15.16:0/1096820915 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f03900654a0 0x7f0390067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:19.661 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.656+0000 7f03ae59c640 1 --2- 172.21.15.16:0/1096820915 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f03900654a0 0x7f0390067950 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f03b00cb3e0 tx=0x7f03a0008040 comp rx=0 tx=0).ready entity=mgr.10542 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:19.662 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.660+0000 7f039effd640 1 -- 172.21.15.16:0/1096820915 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f03a8123020 con 0x7f03b015c580 2023-05-22T16:05:19.819 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.816+0000 7f03b4a44640 1 -- 172.21.15.16:0/1096820915 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f03b004dbe0 con 0x7f03b015c580 2023-05-22T16:05:19.820 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.816+0000 7f039effd640 1 -- 172.21.15.16:0/1096820915 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v192) v1 ==== 92+0+10 (secure 0 0 0) 0x7f03a80ef5e0 con 0x7f03b015c580 2023-05-22T16:05:19.820 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 8 2023-05-22T16:05:19.822 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.820+0000 7f03b4a44640 1 -- 172.21.15.16:0/1096820915 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f03900654a0 msgr2=0x7f0390067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:19.823 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.820+0000 7f03b4a44640 1 --2- 172.21.15.16:0/1096820915 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f03900654a0 0x7f0390067950 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f03b00cb3e0 tx=0x7f03a0008040 comp rx=0 tx=0).stop 2023-05-22T16:05:19.823 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.820+0000 7f03b4a44640 1 -- 172.21.15.16:0/1096820915 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f03b015c580 msgr2=0x7f03b00adfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:19.823 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.820+0000 7f03b4a44640 1 --2- 172.21.15.16:0/1096820915 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f03b015c580 0x7f03b00adfe0 secure :-1 s=READY pgs=2421 cs=0 l=1 rev1=1 crypto rx=0x7f03a801afc0 tx=0x7f03a80196b0 comp rx=0 tx=0).stop 2023-05-22T16:05:19.823 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.820+0000 7f03b4a44640 1 -- 172.21.15.16:0/1096820915 shutdown_connections 2023-05-22T16:05:19.823 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.820+0000 7f03b4a44640 1 --2- 172.21.15.16:0/1096820915 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f03900654a0 0x7f0390067950 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:19.823 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.820+0000 7f03b4a44640 1 --2- 172.21.15.16:0/1096820915 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f03b015c580 0x7f03b00adfe0 unknown :-1 s=CLOSED pgs=2421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:19.823 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.820+0000 7f03b4a44640 1 --2- 172.21.15.16:0/1096820915 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f03b0070c70 0x7f03b014b910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:19.824 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.820+0000 7f03b4a44640 1 --2- 172.21.15.16:0/1096820915 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f03b006b6c0 0x7f03b0159010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:19.824 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.820+0000 7f03b4a44640 1 -- 172.21.15.16:0/1096820915 >> 172.21.15.16:0/1096820915 conn(0x7f03b0069720 msgr2=0x7f03b006fba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:05:19.824 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.820+0000 7f03b4a44640 1 -- 172.21.15.16:0/1096820915 shutdown_connections 2023-05-22T16:05:19.824 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.820+0000 7f03b4a44640 1 -- 172.21.15.16:0/1096820915 wait complete. 2023-05-22T16:05:19.839 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-05-22T16:05:19.959 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.956+0000 7f5c535fe640 1 Processor -- start 2023-05-22T16:05:19.959 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.956+0000 7f5c535fe640 1 -- start start 2023-05-22T16:05:19.959 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.956+0000 7f5c535fe640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f5c4c159a40 0x7f5c4c0c8ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:19.959 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.956+0000 7f5c535fe640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c4c15a0a0 con 0x7f5c4c0b0250 2023-05-22T16:05:19.960 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.956+0000 7f5c535fe640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c4c153580 con 0x7f5c4c1531b0 2023-05-22T16:05:19.960 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.956+0000 7f5c535fe640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c4c159e90 con 0x7f5c4c159a40 2023-05-22T16:05:19.960 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.956+0000 7f5c51b9b640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f5c4c159a40 0x7f5c4c0c8ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:19.960 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.956+0000 7f5c51b9b640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f5c4c159a40 0x7f5c4c0c8ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41548/0 (socket says 172.21.15.16:41548) 2023-05-22T16:05:19.960 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.956+0000 7f5c51b9b640 1 -- 172.21.15.16:0/711141411 learned_addr learned my addr 172.21.15.16:0/711141411 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:05:19.960 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.956+0000 7f5c50b99640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f5c4c0b0250 0x7f5c4c159330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:48422/0 (socket says 172.21.15.16:48422) 2023-05-22T16:05:19.960 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.956+0000 7f5c43fff640 1 -- 172.21.15.16:0/711141411 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 418981437 0 0) 0x7f5c4c15a0a0 con 0x7f5c4c0b0250 2023-05-22T16:05:19.961 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.956+0000 7f5c51b9b640 1 -- 172.21.15.16:0/711141411 >> v1:172.21.15.62:6789/0 conn(0x7f5c4c1531b0 legacy=0x7f5c4c0afb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:19.961 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.956+0000 7f5c51b9b640 1 -- 172.21.15.16:0/711141411 >> v1:172.21.15.16:6789/0 conn(0x7f5c4c0b0250 legacy=0x7f5c4c159330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:19.961 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.956+0000 7f5c51b9b640 1 -- 172.21.15.16:0/711141411 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c4c15d5e0 con 0x7f5c4c159a40 2023-05-22T16:05:19.962 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.960+0000 7f5c51b9b640 1 --2- 172.21.15.16:0/711141411 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5c4c159a40 0x7f5c4c0c8ba0 secure :-1 s=READY pgs=2423 cs=0 l=1 rev1=1 crypto rx=0x7f5c44009e50 tx=0x7f5c4401c9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8f552eb205b978d server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:19.962 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.960+0000 7f5c43fff640 1 -- 172.21.15.16:0/711141411 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c440a58a0 con 0x7f5c4c159a40 2023-05-22T16:05:19.962 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.960+0000 7f5c43fff640 1 -- 172.21.15.16:0/711141411 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c440a5a00 con 0x7f5c4c159a40 2023-05-22T16:05:19.962 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.960+0000 7f5c43fff640 1 -- 172.21.15.16:0/711141411 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c440a5d90 con 0x7f5c4c159a40 2023-05-22T16:05:19.962 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.960+0000 7f5c535fe640 1 -- 172.21.15.16:0/711141411 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5c4c159a40 msgr2=0x7f5c4c0c8ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:19.963 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.960+0000 7f5c535fe640 1 --2- 172.21.15.16:0/711141411 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5c4c159a40 0x7f5c4c0c8ba0 secure :-1 s=READY pgs=2423 cs=0 l=1 rev1=1 crypto rx=0x7f5c44009e50 tx=0x7f5c4401c9e0 comp rx=0 tx=0).stop 2023-05-22T16:05:19.963 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.960+0000 7f5c535fe640 1 -- 172.21.15.16:0/711141411 shutdown_connections 2023-05-22T16:05:19.963 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.960+0000 7f5c535fe640 1 --2- 172.21.15.16:0/711141411 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5c4c159a40 0x7f5c4c0c8ba0 unknown :-1 s=CLOSED pgs=2423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:19.963 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.960+0000 7f5c535fe640 1 -- 172.21.15.16:0/711141411 >> 172.21.15.16:0/711141411 conn(0x7f5c4c069720 msgr2=0x7f5c4c069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:05:19.963 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.960+0000 7f5c535fe640 1 -- 172.21.15.16:0/711141411 shutdown_connections 2023-05-22T16:05:19.963 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.960+0000 7f5c535fe640 1 -- 172.21.15.16:0/711141411 wait complete. 2023-05-22T16:05:19.965 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.960+0000 7f5c535fe640 1 Processor -- start 2023-05-22T16:05:19.966 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.960+0000 7f5c535fe640 1 -- start start 2023-05-22T16:05:19.966 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.960+0000 7f5c535fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5c4c0b0250 0x7f5c4c14c440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:19.966 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.960+0000 7f5c535fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5c4c1531b0 0x7f5c4c0c50f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:19.966 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.960+0000 7f5c535fe640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5c4c159a40 0x7f5c4c167c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:19.966 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.960+0000 7f5c535fe640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c4c166b60 con 0x7f5c4c1531b0 2023-05-22T16:05:19.967 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.960+0000 7f5c535fe640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c4c166e40 con 0x7f5c4c0b0250 2023-05-22T16:05:19.967 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.960+0000 7f5c535fe640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c4c167120 con 0x7f5c4c159a40 2023-05-22T16:05:19.967 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.960+0000 7f5c50b99640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5c4c1531b0 0x7f5c4c0c50f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:19.967 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.964+0000 7f5c50b99640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5c4c1531b0 0x7f5c4c0c50f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41564/0 (socket says 172.21.15.16:41564) 2023-05-22T16:05:19.967 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.964+0000 7f5c50b99640 1 -- 172.21.15.16:0/3164692760 learned_addr learned my addr 172.21.15.16:0/3164692760 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:05:19.967 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.964+0000 7f5c5139a640 1 --2- 172.21.15.16:0/3164692760 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5c4c0b0250 0x7f5c4c14c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:19.967 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.964+0000 7f5c51b9b640 1 --2- 172.21.15.16:0/3164692760 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5c4c159a40 0x7f5c4c167c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:19.968 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.964+0000 7f5c50b99640 1 -- 172.21.15.16:0/3164692760 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5c4c159a40 msgr2=0x7f5c4c167c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:19.968 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.964+0000 7f5c50b99640 1 --2- 172.21.15.16:0/3164692760 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5c4c159a40 0x7f5c4c167c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:19.968 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.964+0000 7f5c50b99640 1 -- 172.21.15.16:0/3164692760 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5c4c0b0250 msgr2=0x7f5c4c14c440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:19.968 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.964+0000 7f5c50b99640 1 --2- 172.21.15.16:0/3164692760 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5c4c0b0250 0x7f5c4c14c440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:19.968 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.964+0000 7f5c50b99640 1 -- 172.21.15.16:0/3164692760 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c44009b30 con 0x7f5c4c1531b0 2023-05-22T16:05:19.968 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.964+0000 7f5c50b99640 1 --2- 172.21.15.16:0/3164692760 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5c4c1531b0 0x7f5c4c0c50f0 secure :-1 s=READY pgs=2424 cs=0 l=1 rev1=1 crypto rx=0x7f5c3c00c3e0 tx=0x7f5c3c00c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:19.969 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.964+0000 7f5c41ffb640 1 -- 172.21.15.16:0/3164692760 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c3c0074e0 con 0x7f5c4c1531b0 2023-05-22T16:05:19.969 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.964+0000 7f5c41ffb640 1 -- 172.21.15.16:0/3164692760 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c3c007640 con 0x7f5c4c1531b0 2023-05-22T16:05:19.969 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.964+0000 7f5c535fe640 1 -- 172.21.15.16:0/3164692760 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c4c15d0c0 con 0x7f5c4c1531b0 2023-05-22T16:05:19.969 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.964+0000 7f5c41ffb640 1 -- 172.21.15.16:0/3164692760 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c3c004a30 con 0x7f5c4c1531b0 2023-05-22T16:05:19.969 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.964+0000 7f5c535fe640 1 -- 172.21.15.16:0/3164692760 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c4c15cde0 con 0x7f5c4c1531b0 2023-05-22T16:05:19.970 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.964+0000 7f5c535fe640 1 -- 172.21.15.16:0/3164692760 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c4c070d40 con 0x7f5c4c1531b0 2023-05-22T16:05:19.970 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.964+0000 7f5c41ffb640 1 -- 172.21.15.16:0/3164692760 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 92) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f5c3c004b90 con 0x7f5c4c1531b0 2023-05-22T16:05:19.971 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.964+0000 7f5c41ffb640 1 --2- 172.21.15.16:0/3164692760 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f5c300654a0 0x7f5c30067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:19.971 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.964+0000 7f5c41ffb640 1 -- 172.21.15.16:0/3164692760 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(192..192 src has 1..192) v4 ==== 5586+0+0 (secure 0 0 0) 0x7f5c3c082960 con 0x7f5c4c1531b0 2023-05-22T16:05:19.971 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.964+0000 7f5c5139a640 1 --2- 172.21.15.16:0/3164692760 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f5c300654a0 0x7f5c30067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:19.971 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.964+0000 7f5c5139a640 1 --2- 172.21.15.16:0/3164692760 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f5c300654a0 0x7f5c30067950 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f5c48007fd0 tx=0x7f5c48009040 comp rx=0 tx=0).ready entity=mgr.10542 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:19.971 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:19.968+0000 7f5c41ffb640 1 -- 172.21.15.16:0/3164692760 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f5c3c0542f0 con 0x7f5c4c1531b0 2023-05-22T16:05:20.128 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:20.124+0000 7f5c535fe640 1 -- 172.21.15.16:0/3164692760 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f5c4c0cc2d0 con 0x7f5c4c1531b0 2023-05-22T16:05:20.129 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:20.128+0000 7f5c41ffb640 1 -- 172.21.15.16:0/3164692760 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v192) v1 ==== 84+0+126 (secure 0 0 0) 0x7f5c3c053e80 con 0x7f5c4c1531b0 2023-05-22T16:05:20.129 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T16:05:20.129 INFO:teuthology.orchestra.run.smithi016.stdout:[ 2023-05-22T16:05:20.129 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T16:05:20.130 INFO:teuthology.orchestra.run.smithi016.stdout: "poolnum": 1, 2023-05-22T16:05:20.130 INFO:teuthology.orchestra.run.smithi016.stdout: "poolname": ".mgr" 2023-05-22T16:05:20.130 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T16:05:20.130 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T16:05:20.130 INFO:teuthology.orchestra.run.smithi016.stdout: "poolnum": 2, 2023-05-22T16:05:20.131 INFO:teuthology.orchestra.run.smithi016.stdout: "poolname": "rbd" 2023-05-22T16:05:20.131 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T16:05:20.131 INFO:teuthology.orchestra.run.smithi016.stdout:] 2023-05-22T16:05:20.131 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:20.128+0000 7f5c535fe640 1 -- 172.21.15.16:0/3164692760 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f5c300654a0 msgr2=0x7f5c30067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:20.132 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:20.128+0000 7f5c535fe640 1 --2- 172.21.15.16:0/3164692760 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f5c300654a0 0x7f5c30067950 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f5c48007fd0 tx=0x7f5c48009040 comp rx=0 tx=0).stop 2023-05-22T16:05:20.132 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:20.128+0000 7f5c535fe640 1 -- 172.21.15.16:0/3164692760 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5c4c1531b0 msgr2=0x7f5c4c0c50f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:20.132 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:20.128+0000 7f5c535fe640 1 --2- 172.21.15.16:0/3164692760 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5c4c1531b0 0x7f5c4c0c50f0 secure :-1 s=READY pgs=2424 cs=0 l=1 rev1=1 crypto rx=0x7f5c3c00c3e0 tx=0x7f5c3c00c8a0 comp rx=0 tx=0).stop 2023-05-22T16:05:20.132 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:20.128+0000 7f5c535fe640 1 -- 172.21.15.16:0/3164692760 shutdown_connections 2023-05-22T16:05:20.132 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:20.128+0000 7f5c535fe640 1 --2- 172.21.15.16:0/3164692760 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5c4c159a40 0x7f5c4c167c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:20.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:20.128+0000 7f5c535fe640 1 --2- 172.21.15.16:0/3164692760 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f5c300654a0 0x7f5c30067950 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:20.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:20.128+0000 7f5c535fe640 1 --2- 172.21.15.16:0/3164692760 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5c4c1531b0 0x7f5c4c0c50f0 unknown :-1 s=CLOSED pgs=2424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:20.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:20.128+0000 7f5c535fe640 1 --2- 172.21.15.16:0/3164692760 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5c4c0b0250 0x7f5c4c14c440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:20.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:20.128+0000 7f5c535fe640 1 -- 172.21.15.16:0/3164692760 >> 172.21.15.16:0/3164692760 conn(0x7f5c4c069720 msgr2=0x7f5c4c0ce000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:05:20.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:20.128+0000 7f5c535fe640 1 -- 172.21.15.16:0/3164692760 shutdown_connections 2023-05-22T16:05:20.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:20.128+0000 7f5c535fe640 1 -- 172.21.15.16:0/3164692760 wait complete. 2023-05-22T16:05:20.147 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-05-22T16:05:20.148 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-05-22T16:05:20.273 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:20.272+0000 7f95f2903640 1 Processor -- start 2023-05-22T16:05:20.273 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:20.272+0000 7f95f2903640 1 -- start start 2023-05-22T16:05:20.274 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:20.272+0000 7f95f2903640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f95ec159ce0 0x7f95ec0c3e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:20.274 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:20.272+0000 7f95f2903640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f95ec0b1d80 0x7f95ec06fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:20.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:20.272+0000 7f95f2903640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f95ec0ccce0 0x7f95ec0cd160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:20.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:20.272+0000 7f95f2903640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95ec0cd6a0 con 0x7f95ec0b1d80 2023-05-22T16:05:20.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:20.272+0000 7f95f2903640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95ec0cf0f0 con 0x7f95ec159ce0 2023-05-22T16:05:20.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:20.272+0000 7f95f2903640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95ec0b2240 con 0x7f95ec0ccce0 2023-05-22T16:05:20.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:20.272+0000 7f95e3fff640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f95ec0b1d80 0x7f95ec06fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:20.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:20.272+0000 7f95e3fff640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f95ec0b1d80 0x7f95ec06fbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41576/0 (socket says 172.21.15.16:41576) 2023-05-22T16:05:20.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:20.272+0000 7f95e3fff640 1 -- 172.21.15.16:0/3473484380 learned_addr learned my addr 172.21.15.16:0/3473484380 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:05:20.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:20.272+0000 7f95ebfff640 1 --2- 172.21.15.16:0/3473484380 >> v2:172.21.15.62:3300/0 conn(0x7f95ec159ce0 0x7f95ec0c3e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:20.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:20.272+0000 7f95f0ea0640 1 --2- 172.21.15.16:0/3473484380 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f95ec0ccce0 0x7f95ec0cd160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:20.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:20.272+0000 7f95e3fff640 1 -- 172.21.15.16:0/3473484380 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f95ec0ccce0 msgr2=0x7f95ec0cd160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:05:20.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:20.272+0000 7f95e3fff640 1 --2- 172.21.15.16:0/3473484380 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f95ec0ccce0 0x7f95ec0cd160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:20.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:20.272+0000 7f95e3fff640 1 -- 172.21.15.16:0/3473484380 >> v2:172.21.15.62:3300/0 conn(0x7f95ec159ce0 msgr2=0x7f95ec0c3e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:05:20.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:20.272+0000 7f95e3fff640 1 --2- 172.21.15.16:0/3473484380 >> v2:172.21.15.62:3300/0 conn(0x7f95ec159ce0 0x7f95ec0c3e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:20.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:20.272+0000 7f95e3fff640 1 -- 172.21.15.16:0/3473484380 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95ec0b2520 con 0x7f95ec0b1d80 2023-05-22T16:05:20.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:20.276+0000 7f95e3fff640 1 --2- 172.21.15.16:0/3473484380 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f95ec0b1d80 0x7f95ec06fbf0 secure :-1 s=READY pgs=2425 cs=0 l=1 rev1=1 crypto rx=0x7f95ec0ca900 tx=0x7f95d401c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=303ad20f8a7f5bc0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:20.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:20.276+0000 7f95eb7fe640 1 -- 172.21.15.16:0/3473484380 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95d40ac600 con 0x7f95ec0b1d80 2023-05-22T16:05:20.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:20.276+0000 7f95eb7fe640 1 -- 172.21.15.16:0/3473484380 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95d40a8050 con 0x7f95ec0b1d80 2023-05-22T16:05:20.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:20.276+0000 7f95eb7fe640 1 -- 172.21.15.16:0/3473484380 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95d40ac980 con 0x7f95ec0b1d80 2023-05-22T16:05:20.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:20.276+0000 7f95f2903640 1 -- 172.21.15.16:0/3473484380 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f95ec0b1d80 msgr2=0x7f95ec06fbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:20.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:20.276+0000 7f95f2903640 1 --2- 172.21.15.16:0/3473484380 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f95ec0b1d80 0x7f95ec06fbf0 secure :-1 s=READY pgs=2425 cs=0 l=1 rev1=1 crypto rx=0x7f95ec0ca900 tx=0x7f95d401c4d0 comp rx=0 tx=0).stop 2023-05-22T16:05:20.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:20.276+0000 7f95f2903640 1 -- 172.21.15.16:0/3473484380 shutdown_connections 2023-05-22T16:05:20.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:20.276+0000 7f95f2903640 1 --2- 172.21.15.16:0/3473484380 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f95ec0b1d80 0x7f95ec06fbf0 unknown :-1 s=CLOSED pgs=2425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:20.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:20.276+0000 7f95f2903640 1 --2- 172.21.15.16:0/3473484380 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f95ec0ccce0 0x7f95ec0cd160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:20.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:20.276+0000 7f95f2903640 1 --2- 172.21.15.16:0/3473484380 >> v2:172.21.15.62:3300/0 conn(0x7f95ec159ce0 0x7f95ec0c3e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:20.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:20.276+0000 7f95f2903640 1 -- 172.21.15.16:0/3473484380 >> 172.21.15.16:0/3473484380 conn(0x7f95ec06d9d0 msgr2=0x7f95ec06ddd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:05:20.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:20.276+0000 7f95f2903640 1 -- 172.21.15.16:0/3473484380 shutdown_connections 2023-05-22T16:05:20.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:20.276+0000 7f95f2903640 1 -- 172.21.15.16:0/3473484380 wait complete. 2023-05-22T16:05:20.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:20.276+0000 7f95f2903640 1 Processor -- start 2023-05-22T16:05:20.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:20.276+0000 7f95f2903640 1 -- start start 2023-05-22T16:05:20.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:20.276+0000 7f95f2903640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f95ec0b1d80 0x7f95ec150340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:20.282 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:20.276+0000 7f95f2903640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f95ec0ccce0 0x7f95ec0b3340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:20.282 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:20.276+0000 7f95f2903640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f95ec159ce0 0x7f95ec0c5730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:20.282 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:20.276+0000 7f95f2903640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95ec0b5d60 con 0x7f95ec0b1d80 2023-05-22T16:05:20.282 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:20.276+0000 7f95f2903640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95ec0b3cb0 con 0x7f95ec0ccce0 2023-05-22T16:05:20.282 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:20.276+0000 7f95f2903640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95ec0b56a0 con 0x7f95ec159ce0 2023-05-22T16:05:20.282 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:20.276+0000 7f95ebfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f95ec0b1d80 0x7f95ec150340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:20.282 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:20.276+0000 7f95ebfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f95ec0b1d80 0x7f95ec150340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41580/0 (socket says 172.21.15.16:41580) 2023-05-22T16:05:20.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:20.276+0000 7f95ebfff640 1 -- 172.21.15.16:0/3865638024 learned_addr learned my addr 172.21.15.16:0/3865638024 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:05:20.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:20.276+0000 7f95f0ea0640 1 --2- 172.21.15.16:0/3865638024 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f95ec159ce0 0x7f95ec0c5730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:20.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:20.276+0000 7f95e3fff640 1 --2- 172.21.15.16:0/3865638024 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f95ec0ccce0 0x7f95ec0b3340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:20.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:20.276+0000 7f95ebfff640 1 -- 172.21.15.16:0/3865638024 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f95ec159ce0 msgr2=0x7f95ec0c5730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:20.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:20.276+0000 7f95ebfff640 1 --2- 172.21.15.16:0/3865638024 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f95ec159ce0 0x7f95ec0c5730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:20.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:20.276+0000 7f95ebfff640 1 -- 172.21.15.16:0/3865638024 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f95ec0ccce0 msgr2=0x7f95ec0b3340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:20.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:20.276+0000 7f95ebfff640 1 --2- 172.21.15.16:0/3865638024 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f95ec0ccce0 0x7f95ec0b3340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:20.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:20.276+0000 7f95ebfff640 1 -- 172.21.15.16:0/3865638024 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95d40094d0 con 0x7f95ec0b1d80 2023-05-22T16:05:20.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:20.276+0000 7f95e3fff640 1 --2- 172.21.15.16:0/3865638024 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f95ec0ccce0 0x7f95ec0b3340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T16:05:20.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:20.276+0000 7f95ebfff640 1 --2- 172.21.15.16:0/3865638024 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f95ec0b1d80 0x7f95ec150340 secure :-1 s=READY pgs=2426 cs=0 l=1 rev1=1 crypto rx=0x7f95dc00d800 tx=0x7f95dc00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:20.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:20.276+0000 7f95e97fa640 1 -- 172.21.15.16:0/3865638024 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95dc00b6c0 con 0x7f95ec0b1d80 2023-05-22T16:05:20.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:20.280+0000 7f95e97fa640 1 -- 172.21.15.16:0/3865638024 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95dc00b820 con 0x7f95ec0b1d80 2023-05-22T16:05:20.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:20.280+0000 7f95f2903640 1 -- 172.21.15.16:0/3865638024 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95ec0c4520 con 0x7f95ec0b1d80 2023-05-22T16:05:20.285 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:20.280+0000 7f95e97fa640 1 -- 172.21.15.16:0/3865638024 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95dc00bb90 con 0x7f95ec0b1d80 2023-05-22T16:05:20.285 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:20.280+0000 7f95f2903640 1 -- 172.21.15.16:0/3865638024 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95ec0c4390 con 0x7f95ec0b1d80 2023-05-22T16:05:20.285 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:20.280+0000 7f95f2903640 1 -- 172.21.15.16:0/3865638024 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95ec070130 con 0x7f95ec0b1d80 2023-05-22T16:05:20.286 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:20.280+0000 7f95e97fa640 1 -- 172.21.15.16:0/3865638024 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 92) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f95dc00bcf0 con 0x7f95ec0b1d80 2023-05-22T16:05:20.286 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:20.280+0000 7f95e97fa640 1 --2- 172.21.15.16:0/3865638024 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f95cc0654a0 0x7f95cc067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:20.286 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:20.280+0000 7f95e97fa640 1 -- 172.21.15.16:0/3865638024 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(192..192 src has 1..192) v4 ==== 5586+0+0 (secure 0 0 0) 0x7f95dc082d60 con 0x7f95ec0b1d80 2023-05-22T16:05:20.286 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:20.280+0000 7f95e3fff640 1 --2- 172.21.15.16:0/3865638024 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f95cc0654a0 0x7f95cc067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:20.286 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:20.280+0000 7f95e3fff640 1 --2- 172.21.15.16:0/3865638024 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f95cc0654a0 0x7f95cc067950 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f95d40ae000 tx=0x7f95d4008040 comp rx=0 tx=0).ready entity=mgr.10542 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:20.287 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:20.284+0000 7f95e97fa640 1 -- 172.21.15.16:0/3865638024 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f95dc0544d0 con 0x7f95ec0b1d80 2023-05-22T16:05:20.439 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:20.436+0000 7f95f2903640 1 -- 172.21.15.16:0/3865638024 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f95ec0b5fd0 con 0x7f95ec0b1d80 2023-05-22T16:05:20.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:20.520+0000 7f95e97fa640 1 -- 172.21.15.16:0/3865638024 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v193) v1 ==== 138+0+0 (secure 0 0 0) 0x7f95dc0540c0 con 0x7f95ec0b1d80 2023-05-22T16:05:20.523 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'cephfs_metadata' created 2023-05-22T16:05:20.527 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:20.524+0000 7f95f2903640 1 -- 172.21.15.16:0/3865638024 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f95cc0654a0 msgr2=0x7f95cc067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:20.527 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:20.524+0000 7f95f2903640 1 --2- 172.21.15.16:0/3865638024 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f95cc0654a0 0x7f95cc067950 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f95d40ae000 tx=0x7f95d4008040 comp rx=0 tx=0).stop 2023-05-22T16:05:20.527 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:20.524+0000 7f95f2903640 1 -- 172.21.15.16:0/3865638024 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f95ec0b1d80 msgr2=0x7f95ec150340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:20.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:20.524+0000 7f95f2903640 1 --2- 172.21.15.16:0/3865638024 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f95ec0b1d80 0x7f95ec150340 secure :-1 s=READY pgs=2426 cs=0 l=1 rev1=1 crypto rx=0x7f95dc00d800 tx=0x7f95dc00dcc0 comp rx=0 tx=0).stop 2023-05-22T16:05:20.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:20.528+0000 7f95f2903640 1 -- 172.21.15.16:0/3865638024 shutdown_connections 2023-05-22T16:05:20.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:20.528+0000 7f95f2903640 1 --2- 172.21.15.16:0/3865638024 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f95ec159ce0 0x7f95ec0c5730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:20.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:20.528+0000 7f95f2903640 1 --2- 172.21.15.16:0/3865638024 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f95ec0ccce0 0x7f95ec0b3340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:20.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:20.528+0000 7f95f2903640 1 --2- 172.21.15.16:0/3865638024 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f95cc0654a0 0x7f95cc067950 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:20.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:20.528+0000 7f95f2903640 1 --2- 172.21.15.16:0/3865638024 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f95ec0b1d80 0x7f95ec150340 unknown :-1 s=CLOSED pgs=2426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:20.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:20.528+0000 7f95f2903640 1 -- 172.21.15.16:0/3865638024 >> 172.21.15.16:0/3865638024 conn(0x7f95ec06d9d0 msgr2=0x7f95ec0ce8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:05:20.530 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:20.528+0000 7f95f2903640 1 -- 172.21.15.16:0/3865638024 shutdown_connections 2023-05-22T16:05:20.530 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:20.528+0000 7f95f2903640 1 -- 172.21.15.16:0/3865638024 wait complete. 2023-05-22T16:05:20.552 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-05-22T16:05:20.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:20.680+0000 7f9da26ad640 1 Processor -- start 2023-05-22T16:05:20.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:20.680+0000 7f9da26ad640 1 -- start start 2023-05-22T16:05:20.683 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:20.680+0000 7f9da26ad640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9d9c15bca0 0x7f9d9c160160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:20.683 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:20.680+0000 7f9da26ad640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f9d9c0b3ee0 0x7f9d9c0b4330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:20.683 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:20.680+0000 7f9da26ad640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f9d9c0b4870 0x7f9d9c0ce900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:20.684 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:20.680+0000 7f9da26ad640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d9c15c070 con 0x7f9d9c0b3ee0 2023-05-22T16:05:20.684 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:20.680+0000 7f9da26ad640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d9c0d0400 con 0x7f9d9c0b4870 2023-05-22T16:05:20.684 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:20.680+0000 7f9da26ad640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d9c1606a0 con 0x7f9d9c15bca0 2023-05-22T16:05:20.684 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:20.680+0000 7f9d9b7fe640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f9d9c0b3ee0 0x7f9d9c0b4330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:20.684 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:20.680+0000 7f9d9b7fe640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f9d9c0b3ee0 0x7f9d9c0b4330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41582/0 (socket says 172.21.15.16:41582) 2023-05-22T16:05:20.685 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:20.680+0000 7f9d9b7fe640 1 -- 172.21.15.16:0/3463309980 learned_addr learned my addr 172.21.15.16:0/3463309980 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:05:20.685 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:20.680+0000 7f9da0c4a640 1 --2- 172.21.15.16:0/3463309980 >> v2:172.21.15.62:3300/0 conn(0x7f9d9c0b4870 0x7f9d9c0ce900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:20.685 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:20.680+0000 7f9d9bfff640 1 --2- 172.21.15.16:0/3463309980 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9d9c15bca0 0x7f9d9c160160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:20.685 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:20.680+0000 7f9d9b7fe640 1 -- 172.21.15.16:0/3463309980 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9d9c15bca0 msgr2=0x7f9d9c160160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:05:20.686 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:20.680+0000 7f9d9b7fe640 1 --2- 172.21.15.16:0/3463309980 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9d9c15bca0 0x7f9d9c160160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:20.686 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:20.680+0000 7f9d9b7fe640 1 -- 172.21.15.16:0/3463309980 >> v2:172.21.15.62:3300/0 conn(0x7f9d9c0b4870 msgr2=0x7f9d9c0ce900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:05:20.686 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:20.680+0000 7f9d9b7fe640 1 --2- 172.21.15.16:0/3463309980 >> v2:172.21.15.62:3300/0 conn(0x7f9d9c0b4870 0x7f9d9c0ce900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:20.686 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:20.680+0000 7f9d9b7fe640 1 -- 172.21.15.16:0/3463309980 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d9c0d00d0 con 0x7f9d9c0b3ee0 2023-05-22T16:05:20.686 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:20.680+0000 7f9d9bfff640 1 --2- 172.21.15.16:0/3463309980 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9d9c15bca0 0x7f9d9c160160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T16:05:20.686 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:20.680+0000 7f9da0c4a640 1 --2- 172.21.15.16:0/3463309980 >> v2:172.21.15.62:3300/0 conn(0x7f9d9c0b4870 0x7f9d9c0ce900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T16:05:20.687 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:20.684+0000 7f9d9b7fe640 1 --2- 172.21.15.16:0/3463309980 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9d9c0b3ee0 0x7f9d9c0b4330 secure :-1 s=READY pgs=2427 cs=0 l=1 rev1=1 crypto rx=0x7f9d9c08b8a0 tx=0x7f9d8401c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7356bd32ba1e34a server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:20.687 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:20.684+0000 7f9d9affd640 1 -- 172.21.15.16:0/3463309980 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d840a5790 con 0x7f9d9c0b3ee0 2023-05-22T16:05:20.688 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:20.684+0000 7f9d9affd640 1 -- 172.21.15.16:0/3463309980 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d840a58f0 con 0x7f9d9c0b3ee0 2023-05-22T16:05:20.688 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:20.684+0000 7f9d9affd640 1 -- 172.21.15.16:0/3463309980 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d840a5c80 con 0x7f9d9c0b3ee0 2023-05-22T16:05:20.688 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:20.684+0000 7f9da26ad640 1 -- 172.21.15.16:0/3463309980 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9d9c0b3ee0 msgr2=0x7f9d9c0b4330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:20.688 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:20.684+0000 7f9da26ad640 1 --2- 172.21.15.16:0/3463309980 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9d9c0b3ee0 0x7f9d9c0b4330 secure :-1 s=READY pgs=2427 cs=0 l=1 rev1=1 crypto rx=0x7f9d9c08b8a0 tx=0x7f9d8401c910 comp rx=0 tx=0).stop 2023-05-22T16:05:20.688 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:20.684+0000 7f9da26ad640 1 -- 172.21.15.16:0/3463309980 shutdown_connections 2023-05-22T16:05:20.688 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:20.684+0000 7f9da26ad640 1 --2- 172.21.15.16:0/3463309980 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9d9c0b3ee0 0x7f9d9c0b4330 unknown :-1 s=CLOSED pgs=2427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:20.688 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:20.684+0000 7f9da26ad640 1 --2- 172.21.15.16:0/3463309980 >> v2:172.21.15.62:3300/0 conn(0x7f9d9c0b4870 0x7f9d9c0ce900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:20.689 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:20.684+0000 7f9da26ad640 1 --2- 172.21.15.16:0/3463309980 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9d9c15bca0 0x7f9d9c160160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:20.689 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:20.684+0000 7f9da26ad640 1 -- 172.21.15.16:0/3463309980 >> 172.21.15.16:0/3463309980 conn(0x7f9d9c0657f0 msgr2=0x7f9d9c070e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:05:20.689 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:20.684+0000 7f9da26ad640 1 -- 172.21.15.16:0/3463309980 shutdown_connections 2023-05-22T16:05:20.689 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:20.684+0000 7f9da26ad640 1 -- 172.21.15.16:0/3463309980 wait complete. 2023-05-22T16:05:20.691 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:20.688+0000 7f9da26ad640 1 Processor -- start 2023-05-22T16:05:20.691 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:20.688+0000 7f9da26ad640 1 -- start start 2023-05-22T16:05:20.691 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:20.688+0000 7f9da26ad640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9d9c0b3ee0 0x7f9d9c14baa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:20.691 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:20.688+0000 7f9da26ad640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9d9c0b4870 0x7f9d9c14ed60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:20.691 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:20.688+0000 7f9da26ad640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9d9c15bca0 0x7f9d9c14f2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:20.692 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:20.688+0000 7f9da26ad640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d9c0b7c40 con 0x7f9d9c0b4870 2023-05-22T16:05:20.692 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:20.688+0000 7f9da26ad640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d9c0b8060 con 0x7f9d9c15bca0 2023-05-22T16:05:20.692 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:20.688+0000 7f9da26ad640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d9c0b6fc0 con 0x7f9d9c0b3ee0 2023-05-22T16:05:20.692 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:20.688+0000 7f9d9b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9d9c0b4870 0x7f9d9c14ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:20.692 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:20.688+0000 7f9d9b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9d9c0b4870 0x7f9d9c14ed60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41594/0 (socket says 172.21.15.16:41594) 2023-05-22T16:05:20.692 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:20.688+0000 7f9d9b7fe640 1 -- 172.21.15.16:0/2469588193 learned_addr learned my addr 172.21.15.16:0/2469588193 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:05:20.692 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:20.688+0000 7f9da0c4a640 1 --2- 172.21.15.16:0/2469588193 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9d9c15bca0 0x7f9d9c14f2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:20.692 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:20.688+0000 7f9d9bfff640 1 --2- 172.21.15.16:0/2469588193 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9d9c0b3ee0 0x7f9d9c14baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:20.693 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:20.688+0000 7f9d9b7fe640 1 -- 172.21.15.16:0/2469588193 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9d9c0b3ee0 msgr2=0x7f9d9c14baa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:20.693 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:20.688+0000 7f9d9b7fe640 1 --2- 172.21.15.16:0/2469588193 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9d9c0b3ee0 0x7f9d9c14baa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:20.693 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:20.688+0000 7f9d9b7fe640 1 -- 172.21.15.16:0/2469588193 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9d9c15bca0 msgr2=0x7f9d9c14f2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:20.693 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:20.688+0000 7f9d9b7fe640 1 --2- 172.21.15.16:0/2469588193 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9d9c15bca0 0x7f9d9c14f2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:20.693 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:20.688+0000 7f9d9b7fe640 1 -- 172.21.15.16:0/2469588193 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d84009b30 con 0x7f9d9c0b4870 2023-05-22T16:05:20.693 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:20.688+0000 7f9d9b7fe640 1 --2- 172.21.15.16:0/2469588193 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9d9c0b4870 0x7f9d9c14ed60 secure :-1 s=READY pgs=2428 cs=0 l=1 rev1=1 crypto rx=0x7f9d8401afc0 tx=0x7f9d840196b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:20.693 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:20.688+0000 7f9d98ff9640 1 -- 172.21.15.16:0/2469588193 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d840b8670 con 0x7f9d9c0b4870 2023-05-22T16:05:20.694 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:20.688+0000 7f9d98ff9640 1 -- 172.21.15.16:0/2469588193 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d840b87d0 con 0x7f9d9c0b4870 2023-05-22T16:05:20.694 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:20.688+0000 7f9d98ff9640 1 -- 172.21.15.16:0/2469588193 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d840b8b80 con 0x7f9d9c0b4870 2023-05-22T16:05:20.694 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:20.688+0000 7f9da26ad640 1 -- 172.21.15.16:0/2469588193 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d9c0cd1a0 con 0x7f9d9c0b4870 2023-05-22T16:05:20.694 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:20.688+0000 7f9da26ad640 1 -- 172.21.15.16:0/2469588193 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d9c0b4c70 con 0x7f9d9c0b4870 2023-05-22T16:05:20.694 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:20.688+0000 7f9da26ad640 1 -- 172.21.15.16:0/2469588193 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d9c0d0870 con 0x7f9d9c0b4870 2023-05-22T16:05:20.695 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:20.688+0000 7f9d98ff9640 1 -- 172.21.15.16:0/2469588193 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 92) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f9d840b8ce0 con 0x7f9d9c0b4870 2023-05-22T16:05:20.695 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:20.692+0000 7f9d98ff9640 1 --2- 172.21.15.16:0/2469588193 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f9d7c0654a0 0x7f9d7c067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:20.695 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:20.692+0000 7f9d98ff9640 1 -- 172.21.15.16:0/2469588193 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(193..193 src has 1..193) v4 ==== 5956+0+0 (secure 0 0 0) 0x7f9d8411e2e0 con 0x7f9d9c0b4870 2023-05-22T16:05:20.695 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:20.692+0000 7f9d9bfff640 1 --2- 172.21.15.16:0/2469588193 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f9d7c0654a0 0x7f9d7c067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:20.695 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:20.692+0000 7f9d9bfff640 1 --2- 172.21.15.16:0/2469588193 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f9d7c0654a0 0x7f9d7c067950 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f9d8c00ba10 tx=0x7f9d8c008040 comp rx=0 tx=0).ready entity=mgr.10542 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:20.696 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:20.696+0000 7f9d98ff9640 1 -- 172.21.15.16:0/2469588193 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f9d840ec020 con 0x7f9d9c0b4870 2023-05-22T16:05:20.853 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:20.852+0000 7f9da26ad640 1 -- 172.21.15.16:0/2469588193 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f9d9c04dbe0 con 0x7f9d9c0b4870 2023-05-22T16:05:21.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:21.544+0000 7f9d98ff9640 1 -- 172.21.15.16:0/2469588193 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v194) v1 ==== 170+0+0 (secure 0 0 0) 0x7f9d840ef640 con 0x7f9d9c0b4870 2023-05-22T16:05:21.550 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'cephfs_data' created 2023-05-22T16:05:21.559 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:21.556+0000 7f9da26ad640 1 -- 172.21.15.16:0/2469588193 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f9d7c0654a0 msgr2=0x7f9d7c067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:21.559 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:21.556+0000 7f9da26ad640 1 --2- 172.21.15.16:0/2469588193 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f9d7c0654a0 0x7f9d7c067950 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f9d8c00ba10 tx=0x7f9d8c008040 comp rx=0 tx=0).stop 2023-05-22T16:05:21.560 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:21.556+0000 7f9da26ad640 1 -- 172.21.15.16:0/2469588193 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9d9c0b4870 msgr2=0x7f9d9c14ed60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:21.560 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:21.556+0000 7f9da26ad640 1 --2- 172.21.15.16:0/2469588193 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9d9c0b4870 0x7f9d9c14ed60 secure :-1 s=READY pgs=2428 cs=0 l=1 rev1=1 crypto rx=0x7f9d8401afc0 tx=0x7f9d840196b0 comp rx=0 tx=0).stop 2023-05-22T16:05:21.561 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:21.560+0000 7f9da26ad640 1 -- 172.21.15.16:0/2469588193 shutdown_connections 2023-05-22T16:05:21.561 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:21.560+0000 7f9da26ad640 1 --2- 172.21.15.16:0/2469588193 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9d9c15bca0 0x7f9d9c14f2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:21.561 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:21.560+0000 7f9da26ad640 1 --2- 172.21.15.16:0/2469588193 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f9d7c0654a0 0x7f9d7c067950 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:21.561 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:21.560+0000 7f9da26ad640 1 --2- 172.21.15.16:0/2469588193 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9d9c0b4870 0x7f9d9c14ed60 unknown :-1 s=CLOSED pgs=2428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:21.561 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:21.560+0000 7f9da26ad640 1 --2- 172.21.15.16:0/2469588193 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9d9c0b3ee0 0x7f9d9c14baa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:21.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:21.560+0000 7f9da26ad640 1 -- 172.21.15.16:0/2469588193 >> 172.21.15.16:0/2469588193 conn(0x7f9d9c0657f0 msgr2=0x7f9d9c0b5fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:05:21.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:21.560+0000 7f9da26ad640 1 -- 172.21.15.16:0/2469588193 shutdown_connections 2023-05-22T16:05:21.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:21.560+0000 7f9da26ad640 1 -- 172.21.15.16:0/2469588193 wait complete. 2023-05-22T16:05:21.593 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-05-22T16:05:21.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:21.716+0000 7fe2e7902640 1 Processor -- start 2023-05-22T16:05:21.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:21.716+0000 7fe2e7902640 1 -- start start 2023-05-22T16:05:21.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:21.716+0000 7fe2e7902640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe2e015bcd0 0x7fe2e015c120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:21.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:21.716+0000 7fe2e7902640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2e0158cd0 con 0x7fe2e0158210 2023-05-22T16:05:21.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:21.716+0000 7fe2e7902640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2e015c790 con 0x7fe2e00c8660 2023-05-22T16:05:21.719 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:21.716+0000 7fe2e7902640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2e015ca70 con 0x7fe2e015bcd0 2023-05-22T16:05:21.719 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:21.716+0000 7fe2e569e640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fe2e0158210 0x7fe2e0159160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:48436/0 (socket says 172.21.15.16:48436) 2023-05-22T16:05:21.719 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:21.716+0000 7fe2e569e640 1 -- 172.21.15.16:0/3630901027 learned_addr learned my addr 172.21.15.16:0/3630901027 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T16:05:21.729 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:21.716+0000 7fe2d7fff640 1 -- 172.21.15.16:0/3630901027 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4249089816 0 0) 0x7fe2e0158cd0 con 0x7fe2e0158210 2023-05-22T16:05:21.730 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:21.716+0000 7fe2e4e9d640 1 --2- 172.21.15.16:0/3630901027 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe2e015bcd0 0x7fe2e015c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:21.731 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:21.716+0000 7fe2d7fff640 1 -- 172.21.15.16:0/3630901027 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe2cc003530 con 0x7fe2e0158210 2023-05-22T16:05:21.731 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:21.716+0000 7fe2d7fff640 1 -- 172.21.15.16:0/3630901027 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3374778548 0 0) 0x7fe2cc003530 con 0x7fe2e0158210 2023-05-22T16:05:21.732 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:21.716+0000 7fe2d7fff640 1 -- 172.21.15.16:0/3630901027 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe2e015bcd0 msgr2=0x7fe2e015c120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:05:21.732 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:21.716+0000 7fe2d7fff640 1 --2- 172.21.15.16:0/3630901027 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe2e015bcd0 0x7fe2e015c120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:21.732 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:21.716+0000 7fe2d7fff640 1 -- 172.21.15.16:0/3630901027 >> v1:172.21.15.62:6789/0 conn(0x7fe2e00c8660 legacy=0x7fe2e00ca690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:21.733 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:21.716+0000 7fe2d7fff640 1 -- 172.21.15.16:0/3630901027 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2e0158960 con 0x7fe2e0158210 2023-05-22T16:05:21.733 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:21.716+0000 7fe2d7fff640 1 -- 172.21.15.16:0/3630901027 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7fe2dc001680 con 0x7fe2e0158210 2023-05-22T16:05:21.733 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:21.716+0000 7fe2d7fff640 1 -- 172.21.15.16:0/3630901027 >> v1:172.21.15.16:6789/0 conn(0x7fe2e0158210 legacy=0x7fe2e0159160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:21.733 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:21.716+0000 7fe2d7fff640 1 --2- 172.21.15.16:0/3630901027 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe2cc0039b0 0x7fe2cc003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:21.734 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:21.716+0000 7fe2d7fff640 1 --2- 172.21.15.16:0/3630901027 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe2cc0047d0 0x7fe2cc004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:21.734 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:21.716+0000 7fe2d7fff640 1 --2- 172.21.15.16:0/3630901027 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe2e015bcd0 0x7fe2cc0051a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:21.734 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:21.716+0000 7fe2d7fff640 1 -- 172.21.15.16:0/3630901027 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2e015ca70 con 0x7fe2cc0039b0 2023-05-22T16:05:21.735 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:21.716+0000 7fe2d7fff640 1 -- 172.21.15.16:0/3630901027 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2cc00ba90 con 0x7fe2cc0047d0 2023-05-22T16:05:21.735 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:21.716+0000 7fe2d7fff640 1 -- 172.21.15.16:0/3630901027 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2cc00bc50 con 0x7fe2e015bcd0 2023-05-22T16:05:21.735 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:21.716+0000 7fe2e569e640 1 --2- 172.21.15.16:0/3630901027 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe2cc0039b0 0x7fe2cc003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:21.736 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:21.716+0000 7fe2e4e9d640 1 --2- 172.21.15.16:0/3630901027 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe2cc0047d0 0x7fe2cc004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:21.736 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:21.716+0000 7fe2e5e9f640 1 --2- 172.21.15.16:0/3630901027 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe2e015bcd0 0x7fe2cc0051a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:21.736 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:21.716+0000 7fe2e569e640 1 -- 172.21.15.16:0/3630901027 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe2e015bcd0 msgr2=0x7fe2cc0051a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:05:21.736 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:21.716+0000 7fe2e569e640 1 --2- 172.21.15.16:0/3630901027 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe2e015bcd0 0x7fe2cc0051a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:21.737 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:21.716+0000 7fe2e569e640 1 -- 172.21.15.16:0/3630901027 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe2cc0047d0 msgr2=0x7fe2cc004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:05:21.737 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:21.716+0000 7fe2e569e640 1 --2- 172.21.15.16:0/3630901027 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe2cc0047d0 0x7fe2cc004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:21.737 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:21.716+0000 7fe2e569e640 1 -- 172.21.15.16:0/3630901027 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2cc003530 con 0x7fe2cc0039b0 2023-05-22T16:05:21.737 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:21.720+0000 7fe2e569e640 1 --2- 172.21.15.16:0/3630901027 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe2cc0039b0 0x7fe2cc003dc0 secure :-1 s=READY pgs=2430 cs=0 l=1 rev1=1 crypto rx=0x7fe2dc00c240 tx=0x7fe2dc01cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9ecf89ae1b9e94c server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:21.737 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:21.720+0000 7fe2d7fff640 1 -- 172.21.15.16:0/3630901027 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2dc005ae0 con 0x7fe2cc0039b0 2023-05-22T16:05:21.737 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:21.720+0000 7fe2d7fff640 1 -- 172.21.15.16:0/3630901027 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2dc005c40 con 0x7fe2cc0039b0 2023-05-22T16:05:21.738 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:21.720+0000 7fe2d7fff640 1 -- 172.21.15.16:0/3630901027 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2dc0a5830 con 0x7fe2cc0039b0 2023-05-22T16:05:21.738 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:21.720+0000 7fe2e7902640 1 -- 172.21.15.16:0/3630901027 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe2cc0039b0 msgr2=0x7fe2cc003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:21.738 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:21.720+0000 7fe2e7902640 1 --2- 172.21.15.16:0/3630901027 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe2cc0039b0 0x7fe2cc003dc0 secure :-1 s=READY pgs=2430 cs=0 l=1 rev1=1 crypto rx=0x7fe2dc00c240 tx=0x7fe2dc01cc40 comp rx=0 tx=0).stop 2023-05-22T16:05:21.738 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:21.720+0000 7fe2e5e9f640 1 -- 172.21.15.16:0/3630901027 reap_dead start 2023-05-22T16:05:21.738 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:21.720+0000 7fe2e7902640 1 -- 172.21.15.16:0/3630901027 shutdown_connections 2023-05-22T16:05:21.738 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:21.720+0000 7fe2e7902640 1 -- 172.21.15.16:0/3630901027 >> 172.21.15.16:0/3630901027 conn(0x7fe2e0069720 msgr2=0x7fe2e0069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:05:21.738 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:21.720+0000 7fe2e7902640 1 -- 172.21.15.16:0/3630901027 shutdown_connections 2023-05-22T16:05:21.738 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:21.720+0000 7fe2e7902640 1 -- 172.21.15.16:0/3630901027 wait complete. 2023-05-22T16:05:21.739 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:21.720+0000 7fe2e7902640 1 Processor -- start 2023-05-22T16:05:21.739 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:21.720+0000 7fe2e7902640 1 -- start start 2023-05-22T16:05:21.739 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:21.724+0000 7fe2e7902640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe2cc0039b0 0x7fe2e01403c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:21.739 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:21.724+0000 7fe2e7902640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe2e0143740 0x7fe2e00c50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:21.739 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:21.724+0000 7fe2e7902640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe2e015db10 0x7fe2e015dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:21.739 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:21.724+0000 7fe2e7902640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2e015e4f0 con 0x7fe2cc0039b0 2023-05-22T16:05:21.739 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:21.724+0000 7fe2e7902640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2e00c70c0 con 0x7fe2e0143740 2023-05-22T16:05:21.740 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:21.724+0000 7fe2e7902640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2e00c73a0 con 0x7fe2e015db10 2023-05-22T16:05:21.740 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:21.724+0000 7fe2e569e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe2cc0039b0 0x7fe2e01403c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:21.740 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:21.724+0000 7fe2e569e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe2cc0039b0 0x7fe2e01403c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41612/0 (socket says 172.21.15.16:41612) 2023-05-22T16:05:21.740 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:21.724+0000 7fe2e569e640 1 -- 172.21.15.16:0/638550351 learned_addr learned my addr 172.21.15.16:0/638550351 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:05:21.740 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:21.724+0000 7fe2e4e9d640 1 --2- 172.21.15.16:0/638550351 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe2e0143740 0x7fe2e00c50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:21.740 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:21.724+0000 7fe2e5e9f640 1 --2- 172.21.15.16:0/638550351 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe2e015db10 0x7fe2e015dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:21.741 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:21.724+0000 7fe2e569e640 1 -- 172.21.15.16:0/638550351 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe2e015db10 msgr2=0x7fe2e015dfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:21.741 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:21.724+0000 7fe2e569e640 1 --2- 172.21.15.16:0/638550351 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe2e015db10 0x7fe2e015dfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:21.741 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:21.724+0000 7fe2e569e640 1 -- 172.21.15.16:0/638550351 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe2e0143740 msgr2=0x7fe2e00c50e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:21.741 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:21.724+0000 7fe2e569e640 1 --2- 172.21.15.16:0/638550351 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe2e0143740 0x7fe2e00c50e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:21.741 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:21.724+0000 7fe2e569e640 1 -- 172.21.15.16:0/638550351 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2e00c9000 con 0x7fe2cc0039b0 2023-05-22T16:05:21.741 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:21.724+0000 7fe2e4e9d640 1 --2- 172.21.15.16:0/638550351 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe2e0143740 0x7fe2e00c50e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T16:05:21.741 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:21.724+0000 7fe2e569e640 1 --2- 172.21.15.16:0/638550351 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe2cc0039b0 0x7fe2e01403c0 secure :-1 s=READY pgs=2431 cs=0 l=1 rev1=1 crypto rx=0x7fe2dc01ca70 tx=0x7fe2cc005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:21.742 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:21.724+0000 7fe2d5ffb640 1 -- 172.21.15.16:0/638550351 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2dc0b4070 con 0x7fe2cc0039b0 2023-05-22T16:05:21.742 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:21.724+0000 7fe2d5ffb640 1 -- 172.21.15.16:0/638550351 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2dc005db0 con 0x7fe2cc0039b0 2023-05-22T16:05:21.742 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:21.724+0000 7fe2e7902640 1 -- 172.21.15.16:0/638550351 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2e00c8e70 con 0x7fe2cc0039b0 2023-05-22T16:05:21.742 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:21.724+0000 7fe2d5ffb640 1 -- 172.21.15.16:0/638550351 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2dc001640 con 0x7fe2cc0039b0 2023-05-22T16:05:21.742 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:21.724+0000 7fe2e7902640 1 -- 172.21.15.16:0/638550351 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2e00cada0 con 0x7fe2cc0039b0 2023-05-22T16:05:21.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:21.724+0000 7fe2e7902640 1 -- 172.21.15.16:0/638550351 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2e004dbe0 con 0x7fe2cc0039b0 2023-05-22T16:05:21.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:21.724+0000 7fe2d5ffb640 1 -- 172.21.15.16:0/638550351 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 92) v1 ==== 84332+0+0 (secure 0 0 0) 0x7fe2dc0a5830 con 0x7fe2cc0039b0 2023-05-22T16:05:21.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:21.724+0000 7fe2d5ffb640 1 --2- 172.21.15.16:0/638550351 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fe2c40654d0 0x7fe2c4067980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:21.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:21.724+0000 7fe2d5ffb640 1 -- 172.21.15.16:0/638550351 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(194..194 src has 1..194) v4 ==== 6338+0+0 (secure 0 0 0) 0x7fe2dc11c3e0 con 0x7fe2cc0039b0 2023-05-22T16:05:21.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:21.724+0000 7fe2e4e9d640 1 --2- 172.21.15.16:0/638550351 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fe2c40654d0 0x7fe2c4067980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:21.744 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:21.728+0000 7fe2e4e9d640 1 --2- 172.21.15.16:0/638550351 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fe2c40654d0 0x7fe2c4067980 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fe2e015e660 tx=0x7fe2d000b3a0 comp rx=0 tx=0).ready entity=mgr.10542 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:21.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:21.728+0000 7fe2d5ffb640 1 -- 172.21.15.16:0/638550351 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fe2dc0e9e10 con 0x7fe2cc0039b0 2023-05-22T16:05:21.919 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:21.916+0000 7fe2e7902640 1 -- 172.21.15.16:0/638550351 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fe2e006c480 con 0x7fe2cc0039b0 2023-05-22T16:05:21.929 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:21.928+0000 7fe2d5ffb640 1 -- 172.21.15.16:0/638550351 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 24 and data pool 25 v77) v1 ==== 174+0+0 (secure 0 0 0) 0x7fe2dc0ed430 con 0x7fe2cc0039b0 2023-05-22T16:05:21.929 INFO:teuthology.orchestra.run.smithi016.stderr:new fs with metadata pool 24 and data pool 25 2023-05-22T16:05:21.935 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:21.932+0000 7fe2e7902640 1 -- 172.21.15.16:0/638550351 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fe2c40654d0 msgr2=0x7fe2c4067980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:21.935 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:21.932+0000 7fe2e7902640 1 --2- 172.21.15.16:0/638550351 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fe2c40654d0 0x7fe2c4067980 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fe2e015e660 tx=0x7fe2d000b3a0 comp rx=0 tx=0).stop 2023-05-22T16:05:21.936 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:21.932+0000 7fe2e7902640 1 -- 172.21.15.16:0/638550351 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe2cc0039b0 msgr2=0x7fe2e01403c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:21.936 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:21.932+0000 7fe2e7902640 1 --2- 172.21.15.16:0/638550351 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe2cc0039b0 0x7fe2e01403c0 secure :-1 s=READY pgs=2431 cs=0 l=1 rev1=1 crypto rx=0x7fe2dc01ca70 tx=0x7fe2cc005800 comp rx=0 tx=0).stop 2023-05-22T16:05:21.938 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:21.932+0000 7fe2e7902640 1 -- 172.21.15.16:0/638550351 shutdown_connections 2023-05-22T16:05:21.938 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:21.932+0000 7fe2e7902640 1 --2- 172.21.15.16:0/638550351 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe2e015db10 0x7fe2e015dfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:21.938 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:21.932+0000 7fe2e7902640 1 --2- 172.21.15.16:0/638550351 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe2e0143740 0x7fe2e00c50e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:21.939 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:21.932+0000 7fe2e7902640 1 --2- 172.21.15.16:0/638550351 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fe2c40654d0 0x7fe2c4067980 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:21.939 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:21.932+0000 7fe2e7902640 1 --2- 172.21.15.16:0/638550351 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe2cc0039b0 0x7fe2e01403c0 unknown :-1 s=CLOSED pgs=2431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:21.939 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:21.932+0000 7fe2e7902640 1 -- 172.21.15.16:0/638550351 >> 172.21.15.16:0/638550351 conn(0x7fe2e0069720 msgr2=0x7fe2e00af2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:05:21.939 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:21.932+0000 7fe2e7902640 1 -- 172.21.15.16:0/638550351 shutdown_connections 2023-05-22T16:05:21.940 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:21.936+0000 7fe2e7902640 1 -- 172.21.15.16:0/638550351 wait complete. 2023-05-22T16:05:21.971 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-05-22T16:05:22.088 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.084+0000 7f1b71f70640 1 Processor -- start 2023-05-22T16:05:22.088 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.084+0000 7f1b71f70640 1 -- start start 2023-05-22T16:05:22.089 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.088+0000 7f1b71f70640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f1b6c0ca4e0 0x7f1b6c15e460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:22.090 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.088+0000 7f1b71f70640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f1b6c0c9b40 0x7f1b6c0afb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:22.090 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.088+0000 7f1b71f70640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b6c0b0650 con 0x7f1b6c070df0 2023-05-22T16:05:22.090 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.088+0000 7f1b71f70640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b6c0b0370 con 0x7f1b6c0c9b40 2023-05-22T16:05:22.090 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.088+0000 7f1b71f70640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b6c15d980 con 0x7f1b6c0ca4e0 2023-05-22T16:05:22.090 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.088+0000 7f1b6affd640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f1b6c0c9b40 0x7f1b6c0afb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:22.091 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.088+0000 7f1b6bfff640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f1b6c070df0 0x7f1b6c071210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:48448/0 (socket says 172.21.15.16:48448) 2023-05-22T16:05:22.091 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.088+0000 7f1b6affd640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f1b6c0c9b40 0x7f1b6c0afb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41622/0 (socket says 172.21.15.16:41622) 2023-05-22T16:05:22.091 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.088+0000 7f1b6bfff640 1 -- 172.21.15.16:0/44231908 learned_addr learned my addr 172.21.15.16:0/44231908 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T16:05:22.091 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.088+0000 7f1b6b7fe640 1 --2- 172.21.15.16:0/44231908 >> v2:172.21.15.62:3300/0 conn(0x7f1b6c0ca4e0 0x7f1b6c15e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:22.091 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.088+0000 7f1b6affd640 1 -- 172.21.15.16:0/44231908 >> v2:172.21.15.62:3300/0 conn(0x7f1b6c0ca4e0 msgr2=0x7f1b6c15e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:05:22.091 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.088+0000 7f1b6affd640 1 --2- 172.21.15.16:0/44231908 >> v2:172.21.15.62:3300/0 conn(0x7f1b6c0ca4e0 0x7f1b6c15e460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:22.091 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.088+0000 7f1b6affd640 1 -- 172.21.15.16:0/44231908 >> v1:172.21.15.16:6789/0 conn(0x7f1b6c070df0 legacy=0x7f1b6c071210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:22.091 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.088+0000 7f1b6affd640 1 -- 172.21.15.16:0/44231908 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b6c15dc60 con 0x7f1b6c0c9b40 2023-05-22T16:05:22.092 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.088+0000 7f1b6a7fc640 1 -- 172.21.15.16:0/44231908 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3757594630 0 0) 0x7f1b6c0b0650 con 0x7f1b6c070df0 2023-05-22T16:05:22.097 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.088+0000 7f1b6affd640 1 --2- 172.21.15.16:0/44231908 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1b6c0c9b40 0x7f1b6c0afb40 secure :-1 s=READY pgs=2433 cs=0 l=1 rev1=1 crypto rx=0x7f1b6c0cad40 tx=0x7f1b5c01c430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff1893335071bf30 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:22.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.088+0000 7f1b6a7fc640 1 -- 172.21.15.16:0/44231908 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b5c0ab610 con 0x7f1b6c0c9b40 2023-05-22T16:05:22.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.088+0000 7f1b6a7fc640 1 -- 172.21.15.16:0/44231908 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b5c0ab770 con 0x7f1b6c0c9b40 2023-05-22T16:05:22.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.088+0000 7f1b6a7fc640 1 -- 172.21.15.16:0/44231908 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b5c0abb00 con 0x7f1b6c0c9b40 2023-05-22T16:05:22.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.088+0000 7f1b71f70640 1 -- 172.21.15.16:0/44231908 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1b6c0c9b40 msgr2=0x7f1b6c0afb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:22.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.088+0000 7f1b71f70640 1 --2- 172.21.15.16:0/44231908 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1b6c0c9b40 0x7f1b6c0afb40 secure :-1 s=READY pgs=2433 cs=0 l=1 rev1=1 crypto rx=0x7f1b6c0cad40 tx=0x7f1b5c01c430 comp rx=0 tx=0).stop 2023-05-22T16:05:22.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.088+0000 7f1b71f70640 1 -- 172.21.15.16:0/44231908 shutdown_connections 2023-05-22T16:05:22.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.088+0000 7f1b71f70640 1 --2- 172.21.15.16:0/44231908 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1b6c0c9b40 0x7f1b6c0afb40 unknown :-1 s=CLOSED pgs=2433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:22.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.088+0000 7f1b71f70640 1 --2- 172.21.15.16:0/44231908 >> v2:172.21.15.62:3300/0 conn(0x7f1b6c0ca4e0 0x7f1b6c15e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:22.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.088+0000 7f1b71f70640 1 -- 172.21.15.16:0/44231908 >> 172.21.15.16:0/44231908 conn(0x7f1b6c069720 msgr2=0x7f1b6c069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:05:22.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.088+0000 7f1b71f70640 1 -- 172.21.15.16:0/44231908 shutdown_connections 2023-05-22T16:05:22.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.088+0000 7f1b71f70640 1 -- 172.21.15.16:0/44231908 wait complete. 2023-05-22T16:05:22.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.092+0000 7f1b71f70640 1 Processor -- start 2023-05-22T16:05:22.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.092+0000 7f1b71f70640 1 -- start start 2023-05-22T16:05:22.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.092+0000 7f1b71f70640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1b6c070df0 0x7f1b6c150600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:22.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.092+0000 7f1b71f70640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1b6c0c9b40 0x7f1b6c151c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:22.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.092+0000 7f1b71f70640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1b6c0ca4e0 0x7f1b6c0cce10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:22.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.092+0000 7f1b71f70640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b6c1521c0 con 0x7f1b6c0c9b40 2023-05-22T16:05:22.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.092+0000 7f1b71f70640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b6c150be0 con 0x7f1b6c0ca4e0 2023-05-22T16:05:22.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.092+0000 7f1b71f70640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b6c150ec0 con 0x7f1b6c070df0 2023-05-22T16:05:22.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.092+0000 7f1b6affd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1b6c0c9b40 0x7f1b6c151c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:22.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.092+0000 7f1b6affd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1b6c0c9b40 0x7f1b6c151c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41636/0 (socket says 172.21.15.16:41636) 2023-05-22T16:05:22.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.092+0000 7f1b6affd640 1 -- 172.21.15.16:0/1206905441 learned_addr learned my addr 172.21.15.16:0/1206905441 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:05:22.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.092+0000 7f1b6bfff640 1 --2- 172.21.15.16:0/1206905441 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1b6c0ca4e0 0x7f1b6c0cce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:22.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.092+0000 7f1b6b7fe640 1 --2- 172.21.15.16:0/1206905441 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1b6c070df0 0x7f1b6c150600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:22.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.092+0000 7f1b6affd640 1 -- 172.21.15.16:0/1206905441 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1b6c070df0 msgr2=0x7f1b6c150600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:22.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.092+0000 7f1b6affd640 1 --2- 172.21.15.16:0/1206905441 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1b6c070df0 0x7f1b6c150600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:22.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.092+0000 7f1b6affd640 1 -- 172.21.15.16:0/1206905441 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1b6c0ca4e0 msgr2=0x7f1b6c0cce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:22.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.092+0000 7f1b6affd640 1 --2- 172.21.15.16:0/1206905441 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1b6c0ca4e0 0x7f1b6c0cce10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:22.103 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.092+0000 7f1b6affd640 1 -- 172.21.15.16:0/1206905441 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b5c009420 con 0x7f1b6c0c9b40 2023-05-22T16:05:22.103 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.092+0000 7f1b6affd640 1 --2- 172.21.15.16:0/1206905441 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1b6c0c9b40 0x7f1b6c151c80 secure :-1 s=READY pgs=2434 cs=0 l=1 rev1=1 crypto rx=0x7f1b5c019040 tx=0x7f1b5c01ac70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:22.103 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.092+0000 7f1b53fff640 1 -- 172.21.15.16:0/1206905441 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b5c0b5810 con 0x7f1b6c0c9b40 2023-05-22T16:05:22.103 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.092+0000 7f1b53fff640 1 -- 172.21.15.16:0/1206905441 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b5c0b5990 con 0x7f1b6c0c9b40 2023-05-22T16:05:22.103 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.092+0000 7f1b71f70640 1 -- 172.21.15.16:0/1206905441 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b6c0c8e80 con 0x7f1b6c0c9b40 2023-05-22T16:05:22.103 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.092+0000 7f1b53fff640 1 -- 172.21.15.16:0/1206905441 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b5c0b5d40 con 0x7f1b6c0c9b40 2023-05-22T16:05:22.103 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.092+0000 7f1b71f70640 1 -- 172.21.15.16:0/1206905441 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b6c0ca8b0 con 0x7f1b6c0c9b40 2023-05-22T16:05:22.104 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.092+0000 7f1b71f70640 1 -- 172.21.15.16:0/1206905441 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b6c0c9010 con 0x7f1b6c0c9b40 2023-05-22T16:05:22.104 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.096+0000 7f1b53fff640 1 -- 172.21.15.16:0/1206905441 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 92) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f1b5c0bf410 con 0x7f1b6c0c9b40 2023-05-22T16:05:22.104 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.096+0000 7f1b53fff640 1 --2- 172.21.15.16:0/1206905441 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f1b580654a0 0x7f1b58067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:22.104 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.096+0000 7f1b53fff640 1 -- 172.21.15.16:0/1206905441 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(195..195 src has 1..195) v4 ==== 6438+0+0 (secure 0 0 0) 0x7f1b5c11da60 con 0x7f1b6c0c9b40 2023-05-22T16:05:22.104 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.096+0000 7f1b6b7fe640 1 --2- 172.21.15.16:0/1206905441 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f1b580654a0 0x7f1b58067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:22.104 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.096+0000 7f1b6b7fe640 1 --2- 172.21.15.16:0/1206905441 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f1b580654a0 0x7f1b58067950 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f1b6c0ad560 tx=0x7f1b60008040 comp rx=0 tx=0).ready entity=mgr.10542 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:22.105 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.100+0000 7f1b53fff640 1 -- 172.21.15.16:0/1206905441 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f1b5c0eb240 con 0x7f1b6c0c9b40 2023-05-22T16:05:22.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.252+0000 7f1b71f70640 1 -- 172.21.15.16:0/1206905441 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1b6c04dbe0 con 0x7f1b6c0c9b40 2023-05-22T16:05:22.258 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.256+0000 7f1b53fff640 1 -- 172.21.15.16:0/1206905441 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v195) v1 ==== 74+0+18153 (secure 0 0 0) 0x7f1b5c0ab5d0 con 0x7f1b6c0c9b40 2023-05-22T16:05:22.259 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T16:05:22.259 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":195,"fsid":"b2719834-7847-4d98-90a6-f65d57f85c49","created":"2023-05-22T15:26:25.259314+0000","modified":"2023-05-22T16:05:21.921492+0000","last_up_change":"2023-05-22T15:26:36.448524+0000","last_in_change":"2023-05-22T15:26:27.675668+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":25,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-05-22T15:26:38.019781+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-05-22T15:26:43.570514+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_metadata","create_time":"2023-05-22T16:05:20.485167+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"195","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":25,"pool_name":"cephfs_data","create_time":"2023-05-22T16:05:20.900001+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"195","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9e7ff854-b25f-4cbb-a5c2-1799d9c53d2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":37021},{"type":"v1","addr":"172.21.15.16:6825","nonce":37021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":37021},{"type":"v1","addr":"172.21.15.16:6827","nonce":37021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":37021},{"type":"v1","addr":"172.21.15.16:6831","nonce":37021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":37021},{"type":"v1","addr":"172.21.15.16:6829","nonce":37021}]},"public_addr":"172.21.15.16:6825/37021","cluster_addr":"172.21.15.16:6827/37021","heartbeat_back_addr":"172.21.15.16:6831/37021","heartbeat_front_addr":"172.21.15.16:6829/37021","state":["exists","up"]},{"osd":1,"uuid":"4702fbc9-a927-4565-aca5-41dc69cc8264","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":37008},{"type":"v1","addr":"172.21.15.16:6801","nonce":37008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":37008},{"type":"v1","addr":"172.21.15.16:6803","nonce":37008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":37008},{"type":"v1","addr":"172.21.15.16:6807","nonce":37008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":37008},{"type":"v1","addr":"172.21.15.16:6805","nonce":37008}]},"public_addr":"172.21.15.16:6801/37008","cluster_addr":"172.21.15.16:6803/37008","heartbeat_back_addr":"172.21.15.16:6807/37008","heartbeat_front_addr":"172.21.15.16:6805/37008","state":["exists","up"]},{"osd":2,"uuid":"1e7bba46-4eba-47ef-bb81-5e2d371638ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":37009},{"type":"v1","addr":"172.21.15.16:6809","nonce":37009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":37009},{"type":"v1","addr":"172.21.15.16:6811","nonce":37009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":37009},{"type":"v1","addr":"172.21.15.16:6815","nonce":37009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":37009},{"type":"v1","addr":"172.21.15.16:6813","nonce":37009}]},"public_addr":"172.21.15.16:6809/37009","cluster_addr":"172.21.15.16:6811/37009","heartbeat_back_addr":"172.21.15.16:6815/37009","heartbeat_front_addr":"172.21.15.16:6813/37009","state":["exists","up"]},{"osd":3,"uuid":"383cacc0-5a88-4904-990e-498ca59ef48a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":37020},{"type":"v1","addr":"172.21.15.16:6817","nonce":37020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":37020},{"type":"v1","addr":"172.21.15.16:6819","nonce":37020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":37020},{"type":"v1","addr":"172.21.15.16:6823","nonce":37020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":37020},{"type":"v1","addr":"172.21.15.16:6821","nonce":37020}]},"public_addr":"172.21.15.16:6817/37020","cluster_addr":"172.21.15.16:6819/37020","heartbeat_back_addr":"172.21.15.16:6823/37020","heartbeat_front_addr":"172.21.15.16:6821/37020","state":["exists","up"]},{"osd":4,"uuid":"970500ef-95b7-42c1-b080-eed4fae7013d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":37353},{"type":"v1","addr":"172.21.15.62:6817","nonce":37353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6819","nonce":37353},{"type":"v1","addr":"172.21.15.62:6822","nonce":37353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37353},{"type":"v1","addr":"172.21.15.62:6831","nonce":37353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37353},{"type":"v1","addr":"172.21.15.62:6826","nonce":37353}]},"public_addr":"172.21.15.62:6817/37353","cluster_addr":"172.21.15.62:6822/37353","heartbeat_back_addr":"172.21.15.62:6831/37353","heartbeat_front_addr":"172.21.15.62:6826/37353","state":["exists","up"]},{"osd":5,"uuid":"022d67db-d4d8-445b-b1b9-8b446a00f5c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":37352},{"type":"v1","addr":"172.21.15.62:6801","nonce":37352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":37352},{"type":"v1","addr":"172.21.15.62:6803","nonce":37352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":37352},{"type":"v1","addr":"172.21.15.62:6807","nonce":37352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":37352},{"type":"v1","addr":"172.21.15.62:6805","nonce":37352}]},"public_addr":"172.21.15.62:6801/37352","cluster_addr":"172.21.15.62:6803/37352","heartbeat_back_addr":"172.21.15.62:6807/37352","heartbeat_front_addr":"172.21.15.62:6805/37352","state":["exists","up"]},{"osd":6,"uuid":"42bd372b-6688-4a1b-8407-e16e8c266e33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":37354},{"type":"v1","addr":"172.21.15.62:6820","nonce":37354}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":37354},{"type":"v1","addr":"172.21.15.62:6823","nonce":37354}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6829","nonce":37354},{"type":"v1","addr":"172.21.15.62:6830","nonce":37354}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6825","nonce":37354},{"type":"v1","addr":"172.21.15.62:6827","nonce":37354}]},"public_addr":"172.21.15.62:6820/37354","cluster_addr":"172.21.15.62:6823/37354","heartbeat_back_addr":"172.21.15.62:6830/37354","heartbeat_front_addr":"172.21.15.62:6827/37354","state":["exists","up"]},{"osd":7,"uuid":"ac70b2a3-f41b-4bf2-b227-7f2cbb55426d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37355},{"type":"v1","addr":"172.21.15.62:6809","nonce":37355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37355},{"type":"v1","addr":"172.21.15.62:6811","nonce":37355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":37355},{"type":"v1","addr":"172.21.15.62:6815","nonce":37355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":37355},{"type":"v1","addr":"172.21.15.62:6813","nonce":37355}]},"public_addr":"172.21.15.62:6809/37355","cluster_addr":"172.21.15.62:6811/37355","heartbeat_back_addr":"172.21.15.62:6815/37355","heartbeat_front_addr":"172.21.15.62:6813/37355","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.368896+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.107273+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.276888+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.287502+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.124791+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.251937+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.255679+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.232797+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":17,"snaps":[{"begin":2,"length":2}]},{"pool":21,"snaps":[{"begin":25,"length":9}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-05-22T16:05:22.260 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.256+0000 7f1b71f70640 1 -- 172.21.15.16:0/1206905441 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f1b580654a0 msgr2=0x7f1b58067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:22.261 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.256+0000 7f1b71f70640 1 --2- 172.21.15.16:0/1206905441 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f1b580654a0 0x7f1b58067950 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f1b6c0ad560 tx=0x7f1b60008040 comp rx=0 tx=0).stop 2023-05-22T16:05:22.261 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.256+0000 7f1b71f70640 1 -- 172.21.15.16:0/1206905441 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1b6c0c9b40 msgr2=0x7f1b6c151c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:22.261 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.256+0000 7f1b71f70640 1 --2- 172.21.15.16:0/1206905441 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1b6c0c9b40 0x7f1b6c151c80 secure :-1 s=READY pgs=2434 cs=0 l=1 rev1=1 crypto rx=0x7f1b5c019040 tx=0x7f1b5c01ac70 comp rx=0 tx=0).stop 2023-05-22T16:05:22.261 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.256+0000 7f1b71f70640 1 -- 172.21.15.16:0/1206905441 shutdown_connections 2023-05-22T16:05:22.261 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.256+0000 7f1b71f70640 1 --2- 172.21.15.16:0/1206905441 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1b6c0ca4e0 0x7f1b6c0cce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:22.261 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.256+0000 7f1b71f70640 1 --2- 172.21.15.16:0/1206905441 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f1b580654a0 0x7f1b58067950 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:22.261 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.256+0000 7f1b71f70640 1 --2- 172.21.15.16:0/1206905441 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1b6c0c9b40 0x7f1b6c151c80 unknown :-1 s=CLOSED pgs=2434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:22.262 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.256+0000 7f1b71f70640 1 --2- 172.21.15.16:0/1206905441 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1b6c070df0 0x7f1b6c150600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:22.262 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.256+0000 7f1b71f70640 1 -- 172.21.15.16:0/1206905441 >> 172.21.15.16:0/1206905441 conn(0x7f1b6c069720 msgr2=0x7f1b6c156640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:05:22.262 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.256+0000 7f1b71f70640 1 -- 172.21.15.16:0/1206905441 shutdown_connections 2023-05-22T16:05:22.262 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.260+0000 7f1b71f70640 1 -- 172.21.15.16:0/1206905441 wait complete. 2023-05-22T16:05:22.278 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-05-22T16:05:22.405 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.404+0000 7f24dd4d8640 1 Processor -- start 2023-05-22T16:05:22.405 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.404+0000 7f24dd4d8640 1 -- start start 2023-05-22T16:05:22.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.404+0000 7f24dd4d8640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f24d80ca4e0 0x7f24d815e460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:22.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.404+0000 7f24dd4d8640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f24d80c9b40 0x7f24d80afb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:22.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.404+0000 7f24dd4d8640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24d80b0650 con 0x7f24d8070df0 2023-05-22T16:05:22.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.404+0000 7f24dd4d8640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24d80b0370 con 0x7f24d80ca4e0 2023-05-22T16:05:22.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.404+0000 7f24dd4d8640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24d815d980 con 0x7f24d80c9b40 2023-05-22T16:05:22.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.404+0000 7f24d77fe640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f24d8070df0 0x7f24d8071210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:48452/0 (socket says 172.21.15.16:48452) 2023-05-22T16:05:22.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.404+0000 7f24d6ffd640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f24d80ca4e0 0x7f24d815e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:22.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.404+0000 7f24d77fe640 1 -- 172.21.15.16:0/2522600713 learned_addr learned my addr 172.21.15.16:0/2522600713 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T16:05:22.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.404+0000 7f24d67fc640 1 --2- 172.21.15.16:0/2522600713 >> v2:172.21.15.62:3300/0 conn(0x7f24d80c9b40 0x7f24d80afb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:22.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.404+0000 7f24d6ffd640 1 -- 172.21.15.16:0/2522600713 >> v2:172.21.15.62:3300/0 conn(0x7f24d80c9b40 msgr2=0x7f24d80afb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:05:22.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.404+0000 7f24d6ffd640 1 --2- 172.21.15.16:0/2522600713 >> v2:172.21.15.62:3300/0 conn(0x7f24d80c9b40 0x7f24d80afb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:22.408 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.404+0000 7f24d6ffd640 1 -- 172.21.15.16:0/2522600713 >> v1:172.21.15.16:6789/0 conn(0x7f24d8070df0 legacy=0x7f24d8071210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:22.408 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.404+0000 7f24d6ffd640 1 -- 172.21.15.16:0/2522600713 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24d815dc60 con 0x7f24d80ca4e0 2023-05-22T16:05:22.409 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.408+0000 7f24d6ffd640 1 --2- 172.21.15.16:0/2522600713 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f24d80ca4e0 0x7f24d815e460 secure :-1 s=READY pgs=2436 cs=0 l=1 rev1=1 crypto rx=0x7f24d806f0e0 tx=0x7f24c001c2f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=662ff6a495152a1d server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:22.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.408+0000 7f24d5ffb640 1 -- 172.21.15.16:0/2522600713 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24c00ab600 con 0x7f24d80ca4e0 2023-05-22T16:05:22.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.408+0000 7f24d5ffb640 1 -- 172.21.15.16:0/2522600713 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24c00a7050 con 0x7f24d80ca4e0 2023-05-22T16:05:22.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.408+0000 7f24d5ffb640 1 -- 172.21.15.16:0/2522600713 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24c00ab980 con 0x7f24d80ca4e0 2023-05-22T16:05:22.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.408+0000 7f24dd4d8640 1 -- 172.21.15.16:0/2522600713 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f24d80ca4e0 msgr2=0x7f24d815e460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:22.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.408+0000 7f24dd4d8640 1 --2- 172.21.15.16:0/2522600713 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f24d80ca4e0 0x7f24d815e460 secure :-1 s=READY pgs=2436 cs=0 l=1 rev1=1 crypto rx=0x7f24d806f0e0 tx=0x7f24c001c2f0 comp rx=0 tx=0).stop 2023-05-22T16:05:22.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.408+0000 7f24dd4d8640 1 -- 172.21.15.16:0/2522600713 shutdown_connections 2023-05-22T16:05:22.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.408+0000 7f24dd4d8640 1 --2- 172.21.15.16:0/2522600713 >> v2:172.21.15.62:3300/0 conn(0x7f24d80c9b40 0x7f24d80afb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:22.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.408+0000 7f24dd4d8640 1 --2- 172.21.15.16:0/2522600713 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f24d80ca4e0 0x7f24d815e460 unknown :-1 s=CLOSED pgs=2436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:22.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.408+0000 7f24dd4d8640 1 -- 172.21.15.16:0/2522600713 >> 172.21.15.16:0/2522600713 conn(0x7f24d8069720 msgr2=0x7f24d8069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:05:22.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.408+0000 7f24dd4d8640 1 -- 172.21.15.16:0/2522600713 shutdown_connections 2023-05-22T16:05:22.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.408+0000 7f24dd4d8640 1 -- 172.21.15.16:0/2522600713 wait complete. 2023-05-22T16:05:22.413 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.408+0000 7f24dd4d8640 1 Processor -- start 2023-05-22T16:05:22.413 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.408+0000 7f24dd4d8640 1 -- start start 2023-05-22T16:05:22.413 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.408+0000 7f24dd4d8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f24d8070df0 0x7f24d814e860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:22.414 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.408+0000 7f24dd4d8640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f24d80c9b40 0x7f24d80c1a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:22.414 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.408+0000 7f24dd4d8640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f24d80ca4e0 0x7f24d80c1f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:22.414 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.408+0000 7f24dd4d8640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24d815bd10 con 0x7f24d8070df0 2023-05-22T16:05:22.414 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.408+0000 7f24dd4d8640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24d815c890 con 0x7f24d80ca4e0 2023-05-22T16:05:22.414 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.408+0000 7f24dd4d8640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24d814eda0 con 0x7f24d80c9b40 2023-05-22T16:05:22.414 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.408+0000 7f24d6ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f24d8070df0 0x7f24d814e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:22.414 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.408+0000 7f24d6ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f24d8070df0 0x7f24d814e860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41656/0 (socket says 172.21.15.16:41656) 2023-05-22T16:05:22.415 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.408+0000 7f24d6ffd640 1 -- 172.21.15.16:0/9824523 learned_addr learned my addr 172.21.15.16:0/9824523 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:05:22.415 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.408+0000 7f24d67fc640 1 --2- 172.21.15.16:0/9824523 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f24d80c9b40 0x7f24d80c1a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:22.415 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.408+0000 7f24d77fe640 1 --2- 172.21.15.16:0/9824523 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f24d80ca4e0 0x7f24d80c1f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:22.415 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.408+0000 7f24d6ffd640 1 -- 172.21.15.16:0/9824523 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f24d80c9b40 msgr2=0x7f24d80c1a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:22.415 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.408+0000 7f24d6ffd640 1 --2- 172.21.15.16:0/9824523 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f24d80c9b40 0x7f24d80c1a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:22.415 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.408+0000 7f24d6ffd640 1 -- 172.21.15.16:0/9824523 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f24d80ca4e0 msgr2=0x7f24d80c1f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:22.415 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.408+0000 7f24d6ffd640 1 --2- 172.21.15.16:0/9824523 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f24d80ca4e0 0x7f24d80c1f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:22.415 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.408+0000 7f24d6ffd640 1 -- 172.21.15.16:0/9824523 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24c0009420 con 0x7f24d8070df0 2023-05-22T16:05:22.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.412+0000 7f24d6ffd640 1 --2- 172.21.15.16:0/9824523 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f24d8070df0 0x7f24d814e860 secure :-1 s=READY pgs=2437 cs=0 l=1 rev1=1 crypto rx=0x7f24c001c7d0 tx=0x7f24c0019090 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:22.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.412+0000 7f24cf7fe640 1 -- 172.21.15.16:0/9824523 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24c00b4810 con 0x7f24d8070df0 2023-05-22T16:05:22.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.412+0000 7f24cf7fe640 1 -- 172.21.15.16:0/9824523 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24c00b4970 con 0x7f24d8070df0 2023-05-22T16:05:22.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.412+0000 7f24dd4d8640 1 -- 172.21.15.16:0/9824523 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24d80b0080 con 0x7f24d8070df0 2023-05-22T16:05:22.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.412+0000 7f24cf7fe640 1 -- 172.21.15.16:0/9824523 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24c00b4d00 con 0x7f24d8070df0 2023-05-22T16:05:22.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.412+0000 7f24dd4d8640 1 -- 172.21.15.16:0/9824523 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24d815df40 con 0x7f24d8070df0 2023-05-22T16:05:22.417 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.412+0000 7f24dd4d8640 1 -- 172.21.15.16:0/9824523 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24d80ca8b0 con 0x7f24d8070df0 2023-05-22T16:05:22.418 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.412+0000 7f24cf7fe640 1 -- 172.21.15.16:0/9824523 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 92) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f24c00b2040 con 0x7f24d8070df0 2023-05-22T16:05:22.418 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.412+0000 7f24cf7fe640 1 --2- 172.21.15.16:0/9824523 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f24b80654a0 0x7f24b8067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:22.418 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.412+0000 7f24cf7fe640 1 -- 172.21.15.16:0/9824523 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(195..195 src has 1..195) v4 ==== 6438+0+0 (secure 0 0 0) 0x7f24c011d090 con 0x7f24d8070df0 2023-05-22T16:05:22.418 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.412+0000 7f24d67fc640 1 --2- 172.21.15.16:0/9824523 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f24b80654a0 0x7f24b8067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:22.418 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.412+0000 7f24d67fc640 1 --2- 172.21.15.16:0/9824523 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f24b80654a0 0x7f24b8067950 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f24d80c9b10 tx=0x7f24c8008040 comp rx=0 tx=0).ready entity=mgr.10542 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:22.419 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.416+0000 7f24cf7fe640 1 -- 172.21.15.16:0/9824523 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f24c00ea970 con 0x7f24d8070df0 2023-05-22T16:05:22.575 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.572+0000 7f24dd4d8640 1 -- 172.21.15.16:0/9824523 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f24d815cd00 con 0x7f24d8070df0 2023-05-22T16:05:22.578 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.576+0000 7f24cf7fe640 1 -- 172.21.15.16:0/9824523 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v195) v1 ==== 74+0+18153 (secure 0 0 0) 0x7f24d815cd00 con 0x7f24d8070df0 2023-05-22T16:05:22.578 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T16:05:22.579 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":195,"fsid":"b2719834-7847-4d98-90a6-f65d57f85c49","created":"2023-05-22T15:26:25.259314+0000","modified":"2023-05-22T16:05:21.921492+0000","last_up_change":"2023-05-22T15:26:36.448524+0000","last_in_change":"2023-05-22T15:26:27.675668+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":25,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-05-22T15:26:38.019781+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-05-22T15:26:43.570514+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_metadata","create_time":"2023-05-22T16:05:20.485167+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"195","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":25,"pool_name":"cephfs_data","create_time":"2023-05-22T16:05:20.900001+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"195","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9e7ff854-b25f-4cbb-a5c2-1799d9c53d2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":37021},{"type":"v1","addr":"172.21.15.16:6825","nonce":37021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":37021},{"type":"v1","addr":"172.21.15.16:6827","nonce":37021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":37021},{"type":"v1","addr":"172.21.15.16:6831","nonce":37021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":37021},{"type":"v1","addr":"172.21.15.16:6829","nonce":37021}]},"public_addr":"172.21.15.16:6825/37021","cluster_addr":"172.21.15.16:6827/37021","heartbeat_back_addr":"172.21.15.16:6831/37021","heartbeat_front_addr":"172.21.15.16:6829/37021","state":["exists","up"]},{"osd":1,"uuid":"4702fbc9-a927-4565-aca5-41dc69cc8264","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":37008},{"type":"v1","addr":"172.21.15.16:6801","nonce":37008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":37008},{"type":"v1","addr":"172.21.15.16:6803","nonce":37008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":37008},{"type":"v1","addr":"172.21.15.16:6807","nonce":37008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":37008},{"type":"v1","addr":"172.21.15.16:6805","nonce":37008}]},"public_addr":"172.21.15.16:6801/37008","cluster_addr":"172.21.15.16:6803/37008","heartbeat_back_addr":"172.21.15.16:6807/37008","heartbeat_front_addr":"172.21.15.16:6805/37008","state":["exists","up"]},{"osd":2,"uuid":"1e7bba46-4eba-47ef-bb81-5e2d371638ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":37009},{"type":"v1","addr":"172.21.15.16:6809","nonce":37009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":37009},{"type":"v1","addr":"172.21.15.16:6811","nonce":37009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":37009},{"type":"v1","addr":"172.21.15.16:6815","nonce":37009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":37009},{"type":"v1","addr":"172.21.15.16:6813","nonce":37009}]},"public_addr":"172.21.15.16:6809/37009","cluster_addr":"172.21.15.16:6811/37009","heartbeat_back_addr":"172.21.15.16:6815/37009","heartbeat_front_addr":"172.21.15.16:6813/37009","state":["exists","up"]},{"osd":3,"uuid":"383cacc0-5a88-4904-990e-498ca59ef48a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":37020},{"type":"v1","addr":"172.21.15.16:6817","nonce":37020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":37020},{"type":"v1","addr":"172.21.15.16:6819","nonce":37020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":37020},{"type":"v1","addr":"172.21.15.16:6823","nonce":37020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":37020},{"type":"v1","addr":"172.21.15.16:6821","nonce":37020}]},"public_addr":"172.21.15.16:6817/37020","cluster_addr":"172.21.15.16:6819/37020","heartbeat_back_addr":"172.21.15.16:6823/37020","heartbeat_front_addr":"172.21.15.16:6821/37020","state":["exists","up"]},{"osd":4,"uuid":"970500ef-95b7-42c1-b080-eed4fae7013d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":37353},{"type":"v1","addr":"172.21.15.62:6817","nonce":37353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6819","nonce":37353},{"type":"v1","addr":"172.21.15.62:6822","nonce":37353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37353},{"type":"v1","addr":"172.21.15.62:6831","nonce":37353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37353},{"type":"v1","addr":"172.21.15.62:6826","nonce":37353}]},"public_addr":"172.21.15.62:6817/37353","cluster_addr":"172.21.15.62:6822/37353","heartbeat_back_addr":"172.21.15.62:6831/37353","heartbeat_front_addr":"172.21.15.62:6826/37353","state":["exists","up"]},{"osd":5,"uuid":"022d67db-d4d8-445b-b1b9-8b446a00f5c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":37352},{"type":"v1","addr":"172.21.15.62:6801","nonce":37352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":37352},{"type":"v1","addr":"172.21.15.62:6803","nonce":37352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":37352},{"type":"v1","addr":"172.21.15.62:6807","nonce":37352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":37352},{"type":"v1","addr":"172.21.15.62:6805","nonce":37352}]},"public_addr":"172.21.15.62:6801/37352","cluster_addr":"172.21.15.62:6803/37352","heartbeat_back_addr":"172.21.15.62:6807/37352","heartbeat_front_addr":"172.21.15.62:6805/37352","state":["exists","up"]},{"osd":6,"uuid":"42bd372b-6688-4a1b-8407-e16e8c266e33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":37354},{"type":"v1","addr":"172.21.15.62:6820","nonce":37354}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":37354},{"type":"v1","addr":"172.21.15.62:6823","nonce":37354}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6829","nonce":37354},{"type":"v1","addr":"172.21.15.62:6830","nonce":37354}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6825","nonce":37354},{"type":"v1","addr":"172.21.15.62:6827","nonce":37354}]},"public_addr":"172.21.15.62:6820/37354","cluster_addr":"172.21.15.62:6823/37354","heartbeat_back_addr":"172.21.15.62:6830/37354","heartbeat_front_addr":"172.21.15.62:6827/37354","state":["exists","up"]},{"osd":7,"uuid":"ac70b2a3-f41b-4bf2-b227-7f2cbb55426d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37355},{"type":"v1","addr":"172.21.15.62:6809","nonce":37355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37355},{"type":"v1","addr":"172.21.15.62:6811","nonce":37355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":37355},{"type":"v1","addr":"172.21.15.62:6815","nonce":37355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":37355},{"type":"v1","addr":"172.21.15.62:6813","nonce":37355}]},"public_addr":"172.21.15.62:6809/37355","cluster_addr":"172.21.15.62:6811/37355","heartbeat_back_addr":"172.21.15.62:6815/37355","heartbeat_front_addr":"172.21.15.62:6813/37355","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.368896+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.107273+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.276888+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.287502+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.124791+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.251937+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.255679+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.232797+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":17,"snaps":[{"begin":2,"length":2}]},{"pool":21,"snaps":[{"begin":25,"length":9}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-05-22T16:05:22.581 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.580+0000 7f24dd4d8640 1 -- 172.21.15.16:0/9824523 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f24b80654a0 msgr2=0x7f24b8067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:22.582 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.580+0000 7f24dd4d8640 1 --2- 172.21.15.16:0/9824523 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f24b80654a0 0x7f24b8067950 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f24d80c9b10 tx=0x7f24c8008040 comp rx=0 tx=0).stop 2023-05-22T16:05:22.582 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.580+0000 7f24dd4d8640 1 -- 172.21.15.16:0/9824523 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f24d8070df0 msgr2=0x7f24d814e860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:22.583 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.580+0000 7f24dd4d8640 1 --2- 172.21.15.16:0/9824523 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f24d8070df0 0x7f24d814e860 secure :-1 s=READY pgs=2437 cs=0 l=1 rev1=1 crypto rx=0x7f24c001c7d0 tx=0x7f24c0019090 comp rx=0 tx=0).stop 2023-05-22T16:05:22.583 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.580+0000 7f24dd4d8640 1 -- 172.21.15.16:0/9824523 shutdown_connections 2023-05-22T16:05:22.583 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.580+0000 7f24dd4d8640 1 --2- 172.21.15.16:0/9824523 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f24d80ca4e0 0x7f24d80c1f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:22.583 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.580+0000 7f24dd4d8640 1 --2- 172.21.15.16:0/9824523 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f24d80c9b40 0x7f24d80c1a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:22.583 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.580+0000 7f24dd4d8640 1 --2- 172.21.15.16:0/9824523 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f24b80654a0 0x7f24b8067950 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:22.584 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.580+0000 7f24dd4d8640 1 --2- 172.21.15.16:0/9824523 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f24d8070df0 0x7f24d814e860 unknown :-1 s=CLOSED pgs=2437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:22.584 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.580+0000 7f24dd4d8640 1 -- 172.21.15.16:0/9824523 >> 172.21.15.16:0/9824523 conn(0x7f24d8069720 msgr2=0x7f24d81583a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:05:22.584 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.580+0000 7f24dd4d8640 1 -- 172.21.15.16:0/9824523 shutdown_connections 2023-05-22T16:05:22.584 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.580+0000 7f24dd4d8640 1 -- 172.21.15.16:0/9824523 wait complete. 2023-05-22T16:05:22.599 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-05-22T16:05:22.725 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.724+0000 7f0cf5ba9640 1 Processor -- start 2023-05-22T16:05:22.725 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.724+0000 7f0cf5ba9640 1 -- start start 2023-05-22T16:05:22.726 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.724+0000 7f0cf5ba9640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f0cf0159970 0x7f0cf0159d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:22.726 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.724+0000 7f0cf5ba9640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f0cf015db00 0x7f0cf006b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:22.726 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.724+0000 7f0cf5ba9640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0cf0070c30 con 0x7f0cf015d680 2023-05-22T16:05:22.727 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.724+0000 7f0cf5ba9640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0cf00cb970 con 0x7f0cf015db00 2023-05-22T16:05:22.727 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.724+0000 7f0cf5ba9640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0cf0070e60 con 0x7f0cf0159970 2023-05-22T16:05:22.727 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.724+0000 7f0ceffff640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f0cf015db00 0x7f0cf006b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:22.727 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.724+0000 7f0ceeffd640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f0cf015d680 0x7f0cf00cb260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:48466/0 (socket says 172.21.15.16:48466) 2023-05-22T16:05:22.727 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.724+0000 7f0ceeffd640 1 -- 172.21.15.16:0/3363541343 learned_addr learned my addr 172.21.15.16:0/3363541343 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T16:05:22.727 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.724+0000 7f0cef7fe640 1 --2- 172.21.15.16:0/3363541343 >> v2:172.21.15.62:3300/0 conn(0x7f0cf0159970 0x7f0cf0159d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:22.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.724+0000 7f0ceffff640 1 -- 172.21.15.16:0/3363541343 >> v2:172.21.15.62:3300/0 conn(0x7f0cf0159970 msgr2=0x7f0cf0159d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:05:22.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.724+0000 7f0ceffff640 1 --2- 172.21.15.16:0/3363541343 >> v2:172.21.15.62:3300/0 conn(0x7f0cf0159970 0x7f0cf0159d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:22.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.724+0000 7f0ceffff640 1 -- 172.21.15.16:0/3363541343 >> v1:172.21.15.16:6789/0 conn(0x7f0cf015d680 legacy=0x7f0cf00cb260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:22.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.724+0000 7f0ceffff640 1 -- 172.21.15.16:0/3363541343 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cf0071140 con 0x7f0cf015db00 2023-05-22T16:05:22.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.724+0000 7f0cee7fc640 1 -- 172.21.15.16:0/3363541343 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2429031021 0 0) 0x7f0cf0070c30 con 0x7f0cf015d680 2023-05-22T16:05:22.729 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.728+0000 7f0ceffff640 1 --2- 172.21.15.16:0/3363541343 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0cf015db00 0x7f0cf006b6c0 secure :-1 s=READY pgs=2439 cs=0 l=1 rev1=1 crypto rx=0x7f0cf006daf0 tx=0x7f0ce801c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ea8375330f07c35 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:22.730 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.728+0000 7f0cee7fc640 1 -- 172.21.15.16:0/3363541343 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ce80a5820 con 0x7f0cf015db00 2023-05-22T16:05:22.730 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.728+0000 7f0cee7fc640 1 -- 172.21.15.16:0/3363541343 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ce80a5980 con 0x7f0cf015db00 2023-05-22T16:05:22.730 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.728+0000 7f0cee7fc640 1 -- 172.21.15.16:0/3363541343 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ce80a5d10 con 0x7f0cf015db00 2023-05-22T16:05:22.730 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.728+0000 7f0cf5ba9640 1 -- 172.21.15.16:0/3363541343 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0cf015db00 msgr2=0x7f0cf006b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:22.730 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.728+0000 7f0cf5ba9640 1 --2- 172.21.15.16:0/3363541343 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0cf015db00 0x7f0cf006b6c0 secure :-1 s=READY pgs=2439 cs=0 l=1 rev1=1 crypto rx=0x7f0cf006daf0 tx=0x7f0ce801c850 comp rx=0 tx=0).stop 2023-05-22T16:05:22.730 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.728+0000 7f0cf5ba9640 1 -- 172.21.15.16:0/3363541343 shutdown_connections 2023-05-22T16:05:22.730 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.728+0000 7f0cf5ba9640 1 --2- 172.21.15.16:0/3363541343 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0cf015db00 0x7f0cf006b6c0 unknown :-1 s=CLOSED pgs=2439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:22.731 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.728+0000 7f0cf5ba9640 1 --2- 172.21.15.16:0/3363541343 >> v2:172.21.15.62:3300/0 conn(0x7f0cf0159970 0x7f0cf0159d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:22.731 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.728+0000 7f0cf5ba9640 1 -- 172.21.15.16:0/3363541343 >> 172.21.15.16:0/3363541343 conn(0x7f0cf0069720 msgr2=0x7f0cf0069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:05:22.731 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.728+0000 7f0cf5ba9640 1 -- 172.21.15.16:0/3363541343 shutdown_connections 2023-05-22T16:05:22.731 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.728+0000 7f0cf5ba9640 1 -- 172.21.15.16:0/3363541343 wait complete. 2023-05-22T16:05:22.732 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.728+0000 7f0cf5ba9640 1 Processor -- start 2023-05-22T16:05:22.732 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.728+0000 7f0cf5ba9640 1 -- start start 2023-05-22T16:05:22.732 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.728+0000 7f0cf5ba9640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0cf0159970 0x7f0cf014efd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:22.733 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.728+0000 7f0cf5ba9640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0cf015d680 0x7f0cf014d2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:22.733 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.728+0000 7f0cf5ba9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0cf015db00 0x7f0cf01534a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:22.733 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.728+0000 7f0cf5ba9640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cf0150940 con 0x7f0cf015db00 2023-05-22T16:05:22.733 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.728+0000 7f0cf5ba9640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cf014ffa0 con 0x7f0cf015d680 2023-05-22T16:05:22.733 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.728+0000 7f0cf5ba9640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0cf0150280 con 0x7f0cf0159970 2023-05-22T16:05:22.733 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.728+0000 7f0ceffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0cf015db00 0x7f0cf01534a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:22.733 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.728+0000 7f0ceffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0cf015db00 0x7f0cf01534a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41686/0 (socket says 172.21.15.16:41686) 2023-05-22T16:05:22.734 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.728+0000 7f0ceffff640 1 -- 172.21.15.16:0/1846298939 learned_addr learned my addr 172.21.15.16:0/1846298939 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:05:22.734 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.728+0000 7f0cef7fe640 1 --2- 172.21.15.16:0/1846298939 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0cf0159970 0x7f0cf014efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:22.734 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.728+0000 7f0ceeffd640 1 --2- 172.21.15.16:0/1846298939 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0cf015d680 0x7f0cf014d2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:22.734 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.728+0000 7f0ceffff640 1 -- 172.21.15.16:0/1846298939 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0cf0159970 msgr2=0x7f0cf014efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:22.734 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.728+0000 7f0ceffff640 1 --2- 172.21.15.16:0/1846298939 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0cf0159970 0x7f0cf014efd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:22.734 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.728+0000 7f0ceffff640 1 -- 172.21.15.16:0/1846298939 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0cf015d680 msgr2=0x7f0cf014d2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:22.734 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.728+0000 7f0ceffff640 1 --2- 172.21.15.16:0/1846298939 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0cf015d680 0x7f0cf014d2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:22.734 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.728+0000 7f0ceffff640 1 -- 172.21.15.16:0/1846298939 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ce8009a70 con 0x7f0cf015db00 2023-05-22T16:05:22.735 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.732+0000 7f0ceffff640 1 --2- 172.21.15.16:0/1846298939 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0cf015db00 0x7f0cf01534a0 secure :-1 s=READY pgs=2440 cs=0 l=1 rev1=1 crypto rx=0x7f0ce80b0040 tx=0x7f0ce80195f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:22.736 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.732+0000 7f0cd7fff640 1 -- 172.21.15.16:0/1846298939 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ce80b9650 con 0x7f0cf015db00 2023-05-22T16:05:22.736 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.732+0000 7f0cd7fff640 1 -- 172.21.15.16:0/1846298939 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ce80b97b0 con 0x7f0cf015db00 2023-05-22T16:05:22.736 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.732+0000 7f0cf5ba9640 1 -- 172.21.15.16:0/1846298939 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0cf00cae20 con 0x7f0cf015db00 2023-05-22T16:05:22.736 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.732+0000 7f0cd7fff640 1 -- 172.21.15.16:0/1846298939 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ce80b9b20 con 0x7f0cf015db00 2023-05-22T16:05:22.736 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.732+0000 7f0cf5ba9640 1 -- 172.21.15.16:0/1846298939 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0cf00cac90 con 0x7f0cf015db00 2023-05-22T16:05:22.736 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.732+0000 7f0cf5ba9640 1 -- 172.21.15.16:0/1846298939 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0cf015adc0 con 0x7f0cf015db00 2023-05-22T16:05:22.737 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.732+0000 7f0cd7fff640 1 -- 172.21.15.16:0/1846298939 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 92) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f0ce80b9c80 con 0x7f0cf015db00 2023-05-22T16:05:22.737 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.732+0000 7f0cd7fff640 1 --2- 172.21.15.16:0/1846298939 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f0cdc0654a0 0x7f0cdc067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:22.737 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.732+0000 7f0cd7fff640 1 -- 172.21.15.16:0/1846298939 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(195..195 src has 1..195) v4 ==== 6438+0+0 (secure 0 0 0) 0x7f0ce811dea0 con 0x7f0cf015db00 2023-05-22T16:05:22.737 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.732+0000 7f0cef7fe640 1 --2- 172.21.15.16:0/1846298939 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f0cdc0654a0 0x7f0cdc067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:22.737 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.732+0000 7f0cef7fe640 1 --2- 172.21.15.16:0/1846298939 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f0cdc0654a0 0x7f0cdc067950 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f0cf0153310 tx=0x7f0ce0008040 comp rx=0 tx=0).ready entity=mgr.10542 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:22.739 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.736+0000 7f0cd7fff640 1 -- 172.21.15.16:0/1846298939 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f0ce80ef2d0 con 0x7f0cf015db00 2023-05-22T16:05:22.930 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:22.928+0000 7f0cf5ba9640 1 -- 172.21.15.16:0/1846298939 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f0cf0150a80 con 0x7f0cf015db00 2023-05-22T16:05:23.450 INFO:tasks.ceph.mds.c.smithi016.stderr:2023-05-22T16:05:23.448+0000 7f4768e9f640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-05-22T16:05:23.944 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:23.940+0000 7f0cd7fff640 1 -- 172.21.15.16:0/1846298939 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v80) v1 ==== 118+0+0 (secure 0 0 0) 0x7f0cf0150a80 con 0x7f0cf015db00 2023-05-22T16:05:23.947 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:23.944+0000 7f0cf5ba9640 1 -- 172.21.15.16:0/1846298939 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f0cdc0654a0 msgr2=0x7f0cdc067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:23.947 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:23.944+0000 7f0cf5ba9640 1 --2- 172.21.15.16:0/1846298939 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f0cdc0654a0 0x7f0cdc067950 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f0cf0153310 tx=0x7f0ce0008040 comp rx=0 tx=0).stop 2023-05-22T16:05:23.947 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:23.944+0000 7f0cf5ba9640 1 -- 172.21.15.16:0/1846298939 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0cf015db00 msgr2=0x7f0cf01534a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:23.947 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:23.944+0000 7f0cf5ba9640 1 --2- 172.21.15.16:0/1846298939 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0cf015db00 0x7f0cf01534a0 secure :-1 s=READY pgs=2440 cs=0 l=1 rev1=1 crypto rx=0x7f0ce80b0040 tx=0x7f0ce80195f0 comp rx=0 tx=0).stop 2023-05-22T16:05:23.947 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:23.944+0000 7f0cf5ba9640 1 -- 172.21.15.16:0/1846298939 shutdown_connections 2023-05-22T16:05:23.948 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:23.944+0000 7f0cf5ba9640 1 --2- 172.21.15.16:0/1846298939 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f0cdc0654a0 0x7f0cdc067950 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:23.948 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:23.944+0000 7f0cf5ba9640 1 --2- 172.21.15.16:0/1846298939 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0cf015db00 0x7f0cf01534a0 unknown :-1 s=CLOSED pgs=2440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:23.948 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:23.944+0000 7f0cf5ba9640 1 --2- 172.21.15.16:0/1846298939 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0cf015d680 0x7f0cf014d2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:23.948 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:23.944+0000 7f0cf5ba9640 1 --2- 172.21.15.16:0/1846298939 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0cf0159970 0x7f0cf014efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:23.948 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:23.944+0000 7f0cf5ba9640 1 -- 172.21.15.16:0/1846298939 >> 172.21.15.16:0/1846298939 conn(0x7f0cf0069720 msgr2=0x7f0cf0157010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:05:23.948 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:23.944+0000 7f0cf5ba9640 1 -- 172.21.15.16:0/1846298939 shutdown_connections 2023-05-22T16:05:23.949 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:23.944+0000 7f0cf5ba9640 1 -- 172.21.15.16:0/1846298939 wait complete. 2023-05-22T16:05:23.967 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2023-05-22T16:05:23.967 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-05-22T16:05:24.092 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:24.088+0000 7f3e26b2a640 1 Processor -- start 2023-05-22T16:05:24.092 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:24.088+0000 7f3e26b2a640 1 -- start start 2023-05-22T16:05:24.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:24.092+0000 7f3e26b2a640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f3e200d2480 0x7f3e200d2900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:24.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:24.092+0000 7f3e26b2a640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f3e200b5340 0x7f3e200b5790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:24.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:24.092+0000 7f3e26b2a640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e200cccc0 con 0x7f3e2015bca0 2023-05-22T16:05:24.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:24.092+0000 7f3e26b2a640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e200cd660 con 0x7f3e200b5340 2023-05-22T16:05:24.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:24.092+0000 7f3e26b2a640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e200ce190 con 0x7f3e200d2480 2023-05-22T16:05:24.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:24.092+0000 7f3e248c6640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f3e2015bca0 0x7f3e200cda80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:48482/0 (socket says 172.21.15.16:48482) 2023-05-22T16:05:24.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:24.092+0000 7f3e250c7640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f3e200b5340 0x7f3e200b5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:24.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:24.092+0000 7f3e248c6640 1 -- 172.21.15.16:0/2843164097 learned_addr learned my addr 172.21.15.16:0/2843164097 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T16:05:24.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:24.092+0000 7f3e1ffff640 1 --2- 172.21.15.16:0/2843164097 >> v2:172.21.15.62:3300/0 conn(0x7f3e200d2480 0x7f3e200d2900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:24.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:24.092+0000 7f3e250c7640 1 -- 172.21.15.16:0/2843164097 >> v2:172.21.15.62:3300/0 conn(0x7f3e200d2480 msgr2=0x7f3e200d2900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:05:24.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:24.092+0000 7f3e250c7640 1 --2- 172.21.15.16:0/2843164097 >> v2:172.21.15.62:3300/0 conn(0x7f3e200d2480 0x7f3e200d2900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:24.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:24.092+0000 7f3e250c7640 1 -- 172.21.15.16:0/2843164097 >> v1:172.21.15.16:6789/0 conn(0x7f3e2015bca0 legacy=0x7f3e200cda80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:24.096 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:24.092+0000 7f3e250c7640 1 -- 172.21.15.16:0/2843164097 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e200ccfa0 con 0x7f3e200b5340 2023-05-22T16:05:24.096 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:24.092+0000 7f3e1f7fe640 1 -- 172.21.15.16:0/2843164097 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3856789875 0 0) 0x7f3e200cccc0 con 0x7f3e2015bca0 2023-05-22T16:05:24.097 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:24.092+0000 7f3e250c7640 1 --2- 172.21.15.16:0/2843164097 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3e200b5340 0x7f3e200b5790 secure :-1 s=READY pgs=2442 cs=0 l=1 rev1=1 crypto rx=0x7f3e20065e80 tx=0x7f3e1801c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c845f0fcae8c3c68 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:24.097 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:24.092+0000 7f3e1f7fe640 1 -- 172.21.15.16:0/2843164097 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e180a58e0 con 0x7f3e200b5340 2023-05-22T16:05:24.097 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:24.092+0000 7f3e1f7fe640 1 -- 172.21.15.16:0/2843164097 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e180a5a40 con 0x7f3e200b5340 2023-05-22T16:05:24.097 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:24.092+0000 7f3e1f7fe640 1 -- 172.21.15.16:0/2843164097 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e180a5dd0 con 0x7f3e200b5340 2023-05-22T16:05:24.097 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:24.096+0000 7f3e26b2a640 1 -- 172.21.15.16:0/2843164097 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3e200b5340 msgr2=0x7f3e200b5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:24.097 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:24.096+0000 7f3e26b2a640 1 --2- 172.21.15.16:0/2843164097 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3e200b5340 0x7f3e200b5790 secure :-1 s=READY pgs=2442 cs=0 l=1 rev1=1 crypto rx=0x7f3e20065e80 tx=0x7f3e1801c910 comp rx=0 tx=0).stop 2023-05-22T16:05:24.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:24.096+0000 7f3e26b2a640 1 -- 172.21.15.16:0/2843164097 shutdown_connections 2023-05-22T16:05:24.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:24.096+0000 7f3e26b2a640 1 --2- 172.21.15.16:0/2843164097 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3e200b5340 0x7f3e200b5790 unknown :-1 s=CLOSED pgs=2442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:24.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:24.096+0000 7f3e26b2a640 1 --2- 172.21.15.16:0/2843164097 >> v2:172.21.15.62:3300/0 conn(0x7f3e200d2480 0x7f3e200d2900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:24.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:24.096+0000 7f3e26b2a640 1 -- 172.21.15.16:0/2843164097 >> 172.21.15.16:0/2843164097 conn(0x7f3e200655c0 msgr2=0x7f3e20070e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:05:24.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:24.096+0000 7f3e26b2a640 1 -- 172.21.15.16:0/2843164097 shutdown_connections 2023-05-22T16:05:24.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:24.096+0000 7f3e26b2a640 1 -- 172.21.15.16:0/2843164097 wait complete. 2023-05-22T16:05:24.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:24.096+0000 7f3e26b2a640 1 Processor -- start 2023-05-22T16:05:24.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:24.096+0000 7f3e26b2a640 1 -- start start 2023-05-22T16:05:24.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:24.096+0000 7f3e26b2a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3e200b5340 0x7f3e20166400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:24.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:24.096+0000 7f3e26b2a640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3e200d2480 0x7f3e20151720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:24.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:24.096+0000 7f3e26b2a640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3e2015bca0 0x7f3e20152b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:24.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:24.096+0000 7f3e26b2a640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e201537c0 con 0x7f3e200b5340 2023-05-22T16:05:24.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:24.096+0000 7f3e26b2a640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e20157180 con 0x7f3e2015bca0 2023-05-22T16:05:24.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:24.096+0000 7f3e26b2a640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e20156ea0 con 0x7f3e200d2480 2023-05-22T16:05:24.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:24.096+0000 7f3e248c6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3e200b5340 0x7f3e20166400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:24.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:24.096+0000 7f3e248c6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3e200b5340 0x7f3e20166400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41696/0 (socket says 172.21.15.16:41696) 2023-05-22T16:05:24.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:24.096+0000 7f3e248c6640 1 -- 172.21.15.16:0/2267367054 learned_addr learned my addr 172.21.15.16:0/2267367054 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:05:24.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:24.096+0000 7f3e250c7640 1 --2- 172.21.15.16:0/2267367054 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3e2015bca0 0x7f3e20152b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:24.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:24.096+0000 7f3e1ffff640 1 --2- 172.21.15.16:0/2267367054 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3e200d2480 0x7f3e20151720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:24.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:24.096+0000 7f3e248c6640 1 -- 172.21.15.16:0/2267367054 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3e200d2480 msgr2=0x7f3e20151720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:24.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:24.096+0000 7f3e248c6640 1 --2- 172.21.15.16:0/2267367054 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3e200d2480 0x7f3e20151720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:24.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:24.096+0000 7f3e248c6640 1 -- 172.21.15.16:0/2267367054 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3e2015bca0 msgr2=0x7f3e20152b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:24.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:24.096+0000 7f3e248c6640 1 --2- 172.21.15.16:0/2267367054 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3e2015bca0 0x7f3e20152b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:24.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:24.096+0000 7f3e248c6640 1 -- 172.21.15.16:0/2267367054 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e18009a70 con 0x7f3e200b5340 2023-05-22T16:05:24.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:24.096+0000 7f3e250c7640 1 --2- 172.21.15.16:0/2267367054 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3e2015bca0 0x7f3e20152b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T16:05:24.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:24.096+0000 7f3e248c6640 1 --2- 172.21.15.16:0/2267367054 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3e200b5340 0x7f3e20166400 secure :-1 s=READY pgs=2443 cs=0 l=1 rev1=1 crypto rx=0x7f3e140011f0 tx=0x7f3e1400c9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:24.103 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:24.096+0000 7f3e1d7fa640 1 -- 172.21.15.16:0/2267367054 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e140074e0 con 0x7f3e200b5340 2023-05-22T16:05:24.103 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:24.096+0000 7f3e1d7fa640 1 -- 172.21.15.16:0/2267367054 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e14007640 con 0x7f3e200b5340 2023-05-22T16:05:24.103 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:24.096+0000 7f3e26b2a640 1 -- 172.21.15.16:0/2267367054 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e20160360 con 0x7f3e200b5340 2023-05-22T16:05:24.103 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:24.096+0000 7f3e1d7fa640 1 -- 172.21.15.16:0/2267367054 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e14004a10 con 0x7f3e200b5340 2023-05-22T16:05:24.103 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:24.096+0000 7f3e26b2a640 1 -- 172.21.15.16:0/2267367054 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e200b5cd0 con 0x7f3e200b5340 2023-05-22T16:05:24.104 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:24.100+0000 7f3e26b2a640 1 -- 172.21.15.16:0/2267367054 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e201604f0 con 0x7f3e200b5340 2023-05-22T16:05:24.104 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:24.100+0000 7f3e1d7fa640 1 -- 172.21.15.16:0/2267367054 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 92) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f3e14004b70 con 0x7f3e200b5340 2023-05-22T16:05:24.105 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:24.100+0000 7f3e1d7fa640 1 --2- 172.21.15.16:0/2267367054 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f3e040654a0 0x7f3e04067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:24.105 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:24.100+0000 7f3e1d7fa640 1 -- 172.21.15.16:0/2267367054 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(196..196 src has 1..196) v4 ==== 6438+0+0 (secure 0 0 0) 0x7f3e14081da0 con 0x7f3e200b5340 2023-05-22T16:05:24.105 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:24.100+0000 7f3e1ffff640 1 --2- 172.21.15.16:0/2267367054 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f3e040654a0 0x7f3e04067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:24.105 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:24.100+0000 7f3e1ffff640 1 --2- 172.21.15.16:0/2267367054 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f3e040654a0 0x7f3e04067950 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f3e2015ecd0 tx=0x7f3e0c008040 comp rx=0 tx=0).ready entity=mgr.10542 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:24.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:24.104+0000 7f3e1d7fa640 1 -- 172.21.15.16:0/2267367054 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f3e1404f820 con 0x7f3e200b5340 2023-05-22T16:05:24.293 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:24.292+0000 7f3e26b2a640 1 -- 172.21.15.16:0/2267367054 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f3e2004dbe0 con 0x7f3e200b5340 2023-05-22T16:05:24.767 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-05-22T16:05:24.770 DEBUG:teuthology.orchestra.run.smithi062:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-05-22T16:05:24.956 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:24.952+0000 7f3e1d7fa640 1 -- 172.21.15.16:0/2267367054 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v81) v1 ==== 122+0+0 (secure 0 0 0) 0x7f3e14018020 con 0x7f3e200b5340 2023-05-22T16:05:24.959 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:24.956+0000 7f3e26b2a640 1 -- 172.21.15.16:0/2267367054 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f3e040654a0 msgr2=0x7f3e04067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:24.960 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:24.956+0000 7f3e26b2a640 1 --2- 172.21.15.16:0/2267367054 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f3e040654a0 0x7f3e04067950 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f3e2015ecd0 tx=0x7f3e0c008040 comp rx=0 tx=0).stop 2023-05-22T16:05:24.960 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:24.956+0000 7f3e26b2a640 1 -- 172.21.15.16:0/2267367054 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3e200b5340 msgr2=0x7f3e20166400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:24.960 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:24.956+0000 7f3e26b2a640 1 --2- 172.21.15.16:0/2267367054 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3e200b5340 0x7f3e20166400 secure :-1 s=READY pgs=2443 cs=0 l=1 rev1=1 crypto rx=0x7f3e140011f0 tx=0x7f3e1400c9b0 comp rx=0 tx=0).stop 2023-05-22T16:05:24.960 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:24.956+0000 7f3e26b2a640 1 -- 172.21.15.16:0/2267367054 shutdown_connections 2023-05-22T16:05:24.960 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:24.956+0000 7f3e26b2a640 1 --2- 172.21.15.16:0/2267367054 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3e2015bca0 0x7f3e20152b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:24.960 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:24.956+0000 7f3e26b2a640 1 --2- 172.21.15.16:0/2267367054 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3e200d2480 0x7f3e20151720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:24.961 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:24.956+0000 7f3e26b2a640 1 --2- 172.21.15.16:0/2267367054 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f3e040654a0 0x7f3e04067950 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:24.961 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:24.956+0000 7f3e26b2a640 1 --2- 172.21.15.16:0/2267367054 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3e200b5340 0x7f3e20166400 unknown :-1 s=CLOSED pgs=2443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:24.961 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:24.956+0000 7f3e26b2a640 1 -- 172.21.15.16:0/2267367054 >> 172.21.15.16:0/2267367054 conn(0x7f3e200655c0 msgr2=0x7f3e20158030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:05:24.961 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:24.956+0000 7f3e26b2a640 1 -- 172.21.15.16:0/2267367054 shutdown_connections 2023-05-22T16:05:24.961 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:24.956+0000 7f3e26b2a640 1 -- 172.21.15.16:0/2267367054 wait complete. 2023-05-22T16:05:24.982 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2023-05-22T16:05:24.982 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-05-22T16:05:25.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.112+0000 7fc9c2e4e640 1 Processor -- start 2023-05-22T16:05:25.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.112+0000 7fc9c2e4e640 1 -- start start 2023-05-22T16:05:25.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.112+0000 7fc9c2e4e640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc9bc0cae20 0x7fc9bc06fef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:25.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.112+0000 7fc9c2e4e640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9bc069720 con 0x7fc9bc153fe0 2023-05-22T16:05:25.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.112+0000 7fc9c2e4e640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9bc1543b0 con 0x7fc9bc070430 2023-05-22T16:05:25.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.112+0000 7fc9c2e4e640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9bc0708d0 con 0x7fc9bc0cae20 2023-05-22T16:05:25.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.112+0000 7fc9c0bea640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fc9bc153fe0 0x7fc9bc06d3f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:48486/0 (socket says 172.21.15.16:48486) 2023-05-22T16:05:25.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.112+0000 7fc9c0bea640 1 -- 172.21.15.16:0/2635759683 learned_addr learned my addr 172.21.15.16:0/2635759683 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T16:05:25.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.112+0000 7fc9bb7fe640 1 -- 172.21.15.16:0/2635759683 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 631438010 0 0) 0x7fc9bc069720 con 0x7fc9bc153fe0 2023-05-22T16:05:25.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.112+0000 7fc9bbfff640 1 --2- 172.21.15.16:0/2635759683 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc9bc0cae20 0x7fc9bc06fef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:25.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.112+0000 7fc9bb7fe640 1 -- 172.21.15.16:0/2635759683 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc9ac003530 con 0x7fc9bc153fe0 2023-05-22T16:05:25.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.112+0000 7fc9bb7fe640 1 -- 172.21.15.16:0/2635759683 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3402754392 0 0) 0x7fc9ac003530 con 0x7fc9bc153fe0 2023-05-22T16:05:25.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.112+0000 7fc9bb7fe640 1 -- 172.21.15.16:0/2635759683 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc9bc0cae20 msgr2=0x7fc9bc06fef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:05:25.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.112+0000 7fc9bb7fe640 1 --2- 172.21.15.16:0/2635759683 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc9bc0cae20 0x7fc9bc06fef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:25.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.112+0000 7fc9bb7fe640 1 -- 172.21.15.16:0/2635759683 >> v1:172.21.15.62:6789/0 conn(0x7fc9bc070430 legacy=0x7fc9bc0ad410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:25.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.112+0000 7fc9bb7fe640 1 -- 172.21.15.16:0/2635759683 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9bc15e360 con 0x7fc9bc153fe0 2023-05-22T16:05:25.118 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.112+0000 7fc9bb7fe640 1 -- 172.21.15.16:0/2635759683 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7fc9b0001680 con 0x7fc9bc153fe0 2023-05-22T16:05:25.118 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.112+0000 7fc9bb7fe640 1 -- 172.21.15.16:0/2635759683 >> v1:172.21.15.16:6789/0 conn(0x7fc9bc153fe0 legacy=0x7fc9bc06d3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:25.118 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.112+0000 7fc9bb7fe640 1 --2- 172.21.15.16:0/2635759683 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc9ac0039b0 0x7fc9ac003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:25.118 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.112+0000 7fc9bb7fe640 1 --2- 172.21.15.16:0/2635759683 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc9ac0047d0 0x7fc9ac004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:25.118 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.112+0000 7fc9bb7fe640 1 --2- 172.21.15.16:0/2635759683 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc9bc0cae20 0x7fc9ac005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:25.119 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.112+0000 7fc9bb7fe640 1 -- 172.21.15.16:0/2635759683 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9ac00ba40 con 0x7fc9ac0039b0 2023-05-22T16:05:25.119 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.112+0000 7fc9bb7fe640 1 -- 172.21.15.16:0/2635759683 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9ac00bb80 con 0x7fc9ac0047d0 2023-05-22T16:05:25.119 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.112+0000 7fc9bb7fe640 1 -- 172.21.15.16:0/2635759683 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9ac00bd10 con 0x7fc9bc0cae20 2023-05-22T16:05:25.119 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.112+0000 7fc9c0bea640 1 --2- 172.21.15.16:0/2635759683 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc9ac0039b0 0x7fc9ac003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:25.120 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.112+0000 7fc9bbfff640 1 --2- 172.21.15.16:0/2635759683 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc9ac0047d0 0x7fc9ac004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:25.120 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.112+0000 7fc9c13eb640 1 --2- 172.21.15.16:0/2635759683 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc9bc0cae20 0x7fc9ac005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:25.120 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.112+0000 7fc9c0bea640 1 -- 172.21.15.16:0/2635759683 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc9bc0cae20 msgr2=0x7fc9ac005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:05:25.120 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.112+0000 7fc9c0bea640 1 --2- 172.21.15.16:0/2635759683 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc9bc0cae20 0x7fc9ac005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:25.121 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.112+0000 7fc9c0bea640 1 -- 172.21.15.16:0/2635759683 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc9ac0047d0 msgr2=0x7fc9ac004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:05:25.121 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.112+0000 7fc9c0bea640 1 --2- 172.21.15.16:0/2635759683 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc9ac0047d0 0x7fc9ac004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:25.121 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.112+0000 7fc9c0bea640 1 -- 172.21.15.16:0/2635759683 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9ac003530 con 0x7fc9ac0039b0 2023-05-22T16:05:25.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.116+0000 7fc9c0bea640 1 --2- 172.21.15.16:0/2635759683 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc9ac0039b0 0x7fc9ac003dc0 secure :-1 s=READY pgs=2445 cs=0 l=1 rev1=1 crypto rx=0x7fc9b000c240 tx=0x7fc9b001cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d54dad5f682f6764 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:25.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.116+0000 7fc9bb7fe640 1 -- 172.21.15.16:0/2635759683 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9b0005ae0 con 0x7fc9ac0039b0 2023-05-22T16:05:25.126 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.116+0000 7fc9bb7fe640 1 -- 172.21.15.16:0/2635759683 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9b0005c60 con 0x7fc9ac0039b0 2023-05-22T16:05:25.126 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.116+0000 7fc9bb7fe640 1 -- 172.21.15.16:0/2635759683 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9b00a5830 con 0x7fc9ac0039b0 2023-05-22T16:05:25.126 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.116+0000 7fc9c2e4e640 1 -- 172.21.15.16:0/2635759683 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc9ac0039b0 msgr2=0x7fc9ac003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:25.126 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.116+0000 7fc9c2e4e640 1 --2- 172.21.15.16:0/2635759683 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc9ac0039b0 0x7fc9ac003dc0 secure :-1 s=READY pgs=2445 cs=0 l=1 rev1=1 crypto rx=0x7fc9b000c240 tx=0x7fc9b001cc40 comp rx=0 tx=0).stop 2023-05-22T16:05:25.127 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.116+0000 7fc9c13eb640 1 -- 172.21.15.16:0/2635759683 reap_dead start 2023-05-22T16:05:25.127 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.116+0000 7fc9c2e4e640 1 -- 172.21.15.16:0/2635759683 shutdown_connections 2023-05-22T16:05:25.127 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.116+0000 7fc9c2e4e640 1 -- 172.21.15.16:0/2635759683 >> 172.21.15.16:0/2635759683 conn(0x7fc9bc0b1110 msgr2=0x7fc9bc0ce830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:05:25.128 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.116+0000 7fc9c2e4e640 1 -- 172.21.15.16:0/2635759683 shutdown_connections 2023-05-22T16:05:25.128 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.116+0000 7fc9c2e4e640 1 -- 172.21.15.16:0/2635759683 wait complete. 2023-05-22T16:05:25.128 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.116+0000 7fc9c2e4e640 1 Processor -- start 2023-05-22T16:05:25.128 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.116+0000 7fc9c2e4e640 1 -- start start 2023-05-22T16:05:25.129 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.116+0000 7fc9c2e4e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc9bc146450 0x7fc9bc0b23e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:25.129 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.116+0000 7fc9c2e4e640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc9bc1468d0 0x7fc9bc15c730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:25.129 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.116+0000 7fc9c2e4e640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc9bc0ce780 0x7fc9bc0cebf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:25.129 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.116+0000 7fc9c2e4e640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9bc0cf130 con 0x7fc9bc146450 2023-05-22T16:05:25.130 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.116+0000 7fc9c2e4e640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9bc15c2a0 con 0x7fc9bc1468d0 2023-05-22T16:05:25.130 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.116+0000 7fc9c2e4e640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9bc147120 con 0x7fc9bc0ce780 2023-05-22T16:05:25.130 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.120+0000 7fc9c0bea640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc9bc146450 0x7fc9bc0b23e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:25.130 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.120+0000 7fc9c0bea640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc9bc146450 0x7fc9bc0b23e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41706/0 (socket says 172.21.15.16:41706) 2023-05-22T16:05:25.130 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.120+0000 7fc9c0bea640 1 -- 172.21.15.16:0/4226809838 learned_addr learned my addr 172.21.15.16:0/4226809838 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:05:25.130 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.120+0000 7fc9bbfff640 1 --2- 172.21.15.16:0/4226809838 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc9bc1468d0 0x7fc9bc15c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:25.131 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.120+0000 7fc9c13eb640 1 --2- 172.21.15.16:0/4226809838 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc9bc0ce780 0x7fc9bc0cebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:25.131 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.120+0000 7fc9c0bea640 1 -- 172.21.15.16:0/4226809838 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc9bc0ce780 msgr2=0x7fc9bc0cebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:25.131 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.120+0000 7fc9c0bea640 1 --2- 172.21.15.16:0/4226809838 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc9bc0ce780 0x7fc9bc0cebf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:25.131 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.120+0000 7fc9c0bea640 1 -- 172.21.15.16:0/4226809838 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc9bc1468d0 msgr2=0x7fc9bc15c730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:25.131 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.120+0000 7fc9c0bea640 1 --2- 172.21.15.16:0/4226809838 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc9bc1468d0 0x7fc9bc15c730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:25.132 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.120+0000 7fc9c0bea640 1 -- 172.21.15.16:0/4226809838 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9bc15de40 con 0x7fc9bc146450 2023-05-22T16:05:25.132 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.120+0000 7fc9c0bea640 1 --2- 172.21.15.16:0/4226809838 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc9bc146450 0x7fc9bc0b23e0 secure :-1 s=READY pgs=2446 cs=0 l=1 rev1=1 crypto rx=0x7fc9b0001940 tx=0x7fc9ac005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:25.132 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.120+0000 7fc9b97fa640 1 -- 172.21.15.16:0/4226809838 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9b00b4070 con 0x7fc9bc146450 2023-05-22T16:05:25.132 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.120+0000 7fc9b97fa640 1 -- 172.21.15.16:0/4226809838 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9b0000ee0 con 0x7fc9bc146450 2023-05-22T16:05:25.132 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.120+0000 7fc9c2e4e640 1 -- 172.21.15.16:0/4226809838 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9bc15db60 con 0x7fc9bc146450 2023-05-22T16:05:25.132 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.120+0000 7fc9b97fa640 1 -- 172.21.15.16:0/4226809838 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9b0001660 con 0x7fc9bc146450 2023-05-22T16:05:25.132 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.120+0000 7fc9c2e4e640 1 -- 172.21.15.16:0/4226809838 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9bc06dd20 con 0x7fc9bc146450 2023-05-22T16:05:25.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.120+0000 7fc9c2e4e640 1 -- 172.21.15.16:0/4226809838 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9bc04dbe0 con 0x7fc9bc146450 2023-05-22T16:05:25.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.120+0000 7fc9b97fa640 1 -- 172.21.15.16:0/4226809838 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 92) v1 ==== 84332+0+0 (secure 0 0 0) 0x7fc9b00048b0 con 0x7fc9bc146450 2023-05-22T16:05:25.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.120+0000 7fc9b97fa640 1 --2- 172.21.15.16:0/4226809838 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fc9a00654d0 0x7fc9a0067980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:25.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.120+0000 7fc9b97fa640 1 -- 172.21.15.16:0/4226809838 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(196..196 src has 1..196) v4 ==== 6438+0+0 (secure 0 0 0) 0x7fc9b011c770 con 0x7fc9bc146450 2023-05-22T16:05:25.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.120+0000 7fc9bbfff640 1 --2- 172.21.15.16:0/4226809838 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fc9a00654d0 0x7fc9a0067980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:25.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.124+0000 7fc9bbfff640 1 --2- 172.21.15.16:0/4226809838 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fc9a00654d0 0x7fc9a0067980 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fc9bc161640 tx=0x7fc9a8008040 comp rx=0 tx=0).ready entity=mgr.10542 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:25.134 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.124+0000 7fc9b97fa640 1 -- 172.21.15.16:0/4226809838 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fc9b00ea1d0 con 0x7fc9bc146450 2023-05-22T16:05:25.314 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.312+0000 7fc9c2e4e640 1 -- 172.21.15.16:0/4226809838 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc9bc14d9c0 con 0x7fc9bc146450 2023-05-22T16:05:25.315 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.312+0000 7fc9b97fa640 1 -- 172.21.15.16:0/4226809838 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 81 v81) v1 ==== 94+0+4653 (secure 0 0 0) 0x7fc9b00ed7f0 con 0x7fc9bc146450 2023-05-22T16:05:25.315 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T16:05:25.316 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":81,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7979,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/1053276484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":1053276484},{"type":"v1","addr":"172.21.15.62:6835","nonce":1053276484}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":76},{"gid":8983,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6833/1372552690","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":1372552690},{"type":"v1","addr":"172.21.15.62:6833","nonce":1372552690}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":76},{"gid":10776,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/982492476","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":982492476},{"type":"v1","addr":"172.21.15.16:6835","nonce":982492476}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":76}],"filesystems":[{"mdsmap":{"epoch":81,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-05-22T16:05:21.922031+0000","modified":"2023-05-22T16:05:24.940696+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10782},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10782":{"gid":10782,"name":"c","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.16:6837/1705367575","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":1705367575},{"type":"v1","addr":"172.21.15.16:6837","nonce":1705367575}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[25],"metadata_pool":24,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":11}]} 2023-05-22T16:05:25.316 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 81 2023-05-22T16:05:25.317 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.316+0000 7fc9c2e4e640 1 -- 172.21.15.16:0/4226809838 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fc9a00654d0 msgr2=0x7fc9a0067980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:25.317 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.316+0000 7fc9c2e4e640 1 --2- 172.21.15.16:0/4226809838 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fc9a00654d0 0x7fc9a0067980 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fc9bc161640 tx=0x7fc9a8008040 comp rx=0 tx=0).stop 2023-05-22T16:05:25.317 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.316+0000 7fc9c2e4e640 1 -- 172.21.15.16:0/4226809838 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc9bc146450 msgr2=0x7fc9bc0b23e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:25.318 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.316+0000 7fc9c2e4e640 1 --2- 172.21.15.16:0/4226809838 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc9bc146450 0x7fc9bc0b23e0 secure :-1 s=READY pgs=2446 cs=0 l=1 rev1=1 crypto rx=0x7fc9b0001940 tx=0x7fc9ac005800 comp rx=0 tx=0).stop 2023-05-22T16:05:25.318 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.316+0000 7fc9c2e4e640 1 -- 172.21.15.16:0/4226809838 shutdown_connections 2023-05-22T16:05:25.318 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.316+0000 7fc9c2e4e640 1 --2- 172.21.15.16:0/4226809838 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc9bc0ce780 0x7fc9bc0cebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:25.318 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.316+0000 7fc9c2e4e640 1 --2- 172.21.15.16:0/4226809838 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc9bc1468d0 0x7fc9bc15c730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:25.318 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.316+0000 7fc9c2e4e640 1 --2- 172.21.15.16:0/4226809838 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fc9a00654d0 0x7fc9a0067980 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:25.318 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.316+0000 7fc9c2e4e640 1 --2- 172.21.15.16:0/4226809838 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc9bc146450 0x7fc9bc0b23e0 unknown :-1 s=CLOSED pgs=2446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:25.319 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.316+0000 7fc9c2e4e640 1 -- 172.21.15.16:0/4226809838 >> 172.21.15.16:0/4226809838 conn(0x7fc9bc0b1110 msgr2=0x7fc9bc1517b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:05:25.319 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.316+0000 7fc9c2e4e640 1 -- 172.21.15.16:0/4226809838 shutdown_connections 2023-05-22T16:05:25.319 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.316+0000 7fc9c2e4e640 1 -- 172.21.15.16:0/4226809838 wait complete. 2023-05-22T16:05:25.334 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-05-22T16:05:25.460 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.456+0000 7f71c591f640 1 Processor -- start 2023-05-22T16:05:25.460 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.456+0000 7f71c591f640 1 -- start start 2023-05-22T16:05:25.461 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.456+0000 7f71c591f640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f71c00cbf80 0x7f71c006b210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:25.461 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.456+0000 7f71c591f640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f71c00b22e0 0x7f71c006bec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:25.461 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.456+0000 7f71c591f640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71c006db40 con 0x7f71c006c400 2023-05-22T16:05:25.462 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.456+0000 7f71c591f640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71c006c880 con 0x7f71c00b22e0 2023-05-22T16:05:25.462 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.456+0000 7f71c591f640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71c00cf3f0 con 0x7f71c00cbf80 2023-05-22T16:05:25.462 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.460+0000 7f71be7fc640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f71c00b22e0 0x7f71c006bec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:25.462 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.460+0000 7f71be7fc640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f71c00b22e0 0x7f71c006bec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41712/0 (socket says 172.21.15.16:41712) 2023-05-22T16:05:25.462 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.460+0000 7f71be7fc640 1 -- 172.21.15.16:0/856421858 learned_addr learned my addr 172.21.15.16:0/856421858 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:05:25.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.460+0000 7f71beffd640 1 --2- 172.21.15.16:0/856421858 >> v2:172.21.15.62:3300/0 conn(0x7f71c00cbf80 0x7f71c006b210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:25.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.460+0000 7f71be7fc640 1 -- 172.21.15.16:0/856421858 >> v2:172.21.15.62:3300/0 conn(0x7f71c00cbf80 msgr2=0x7f71c006b210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:05:25.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.460+0000 7f71be7fc640 1 --2- 172.21.15.16:0/856421858 >> v2:172.21.15.62:3300/0 conn(0x7f71c00cbf80 0x7f71c006b210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:25.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.460+0000 7f71be7fc640 1 -- 172.21.15.16:0/856421858 >> v1:172.21.15.62:6789/0 conn(0x7f71c006c400 legacy=0x7f71c015f810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:05:25.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.460+0000 7f71be7fc640 1 -- 172.21.15.16:0/856421858 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71c015c080 con 0x7f71c00b22e0 2023-05-22T16:05:25.464 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.460+0000 7f71be7fc640 1 --2- 172.21.15.16:0/856421858 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f71c00b22e0 0x7f71c006bec0 secure :-1 s=READY pgs=2447 cs=0 l=1 rev1=1 crypto rx=0x7f71c00d1300 tx=0x7f71b401c400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6fd3fa8bccb4ede server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:25.464 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.460+0000 7f71bdffb640 1 -- 172.21.15.16:0/856421858 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71b40ab600 con 0x7f71c00b22e0 2023-05-22T16:05:25.465 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.460+0000 7f71bdffb640 1 -- 172.21.15.16:0/856421858 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71b40ab760 con 0x7f71c00b22e0 2023-05-22T16:05:25.465 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.460+0000 7f71bdffb640 1 -- 172.21.15.16:0/856421858 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71b40abaf0 con 0x7f71c00b22e0 2023-05-22T16:05:25.465 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.460+0000 7f71c591f640 1 -- 172.21.15.16:0/856421858 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f71c00b22e0 msgr2=0x7f71c006bec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:25.465 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.460+0000 7f71c591f640 1 --2- 172.21.15.16:0/856421858 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f71c00b22e0 0x7f71c006bec0 secure :-1 s=READY pgs=2447 cs=0 l=1 rev1=1 crypto rx=0x7f71c00d1300 tx=0x7f71b401c400 comp rx=0 tx=0).stop 2023-05-22T16:05:25.465 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.460+0000 7f71c591f640 1 -- 172.21.15.16:0/856421858 shutdown_connections 2023-05-22T16:05:25.465 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.460+0000 7f71c591f640 1 --2- 172.21.15.16:0/856421858 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f71c00b22e0 0x7f71c006bec0 unknown :-1 s=CLOSED pgs=2447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:25.465 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.460+0000 7f71c591f640 1 --2- 172.21.15.16:0/856421858 >> v2:172.21.15.62:3300/0 conn(0x7f71c00cbf80 0x7f71c006b210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:25.466 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.460+0000 7f71c591f640 1 -- 172.21.15.16:0/856421858 >> 172.21.15.16:0/856421858 conn(0x7f71c006f560 msgr2=0x7f71c006f960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:05:25.466 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.460+0000 7f71c591f640 1 -- 172.21.15.16:0/856421858 shutdown_connections 2023-05-22T16:05:25.466 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.460+0000 7f71c591f640 1 -- 172.21.15.16:0/856421858 wait complete. 2023-05-22T16:05:25.467 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.464+0000 7f71c591f640 1 Processor -- start 2023-05-22T16:05:25.468 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.464+0000 7f71c591f640 1 -- start start 2023-05-22T16:05:25.468 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.464+0000 7f71c591f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f71c006c400 0x7f71c0151be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:25.468 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.464+0000 7f71c591f640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f71c00b22e0 0x7f71c00b0a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:25.468 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.464+0000 7f71c591f640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f71c00cbf80 0x7f71c00b0f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:25.469 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.464+0000 7f71c591f640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71c0154170 con 0x7f71c006c400 2023-05-22T16:05:25.469 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.464+0000 7f71c591f640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71c01543c0 con 0x7f71c00cbf80 2023-05-22T16:05:25.469 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.464+0000 7f71c591f640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71c01546a0 con 0x7f71c00b22e0 2023-05-22T16:05:25.469 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.464+0000 7f71beffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f71c006c400 0x7f71c0151be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:25.469 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.464+0000 7f71beffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f71c006c400 0x7f71c0151be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41716/0 (socket says 172.21.15.16:41716) 2023-05-22T16:05:25.470 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.464+0000 7f71beffd640 1 -- 172.21.15.16:0/843758732 learned_addr learned my addr 172.21.15.16:0/843758732 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:05:25.470 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.464+0000 7f71bf7fe640 1 --2- 172.21.15.16:0/843758732 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f71c00cbf80 0x7f71c00b0f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:25.470 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.464+0000 7f71be7fc640 1 --2- 172.21.15.16:0/843758732 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f71c00b22e0 0x7f71c00b0a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:25.470 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.464+0000 7f71beffd640 1 -- 172.21.15.16:0/843758732 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f71c00b22e0 msgr2=0x7f71c00b0a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:25.470 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.464+0000 7f71beffd640 1 --2- 172.21.15.16:0/843758732 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f71c00b22e0 0x7f71c00b0a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:25.470 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.464+0000 7f71beffd640 1 -- 172.21.15.16:0/843758732 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f71c00cbf80 msgr2=0x7f71c00b0f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:25.471 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.464+0000 7f71beffd640 1 --2- 172.21.15.16:0/843758732 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f71c00cbf80 0x7f71c00b0f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:25.471 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.464+0000 7f71beffd640 1 -- 172.21.15.16:0/843758732 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71b40094d0 con 0x7f71c006c400 2023-05-22T16:05:25.471 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.464+0000 7f71be7fc640 1 --2- 172.21.15.16:0/843758732 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f71c00b22e0 0x7f71c00b0a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T16:05:25.471 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.464+0000 7f71beffd640 1 --2- 172.21.15.16:0/843758732 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f71c006c400 0x7f71c0151be0 secure :-1 s=READY pgs=2448 cs=0 l=1 rev1=1 crypto rx=0x7f71b000d5f0 tx=0x7f71b000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:25.471 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.464+0000 7f71a7fff640 1 -- 172.21.15.16:0/843758732 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71b00125f0 con 0x7f71c006c400 2023-05-22T16:05:25.471 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.464+0000 7f71a7fff640 1 -- 172.21.15.16:0/843758732 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71b0012750 con 0x7f71c006c400 2023-05-22T16:05:25.472 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.464+0000 7f71a7fff640 1 -- 172.21.15.16:0/843758732 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71b0012ae0 con 0x7f71c006c400 2023-05-22T16:05:25.472 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.464+0000 7f71c591f640 1 -- 172.21.15.16:0/843758732 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71c006b750 con 0x7f71c006c400 2023-05-22T16:05:25.472 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.464+0000 7f71c591f640 1 -- 172.21.15.16:0/843758732 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71c006e380 con 0x7f71c006c400 2023-05-22T16:05:25.473 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.464+0000 7f71c591f640 1 -- 172.21.15.16:0/843758732 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71c00cc350 con 0x7f71c006c400 2023-05-22T16:05:25.475 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.464+0000 7f71a7fff640 1 -- 172.21.15.16:0/843758732 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 92) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f71b0012c40 con 0x7f71c006c400 2023-05-22T16:05:25.475 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.468+0000 7f71a7fff640 1 --2- 172.21.15.16:0/843758732 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f71ac0654a0 0x7f71ac067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:25.475 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.468+0000 7f71a7fff640 1 -- 172.21.15.16:0/843758732 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(196..196 src has 1..196) v4 ==== 6438+0+0 (secure 0 0 0) 0x7f71b0083040 con 0x7f71c006c400 2023-05-22T16:05:25.476 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.468+0000 7f71be7fc640 1 --2- 172.21.15.16:0/843758732 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f71ac0654a0 0x7f71ac067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:25.476 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.468+0000 7f71be7fc640 1 --2- 172.21.15.16:0/843758732 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f71ac0654a0 0x7f71ac067950 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f71b40af000 tx=0x7f71b4008040 comp rx=0 tx=0).ready entity=mgr.10542 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:25.476 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.472+0000 7f71a7fff640 1 -- 172.21.15.16:0/843758732 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f71b00509d0 con 0x7f71c006c400 2023-05-22T16:05:25.623 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.620+0000 7f71c591f640 1 -- 172.21.15.16:0/843758732 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f71c004dbe0 con 0x7f71c006c400 2023-05-22T16:05:25.625 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.624+0000 7f71a7fff640 1 -- 172.21.15.16:0/843758732 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v196) v1 ==== 74+0+18140 (secure 0 0 0) 0x7f71b0053ff0 con 0x7f71c006c400 2023-05-22T16:05:25.626 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T16:05:25.626 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":196,"fsid":"b2719834-7847-4d98-90a6-f65d57f85c49","created":"2023-05-22T15:26:25.259314+0000","modified":"2023-05-22T16:05:22.925970+0000","last_up_change":"2023-05-22T15:26:36.448524+0000","last_in_change":"2023-05-22T15:26:27.675668+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":25,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-05-22T15:26:38.019781+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-05-22T15:26:43.570514+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_metadata","create_time":"2023-05-22T16:05:20.485167+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"195","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":25,"pool_name":"cephfs_data","create_time":"2023-05-22T16:05:20.900001+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"196","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9e7ff854-b25f-4cbb-a5c2-1799d9c53d2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":37021},{"type":"v1","addr":"172.21.15.16:6825","nonce":37021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":37021},{"type":"v1","addr":"172.21.15.16:6827","nonce":37021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":37021},{"type":"v1","addr":"172.21.15.16:6831","nonce":37021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":37021},{"type":"v1","addr":"172.21.15.16:6829","nonce":37021}]},"public_addr":"172.21.15.16:6825/37021","cluster_addr":"172.21.15.16:6827/37021","heartbeat_back_addr":"172.21.15.16:6831/37021","heartbeat_front_addr":"172.21.15.16:6829/37021","state":["exists","up"]},{"osd":1,"uuid":"4702fbc9-a927-4565-aca5-41dc69cc8264","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":37008},{"type":"v1","addr":"172.21.15.16:6801","nonce":37008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":37008},{"type":"v1","addr":"172.21.15.16:6803","nonce":37008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":37008},{"type":"v1","addr":"172.21.15.16:6807","nonce":37008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":37008},{"type":"v1","addr":"172.21.15.16:6805","nonce":37008}]},"public_addr":"172.21.15.16:6801/37008","cluster_addr":"172.21.15.16:6803/37008","heartbeat_back_addr":"172.21.15.16:6807/37008","heartbeat_front_addr":"172.21.15.16:6805/37008","state":["exists","up"]},{"osd":2,"uuid":"1e7bba46-4eba-47ef-bb81-5e2d371638ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":37009},{"type":"v1","addr":"172.21.15.16:6809","nonce":37009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":37009},{"type":"v1","addr":"172.21.15.16:6811","nonce":37009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":37009},{"type":"v1","addr":"172.21.15.16:6815","nonce":37009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":37009},{"type":"v1","addr":"172.21.15.16:6813","nonce":37009}]},"public_addr":"172.21.15.16:6809/37009","cluster_addr":"172.21.15.16:6811/37009","heartbeat_back_addr":"172.21.15.16:6815/37009","heartbeat_front_addr":"172.21.15.16:6813/37009","state":["exists","up"]},{"osd":3,"uuid":"383cacc0-5a88-4904-990e-498ca59ef48a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":37020},{"type":"v1","addr":"172.21.15.16:6817","nonce":37020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":37020},{"type":"v1","addr":"172.21.15.16:6819","nonce":37020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":37020},{"type":"v1","addr":"172.21.15.16:6823","nonce":37020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":37020},{"type":"v1","addr":"172.21.15.16:6821","nonce":37020}]},"public_addr":"172.21.15.16:6817/37020","cluster_addr":"172.21.15.16:6819/37020","heartbeat_back_addr":"172.21.15.16:6823/37020","heartbeat_front_addr":"172.21.15.16:6821/37020","state":["exists","up"]},{"osd":4,"uuid":"970500ef-95b7-42c1-b080-eed4fae7013d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":37353},{"type":"v1","addr":"172.21.15.62:6817","nonce":37353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6819","nonce":37353},{"type":"v1","addr":"172.21.15.62:6822","nonce":37353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37353},{"type":"v1","addr":"172.21.15.62:6831","nonce":37353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37353},{"type":"v1","addr":"172.21.15.62:6826","nonce":37353}]},"public_addr":"172.21.15.62:6817/37353","cluster_addr":"172.21.15.62:6822/37353","heartbeat_back_addr":"172.21.15.62:6831/37353","heartbeat_front_addr":"172.21.15.62:6826/37353","state":["exists","up"]},{"osd":5,"uuid":"022d67db-d4d8-445b-b1b9-8b446a00f5c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":37352},{"type":"v1","addr":"172.21.15.62:6801","nonce":37352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":37352},{"type":"v1","addr":"172.21.15.62:6803","nonce":37352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":37352},{"type":"v1","addr":"172.21.15.62:6807","nonce":37352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":37352},{"type":"v1","addr":"172.21.15.62:6805","nonce":37352}]},"public_addr":"172.21.15.62:6801/37352","cluster_addr":"172.21.15.62:6803/37352","heartbeat_back_addr":"172.21.15.62:6807/37352","heartbeat_front_addr":"172.21.15.62:6805/37352","state":["exists","up"]},{"osd":6,"uuid":"42bd372b-6688-4a1b-8407-e16e8c266e33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":37354},{"type":"v1","addr":"172.21.15.62:6820","nonce":37354}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":37354},{"type":"v1","addr":"172.21.15.62:6823","nonce":37354}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6829","nonce":37354},{"type":"v1","addr":"172.21.15.62:6830","nonce":37354}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6825","nonce":37354},{"type":"v1","addr":"172.21.15.62:6827","nonce":37354}]},"public_addr":"172.21.15.62:6820/37354","cluster_addr":"172.21.15.62:6823/37354","heartbeat_back_addr":"172.21.15.62:6830/37354","heartbeat_front_addr":"172.21.15.62:6827/37354","state":["exists","up"]},{"osd":7,"uuid":"ac70b2a3-f41b-4bf2-b227-7f2cbb55426d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37355},{"type":"v1","addr":"172.21.15.62:6809","nonce":37355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37355},{"type":"v1","addr":"172.21.15.62:6811","nonce":37355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":37355},{"type":"v1","addr":"172.21.15.62:6815","nonce":37355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":37355},{"type":"v1","addr":"172.21.15.62:6813","nonce":37355}]},"public_addr":"172.21.15.62:6809/37355","cluster_addr":"172.21.15.62:6811/37355","heartbeat_back_addr":"172.21.15.62:6815/37355","heartbeat_front_addr":"172.21.15.62:6813/37355","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.368896+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.107273+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.276888+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.287502+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.124791+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.251937+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.255679+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.232797+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":17,"snaps":[{"begin":2,"length":2}]},{"pool":21,"snaps":[{"begin":25,"length":9}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-05-22T16:05:25.628 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.624+0000 7f71c591f640 1 -- 172.21.15.16:0/843758732 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f71ac0654a0 msgr2=0x7f71ac067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:25.628 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.624+0000 7f71c591f640 1 --2- 172.21.15.16:0/843758732 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f71ac0654a0 0x7f71ac067950 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f71b40af000 tx=0x7f71b4008040 comp rx=0 tx=0).stop 2023-05-22T16:05:25.628 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.624+0000 7f71c591f640 1 -- 172.21.15.16:0/843758732 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f71c006c400 msgr2=0x7f71c0151be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:25.628 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.624+0000 7f71c591f640 1 --2- 172.21.15.16:0/843758732 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f71c006c400 0x7f71c0151be0 secure :-1 s=READY pgs=2448 cs=0 l=1 rev1=1 crypto rx=0x7f71b000d5f0 tx=0x7f71b000dab0 comp rx=0 tx=0).stop 2023-05-22T16:05:25.629 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.624+0000 7f71c591f640 1 -- 172.21.15.16:0/843758732 shutdown_connections 2023-05-22T16:05:25.629 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.624+0000 7f71c591f640 1 --2- 172.21.15.16:0/843758732 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f71c00cbf80 0x7f71c00b0f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:25.629 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.624+0000 7f71c591f640 1 --2- 172.21.15.16:0/843758732 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f71c00b22e0 0x7f71c00b0a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:25.629 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.624+0000 7f71c591f640 1 --2- 172.21.15.16:0/843758732 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f71ac0654a0 0x7f71ac067950 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:25.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.624+0000 7f71c591f640 1 --2- 172.21.15.16:0/843758732 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f71c006c400 0x7f71c0151be0 unknown :-1 s=CLOSED pgs=2448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:25.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.624+0000 7f71c591f640 1 -- 172.21.15.16:0/843758732 >> 172.21.15.16:0/843758732 conn(0x7f71c006f560 msgr2=0x7f71c006ba90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:05:25.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.624+0000 7f71c591f640 1 -- 172.21.15.16:0/843758732 shutdown_connections 2023-05-22T16:05:25.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.624+0000 7f71c591f640 1 -- 172.21.15.16:0/843758732 wait complete. 2023-05-22T16:05:25.644 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-05-22T16:05:25.645 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-05-22T16:05:25.765 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.764+0000 7fbbb0237640 1 Processor -- start 2023-05-22T16:05:25.765 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.764+0000 7fbbb0237640 1 -- start start 2023-05-22T16:05:25.766 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.764+0000 7fbbb0237640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbba80b24a0 0x7fbba806dd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:25.766 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.764+0000 7fbbb0237640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fbba806e270 0x7fbba80cb1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:25.766 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.764+0000 7fbbb0237640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbba806bfd0 con 0x7fbba815c130 2023-05-22T16:05:25.766 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.764+0000 7fbbb0237640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbba80d1170 con 0x7fbba806e270 2023-05-22T16:05:25.767 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.764+0000 7fbbb0237640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbba806e6f0 con 0x7fbba80b24a0 2023-05-22T16:05:25.767 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.764+0000 7fbbad7d2640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbba80b24a0 0x7fbba806dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:25.767 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.764+0000 7fbbae7d4640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fbba806e270 0x7fbba80cb1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:25.768 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.764+0000 7fbbadfd3640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7fbba815c130 0x7fbba80ceb70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.16:57644/0 (socket says 172.21.15.16:57644) 2023-05-22T16:05:25.768 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.764+0000 7fbbadfd3640 1 -- 172.21.15.16:0/1327224234 learned_addr learned my addr 172.21.15.16:0/1327224234 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T16:05:25.768 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.764+0000 7fbbad7d2640 1 -- 172.21.15.16:0/1327224234 >> v2:172.21.15.62:3300/0 conn(0x7fbba806e270 msgr2=0x7fbba80cb1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:05:25.768 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.764+0000 7fbbad7d2640 1 --2- 172.21.15.16:0/1327224234 >> v2:172.21.15.62:3300/0 conn(0x7fbba806e270 0x7fbba80cb1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:25.768 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.764+0000 7fbbad7d2640 1 -- 172.21.15.16:0/1327224234 >> v1:172.21.15.62:6789/0 conn(0x7fbba815c130 legacy=0x7fbba80ceb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:25.769 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.764+0000 7fbbad7d2640 1 -- 172.21.15.16:0/1327224234 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbba806c830 con 0x7fbba80b24a0 2023-05-22T16:05:25.769 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.768+0000 7fbbad7d2640 1 --2- 172.21.15.16:0/1327224234 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbba80b24a0 0x7fbba806dd30 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fbba40097c0 tx=0x7fbba401c3f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5ac5d3ecf8a2ed37 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:25.771 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.768+0000 7fbbacfd1640 1 -- 172.21.15.16:0/1327224234 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbba40ab600 con 0x7fbba80b24a0 2023-05-22T16:05:25.771 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.768+0000 7fbbacfd1640 1 -- 172.21.15.16:0/1327224234 <== mon.2 v2:172.21.15.62:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbba40a7050 con 0x7fbba80b24a0 2023-05-22T16:05:25.772 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.768+0000 7fbbacfd1640 1 -- 172.21.15.16:0/1327224234 <== mon.2 v2:172.21.15.62:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbba40ab980 con 0x7fbba80b24a0 2023-05-22T16:05:25.772 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.768+0000 7fbbb0237640 1 -- 172.21.15.16:0/1327224234 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbba80b24a0 msgr2=0x7fbba806dd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:25.772 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.768+0000 7fbbb0237640 1 --2- 172.21.15.16:0/1327224234 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbba80b24a0 0x7fbba806dd30 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fbba40097c0 tx=0x7fbba401c3f0 comp rx=0 tx=0).stop 2023-05-22T16:05:25.773 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.768+0000 7fbbb0237640 1 -- 172.21.15.16:0/1327224234 shutdown_connections 2023-05-22T16:05:25.773 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.768+0000 7fbbb0237640 1 --2- 172.21.15.16:0/1327224234 >> v2:172.21.15.62:3300/0 conn(0x7fbba806e270 0x7fbba80cb1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:25.773 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.768+0000 7fbbb0237640 1 --2- 172.21.15.16:0/1327224234 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbba80b24a0 0x7fbba806dd30 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:25.774 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.768+0000 7fbbb0237640 1 -- 172.21.15.16:0/1327224234 >> 172.21.15.16:0/1327224234 conn(0x7fbba806f5c0 msgr2=0x7fbba806f9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:05:25.774 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.768+0000 7fbbb0237640 1 -- 172.21.15.16:0/1327224234 shutdown_connections 2023-05-22T16:05:25.774 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.768+0000 7fbbb0237640 1 -- 172.21.15.16:0/1327224234 wait complete. 2023-05-22T16:05:25.777 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.768+0000 7fbbb0237640 1 Processor -- start 2023-05-22T16:05:25.778 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.768+0000 7fbbb0237640 1 -- start start 2023-05-22T16:05:25.778 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.772+0000 7fbbb0237640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbba806e270 0x7fbba81530a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:25.778 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.772+0000 7fbbb0237640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbba80b24a0 0x7fbba80c6020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:25.778 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.772+0000 7fbbb0237640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbba815c130 0x7fbba80c5420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:25.779 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.772+0000 7fbbb0237640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbba80c9370 con 0x7fbba80b24a0 2023-05-22T16:05:25.779 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.772+0000 7fbbb0237640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbba80c6560 con 0x7fbba806e270 2023-05-22T16:05:25.779 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.772+0000 7fbbb0237640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbba80c8970 con 0x7fbba815c130 2023-05-22T16:05:25.779 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.772+0000 7fbbad7d2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbba80b24a0 0x7fbba80c6020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:25.779 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.772+0000 7fbbad7d2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbba80b24a0 0x7fbba80c6020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41720/0 (socket says 172.21.15.16:41720) 2023-05-22T16:05:25.779 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.772+0000 7fbbad7d2640 1 -- 172.21.15.16:0/89969264 learned_addr learned my addr 172.21.15.16:0/89969264 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:05:25.780 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.772+0000 7fbbadfd3640 1 --2- 172.21.15.16:0/89969264 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbba806e270 0x7fbba81530a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:25.780 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.772+0000 7fbbae7d4640 1 --2- 172.21.15.16:0/89969264 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbba815c130 0x7fbba80c5420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:25.780 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.772+0000 7fbbad7d2640 1 -- 172.21.15.16:0/89969264 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbba815c130 msgr2=0x7fbba80c5420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:25.780 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.772+0000 7fbbad7d2640 1 --2- 172.21.15.16:0/89969264 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbba815c130 0x7fbba80c5420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:25.780 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.772+0000 7fbbad7d2640 1 -- 172.21.15.16:0/89969264 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbba806e270 msgr2=0x7fbba81530a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:25.780 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.772+0000 7fbbad7d2640 1 --2- 172.21.15.16:0/89969264 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbba806e270 0x7fbba81530a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:25.781 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.772+0000 7fbbad7d2640 1 -- 172.21.15.16:0/89969264 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbba4009420 con 0x7fbba80b24a0 2023-05-22T16:05:25.781 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.772+0000 7fbbae7d4640 1 --2- 172.21.15.16:0/89969264 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbba815c130 0x7fbba80c5420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T16:05:25.781 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.772+0000 7fbbad7d2640 1 --2- 172.21.15.16:0/89969264 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbba80b24a0 0x7fbba80c6020 secure :-1 s=READY pgs=2449 cs=0 l=1 rev1=1 crypto rx=0x7fbba40ae040 tx=0x7fbba4019190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:25.781 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.772+0000 7fbb9e7fc640 1 -- 172.21.15.16:0/89969264 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbba40b4840 con 0x7fbba80b24a0 2023-05-22T16:05:25.781 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.772+0000 7fbb9e7fc640 1 -- 172.21.15.16:0/89969264 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbba40b49c0 con 0x7fbba80b24a0 2023-05-22T16:05:25.781 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.772+0000 7fbbb0237640 1 -- 172.21.15.16:0/89969264 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbba80ca9e0 con 0x7fbba80b24a0 2023-05-22T16:05:25.782 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.772+0000 7fbb9e7fc640 1 -- 172.21.15.16:0/89969264 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbba40b4d70 con 0x7fbba80b24a0 2023-05-22T16:05:25.782 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.772+0000 7fbbb0237640 1 -- 172.21.15.16:0/89969264 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbba806c550 con 0x7fbba80b24a0 2023-05-22T16:05:25.782 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.772+0000 7fbbb0237640 1 -- 172.21.15.16:0/89969264 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbba80cacc0 con 0x7fbba80b24a0 2023-05-22T16:05:25.782 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.772+0000 7fbb9e7fc640 1 -- 172.21.15.16:0/89969264 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 92) v1 ==== 84332+0+0 (secure 0 0 0) 0x7fbba40abd50 con 0x7fbba80b24a0 2023-05-22T16:05:25.782 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.772+0000 7fbb9e7fc640 1 --2- 172.21.15.16:0/89969264 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fbb8c0654a0 0x7fbb8c067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:25.782 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.772+0000 7fbb9e7fc640 1 -- 172.21.15.16:0/89969264 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(196..196 src has 1..196) v4 ==== 6438+0+0 (secure 0 0 0) 0x7fbba411d280 con 0x7fbba80b24a0 2023-05-22T16:05:25.783 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.772+0000 7fbbadfd3640 1 --2- 172.21.15.16:0/89969264 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fbb8c0654a0 0x7fbb8c067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:25.783 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.772+0000 7fbbadfd3640 1 --2- 172.21.15.16:0/89969264 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fbb8c0654a0 0x7fbb8c067950 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fbb98005890 tx=0x7fbb98009040 comp rx=0 tx=0).ready entity=mgr.10542 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:25.784 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.776+0000 7fbb9e7fc640 1 -- 172.21.15.16:0/89969264 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fbba40eacd0 con 0x7fbba80b24a0 2023-05-22T16:05:25.928 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.928+0000 7fbbb0237640 1 -- 172.21.15.16:0/89969264 --> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fbba804dbe0 con 0x7fbb8c0654a0 2023-05-22T16:05:25.945 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.944+0000 7fbb9e7fc640 1 -- 172.21.15.16:0/89969264 <== mgr.10542 v2:172.21.15.16:6832/65890 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+309683 (secure 0 0 0) 0x7fbba804dbe0 con 0x7fbb8c0654a0 2023-05-22T16:05:25.947 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T16:05:25.949 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2023-05-22T16:05:25.950 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.948+0000 7fbbb0237640 1 -- 172.21.15.16:0/89969264 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fbb8c0654a0 msgr2=0x7fbb8c067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:25.950 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.948+0000 7fbbb0237640 1 --2- 172.21.15.16:0/89969264 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fbb8c0654a0 0x7fbb8c067950 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fbb98005890 tx=0x7fbb98009040 comp rx=0 tx=0).stop 2023-05-22T16:05:25.950 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.948+0000 7fbbb0237640 1 -- 172.21.15.16:0/89969264 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbba80b24a0 msgr2=0x7fbba80c6020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:25.951 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.948+0000 7fbbb0237640 1 --2- 172.21.15.16:0/89969264 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbba80b24a0 0x7fbba80c6020 secure :-1 s=READY pgs=2449 cs=0 l=1 rev1=1 crypto rx=0x7fbba40ae040 tx=0x7fbba4019190 comp rx=0 tx=0).stop 2023-05-22T16:05:25.951 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.948+0000 7fbbb0237640 1 -- 172.21.15.16:0/89969264 shutdown_connections 2023-05-22T16:05:25.951 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.948+0000 7fbbb0237640 1 --2- 172.21.15.16:0/89969264 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbba815c130 0x7fbba80c5420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:25.951 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.948+0000 7fbbb0237640 1 --2- 172.21.15.16:0/89969264 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fbb8c0654a0 0x7fbb8c067950 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:25.951 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.948+0000 7fbbb0237640 1 --2- 172.21.15.16:0/89969264 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbba80b24a0 0x7fbba80c6020 unknown :-1 s=CLOSED pgs=2449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:25.951 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.948+0000 7fbbb0237640 1 --2- 172.21.15.16:0/89969264 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbba806e270 0x7fbba81530a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:25.952 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.948+0000 7fbbb0237640 1 -- 172.21.15.16:0/89969264 >> 172.21.15.16:0/89969264 conn(0x7fbba806f5c0 msgr2=0x7fbba806f990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:05:25.952 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.948+0000 7fbbb0237640 1 -- 172.21.15.16:0/89969264 shutdown_connections 2023-05-22T16:05:25.952 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:25.948+0000 7fbbb0237640 1 -- 172.21.15.16:0/89969264 wait complete. 2023-05-22T16:05:25.968 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":25,"stamp":"2023-05-22T16:05:24.821335+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592698,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1940,"num_read_kb":4014,"num_write":583,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":364,"ondisk_log_size":364,"up":210,"acting":210,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":111,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":247860,"kb_used_data":20276,"kb_used_omap":0,"kb_used_meta":227520,"kb_avail":754726860,"statfs":{"total":773094113280,"available":772840304640,"internally_reserved":0,"allocated":20762624,"data_stored":11630668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":232980480},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":75,"apply_latency_ms":75,"commit_latency_ns":75000000,"apply_latency_ns":75000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2330,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.301670"},"pg_stats":[{"pgid":"25.3f","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2023-05-22T16:05:22.004814+0000","last_change":"2023-05-22T16:05:21.996989+0000","last_active":"2023-05-22T16:05:22.004814+0000","last_peered":"2023-05-22T16:05:22.004814+0000","last_clean":"2023-05-22T16:05:22.004814+0000","last_became_active":"2023-05-22T16:05:21.996486+0000","last_became_peered":"2023-05-22T16:05:21.996486+0000","last_unstale":"2023-05-22T16:05:22.004814+0000","last_undegraded":"2023-05-22T16:05:22.004814+0000","last_fullsized":"2023-05-22T16:05:22.004814+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_clean_scrub_stamp":"2023-05-22T16:05:21.519005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:17:20.015836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.3e","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2023-05-22T16:05:21.983740+0000","last_change":"2023-05-22T16:05:21.959893+0000","last_active":"2023-05-22T16:05:21.983740+0000","last_peered":"2023-05-22T16:05:21.983740+0000","last_clean":"2023-05-22T16:05:21.983740+0000","last_became_active":"2023-05-22T16:05:21.959111+0000","last_became_peered":"2023-05-22T16:05:21.959111+0000","last_unstale":"2023-05-22T16:05:21.983740+0000","last_undegraded":"2023-05-22T16:05:21.983740+0000","last_fullsized":"2023-05-22T16:05:21.983740+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_clean_scrub_stamp":"2023-05-22T16:05:21.519005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:10:02.644633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.3d","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2023-05-22T16:05:21.977760+0000","last_change":"2023-05-22T16:05:21.976957+0000","last_active":"2023-05-22T16:05:21.977760+0000","last_peered":"2023-05-22T16:05:21.977760+0000","last_clean":"2023-05-22T16:05:21.977760+0000","last_became_active":"2023-05-22T16:05:21.976584+0000","last_became_peered":"2023-05-22T16:05:21.976584+0000","last_unstale":"2023-05-22T16:05:21.977760+0000","last_undegraded":"2023-05-22T16:05:21.977760+0000","last_fullsized":"2023-05-22T16:05:21.977760+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_clean_scrub_stamp":"2023-05-22T16:05:21.519005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:22:21.749692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.3c","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2023-05-22T16:05:21.986529+0000","last_change":"2023-05-22T16:05:21.959819+0000","last_active":"2023-05-22T16:05:21.986529+0000","last_peered":"2023-05-22T16:05:21.986529+0000","last_clean":"2023-05-22T16:05:21.986529+0000","last_became_active":"2023-05-22T16:05:21.958880+0000","last_became_peered":"2023-05-22T16:05:21.958880+0000","last_unstale":"2023-05-22T16:05:21.986529+0000","last_undegraded":"2023-05-22T16:05:21.986529+0000","last_fullsized":"2023-05-22T16:05:21.986529+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_clean_scrub_stamp":"2023-05-22T16:05:21.519005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:18:24.385500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.3b","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2023-05-22T16:05:21.970062+0000","last_change":"2023-05-22T16:05:21.969331+0000","last_active":"2023-05-22T16:05:21.970062+0000","last_peered":"2023-05-22T16:05:21.970062+0000","last_clean":"2023-05-22T16:05:21.970062+0000","last_became_active":"2023-05-22T16:05:21.969017+0000","last_became_peered":"2023-05-22T16:05:21.969017+0000","last_unstale":"2023-05-22T16:05:21.970062+0000","last_undegraded":"2023-05-22T16:05:21.970062+0000","last_fullsized":"2023-05-22T16:05:21.970062+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_clean_scrub_stamp":"2023-05-22T16:05:21.519005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T16:16:17.806836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.3a","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2023-05-22T16:05:21.983844+0000","last_change":"2023-05-22T16:05:21.982911+0000","last_active":"2023-05-22T16:05:21.983844+0000","last_peered":"2023-05-22T16:05:21.983844+0000","last_clean":"2023-05-22T16:05:21.983844+0000","last_became_active":"2023-05-22T16:05:21.982587+0000","last_became_peered":"2023-05-22T16:05:21.982587+0000","last_unstale":"2023-05-22T16:05:21.983844+0000","last_undegraded":"2023-05-22T16:05:21.983844+0000","last_fullsized":"2023-05-22T16:05:21.983844+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_clean_scrub_stamp":"2023-05-22T16:05:21.519005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:16:57.793243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.39","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2023-05-22T16:05:21.992244+0000","last_change":"2023-05-22T16:05:21.985013+0000","last_active":"2023-05-22T16:05:21.992244+0000","last_peered":"2023-05-22T16:05:21.992244+0000","last_clean":"2023-05-22T16:05:21.992244+0000","last_became_active":"2023-05-22T16:05:21.984286+0000","last_became_peered":"2023-05-22T16:05:21.984286+0000","last_unstale":"2023-05-22T16:05:21.992244+0000","last_undegraded":"2023-05-22T16:05:21.992244+0000","last_fullsized":"2023-05-22T16:05:21.992244+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_clean_scrub_stamp":"2023-05-22T16:05:21.519005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T18:38:48.345515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.38","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2023-05-22T16:05:21.991226+0000","last_change":"2023-05-22T16:05:21.990505+0000","last_active":"2023-05-22T16:05:21.991226+0000","last_peered":"2023-05-22T16:05:21.991226+0000","last_clean":"2023-05-22T16:05:21.991226+0000","last_became_active":"2023-05-22T16:05:21.990152+0000","last_became_peered":"2023-05-22T16:05:21.990152+0000","last_unstale":"2023-05-22T16:05:21.991226+0000","last_undegraded":"2023-05-22T16:05:21.991226+0000","last_fullsized":"2023-05-22T16:05:21.991226+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_clean_scrub_stamp":"2023-05-22T16:05:21.519005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:41:43.814799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.37","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2023-05-22T16:05:21.992939+0000","last_change":"2023-05-22T16:05:21.986578+0000","last_active":"2023-05-22T16:05:21.992939+0000","last_peered":"2023-05-22T16:05:21.992939+0000","last_clean":"2023-05-22T16:05:21.992939+0000","last_became_active":"2023-05-22T16:05:21.985854+0000","last_became_peered":"2023-05-22T16:05:21.985854+0000","last_unstale":"2023-05-22T16:05:21.992939+0000","last_undegraded":"2023-05-22T16:05:21.992939+0000","last_fullsized":"2023-05-22T16:05:21.992939+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_clean_scrub_stamp":"2023-05-22T16:05:21.519005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:49:39.424358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.36","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2023-05-22T16:05:21.952784+0000","last_change":"2023-05-22T16:05:21.950885+0000","last_active":"2023-05-22T16:05:21.952784+0000","last_peered":"2023-05-22T16:05:21.952784+0000","last_clean":"2023-05-22T16:05:21.952784+0000","last_became_active":"2023-05-22T16:05:21.950609+0000","last_became_peered":"2023-05-22T16:05:21.950609+0000","last_unstale":"2023-05-22T16:05:21.952784+0000","last_undegraded":"2023-05-22T16:05:21.952784+0000","last_fullsized":"2023-05-22T16:05:21.952784+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_clean_scrub_stamp":"2023-05-22T16:05:21.519005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T16:29:01.531219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.35","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2023-05-22T16:05:22.000575+0000","last_change":"2023-05-22T16:05:21.997776+0000","last_active":"2023-05-22T16:05:22.000575+0000","last_peered":"2023-05-22T16:05:22.000575+0000","last_clean":"2023-05-22T16:05:22.000575+0000","last_became_active":"2023-05-22T16:05:21.997230+0000","last_became_peered":"2023-05-22T16:05:21.997230+0000","last_unstale":"2023-05-22T16:05:22.000575+0000","last_undegraded":"2023-05-22T16:05:22.000575+0000","last_fullsized":"2023-05-22T16:05:22.000575+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_clean_scrub_stamp":"2023-05-22T16:05:21.519005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:54:30.440182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.34","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2023-05-22T16:05:22.002068+0000","last_change":"2023-05-22T16:05:21.997195+0000","last_active":"2023-05-22T16:05:22.002068+0000","last_peered":"2023-05-22T16:05:22.002068+0000","last_clean":"2023-05-22T16:05:22.002068+0000","last_became_active":"2023-05-22T16:05:21.996648+0000","last_became_peered":"2023-05-22T16:05:21.996648+0000","last_unstale":"2023-05-22T16:05:22.002068+0000","last_undegraded":"2023-05-22T16:05:22.002068+0000","last_fullsized":"2023-05-22T16:05:22.002068+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_clean_scrub_stamp":"2023-05-22T16:05:21.519005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:02:05.951854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.33","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2023-05-22T16:05:21.983828+0000","last_change":"2023-05-22T16:05:21.964774+0000","last_active":"2023-05-22T16:05:21.983828+0000","last_peered":"2023-05-22T16:05:21.983828+0000","last_clean":"2023-05-22T16:05:21.983828+0000","last_became_active":"2023-05-22T16:05:21.963941+0000","last_became_peered":"2023-05-22T16:05:21.963941+0000","last_unstale":"2023-05-22T16:05:21.983828+0000","last_undegraded":"2023-05-22T16:05:21.983828+0000","last_fullsized":"2023-05-22T16:05:21.983828+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_clean_scrub_stamp":"2023-05-22T16:05:21.519005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:52:00.554956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.32","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2023-05-22T16:05:21.987271+0000","last_change":"2023-05-22T16:05:21.967310+0000","last_active":"2023-05-22T16:05:21.987271+0000","last_peered":"2023-05-22T16:05:21.987271+0000","last_clean":"2023-05-22T16:05:21.987271+0000","last_became_active":"2023-05-22T16:05:21.966561+0000","last_became_peered":"2023-05-22T16:05:21.966561+0000","last_unstale":"2023-05-22T16:05:21.987271+0000","last_undegraded":"2023-05-22T16:05:21.987271+0000","last_fullsized":"2023-05-22T16:05:21.987271+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_clean_scrub_stamp":"2023-05-22T16:05:21.519005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T16:22:43.843829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.31","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2023-05-22T16:05:21.994726+0000","last_change":"2023-05-22T16:05:21.982875+0000","last_active":"2023-05-22T16:05:21.994726+0000","last_peered":"2023-05-22T16:05:21.994726+0000","last_clean":"2023-05-22T16:05:21.994726+0000","last_became_active":"2023-05-22T16:05:21.982163+0000","last_became_peered":"2023-05-22T16:05:21.982163+0000","last_unstale":"2023-05-22T16:05:21.994726+0000","last_undegraded":"2023-05-22T16:05:21.994726+0000","last_fullsized":"2023-05-22T16:05:21.994726+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_clean_scrub_stamp":"2023-05-22T16:05:21.519005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:16:57.793243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.30","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2023-05-22T16:05:21.972422+0000","last_change":"2023-05-22T16:05:21.971263+0000","last_active":"2023-05-22T16:05:21.972422+0000","last_peered":"2023-05-22T16:05:21.972422+0000","last_clean":"2023-05-22T16:05:21.972422+0000","last_became_active":"2023-05-22T16:05:21.970948+0000","last_became_peered":"2023-05-22T16:05:21.970948+0000","last_unstale":"2023-05-22T16:05:21.972422+0000","last_undegraded":"2023-05-22T16:05:21.972422+0000","last_fullsized":"2023-05-22T16:05:21.972422+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_clean_scrub_stamp":"2023-05-22T16:05:21.519005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T18:49:41.457080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.2f","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2023-05-22T16:05:21.983669+0000","last_change":"2023-05-22T16:05:21.982607+0000","last_active":"2023-05-22T16:05:21.983669+0000","last_peered":"2023-05-22T16:05:21.983669+0000","last_clean":"2023-05-22T16:05:21.983669+0000","last_became_active":"2023-05-22T16:05:21.982207+0000","last_became_peered":"2023-05-22T16:05:21.982207+0000","last_unstale":"2023-05-22T16:05:21.983669+0000","last_undegraded":"2023-05-22T16:05:21.983669+0000","last_fullsized":"2023-05-22T16:05:21.983669+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_clean_scrub_stamp":"2023-05-22T16:05:21.519005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:22:44.849706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.2e","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2023-05-22T16:05:22.004866+0000","last_change":"2023-05-22T16:05:21.997190+0000","last_active":"2023-05-22T16:05:22.004866+0000","last_peered":"2023-05-22T16:05:22.004866+0000","last_clean":"2023-05-22T16:05:22.004866+0000","last_became_active":"2023-05-22T16:05:21.996472+0000","last_became_peered":"2023-05-22T16:05:21.996472+0000","last_unstale":"2023-05-22T16:05:22.004866+0000","last_undegraded":"2023-05-22T16:05:22.004866+0000","last_fullsized":"2023-05-22T16:05:22.004866+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_clean_scrub_stamp":"2023-05-22T16:05:21.519005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:58:14.389065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.2d","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2023-05-22T16:05:21.977785+0000","last_change":"2023-05-22T16:05:21.976684+0000","last_active":"2023-05-22T16:05:21.977785+0000","last_peered":"2023-05-22T16:05:21.977785+0000","last_clean":"2023-05-22T16:05:21.977785+0000","last_became_active":"2023-05-22T16:05:21.976411+0000","last_became_peered":"2023-05-22T16:05:21.976411+0000","last_unstale":"2023-05-22T16:05:21.977785+0000","last_undegraded":"2023-05-22T16:05:21.977785+0000","last_fullsized":"2023-05-22T16:05:21.977785+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_clean_scrub_stamp":"2023-05-22T16:05:21.519005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:40:32.750594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.2c","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2023-05-22T16:05:21.990873+0000","last_change":"2023-05-22T16:05:21.990043+0000","last_active":"2023-05-22T16:05:21.990873+0000","last_peered":"2023-05-22T16:05:21.990873+0000","last_clean":"2023-05-22T16:05:21.990873+0000","last_became_active":"2023-05-22T16:05:21.989695+0000","last_became_peered":"2023-05-22T16:05:21.989695+0000","last_unstale":"2023-05-22T16:05:21.990873+0000","last_undegraded":"2023-05-22T16:05:21.990873+0000","last_fullsized":"2023-05-22T16:05:21.990873+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_clean_scrub_stamp":"2023-05-22T16:05:21.519005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T16:24:50.702827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.2b","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2023-05-22T16:05:21.970041+0000","last_change":"2023-05-22T16:05:21.969335+0000","last_active":"2023-05-22T16:05:21.970041+0000","last_peered":"2023-05-22T16:05:21.970041+0000","last_clean":"2023-05-22T16:05:21.970041+0000","last_became_active":"2023-05-22T16:05:21.969041+0000","last_became_peered":"2023-05-22T16:05:21.969041+0000","last_unstale":"2023-05-22T16:05:21.970041+0000","last_undegraded":"2023-05-22T16:05:21.970041+0000","last_fullsized":"2023-05-22T16:05:21.970041+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_clean_scrub_stamp":"2023-05-22T16:05:21.519005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:48:52.945903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.2a","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2023-05-22T16:05:21.952486+0000","last_change":"2023-05-22T16:05:21.950635+0000","last_active":"2023-05-22T16:05:21.952486+0000","last_peered":"2023-05-22T16:05:21.952486+0000","last_clean":"2023-05-22T16:05:21.952486+0000","last_became_active":"2023-05-22T16:05:21.950295+0000","last_became_peered":"2023-05-22T16:05:21.950295+0000","last_unstale":"2023-05-22T16:05:21.952486+0000","last_undegraded":"2023-05-22T16:05:21.952486+0000","last_fullsized":"2023-05-22T16:05:21.952486+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_clean_scrub_stamp":"2023-05-22T16:05:21.519005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:31:49.575445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.29","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2023-05-22T16:05:21.954883+0000","last_change":"2023-05-22T16:05:21.952067+0000","last_active":"2023-05-22T16:05:21.954883+0000","last_peered":"2023-05-22T16:05:21.954883+0000","last_clean":"2023-05-22T16:05:21.954883+0000","last_became_active":"2023-05-22T16:05:21.951583+0000","last_became_peered":"2023-05-22T16:05:21.951583+0000","last_unstale":"2023-05-22T16:05:21.954883+0000","last_undegraded":"2023-05-22T16:05:21.954883+0000","last_fullsized":"2023-05-22T16:05:21.954883+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_clean_scrub_stamp":"2023-05-22T16:05:21.519005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:59:39.304277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.28","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2023-05-22T16:05:21.955519+0000","last_change":"2023-05-22T16:05:21.954365+0000","last_active":"2023-05-22T16:05:21.955519+0000","last_peered":"2023-05-22T16:05:21.955519+0000","last_clean":"2023-05-22T16:05:21.955519+0000","last_became_active":"2023-05-22T16:05:21.953868+0000","last_became_peered":"2023-05-22T16:05:21.953868+0000","last_unstale":"2023-05-22T16:05:21.955519+0000","last_undegraded":"2023-05-22T16:05:21.955519+0000","last_fullsized":"2023-05-22T16:05:21.955519+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_clean_scrub_stamp":"2023-05-22T16:05:21.519005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:58:14.389065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.27","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2023-05-22T16:05:21.983787+0000","last_change":"2023-05-22T16:05:21.960815+0000","last_active":"2023-05-22T16:05:21.983787+0000","last_peered":"2023-05-22T16:05:21.983787+0000","last_clean":"2023-05-22T16:05:21.983787+0000","last_became_active":"2023-05-22T16:05:21.959608+0000","last_became_peered":"2023-05-22T16:05:21.959608+0000","last_unstale":"2023-05-22T16:05:21.983787+0000","last_undegraded":"2023-05-22T16:05:21.983787+0000","last_fullsized":"2023-05-22T16:05:21.983787+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_clean_scrub_stamp":"2023-05-22T16:05:21.519005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T16:17:02.634957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.26","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2023-05-22T16:05:21.987082+0000","last_change":"2023-05-22T16:05:21.963335+0000","last_active":"2023-05-22T16:05:21.987082+0000","last_peered":"2023-05-22T16:05:21.987082+0000","last_clean":"2023-05-22T16:05:21.987082+0000","last_became_active":"2023-05-22T16:05:21.959751+0000","last_became_peered":"2023-05-22T16:05:21.959751+0000","last_unstale":"2023-05-22T16:05:21.987082+0000","last_undegraded":"2023-05-22T16:05:21.987082+0000","last_fullsized":"2023-05-22T16:05:21.987082+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_clean_scrub_stamp":"2023-05-22T16:05:21.519005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:05:02.241625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.25","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2023-05-22T16:05:22.000126+0000","last_change":"2023-05-22T16:05:21.999427+0000","last_active":"2023-05-22T16:05:22.000126+0000","last_peered":"2023-05-22T16:05:22.000126+0000","last_clean":"2023-05-22T16:05:22.000126+0000","last_became_active":"2023-05-22T16:05:21.999144+0000","last_became_peered":"2023-05-22T16:05:21.999144+0000","last_unstale":"2023-05-22T16:05:22.000126+0000","last_undegraded":"2023-05-22T16:05:22.000126+0000","last_fullsized":"2023-05-22T16:05:22.000126+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_clean_scrub_stamp":"2023-05-22T16:05:21.519005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:20:08.909225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.24","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2023-05-22T16:05:21.943911+0000","last_change":"2023-05-22T16:05:21.942712+0000","last_active":"2023-05-22T16:05:21.943911+0000","last_peered":"2023-05-22T16:05:21.943911+0000","last_clean":"2023-05-22T16:05:21.943911+0000","last_became_active":"2023-05-22T16:05:21.941658+0000","last_became_peered":"2023-05-22T16:05:21.941658+0000","last_unstale":"2023-05-22T16:05:21.943911+0000","last_undegraded":"2023-05-22T16:05:21.943911+0000","last_fullsized":"2023-05-22T16:05:21.943911+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_clean_scrub_stamp":"2023-05-22T16:05:21.519005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:37:47.243695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.23","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2023-05-22T16:05:21.994666+0000","last_change":"2023-05-22T16:05:21.981609+0000","last_active":"2023-05-22T16:05:21.994666+0000","last_peered":"2023-05-22T16:05:21.994666+0000","last_clean":"2023-05-22T16:05:21.994666+0000","last_became_active":"2023-05-22T16:05:21.980845+0000","last_became_peered":"2023-05-22T16:05:21.980845+0000","last_unstale":"2023-05-22T16:05:21.994666+0000","last_undegraded":"2023-05-22T16:05:21.994666+0000","last_fullsized":"2023-05-22T16:05:21.994666+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_clean_scrub_stamp":"2023-05-22T16:05:21.519005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T18:05:43.762192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.22","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2023-05-22T16:05:21.991036+0000","last_change":"2023-05-22T16:05:21.981856+0000","last_active":"2023-05-22T16:05:21.991036+0000","last_peered":"2023-05-22T16:05:21.991036+0000","last_clean":"2023-05-22T16:05:21.991036+0000","last_became_active":"2023-05-22T16:05:21.981105+0000","last_became_peered":"2023-05-22T16:05:21.981105+0000","last_unstale":"2023-05-22T16:05:21.991036+0000","last_undegraded":"2023-05-22T16:05:21.991036+0000","last_fullsized":"2023-05-22T16:05:21.991036+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_clean_scrub_stamp":"2023-05-22T16:05:21.519005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T18:11:18.469510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.21","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2023-05-22T16:05:21.988164+0000","last_change":"2023-05-22T16:05:21.987465+0000","last_active":"2023-05-22T16:05:21.988164+0000","last_peered":"2023-05-22T16:05:21.988164+0000","last_clean":"2023-05-22T16:05:21.988164+0000","last_became_active":"2023-05-22T16:05:21.987270+0000","last_became_peered":"2023-05-22T16:05:21.987270+0000","last_unstale":"2023-05-22T16:05:21.988164+0000","last_undegraded":"2023-05-22T16:05:21.988164+0000","last_fullsized":"2023-05-22T16:05:21.988164+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_clean_scrub_stamp":"2023-05-22T16:05:21.519005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T16:12:56.401350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.20","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2023-05-22T16:05:21.966709+0000","last_change":"2023-05-22T16:05:21.966136+0000","last_active":"2023-05-22T16:05:21.966709+0000","last_peered":"2023-05-22T16:05:21.966709+0000","last_clean":"2023-05-22T16:05:21.966709+0000","last_became_active":"2023-05-22T16:05:21.965842+0000","last_became_peered":"2023-05-22T16:05:21.965842+0000","last_unstale":"2023-05-22T16:05:21.966709+0000","last_undegraded":"2023-05-22T16:05:21.966709+0000","last_fullsized":"2023-05-22T16:05:21.966709+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_clean_scrub_stamp":"2023-05-22T16:05:21.519005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:55:10.528852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.16","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2023-05-22T16:05:21.953254+0000","last_change":"2023-05-22T16:05:21.950299+0000","last_active":"2023-05-22T16:05:21.953254+0000","last_peered":"2023-05-22T16:05:21.953254+0000","last_clean":"2023-05-22T16:05:21.953254+0000","last_became_active":"2023-05-22T16:05:21.950029+0000","last_became_peered":"2023-05-22T16:05:21.950029+0000","last_unstale":"2023-05-22T16:05:21.953254+0000","last_undegraded":"2023-05-22T16:05:21.953254+0000","last_fullsized":"2023-05-22T16:05:21.953254+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_clean_scrub_stamp":"2023-05-22T16:05:21.519005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:38:50.322794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.17","version":"0'0","reported_seq":10,"reported_epoch":195,"state":"active+clean","last_fresh":"2023-05-22T16:05:21.940688+0000","last_change":"2023-05-22T16:05:21.589641+0000","last_active":"2023-05-22T16:05:21.940688+0000","last_peered":"2023-05-22T16:05:21.940688+0000","last_clean":"2023-05-22T16:05:21.940688+0000","last_became_active":"2023-05-22T16:05:21.588996+0000","last_became_peered":"2023-05-22T16:05:21.588996+0000","last_unstale":"2023-05-22T16:05:21.940688+0000","last_undegraded":"2023-05-22T16:05:21.940688+0000","last_fullsized":"2023-05-22T16:05:21.940688+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:05:20.505200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:05:20.505200+0000","last_clean_scrub_stamp":"2023-05-22T16:05:20.505200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T16:32:24.571367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.17","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2023-05-22T16:05:21.955267+0000","last_change":"2023-05-22T16:05:21.953210+0000","last_active":"2023-05-22T16:05:21.955267+0000","last_peered":"2023-05-22T16:05:21.955267+0000","last_clean":"2023-05-22T16:05:21.955267+0000","last_became_active":"2023-05-22T16:05:21.952894+0000","last_became_peered":"2023-05-22T16:05:21.952894+0000","last_unstale":"2023-05-22T16:05:21.955267+0000","last_undegraded":"2023-05-22T16:05:21.955267+0000","last_fullsized":"2023-05-22T16:05:21.955267+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_clean_scrub_stamp":"2023-05-22T16:05:21.519005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:30:02.698847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.16","version":"0'0","reported_seq":10,"reported_epoch":195,"state":"active+clean","last_fresh":"2023-05-22T16:05:21.929668+0000","last_change":"2023-05-22T16:05:21.575900+0000","last_active":"2023-05-22T16:05:21.929668+0000","last_peered":"2023-05-22T16:05:21.929668+0000","last_clean":"2023-05-22T16:05:21.929668+0000","last_became_active":"2023-05-22T16:05:21.575461+0000","last_became_peered":"2023-05-22T16:05:21.575461+0000","last_unstale":"2023-05-22T16:05:21.929668+0000","last_undegraded":"2023-05-22T16:05:21.929668+0000","last_fullsized":"2023-05-22T16:05:21.929668+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:05:20.505200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:05:20.505200+0000","last_clean_scrub_stamp":"2023-05-22T16:05:20.505200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:02:08.546352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.14","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2023-05-22T16:05:21.987803+0000","last_change":"2023-05-22T16:05:21.972774+0000","last_active":"2023-05-22T16:05:21.987803+0000","last_peered":"2023-05-22T16:05:21.987803+0000","last_clean":"2023-05-22T16:05:21.987803+0000","last_became_active":"2023-05-22T16:05:21.971633+0000","last_became_peered":"2023-05-22T16:05:21.971633+0000","last_unstale":"2023-05-22T16:05:21.987803+0000","last_undegraded":"2023-05-22T16:05:21.987803+0000","last_fullsized":"2023-05-22T16:05:21.987803+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_clean_scrub_stamp":"2023-05-22T16:05:21.519005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:56:29.590805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.15","version":"195'3","reported_seq":13,"reported_epoch":195,"state":"active+clean","last_fresh":"2023-05-22T16:05:21.938841+0000","last_change":"2023-05-22T16:05:21.569294+0000","last_active":"2023-05-22T16:05:21.938841+0000","last_peered":"2023-05-22T16:05:21.938841+0000","last_clean":"2023-05-22T16:05:21.938841+0000","last_became_active":"2023-05-22T16:05:21.568914+0000","last_became_peered":"2023-05-22T16:05:21.568914+0000","last_unstale":"2023-05-22T16:05:21.938841+0000","last_undegraded":"2023-05-22T16:05:21.938841+0000","last_fullsized":"2023-05-22T16:05:21.938841+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:05:20.505200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:05:20.505200+0000","last_clean_scrub_stamp":"2023-05-22T16:05:20.505200+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:47:57.120748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.15","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2023-05-22T16:05:21.955179+0000","last_change":"2023-05-22T16:05:21.950957+0000","last_active":"2023-05-22T16:05:21.955179+0000","last_peered":"2023-05-22T16:05:21.955179+0000","last_clean":"2023-05-22T16:05:21.955179+0000","last_became_active":"2023-05-22T16:05:21.950627+0000","last_became_peered":"2023-05-22T16:05:21.950627+0000","last_unstale":"2023-05-22T16:05:21.955179+0000","last_undegraded":"2023-05-22T16:05:21.955179+0000","last_fullsized":"2023-05-22T16:05:21.955179+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_clean_scrub_stamp":"2023-05-22T16:05:21.519005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:23:31.666673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.14","version":"195'6","reported_seq":16,"reported_epoch":195,"state":"active+clean","last_fresh":"2023-05-22T16:05:21.997131+0000","last_change":"2023-05-22T16:05:21.575523+0000","last_active":"2023-05-22T16:05:21.997131+0000","last_peered":"2023-05-22T16:05:21.997131+0000","last_clean":"2023-05-22T16:05:21.997131+0000","last_became_active":"2023-05-22T16:05:21.575155+0000","last_became_peered":"2023-05-22T16:05:21.575155+0000","last_unstale":"2023-05-22T16:05:21.997131+0000","last_undegraded":"2023-05-22T16:05:21.997131+0000","last_fullsized":"2023-05-22T16:05:21.997131+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:05:20.505200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:05:20.505200+0000","last_clean_scrub_stamp":"2023-05-22T16:05:20.505200+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:49:38.410553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1548,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.12","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2023-05-22T16:05:21.954863+0000","last_change":"2023-05-22T16:05:21.953032+0000","last_active":"2023-05-22T16:05:21.954863+0000","last_peered":"2023-05-22T16:05:21.954863+0000","last_clean":"2023-05-22T16:05:21.954863+0000","last_became_active":"2023-05-22T16:05:21.952770+0000","last_became_peered":"2023-05-22T16:05:21.952770+0000","last_unstale":"2023-05-22T16:05:21.954863+0000","last_undegraded":"2023-05-22T16:05:21.954863+0000","last_fullsized":"2023-05-22T16:05:21.954863+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_clean_scrub_stamp":"2023-05-22T16:05:21.519005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T16:32:25.585172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.13","version":"195'2","reported_seq":12,"reported_epoch":195,"state":"active+clean","last_fresh":"2023-05-22T16:05:21.996956+0000","last_change":"2023-05-22T16:05:21.566159+0000","last_active":"2023-05-22T16:05:21.996956+0000","last_peered":"2023-05-22T16:05:21.996956+0000","last_clean":"2023-05-22T16:05:21.996956+0000","last_became_active":"2023-05-22T16:05:21.565413+0000","last_became_peered":"2023-05-22T16:05:21.565413+0000","last_unstale":"2023-05-22T16:05:21.996956+0000","last_undegraded":"2023-05-22T16:05:21.996956+0000","last_fullsized":"2023-05-22T16:05:21.996956+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:05:20.505200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:05:20.505200+0000","last_clean_scrub_stamp":"2023-05-22T16:05:20.505200+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:21:03.448650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.13","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2023-05-22T16:05:21.987136+0000","last_change":"2023-05-22T16:05:21.963650+0000","last_active":"2023-05-22T16:05:21.987136+0000","last_peered":"2023-05-22T16:05:21.987136+0000","last_clean":"2023-05-22T16:05:21.987136+0000","last_became_active":"2023-05-22T16:05:21.962778+0000","last_became_peered":"2023-05-22T16:05:21.962778+0000","last_unstale":"2023-05-22T16:05:21.987136+0000","last_undegraded":"2023-05-22T16:05:21.987136+0000","last_fullsized":"2023-05-22T16:05:21.987136+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_clean_scrub_stamp":"2023-05-22T16:05:21.519005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:52:47.830248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.12","version":"0'0","reported_seq":10,"reported_epoch":195,"state":"active+clean","last_fresh":"2023-05-22T16:05:21.937229+0000","last_change":"2023-05-22T16:05:21.570545+0000","last_active":"2023-05-22T16:05:21.937229+0000","last_peered":"2023-05-22T16:05:21.937229+0000","last_clean":"2023-05-22T16:05:21.937229+0000","last_became_active":"2023-05-22T16:05:21.570175+0000","last_became_peered":"2023-05-22T16:05:21.570175+0000","last_unstale":"2023-05-22T16:05:21.937229+0000","last_undegraded":"2023-05-22T16:05:21.937229+0000","last_fullsized":"2023-05-22T16:05:21.937229+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:05:20.505200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:05:20.505200+0000","last_clean_scrub_stamp":"2023-05-22T16:05:20.505200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:56:28.577000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.10","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2023-05-22T16:05:21.986843+0000","last_change":"2023-05-22T16:05:21.967603+0000","last_active":"2023-05-22T16:05:21.986843+0000","last_peered":"2023-05-22T16:05:21.986843+0000","last_clean":"2023-05-22T16:05:21.986843+0000","last_became_active":"2023-05-22T16:05:21.966720+0000","last_became_peered":"2023-05-22T16:05:21.966720+0000","last_unstale":"2023-05-22T16:05:21.986843+0000","last_undegraded":"2023-05-22T16:05:21.986843+0000","last_fullsized":"2023-05-22T16:05:21.986843+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_clean_scrub_stamp":"2023-05-22T16:05:21.519005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:38:47.496733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.11","version":"0'0","reported_seq":10,"reported_epoch":195,"state":"active+clean","last_fresh":"2023-05-22T16:05:21.939652+0000","last_change":"2023-05-22T16:05:21.577260+0000","last_active":"2023-05-22T16:05:21.939652+0000","last_peered":"2023-05-22T16:05:21.939652+0000","last_clean":"2023-05-22T16:05:21.939652+0000","last_became_active":"2023-05-22T16:05:21.576960+0000","last_became_peered":"2023-05-22T16:05:21.576960+0000","last_unstale":"2023-05-22T16:05:21.939652+0000","last_undegraded":"2023-05-22T16:05:21.939652+0000","last_fullsized":"2023-05-22T16:05:21.939652+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:05:20.505200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:05:20.505200+0000","last_clean_scrub_stamp":"2023-05-22T16:05:20.505200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:56:28.577000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.11","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2023-05-22T16:05:21.989716+0000","last_change":"2023-05-22T16:05:21.988706+0000","last_active":"2023-05-22T16:05:21.989716+0000","last_peered":"2023-05-22T16:05:21.989716+0000","last_clean":"2023-05-22T16:05:21.989716+0000","last_became_active":"2023-05-22T16:05:21.988206+0000","last_became_peered":"2023-05-22T16:05:21.988206+0000","last_unstale":"2023-05-22T16:05:21.989716+0000","last_undegraded":"2023-05-22T16:05:21.989716+0000","last_fullsized":"2023-05-22T16:05:21.989716+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_clean_scrub_stamp":"2023-05-22T16:05:21.519005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:54:46.681293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.10","version":"195'8","reported_seq":18,"reported_epoch":195,"state":"active+clean","last_fresh":"2023-05-22T16:05:22.003213+0000","last_change":"2023-05-22T16:05:21.565870+0000","last_active":"2023-05-22T16:05:22.003213+0000","last_peered":"2023-05-22T16:05:22.003213+0000","last_clean":"2023-05-22T16:05:22.003213+0000","last_became_active":"2023-05-22T16:05:21.565036+0000","last_became_peered":"2023-05-22T16:05:21.565036+0000","last_unstale":"2023-05-22T16:05:22.003213+0000","last_undegraded":"2023-05-22T16:05:22.003213+0000","last_fullsized":"2023-05-22T16:05:22.003213+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:05:20.505200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:05:20.505200+0000","last_clean_scrub_stamp":"2023-05-22T16:05:20.505200+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:29:44.137982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.e","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2023-05-22T16:05:21.986809+0000","last_change":"2023-05-22T16:05:21.986108+0000","last_active":"2023-05-22T16:05:21.986809+0000","last_peered":"2023-05-22T16:05:21.986809+0000","last_clean":"2023-05-22T16:05:21.986809+0000","last_became_active":"2023-05-22T16:05:21.985777+0000","last_became_peered":"2023-05-22T16:05:21.985777+0000","last_unstale":"2023-05-22T16:05:21.986809+0000","last_undegraded":"2023-05-22T16:05:21.986809+0000","last_fullsized":"2023-05-22T16:05:21.986809+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_clean_scrub_stamp":"2023-05-22T16:05:21.519005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:10:10.702604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.f","version":"195'1","reported_seq":11,"reported_epoch":195,"state":"active+clean","last_fresh":"2023-05-22T16:05:21.945708+0000","last_change":"2023-05-22T16:05:21.571450+0000","last_active":"2023-05-22T16:05:21.945708+0000","last_peered":"2023-05-22T16:05:21.945708+0000","last_clean":"2023-05-22T16:05:21.945708+0000","last_became_active":"2023-05-22T16:05:21.570848+0000","last_became_peered":"2023-05-22T16:05:21.570848+0000","last_unstale":"2023-05-22T16:05:21.945708+0000","last_undegraded":"2023-05-22T16:05:21.945708+0000","last_fullsized":"2023-05-22T16:05:21.945708+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:05:20.505200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:05:20.505200+0000","last_clean_scrub_stamp":"2023-05-22T16:05:20.505200+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:19:47.972327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.f","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2023-05-22T16:05:21.986302+0000","last_change":"2023-05-22T16:05:21.965884+0000","last_active":"2023-05-22T16:05:21.986302+0000","last_peered":"2023-05-22T16:05:21.986302+0000","last_clean":"2023-05-22T16:05:21.986302+0000","last_became_active":"2023-05-22T16:05:21.964971+0000","last_became_peered":"2023-05-22T16:05:21.964971+0000","last_unstale":"2023-05-22T16:05:21.986302+0000","last_undegraded":"2023-05-22T16:05:21.986302+0000","last_fullsized":"2023-05-22T16:05:21.986302+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_clean_scrub_stamp":"2023-05-22T16:05:21.519005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:32:05.958044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.e","version":"195'2","reported_seq":12,"reported_epoch":195,"state":"active+clean","last_fresh":"2023-05-22T16:05:21.989816+0000","last_change":"2023-05-22T16:05:21.576915+0000","last_active":"2023-05-22T16:05:21.989816+0000","last_peered":"2023-05-22T16:05:21.989816+0000","last_clean":"2023-05-22T16:05:21.989816+0000","last_became_active":"2023-05-22T16:05:21.576583+0000","last_became_peered":"2023-05-22T16:05:21.576583+0000","last_unstale":"2023-05-22T16:05:21.989816+0000","last_undegraded":"2023-05-22T16:05:21.989816+0000","last_fullsized":"2023-05-22T16:05:21.989816+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:05:20.505200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:05:20.505200+0000","last_clean_scrub_stamp":"2023-05-22T16:05:20.505200+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:54:29.426377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.c","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2023-05-22T16:05:22.002846+0000","last_change":"2023-05-22T16:05:21.998833+0000","last_active":"2023-05-22T16:05:22.002846+0000","last_peered":"2023-05-22T16:05:22.002846+0000","last_clean":"2023-05-22T16:05:22.002846+0000","last_became_active":"2023-05-22T16:05:21.997997+0000","last_became_peered":"2023-05-22T16:05:21.997997+0000","last_unstale":"2023-05-22T16:05:22.002846+0000","last_undegraded":"2023-05-22T16:05:22.002846+0000","last_fullsized":"2023-05-22T16:05:22.002846+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_clean_scrub_stamp":"2023-05-22T16:05:21.519005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T16:53:07.107628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.d","version":"195'1","reported_seq":11,"reported_epoch":195,"state":"active+clean","last_fresh":"2023-05-22T16:05:21.996305+0000","last_change":"2023-05-22T16:05:21.568161+0000","last_active":"2023-05-22T16:05:21.996305+0000","last_peered":"2023-05-22T16:05:21.996305+0000","last_clean":"2023-05-22T16:05:21.996305+0000","last_became_active":"2023-05-22T16:05:21.567369+0000","last_became_peered":"2023-05-22T16:05:21.567369+0000","last_unstale":"2023-05-22T16:05:21.996305+0000","last_undegraded":"2023-05-22T16:05:21.996305+0000","last_fullsized":"2023-05-22T16:05:21.996305+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:05:20.505200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:05:20.505200+0000","last_clean_scrub_stamp":"2023-05-22T16:05:20.505200+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:14:03.680350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.d","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2023-05-22T16:05:21.983853+0000","last_change":"2023-05-22T16:05:21.959490+0000","last_active":"2023-05-22T16:05:21.983853+0000","last_peered":"2023-05-22T16:05:21.983853+0000","last_clean":"2023-05-22T16:05:21.983853+0000","last_became_active":"2023-05-22T16:05:21.958689+0000","last_became_peered":"2023-05-22T16:05:21.958689+0000","last_unstale":"2023-05-22T16:05:21.983853+0000","last_undegraded":"2023-05-22T16:05:21.983853+0000","last_fullsized":"2023-05-22T16:05:21.983853+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_clean_scrub_stamp":"2023-05-22T16:05:21.519005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:33:18.037523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.c","version":"0'0","reported_seq":10,"reported_epoch":195,"state":"active+clean","last_fresh":"2023-05-22T16:05:21.940426+0000","last_change":"2023-05-22T16:05:21.573321+0000","last_active":"2023-05-22T16:05:21.940426+0000","last_peered":"2023-05-22T16:05:21.940426+0000","last_clean":"2023-05-22T16:05:21.940426+0000","last_became_active":"2023-05-22T16:05:21.572288+0000","last_became_peered":"2023-05-22T16:05:21.572288+0000","last_unstale":"2023-05-22T16:05:21.940426+0000","last_undegraded":"2023-05-22T16:05:21.940426+0000","last_fullsized":"2023-05-22T16:05:21.940426+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:05:20.505200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:05:20.505200+0000","last_clean_scrub_stamp":"2023-05-22T16:05:20.505200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:59:38.290472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.a","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2023-05-22T16:05:21.950544+0000","last_change":"2023-05-22T16:05:21.942660+0000","last_active":"2023-05-22T16:05:21.950544+0000","last_peered":"2023-05-22T16:05:21.950544+0000","last_clean":"2023-05-22T16:05:21.950544+0000","last_became_active":"2023-05-22T16:05:21.942278+0000","last_became_peered":"2023-05-22T16:05:21.942278+0000","last_unstale":"2023-05-22T16:05:21.950544+0000","last_undegraded":"2023-05-22T16:05:21.950544+0000","last_fullsized":"2023-05-22T16:05:21.950544+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_clean_scrub_stamp":"2023-05-22T16:05:21.519005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:03:30.813740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.b","version":"195'2","reported_seq":12,"reported_epoch":195,"state":"active+clean","last_fresh":"2023-05-22T16:05:21.952313+0000","last_change":"2023-05-22T16:05:21.554180+0000","last_active":"2023-05-22T16:05:21.952313+0000","last_peered":"2023-05-22T16:05:21.952313+0000","last_clean":"2023-05-22T16:05:21.952313+0000","last_became_active":"2023-05-22T16:05:21.553865+0000","last_became_peered":"2023-05-22T16:05:21.553865+0000","last_unstale":"2023-05-22T16:05:21.952313+0000","last_undegraded":"2023-05-22T16:05:21.952313+0000","last_fullsized":"2023-05-22T16:05:21.952313+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:05:20.505200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:05:20.505200+0000","last_clean_scrub_stamp":"2023-05-22T16:05:20.505200+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:38:46.482928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.b","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2023-05-22T16:05:21.987004+0000","last_change":"2023-05-22T16:05:21.959708+0000","last_active":"2023-05-22T16:05:21.987004+0000","last_peered":"2023-05-22T16:05:21.987004+0000","last_clean":"2023-05-22T16:05:21.987004+0000","last_became_active":"2023-05-22T16:05:21.958059+0000","last_became_peered":"2023-05-22T16:05:21.958059+0000","last_unstale":"2023-05-22T16:05:21.987004+0000","last_undegraded":"2023-05-22T16:05:21.987004+0000","last_fullsized":"2023-05-22T16:05:21.987004+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_clean_scrub_stamp":"2023-05-22T16:05:21.519005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:29:45.151787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.a","version":"0'0","reported_seq":10,"reported_epoch":195,"state":"active+clean","last_fresh":"2023-05-22T16:05:21.928184+0000","last_change":"2023-05-22T16:05:21.572978+0000","last_active":"2023-05-22T16:05:21.928184+0000","last_peered":"2023-05-22T16:05:21.928184+0000","last_clean":"2023-05-22T16:05:21.928184+0000","last_became_active":"2023-05-22T16:05:21.572735+0000","last_became_peered":"2023-05-22T16:05:21.572735+0000","last_unstale":"2023-05-22T16:05:21.928184+0000","last_undegraded":"2023-05-22T16:05:21.928184+0000","last_fullsized":"2023-05-22T16:05:21.928184+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:05:20.505200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:05:20.505200+0000","last_clean_scrub_stamp":"2023-05-22T16:05:20.505200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T16:16:12.432825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.8","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2023-05-22T16:05:21.994865+0000","last_change":"2023-05-22T16:05:21.985741+0000","last_active":"2023-05-22T16:05:21.994865+0000","last_peered":"2023-05-22T16:05:21.994865+0000","last_clean":"2023-05-22T16:05:21.994865+0000","last_became_active":"2023-05-22T16:05:21.984619+0000","last_became_peered":"2023-05-22T16:05:21.984619+0000","last_unstale":"2023-05-22T16:05:21.994865+0000","last_undegraded":"2023-05-22T16:05:21.994865+0000","last_fullsized":"2023-05-22T16:05:21.994865+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_clean_scrub_stamp":"2023-05-22T16:05:21.519005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:20:08.909225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.9","version":"0'0","reported_seq":10,"reported_epoch":195,"state":"active+clean","last_fresh":"2023-05-22T16:05:21.943510+0000","last_change":"2023-05-22T16:05:21.565141+0000","last_active":"2023-05-22T16:05:21.943510+0000","last_peered":"2023-05-22T16:05:21.943510+0000","last_clean":"2023-05-22T16:05:21.943510+0000","last_became_active":"2023-05-22T16:05:21.564803+0000","last_became_peered":"2023-05-22T16:05:21.564803+0000","last_unstale":"2023-05-22T16:05:21.943510+0000","last_undegraded":"2023-05-22T16:05:21.943510+0000","last_fullsized":"2023-05-22T16:05:21.943510+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:05:20.505200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:05:20.505200+0000","last_clean_scrub_stamp":"2023-05-22T16:05:20.505200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T16:43:05.102031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.9","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2023-05-22T16:05:21.954713+0000","last_change":"2023-05-22T16:05:21.952574+0000","last_active":"2023-05-22T16:05:21.954713+0000","last_peered":"2023-05-22T16:05:21.954713+0000","last_clean":"2023-05-22T16:05:21.954713+0000","last_became_active":"2023-05-22T16:05:21.950472+0000","last_became_peered":"2023-05-22T16:05:21.950472+0000","last_unstale":"2023-05-22T16:05:21.954713+0000","last_undegraded":"2023-05-22T16:05:21.954713+0000","last_fullsized":"2023-05-22T16:05:21.954713+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_clean_scrub_stamp":"2023-05-22T16:05:21.519005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:06:43.269527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.8","version":"0'0","reported_seq":10,"reported_epoch":195,"state":"active+clean","last_fresh":"2023-05-22T16:05:21.957022+0000","last_change":"2023-05-22T16:05:21.579600+0000","last_active":"2023-05-22T16:05:21.957022+0000","last_peered":"2023-05-22T16:05:21.957022+0000","last_clean":"2023-05-22T16:05:21.957022+0000","last_became_active":"2023-05-22T16:05:21.578847+0000","last_became_peered":"2023-05-22T16:05:21.578847+0000","last_unstale":"2023-05-22T16:05:21.957022+0000","last_undegraded":"2023-05-22T16:05:21.957022+0000","last_fullsized":"2023-05-22T16:05:21.957022+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:05:20.505200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:05:20.505200+0000","last_clean_scrub_stamp":"2023-05-22T16:05:20.505200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T16:16:16.793031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.6","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2023-05-22T16:05:21.954765+0000","last_change":"2023-05-22T16:05:21.953258+0000","last_active":"2023-05-22T16:05:21.954765+0000","last_peered":"2023-05-22T16:05:21.954765+0000","last_clean":"2023-05-22T16:05:21.954765+0000","last_became_active":"2023-05-22T16:05:21.952881+0000","last_became_peered":"2023-05-22T16:05:21.952881+0000","last_unstale":"2023-05-22T16:05:21.954765+0000","last_undegraded":"2023-05-22T16:05:21.954765+0000","last_fullsized":"2023-05-22T16:05:21.954765+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_clean_scrub_stamp":"2023-05-22T16:05:21.519005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:42:58.578446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.7","version":"195'2","reported_seq":12,"reported_epoch":195,"state":"active+clean","last_fresh":"2023-05-22T16:05:21.997383+0000","last_change":"2023-05-22T16:05:21.568026+0000","last_active":"2023-05-22T16:05:21.997383+0000","last_peered":"2023-05-22T16:05:21.997383+0000","last_clean":"2023-05-22T16:05:21.997383+0000","last_became_active":"2023-05-22T16:05:21.567033+0000","last_became_peered":"2023-05-22T16:05:21.567033+0000","last_unstale":"2023-05-22T16:05:21.997383+0000","last_undegraded":"2023-05-22T16:05:21.997383+0000","last_fullsized":"2023-05-22T16:05:21.997383+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:05:20.505200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:05:20.505200+0000","last_clean_scrub_stamp":"2023-05-22T16:05:20.505200+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:05:01.227820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.7","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2023-05-22T16:05:21.984485+0000","last_change":"2023-05-22T16:05:21.983608+0000","last_active":"2023-05-22T16:05:21.984485+0000","last_peered":"2023-05-22T16:05:21.984485+0000","last_clean":"2023-05-22T16:05:21.984485+0000","last_became_active":"2023-05-22T16:05:21.983327+0000","last_became_peered":"2023-05-22T16:05:21.983327+0000","last_unstale":"2023-05-22T16:05:21.984485+0000","last_undegraded":"2023-05-22T16:05:21.984485+0000","last_fullsized":"2023-05-22T16:05:21.984485+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_clean_scrub_stamp":"2023-05-22T16:05:21.519005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:31:27.332257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.6","version":"0'0","reported_seq":10,"reported_epoch":195,"state":"active+clean","last_fresh":"2023-05-22T16:05:21.935195+0000","last_change":"2023-05-22T16:05:21.570917+0000","last_active":"2023-05-22T16:05:21.935195+0000","last_peered":"2023-05-22T16:05:21.935195+0000","last_clean":"2023-05-22T16:05:21.935195+0000","last_became_active":"2023-05-22T16:05:21.570604+0000","last_became_peered":"2023-05-22T16:05:21.570604+0000","last_unstale":"2023-05-22T16:05:21.935195+0000","last_undegraded":"2023-05-22T16:05:21.935195+0000","last_fullsized":"2023-05-22T16:05:21.935195+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:05:20.505200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:05:20.505200+0000","last_clean_scrub_stamp":"2023-05-22T16:05:20.505200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:52:46.816443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.4","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2023-05-22T16:05:21.955023+0000","last_change":"2023-05-22T16:05:21.950184+0000","last_active":"2023-05-22T16:05:21.955023+0000","last_peered":"2023-05-22T16:05:21.955023+0000","last_clean":"2023-05-22T16:05:21.955023+0000","last_became_active":"2023-05-22T16:05:21.949891+0000","last_became_peered":"2023-05-22T16:05:21.949891+0000","last_unstale":"2023-05-22T16:05:21.955023+0000","last_undegraded":"2023-05-22T16:05:21.955023+0000","last_fullsized":"2023-05-22T16:05:21.955023+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_clean_scrub_stamp":"2023-05-22T16:05:21.519005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:35:44.913595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.5","version":"0'0","reported_seq":10,"reported_epoch":195,"state":"active+clean","last_fresh":"2023-05-22T16:05:21.934745+0000","last_change":"2023-05-22T16:05:21.574565+0000","last_active":"2023-05-22T16:05:21.934745+0000","last_peered":"2023-05-22T16:05:21.934745+0000","last_clean":"2023-05-22T16:05:21.934745+0000","last_became_active":"2023-05-22T16:05:21.574090+0000","last_became_peered":"2023-05-22T16:05:21.574090+0000","last_unstale":"2023-05-22T16:05:21.934745+0000","last_undegraded":"2023-05-22T16:05:21.934745+0000","last_fullsized":"2023-05-22T16:05:21.934745+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:05:20.505200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:05:20.505200+0000","last_clean_scrub_stamp":"2023-05-22T16:05:20.505200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T18:38:47.331710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.5","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2023-05-22T16:05:21.977181+0000","last_change":"2023-05-22T16:05:21.975813+0000","last_active":"2023-05-22T16:05:21.977181+0000","last_peered":"2023-05-22T16:05:21.977181+0000","last_clean":"2023-05-22T16:05:21.977181+0000","last_became_active":"2023-05-22T16:05:21.975554+0000","last_became_peered":"2023-05-22T16:05:21.975554+0000","last_unstale":"2023-05-22T16:05:21.977181+0000","last_undegraded":"2023-05-22T16:05:21.977181+0000","last_fullsized":"2023-05-22T16:05:21.977181+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_clean_scrub_stamp":"2023-05-22T16:05:21.519005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:03:30.813740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.4","version":"0'0","reported_seq":10,"reported_epoch":195,"state":"active+clean","last_fresh":"2023-05-22T16:05:21.940406+0000","last_change":"2023-05-22T16:05:21.589075+0000","last_active":"2023-05-22T16:05:21.940406+0000","last_peered":"2023-05-22T16:05:21.940406+0000","last_clean":"2023-05-22T16:05:21.940406+0000","last_became_active":"2023-05-22T16:05:21.588792+0000","last_became_peered":"2023-05-22T16:05:21.588792+0000","last_unstale":"2023-05-22T16:05:21.940406+0000","last_undegraded":"2023-05-22T16:05:21.940406+0000","last_fullsized":"2023-05-22T16:05:21.940406+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:05:20.505200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:05:20.505200+0000","last_clean_scrub_stamp":"2023-05-22T16:05:20.505200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:52:46.816443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.2","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2023-05-22T16:05:22.004893+0000","last_change":"2023-05-22T16:05:21.997428+0000","last_active":"2023-05-22T16:05:22.004893+0000","last_peered":"2023-05-22T16:05:22.004893+0000","last_clean":"2023-05-22T16:05:22.004893+0000","last_became_active":"2023-05-22T16:05:21.996749+0000","last_became_peered":"2023-05-22T16:05:21.996749+0000","last_unstale":"2023-05-22T16:05:22.004893+0000","last_undegraded":"2023-05-22T16:05:22.004893+0000","last_fullsized":"2023-05-22T16:05:22.004893+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_clean_scrub_stamp":"2023-05-22T16:05:21.519005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:38:35.770174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.3","version":"0'0","reported_seq":10,"reported_epoch":195,"state":"active+clean","last_fresh":"2023-05-22T16:05:21.957343+0000","last_change":"2023-05-22T16:05:21.588428+0000","last_active":"2023-05-22T16:05:21.957343+0000","last_peered":"2023-05-22T16:05:21.957343+0000","last_clean":"2023-05-22T16:05:21.957343+0000","last_became_active":"2023-05-22T16:05:21.587723+0000","last_became_peered":"2023-05-22T16:05:21.587723+0000","last_unstale":"2023-05-22T16:05:21.957343+0000","last_undegraded":"2023-05-22T16:05:21.957343+0000","last_fullsized":"2023-05-22T16:05:21.957343+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:05:20.505200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:05:20.505200+0000","last_clean_scrub_stamp":"2023-05-22T16:05:20.505200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:48:51.932099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.3","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2023-05-22T16:05:22.004844+0000","last_change":"2023-05-22T16:05:21.997207+0000","last_active":"2023-05-22T16:05:22.004844+0000","last_peered":"2023-05-22T16:05:22.004844+0000","last_clean":"2023-05-22T16:05:22.004844+0000","last_became_active":"2023-05-22T16:05:21.996528+0000","last_became_peered":"2023-05-22T16:05:21.996528+0000","last_unstale":"2023-05-22T16:05:22.004844+0000","last_undegraded":"2023-05-22T16:05:22.004844+0000","last_fullsized":"2023-05-22T16:05:22.004844+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_clean_scrub_stamp":"2023-05-22T16:05:21.519005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T03:24:08.639357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.2","version":"0'0","reported_seq":10,"reported_epoch":195,"state":"active+clean","last_fresh":"2023-05-22T16:05:21.958132+0000","last_change":"2023-05-22T16:05:21.573511+0000","last_active":"2023-05-22T16:05:21.958132+0000","last_peered":"2023-05-22T16:05:21.958132+0000","last_clean":"2023-05-22T16:05:21.958132+0000","last_became_active":"2023-05-22T16:05:21.571649+0000","last_became_peered":"2023-05-22T16:05:21.571649+0000","last_unstale":"2023-05-22T16:05:21.958132+0000","last_undegraded":"2023-05-22T16:05:21.958132+0000","last_fullsized":"2023-05-22T16:05:21.958132+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:05:20.505200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:05:20.505200+0000","last_clean_scrub_stamp":"2023-05-22T16:05:20.505200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:42:57.564641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.0","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2023-05-22T16:05:21.994657+0000","last_change":"2023-05-22T16:05:21.981143+0000","last_active":"2023-05-22T16:05:21.994657+0000","last_peered":"2023-05-22T16:05:21.994657+0000","last_clean":"2023-05-22T16:05:21.994657+0000","last_became_active":"2023-05-22T16:05:21.980177+0000","last_became_peered":"2023-05-22T16:05:21.980177+0000","last_unstale":"2023-05-22T16:05:21.994657+0000","last_undegraded":"2023-05-22T16:05:21.994657+0000","last_fullsized":"2023-05-22T16:05:21.994657+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_clean_scrub_stamp":"2023-05-22T16:05:21.519005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:59:51.654460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.1","version":"0'0","reported_seq":10,"reported_epoch":195,"state":"active+clean","last_fresh":"2023-05-22T16:05:21.938576+0000","last_change":"2023-05-22T16:05:21.578903+0000","last_active":"2023-05-22T16:05:21.938576+0000","last_peered":"2023-05-22T16:05:21.938576+0000","last_clean":"2023-05-22T16:05:21.938576+0000","last_became_active":"2023-05-22T16:05:21.578160+0000","last_became_peered":"2023-05-22T16:05:21.578160+0000","last_unstale":"2023-05-22T16:05:21.938576+0000","last_undegraded":"2023-05-22T16:05:21.938576+0000","last_fullsized":"2023-05-22T16:05:21.938576+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:05:20.505200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:05:20.505200+0000","last_clean_scrub_stamp":"2023-05-22T16:05:20.505200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:49:38.410553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.1","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2023-05-22T16:05:21.965326+0000","last_change":"2023-05-22T16:05:21.964514+0000","last_active":"2023-05-22T16:05:21.965326+0000","last_peered":"2023-05-22T16:05:21.965326+0000","last_clean":"2023-05-22T16:05:21.965326+0000","last_became_active":"2023-05-22T16:05:21.964217+0000","last_became_peered":"2023-05-22T16:05:21.964217+0000","last_unstale":"2023-05-22T16:05:21.965326+0000","last_undegraded":"2023-05-22T16:05:21.965326+0000","last_fullsized":"2023-05-22T16:05:21.965326+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_clean_scrub_stamp":"2023-05-22T16:05:21.519005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:10:09.169722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.0","version":"0'0","reported_seq":10,"reported_epoch":195,"state":"active+clean","last_fresh":"2023-05-22T16:05:21.946515+0000","last_change":"2023-05-22T16:05:21.566366+0000","last_active":"2023-05-22T16:05:21.946515+0000","last_peered":"2023-05-22T16:05:21.946515+0000","last_clean":"2023-05-22T16:05:21.946515+0000","last_became_active":"2023-05-22T16:05:21.566072+0000","last_became_peered":"2023-05-22T16:05:21.566072+0000","last_unstale":"2023-05-22T16:05:21.946515+0000","last_undegraded":"2023-05-22T16:05:21.946515+0000","last_fullsized":"2023-05-22T16:05:21.946515+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:05:20.505200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:05:20.505200+0000","last_clean_scrub_stamp":"2023-05-22T16:05:20.505200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T16:53:06.093823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.18","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2023-05-22T16:05:21.984466+0000","last_change":"2023-05-22T16:05:21.983501+0000","last_active":"2023-05-22T16:05:21.984466+0000","last_peered":"2023-05-22T16:05:21.984466+0000","last_clean":"2023-05-22T16:05:21.984466+0000","last_became_active":"2023-05-22T16:05:21.983182+0000","last_became_peered":"2023-05-22T16:05:21.983182+0000","last_unstale":"2023-05-22T16:05:21.984466+0000","last_undegraded":"2023-05-22T16:05:21.984466+0000","last_fullsized":"2023-05-22T16:05:21.984466+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_clean_scrub_stamp":"2023-05-22T16:05:21.519005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:24:20.405612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.19","version":"0'0","reported_seq":10,"reported_epoch":195,"state":"active+clean","last_fresh":"2023-05-22T16:05:21.965145+0000","last_change":"2023-05-22T16:05:21.579892+0000","last_active":"2023-05-22T16:05:21.965145+0000","last_peered":"2023-05-22T16:05:21.965145+0000","last_clean":"2023-05-22T16:05:21.965145+0000","last_became_active":"2023-05-22T16:05:21.579210+0000","last_became_peered":"2023-05-22T16:05:21.579210+0000","last_unstale":"2023-05-22T16:05:21.965145+0000","last_undegraded":"2023-05-22T16:05:21.965145+0000","last_fullsized":"2023-05-22T16:05:21.965145+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:05:20.505200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:05:20.505200+0000","last_clean_scrub_stamp":"2023-05-22T16:05:20.505200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:56:28.577000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"183'326","reported_seq":1078,"reported_epoch":194,"state":"active+clean","last_fresh":"2023-05-22T16:05:21.542676+0000","last_change":"2023-05-22T15:26:39.318204+0000","last_active":"2023-05-22T16:05:21.542676+0000","last_peered":"2023-05-22T16:05:21.542676+0000","last_clean":"2023-05-22T16:05:21.542676+0000","last_became_active":"2023-05-22T15:26:39.315370+0000","last_became_peered":"2023-05-22T15:26:39.315370+0000","last_unstale":"2023-05-22T16:05:21.542676+0000","last_undegraded":"2023-05-22T16:05:21.542676+0000","last_fullsized":"2023-05-22T16:05:21.542676+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:38.020076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:38.020076+0000","last_clean_scrub_stamp":"2023-05-22T15:26:38.020076+0000","objects_scrubbed":0,"log_size":326,"log_dups_size":0,"ondisk_log_size":326,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:06:39.207543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1940,"num_read_kb":4014,"num_write":539,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":195,"reported_epoch":194,"state":"active+clean","last_fresh":"2023-05-22T16:05:21.555321+0000","last_change":"2023-05-22T15:26:44.971951+0000","last_active":"2023-05-22T16:05:21.555321+0000","last_peered":"2023-05-22T16:05:21.555321+0000","last_clean":"2023-05-22T16:05:21.555321+0000","last_became_active":"2023-05-22T15:26:44.970872+0000","last_became_peered":"2023-05-22T15:26:44.970872+0000","last_unstale":"2023-05-22T16:05:21.555321+0000","last_undegraded":"2023-05-22T16:05:21.555321+0000","last_fullsized":"2023-05-22T16:05:21.555321+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:55:34.752949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.1b","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2023-05-22T16:05:21.973217+0000","last_change":"2023-05-22T16:05:21.971211+0000","last_active":"2023-05-22T16:05:21.973217+0000","last_peered":"2023-05-22T16:05:21.973217+0000","last_clean":"2023-05-22T16:05:21.973217+0000","last_became_active":"2023-05-22T16:05:21.970857+0000","last_became_peered":"2023-05-22T16:05:21.970857+0000","last_unstale":"2023-05-22T16:05:21.973217+0000","last_undegraded":"2023-05-22T16:05:21.973217+0000","last_fullsized":"2023-05-22T16:05:21.973217+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_clean_scrub_stamp":"2023-05-22T16:05:21.519005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:24:20.405612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1a","version":"195'1","reported_seq":11,"reported_epoch":195,"state":"active+clean","last_fresh":"2023-05-22T16:05:21.949529+0000","last_change":"2023-05-22T16:05:21.566496+0000","last_active":"2023-05-22T16:05:21.949529+0000","last_peered":"2023-05-22T16:05:21.949529+0000","last_clean":"2023-05-22T16:05:21.949529+0000","last_became_active":"2023-05-22T16:05:21.566147+0000","last_became_peered":"2023-05-22T16:05:21.566147+0000","last_unstale":"2023-05-22T16:05:21.949529+0000","last_undegraded":"2023-05-22T16:05:21.949529+0000","last_fullsized":"2023-05-22T16:05:21.949529+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:05:20.505200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:05:20.505200+0000","last_clean_scrub_stamp":"2023-05-22T16:05:20.505200+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:05:01.227820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":184,"reported_epoch":194,"state":"active+clean","last_fresh":"2023-05-22T16:05:21.542445+0000","last_change":"2023-05-22T15:26:44.977198+0000","last_active":"2023-05-22T16:05:21.542445+0000","last_peered":"2023-05-22T16:05:21.542445+0000","last_clean":"2023-05-22T16:05:21.542445+0000","last_became_active":"2023-05-22T15:26:44.976445+0000","last_became_peered":"2023-05-22T15:26:44.976445+0000","last_unstale":"2023-05-22T16:05:21.542445+0000","last_undegraded":"2023-05-22T16:05:21.542445+0000","last_fullsized":"2023-05-22T16:05:21.542445+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:12:41.464054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.1a","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2023-05-22T16:05:21.952457+0000","last_change":"2023-05-22T16:05:21.950222+0000","last_active":"2023-05-22T16:05:21.952457+0000","last_peered":"2023-05-22T16:05:21.952457+0000","last_clean":"2023-05-22T16:05:21.952457+0000","last_became_active":"2023-05-22T16:05:21.948530+0000","last_became_peered":"2023-05-22T16:05:21.948530+0000","last_unstale":"2023-05-22T16:05:21.952457+0000","last_undegraded":"2023-05-22T16:05:21.952457+0000","last_fullsized":"2023-05-22T16:05:21.952457+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_clean_scrub_stamp":"2023-05-22T16:05:21.519005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:51:31.114141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.1b","version":"195'1","reported_seq":11,"reported_epoch":195,"state":"active+clean","last_fresh":"2023-05-22T16:05:21.941700+0000","last_change":"2023-05-22T16:05:21.571275+0000","last_active":"2023-05-22T16:05:21.941700+0000","last_peered":"2023-05-22T16:05:21.941700+0000","last_clean":"2023-05-22T16:05:21.941700+0000","last_became_active":"2023-05-22T16:05:21.570988+0000","last_became_peered":"2023-05-22T16:05:21.570988+0000","last_unstale":"2023-05-22T16:05:21.941700+0000","last_undegraded":"2023-05-22T16:05:21.941700+0000","last_fullsized":"2023-05-22T16:05:21.941700+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:05:20.505200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:05:20.505200+0000","last_clean_scrub_stamp":"2023-05-22T16:05:20.505200+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:54:20.347388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":188,"reported_epoch":194,"state":"active+clean","last_fresh":"2023-05-22T16:05:21.537715+0000","last_change":"2023-05-22T15:26:44.969469+0000","last_active":"2023-05-22T16:05:21.537715+0000","last_peered":"2023-05-22T16:05:21.537715+0000","last_clean":"2023-05-22T16:05:21.537715+0000","last_became_active":"2023-05-22T15:26:44.969083+0000","last_became_peered":"2023-05-22T15:26:44.969083+0000","last_unstale":"2023-05-22T16:05:21.537715+0000","last_undegraded":"2023-05-22T16:05:21.537715+0000","last_fullsized":"2023-05-22T16:05:21.537715+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:29:23.670067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.19","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2023-05-22T16:05:21.992099+0000","last_change":"2023-05-22T16:05:21.983557+0000","last_active":"2023-05-22T16:05:21.992099+0000","last_peered":"2023-05-22T16:05:21.992099+0000","last_clean":"2023-05-22T16:05:21.992099+0000","last_became_active":"2023-05-22T16:05:21.982644+0000","last_became_peered":"2023-05-22T16:05:21.982644+0000","last_unstale":"2023-05-22T16:05:21.992099+0000","last_undegraded":"2023-05-22T16:05:21.992099+0000","last_fullsized":"2023-05-22T16:05:21.992099+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_clean_scrub_stamp":"2023-05-22T16:05:21.519005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T03:02:26.541310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.18","version":"0'0","reported_seq":10,"reported_epoch":195,"state":"active+clean","last_fresh":"2023-05-22T16:05:21.956111+0000","last_change":"2023-05-22T16:05:21.568215+0000","last_active":"2023-05-22T16:05:21.956111+0000","last_peered":"2023-05-22T16:05:21.956111+0000","last_clean":"2023-05-22T16:05:21.956111+0000","last_became_active":"2023-05-22T16:05:21.567241+0000","last_became_peered":"2023-05-22T16:05:21.567241+0000","last_unstale":"2023-05-22T16:05:21.956111+0000","last_undegraded":"2023-05-22T16:05:21.956111+0000","last_fullsized":"2023-05-22T16:05:21.956111+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:05:20.505200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:05:20.505200+0000","last_clean_scrub_stamp":"2023-05-22T16:05:20.505200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:54:29.426377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":195,"reported_epoch":194,"state":"active+clean","last_fresh":"2023-05-22T16:05:21.555395+0000","last_change":"2023-05-22T15:26:44.976632+0000","last_active":"2023-05-22T16:05:21.555395+0000","last_peered":"2023-05-22T16:05:21.555395+0000","last_clean":"2023-05-22T16:05:21.555395+0000","last_became_active":"2023-05-22T15:26:44.975546+0000","last_became_peered":"2023-05-22T15:26:44.975546+0000","last_unstale":"2023-05-22T16:05:21.555395+0000","last_undegraded":"2023-05-22T16:05:21.555395+0000","last_fullsized":"2023-05-22T16:05:21.555395+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:42:57.973030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.1f","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2023-05-22T16:05:21.950112+0000","last_change":"2023-05-22T16:05:21.942024+0000","last_active":"2023-05-22T16:05:21.950112+0000","last_peered":"2023-05-22T16:05:21.950112+0000","last_clean":"2023-05-22T16:05:21.950112+0000","last_became_active":"2023-05-22T16:05:21.941728+0000","last_became_peered":"2023-05-22T16:05:21.941728+0000","last_unstale":"2023-05-22T16:05:21.950112+0000","last_undegraded":"2023-05-22T16:05:21.950112+0000","last_fullsized":"2023-05-22T16:05:21.950112+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_clean_scrub_stamp":"2023-05-22T16:05:21.519005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:41:43.814799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.1e","version":"195'1","reported_seq":11,"reported_epoch":195,"state":"active+clean","last_fresh":"2023-05-22T16:05:21.997614+0000","last_change":"2023-05-22T16:05:21.574643+0000","last_active":"2023-05-22T16:05:21.997614+0000","last_peered":"2023-05-22T16:05:21.997614+0000","last_clean":"2023-05-22T16:05:21.997614+0000","last_became_active":"2023-05-22T16:05:21.573908+0000","last_became_peered":"2023-05-22T16:05:21.573908+0000","last_unstale":"2023-05-22T16:05:21.997614+0000","last_undegraded":"2023-05-22T16:05:21.997614+0000","last_fullsized":"2023-05-22T16:05:21.997614+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:05:20.505200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:05:20.505200+0000","last_clean_scrub_stamp":"2023-05-22T16:05:20.505200+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:22:43.835901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":173,"reported_epoch":194,"state":"active+clean","last_fresh":"2023-05-22T16:05:21.547232+0000","last_change":"2023-05-22T15:26:45.091475+0000","last_active":"2023-05-22T16:05:21.547232+0000","last_peered":"2023-05-22T16:05:21.547232+0000","last_clean":"2023-05-22T16:05:21.547232+0000","last_became_active":"2023-05-22T15:26:45.091143+0000","last_became_peered":"2023-05-22T15:26:45.091143+0000","last_unstale":"2023-05-22T16:05:21.547232+0000","last_undegraded":"2023-05-22T16:05:21.547232+0000","last_fullsized":"2023-05-22T16:05:21.547232+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:12:41.464054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.1e","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2023-05-22T16:05:21.972457+0000","last_change":"2023-05-22T16:05:21.971302+0000","last_active":"2023-05-22T16:05:21.972457+0000","last_peered":"2023-05-22T16:05:21.972457+0000","last_clean":"2023-05-22T16:05:21.972457+0000","last_became_active":"2023-05-22T16:05:21.970903+0000","last_became_peered":"2023-05-22T16:05:21.970903+0000","last_unstale":"2023-05-22T16:05:21.972457+0000","last_undegraded":"2023-05-22T16:05:21.972457+0000","last_fullsized":"2023-05-22T16:05:21.972457+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_clean_scrub_stamp":"2023-05-22T16:05:21.519005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:46:30.583073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1f","version":"195'4","reported_seq":14,"reported_epoch":195,"state":"active+clean","last_fresh":"2023-05-22T16:05:21.985933+0000","last_change":"2023-05-22T16:05:21.577808+0000","last_active":"2023-05-22T16:05:21.985933+0000","last_peered":"2023-05-22T16:05:21.985933+0000","last_clean":"2023-05-22T16:05:21.985933+0000","last_became_active":"2023-05-22T16:05:21.577440+0000","last_became_peered":"2023-05-22T16:05:21.577440+0000","last_unstale":"2023-05-22T16:05:21.985933+0000","last_undegraded":"2023-05-22T16:05:21.985933+0000","last_fullsized":"2023-05-22T16:05:21.985933+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:05:20.505200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:05:20.505200+0000","last_clean_scrub_stamp":"2023-05-22T16:05:20.505200+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:44:33.016687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":566,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":183,"reported_epoch":194,"state":"active+clean","last_fresh":"2023-05-22T16:05:21.543449+0000","last_change":"2023-05-22T15:26:45.092367+0000","last_active":"2023-05-22T16:05:21.543449+0000","last_peered":"2023-05-22T16:05:21.543449+0000","last_clean":"2023-05-22T16:05:21.543449+0000","last_became_active":"2023-05-22T15:26:45.091951+0000","last_became_peered":"2023-05-22T15:26:45.091951+0000","last_unstale":"2023-05-22T16:05:21.543449+0000","last_undegraded":"2023-05-22T16:05:21.543449+0000","last_fullsized":"2023-05-22T16:05:21.543449+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:42:57.973030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.1d","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2023-05-22T16:05:21.992938+0000","last_change":"2023-05-22T16:05:21.988728+0000","last_active":"2023-05-22T16:05:21.992938+0000","last_peered":"2023-05-22T16:05:21.992938+0000","last_clean":"2023-05-22T16:05:21.992938+0000","last_became_active":"2023-05-22T16:05:21.988029+0000","last_became_peered":"2023-05-22T16:05:21.988029+0000","last_unstale":"2023-05-22T16:05:21.992938+0000","last_undegraded":"2023-05-22T16:05:21.992938+0000","last_fullsized":"2023-05-22T16:05:21.992938+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_clean_scrub_stamp":"2023-05-22T16:05:21.519005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:56:19.066496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.1c","version":"195'2","reported_seq":12,"reported_epoch":195,"state":"active+clean","last_fresh":"2023-05-22T16:05:21.986407+0000","last_change":"2023-05-22T16:05:21.583033+0000","last_active":"2023-05-22T16:05:21.986407+0000","last_peered":"2023-05-22T16:05:21.986407+0000","last_clean":"2023-05-22T16:05:21.986407+0000","last_became_active":"2023-05-22T16:05:21.582280+0000","last_became_peered":"2023-05-22T16:05:21.582280+0000","last_unstale":"2023-05-22T16:05:21.986407+0000","last_undegraded":"2023-05-22T16:05:21.986407+0000","last_fullsized":"2023-05-22T16:05:21.986407+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:05:20.505200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:05:20.505200+0000","last_clean_scrub_stamp":"2023-05-22T16:05:20.505200+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T18:11:17.455705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":172,"reported_epoch":194,"state":"active+clean","last_fresh":"2023-05-22T16:05:21.547475+0000","last_change":"2023-05-22T15:26:44.978858+0000","last_active":"2023-05-22T16:05:21.547475+0000","last_peered":"2023-05-22T16:05:21.547475+0000","last_clean":"2023-05-22T16:05:21.547475+0000","last_became_active":"2023-05-22T15:26:44.978443+0000","last_became_peered":"2023-05-22T15:26:44.978443+0000","last_unstale":"2023-05-22T16:05:21.547475+0000","last_undegraded":"2023-05-22T16:05:21.547475+0000","last_fullsized":"2023-05-22T16:05:21.547475+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:46:29.290243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.1c","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2023-05-22T16:05:21.987202+0000","last_change":"2023-05-22T16:05:21.960112+0000","last_active":"2023-05-22T16:05:21.987202+0000","last_peered":"2023-05-22T16:05:21.987202+0000","last_clean":"2023-05-22T16:05:21.987202+0000","last_became_active":"2023-05-22T16:05:21.959360+0000","last_became_peered":"2023-05-22T16:05:21.959360+0000","last_unstale":"2023-05-22T16:05:21.987202+0000","last_undegraded":"2023-05-22T16:05:21.987202+0000","last_fullsized":"2023-05-22T16:05:21.987202+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_clean_scrub_stamp":"2023-05-22T16:05:21.519005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:21:04.462454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.1d","version":"195'2","reported_seq":12,"reported_epoch":195,"state":"active+clean","last_fresh":"2023-05-22T16:05:21.952541+0000","last_change":"2023-05-22T16:05:21.574994+0000","last_active":"2023-05-22T16:05:21.952541+0000","last_peered":"2023-05-22T16:05:21.952541+0000","last_clean":"2023-05-22T16:05:21.952541+0000","last_became_active":"2023-05-22T16:05:21.574678+0000","last_became_peered":"2023-05-22T16:05:21.574678+0000","last_unstale":"2023-05-22T16:05:21.952541+0000","last_undegraded":"2023-05-22T16:05:21.952541+0000","last_fullsized":"2023-05-22T16:05:21.952541+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:05:20.505200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:05:20.505200+0000","last_clean_scrub_stamp":"2023-05-22T16:05:20.505200+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:32:04.944239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":171,"reported_epoch":194,"state":"active+clean","last_fresh":"2023-05-22T16:05:21.564660+0000","last_change":"2023-05-22T15:26:44.982421+0000","last_active":"2023-05-22T16:05:21.564660+0000","last_peered":"2023-05-22T16:05:21.564660+0000","last_clean":"2023-05-22T16:05:21.564660+0000","last_became_active":"2023-05-22T15:26:44.981586+0000","last_became_peered":"2023-05-22T15:26:44.981586+0000","last_unstale":"2023-05-22T16:05:21.564660+0000","last_undegraded":"2023-05-22T16:05:21.564660+0000","last_fullsized":"2023-05-22T16:05:21.564660+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:14:11.156544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":25,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":24,"num_pg":32,"stat_sum":{"num_bytes":2330,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1940,"num_read_kb":4014,"num_write":539,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":326,"ondisk_log_size":326,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575307,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29072,"kb_used_data":2952,"kb_used_omap":0,"kb_used_meta":26112,"kb_avail":94342768,"statfs":{"total":96636764160,"available":96606994432,"internally_reserved":0,"allocated":3022848,"data_stored":1889474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":26738688},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 16:04:48 2023","interfaces":[{"interface":"back","average":{"1min":0.81100000000000005,"5min":0.86199999999999999,"15min":0.86299999999999999},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51100000000000001},"max":{"1min":1.0660000000000001,"5min":1.3069999999999999,"15min":1.3069999999999999},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.87,"5min":0.88,"15min":0.88},"min":{"1min":0.63200000000000001,"5min":0.501,"15min":0.501},"max":{"1min":1.1439999999999999,"5min":1.2529999999999999,"15min":1.2529999999999999},"last":0.72699999999999998}]},{"osd":1,"last update":"Mon May 22 16:04:48 2023","interfaces":[{"interface":"back","average":{"1min":0.98499999999999999,"5min":0.99099999999999999,"15min":0.99399999999999999},"min":{"1min":0.73199999999999998,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.2450000000000001,"5min":1.431,"15min":1.431},"last":0.93700000000000006},{"interface":"front","average":{"1min":1.0089999999999999,"5min":1.04,"15min":1.0349999999999999},"min":{"1min":0.71099999999999997,"5min":0.66300000000000003,"15min":0.626},"max":{"1min":1.262,"5min":1.345,"15min":1.345},"last":0.67000000000000004}]},{"osd":2,"last update":"Mon May 22 16:04:48 2023","interfaces":[{"interface":"back","average":{"1min":0.92400000000000004,"5min":0.92000000000000004,"15min":0.90900000000000003},"min":{"1min":0.65000000000000002,"5min":0.63200000000000001,"15min":0.63200000000000001},"max":{"1min":1.1910000000000001,"5min":1.286,"15min":1.407},"last":0.81499999999999995},{"interface":"front","average":{"1min":0.90900000000000003,"5min":0.96499999999999997,"15min":0.96299999999999997},"min":{"1min":0.69499999999999995,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":1.089,"5min":1.246,"15min":1.2829999999999999},"last":0.628}]},{"osd":3,"last update":"Mon May 22 16:04:48 2023","interfaces":[{"interface":"back","average":{"1min":0.90500000000000003,"5min":0.93000000000000005,"15min":0.94499999999999995},"min":{"1min":0.61099999999999999,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.252,"5min":1.2909999999999999,"15min":1.401},"last":0.78200000000000003},{"interface":"front","average":{"1min":0.92500000000000004,"5min":1.0009999999999999,"15min":0.98199999999999998},"min":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":1.286,"5min":1.411,"15min":1.411},"last":1.036}]},{"osd":4,"last update":"Mon May 22 16:04:48 2023","interfaces":[{"interface":"back","average":{"1min":0.86899999999999999,"5min":0.88,"15min":0.88700000000000001},"min":{"1min":0.57299999999999995,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":1.127,"5min":1.3440000000000001,"15min":1.3440000000000001},"last":0.85999999999999999},{"interface":"front","average":{"1min":0.95199999999999996,"5min":0.98499999999999999,"15min":0.96499999999999997},"min":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":1.1819999999999999,"5min":1.405,"15min":1.405},"last":0.88100000000000001}]},{"osd":5,"last update":"Mon May 22 16:04:48 2023","interfaces":[{"interface":"back","average":{"1min":1.002,"5min":0.98999999999999999,"15min":0.99299999999999999},"min":{"1min":0.66200000000000003,"5min":0.623,"15min":0.441},"max":{"1min":1.25,"5min":1.2909999999999999,"15min":1.3759999999999999},"last":1.129},{"interface":"front","average":{"1min":1.159,"5min":1.1339999999999999,"15min":1.131},"min":{"1min":0.89500000000000002,"5min":0.72299999999999998,"15min":0.59799999999999998},"max":{"1min":1.345,"5min":1.3839999999999999,"15min":1.413},"last":0.91700000000000004}]},{"osd":6,"last update":"Mon May 22 16:04:48 2023","interfaces":[{"interface":"back","average":{"1min":1.0489999999999999,"5min":1.0469999999999999,"15min":1.0580000000000001},"min":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":1.256,"5min":1.3169999999999999,"15min":1.389},"last":0.97899999999999998},{"interface":"front","average":{"1min":1.127,"5min":1.103,"15min":1.093},"min":{"1min":0.81699999999999995,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":1.3500000000000001,"5min":1.391,"15min":1.7050000000000001},"last":1.002}]}]},{"osd":6,"up_from":13,"seq":55834575308,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":22616,"kb_used_data":2384,"kb_used_omap":0,"kb_used_meta":20224,"kb_avail":94349224,"statfs":{"total":96636764160,"available":96613605376,"internally_reserved":0,"allocated":2441216,"data_stored":1306610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":20709376},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 16:04:55 2023","interfaces":[{"interface":"back","average":{"1min":0.93899999999999995,"5min":0.873,"15min":0.90300000000000002},"min":{"1min":0.65500000000000003,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":1.3089999999999999,"5min":1.3089999999999999,"15min":1.4399999999999999},"last":2.6469999999999998},{"interface":"front","average":{"1min":0.91200000000000003,"5min":0.89600000000000002,"15min":0.91200000000000003},"min":{"1min":0.621,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":1.2010000000000001,"5min":1.427,"15min":1.427},"last":6.7439999999999998}]},{"osd":1,"last update":"Mon May 22 16:04:55 2023","interfaces":[{"interface":"back","average":{"1min":0.94499999999999995,"5min":0.91700000000000004,"15min":0.93999999999999995},"min":{"1min":0.61899999999999999,"5min":0.55300000000000005,"15min":0.51100000000000001},"max":{"1min":1.2949999999999999,"5min":1.3100000000000001,"15min":1.3380000000000001},"last":2.3290000000000002},{"interface":"front","average":{"1min":0.92800000000000005,"5min":0.92300000000000004,"15min":0.93000000000000005},"min":{"1min":0.68600000000000005,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.1279999999999999,"5min":1.2849999999999999,"15min":1.2849999999999999},"last":6.5010000000000003}]},{"osd":2,"last update":"Mon May 22 16:04:55 2023","interfaces":[{"interface":"back","average":{"1min":0.98099999999999998,"5min":0.97799999999999998,"15min":0.996},"min":{"1min":0.63700000000000001,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":1.2589999999999999,"5min":1.423,"15min":1.423},"last":7.0170000000000003},{"interface":"front","average":{"1min":1.04,"5min":1.056,"15min":1.071},"min":{"1min":0.72799999999999998,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":1.242,"5min":1.6599999999999999,"15min":1.6599999999999999},"last":2.403}]},{"osd":3,"last update":"Mon May 22 16:04:55 2023","interfaces":[{"interface":"back","average":{"1min":1.0009999999999999,"5min":0.998,"15min":0.995},"min":{"1min":0.76400000000000001,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.266,"5min":1.51,"15min":1.51},"last":6.6699999999999999},{"interface":"front","average":{"1min":0.95399999999999996,"5min":0.96599999999999997,"15min":0.97199999999999998},"min":{"1min":0.76800000000000002,"5min":0.622,"15min":0.622},"max":{"1min":1.286,"5min":1.286,"15min":1.3779999999999999},"last":7.1260000000000003}]},{"osd":4,"last update":"Mon May 22 16:04:55 2023","interfaces":[{"interface":"back","average":{"1min":0.92700000000000005,"5min":0.92600000000000005,"15min":0.95599999999999996},"min":{"1min":0.63300000000000001,"5min":0.62,"15min":0.61499999999999999},"max":{"1min":1.3540000000000001,"5min":1.4550000000000001,"15min":1.5409999999999999},"last":4.0709999999999997},{"interface":"front","average":{"1min":1.0389999999999999,"5min":1.0600000000000001,"15min":1.04},"min":{"1min":0.67000000000000004,"5min":0.63300000000000001,"15min":0.622},"max":{"1min":1.278,"5min":1.589,"15min":1.589},"last":11.581}]},{"osd":5,"last update":"Mon May 22 16:04:55 2023","interfaces":[{"interface":"back","average":{"1min":0.96099999999999997,"5min":0.98999999999999999,"15min":0.98899999999999999},"min":{"1min":0.71199999999999997,"5min":0.64300000000000002,"15min":0.61799999999999999},"max":{"1min":1.363,"5min":1.397,"15min":1.458},"last":17.587},{"interface":"front","average":{"1min":1.0649999999999999,"5min":1.093,"15min":1.1200000000000001},"min":{"1min":0.78600000000000003,"5min":0.748,"15min":0.69699999999999995},"max":{"1min":1.425,"5min":2.0390000000000001,"15min":2.0390000000000001},"last":2.9620000000000002}]},{"osd":7,"last update":"Mon May 22 16:04:55 2023","interfaces":[{"interface":"back","average":{"1min":1.1579999999999999,"5min":1.139,"15min":1.1339999999999999},"min":{"1min":0.95199999999999996,"5min":0.64200000000000002,"15min":0.58599999999999997},"max":{"1min":1.3580000000000001,"5min":1.746,"15min":1.746},"last":6.8339999999999996},{"interface":"front","average":{"1min":1.0669999999999999,"5min":1.1220000000000001,"15min":1.133},"min":{"1min":0.84999999999999998,"5min":0.78800000000000003,"15min":0.746},"max":{"1min":1.325,"5min":1.605,"15min":1.669},"last":8.5990000000000002}]}]},{"osd":5,"up_from":13,"seq":55834575308,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32728,"kb_used_data":2384,"kb_used_omap":0,"kb_used_meta":30336,"kb_avail":94339112,"statfs":{"total":96636764160,"available":96603250688,"internally_reserved":0,"allocated":2441216,"data_stored":1306610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31064064},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 16:04:49 2023","interfaces":[{"interface":"back","average":{"1min":0.80900000000000005,"5min":0.877,"15min":0.88800000000000001},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.1180000000000001,"5min":1.278,"15min":1.3959999999999999},"last":0.47899999999999998},{"interface":"front","average":{"1min":1.044,"5min":0.95199999999999996,"15min":0.94599999999999995},"min":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.502},"max":{"1min":1.27,"5min":1.325,"15min":1.325},"last":0.95899999999999996}]},{"osd":1,"last update":"Mon May 22 16:04:49 2023","interfaces":[{"interface":"back","average":{"1min":0.96599999999999997,"5min":1,"15min":1.012},"min":{"1min":0.69299999999999995,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":1.234,"5min":1.365,"15min":1.365},"last":0.65200000000000002},{"interface":"front","average":{"1min":1.0149999999999999,"5min":1.0629999999999999,"15min":1.073},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":1.3220000000000001,"5min":1.3520000000000001,"15min":1.4630000000000001},"last":0.70599999999999996}]},{"osd":2,"last update":"Mon May 22 16:04:49 2023","interfaces":[{"interface":"back","average":{"1min":0.92900000000000005,"5min":0.90800000000000003,"15min":0.92500000000000004},"min":{"1min":0.48699999999999999,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.3160000000000001,"5min":1.3200000000000001,"15min":1.3740000000000001},"last":0.81000000000000005},{"interface":"front","average":{"1min":0.94699999999999995,"5min":0.97399999999999998,"15min":0.95299999999999996},"min":{"1min":0.65600000000000003,"5min":0.64700000000000002,"15min":0.64700000000000002},"max":{"1min":1.244,"5min":1.4350000000000001,"15min":1.4350000000000001},"last":0.92600000000000005}]},{"osd":3,"last update":"Mon May 22 16:04:49 2023","interfaces":[{"interface":"back","average":{"1min":0.95699999999999996,"5min":0.97399999999999998,"15min":0.97899999999999998},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":1.349,"5min":1.349,"15min":1.367},"last":0.88800000000000001},{"interface":"front","average":{"1min":1.0840000000000001,"5min":1.0429999999999999,"15min":1.054},"min":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.3899999999999999,"5min":1.3899999999999999,"15min":1.3899999999999999},"last":1.03}]},{"osd":4,"last update":"Mon May 22 16:04:49 2023","interfaces":[{"interface":"back","average":{"1min":0.96299999999999997,"5min":0.93500000000000005,"15min":0.93100000000000005},"min":{"1min":0.66900000000000004,"5min":0.61299999999999999,"15min":0.61299999999999999},"max":{"1min":1.429,"5min":1.6000000000000001,"15min":1.6000000000000001},"last":0.78000000000000003},{"interface":"front","average":{"1min":0.997,"5min":0.99299999999999999,"15min":1.016},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":1.409,"5min":1.4630000000000001,"15min":1.4630000000000001},"last":1.0529999999999999}]},{"osd":6,"last update":"Mon May 22 16:04:49 2023","interfaces":[{"interface":"back","average":{"1min":1.028,"5min":1.004,"15min":1.0189999999999999},"min":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.59799999999999998},"max":{"1min":1.4470000000000001,"5min":1.4630000000000001,"15min":1.4790000000000001},"last":1.145},{"interface":"front","average":{"1min":1.0820000000000001,"5min":1.083,"15min":1.0880000000000001},"min":{"1min":0.748,"5min":0.67300000000000004,"15min":0.67300000000000004},"max":{"1min":1.6080000000000001,"5min":1.6080000000000001,"15min":1.6080000000000001},"last":0.85499999999999998}]},{"osd":7,"last update":"Mon May 22 16:04:49 2023","interfaces":[{"interface":"back","average":{"1min":1.1220000000000001,"5min":1.1200000000000001,"15min":1.129},"min":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"max":{"1min":1.51,"5min":1.51,"15min":1.5109999999999999},"last":0.995},{"interface":"front","average":{"1min":1.2370000000000001,"5min":1.1890000000000001,"15min":1.1970000000000001},"min":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":1.6679999999999999,"5min":1.6679999999999999,"15min":1.7210000000000001},"last":1.353}]}]},{"osd":4,"up_from":13,"seq":55834575306,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35304,"kb_used_data":2400,"kb_used_omap":0,"kb_used_meta":32896,"kb_avail":94336536,"statfs":{"total":96636764160,"available":96600612864,"internally_reserved":0,"allocated":2457600,"data_stored":1308724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":33685504},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 16:05:02 2023","interfaces":[{"interface":"back","average":{"1min":0.90400000000000003,"5min":0.89600000000000002,"15min":0.90000000000000002},"min":{"1min":0.59999999999999998,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.1870000000000001,"5min":1.7210000000000001,"15min":1.7210000000000001},"last":1.1439999999999999},{"interface":"front","average":{"1min":0.995,"5min":0.96599999999999997,"15min":0.95199999999999996},"min":{"1min":0.70899999999999996,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":1.3360000000000001,"5min":1.7290000000000001,"15min":1.7290000000000001},"last":0.67700000000000005}]},{"osd":1,"last update":"Mon May 22 16:05:02 2023","interfaces":[{"interface":"back","average":{"1min":1.0269999999999999,"5min":0.97599999999999998,"15min":0.95899999999999996},"min":{"1min":0.76900000000000002,"5min":0.314,"15min":0.314},"max":{"1min":1.258,"5min":1.9339999999999999,"15min":1.9339999999999999},"last":0.76900000000000002},{"interface":"front","average":{"1min":0.90000000000000002,"5min":0.92800000000000005,"15min":0.93999999999999995},"min":{"1min":0.68799999999999994,"5min":0.504,"15min":0.504},"max":{"1min":1.0900000000000001,"5min":1.365,"15min":1.365},"last":0.89200000000000002}]},{"osd":2,"last update":"Mon May 22 16:05:02 2023","interfaces":[{"interface":"back","average":{"1min":0.98599999999999999,"5min":0.95499999999999996,"15min":0.94999999999999996},"min":{"1min":0.71799999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":1.28,"5min":1.4990000000000001,"15min":1.4990000000000001},"last":0.81000000000000005},{"interface":"front","average":{"1min":1.056,"5min":1.036,"15min":1.052},"min":{"1min":0.76000000000000001,"5min":0.44,"15min":0.44},"max":{"1min":1.2529999999999999,"5min":2.0249999999999999,"15min":2.0249999999999999},"last":1.0329999999999999}]},{"osd":3,"last update":"Mon May 22 16:05:02 2023","interfaces":[{"interface":"back","average":{"1min":1.0329999999999999,"5min":0.96499999999999997,"15min":0.96599999999999997},"min":{"1min":0.79200000000000004,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":1.258,"5min":1.4610000000000001,"15min":1.4610000000000001},"last":1.1890000000000001},{"interface":"front","average":{"1min":1.0760000000000001,"5min":1.0609999999999999,"15min":1.0489999999999999},"min":{"1min":0.83899999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":1.323,"5min":1.349,"15min":1.484},"last":1.278}]},{"osd":5,"last update":"Mon May 22 16:05:02 2023","interfaces":[{"interface":"back","average":{"1min":1.004,"5min":1.036,"15min":1.002},"min":{"1min":0.69599999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":1.325,"5min":2.1110000000000002,"15min":2.1110000000000002},"last":0.94299999999999995},{"interface":"front","average":{"1min":1.0309999999999999,"5min":1.0189999999999999,"15min":0.996},"min":{"1min":0.69999999999999996,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.1990000000000001,"5min":1.5449999999999999,"15min":1.5449999999999999},"last":1.0620000000000001}]},{"osd":6,"last update":"Mon May 22 16:05:02 2023","interfaces":[{"interface":"back","average":{"1min":1.0669999999999999,"5min":1.036,"15min":1.0009999999999999},"min":{"1min":0.75800000000000001,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":1.387,"5min":1.397,"15min":1.4339999999999999},"last":0.98299999999999998},{"interface":"front","average":{"1min":1.1599999999999999,"5min":1.1599999999999999,"15min":1.137},"min":{"1min":0.875,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.3340000000000001,"5min":2.2690000000000001,"15min":2.2690000000000001},"last":1.1160000000000001}]},{"osd":7,"last update":"Mon May 22 16:05:02 2023","interfaces":[{"interface":"back","average":{"1min":1.093,"5min":1.0960000000000001,"15min":1.069},"min":{"1min":0.83599999999999997,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":1.3029999999999999,"5min":1.623,"15min":1.623},"last":1.2370000000000001},{"interface":"front","average":{"1min":1.2150000000000001,"5min":1.1879999999999999,"15min":1.1599999999999999},"min":{"1min":1.0169999999999999,"5min":0.71199999999999997,"15min":0.68100000000000005},"max":{"1min":1.4390000000000001,"5min":1.448,"15min":1.5089999999999999},"last":1.367}]}]},{"osd":3,"up_from":13,"seq":55834575306,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":2396,"kb_used_omap":0,"kb_used_meta":24832,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":2453504,"data_stored":1306804,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":25427968},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 16:04:38 2023","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.63100000000000001,"15min":0.63600000000000001},"min":{"1min":0.374,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.91800000000000004,"5min":1.0049999999999999,"15min":1.0229999999999999},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.82599999999999996,"5min":0.77900000000000003,"15min":0.77400000000000002},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":1.7230000000000001,"5min":1.7230000000000001,"15min":1.7230000000000001},"last":0.50900000000000001}]},{"osd":1,"last update":"Mon May 22 16:04:38 2023","interfaces":[{"interface":"back","average":{"1min":0.81299999999999994,"5min":0.76800000000000002,"15min":0.72199999999999998},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":2.1280000000000001,"5min":2.4969999999999999,"15min":2.4969999999999999},"last":1.056},{"interface":"front","average":{"1min":0.79800000000000004,"5min":0.80600000000000005,"15min":0.80600000000000005},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":1.395,"5min":2.4769999999999999,"15min":2.4769999999999999},"last":0.77200000000000002}]},{"osd":2,"last update":"Mon May 22 16:04:38 2023","interfaces":[{"interface":"back","average":{"1min":0.88100000000000001,"5min":0.85399999999999998,"15min":0.85699999999999998},"min":{"1min":0.47699999999999998,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":2.2250000000000001,"5min":2.2250000000000001,"15min":2.2250000000000001},"last":0.67700000000000005},{"interface":"front","average":{"1min":0.84699999999999998,"5min":0.82699999999999996,"15min":0.83499999999999996},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":2.1680000000000001,"5min":2.1680000000000001,"15min":2.1680000000000001},"last":0.79800000000000004}]},{"osd":4,"last update":"Mon May 22 16:04:38 2023","interfaces":[{"interface":"back","average":{"1min":1.042,"5min":0.98499999999999999,"15min":0.98099999999999998},"min":{"1min":0.69299999999999995,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":2.3479999999999999,"5min":2.3479999999999999,"15min":2.3479999999999999},"last":1.226},{"interface":"front","average":{"1min":1.0349999999999999,"5min":1.0229999999999999,"15min":1.034},"min":{"1min":0.54300000000000004,"5min":0.37,"15min":0.37},"max":{"1min":1.788,"5min":1.788,"15min":1.788},"last":0.83099999999999996}]},{"osd":5,"last update":"Mon May 22 16:04:38 2023","interfaces":[{"interface":"back","average":{"1min":1.036,"5min":1.054,"15min":1.0549999999999999},"min":{"1min":0.48699999999999999,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":1.8129999999999999,"5min":1.8129999999999999,"15min":1.8129999999999999},"last":1.1870000000000001},{"interface":"front","average":{"1min":1,"5min":1.038,"15min":1.0469999999999999},"min":{"1min":0.57899999999999996,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":1.347,"5min":1.571,"15min":1.571},"last":0.95599999999999996}]},{"osd":6,"last update":"Mon May 22 16:04:38 2023","interfaces":[{"interface":"back","average":{"1min":1.0820000000000001,"5min":1.093,"15min":1.095},"min":{"1min":0.57099999999999995,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":1.76,"5min":1.76,"15min":1.76},"last":1.274},{"interface":"front","average":{"1min":1.0720000000000001,"5min":1.0940000000000001,"15min":1.105},"min":{"1min":0.64200000000000002,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":1.4930000000000001,"5min":1.754,"15min":1.754},"last":1.107}]},{"osd":7,"last update":"Mon May 22 16:04:38 2023","interfaces":[{"interface":"back","average":{"1min":1.1040000000000001,"5min":1.1240000000000001,"15min":1.133},"min":{"1min":0.60399999999999998,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.421,"5min":1.673,"15min":1.673},"last":1.2929999999999999},{"interface":"front","average":{"1min":1.1459999999999999,"5min":1.1240000000000001,"15min":1.115},"min":{"1min":0.79600000000000004,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":2.3079999999999998,"5min":2.3079999999999998,"15min":2.3079999999999998},"last":1.3839999999999999}]}]},{"osd":2,"up_from":13,"seq":55834575307,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31268,"kb_used_data":2396,"kb_used_omap":0,"kb_used_meta":28864,"kb_avail":94340572,"statfs":{"total":96636764160,"available":96604745728,"internally_reserved":0,"allocated":2453504,"data_stored":1307288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":29556736},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 16:04:32 2023","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.64800000000000002,"15min":0.67600000000000005},"min":{"1min":0.38,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.85999999999999999,"5min":1.1180000000000001,"15min":1.1519999999999999},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.745,"5min":0.74299999999999999,"15min":0.752},"min":{"1min":0.495,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":1.1659999999999999,"5min":1.1659999999999999,"15min":1.2090000000000001},"last":0.52700000000000002}]},{"osd":1,"last update":"Mon May 22 16:04:32 2023","interfaces":[{"interface":"back","average":{"1min":0.72199999999999998,"5min":0.75600000000000001,"15min":0.75900000000000001},"min":{"1min":0.42599999999999999,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":1.069,"5min":1.0800000000000001,"15min":1.099},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.80700000000000005,"5min":0.83499999999999996,"15min":0.82399999999999995},"min":{"1min":0.39200000000000002,"5min":0.311,"15min":0.311},"max":{"1min":1.099,"5min":1.1739999999999999,"15min":1.2589999999999999},"last":0.67600000000000005}]},{"osd":3,"last update":"Mon May 22 16:04:32 2023","interfaces":[{"interface":"back","average":{"1min":0.73499999999999999,"5min":0.76700000000000002,"15min":0.76000000000000001},"min":{"1min":0.36799999999999999,"5min":0.315,"15min":0.315},"max":{"1min":1.046,"5min":1.3540000000000001,"15min":1.3540000000000001},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.875,"5min":0.872,"15min":0.89800000000000002},"min":{"1min":0.45400000000000001,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":1.2949999999999999,"5min":1.2949999999999999,"15min":1.2949999999999999},"last":0.50800000000000001}]},{"osd":4,"last update":"Mon May 22 16:04:32 2023","interfaces":[{"interface":"back","average":{"1min":1.032,"5min":1.0269999999999999,"15min":1.0409999999999999},"min":{"1min":0.59099999999999997,"5min":0.441,"15min":0.441},"max":{"1min":1.516,"5min":1.516,"15min":1.6539999999999999},"last":0.70599999999999996},{"interface":"front","average":{"1min":1.036,"5min":1.0209999999999999,"15min":1.0429999999999999},"min":{"1min":0.76500000000000001,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.4299999999999999,"5min":1.4299999999999999,"15min":1.538},"last":0.89700000000000002}]},{"osd":5,"last update":"Mon May 22 16:04:32 2023","interfaces":[{"interface":"back","average":{"1min":1.024,"5min":1.026,"15min":1.0580000000000001},"min":{"1min":0.625,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":1.2829999999999999,"5min":1.3300000000000001,"15min":1.605},"last":1.0129999999999999},{"interface":"front","average":{"1min":1.0249999999999999,"5min":1.0269999999999999,"15min":1.034},"min":{"1min":0.78400000000000003,"5min":0.67700000000000005,"15min":0.67700000000000005},"max":{"1min":1.534,"5min":1.534,"15min":1.534},"last":0.79800000000000004}]},{"osd":6,"last update":"Mon May 22 16:04:32 2023","interfaces":[{"interface":"back","average":{"1min":1.0469999999999999,"5min":1.04,"15min":1.03},"min":{"1min":0.64400000000000002,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":1.4219999999999999,"5min":1.429,"15min":1.462},"last":0.96299999999999997},{"interface":"front","average":{"1min":1.1100000000000001,"5min":1.0980000000000001,"15min":1.135},"min":{"1min":0.69999999999999996,"5min":0.57099999999999995,"15min":0.57099999999999995},"max":{"1min":1.621,"5min":1.621,"15min":1.7110000000000001},"last":0.98899999999999999}]},{"osd":7,"last update":"Mon May 22 16:04:32 2023","interfaces":[{"interface":"back","average":{"1min":1.1279999999999999,"5min":1.095,"15min":1.1259999999999999},"min":{"1min":0.77600000000000002,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":1.393,"5min":1.466,"15min":1.6399999999999999},"last":1.052},{"interface":"front","average":{"1min":1.083,"5min":1.103,"15min":1.117},"min":{"1min":0.60999999999999999,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":1.3440000000000001,"5min":1.673,"15min":1.673},"last":0.872}]}]},{"osd":1,"up_from":13,"seq":55834575307,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37092,"kb_used_data":2396,"kb_used_omap":0,"kb_used_meta":34688,"kb_avail":94334748,"statfs":{"total":96636764160,"available":96598781952,"internally_reserved":0,"allocated":2453504,"data_stored":1308158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":35520512},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 16:04:55 2023","interfaces":[{"interface":"back","average":{"1min":0.71799999999999997,"5min":0.65500000000000003,"15min":0.66300000000000003},"min":{"1min":0.497,"5min":0.308,"15min":0.308},"max":{"1min":0.96099999999999997,"5min":1.151,"15min":1.329},"last":0.71099999999999997},{"interface":"front","average":{"1min":0.755,"5min":0.74399999999999999,"15min":0.76800000000000002},"min":{"1min":0.56299999999999994,"5min":0.502,"15min":0.502},"max":{"1min":1.25,"5min":1.25,"15min":1.268},"last":0.68200000000000005}]},{"osd":2,"last update":"Mon May 22 16:04:55 2023","interfaces":[{"interface":"back","average":{"1min":0.72399999999999998,"5min":0.74399999999999999,"15min":0.75800000000000001},"min":{"1min":0.44800000000000001,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":1.0700000000000001,"5min":1.0880000000000001,"15min":1.26},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.81100000000000005,"5min":0.77300000000000002,"15min":0.78200000000000003},"min":{"1min":0.65600000000000003,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":1.0660000000000001,"5min":1.1299999999999999,"15min":1.268},"last":0.77500000000000002}]},{"osd":3,"last update":"Mon May 22 16:04:55 2023","interfaces":[{"interface":"back","average":{"1min":0.86499999999999999,"5min":0.86299999999999999,"15min":0.88300000000000001},"min":{"1min":0.60399999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":1.1859999999999999,"5min":1.2829999999999999,"15min":1.373},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.86499999999999999,"5min":0.82999999999999996,"15min":0.82899999999999996},"min":{"1min":0.57099999999999995,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":1.171,"5min":1.2749999999999999,"15min":1.2749999999999999},"last":0.74199999999999999}]},{"osd":4,"last update":"Mon May 22 16:04:55 2023","interfaces":[{"interface":"back","average":{"1min":0.97999999999999998,"5min":0.97499999999999998,"15min":0.98599999999999999},"min":{"1min":0.73299999999999998,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":1.155,"5min":1.542,"15min":1.542},"last":0.81999999999999995},{"interface":"front","average":{"1min":0.95999999999999996,"5min":1.006,"15min":1.026},"min":{"1min":0.76800000000000002,"5min":0.434,"15min":0.434},"max":{"1min":1.153,"5min":1.327,"15min":1.4650000000000001},"last":1.264}]},{"osd":5,"last update":"Mon May 22 16:04:55 2023","interfaces":[{"interface":"back","average":{"1min":1.0029999999999999,"5min":1.04,"15min":1.05},"min":{"1min":0.81299999999999994,"5min":0.70399999999999996,"15min":0.70399999999999996},"max":{"1min":1.224,"5min":1.605,"15min":1.605},"last":1.3069999999999999},{"interface":"front","average":{"1min":1.093,"5min":1.0669999999999999,"15min":1.073},"min":{"1min":0.81299999999999994,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":1.3380000000000001,"5min":1.3919999999999999,"15min":1.5389999999999999},"last":0.86099999999999999}]},{"osd":6,"last update":"Mon May 22 16:04:55 2023","interfaces":[{"interface":"back","average":{"1min":1.0629999999999999,"5min":1.054,"15min":1.0669999999999999},"min":{"1min":0.92900000000000005,"5min":0.56799999999999995,"15min":0.56799999999999995},"max":{"1min":1.167,"5min":1.371,"15min":1.5249999999999999},"last":1.3640000000000001},{"interface":"front","average":{"1min":1.069,"5min":1.105,"15min":1.113},"min":{"1min":0.88100000000000001,"5min":0.77600000000000002,"15min":0.75700000000000001},"max":{"1min":1.304,"5min":1.4359999999999999,"15min":1.6479999999999999},"last":1.242}]},{"osd":7,"last update":"Mon May 22 16:04:55 2023","interfaces":[{"interface":"back","average":{"1min":1.1220000000000001,"5min":1.147,"15min":1.159},"min":{"1min":0.92900000000000005,"5min":0.72199999999999998,"15min":0.72199999999999998},"max":{"1min":1.3819999999999999,"5min":1.7030000000000001,"15min":1.7030000000000001},"last":1.1799999999999999},{"interface":"front","average":{"1min":1.131,"5min":1.1339999999999999,"15min":1.1479999999999999},"min":{"1min":0.82699999999999996,"5min":0.73599999999999999,"15min":0.73599999999999999},"max":{"1min":1.395,"5min":1.6439999999999999,"15min":1.6439999999999999},"last":1.036}]}]},{"osd":0,"up_from":13,"seq":55834575308,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32544,"kb_used_data":2968,"kb_used_omap":0,"kb_used_meta":29568,"kb_avail":94339296,"statfs":{"total":96636764160,"available":96603439104,"internally_reserved":0,"allocated":3039232,"data_stored":1897000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":30277632},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon May 22 16:04:59 2023","interfaces":[{"interface":"back","average":{"1min":0.754,"5min":0.71299999999999997,"15min":0.71099999999999997},"min":{"1min":0.55600000000000005,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":1.075,"5min":1.296,"15min":1.296},"last":0.70799999999999996},{"interface":"front","average":{"1min":0.81000000000000005,"5min":0.76200000000000001,"15min":0.75},"min":{"1min":0.53500000000000003,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.0840000000000001,"5min":1.181,"15min":1.3149999999999999},"last":0.80100000000000005}]},{"osd":2,"last update":"Mon May 22 16:04:59 2023","interfaces":[{"interface":"back","average":{"1min":0.81799999999999995,"5min":0.79800000000000004,"15min":0.78100000000000003},"min":{"1min":0.59799999999999998,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":1.2150000000000001,"5min":1.22,"15min":1.284},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.78900000000000003,"5min":0.81899999999999995,"15min":0.81799999999999995},"min":{"1min":0.58199999999999996,"5min":0.437,"15min":0.437},"max":{"1min":1.0009999999999999,"5min":1.323,"15min":1.323},"last":0.92700000000000005}]},{"osd":3,"last update":"Mon May 22 16:04:59 2023","interfaces":[{"interface":"back","average":{"1min":0.79000000000000004,"5min":0.77000000000000002,"15min":0.77800000000000002},"min":{"1min":0.54800000000000004,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.1859999999999999,"5min":1.2350000000000001,"15min":1.2350000000000001},"last":0.74299999999999999},{"interface":"front","average":{"1min":0.84599999999999997,"5min":0.879,"15min":0.88100000000000001},"min":{"1min":0.66800000000000004,"5min":0.377,"15min":0.377},"max":{"1min":1.0860000000000001,"5min":1.2949999999999999,"15min":1.331},"last":0.96199999999999997}]},{"osd":4,"last update":"Mon May 22 16:04:59 2023","interfaces":[{"interface":"back","average":{"1min":0.94799999999999995,"5min":0.95599999999999996,"15min":0.95699999999999996},"min":{"1min":0.76600000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"max":{"1min":1.1579999999999999,"5min":1.264,"15min":1.4059999999999999},"last":0.98599999999999999},{"interface":"front","average":{"1min":1.0960000000000001,"5min":1.073,"15min":1.0569999999999999},"min":{"1min":0.90500000000000003,"5min":0.63900000000000001,"15min":0.63900000000000001},"max":{"1min":1.343,"5min":1.401,"15min":1.482},"last":1.1200000000000001}]},{"osd":5,"last update":"Mon May 22 16:04:59 2023","interfaces":[{"interface":"back","average":{"1min":1.0349999999999999,"5min":1.0329999999999999,"15min":1.0429999999999999},"min":{"1min":0.79100000000000004,"5min":0.64600000000000002,"15min":0.64600000000000002},"max":{"1min":1.2190000000000001,"5min":1.325,"15min":1.6419999999999999},"last":1.0940000000000001},{"interface":"front","average":{"1min":1.0609999999999999,"5min":1.032,"15min":1.02},"min":{"1min":0.83799999999999997,"5min":0.66200000000000003,"15min":0.66200000000000003},"max":{"1min":1.252,"5min":1.413,"15min":1.458},"last":0.89400000000000002}]},{"osd":6,"last update":"Mon May 22 16:04:59 2023","interfaces":[{"interface":"back","average":{"1min":1.129,"5min":1.1160000000000001,"15min":1.117},"min":{"1min":0.88500000000000001,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":1.401,"5min":1.407,"15min":1.77},"last":0.85199999999999998},{"interface":"front","average":{"1min":1.141,"5min":1.1319999999999999,"15min":1.111},"min":{"1min":0.94399999999999995,"5min":0.68300000000000005,"15min":0.68300000000000005},"max":{"1min":1.5409999999999999,"5min":1.5409999999999999,"15min":1.5409999999999999},"last":1.0089999999999999}]},{"osd":7,"last update":"Mon May 22 16:04:59 2023","interfaces":[{"interface":"back","average":{"1min":1.2110000000000001,"5min":1.151,"15min":1.141},"min":{"1min":1.0309999999999999,"5min":0.63900000000000001,"15min":0.63900000000000001},"max":{"1min":1.6020000000000001,"5min":1.6020000000000001,"15min":1.6020000000000001},"last":1.145},{"interface":"front","average":{"1min":1.131,"5min":1.0960000000000001,"15min":1.1040000000000001},"min":{"1min":0.77400000000000002,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.4119999999999999,"5min":1.4650000000000001,"15min":1.4990000000000001},"last":1.0549999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2114,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-05-22T16:05:25.975 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-05-22T16:05:26.134 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.132+0000 7fdc8ea10640 1 Processor -- start 2023-05-22T16:05:26.134 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.132+0000 7fdc8ea10640 1 -- start start 2023-05-22T16:05:26.135 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.132+0000 7fdc8ea10640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fdc88070fb0 0x7fdc880cd1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:26.135 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.132+0000 7fdc8ea10640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc880cd8d0 con 0x7fdc88160400 2023-05-22T16:05:26.135 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.132+0000 7fdc8ea10640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc88164de0 con 0x7fdc880afb40 2023-05-22T16:05:26.135 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.132+0000 7fdc8ea10640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc880ca400 con 0x7fdc88070fb0 2023-05-22T16:05:26.136 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.132+0000 7fdc8cfad640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fdc88160400 0x7fdc880cddb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:48500/0 (socket says 172.21.15.16:48500) 2023-05-22T16:05:26.136 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.132+0000 7fdc8cfad640 1 -- 172.21.15.16:0/875200853 learned_addr learned my addr 172.21.15.16:0/875200853 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T16:05:26.138 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.132+0000 7fdc86ffd640 1 -- 172.21.15.16:0/875200853 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 438592460 0 0) 0x7fdc880cd8d0 con 0x7fdc88160400 2023-05-22T16:05:26.138 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.132+0000 7fdc87fff640 1 --2- 172.21.15.16:0/875200853 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fdc88070fb0 0x7fdc880cd1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:26.138 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.132+0000 7fdc86ffd640 1 -- 172.21.15.16:0/875200853 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdc70003530 con 0x7fdc88160400 2023-05-22T16:05:26.138 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.132+0000 7fdc86ffd640 1 -- 172.21.15.16:0/875200853 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 360420614 0 0) 0x7fdc70003530 con 0x7fdc88160400 2023-05-22T16:05:26.138 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.132+0000 7fdc86ffd640 1 -- 172.21.15.16:0/875200853 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fdc88070fb0 msgr2=0x7fdc880cd1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:05:26.138 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.132+0000 7fdc86ffd640 1 --2- 172.21.15.16:0/875200853 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fdc88070fb0 0x7fdc880cd1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:26.138 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.132+0000 7fdc86ffd640 1 -- 172.21.15.16:0/875200853 >> v1:172.21.15.62:6789/0 conn(0x7fdc880afb40 legacy=0x7fdc8815fcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:26.139 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.132+0000 7fdc86ffd640 1 -- 172.21.15.16:0/875200853 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc880ca6e0 con 0x7fdc88160400 2023-05-22T16:05:26.139 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.132+0000 7fdc86ffd640 1 -- 172.21.15.16:0/875200853 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7fdc80001a00 con 0x7fdc88160400 2023-05-22T16:05:26.139 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.132+0000 7fdc86ffd640 1 -- 172.21.15.16:0/875200853 >> v1:172.21.15.16:6789/0 conn(0x7fdc88160400 legacy=0x7fdc880cddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:26.139 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.132+0000 7fdc86ffd640 1 --2- 172.21.15.16:0/875200853 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdc700039b0 0x7fdc70003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:26.139 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.132+0000 7fdc86ffd640 1 --2- 172.21.15.16:0/875200853 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fdc88070fb0 0x7fdc700047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:26.139 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.132+0000 7fdc86ffd640 1 --2- 172.21.15.16:0/875200853 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdc70004d10 0x7fdc70005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:26.139 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.132+0000 7fdc86ffd640 1 -- 172.21.15.16:0/875200853 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc880ca400 con 0x7fdc700039b0 2023-05-22T16:05:26.140 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.132+0000 7fdc86ffd640 1 -- 172.21.15.16:0/875200853 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc7000baa0 con 0x7fdc70004d10 2023-05-22T16:05:26.140 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.132+0000 7fdc86ffd640 1 -- 172.21.15.16:0/875200853 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc7000bc60 con 0x7fdc88070fb0 2023-05-22T16:05:26.140 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.132+0000 7fdc87fff640 1 --2- 172.21.15.16:0/875200853 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdc700039b0 0x7fdc70003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:26.140 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.136+0000 7fdc877fe640 1 --2- 172.21.15.16:0/875200853 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fdc88070fb0 0x7fdc700047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:26.140 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.136+0000 7fdc8cfad640 1 --2- 172.21.15.16:0/875200853 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdc70004d10 0x7fdc70005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:26.140 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.136+0000 7fdc87fff640 1 -- 172.21.15.16:0/875200853 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fdc88070fb0 msgr2=0x7fdc700047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:05:26.140 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.136+0000 7fdc87fff640 1 --2- 172.21.15.16:0/875200853 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fdc88070fb0 0x7fdc700047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:26.140 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.136+0000 7fdc87fff640 1 -- 172.21.15.16:0/875200853 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdc70004d10 msgr2=0x7fdc70005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:05:26.141 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.136+0000 7fdc87fff640 1 --2- 172.21.15.16:0/875200853 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdc70004d10 0x7fdc70005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:26.141 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.136+0000 7fdc87fff640 1 -- 172.21.15.16:0/875200853 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc70003530 con 0x7fdc700039b0 2023-05-22T16:05:26.141 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.136+0000 7fdc8cfad640 1 --2- 172.21.15.16:0/875200853 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdc70004d10 0x7fdc70005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T16:05:26.141 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.136+0000 7fdc877fe640 1 --2- 172.21.15.16:0/875200853 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fdc88070fb0 0x7fdc700047d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T16:05:26.143 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.136+0000 7fdc87fff640 1 --2- 172.21.15.16:0/875200853 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdc700039b0 0x7fdc70003dc0 secure :-1 s=READY pgs=2451 cs=0 l=1 rev1=1 crypto rx=0x7fdc7c00ba30 tx=0x7fdc7c01c640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16be53bc77de02f8 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:26.144 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.136+0000 7fdc86ffd640 1 -- 172.21.15.16:0/875200853 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc7c0a9660 con 0x7fdc700039b0 2023-05-22T16:05:26.144 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.136+0000 7fdc86ffd640 1 -- 172.21.15.16:0/875200853 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc7c0a97c0 con 0x7fdc700039b0 2023-05-22T16:05:26.144 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.136+0000 7fdc86ffd640 1 -- 172.21.15.16:0/875200853 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc7c0a9b70 con 0x7fdc700039b0 2023-05-22T16:05:26.144 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.136+0000 7fdc8ea10640 1 -- 172.21.15.16:0/875200853 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdc700039b0 msgr2=0x7fdc70003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:26.144 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.136+0000 7fdc8ea10640 1 --2- 172.21.15.16:0/875200853 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdc700039b0 0x7fdc70003dc0 secure :-1 s=READY pgs=2451 cs=0 l=1 rev1=1 crypto rx=0x7fdc7c00ba30 tx=0x7fdc7c01c640 comp rx=0 tx=0).stop 2023-05-22T16:05:26.144 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.136+0000 7fdc8cfad640 1 -- 172.21.15.16:0/875200853 reap_dead start 2023-05-22T16:05:26.144 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.136+0000 7fdc8ea10640 1 -- 172.21.15.16:0/875200853 shutdown_connections 2023-05-22T16:05:26.145 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.136+0000 7fdc8ea10640 1 -- 172.21.15.16:0/875200853 >> 172.21.15.16:0/875200853 conn(0x7fdc88069720 msgr2=0x7fdc88069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:05:26.145 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.136+0000 7fdc8ea10640 1 -- 172.21.15.16:0/875200853 shutdown_connections 2023-05-22T16:05:26.145 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.136+0000 7fdc8ea10640 1 -- 172.21.15.16:0/875200853 wait complete. 2023-05-22T16:05:26.145 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.140+0000 7fdc8ea10640 1 Processor -- start 2023-05-22T16:05:26.145 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.140+0000 7fdc8ea10640 1 -- start start 2023-05-22T16:05:26.145 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.140+0000 7fdc8ea10640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fdc700039b0 0x7fdc880b3080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:26.145 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.140+0000 7fdc8ea10640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdc8814e740 0x7fdc880b35c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:26.145 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.140+0000 7fdc8ea10640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdc880c1a40 0x7fdc880c1eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:26.146 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.140+0000 7fdc8ea10640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc880c3f20 con 0x7fdc880c1a40 2023-05-22T16:05:26.146 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.140+0000 7fdc8ea10640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc880c23f0 con 0x7fdc8814e740 2023-05-22T16:05:26.146 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.140+0000 7fdc8ea10640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc880c2660 con 0x7fdc700039b0 2023-05-22T16:05:26.146 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.140+0000 7fdc8cfad640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdc880c1a40 0x7fdc880c1eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:26.146 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.140+0000 7fdc8cfad640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdc880c1a40 0x7fdc880c1eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41738/0 (socket says 172.21.15.16:41738) 2023-05-22T16:05:26.146 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.140+0000 7fdc8cfad640 1 -- 172.21.15.16:0/1537566114 learned_addr learned my addr 172.21.15.16:0/1537566114 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:05:26.146 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.140+0000 7fdc877fe640 1 --2- 172.21.15.16:0/1537566114 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdc8814e740 0x7fdc880b35c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:26.147 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.140+0000 7fdc87fff640 1 --2- 172.21.15.16:0/1537566114 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fdc700039b0 0x7fdc880b3080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:26.147 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.140+0000 7fdc8cfad640 1 -- 172.21.15.16:0/1537566114 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fdc700039b0 msgr2=0x7fdc880b3080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:26.147 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.140+0000 7fdc8cfad640 1 --2- 172.21.15.16:0/1537566114 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fdc700039b0 0x7fdc880b3080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:26.147 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.140+0000 7fdc8cfad640 1 -- 172.21.15.16:0/1537566114 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdc8814e740 msgr2=0x7fdc880b35c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:26.147 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.140+0000 7fdc8cfad640 1 --2- 172.21.15.16:0/1537566114 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdc8814e740 0x7fdc880b35c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:26.147 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.140+0000 7fdc8cfad640 1 -- 172.21.15.16:0/1537566114 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc880ca040 con 0x7fdc880c1a40 2023-05-22T16:05:26.147 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.140+0000 7fdc87fff640 1 --2- 172.21.15.16:0/1537566114 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fdc700039b0 0x7fdc880b3080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T16:05:26.147 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.140+0000 7fdc8cfad640 1 --2- 172.21.15.16:0/1537566114 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdc880c1a40 0x7fdc880c1eb0 secure :-1 s=READY pgs=2452 cs=0 l=1 rev1=1 crypto rx=0x7fdc8806f3e0 tx=0x7fdc8000dce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:26.148 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.140+0000 7fdc84ff9640 1 -- 172.21.15.16:0/1537566114 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc80007070 con 0x7fdc880c1a40 2023-05-22T16:05:26.148 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.140+0000 7fdc84ff9640 1 -- 172.21.15.16:0/1537566114 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc800071d0 con 0x7fdc880c1a40 2023-05-22T16:05:26.148 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.140+0000 7fdc8ea10640 1 -- 172.21.15.16:0/1537566114 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc88160820 con 0x7fdc880c1a40 2023-05-22T16:05:26.148 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.140+0000 7fdc84ff9640 1 -- 172.21.15.16:0/1537566114 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc80007580 con 0x7fdc880c1a40 2023-05-22T16:05:26.148 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.140+0000 7fdc8ea10640 1 -- 172.21.15.16:0/1537566114 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc880ca1d0 con 0x7fdc880c1a40 2023-05-22T16:05:26.148 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.140+0000 7fdc8ea10640 1 -- 172.21.15.16:0/1537566114 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc881629f0 con 0x7fdc880c1a40 2023-05-22T16:05:26.148 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.140+0000 7fdc84ff9640 1 -- 172.21.15.16:0/1537566114 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 92) v1 ==== 84332+0+0 (secure 0 0 0) 0x7fdc800076e0 con 0x7fdc880c1a40 2023-05-22T16:05:26.149 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.140+0000 7fdc84ff9640 1 --2- 172.21.15.16:0/1537566114 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fdc740654a0 0x7fdc74067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:26.149 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.140+0000 7fdc84ff9640 1 -- 172.21.15.16:0/1537566114 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(196..196 src has 1..196) v4 ==== 6438+0+0 (secure 0 0 0) 0x7fdc800827a0 con 0x7fdc880c1a40 2023-05-22T16:05:26.149 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.140+0000 7fdc87fff640 1 --2- 172.21.15.16:0/1537566114 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fdc740654a0 0x7fdc74067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:26.150 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.144+0000 7fdc87fff640 1 --2- 172.21.15.16:0/1537566114 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fdc740654a0 0x7fdc74067950 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fdc7c0b0000 tx=0x7fdc7c0ac040 comp rx=0 tx=0).ready entity=mgr.10542 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:26.150 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.144+0000 7fdc84ff9640 1 -- 172.21.15.16:0/1537566114 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fdc8004ffe0 con 0x7fdc880c1a40 2023-05-22T16:05:26.293 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.292+0000 7fdc8ea10640 1 -- 172.21.15.16:0/1537566114 --> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fdc8804dbe0 con 0x7fdc740654a0 2023-05-22T16:05:26.317 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.312+0000 7fdc84ff9640 1 -- 172.21.15.16:0/1537566114 <== mgr.10542 v2:172.21.15.16:6832/65890 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+309683 (secure 0 0 0) 0x7fdc8804dbe0 con 0x7fdc740654a0 2023-05-22T16:05:26.319 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T16:05:26.320 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2023-05-22T16:05:26.322 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.316+0000 7fdc8ea10640 1 -- 172.21.15.16:0/1537566114 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fdc740654a0 msgr2=0x7fdc74067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:26.322 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.316+0000 7fdc8ea10640 1 --2- 172.21.15.16:0/1537566114 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fdc740654a0 0x7fdc74067950 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fdc7c0b0000 tx=0x7fdc7c0ac040 comp rx=0 tx=0).stop 2023-05-22T16:05:26.322 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.316+0000 7fdc8ea10640 1 -- 172.21.15.16:0/1537566114 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdc880c1a40 msgr2=0x7fdc880c1eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:26.322 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.316+0000 7fdc8ea10640 1 --2- 172.21.15.16:0/1537566114 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdc880c1a40 0x7fdc880c1eb0 secure :-1 s=READY pgs=2452 cs=0 l=1 rev1=1 crypto rx=0x7fdc8806f3e0 tx=0x7fdc8000dce0 comp rx=0 tx=0).stop 2023-05-22T16:05:26.322 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.316+0000 7fdc8ea10640 1 -- 172.21.15.16:0/1537566114 shutdown_connections 2023-05-22T16:05:26.322 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.316+0000 7fdc8ea10640 1 --2- 172.21.15.16:0/1537566114 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fdc740654a0 0x7fdc74067950 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:26.323 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.316+0000 7fdc8ea10640 1 --2- 172.21.15.16:0/1537566114 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdc880c1a40 0x7fdc880c1eb0 unknown :-1 s=CLOSED pgs=2452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:26.323 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.316+0000 7fdc8ea10640 1 --2- 172.21.15.16:0/1537566114 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdc8814e740 0x7fdc880b35c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:26.323 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.316+0000 7fdc8ea10640 1 --2- 172.21.15.16:0/1537566114 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fdc700039b0 0x7fdc880b3080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:26.323 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.316+0000 7fdc8ea10640 1 -- 172.21.15.16:0/1537566114 >> 172.21.15.16:0/1537566114 conn(0x7fdc88069720 msgr2=0x7fdc880cdcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:05:26.324 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.316+0000 7fdc8ea10640 1 -- 172.21.15.16:0/1537566114 shutdown_connections 2023-05-22T16:05:26.324 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.316+0000 7fdc8ea10640 1 -- 172.21.15.16:0/1537566114 wait complete. 2023-05-22T16:05:26.336 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":25,"stamp":"2023-05-22T16:05:24.821335+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592698,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1940,"num_read_kb":4014,"num_write":583,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":364,"ondisk_log_size":364,"up":210,"acting":210,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":111,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":247860,"kb_used_data":20276,"kb_used_omap":0,"kb_used_meta":227520,"kb_avail":754726860,"statfs":{"total":773094113280,"available":772840304640,"internally_reserved":0,"allocated":20762624,"data_stored":11630668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":232980480},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":75,"apply_latency_ms":75,"commit_latency_ns":75000000,"apply_latency_ns":75000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2330,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.301670"},"pg_stats":[{"pgid":"25.3f","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2023-05-22T16:05:22.004814+0000","last_change":"2023-05-22T16:05:21.996989+0000","last_active":"2023-05-22T16:05:22.004814+0000","last_peered":"2023-05-22T16:05:22.004814+0000","last_clean":"2023-05-22T16:05:22.004814+0000","last_became_active":"2023-05-22T16:05:21.996486+0000","last_became_peered":"2023-05-22T16:05:21.996486+0000","last_unstale":"2023-05-22T16:05:22.004814+0000","last_undegraded":"2023-05-22T16:05:22.004814+0000","last_fullsized":"2023-05-22T16:05:22.004814+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_clean_scrub_stamp":"2023-05-22T16:05:21.519005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:17:20.015836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.3e","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2023-05-22T16:05:21.983740+0000","last_change":"2023-05-22T16:05:21.959893+0000","last_active":"2023-05-22T16:05:21.983740+0000","last_peered":"2023-05-22T16:05:21.983740+0000","last_clean":"2023-05-22T16:05:21.983740+0000","last_became_active":"2023-05-22T16:05:21.959111+0000","last_became_peered":"2023-05-22T16:05:21.959111+0000","last_unstale":"2023-05-22T16:05:21.983740+0000","last_undegraded":"2023-05-22T16:05:21.983740+0000","last_fullsized":"2023-05-22T16:05:21.983740+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_clean_scrub_stamp":"2023-05-22T16:05:21.519005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:10:02.644633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.3d","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2023-05-22T16:05:21.977760+0000","last_change":"2023-05-22T16:05:21.976957+0000","last_active":"2023-05-22T16:05:21.977760+0000","last_peered":"2023-05-22T16:05:21.977760+0000","last_clean":"2023-05-22T16:05:21.977760+0000","last_became_active":"2023-05-22T16:05:21.976584+0000","last_became_peered":"2023-05-22T16:05:21.976584+0000","last_unstale":"2023-05-22T16:05:21.977760+0000","last_undegraded":"2023-05-22T16:05:21.977760+0000","last_fullsized":"2023-05-22T16:05:21.977760+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_clean_scrub_stamp":"2023-05-22T16:05:21.519005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:22:21.749692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.3c","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2023-05-22T16:05:21.986529+0000","last_change":"2023-05-22T16:05:21.959819+0000","last_active":"2023-05-22T16:05:21.986529+0000","last_peered":"2023-05-22T16:05:21.986529+0000","last_clean":"2023-05-22T16:05:21.986529+0000","last_became_active":"2023-05-22T16:05:21.958880+0000","last_became_peered":"2023-05-22T16:05:21.958880+0000","last_unstale":"2023-05-22T16:05:21.986529+0000","last_undegraded":"2023-05-22T16:05:21.986529+0000","last_fullsized":"2023-05-22T16:05:21.986529+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_clean_scrub_stamp":"2023-05-22T16:05:21.519005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:18:24.385500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.3b","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2023-05-22T16:05:21.970062+0000","last_change":"2023-05-22T16:05:21.969331+0000","last_active":"2023-05-22T16:05:21.970062+0000","last_peered":"2023-05-22T16:05:21.970062+0000","last_clean":"2023-05-22T16:05:21.970062+0000","last_became_active":"2023-05-22T16:05:21.969017+0000","last_became_peered":"2023-05-22T16:05:21.969017+0000","last_unstale":"2023-05-22T16:05:21.970062+0000","last_undegraded":"2023-05-22T16:05:21.970062+0000","last_fullsized":"2023-05-22T16:05:21.970062+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_clean_scrub_stamp":"2023-05-22T16:05:21.519005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T16:16:17.806836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.3a","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2023-05-22T16:05:21.983844+0000","last_change":"2023-05-22T16:05:21.982911+0000","last_active":"2023-05-22T16:05:21.983844+0000","last_peered":"2023-05-22T16:05:21.983844+0000","last_clean":"2023-05-22T16:05:21.983844+0000","last_became_active":"2023-05-22T16:05:21.982587+0000","last_became_peered":"2023-05-22T16:05:21.982587+0000","last_unstale":"2023-05-22T16:05:21.983844+0000","last_undegraded":"2023-05-22T16:05:21.983844+0000","last_fullsized":"2023-05-22T16:05:21.983844+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_clean_scrub_stamp":"2023-05-22T16:05:21.519005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:16:57.793243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.39","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2023-05-22T16:05:21.992244+0000","last_change":"2023-05-22T16:05:21.985013+0000","last_active":"2023-05-22T16:05:21.992244+0000","last_peered":"2023-05-22T16:05:21.992244+0000","last_clean":"2023-05-22T16:05:21.992244+0000","last_became_active":"2023-05-22T16:05:21.984286+0000","last_became_peered":"2023-05-22T16:05:21.984286+0000","last_unstale":"2023-05-22T16:05:21.992244+0000","last_undegraded":"2023-05-22T16:05:21.992244+0000","last_fullsized":"2023-05-22T16:05:21.992244+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_clean_scrub_stamp":"2023-05-22T16:05:21.519005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T18:38:48.345515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.38","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2023-05-22T16:05:21.991226+0000","last_change":"2023-05-22T16:05:21.990505+0000","last_active":"2023-05-22T16:05:21.991226+0000","last_peered":"2023-05-22T16:05:21.991226+0000","last_clean":"2023-05-22T16:05:21.991226+0000","last_became_active":"2023-05-22T16:05:21.990152+0000","last_became_peered":"2023-05-22T16:05:21.990152+0000","last_unstale":"2023-05-22T16:05:21.991226+0000","last_undegraded":"2023-05-22T16:05:21.991226+0000","last_fullsized":"2023-05-22T16:05:21.991226+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_clean_scrub_stamp":"2023-05-22T16:05:21.519005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:41:43.814799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.37","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2023-05-22T16:05:21.992939+0000","last_change":"2023-05-22T16:05:21.986578+0000","last_active":"2023-05-22T16:05:21.992939+0000","last_peered":"2023-05-22T16:05:21.992939+0000","last_clean":"2023-05-22T16:05:21.992939+0000","last_became_active":"2023-05-22T16:05:21.985854+0000","last_became_peered":"2023-05-22T16:05:21.985854+0000","last_unstale":"2023-05-22T16:05:21.992939+0000","last_undegraded":"2023-05-22T16:05:21.992939+0000","last_fullsized":"2023-05-22T16:05:21.992939+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_clean_scrub_stamp":"2023-05-22T16:05:21.519005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:49:39.424358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.36","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2023-05-22T16:05:21.952784+0000","last_change":"2023-05-22T16:05:21.950885+0000","last_active":"2023-05-22T16:05:21.952784+0000","last_peered":"2023-05-22T16:05:21.952784+0000","last_clean":"2023-05-22T16:05:21.952784+0000","last_became_active":"2023-05-22T16:05:21.950609+0000","last_became_peered":"2023-05-22T16:05:21.950609+0000","last_unstale":"2023-05-22T16:05:21.952784+0000","last_undegraded":"2023-05-22T16:05:21.952784+0000","last_fullsized":"2023-05-22T16:05:21.952784+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_clean_scrub_stamp":"2023-05-22T16:05:21.519005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T16:29:01.531219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.35","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2023-05-22T16:05:22.000575+0000","last_change":"2023-05-22T16:05:21.997776+0000","last_active":"2023-05-22T16:05:22.000575+0000","last_peered":"2023-05-22T16:05:22.000575+0000","last_clean":"2023-05-22T16:05:22.000575+0000","last_became_active":"2023-05-22T16:05:21.997230+0000","last_became_peered":"2023-05-22T16:05:21.997230+0000","last_unstale":"2023-05-22T16:05:22.000575+0000","last_undegraded":"2023-05-22T16:05:22.000575+0000","last_fullsized":"2023-05-22T16:05:22.000575+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_clean_scrub_stamp":"2023-05-22T16:05:21.519005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:54:30.440182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.34","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2023-05-22T16:05:22.002068+0000","last_change":"2023-05-22T16:05:21.997195+0000","last_active":"2023-05-22T16:05:22.002068+0000","last_peered":"2023-05-22T16:05:22.002068+0000","last_clean":"2023-05-22T16:05:22.002068+0000","last_became_active":"2023-05-22T16:05:21.996648+0000","last_became_peered":"2023-05-22T16:05:21.996648+0000","last_unstale":"2023-05-22T16:05:22.002068+0000","last_undegraded":"2023-05-22T16:05:22.002068+0000","last_fullsized":"2023-05-22T16:05:22.002068+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_clean_scrub_stamp":"2023-05-22T16:05:21.519005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:02:05.951854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.33","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2023-05-22T16:05:21.983828+0000","last_change":"2023-05-22T16:05:21.964774+0000","last_active":"2023-05-22T16:05:21.983828+0000","last_peered":"2023-05-22T16:05:21.983828+0000","last_clean":"2023-05-22T16:05:21.983828+0000","last_became_active":"2023-05-22T16:05:21.963941+0000","last_became_peered":"2023-05-22T16:05:21.963941+0000","last_unstale":"2023-05-22T16:05:21.983828+0000","last_undegraded":"2023-05-22T16:05:21.983828+0000","last_fullsized":"2023-05-22T16:05:21.983828+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_clean_scrub_stamp":"2023-05-22T16:05:21.519005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:52:00.554956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.32","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2023-05-22T16:05:21.987271+0000","last_change":"2023-05-22T16:05:21.967310+0000","last_active":"2023-05-22T16:05:21.987271+0000","last_peered":"2023-05-22T16:05:21.987271+0000","last_clean":"2023-05-22T16:05:21.987271+0000","last_became_active":"2023-05-22T16:05:21.966561+0000","last_became_peered":"2023-05-22T16:05:21.966561+0000","last_unstale":"2023-05-22T16:05:21.987271+0000","last_undegraded":"2023-05-22T16:05:21.987271+0000","last_fullsized":"2023-05-22T16:05:21.987271+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_clean_scrub_stamp":"2023-05-22T16:05:21.519005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T16:22:43.843829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.31","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2023-05-22T16:05:21.994726+0000","last_change":"2023-05-22T16:05:21.982875+0000","last_active":"2023-05-22T16:05:21.994726+0000","last_peered":"2023-05-22T16:05:21.994726+0000","last_clean":"2023-05-22T16:05:21.994726+0000","last_became_active":"2023-05-22T16:05:21.982163+0000","last_became_peered":"2023-05-22T16:05:21.982163+0000","last_unstale":"2023-05-22T16:05:21.994726+0000","last_undegraded":"2023-05-22T16:05:21.994726+0000","last_fullsized":"2023-05-22T16:05:21.994726+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_clean_scrub_stamp":"2023-05-22T16:05:21.519005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:16:57.793243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.30","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2023-05-22T16:05:21.972422+0000","last_change":"2023-05-22T16:05:21.971263+0000","last_active":"2023-05-22T16:05:21.972422+0000","last_peered":"2023-05-22T16:05:21.972422+0000","last_clean":"2023-05-22T16:05:21.972422+0000","last_became_active":"2023-05-22T16:05:21.970948+0000","last_became_peered":"2023-05-22T16:05:21.970948+0000","last_unstale":"2023-05-22T16:05:21.972422+0000","last_undegraded":"2023-05-22T16:05:21.972422+0000","last_fullsized":"2023-05-22T16:05:21.972422+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_clean_scrub_stamp":"2023-05-22T16:05:21.519005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T18:49:41.457080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.2f","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2023-05-22T16:05:21.983669+0000","last_change":"2023-05-22T16:05:21.982607+0000","last_active":"2023-05-22T16:05:21.983669+0000","last_peered":"2023-05-22T16:05:21.983669+0000","last_clean":"2023-05-22T16:05:21.983669+0000","last_became_active":"2023-05-22T16:05:21.982207+0000","last_became_peered":"2023-05-22T16:05:21.982207+0000","last_unstale":"2023-05-22T16:05:21.983669+0000","last_undegraded":"2023-05-22T16:05:21.983669+0000","last_fullsized":"2023-05-22T16:05:21.983669+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_clean_scrub_stamp":"2023-05-22T16:05:21.519005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:22:44.849706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.2e","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2023-05-22T16:05:22.004866+0000","last_change":"2023-05-22T16:05:21.997190+0000","last_active":"2023-05-22T16:05:22.004866+0000","last_peered":"2023-05-22T16:05:22.004866+0000","last_clean":"2023-05-22T16:05:22.004866+0000","last_became_active":"2023-05-22T16:05:21.996472+0000","last_became_peered":"2023-05-22T16:05:21.996472+0000","last_unstale":"2023-05-22T16:05:22.004866+0000","last_undegraded":"2023-05-22T16:05:22.004866+0000","last_fullsized":"2023-05-22T16:05:22.004866+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_clean_scrub_stamp":"2023-05-22T16:05:21.519005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:58:14.389065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.2d","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2023-05-22T16:05:21.977785+0000","last_change":"2023-05-22T16:05:21.976684+0000","last_active":"2023-05-22T16:05:21.977785+0000","last_peered":"2023-05-22T16:05:21.977785+0000","last_clean":"2023-05-22T16:05:21.977785+0000","last_became_active":"2023-05-22T16:05:21.976411+0000","last_became_peered":"2023-05-22T16:05:21.976411+0000","last_unstale":"2023-05-22T16:05:21.977785+0000","last_undegraded":"2023-05-22T16:05:21.977785+0000","last_fullsized":"2023-05-22T16:05:21.977785+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_clean_scrub_stamp":"2023-05-22T16:05:21.519005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:40:32.750594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.2c","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2023-05-22T16:05:21.990873+0000","last_change":"2023-05-22T16:05:21.990043+0000","last_active":"2023-05-22T16:05:21.990873+0000","last_peered":"2023-05-22T16:05:21.990873+0000","last_clean":"2023-05-22T16:05:21.990873+0000","last_became_active":"2023-05-22T16:05:21.989695+0000","last_became_peered":"2023-05-22T16:05:21.989695+0000","last_unstale":"2023-05-22T16:05:21.990873+0000","last_undegraded":"2023-05-22T16:05:21.990873+0000","last_fullsized":"2023-05-22T16:05:21.990873+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_clean_scrub_stamp":"2023-05-22T16:05:21.519005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T16:24:50.702827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.2b","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2023-05-22T16:05:21.970041+0000","last_change":"2023-05-22T16:05:21.969335+0000","last_active":"2023-05-22T16:05:21.970041+0000","last_peered":"2023-05-22T16:05:21.970041+0000","last_clean":"2023-05-22T16:05:21.970041+0000","last_became_active":"2023-05-22T16:05:21.969041+0000","last_became_peered":"2023-05-22T16:05:21.969041+0000","last_unstale":"2023-05-22T16:05:21.970041+0000","last_undegraded":"2023-05-22T16:05:21.970041+0000","last_fullsized":"2023-05-22T16:05:21.970041+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_clean_scrub_stamp":"2023-05-22T16:05:21.519005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:48:52.945903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.2a","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2023-05-22T16:05:21.952486+0000","last_change":"2023-05-22T16:05:21.950635+0000","last_active":"2023-05-22T16:05:21.952486+0000","last_peered":"2023-05-22T16:05:21.952486+0000","last_clean":"2023-05-22T16:05:21.952486+0000","last_became_active":"2023-05-22T16:05:21.950295+0000","last_became_peered":"2023-05-22T16:05:21.950295+0000","last_unstale":"2023-05-22T16:05:21.952486+0000","last_undegraded":"2023-05-22T16:05:21.952486+0000","last_fullsized":"2023-05-22T16:05:21.952486+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_clean_scrub_stamp":"2023-05-22T16:05:21.519005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:31:49.575445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.29","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2023-05-22T16:05:21.954883+0000","last_change":"2023-05-22T16:05:21.952067+0000","last_active":"2023-05-22T16:05:21.954883+0000","last_peered":"2023-05-22T16:05:21.954883+0000","last_clean":"2023-05-22T16:05:21.954883+0000","last_became_active":"2023-05-22T16:05:21.951583+0000","last_became_peered":"2023-05-22T16:05:21.951583+0000","last_unstale":"2023-05-22T16:05:21.954883+0000","last_undegraded":"2023-05-22T16:05:21.954883+0000","last_fullsized":"2023-05-22T16:05:21.954883+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_clean_scrub_stamp":"2023-05-22T16:05:21.519005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:59:39.304277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.28","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2023-05-22T16:05:21.955519+0000","last_change":"2023-05-22T16:05:21.954365+0000","last_active":"2023-05-22T16:05:21.955519+0000","last_peered":"2023-05-22T16:05:21.955519+0000","last_clean":"2023-05-22T16:05:21.955519+0000","last_became_active":"2023-05-22T16:05:21.953868+0000","last_became_peered":"2023-05-22T16:05:21.953868+0000","last_unstale":"2023-05-22T16:05:21.955519+0000","last_undegraded":"2023-05-22T16:05:21.955519+0000","last_fullsized":"2023-05-22T16:05:21.955519+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_clean_scrub_stamp":"2023-05-22T16:05:21.519005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:58:14.389065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.27","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2023-05-22T16:05:21.983787+0000","last_change":"2023-05-22T16:05:21.960815+0000","last_active":"2023-05-22T16:05:21.983787+0000","last_peered":"2023-05-22T16:05:21.983787+0000","last_clean":"2023-05-22T16:05:21.983787+0000","last_became_active":"2023-05-22T16:05:21.959608+0000","last_became_peered":"2023-05-22T16:05:21.959608+0000","last_unstale":"2023-05-22T16:05:21.983787+0000","last_undegraded":"2023-05-22T16:05:21.983787+0000","last_fullsized":"2023-05-22T16:05:21.983787+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_clean_scrub_stamp":"2023-05-22T16:05:21.519005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T16:17:02.634957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.26","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2023-05-22T16:05:21.987082+0000","last_change":"2023-05-22T16:05:21.963335+0000","last_active":"2023-05-22T16:05:21.987082+0000","last_peered":"2023-05-22T16:05:21.987082+0000","last_clean":"2023-05-22T16:05:21.987082+0000","last_became_active":"2023-05-22T16:05:21.959751+0000","last_became_peered":"2023-05-22T16:05:21.959751+0000","last_unstale":"2023-05-22T16:05:21.987082+0000","last_undegraded":"2023-05-22T16:05:21.987082+0000","last_fullsized":"2023-05-22T16:05:21.987082+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_clean_scrub_stamp":"2023-05-22T16:05:21.519005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:05:02.241625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.25","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2023-05-22T16:05:22.000126+0000","last_change":"2023-05-22T16:05:21.999427+0000","last_active":"2023-05-22T16:05:22.000126+0000","last_peered":"2023-05-22T16:05:22.000126+0000","last_clean":"2023-05-22T16:05:22.000126+0000","last_became_active":"2023-05-22T16:05:21.999144+0000","last_became_peered":"2023-05-22T16:05:21.999144+0000","last_unstale":"2023-05-22T16:05:22.000126+0000","last_undegraded":"2023-05-22T16:05:22.000126+0000","last_fullsized":"2023-05-22T16:05:22.000126+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_clean_scrub_stamp":"2023-05-22T16:05:21.519005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:20:08.909225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.24","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2023-05-22T16:05:21.943911+0000","last_change":"2023-05-22T16:05:21.942712+0000","last_active":"2023-05-22T16:05:21.943911+0000","last_peered":"2023-05-22T16:05:21.943911+0000","last_clean":"2023-05-22T16:05:21.943911+0000","last_became_active":"2023-05-22T16:05:21.941658+0000","last_became_peered":"2023-05-22T16:05:21.941658+0000","last_unstale":"2023-05-22T16:05:21.943911+0000","last_undegraded":"2023-05-22T16:05:21.943911+0000","last_fullsized":"2023-05-22T16:05:21.943911+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_clean_scrub_stamp":"2023-05-22T16:05:21.519005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:37:47.243695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.23","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2023-05-22T16:05:21.994666+0000","last_change":"2023-05-22T16:05:21.981609+0000","last_active":"2023-05-22T16:05:21.994666+0000","last_peered":"2023-05-22T16:05:21.994666+0000","last_clean":"2023-05-22T16:05:21.994666+0000","last_became_active":"2023-05-22T16:05:21.980845+0000","last_became_peered":"2023-05-22T16:05:21.980845+0000","last_unstale":"2023-05-22T16:05:21.994666+0000","last_undegraded":"2023-05-22T16:05:21.994666+0000","last_fullsized":"2023-05-22T16:05:21.994666+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_clean_scrub_stamp":"2023-05-22T16:05:21.519005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T18:05:43.762192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.22","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2023-05-22T16:05:21.991036+0000","last_change":"2023-05-22T16:05:21.981856+0000","last_active":"2023-05-22T16:05:21.991036+0000","last_peered":"2023-05-22T16:05:21.991036+0000","last_clean":"2023-05-22T16:05:21.991036+0000","last_became_active":"2023-05-22T16:05:21.981105+0000","last_became_peered":"2023-05-22T16:05:21.981105+0000","last_unstale":"2023-05-22T16:05:21.991036+0000","last_undegraded":"2023-05-22T16:05:21.991036+0000","last_fullsized":"2023-05-22T16:05:21.991036+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_clean_scrub_stamp":"2023-05-22T16:05:21.519005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T18:11:18.469510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.21","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2023-05-22T16:05:21.988164+0000","last_change":"2023-05-22T16:05:21.987465+0000","last_active":"2023-05-22T16:05:21.988164+0000","last_peered":"2023-05-22T16:05:21.988164+0000","last_clean":"2023-05-22T16:05:21.988164+0000","last_became_active":"2023-05-22T16:05:21.987270+0000","last_became_peered":"2023-05-22T16:05:21.987270+0000","last_unstale":"2023-05-22T16:05:21.988164+0000","last_undegraded":"2023-05-22T16:05:21.988164+0000","last_fullsized":"2023-05-22T16:05:21.988164+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_clean_scrub_stamp":"2023-05-22T16:05:21.519005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T16:12:56.401350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.20","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2023-05-22T16:05:21.966709+0000","last_change":"2023-05-22T16:05:21.966136+0000","last_active":"2023-05-22T16:05:21.966709+0000","last_peered":"2023-05-22T16:05:21.966709+0000","last_clean":"2023-05-22T16:05:21.966709+0000","last_became_active":"2023-05-22T16:05:21.965842+0000","last_became_peered":"2023-05-22T16:05:21.965842+0000","last_unstale":"2023-05-22T16:05:21.966709+0000","last_undegraded":"2023-05-22T16:05:21.966709+0000","last_fullsized":"2023-05-22T16:05:21.966709+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_clean_scrub_stamp":"2023-05-22T16:05:21.519005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:55:10.528852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.16","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2023-05-22T16:05:21.953254+0000","last_change":"2023-05-22T16:05:21.950299+0000","last_active":"2023-05-22T16:05:21.953254+0000","last_peered":"2023-05-22T16:05:21.953254+0000","last_clean":"2023-05-22T16:05:21.953254+0000","last_became_active":"2023-05-22T16:05:21.950029+0000","last_became_peered":"2023-05-22T16:05:21.950029+0000","last_unstale":"2023-05-22T16:05:21.953254+0000","last_undegraded":"2023-05-22T16:05:21.953254+0000","last_fullsized":"2023-05-22T16:05:21.953254+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_clean_scrub_stamp":"2023-05-22T16:05:21.519005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:38:50.322794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.17","version":"0'0","reported_seq":10,"reported_epoch":195,"state":"active+clean","last_fresh":"2023-05-22T16:05:21.940688+0000","last_change":"2023-05-22T16:05:21.589641+0000","last_active":"2023-05-22T16:05:21.940688+0000","last_peered":"2023-05-22T16:05:21.940688+0000","last_clean":"2023-05-22T16:05:21.940688+0000","last_became_active":"2023-05-22T16:05:21.588996+0000","last_became_peered":"2023-05-22T16:05:21.588996+0000","last_unstale":"2023-05-22T16:05:21.940688+0000","last_undegraded":"2023-05-22T16:05:21.940688+0000","last_fullsized":"2023-05-22T16:05:21.940688+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:05:20.505200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:05:20.505200+0000","last_clean_scrub_stamp":"2023-05-22T16:05:20.505200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T16:32:24.571367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.17","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2023-05-22T16:05:21.955267+0000","last_change":"2023-05-22T16:05:21.953210+0000","last_active":"2023-05-22T16:05:21.955267+0000","last_peered":"2023-05-22T16:05:21.955267+0000","last_clean":"2023-05-22T16:05:21.955267+0000","last_became_active":"2023-05-22T16:05:21.952894+0000","last_became_peered":"2023-05-22T16:05:21.952894+0000","last_unstale":"2023-05-22T16:05:21.955267+0000","last_undegraded":"2023-05-22T16:05:21.955267+0000","last_fullsized":"2023-05-22T16:05:21.955267+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_clean_scrub_stamp":"2023-05-22T16:05:21.519005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:30:02.698847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.16","version":"0'0","reported_seq":10,"reported_epoch":195,"state":"active+clean","last_fresh":"2023-05-22T16:05:21.929668+0000","last_change":"2023-05-22T16:05:21.575900+0000","last_active":"2023-05-22T16:05:21.929668+0000","last_peered":"2023-05-22T16:05:21.929668+0000","last_clean":"2023-05-22T16:05:21.929668+0000","last_became_active":"2023-05-22T16:05:21.575461+0000","last_became_peered":"2023-05-22T16:05:21.575461+0000","last_unstale":"2023-05-22T16:05:21.929668+0000","last_undegraded":"2023-05-22T16:05:21.929668+0000","last_fullsized":"2023-05-22T16:05:21.929668+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:05:20.505200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:05:20.505200+0000","last_clean_scrub_stamp":"2023-05-22T16:05:20.505200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:02:08.546352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.14","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2023-05-22T16:05:21.987803+0000","last_change":"2023-05-22T16:05:21.972774+0000","last_active":"2023-05-22T16:05:21.987803+0000","last_peered":"2023-05-22T16:05:21.987803+0000","last_clean":"2023-05-22T16:05:21.987803+0000","last_became_active":"2023-05-22T16:05:21.971633+0000","last_became_peered":"2023-05-22T16:05:21.971633+0000","last_unstale":"2023-05-22T16:05:21.987803+0000","last_undegraded":"2023-05-22T16:05:21.987803+0000","last_fullsized":"2023-05-22T16:05:21.987803+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_clean_scrub_stamp":"2023-05-22T16:05:21.519005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:56:29.590805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.15","version":"195'3","reported_seq":13,"reported_epoch":195,"state":"active+clean","last_fresh":"2023-05-22T16:05:21.938841+0000","last_change":"2023-05-22T16:05:21.569294+0000","last_active":"2023-05-22T16:05:21.938841+0000","last_peered":"2023-05-22T16:05:21.938841+0000","last_clean":"2023-05-22T16:05:21.938841+0000","last_became_active":"2023-05-22T16:05:21.568914+0000","last_became_peered":"2023-05-22T16:05:21.568914+0000","last_unstale":"2023-05-22T16:05:21.938841+0000","last_undegraded":"2023-05-22T16:05:21.938841+0000","last_fullsized":"2023-05-22T16:05:21.938841+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:05:20.505200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:05:20.505200+0000","last_clean_scrub_stamp":"2023-05-22T16:05:20.505200+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:47:57.120748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.15","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2023-05-22T16:05:21.955179+0000","last_change":"2023-05-22T16:05:21.950957+0000","last_active":"2023-05-22T16:05:21.955179+0000","last_peered":"2023-05-22T16:05:21.955179+0000","last_clean":"2023-05-22T16:05:21.955179+0000","last_became_active":"2023-05-22T16:05:21.950627+0000","last_became_peered":"2023-05-22T16:05:21.950627+0000","last_unstale":"2023-05-22T16:05:21.955179+0000","last_undegraded":"2023-05-22T16:05:21.955179+0000","last_fullsized":"2023-05-22T16:05:21.955179+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_clean_scrub_stamp":"2023-05-22T16:05:21.519005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:23:31.666673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.14","version":"195'6","reported_seq":16,"reported_epoch":195,"state":"active+clean","last_fresh":"2023-05-22T16:05:21.997131+0000","last_change":"2023-05-22T16:05:21.575523+0000","last_active":"2023-05-22T16:05:21.997131+0000","last_peered":"2023-05-22T16:05:21.997131+0000","last_clean":"2023-05-22T16:05:21.997131+0000","last_became_active":"2023-05-22T16:05:21.575155+0000","last_became_peered":"2023-05-22T16:05:21.575155+0000","last_unstale":"2023-05-22T16:05:21.997131+0000","last_undegraded":"2023-05-22T16:05:21.997131+0000","last_fullsized":"2023-05-22T16:05:21.997131+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:05:20.505200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:05:20.505200+0000","last_clean_scrub_stamp":"2023-05-22T16:05:20.505200+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:49:38.410553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1548,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.12","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2023-05-22T16:05:21.954863+0000","last_change":"2023-05-22T16:05:21.953032+0000","last_active":"2023-05-22T16:05:21.954863+0000","last_peered":"2023-05-22T16:05:21.954863+0000","last_clean":"2023-05-22T16:05:21.954863+0000","last_became_active":"2023-05-22T16:05:21.952770+0000","last_became_peered":"2023-05-22T16:05:21.952770+0000","last_unstale":"2023-05-22T16:05:21.954863+0000","last_undegraded":"2023-05-22T16:05:21.954863+0000","last_fullsized":"2023-05-22T16:05:21.954863+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_clean_scrub_stamp":"2023-05-22T16:05:21.519005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T16:32:25.585172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.13","version":"195'2","reported_seq":12,"reported_epoch":195,"state":"active+clean","last_fresh":"2023-05-22T16:05:21.996956+0000","last_change":"2023-05-22T16:05:21.566159+0000","last_active":"2023-05-22T16:05:21.996956+0000","last_peered":"2023-05-22T16:05:21.996956+0000","last_clean":"2023-05-22T16:05:21.996956+0000","last_became_active":"2023-05-22T16:05:21.565413+0000","last_became_peered":"2023-05-22T16:05:21.565413+0000","last_unstale":"2023-05-22T16:05:21.996956+0000","last_undegraded":"2023-05-22T16:05:21.996956+0000","last_fullsized":"2023-05-22T16:05:21.996956+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:05:20.505200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:05:20.505200+0000","last_clean_scrub_stamp":"2023-05-22T16:05:20.505200+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:21:03.448650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.13","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2023-05-22T16:05:21.987136+0000","last_change":"2023-05-22T16:05:21.963650+0000","last_active":"2023-05-22T16:05:21.987136+0000","last_peered":"2023-05-22T16:05:21.987136+0000","last_clean":"2023-05-22T16:05:21.987136+0000","last_became_active":"2023-05-22T16:05:21.962778+0000","last_became_peered":"2023-05-22T16:05:21.962778+0000","last_unstale":"2023-05-22T16:05:21.987136+0000","last_undegraded":"2023-05-22T16:05:21.987136+0000","last_fullsized":"2023-05-22T16:05:21.987136+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_clean_scrub_stamp":"2023-05-22T16:05:21.519005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:52:47.830248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.12","version":"0'0","reported_seq":10,"reported_epoch":195,"state":"active+clean","last_fresh":"2023-05-22T16:05:21.937229+0000","last_change":"2023-05-22T16:05:21.570545+0000","last_active":"2023-05-22T16:05:21.937229+0000","last_peered":"2023-05-22T16:05:21.937229+0000","last_clean":"2023-05-22T16:05:21.937229+0000","last_became_active":"2023-05-22T16:05:21.570175+0000","last_became_peered":"2023-05-22T16:05:21.570175+0000","last_unstale":"2023-05-22T16:05:21.937229+0000","last_undegraded":"2023-05-22T16:05:21.937229+0000","last_fullsized":"2023-05-22T16:05:21.937229+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:05:20.505200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:05:20.505200+0000","last_clean_scrub_stamp":"2023-05-22T16:05:20.505200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:56:28.577000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.10","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2023-05-22T16:05:21.986843+0000","last_change":"2023-05-22T16:05:21.967603+0000","last_active":"2023-05-22T16:05:21.986843+0000","last_peered":"2023-05-22T16:05:21.986843+0000","last_clean":"2023-05-22T16:05:21.986843+0000","last_became_active":"2023-05-22T16:05:21.966720+0000","last_became_peered":"2023-05-22T16:05:21.966720+0000","last_unstale":"2023-05-22T16:05:21.986843+0000","last_undegraded":"2023-05-22T16:05:21.986843+0000","last_fullsized":"2023-05-22T16:05:21.986843+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_clean_scrub_stamp":"2023-05-22T16:05:21.519005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:38:47.496733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.11","version":"0'0","reported_seq":10,"reported_epoch":195,"state":"active+clean","last_fresh":"2023-05-22T16:05:21.939652+0000","last_change":"2023-05-22T16:05:21.577260+0000","last_active":"2023-05-22T16:05:21.939652+0000","last_peered":"2023-05-22T16:05:21.939652+0000","last_clean":"2023-05-22T16:05:21.939652+0000","last_became_active":"2023-05-22T16:05:21.576960+0000","last_became_peered":"2023-05-22T16:05:21.576960+0000","last_unstale":"2023-05-22T16:05:21.939652+0000","last_undegraded":"2023-05-22T16:05:21.939652+0000","last_fullsized":"2023-05-22T16:05:21.939652+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:05:20.505200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:05:20.505200+0000","last_clean_scrub_stamp":"2023-05-22T16:05:20.505200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:56:28.577000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.11","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2023-05-22T16:05:21.989716+0000","last_change":"2023-05-22T16:05:21.988706+0000","last_active":"2023-05-22T16:05:21.989716+0000","last_peered":"2023-05-22T16:05:21.989716+0000","last_clean":"2023-05-22T16:05:21.989716+0000","last_became_active":"2023-05-22T16:05:21.988206+0000","last_became_peered":"2023-05-22T16:05:21.988206+0000","last_unstale":"2023-05-22T16:05:21.989716+0000","last_undegraded":"2023-05-22T16:05:21.989716+0000","last_fullsized":"2023-05-22T16:05:21.989716+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_clean_scrub_stamp":"2023-05-22T16:05:21.519005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:54:46.681293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.10","version":"195'8","reported_seq":18,"reported_epoch":195,"state":"active+clean","last_fresh":"2023-05-22T16:05:22.003213+0000","last_change":"2023-05-22T16:05:21.565870+0000","last_active":"2023-05-22T16:05:22.003213+0000","last_peered":"2023-05-22T16:05:22.003213+0000","last_clean":"2023-05-22T16:05:22.003213+0000","last_became_active":"2023-05-22T16:05:21.565036+0000","last_became_peered":"2023-05-22T16:05:21.565036+0000","last_unstale":"2023-05-22T16:05:22.003213+0000","last_undegraded":"2023-05-22T16:05:22.003213+0000","last_fullsized":"2023-05-22T16:05:22.003213+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:05:20.505200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:05:20.505200+0000","last_clean_scrub_stamp":"2023-05-22T16:05:20.505200+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:29:44.137982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.e","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2023-05-22T16:05:21.986809+0000","last_change":"2023-05-22T16:05:21.986108+0000","last_active":"2023-05-22T16:05:21.986809+0000","last_peered":"2023-05-22T16:05:21.986809+0000","last_clean":"2023-05-22T16:05:21.986809+0000","last_became_active":"2023-05-22T16:05:21.985777+0000","last_became_peered":"2023-05-22T16:05:21.985777+0000","last_unstale":"2023-05-22T16:05:21.986809+0000","last_undegraded":"2023-05-22T16:05:21.986809+0000","last_fullsized":"2023-05-22T16:05:21.986809+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_clean_scrub_stamp":"2023-05-22T16:05:21.519005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:10:10.702604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.f","version":"195'1","reported_seq":11,"reported_epoch":195,"state":"active+clean","last_fresh":"2023-05-22T16:05:21.945708+0000","last_change":"2023-05-22T16:05:21.571450+0000","last_active":"2023-05-22T16:05:21.945708+0000","last_peered":"2023-05-22T16:05:21.945708+0000","last_clean":"2023-05-22T16:05:21.945708+0000","last_became_active":"2023-05-22T16:05:21.570848+0000","last_became_peered":"2023-05-22T16:05:21.570848+0000","last_unstale":"2023-05-22T16:05:21.945708+0000","last_undegraded":"2023-05-22T16:05:21.945708+0000","last_fullsized":"2023-05-22T16:05:21.945708+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:05:20.505200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:05:20.505200+0000","last_clean_scrub_stamp":"2023-05-22T16:05:20.505200+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:19:47.972327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.f","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2023-05-22T16:05:21.986302+0000","last_change":"2023-05-22T16:05:21.965884+0000","last_active":"2023-05-22T16:05:21.986302+0000","last_peered":"2023-05-22T16:05:21.986302+0000","last_clean":"2023-05-22T16:05:21.986302+0000","last_became_active":"2023-05-22T16:05:21.964971+0000","last_became_peered":"2023-05-22T16:05:21.964971+0000","last_unstale":"2023-05-22T16:05:21.986302+0000","last_undegraded":"2023-05-22T16:05:21.986302+0000","last_fullsized":"2023-05-22T16:05:21.986302+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_clean_scrub_stamp":"2023-05-22T16:05:21.519005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:32:05.958044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.e","version":"195'2","reported_seq":12,"reported_epoch":195,"state":"active+clean","last_fresh":"2023-05-22T16:05:21.989816+0000","last_change":"2023-05-22T16:05:21.576915+0000","last_active":"2023-05-22T16:05:21.989816+0000","last_peered":"2023-05-22T16:05:21.989816+0000","last_clean":"2023-05-22T16:05:21.989816+0000","last_became_active":"2023-05-22T16:05:21.576583+0000","last_became_peered":"2023-05-22T16:05:21.576583+0000","last_unstale":"2023-05-22T16:05:21.989816+0000","last_undegraded":"2023-05-22T16:05:21.989816+0000","last_fullsized":"2023-05-22T16:05:21.989816+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:05:20.505200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:05:20.505200+0000","last_clean_scrub_stamp":"2023-05-22T16:05:20.505200+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:54:29.426377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.c","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2023-05-22T16:05:22.002846+0000","last_change":"2023-05-22T16:05:21.998833+0000","last_active":"2023-05-22T16:05:22.002846+0000","last_peered":"2023-05-22T16:05:22.002846+0000","last_clean":"2023-05-22T16:05:22.002846+0000","last_became_active":"2023-05-22T16:05:21.997997+0000","last_became_peered":"2023-05-22T16:05:21.997997+0000","last_unstale":"2023-05-22T16:05:22.002846+0000","last_undegraded":"2023-05-22T16:05:22.002846+0000","last_fullsized":"2023-05-22T16:05:22.002846+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_clean_scrub_stamp":"2023-05-22T16:05:21.519005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T16:53:07.107628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.d","version":"195'1","reported_seq":11,"reported_epoch":195,"state":"active+clean","last_fresh":"2023-05-22T16:05:21.996305+0000","last_change":"2023-05-22T16:05:21.568161+0000","last_active":"2023-05-22T16:05:21.996305+0000","last_peered":"2023-05-22T16:05:21.996305+0000","last_clean":"2023-05-22T16:05:21.996305+0000","last_became_active":"2023-05-22T16:05:21.567369+0000","last_became_peered":"2023-05-22T16:05:21.567369+0000","last_unstale":"2023-05-22T16:05:21.996305+0000","last_undegraded":"2023-05-22T16:05:21.996305+0000","last_fullsized":"2023-05-22T16:05:21.996305+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:05:20.505200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:05:20.505200+0000","last_clean_scrub_stamp":"2023-05-22T16:05:20.505200+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:14:03.680350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.d","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2023-05-22T16:05:21.983853+0000","last_change":"2023-05-22T16:05:21.959490+0000","last_active":"2023-05-22T16:05:21.983853+0000","last_peered":"2023-05-22T16:05:21.983853+0000","last_clean":"2023-05-22T16:05:21.983853+0000","last_became_active":"2023-05-22T16:05:21.958689+0000","last_became_peered":"2023-05-22T16:05:21.958689+0000","last_unstale":"2023-05-22T16:05:21.983853+0000","last_undegraded":"2023-05-22T16:05:21.983853+0000","last_fullsized":"2023-05-22T16:05:21.983853+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_clean_scrub_stamp":"2023-05-22T16:05:21.519005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:33:18.037523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.c","version":"0'0","reported_seq":10,"reported_epoch":195,"state":"active+clean","last_fresh":"2023-05-22T16:05:21.940426+0000","last_change":"2023-05-22T16:05:21.573321+0000","last_active":"2023-05-22T16:05:21.940426+0000","last_peered":"2023-05-22T16:05:21.940426+0000","last_clean":"2023-05-22T16:05:21.940426+0000","last_became_active":"2023-05-22T16:05:21.572288+0000","last_became_peered":"2023-05-22T16:05:21.572288+0000","last_unstale":"2023-05-22T16:05:21.940426+0000","last_undegraded":"2023-05-22T16:05:21.940426+0000","last_fullsized":"2023-05-22T16:05:21.940426+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:05:20.505200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:05:20.505200+0000","last_clean_scrub_stamp":"2023-05-22T16:05:20.505200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:59:38.290472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.a","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2023-05-22T16:05:21.950544+0000","last_change":"2023-05-22T16:05:21.942660+0000","last_active":"2023-05-22T16:05:21.950544+0000","last_peered":"2023-05-22T16:05:21.950544+0000","last_clean":"2023-05-22T16:05:21.950544+0000","last_became_active":"2023-05-22T16:05:21.942278+0000","last_became_peered":"2023-05-22T16:05:21.942278+0000","last_unstale":"2023-05-22T16:05:21.950544+0000","last_undegraded":"2023-05-22T16:05:21.950544+0000","last_fullsized":"2023-05-22T16:05:21.950544+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_clean_scrub_stamp":"2023-05-22T16:05:21.519005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:03:30.813740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.b","version":"195'2","reported_seq":12,"reported_epoch":195,"state":"active+clean","last_fresh":"2023-05-22T16:05:21.952313+0000","last_change":"2023-05-22T16:05:21.554180+0000","last_active":"2023-05-22T16:05:21.952313+0000","last_peered":"2023-05-22T16:05:21.952313+0000","last_clean":"2023-05-22T16:05:21.952313+0000","last_became_active":"2023-05-22T16:05:21.553865+0000","last_became_peered":"2023-05-22T16:05:21.553865+0000","last_unstale":"2023-05-22T16:05:21.952313+0000","last_undegraded":"2023-05-22T16:05:21.952313+0000","last_fullsized":"2023-05-22T16:05:21.952313+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:05:20.505200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:05:20.505200+0000","last_clean_scrub_stamp":"2023-05-22T16:05:20.505200+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:38:46.482928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.b","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2023-05-22T16:05:21.987004+0000","last_change":"2023-05-22T16:05:21.959708+0000","last_active":"2023-05-22T16:05:21.987004+0000","last_peered":"2023-05-22T16:05:21.987004+0000","last_clean":"2023-05-22T16:05:21.987004+0000","last_became_active":"2023-05-22T16:05:21.958059+0000","last_became_peered":"2023-05-22T16:05:21.958059+0000","last_unstale":"2023-05-22T16:05:21.987004+0000","last_undegraded":"2023-05-22T16:05:21.987004+0000","last_fullsized":"2023-05-22T16:05:21.987004+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_clean_scrub_stamp":"2023-05-22T16:05:21.519005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:29:45.151787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.a","version":"0'0","reported_seq":10,"reported_epoch":195,"state":"active+clean","last_fresh":"2023-05-22T16:05:21.928184+0000","last_change":"2023-05-22T16:05:21.572978+0000","last_active":"2023-05-22T16:05:21.928184+0000","last_peered":"2023-05-22T16:05:21.928184+0000","last_clean":"2023-05-22T16:05:21.928184+0000","last_became_active":"2023-05-22T16:05:21.572735+0000","last_became_peered":"2023-05-22T16:05:21.572735+0000","last_unstale":"2023-05-22T16:05:21.928184+0000","last_undegraded":"2023-05-22T16:05:21.928184+0000","last_fullsized":"2023-05-22T16:05:21.928184+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:05:20.505200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:05:20.505200+0000","last_clean_scrub_stamp":"2023-05-22T16:05:20.505200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T16:16:12.432825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.8","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2023-05-22T16:05:21.994865+0000","last_change":"2023-05-22T16:05:21.985741+0000","last_active":"2023-05-22T16:05:21.994865+0000","last_peered":"2023-05-22T16:05:21.994865+0000","last_clean":"2023-05-22T16:05:21.994865+0000","last_became_active":"2023-05-22T16:05:21.984619+0000","last_became_peered":"2023-05-22T16:05:21.984619+0000","last_unstale":"2023-05-22T16:05:21.994865+0000","last_undegraded":"2023-05-22T16:05:21.994865+0000","last_fullsized":"2023-05-22T16:05:21.994865+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_clean_scrub_stamp":"2023-05-22T16:05:21.519005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:20:08.909225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.9","version":"0'0","reported_seq":10,"reported_epoch":195,"state":"active+clean","last_fresh":"2023-05-22T16:05:21.943510+0000","last_change":"2023-05-22T16:05:21.565141+0000","last_active":"2023-05-22T16:05:21.943510+0000","last_peered":"2023-05-22T16:05:21.943510+0000","last_clean":"2023-05-22T16:05:21.943510+0000","last_became_active":"2023-05-22T16:05:21.564803+0000","last_became_peered":"2023-05-22T16:05:21.564803+0000","last_unstale":"2023-05-22T16:05:21.943510+0000","last_undegraded":"2023-05-22T16:05:21.943510+0000","last_fullsized":"2023-05-22T16:05:21.943510+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:05:20.505200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:05:20.505200+0000","last_clean_scrub_stamp":"2023-05-22T16:05:20.505200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T16:43:05.102031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.9","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2023-05-22T16:05:21.954713+0000","last_change":"2023-05-22T16:05:21.952574+0000","last_active":"2023-05-22T16:05:21.954713+0000","last_peered":"2023-05-22T16:05:21.954713+0000","last_clean":"2023-05-22T16:05:21.954713+0000","last_became_active":"2023-05-22T16:05:21.950472+0000","last_became_peered":"2023-05-22T16:05:21.950472+0000","last_unstale":"2023-05-22T16:05:21.954713+0000","last_undegraded":"2023-05-22T16:05:21.954713+0000","last_fullsized":"2023-05-22T16:05:21.954713+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_clean_scrub_stamp":"2023-05-22T16:05:21.519005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:06:43.269527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.8","version":"0'0","reported_seq":10,"reported_epoch":195,"state":"active+clean","last_fresh":"2023-05-22T16:05:21.957022+0000","last_change":"2023-05-22T16:05:21.579600+0000","last_active":"2023-05-22T16:05:21.957022+0000","last_peered":"2023-05-22T16:05:21.957022+0000","last_clean":"2023-05-22T16:05:21.957022+0000","last_became_active":"2023-05-22T16:05:21.578847+0000","last_became_peered":"2023-05-22T16:05:21.578847+0000","last_unstale":"2023-05-22T16:05:21.957022+0000","last_undegraded":"2023-05-22T16:05:21.957022+0000","last_fullsized":"2023-05-22T16:05:21.957022+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:05:20.505200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:05:20.505200+0000","last_clean_scrub_stamp":"2023-05-22T16:05:20.505200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T16:16:16.793031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.6","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2023-05-22T16:05:21.954765+0000","last_change":"2023-05-22T16:05:21.953258+0000","last_active":"2023-05-22T16:05:21.954765+0000","last_peered":"2023-05-22T16:05:21.954765+0000","last_clean":"2023-05-22T16:05:21.954765+0000","last_became_active":"2023-05-22T16:05:21.952881+0000","last_became_peered":"2023-05-22T16:05:21.952881+0000","last_unstale":"2023-05-22T16:05:21.954765+0000","last_undegraded":"2023-05-22T16:05:21.954765+0000","last_fullsized":"2023-05-22T16:05:21.954765+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_clean_scrub_stamp":"2023-05-22T16:05:21.519005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:42:58.578446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.7","version":"195'2","reported_seq":12,"reported_epoch":195,"state":"active+clean","last_fresh":"2023-05-22T16:05:21.997383+0000","last_change":"2023-05-22T16:05:21.568026+0000","last_active":"2023-05-22T16:05:21.997383+0000","last_peered":"2023-05-22T16:05:21.997383+0000","last_clean":"2023-05-22T16:05:21.997383+0000","last_became_active":"2023-05-22T16:05:21.567033+0000","last_became_peered":"2023-05-22T16:05:21.567033+0000","last_unstale":"2023-05-22T16:05:21.997383+0000","last_undegraded":"2023-05-22T16:05:21.997383+0000","last_fullsized":"2023-05-22T16:05:21.997383+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:05:20.505200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:05:20.505200+0000","last_clean_scrub_stamp":"2023-05-22T16:05:20.505200+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:05:01.227820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.7","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2023-05-22T16:05:21.984485+0000","last_change":"2023-05-22T16:05:21.983608+0000","last_active":"2023-05-22T16:05:21.984485+0000","last_peered":"2023-05-22T16:05:21.984485+0000","last_clean":"2023-05-22T16:05:21.984485+0000","last_became_active":"2023-05-22T16:05:21.983327+0000","last_became_peered":"2023-05-22T16:05:21.983327+0000","last_unstale":"2023-05-22T16:05:21.984485+0000","last_undegraded":"2023-05-22T16:05:21.984485+0000","last_fullsized":"2023-05-22T16:05:21.984485+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_clean_scrub_stamp":"2023-05-22T16:05:21.519005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:31:27.332257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.6","version":"0'0","reported_seq":10,"reported_epoch":195,"state":"active+clean","last_fresh":"2023-05-22T16:05:21.935195+0000","last_change":"2023-05-22T16:05:21.570917+0000","last_active":"2023-05-22T16:05:21.935195+0000","last_peered":"2023-05-22T16:05:21.935195+0000","last_clean":"2023-05-22T16:05:21.935195+0000","last_became_active":"2023-05-22T16:05:21.570604+0000","last_became_peered":"2023-05-22T16:05:21.570604+0000","last_unstale":"2023-05-22T16:05:21.935195+0000","last_undegraded":"2023-05-22T16:05:21.935195+0000","last_fullsized":"2023-05-22T16:05:21.935195+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:05:20.505200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:05:20.505200+0000","last_clean_scrub_stamp":"2023-05-22T16:05:20.505200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:52:46.816443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.4","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2023-05-22T16:05:21.955023+0000","last_change":"2023-05-22T16:05:21.950184+0000","last_active":"2023-05-22T16:05:21.955023+0000","last_peered":"2023-05-22T16:05:21.955023+0000","last_clean":"2023-05-22T16:05:21.955023+0000","last_became_active":"2023-05-22T16:05:21.949891+0000","last_became_peered":"2023-05-22T16:05:21.949891+0000","last_unstale":"2023-05-22T16:05:21.955023+0000","last_undegraded":"2023-05-22T16:05:21.955023+0000","last_fullsized":"2023-05-22T16:05:21.955023+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_clean_scrub_stamp":"2023-05-22T16:05:21.519005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:35:44.913595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.5","version":"0'0","reported_seq":10,"reported_epoch":195,"state":"active+clean","last_fresh":"2023-05-22T16:05:21.934745+0000","last_change":"2023-05-22T16:05:21.574565+0000","last_active":"2023-05-22T16:05:21.934745+0000","last_peered":"2023-05-22T16:05:21.934745+0000","last_clean":"2023-05-22T16:05:21.934745+0000","last_became_active":"2023-05-22T16:05:21.574090+0000","last_became_peered":"2023-05-22T16:05:21.574090+0000","last_unstale":"2023-05-22T16:05:21.934745+0000","last_undegraded":"2023-05-22T16:05:21.934745+0000","last_fullsized":"2023-05-22T16:05:21.934745+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:05:20.505200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:05:20.505200+0000","last_clean_scrub_stamp":"2023-05-22T16:05:20.505200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T18:38:47.331710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.5","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2023-05-22T16:05:21.977181+0000","last_change":"2023-05-22T16:05:21.975813+0000","last_active":"2023-05-22T16:05:21.977181+0000","last_peered":"2023-05-22T16:05:21.977181+0000","last_clean":"2023-05-22T16:05:21.977181+0000","last_became_active":"2023-05-22T16:05:21.975554+0000","last_became_peered":"2023-05-22T16:05:21.975554+0000","last_unstale":"2023-05-22T16:05:21.977181+0000","last_undegraded":"2023-05-22T16:05:21.977181+0000","last_fullsized":"2023-05-22T16:05:21.977181+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_clean_scrub_stamp":"2023-05-22T16:05:21.519005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:03:30.813740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.4","version":"0'0","reported_seq":10,"reported_epoch":195,"state":"active+clean","last_fresh":"2023-05-22T16:05:21.940406+0000","last_change":"2023-05-22T16:05:21.589075+0000","last_active":"2023-05-22T16:05:21.940406+0000","last_peered":"2023-05-22T16:05:21.940406+0000","last_clean":"2023-05-22T16:05:21.940406+0000","last_became_active":"2023-05-22T16:05:21.588792+0000","last_became_peered":"2023-05-22T16:05:21.588792+0000","last_unstale":"2023-05-22T16:05:21.940406+0000","last_undegraded":"2023-05-22T16:05:21.940406+0000","last_fullsized":"2023-05-22T16:05:21.940406+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:05:20.505200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:05:20.505200+0000","last_clean_scrub_stamp":"2023-05-22T16:05:20.505200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:52:46.816443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.2","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2023-05-22T16:05:22.004893+0000","last_change":"2023-05-22T16:05:21.997428+0000","last_active":"2023-05-22T16:05:22.004893+0000","last_peered":"2023-05-22T16:05:22.004893+0000","last_clean":"2023-05-22T16:05:22.004893+0000","last_became_active":"2023-05-22T16:05:21.996749+0000","last_became_peered":"2023-05-22T16:05:21.996749+0000","last_unstale":"2023-05-22T16:05:22.004893+0000","last_undegraded":"2023-05-22T16:05:22.004893+0000","last_fullsized":"2023-05-22T16:05:22.004893+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_clean_scrub_stamp":"2023-05-22T16:05:21.519005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:38:35.770174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.3","version":"0'0","reported_seq":10,"reported_epoch":195,"state":"active+clean","last_fresh":"2023-05-22T16:05:21.957343+0000","last_change":"2023-05-22T16:05:21.588428+0000","last_active":"2023-05-22T16:05:21.957343+0000","last_peered":"2023-05-22T16:05:21.957343+0000","last_clean":"2023-05-22T16:05:21.957343+0000","last_became_active":"2023-05-22T16:05:21.587723+0000","last_became_peered":"2023-05-22T16:05:21.587723+0000","last_unstale":"2023-05-22T16:05:21.957343+0000","last_undegraded":"2023-05-22T16:05:21.957343+0000","last_fullsized":"2023-05-22T16:05:21.957343+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:05:20.505200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:05:20.505200+0000","last_clean_scrub_stamp":"2023-05-22T16:05:20.505200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:48:51.932099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.3","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2023-05-22T16:05:22.004844+0000","last_change":"2023-05-22T16:05:21.997207+0000","last_active":"2023-05-22T16:05:22.004844+0000","last_peered":"2023-05-22T16:05:22.004844+0000","last_clean":"2023-05-22T16:05:22.004844+0000","last_became_active":"2023-05-22T16:05:21.996528+0000","last_became_peered":"2023-05-22T16:05:21.996528+0000","last_unstale":"2023-05-22T16:05:22.004844+0000","last_undegraded":"2023-05-22T16:05:22.004844+0000","last_fullsized":"2023-05-22T16:05:22.004844+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_clean_scrub_stamp":"2023-05-22T16:05:21.519005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T03:24:08.639357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.2","version":"0'0","reported_seq":10,"reported_epoch":195,"state":"active+clean","last_fresh":"2023-05-22T16:05:21.958132+0000","last_change":"2023-05-22T16:05:21.573511+0000","last_active":"2023-05-22T16:05:21.958132+0000","last_peered":"2023-05-22T16:05:21.958132+0000","last_clean":"2023-05-22T16:05:21.958132+0000","last_became_active":"2023-05-22T16:05:21.571649+0000","last_became_peered":"2023-05-22T16:05:21.571649+0000","last_unstale":"2023-05-22T16:05:21.958132+0000","last_undegraded":"2023-05-22T16:05:21.958132+0000","last_fullsized":"2023-05-22T16:05:21.958132+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:05:20.505200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:05:20.505200+0000","last_clean_scrub_stamp":"2023-05-22T16:05:20.505200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:42:57.564641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.0","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2023-05-22T16:05:21.994657+0000","last_change":"2023-05-22T16:05:21.981143+0000","last_active":"2023-05-22T16:05:21.994657+0000","last_peered":"2023-05-22T16:05:21.994657+0000","last_clean":"2023-05-22T16:05:21.994657+0000","last_became_active":"2023-05-22T16:05:21.980177+0000","last_became_peered":"2023-05-22T16:05:21.980177+0000","last_unstale":"2023-05-22T16:05:21.994657+0000","last_undegraded":"2023-05-22T16:05:21.994657+0000","last_fullsized":"2023-05-22T16:05:21.994657+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_clean_scrub_stamp":"2023-05-22T16:05:21.519005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:59:51.654460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.1","version":"0'0","reported_seq":10,"reported_epoch":195,"state":"active+clean","last_fresh":"2023-05-22T16:05:21.938576+0000","last_change":"2023-05-22T16:05:21.578903+0000","last_active":"2023-05-22T16:05:21.938576+0000","last_peered":"2023-05-22T16:05:21.938576+0000","last_clean":"2023-05-22T16:05:21.938576+0000","last_became_active":"2023-05-22T16:05:21.578160+0000","last_became_peered":"2023-05-22T16:05:21.578160+0000","last_unstale":"2023-05-22T16:05:21.938576+0000","last_undegraded":"2023-05-22T16:05:21.938576+0000","last_fullsized":"2023-05-22T16:05:21.938576+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:05:20.505200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:05:20.505200+0000","last_clean_scrub_stamp":"2023-05-22T16:05:20.505200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:49:38.410553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.1","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2023-05-22T16:05:21.965326+0000","last_change":"2023-05-22T16:05:21.964514+0000","last_active":"2023-05-22T16:05:21.965326+0000","last_peered":"2023-05-22T16:05:21.965326+0000","last_clean":"2023-05-22T16:05:21.965326+0000","last_became_active":"2023-05-22T16:05:21.964217+0000","last_became_peered":"2023-05-22T16:05:21.964217+0000","last_unstale":"2023-05-22T16:05:21.965326+0000","last_undegraded":"2023-05-22T16:05:21.965326+0000","last_fullsized":"2023-05-22T16:05:21.965326+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_clean_scrub_stamp":"2023-05-22T16:05:21.519005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T20:10:09.169722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.0","version":"0'0","reported_seq":10,"reported_epoch":195,"state":"active+clean","last_fresh":"2023-05-22T16:05:21.946515+0000","last_change":"2023-05-22T16:05:21.566366+0000","last_active":"2023-05-22T16:05:21.946515+0000","last_peered":"2023-05-22T16:05:21.946515+0000","last_clean":"2023-05-22T16:05:21.946515+0000","last_became_active":"2023-05-22T16:05:21.566072+0000","last_became_peered":"2023-05-22T16:05:21.566072+0000","last_unstale":"2023-05-22T16:05:21.946515+0000","last_undegraded":"2023-05-22T16:05:21.946515+0000","last_fullsized":"2023-05-22T16:05:21.946515+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:05:20.505200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:05:20.505200+0000","last_clean_scrub_stamp":"2023-05-22T16:05:20.505200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T16:53:06.093823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.18","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2023-05-22T16:05:21.984466+0000","last_change":"2023-05-22T16:05:21.983501+0000","last_active":"2023-05-22T16:05:21.984466+0000","last_peered":"2023-05-22T16:05:21.984466+0000","last_clean":"2023-05-22T16:05:21.984466+0000","last_became_active":"2023-05-22T16:05:21.983182+0000","last_became_peered":"2023-05-22T16:05:21.983182+0000","last_unstale":"2023-05-22T16:05:21.984466+0000","last_undegraded":"2023-05-22T16:05:21.984466+0000","last_fullsized":"2023-05-22T16:05:21.984466+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_clean_scrub_stamp":"2023-05-22T16:05:21.519005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:24:20.405612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.19","version":"0'0","reported_seq":10,"reported_epoch":195,"state":"active+clean","last_fresh":"2023-05-22T16:05:21.965145+0000","last_change":"2023-05-22T16:05:21.579892+0000","last_active":"2023-05-22T16:05:21.965145+0000","last_peered":"2023-05-22T16:05:21.965145+0000","last_clean":"2023-05-22T16:05:21.965145+0000","last_became_active":"2023-05-22T16:05:21.579210+0000","last_became_peered":"2023-05-22T16:05:21.579210+0000","last_unstale":"2023-05-22T16:05:21.965145+0000","last_undegraded":"2023-05-22T16:05:21.965145+0000","last_fullsized":"2023-05-22T16:05:21.965145+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:05:20.505200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:05:20.505200+0000","last_clean_scrub_stamp":"2023-05-22T16:05:20.505200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:56:28.577000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"183'326","reported_seq":1078,"reported_epoch":194,"state":"active+clean","last_fresh":"2023-05-22T16:05:21.542676+0000","last_change":"2023-05-22T15:26:39.318204+0000","last_active":"2023-05-22T16:05:21.542676+0000","last_peered":"2023-05-22T16:05:21.542676+0000","last_clean":"2023-05-22T16:05:21.542676+0000","last_became_active":"2023-05-22T15:26:39.315370+0000","last_became_peered":"2023-05-22T15:26:39.315370+0000","last_unstale":"2023-05-22T16:05:21.542676+0000","last_undegraded":"2023-05-22T16:05:21.542676+0000","last_fullsized":"2023-05-22T16:05:21.542676+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:38.020076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:38.020076+0000","last_clean_scrub_stamp":"2023-05-22T15:26:38.020076+0000","objects_scrubbed":0,"log_size":326,"log_dups_size":0,"ondisk_log_size":326,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:06:39.207543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1940,"num_read_kb":4014,"num_write":539,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":195,"reported_epoch":194,"state":"active+clean","last_fresh":"2023-05-22T16:05:21.555321+0000","last_change":"2023-05-22T15:26:44.971951+0000","last_active":"2023-05-22T16:05:21.555321+0000","last_peered":"2023-05-22T16:05:21.555321+0000","last_clean":"2023-05-22T16:05:21.555321+0000","last_became_active":"2023-05-22T15:26:44.970872+0000","last_became_peered":"2023-05-22T15:26:44.970872+0000","last_unstale":"2023-05-22T16:05:21.555321+0000","last_undegraded":"2023-05-22T16:05:21.555321+0000","last_fullsized":"2023-05-22T16:05:21.555321+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:55:34.752949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.1b","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2023-05-22T16:05:21.973217+0000","last_change":"2023-05-22T16:05:21.971211+0000","last_active":"2023-05-22T16:05:21.973217+0000","last_peered":"2023-05-22T16:05:21.973217+0000","last_clean":"2023-05-22T16:05:21.973217+0000","last_became_active":"2023-05-22T16:05:21.970857+0000","last_became_peered":"2023-05-22T16:05:21.970857+0000","last_unstale":"2023-05-22T16:05:21.973217+0000","last_undegraded":"2023-05-22T16:05:21.973217+0000","last_fullsized":"2023-05-22T16:05:21.973217+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_clean_scrub_stamp":"2023-05-22T16:05:21.519005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:24:20.405612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1a","version":"195'1","reported_seq":11,"reported_epoch":195,"state":"active+clean","last_fresh":"2023-05-22T16:05:21.949529+0000","last_change":"2023-05-22T16:05:21.566496+0000","last_active":"2023-05-22T16:05:21.949529+0000","last_peered":"2023-05-22T16:05:21.949529+0000","last_clean":"2023-05-22T16:05:21.949529+0000","last_became_active":"2023-05-22T16:05:21.566147+0000","last_became_peered":"2023-05-22T16:05:21.566147+0000","last_unstale":"2023-05-22T16:05:21.949529+0000","last_undegraded":"2023-05-22T16:05:21.949529+0000","last_fullsized":"2023-05-22T16:05:21.949529+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:05:20.505200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:05:20.505200+0000","last_clean_scrub_stamp":"2023-05-22T16:05:20.505200+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:05:01.227820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":184,"reported_epoch":194,"state":"active+clean","last_fresh":"2023-05-22T16:05:21.542445+0000","last_change":"2023-05-22T15:26:44.977198+0000","last_active":"2023-05-22T16:05:21.542445+0000","last_peered":"2023-05-22T16:05:21.542445+0000","last_clean":"2023-05-22T16:05:21.542445+0000","last_became_active":"2023-05-22T15:26:44.976445+0000","last_became_peered":"2023-05-22T15:26:44.976445+0000","last_unstale":"2023-05-22T16:05:21.542445+0000","last_undegraded":"2023-05-22T16:05:21.542445+0000","last_fullsized":"2023-05-22T16:05:21.542445+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:12:41.464054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.1a","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2023-05-22T16:05:21.952457+0000","last_change":"2023-05-22T16:05:21.950222+0000","last_active":"2023-05-22T16:05:21.952457+0000","last_peered":"2023-05-22T16:05:21.952457+0000","last_clean":"2023-05-22T16:05:21.952457+0000","last_became_active":"2023-05-22T16:05:21.948530+0000","last_became_peered":"2023-05-22T16:05:21.948530+0000","last_unstale":"2023-05-22T16:05:21.952457+0000","last_undegraded":"2023-05-22T16:05:21.952457+0000","last_fullsized":"2023-05-22T16:05:21.952457+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_clean_scrub_stamp":"2023-05-22T16:05:21.519005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:51:31.114141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.1b","version":"195'1","reported_seq":11,"reported_epoch":195,"state":"active+clean","last_fresh":"2023-05-22T16:05:21.941700+0000","last_change":"2023-05-22T16:05:21.571275+0000","last_active":"2023-05-22T16:05:21.941700+0000","last_peered":"2023-05-22T16:05:21.941700+0000","last_clean":"2023-05-22T16:05:21.941700+0000","last_became_active":"2023-05-22T16:05:21.570988+0000","last_became_peered":"2023-05-22T16:05:21.570988+0000","last_unstale":"2023-05-22T16:05:21.941700+0000","last_undegraded":"2023-05-22T16:05:21.941700+0000","last_fullsized":"2023-05-22T16:05:21.941700+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:05:20.505200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:05:20.505200+0000","last_clean_scrub_stamp":"2023-05-22T16:05:20.505200+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:54:20.347388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":188,"reported_epoch":194,"state":"active+clean","last_fresh":"2023-05-22T16:05:21.537715+0000","last_change":"2023-05-22T15:26:44.969469+0000","last_active":"2023-05-22T16:05:21.537715+0000","last_peered":"2023-05-22T16:05:21.537715+0000","last_clean":"2023-05-22T16:05:21.537715+0000","last_became_active":"2023-05-22T15:26:44.969083+0000","last_became_peered":"2023-05-22T15:26:44.969083+0000","last_unstale":"2023-05-22T16:05:21.537715+0000","last_undegraded":"2023-05-22T16:05:21.537715+0000","last_fullsized":"2023-05-22T16:05:21.537715+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:29:23.670067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.19","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2023-05-22T16:05:21.992099+0000","last_change":"2023-05-22T16:05:21.983557+0000","last_active":"2023-05-22T16:05:21.992099+0000","last_peered":"2023-05-22T16:05:21.992099+0000","last_clean":"2023-05-22T16:05:21.992099+0000","last_became_active":"2023-05-22T16:05:21.982644+0000","last_became_peered":"2023-05-22T16:05:21.982644+0000","last_unstale":"2023-05-22T16:05:21.992099+0000","last_undegraded":"2023-05-22T16:05:21.992099+0000","last_fullsized":"2023-05-22T16:05:21.992099+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_clean_scrub_stamp":"2023-05-22T16:05:21.519005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T03:02:26.541310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.18","version":"0'0","reported_seq":10,"reported_epoch":195,"state":"active+clean","last_fresh":"2023-05-22T16:05:21.956111+0000","last_change":"2023-05-22T16:05:21.568215+0000","last_active":"2023-05-22T16:05:21.956111+0000","last_peered":"2023-05-22T16:05:21.956111+0000","last_clean":"2023-05-22T16:05:21.956111+0000","last_became_active":"2023-05-22T16:05:21.567241+0000","last_became_peered":"2023-05-22T16:05:21.567241+0000","last_unstale":"2023-05-22T16:05:21.956111+0000","last_undegraded":"2023-05-22T16:05:21.956111+0000","last_fullsized":"2023-05-22T16:05:21.956111+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:05:20.505200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:05:20.505200+0000","last_clean_scrub_stamp":"2023-05-22T16:05:20.505200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T01:54:29.426377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":195,"reported_epoch":194,"state":"active+clean","last_fresh":"2023-05-22T16:05:21.555395+0000","last_change":"2023-05-22T15:26:44.976632+0000","last_active":"2023-05-22T16:05:21.555395+0000","last_peered":"2023-05-22T16:05:21.555395+0000","last_clean":"2023-05-22T16:05:21.555395+0000","last_became_active":"2023-05-22T15:26:44.975546+0000","last_became_peered":"2023-05-22T15:26:44.975546+0000","last_unstale":"2023-05-22T16:05:21.555395+0000","last_undegraded":"2023-05-22T16:05:21.555395+0000","last_fullsized":"2023-05-22T16:05:21.555395+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:42:57.973030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.1f","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2023-05-22T16:05:21.950112+0000","last_change":"2023-05-22T16:05:21.942024+0000","last_active":"2023-05-22T16:05:21.950112+0000","last_peered":"2023-05-22T16:05:21.950112+0000","last_clean":"2023-05-22T16:05:21.950112+0000","last_became_active":"2023-05-22T16:05:21.941728+0000","last_became_peered":"2023-05-22T16:05:21.941728+0000","last_unstale":"2023-05-22T16:05:21.950112+0000","last_undegraded":"2023-05-22T16:05:21.950112+0000","last_fullsized":"2023-05-22T16:05:21.950112+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_clean_scrub_stamp":"2023-05-22T16:05:21.519005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:41:43.814799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.1e","version":"195'1","reported_seq":11,"reported_epoch":195,"state":"active+clean","last_fresh":"2023-05-22T16:05:21.997614+0000","last_change":"2023-05-22T16:05:21.574643+0000","last_active":"2023-05-22T16:05:21.997614+0000","last_peered":"2023-05-22T16:05:21.997614+0000","last_clean":"2023-05-22T16:05:21.997614+0000","last_became_active":"2023-05-22T16:05:21.573908+0000","last_became_peered":"2023-05-22T16:05:21.573908+0000","last_unstale":"2023-05-22T16:05:21.997614+0000","last_undegraded":"2023-05-22T16:05:21.997614+0000","last_fullsized":"2023-05-22T16:05:21.997614+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:05:20.505200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:05:20.505200+0000","last_clean_scrub_stamp":"2023-05-22T16:05:20.505200+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:22:43.835901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":173,"reported_epoch":194,"state":"active+clean","last_fresh":"2023-05-22T16:05:21.547232+0000","last_change":"2023-05-22T15:26:45.091475+0000","last_active":"2023-05-22T16:05:21.547232+0000","last_peered":"2023-05-22T16:05:21.547232+0000","last_clean":"2023-05-22T16:05:21.547232+0000","last_became_active":"2023-05-22T15:26:45.091143+0000","last_became_peered":"2023-05-22T15:26:45.091143+0000","last_unstale":"2023-05-22T16:05:21.547232+0000","last_undegraded":"2023-05-22T16:05:21.547232+0000","last_fullsized":"2023-05-22T16:05:21.547232+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:12:41.464054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.1e","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2023-05-22T16:05:21.972457+0000","last_change":"2023-05-22T16:05:21.971302+0000","last_active":"2023-05-22T16:05:21.972457+0000","last_peered":"2023-05-22T16:05:21.972457+0000","last_clean":"2023-05-22T16:05:21.972457+0000","last_became_active":"2023-05-22T16:05:21.970903+0000","last_became_peered":"2023-05-22T16:05:21.970903+0000","last_unstale":"2023-05-22T16:05:21.972457+0000","last_undegraded":"2023-05-22T16:05:21.972457+0000","last_fullsized":"2023-05-22T16:05:21.972457+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_clean_scrub_stamp":"2023-05-22T16:05:21.519005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T22:46:30.583073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1f","version":"195'4","reported_seq":14,"reported_epoch":195,"state":"active+clean","last_fresh":"2023-05-22T16:05:21.985933+0000","last_change":"2023-05-22T16:05:21.577808+0000","last_active":"2023-05-22T16:05:21.985933+0000","last_peered":"2023-05-22T16:05:21.985933+0000","last_clean":"2023-05-22T16:05:21.985933+0000","last_became_active":"2023-05-22T16:05:21.577440+0000","last_became_peered":"2023-05-22T16:05:21.577440+0000","last_unstale":"2023-05-22T16:05:21.985933+0000","last_undegraded":"2023-05-22T16:05:21.985933+0000","last_fullsized":"2023-05-22T16:05:21.985933+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:05:20.505200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:05:20.505200+0000","last_clean_scrub_stamp":"2023-05-22T16:05:20.505200+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:44:33.016687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":566,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":183,"reported_epoch":194,"state":"active+clean","last_fresh":"2023-05-22T16:05:21.543449+0000","last_change":"2023-05-22T15:26:45.092367+0000","last_active":"2023-05-22T16:05:21.543449+0000","last_peered":"2023-05-22T16:05:21.543449+0000","last_clean":"2023-05-22T16:05:21.543449+0000","last_became_active":"2023-05-22T15:26:45.091951+0000","last_became_peered":"2023-05-22T15:26:45.091951+0000","last_unstale":"2023-05-22T16:05:21.543449+0000","last_undegraded":"2023-05-22T16:05:21.543449+0000","last_fullsized":"2023-05-22T16:05:21.543449+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:42:57.973030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.1d","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2023-05-22T16:05:21.992938+0000","last_change":"2023-05-22T16:05:21.988728+0000","last_active":"2023-05-22T16:05:21.992938+0000","last_peered":"2023-05-22T16:05:21.992938+0000","last_clean":"2023-05-22T16:05:21.992938+0000","last_became_active":"2023-05-22T16:05:21.988029+0000","last_became_peered":"2023-05-22T16:05:21.988029+0000","last_unstale":"2023-05-22T16:05:21.992938+0000","last_undegraded":"2023-05-22T16:05:21.992938+0000","last_fullsized":"2023-05-22T16:05:21.992938+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_clean_scrub_stamp":"2023-05-22T16:05:21.519005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:56:19.066496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.1c","version":"195'2","reported_seq":12,"reported_epoch":195,"state":"active+clean","last_fresh":"2023-05-22T16:05:21.986407+0000","last_change":"2023-05-22T16:05:21.583033+0000","last_active":"2023-05-22T16:05:21.986407+0000","last_peered":"2023-05-22T16:05:21.986407+0000","last_clean":"2023-05-22T16:05:21.986407+0000","last_became_active":"2023-05-22T16:05:21.582280+0000","last_became_peered":"2023-05-22T16:05:21.582280+0000","last_unstale":"2023-05-22T16:05:21.986407+0000","last_undegraded":"2023-05-22T16:05:21.986407+0000","last_fullsized":"2023-05-22T16:05:21.986407+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:05:20.505200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:05:20.505200+0000","last_clean_scrub_stamp":"2023-05-22T16:05:20.505200+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T18:11:17.455705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":172,"reported_epoch":194,"state":"active+clean","last_fresh":"2023-05-22T16:05:21.547475+0000","last_change":"2023-05-22T15:26:44.978858+0000","last_active":"2023-05-22T16:05:21.547475+0000","last_peered":"2023-05-22T16:05:21.547475+0000","last_clean":"2023-05-22T16:05:21.547475+0000","last_became_active":"2023-05-22T15:26:44.978443+0000","last_became_peered":"2023-05-22T15:26:44.978443+0000","last_unstale":"2023-05-22T16:05:21.547475+0000","last_undegraded":"2023-05-22T16:05:21.547475+0000","last_fullsized":"2023-05-22T16:05:21.547475+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:46:29.290243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.1c","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2023-05-22T16:05:21.987202+0000","last_change":"2023-05-22T16:05:21.960112+0000","last_active":"2023-05-22T16:05:21.987202+0000","last_peered":"2023-05-22T16:05:21.987202+0000","last_clean":"2023-05-22T16:05:21.987202+0000","last_became_active":"2023-05-22T16:05:21.959360+0000","last_became_peered":"2023-05-22T16:05:21.959360+0000","last_unstale":"2023-05-22T16:05:21.987202+0000","last_undegraded":"2023-05-22T16:05:21.987202+0000","last_fullsized":"2023-05-22T16:05:21.987202+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:05:21.519005+0000","last_clean_scrub_stamp":"2023-05-22T16:05:21.519005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:21:04.462454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.1d","version":"195'2","reported_seq":12,"reported_epoch":195,"state":"active+clean","last_fresh":"2023-05-22T16:05:21.952541+0000","last_change":"2023-05-22T16:05:21.574994+0000","last_active":"2023-05-22T16:05:21.952541+0000","last_peered":"2023-05-22T16:05:21.952541+0000","last_clean":"2023-05-22T16:05:21.952541+0000","last_became_active":"2023-05-22T16:05:21.574678+0000","last_became_peered":"2023-05-22T16:05:21.574678+0000","last_unstale":"2023-05-22T16:05:21.952541+0000","last_undegraded":"2023-05-22T16:05:21.952541+0000","last_fullsized":"2023-05-22T16:05:21.952541+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:05:20.505200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:05:20.505200+0000","last_clean_scrub_stamp":"2023-05-22T16:05:20.505200+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:32:04.944239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":171,"reported_epoch":194,"state":"active+clean","last_fresh":"2023-05-22T16:05:21.564660+0000","last_change":"2023-05-22T15:26:44.982421+0000","last_active":"2023-05-22T16:05:21.564660+0000","last_peered":"2023-05-22T16:05:21.564660+0000","last_clean":"2023-05-22T16:05:21.564660+0000","last_became_active":"2023-05-22T15:26:44.981586+0000","last_became_peered":"2023-05-22T15:26:44.981586+0000","last_unstale":"2023-05-22T16:05:21.564660+0000","last_undegraded":"2023-05-22T16:05:21.564660+0000","last_fullsized":"2023-05-22T16:05:21.564660+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:14:11.156544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":25,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":24,"num_pg":32,"stat_sum":{"num_bytes":2330,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1940,"num_read_kb":4014,"num_write":539,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":326,"ondisk_log_size":326,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575307,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29072,"kb_used_data":2952,"kb_used_omap":0,"kb_used_meta":26112,"kb_avail":94342768,"statfs":{"total":96636764160,"available":96606994432,"internally_reserved":0,"allocated":3022848,"data_stored":1889474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":26738688},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 16:04:48 2023","interfaces":[{"interface":"back","average":{"1min":0.81100000000000005,"5min":0.86199999999999999,"15min":0.86299999999999999},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51100000000000001},"max":{"1min":1.0660000000000001,"5min":1.3069999999999999,"15min":1.3069999999999999},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.87,"5min":0.88,"15min":0.88},"min":{"1min":0.63200000000000001,"5min":0.501,"15min":0.501},"max":{"1min":1.1439999999999999,"5min":1.2529999999999999,"15min":1.2529999999999999},"last":0.72699999999999998}]},{"osd":1,"last update":"Mon May 22 16:04:48 2023","interfaces":[{"interface":"back","average":{"1min":0.98499999999999999,"5min":0.99099999999999999,"15min":0.99399999999999999},"min":{"1min":0.73199999999999998,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.2450000000000001,"5min":1.431,"15min":1.431},"last":0.93700000000000006},{"interface":"front","average":{"1min":1.0089999999999999,"5min":1.04,"15min":1.0349999999999999},"min":{"1min":0.71099999999999997,"5min":0.66300000000000003,"15min":0.626},"max":{"1min":1.262,"5min":1.345,"15min":1.345},"last":0.67000000000000004}]},{"osd":2,"last update":"Mon May 22 16:04:48 2023","interfaces":[{"interface":"back","average":{"1min":0.92400000000000004,"5min":0.92000000000000004,"15min":0.90900000000000003},"min":{"1min":0.65000000000000002,"5min":0.63200000000000001,"15min":0.63200000000000001},"max":{"1min":1.1910000000000001,"5min":1.286,"15min":1.407},"last":0.81499999999999995},{"interface":"front","average":{"1min":0.90900000000000003,"5min":0.96499999999999997,"15min":0.96299999999999997},"min":{"1min":0.69499999999999995,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":1.089,"5min":1.246,"15min":1.2829999999999999},"last":0.628}]},{"osd":3,"last update":"Mon May 22 16:04:48 2023","interfaces":[{"interface":"back","average":{"1min":0.90500000000000003,"5min":0.93000000000000005,"15min":0.94499999999999995},"min":{"1min":0.61099999999999999,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.252,"5min":1.2909999999999999,"15min":1.401},"last":0.78200000000000003},{"interface":"front","average":{"1min":0.92500000000000004,"5min":1.0009999999999999,"15min":0.98199999999999998},"min":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":1.286,"5min":1.411,"15min":1.411},"last":1.036}]},{"osd":4,"last update":"Mon May 22 16:04:48 2023","interfaces":[{"interface":"back","average":{"1min":0.86899999999999999,"5min":0.88,"15min":0.88700000000000001},"min":{"1min":0.57299999999999995,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":1.127,"5min":1.3440000000000001,"15min":1.3440000000000001},"last":0.85999999999999999},{"interface":"front","average":{"1min":0.95199999999999996,"5min":0.98499999999999999,"15min":0.96499999999999997},"min":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":1.1819999999999999,"5min":1.405,"15min":1.405},"last":0.88100000000000001}]},{"osd":5,"last update":"Mon May 22 16:04:48 2023","interfaces":[{"interface":"back","average":{"1min":1.002,"5min":0.98999999999999999,"15min":0.99299999999999999},"min":{"1min":0.66200000000000003,"5min":0.623,"15min":0.441},"max":{"1min":1.25,"5min":1.2909999999999999,"15min":1.3759999999999999},"last":1.129},{"interface":"front","average":{"1min":1.159,"5min":1.1339999999999999,"15min":1.131},"min":{"1min":0.89500000000000002,"5min":0.72299999999999998,"15min":0.59799999999999998},"max":{"1min":1.345,"5min":1.3839999999999999,"15min":1.413},"last":0.91700000000000004}]},{"osd":6,"last update":"Mon May 22 16:04:48 2023","interfaces":[{"interface":"back","average":{"1min":1.0489999999999999,"5min":1.0469999999999999,"15min":1.0580000000000001},"min":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":1.256,"5min":1.3169999999999999,"15min":1.389},"last":0.97899999999999998},{"interface":"front","average":{"1min":1.127,"5min":1.103,"15min":1.093},"min":{"1min":0.81699999999999995,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":1.3500000000000001,"5min":1.391,"15min":1.7050000000000001},"last":1.002}]}]},{"osd":6,"up_from":13,"seq":55834575308,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":22616,"kb_used_data":2384,"kb_used_omap":0,"kb_used_meta":20224,"kb_avail":94349224,"statfs":{"total":96636764160,"available":96613605376,"internally_reserved":0,"allocated":2441216,"data_stored":1306610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":20709376},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 16:04:55 2023","interfaces":[{"interface":"back","average":{"1min":0.93899999999999995,"5min":0.873,"15min":0.90300000000000002},"min":{"1min":0.65500000000000003,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":1.3089999999999999,"5min":1.3089999999999999,"15min":1.4399999999999999},"last":2.6469999999999998},{"interface":"front","average":{"1min":0.91200000000000003,"5min":0.89600000000000002,"15min":0.91200000000000003},"min":{"1min":0.621,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":1.2010000000000001,"5min":1.427,"15min":1.427},"last":6.7439999999999998}]},{"osd":1,"last update":"Mon May 22 16:04:55 2023","interfaces":[{"interface":"back","average":{"1min":0.94499999999999995,"5min":0.91700000000000004,"15min":0.93999999999999995},"min":{"1min":0.61899999999999999,"5min":0.55300000000000005,"15min":0.51100000000000001},"max":{"1min":1.2949999999999999,"5min":1.3100000000000001,"15min":1.3380000000000001},"last":2.3290000000000002},{"interface":"front","average":{"1min":0.92800000000000005,"5min":0.92300000000000004,"15min":0.93000000000000005},"min":{"1min":0.68600000000000005,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.1279999999999999,"5min":1.2849999999999999,"15min":1.2849999999999999},"last":6.5010000000000003}]},{"osd":2,"last update":"Mon May 22 16:04:55 2023","interfaces":[{"interface":"back","average":{"1min":0.98099999999999998,"5min":0.97799999999999998,"15min":0.996},"min":{"1min":0.63700000000000001,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":1.2589999999999999,"5min":1.423,"15min":1.423},"last":7.0170000000000003},{"interface":"front","average":{"1min":1.04,"5min":1.056,"15min":1.071},"min":{"1min":0.72799999999999998,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":1.242,"5min":1.6599999999999999,"15min":1.6599999999999999},"last":2.403}]},{"osd":3,"last update":"Mon May 22 16:04:55 2023","interfaces":[{"interface":"back","average":{"1min":1.0009999999999999,"5min":0.998,"15min":0.995},"min":{"1min":0.76400000000000001,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.266,"5min":1.51,"15min":1.51},"last":6.6699999999999999},{"interface":"front","average":{"1min":0.95399999999999996,"5min":0.96599999999999997,"15min":0.97199999999999998},"min":{"1min":0.76800000000000002,"5min":0.622,"15min":0.622},"max":{"1min":1.286,"5min":1.286,"15min":1.3779999999999999},"last":7.1260000000000003}]},{"osd":4,"last update":"Mon May 22 16:04:55 2023","interfaces":[{"interface":"back","average":{"1min":0.92700000000000005,"5min":0.92600000000000005,"15min":0.95599999999999996},"min":{"1min":0.63300000000000001,"5min":0.62,"15min":0.61499999999999999},"max":{"1min":1.3540000000000001,"5min":1.4550000000000001,"15min":1.5409999999999999},"last":4.0709999999999997},{"interface":"front","average":{"1min":1.0389999999999999,"5min":1.0600000000000001,"15min":1.04},"min":{"1min":0.67000000000000004,"5min":0.63300000000000001,"15min":0.622},"max":{"1min":1.278,"5min":1.589,"15min":1.589},"last":11.581}]},{"osd":5,"last update":"Mon May 22 16:04:55 2023","interfaces":[{"interface":"back","average":{"1min":0.96099999999999997,"5min":0.98999999999999999,"15min":0.98899999999999999},"min":{"1min":0.71199999999999997,"5min":0.64300000000000002,"15min":0.61799999999999999},"max":{"1min":1.363,"5min":1.397,"15min":1.458},"last":17.587},{"interface":"front","average":{"1min":1.0649999999999999,"5min":1.093,"15min":1.1200000000000001},"min":{"1min":0.78600000000000003,"5min":0.748,"15min":0.69699999999999995},"max":{"1min":1.425,"5min":2.0390000000000001,"15min":2.0390000000000001},"last":2.9620000000000002}]},{"osd":7,"last update":"Mon May 22 16:04:55 2023","interfaces":[{"interface":"back","average":{"1min":1.1579999999999999,"5min":1.139,"15min":1.1339999999999999},"min":{"1min":0.95199999999999996,"5min":0.64200000000000002,"15min":0.58599999999999997},"max":{"1min":1.3580000000000001,"5min":1.746,"15min":1.746},"last":6.8339999999999996},{"interface":"front","average":{"1min":1.0669999999999999,"5min":1.1220000000000001,"15min":1.133},"min":{"1min":0.84999999999999998,"5min":0.78800000000000003,"15min":0.746},"max":{"1min":1.325,"5min":1.605,"15min":1.669},"last":8.5990000000000002}]}]},{"osd":5,"up_from":13,"seq":55834575308,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32728,"kb_used_data":2384,"kb_used_omap":0,"kb_used_meta":30336,"kb_avail":94339112,"statfs":{"total":96636764160,"available":96603250688,"internally_reserved":0,"allocated":2441216,"data_stored":1306610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31064064},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 16:04:49 2023","interfaces":[{"interface":"back","average":{"1min":0.80900000000000005,"5min":0.877,"15min":0.88800000000000001},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.1180000000000001,"5min":1.278,"15min":1.3959999999999999},"last":0.47899999999999998},{"interface":"front","average":{"1min":1.044,"5min":0.95199999999999996,"15min":0.94599999999999995},"min":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.502},"max":{"1min":1.27,"5min":1.325,"15min":1.325},"last":0.95899999999999996}]},{"osd":1,"last update":"Mon May 22 16:04:49 2023","interfaces":[{"interface":"back","average":{"1min":0.96599999999999997,"5min":1,"15min":1.012},"min":{"1min":0.69299999999999995,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":1.234,"5min":1.365,"15min":1.365},"last":0.65200000000000002},{"interface":"front","average":{"1min":1.0149999999999999,"5min":1.0629999999999999,"15min":1.073},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":1.3220000000000001,"5min":1.3520000000000001,"15min":1.4630000000000001},"last":0.70599999999999996}]},{"osd":2,"last update":"Mon May 22 16:04:49 2023","interfaces":[{"interface":"back","average":{"1min":0.92900000000000005,"5min":0.90800000000000003,"15min":0.92500000000000004},"min":{"1min":0.48699999999999999,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.3160000000000001,"5min":1.3200000000000001,"15min":1.3740000000000001},"last":0.81000000000000005},{"interface":"front","average":{"1min":0.94699999999999995,"5min":0.97399999999999998,"15min":0.95299999999999996},"min":{"1min":0.65600000000000003,"5min":0.64700000000000002,"15min":0.64700000000000002},"max":{"1min":1.244,"5min":1.4350000000000001,"15min":1.4350000000000001},"last":0.92600000000000005}]},{"osd":3,"last update":"Mon May 22 16:04:49 2023","interfaces":[{"interface":"back","average":{"1min":0.95699999999999996,"5min":0.97399999999999998,"15min":0.97899999999999998},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":1.349,"5min":1.349,"15min":1.367},"last":0.88800000000000001},{"interface":"front","average":{"1min":1.0840000000000001,"5min":1.0429999999999999,"15min":1.054},"min":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.3899999999999999,"5min":1.3899999999999999,"15min":1.3899999999999999},"last":1.03}]},{"osd":4,"last update":"Mon May 22 16:04:49 2023","interfaces":[{"interface":"back","average":{"1min":0.96299999999999997,"5min":0.93500000000000005,"15min":0.93100000000000005},"min":{"1min":0.66900000000000004,"5min":0.61299999999999999,"15min":0.61299999999999999},"max":{"1min":1.429,"5min":1.6000000000000001,"15min":1.6000000000000001},"last":0.78000000000000003},{"interface":"front","average":{"1min":0.997,"5min":0.99299999999999999,"15min":1.016},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":1.409,"5min":1.4630000000000001,"15min":1.4630000000000001},"last":1.0529999999999999}]},{"osd":6,"last update":"Mon May 22 16:04:49 2023","interfaces":[{"interface":"back","average":{"1min":1.028,"5min":1.004,"15min":1.0189999999999999},"min":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.59799999999999998},"max":{"1min":1.4470000000000001,"5min":1.4630000000000001,"15min":1.4790000000000001},"last":1.145},{"interface":"front","average":{"1min":1.0820000000000001,"5min":1.083,"15min":1.0880000000000001},"min":{"1min":0.748,"5min":0.67300000000000004,"15min":0.67300000000000004},"max":{"1min":1.6080000000000001,"5min":1.6080000000000001,"15min":1.6080000000000001},"last":0.85499999999999998}]},{"osd":7,"last update":"Mon May 22 16:04:49 2023","interfaces":[{"interface":"back","average":{"1min":1.1220000000000001,"5min":1.1200000000000001,"15min":1.129},"min":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"max":{"1min":1.51,"5min":1.51,"15min":1.5109999999999999},"last":0.995},{"interface":"front","average":{"1min":1.2370000000000001,"5min":1.1890000000000001,"15min":1.1970000000000001},"min":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":1.6679999999999999,"5min":1.6679999999999999,"15min":1.7210000000000001},"last":1.353}]}]},{"osd":4,"up_from":13,"seq":55834575306,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35304,"kb_used_data":2400,"kb_used_omap":0,"kb_used_meta":32896,"kb_avail":94336536,"statfs":{"total":96636764160,"available":96600612864,"internally_reserved":0,"allocated":2457600,"data_stored":1308724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":33685504},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 16:05:02 2023","interfaces":[{"interface":"back","average":{"1min":0.90400000000000003,"5min":0.89600000000000002,"15min":0.90000000000000002},"min":{"1min":0.59999999999999998,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.1870000000000001,"5min":1.7210000000000001,"15min":1.7210000000000001},"last":1.1439999999999999},{"interface":"front","average":{"1min":0.995,"5min":0.96599999999999997,"15min":0.95199999999999996},"min":{"1min":0.70899999999999996,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":1.3360000000000001,"5min":1.7290000000000001,"15min":1.7290000000000001},"last":0.67700000000000005}]},{"osd":1,"last update":"Mon May 22 16:05:02 2023","interfaces":[{"interface":"back","average":{"1min":1.0269999999999999,"5min":0.97599999999999998,"15min":0.95899999999999996},"min":{"1min":0.76900000000000002,"5min":0.314,"15min":0.314},"max":{"1min":1.258,"5min":1.9339999999999999,"15min":1.9339999999999999},"last":0.76900000000000002},{"interface":"front","average":{"1min":0.90000000000000002,"5min":0.92800000000000005,"15min":0.93999999999999995},"min":{"1min":0.68799999999999994,"5min":0.504,"15min":0.504},"max":{"1min":1.0900000000000001,"5min":1.365,"15min":1.365},"last":0.89200000000000002}]},{"osd":2,"last update":"Mon May 22 16:05:02 2023","interfaces":[{"interface":"back","average":{"1min":0.98599999999999999,"5min":0.95499999999999996,"15min":0.94999999999999996},"min":{"1min":0.71799999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":1.28,"5min":1.4990000000000001,"15min":1.4990000000000001},"last":0.81000000000000005},{"interface":"front","average":{"1min":1.056,"5min":1.036,"15min":1.052},"min":{"1min":0.76000000000000001,"5min":0.44,"15min":0.44},"max":{"1min":1.2529999999999999,"5min":2.0249999999999999,"15min":2.0249999999999999},"last":1.0329999999999999}]},{"osd":3,"last update":"Mon May 22 16:05:02 2023","interfaces":[{"interface":"back","average":{"1min":1.0329999999999999,"5min":0.96499999999999997,"15min":0.96599999999999997},"min":{"1min":0.79200000000000004,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":1.258,"5min":1.4610000000000001,"15min":1.4610000000000001},"last":1.1890000000000001},{"interface":"front","average":{"1min":1.0760000000000001,"5min":1.0609999999999999,"15min":1.0489999999999999},"min":{"1min":0.83899999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":1.323,"5min":1.349,"15min":1.484},"last":1.278}]},{"osd":5,"last update":"Mon May 22 16:05:02 2023","interfaces":[{"interface":"back","average":{"1min":1.004,"5min":1.036,"15min":1.002},"min":{"1min":0.69599999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":1.325,"5min":2.1110000000000002,"15min":2.1110000000000002},"last":0.94299999999999995},{"interface":"front","average":{"1min":1.0309999999999999,"5min":1.0189999999999999,"15min":0.996},"min":{"1min":0.69999999999999996,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.1990000000000001,"5min":1.5449999999999999,"15min":1.5449999999999999},"last":1.0620000000000001}]},{"osd":6,"last update":"Mon May 22 16:05:02 2023","interfaces":[{"interface":"back","average":{"1min":1.0669999999999999,"5min":1.036,"15min":1.0009999999999999},"min":{"1min":0.75800000000000001,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":1.387,"5min":1.397,"15min":1.4339999999999999},"last":0.98299999999999998},{"interface":"front","average":{"1min":1.1599999999999999,"5min":1.1599999999999999,"15min":1.137},"min":{"1min":0.875,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.3340000000000001,"5min":2.2690000000000001,"15min":2.2690000000000001},"last":1.1160000000000001}]},{"osd":7,"last update":"Mon May 22 16:05:02 2023","interfaces":[{"interface":"back","average":{"1min":1.093,"5min":1.0960000000000001,"15min":1.069},"min":{"1min":0.83599999999999997,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":1.3029999999999999,"5min":1.623,"15min":1.623},"last":1.2370000000000001},{"interface":"front","average":{"1min":1.2150000000000001,"5min":1.1879999999999999,"15min":1.1599999999999999},"min":{"1min":1.0169999999999999,"5min":0.71199999999999997,"15min":0.68100000000000005},"max":{"1min":1.4390000000000001,"5min":1.448,"15min":1.5089999999999999},"last":1.367}]}]},{"osd":3,"up_from":13,"seq":55834575306,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":2396,"kb_used_omap":0,"kb_used_meta":24832,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":2453504,"data_stored":1306804,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":25427968},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 16:04:38 2023","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.63100000000000001,"15min":0.63600000000000001},"min":{"1min":0.374,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.91800000000000004,"5min":1.0049999999999999,"15min":1.0229999999999999},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.82599999999999996,"5min":0.77900000000000003,"15min":0.77400000000000002},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":1.7230000000000001,"5min":1.7230000000000001,"15min":1.7230000000000001},"last":0.50900000000000001}]},{"osd":1,"last update":"Mon May 22 16:04:38 2023","interfaces":[{"interface":"back","average":{"1min":0.81299999999999994,"5min":0.76800000000000002,"15min":0.72199999999999998},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":2.1280000000000001,"5min":2.4969999999999999,"15min":2.4969999999999999},"last":1.056},{"interface":"front","average":{"1min":0.79800000000000004,"5min":0.80600000000000005,"15min":0.80600000000000005},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":1.395,"5min":2.4769999999999999,"15min":2.4769999999999999},"last":0.77200000000000002}]},{"osd":2,"last update":"Mon May 22 16:04:38 2023","interfaces":[{"interface":"back","average":{"1min":0.88100000000000001,"5min":0.85399999999999998,"15min":0.85699999999999998},"min":{"1min":0.47699999999999998,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":2.2250000000000001,"5min":2.2250000000000001,"15min":2.2250000000000001},"last":0.67700000000000005},{"interface":"front","average":{"1min":0.84699999999999998,"5min":0.82699999999999996,"15min":0.83499999999999996},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":2.1680000000000001,"5min":2.1680000000000001,"15min":2.1680000000000001},"last":0.79800000000000004}]},{"osd":4,"last update":"Mon May 22 16:04:38 2023","interfaces":[{"interface":"back","average":{"1min":1.042,"5min":0.98499999999999999,"15min":0.98099999999999998},"min":{"1min":0.69299999999999995,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":2.3479999999999999,"5min":2.3479999999999999,"15min":2.3479999999999999},"last":1.226},{"interface":"front","average":{"1min":1.0349999999999999,"5min":1.0229999999999999,"15min":1.034},"min":{"1min":0.54300000000000004,"5min":0.37,"15min":0.37},"max":{"1min":1.788,"5min":1.788,"15min":1.788},"last":0.83099999999999996}]},{"osd":5,"last update":"Mon May 22 16:04:38 2023","interfaces":[{"interface":"back","average":{"1min":1.036,"5min":1.054,"15min":1.0549999999999999},"min":{"1min":0.48699999999999999,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":1.8129999999999999,"5min":1.8129999999999999,"15min":1.8129999999999999},"last":1.1870000000000001},{"interface":"front","average":{"1min":1,"5min":1.038,"15min":1.0469999999999999},"min":{"1min":0.57899999999999996,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":1.347,"5min":1.571,"15min":1.571},"last":0.95599999999999996}]},{"osd":6,"last update":"Mon May 22 16:04:38 2023","interfaces":[{"interface":"back","average":{"1min":1.0820000000000001,"5min":1.093,"15min":1.095},"min":{"1min":0.57099999999999995,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":1.76,"5min":1.76,"15min":1.76},"last":1.274},{"interface":"front","average":{"1min":1.0720000000000001,"5min":1.0940000000000001,"15min":1.105},"min":{"1min":0.64200000000000002,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":1.4930000000000001,"5min":1.754,"15min":1.754},"last":1.107}]},{"osd":7,"last update":"Mon May 22 16:04:38 2023","interfaces":[{"interface":"back","average":{"1min":1.1040000000000001,"5min":1.1240000000000001,"15min":1.133},"min":{"1min":0.60399999999999998,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.421,"5min":1.673,"15min":1.673},"last":1.2929999999999999},{"interface":"front","average":{"1min":1.1459999999999999,"5min":1.1240000000000001,"15min":1.115},"min":{"1min":0.79600000000000004,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":2.3079999999999998,"5min":2.3079999999999998,"15min":2.3079999999999998},"last":1.3839999999999999}]}]},{"osd":2,"up_from":13,"seq":55834575307,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31268,"kb_used_data":2396,"kb_used_omap":0,"kb_used_meta":28864,"kb_avail":94340572,"statfs":{"total":96636764160,"available":96604745728,"internally_reserved":0,"allocated":2453504,"data_stored":1307288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":29556736},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 16:04:32 2023","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.64800000000000002,"15min":0.67600000000000005},"min":{"1min":0.38,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.85999999999999999,"5min":1.1180000000000001,"15min":1.1519999999999999},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.745,"5min":0.74299999999999999,"15min":0.752},"min":{"1min":0.495,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":1.1659999999999999,"5min":1.1659999999999999,"15min":1.2090000000000001},"last":0.52700000000000002}]},{"osd":1,"last update":"Mon May 22 16:04:32 2023","interfaces":[{"interface":"back","average":{"1min":0.72199999999999998,"5min":0.75600000000000001,"15min":0.75900000000000001},"min":{"1min":0.42599999999999999,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":1.069,"5min":1.0800000000000001,"15min":1.099},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.80700000000000005,"5min":0.83499999999999996,"15min":0.82399999999999995},"min":{"1min":0.39200000000000002,"5min":0.311,"15min":0.311},"max":{"1min":1.099,"5min":1.1739999999999999,"15min":1.2589999999999999},"last":0.67600000000000005}]},{"osd":3,"last update":"Mon May 22 16:04:32 2023","interfaces":[{"interface":"back","average":{"1min":0.73499999999999999,"5min":0.76700000000000002,"15min":0.76000000000000001},"min":{"1min":0.36799999999999999,"5min":0.315,"15min":0.315},"max":{"1min":1.046,"5min":1.3540000000000001,"15min":1.3540000000000001},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.875,"5min":0.872,"15min":0.89800000000000002},"min":{"1min":0.45400000000000001,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":1.2949999999999999,"5min":1.2949999999999999,"15min":1.2949999999999999},"last":0.50800000000000001}]},{"osd":4,"last update":"Mon May 22 16:04:32 2023","interfaces":[{"interface":"back","average":{"1min":1.032,"5min":1.0269999999999999,"15min":1.0409999999999999},"min":{"1min":0.59099999999999997,"5min":0.441,"15min":0.441},"max":{"1min":1.516,"5min":1.516,"15min":1.6539999999999999},"last":0.70599999999999996},{"interface":"front","average":{"1min":1.036,"5min":1.0209999999999999,"15min":1.0429999999999999},"min":{"1min":0.76500000000000001,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.4299999999999999,"5min":1.4299999999999999,"15min":1.538},"last":0.89700000000000002}]},{"osd":5,"last update":"Mon May 22 16:04:32 2023","interfaces":[{"interface":"back","average":{"1min":1.024,"5min":1.026,"15min":1.0580000000000001},"min":{"1min":0.625,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":1.2829999999999999,"5min":1.3300000000000001,"15min":1.605},"last":1.0129999999999999},{"interface":"front","average":{"1min":1.0249999999999999,"5min":1.0269999999999999,"15min":1.034},"min":{"1min":0.78400000000000003,"5min":0.67700000000000005,"15min":0.67700000000000005},"max":{"1min":1.534,"5min":1.534,"15min":1.534},"last":0.79800000000000004}]},{"osd":6,"last update":"Mon May 22 16:04:32 2023","interfaces":[{"interface":"back","average":{"1min":1.0469999999999999,"5min":1.04,"15min":1.03},"min":{"1min":0.64400000000000002,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":1.4219999999999999,"5min":1.429,"15min":1.462},"last":0.96299999999999997},{"interface":"front","average":{"1min":1.1100000000000001,"5min":1.0980000000000001,"15min":1.135},"min":{"1min":0.69999999999999996,"5min":0.57099999999999995,"15min":0.57099999999999995},"max":{"1min":1.621,"5min":1.621,"15min":1.7110000000000001},"last":0.98899999999999999}]},{"osd":7,"last update":"Mon May 22 16:04:32 2023","interfaces":[{"interface":"back","average":{"1min":1.1279999999999999,"5min":1.095,"15min":1.1259999999999999},"min":{"1min":0.77600000000000002,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":1.393,"5min":1.466,"15min":1.6399999999999999},"last":1.052},{"interface":"front","average":{"1min":1.083,"5min":1.103,"15min":1.117},"min":{"1min":0.60999999999999999,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":1.3440000000000001,"5min":1.673,"15min":1.673},"last":0.872}]}]},{"osd":1,"up_from":13,"seq":55834575307,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37092,"kb_used_data":2396,"kb_used_omap":0,"kb_used_meta":34688,"kb_avail":94334748,"statfs":{"total":96636764160,"available":96598781952,"internally_reserved":0,"allocated":2453504,"data_stored":1308158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":35520512},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 16:04:55 2023","interfaces":[{"interface":"back","average":{"1min":0.71799999999999997,"5min":0.65500000000000003,"15min":0.66300000000000003},"min":{"1min":0.497,"5min":0.308,"15min":0.308},"max":{"1min":0.96099999999999997,"5min":1.151,"15min":1.329},"last":0.71099999999999997},{"interface":"front","average":{"1min":0.755,"5min":0.74399999999999999,"15min":0.76800000000000002},"min":{"1min":0.56299999999999994,"5min":0.502,"15min":0.502},"max":{"1min":1.25,"5min":1.25,"15min":1.268},"last":0.68200000000000005}]},{"osd":2,"last update":"Mon May 22 16:04:55 2023","interfaces":[{"interface":"back","average":{"1min":0.72399999999999998,"5min":0.74399999999999999,"15min":0.75800000000000001},"min":{"1min":0.44800000000000001,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":1.0700000000000001,"5min":1.0880000000000001,"15min":1.26},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.81100000000000005,"5min":0.77300000000000002,"15min":0.78200000000000003},"min":{"1min":0.65600000000000003,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":1.0660000000000001,"5min":1.1299999999999999,"15min":1.268},"last":0.77500000000000002}]},{"osd":3,"last update":"Mon May 22 16:04:55 2023","interfaces":[{"interface":"back","average":{"1min":0.86499999999999999,"5min":0.86299999999999999,"15min":0.88300000000000001},"min":{"1min":0.60399999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":1.1859999999999999,"5min":1.2829999999999999,"15min":1.373},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.86499999999999999,"5min":0.82999999999999996,"15min":0.82899999999999996},"min":{"1min":0.57099999999999995,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":1.171,"5min":1.2749999999999999,"15min":1.2749999999999999},"last":0.74199999999999999}]},{"osd":4,"last update":"Mon May 22 16:04:55 2023","interfaces":[{"interface":"back","average":{"1min":0.97999999999999998,"5min":0.97499999999999998,"15min":0.98599999999999999},"min":{"1min":0.73299999999999998,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":1.155,"5min":1.542,"15min":1.542},"last":0.81999999999999995},{"interface":"front","average":{"1min":0.95999999999999996,"5min":1.006,"15min":1.026},"min":{"1min":0.76800000000000002,"5min":0.434,"15min":0.434},"max":{"1min":1.153,"5min":1.327,"15min":1.4650000000000001},"last":1.264}]},{"osd":5,"last update":"Mon May 22 16:04:55 2023","interfaces":[{"interface":"back","average":{"1min":1.0029999999999999,"5min":1.04,"15min":1.05},"min":{"1min":0.81299999999999994,"5min":0.70399999999999996,"15min":0.70399999999999996},"max":{"1min":1.224,"5min":1.605,"15min":1.605},"last":1.3069999999999999},{"interface":"front","average":{"1min":1.093,"5min":1.0669999999999999,"15min":1.073},"min":{"1min":0.81299999999999994,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":1.3380000000000001,"5min":1.3919999999999999,"15min":1.5389999999999999},"last":0.86099999999999999}]},{"osd":6,"last update":"Mon May 22 16:04:55 2023","interfaces":[{"interface":"back","average":{"1min":1.0629999999999999,"5min":1.054,"15min":1.0669999999999999},"min":{"1min":0.92900000000000005,"5min":0.56799999999999995,"15min":0.56799999999999995},"max":{"1min":1.167,"5min":1.371,"15min":1.5249999999999999},"last":1.3640000000000001},{"interface":"front","average":{"1min":1.069,"5min":1.105,"15min":1.113},"min":{"1min":0.88100000000000001,"5min":0.77600000000000002,"15min":0.75700000000000001},"max":{"1min":1.304,"5min":1.4359999999999999,"15min":1.6479999999999999},"last":1.242}]},{"osd":7,"last update":"Mon May 22 16:04:55 2023","interfaces":[{"interface":"back","average":{"1min":1.1220000000000001,"5min":1.147,"15min":1.159},"min":{"1min":0.92900000000000005,"5min":0.72199999999999998,"15min":0.72199999999999998},"max":{"1min":1.3819999999999999,"5min":1.7030000000000001,"15min":1.7030000000000001},"last":1.1799999999999999},{"interface":"front","average":{"1min":1.131,"5min":1.1339999999999999,"15min":1.1479999999999999},"min":{"1min":0.82699999999999996,"5min":0.73599999999999999,"15min":0.73599999999999999},"max":{"1min":1.395,"5min":1.6439999999999999,"15min":1.6439999999999999},"last":1.036}]}]},{"osd":0,"up_from":13,"seq":55834575308,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32544,"kb_used_data":2968,"kb_used_omap":0,"kb_used_meta":29568,"kb_avail":94339296,"statfs":{"total":96636764160,"available":96603439104,"internally_reserved":0,"allocated":3039232,"data_stored":1897000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":30277632},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon May 22 16:04:59 2023","interfaces":[{"interface":"back","average":{"1min":0.754,"5min":0.71299999999999997,"15min":0.71099999999999997},"min":{"1min":0.55600000000000005,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":1.075,"5min":1.296,"15min":1.296},"last":0.70799999999999996},{"interface":"front","average":{"1min":0.81000000000000005,"5min":0.76200000000000001,"15min":0.75},"min":{"1min":0.53500000000000003,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.0840000000000001,"5min":1.181,"15min":1.3149999999999999},"last":0.80100000000000005}]},{"osd":2,"last update":"Mon May 22 16:04:59 2023","interfaces":[{"interface":"back","average":{"1min":0.81799999999999995,"5min":0.79800000000000004,"15min":0.78100000000000003},"min":{"1min":0.59799999999999998,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":1.2150000000000001,"5min":1.22,"15min":1.284},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.78900000000000003,"5min":0.81899999999999995,"15min":0.81799999999999995},"min":{"1min":0.58199999999999996,"5min":0.437,"15min":0.437},"max":{"1min":1.0009999999999999,"5min":1.323,"15min":1.323},"last":0.92700000000000005}]},{"osd":3,"last update":"Mon May 22 16:04:59 2023","interfaces":[{"interface":"back","average":{"1min":0.79000000000000004,"5min":0.77000000000000002,"15min":0.77800000000000002},"min":{"1min":0.54800000000000004,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.1859999999999999,"5min":1.2350000000000001,"15min":1.2350000000000001},"last":0.74299999999999999},{"interface":"front","average":{"1min":0.84599999999999997,"5min":0.879,"15min":0.88100000000000001},"min":{"1min":0.66800000000000004,"5min":0.377,"15min":0.377},"max":{"1min":1.0860000000000001,"5min":1.2949999999999999,"15min":1.331},"last":0.96199999999999997}]},{"osd":4,"last update":"Mon May 22 16:04:59 2023","interfaces":[{"interface":"back","average":{"1min":0.94799999999999995,"5min":0.95599999999999996,"15min":0.95699999999999996},"min":{"1min":0.76600000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"max":{"1min":1.1579999999999999,"5min":1.264,"15min":1.4059999999999999},"last":0.98599999999999999},{"interface":"front","average":{"1min":1.0960000000000001,"5min":1.073,"15min":1.0569999999999999},"min":{"1min":0.90500000000000003,"5min":0.63900000000000001,"15min":0.63900000000000001},"max":{"1min":1.343,"5min":1.401,"15min":1.482},"last":1.1200000000000001}]},{"osd":5,"last update":"Mon May 22 16:04:59 2023","interfaces":[{"interface":"back","average":{"1min":1.0349999999999999,"5min":1.0329999999999999,"15min":1.0429999999999999},"min":{"1min":0.79100000000000004,"5min":0.64600000000000002,"15min":0.64600000000000002},"max":{"1min":1.2190000000000001,"5min":1.325,"15min":1.6419999999999999},"last":1.0940000000000001},{"interface":"front","average":{"1min":1.0609999999999999,"5min":1.032,"15min":1.02},"min":{"1min":0.83799999999999997,"5min":0.66200000000000003,"15min":0.66200000000000003},"max":{"1min":1.252,"5min":1.413,"15min":1.458},"last":0.89400000000000002}]},{"osd":6,"last update":"Mon May 22 16:04:59 2023","interfaces":[{"interface":"back","average":{"1min":1.129,"5min":1.1160000000000001,"15min":1.117},"min":{"1min":0.88500000000000001,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":1.401,"5min":1.407,"15min":1.77},"last":0.85199999999999998},{"interface":"front","average":{"1min":1.141,"5min":1.1319999999999999,"15min":1.111},"min":{"1min":0.94399999999999995,"5min":0.68300000000000005,"15min":0.68300000000000005},"max":{"1min":1.5409999999999999,"5min":1.5409999999999999,"15min":1.5409999999999999},"last":1.0089999999999999}]},{"osd":7,"last update":"Mon May 22 16:04:59 2023","interfaces":[{"interface":"back","average":{"1min":1.2110000000000001,"5min":1.151,"15min":1.141},"min":{"1min":1.0309999999999999,"5min":0.63900000000000001,"15min":0.63900000000000001},"max":{"1min":1.6020000000000001,"5min":1.6020000000000001,"15min":1.6020000000000001},"last":1.145},{"interface":"front","average":{"1min":1.131,"5min":1.0960000000000001,"15min":1.1040000000000001},"min":{"1min":0.77400000000000002,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.4119999999999999,"5min":1.4650000000000001,"15min":1.4990000000000001},"last":1.0549999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2114,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-05-22T16:05:26.343 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-05-22T16:05:26.344 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2023-05-22T16:05:26.516 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.512+0000 7f801bb10640 1 Processor -- start 2023-05-22T16:05:26.516 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.512+0000 7f801bb10640 1 -- start start 2023-05-22T16:05:26.519 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.516+0000 7f801bb10640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8014159990 0x7f80140cf0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:26.519 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.516+0000 7f801bb10640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f801415e580 con 0x7f801415e0f0 2023-05-22T16:05:26.519 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.516+0000 7f801bb10640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80140cc090 con 0x7f80140ceac0 2023-05-22T16:05:26.519 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.516+0000 7f801bb10640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80140cc370 con 0x7f8014159990 2023-05-22T16:05:26.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.516+0000 7f80190ab640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f801415e0f0 0x7f80140cdeb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:48508/0 (socket says 172.21.15.16:48508) 2023-05-22T16:05:26.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.516+0000 7f80190ab640 1 -- 172.21.15.16:0/1696443949 learned_addr learned my addr 172.21.15.16:0/1696443949 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T16:05:26.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.516+0000 7f80198ac640 1 --2- 172.21.15.16:0/1696443949 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8014159990 0x7f80140cf0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:26.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.516+0000 7f80188aa640 1 -- 172.21.15.16:0/1696443949 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3315561163 0 0) 0x7f801415e580 con 0x7f801415e0f0 2023-05-22T16:05:26.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.516+0000 7f80188aa640 1 -- 172.21.15.16:0/1696443949 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8000003530 con 0x7f801415e0f0 2023-05-22T16:05:26.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.516+0000 7f80188aa640 1 -- 172.21.15.16:0/1696443949 <== mon.1 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3130448145 0 0) 0x7f80140cc090 con 0x7f80140ceac0 2023-05-22T16:05:26.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.516+0000 7f80188aa640 1 -- 172.21.15.16:0/1696443949 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f801415e580 con 0x7f80140ceac0 2023-05-22T16:05:26.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.516+0000 7f80188aa640 1 -- 172.21.15.16:0/1696443949 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1806578541 0 0) 0x7f8000003530 con 0x7f801415e0f0 2023-05-22T16:05:26.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.516+0000 7f80198ac640 1 -- 172.21.15.16:0/1696443949 >> v1:172.21.15.62:6789/0 conn(0x7f80140ceac0 legacy=0x7f80140cce50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:26.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.516+0000 7f80198ac640 1 -- 172.21.15.16:0/1696443949 >> v1:172.21.15.16:6789/0 conn(0x7f801415e0f0 legacy=0x7f80140cdeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:26.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.516+0000 7f80198ac640 1 -- 172.21.15.16:0/1696443949 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80140cc650 con 0x7f8014159990 2023-05-22T16:05:26.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.516+0000 7f80198ac640 1 --2- 172.21.15.16:0/1696443949 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8014159990 0x7f80140cf0d0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f80100097f0 tx=0x7f801001c3b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2e8a47e936c5b763 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:26.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.520+0000 7f80188aa640 1 -- 172.21.15.16:0/1696443949 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8004001680 con 0x7f8014159990 2023-05-22T16:05:26.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.520+0000 7f80188aa640 1 -- 172.21.15.16:0/1696443949 <== mon.2 v2:172.21.15.62:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80100a7050 con 0x7f8014159990 2023-05-22T16:05:26.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.520+0000 7f80188aa640 1 -- 172.21.15.16:0/1696443949 <== mon.2 v2:172.21.15.62:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8010004230 con 0x7f8014159990 2023-05-22T16:05:26.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.520+0000 7f801bb10640 1 -- 172.21.15.16:0/1696443949 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8014159990 msgr2=0x7f80140cf0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:26.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.520+0000 7f801bb10640 1 --2- 172.21.15.16:0/1696443949 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8014159990 0x7f80140cf0d0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f80100097f0 tx=0x7f801001c3b0 comp rx=0 tx=0).stop 2023-05-22T16:05:26.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.520+0000 7f801bb10640 1 -- 172.21.15.16:0/1696443949 shutdown_connections 2023-05-22T16:05:26.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.520+0000 7f801bb10640 1 --2- 172.21.15.16:0/1696443949 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8014159990 0x7f80140cf0d0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:26.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.520+0000 7f801bb10640 1 -- 172.21.15.16:0/1696443949 >> 172.21.15.16:0/1696443949 conn(0x7f801406dac0 msgr2=0x7f801406dec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:05:26.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.520+0000 7f801bb10640 1 -- 172.21.15.16:0/1696443949 shutdown_connections 2023-05-22T16:05:26.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.520+0000 7f801bb10640 1 -- 172.21.15.16:0/1696443949 wait complete. 2023-05-22T16:05:26.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.520+0000 7f801bb10640 1 Processor -- start 2023-05-22T16:05:26.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.520+0000 7f801bb10640 1 -- start start 2023-05-22T16:05:26.526 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.520+0000 7f801bb10640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f80140ceac0 0x7f8014150780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:26.526 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.520+0000 7f801bb10640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8014159990 0x7f8014150cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:26.526 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.520+0000 7f801bb10640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f801415e0f0 0x7f80141518b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:26.526 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.520+0000 7f801bb10640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8014145230 con 0x7f8014159990 2023-05-22T16:05:26.526 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.520+0000 7f801bb10640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f801415bf60 con 0x7f801415e0f0 2023-05-22T16:05:26.526 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.520+0000 7f801bb10640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f801415b350 con 0x7f80140ceac0 2023-05-22T16:05:26.527 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.520+0000 7f80190ab640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8014159990 0x7f8014150cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:26.527 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.520+0000 7f80190ab640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8014159990 0x7f8014150cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41740/0 (socket says 172.21.15.16:41740) 2023-05-22T16:05:26.527 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.520+0000 7f80190ab640 1 -- 172.21.15.16:0/865744185 learned_addr learned my addr 172.21.15.16:0/865744185 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:05:26.527 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.520+0000 7f80198ac640 1 --2- 172.21.15.16:0/865744185 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f80140ceac0 0x7f8014150780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:26.527 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.520+0000 7f801a0ad640 1 --2- 172.21.15.16:0/865744185 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f801415e0f0 0x7f80141518b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:26.527 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.520+0000 7f80190ab640 1 -- 172.21.15.16:0/865744185 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f80140ceac0 msgr2=0x7f8014150780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:26.527 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.520+0000 7f80190ab640 1 --2- 172.21.15.16:0/865744185 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f80140ceac0 0x7f8014150780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:26.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.520+0000 7f80190ab640 1 -- 172.21.15.16:0/865744185 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f801415e0f0 msgr2=0x7f80141518b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:26.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.520+0000 7f80190ab640 1 --2- 172.21.15.16:0/865744185 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f801415e0f0 0x7f80141518b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:26.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.520+0000 7f80190ab640 1 -- 172.21.15.16:0/865744185 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8010009420 con 0x7f8014159990 2023-05-22T16:05:26.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.520+0000 7f80198ac640 1 --2- 172.21.15.16:0/865744185 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f80140ceac0 0x7f8014150780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T16:05:26.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.520+0000 7f80190ab640 1 --2- 172.21.15.16:0/865744185 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8014159990 0x7f8014150cc0 secure :-1 s=READY pgs=2454 cs=0 l=1 rev1=1 crypto rx=0x7f800400c700 tx=0x7f800400cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:26.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.520+0000 7f800a7fc640 1 -- 172.21.15.16:0/865744185 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8004005ae0 con 0x7f8014159990 2023-05-22T16:05:26.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.520+0000 7f800a7fc640 1 -- 172.21.15.16:0/865744185 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8004005c40 con 0x7f8014159990 2023-05-22T16:05:26.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.520+0000 7f801bb10640 1 -- 172.21.15.16:0/865744185 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f801415af00 con 0x7f8014159990 2023-05-22T16:05:26.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.520+0000 7f800a7fc640 1 -- 172.21.15.16:0/865744185 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8004007070 con 0x7f8014159990 2023-05-22T16:05:26.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.520+0000 7f801bb10640 1 -- 172.21.15.16:0/865744185 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f801415ad70 con 0x7f8014159990 2023-05-22T16:05:26.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.524+0000 7f801bb10640 1 -- 172.21.15.16:0/865744185 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80140ce5c0 con 0x7f8014159990 2023-05-22T16:05:26.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.524+0000 7f800a7fc640 1 -- 172.21.15.16:0/865744185 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 92) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f80040071d0 con 0x7f8014159990 2023-05-22T16:05:26.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.524+0000 7f800a7fc640 1 --2- 172.21.15.16:0/865744185 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f7ff80654a0 0x7f7ff8067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:26.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.524+0000 7f800a7fc640 1 -- 172.21.15.16:0/865744185 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(196..196 src has 1..196) v4 ==== 6438+0+0 (secure 0 0 0) 0x7f8004082090 con 0x7f8014159990 2023-05-22T16:05:26.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.524+0000 7f80198ac640 1 --2- 172.21.15.16:0/865744185 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f7ff80654a0 0x7f7ff8067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:26.530 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.524+0000 7f80198ac640 1 --2- 172.21.15.16:0/865744185 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f7ff80654a0 0x7f7ff8067950 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f80100ad000 tx=0x7f8010008040 comp rx=0 tx=0).ready entity=mgr.10542 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:26.530 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.528+0000 7f800a7fc640 1 -- 172.21.15.16:0/865744185 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f800404a620 con 0x7f8014159990 2023-05-22T16:05:26.731 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.728+0000 7f801bb10640 1 -- 172.21.15.16:0/865744185 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f80140738f0 con 0x7f8014159990 2023-05-22T16:05:26.739 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.736+0000 7f800a7fc640 1 -- 172.21.15.16:0/865744185 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v34) v1 ==== 190+0+0 (secure 0 0 0) 0x7f80040535f0 con 0x7f8014159990 2023-05-22T16:05:26.740 INFO:teuthology.orchestra.run.smithi016.stderr:updated caps for client.0 2023-05-22T16:05:26.741 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.740+0000 7f801bb10640 1 -- 172.21.15.16:0/865744185 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f7ff80654a0 msgr2=0x7f7ff8067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:26.742 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.740+0000 7f801bb10640 1 --2- 172.21.15.16:0/865744185 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f7ff80654a0 0x7f7ff8067950 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f80100ad000 tx=0x7f8010008040 comp rx=0 tx=0).stop 2023-05-22T16:05:26.742 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.740+0000 7f801bb10640 1 -- 172.21.15.16:0/865744185 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8014159990 msgr2=0x7f8014150cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:26.742 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.740+0000 7f801bb10640 1 --2- 172.21.15.16:0/865744185 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8014159990 0x7f8014150cc0 secure :-1 s=READY pgs=2454 cs=0 l=1 rev1=1 crypto rx=0x7f800400c700 tx=0x7f800400cbc0 comp rx=0 tx=0).stop 2023-05-22T16:05:26.742 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.740+0000 7f801bb10640 1 -- 172.21.15.16:0/865744185 shutdown_connections 2023-05-22T16:05:26.742 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.740+0000 7f801bb10640 1 --2- 172.21.15.16:0/865744185 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f801415e0f0 0x7f80141518b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:26.742 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.740+0000 7f801bb10640 1 --2- 172.21.15.16:0/865744185 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f7ff80654a0 0x7f7ff8067950 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:26.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.740+0000 7f801bb10640 1 --2- 172.21.15.16:0/865744185 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8014159990 0x7f8014150cc0 unknown :-1 s=CLOSED pgs=2454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:26.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.740+0000 7f801bb10640 1 --2- 172.21.15.16:0/865744185 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f80140ceac0 0x7f8014150780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:26.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.740+0000 7f801bb10640 1 -- 172.21.15.16:0/865744185 >> 172.21.15.16:0/865744185 conn(0x7f801406dac0 msgr2=0x7f80140d0980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:05:26.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.740+0000 7f801bb10640 1 -- 172.21.15.16:0/865744185 shutdown_connections 2023-05-22T16:05:26.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.740+0000 7f801bb10640 1 -- 172.21.15.16:0/865744185 wait complete. 2023-05-22T16:05:26.757 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-05-22T16:05:26.880 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.876+0000 7f91bd0ae640 1 Processor -- start 2023-05-22T16:05:26.881 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.876+0000 7f91bd0ae640 1 -- start start 2023-05-22T16:05:26.882 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.880+0000 7f91bd0ae640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f91b80ce480 0x7f91b8071640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:26.882 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.880+0000 7f91bd0ae640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f91b80ce850 0x7f91b8160090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:26.882 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.880+0000 7f91bd0ae640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f91b81605d0 0x7f91b815c270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:26.882 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.880+0000 7f91bd0ae640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91b815f990 con 0x7f91b80ce480 2023-05-22T16:05:26.883 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.880+0000 7f91bd0ae640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91b8069ae0 con 0x7f91b80ce850 2023-05-22T16:05:26.883 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.880+0000 7f91bd0ae640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91b8069dc0 con 0x7f91b81605d0 2023-05-22T16:05:26.883 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.880+0000 7f91b6d9d640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f91b80ce480 0x7f91b8071640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:26.883 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.880+0000 7f91b6d9d640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f91b80ce480 0x7f91b8071640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41756/0 (socket says 172.21.15.16:41756) 2023-05-22T16:05:26.883 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.880+0000 7f91b6d9d640 1 -- 172.21.15.16:0/1882758202 learned_addr learned my addr 172.21.15.16:0/1882758202 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:05:26.883 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.880+0000 7f91ae59c640 1 --2- 172.21.15.16:0/1882758202 >> v2:172.21.15.62:3300/0 conn(0x7f91b80ce850 0x7f91b8160090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:26.883 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.880+0000 7f91b759e640 1 --2- 172.21.15.16:0/1882758202 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f91b81605d0 0x7f91b815c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:26.883 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.880+0000 7f91b6d9d640 1 -- 172.21.15.16:0/1882758202 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f91b81605d0 msgr2=0x7f91b815c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:05:26.884 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.880+0000 7f91b6d9d640 1 --2- 172.21.15.16:0/1882758202 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f91b81605d0 0x7f91b815c270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:26.884 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.880+0000 7f91b6d9d640 1 -- 172.21.15.16:0/1882758202 >> v2:172.21.15.62:3300/0 conn(0x7f91b80ce850 msgr2=0x7f91b8160090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:05:26.884 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.880+0000 7f91b6d9d640 1 --2- 172.21.15.16:0/1882758202 >> v2:172.21.15.62:3300/0 conn(0x7f91b80ce850 0x7f91b8160090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:26.884 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.880+0000 7f91b6d9d640 1 -- 172.21.15.16:0/1882758202 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91b806a0a0 con 0x7f91b80ce480 2023-05-22T16:05:26.885 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.880+0000 7f91b6d9d640 1 --2- 172.21.15.16:0/1882758202 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f91b80ce480 0x7f91b8071640 secure :-1 s=READY pgs=2455 cs=0 l=1 rev1=1 crypto rx=0x7f91b80b1d50 tx=0x7f91a801c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e07ac49c361c617d server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:26.885 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.880+0000 7f91b659c640 1 -- 172.21.15.16:0/1882758202 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91a80ac600 con 0x7f91b80ce480 2023-05-22T16:05:26.885 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.884+0000 7f91b659c640 1 -- 172.21.15.16:0/1882758202 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91a80a8050 con 0x7f91b80ce480 2023-05-22T16:05:26.885 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.884+0000 7f91b659c640 1 -- 172.21.15.16:0/1882758202 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91a80ac980 con 0x7f91b80ce480 2023-05-22T16:05:26.886 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.884+0000 7f91bd0ae640 1 -- 172.21.15.16:0/1882758202 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f91b80ce480 msgr2=0x7f91b8071640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:26.886 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.884+0000 7f91bd0ae640 1 --2- 172.21.15.16:0/1882758202 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f91b80ce480 0x7f91b8071640 secure :-1 s=READY pgs=2455 cs=0 l=1 rev1=1 crypto rx=0x7f91b80b1d50 tx=0x7f91a801c4d0 comp rx=0 tx=0).stop 2023-05-22T16:05:26.886 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.884+0000 7f91bd0ae640 1 -- 172.21.15.16:0/1882758202 shutdown_connections 2023-05-22T16:05:26.886 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.884+0000 7f91bd0ae640 1 --2- 172.21.15.16:0/1882758202 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f91b81605d0 0x7f91b815c270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:26.886 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.884+0000 7f91bd0ae640 1 --2- 172.21.15.16:0/1882758202 >> v2:172.21.15.62:3300/0 conn(0x7f91b80ce850 0x7f91b8160090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:26.886 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.884+0000 7f91bd0ae640 1 --2- 172.21.15.16:0/1882758202 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f91b80ce480 0x7f91b8071640 unknown :-1 s=CLOSED pgs=2455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:26.886 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.884+0000 7f91bd0ae640 1 -- 172.21.15.16:0/1882758202 >> 172.21.15.16:0/1882758202 conn(0x7f91b80cc870 msgr2=0x7f91b80cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:05:26.886 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.884+0000 7f91bd0ae640 1 -- 172.21.15.16:0/1882758202 shutdown_connections 2023-05-22T16:05:26.887 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.884+0000 7f91bd0ae640 1 -- 172.21.15.16:0/1882758202 wait complete. 2023-05-22T16:05:26.888 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.884+0000 7f91bd0ae640 1 Processor -- start 2023-05-22T16:05:26.888 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.884+0000 7f91bd0ae640 1 -- start start 2023-05-22T16:05:26.888 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.884+0000 7f91bd0ae640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f91b80ce480 0x7f91b814f8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:26.889 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.884+0000 7f91bd0ae640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f91b80ce850 0x7f91b8156740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:26.889 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.884+0000 7f91bd0ae640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f91b81605d0 0x7f91b806db00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:26.889 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.884+0000 7f91bd0ae640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91b80af900 con 0x7f91b80ce480 2023-05-22T16:05:26.889 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.884+0000 7f91bd0ae640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91b8157040 con 0x7f91b80ce850 2023-05-22T16:05:26.889 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.884+0000 7f91bd0ae640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91b8157320 con 0x7f91b81605d0 2023-05-22T16:05:26.889 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.884+0000 7f91b6d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f91b80ce480 0x7f91b814f8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:26.889 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.884+0000 7f91b6d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f91b80ce480 0x7f91b814f8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41770/0 (socket says 172.21.15.16:41770) 2023-05-22T16:05:26.890 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.884+0000 7f91b6d9d640 1 -- 172.21.15.16:0/1177495777 learned_addr learned my addr 172.21.15.16:0/1177495777 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:05:26.890 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.884+0000 7f91ae59c640 1 --2- 172.21.15.16:0/1177495777 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f91b80ce850 0x7f91b8156740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:26.890 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.884+0000 7f91b759e640 1 --2- 172.21.15.16:0/1177495777 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f91b81605d0 0x7f91b806db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:26.890 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.884+0000 7f91b6d9d640 1 -- 172.21.15.16:0/1177495777 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f91b81605d0 msgr2=0x7f91b806db00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:26.890 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.884+0000 7f91b6d9d640 1 --2- 172.21.15.16:0/1177495777 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f91b81605d0 0x7f91b806db00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:26.890 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.884+0000 7f91b6d9d640 1 -- 172.21.15.16:0/1177495777 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f91b80ce850 msgr2=0x7f91b8156740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:26.890 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.884+0000 7f91b6d9d640 1 --2- 172.21.15.16:0/1177495777 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f91b80ce850 0x7f91b8156740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:26.890 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.884+0000 7f91b6d9d640 1 -- 172.21.15.16:0/1177495777 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91a80094d0 con 0x7f91b80ce480 2023-05-22T16:05:26.891 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.884+0000 7f91b6d9d640 1 --2- 172.21.15.16:0/1177495777 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f91b80ce480 0x7f91b814f8a0 secure :-1 s=READY pgs=2456 cs=0 l=1 rev1=1 crypto rx=0x7f91a801ab80 tx=0x7f91a801ad10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:26.891 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.884+0000 7f91affff640 1 -- 172.21.15.16:0/1177495777 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91a80b7650 con 0x7f91b80ce480 2023-05-22T16:05:26.891 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.884+0000 7f91affff640 1 -- 172.21.15.16:0/1177495777 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91a80ace90 con 0x7f91b80ce480 2023-05-22T16:05:26.891 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.884+0000 7f91bd0ae640 1 -- 172.21.15.16:0/1177495777 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91b80c6000 con 0x7f91b80ce480 2023-05-22T16:05:26.891 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.884+0000 7f91affff640 1 -- 172.21.15.16:0/1177495777 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91a80b79b0 con 0x7f91b80ce480 2023-05-22T16:05:26.891 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.884+0000 7f91bd0ae640 1 -- 172.21.15.16:0/1177495777 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91b80c5e70 con 0x7f91b80ce480 2023-05-22T16:05:26.892 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.888+0000 7f91bd0ae640 1 -- 172.21.15.16:0/1177495777 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91b80cec50 con 0x7f91b80ce480 2023-05-22T16:05:26.893 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.888+0000 7f91affff640 1 -- 172.21.15.16:0/1177495777 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 92) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f91a80b3070 con 0x7f91b80ce480 2023-05-22T16:05:26.893 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.888+0000 7f91affff640 1 --2- 172.21.15.16:0/1177495777 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f91980654a0 0x7f9198067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:26.893 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.888+0000 7f91affff640 1 -- 172.21.15.16:0/1177495777 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(196..196 src has 1..196) v4 ==== 6438+0+0 (secure 0 0 0) 0x7f91a811d330 con 0x7f91b80ce480 2023-05-22T16:05:26.893 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.888+0000 7f91ae59c640 1 --2- 172.21.15.16:0/1177495777 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f91980654a0 0x7f9198067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:26.893 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.888+0000 7f91ae59c640 1 --2- 172.21.15.16:0/1177495777 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f91980654a0 0x7f9198067950 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f91b8159a20 tx=0x7f91a0008040 comp rx=0 tx=0).ready entity=mgr.10542 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:26.894 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:26.892+0000 7f91affff640 1 -- 172.21.15.16:0/1177495777 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f91a80eaca0 con 0x7f91b80ce480 2023-05-22T16:05:27.083 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.080+0000 7f91bd0ae640 1 -- 172.21.15.16:0/1177495777 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f91b804dbe0 con 0x7f91b80ce480 2023-05-22T16:05:27.084 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.084+0000 7f91affff640 1 -- 172.21.15.16:0/1177495777 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 81 v81) v1 ==== 94+0+4653 (secure 0 0 0) 0x7f91a80ee2c0 con 0x7f91b80ce480 2023-05-22T16:05:27.085 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T16:05:27.085 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":81,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7979,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/1053276484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":1053276484},{"type":"v1","addr":"172.21.15.62:6835","nonce":1053276484}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":76},{"gid":8983,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6833/1372552690","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":1372552690},{"type":"v1","addr":"172.21.15.62:6833","nonce":1372552690}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":76},{"gid":10776,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/982492476","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":982492476},{"type":"v1","addr":"172.21.15.16:6835","nonce":982492476}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":76}],"filesystems":[{"mdsmap":{"epoch":81,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-05-22T16:05:21.922031+0000","modified":"2023-05-22T16:05:24.940696+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10782},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10782":{"gid":10782,"name":"c","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.16:6837/1705367575","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":1705367575},{"type":"v1","addr":"172.21.15.16:6837","nonce":1705367575}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[25],"metadata_pool":24,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":11}]} 2023-05-22T16:05:27.085 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 81 2023-05-22T16:05:27.087 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.084+0000 7f91bd0ae640 1 -- 172.21.15.16:0/1177495777 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f91980654a0 msgr2=0x7f9198067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:27.087 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.084+0000 7f91bd0ae640 1 --2- 172.21.15.16:0/1177495777 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f91980654a0 0x7f9198067950 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f91b8159a20 tx=0x7f91a0008040 comp rx=0 tx=0).stop 2023-05-22T16:05:27.087 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.084+0000 7f91bd0ae640 1 -- 172.21.15.16:0/1177495777 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f91b80ce480 msgr2=0x7f91b814f8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:27.087 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.084+0000 7f91bd0ae640 1 --2- 172.21.15.16:0/1177495777 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f91b80ce480 0x7f91b814f8a0 secure :-1 s=READY pgs=2456 cs=0 l=1 rev1=1 crypto rx=0x7f91a801ab80 tx=0x7f91a801ad10 comp rx=0 tx=0).stop 2023-05-22T16:05:27.088 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.084+0000 7f91bd0ae640 1 -- 172.21.15.16:0/1177495777 shutdown_connections 2023-05-22T16:05:27.088 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.084+0000 7f91bd0ae640 1 --2- 172.21.15.16:0/1177495777 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f91b81605d0 0x7f91b806db00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:27.088 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.084+0000 7f91bd0ae640 1 --2- 172.21.15.16:0/1177495777 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f91b80ce850 0x7f91b8156740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:27.088 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.084+0000 7f91bd0ae640 1 --2- 172.21.15.16:0/1177495777 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f91980654a0 0x7f9198067950 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:27.088 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.084+0000 7f91bd0ae640 1 --2- 172.21.15.16:0/1177495777 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f91b80ce480 0x7f91b814f8a0 unknown :-1 s=CLOSED pgs=2456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:27.089 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.084+0000 7f91bd0ae640 1 -- 172.21.15.16:0/1177495777 >> 172.21.15.16:0/1177495777 conn(0x7f91b80cc870 msgr2=0x7f91b815f110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:05:27.089 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.084+0000 7f91bd0ae640 1 -- 172.21.15.16:0/1177495777 shutdown_connections 2023-05-22T16:05:27.089 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.084+0000 7f91bd0ae640 1 -- 172.21.15.16:0/1177495777 wait complete. 2023-05-22T16:05:27.506 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 81, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-05-22T16:05:21.922031+0000', 'modified': '2023-05-22T16:05:24.940696+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 10782}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10782': {'gid': 10782, 'name': 'c', 'rank': 0, 'incarnation': 77, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.16:6837/1705367575', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6836', 'nonce': 1705367575}, {'type': 'v1', 'addr': '172.21.15.16:6837', 'nonce': 1705367575}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [25], 'metadata_pool': 24, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-05-22T16:05:27.507 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-05-22T16:05:27.507 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2023-05-22T16:05:27.646 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.644+0000 7f6309717640 1 Processor -- start 2023-05-22T16:05:27.646 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.644+0000 7f6309717640 1 -- start start 2023-05-22T16:05:27.651 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.644+0000 7f6309717640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f63040ca4e0 0x7f630415e460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:27.651 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.644+0000 7f6309717640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f63040c9b40 0x7f63040afb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:27.651 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.644+0000 7f6309717640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63040b0650 con 0x7f6304070df0 2023-05-22T16:05:27.652 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.644+0000 7f6309717640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63040b0370 con 0x7f63040ca4e0 2023-05-22T16:05:27.652 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.644+0000 7f6309717640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f630415d980 con 0x7f63040c9b40 2023-05-22T16:05:27.652 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.644+0000 7f63037fe640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f6304070df0 0x7f6304071210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:48512/0 (socket says 172.21.15.16:48512) 2023-05-22T16:05:27.652 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.644+0000 7f63037fe640 1 -- 172.21.15.16:0/682563745 learned_addr learned my addr 172.21.15.16:0/682563745 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T16:05:27.653 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.644+0000 7f63027fc640 1 --2- 172.21.15.16:0/682563745 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f63040c9b40 0x7f63040afb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:27.653 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.644+0000 7f6302ffd640 1 --2- 172.21.15.16:0/682563745 >> v2:172.21.15.62:3300/0 conn(0x7f63040ca4e0 0x7f630415e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:27.653 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.644+0000 7f6301ffb640 1 -- 172.21.15.16:0/682563745 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1865291630 0 0) 0x7f63040b0650 con 0x7f6304070df0 2023-05-22T16:05:27.653 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.644+0000 7f6301ffb640 1 -- 172.21.15.16:0/682563745 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f62ec003530 con 0x7f6304070df0 2023-05-22T16:05:27.654 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.644+0000 7f6301ffb640 1 -- 172.21.15.16:0/682563745 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1567027566 0 0) 0x7f62ec003530 con 0x7f6304070df0 2023-05-22T16:05:27.654 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.644+0000 7f6301ffb640 1 -- 172.21.15.16:0/682563745 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f63040c9b40 msgr2=0x7f63040afb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:05:27.654 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.644+0000 7f6301ffb640 1 --2- 172.21.15.16:0/682563745 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f63040c9b40 0x7f63040afb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:27.654 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.644+0000 7f6301ffb640 1 -- 172.21.15.16:0/682563745 >> v2:172.21.15.62:3300/0 conn(0x7f63040ca4e0 msgr2=0x7f630415e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:05:27.654 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.644+0000 7f6301ffb640 1 --2- 172.21.15.16:0/682563745 >> v2:172.21.15.62:3300/0 conn(0x7f63040ca4e0 0x7f630415e460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:27.655 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.644+0000 7f6301ffb640 1 -- 172.21.15.16:0/682563745 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f630415dc60 con 0x7f6304070df0 2023-05-22T16:05:27.655 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.644+0000 7f6301ffb640 1 -- 172.21.15.16:0/682563745 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7f62fc001680 con 0x7f6304070df0 2023-05-22T16:05:27.655 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.644+0000 7f6301ffb640 1 -- 172.21.15.16:0/682563745 >> v1:172.21.15.16:6789/0 conn(0x7f6304070df0 legacy=0x7f6304071210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:27.655 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.644+0000 7f6301ffb640 1 --2- 172.21.15.16:0/682563745 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f62ec003980 0x7f62ec003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:27.655 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.644+0000 7f6301ffb640 1 --2- 172.21.15.16:0/682563745 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f62ec0043b0 0x7f62ec004870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:27.655 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.644+0000 7f6301ffb640 1 --2- 172.21.15.16:0/682563745 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f63040c9b40 0x7f62ec004fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:27.655 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.644+0000 7f6301ffb640 1 -- 172.21.15.16:0/682563745 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63040b0370 con 0x7f62ec003980 2023-05-22T16:05:27.656 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.644+0000 7f6301ffb640 1 -- 172.21.15.16:0/682563745 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f630415d980 con 0x7f62ec0043b0 2023-05-22T16:05:27.656 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.644+0000 7f6301ffb640 1 -- 172.21.15.16:0/682563745 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62ec00b990 con 0x7f63040c9b40 2023-05-22T16:05:27.656 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.644+0000 7f6302ffd640 1 --2- 172.21.15.16:0/682563745 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f62ec003980 0x7f62ec003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:27.656 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.644+0000 7f63037fe640 1 --2- 172.21.15.16:0/682563745 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f63040c9b40 0x7f62ec004fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:27.656 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.644+0000 7f63027fc640 1 --2- 172.21.15.16:0/682563745 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f62ec0043b0 0x7f62ec004870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:27.656 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.644+0000 7f6302ffd640 1 -- 172.21.15.16:0/682563745 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f63040c9b40 msgr2=0x7f62ec004fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:05:27.657 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.644+0000 7f6302ffd640 1 --2- 172.21.15.16:0/682563745 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f63040c9b40 0x7f62ec004fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:27.657 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.644+0000 7f6302ffd640 1 -- 172.21.15.16:0/682563745 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f62ec0043b0 msgr2=0x7f62ec004870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:05:27.657 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.644+0000 7f6302ffd640 1 --2- 172.21.15.16:0/682563745 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f62ec0043b0 0x7f62ec004870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:27.657 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.644+0000 7f6302ffd640 1 -- 172.21.15.16:0/682563745 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62ec003530 con 0x7f62ec003980 2023-05-22T16:05:27.657 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.648+0000 7f6302ffd640 1 --2- 172.21.15.16:0/682563745 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f62ec003980 0x7f62ec003d90 secure :-1 s=READY pgs=2458 cs=0 l=1 rev1=1 crypto rx=0x7f62f4009b10 tx=0x7f62f401ca60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87724a63a31cc1bd server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:27.657 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.648+0000 7f6301ffb640 1 -- 172.21.15.16:0/682563745 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62f40a5550 con 0x7f62ec003980 2023-05-22T16:05:27.657 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.648+0000 7f6301ffb640 1 -- 172.21.15.16:0/682563745 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62f40a56b0 con 0x7f62ec003980 2023-05-22T16:05:27.658 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.648+0000 7f6301ffb640 1 -- 172.21.15.16:0/682563745 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62f40a5a60 con 0x7f62ec003980 2023-05-22T16:05:27.658 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.648+0000 7f6309717640 1 -- 172.21.15.16:0/682563745 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f62ec003980 msgr2=0x7f62ec003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:27.658 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.648+0000 7f6309717640 1 --2- 172.21.15.16:0/682563745 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f62ec003980 0x7f62ec003d90 secure :-1 s=READY pgs=2458 cs=0 l=1 rev1=1 crypto rx=0x7f62f4009b10 tx=0x7f62f401ca60 comp rx=0 tx=0).stop 2023-05-22T16:05:27.658 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.648+0000 7f63037fe640 1 -- 172.21.15.16:0/682563745 reap_dead start 2023-05-22T16:05:27.658 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.648+0000 7f6309717640 1 -- 172.21.15.16:0/682563745 shutdown_connections 2023-05-22T16:05:27.659 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.648+0000 7f6309717640 1 -- 172.21.15.16:0/682563745 >> 172.21.15.16:0/682563745 conn(0x7f6304069720 msgr2=0x7f6304069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:05:27.659 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.648+0000 7f6309717640 1 -- 172.21.15.16:0/682563745 shutdown_connections 2023-05-22T16:05:27.659 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.648+0000 7f6309717640 1 -- 172.21.15.16:0/682563745 wait complete. 2023-05-22T16:05:27.666 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.648+0000 7f6309717640 1 Processor -- start 2023-05-22T16:05:27.666 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.648+0000 7f6309717640 1 -- start start 2023-05-22T16:05:27.666 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.652+0000 7f6309717640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f62ec003980 0x7f630414d050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:27.666 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.652+0000 7f6309717640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f630414fe20 0x7f6304150290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:27.666 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.652+0000 7f6309717640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f63040b2300 0x7f63040b2770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:27.667 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.652+0000 7f6309717640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63040b2cb0 con 0x7f62ec003980 2023-05-22T16:05:27.667 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.652+0000 7f6309717640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6304150920 con 0x7f630414fe20 2023-05-22T16:05:27.667 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.652+0000 7f6309717640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63040c1720 con 0x7f63040b2300 2023-05-22T16:05:27.667 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.652+0000 7f6302ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f62ec003980 0x7f630414d050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:27.667 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.652+0000 7f6302ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f62ec003980 0x7f630414d050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41792/0 (socket says 172.21.15.16:41792) 2023-05-22T16:05:27.667 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.652+0000 7f6302ffd640 1 -- 172.21.15.16:0/1988014886 learned_addr learned my addr 172.21.15.16:0/1988014886 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:05:27.667 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.652+0000 7f63037fe640 1 --2- 172.21.15.16:0/1988014886 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f63040b2300 0x7f63040b2770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:27.667 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.652+0000 7f63027fc640 1 --2- 172.21.15.16:0/1988014886 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f630414fe20 0x7f6304150290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:27.668 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.652+0000 7f6302ffd640 1 -- 172.21.15.16:0/1988014886 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f63040b2300 msgr2=0x7f63040b2770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:27.668 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.652+0000 7f6302ffd640 1 --2- 172.21.15.16:0/1988014886 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f63040b2300 0x7f63040b2770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:27.668 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.652+0000 7f6302ffd640 1 -- 172.21.15.16:0/1988014886 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f630414fe20 msgr2=0x7f6304150290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:27.668 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.652+0000 7f6302ffd640 1 --2- 172.21.15.16:0/1988014886 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f630414fe20 0x7f6304150290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:27.668 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.652+0000 7f6302ffd640 1 -- 172.21.15.16:0/1988014886 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63040b00e0 con 0x7f62ec003980 2023-05-22T16:05:27.668 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.652+0000 7f6302ffd640 1 --2- 172.21.15.16:0/1988014886 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f62ec003980 0x7f630414d050 secure :-1 s=READY pgs=2459 cs=0 l=1 rev1=1 crypto rx=0x7f62f401c890 tx=0x7f62ec005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:27.669 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.652+0000 7f62eb7fe640 1 -- 172.21.15.16:0/1988014886 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62f40a5e40 con 0x7f62ec003980 2023-05-22T16:05:27.669 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.652+0000 7f62eb7fe640 1 -- 172.21.15.16:0/1988014886 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62f4004020 con 0x7f62ec003980 2023-05-22T16:05:27.669 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.652+0000 7f6309717640 1 -- 172.21.15.16:0/1988014886 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63040aff50 con 0x7f62ec003980 2023-05-22T16:05:27.669 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.652+0000 7f62eb7fe640 1 -- 172.21.15.16:0/1988014886 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62f40043d0 con 0x7f62ec003980 2023-05-22T16:05:27.669 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.652+0000 7f6309717640 1 -- 172.21.15.16:0/1988014886 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63040c8e80 con 0x7f62ec003980 2023-05-22T16:05:27.669 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.652+0000 7f62eb7fe640 1 -- 172.21.15.16:0/1988014886 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 92) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f62f40b4040 con 0x7f62ec003980 2023-05-22T16:05:27.669 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.656+0000 7f62eb7fe640 1 --2- 172.21.15.16:0/1988014886 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f62f00654d0 0x7f62f0067980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:27.670 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.656+0000 7f62eb7fe640 1 -- 172.21.15.16:0/1988014886 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(196..196 src has 1..196) v4 ==== 6438+0+0 (secure 0 0 0) 0x7f62f411cae0 con 0x7f62ec003980 2023-05-22T16:05:27.670 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.656+0000 7f63027fc640 1 --2- 172.21.15.16:0/1988014886 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f62f00654d0 0x7f62f0067980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:27.670 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.656+0000 7f630c839000 1 Processor -- start 2023-05-22T16:05:27.670 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.656+0000 7f63027fc640 1 --2- 172.21.15.16:0/1988014886 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f62f00654d0 0x7f62f0067980 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f6304160600 tx=0x7f62f800d040 comp rx=0 tx=0).ready entity=mgr.10542 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:27.670 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.656+0000 7f630c839000 1 -- start start 2023-05-22T16:05:27.670 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.656+0000 7f630c839000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fc0ca8afd0 0x55fc0cd02090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:27.671 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.656+0000 7f630c839000 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x55fc0cd025d0 0x55fc0cd029a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:27.671 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.656+0000 7f630c839000 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fc0ccb5e20 0x55fc0ccb61f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:27.671 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.656+0000 7f630c839000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fc0cb7bbd0 con 0x55fc0ca8afd0 2023-05-22T16:05:27.671 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.656+0000 7f630c839000 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fc0ca98dd0 con 0x55fc0ccb5e20 2023-05-22T16:05:27.671 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.656+0000 7f630c839000 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x55fc0cb09640 con 0x55fc0cd025d0 2023-05-22T16:05:27.671 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.656+0000 7f6302ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fc0ca8afd0 0x55fc0cd02090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:27.671 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.656+0000 7f6302ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fc0ca8afd0 0x55fc0cd02090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41794/0 (socket says 172.21.15.16:41794) 2023-05-22T16:05:27.672 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.656+0000 7f6302ffd640 1 -- 172.21.15.16:0/1316086019 learned_addr learned my addr 172.21.15.16:0/1316086019 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:05:27.672 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.656+0000 7f63027fc640 1 --2- 172.21.15.16:0/1316086019 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x55fc0cd025d0 0x55fc0cd029a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:27.672 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.656+0000 7f6302ffd640 1 -- 172.21.15.16:0/1316086019 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x55fc0cd025d0 msgr2=0x55fc0cd029a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:05:27.672 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.656+0000 7f63037fe640 1 --2- 172.21.15.16:0/1316086019 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fc0ccb5e20 0x55fc0ccb61f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:27.672 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.656+0000 7f6302ffd640 1 --2- 172.21.15.16:0/1316086019 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x55fc0cd025d0 0x55fc0cd029a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:27.672 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.656+0000 7f6302ffd640 1 -- 172.21.15.16:0/1316086019 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fc0ccb5e20 msgr2=0x55fc0ccb61f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:05:27.673 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.656+0000 7f6302ffd640 1 --2- 172.21.15.16:0/1316086019 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fc0ccb5e20 0x55fc0ccb61f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:27.673 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.656+0000 7f6302ffd640 1 -- 172.21.15.16:0/1316086019 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fc0cac80b0 con 0x55fc0ca8afd0 2023-05-22T16:05:27.673 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.656+0000 7f6302ffd640 1 --2- 172.21.15.16:0/1316086019 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fc0ca8afd0 0x55fc0cd02090 secure :-1 s=READY pgs=2460 cs=0 l=1 rev1=1 crypto rx=0x7f62f40ee200 tx=0x7f62f4000bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c45980e88ed9969a server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:27.673 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.656+0000 7f62e8ff9640 1 -- 172.21.15.16:0/1316086019 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62f40096b0 con 0x55fc0ca8afd0 2023-05-22T16:05:27.673 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.656+0000 7f62e8ff9640 1 -- 172.21.15.16:0/1316086019 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62f411ad20 con 0x55fc0ca8afd0 2023-05-22T16:05:27.673 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.656+0000 7f62e8ff9640 1 -- 172.21.15.16:0/1316086019 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62f411ae90 con 0x55fc0ca8afd0 2023-05-22T16:05:27.673 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.656+0000 7f630c839000 1 -- 172.21.15.16:0/1316086019 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fc0ca8afd0 msgr2=0x55fc0cd02090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:27.674 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.656+0000 7f630c839000 1 --2- 172.21.15.16:0/1316086019 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fc0ca8afd0 0x55fc0cd02090 secure :-1 s=READY pgs=2460 cs=0 l=1 rev1=1 crypto rx=0x7f62f40ee200 tx=0x7f62f4000bd0 comp rx=0 tx=0).stop 2023-05-22T16:05:27.674 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.656+0000 7f630c839000 1 -- 172.21.15.16:0/1316086019 shutdown_connections 2023-05-22T16:05:27.674 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.656+0000 7f630c839000 1 --2- 172.21.15.16:0/1316086019 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fc0ccb5e20 0x55fc0ccb61f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:27.674 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.656+0000 7f630c839000 1 --2- 172.21.15.16:0/1316086019 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x55fc0cd025d0 0x55fc0cd029a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:27.674 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.656+0000 7f630c839000 1 --2- 172.21.15.16:0/1316086019 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fc0ca8afd0 0x55fc0cd02090 unknown :-1 s=CLOSED pgs=2460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:27.674 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.656+0000 7f630c839000 1 -- 172.21.15.16:0/1316086019 >> 172.21.15.16:0/1316086019 conn(0x55fc0cb5a270 msgr2=0x55fc0cce5680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:05:27.674 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.656+0000 7f630c839000 1 -- 172.21.15.16:0/1316086019 shutdown_connections 2023-05-22T16:05:27.675 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.656+0000 7f630c839000 1 -- 172.21.15.16:0/1316086019 wait complete. 2023-05-22T16:05:27.675 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.656+0000 7f630c839000 1 -- start start 2023-05-22T16:05:27.675 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.656+0000 7f630c839000 1 Processor -- start 2023-05-22T16:05:27.675 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.660+0000 7f630c839000 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x55fc0ca8afd0 0x55fc0ccd4550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:27.675 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.660+0000 7f630c839000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fc0ccb5e20 0x55fc0ccd4a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:27.675 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.660+0000 7f630c839000 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fc0cd025d0 0x55fc0ccd4fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:27.676 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.660+0000 7f630c839000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fc0cb02780 con 0x55fc0ccb5e20 2023-05-22T16:05:27.676 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.660+0000 7f630c839000 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fc0cac0250 con 0x55fc0cd025d0 2023-05-22T16:05:27.676 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.660+0000 7f630c839000 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x55fc0cabfef0 con 0x55fc0ca8afd0 2023-05-22T16:05:27.676 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.660+0000 7f630c839000 10 client.0 _ll_register_callbacks cb 0x55fc0cc9fb50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-05-22T16:05:27.676 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.660+0000 7f63027fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fc0ccb5e20 0x55fc0ccd4a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:27.676 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.660+0000 7f63027fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fc0ccb5e20 0x55fc0ccd4a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41808/0 (socket says 172.21.15.16:41808) 2023-05-22T16:05:27.676 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.660+0000 7f63027fc640 1 -- 172.21.15.16:0/2386316222 learned_addr learned my addr 172.21.15.16:0/2386316222 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:05:27.677 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.660+0000 7f63037fe640 1 --2- 172.21.15.16:0/2386316222 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fc0cd025d0 0x55fc0ccd4fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:27.677 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.660+0000 7f6302ffd640 1 --2- 172.21.15.16:0/2386316222 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x55fc0ca8afd0 0x55fc0ccd4550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:27.677 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.660+0000 7f63027fc640 1 -- 172.21.15.16:0/2386316222 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x55fc0ca8afd0 msgr2=0x55fc0ccd4550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:05:27.677 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.660+0000 7f63027fc640 1 --2- 172.21.15.16:0/2386316222 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x55fc0ca8afd0 0x55fc0ccd4550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:27.677 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.660+0000 7f63027fc640 1 -- 172.21.15.16:0/2386316222 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fc0cd025d0 msgr2=0x55fc0ccd4fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:05:27.677 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.660+0000 7f63027fc640 1 --2- 172.21.15.16:0/2386316222 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fc0cd025d0 0x55fc0ccd4fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:27.678 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.660+0000 7f63027fc640 1 -- 172.21.15.16:0/2386316222 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63040c8e80 con 0x55fc0ccb5e20 2023-05-22T16:05:27.678 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.660+0000 7f63037fe640 1 --2- 172.21.15.16:0/2386316222 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fc0cd025d0 0x55fc0ccd4fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T16:05:27.678 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.660+0000 7f63027fc640 1 --2- 172.21.15.16:0/2386316222 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fc0ccb5e20 0x55fc0ccd4a90 secure :-1 s=READY pgs=2461 cs=0 l=1 rev1=1 crypto rx=0x7f62f8018b90 tx=0x7f62f8004ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1129bd47d1ece212 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:27.678 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.660+0000 7f62e1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2023-05-22T16:05:27.678 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.660+0000 7f62e1ffb640 1 -- 172.21.15.16:0/2386316222 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62f8023410 con 0x55fc0ccb5e20 2023-05-22T16:05:27.678 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.660+0000 7f62e1ffb640 1 -- 172.21.15.16:0/2386316222 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(196..196 src has 1..196) v4 ==== 6438+0+0 (secure 0 0 0) 0x7f62f8023570 con 0x55fc0ccb5e20 2023-05-22T16:05:27.678 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.660+0000 7f630c839000 1 -- 172.21.15.16:0/2386316222 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55fc0cac6c60 con 0x55fc0ccb5e20 2023-05-22T16:05:27.679 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.660+0000 7f62e1ffb640 1 -- 172.21.15.16:0/2386316222 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=81) v2 ==== 24+0+0 (secure 0 0 0) 0x7f62f8023730 con 0x55fc0ccb5e20 2023-05-22T16:05:27.679 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.660+0000 7f630c839000 10 client.10932 fetch_fsmap learned FSMap version 81 2023-05-22T16:05:27.679 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.660+0000 7f630c839000 1 -- 172.21.15.16:0/2386316222 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=81,monmap=2+}) v3 -- 0x55fc0c7ff890 con 0x55fc0ccb5e20 2023-05-22T16:05:27.679 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.660+0000 7f62e1ffb640 1 -- 172.21.15.16:0/2386316222 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55fc0cac6c60 con 0x55fc0ccb5e20 2023-05-22T16:05:27.679 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.660+0000 7f62e1ffb640 1 -- 172.21.15.16:0/2386316222 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 81) v1 ==== 3106+0+0 (secure 0 0 0) 0x7f62f8023890 con 0x55fc0ccb5e20 2023-05-22T16:05:27.679 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.660+0000 7f630c839000 10 client.10932 fetch_fsmap finished waiting for FSMap version 81 2023-05-22T16:05:27.679 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.660+0000 7f630c839000 10 client.10932 resolve_mds: resolved name 'c' to daemon mds.c 2023-05-22T16:05:27.680 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.660+0000 7f630c839000 20 client.10932 populate_metadata read hostname 'smithi016' 2023-05-22T16:05:27.680 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.660+0000 7f630c839000 1 --2- 172.21.15.16:0/2386316222 >> [v2:172.21.15.16:6836/1705367575,v1:172.21.15.16:6837/1705367575] conn(0x55fc0ccd5870 0x55fc0cd13210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:27.680 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.660+0000 7f630c839000 4 client.10932 mds_command: new command op to 10782 tid=0[{"prefix": "get_command_descriptions"}] 2023-05-22T16:05:27.680 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.660+0000 7f630c839000 1 -- 172.21.15.16:0/2386316222 --> [v2:172.21.15.16:6836/1705367575,v1:172.21.15.16:6837/1705367575] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55fc0cb010d0 con 0x55fc0ccd5870 2023-05-22T16:05:27.680 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.660+0000 7f6302ffd640 1 --2- 172.21.15.16:0/2386316222 >> [v2:172.21.15.16:6836/1705367575,v1:172.21.15.16:6837/1705367575] conn(0x55fc0ccd5870 0x55fc0cd13210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:27.680 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.660+0000 7f6302ffd640 1 --2- 172.21.15.16:0/2386316222 >> [v2:172.21.15.16:6836/1705367575,v1:172.21.15.16:6837/1705367575] conn(0x55fc0ccd5870 0x55fc0cd13210 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=60a13d1d70e32cbe server_cookie=2f5684abf4e71193 in_seq=0 out_seq=0 2023-05-22T16:05:27.680 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.660+0000 7f62e1ffb640 10 client.10932 ms_handle_connect on v2:172.21.15.16:6836/1705367575 2023-05-22T16:05:27.681 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.660+0000 7f62e1ffb640 1 -- 172.21.15.16:0/2386316222 <== mds.0 v2:172.21.15.16:6836/1705367575 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x55fc0cb010d0 con 0x55fc0ccd5870 2023-05-22T16:05:27.681 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.660+0000 7f62e1ffb640 10 client.10932 handle_command_reply: tid=0 2023-05-22T16:05:27.681 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.660+0000 7f630c839000 1 client.10932 shutdown 2023-05-22T16:05:27.681 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.660+0000 7f630c839000 1 -- 172.21.15.16:0/2386316222 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fc0ccb5e20 msgr2=0x55fc0ccd4a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:27.681 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.660+0000 7f630c839000 1 --2- 172.21.15.16:0/2386316222 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fc0ccb5e20 0x55fc0ccd4a90 secure :-1 s=READY pgs=2461 cs=0 l=1 rev1=1 crypto rx=0x7f62f8018b90 tx=0x7f62f8004ac0 comp rx=0 tx=0).stop 2023-05-22T16:05:27.681 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.660+0000 7f630c839000 1 -- 172.21.15.16:0/2386316222 shutdown_connections 2023-05-22T16:05:27.681 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.660+0000 7f630c839000 1 --2- 172.21.15.16:0/2386316222 >> [v2:172.21.15.16:6836/1705367575,v1:172.21.15.16:6837/1705367575] conn(0x55fc0ccd5870 0x55fc0cd13210 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:27.681 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.660+0000 7f630c839000 1 --2- 172.21.15.16:0/2386316222 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fc0cd025d0 0x55fc0ccd4fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:27.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.660+0000 7f630c839000 1 --2- 172.21.15.16:0/2386316222 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fc0ccb5e20 0x55fc0ccd4a90 unknown :-1 s=CLOSED pgs=2461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:27.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.660+0000 7f62e1ffb640 0 client.10932 ms_handle_reset on v2:172.21.15.16:6836/1705367575 2023-05-22T16:05:27.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.660+0000 7f630c839000 1 --2- 172.21.15.16:0/2386316222 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x55fc0ca8afd0 0x55fc0ccd4550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:27.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.660+0000 7f630c839000 1 -- 172.21.15.16:0/2386316222 >> 172.21.15.16:0/2386316222 conn(0x55fc0cb5a270 msgr2=0x55fc0ccb8200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:05:27.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.660+0000 7f630c839000 1 -- 172.21.15.16:0/2386316222 shutdown_connections 2023-05-22T16:05:27.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.660+0000 7f630c839000 1 -- 172.21.15.16:0/2386316222 wait complete. 2023-05-22T16:05:27.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.660+0000 7f630c839000 20 client.10932 trim_cache size 0 max 16384 2023-05-22T16:05:27.687 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.676+0000 7f630c839000 1 Processor -- start 2023-05-22T16:05:27.687 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.676+0000 7f630c839000 1 -- start start 2023-05-22T16:05:27.687 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.676+0000 7f630c839000 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fc0ca8afd0 0x55fc0cd59d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:27.687 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.676+0000 7f630c839000 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x55fc0ccb5e20 0x55fc0cd5a290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:27.687 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.676+0000 7f630c839000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fc0cd025d0 0x55fc0cd5a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:27.688 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.676+0000 7f630c839000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fc0c7c0940 con 0x55fc0cd025d0 2023-05-22T16:05:27.688 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.676+0000 7f630c839000 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fc0cce7e40 con 0x55fc0ca8afd0 2023-05-22T16:05:27.688 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.676+0000 7f630c839000 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x55fc0cb01610 con 0x55fc0ccb5e20 2023-05-22T16:05:27.689 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.676+0000 7f63037fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fc0cd025d0 0x55fc0cd5a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:27.689 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.676+0000 7f63037fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fc0cd025d0 0x55fc0cd5a7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41816/0 (socket says 172.21.15.16:41816) 2023-05-22T16:05:27.689 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.676+0000 7f63037fe640 1 -- 172.21.15.16:0/965217891 learned_addr learned my addr 172.21.15.16:0/965217891 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:05:27.689 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.676+0000 7f6302ffd640 1 --2- 172.21.15.16:0/965217891 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fc0ca8afd0 0x55fc0cd59d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:27.690 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.676+0000 7f63027fc640 1 --2- 172.21.15.16:0/965217891 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x55fc0ccb5e20 0x55fc0cd5a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:27.690 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.676+0000 7f63037fe640 1 -- 172.21.15.16:0/965217891 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x55fc0ccb5e20 msgr2=0x55fc0cd5a290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:05:27.690 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.676+0000 7f63037fe640 1 --2- 172.21.15.16:0/965217891 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x55fc0ccb5e20 0x55fc0cd5a290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:27.690 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.676+0000 7f63037fe640 1 -- 172.21.15.16:0/965217891 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fc0ca8afd0 msgr2=0x55fc0cd59d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:05:27.691 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.676+0000 7f63037fe640 1 --2- 172.21.15.16:0/965217891 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fc0ca8afd0 0x55fc0cd59d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:27.691 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.676+0000 7f63037fe640 1 -- 172.21.15.16:0/965217891 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fc0cac8360 con 0x55fc0cd025d0 2023-05-22T16:05:27.691 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.676+0000 7f6302ffd640 1 --2- 172.21.15.16:0/965217891 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fc0ca8afd0 0x55fc0cd59d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T16:05:27.691 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.676+0000 7f63037fe640 1 --2- 172.21.15.16:0/965217891 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fc0cd025d0 0x55fc0cd5a7d0 secure :-1 s=READY pgs=2462 cs=0 l=1 rev1=1 crypto rx=0x7f63040cdef0 tx=0x7f62fc009040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65c3e4b328a9655f server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:27.691 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.676+0000 7f62e97fa640 1 -- 172.21.15.16:0/965217891 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62fc01a070 con 0x55fc0cd025d0 2023-05-22T16:05:27.692 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.676+0000 7f62e97fa640 1 -- 172.21.15.16:0/965217891 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62fc00d070 con 0x55fc0cd025d0 2023-05-22T16:05:27.692 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.676+0000 7f62e97fa640 1 -- 172.21.15.16:0/965217891 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62fc005e60 con 0x55fc0cd025d0 2023-05-22T16:05:27.692 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.680+0000 7f630c839000 1 -- 172.21.15.16:0/965217891 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fc0cd025d0 msgr2=0x55fc0cd5a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:27.692 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.680+0000 7f630c839000 1 --2- 172.21.15.16:0/965217891 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fc0cd025d0 0x55fc0cd5a7d0 secure :-1 s=READY pgs=2462 cs=0 l=1 rev1=1 crypto rx=0x7f63040cdef0 tx=0x7f62fc009040 comp rx=0 tx=0).stop 2023-05-22T16:05:27.693 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.680+0000 7f630c839000 1 -- 172.21.15.16:0/965217891 shutdown_connections 2023-05-22T16:05:27.693 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.680+0000 7f630c839000 1 --2- 172.21.15.16:0/965217891 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fc0cd025d0 0x55fc0cd5a7d0 unknown :-1 s=CLOSED pgs=2462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:27.693 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.680+0000 7f630c839000 1 --2- 172.21.15.16:0/965217891 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x55fc0ccb5e20 0x55fc0cd5a290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:27.693 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.680+0000 7f630c839000 1 --2- 172.21.15.16:0/965217891 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fc0ca8afd0 0x55fc0cd59d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:27.693 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.680+0000 7f630c839000 1 -- 172.21.15.16:0/965217891 >> 172.21.15.16:0/965217891 conn(0x55fc0cb5a270 msgr2=0x55fc0ccba440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:05:27.693 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.680+0000 7f630c839000 1 -- 172.21.15.16:0/965217891 shutdown_connections 2023-05-22T16:05:27.694 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.680+0000 7f630c839000 1 -- 172.21.15.16:0/965217891 wait complete. 2023-05-22T16:05:27.694 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.680+0000 7f630c839000 1 -- start start 2023-05-22T16:05:27.694 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.680+0000 7f630c839000 1 Processor -- start 2023-05-22T16:05:27.694 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.680+0000 7f630c839000 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fc0ca8afd0 0x55fc0cd59640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:27.695 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.680+0000 7f630c839000 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fc0ccb5e20 0x55fc0cd59f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:27.695 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.680+0000 7f630c839000 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x55fc0cd025d0 0x55fc0cd5a4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:27.695 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.680+0000 7f630c839000 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fc0c8001b0 con 0x55fc0ccb5e20 2023-05-22T16:05:27.695 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.680+0000 7f630c839000 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fc0caffb80 con 0x55fc0ca8afd0 2023-05-22T16:05:27.695 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.680+0000 7f630c839000 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x55fc0ccabe80 con 0x55fc0cd025d0 2023-05-22T16:05:27.696 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.680+0000 7f630c839000 10 client.0 _ll_register_callbacks cb 0x55fc0cb52eb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-05-22T16:05:27.696 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.680+0000 7f63027fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fc0ccb5e20 0x55fc0cd59f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:27.696 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.680+0000 7f63027fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fc0ccb5e20 0x55fc0cd59f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41828/0 (socket says 172.21.15.16:41828) 2023-05-22T16:05:27.696 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.680+0000 7f63027fc640 1 -- 172.21.15.16:0/479609980 learned_addr learned my addr 172.21.15.16:0/479609980 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:05:27.697 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.680+0000 7f63037fe640 1 --2- 172.21.15.16:0/479609980 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x55fc0cd025d0 0x55fc0cd5a4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:27.697 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.680+0000 7f6302ffd640 1 --2- 172.21.15.16:0/479609980 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fc0ca8afd0 0x55fc0cd59640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:27.697 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.680+0000 7f63027fc640 1 -- 172.21.15.16:0/479609980 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x55fc0cd025d0 msgr2=0x55fc0cd5a4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:05:27.697 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.680+0000 7f63027fc640 1 --2- 172.21.15.16:0/479609980 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x55fc0cd025d0 0x55fc0cd5a4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:27.698 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.680+0000 7f63027fc640 1 -- 172.21.15.16:0/479609980 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fc0ca8afd0 msgr2=0x55fc0cd59640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:05:27.698 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.680+0000 7f63027fc640 1 --2- 172.21.15.16:0/479609980 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fc0ca8afd0 0x55fc0cd59640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:27.698 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.680+0000 7f63027fc640 1 -- 172.21.15.16:0/479609980 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f630415dc60 con 0x55fc0ccb5e20 2023-05-22T16:05:27.698 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.680+0000 7f63037fe640 1 --2- 172.21.15.16:0/479609980 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x55fc0cd025d0 0x55fc0cd5a4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T16:05:27.698 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.680+0000 7f63027fc640 1 --2- 172.21.15.16:0/479609980 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fc0ccb5e20 0x55fc0cd59f60 secure :-1 s=READY pgs=2463 cs=0 l=1 rev1=1 crypto rx=0x7f62f8009a40 tx=0x7f62f800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5540aa53bbdb9654 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:27.699 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.680+0000 7f62e1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2023-05-22T16:05:27.699 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.680+0000 7f62e1ffb640 1 -- 172.21.15.16:0/479609980 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62f8023610 con 0x55fc0ccb5e20 2023-05-22T16:05:27.699 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.680+0000 7f62e1ffb640 1 -- 172.21.15.16:0/479609980 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(196..196 src has 1..196) v4 ==== 6438+0+0 (secure 0 0 0) 0x7f62f8023df0 con 0x55fc0ccb5e20 2023-05-22T16:05:27.699 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.680+0000 7f630c839000 1 -- 172.21.15.16:0/479609980 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55fc0cce8a00 con 0x55fc0ccb5e20 2023-05-22T16:05:27.699 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.680+0000 7f62e1ffb640 1 -- 172.21.15.16:0/479609980 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=81) v2 ==== 24+0+0 (secure 0 0 0) 0x7f62f8016860 con 0x55fc0ccb5e20 2023-05-22T16:05:27.700 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.680+0000 7f630c839000 10 client.10938 fetch_fsmap learned FSMap version 81 2023-05-22T16:05:27.700 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.680+0000 7f630c839000 1 -- 172.21.15.16:0/479609980 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=81,monmap=2+}) v3 -- 0x55fc0caadbe0 con 0x55fc0ccb5e20 2023-05-22T16:05:27.700 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.680+0000 7f62e1ffb640 1 -- 172.21.15.16:0/479609980 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55fc0cce8a00 con 0x55fc0ccb5e20 2023-05-22T16:05:27.700 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.680+0000 7f62e1ffb640 1 -- 172.21.15.16:0/479609980 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 81) v1 ==== 3106+0+0 (secure 0 0 0) 0x7f62f8023770 con 0x55fc0ccb5e20 2023-05-22T16:05:27.701 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.680+0000 7f630c839000 10 client.10938 fetch_fsmap finished waiting for FSMap version 81 2023-05-22T16:05:27.701 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.680+0000 7f630c839000 10 client.10938 resolve_mds: resolved name 'c' to daemon mds.c 2023-05-22T16:05:27.701 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.680+0000 7f630c839000 20 client.10938 populate_metadata read hostname 'smithi016' 2023-05-22T16:05:27.701 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.680+0000 7f630c839000 1 --2- 172.21.15.16:0/479609980 >> [v2:172.21.15.16:6836/1705367575,v1:172.21.15.16:6837/1705367575] conn(0x55fc0cd65520 0x55fc0cd67900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:27.701 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.680+0000 7f630c839000 4 client.10938 mds_command: new command op to 10782 tid=0[{"prefix": "status"}] 2023-05-22T16:05:27.702 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.680+0000 7f630c839000 1 -- 172.21.15.16:0/479609980 --> [v2:172.21.15.16:6836/1705367575,v1:172.21.15.16:6837/1705367575] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55fc0caa94d0 con 0x55fc0cd65520 2023-05-22T16:05:27.702 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.680+0000 7f6302ffd640 1 --2- 172.21.15.16:0/479609980 >> [v2:172.21.15.16:6836/1705367575,v1:172.21.15.16:6837/1705367575] conn(0x55fc0cd65520 0x55fc0cd67900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:27.702 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.680+0000 7f6302ffd640 1 --2- 172.21.15.16:0/479609980 >> [v2:172.21.15.16:6836/1705367575,v1:172.21.15.16:6837/1705367575] conn(0x55fc0cd65520 0x55fc0cd67900 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=884610893eb939ed server_cookie=dcea83be5d3803a9 in_seq=0 out_seq=0 2023-05-22T16:05:27.702 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.680+0000 7f62e1ffb640 10 client.10938 ms_handle_connect on v2:172.21.15.16:6836/1705367575 2023-05-22T16:05:27.702 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.684+0000 7f62e1ffb640 1 -- 172.21.15.16:0/479609980 <== mds.0 v2:172.21.15.16:6836/1705367575 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+334 (crc 0 0 0) 0x55fc0caa94d0 con 0x55fc0cd65520 2023-05-22T16:05:27.702 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.684+0000 7f62e1ffb640 10 client.10938 handle_command_reply: tid=0 2023-05-22T16:05:27.703 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.684+0000 7f630c839000 1 client.10938 shutdown 2023-05-22T16:05:27.703 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.684+0000 7f630c839000 1 -- 172.21.15.16:0/479609980 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fc0ccb5e20 msgr2=0x55fc0cd59f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:27.703 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.684+0000 7f630c839000 1 --2- 172.21.15.16:0/479609980 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fc0ccb5e20 0x55fc0cd59f60 secure :-1 s=READY pgs=2463 cs=0 l=1 rev1=1 crypto rx=0x7f62f8009a40 tx=0x7f62f800b040 comp rx=0 tx=0).stop 2023-05-22T16:05:27.703 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.684+0000 7f630c839000 1 -- 172.21.15.16:0/479609980 shutdown_connections 2023-05-22T16:05:27.703 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.684+0000 7f630c839000 1 --2- 172.21.15.16:0/479609980 >> [v2:172.21.15.16:6836/1705367575,v1:172.21.15.16:6837/1705367575] conn(0x55fc0cd65520 0x55fc0cd67900 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:27.703 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.684+0000 7f630c839000 1 --2- 172.21.15.16:0/479609980 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x55fc0cd025d0 0x55fc0cd5a4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:27.703 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.684+0000 7f630c839000 1 --2- 172.21.15.16:0/479609980 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fc0ccb5e20 0x55fc0cd59f60 unknown :-1 s=CLOSED pgs=2463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:27.703 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.684+0000 7f630c839000 1 --2- 172.21.15.16:0/479609980 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fc0ca8afd0 0x55fc0cd59640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:27.704 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.684+0000 7f62e1ffb640 0 client.10938 ms_handle_reset on v2:172.21.15.16:6836/1705367575 2023-05-22T16:05:27.704 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.684+0000 7f630c839000 1 -- 172.21.15.16:0/479609980 >> 172.21.15.16:0/479609980 conn(0x55fc0cb5a270 msgr2=0x55fc0ccd3730 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:05:27.704 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.684+0000 7f630c839000 1 -- 172.21.15.16:0/479609980 shutdown_connections 2023-05-22T16:05:27.704 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.684+0000 7f630c839000 1 -- 172.21.15.16:0/479609980 wait complete. 2023-05-22T16:05:27.704 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.684+0000 7f630c839000 20 client.10938 trim_cache size 0 max 16384 2023-05-22T16:05:27.705 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-05-22T16:05:27.705 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_fsid": "b2719834-7847-4d98-90a6-f65d57f85c49", 2023-05-22T16:05:27.705 INFO:teuthology.orchestra.run.smithi016.stdout: "whoami": 0, 2023-05-22T16:05:27.705 INFO:teuthology.orchestra.run.smithi016.stdout: "id": 10782, 2023-05-22T16:05:27.705 INFO:teuthology.orchestra.run.smithi016.stdout: "want_state": "up:active", 2023-05-22T16:05:27.705 INFO:teuthology.orchestra.run.smithi016.stdout: "state": "up:active", 2023-05-22T16:05:27.705 INFO:teuthology.orchestra.run.smithi016.stdout: "fs_name": "cephfs", 2023-05-22T16:05:27.706 INFO:teuthology.orchestra.run.smithi016.stdout: "rank_uptime": 5.7599064999999996, 2023-05-22T16:05:27.706 INFO:teuthology.orchestra.run.smithi016.stdout: "mdsmap_epoch": 81, 2023-05-22T16:05:27.706 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch": 196, 2023-05-22T16:05:27.706 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch_barrier": 196, 2023-05-22T16:05:27.706 INFO:teuthology.orchestra.run.smithi016.stdout: "uptime": 10.234372650999999 2023-05-22T16:05:27.706 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-05-22T16:05:27.707 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.684+0000 7f6309717640 1 -- 172.21.15.16:0/1988014886 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f62f00654d0 msgr2=0x7f62f0067980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:27.707 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.684+0000 7f6309717640 1 --2- 172.21.15.16:0/1988014886 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f62f00654d0 0x7f62f0067980 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f6304160600 tx=0x7f62f800d040 comp rx=0 tx=0).stop 2023-05-22T16:05:27.708 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.684+0000 7f6309717640 1 -- 172.21.15.16:0/1988014886 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f62ec003980 msgr2=0x7f630414d050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:27.708 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.684+0000 7f6309717640 1 --2- 172.21.15.16:0/1988014886 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f62ec003980 0x7f630414d050 secure :-1 s=READY pgs=2459 cs=0 l=1 rev1=1 crypto rx=0x7f62f401c890 tx=0x7f62ec005800 comp rx=0 tx=0).stop 2023-05-22T16:05:27.708 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.684+0000 7f6309717640 1 -- 172.21.15.16:0/1988014886 shutdown_connections 2023-05-22T16:05:27.708 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.684+0000 7f6309717640 1 --2- 172.21.15.16:0/1988014886 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f63040b2300 0x7f63040b2770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:27.708 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.684+0000 7f6309717640 1 --2- 172.21.15.16:0/1988014886 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f630414fe20 0x7f6304150290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:27.708 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.684+0000 7f6309717640 1 --2- 172.21.15.16:0/1988014886 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f62f00654d0 0x7f62f0067980 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:27.709 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.684+0000 7f6309717640 1 --2- 172.21.15.16:0/1988014886 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f62ec003980 0x7f630414d050 unknown :-1 s=CLOSED pgs=2459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:27.709 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.684+0000 7f6309717640 1 -- 172.21.15.16:0/1988014886 >> 172.21.15.16:0/1988014886 conn(0x7f6304069720 msgr2=0x7f6304156db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:05:27.709 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.684+0000 7f6309717640 1 -- 172.21.15.16:0/1988014886 shutdown_connections 2023-05-22T16:05:27.709 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.684+0000 7f6309717640 1 -- 172.21.15.16:0/1988014886 wait complete. 2023-05-22T16:05:27.709 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-05-22T16:05:27.820 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.816+0000 7fc13981d640 1 Processor -- start 2023-05-22T16:05:27.820 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.816+0000 7fc13981d640 1 -- start start 2023-05-22T16:05:27.821 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.820+0000 7fc13981d640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc1340c8250 0x7fc13406c220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:27.822 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.820+0000 7fc13981d640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc1340cc6c0 0x7fc13406c760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:27.822 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.820+0000 7fc13981d640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc13415c210 con 0x7fc1341563c0 2023-05-22T16:05:27.822 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.820+0000 7fc13981d640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc134160700 con 0x7fc1340cc6c0 2023-05-22T16:05:27.822 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.820+0000 7fc13981d640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc13415c480 con 0x7fc1340c8250 2023-05-22T16:05:27.822 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.820+0000 7fc132ffd640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fc1341563c0 0x7fc1340c7b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:48526/0 (socket says 172.21.15.16:48526) 2023-05-22T16:05:27.822 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.820+0000 7fc132ffd640 1 -- 172.21.15.16:0/2381157462 learned_addr learned my addr 172.21.15.16:0/2381157462 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T16:05:27.823 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.820+0000 7fc1337fe640 1 --2- 172.21.15.16:0/2381157462 >> v2:172.21.15.16:3300/0 conn(0x7fc1340cc6c0 0x7fc13406c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:27.823 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.820+0000 7fc1327fc640 1 --2- 172.21.15.16:0/2381157462 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc1340c8250 0x7fc13406c220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:27.823 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.820+0000 7fc1337fe640 1 -- 172.21.15.16:0/2381157462 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc1340c8250 msgr2=0x7fc13406c220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:05:27.823 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.820+0000 7fc1337fe640 1 --2- 172.21.15.16:0/2381157462 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc1340c8250 0x7fc13406c220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:27.823 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.820+0000 7fc1337fe640 1 -- 172.21.15.16:0/2381157462 >> v1:172.21.15.16:6789/0 conn(0x7fc1341563c0 legacy=0x7fc1340c7b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:27.823 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.820+0000 7fc131ffb640 1 -- 172.21.15.16:0/2381157462 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2461607850 0 0) 0x7fc13415c210 con 0x7fc1341563c0 2023-05-22T16:05:27.823 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.820+0000 7fc1337fe640 1 -- 172.21.15.16:0/2381157462 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc13415c760 con 0x7fc1340cc6c0 2023-05-22T16:05:27.823 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.820+0000 7fc1327fc640 1 --2- 172.21.15.16:0/2381157462 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc1340c8250 0x7fc13406c220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T16:05:27.825 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.820+0000 7fc1337fe640 1 --2- 172.21.15.16:0/2381157462 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc1340cc6c0 0x7fc13406c760 secure :-1 s=READY pgs=2465 cs=0 l=1 rev1=1 crypto rx=0x7fc12c0097c0 tx=0x7fc12c01c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9be2309fce4659b1 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:27.825 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.820+0000 7fc131ffb640 1 -- 172.21.15.16:0/2381157462 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc12c0ab650 con 0x7fc1340cc6c0 2023-05-22T16:05:27.826 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.820+0000 7fc131ffb640 1 -- 172.21.15.16:0/2381157462 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc12c0ab7b0 con 0x7fc1340cc6c0 2023-05-22T16:05:27.826 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.820+0000 7fc131ffb640 1 -- 172.21.15.16:0/2381157462 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc12c0abb40 con 0x7fc1340cc6c0 2023-05-22T16:05:27.826 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.824+0000 7fc13981d640 1 -- 172.21.15.16:0/2381157462 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc1340cc6c0 msgr2=0x7fc13406c760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:27.826 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.824+0000 7fc13981d640 1 --2- 172.21.15.16:0/2381157462 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc1340cc6c0 0x7fc13406c760 secure :-1 s=READY pgs=2465 cs=0 l=1 rev1=1 crypto rx=0x7fc12c0097c0 tx=0x7fc12c01c450 comp rx=0 tx=0).stop 2023-05-22T16:05:27.827 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.824+0000 7fc13981d640 1 -- 172.21.15.16:0/2381157462 shutdown_connections 2023-05-22T16:05:27.827 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.824+0000 7fc13981d640 1 --2- 172.21.15.16:0/2381157462 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc1340cc6c0 0x7fc13406c760 unknown :-1 s=CLOSED pgs=2465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:27.827 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.824+0000 7fc13981d640 1 --2- 172.21.15.16:0/2381157462 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc1340c8250 0x7fc13406c220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:27.828 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.824+0000 7fc13981d640 1 -- 172.21.15.16:0/2381157462 >> 172.21.15.16:0/2381157462 conn(0x7fc1340b1110 msgr2=0x7fc1340ce830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:05:27.828 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.824+0000 7fc13981d640 1 -- 172.21.15.16:0/2381157462 shutdown_connections 2023-05-22T16:05:27.829 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.824+0000 7fc13981d640 1 -- 172.21.15.16:0/2381157462 wait complete. 2023-05-22T16:05:27.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.824+0000 7fc13981d640 1 Processor -- start 2023-05-22T16:05:27.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.824+0000 7fc13981d640 1 -- start start 2023-05-22T16:05:27.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.824+0000 7fc13981d640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc1340c8250 0x7fc134146cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:27.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.824+0000 7fc13981d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc1340cc6c0 0x7fc134154c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:27.834 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.824+0000 7fc13981d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc13415ef20 0x7fc134144ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:27.834 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.824+0000 7fc13981d640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1341474c0 con 0x7fc1340cc6c0 2023-05-22T16:05:27.834 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.824+0000 7fc13981d640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc134153810 con 0x7fc13415ef20 2023-05-22T16:05:27.834 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.824+0000 7fc13981d640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc134153250 con 0x7fc1340c8250 2023-05-22T16:05:27.834 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.824+0000 7fc1327fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc1340cc6c0 0x7fc134154c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:27.835 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.824+0000 7fc1327fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc1340cc6c0 0x7fc134154c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41840/0 (socket says 172.21.15.16:41840) 2023-05-22T16:05:27.835 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.824+0000 7fc1327fc640 1 -- 172.21.15.16:0/4208979999 learned_addr learned my addr 172.21.15.16:0/4208979999 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:05:27.835 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.824+0000 7fc132ffd640 1 --2- 172.21.15.16:0/4208979999 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc1340c8250 0x7fc134146cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:27.835 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.824+0000 7fc1337fe640 1 --2- 172.21.15.16:0/4208979999 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc13415ef20 0x7fc134144ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:27.835 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.824+0000 7fc1327fc640 1 -- 172.21.15.16:0/4208979999 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc1340c8250 msgr2=0x7fc134146cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:27.836 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.824+0000 7fc1327fc640 1 --2- 172.21.15.16:0/4208979999 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc1340c8250 0x7fc134146cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:27.836 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.824+0000 7fc1327fc640 1 -- 172.21.15.16:0/4208979999 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc13415ef20 msgr2=0x7fc134144ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:27.836 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.824+0000 7fc1327fc640 1 --2- 172.21.15.16:0/4208979999 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc13415ef20 0x7fc134144ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:27.836 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.824+0000 7fc1327fc640 1 -- 172.21.15.16:0/4208979999 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc12c009420 con 0x7fc1340cc6c0 2023-05-22T16:05:27.837 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.824+0000 7fc1327fc640 1 --2- 172.21.15.16:0/4208979999 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc1340cc6c0 0x7fc134154c40 secure :-1 s=READY pgs=2466 cs=0 l=1 rev1=1 crypto rx=0x7fc12800cf30 tx=0x7fc12800a7d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:27.837 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.824+0000 7fc11bfff640 1 -- 172.21.15.16:0/4208979999 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc12800d840 con 0x7fc1340cc6c0 2023-05-22T16:05:27.837 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.824+0000 7fc11bfff640 1 -- 172.21.15.16:0/4208979999 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc12800d9a0 con 0x7fc1340cc6c0 2023-05-22T16:05:27.837 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.824+0000 7fc11bfff640 1 -- 172.21.15.16:0/4208979999 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc12800dd30 con 0x7fc1340cc6c0 2023-05-22T16:05:27.837 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.824+0000 7fc13981d640 1 -- 172.21.15.16:0/4208979999 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc13406ce30 con 0x7fc1340cc6c0 2023-05-22T16:05:27.837 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.824+0000 7fc13981d640 1 -- 172.21.15.16:0/4208979999 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc13406cca0 con 0x7fc1340cc6c0 2023-05-22T16:05:27.837 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.828+0000 7fc13981d640 1 -- 172.21.15.16:0/4208979999 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc13415ff00 con 0x7fc1340cc6c0 2023-05-22T16:05:27.838 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.828+0000 7fc11bfff640 1 -- 172.21.15.16:0/4208979999 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 92) v1 ==== 84332+0+0 (secure 0 0 0) 0x7fc128015440 con 0x7fc1340cc6c0 2023-05-22T16:05:27.838 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.828+0000 7fc11bfff640 1 --2- 172.21.15.16:0/4208979999 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fc1200654a0 0x7fc120067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:27.838 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.828+0000 7fc11bfff640 1 -- 172.21.15.16:0/4208979999 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(196..196 src has 1..196) v4 ==== 6438+0+0 (secure 0 0 0) 0x7fc1280834b0 con 0x7fc1340cc6c0 2023-05-22T16:05:27.838 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.828+0000 7fc132ffd640 1 --2- 172.21.15.16:0/4208979999 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fc1200654a0 0x7fc120067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:27.838 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.828+0000 7fc132ffd640 1 --2- 172.21.15.16:0/4208979999 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fc1200654a0 0x7fc120067950 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fc124000f70 tx=0x7fc124007c10 comp rx=0 tx=0).ready entity=mgr.10542 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:27.839 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:27.832+0000 7fc11bfff640 1 -- 172.21.15.16:0/4208979999 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fc12804d620 con 0x7fc1340cc6c0 2023-05-22T16:05:28.019 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:28.016+0000 7fc13981d640 1 -- 172.21.15.16:0/4208979999 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fc13404dbe0 con 0x7fc1340cc6c0 2023-05-22T16:05:28.019 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:28.016+0000 7fc11bfff640 1 -- 172.21.15.16:0/4208979999 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v81) v1 ==== 53+0+73 (secure 0 0 0) 0x7fc128050ee0 con 0x7fc1340cc6c0 2023-05-22T16:05:28.020 INFO:teuthology.orchestra.run.smithi016.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-05-22T16:05:28.021 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:28.020+0000 7fc13981d640 1 -- 172.21.15.16:0/4208979999 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fc1200654a0 msgr2=0x7fc120067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:28.022 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:28.020+0000 7fc13981d640 1 --2- 172.21.15.16:0/4208979999 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fc1200654a0 0x7fc120067950 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fc124000f70 tx=0x7fc124007c10 comp rx=0 tx=0).stop 2023-05-22T16:05:28.022 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:28.020+0000 7fc13981d640 1 -- 172.21.15.16:0/4208979999 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc1340cc6c0 msgr2=0x7fc134154c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:28.022 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:28.020+0000 7fc13981d640 1 --2- 172.21.15.16:0/4208979999 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc1340cc6c0 0x7fc134154c40 secure :-1 s=READY pgs=2466 cs=0 l=1 rev1=1 crypto rx=0x7fc12800cf30 tx=0x7fc12800a7d0 comp rx=0 tx=0).stop 2023-05-22T16:05:28.022 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:28.020+0000 7fc13981d640 1 -- 172.21.15.16:0/4208979999 shutdown_connections 2023-05-22T16:05:28.022 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:28.020+0000 7fc13981d640 1 --2- 172.21.15.16:0/4208979999 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc13415ef20 0x7fc134144ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:28.022 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:28.020+0000 7fc13981d640 1 --2- 172.21.15.16:0/4208979999 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fc1200654a0 0x7fc120067950 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:28.023 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:28.020+0000 7fc13981d640 1 --2- 172.21.15.16:0/4208979999 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc1340cc6c0 0x7fc134154c40 unknown :-1 s=CLOSED pgs=2466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:28.023 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:28.020+0000 7fc13981d640 1 --2- 172.21.15.16:0/4208979999 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc1340c8250 0x7fc134146cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:28.023 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:28.020+0000 7fc13981d640 1 -- 172.21.15.16:0/4208979999 >> 172.21.15.16:0/4208979999 conn(0x7fc1340b1110 msgr2=0x7fc1340c7b40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:05:28.023 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:28.020+0000 7fc13981d640 1 -- 172.21.15.16:0/4208979999 shutdown_connections 2023-05-22T16:05:28.023 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:28.020+0000 7fc13981d640 1 -- 172.21.15.16:0/4208979999 wait complete. 2023-05-22T16:05:28.038 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-05-22T16:05:28.039 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-05-22T16:05:28.039 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi016.front.sepia.ceph.com 2023-05-22T16:05:28.039 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-05-22T16:05:28.040 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-05-22T16:05:28.040 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-05-22T16:05:28.040 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-05-22T16:05:28.040 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-05-22T16:05:28.041 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-05-22T16:05:28.041 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T16:05:28.041 DEBUG:teuthology.orchestra.run.smithi016:> ip addr 2023-05-22T16:05:28.047 INFO:teuthology.orchestra.run.smithi016.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-05-22T16:05:28.048 INFO:teuthology.orchestra.run.smithi016.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-05-22T16:05:28.048 INFO:teuthology.orchestra.run.smithi016.stdout: inet 127.0.0.1/8 scope host lo 2023-05-22T16:05:28.048 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-05-22T16:05:28.048 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 ::1/128 scope host 2023-05-22T16:05:28.048 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-05-22T16:05:28.048 INFO:teuthology.orchestra.run.smithi016.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-05-22T16:05:28.048 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:6b:fd:e0 brd ff:ff:ff:ff:ff:ff 2023-05-22T16:05:28.049 INFO:teuthology.orchestra.run.smithi016.stdout: altname enp7s0f0 2023-05-22T16:05:28.049 INFO:teuthology.orchestra.run.smithi016.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-05-22T16:05:28.049 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:6b:fd:e1 brd ff:ff:ff:ff:ff:ff 2023-05-22T16:05:28.049 INFO:teuthology.orchestra.run.smithi016.stdout: altname enp7s0f1 2023-05-22T16:05:28.049 INFO:teuthology.orchestra.run.smithi016.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2023-05-22T16:05:28.049 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:bd:15:64 brd ff:ff:ff:ff:ff:ff 2023-05-22T16:05:28.049 INFO:teuthology.orchestra.run.smithi016.stdout: altname enp1s0f0 2023-05-22T16:05:28.049 INFO:teuthology.orchestra.run.smithi016.stdout: inet 172.21.15.16/20 brd 172.21.15.255 scope global dynamic ens1f0 2023-05-22T16:05:28.050 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft 40188sec preferred_lft 40188sec 2023-05-22T16:05:28.050 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::ec4:7aff:febd:1564/64 scope link 2023-05-22T16:05:28.050 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-05-22T16:05:28.050 INFO:teuthology.orchestra.run.smithi016.stdout:5: ens1f1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-05-22T16:05:28.050 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:bd:15:65 brd ff:ff:ff:ff:ff:ff 2023-05-22T16:05:28.050 INFO:teuthology.orchestra.run.smithi016.stdout: altname enp1s0f1 2023-05-22T16:05:28.050 INFO:teuthology.orchestra.run.smithi016.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2023-05-22T16:05:28.051 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 02:42:1f:b8:98:0a brd ff:ff:ff:ff:ff:ff 2023-05-22T16:05:28.051 INFO:teuthology.orchestra.run.smithi016.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2023-05-22T16:05:28.051 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-05-22T16:05:28.051 INFO:teuthology.orchestra.run.smithi016.stdout:9: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-05-22T16:05:28.051 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 5e:1f:84:6d:79:ae brd ff:ff:ff:ff:ff:ff 2023-05-22T16:05:28.052 INFO:teuthology.orchestra.run.smithi016.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-05-22T16:05:28.052 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-05-22T16:05:28.052 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::5c1f:84ff:fe6d:79ae/64 scope link 2023-05-22T16:05:28.052 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-05-22T16:05:28.052 INFO:teuthology.orchestra.run.smithi016.stdout:10: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-05-22T16:05:28.052 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0e:5c:f6:1f:82:2f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-05-22T16:05:28.052 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::c5c:f6ff:fe1f:822f/64 scope link 2023-05-22T16:05:28.053 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-05-22T16:05:28.053 INFO:teuthology.orchestra.run.smithi016.stdout:11: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-05-22T16:05:28.053 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether c6:a6:3d:69:d0:db brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-05-22T16:05:28.053 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::c4a6:3dff:fe69:d0db/64 scope link 2023-05-22T16:05:28.053 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-05-22T16:05:28.053 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T16:05:28.054 DEBUG:teuthology.orchestra.run.smithi016:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-05-22T16:05:28.098 INFO:teuthology.orchestra.run.smithi016.stdout:1 2023-05-22T16:05:28.099 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T16:05:28.100 DEBUG:teuthology.orchestra.run.smithi016:> route 2023-05-22T16:05:28.145 INFO:teuthology.orchestra.run.smithi016.stdout:Kernel IP routing table 2023-05-22T16:05:28.146 INFO:teuthology.orchestra.run.smithi016.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-05-22T16:05:28.146 INFO:teuthology.orchestra.run.smithi016.stdout:default _gateway 0.0.0.0 UG 0 0 0 ens1f0 2023-05-22T16:05:28.146 INFO:teuthology.orchestra.run.smithi016.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2023-05-22T16:05:28.146 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 0 0 0 ens1f0 2023-05-22T16:05:28.146 INFO:teuthology.orchestra.run.smithi016.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-05-22T16:05:28.146 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T16:05:28.147 DEBUG:teuthology.orchestra.run.smithi016:> (cd / && exec bash -c ' 2023-05-22T16:05:28.147 DEBUG:teuthology.orchestra.run.smithi016:> set -e 2023-05-22T16:05:28.147 DEBUG:teuthology.orchestra.run.smithi016:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2023-05-22T16:05:28.148 DEBUG:teuthology.orchestra.run.smithi016:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2023-05-22T16:05:28.148 DEBUG:teuthology.orchestra.run.smithi016:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2023-05-22T16:05:28.148 DEBUG:teuthology.orchestra.run.smithi016:> ') 2023-05-22T16:05:28.220 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T16:05:28.221 DEBUG:teuthology.orchestra.run.smithi016:> ip netns list 2023-05-22T16:05:28.269 INFO:teuthology.orchestra.run.smithi016.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-05-22T16:05:28.269 INFO:teuthology.orchestra.run.smithi016.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-05-22T16:05:28.270 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-05-22T16:05:28.270 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T16:05:28.270 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip link set brx.0 up 2023-05-22T16:05:28.326 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2023-05-22T16:05:28.326 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-05-22T16:05:28.327 DEBUG:teuthology.orchestra.run.smithi016:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-05-22T16:05:28.374 INFO:teuthology.orchestra.run.smithi016.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-05-22T16:05:28.375 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-05-22T16:05:28.375 DEBUG:teuthology.orchestra.run.smithi016:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-05-22T16:05:28.422 DEBUG:teuthology.orchestra.run.smithi016:> sudo modprobe fuse 2023-05-22T16:05:28.479 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-05-22T16:05:28.480 DEBUG:teuthology.orchestra.run.smithi016:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-05-22T16:05:28.533 INFO:teuthology.orchestra.run.smithi016.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-05-22T16:05:28.535 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-05-22T16:05:28.536 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T16:05:28.536 DEBUG:teuthology.orchestra.run.smithi016:> ls /sys/fs/fuse/connections 2023-05-22T16:05:28.582 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2023-05-22T16:05:28.583 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2023-05-22T16:05:28.627 DEBUG:teuthology.orchestra.run.smithi016:> sudo modprobe fuse 2023-05-22T16:05:28.644 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-05-22T16:05:28.645 DEBUG:teuthology.orchestra.run.smithi016:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-05-22T16:05:28.699 INFO:teuthology.orchestra.run.smithi016.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-05-22T16:05:28.700 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-05-22T16:05:28.701 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T16:05:28.702 DEBUG:teuthology.orchestra.run.smithi016:> ls /sys/fs/fuse/connections 2023-05-22T16:05:28.715 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi016.stderr:2023-05-22T16:05:28.712+0000 7f2f5333d580 -1 init, newargv = 0x55c0a3efcc60 newargc=15 2023-05-22T16:05:28.715 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi016.stderr:ceph-fuse[69985]: starting ceph client 2023-05-22T16:05:28.721 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi016.stderr:ceph-fuse[69985]: starting fuse 2023-05-22T16:05:29.819 DEBUG:teuthology.orchestra.run.smithi016:> sudo modprobe fuse 2023-05-22T16:05:29.836 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-05-22T16:05:29.836 DEBUG:teuthology.orchestra.run.smithi016:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-05-22T16:05:29.893 INFO:teuthology.orchestra.run.smithi016.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-05-22T16:05:29.894 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-05-22T16:05:29.894 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T16:05:29.895 DEBUG:teuthology.orchestra.run.smithi016:> ls /sys/fs/fuse/connections 2023-05-22T16:05:29.943 INFO:teuthology.orchestra.run.smithi016.stdout:45 2023-05-22T16:05:29.943 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [45] 2023-05-22T16:05:29.944 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2023-05-22T16:05:29.944 DEBUG:teuthology.orchestra.run.smithi016:> import glob 2023-05-22T16:05:29.944 DEBUG:teuthology.orchestra.run.smithi016:> import re 2023-05-22T16:05:29.945 DEBUG:teuthology.orchestra.run.smithi016:> import os 2023-05-22T16:05:29.945 DEBUG:teuthology.orchestra.run.smithi016:> import subprocess 2023-05-22T16:05:29.945 DEBUG:teuthology.orchestra.run.smithi016:> 2023-05-22T16:05:29.946 DEBUG:teuthology.orchestra.run.smithi016:> def _find_admin_socket(client_name): 2023-05-22T16:05:29.946 DEBUG:teuthology.orchestra.run.smithi016:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2023-05-22T16:05:29.946 DEBUG:teuthology.orchestra.run.smithi016:> files = glob.glob(asok_path) 2023-05-22T16:05:29.947 DEBUG:teuthology.orchestra.run.smithi016:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2023-05-22T16:05:29.947 DEBUG:teuthology.orchestra.run.smithi016:> 2023-05-22T16:05:29.947 DEBUG:teuthology.orchestra.run.smithi016:> # Given a non-glob path, it better be there 2023-05-22T16:05:29.947 DEBUG:teuthology.orchestra.run.smithi016:> if "*" not in asok_path: 2023-05-22T16:05:29.948 DEBUG:teuthology.orchestra.run.smithi016:> assert(len(files) == 1) 2023-05-22T16:05:29.948 DEBUG:teuthology.orchestra.run.smithi016:> return files[0] 2023-05-22T16:05:29.948 DEBUG:teuthology.orchestra.run.smithi016:> 2023-05-22T16:05:29.949 DEBUG:teuthology.orchestra.run.smithi016:> for f in files: 2023-05-22T16:05:29.949 DEBUG:teuthology.orchestra.run.smithi016:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-05-22T16:05:29.949 DEBUG:teuthology.orchestra.run.smithi016:> if os.path.exists("/proc/{0}".format(pid)): 2023-05-22T16:05:29.950 DEBUG:teuthology.orchestra.run.smithi016:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-05-22T16:05:29.950 DEBUG:teuthology.orchestra.run.smithi016:> contents = proc_f.read() 2023-05-22T16:05:29.951 DEBUG:teuthology.orchestra.run.smithi016:> if mountpoint in contents: 2023-05-22T16:05:29.951 DEBUG:teuthology.orchestra.run.smithi016:> return f 2023-05-22T16:05:29.951 DEBUG:teuthology.orchestra.run.smithi016:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-05-22T16:05:29.952 DEBUG:teuthology.orchestra.run.smithi016:> 2023-05-22T16:05:29.952 DEBUG:teuthology.orchestra.run.smithi016:> print(_find_admin_socket("client.0")) 2023-05-22T16:05:29.952 DEBUG:teuthology.orchestra.run.smithi016:> ' 2023-05-22T16:05:30.065 INFO:teuthology.orchestra.run.smithi016.stdout:/var/run/ceph/ceph-client.0.69985.asok 2023-05-22T16:05:30.248 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.69985.asok 2023-05-22T16:05:30.249 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T16:05:30.250 DEBUG:teuthology.orchestra.run.smithi016:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.69985.asok status 2023-05-22T16:05:30.415 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-05-22T16:05:30.416 INFO:teuthology.orchestra.run.smithi016.stdout: "metadata": { 2023-05-22T16:05:30.416 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph_sha1": "9c9ba4df22024d9d8d6cf9f98a4b96751fc78d0c", 2023-05-22T16:05:30.416 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph_version": "ceph version 18.0.0-3192-g9c9ba4df (9c9ba4df22024d9d8d6cf9f98a4b96751fc78d0c) reef (dev)", 2023-05-22T16:05:30.416 INFO:teuthology.orchestra.run.smithi016.stdout: "entity_id": "0", 2023-05-22T16:05:30.416 INFO:teuthology.orchestra.run.smithi016.stdout: "hostname": "smithi016", 2023-05-22T16:05:30.416 INFO:teuthology.orchestra.run.smithi016.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2023-05-22T16:05:30.417 INFO:teuthology.orchestra.run.smithi016.stdout: "pid": "69985", 2023-05-22T16:05:30.417 INFO:teuthology.orchestra.run.smithi016.stdout: "root": "/" 2023-05-22T16:05:30.417 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T16:05:30.417 INFO:teuthology.orchestra.run.smithi016.stdout: "dentry_count": 0, 2023-05-22T16:05:30.418 INFO:teuthology.orchestra.run.smithi016.stdout: "dentry_pinned_count": 0, 2023-05-22T16:05:30.418 INFO:teuthology.orchestra.run.smithi016.stdout: "id": 10953, 2023-05-22T16:05:30.418 INFO:teuthology.orchestra.run.smithi016.stdout: "inst": { 2023-05-22T16:05:30.418 INFO:teuthology.orchestra.run.smithi016.stdout: "name": { 2023-05-22T16:05:30.418 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "client", 2023-05-22T16:05:30.418 INFO:teuthology.orchestra.run.smithi016.stdout: "num": 10953 2023-05-22T16:05:30.418 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T16:05:30.419 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": { 2023-05-22T16:05:30.419 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-05-22T16:05:30.419 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "192.168.0.1:0", 2023-05-22T16:05:30.419 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 1291094344 2023-05-22T16:05:30.419 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T16:05:30.419 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T16:05:30.419 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": { 2023-05-22T16:05:30.420 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-05-22T16:05:30.420 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "192.168.0.1:0", 2023-05-22T16:05:30.420 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 1291094344 2023-05-22T16:05:30.420 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T16:05:30.420 INFO:teuthology.orchestra.run.smithi016.stdout: "inst_str": "client.10953 192.168.0.1:0/1291094344", 2023-05-22T16:05:30.420 INFO:teuthology.orchestra.run.smithi016.stdout: "addr_str": "192.168.0.1:0/1291094344", 2023-05-22T16:05:30.421 INFO:teuthology.orchestra.run.smithi016.stdout: "inode_count": 1, 2023-05-22T16:05:30.421 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_epoch": 81, 2023-05-22T16:05:30.421 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_epoch": 196, 2023-05-22T16:05:30.421 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_epoch_barrier": 0, 2023-05-22T16:05:30.421 INFO:teuthology.orchestra.run.smithi016.stdout: "blocklisted": false, 2023-05-22T16:05:30.421 INFO:teuthology.orchestra.run.smithi016.stdout: "fs_name": "cephfs" 2023-05-22T16:05:30.421 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-05-22T16:05:30.429 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T16:05:30.430 DEBUG:teuthology.orchestra.run.smithi016:> stat --file-system '--printf=%T 2023-05-22T16:05:30.430 DEBUG:teuthology.orchestra.run.smithi016:> ' -- /home/ubuntu/cephtest/mnt.0 2023-05-22T16:05:30.436 INFO:teuthology.orchestra.run.smithi016.stdout:fuseblk 2023-05-22T16:05:30.436 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2023-05-22T16:05:30.437 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T16:05:30.437 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-05-22T16:05:30.493 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2023-05-22T16:05:30.654 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:30.652+0000 7fdaa1884640 1 Processor -- start 2023-05-22T16:05:30.655 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:30.652+0000 7fdaa1884640 1 -- start start 2023-05-22T16:05:30.656 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:30.652+0000 7fdaa1884640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fda9c0ca4e0 0x7fda9c15e460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:30.656 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:30.652+0000 7fdaa1884640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fda9c070df0 0x7fda9c071210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:30.656 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:30.652+0000 7fdaa1884640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda9c0b0650 con 0x7fda9c0c9b40 2023-05-22T16:05:30.656 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:30.652+0000 7fdaa1884640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda9c0b0370 con 0x7fda9c070df0 2023-05-22T16:05:30.656 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:30.652+0000 7fdaa1884640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda9c15d980 con 0x7fda9c0ca4e0 2023-05-22T16:05:30.656 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:30.652+0000 7fda9b7fe640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fda9c070df0 0x7fda9c071210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:30.657 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:30.652+0000 7fda9b7fe640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fda9c070df0 0x7fda9c071210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42866/0 (socket says 172.21.15.16:42866) 2023-05-22T16:05:30.657 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:30.652+0000 7fda9b7fe640 1 -- 172.21.15.16:0/365608205 learned_addr learned my addr 172.21.15.16:0/365608205 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:05:30.657 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:30.652+0000 7fda9affd640 1 --2- 172.21.15.16:0/365608205 >> v2:172.21.15.62:3300/0 conn(0x7fda9c0ca4e0 0x7fda9c15e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:30.657 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:30.652+0000 7fda9b7fe640 1 -- 172.21.15.16:0/365608205 >> v2:172.21.15.62:3300/0 conn(0x7fda9c0ca4e0 msgr2=0x7fda9c15e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:05:30.657 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:30.652+0000 7fda9b7fe640 1 --2- 172.21.15.16:0/365608205 >> v2:172.21.15.62:3300/0 conn(0x7fda9c0ca4e0 0x7fda9c15e460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:30.658 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:30.652+0000 7fda9b7fe640 1 -- 172.21.15.16:0/365608205 >> v1:172.21.15.62:6789/0 conn(0x7fda9c0c9b40 legacy=0x7fda9c0afb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:05:30.658 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:30.652+0000 7fda9b7fe640 1 -- 172.21.15.16:0/365608205 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda9c15dc60 con 0x7fda9c070df0 2023-05-22T16:05:30.659 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:30.656+0000 7fda9b7fe640 1 --2- 172.21.15.16:0/365608205 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fda9c070df0 0x7fda9c071210 secure :-1 s=READY pgs=2470 cs=0 l=1 rev1=1 crypto rx=0x7fda9c0cad40 tx=0x7fda9401c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d258d80ac43f7c6f server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:30.659 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:30.656+0000 7fda99ffb640 1 -- 172.21.15.16:0/365608205 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda940a5780 con 0x7fda9c070df0 2023-05-22T16:05:30.659 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:30.656+0000 7fda99ffb640 1 -- 172.21.15.16:0/365608205 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda940a58e0 con 0x7fda9c070df0 2023-05-22T16:05:30.659 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:30.656+0000 7fda99ffb640 1 -- 172.21.15.16:0/365608205 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda940a5c70 con 0x7fda9c070df0 2023-05-22T16:05:30.659 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:30.656+0000 7fdaa1884640 1 -- 172.21.15.16:0/365608205 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fda9c070df0 msgr2=0x7fda9c071210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:30.659 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:30.656+0000 7fdaa1884640 1 --2- 172.21.15.16:0/365608205 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fda9c070df0 0x7fda9c071210 secure :-1 s=READY pgs=2470 cs=0 l=1 rev1=1 crypto rx=0x7fda9c0cad40 tx=0x7fda9401c960 comp rx=0 tx=0).stop 2023-05-22T16:05:30.660 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:30.656+0000 7fdaa1884640 1 -- 172.21.15.16:0/365608205 shutdown_connections 2023-05-22T16:05:30.660 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:30.656+0000 7fdaa1884640 1 --2- 172.21.15.16:0/365608205 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fda9c070df0 0x7fda9c071210 unknown :-1 s=CLOSED pgs=2470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:30.660 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:30.656+0000 7fdaa1884640 1 --2- 172.21.15.16:0/365608205 >> v2:172.21.15.62:3300/0 conn(0x7fda9c0ca4e0 0x7fda9c15e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:30.660 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:30.656+0000 7fdaa1884640 1 -- 172.21.15.16:0/365608205 >> 172.21.15.16:0/365608205 conn(0x7fda9c069720 msgr2=0x7fda9c069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:05:30.660 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:30.656+0000 7fdaa1884640 1 -- 172.21.15.16:0/365608205 shutdown_connections 2023-05-22T16:05:30.660 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:30.656+0000 7fdaa1884640 1 -- 172.21.15.16:0/365608205 wait complete. 2023-05-22T16:05:30.662 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:30.656+0000 7fdaa1884640 1 Processor -- start 2023-05-22T16:05:30.662 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:30.656+0000 7fdaa1884640 1 -- start start 2023-05-22T16:05:30.662 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:30.660+0000 7fdaa1884640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fda9c070df0 0x7fda9c151240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:30.662 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:30.660+0000 7fdaa1884640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fda9c0c9b40 0x7fda9c1533c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:30.662 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:30.660+0000 7fdaa1884640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fda9c0ca4e0 0x7fda9c0cd260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:30.662 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:30.660+0000 7fdaa1884640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda9c0ccdd0 con 0x7fda9c0ca4e0 2023-05-22T16:05:30.663 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:30.660+0000 7fdaa1884640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda9c0cc1e0 con 0x7fda9c0c9b40 2023-05-22T16:05:30.663 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:30.660+0000 7fdaa1884640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda9c0cc4a0 con 0x7fda9c070df0 2023-05-22T16:05:30.663 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:30.660+0000 7fda9b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fda9c0ca4e0 0x7fda9c0cd260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:30.663 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:30.660+0000 7fda9b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fda9c0ca4e0 0x7fda9c0cd260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42880/0 (socket says 172.21.15.16:42880) 2023-05-22T16:05:30.663 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:30.660+0000 7fda9b7fe640 1 -- 172.21.15.16:0/2056641328 learned_addr learned my addr 172.21.15.16:0/2056641328 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:05:30.663 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:30.660+0000 7fda9affd640 1 --2- 172.21.15.16:0/2056641328 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fda9c070df0 0x7fda9c151240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:30.663 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:30.660+0000 7fda9a7fc640 1 --2- 172.21.15.16:0/2056641328 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fda9c0c9b40 0x7fda9c1533c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:30.663 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:30.660+0000 7fda9b7fe640 1 -- 172.21.15.16:0/2056641328 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fda9c070df0 msgr2=0x7fda9c151240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:30.664 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:30.660+0000 7fda9b7fe640 1 --2- 172.21.15.16:0/2056641328 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fda9c070df0 0x7fda9c151240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:30.664 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:30.660+0000 7fda9b7fe640 1 -- 172.21.15.16:0/2056641328 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fda9c0c9b40 msgr2=0x7fda9c1533c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:30.664 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:30.660+0000 7fda9b7fe640 1 --2- 172.21.15.16:0/2056641328 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fda9c0c9b40 0x7fda9c1533c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:30.664 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:30.660+0000 7fda9b7fe640 1 -- 172.21.15.16:0/2056641328 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda94009b30 con 0x7fda9c0ca4e0 2023-05-22T16:05:30.664 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:30.660+0000 7fda9b7fe640 1 --2- 172.21.15.16:0/2056641328 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fda9c0ca4e0 0x7fda9c0cd260 secure :-1 s=READY pgs=2471 cs=0 l=1 rev1=1 crypto rx=0x7fda940af040 tx=0x7fda94019700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:30.664 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:30.660+0000 7fda93fff640 1 -- 172.21.15.16:0/2056641328 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda940b8670 con 0x7fda9c0ca4e0 2023-05-22T16:05:30.664 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:30.660+0000 7fda93fff640 1 -- 172.21.15.16:0/2056641328 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda940b87d0 con 0x7fda9c0ca4e0 2023-05-22T16:05:30.665 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:30.660+0000 7fda93fff640 1 -- 172.21.15.16:0/2056641328 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda940b8b80 con 0x7fda9c0ca4e0 2023-05-22T16:05:30.665 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:30.660+0000 7fdaa1884640 1 -- 172.21.15.16:0/2056641328 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda9c0c8e80 con 0x7fda9c0ca4e0 2023-05-22T16:05:30.666 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:30.660+0000 7fdaa1884640 1 -- 172.21.15.16:0/2056641328 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda9c0ca8b0 con 0x7fda9c0ca4e0 2023-05-22T16:05:30.666 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:30.660+0000 7fdaa1884640 1 -- 172.21.15.16:0/2056641328 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda9c0c9010 con 0x7fda9c0ca4e0 2023-05-22T16:05:30.666 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:30.664+0000 7fda93fff640 1 -- 172.21.15.16:0/2056641328 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 92) v1 ==== 84332+0+0 (secure 0 0 0) 0x7fda940b8ce0 con 0x7fda9c0ca4e0 2023-05-22T16:05:30.666 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:30.664+0000 7fda93fff640 1 --2- 172.21.15.16:0/2056641328 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fda7c0654a0 0x7fda7c067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:30.666 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:30.664+0000 7fda93fff640 1 -- 172.21.15.16:0/2056641328 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(196..196 src has 1..196) v4 ==== 6438+0+0 (secure 0 0 0) 0x7fda9411e1a0 con 0x7fda9c0ca4e0 2023-05-22T16:05:30.666 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:30.664+0000 7fda9affd640 1 --2- 172.21.15.16:0/2056641328 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fda7c0654a0 0x7fda7c067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:30.666 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:30.664+0000 7fda9affd640 1 --2- 172.21.15.16:0/2056641328 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fda7c0654a0 0x7fda7c067950 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fda9c1531b0 tx=0x7fda8c008040 comp rx=0 tx=0).ready entity=mgr.10542 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:30.669 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:30.668+0000 7fda93fff640 1 -- 172.21.15.16:0/2056641328 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fda940ebbb0 con 0x7fda9c0ca4e0 2023-05-22T16:05:30.843 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:30.840+0000 7fdaa1884640 1 -- 172.21.15.16:0/2056641328 --> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fda9c0c9010 con 0x7fda7c0654a0 2023-05-22T16:05:30.845 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:30.844+0000 7fda93fff640 1 -- 172.21.15.16:0/2056641328 <== mgr.10542 v2:172.21.15.16:6832/65890 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7fda9c0c9010 con 0x7fda7c0654a0 2023-05-22T16:05:30.846 INFO:teuthology.orchestra.run.smithi016.stdout:[ 2023-05-22T16:05:30.846 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-05-22T16:05:30.846 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "cephfs" 2023-05-22T16:05:30.846 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T16:05:30.846 INFO:teuthology.orchestra.run.smithi016.stdout:] 2023-05-22T16:05:30.848 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:30.844+0000 7fdaa1884640 1 -- 172.21.15.16:0/2056641328 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fda7c0654a0 msgr2=0x7fda7c067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:30.848 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:30.844+0000 7fdaa1884640 1 --2- 172.21.15.16:0/2056641328 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fda7c0654a0 0x7fda7c067950 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fda9c1531b0 tx=0x7fda8c008040 comp rx=0 tx=0).stop 2023-05-22T16:05:30.848 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:30.844+0000 7fdaa1884640 1 -- 172.21.15.16:0/2056641328 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fda9c0ca4e0 msgr2=0x7fda9c0cd260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:30.848 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:30.844+0000 7fdaa1884640 1 --2- 172.21.15.16:0/2056641328 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fda9c0ca4e0 0x7fda9c0cd260 secure :-1 s=READY pgs=2471 cs=0 l=1 rev1=1 crypto rx=0x7fda940af040 tx=0x7fda94019700 comp rx=0 tx=0).stop 2023-05-22T16:05:30.848 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:30.844+0000 7fdaa1884640 1 -- 172.21.15.16:0/2056641328 shutdown_connections 2023-05-22T16:05:30.849 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:30.844+0000 7fdaa1884640 1 --2- 172.21.15.16:0/2056641328 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fda7c0654a0 0x7fda7c067950 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:30.849 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:30.844+0000 7fdaa1884640 1 --2- 172.21.15.16:0/2056641328 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fda9c0ca4e0 0x7fda9c0cd260 unknown :-1 s=CLOSED pgs=2471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:30.849 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:30.844+0000 7fdaa1884640 1 --2- 172.21.15.16:0/2056641328 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fda9c0c9b40 0x7fda9c1533c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:30.849 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:30.844+0000 7fdaa1884640 1 --2- 172.21.15.16:0/2056641328 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fda9c070df0 0x7fda9c151240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:30.849 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:30.844+0000 7fdaa1884640 1 -- 172.21.15.16:0/2056641328 >> 172.21.15.16:0/2056641328 conn(0x7fda9c069720 msgr2=0x7fda9c06bf30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:05:30.849 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:30.844+0000 7fdaa1884640 1 -- 172.21.15.16:0/2056641328 shutdown_connections 2023-05-22T16:05:30.849 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:30.844+0000 7fdaa1884640 1 -- 172.21.15.16:0/2056641328 wait complete. 2023-05-22T16:05:30.865 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2023-05-22T16:05:30.988 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:30.984+0000 7f963cf38640 1 Processor -- start 2023-05-22T16:05:30.989 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:30.984+0000 7f963cf38640 1 -- start start 2023-05-22T16:05:30.990 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:30.988+0000 7f963cf38640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f96380c67d0 0x7f96380cf730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:30.991 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:30.988+0000 7f963cf38640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f96380cc6f0 0x7f96380c3db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:30.991 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:30.988+0000 7f963cf38640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96380cfc70 con 0x7f96380b2880 2023-05-22T16:05:30.991 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:30.988+0000 7f963cf38640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f963815a300 con 0x7f96380c67d0 2023-05-22T16:05:30.991 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:30.988+0000 7f963cf38640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96380c6ba0 con 0x7f96380cc6f0 2023-05-22T16:05:30.991 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:30.988+0000 7f963659c640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f96380c67d0 0x7f96380cf730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:30.991 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:30.988+0000 7f963659c640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f96380c67d0 0x7f96380cf730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42892/0 (socket says 172.21.15.16:42892) 2023-05-22T16:05:30.992 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:30.988+0000 7f963659c640 1 -- 172.21.15.16:0/3135165283 learned_addr learned my addr 172.21.15.16:0/3135165283 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:05:30.992 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:30.988+0000 7f9636d9d640 1 --2- 172.21.15.16:0/3135165283 >> v2:172.21.15.62:3300/0 conn(0x7f96380cc6f0 0x7f96380c3db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:30.992 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:30.988+0000 7f963659c640 1 -- 172.21.15.16:0/3135165283 >> v2:172.21.15.62:3300/0 conn(0x7f96380cc6f0 msgr2=0x7f96380c3db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:05:30.992 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:30.988+0000 7f963659c640 1 --2- 172.21.15.16:0/3135165283 >> v2:172.21.15.62:3300/0 conn(0x7f96380cc6f0 0x7f96380c3db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:30.992 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:30.988+0000 7f963659c640 1 -- 172.21.15.16:0/3135165283 >> v1:172.21.15.62:6789/0 conn(0x7f96380b2880 legacy=0x7f96380cbfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:05:30.992 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:30.988+0000 7f963659c640 1 -- 172.21.15.16:0/3135165283 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96380cfef0 con 0x7f96380c67d0 2023-05-22T16:05:30.993 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:30.988+0000 7f963659c640 1 --2- 172.21.15.16:0/3135165283 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f96380c67d0 0x7f96380cf730 secure :-1 s=READY pgs=2472 cs=0 l=1 rev1=1 crypto rx=0x7f96380b6360 tx=0x7f962801c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b83d87f4be5e75ff server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:30.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:30.992+0000 7f963559a640 1 -- 172.21.15.16:0/3135165283 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96280a56a0 con 0x7f96380c67d0 2023-05-22T16:05:30.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:30.992+0000 7f963559a640 1 -- 172.21.15.16:0/3135165283 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96280a5800 con 0x7f96380c67d0 2023-05-22T16:05:30.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:30.992+0000 7f963559a640 1 -- 172.21.15.16:0/3135165283 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96280a5b90 con 0x7f96380c67d0 2023-05-22T16:05:30.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:30.992+0000 7f963cf38640 1 -- 172.21.15.16:0/3135165283 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f96380c67d0 msgr2=0x7f96380cf730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:30.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:30.992+0000 7f963cf38640 1 --2- 172.21.15.16:0/3135165283 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f96380c67d0 0x7f96380cf730 secure :-1 s=READY pgs=2472 cs=0 l=1 rev1=1 crypto rx=0x7f96380b6360 tx=0x7f962801c8d0 comp rx=0 tx=0).stop 2023-05-22T16:05:30.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:30.992+0000 7f963cf38640 1 -- 172.21.15.16:0/3135165283 shutdown_connections 2023-05-22T16:05:30.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:30.992+0000 7f963cf38640 1 --2- 172.21.15.16:0/3135165283 >> v2:172.21.15.62:3300/0 conn(0x7f96380cc6f0 0x7f96380c3db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:30.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:30.992+0000 7f963cf38640 1 --2- 172.21.15.16:0/3135165283 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f96380c67d0 0x7f96380cf730 unknown :-1 s=CLOSED pgs=2472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:30.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:30.992+0000 7f963cf38640 1 -- 172.21.15.16:0/3135165283 >> 172.21.15.16:0/3135165283 conn(0x7f963806f5a0 msgr2=0x7f963806f9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:05:30.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:30.992+0000 7f963cf38640 1 -- 172.21.15.16:0/3135165283 shutdown_connections 2023-05-22T16:05:30.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:30.992+0000 7f963cf38640 1 -- 172.21.15.16:0/3135165283 wait complete. 2023-05-22T16:05:30.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:30.992+0000 7f963cf38640 1 Processor -- start 2023-05-22T16:05:30.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:30.992+0000 7f963cf38640 1 -- start start 2023-05-22T16:05:30.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:30.992+0000 7f963cf38640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f96380b2880 0x7f96381457d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:30.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:30.992+0000 7f963cf38640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f96380c67d0 0x7f963815dec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:30.998 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:30.992+0000 7f963cf38640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f96380cc6f0 0x7f96381542e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:30.998 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:30.992+0000 7f963cf38640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f963815e400 con 0x7f96380c67d0 2023-05-22T16:05:30.998 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:30.992+0000 7f963cf38640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9638155690 con 0x7f96380cc6f0 2023-05-22T16:05:30.998 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:30.992+0000 7f963cf38640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9638154820 con 0x7f96380b2880 2023-05-22T16:05:30.998 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:30.992+0000 7f9635d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f96380c67d0 0x7f963815dec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:30.998 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:30.992+0000 7f9635d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f96380c67d0 0x7f963815dec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42902/0 (socket says 172.21.15.16:42902) 2023-05-22T16:05:30.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:30.992+0000 7f9635d9b640 1 -- 172.21.15.16:0/991745961 learned_addr learned my addr 172.21.15.16:0/991745961 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:05:30.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:30.992+0000 7f963659c640 1 --2- 172.21.15.16:0/991745961 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f96380b2880 0x7f96381457d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:30.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:30.992+0000 7f9636d9d640 1 --2- 172.21.15.16:0/991745961 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f96380cc6f0 0x7f96381542e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:30.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:30.992+0000 7f9635d9b640 1 -- 172.21.15.16:0/991745961 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f96380b2880 msgr2=0x7f96381457d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:30.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:30.992+0000 7f9635d9b640 1 --2- 172.21.15.16:0/991745961 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f96380b2880 0x7f96381457d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:30.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:30.992+0000 7f9635d9b640 1 -- 172.21.15.16:0/991745961 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f96380cc6f0 msgr2=0x7f96381542e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:30.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:30.992+0000 7f9635d9b640 1 --2- 172.21.15.16:0/991745961 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f96380cc6f0 0x7f96381542e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:31.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:30.992+0000 7f9635d9b640 1 -- 172.21.15.16:0/991745961 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9628009b30 con 0x7f96380c67d0 2023-05-22T16:05:31.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:30.992+0000 7f9635d9b640 1 --2- 172.21.15.16:0/991745961 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f96380c67d0 0x7f963815dec0 secure :-1 s=READY pgs=2473 cs=0 l=1 rev1=1 crypto rx=0x7f962c00b410 tx=0x7f962c00b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:31.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:30.992+0000 7f9626ffd640 1 -- 172.21.15.16:0/991745961 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f962c004020 con 0x7f96380c67d0 2023-05-22T16:05:31.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:30.996+0000 7f9626ffd640 1 -- 172.21.15.16:0/991745961 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f962c004180 con 0x7f96380c67d0 2023-05-22T16:05:31.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:30.996+0000 7f963cf38640 1 -- 172.21.15.16:0/991745961 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96380d0a70 con 0x7f96380c67d0 2023-05-22T16:05:31.001 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:30.996+0000 7f9626ffd640 1 -- 172.21.15.16:0/991745961 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f962c011630 con 0x7f96380c67d0 2023-05-22T16:05:31.001 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:30.996+0000 7f963cf38640 1 -- 172.21.15.16:0/991745961 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96380d0790 con 0x7f96380c67d0 2023-05-22T16:05:31.001 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:30.996+0000 7f963cf38640 1 -- 172.21.15.16:0/991745961 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96380d0f90 con 0x7f96380c67d0 2023-05-22T16:05:31.002 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:30.996+0000 7f9626ffd640 1 -- 172.21.15.16:0/991745961 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 92) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f962c011790 con 0x7f96380c67d0 2023-05-22T16:05:31.003 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:30.996+0000 7f9626ffd640 1 --2- 172.21.15.16:0/991745961 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f96180654a0 0x7f9618067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:31.003 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:30.996+0000 7f9626ffd640 1 -- 172.21.15.16:0/991745961 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(196..196 src has 1..196) v4 ==== 6438+0+0 (secure 0 0 0) 0x7f962c082890 con 0x7f96380c67d0 2023-05-22T16:05:31.003 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:30.996+0000 7f963659c640 1 --2- 172.21.15.16:0/991745961 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f96180654a0 0x7f9618067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:31.003 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:30.996+0000 7f963659c640 1 --2- 172.21.15.16:0/991745961 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f96180654a0 0x7f9618067950 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f96280ae000 tx=0x7f96280a7070 comp rx=0 tx=0).ready entity=mgr.10542 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:31.004 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:31.000+0000 7f9626ffd640 1 -- 172.21.15.16:0/991745961 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f962c050280 con 0x7f96380c67d0 2023-05-22T16:05:31.198 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:31.196+0000 7f963cf38640 1 -- 172.21.15.16:0/991745961 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7f96380cae70 con 0x7f96380c67d0 2023-05-22T16:05:32.210 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:32.208+0000 7f9626ffd640 1 -- 172.21.15.16:0/991745961 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mgrmap(e 93) v1 ==== 84349+0+0 (secure 0 0 0) 0x7f962c080ad0 con 0x7f96380c67d0 2023-05-22T16:05:32.211 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:32.208+0000 7f9626ffd640 1 -- 172.21.15.16:0/991745961 <== mon.0 v2:172.21.15.16:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v93) v1 ==== 92+0+0 (secure 0 0 0) 0x7f962c050280 con 0x7f96380c67d0 2023-05-22T16:05:32.214 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:32.212+0000 7f963cf38640 1 -- 172.21.15.16:0/991745961 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f96180654a0 msgr2=0x7f9618067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:32.214 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:32.212+0000 7f963cf38640 1 --2- 172.21.15.16:0/991745961 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f96180654a0 0x7f9618067950 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f96280ae000 tx=0x7f96280a7070 comp rx=0 tx=0).stop 2023-05-22T16:05:32.214 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:32.212+0000 7f963cf38640 1 -- 172.21.15.16:0/991745961 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f96380c67d0 msgr2=0x7f963815dec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:32.215 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:32.212+0000 7f963cf38640 1 --2- 172.21.15.16:0/991745961 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f96380c67d0 0x7f963815dec0 secure :-1 s=READY pgs=2473 cs=0 l=1 rev1=1 crypto rx=0x7f962c00b410 tx=0x7f962c00b8d0 comp rx=0 tx=0).stop 2023-05-22T16:05:32.215 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:32.212+0000 7f963cf38640 1 -- 172.21.15.16:0/991745961 shutdown_connections 2023-05-22T16:05:32.215 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:32.212+0000 7f963cf38640 1 --2- 172.21.15.16:0/991745961 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f96380cc6f0 0x7f96381542e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:32.216 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:32.212+0000 7f963cf38640 1 --2- 172.21.15.16:0/991745961 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f96180654a0 0x7f9618067950 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:32.216 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:32.212+0000 7f963cf38640 1 --2- 172.21.15.16:0/991745961 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f96380c67d0 0x7f963815dec0 unknown :-1 s=CLOSED pgs=2473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:32.216 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:32.212+0000 7f963cf38640 1 --2- 172.21.15.16:0/991745961 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f96380b2880 0x7f96381457d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:32.217 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:32.212+0000 7f963cf38640 1 -- 172.21.15.16:0/991745961 >> 172.21.15.16:0/991745961 conn(0x7f963806f5a0 msgr2=0x7f9638156c00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:05:32.217 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:32.212+0000 7f963cf38640 1 -- 172.21.15.16:0/991745961 shutdown_connections 2023-05-22T16:05:32.217 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:32.212+0000 7f963cf38640 1 -- 172.21.15.16:0/991745961 wait complete. 2023-05-22T16:05:32.234 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2023-05-22T16:05:32.334 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T16:05:32.331+0000 7ff926cc3180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-05-22T16:05:32.340 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T16:05:32.340+0000 7f460e0e4180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-05-22T16:05:32.355 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:32.352+0000 7fc861b1f640 1 Processor -- start 2023-05-22T16:05:32.355 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:32.352+0000 7fc861b1f640 1 -- start start 2023-05-22T16:05:32.356 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:32.352+0000 7fc861b1f640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fc85c071530 0x7fc85c15c580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:32.356 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:32.352+0000 7fc861b1f640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc85c06b6c0 0x7fc85c06bb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:32.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:32.352+0000 7fc861b1f640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc85c15d520 con 0x7fc85c070c70 2023-05-22T16:05:32.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:32.352+0000 7fc861b1f640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc85c071980 con 0x7fc85c071530 2023-05-22T16:05:32.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:32.352+0000 7fc861b1f640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc85c06c040 con 0x7fc85c06b6c0 2023-05-22T16:05:32.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:32.352+0000 7fc85b7fe640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fc85c070c70 0x7fc85c06fba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:57150/0 (socket says 172.21.15.16:57150) 2023-05-22T16:05:32.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:32.352+0000 7fc85b7fe640 1 -- 172.21.15.16:0/3621133348 learned_addr learned my addr 172.21.15.16:0/3621133348 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T16:05:32.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:32.356+0000 7fc85affd640 1 --2- 172.21.15.16:0/3621133348 >> v2:172.21.15.62:3300/0 conn(0x7fc85c071530 0x7fc85c15c580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:32.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:32.356+0000 7fc85bfff640 1 --2- 172.21.15.16:0/3621133348 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc85c06b6c0 0x7fc85c06bb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:32.358 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:32.356+0000 7fc85a7fc640 1 -- 172.21.15.16:0/3621133348 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1676173074 0 0) 0x7fc85c15d520 con 0x7fc85c070c70 2023-05-22T16:05:32.358 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:32.356+0000 7fc85a7fc640 1 -- 172.21.15.16:0/3621133348 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc844003530 con 0x7fc85c070c70 2023-05-22T16:05:32.359 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:32.356+0000 7fc85a7fc640 1 -- 172.21.15.16:0/3621133348 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1926475807 0 0) 0x7fc844003530 con 0x7fc85c070c70 2023-05-22T16:05:32.359 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:32.356+0000 7fc85a7fc640 1 -- 172.21.15.16:0/3621133348 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc85c06b6c0 msgr2=0x7fc85c06bb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:05:32.359 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:32.356+0000 7fc85a7fc640 1 --2- 172.21.15.16:0/3621133348 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc85c06b6c0 0x7fc85c06bb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:32.360 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:32.356+0000 7fc85a7fc640 1 -- 172.21.15.16:0/3621133348 >> v2:172.21.15.62:3300/0 conn(0x7fc85c071530 msgr2=0x7fc85c15c580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:05:32.360 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:32.356+0000 7fc85a7fc640 1 --2- 172.21.15.16:0/3621133348 >> v2:172.21.15.62:3300/0 conn(0x7fc85c071530 0x7fc85c15c580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:32.360 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:32.356+0000 7fc85a7fc640 1 -- 172.21.15.16:0/3621133348 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc85c0c8b10 con 0x7fc85c070c70 2023-05-22T16:05:32.360 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:32.356+0000 7fc85a7fc640 1 -- 172.21.15.16:0/3621133348 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7fc84c001a00 con 0x7fc85c070c70 2023-05-22T16:05:32.360 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:32.356+0000 7fc85affd640 1 --2- 172.21.15.16:0/3621133348 >> v2:172.21.15.62:3300/0 conn(0x7fc85c071530 0x7fc85c15c580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-05-22T16:05:32.360 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:32.356+0000 7fc85a7fc640 1 -- 172.21.15.16:0/3621133348 >> v1:172.21.15.16:6789/0 conn(0x7fc85c070c70 legacy=0x7fc85c06fba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:32.360 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:32.356+0000 7fc85a7fc640 1 --2- 172.21.15.16:0/3621133348 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc844003980 0x7fc844003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:32.361 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:32.356+0000 7fc85a7fc640 1 --2- 172.21.15.16:0/3621133348 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc85c06b6c0 0x7fc8440047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:32.361 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:32.356+0000 7fc85a7fc640 1 --2- 172.21.15.16:0/3621133348 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc844004d10 0x7fc844005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:32.361 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:32.356+0000 7fc85a7fc640 1 -- 172.21.15.16:0/3621133348 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc85c071980 con 0x7fc844004d10 2023-05-22T16:05:32.361 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:32.356+0000 7fc85a7fc640 1 -- 172.21.15.16:0/3621133348 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc85c06c040 con 0x7fc844003980 2023-05-22T16:05:32.361 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:32.356+0000 7fc85a7fc640 1 -- 172.21.15.16:0/3621133348 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc84400b990 con 0x7fc85c06b6c0 2023-05-22T16:05:32.361 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:32.356+0000 7fc85bfff640 1 --2- 172.21.15.16:0/3621133348 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc844004d10 0x7fc844005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:32.361 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:32.356+0000 7fc85b7fe640 1 --2- 172.21.15.16:0/3621133348 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc844003980 0x7fc844003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:32.362 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:32.356+0000 7fc85affd640 1 --2- 172.21.15.16:0/3621133348 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc85c06b6c0 0x7fc8440047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:32.362 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:32.356+0000 7fc85bfff640 1 -- 172.21.15.16:0/3621133348 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc85c06b6c0 msgr2=0x7fc8440047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:05:32.362 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:32.356+0000 7fc85bfff640 1 --2- 172.21.15.16:0/3621133348 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc85c06b6c0 0x7fc8440047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:32.362 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:32.356+0000 7fc85bfff640 1 -- 172.21.15.16:0/3621133348 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc844003980 msgr2=0x7fc844003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:05:32.362 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:32.356+0000 7fc85bfff640 1 --2- 172.21.15.16:0/3621133348 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc844003980 0x7fc844003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:32.362 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:32.356+0000 7fc85bfff640 1 -- 172.21.15.16:0/3621133348 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc844003530 con 0x7fc844004d10 2023-05-22T16:05:32.362 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:32.356+0000 7fc85b7fe640 1 --2- 172.21.15.16:0/3621133348 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc844003980 0x7fc844003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T16:05:32.363 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:32.356+0000 7fc85bfff640 1 --2- 172.21.15.16:0/3621133348 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc844004d10 0x7fc844005180 secure :-1 s=READY pgs=2477 cs=0 l=1 rev1=1 crypto rx=0x7fc854009ae0 tx=0x7fc85401ca50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f19ea693871d6b1d server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:32.365 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:32.356+0000 7fc85a7fc640 1 -- 172.21.15.16:0/3621133348 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8540a5640 con 0x7fc844004d10 2023-05-22T16:05:32.365 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:32.356+0000 7fc85a7fc640 1 -- 172.21.15.16:0/3621133348 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8540a57a0 con 0x7fc844004d10 2023-05-22T16:05:32.365 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:32.356+0000 7fc85a7fc640 1 -- 172.21.15.16:0/3621133348 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8540a5b50 con 0x7fc844004d10 2023-05-22T16:05:32.365 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:32.360+0000 7fc861b1f640 1 -- 172.21.15.16:0/3621133348 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc844004d10 msgr2=0x7fc844005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:32.365 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:32.360+0000 7fc861b1f640 1 --2- 172.21.15.16:0/3621133348 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc844004d10 0x7fc844005180 secure :-1 s=READY pgs=2477 cs=0 l=1 rev1=1 crypto rx=0x7fc854009ae0 tx=0x7fc85401ca50 comp rx=0 tx=0).stop 2023-05-22T16:05:32.365 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:32.360+0000 7fc85bfff640 1 -- 172.21.15.16:0/3621133348 reap_dead start 2023-05-22T16:05:32.365 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:32.360+0000 7fc861b1f640 1 -- 172.21.15.16:0/3621133348 shutdown_connections 2023-05-22T16:05:32.366 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:32.360+0000 7fc861b1f640 1 -- 172.21.15.16:0/3621133348 >> 172.21.15.16:0/3621133348 conn(0x7fc85c069720 msgr2=0x7fc85c069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:05:32.366 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:32.360+0000 7fc861b1f640 1 -- 172.21.15.16:0/3621133348 shutdown_connections 2023-05-22T16:05:32.366 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:32.360+0000 7fc861b1f640 1 -- 172.21.15.16:0/3621133348 wait complete. 2023-05-22T16:05:32.366 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:32.360+0000 7fc861b1f640 1 Processor -- start 2023-05-22T16:05:32.366 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:32.360+0000 7fc861b1f640 1 -- start start 2023-05-22T16:05:32.366 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:32.360+0000 7fc861b1f640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc844004d10 0x7fc85c1524e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:32.366 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:32.360+0000 7fc861b1f640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc85c159320 0x7fc85c159790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:32.367 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:32.360+0000 7fc861b1f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc85c069b20 0x7fc85c069f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:32.367 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:32.360+0000 7fc861b1f640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc85c0c7000 con 0x7fc85c069b20 2023-05-22T16:05:32.367 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:32.360+0000 7fc861b1f640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc85c159cd0 con 0x7fc844004d10 2023-05-22T16:05:32.367 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:32.360+0000 7fc861b1f640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc85c06a4d0 con 0x7fc85c159320 2023-05-22T16:05:32.367 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:32.360+0000 7fc85bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc85c069b20 0x7fc85c069f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:32.367 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:32.360+0000 7fc85bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc85c069b20 0x7fc85c069f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42930/0 (socket says 172.21.15.16:42930) 2023-05-22T16:05:32.367 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:32.360+0000 7fc85bfff640 1 -- 172.21.15.16:0/223978444 learned_addr learned my addr 172.21.15.16:0/223978444 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:05:32.368 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:32.360+0000 7fc85b7fe640 1 --2- 172.21.15.16:0/223978444 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc844004d10 0x7fc85c1524e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:32.368 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:32.360+0000 7fc85affd640 1 --2- 172.21.15.16:0/223978444 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc85c159320 0x7fc85c159790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:32.368 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:32.360+0000 7fc85bfff640 1 -- 172.21.15.16:0/223978444 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc85c159320 msgr2=0x7fc85c159790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:32.368 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:32.360+0000 7fc85bfff640 1 --2- 172.21.15.16:0/223978444 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc85c159320 0x7fc85c159790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:32.368 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:32.360+0000 7fc85bfff640 1 -- 172.21.15.16:0/223978444 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc844004d10 msgr2=0x7fc85c1524e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:32.368 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:32.360+0000 7fc85bfff640 1 --2- 172.21.15.16:0/223978444 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc844004d10 0x7fc85c1524e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:32.368 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:32.360+0000 7fc85bfff640 1 -- 172.21.15.16:0/223978444 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc85c15cf60 con 0x7fc85c069b20 2023-05-22T16:05:32.368 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:32.360+0000 7fc85bfff640 1 --2- 172.21.15.16:0/223978444 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc85c069b20 0x7fc85c069f90 secure :-1 s=READY pgs=2478 cs=0 l=1 rev1=1 crypto rx=0x7fc854009ae0 tx=0x7fc844005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:32.369 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:32.360+0000 7fc843fff640 1 -- 172.21.15.16:0/223978444 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8540a5640 con 0x7fc85c069b20 2023-05-22T16:05:32.369 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:32.360+0000 7fc843fff640 1 -- 172.21.15.16:0/223978444 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc854004220 con 0x7fc85c069b20 2023-05-22T16:05:32.369 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:32.360+0000 7fc861b1f640 1 -- 172.21.15.16:0/223978444 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc85c15cc80 con 0x7fc85c069b20 2023-05-22T16:05:32.369 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:32.360+0000 7fc843fff640 1 -- 172.21.15.16:0/223978444 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc854003620 con 0x7fc85c069b20 2023-05-22T16:05:32.369 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:32.360+0000 7fc861b1f640 1 -- 172.21.15.16:0/223978444 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc85c15d240 con 0x7fc85c069b20 2023-05-22T16:05:32.369 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:32.364+0000 7fc861b1f640 1 -- 172.21.15.16:0/223978444 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc85c0c6130 con 0x7fc85c069b20 2023-05-22T16:05:32.370 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:32.364+0000 7fc843fff640 1 -- 172.21.15.16:0/223978444 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 93) v1 ==== 84349+0+0 (secure 0 0 0) 0x7fc8540b4070 con 0x7fc85c069b20 2023-05-22T16:05:32.370 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:32.364+0000 7fc843fff640 1 --2- 172.21.15.16:0/223978444 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fc8480654f0 0x7fc8480679a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:32.370 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:32.364+0000 7fc843fff640 1 -- 172.21.15.16:0/223978444 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(196..196 src has 1..196) v4 ==== 6438+0+0 (secure 0 0 0) 0x7fc85411cac0 con 0x7fc85c069b20 2023-05-22T16:05:32.370 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:32.364+0000 7fc85b7fe640 1 -- 172.21.15.16:0/223978444 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fc8480654f0 msgr2=0x7fc8480679a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/65890 2023-05-22T16:05:32.371 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:32.364+0000 7fc85b7fe640 1 --2- 172.21.15.16:0/223978444 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fc8480654f0 0x7fc8480679a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-05-22T16:05:32.371 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:32.368+0000 7fc843fff640 1 -- 172.21.15.16:0/223978444 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fc8540ea4d0 con 0x7fc85c069b20 2023-05-22T16:05:32.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:32.520+0000 7fc861b1f640 1 -- 172.21.15.16:0/223978444 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7fc85c04dbe0 con 0x7fc85c069b20 2023-05-22T16:05:32.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:32.528+0000 7fc843fff640 1 -- 172.21.15.16:0/223978444 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v65)=0 v65) v1 ==== 138+0+0 (secure 0 0 0) 0x7fc85411ce60 con 0x7fc85c069b20 2023-05-22T16:05:32.531 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:32.528+0000 7fc861b1f640 1 -- 172.21.15.16:0/223978444 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fc8480654f0 msgr2=0x7fc8480679a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-05-22T16:05:32.531 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:32.528+0000 7fc861b1f640 1 --2- 172.21.15.16:0/223978444 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fc8480654f0 0x7fc8480679a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:32.531 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:32.528+0000 7fc861b1f640 1 -- 172.21.15.16:0/223978444 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc85c069b20 msgr2=0x7fc85c069f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:32.531 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:32.528+0000 7fc861b1f640 1 --2- 172.21.15.16:0/223978444 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc85c069b20 0x7fc85c069f90 secure :-1 s=READY pgs=2478 cs=0 l=1 rev1=1 crypto rx=0x7fc854009ae0 tx=0x7fc844005800 comp rx=0 tx=0).stop 2023-05-22T16:05:32.531 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:32.528+0000 7fc861b1f640 1 -- 172.21.15.16:0/223978444 shutdown_connections 2023-05-22T16:05:32.532 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:32.528+0000 7fc861b1f640 1 --2- 172.21.15.16:0/223978444 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fc8480654f0 0x7fc8480679a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:32.532 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:32.528+0000 7fc861b1f640 1 --2- 172.21.15.16:0/223978444 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc85c069b20 0x7fc85c069f90 unknown :-1 s=CLOSED pgs=2478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:32.532 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:32.528+0000 7fc861b1f640 1 --2- 172.21.15.16:0/223978444 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc85c159320 0x7fc85c159790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:32.532 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:32.528+0000 7fc861b1f640 1 --2- 172.21.15.16:0/223978444 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc844004d10 0x7fc85c1524e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:32.532 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:32.528+0000 7fc861b1f640 1 -- 172.21.15.16:0/223978444 >> 172.21.15.16:0/223978444 conn(0x7fc85c069720 msgr2=0x7fc85c06fba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:05:32.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:32.528+0000 7fc861b1f640 1 -- 172.21.15.16:0/223978444 shutdown_connections 2023-05-22T16:05:32.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:32.528+0000 7fc861b1f640 1 -- 172.21.15.16:0/223978444 wait complete. 2023-05-22T16:05:32.548 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2023-05-22T16:05:32.665 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:32.664+0000 7ff39b009640 1 Processor -- start 2023-05-22T16:05:32.665 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:32.664+0000 7ff39b009640 1 -- start start 2023-05-22T16:05:32.666 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:32.664+0000 7ff39b009640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff3940b2880 0x7ff3940cbfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:32.667 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:32.664+0000 7ff39b009640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7ff3940cc520 0x7ff3940c3db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:32.667 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:32.664+0000 7ff39b009640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3940c6ba0 con 0x7ff3940c67d0 2023-05-22T16:05:32.667 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:32.664+0000 7ff39b009640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff39415a300 con 0x7ff3940b2880 2023-05-22T16:05:32.667 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:32.664+0000 7ff39b009640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3940cfef0 con 0x7ff3940cc520 2023-05-22T16:05:32.667 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:32.664+0000 7ff393fff640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff3940b2880 0x7ff3940cbfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:32.668 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:32.664+0000 7ff393fff640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff3940b2880 0x7ff3940cbfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42940/0 (socket says 172.21.15.16:42940) 2023-05-22T16:05:32.668 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:32.664+0000 7ff393fff640 1 -- 172.21.15.16:0/942853954 learned_addr learned my addr 172.21.15.16:0/942853954 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:05:32.668 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:32.664+0000 7ff398da5640 1 --1- 172.21.15.16:0/942853954 >> v1:172.21.15.16:6789/0 conn(0x7ff3940c67d0 0x7ff3940cf730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:57156/0 (socket says 172.21.15.16:57156) 2023-05-22T16:05:32.668 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:32.664+0000 7ff3995a6640 1 --2- 172.21.15.16:0/942853954 >> v2:172.21.15.62:3300/0 conn(0x7ff3940cc520 0x7ff3940c3db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:32.668 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:32.664+0000 7ff393fff640 1 -- 172.21.15.16:0/942853954 >> v2:172.21.15.62:3300/0 conn(0x7ff3940cc520 msgr2=0x7ff3940c3db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:05:32.668 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:32.664+0000 7ff393fff640 1 --2- 172.21.15.16:0/942853954 >> v2:172.21.15.62:3300/0 conn(0x7ff3940cc520 0x7ff3940c3db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:32.668 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:32.664+0000 7ff393fff640 1 -- 172.21.15.16:0/942853954 >> v1:172.21.15.16:6789/0 conn(0x7ff3940c67d0 legacy=0x7ff3940cf730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:32.669 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:32.664+0000 7ff3937fe640 1 -- 172.21.15.16:0/942853954 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3728252718 0 0) 0x7ff3940c6ba0 con 0x7ff3940c67d0 2023-05-22T16:05:32.669 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:32.664+0000 7ff393fff640 1 -- 172.21.15.16:0/942853954 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3940d01d0 con 0x7ff3940b2880 2023-05-22T16:05:32.670 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:32.664+0000 7ff393fff640 1 --2- 172.21.15.16:0/942853954 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff3940b2880 0x7ff3940cbfe0 secure :-1 s=READY pgs=2480 cs=0 l=1 rev1=1 crypto rx=0x7ff3940b6360 tx=0x7ff38001c440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78486620d1322c2a server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:32.670 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:32.668+0000 7ff3937fe640 1 -- 172.21.15.16:0/942853954 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3800ab640 con 0x7ff3940b2880 2023-05-22T16:05:32.670 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:32.668+0000 7ff3937fe640 1 -- 172.21.15.16:0/942853954 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3800a7070 con 0x7ff3940b2880 2023-05-22T16:05:32.670 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:32.668+0000 7ff3937fe640 1 -- 172.21.15.16:0/942853954 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3800ab9c0 con 0x7ff3940b2880 2023-05-22T16:05:32.670 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:32.668+0000 7ff39b009640 1 -- 172.21.15.16:0/942853954 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff3940b2880 msgr2=0x7ff3940cbfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:32.671 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:32.668+0000 7ff39b009640 1 --2- 172.21.15.16:0/942853954 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff3940b2880 0x7ff3940cbfe0 secure :-1 s=READY pgs=2480 cs=0 l=1 rev1=1 crypto rx=0x7ff3940b6360 tx=0x7ff38001c440 comp rx=0 tx=0).stop 2023-05-22T16:05:32.671 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:32.668+0000 7ff39b009640 1 -- 172.21.15.16:0/942853954 shutdown_connections 2023-05-22T16:05:32.671 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:32.668+0000 7ff39b009640 1 --2- 172.21.15.16:0/942853954 >> v2:172.21.15.62:3300/0 conn(0x7ff3940cc520 0x7ff3940c3db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:32.671 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:32.668+0000 7ff39b009640 1 --2- 172.21.15.16:0/942853954 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff3940b2880 0x7ff3940cbfe0 unknown :-1 s=CLOSED pgs=2480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:32.672 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:32.668+0000 7ff39b009640 1 -- 172.21.15.16:0/942853954 >> 172.21.15.16:0/942853954 conn(0x7ff39406f5a0 msgr2=0x7ff39406f9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:05:32.672 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:32.668+0000 7ff39b009640 1 -- 172.21.15.16:0/942853954 shutdown_connections 2023-05-22T16:05:32.672 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:32.668+0000 7ff39b009640 1 -- 172.21.15.16:0/942853954 wait complete. 2023-05-22T16:05:32.676 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:32.668+0000 7ff39b009640 1 Processor -- start 2023-05-22T16:05:32.676 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:32.668+0000 7ff39b009640 1 -- start start 2023-05-22T16:05:32.677 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:32.668+0000 7ff39b009640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff3940b2880 0x7ff39406aa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:32.677 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:32.668+0000 7ff39b009640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff3940c67d0 0x7ff39414e600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:32.677 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:32.668+0000 7ff39b009640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff3940cc520 0x7ff39414eb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:32.677 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:32.668+0000 7ff39b009640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3940c7db0 con 0x7ff3940c67d0 2023-05-22T16:05:32.677 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:32.668+0000 7ff39b009640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3940b5250 con 0x7ff3940cc520 2023-05-22T16:05:32.677 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:32.668+0000 7ff39b009640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3940b5530 con 0x7ff3940b2880 2023-05-22T16:05:32.678 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:32.668+0000 7ff393fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff3940c67d0 0x7ff39414e600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:32.678 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:32.668+0000 7ff393fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff3940c67d0 0x7ff39414e600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42948/0 (socket says 172.21.15.16:42948) 2023-05-22T16:05:32.678 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:32.668+0000 7ff393fff640 1 -- 172.21.15.16:0/4020436958 learned_addr learned my addr 172.21.15.16:0/4020436958 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:05:32.678 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:32.668+0000 7ff398da5640 1 --2- 172.21.15.16:0/4020436958 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff3940b2880 0x7ff39406aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:32.678 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:32.668+0000 7ff3995a6640 1 --2- 172.21.15.16:0/4020436958 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff3940cc520 0x7ff39414eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:32.678 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:32.668+0000 7ff393fff640 1 -- 172.21.15.16:0/4020436958 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff3940b2880 msgr2=0x7ff39406aa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:32.678 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:32.668+0000 7ff393fff640 1 --2- 172.21.15.16:0/4020436958 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff3940b2880 0x7ff39406aa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:32.679 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:32.668+0000 7ff393fff640 1 -- 172.21.15.16:0/4020436958 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff3940cc520 msgr2=0x7ff39414eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:32.679 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:32.668+0000 7ff393fff640 1 --2- 172.21.15.16:0/4020436958 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff3940cc520 0x7ff39414eb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:32.679 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:32.668+0000 7ff393fff640 1 -- 172.21.15.16:0/4020436958 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3800094d0 con 0x7ff3940c67d0 2023-05-22T16:05:32.679 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:32.668+0000 7ff398da5640 1 --2- 172.21.15.16:0/4020436958 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff3940b2880 0x7ff39406aa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T16:05:32.680 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:32.668+0000 7ff393fff640 1 --2- 172.21.15.16:0/4020436958 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff3940c67d0 0x7ff39414e600 secure :-1 s=READY pgs=2481 cs=0 l=1 rev1=1 crypto rx=0x7ff380019050 tx=0x7ff38001ac80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:32.680 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:32.668+0000 7ff3917fa640 1 -- 172.21.15.16:0/4020436958 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3800be3f0 con 0x7ff3940c67d0 2023-05-22T16:05:32.680 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:32.668+0000 7ff3917fa640 1 -- 172.21.15.16:0/4020436958 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3800be550 con 0x7ff3940c67d0 2023-05-22T16:05:32.680 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:32.668+0000 7ff3917fa640 1 -- 172.21.15.16:0/4020436958 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3800be900 con 0x7ff3940c67d0 2023-05-22T16:05:32.680 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:32.668+0000 7ff39b009640 1 -- 172.21.15.16:0/4020436958 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3940d0f90 con 0x7ff3940c67d0 2023-05-22T16:05:32.681 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:32.668+0000 7ff39b009640 1 -- 172.21.15.16:0/4020436958 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3940d0a70 con 0x7ff3940c67d0 2023-05-22T16:05:32.681 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:32.672+0000 7ff39b009640 1 -- 172.21.15.16:0/4020436958 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3940cc8f0 con 0x7ff3940c67d0 2023-05-22T16:05:32.681 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:32.672+0000 7ff3917fa640 1 -- 172.21.15.16:0/4020436958 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 93) v1 ==== 84349+0+0 (secure 0 0 0) 0x7ff3800b43f0 con 0x7ff3940c67d0 2023-05-22T16:05:32.681 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:32.672+0000 7ff3917fa640 1 --2- 172.21.15.16:0/4020436958 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7ff3780654f0 0x7ff3780679a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:32.681 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:32.672+0000 7ff398da5640 1 -- 172.21.15.16:0/4020436958 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7ff3780654f0 msgr2=0x7ff3780679a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/65890 2023-05-22T16:05:32.681 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:32.672+0000 7ff3917fa640 1 -- 172.21.15.16:0/4020436958 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(196..196 src has 1..196) v4 ==== 6438+0+0 (secure 0 0 0) 0x7ff38011d310 con 0x7ff3940c67d0 2023-05-22T16:05:32.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:32.672+0000 7ff398da5640 1 --2- 172.21.15.16:0/4020436958 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7ff3780654f0 0x7ff3780679a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-05-22T16:05:32.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:32.676+0000 7ff3917fa640 1 -- 172.21.15.16:0/4020436958 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7ff3800eaba0 con 0x7ff3940c67d0 2023-05-22T16:05:32.726 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T16:05:32.719+0000 7ff926cc3180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-05-22T16:05:32.762 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T16:05:32.756+0000 7f460e0e4180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-05-22T16:05:32.845 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:32.844+0000 7ff39b009640 1 -- 172.21.15.16:0/4020436958 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7ff39404dbe0 con 0x7ff3940c67d0 2023-05-22T16:05:32.851 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:32.848+0000 7ff3917fa640 1 -- 172.21.15.16:0/4020436958 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v66)=0 v66) v1 ==== 133+0+0 (secure 0 0 0) 0x7ff3800ee1c0 con 0x7ff3940c67d0 2023-05-22T16:05:32.854 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:32.852+0000 7ff39b009640 1 -- 172.21.15.16:0/4020436958 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7ff3780654f0 msgr2=0x7ff3780679a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-05-22T16:05:32.854 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:32.852+0000 7ff39b009640 1 --2- 172.21.15.16:0/4020436958 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7ff3780654f0 0x7ff3780679a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:32.854 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:32.852+0000 7ff39b009640 1 -- 172.21.15.16:0/4020436958 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff3940c67d0 msgr2=0x7ff39414e600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:32.854 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:32.852+0000 7ff39b009640 1 --2- 172.21.15.16:0/4020436958 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff3940c67d0 0x7ff39414e600 secure :-1 s=READY pgs=2481 cs=0 l=1 rev1=1 crypto rx=0x7ff380019050 tx=0x7ff38001ac80 comp rx=0 tx=0).stop 2023-05-22T16:05:32.854 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:32.852+0000 7ff39b009640 1 -- 172.21.15.16:0/4020436958 shutdown_connections 2023-05-22T16:05:32.855 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:32.852+0000 7ff39b009640 1 --2- 172.21.15.16:0/4020436958 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff3940cc520 0x7ff39414eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:32.855 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:32.852+0000 7ff39b009640 1 --2- 172.21.15.16:0/4020436958 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7ff3780654f0 0x7ff3780679a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:32.855 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:32.852+0000 7ff39b009640 1 --2- 172.21.15.16:0/4020436958 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff3940c67d0 0x7ff39414e600 unknown :-1 s=CLOSED pgs=2481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:32.855 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:32.852+0000 7ff39b009640 1 --2- 172.21.15.16:0/4020436958 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff3940b2880 0x7ff39406aa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:32.855 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:32.852+0000 7ff39b009640 1 -- 172.21.15.16:0/4020436958 >> 172.21.15.16:0/4020436958 conn(0x7ff39406f5a0 msgr2=0x7ff39406b7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:05:32.855 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:32.852+0000 7ff39b009640 1 -- 172.21.15.16:0/4020436958 shutdown_connections 2023-05-22T16:05:32.856 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:32.852+0000 7ff39b009640 1 -- 172.21.15.16:0/4020436958 wait complete. 2023-05-22T16:05:32.860 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T16:05:32.855+0000 7ff926cc3180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-05-22T16:05:32.870 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T16:05:32.871 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1) 2023-05-22T16:05:32.880 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add --fs cephfs snap_test_dir1 1M 2023-05-22T16:05:32.899 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T16:05:32.896+0000 7f460e0e4180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-05-22T16:05:33.021 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:33.020+0000 7fb70a9f1640 1 Processor -- start 2023-05-22T16:05:33.021 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:33.020+0000 7fb70a9f1640 1 -- start start 2023-05-22T16:05:33.022 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:33.020+0000 7fb70a9f1640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb70415b9e0 0x7fb7040adca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:33.022 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:33.020+0000 7fb70a9f1640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb70415bdb0 0x7fb7040cad80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:33.022 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:33.020+0000 7fb70a9f1640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7040ce740 con 0x7fb70406fef0 2023-05-22T16:05:33.022 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:33.020+0000 7fb70a9f1640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7040ce460 con 0x7fb70415b9e0 2023-05-22T16:05:33.022 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:33.020+0000 7fb70a9f1640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7040caae0 con 0x7fb70415bdb0 2023-05-22T16:05:33.022 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:33.020+0000 7fb703fff640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb70415b9e0 0x7fb7040adca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:33.023 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:33.020+0000 7fb703fff640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb70415b9e0 0x7fb7040adca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42958/0 (socket says 172.21.15.16:42958) 2023-05-22T16:05:33.023 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:33.020+0000 7fb703fff640 1 -- 172.21.15.16:0/2770688510 learned_addr learned my addr 172.21.15.16:0/2770688510 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:05:33.023 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:33.020+0000 7fb703fff640 1 -- 172.21.15.16:0/2770688510 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb70415bdb0 msgr2=0x7fb7040cad80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:05:33.023 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:33.020+0000 7fb7037fe640 1 --2- 172.21.15.16:0/2770688510 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb70415bdb0 0x7fb7040cad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:33.024 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:33.020+0000 7fb703fff640 1 --2- 172.21.15.16:0/2770688510 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb70415bdb0 0x7fb7040cad80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:33.024 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:33.020+0000 7fb703fff640 1 -- 172.21.15.16:0/2770688510 >> v1:172.21.15.62:6789/0 conn(0x7fb70406fef0 legacy=0x7fb704069a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:05:33.024 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:33.020+0000 7fb703fff640 1 -- 172.21.15.16:0/2770688510 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7040cde10 con 0x7fb70415b9e0 2023-05-22T16:05:33.025 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:33.020+0000 7fb703fff640 1 --2- 172.21.15.16:0/2770688510 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb70415b9e0 0x7fb7040adca0 secure :-1 s=READY pgs=2482 cs=0 l=1 rev1=1 crypto rx=0x7fb6f4009870 tx=0x7fb6f401c470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=925dc233497cdf11 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:33.025 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:33.024+0000 7fb702ffd640 1 -- 172.21.15.16:0/2770688510 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6f40ab600 con 0x7fb70415b9e0 2023-05-22T16:05:33.026 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:33.024+0000 7fb702ffd640 1 -- 172.21.15.16:0/2770688510 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6f40a7050 con 0x7fb70415b9e0 2023-05-22T16:05:33.026 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:33.024+0000 7fb702ffd640 1 -- 172.21.15.16:0/2770688510 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6f40ab980 con 0x7fb70415b9e0 2023-05-22T16:05:33.026 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:33.024+0000 7fb70a9f1640 1 -- 172.21.15.16:0/2770688510 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb70415b9e0 msgr2=0x7fb7040adca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:33.026 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:33.024+0000 7fb70a9f1640 1 --2- 172.21.15.16:0/2770688510 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb70415b9e0 0x7fb7040adca0 secure :-1 s=READY pgs=2482 cs=0 l=1 rev1=1 crypto rx=0x7fb6f4009870 tx=0x7fb6f401c470 comp rx=0 tx=0).stop 2023-05-22T16:05:33.026 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:33.024+0000 7fb70a9f1640 1 -- 172.21.15.16:0/2770688510 shutdown_connections 2023-05-22T16:05:33.026 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:33.024+0000 7fb70a9f1640 1 --2- 172.21.15.16:0/2770688510 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb70415bdb0 0x7fb7040cad80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:33.026 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:33.024+0000 7fb70a9f1640 1 --2- 172.21.15.16:0/2770688510 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb70415b9e0 0x7fb7040adca0 unknown :-1 s=CLOSED pgs=2482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:33.027 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:33.024+0000 7fb70a9f1640 1 -- 172.21.15.16:0/2770688510 >> 172.21.15.16:0/2770688510 conn(0x7fb7040ce8d0 msgr2=0x7fb7040cecd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:05:33.027 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:33.024+0000 7fb70a9f1640 1 -- 172.21.15.16:0/2770688510 shutdown_connections 2023-05-22T16:05:33.027 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:33.024+0000 7fb70a9f1640 1 -- 172.21.15.16:0/2770688510 wait complete. 2023-05-22T16:05:33.028 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:33.024+0000 7fb70a9f1640 1 Processor -- start 2023-05-22T16:05:33.028 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:33.024+0000 7fb70a9f1640 1 -- start start 2023-05-22T16:05:33.029 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:33.024+0000 7fb70a9f1640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb70406fef0 0x7fb7040b40e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:33.029 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:33.024+0000 7fb70a9f1640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb70415b9e0 0x7fb7041f5aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:33.029 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:33.024+0000 7fb70a9f1640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb70415bdb0 0x7fb7040b1700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:33.029 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:33.024+0000 7fb70a9f1640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7040b4620 con 0x7fb70415bdb0 2023-05-22T16:05:33.029 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:33.024+0000 7fb70a9f1640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7041f5610 con 0x7fb70406fef0 2023-05-22T16:05:33.029 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:33.024+0000 7fb70a9f1640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7041f5fe0 con 0x7fb70415b9e0 2023-05-22T16:05:33.029 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:33.024+0000 7fb708f8e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb70415bdb0 0x7fb7040b1700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:33.030 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:33.024+0000 7fb708f8e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb70415bdb0 0x7fb7040b1700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42960/0 (socket says 172.21.15.16:42960) 2023-05-22T16:05:33.030 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:33.024+0000 7fb708f8e640 1 -- 172.21.15.16:0/1550809692 learned_addr learned my addr 172.21.15.16:0/1550809692 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:05:33.030 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:33.024+0000 7fb7037fe640 1 --2- 172.21.15.16:0/1550809692 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb70415b9e0 0x7fb7041f5aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:33.030 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:33.024+0000 7fb703fff640 1 --2- 172.21.15.16:0/1550809692 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb70406fef0 0x7fb7040b40e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:33.030 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:33.024+0000 7fb708f8e640 1 -- 172.21.15.16:0/1550809692 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb70415b9e0 msgr2=0x7fb7041f5aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:33.030 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:33.024+0000 7fb708f8e640 1 --2- 172.21.15.16:0/1550809692 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb70415b9e0 0x7fb7041f5aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:33.030 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:33.024+0000 7fb708f8e640 1 -- 172.21.15.16:0/1550809692 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb70406fef0 msgr2=0x7fb7040b40e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:33.031 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:33.024+0000 7fb708f8e640 1 --2- 172.21.15.16:0/1550809692 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb70406fef0 0x7fb7040b40e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:33.031 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:33.024+0000 7fb708f8e640 1 -- 172.21.15.16:0/1550809692 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6f40094d0 con 0x7fb70415bdb0 2023-05-22T16:05:33.031 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:33.024+0000 7fb708f8e640 1 --2- 172.21.15.16:0/1550809692 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb70415bdb0 0x7fb7040b1700 secure :-1 s=READY pgs=2483 cs=0 l=1 rev1=1 crypto rx=0x7fb6fc00b970 tx=0x7fb6fc00be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:33.031 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:33.024+0000 7fb700ff9640 1 -- 172.21.15.16:0/1550809692 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6fc00cb40 con 0x7fb70415bdb0 2023-05-22T16:05:33.031 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:33.024+0000 7fb700ff9640 1 -- 172.21.15.16:0/1550809692 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6fc00cca0 con 0x7fb70415bdb0 2023-05-22T16:05:33.031 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:33.024+0000 7fb70a9f1640 1 -- 172.21.15.16:0/1550809692 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb70406a520 con 0x7fb70415bdb0 2023-05-22T16:05:33.031 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:33.024+0000 7fb700ff9640 1 -- 172.21.15.16:0/1550809692 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6fc014680 con 0x7fb70415bdb0 2023-05-22T16:05:33.031 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:33.024+0000 7fb70a9f1640 1 -- 172.21.15.16:0/1550809692 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb704070780 con 0x7fb70415bdb0 2023-05-22T16:05:33.032 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:33.028+0000 7fb70a9f1640 1 -- 172.21.15.16:0/1550809692 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7040b1330 con 0x7fb70415bdb0 2023-05-22T16:05:33.032 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:33.028+0000 7fb700ff9640 1 -- 172.21.15.16:0/1550809692 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 93) v1 ==== 84349+0+0 (secure 0 0 0) 0x7fb6fc01d3b0 con 0x7fb70415bdb0 2023-05-22T16:05:33.033 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:33.028+0000 7fb700ff9640 1 --2- 172.21.15.16:0/1550809692 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fb6f00654f0 0x7fb6f00679a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:33.033 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:33.028+0000 7fb703fff640 1 -- 172.21.15.16:0/1550809692 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fb6f00654f0 msgr2=0x7fb6f00679a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/65890 2023-05-22T16:05:33.033 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:33.028+0000 7fb700ff9640 1 -- 172.21.15.16:0/1550809692 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(196..196 src has 1..196) v4 ==== 6438+0+0 (secure 0 0 0) 0x7fb6fc082b20 con 0x7fb70415bdb0 2023-05-22T16:05:33.033 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:33.028+0000 7fb703fff640 1 --2- 172.21.15.16:0/1550809692 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fb6f00654f0 0x7fb6f00679a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-05-22T16:05:33.034 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:33.032+0000 7fb700ff9640 1 -- 172.21.15.16:0/1550809692 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fb6fc050490 con 0x7fb70415bdb0 2023-05-22T16:05:33.200 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:33.196+0000 7fb70a9f1640 1 -- 172.21.15.16:0/1550809692 --> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "snap_test_dir1", "snap_schedule": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7fb70406a6b0 con 0x7fb6f00654f0 2023-05-22T16:05:33.232 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:33.232+0000 7fb703fff640 1 -- 172.21.15.16:0/1550809692 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fb6f00654f0 msgr2=0x7fb6f00679a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/65890 2023-05-22T16:05:33.233 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:33.232+0000 7fb703fff640 1 --2- 172.21.15.16:0/1550809692 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fb6f00654f0 0x7fb6f00679a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-05-22T16:05:33.302 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T16:05:33.295+0000 7ff926cc3180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-05-22T16:05:33.345 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T16:05:33.344+0000 7f460e0e4180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-05-22T16:05:33.355 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T16:05:33.351+0000 7ff926cc3180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-05-22T16:05:33.398 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T16:05:33.396+0000 7f460e0e4180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-05-22T16:05:33.404 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T16:05:33.399+0000 7ff926cc3180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-05-22T16:05:33.448 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T16:05:33.444+0000 7f460e0e4180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-05-22T16:05:33.483 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T16:05:33.475+0000 7ff926cc3180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-05-22T16:05:33.525 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T16:05:33.524+0000 7f460e0e4180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-05-22T16:05:33.533 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T16:05:33.527+0000 7ff926cc3180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-05-22T16:05:33.576 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T16:05:33.572+0000 7f460e0e4180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-05-22T16:05:33.609 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T16:05:33.603+0000 7ff926cc3180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-05-22T16:05:33.636 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:33.636+0000 7fb703fff640 1 -- 172.21.15.16:0/1550809692 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fb6f00654f0 msgr2=0x7fb6f00679a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/65890 2023-05-22T16:05:33.637 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:33.636+0000 7fb703fff640 1 --2- 172.21.15.16:0/1550809692 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fb6f00654f0 0x7fb6f00679a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-05-22T16:05:33.653 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T16:05:33.652+0000 7f460e0e4180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-05-22T16:05:33.661 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T16:05:33.655+0000 7ff926cc3180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-05-22T16:05:33.704 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T16:05:33.700+0000 7f460e0e4180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-05-22T16:05:33.793 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T16:05:33.787+0000 7ff926cc3180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-05-22T16:05:33.836 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T16:05:33.832+0000 7f460e0e4180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-05-22T16:05:33.845 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T16:05:33.839+0000 7ff926cc3180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-05-22T16:05:33.888 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T16:05:33.884+0000 7f460e0e4180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-05-22T16:05:34.069 INFO:tasks.ceph.mgr.x.smithi062.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2023-05-22T16:05:34.069 INFO:tasks.ceph.mgr.x.smithi062.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2023-05-22T16:05:34.070 INFO:tasks.ceph.mgr.x.smithi062.stderr: from numpy import show_config as show_numpy_config 2023-05-22T16:05:34.077 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T16:05:34.071+0000 7ff926cc3180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-05-22T16:05:34.112 INFO:tasks.ceph.mgr.y.smithi016.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2023-05-22T16:05:34.113 INFO:tasks.ceph.mgr.y.smithi016.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2023-05-22T16:05:34.113 INFO:tasks.ceph.mgr.y.smithi016.stderr: from numpy import show_config as show_numpy_config 2023-05-22T16:05:34.120 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T16:05:34.116+0000 7f460e0e4180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-05-22T16:05:34.201 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T16:05:34.195+0000 7ff926cc3180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-05-22T16:05:34.245 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T16:05:34.244+0000 7f460e0e4180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-05-22T16:05:34.315 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T16:05:34.311+0000 7ff926cc3180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-05-22T16:05:34.358 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T16:05:34.356+0000 7f460e0e4180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-05-22T16:05:34.378 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T16:05:34.375+0000 7ff926cc3180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-05-22T16:05:34.420 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T16:05:34.420+0000 7f460e0e4180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-05-22T16:05:34.440 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:34.440+0000 7fb703fff640 1 -- 172.21.15.16:0/1550809692 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fb6f00654f0 msgr2=0x7fb6f00679a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/65890 2023-05-22T16:05:34.441 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:34.440+0000 7fb703fff640 1 --2- 172.21.15.16:0/1550809692 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fb6f00654f0 0x7fb6f00679a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-05-22T16:05:34.504 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T16:05:34.499+0000 7ff926cc3180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-05-22T16:05:34.545 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T16:05:34.544+0000 7f460e0e4180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-05-22T16:05:34.997 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T16:05:34.991+0000 7ff926cc3180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-05-22T16:05:35.036 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T16:05:35.032+0000 7f460e0e4180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-05-22T16:05:35.073 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T16:05:35.067+0000 7ff926cc3180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-05-22T16:05:35.112 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T16:05:35.108+0000 7f460e0e4180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-05-22T16:05:35.211 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T16:05:35.207+0000 7ff926cc3180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-05-22T16:05:35.250 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T16:05:35.248+0000 7f460e0e4180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-05-22T16:05:35.288 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T16:05:35.283+0000 7ff926cc3180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-05-22T16:05:35.325 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T16:05:35.324+0000 7f460e0e4180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-05-22T16:05:35.535 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T16:05:35.531+0000 7ff926cc3180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-05-22T16:05:35.572 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T16:05:35.572+0000 7f460e0e4180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-05-22T16:05:35.590 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T16:05:35.583+0000 7ff926cc3180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-05-22T16:05:35.627 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T16:05:35.624+0000 7f460e0e4180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-05-22T16:05:35.662 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:35.660+0000 7fb700ff9640 1 -- 172.21.15.16:0/1550809692 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mgrmap(e 94) v1 ==== 83958+0+0 (secure 0 0 0) 0x7fb6fc050860 con 0x7fb70415bdb0 2023-05-22T16:05:35.662 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:35.660+0000 7fb700ff9640 1 -- 172.21.15.16:0/1550809692 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fb6f00654f0 msgr2=0x7fb6f00679a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-05-22T16:05:35.662 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:35.660+0000 7fb700ff9640 1 --2- 172.21.15.16:0/1550809692 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fb6f00654f0 0x7fb6f00679a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:35.805 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T16:05:35.804+0000 7f458d25b640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T16:05:35.805 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T16:05:35.804+0000 7f458d25b640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T16:05:35.805 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T16:05:35.804+0000 7f458d25b640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T16:05:35.805 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T16:05:35.804+0000 7f458d25b640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T16:05:35.805 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T16:05:35.804+0000 7f458d25b640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T16:05:35.805 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T16:05:35.804+0000 7f458f25f640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T16:05:35.806 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T16:05:35.804+0000 7f458f25f640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T16:05:35.806 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T16:05:35.804+0000 7f458f25f640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T16:05:35.806 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T16:05:35.804+0000 7f458f25f640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T16:05:35.806 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T16:05:35.804+0000 7f458f25f640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T16:05:36.699 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:36.696+0000 7fb700ff9640 1 -- 172.21.15.16:0/1550809692 <== mon.0 v2:172.21.15.16:3300/0 8 ==== mgrmap(e 95) v1 ==== 84085+0+0 (secure 0 0 0) 0x7fb6fc080d40 con 0x7fb70415bdb0 2023-05-22T16:05:36.699 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:36.696+0000 7fb700ff9640 1 --2- 172.21.15.16:0/1550809692 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fb6f00654f0 0x7fb6f0067920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:36.699 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:36.696+0000 7fb700ff9640 1 -- 172.21.15.16:0/1550809692 --> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "snap_test_dir1", "snap_schedule": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7fb70406a6b0 con 0x7fb6f00654f0 2023-05-22T16:05:36.703 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:36.700+0000 7fb703fff640 1 --2- 172.21.15.16:0/1550809692 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fb6f00654f0 0x7fb6f0067920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:36.707 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:36.704+0000 7fb703fff640 1 --2- 172.21.15.16:0/1550809692 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fb6f00654f0 0x7fb6f0067920 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fb6f40ad000 tx=0x7fb6f40a6000 comp rx=0 tx=0).ready entity=mgr.10974 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:36.842 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:36.840+0000 7fb700ff9640 1 -- 172.21.15.16:0/1550809692 <== mgr.10974 v2:172.21.15.16:6832/65890 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fb70406a6b0 con 0x7fb6f00654f0 2023-05-22T16:05:36.843 INFO:teuthology.orchestra.run.smithi016.stdout:Schedule set for path snap_test_dir1 2023-05-22T16:05:36.846 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:36.844+0000 7fb70a9f1640 1 -- 172.21.15.16:0/1550809692 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fb6f00654f0 msgr2=0x7fb6f0067920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:36.846 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:36.844+0000 7fb70a9f1640 1 --2- 172.21.15.16:0/1550809692 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fb6f00654f0 0x7fb6f0067920 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fb6f40ad000 tx=0x7fb6f40a6000 comp rx=0 tx=0).stop 2023-05-22T16:05:36.846 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:36.844+0000 7fb70a9f1640 1 -- 172.21.15.16:0/1550809692 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb70415bdb0 msgr2=0x7fb7040b1700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:36.847 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:36.844+0000 7fb70a9f1640 1 --2- 172.21.15.16:0/1550809692 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb70415bdb0 0x7fb7040b1700 secure :-1 s=READY pgs=2483 cs=0 l=1 rev1=1 crypto rx=0x7fb6fc00b970 tx=0x7fb6fc00be30 comp rx=0 tx=0).stop 2023-05-22T16:05:36.848 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:36.844+0000 7fb70a9f1640 1 -- 172.21.15.16:0/1550809692 shutdown_connections 2023-05-22T16:05:36.848 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:36.844+0000 7fb70a9f1640 1 --2- 172.21.15.16:0/1550809692 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fb6f00654f0 0x7fb6f0067920 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:36.848 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:36.844+0000 7fb70a9f1640 1 --2- 172.21.15.16:0/1550809692 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb70415bdb0 0x7fb7040b1700 unknown :-1 s=CLOSED pgs=2483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:36.849 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:36.844+0000 7fb70a9f1640 1 --2- 172.21.15.16:0/1550809692 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb70415b9e0 0x7fb7041f5aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:36.849 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:36.844+0000 7fb70a9f1640 1 --2- 172.21.15.16:0/1550809692 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb70406fef0 0x7fb7040b40e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:36.849 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:36.844+0000 7fb70a9f1640 1 -- 172.21.15.16:0/1550809692 >> 172.21.15.16:0/1550809692 conn(0x7fb7040ce8d0 msgr2=0x7fb70415cae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:05:36.849 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:36.844+0000 7fb70a9f1640 1 -- 172.21.15.16:0/1550809692 shutdown_connections 2023-05-22T16:05:36.849 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:36.844+0000 7fb70a9f1640 1 -- 172.21.15.16:0/1550809692 wait complete. 2023-05-22T16:05:36.871 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1 2023-05-22T16:05:36.873 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule retention add --fs cephfs snap_test_dir1 1M 2023-05-22T16:05:37.010 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.008+0000 7fe2b05a8640 1 Processor -- start 2023-05-22T16:05:37.010 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.008+0000 7fe2b05a8640 1 -- start start 2023-05-22T16:05:37.014 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.008+0000 7fe2b05a8640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe2a806c5f0 0x7fe2a80cc7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:37.014 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.008+0000 7fe2b05a8640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fe2a806a830 0x7fe2a815fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:37.014 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.008+0000 7fe2b05a8640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2a815bfc0 con 0x7fe2a806db10 2023-05-22T16:05:37.015 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.008+0000 7fe2b05a8640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2a815f790 con 0x7fe2a806a830 2023-05-22T16:05:37.015 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.008+0000 7fe2b05a8640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2a80ccce0 con 0x7fe2a806c5f0 2023-05-22T16:05:37.015 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.008+0000 7fe2ae344640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fe2a806db10 0x7fe2a806dee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:57172/0 (socket says 172.21.15.16:57172) 2023-05-22T16:05:37.015 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.008+0000 7fe2ae344640 1 -- 172.21.15.16:0/1891147731 learned_addr learned my addr 172.21.15.16:0/1891147731 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T16:05:37.015 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.008+0000 7fe2aeb45640 1 --2- 172.21.15.16:0/1891147731 >> v2:172.21.15.62:3300/0 conn(0x7fe2a806a830 0x7fe2a815fbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:37.015 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.008+0000 7fe2adb43640 1 --2- 172.21.15.16:0/1891147731 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe2a806c5f0 0x7fe2a80cc7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:37.016 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.008+0000 7fe2ad342640 1 -- 172.21.15.16:0/1891147731 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3583123375 0 0) 0x7fe2a4001120 con 0x7fe2a806db10 2023-05-22T16:05:37.016 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.008+0000 7fe2ad342640 1 -- 172.21.15.16:0/1891147731 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe294003530 con 0x7fe2a806db10 2023-05-22T16:05:37.016 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.008+0000 7fe2ad342640 1 -- 172.21.15.16:0/1891147731 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1176786936 0 0) 0x7fe294003530 con 0x7fe2a806db10 2023-05-22T16:05:37.016 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.008+0000 7fe2ad342640 1 -- 172.21.15.16:0/1891147731 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe2a806c5f0 msgr2=0x7fe2a80cc7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:05:37.016 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.008+0000 7fe2ad342640 1 --2- 172.21.15.16:0/1891147731 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe2a806c5f0 0x7fe2a80cc7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:37.016 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.008+0000 7fe2ad342640 1 -- 172.21.15.16:0/1891147731 >> v2:172.21.15.62:3300/0 conn(0x7fe2a806a830 msgr2=0x7fe2a815fbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:05:37.016 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.008+0000 7fe2ad342640 1 --2- 172.21.15.16:0/1891147731 >> v2:172.21.15.62:3300/0 conn(0x7fe2a806a830 0x7fe2a815fbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:37.017 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.008+0000 7fe2ad342640 1 -- 172.21.15.16:0/1891147731 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2a80cf620 con 0x7fe2a806db10 2023-05-22T16:05:37.017 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.008+0000 7fe2ad342640 1 -- 172.21.15.16:0/1891147731 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7fe2a4001810 con 0x7fe2a806db10 2023-05-22T16:05:37.017 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.008+0000 7fe2ad342640 1 -- 172.21.15.16:0/1891147731 >> v1:172.21.15.16:6789/0 conn(0x7fe2a806db10 legacy=0x7fe2a806dee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:37.017 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.008+0000 7fe2ad342640 1 --2- 172.21.15.16:0/1891147731 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe2a806c5f0 0x7fe294003980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:37.017 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.008+0000 7fe2ad342640 1 --2- 172.21.15.16:0/1891147731 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe294003f00 0x7fe2940047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:37.018 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.008+0000 7fe2ad342640 1 --2- 172.21.15.16:0/1891147731 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe294004d10 0x7fe294005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:37.018 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.008+0000 7fe2ad342640 1 -- 172.21.15.16:0/1891147731 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2a815f790 con 0x7fe294004d10 2023-05-22T16:05:37.018 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.008+0000 7fe2ad342640 1 -- 172.21.15.16:0/1891147731 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2a80ccce0 con 0x7fe294003f00 2023-05-22T16:05:37.018 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.008+0000 7fe2ad342640 1 -- 172.21.15.16:0/1891147731 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe29400b990 con 0x7fe2a806c5f0 2023-05-22T16:05:37.018 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.008+0000 7fe2aeb45640 1 --2- 172.21.15.16:0/1891147731 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe294004d10 0x7fe294005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:37.018 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.008+0000 7fe2ae344640 1 --2- 172.21.15.16:0/1891147731 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe2a806c5f0 0x7fe294003980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:37.019 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.008+0000 7fe2adb43640 1 --2- 172.21.15.16:0/1891147731 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe294003f00 0x7fe2940047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:37.019 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.008+0000 7fe2aeb45640 1 -- 172.21.15.16:0/1891147731 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe2a806c5f0 msgr2=0x7fe294003980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:05:37.019 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.008+0000 7fe2aeb45640 1 --2- 172.21.15.16:0/1891147731 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe2a806c5f0 0x7fe294003980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:37.020 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.008+0000 7fe2aeb45640 1 -- 172.21.15.16:0/1891147731 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe294003f00 msgr2=0x7fe2940047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:05:37.020 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.008+0000 7fe2aeb45640 1 --2- 172.21.15.16:0/1891147731 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe294003f00 0x7fe2940047d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:37.020 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.008+0000 7fe2aeb45640 1 -- 172.21.15.16:0/1891147731 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe294003530 con 0x7fe294004d10 2023-05-22T16:05:37.023 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.012+0000 7fe2aeb45640 1 --2- 172.21.15.16:0/1891147731 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe294004d10 0x7fe294005180 secure :-1 s=READY pgs=2500 cs=0 l=1 rev1=1 crypto rx=0x7fe2a0009fc0 tx=0x7fe2a001cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e1788f5fde43c19 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:37.023 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.012+0000 7fe2ad342640 1 -- 172.21.15.16:0/1891147731 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2a00a5cb0 con 0x7fe294004d10 2023-05-22T16:05:37.024 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.012+0000 7fe2ad342640 1 -- 172.21.15.16:0/1891147731 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2a000f040 con 0x7fe294004d10 2023-05-22T16:05:37.024 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.012+0000 7fe2ad342640 1 -- 172.21.15.16:0/1891147731 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2a00ad680 con 0x7fe294004d10 2023-05-22T16:05:37.024 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.012+0000 7fe2b05a8640 1 -- 172.21.15.16:0/1891147731 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe294004d10 msgr2=0x7fe294005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:37.024 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.012+0000 7fe2b05a8640 1 --2- 172.21.15.16:0/1891147731 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe294004d10 0x7fe294005180 secure :-1 s=READY pgs=2500 cs=0 l=1 rev1=1 crypto rx=0x7fe2a0009fc0 tx=0x7fe2a001cca0 comp rx=0 tx=0).stop 2023-05-22T16:05:37.025 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.012+0000 7fe2aeb45640 1 -- 172.21.15.16:0/1891147731 reap_dead start 2023-05-22T16:05:37.025 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.012+0000 7fe2b05a8640 1 -- 172.21.15.16:0/1891147731 shutdown_connections 2023-05-22T16:05:37.025 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.012+0000 7fe2b05a8640 1 -- 172.21.15.16:0/1891147731 >> 172.21.15.16:0/1891147731 conn(0x7fe2a806f4f0 msgr2=0x7fe2a806f8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:05:37.025 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.012+0000 7fe2b05a8640 1 -- 172.21.15.16:0/1891147731 shutdown_connections 2023-05-22T16:05:37.025 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.012+0000 7fe2b05a8640 1 -- 172.21.15.16:0/1891147731 wait complete. 2023-05-22T16:05:37.025 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.012+0000 7fe2b05a8640 1 Processor -- start 2023-05-22T16:05:37.025 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.016+0000 7fe2b05a8640 1 -- start start 2023-05-22T16:05:37.026 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.016+0000 7fe2b05a8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe294004d10 0x7fe2a80c68f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:37.026 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.016+0000 7fe2b05a8640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe2a80c6e30 0x7fe2a80ad070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:37.026 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.016+0000 7fe2b05a8640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe2a81599e0 0x7fe2a80ad5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:37.026 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.016+0000 7fe2b05a8640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2a80ac5a0 con 0x7fe294004d10 2023-05-22T16:05:37.027 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.016+0000 7fe2b05a8640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2a80ac880 con 0x7fe2a80c6e30 2023-05-22T16:05:37.027 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.016+0000 7fe2b05a8640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2a80acb60 con 0x7fe2a81599e0 2023-05-22T16:05:37.027 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.016+0000 7fe2ae344640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe294004d10 0x7fe2a80c68f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:37.027 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.016+0000 7fe2ae344640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe294004d10 0x7fe2a80c68f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43088/0 (socket says 172.21.15.16:43088) 2023-05-22T16:05:37.027 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.016+0000 7fe2ae344640 1 -- 172.21.15.16:0/192093236 learned_addr learned my addr 172.21.15.16:0/192093236 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:05:37.027 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.016+0000 7fe2adb43640 1 --2- 172.21.15.16:0/192093236 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe2a80c6e30 0x7fe2a80ad070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:37.027 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.016+0000 7fe2aeb45640 1 --2- 172.21.15.16:0/192093236 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe2a81599e0 0x7fe2a80ad5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:37.028 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.016+0000 7fe2ae344640 1 -- 172.21.15.16:0/192093236 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe2a81599e0 msgr2=0x7fe2a80ad5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:37.028 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.016+0000 7fe2ae344640 1 --2- 172.21.15.16:0/192093236 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe2a81599e0 0x7fe2a80ad5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:37.028 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.016+0000 7fe2ae344640 1 -- 172.21.15.16:0/192093236 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe2a80c6e30 msgr2=0x7fe2a80ad070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:37.028 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.016+0000 7fe2ae344640 1 --2- 172.21.15.16:0/192093236 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe2a80c6e30 0x7fe2a80ad070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:37.028 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.016+0000 7fe2ae344640 1 -- 172.21.15.16:0/192093236 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2a80cfbe0 con 0x7fe294004d10 2023-05-22T16:05:37.028 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.016+0000 7fe2aeb45640 1 --2- 172.21.15.16:0/192093236 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe2a81599e0 0x7fe2a80ad5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T16:05:37.028 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.016+0000 7fe2ae344640 1 --2- 172.21.15.16:0/192093236 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe294004d10 0x7fe2a80c68f0 secure :-1 s=READY pgs=2501 cs=0 l=1 rev1=1 crypto rx=0x7fe2a4007330 tx=0x7fe2a4007730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:37.029 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.016+0000 7fe29effd640 1 -- 172.21.15.16:0/192093236 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2a400e950 con 0x7fe294004d10 2023-05-22T16:05:37.029 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.016+0000 7fe29effd640 1 -- 172.21.15.16:0/192093236 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2a400eab0 con 0x7fe294004d10 2023-05-22T16:05:37.029 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.016+0000 7fe2b05a8640 1 -- 172.21.15.16:0/192093236 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2a80cf900 con 0x7fe294004d10 2023-05-22T16:05:37.029 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.016+0000 7fe29effd640 1 -- 172.21.15.16:0/192093236 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2a400ee60 con 0x7fe294004d10 2023-05-22T16:05:37.030 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.016+0000 7fe2b05a8640 1 -- 172.21.15.16:0/192093236 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2a80cfec0 con 0x7fe294004d10 2023-05-22T16:05:37.030 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.016+0000 7fe2b05a8640 1 -- 172.21.15.16:0/192093236 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2a80acda0 con 0x7fe294004d10 2023-05-22T16:05:37.030 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.016+0000 7fe29effd640 1 -- 172.21.15.16:0/192093236 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 95) v1 ==== 84085+0+0 (secure 0 0 0) 0x7fe2a401d460 con 0x7fe294004d10 2023-05-22T16:05:37.030 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.016+0000 7fe29effd640 1 --2- 172.21.15.16:0/192093236 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fe28c065220 0x7fe28c0676d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:37.030 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.016+0000 7fe29effd640 1 -- 172.21.15.16:0/192093236 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(197..197 src has 1..197) v4 ==== 6782+0+0 (secure 0 0 0) 0x7fe2a4083100 con 0x7fe294004d10 2023-05-22T16:05:37.031 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.016+0000 7fe2adb43640 1 --2- 172.21.15.16:0/192093236 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fe28c065220 0x7fe28c0676d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:37.031 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.020+0000 7fe2adb43640 1 --2- 172.21.15.16:0/192093236 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fe28c065220 0x7fe28c0676d0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fe2a80aeeb0 tx=0x7fe298008040 comp rx=0 tx=0).ready entity=mgr.10974 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:37.031 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.020+0000 7fe29effd640 1 -- 172.21.15.16:0/192093236 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fe2a4050e50 con 0x7fe294004d10 2023-05-22T16:05:37.197 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.196+0000 7fe2b05a8640 1 -- 172.21.15.16:0/192093236 --> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] -- mgr_command(tid 0: {"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "snap_test_dir1", "retention_spec_or_period": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7fe2a815c4e0 con 0x7fe28c065220 2023-05-22T16:05:37.289 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.288+0000 7fe29effd640 1 -- 172.21.15.16:0/192093236 <== mgr.10974 v2:172.21.15.16:6832/65890 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+39 (secure 0 0 0) 0x7fe2a815c4e0 con 0x7fe28c065220 2023-05-22T16:05:37.289 INFO:teuthology.orchestra.run.smithi016.stdout:Retention added to path snap_test_dir1 2023-05-22T16:05:37.292 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.288+0000 7fe2b05a8640 1 -- 172.21.15.16:0/192093236 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fe28c065220 msgr2=0x7fe28c0676d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:37.292 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.288+0000 7fe2b05a8640 1 --2- 172.21.15.16:0/192093236 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fe28c065220 0x7fe28c0676d0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fe2a80aeeb0 tx=0x7fe298008040 comp rx=0 tx=0).stop 2023-05-22T16:05:37.292 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.288+0000 7fe2b05a8640 1 -- 172.21.15.16:0/192093236 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe294004d10 msgr2=0x7fe2a80c68f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:37.292 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.288+0000 7fe2b05a8640 1 --2- 172.21.15.16:0/192093236 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe294004d10 0x7fe2a80c68f0 secure :-1 s=READY pgs=2501 cs=0 l=1 rev1=1 crypto rx=0x7fe2a4007330 tx=0x7fe2a4007730 comp rx=0 tx=0).stop 2023-05-22T16:05:37.293 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.288+0000 7fe2b05a8640 1 -- 172.21.15.16:0/192093236 shutdown_connections 2023-05-22T16:05:37.293 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.288+0000 7fe2b05a8640 1 --2- 172.21.15.16:0/192093236 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe2a81599e0 0x7fe2a80ad5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:37.293 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.288+0000 7fe2b05a8640 1 --2- 172.21.15.16:0/192093236 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe2a80c6e30 0x7fe2a80ad070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:37.293 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.288+0000 7fe2b05a8640 1 --2- 172.21.15.16:0/192093236 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fe28c065220 0x7fe28c0676d0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:37.293 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.288+0000 7fe2b05a8640 1 --2- 172.21.15.16:0/192093236 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe294004d10 0x7fe2a80c68f0 unknown :-1 s=CLOSED pgs=2501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:37.294 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.288+0000 7fe2b05a8640 1 -- 172.21.15.16:0/192093236 >> 172.21.15.16:0/192093236 conn(0x7fe2a806f4f0 msgr2=0x7fe2a815b600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:05:37.294 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.288+0000 7fe2b05a8640 1 -- 172.21.15.16:0/192093236 shutdown_connections 2023-05-22T16:05:37.294 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.288+0000 7fe2b05a8640 1 -- 172.21.15.16:0/192093236 wait complete. 2023-05-22T16:05:37.311 DEBUG:tasks.cephfs.test_snap_schedules:res=Retention added to path snap_test_dir1 2023-05-22T16:05:37.317 DEBUG:teuthology.orchestra.run.smithi016:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-05-22T16:05:37.320 DEBUG:teuthology.orchestra.run.smithi062:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-05-22T16:05:37.343 INFO:teuthology.orchestra.run.smithi016.stdout:[global] 2023-05-22T16:05:37.344 INFO:teuthology.orchestra.run.smithi016.stdout: chdir = "" 2023-05-22T16:05:37.344 INFO:teuthology.orchestra.run.smithi016.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-05-22T16:05:37.344 INFO:teuthology.orchestra.run.smithi016.stdout: auth supported = cephx 2023-05-22T16:05:37.344 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T16:05:37.344 INFO:teuthology.orchestra.run.smithi016.stdout: filestore xattr use omap = true 2023-05-22T16:05:37.345 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T16:05:37.345 INFO:teuthology.orchestra.run.smithi016.stdout: mon clock drift allowed = 1.000 2023-05-22T16:05:37.345 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T16:05:37.345 INFO:teuthology.orchestra.run.smithi016.stdout: osd crush chooseleaf type = 0 2023-05-22T16:05:37.346 INFO:teuthology.orchestra.run.smithi016.stdout: auth debug = true 2023-05-22T16:05:37.346 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T16:05:37.346 INFO:teuthology.orchestra.run.smithi016.stdout: ms die on old message = true 2023-05-22T16:05:37.346 INFO:teuthology.orchestra.run.smithi016.stdout: ms die on bug = true 2023-05-22T16:05:37.346 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T16:05:37.346 INFO:teuthology.orchestra.run.smithi016.stdout: mon max pg per osd = 10000 # >= luminous 2023-05-22T16:05:37.347 INFO:teuthology.orchestra.run.smithi016.stdout: mon pg warn max object skew = 0 2023-05-22T16:05:37.347 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T16:05:37.347 INFO:teuthology.orchestra.run.smithi016.stdout: # disable pg_autoscaler by default for new pools 2023-05-22T16:05:37.347 INFO:teuthology.orchestra.run.smithi016.stdout: osd_pool_default_pg_autoscale_mode = off 2023-05-22T16:05:37.347 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T16:05:37.347 INFO:teuthology.orchestra.run.smithi016.stdout: osd pool default size = 2 2023-05-22T16:05:37.348 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T16:05:37.348 INFO:teuthology.orchestra.run.smithi016.stdout: mon osd allow primary affinity = true 2023-05-22T16:05:37.348 INFO:teuthology.orchestra.run.smithi016.stdout: mon osd allow pg remap = true 2023-05-22T16:05:37.348 INFO:teuthology.orchestra.run.smithi016.stdout: mon warn on legacy crush tunables = false 2023-05-22T16:05:37.348 INFO:teuthology.orchestra.run.smithi016.stdout: mon warn on crush straw calc version zero = false 2023-05-22T16:05:37.348 INFO:teuthology.orchestra.run.smithi016.stdout: mon warn on no sortbitwise = false 2023-05-22T16:05:37.348 INFO:teuthology.orchestra.run.smithi016.stdout: mon warn on osd down out interval zero = false 2023-05-22T16:05:37.349 INFO:teuthology.orchestra.run.smithi016.stdout: mon warn on too few osds = false 2023-05-22T16:05:37.349 INFO:teuthology.orchestra.run.smithi016.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-05-22T16:05:37.349 INFO:teuthology.orchestra.run.smithi016.stdout: mon_warn_on_pool_no_redundancy = false 2023-05-22T16:05:37.349 INFO:teuthology.orchestra.run.smithi016.stdout: mon_allow_pool_size_one = true 2023-05-22T16:05:37.349 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T16:05:37.349 INFO:teuthology.orchestra.run.smithi016.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-05-22T16:05:37.349 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T16:05:37.350 INFO:teuthology.orchestra.run.smithi016.stdout: osd default data pool replay window = 5 2023-05-22T16:05:37.350 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T16:05:37.350 INFO:teuthology.orchestra.run.smithi016.stdout: mon allow pool delete = true 2023-05-22T16:05:37.350 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T16:05:37.350 INFO:teuthology.orchestra.run.smithi016.stdout: mon cluster log file level = debug 2023-05-22T16:05:37.350 INFO:teuthology.orchestra.run.smithi016.stdout: debug asserts on shutdown = true 2023-05-22T16:05:37.350 INFO:teuthology.orchestra.run.smithi016.stdout: mon health detail to clog = false 2023-05-22T16:05:37.350 INFO:teuthology.orchestra.run.smithi016.stdout: mon host = "172.21.15.16,172.21.15.62,[v2:172.21.15.62:3301,v1:172.21.15.62:6790]" 2023-05-22T16:05:37.351 INFO:teuthology.orchestra.run.smithi016.stdout: fsid = b2719834-7847-4d98-90a6-f65d57f85c49 2023-05-22T16:05:37.351 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T16:05:37.351 INFO:teuthology.orchestra.run.smithi016.stdout:[osd] 2023-05-22T16:05:37.351 INFO:teuthology.orchestra.run.smithi016.stdout: osd journal size = 100 2023-05-22T16:05:37.351 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T16:05:37.351 INFO:teuthology.orchestra.run.smithi016.stdout: osd scrub load threshold = 5.0 2023-05-22T16:05:37.351 INFO:teuthology.orchestra.run.smithi016.stdout: osd scrub max interval = 600 2023-05-22T16:05:37.351 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T16:05:37.352 INFO:teuthology.orchestra.run.smithi016.stdout: osd recover clone overlap = true 2023-05-22T16:05:37.352 INFO:teuthology.orchestra.run.smithi016.stdout: osd recovery max chunk = 1048576 2023-05-22T16:05:37.352 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T16:05:37.352 INFO:teuthology.orchestra.run.smithi016.stdout: osd debug shutdown = true 2023-05-22T16:05:37.352 INFO:teuthology.orchestra.run.smithi016.stdout: osd debug op order = true 2023-05-22T16:05:37.352 INFO:teuthology.orchestra.run.smithi016.stdout: osd debug verify stray on activate = true 2023-05-22T16:05:37.352 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T16:05:37.353 INFO:teuthology.orchestra.run.smithi016.stdout: osd open classes on start = true 2023-05-22T16:05:37.353 INFO:teuthology.orchestra.run.smithi016.stdout: osd debug pg log writeout = true 2023-05-22T16:05:37.353 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T16:05:37.353 INFO:teuthology.orchestra.run.smithi016.stdout: osd deep scrub update digest min age = 30 2023-05-22T16:05:37.353 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T16:05:37.353 INFO:teuthology.orchestra.run.smithi016.stdout: osd map max advance = 10 2023-05-22T16:05:37.353 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T16:05:37.353 INFO:teuthology.orchestra.run.smithi016.stdout: journal zero on create = true 2023-05-22T16:05:37.354 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T16:05:37.354 INFO:teuthology.orchestra.run.smithi016.stdout: filestore ondisk finisher threads = 3 2023-05-22T16:05:37.354 INFO:teuthology.orchestra.run.smithi016.stdout: filestore apply finisher threads = 3 2023-05-22T16:05:37.354 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T16:05:37.354 INFO:teuthology.orchestra.run.smithi016.stdout: bdev debug aio = true 2023-05-22T16:05:37.354 INFO:teuthology.orchestra.run.smithi016.stdout: osd debug misdirected ops = true 2023-05-22T16:05:37.354 INFO:teuthology.orchestra.run.smithi016.stdout: bluestore block size = 96636764160 2023-05-22T16:05:37.354 INFO:teuthology.orchestra.run.smithi016.stdout: bluestore fsck on mount = True 2023-05-22T16:05:37.355 INFO:teuthology.orchestra.run.smithi016.stdout: debug bluefs = 20 2023-05-22T16:05:37.355 INFO:teuthology.orchestra.run.smithi016.stdout: debug bluestore = 20 2023-05-22T16:05:37.355 INFO:teuthology.orchestra.run.smithi016.stdout: debug ms = 1 2023-05-22T16:05:37.355 INFO:teuthology.orchestra.run.smithi016.stdout: debug osd = 20 2023-05-22T16:05:37.355 INFO:teuthology.orchestra.run.smithi016.stdout: debug rocksdb = 10 2023-05-22T16:05:37.355 INFO:teuthology.orchestra.run.smithi016.stdout: mon osd backfillfull_ratio = 0.85 2023-05-22T16:05:37.355 INFO:teuthology.orchestra.run.smithi016.stdout: mon osd full ratio = 0.9 2023-05-22T16:05:37.355 INFO:teuthology.orchestra.run.smithi016.stdout: mon osd nearfull ratio = 0.8 2023-05-22T16:05:37.356 INFO:teuthology.orchestra.run.smithi016.stdout: osd failsafe full ratio = 0.95 2023-05-22T16:05:37.356 INFO:teuthology.orchestra.run.smithi016.stdout: osd objectstore = bluestore 2023-05-22T16:05:37.356 INFO:teuthology.orchestra.run.smithi016.stdout: osd op complaint time = 180 2023-05-22T16:05:37.356 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T16:05:37.357 INFO:teuthology.orchestra.run.smithi016.stdout:[mgr] 2023-05-22T16:05:37.357 INFO:teuthology.orchestra.run.smithi016.stdout: debug ms = 1 2023-05-22T16:05:37.357 INFO:teuthology.orchestra.run.smithi016.stdout: debug mgr = 20 2023-05-22T16:05:37.357 INFO:teuthology.orchestra.run.smithi016.stdout: debug mon = 20 2023-05-22T16:05:37.357 INFO:teuthology.orchestra.run.smithi016.stdout: debug auth = 20 2023-05-22T16:05:37.357 INFO:teuthology.orchestra.run.smithi016.stdout: mon reweight min pgs per osd = 4 2023-05-22T16:05:37.358 INFO:teuthology.orchestra.run.smithi016.stdout: mon reweight min bytes per osd = 10 2023-05-22T16:05:37.358 INFO:teuthology.orchestra.run.smithi016.stdout: mgr/telemetry/nag = false 2023-05-22T16:05:37.358 INFO:teuthology.orchestra.run.smithi016.stdout: debug client = 20 2023-05-22T16:05:37.358 INFO:teuthology.orchestra.run.smithi016.stdout: debug finisher = 20 2023-05-22T16:05:37.358 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T16:05:37.358 INFO:teuthology.orchestra.run.smithi016.stdout:[mon] 2023-05-22T16:05:37.359 INFO:teuthology.orchestra.run.smithi016.stdout: debug ms = 1 2023-05-22T16:05:37.359 INFO:teuthology.orchestra.run.smithi016.stdout: debug mon = 20 2023-05-22T16:05:37.359 INFO:teuthology.orchestra.run.smithi016.stdout: debug paxos = 20 2023-05-22T16:05:37.359 INFO:teuthology.orchestra.run.smithi016.stdout: debug auth = 20 2023-05-22T16:05:37.359 INFO:teuthology.orchestra.run.smithi016.stdout: mon data avail warn = 5 2023-05-22T16:05:37.359 INFO:teuthology.orchestra.run.smithi016.stdout: mon mgr mkfs grace = 240 2023-05-22T16:05:37.359 INFO:teuthology.orchestra.run.smithi016.stdout: mon reweight min pgs per osd = 4 2023-05-22T16:05:37.359 INFO:teuthology.orchestra.run.smithi016.stdout: mon osd reporter subtree level = osd 2023-05-22T16:05:37.360 INFO:teuthology.orchestra.run.smithi016.stdout: mon osd prime pg temp = true 2023-05-22T16:05:37.360 INFO:teuthology.orchestra.run.smithi016.stdout: mon reweight min bytes per osd = 10 2023-05-22T16:05:37.360 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T16:05:37.360 INFO:teuthology.orchestra.run.smithi016.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-05-22T16:05:37.360 INFO:teuthology.orchestra.run.smithi016.stdout: auth mon ticket ttl = 660 # 11m 2023-05-22T16:05:37.360 INFO:teuthology.orchestra.run.smithi016.stdout: auth service ticket ttl = 240 # 4m 2023-05-22T16:05:37.360 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T16:05:37.361 INFO:teuthology.orchestra.run.smithi016.stdout: # don't complain about insecure global_id in the test suite 2023-05-22T16:05:37.361 INFO:teuthology.orchestra.run.smithi016.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-05-22T16:05:37.361 INFO:teuthology.orchestra.run.smithi016.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-05-22T16:05:37.361 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T16:05:37.361 INFO:teuthology.orchestra.run.smithi016.stdout: # 1m isn't quite enough 2023-05-22T16:05:37.361 INFO:teuthology.orchestra.run.smithi016.stdout: mon_down_mkfs_grace = 2m 2023-05-22T16:05:37.361 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T16:05:37.361 INFO:teuthology.orchestra.run.smithi016.stdout: mon_warn_on_filestore_osds = false 2023-05-22T16:05:37.362 INFO:teuthology.orchestra.run.smithi016.stdout: mon op complaint time = 120 2023-05-22T16:05:37.362 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T16:05:37.363 INFO:teuthology.orchestra.run.smithi016.stdout:[client] 2023-05-22T16:05:37.363 INFO:teuthology.orchestra.run.smithi016.stdout: rgw cache enabled = true 2023-05-22T16:05:37.363 INFO:teuthology.orchestra.run.smithi016.stdout: rgw enable ops log = true 2023-05-22T16:05:37.364 INFO:teuthology.orchestra.run.smithi016.stdout: rgw enable usage log = true 2023-05-22T16:05:37.364 INFO:teuthology.orchestra.run.smithi016.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-05-22T16:05:37.364 INFO:teuthology.orchestra.run.smithi016.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-05-22T16:05:37.364 INFO:teuthology.orchestra.run.smithi016.stdout: client mount timeout = 600 2023-05-22T16:05:37.365 INFO:teuthology.orchestra.run.smithi016.stdout: debug client = 20 2023-05-22T16:05:37.365 INFO:teuthology.orchestra.run.smithi016.stdout: debug ms = 1 2023-05-22T16:05:37.365 INFO:teuthology.orchestra.run.smithi016.stdout: pid file = "" 2023-05-22T16:05:37.366 INFO:teuthology.orchestra.run.smithi016.stdout: rados mon op timeout = 900 2023-05-22T16:05:37.366 INFO:teuthology.orchestra.run.smithi016.stdout: rados osd op timeout = 900 2023-05-22T16:05:37.366 INFO:teuthology.orchestra.run.smithi016.stdout: client snapdir = .snap 2023-05-22T16:05:37.367 INFO:teuthology.orchestra.run.smithi016.stdout:[mon.a] 2023-05-22T16:05:37.367 INFO:teuthology.orchestra.run.smithi016.stdout:[mon.b] 2023-05-22T16:05:37.367 INFO:teuthology.orchestra.run.smithi016.stdout:[mon.c] 2023-05-22T16:05:37.367 INFO:teuthology.orchestra.run.smithi016.stdout:[mds.a] 2023-05-22T16:05:37.368 INFO:teuthology.orchestra.run.smithi016.stdout:[mds.c] 2023-05-22T16:05:37.368 INFO:teuthology.orchestra.run.smithi016.stdout:[mds.b] 2023-05-22T16:05:37.368 INFO:teuthology.orchestra.run.smithi016.stdout:[mds.d] 2023-05-22T16:05:37.368 INFO:teuthology.orchestra.run.smithi016.stdout:[mds] 2023-05-22T16:05:37.368 INFO:teuthology.orchestra.run.smithi016.stdout: debug mds = 20 2023-05-22T16:05:37.369 INFO:teuthology.orchestra.run.smithi016.stdout: debug mds balancer = 20 2023-05-22T16:05:37.369 INFO:teuthology.orchestra.run.smithi016.stdout: debug ms = 1 2023-05-22T16:05:37.369 INFO:teuthology.orchestra.run.smithi016.stdout: mds debug frag = True 2023-05-22T16:05:37.369 INFO:teuthology.orchestra.run.smithi016.stdout: mds debug scatterstat = True 2023-05-22T16:05:37.369 INFO:teuthology.orchestra.run.smithi016.stdout: mds op complaint time = 180 2023-05-22T16:05:37.369 INFO:teuthology.orchestra.run.smithi016.stdout: mds verify scatter = True 2023-05-22T16:05:37.369 INFO:teuthology.orchestra.run.smithi016.stdout: osd op complaint time = 180 2023-05-22T16:05:37.370 INFO:teuthology.orchestra.run.smithi016.stdout: rados mon op timeout = 900 2023-05-22T16:05:37.370 INFO:teuthology.orchestra.run.smithi016.stdout: rados osd op timeout = 900 2023-05-22T16:05:37.371 INFO:teuthology.orchestra.run.smithi062.stdout:[global] 2023-05-22T16:05:37.372 INFO:teuthology.orchestra.run.smithi062.stdout: chdir = "" 2023-05-22T16:05:37.372 INFO:teuthology.orchestra.run.smithi062.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-05-22T16:05:37.372 INFO:teuthology.orchestra.run.smithi062.stdout: auth supported = cephx 2023-05-22T16:05:37.372 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-05-22T16:05:37.373 INFO:teuthology.orchestra.run.smithi062.stdout: filestore xattr use omap = true 2023-05-22T16:05:37.373 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-05-22T16:05:37.373 INFO:teuthology.orchestra.run.smithi062.stdout: mon clock drift allowed = 1.000 2023-05-22T16:05:37.373 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-05-22T16:05:37.373 INFO:teuthology.orchestra.run.smithi062.stdout: osd crush chooseleaf type = 0 2023-05-22T16:05:37.373 INFO:teuthology.orchestra.run.smithi062.stdout: auth debug = true 2023-05-22T16:05:37.374 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-05-22T16:05:37.374 INFO:teuthology.orchestra.run.smithi062.stdout: ms die on old message = true 2023-05-22T16:05:37.374 INFO:teuthology.orchestra.run.smithi062.stdout: ms die on bug = true 2023-05-22T16:05:37.374 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-05-22T16:05:37.374 INFO:teuthology.orchestra.run.smithi062.stdout: mon max pg per osd = 10000 # >= luminous 2023-05-22T16:05:37.375 INFO:teuthology.orchestra.run.smithi062.stdout: mon pg warn max object skew = 0 2023-05-22T16:05:37.375 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-05-22T16:05:37.375 INFO:teuthology.orchestra.run.smithi062.stdout: # disable pg_autoscaler by default for new pools 2023-05-22T16:05:37.375 INFO:teuthology.orchestra.run.smithi062.stdout: osd_pool_default_pg_autoscale_mode = off 2023-05-22T16:05:37.376 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-05-22T16:05:37.376 INFO:teuthology.orchestra.run.smithi062.stdout: osd pool default size = 2 2023-05-22T16:05:37.376 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-05-22T16:05:37.377 INFO:teuthology.orchestra.run.smithi062.stdout: mon osd allow primary affinity = true 2023-05-22T16:05:37.377 INFO:teuthology.orchestra.run.smithi062.stdout: mon osd allow pg remap = true 2023-05-22T16:05:37.377 INFO:teuthology.orchestra.run.smithi062.stdout: mon warn on legacy crush tunables = false 2023-05-22T16:05:37.377 INFO:teuthology.orchestra.run.smithi062.stdout: mon warn on crush straw calc version zero = false 2023-05-22T16:05:37.378 INFO:teuthology.orchestra.run.smithi062.stdout: mon warn on no sortbitwise = false 2023-05-22T16:05:37.378 INFO:teuthology.orchestra.run.smithi062.stdout: mon warn on osd down out interval zero = false 2023-05-22T16:05:37.378 INFO:teuthology.orchestra.run.smithi062.stdout: mon warn on too few osds = false 2023-05-22T16:05:37.379 INFO:teuthology.orchestra.run.smithi062.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-05-22T16:05:37.379 INFO:teuthology.orchestra.run.smithi062.stdout: mon_warn_on_pool_no_redundancy = false 2023-05-22T16:05:37.379 INFO:teuthology.orchestra.run.smithi062.stdout: mon_allow_pool_size_one = true 2023-05-22T16:05:37.379 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-05-22T16:05:37.379 INFO:teuthology.orchestra.run.smithi062.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-05-22T16:05:37.380 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-05-22T16:05:37.380 INFO:teuthology.orchestra.run.smithi062.stdout: osd default data pool replay window = 5 2023-05-22T16:05:37.380 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-05-22T16:05:37.380 INFO:teuthology.orchestra.run.smithi062.stdout: mon allow pool delete = true 2023-05-22T16:05:37.380 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-05-22T16:05:37.381 INFO:teuthology.orchestra.run.smithi062.stdout: mon cluster log file level = debug 2023-05-22T16:05:37.381 INFO:teuthology.orchestra.run.smithi062.stdout: debug asserts on shutdown = true 2023-05-22T16:05:37.381 INFO:teuthology.orchestra.run.smithi062.stdout: mon health detail to clog = false 2023-05-22T16:05:37.382 INFO:teuthology.orchestra.run.smithi062.stdout: mon host = "172.21.15.16,172.21.15.62,[v2:172.21.15.62:3301,v1:172.21.15.62:6790]" 2023-05-22T16:05:37.382 INFO:teuthology.orchestra.run.smithi062.stdout: fsid = b2719834-7847-4d98-90a6-f65d57f85c49 2023-05-22T16:05:37.382 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-05-22T16:05:37.382 INFO:teuthology.orchestra.run.smithi062.stdout:[osd] 2023-05-22T16:05:37.382 INFO:teuthology.orchestra.run.smithi062.stdout: osd journal size = 100 2023-05-22T16:05:37.383 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-05-22T16:05:37.383 INFO:teuthology.orchestra.run.smithi062.stdout: osd scrub load threshold = 5.0 2023-05-22T16:05:37.383 INFO:teuthology.orchestra.run.smithi062.stdout: osd scrub max interval = 600 2023-05-22T16:05:37.383 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-05-22T16:05:37.383 INFO:teuthology.orchestra.run.smithi062.stdout: osd recover clone overlap = true 2023-05-22T16:05:37.383 INFO:teuthology.orchestra.run.smithi062.stdout: osd recovery max chunk = 1048576 2023-05-22T16:05:37.383 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-05-22T16:05:37.384 INFO:teuthology.orchestra.run.smithi062.stdout: osd debug shutdown = true 2023-05-22T16:05:37.384 INFO:teuthology.orchestra.run.smithi062.stdout: osd debug op order = true 2023-05-22T16:05:37.384 INFO:teuthology.orchestra.run.smithi062.stdout: osd debug verify stray on activate = true 2023-05-22T16:05:37.384 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-05-22T16:05:37.384 INFO:teuthology.orchestra.run.smithi062.stdout: osd open classes on start = true 2023-05-22T16:05:37.384 INFO:teuthology.orchestra.run.smithi062.stdout: osd debug pg log writeout = true 2023-05-22T16:05:37.384 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-05-22T16:05:37.385 INFO:teuthology.orchestra.run.smithi062.stdout: osd deep scrub update digest min age = 30 2023-05-22T16:05:37.385 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-05-22T16:05:37.385 INFO:teuthology.orchestra.run.smithi062.stdout: osd map max advance = 10 2023-05-22T16:05:37.385 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-05-22T16:05:37.385 INFO:teuthology.orchestra.run.smithi062.stdout: journal zero on create = true 2023-05-22T16:05:37.385 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-05-22T16:05:37.385 INFO:teuthology.orchestra.run.smithi062.stdout: filestore ondisk finisher threads = 3 2023-05-22T16:05:37.386 INFO:teuthology.orchestra.run.smithi062.stdout: filestore apply finisher threads = 3 2023-05-22T16:05:37.386 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-05-22T16:05:37.386 INFO:teuthology.orchestra.run.smithi062.stdout: bdev debug aio = true 2023-05-22T16:05:37.386 INFO:teuthology.orchestra.run.smithi062.stdout: osd debug misdirected ops = true 2023-05-22T16:05:37.386 INFO:teuthology.orchestra.run.smithi062.stdout: bluestore block size = 96636764160 2023-05-22T16:05:37.386 INFO:teuthology.orchestra.run.smithi062.stdout: bluestore fsck on mount = True 2023-05-22T16:05:37.386 INFO:teuthology.orchestra.run.smithi062.stdout: debug bluefs = 20 2023-05-22T16:05:37.386 INFO:teuthology.orchestra.run.smithi062.stdout: debug bluestore = 20 2023-05-22T16:05:37.387 INFO:teuthology.orchestra.run.smithi062.stdout: debug ms = 1 2023-05-22T16:05:37.387 INFO:teuthology.orchestra.run.smithi062.stdout: debug osd = 20 2023-05-22T16:05:37.387 INFO:teuthology.orchestra.run.smithi062.stdout: debug rocksdb = 10 2023-05-22T16:05:37.387 INFO:teuthology.orchestra.run.smithi062.stdout: mon osd backfillfull_ratio = 0.85 2023-05-22T16:05:37.387 INFO:teuthology.orchestra.run.smithi062.stdout: mon osd full ratio = 0.9 2023-05-22T16:05:37.387 INFO:teuthology.orchestra.run.smithi062.stdout: mon osd nearfull ratio = 0.8 2023-05-22T16:05:37.387 INFO:teuthology.orchestra.run.smithi062.stdout: osd failsafe full ratio = 0.95 2023-05-22T16:05:37.387 INFO:teuthology.orchestra.run.smithi062.stdout: osd objectstore = bluestore 2023-05-22T16:05:37.388 INFO:teuthology.orchestra.run.smithi062.stdout: osd op complaint time = 180 2023-05-22T16:05:37.388 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-05-22T16:05:37.388 INFO:teuthology.orchestra.run.smithi062.stdout:[mgr] 2023-05-22T16:05:37.388 INFO:teuthology.orchestra.run.smithi062.stdout: debug ms = 1 2023-05-22T16:05:37.388 INFO:teuthology.orchestra.run.smithi062.stdout: debug mgr = 20 2023-05-22T16:05:37.388 INFO:teuthology.orchestra.run.smithi062.stdout: debug mon = 20 2023-05-22T16:05:37.388 INFO:teuthology.orchestra.run.smithi062.stdout: debug auth = 20 2023-05-22T16:05:37.389 INFO:teuthology.orchestra.run.smithi062.stdout: mon reweight min pgs per osd = 4 2023-05-22T16:05:37.389 INFO:teuthology.orchestra.run.smithi062.stdout: mon reweight min bytes per osd = 10 2023-05-22T16:05:37.389 INFO:teuthology.orchestra.run.smithi062.stdout: mgr/telemetry/nag = false 2023-05-22T16:05:37.389 INFO:teuthology.orchestra.run.smithi062.stdout: debug client = 20 2023-05-22T16:05:37.389 INFO:teuthology.orchestra.run.smithi062.stdout: debug finisher = 20 2023-05-22T16:05:37.389 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-05-22T16:05:37.389 INFO:teuthology.orchestra.run.smithi062.stdout:[mon] 2023-05-22T16:05:37.389 INFO:teuthology.orchestra.run.smithi062.stdout: debug ms = 1 2023-05-22T16:05:37.390 INFO:teuthology.orchestra.run.smithi062.stdout: debug mon = 20 2023-05-22T16:05:37.390 INFO:teuthology.orchestra.run.smithi062.stdout: debug paxos = 20 2023-05-22T16:05:37.390 INFO:teuthology.orchestra.run.smithi062.stdout: debug auth = 20 2023-05-22T16:05:37.390 INFO:teuthology.orchestra.run.smithi062.stdout: mon data avail warn = 5 2023-05-22T16:05:37.390 INFO:teuthology.orchestra.run.smithi062.stdout: mon mgr mkfs grace = 240 2023-05-22T16:05:37.390 INFO:teuthology.orchestra.run.smithi062.stdout: mon reweight min pgs per osd = 4 2023-05-22T16:05:37.390 INFO:teuthology.orchestra.run.smithi062.stdout: mon osd reporter subtree level = osd 2023-05-22T16:05:37.390 INFO:teuthology.orchestra.run.smithi062.stdout: mon osd prime pg temp = true 2023-05-22T16:05:37.391 INFO:teuthology.orchestra.run.smithi062.stdout: mon reweight min bytes per osd = 10 2023-05-22T16:05:37.391 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-05-22T16:05:37.391 INFO:teuthology.orchestra.run.smithi062.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-05-22T16:05:37.391 INFO:teuthology.orchestra.run.smithi062.stdout: auth mon ticket ttl = 660 # 11m 2023-05-22T16:05:37.391 INFO:teuthology.orchestra.run.smithi062.stdout: auth service ticket ttl = 240 # 4m 2023-05-22T16:05:37.391 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-05-22T16:05:37.391 INFO:teuthology.orchestra.run.smithi062.stdout: # don't complain about insecure global_id in the test suite 2023-05-22T16:05:37.392 INFO:teuthology.orchestra.run.smithi062.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-05-22T16:05:37.392 INFO:teuthology.orchestra.run.smithi062.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-05-22T16:05:37.392 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-05-22T16:05:37.392 INFO:teuthology.orchestra.run.smithi062.stdout: # 1m isn't quite enough 2023-05-22T16:05:37.392 INFO:teuthology.orchestra.run.smithi062.stdout: mon_down_mkfs_grace = 2m 2023-05-22T16:05:37.392 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-05-22T16:05:37.392 INFO:teuthology.orchestra.run.smithi062.stdout: mon_warn_on_filestore_osds = false 2023-05-22T16:05:37.392 INFO:teuthology.orchestra.run.smithi062.stdout: mon op complaint time = 120 2023-05-22T16:05:37.393 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-05-22T16:05:37.393 INFO:teuthology.orchestra.run.smithi062.stdout:[client] 2023-05-22T16:05:37.393 INFO:teuthology.orchestra.run.smithi062.stdout: rgw cache enabled = true 2023-05-22T16:05:37.393 INFO:teuthology.orchestra.run.smithi062.stdout: rgw enable ops log = true 2023-05-22T16:05:37.393 INFO:teuthology.orchestra.run.smithi062.stdout: rgw enable usage log = true 2023-05-22T16:05:37.393 INFO:teuthology.orchestra.run.smithi062.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-05-22T16:05:37.393 INFO:teuthology.orchestra.run.smithi062.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-05-22T16:05:37.394 INFO:teuthology.orchestra.run.smithi062.stdout: client mount timeout = 600 2023-05-22T16:05:37.394 INFO:teuthology.orchestra.run.smithi062.stdout: debug client = 20 2023-05-22T16:05:37.394 INFO:teuthology.orchestra.run.smithi062.stdout: debug ms = 1 2023-05-22T16:05:37.394 INFO:teuthology.orchestra.run.smithi062.stdout: pid file = "" 2023-05-22T16:05:37.394 INFO:teuthology.orchestra.run.smithi062.stdout: rados mon op timeout = 900 2023-05-22T16:05:37.394 INFO:teuthology.orchestra.run.smithi062.stdout: rados osd op timeout = 900 2023-05-22T16:05:37.394 INFO:teuthology.orchestra.run.smithi062.stdout: client snapdir = .snap 2023-05-22T16:05:37.394 INFO:teuthology.orchestra.run.smithi062.stdout:[mon.a] 2023-05-22T16:05:37.395 INFO:teuthology.orchestra.run.smithi062.stdout:[mon.b] 2023-05-22T16:05:37.395 INFO:teuthology.orchestra.run.smithi062.stdout:[mon.c] 2023-05-22T16:05:37.395 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.a] 2023-05-22T16:05:37.395 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.c] 2023-05-22T16:05:37.395 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.b] 2023-05-22T16:05:37.395 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.d] 2023-05-22T16:05:37.395 INFO:teuthology.orchestra.run.smithi062.stdout:[mds] 2023-05-22T16:05:37.395 INFO:teuthology.orchestra.run.smithi062.stdout: debug mds = 20 2023-05-22T16:05:37.396 INFO:teuthology.orchestra.run.smithi062.stdout: debug mds balancer = 20 2023-05-22T16:05:37.396 INFO:teuthology.orchestra.run.smithi062.stdout: debug ms = 1 2023-05-22T16:05:37.396 INFO:teuthology.orchestra.run.smithi062.stdout: mds debug frag = True 2023-05-22T16:05:37.396 INFO:teuthology.orchestra.run.smithi062.stdout: mds debug scatterstat = True 2023-05-22T16:05:37.396 INFO:teuthology.orchestra.run.smithi062.stdout: mds op complaint time = 180 2023-05-22T16:05:37.396 INFO:teuthology.orchestra.run.smithi062.stdout: mds verify scatter = True 2023-05-22T16:05:37.396 INFO:teuthology.orchestra.run.smithi062.stdout: osd op complaint time = 180 2023-05-22T16:05:37.396 INFO:teuthology.orchestra.run.smithi062.stdout: rados mon op timeout = 900 2023-05-22T16:05:37.397 INFO:teuthology.orchestra.run.smithi062.stdout: rados osd op timeout = 900 2023-05-22T16:05:37.397 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-05-22T16:05:37.397 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2023-05-22T16:05:37.417 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-05-22T16:05:37.596 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.592+0000 7f4be3bc6640 1 Processor -- start 2023-05-22T16:05:37.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.596+0000 7f4be3bc6640 1 -- start start 2023-05-22T16:05:37.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.596+0000 7f4be3bc6640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f4bdc1603d0 0x7f4bdc0cda80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:37.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.596+0000 7f4be3bc6640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f4bdc06cdb0 0x7f4bdc0afb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:37.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.596+0000 7f4be3bc6640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4bdc06bee0 con 0x7f4bdc160000 2023-05-22T16:05:37.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.596+0000 7f4be3bc6640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4bdc06d520 con 0x7f4bdc1603d0 2023-05-22T16:05:37.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.596+0000 7f4be3bc6640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4bdc0c9fd0 con 0x7f4bdc06cdb0 2023-05-22T16:05:37.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.596+0000 7f4be1161640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f4bdc1603d0 0x7f4bdc0cda80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:37.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.596+0000 7f4be1962640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f4bdc160000 0x7f4bdc06b7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:57174/0 (socket says 172.21.15.16:57174) 2023-05-22T16:05:37.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.596+0000 7f4be1161640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f4bdc1603d0 0x7f4bdc0cda80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43090/0 (socket says 172.21.15.16:43090) 2023-05-22T16:05:37.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.596+0000 7f4be1962640 1 -- 172.21.15.16:0/2879628652 learned_addr learned my addr 172.21.15.16:0/2879628652 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T16:05:37.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.596+0000 7f4be2163640 1 --2- 172.21.15.16:0/2879628652 >> v2:172.21.15.62:3300/0 conn(0x7f4bdc06cdb0 0x7f4bdc0afb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:37.600 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.596+0000 7f4be1161640 1 -- 172.21.15.16:0/2879628652 >> v2:172.21.15.62:3300/0 conn(0x7f4bdc06cdb0 msgr2=0x7f4bdc0afb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:05:37.600 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.596+0000 7f4be1161640 1 --2- 172.21.15.16:0/2879628652 >> v2:172.21.15.62:3300/0 conn(0x7f4bdc06cdb0 0x7f4bdc0afb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:37.601 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.596+0000 7f4be1161640 1 -- 172.21.15.16:0/2879628652 >> v1:172.21.15.16:6789/0 conn(0x7f4bdc160000 legacy=0x7f4bdc06b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:37.601 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.596+0000 7f4be1161640 1 -- 172.21.15.16:0/2879628652 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bdc163cb0 con 0x7f4bdc1603d0 2023-05-22T16:05:37.601 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.596+0000 7f4be1161640 1 --2- 172.21.15.16:0/2879628652 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4bdc1603d0 0x7f4bdc0cda80 secure :-1 s=READY pgs=2503 cs=0 l=1 rev1=1 crypto rx=0x7f4bdc0b0c10 tx=0x7f4bd001c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5cc595aa6cb4e88b server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:37.601 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.596+0000 7f4be0960640 1 -- 172.21.15.16:0/2879628652 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bd00ab600 con 0x7f4bdc1603d0 2023-05-22T16:05:37.601 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.600+0000 7f4be0960640 1 -- 172.21.15.16:0/2879628652 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bd00a7050 con 0x7f4bdc1603d0 2023-05-22T16:05:37.601 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.600+0000 7f4be0960640 1 -- 172.21.15.16:0/2879628652 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bd00ab980 con 0x7f4bdc1603d0 2023-05-22T16:05:37.602 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.600+0000 7f4be3bc6640 1 -- 172.21.15.16:0/2879628652 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4bdc1603d0 msgr2=0x7f4bdc0cda80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:37.603 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.600+0000 7f4be3bc6640 1 --2- 172.21.15.16:0/2879628652 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4bdc1603d0 0x7f4bdc0cda80 secure :-1 s=READY pgs=2503 cs=0 l=1 rev1=1 crypto rx=0x7f4bdc0b0c10 tx=0x7f4bd001c460 comp rx=0 tx=0).stop 2023-05-22T16:05:37.603 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.600+0000 7f4be3bc6640 1 -- 172.21.15.16:0/2879628652 shutdown_connections 2023-05-22T16:05:37.603 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.600+0000 7f4be3bc6640 1 --2- 172.21.15.16:0/2879628652 >> v2:172.21.15.62:3300/0 conn(0x7f4bdc06cdb0 0x7f4bdc0afb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:37.603 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.600+0000 7f4be3bc6640 1 --2- 172.21.15.16:0/2879628652 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4bdc1603d0 0x7f4bdc0cda80 unknown :-1 s=CLOSED pgs=2503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:37.604 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.600+0000 7f4be3bc6640 1 -- 172.21.15.16:0/2879628652 >> 172.21.15.16:0/2879628652 conn(0x7f4bdc0cab50 msgr2=0x7f4bdc0caf50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:05:37.604 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.600+0000 7f4be3bc6640 1 -- 172.21.15.16:0/2879628652 shutdown_connections 2023-05-22T16:05:37.604 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.600+0000 7f4be3bc6640 1 -- 172.21.15.16:0/2879628652 wait complete. 2023-05-22T16:05:37.604 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.600+0000 7f4be3bc6640 1 Processor -- start 2023-05-22T16:05:37.604 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.600+0000 7f4be3bc6640 1 -- start start 2023-05-22T16:05:37.606 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.600+0000 7f4be3bc6640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4bdc06cdb0 0x7f4bdc15f440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:37.606 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.600+0000 7f4be3bc6640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4bdc160000 0x7f4bdc151c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:37.606 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.600+0000 7f4be3bc6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4bdc1521d0 0x7f4bdc15cb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:37.606 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.600+0000 7f4be3bc6640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bdc152670 con 0x7f4bdc1521d0 2023-05-22T16:05:37.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.600+0000 7f4be3bc6640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bdc152ef0 con 0x7f4bdc06cdb0 2023-05-22T16:05:37.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.600+0000 7f4be3bc6640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4bdc153140 con 0x7f4bdc160000 2023-05-22T16:05:37.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.600+0000 7f4be2163640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4bdc1521d0 0x7f4bdc15cb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:37.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.600+0000 7f4be2163640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4bdc1521d0 0x7f4bdc15cb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43100/0 (socket says 172.21.15.16:43100) 2023-05-22T16:05:37.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.600+0000 7f4be2163640 1 -- 172.21.15.16:0/707049011 learned_addr learned my addr 172.21.15.16:0/707049011 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:05:37.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.600+0000 7f4be1161640 1 --2- 172.21.15.16:0/707049011 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4bdc160000 0x7f4bdc151c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:37.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.600+0000 7f4be1962640 1 --2- 172.21.15.16:0/707049011 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4bdc06cdb0 0x7f4bdc15f440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:37.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.600+0000 7f4be2163640 1 -- 172.21.15.16:0/707049011 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4bdc160000 msgr2=0x7f4bdc151c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:37.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.600+0000 7f4be2163640 1 --2- 172.21.15.16:0/707049011 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4bdc160000 0x7f4bdc151c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:37.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.600+0000 7f4be2163640 1 -- 172.21.15.16:0/707049011 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4bdc06cdb0 msgr2=0x7f4bdc15f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:37.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.600+0000 7f4be2163640 1 --2- 172.21.15.16:0/707049011 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4bdc06cdb0 0x7f4bdc15f440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:37.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.600+0000 7f4be2163640 1 -- 172.21.15.16:0/707049011 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bd0009420 con 0x7f4bdc1521d0 2023-05-22T16:05:37.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.600+0000 7f4be2163640 1 --2- 172.21.15.16:0/707049011 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4bdc1521d0 0x7f4bdc15cb20 secure :-1 s=READY pgs=2504 cs=0 l=1 rev1=1 crypto rx=0x7f4bcc00d800 tx=0x7f4bcc00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:37.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.600+0000 7f4bda7fc640 1 -- 172.21.15.16:0/707049011 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bcc00b6c0 con 0x7f4bdc1521d0 2023-05-22T16:05:37.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.600+0000 7f4bda7fc640 1 -- 172.21.15.16:0/707049011 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bcc00b820 con 0x7f4bdc1521d0 2023-05-22T16:05:37.609 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.600+0000 7f4be3bc6640 1 -- 172.21.15.16:0/707049011 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4bdc06d1b0 con 0x7f4bdc1521d0 2023-05-22T16:05:37.609 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.600+0000 7f4bda7fc640 1 -- 172.21.15.16:0/707049011 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bcc00bbd0 con 0x7f4bdc1521d0 2023-05-22T16:05:37.609 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.600+0000 7f4be3bc6640 1 -- 172.21.15.16:0/707049011 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4bdc0ca830 con 0x7f4bdc1521d0 2023-05-22T16:05:37.609 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.604+0000 7f4be3bc6640 1 -- 172.21.15.16:0/707049011 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4bdc0c0cd0 con 0x7f4bdc1521d0 2023-05-22T16:05:37.609 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.604+0000 7f4bda7fc640 1 -- 172.21.15.16:0/707049011 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 95) v1 ==== 84085+0+0 (secure 0 0 0) 0x7f4bcc00bd30 con 0x7f4bdc1521d0 2023-05-22T16:05:37.609 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.604+0000 7f4bda7fc640 1 --2- 172.21.15.16:0/707049011 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f4bc8065220 0x7f4bc80676d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:37.609 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.604+0000 7f4be1962640 1 --2- 172.21.15.16:0/707049011 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f4bc8065220 0x7f4bc80676d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:37.609 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.604+0000 7f4bda7fc640 1 -- 172.21.15.16:0/707049011 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(197..197 src has 1..197) v4 ==== 6782+0+0 (secure 0 0 0) 0x7f4bcc082d20 con 0x7f4bdc1521d0 2023-05-22T16:05:37.610 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.604+0000 7f4be1962640 1 --2- 172.21.15.16:0/707049011 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f4bc8065220 0x7f4bc80676d0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f4bd4005c10 tx=0x7f4bd4009040 comp rx=0 tx=0).ready entity=mgr.10974 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:37.610 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.608+0000 7f4bda7fc640 1 -- 172.21.15.16:0/707049011 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f4bcc050a10 con 0x7f4bdc1521d0 2023-05-22T16:05:37.704 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.700+0000 7f4bda7fc640 1 -- 172.21.15.16:0/707049011 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mgrmap(e 96) v1 ==== 84406+0+0 (secure 0 0 0) 0x7f4bcc050de0 con 0x7f4bdc1521d0 2023-05-22T16:05:37.768 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.768+0000 7f4be3bc6640 1 -- 172.21.15.16:0/707049011 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4bdc071a90 con 0x7f4bdc1521d0 2023-05-22T16:05:37.771 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.768+0000 7f4bda7fc640 1 -- 172.21.15.16:0/707049011 <== mon.0 v2:172.21.15.16:3300/0 8 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v197) v1 ==== 74+0+18629 (secure 0 0 0) 0x7f4bcc054440 con 0x7f4bdc1521d0 2023-05-22T16:05:37.772 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T16:05:37.772 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":197,"fsid":"b2719834-7847-4d98-90a6-f65d57f85c49","created":"2023-05-22T15:26:25.259314+0000","modified":"2023-05-22T16:05:35.647639+0000","last_up_change":"2023-05-22T15:26:36.448524+0000","last_in_change":"2023-05-22T15:26:27.675668+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":25,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-05-22T15:26:38.019781+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-05-22T15:26:43.570514+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_metadata","create_time":"2023-05-22T16:05:20.485167+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"195","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":25,"pool_name":"cephfs_data","create_time":"2023-05-22T16:05:20.900001+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"196","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9e7ff854-b25f-4cbb-a5c2-1799d9c53d2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":37021},{"type":"v1","addr":"172.21.15.16:6825","nonce":37021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":37021},{"type":"v1","addr":"172.21.15.16:6827","nonce":37021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":37021},{"type":"v1","addr":"172.21.15.16:6831","nonce":37021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":37021},{"type":"v1","addr":"172.21.15.16:6829","nonce":37021}]},"public_addr":"172.21.15.16:6825/37021","cluster_addr":"172.21.15.16:6827/37021","heartbeat_back_addr":"172.21.15.16:6831/37021","heartbeat_front_addr":"172.21.15.16:6829/37021","state":["exists","up"]},{"osd":1,"uuid":"4702fbc9-a927-4565-aca5-41dc69cc8264","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":37008},{"type":"v1","addr":"172.21.15.16:6801","nonce":37008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":37008},{"type":"v1","addr":"172.21.15.16:6803","nonce":37008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":37008},{"type":"v1","addr":"172.21.15.16:6807","nonce":37008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":37008},{"type":"v1","addr":"172.21.15.16:6805","nonce":37008}]},"public_addr":"172.21.15.16:6801/37008","cluster_addr":"172.21.15.16:6803/37008","heartbeat_back_addr":"172.21.15.16:6807/37008","heartbeat_front_addr":"172.21.15.16:6805/37008","state":["exists","up"]},{"osd":2,"uuid":"1e7bba46-4eba-47ef-bb81-5e2d371638ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":37009},{"type":"v1","addr":"172.21.15.16:6809","nonce":37009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":37009},{"type":"v1","addr":"172.21.15.16:6811","nonce":37009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":37009},{"type":"v1","addr":"172.21.15.16:6815","nonce":37009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":37009},{"type":"v1","addr":"172.21.15.16:6813","nonce":37009}]},"public_addr":"172.21.15.16:6809/37009","cluster_addr":"172.21.15.16:6811/37009","heartbeat_back_addr":"172.21.15.16:6815/37009","heartbeat_front_addr":"172.21.15.16:6813/37009","state":["exists","up"]},{"osd":3,"uuid":"383cacc0-5a88-4904-990e-498ca59ef48a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":37020},{"type":"v1","addr":"172.21.15.16:6817","nonce":37020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":37020},{"type":"v1","addr":"172.21.15.16:6819","nonce":37020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":37020},{"type":"v1","addr":"172.21.15.16:6823","nonce":37020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":37020},{"type":"v1","addr":"172.21.15.16:6821","nonce":37020}]},"public_addr":"172.21.15.16:6817/37020","cluster_addr":"172.21.15.16:6819/37020","heartbeat_back_addr":"172.21.15.16:6823/37020","heartbeat_front_addr":"172.21.15.16:6821/37020","state":["exists","up"]},{"osd":4,"uuid":"970500ef-95b7-42c1-b080-eed4fae7013d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":37353},{"type":"v1","addr":"172.21.15.62:6817","nonce":37353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6819","nonce":37353},{"type":"v1","addr":"172.21.15.62:6822","nonce":37353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37353},{"type":"v1","addr":"172.21.15.62:6831","nonce":37353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37353},{"type":"v1","addr":"172.21.15.62:6826","nonce":37353}]},"public_addr":"172.21.15.62:6817/37353","cluster_addr":"172.21.15.62:6822/37353","heartbeat_back_addr":"172.21.15.62:6831/37353","heartbeat_front_addr":"172.21.15.62:6826/37353","state":["exists","up"]},{"osd":5,"uuid":"022d67db-d4d8-445b-b1b9-8b446a00f5c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":37352},{"type":"v1","addr":"172.21.15.62:6801","nonce":37352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":37352},{"type":"v1","addr":"172.21.15.62:6803","nonce":37352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":37352},{"type":"v1","addr":"172.21.15.62:6807","nonce":37352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":37352},{"type":"v1","addr":"172.21.15.62:6805","nonce":37352}]},"public_addr":"172.21.15.62:6801/37352","cluster_addr":"172.21.15.62:6803/37352","heartbeat_back_addr":"172.21.15.62:6807/37352","heartbeat_front_addr":"172.21.15.62:6805/37352","state":["exists","up"]},{"osd":6,"uuid":"42bd372b-6688-4a1b-8407-e16e8c266e33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":37354},{"type":"v1","addr":"172.21.15.62:6820","nonce":37354}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":37354},{"type":"v1","addr":"172.21.15.62:6823","nonce":37354}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6829","nonce":37354},{"type":"v1","addr":"172.21.15.62:6830","nonce":37354}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6825","nonce":37354},{"type":"v1","addr":"172.21.15.62:6827","nonce":37354}]},"public_addr":"172.21.15.62:6820/37354","cluster_addr":"172.21.15.62:6823/37354","heartbeat_back_addr":"172.21.15.62:6830/37354","heartbeat_front_addr":"172.21.15.62:6827/37354","state":["exists","up"]},{"osd":7,"uuid":"ac70b2a3-f41b-4bf2-b227-7f2cbb55426d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37355},{"type":"v1","addr":"172.21.15.62:6809","nonce":37355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37355},{"type":"v1","addr":"172.21.15.62:6811","nonce":37355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":37355},{"type":"v1","addr":"172.21.15.62:6815","nonce":37355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":37355},{"type":"v1","addr":"172.21.15.62:6813","nonce":37355}]},"public_addr":"172.21.15.62:6809/37355","cluster_addr":"172.21.15.62:6811/37355","heartbeat_back_addr":"172.21.15.62:6815/37355","heartbeat_front_addr":"172.21.15.62:6813/37355","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.368896+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.107273+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.276888+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.287502+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.124791+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.251937+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.255679+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.232797+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:0/1961044213":"2023-05-23T16:05:35.647562+0000","172.21.15.16:0/238292614":"2023-05-23T16:05:35.647562+0000","172.21.15.16:6833/65890":"2023-05-23T16:05:35.647562+0000","172.21.15.16:0/1148602276":"2023-05-23T16:05:35.647562+0000","172.21.15.16:6832/65890":"2023-05-23T16:05:35.647562+0000","172.21.15.16:0/456460873":"2023-05-23T16:05:35.647562+0000","172.21.15.16:0/2766506027":"2023-05-23T16:05:35.647562+0000","172.21.15.16:0/2214029832":"2023-05-23T16:05:35.647562+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":17,"snaps":[{"begin":2,"length":2}]},{"pool":21,"snaps":[{"begin":25,"length":9}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-05-22T16:05:37.774 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.772+0000 7f4be3bc6640 1 -- 172.21.15.16:0/707049011 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f4bc8065220 msgr2=0x7f4bc80676d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:37.774 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.772+0000 7f4be3bc6640 1 --2- 172.21.15.16:0/707049011 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f4bc8065220 0x7f4bc80676d0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f4bd4005c10 tx=0x7f4bd4009040 comp rx=0 tx=0).stop 2023-05-22T16:05:37.774 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.772+0000 7f4be3bc6640 1 -- 172.21.15.16:0/707049011 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4bdc1521d0 msgr2=0x7f4bdc15cb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:37.775 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.772+0000 7f4be3bc6640 1 --2- 172.21.15.16:0/707049011 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4bdc1521d0 0x7f4bdc15cb20 secure :-1 s=READY pgs=2504 cs=0 l=1 rev1=1 crypto rx=0x7f4bcc00d800 tx=0x7f4bcc00dcc0 comp rx=0 tx=0).stop 2023-05-22T16:05:37.775 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.772+0000 7f4be3bc6640 1 -- 172.21.15.16:0/707049011 shutdown_connections 2023-05-22T16:05:37.775 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.772+0000 7f4be3bc6640 1 --2- 172.21.15.16:0/707049011 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f4bc8065220 0x7f4bc80676d0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:37.775 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.772+0000 7f4be3bc6640 1 --2- 172.21.15.16:0/707049011 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4bdc1521d0 0x7f4bdc15cb20 unknown :-1 s=CLOSED pgs=2504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:37.775 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.772+0000 7f4be3bc6640 1 --2- 172.21.15.16:0/707049011 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4bdc160000 0x7f4bdc151c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:37.776 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.772+0000 7f4be3bc6640 1 --2- 172.21.15.16:0/707049011 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4bdc06cdb0 0x7f4bdc15f440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:37.776 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.772+0000 7f4be3bc6640 1 -- 172.21.15.16:0/707049011 >> 172.21.15.16:0/707049011 conn(0x7f4bdc0cab50 msgr2=0x7f4bdc0adb40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:05:37.776 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.772+0000 7f4be3bc6640 1 -- 172.21.15.16:0/707049011 shutdown_connections 2023-05-22T16:05:37.776 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.772+0000 7f4be3bc6640 1 -- 172.21.15.16:0/707049011 wait complete. 2023-05-22T16:05:37.792 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-05-22T15:26:38.019781+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-05-22T15:26:43.570514+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 24, 'pool_name': 'cephfs_metadata', 'create_time': '2023-05-22T16:05:20.485167+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '195', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 25, 'pool_name': 'cephfs_data', 'create_time': '2023-05-22T16:05:20.900001+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '196', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-05-22T16:05:37.793 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-05-22T16:05:37.913 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.912+0000 7f6e2a618640 1 Processor -- start 2023-05-22T16:05:37.914 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.912+0000 7f6e2a618640 1 -- start start 2023-05-22T16:05:37.917 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.912+0000 7f6e2a618640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6e2406e200 0x7f6e240ce580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:37.917 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.912+0000 7f6e2a618640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e24071370 con 0x7f6e24069eb0 2023-05-22T16:05:37.917 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.912+0000 7f6e2a618640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e24071650 con 0x7f6e2406ddd0 2023-05-22T16:05:37.917 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.912+0000 7f6e2a618640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e2415a2f0 con 0x7f6e2406e200 2023-05-22T16:05:37.918 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.912+0000 7f6e23fff640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f6e24069eb0 0x7f6e240b1680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:57190/0 (socket says 172.21.15.16:57190) 2023-05-22T16:05:37.918 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.912+0000 7f6e23fff640 1 -- 172.21.15.16:0/3200353561 learned_addr learned my addr 172.21.15.16:0/3200353561 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T16:05:37.918 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.912+0000 7f6e28bb5640 1 --2- 172.21.15.16:0/3200353561 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6e2406e200 0x7f6e240ce580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:37.918 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.912+0000 7f6e22ffd640 1 -- 172.21.15.16:0/3200353561 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 773410327 0 0) 0x7f6e24071370 con 0x7f6e24069eb0 2023-05-22T16:05:37.918 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.912+0000 7f6e22ffd640 1 -- 172.21.15.16:0/3200353561 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e10003530 con 0x7f6e24069eb0 2023-05-22T16:05:37.918 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.912+0000 7f6e22ffd640 1 -- 172.21.15.16:0/3200353561 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2263658289 0 0) 0x7f6e10003530 con 0x7f6e24069eb0 2023-05-22T16:05:37.918 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.912+0000 7f6e22ffd640 1 -- 172.21.15.16:0/3200353561 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6e2406e200 msgr2=0x7f6e240ce580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:05:37.919 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.912+0000 7f6e22ffd640 1 --2- 172.21.15.16:0/3200353561 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6e2406e200 0x7f6e240ce580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:37.919 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.912+0000 7f6e22ffd640 1 -- 172.21.15.16:0/3200353561 >> v1:172.21.15.62:6789/0 conn(0x7f6e2406ddd0 legacy=0x7f6e2415df00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:37.919 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.912+0000 7f6e22ffd640 1 -- 172.21.15.16:0/3200353561 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e24071930 con 0x7f6e24069eb0 2023-05-22T16:05:37.919 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.912+0000 7f6e22ffd640 1 -- 172.21.15.16:0/3200353561 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7f6e0c001680 con 0x7f6e24069eb0 2023-05-22T16:05:37.919 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.912+0000 7f6e22ffd640 1 -- 172.21.15.16:0/3200353561 >> v1:172.21.15.16:6789/0 conn(0x7f6e24069eb0 legacy=0x7f6e240b1680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:37.919 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.912+0000 7f6e22ffd640 1 --2- 172.21.15.16:0/3200353561 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6e100039b0 0x7f6e10003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:37.919 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.912+0000 7f6e22ffd640 1 --2- 172.21.15.16:0/3200353561 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6e2406e200 0x7f6e100047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:37.920 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.912+0000 7f6e22ffd640 1 --2- 172.21.15.16:0/3200353561 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6e10004d10 0x7f6e10005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:37.920 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.912+0000 7f6e22ffd640 1 -- 172.21.15.16:0/3200353561 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e1000bad0 con 0x7f6e100039b0 2023-05-22T16:05:37.920 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.912+0000 7f6e22ffd640 1 -- 172.21.15.16:0/3200353561 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e1000bc10 con 0x7f6e10004d10 2023-05-22T16:05:37.920 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.912+0000 7f6e22ffd640 1 -- 172.21.15.16:0/3200353561 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e1000bda0 con 0x7f6e2406e200 2023-05-22T16:05:37.920 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.912+0000 7f6e23fff640 1 --2- 172.21.15.16:0/3200353561 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6e100039b0 0x7f6e10003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:37.921 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.912+0000 7f6e28bb5640 1 --2- 172.21.15.16:0/3200353561 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6e10004d10 0x7f6e10005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:37.921 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.912+0000 7f6e237fe640 1 --2- 172.21.15.16:0/3200353561 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6e2406e200 0x7f6e100047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:37.921 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.912+0000 7f6e23fff640 1 -- 172.21.15.16:0/3200353561 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6e2406e200 msgr2=0x7f6e100047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:05:37.921 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.912+0000 7f6e23fff640 1 --2- 172.21.15.16:0/3200353561 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6e2406e200 0x7f6e100047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:37.922 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.912+0000 7f6e23fff640 1 -- 172.21.15.16:0/3200353561 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6e10004d10 msgr2=0x7f6e10005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:05:37.922 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.912+0000 7f6e23fff640 1 --2- 172.21.15.16:0/3200353561 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6e10004d10 0x7f6e10005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:37.922 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.912+0000 7f6e23fff640 1 -- 172.21.15.16:0/3200353561 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e10003530 con 0x7f6e100039b0 2023-05-22T16:05:37.922 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.912+0000 7f6e237fe640 1 --2- 172.21.15.16:0/3200353561 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6e2406e200 0x7f6e100047d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T16:05:37.924 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.916+0000 7f6e23fff640 1 --2- 172.21.15.16:0/3200353561 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6e100039b0 0x7f6e10003dc0 secure :-1 s=READY pgs=2506 cs=0 l=1 rev1=1 crypto rx=0x7f6e0c00c240 tx=0x7f6e0c01f3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fae68eac01ab1537 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:37.925 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.916+0000 7f6e22ffd640 1 -- 172.21.15.16:0/3200353561 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e0c005ae0 con 0x7f6e100039b0 2023-05-22T16:05:37.925 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.916+0000 7f6e22ffd640 1 -- 172.21.15.16:0/3200353561 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e0c0ab070 con 0x7f6e100039b0 2023-05-22T16:05:37.925 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.916+0000 7f6e22ffd640 1 -- 172.21.15.16:0/3200353561 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e0c005ea0 con 0x7f6e100039b0 2023-05-22T16:05:37.925 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.916+0000 7f6e2a618640 1 -- 172.21.15.16:0/3200353561 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6e100039b0 msgr2=0x7f6e10003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:37.925 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.916+0000 7f6e2a618640 1 --2- 172.21.15.16:0/3200353561 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6e100039b0 0x7f6e10003dc0 secure :-1 s=READY pgs=2506 cs=0 l=1 rev1=1 crypto rx=0x7f6e0c00c240 tx=0x7f6e0c01f3b0 comp rx=0 tx=0).stop 2023-05-22T16:05:37.926 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.916+0000 7f6e28bb5640 1 -- 172.21.15.16:0/3200353561 reap_dead start 2023-05-22T16:05:37.926 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.916+0000 7f6e2a618640 1 -- 172.21.15.16:0/3200353561 shutdown_connections 2023-05-22T16:05:37.926 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.916+0000 7f6e2a618640 1 -- 172.21.15.16:0/3200353561 >> 172.21.15.16:0/3200353561 conn(0x7f6e2406d5b0 msgr2=0x7f6e240b0de0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:05:37.926 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.916+0000 7f6e2a618640 1 -- 172.21.15.16:0/3200353561 shutdown_connections 2023-05-22T16:05:37.926 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.916+0000 7f6e2a618640 1 -- 172.21.15.16:0/3200353561 wait complete. 2023-05-22T16:05:37.926 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.916+0000 7f6e2a618640 1 Processor -- start 2023-05-22T16:05:37.926 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.916+0000 7f6e2a618640 1 -- start start 2023-05-22T16:05:37.927 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.920+0000 7f6e2a618640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6e100039b0 0x7f6e2415a700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:37.927 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.920+0000 7f6e2a618640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6e241f5850 0x7f6e240ac140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:37.927 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.920+0000 7f6e2a618640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6e240ac680 0x7f6e2415c200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:37.927 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.920+0000 7f6e2a618640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e240acad0 con 0x7f6e100039b0 2023-05-22T16:05:37.927 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.920+0000 7f6e2a618640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e24150d40 con 0x7f6e241f5850 2023-05-22T16:05:37.927 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.920+0000 7f6e2a618640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e2415ce50 con 0x7f6e240ac680 2023-05-22T16:05:37.927 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.920+0000 7f6e23fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6e100039b0 0x7f6e2415a700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:37.928 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.920+0000 7f6e23fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6e100039b0 0x7f6e2415a700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43116/0 (socket says 172.21.15.16:43116) 2023-05-22T16:05:37.928 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.920+0000 7f6e23fff640 1 -- 172.21.15.16:0/2332631706 learned_addr learned my addr 172.21.15.16:0/2332631706 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:05:37.928 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.920+0000 7f6e237fe640 1 --2- 172.21.15.16:0/2332631706 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6e241f5850 0x7f6e240ac140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:37.928 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.920+0000 7f6e28bb5640 1 --2- 172.21.15.16:0/2332631706 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6e240ac680 0x7f6e2415c200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:37.928 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.920+0000 7f6e23fff640 1 -- 172.21.15.16:0/2332631706 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6e240ac680 msgr2=0x7f6e2415c200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:37.929 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.920+0000 7f6e23fff640 1 --2- 172.21.15.16:0/2332631706 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6e240ac680 0x7f6e2415c200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:37.929 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.920+0000 7f6e23fff640 1 -- 172.21.15.16:0/2332631706 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6e241f5850 msgr2=0x7f6e240ac140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:37.929 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.920+0000 7f6e23fff640 1 --2- 172.21.15.16:0/2332631706 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6e241f5850 0x7f6e240ac140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:37.929 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.920+0000 7f6e23fff640 1 -- 172.21.15.16:0/2332631706 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e2415d9e0 con 0x7f6e100039b0 2023-05-22T16:05:37.929 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.920+0000 7f6e23fff640 1 --2- 172.21.15.16:0/2332631706 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6e100039b0 0x7f6e2415a700 secure :-1 s=READY pgs=2507 cs=0 l=1 rev1=1 crypto rx=0x7f6e0c026020 tx=0x7f6e10005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:37.929 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.920+0000 7f6e20ff9640 1 -- 172.21.15.16:0/2332631706 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e0c001640 con 0x7f6e100039b0 2023-05-22T16:05:37.929 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.920+0000 7f6e20ff9640 1 -- 172.21.15.16:0/2332631706 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e0c007070 con 0x7f6e100039b0 2023-05-22T16:05:37.930 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.920+0000 7f6e2a618640 1 -- 172.21.15.16:0/2332631706 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e24071c10 con 0x7f6e100039b0 2023-05-22T16:05:37.930 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.920+0000 7f6e20ff9640 1 -- 172.21.15.16:0/2332631706 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e0c007400 con 0x7f6e100039b0 2023-05-22T16:05:37.930 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.920+0000 7f6e2a618640 1 -- 172.21.15.16:0/2332631706 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e24070f90 con 0x7f6e100039b0 2023-05-22T16:05:37.930 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.920+0000 7f6e2a618640 1 -- 172.21.15.16:0/2332631706 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e241f4bb0 con 0x7f6e100039b0 2023-05-22T16:05:37.930 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.920+0000 7f6e20ff9640 1 -- 172.21.15.16:0/2332631706 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 96) v1 ==== 84406+0+0 (secure 0 0 0) 0x7f6e0c007560 con 0x7f6e100039b0 2023-05-22T16:05:37.930 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.920+0000 7f6e20ff9640 1 --2- 172.21.15.16:0/2332631706 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f6e040655b0 0x7f6e04067a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:37.930 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.920+0000 7f6e20ff9640 1 -- 172.21.15.16:0/2332631706 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(197..197 src has 1..197) v4 ==== 6782+0+0 (secure 0 0 0) 0x7f6e0c11f830 con 0x7f6e100039b0 2023-05-22T16:05:37.931 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.920+0000 7f6e237fe640 1 --2- 172.21.15.16:0/2332631706 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f6e040655b0 0x7f6e04067a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:37.931 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.920+0000 7f6e237fe640 1 --2- 172.21.15.16:0/2332631706 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f6e040655b0 0x7f6e04067a60 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f6e240c6690 tx=0x7f6e14009040 comp rx=0 tx=0).ready entity=mgr.10974 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:37.931 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:37.924+0000 7f6e20ff9640 1 -- 172.21.15.16:0/2332631706 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f6e0c0ecdf0 con 0x7f6e100039b0 2023-05-22T16:05:38.083 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.080+0000 7f6e2a618640 1 -- 172.21.15.16:0/2332631706 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f6e240aca50 con 0x7f6e100039b0 2023-05-22T16:05:38.084 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.084+0000 7f6e20ff9640 1 -- 172.21.15.16:0/2332631706 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v197) v1 ==== 93+0+10 (secure 0 0 0) 0x7f6e0c0f0410 con 0x7f6e100039b0 2023-05-22T16:05:38.085 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2023-05-22T16:05:38.087 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.084+0000 7f6e2a618640 1 -- 172.21.15.16:0/2332631706 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f6e040655b0 msgr2=0x7f6e04067a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:38.087 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.084+0000 7f6e2a618640 1 --2- 172.21.15.16:0/2332631706 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f6e040655b0 0x7f6e04067a60 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f6e240c6690 tx=0x7f6e14009040 comp rx=0 tx=0).stop 2023-05-22T16:05:38.087 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.084+0000 7f6e2a618640 1 -- 172.21.15.16:0/2332631706 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6e100039b0 msgr2=0x7f6e2415a700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:38.088 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.084+0000 7f6e2a618640 1 --2- 172.21.15.16:0/2332631706 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6e100039b0 0x7f6e2415a700 secure :-1 s=READY pgs=2507 cs=0 l=1 rev1=1 crypto rx=0x7f6e0c026020 tx=0x7f6e10005800 comp rx=0 tx=0).stop 2023-05-22T16:05:38.088 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.084+0000 7f6e2a618640 1 -- 172.21.15.16:0/2332631706 shutdown_connections 2023-05-22T16:05:38.088 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.084+0000 7f6e2a618640 1 --2- 172.21.15.16:0/2332631706 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6e240ac680 0x7f6e2415c200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:38.088 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.084+0000 7f6e2a618640 1 --2- 172.21.15.16:0/2332631706 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6e241f5850 0x7f6e240ac140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:38.088 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.084+0000 7f6e2a618640 1 --2- 172.21.15.16:0/2332631706 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f6e040655b0 0x7f6e04067a60 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:38.088 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.084+0000 7f6e2a618640 1 --2- 172.21.15.16:0/2332631706 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6e100039b0 0x7f6e2415a700 unknown :-1 s=CLOSED pgs=2507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:38.089 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.084+0000 7f6e2a618640 1 -- 172.21.15.16:0/2332631706 >> 172.21.15.16:0/2332631706 conn(0x7f6e2406d5b0 msgr2=0x7f6e2415de00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:05:38.089 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.084+0000 7f6e2a618640 1 -- 172.21.15.16:0/2332631706 shutdown_connections 2023-05-22T16:05:38.089 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.084+0000 7f6e2a618640 1 -- 172.21.15.16:0/2332631706 wait complete. 2023-05-22T16:05:38.105 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-05-22T16:05:38.233 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.232+0000 7f4bb47f5640 1 Processor -- start 2023-05-22T16:05:38.234 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.232+0000 7f4bb47f5640 1 -- start start 2023-05-22T16:05:38.235 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.232+0000 7f4bb47f5640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4bac0afbf0 0x7f4bac15e380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:38.235 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.232+0000 7f4bb47f5640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f4bac071530 0x7f4bac0cdda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:38.235 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.232+0000 7f4bb47f5640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4bac15e8c0 con 0x7f4bac15a020 2023-05-22T16:05:38.235 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.232+0000 7f4bb47f5640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4bac0c4b90 con 0x7f4bac071530 2023-05-22T16:05:38.235 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.232+0000 7f4bb47f5640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4bac06f8a0 con 0x7f4bac0afbf0 2023-05-22T16:05:38.236 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.232+0000 7f4bb2d92640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f4bac071530 0x7f4bac0cdda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:38.236 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.232+0000 7f4bb2591640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f4bac15a020 0x7f4bac070e20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:57194/0 (socket says 172.21.15.16:57194) 2023-05-22T16:05:38.236 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.232+0000 7f4bb2591640 1 -- 172.21.15.16:0/2782188849 learned_addr learned my addr 172.21.15.16:0/2782188849 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T16:05:38.236 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.232+0000 7f4bb1d90640 1 --2- 172.21.15.16:0/2782188849 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4bac0afbf0 0x7f4bac15e380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:38.237 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.232+0000 7f4bb2d92640 1 -- 172.21.15.16:0/2782188849 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4bac0afbf0 msgr2=0x7f4bac15e380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:05:38.237 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.232+0000 7f4bb2d92640 1 --2- 172.21.15.16:0/2782188849 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4bac0afbf0 0x7f4bac15e380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:38.237 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.232+0000 7f4bb2d92640 1 -- 172.21.15.16:0/2782188849 >> v1:172.21.15.16:6789/0 conn(0x7f4bac15a020 legacy=0x7f4bac070e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:38.237 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.232+0000 7f4bb2d92640 1 -- 172.21.15.16:0/2782188849 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bac0b0080 con 0x7f4bac071530 2023-05-22T16:05:38.237 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.232+0000 7f4bb158f640 1 -- 172.21.15.16:0/2782188849 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 432044096 0 0) 0x7f4ba8001120 con 0x7f4bac15a020 2023-05-22T16:05:38.238 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.236+0000 7f4bb2d92640 1 --2- 172.21.15.16:0/2782188849 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4bac071530 0x7f4bac0cdda0 secure :-1 s=READY pgs=2509 cs=0 l=1 rev1=1 crypto rx=0x7f4ba4009d90 tx=0x7f4ba401c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=511bd873b1d105ae server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:38.238 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.236+0000 7f4bb158f640 1 -- 172.21.15.16:0/2782188849 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ba40a59e0 con 0x7f4bac071530 2023-05-22T16:05:38.238 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.236+0000 7f4bb158f640 1 -- 172.21.15.16:0/2782188849 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ba40a5b40 con 0x7f4bac071530 2023-05-22T16:05:38.238 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.236+0000 7f4bb158f640 1 -- 172.21.15.16:0/2782188849 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ba40ad440 con 0x7f4bac071530 2023-05-22T16:05:38.239 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.236+0000 7f4bb47f5640 1 -- 172.21.15.16:0/2782188849 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4bac071530 msgr2=0x7f4bac0cdda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:38.239 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.236+0000 7f4bb47f5640 1 --2- 172.21.15.16:0/2782188849 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4bac071530 0x7f4bac0cdda0 secure :-1 s=READY pgs=2509 cs=0 l=1 rev1=1 crypto rx=0x7f4ba4009d90 tx=0x7f4ba401c920 comp rx=0 tx=0).stop 2023-05-22T16:05:38.239 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.236+0000 7f4bb47f5640 1 -- 172.21.15.16:0/2782188849 shutdown_connections 2023-05-22T16:05:38.239 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.236+0000 7f4bb47f5640 1 --2- 172.21.15.16:0/2782188849 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4bac071530 0x7f4bac0cdda0 unknown :-1 s=CLOSED pgs=2509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:38.240 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.236+0000 7f4bb47f5640 1 --2- 172.21.15.16:0/2782188849 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4bac0afbf0 0x7f4bac15e380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:38.240 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.236+0000 7f4bb47f5640 1 -- 172.21.15.16:0/2782188849 >> 172.21.15.16:0/2782188849 conn(0x7f4bac069800 msgr2=0x7f4bac069c00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:05:38.240 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.236+0000 7f4bb47f5640 1 -- 172.21.15.16:0/2782188849 shutdown_connections 2023-05-22T16:05:38.240 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.236+0000 7f4bb47f5640 1 -- 172.21.15.16:0/2782188849 wait complete. 2023-05-22T16:05:38.241 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.236+0000 7f4bb47f5640 1 Processor -- start 2023-05-22T16:05:38.241 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.236+0000 7f4bb47f5640 1 -- start start 2023-05-22T16:05:38.241 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.240+0000 7f4bb47f5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4bac071530 0x7f4bac1f9450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:38.241 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.240+0000 7f4bb47f5640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4bac0afbf0 0x7f4bac1f81c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:38.241 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.240+0000 7f4bb47f5640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4bac15c4a0 0x7f4bac15c940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:38.242 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.240+0000 7f4bb47f5640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bac15d2b0 con 0x7f4bac071530 2023-05-22T16:05:38.242 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.240+0000 7f4bb47f5640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bac1f89b0 con 0x7f4bac0afbf0 2023-05-22T16:05:38.242 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.240+0000 7f4bb47f5640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4bac1f8700 con 0x7f4bac15c4a0 2023-05-22T16:05:38.242 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.240+0000 7f4bb2591640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4bac071530 0x7f4bac1f9450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:38.242 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.240+0000 7f4bb2591640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4bac071530 0x7f4bac1f9450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43144/0 (socket says 172.21.15.16:43144) 2023-05-22T16:05:38.242 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.240+0000 7f4bb2591640 1 -- 172.21.15.16:0/166642883 learned_addr learned my addr 172.21.15.16:0/166642883 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:05:38.242 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.240+0000 7f4bb2d92640 1 --2- 172.21.15.16:0/166642883 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4bac15c4a0 0x7f4bac15c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:38.243 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.240+0000 7f4bb1d90640 1 --2- 172.21.15.16:0/166642883 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4bac0afbf0 0x7f4bac1f81c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:38.244 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.240+0000 7f4bb2591640 1 -- 172.21.15.16:0/166642883 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4bac15c4a0 msgr2=0x7f4bac15c940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:38.244 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.240+0000 7f4bb2591640 1 --2- 172.21.15.16:0/166642883 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4bac15c4a0 0x7f4bac15c940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:38.244 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.240+0000 7f4bb2591640 1 -- 172.21.15.16:0/166642883 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4bac0afbf0 msgr2=0x7f4bac1f81c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:38.244 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.240+0000 7f4bb2591640 1 --2- 172.21.15.16:0/166642883 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4bac0afbf0 0x7f4bac1f81c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:38.244 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.240+0000 7f4bb2591640 1 -- 172.21.15.16:0/166642883 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ba4009a70 con 0x7f4bac071530 2023-05-22T16:05:38.245 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.240+0000 7f4bb2591640 1 --2- 172.21.15.16:0/166642883 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4bac071530 0x7f4bac1f9450 secure :-1 s=READY pgs=2510 cs=0 l=1 rev1=1 crypto rx=0x7f4ba800c480 tx=0x7f4ba800c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:38.245 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.240+0000 7f4ba2ffd640 1 -- 172.21.15.16:0/166642883 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ba80074e0 con 0x7f4bac071530 2023-05-22T16:05:38.245 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.240+0000 7f4ba2ffd640 1 -- 172.21.15.16:0/166642883 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ba8007660 con 0x7f4bac071530 2023-05-22T16:05:38.245 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.240+0000 7f4bb47f5640 1 -- 172.21.15.16:0/166642883 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4bac0cd880 con 0x7f4bac071530 2023-05-22T16:05:38.245 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.240+0000 7f4ba2ffd640 1 -- 172.21.15.16:0/166642883 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ba8004a30 con 0x7f4bac071530 2023-05-22T16:05:38.245 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.240+0000 7f4bb47f5640 1 -- 172.21.15.16:0/166642883 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4bac0cd5a0 con 0x7f4bac071530 2023-05-22T16:05:38.245 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.240+0000 7f4bb47f5640 1 -- 172.21.15.16:0/166642883 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4bac0ceaf0 con 0x7f4bac071530 2023-05-22T16:05:38.245 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.240+0000 7f4ba2ffd640 1 -- 172.21.15.16:0/166642883 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 96) v1 ==== 84406+0+0 (secure 0 0 0) 0x7f4ba8004b90 con 0x7f4bac071530 2023-05-22T16:05:38.246 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.240+0000 7f4ba2ffd640 1 --2- 172.21.15.16:0/166642883 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f4b900655b0 0x7f4b90067a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:38.246 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.240+0000 7f4ba2ffd640 1 -- 172.21.15.16:0/166642883 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(197..197 src has 1..197) v4 ==== 6782+0+0 (secure 0 0 0) 0x7f4ba8082160 con 0x7f4bac071530 2023-05-22T16:05:38.246 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.240+0000 7f4bb1d90640 1 --2- 172.21.15.16:0/166642883 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f4b900655b0 0x7f4b90067a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:38.246 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.244+0000 7f4bb1d90640 1 --2- 172.21.15.16:0/166642883 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f4b900655b0 0x7f4b90067a60 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f4bac0cdea0 tx=0x7f4b9c008040 comp rx=0 tx=0).ready entity=mgr.10974 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:38.248 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.244+0000 7f4ba2ffd640 1 -- 172.21.15.16:0/166642883 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f4ba804f920 con 0x7f4bac071530 2023-05-22T16:05:38.400 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.396+0000 7f4bb47f5640 1 -- 172.21.15.16:0/166642883 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f4bac15e8c0 con 0x7f4bac071530 2023-05-22T16:05:38.401 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.400+0000 7f4ba2ffd640 1 -- 172.21.15.16:0/166642883 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v197) v1 ==== 92+0+10 (secure 0 0 0) 0x7f4bac15e8c0 con 0x7f4bac071530 2023-05-22T16:05:38.401 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 8 2023-05-22T16:05:38.403 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.400+0000 7f4bb47f5640 1 -- 172.21.15.16:0/166642883 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f4b900655b0 msgr2=0x7f4b90067a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:38.403 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.400+0000 7f4bb47f5640 1 --2- 172.21.15.16:0/166642883 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f4b900655b0 0x7f4b90067a60 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f4bac0cdea0 tx=0x7f4b9c008040 comp rx=0 tx=0).stop 2023-05-22T16:05:38.403 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.400+0000 7f4bb47f5640 1 -- 172.21.15.16:0/166642883 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4bac071530 msgr2=0x7f4bac1f9450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:38.403 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.400+0000 7f4bb47f5640 1 --2- 172.21.15.16:0/166642883 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4bac071530 0x7f4bac1f9450 secure :-1 s=READY pgs=2510 cs=0 l=1 rev1=1 crypto rx=0x7f4ba800c480 tx=0x7f4ba800c940 comp rx=0 tx=0).stop 2023-05-22T16:05:38.404 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.400+0000 7f4bb47f5640 1 -- 172.21.15.16:0/166642883 shutdown_connections 2023-05-22T16:05:38.404 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.400+0000 7f4bb47f5640 1 --2- 172.21.15.16:0/166642883 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4bac15c4a0 0x7f4bac15c940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:38.404 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.400+0000 7f4bb47f5640 1 --2- 172.21.15.16:0/166642883 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4bac0afbf0 0x7f4bac1f81c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:38.404 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.400+0000 7f4bb47f5640 1 --2- 172.21.15.16:0/166642883 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f4b900655b0 0x7f4b90067a60 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:38.404 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.400+0000 7f4bb47f5640 1 --2- 172.21.15.16:0/166642883 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4bac071530 0x7f4bac1f9450 unknown :-1 s=CLOSED pgs=2510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:38.404 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.400+0000 7f4bb47f5640 1 -- 172.21.15.16:0/166642883 >> 172.21.15.16:0/166642883 conn(0x7f4bac069800 msgr2=0x7f4bac069bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:05:38.404 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.400+0000 7f4bb47f5640 1 -- 172.21.15.16:0/166642883 shutdown_connections 2023-05-22T16:05:38.405 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.400+0000 7f4bb47f5640 1 -- 172.21.15.16:0/166642883 wait complete. 2023-05-22T16:05:38.421 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-05-22T16:05:38.549 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.548+0000 7f87b1fec640 1 Processor -- start 2023-05-22T16:05:38.549 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.548+0000 7f87b1fec640 1 -- start start 2023-05-22T16:05:38.551 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.548+0000 7f87b1fec640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f87ac15d3d0 0x7f87ac0cde00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:38.551 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.548+0000 7f87b1fec640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f87ac0ca880 0x7f87ac0ce340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:38.551 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.548+0000 7f87b1fec640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87ac0cb7a0 con 0x7f87ac0ce880 2023-05-22T16:05:38.551 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.548+0000 7f87b1fec640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87ac0afa60 con 0x7f87ac15d3d0 2023-05-22T16:05:38.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.548+0000 7f87b1fec640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87ac0b2770 con 0x7f87ac0ca880 2023-05-22T16:05:38.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.548+0000 7f87a2ffd640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f87ac0ca880 0x7f87ac0ce340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:38.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.548+0000 7f87ab7fe640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f87ac15d3d0 0x7f87ac0cde00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:38.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.548+0000 7f87a2ffd640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f87ac0ca880 0x7f87ac0ce340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3301/0 says I am v2:172.21.15.16:34658/0 (socket says 172.21.15.16:34658) 2023-05-22T16:05:38.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.548+0000 7f87a2ffd640 1 -- 172.21.15.16:0/3083541092 learned_addr learned my addr 172.21.15.16:0/3083541092 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:05:38.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.548+0000 7f87a2ffd640 1 -- 172.21.15.16:0/3083541092 >> v2:172.21.15.62:3300/0 conn(0x7f87ac15d3d0 msgr2=0x7f87ac0cde00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:05:38.554 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.548+0000 7f87a2ffd640 1 --2- 172.21.15.16:0/3083541092 >> v2:172.21.15.62:3300/0 conn(0x7f87ac15d3d0 0x7f87ac0cde00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:38.554 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.548+0000 7f87a2ffd640 1 -- 172.21.15.16:0/3083541092 >> v1:172.21.15.62:6789/0 conn(0x7f87ac0ce880 legacy=0x7f87ac15d8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:38.555 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.548+0000 7f87a2ffd640 1 -- 172.21.15.16:0/3083541092 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87ac0cc600 con 0x7f87ac0ca880 2023-05-22T16:05:38.555 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.548+0000 7f87ab7fe640 1 --2- 172.21.15.16:0/3083541092 >> v2:172.21.15.62:3300/0 conn(0x7f87ac15d3d0 0x7f87ac0cde00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-05-22T16:05:38.555 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.552+0000 7f87a2ffd640 1 --2- 172.21.15.16:0/3083541092 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f87ac0ca880 0x7f87ac0ce340 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f8794009870 tx=0x7f879401c540 comp rx=0 tx=0).ready entity=mon.2 client_cookie=60ce75eb22070279 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:38.555 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.552+0000 7f87aaffd640 1 -- 172.21.15.16:0/3083541092 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87940ab600 con 0x7f87ac0ca880 2023-05-22T16:05:38.556 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.552+0000 7f87aaffd640 1 -- 172.21.15.16:0/3083541092 <== mon.2 v2:172.21.15.62:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87940a7050 con 0x7f87ac0ca880 2023-05-22T16:05:38.556 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.552+0000 7f87aaffd640 1 -- 172.21.15.16:0/3083541092 <== mon.2 v2:172.21.15.62:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87940ab980 con 0x7f87ac0ca880 2023-05-22T16:05:38.556 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.552+0000 7f87b1fec640 1 -- 172.21.15.16:0/3083541092 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f87ac0ca880 msgr2=0x7f87ac0ce340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:38.557 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.552+0000 7f87b1fec640 1 --2- 172.21.15.16:0/3083541092 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f87ac0ca880 0x7f87ac0ce340 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f8794009870 tx=0x7f879401c540 comp rx=0 tx=0).stop 2023-05-22T16:05:38.557 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.552+0000 7f87b1fec640 1 -- 172.21.15.16:0/3083541092 shutdown_connections 2023-05-22T16:05:38.557 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.552+0000 7f87b1fec640 1 --2- 172.21.15.16:0/3083541092 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f87ac0ca880 0x7f87ac0ce340 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:38.557 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.552+0000 7f87b1fec640 1 --2- 172.21.15.16:0/3083541092 >> v2:172.21.15.62:3300/0 conn(0x7f87ac15d3d0 0x7f87ac0cde00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:38.557 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.552+0000 7f87b1fec640 1 -- 172.21.15.16:0/3083541092 >> 172.21.15.16:0/3083541092 conn(0x7f87ac099620 msgr2=0x7f87ac0d02c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:05:38.557 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.552+0000 7f87b1fec640 1 -- 172.21.15.16:0/3083541092 shutdown_connections 2023-05-22T16:05:38.557 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.552+0000 7f87b1fec640 1 -- 172.21.15.16:0/3083541092 wait complete. 2023-05-22T16:05:38.558 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.556+0000 7f87b1fec640 1 Processor -- start 2023-05-22T16:05:38.558 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.556+0000 7f87b1fec640 1 -- start start 2023-05-22T16:05:38.559 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.556+0000 7f87b1fec640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f87ac0ca880 0x7f87ac0c6890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:38.559 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.556+0000 7f87b1fec640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f87ac0ce880 0x7f87ac0ade60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:38.560 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.556+0000 7f87b1fec640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f87ac15d3d0 0x7f87ac0ae3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:38.560 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.556+0000 7f87b1fec640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87ac0b4d30 con 0x7f87ac15d3d0 2023-05-22T16:05:38.560 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.556+0000 7f87b1fec640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87ac0b5b90 con 0x7f87ac0ce880 2023-05-22T16:05:38.560 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.556+0000 7f87b1fec640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87ac0c6dd0 con 0x7f87ac0ca880 2023-05-22T16:05:38.560 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.556+0000 7f87abfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f87ac15d3d0 0x7f87ac0ae3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:38.560 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.556+0000 7f87abfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f87ac15d3d0 0x7f87ac0ae3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43158/0 (socket says 172.21.15.16:43158) 2023-05-22T16:05:38.561 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.556+0000 7f87abfff640 1 -- 172.21.15.16:0/930809235 learned_addr learned my addr 172.21.15.16:0/930809235 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:05:38.561 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.556+0000 7f87ab7fe640 1 --2- 172.21.15.16:0/930809235 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f87ac0ca880 0x7f87ac0c6890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:38.561 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.556+0000 7f87a2ffd640 1 --2- 172.21.15.16:0/930809235 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f87ac0ce880 0x7f87ac0ade60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:38.561 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.556+0000 7f87abfff640 1 -- 172.21.15.16:0/930809235 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f87ac0ca880 msgr2=0x7f87ac0c6890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:38.561 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.556+0000 7f87abfff640 1 --2- 172.21.15.16:0/930809235 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f87ac0ca880 0x7f87ac0c6890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:38.561 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.556+0000 7f87abfff640 1 -- 172.21.15.16:0/930809235 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f87ac0ce880 msgr2=0x7f87ac0ade60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:38.561 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.556+0000 7f87abfff640 1 --2- 172.21.15.16:0/930809235 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f87ac0ce880 0x7f87ac0ade60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:38.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.556+0000 7f87abfff640 1 -- 172.21.15.16:0/930809235 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f879c0093b0 con 0x7f87ac15d3d0 2023-05-22T16:05:38.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.556+0000 7f87a2ffd640 1 --2- 172.21.15.16:0/930809235 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f87ac0ce880 0x7f87ac0ade60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T16:05:38.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.556+0000 7f87abfff640 1 --2- 172.21.15.16:0/930809235 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f87ac15d3d0 0x7f87ac0ae3a0 secure :-1 s=READY pgs=2511 cs=0 l=1 rev1=1 crypto rx=0x7f87a400c770 tx=0x7f87a400cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:38.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.556+0000 7f87a8ff9640 1 -- 172.21.15.16:0/930809235 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87a4003b80 con 0x7f87ac15d3d0 2023-05-22T16:05:38.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.556+0000 7f87a8ff9640 1 -- 172.21.15.16:0/930809235 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87a4003ce0 con 0x7f87ac15d3d0 2023-05-22T16:05:38.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.556+0000 7f87b1fec640 1 -- 172.21.15.16:0/930809235 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87940094d0 con 0x7f87ac15d3d0 2023-05-22T16:05:38.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.556+0000 7f87a8ff9640 1 -- 172.21.15.16:0/930809235 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87a400aa90 con 0x7f87ac15d3d0 2023-05-22T16:05:38.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.556+0000 7f87b1fec640 1 -- 172.21.15.16:0/930809235 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87ac0cc290 con 0x7f87ac15d3d0 2023-05-22T16:05:38.565 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.560+0000 7f87b1fec640 1 -- 172.21.15.16:0/930809235 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87ac0cbfb0 con 0x7f87ac15d3d0 2023-05-22T16:05:38.565 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.560+0000 7f87a8ff9640 1 -- 172.21.15.16:0/930809235 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 96) v1 ==== 84406+0+0 (secure 0 0 0) 0x7f87a400abf0 con 0x7f87ac15d3d0 2023-05-22T16:05:38.565 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.560+0000 7f87a8ff9640 1 --2- 172.21.15.16:0/930809235 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f878c065580 0x7f878c067a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:38.566 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.560+0000 7f87ab7fe640 1 --2- 172.21.15.16:0/930809235 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f878c065580 0x7f878c067a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:38.566 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.560+0000 7f87a8ff9640 1 -- 172.21.15.16:0/930809235 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(197..197 src has 1..197) v4 ==== 6782+0+0 (secure 0 0 0) 0x7f87a4083720 con 0x7f87ac15d3d0 2023-05-22T16:05:38.566 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.560+0000 7f87ab7fe640 1 --2- 172.21.15.16:0/930809235 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f878c065580 0x7f878c067a30 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f87ac0afe20 tx=0x7f879c006f90 comp rx=0 tx=0).ready entity=mgr.10974 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:38.567 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.564+0000 7f87a8ff9640 1 -- 172.21.15.16:0/930809235 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f87a4050ec0 con 0x7f87ac15d3d0 2023-05-22T16:05:38.722 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.720+0000 7f87b1fec640 1 -- 172.21.15.16:0/930809235 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f87ac071a80 con 0x7f87ac15d3d0 2023-05-22T16:05:38.722 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.720+0000 7f87a8ff9640 1 -- 172.21.15.16:0/930809235 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v197) v1 ==== 104+0+11 (secure 0 0 0) 0x7f87a40544e0 con 0x7f87ac15d3d0 2023-05-22T16:05:38.723 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2023-05-22T16:05:38.725 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.724+0000 7f87b1fec640 1 -- 172.21.15.16:0/930809235 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f878c065580 msgr2=0x7f878c067a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:38.725 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.724+0000 7f87b1fec640 1 --2- 172.21.15.16:0/930809235 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f878c065580 0x7f878c067a30 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f87ac0afe20 tx=0x7f879c006f90 comp rx=0 tx=0).stop 2023-05-22T16:05:38.725 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.724+0000 7f87b1fec640 1 -- 172.21.15.16:0/930809235 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f87ac15d3d0 msgr2=0x7f87ac0ae3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:38.726 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.724+0000 7f87b1fec640 1 --2- 172.21.15.16:0/930809235 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f87ac15d3d0 0x7f87ac0ae3a0 secure :-1 s=READY pgs=2511 cs=0 l=1 rev1=1 crypto rx=0x7f87a400c770 tx=0x7f87a400cc30 comp rx=0 tx=0).stop 2023-05-22T16:05:38.726 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.724+0000 7f87b1fec640 1 -- 172.21.15.16:0/930809235 shutdown_connections 2023-05-22T16:05:38.726 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.724+0000 7f87b1fec640 1 --2- 172.21.15.16:0/930809235 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f878c065580 0x7f878c067a30 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:38.726 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.724+0000 7f87b1fec640 1 --2- 172.21.15.16:0/930809235 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f87ac15d3d0 0x7f87ac0ae3a0 unknown :-1 s=CLOSED pgs=2511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:38.726 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.724+0000 7f87b1fec640 1 --2- 172.21.15.16:0/930809235 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f87ac0ce880 0x7f87ac0ade60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:38.726 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.724+0000 7f87b1fec640 1 --2- 172.21.15.16:0/930809235 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f87ac0ca880 0x7f87ac0c6890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:38.726 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.724+0000 7f87b1fec640 1 -- 172.21.15.16:0/930809235 >> 172.21.15.16:0/930809235 conn(0x7f87ac099620 msgr2=0x7f87ac0cfff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:05:38.727 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.724+0000 7f87b1fec640 1 -- 172.21.15.16:0/930809235 shutdown_connections 2023-05-22T16:05:38.727 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.724+0000 7f87b1fec640 1 -- 172.21.15.16:0/930809235 wait complete. 2023-05-22T16:05:38.743 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-05-22T16:05:38.865 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.864+0000 7f6d9989f640 1 Processor -- start 2023-05-22T16:05:38.866 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.864+0000 7f6d9989f640 1 -- start start 2023-05-22T16:05:38.867 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.864+0000 7f6d9989f640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6d9415d3d0 0x7f6d940cde00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:38.867 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.864+0000 7f6d9989f640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6d940ace10 0x7f6d940ad290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:38.867 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.864+0000 7f6d9989f640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d940cb7a0 con 0x7f6d940ca880 2023-05-22T16:05:38.867 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.864+0000 7f6d9989f640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d940afa60 con 0x7f6d940ace10 2023-05-22T16:05:38.868 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.864+0000 7f6d9989f640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d940b2770 con 0x7f6d9415d3d0 2023-05-22T16:05:38.868 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.864+0000 7f6d937fe640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6d940ace10 0x7f6d940ad290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:38.868 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.864+0000 7f6d937fe640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6d940ace10 0x7f6d940ad290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43172/0 (socket says 172.21.15.16:43172) 2023-05-22T16:05:38.868 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.864+0000 7f6d937fe640 1 -- 172.21.15.16:0/2342057570 learned_addr learned my addr 172.21.15.16:0/2342057570 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:05:38.869 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.864+0000 7f6d92ffd640 1 --2- 172.21.15.16:0/2342057570 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6d9415d3d0 0x7f6d940cde00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:38.869 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.864+0000 7f6d937fe640 1 -- 172.21.15.16:0/2342057570 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6d9415d3d0 msgr2=0x7f6d940cde00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:05:38.869 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.864+0000 7f6d937fe640 1 --2- 172.21.15.16:0/2342057570 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6d9415d3d0 0x7f6d940cde00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:38.869 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.864+0000 7f6d937fe640 1 -- 172.21.15.16:0/2342057570 >> v1:172.21.15.62:6789/0 conn(0x7f6d940ca880 legacy=0x7f6d940ce340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:05:38.870 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.864+0000 7f6d937fe640 1 -- 172.21.15.16:0/2342057570 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d940cc600 con 0x7f6d940ace10 2023-05-22T16:05:38.871 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.868+0000 7f6d937fe640 1 --2- 172.21.15.16:0/2342057570 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6d940ace10 0x7f6d940ad290 secure :-1 s=READY pgs=2512 cs=0 l=1 rev1=1 crypto rx=0x7f6d8c009870 tx=0x7f6d8c01c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cbaa81d49fbca653 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:38.871 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.868+0000 7f6d91ffb640 1 -- 172.21.15.16:0/2342057570 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d8c0ab600 con 0x7f6d940ace10 2023-05-22T16:05:38.872 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.868+0000 7f6d91ffb640 1 -- 172.21.15.16:0/2342057570 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d8c0a7050 con 0x7f6d940ace10 2023-05-22T16:05:38.872 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.868+0000 7f6d91ffb640 1 -- 172.21.15.16:0/2342057570 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d8c0ab980 con 0x7f6d940ace10 2023-05-22T16:05:38.872 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.868+0000 7f6d9989f640 1 -- 172.21.15.16:0/2342057570 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6d940ace10 msgr2=0x7f6d940ad290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:38.873 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.868+0000 7f6d9989f640 1 --2- 172.21.15.16:0/2342057570 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6d940ace10 0x7f6d940ad290 secure :-1 s=READY pgs=2512 cs=0 l=1 rev1=1 crypto rx=0x7f6d8c009870 tx=0x7f6d8c01c500 comp rx=0 tx=0).stop 2023-05-22T16:05:38.873 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.868+0000 7f6d9989f640 1 -- 172.21.15.16:0/2342057570 shutdown_connections 2023-05-22T16:05:38.873 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.868+0000 7f6d9989f640 1 --2- 172.21.15.16:0/2342057570 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6d940ace10 0x7f6d940ad290 unknown :-1 s=CLOSED pgs=2512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:38.873 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.868+0000 7f6d9989f640 1 --2- 172.21.15.16:0/2342057570 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6d9415d3d0 0x7f6d940cde00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:38.874 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.868+0000 7f6d9989f640 1 -- 172.21.15.16:0/2342057570 >> 172.21.15.16:0/2342057570 conn(0x7f6d94099620 msgr2=0x7f6d940d02c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:05:38.874 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.868+0000 7f6d9989f640 1 -- 172.21.15.16:0/2342057570 shutdown_connections 2023-05-22T16:05:38.874 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.868+0000 7f6d9989f640 1 -- 172.21.15.16:0/2342057570 wait complete. 2023-05-22T16:05:38.878 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.868+0000 7f6d9989f640 1 Processor -- start 2023-05-22T16:05:38.878 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.868+0000 7f6d9989f640 1 -- start start 2023-05-22T16:05:38.878 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.868+0000 7f6d9989f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6d940ace10 0x7f6d940b4ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:38.878 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.868+0000 7f6d9989f640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6d940ca880 0x7f6d940b30f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:38.878 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.868+0000 7f6d9989f640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6d9415d3d0 0x7f6d9415d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:38.879 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.868+0000 7f6d9989f640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d940b4410 con 0x7f6d940ace10 2023-05-22T16:05:38.879 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.868+0000 7f6d9989f640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d940b3b70 con 0x7f6d940ca880 2023-05-22T16:05:38.879 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.868+0000 7f6d9989f640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d940b3e50 con 0x7f6d9415d3d0 2023-05-22T16:05:38.879 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.872+0000 7f6d92ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6d940ace10 0x7f6d940b4ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:38.879 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.872+0000 7f6d92ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6d940ace10 0x7f6d940b4ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43180/0 (socket says 172.21.15.16:43180) 2023-05-22T16:05:38.879 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.872+0000 7f6d92ffd640 1 -- 172.21.15.16:0/2626615951 learned_addr learned my addr 172.21.15.16:0/2626615951 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:05:38.880 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.872+0000 7f6d937fe640 1 --2- 172.21.15.16:0/2626615951 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6d9415d3d0 0x7f6d9415d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:38.880 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.872+0000 7f6d927fc640 1 --2- 172.21.15.16:0/2626615951 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6d940ca880 0x7f6d940b30f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:38.880 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.872+0000 7f6d92ffd640 1 -- 172.21.15.16:0/2626615951 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6d9415d3d0 msgr2=0x7f6d9415d8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:38.880 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.872+0000 7f6d92ffd640 1 --2- 172.21.15.16:0/2626615951 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6d9415d3d0 0x7f6d9415d8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:38.880 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.872+0000 7f6d92ffd640 1 -- 172.21.15.16:0/2626615951 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6d940ca880 msgr2=0x7f6d940b30f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:38.880 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.872+0000 7f6d92ffd640 1 --2- 172.21.15.16:0/2626615951 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6d940ca880 0x7f6d940b30f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:38.881 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.872+0000 7f6d92ffd640 1 -- 172.21.15.16:0/2626615951 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d8c0094d0 con 0x7f6d940ace10 2023-05-22T16:05:38.881 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.872+0000 7f6d92ffd640 1 --2- 172.21.15.16:0/2626615951 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6d940ace10 0x7f6d940b4ea0 secure :-1 s=READY pgs=2513 cs=0 l=1 rev1=1 crypto rx=0x7f6d84007ef0 tx=0x7f6d8400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:38.881 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.872+0000 7f6d7bfff640 1 -- 172.21.15.16:0/2626615951 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d84014620 con 0x7f6d940ace10 2023-05-22T16:05:38.881 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.872+0000 7f6d7bfff640 1 -- 172.21.15.16:0/2626615951 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d8400ee90 con 0x7f6d940ace10 2023-05-22T16:05:38.881 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.872+0000 7f6d7bfff640 1 -- 172.21.15.16:0/2626615951 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d840149c0 con 0x7f6d940ace10 2023-05-22T16:05:38.881 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.872+0000 7f6d9989f640 1 -- 172.21.15.16:0/2626615951 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d940cc290 con 0x7f6d940ace10 2023-05-22T16:05:38.881 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.872+0000 7f6d9989f640 1 -- 172.21.15.16:0/2626615951 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d940cbfb0 con 0x7f6d940ace10 2023-05-22T16:05:38.881 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.872+0000 7f6d9989f640 1 -- 172.21.15.16:0/2626615951 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d940cea50 con 0x7f6d940ace10 2023-05-22T16:05:38.882 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.872+0000 7f6d7bfff640 1 -- 172.21.15.16:0/2626615951 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 96) v1 ==== 84406+0+0 (secure 0 0 0) 0x7f6d84006d50 con 0x7f6d940ace10 2023-05-22T16:05:38.882 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.872+0000 7f6d7bfff640 1 --2- 172.21.15.16:0/2626615951 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f6d800655b0 0x7f6d80067a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:38.882 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.872+0000 7f6d7bfff640 1 -- 172.21.15.16:0/2626615951 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(197..197 src has 1..197) v4 ==== 6782+0+0 (secure 0 0 0) 0x7f6d84083520 con 0x7f6d940ace10 2023-05-22T16:05:38.882 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.872+0000 7f6d927fc640 1 --2- 172.21.15.16:0/2626615951 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f6d800655b0 0x7f6d80067a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:38.882 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.872+0000 7f6d927fc640 1 --2- 172.21.15.16:0/2626615951 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f6d800655b0 0x7f6d80067a60 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f6d9409e160 tx=0x7f6d88009040 comp rx=0 tx=0).ready entity=mgr.10974 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:38.883 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:38.876+0000 7f6d7bfff640 1 -- 172.21.15.16:0/2626615951 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f6d84050ce0 con 0x7f6d940ace10 2023-05-22T16:05:39.032 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.028+0000 7f6d9989f640 1 -- 172.21.15.16:0/2626615951 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f6d940b4550 con 0x7f6d940ace10 2023-05-22T16:05:39.033 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.032+0000 7f6d7bfff640 1 -- 172.21.15.16:0/2626615951 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v197) v1 ==== 100+0+11 (secure 0 0 0) 0x7f6d940b4550 con 0x7f6d940ace10 2023-05-22T16:05:39.033 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 64 2023-05-22T16:05:39.036 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.032+0000 7f6d9989f640 1 -- 172.21.15.16:0/2626615951 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f6d800655b0 msgr2=0x7f6d80067a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:39.036 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.032+0000 7f6d9989f640 1 --2- 172.21.15.16:0/2626615951 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f6d800655b0 0x7f6d80067a60 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f6d9409e160 tx=0x7f6d88009040 comp rx=0 tx=0).stop 2023-05-22T16:05:39.036 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.032+0000 7f6d9989f640 1 -- 172.21.15.16:0/2626615951 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6d940ace10 msgr2=0x7f6d940b4ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:39.037 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.032+0000 7f6d9989f640 1 --2- 172.21.15.16:0/2626615951 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6d940ace10 0x7f6d940b4ea0 secure :-1 s=READY pgs=2513 cs=0 l=1 rev1=1 crypto rx=0x7f6d84007ef0 tx=0x7f6d8400e4b0 comp rx=0 tx=0).stop 2023-05-22T16:05:39.037 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.032+0000 7f6d9989f640 1 -- 172.21.15.16:0/2626615951 shutdown_connections 2023-05-22T16:05:39.037 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.032+0000 7f6d9989f640 1 --2- 172.21.15.16:0/2626615951 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6d9415d3d0 0x7f6d9415d8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:39.037 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.032+0000 7f6d9989f640 1 --2- 172.21.15.16:0/2626615951 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6d940ca880 0x7f6d940b30f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:39.038 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.032+0000 7f6d9989f640 1 --2- 172.21.15.16:0/2626615951 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f6d800655b0 0x7f6d80067a60 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:39.038 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.032+0000 7f6d9989f640 1 --2- 172.21.15.16:0/2626615951 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6d940ace10 0x7f6d940b4ea0 unknown :-1 s=CLOSED pgs=2513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:39.038 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.032+0000 7f6d9989f640 1 -- 172.21.15.16:0/2626615951 >> 172.21.15.16:0/2626615951 conn(0x7f6d94099620 msgr2=0x7f6d940cde00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:05:39.039 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.032+0000 7f6d9989f640 1 -- 172.21.15.16:0/2626615951 shutdown_connections 2023-05-22T16:05:39.039 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.032+0000 7f6d9989f640 1 -- 172.21.15.16:0/2626615951 wait complete. 2023-05-22T16:05:39.053 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-05-22T16:05:39.173 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.172+0000 7f7f8e9df640 1 Processor -- start 2023-05-22T16:05:39.174 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.172+0000 7f7f8e9df640 1 -- start start 2023-05-22T16:05:39.174 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.172+0000 7f7f8e9df640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f7f88160110 0x7f7f8815c9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:39.175 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.172+0000 7f7f8e9df640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7f88160590 0x7f7f880ccb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:39.175 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.172+0000 7f7f8e9df640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f880d0020 con 0x7f7f8815fd40 2023-05-22T16:05:39.175 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.172+0000 7f7f8e9df640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f8815c2e0 con 0x7f7f88160110 2023-05-22T16:05:39.175 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.172+0000 7f7f8e9df640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f880d0830 con 0x7f7f88160590 2023-05-22T16:05:39.176 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.172+0000 7f7f877fe640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f7f88160110 0x7f7f8815c9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:39.176 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.172+0000 7f7f8cf7c640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7f88160590 0x7f7f880ccb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:39.176 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.172+0000 7f7f87fff640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f7f8815fd40 0x7f7f880d1c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.16:54304/0 (socket says 172.21.15.16:54304) 2023-05-22T16:05:39.176 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.172+0000 7f7f87fff640 1 -- 172.21.15.16:0/3497437853 learned_addr learned my addr 172.21.15.16:0/3497437853 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T16:05:39.176 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.172+0000 7f7f8cf7c640 1 -- 172.21.15.16:0/3497437853 >> v2:172.21.15.62:3300/0 conn(0x7f7f88160110 msgr2=0x7f7f8815c9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:05:39.177 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.172+0000 7f7f8cf7c640 1 --2- 172.21.15.16:0/3497437853 >> v2:172.21.15.62:3300/0 conn(0x7f7f88160110 0x7f7f8815c9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:39.177 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.172+0000 7f7f8cf7c640 1 -- 172.21.15.16:0/3497437853 >> v1:172.21.15.62:6789/0 conn(0x7f7f8815fd40 legacy=0x7f7f880d1c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:39.177 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.172+0000 7f7f8cf7c640 1 -- 172.21.15.16:0/3497437853 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f880d0270 con 0x7f7f88160590 2023-05-22T16:05:39.177 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.172+0000 7f7f877fe640 1 --2- 172.21.15.16:0/3497437853 >> v2:172.21.15.62:3300/0 conn(0x7f7f88160110 0x7f7f8815c9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-05-22T16:05:39.178 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.176+0000 7f7f8cf7c640 1 --2- 172.21.15.16:0/3497437853 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7f88160590 0x7f7f880ccb30 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f7f80009d90 tx=0x7f7f8001c8c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c8c9a3f0393091d2 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:39.178 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.176+0000 7f7f86ffd640 1 -- 172.21.15.16:0/3497437853 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f800a5890 con 0x7f7f88160590 2023-05-22T16:05:39.179 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.176+0000 7f7f86ffd640 1 -- 172.21.15.16:0/3497437853 <== mon.2 v2:172.21.15.62:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f800a59f0 con 0x7f7f88160590 2023-05-22T16:05:39.179 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.176+0000 7f7f86ffd640 1 -- 172.21.15.16:0/3497437853 <== mon.2 v2:172.21.15.62:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f800a5d80 con 0x7f7f88160590 2023-05-22T16:05:39.179 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.176+0000 7f7f8e9df640 1 -- 172.21.15.16:0/3497437853 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7f88160590 msgr2=0x7f7f880ccb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:39.180 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.176+0000 7f7f8e9df640 1 --2- 172.21.15.16:0/3497437853 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7f88160590 0x7f7f880ccb30 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f7f80009d90 tx=0x7f7f8001c8c0 comp rx=0 tx=0).stop 2023-05-22T16:05:39.180 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.176+0000 7f7f8e9df640 1 -- 172.21.15.16:0/3497437853 shutdown_connections 2023-05-22T16:05:39.180 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.176+0000 7f7f8e9df640 1 --2- 172.21.15.16:0/3497437853 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7f88160590 0x7f7f880ccb30 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:39.181 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.176+0000 7f7f8e9df640 1 --2- 172.21.15.16:0/3497437853 >> v2:172.21.15.62:3300/0 conn(0x7f7f88160110 0x7f7f8815c9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:39.181 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.176+0000 7f7f8e9df640 1 -- 172.21.15.16:0/3497437853 >> 172.21.15.16:0/3497437853 conn(0x7f7f880655c0 msgr2=0x7f7f88070f30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:05:39.181 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.176+0000 7f7f8e9df640 1 -- 172.21.15.16:0/3497437853 shutdown_connections 2023-05-22T16:05:39.181 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.176+0000 7f7f8e9df640 1 -- 172.21.15.16:0/3497437853 wait complete. 2023-05-22T16:05:39.182 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.180+0000 7f7f8e9df640 1 Processor -- start 2023-05-22T16:05:39.183 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.180+0000 7f7f8e9df640 1 -- start start 2023-05-22T16:05:39.183 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.180+0000 7f7f8e9df640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7f8815fd40 0x7f7f881545e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:39.183 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.180+0000 7f7f8e9df640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7f88160110 0x7f7f8814f9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:39.183 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.180+0000 7f7f8e9df640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7f88160590 0x7f7f88150810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:39.183 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.180+0000 7f7f8e9df640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f88153580 con 0x7f7f88160590 2023-05-22T16:05:39.183 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.180+0000 7f7f8e9df640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f88153820 con 0x7f7f8815fd40 2023-05-22T16:05:39.183 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.180+0000 7f7f8e9df640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f88153b00 con 0x7f7f88160110 2023-05-22T16:05:39.184 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.180+0000 7f7f8cf7c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7f88160590 0x7f7f88150810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:39.184 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.180+0000 7f7f8cf7c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7f88160590 0x7f7f88150810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59274/0 (socket says 172.21.15.16:59274) 2023-05-22T16:05:39.184 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.180+0000 7f7f8cf7c640 1 -- 172.21.15.16:0/942939513 learned_addr learned my addr 172.21.15.16:0/942939513 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:05:39.184 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.180+0000 7f7f87fff640 1 --2- 172.21.15.16:0/942939513 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7f8815fd40 0x7f7f881545e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:39.184 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.180+0000 7f7f877fe640 1 --2- 172.21.15.16:0/942939513 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7f88160110 0x7f7f8814f9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:39.184 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.180+0000 7f7f8cf7c640 1 -- 172.21.15.16:0/942939513 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7f88160110 msgr2=0x7f7f8814f9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:39.184 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.180+0000 7f7f8cf7c640 1 --2- 172.21.15.16:0/942939513 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7f88160110 0x7f7f8814f9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:39.184 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.180+0000 7f7f8cf7c640 1 -- 172.21.15.16:0/942939513 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7f8815fd40 msgr2=0x7f7f881545e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:39.185 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.180+0000 7f7f8cf7c640 1 --2- 172.21.15.16:0/942939513 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7f8815fd40 0x7f7f881545e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:39.185 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.180+0000 7f7f8cf7c640 1 -- 172.21.15.16:0/942939513 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f7c009420 con 0x7f7f88160590 2023-05-22T16:05:39.185 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.180+0000 7f7f8cf7c640 1 --2- 172.21.15.16:0/942939513 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7f88160590 0x7f7f88150810 secure :-1 s=READY pgs=2514 cs=0 l=1 rev1=1 crypto rx=0x7f7f8001a4f0 tx=0x7f7f80019660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:39.185 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.180+0000 7f7f84ff9640 1 -- 172.21.15.16:0/942939513 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f800ad640 con 0x7f7f88160590 2023-05-22T16:05:39.185 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.180+0000 7f7f84ff9640 1 -- 172.21.15.16:0/942939513 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f800ad7a0 con 0x7f7f88160590 2023-05-22T16:05:39.185 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.180+0000 7f7f84ff9640 1 -- 172.21.15.16:0/942939513 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f800adb30 con 0x7f7f88160590 2023-05-22T16:05:39.185 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.180+0000 7f7f8e9df640 1 -- 172.21.15.16:0/942939513 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f80009a70 con 0x7f7f88160590 2023-05-22T16:05:39.186 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.180+0000 7f7f8e9df640 1 -- 172.21.15.16:0/942939513 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f880cf890 con 0x7f7f88160590 2023-05-22T16:05:39.186 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.180+0000 7f7f8e9df640 1 -- 172.21.15.16:0/942939513 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f880cf700 con 0x7f7f88160590 2023-05-22T16:05:39.187 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.180+0000 7f7f84ff9640 1 -- 172.21.15.16:0/942939513 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 96) v1 ==== 84406+0+0 (secure 0 0 0) 0x7f7f800b4070 con 0x7f7f88160590 2023-05-22T16:05:39.187 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.184+0000 7f7f84ff9640 1 --2- 172.21.15.16:0/942939513 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f7f740655b0 0x7f7f74067a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:39.187 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.184+0000 7f7f84ff9640 1 -- 172.21.15.16:0/942939513 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(197..197 src has 1..197) v4 ==== 6782+0+0 (secure 0 0 0) 0x7f7f8011dee0 con 0x7f7f88160590 2023-05-22T16:05:39.187 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.184+0000 7f7f87fff640 1 --2- 172.21.15.16:0/942939513 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f7f740655b0 0x7f7f74067a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:39.187 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.184+0000 7f7f87fff640 1 --2- 172.21.15.16:0/942939513 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f7f740655b0 0x7f7f74067a60 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f7f8815f3b0 tx=0x7f7f78009040 comp rx=0 tx=0).ready entity=mgr.10974 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:39.189 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.188+0000 7f7f84ff9640 1 -- 172.21.15.16:0/942939513 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f7f800eb580 con 0x7f7f88160590 2023-05-22T16:05:39.360 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.356+0000 7f7f8e9df640 1 -- 172.21.15.16:0/942939513 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f7f880b7350 con 0x7f7f88160590 2023-05-22T16:05:39.361 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.360+0000 7f7f84ff9640 1 -- 172.21.15.16:0/942939513 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 8 entries v197) v1 ==== 80+0+458 (secure 0 0 0) 0x7f7f800b2080 con 0x7f7f88160590 2023-05-22T16:05:39.362 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/1961044213 2023-05-23T16:05:35.647562+0000 2023-05-22T16:05:39.362 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/238292614 2023-05-23T16:05:35.647562+0000 2023-05-22T16:05:39.362 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:6833/65890 2023-05-23T16:05:35.647562+0000 2023-05-22T16:05:39.362 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/1148602276 2023-05-23T16:05:35.647562+0000 2023-05-22T16:05:39.362 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:6832/65890 2023-05-23T16:05:35.647562+0000 2023-05-22T16:05:39.363 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/456460873 2023-05-23T16:05:35.647562+0000 2023-05-22T16:05:39.363 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/2766506027 2023-05-23T16:05:35.647562+0000 2023-05-22T16:05:39.363 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/2214029832 2023-05-23T16:05:35.647562+0000 2023-05-22T16:05:39.364 INFO:teuthology.orchestra.run.smithi016.stderr:listed 8 entries 2023-05-22T16:05:39.364 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.360+0000 7f7f8e9df640 1 -- 172.21.15.16:0/942939513 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f7f740655b0 msgr2=0x7f7f74067a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:39.364 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.360+0000 7f7f8e9df640 1 --2- 172.21.15.16:0/942939513 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f7f740655b0 0x7f7f74067a60 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f7f8815f3b0 tx=0x7f7f78009040 comp rx=0 tx=0).stop 2023-05-22T16:05:39.364 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.360+0000 7f7f8e9df640 1 -- 172.21.15.16:0/942939513 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7f88160590 msgr2=0x7f7f88150810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:39.365 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.360+0000 7f7f8e9df640 1 --2- 172.21.15.16:0/942939513 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7f88160590 0x7f7f88150810 secure :-1 s=READY pgs=2514 cs=0 l=1 rev1=1 crypto rx=0x7f7f8001a4f0 tx=0x7f7f80019660 comp rx=0 tx=0).stop 2023-05-22T16:05:39.365 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.360+0000 7f7f8e9df640 1 -- 172.21.15.16:0/942939513 shutdown_connections 2023-05-22T16:05:39.365 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.360+0000 7f7f8e9df640 1 --2- 172.21.15.16:0/942939513 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f7f740655b0 0x7f7f74067a60 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:39.365 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.360+0000 7f7f8e9df640 1 --2- 172.21.15.16:0/942939513 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7f88160590 0x7f7f88150810 unknown :-1 s=CLOSED pgs=2514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:39.365 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.360+0000 7f7f8e9df640 1 --2- 172.21.15.16:0/942939513 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7f88160110 0x7f7f8814f9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:39.365 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.360+0000 7f7f8e9df640 1 --2- 172.21.15.16:0/942939513 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7f8815fd40 0x7f7f881545e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:39.365 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.360+0000 7f7f8e9df640 1 -- 172.21.15.16:0/942939513 >> 172.21.15.16:0/942939513 conn(0x7f7f880655c0 msgr2=0x7f7f8815e760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:05:39.366 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.360+0000 7f7f8e9df640 1 -- 172.21.15.16:0/942939513 shutdown_connections 2023-05-22T16:05:39.366 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.360+0000 7f7f8e9df640 1 -- 172.21.15.16:0/942939513 wait complete. 2023-05-22T16:05:39.381 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-05-22T16:05:39.382 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2023-05-22T16:05:39.387 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-05-22T16:05:39.549 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.548+0000 7f7ff6fe1640 1 Processor -- start 2023-05-22T16:05:39.550 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.548+0000 7f7ff6fe1640 1 -- start start 2023-05-22T16:05:39.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.548+0000 7f7ff6fe1640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f7ff00cfa20 0x7f7ff00d0020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:39.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.548+0000 7f7ff6fe1640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f7ff00d0560 0x7f7ff00d20c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:39.554 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.548+0000 7f7ff6fe1640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7ff00d2600 0x7f7ff00b23c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:39.554 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.548+0000 7f7ff6fe1640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ff015c4f0 con 0x7f7ff00cfa20 2023-05-22T16:05:39.554 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.548+0000 7f7ff6fe1640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ff00d0990 con 0x7f7ff00d0560 2023-05-22T16:05:39.554 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.548+0000 7f7ff6fe1640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ff00cf190 con 0x7f7ff00d2600 2023-05-22T16:05:39.555 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.548+0000 7f7ff4d7d640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f7ff00cfa20 0x7f7ff00d0020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:39.555 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.548+0000 7f7ff4d7d640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f7ff00cfa20 0x7f7ff00d0020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59284/0 (socket says 172.21.15.16:59284) 2023-05-22T16:05:39.555 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.548+0000 7f7ff4d7d640 1 -- 172.21.15.16:0/423998977 learned_addr learned my addr 172.21.15.16:0/423998977 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:05:39.555 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.548+0000 7f7feffff640 1 --2- 172.21.15.16:0/423998977 >> v2:172.21.15.62:3300/0 conn(0x7f7ff00d0560 0x7f7ff00d20c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:39.555 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.548+0000 7f7ff4d7d640 1 -- 172.21.15.16:0/423998977 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7ff00d2600 msgr2=0x7f7ff00b23c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:05:39.555 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.548+0000 7f7ff557e640 1 --2- 172.21.15.16:0/423998977 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7ff00d2600 0x7f7ff00b23c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:39.555 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.548+0000 7f7ff4d7d640 1 --2- 172.21.15.16:0/423998977 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7ff00d2600 0x7f7ff00b23c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:39.556 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.548+0000 7f7ff4d7d640 1 -- 172.21.15.16:0/423998977 >> v2:172.21.15.62:3300/0 conn(0x7f7ff00d0560 msgr2=0x7f7ff00d20c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:05:39.556 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.548+0000 7f7ff4d7d640 1 --2- 172.21.15.16:0/423998977 >> v2:172.21.15.62:3300/0 conn(0x7f7ff00d0560 0x7f7ff00d20c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:39.556 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.548+0000 7f7ff4d7d640 1 -- 172.21.15.16:0/423998977 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ff00d3360 con 0x7f7ff00cfa20 2023-05-22T16:05:39.556 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.548+0000 7f7feffff640 1 --2- 172.21.15.16:0/423998977 >> v2:172.21.15.62:3300/0 conn(0x7f7ff00d0560 0x7f7ff00d20c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T16:05:39.558 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.552+0000 7f7ff4d7d640 1 --2- 172.21.15.16:0/423998977 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ff00cfa20 0x7f7ff00d0020 secure :-1 s=READY pgs=2515 cs=0 l=1 rev1=1 crypto rx=0x7f7ff00cff40 tx=0x7f7fe801ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c198ecab3768cc0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:39.558 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.552+0000 7f7fef7fe640 1 -- 172.21.15.16:0/423998977 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fe80a89b0 con 0x7f7ff00cfa20 2023-05-22T16:05:39.558 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.552+0000 7f7fef7fe640 1 -- 172.21.15.16:0/423998977 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fe80a8b10 con 0x7f7ff00cfa20 2023-05-22T16:05:39.558 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.552+0000 7f7fef7fe640 1 -- 172.21.15.16:0/423998977 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fe80a8ea0 con 0x7f7ff00cfa20 2023-05-22T16:05:39.559 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.552+0000 7f7ff6fe1640 1 -- 172.21.15.16:0/423998977 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ff00cfa20 msgr2=0x7f7ff00d0020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:39.559 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.552+0000 7f7ff6fe1640 1 --2- 172.21.15.16:0/423998977 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ff00cfa20 0x7f7ff00d0020 secure :-1 s=READY pgs=2515 cs=0 l=1 rev1=1 crypto rx=0x7f7ff00cff40 tx=0x7f7fe801ecf0 comp rx=0 tx=0).stop 2023-05-22T16:05:39.559 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.552+0000 7f7ff6fe1640 1 -- 172.21.15.16:0/423998977 shutdown_connections 2023-05-22T16:05:39.559 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.552+0000 7f7ff6fe1640 1 --2- 172.21.15.16:0/423998977 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7ff00d2600 0x7f7ff00b23c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:39.559 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.552+0000 7f7ff6fe1640 1 --2- 172.21.15.16:0/423998977 >> v2:172.21.15.62:3300/0 conn(0x7f7ff00d0560 0x7f7ff00d20c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:39.559 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.552+0000 7f7ff6fe1640 1 --2- 172.21.15.16:0/423998977 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ff00cfa20 0x7f7ff00d0020 unknown :-1 s=CLOSED pgs=2515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:39.559 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.552+0000 7f7ff6fe1640 1 -- 172.21.15.16:0/423998977 >> 172.21.15.16:0/423998977 conn(0x7f7ff0071850 msgr2=0x7f7ff009b830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:05:39.559 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.552+0000 7f7ff6fe1640 1 -- 172.21.15.16:0/423998977 shutdown_connections 2023-05-22T16:05:39.560 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.552+0000 7f7ff6fe1640 1 -- 172.21.15.16:0/423998977 wait complete. 2023-05-22T16:05:39.560 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.552+0000 7f7ff6fe1640 1 Processor -- start 2023-05-22T16:05:39.560 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.552+0000 7f7ff6fe1640 1 -- start start 2023-05-22T16:05:39.560 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.556+0000 7f7ff6fe1640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ff00cfa20 0x7f7ff015a9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:39.560 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.556+0000 7f7ff6fe1640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7ff00d0560 0x7f7ff00b69b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:39.560 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.556+0000 7f7ff6fe1640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7ff00d2600 0x7f7ff00b23c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:39.560 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.556+0000 7f7ff6fe1640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ff00b5fc0 con 0x7f7ff00cfa20 2023-05-22T16:05:39.561 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.556+0000 7f7ff6fe1640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ff00b53b0 con 0x7f7ff00d2600 2023-05-22T16:05:39.561 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.556+0000 7f7ff6fe1640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ff00b5690 con 0x7f7ff00d0560 2023-05-22T16:05:39.561 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.556+0000 7f7ff4d7d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ff00cfa20 0x7f7ff015a9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:39.561 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.556+0000 7f7ff4d7d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ff00cfa20 0x7f7ff015a9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59290/0 (socket says 172.21.15.16:59290) 2023-05-22T16:05:39.561 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.556+0000 7f7ff4d7d640 1 -- 172.21.15.16:0/2394957183 learned_addr learned my addr 172.21.15.16:0/2394957183 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:05:39.561 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.556+0000 7f7feffff640 1 --2- 172.21.15.16:0/2394957183 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7ff00d0560 0x7f7ff00b69b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:39.561 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.556+0000 7f7ff557e640 1 --2- 172.21.15.16:0/2394957183 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7ff00d2600 0x7f7ff00b23c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:39.561 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.556+0000 7f7ff4d7d640 1 -- 172.21.15.16:0/2394957183 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7ff00d0560 msgr2=0x7f7ff00b69b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:39.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.556+0000 7f7ff4d7d640 1 --2- 172.21.15.16:0/2394957183 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7ff00d0560 0x7f7ff00b69b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:39.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.556+0000 7f7ff4d7d640 1 -- 172.21.15.16:0/2394957183 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7ff00d2600 msgr2=0x7f7ff00b23c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:39.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.556+0000 7f7ff4d7d640 1 --2- 172.21.15.16:0/2394957183 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7ff00d2600 0x7f7ff00b23c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:39.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.556+0000 7f7ff4d7d640 1 -- 172.21.15.16:0/2394957183 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fe8009480 con 0x7f7ff00cfa20 2023-05-22T16:05:39.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.556+0000 7f7ff4d7d640 1 --2- 172.21.15.16:0/2394957183 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ff00cfa20 0x7f7ff015a9f0 secure :-1 s=READY pgs=2516 cs=0 l=1 rev1=1 crypto rx=0x7f7fe801f590 tx=0x7f7fe801ba90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:39.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.556+0000 7f7fed7fa640 1 -- 172.21.15.16:0/2394957183 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fe8005db0 con 0x7f7ff00cfa20 2023-05-22T16:05:39.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.556+0000 7f7fed7fa640 1 -- 172.21.15.16:0/2394957183 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fe8003620 con 0x7f7ff00cfa20 2023-05-22T16:05:39.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.556+0000 7f7ff6fe1640 1 -- 172.21.15.16:0/2394957183 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ff00d3080 con 0x7f7ff00cfa20 2023-05-22T16:05:39.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.556+0000 7f7fed7fa640 1 -- 172.21.15.16:0/2394957183 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fe80039d0 con 0x7f7ff00cfa20 2023-05-22T16:05:39.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.556+0000 7f7ff6fe1640 1 -- 172.21.15.16:0/2394957183 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ff00d2da0 con 0x7f7ff00cfa20 2023-05-22T16:05:39.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.556+0000 7f7ff6fe1640 1 -- 172.21.15.16:0/2394957183 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ff00ccb40 con 0x7f7ff00cfa20 2023-05-22T16:05:39.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.556+0000 7f7fed7fa640 1 -- 172.21.15.16:0/2394957183 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 96) v1 ==== 84406+0+0 (secure 0 0 0) 0x7f7fe80b7070 con 0x7f7ff00cfa20 2023-05-22T16:05:39.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.556+0000 7f7fed7fa640 1 --2- 172.21.15.16:0/2394957183 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f7fd40655b0 0x7f7fd4067a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:39.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.556+0000 7f7feffff640 1 --2- 172.21.15.16:0/2394957183 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f7fd40655b0 0x7f7fd4067a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:39.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.556+0000 7f7fed7fa640 1 -- 172.21.15.16:0/2394957183 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(197..197 src has 1..197) v4 ==== 6782+0+0 (secure 0 0 0) 0x7f7fe811fcc0 con 0x7f7ff00cfa20 2023-05-22T16:05:39.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.560+0000 7f7feffff640 1 --2- 172.21.15.16:0/2394957183 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f7fd40655b0 0x7f7fd4067a60 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f7ff00a0370 tx=0x7f7fd8008460 comp rx=0 tx=0).ready entity=mgr.10974 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:39.565 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.560+0000 7f7fed7fa640 1 -- 172.21.15.16:0/2394957183 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f7fe80ed480 con 0x7f7ff00cfa20 2023-05-22T16:05:39.722 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.720+0000 7f7ff6fe1640 1 -- 172.21.15.16:0/2394957183 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7ff0073830 con 0x7f7ff00cfa20 2023-05-22T16:05:39.724 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.724+0000 7f7fed7fa640 1 -- 172.21.15.16:0/2394957183 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v197) v1 ==== 74+0+18629 (secure 0 0 0) 0x7f7fe80f0aa0 con 0x7f7ff00cfa20 2023-05-22T16:05:39.725 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T16:05:39.725 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":197,"fsid":"b2719834-7847-4d98-90a6-f65d57f85c49","created":"2023-05-22T15:26:25.259314+0000","modified":"2023-05-22T16:05:35.647639+0000","last_up_change":"2023-05-22T15:26:36.448524+0000","last_in_change":"2023-05-22T15:26:27.675668+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":25,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-05-22T15:26:38.019781+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-05-22T15:26:43.570514+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_metadata","create_time":"2023-05-22T16:05:20.485167+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"195","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":25,"pool_name":"cephfs_data","create_time":"2023-05-22T16:05:20.900001+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"196","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9e7ff854-b25f-4cbb-a5c2-1799d9c53d2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":37021},{"type":"v1","addr":"172.21.15.16:6825","nonce":37021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":37021},{"type":"v1","addr":"172.21.15.16:6827","nonce":37021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":37021},{"type":"v1","addr":"172.21.15.16:6831","nonce":37021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":37021},{"type":"v1","addr":"172.21.15.16:6829","nonce":37021}]},"public_addr":"172.21.15.16:6825/37021","cluster_addr":"172.21.15.16:6827/37021","heartbeat_back_addr":"172.21.15.16:6831/37021","heartbeat_front_addr":"172.21.15.16:6829/37021","state":["exists","up"]},{"osd":1,"uuid":"4702fbc9-a927-4565-aca5-41dc69cc8264","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":37008},{"type":"v1","addr":"172.21.15.16:6801","nonce":37008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":37008},{"type":"v1","addr":"172.21.15.16:6803","nonce":37008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":37008},{"type":"v1","addr":"172.21.15.16:6807","nonce":37008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":37008},{"type":"v1","addr":"172.21.15.16:6805","nonce":37008}]},"public_addr":"172.21.15.16:6801/37008","cluster_addr":"172.21.15.16:6803/37008","heartbeat_back_addr":"172.21.15.16:6807/37008","heartbeat_front_addr":"172.21.15.16:6805/37008","state":["exists","up"]},{"osd":2,"uuid":"1e7bba46-4eba-47ef-bb81-5e2d371638ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":37009},{"type":"v1","addr":"172.21.15.16:6809","nonce":37009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":37009},{"type":"v1","addr":"172.21.15.16:6811","nonce":37009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":37009},{"type":"v1","addr":"172.21.15.16:6815","nonce":37009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":37009},{"type":"v1","addr":"172.21.15.16:6813","nonce":37009}]},"public_addr":"172.21.15.16:6809/37009","cluster_addr":"172.21.15.16:6811/37009","heartbeat_back_addr":"172.21.15.16:6815/37009","heartbeat_front_addr":"172.21.15.16:6813/37009","state":["exists","up"]},{"osd":3,"uuid":"383cacc0-5a88-4904-990e-498ca59ef48a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":37020},{"type":"v1","addr":"172.21.15.16:6817","nonce":37020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":37020},{"type":"v1","addr":"172.21.15.16:6819","nonce":37020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":37020},{"type":"v1","addr":"172.21.15.16:6823","nonce":37020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":37020},{"type":"v1","addr":"172.21.15.16:6821","nonce":37020}]},"public_addr":"172.21.15.16:6817/37020","cluster_addr":"172.21.15.16:6819/37020","heartbeat_back_addr":"172.21.15.16:6823/37020","heartbeat_front_addr":"172.21.15.16:6821/37020","state":["exists","up"]},{"osd":4,"uuid":"970500ef-95b7-42c1-b080-eed4fae7013d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":37353},{"type":"v1","addr":"172.21.15.62:6817","nonce":37353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6819","nonce":37353},{"type":"v1","addr":"172.21.15.62:6822","nonce":37353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37353},{"type":"v1","addr":"172.21.15.62:6831","nonce":37353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37353},{"type":"v1","addr":"172.21.15.62:6826","nonce":37353}]},"public_addr":"172.21.15.62:6817/37353","cluster_addr":"172.21.15.62:6822/37353","heartbeat_back_addr":"172.21.15.62:6831/37353","heartbeat_front_addr":"172.21.15.62:6826/37353","state":["exists","up"]},{"osd":5,"uuid":"022d67db-d4d8-445b-b1b9-8b446a00f5c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":37352},{"type":"v1","addr":"172.21.15.62:6801","nonce":37352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":37352},{"type":"v1","addr":"172.21.15.62:6803","nonce":37352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":37352},{"type":"v1","addr":"172.21.15.62:6807","nonce":37352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":37352},{"type":"v1","addr":"172.21.15.62:6805","nonce":37352}]},"public_addr":"172.21.15.62:6801/37352","cluster_addr":"172.21.15.62:6803/37352","heartbeat_back_addr":"172.21.15.62:6807/37352","heartbeat_front_addr":"172.21.15.62:6805/37352","state":["exists","up"]},{"osd":6,"uuid":"42bd372b-6688-4a1b-8407-e16e8c266e33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":37354},{"type":"v1","addr":"172.21.15.62:6820","nonce":37354}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":37354},{"type":"v1","addr":"172.21.15.62:6823","nonce":37354}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6829","nonce":37354},{"type":"v1","addr":"172.21.15.62:6830","nonce":37354}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6825","nonce":37354},{"type":"v1","addr":"172.21.15.62:6827","nonce":37354}]},"public_addr":"172.21.15.62:6820/37354","cluster_addr":"172.21.15.62:6823/37354","heartbeat_back_addr":"172.21.15.62:6830/37354","heartbeat_front_addr":"172.21.15.62:6827/37354","state":["exists","up"]},{"osd":7,"uuid":"ac70b2a3-f41b-4bf2-b227-7f2cbb55426d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37355},{"type":"v1","addr":"172.21.15.62:6809","nonce":37355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37355},{"type":"v1","addr":"172.21.15.62:6811","nonce":37355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":37355},{"type":"v1","addr":"172.21.15.62:6815","nonce":37355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":37355},{"type":"v1","addr":"172.21.15.62:6813","nonce":37355}]},"public_addr":"172.21.15.62:6809/37355","cluster_addr":"172.21.15.62:6811/37355","heartbeat_back_addr":"172.21.15.62:6815/37355","heartbeat_front_addr":"172.21.15.62:6813/37355","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.368896+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.107273+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.276888+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.287502+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.124791+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.251937+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.255679+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.232797+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:0/1961044213":"2023-05-23T16:05:35.647562+0000","172.21.15.16:0/238292614":"2023-05-23T16:05:35.647562+0000","172.21.15.16:6833/65890":"2023-05-23T16:05:35.647562+0000","172.21.15.16:0/1148602276":"2023-05-23T16:05:35.647562+0000","172.21.15.16:6832/65890":"2023-05-23T16:05:35.647562+0000","172.21.15.16:0/456460873":"2023-05-23T16:05:35.647562+0000","172.21.15.16:0/2766506027":"2023-05-23T16:05:35.647562+0000","172.21.15.16:0/2214029832":"2023-05-23T16:05:35.647562+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":17,"snaps":[{"begin":2,"length":2}]},{"pool":21,"snaps":[{"begin":25,"length":9}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-05-22T16:05:39.727 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.724+0000 7f7ff6fe1640 1 -- 172.21.15.16:0/2394957183 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f7fd40655b0 msgr2=0x7f7fd4067a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:39.727 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.724+0000 7f7ff6fe1640 1 --2- 172.21.15.16:0/2394957183 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f7fd40655b0 0x7f7fd4067a60 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f7ff00a0370 tx=0x7f7fd8008460 comp rx=0 tx=0).stop 2023-05-22T16:05:39.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.724+0000 7f7ff6fe1640 1 -- 172.21.15.16:0/2394957183 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ff00cfa20 msgr2=0x7f7ff015a9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:39.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.724+0000 7f7ff6fe1640 1 --2- 172.21.15.16:0/2394957183 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ff00cfa20 0x7f7ff015a9f0 secure :-1 s=READY pgs=2516 cs=0 l=1 rev1=1 crypto rx=0x7f7fe801f590 tx=0x7f7fe801ba90 comp rx=0 tx=0).stop 2023-05-22T16:05:39.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.724+0000 7f7ff6fe1640 1 -- 172.21.15.16:0/2394957183 shutdown_connections 2023-05-22T16:05:39.729 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.724+0000 7f7ff6fe1640 1 --2- 172.21.15.16:0/2394957183 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7ff00d2600 0x7f7ff00b23c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:39.729 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.724+0000 7f7ff6fe1640 1 --2- 172.21.15.16:0/2394957183 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7ff00d0560 0x7f7ff00b69b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:39.729 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.724+0000 7f7ff6fe1640 1 --2- 172.21.15.16:0/2394957183 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f7fd40655b0 0x7f7fd4067a60 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:39.729 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.724+0000 7f7ff6fe1640 1 --2- 172.21.15.16:0/2394957183 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ff00cfa20 0x7f7ff015a9f0 unknown :-1 s=CLOSED pgs=2516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:39.729 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.724+0000 7f7ff6fe1640 1 -- 172.21.15.16:0/2394957183 >> 172.21.15.16:0/2394957183 conn(0x7f7ff0071850 msgr2=0x7f7ff015b9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:05:39.729 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.724+0000 7f7ff6fe1640 1 -- 172.21.15.16:0/2394957183 shutdown_connections 2023-05-22T16:05:39.729 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.724+0000 7f7ff6fe1640 1 -- 172.21.15.16:0/2394957183 wait complete. 2023-05-22T16:05:39.744 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-05-22T15:26:38.019781+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-05-22T15:26:43.570514+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 24, 'pool_name': 'cephfs_metadata', 'create_time': '2023-05-22T16:05:20.485167+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '195', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 25, 'pool_name': 'cephfs_data', 'create_time': '2023-05-22T16:05:20.900001+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '196', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-05-22T16:05:39.745 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-05-22T16:05:39.865 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.864+0000 7f9bfd26d640 1 Processor -- start 2023-05-22T16:05:39.865 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.864+0000 7f9bfd26d640 1 -- start start 2023-05-22T16:05:39.868 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.864+0000 7f9bfd26d640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9bf80ca880 0x7f9bf80ce510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:39.868 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.864+0000 7f9bfd26d640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9bf80cb7a0 con 0x7f9bf815d3d0 2023-05-22T16:05:39.868 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.864+0000 7f9bfd26d640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9bf80cc600 con 0x7f9bf80ace10 2023-05-22T16:05:39.868 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.864+0000 7f9bfd26d640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9bf80afa60 con 0x7f9bf80ca880 2023-05-22T16:05:39.869 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.864+0000 7f9bf6ffd640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f9bf815d3d0 0x7f9bf80cde00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:54254/0 (socket says 172.21.15.16:54254) 2023-05-22T16:05:39.869 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.864+0000 7f9bf6ffd640 1 -- 172.21.15.16:0/1417282142 learned_addr learned my addr 172.21.15.16:0/1417282142 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T16:05:39.869 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.864+0000 7f9bf67fc640 1 --2- 172.21.15.16:0/1417282142 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9bf80ca880 0x7f9bf80ce510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:39.869 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.864+0000 7f9bf5ffb640 1 -- 172.21.15.16:0/1417282142 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3811078888 0 0) 0x7f9bf80cb7a0 con 0x7f9bf815d3d0 2023-05-22T16:05:39.869 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.864+0000 7f9bf5ffb640 1 -- 172.21.15.16:0/1417282142 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9be4003530 con 0x7f9bf815d3d0 2023-05-22T16:05:39.869 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.864+0000 7f9bf5ffb640 1 -- 172.21.15.16:0/1417282142 <== mon.1 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2468040474 0 0) 0x7f9bf80cc600 con 0x7f9bf80ace10 2023-05-22T16:05:39.870 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.864+0000 7f9bf5ffb640 1 -- 172.21.15.16:0/1417282142 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9bf80cb7a0 con 0x7f9bf80ace10 2023-05-22T16:05:39.870 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.864+0000 7f9bf5ffb640 1 -- 172.21.15.16:0/1417282142 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1408968353 0 0) 0x7f9be4003530 con 0x7f9bf815d3d0 2023-05-22T16:05:39.870 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.864+0000 7f9bf5ffb640 1 -- 172.21.15.16:0/1417282142 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9bf80ca880 msgr2=0x7f9bf80ce510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:05:39.870 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.864+0000 7f9bf5ffb640 1 --2- 172.21.15.16:0/1417282142 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9bf80ca880 0x7f9bf80ce510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:39.870 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.864+0000 7f9bf5ffb640 1 -- 172.21.15.16:0/1417282142 >> v1:172.21.15.62:6789/0 conn(0x7f9bf80ace10 legacy=0x7f9bf80ad290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:39.870 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.864+0000 7f9bf5ffb640 1 -- 172.21.15.16:0/1417282142 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bf80cca90 con 0x7f9bf815d3d0 2023-05-22T16:05:39.870 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.864+0000 7f9bf5ffb640 1 -- 172.21.15.16:0/1417282142 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7f9be0001800 con 0x7f9bf815d3d0 2023-05-22T16:05:39.870 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.864+0000 7f9bf5ffb640 1 -- 172.21.15.16:0/1417282142 >> v1:172.21.15.16:6789/0 conn(0x7f9bf815d3d0 legacy=0x7f9bf80cde00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:39.871 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.864+0000 7f9bf5ffb640 1 --2- 172.21.15.16:0/1417282142 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9bf80ca880 0x7f9be4003b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:39.871 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.864+0000 7f9bf5ffb640 1 --2- 172.21.15.16:0/1417282142 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9be40040a0 0x7f9be40047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:39.871 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.864+0000 7f9bf5ffb640 1 --2- 172.21.15.16:0/1417282142 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9be4004d10 0x7f9be4005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:39.871 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.864+0000 7f9bf5ffb640 1 -- 172.21.15.16:0/1417282142 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9be400ba40 con 0x7f9be4004d10 2023-05-22T16:05:39.871 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.864+0000 7f9bf5ffb640 1 -- 172.21.15.16:0/1417282142 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9be400bb80 con 0x7f9be40040a0 2023-05-22T16:05:39.871 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.864+0000 7f9bf5ffb640 1 -- 172.21.15.16:0/1417282142 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9be400bd40 con 0x7f9bf80ca880 2023-05-22T16:05:39.871 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.864+0000 7f9bf77fe640 1 --2- 172.21.15.16:0/1417282142 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9be4004d10 0x7f9be4005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:39.872 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.864+0000 7f9bf67fc640 1 --2- 172.21.15.16:0/1417282142 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9be40040a0 0x7f9be40047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:39.872 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.864+0000 7f9bf6ffd640 1 --2- 172.21.15.16:0/1417282142 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9bf80ca880 0x7f9be4003b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:39.872 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.864+0000 7f9bf77fe640 1 -- 172.21.15.16:0/1417282142 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9bf80ca880 msgr2=0x7f9be4003b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:05:39.872 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.864+0000 7f9bf77fe640 1 --2- 172.21.15.16:0/1417282142 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9bf80ca880 0x7f9be4003b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:39.872 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.864+0000 7f9bf77fe640 1 -- 172.21.15.16:0/1417282142 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9be40040a0 msgr2=0x7f9be40047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:05:39.873 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.864+0000 7f9bf77fe640 1 --2- 172.21.15.16:0/1417282142 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9be40040a0 0x7f9be40047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:39.873 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.864+0000 7f9bf77fe640 1 -- 172.21.15.16:0/1417282142 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9be4003530 con 0x7f9be4004d10 2023-05-22T16:05:39.873 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.864+0000 7f9bf67fc640 1 --2- 172.21.15.16:0/1417282142 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9be40040a0 0x7f9be40047d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T16:05:39.877 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.868+0000 7f9bf77fe640 1 --2- 172.21.15.16:0/1417282142 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9be4004d10 0x7f9be4005180 secure :-1 s=READY pgs=2518 cs=0 l=1 rev1=1 crypto rx=0x7f9bf0007fd0 tx=0x7f9bf001c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24e2a3a8a3433157 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:39.878 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.868+0000 7f9bf5ffb640 1 -- 172.21.15.16:0/1417282142 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bf0003b80 con 0x7f9be4004d10 2023-05-22T16:05:39.878 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.868+0000 7f9bf5ffb640 1 -- 172.21.15.16:0/1417282142 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9bf0003d00 con 0x7f9be4004d10 2023-05-22T16:05:39.878 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.868+0000 7f9bf5ffb640 1 -- 172.21.15.16:0/1417282142 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bf0005310 con 0x7f9be4004d10 2023-05-22T16:05:39.879 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.868+0000 7f9bfd26d640 1 -- 172.21.15.16:0/1417282142 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9be4004d10 msgr2=0x7f9be4005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:39.879 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.868+0000 7f9bfd26d640 1 --2- 172.21.15.16:0/1417282142 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9be4004d10 0x7f9be4005180 secure :-1 s=READY pgs=2518 cs=0 l=1 rev1=1 crypto rx=0x7f9bf0007fd0 tx=0x7f9bf001c8c0 comp rx=0 tx=0).stop 2023-05-22T16:05:39.879 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.868+0000 7f9bf77fe640 1 -- 172.21.15.16:0/1417282142 reap_dead start 2023-05-22T16:05:39.879 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.868+0000 7f9bfd26d640 1 -- 172.21.15.16:0/1417282142 shutdown_connections 2023-05-22T16:05:39.880 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.868+0000 7f9bfd26d640 1 -- 172.21.15.16:0/1417282142 >> 172.21.15.16:0/1417282142 conn(0x7f9bf8099620 msgr2=0x7f9bf80d02c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:05:39.880 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.868+0000 7f9bfd26d640 1 -- 172.21.15.16:0/1417282142 shutdown_connections 2023-05-22T16:05:39.880 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.868+0000 7f9bfd26d640 1 -- 172.21.15.16:0/1417282142 wait complete. 2023-05-22T16:05:39.880 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.868+0000 7f9bfd26d640 1 Processor -- start 2023-05-22T16:05:39.881 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.868+0000 7f9bfd26d640 1 -- start start 2023-05-22T16:05:39.881 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.868+0000 7f9bfd26d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9bf806db50 0x7f9bf80c64d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:39.881 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.868+0000 7f9bfd26d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9bf80c9660 0x7f9bf8153e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:39.881 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.868+0000 7f9bfd26d640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9bf806df20 0x7f9bf81543d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:39.882 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.868+0000 7f9bfd26d640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bf80c9af0 con 0x7f9bf806db50 2023-05-22T16:05:39.882 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.872+0000 7f9bfd26d640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bf80c9dd0 con 0x7f9bf80c9660 2023-05-22T16:05:39.882 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.872+0000 7f9bfd26d640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9bf80ca0b0 con 0x7f9bf806df20 2023-05-22T16:05:39.883 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.872+0000 7f9bf6ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9bf806db50 0x7f9bf80c64d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:39.883 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.872+0000 7f9bf6ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9bf806db50 0x7f9bf80c64d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59306/0 (socket says 172.21.15.16:59306) 2023-05-22T16:05:39.883 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.872+0000 7f9bf6ffd640 1 -- 172.21.15.16:0/4275905403 learned_addr learned my addr 172.21.15.16:0/4275905403 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:05:39.883 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.872+0000 7f9bf77fe640 1 --2- 172.21.15.16:0/4275905403 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9bf806df20 0x7f9bf81543d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:39.883 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.872+0000 7f9bf67fc640 1 --2- 172.21.15.16:0/4275905403 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9bf80c9660 0x7f9bf8153e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:39.884 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.872+0000 7f9bf6ffd640 1 -- 172.21.15.16:0/4275905403 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9bf806df20 msgr2=0x7f9bf81543d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:39.884 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.872+0000 7f9bf6ffd640 1 --2- 172.21.15.16:0/4275905403 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9bf806df20 0x7f9bf81543d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:39.884 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.872+0000 7f9bf6ffd640 1 -- 172.21.15.16:0/4275905403 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9bf80c9660 msgr2=0x7f9bf8153e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:39.884 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.872+0000 7f9bf6ffd640 1 --2- 172.21.15.16:0/4275905403 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9bf80c9660 0x7f9bf8153e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:39.885 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.872+0000 7f9bf6ffd640 1 -- 172.21.15.16:0/4275905403 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bf80cbcd0 con 0x7f9bf806db50 2023-05-22T16:05:39.885 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.872+0000 7f9bf6ffd640 1 --2- 172.21.15.16:0/4275905403 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9bf806db50 0x7f9bf80c64d0 secure :-1 s=READY pgs=2519 cs=0 l=1 rev1=1 crypto rx=0x7f9be0001ec0 tx=0x7f9be000e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:39.885 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.872+0000 7f9bef7fe640 1 -- 172.21.15.16:0/4275905403 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9be0014600 con 0x7f9bf806db50 2023-05-22T16:05:39.886 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.872+0000 7f9bef7fe640 1 -- 172.21.15.16:0/4275905403 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9be000d030 con 0x7f9bf806db50 2023-05-22T16:05:39.886 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.872+0000 7f9bfd26d640 1 -- 172.21.15.16:0/4275905403 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9bf80cb9f0 con 0x7f9bf806db50 2023-05-22T16:05:39.886 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.872+0000 7f9bef7fe640 1 -- 172.21.15.16:0/4275905403 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9be00149c0 con 0x7f9bf806db50 2023-05-22T16:05:39.886 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.872+0000 7f9bfd26d640 1 -- 172.21.15.16:0/4275905403 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9bf80cbfb0 con 0x7f9bf806db50 2023-05-22T16:05:39.887 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.872+0000 7f9bfd26d640 1 -- 172.21.15.16:0/4275905403 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9bf80af6d0 con 0x7f9bf806db50 2023-05-22T16:05:39.887 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.872+0000 7f9bef7fe640 1 -- 172.21.15.16:0/4275905403 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 96) v1 ==== 84406+0+0 (secure 0 0 0) 0x7f9be0014b20 con 0x7f9bf806db50 2023-05-22T16:05:39.887 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.872+0000 7f9bef7fe640 1 --2- 172.21.15.16:0/4275905403 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f9bd8065580 0x7f9bd8067a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:39.887 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.872+0000 7f9bef7fe640 1 -- 172.21.15.16:0/4275905403 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(197..197 src has 1..197) v4 ==== 6782+0+0 (secure 0 0 0) 0x7f9be0085c80 con 0x7f9bf806db50 2023-05-22T16:05:39.888 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.872+0000 7f9bf67fc640 1 --2- 172.21.15.16:0/4275905403 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f9bd8065580 0x7f9bd8067a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:39.888 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.872+0000 7f9bf67fc640 1 --2- 172.21.15.16:0/4275905403 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f9bd8065580 0x7f9bd8067a30 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f9bf806f890 tx=0x7f9be8008040 comp rx=0 tx=0).ready entity=mgr.10974 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:39.889 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:39.876+0000 7f9bef7fe640 1 -- 172.21.15.16:0/4275905403 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f9be0053170 con 0x7f9bf806db50 2023-05-22T16:05:40.029 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.028+0000 7f9bfd26d640 1 -- 172.21.15.16:0/4275905403 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f9bf80c6f80 con 0x7f9bf806db50 2023-05-22T16:05:40.030 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.028+0000 7f9bef7fe640 1 -- 172.21.15.16:0/4275905403 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v197) v1 ==== 93+0+10 (secure 0 0 0) 0x7f9be0018030 con 0x7f9bf806db50 2023-05-22T16:05:40.030 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2023-05-22T16:05:40.032 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.028+0000 7f9bfd26d640 1 -- 172.21.15.16:0/4275905403 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f9bd8065580 msgr2=0x7f9bd8067a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:40.033 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.028+0000 7f9bfd26d640 1 --2- 172.21.15.16:0/4275905403 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f9bd8065580 0x7f9bd8067a30 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f9bf806f890 tx=0x7f9be8008040 comp rx=0 tx=0).stop 2023-05-22T16:05:40.033 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.028+0000 7f9bfd26d640 1 -- 172.21.15.16:0/4275905403 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9bf806db50 msgr2=0x7f9bf80c64d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:40.033 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.028+0000 7f9bfd26d640 1 --2- 172.21.15.16:0/4275905403 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9bf806db50 0x7f9bf80c64d0 secure :-1 s=READY pgs=2519 cs=0 l=1 rev1=1 crypto rx=0x7f9be0001ec0 tx=0x7f9be000e530 comp rx=0 tx=0).stop 2023-05-22T16:05:40.033 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.028+0000 7f9bfd26d640 1 -- 172.21.15.16:0/4275905403 shutdown_connections 2023-05-22T16:05:40.034 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.028+0000 7f9bfd26d640 1 --2- 172.21.15.16:0/4275905403 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9bf806df20 0x7f9bf81543d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:40.034 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.028+0000 7f9bfd26d640 1 --2- 172.21.15.16:0/4275905403 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9bf80c9660 0x7f9bf8153e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:40.034 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.028+0000 7f9bfd26d640 1 --2- 172.21.15.16:0/4275905403 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f9bd8065580 0x7f9bd8067a30 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:40.034 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.028+0000 7f9bfd26d640 1 --2- 172.21.15.16:0/4275905403 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9bf806db50 0x7f9bf80c64d0 unknown :-1 s=CLOSED pgs=2519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:40.035 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.028+0000 7f9bfd26d640 1 -- 172.21.15.16:0/4275905403 >> 172.21.15.16:0/4275905403 conn(0x7f9bf8099620 msgr2=0x7f9bf80cd5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:05:40.035 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.028+0000 7f9bfd26d640 1 -- 172.21.15.16:0/4275905403 shutdown_connections 2023-05-22T16:05:40.035 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.028+0000 7f9bfd26d640 1 -- 172.21.15.16:0/4275905403 wait complete. 2023-05-22T16:05:40.048 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-05-22T16:05:40.169 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.168+0000 7fc4be442640 1 Processor -- start 2023-05-22T16:05:40.169 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.168+0000 7fc4be442640 1 -- start start 2023-05-22T16:05:40.170 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.168+0000 7fc4be442640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc4b8069800 0x7fc4b815cf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:40.170 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.168+0000 7fc4be442640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4b815ca90 con 0x7fc4b8070a50 2023-05-22T16:05:40.170 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.168+0000 7fc4be442640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4b8069c30 con 0x7fc4b8071360 2023-05-22T16:05:40.170 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.168+0000 7fc4be442640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4b8069ec0 con 0x7fc4b8069800 2023-05-22T16:05:40.171 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.168+0000 7fc4bc9df640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fc4b8070a50 0x7fc4b806e090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:54260/0 (socket says 172.21.15.16:54260) 2023-05-22T16:05:40.171 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.168+0000 7fc4bc9df640 1 -- 172.21.15.16:0/2506529469 learned_addr learned my addr 172.21.15.16:0/2506529469 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T16:05:40.171 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.168+0000 7fc4affff640 1 --2- 172.21.15.16:0/2506529469 >> v2:172.21.15.16:3300/0 conn(0x7fc4b8069800 0x7fc4b815cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:40.171 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.168+0000 7fc4b77fe640 1 -- 172.21.15.16:0/2506529469 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2837217098 0 0) 0x7fc4b815ca90 con 0x7fc4b8070a50 2023-05-22T16:05:40.172 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.168+0000 7fc4b77fe640 1 -- 172.21.15.16:0/2506529469 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc4a4003530 con 0x7fc4b8070a50 2023-05-22T16:05:40.172 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.168+0000 7fc4affff640 1 -- 172.21.15.16:0/2506529469 >> v1:172.21.15.62:6789/0 conn(0x7fc4b8071360 legacy=0x7fc4b8071730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:40.172 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.168+0000 7fc4affff640 1 -- 172.21.15.16:0/2506529469 >> v1:172.21.15.16:6789/0 conn(0x7fc4b8070a50 legacy=0x7fc4b806e090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:40.172 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.168+0000 7fc4affff640 1 -- 172.21.15.16:0/2506529469 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4b806a1a0 con 0x7fc4b8069800 2023-05-22T16:05:40.173 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.172+0000 7fc4affff640 1 --2- 172.21.15.16:0/2506529469 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4b8069800 0x7fc4b815cf20 secure :-1 s=READY pgs=2521 cs=0 l=1 rev1=1 crypto rx=0x7fc4a00097c0 tx=0x7fc4a001c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=139b172607ad78bf server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:40.173 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.172+0000 7fc4b77fe640 1 -- 172.21.15.16:0/2506529469 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4a00ab670 con 0x7fc4b8069800 2023-05-22T16:05:40.173 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.172+0000 7fc4b77fe640 1 -- 172.21.15.16:0/2506529469 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4a00ab7d0 con 0x7fc4b8069800 2023-05-22T16:05:40.173 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.172+0000 7fc4b77fe640 1 -- 172.21.15.16:0/2506529469 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4a00abb60 con 0x7fc4b8069800 2023-05-22T16:05:40.174 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.172+0000 7fc4be442640 1 -- 172.21.15.16:0/2506529469 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4b8069800 msgr2=0x7fc4b815cf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:40.174 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.172+0000 7fc4be442640 1 --2- 172.21.15.16:0/2506529469 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4b8069800 0x7fc4b815cf20 secure :-1 s=READY pgs=2521 cs=0 l=1 rev1=1 crypto rx=0x7fc4a00097c0 tx=0x7fc4a001c450 comp rx=0 tx=0).stop 2023-05-22T16:05:40.176 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.172+0000 7fc4be442640 1 -- 172.21.15.16:0/2506529469 shutdown_connections 2023-05-22T16:05:40.176 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.172+0000 7fc4be442640 1 --2- 172.21.15.16:0/2506529469 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4b8069800 0x7fc4b815cf20 unknown :-1 s=CLOSED pgs=2521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:40.176 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.172+0000 7fc4be442640 1 -- 172.21.15.16:0/2506529469 >> 172.21.15.16:0/2506529469 conn(0x7fc4b806d5b0 msgr2=0x7fc4b80b0de0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:05:40.176 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.172+0000 7fc4be442640 1 -- 172.21.15.16:0/2506529469 shutdown_connections 2023-05-22T16:05:40.177 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.172+0000 7fc4be442640 1 -- 172.21.15.16:0/2506529469 wait complete. 2023-05-22T16:05:40.178 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.172+0000 7fc4be442640 1 Processor -- start 2023-05-22T16:05:40.178 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.172+0000 7fc4be442640 1 -- start start 2023-05-22T16:05:40.178 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.172+0000 7fc4be442640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4b8069800 0x7fc4b8153350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:40.179 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.172+0000 7fc4be442640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc4b8070a50 0x7fc4b80c4b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:40.179 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.172+0000 7fc4be442640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc4b8071360 0x7fc4b80c50a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:40.179 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.172+0000 7fc4be442640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4b80b4be0 con 0x7fc4b8069800 2023-05-22T16:05:40.179 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.172+0000 7fc4be442640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4b8153890 con 0x7fc4b8070a50 2023-05-22T16:05:40.179 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.172+0000 7fc4be442640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4b815adb0 con 0x7fc4b8071360 2023-05-22T16:05:40.179 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.172+0000 7fc4b7fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4b8069800 0x7fc4b8153350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:40.180 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.172+0000 7fc4b7fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4b8069800 0x7fc4b8153350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59324/0 (socket says 172.21.15.16:59324) 2023-05-22T16:05:40.180 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.172+0000 7fc4b7fff640 1 -- 172.21.15.16:0/3650320387 learned_addr learned my addr 172.21.15.16:0/3650320387 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:05:40.180 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.172+0000 7fc4affff640 1 --2- 172.21.15.16:0/3650320387 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc4b8070a50 0x7fc4b80c4b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:40.181 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.172+0000 7fc4bc9df640 1 --2- 172.21.15.16:0/3650320387 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc4b8071360 0x7fc4b80c50a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:40.181 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.172+0000 7fc4b7fff640 1 -- 172.21.15.16:0/3650320387 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc4b8071360 msgr2=0x7fc4b80c50a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:40.181 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.172+0000 7fc4b7fff640 1 --2- 172.21.15.16:0/3650320387 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc4b8071360 0x7fc4b80c50a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:40.181 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.172+0000 7fc4b7fff640 1 -- 172.21.15.16:0/3650320387 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc4b8070a50 msgr2=0x7fc4b80c4b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:40.181 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.172+0000 7fc4b7fff640 1 --2- 172.21.15.16:0/3650320387 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc4b8070a50 0x7fc4b80c4b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:40.181 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.172+0000 7fc4b7fff640 1 -- 172.21.15.16:0/3650320387 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4a0009420 con 0x7fc4b8069800 2023-05-22T16:05:40.181 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.172+0000 7fc4affff640 1 --2- 172.21.15.16:0/3650320387 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc4b8070a50 0x7fc4b80c4b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T16:05:40.181 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.176+0000 7fc4b7fff640 1 --2- 172.21.15.16:0/3650320387 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4b8069800 0x7fc4b8153350 secure :-1 s=READY pgs=2522 cs=0 l=1 rev1=1 crypto rx=0x7fc4a800c530 tx=0x7fc4a800c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:40.182 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.176+0000 7fc4b57fa640 1 -- 172.21.15.16:0/3650320387 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4a8005220 con 0x7fc4b8069800 2023-05-22T16:05:40.182 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.176+0000 7fc4b57fa640 1 -- 172.21.15.16:0/3650320387 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4a8005380 con 0x7fc4b8069800 2023-05-22T16:05:40.182 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.176+0000 7fc4b57fa640 1 -- 172.21.15.16:0/3650320387 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4a8007270 con 0x7fc4b8069800 2023-05-22T16:05:40.182 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.176+0000 7fc4be442640 1 -- 172.21.15.16:0/3650320387 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4b815c350 con 0x7fc4b8069800 2023-05-22T16:05:40.182 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.176+0000 7fc4be442640 1 -- 172.21.15.16:0/3650320387 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4b806ebb0 con 0x7fc4b8069800 2023-05-22T16:05:40.182 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.176+0000 7fc4be442640 1 -- 172.21.15.16:0/3650320387 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4b815c4e0 con 0x7fc4b8069800 2023-05-22T16:05:40.182 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.176+0000 7fc4b57fa640 1 -- 172.21.15.16:0/3650320387 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 96) v1 ==== 84406+0+0 (secure 0 0 0) 0x7fc4a80073d0 con 0x7fc4b8069800 2023-05-22T16:05:40.182 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.176+0000 7fc4b57fa640 1 --2- 172.21.15.16:0/3650320387 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fc498065580 0x7fc498067a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:40.183 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.176+0000 7fc4affff640 1 --2- 172.21.15.16:0/3650320387 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fc498065580 0x7fc498067a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:40.183 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.176+0000 7fc4b57fa640 1 -- 172.21.15.16:0/3650320387 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(197..197 src has 1..197) v4 ==== 6782+0+0 (secure 0 0 0) 0x7fc4a80833c0 con 0x7fc4b8069800 2023-05-22T16:05:40.183 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.176+0000 7fc4affff640 1 --2- 172.21.15.16:0/3650320387 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fc498065580 0x7fc498067a30 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fc4a00af000 tx=0x7fc4a0008040 comp rx=0 tx=0).ready entity=mgr.10974 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:40.184 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.180+0000 7fc4b57fa640 1 -- 172.21.15.16:0/3650320387 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fc4a8050b60 con 0x7fc4b8069800 2023-05-22T16:05:40.336 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.332+0000 7fc4be442640 1 -- 172.21.15.16:0/3650320387 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fc4b815c4e0 con 0x7fc4b8069800 2023-05-22T16:05:40.337 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.336+0000 7fc4b57fa640 1 -- 172.21.15.16:0/3650320387 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v197) v1 ==== 92+0+10 (secure 0 0 0) 0x7fc4b815c4e0 con 0x7fc4b8069800 2023-05-22T16:05:40.337 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 8 2023-05-22T16:05:40.339 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.336+0000 7fc4be442640 1 -- 172.21.15.16:0/3650320387 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fc498065580 msgr2=0x7fc498067a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:40.339 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.336+0000 7fc4be442640 1 --2- 172.21.15.16:0/3650320387 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fc498065580 0x7fc498067a30 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fc4a00af000 tx=0x7fc4a0008040 comp rx=0 tx=0).stop 2023-05-22T16:05:40.340 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.336+0000 7fc4be442640 1 -- 172.21.15.16:0/3650320387 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4b8069800 msgr2=0x7fc4b8153350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:40.340 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.336+0000 7fc4be442640 1 --2- 172.21.15.16:0/3650320387 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4b8069800 0x7fc4b8153350 secure :-1 s=READY pgs=2522 cs=0 l=1 rev1=1 crypto rx=0x7fc4a800c530 tx=0x7fc4a800c9f0 comp rx=0 tx=0).stop 2023-05-22T16:05:40.340 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.336+0000 7fc4be442640 1 -- 172.21.15.16:0/3650320387 shutdown_connections 2023-05-22T16:05:40.340 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.336+0000 7fc4be442640 1 --2- 172.21.15.16:0/3650320387 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc4b8071360 0x7fc4b80c50a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:40.340 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.336+0000 7fc4be442640 1 --2- 172.21.15.16:0/3650320387 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc4b8070a50 0x7fc4b80c4b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:40.340 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.336+0000 7fc4be442640 1 --2- 172.21.15.16:0/3650320387 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fc498065580 0x7fc498067a30 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:40.340 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.336+0000 7fc4be442640 1 --2- 172.21.15.16:0/3650320387 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4b8069800 0x7fc4b8153350 unknown :-1 s=CLOSED pgs=2522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:40.341 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.336+0000 7fc4be442640 1 -- 172.21.15.16:0/3650320387 >> 172.21.15.16:0/3650320387 conn(0x7fc4b806d5b0 msgr2=0x7fc4b80cf9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:05:40.341 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.336+0000 7fc4be442640 1 -- 172.21.15.16:0/3650320387 shutdown_connections 2023-05-22T16:05:40.341 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.336+0000 7fc4be442640 1 -- 172.21.15.16:0/3650320387 wait complete. 2023-05-22T16:05:40.357 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-05-22T16:05:40.485 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.484+0000 7f2488ddc640 1 Processor -- start 2023-05-22T16:05:40.485 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.484+0000 7f2488ddc640 1 -- start start 2023-05-22T16:05:40.486 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.484+0000 7f2488ddc640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f24840b3550 0x7f248415e240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:40.486 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.484+0000 7f2488ddc640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f24840cb720 0x7f2484070d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:40.486 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.484+0000 7f2488ddc640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24840b3920 con 0x7f24840b3ae0 2023-05-22T16:05:40.486 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.484+0000 7f2488ddc640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24840b06a0 con 0x7f24840cb720 2023-05-22T16:05:40.486 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.484+0000 7f2488ddc640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24840b0980 con 0x7f24840b3550 2023-05-22T16:05:40.488 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.484+0000 7f2481d9b640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f24840cb720 0x7f2484070d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:40.488 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.484+0000 7f2481d9b640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f24840cb720 0x7f2484070d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59326/0 (socket says 172.21.15.16:59326) 2023-05-22T16:05:40.488 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.484+0000 7f2481d9b640 1 -- 172.21.15.16:0/4127021592 learned_addr learned my addr 172.21.15.16:0/4127021592 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:05:40.488 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.484+0000 7f248259c640 1 --2- 172.21.15.16:0/4127021592 >> v2:172.21.15.62:3300/0 conn(0x7f24840b3550 0x7f248415e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:40.488 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.484+0000 7f2481d9b640 1 -- 172.21.15.16:0/4127021592 >> v2:172.21.15.62:3300/0 conn(0x7f24840b3550 msgr2=0x7f248415e240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:05:40.489 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.484+0000 7f2481d9b640 1 --2- 172.21.15.16:0/4127021592 >> v2:172.21.15.62:3300/0 conn(0x7f24840b3550 0x7f248415e240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:40.489 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.484+0000 7f2481d9b640 1 -- 172.21.15.16:0/4127021592 >> v1:172.21.15.62:6789/0 conn(0x7f24840b3ae0 legacy=0x7f24840cd620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:40.489 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.484+0000 7f2481d9b640 1 -- 172.21.15.16:0/4127021592 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f248415d760 con 0x7f24840cb720 2023-05-22T16:05:40.489 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.484+0000 7f248259c640 1 --2- 172.21.15.16:0/4127021592 >> v2:172.21.15.62:3300/0 conn(0x7f24840b3550 0x7f248415e240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T16:05:40.490 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.488+0000 7f2481d9b640 1 --2- 172.21.15.16:0/4127021592 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f24840cb720 0x7f2484070d60 secure :-1 s=READY pgs=2523 cs=0 l=1 rev1=1 crypto rx=0x7f24840b0ca0 tx=0x7f247801c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6262838eda037a1 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:40.490 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.488+0000 7f248159a640 1 -- 172.21.15.16:0/4127021592 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24780aa430 con 0x7f24840cb720 2023-05-22T16:05:40.490 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.488+0000 7f248159a640 1 -- 172.21.15.16:0/4127021592 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24780aa590 con 0x7f24840cb720 2023-05-22T16:05:40.490 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.488+0000 7f248159a640 1 -- 172.21.15.16:0/4127021592 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24780aa920 con 0x7f24840cb720 2023-05-22T16:05:40.490 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.488+0000 7f2488ddc640 1 -- 172.21.15.16:0/4127021592 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f24840cb720 msgr2=0x7f2484070d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:40.491 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.488+0000 7f2488ddc640 1 --2- 172.21.15.16:0/4127021592 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f24840cb720 0x7f2484070d60 secure :-1 s=READY pgs=2523 cs=0 l=1 rev1=1 crypto rx=0x7f24840b0ca0 tx=0x7f247801c4d0 comp rx=0 tx=0).stop 2023-05-22T16:05:40.491 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.488+0000 7f2488ddc640 1 -- 172.21.15.16:0/4127021592 shutdown_connections 2023-05-22T16:05:40.491 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.488+0000 7f2488ddc640 1 --2- 172.21.15.16:0/4127021592 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f24840cb720 0x7f2484070d60 unknown :-1 s=CLOSED pgs=2523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:40.491 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.488+0000 7f2488ddc640 1 --2- 172.21.15.16:0/4127021592 >> v2:172.21.15.62:3300/0 conn(0x7f24840b3550 0x7f248415e240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:40.491 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.488+0000 7f2488ddc640 1 -- 172.21.15.16:0/4127021592 >> 172.21.15.16:0/4127021592 conn(0x7f2484069800 msgr2=0x7f2484069c00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:05:40.491 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.488+0000 7f2488ddc640 1 -- 172.21.15.16:0/4127021592 shutdown_connections 2023-05-22T16:05:40.492 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.488+0000 7f2488ddc640 1 -- 172.21.15.16:0/4127021592 wait complete. 2023-05-22T16:05:40.493 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.488+0000 7f2488ddc640 1 Processor -- start 2023-05-22T16:05:40.493 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.488+0000 7f2488ddc640 1 -- start start 2023-05-22T16:05:40.494 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.488+0000 7f2488ddc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f24840b3550 0x7f24840af6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:40.494 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.488+0000 7f2488ddc640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f24840b3ae0 0x7f2484155f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:40.494 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.488+0000 7f2488ddc640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f24840cb720 0x7f2484151670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:40.494 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.488+0000 7f2488ddc640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24840c6b90 con 0x7f24840b3550 2023-05-22T16:05:40.494 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.488+0000 7f2488ddc640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24840c1c80 con 0x7f24840b3ae0 2023-05-22T16:05:40.494 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.488+0000 7f2488ddc640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24840c5c30 con 0x7f24840cb720 2023-05-22T16:05:40.494 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.488+0000 7f248259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f24840b3550 0x7f24840af6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:40.495 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.488+0000 7f248259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f24840b3550 0x7f24840af6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59332/0 (socket says 172.21.15.16:59332) 2023-05-22T16:05:40.495 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.488+0000 7f248259c640 1 -- 172.21.15.16:0/944424037 learned_addr learned my addr 172.21.15.16:0/944424037 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:05:40.495 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.492+0000 7f2481d9b640 1 --2- 172.21.15.16:0/944424037 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f24840b3ae0 0x7f2484155f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:40.495 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.492+0000 7f2482d9d640 1 --2- 172.21.15.16:0/944424037 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f24840cb720 0x7f2484151670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:40.495 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.492+0000 7f248259c640 1 -- 172.21.15.16:0/944424037 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f24840cb720 msgr2=0x7f2484151670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:40.495 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.492+0000 7f248259c640 1 --2- 172.21.15.16:0/944424037 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f24840cb720 0x7f2484151670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:40.495 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.492+0000 7f248259c640 1 -- 172.21.15.16:0/944424037 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f24840b3ae0 msgr2=0x7f2484155f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:40.495 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.492+0000 7f248259c640 1 --2- 172.21.15.16:0/944424037 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f24840b3ae0 0x7f2484155f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:40.496 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.492+0000 7f248259c640 1 -- 172.21.15.16:0/944424037 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24780094d0 con 0x7f24840b3550 2023-05-22T16:05:40.496 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.492+0000 7f248259c640 1 --2- 172.21.15.16:0/944424037 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f24840b3550 0x7f24840af6e0 secure :-1 s=READY pgs=2524 cs=0 l=1 rev1=1 crypto rx=0x7f247400e810 tx=0x7f247400ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:40.496 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.492+0000 7f2472ffd640 1 -- 172.21.15.16:0/944424037 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f247400c8d0 con 0x7f24840b3550 2023-05-22T16:05:40.496 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.492+0000 7f2472ffd640 1 -- 172.21.15.16:0/944424037 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f247400ca30 con 0x7f24840b3550 2023-05-22T16:05:40.496 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.492+0000 7f2488ddc640 1 -- 172.21.15.16:0/944424037 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f248406f6a0 con 0x7f24840b3550 2023-05-22T16:05:40.497 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.492+0000 7f2472ffd640 1 -- 172.21.15.16:0/944424037 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f247400cde0 con 0x7f24840b3550 2023-05-22T16:05:40.497 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.492+0000 7f2488ddc640 1 -- 172.21.15.16:0/944424037 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24840716b0 con 0x7f24840b3550 2023-05-22T16:05:40.498 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.492+0000 7f2488ddc640 1 -- 172.21.15.16:0/944424037 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f248406f830 con 0x7f24840b3550 2023-05-22T16:05:40.498 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.492+0000 7f2472ffd640 1 -- 172.21.15.16:0/944424037 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 96) v1 ==== 84406+0+0 (secure 0 0 0) 0x7f2474026480 con 0x7f24840b3550 2023-05-22T16:05:40.498 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.492+0000 7f2472ffd640 1 --2- 172.21.15.16:0/944424037 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f2464065580 0x7f2464067a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:40.498 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.492+0000 7f2472ffd640 1 -- 172.21.15.16:0/944424037 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(197..197 src has 1..197) v4 ==== 6782+0+0 (secure 0 0 0) 0x7f2474083340 con 0x7f24840b3550 2023-05-22T16:05:40.498 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.492+0000 7f2481d9b640 1 --2- 172.21.15.16:0/944424037 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f2464065580 0x7f2464067a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:40.498 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.496+0000 7f2481d9b640 1 --2- 172.21.15.16:0/944424037 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f2464065580 0x7f2464067a30 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f24780ae000 tx=0x7f24780a7000 comp rx=0 tx=0).ready entity=mgr.10974 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:40.499 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.496+0000 7f2472ffd640 1 -- 172.21.15.16:0/944424037 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f2474050950 con 0x7f24840b3550 2023-05-22T16:05:40.654 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.652+0000 7f2488ddc640 1 -- 172.21.15.16:0/944424037 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f24840c6eb0 con 0x7f24840b3550 2023-05-22T16:05:40.655 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.652+0000 7f2472ffd640 1 -- 172.21.15.16:0/944424037 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v197) v1 ==== 104+0+11 (secure 0 0 0) 0x7f2474053f70 con 0x7f24840b3550 2023-05-22T16:05:40.656 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2023-05-22T16:05:40.658 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.656+0000 7f2488ddc640 1 -- 172.21.15.16:0/944424037 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f2464065580 msgr2=0x7f2464067a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:40.659 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.656+0000 7f2488ddc640 1 --2- 172.21.15.16:0/944424037 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f2464065580 0x7f2464067a30 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f24780ae000 tx=0x7f24780a7000 comp rx=0 tx=0).stop 2023-05-22T16:05:40.659 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.656+0000 7f2488ddc640 1 -- 172.21.15.16:0/944424037 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f24840b3550 msgr2=0x7f24840af6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:40.659 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.656+0000 7f2488ddc640 1 --2- 172.21.15.16:0/944424037 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f24840b3550 0x7f24840af6e0 secure :-1 s=READY pgs=2524 cs=0 l=1 rev1=1 crypto rx=0x7f247400e810 tx=0x7f247400ecd0 comp rx=0 tx=0).stop 2023-05-22T16:05:40.659 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.656+0000 7f2488ddc640 1 -- 172.21.15.16:0/944424037 shutdown_connections 2023-05-22T16:05:40.659 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.656+0000 7f2488ddc640 1 --2- 172.21.15.16:0/944424037 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f24840cb720 0x7f2484151670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:40.659 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.656+0000 7f2488ddc640 1 --2- 172.21.15.16:0/944424037 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f24840b3ae0 0x7f2484155f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:40.660 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.656+0000 7f2488ddc640 1 --2- 172.21.15.16:0/944424037 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f2464065580 0x7f2464067a30 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:40.660 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.656+0000 7f2488ddc640 1 --2- 172.21.15.16:0/944424037 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f24840b3550 0x7f24840af6e0 unknown :-1 s=CLOSED pgs=2524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:40.660 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.656+0000 7f2488ddc640 1 -- 172.21.15.16:0/944424037 >> 172.21.15.16:0/944424037 conn(0x7f2484069800 msgr2=0x7f2484156720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:05:40.660 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.656+0000 7f2488ddc640 1 -- 172.21.15.16:0/944424037 shutdown_connections 2023-05-22T16:05:40.660 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.656+0000 7f2488ddc640 1 -- 172.21.15.16:0/944424037 wait complete. 2023-05-22T16:05:40.675 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-05-22T16:05:40.800 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.800+0000 7f615f2da640 1 Processor -- start 2023-05-22T16:05:40.801 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.800+0000 7f615f2da640 1 -- start start 2023-05-22T16:05:40.802 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.800+0000 7f615f2da640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f6158069eb0 0x7f61580b1680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:40.802 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.800+0000 7f615f2da640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6158071370 con 0x7f615806e200 2023-05-22T16:05:40.803 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.800+0000 7f615f2da640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6158071650 con 0x7f615806ddd0 2023-05-22T16:05:40.803 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.800+0000 7f615f2da640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f615815a2f0 con 0x7f6158069eb0 2023-05-22T16:05:40.803 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.800+0000 7f615d877640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f615806e200 0x7f61580ce580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:54266/0 (socket says 172.21.15.16:54266) 2023-05-22T16:05:40.803 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.800+0000 7f615d877640 1 -- 172.21.15.16:0/3817423197 learned_addr learned my addr 172.21.15.16:0/3817423197 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T16:05:40.803 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.800+0000 7f615d076640 1 --2- 172.21.15.16:0/3817423197 >> v2:172.21.15.62:3300/0 conn(0x7f6158069eb0 0x7f61580b1680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:40.803 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.800+0000 7f614ffff640 1 -- 172.21.15.16:0/3817423197 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3826065337 0 0) 0x7f6158071370 con 0x7f615806e200 2023-05-22T16:05:40.803 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.800+0000 7f614ffff640 1 -- 172.21.15.16:0/3817423197 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6144003530 con 0x7f615806e200 2023-05-22T16:05:40.806 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.800+0000 7f614ffff640 1 -- 172.21.15.16:0/3817423197 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 291455039 0 0) 0x7f6144003530 con 0x7f615806e200 2023-05-22T16:05:40.806 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.800+0000 7f614ffff640 1 -- 172.21.15.16:0/3817423197 >> v2:172.21.15.62:3300/0 conn(0x7f6158069eb0 msgr2=0x7f61580b1680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:05:40.806 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.800+0000 7f614ffff640 1 --2- 172.21.15.16:0/3817423197 >> v2:172.21.15.62:3300/0 conn(0x7f6158069eb0 0x7f61580b1680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:40.806 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.800+0000 7f614ffff640 1 -- 172.21.15.16:0/3817423197 >> v1:172.21.15.62:6789/0 conn(0x7f615806ddd0 legacy=0x7f615815df00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:40.806 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.800+0000 7f614ffff640 1 -- 172.21.15.16:0/3817423197 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6158071930 con 0x7f615806e200 2023-05-22T16:05:40.807 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.800+0000 7f614ffff640 1 -- 172.21.15.16:0/3817423197 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7f6150001680 con 0x7f615806e200 2023-05-22T16:05:40.807 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.800+0000 7f614ffff640 1 -- 172.21.15.16:0/3817423197 >> v1:172.21.15.16:6789/0 conn(0x7f615806e200 legacy=0x7f61580ce580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:40.807 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.800+0000 7f614ffff640 1 --2- 172.21.15.16:0/3817423197 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6144003980 0x7f6144003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:40.807 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.800+0000 7f614ffff640 1 --2- 172.21.15.16:0/3817423197 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f61440047d0 0x7f6144004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:40.807 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.800+0000 7f614ffff640 1 --2- 172.21.15.16:0/3817423197 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6144005180 0x7f614400b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:40.808 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.800+0000 7f614ffff640 1 -- 172.21.15.16:0/3817423197 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f614400bfb0 con 0x7f61440047d0 2023-05-22T16:05:40.808 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.800+0000 7f614ffff640 1 -- 172.21.15.16:0/3817423197 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f614400c120 con 0x7f6144003980 2023-05-22T16:05:40.808 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.800+0000 7f614ffff640 1 -- 172.21.15.16:0/3817423197 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f614400c2e0 con 0x7f6144005180 2023-05-22T16:05:40.808 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.800+0000 7f615c875640 1 --2- 172.21.15.16:0/3817423197 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f61440047d0 0x7f6144004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:40.808 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.800+0000 7f615d076640 1 --2- 172.21.15.16:0/3817423197 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6144003980 0x7f6144003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:40.809 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.800+0000 7f615d877640 1 --2- 172.21.15.16:0/3817423197 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6144005180 0x7f614400b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:40.809 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.800+0000 7f615c875640 1 -- 172.21.15.16:0/3817423197 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6144005180 msgr2=0x7f614400b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:05:40.809 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.800+0000 7f615c875640 1 --2- 172.21.15.16:0/3817423197 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6144005180 0x7f614400b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:40.809 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.800+0000 7f615c875640 1 -- 172.21.15.16:0/3817423197 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6144003980 msgr2=0x7f6144003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:05:40.810 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.800+0000 7f615c875640 1 --2- 172.21.15.16:0/3817423197 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6144003980 0x7f6144003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:40.810 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.800+0000 7f615c875640 1 -- 172.21.15.16:0/3817423197 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6144003530 con 0x7f61440047d0 2023-05-22T16:05:40.810 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.800+0000 7f615d877640 1 -- 172.21.15.16:0/3817423197 reap_dead start 2023-05-22T16:05:40.810 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.800+0000 7f615d076640 1 --2- 172.21.15.16:0/3817423197 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6144003980 0x7f6144003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T16:05:40.810 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.804+0000 7f615c875640 1 --2- 172.21.15.16:0/3817423197 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f61440047d0 0x7f6144004c40 secure :-1 s=READY pgs=2526 cs=0 l=1 rev1=1 crypto rx=0x7f6148001780 tx=0x7f614801c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7bda9e18ac1a4584 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:40.811 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.804+0000 7f614ffff640 1 -- 172.21.15.16:0/3817423197 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61480052d0 con 0x7f61440047d0 2023-05-22T16:05:40.811 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.804+0000 7f614ffff640 1 -- 172.21.15.16:0/3817423197 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61480a7050 con 0x7f61440047d0 2023-05-22T16:05:40.811 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.804+0000 7f614ffff640 1 -- 172.21.15.16:0/3817423197 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6148003600 con 0x7f61440047d0 2023-05-22T16:05:40.816 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.804+0000 7f615f2da640 1 -- 172.21.15.16:0/3817423197 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f61440047d0 msgr2=0x7f6144004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:40.816 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.804+0000 7f615f2da640 1 --2- 172.21.15.16:0/3817423197 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f61440047d0 0x7f6144004c40 secure :-1 s=READY pgs=2526 cs=0 l=1 rev1=1 crypto rx=0x7f6148001780 tx=0x7f614801c3e0 comp rx=0 tx=0).stop 2023-05-22T16:05:40.816 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.804+0000 7f615f2da640 1 -- 172.21.15.16:0/3817423197 shutdown_connections 2023-05-22T16:05:40.817 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.804+0000 7f615f2da640 1 --2- 172.21.15.16:0/3817423197 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f61440047d0 0x7f6144004c40 unknown :-1 s=CLOSED pgs=2526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:40.817 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.804+0000 7f615f2da640 1 -- 172.21.15.16:0/3817423197 >> 172.21.15.16:0/3817423197 conn(0x7f615806d5b0 msgr2=0x7f61580b0de0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:05:40.817 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.804+0000 7f615f2da640 1 -- 172.21.15.16:0/3817423197 shutdown_connections 2023-05-22T16:05:40.817 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.804+0000 7f615f2da640 1 -- 172.21.15.16:0/3817423197 wait complete. 2023-05-22T16:05:40.817 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.804+0000 7f615f2da640 1 Processor -- start 2023-05-22T16:05:40.817 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.804+0000 7f615f2da640 1 -- start start 2023-05-22T16:05:40.818 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.808+0000 7f615f2da640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f61440047d0 0x7f615815a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:40.818 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.808+0000 7f615f2da640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f615815c0a0 0x7f61580c88e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:40.818 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.808+0000 7f615f2da640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f61580a7740 0x7f61580adb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:40.818 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.808+0000 7f615f2da640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f615815c4f0 con 0x7f61440047d0 2023-05-22T16:05:40.819 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.808+0000 7f615f2da640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61580c8e20 con 0x7f61580a7740 2023-05-22T16:05:40.819 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.808+0000 7f615f2da640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f615815b020 con 0x7f615815c0a0 2023-05-22T16:05:40.819 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.808+0000 7f615d076640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f61440047d0 0x7f615815a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:40.819 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.808+0000 7f615d076640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f61440047d0 0x7f615815a9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59360/0 (socket says 172.21.15.16:59360) 2023-05-22T16:05:40.820 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.808+0000 7f615d076640 1 -- 172.21.15.16:0/885451011 learned_addr learned my addr 172.21.15.16:0/885451011 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:05:40.820 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.808+0000 7f615d877640 1 --2- 172.21.15.16:0/885451011 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f61580a7740 0x7f61580adb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:40.820 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.808+0000 7f615c875640 1 --2- 172.21.15.16:0/885451011 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f615815c0a0 0x7f61580c88e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:40.820 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.808+0000 7f615d076640 1 -- 172.21.15.16:0/885451011 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f615815c0a0 msgr2=0x7f61580c88e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:40.821 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.808+0000 7f615d076640 1 --2- 172.21.15.16:0/885451011 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f615815c0a0 0x7f61580c88e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:40.821 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.808+0000 7f615d076640 1 -- 172.21.15.16:0/885451011 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f61580a7740 msgr2=0x7f61580adb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:40.821 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.808+0000 7f615d076640 1 --2- 172.21.15.16:0/885451011 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f61580a7740 0x7f61580adb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:40.821 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.808+0000 7f615d076640 1 -- 172.21.15.16:0/885451011 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f615815d9e0 con 0x7f61440047d0 2023-05-22T16:05:40.822 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.808+0000 7f615d076640 1 --2- 172.21.15.16:0/885451011 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f61440047d0 0x7f615815a9c0 secure :-1 s=READY pgs=2527 cs=0 l=1 rev1=1 crypto rx=0x7f6154009f90 tx=0x7f61540101c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:40.822 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.808+0000 7f614dffb640 1 -- 172.21.15.16:0/885451011 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f615400e050 con 0x7f61440047d0 2023-05-22T16:05:40.822 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.808+0000 7f614dffb640 1 -- 172.21.15.16:0/885451011 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f615400f040 con 0x7f61440047d0 2023-05-22T16:05:40.822 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.808+0000 7f615f2da640 1 -- 172.21.15.16:0/885451011 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6158071c10 con 0x7f61440047d0 2023-05-22T16:05:40.823 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.808+0000 7f614dffb640 1 -- 172.21.15.16:0/885451011 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6154010da0 con 0x7f61440047d0 2023-05-22T16:05:40.823 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.808+0000 7f615f2da640 1 -- 172.21.15.16:0/885451011 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6158070f90 con 0x7f61440047d0 2023-05-22T16:05:40.823 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.808+0000 7f615f2da640 1 -- 172.21.15.16:0/885451011 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f615806f0e0 con 0x7f61440047d0 2023-05-22T16:05:40.823 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.808+0000 7f614dffb640 1 -- 172.21.15.16:0/885451011 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 96) v1 ==== 84406+0+0 (secure 0 0 0) 0x7f61540039a0 con 0x7f61440047d0 2023-05-22T16:05:40.824 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.808+0000 7f614dffb640 1 --2- 172.21.15.16:0/885451011 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f613c065580 0x7f613c067a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:40.824 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.808+0000 7f615c875640 1 --2- 172.21.15.16:0/885451011 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f613c065580 0x7f613c067a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:40.824 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.808+0000 7f614dffb640 1 -- 172.21.15.16:0/885451011 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(197..197 src has 1..197) v4 ==== 6782+0+0 (secure 0 0 0) 0x7f6154083880 con 0x7f61440047d0 2023-05-22T16:05:40.824 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.808+0000 7f615c875640 1 --2- 172.21.15.16:0/885451011 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f613c065580 0x7f613c067a30 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f6148001750 tx=0x7f61480011a0 comp rx=0 tx=0).ready entity=mgr.10974 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:40.825 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.812+0000 7f614dffb640 1 -- 172.21.15.16:0/885451011 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f6154050f30 con 0x7f61440047d0 2023-05-22T16:05:40.967 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.964+0000 7f615f2da640 1 -- 172.21.15.16:0/885451011 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f6158158400 con 0x7f61440047d0 2023-05-22T16:05:40.968 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.964+0000 7f614dffb640 1 -- 172.21.15.16:0/885451011 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v197) v1 ==== 100+0+11 (secure 0 0 0) 0x7f6154054550 con 0x7f61440047d0 2023-05-22T16:05:40.968 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 64 2023-05-22T16:05:40.970 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.968+0000 7f615f2da640 1 -- 172.21.15.16:0/885451011 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f613c065580 msgr2=0x7f613c067a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:40.970 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.968+0000 7f615f2da640 1 --2- 172.21.15.16:0/885451011 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f613c065580 0x7f613c067a30 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f6148001750 tx=0x7f61480011a0 comp rx=0 tx=0).stop 2023-05-22T16:05:40.970 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.968+0000 7f615f2da640 1 -- 172.21.15.16:0/885451011 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f61440047d0 msgr2=0x7f615815a9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:40.970 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.968+0000 7f615f2da640 1 --2- 172.21.15.16:0/885451011 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f61440047d0 0x7f615815a9c0 secure :-1 s=READY pgs=2527 cs=0 l=1 rev1=1 crypto rx=0x7f6154009f90 tx=0x7f61540101c0 comp rx=0 tx=0).stop 2023-05-22T16:05:40.971 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.968+0000 7f615f2da640 1 -- 172.21.15.16:0/885451011 shutdown_connections 2023-05-22T16:05:40.971 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.968+0000 7f615f2da640 1 --2- 172.21.15.16:0/885451011 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f61580a7740 0x7f61580adb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:40.971 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.968+0000 7f615f2da640 1 --2- 172.21.15.16:0/885451011 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f615815c0a0 0x7f61580c88e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:40.971 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.968+0000 7f615f2da640 1 --2- 172.21.15.16:0/885451011 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f613c065580 0x7f613c067a30 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:40.971 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.968+0000 7f615f2da640 1 --2- 172.21.15.16:0/885451011 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f61440047d0 0x7f615815a9c0 unknown :-1 s=CLOSED pgs=2527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:40.971 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.968+0000 7f615f2da640 1 -- 172.21.15.16:0/885451011 >> 172.21.15.16:0/885451011 conn(0x7f615806d5b0 msgr2=0x7f61580b0a70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:05:40.972 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.968+0000 7f615f2da640 1 -- 172.21.15.16:0/885451011 shutdown_connections 2023-05-22T16:05:40.972 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:40.968+0000 7f615f2da640 1 -- 172.21.15.16:0/885451011 wait complete. 2023-05-22T16:05:40.988 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-05-22T16:05:41.110 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:41.108+0000 7f8e5b0db640 1 Processor -- start 2023-05-22T16:05:41.110 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:41.108+0000 7f8e5b0db640 1 -- start start 2023-05-22T16:05:41.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:41.108+0000 7f8e5b0db640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f8e54154170 0x7f8e5406f630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:41.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:41.108+0000 7f8e5b0db640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f8e540afb50 0x7f8e540affb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:41.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:41.108+0000 7f8e5b0db640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e54071b40 con 0x7f8e5406cc80 2023-05-22T16:05:41.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:41.108+0000 7f8e5b0db640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e540ca9f0 con 0x7f8e540afb50 2023-05-22T16:05:41.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:41.108+0000 7f8e5b0db640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e540ca150 con 0x7f8e54154170 2023-05-22T16:05:41.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:41.108+0000 7f8e59678640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f8e540afb50 0x7f8e540affb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:41.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:41.108+0000 7f8e59678640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f8e540afb50 0x7f8e540affb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59366/0 (socket says 172.21.15.16:59366) 2023-05-22T16:05:41.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:41.108+0000 7f8e59678640 1 -- 172.21.15.16:0/3568637410 learned_addr learned my addr 172.21.15.16:0/3568637410 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:05:41.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:41.108+0000 7f8e59678640 1 -- 172.21.15.16:0/3568637410 >> v2:172.21.15.62:3300/0 conn(0x7f8e54154170 msgr2=0x7f8e5406f630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:05:41.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:41.108+0000 7f8e59678640 1 --2- 172.21.15.16:0/3568637410 >> v2:172.21.15.62:3300/0 conn(0x7f8e54154170 0x7f8e5406f630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:41.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:41.108+0000 7f8e59678640 1 -- 172.21.15.16:0/3568637410 >> v1:172.21.15.62:6789/0 conn(0x7f8e5406cc80 legacy=0x7f8e5406be90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:05:41.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:41.108+0000 7f8e59678640 1 -- 172.21.15.16:0/3568637410 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e540b04f0 con 0x7f8e540afb50 2023-05-22T16:05:41.119 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:41.112+0000 7f8e59678640 1 --2- 172.21.15.16:0/3568637410 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8e540afb50 0x7f8e540affb0 secure :-1 s=READY pgs=2528 cs=0 l=1 rev1=1 crypto rx=0x7f8e540b0bb0 tx=0x7f8e4c01c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50e22abc1d3fe7e8 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:41.119 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:41.112+0000 7f8e537fe640 1 -- 172.21.15.16:0/3568637410 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e4c0a5820 con 0x7f8e540afb50 2023-05-22T16:05:41.119 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:41.112+0000 7f8e537fe640 1 -- 172.21.15.16:0/3568637410 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e4c0a5980 con 0x7f8e540afb50 2023-05-22T16:05:41.119 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:41.112+0000 7f8e537fe640 1 -- 172.21.15.16:0/3568637410 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e4c0a5d10 con 0x7f8e540afb50 2023-05-22T16:05:41.120 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:41.112+0000 7f8e5b0db640 1 -- 172.21.15.16:0/3568637410 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8e540afb50 msgr2=0x7f8e540affb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:41.120 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:41.112+0000 7f8e5b0db640 1 --2- 172.21.15.16:0/3568637410 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8e540afb50 0x7f8e540affb0 secure :-1 s=READY pgs=2528 cs=0 l=1 rev1=1 crypto rx=0x7f8e540b0bb0 tx=0x7f8e4c01c960 comp rx=0 tx=0).stop 2023-05-22T16:05:41.120 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:41.112+0000 7f8e5b0db640 1 -- 172.21.15.16:0/3568637410 shutdown_connections 2023-05-22T16:05:41.120 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:41.112+0000 7f8e5b0db640 1 --2- 172.21.15.16:0/3568637410 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8e540afb50 0x7f8e540affb0 unknown :-1 s=CLOSED pgs=2528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:41.121 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:41.112+0000 7f8e5b0db640 1 --2- 172.21.15.16:0/3568637410 >> v2:172.21.15.62:3300/0 conn(0x7f8e54154170 0x7f8e5406f630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:41.121 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:41.112+0000 7f8e5b0db640 1 -- 172.21.15.16:0/3568637410 >> 172.21.15.16:0/3568637410 conn(0x7f8e540cab50 msgr2=0x7f8e540caf50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:05:41.121 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:41.112+0000 7f8e5b0db640 1 -- 172.21.15.16:0/3568637410 shutdown_connections 2023-05-22T16:05:41.121 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:41.112+0000 7f8e5b0db640 1 -- 172.21.15.16:0/3568637410 wait complete. 2023-05-22T16:05:41.122 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:41.112+0000 7f8e5b0db640 1 Processor -- start 2023-05-22T16:05:41.122 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:41.112+0000 7f8e5b0db640 1 -- start start 2023-05-22T16:05:41.122 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:41.112+0000 7f8e5b0db640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8e5406cc80 0x7f8e5414c360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:41.122 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:41.112+0000 7f8e5b0db640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8e540afb50 0x7f8e54149cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:41.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:41.112+0000 7f8e5b0db640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8e54154170 0x7f8e540b2a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:41.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:41.112+0000 7f8e5b0db640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e540b1210 con 0x7f8e54154170 2023-05-22T16:05:41.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:41.116+0000 7f8e5b0db640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e540b0f00 con 0x7f8e540afb50 2023-05-22T16:05:41.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:41.116+0000 7f8e5b0db640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e5414c8a0 con 0x7f8e5406cc80 2023-05-22T16:05:41.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:41.116+0000 7f8e59678640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8e54154170 0x7f8e540b2a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:41.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:41.116+0000 7f8e59678640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8e54154170 0x7f8e540b2a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59380/0 (socket says 172.21.15.16:59380) 2023-05-22T16:05:41.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:41.116+0000 7f8e59678640 1 -- 172.21.15.16:0/1912578436 learned_addr learned my addr 172.21.15.16:0/1912578436 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:05:41.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:41.116+0000 7f8e53fff640 1 --2- 172.21.15.16:0/1912578436 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8e540afb50 0x7f8e54149cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:41.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:41.116+0000 7f8e58e77640 1 --2- 172.21.15.16:0/1912578436 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8e5406cc80 0x7f8e5414c360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:41.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:41.116+0000 7f8e59678640 1 -- 172.21.15.16:0/1912578436 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8e5406cc80 msgr2=0x7f8e5414c360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:41.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:41.116+0000 7f8e59678640 1 --2- 172.21.15.16:0/1912578436 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8e5406cc80 0x7f8e5414c360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:41.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:41.116+0000 7f8e59678640 1 -- 172.21.15.16:0/1912578436 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8e540afb50 msgr2=0x7f8e54149cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:41.126 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:41.116+0000 7f8e59678640 1 --2- 172.21.15.16:0/1912578436 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8e540afb50 0x7f8e54149cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:41.126 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:41.116+0000 7f8e59678640 1 -- 172.21.15.16:0/1912578436 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e4c009b30 con 0x7f8e54154170 2023-05-22T16:05:41.126 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:41.116+0000 7f8e59678640 1 --2- 172.21.15.16:0/1912578436 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8e54154170 0x7f8e540b2a80 secure :-1 s=READY pgs=2529 cs=0 l=1 rev1=1 crypto rx=0x7f8e4c01a590 tx=0x7f8e4c019700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:41.126 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:41.116+0000 7f8e517fa640 1 -- 172.21.15.16:0/1912578436 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e4c0b8670 con 0x7f8e54154170 2023-05-22T16:05:41.127 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:41.116+0000 7f8e517fa640 1 -- 172.21.15.16:0/1912578436 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e4c0b87f0 con 0x7f8e54154170 2023-05-22T16:05:41.127 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:41.116+0000 7f8e5b0db640 1 -- 172.21.15.16:0/1912578436 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e5406c5a0 con 0x7f8e54154170 2023-05-22T16:05:41.127 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:41.116+0000 7f8e517fa640 1 -- 172.21.15.16:0/1912578436 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e4c0b8ba0 con 0x7f8e54154170 2023-05-22T16:05:41.127 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:41.116+0000 7f8e5b0db640 1 -- 172.21.15.16:0/1912578436 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e5406b970 con 0x7f8e54154170 2023-05-22T16:05:41.128 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:41.116+0000 7f8e5b0db640 1 -- 172.21.15.16:0/1912578436 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e54154540 con 0x7f8e54154170 2023-05-22T16:05:41.128 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:41.116+0000 7f8e517fa640 1 -- 172.21.15.16:0/1912578436 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 96) v1 ==== 84406+0+0 (secure 0 0 0) 0x7f8e4c0b8d00 con 0x7f8e54154170 2023-05-22T16:05:41.129 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:41.116+0000 7f8e517fa640 1 --2- 172.21.15.16:0/1912578436 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f8e34065580 0x7f8e34067a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:41.129 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:41.116+0000 7f8e517fa640 1 -- 172.21.15.16:0/1912578436 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(197..197 src has 1..197) v4 ==== 6782+0+0 (secure 0 0 0) 0x7f8e4c11e930 con 0x7f8e54154170 2023-05-22T16:05:41.130 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:41.116+0000 7f8e58e77640 1 --2- 172.21.15.16:0/1912578436 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f8e34065580 0x7f8e34067a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:41.130 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:41.116+0000 7f8e58e77640 1 --2- 172.21.15.16:0/1912578436 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f8e34065580 0x7f8e34067a30 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f8e3c006fd0 tx=0x7f8e3c008040 comp rx=0 tx=0).ready entity=mgr.10974 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:41.130 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:41.120+0000 7f8e517fa640 1 -- 172.21.15.16:0/1912578436 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f8e4c0ec1d0 con 0x7f8e54154170 2023-05-22T16:05:41.290 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:41.288+0000 7f8e5b0db640 1 -- 172.21.15.16:0/1912578436 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f8e540b1350 con 0x7f8e54154170 2023-05-22T16:05:41.291 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:41.288+0000 7f8e517fa640 1 -- 172.21.15.16:0/1912578436 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 8 entries v197) v1 ==== 80+0+458 (secure 0 0 0) 0x7f8e540b1350 con 0x7f8e54154170 2023-05-22T16:05:41.292 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/1961044213 2023-05-23T16:05:35.647562+0000 2023-05-22T16:05:41.292 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/238292614 2023-05-23T16:05:35.647562+0000 2023-05-22T16:05:41.292 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:6833/65890 2023-05-23T16:05:35.647562+0000 2023-05-22T16:05:41.292 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/1148602276 2023-05-23T16:05:35.647562+0000 2023-05-22T16:05:41.293 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:6832/65890 2023-05-23T16:05:35.647562+0000 2023-05-22T16:05:41.293 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/456460873 2023-05-23T16:05:35.647562+0000 2023-05-22T16:05:41.293 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/2766506027 2023-05-23T16:05:35.647562+0000 2023-05-22T16:05:41.293 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/2214029832 2023-05-23T16:05:35.647562+0000 2023-05-22T16:05:41.294 INFO:teuthology.orchestra.run.smithi016.stderr:listed 8 entries 2023-05-22T16:05:41.294 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:41.292+0000 7f8e5b0db640 1 -- 172.21.15.16:0/1912578436 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f8e34065580 msgr2=0x7f8e34067a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:41.295 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:41.292+0000 7f8e5b0db640 1 --2- 172.21.15.16:0/1912578436 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f8e34065580 0x7f8e34067a30 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f8e3c006fd0 tx=0x7f8e3c008040 comp rx=0 tx=0).stop 2023-05-22T16:05:41.295 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:41.292+0000 7f8e5b0db640 1 -- 172.21.15.16:0/1912578436 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8e54154170 msgr2=0x7f8e540b2a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:41.295 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:41.292+0000 7f8e5b0db640 1 --2- 172.21.15.16:0/1912578436 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8e54154170 0x7f8e540b2a80 secure :-1 s=READY pgs=2529 cs=0 l=1 rev1=1 crypto rx=0x7f8e4c01a590 tx=0x7f8e4c019700 comp rx=0 tx=0).stop 2023-05-22T16:05:41.295 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:41.292+0000 7f8e5b0db640 1 -- 172.21.15.16:0/1912578436 shutdown_connections 2023-05-22T16:05:41.295 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:41.292+0000 7f8e5b0db640 1 --2- 172.21.15.16:0/1912578436 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f8e34065580 0x7f8e34067a30 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:41.295 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:41.292+0000 7f8e5b0db640 1 --2- 172.21.15.16:0/1912578436 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8e54154170 0x7f8e540b2a80 unknown :-1 s=CLOSED pgs=2529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:41.296 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:41.292+0000 7f8e5b0db640 1 --2- 172.21.15.16:0/1912578436 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8e540afb50 0x7f8e54149cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:41.296 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:41.292+0000 7f8e5b0db640 1 --2- 172.21.15.16:0/1912578436 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8e5406cc80 0x7f8e5414c360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:41.296 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:41.292+0000 7f8e5b0db640 1 -- 172.21.15.16:0/1912578436 >> 172.21.15.16:0/1912578436 conn(0x7f8e540cab50 msgr2=0x7f8e54152e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:05:41.296 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:41.292+0000 7f8e5b0db640 1 -- 172.21.15.16:0/1912578436 shutdown_connections 2023-05-22T16:05:41.296 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:41.292+0000 7f8e5b0db640 1 -- 172.21.15.16:0/1912578436 wait complete. 2023-05-22T16:05:41.309 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi016.front.sepia.ceph.com... 2023-05-22T16:05:41.310 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T16:05:41.310 DEBUG:teuthology.orchestra.run.smithi016:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2023-05-22T16:05:41.321 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi016.stderr:ceph-fuse[69985]: fuse finished with error 0 and tester_r 0 2023-05-22T16:05:41.322 INFO:teuthology.orchestra.run:waiting for 300 2023-05-22T16:05:47.424 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi016.front.sepia.ceph.com 2023-05-22T16:05:47.424 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T16:05:47.425 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-05-22T16:05:47.432 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-05-22T16:05:47.433 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2023-05-22T16:05:47.481 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16345192k,nr_inodes=4086298,mode=755,inode64 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3276544k,mode=755,inode64 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev,inode64 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=15732 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3276540k,nr_inodes=819135,mode=700,uid=1000,gid=1271,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3276544k,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-05-22T16:05:47.482 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-05-22T16:05:47.661 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:47.660+0000 7fd0cd845640 1 Processor -- start 2023-05-22T16:05:47.661 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:47.660+0000 7fd0cd845640 1 -- start start 2023-05-22T16:05:47.662 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:47.660+0000 7fd0cd845640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd0c80cd810 0x7fd0c80cf400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:47.662 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:47.660+0000 7fd0cd845640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0c80d00a0 con 0x7fd0c8159ba0 2023-05-22T16:05:47.662 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:47.660+0000 7fd0cd845640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0c806def0 con 0x7fd0c80b21d0 2023-05-22T16:05:47.662 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:47.660+0000 7fd0cd845640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0c8159f70 con 0x7fd0c80cd810 2023-05-22T16:05:47.664 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:47.660+0000 7fd0c6ffd640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fd0c8159ba0 0x7fd0c806e380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:54274/0 (socket says 172.21.15.16:54274) 2023-05-22T16:05:47.664 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:47.660+0000 7fd0c6ffd640 1 -- 172.21.15.16:0/2073413566 learned_addr learned my addr 172.21.15.16:0/2073413566 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T16:05:47.664 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:47.660+0000 7fd0bffff640 1 --2- 172.21.15.16:0/2073413566 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd0c80cd810 0x7fd0c80cf400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:47.664 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:47.660+0000 7fd0c67fc640 1 -- 172.21.15.16:0/2073413566 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4016057996 0 0) 0x7fd0c80d00a0 con 0x7fd0c8159ba0 2023-05-22T16:05:47.664 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:47.660+0000 7fd0c67fc640 1 -- 172.21.15.16:0/2073413566 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd0b4003530 con 0x7fd0c8159ba0 2023-05-22T16:05:47.665 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:47.660+0000 7fd0c67fc640 1 -- 172.21.15.16:0/2073413566 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2473886209 0 0) 0x7fd0b4003530 con 0x7fd0c8159ba0 2023-05-22T16:05:47.665 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:47.660+0000 7fd0c67fc640 1 -- 172.21.15.16:0/2073413566 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd0c80cd810 msgr2=0x7fd0c80cf400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:05:47.665 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:47.660+0000 7fd0c67fc640 1 --2- 172.21.15.16:0/2073413566 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd0c80cd810 0x7fd0c80cf400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:47.665 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:47.660+0000 7fd0c67fc640 1 -- 172.21.15.16:0/2073413566 >> v1:172.21.15.62:6789/0 conn(0x7fd0c80b21d0 legacy=0x7fd0c815d670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:47.665 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:47.660+0000 7fd0c67fc640 1 -- 172.21.15.16:0/2073413566 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0c80cca60 con 0x7fd0c8159ba0 2023-05-22T16:05:47.665 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:47.660+0000 7fd0c67fc640 1 -- 172.21.15.16:0/2073413566 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7fd0b8001680 con 0x7fd0c8159ba0 2023-05-22T16:05:47.665 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:47.660+0000 7fd0c67fc640 1 -- 172.21.15.16:0/2073413566 >> v1:172.21.15.16:6789/0 conn(0x7fd0c8159ba0 legacy=0x7fd0c806e380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:47.666 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:47.660+0000 7fd0c67fc640 1 --2- 172.21.15.16:0/2073413566 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd0c80cd810 0x7fd0b40039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:47.666 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:47.660+0000 7fd0c67fc640 1 --2- 172.21.15.16:0/2073413566 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd0b4003ef0 0x7fd0b40047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:47.666 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:47.660+0000 7fd0c67fc640 1 --2- 172.21.15.16:0/2073413566 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd0b4004d10 0x7fd0b4005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:47.666 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:47.660+0000 7fd0c67fc640 1 -- 172.21.15.16:0/2073413566 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0c8159f70 con 0x7fd0b4003ef0 2023-05-22T16:05:47.667 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:47.660+0000 7fd0c67fc640 1 -- 172.21.15.16:0/2073413566 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0b400ba70 con 0x7fd0b4004d10 2023-05-22T16:05:47.667 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:47.660+0000 7fd0c67fc640 1 -- 172.21.15.16:0/2073413566 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0b400bc30 con 0x7fd0c80cd810 2023-05-22T16:05:47.667 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:47.660+0000 7fd0bffff640 1 --2- 172.21.15.16:0/2073413566 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd0b4003ef0 0x7fd0b40047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:47.667 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:47.660+0000 7fd0c6ffd640 1 --2- 172.21.15.16:0/2073413566 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd0c80cd810 0x7fd0b40039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:47.667 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:47.660+0000 7fd0c77fe640 1 --2- 172.21.15.16:0/2073413566 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd0b4004d10 0x7fd0b4005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:47.668 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:47.660+0000 7fd0bffff640 1 -- 172.21.15.16:0/2073413566 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd0c80cd810 msgr2=0x7fd0b40039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:05:47.668 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:47.660+0000 7fd0bffff640 1 --2- 172.21.15.16:0/2073413566 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd0c80cd810 0x7fd0b40039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:47.668 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:47.660+0000 7fd0bffff640 1 -- 172.21.15.16:0/2073413566 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd0b4004d10 msgr2=0x7fd0b4005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:05:47.668 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:47.660+0000 7fd0bffff640 1 --2- 172.21.15.16:0/2073413566 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd0b4004d10 0x7fd0b4005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:47.668 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:47.660+0000 7fd0bffff640 1 -- 172.21.15.16:0/2073413566 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0b4003530 con 0x7fd0b4003ef0 2023-05-22T16:05:47.671 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:47.664+0000 7fd0bffff640 1 --2- 172.21.15.16:0/2073413566 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd0b4003ef0 0x7fd0b40047d0 secure :-1 s=READY pgs=2531 cs=0 l=1 rev1=1 crypto rx=0x7fd0b0009ab0 tx=0x7fd0b001ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3b2c8678e3e3da5 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:47.672 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:47.664+0000 7fd0c67fc640 1 -- 172.21.15.16:0/2073413566 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0b00a5620 con 0x7fd0b4003ef0 2023-05-22T16:05:47.672 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:47.664+0000 7fd0c67fc640 1 -- 172.21.15.16:0/2073413566 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0b00a5780 con 0x7fd0b4003ef0 2023-05-22T16:05:47.672 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:47.664+0000 7fd0c67fc640 1 -- 172.21.15.16:0/2073413566 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0b00a5b30 con 0x7fd0b4003ef0 2023-05-22T16:05:47.672 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:47.664+0000 7fd0cd845640 1 -- 172.21.15.16:0/2073413566 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd0b4003ef0 msgr2=0x7fd0b40047d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:47.672 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:47.664+0000 7fd0cd845640 1 --2- 172.21.15.16:0/2073413566 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd0b4003ef0 0x7fd0b40047d0 secure :-1 s=READY pgs=2531 cs=0 l=1 rev1=1 crypto rx=0x7fd0b0009ab0 tx=0x7fd0b001ca30 comp rx=0 tx=0).stop 2023-05-22T16:05:47.673 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:47.664+0000 7fd0c77fe640 1 -- 172.21.15.16:0/2073413566 reap_dead start 2023-05-22T16:05:47.673 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:47.664+0000 7fd0cd845640 1 -- 172.21.15.16:0/2073413566 shutdown_connections 2023-05-22T16:05:47.673 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:47.664+0000 7fd0cd845640 1 -- 172.21.15.16:0/2073413566 >> 172.21.15.16:0/2073413566 conn(0x7fd0c80653f0 msgr2=0x7fd0c80657f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:05:47.673 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:47.664+0000 7fd0cd845640 1 -- 172.21.15.16:0/2073413566 shutdown_connections 2023-05-22T16:05:47.673 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:47.664+0000 7fd0cd845640 1 -- 172.21.15.16:0/2073413566 wait complete. 2023-05-22T16:05:47.674 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:47.664+0000 7fd0cd845640 1 Processor -- start 2023-05-22T16:05:47.674 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:47.664+0000 7fd0cd845640 1 -- start start 2023-05-22T16:05:47.674 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:47.668+0000 7fd0cd845640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd0b4003ef0 0x7fd0c80c74f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:47.674 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:47.668+0000 7fd0cd845640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd0c8150860 0x7fd0c80c8db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:47.674 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:47.668+0000 7fd0cd845640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd0c81547f0 0x7fd0c8154c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:47.674 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:47.668+0000 7fd0cd845640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0c81551a0 con 0x7fd0b4003ef0 2023-05-22T16:05:47.675 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:47.668+0000 7fd0cd845640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0c80c9690 con 0x7fd0c81547f0 2023-05-22T16:05:47.675 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:47.668+0000 7fd0cd845640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0c80c9970 con 0x7fd0c8150860 2023-05-22T16:05:47.675 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:47.668+0000 7fd0c6ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd0b4003ef0 0x7fd0c80c74f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:47.675 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:47.668+0000 7fd0c6ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd0b4003ef0 0x7fd0c80c74f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59392/0 (socket says 172.21.15.16:59392) 2023-05-22T16:05:47.675 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:47.668+0000 7fd0c6ffd640 1 -- 172.21.15.16:0/149834554 learned_addr learned my addr 172.21.15.16:0/149834554 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:05:47.675 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:47.668+0000 7fd0c77fe640 1 --2- 172.21.15.16:0/149834554 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd0c81547f0 0x7fd0c8154c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:47.676 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:47.668+0000 7fd0bffff640 1 --2- 172.21.15.16:0/149834554 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd0c8150860 0x7fd0c80c8db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:47.676 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:47.668+0000 7fd0c6ffd640 1 -- 172.21.15.16:0/149834554 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd0c8150860 msgr2=0x7fd0c80c8db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:47.676 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:47.668+0000 7fd0c6ffd640 1 --2- 172.21.15.16:0/149834554 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd0c8150860 0x7fd0c80c8db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:47.676 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:47.668+0000 7fd0c6ffd640 1 -- 172.21.15.16:0/149834554 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd0c81547f0 msgr2=0x7fd0c8154c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:47.677 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:47.668+0000 7fd0c6ffd640 1 --2- 172.21.15.16:0/149834554 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd0c81547f0 0x7fd0c8154c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:47.677 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:47.668+0000 7fd0c6ffd640 1 -- 172.21.15.16:0/149834554 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0c80cfdc0 con 0x7fd0b4003ef0 2023-05-22T16:05:47.677 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:47.668+0000 7fd0bffff640 1 --2- 172.21.15.16:0/149834554 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd0c8150860 0x7fd0c80c8db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T16:05:47.677 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:47.668+0000 7fd0c6ffd640 1 --2- 172.21.15.16:0/149834554 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd0b4003ef0 0x7fd0c80c74f0 secure :-1 s=READY pgs=2532 cs=0 l=1 rev1=1 crypto rx=0x7fd0b800e780 tx=0x7fd0b800ec40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:47.678 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:47.668+0000 7fd0c4ff9640 1 -- 172.21.15.16:0/149834554 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0b800ee90 con 0x7fd0b4003ef0 2023-05-22T16:05:47.678 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:47.668+0000 7fd0c4ff9640 1 -- 172.21.15.16:0/149834554 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0b800c8d0 con 0x7fd0b4003ef0 2023-05-22T16:05:47.678 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:47.668+0000 7fd0cd845640 1 -- 172.21.15.16:0/149834554 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0c80cfae0 con 0x7fd0b4003ef0 2023-05-22T16:05:47.679 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:47.668+0000 7fd0c4ff9640 1 -- 172.21.15.16:0/149834554 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0b800cc80 con 0x7fd0b4003ef0 2023-05-22T16:05:47.679 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:47.668+0000 7fd0cd845640 1 -- 172.21.15.16:0/149834554 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0c806db80 con 0x7fd0b4003ef0 2023-05-22T16:05:47.679 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:47.668+0000 7fd0cd845640 1 -- 172.21.15.16:0/149834554 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0c8073830 con 0x7fd0b4003ef0 2023-05-22T16:05:47.680 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:47.668+0000 7fd0c4ff9640 1 -- 172.21.15.16:0/149834554 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 96) v1 ==== 84406+0+0 (secure 0 0 0) 0x7fd0b801d460 con 0x7fd0b4003ef0 2023-05-22T16:05:47.680 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:47.668+0000 7fd0c4ff9640 1 --2- 172.21.15.16:0/149834554 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fd0a8065580 0x7fd0a8067a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:47.680 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:47.668+0000 7fd0bffff640 1 --2- 172.21.15.16:0/149834554 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fd0a8065580 0x7fd0a8067a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:47.680 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:47.668+0000 7fd0c4ff9640 1 -- 172.21.15.16:0/149834554 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(197..197 src has 1..197) v4 ==== 6782+0+0 (secure 0 0 0) 0x7fd0b8082ee0 con 0x7fd0b4003ef0 2023-05-22T16:05:47.681 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:47.668+0000 7fd0bffff640 1 --2- 172.21.15.16:0/149834554 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fd0a8065580 0x7fd0a8067a30 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fd0b00af000 tx=0x7fd0b00046d0 comp rx=0 tx=0).ready entity=mgr.10974 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:47.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:47.672+0000 7fd0c4ff9640 1 -- 172.21.15.16:0/149834554 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fd0b80505b0 con 0x7fd0b4003ef0 2023-05-22T16:05:47.865 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:47.864+0000 7fd0cd845640 1 -- 172.21.15.16:0/149834554 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fd0c80cf550 con 0x7fd0b4003ef0 2023-05-22T16:05:47.866 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:47.864+0000 7fd0c4ff9640 1 -- 172.21.15.16:0/149834554 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v81) v1 ==== 53+0+73 (secure 0 0 0) 0x7fd0b8053bd0 con 0x7fd0b4003ef0 2023-05-22T16:05:47.866 INFO:teuthology.orchestra.run.smithi016.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-05-22T16:05:47.868 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:47.864+0000 7fd0cd845640 1 -- 172.21.15.16:0/149834554 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fd0a8065580 msgr2=0x7fd0a8067a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:47.868 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:47.864+0000 7fd0cd845640 1 --2- 172.21.15.16:0/149834554 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fd0a8065580 0x7fd0a8067a30 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fd0b00af000 tx=0x7fd0b00046d0 comp rx=0 tx=0).stop 2023-05-22T16:05:47.869 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:47.864+0000 7fd0cd845640 1 -- 172.21.15.16:0/149834554 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd0b4003ef0 msgr2=0x7fd0c80c74f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:47.869 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:47.864+0000 7fd0cd845640 1 --2- 172.21.15.16:0/149834554 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd0b4003ef0 0x7fd0c80c74f0 secure :-1 s=READY pgs=2532 cs=0 l=1 rev1=1 crypto rx=0x7fd0b800e780 tx=0x7fd0b800ec40 comp rx=0 tx=0).stop 2023-05-22T16:05:47.869 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:47.864+0000 7fd0cd845640 1 -- 172.21.15.16:0/149834554 shutdown_connections 2023-05-22T16:05:47.869 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:47.864+0000 7fd0cd845640 1 --2- 172.21.15.16:0/149834554 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd0c81547f0 0x7fd0c8154c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:47.869 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:47.864+0000 7fd0cd845640 1 --2- 172.21.15.16:0/149834554 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd0c8150860 0x7fd0c80c8db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:47.869 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:47.864+0000 7fd0cd845640 1 --2- 172.21.15.16:0/149834554 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fd0a8065580 0x7fd0a8067a30 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:47.869 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:47.864+0000 7fd0cd845640 1 --2- 172.21.15.16:0/149834554 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd0b4003ef0 0x7fd0c80c74f0 unknown :-1 s=CLOSED pgs=2532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:47.870 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:47.864+0000 7fd0cd845640 1 -- 172.21.15.16:0/149834554 >> 172.21.15.16:0/149834554 conn(0x7fd0c80653f0 msgr2=0x7fd0c80cbce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:05:47.870 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:47.868+0000 7fd0cd845640 1 -- 172.21.15.16:0/149834554 shutdown_connections 2023-05-22T16:05:47.870 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:47.868+0000 7fd0cd845640 1 -- 172.21.15.16:0/149834554 wait complete. 2023-05-22T16:05:47.886 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-05-22T16:05:47.886 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-05-22T16:05:47.886 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi016.front.sepia.ceph.com 2023-05-22T16:05:47.887 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-05-22T16:05:47.887 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-05-22T16:05:47.887 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-05-22T16:05:47.888 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-05-22T16:05:47.888 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-05-22T16:05:47.888 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-05-22T16:05:47.888 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T16:05:47.889 DEBUG:teuthology.orchestra.run.smithi016:> ip addr 2023-05-22T16:05:47.895 INFO:teuthology.orchestra.run.smithi016.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-05-22T16:05:47.896 INFO:teuthology.orchestra.run.smithi016.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-05-22T16:05:47.896 INFO:teuthology.orchestra.run.smithi016.stdout: inet 127.0.0.1/8 scope host lo 2023-05-22T16:05:47.896 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-05-22T16:05:47.896 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 ::1/128 scope host 2023-05-22T16:05:47.896 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-05-22T16:05:47.896 INFO:teuthology.orchestra.run.smithi016.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-05-22T16:05:47.896 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:6b:fd:e0 brd ff:ff:ff:ff:ff:ff 2023-05-22T16:05:47.897 INFO:teuthology.orchestra.run.smithi016.stdout: altname enp7s0f0 2023-05-22T16:05:47.898 INFO:teuthology.orchestra.run.smithi016.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-05-22T16:05:47.898 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:6b:fd:e1 brd ff:ff:ff:ff:ff:ff 2023-05-22T16:05:47.898 INFO:teuthology.orchestra.run.smithi016.stdout: altname enp7s0f1 2023-05-22T16:05:47.898 INFO:teuthology.orchestra.run.smithi016.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2023-05-22T16:05:47.898 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:bd:15:64 brd ff:ff:ff:ff:ff:ff 2023-05-22T16:05:47.898 INFO:teuthology.orchestra.run.smithi016.stdout: altname enp1s0f0 2023-05-22T16:05:47.898 INFO:teuthology.orchestra.run.smithi016.stdout: inet 172.21.15.16/20 brd 172.21.15.255 scope global dynamic ens1f0 2023-05-22T16:05:47.898 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft 40168sec preferred_lft 40168sec 2023-05-22T16:05:47.899 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::ec4:7aff:febd:1564/64 scope link 2023-05-22T16:05:47.899 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-05-22T16:05:47.899 INFO:teuthology.orchestra.run.smithi016.stdout:5: ens1f1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-05-22T16:05:47.899 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:bd:15:65 brd ff:ff:ff:ff:ff:ff 2023-05-22T16:05:47.899 INFO:teuthology.orchestra.run.smithi016.stdout: altname enp1s0f1 2023-05-22T16:05:47.899 INFO:teuthology.orchestra.run.smithi016.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2023-05-22T16:05:47.899 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 02:42:1f:b8:98:0a brd ff:ff:ff:ff:ff:ff 2023-05-22T16:05:47.900 INFO:teuthology.orchestra.run.smithi016.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2023-05-22T16:05:47.900 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-05-22T16:05:47.900 INFO:teuthology.orchestra.run.smithi016.stdout:9: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-05-22T16:05:47.900 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 5e:1f:84:6d:79:ae brd ff:ff:ff:ff:ff:ff 2023-05-22T16:05:47.900 INFO:teuthology.orchestra.run.smithi016.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-05-22T16:05:47.900 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-05-22T16:05:47.900 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::5c1f:84ff:fe6d:79ae/64 scope link 2023-05-22T16:05:47.901 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-05-22T16:05:47.901 INFO:teuthology.orchestra.run.smithi016.stdout:10: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-05-22T16:05:47.901 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0e:5c:f6:1f:82:2f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-05-22T16:05:47.901 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::c5c:f6ff:fe1f:822f/64 scope link 2023-05-22T16:05:47.901 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-05-22T16:05:47.901 INFO:teuthology.orchestra.run.smithi016.stdout:11: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-05-22T16:05:47.901 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether c6:a6:3d:69:d0:db brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-05-22T16:05:47.901 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::c4a6:3dff:fe69:d0db/64 scope link 2023-05-22T16:05:47.902 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-05-22T16:05:47.902 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T16:05:47.902 DEBUG:teuthology.orchestra.run.smithi016:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-05-22T16:05:47.952 INFO:teuthology.orchestra.run.smithi016.stdout:1 2023-05-22T16:05:47.954 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T16:05:47.954 DEBUG:teuthology.orchestra.run.smithi016:> route 2023-05-22T16:05:48.003 INFO:teuthology.orchestra.run.smithi016.stdout:Kernel IP routing table 2023-05-22T16:05:48.004 INFO:teuthology.orchestra.run.smithi016.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-05-22T16:05:48.004 INFO:teuthology.orchestra.run.smithi016.stdout:default _gateway 0.0.0.0 UG 0 0 0 ens1f0 2023-05-22T16:05:48.004 INFO:teuthology.orchestra.run.smithi016.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2023-05-22T16:05:48.004 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 0 0 0 ens1f0 2023-05-22T16:05:48.004 INFO:teuthology.orchestra.run.smithi016.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-05-22T16:05:48.005 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T16:05:48.005 DEBUG:teuthology.orchestra.run.smithi016:> (cd / && exec bash -c ' 2023-05-22T16:05:48.005 DEBUG:teuthology.orchestra.run.smithi016:> set -e 2023-05-22T16:05:48.006 DEBUG:teuthology.orchestra.run.smithi016:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2023-05-22T16:05:48.006 DEBUG:teuthology.orchestra.run.smithi016:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2023-05-22T16:05:48.006 DEBUG:teuthology.orchestra.run.smithi016:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2023-05-22T16:05:48.007 DEBUG:teuthology.orchestra.run.smithi016:> ') 2023-05-22T16:05:48.085 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T16:05:48.086 DEBUG:teuthology.orchestra.run.smithi016:> ip netns list 2023-05-22T16:05:48.133 INFO:teuthology.orchestra.run.smithi016.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-05-22T16:05:48.134 INFO:teuthology.orchestra.run.smithi016.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-05-22T16:05:48.134 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-05-22T16:05:48.135 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T16:05:48.135 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip link set brx.0 up 2023-05-22T16:05:48.190 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2023-05-22T16:05:48.190 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-05-22T16:05:48.191 DEBUG:teuthology.orchestra.run.smithi016:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-05-22T16:05:48.238 INFO:teuthology.orchestra.run.smithi016.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-05-22T16:05:48.239 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-05-22T16:05:48.239 DEBUG:teuthology.orchestra.run.smithi016:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-05-22T16:05:48.289 DEBUG:teuthology.orchestra.run.smithi016:> sudo modprobe fuse 2023-05-22T16:05:48.347 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-05-22T16:05:48.348 DEBUG:teuthology.orchestra.run.smithi016:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-05-22T16:05:48.401 INFO:teuthology.orchestra.run.smithi016.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-05-22T16:05:48.402 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-05-22T16:05:48.403 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T16:05:48.403 DEBUG:teuthology.orchestra.run.smithi016:> ls /sys/fs/fuse/connections 2023-05-22T16:05:48.449 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2023-05-22T16:05:48.450 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2023-05-22T16:05:48.494 DEBUG:teuthology.orchestra.run.smithi016:> sudo modprobe fuse 2023-05-22T16:05:48.509 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-05-22T16:05:48.510 DEBUG:teuthology.orchestra.run.smithi016:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-05-22T16:05:48.562 INFO:teuthology.orchestra.run.smithi016.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-05-22T16:05:48.563 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-05-22T16:05:48.564 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T16:05:48.565 DEBUG:teuthology.orchestra.run.smithi016:> ls /sys/fs/fuse/connections 2023-05-22T16:05:48.574 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi016.stderr:2023-05-22T16:05:48.572+0000 7f9e0c587580 -1 init, newargv = 0x55e28b890050 newargc=15 2023-05-22T16:05:48.574 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi016.stderr:ceph-fuse[70744]: starting ceph client 2023-05-22T16:05:48.580 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi016.stderr:ceph-fuse[70744]: starting fuse 2023-05-22T16:05:49.679 DEBUG:teuthology.orchestra.run.smithi016:> sudo modprobe fuse 2023-05-22T16:05:49.696 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-05-22T16:05:49.698 DEBUG:teuthology.orchestra.run.smithi016:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-05-22T16:05:49.753 INFO:teuthology.orchestra.run.smithi016.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-05-22T16:05:49.755 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-05-22T16:05:49.755 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T16:05:49.756 DEBUG:teuthology.orchestra.run.smithi016:> ls /sys/fs/fuse/connections 2023-05-22T16:05:49.806 INFO:teuthology.orchestra.run.smithi016.stdout:45 2023-05-22T16:05:49.807 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [45] 2023-05-22T16:05:49.807 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2023-05-22T16:05:49.807 DEBUG:teuthology.orchestra.run.smithi016:> import glob 2023-05-22T16:05:49.808 DEBUG:teuthology.orchestra.run.smithi016:> import re 2023-05-22T16:05:49.808 DEBUG:teuthology.orchestra.run.smithi016:> import os 2023-05-22T16:05:49.808 DEBUG:teuthology.orchestra.run.smithi016:> import subprocess 2023-05-22T16:05:49.809 DEBUG:teuthology.orchestra.run.smithi016:> 2023-05-22T16:05:49.809 DEBUG:teuthology.orchestra.run.smithi016:> def _find_admin_socket(client_name): 2023-05-22T16:05:49.809 DEBUG:teuthology.orchestra.run.smithi016:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2023-05-22T16:05:49.810 DEBUG:teuthology.orchestra.run.smithi016:> files = glob.glob(asok_path) 2023-05-22T16:05:49.810 DEBUG:teuthology.orchestra.run.smithi016:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2023-05-22T16:05:49.810 DEBUG:teuthology.orchestra.run.smithi016:> 2023-05-22T16:05:49.810 DEBUG:teuthology.orchestra.run.smithi016:> # Given a non-glob path, it better be there 2023-05-22T16:05:49.811 DEBUG:teuthology.orchestra.run.smithi016:> if "*" not in asok_path: 2023-05-22T16:05:49.811 DEBUG:teuthology.orchestra.run.smithi016:> assert(len(files) == 1) 2023-05-22T16:05:49.811 DEBUG:teuthology.orchestra.run.smithi016:> return files[0] 2023-05-22T16:05:49.812 DEBUG:teuthology.orchestra.run.smithi016:> 2023-05-22T16:05:49.812 DEBUG:teuthology.orchestra.run.smithi016:> for f in files: 2023-05-22T16:05:49.812 DEBUG:teuthology.orchestra.run.smithi016:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-05-22T16:05:49.813 DEBUG:teuthology.orchestra.run.smithi016:> if os.path.exists("/proc/{0}".format(pid)): 2023-05-22T16:05:49.813 DEBUG:teuthology.orchestra.run.smithi016:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-05-22T16:05:49.813 DEBUG:teuthology.orchestra.run.smithi016:> contents = proc_f.read() 2023-05-22T16:05:49.813 DEBUG:teuthology.orchestra.run.smithi016:> if mountpoint in contents: 2023-05-22T16:05:49.814 DEBUG:teuthology.orchestra.run.smithi016:> return f 2023-05-22T16:05:49.814 DEBUG:teuthology.orchestra.run.smithi016:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-05-22T16:05:49.814 DEBUG:teuthology.orchestra.run.smithi016:> 2023-05-22T16:05:49.815 DEBUG:teuthology.orchestra.run.smithi016:> print(_find_admin_socket("client.0")) 2023-05-22T16:05:49.815 DEBUG:teuthology.orchestra.run.smithi016:> ' 2023-05-22T16:05:49.931 INFO:teuthology.orchestra.run.smithi016.stdout:/var/run/ceph/ceph-client.0.70744.asok 2023-05-22T16:05:50.113 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.70744.asok 2023-05-22T16:05:50.113 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T16:05:50.114 DEBUG:teuthology.orchestra.run.smithi016:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.70744.asok status 2023-05-22T16:05:50.272 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-05-22T16:05:50.272 INFO:teuthology.orchestra.run.smithi016.stdout: "metadata": { 2023-05-22T16:05:50.273 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph_sha1": "9c9ba4df22024d9d8d6cf9f98a4b96751fc78d0c", 2023-05-22T16:05:50.273 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph_version": "ceph version 18.0.0-3192-g9c9ba4df (9c9ba4df22024d9d8d6cf9f98a4b96751fc78d0c) reef (dev)", 2023-05-22T16:05:50.273 INFO:teuthology.orchestra.run.smithi016.stdout: "entity_id": "0", 2023-05-22T16:05:50.273 INFO:teuthology.orchestra.run.smithi016.stdout: "hostname": "smithi016", 2023-05-22T16:05:50.273 INFO:teuthology.orchestra.run.smithi016.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2023-05-22T16:05:50.273 INFO:teuthology.orchestra.run.smithi016.stdout: "pid": "70744", 2023-05-22T16:05:50.273 INFO:teuthology.orchestra.run.smithi016.stdout: "root": "/" 2023-05-22T16:05:50.274 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T16:05:50.274 INFO:teuthology.orchestra.run.smithi016.stdout: "dentry_count": 0, 2023-05-22T16:05:50.274 INFO:teuthology.orchestra.run.smithi016.stdout: "dentry_pinned_count": 0, 2023-05-22T16:05:50.274 INFO:teuthology.orchestra.run.smithi016.stdout: "id": 11130, 2023-05-22T16:05:50.274 INFO:teuthology.orchestra.run.smithi016.stdout: "inst": { 2023-05-22T16:05:50.274 INFO:teuthology.orchestra.run.smithi016.stdout: "name": { 2023-05-22T16:05:50.274 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "client", 2023-05-22T16:05:50.274 INFO:teuthology.orchestra.run.smithi016.stdout: "num": 11130 2023-05-22T16:05:50.275 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T16:05:50.275 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": { 2023-05-22T16:05:50.275 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-05-22T16:05:50.275 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "192.168.0.1:0", 2023-05-22T16:05:50.275 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 338358933 2023-05-22T16:05:50.275 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-05-22T16:05:50.275 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T16:05:50.275 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": { 2023-05-22T16:05:50.276 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-05-22T16:05:50.276 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "192.168.0.1:0", 2023-05-22T16:05:50.276 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 338358933 2023-05-22T16:05:50.276 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-05-22T16:05:50.276 INFO:teuthology.orchestra.run.smithi016.stdout: "inst_str": "client.11130 192.168.0.1:0/338358933", 2023-05-22T16:05:50.276 INFO:teuthology.orchestra.run.smithi016.stdout: "addr_str": "192.168.0.1:0/338358933", 2023-05-22T16:05:50.276 INFO:teuthology.orchestra.run.smithi016.stdout: "inode_count": 1, 2023-05-22T16:05:50.277 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_epoch": 81, 2023-05-22T16:05:50.277 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_epoch": 197, 2023-05-22T16:05:50.277 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_epoch_barrier": 0, 2023-05-22T16:05:50.277 INFO:teuthology.orchestra.run.smithi016.stdout: "blocklisted": false, 2023-05-22T16:05:50.277 INFO:teuthology.orchestra.run.smithi016.stdout: "fs_name": "cephfs" 2023-05-22T16:05:50.277 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-05-22T16:05:50.287 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T16:05:50.287 DEBUG:teuthology.orchestra.run.smithi016:> stat --file-system '--printf=%T 2023-05-22T16:05:50.288 DEBUG:teuthology.orchestra.run.smithi016:> ' -- /home/ubuntu/cephtest/mnt.0 2023-05-22T16:05:50.294 INFO:teuthology.orchestra.run.smithi016.stdout:fuseblk 2023-05-22T16:05:50.295 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2023-05-22T16:05:50.295 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T16:05:50.295 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-05-22T16:05:50.348 INFO:tasks.cephfs.test_snap_schedules:expecting snap snap_test_dir1/.snap/scheduled-2023-05-22-16_06_00 in ~23s... 2023-05-22T16:05:50.348 DEBUG:tasks.cephfs.test_snap_schedules:expected_schedule: ['1M'], expected_retention: [{'M': 1}] 2023-05-22T16:05:50.348 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule list --fs cephfs --format json snap_test_dir1 2023-05-22T16:05:50.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:50.519+0000 7f743d713640 1 Processor -- start 2023-05-22T16:05:50.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:50.519+0000 7f743d713640 1 -- start start 2023-05-22T16:05:50.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:50.523+0000 7f743d713640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f74380b3f70 0x7f74380b43d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:50.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:50.523+0000 7f743d713640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f74380d03d0 0x7f74380d0810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:50.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:50.523+0000 7f743d713640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74380b4910 con 0x7f74380b7ba0 2023-05-22T16:05:50.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:50.523+0000 7f743d713640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74380ccab0 con 0x7f74380d03d0 2023-05-22T16:05:50.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:50.523+0000 7f743d713640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74380cd310 con 0x7f74380b3f70 2023-05-22T16:05:50.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:50.523+0000 7f74377fe640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f74380d03d0 0x7f74380d0810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:50.526 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:50.523+0000 7f74377fe640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f74380d03d0 0x7f74380d0810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38256/0 (socket says 172.21.15.16:38256) 2023-05-22T16:05:50.526 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:50.523+0000 7f74377fe640 1 -- 172.21.15.16:0/3136119811 learned_addr learned my addr 172.21.15.16:0/3136119811 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:05:50.526 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:50.523+0000 7f74367fc640 1 --2- 172.21.15.16:0/3136119811 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f74380b3f70 0x7f74380b43d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:50.526 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:50.523+0000 7f74377fe640 1 -- 172.21.15.16:0/3136119811 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f74380b3f70 msgr2=0x7f74380b43d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:05:50.527 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:50.523+0000 7f74377fe640 1 --2- 172.21.15.16:0/3136119811 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f74380b3f70 0x7f74380b43d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:50.527 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:50.523+0000 7f74377fe640 1 -- 172.21.15.16:0/3136119811 >> v1:172.21.15.62:6789/0 conn(0x7f74380b7ba0 legacy=0x7f74380ce7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:50.527 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:50.523+0000 7f74377fe640 1 -- 172.21.15.16:0/3136119811 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7438160820 con 0x7f74380d03d0 2023-05-22T16:05:50.527 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:50.523+0000 7f74367fc640 1 --2- 172.21.15.16:0/3136119811 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f74380b3f70 0x7f74380b43d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T16:05:50.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:50.523+0000 7f74377fe640 1 --2- 172.21.15.16:0/3136119811 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f74380d03d0 0x7f74380d0810 secure :-1 s=READY pgs=2535 cs=0 l=1 rev1=1 crypto rx=0x7f7430009e50 tx=0x7f743001c9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28480ddd03ecc580 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:50.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:50.523+0000 7f7435ffb640 1 -- 172.21.15.16:0/3136119811 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74300a58a0 con 0x7f74380d03d0 2023-05-22T16:05:50.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:50.523+0000 7f7435ffb640 1 -- 172.21.15.16:0/3136119811 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74300a5a00 con 0x7f74380d03d0 2023-05-22T16:05:50.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:50.523+0000 7f7435ffb640 1 -- 172.21.15.16:0/3136119811 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74300a5d90 con 0x7f74380d03d0 2023-05-22T16:05:50.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:50.523+0000 7f743d713640 1 -- 172.21.15.16:0/3136119811 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f74380d03d0 msgr2=0x7f74380d0810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:50.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:50.523+0000 7f743d713640 1 --2- 172.21.15.16:0/3136119811 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f74380d03d0 0x7f74380d0810 secure :-1 s=READY pgs=2535 cs=0 l=1 rev1=1 crypto rx=0x7f7430009e50 tx=0x7f743001c9e0 comp rx=0 tx=0).stop 2023-05-22T16:05:50.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:50.523+0000 7f743d713640 1 -- 172.21.15.16:0/3136119811 shutdown_connections 2023-05-22T16:05:50.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:50.523+0000 7f743d713640 1 --2- 172.21.15.16:0/3136119811 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f74380d03d0 0x7f74380d0810 unknown :-1 s=CLOSED pgs=2535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:50.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:50.523+0000 7f743d713640 1 --2- 172.21.15.16:0/3136119811 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f74380b3f70 0x7f74380b43d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:50.530 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:50.523+0000 7f743d713640 1 -- 172.21.15.16:0/3136119811 >> 172.21.15.16:0/3136119811 conn(0x7f74380d24f0 msgr2=0x7f74380d28f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:05:50.530 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:50.527+0000 7f743d713640 1 -- 172.21.15.16:0/3136119811 shutdown_connections 2023-05-22T16:05:50.530 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:50.527+0000 7f743d713640 1 -- 172.21.15.16:0/3136119811 wait complete. 2023-05-22T16:05:50.532 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:50.527+0000 7f743d713640 1 Processor -- start 2023-05-22T16:05:50.532 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:50.527+0000 7f743d713640 1 -- start start 2023-05-22T16:05:50.532 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:50.527+0000 7f743d713640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f74380b3f70 0x7f74380c8600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:50.532 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:50.527+0000 7f743d713640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f74380b7ba0 0x7f743815e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:50.532 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:50.527+0000 7f743d713640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f74380d03d0 0x7f743815e560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:50.532 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:50.527+0000 7f743d713640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74380b6b10 con 0x7f74380b7ba0 2023-05-22T16:05:50.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:50.527+0000 7f743d713640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f743815ae70 con 0x7f74380d03d0 2023-05-22T16:05:50.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:50.527+0000 7f743d713640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f743815b150 con 0x7f74380b3f70 2023-05-22T16:05:50.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:50.527+0000 7f74367fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f74380b7ba0 0x7f743815e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:50.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:50.527+0000 7f74367fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f74380b7ba0 0x7f743815e020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38272/0 (socket says 172.21.15.16:38272) 2023-05-22T16:05:50.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:50.527+0000 7f74367fc640 1 -- 172.21.15.16:0/1405328773 learned_addr learned my addr 172.21.15.16:0/1405328773 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:05:50.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:50.527+0000 7f74377fe640 1 --2- 172.21.15.16:0/1405328773 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f74380d03d0 0x7f743815e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:50.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:50.527+0000 7f7436ffd640 1 --2- 172.21.15.16:0/1405328773 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f74380b3f70 0x7f74380c8600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:50.534 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:50.527+0000 7f74367fc640 1 -- 172.21.15.16:0/1405328773 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f74380b3f70 msgr2=0x7f74380c8600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:50.534 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:50.527+0000 7f74367fc640 1 --2- 172.21.15.16:0/1405328773 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f74380b3f70 0x7f74380c8600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:50.534 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:50.527+0000 7f74367fc640 1 -- 172.21.15.16:0/1405328773 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f74380d03d0 msgr2=0x7f743815e560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:50.534 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:50.527+0000 7f74367fc640 1 --2- 172.21.15.16:0/1405328773 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f74380d03d0 0x7f743815e560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:50.534 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:50.527+0000 7f74367fc640 1 -- 172.21.15.16:0/1405328773 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7430009b30 con 0x7f74380b7ba0 2023-05-22T16:05:50.534 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:50.527+0000 7f74367fc640 1 --2- 172.21.15.16:0/1405328773 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f74380b7ba0 0x7f743815e020 secure :-1 s=READY pgs=2536 cs=0 l=1 rev1=1 crypto rx=0x7f742c00e810 tx=0x7f742c00ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:50.534 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:50.527+0000 7f741f7fe640 1 -- 172.21.15.16:0/1405328773 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f742c00c8d0 con 0x7f74380b7ba0 2023-05-22T16:05:50.534 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:50.527+0000 7f741f7fe640 1 -- 172.21.15.16:0/1405328773 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f742c00ca30 con 0x7f74380b7ba0 2023-05-22T16:05:50.535 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:50.527+0000 7f743d713640 1 -- 172.21.15.16:0/1405328773 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7438160460 con 0x7f74380b7ba0 2023-05-22T16:05:50.535 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:50.527+0000 7f741f7fe640 1 -- 172.21.15.16:0/1405328773 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f742c00cde0 con 0x7f74380b7ba0 2023-05-22T16:05:50.535 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:50.527+0000 7f743d713640 1 -- 172.21.15.16:0/1405328773 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74380b1760 con 0x7f74380b7ba0 2023-05-22T16:05:50.536 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:50.527+0000 7f743d713640 1 -- 172.21.15.16:0/1405328773 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74381605f0 con 0x7f74380b7ba0 2023-05-22T16:05:50.536 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:50.531+0000 7f741f7fe640 1 -- 172.21.15.16:0/1405328773 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 96) v1 ==== 84406+0+0 (secure 0 0 0) 0x7f742c013410 con 0x7f74380b7ba0 2023-05-22T16:05:50.536 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:50.531+0000 7f741f7fe640 1 --2- 172.21.15.16:0/1405328773 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f7424065580 0x7f7424067a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:50.536 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:50.531+0000 7f741f7fe640 1 -- 172.21.15.16:0/1405328773 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(197..197 src has 1..197) v4 ==== 6782+0+0 (secure 0 0 0) 0x7f742c082ee0 con 0x7f74380b7ba0 2023-05-22T16:05:50.536 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:50.531+0000 7f7436ffd640 1 --2- 172.21.15.16:0/1405328773 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f7424065580 0x7f7424067a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:50.536 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:50.531+0000 7f7436ffd640 1 --2- 172.21.15.16:0/1405328773 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f7424065580 0x7f7424067a30 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f74380c9460 tx=0x7f7428009040 comp rx=0 tx=0).ready entity=mgr.10974 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:50.537 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:50.535+0000 7f741f7fe640 1 -- 172.21.15.16:0/1405328773 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f742c0504a0 con 0x7f74380b7ba0 2023-05-22T16:05:50.711 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:50.707+0000 7f743d713640 1 -- 172.21.15.16:0/1405328773 --> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "snap_test_dir1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f743815b2c0 con 0x7f7424065580 2023-05-22T16:05:50.721 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:50.719+0000 7f741f7fe640 1 -- 172.21.15.16:0/1405328773 <== mgr.10974 v2:172.21.15.16:6832/65890 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+72 (secure 0 0 0) 0x7f743815b2c0 con 0x7f7424065580 2023-05-22T16:05:50.721 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T16:05:50.721 INFO:teuthology.orchestra.run.smithi016.stdout:{"path": "snap_test_dir1", "schedule": ["1M"], "retention": [{"M": 1}]} 2023-05-22T16:05:50.723 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:50.719+0000 7f743d713640 1 -- 172.21.15.16:0/1405328773 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f7424065580 msgr2=0x7f7424067a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:50.723 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:50.719+0000 7f743d713640 1 --2- 172.21.15.16:0/1405328773 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f7424065580 0x7f7424067a30 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f74380c9460 tx=0x7f7428009040 comp rx=0 tx=0).stop 2023-05-22T16:05:50.723 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:50.719+0000 7f743d713640 1 -- 172.21.15.16:0/1405328773 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f74380b7ba0 msgr2=0x7f743815e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:50.723 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:50.719+0000 7f743d713640 1 --2- 172.21.15.16:0/1405328773 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f74380b7ba0 0x7f743815e020 secure :-1 s=READY pgs=2536 cs=0 l=1 rev1=1 crypto rx=0x7f742c00e810 tx=0x7f742c00ecd0 comp rx=0 tx=0).stop 2023-05-22T16:05:50.724 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:50.719+0000 7f743d713640 1 -- 172.21.15.16:0/1405328773 shutdown_connections 2023-05-22T16:05:50.724 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:50.719+0000 7f743d713640 1 --2- 172.21.15.16:0/1405328773 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f74380d03d0 0x7f743815e560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:50.724 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:50.719+0000 7f743d713640 1 --2- 172.21.15.16:0/1405328773 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f7424065580 0x7f7424067a30 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:50.724 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:50.719+0000 7f743d713640 1 --2- 172.21.15.16:0/1405328773 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f74380b7ba0 0x7f743815e020 unknown :-1 s=CLOSED pgs=2536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:50.724 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:50.719+0000 7f743d713640 1 --2- 172.21.15.16:0/1405328773 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f74380b3f70 0x7f74380c8600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:50.724 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:50.719+0000 7f743d713640 1 -- 172.21.15.16:0/1405328773 >> 172.21.15.16:0/1405328773 conn(0x7f74380d24f0 msgr2=0x7f7438155ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:05:50.725 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:50.719+0000 7f743d713640 1 -- 172.21.15.16:0/1405328773 shutdown_connections 2023-05-22T16:05:50.725 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:50.719+0000 7f743d713640 1 -- 172.21.15.16:0/1405328773 wait complete. 2023-05-22T16:05:50.741 DEBUG:tasks.cephfs.test_snap_schedules:res= {"path": "snap_test_dir1", "schedule": ["1M"], "retention": [{"M": 1}]} 2023-05-22T16:05:50.741 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'path': 'snap_test_dir1', 'schedule': ['1M'], 'retention': [{'M': 1}]} 2023-05-22T16:05:50.742 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove --fs cephfs snap_test_dir1 2023-05-22T16:05:50.874 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:50.871+0000 7f5a233cb640 1 Processor -- start 2023-05-22T16:05:50.875 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:50.871+0000 7f5a233cb640 1 -- start start 2023-05-22T16:05:50.875 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:50.871+0000 7f5a233cb640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5a1c15bec0 0x7f5a1c0b53d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:50.875 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:50.871+0000 7f5a233cb640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f5a1c15c290 0x7f5a1c0b1120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:50.875 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:50.871+0000 7f5a233cb640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a1c0b1e30 con 0x7f5a1c0b1660 2023-05-22T16:05:50.875 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:50.871+0000 7f5a233cb640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a1c0b1b50 con 0x7f5a1c15c290 2023-05-22T16:05:50.876 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:50.871+0000 7f5a233cb640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a1c0d0490 con 0x7f5a1c15bec0 2023-05-22T16:05:50.876 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:50.871+0000 7f5a21167640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5a1c15bec0 0x7f5a1c0b53d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:50.876 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:50.871+0000 7f5a21167640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5a1c15bec0 0x7f5a1c0b53d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3301/0 says I am v2:172.21.15.16:42236/0 (socket says 172.21.15.16:42236) 2023-05-22T16:05:50.877 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:50.871+0000 7f5a21167640 1 -- 172.21.15.16:0/4209394885 learned_addr learned my addr 172.21.15.16:0/4209394885 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:05:50.877 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:50.871+0000 7f5a20966640 1 --2- 172.21.15.16:0/4209394885 >> v2:172.21.15.62:3300/0 conn(0x7f5a1c15c290 0x7f5a1c0b1120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:50.877 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:50.871+0000 7f5a21167640 1 -- 172.21.15.16:0/4209394885 >> v2:172.21.15.62:3300/0 conn(0x7f5a1c15c290 msgr2=0x7f5a1c0b1120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:05:50.877 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:50.871+0000 7f5a21167640 1 --2- 172.21.15.16:0/4209394885 >> v2:172.21.15.62:3300/0 conn(0x7f5a1c15c290 0x7f5a1c0b1120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:50.877 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:50.871+0000 7f5a21167640 1 -- 172.21.15.16:0/4209394885 >> v1:172.21.15.62:6789/0 conn(0x7f5a1c0b1660 legacy=0x7f5a1c0b4000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:50.877 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:50.871+0000 7f5a21167640 1 -- 172.21.15.16:0/4209394885 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a1c0d0770 con 0x7f5a1c15bec0 2023-05-22T16:05:50.878 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:50.875+0000 7f5a21167640 1 --2- 172.21.15.16:0/4209394885 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5a1c15bec0 0x7f5a1c0b53d0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f5a18009870 tx=0x7f5a1801c480 comp rx=0 tx=0).ready entity=mon.2 client_cookie=65ab62a91708b062 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:50.879 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:50.875+0000 7f5a13fff640 1 -- 172.21.15.16:0/4209394885 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a180a6040 con 0x7f5a1c15bec0 2023-05-22T16:05:50.879 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:50.875+0000 7f5a13fff640 1 -- 172.21.15.16:0/4209394885 <== mon.2 v2:172.21.15.62:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a180aa600 con 0x7f5a1c15bec0 2023-05-22T16:05:50.879 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:50.875+0000 7f5a13fff640 1 -- 172.21.15.16:0/4209394885 <== mon.2 v2:172.21.15.62:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a180aa990 con 0x7f5a1c15bec0 2023-05-22T16:05:50.880 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:50.875+0000 7f5a233cb640 1 -- 172.21.15.16:0/4209394885 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5a1c15bec0 msgr2=0x7f5a1c0b53d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:50.880 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:50.875+0000 7f5a233cb640 1 --2- 172.21.15.16:0/4209394885 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5a1c15bec0 0x7f5a1c0b53d0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f5a18009870 tx=0x7f5a1801c480 comp rx=0 tx=0).stop 2023-05-22T16:05:50.880 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:50.875+0000 7f5a233cb640 1 -- 172.21.15.16:0/4209394885 shutdown_connections 2023-05-22T16:05:50.881 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:50.875+0000 7f5a233cb640 1 --2- 172.21.15.16:0/4209394885 >> v2:172.21.15.62:3300/0 conn(0x7f5a1c15c290 0x7f5a1c0b1120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:50.881 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:50.875+0000 7f5a233cb640 1 --2- 172.21.15.16:0/4209394885 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5a1c15bec0 0x7f5a1c0b53d0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:50.881 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:50.875+0000 7f5a233cb640 1 -- 172.21.15.16:0/4209394885 >> 172.21.15.16:0/4209394885 conn(0x7f5a1c0cf000 msgr2=0x7f5a1c0cf400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:05:50.881 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:50.875+0000 7f5a233cb640 1 -- 172.21.15.16:0/4209394885 shutdown_connections 2023-05-22T16:05:50.881 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:50.875+0000 7f5a233cb640 1 -- 172.21.15.16:0/4209394885 wait complete. 2023-05-22T16:05:50.882 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:50.879+0000 7f5a233cb640 1 Processor -- start 2023-05-22T16:05:50.882 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:50.879+0000 7f5a233cb640 1 -- start start 2023-05-22T16:05:50.882 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:50.879+0000 7f5a233cb640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5a1c0b1660 0x7f5a1c149530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:50.883 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:50.879+0000 7f5a233cb640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5a1c15bec0 0x7f5a1c14ad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:50.883 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:50.879+0000 7f5a233cb640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5a1c14bdc0 0x7f5a1c0c5660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:50.883 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:50.879+0000 7f5a233cb640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a1c149a70 con 0x7f5a1c0b1660 2023-05-22T16:05:50.883 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:50.879+0000 7f5a233cb640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a1c0b4690 con 0x7f5a1c14bdc0 2023-05-22T16:05:50.883 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:50.879+0000 7f5a233cb640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a1c0c51d0 con 0x7f5a1c15bec0 2023-05-22T16:05:50.883 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:50.879+0000 7f5a21167640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5a1c0b1660 0x7f5a1c149530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:50.883 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:50.879+0000 7f5a21167640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5a1c0b1660 0x7f5a1c149530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38278/0 (socket says 172.21.15.16:38278) 2023-05-22T16:05:50.884 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:50.879+0000 7f5a21167640 1 -- 172.21.15.16:0/80128173 learned_addr learned my addr 172.21.15.16:0/80128173 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:05:50.884 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:50.879+0000 7f5a20966640 1 --2- 172.21.15.16:0/80128173 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5a1c15bec0 0x7f5a1c14ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:50.884 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:50.879+0000 7f5a21968640 1 --2- 172.21.15.16:0/80128173 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5a1c14bdc0 0x7f5a1c0c5660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:50.884 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:50.879+0000 7f5a21167640 1 -- 172.21.15.16:0/80128173 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5a1c15bec0 msgr2=0x7f5a1c14ad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:50.884 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:50.879+0000 7f5a21167640 1 --2- 172.21.15.16:0/80128173 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5a1c15bec0 0x7f5a1c14ad70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:50.884 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:50.879+0000 7f5a21167640 1 -- 172.21.15.16:0/80128173 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5a1c14bdc0 msgr2=0x7f5a1c0c5660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:50.884 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:50.879+0000 7f5a21167640 1 --2- 172.21.15.16:0/80128173 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5a1c14bdc0 0x7f5a1c0c5660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:50.885 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:50.879+0000 7f5a21167640 1 -- 172.21.15.16:0/80128173 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a180094d0 con 0x7f5a1c0b1660 2023-05-22T16:05:50.886 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:50.879+0000 7f5a21167640 1 --2- 172.21.15.16:0/80128173 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5a1c0b1660 0x7f5a1c149530 secure :-1 s=READY pgs=2537 cs=0 l=1 rev1=1 crypto rx=0x7f5a1801ab30 tx=0x7f5a1801acc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:50.886 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:50.879+0000 7f5a11ffb640 1 -- 172.21.15.16:0/80128173 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a180b3600 con 0x7f5a1c0b1660 2023-05-22T16:05:50.886 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:50.879+0000 7f5a11ffb640 1 -- 172.21.15.16:0/80128173 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a180b3780 con 0x7f5a1c0b1660 2023-05-22T16:05:50.887 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:50.879+0000 7f5a233cb640 1 -- 172.21.15.16:0/80128173 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a1c0b5aa0 con 0x7f5a1c0b1660 2023-05-22T16:05:50.887 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:50.879+0000 7f5a11ffb640 1 -- 172.21.15.16:0/80128173 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a180b3b30 con 0x7f5a1c0b1660 2023-05-22T16:05:50.887 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:50.879+0000 7f5a233cb640 1 -- 172.21.15.16:0/80128173 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a1c0b5910 con 0x7f5a1c0b1660 2023-05-22T16:05:50.887 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:50.879+0000 7f5a233cb640 1 -- 172.21.15.16:0/80128173 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a1c0d1780 con 0x7f5a1c0b1660 2023-05-22T16:05:50.887 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:50.883+0000 7f5a11ffb640 1 -- 172.21.15.16:0/80128173 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 96) v1 ==== 84406+0+0 (secure 0 0 0) 0x7f5a180b3c90 con 0x7f5a1c0b1660 2023-05-22T16:05:50.887 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:50.883+0000 7f5a11ffb640 1 --2- 172.21.15.16:0/80128173 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f5a000655b0 0x7f5a00067a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:50.887 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:50.883+0000 7f5a11ffb640 1 -- 172.21.15.16:0/80128173 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(197..197 src has 1..197) v4 ==== 6782+0+0 (secure 0 0 0) 0x7f5a1811d7e0 con 0x7f5a1c0b1660 2023-05-22T16:05:50.888 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:50.883+0000 7f5a20966640 1 --2- 172.21.15.16:0/80128173 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f5a000655b0 0x7f5a00067a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:50.888 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:50.883+0000 7f5a20966640 1 --2- 172.21.15.16:0/80128173 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f5a000655b0 0x7f5a00067a60 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f5a1c155d60 tx=0x7f5a0c008040 comp rx=0 tx=0).ready entity=mgr.10974 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:50.889 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:50.887+0000 7f5a11ffb640 1 -- 172.21.15.16:0/80128173 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f5a180ead70 con 0x7f5a1c0b1660 2023-05-22T16:05:51.062 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:51.059+0000 7f5a233cb640 1 -- 172.21.15.16:0/80128173 --> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "fs": "cephfs", "path": "snap_test_dir1", "target": ["mon-mgr", ""]}) v1 -- 0x7f5a1c066b30 con 0x7f5a000655b0 2023-05-22T16:05:51.188 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:51.183+0000 7f5a11ffb640 1 -- 172.21.15.16:0/80128173 <== mgr.10974 v2:172.21.15.16:6832/65890 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f5a1c066b30 con 0x7f5a000655b0 2023-05-22T16:05:51.188 INFO:teuthology.orchestra.run.smithi016.stdout:Schedule removed for path snap_test_dir1 2023-05-22T16:05:51.192 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:51.187+0000 7f5a233cb640 1 -- 172.21.15.16:0/80128173 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f5a000655b0 msgr2=0x7f5a00067a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:51.192 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:51.187+0000 7f5a233cb640 1 --2- 172.21.15.16:0/80128173 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f5a000655b0 0x7f5a00067a60 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f5a1c155d60 tx=0x7f5a0c008040 comp rx=0 tx=0).stop 2023-05-22T16:05:51.193 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:51.187+0000 7f5a233cb640 1 -- 172.21.15.16:0/80128173 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5a1c0b1660 msgr2=0x7f5a1c149530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:51.193 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:51.187+0000 7f5a233cb640 1 --2- 172.21.15.16:0/80128173 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5a1c0b1660 0x7f5a1c149530 secure :-1 s=READY pgs=2537 cs=0 l=1 rev1=1 crypto rx=0x7f5a1801ab30 tx=0x7f5a1801acc0 comp rx=0 tx=0).stop 2023-05-22T16:05:51.193 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:51.187+0000 7f5a233cb640 1 -- 172.21.15.16:0/80128173 shutdown_connections 2023-05-22T16:05:51.193 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:51.187+0000 7f5a233cb640 1 --2- 172.21.15.16:0/80128173 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5a1c14bdc0 0x7f5a1c0c5660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:51.194 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:51.187+0000 7f5a233cb640 1 --2- 172.21.15.16:0/80128173 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5a1c15bec0 0x7f5a1c14ad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:51.194 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:51.187+0000 7f5a233cb640 1 --2- 172.21.15.16:0/80128173 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f5a000655b0 0x7f5a00067a60 secure :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f5a1c155d60 tx=0x7f5a0c008040 comp rx=0 tx=0).stop 2023-05-22T16:05:51.194 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:51.187+0000 7f5a233cb640 1 --2- 172.21.15.16:0/80128173 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5a1c0b1660 0x7f5a1c149530 unknown :-1 s=CLOSED pgs=2537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:51.194 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:51.187+0000 7f5a233cb640 1 -- 172.21.15.16:0/80128173 >> 172.21.15.16:0/80128173 conn(0x7f5a1c0cf000 msgr2=0x7f5a1c0b66d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:05:51.195 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:51.187+0000 7f5a233cb640 1 -- 172.21.15.16:0/80128173 shutdown_connections 2023-05-22T16:05:51.195 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:51.187+0000 7f5a233cb640 1 -- 172.21.15.16:0/80128173 wait complete. 2023-05-22T16:05:51.211 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path snap_test_dir1 2023-05-22T16:05:51.212 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T16:05:51.212 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-05-22T16:05:51.224 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T16:05:51.224 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1) 2023-05-22T16:05:51.274 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2023-05-22T16:05:51.449 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:51.447+0000 7fd6821e5640 1 Processor -- start 2023-05-22T16:05:51.449 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:51.447+0000 7fd6821e5640 1 -- start start 2023-05-22T16:05:51.450 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:51.447+0000 7fd6821e5640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fd67c15d3d0 0x7fd67c0cde00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:51.450 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:51.447+0000 7fd6821e5640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd67c0ca880 0x7fd67c0ce340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:51.451 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:51.447+0000 7fd6821e5640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd67c0cb7a0 con 0x7fd67c0ce880 2023-05-22T16:05:51.451 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:51.447+0000 7fd6821e5640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd67c0afa60 con 0x7fd67c0ca880 2023-05-22T16:05:51.451 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:51.447+0000 7fd6821e5640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd67c0b2770 con 0x7fd67c15d3d0 2023-05-22T16:05:51.451 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:51.447+0000 7fd67affd640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd67c0ca880 0x7fd67c0ce340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:51.451 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:51.447+0000 7fd67affd640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd67c0ca880 0x7fd67c0ce340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38292/0 (socket says 172.21.15.16:38292) 2023-05-22T16:05:51.451 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:51.447+0000 7fd67affd640 1 -- 172.21.15.16:0/2307883452 learned_addr learned my addr 172.21.15.16:0/2307883452 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:05:51.451 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:51.447+0000 7fd67b7fe640 1 --2- 172.21.15.16:0/2307883452 >> v2:172.21.15.62:3300/0 conn(0x7fd67c15d3d0 0x7fd67c0cde00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:51.452 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:51.447+0000 7fd67affd640 1 -- 172.21.15.16:0/2307883452 >> v2:172.21.15.62:3300/0 conn(0x7fd67c15d3d0 msgr2=0x7fd67c0cde00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:05:51.452 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:51.447+0000 7fd67affd640 1 --2- 172.21.15.16:0/2307883452 >> v2:172.21.15.62:3300/0 conn(0x7fd67c15d3d0 0x7fd67c0cde00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:51.452 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:51.447+0000 7fd67affd640 1 -- 172.21.15.16:0/2307883452 >> v1:172.21.15.62:6789/0 conn(0x7fd67c0ce880 legacy=0x7fd67c15d8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:51.452 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:51.447+0000 7fd67affd640 1 -- 172.21.15.16:0/2307883452 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd67c0cc600 con 0x7fd67c0ca880 2023-05-22T16:05:51.453 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:51.451+0000 7fd67affd640 1 --2- 172.21.15.16:0/2307883452 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd67c0ca880 0x7fd67c0ce340 secure :-1 s=READY pgs=2538 cs=0 l=1 rev1=1 crypto rx=0x7fd67c08a7e0 tx=0x7fd67001c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4714545ef30e4ff8 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:51.453 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:51.451+0000 7fd67a7fc640 1 -- 172.21.15.16:0/2307883452 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6700ab600 con 0x7fd67c0ca880 2023-05-22T16:05:51.454 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:51.451+0000 7fd67a7fc640 1 -- 172.21.15.16:0/2307883452 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6700a7050 con 0x7fd67c0ca880 2023-05-22T16:05:51.454 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:51.451+0000 7fd67a7fc640 1 -- 172.21.15.16:0/2307883452 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6700ab980 con 0x7fd67c0ca880 2023-05-22T16:05:51.454 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:51.451+0000 7fd6821e5640 1 -- 172.21.15.16:0/2307883452 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd67c0ca880 msgr2=0x7fd67c0ce340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:51.455 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:51.451+0000 7fd6821e5640 1 --2- 172.21.15.16:0/2307883452 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd67c0ca880 0x7fd67c0ce340 secure :-1 s=READY pgs=2538 cs=0 l=1 rev1=1 crypto rx=0x7fd67c08a7e0 tx=0x7fd67001c500 comp rx=0 tx=0).stop 2023-05-22T16:05:51.455 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:51.451+0000 7fd6821e5640 1 -- 172.21.15.16:0/2307883452 shutdown_connections 2023-05-22T16:05:51.455 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:51.451+0000 7fd6821e5640 1 --2- 172.21.15.16:0/2307883452 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd67c0ca880 0x7fd67c0ce340 unknown :-1 s=CLOSED pgs=2538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:51.455 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:51.451+0000 7fd6821e5640 1 --2- 172.21.15.16:0/2307883452 >> v2:172.21.15.62:3300/0 conn(0x7fd67c15d3d0 0x7fd67c0cde00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:51.455 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:51.451+0000 7fd6821e5640 1 -- 172.21.15.16:0/2307883452 >> 172.21.15.16:0/2307883452 conn(0x7fd67c099620 msgr2=0x7fd67c0d02c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:05:51.455 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:51.451+0000 7fd6821e5640 1 -- 172.21.15.16:0/2307883452 shutdown_connections 2023-05-22T16:05:51.455 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:51.451+0000 7fd6821e5640 1 -- 172.21.15.16:0/2307883452 wait complete. 2023-05-22T16:05:51.457 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:51.451+0000 7fd6821e5640 1 Processor -- start 2023-05-22T16:05:51.457 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:51.451+0000 7fd6821e5640 1 -- start start 2023-05-22T16:05:51.457 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:51.451+0000 7fd6821e5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd67c0ca880 0x7fd67c157be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:51.457 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:51.451+0000 7fd6821e5640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd67c0ce880 0x7fd67c0c7300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:51.457 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:51.451+0000 7fd6821e5640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd67c15d3d0 0x7fd67c0b5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:51.458 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:51.451+0000 7fd6821e5640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd67c0b0b70 con 0x7fd67c0ca880 2023-05-22T16:05:51.458 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:51.451+0000 7fd6821e5640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd67c158120 con 0x7fd67c15d3d0 2023-05-22T16:05:51.458 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:51.451+0000 7fd6821e5640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd67c0afef0 con 0x7fd67c0ce880 2023-05-22T16:05:51.458 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:51.455+0000 7fd67b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd67c0ca880 0x7fd67c157be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:51.458 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:51.455+0000 7fd67b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd67c0ca880 0x7fd67c157be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38296/0 (socket says 172.21.15.16:38296) 2023-05-22T16:05:51.458 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:51.455+0000 7fd67b7fe640 1 -- 172.21.15.16:0/3352507575 learned_addr learned my addr 172.21.15.16:0/3352507575 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:05:51.458 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:51.455+0000 7fd67affd640 1 --2- 172.21.15.16:0/3352507575 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd67c0ce880 0x7fd67c0c7300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:51.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:51.455+0000 7fd67bfff640 1 --2- 172.21.15.16:0/3352507575 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd67c15d3d0 0x7fd67c0b5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:51.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:51.455+0000 7fd67b7fe640 1 -- 172.21.15.16:0/3352507575 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd67c0ce880 msgr2=0x7fd67c0c7300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:51.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:51.455+0000 7fd67b7fe640 1 --2- 172.21.15.16:0/3352507575 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd67c0ce880 0x7fd67c0c7300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:51.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:51.455+0000 7fd67b7fe640 1 -- 172.21.15.16:0/3352507575 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd67c15d3d0 msgr2=0x7fd67c0b5870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:51.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:51.455+0000 7fd67b7fe640 1 --2- 172.21.15.16:0/3352507575 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd67c15d3d0 0x7fd67c0b5870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:51.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:51.455+0000 7fd67b7fe640 1 -- 172.21.15.16:0/3352507575 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6700094d0 con 0x7fd67c0ca880 2023-05-22T16:05:51.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:51.455+0000 7fd67bfff640 1 --2- 172.21.15.16:0/3352507575 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd67c15d3d0 0x7fd67c0b5870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T16:05:51.460 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:51.455+0000 7fd67b7fe640 1 --2- 172.21.15.16:0/3352507575 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd67c0ca880 0x7fd67c157be0 secure :-1 s=READY pgs=2539 cs=0 l=1 rev1=1 crypto rx=0x7fd66c00d7d0 tx=0x7fd66c00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:51.461 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:51.455+0000 7fd663fff640 1 -- 172.21.15.16:0/3352507575 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd66c00b6c0 con 0x7fd67c0ca880 2023-05-22T16:05:51.461 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:51.455+0000 7fd663fff640 1 -- 172.21.15.16:0/3352507575 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd66c00b820 con 0x7fd67c0ca880 2023-05-22T16:05:51.461 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:51.455+0000 7fd6821e5640 1 -- 172.21.15.16:0/3352507575 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd67c0cc290 con 0x7fd67c0ca880 2023-05-22T16:05:51.461 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:51.455+0000 7fd663fff640 1 -- 172.21.15.16:0/3352507575 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd66c00bbd0 con 0x7fd67c0ca880 2023-05-22T16:05:51.461 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:51.455+0000 7fd6821e5640 1 -- 172.21.15.16:0/3352507575 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd67c0cbfb0 con 0x7fd67c0ca880 2023-05-22T16:05:51.461 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:51.455+0000 7fd6821e5640 1 -- 172.21.15.16:0/3352507575 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd67c071a80 con 0x7fd67c0ca880 2023-05-22T16:05:51.462 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:51.455+0000 7fd663fff640 1 -- 172.21.15.16:0/3352507575 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 96) v1 ==== 84406+0+0 (secure 0 0 0) 0x7fd66c00bd30 con 0x7fd67c0ca880 2023-05-22T16:05:51.462 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:51.455+0000 7fd663fff640 1 --2- 172.21.15.16:0/3352507575 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fd668065580 0x7fd668067a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:51.462 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:51.455+0000 7fd663fff640 1 -- 172.21.15.16:0/3352507575 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(197..197 src has 1..197) v4 ==== 6782+0+0 (secure 0 0 0) 0x7fd66c083320 con 0x7fd67c0ca880 2023-05-22T16:05:51.462 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:51.455+0000 7fd67affd640 1 --2- 172.21.15.16:0/3352507575 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fd668065580 0x7fd668067a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:51.462 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:51.455+0000 7fd67affd640 1 --2- 172.21.15.16:0/3352507575 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fd668065580 0x7fd668067a30 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fd6700ad000 tx=0x7fd6700a6000 comp rx=0 tx=0).ready entity=mgr.10974 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:51.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:51.459+0000 7fd663fff640 1 -- 172.21.15.16:0/3352507575 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fd66c050ae0 con 0x7fd67c0ca880 2023-05-22T16:05:51.659 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:51.655+0000 7fd6821e5640 1 -- 172.21.15.16:0/3352507575 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7fd67c0b0de0 con 0x7fd67c0ca880 2023-05-22T16:05:52.238 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:52.235+0000 7fd663fff640 1 -- 172.21.15.16:0/3352507575 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mgrmap(e 97) v1 ==== 84389+0+0 (secure 0 0 0) 0x7fd66c050eb0 con 0x7fd67c0ca880 2023-05-22T16:05:52.240 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:52.235+0000 7fd663fff640 1 -- 172.21.15.16:0/3352507575 <== mon.0 v2:172.21.15.16:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v97) v1 ==== 93+0+0 (secure 0 0 0) 0x7fd66c050ae0 con 0x7fd67c0ca880 2023-05-22T16:05:52.243 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:52.239+0000 7fd6821e5640 1 -- 172.21.15.16:0/3352507575 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fd668065580 msgr2=0x7fd668067a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:52.243 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:52.239+0000 7fd6821e5640 1 --2- 172.21.15.16:0/3352507575 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fd668065580 0x7fd668067a30 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fd6700ad000 tx=0x7fd6700a6000 comp rx=0 tx=0).stop 2023-05-22T16:05:52.243 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:52.239+0000 7fd6821e5640 1 -- 172.21.15.16:0/3352507575 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd67c0ca880 msgr2=0x7fd67c157be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:52.243 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:52.239+0000 7fd6821e5640 1 --2- 172.21.15.16:0/3352507575 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd67c0ca880 0x7fd67c157be0 secure :-1 s=READY pgs=2539 cs=0 l=1 rev1=1 crypto rx=0x7fd66c00d7d0 tx=0x7fd66c00dc90 comp rx=0 tx=0).stop 2023-05-22T16:05:52.243 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:52.239+0000 7fd6821e5640 1 -- 172.21.15.16:0/3352507575 shutdown_connections 2023-05-22T16:05:52.243 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:52.239+0000 7fd6821e5640 1 --2- 172.21.15.16:0/3352507575 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd67c15d3d0 0x7fd67c0b5870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:52.244 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:52.239+0000 7fd6821e5640 1 --2- 172.21.15.16:0/3352507575 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd67c0ce880 0x7fd67c0c7300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:52.244 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:52.239+0000 7fd6821e5640 1 --2- 172.21.15.16:0/3352507575 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fd668065580 0x7fd668067a30 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:52.244 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:52.239+0000 7fd6821e5640 1 --2- 172.21.15.16:0/3352507575 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd67c0ca880 0x7fd67c157be0 unknown :-1 s=CLOSED pgs=2539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:52.244 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:52.239+0000 7fd6821e5640 1 -- 172.21.15.16:0/3352507575 >> 172.21.15.16:0/3352507575 conn(0x7fd67c099620 msgr2=0x7fd67c0cac70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:05:52.244 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:52.239+0000 7fd6821e5640 1 -- 172.21.15.16:0/3352507575 shutdown_connections 2023-05-22T16:05:52.244 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:52.239+0000 7fd6821e5640 1 -- 172.21.15.16:0/3352507575 wait complete. 2023-05-22T16:05:52.263 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-05-22T16:05:52.263 DEBUG:teuthology.orchestra.run.smithi016:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-05-22T16:05:52.310 DEBUG:teuthology.orchestra.run.smithi062:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-05-22T16:05:52.352 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-05-22T16:05:52.352 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-05-22T16:05:52.352 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2023-05-22T16:05:52.358 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-05-22T16:05:52.364 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T16:05:52.359+0000 7f69a9b59180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-05-22T16:05:52.369 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T16:05:52.367+0000 7fed27eaa180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-05-22T16:05:52.494 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:52.491+0000 7fa58cc30640 1 Processor -- start 2023-05-22T16:05:52.494 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:52.491+0000 7fa58cc30640 1 -- start start 2023-05-22T16:05:52.495 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:52.491+0000 7fa58cc30640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fa58815d960 0x7fa5880cc410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:52.495 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:52.491+0000 7fa58cc30640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5880cd240 con 0x7fa58815dd30 2023-05-22T16:05:52.495 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:52.491+0000 7fa58cc30640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5880cd520 con 0x7fa588159c70 2023-05-22T16:05:52.496 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:52.491+0000 7fa58cc30640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5880cd800 con 0x7fa58815d960 2023-05-22T16:05:52.496 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:52.491+0000 7fa58659c640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fa58815d960 0x7fa5880cc410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:52.496 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:52.491+0000 7fa58659c640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fa58815d960 0x7fa5880cc410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38320/0 (socket says 172.21.15.16:38320) 2023-05-22T16:05:52.496 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:52.491+0000 7fa58659c640 1 -- 172.21.15.16:0/3119443386 learned_addr learned my addr 172.21.15.16:0/3119443386 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:05:52.496 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:52.491+0000 7fa58659c640 1 -- 172.21.15.16:0/3119443386 >> v1:172.21.15.62:6789/0 conn(0x7fa588159c70 legacy=0x7fa58815a0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:05:52.497 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:52.491+0000 7fa58659c640 1 -- 172.21.15.16:0/3119443386 >> v1:172.21.15.16:6789/0 conn(0x7fa58815dd30 legacy=0x7fa5880cca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:52.497 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:52.491+0000 7fa58659c640 1 -- 172.21.15.16:0/3119443386 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5880c5260 con 0x7fa58815d960 2023-05-22T16:05:52.497 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:52.495+0000 7fa58659c640 1 --2- 172.21.15.16:0/3119443386 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa58815d960 0x7fa5880cc410 secure :-1 s=READY pgs=2542 cs=0 l=1 rev1=1 crypto rx=0x7fa578009870 tx=0x7fa57801c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c23dc04dbf4496a2 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:52.498 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:52.495+0000 7fa58559a640 1 -- 172.21.15.16:0/3119443386 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5780ab610 con 0x7fa58815d960 2023-05-22T16:05:52.498 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:52.495+0000 7fa58559a640 1 -- 172.21.15.16:0/3119443386 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5780ab770 con 0x7fa58815d960 2023-05-22T16:05:52.498 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:52.495+0000 7fa58559a640 1 -- 172.21.15.16:0/3119443386 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5780abb00 con 0x7fa58815d960 2023-05-22T16:05:52.499 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:52.495+0000 7fa58cc30640 1 -- 172.21.15.16:0/3119443386 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa58815d960 msgr2=0x7fa5880cc410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:52.499 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:52.495+0000 7fa58cc30640 1 --2- 172.21.15.16:0/3119443386 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa58815d960 0x7fa5880cc410 secure :-1 s=READY pgs=2542 cs=0 l=1 rev1=1 crypto rx=0x7fa578009870 tx=0x7fa57801c500 comp rx=0 tx=0).stop 2023-05-22T16:05:52.499 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:52.495+0000 7fa58cc30640 1 -- 172.21.15.16:0/3119443386 shutdown_connections 2023-05-22T16:05:52.499 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:52.495+0000 7fa58cc30640 1 --2- 172.21.15.16:0/3119443386 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa58815d960 0x7fa5880cc410 unknown :-1 s=CLOSED pgs=2542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:52.500 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:52.495+0000 7fa58cc30640 1 -- 172.21.15.16:0/3119443386 >> 172.21.15.16:0/3119443386 conn(0x7fa58806db60 msgr2=0x7fa58806df60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:05:52.500 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:52.495+0000 7fa58cc30640 1 -- 172.21.15.16:0/3119443386 shutdown_connections 2023-05-22T16:05:52.500 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:52.495+0000 7fa58cc30640 1 -- 172.21.15.16:0/3119443386 wait complete. 2023-05-22T16:05:52.500 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:52.495+0000 7fa58cc30640 1 Processor -- start 2023-05-22T16:05:52.500 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:52.495+0000 7fa58cc30640 1 -- start start 2023-05-22T16:05:52.502 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:52.499+0000 7fa58cc30640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa588159c70 0x7fa588152e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:52.502 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:52.499+0000 7fa58cc30640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa58815d960 0x7fa58814ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:52.502 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:52.499+0000 7fa58cc30640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa58815dd30 0x7fa5881547d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:52.502 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:52.499+0000 7fa58cc30640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa588157df0 con 0x7fa58815dd30 2023-05-22T16:05:52.502 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:52.499+0000 7fa58cc30640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa588153380 con 0x7fa58815d960 2023-05-22T16:05:52.502 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:52.499+0000 7fa58cc30640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa58814e330 con 0x7fa588159c70 2023-05-22T16:05:52.503 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:52.499+0000 7fa586d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa58815dd30 0x7fa5881547d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:52.503 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:52.499+0000 7fa586d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa58815dd30 0x7fa5881547d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38326/0 (socket says 172.21.15.16:38326) 2023-05-22T16:05:52.503 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:52.499+0000 7fa586d9d640 1 -- 172.21.15.16:0/3083865684 learned_addr learned my addr 172.21.15.16:0/3083865684 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:05:52.503 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:52.499+0000 7fa58659c640 1 --2- 172.21.15.16:0/3083865684 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa588159c70 0x7fa588152e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:52.503 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:52.499+0000 7fa585d9b640 1 --2- 172.21.15.16:0/3083865684 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa58815d960 0x7fa58814ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:52.503 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:52.499+0000 7fa586d9d640 1 -- 172.21.15.16:0/3083865684 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa588159c70 msgr2=0x7fa588152e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:52.503 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:52.499+0000 7fa586d9d640 1 --2- 172.21.15.16:0/3083865684 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa588159c70 0x7fa588152e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:52.504 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:52.499+0000 7fa586d9d640 1 -- 172.21.15.16:0/3083865684 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa58815d960 msgr2=0x7fa58814ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:52.504 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:52.499+0000 7fa586d9d640 1 --2- 172.21.15.16:0/3083865684 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa58815d960 0x7fa58814ddf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:52.504 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:52.499+0000 7fa586d9d640 1 -- 172.21.15.16:0/3083865684 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5780094d0 con 0x7fa58815dd30 2023-05-22T16:05:52.504 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:52.499+0000 7fa586d9d640 1 --2- 172.21.15.16:0/3083865684 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa58815dd30 0x7fa5881547d0 secure :-1 s=READY pgs=2543 cs=0 l=1 rev1=1 crypto rx=0x7fa58000c7e0 tx=0x7fa58000cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:52.504 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:52.499+0000 7fa576ffd640 1 -- 172.21.15.16:0/3083865684 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa580004b70 con 0x7fa58815dd30 2023-05-22T16:05:52.504 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:52.499+0000 7fa576ffd640 1 -- 172.21.15.16:0/3083865684 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa580004cd0 con 0x7fa58815dd30 2023-05-22T16:05:52.504 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:52.499+0000 7fa58cc30640 1 -- 172.21.15.16:0/3083865684 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa588167c40 con 0x7fa58815dd30 2023-05-22T16:05:52.505 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:52.499+0000 7fa576ffd640 1 -- 172.21.15.16:0/3083865684 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa58000aa90 con 0x7fa58815dd30 2023-05-22T16:05:52.505 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:52.499+0000 7fa58cc30640 1 -- 172.21.15.16:0/3083865684 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa588167ab0 con 0x7fa58815dd30 2023-05-22T16:05:52.505 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:52.499+0000 7fa58cc30640 1 -- 172.21.15.16:0/3083865684 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa588167dd0 con 0x7fa58815dd30 2023-05-22T16:05:52.506 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:52.499+0000 7fa576ffd640 1 -- 172.21.15.16:0/3083865684 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 97) v1 ==== 84389+0+0 (secure 0 0 0) 0x7fa58000abf0 con 0x7fa58815dd30 2023-05-22T16:05:52.506 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:52.499+0000 7fa576ffd640 1 --2- 172.21.15.16:0/3083865684 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fa568065530 0x7fa5680679e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:52.506 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:52.499+0000 7fa58659c640 1 -- 172.21.15.16:0/3083865684 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fa568065530 msgr2=0x7fa5680679e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/65890 2023-05-22T16:05:52.506 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:52.499+0000 7fa576ffd640 1 -- 172.21.15.16:0/3083865684 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(197..197 src has 1..197) v4 ==== 6782+0+0 (secure 0 0 0) 0x7fa580083250 con 0x7fa58815dd30 2023-05-22T16:05:52.506 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:52.499+0000 7fa58659c640 1 --2- 172.21.15.16:0/3083865684 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fa568065530 0x7fa5680679e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-05-22T16:05:52.508 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:52.503+0000 7fa576ffd640 1 -- 172.21.15.16:0/3083865684 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fa580050990 con 0x7fa58815dd30 2023-05-22T16:05:52.661 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:52.659+0000 7fa58cc30640 1 -- 172.21.15.16:0/3083865684 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa5880c6fb0 con 0x7fa58815dd30 2023-05-22T16:05:52.662 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:52.659+0000 7fa576ffd640 1 -- 172.21.15.16:0/3083865684 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v197) v1 ==== 74+0+18629 (secure 0 0 0) 0x7fa580053fb0 con 0x7fa58815dd30 2023-05-22T16:05:52.663 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T16:05:52.663 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":197,"fsid":"b2719834-7847-4d98-90a6-f65d57f85c49","created":"2023-05-22T15:26:25.259314+0000","modified":"2023-05-22T16:05:35.647639+0000","last_up_change":"2023-05-22T15:26:36.448524+0000","last_in_change":"2023-05-22T15:26:27.675668+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":25,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-05-22T15:26:38.019781+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-05-22T15:26:43.570514+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_metadata","create_time":"2023-05-22T16:05:20.485167+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"195","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":25,"pool_name":"cephfs_data","create_time":"2023-05-22T16:05:20.900001+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"196","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9e7ff854-b25f-4cbb-a5c2-1799d9c53d2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":37021},{"type":"v1","addr":"172.21.15.16:6825","nonce":37021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":37021},{"type":"v1","addr":"172.21.15.16:6827","nonce":37021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":37021},{"type":"v1","addr":"172.21.15.16:6831","nonce":37021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":37021},{"type":"v1","addr":"172.21.15.16:6829","nonce":37021}]},"public_addr":"172.21.15.16:6825/37021","cluster_addr":"172.21.15.16:6827/37021","heartbeat_back_addr":"172.21.15.16:6831/37021","heartbeat_front_addr":"172.21.15.16:6829/37021","state":["exists","up"]},{"osd":1,"uuid":"4702fbc9-a927-4565-aca5-41dc69cc8264","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":37008},{"type":"v1","addr":"172.21.15.16:6801","nonce":37008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":37008},{"type":"v1","addr":"172.21.15.16:6803","nonce":37008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":37008},{"type":"v1","addr":"172.21.15.16:6807","nonce":37008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":37008},{"type":"v1","addr":"172.21.15.16:6805","nonce":37008}]},"public_addr":"172.21.15.16:6801/37008","cluster_addr":"172.21.15.16:6803/37008","heartbeat_back_addr":"172.21.15.16:6807/37008","heartbeat_front_addr":"172.21.15.16:6805/37008","state":["exists","up"]},{"osd":2,"uuid":"1e7bba46-4eba-47ef-bb81-5e2d371638ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":37009},{"type":"v1","addr":"172.21.15.16:6809","nonce":37009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":37009},{"type":"v1","addr":"172.21.15.16:6811","nonce":37009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":37009},{"type":"v1","addr":"172.21.15.16:6815","nonce":37009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":37009},{"type":"v1","addr":"172.21.15.16:6813","nonce":37009}]},"public_addr":"172.21.15.16:6809/37009","cluster_addr":"172.21.15.16:6811/37009","heartbeat_back_addr":"172.21.15.16:6815/37009","heartbeat_front_addr":"172.21.15.16:6813/37009","state":["exists","up"]},{"osd":3,"uuid":"383cacc0-5a88-4904-990e-498ca59ef48a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":37020},{"type":"v1","addr":"172.21.15.16:6817","nonce":37020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":37020},{"type":"v1","addr":"172.21.15.16:6819","nonce":37020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":37020},{"type":"v1","addr":"172.21.15.16:6823","nonce":37020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":37020},{"type":"v1","addr":"172.21.15.16:6821","nonce":37020}]},"public_addr":"172.21.15.16:6817/37020","cluster_addr":"172.21.15.16:6819/37020","heartbeat_back_addr":"172.21.15.16:6823/37020","heartbeat_front_addr":"172.21.15.16:6821/37020","state":["exists","up"]},{"osd":4,"uuid":"970500ef-95b7-42c1-b080-eed4fae7013d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":37353},{"type":"v1","addr":"172.21.15.62:6817","nonce":37353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6819","nonce":37353},{"type":"v1","addr":"172.21.15.62:6822","nonce":37353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37353},{"type":"v1","addr":"172.21.15.62:6831","nonce":37353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37353},{"type":"v1","addr":"172.21.15.62:6826","nonce":37353}]},"public_addr":"172.21.15.62:6817/37353","cluster_addr":"172.21.15.62:6822/37353","heartbeat_back_addr":"172.21.15.62:6831/37353","heartbeat_front_addr":"172.21.15.62:6826/37353","state":["exists","up"]},{"osd":5,"uuid":"022d67db-d4d8-445b-b1b9-8b446a00f5c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":37352},{"type":"v1","addr":"172.21.15.62:6801","nonce":37352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":37352},{"type":"v1","addr":"172.21.15.62:6803","nonce":37352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":37352},{"type":"v1","addr":"172.21.15.62:6807","nonce":37352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":37352},{"type":"v1","addr":"172.21.15.62:6805","nonce":37352}]},"public_addr":"172.21.15.62:6801/37352","cluster_addr":"172.21.15.62:6803/37352","heartbeat_back_addr":"172.21.15.62:6807/37352","heartbeat_front_addr":"172.21.15.62:6805/37352","state":["exists","up"]},{"osd":6,"uuid":"42bd372b-6688-4a1b-8407-e16e8c266e33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":37354},{"type":"v1","addr":"172.21.15.62:6820","nonce":37354}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":37354},{"type":"v1","addr":"172.21.15.62:6823","nonce":37354}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6829","nonce":37354},{"type":"v1","addr":"172.21.15.62:6830","nonce":37354}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6825","nonce":37354},{"type":"v1","addr":"172.21.15.62:6827","nonce":37354}]},"public_addr":"172.21.15.62:6820/37354","cluster_addr":"172.21.15.62:6823/37354","heartbeat_back_addr":"172.21.15.62:6830/37354","heartbeat_front_addr":"172.21.15.62:6827/37354","state":["exists","up"]},{"osd":7,"uuid":"ac70b2a3-f41b-4bf2-b227-7f2cbb55426d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37355},{"type":"v1","addr":"172.21.15.62:6809","nonce":37355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37355},{"type":"v1","addr":"172.21.15.62:6811","nonce":37355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":37355},{"type":"v1","addr":"172.21.15.62:6815","nonce":37355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":37355},{"type":"v1","addr":"172.21.15.62:6813","nonce":37355}]},"public_addr":"172.21.15.62:6809/37355","cluster_addr":"172.21.15.62:6811/37355","heartbeat_back_addr":"172.21.15.62:6815/37355","heartbeat_front_addr":"172.21.15.62:6813/37355","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.368896+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.107273+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.276888+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.287502+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.124791+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.251937+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.255679+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.232797+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:0/1961044213":"2023-05-23T16:05:35.647562+0000","172.21.15.16:0/238292614":"2023-05-23T16:05:35.647562+0000","172.21.15.16:6833/65890":"2023-05-23T16:05:35.647562+0000","172.21.15.16:0/1148602276":"2023-05-23T16:05:35.647562+0000","172.21.15.16:6832/65890":"2023-05-23T16:05:35.647562+0000","172.21.15.16:0/456460873":"2023-05-23T16:05:35.647562+0000","172.21.15.16:0/2766506027":"2023-05-23T16:05:35.647562+0000","172.21.15.16:0/2214029832":"2023-05-23T16:05:35.647562+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":17,"snaps":[{"begin":2,"length":2}]},{"pool":21,"snaps":[{"begin":25,"length":9}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-05-22T16:05:52.665 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:52.663+0000 7fa58cc30640 1 -- 172.21.15.16:0/3083865684 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fa568065530 msgr2=0x7fa5680679e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-05-22T16:05:52.665 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:52.663+0000 7fa58cc30640 1 --2- 172.21.15.16:0/3083865684 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fa568065530 0x7fa5680679e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:52.665 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:52.663+0000 7fa58cc30640 1 -- 172.21.15.16:0/3083865684 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa58815dd30 msgr2=0x7fa5881547d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:52.665 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:52.663+0000 7fa58cc30640 1 --2- 172.21.15.16:0/3083865684 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa58815dd30 0x7fa5881547d0 secure :-1 s=READY pgs=2543 cs=0 l=1 rev1=1 crypto rx=0x7fa58000c7e0 tx=0x7fa58000cca0 comp rx=0 tx=0).stop 2023-05-22T16:05:52.665 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:52.663+0000 7fa58cc30640 1 -- 172.21.15.16:0/3083865684 shutdown_connections 2023-05-22T16:05:52.666 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:52.663+0000 7fa58cc30640 1 --2- 172.21.15.16:0/3083865684 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fa568065530 0x7fa5680679e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:52.666 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:52.663+0000 7fa58cc30640 1 --2- 172.21.15.16:0/3083865684 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa58815dd30 0x7fa5881547d0 unknown :-1 s=CLOSED pgs=2543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:52.666 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:52.663+0000 7fa58cc30640 1 --2- 172.21.15.16:0/3083865684 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa58815d960 0x7fa58814ddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:52.666 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:52.663+0000 7fa58cc30640 1 --2- 172.21.15.16:0/3083865684 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa588159c70 0x7fa588152e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:52.666 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:52.663+0000 7fa58cc30640 1 -- 172.21.15.16:0/3083865684 >> 172.21.15.16:0/3083865684 conn(0x7fa58806db60 msgr2=0x7fa5880b1f30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:05:52.666 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:52.663+0000 7fa58cc30640 1 -- 172.21.15.16:0/3083865684 shutdown_connections 2023-05-22T16:05:52.666 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:52.663+0000 7fa58cc30640 1 -- 172.21.15.16:0/3083865684 wait complete. 2023-05-22T16:05:52.682 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-05-22T15:26:38.019781+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-05-22T15:26:43.570514+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 24, 'pool_name': 'cephfs_metadata', 'create_time': '2023-05-22T16:05:20.485167+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '195', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 25, 'pool_name': 'cephfs_data', 'create_time': '2023-05-22T16:05:20.900001+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '196', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-05-22T16:05:52.682 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-05-22T16:05:52.750 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T16:05:52.747+0000 7f69a9b59180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-05-22T16:05:52.772 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T16:05:52.767+0000 7fed27eaa180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-05-22T16:05:52.800 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:52.795+0000 7f718310f640 1 Processor -- start 2023-05-22T16:05:52.801 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:52.799+0000 7f718310f640 1 -- start start 2023-05-22T16:05:52.801 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:52.799+0000 7f718310f640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f717c158b60 0x7f717c0cd1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:52.802 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:52.799+0000 7f718310f640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f717c15d1a0 0x7f717c0cd7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:52.802 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:52.799+0000 7f718310f640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f717c15c750 con 0x7f717c0cdcf0 2023-05-22T16:05:52.802 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:52.799+0000 7f718310f640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f717c0ce170 con 0x7f717c15d1a0 2023-05-22T16:05:52.802 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:52.799+0000 7f718310f640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f717c15cd10 con 0x7f717c158b60 2023-05-22T16:05:52.802 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:52.799+0000 7f71816ac640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f717c0cdcf0 0x7f717c06dbc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:45734/0 (socket says 172.21.15.16:45734) 2023-05-22T16:05:52.802 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:52.799+0000 7f71816ac640 1 -- 172.21.15.16:0/492148099 learned_addr learned my addr 172.21.15.16:0/492148099 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T16:05:52.804 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:52.799+0000 7f717bfff640 1 --2- 172.21.15.16:0/492148099 >> v2:172.21.15.62:3300/0 conn(0x7f717c15d1a0 0x7f717c0cd7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:52.804 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:52.799+0000 7f7180eab640 1 --2- 172.21.15.16:0/492148099 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f717c158b60 0x7f717c0cd1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:52.804 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:52.799+0000 7f717b7fe640 1 -- 172.21.15.16:0/492148099 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3660957081 0 0) 0x7f717c15c750 con 0x7f717c0cdcf0 2023-05-22T16:05:52.804 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:52.799+0000 7f717b7fe640 1 -- 172.21.15.16:0/492148099 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7168003530 con 0x7f717c0cdcf0 2023-05-22T16:05:52.804 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:52.799+0000 7f717b7fe640 1 -- 172.21.15.16:0/492148099 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 372175133 0 0) 0x7f7168003530 con 0x7f717c0cdcf0 2023-05-22T16:05:52.805 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:52.799+0000 7f717b7fe640 1 -- 172.21.15.16:0/492148099 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f717c158b60 msgr2=0x7f717c0cd1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:05:52.805 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:52.799+0000 7f717b7fe640 1 --2- 172.21.15.16:0/492148099 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f717c158b60 0x7f717c0cd1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:52.805 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:52.799+0000 7f717b7fe640 1 -- 172.21.15.16:0/492148099 >> v2:172.21.15.62:3300/0 conn(0x7f717c15d1a0 msgr2=0x7f717c0cd7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:05:52.805 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:52.799+0000 7f717b7fe640 1 --2- 172.21.15.16:0/492148099 >> v2:172.21.15.62:3300/0 conn(0x7f717c15d1a0 0x7f717c0cd7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:52.805 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:52.799+0000 7f717b7fe640 1 -- 172.21.15.16:0/492148099 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f717c15d5d0 con 0x7f717c0cdcf0 2023-05-22T16:05:52.805 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:52.799+0000 7f717b7fe640 1 -- 172.21.15.16:0/492148099 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7f7174001680 con 0x7f717c0cdcf0 2023-05-22T16:05:52.805 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:52.799+0000 7f717b7fe640 1 -- 172.21.15.16:0/492148099 >> v1:172.21.15.16:6789/0 conn(0x7f717c0cdcf0 legacy=0x7f717c06dbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:52.806 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:52.799+0000 7f717b7fe640 1 --2- 172.21.15.16:0/492148099 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7168003980 0x7f7168003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:52.806 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:52.799+0000 7f717b7fe640 1 --2- 172.21.15.16:0/492148099 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f71680043b0 0x7f7168004870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:52.806 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:52.799+0000 7f717b7fe640 1 --2- 172.21.15.16:0/492148099 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f717c158b60 0x7f7168004fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:52.806 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:52.799+0000 7f717b7fe640 1 -- 172.21.15.16:0/492148099 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f717c0ce170 con 0x7f71680043b0 2023-05-22T16:05:52.806 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:52.799+0000 7f717b7fe640 1 -- 172.21.15.16:0/492148099 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f717c15cd10 con 0x7f7168003980 2023-05-22T16:05:52.806 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:52.799+0000 7f717b7fe640 1 -- 172.21.15.16:0/492148099 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f716800b990 con 0x7f717c158b60 2023-05-22T16:05:52.806 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:52.799+0000 7f717bfff640 1 --2- 172.21.15.16:0/492148099 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f71680043b0 0x7f7168004870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:52.806 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:52.799+0000 7f71816ac640 1 --2- 172.21.15.16:0/492148099 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f717c158b60 0x7f7168004fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:52.807 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:52.799+0000 7f7180eab640 1 --2- 172.21.15.16:0/492148099 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7168003980 0x7f7168003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:52.807 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:52.799+0000 7f717bfff640 1 -- 172.21.15.16:0/492148099 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f717c158b60 msgr2=0x7f7168004fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:05:52.807 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:52.799+0000 7f717bfff640 1 --2- 172.21.15.16:0/492148099 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f717c158b60 0x7f7168004fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:52.807 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:52.799+0000 7f717bfff640 1 -- 172.21.15.16:0/492148099 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7168003980 msgr2=0x7f7168003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:05:52.807 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:52.799+0000 7f717bfff640 1 --2- 172.21.15.16:0/492148099 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7168003980 0x7f7168003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:52.807 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:52.799+0000 7f717bfff640 1 -- 172.21.15.16:0/492148099 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7168003530 con 0x7f71680043b0 2023-05-22T16:05:52.809 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:52.803+0000 7f717bfff640 1 --2- 172.21.15.16:0/492148099 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f71680043b0 0x7f7168004870 secure :-1 s=READY pgs=2545 cs=0 l=1 rev1=1 crypto rx=0x7f7164009b10 tx=0x7f716401ca60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff08a6dea5e83133 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:52.809 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:52.803+0000 7f717b7fe640 1 -- 172.21.15.16:0/492148099 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71640a5540 con 0x7f71680043b0 2023-05-22T16:05:52.809 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:52.803+0000 7f717b7fe640 1 -- 172.21.15.16:0/492148099 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71640a56a0 con 0x7f71680043b0 2023-05-22T16:05:52.810 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:52.803+0000 7f717b7fe640 1 -- 172.21.15.16:0/492148099 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71640a5a50 con 0x7f71680043b0 2023-05-22T16:05:52.810 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:52.803+0000 7f718310f640 1 -- 172.21.15.16:0/492148099 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f71680043b0 msgr2=0x7f7168004870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:52.810 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:52.803+0000 7f718310f640 1 --2- 172.21.15.16:0/492148099 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f71680043b0 0x7f7168004870 secure :-1 s=READY pgs=2545 cs=0 l=1 rev1=1 crypto rx=0x7f7164009b10 tx=0x7f716401ca60 comp rx=0 tx=0).stop 2023-05-22T16:05:52.810 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:52.803+0000 7f71816ac640 1 -- 172.21.15.16:0/492148099 reap_dead start 2023-05-22T16:05:52.810 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:52.803+0000 7f718310f640 1 -- 172.21.15.16:0/492148099 shutdown_connections 2023-05-22T16:05:52.810 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:52.803+0000 7f718310f640 1 -- 172.21.15.16:0/492148099 >> 172.21.15.16:0/492148099 conn(0x7f717c06d5b0 msgr2=0x7f717c0b0de0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:05:52.810 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:52.803+0000 7f718310f640 1 -- 172.21.15.16:0/492148099 shutdown_connections 2023-05-22T16:05:52.811 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:52.803+0000 7f718310f640 1 -- 172.21.15.16:0/492148099 wait complete. 2023-05-22T16:05:52.811 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:52.803+0000 7f718310f640 1 Processor -- start 2023-05-22T16:05:52.811 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:52.803+0000 7f718310f640 1 -- start start 2023-05-22T16:05:52.811 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:52.803+0000 7f718310f640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f71680043b0 0x7f717c163160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:52.811 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:52.803+0000 7f718310f640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f717c157390 0x7f717c0c7200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:52.811 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:52.803+0000 7f718310f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f717c156c70 0x7f717c0ca390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:52.811 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:52.803+0000 7f718310f640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f717c0c7740 con 0x7f717c156c70 2023-05-22T16:05:52.811 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:52.803+0000 7f718310f640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f717c157c30 con 0x7f717c157390 2023-05-22T16:05:52.812 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:52.803+0000 7f718310f640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f717c0c6d70 con 0x7f71680043b0 2023-05-22T16:05:52.812 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:52.803+0000 7f71816ac640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f717c156c70 0x7f717c0ca390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:52.812 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:52.807+0000 7f71816ac640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f717c156c70 0x7f717c0ca390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38350/0 (socket says 172.21.15.16:38350) 2023-05-22T16:05:52.812 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:52.807+0000 7f71816ac640 1 -- 172.21.15.16:0/337075029 learned_addr learned my addr 172.21.15.16:0/337075029 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:05:52.812 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:52.807+0000 7f717bfff640 1 --2- 172.21.15.16:0/337075029 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f717c157390 0x7f717c0c7200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:52.812 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:52.807+0000 7f7180eab640 1 --2- 172.21.15.16:0/337075029 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f71680043b0 0x7f717c163160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:52.812 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:52.807+0000 7f71816ac640 1 -- 172.21.15.16:0/337075029 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f71680043b0 msgr2=0x7f717c163160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:52.813 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:52.807+0000 7f71816ac640 1 --2- 172.21.15.16:0/337075029 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f71680043b0 0x7f717c163160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:52.813 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:52.807+0000 7f71816ac640 1 -- 172.21.15.16:0/337075029 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f717c157390 msgr2=0x7f717c0c7200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:52.813 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:52.807+0000 7f71816ac640 1 --2- 172.21.15.16:0/337075029 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f717c157390 0x7f717c0c7200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:52.813 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:52.807+0000 7f71816ac640 1 -- 172.21.15.16:0/337075029 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f717c0b14f0 con 0x7f717c156c70 2023-05-22T16:05:52.813 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:52.807+0000 7f717bfff640 1 --2- 172.21.15.16:0/337075029 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f717c157390 0x7f717c0c7200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T16:05:52.813 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:52.807+0000 7f71816ac640 1 --2- 172.21.15.16:0/337075029 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f717c156c70 0x7f717c0ca390 secure :-1 s=READY pgs=2546 cs=0 l=1 rev1=1 crypto rx=0x7f717c0b5200 tx=0x7f717400e7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:52.813 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:52.807+0000 7f71797fa640 1 -- 172.21.15.16:0/337075029 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f717400eda0 con 0x7f717c156c70 2023-05-22T16:05:52.813 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:52.807+0000 7f71797fa640 1 -- 172.21.15.16:0/337075029 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7174014430 con 0x7f717c156c70 2023-05-22T16:05:52.814 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:52.807+0000 7f71797fa640 1 -- 172.21.15.16:0/337075029 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71740147e0 con 0x7f717c156c70 2023-05-22T16:05:52.814 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:52.807+0000 7f718310f640 1 -- 172.21.15.16:0/337075029 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f717c15c9a0 con 0x7f717c156c70 2023-05-22T16:05:52.815 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:52.807+0000 7f718310f640 1 -- 172.21.15.16:0/337075029 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f717c158f30 con 0x7f717c156c70 2023-05-22T16:05:52.815 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:52.807+0000 7f718310f640 1 -- 172.21.15.16:0/337075029 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f717c0c6eb0 con 0x7f717c156c70 2023-05-22T16:05:52.815 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:52.807+0000 7f71797fa640 1 -- 172.21.15.16:0/337075029 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 97) v1 ==== 84389+0+0 (secure 0 0 0) 0x7f7174014940 con 0x7f717c156c70 2023-05-22T16:05:52.815 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:52.807+0000 7f71797fa640 1 --2- 172.21.15.16:0/337075029 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f715c065530 0x7f715c0679e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:52.815 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:52.807+0000 7f7180eab640 1 -- 172.21.15.16:0/337075029 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f715c065530 msgr2=0x7f715c0679e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/65890 2023-05-22T16:05:52.815 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:52.807+0000 7f71797fa640 1 -- 172.21.15.16:0/337075029 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(197..197 src has 1..197) v4 ==== 6782+0+0 (secure 0 0 0) 0x7f7174081e30 con 0x7f717c156c70 2023-05-22T16:05:52.816 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:52.807+0000 7f7180eab640 1 --2- 172.21.15.16:0/337075029 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f715c065530 0x7f715c0679e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-05-22T16:05:52.816 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:52.811+0000 7f71797fa640 1 -- 172.21.15.16:0/337075029 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f717404b620 con 0x7f717c156c70 2023-05-22T16:05:52.888 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T16:05:52.883+0000 7f69a9b59180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-05-22T16:05:52.913 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T16:05:52.911+0000 7fed27eaa180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-05-22T16:05:52.972 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:52.967+0000 7f718310f640 1 -- 172.21.15.16:0/337075029 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f717c166a80 con 0x7f717c156c70 2023-05-22T16:05:52.973 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:52.971+0000 7f71797fa640 1 -- 172.21.15.16:0/337075029 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v197) v1 ==== 93+0+10 (secure 0 0 0) 0x7f7174058030 con 0x7f717c156c70 2023-05-22T16:05:52.973 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2023-05-22T16:05:52.976 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:52.971+0000 7f718310f640 1 -- 172.21.15.16:0/337075029 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f715c065530 msgr2=0x7f715c0679e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-05-22T16:05:52.976 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:52.971+0000 7f718310f640 1 --2- 172.21.15.16:0/337075029 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f715c065530 0x7f715c0679e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:52.976 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:52.971+0000 7f718310f640 1 -- 172.21.15.16:0/337075029 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f717c156c70 msgr2=0x7f717c0ca390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:52.976 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:52.971+0000 7f718310f640 1 --2- 172.21.15.16:0/337075029 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f717c156c70 0x7f717c0ca390 secure :-1 s=READY pgs=2546 cs=0 l=1 rev1=1 crypto rx=0x7f717c0b5200 tx=0x7f717400e7c0 comp rx=0 tx=0).stop 2023-05-22T16:05:52.976 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:52.971+0000 7f718310f640 1 -- 172.21.15.16:0/337075029 shutdown_connections 2023-05-22T16:05:52.976 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:52.971+0000 7f718310f640 1 --2- 172.21.15.16:0/337075029 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f715c065530 0x7f715c0679e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:52.976 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:52.971+0000 7f718310f640 1 --2- 172.21.15.16:0/337075029 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f717c156c70 0x7f717c0ca390 unknown :-1 s=CLOSED pgs=2546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:52.977 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:52.971+0000 7f718310f640 1 --2- 172.21.15.16:0/337075029 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f717c157390 0x7f717c0c7200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:52.977 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:52.971+0000 7f718310f640 1 --2- 172.21.15.16:0/337075029 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f71680043b0 0x7f717c163160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:52.977 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:52.971+0000 7f718310f640 1 -- 172.21.15.16:0/337075029 >> 172.21.15.16:0/337075029 conn(0x7f717c06d5b0 msgr2=0x7f717c06e6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:05:52.977 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:52.971+0000 7f718310f640 1 -- 172.21.15.16:0/337075029 shutdown_connections 2023-05-22T16:05:52.977 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:52.971+0000 7f718310f640 1 -- 172.21.15.16:0/337075029 wait complete. 2023-05-22T16:05:52.992 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-05-22T16:05:53.108 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.103+0000 7fc8d0eba640 1 Processor -- start 2023-05-22T16:05:53.108 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.103+0000 7fc8d0eba640 1 -- start start 2023-05-22T16:05:53.109 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.107+0000 7fc8d0eba640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc8cc0cbdb0 0x7fc8cc0cc230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:53.109 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.107+0000 7fc8d0eba640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8cc06b5f0 con 0x7fc8cc15f480 2023-05-22T16:05:53.109 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.107+0000 7fc8d0eba640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8cc0cad10 con 0x7fc8cc0ca880 2023-05-22T16:05:53.109 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.107+0000 7fc8d0eba640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8cc06bf70 con 0x7fc8cc0cbdb0 2023-05-22T16:05:53.110 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.107+0000 7fc8ca59c640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fc8cc15f480 0x7fc8cc0b3110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:45744/0 (socket says 172.21.15.16:45744) 2023-05-22T16:05:53.110 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.107+0000 7fc8ca59c640 1 -- 172.21.15.16:0/1047969889 learned_addr learned my addr 172.21.15.16:0/1047969889 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T16:05:53.110 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.107+0000 7fc8cad9d640 1 --2- 172.21.15.16:0/1047969889 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc8cc0cbdb0 0x7fc8cc0cc230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:53.110 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.107+0000 7fc8c959a640 1 -- 172.21.15.16:0/1047969889 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4023860021 0 0) 0x7fc8cc06b5f0 con 0x7fc8cc15f480 2023-05-22T16:05:53.110 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.107+0000 7fc8c959a640 1 -- 172.21.15.16:0/1047969889 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc8b4003530 con 0x7fc8cc15f480 2023-05-22T16:05:53.111 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.107+0000 7fc8c959a640 1 -- 172.21.15.16:0/1047969889 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2831785142 0 0) 0x7fc8b4003530 con 0x7fc8cc15f480 2023-05-22T16:05:53.111 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.107+0000 7fc8c959a640 1 -- 172.21.15.16:0/1047969889 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc8cc0cbdb0 msgr2=0x7fc8cc0cc230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:05:53.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.107+0000 7fc8c959a640 1 --2- 172.21.15.16:0/1047969889 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc8cc0cbdb0 0x7fc8cc0cc230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:53.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.107+0000 7fc8c959a640 1 -- 172.21.15.16:0/1047969889 >> v1:172.21.15.62:6789/0 conn(0x7fc8cc0ca880 legacy=0x7fc8cc0cee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:53.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.107+0000 7fc8c959a640 1 -- 172.21.15.16:0/1047969889 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8cc0caff0 con 0x7fc8cc15f480 2023-05-22T16:05:53.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.107+0000 7fc8c959a640 1 -- 172.21.15.16:0/1047969889 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7fc8bc001800 con 0x7fc8cc15f480 2023-05-22T16:05:53.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.107+0000 7fc8c959a640 1 -- 172.21.15.16:0/1047969889 >> v1:172.21.15.16:6789/0 conn(0x7fc8cc15f480 legacy=0x7fc8cc0b3110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:53.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.107+0000 7fc8c959a640 1 --2- 172.21.15.16:0/1047969889 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc8b40039b0 0x7fc8b4003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:53.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.107+0000 7fc8c959a640 1 --2- 172.21.15.16:0/1047969889 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc8cc0cbdb0 0x7fc8b40047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:53.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.107+0000 7fc8c959a640 1 --2- 172.21.15.16:0/1047969889 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc8b4004d10 0x7fc8b4005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:53.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.107+0000 7fc8c959a640 1 -- 172.21.15.16:0/1047969889 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8cc06bf70 con 0x7fc8b4004d10 2023-05-22T16:05:53.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.107+0000 7fc8c959a640 1 -- 172.21.15.16:0/1047969889 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8b400ba40 con 0x7fc8b40039b0 2023-05-22T16:05:53.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.107+0000 7fc8c959a640 1 -- 172.21.15.16:0/1047969889 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8b400bbd0 con 0x7fc8cc0cbdb0 2023-05-22T16:05:53.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.107+0000 7fc8cad9d640 1 --2- 172.21.15.16:0/1047969889 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc8b4004d10 0x7fc8b4005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:53.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.107+0000 7fc8ca59c640 1 --2- 172.21.15.16:0/1047969889 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc8b40039b0 0x7fc8b4003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:53.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.107+0000 7fc8c9d9b640 1 --2- 172.21.15.16:0/1047969889 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc8cc0cbdb0 0x7fc8b40047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:53.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.107+0000 7fc8cad9d640 1 -- 172.21.15.16:0/1047969889 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc8cc0cbdb0 msgr2=0x7fc8b40047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:05:53.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.107+0000 7fc8cad9d640 1 --2- 172.21.15.16:0/1047969889 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc8cc0cbdb0 0x7fc8b40047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:53.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.107+0000 7fc8cad9d640 1 -- 172.21.15.16:0/1047969889 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc8b40039b0 msgr2=0x7fc8b4003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:05:53.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.107+0000 7fc8cad9d640 1 --2- 172.21.15.16:0/1047969889 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc8b40039b0 0x7fc8b4003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:53.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.107+0000 7fc8cad9d640 1 -- 172.21.15.16:0/1047969889 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8b4003530 con 0x7fc8b4004d10 2023-05-22T16:05:53.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.107+0000 7fc8c9d9b640 1 --2- 172.21.15.16:0/1047969889 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc8cc0cbdb0 0x7fc8b40047d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T16:05:53.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.107+0000 7fc8cad9d640 1 --2- 172.21.15.16:0/1047969889 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc8b4004d10 0x7fc8b4005180 secure :-1 s=READY pgs=2548 cs=0 l=1 rev1=1 crypto rx=0x7fc8c4009fc0 tx=0x7fc8c401cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e0672389a51a6ef server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:53.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.107+0000 7fc8c959a640 1 -- 172.21.15.16:0/1047969889 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8c40a5c70 con 0x7fc8b4004d10 2023-05-22T16:05:53.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.107+0000 7fc8c959a640 1 -- 172.21.15.16:0/1047969889 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8c400f040 con 0x7fc8b4004d10 2023-05-22T16:05:53.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.107+0000 7fc8c959a640 1 -- 172.21.15.16:0/1047969889 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8c40ad440 con 0x7fc8b4004d10 2023-05-22T16:05:53.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.111+0000 7fc8d0eba640 1 -- 172.21.15.16:0/1047969889 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc8b4004d10 msgr2=0x7fc8b4005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:53.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.111+0000 7fc8d0eba640 1 --2- 172.21.15.16:0/1047969889 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc8b4004d10 0x7fc8b4005180 secure :-1 s=READY pgs=2548 cs=0 l=1 rev1=1 crypto rx=0x7fc8c4009fc0 tx=0x7fc8c401cc60 comp rx=0 tx=0).stop 2023-05-22T16:05:53.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.111+0000 7fc8cad9d640 1 -- 172.21.15.16:0/1047969889 reap_dead start 2023-05-22T16:05:53.118 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.111+0000 7fc8d0eba640 1 -- 172.21.15.16:0/1047969889 shutdown_connections 2023-05-22T16:05:53.118 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.111+0000 7fc8d0eba640 1 -- 172.21.15.16:0/1047969889 >> 172.21.15.16:0/1047969889 conn(0x7fc8cc099620 msgr2=0x7fc8cc0d02c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:05:53.118 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.111+0000 7fc8d0eba640 1 -- 172.21.15.16:0/1047969889 shutdown_connections 2023-05-22T16:05:53.118 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.111+0000 7fc8d0eba640 1 -- 172.21.15.16:0/1047969889 wait complete. 2023-05-22T16:05:53.118 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.111+0000 7fc8d0eba640 1 Processor -- start 2023-05-22T16:05:53.118 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.111+0000 7fc8d0eba640 1 -- start start 2023-05-22T16:05:53.118 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.111+0000 7fc8d0eba640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc8b4004d10 0x7fc8cc1506c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:53.119 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.111+0000 7fc8d0eba640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc8cc15c830 0x7fc8cc1e58f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:53.119 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.111+0000 7fc8d0eba640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc8cc15e670 0x7fc8cc1e5e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:53.119 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.111+0000 7fc8d0eba640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8cc15e490 con 0x7fc8cc15e670 2023-05-22T16:05:53.119 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.111+0000 7fc8d0eba640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8cc15ded0 con 0x7fc8cc15c830 2023-05-22T16:05:53.119 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.111+0000 7fc8d0eba640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8cc15e1b0 con 0x7fc8b4004d10 2023-05-22T16:05:53.119 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.111+0000 7fc8cad9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc8cc15e670 0x7fc8cc1e5e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:53.120 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.111+0000 7fc8cad9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc8cc15e670 0x7fc8cc1e5e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38372/0 (socket says 172.21.15.16:38372) 2023-05-22T16:05:53.120 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.111+0000 7fc8cad9d640 1 -- 172.21.15.16:0/4022092759 learned_addr learned my addr 172.21.15.16:0/4022092759 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:05:53.120 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.111+0000 7fc8c9d9b640 1 --2- 172.21.15.16:0/4022092759 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc8cc15c830 0x7fc8cc1e58f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:53.120 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.111+0000 7fc8ca59c640 1 --2- 172.21.15.16:0/4022092759 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc8b4004d10 0x7fc8cc1506c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:53.120 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.111+0000 7fc8cad9d640 1 -- 172.21.15.16:0/4022092759 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc8b4004d10 msgr2=0x7fc8cc1506c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:53.120 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.111+0000 7fc8cad9d640 1 --2- 172.21.15.16:0/4022092759 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc8b4004d10 0x7fc8cc1506c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:53.120 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.111+0000 7fc8cad9d640 1 -- 172.21.15.16:0/4022092759 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc8cc15c830 msgr2=0x7fc8cc1e58f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:53.121 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.111+0000 7fc8cad9d640 1 --2- 172.21.15.16:0/4022092759 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc8cc15c830 0x7fc8cc1e58f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:53.121 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.111+0000 7fc8cad9d640 1 -- 172.21.15.16:0/4022092759 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8cc0cb5b0 con 0x7fc8cc15e670 2023-05-22T16:05:53.121 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.111+0000 7fc8ca59c640 1 --2- 172.21.15.16:0/4022092759 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc8b4004d10 0x7fc8cc1506c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T16:05:53.121 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.111+0000 7fc8c9d9b640 1 --2- 172.21.15.16:0/4022092759 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc8cc15c830 0x7fc8cc1e58f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T16:05:53.121 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.111+0000 7fc8cad9d640 1 --2- 172.21.15.16:0/4022092759 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc8cc15e670 0x7fc8cc1e5e30 secure :-1 s=READY pgs=2549 cs=0 l=1 rev1=1 crypto rx=0x7fc8c40b0040 tx=0x7fc8c4009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:53.121 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.111+0000 7fc8baffd640 1 -- 172.21.15.16:0/4022092759 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8c40bc070 con 0x7fc8cc15e670 2023-05-22T16:05:53.121 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.111+0000 7fc8baffd640 1 -- 172.21.15.16:0/4022092759 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8c40039a0 con 0x7fc8cc15e670 2023-05-22T16:05:53.121 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.111+0000 7fc8d0eba640 1 -- 172.21.15.16:0/4022092759 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8cc0cb2d0 con 0x7fc8cc15e670 2023-05-22T16:05:53.122 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.111+0000 7fc8baffd640 1 -- 172.21.15.16:0/4022092759 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8c4003b30 con 0x7fc8cc15e670 2023-05-22T16:05:53.122 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.111+0000 7fc8d0eba640 1 -- 172.21.15.16:0/4022092759 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8cc0cb890 con 0x7fc8cc15e670 2023-05-22T16:05:53.122 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.115+0000 7fc8d0eba640 1 -- 172.21.15.16:0/4022092759 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8cc0c86c0 con 0x7fc8cc15e670 2023-05-22T16:05:53.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.115+0000 7fc8baffd640 1 -- 172.21.15.16:0/4022092759 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 97) v1 ==== 84389+0+0 (secure 0 0 0) 0x7fc8c40b4040 con 0x7fc8cc15e670 2023-05-22T16:05:53.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.115+0000 7fc8baffd640 1 --2- 172.21.15.16:0/4022092759 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fc8ac065530 0x7fc8ac0679e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:53.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.115+0000 7fc8ca59c640 1 -- 172.21.15.16:0/4022092759 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fc8ac065530 msgr2=0x7fc8ac0679e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/65890 2023-05-22T16:05:53.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.115+0000 7fc8baffd640 1 -- 172.21.15.16:0/4022092759 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(197..197 src has 1..197) v4 ==== 6782+0+0 (secure 0 0 0) 0x7fc8c411d930 con 0x7fc8cc15e670 2023-05-22T16:05:53.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.115+0000 7fc8ca59c640 1 --2- 172.21.15.16:0/4022092759 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fc8ac065530 0x7fc8ac0679e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-05-22T16:05:53.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.119+0000 7fc8baffd640 1 -- 172.21.15.16:0/4022092759 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fc8c40eb180 con 0x7fc8cc15e670 2023-05-22T16:05:53.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.271+0000 7fc8d0eba640 1 -- 172.21.15.16:0/4022092759 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fc8cc0c86c0 con 0x7fc8cc15e670 2023-05-22T16:05:53.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.271+0000 7fc8baffd640 1 -- 172.21.15.16:0/4022092759 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v197) v1 ==== 92+0+10 (secure 0 0 0) 0x7fc8cc0c86c0 con 0x7fc8cc15e670 2023-05-22T16:05:53.276 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 8 2023-05-22T16:05:53.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.275+0000 7fc8d0eba640 1 -- 172.21.15.16:0/4022092759 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fc8ac065530 msgr2=0x7fc8ac0679e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-05-22T16:05:53.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.275+0000 7fc8d0eba640 1 --2- 172.21.15.16:0/4022092759 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fc8ac065530 0x7fc8ac0679e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:53.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.275+0000 7fc8d0eba640 1 -- 172.21.15.16:0/4022092759 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc8cc15e670 msgr2=0x7fc8cc1e5e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:53.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.275+0000 7fc8d0eba640 1 --2- 172.21.15.16:0/4022092759 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc8cc15e670 0x7fc8cc1e5e30 secure :-1 s=READY pgs=2549 cs=0 l=1 rev1=1 crypto rx=0x7fc8c40b0040 tx=0x7fc8c4009750 comp rx=0 tx=0).stop 2023-05-22T16:05:53.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.275+0000 7fc8d0eba640 1 -- 172.21.15.16:0/4022092759 shutdown_connections 2023-05-22T16:05:53.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.275+0000 7fc8d0eba640 1 --2- 172.21.15.16:0/4022092759 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fc8ac065530 0x7fc8ac0679e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:53.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.275+0000 7fc8d0eba640 1 --2- 172.21.15.16:0/4022092759 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc8cc15e670 0x7fc8cc1e5e30 unknown :-1 s=CLOSED pgs=2549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:53.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.275+0000 7fc8d0eba640 1 --2- 172.21.15.16:0/4022092759 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc8cc15c830 0x7fc8cc1e58f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:53.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.275+0000 7fc8d0eba640 1 --2- 172.21.15.16:0/4022092759 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc8b4004d10 0x7fc8cc1506c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:53.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.275+0000 7fc8d0eba640 1 -- 172.21.15.16:0/4022092759 >> 172.21.15.16:0/4022092759 conn(0x7fc8cc099620 msgr2=0x7fc8cc06d8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:05:53.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.275+0000 7fc8d0eba640 1 -- 172.21.15.16:0/4022092759 shutdown_connections 2023-05-22T16:05:53.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.275+0000 7fc8d0eba640 1 -- 172.21.15.16:0/4022092759 wait complete. 2023-05-22T16:05:53.294 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-05-22T16:05:53.333 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T16:05:53.327+0000 7f69a9b59180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-05-22T16:05:53.368 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T16:05:53.363+0000 7fed27eaa180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-05-22T16:05:53.386 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T16:05:53.379+0000 7f69a9b59180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-05-22T16:05:53.415 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.411+0000 7f5dfb765640 1 Processor -- start 2023-05-22T16:05:53.415 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.411+0000 7f5dfb765640 1 -- start start 2023-05-22T16:05:53.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.411+0000 7f5dfb765640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5df406ddd0 0x7f5df406e200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:53.417 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.411+0000 7f5dfb765640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f5df415df00 0x7f5df415e380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:53.417 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.411+0000 7f5dfb765640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5df4071370 con 0x7f5df4069eb0 2023-05-22T16:05:53.417 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.411+0000 7f5dfb765640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5df415a2f0 con 0x7f5df415df00 2023-05-22T16:05:53.417 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.411+0000 7f5dfb765640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5df406a280 con 0x7f5df406ddd0 2023-05-22T16:05:53.417 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.415+0000 7f5df9d02640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f5df415df00 0x7f5df415e380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:53.417 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.415+0000 7f5df9d02640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f5df415df00 0x7f5df415e380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38376/0 (socket says 172.21.15.16:38376) 2023-05-22T16:05:53.418 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.415+0000 7f5df9d02640 1 -- 172.21.15.16:0/1233067878 learned_addr learned my addr 172.21.15.16:0/1233067878 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:05:53.418 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.415+0000 7f5df8d00640 1 --2- 172.21.15.16:0/1233067878 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5df406ddd0 0x7f5df406e200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:53.418 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.415+0000 7f5df9d02640 1 -- 172.21.15.16:0/1233067878 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5df406ddd0 msgr2=0x7f5df406e200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:05:53.418 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.415+0000 7f5df9d02640 1 --2- 172.21.15.16:0/1233067878 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5df406ddd0 0x7f5df406e200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:53.418 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.415+0000 7f5df9d02640 1 -- 172.21.15.16:0/1233067878 >> v1:172.21.15.16:6789/0 conn(0x7f5df4069eb0 legacy=0x7f5df40b1680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:53.418 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.415+0000 7f5df9d02640 1 -- 172.21.15.16:0/1233067878 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5df4071650 con 0x7f5df415df00 2023-05-22T16:05:53.418 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.415+0000 7f5df8d00640 1 --2- 172.21.15.16:0/1233067878 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5df406ddd0 0x7f5df406e200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T16:05:53.419 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.415+0000 7f5df9d02640 1 --2- 172.21.15.16:0/1233067878 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5df415df00 0x7f5df415e380 secure :-1 s=READY pgs=2551 cs=0 l=1 rev1=1 crypto rx=0x7f5dec009e50 tx=0x7f5dec01f1f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5b965fe46cc3332 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:53.419 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.415+0000 7f5debfff640 1 -- 172.21.15.16:0/1233067878 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dec0af400 con 0x7f5df415df00 2023-05-22T16:05:53.419 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.415+0000 7f5debfff640 1 -- 172.21.15.16:0/1233067878 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5dec0ab050 con 0x7f5df415df00 2023-05-22T16:05:53.419 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.415+0000 7f5debfff640 1 -- 172.21.15.16:0/1233067878 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dec0af780 con 0x7f5df415df00 2023-05-22T16:05:53.420 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.415+0000 7f5dfb765640 1 -- 172.21.15.16:0/1233067878 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5df415df00 msgr2=0x7f5df415e380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:53.420 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.415+0000 7f5dfb765640 1 --2- 172.21.15.16:0/1233067878 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5df415df00 0x7f5df415e380 secure :-1 s=READY pgs=2551 cs=0 l=1 rev1=1 crypto rx=0x7f5dec009e50 tx=0x7f5dec01f1f0 comp rx=0 tx=0).stop 2023-05-22T16:05:53.420 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.415+0000 7f5dfb765640 1 -- 172.21.15.16:0/1233067878 shutdown_connections 2023-05-22T16:05:53.420 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.415+0000 7f5dfb765640 1 --2- 172.21.15.16:0/1233067878 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5df415df00 0x7f5df415e380 unknown :-1 s=CLOSED pgs=2551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:53.421 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.415+0000 7f5dfb765640 1 --2- 172.21.15.16:0/1233067878 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5df406ddd0 0x7f5df406e200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:53.421 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.415+0000 7f5dfb765640 1 -- 172.21.15.16:0/1233067878 >> 172.21.15.16:0/1233067878 conn(0x7f5df406d5b0 msgr2=0x7f5df40b0de0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:05:53.421 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.415+0000 7f5dfb765640 1 -- 172.21.15.16:0/1233067878 shutdown_connections 2023-05-22T16:05:53.421 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.415+0000 7f5dfb765640 1 -- 172.21.15.16:0/1233067878 wait complete. 2023-05-22T16:05:53.422 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.419+0000 7f5dfb765640 1 Processor -- start 2023-05-22T16:05:53.422 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.419+0000 7f5dfb765640 1 -- start start 2023-05-22T16:05:53.423 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.419+0000 7f5dfb765640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5df4069eb0 0x7f5df40c4530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:53.423 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.419+0000 7f5dfb765640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5df406ddd0 0x7f5df41ef780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:53.423 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.419+0000 7f5dfb765640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5df415df00 0x7f5df415cae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:53.423 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.419+0000 7f5dfb765640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5df40c3b30 con 0x7f5df415df00 2023-05-22T16:05:53.423 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.419+0000 7f5dfb765640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5df415c370 con 0x7f5df4069eb0 2023-05-22T16:05:53.423 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.419+0000 7f5dfb765640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5df41effb0 con 0x7f5df406ddd0 2023-05-22T16:05:53.423 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.419+0000 7f5df9d02640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5df415df00 0x7f5df415cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:53.424 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.419+0000 7f5df9d02640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5df415df00 0x7f5df415cae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38392/0 (socket says 172.21.15.16:38392) 2023-05-22T16:05:53.424 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.419+0000 7f5df9d02640 1 -- 172.21.15.16:0/1574804059 learned_addr learned my addr 172.21.15.16:0/1574804059 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:05:53.424 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.419+0000 7f5df8d00640 1 --2- 172.21.15.16:0/1574804059 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5df406ddd0 0x7f5df41ef780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:53.424 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.419+0000 7f5df9501640 1 --2- 172.21.15.16:0/1574804059 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5df4069eb0 0x7f5df40c4530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:53.424 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.419+0000 7f5df9d02640 1 -- 172.21.15.16:0/1574804059 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5df406ddd0 msgr2=0x7f5df41ef780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:53.424 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.419+0000 7f5df9d02640 1 --2- 172.21.15.16:0/1574804059 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5df406ddd0 0x7f5df41ef780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:53.424 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.419+0000 7f5df9d02640 1 -- 172.21.15.16:0/1574804059 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5df4069eb0 msgr2=0x7f5df40c4530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:53.425 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.419+0000 7f5df9d02640 1 --2- 172.21.15.16:0/1574804059 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5df4069eb0 0x7f5df40c4530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:53.425 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.419+0000 7f5df9d02640 1 -- 172.21.15.16:0/1574804059 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5dec009b30 con 0x7f5df415df00 2023-05-22T16:05:53.425 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.419+0000 7f5df9d02640 1 --2- 172.21.15.16:0/1574804059 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5df415df00 0x7f5df415cae0 secure :-1 s=READY pgs=2552 cs=0 l=1 rev1=1 crypto rx=0x7f5dec01ce20 tx=0x7f5dec01bf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:53.425 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T16:05:53.419+0000 7fed27eaa180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-05-22T16:05:53.426 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.419+0000 7f5de9ffb640 1 -- 172.21.15.16:0/1574804059 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dec0afde0 con 0x7f5df415df00 2023-05-22T16:05:53.426 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.419+0000 7f5de9ffb640 1 -- 172.21.15.16:0/1574804059 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5dec009510 con 0x7f5df415df00 2023-05-22T16:05:53.426 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.419+0000 7f5de9ffb640 1 -- 172.21.15.16:0/1574804059 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dec0b8640 con 0x7f5df415df00 2023-05-22T16:05:53.427 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.419+0000 7f5dfb765640 1 -- 172.21.15.16:0/1574804059 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5df4071c10 con 0x7f5df415df00 2023-05-22T16:05:53.427 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.419+0000 7f5dfb765640 1 -- 172.21.15.16:0/1574804059 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5df4071930 con 0x7f5df415df00 2023-05-22T16:05:53.427 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.419+0000 7f5dfb765640 1 -- 172.21.15.16:0/1574804059 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5df415d9e0 con 0x7f5df415df00 2023-05-22T16:05:53.427 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.419+0000 7f5de9ffb640 1 -- 172.21.15.16:0/1574804059 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 97) v1 ==== 84389+0+0 (secure 0 0 0) 0x7f5dec0b6040 con 0x7f5df415df00 2023-05-22T16:05:53.427 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.423+0000 7f5de9ffb640 1 --2- 172.21.15.16:0/1574804059 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f5dd8065530 0x7f5dd80679e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:53.427 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.423+0000 7f5df9501640 1 -- 172.21.15.16:0/1574804059 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f5dd8065530 msgr2=0x7f5dd80679e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/65890 2023-05-22T16:05:53.427 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.423+0000 7f5de9ffb640 1 -- 172.21.15.16:0/1574804059 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(197..197 src has 1..197) v4 ==== 6782+0+0 (secure 0 0 0) 0x7f5dec120f70 con 0x7f5df415df00 2023-05-22T16:05:53.428 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.423+0000 7f5df9501640 1 --2- 172.21.15.16:0/1574804059 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f5dd8065530 0x7f5dd80679e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-05-22T16:05:53.428 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.427+0000 7f5de9ffb640 1 -- 172.21.15.16:0/1574804059 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f5dec0ee650 con 0x7f5df415df00 2023-05-22T16:05:53.436 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T16:05:53.431+0000 7f69a9b59180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-05-22T16:05:53.474 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T16:05:53.471+0000 7fed27eaa180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-05-22T16:05:53.515 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T16:05:53.511+0000 7f69a9b59180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-05-22T16:05:53.553 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T16:05:53.551+0000 7fed27eaa180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-05-22T16:05:53.566 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T16:05:53.563+0000 7f69a9b59180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-05-22T16:05:53.583 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.579+0000 7f5dfb765640 1 -- 172.21.15.16:0/1574804059 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f5df40b3090 con 0x7f5df415df00 2023-05-22T16:05:53.584 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.579+0000 7f5de9ffb640 1 -- 172.21.15.16:0/1574804059 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v197) v1 ==== 104+0+11 (secure 0 0 0) 0x7f5dec0f1c70 con 0x7f5df415df00 2023-05-22T16:05:53.584 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2023-05-22T16:05:53.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.583+0000 7f5dfb765640 1 -- 172.21.15.16:0/1574804059 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f5dd8065530 msgr2=0x7f5dd80679e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-05-22T16:05:53.587 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.583+0000 7f5dfb765640 1 --2- 172.21.15.16:0/1574804059 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f5dd8065530 0x7f5dd80679e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:53.587 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.583+0000 7f5dfb765640 1 -- 172.21.15.16:0/1574804059 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5df415df00 msgr2=0x7f5df415cae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:53.587 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.583+0000 7f5dfb765640 1 --2- 172.21.15.16:0/1574804059 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5df415df00 0x7f5df415cae0 secure :-1 s=READY pgs=2552 cs=0 l=1 rev1=1 crypto rx=0x7f5dec01ce20 tx=0x7f5dec01bf90 comp rx=0 tx=0).stop 2023-05-22T16:05:53.588 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.583+0000 7f5dfb765640 1 -- 172.21.15.16:0/1574804059 shutdown_connections 2023-05-22T16:05:53.588 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.583+0000 7f5dfb765640 1 --2- 172.21.15.16:0/1574804059 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f5dd8065530 0x7f5dd80679e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:53.588 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.583+0000 7f5dfb765640 1 --2- 172.21.15.16:0/1574804059 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5df415df00 0x7f5df415cae0 unknown :-1 s=CLOSED pgs=2552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:53.588 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.583+0000 7f5dfb765640 1 --2- 172.21.15.16:0/1574804059 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5df406ddd0 0x7f5df41ef780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:53.589 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.583+0000 7f5dfb765640 1 --2- 172.21.15.16:0/1574804059 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5df4069eb0 0x7f5df40c4530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:53.589 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.583+0000 7f5dfb765640 1 -- 172.21.15.16:0/1574804059 >> 172.21.15.16:0/1574804059 conn(0x7f5df406d5b0 msgr2=0x7f5df40b2100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:05:53.589 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.583+0000 7f5dfb765640 1 -- 172.21.15.16:0/1574804059 shutdown_connections 2023-05-22T16:05:53.590 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.583+0000 7f5dfb765640 1 -- 172.21.15.16:0/1574804059 wait complete. 2023-05-22T16:05:53.603 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-05-22T16:05:53.608 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T16:05:53.603+0000 7fed27eaa180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-05-22T16:05:53.644 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T16:05:53.639+0000 7f69a9b59180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-05-22T16:05:53.688 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T16:05:53.683+0000 7fed27eaa180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-05-22T16:05:53.697 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T16:05:53.691+0000 7f69a9b59180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-05-22T16:05:53.722 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.719+0000 7fc2e3c21640 1 Processor -- start 2023-05-22T16:05:53.722 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.719+0000 7fc2e3c21640 1 -- start start 2023-05-22T16:05:53.724 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.719+0000 7fc2e3c21640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc2dc069eb0 0x7fc2dc0b1680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:53.724 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.719+0000 7fc2e3c21640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc2dc06e200 0x7fc2dc0ce580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:53.725 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.719+0000 7fc2e3c21640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2dc071370 con 0x7fc2dc06ddd0 2023-05-22T16:05:53.725 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.719+0000 7fc2e3c21640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2dc15a2f0 con 0x7fc2dc069eb0 2023-05-22T16:05:53.725 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.719+0000 7fc2e3c21640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2dc06a280 con 0x7fc2dc06e200 2023-05-22T16:05:53.725 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.719+0000 7fc2e19bd640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc2dc069eb0 0x7fc2dc0b1680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:53.726 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.719+0000 7fc2e19bd640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc2dc069eb0 0x7fc2dc0b1680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38404/0 (socket says 172.21.15.16:38404) 2023-05-22T16:05:53.726 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.719+0000 7fc2e19bd640 1 -- 172.21.15.16:0/1667376734 learned_addr learned my addr 172.21.15.16:0/1667376734 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:05:53.726 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.719+0000 7fc2e21be640 1 --2- 172.21.15.16:0/1667376734 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc2dc06e200 0x7fc2dc0ce580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:53.726 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.719+0000 7fc2e19bd640 1 -- 172.21.15.16:0/1667376734 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc2dc06e200 msgr2=0x7fc2dc0ce580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:05:53.727 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.719+0000 7fc2e19bd640 1 --2- 172.21.15.16:0/1667376734 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc2dc06e200 0x7fc2dc0ce580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:53.727 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.719+0000 7fc2e19bd640 1 -- 172.21.15.16:0/1667376734 >> v1:172.21.15.62:6789/0 conn(0x7fc2dc06ddd0 legacy=0x7fc2dc15df00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:53.727 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.719+0000 7fc2e19bd640 1 -- 172.21.15.16:0/1667376734 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2dc071650 con 0x7fc2dc069eb0 2023-05-22T16:05:53.731 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.723+0000 7fc2e19bd640 1 --2- 172.21.15.16:0/1667376734 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc2dc069eb0 0x7fc2dc0b1680 secure :-1 s=READY pgs=2553 cs=0 l=1 rev1=1 crypto rx=0x7fc2cc009870 tx=0x7fc2cc01ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e4db9f668a34244 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:53.731 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.723+0000 7fc2e09bb640 1 -- 172.21.15.16:0/1667376734 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2cc0a8a30 con 0x7fc2dc069eb0 2023-05-22T16:05:53.732 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.723+0000 7fc2e09bb640 1 -- 172.21.15.16:0/1667376734 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2cc0a8b90 con 0x7fc2dc069eb0 2023-05-22T16:05:53.732 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.723+0000 7fc2e09bb640 1 -- 172.21.15.16:0/1667376734 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2cc0b0440 con 0x7fc2dc069eb0 2023-05-22T16:05:53.732 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.723+0000 7fc2e3c21640 1 -- 172.21.15.16:0/1667376734 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc2dc069eb0 msgr2=0x7fc2dc0b1680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:53.732 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.723+0000 7fc2e3c21640 1 --2- 172.21.15.16:0/1667376734 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc2dc069eb0 0x7fc2dc0b1680 secure :-1 s=READY pgs=2553 cs=0 l=1 rev1=1 crypto rx=0x7fc2cc009870 tx=0x7fc2cc01ed30 comp rx=0 tx=0).stop 2023-05-22T16:05:53.733 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.723+0000 7fc2e3c21640 1 -- 172.21.15.16:0/1667376734 shutdown_connections 2023-05-22T16:05:53.733 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.723+0000 7fc2e3c21640 1 --2- 172.21.15.16:0/1667376734 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc2dc06e200 0x7fc2dc0ce580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:53.733 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.723+0000 7fc2e3c21640 1 --2- 172.21.15.16:0/1667376734 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc2dc069eb0 0x7fc2dc0b1680 unknown :-1 s=CLOSED pgs=2553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:53.734 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.723+0000 7fc2e3c21640 1 -- 172.21.15.16:0/1667376734 >> 172.21.15.16:0/1667376734 conn(0x7fc2dc06d5b0 msgr2=0x7fc2dc0b0de0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:05:53.734 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.723+0000 7fc2e3c21640 1 -- 172.21.15.16:0/1667376734 shutdown_connections 2023-05-22T16:05:53.734 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.723+0000 7fc2e3c21640 1 -- 172.21.15.16:0/1667376734 wait complete. 2023-05-22T16:05:53.734 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.723+0000 7fc2e3c21640 1 Processor -- start 2023-05-22T16:05:53.735 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.723+0000 7fc2e3c21640 1 -- start start 2023-05-22T16:05:53.735 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.727+0000 7fc2e3c21640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc2dc069eb0 0x7fc2dc0a7ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:53.735 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.727+0000 7fc2e3c21640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc2dc06ddd0 0x7fc2dc15c610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:53.736 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.727+0000 7fc2e3c21640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc2dc06e200 0x7fc2dc15cb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:53.736 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.727+0000 7fc2e3c21640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2dc0c9270 con 0x7fc2dc069eb0 2023-05-22T16:05:53.736 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.727+0000 7fc2e3c21640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2dc0c86f0 con 0x7fc2dc06e200 2023-05-22T16:05:53.736 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.727+0000 7fc2e3c21640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2dc0c89d0 con 0x7fc2dc06ddd0 2023-05-22T16:05:53.737 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.727+0000 7fc2e19bd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc2dc069eb0 0x7fc2dc0a7ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:53.737 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.727+0000 7fc2e19bd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc2dc069eb0 0x7fc2dc0a7ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38418/0 (socket says 172.21.15.16:38418) 2023-05-22T16:05:53.737 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.727+0000 7fc2e19bd640 1 -- 172.21.15.16:0/3728241438 learned_addr learned my addr 172.21.15.16:0/3728241438 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:05:53.737 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.727+0000 7fc2e21be640 1 --2- 172.21.15.16:0/3728241438 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc2dc06e200 0x7fc2dc15cb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:53.738 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.727+0000 7fc2e11bc640 1 --2- 172.21.15.16:0/3728241438 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc2dc06ddd0 0x7fc2dc15c610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:53.738 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.727+0000 7fc2e19bd640 1 -- 172.21.15.16:0/3728241438 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc2dc06ddd0 msgr2=0x7fc2dc15c610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:53.738 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.727+0000 7fc2e19bd640 1 --2- 172.21.15.16:0/3728241438 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc2dc06ddd0 0x7fc2dc15c610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:53.739 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.727+0000 7fc2e19bd640 1 -- 172.21.15.16:0/3728241438 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc2dc06e200 msgr2=0x7fc2dc15cb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:53.739 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.727+0000 7fc2e19bd640 1 --2- 172.21.15.16:0/3728241438 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc2dc06e200 0x7fc2dc15cb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:53.739 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.727+0000 7fc2e19bd640 1 -- 172.21.15.16:0/3728241438 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2cc0094d0 con 0x7fc2dc069eb0 2023-05-22T16:05:53.739 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.727+0000 7fc2e19bd640 1 --2- 172.21.15.16:0/3728241438 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc2dc069eb0 0x7fc2dc0a7ec0 secure :-1 s=READY pgs=2554 cs=0 l=1 rev1=1 crypto rx=0x7fc2cc0b3040 tx=0x7fc2cc01bad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:53.740 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.727+0000 7fc2d27fc640 1 -- 172.21.15.16:0/3728241438 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2cc0be070 con 0x7fc2dc069eb0 2023-05-22T16:05:53.740 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.727+0000 7fc2d27fc640 1 -- 172.21.15.16:0/3728241438 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2cc0ba400 con 0x7fc2dc069eb0 2023-05-22T16:05:53.740 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.727+0000 7fc2e3c21640 1 -- 172.21.15.16:0/3728241438 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2dc071c10 con 0x7fc2dc069eb0 2023-05-22T16:05:53.740 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.727+0000 7fc2d27fc640 1 -- 172.21.15.16:0/3728241438 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2cc0ba7d0 con 0x7fc2dc069eb0 2023-05-22T16:05:53.741 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.727+0000 7fc2e3c21640 1 -- 172.21.15.16:0/3728241438 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2dc071930 con 0x7fc2dc069eb0 2023-05-22T16:05:53.741 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.727+0000 7fc2e3c21640 1 -- 172.21.15.16:0/3728241438 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2dc15d9e0 con 0x7fc2dc069eb0 2023-05-22T16:05:53.741 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.727+0000 7fc2d27fc640 1 -- 172.21.15.16:0/3728241438 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 97) v1 ==== 84389+0+0 (secure 0 0 0) 0x7fc2cc0c3410 con 0x7fc2dc069eb0 2023-05-22T16:05:53.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.727+0000 7fc2d27fc640 1 --2- 172.21.15.16:0/3728241438 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fc2c0065530 0x7fc2c00679e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:53.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.727+0000 7fc2e11bc640 1 -- 172.21.15.16:0/3728241438 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fc2c0065530 msgr2=0x7fc2c00679e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/65890 2023-05-22T16:05:53.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.727+0000 7fc2d27fc640 1 -- 172.21.15.16:0/3728241438 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(197..197 src has 1..197) v4 ==== 6782+0+0 (secure 0 0 0) 0x7fc2cc1208f0 con 0x7fc2dc069eb0 2023-05-22T16:05:53.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.727+0000 7fc2e11bc640 1 --2- 172.21.15.16:0/3728241438 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fc2c0065530 0x7fc2c00679e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-05-22T16:05:53.744 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.731+0000 7fc2d27fc640 1 -- 172.21.15.16:0/3728241438 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fc2cc0ee050 con 0x7fc2dc069eb0 2023-05-22T16:05:53.744 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T16:05:53.739+0000 7fed27eaa180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-05-22T16:05:53.831 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T16:05:53.827+0000 7f69a9b59180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-05-22T16:05:53.876 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T16:05:53.875+0000 7fed27eaa180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-05-22T16:05:53.884 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T16:05:53.879+0000 7f69a9b59180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-05-22T16:05:53.886 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.883+0000 7fc2e3c21640 1 -- 172.21.15.16:0/3728241438 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fc2dc0c8b10 con 0x7fc2dc069eb0 2023-05-22T16:05:53.887 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.883+0000 7fc2d27fc640 1 -- 172.21.15.16:0/3728241438 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v197) v1 ==== 100+0+11 (secure 0 0 0) 0x7fc2dc0c8b10 con 0x7fc2dc069eb0 2023-05-22T16:05:53.887 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 64 2023-05-22T16:05:53.890 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.887+0000 7fc2e3c21640 1 -- 172.21.15.16:0/3728241438 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fc2c0065530 msgr2=0x7fc2c00679e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-05-22T16:05:53.890 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.887+0000 7fc2e3c21640 1 --2- 172.21.15.16:0/3728241438 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fc2c0065530 0x7fc2c00679e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:53.891 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.887+0000 7fc2e3c21640 1 -- 172.21.15.16:0/3728241438 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc2dc069eb0 msgr2=0x7fc2dc0a7ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:53.891 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.887+0000 7fc2e3c21640 1 --2- 172.21.15.16:0/3728241438 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc2dc069eb0 0x7fc2dc0a7ec0 secure :-1 s=READY pgs=2554 cs=0 l=1 rev1=1 crypto rx=0x7fc2cc0b3040 tx=0x7fc2cc01bad0 comp rx=0 tx=0).stop 2023-05-22T16:05:53.891 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.887+0000 7fc2e3c21640 1 -- 172.21.15.16:0/3728241438 shutdown_connections 2023-05-22T16:05:53.891 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.887+0000 7fc2e3c21640 1 --2- 172.21.15.16:0/3728241438 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc2dc06e200 0x7fc2dc15cb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:53.892 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.887+0000 7fc2e3c21640 1 --2- 172.21.15.16:0/3728241438 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc2dc06ddd0 0x7fc2dc15c610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:53.892 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.887+0000 7fc2e3c21640 1 --2- 172.21.15.16:0/3728241438 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fc2c0065530 0x7fc2c00679e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:53.892 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.887+0000 7fc2e3c21640 1 --2- 172.21.15.16:0/3728241438 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc2dc069eb0 0x7fc2dc0a7ec0 unknown :-1 s=CLOSED pgs=2554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:53.892 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.887+0000 7fc2e3c21640 1 -- 172.21.15.16:0/3728241438 >> 172.21.15.16:0/3728241438 conn(0x7fc2dc06d5b0 msgr2=0x7fc2dc0b0a20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:05:53.893 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.887+0000 7fc2e3c21640 1 -- 172.21.15.16:0/3728241438 shutdown_connections 2023-05-22T16:05:53.893 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:53.887+0000 7fc2e3c21640 1 -- 172.21.15.16:0/3728241438 wait complete. 2023-05-22T16:05:53.906 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-05-22T16:05:53.931 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T16:05:53.927+0000 7fed27eaa180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-05-22T16:05:54.062 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:54.059+0000 7f36d50da640 1 Processor -- start 2023-05-22T16:05:54.062 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:54.059+0000 7f36d50da640 1 -- start start 2023-05-22T16:05:54.065 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:54.059+0000 7f36d50da640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f36d00cb270 0x7f36d015e450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:54.065 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:54.059+0000 7f36d50da640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f36d0069f10 0x7f36d0070860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:54.065 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:54.059+0000 7f36d50da640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36d015da00 con 0x7f36d00cb640 2023-05-22T16:05:54.066 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:54.059+0000 7f36ced9d640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f36d00cb270 0x7f36d015e450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:54.066 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:54.059+0000 7f36ced9d640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f36d00cb270 0x7f36d015e450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38430/0 (socket says 172.21.15.16:38430) 2023-05-22T16:05:54.066 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:54.059+0000 7f36ced9d640 1 -- 172.21.15.16:0/4291933234 learned_addr learned my addr 172.21.15.16:0/4291933234 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:05:54.067 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:54.059+0000 7f36d50da640 1 -- 172.21.15.16:0/4291933234 --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36d015dfc0 con 0x7f36d00cb270 2023-05-22T16:05:54.067 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:54.059+0000 7f36cf59e640 1 --2- 172.21.15.16:0/4291933234 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f36d0069f10 0x7f36d0070860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:54.067 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:54.059+0000 7f36d50da640 1 -- 172.21.15.16:0/4291933234 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36d015a310 con 0x7f36d0069f10 2023-05-22T16:05:54.068 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:54.063+0000 7f36cdd9b640 1 -- 172.21.15.16:0/4291933234 <== mon.1 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1639395886 0 0) 0x7f36d015da00 con 0x7f36d00cb640 2023-05-22T16:05:54.068 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:54.063+0000 7f36cdd9b640 1 -- 172.21.15.16:0/4291933234 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36bc003530 con 0x7f36d00cb640 2023-05-22T16:05:54.068 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:54.063+0000 7f36ced9d640 1 -- 172.21.15.16:0/4291933234 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f36d0069f10 msgr2=0x7f36d0070860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:05:54.068 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:54.063+0000 7f36ced9d640 1 --2- 172.21.15.16:0/4291933234 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f36d0069f10 0x7f36d0070860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:54.069 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:54.063+0000 7f36ced9d640 1 -- 172.21.15.16:0/4291933234 >> v1:172.21.15.62:6789/0 conn(0x7f36d00cb640 legacy=0x7f36d0069800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:54.069 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:54.063+0000 7f36ced9d640 1 -- 172.21.15.16:0/4291933234 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36d00cba70 con 0x7f36d00cb270 2023-05-22T16:05:54.070 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:54.067+0000 7f36ced9d640 1 --2- 172.21.15.16:0/4291933234 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f36d00cb270 0x7f36d015e450 secure :-1 s=READY pgs=2555 cs=0 l=1 rev1=1 crypto rx=0x7f36c8009e50 tx=0x7f36c801ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50c2e2f5127a5342 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:54.070 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:54.067+0000 7f36cdd9b640 1 -- 172.21.15.16:0/4291933234 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36d00715b0 con 0x7f36d00cb270 2023-05-22T16:05:54.070 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:54.067+0000 7f36cdd9b640 1 -- 172.21.15.16:0/4291933234 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36c80a57d0 con 0x7f36d00cb270 2023-05-22T16:05:54.071 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:54.067+0000 7f36cdd9b640 1 -- 172.21.15.16:0/4291933234 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36c80a5b60 con 0x7f36d00cb270 2023-05-22T16:05:54.072 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:54.067+0000 7f36d50da640 1 -- 172.21.15.16:0/4291933234 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f36d00cb270 msgr2=0x7f36d015e450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:54.072 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:54.067+0000 7f36d50da640 1 --2- 172.21.15.16:0/4291933234 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f36d00cb270 0x7f36d015e450 secure :-1 s=READY pgs=2555 cs=0 l=1 rev1=1 crypto rx=0x7f36c8009e50 tx=0x7f36c801ca10 comp rx=0 tx=0).stop 2023-05-22T16:05:54.072 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:54.067+0000 7f36d50da640 1 -- 172.21.15.16:0/4291933234 shutdown_connections 2023-05-22T16:05:54.072 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:54.067+0000 7f36d50da640 1 --2- 172.21.15.16:0/4291933234 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f36d0069f10 0x7f36d0070860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:54.072 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:54.067+0000 7f36d50da640 1 --2- 172.21.15.16:0/4291933234 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f36d00cb270 0x7f36d015e450 unknown :-1 s=CLOSED pgs=2555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:54.073 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:54.067+0000 7f36d50da640 1 -- 172.21.15.16:0/4291933234 >> 172.21.15.16:0/4291933234 conn(0x7f36d00cc9f0 msgr2=0x7f36d00cf2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:05:54.073 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:54.067+0000 7f36d50da640 1 -- 172.21.15.16:0/4291933234 shutdown_connections 2023-05-22T16:05:54.073 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:54.067+0000 7f36d50da640 1 -- 172.21.15.16:0/4291933234 wait complete. 2023-05-22T16:05:54.073 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:54.071+0000 7f36d50da640 1 Processor -- start 2023-05-22T16:05:54.074 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:54.071+0000 7f36d50da640 1 -- start start 2023-05-22T16:05:54.074 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:54.071+0000 7f36d50da640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f36d0069f10 0x7f36d0150930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:54.075 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:54.071+0000 7f36d50da640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f36d00cb270 0x7f36d01f3d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:54.075 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:54.071+0000 7f36d50da640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f36d00cb640 0x7f36d01f4270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:54.075 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:54.071+0000 7f36d50da640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36d0154990 con 0x7f36d00cb270 2023-05-22T16:05:54.075 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:54.071+0000 7f36d50da640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36d0153d10 con 0x7f36d0069f10 2023-05-22T16:05:54.075 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:54.071+0000 7f36d50da640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36d0153ff0 con 0x7f36d00cb640 2023-05-22T16:05:54.076 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:54.071+0000 7f36ced9d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f36d0069f10 0x7f36d0150930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:54.076 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:54.071+0000 7f36cf59e640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f36d00cb640 0x7f36d01f4270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:54.076 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:54.071+0000 7f36ced9d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f36d0069f10 0x7f36d0150930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.16:44756/0 (socket says 172.21.15.16:44756) 2023-05-22T16:05:54.076 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:54.071+0000 7f36ce59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f36d00cb270 0x7f36d01f3d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:54.076 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:54.071+0000 7f36ced9d640 1 -- 172.21.15.16:0/2252446372 learned_addr learned my addr 172.21.15.16:0/2252446372 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:05:54.077 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:54.071+0000 7f36ced9d640 1 -- 172.21.15.16:0/2252446372 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f36d00cb640 msgr2=0x7f36d01f4270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:54.077 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:54.071+0000 7f36ced9d640 1 --2- 172.21.15.16:0/2252446372 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f36d00cb640 0x7f36d01f4270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:54.077 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:54.071+0000 7f36ced9d640 1 -- 172.21.15.16:0/2252446372 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f36d00cb270 msgr2=0x7f36d01f3d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:54.077 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:54.071+0000 7f36ced9d640 1 --2- 172.21.15.16:0/2252446372 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f36d00cb270 0x7f36d01f3d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:54.078 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:54.071+0000 7f36ced9d640 1 -- 172.21.15.16:0/2252446372 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36c8009b30 con 0x7f36d0069f10 2023-05-22T16:05:54.078 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:54.071+0000 7f36cf59e640 1 --2- 172.21.15.16:0/2252446372 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f36d00cb640 0x7f36d01f4270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-05-22T16:05:54.078 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:54.071+0000 7f36ced9d640 1 --2- 172.21.15.16:0/2252446372 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f36d0069f10 0x7f36d0150930 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f36c801a640 tx=0x7f36c80197b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:54.078 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:54.075+0000 7f36c77fe640 1 -- 172.21.15.16:0/2252446372 <== mon.1 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36c80b8670 con 0x7f36d0069f10 2023-05-22T16:05:54.078 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:54.075+0000 7f36c77fe640 1 -- 172.21.15.16:0/2252446372 <== mon.1 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36c80b87d0 con 0x7f36d0069f10 2023-05-22T16:05:54.078 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:54.075+0000 7f36c77fe640 1 -- 172.21.15.16:0/2252446372 <== mon.1 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36c80b8b80 con 0x7f36d0069f10 2023-05-22T16:05:54.078 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:54.075+0000 7f36d50da640 1 -- 172.21.15.16:0/2252446372 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36d006e2e0 con 0x7f36d0069f10 2023-05-22T16:05:54.079 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:54.075+0000 7f36d50da640 1 -- 172.21.15.16:0/2252446372 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36d015dc50 con 0x7f36d0069f10 2023-05-22T16:05:54.079 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:54.075+0000 7f36d50da640 1 -- 172.21.15.16:0/2252446372 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36d006e470 con 0x7f36d0069f10 2023-05-22T16:05:54.080 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:54.075+0000 7f36c77fe640 1 -- 172.21.15.16:0/2252446372 <== mon.1 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 97) v1 ==== 84389+0+0 (secure 0 0 0) 0x7f36c80b8ce0 con 0x7f36d0069f10 2023-05-22T16:05:54.081 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:54.079+0000 7f36c77fe640 1 --2- 172.21.15.16:0/2252446372 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f36b0065580 0x7f36b0067a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:54.081 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:54.079+0000 7f36ce59c640 1 -- 172.21.15.16:0/2252446372 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f36b0065580 msgr2=0x7f36b0067a30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/65890 2023-05-22T16:05:54.081 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:54.079+0000 7f36c77fe640 1 -- 172.21.15.16:0/2252446372 <== mon.1 v2:172.21.15.62:3300/0 5 ==== osd_map(197..197 src has 1..197) v4 ==== 6782+0+0 (secure 0 0 0) 0x7f36c811e170 con 0x7f36d0069f10 2023-05-22T16:05:54.081 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:54.079+0000 7f36ce59c640 1 --2- 172.21.15.16:0/2252446372 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f36b0065580 0x7f36b0067a30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-05-22T16:05:54.090 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:54.087+0000 7f36c77fe640 1 -- 172.21.15.16:0/2252446372 <== mon.1 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f36c80ebaa0 con 0x7f36d0069f10 2023-05-22T16:05:54.115 INFO:tasks.ceph.mgr.x.smithi062.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2023-05-22T16:05:54.115 INFO:tasks.ceph.mgr.x.smithi062.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2023-05-22T16:05:54.115 INFO:tasks.ceph.mgr.x.smithi062.stderr: from numpy import show_config as show_numpy_config 2023-05-22T16:05:54.122 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T16:05:54.119+0000 7f69a9b59180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-05-22T16:05:54.168 INFO:tasks.ceph.mgr.y.smithi016.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2023-05-22T16:05:54.168 INFO:tasks.ceph.mgr.y.smithi016.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2023-05-22T16:05:54.168 INFO:tasks.ceph.mgr.y.smithi016.stderr: from numpy import show_config as show_numpy_config 2023-05-22T16:05:54.175 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T16:05:54.171+0000 7fed27eaa180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-05-22T16:05:54.249 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T16:05:54.243+0000 7f69a9b59180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-05-22T16:05:54.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:54.283+0000 7f36ce59c640 1 -- 172.21.15.16:0/2252446372 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f36b0065580 msgr2=0x7f36b0067a30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/65890 2023-05-22T16:05:54.285 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:54.283+0000 7f36ce59c640 1 --2- 172.21.15.16:0/2252446372 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f36b0065580 0x7f36b0067a30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-05-22T16:05:54.290 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:54.287+0000 7f36d50da640 1 -- 172.21.15.16:0/2252446372 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f36d0073050 con 0x7f36d0069f10 2023-05-22T16:05:54.291 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:54.287+0000 7f36c77fe640 1 -- 172.21.15.16:0/2252446372 <== mon.1 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 8 entries v197) v1 ==== 80+0+458 (secure 0 0 0) 0x7f36c80ef0c0 con 0x7f36d0069f10 2023-05-22T16:05:54.292 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/1961044213 2023-05-23T16:05:35.647562+0000 2023-05-22T16:05:54.292 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/238292614 2023-05-23T16:05:35.647562+0000 2023-05-22T16:05:54.292 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:6833/65890 2023-05-23T16:05:35.647562+0000 2023-05-22T16:05:54.292 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/1148602276 2023-05-23T16:05:35.647562+0000 2023-05-22T16:05:54.292 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:6832/65890 2023-05-23T16:05:35.647562+0000 2023-05-22T16:05:54.293 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/456460873 2023-05-23T16:05:35.647562+0000 2023-05-22T16:05:54.293 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/2766506027 2023-05-23T16:05:35.647562+0000 2023-05-22T16:05:54.293 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/2214029832 2023-05-23T16:05:35.647562+0000 2023-05-22T16:05:54.293 INFO:teuthology.orchestra.run.smithi016.stderr:listed 8 entries 2023-05-22T16:05:54.294 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:54.291+0000 7f36d50da640 1 -- 172.21.15.16:0/2252446372 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f36b0065580 msgr2=0x7f36b0067a30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-05-22T16:05:54.294 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:54.291+0000 7f36d50da640 1 --2- 172.21.15.16:0/2252446372 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f36b0065580 0x7f36b0067a30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:54.294 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:54.291+0000 7f36d50da640 1 -- 172.21.15.16:0/2252446372 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f36d0069f10 msgr2=0x7f36d0150930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:54.294 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:54.291+0000 7f36d50da640 1 --2- 172.21.15.16:0/2252446372 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f36d0069f10 0x7f36d0150930 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f36c801a640 tx=0x7f36c80197b0 comp rx=0 tx=0).stop 2023-05-22T16:05:54.294 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:54.291+0000 7f36d50da640 1 -- 172.21.15.16:0/2252446372 shutdown_connections 2023-05-22T16:05:54.294 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:54.291+0000 7f36d50da640 1 --2- 172.21.15.16:0/2252446372 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f36d00cb640 0x7f36d01f4270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:54.295 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:54.291+0000 7f36d50da640 1 --2- 172.21.15.16:0/2252446372 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f36b0065580 0x7f36b0067a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:54.295 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:54.291+0000 7f36d50da640 1 --2- 172.21.15.16:0/2252446372 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f36d00cb270 0x7f36d01f3d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:54.295 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:54.291+0000 7f36d50da640 1 --2- 172.21.15.16:0/2252446372 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f36d0069f10 0x7f36d0150930 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:54.295 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:54.291+0000 7f36d50da640 1 -- 172.21.15.16:0/2252446372 >> 172.21.15.16:0/2252446372 conn(0x7f36d00cc9f0 msgr2=0x7f36d00ce470 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:05:54.295 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:54.291+0000 7f36d50da640 1 -- 172.21.15.16:0/2252446372 shutdown_connections 2023-05-22T16:05:54.295 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:54.291+0000 7f36d50da640 1 -- 172.21.15.16:0/2252446372 wait complete. 2023-05-22T16:05:54.304 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T16:05:54.299+0000 7fed27eaa180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-05-22T16:05:54.309 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi016.front.sepia.ceph.com... 2023-05-22T16:05:54.309 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T16:05:54.310 DEBUG:teuthology.orchestra.run.smithi016:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2023-05-22T16:05:54.318 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi016.stderr:ceph-fuse[70744]: fuse finished with error 0 and tester_r 0 2023-05-22T16:05:54.319 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi016.front.sepia.ceph.com 2023-05-22T16:05:54.319 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T16:05:54.320 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-05-22T16:05:54.362 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T16:05:54.359+0000 7f69a9b59180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-05-22T16:05:54.365 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2023-05-22T16:05:54.366 DEBUG:teuthology.orchestra.run.smithi062:> dd if=/proc/self/mounts of=/dev/stdout 2023-05-22T16:05:54.371 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16342116k,nr_inodes=4085529,mode=755,inode64 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275932k,mode=755,inode64 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev,inode64 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=13250 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275928k,nr_inodes=818982,mode=700,uid=1000,gid=1271,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275932k,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-05-22T16:05:54.371 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi062.front.sepia.ceph.com 2023-05-22T16:05:54.372 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T16:05:54.372 DEBUG:teuthology.orchestra.run.smithi062:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2023-05-22T16:05:54.418 INFO:teuthology.orchestra.run.smithi062.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2023-05-22T16:05:54.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-05-22T16:05:54.420 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-05-22T16:05:54.420 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2023-05-22T16:05:54.423 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T16:05:54.419+0000 7fed27eaa180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-05-22T16:05:54.425 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16345192k,nr_inodes=4086298,mode=755,inode64 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3276544k,mode=755,inode64 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev,inode64 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=15732 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3276540k,nr_inodes=819135,mode=700,uid=1000,gid=1271,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3276544k,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-05-22T16:05:54.426 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi016.front.sepia.ceph.com 2023-05-22T16:05:54.426 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T16:05:54.426 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2023-05-22T16:05:54.427 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T16:05:54.423+0000 7f69a9b59180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-05-22T16:05:54.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-05-22T16:05:54.474 INFO:teuthology.orchestra.run.smithi016.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2023-05-22T16:05:54.475 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2023-05-22T16:05:54.476 DEBUG:teuthology.orchestra.run.smithi062:> dd if=/proc/self/mounts of=/dev/stdout 2023-05-22T16:05:54.481 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16342116k,nr_inodes=4085529,mode=755,inode64 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275932k,mode=755,inode64 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev,inode64 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=13250 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275928k,nr_inodes=818982,mode=700,uid=1000,gid=1271,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275932k,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-05-22T16:05:54.482 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi062.front.sepia.ceph.com 2023-05-22T16:05:54.482 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T16:05:54.482 DEBUG:teuthology.orchestra.run.smithi062:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2023-05-22T16:05:54.487 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T16:05:54.483+0000 7fed27eaa180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-05-22T16:05:54.527 INFO:teuthology.orchestra.run.smithi062.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2023-05-22T16:05:54.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-05-22T16:05:54.528 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-05-22T16:05:54.555 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T16:05:54.551+0000 7f69a9b59180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-05-22T16:05:54.616 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T16:05:54.611+0000 7fed27eaa180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-05-22T16:05:54.640 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:54.635+0000 7fd339609640 1 Processor -- start 2023-05-22T16:05:54.640 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:54.635+0000 7fd339609640 1 -- start start 2023-05-22T16:05:54.641 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:54.639+0000 7fd339609640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fd334070700 0x7fd334166c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:54.641 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:54.639+0000 7fd339609640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3340ce860 con 0x7fd33406add0 2023-05-22T16:05:54.641 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:54.639+0000 7fd339609640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd33406c050 con 0x7fd33415bd30 2023-05-22T16:05:54.642 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:54.639+0000 7fd339609640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd33406df40 con 0x7fd334070700 2023-05-22T16:05:54.642 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:54.639+0000 7fd3327fc640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fd33406add0 0x7fd33406fff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:45762/0 (socket says 172.21.15.16:45762) 2023-05-22T16:05:54.642 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:54.639+0000 7fd3327fc640 1 -- 172.21.15.16:0/933870503 learned_addr learned my addr 172.21.15.16:0/933870503 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T16:05:54.642 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:54.639+0000 7fd3337fe640 1 --2- 172.21.15.16:0/933870503 >> v2:172.21.15.62:3300/0 conn(0x7fd334070700 0x7fd334166c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:54.643 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:54.639+0000 7fd331ffb640 1 -- 172.21.15.16:0/933870503 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2165804198 0 0) 0x7fd3340ce860 con 0x7fd33406add0 2023-05-22T16:05:54.643 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:54.639+0000 7fd331ffb640 1 -- 172.21.15.16:0/933870503 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd31c003530 con 0x7fd33406add0 2023-05-22T16:05:54.643 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:54.639+0000 7fd3337fe640 1 -- 172.21.15.16:0/933870503 >> v1:172.21.15.62:6789/0 conn(0x7fd33415bd30 legacy=0x7fd33406d730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:54.643 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:54.639+0000 7fd3337fe640 1 -- 172.21.15.16:0/933870503 >> v1:172.21.15.16:6789/0 conn(0x7fd33406add0 legacy=0x7fd33406fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:54.643 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:54.639+0000 7fd3337fe640 1 -- 172.21.15.16:0/933870503 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3340cdd20 con 0x7fd334070700 2023-05-22T16:05:54.644 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:54.639+0000 7fd3337fe640 1 --2- 172.21.15.16:0/933870503 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd334070700 0x7fd334166c50 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7fd32c0097c0 tx=0x7fd32c01c330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=26b183ca05c12161 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:54.644 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:54.639+0000 7fd331ffb640 1 -- 172.21.15.16:0/933870503 <== mon.1 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd32c0ab600 con 0x7fd334070700 2023-05-22T16:05:54.644 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:54.639+0000 7fd331ffb640 1 -- 172.21.15.16:0/933870503 <== mon.1 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd32c0a7050 con 0x7fd334070700 2023-05-22T16:05:54.645 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:54.639+0000 7fd331ffb640 1 -- 172.21.15.16:0/933870503 <== mon.1 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd32c0ab960 con 0x7fd334070700 2023-05-22T16:05:54.645 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:54.643+0000 7fd339609640 1 -- 172.21.15.16:0/933870503 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd334070700 msgr2=0x7fd334166c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:54.645 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:54.643+0000 7fd339609640 1 --2- 172.21.15.16:0/933870503 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd334070700 0x7fd334166c50 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7fd32c0097c0 tx=0x7fd32c01c330 comp rx=0 tx=0).stop 2023-05-22T16:05:54.646 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:54.643+0000 7fd339609640 1 -- 172.21.15.16:0/933870503 shutdown_connections 2023-05-22T16:05:54.646 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:54.643+0000 7fd339609640 1 --2- 172.21.15.16:0/933870503 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd334070700 0x7fd334166c50 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:54.646 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:54.643+0000 7fd339609640 1 -- 172.21.15.16:0/933870503 >> 172.21.15.16:0/933870503 conn(0x7fd3340cea90 msgr2=0x7fd3340cee90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:05:54.646 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:54.643+0000 7fd339609640 1 -- 172.21.15.16:0/933870503 shutdown_connections 2023-05-22T16:05:54.646 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:54.643+0000 7fd339609640 1 -- 172.21.15.16:0/933870503 wait complete. 2023-05-22T16:05:54.647 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:54.643+0000 7fd339609640 1 Processor -- start 2023-05-22T16:05:54.648 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:54.643+0000 7fd339609640 1 -- start start 2023-05-22T16:05:54.648 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:54.643+0000 7fd339609640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd33406add0 0x7fd3341518d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:54.648 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:54.643+0000 7fd339609640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd334070700 0x7fd33415af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:54.648 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:54.643+0000 7fd339609640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd33415bd30 0x7fd3341549a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:54.648 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:54.643+0000 7fd339609640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3340c3b90 con 0x7fd33415bd30 2023-05-22T16:05:54.648 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:54.643+0000 7fd339609640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3340b4aa0 con 0x7fd33406add0 2023-05-22T16:05:54.648 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:54.643+0000 7fd339609640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3340c3700 con 0x7fd334070700 2023-05-22T16:05:54.649 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:54.643+0000 7fd3337fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd33415bd30 0x7fd3341549a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:54.649 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:54.643+0000 7fd3337fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd33415bd30 0x7fd3341549a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38456/0 (socket says 172.21.15.16:38456) 2023-05-22T16:05:54.649 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:54.643+0000 7fd3337fe640 1 -- 172.21.15.16:0/546861501 learned_addr learned my addr 172.21.15.16:0/546861501 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:05:54.649 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:54.643+0000 7fd332ffd640 1 --2- 172.21.15.16:0/546861501 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd33406add0 0x7fd3341518d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:54.649 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:54.643+0000 7fd3327fc640 1 --2- 172.21.15.16:0/546861501 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd334070700 0x7fd33415af40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:54.649 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:54.643+0000 7fd3337fe640 1 -- 172.21.15.16:0/546861501 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd334070700 msgr2=0x7fd33415af40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:54.649 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:54.643+0000 7fd3337fe640 1 --2- 172.21.15.16:0/546861501 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd334070700 0x7fd33415af40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:54.650 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:54.643+0000 7fd3337fe640 1 -- 172.21.15.16:0/546861501 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd33406add0 msgr2=0x7fd3341518d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:54.650 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:54.643+0000 7fd3337fe640 1 --2- 172.21.15.16:0/546861501 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd33406add0 0x7fd3341518d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:54.650 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:54.643+0000 7fd3337fe640 1 -- 172.21.15.16:0/546861501 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd32c009420 con 0x7fd33415bd30 2023-05-22T16:05:54.650 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:54.643+0000 7fd3337fe640 1 --2- 172.21.15.16:0/546861501 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd33415bd30 0x7fd3341549a0 secure :-1 s=READY pgs=2557 cs=0 l=1 rev1=1 crypto rx=0x7fd32c0097c0 tx=0x7fd32c0190d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:54.651 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:54.643+0000 7fd31b7fe640 1 -- 172.21.15.16:0/546861501 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd32c004030 con 0x7fd33415bd30 2023-05-22T16:05:54.651 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:54.643+0000 7fd31b7fe640 1 -- 172.21.15.16:0/546861501 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd32c004190 con 0x7fd33415bd30 2023-05-22T16:05:54.651 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:54.643+0000 7fd339609640 1 -- 172.21.15.16:0/546861501 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd33415c100 con 0x7fd33415bd30 2023-05-22T16:05:54.651 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:54.643+0000 7fd31b7fe640 1 -- 172.21.15.16:0/546861501 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd32c004500 con 0x7fd33415bd30 2023-05-22T16:05:54.652 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:54.643+0000 7fd339609640 1 -- 172.21.15.16:0/546861501 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3340ce580 con 0x7fd33415bd30 2023-05-22T16:05:54.652 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:54.647+0000 7fd339609640 1 -- 172.21.15.16:0/546861501 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd33406aaf0 con 0x7fd33415bd30 2023-05-22T16:05:54.652 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:54.647+0000 7fd31b7fe640 1 -- 172.21.15.16:0/546861501 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 97) v1 ==== 84389+0+0 (secure 0 0 0) 0x7fd32c0b2040 con 0x7fd33415bd30 2023-05-22T16:05:54.652 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:54.647+0000 7fd31b7fe640 1 --2- 172.21.15.16:0/546861501 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fd320065530 0x7fd3200679e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:54.652 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:54.647+0000 7fd332ffd640 1 -- 172.21.15.16:0/546861501 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fd320065530 msgr2=0x7fd3200679e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/65890 2023-05-22T16:05:54.652 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:54.647+0000 7fd31b7fe640 1 -- 172.21.15.16:0/546861501 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(197..197 src has 1..197) v4 ==== 6782+0+0 (secure 0 0 0) 0x7fd32c11d700 con 0x7fd33415bd30 2023-05-22T16:05:54.652 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:54.647+0000 7fd332ffd640 1 --2- 172.21.15.16:0/546861501 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fd320065530 0x7fd3200679e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-05-22T16:05:54.653 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:54.651+0000 7fd31b7fe640 1 -- 172.21.15.16:0/546861501 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fd32c0eaca0 con 0x7fd33415bd30 2023-05-22T16:05:54.825 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-05-22T16:05:54.828 DEBUG:teuthology.orchestra.run.smithi062:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-05-22T16:05:54.836 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:54.831+0000 7fd339609640 1 -- 172.21.15.16:0/546861501 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd334073050 con 0x7fd33415bd30 2023-05-22T16:05:54.837 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:54.835+0000 7fd31b7fe640 1 -- 172.21.15.16:0/546861501 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 81 v81) v1 ==== 94+0+4653 (secure 0 0 0) 0x7fd32c0b0090 con 0x7fd33415bd30 2023-05-22T16:05:54.837 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T16:05:54.837 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":81,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7979,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/1053276484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":1053276484},{"type":"v1","addr":"172.21.15.62:6835","nonce":1053276484}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":76},{"gid":8983,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6833/1372552690","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":1372552690},{"type":"v1","addr":"172.21.15.62:6833","nonce":1372552690}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":76},{"gid":10776,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/982492476","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":982492476},{"type":"v1","addr":"172.21.15.16:6835","nonce":982492476}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":76}],"filesystems":[{"mdsmap":{"epoch":81,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-05-22T16:05:21.922031+0000","modified":"2023-05-22T16:05:24.940696+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10782},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10782":{"gid":10782,"name":"c","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.16:6837/1705367575","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":1705367575},{"type":"v1","addr":"172.21.15.16:6837","nonce":1705367575}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[25],"metadata_pool":24,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":11}]} 2023-05-22T16:05:54.838 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 81 2023-05-22T16:05:54.839 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:54.835+0000 7fd339609640 1 -- 172.21.15.16:0/546861501 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fd320065530 msgr2=0x7fd3200679e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-05-22T16:05:54.839 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:54.835+0000 7fd339609640 1 --2- 172.21.15.16:0/546861501 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fd320065530 0x7fd3200679e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:54.839 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:54.835+0000 7fd339609640 1 -- 172.21.15.16:0/546861501 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd33415bd30 msgr2=0x7fd3341549a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:54.840 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:54.835+0000 7fd339609640 1 --2- 172.21.15.16:0/546861501 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd33415bd30 0x7fd3341549a0 secure :-1 s=READY pgs=2557 cs=0 l=1 rev1=1 crypto rx=0x7fd32c0097c0 tx=0x7fd32c0190d0 comp rx=0 tx=0).stop 2023-05-22T16:05:54.840 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:54.835+0000 7fd339609640 1 -- 172.21.15.16:0/546861501 shutdown_connections 2023-05-22T16:05:54.840 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:54.835+0000 7fd339609640 1 --2- 172.21.15.16:0/546861501 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fd320065530 0x7fd3200679e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:54.840 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:54.835+0000 7fd339609640 1 --2- 172.21.15.16:0/546861501 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd33415bd30 0x7fd3341549a0 unknown :-1 s=CLOSED pgs=2557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:54.840 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:54.835+0000 7fd339609640 1 --2- 172.21.15.16:0/546861501 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd334070700 0x7fd33415af40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:54.840 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:54.835+0000 7fd339609640 1 --2- 172.21.15.16:0/546861501 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd33406add0 0x7fd3341518d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:54.841 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:54.835+0000 7fd339609640 1 -- 172.21.15.16:0/546861501 >> 172.21.15.16:0/546861501 conn(0x7fd3340cea90 msgr2=0x7fd3340ae200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:05:54.841 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:54.835+0000 7fd339609640 1 -- 172.21.15.16:0/546861501 shutdown_connections 2023-05-22T16:05:54.841 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:54.835+0000 7fd339609640 1 -- 172.21.15.16:0/546861501 wait complete. 2023-05-22T16:05:54.855 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-05-22T16:05:55.027 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.023+0000 7f41ba596640 1 Processor -- start 2023-05-22T16:05:55.028 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.023+0000 7f41ba596640 1 -- start start 2023-05-22T16:05:55.029 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.027+0000 7f41ba596640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f41b40ce510 0x7f41b40ce940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:55.029 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.027+0000 7f41ba596640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41b406a2a0 con 0x7f41b406ed50 2023-05-22T16:05:55.029 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.027+0000 7f41ba596640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41b4160620 con 0x7f41b4070860 2023-05-22T16:05:55.029 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.027+0000 7f41ba596640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41b4160ab0 con 0x7f41b40ce510 2023-05-22T16:05:55.029 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.027+0000 7f41b37fe640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f41b40ce510 0x7f41b40ce940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:55.030 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.027+0000 7f41b3fff640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f41b406ed50 0x7f41b4069800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:45766/0 (socket says 172.21.15.16:45766) 2023-05-22T16:05:55.030 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.027+0000 7f41b3fff640 1 -- 172.21.15.16:0/1325252279 learned_addr learned my addr 172.21.15.16:0/1325252279 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T16:05:55.030 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.027+0000 7f41b37fe640 1 -- 172.21.15.16:0/1325252279 >> v1:172.21.15.62:6789/0 conn(0x7f41b4070860 legacy=0x7f41b4070cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:05:55.030 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.027+0000 7f41b37fe640 1 -- 172.21.15.16:0/1325252279 >> v1:172.21.15.16:6789/0 conn(0x7f41b406ed50 legacy=0x7f41b4069800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:55.030 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.027+0000 7f41b2ffd640 1 -- 172.21.15.16:0/1325252279 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2549955437 0 0) 0x7f41b406a2a0 con 0x7f41b406ed50 2023-05-22T16:05:55.031 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.027+0000 7f41b37fe640 1 -- 172.21.15.16:0/1325252279 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41b415fcf0 con 0x7f41b40ce510 2023-05-22T16:05:55.031 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.027+0000 7f41b37fe640 1 --2- 172.21.15.16:0/1325252279 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f41b40ce510 0x7f41b40ce940 secure :-1 s=READY pgs=2559 cs=0 l=1 rev1=1 crypto rx=0x7f419c0097c0 tx=0x7f419c01c3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37fe13fab3abf44a server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:55.032 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.027+0000 7f41b2ffd640 1 -- 172.21.15.16:0/1325252279 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f419c0ab600 con 0x7f41b40ce510 2023-05-22T16:05:55.032 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.027+0000 7f41b2ffd640 1 -- 172.21.15.16:0/1325252279 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f419c0a7050 con 0x7f41b40ce510 2023-05-22T16:05:55.032 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.027+0000 7f41b2ffd640 1 -- 172.21.15.16:0/1325252279 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f419c0ab980 con 0x7f41b40ce510 2023-05-22T16:05:55.032 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.027+0000 7f41ba596640 1 -- 172.21.15.16:0/1325252279 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f41b40ce510 msgr2=0x7f41b40ce940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:55.032 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.027+0000 7f41ba596640 1 --2- 172.21.15.16:0/1325252279 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f41b40ce510 0x7f41b40ce940 secure :-1 s=READY pgs=2559 cs=0 l=1 rev1=1 crypto rx=0x7f419c0097c0 tx=0x7f419c01c3b0 comp rx=0 tx=0).stop 2023-05-22T16:05:55.032 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.027+0000 7f41ba596640 1 -- 172.21.15.16:0/1325252279 shutdown_connections 2023-05-22T16:05:55.033 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.027+0000 7f41ba596640 1 --2- 172.21.15.16:0/1325252279 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f41b40ce510 0x7f41b40ce940 unknown :-1 s=CLOSED pgs=2559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:55.033 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.027+0000 7f41ba596640 1 -- 172.21.15.16:0/1325252279 >> 172.21.15.16:0/1325252279 conn(0x7f41b40cc9f0 msgr2=0x7f41b40cf2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:05:55.033 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.027+0000 7f41ba596640 1 -- 172.21.15.16:0/1325252279 shutdown_connections 2023-05-22T16:05:55.033 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.027+0000 7f41ba596640 1 -- 172.21.15.16:0/1325252279 wait complete. 2023-05-22T16:05:55.035 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.031+0000 7f41ba596640 1 Processor -- start 2023-05-22T16:05:55.035 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.031+0000 7f41ba596640 1 -- start start 2023-05-22T16:05:55.035 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.031+0000 7f41ba596640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f41b406ed50 0x7f41b414e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:55.035 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.031+0000 7f41ba596640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f41b4070860 0x7f41b4159300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:55.035 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.031+0000 7f41ba596640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f41b40ce510 0x7f41b4159840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:55.035 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.031+0000 7f41ba596640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41b40ab8f0 con 0x7f41b40ce510 2023-05-22T16:05:55.035 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.031+0000 7f41ba596640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41b414f030 con 0x7f41b4070860 2023-05-22T16:05:55.036 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.031+0000 7f41ba596640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41b414eb00 con 0x7f41b406ed50 2023-05-22T16:05:55.036 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.031+0000 7f41b8b33640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f41b40ce510 0x7f41b4159840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:55.036 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.031+0000 7f41b8b33640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f41b40ce510 0x7f41b4159840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38478/0 (socket says 172.21.15.16:38478) 2023-05-22T16:05:55.036 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.031+0000 7f41b8b33640 1 -- 172.21.15.16:0/3666707738 learned_addr learned my addr 172.21.15.16:0/3666707738 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:05:55.036 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.031+0000 7f41b37fe640 1 --2- 172.21.15.16:0/3666707738 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f41b4070860 0x7f41b4159300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:55.036 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.031+0000 7f41b3fff640 1 --2- 172.21.15.16:0/3666707738 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f41b406ed50 0x7f41b414e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:55.037 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.031+0000 7f41b8b33640 1 -- 172.21.15.16:0/3666707738 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f41b406ed50 msgr2=0x7f41b414e5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:55.037 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.031+0000 7f41b8b33640 1 --2- 172.21.15.16:0/3666707738 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f41b406ed50 0x7f41b414e5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:55.037 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.031+0000 7f41b8b33640 1 -- 172.21.15.16:0/3666707738 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f41b4070860 msgr2=0x7f41b4159300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:55.037 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.031+0000 7f41b8b33640 1 --2- 172.21.15.16:0/3666707738 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f41b4070860 0x7f41b4159300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:55.037 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.031+0000 7f41b8b33640 1 -- 172.21.15.16:0/3666707738 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f419c009420 con 0x7f41b40ce510 2023-05-22T16:05:55.038 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.031+0000 7f41b8b33640 1 --2- 172.21.15.16:0/3666707738 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f41b40ce510 0x7f41b4159840 secure :-1 s=READY pgs=2560 cs=0 l=1 rev1=1 crypto rx=0x7f41a400c7b0 tx=0x7f41a400cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:55.038 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.031+0000 7f41b0ff9640 1 -- 172.21.15.16:0/3666707738 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41a4004b70 con 0x7f41b40ce510 2023-05-22T16:05:55.038 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.031+0000 7f41b0ff9640 1 -- 172.21.15.16:0/3666707738 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41a4004cd0 con 0x7f41b40ce510 2023-05-22T16:05:55.038 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.031+0000 7f41ba596640 1 -- 172.21.15.16:0/3666707738 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41b4069f10 con 0x7f41b40ce510 2023-05-22T16:05:55.038 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.031+0000 7f41b0ff9640 1 -- 172.21.15.16:0/3666707738 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41a400aa90 con 0x7f41b40ce510 2023-05-22T16:05:55.038 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.031+0000 7f41ba596640 1 -- 172.21.15.16:0/3666707738 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41b40cee80 con 0x7f41b40ce510 2023-05-22T16:05:55.039 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.031+0000 7f41ba596640 1 -- 172.21.15.16:0/3666707738 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41b40717e0 con 0x7f41b40ce510 2023-05-22T16:05:55.039 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.031+0000 7f41b0ff9640 1 -- 172.21.15.16:0/3666707738 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 97) v1 ==== 84389+0+0 (secure 0 0 0) 0x7f41a400abf0 con 0x7f41b40ce510 2023-05-22T16:05:55.040 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.035+0000 7f41b0ff9640 1 --2- 172.21.15.16:0/3666707738 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f4194065530 0x7f41940679e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:55.040 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.035+0000 7f41b3fff640 1 -- 172.21.15.16:0/3666707738 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f4194065530 msgr2=0x7f41940679e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/65890 2023-05-22T16:05:55.040 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.035+0000 7f41b0ff9640 1 -- 172.21.15.16:0/3666707738 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(197..197 src has 1..197) v4 ==== 6782+0+0 (secure 0 0 0) 0x7f41a4082f40 con 0x7f41b40ce510 2023-05-22T16:05:55.040 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.035+0000 7f41b3fff640 1 --2- 172.21.15.16:0/3666707738 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f4194065530 0x7f41940679e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-05-22T16:05:55.041 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.039+0000 7f41b0ff9640 1 -- 172.21.15.16:0/3666707738 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f41a4050700 con 0x7f41b40ce510 2023-05-22T16:05:55.054 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T16:05:55.051+0000 7f69a9b59180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-05-22T16:05:55.128 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T16:05:55.123+0000 7fed27eaa180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-05-22T16:05:55.131 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T16:05:55.127+0000 7f69a9b59180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-05-22T16:05:55.193 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.191+0000 7f41ba596640 1 -- 172.21.15.16:0/3666707738 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f41b4073050 con 0x7f41b40ce510 2023-05-22T16:05:55.196 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.191+0000 7f41b0ff9640 1 -- 172.21.15.16:0/3666707738 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v197) v1 ==== 74+0+18629 (secure 0 0 0) 0x7f41a4053d20 con 0x7f41b40ce510 2023-05-22T16:05:55.197 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T16:05:55.197 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":197,"fsid":"b2719834-7847-4d98-90a6-f65d57f85c49","created":"2023-05-22T15:26:25.259314+0000","modified":"2023-05-22T16:05:35.647639+0000","last_up_change":"2023-05-22T15:26:36.448524+0000","last_in_change":"2023-05-22T15:26:27.675668+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":25,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-05-22T15:26:38.019781+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-05-22T15:26:43.570514+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_metadata","create_time":"2023-05-22T16:05:20.485167+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"195","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":25,"pool_name":"cephfs_data","create_time":"2023-05-22T16:05:20.900001+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"196","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9e7ff854-b25f-4cbb-a5c2-1799d9c53d2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":37021},{"type":"v1","addr":"172.21.15.16:6825","nonce":37021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":37021},{"type":"v1","addr":"172.21.15.16:6827","nonce":37021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":37021},{"type":"v1","addr":"172.21.15.16:6831","nonce":37021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":37021},{"type":"v1","addr":"172.21.15.16:6829","nonce":37021}]},"public_addr":"172.21.15.16:6825/37021","cluster_addr":"172.21.15.16:6827/37021","heartbeat_back_addr":"172.21.15.16:6831/37021","heartbeat_front_addr":"172.21.15.16:6829/37021","state":["exists","up"]},{"osd":1,"uuid":"4702fbc9-a927-4565-aca5-41dc69cc8264","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":37008},{"type":"v1","addr":"172.21.15.16:6801","nonce":37008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":37008},{"type":"v1","addr":"172.21.15.16:6803","nonce":37008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":37008},{"type":"v1","addr":"172.21.15.16:6807","nonce":37008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":37008},{"type":"v1","addr":"172.21.15.16:6805","nonce":37008}]},"public_addr":"172.21.15.16:6801/37008","cluster_addr":"172.21.15.16:6803/37008","heartbeat_back_addr":"172.21.15.16:6807/37008","heartbeat_front_addr":"172.21.15.16:6805/37008","state":["exists","up"]},{"osd":2,"uuid":"1e7bba46-4eba-47ef-bb81-5e2d371638ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":37009},{"type":"v1","addr":"172.21.15.16:6809","nonce":37009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":37009},{"type":"v1","addr":"172.21.15.16:6811","nonce":37009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":37009},{"type":"v1","addr":"172.21.15.16:6815","nonce":37009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":37009},{"type":"v1","addr":"172.21.15.16:6813","nonce":37009}]},"public_addr":"172.21.15.16:6809/37009","cluster_addr":"172.21.15.16:6811/37009","heartbeat_back_addr":"172.21.15.16:6815/37009","heartbeat_front_addr":"172.21.15.16:6813/37009","state":["exists","up"]},{"osd":3,"uuid":"383cacc0-5a88-4904-990e-498ca59ef48a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":37020},{"type":"v1","addr":"172.21.15.16:6817","nonce":37020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":37020},{"type":"v1","addr":"172.21.15.16:6819","nonce":37020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":37020},{"type":"v1","addr":"172.21.15.16:6823","nonce":37020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":37020},{"type":"v1","addr":"172.21.15.16:6821","nonce":37020}]},"public_addr":"172.21.15.16:6817/37020","cluster_addr":"172.21.15.16:6819/37020","heartbeat_back_addr":"172.21.15.16:6823/37020","heartbeat_front_addr":"172.21.15.16:6821/37020","state":["exists","up"]},{"osd":4,"uuid":"970500ef-95b7-42c1-b080-eed4fae7013d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":37353},{"type":"v1","addr":"172.21.15.62:6817","nonce":37353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6819","nonce":37353},{"type":"v1","addr":"172.21.15.62:6822","nonce":37353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37353},{"type":"v1","addr":"172.21.15.62:6831","nonce":37353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37353},{"type":"v1","addr":"172.21.15.62:6826","nonce":37353}]},"public_addr":"172.21.15.62:6817/37353","cluster_addr":"172.21.15.62:6822/37353","heartbeat_back_addr":"172.21.15.62:6831/37353","heartbeat_front_addr":"172.21.15.62:6826/37353","state":["exists","up"]},{"osd":5,"uuid":"022d67db-d4d8-445b-b1b9-8b446a00f5c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":37352},{"type":"v1","addr":"172.21.15.62:6801","nonce":37352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":37352},{"type":"v1","addr":"172.21.15.62:6803","nonce":37352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":37352},{"type":"v1","addr":"172.21.15.62:6807","nonce":37352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":37352},{"type":"v1","addr":"172.21.15.62:6805","nonce":37352}]},"public_addr":"172.21.15.62:6801/37352","cluster_addr":"172.21.15.62:6803/37352","heartbeat_back_addr":"172.21.15.62:6807/37352","heartbeat_front_addr":"172.21.15.62:6805/37352","state":["exists","up"]},{"osd":6,"uuid":"42bd372b-6688-4a1b-8407-e16e8c266e33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":37354},{"type":"v1","addr":"172.21.15.62:6820","nonce":37354}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":37354},{"type":"v1","addr":"172.21.15.62:6823","nonce":37354}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6829","nonce":37354},{"type":"v1","addr":"172.21.15.62:6830","nonce":37354}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6825","nonce":37354},{"type":"v1","addr":"172.21.15.62:6827","nonce":37354}]},"public_addr":"172.21.15.62:6820/37354","cluster_addr":"172.21.15.62:6823/37354","heartbeat_back_addr":"172.21.15.62:6830/37354","heartbeat_front_addr":"172.21.15.62:6827/37354","state":["exists","up"]},{"osd":7,"uuid":"ac70b2a3-f41b-4bf2-b227-7f2cbb55426d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37355},{"type":"v1","addr":"172.21.15.62:6809","nonce":37355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37355},{"type":"v1","addr":"172.21.15.62:6811","nonce":37355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":37355},{"type":"v1","addr":"172.21.15.62:6815","nonce":37355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":37355},{"type":"v1","addr":"172.21.15.62:6813","nonce":37355}]},"public_addr":"172.21.15.62:6809/37355","cluster_addr":"172.21.15.62:6811/37355","heartbeat_back_addr":"172.21.15.62:6815/37355","heartbeat_front_addr":"172.21.15.62:6813/37355","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.368896+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.107273+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.276888+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.287502+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.124791+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.251937+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.255679+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.232797+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:0/1961044213":"2023-05-23T16:05:35.647562+0000","172.21.15.16:0/238292614":"2023-05-23T16:05:35.647562+0000","172.21.15.16:6833/65890":"2023-05-23T16:05:35.647562+0000","172.21.15.16:0/1148602276":"2023-05-23T16:05:35.647562+0000","172.21.15.16:6832/65890":"2023-05-23T16:05:35.647562+0000","172.21.15.16:0/456460873":"2023-05-23T16:05:35.647562+0000","172.21.15.16:0/2766506027":"2023-05-23T16:05:35.647562+0000","172.21.15.16:0/2214029832":"2023-05-23T16:05:35.647562+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":17,"snaps":[{"begin":2,"length":2}]},{"pool":21,"snaps":[{"begin":25,"length":9}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-05-22T16:05:55.198 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.195+0000 7f41ba596640 1 -- 172.21.15.16:0/3666707738 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f4194065530 msgr2=0x7f41940679e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-05-22T16:05:55.198 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.195+0000 7f41ba596640 1 --2- 172.21.15.16:0/3666707738 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f4194065530 0x7f41940679e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:55.199 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.195+0000 7f41ba596640 1 -- 172.21.15.16:0/3666707738 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f41b40ce510 msgr2=0x7f41b4159840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:55.199 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.195+0000 7f41ba596640 1 --2- 172.21.15.16:0/3666707738 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f41b40ce510 0x7f41b4159840 secure :-1 s=READY pgs=2560 cs=0 l=1 rev1=1 crypto rx=0x7f41a400c7b0 tx=0x7f41a400cc70 comp rx=0 tx=0).stop 2023-05-22T16:05:55.199 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.195+0000 7f41ba596640 1 -- 172.21.15.16:0/3666707738 shutdown_connections 2023-05-22T16:05:55.199 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.195+0000 7f41ba596640 1 --2- 172.21.15.16:0/3666707738 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f4194065530 0x7f41940679e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:55.199 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.195+0000 7f41ba596640 1 --2- 172.21.15.16:0/3666707738 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f41b40ce510 0x7f41b4159840 unknown :-1 s=CLOSED pgs=2560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:55.199 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.195+0000 7f41ba596640 1 --2- 172.21.15.16:0/3666707738 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f41b4070860 0x7f41b4159300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:55.199 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.195+0000 7f41ba596640 1 --2- 172.21.15.16:0/3666707738 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f41b406ed50 0x7f41b414e5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:55.200 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.195+0000 7f41ba596640 1 -- 172.21.15.16:0/3666707738 >> 172.21.15.16:0/3666707738 conn(0x7f41b40cc9f0 msgr2=0x7f41b4069800 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:05:55.200 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.195+0000 7f41ba596640 1 -- 172.21.15.16:0/3666707738 shutdown_connections 2023-05-22T16:05:55.200 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.195+0000 7f41ba596640 1 -- 172.21.15.16:0/3666707738 wait complete. 2023-05-22T16:05:55.205 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T16:05:55.203+0000 7fed27eaa180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-05-22T16:05:55.215 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-05-22T15:26:38.019781+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-05-22T15:26:43.570514+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 24, 'pool_name': 'cephfs_metadata', 'create_time': '2023-05-22T16:05:20.485167+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '195', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 25, 'pool_name': 'cephfs_data', 'create_time': '2023-05-22T16:05:20.900001+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '196', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-05-22T16:05:55.217 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-05-22T16:05:55.272 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T16:05:55.267+0000 7f69a9b59180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-05-22T16:05:55.337 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.335+0000 7f0d3325e640 1 Processor -- start 2023-05-22T16:05:55.337 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.335+0000 7f0d3325e640 1 -- start start 2023-05-22T16:05:55.338 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.335+0000 7f0d3325e640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f0d2c15ce80 0x7f0d2c070300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:55.338 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.335+0000 7f0d3325e640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d2c06d7d0 con 0x7f0d2c06fe80 2023-05-22T16:05:55.338 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.335+0000 7f0d3325e640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d2c069800 con 0x7f0d2c15baa0 2023-05-22T16:05:55.338 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.335+0000 7f0d3325e640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d2c070840 con 0x7f0d2c15ce80 2023-05-22T16:05:55.338 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.335+0000 7f0d317fb640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f0d2c15ce80 0x7f0d2c070300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:55.339 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.335+0000 7f0d2bfff640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f0d2c06fe80 0x7f0d2c15f770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:45774/0 (socket says 172.21.15.16:45774) 2023-05-22T16:05:55.339 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.335+0000 7f0d2bfff640 1 -- 172.21.15.16:0/1646045778 learned_addr learned my addr 172.21.15.16:0/1646045778 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T16:05:55.339 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.335+0000 7f0d317fb640 1 -- 172.21.15.16:0/1646045778 >> v1:172.21.15.62:6789/0 conn(0x7f0d2c15baa0 legacy=0x7f0d2c06cd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:55.340 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.335+0000 7f0d317fb640 1 -- 172.21.15.16:0/1646045778 >> v1:172.21.15.16:6789/0 conn(0x7f0d2c06fe80 legacy=0x7f0d2c15f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:55.340 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.335+0000 7f0d317fb640 1 -- 172.21.15.16:0/1646045778 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d2c069ae0 con 0x7f0d2c15ce80 2023-05-22T16:05:55.340 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.335+0000 7f0d317fb640 1 --2- 172.21.15.16:0/1646045778 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d2c15ce80 0x7f0d2c070300 secure :-1 s=READY pgs=2562 cs=0 l=1 rev1=1 crypto rx=0x7f0d24009d90 tx=0x7f0d2401c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9de699dea5c30f server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:55.341 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.339+0000 7f0d2b7fe640 1 -- 172.21.15.16:0/1646045778 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d240a59e0 con 0x7f0d2c15ce80 2023-05-22T16:05:55.341 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.339+0000 7f0d2b7fe640 1 -- 172.21.15.16:0/1646045778 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d240a5b40 con 0x7f0d2c15ce80 2023-05-22T16:05:55.341 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.339+0000 7f0d2b7fe640 1 -- 172.21.15.16:0/1646045778 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d240ad440 con 0x7f0d2c15ce80 2023-05-22T16:05:55.342 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.339+0000 7f0d3325e640 1 -- 172.21.15.16:0/1646045778 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d2c15ce80 msgr2=0x7f0d2c070300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:55.342 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.339+0000 7f0d3325e640 1 --2- 172.21.15.16:0/1646045778 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d2c15ce80 0x7f0d2c070300 secure :-1 s=READY pgs=2562 cs=0 l=1 rev1=1 crypto rx=0x7f0d24009d90 tx=0x7f0d2401c920 comp rx=0 tx=0).stop 2023-05-22T16:05:55.342 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.339+0000 7f0d3325e640 1 -- 172.21.15.16:0/1646045778 shutdown_connections 2023-05-22T16:05:55.342 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.339+0000 7f0d3325e640 1 --2- 172.21.15.16:0/1646045778 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d2c15ce80 0x7f0d2c070300 unknown :-1 s=CLOSED pgs=2562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:55.342 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.339+0000 7f0d3325e640 1 -- 172.21.15.16:0/1646045778 >> 172.21.15.16:0/1646045778 conn(0x7f0d2c0ce930 msgr2=0x7f0d2c0ced30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:05:55.342 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.339+0000 7f0d3325e640 1 -- 172.21.15.16:0/1646045778 shutdown_connections 2023-05-22T16:05:55.343 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.339+0000 7f0d3325e640 1 -- 172.21.15.16:0/1646045778 wait complete. 2023-05-22T16:05:55.345 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.339+0000 7f0d3325e640 1 Processor -- start 2023-05-22T16:05:55.345 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.339+0000 7f0d3325e640 1 -- start start 2023-05-22T16:05:55.345 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.339+0000 7f0d3325e640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0d2c06fe80 0x7f0d2c150300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:55.345 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.339+0000 7f0d3325e640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0d2c15baa0 0x7f0d2c159db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:55.346 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.339+0000 7f0d3325e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d2c15ce80 0x7f0d2c0b2f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:55.346 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.339+0000 7f0d3325e640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d2c1589c0 con 0x7f0d2c15ce80 2023-05-22T16:05:55.346 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.339+0000 7f0d3325e640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d2c158c60 con 0x7f0d2c15baa0 2023-05-22T16:05:55.346 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.339+0000 7f0d3325e640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d2c158f40 con 0x7f0d2c06fe80 2023-05-22T16:05:55.346 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.339+0000 7f0d317fb640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d2c15ce80 0x7f0d2c0b2f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:55.346 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.339+0000 7f0d317fb640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d2c15ce80 0x7f0d2c0b2f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38496/0 (socket says 172.21.15.16:38496) 2023-05-22T16:05:55.346 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.339+0000 7f0d317fb640 1 -- 172.21.15.16:0/309724882 learned_addr learned my addr 172.21.15.16:0/309724882 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:05:55.347 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.339+0000 7f0d2bfff640 1 --2- 172.21.15.16:0/309724882 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0d2c15baa0 0x7f0d2c159db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:55.347 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.339+0000 7f0d30ffa640 1 --2- 172.21.15.16:0/309724882 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0d2c06fe80 0x7f0d2c150300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:55.347 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.339+0000 7f0d317fb640 1 -- 172.21.15.16:0/309724882 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0d2c06fe80 msgr2=0x7f0d2c150300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:55.347 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.339+0000 7f0d317fb640 1 --2- 172.21.15.16:0/309724882 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0d2c06fe80 0x7f0d2c150300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:55.347 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.339+0000 7f0d317fb640 1 -- 172.21.15.16:0/309724882 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0d2c15baa0 msgr2=0x7f0d2c159db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:55.347 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.339+0000 7f0d317fb640 1 --2- 172.21.15.16:0/309724882 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0d2c15baa0 0x7f0d2c159db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:55.347 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.339+0000 7f0d317fb640 1 -- 172.21.15.16:0/309724882 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d24009a70 con 0x7f0d2c15ce80 2023-05-22T16:05:55.348 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.339+0000 7f0d2bfff640 1 --2- 172.21.15.16:0/309724882 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0d2c15baa0 0x7f0d2c159db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T16:05:55.348 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.339+0000 7f0d30ffa640 1 --2- 172.21.15.16:0/309724882 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0d2c06fe80 0x7f0d2c150300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T16:05:55.348 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.339+0000 7f0d317fb640 1 --2- 172.21.15.16:0/309724882 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d2c15ce80 0x7f0d2c0b2f00 secure :-1 s=READY pgs=2563 cs=0 l=1 rev1=1 crypto rx=0x7f0d2401a550 tx=0x7f0d240196c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:55.348 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.339+0000 7f0d297fa640 1 -- 172.21.15.16:0/309724882 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d240addc0 con 0x7f0d2c15ce80 2023-05-22T16:05:55.348 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.339+0000 7f0d297fa640 1 -- 172.21.15.16:0/309724882 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d240b9450 con 0x7f0d2c15ce80 2023-05-22T16:05:55.348 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.339+0000 7f0d3325e640 1 -- 172.21.15.16:0/309724882 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d2c06d440 con 0x7f0d2c15ce80 2023-05-22T16:05:55.348 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.339+0000 7f0d297fa640 1 -- 172.21.15.16:0/309724882 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d240b97e0 con 0x7f0d2c15ce80 2023-05-22T16:05:55.348 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.339+0000 7f0d3325e640 1 -- 172.21.15.16:0/309724882 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d2c06a5c0 con 0x7f0d2c15ce80 2023-05-22T16:05:55.349 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.343+0000 7f0d3325e640 1 -- 172.21.15.16:0/309724882 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d2c0ce5d0 con 0x7f0d2c15ce80 2023-05-22T16:05:55.349 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.343+0000 7f0d297fa640 1 -- 172.21.15.16:0/309724882 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 97) v1 ==== 84389+0+0 (secure 0 0 0) 0x7f0d240b9940 con 0x7f0d2c15ce80 2023-05-22T16:05:55.350 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.343+0000 7f0d297fa640 1 --2- 172.21.15.16:0/309724882 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f0d18065530 0x7f0d180679e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:55.350 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.343+0000 7f0d30ffa640 1 -- 172.21.15.16:0/309724882 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f0d18065530 msgr2=0x7f0d180679e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/65890 2023-05-22T16:05:55.350 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.343+0000 7f0d297fa640 1 -- 172.21.15.16:0/309724882 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(197..197 src has 1..197) v4 ==== 6782+0+0 (secure 0 0 0) 0x7f0d2411ef60 con 0x7f0d2c15ce80 2023-05-22T16:05:55.351 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.343+0000 7f0d30ffa640 1 --2- 172.21.15.16:0/309724882 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f0d18065530 0x7f0d180679e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-05-22T16:05:55.351 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.347+0000 7f0d297fa640 1 -- 172.21.15.16:0/309724882 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f0d240ec770 con 0x7f0d2c15ce80 2023-05-22T16:05:55.352 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T16:05:55.343+0000 7f69a9b59180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-05-22T16:05:55.352 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T16:05:55.347+0000 7fed27eaa180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-05-22T16:05:55.428 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T16:05:55.427+0000 7fed27eaa180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-05-22T16:05:55.497 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.495+0000 7f0d3325e640 1 -- 172.21.15.16:0/309724882 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f0d2c160690 con 0x7f0d2c15ce80 2023-05-22T16:05:55.497 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.495+0000 7f0d297fa640 1 -- 172.21.15.16:0/309724882 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v197) v1 ==== 93+0+10 (secure 0 0 0) 0x7f0d2c160690 con 0x7f0d2c15ce80 2023-05-22T16:05:55.498 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2023-05-22T16:05:55.500 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.495+0000 7f0d3325e640 1 -- 172.21.15.16:0/309724882 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f0d18065530 msgr2=0x7f0d180679e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-05-22T16:05:55.500 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.495+0000 7f0d3325e640 1 --2- 172.21.15.16:0/309724882 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f0d18065530 0x7f0d180679e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:55.501 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.495+0000 7f0d3325e640 1 -- 172.21.15.16:0/309724882 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d2c15ce80 msgr2=0x7f0d2c0b2f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:55.501 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.495+0000 7f0d3325e640 1 --2- 172.21.15.16:0/309724882 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d2c15ce80 0x7f0d2c0b2f00 secure :-1 s=READY pgs=2563 cs=0 l=1 rev1=1 crypto rx=0x7f0d2401a550 tx=0x7f0d240196c0 comp rx=0 tx=0).stop 2023-05-22T16:05:55.501 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.495+0000 7f0d3325e640 1 -- 172.21.15.16:0/309724882 shutdown_connections 2023-05-22T16:05:55.501 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.495+0000 7f0d3325e640 1 --2- 172.21.15.16:0/309724882 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f0d18065530 0x7f0d180679e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:55.502 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.495+0000 7f0d3325e640 1 --2- 172.21.15.16:0/309724882 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d2c15ce80 0x7f0d2c0b2f00 unknown :-1 s=CLOSED pgs=2563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:55.502 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.495+0000 7f0d3325e640 1 --2- 172.21.15.16:0/309724882 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0d2c15baa0 0x7f0d2c159db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:55.502 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.495+0000 7f0d3325e640 1 --2- 172.21.15.16:0/309724882 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0d2c06fe80 0x7f0d2c150300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:55.502 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.495+0000 7f0d3325e640 1 -- 172.21.15.16:0/309724882 >> 172.21.15.16:0/309724882 conn(0x7f0d2c0ce930 msgr2=0x7f0d2c06b730 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:05:55.502 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.495+0000 7f0d3325e640 1 -- 172.21.15.16:0/309724882 shutdown_connections 2023-05-22T16:05:55.503 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.495+0000 7f0d3325e640 1 -- 172.21.15.16:0/309724882 wait complete. 2023-05-22T16:05:55.516 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-05-22T16:05:55.598 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T16:05:55.591+0000 7f69a9b59180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-05-22T16:05:55.640 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.631+0000 7f489f5ae640 1 Processor -- start 2023-05-22T16:05:55.641 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.631+0000 7f489f5ae640 1 -- start start 2023-05-22T16:05:55.641 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.631+0000 7f489f5ae640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f489815baa0 0x7f489806cd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:55.641 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.631+0000 7f489f5ae640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4898070590 0x7f489815f770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:55.641 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.631+0000 7f489f5ae640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f489806d7d0 con 0x7f489806d270 2023-05-22T16:05:55.641 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.631+0000 7f489f5ae640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4898069800 con 0x7f489815baa0 2023-05-22T16:05:55.642 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.631+0000 7f489f5ae640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4898069ae0 con 0x7f4898070590 2023-05-22T16:05:55.642 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.631+0000 7f489d34a640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f489815baa0 0x7f489806cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:55.642 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.631+0000 7f489d34a640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f489815baa0 0x7f489806cd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38500/0 (socket says 172.21.15.16:38500) 2023-05-22T16:05:55.642 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.631+0000 7f489d34a640 1 -- 172.21.15.16:0/2615653224 learned_addr learned my addr 172.21.15.16:0/2615653224 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:05:55.642 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.631+0000 7f489db4b640 1 --2- 172.21.15.16:0/2615653224 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4898070590 0x7f489815f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:55.642 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.631+0000 7f489d34a640 1 -- 172.21.15.16:0/2615653224 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4898070590 msgr2=0x7f489815f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:05:55.642 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.631+0000 7f489d34a640 1 --2- 172.21.15.16:0/2615653224 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4898070590 0x7f489815f770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:55.643 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.631+0000 7f489d34a640 1 -- 172.21.15.16:0/2615653224 >> v1:172.21.15.62:6789/0 conn(0x7f489806d270 legacy=0x7f489806fe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:55.643 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.631+0000 7f489d34a640 1 -- 172.21.15.16:0/2615653224 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4898069dc0 con 0x7f489815baa0 2023-05-22T16:05:55.643 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.635+0000 7f489d34a640 1 --2- 172.21.15.16:0/2615653224 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f489815baa0 0x7f489806cd30 secure :-1 s=READY pgs=2564 cs=0 l=1 rev1=1 crypto rx=0x7f4894009870 tx=0x7f489401c510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=deb0c6b854cd4fbf server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:55.643 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.635+0000 7f488ffff640 1 -- 172.21.15.16:0/2615653224 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48940a95f0 con 0x7f489815baa0 2023-05-22T16:05:55.643 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.635+0000 7f488ffff640 1 -- 172.21.15.16:0/2615653224 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48940a9750 con 0x7f489815baa0 2023-05-22T16:05:55.643 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.635+0000 7f488ffff640 1 -- 172.21.15.16:0/2615653224 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48940a9ae0 con 0x7f489815baa0 2023-05-22T16:05:55.643 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.635+0000 7f489f5ae640 1 -- 172.21.15.16:0/2615653224 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f489815baa0 msgr2=0x7f489806cd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:55.644 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.635+0000 7f489f5ae640 1 --2- 172.21.15.16:0/2615653224 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f489815baa0 0x7f489806cd30 secure :-1 s=READY pgs=2564 cs=0 l=1 rev1=1 crypto rx=0x7f4894009870 tx=0x7f489401c510 comp rx=0 tx=0).stop 2023-05-22T16:05:55.644 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.635+0000 7f489f5ae640 1 -- 172.21.15.16:0/2615653224 shutdown_connections 2023-05-22T16:05:55.644 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.635+0000 7f489f5ae640 1 --2- 172.21.15.16:0/2615653224 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4898070590 0x7f489815f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:55.644 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.635+0000 7f489f5ae640 1 --2- 172.21.15.16:0/2615653224 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f489815baa0 0x7f489806cd30 unknown :-1 s=CLOSED pgs=2564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:55.644 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.635+0000 7f489f5ae640 1 -- 172.21.15.16:0/2615653224 >> 172.21.15.16:0/2615653224 conn(0x7f48980ce930 msgr2=0x7f48980ced30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:05:55.644 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.635+0000 7f489f5ae640 1 -- 172.21.15.16:0/2615653224 shutdown_connections 2023-05-22T16:05:55.644 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.635+0000 7f489f5ae640 1 -- 172.21.15.16:0/2615653224 wait complete. 2023-05-22T16:05:55.645 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.635+0000 7f489f5ae640 1 Processor -- start 2023-05-22T16:05:55.645 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.635+0000 7f489f5ae640 1 -- start start 2023-05-22T16:05:55.645 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.635+0000 7f489f5ae640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f489806d270 0x7f48981522d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:55.645 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.635+0000 7f489f5ae640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4898070590 0x7f489815df60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:55.645 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.635+0000 7f489f5ae640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f489815baa0 0x7f48980c3870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:55.645 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.635+0000 7f489f5ae640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48980c33e0 con 0x7f4898070590 2023-05-22T16:05:55.645 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.635+0000 7f489f5ae640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48980c27d0 con 0x7f489806d270 2023-05-22T16:05:55.645 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.635+0000 7f489f5ae640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48980c2ab0 con 0x7f489815baa0 2023-05-22T16:05:55.646 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.635+0000 7f489cb49640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4898070590 0x7f489815df60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:55.646 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.635+0000 7f489cb49640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4898070590 0x7f489815df60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38504/0 (socket says 172.21.15.16:38504) 2023-05-22T16:05:55.646 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.635+0000 7f489cb49640 1 -- 172.21.15.16:0/1193512439 learned_addr learned my addr 172.21.15.16:0/1193512439 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:05:55.646 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.635+0000 7f489db4b640 1 --2- 172.21.15.16:0/1193512439 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f489815baa0 0x7f48980c3870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:55.646 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.635+0000 7f489d34a640 1 --2- 172.21.15.16:0/1193512439 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f489806d270 0x7f48981522d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:55.646 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.635+0000 7f489cb49640 1 -- 172.21.15.16:0/1193512439 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f489815baa0 msgr2=0x7f48980c3870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:55.646 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.635+0000 7f489cb49640 1 --2- 172.21.15.16:0/1193512439 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f489815baa0 0x7f48980c3870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:55.647 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.635+0000 7f489cb49640 1 -- 172.21.15.16:0/1193512439 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f489806d270 msgr2=0x7f48981522d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:55.647 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.635+0000 7f489cb49640 1 --2- 172.21.15.16:0/1193512439 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f489806d270 0x7f48981522d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:55.647 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.635+0000 7f489cb49640 1 -- 172.21.15.16:0/1193512439 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48940094d0 con 0x7f4898070590 2023-05-22T16:05:55.647 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.635+0000 7f489cb49640 1 --2- 172.21.15.16:0/1193512439 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4898070590 0x7f489815df60 secure :-1 s=READY pgs=2565 cs=0 l=1 rev1=1 crypto rx=0x7f488800c500 tx=0x7f488800c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:55.647 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.635+0000 7f488dffb640 1 -- 172.21.15.16:0/1193512439 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4888005220 con 0x7f4898070590 2023-05-22T16:05:55.647 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.635+0000 7f488dffb640 1 -- 172.21.15.16:0/1193512439 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4888005380 con 0x7f4898070590 2023-05-22T16:05:55.647 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.639+0000 7f489f5ae640 1 -- 172.21.15.16:0/1193512439 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48980ce5d0 con 0x7f4898070590 2023-05-22T16:05:55.648 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.639+0000 7f488dffb640 1 -- 172.21.15.16:0/1193512439 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4888007270 con 0x7f4898070590 2023-05-22T16:05:55.648 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.639+0000 7f489f5ae640 1 -- 172.21.15.16:0/1193512439 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f489815be70 con 0x7f4898070590 2023-05-22T16:05:55.649 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.639+0000 7f489f5ae640 1 -- 172.21.15.16:0/1193512439 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48980ce760 con 0x7f4898070590 2023-05-22T16:05:55.649 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.639+0000 7f488dffb640 1 -- 172.21.15.16:0/1193512439 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 97) v1 ==== 84389+0+0 (secure 0 0 0) 0x7f48880073d0 con 0x7f4898070590 2023-05-22T16:05:55.649 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.639+0000 7f488dffb640 1 --2- 172.21.15.16:0/1193512439 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f487c065530 0x7f487c0679e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:55.650 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.639+0000 7f489d34a640 1 -- 172.21.15.16:0/1193512439 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f487c065530 msgr2=0x7f487c0679e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/65890 2023-05-22T16:05:55.650 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.639+0000 7f488dffb640 1 -- 172.21.15.16:0/1193512439 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(197..197 src has 1..197) v4 ==== 6782+0+0 (secure 0 0 0) 0x7f4888082220 con 0x7f4898070590 2023-05-22T16:05:55.650 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.639+0000 7f489d34a640 1 --2- 172.21.15.16:0/1193512439 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f487c065530 0x7f487c0679e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-05-22T16:05:55.650 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.643+0000 7f488dffb640 1 -- 172.21.15.16:0/1193512439 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f488804a620 con 0x7f4898070590 2023-05-22T16:05:55.653 INFO:tasks.ceph.mgr.x.smithi062.stderr:2023-05-22T16:05:55.647+0000 7f69a9b59180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-05-22T16:05:55.683 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T16:05:55.679+0000 7fed27eaa180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-05-22T16:05:55.739 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T16:05:55.735+0000 7fed27eaa180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-05-22T16:05:55.772 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.767+0000 7f488dffb640 1 -- 172.21.15.16:0/1193512439 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mgrmap(e 98) v1 ==== 83941+0+0 (secure 0 0 0) 0x7f488804fd30 con 0x7f4898070590 2023-05-22T16:05:55.772 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.767+0000 7f488dffb640 1 -- 172.21.15.16:0/1193512439 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f487c065530 msgr2=0x7f487c0679e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-05-22T16:05:55.772 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.767+0000 7f488dffb640 1 --2- 172.21.15.16:0/1193512439 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f487c065530 0x7f487c0679e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:55.805 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.803+0000 7f489f5ae640 1 -- 172.21.15.16:0/1193512439 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f48980ce760 con 0x7f4898070590 2023-05-22T16:05:55.806 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.803+0000 7f488dffb640 1 -- 172.21.15.16:0/1193512439 <== mon.0 v2:172.21.15.16:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v198) v1 ==== 92+0+10 (secure 0 0 0) 0x7f48980ce760 con 0x7f4898070590 2023-05-22T16:05:55.806 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 8 2023-05-22T16:05:55.808 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.803+0000 7f489f5ae640 1 -- 172.21.15.16:0/1193512439 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4898070590 msgr2=0x7f489815df60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:55.808 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.803+0000 7f489f5ae640 1 --2- 172.21.15.16:0/1193512439 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4898070590 0x7f489815df60 secure :-1 s=READY pgs=2565 cs=0 l=1 rev1=1 crypto rx=0x7f488800c500 tx=0x7f488800c9c0 comp rx=0 tx=0).stop 2023-05-22T16:05:55.808 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.803+0000 7f489f5ae640 1 -- 172.21.15.16:0/1193512439 shutdown_connections 2023-05-22T16:05:55.808 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.803+0000 7f489f5ae640 1 --2- 172.21.15.16:0/1193512439 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f489815baa0 0x7f48980c3870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:55.808 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.803+0000 7f489f5ae640 1 --2- 172.21.15.16:0/1193512439 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f487c065530 0x7f487c0679e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:55.809 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.803+0000 7f489f5ae640 1 --2- 172.21.15.16:0/1193512439 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4898070590 0x7f489815df60 unknown :-1 s=CLOSED pgs=2565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:55.809 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.803+0000 7f489f5ae640 1 --2- 172.21.15.16:0/1193512439 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f489806d270 0x7f48981522d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:55.809 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.803+0000 7f489f5ae640 1 -- 172.21.15.16:0/1193512439 >> 172.21.15.16:0/1193512439 conn(0x7f48980ce930 msgr2=0x7f489806fd80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:05:55.809 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.803+0000 7f489f5ae640 1 -- 172.21.15.16:0/1193512439 shutdown_connections 2023-05-22T16:05:55.809 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.803+0000 7f489f5ae640 1 -- 172.21.15.16:0/1193512439 wait complete. 2023-05-22T16:05:55.823 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-05-22T16:05:55.865 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T16:05:55.863+0000 7fecb3038640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T16:05:55.865 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T16:05:55.863+0000 7fecb3038640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T16:05:55.865 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T16:05:55.863+0000 7fecb3038640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T16:05:55.865 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T16:05:55.863+0000 7fecb3038640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T16:05:55.865 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T16:05:55.863+0000 7fecb3038640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T16:05:55.866 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T16:05:55.863+0000 7fecb0032640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T16:05:55.866 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T16:05:55.863+0000 7fecb0032640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T16:05:55.866 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T16:05:55.863+0000 7fecb0032640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T16:05:55.866 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T16:05:55.863+0000 7fecb0032640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T16:05:55.866 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-05-22T16:05:55.863+0000 7fecb0032640 -1 client.0 error registering admin socket command: (17) File exists 2023-05-22T16:05:55.946 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.943+0000 7f88cbdf7640 1 Processor -- start 2023-05-22T16:05:55.946 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.943+0000 7f88cbdf7640 1 -- start start 2023-05-22T16:05:55.947 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.943+0000 7f88cbdf7640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f88c4158c60 0x7f88c415c6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:55.948 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.943+0000 7f88cbdf7640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f88c40c9890 0x7f88c40afbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:55.948 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.943+0000 7f88cbdf7640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f88c40b0130 0x7f88c40cab30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:55.948 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.943+0000 7f88cbdf7640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88c40b05b0 con 0x7f88c40b0130 2023-05-22T16:05:55.948 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.943+0000 7f88cbdf7640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88c4159030 con 0x7f88c4158c60 2023-05-22T16:05:55.948 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.943+0000 7f88cbdf7640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88c40c9ce0 con 0x7f88c40c9890 2023-05-22T16:05:55.948 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.943+0000 7f88ca394640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f88c40b0130 0x7f88c40cab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:55.949 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.943+0000 7f88ca394640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f88c40b0130 0x7f88c40cab30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38612/0 (socket says 172.21.15.16:38612) 2023-05-22T16:05:55.949 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.943+0000 7f88ca394640 1 -- 172.21.15.16:0/1866626600 learned_addr learned my addr 172.21.15.16:0/1866626600 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:05:55.949 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.943+0000 7f88c9b93640 1 --2- 172.21.15.16:0/1866626600 >> v2:172.21.15.62:3300/0 conn(0x7f88c4158c60 0x7f88c415c6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:55.949 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.943+0000 7f88ca394640 1 -- 172.21.15.16:0/1866626600 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f88c40c9890 msgr2=0x7f88c40afbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:05:55.949 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.943+0000 7f88ca394640 1 --2- 172.21.15.16:0/1866626600 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f88c40c9890 0x7f88c40afbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:55.950 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.943+0000 7f88ca394640 1 -- 172.21.15.16:0/1866626600 >> v2:172.21.15.62:3300/0 conn(0x7f88c4158c60 msgr2=0x7f88c415c6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:05:55.950 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.943+0000 7f88ca394640 1 --2- 172.21.15.16:0/1866626600 >> v2:172.21.15.62:3300/0 conn(0x7f88c4158c60 0x7f88c415c6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:55.950 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.943+0000 7f88ca394640 1 -- 172.21.15.16:0/1866626600 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88c406fd70 con 0x7f88c40b0130 2023-05-22T16:05:55.950 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.943+0000 7f88c9b93640 1 --2- 172.21.15.16:0/1866626600 >> v2:172.21.15.62:3300/0 conn(0x7f88c4158c60 0x7f88c415c6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T16:05:55.951 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.947+0000 7f88ca394640 1 --2- 172.21.15.16:0/1866626600 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f88c40b0130 0x7f88c40cab30 secure :-1 s=READY pgs=2578 cs=0 l=1 rev1=1 crypto rx=0x7f88c40b0c60 tx=0x7f88bc01c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f38d454cb14ff992 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:55.951 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.947+0000 7f88c8b91640 1 -- 172.21.15.16:0/1866626600 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88bc0ac410 con 0x7f88c40b0130 2023-05-22T16:05:55.951 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.947+0000 7f88c8b91640 1 -- 172.21.15.16:0/1866626600 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88bc0ac570 con 0x7f88c40b0130 2023-05-22T16:05:55.951 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.947+0000 7f88c8b91640 1 -- 172.21.15.16:0/1866626600 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88bc0ac900 con 0x7f88c40b0130 2023-05-22T16:05:55.952 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.947+0000 7f88cbdf7640 1 -- 172.21.15.16:0/1866626600 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f88c40b0130 msgr2=0x7f88c40cab30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:55.952 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.947+0000 7f88cbdf7640 1 --2- 172.21.15.16:0/1866626600 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f88c40b0130 0x7f88c40cab30 secure :-1 s=READY pgs=2578 cs=0 l=1 rev1=1 crypto rx=0x7f88c40b0c60 tx=0x7f88bc01c500 comp rx=0 tx=0).stop 2023-05-22T16:05:55.952 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.947+0000 7f88cbdf7640 1 -- 172.21.15.16:0/1866626600 shutdown_connections 2023-05-22T16:05:55.952 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.947+0000 7f88cbdf7640 1 --2- 172.21.15.16:0/1866626600 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f88c40b0130 0x7f88c40cab30 unknown :-1 s=CLOSED pgs=2578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:55.952 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.947+0000 7f88cbdf7640 1 --2- 172.21.15.16:0/1866626600 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f88c40c9890 0x7f88c40afbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:55.952 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.947+0000 7f88cbdf7640 1 --2- 172.21.15.16:0/1866626600 >> v2:172.21.15.62:3300/0 conn(0x7f88c4158c60 0x7f88c415c6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:55.952 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.947+0000 7f88cbdf7640 1 -- 172.21.15.16:0/1866626600 >> 172.21.15.16:0/1866626600 conn(0x7f88c4069800 msgr2=0x7f88c4069c00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:05:55.953 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.947+0000 7f88cbdf7640 1 -- 172.21.15.16:0/1866626600 shutdown_connections 2023-05-22T16:05:55.953 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.947+0000 7f88cbdf7640 1 -- 172.21.15.16:0/1866626600 wait complete. 2023-05-22T16:05:55.954 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.947+0000 7f88cbdf7640 1 Processor -- start 2023-05-22T16:05:55.955 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.947+0000 7f88cbdf7640 1 -- start start 2023-05-22T16:05:55.955 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.951+0000 7f88cbdf7640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f88c40b0130 0x7f88c41497b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:55.955 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.951+0000 7f88cbdf7640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f88c40c9890 0x7f88c40c2f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:55.955 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.951+0000 7f88cbdf7640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f88c4158c60 0x7f88c40c3450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:55.955 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.951+0000 7f88cbdf7640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88c40c2a80 con 0x7f88c40b0130 2023-05-22T16:05:55.955 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.951+0000 7f88cbdf7640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88c4152e40 con 0x7f88c40c9890 2023-05-22T16:05:55.955 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.951+0000 7f88cbdf7640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88c4152b60 con 0x7f88c4158c60 2023-05-22T16:05:55.956 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.951+0000 7f88c9b93640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f88c40b0130 0x7f88c41497b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:55.956 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.951+0000 7f88c9b93640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f88c40b0130 0x7f88c41497b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38622/0 (socket says 172.21.15.16:38622) 2023-05-22T16:05:55.956 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.951+0000 7f88c9b93640 1 -- 172.21.15.16:0/3721144213 learned_addr learned my addr 172.21.15.16:0/3721144213 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:05:55.956 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.951+0000 7f88c9392640 1 --2- 172.21.15.16:0/3721144213 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f88c40c9890 0x7f88c40c2f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:55.956 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.951+0000 7f88ca394640 1 --2- 172.21.15.16:0/3721144213 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f88c4158c60 0x7f88c40c3450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:55.956 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.951+0000 7f88c9b93640 1 -- 172.21.15.16:0/3721144213 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f88c4158c60 msgr2=0x7f88c40c3450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:55.956 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.951+0000 7f88c9b93640 1 --2- 172.21.15.16:0/3721144213 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f88c4158c60 0x7f88c40c3450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:55.956 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.951+0000 7f88c9b93640 1 -- 172.21.15.16:0/3721144213 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f88c40c9890 msgr2=0x7f88c40c2f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:55.957 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.951+0000 7f88c9b93640 1 --2- 172.21.15.16:0/3721144213 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f88c40c9890 0x7f88c40c2f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:55.957 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.951+0000 7f88c9b93640 1 -- 172.21.15.16:0/3721144213 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88bc0094d0 con 0x7f88c40b0130 2023-05-22T16:05:55.957 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.951+0000 7f88c9b93640 1 --2- 172.21.15.16:0/3721144213 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f88c40b0130 0x7f88c41497b0 secure :-1 s=READY pgs=2579 cs=0 l=1 rev1=1 crypto rx=0x7f88c000ce00 tx=0x7f88c000a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:55.957 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.951+0000 7f88ba7fc640 1 -- 172.21.15.16:0/3721144213 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88c000f890 con 0x7f88c40b0130 2023-05-22T16:05:55.957 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.951+0000 7f88ba7fc640 1 -- 172.21.15.16:0/3721144213 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88c000f9f0 con 0x7f88c40b0130 2023-05-22T16:05:55.957 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.951+0000 7f88ba7fc640 1 -- 172.21.15.16:0/3721144213 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88c000fdc0 con 0x7f88c40b0130 2023-05-22T16:05:55.957 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.951+0000 7f88cbdf7640 1 -- 172.21.15.16:0/3721144213 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88c415d540 con 0x7f88c40b0130 2023-05-22T16:05:55.957 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.951+0000 7f88cbdf7640 1 -- 172.21.15.16:0/3721144213 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88c415d260 con 0x7f88c40b0130 2023-05-22T16:05:55.958 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.951+0000 7f88cbdf7640 1 -- 172.21.15.16:0/3721144213 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88c40cdbe0 con 0x7f88c40b0130 2023-05-22T16:05:55.958 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.951+0000 7f88ba7fc640 1 -- 172.21.15.16:0/3721144213 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 98) v1 ==== 83941+0+0 (secure 0 0 0) 0x7f88c000b040 con 0x7f88c40b0130 2023-05-22T16:05:55.958 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.951+0000 7f88ba7fc640 1 -- 172.21.15.16:0/3721144213 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(198..198 src has 1..198) v4 ==== 7083+0+0 (secure 0 0 0) 0x7f88c00830e0 con 0x7f88c40b0130 2023-05-22T16:05:55.959 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:55.955+0000 7f88ba7fc640 1 -- 172.21.15.16:0/3721144213 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f88c0050db0 con 0x7f88c40b0130 2023-05-22T16:05:56.107 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.103+0000 7f88cbdf7640 1 -- 172.21.15.16:0/3721144213 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f88c415b470 con 0x7f88c40b0130 2023-05-22T16:05:56.108 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.103+0000 7f88ba7fc640 1 -- 172.21.15.16:0/3721144213 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v198) v1 ==== 104+0+11 (secure 0 0 0) 0x7f88c00543d0 con 0x7f88c40b0130 2023-05-22T16:05:56.108 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2023-05-22T16:05:56.110 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.107+0000 7f88cbdf7640 1 -- 172.21.15.16:0/3721144213 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f88c40b0130 msgr2=0x7f88c41497b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:56.110 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.107+0000 7f88cbdf7640 1 --2- 172.21.15.16:0/3721144213 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f88c40b0130 0x7f88c41497b0 secure :-1 s=READY pgs=2579 cs=0 l=1 rev1=1 crypto rx=0x7f88c000ce00 tx=0x7f88c000a650 comp rx=0 tx=0).stop 2023-05-22T16:05:56.110 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.107+0000 7f88cbdf7640 1 -- 172.21.15.16:0/3721144213 shutdown_connections 2023-05-22T16:05:56.111 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.107+0000 7f88cbdf7640 1 --2- 172.21.15.16:0/3721144213 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f88c4158c60 0x7f88c40c3450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:56.111 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.107+0000 7f88cbdf7640 1 --2- 172.21.15.16:0/3721144213 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f88c40c9890 0x7f88c40c2f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:56.111 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.107+0000 7f88cbdf7640 1 --2- 172.21.15.16:0/3721144213 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f88c40b0130 0x7f88c41497b0 unknown :-1 s=CLOSED pgs=2579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:56.111 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.107+0000 7f88cbdf7640 1 -- 172.21.15.16:0/3721144213 >> 172.21.15.16:0/3721144213 conn(0x7f88c4069800 msgr2=0x7f88c40ce130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:05:56.111 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.107+0000 7f88cbdf7640 1 -- 172.21.15.16:0/3721144213 shutdown_connections 2023-05-22T16:05:56.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.107+0000 7f88cbdf7640 1 -- 172.21.15.16:0/3721144213 wait complete. 2023-05-22T16:05:56.125 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-05-22T16:05:56.285 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.283+0000 7f8230c8d640 1 Processor -- start 2023-05-22T16:05:56.285 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.283+0000 7f8230c8d640 1 -- start start 2023-05-22T16:05:56.286 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.283+0000 7f8230c8d640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f822c0ad0d0 0x7f822c0afbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:56.286 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.283+0000 7f8230c8d640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f822c0b0130 0x7f822c0c9420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:56.286 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.283+0000 7f8230c8d640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f822c0ad530 con 0x7f822c15c040 2023-05-22T16:05:56.286 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.283+0000 7f8230c8d640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f822c0ada70 con 0x7f822c0b0130 2023-05-22T16:05:56.286 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.283+0000 7f8230c8d640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f822c0b0530 con 0x7f822c0ad0d0 2023-05-22T16:05:56.287 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.283+0000 7f822ad9d640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f822c0b0130 0x7f822c0c9420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:56.287 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.283+0000 7f822ad9d640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f822c0b0130 0x7f822c0c9420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38628/0 (socket says 172.21.15.16:38628) 2023-05-22T16:05:56.287 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.283+0000 7f822ad9d640 1 -- 172.21.15.16:0/3665535388 learned_addr learned my addr 172.21.15.16:0/3665535388 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:05:56.287 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.283+0000 7f8221d9b640 1 --2- 172.21.15.16:0/3665535388 >> v2:172.21.15.62:3300/0 conn(0x7f822c0ad0d0 0x7f822c0afbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:56.288 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.283+0000 7f822ad9d640 1 -- 172.21.15.16:0/3665535388 >> v2:172.21.15.62:3300/0 conn(0x7f822c0ad0d0 msgr2=0x7f822c0afbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:05:56.288 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.283+0000 7f822ad9d640 1 --2- 172.21.15.16:0/3665535388 >> v2:172.21.15.62:3300/0 conn(0x7f822c0ad0d0 0x7f822c0afbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:56.288 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.283+0000 7f822ad9d640 1 -- 172.21.15.16:0/3665535388 >> v1:172.21.15.62:6789/0 conn(0x7f822c15c040 legacy=0x7f822c1603e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:05:56.288 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.283+0000 7f822ad9d640 1 -- 172.21.15.16:0/3665535388 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f822c0ad790 con 0x7f822c0b0130 2023-05-22T16:05:56.289 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.287+0000 7f822ad9d640 1 --2- 172.21.15.16:0/3665535388 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f822c0b0130 0x7f822c0c9420 secure :-1 s=READY pgs=2580 cs=0 l=1 rev1=1 crypto rx=0x7f822c0b42c0 tx=0x7f822401c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c15f4215a4821d5 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:56.289 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.287+0000 7f8229d9b640 1 -- 172.21.15.16:0/3665535388 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82240a5830 con 0x7f822c0b0130 2023-05-22T16:05:56.289 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.287+0000 7f8229d9b640 1 -- 172.21.15.16:0/3665535388 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82240a5990 con 0x7f822c0b0130 2023-05-22T16:05:56.289 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.287+0000 7f8229d9b640 1 -- 172.21.15.16:0/3665535388 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82240a5d20 con 0x7f822c0b0130 2023-05-22T16:05:56.290 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.287+0000 7f8230c8d640 1 -- 172.21.15.16:0/3665535388 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f822c0b0130 msgr2=0x7f822c0c9420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:56.290 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.287+0000 7f8230c8d640 1 --2- 172.21.15.16:0/3665535388 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f822c0b0130 0x7f822c0c9420 secure :-1 s=READY pgs=2580 cs=0 l=1 rev1=1 crypto rx=0x7f822c0b42c0 tx=0x7f822401c970 comp rx=0 tx=0).stop 2023-05-22T16:05:56.290 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.287+0000 7f8230c8d640 1 -- 172.21.15.16:0/3665535388 shutdown_connections 2023-05-22T16:05:56.290 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.287+0000 7f8230c8d640 1 --2- 172.21.15.16:0/3665535388 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f822c0b0130 0x7f822c0c9420 unknown :-1 s=CLOSED pgs=2580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:56.290 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.287+0000 7f8230c8d640 1 --2- 172.21.15.16:0/3665535388 >> v2:172.21.15.62:3300/0 conn(0x7f822c0ad0d0 0x7f822c0afbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:56.291 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.287+0000 7f8230c8d640 1 -- 172.21.15.16:0/3665535388 >> 172.21.15.16:0/3665535388 conn(0x7f822c069800 msgr2=0x7f822c069c00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:05:56.291 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.287+0000 7f8230c8d640 1 -- 172.21.15.16:0/3665535388 shutdown_connections 2023-05-22T16:05:56.291 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.287+0000 7f8230c8d640 1 -- 172.21.15.16:0/3665535388 wait complete. 2023-05-22T16:05:56.292 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.287+0000 7f8230c8d640 1 Processor -- start 2023-05-22T16:05:56.292 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.287+0000 7f8230c8d640 1 -- start start 2023-05-22T16:05:56.293 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.287+0000 7f8230c8d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f822c0ad0d0 0x7f822c159090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:56.293 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.287+0000 7f8230c8d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f822c0b0130 0x7f822c0c1c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:56.293 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.287+0000 7f8230c8d640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f822c15c040 0x7f822c0af3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:56.293 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.287+0000 7f8230c8d640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f822c0c5040 con 0x7f822c0b0130 2023-05-22T16:05:56.294 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.287+0000 7f8230c8d640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f822c158f10 con 0x7f822c0ad0d0 2023-05-22T16:05:56.294 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.287+0000 7f8230c8d640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f822c0c21c0 con 0x7f822c15c040 2023-05-22T16:05:56.294 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.287+0000 7f8221d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f822c0b0130 0x7f822c0c1c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:56.294 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.287+0000 7f8221d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f822c0b0130 0x7f822c0c1c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38638/0 (socket says 172.21.15.16:38638) 2023-05-22T16:05:56.294 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.287+0000 7f8221d9b640 1 -- 172.21.15.16:0/1288963003 learned_addr learned my addr 172.21.15.16:0/1288963003 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:05:56.294 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.287+0000 7f822ad9d640 1 --2- 172.21.15.16:0/1288963003 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f822c15c040 0x7f822c0af3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:56.294 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.287+0000 7f822a59c640 1 --2- 172.21.15.16:0/1288963003 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f822c0ad0d0 0x7f822c159090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:56.294 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.287+0000 7f8221d9b640 1 -- 172.21.15.16:0/1288963003 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f822c15c040 msgr2=0x7f822c0af3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:56.295 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.287+0000 7f8221d9b640 1 --2- 172.21.15.16:0/1288963003 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f822c15c040 0x7f822c0af3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:56.295 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.287+0000 7f8221d9b640 1 -- 172.21.15.16:0/1288963003 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f822c0ad0d0 msgr2=0x7f822c159090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:56.295 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.287+0000 7f8221d9b640 1 --2- 172.21.15.16:0/1288963003 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f822c0ad0d0 0x7f822c159090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:56.295 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.287+0000 7f8221d9b640 1 -- 172.21.15.16:0/1288963003 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8224009b30 con 0x7f822c0b0130 2023-05-22T16:05:56.295 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.287+0000 7f8221d9b640 1 --2- 172.21.15.16:0/1288963003 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f822c0b0130 0x7f822c0c1c80 secure :-1 s=READY pgs=2581 cs=0 l=1 rev1=1 crypto rx=0x7f821400d620 tx=0x7f821400dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:56.295 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.287+0000 7f82237fe640 1 -- 172.21.15.16:0/1288963003 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f821400b620 con 0x7f822c0b0130 2023-05-22T16:05:56.295 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.287+0000 7f82237fe640 1 -- 172.21.15.16:0/1288963003 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f821400b780 con 0x7f822c0b0130 2023-05-22T16:05:56.296 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.287+0000 7f8230c8d640 1 -- 172.21.15.16:0/1288963003 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f822c0cc050 con 0x7f822c0b0130 2023-05-22T16:05:56.296 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.287+0000 7f82237fe640 1 -- 172.21.15.16:0/1288963003 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f821400bb10 con 0x7f822c0b0130 2023-05-22T16:05:56.296 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.287+0000 7f8230c8d640 1 -- 172.21.15.16:0/1288963003 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f822c0c8f00 con 0x7f822c0b0130 2023-05-22T16:05:56.296 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.291+0000 7f8230c8d640 1 -- 172.21.15.16:0/1288963003 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f822c0cc1e0 con 0x7f822c0b0130 2023-05-22T16:05:56.297 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.291+0000 7f82237fe640 1 -- 172.21.15.16:0/1288963003 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 98) v1 ==== 83941+0+0 (secure 0 0 0) 0x7f821400bc70 con 0x7f822c0b0130 2023-05-22T16:05:56.297 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.291+0000 7f82237fe640 1 -- 172.21.15.16:0/1288963003 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(198..198 src has 1..198) v4 ==== 7083+0+0 (secure 0 0 0) 0x7f8214082be0 con 0x7f822c0b0130 2023-05-22T16:05:56.298 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.295+0000 7f82237fe640 1 -- 172.21.15.16:0/1288963003 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f8214050850 con 0x7f822c0b0130 2023-05-22T16:05:56.449 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.447+0000 7f8230c8d640 1 -- 172.21.15.16:0/1288963003 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f822c0cadb0 con 0x7f822c0b0130 2023-05-22T16:05:56.450 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.447+0000 7f82237fe640 1 -- 172.21.15.16:0/1288963003 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v198) v1 ==== 100+0+11 (secure 0 0 0) 0x7f8214007380 con 0x7f822c0b0130 2023-05-22T16:05:56.450 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 64 2023-05-22T16:05:56.451 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.447+0000 7f8230c8d640 1 -- 172.21.15.16:0/1288963003 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f822c0b0130 msgr2=0x7f822c0c1c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:56.452 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.447+0000 7f8230c8d640 1 --2- 172.21.15.16:0/1288963003 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f822c0b0130 0x7f822c0c1c80 secure :-1 s=READY pgs=2581 cs=0 l=1 rev1=1 crypto rx=0x7f821400d620 tx=0x7f821400dae0 comp rx=0 tx=0).stop 2023-05-22T16:05:56.452 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.447+0000 7f8230c8d640 1 -- 172.21.15.16:0/1288963003 shutdown_connections 2023-05-22T16:05:56.452 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.447+0000 7f8230c8d640 1 --2- 172.21.15.16:0/1288963003 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f822c15c040 0x7f822c0af3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:56.453 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.447+0000 7f8230c8d640 1 --2- 172.21.15.16:0/1288963003 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f822c0b0130 0x7f822c0c1c80 unknown :-1 s=CLOSED pgs=2581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:56.453 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.447+0000 7f8230c8d640 1 --2- 172.21.15.16:0/1288963003 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f822c0ad0d0 0x7f822c159090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:56.453 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.447+0000 7f8230c8d640 1 -- 172.21.15.16:0/1288963003 >> 172.21.15.16:0/1288963003 conn(0x7f822c069800 msgr2=0x7f822c0ae570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:05:56.453 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.447+0000 7f8230c8d640 1 -- 172.21.15.16:0/1288963003 shutdown_connections 2023-05-22T16:05:56.453 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.447+0000 7f8230c8d640 1 -- 172.21.15.16:0/1288963003 wait complete. 2023-05-22T16:05:56.469 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-05-22T16:05:56.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.583+0000 7fc29c3e2640 1 Processor -- start 2023-05-22T16:05:56.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.583+0000 7fc29c3e2640 1 -- start start 2023-05-22T16:05:56.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.583+0000 7fc29c3e2640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc2940ce350 0x7fc294166b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:56.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.583+0000 7fc29c3e2640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2940cf190 con 0x7fc2940d0730 2023-05-22T16:05:56.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.583+0000 7fc29c3e2640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2940d3360 con 0x7fc2940cfa20 2023-05-22T16:05:56.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.583+0000 7fc29c3e2640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc29415c4f0 con 0x7fc2940ce350 2023-05-22T16:05:56.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.583+0000 7fc29997d640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fc2940d0730 0x7fc2940d20c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:45784/0 (socket says 172.21.15.16:45784) 2023-05-22T16:05:56.587 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.583+0000 7fc29997d640 1 -- 172.21.15.16:0/1956983674 learned_addr learned my addr 172.21.15.16:0/1956983674 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T16:05:56.588 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.583+0000 7fc29a97f640 1 --2- 172.21.15.16:0/1956983674 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc2940ce350 0x7fc294166b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:56.588 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.583+0000 7fc29917c640 1 -- 172.21.15.16:0/1956983674 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 930371744 0 0) 0x7fc2940cf190 con 0x7fc2940d0730 2023-05-22T16:05:56.589 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.583+0000 7fc29917c640 1 -- 172.21.15.16:0/1956983674 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc280003530 con 0x7fc2940d0730 2023-05-22T16:05:56.589 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.583+0000 7fc29917c640 1 -- 172.21.15.16:0/1956983674 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2803533406 0 0) 0x7fc280003530 con 0x7fc2940d0730 2023-05-22T16:05:56.589 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.583+0000 7fc29917c640 1 -- 172.21.15.16:0/1956983674 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc2940ce350 msgr2=0x7fc294166b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:05:56.589 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.583+0000 7fc29917c640 1 --2- 172.21.15.16:0/1956983674 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc2940ce350 0x7fc294166b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:56.589 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.583+0000 7fc29917c640 1 -- 172.21.15.16:0/1956983674 >> v1:172.21.15.62:6789/0 conn(0x7fc2940cfa20 legacy=0x7fc2940d0020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:56.589 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.583+0000 7fc29917c640 1 -- 172.21.15.16:0/1956983674 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2940d2da0 con 0x7fc2940d0730 2023-05-22T16:05:56.589 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.583+0000 7fc29917c640 1 -- 172.21.15.16:0/1956983674 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7fc290001680 con 0x7fc2940d0730 2023-05-22T16:05:56.589 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.583+0000 7fc29917c640 1 -- 172.21.15.16:0/1956983674 >> v1:172.21.15.16:6789/0 conn(0x7fc2940d0730 legacy=0x7fc2940d20c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:56.590 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.583+0000 7fc29917c640 1 --2- 172.21.15.16:0/1956983674 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc2800039b0 0x7fc280003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:56.590 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.583+0000 7fc29917c640 1 --2- 172.21.15.16:0/1956983674 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc2940ce350 0x7fc2800047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:56.590 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.583+0000 7fc29917c640 1 --2- 172.21.15.16:0/1956983674 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc280004d10 0x7fc280005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:56.590 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.583+0000 7fc29917c640 1 -- 172.21.15.16:0/1956983674 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc29415c4f0 con 0x7fc280004d10 2023-05-22T16:05:56.590 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.583+0000 7fc29917c640 1 -- 172.21.15.16:0/1956983674 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc28000bad0 con 0x7fc2800039b0 2023-05-22T16:05:56.590 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.583+0000 7fc29917c640 1 -- 172.21.15.16:0/1956983674 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc28000bc60 con 0x7fc2940ce350 2023-05-22T16:05:56.590 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.583+0000 7fc29a97f640 1 --2- 172.21.15.16:0/1956983674 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc280004d10 0x7fc280005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:56.591 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.583+0000 7fc29a17e640 1 --2- 172.21.15.16:0/1956983674 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc2800039b0 0x7fc280003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:56.591 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.583+0000 7fc29997d640 1 --2- 172.21.15.16:0/1956983674 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc2940ce350 0x7fc2800047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:56.591 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.583+0000 7fc29a97f640 1 -- 172.21.15.16:0/1956983674 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc2940ce350 msgr2=0x7fc2800047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:05:56.591 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.583+0000 7fc29a97f640 1 --2- 172.21.15.16:0/1956983674 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc2940ce350 0x7fc2800047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:56.591 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.583+0000 7fc29a97f640 1 -- 172.21.15.16:0/1956983674 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc2800039b0 msgr2=0x7fc280003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:05:56.591 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.583+0000 7fc29a97f640 1 --2- 172.21.15.16:0/1956983674 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc2800039b0 0x7fc280003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:56.591 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.583+0000 7fc29a97f640 1 -- 172.21.15.16:0/1956983674 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc280003530 con 0x7fc280004d10 2023-05-22T16:05:56.591 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.583+0000 7fc29a17e640 1 --2- 172.21.15.16:0/1956983674 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc2800039b0 0x7fc280003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T16:05:56.595 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.587+0000 7fc29a97f640 1 --2- 172.21.15.16:0/1956983674 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc280004d10 0x7fc280005180 secure :-1 s=READY pgs=2583 cs=0 l=1 rev1=1 crypto rx=0x7fc28c009fc0 tx=0x7fc28c01cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36827aae067f3b82 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:56.595 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.587+0000 7fc29917c640 1 -- 172.21.15.16:0/1956983674 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc28c0a5c70 con 0x7fc280004d10 2023-05-22T16:05:56.596 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.587+0000 7fc29917c640 1 -- 172.21.15.16:0/1956983674 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc28c00f040 con 0x7fc280004d10 2023-05-22T16:05:56.596 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.587+0000 7fc29917c640 1 -- 172.21.15.16:0/1956983674 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc28c0ad440 con 0x7fc280004d10 2023-05-22T16:05:56.596 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.587+0000 7fc29c3e2640 1 -- 172.21.15.16:0/1956983674 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc280004d10 msgr2=0x7fc280005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:56.596 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.587+0000 7fc29c3e2640 1 --2- 172.21.15.16:0/1956983674 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc280004d10 0x7fc280005180 secure :-1 s=READY pgs=2583 cs=0 l=1 rev1=1 crypto rx=0x7fc28c009fc0 tx=0x7fc28c01cc60 comp rx=0 tx=0).stop 2023-05-22T16:05:56.596 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.587+0000 7fc29a97f640 1 -- 172.21.15.16:0/1956983674 reap_dead start 2023-05-22T16:05:56.596 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.587+0000 7fc29c3e2640 1 -- 172.21.15.16:0/1956983674 shutdown_connections 2023-05-22T16:05:56.596 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.587+0000 7fc29c3e2640 1 -- 172.21.15.16:0/1956983674 >> 172.21.15.16:0/1956983674 conn(0x7fc294071850 msgr2=0x7fc29409b830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:05:56.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.587+0000 7fc29c3e2640 1 -- 172.21.15.16:0/1956983674 shutdown_connections 2023-05-22T16:05:56.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.587+0000 7fc29c3e2640 1 -- 172.21.15.16:0/1956983674 wait complete. 2023-05-22T16:05:56.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.587+0000 7fc29c3e2640 1 Processor -- start 2023-05-22T16:05:56.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.587+0000 7fc29c3e2640 1 -- start start 2023-05-22T16:05:56.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.591+0000 7fc29c3e2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc280004d10 0x7fc29415a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:56.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.591+0000 7fc29c3e2640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc2940cab00 0x7fc2940d20c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:56.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.591+0000 7fc29c3e2640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc2940caf70 0x7fc2940d2600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:56.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.591+0000 7fc29c3e2640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2941f4f50 con 0x7fc280004d10 2023-05-22T16:05:56.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.591+0000 7fc29c3e2640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2940b5320 con 0x7fc2940cab00 2023-05-22T16:05:56.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.591+0000 7fc29c3e2640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2940b7fa0 con 0x7fc2940caf70 2023-05-22T16:05:56.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.591+0000 7fc29a17e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc280004d10 0x7fc29415a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:56.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.591+0000 7fc29a17e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc280004d10 0x7fc29415a7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38658/0 (socket says 172.21.15.16:38658) 2023-05-22T16:05:56.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.591+0000 7fc29a17e640 1 -- 172.21.15.16:0/3587447736 learned_addr learned my addr 172.21.15.16:0/3587447736 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:05:56.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.591+0000 7fc29997d640 1 --2- 172.21.15.16:0/3587447736 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc2940cab00 0x7fc2940d20c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:56.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.591+0000 7fc29a17e640 1 -- 172.21.15.16:0/3587447736 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc2940caf70 msgr2=0x7fc2940d2600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:56.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.591+0000 7fc29a97f640 1 --2- 172.21.15.16:0/3587447736 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc2940caf70 0x7fc2940d2600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:56.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.591+0000 7fc29a17e640 1 --2- 172.21.15.16:0/3587447736 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc2940caf70 0x7fc2940d2600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:56.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.591+0000 7fc29a17e640 1 -- 172.21.15.16:0/3587447736 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc2940cab00 msgr2=0x7fc2940d20c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:56.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.591+0000 7fc29a17e640 1 --2- 172.21.15.16:0/3587447736 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc2940cab00 0x7fc2940d20c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:56.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.591+0000 7fc29a17e640 1 -- 172.21.15.16:0/3587447736 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2940ccb40 con 0x7fc280004d10 2023-05-22T16:05:56.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.591+0000 7fc29a17e640 1 --2- 172.21.15.16:0/3587447736 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc280004d10 0x7fc29415a7a0 secure :-1 s=READY pgs=2584 cs=0 l=1 rev1=1 crypto rx=0x7fc28400e580 tx=0x7fc28400ea40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:56.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.591+0000 7fc28affd640 1 -- 172.21.15.16:0/3587447736 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2840052e0 con 0x7fc280004d10 2023-05-22T16:05:56.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.591+0000 7fc28affd640 1 -- 172.21.15.16:0/3587447736 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc284005440 con 0x7fc280004d10 2023-05-22T16:05:56.600 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.591+0000 7fc29c3e2640 1 -- 172.21.15.16:0/3587447736 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2940d3080 con 0x7fc280004d10 2023-05-22T16:05:56.600 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.591+0000 7fc28affd640 1 -- 172.21.15.16:0/3587447736 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc284003890 con 0x7fc280004d10 2023-05-22T16:05:56.600 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.591+0000 7fc29c3e2640 1 -- 172.21.15.16:0/3587447736 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2940cce20 con 0x7fc280004d10 2023-05-22T16:05:56.600 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.591+0000 7fc29c3e2640 1 -- 172.21.15.16:0/3587447736 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc294073830 con 0x7fc280004d10 2023-05-22T16:05:56.600 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.591+0000 7fc28affd640 1 -- 172.21.15.16:0/3587447736 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 98) v1 ==== 83941+0+0 (secure 0 0 0) 0x7fc2840039f0 con 0x7fc280004d10 2023-05-22T16:05:56.600 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.591+0000 7fc28affd640 1 -- 172.21.15.16:0/3587447736 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(198..198 src has 1..198) v4 ==== 7083+0+0 (secure 0 0 0) 0x7fc284082550 con 0x7fc280004d10 2023-05-22T16:05:56.601 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.595+0000 7fc28affd640 1 -- 172.21.15.16:0/3587447736 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fc284050140 con 0x7fc280004d10 2023-05-22T16:05:56.782 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.779+0000 7fc28affd640 1 -- 172.21.15.16:0/3587447736 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mgrmap(e 99) v1 ==== 84068+0+0 (secure 0 0 0) 0x7fc28404fbf0 con 0x7fc280004d10 2023-05-22T16:05:56.783 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.779+0000 7fc28affd640 1 --2- 172.21.15.16:0/3587447736 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fc27806b760 0x7fc27806db40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:56.785 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.783+0000 7fc29997d640 1 --2- 172.21.15.16:0/3587447736 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fc27806b760 0x7fc27806db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:56.789 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.787+0000 7fc29997d640 1 --2- 172.21.15.16:0/3587447736 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fc27806b760 0x7fc27806db40 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fc2941e2970 tx=0x7fc290009040 comp rx=0 tx=0).ready entity=mgr.8195 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:56.796 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.795+0000 7fc29c3e2640 1 -- 172.21.15.16:0/3587447736 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc29406b5f0 con 0x7fc280004d10 2023-05-22T16:05:56.797 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.795+0000 7fc28affd640 1 -- 172.21.15.16:0/3587447736 <== mon.0 v2:172.21.15.16:3300/0 8 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 81 v81) v1 ==== 94+0+4653 (secure 0 0 0) 0x7fc28406d230 con 0x7fc280004d10 2023-05-22T16:05:56.798 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T16:05:56.798 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":81,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7979,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/1053276484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":1053276484},{"type":"v1","addr":"172.21.15.62:6835","nonce":1053276484}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":76},{"gid":8983,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6833/1372552690","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":1372552690},{"type":"v1","addr":"172.21.15.62:6833","nonce":1372552690}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":76},{"gid":10776,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/982492476","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":982492476},{"type":"v1","addr":"172.21.15.16:6835","nonce":982492476}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":76}],"filesystems":[{"mdsmap":{"epoch":81,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-05-22T16:05:21.922031+0000","modified":"2023-05-22T16:05:24.940696+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10782},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10782":{"gid":10782,"name":"c","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.16:6837/1705367575","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":1705367575},{"type":"v1","addr":"172.21.15.16:6837","nonce":1705367575}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[25],"metadata_pool":24,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":11}]} 2023-05-22T16:05:56.798 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 81 2023-05-22T16:05:56.799 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.795+0000 7fc29c3e2640 1 -- 172.21.15.16:0/3587447736 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fc27806b760 msgr2=0x7fc27806db40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:56.799 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.795+0000 7fc29c3e2640 1 --2- 172.21.15.16:0/3587447736 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fc27806b760 0x7fc27806db40 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fc2941e2970 tx=0x7fc290009040 comp rx=0 tx=0).stop 2023-05-22T16:05:56.799 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.795+0000 7fc29c3e2640 1 -- 172.21.15.16:0/3587447736 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc280004d10 msgr2=0x7fc29415a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:56.799 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.795+0000 7fc29c3e2640 1 --2- 172.21.15.16:0/3587447736 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc280004d10 0x7fc29415a7a0 secure :-1 s=READY pgs=2584 cs=0 l=1 rev1=1 crypto rx=0x7fc28400e580 tx=0x7fc28400ea40 comp rx=0 tx=0).stop 2023-05-22T16:05:56.800 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.795+0000 7fc29c3e2640 1 -- 172.21.15.16:0/3587447736 shutdown_connections 2023-05-22T16:05:56.800 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.795+0000 7fc29c3e2640 1 --2- 172.21.15.16:0/3587447736 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc2940caf70 0x7fc2940d2600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:56.800 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.795+0000 7fc29c3e2640 1 --2- 172.21.15.16:0/3587447736 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc2940cab00 0x7fc2940d20c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:56.800 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.795+0000 7fc29c3e2640 1 --2- 172.21.15.16:0/3587447736 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fc27806b760 0x7fc27806db40 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:56.800 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.795+0000 7fc29c3e2640 1 --2- 172.21.15.16:0/3587447736 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc280004d10 0x7fc29415a7a0 unknown :-1 s=CLOSED pgs=2584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:56.800 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.795+0000 7fc29c3e2640 1 -- 172.21.15.16:0/3587447736 >> 172.21.15.16:0/3587447736 conn(0x7fc294071850 msgr2=0x7fc2940cd240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:05:56.800 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.795+0000 7fc29c3e2640 1 -- 172.21.15.16:0/3587447736 shutdown_connections 2023-05-22T16:05:56.801 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.795+0000 7fc29c3e2640 1 -- 172.21.15.16:0/3587447736 wait complete. 2023-05-22T16:05:56.814 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-05-22T16:05:56.932 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.931+0000 7f8959433640 1 Processor -- start 2023-05-22T16:05:56.933 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.931+0000 7f8959433640 1 -- start start 2023-05-22T16:05:56.934 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.931+0000 7f8959433640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f895406cc80 0x7f895406f630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:56.934 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.931+0000 7f8959433640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8954071b40 con 0x7f89540afb50 2023-05-22T16:05:56.934 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.931+0000 7f8959433640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89540ca150 con 0x7f8954154170 2023-05-22T16:05:56.934 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.931+0000 7f8959433640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89540ca9f0 con 0x7f895406cc80 2023-05-22T16:05:56.934 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.931+0000 7f89527fc640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f895406cc80 0x7f895406f630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:56.934 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.931+0000 7f89527fc640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f895406cc80 0x7f895406f630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38674/0 (socket says 172.21.15.16:38674) 2023-05-22T16:05:56.934 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.931+0000 7f89527fc640 1 -- 172.21.15.16:0/2821291807 learned_addr learned my addr 172.21.15.16:0/2821291807 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:05:56.935 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.931+0000 7f89527fc640 1 -- 172.21.15.16:0/2821291807 >> v1:172.21.15.62:6789/0 conn(0x7f8954154170 legacy=0x7f895406be90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:05:56.935 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.931+0000 7f89527fc640 1 -- 172.21.15.16:0/2821291807 >> v1:172.21.15.16:6789/0 conn(0x7f89540afb50 legacy=0x7f895415a0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:56.935 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.931+0000 7f89527fc640 1 -- 172.21.15.16:0/2821291807 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89540ca430 con 0x7f895406cc80 2023-05-22T16:05:56.937 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.935+0000 7f89527fc640 1 --2- 172.21.15.16:0/2821291807 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f895406cc80 0x7f895406f630 secure :-1 s=READY pgs=2586 cs=0 l=1 rev1=1 crypto rx=0x7f8948009870 tx=0x7f894801c510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ec6c2e3fb30594f server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:56.937 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.935+0000 7f8951ffb640 1 -- 172.21.15.16:0/2821291807 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89480aa670 con 0x7f895406cc80 2023-05-22T16:05:56.937 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.935+0000 7f8951ffb640 1 -- 172.21.15.16:0/2821291807 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89480aa7d0 con 0x7f895406cc80 2023-05-22T16:05:56.937 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.935+0000 7f8951ffb640 1 -- 172.21.15.16:0/2821291807 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89480aab60 con 0x7f895406cc80 2023-05-22T16:05:56.938 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.935+0000 7f8959433640 1 -- 172.21.15.16:0/2821291807 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f895406cc80 msgr2=0x7f895406f630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:56.938 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.935+0000 7f8959433640 1 --2- 172.21.15.16:0/2821291807 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f895406cc80 0x7f895406f630 secure :-1 s=READY pgs=2586 cs=0 l=1 rev1=1 crypto rx=0x7f8948009870 tx=0x7f894801c510 comp rx=0 tx=0).stop 2023-05-22T16:05:56.938 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.935+0000 7f8959433640 1 -- 172.21.15.16:0/2821291807 shutdown_connections 2023-05-22T16:05:56.939 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.935+0000 7f8959433640 1 --2- 172.21.15.16:0/2821291807 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f895406cc80 0x7f895406f630 unknown :-1 s=CLOSED pgs=2586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:56.939 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.935+0000 7f8959433640 1 -- 172.21.15.16:0/2821291807 >> 172.21.15.16:0/2821291807 conn(0x7f89540cab50 msgr2=0x7f89540caf50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:05:56.939 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.935+0000 7f8959433640 1 -- 172.21.15.16:0/2821291807 shutdown_connections 2023-05-22T16:05:56.939 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.935+0000 7f8959433640 1 -- 172.21.15.16:0/2821291807 wait complete. 2023-05-22T16:05:56.940 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.935+0000 7f8959433640 1 Processor -- start 2023-05-22T16:05:56.940 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.935+0000 7f8959433640 1 -- start start 2023-05-22T16:05:56.940 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.935+0000 7f8959433640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f895406cc80 0x7f895414bcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:56.940 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.935+0000 7f8959433640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f89540afb50 0x7f89540b19d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:56.940 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.935+0000 7f8959433640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8954154170 0x7f895414b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:56.940 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.935+0000 7f8959433640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89540b1f10 con 0x7f89540afb50 2023-05-22T16:05:56.941 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.935+0000 7f8959433640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8954158270 con 0x7f8954154170 2023-05-22T16:05:56.941 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.935+0000 7f8959433640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8954158550 con 0x7f895406cc80 2023-05-22T16:05:56.941 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.935+0000 7f89527fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f89540afb50 0x7f89540b19d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:56.941 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.935+0000 7f89527fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f89540afb50 0x7f89540b19d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38678/0 (socket says 172.21.15.16:38678) 2023-05-22T16:05:56.941 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.935+0000 7f89527fc640 1 -- 172.21.15.16:0/597836425 learned_addr learned my addr 172.21.15.16:0/597836425 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:05:56.942 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.935+0000 7f89527fc640 1 -- 172.21.15.16:0/597836425 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f895406cc80 msgr2=0x7f895414bcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:56.943 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.935+0000 7f8952ffd640 1 --2- 172.21.15.16:0/597836425 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f895406cc80 0x7f895414bcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:56.943 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.939+0000 7f89537fe640 1 --2- 172.21.15.16:0/597836425 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8954154170 0x7f895414b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:56.943 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.939+0000 7f89527fc640 1 --2- 172.21.15.16:0/597836425 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f895406cc80 0x7f895414bcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:56.943 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.939+0000 7f89527fc640 1 -- 172.21.15.16:0/597836425 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8954154170 msgr2=0x7f895414b480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:56.944 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.939+0000 7f89527fc640 1 --2- 172.21.15.16:0/597836425 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8954154170 0x7f895414b480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:56.944 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.939+0000 7f89527fc640 1 -- 172.21.15.16:0/597836425 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89480094d0 con 0x7f89540afb50 2023-05-22T16:05:56.944 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.939+0000 7f8952ffd640 1 --2- 172.21.15.16:0/597836425 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f895406cc80 0x7f895414bcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T16:05:56.944 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.939+0000 7f89527fc640 1 --2- 172.21.15.16:0/597836425 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f89540afb50 0x7f89540b19d0 secure :-1 s=READY pgs=2587 cs=0 l=1 rev1=1 crypto rx=0x7f8948019120 tx=0x7f894801ad50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:56.944 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.939+0000 7f893b7fe640 1 -- 172.21.15.16:0/597836425 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89480b38c0 con 0x7f89540afb50 2023-05-22T16:05:56.944 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.939+0000 7f893b7fe640 1 -- 172.21.15.16:0/597836425 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89480b3a20 con 0x7f89540afb50 2023-05-22T16:05:56.944 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.939+0000 7f8959433640 1 -- 172.21.15.16:0/597836425 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8954154540 con 0x7f89540afb50 2023-05-22T16:05:56.945 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.939+0000 7f893b7fe640 1 -- 172.21.15.16:0/597836425 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89480b3dd0 con 0x7f89540afb50 2023-05-22T16:05:56.945 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.939+0000 7f8959433640 1 -- 172.21.15.16:0/597836425 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f895406c5a0 con 0x7f89540afb50 2023-05-22T16:05:56.945 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.939+0000 7f8959433640 1 -- 172.21.15.16:0/597836425 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89540c9d90 con 0x7f89540afb50 2023-05-22T16:05:56.945 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.939+0000 7f893b7fe640 1 -- 172.21.15.16:0/597836425 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 99) v1 ==== 84068+0+0 (secure 0 0 0) 0x7f89480c6390 con 0x7f89540afb50 2023-05-22T16:05:56.945 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.939+0000 7f893b7fe640 1 --2- 172.21.15.16:0/597836425 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f89400651d0 0x7f8940067680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:56.945 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.939+0000 7f893b7fe640 1 -- 172.21.15.16:0/597836425 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(198..198 src has 1..198) v4 ==== 7083+0+0 (secure 0 0 0) 0x7f894811d670 con 0x7f89540afb50 2023-05-22T16:05:56.945 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.939+0000 7f8952ffd640 1 --2- 172.21.15.16:0/597836425 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f89400651d0 0x7f8940067680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:56.946 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.939+0000 7f8952ffd640 1 --2- 172.21.15.16:0/597836425 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f89400651d0 0x7f8940067680 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f89541658e0 tx=0x7f8944009040 comp rx=0 tx=0).ready entity=mgr.8195 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:56.946 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:56.943+0000 7f893b7fe640 1 -- 172.21.15.16:0/597836425 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f89480eb1d0 con 0x7f89540afb50 2023-05-22T16:05:57.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.095+0000 7f8959433640 1 -- 172.21.15.16:0/597836425 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8954073830 con 0x7f89540afb50 2023-05-22T16:05:57.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.099+0000 7f893b7fe640 1 -- 172.21.15.16:0/597836425 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v198) v1 ==== 74+0+19062 (secure 0 0 0) 0x7f89480ee7f0 con 0x7f89540afb50 2023-05-22T16:05:57.102 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T16:05:57.103 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":198,"fsid":"b2719834-7847-4d98-90a6-f65d57f85c49","created":"2023-05-22T15:26:25.259314+0000","modified":"2023-05-22T16:05:55.761209+0000","last_up_change":"2023-05-22T15:26:36.448524+0000","last_in_change":"2023-05-22T15:26:27.675668+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":25,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-05-22T15:26:38.019781+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-05-22T15:26:43.570514+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_metadata","create_time":"2023-05-22T16:05:20.485167+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"195","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":25,"pool_name":"cephfs_data","create_time":"2023-05-22T16:05:20.900001+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"196","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9e7ff854-b25f-4cbb-a5c2-1799d9c53d2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":37021},{"type":"v1","addr":"172.21.15.16:6825","nonce":37021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":37021},{"type":"v1","addr":"172.21.15.16:6827","nonce":37021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":37021},{"type":"v1","addr":"172.21.15.16:6831","nonce":37021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":37021},{"type":"v1","addr":"172.21.15.16:6829","nonce":37021}]},"public_addr":"172.21.15.16:6825/37021","cluster_addr":"172.21.15.16:6827/37021","heartbeat_back_addr":"172.21.15.16:6831/37021","heartbeat_front_addr":"172.21.15.16:6829/37021","state":["exists","up"]},{"osd":1,"uuid":"4702fbc9-a927-4565-aca5-41dc69cc8264","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":37008},{"type":"v1","addr":"172.21.15.16:6801","nonce":37008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":37008},{"type":"v1","addr":"172.21.15.16:6803","nonce":37008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":37008},{"type":"v1","addr":"172.21.15.16:6807","nonce":37008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":37008},{"type":"v1","addr":"172.21.15.16:6805","nonce":37008}]},"public_addr":"172.21.15.16:6801/37008","cluster_addr":"172.21.15.16:6803/37008","heartbeat_back_addr":"172.21.15.16:6807/37008","heartbeat_front_addr":"172.21.15.16:6805/37008","state":["exists","up"]},{"osd":2,"uuid":"1e7bba46-4eba-47ef-bb81-5e2d371638ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":37009},{"type":"v1","addr":"172.21.15.16:6809","nonce":37009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":37009},{"type":"v1","addr":"172.21.15.16:6811","nonce":37009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":37009},{"type":"v1","addr":"172.21.15.16:6815","nonce":37009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":37009},{"type":"v1","addr":"172.21.15.16:6813","nonce":37009}]},"public_addr":"172.21.15.16:6809/37009","cluster_addr":"172.21.15.16:6811/37009","heartbeat_back_addr":"172.21.15.16:6815/37009","heartbeat_front_addr":"172.21.15.16:6813/37009","state":["exists","up"]},{"osd":3,"uuid":"383cacc0-5a88-4904-990e-498ca59ef48a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":37020},{"type":"v1","addr":"172.21.15.16:6817","nonce":37020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":37020},{"type":"v1","addr":"172.21.15.16:6819","nonce":37020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":37020},{"type":"v1","addr":"172.21.15.16:6823","nonce":37020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":37020},{"type":"v1","addr":"172.21.15.16:6821","nonce":37020}]},"public_addr":"172.21.15.16:6817/37020","cluster_addr":"172.21.15.16:6819/37020","heartbeat_back_addr":"172.21.15.16:6823/37020","heartbeat_front_addr":"172.21.15.16:6821/37020","state":["exists","up"]},{"osd":4,"uuid":"970500ef-95b7-42c1-b080-eed4fae7013d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":37353},{"type":"v1","addr":"172.21.15.62:6817","nonce":37353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6819","nonce":37353},{"type":"v1","addr":"172.21.15.62:6822","nonce":37353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37353},{"type":"v1","addr":"172.21.15.62:6831","nonce":37353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37353},{"type":"v1","addr":"172.21.15.62:6826","nonce":37353}]},"public_addr":"172.21.15.62:6817/37353","cluster_addr":"172.21.15.62:6822/37353","heartbeat_back_addr":"172.21.15.62:6831/37353","heartbeat_front_addr":"172.21.15.62:6826/37353","state":["exists","up"]},{"osd":5,"uuid":"022d67db-d4d8-445b-b1b9-8b446a00f5c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":37352},{"type":"v1","addr":"172.21.15.62:6801","nonce":37352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":37352},{"type":"v1","addr":"172.21.15.62:6803","nonce":37352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":37352},{"type":"v1","addr":"172.21.15.62:6807","nonce":37352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":37352},{"type":"v1","addr":"172.21.15.62:6805","nonce":37352}]},"public_addr":"172.21.15.62:6801/37352","cluster_addr":"172.21.15.62:6803/37352","heartbeat_back_addr":"172.21.15.62:6807/37352","heartbeat_front_addr":"172.21.15.62:6805/37352","state":["exists","up"]},{"osd":6,"uuid":"42bd372b-6688-4a1b-8407-e16e8c266e33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":37354},{"type":"v1","addr":"172.21.15.62:6820","nonce":37354}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":37354},{"type":"v1","addr":"172.21.15.62:6823","nonce":37354}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6829","nonce":37354},{"type":"v1","addr":"172.21.15.62:6830","nonce":37354}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6825","nonce":37354},{"type":"v1","addr":"172.21.15.62:6827","nonce":37354}]},"public_addr":"172.21.15.62:6820/37354","cluster_addr":"172.21.15.62:6823/37354","heartbeat_back_addr":"172.21.15.62:6830/37354","heartbeat_front_addr":"172.21.15.62:6827/37354","state":["exists","up"]},{"osd":7,"uuid":"ac70b2a3-f41b-4bf2-b227-7f2cbb55426d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37355},{"type":"v1","addr":"172.21.15.62:6809","nonce":37355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37355},{"type":"v1","addr":"172.21.15.62:6811","nonce":37355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":37355},{"type":"v1","addr":"172.21.15.62:6815","nonce":37355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":37355},{"type":"v1","addr":"172.21.15.62:6813","nonce":37355}]},"public_addr":"172.21.15.62:6809/37355","cluster_addr":"172.21.15.62:6811/37355","heartbeat_back_addr":"172.21.15.62:6815/37355","heartbeat_front_addr":"172.21.15.62:6813/37355","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.368896+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.107273+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.276888+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.287502+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.124791+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.251937+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.255679+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.232797+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:0/3256922077":"2023-05-23T16:05:55.761117+0000","172.21.15.16:0/1351897283":"2023-05-23T16:05:55.761117+0000","172.21.15.16:0/168959634":"2023-05-23T16:05:55.761117+0000","172.21.15.16:0/3781682956":"2023-05-23T16:05:55.761117+0000","172.21.15.16:0/1599584790":"2023-05-23T16:05:55.761117+0000","172.21.15.16:0/1961044213":"2023-05-23T16:05:35.647562+0000","172.21.15.16:0/238292614":"2023-05-23T16:05:35.647562+0000","172.21.15.16:0/1185672517":"2023-05-23T16:05:55.761117+0000","172.21.15.16:0/2111070480":"2023-05-23T16:05:55.761117+0000","172.21.15.16:6833/65890":"2023-05-23T16:05:35.647562+0000","172.21.15.16:0/1148602276":"2023-05-23T16:05:35.647562+0000","172.21.15.16:6832/65890":"2023-05-23T16:05:35.647562+0000","172.21.15.16:0/456460873":"2023-05-23T16:05:35.647562+0000","172.21.15.16:0/2766506027":"2023-05-23T16:05:35.647562+0000","172.21.15.16:0/2214029832":"2023-05-23T16:05:35.647562+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":17,"snaps":[{"begin":2,"length":2}]},{"pool":21,"snaps":[{"begin":25,"length":9}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-05-22T16:05:57.104 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.099+0000 7f8959433640 1 -- 172.21.15.16:0/597836425 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f89400651d0 msgr2=0x7f8940067680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:57.104 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.099+0000 7f8959433640 1 --2- 172.21.15.16:0/597836425 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f89400651d0 0x7f8940067680 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f89541658e0 tx=0x7f8944009040 comp rx=0 tx=0).stop 2023-05-22T16:05:57.104 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.099+0000 7f8959433640 1 -- 172.21.15.16:0/597836425 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f89540afb50 msgr2=0x7f89540b19d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:57.105 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.099+0000 7f8959433640 1 --2- 172.21.15.16:0/597836425 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f89540afb50 0x7f89540b19d0 secure :-1 s=READY pgs=2587 cs=0 l=1 rev1=1 crypto rx=0x7f8948019120 tx=0x7f894801ad50 comp rx=0 tx=0).stop 2023-05-22T16:05:57.105 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.099+0000 7f8959433640 1 -- 172.21.15.16:0/597836425 shutdown_connections 2023-05-22T16:05:57.105 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.099+0000 7f8959433640 1 --2- 172.21.15.16:0/597836425 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8954154170 0x7f895414b480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:57.105 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.099+0000 7f8959433640 1 --2- 172.21.15.16:0/597836425 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f89400651d0 0x7f8940067680 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:57.105 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.099+0000 7f8959433640 1 --2- 172.21.15.16:0/597836425 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f89540afb50 0x7f89540b19d0 unknown :-1 s=CLOSED pgs=2587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:57.105 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.099+0000 7f8959433640 1 --2- 172.21.15.16:0/597836425 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f895406cc80 0x7f895414bcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:57.105 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.099+0000 7f8959433640 1 -- 172.21.15.16:0/597836425 >> 172.21.15.16:0/597836425 conn(0x7f89540cab50 msgr2=0x7f89541515a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:05:57.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.103+0000 7f8959433640 1 -- 172.21.15.16:0/597836425 shutdown_connections 2023-05-22T16:05:57.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.103+0000 7f8959433640 1 -- 172.21.15.16:0/597836425 wait complete. 2023-05-22T16:05:57.123 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-05-22T16:05:57.124 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-05-22T16:05:57.248 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.243+0000 7fcce0cb8640 1 Processor -- start 2023-05-22T16:05:57.249 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.247+0000 7fcce0cb8640 1 -- start start 2023-05-22T16:05:57.250 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.247+0000 7fcce0cb8640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fccdc06c8e0 0x7fccdc15cf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:57.250 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.247+0000 7fcce0cb8640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fccdc158be0 0x7fccdc06ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:57.250 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.247+0000 7fcce0cb8640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fccdc15d480 con 0x7fccdc0c8840 2023-05-22T16:05:57.250 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.247+0000 7fcce0cb8640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fccdc06d1f0 con 0x7fccdc06c8e0 2023-05-22T16:05:57.250 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.247+0000 7fcce0cb8640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fccdc0c8e80 con 0x7fccdc158be0 2023-05-22T16:05:57.250 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.247+0000 7fccda59c640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fccdc06c8e0 0x7fccdc15cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:57.251 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.247+0000 7fccda59c640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fccdc06c8e0 0x7fccdc15cf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38694/0 (socket says 172.21.15.16:38694) 2023-05-22T16:05:57.251 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.247+0000 7fccda59c640 1 -- 172.21.15.16:0/2987784676 learned_addr learned my addr 172.21.15.16:0/2987784676 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:05:57.251 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.247+0000 7fccd9d9b640 1 --2- 172.21.15.16:0/2987784676 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fccdc158be0 0x7fccdc06ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:57.251 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.247+0000 7fccda59c640 1 -- 172.21.15.16:0/2987784676 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fccdc158be0 msgr2=0x7fccdc06ccb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:05:57.252 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.247+0000 7fccda59c640 1 --2- 172.21.15.16:0/2987784676 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fccdc158be0 0x7fccdc06ccb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:57.252 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.247+0000 7fccda59c640 1 -- 172.21.15.16:0/2987784676 >> v1:172.21.15.62:6789/0 conn(0x7fccdc0c8840 legacy=0x7fccdc0afb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:57.252 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.247+0000 7fccda59c640 1 -- 172.21.15.16:0/2987784676 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccdc0c9160 con 0x7fccdc06c8e0 2023-05-22T16:05:57.253 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.247+0000 7fccda59c640 1 --2- 172.21.15.16:0/2987784676 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fccdc06c8e0 0x7fccdc15cf40 secure :-1 s=READY pgs=2588 cs=0 l=1 rev1=1 crypto rx=0x7fcccc009870 tx=0x7fcccc01c540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e4d2161ba612d10 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:57.253 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.247+0000 7fccd959a640 1 -- 172.21.15.16:0/2987784676 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcccc0ab600 con 0x7fccdc06c8e0 2023-05-22T16:05:57.254 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.251+0000 7fccd959a640 1 -- 172.21.15.16:0/2987784676 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcccc0a7050 con 0x7fccdc06c8e0 2023-05-22T16:05:57.254 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.251+0000 7fccd959a640 1 -- 172.21.15.16:0/2987784676 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcccc0ab980 con 0x7fccdc06c8e0 2023-05-22T16:05:57.254 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.251+0000 7fcce0cb8640 1 -- 172.21.15.16:0/2987784676 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fccdc06c8e0 msgr2=0x7fccdc15cf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:57.254 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.251+0000 7fcce0cb8640 1 --2- 172.21.15.16:0/2987784676 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fccdc06c8e0 0x7fccdc15cf40 secure :-1 s=READY pgs=2588 cs=0 l=1 rev1=1 crypto rx=0x7fcccc009870 tx=0x7fcccc01c540 comp rx=0 tx=0).stop 2023-05-22T16:05:57.254 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.251+0000 7fcce0cb8640 1 -- 172.21.15.16:0/2987784676 shutdown_connections 2023-05-22T16:05:57.254 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.251+0000 7fcce0cb8640 1 --2- 172.21.15.16:0/2987784676 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fccdc158be0 0x7fccdc06ccb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:57.254 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.251+0000 7fcce0cb8640 1 --2- 172.21.15.16:0/2987784676 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fccdc06c8e0 0x7fccdc15cf40 unknown :-1 s=CLOSED pgs=2588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:57.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.251+0000 7fcce0cb8640 1 -- 172.21.15.16:0/2987784676 >> 172.21.15.16:0/2987784676 conn(0x7fccdc0cab50 msgr2=0x7fccdc0caf50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:05:57.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.251+0000 7fcce0cb8640 1 -- 172.21.15.16:0/2987784676 shutdown_connections 2023-05-22T16:05:57.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.251+0000 7fcce0cb8640 1 -- 172.21.15.16:0/2987784676 wait complete. 2023-05-22T16:05:57.256 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.251+0000 7fcce0cb8640 1 Processor -- start 2023-05-22T16:05:57.256 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.251+0000 7fcce0cb8640 1 -- start start 2023-05-22T16:05:57.257 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.251+0000 7fcce0cb8640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fccdc06c8e0 0x7fccdc155a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:57.257 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.251+0000 7fcce0cb8640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fccdc0c8840 0x7fccdc151b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:57.257 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.251+0000 7fcce0cb8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fccdc158be0 0x7fccdc154370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:57.257 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.251+0000 7fcce0cb8640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccdc152430 con 0x7fccdc158be0 2023-05-22T16:05:57.257 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.251+0000 7fcce0cb8640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccdc152710 con 0x7fccdc0c8840 2023-05-22T16:05:57.257 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.251+0000 7fcce0cb8640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fccdc1529f0 con 0x7fccdc06c8e0 2023-05-22T16:05:57.257 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.251+0000 7fccdad9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fccdc158be0 0x7fccdc154370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:57.258 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.251+0000 7fccdad9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fccdc158be0 0x7fccdc154370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38696/0 (socket says 172.21.15.16:38696) 2023-05-22T16:05:57.258 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.251+0000 7fccdad9d640 1 -- 172.21.15.16:0/2630089861 learned_addr learned my addr 172.21.15.16:0/2630089861 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:05:57.258 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.251+0000 7fccda59c640 1 --2- 172.21.15.16:0/2630089861 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fccdc06c8e0 0x7fccdc155a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:57.258 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.251+0000 7fccd9d9b640 1 --2- 172.21.15.16:0/2630089861 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fccdc0c8840 0x7fccdc151b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:57.258 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.251+0000 7fccdad9d640 1 -- 172.21.15.16:0/2630089861 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fccdc06c8e0 msgr2=0x7fccdc155a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:57.258 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.251+0000 7fccdad9d640 1 --2- 172.21.15.16:0/2630089861 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fccdc06c8e0 0x7fccdc155a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:57.258 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.251+0000 7fccdad9d640 1 -- 172.21.15.16:0/2630089861 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fccdc0c8840 msgr2=0x7fccdc151b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:57.258 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.251+0000 7fccdad9d640 1 --2- 172.21.15.16:0/2630089861 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fccdc0c8840 0x7fccdc151b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:57.259 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.251+0000 7fccdad9d640 1 -- 172.21.15.16:0/2630089861 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcccc0094d0 con 0x7fccdc158be0 2023-05-22T16:05:57.259 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.251+0000 7fccdad9d640 1 --2- 172.21.15.16:0/2630089861 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fccdc158be0 0x7fccdc154370 secure :-1 s=READY pgs=2589 cs=0 l=1 rev1=1 crypto rx=0x7fccd400c770 tx=0x7fccd400cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:57.259 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.251+0000 7fcccaffd640 1 -- 172.21.15.16:0/2630089861 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccd4003b80 con 0x7fccdc158be0 2023-05-22T16:05:57.259 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.251+0000 7fcccaffd640 1 -- 172.21.15.16:0/2630089861 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccd4003ce0 con 0x7fccdc158be0 2023-05-22T16:05:57.259 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.251+0000 7fcccaffd640 1 -- 172.21.15.16:0/2630089861 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccd400aa90 con 0x7fccdc158be0 2023-05-22T16:05:57.259 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.251+0000 7fcce0cb8640 1 -- 172.21.15.16:0/2630089861 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fccdc06b7d0 con 0x7fccdc158be0 2023-05-22T16:05:57.259 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.251+0000 7fcce0cb8640 1 -- 172.21.15.16:0/2630089861 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fccdc0c9a00 con 0x7fccdc158be0 2023-05-22T16:05:57.260 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.255+0000 7fcce0cb8640 1 -- 172.21.15.16:0/2630089861 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fccdc06b960 con 0x7fccdc158be0 2023-05-22T16:05:57.260 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.255+0000 7fcccaffd640 1 -- 172.21.15.16:0/2630089861 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 99) v1 ==== 84068+0+0 (secure 0 0 0) 0x7fccd400abf0 con 0x7fccdc158be0 2023-05-22T16:05:57.260 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.255+0000 7fcccaffd640 1 --2- 172.21.15.16:0/2630089861 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fccbc0651d0 0x7fccbc067680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:57.261 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.255+0000 7fccda59c640 1 --2- 172.21.15.16:0/2630089861 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fccbc0651d0 0x7fccbc067680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:57.261 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.255+0000 7fcccaffd640 1 -- 172.21.15.16:0/2630089861 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(198..198 src has 1..198) v4 ==== 7083+0+0 (secure 0 0 0) 0x7fccd4082610 con 0x7fccdc158be0 2023-05-22T16:05:57.261 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.255+0000 7fccda59c640 1 --2- 172.21.15.16:0/2630089861 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fccbc0651d0 0x7fccbc067680 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fccdc0b0ca0 tx=0x7fcccc008040 comp rx=0 tx=0).ready entity=mgr.8195 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:57.262 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.259+0000 7fcccaffd640 1 -- 172.21.15.16:0/2630089861 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fccd40501d0 con 0x7fccdc158be0 2023-05-22T16:05:57.450 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.447+0000 7fcce0cb8640 1 -- 172.21.15.16:0/2630089861 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fccdc073830 con 0x7fccdc158be0 2023-05-22T16:05:57.450 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.447+0000 7fcccaffd640 1 -- 172.21.15.16:0/2630089861 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 81 v81) v1 ==== 94+0+4653 (secure 0 0 0) 0x7fccd40537f0 con 0x7fccdc158be0 2023-05-22T16:05:57.451 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T16:05:57.451 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":81,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7979,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/1053276484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":1053276484},{"type":"v1","addr":"172.21.15.62:6835","nonce":1053276484}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":76},{"gid":8983,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6833/1372552690","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":1372552690},{"type":"v1","addr":"172.21.15.62:6833","nonce":1372552690}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":76},{"gid":10776,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/982492476","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":982492476},{"type":"v1","addr":"172.21.15.16:6835","nonce":982492476}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":76}],"filesystems":[{"mdsmap":{"epoch":81,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-05-22T16:05:21.922031+0000","modified":"2023-05-22T16:05:24.940696+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10782},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10782":{"gid":10782,"name":"c","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.16:6837/1705367575","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":1705367575},{"type":"v1","addr":"172.21.15.16:6837","nonce":1705367575}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[25],"metadata_pool":24,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":11}]} 2023-05-22T16:05:57.451 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 81 2023-05-22T16:05:57.453 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.451+0000 7fcce0cb8640 1 -- 172.21.15.16:0/2630089861 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fccbc0651d0 msgr2=0x7fccbc067680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:57.453 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.451+0000 7fcce0cb8640 1 --2- 172.21.15.16:0/2630089861 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fccbc0651d0 0x7fccbc067680 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fccdc0b0ca0 tx=0x7fcccc008040 comp rx=0 tx=0).stop 2023-05-22T16:05:57.453 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.451+0000 7fcce0cb8640 1 -- 172.21.15.16:0/2630089861 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fccdc158be0 msgr2=0x7fccdc154370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:57.453 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.451+0000 7fcce0cb8640 1 --2- 172.21.15.16:0/2630089861 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fccdc158be0 0x7fccdc154370 secure :-1 s=READY pgs=2589 cs=0 l=1 rev1=1 crypto rx=0x7fccd400c770 tx=0x7fccd400cc30 comp rx=0 tx=0).stop 2023-05-22T16:05:57.453 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.451+0000 7fcce0cb8640 1 -- 172.21.15.16:0/2630089861 shutdown_connections 2023-05-22T16:05:57.453 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.451+0000 7fcce0cb8640 1 --2- 172.21.15.16:0/2630089861 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fccbc0651d0 0x7fccbc067680 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:57.454 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.451+0000 7fcce0cb8640 1 --2- 172.21.15.16:0/2630089861 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fccdc158be0 0x7fccdc154370 unknown :-1 s=CLOSED pgs=2589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:57.454 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.451+0000 7fcce0cb8640 1 --2- 172.21.15.16:0/2630089861 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fccdc0c8840 0x7fccdc151b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:57.454 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.451+0000 7fcce0cb8640 1 --2- 172.21.15.16:0/2630089861 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fccdc06c8e0 0x7fccdc155a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:57.454 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.451+0000 7fcce0cb8640 1 -- 172.21.15.16:0/2630089861 >> 172.21.15.16:0/2630089861 conn(0x7fccdc0cab50 msgr2=0x7fccdc0caf20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:05:57.454 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.451+0000 7fcce0cb8640 1 -- 172.21.15.16:0/2630089861 shutdown_connections 2023-05-22T16:05:57.454 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.451+0000 7fcce0cb8640 1 -- 172.21.15.16:0/2630089861 wait complete. 2023-05-22T16:05:57.471 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-05-22T16:05:57.595 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.591+0000 7fed78b9c640 1 Processor -- start 2023-05-22T16:05:57.595 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.591+0000 7fed78b9c640 1 -- start start 2023-05-22T16:05:57.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.591+0000 7fed78b9c640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fed740cfa20 0x7fed740d0020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:57.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.591+0000 7fed78b9c640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fed740d0560 0x7fed740d20c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:57.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.591+0000 7fed78b9c640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed740d0990 con 0x7fed740d2600 2023-05-22T16:05:57.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.591+0000 7fed78b9c640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed7415c4f0 con 0x7fed740cfa20 2023-05-22T16:05:57.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.591+0000 7fed78b9c640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed740cf190 con 0x7fed740d0560 2023-05-22T16:05:57.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.591+0000 7fed72d9d640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fed740d2600 0x7fed740b23c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:45802/0 (socket says 172.21.15.16:45802) 2023-05-22T16:05:57.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.591+0000 7fed72d9d640 1 -- 172.21.15.16:0/2482831345 learned_addr learned my addr 172.21.15.16:0/2482831345 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T16:05:57.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.591+0000 7fed71d9b640 1 --2- 172.21.15.16:0/2482831345 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fed740d0560 0x7fed740d20c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:57.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.591+0000 7fed7259c640 1 --2- 172.21.15.16:0/2482831345 >> v2:172.21.15.62:3300/0 conn(0x7fed740cfa20 0x7fed740d0020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:57.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.591+0000 7fed7159a640 1 -- 172.21.15.16:0/2482831345 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4195205947 0 0) 0x7fed740d0990 con 0x7fed740d2600 2023-05-22T16:05:57.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.591+0000 7fed7159a640 1 -- 172.21.15.16:0/2482831345 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed5c003530 con 0x7fed740d2600 2023-05-22T16:05:57.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.591+0000 7fed7159a640 1 -- 172.21.15.16:0/2482831345 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 448214039 0 0) 0x7fed5c003530 con 0x7fed740d2600 2023-05-22T16:05:57.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.591+0000 7fed7159a640 1 -- 172.21.15.16:0/2482831345 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fed740d0560 msgr2=0x7fed740d20c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:05:57.600 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.591+0000 7fed7159a640 1 --2- 172.21.15.16:0/2482831345 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fed740d0560 0x7fed740d20c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:57.600 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.591+0000 7fed7159a640 1 -- 172.21.15.16:0/2482831345 >> v2:172.21.15.62:3300/0 conn(0x7fed740cfa20 msgr2=0x7fed740d0020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:05:57.600 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.591+0000 7fed7159a640 1 --2- 172.21.15.16:0/2482831345 >> v2:172.21.15.62:3300/0 conn(0x7fed740cfa20 0x7fed740d0020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:57.600 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.591+0000 7fed7159a640 1 -- 172.21.15.16:0/2482831345 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed740d3360 con 0x7fed740d2600 2023-05-22T16:05:57.600 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.591+0000 7fed7159a640 1 -- 172.21.15.16:0/2482831345 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7fed6c001a50 con 0x7fed740d2600 2023-05-22T16:05:57.600 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.591+0000 7fed7159a640 1 -- 172.21.15.16:0/2482831345 >> v1:172.21.15.16:6789/0 conn(0x7fed740d2600 legacy=0x7fed740b23c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:57.600 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.595+0000 7fed7159a640 1 --2- 172.21.15.16:0/2482831345 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed5c003980 0x7fed5c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:57.601 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.595+0000 7fed7159a640 1 --2- 172.21.15.16:0/2482831345 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fed740d0560 0x7fed5c0047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:57.601 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.595+0000 7fed7159a640 1 --2- 172.21.15.16:0/2482831345 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fed5c004d10 0x7fed5c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:57.601 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.595+0000 7fed7159a640 1 -- 172.21.15.16:0/2482831345 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed7415c4f0 con 0x7fed5c003980 2023-05-22T16:05:57.601 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.595+0000 7fed7159a640 1 -- 172.21.15.16:0/2482831345 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed740cf190 con 0x7fed5c004d10 2023-05-22T16:05:57.601 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.595+0000 7fed7159a640 1 -- 172.21.15.16:0/2482831345 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed5c00b990 con 0x7fed740d0560 2023-05-22T16:05:57.601 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.595+0000 7fed7259c640 1 --2- 172.21.15.16:0/2482831345 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed5c003980 0x7fed5c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:57.601 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.595+0000 7fed72d9d640 1 --2- 172.21.15.16:0/2482831345 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fed5c004d10 0x7fed5c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:57.602 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.595+0000 7fed71d9b640 1 --2- 172.21.15.16:0/2482831345 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fed740d0560 0x7fed5c0047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:57.602 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.595+0000 7fed7259c640 1 -- 172.21.15.16:0/2482831345 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fed740d0560 msgr2=0x7fed5c0047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:05:57.602 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.595+0000 7fed7259c640 1 --2- 172.21.15.16:0/2482831345 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fed740d0560 0x7fed5c0047d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:57.602 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.595+0000 7fed7259c640 1 -- 172.21.15.16:0/2482831345 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fed5c004d10 msgr2=0x7fed5c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:05:57.602 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.595+0000 7fed7259c640 1 --2- 172.21.15.16:0/2482831345 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fed5c004d10 0x7fed5c005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:57.602 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.595+0000 7fed7259c640 1 -- 172.21.15.16:0/2482831345 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed5c003530 con 0x7fed5c003980 2023-05-22T16:05:57.602 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.595+0000 7fed72d9d640 1 --2- 172.21.15.16:0/2482831345 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fed5c004d10 0x7fed5c005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T16:05:57.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.595+0000 7fed7259c640 1 --2- 172.21.15.16:0/2482831345 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed5c003980 0x7fed5c003d90 secure :-1 s=READY pgs=2591 cs=0 l=1 rev1=1 crypto rx=0x7fed64009af0 tx=0x7fed6401c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2feb551746cd909 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:57.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.595+0000 7fed7159a640 1 -- 172.21.15.16:0/2482831345 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed640059a0 con 0x7fed5c003980 2023-05-22T16:05:57.606 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.595+0000 7fed7159a640 1 -- 172.21.15.16:0/2482831345 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed64005b00 con 0x7fed5c003980 2023-05-22T16:05:57.606 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.595+0000 7fed7159a640 1 -- 172.21.15.16:0/2482831345 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed64006030 con 0x7fed5c003980 2023-05-22T16:05:57.606 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.595+0000 7fed78b9c640 1 -- 172.21.15.16:0/2482831345 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed5c003980 msgr2=0x7fed5c003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:57.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.595+0000 7fed78b9c640 1 --2- 172.21.15.16:0/2482831345 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed5c003980 0x7fed5c003d90 secure :-1 s=READY pgs=2591 cs=0 l=1 rev1=1 crypto rx=0x7fed64009af0 tx=0x7fed6401c9f0 comp rx=0 tx=0).stop 2023-05-22T16:05:57.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.595+0000 7fed72d9d640 1 -- 172.21.15.16:0/2482831345 reap_dead start 2023-05-22T16:05:57.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.595+0000 7fed78b9c640 1 -- 172.21.15.16:0/2482831345 shutdown_connections 2023-05-22T16:05:57.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.595+0000 7fed78b9c640 1 -- 172.21.15.16:0/2482831345 >> 172.21.15.16:0/2482831345 conn(0x7fed74071850 msgr2=0x7fed7409b830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:05:57.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.595+0000 7fed78b9c640 1 -- 172.21.15.16:0/2482831345 shutdown_connections 2023-05-22T16:05:57.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.595+0000 7fed78b9c640 1 -- 172.21.15.16:0/2482831345 wait complete. 2023-05-22T16:05:57.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.599+0000 7fed78b9c640 1 Processor -- start 2023-05-22T16:05:57.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.599+0000 7fed78b9c640 1 -- start start 2023-05-22T16:05:57.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.599+0000 7fed78b9c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed5c003980 0x7fed7415a810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:57.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.599+0000 7fed78b9c640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fed740b5b00 0x7fed740b5f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:57.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.599+0000 7fed78b9c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fed740c7ae0 0x7fed740c7f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:57.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.599+0000 7fed78b9c640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed740c9520 con 0x7fed5c003980 2023-05-22T16:05:57.609 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.599+0000 7fed78b9c640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed740b64b0 con 0x7fed740c7ae0 2023-05-22T16:05:57.609 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.599+0000 7fed78b9c640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed740c8880 con 0x7fed740b5b00 2023-05-22T16:05:57.609 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.599+0000 7fed7259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed5c003980 0x7fed7415a810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:57.609 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.599+0000 7fed7259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed5c003980 0x7fed7415a810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38702/0 (socket says 172.21.15.16:38702) 2023-05-22T16:05:57.609 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.599+0000 7fed7259c640 1 -- 172.21.15.16:0/1849028019 learned_addr learned my addr 172.21.15.16:0/1849028019 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:05:57.609 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.599+0000 7fed71d9b640 1 --2- 172.21.15.16:0/1849028019 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fed740b5b00 0x7fed740b5f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:57.609 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.599+0000 7fed7259c640 1 -- 172.21.15.16:0/1849028019 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fed740b5b00 msgr2=0x7fed740b5f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:57.609 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.599+0000 7fed72d9d640 1 --2- 172.21.15.16:0/1849028019 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fed740c7ae0 0x7fed740c7f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:57.610 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.599+0000 7fed7259c640 1 --2- 172.21.15.16:0/1849028019 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fed740b5b00 0x7fed740b5f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:57.610 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.599+0000 7fed7259c640 1 -- 172.21.15.16:0/1849028019 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fed740c7ae0 msgr2=0x7fed740c7f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:57.610 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.599+0000 7fed7259c640 1 --2- 172.21.15.16:0/1849028019 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fed740c7ae0 0x7fed740c7f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:57.610 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.599+0000 7fed7259c640 1 -- 172.21.15.16:0/1849028019 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed740d3080 con 0x7fed5c003980 2023-05-22T16:05:57.610 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.599+0000 7fed71d9b640 1 --2- 172.21.15.16:0/1849028019 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fed740b5b00 0x7fed740b5f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T16:05:57.610 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.599+0000 7fed72d9d640 1 --2- 172.21.15.16:0/1849028019 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fed740c7ae0 0x7fed740c7f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T16:05:57.610 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.599+0000 7fed7259c640 1 --2- 172.21.15.16:0/1849028019 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed5c003980 0x7fed7415a810 secure :-1 s=READY pgs=2592 cs=0 l=1 rev1=1 crypto rx=0x7fed64009af0 tx=0x7fed5c005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:57.611 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.599+0000 7fed62ffd640 1 -- 172.21.15.16:0/1849028019 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed640068d0 con 0x7fed5c003980 2023-05-22T16:05:57.611 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.599+0000 7fed62ffd640 1 -- 172.21.15.16:0/1849028019 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed64005c70 con 0x7fed5c003980 2023-05-22T16:05:57.611 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.599+0000 7fed78b9c640 1 -- 172.21.15.16:0/1849028019 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed740d2da0 con 0x7fed5c003980 2023-05-22T16:05:57.611 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.599+0000 7fed62ffd640 1 -- 172.21.15.16:0/1849028019 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed64004020 con 0x7fed5c003980 2023-05-22T16:05:57.611 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.599+0000 7fed78b9c640 1 -- 172.21.15.16:0/1849028019 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed740ccb40 con 0x7fed5c003980 2023-05-22T16:05:57.611 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.599+0000 7fed78b9c640 1 -- 172.21.15.16:0/1849028019 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed74073830 con 0x7fed5c003980 2023-05-22T16:05:57.611 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.603+0000 7fed62ffd640 1 -- 172.21.15.16:0/1849028019 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 99) v1 ==== 84068+0+0 (secure 0 0 0) 0x7fed64004180 con 0x7fed5c003980 2023-05-22T16:05:57.611 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.603+0000 7fed62ffd640 1 --2- 172.21.15.16:0/1849028019 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fed54065200 0x7fed540676b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:57.612 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.603+0000 7fed62ffd640 1 -- 172.21.15.16:0/1849028019 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(198..198 src has 1..198) v4 ==== 7083+0+0 (secure 0 0 0) 0x7fed6411d1a0 con 0x7fed5c003980 2023-05-22T16:05:57.612 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.603+0000 7fed71d9b640 1 --2- 172.21.15.16:0/1849028019 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fed54065200 0x7fed540676b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:57.612 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.603+0000 7fed71d9b640 1 --2- 172.21.15.16:0/1849028019 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fed54065200 0x7fed540676b0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fed740cea70 tx=0x7fed68008040 comp rx=0 tx=0).ready entity=mgr.8195 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:57.612 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.607+0000 7fed62ffd640 1 -- 172.21.15.16:0/1849028019 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fed640ead60 con 0x7fed5c003980 2023-05-22T16:05:57.790 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.787+0000 7fed78b9c640 1 -- 172.21.15.16:0/1849028019 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fed7406a900 con 0x7fed5c003980 2023-05-22T16:05:57.791 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.787+0000 7fed62ffd640 1 -- 172.21.15.16:0/1849028019 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 81 v81) v1 ==== 94+0+4653 (secure 0 0 0) 0x7fed640ee380 con 0x7fed5c003980 2023-05-22T16:05:57.792 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T16:05:57.792 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":81,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7979,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/1053276484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":1053276484},{"type":"v1","addr":"172.21.15.62:6835","nonce":1053276484}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":76},{"gid":8983,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6833/1372552690","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":1372552690},{"type":"v1","addr":"172.21.15.62:6833","nonce":1372552690}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":76},{"gid":10776,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/982492476","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":982492476},{"type":"v1","addr":"172.21.15.16:6835","nonce":982492476}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":76}],"filesystems":[{"mdsmap":{"epoch":81,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-05-22T16:05:21.922031+0000","modified":"2023-05-22T16:05:24.940696+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10782},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10782":{"gid":10782,"name":"c","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.16:6837/1705367575","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":1705367575},{"type":"v1","addr":"172.21.15.16:6837","nonce":1705367575}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[25],"metadata_pool":24,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":11}]} 2023-05-22T16:05:57.792 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 81 2023-05-22T16:05:57.794 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.791+0000 7fed78b9c640 1 -- 172.21.15.16:0/1849028019 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fed54065200 msgr2=0x7fed540676b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:57.794 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.791+0000 7fed78b9c640 1 --2- 172.21.15.16:0/1849028019 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fed54065200 0x7fed540676b0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fed740cea70 tx=0x7fed68008040 comp rx=0 tx=0).stop 2023-05-22T16:05:57.794 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.791+0000 7fed78b9c640 1 -- 172.21.15.16:0/1849028019 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed5c003980 msgr2=0x7fed7415a810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:57.794 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.791+0000 7fed78b9c640 1 --2- 172.21.15.16:0/1849028019 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed5c003980 0x7fed7415a810 secure :-1 s=READY pgs=2592 cs=0 l=1 rev1=1 crypto rx=0x7fed64009af0 tx=0x7fed5c005800 comp rx=0 tx=0).stop 2023-05-22T16:05:57.795 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.791+0000 7fed78b9c640 1 -- 172.21.15.16:0/1849028019 shutdown_connections 2023-05-22T16:05:57.795 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.791+0000 7fed78b9c640 1 --2- 172.21.15.16:0/1849028019 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fed740c7ae0 0x7fed740c7f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:57.795 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.791+0000 7fed78b9c640 1 --2- 172.21.15.16:0/1849028019 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fed740b5b00 0x7fed740b5f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:57.795 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.791+0000 7fed78b9c640 1 --2- 172.21.15.16:0/1849028019 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fed54065200 0x7fed540676b0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:57.796 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.791+0000 7fed78b9c640 1 --2- 172.21.15.16:0/1849028019 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed5c003980 0x7fed7415a810 unknown :-1 s=CLOSED pgs=2592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:57.796 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.791+0000 7fed78b9c640 1 -- 172.21.15.16:0/1849028019 >> 172.21.15.16:0/1849028019 conn(0x7fed74071850 msgr2=0x7fed7415b790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:05:57.796 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.791+0000 7fed78b9c640 1 -- 172.21.15.16:0/1849028019 shutdown_connections 2023-05-22T16:05:57.797 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.791+0000 7fed78b9c640 1 -- 172.21.15.16:0/1849028019 wait complete. 2023-05-22T16:05:57.809 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-05-22T16:05:57.932 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.927+0000 7f1c1962d640 1 Processor -- start 2023-05-22T16:05:57.932 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.927+0000 7f1c1962d640 1 -- start start 2023-05-22T16:05:57.933 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.931+0000 7f1c1962d640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f1c14154170 0x7f1c1406f630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:57.933 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.931+0000 7f1c1962d640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f1c1406cc80 0x7f1c1406be90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:57.933 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.931+0000 7f1c1962d640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c14071b40 con 0x7f1c1406c3d0 2023-05-22T16:05:57.933 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.931+0000 7f1c1962d640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c140ca9f0 con 0x7f1c14154170 2023-05-22T16:05:57.934 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.931+0000 7f1c1962d640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c140ca150 con 0x7f1c1406cc80 2023-05-22T16:05:57.934 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.931+0000 7f1c12ffd640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f1c14154170 0x7f1c1406f630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:57.934 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.931+0000 7f1c12ffd640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f1c14154170 0x7f1c1406f630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38718/0 (socket says 172.21.15.16:38718) 2023-05-22T16:05:57.934 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.931+0000 7f1c12ffd640 1 -- 172.21.15.16:0/3520767103 learned_addr learned my addr 172.21.15.16:0/3520767103 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:05:57.934 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.931+0000 7f1c12ffd640 1 -- 172.21.15.16:0/3520767103 >> v2:172.21.15.62:3300/0 conn(0x7f1c1406cc80 msgr2=0x7f1c1406be90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:05:57.935 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.931+0000 7f1c127fc640 1 --2- 172.21.15.16:0/3520767103 >> v2:172.21.15.62:3300/0 conn(0x7f1c1406cc80 0x7f1c1406be90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:57.935 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.931+0000 7f1c12ffd640 1 --2- 172.21.15.16:0/3520767103 >> v2:172.21.15.62:3300/0 conn(0x7f1c1406cc80 0x7f1c1406be90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:57.935 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.931+0000 7f1c12ffd640 1 -- 172.21.15.16:0/3520767103 >> v1:172.21.15.62:6789/0 conn(0x7f1c1406c3d0 legacy=0x7f1c140afb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:05:57.935 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.931+0000 7f1c12ffd640 1 -- 172.21.15.16:0/3520767103 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c140ca430 con 0x7f1c14154170 2023-05-22T16:05:57.936 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.931+0000 7f1c12ffd640 1 --2- 172.21.15.16:0/3520767103 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c14154170 0x7f1c1406f630 secure :-1 s=READY pgs=2593 cs=0 l=1 rev1=1 crypto rx=0x7f1c140cbc50 tx=0x7f1c0401c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=951954820f93580f server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:57.936 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.931+0000 7f1c11ffb640 1 -- 172.21.15.16:0/3520767103 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c040aa5f0 con 0x7f1c14154170 2023-05-22T16:05:57.936 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.931+0000 7f1c11ffb640 1 -- 172.21.15.16:0/3520767103 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c040aa750 con 0x7f1c14154170 2023-05-22T16:05:57.936 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.931+0000 7f1c11ffb640 1 -- 172.21.15.16:0/3520767103 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c040aaae0 con 0x7f1c14154170 2023-05-22T16:05:57.937 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.935+0000 7f1c1962d640 1 -- 172.21.15.16:0/3520767103 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c14154170 msgr2=0x7f1c1406f630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:57.937 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.935+0000 7f1c1962d640 1 --2- 172.21.15.16:0/3520767103 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c14154170 0x7f1c1406f630 secure :-1 s=READY pgs=2593 cs=0 l=1 rev1=1 crypto rx=0x7f1c140cbc50 tx=0x7f1c0401c490 comp rx=0 tx=0).stop 2023-05-22T16:05:57.938 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.935+0000 7f1c1962d640 1 -- 172.21.15.16:0/3520767103 shutdown_connections 2023-05-22T16:05:57.938 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.935+0000 7f1c1962d640 1 --2- 172.21.15.16:0/3520767103 >> v2:172.21.15.62:3300/0 conn(0x7f1c1406cc80 0x7f1c1406be90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:57.938 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.935+0000 7f1c1962d640 1 --2- 172.21.15.16:0/3520767103 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c14154170 0x7f1c1406f630 unknown :-1 s=CLOSED pgs=2593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:57.938 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.935+0000 7f1c1962d640 1 -- 172.21.15.16:0/3520767103 >> 172.21.15.16:0/3520767103 conn(0x7f1c140cab50 msgr2=0x7f1c140caf50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:05:57.938 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.935+0000 7f1c1962d640 1 -- 172.21.15.16:0/3520767103 shutdown_connections 2023-05-22T16:05:57.938 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.935+0000 7f1c1962d640 1 -- 172.21.15.16:0/3520767103 wait complete. 2023-05-22T16:05:57.940 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.935+0000 7f1c1962d640 1 Processor -- start 2023-05-22T16:05:57.940 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.935+0000 7f1c1962d640 1 -- start start 2023-05-22T16:05:57.941 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.935+0000 7f1c1962d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c1406c3d0 0x7f1c1414ed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:57.941 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.935+0000 7f1c1962d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1c1406cc80 0x7f1c14147400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:57.941 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.935+0000 7f1c1962d640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1c14154170 0x7f1c14147940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:57.941 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.935+0000 7f1c1962d640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c14148420 con 0x7f1c1406c3d0 2023-05-22T16:05:57.941 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.935+0000 7f1c1962d640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c14147e80 con 0x7f1c1406cc80 2023-05-22T16:05:57.941 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.935+0000 7f1c1962d640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c14148080 con 0x7f1c14154170 2023-05-22T16:05:57.942 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.935+0000 7f1c12ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c1406c3d0 0x7f1c1414ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:57.942 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.935+0000 7f1c12ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c1406c3d0 0x7f1c1414ed40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38724/0 (socket says 172.21.15.16:38724) 2023-05-22T16:05:57.942 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.935+0000 7f1c12ffd640 1 -- 172.21.15.16:0/2606116251 learned_addr learned my addr 172.21.15.16:0/2606116251 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:05:57.942 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.935+0000 7f1c127fc640 1 --2- 172.21.15.16:0/2606116251 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1c1406cc80 0x7f1c14147400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:57.942 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.935+0000 7f1c137fe640 1 --2- 172.21.15.16:0/2606116251 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1c14154170 0x7f1c14147940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:57.942 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.935+0000 7f1c12ffd640 1 -- 172.21.15.16:0/2606116251 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1c14154170 msgr2=0x7f1c14147940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:57.942 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.935+0000 7f1c12ffd640 1 --2- 172.21.15.16:0/2606116251 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1c14154170 0x7f1c14147940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:57.943 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.935+0000 7f1c12ffd640 1 -- 172.21.15.16:0/2606116251 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1c1406cc80 msgr2=0x7f1c14147400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:57.943 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.935+0000 7f1c12ffd640 1 --2- 172.21.15.16:0/2606116251 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1c1406cc80 0x7f1c14147400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:57.943 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.935+0000 7f1c12ffd640 1 -- 172.21.15.16:0/2606116251 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c040094d0 con 0x7f1c1406c3d0 2023-05-22T16:05:57.943 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.935+0000 7f1c127fc640 1 --2- 172.21.15.16:0/2606116251 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1c1406cc80 0x7f1c14147400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T16:05:57.943 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.935+0000 7f1c12ffd640 1 --2- 172.21.15.16:0/2606116251 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c1406c3d0 0x7f1c1414ed40 secure :-1 s=READY pgs=2594 cs=0 l=1 rev1=1 crypto rx=0x7f1c0401a0c0 tx=0x7f1c0401acd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:57.943 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.935+0000 7f1bfb7fe640 1 -- 172.21.15.16:0/2606116251 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c040b3800 con 0x7f1c1406c3d0 2023-05-22T16:05:57.943 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.935+0000 7f1bfb7fe640 1 -- 172.21.15.16:0/2606116251 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c040b3980 con 0x7f1c1406c3d0 2023-05-22T16:05:57.943 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.935+0000 7f1bfb7fe640 1 -- 172.21.15.16:0/2606116251 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c040b3d30 con 0x7f1c1406c3d0 2023-05-22T16:05:57.944 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.935+0000 7f1c1962d640 1 -- 172.21.15.16:0/2606116251 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c140c9d90 con 0x7f1c1406c3d0 2023-05-22T16:05:57.944 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.935+0000 7f1c1962d640 1 -- 172.21.15.16:0/2606116251 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c14154540 con 0x7f1c1406c3d0 2023-05-22T16:05:57.944 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.939+0000 7f1c1962d640 1 -- 172.21.15.16:0/2606116251 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c140c9f20 con 0x7f1c1406c3d0 2023-05-22T16:05:57.945 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.939+0000 7f1bfb7fe640 1 -- 172.21.15.16:0/2606116251 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 99) v1 ==== 84068+0+0 (secure 0 0 0) 0x7f1c040b2380 con 0x7f1c1406c3d0 2023-05-22T16:05:57.945 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.939+0000 7f1bfb7fe640 1 --2- 172.21.15.16:0/2606116251 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f1c000651d0 0x7f1c00067680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:57.945 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.939+0000 7f1bfb7fe640 1 -- 172.21.15.16:0/2606116251 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(198..198 src has 1..198) v4 ==== 7083+0+0 (secure 0 0 0) 0x7f1c040a7070 con 0x7f1c1406c3d0 2023-05-22T16:05:57.945 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.939+0000 7f1c127fc640 1 --2- 172.21.15.16:0/2606116251 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f1c000651d0 0x7f1c00067680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:57.946 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.939+0000 7f1c127fc640 1 --2- 172.21.15.16:0/2606116251 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f1c000651d0 0x7f1c00067680 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f1c140b4130 tx=0x7f1c08008040 comp rx=0 tx=0).ready entity=mgr.8195 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:57.946 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:57.943+0000 7f1bfb7fe640 1 -- 172.21.15.16:0/2606116251 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f1c040ea950 con 0x7f1c1406c3d0 2023-05-22T16:05:58.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:58.095+0000 7f1c1962d640 1 -- 172.21.15.16:0/2606116251 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1c14148560 con 0x7f1c1406c3d0 2023-05-22T16:05:58.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:58.095+0000 7f1bfb7fe640 1 -- 172.21.15.16:0/2606116251 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v198) v1 ==== 74+0+19062 (secure 0 0 0) 0x7f1c14148560 con 0x7f1c1406c3d0 2023-05-22T16:05:58.101 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T16:05:58.101 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":198,"fsid":"b2719834-7847-4d98-90a6-f65d57f85c49","created":"2023-05-22T15:26:25.259314+0000","modified":"2023-05-22T16:05:55.761209+0000","last_up_change":"2023-05-22T15:26:36.448524+0000","last_in_change":"2023-05-22T15:26:27.675668+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":25,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-05-22T15:26:38.019781+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-05-22T15:26:43.570514+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_metadata","create_time":"2023-05-22T16:05:20.485167+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"195","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":25,"pool_name":"cephfs_data","create_time":"2023-05-22T16:05:20.900001+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"196","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9e7ff854-b25f-4cbb-a5c2-1799d9c53d2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":37021},{"type":"v1","addr":"172.21.15.16:6825","nonce":37021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":37021},{"type":"v1","addr":"172.21.15.16:6827","nonce":37021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":37021},{"type":"v1","addr":"172.21.15.16:6831","nonce":37021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":37021},{"type":"v1","addr":"172.21.15.16:6829","nonce":37021}]},"public_addr":"172.21.15.16:6825/37021","cluster_addr":"172.21.15.16:6827/37021","heartbeat_back_addr":"172.21.15.16:6831/37021","heartbeat_front_addr":"172.21.15.16:6829/37021","state":["exists","up"]},{"osd":1,"uuid":"4702fbc9-a927-4565-aca5-41dc69cc8264","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":37008},{"type":"v1","addr":"172.21.15.16:6801","nonce":37008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":37008},{"type":"v1","addr":"172.21.15.16:6803","nonce":37008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":37008},{"type":"v1","addr":"172.21.15.16:6807","nonce":37008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":37008},{"type":"v1","addr":"172.21.15.16:6805","nonce":37008}]},"public_addr":"172.21.15.16:6801/37008","cluster_addr":"172.21.15.16:6803/37008","heartbeat_back_addr":"172.21.15.16:6807/37008","heartbeat_front_addr":"172.21.15.16:6805/37008","state":["exists","up"]},{"osd":2,"uuid":"1e7bba46-4eba-47ef-bb81-5e2d371638ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":37009},{"type":"v1","addr":"172.21.15.16:6809","nonce":37009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":37009},{"type":"v1","addr":"172.21.15.16:6811","nonce":37009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":37009},{"type":"v1","addr":"172.21.15.16:6815","nonce":37009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":37009},{"type":"v1","addr":"172.21.15.16:6813","nonce":37009}]},"public_addr":"172.21.15.16:6809/37009","cluster_addr":"172.21.15.16:6811/37009","heartbeat_back_addr":"172.21.15.16:6815/37009","heartbeat_front_addr":"172.21.15.16:6813/37009","state":["exists","up"]},{"osd":3,"uuid":"383cacc0-5a88-4904-990e-498ca59ef48a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":37020},{"type":"v1","addr":"172.21.15.16:6817","nonce":37020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":37020},{"type":"v1","addr":"172.21.15.16:6819","nonce":37020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":37020},{"type":"v1","addr":"172.21.15.16:6823","nonce":37020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":37020},{"type":"v1","addr":"172.21.15.16:6821","nonce":37020}]},"public_addr":"172.21.15.16:6817/37020","cluster_addr":"172.21.15.16:6819/37020","heartbeat_back_addr":"172.21.15.16:6823/37020","heartbeat_front_addr":"172.21.15.16:6821/37020","state":["exists","up"]},{"osd":4,"uuid":"970500ef-95b7-42c1-b080-eed4fae7013d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":37353},{"type":"v1","addr":"172.21.15.62:6817","nonce":37353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6819","nonce":37353},{"type":"v1","addr":"172.21.15.62:6822","nonce":37353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37353},{"type":"v1","addr":"172.21.15.62:6831","nonce":37353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37353},{"type":"v1","addr":"172.21.15.62:6826","nonce":37353}]},"public_addr":"172.21.15.62:6817/37353","cluster_addr":"172.21.15.62:6822/37353","heartbeat_back_addr":"172.21.15.62:6831/37353","heartbeat_front_addr":"172.21.15.62:6826/37353","state":["exists","up"]},{"osd":5,"uuid":"022d67db-d4d8-445b-b1b9-8b446a00f5c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":37352},{"type":"v1","addr":"172.21.15.62:6801","nonce":37352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":37352},{"type":"v1","addr":"172.21.15.62:6803","nonce":37352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":37352},{"type":"v1","addr":"172.21.15.62:6807","nonce":37352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":37352},{"type":"v1","addr":"172.21.15.62:6805","nonce":37352}]},"public_addr":"172.21.15.62:6801/37352","cluster_addr":"172.21.15.62:6803/37352","heartbeat_back_addr":"172.21.15.62:6807/37352","heartbeat_front_addr":"172.21.15.62:6805/37352","state":["exists","up"]},{"osd":6,"uuid":"42bd372b-6688-4a1b-8407-e16e8c266e33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":37354},{"type":"v1","addr":"172.21.15.62:6820","nonce":37354}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":37354},{"type":"v1","addr":"172.21.15.62:6823","nonce":37354}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6829","nonce":37354},{"type":"v1","addr":"172.21.15.62:6830","nonce":37354}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6825","nonce":37354},{"type":"v1","addr":"172.21.15.62:6827","nonce":37354}]},"public_addr":"172.21.15.62:6820/37354","cluster_addr":"172.21.15.62:6823/37354","heartbeat_back_addr":"172.21.15.62:6830/37354","heartbeat_front_addr":"172.21.15.62:6827/37354","state":["exists","up"]},{"osd":7,"uuid":"ac70b2a3-f41b-4bf2-b227-7f2cbb55426d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37355},{"type":"v1","addr":"172.21.15.62:6809","nonce":37355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37355},{"type":"v1","addr":"172.21.15.62:6811","nonce":37355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":37355},{"type":"v1","addr":"172.21.15.62:6815","nonce":37355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":37355},{"type":"v1","addr":"172.21.15.62:6813","nonce":37355}]},"public_addr":"172.21.15.62:6809/37355","cluster_addr":"172.21.15.62:6811/37355","heartbeat_back_addr":"172.21.15.62:6815/37355","heartbeat_front_addr":"172.21.15.62:6813/37355","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.368896+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.107273+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.276888+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.287502+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.124791+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.251937+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.255679+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.232797+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:0/3256922077":"2023-05-23T16:05:55.761117+0000","172.21.15.16:0/1351897283":"2023-05-23T16:05:55.761117+0000","172.21.15.16:0/168959634":"2023-05-23T16:05:55.761117+0000","172.21.15.16:0/3781682956":"2023-05-23T16:05:55.761117+0000","172.21.15.16:0/1599584790":"2023-05-23T16:05:55.761117+0000","172.21.15.16:0/1961044213":"2023-05-23T16:05:35.647562+0000","172.21.15.16:0/238292614":"2023-05-23T16:05:35.647562+0000","172.21.15.16:0/1185672517":"2023-05-23T16:05:55.761117+0000","172.21.15.16:0/2111070480":"2023-05-23T16:05:55.761117+0000","172.21.15.16:6833/65890":"2023-05-23T16:05:35.647562+0000","172.21.15.16:0/1148602276":"2023-05-23T16:05:35.647562+0000","172.21.15.16:6832/65890":"2023-05-23T16:05:35.647562+0000","172.21.15.16:0/456460873":"2023-05-23T16:05:35.647562+0000","172.21.15.16:0/2766506027":"2023-05-23T16:05:35.647562+0000","172.21.15.16:0/2214029832":"2023-05-23T16:05:35.647562+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":17,"snaps":[{"begin":2,"length":2}]},{"pool":21,"snaps":[{"begin":25,"length":9}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-05-22T16:05:58.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:58.099+0000 7f1c1962d640 1 -- 172.21.15.16:0/2606116251 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f1c000651d0 msgr2=0x7f1c00067680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:58.103 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:58.099+0000 7f1c1962d640 1 --2- 172.21.15.16:0/2606116251 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f1c000651d0 0x7f1c00067680 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f1c140b4130 tx=0x7f1c08008040 comp rx=0 tx=0).stop 2023-05-22T16:05:58.103 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:58.099+0000 7f1c1962d640 1 -- 172.21.15.16:0/2606116251 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c1406c3d0 msgr2=0x7f1c1414ed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:58.103 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:58.099+0000 7f1c1962d640 1 --2- 172.21.15.16:0/2606116251 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c1406c3d0 0x7f1c1414ed40 secure :-1 s=READY pgs=2594 cs=0 l=1 rev1=1 crypto rx=0x7f1c0401a0c0 tx=0x7f1c0401acd0 comp rx=0 tx=0).stop 2023-05-22T16:05:58.103 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:58.099+0000 7f1c1962d640 1 -- 172.21.15.16:0/2606116251 shutdown_connections 2023-05-22T16:05:58.103 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:58.099+0000 7f1c1962d640 1 --2- 172.21.15.16:0/2606116251 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1c14154170 0x7f1c14147940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:58.104 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:58.099+0000 7f1c1962d640 1 --2- 172.21.15.16:0/2606116251 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1c1406cc80 0x7f1c14147400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:58.104 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:58.099+0000 7f1c1962d640 1 --2- 172.21.15.16:0/2606116251 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f1c000651d0 0x7f1c00067680 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:58.104 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:58.099+0000 7f1c1962d640 1 --2- 172.21.15.16:0/2606116251 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c1406c3d0 0x7f1c1414ed40 unknown :-1 s=CLOSED pgs=2594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:58.104 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:58.099+0000 7f1c1962d640 1 -- 172.21.15.16:0/2606116251 >> 172.21.15.16:0/2606116251 conn(0x7f1c140cab50 msgr2=0x7f1c14157d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:05:58.105 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:58.099+0000 7f1c1962d640 1 -- 172.21.15.16:0/2606116251 shutdown_connections 2023-05-22T16:05:58.105 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:58.099+0000 7f1c1962d640 1 -- 172.21.15.16:0/2606116251 wait complete. 2023-05-22T16:05:58.120 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-05-22T16:05:58.246 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:58.243+0000 7fb33a467640 1 Processor -- start 2023-05-22T16:05:58.246 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:58.243+0000 7fb33a467640 1 -- start start 2023-05-22T16:05:58.247 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:58.243+0000 7fb33a467640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb334158a50 0x7fb33406c370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:58.247 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:58.243+0000 7fb33a467640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb334158e20 0x7fb3340ca0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:58.248 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:58.243+0000 7fb33a467640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fb3340ccb90 0x7fb3340cd070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:58.248 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:58.243+0000 7fb33a467640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb33406a240 con 0x7fb334158a50 2023-05-22T16:05:58.248 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:58.243+0000 7fb33a467640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3340ca9e0 con 0x7fb3340ccb90 2023-05-22T16:05:58.248 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:58.243+0000 7fb33a467640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3340698a0 con 0x7fb334158e20 2023-05-22T16:05:58.248 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:58.243+0000 7fb333fff640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb334158a50 0x7fb33406c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:58.248 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:58.243+0000 7fb333fff640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb334158a50 0x7fb33406c370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38736/0 (socket says 172.21.15.16:38736) 2023-05-22T16:05:58.249 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:58.243+0000 7fb333fff640 1 -- 172.21.15.16:0/1383083270 learned_addr learned my addr 172.21.15.16:0/1383083270 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:05:58.249 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:58.243+0000 7fb3337fe640 1 --2- 172.21.15.16:0/1383083270 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb334158e20 0x7fb3340ca0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:58.249 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:58.243+0000 7fb338a04640 1 --2- 172.21.15.16:0/1383083270 >> v2:172.21.15.62:3300/0 conn(0x7fb3340ccb90 0x7fb3340cd070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:58.249 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:58.243+0000 7fb333fff640 1 -- 172.21.15.16:0/1383083270 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb334158e20 msgr2=0x7fb3340ca0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:05:58.249 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:58.243+0000 7fb333fff640 1 --2- 172.21.15.16:0/1383083270 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb334158e20 0x7fb3340ca0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:58.249 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:58.243+0000 7fb333fff640 1 -- 172.21.15.16:0/1383083270 >> v2:172.21.15.62:3300/0 conn(0x7fb3340ccb90 msgr2=0x7fb3340cd070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:05:58.249 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:58.243+0000 7fb333fff640 1 --2- 172.21.15.16:0/1383083270 >> v2:172.21.15.62:3300/0 conn(0x7fb3340ccb90 0x7fb3340cd070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:58.250 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:58.243+0000 7fb333fff640 1 -- 172.21.15.16:0/1383083270 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb334069b80 con 0x7fb334158a50 2023-05-22T16:05:58.251 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:58.247+0000 7fb333fff640 1 --2- 172.21.15.16:0/1383083270 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb334158a50 0x7fb33406c370 secure :-1 s=READY pgs=2595 cs=0 l=1 rev1=1 crypto rx=0x7fb33406fe40 tx=0x7fb32c01c9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6022fff5331a06ad server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:58.251 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:58.247+0000 7fb332ffd640 1 -- 172.21.15.16:0/1383083270 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb32c0a5730 con 0x7fb334158a50 2023-05-22T16:05:58.251 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:58.247+0000 7fb332ffd640 1 -- 172.21.15.16:0/1383083270 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb32c0a5890 con 0x7fb334158a50 2023-05-22T16:05:58.251 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:58.247+0000 7fb332ffd640 1 -- 172.21.15.16:0/1383083270 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb32c0a5c20 con 0x7fb334158a50 2023-05-22T16:05:58.251 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:58.247+0000 7fb33a467640 1 -- 172.21.15.16:0/1383083270 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb334158a50 msgr2=0x7fb33406c370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:58.251 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:58.247+0000 7fb33a467640 1 --2- 172.21.15.16:0/1383083270 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb334158a50 0x7fb33406c370 secure :-1 s=READY pgs=2595 cs=0 l=1 rev1=1 crypto rx=0x7fb33406fe40 tx=0x7fb32c01c9a0 comp rx=0 tx=0).stop 2023-05-22T16:05:58.252 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:58.247+0000 7fb33a467640 1 -- 172.21.15.16:0/1383083270 shutdown_connections 2023-05-22T16:05:58.252 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:58.247+0000 7fb33a467640 1 --2- 172.21.15.16:0/1383083270 >> v2:172.21.15.62:3300/0 conn(0x7fb3340ccb90 0x7fb3340cd070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:58.252 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:58.247+0000 7fb33a467640 1 --2- 172.21.15.16:0/1383083270 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb334158e20 0x7fb3340ca0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:58.252 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:58.247+0000 7fb33a467640 1 --2- 172.21.15.16:0/1383083270 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb334158a50 0x7fb33406c370 unknown :-1 s=CLOSED pgs=2595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:58.252 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:58.247+0000 7fb33a467640 1 -- 172.21.15.16:0/1383083270 >> 172.21.15.16:0/1383083270 conn(0x7fb3340cea90 msgr2=0x7fb3340cee90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:05:58.252 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:58.247+0000 7fb33a467640 1 -- 172.21.15.16:0/1383083270 shutdown_connections 2023-05-22T16:05:58.252 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:58.247+0000 7fb33a467640 1 -- 172.21.15.16:0/1383083270 wait complete. 2023-05-22T16:05:58.254 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:58.247+0000 7fb33a467640 1 Processor -- start 2023-05-22T16:05:58.254 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:58.247+0000 7fb33a467640 1 -- start start 2023-05-22T16:05:58.254 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:58.251+0000 7fb33a467640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb3340ccb90 0x7fb3341377a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:58.254 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:58.251+0000 7fb33a467640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb334158a50 0x7fb334138880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:58.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:58.251+0000 7fb33a467640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb334158e20 0x7fb334150c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:58.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:58.251+0000 7fb33a467640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3341507b0 con 0x7fb3340ccb90 2023-05-22T16:05:58.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:58.251+0000 7fb33a467640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3341578b0 con 0x7fb334158a50 2023-05-22T16:05:58.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:58.251+0000 7fb33a467640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb334157b90 con 0x7fb334158e20 2023-05-22T16:05:58.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:58.251+0000 7fb333fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb3340ccb90 0x7fb3341377a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:58.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:58.251+0000 7fb333fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb3340ccb90 0x7fb3341377a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38740/0 (socket says 172.21.15.16:38740) 2023-05-22T16:05:58.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:58.251+0000 7fb333fff640 1 -- 172.21.15.16:0/2780684057 learned_addr learned my addr 172.21.15.16:0/2780684057 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:05:58.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:58.251+0000 7fb338a04640 1 --2- 172.21.15.16:0/2780684057 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb334158e20 0x7fb334150c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:58.256 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:58.251+0000 7fb3337fe640 1 --2- 172.21.15.16:0/2780684057 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb334158a50 0x7fb334138880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:58.256 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:58.251+0000 7fb333fff640 1 -- 172.21.15.16:0/2780684057 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb334158e20 msgr2=0x7fb334150c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:58.256 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:58.251+0000 7fb333fff640 1 --2- 172.21.15.16:0/2780684057 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb334158e20 0x7fb334150c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:58.256 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:58.251+0000 7fb333fff640 1 -- 172.21.15.16:0/2780684057 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb334158a50 msgr2=0x7fb334138880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:58.256 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:58.251+0000 7fb333fff640 1 --2- 172.21.15.16:0/2780684057 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb334158a50 0x7fb334138880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:58.256 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:58.251+0000 7fb333fff640 1 -- 172.21.15.16:0/2780684057 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb32c009b30 con 0x7fb3340ccb90 2023-05-22T16:05:58.256 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:58.251+0000 7fb333fff640 1 --2- 172.21.15.16:0/2780684057 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb3340ccb90 0x7fb3341377a0 secure :-1 s=READY pgs=2596 cs=0 l=1 rev1=1 crypto rx=0x7fb32c0af040 tx=0x7fb32c019740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:58.257 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:58.251+0000 7fb330ff9640 1 -- 172.21.15.16:0/2780684057 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb32c0b8670 con 0x7fb3340ccb90 2023-05-22T16:05:58.257 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:58.251+0000 7fb330ff9640 1 -- 172.21.15.16:0/2780684057 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb32c0b87f0 con 0x7fb3340ccb90 2023-05-22T16:05:58.257 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:58.251+0000 7fb33a467640 1 -- 172.21.15.16:0/2780684057 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3340ca780 con 0x7fb3340ccb90 2023-05-22T16:05:58.257 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:58.251+0000 7fb330ff9640 1 -- 172.21.15.16:0/2780684057 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb32c0b8ba0 con 0x7fb3340ccb90 2023-05-22T16:05:58.257 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:58.251+0000 7fb33a467640 1 -- 172.21.15.16:0/2780684057 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3340ca5f0 con 0x7fb3340ccb90 2023-05-22T16:05:58.257 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:58.251+0000 7fb33a467640 1 -- 172.21.15.16:0/2780684057 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb33406ae70 con 0x7fb3340ccb90 2023-05-22T16:05:58.258 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:58.251+0000 7fb330ff9640 1 -- 172.21.15.16:0/2780684057 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 99) v1 ==== 84068+0+0 (secure 0 0 0) 0x7fb32c0b8d00 con 0x7fb3340ccb90 2023-05-22T16:05:58.258 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:58.251+0000 7fb330ff9640 1 --2- 172.21.15.16:0/2780684057 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fb314065200 0x7fb3140676b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:58.258 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:58.251+0000 7fb330ff9640 1 -- 172.21.15.16:0/2780684057 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(198..198 src has 1..198) v4 ==== 7083+0+0 (secure 0 0 0) 0x7fb32c11d8b0 con 0x7fb3340ccb90 2023-05-22T16:05:58.258 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:58.251+0000 7fb3337fe640 1 --2- 172.21.15.16:0/2780684057 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fb314065200 0x7fb3140676b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:58.258 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:58.251+0000 7fb3337fe640 1 --2- 172.21.15.16:0/2780684057 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fb314065200 0x7fb3140676b0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fb33415bf10 tx=0x7fb31c008040 comp rx=0 tx=0).ready entity=mgr.8195 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:58.259 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:58.255+0000 7fb330ff9640 1 -- 172.21.15.16:0/2780684057 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fb32c0eb520 con 0x7fb3340ccb90 2023-05-22T16:05:58.451 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:58.447+0000 7fb33a467640 1 -- 172.21.15.16:0/2780684057 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fb334073050 con 0x7fb3340ccb90 2023-05-22T16:05:58.798 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:58.795+0000 7fb330ff9640 1 -- 172.21.15.16:0/2780684057 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mgrmap(e 100) v1 ==== 84332+0+0 (secure 0 0 0) 0x7fb32c0eb8f0 con 0x7fb3340ccb90 2023-05-22T16:05:59.240 INFO:tasks.ceph.mon.a.smithi016.stderr:2023-05-22T16:05:59.239+0000 7f95dd48a640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-05-22T16:05:59.251 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:59.247+0000 7fb330ff9640 1 -- 172.21.15.16:0/2780684057 <== mon.0 v2:172.21.15.16:3300/0 8 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v82) v1 ==== 161+0+0 (secure 0 0 0) 0x7fb32c0eb520 con 0x7fb3340ccb90 2023-05-22T16:05:59.251 INFO:teuthology.orchestra.run.smithi016.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-05-22T16:05:59.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:59.251+0000 7fb33a467640 1 -- 172.21.15.16:0/2780684057 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fb314065200 msgr2=0x7fb3140676b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:59.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:59.251+0000 7fb33a467640 1 --2- 172.21.15.16:0/2780684057 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fb314065200 0x7fb3140676b0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fb33415bf10 tx=0x7fb31c008040 comp rx=0 tx=0).stop 2023-05-22T16:05:59.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:59.251+0000 7fb33a467640 1 -- 172.21.15.16:0/2780684057 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb3340ccb90 msgr2=0x7fb3341377a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:59.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:59.251+0000 7fb33a467640 1 --2- 172.21.15.16:0/2780684057 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb3340ccb90 0x7fb3341377a0 secure :-1 s=READY pgs=2596 cs=0 l=1 rev1=1 crypto rx=0x7fb32c0af040 tx=0x7fb32c019740 comp rx=0 tx=0).stop 2023-05-22T16:05:59.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:59.251+0000 7fb33a467640 1 -- 172.21.15.16:0/2780684057 shutdown_connections 2023-05-22T16:05:59.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:59.251+0000 7fb33a467640 1 --2- 172.21.15.16:0/2780684057 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb334158e20 0x7fb334150c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:59.256 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:59.251+0000 7fb33a467640 1 --2- 172.21.15.16:0/2780684057 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb334158a50 0x7fb334138880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:59.256 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:59.251+0000 7fb33a467640 1 --2- 172.21.15.16:0/2780684057 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fb314065200 0x7fb3140676b0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:59.256 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:59.251+0000 7fb33a467640 1 --2- 172.21.15.16:0/2780684057 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb3340ccb90 0x7fb3341377a0 unknown :-1 s=CLOSED pgs=2596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:59.256 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:59.251+0000 7fb33a467640 1 -- 172.21.15.16:0/2780684057 >> 172.21.15.16:0/2780684057 conn(0x7fb3340cea90 msgr2=0x7fb334155f30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:05:59.256 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:59.251+0000 7fb33a467640 1 -- 172.21.15.16:0/2780684057 shutdown_connections 2023-05-22T16:05:59.256 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:59.251+0000 7fb33a467640 1 -- 172.21.15.16:0/2780684057 wait complete. 2023-05-22T16:05:59.257 INFO:tasks.ceph.mds.c.smithi016.stderr: -4227> 2023-05-22T16:05:23.448+0000 7f4768e9f640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-05-22T16:05:59.279 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-05-22T16:05:59.298 INFO:tasks.ceph.mds.c.smithi016.stdout:starting mds.c at 2023-05-22T16:05:59.403 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:59.399+0000 7f3fbb74b640 1 Processor -- start 2023-05-22T16:05:59.403 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:59.399+0000 7f3fbb74b640 1 -- start start 2023-05-22T16:05:59.404 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:59.399+0000 7f3fbb74b640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f3fb40ca880 0x7f3fb40cee10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:59.404 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:59.399+0000 7f3fbb74b640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3fb40cbdb0 0x7f3fb40cc230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:59.404 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:59.399+0000 7f3fbb74b640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fb406b5f0 con 0x7f3fb415f480 2023-05-22T16:05:59.404 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:59.399+0000 7f3fbb74b640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3fb406bf70 con 0x7f3fb40ca880 2023-05-22T16:05:59.405 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:59.399+0000 7f3fbb74b640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3fb40cad10 con 0x7f3fb40cbdb0 2023-05-22T16:05:59.405 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:59.399+0000 7f3fb94e7640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f3fb415f480 0x7f3fb40b3110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:49778/0 (socket says 172.21.15.16:49778) 2023-05-22T16:05:59.405 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:59.399+0000 7f3fb94e7640 1 -- 172.21.15.16:0/38951377 learned_addr learned my addr 172.21.15.16:0/38951377 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T16:05:59.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:59.399+0000 7f3fb9ce8640 1 --2- 172.21.15.16:0/38951377 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3fb40cbdb0 0x7f3fb40cc230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:59.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:59.399+0000 7f3fb8ce6640 1 --2- 172.21.15.16:0/38951377 >> v2:172.21.15.62:3300/0 conn(0x7f3fb40ca880 0x7f3fb40cee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:59.408 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:59.399+0000 7f3fabfff640 1 -- 172.21.15.16:0/38951377 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2744439015 0 0) 0x7f3fb406b5f0 con 0x7f3fb415f480 2023-05-22T16:05:59.408 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:59.403+0000 7f3fabfff640 1 -- 172.21.15.16:0/38951377 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3fa0003530 con 0x7f3fb415f480 2023-05-22T16:05:59.408 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:59.403+0000 7f3fabfff640 1 -- 172.21.15.16:0/38951377 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3372457204 0 0) 0x7f3fa0003530 con 0x7f3fb415f480 2023-05-22T16:05:59.408 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:59.403+0000 7f3fabfff640 1 -- 172.21.15.16:0/38951377 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3fb40cbdb0 msgr2=0x7f3fb40cc230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:05:59.408 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:59.403+0000 7f3fabfff640 1 --2- 172.21.15.16:0/38951377 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3fb40cbdb0 0x7f3fb40cc230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:59.408 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:59.403+0000 7f3fabfff640 1 -- 172.21.15.16:0/38951377 >> v2:172.21.15.62:3300/0 conn(0x7f3fb40ca880 msgr2=0x7f3fb40cee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:05:59.409 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:59.403+0000 7f3fabfff640 1 --2- 172.21.15.16:0/38951377 >> v2:172.21.15.62:3300/0 conn(0x7f3fb40ca880 0x7f3fb40cee10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:59.409 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:59.403+0000 7f3fabfff640 1 -- 172.21.15.16:0/38951377 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fb40caff0 con 0x7f3fb415f480 2023-05-22T16:05:59.409 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:59.403+0000 7f3fabfff640 1 -- 172.21.15.16:0/38951377 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7f3fb0001800 con 0x7f3fb415f480 2023-05-22T16:05:59.409 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:59.403+0000 7f3fabfff640 1 -- 172.21.15.16:0/38951377 >> v1:172.21.15.16:6789/0 conn(0x7f3fb415f480 legacy=0x7f3fb40b3110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:59.409 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:59.403+0000 7f3fabfff640 1 --2- 172.21.15.16:0/38951377 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3fa0003980 0x7f3fa0003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:59.409 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:59.403+0000 7f3fabfff640 1 --2- 172.21.15.16:0/38951377 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3fb40cbdb0 0x7f3fa00047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:59.409 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:59.403+0000 7f3fabfff640 1 --2- 172.21.15.16:0/38951377 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3fa0004d10 0x7f3fa0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:59.409 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:59.403+0000 7f3fabfff640 1 -- 172.21.15.16:0/38951377 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fb406bf70 con 0x7f3fa0003980 2023-05-22T16:05:59.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:59.403+0000 7f3fabfff640 1 -- 172.21.15.16:0/38951377 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fb40cad10 con 0x7f3fa0004d10 2023-05-22T16:05:59.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:59.403+0000 7f3fabfff640 1 -- 172.21.15.16:0/38951377 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3fa000b990 con 0x7f3fb40cbdb0 2023-05-22T16:05:59.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:59.403+0000 7f3fb94e7640 1 --2- 172.21.15.16:0/38951377 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3fa0003980 0x7f3fa0003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:59.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:59.403+0000 7f3fb8ce6640 1 --2- 172.21.15.16:0/38951377 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3fb40cbdb0 0x7f3fa00047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:59.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:59.403+0000 7f3fb9ce8640 1 --2- 172.21.15.16:0/38951377 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3fa0004d10 0x7f3fa0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:59.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:59.403+0000 7f3fb94e7640 1 -- 172.21.15.16:0/38951377 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3fb40cbdb0 msgr2=0x7f3fa00047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:05:59.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:59.403+0000 7f3fb94e7640 1 --2- 172.21.15.16:0/38951377 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3fb40cbdb0 0x7f3fa00047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:59.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:59.403+0000 7f3fb94e7640 1 -- 172.21.15.16:0/38951377 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3fa0004d10 msgr2=0x7f3fa0005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:05:59.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:59.403+0000 7f3fb94e7640 1 --2- 172.21.15.16:0/38951377 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3fa0004d10 0x7f3fa0005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:59.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:59.403+0000 7f3fb94e7640 1 -- 172.21.15.16:0/38951377 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fa0003530 con 0x7f3fa0003980 2023-05-22T16:05:59.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:59.403+0000 7f3fb9ce8640 1 --2- 172.21.15.16:0/38951377 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3fa0004d10 0x7f3fa0005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T16:05:59.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:59.403+0000 7f3fb8ce6640 1 --2- 172.21.15.16:0/38951377 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3fb40cbdb0 0x7f3fa00047d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T16:05:59.412 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:59.403+0000 7f3fb94e7640 1 --2- 172.21.15.16:0/38951377 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3fa0003980 0x7f3fa0003d90 secure :-1 s=READY pgs=2599 cs=0 l=1 rev1=1 crypto rx=0x7f3fb000c210 tx=0x7f3fb001cbd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4ed535bbe44f35f server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:59.412 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:59.403+0000 7f3fabfff640 1 -- 172.21.15.16:0/38951377 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fb00a5910 con 0x7f3fa0003980 2023-05-22T16:05:59.412 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:59.403+0000 7f3fabfff640 1 -- 172.21.15.16:0/38951377 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fb00a5a70 con 0x7f3fa0003980 2023-05-22T16:05:59.412 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:59.403+0000 7f3fabfff640 1 -- 172.21.15.16:0/38951377 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fb00a5e20 con 0x7f3fa0003980 2023-05-22T16:05:59.415 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:59.403+0000 7f3fbb74b640 1 -- 172.21.15.16:0/38951377 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3fa0003980 msgr2=0x7f3fa0003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:59.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:59.403+0000 7f3fbb74b640 1 --2- 172.21.15.16:0/38951377 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3fa0003980 0x7f3fa0003d90 secure :-1 s=READY pgs=2599 cs=0 l=1 rev1=1 crypto rx=0x7f3fb000c210 tx=0x7f3fb001cbd0 comp rx=0 tx=0).stop 2023-05-22T16:05:59.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:59.403+0000 7f3fb9ce8640 1 -- 172.21.15.16:0/38951377 reap_dead start 2023-05-22T16:05:59.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:59.403+0000 7f3fbb74b640 1 -- 172.21.15.16:0/38951377 shutdown_connections 2023-05-22T16:05:59.417 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:59.403+0000 7f3fbb74b640 1 -- 172.21.15.16:0/38951377 >> 172.21.15.16:0/38951377 conn(0x7f3fb4099620 msgr2=0x7f3fb40d02c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:05:59.417 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:59.403+0000 7f3fbb74b640 1 -- 172.21.15.16:0/38951377 shutdown_connections 2023-05-22T16:05:59.417 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:59.403+0000 7f3fbb74b640 1 -- 172.21.15.16:0/38951377 wait complete. 2023-05-22T16:05:59.417 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:59.407+0000 7f3fbb74b640 1 Processor -- start 2023-05-22T16:05:59.418 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:59.407+0000 7f3fbb74b640 1 -- start start 2023-05-22T16:05:59.418 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:59.407+0000 7f3fbb74b640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3fa0003980 0x7f3fb4150850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:59.418 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:59.407+0000 7f3fbb74b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3fb4158490 0x7f3fb4159e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:59.419 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:59.407+0000 7f3fbb74b640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3fb4158900 0x7f3fb415a390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:59.419 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:59.407+0000 7f3fbb74b640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fb4150d90 con 0x7f3fb4158490 2023-05-22T16:05:59.419 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:59.407+0000 7f3fbb74b640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fb4151da0 con 0x7f3fa0003980 2023-05-22T16:05:59.419 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:59.407+0000 7f3fbb74b640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3fb4158e70 con 0x7f3fb4158900 2023-05-22T16:05:59.420 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:59.407+0000 7f3fb8ce6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3fb4158490 0x7f3fb4159e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:59.420 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:59.407+0000 7f3fb8ce6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3fb4158490 0x7f3fb4159e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43332/0 (socket says 172.21.15.16:43332) 2023-05-22T16:05:59.420 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:59.407+0000 7f3fb8ce6640 1 -- 172.21.15.16:0/2602690024 learned_addr learned my addr 172.21.15.16:0/2602690024 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:05:59.421 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:59.407+0000 7f3fb94e7640 1 --2- 172.21.15.16:0/2602690024 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3fa0003980 0x7f3fb4150850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:59.421 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:59.407+0000 7f3fb9ce8640 1 --2- 172.21.15.16:0/2602690024 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3fb4158900 0x7f3fb415a390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:59.421 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:59.407+0000 7f3fb8ce6640 1 -- 172.21.15.16:0/2602690024 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3fb4158900 msgr2=0x7f3fb415a390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:59.421 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:59.407+0000 7f3fb8ce6640 1 --2- 172.21.15.16:0/2602690024 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3fb4158900 0x7f3fb415a390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:59.422 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:59.407+0000 7f3fb8ce6640 1 -- 172.21.15.16:0/2602690024 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3fa0003980 msgr2=0x7f3fb4150850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:05:59.422 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:59.407+0000 7f3fb8ce6640 1 --2- 172.21.15.16:0/2602690024 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3fa0003980 0x7f3fb4150850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:05:59.422 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:59.407+0000 7f3fb8ce6640 1 -- 172.21.15.16:0/2602690024 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fb40b3820 con 0x7f3fb4158490 2023-05-22T16:05:59.423 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:59.407+0000 7f3fb94e7640 1 --2- 172.21.15.16:0/2602690024 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3fa0003980 0x7f3fb4150850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T16:05:59.423 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:59.407+0000 7f3fb9ce8640 1 --2- 172.21.15.16:0/2602690024 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3fb4158900 0x7f3fb415a390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T16:05:59.423 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:59.407+0000 7f3fb8ce6640 1 --2- 172.21.15.16:0/2602690024 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3fb4158490 0x7f3fb4159e50 secure :-1 s=READY pgs=2600 cs=0 l=1 rev1=1 crypto rx=0x7f3fa40108d0 tx=0x7f3fa4010d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:59.424 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:59.407+0000 7f3fa9ffb640 1 -- 172.21.15.16:0/2602690024 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fa4004220 con 0x7f3fb4158490 2023-05-22T16:05:59.424 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:59.407+0000 7f3fa9ffb640 1 -- 172.21.15.16:0/2602690024 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fa4004380 con 0x7f3fb4158490 2023-05-22T16:05:59.424 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:59.407+0000 7f3fbb74b640 1 -- 172.21.15.16:0/2602690024 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3fb40cb890 con 0x7f3fb4158490 2023-05-22T16:05:59.424 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:59.407+0000 7f3fa9ffb640 1 -- 172.21.15.16:0/2602690024 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fa400eba0 con 0x7f3fb4158490 2023-05-22T16:05:59.424 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:59.407+0000 7f3fbb74b640 1 -- 172.21.15.16:0/2602690024 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3fb40cc5c0 con 0x7f3fb4158490 2023-05-22T16:05:59.424 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:59.407+0000 7f3fbb74b640 1 -- 172.21.15.16:0/2602690024 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3fb4151b70 con 0x7f3fb4158490 2023-05-22T16:05:59.425 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:59.411+0000 7f3fa9ffb640 1 -- 172.21.15.16:0/2602690024 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 100) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f3fa400ed00 con 0x7f3fb4158490 2023-05-22T16:05:59.425 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:59.411+0000 7f3fa9ffb640 1 --2- 172.21.15.16:0/2602690024 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f3f980654d0 0x7f3f98067980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:05:59.425 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:59.411+0000 7f3fa9ffb640 1 -- 172.21.15.16:0/2602690024 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(199..199 src has 1..199) v4 ==== 7169+0+0 (secure 0 0 0) 0x7f3fa4082d10 con 0x7f3fb4158490 2023-05-22T16:05:59.425 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:59.411+0000 7f3fb94e7640 1 --2- 172.21.15.16:0/2602690024 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f3f980654d0 0x7f3f98067980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:05:59.425 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:59.411+0000 7f3fb94e7640 1 --2- 172.21.15.16:0/2602690024 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f3f980654d0 0x7f3f98067980 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f3fb00aa000 tx=0x7f3fb00b0040 comp rx=0 tx=0).ready entity=mgr.8195 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:05:59.426 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:59.415+0000 7f3fa9ffb640 1 -- 172.21.15.16:0/2602690024 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f3fb4151b70 con 0x7f3fb4158490 2023-05-22T16:05:59.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:05:59.603+0000 7f3fbb74b640 1 -- 172.21.15.16:0/2602690024 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f3fb4151ee0 con 0x7f3fb4158490 2023-05-22T16:06:00.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:00.271+0000 7f3fa9ffb640 1 -- 172.21.15.16:0/2602690024 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v83) v1 ==== 104+0+0 (secure 0 0 0) 0x7f3fa4050420 con 0x7f3fb4158490 2023-05-22T16:06:00.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:00.275+0000 7f3fbb74b640 1 -- 172.21.15.16:0/2602690024 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f3f980654d0 msgr2=0x7f3f98067980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:06:00.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:00.275+0000 7f3fbb74b640 1 --2- 172.21.15.16:0/2602690024 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f3f980654d0 0x7f3f98067980 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f3fb00aa000 tx=0x7f3fb00b0040 comp rx=0 tx=0).stop 2023-05-22T16:06:00.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:00.275+0000 7f3fbb74b640 1 -- 172.21.15.16:0/2602690024 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3fb4158490 msgr2=0x7f3fb4159e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:06:00.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:00.275+0000 7f3fbb74b640 1 --2- 172.21.15.16:0/2602690024 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3fb4158490 0x7f3fb4159e50 secure :-1 s=READY pgs=2600 cs=0 l=1 rev1=1 crypto rx=0x7f3fa40108d0 tx=0x7f3fa4010d90 comp rx=0 tx=0).stop 2023-05-22T16:06:00.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:00.275+0000 7f3fbb74b640 1 -- 172.21.15.16:0/2602690024 shutdown_connections 2023-05-22T16:06:00.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:00.275+0000 7f3fbb74b640 1 --2- 172.21.15.16:0/2602690024 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3fb4158900 0x7f3fb415a390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:00.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:00.275+0000 7f3fbb74b640 1 --2- 172.21.15.16:0/2602690024 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f3f980654d0 0x7f3f98067980 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:00.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:00.275+0000 7f3fbb74b640 1 --2- 172.21.15.16:0/2602690024 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3fb4158490 0x7f3fb4159e50 unknown :-1 s=CLOSED pgs=2600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:00.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:00.275+0000 7f3fbb74b640 1 --2- 172.21.15.16:0/2602690024 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3fa0003980 0x7f3fb4150850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:00.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:00.275+0000 7f3fbb74b640 1 -- 172.21.15.16:0/2602690024 >> 172.21.15.16:0/2602690024 conn(0x7f3fb4099620 msgr2=0x7f3fb40b2180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:06:00.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:00.275+0000 7f3fbb74b640 1 -- 172.21.15.16:0/2602690024 shutdown_connections 2023-05-22T16:06:00.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:00.275+0000 7f3fbb74b640 1 -- 172.21.15.16:0/2602690024 wait complete. 2023-05-22T16:06:00.298 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-05-22T16:06:00.475 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:00.471+0000 7f730fe37640 1 Processor -- start 2023-05-22T16:06:00.475 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:00.471+0000 7f730fe37640 1 -- start start 2023-05-22T16:06:00.476 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:00.471+0000 7f730fe37640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f7308069e70 0x7f730806ea30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:00.476 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:00.471+0000 7f730fe37640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f730806ef70 0x7f730815fab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:00.477 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:00.471+0000 7f730fe37640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f730806e5a0 con 0x7f730815fff0 2023-05-22T16:06:00.477 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:00.471+0000 7f730fe37640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7308160410 con 0x7f730806ef70 2023-05-22T16:06:00.477 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:00.471+0000 7f730fe37640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73080ccb30 con 0x7f7308069e70 2023-05-22T16:06:00.477 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:00.471+0000 7f730d3d2640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f730806ef70 0x7f730815fab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:00.477 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:00.471+0000 7f730d3d2640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f730806ef70 0x7f730815fab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43344/0 (socket says 172.21.15.16:43344) 2023-05-22T16:06:00.477 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:00.471+0000 7f730d3d2640 1 -- 172.21.15.16:0/3419597511 learned_addr learned my addr 172.21.15.16:0/3419597511 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:06:00.477 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:00.471+0000 7f730dbd3640 1 --2- 172.21.15.16:0/3419597511 >> v2:172.21.15.62:3300/0 conn(0x7f7308069e70 0x7f730806ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:00.478 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:00.471+0000 7f730d3d2640 1 -- 172.21.15.16:0/3419597511 >> v2:172.21.15.62:3300/0 conn(0x7f7308069e70 msgr2=0x7f730806ea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:06:00.478 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:00.471+0000 7f730d3d2640 1 --2- 172.21.15.16:0/3419597511 >> v2:172.21.15.62:3300/0 conn(0x7f7308069e70 0x7f730806ea30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:00.478 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:00.471+0000 7f730d3d2640 1 -- 172.21.15.16:0/3419597511 >> v1:172.21.15.62:6789/0 conn(0x7f730815fff0 legacy=0x7f73080707d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:06:00.478 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:00.471+0000 7f730d3d2640 1 -- 172.21.15.16:0/3419597511 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7308160970 con 0x7f730806ef70 2023-05-22T16:06:00.478 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:00.475+0000 7f730d3d2640 1 --2- 172.21.15.16:0/3419597511 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f730806ef70 0x7f730815fab0 secure :-1 s=READY pgs=2601 cs=0 l=1 rev1=1 crypto rx=0x7f730806a720 tx=0x7f72fc01c4c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2bb1de1ba612f691 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:06:00.479 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:00.475+0000 7f730cbd1640 1 -- 172.21.15.16:0/3419597511 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72fc0aa620 con 0x7f730806ef70 2023-05-22T16:06:00.479 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:00.475+0000 7f730cbd1640 1 -- 172.21.15.16:0/3419597511 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72fc0aa780 con 0x7f730806ef70 2023-05-22T16:06:00.479 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:00.475+0000 7f730cbd1640 1 -- 172.21.15.16:0/3419597511 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72fc0aab10 con 0x7f730806ef70 2023-05-22T16:06:00.480 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:00.475+0000 7f730fe37640 1 -- 172.21.15.16:0/3419597511 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f730806ef70 msgr2=0x7f730815fab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:06:00.480 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:00.475+0000 7f730fe37640 1 --2- 172.21.15.16:0/3419597511 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f730806ef70 0x7f730815fab0 secure :-1 s=READY pgs=2601 cs=0 l=1 rev1=1 crypto rx=0x7f730806a720 tx=0x7f72fc01c4c0 comp rx=0 tx=0).stop 2023-05-22T16:06:00.481 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:00.475+0000 7f730fe37640 1 -- 172.21.15.16:0/3419597511 shutdown_connections 2023-05-22T16:06:00.481 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:00.475+0000 7f730fe37640 1 --2- 172.21.15.16:0/3419597511 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f730806ef70 0x7f730815fab0 unknown :-1 s=CLOSED pgs=2601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:00.481 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:00.475+0000 7f730fe37640 1 --2- 172.21.15.16:0/3419597511 >> v2:172.21.15.62:3300/0 conn(0x7f7308069e70 0x7f730806ea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:00.481 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:00.475+0000 7f730fe37640 1 -- 172.21.15.16:0/3419597511 >> 172.21.15.16:0/3419597511 conn(0x7f73080cbd00 msgr2=0x7f73080cc100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:06:00.481 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:00.475+0000 7f730fe37640 1 -- 172.21.15.16:0/3419597511 shutdown_connections 2023-05-22T16:06:00.482 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:00.475+0000 7f730fe37640 1 -- 172.21.15.16:0/3419597511 wait complete. 2023-05-22T16:06:00.482 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:00.479+0000 7f730fe37640 1 Processor -- start 2023-05-22T16:06:00.482 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:00.479+0000 7f730fe37640 1 -- start start 2023-05-22T16:06:00.484 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:00.479+0000 7f730fe37640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7308069e70 0x7f7308155320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:00.484 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:00.479+0000 7f730fe37640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f730806ef70 0x7f730815aca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:00.484 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:00.479+0000 7f730fe37640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f730815fff0 0x7f730815b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:00.484 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:00.479+0000 7f730fe37640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73080b2850 con 0x7f730815fff0 2023-05-22T16:06:00.484 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:00.479+0000 7f730fe37640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73080aece0 con 0x7f730806ef70 2023-05-22T16:06:00.484 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:00.479+0000 7f730fe37640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73080aef30 con 0x7f7308069e70 2023-05-22T16:06:00.485 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:00.479+0000 7f730e3d4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f730815fff0 0x7f730815b1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:00.485 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:00.479+0000 7f730e3d4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f730815fff0 0x7f730815b1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43352/0 (socket says 172.21.15.16:43352) 2023-05-22T16:06:00.485 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:00.479+0000 7f730e3d4640 1 -- 172.21.15.16:0/11792668 learned_addr learned my addr 172.21.15.16:0/11792668 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:06:00.485 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:00.479+0000 7f730d3d2640 1 --2- 172.21.15.16:0/11792668 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f730806ef70 0x7f730815aca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:00.485 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:00.479+0000 7f730dbd3640 1 --2- 172.21.15.16:0/11792668 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7308069e70 0x7f7308155320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:00.485 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:00.479+0000 7f730e3d4640 1 -- 172.21.15.16:0/11792668 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7308069e70 msgr2=0x7f7308155320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:06:00.485 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:00.479+0000 7f730e3d4640 1 --2- 172.21.15.16:0/11792668 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7308069e70 0x7f7308155320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:00.486 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:00.479+0000 7f730e3d4640 1 -- 172.21.15.16:0/11792668 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f730806ef70 msgr2=0x7f730815aca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:06:00.486 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:00.479+0000 7f730e3d4640 1 --2- 172.21.15.16:0/11792668 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f730806ef70 0x7f730815aca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:00.486 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:00.479+0000 7f730e3d4640 1 -- 172.21.15.16:0/11792668 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72fc0094d0 con 0x7f730815fff0 2023-05-22T16:06:00.486 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:00.479+0000 7f730e3d4640 1 --2- 172.21.15.16:0/11792668 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f730815fff0 0x7f730815b1e0 secure :-1 s=READY pgs=2602 cs=0 l=1 rev1=1 crypto rx=0x7f730000c780 tx=0x7f730000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:06:00.486 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:00.479+0000 7f72fa7fc640 1 -- 172.21.15.16:0/11792668 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7300004b70 con 0x7f730815fff0 2023-05-22T16:06:00.486 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:00.479+0000 7f72fa7fc640 1 -- 172.21.15.16:0/11792668 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f730000ce90 con 0x7f730815fff0 2023-05-22T16:06:00.486 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:00.479+0000 7f730fe37640 1 -- 172.21.15.16:0/11792668 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f730806a240 con 0x7f730815fff0 2023-05-22T16:06:00.487 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:00.479+0000 7f72fa7fc640 1 -- 172.21.15.16:0/11792668 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f730000a850 con 0x7f730815fff0 2023-05-22T16:06:00.487 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:00.479+0000 7f730fe37640 1 -- 172.21.15.16:0/11792668 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f730806e230 con 0x7f730815fff0 2023-05-22T16:06:00.487 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:00.479+0000 7f730fe37640 1 -- 172.21.15.16:0/11792668 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f730806a3d0 con 0x7f730815fff0 2023-05-22T16:06:00.487 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:00.479+0000 7f72fa7fc640 1 -- 172.21.15.16:0/11792668 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 100) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f730000a9b0 con 0x7f730815fff0 2023-05-22T16:06:00.488 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:00.483+0000 7f72fa7fc640 1 --2- 172.21.15.16:0/11792668 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f72ec0654a0 0x7f72ec067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:00.488 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:00.483+0000 7f72fa7fc640 1 -- 172.21.15.16:0/11792668 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(199..199 src has 1..199) v4 ==== 7169+0+0 (secure 0 0 0) 0x7f7300083010 con 0x7f730815fff0 2023-05-22T16:06:00.488 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:00.483+0000 7f730dbd3640 1 --2- 172.21.15.16:0/11792668 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f72ec0654a0 0x7f72ec067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:00.488 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:00.483+0000 7f730dbd3640 1 --2- 172.21.15.16:0/11792668 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f72ec0654a0 0x7f72ec067950 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f730400b930 tx=0x7f7304008040 comp rx=0 tx=0).ready entity=mgr.8195 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:06:00.488 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:00.487+0000 7f72fa7fc640 1 -- 172.21.15.16:0/11792668 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f7300054150 con 0x7f730815fff0 2023-05-22T16:06:00.642 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:00.639+0000 7f730fe37640 1 -- 172.21.15.16:0/11792668 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f730804dcc0 con 0x7f730815fff0 2023-05-22T16:06:01.347 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:01.339+0000 7f72fa7fc640 1 -- 172.21.15.16:0/11792668 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v200) v1 ==== 181+0+0 (secure 0 0 0) 0x7f7300053d40 con 0x7f730815fff0 2023-05-22T16:06:01.347 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'cephfs_metadata' removed 2023-05-22T16:06:01.352 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:01.347+0000 7f730fe37640 1 -- 172.21.15.16:0/11792668 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f72ec0654a0 msgr2=0x7f72ec067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:06:01.353 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:01.347+0000 7f730fe37640 1 --2- 172.21.15.16:0/11792668 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f72ec0654a0 0x7f72ec067950 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f730400b930 tx=0x7f7304008040 comp rx=0 tx=0).stop 2023-05-22T16:06:01.353 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:01.351+0000 7f730fe37640 1 -- 172.21.15.16:0/11792668 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f730815fff0 msgr2=0x7f730815b1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:06:01.353 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:01.351+0000 7f730fe37640 1 --2- 172.21.15.16:0/11792668 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f730815fff0 0x7f730815b1e0 secure :-1 s=READY pgs=2602 cs=0 l=1 rev1=1 crypto rx=0x7f730000c780 tx=0x7f730000cc40 comp rx=0 tx=0).stop 2023-05-22T16:06:01.354 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:01.351+0000 7f730fe37640 1 -- 172.21.15.16:0/11792668 shutdown_connections 2023-05-22T16:06:01.354 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:01.351+0000 7f730fe37640 1 --2- 172.21.15.16:0/11792668 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f72ec0654a0 0x7f72ec067950 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:01.354 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:01.351+0000 7f730fe37640 1 --2- 172.21.15.16:0/11792668 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f730815fff0 0x7f730815b1e0 unknown :-1 s=CLOSED pgs=2602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:01.354 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:01.351+0000 7f730fe37640 1 --2- 172.21.15.16:0/11792668 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f730806ef70 0x7f730815aca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:01.354 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:01.351+0000 7f730fe37640 1 --2- 172.21.15.16:0/11792668 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7308069e70 0x7f7308155320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:01.354 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:01.351+0000 7f730fe37640 1 -- 172.21.15.16:0/11792668 >> 172.21.15.16:0/11792668 conn(0x7f73080cbd00 msgr2=0x7f7308071350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:06:01.355 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:01.351+0000 7f730fe37640 1 -- 172.21.15.16:0/11792668 shutdown_connections 2023-05-22T16:06:01.355 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:01.351+0000 7f730fe37640 1 -- 172.21.15.16:0/11792668 wait complete. 2023-05-22T16:06:01.376 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-05-22T16:06:01.509 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:01.507+0000 7fc608445640 1 Processor -- start 2023-05-22T16:06:01.509 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:01.507+0000 7fc608445640 1 -- start start 2023-05-22T16:06:01.511 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:01.507+0000 7fc608445640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc600159a50 0x7fc600070d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:01.511 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:01.507+0000 7fc608445640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc60006c060 0x7fc6000712c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:01.511 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:01.507+0000 7fc608445640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc600071800 con 0x7fc6000cea90 2023-05-22T16:06:01.511 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:01.507+0000 7fc608445640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc60015d800 con 0x7fc60006c060 2023-05-22T16:06:01.511 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:01.507+0000 7fc608445640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc60015dae0 con 0x7fc600159a50 2023-05-22T16:06:01.511 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:01.507+0000 7fc6069e2640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc60006c060 0x7fc6000712c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:01.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:01.507+0000 7fc6059e0640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fc6000cea90 0x7fc6000c8740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:49784/0 (socket says 172.21.15.16:49784) 2023-05-22T16:06:01.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:01.507+0000 7fc6069e2640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc60006c060 0x7fc6000712c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43360/0 (socket says 172.21.15.16:43360) 2023-05-22T16:06:01.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:01.507+0000 7fc6059e0640 1 -- 172.21.15.16:0/85005052 learned_addr learned my addr 172.21.15.16:0/85005052 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T16:06:01.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:01.507+0000 7fc6061e1640 1 --2- 172.21.15.16:0/85005052 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc600159a50 0x7fc600070d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:01.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:01.507+0000 7fc6069e2640 1 -- 172.21.15.16:0/85005052 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc600159a50 msgr2=0x7fc600070d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:06:01.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:01.507+0000 7fc6069e2640 1 --2- 172.21.15.16:0/85005052 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc600159a50 0x7fc600070d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:01.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:01.507+0000 7fc6069e2640 1 -- 172.21.15.16:0/85005052 >> v1:172.21.15.16:6789/0 conn(0x7fc6000cea90 legacy=0x7fc6000c8740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:06:01.513 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:01.507+0000 7fc6051df640 1 -- 172.21.15.16:0/85005052 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2999228595 0 0) 0x7fc600071800 con 0x7fc6000cea90 2023-05-22T16:06:01.513 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:01.507+0000 7fc6069e2640 1 -- 172.21.15.16:0/85005052 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc60015ddc0 con 0x7fc60006c060 2023-05-22T16:06:01.513 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:01.507+0000 7fc6061e1640 1 --2- 172.21.15.16:0/85005052 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc600159a50 0x7fc600070d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T16:06:01.514 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:01.511+0000 7fc6069e2640 1 --2- 172.21.15.16:0/85005052 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc60006c060 0x7fc6000712c0 secure :-1 s=READY pgs=2604 cs=0 l=1 rev1=1 crypto rx=0x7fc5f8009d90 tx=0x7fc5f801c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a0956b8ee15717d server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:06:01.514 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:01.511+0000 7fc6051df640 1 -- 172.21.15.16:0/85005052 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5f80a57b0 con 0x7fc60006c060 2023-05-22T16:06:01.514 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:01.511+0000 7fc6051df640 1 -- 172.21.15.16:0/85005052 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5f80a5910 con 0x7fc60006c060 2023-05-22T16:06:01.514 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:01.511+0000 7fc6051df640 1 -- 172.21.15.16:0/85005052 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5f80a5ca0 con 0x7fc60006c060 2023-05-22T16:06:01.514 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:01.511+0000 7fc608445640 1 -- 172.21.15.16:0/85005052 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc60006c060 msgr2=0x7fc6000712c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:06:01.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:01.511+0000 7fc608445640 1 --2- 172.21.15.16:0/85005052 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc60006c060 0x7fc6000712c0 secure :-1 s=READY pgs=2604 cs=0 l=1 rev1=1 crypto rx=0x7fc5f8009d90 tx=0x7fc5f801c8c0 comp rx=0 tx=0).stop 2023-05-22T16:06:01.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:01.511+0000 7fc608445640 1 -- 172.21.15.16:0/85005052 shutdown_connections 2023-05-22T16:06:01.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:01.511+0000 7fc608445640 1 --2- 172.21.15.16:0/85005052 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc60006c060 0x7fc6000712c0 unknown :-1 s=CLOSED pgs=2604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:01.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:01.511+0000 7fc608445640 1 --2- 172.21.15.16:0/85005052 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc600159a50 0x7fc600070d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:01.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:01.511+0000 7fc608445640 1 -- 172.21.15.16:0/85005052 >> 172.21.15.16:0/85005052 conn(0x7fc600069800 msgr2=0x7fc600069c00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:06:01.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:01.511+0000 7fc608445640 1 -- 172.21.15.16:0/85005052 shutdown_connections 2023-05-22T16:06:01.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:01.511+0000 7fc608445640 1 -- 172.21.15.16:0/85005052 wait complete. 2023-05-22T16:06:01.517 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:01.511+0000 7fc608445640 1 Processor -- start 2023-05-22T16:06:01.517 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:01.511+0000 7fc608445640 1 -- start start 2023-05-22T16:06:01.517 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:01.511+0000 7fc608445640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc60006c060 0x7fc6000cd280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:01.517 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:01.511+0000 7fc608445640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc6000cea90 0x7fc6000b2ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:01.517 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:01.511+0000 7fc608445640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc600159a50 0x7fc60015cc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:01.518 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:01.511+0000 7fc608445640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6000cd7c0 con 0x7fc6000cea90 2023-05-22T16:06:01.518 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:01.511+0000 7fc608445640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6000b1f60 con 0x7fc60006c060 2023-05-22T16:06:01.518 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:01.511+0000 7fc608445640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6000b3220 con 0x7fc600159a50 2023-05-22T16:06:01.518 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:01.511+0000 7fc6059e0640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc6000cea90 0x7fc6000b2ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:01.518 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:01.511+0000 7fc6059e0640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc6000cea90 0x7fc6000b2ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43376/0 (socket says 172.21.15.16:43376) 2023-05-22T16:06:01.518 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:01.511+0000 7fc6059e0640 1 -- 172.21.15.16:0/1989941990 learned_addr learned my addr 172.21.15.16:0/1989941990 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:06:01.518 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:01.511+0000 7fc6069e2640 1 --2- 172.21.15.16:0/1989941990 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc600159a50 0x7fc60015cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:01.519 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:01.511+0000 7fc6061e1640 1 --2- 172.21.15.16:0/1989941990 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc60006c060 0x7fc6000cd280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:01.519 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:01.511+0000 7fc6059e0640 1 -- 172.21.15.16:0/1989941990 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc600159a50 msgr2=0x7fc60015cc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:06:01.519 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:01.511+0000 7fc6059e0640 1 --2- 172.21.15.16:0/1989941990 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc600159a50 0x7fc60015cc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:01.519 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:01.511+0000 7fc6059e0640 1 -- 172.21.15.16:0/1989941990 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc60006c060 msgr2=0x7fc6000cd280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:06:01.519 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:01.511+0000 7fc6059e0640 1 --2- 172.21.15.16:0/1989941990 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc60006c060 0x7fc6000cd280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:01.519 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:01.511+0000 7fc6059e0640 1 -- 172.21.15.16:0/1989941990 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5f8009a70 con 0x7fc6000cea90 2023-05-22T16:06:01.519 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:01.511+0000 7fc6059e0640 1 --2- 172.21.15.16:0/1989941990 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc6000cea90 0x7fc6000b2ce0 secure :-1 s=READY pgs=2605 cs=0 l=1 rev1=1 crypto rx=0x7fc5f00011f0 tx=0x7fc5f000c9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:06:01.519 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:01.515+0000 7fc5f6ffd640 1 -- 172.21.15.16:0/1989941990 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5f00074e0 con 0x7fc6000cea90 2023-05-22T16:06:01.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:01.515+0000 7fc5f6ffd640 1 -- 172.21.15.16:0/1989941990 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5f0007640 con 0x7fc6000cea90 2023-05-22T16:06:01.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:01.515+0000 7fc5f6ffd640 1 -- 172.21.15.16:0/1989941990 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5f0004a10 con 0x7fc6000cea90 2023-05-22T16:06:01.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:01.515+0000 7fc608445640 1 -- 172.21.15.16:0/1989941990 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6000c9db0 con 0x7fc6000cea90 2023-05-22T16:06:01.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:01.515+0000 7fc608445640 1 -- 172.21.15.16:0/1989941990 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc600071a20 con 0x7fc6000cea90 2023-05-22T16:06:01.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:01.515+0000 7fc608445640 1 -- 172.21.15.16:0/1989941990 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6000c9f40 con 0x7fc6000cea90 2023-05-22T16:06:01.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:01.515+0000 7fc5f6ffd640 1 -- 172.21.15.16:0/1989941990 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 100) v1 ==== 84332+0+0 (secure 0 0 0) 0x7fc5f0004b70 con 0x7fc6000cea90 2023-05-22T16:06:01.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:01.515+0000 7fc5f6ffd640 1 --2- 172.21.15.16:0/1989941990 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fc5e40654d0 0x7fc5e4067980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:01.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:01.515+0000 7fc5f6ffd640 1 -- 172.21.15.16:0/1989941990 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(200..200 src has 1..200) v4 ==== 6731+0+0 (secure 0 0 0) 0x7fc5f0081e20 con 0x7fc6000cea90 2023-05-22T16:06:01.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:01.515+0000 7fc6061e1640 1 --2- 172.21.15.16:0/1989941990 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fc5e40654d0 0x7fc5e4067980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:01.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:01.515+0000 7fc6061e1640 1 --2- 172.21.15.16:0/1989941990 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fc5e40654d0 0x7fc5e4067980 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fc5fc006fd0 tx=0x7fc5fc008040 comp rx=0 tx=0).ready entity=mgr.8195 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:06:01.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:01.519+0000 7fc5f6ffd640 1 -- 172.21.15.16:0/1989941990 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fc6000c9f40 con 0x7fc6000cea90 2023-05-22T16:06:01.679 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:01.675+0000 7fc608445640 1 -- 172.21.15.16:0/1989941990 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fc6000c9f40 con 0x7fc6000cea90 2023-05-22T16:06:02.363 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:02.359+0000 7fc5f6ffd640 1 -- 172.21.15.16:0/1989941990 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v201) v1 ==== 169+0+0 (secure 0 0 0) 0x7fc6000c9f40 con 0x7fc6000cea90 2023-05-22T16:06:02.364 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'cephfs_data' removed 2023-05-22T16:06:02.370 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:02.367+0000 7fc608445640 1 -- 172.21.15.16:0/1989941990 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fc5e40654d0 msgr2=0x7fc5e4067980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:06:02.371 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:02.367+0000 7fc608445640 1 --2- 172.21.15.16:0/1989941990 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fc5e40654d0 0x7fc5e4067980 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fc5fc006fd0 tx=0x7fc5fc008040 comp rx=0 tx=0).stop 2023-05-22T16:06:02.371 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:02.367+0000 7fc608445640 1 -- 172.21.15.16:0/1989941990 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc6000cea90 msgr2=0x7fc6000b2ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:06:02.371 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:02.367+0000 7fc608445640 1 --2- 172.21.15.16:0/1989941990 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc6000cea90 0x7fc6000b2ce0 secure :-1 s=READY pgs=2605 cs=0 l=1 rev1=1 crypto rx=0x7fc5f00011f0 tx=0x7fc5f000c9b0 comp rx=0 tx=0).stop 2023-05-22T16:06:02.372 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:02.371+0000 7fc608445640 1 -- 172.21.15.16:0/1989941990 shutdown_connections 2023-05-22T16:06:02.373 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:02.371+0000 7fc608445640 1 --2- 172.21.15.16:0/1989941990 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc600159a50 0x7fc60015cc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:02.373 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:02.371+0000 7fc608445640 1 --2- 172.21.15.16:0/1989941990 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fc5e40654d0 0x7fc5e4067980 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:02.373 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:02.371+0000 7fc608445640 1 --2- 172.21.15.16:0/1989941990 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc6000cea90 0x7fc6000b2ce0 unknown :-1 s=CLOSED pgs=2605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:02.373 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:02.371+0000 7fc608445640 1 --2- 172.21.15.16:0/1989941990 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc60006c060 0x7fc6000cd280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:02.373 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:02.371+0000 7fc608445640 1 -- 172.21.15.16:0/1989941990 >> 172.21.15.16:0/1989941990 conn(0x7fc600069800 msgr2=0x7fc6001635b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:06:02.374 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:02.371+0000 7fc608445640 1 -- 172.21.15.16:0/1989941990 shutdown_connections 2023-05-22T16:06:02.374 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:02.371+0000 7fc608445640 1 -- 172.21.15.16:0/1989941990 wait complete. 2023-05-22T16:06:02.396 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2023-05-22T16:06:02.526 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:02.523+0000 7fee61545640 1 Processor -- start 2023-05-22T16:06:02.527 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:02.523+0000 7fee61545640 1 -- start start 2023-05-22T16:06:02.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:02.523+0000 7fee61545640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fee5c0cb4e0 0x7fee5c0cd880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:02.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:02.523+0000 7fee61545640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fee5c0afbf0 0x7fee5c0b0030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:02.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:02.523+0000 7fee61545640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee5c0b0570 con 0x7fee5c1602b0 2023-05-22T16:06:02.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:02.523+0000 7fee61545640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee5c0c8ac0 con 0x7fee5c0cb4e0 2023-05-22T16:06:02.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:02.523+0000 7fee61545640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee5c071250 con 0x7fee5c0afbf0 2023-05-22T16:06:02.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:02.523+0000 7fee5affd640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fee5c1602b0 0x7fee5c1645d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:49800/0 (socket says 172.21.15.16:49800) 2023-05-22T16:06:02.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:02.523+0000 7fee5a7fc640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fee5c0cb4e0 0x7fee5c0cd880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:02.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:02.523+0000 7fee5affd640 1 -- 172.21.15.16:0/963068374 learned_addr learned my addr 172.21.15.16:0/963068374 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T16:06:02.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:02.523+0000 7fee5b7fe640 1 --2- 172.21.15.16:0/963068374 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fee5c0afbf0 0x7fee5c0b0030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:02.530 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:02.527+0000 7fee5a7fc640 1 -- 172.21.15.16:0/963068374 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fee5c0afbf0 msgr2=0x7fee5c0b0030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:06:02.530 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:02.527+0000 7fee5a7fc640 1 --2- 172.21.15.16:0/963068374 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fee5c0afbf0 0x7fee5c0b0030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:02.530 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:02.527+0000 7fee5a7fc640 1 -- 172.21.15.16:0/963068374 >> v1:172.21.15.16:6789/0 conn(0x7fee5c1602b0 legacy=0x7fee5c1645d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:06:02.530 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:02.527+0000 7fee5a7fc640 1 -- 172.21.15.16:0/963068374 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee5c164ce0 con 0x7fee5c0cb4e0 2023-05-22T16:06:02.530 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:02.527+0000 7fee59ffb640 1 -- 172.21.15.16:0/963068374 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1763210892 0 0) 0x7fee5c0b0570 con 0x7fee5c1602b0 2023-05-22T16:06:02.531 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:02.527+0000 7fee5a7fc640 1 --2- 172.21.15.16:0/963068374 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fee5c0cb4e0 0x7fee5c0cd880 secure :-1 s=READY pgs=2607 cs=0 l=1 rev1=1 crypto rx=0x7fee500097c0 tx=0x7fee5001c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d534fa5abd705a7e server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:06:02.531 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:02.527+0000 7fee59ffb640 1 -- 172.21.15.16:0/963068374 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee500ab600 con 0x7fee5c0cb4e0 2023-05-22T16:06:02.531 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:02.527+0000 7fee59ffb640 1 -- 172.21.15.16:0/963068374 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee500a7050 con 0x7fee5c0cb4e0 2023-05-22T16:06:02.531 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:02.527+0000 7fee59ffb640 1 -- 172.21.15.16:0/963068374 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee500ab980 con 0x7fee5c0cb4e0 2023-05-22T16:06:02.532 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:02.527+0000 7fee61545640 1 -- 172.21.15.16:0/963068374 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fee5c0cb4e0 msgr2=0x7fee5c0cd880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:06:02.532 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:02.527+0000 7fee61545640 1 --2- 172.21.15.16:0/963068374 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fee5c0cb4e0 0x7fee5c0cd880 secure :-1 s=READY pgs=2607 cs=0 l=1 rev1=1 crypto rx=0x7fee500097c0 tx=0x7fee5001c480 comp rx=0 tx=0).stop 2023-05-22T16:06:02.532 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:02.527+0000 7fee61545640 1 -- 172.21.15.16:0/963068374 shutdown_connections 2023-05-22T16:06:02.532 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:02.527+0000 7fee61545640 1 --2- 172.21.15.16:0/963068374 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fee5c0afbf0 0x7fee5c0b0030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:02.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:02.527+0000 7fee61545640 1 --2- 172.21.15.16:0/963068374 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fee5c0cb4e0 0x7fee5c0cd880 unknown :-1 s=CLOSED pgs=2607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:02.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:02.527+0000 7fee61545640 1 -- 172.21.15.16:0/963068374 >> 172.21.15.16:0/963068374 conn(0x7fee5c069800 msgr2=0x7fee5c069c00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:06:02.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:02.527+0000 7fee61545640 1 -- 172.21.15.16:0/963068374 shutdown_connections 2023-05-22T16:06:02.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:02.527+0000 7fee61545640 1 -- 172.21.15.16:0/963068374 wait complete. 2023-05-22T16:06:02.534 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:02.531+0000 7fee61545640 1 Processor -- start 2023-05-22T16:06:02.534 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:02.531+0000 7fee61545640 1 -- start start 2023-05-22T16:06:02.534 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:02.531+0000 7fee61545640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fee5c0afbf0 0x7fee5c1577e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:02.535 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:02.531+0000 7fee61545640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fee5c0cb4e0 0x7fee5c0b2d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:02.535 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:02.531+0000 7fee61545640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fee5c1602b0 0x7fee5c0c3390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:02.535 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:02.531+0000 7fee61545640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee5c0b12f0 con 0x7fee5c0cb4e0 2023-05-22T16:06:02.535 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:02.531+0000 7fee61545640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee5c0b3730 con 0x7fee5c0afbf0 2023-05-22T16:06:02.535 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:02.531+0000 7fee61545640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee5c0b3a10 con 0x7fee5c1602b0 2023-05-22T16:06:02.535 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:02.531+0000 7fee5a7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fee5c0cb4e0 0x7fee5c0b2d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:02.535 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:02.531+0000 7fee5a7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fee5c0cb4e0 0x7fee5c0b2d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43398/0 (socket says 172.21.15.16:43398) 2023-05-22T16:06:02.535 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:02.531+0000 7fee5a7fc640 1 -- 172.21.15.16:0/974189327 learned_addr learned my addr 172.21.15.16:0/974189327 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:06:02.536 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:02.531+0000 7fee5affd640 1 --2- 172.21.15.16:0/974189327 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fee5c0afbf0 0x7fee5c1577e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:02.536 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:02.531+0000 7fee5b7fe640 1 --2- 172.21.15.16:0/974189327 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fee5c1602b0 0x7fee5c0c3390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:02.536 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:02.531+0000 7fee5a7fc640 1 -- 172.21.15.16:0/974189327 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fee5c1602b0 msgr2=0x7fee5c0c3390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:06:02.536 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:02.531+0000 7fee5a7fc640 1 --2- 172.21.15.16:0/974189327 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fee5c1602b0 0x7fee5c0c3390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:02.536 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:02.531+0000 7fee5a7fc640 1 -- 172.21.15.16:0/974189327 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fee5c0afbf0 msgr2=0x7fee5c1577e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:06:02.536 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:02.531+0000 7fee5a7fc640 1 --2- 172.21.15.16:0/974189327 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fee5c0afbf0 0x7fee5c1577e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:02.536 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:02.531+0000 7fee5a7fc640 1 -- 172.21.15.16:0/974189327 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee50009420 con 0x7fee5c0cb4e0 2023-05-22T16:06:02.537 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:02.531+0000 7fee5b7fe640 1 --2- 172.21.15.16:0/974189327 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fee5c1602b0 0x7fee5c0c3390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T16:06:02.537 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:02.531+0000 7fee5a7fc640 1 --2- 172.21.15.16:0/974189327 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fee5c0cb4e0 0x7fee5c0b2d50 secure :-1 s=READY pgs=2608 cs=0 l=1 rev1=1 crypto rx=0x7fee500ae040 tx=0x7fee50019220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:06:02.538 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:02.531+0000 7fee437fe640 1 -- 172.21.15.16:0/974189327 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee500b4840 con 0x7fee5c0cb4e0 2023-05-22T16:06:02.538 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:02.531+0000 7fee437fe640 1 -- 172.21.15.16:0/974189327 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee500b49c0 con 0x7fee5c0cb4e0 2023-05-22T16:06:02.538 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:02.531+0000 7fee437fe640 1 -- 172.21.15.16:0/974189327 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee500b4d70 con 0x7fee5c0cb4e0 2023-05-22T16:06:02.538 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:02.531+0000 7fee61545640 1 -- 172.21.15.16:0/974189327 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee5c0c1560 con 0x7fee5c0cb4e0 2023-05-22T16:06:02.538 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:02.531+0000 7fee61545640 1 -- 172.21.15.16:0/974189327 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee5c1640b0 con 0x7fee5c0cb4e0 2023-05-22T16:06:02.538 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:02.531+0000 7fee61545640 1 -- 172.21.15.16:0/974189327 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee5c0c16f0 con 0x7fee5c0cb4e0 2023-05-22T16:06:02.539 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:02.531+0000 7fee437fe640 1 -- 172.21.15.16:0/974189327 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 100) v1 ==== 84332+0+0 (secure 0 0 0) 0x7fee500b2040 con 0x7fee5c0cb4e0 2023-05-22T16:06:02.539 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:02.535+0000 7fee437fe640 1 --2- 172.21.15.16:0/974189327 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fee480654a0 0x7fee48067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:02.539 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:02.535+0000 7fee437fe640 1 -- 172.21.15.16:0/974189327 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(201..201 src has 1..201) v4 ==== 6317+0+0 (secure 0 0 0) 0x7fee5011cf50 con 0x7fee5c0cb4e0 2023-05-22T16:06:02.539 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:02.535+0000 7fee5affd640 1 --2- 172.21.15.16:0/974189327 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fee480654a0 0x7fee48067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:02.539 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:02.535+0000 7fee5affd640 1 --2- 172.21.15.16:0/974189327 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fee480654a0 0x7fee48067950 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fee4c000f70 tx=0x7fee4c007b60 comp rx=0 tx=0).ready entity=mgr.8195 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:06:02.541 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:02.539+0000 7fee437fe640 1 -- 172.21.15.16:0/974189327 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fee500ea8b0 con 0x7fee5c0cb4e0 2023-05-22T16:06:02.699 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:02.695+0000 7fee61545640 1 -- 172.21.15.16:0/974189327 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7fee5c0c3170 con 0x7fee5c0cb4e0 2023-05-22T16:06:02.709 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:02.707+0000 7fee437fe640 1 -- 172.21.15.16:0/974189327 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v67) v1 ==== 120+0+0 (secure 0 0 0) 0x7fee500b00b0 con 0x7fee5c0cb4e0 2023-05-22T16:06:02.712 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:02.707+0000 7fee61545640 1 -- 172.21.15.16:0/974189327 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fee480654a0 msgr2=0x7fee48067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:06:02.712 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:02.707+0000 7fee61545640 1 --2- 172.21.15.16:0/974189327 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fee480654a0 0x7fee48067950 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fee4c000f70 tx=0x7fee4c007b60 comp rx=0 tx=0).stop 2023-05-22T16:06:02.713 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:02.707+0000 7fee61545640 1 -- 172.21.15.16:0/974189327 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fee5c0cb4e0 msgr2=0x7fee5c0b2d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:06:02.713 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:02.707+0000 7fee61545640 1 --2- 172.21.15.16:0/974189327 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fee5c0cb4e0 0x7fee5c0b2d50 secure :-1 s=READY pgs=2608 cs=0 l=1 rev1=1 crypto rx=0x7fee500ae040 tx=0x7fee50019220 comp rx=0 tx=0).stop 2023-05-22T16:06:02.713 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:02.707+0000 7fee61545640 1 -- 172.21.15.16:0/974189327 shutdown_connections 2023-05-22T16:06:02.713 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:02.707+0000 7fee61545640 1 --2- 172.21.15.16:0/974189327 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fee5c1602b0 0x7fee5c0c3390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:02.713 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:02.707+0000 7fee61545640 1 --2- 172.21.15.16:0/974189327 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fee480654a0 0x7fee48067950 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:02.714 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:02.707+0000 7fee61545640 1 --2- 172.21.15.16:0/974189327 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fee5c0cb4e0 0x7fee5c0b2d50 unknown :-1 s=CLOSED pgs=2608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:02.714 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:02.707+0000 7fee61545640 1 --2- 172.21.15.16:0/974189327 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fee5c0afbf0 0x7fee5c1577e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:02.714 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:02.707+0000 7fee61545640 1 -- 172.21.15.16:0/974189327 >> 172.21.15.16:0/974189327 conn(0x7fee5c069800 msgr2=0x7fee5c0ada90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:06:02.714 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:02.711+0000 7fee61545640 1 -- 172.21.15.16:0/974189327 shutdown_connections 2023-05-22T16:06:02.715 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:02.711+0000 7fee61545640 1 -- 172.21.15.16:0/974189327 wait complete. 2023-05-22T16:06:02.731 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2023-05-22T16:06:02.859 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:02.855+0000 7fee8f626640 1 Processor -- start 2023-05-22T16:06:02.859 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:02.855+0000 7fee8f626640 1 -- start start 2023-05-22T16:06:02.860 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:02.855+0000 7fee8f626640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fee8815c040 0x7fee880ad0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:02.861 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:02.855+0000 7fee8f626640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fee881603e0 0x7fee880afbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:02.861 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:02.855+0000 7fee8f626640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee880b0550 con 0x7fee880b0130 2023-05-22T16:06:02.861 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:02.855+0000 7fee8f626640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee880ada70 con 0x7fee881603e0 2023-05-22T16:06:02.861 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:02.855+0000 7fee8f626640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee880ad790 con 0x7fee8815c040 2023-05-22T16:06:02.861 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:02.855+0000 7fee8cbc1640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fee881603e0 0x7fee880afbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:02.861 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:02.859+0000 7fee8cbc1640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fee881603e0 0x7fee880afbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43412/0 (socket says 172.21.15.16:43412) 2023-05-22T16:06:02.862 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:02.859+0000 7fee8cbc1640 1 -- 172.21.15.16:0/4261565787 learned_addr learned my addr 172.21.15.16:0/4261565787 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:06:02.862 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:02.859+0000 7fee8d3c2640 1 --2- 172.21.15.16:0/4261565787 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fee8815c040 0x7fee880ad0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:02.862 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:02.859+0000 7fee8cbc1640 1 -- 172.21.15.16:0/4261565787 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fee8815c040 msgr2=0x7fee880ad0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:06:02.862 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:02.859+0000 7fee8cbc1640 1 --2- 172.21.15.16:0/4261565787 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fee8815c040 0x7fee880ad0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:02.862 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:02.859+0000 7fee8cbc1640 1 -- 172.21.15.16:0/4261565787 >> v1:172.21.15.62:6789/0 conn(0x7fee880b0130 legacy=0x7fee880ce0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:06:02.862 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:02.859+0000 7fee8cbc1640 1 -- 172.21.15.16:0/4261565787 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee880c8940 con 0x7fee881603e0 2023-05-22T16:06:02.863 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:02.859+0000 7fee8d3c2640 1 --2- 172.21.15.16:0/4261565787 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fee8815c040 0x7fee880ad0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T16:06:02.864 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:02.859+0000 7fee8cbc1640 1 --2- 172.21.15.16:0/4261565787 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fee881603e0 0x7fee880afbf0 secure :-1 s=READY pgs=2609 cs=0 l=1 rev1=1 crypto rx=0x7fee78009870 tx=0x7fee7801c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f40b8875722d5eee server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:06:02.864 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:02.859+0000 7fee7ffff640 1 -- 172.21.15.16:0/4261565787 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee780ab600 con 0x7fee881603e0 2023-05-22T16:06:02.864 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:02.859+0000 7fee7ffff640 1 -- 172.21.15.16:0/4261565787 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee780ab760 con 0x7fee881603e0 2023-05-22T16:06:02.864 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:02.859+0000 7fee7ffff640 1 -- 172.21.15.16:0/4261565787 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee780abaf0 con 0x7fee881603e0 2023-05-22T16:06:02.864 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:02.859+0000 7fee8f626640 1 -- 172.21.15.16:0/4261565787 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fee881603e0 msgr2=0x7fee880afbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:06:02.864 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:02.859+0000 7fee8f626640 1 --2- 172.21.15.16:0/4261565787 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fee881603e0 0x7fee880afbf0 secure :-1 s=READY pgs=2609 cs=0 l=1 rev1=1 crypto rx=0x7fee78009870 tx=0x7fee7801c500 comp rx=0 tx=0).stop 2023-05-22T16:06:02.865 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:02.859+0000 7fee8f626640 1 -- 172.21.15.16:0/4261565787 shutdown_connections 2023-05-22T16:06:02.865 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:02.859+0000 7fee8f626640 1 --2- 172.21.15.16:0/4261565787 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fee881603e0 0x7fee880afbf0 unknown :-1 s=CLOSED pgs=2609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:02.865 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:02.859+0000 7fee8f626640 1 --2- 172.21.15.16:0/4261565787 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fee8815c040 0x7fee880ad0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:02.865 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:02.859+0000 7fee8f626640 1 -- 172.21.15.16:0/4261565787 >> 172.21.15.16:0/4261565787 conn(0x7fee88069800 msgr2=0x7fee88069c00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:06:02.865 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:02.859+0000 7fee8f626640 1 -- 172.21.15.16:0/4261565787 shutdown_connections 2023-05-22T16:06:02.865 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:02.859+0000 7fee8f626640 1 -- 172.21.15.16:0/4261565787 wait complete. 2023-05-22T16:06:02.867 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:02.863+0000 7fee8f626640 1 Processor -- start 2023-05-22T16:06:02.867 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:02.863+0000 7fee8f626640 1 -- start start 2023-05-22T16:06:02.868 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:02.863+0000 7fee8f626640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fee880b0130 0x7fee880c5c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:02.868 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:02.863+0000 7fee8f626640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fee8815c040 0x7fee881513e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:02.868 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:02.863+0000 7fee8f626640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fee881603e0 0x7fee880c7440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:02.868 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:02.863+0000 7fee8f626640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee88153990 con 0x7fee880b0130 2023-05-22T16:06:02.868 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:02.863+0000 7fee8f626640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee880adbb0 con 0x7fee8815c040 2023-05-22T16:06:02.868 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:02.863+0000 7fee8f626640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee880b3820 con 0x7fee881603e0 2023-05-22T16:06:02.869 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:02.863+0000 7fee8d3c2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fee880b0130 0x7fee880c5c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:02.869 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:02.863+0000 7fee8d3c2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fee880b0130 0x7fee880c5c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43422/0 (socket says 172.21.15.16:43422) 2023-05-22T16:06:02.869 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:02.863+0000 7fee8d3c2640 1 -- 172.21.15.16:0/275025802 learned_addr learned my addr 172.21.15.16:0/275025802 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:06:02.869 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:02.863+0000 7fee8cbc1640 1 --2- 172.21.15.16:0/275025802 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fee8815c040 0x7fee881513e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:02.869 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:02.863+0000 7fee8dbc3640 1 --2- 172.21.15.16:0/275025802 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fee881603e0 0x7fee880c7440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:02.869 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:02.863+0000 7fee8d3c2640 1 -- 172.21.15.16:0/275025802 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fee881603e0 msgr2=0x7fee880c7440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:06:02.869 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:02.863+0000 7fee8d3c2640 1 --2- 172.21.15.16:0/275025802 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fee881603e0 0x7fee880c7440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:02.869 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:02.863+0000 7fee8d3c2640 1 -- 172.21.15.16:0/275025802 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fee8815c040 msgr2=0x7fee881513e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:06:02.870 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:02.863+0000 7fee8d3c2640 1 --2- 172.21.15.16:0/275025802 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fee8815c040 0x7fee881513e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:02.870 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:02.863+0000 7fee8d3c2640 1 -- 172.21.15.16:0/275025802 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee780094d0 con 0x7fee880b0130 2023-05-22T16:06:02.870 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:02.863+0000 7fee8dbc3640 1 --2- 172.21.15.16:0/275025802 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fee881603e0 0x7fee880c7440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T16:06:02.870 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:02.863+0000 7fee8d3c2640 1 --2- 172.21.15.16:0/275025802 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fee880b0130 0x7fee880c5c30 secure :-1 s=READY pgs=2610 cs=0 l=1 rev1=1 crypto rx=0x7fee8400c7c0 tx=0x7fee8400cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:06:02.870 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:02.863+0000 7fee7dffb640 1 -- 172.21.15.16:0/275025802 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee84007a30 con 0x7fee880b0130 2023-05-22T16:06:02.870 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:02.863+0000 7fee7dffb640 1 -- 172.21.15.16:0/275025802 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee84007b90 con 0x7fee880b0130 2023-05-22T16:06:02.870 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:02.863+0000 7fee7dffb640 1 -- 172.21.15.16:0/275025802 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee8400ce70 con 0x7fee880b0130 2023-05-22T16:06:02.871 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:02.863+0000 7fee8f626640 1 -- 172.21.15.16:0/275025802 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee880cc050 con 0x7fee880b0130 2023-05-22T16:06:02.871 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:02.863+0000 7fee8f626640 1 -- 172.21.15.16:0/275025802 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee88160810 con 0x7fee880b0130 2023-05-22T16:06:02.871 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:02.863+0000 7fee8f626640 1 -- 172.21.15.16:0/275025802 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee880cc1e0 con 0x7fee880b0130 2023-05-22T16:06:02.872 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:02.867+0000 7fee7dffb640 1 -- 172.21.15.16:0/275025802 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 100) v1 ==== 84332+0+0 (secure 0 0 0) 0x7fee840113f0 con 0x7fee880b0130 2023-05-22T16:06:02.872 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:02.867+0000 7fee7dffb640 1 --2- 172.21.15.16:0/275025802 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fee6c0654a0 0x7fee6c067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:02.872 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:02.867+0000 7fee7dffb640 1 -- 172.21.15.16:0/275025802 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(201..201 src has 1..201) v4 ==== 6317+0+0 (secure 0 0 0) 0x7fee84083070 con 0x7fee880b0130 2023-05-22T16:06:02.872 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:02.867+0000 7fee8cbc1640 1 --2- 172.21.15.16:0/275025802 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fee6c0654a0 0x7fee6c067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:02.872 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:02.867+0000 7fee8cbc1640 1 --2- 172.21.15.16:0/275025802 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fee6c0654a0 0x7fee6c067950 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fee780af000 tx=0x7fee780a7000 comp rx=0 tx=0).ready entity=mgr.8195 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:06:02.873 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:02.871+0000 7fee7dffb640 1 -- 172.21.15.16:0/275025802 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fee84050970 con 0x7fee880b0130 2023-05-22T16:06:03.026 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:03.023+0000 7fee8f626640 1 -- 172.21.15.16:0/275025802 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fee880b1f40 con 0x7fee880b0130 2023-05-22T16:06:03.033 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:03.031+0000 7fee7dffb640 1 -- 172.21.15.16:0/275025802 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v68) v1 ==== 104+0+0 (secure 0 0 0) 0x7fee880b1f40 con 0x7fee880b0130 2023-05-22T16:06:03.035 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:03.031+0000 7fee8f626640 1 -- 172.21.15.16:0/275025802 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fee6c0654a0 msgr2=0x7fee6c067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:06:03.035 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:03.031+0000 7fee8f626640 1 --2- 172.21.15.16:0/275025802 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fee6c0654a0 0x7fee6c067950 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fee780af000 tx=0x7fee780a7000 comp rx=0 tx=0).stop 2023-05-22T16:06:03.035 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:03.031+0000 7fee8f626640 1 -- 172.21.15.16:0/275025802 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fee880b0130 msgr2=0x7fee880c5c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:06:03.035 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:03.031+0000 7fee8f626640 1 --2- 172.21.15.16:0/275025802 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fee880b0130 0x7fee880c5c30 secure :-1 s=READY pgs=2610 cs=0 l=1 rev1=1 crypto rx=0x7fee8400c7c0 tx=0x7fee8400cc80 comp rx=0 tx=0).stop 2023-05-22T16:06:03.036 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:03.031+0000 7fee8f626640 1 -- 172.21.15.16:0/275025802 shutdown_connections 2023-05-22T16:06:03.036 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:03.031+0000 7fee8f626640 1 --2- 172.21.15.16:0/275025802 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fee881603e0 0x7fee880c7440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:03.037 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:03.031+0000 7fee8f626640 1 --2- 172.21.15.16:0/275025802 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fee8815c040 0x7fee881513e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:03.037 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:03.031+0000 7fee8f626640 1 --2- 172.21.15.16:0/275025802 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fee6c0654a0 0x7fee6c067950 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:03.037 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:03.031+0000 7fee8f626640 1 --2- 172.21.15.16:0/275025802 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fee880b0130 0x7fee880c5c30 unknown :-1 s=CLOSED pgs=2610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:03.038 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:03.031+0000 7fee8f626640 1 -- 172.21.15.16:0/275025802 >> 172.21.15.16:0/275025802 conn(0x7fee88069800 msgr2=0x7fee88070b00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:06:03.038 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:03.031+0000 7fee8f626640 1 -- 172.21.15.16:0/275025802 shutdown_connections 2023-05-22T16:06:03.038 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:03.031+0000 7fee8f626640 1 -- 172.21.15.16:0/275025802 wait complete. 2023-05-22T16:06:03.053 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2023-05-22T16:06:03.175 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:03.171+0000 7f914b8c6640 1 Processor -- start 2023-05-22T16:06:03.176 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:03.171+0000 7f914b8c6640 1 -- start start 2023-05-22T16:06:03.176 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:03.175+0000 7f914b8c6640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f914415d3d0 0x7f91440cde00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:03.177 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:03.175+0000 7f914b8c6640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f91440ca880 0x7f91440ce340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:03.177 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:03.175+0000 7f914b8c6640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91440cb7a0 con 0x7f91440ce880 2023-05-22T16:06:03.177 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:03.175+0000 7f914b8c6640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91440afa60 con 0x7f914415d3d0 2023-05-22T16:06:03.177 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:03.175+0000 7f914b8c6640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91440b2770 con 0x7f91440ca880 2023-05-22T16:06:03.178 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:03.175+0000 7f9149662640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f914415d3d0 0x7f91440cde00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:03.178 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:03.175+0000 7f9148e61640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f91440ca880 0x7f91440ce340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:03.178 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:03.175+0000 7f9149662640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f914415d3d0 0x7f91440cde00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.16:43306/0 (socket says 172.21.15.16:43306) 2023-05-22T16:06:03.178 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:03.175+0000 7f9149662640 1 -- 172.21.15.16:0/3195350574 learned_addr learned my addr 172.21.15.16:0/3195350574 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:06:03.179 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:03.175+0000 7f9149662640 1 -- 172.21.15.16:0/3195350574 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f91440ca880 msgr2=0x7f91440ce340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:06:03.179 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:03.175+0000 7f9149662640 1 --2- 172.21.15.16:0/3195350574 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f91440ca880 0x7f91440ce340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:03.179 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:03.175+0000 7f9149662640 1 -- 172.21.15.16:0/3195350574 >> v1:172.21.15.62:6789/0 conn(0x7f91440ce880 legacy=0x7f914415d8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:06:03.179 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:03.175+0000 7f9149662640 1 -- 172.21.15.16:0/3195350574 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91440cc600 con 0x7f914415d3d0 2023-05-22T16:06:03.180 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:03.175+0000 7f9149662640 1 --2- 172.21.15.16:0/3195350574 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f914415d3d0 0x7f91440cde00 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f9140006fd0 tx=0x7f914001c520 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e19af25300ce3e8b server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:06:03.181 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:03.179+0000 7f913bfff640 1 -- 172.21.15.16:0/3195350574 <== mon.1 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91400ab600 con 0x7f914415d3d0 2023-05-22T16:06:03.181 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:03.179+0000 7f913bfff640 1 -- 172.21.15.16:0/3195350574 <== mon.1 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91400a7050 con 0x7f914415d3d0 2023-05-22T16:06:03.181 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:03.179+0000 7f913bfff640 1 -- 172.21.15.16:0/3195350574 <== mon.1 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91400ab980 con 0x7f914415d3d0 2023-05-22T16:06:03.182 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:03.179+0000 7f914b8c6640 1 -- 172.21.15.16:0/3195350574 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f914415d3d0 msgr2=0x7f91440cde00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:06:03.182 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:03.179+0000 7f914b8c6640 1 --2- 172.21.15.16:0/3195350574 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f914415d3d0 0x7f91440cde00 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f9140006fd0 tx=0x7f914001c520 comp rx=0 tx=0).stop 2023-05-22T16:06:03.182 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:03.179+0000 7f914b8c6640 1 -- 172.21.15.16:0/3195350574 shutdown_connections 2023-05-22T16:06:03.182 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:03.179+0000 7f914b8c6640 1 --2- 172.21.15.16:0/3195350574 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f91440ca880 0x7f91440ce340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:03.182 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:03.179+0000 7f914b8c6640 1 --2- 172.21.15.16:0/3195350574 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f914415d3d0 0x7f91440cde00 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:03.182 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:03.179+0000 7f914b8c6640 1 -- 172.21.15.16:0/3195350574 >> 172.21.15.16:0/3195350574 conn(0x7f9144099620 msgr2=0x7f91440d02c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:06:03.182 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:03.179+0000 7f914b8c6640 1 -- 172.21.15.16:0/3195350574 shutdown_connections 2023-05-22T16:06:03.183 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:03.179+0000 7f914b8c6640 1 -- 172.21.15.16:0/3195350574 wait complete. 2023-05-22T16:06:03.184 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:03.179+0000 7f914b8c6640 1 Processor -- start 2023-05-22T16:06:03.184 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:03.179+0000 7f914b8c6640 1 -- start start 2023-05-22T16:06:03.185 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:03.179+0000 7f914b8c6640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f91440ca880 0x7f9144158470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:03.185 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:03.179+0000 7f914b8c6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f91440ce880 0x7f91440b5cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:03.185 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:03.179+0000 7f914b8c6640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f914415d3d0 0x7f9144157d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:03.185 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:03.179+0000 7f914b8c6640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91440b61f0 con 0x7f91440ce880 2023-05-22T16:06:03.186 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:03.179+0000 7f914b8c6640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91441582c0 con 0x7f914415d3d0 2023-05-22T16:06:03.186 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:03.179+0000 7f914b8c6640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f914406adf0 con 0x7f91440ca880 2023-05-22T16:06:03.186 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:03.179+0000 7f9148e61640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f91440ce880 0x7f91440b5cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:03.186 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:03.179+0000 7f9148e61640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f91440ce880 0x7f91440b5cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43428/0 (socket says 172.21.15.16:43428) 2023-05-22T16:06:03.186 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:03.179+0000 7f9148e61640 1 -- 172.21.15.16:0/1988764206 learned_addr learned my addr 172.21.15.16:0/1988764206 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:06:03.186 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:03.179+0000 7f9149e63640 1 --2- 172.21.15.16:0/1988764206 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f914415d3d0 0x7f9144157d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:03.187 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:03.179+0000 7f9149662640 1 --2- 172.21.15.16:0/1988764206 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f91440ca880 0x7f9144158470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:03.187 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:03.179+0000 7f9148e61640 1 -- 172.21.15.16:0/1988764206 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f91440ca880 msgr2=0x7f9144158470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:06:03.187 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:03.179+0000 7f9148e61640 1 --2- 172.21.15.16:0/1988764206 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f91440ca880 0x7f9144158470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:03.187 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:03.179+0000 7f9148e61640 1 -- 172.21.15.16:0/1988764206 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f914415d3d0 msgr2=0x7f9144157d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:06:03.187 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:03.179+0000 7f9148e61640 1 --2- 172.21.15.16:0/1988764206 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f914415d3d0 0x7f9144157d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:03.187 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:03.179+0000 7f9148e61640 1 -- 172.21.15.16:0/1988764206 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9140009480 con 0x7f91440ce880 2023-05-22T16:06:03.188 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:03.179+0000 7f9149662640 1 --2- 172.21.15.16:0/1988764206 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f91440ca880 0x7f9144158470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T16:06:03.188 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:03.183+0000 7f9148e61640 1 --2- 172.21.15.16:0/1988764206 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f91440ce880 0x7f91440b5cb0 secure :-1 s=READY pgs=2611 cs=0 l=1 rev1=1 crypto rx=0x7f913400e930 tx=0x7f913400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:06:03.188 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:03.183+0000 7f9139ffb640 1 -- 172.21.15.16:0/1988764206 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9134009690 con 0x7f91440ce880 2023-05-22T16:06:03.188 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:03.183+0000 7f9139ffb640 1 -- 172.21.15.16:0/1988764206 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f913400ca00 con 0x7f91440ce880 2023-05-22T16:06:03.188 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:03.183+0000 7f914b8c6640 1 -- 172.21.15.16:0/1988764206 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91440cbfb0 con 0x7f91440ce880 2023-05-22T16:06:03.188 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:03.183+0000 7f9139ffb640 1 -- 172.21.15.16:0/1988764206 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f913400cd70 con 0x7f91440ce880 2023-05-22T16:06:03.189 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:03.183+0000 7f914b8c6640 1 -- 172.21.15.16:0/1988764206 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91440cca90 con 0x7f91440ce880 2023-05-22T16:06:03.189 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:03.183+0000 7f914b8c6640 1 -- 172.21.15.16:0/1988764206 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91440cc290 con 0x7f91440ce880 2023-05-22T16:06:03.190 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:03.183+0000 7f9139ffb640 1 -- 172.21.15.16:0/1988764206 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 100) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f9134013470 con 0x7f91440ce880 2023-05-22T16:06:03.190 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:03.183+0000 7f9139ffb640 1 --2- 172.21.15.16:0/1988764206 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f91280654a0 0x7f9128067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:03.190 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:03.183+0000 7f9139ffb640 1 -- 172.21.15.16:0/1988764206 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(201..201 src has 1..201) v4 ==== 6317+0+0 (secure 0 0 0) 0x7f9134082c70 con 0x7f91440ce880 2023-05-22T16:06:03.190 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:03.183+0000 7f9149662640 1 --2- 172.21.15.16:0/1988764206 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f91280654a0 0x7f9128067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:03.190 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:03.183+0000 7f9149662640 1 --2- 172.21.15.16:0/1988764206 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f91280654a0 0x7f9128067950 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f91400ad000 tx=0x7f91400a6000 comp rx=0 tx=0).ready entity=mgr.8195 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:06:03.192 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:03.187+0000 7f9139ffb640 1 -- 172.21.15.16:0/1988764206 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f91340507f0 con 0x7f91440ce880 2023-05-22T16:06:03.353 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:03.351+0000 7f914b8c6640 1 -- 172.21.15.16:0/1988764206 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7f9144071e00 con 0x7f91440ce880 2023-05-22T16:06:03.359 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:03.355+0000 7f9139ffb640 1 -- 172.21.15.16:0/1988764206 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v69) v1 ==== 115+0+0 (secure 0 0 0) 0x7f9134053e10 con 0x7f91440ce880 2023-05-22T16:06:03.361 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:03.359+0000 7f914b8c6640 1 -- 172.21.15.16:0/1988764206 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f91280654a0 msgr2=0x7f9128067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:06:03.362 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:03.359+0000 7f914b8c6640 1 --2- 172.21.15.16:0/1988764206 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f91280654a0 0x7f9128067950 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f91400ad000 tx=0x7f91400a6000 comp rx=0 tx=0).stop 2023-05-22T16:06:03.362 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:03.359+0000 7f914b8c6640 1 -- 172.21.15.16:0/1988764206 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f91440ce880 msgr2=0x7f91440b5cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:06:03.362 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:03.359+0000 7f914b8c6640 1 --2- 172.21.15.16:0/1988764206 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f91440ce880 0x7f91440b5cb0 secure :-1 s=READY pgs=2611 cs=0 l=1 rev1=1 crypto rx=0x7f913400e930 tx=0x7f913400edf0 comp rx=0 tx=0).stop 2023-05-22T16:06:03.362 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:03.359+0000 7f914b8c6640 1 -- 172.21.15.16:0/1988764206 shutdown_connections 2023-05-22T16:06:03.363 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:03.359+0000 7f914b8c6640 1 --2- 172.21.15.16:0/1988764206 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f914415d3d0 0x7f9144157d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:03.363 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:03.359+0000 7f914b8c6640 1 --2- 172.21.15.16:0/1988764206 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f91280654a0 0x7f9128067950 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:03.363 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:03.359+0000 7f914b8c6640 1 --2- 172.21.15.16:0/1988764206 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f91440ce880 0x7f91440b5cb0 unknown :-1 s=CLOSED pgs=2611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:03.363 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:03.359+0000 7f914b8c6640 1 --2- 172.21.15.16:0/1988764206 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f91440ca880 0x7f9144158470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:03.364 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:03.359+0000 7f914b8c6640 1 -- 172.21.15.16:0/1988764206 >> 172.21.15.16:0/1988764206 conn(0x7f9144099620 msgr2=0x7f91440cfff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:06:03.364 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:03.359+0000 7f914b8c6640 1 -- 172.21.15.16:0/1988764206 shutdown_connections 2023-05-22T16:06:03.364 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:03.359+0000 7f914b8c6640 1 -- 172.21.15.16:0/1988764206 wait complete. 2023-05-22T16:06:03.381 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedulesSnapdir.test_snap_dir_name' 2023-05-22T16:06:03.500 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:03.495+0000 7f4785767640 1 Processor -- start 2023-05-22T16:06:03.501 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:03.495+0000 7f4785767640 1 -- start start 2023-05-22T16:06:03.503 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:03.499+0000 7f4785767640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f47800cfa20 0x7f47800d0020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:03.504 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:03.499+0000 7f4785767640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47800d0990 con 0x7f47800d27d0 2023-05-22T16:06:03.504 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:03.499+0000 7f4785767640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47800cf190 con 0x7f47800d0560 2023-05-22T16:06:03.504 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:03.499+0000 7f4785767640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f478015c4f0 con 0x7f47800cfa20 2023-05-22T16:06:03.504 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:03.499+0000 7f477f7fe640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f47800d27d0 0x7f47800cd340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:49814/0 (socket says 172.21.15.16:49814) 2023-05-22T16:06:03.504 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:03.499+0000 7f477f7fe640 1 -- 172.21.15.16:0/805881651 learned_addr learned my addr 172.21.15.16:0/805881651 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T16:06:03.505 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:03.499+0000 7f477effd640 1 --2- 172.21.15.16:0/805881651 >> v2:172.21.15.62:3300/0 conn(0x7f47800cfa20 0x7f47800d0020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:03.505 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:03.499+0000 7f477e7fc640 1 -- 172.21.15.16:0/805881651 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1705628564 0 0) 0x7f47800d0990 con 0x7f47800d27d0 2023-05-22T16:06:03.505 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:03.499+0000 7f477e7fc640 1 -- 172.21.15.16:0/805881651 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f476c003530 con 0x7f47800d27d0 2023-05-22T16:06:03.505 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:03.499+0000 7f477e7fc640 1 -- 172.21.15.16:0/805881651 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3353573821 0 0) 0x7f476c003530 con 0x7f47800d27d0 2023-05-22T16:06:03.505 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:03.499+0000 7f477e7fc640 1 -- 172.21.15.16:0/805881651 >> v2:172.21.15.62:3300/0 conn(0x7f47800cfa20 msgr2=0x7f47800d0020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:06:03.505 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:03.499+0000 7f477e7fc640 1 --2- 172.21.15.16:0/805881651 >> v2:172.21.15.62:3300/0 conn(0x7f47800cfa20 0x7f47800d0020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:03.505 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:03.499+0000 7f477e7fc640 1 -- 172.21.15.16:0/805881651 >> v1:172.21.15.62:6789/0 conn(0x7f47800d0560 legacy=0x7f47800d20c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:06:03.505 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:03.499+0000 7f477e7fc640 1 -- 172.21.15.16:0/805881651 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47800d3360 con 0x7f47800d27d0 2023-05-22T16:06:03.506 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:03.499+0000 7f477e7fc640 1 -- 172.21.15.16:0/805881651 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7f4778001a50 con 0x7f47800d27d0 2023-05-22T16:06:03.506 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:03.499+0000 7f477e7fc640 1 -- 172.21.15.16:0/805881651 >> v1:172.21.15.16:6789/0 conn(0x7f47800d27d0 legacy=0x7f47800cd340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:06:03.506 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:03.499+0000 7f477e7fc640 1 --2- 172.21.15.16:0/805881651 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f476c003980 0x7f476c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:03.506 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:03.499+0000 7f477e7fc640 1 --2- 172.21.15.16:0/805881651 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f476c0047d0 0x7f476c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:03.506 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:03.499+0000 7f477e7fc640 1 --2- 172.21.15.16:0/805881651 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f476c005180 0x7f476c00b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:03.506 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:03.499+0000 7f477e7fc640 1 -- 172.21.15.16:0/805881651 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f478015c4f0 con 0x7f476c005180 2023-05-22T16:06:03.506 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:03.499+0000 7f477e7fc640 1 -- 172.21.15.16:0/805881651 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f476c00bfe0 con 0x7f476c0047d0 2023-05-22T16:06:03.507 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:03.499+0000 7f477e7fc640 1 -- 172.21.15.16:0/805881651 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f476c00c1a0 con 0x7f476c003980 2023-05-22T16:06:03.507 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:03.499+0000 7f477f7fe640 1 --2- 172.21.15.16:0/805881651 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f476c005180 0x7f476c00b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:03.507 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:03.499+0000 7f4777fff640 1 --2- 172.21.15.16:0/805881651 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f476c0047d0 0x7f476c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:03.507 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:03.499+0000 7f477effd640 1 --2- 172.21.15.16:0/805881651 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f476c003980 0x7f476c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:03.507 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:03.499+0000 7f477f7fe640 1 -- 172.21.15.16:0/805881651 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f476c003980 msgr2=0x7f476c003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:06:03.507 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:03.499+0000 7f477f7fe640 1 --2- 172.21.15.16:0/805881651 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f476c003980 0x7f476c003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:03.507 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:03.499+0000 7f477f7fe640 1 -- 172.21.15.16:0/805881651 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f476c0047d0 msgr2=0x7f476c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:06:03.507 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:03.499+0000 7f477f7fe640 1 --2- 172.21.15.16:0/805881651 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f476c0047d0 0x7f476c004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:03.508 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:03.499+0000 7f477f7fe640 1 -- 172.21.15.16:0/805881651 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f476c003530 con 0x7f476c005180 2023-05-22T16:06:03.508 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:03.499+0000 7f4777fff640 1 --2- 172.21.15.16:0/805881651 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f476c0047d0 0x7f476c004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T16:06:03.511 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:03.503+0000 7f477f7fe640 1 -- 172.21.15.16:0/805881651 reap_dead start 2023-05-22T16:06:03.511 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:03.503+0000 7f477f7fe640 1 --2- 172.21.15.16:0/805881651 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f476c005180 0x7f476c00b990 secure :-1 s=READY pgs=2613 cs=0 l=1 rev1=1 crypto rx=0x7f477800a7d0 tx=0x7f477801d2e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3a32f0b6e414147 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:06:03.511 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:03.503+0000 7f477e7fc640 1 -- 172.21.15.16:0/805881651 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47780ac620 con 0x7f476c005180 2023-05-22T16:06:03.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:03.503+0000 7f477e7fc640 1 -- 172.21.15.16:0/805881651 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47780a8070 con 0x7f476c005180 2023-05-22T16:06:03.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:03.503+0000 7f477e7fc640 1 -- 172.21.15.16:0/805881651 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47780aca00 con 0x7f476c005180 2023-05-22T16:06:03.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:03.503+0000 7f4785767640 1 -- 172.21.15.16:0/805881651 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f476c005180 msgr2=0x7f476c00b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:06:03.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:03.503+0000 7f4785767640 1 --2- 172.21.15.16:0/805881651 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f476c005180 0x7f476c00b990 secure :-1 s=READY pgs=2613 cs=0 l=1 rev1=1 crypto rx=0x7f477800a7d0 tx=0x7f477801d2e0 comp rx=0 tx=0).stop 2023-05-22T16:06:03.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:03.503+0000 7f4785767640 1 -- 172.21.15.16:0/805881651 shutdown_connections 2023-05-22T16:06:03.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:03.503+0000 7f4785767640 1 --2- 172.21.15.16:0/805881651 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f476c005180 0x7f476c00b990 unknown :-1 s=CLOSED pgs=2613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:03.513 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:03.503+0000 7f4785767640 1 -- 172.21.15.16:0/805881651 >> 172.21.15.16:0/805881651 conn(0x7f4780071850 msgr2=0x7f478009b830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:06:03.513 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:03.503+0000 7f4785767640 1 -- 172.21.15.16:0/805881651 shutdown_connections 2023-05-22T16:06:03.513 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:03.503+0000 7f4785767640 1 -- 172.21.15.16:0/805881651 wait complete. 2023-05-22T16:06:03.513 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:03.503+0000 7f4785767640 1 Processor -- start 2023-05-22T16:06:03.513 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:03.503+0000 7f4785767640 1 -- start start 2023-05-22T16:06:03.513 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:03.507+0000 7f4785767640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f476c005180 0x7f47800b7bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:03.514 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:03.507+0000 7f4785767640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f47800b8100 0x7f478014ffe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:03.514 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:03.507+0000 7f4785767640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4780150520 0x7f47800b23c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:03.514 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:03.507+0000 7f4785767640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47800b6630 con 0x7f476c005180 2023-05-22T16:06:03.514 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:03.507+0000 7f4785767640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47801f2040 con 0x7f4780150520 2023-05-22T16:06:03.514 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:03.507+0000 7f4785767640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47801f25b0 con 0x7f47800b8100 2023-05-22T16:06:03.514 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:03.507+0000 7f477effd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f476c005180 0x7f47800b7bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:03.514 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:03.507+0000 7f477effd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f476c005180 0x7f47800b7bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43450/0 (socket says 172.21.15.16:43450) 2023-05-22T16:06:03.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:03.507+0000 7f477effd640 1 -- 172.21.15.16:0/2241349474 learned_addr learned my addr 172.21.15.16:0/2241349474 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:06:03.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:03.507+0000 7f4777fff640 1 --2- 172.21.15.16:0/2241349474 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f47800b8100 0x7f478014ffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:03.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:03.507+0000 7f477f7fe640 1 --2- 172.21.15.16:0/2241349474 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4780150520 0x7f47800b23c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:03.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:03.507+0000 7f477effd640 1 -- 172.21.15.16:0/2241349474 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f47800b8100 msgr2=0x7f478014ffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:06:03.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:03.507+0000 7f477effd640 1 --2- 172.21.15.16:0/2241349474 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f47800b8100 0x7f478014ffe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:03.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:03.507+0000 7f477effd640 1 -- 172.21.15.16:0/2241349474 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4780150520 msgr2=0x7f47800b23c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:06:03.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:03.507+0000 7f477effd640 1 --2- 172.21.15.16:0/2241349474 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4780150520 0x7f47800b23c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:03.516 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:03.507+0000 7f477effd640 1 -- 172.21.15.16:0/2241349474 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47800ccb40 con 0x7f476c005180 2023-05-22T16:06:03.516 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:03.507+0000 7f4777fff640 1 --2- 172.21.15.16:0/2241349474 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f47800b8100 0x7f478014ffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T16:06:03.516 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:03.507+0000 7f477effd640 1 --2- 172.21.15.16:0/2241349474 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f476c005180 0x7f47800b7bc0 secure :-1 s=READY pgs=2614 cs=0 l=1 rev1=1 crypto rx=0x7f47700115c0 tx=0x7f4770011a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:06:03.516 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:03.507+0000 7f47777fe640 1 -- 172.21.15.16:0/2241349474 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f477000f8f0 con 0x7f476c005180 2023-05-22T16:06:03.516 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:03.507+0000 7f47777fe640 1 -- 172.21.15.16:0/2241349474 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f477000fa50 con 0x7f476c005180 2023-05-22T16:06:03.516 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:03.507+0000 7f4785767640 1 -- 172.21.15.16:0/2241349474 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47800d3080 con 0x7f476c005180 2023-05-22T16:06:03.516 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:03.507+0000 7f47777fe640 1 -- 172.21.15.16:0/2241349474 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f477000fdc0 con 0x7f476c005180 2023-05-22T16:06:03.517 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:03.507+0000 7f4785767640 1 -- 172.21.15.16:0/2241349474 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47800cce20 con 0x7f476c005180 2023-05-22T16:06:03.517 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:03.507+0000 7f4785767640 1 -- 172.21.15.16:0/2241349474 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4780150920 con 0x7f476c005180 2023-05-22T16:06:03.517 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:03.507+0000 7f47777fe640 1 -- 172.21.15.16:0/2241349474 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 100) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f4770015460 con 0x7f476c005180 2023-05-22T16:06:03.518 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:03.507+0000 7f47777fe640 1 --2- 172.21.15.16:0/2241349474 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f47600654a0 0x7f4760067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:03.518 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:03.507+0000 7f47777fe640 1 -- 172.21.15.16:0/2241349474 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(201..201 src has 1..201) v4 ==== 6317+0+0 (secure 0 0 0) 0x7f4770083300 con 0x7f476c005180 2023-05-22T16:06:03.518 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:03.507+0000 7f4777fff640 1 --2- 172.21.15.16:0/2241349474 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f47600654a0 0x7f4760067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:03.519 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:03.507+0000 7f4777fff640 1 --2- 172.21.15.16:0/2241349474 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f47600654a0 0x7f4760067950 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f4768007fd0 tx=0x7f4768009040 comp rx=0 tx=0).ready entity=mgr.8195 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:06:03.519 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:03.511+0000 7f47777fe640 1 -- 172.21.15.16:0/2241349474 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f4770050d10 con 0x7f476c005180 2023-05-22T16:06:03.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:03.723+0000 7f4785767640 1 -- 172.21.15.16:0/2241349474 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedulesSnapdir.test_snap_dir_name"]} v 0) v1 -- 0x7f47800b3270 con 0x7f476c005180 2023-05-22T16:06:04.354 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:04.351+0000 7f47777fe640 1 -- 172.21.15.16:0/2241349474 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedulesSnapdir.test_snap_dir_name"]}]=0 v1403) v1 ==== 155+0+0 (secure 0 0 0) 0x7f4770054330 con 0x7f476c005180 2023-05-22T16:06:04.358 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:04.355+0000 7f4785767640 1 -- 172.21.15.16:0/2241349474 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f47600654a0 msgr2=0x7f4760067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:06:04.358 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:04.355+0000 7f4785767640 1 --2- 172.21.15.16:0/2241349474 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f47600654a0 0x7f4760067950 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f4768007fd0 tx=0x7f4768009040 comp rx=0 tx=0).stop 2023-05-22T16:06:04.358 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:04.355+0000 7f4785767640 1 -- 172.21.15.16:0/2241349474 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f476c005180 msgr2=0x7f47800b7bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:06:04.359 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:04.355+0000 7f4785767640 1 --2- 172.21.15.16:0/2241349474 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f476c005180 0x7f47800b7bc0 secure :-1 s=READY pgs=2614 cs=0 l=1 rev1=1 crypto rx=0x7f47700115c0 tx=0x7f4770011a80 comp rx=0 tx=0).stop 2023-05-22T16:06:04.359 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:04.355+0000 7f4785767640 1 -- 172.21.15.16:0/2241349474 shutdown_connections 2023-05-22T16:06:04.359 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:04.355+0000 7f4785767640 1 --2- 172.21.15.16:0/2241349474 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4780150520 0x7f47800b23c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:04.359 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:04.355+0000 7f4785767640 1 --2- 172.21.15.16:0/2241349474 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f47800b8100 0x7f478014ffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:04.359 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:04.355+0000 7f4785767640 1 --2- 172.21.15.16:0/2241349474 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f47600654a0 0x7f4760067950 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:04.359 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:04.355+0000 7f4785767640 1 --2- 172.21.15.16:0/2241349474 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f476c005180 0x7f47800b7bc0 unknown :-1 s=CLOSED pgs=2614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:04.359 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:04.355+0000 7f4785767640 1 -- 172.21.15.16:0/2241349474 >> 172.21.15.16:0/2241349474 conn(0x7f4780071850 msgr2=0x7f47800ce350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:06:04.360 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:04.355+0000 7f4785767640 1 -- 172.21.15.16:0/2241349474 shutdown_connections 2023-05-22T16:06:04.360 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:04.355+0000 7f4785767640 1 -- 172.21.15.16:0/2241349474 wait complete. 2023-05-22T16:06:04.384 INFO:tasks.cephfs_test_runner:Test the correctness of snap directory name ... ok 2023-05-22T16:06:04.385 INFO:tasks.cephfs_test_runner: 2023-05-22T16:06:04.385 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2023-05-22T16:06:04.385 INFO:tasks.cephfs_test_runner:Ran 10 tests in 2306.385s 2023-05-22T16:06:04.386 INFO:tasks.cephfs_test_runner: 2023-05-22T16:06:04.386 INFO:tasks.cephfs_test_runner:OK 2023-05-22T16:06:04.386 DEBUG:teuthology.run_tasks:Unwinding manager cephfs_test_runner 2023-05-22T16:06:04.397 DEBUG:teuthology.run_tasks:Unwinding manager ceph-fuse 2023-05-22T16:06:04.414 INFO:tasks.ceph_fuse:Unmounting ceph-fuse clients... 2023-05-22T16:06:04.415 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-05-22T16:06:04.415 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2023-05-22T16:06:04.421 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16345192k,nr_inodes=4086298,mode=755,inode64 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3276544k,mode=755,inode64 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev,inode64 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=15732 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3276540k,nr_inodes=819135,mode=700,uid=1000,gid=1271,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3276544k,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-05-22T16:06:04.422 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-05-22T16:06:04.422 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2023-05-22T16:06:04.469 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16345192k,nr_inodes=4086298,mode=755,inode64 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3276544k,mode=755,inode64 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev,inode64 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=15732 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3276540k,nr_inodes=819135,mode=700,uid=1000,gid=1271,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3276544k,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-05-22T16:06:04.470 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2023-05-22T16:06:04.470 DEBUG:teuthology.orchestra.run.smithi062:> dd if=/proc/self/mounts of=/dev/stdout 2023-05-22T16:06:04.476 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16342116k,nr_inodes=4085529,mode=755,inode64 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275932k,mode=755,inode64 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev,inode64 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=13250 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275928k,nr_inodes=818982,mode=700,uid=1000,gid=1271,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275932k,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-05-22T16:06:04.476 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2023-05-22T16:06:04.477 DEBUG:teuthology.orchestra.run.smithi062:> dd if=/proc/self/mounts of=/dev/stdout 2023-05-22T16:06:04.523 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16342116k,nr_inodes=4085529,mode=755,inode64 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275932k,mode=755,inode64 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev,inode64 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=13250 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275928k,nr_inodes=818982,mode=700,uid=1000,gid=1271,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275932k,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-05-22T16:06:04.523 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T16:06:04.524 DEBUG:teuthology.orchestra.run.smithi062:> ip netns list 2023-05-22T16:06:04.571 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2023-05-22T16:06:04.571 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2023-05-22T16:06:04.572 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T16:06:04.572 DEBUG:teuthology.orchestra.run.smithi062:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.3 2023-05-22T16:06:04.628 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T16:06:04.629 DEBUG:teuthology.orchestra.run.smithi062:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.1 2023-05-22T16:06:04.724 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T16:06:04.724 DEBUG:teuthology.orchestra.run.smithi062:> sudo ip link delete ceph-brx 2023-05-22T16:06:04.852 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T16:06:04.852 DEBUG:teuthology.orchestra.run.smithi016:> ip netns list 2023-05-22T16:06:04.859 INFO:teuthology.orchestra.run.smithi016.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-05-22T16:06:04.859 INFO:teuthology.orchestra.run.smithi016.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-05-22T16:06:04.860 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T16:06:04.860 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.2 2023-05-22T16:06:04.914 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T16:06:04.915 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2023-05-22T16:06:05.006 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-05-22T16:06:05.007 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip link delete ceph-brx 2023-05-22T16:06:05.158 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2023-05-22T16:06:05.172 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-05-22T16:06:05.173 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-05-22T16:06:05.305 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.303+0000 7fd898d18640 1 Processor -- start 2023-05-22T16:06:05.306 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.303+0000 7fd898d18640 1 -- start start 2023-05-22T16:06:05.306 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.303+0000 7fd898d18640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd894070c40 0x7fd89406d2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:05.307 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.303+0000 7fd898d18640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8940b46b0 con 0x7fd894159da0 2023-05-22T16:06:05.307 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.303+0000 7fd898d18640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd89406c0f0 con 0x7fd894070700 2023-05-22T16:06:05.307 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.303+0000 7fd898d18640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd89406c3d0 con 0x7fd894070c40 2023-05-22T16:06:05.307 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.303+0000 7fd89259c640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fd894159da0 0x7fd89406fff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:49822/0 (socket says 172.21.15.16:49822) 2023-05-22T16:06:05.307 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.303+0000 7fd89259c640 1 -- 172.21.15.16:0/2780637030 learned_addr learned my addr 172.21.15.16:0/2780637030 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T16:06:05.309 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.303+0000 7fd892d9d640 1 --2- 172.21.15.16:0/2780637030 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd894070c40 0x7fd89406d2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:05.309 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.303+0000 7fd89159a640 1 -- 172.21.15.16:0/2780637030 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1772778218 0 0) 0x7fd8940b46b0 con 0x7fd894159da0 2023-05-22T16:06:05.309 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.303+0000 7fd89159a640 1 -- 172.21.15.16:0/2780637030 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd880003530 con 0x7fd894159da0 2023-05-22T16:06:05.309 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.303+0000 7fd89159a640 1 -- 172.21.15.16:0/2780637030 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2771934809 0 0) 0x7fd880003530 con 0x7fd894159da0 2023-05-22T16:06:05.310 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.303+0000 7fd89159a640 1 -- 172.21.15.16:0/2780637030 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd894070c40 msgr2=0x7fd89406d2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:06:05.310 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.303+0000 7fd89159a640 1 --2- 172.21.15.16:0/2780637030 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd894070c40 0x7fd89406d2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:05.310 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.303+0000 7fd89159a640 1 -- 172.21.15.16:0/2780637030 >> v1:172.21.15.62:6789/0 conn(0x7fd894070700 legacy=0x7fd89406cbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:06:05.310 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.303+0000 7fd89159a640 1 -- 172.21.15.16:0/2780637030 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd89406c6b0 con 0x7fd894159da0 2023-05-22T16:06:05.310 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.303+0000 7fd89159a640 1 -- 172.21.15.16:0/2780637030 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7fd884001680 con 0x7fd894159da0 2023-05-22T16:06:05.311 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.303+0000 7fd892d9d640 1 --2- 172.21.15.16:0/2780637030 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd894070c40 0x7fd89406d2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T16:06:05.311 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.303+0000 7fd89159a640 1 -- 172.21.15.16:0/2780637030 >> v1:172.21.15.16:6789/0 conn(0x7fd894159da0 legacy=0x7fd89406fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:06:05.311 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.303+0000 7fd89159a640 1 --2- 172.21.15.16:0/2780637030 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd894070c40 0x7fd8800039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:05.311 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.303+0000 7fd89159a640 1 --2- 172.21.15.16:0/2780637030 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd880003fa0 0x7fd8800047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:05.311 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.303+0000 7fd89159a640 1 --2- 172.21.15.16:0/2780637030 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd880004d10 0x7fd880005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:05.311 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.303+0000 7fd89159a640 1 -- 172.21.15.16:0/2780637030 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd89406c3d0 con 0x7fd880004d10 2023-05-22T16:06:05.312 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.303+0000 7fd89159a640 1 -- 172.21.15.16:0/2780637030 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd88000bb60 con 0x7fd880003fa0 2023-05-22T16:06:05.312 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.303+0000 7fd89159a640 1 -- 172.21.15.16:0/2780637030 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd88000bd20 con 0x7fd894070c40 2023-05-22T16:06:05.312 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.303+0000 7fd892d9d640 1 --2- 172.21.15.16:0/2780637030 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd880004d10 0x7fd880005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:05.312 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.303+0000 7fd89259c640 1 --2- 172.21.15.16:0/2780637030 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd894070c40 0x7fd8800039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:05.312 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.303+0000 7fd892d9d640 1 -- 172.21.15.16:0/2780637030 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd894070c40 msgr2=0x7fd8800039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:06:05.312 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.303+0000 7fd892d9d640 1 --2- 172.21.15.16:0/2780637030 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd894070c40 0x7fd8800039b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:05.313 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.303+0000 7fd892d9d640 1 -- 172.21.15.16:0/2780637030 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd880003fa0 msgr2=0x7fd8800047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:06:05.313 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.303+0000 7fd891d9b640 1 --2- 172.21.15.16:0/2780637030 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd880003fa0 0x7fd8800047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:05.313 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.303+0000 7fd892d9d640 1 --2- 172.21.15.16:0/2780637030 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd880003fa0 0x7fd8800047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:05.313 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.303+0000 7fd892d9d640 1 -- 172.21.15.16:0/2780637030 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd880003530 con 0x7fd880004d10 2023-05-22T16:06:05.313 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.303+0000 7fd891d9b640 1 --2- 172.21.15.16:0/2780637030 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd880003fa0 0x7fd8800047d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T16:06:05.315 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.307+0000 7fd892d9d640 1 --2- 172.21.15.16:0/2780637030 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd880004d10 0x7fd880005180 secure :-1 s=READY pgs=2616 cs=0 l=1 rev1=1 crypto rx=0x7fd88c0073e0 tx=0x7fd88c01cea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2daa6319e3f7a5a8 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:06:05.316 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.307+0000 7fd89159a640 1 -- 172.21.15.16:0/2780637030 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd88c0a5da0 con 0x7fd880004d10 2023-05-22T16:06:05.316 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.307+0000 7fd89159a640 1 -- 172.21.15.16:0/2780637030 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd88c0ad440 con 0x7fd880004d10 2023-05-22T16:06:05.316 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.307+0000 7fd89159a640 1 -- 172.21.15.16:0/2780637030 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd88c0ad7b0 con 0x7fd880004d10 2023-05-22T16:06:05.316 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.307+0000 7fd898d18640 1 -- 172.21.15.16:0/2780637030 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd880004d10 msgr2=0x7fd880005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:06:05.316 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.307+0000 7fd898d18640 1 --2- 172.21.15.16:0/2780637030 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd880004d10 0x7fd880005180 secure :-1 s=READY pgs=2616 cs=0 l=1 rev1=1 crypto rx=0x7fd88c0073e0 tx=0x7fd88c01cea0 comp rx=0 tx=0).stop 2023-05-22T16:06:05.316 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.307+0000 7fd892d9d640 1 -- 172.21.15.16:0/2780637030 reap_dead start 2023-05-22T16:06:05.317 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.307+0000 7fd898d18640 1 -- 172.21.15.16:0/2780637030 shutdown_connections 2023-05-22T16:06:05.317 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.307+0000 7fd898d18640 1 -- 172.21.15.16:0/2780637030 >> 172.21.15.16:0/2780637030 conn(0x7fd8940cea90 msgr2=0x7fd8940cee90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:06:05.317 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.307+0000 7fd898d18640 1 -- 172.21.15.16:0/2780637030 shutdown_connections 2023-05-22T16:06:05.317 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.307+0000 7fd898d18640 1 -- 172.21.15.16:0/2780637030 wait complete. 2023-05-22T16:06:05.317 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.307+0000 7fd898d18640 1 Processor -- start 2023-05-22T16:06:05.317 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.307+0000 7fd898d18640 1 -- start start 2023-05-22T16:06:05.317 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.311+0000 7fd898d18640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd880004d10 0x7fd8940c2ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:05.317 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.311+0000 7fd898d18640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd894158570 0x7fd8940c5b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:05.318 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.311+0000 7fd898d18640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd8940c21b0 0x7fd8940cb0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:05.318 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.311+0000 7fd898d18640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd89415c1c0 con 0x7fd880004d10 2023-05-22T16:06:05.318 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.311+0000 7fd898d18640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8940c3090 con 0x7fd894158570 2023-05-22T16:06:05.318 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.311+0000 7fd898d18640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd89415b820 con 0x7fd8940c21b0 2023-05-22T16:06:05.318 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.311+0000 7fd89259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd880004d10 0x7fd8940c2ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:05.318 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.311+0000 7fd89259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd880004d10 0x7fd8940c2ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43470/0 (socket says 172.21.15.16:43470) 2023-05-22T16:06:05.318 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.311+0000 7fd89259c640 1 -- 172.21.15.16:0/1441719639 learned_addr learned my addr 172.21.15.16:0/1441719639 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:06:05.319 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.311+0000 7fd891d9b640 1 --2- 172.21.15.16:0/1441719639 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd894158570 0x7fd8940c5b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:05.319 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.311+0000 7fd892d9d640 1 --2- 172.21.15.16:0/1441719639 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd8940c21b0 0x7fd8940cb0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:05.319 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.311+0000 7fd89259c640 1 -- 172.21.15.16:0/1441719639 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd8940c21b0 msgr2=0x7fd8940cb0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:06:05.319 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.311+0000 7fd89259c640 1 --2- 172.21.15.16:0/1441719639 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd8940c21b0 0x7fd8940cb0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:05.319 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.311+0000 7fd89259c640 1 -- 172.21.15.16:0/1441719639 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd894158570 msgr2=0x7fd8940c5b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:06:05.319 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.311+0000 7fd89259c640 1 --2- 172.21.15.16:0/1441719639 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd894158570 0x7fd8940c5b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:05.319 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.311+0000 7fd89259c640 1 -- 172.21.15.16:0/1441719639 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8940b4340 con 0x7fd880004d10 2023-05-22T16:06:05.319 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.311+0000 7fd891d9b640 1 --2- 172.21.15.16:0/1441719639 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd894158570 0x7fd8940c5b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T16:06:05.320 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.311+0000 7fd89259c640 1 --2- 172.21.15.16:0/1441719639 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd880004d10 0x7fd8940c2ab0 secure :-1 s=READY pgs=2617 cs=0 l=1 rev1=1 crypto rx=0x7fd884001210 tx=0x7fd88400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:06:05.320 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.311+0000 7fd88affd640 1 -- 172.21.15.16:0/1441719639 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8840074e0 con 0x7fd880004d10 2023-05-22T16:06:05.320 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.311+0000 7fd88affd640 1 -- 172.21.15.16:0/1441719639 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd884007640 con 0x7fd880004d10 2023-05-22T16:06:05.320 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.311+0000 7fd898d18640 1 -- 172.21.15.16:0/1441719639 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8940b4990 con 0x7fd880004d10 2023-05-22T16:06:05.320 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.311+0000 7fd88affd640 1 -- 172.21.15.16:0/1441719639 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8840049f0 con 0x7fd880004d10 2023-05-22T16:06:05.320 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.311+0000 7fd898d18640 1 -- 172.21.15.16:0/1441719639 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8940add40 con 0x7fd880004d10 2023-05-22T16:06:05.321 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.311+0000 7fd898d18640 1 -- 172.21.15.16:0/1441719639 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd894158940 con 0x7fd880004d10 2023-05-22T16:06:05.321 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.311+0000 7fd88affd640 1 -- 172.21.15.16:0/1441719639 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 100) v1 ==== 84332+0+0 (secure 0 0 0) 0x7fd88400b680 con 0x7fd880004d10 2023-05-22T16:06:05.321 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.311+0000 7fd88affd640 1 --2- 172.21.15.16:0/1441719639 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fd8740654a0 0x7fd874067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:05.321 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.311+0000 7fd88affd640 1 -- 172.21.15.16:0/1441719639 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(201..201 src has 1..201) v4 ==== 6317+0+0 (secure 0 0 0) 0x7fd884082ee0 con 0x7fd880004d10 2023-05-22T16:06:05.321 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.311+0000 7fd891d9b640 1 --2- 172.21.15.16:0/1441719639 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fd8740654a0 0x7fd874067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:05.322 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.311+0000 7fd891d9b640 1 --2- 172.21.15.16:0/1441719639 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fd8740654a0 0x7fd874067950 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fd8940cbd90 tx=0x7fd87c009040 comp rx=0 tx=0).ready entity=mgr.8195 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:06:05.322 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.315+0000 7fd88affd640 1 -- 172.21.15.16:0/1441719639 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fd884050a00 con 0x7fd880004d10 2023-05-22T16:06:05.472 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.467+0000 7fd898d18640 1 -- 172.21.15.16:0/1441719639 --> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd894073050 con 0x7fd8740654a0 2023-05-22T16:06:05.479 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.475+0000 7fd88affd640 1 -- 172.21.15.16:0/1441719639 <== mgr.8195 v2:172.21.15.16:6832/65890 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+71195 (secure 0 0 0) 0x7fd894073050 con 0x7fd8740654a0 2023-05-22T16:06:05.480 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T16:06:05.481 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2023-05-22T16:06:05.482 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.479+0000 7fd898d18640 1 -- 172.21.15.16:0/1441719639 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fd8740654a0 msgr2=0x7fd874067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:06:05.482 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.479+0000 7fd898d18640 1 --2- 172.21.15.16:0/1441719639 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fd8740654a0 0x7fd874067950 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fd8940cbd90 tx=0x7fd87c009040 comp rx=0 tx=0).stop 2023-05-22T16:06:05.482 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.479+0000 7fd898d18640 1 -- 172.21.15.16:0/1441719639 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd880004d10 msgr2=0x7fd8940c2ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:06:05.482 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.479+0000 7fd898d18640 1 --2- 172.21.15.16:0/1441719639 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd880004d10 0x7fd8940c2ab0 secure :-1 s=READY pgs=2617 cs=0 l=1 rev1=1 crypto rx=0x7fd884001210 tx=0x7fd88400dc90 comp rx=0 tx=0).stop 2023-05-22T16:06:05.483 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.479+0000 7fd898d18640 1 -- 172.21.15.16:0/1441719639 shutdown_connections 2023-05-22T16:06:05.483 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.479+0000 7fd898d18640 1 --2- 172.21.15.16:0/1441719639 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd8940c21b0 0x7fd8940cb0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:05.483 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.479+0000 7fd898d18640 1 --2- 172.21.15.16:0/1441719639 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd894158570 0x7fd8940c5b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:05.483 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.479+0000 7fd898d18640 1 --2- 172.21.15.16:0/1441719639 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fd8740654a0 0x7fd874067950 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:05.483 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.479+0000 7fd898d18640 1 --2- 172.21.15.16:0/1441719639 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd880004d10 0x7fd8940c2ab0 unknown :-1 s=CLOSED pgs=2617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:05.483 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.479+0000 7fd898d18640 1 -- 172.21.15.16:0/1441719639 >> 172.21.15.16:0/1441719639 conn(0x7fd8940cea90 msgr2=0x7fd8940ab8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:06:05.483 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.479+0000 7fd898d18640 1 -- 172.21.15.16:0/1441719639 shutdown_connections 2023-05-22T16:06:05.484 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.479+0000 7fd898d18640 1 -- 172.21.15.16:0/1441719639 wait complete. 2023-05-22T16:06:05.499 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":10,"stamp":"2023-05-22T16:06:03.797024+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2132,"num_read_kb":4426,"num_write":579,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":348,"ondisk_log_size":348,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":210,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":259176,"kb_used_data":21992,"kb_used_omap":0,"kb_used_meta":237120,"kb_avail":754715544,"statfs":{"total":773094113280,"available":772828717056,"internally_reserved":0,"allocated":22519808,"data_stored":13208106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":242810880},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-667690,"num_objects":-25,"num_object_clones":0,"num_object_copies":-50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-25,"num_whiteouts":0,"num_read":-317,"num_read_kb":-281,"num_write":-287,"num_write_kb":-2056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002017"},"pg_stats":[{"pgid":"1.0","version":"198'348","reported_seq":1160,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-05-22T16:05:55.897220+0000","last_change":"2023-05-22T15:26:39.318204+0000","last_active":"2023-05-22T16:05:55.897220+0000","last_peered":"2023-05-22T16:05:55.897220+0000","last_clean":"2023-05-22T16:05:55.897220+0000","last_became_active":"2023-05-22T15:26:39.315370+0000","last_became_peered":"2023-05-22T15:26:39.315370+0000","last_unstale":"2023-05-22T16:05:55.897220+0000","last_undegraded":"2023-05-22T16:05:55.897220+0000","last_fullsized":"2023-05-22T16:05:55.897220+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:38.020076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:38.020076+0000","last_clean_scrub_stamp":"2023-05-22T15:26:38.020076+0000","objects_scrubbed":0,"log_size":348,"log_dups_size":0,"ondisk_log_size":348,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:06:39.207543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2132,"num_read_kb":4426,"num_write":579,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":199,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-05-22T16:05:55.773610+0000","last_change":"2023-05-22T15:26:44.971951+0000","last_active":"2023-05-22T16:05:55.773610+0000","last_peered":"2023-05-22T16:05:55.773610+0000","last_clean":"2023-05-22T16:05:55.773610+0000","last_became_active":"2023-05-22T15:26:44.970872+0000","last_became_peered":"2023-05-22T15:26:44.970872+0000","last_unstale":"2023-05-22T16:05:55.773610+0000","last_undegraded":"2023-05-22T16:05:55.773610+0000","last_fullsized":"2023-05-22T16:05:55.773610+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:55:34.752949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":188,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-05-22T16:05:55.833896+0000","last_change":"2023-05-22T15:26:44.977198+0000","last_active":"2023-05-22T16:05:55.833896+0000","last_peered":"2023-05-22T16:05:55.833896+0000","last_clean":"2023-05-22T16:05:55.833896+0000","last_became_active":"2023-05-22T15:26:44.976445+0000","last_became_peered":"2023-05-22T15:26:44.976445+0000","last_unstale":"2023-05-22T16:05:55.833896+0000","last_undegraded":"2023-05-22T16:05:55.833896+0000","last_fullsized":"2023-05-22T16:05:55.833896+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:12:41.464054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":193,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-05-22T16:05:58.459713+0000","last_change":"2023-05-22T15:26:44.969469+0000","last_active":"2023-05-22T16:05:58.459713+0000","last_peered":"2023-05-22T16:05:58.459713+0000","last_clean":"2023-05-22T16:05:58.459713+0000","last_became_active":"2023-05-22T15:26:44.969083+0000","last_became_peered":"2023-05-22T15:26:44.969083+0000","last_unstale":"2023-05-22T16:05:58.459713+0000","last_undegraded":"2023-05-22T16:05:58.459713+0000","last_fullsized":"2023-05-22T16:05:58.459713+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:29:23.670067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":199,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-05-22T16:05:55.774421+0000","last_change":"2023-05-22T15:26:44.976632+0000","last_active":"2023-05-22T16:05:55.774421+0000","last_peered":"2023-05-22T16:05:55.774421+0000","last_clean":"2023-05-22T16:05:55.774421+0000","last_became_active":"2023-05-22T15:26:44.975546+0000","last_became_peered":"2023-05-22T15:26:44.975546+0000","last_unstale":"2023-05-22T16:05:55.774421+0000","last_undegraded":"2023-05-22T16:05:55.774421+0000","last_fullsized":"2023-05-22T16:05:55.774421+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:42:57.973030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":178,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-05-22T16:05:58.461430+0000","last_change":"2023-05-22T15:26:45.091475+0000","last_active":"2023-05-22T16:05:58.461430+0000","last_peered":"2023-05-22T16:05:58.461430+0000","last_clean":"2023-05-22T16:05:58.461430+0000","last_became_active":"2023-05-22T15:26:45.091143+0000","last_became_peered":"2023-05-22T15:26:45.091143+0000","last_unstale":"2023-05-22T16:05:58.461430+0000","last_undegraded":"2023-05-22T16:05:58.461430+0000","last_fullsized":"2023-05-22T16:05:58.461430+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:12:41.464054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":187,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-05-22T16:05:55.834651+0000","last_change":"2023-05-22T15:26:45.092367+0000","last_active":"2023-05-22T16:05:55.834651+0000","last_peered":"2023-05-22T16:05:55.834651+0000","last_clean":"2023-05-22T16:05:55.834651+0000","last_became_active":"2023-05-22T15:26:45.091951+0000","last_became_peered":"2023-05-22T15:26:45.091951+0000","last_unstale":"2023-05-22T16:05:55.834651+0000","last_undegraded":"2023-05-22T16:05:55.834651+0000","last_fullsized":"2023-05-22T16:05:55.834651+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:42:57.973030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":178,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-05-22T16:05:58.461984+0000","last_change":"2023-05-22T15:26:44.978858+0000","last_active":"2023-05-22T16:05:58.461984+0000","last_peered":"2023-05-22T16:05:58.461984+0000","last_clean":"2023-05-22T16:05:58.461984+0000","last_became_active":"2023-05-22T15:26:44.978443+0000","last_became_peered":"2023-05-22T15:26:44.978443+0000","last_unstale":"2023-05-22T16:05:58.461984+0000","last_undegraded":"2023-05-22T16:05:58.461984+0000","last_fullsized":"2023-05-22T16:05:58.461984+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:46:29.290243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":176,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-05-22T16:05:55.915820+0000","last_change":"2023-05-22T15:26:44.982421+0000","last_active":"2023-05-22T16:05:55.915820+0000","last_peered":"2023-05-22T16:05:55.915820+0000","last_clean":"2023-05-22T16:05:55.915820+0000","last_became_active":"2023-05-22T15:26:44.981586+0000","last_became_peered":"2023-05-22T15:26:44.981586+0000","last_unstale":"2023-05-22T16:05:55.915820+0000","last_undegraded":"2023-05-22T16:05:55.915820+0000","last_fullsized":"2023-05-22T16:05:55.915820+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:14:11.156544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2132,"num_read_kb":4426,"num_write":579,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":348,"ondisk_log_size":348,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575314,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29912,"kb_used_data":3344,"kb_used_omap":0,"kb_used_meta":26560,"kb_avail":94341928,"statfs":{"total":96636764160,"available":96606134272,"internally_reserved":0,"allocated":3424256,"data_stored":2254794,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27197440},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 16:05:53 2023","interfaces":[{"interface":"back","average":{"1min":0.81999999999999995,"5min":0.85099999999999998,"15min":0.86099999999999999},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":1.222,"5min":1.3069999999999999,"15min":1.3069999999999999},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.84999999999999998,"5min":0.88200000000000001,"15min":0.877},"min":{"1min":0.61299999999999999,"5min":0.501,"15min":0.501},"max":{"1min":1.296,"5min":1.296,"15min":1.296},"last":1.036}]},{"osd":1,"last update":"Mon May 22 16:05:53 2023","interfaces":[{"interface":"back","average":{"1min":0.90400000000000003,"5min":0.97299999999999998,"15min":0.98999999999999999},"min":{"1min":0.66000000000000003,"5min":0.63400000000000001,"15min":0.58299999999999996},"max":{"1min":1.274,"5min":1.431,"15min":1.431},"last":1.129},{"interface":"front","average":{"1min":0.95499999999999996,"5min":1.026,"15min":1.026},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":1.2909999999999999,"5min":1.345,"15min":1.345},"last":1.216}]},{"osd":2,"last update":"Mon May 22 16:05:53 2023","interfaces":[{"interface":"back","average":{"1min":0.83899999999999997,"5min":0.89900000000000002,"15min":0.90400000000000003},"min":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.171,"5min":1.286,"15min":1.407},"last":0.73399999999999999},{"interface":"front","average":{"1min":0.89100000000000001,"5min":0.94899999999999995,"15min":0.95899999999999996},"min":{"1min":0.53200000000000003,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":1.3899999999999999,"5min":1.3899999999999999,"15min":1.3899999999999999},"last":0.90500000000000003}]},{"osd":3,"last update":"Mon May 22 16:05:53 2023","interfaces":[{"interface":"back","average":{"1min":0.96399999999999997,"5min":0.93300000000000005,"15min":0.94799999999999995},"min":{"1min":0.755,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.2,"5min":1.2909999999999999,"15min":1.401},"last":1.0700000000000001},{"interface":"front","average":{"1min":0.97199999999999998,"5min":0.98699999999999999,"15min":0.98099999999999998},"min":{"1min":0.60299999999999998,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":1.4179999999999999,"5min":1.4179999999999999,"15min":1.4179999999999999},"last":0.85199999999999998}]},{"osd":4,"last update":"Mon May 22 16:05:53 2023","interfaces":[{"interface":"back","average":{"1min":0.875,"5min":0.88900000000000001,"15min":0.88600000000000001},"min":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51000000000000001},"max":{"1min":1.2310000000000001,"5min":1.3440000000000001,"15min":1.3440000000000001},"last":0.77600000000000002},{"interface":"front","average":{"1min":0.95599999999999996,"5min":0.99099999999999999,"15min":0.96499999999999997},"min":{"1min":0.60899999999999999,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":1.3480000000000001,"5min":1.405,"15min":1.405},"last":1.165}]},{"osd":5,"last update":"Mon May 22 16:05:53 2023","interfaces":[{"interface":"back","average":{"1min":0.96499999999999997,"5min":0.98899999999999999,"15min":0.98999999999999999},"min":{"1min":0.66200000000000003,"5min":0.623,"15min":0.441},"max":{"1min":1.159,"5min":1.2909999999999999,"15min":1.3759999999999999},"last":0.95799999999999996},{"interface":"front","average":{"1min":1.1180000000000001,"5min":1.1339999999999999,"15min":1.135},"min":{"1min":0.91000000000000003,"5min":0.72299999999999998,"15min":0.66600000000000004},"max":{"1min":1.468,"5min":1.468,"15min":1.468},"last":1.194}]},{"osd":6,"last update":"Mon May 22 16:05:53 2023","interfaces":[{"interface":"back","average":{"1min":1.048,"5min":1.0600000000000001,"15min":1.0569999999999999},"min":{"1min":0.72799999999999998,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":1.288,"5min":1.3169999999999999,"15min":1.389},"last":1.105},{"interface":"front","average":{"1min":1.008,"5min":1.0940000000000001,"15min":1.0880000000000001},"min":{"1min":0.70399999999999996,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":1.3149999999999999,"5min":1.391,"15min":1.7050000000000001},"last":0.99399999999999999}]}]},{"osd":6,"up_from":13,"seq":55834575315,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":22668,"kb_used_data":2436,"kb_used_omap":0,"kb_used_meta":20224,"kb_avail":94349172,"statfs":{"total":96636764160,"available":96613552128,"internally_reserved":0,"allocated":2494464,"data_stored":1336191,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":20709376},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 16:05:56 2023","interfaces":[{"interface":"back","average":{"1min":0.85999999999999999,"5min":0.88,"15min":0.89600000000000002},"min":{"1min":0.48599999999999999,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":2.6469999999999998,"5min":2.6469999999999998,"15min":2.6469999999999998},"last":1.093},{"interface":"front","average":{"1min":1.254,"5min":0.96899999999999997,"15min":0.92800000000000005},"min":{"1min":0.64600000000000002,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":6.7439999999999998,"5min":6.7439999999999998,"15min":6.7439999999999998},"last":0.95399999999999996}]},{"osd":1,"last update":"Mon May 22 16:05:56 2023","interfaces":[{"interface":"back","average":{"1min":1.0009999999999999,"5min":0.93000000000000005,"15min":0.94199999999999995},"min":{"1min":0.57399999999999995,"5min":0.55300000000000005,"15min":0.51100000000000001},"max":{"1min":2.3290000000000002,"5min":2.3290000000000002,"15min":2.3290000000000002},"last":0.93200000000000005},{"interface":"front","average":{"1min":1.2330000000000001,"5min":0.97899999999999998,"15min":0.94699999999999995},"min":{"1min":0.58199999999999996,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":6.5010000000000003,"5min":6.5010000000000003,"15min":6.5010000000000003},"last":1.1839999999999999}]},{"osd":2,"last update":"Mon May 22 16:05:56 2023","interfaces":[{"interface":"back","average":{"1min":1.2210000000000001,"5min":1.012,"15min":1.0109999999999999},"min":{"1min":0.57399999999999995,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":7.0170000000000003,"5min":7.0170000000000003,"15min":7.0170000000000003},"last":0.98999999999999999},{"interface":"front","average":{"1min":1.0589999999999999,"5min":1.05,"15min":1.0720000000000001},"min":{"1min":0.66200000000000003,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":2.403,"5min":2.403,"15min":2.403},"last":1.02}]},{"osd":3,"last update":"Mon May 22 16:05:56 2023","interfaces":[{"interface":"back","average":{"1min":1.278,"5min":1.054,"15min":1.0129999999999999},"min":{"1min":0.63800000000000001,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":6.6699999999999999,"5min":6.6699999999999999,"15min":6.6699999999999999},"last":1.133},{"interface":"front","average":{"1min":1.244,"5min":1.02,"15min":0.98899999999999999},"min":{"1min":0.68000000000000005,"5min":0.622,"15min":0.622},"max":{"1min":7.1260000000000003,"5min":7.1260000000000003,"15min":7.1260000000000003},"last":0.90400000000000003}]},{"osd":4,"last update":"Mon May 22 16:05:56 2023","interfaces":[{"interface":"back","average":{"1min":1.161,"5min":0.98099999999999998,"15min":0.96999999999999997},"min":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"max":{"1min":4.0709999999999997,"5min":4.0709999999999997,"15min":4.0709999999999997},"last":0.73599999999999999},{"interface":"front","average":{"1min":1.583,"5min":1.1619999999999999,"15min":1.075},"min":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":11.581,"5min":11.581,"15min":11.581},"last":1.2330000000000001}]},{"osd":5,"last update":"Mon May 22 16:05:56 2023","interfaces":[{"interface":"back","average":{"1min":1.925,"5min":1.1739999999999999,"15min":1.056},"min":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":17.587,"5min":17.587,"15min":17.587},"last":0.70699999999999996},{"interface":"front","average":{"1min":1.204,"5min":1.1180000000000001,"15min":1.1240000000000001},"min":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"max":{"1min":2.9620000000000002,"5min":2.9620000000000002,"15min":2.9620000000000002},"last":0.83099999999999996}]},{"osd":7,"last update":"Mon May 22 16:05:56 2023","interfaces":[{"interface":"back","average":{"1min":1.4319999999999999,"5min":1.202,"15min":1.1559999999999999},"min":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.58599999999999997},"max":{"1min":6.8339999999999996,"5min":6.8339999999999996,"15min":6.8339999999999996},"last":1.0629999999999999},{"interface":"front","average":{"1min":1.5429999999999999,"5min":1.2090000000000001,"15min":1.163},"min":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":8.5990000000000002,"5min":8.5990000000000002,"15min":8.5990000000000002},"last":0.78600000000000003}]}]},{"osd":5,"up_from":13,"seq":55834575315,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32780,"kb_used_data":2436,"kb_used_omap":0,"kb_used_meta":30336,"kb_avail":94339060,"statfs":{"total":96636764160,"available":96603197440,"internally_reserved":0,"allocated":2494464,"data_stored":1336191,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31064064},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 16:05:54 2023","interfaces":[{"interface":"back","average":{"1min":0.76000000000000001,"5min":0.85199999999999998,"15min":0.875},"min":{"1min":0.47899999999999998,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.0820000000000001,"5min":1.2749999999999999,"15min":1.3959999999999999},"last":0.98599999999999999},{"interface":"front","average":{"1min":0.86499999999999999,"5min":0.93999999999999995,"15min":0.94299999999999995},"min":{"1min":0.56499999999999995,"5min":0.56200000000000006,"15min":0.502},"max":{"1min":1.165,"5min":1.325,"15min":1.325},"last":1.0860000000000001}]},{"osd":1,"last update":"Mon May 22 16:05:54 2023","interfaces":[{"interface":"back","average":{"1min":0.86599999999999999,"5min":0.95999999999999996,"15min":1.0009999999999999},"min":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.2909999999999999,"5min":1.365,"15min":1.365},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.89600000000000002,"5min":1.028,"15min":1.0640000000000001},"min":{"1min":0.46899999999999997,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":1.216,"5min":1.3520000000000001,"15min":1.4630000000000001},"last":1.0569999999999999}]},{"osd":2,"last update":"Mon May 22 16:05:54 2023","interfaces":[{"interface":"back","average":{"1min":0.85299999999999998,"5min":0.90500000000000003,"15min":0.92000000000000004},"min":{"1min":0.60899999999999999,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.1859999999999999,"5min":1.3200000000000001,"15min":1.3740000000000001},"last":1.0149999999999999},{"interface":"front","average":{"1min":0.86799999999999999,"5min":0.94699999999999995,"15min":0.94599999999999995},"min":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":1.117,"5min":1.4350000000000001,"15min":1.4350000000000001},"last":0.92700000000000005}]},{"osd":3,"last update":"Mon May 22 16:05:54 2023","interfaces":[{"interface":"back","average":{"1min":0.91300000000000003,"5min":0.97699999999999998,"15min":0.97599999999999998},"min":{"1min":0.63400000000000001,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":1.1759999999999999,"5min":1.349,"15min":1.367},"last":0.83699999999999997},{"interface":"front","average":{"1min":0.97999999999999998,"5min":1.0329999999999999,"15min":1.048},"min":{"1min":0.67600000000000005,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.1779999999999999,"5min":1.3899999999999999,"15min":1.3899999999999999},"last":0.85599999999999998}]},{"osd":4,"last update":"Mon May 22 16:05:54 2023","interfaces":[{"interface":"back","average":{"1min":0.91200000000000003,"5min":0.93600000000000005,"15min":0.93100000000000005},"min":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.2170000000000001,"5min":1.6000000000000001,"15min":1.6000000000000001},"last":0.79900000000000004},{"interface":"front","average":{"1min":0.98899999999999999,"5min":0.98499999999999999,"15min":1.01},"min":{"1min":0.57899999999999996,"5min":0.498,"15min":0.498},"max":{"1min":1.371,"5min":1.4630000000000001,"15min":1.4630000000000001},"last":0.89900000000000002}]},{"osd":6,"last update":"Mon May 22 16:05:54 2023","interfaces":[{"interface":"back","average":{"1min":1.054,"5min":1.0129999999999999,"15min":1.0209999999999999},"min":{"1min":0.68999999999999995,"5min":0.63200000000000001,"15min":0.59799999999999998},"max":{"1min":1.2509999999999999,"5min":1.4630000000000001,"15min":1.4790000000000001},"last":1.147},{"interface":"front","average":{"1min":1.0449999999999999,"5min":1.073,"15min":1.081},"min":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":1.3440000000000001,"5min":1.6080000000000001,"15min":1.6080000000000001},"last":1.0349999999999999}]},{"osd":7,"last update":"Mon May 22 16:05:54 2023","interfaces":[{"interface":"back","average":{"1min":1.133,"5min":1.1180000000000001,"15min":1.1259999999999999},"min":{"1min":0.83899999999999997,"5min":0.57699999999999996,"15min":0.57699999999999996},"max":{"1min":1.3680000000000001,"5min":1.51,"15min":1.5109999999999999},"last":0.77300000000000002},{"interface":"front","average":{"1min":1.1739999999999999,"5min":1.1859999999999999,"15min":1.1930000000000001},"min":{"1min":0.91000000000000003,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":1.353,"5min":1.6679999999999999,"15min":1.7210000000000001},"last":0.95299999999999996}]}]},{"osd":4,"up_from":13,"seq":55834575313,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39584,"kb_used_data":2456,"kb_used_omap":0,"kb_used_meta":37120,"kb_avail":94332256,"statfs":{"total":96636764160,"available":96596230144,"internally_reserved":0,"allocated":2514944,"data_stored":1349186,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":38010880},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 16:05:02 2023","interfaces":[{"interface":"back","average":{"1min":0.90400000000000003,"5min":0.89600000000000002,"15min":0.90000000000000002},"min":{"1min":0.59999999999999998,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.1870000000000001,"5min":1.7210000000000001,"15min":1.7210000000000001},"last":0.72299999999999998},{"interface":"front","average":{"1min":0.995,"5min":0.96599999999999997,"15min":0.95199999999999996},"min":{"1min":0.70899999999999996,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":1.3360000000000001,"5min":1.7290000000000001,"15min":1.7290000000000001},"last":0.57099999999999995}]},{"osd":1,"last update":"Mon May 22 16:05:02 2023","interfaces":[{"interface":"back","average":{"1min":1.0269999999999999,"5min":0.97599999999999998,"15min":0.95899999999999996},"min":{"1min":0.76900000000000002,"5min":0.314,"15min":0.314},"max":{"1min":1.258,"5min":1.9339999999999999,"15min":1.9339999999999999},"last":0.96299999999999997},{"interface":"front","average":{"1min":0.90000000000000002,"5min":0.92800000000000005,"15min":0.93999999999999995},"min":{"1min":0.68799999999999994,"5min":0.504,"15min":0.504},"max":{"1min":1.0900000000000001,"5min":1.365,"15min":1.365},"last":0.79400000000000004}]},{"osd":2,"last update":"Mon May 22 16:05:02 2023","interfaces":[{"interface":"back","average":{"1min":0.98599999999999999,"5min":0.95499999999999996,"15min":0.94999999999999996},"min":{"1min":0.71799999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":1.28,"5min":1.4990000000000001,"15min":1.4990000000000001},"last":1.0129999999999999},{"interface":"front","average":{"1min":1.056,"5min":1.036,"15min":1.052},"min":{"1min":0.76000000000000001,"5min":0.44,"15min":0.44},"max":{"1min":1.2529999999999999,"5min":2.0249999999999999,"15min":2.0249999999999999},"last":0.84899999999999998}]},{"osd":3,"last update":"Mon May 22 16:05:02 2023","interfaces":[{"interface":"back","average":{"1min":1.0329999999999999,"5min":0.96499999999999997,"15min":0.96599999999999997},"min":{"1min":0.79200000000000004,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":1.258,"5min":1.4610000000000001,"15min":1.4610000000000001},"last":0.61699999999999999},{"interface":"front","average":{"1min":1.0760000000000001,"5min":1.0609999999999999,"15min":1.0489999999999999},"min":{"1min":0.83899999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":1.323,"5min":1.349,"15min":1.484},"last":1.125}]},{"osd":5,"last update":"Mon May 22 16:05:02 2023","interfaces":[{"interface":"back","average":{"1min":1.004,"5min":1.036,"15min":1.002},"min":{"1min":0.69599999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":1.325,"5min":2.1110000000000002,"15min":2.1110000000000002},"last":1.0880000000000001},{"interface":"front","average":{"1min":1.0309999999999999,"5min":1.0189999999999999,"15min":0.996},"min":{"1min":0.69999999999999996,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.1990000000000001,"5min":1.5449999999999999,"15min":1.5449999999999999},"last":1.0620000000000001}]},{"osd":6,"last update":"Mon May 22 16:05:02 2023","interfaces":[{"interface":"back","average":{"1min":1.0669999999999999,"5min":1.036,"15min":1.0009999999999999},"min":{"1min":0.75800000000000001,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":1.387,"5min":1.397,"15min":1.4339999999999999},"last":1.2},{"interface":"front","average":{"1min":1.1599999999999999,"5min":1.1599999999999999,"15min":1.137},"min":{"1min":0.875,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.3340000000000001,"5min":2.2690000000000001,"15min":2.2690000000000001},"last":1.1639999999999999}]},{"osd":7,"last update":"Mon May 22 16:05:02 2023","interfaces":[{"interface":"back","average":{"1min":1.093,"5min":1.0960000000000001,"15min":1.069},"min":{"1min":0.83599999999999997,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":1.3029999999999999,"5min":1.623,"15min":1.623},"last":0.91300000000000003},{"interface":"front","average":{"1min":1.2150000000000001,"5min":1.1879999999999999,"15min":1.1599999999999999},"min":{"1min":1.0169999999999999,"5min":0.71199999999999997,"15min":0.68100000000000005},"max":{"1min":1.4390000000000001,"5min":1.448,"15min":1.5089999999999999},"last":1.2529999999999999}]}]},{"osd":3,"up_from":13,"seq":55834575313,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27928,"kb_used_data":2768,"kb_used_omap":0,"kb_used_meta":25152,"kb_avail":94343912,"statfs":{"total":96636764160,"available":96608165888,"internally_reserved":0,"allocated":2834432,"data_stored":1664516,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":25755648},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 16:05:38 2023","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"min":{"1min":0.29299999999999998,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.93700000000000006,"5min":1.0049999999999999,"15min":1.0229999999999999},"last":0.81899999999999995},{"interface":"front","average":{"1min":0.70299999999999996,"5min":0.76200000000000001,"15min":0.76900000000000002},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":1.2749999999999999,"5min":1.7230000000000001,"15min":1.7230000000000001},"last":0.45900000000000002}]},{"osd":1,"last update":"Mon May 22 16:05:38 2023","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.75600000000000001,"15min":0.71999999999999997},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":1.2190000000000001,"5min":2.4969999999999999,"15min":2.4969999999999999},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.76000000000000001,"5min":0.80700000000000005,"15min":0.80100000000000005},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":1.3520000000000001,"5min":2.4769999999999999,"15min":2.4769999999999999},"last":0.66200000000000003}]},{"osd":2,"last update":"Mon May 22 16:05:38 2023","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":0.83699999999999997,"15min":0.84899999999999998},"min":{"1min":0.52300000000000002,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.94799999999999995,"5min":2.2250000000000001,"15min":2.2250000000000001},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.87,"5min":0.83599999999999997,"15min":0.83299999999999996},"min":{"1min":0.55500000000000005,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":1.129,"5min":2.1680000000000001,"15min":2.1680000000000001},"last":0.76400000000000001}]},{"osd":4,"last update":"Mon May 22 16:05:38 2023","interfaces":[{"interface":"back","average":{"1min":0.95199999999999996,"5min":0.98099999999999998,"15min":0.97399999999999998},"min":{"1min":0.36799999999999999,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":1.357,"5min":2.3479999999999999,"15min":2.3479999999999999},"last":1.0009999999999999},{"interface":"front","average":{"1min":1.0009999999999999,"5min":1.022,"15min":1.03},"min":{"1min":0.39400000000000002,"5min":0.37,"15min":0.37},"max":{"1min":1.2210000000000001,"5min":1.788,"15min":1.788},"last":1.135}]},{"osd":5,"last update":"Mon May 22 16:05:38 2023","interfaces":[{"interface":"back","average":{"1min":1.0069999999999999,"5min":1.0409999999999999,"15min":1.05},"min":{"1min":0.496,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":1.21,"5min":1.8129999999999999,"15min":1.8129999999999999},"last":1.0409999999999999},{"interface":"front","average":{"1min":0.98299999999999998,"5min":1.0169999999999999,"15min":1.0369999999999999},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.3140000000000001,"5min":1.571,"15min":1.571},"last":0.86599999999999999}]},{"osd":6,"last update":"Mon May 22 16:05:38 2023","interfaces":[{"interface":"back","average":{"1min":1.0109999999999999,"5min":1.0680000000000001,"15min":1.0880000000000001},"min":{"1min":0.45600000000000002,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":1.274,"5min":1.76,"15min":1.76},"last":1.2090000000000001},{"interface":"front","average":{"1min":1.0609999999999999,"5min":1.0820000000000001,"15min":1.099},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":1.4139999999999999,"5min":1.754,"15min":1.754},"last":0.91000000000000003}]},{"osd":7,"last update":"Mon May 22 16:05:38 2023","interfaces":[{"interface":"back","average":{"1min":1.1020000000000001,"5min":1.111,"15min":1.127},"min":{"1min":0.53800000000000003,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.5049999999999999,"5min":1.673,"15min":1.673},"last":1.081},{"interface":"front","average":{"1min":1.1040000000000001,"5min":1.1160000000000001,"15min":1.109},"min":{"1min":0.41099999999999998,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":1.4670000000000001,"5min":2.3079999999999998,"15min":2.3079999999999998},"last":0.88}]}]},{"osd":2,"up_from":13,"seq":55834575314,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31316,"kb_used_data":2444,"kb_used_omap":0,"kb_used_meta":28864,"kb_avail":94340524,"statfs":{"total":96636764160,"available":96604696576,"internally_reserved":0,"allocated":2502656,"data_stored":1336768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":29556736},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 16:05:35 2023","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.629,"15min":0.67000000000000004},"min":{"1min":0.28599999999999998,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.90100000000000002,"5min":1.0549999999999999,"15min":1.1519999999999999},"last":0.78400000000000003},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.71799999999999997,"15min":0.74199999999999999},"min":{"1min":0.35299999999999998,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":1.0309999999999999,"5min":1.1659999999999999,"15min":1.2090000000000001},"last":0.66100000000000003}]},{"osd":1,"last update":"Mon May 22 16:05:35 2023","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.74399999999999999,"15min":0.75700000000000001},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.875,"5min":1.071,"15min":1.099},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.79300000000000004,"5min":0.82099999999999995,"15min":0.82599999999999996},"min":{"1min":0.39500000000000002,"5min":0.311,"15min":0.311},"max":{"1min":1.069,"5min":1.1739999999999999,"15min":1.2589999999999999},"last":0.875}]},{"osd":3,"last update":"Mon May 22 16:05:35 2023","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":0.75600000000000001,"15min":0.75800000000000001},"min":{"1min":0.32100000000000001,"5min":0.315,"15min":0.315},"max":{"1min":0.91400000000000003,"5min":1.3540000000000001,"15min":1.3540000000000001},"last":0.83899999999999997},{"interface":"front","average":{"1min":0.85199999999999998,"5min":0.87,"15min":0.89300000000000002},"min":{"1min":0.41399999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":1.1859999999999999,"5min":1.2949999999999999,"15min":1.2949999999999999},"last":0.95099999999999996}]},{"osd":4,"last update":"Mon May 22 16:05:35 2023","interfaces":[{"interface":"back","average":{"1min":1.02,"5min":1.024,"15min":1.04},"min":{"1min":0.66100000000000003,"5min":0.441,"15min":0.441},"max":{"1min":1.4590000000000001,"5min":1.516,"15min":1.6539999999999999},"last":1.0249999999999999},{"interface":"front","average":{"1min":0.996,"5min":1.0089999999999999,"15min":1.04},"min":{"1min":0.55600000000000005,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.419,"5min":1.4299999999999999,"15min":1.538},"last":1.1060000000000001}]},{"osd":5,"last update":"Mon May 22 16:05:35 2023","interfaces":[{"interface":"back","average":{"1min":1.03,"5min":1.024,"15min":1.0549999999999999},"min":{"1min":0.53600000000000003,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":1.4139999999999999,"5min":1.4139999999999999,"15min":1.605},"last":1.2350000000000001},{"interface":"front","average":{"1min":1.0069999999999999,"5min":1.038,"15min":1.03},"min":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":1.339,"5min":1.534,"15min":1.534},"last":0.92200000000000004}]},{"osd":6,"last update":"Mon May 22 16:05:35 2023","interfaces":[{"interface":"back","average":{"1min":1.095,"5min":1.0589999999999999,"15min":1.0349999999999999},"min":{"1min":0.72699999999999998,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":1.4259999999999999,"5min":1.429,"15min":1.462},"last":0.99399999999999999},{"interface":"front","average":{"1min":1.127,"5min":1.105,"15min":1.1339999999999999},"min":{"1min":0.59199999999999997,"5min":0.57099999999999995,"15min":0.57099999999999995},"max":{"1min":1.5269999999999999,"5min":1.621,"15min":1.7110000000000001},"last":1.0760000000000001}]},{"osd":7,"last update":"Mon May 22 16:05:35 2023","interfaces":[{"interface":"back","average":{"1min":1.0940000000000001,"5min":1.0860000000000001,"15min":1.1220000000000001},"min":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.5680000000000001,"5min":1.5680000000000001,"15min":1.6399999999999999},"last":1.179},{"interface":"front","average":{"1min":1.0940000000000001,"5min":1.107,"15min":1.1140000000000001},"min":{"1min":0.629,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":1.4610000000000001,"5min":1.673,"15min":1.673},"last":1.0389999999999999}]}]},{"osd":1,"up_from":13,"seq":55834575314,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37724,"kb_used_data":2772,"kb_used_omap":0,"kb_used_meta":34944,"kb_avail":94334116,"statfs":{"total":96636764160,"available":96598134784,"internally_reserved":0,"allocated":2838528,"data_stored":1676300,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":35782656},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 16:05:58 2023","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.64800000000000002,"15min":0.66200000000000003},"min":{"1min":0.34499999999999997,"5min":0.308,"15min":0.308},"max":{"1min":1.0449999999999999,"5min":1.0449999999999999,"15min":1.329},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.72199999999999998,"15min":0.76300000000000001},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.95499999999999996,"5min":1.25,"15min":1.268},"last":0.85099999999999998}]},{"osd":2,"last update":"Mon May 22 16:05:58 2023","interfaces":[{"interface":"back","average":{"1min":0.70999999999999996,"5min":0.73399999999999999,"15min":0.75600000000000001},"min":{"1min":0.372,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":1.141,"5min":1.141,"15min":1.26},"last":0.96099999999999997},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.76300000000000001,"15min":0.77700000000000002},"min":{"1min":0.46999999999999997,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":1.0249999999999999,"5min":1.127,"15min":1.268},"last":0.78900000000000003}]},{"osd":3,"last update":"Mon May 22 16:05:58 2023","interfaces":[{"interface":"back","average":{"1min":0.78300000000000003,"5min":0.83799999999999997,"15min":0.878},"min":{"1min":0.39900000000000002,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":1.1100000000000001,"5min":1.2829999999999999,"15min":1.2829999999999999},"last":1.1100000000000001},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.80300000000000005,"15min":0.81899999999999995},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":1.2050000000000001,"5min":1.2749999999999999,"15min":1.2749999999999999},"last":0.38500000000000001}]},{"osd":4,"last update":"Mon May 22 16:05:58 2023","interfaces":[{"interface":"back","average":{"1min":0.93899999999999995,"5min":0.96599999999999997,"15min":0.98399999999999999},"min":{"1min":0.58499999999999996,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":1.3500000000000001,"5min":1.542,"15min":1.542},"last":1.046},{"interface":"front","average":{"1min":1.006,"5min":0.997,"15min":1.024},"min":{"1min":0.71499999999999997,"5min":0.434,"15min":0.434},"max":{"1min":1.2729999999999999,"5min":1.327,"15min":1.4650000000000001},"last":1.2390000000000001}]},{"osd":5,"last update":"Mon May 22 16:05:58 2023","interfaces":[{"interface":"back","average":{"1min":1.0309999999999999,"5min":1.0349999999999999,"15min":1.048},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.319,"5min":1.605,"15min":1.605},"last":0.997},{"interface":"front","average":{"1min":1.0589999999999999,"5min":1.0620000000000001,"15min":1.0760000000000001},"min":{"1min":0.75900000000000001,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":1.4370000000000001,"5min":1.4370000000000001,"15min":1.5389999999999999},"last":1.151}]},{"osd":6,"last update":"Mon May 22 16:05:58 2023","interfaces":[{"interface":"back","average":{"1min":1.03,"5min":1.0469999999999999,"15min":1.0640000000000001},"min":{"1min":0.56899999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"max":{"1min":1.4770000000000001,"5min":1.4770000000000001,"15min":1.5249999999999999},"last":1.3580000000000001},{"interface":"front","average":{"1min":1.0589999999999999,"5min":1.099,"15min":1.1100000000000001},"min":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"max":{"1min":1.387,"5min":1.4359999999999999,"15min":1.6479999999999999},"last":1.1890000000000001}]},{"osd":7,"last update":"Mon May 22 16:05:58 2023","interfaces":[{"interface":"back","average":{"1min":1.0509999999999999,"5min":1.123,"15min":1.153},"min":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":1.484,"5min":1.7030000000000001,"15min":1.7030000000000001},"last":1.2829999999999999},{"interface":"front","average":{"1min":1.1339999999999999,"5min":1.131,"15min":1.151},"min":{"1min":0.78400000000000003,"5min":0.73599999999999999,"15min":0.73599999999999999},"max":{"1min":1.4810000000000001,"5min":1.6439999999999999,"15min":1.6439999999999999},"last":1.319}]}]},{"osd":0,"up_from":13,"seq":55834575315,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37264,"kb_used_data":3336,"kb_used_omap":0,"kb_used_meta":33920,"kb_avail":94334576,"statfs":{"total":96636764160,"available":96598605824,"internally_reserved":0,"allocated":3416064,"data_stored":2254160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":34734080},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon May 22 16:06:01 2023","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.72399999999999998,"15min":0.71099999999999997},"min":{"1min":0.42299999999999999,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":1.1140000000000001,"5min":1.296,"15min":1.296},"last":0.73799999999999999},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.72999999999999998,"15min":0.73899999999999999},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.86899999999999999,"5min":1.181,"15min":1.3149999999999999},"last":0.60699999999999998}]},{"osd":2,"last update":"Mon May 22 16:06:01 2023","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.751,"15min":0.76900000000000002},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":1.071,"5min":1.22,"15min":1.284},"last":0.877},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.78700000000000003,"15min":0.81299999999999994},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":1.042,"5min":1.323,"15min":1.323},"last":0.70399999999999996}]},{"osd":3,"last update":"Mon May 22 16:06:01 2023","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.746,"15min":0.77100000000000002},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":1.228,"5min":1.2350000000000001,"15min":1.2350000000000001},"last":0.83099999999999996},{"interface":"front","average":{"1min":0.78300000000000003,"5min":0.85899999999999999,"15min":0.874},"min":{"1min":0.43099999999999999,"5min":0.377,"15min":0.377},"max":{"1min":1.22,"5min":1.2949999999999999,"15min":1.331},"last":1.0600000000000001}]},{"osd":4,"last update":"Mon May 22 16:06:01 2023","interfaces":[{"interface":"back","average":{"1min":0.90200000000000002,"5min":0.94599999999999995,"15min":0.95199999999999996},"min":{"1min":0.64100000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"max":{"1min":1.1819999999999999,"5min":1.264,"15min":1.4059999999999999},"last":1.085},{"interface":"front","average":{"1min":0.91500000000000004,"5min":1.042,"15min":1.05},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.1619999999999999,"5min":1.401,"15min":1.482},"last":0.93999999999999995}]},{"osd":5,"last update":"Mon May 22 16:06:01 2023","interfaces":[{"interface":"back","average":{"1min":0.92600000000000005,"5min":1.014,"15min":1.0349999999999999},"min":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":1.1830000000000001,"5min":1.325,"15min":1.6419999999999999},"last":1.016},{"interface":"front","average":{"1min":0.90200000000000002,"5min":1.0009999999999999,"15min":1.0109999999999999},"min":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"max":{"1min":1.1859999999999999,"5min":1.413,"15min":1.458},"last":0.96899999999999997}]},{"osd":6,"last update":"Mon May 22 16:06:01 2023","interfaces":[{"interface":"back","average":{"1min":0.98399999999999999,"5min":1.091,"15min":1.1060000000000001},"min":{"1min":0.66400000000000003,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":1.2729999999999999,"5min":1.407,"15min":1.77},"last":1.1479999999999999},{"interface":"front","average":{"1min":0.96399999999999997,"5min":1.093,"15min":1.1040000000000001},"min":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"max":{"1min":1.2589999999999999,"5min":1.5409999999999999,"15min":1.5409999999999999},"last":1.2070000000000001}]},{"osd":7,"last update":"Mon May 22 16:06:01 2023","interfaces":[{"interface":"back","average":{"1min":1.034,"5min":1.1279999999999999,"15min":1.1359999999999999},"min":{"1min":0.69299999999999995,"5min":0.63900000000000001,"15min":0.63900000000000001},"max":{"1min":1.341,"5min":1.6020000000000001,"15min":1.6020000000000001},"last":1.1259999999999999},{"interface":"front","average":{"1min":1.0109999999999999,"5min":1.0780000000000001,"15min":1.095},"min":{"1min":0.68999999999999995,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.2989999999999999,"5min":1.4650000000000001,"15min":1.4990000000000001},"last":1.173}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-05-22T16:06:05.502 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-05-22T16:06:05.628 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.623+0000 7f0b1587c640 1 Processor -- start 2023-05-22T16:06:05.628 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.623+0000 7f0b1587c640 1 -- start start 2023-05-22T16:06:05.629 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.627+0000 7f0b1587c640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f0b1006c8e0 0x7f0b1015cf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:05.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.627+0000 7f0b1587c640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f0b10158be0 0x7f0b1006ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:05.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.627+0000 7f0b1587c640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b1015d480 con 0x7f0b100c8840 2023-05-22T16:06:05.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.627+0000 7f0b1587c640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b1006d1f0 con 0x7f0b10158be0 2023-05-22T16:06:05.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.627+0000 7f0b1587c640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b100c8e80 con 0x7f0b1006c8e0 2023-05-22T16:06:05.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.627+0000 7f0b0e7fc640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f0b10158be0 0x7f0b1006ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:05.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.627+0000 7f0b0e7fc640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f0b10158be0 0x7f0b1006ccb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43476/0 (socket says 172.21.15.16:43476) 2023-05-22T16:06:05.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.627+0000 7f0b0e7fc640 1 -- 172.21.15.16:0/2738138499 learned_addr learned my addr 172.21.15.16:0/2738138499 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:06:05.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.627+0000 7f0b0effd640 1 --2- 172.21.15.16:0/2738138499 >> v2:172.21.15.62:3300/0 conn(0x7f0b1006c8e0 0x7f0b1015cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:05.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.627+0000 7f0b0e7fc640 1 -- 172.21.15.16:0/2738138499 >> v2:172.21.15.62:3300/0 conn(0x7f0b1006c8e0 msgr2=0x7f0b1015cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:06:05.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.627+0000 7f0b0e7fc640 1 --2- 172.21.15.16:0/2738138499 >> v2:172.21.15.62:3300/0 conn(0x7f0b1006c8e0 0x7f0b1015cf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:05.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.627+0000 7f0b0e7fc640 1 -- 172.21.15.16:0/2738138499 >> v1:172.21.15.62:6789/0 conn(0x7f0b100c8840 legacy=0x7f0b100afb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:06:05.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.627+0000 7f0b0e7fc640 1 -- 172.21.15.16:0/2738138499 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b100c9160 con 0x7f0b10158be0 2023-05-22T16:06:05.632 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.627+0000 7f0b0e7fc640 1 --2- 172.21.15.16:0/2738138499 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0b10158be0 0x7f0b1006ccb0 secure :-1 s=READY pgs=2618 cs=0 l=1 rev1=1 crypto rx=0x7f0b100cb910 tx=0x7f0b0401c4c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9ad2076a3abc795 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:06:05.633 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.627+0000 7f0b0dffb640 1 -- 172.21.15.16:0/2738138499 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b040ab600 con 0x7f0b10158be0 2023-05-22T16:06:05.633 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.627+0000 7f0b0dffb640 1 -- 172.21.15.16:0/2738138499 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b040a7050 con 0x7f0b10158be0 2023-05-22T16:06:05.633 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.627+0000 7f0b0dffb640 1 -- 172.21.15.16:0/2738138499 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b040ab980 con 0x7f0b10158be0 2023-05-22T16:06:05.633 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.631+0000 7f0b1587c640 1 -- 172.21.15.16:0/2738138499 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0b10158be0 msgr2=0x7f0b1006ccb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:06:05.633 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.631+0000 7f0b1587c640 1 --2- 172.21.15.16:0/2738138499 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0b10158be0 0x7f0b1006ccb0 secure :-1 s=READY pgs=2618 cs=0 l=1 rev1=1 crypto rx=0x7f0b100cb910 tx=0x7f0b0401c4c0 comp rx=0 tx=0).stop 2023-05-22T16:06:05.633 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.631+0000 7f0b1587c640 1 -- 172.21.15.16:0/2738138499 shutdown_connections 2023-05-22T16:06:05.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.631+0000 7f0b1587c640 1 --2- 172.21.15.16:0/2738138499 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0b10158be0 0x7f0b1006ccb0 unknown :-1 s=CLOSED pgs=2618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:05.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.631+0000 7f0b1587c640 1 --2- 172.21.15.16:0/2738138499 >> v2:172.21.15.62:3300/0 conn(0x7f0b1006c8e0 0x7f0b1015cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:05.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.631+0000 7f0b1587c640 1 -- 172.21.15.16:0/2738138499 >> 172.21.15.16:0/2738138499 conn(0x7f0b100cab50 msgr2=0x7f0b100caf50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:06:05.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.631+0000 7f0b1587c640 1 -- 172.21.15.16:0/2738138499 shutdown_connections 2023-05-22T16:06:05.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.631+0000 7f0b1587c640 1 -- 172.21.15.16:0/2738138499 wait complete. 2023-05-22T16:06:05.635 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.631+0000 7f0b1587c640 1 Processor -- start 2023-05-22T16:06:05.636 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.631+0000 7f0b1587c640 1 -- start start 2023-05-22T16:06:05.636 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.631+0000 7f0b1587c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0b1006c8e0 0x7f0b100ae2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:05.636 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.631+0000 7f0b1587c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0b100c8840 0x7f0b1014cf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:05.636 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.631+0000 7f0b1587c640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0b10158be0 0x7f0b1014d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:05.637 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.631+0000 7f0b1587c640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b101512d0 con 0x7f0b1006c8e0 2023-05-22T16:06:05.637 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.631+0000 7f0b1587c640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b101f91e0 con 0x7f0b100c8840 2023-05-22T16:06:05.637 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.631+0000 7f0b1587c640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b1014f3a0 con 0x7f0b10158be0 2023-05-22T16:06:05.637 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.631+0000 7f0b0effd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0b1006c8e0 0x7f0b100ae2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:05.637 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.631+0000 7f0b0effd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0b1006c8e0 0x7f0b100ae2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43488/0 (socket says 172.21.15.16:43488) 2023-05-22T16:06:05.637 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.631+0000 7f0b0effd640 1 -- 172.21.15.16:0/3092078221 learned_addr learned my addr 172.21.15.16:0/3092078221 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:06:05.637 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.631+0000 7f0b0e7fc640 1 --2- 172.21.15.16:0/3092078221 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0b100c8840 0x7f0b1014cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:05.638 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.631+0000 7f0b0f7fe640 1 --2- 172.21.15.16:0/3092078221 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0b10158be0 0x7f0b1014d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:05.638 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.631+0000 7f0b0effd640 1 -- 172.21.15.16:0/3092078221 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0b10158be0 msgr2=0x7f0b1014d480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:06:05.638 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.631+0000 7f0b0effd640 1 --2- 172.21.15.16:0/3092078221 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0b10158be0 0x7f0b1014d480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:05.638 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.631+0000 7f0b0effd640 1 -- 172.21.15.16:0/3092078221 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0b100c8840 msgr2=0x7f0b1014cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:06:05.638 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.631+0000 7f0b0effd640 1 --2- 172.21.15.16:0/3092078221 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0b100c8840 0x7f0b1014cf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:05.638 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.631+0000 7f0b0effd640 1 -- 172.21.15.16:0/3092078221 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b040094d0 con 0x7f0b1006c8e0 2023-05-22T16:06:05.638 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.631+0000 7f0b0effd640 1 --2- 172.21.15.16:0/3092078221 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0b1006c8e0 0x7f0b100ae2a0 secure :-1 s=READY pgs=2619 cs=0 l=1 rev1=1 crypto rx=0x7f0b0000c820 tx=0x7f0b0000cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:06:05.638 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.631+0000 7f0af7fff640 1 -- 172.21.15.16:0/3092078221 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b00004220 con 0x7f0b1006c8e0 2023-05-22T16:06:05.639 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.631+0000 7f0af7fff640 1 -- 172.21.15.16:0/3092078221 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b000043a0 con 0x7f0b1006c8e0 2023-05-22T16:06:05.639 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.631+0000 7f0b1587c640 1 -- 172.21.15.16:0/3092078221 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b1006b7d0 con 0x7f0b1006c8e0 2023-05-22T16:06:05.639 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.631+0000 7f0af7fff640 1 -- 172.21.15.16:0/3092078221 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b00004510 con 0x7f0b1006c8e0 2023-05-22T16:06:05.639 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.631+0000 7f0b1587c640 1 -- 172.21.15.16:0/3092078221 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b100c9a00 con 0x7f0b1006c8e0 2023-05-22T16:06:05.640 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.635+0000 7f0b1587c640 1 -- 172.21.15.16:0/3092078221 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b1006b960 con 0x7f0b1006c8e0 2023-05-22T16:06:05.640 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.635+0000 7f0af7fff640 1 -- 172.21.15.16:0/3092078221 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 100) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f0b00007a70 con 0x7f0b1006c8e0 2023-05-22T16:06:05.640 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.635+0000 7f0af7fff640 1 --2- 172.21.15.16:0/3092078221 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f0afc0654a0 0x7f0afc067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:05.640 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.635+0000 7f0af7fff640 1 -- 172.21.15.16:0/3092078221 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(201..201 src has 1..201) v4 ==== 6317+0+0 (secure 0 0 0) 0x7f0b00082990 con 0x7f0b1006c8e0 2023-05-22T16:06:05.640 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.635+0000 7f0b0e7fc640 1 --2- 172.21.15.16:0/3092078221 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f0afc0654a0 0x7f0afc067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:05.641 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.635+0000 7f0b0e7fc640 1 --2- 172.21.15.16:0/3092078221 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f0afc0654a0 0x7f0afc067950 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f0b040ad000 tx=0x7f0b040a6000 comp rx=0 tx=0).ready entity=mgr.8195 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:06:05.642 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.639+0000 7f0af7fff640 1 -- 172.21.15.16:0/3092078221 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f0b000502f0 con 0x7f0b1006c8e0 2023-05-22T16:06:05.793 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.791+0000 7f0b1587c640 1 -- 172.21.15.16:0/3092078221 --> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f0b10151050 con 0x7f0afc0654a0 2023-05-22T16:06:05.801 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.799+0000 7f0af7fff640 1 -- 172.21.15.16:0/3092078221 <== mgr.8195 v2:172.21.15.16:6832/65890 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+71195 (secure 0 0 0) 0x7f0b10151050 con 0x7f0afc0654a0 2023-05-22T16:06:05.802 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T16:06:05.803 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2023-05-22T16:06:05.804 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.799+0000 7f0b1587c640 1 -- 172.21.15.16:0/3092078221 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f0afc0654a0 msgr2=0x7f0afc067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:06:05.804 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.799+0000 7f0b1587c640 1 --2- 172.21.15.16:0/3092078221 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f0afc0654a0 0x7f0afc067950 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f0b040ad000 tx=0x7f0b040a6000 comp rx=0 tx=0).stop 2023-05-22T16:06:05.804 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.799+0000 7f0b1587c640 1 -- 172.21.15.16:0/3092078221 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0b1006c8e0 msgr2=0x7f0b100ae2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:06:05.804 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.799+0000 7f0b1587c640 1 --2- 172.21.15.16:0/3092078221 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0b1006c8e0 0x7f0b100ae2a0 secure :-1 s=READY pgs=2619 cs=0 l=1 rev1=1 crypto rx=0x7f0b0000c820 tx=0x7f0b0000cce0 comp rx=0 tx=0).stop 2023-05-22T16:06:05.805 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.799+0000 7f0b1587c640 1 -- 172.21.15.16:0/3092078221 shutdown_connections 2023-05-22T16:06:05.805 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.799+0000 7f0b1587c640 1 --2- 172.21.15.16:0/3092078221 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0b10158be0 0x7f0b1014d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:05.805 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.799+0000 7f0b1587c640 1 --2- 172.21.15.16:0/3092078221 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0b100c8840 0x7f0b1014cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:05.805 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.799+0000 7f0b1587c640 1 --2- 172.21.15.16:0/3092078221 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f0afc0654a0 0x7f0afc067950 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:05.805 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.799+0000 7f0b1587c640 1 --2- 172.21.15.16:0/3092078221 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0b1006c8e0 0x7f0b100ae2a0 unknown :-1 s=CLOSED pgs=2619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:05.805 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.799+0000 7f0b1587c640 1 -- 172.21.15.16:0/3092078221 >> 172.21.15.16:0/3092078221 conn(0x7f0b100cab50 msgr2=0x7f0b1006bd70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:06:05.806 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.799+0000 7f0b1587c640 1 -- 172.21.15.16:0/3092078221 shutdown_connections 2023-05-22T16:06:05.806 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.799+0000 7f0b1587c640 1 -- 172.21.15.16:0/3092078221 wait complete. 2023-05-22T16:06:05.820 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":10,"stamp":"2023-05-22T16:06:03.797024+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2132,"num_read_kb":4426,"num_write":579,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":348,"ondisk_log_size":348,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":210,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":259176,"kb_used_data":21992,"kb_used_omap":0,"kb_used_meta":237120,"kb_avail":754715544,"statfs":{"total":773094113280,"available":772828717056,"internally_reserved":0,"allocated":22519808,"data_stored":13208106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":242810880},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-667690,"num_objects":-25,"num_object_clones":0,"num_object_copies":-50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-25,"num_whiteouts":0,"num_read":-317,"num_read_kb":-281,"num_write":-287,"num_write_kb":-2056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002017"},"pg_stats":[{"pgid":"1.0","version":"198'348","reported_seq":1160,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-05-22T16:05:55.897220+0000","last_change":"2023-05-22T15:26:39.318204+0000","last_active":"2023-05-22T16:05:55.897220+0000","last_peered":"2023-05-22T16:05:55.897220+0000","last_clean":"2023-05-22T16:05:55.897220+0000","last_became_active":"2023-05-22T15:26:39.315370+0000","last_became_peered":"2023-05-22T15:26:39.315370+0000","last_unstale":"2023-05-22T16:05:55.897220+0000","last_undegraded":"2023-05-22T16:05:55.897220+0000","last_fullsized":"2023-05-22T16:05:55.897220+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:38.020076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:38.020076+0000","last_clean_scrub_stamp":"2023-05-22T15:26:38.020076+0000","objects_scrubbed":0,"log_size":348,"log_dups_size":0,"ondisk_log_size":348,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:06:39.207543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2132,"num_read_kb":4426,"num_write":579,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":199,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-05-22T16:05:55.773610+0000","last_change":"2023-05-22T15:26:44.971951+0000","last_active":"2023-05-22T16:05:55.773610+0000","last_peered":"2023-05-22T16:05:55.773610+0000","last_clean":"2023-05-22T16:05:55.773610+0000","last_became_active":"2023-05-22T15:26:44.970872+0000","last_became_peered":"2023-05-22T15:26:44.970872+0000","last_unstale":"2023-05-22T16:05:55.773610+0000","last_undegraded":"2023-05-22T16:05:55.773610+0000","last_fullsized":"2023-05-22T16:05:55.773610+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:55:34.752949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":188,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-05-22T16:05:55.833896+0000","last_change":"2023-05-22T15:26:44.977198+0000","last_active":"2023-05-22T16:05:55.833896+0000","last_peered":"2023-05-22T16:05:55.833896+0000","last_clean":"2023-05-22T16:05:55.833896+0000","last_became_active":"2023-05-22T15:26:44.976445+0000","last_became_peered":"2023-05-22T15:26:44.976445+0000","last_unstale":"2023-05-22T16:05:55.833896+0000","last_undegraded":"2023-05-22T16:05:55.833896+0000","last_fullsized":"2023-05-22T16:05:55.833896+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:12:41.464054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":193,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-05-22T16:05:58.459713+0000","last_change":"2023-05-22T15:26:44.969469+0000","last_active":"2023-05-22T16:05:58.459713+0000","last_peered":"2023-05-22T16:05:58.459713+0000","last_clean":"2023-05-22T16:05:58.459713+0000","last_became_active":"2023-05-22T15:26:44.969083+0000","last_became_peered":"2023-05-22T15:26:44.969083+0000","last_unstale":"2023-05-22T16:05:58.459713+0000","last_undegraded":"2023-05-22T16:05:58.459713+0000","last_fullsized":"2023-05-22T16:05:58.459713+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:29:23.670067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":199,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-05-22T16:05:55.774421+0000","last_change":"2023-05-22T15:26:44.976632+0000","last_active":"2023-05-22T16:05:55.774421+0000","last_peered":"2023-05-22T16:05:55.774421+0000","last_clean":"2023-05-22T16:05:55.774421+0000","last_became_active":"2023-05-22T15:26:44.975546+0000","last_became_peered":"2023-05-22T15:26:44.975546+0000","last_unstale":"2023-05-22T16:05:55.774421+0000","last_undegraded":"2023-05-22T16:05:55.774421+0000","last_fullsized":"2023-05-22T16:05:55.774421+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:42:57.973030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":178,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-05-22T16:05:58.461430+0000","last_change":"2023-05-22T15:26:45.091475+0000","last_active":"2023-05-22T16:05:58.461430+0000","last_peered":"2023-05-22T16:05:58.461430+0000","last_clean":"2023-05-22T16:05:58.461430+0000","last_became_active":"2023-05-22T15:26:45.091143+0000","last_became_peered":"2023-05-22T15:26:45.091143+0000","last_unstale":"2023-05-22T16:05:58.461430+0000","last_undegraded":"2023-05-22T16:05:58.461430+0000","last_fullsized":"2023-05-22T16:05:58.461430+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:12:41.464054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":187,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-05-22T16:05:55.834651+0000","last_change":"2023-05-22T15:26:45.092367+0000","last_active":"2023-05-22T16:05:55.834651+0000","last_peered":"2023-05-22T16:05:55.834651+0000","last_clean":"2023-05-22T16:05:55.834651+0000","last_became_active":"2023-05-22T15:26:45.091951+0000","last_became_peered":"2023-05-22T15:26:45.091951+0000","last_unstale":"2023-05-22T16:05:55.834651+0000","last_undegraded":"2023-05-22T16:05:55.834651+0000","last_fullsized":"2023-05-22T16:05:55.834651+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:42:57.973030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":178,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-05-22T16:05:58.461984+0000","last_change":"2023-05-22T15:26:44.978858+0000","last_active":"2023-05-22T16:05:58.461984+0000","last_peered":"2023-05-22T16:05:58.461984+0000","last_clean":"2023-05-22T16:05:58.461984+0000","last_became_active":"2023-05-22T15:26:44.978443+0000","last_became_peered":"2023-05-22T15:26:44.978443+0000","last_unstale":"2023-05-22T16:05:58.461984+0000","last_undegraded":"2023-05-22T16:05:58.461984+0000","last_fullsized":"2023-05-22T16:05:58.461984+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:46:29.290243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":176,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-05-22T16:05:55.915820+0000","last_change":"2023-05-22T15:26:44.982421+0000","last_active":"2023-05-22T16:05:55.915820+0000","last_peered":"2023-05-22T16:05:55.915820+0000","last_clean":"2023-05-22T16:05:55.915820+0000","last_became_active":"2023-05-22T15:26:44.981586+0000","last_became_peered":"2023-05-22T15:26:44.981586+0000","last_unstale":"2023-05-22T16:05:55.915820+0000","last_undegraded":"2023-05-22T16:05:55.915820+0000","last_fullsized":"2023-05-22T16:05:55.915820+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:14:11.156544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2132,"num_read_kb":4426,"num_write":579,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":348,"ondisk_log_size":348,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575314,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29912,"kb_used_data":3344,"kb_used_omap":0,"kb_used_meta":26560,"kb_avail":94341928,"statfs":{"total":96636764160,"available":96606134272,"internally_reserved":0,"allocated":3424256,"data_stored":2254794,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27197440},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 16:05:53 2023","interfaces":[{"interface":"back","average":{"1min":0.81999999999999995,"5min":0.85099999999999998,"15min":0.86099999999999999},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":1.222,"5min":1.3069999999999999,"15min":1.3069999999999999},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.84999999999999998,"5min":0.88200000000000001,"15min":0.877},"min":{"1min":0.61299999999999999,"5min":0.501,"15min":0.501},"max":{"1min":1.296,"5min":1.296,"15min":1.296},"last":1.036}]},{"osd":1,"last update":"Mon May 22 16:05:53 2023","interfaces":[{"interface":"back","average":{"1min":0.90400000000000003,"5min":0.97299999999999998,"15min":0.98999999999999999},"min":{"1min":0.66000000000000003,"5min":0.63400000000000001,"15min":0.58299999999999996},"max":{"1min":1.274,"5min":1.431,"15min":1.431},"last":1.129},{"interface":"front","average":{"1min":0.95499999999999996,"5min":1.026,"15min":1.026},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":1.2909999999999999,"5min":1.345,"15min":1.345},"last":1.216}]},{"osd":2,"last update":"Mon May 22 16:05:53 2023","interfaces":[{"interface":"back","average":{"1min":0.83899999999999997,"5min":0.89900000000000002,"15min":0.90400000000000003},"min":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.171,"5min":1.286,"15min":1.407},"last":0.73399999999999999},{"interface":"front","average":{"1min":0.89100000000000001,"5min":0.94899999999999995,"15min":0.95899999999999996},"min":{"1min":0.53200000000000003,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":1.3899999999999999,"5min":1.3899999999999999,"15min":1.3899999999999999},"last":0.90500000000000003}]},{"osd":3,"last update":"Mon May 22 16:05:53 2023","interfaces":[{"interface":"back","average":{"1min":0.96399999999999997,"5min":0.93300000000000005,"15min":0.94799999999999995},"min":{"1min":0.755,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.2,"5min":1.2909999999999999,"15min":1.401},"last":1.0700000000000001},{"interface":"front","average":{"1min":0.97199999999999998,"5min":0.98699999999999999,"15min":0.98099999999999998},"min":{"1min":0.60299999999999998,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":1.4179999999999999,"5min":1.4179999999999999,"15min":1.4179999999999999},"last":0.85199999999999998}]},{"osd":4,"last update":"Mon May 22 16:05:53 2023","interfaces":[{"interface":"back","average":{"1min":0.875,"5min":0.88900000000000001,"15min":0.88600000000000001},"min":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51000000000000001},"max":{"1min":1.2310000000000001,"5min":1.3440000000000001,"15min":1.3440000000000001},"last":0.77600000000000002},{"interface":"front","average":{"1min":0.95599999999999996,"5min":0.99099999999999999,"15min":0.96499999999999997},"min":{"1min":0.60899999999999999,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":1.3480000000000001,"5min":1.405,"15min":1.405},"last":1.165}]},{"osd":5,"last update":"Mon May 22 16:05:53 2023","interfaces":[{"interface":"back","average":{"1min":0.96499999999999997,"5min":0.98899999999999999,"15min":0.98999999999999999},"min":{"1min":0.66200000000000003,"5min":0.623,"15min":0.441},"max":{"1min":1.159,"5min":1.2909999999999999,"15min":1.3759999999999999},"last":0.95799999999999996},{"interface":"front","average":{"1min":1.1180000000000001,"5min":1.1339999999999999,"15min":1.135},"min":{"1min":0.91000000000000003,"5min":0.72299999999999998,"15min":0.66600000000000004},"max":{"1min":1.468,"5min":1.468,"15min":1.468},"last":1.194}]},{"osd":6,"last update":"Mon May 22 16:05:53 2023","interfaces":[{"interface":"back","average":{"1min":1.048,"5min":1.0600000000000001,"15min":1.0569999999999999},"min":{"1min":0.72799999999999998,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":1.288,"5min":1.3169999999999999,"15min":1.389},"last":1.105},{"interface":"front","average":{"1min":1.008,"5min":1.0940000000000001,"15min":1.0880000000000001},"min":{"1min":0.70399999999999996,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":1.3149999999999999,"5min":1.391,"15min":1.7050000000000001},"last":0.99399999999999999}]}]},{"osd":6,"up_from":13,"seq":55834575315,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":22668,"kb_used_data":2436,"kb_used_omap":0,"kb_used_meta":20224,"kb_avail":94349172,"statfs":{"total":96636764160,"available":96613552128,"internally_reserved":0,"allocated":2494464,"data_stored":1336191,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":20709376},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 16:05:56 2023","interfaces":[{"interface":"back","average":{"1min":0.85999999999999999,"5min":0.88,"15min":0.89600000000000002},"min":{"1min":0.48599999999999999,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":2.6469999999999998,"5min":2.6469999999999998,"15min":2.6469999999999998},"last":1.093},{"interface":"front","average":{"1min":1.254,"5min":0.96899999999999997,"15min":0.92800000000000005},"min":{"1min":0.64600000000000002,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":6.7439999999999998,"5min":6.7439999999999998,"15min":6.7439999999999998},"last":0.95399999999999996}]},{"osd":1,"last update":"Mon May 22 16:05:56 2023","interfaces":[{"interface":"back","average":{"1min":1.0009999999999999,"5min":0.93000000000000005,"15min":0.94199999999999995},"min":{"1min":0.57399999999999995,"5min":0.55300000000000005,"15min":0.51100000000000001},"max":{"1min":2.3290000000000002,"5min":2.3290000000000002,"15min":2.3290000000000002},"last":0.93200000000000005},{"interface":"front","average":{"1min":1.2330000000000001,"5min":0.97899999999999998,"15min":0.94699999999999995},"min":{"1min":0.58199999999999996,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":6.5010000000000003,"5min":6.5010000000000003,"15min":6.5010000000000003},"last":1.1839999999999999}]},{"osd":2,"last update":"Mon May 22 16:05:56 2023","interfaces":[{"interface":"back","average":{"1min":1.2210000000000001,"5min":1.012,"15min":1.0109999999999999},"min":{"1min":0.57399999999999995,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":7.0170000000000003,"5min":7.0170000000000003,"15min":7.0170000000000003},"last":0.98999999999999999},{"interface":"front","average":{"1min":1.0589999999999999,"5min":1.05,"15min":1.0720000000000001},"min":{"1min":0.66200000000000003,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":2.403,"5min":2.403,"15min":2.403},"last":1.02}]},{"osd":3,"last update":"Mon May 22 16:05:56 2023","interfaces":[{"interface":"back","average":{"1min":1.278,"5min":1.054,"15min":1.0129999999999999},"min":{"1min":0.63800000000000001,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":6.6699999999999999,"5min":6.6699999999999999,"15min":6.6699999999999999},"last":1.133},{"interface":"front","average":{"1min":1.244,"5min":1.02,"15min":0.98899999999999999},"min":{"1min":0.68000000000000005,"5min":0.622,"15min":0.622},"max":{"1min":7.1260000000000003,"5min":7.1260000000000003,"15min":7.1260000000000003},"last":0.90400000000000003}]},{"osd":4,"last update":"Mon May 22 16:05:56 2023","interfaces":[{"interface":"back","average":{"1min":1.161,"5min":0.98099999999999998,"15min":0.96999999999999997},"min":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"max":{"1min":4.0709999999999997,"5min":4.0709999999999997,"15min":4.0709999999999997},"last":0.73599999999999999},{"interface":"front","average":{"1min":1.583,"5min":1.1619999999999999,"15min":1.075},"min":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":11.581,"5min":11.581,"15min":11.581},"last":1.2330000000000001}]},{"osd":5,"last update":"Mon May 22 16:05:56 2023","interfaces":[{"interface":"back","average":{"1min":1.925,"5min":1.1739999999999999,"15min":1.056},"min":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":17.587,"5min":17.587,"15min":17.587},"last":0.70699999999999996},{"interface":"front","average":{"1min":1.204,"5min":1.1180000000000001,"15min":1.1240000000000001},"min":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"max":{"1min":2.9620000000000002,"5min":2.9620000000000002,"15min":2.9620000000000002},"last":0.83099999999999996}]},{"osd":7,"last update":"Mon May 22 16:05:56 2023","interfaces":[{"interface":"back","average":{"1min":1.4319999999999999,"5min":1.202,"15min":1.1559999999999999},"min":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.58599999999999997},"max":{"1min":6.8339999999999996,"5min":6.8339999999999996,"15min":6.8339999999999996},"last":1.0629999999999999},{"interface":"front","average":{"1min":1.5429999999999999,"5min":1.2090000000000001,"15min":1.163},"min":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":8.5990000000000002,"5min":8.5990000000000002,"15min":8.5990000000000002},"last":0.78600000000000003}]}]},{"osd":5,"up_from":13,"seq":55834575315,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32780,"kb_used_data":2436,"kb_used_omap":0,"kb_used_meta":30336,"kb_avail":94339060,"statfs":{"total":96636764160,"available":96603197440,"internally_reserved":0,"allocated":2494464,"data_stored":1336191,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31064064},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 16:05:54 2023","interfaces":[{"interface":"back","average":{"1min":0.76000000000000001,"5min":0.85199999999999998,"15min":0.875},"min":{"1min":0.47899999999999998,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.0820000000000001,"5min":1.2749999999999999,"15min":1.3959999999999999},"last":0.98599999999999999},{"interface":"front","average":{"1min":0.86499999999999999,"5min":0.93999999999999995,"15min":0.94299999999999995},"min":{"1min":0.56499999999999995,"5min":0.56200000000000006,"15min":0.502},"max":{"1min":1.165,"5min":1.325,"15min":1.325},"last":1.0860000000000001}]},{"osd":1,"last update":"Mon May 22 16:05:54 2023","interfaces":[{"interface":"back","average":{"1min":0.86599999999999999,"5min":0.95999999999999996,"15min":1.0009999999999999},"min":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.2909999999999999,"5min":1.365,"15min":1.365},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.89600000000000002,"5min":1.028,"15min":1.0640000000000001},"min":{"1min":0.46899999999999997,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":1.216,"5min":1.3520000000000001,"15min":1.4630000000000001},"last":1.0569999999999999}]},{"osd":2,"last update":"Mon May 22 16:05:54 2023","interfaces":[{"interface":"back","average":{"1min":0.85299999999999998,"5min":0.90500000000000003,"15min":0.92000000000000004},"min":{"1min":0.60899999999999999,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.1859999999999999,"5min":1.3200000000000001,"15min":1.3740000000000001},"last":1.0149999999999999},{"interface":"front","average":{"1min":0.86799999999999999,"5min":0.94699999999999995,"15min":0.94599999999999995},"min":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":1.117,"5min":1.4350000000000001,"15min":1.4350000000000001},"last":0.92700000000000005}]},{"osd":3,"last update":"Mon May 22 16:05:54 2023","interfaces":[{"interface":"back","average":{"1min":0.91300000000000003,"5min":0.97699999999999998,"15min":0.97599999999999998},"min":{"1min":0.63400000000000001,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":1.1759999999999999,"5min":1.349,"15min":1.367},"last":0.83699999999999997},{"interface":"front","average":{"1min":0.97999999999999998,"5min":1.0329999999999999,"15min":1.048},"min":{"1min":0.67600000000000005,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.1779999999999999,"5min":1.3899999999999999,"15min":1.3899999999999999},"last":0.85599999999999998}]},{"osd":4,"last update":"Mon May 22 16:05:54 2023","interfaces":[{"interface":"back","average":{"1min":0.91200000000000003,"5min":0.93600000000000005,"15min":0.93100000000000005},"min":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.2170000000000001,"5min":1.6000000000000001,"15min":1.6000000000000001},"last":0.79900000000000004},{"interface":"front","average":{"1min":0.98899999999999999,"5min":0.98499999999999999,"15min":1.01},"min":{"1min":0.57899999999999996,"5min":0.498,"15min":0.498},"max":{"1min":1.371,"5min":1.4630000000000001,"15min":1.4630000000000001},"last":0.89900000000000002}]},{"osd":6,"last update":"Mon May 22 16:05:54 2023","interfaces":[{"interface":"back","average":{"1min":1.054,"5min":1.0129999999999999,"15min":1.0209999999999999},"min":{"1min":0.68999999999999995,"5min":0.63200000000000001,"15min":0.59799999999999998},"max":{"1min":1.2509999999999999,"5min":1.4630000000000001,"15min":1.4790000000000001},"last":1.147},{"interface":"front","average":{"1min":1.0449999999999999,"5min":1.073,"15min":1.081},"min":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":1.3440000000000001,"5min":1.6080000000000001,"15min":1.6080000000000001},"last":1.0349999999999999}]},{"osd":7,"last update":"Mon May 22 16:05:54 2023","interfaces":[{"interface":"back","average":{"1min":1.133,"5min":1.1180000000000001,"15min":1.1259999999999999},"min":{"1min":0.83899999999999997,"5min":0.57699999999999996,"15min":0.57699999999999996},"max":{"1min":1.3680000000000001,"5min":1.51,"15min":1.5109999999999999},"last":0.77300000000000002},{"interface":"front","average":{"1min":1.1739999999999999,"5min":1.1859999999999999,"15min":1.1930000000000001},"min":{"1min":0.91000000000000003,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":1.353,"5min":1.6679999999999999,"15min":1.7210000000000001},"last":0.95299999999999996}]}]},{"osd":4,"up_from":13,"seq":55834575313,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39584,"kb_used_data":2456,"kb_used_omap":0,"kb_used_meta":37120,"kb_avail":94332256,"statfs":{"total":96636764160,"available":96596230144,"internally_reserved":0,"allocated":2514944,"data_stored":1349186,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":38010880},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 16:05:02 2023","interfaces":[{"interface":"back","average":{"1min":0.90400000000000003,"5min":0.89600000000000002,"15min":0.90000000000000002},"min":{"1min":0.59999999999999998,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.1870000000000001,"5min":1.7210000000000001,"15min":1.7210000000000001},"last":0.72299999999999998},{"interface":"front","average":{"1min":0.995,"5min":0.96599999999999997,"15min":0.95199999999999996},"min":{"1min":0.70899999999999996,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":1.3360000000000001,"5min":1.7290000000000001,"15min":1.7290000000000001},"last":0.57099999999999995}]},{"osd":1,"last update":"Mon May 22 16:05:02 2023","interfaces":[{"interface":"back","average":{"1min":1.0269999999999999,"5min":0.97599999999999998,"15min":0.95899999999999996},"min":{"1min":0.76900000000000002,"5min":0.314,"15min":0.314},"max":{"1min":1.258,"5min":1.9339999999999999,"15min":1.9339999999999999},"last":0.96299999999999997},{"interface":"front","average":{"1min":0.90000000000000002,"5min":0.92800000000000005,"15min":0.93999999999999995},"min":{"1min":0.68799999999999994,"5min":0.504,"15min":0.504},"max":{"1min":1.0900000000000001,"5min":1.365,"15min":1.365},"last":0.79400000000000004}]},{"osd":2,"last update":"Mon May 22 16:05:02 2023","interfaces":[{"interface":"back","average":{"1min":0.98599999999999999,"5min":0.95499999999999996,"15min":0.94999999999999996},"min":{"1min":0.71799999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":1.28,"5min":1.4990000000000001,"15min":1.4990000000000001},"last":1.0129999999999999},{"interface":"front","average":{"1min":1.056,"5min":1.036,"15min":1.052},"min":{"1min":0.76000000000000001,"5min":0.44,"15min":0.44},"max":{"1min":1.2529999999999999,"5min":2.0249999999999999,"15min":2.0249999999999999},"last":0.84899999999999998}]},{"osd":3,"last update":"Mon May 22 16:05:02 2023","interfaces":[{"interface":"back","average":{"1min":1.0329999999999999,"5min":0.96499999999999997,"15min":0.96599999999999997},"min":{"1min":0.79200000000000004,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":1.258,"5min":1.4610000000000001,"15min":1.4610000000000001},"last":0.61699999999999999},{"interface":"front","average":{"1min":1.0760000000000001,"5min":1.0609999999999999,"15min":1.0489999999999999},"min":{"1min":0.83899999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":1.323,"5min":1.349,"15min":1.484},"last":1.125}]},{"osd":5,"last update":"Mon May 22 16:05:02 2023","interfaces":[{"interface":"back","average":{"1min":1.004,"5min":1.036,"15min":1.002},"min":{"1min":0.69599999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":1.325,"5min":2.1110000000000002,"15min":2.1110000000000002},"last":1.0880000000000001},{"interface":"front","average":{"1min":1.0309999999999999,"5min":1.0189999999999999,"15min":0.996},"min":{"1min":0.69999999999999996,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.1990000000000001,"5min":1.5449999999999999,"15min":1.5449999999999999},"last":1.0620000000000001}]},{"osd":6,"last update":"Mon May 22 16:05:02 2023","interfaces":[{"interface":"back","average":{"1min":1.0669999999999999,"5min":1.036,"15min":1.0009999999999999},"min":{"1min":0.75800000000000001,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":1.387,"5min":1.397,"15min":1.4339999999999999},"last":1.2},{"interface":"front","average":{"1min":1.1599999999999999,"5min":1.1599999999999999,"15min":1.137},"min":{"1min":0.875,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.3340000000000001,"5min":2.2690000000000001,"15min":2.2690000000000001},"last":1.1639999999999999}]},{"osd":7,"last update":"Mon May 22 16:05:02 2023","interfaces":[{"interface":"back","average":{"1min":1.093,"5min":1.0960000000000001,"15min":1.069},"min":{"1min":0.83599999999999997,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":1.3029999999999999,"5min":1.623,"15min":1.623},"last":0.91300000000000003},{"interface":"front","average":{"1min":1.2150000000000001,"5min":1.1879999999999999,"15min":1.1599999999999999},"min":{"1min":1.0169999999999999,"5min":0.71199999999999997,"15min":0.68100000000000005},"max":{"1min":1.4390000000000001,"5min":1.448,"15min":1.5089999999999999},"last":1.2529999999999999}]}]},{"osd":3,"up_from":13,"seq":55834575313,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27928,"kb_used_data":2768,"kb_used_omap":0,"kb_used_meta":25152,"kb_avail":94343912,"statfs":{"total":96636764160,"available":96608165888,"internally_reserved":0,"allocated":2834432,"data_stored":1664516,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":25755648},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 16:05:38 2023","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"min":{"1min":0.29299999999999998,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.93700000000000006,"5min":1.0049999999999999,"15min":1.0229999999999999},"last":0.81899999999999995},{"interface":"front","average":{"1min":0.70299999999999996,"5min":0.76200000000000001,"15min":0.76900000000000002},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":1.2749999999999999,"5min":1.7230000000000001,"15min":1.7230000000000001},"last":0.45900000000000002}]},{"osd":1,"last update":"Mon May 22 16:05:38 2023","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.75600000000000001,"15min":0.71999999999999997},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":1.2190000000000001,"5min":2.4969999999999999,"15min":2.4969999999999999},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.76000000000000001,"5min":0.80700000000000005,"15min":0.80100000000000005},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":1.3520000000000001,"5min":2.4769999999999999,"15min":2.4769999999999999},"last":0.66200000000000003}]},{"osd":2,"last update":"Mon May 22 16:05:38 2023","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":0.83699999999999997,"15min":0.84899999999999998},"min":{"1min":0.52300000000000002,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.94799999999999995,"5min":2.2250000000000001,"15min":2.2250000000000001},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.87,"5min":0.83599999999999997,"15min":0.83299999999999996},"min":{"1min":0.55500000000000005,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":1.129,"5min":2.1680000000000001,"15min":2.1680000000000001},"last":0.76400000000000001}]},{"osd":4,"last update":"Mon May 22 16:05:38 2023","interfaces":[{"interface":"back","average":{"1min":0.95199999999999996,"5min":0.98099999999999998,"15min":0.97399999999999998},"min":{"1min":0.36799999999999999,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":1.357,"5min":2.3479999999999999,"15min":2.3479999999999999},"last":1.0009999999999999},{"interface":"front","average":{"1min":1.0009999999999999,"5min":1.022,"15min":1.03},"min":{"1min":0.39400000000000002,"5min":0.37,"15min":0.37},"max":{"1min":1.2210000000000001,"5min":1.788,"15min":1.788},"last":1.135}]},{"osd":5,"last update":"Mon May 22 16:05:38 2023","interfaces":[{"interface":"back","average":{"1min":1.0069999999999999,"5min":1.0409999999999999,"15min":1.05},"min":{"1min":0.496,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":1.21,"5min":1.8129999999999999,"15min":1.8129999999999999},"last":1.0409999999999999},{"interface":"front","average":{"1min":0.98299999999999998,"5min":1.0169999999999999,"15min":1.0369999999999999},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.3140000000000001,"5min":1.571,"15min":1.571},"last":0.86599999999999999}]},{"osd":6,"last update":"Mon May 22 16:05:38 2023","interfaces":[{"interface":"back","average":{"1min":1.0109999999999999,"5min":1.0680000000000001,"15min":1.0880000000000001},"min":{"1min":0.45600000000000002,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":1.274,"5min":1.76,"15min":1.76},"last":1.2090000000000001},{"interface":"front","average":{"1min":1.0609999999999999,"5min":1.0820000000000001,"15min":1.099},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":1.4139999999999999,"5min":1.754,"15min":1.754},"last":0.91000000000000003}]},{"osd":7,"last update":"Mon May 22 16:05:38 2023","interfaces":[{"interface":"back","average":{"1min":1.1020000000000001,"5min":1.111,"15min":1.127},"min":{"1min":0.53800000000000003,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.5049999999999999,"5min":1.673,"15min":1.673},"last":1.081},{"interface":"front","average":{"1min":1.1040000000000001,"5min":1.1160000000000001,"15min":1.109},"min":{"1min":0.41099999999999998,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":1.4670000000000001,"5min":2.3079999999999998,"15min":2.3079999999999998},"last":0.88}]}]},{"osd":2,"up_from":13,"seq":55834575314,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31316,"kb_used_data":2444,"kb_used_omap":0,"kb_used_meta":28864,"kb_avail":94340524,"statfs":{"total":96636764160,"available":96604696576,"internally_reserved":0,"allocated":2502656,"data_stored":1336768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":29556736},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 16:05:35 2023","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.629,"15min":0.67000000000000004},"min":{"1min":0.28599999999999998,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.90100000000000002,"5min":1.0549999999999999,"15min":1.1519999999999999},"last":0.78400000000000003},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.71799999999999997,"15min":0.74199999999999999},"min":{"1min":0.35299999999999998,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":1.0309999999999999,"5min":1.1659999999999999,"15min":1.2090000000000001},"last":0.66100000000000003}]},{"osd":1,"last update":"Mon May 22 16:05:35 2023","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.74399999999999999,"15min":0.75700000000000001},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.875,"5min":1.071,"15min":1.099},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.79300000000000004,"5min":0.82099999999999995,"15min":0.82599999999999996},"min":{"1min":0.39500000000000002,"5min":0.311,"15min":0.311},"max":{"1min":1.069,"5min":1.1739999999999999,"15min":1.2589999999999999},"last":0.875}]},{"osd":3,"last update":"Mon May 22 16:05:35 2023","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":0.75600000000000001,"15min":0.75800000000000001},"min":{"1min":0.32100000000000001,"5min":0.315,"15min":0.315},"max":{"1min":0.91400000000000003,"5min":1.3540000000000001,"15min":1.3540000000000001},"last":0.83899999999999997},{"interface":"front","average":{"1min":0.85199999999999998,"5min":0.87,"15min":0.89300000000000002},"min":{"1min":0.41399999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":1.1859999999999999,"5min":1.2949999999999999,"15min":1.2949999999999999},"last":0.95099999999999996}]},{"osd":4,"last update":"Mon May 22 16:05:35 2023","interfaces":[{"interface":"back","average":{"1min":1.02,"5min":1.024,"15min":1.04},"min":{"1min":0.66100000000000003,"5min":0.441,"15min":0.441},"max":{"1min":1.4590000000000001,"5min":1.516,"15min":1.6539999999999999},"last":1.0249999999999999},{"interface":"front","average":{"1min":0.996,"5min":1.0089999999999999,"15min":1.04},"min":{"1min":0.55600000000000005,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.419,"5min":1.4299999999999999,"15min":1.538},"last":1.1060000000000001}]},{"osd":5,"last update":"Mon May 22 16:05:35 2023","interfaces":[{"interface":"back","average":{"1min":1.03,"5min":1.024,"15min":1.0549999999999999},"min":{"1min":0.53600000000000003,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":1.4139999999999999,"5min":1.4139999999999999,"15min":1.605},"last":1.2350000000000001},{"interface":"front","average":{"1min":1.0069999999999999,"5min":1.038,"15min":1.03},"min":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":1.339,"5min":1.534,"15min":1.534},"last":0.92200000000000004}]},{"osd":6,"last update":"Mon May 22 16:05:35 2023","interfaces":[{"interface":"back","average":{"1min":1.095,"5min":1.0589999999999999,"15min":1.0349999999999999},"min":{"1min":0.72699999999999998,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":1.4259999999999999,"5min":1.429,"15min":1.462},"last":0.99399999999999999},{"interface":"front","average":{"1min":1.127,"5min":1.105,"15min":1.1339999999999999},"min":{"1min":0.59199999999999997,"5min":0.57099999999999995,"15min":0.57099999999999995},"max":{"1min":1.5269999999999999,"5min":1.621,"15min":1.7110000000000001},"last":1.0760000000000001}]},{"osd":7,"last update":"Mon May 22 16:05:35 2023","interfaces":[{"interface":"back","average":{"1min":1.0940000000000001,"5min":1.0860000000000001,"15min":1.1220000000000001},"min":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.5680000000000001,"5min":1.5680000000000001,"15min":1.6399999999999999},"last":1.179},{"interface":"front","average":{"1min":1.0940000000000001,"5min":1.107,"15min":1.1140000000000001},"min":{"1min":0.629,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":1.4610000000000001,"5min":1.673,"15min":1.673},"last":1.0389999999999999}]}]},{"osd":1,"up_from":13,"seq":55834575314,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37724,"kb_used_data":2772,"kb_used_omap":0,"kb_used_meta":34944,"kb_avail":94334116,"statfs":{"total":96636764160,"available":96598134784,"internally_reserved":0,"allocated":2838528,"data_stored":1676300,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":35782656},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 16:05:58 2023","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.64800000000000002,"15min":0.66200000000000003},"min":{"1min":0.34499999999999997,"5min":0.308,"15min":0.308},"max":{"1min":1.0449999999999999,"5min":1.0449999999999999,"15min":1.329},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.72199999999999998,"15min":0.76300000000000001},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.95499999999999996,"5min":1.25,"15min":1.268},"last":0.85099999999999998}]},{"osd":2,"last update":"Mon May 22 16:05:58 2023","interfaces":[{"interface":"back","average":{"1min":0.70999999999999996,"5min":0.73399999999999999,"15min":0.75600000000000001},"min":{"1min":0.372,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":1.141,"5min":1.141,"15min":1.26},"last":0.96099999999999997},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.76300000000000001,"15min":0.77700000000000002},"min":{"1min":0.46999999999999997,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":1.0249999999999999,"5min":1.127,"15min":1.268},"last":0.78900000000000003}]},{"osd":3,"last update":"Mon May 22 16:05:58 2023","interfaces":[{"interface":"back","average":{"1min":0.78300000000000003,"5min":0.83799999999999997,"15min":0.878},"min":{"1min":0.39900000000000002,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":1.1100000000000001,"5min":1.2829999999999999,"15min":1.2829999999999999},"last":1.1100000000000001},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.80300000000000005,"15min":0.81899999999999995},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":1.2050000000000001,"5min":1.2749999999999999,"15min":1.2749999999999999},"last":0.38500000000000001}]},{"osd":4,"last update":"Mon May 22 16:05:58 2023","interfaces":[{"interface":"back","average":{"1min":0.93899999999999995,"5min":0.96599999999999997,"15min":0.98399999999999999},"min":{"1min":0.58499999999999996,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":1.3500000000000001,"5min":1.542,"15min":1.542},"last":1.046},{"interface":"front","average":{"1min":1.006,"5min":0.997,"15min":1.024},"min":{"1min":0.71499999999999997,"5min":0.434,"15min":0.434},"max":{"1min":1.2729999999999999,"5min":1.327,"15min":1.4650000000000001},"last":1.2390000000000001}]},{"osd":5,"last update":"Mon May 22 16:05:58 2023","interfaces":[{"interface":"back","average":{"1min":1.0309999999999999,"5min":1.0349999999999999,"15min":1.048},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.319,"5min":1.605,"15min":1.605},"last":0.997},{"interface":"front","average":{"1min":1.0589999999999999,"5min":1.0620000000000001,"15min":1.0760000000000001},"min":{"1min":0.75900000000000001,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":1.4370000000000001,"5min":1.4370000000000001,"15min":1.5389999999999999},"last":1.151}]},{"osd":6,"last update":"Mon May 22 16:05:58 2023","interfaces":[{"interface":"back","average":{"1min":1.03,"5min":1.0469999999999999,"15min":1.0640000000000001},"min":{"1min":0.56899999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"max":{"1min":1.4770000000000001,"5min":1.4770000000000001,"15min":1.5249999999999999},"last":1.3580000000000001},{"interface":"front","average":{"1min":1.0589999999999999,"5min":1.099,"15min":1.1100000000000001},"min":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"max":{"1min":1.387,"5min":1.4359999999999999,"15min":1.6479999999999999},"last":1.1890000000000001}]},{"osd":7,"last update":"Mon May 22 16:05:58 2023","interfaces":[{"interface":"back","average":{"1min":1.0509999999999999,"5min":1.123,"15min":1.153},"min":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":1.484,"5min":1.7030000000000001,"15min":1.7030000000000001},"last":1.2829999999999999},{"interface":"front","average":{"1min":1.1339999999999999,"5min":1.131,"15min":1.151},"min":{"1min":0.78400000000000003,"5min":0.73599999999999999,"15min":0.73599999999999999},"max":{"1min":1.4810000000000001,"5min":1.6439999999999999,"15min":1.6439999999999999},"last":1.319}]}]},{"osd":0,"up_from":13,"seq":55834575315,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37264,"kb_used_data":3336,"kb_used_omap":0,"kb_used_meta":33920,"kb_avail":94334576,"statfs":{"total":96636764160,"available":96598605824,"internally_reserved":0,"allocated":3416064,"data_stored":2254160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":34734080},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon May 22 16:06:01 2023","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.72399999999999998,"15min":0.71099999999999997},"min":{"1min":0.42299999999999999,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":1.1140000000000001,"5min":1.296,"15min":1.296},"last":0.73799999999999999},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.72999999999999998,"15min":0.73899999999999999},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.86899999999999999,"5min":1.181,"15min":1.3149999999999999},"last":0.60699999999999998}]},{"osd":2,"last update":"Mon May 22 16:06:01 2023","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.751,"15min":0.76900000000000002},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":1.071,"5min":1.22,"15min":1.284},"last":0.877},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.78700000000000003,"15min":0.81299999999999994},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":1.042,"5min":1.323,"15min":1.323},"last":0.70399999999999996}]},{"osd":3,"last update":"Mon May 22 16:06:01 2023","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.746,"15min":0.77100000000000002},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":1.228,"5min":1.2350000000000001,"15min":1.2350000000000001},"last":0.83099999999999996},{"interface":"front","average":{"1min":0.78300000000000003,"5min":0.85899999999999999,"15min":0.874},"min":{"1min":0.43099999999999999,"5min":0.377,"15min":0.377},"max":{"1min":1.22,"5min":1.2949999999999999,"15min":1.331},"last":1.0600000000000001}]},{"osd":4,"last update":"Mon May 22 16:06:01 2023","interfaces":[{"interface":"back","average":{"1min":0.90200000000000002,"5min":0.94599999999999995,"15min":0.95199999999999996},"min":{"1min":0.64100000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"max":{"1min":1.1819999999999999,"5min":1.264,"15min":1.4059999999999999},"last":1.085},{"interface":"front","average":{"1min":0.91500000000000004,"5min":1.042,"15min":1.05},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.1619999999999999,"5min":1.401,"15min":1.482},"last":0.93999999999999995}]},{"osd":5,"last update":"Mon May 22 16:06:01 2023","interfaces":[{"interface":"back","average":{"1min":0.92600000000000005,"5min":1.014,"15min":1.0349999999999999},"min":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":1.1830000000000001,"5min":1.325,"15min":1.6419999999999999},"last":1.016},{"interface":"front","average":{"1min":0.90200000000000002,"5min":1.0009999999999999,"15min":1.0109999999999999},"min":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"max":{"1min":1.1859999999999999,"5min":1.413,"15min":1.458},"last":0.96899999999999997}]},{"osd":6,"last update":"Mon May 22 16:06:01 2023","interfaces":[{"interface":"back","average":{"1min":0.98399999999999999,"5min":1.091,"15min":1.1060000000000001},"min":{"1min":0.66400000000000003,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":1.2729999999999999,"5min":1.407,"15min":1.77},"last":1.1479999999999999},{"interface":"front","average":{"1min":0.96399999999999997,"5min":1.093,"15min":1.1040000000000001},"min":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"max":{"1min":1.2589999999999999,"5min":1.5409999999999999,"15min":1.5409999999999999},"last":1.2070000000000001}]},{"osd":7,"last update":"Mon May 22 16:06:01 2023","interfaces":[{"interface":"back","average":{"1min":1.034,"5min":1.1279999999999999,"15min":1.1359999999999999},"min":{"1min":0.69299999999999995,"5min":0.63900000000000001,"15min":0.63900000000000001},"max":{"1min":1.341,"5min":1.6020000000000001,"15min":1.6020000000000001},"last":1.1259999999999999},{"interface":"front","average":{"1min":1.0109999999999999,"5min":1.0780000000000001,"15min":1.095},"min":{"1min":0.68999999999999995,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.2989999999999999,"5min":1.4650000000000001,"15min":1.4990000000000001},"last":1.173}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-05-22T16:06:05.823 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-05-22T16:06:05.824 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-05-22T16:06:05.948 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.947+0000 7fb1c5085640 1 Processor -- start 2023-05-22T16:06:05.949 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.947+0000 7fb1c5085640 1 -- start start 2023-05-22T16:06:05.949 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.947+0000 7fb1c5085640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb1c01565d0 0x7fb1c00d0090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:05.950 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.947+0000 7fb1c5085640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fb1c015cdb0 0x7fb1c00b0440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:05.950 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.947+0000 7fb1c5085640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1c00cd650 con 0x7fb1c00d05d0 2023-05-22T16:06:05.950 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.947+0000 7fb1c5085640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1c00ccb10 con 0x7fb1c01565d0 2023-05-22T16:06:05.950 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.947+0000 7fb1c5085640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1c00d0a50 con 0x7fb1c015cdb0 2023-05-22T16:06:05.951 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.947+0000 7fb1bed9d640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb1c01565d0 0x7fb1c00d0090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:05.952 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.947+0000 7fb1bed9d640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb1c01565d0 0x7fb1c00d0090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43504/0 (socket says 172.21.15.16:43504) 2023-05-22T16:06:05.952 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.947+0000 7fb1bed9d640 1 -- 172.21.15.16:0/2728401646 learned_addr learned my addr 172.21.15.16:0/2728401646 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:06:05.952 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.947+0000 7fb1bf59e640 1 --2- 172.21.15.16:0/2728401646 >> v2:172.21.15.62:3300/0 conn(0x7fb1c015cdb0 0x7fb1c00b0440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:05.952 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.947+0000 7fb1bed9d640 1 -- 172.21.15.16:0/2728401646 >> v2:172.21.15.62:3300/0 conn(0x7fb1c015cdb0 msgr2=0x7fb1c00b0440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:06:05.952 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.947+0000 7fb1bed9d640 1 --2- 172.21.15.16:0/2728401646 >> v2:172.21.15.62:3300/0 conn(0x7fb1c015cdb0 0x7fb1c00b0440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:05.952 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.947+0000 7fb1bed9d640 1 -- 172.21.15.16:0/2728401646 >> v1:172.21.15.62:6789/0 conn(0x7fb1c00d05d0 legacy=0x7fb1c015c6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:06:05.953 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.947+0000 7fb1bed9d640 1 -- 172.21.15.16:0/2728401646 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1c00ccd20 con 0x7fb1c01565d0 2023-05-22T16:06:05.954 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.951+0000 7fb1bed9d640 1 --2- 172.21.15.16:0/2728401646 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb1c01565d0 0x7fb1c00d0090 secure :-1 s=READY pgs=2620 cs=0 l=1 rev1=1 crypto rx=0x7fb1c008b820 tx=0x7fb1b801c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3484052abcf8406 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:06:05.954 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.951+0000 7fb1bdd9b640 1 -- 172.21.15.16:0/2728401646 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1b80ab5f0 con 0x7fb1c01565d0 2023-05-22T16:06:05.954 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.951+0000 7fb1bdd9b640 1 -- 172.21.15.16:0/2728401646 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1b80ab750 con 0x7fb1c01565d0 2023-05-22T16:06:05.954 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.951+0000 7fb1bdd9b640 1 -- 172.21.15.16:0/2728401646 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1b80abae0 con 0x7fb1c01565d0 2023-05-22T16:06:05.954 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.951+0000 7fb1c5085640 1 -- 172.21.15.16:0/2728401646 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb1c01565d0 msgr2=0x7fb1c00d0090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:06:05.955 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.951+0000 7fb1c5085640 1 --2- 172.21.15.16:0/2728401646 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb1c01565d0 0x7fb1c00d0090 secure :-1 s=READY pgs=2620 cs=0 l=1 rev1=1 crypto rx=0x7fb1c008b820 tx=0x7fb1b801c500 comp rx=0 tx=0).stop 2023-05-22T16:06:05.955 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.951+0000 7fb1c5085640 1 -- 172.21.15.16:0/2728401646 shutdown_connections 2023-05-22T16:06:05.955 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.951+0000 7fb1c5085640 1 --2- 172.21.15.16:0/2728401646 >> v2:172.21.15.62:3300/0 conn(0x7fb1c015cdb0 0x7fb1c00b0440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:05.955 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.951+0000 7fb1c5085640 1 --2- 172.21.15.16:0/2728401646 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb1c01565d0 0x7fb1c00d0090 unknown :-1 s=CLOSED pgs=2620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:05.955 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.951+0000 7fb1c5085640 1 -- 172.21.15.16:0/2728401646 >> 172.21.15.16:0/2728401646 conn(0x7fb1c0071850 msgr2=0x7fb1c009b830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:06:05.955 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.951+0000 7fb1c5085640 1 -- 172.21.15.16:0/2728401646 shutdown_connections 2023-05-22T16:06:05.955 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.951+0000 7fb1c5085640 1 -- 172.21.15.16:0/2728401646 wait complete. 2023-05-22T16:06:05.957 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.951+0000 7fb1c5085640 1 Processor -- start 2023-05-22T16:06:05.958 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.951+0000 7fb1c5085640 1 -- start start 2023-05-22T16:06:05.958 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.951+0000 7fb1c5085640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb1c00d05d0 0x7fb1c00b6ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:05.958 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.951+0000 7fb1c5085640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb1c01565d0 0x7fb1c00c9c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:05.958 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.951+0000 7fb1c5085640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb1c015cdb0 0x7fb1c00b5d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:05.958 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.951+0000 7fb1c5085640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1c00b1bc0 con 0x7fb1c00d05d0 2023-05-22T16:06:05.958 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.951+0000 7fb1c5085640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1c00b1ea0 con 0x7fb1c015cdb0 2023-05-22T16:06:05.958 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.951+0000 7fb1c5085640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1c00b2180 con 0x7fb1c01565d0 2023-05-22T16:06:05.959 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.951+0000 7fb1bed9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb1c00d05d0 0x7fb1c00b6ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:05.959 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.951+0000 7fb1bed9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb1c00d05d0 0x7fb1c00b6ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43506/0 (socket says 172.21.15.16:43506) 2023-05-22T16:06:05.959 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.951+0000 7fb1bed9d640 1 -- 172.21.15.16:0/2355850630 learned_addr learned my addr 172.21.15.16:0/2355850630 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:06:05.959 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.951+0000 7fb1bf59e640 1 --2- 172.21.15.16:0/2355850630 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb1c015cdb0 0x7fb1c00b5d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:05.959 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.955+0000 7fb1bed9d640 1 -- 172.21.15.16:0/2355850630 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb1c01565d0 msgr2=0x7fb1c00c9c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:06:05.959 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.955+0000 7fb1be59c640 1 --2- 172.21.15.16:0/2355850630 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb1c01565d0 0x7fb1c00c9c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:05.959 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.955+0000 7fb1bed9d640 1 --2- 172.21.15.16:0/2355850630 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb1c01565d0 0x7fb1c00c9c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:05.960 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.955+0000 7fb1bed9d640 1 -- 172.21.15.16:0/2355850630 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb1c015cdb0 msgr2=0x7fb1c00b5d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:06:05.960 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.955+0000 7fb1bed9d640 1 --2- 172.21.15.16:0/2355850630 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb1c015cdb0 0x7fb1c00b5d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:05.960 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.955+0000 7fb1bed9d640 1 -- 172.21.15.16:0/2355850630 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1b8009470 con 0x7fb1c00d05d0 2023-05-22T16:06:05.960 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.955+0000 7fb1bed9d640 1 --2- 172.21.15.16:0/2355850630 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb1c00d05d0 0x7fb1c00b6ad0 secure :-1 s=READY pgs=2621 cs=0 l=1 rev1=1 crypto rx=0x7fb1b80ad040 tx=0x7fb1b801ad40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:06:05.960 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.955+0000 7fb1a77fe640 1 -- 172.21.15.16:0/2355850630 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1b80be5f0 con 0x7fb1c00d05d0 2023-05-22T16:06:05.960 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.955+0000 7fb1a77fe640 1 -- 172.21.15.16:0/2355850630 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1b80be750 con 0x7fb1c00d05d0 2023-05-22T16:06:05.960 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.955+0000 7fb1a77fe640 1 -- 172.21.15.16:0/2355850630 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1b80beb00 con 0x7fb1c00d05d0 2023-05-22T16:06:05.961 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.955+0000 7fb1c5085640 1 -- 172.21.15.16:0/2355850630 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1c00af220 con 0x7fb1c00d05d0 2023-05-22T16:06:05.961 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.955+0000 7fb1c5085640 1 -- 172.21.15.16:0/2355850630 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1c00af090 con 0x7fb1c00d05d0 2023-05-22T16:06:05.962 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.955+0000 7fb1c5085640 1 -- 172.21.15.16:0/2355850630 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1c00cdae0 con 0x7fb1c00d05d0 2023-05-22T16:06:05.962 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.955+0000 7fb1a77fe640 1 -- 172.21.15.16:0/2355850630 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 100) v1 ==== 84332+0+0 (secure 0 0 0) 0x7fb1b80b43f0 con 0x7fb1c00d05d0 2023-05-22T16:06:05.962 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.955+0000 7fb1a77fe640 1 --2- 172.21.15.16:0/2355850630 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fb1ac0654a0 0x7fb1ac067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:05.962 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.955+0000 7fb1a77fe640 1 -- 172.21.15.16:0/2355850630 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(201..201 src has 1..201) v4 ==== 6317+0+0 (secure 0 0 0) 0x7fb1b80a8050 con 0x7fb1c00d05d0 2023-05-22T16:06:05.962 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.955+0000 7fb1be59c640 1 --2- 172.21.15.16:0/2355850630 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fb1ac0654a0 0x7fb1ac067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:05.962 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.955+0000 7fb1be59c640 1 --2- 172.21.15.16:0/2355850630 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fb1ac0654a0 0x7fb1ac067950 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fb1c00ceac0 tx=0x7fb1b4009040 comp rx=0 tx=0).ready entity=mgr.8195 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:06:05.963 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:05.959+0000 7fb1a77fe640 1 -- 172.21.15.16:0/2355850630 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fb1b80eb1c0 con 0x7fb1c00d05d0 2023-05-22T16:06:06.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:06.111+0000 7fb1c5085640 1 -- 172.21.15.16:0/2355850630 --> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb1c0155f50 con 0x7fb1ac0654a0 2023-05-22T16:06:06.121 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:06.119+0000 7fb1a77fe640 1 -- 172.21.15.16:0/2355850630 <== mgr.8195 v2:172.21.15.16:6832/65890 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+71195 (secure 0 0 0) 0x7fb1c0155f50 con 0x7fb1ac0654a0 2023-05-22T16:06:06.122 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T16:06:06.123 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2023-05-22T16:06:06.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:06.119+0000 7fb1c5085640 1 -- 172.21.15.16:0/2355850630 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fb1ac0654a0 msgr2=0x7fb1ac067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:06:06.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:06.119+0000 7fb1c5085640 1 --2- 172.21.15.16:0/2355850630 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fb1ac0654a0 0x7fb1ac067950 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fb1c00ceac0 tx=0x7fb1b4009040 comp rx=0 tx=0).stop 2023-05-22T16:06:06.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:06.119+0000 7fb1c5085640 1 -- 172.21.15.16:0/2355850630 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb1c00d05d0 msgr2=0x7fb1c00b6ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:06:06.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:06.119+0000 7fb1c5085640 1 --2- 172.21.15.16:0/2355850630 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb1c00d05d0 0x7fb1c00b6ad0 secure :-1 s=READY pgs=2621 cs=0 l=1 rev1=1 crypto rx=0x7fb1b80ad040 tx=0x7fb1b801ad40 comp rx=0 tx=0).stop 2023-05-22T16:06:06.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:06.119+0000 7fb1c5085640 1 -- 172.21.15.16:0/2355850630 shutdown_connections 2023-05-22T16:06:06.126 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:06.119+0000 7fb1c5085640 1 --2- 172.21.15.16:0/2355850630 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb1c015cdb0 0x7fb1c00b5d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:06.126 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:06.119+0000 7fb1c5085640 1 --2- 172.21.15.16:0/2355850630 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb1c01565d0 0x7fb1c00c9c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:06.126 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:06.119+0000 7fb1c5085640 1 --2- 172.21.15.16:0/2355850630 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fb1ac0654a0 0x7fb1ac067950 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:06.126 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:06.119+0000 7fb1c5085640 1 --2- 172.21.15.16:0/2355850630 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb1c00d05d0 0x7fb1c00b6ad0 unknown :-1 s=CLOSED pgs=2621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:06.126 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:06.119+0000 7fb1c5085640 1 -- 172.21.15.16:0/2355850630 >> 172.21.15.16:0/2355850630 conn(0x7fb1c0071850 msgr2=0x7fb1c00ce2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:06:06.126 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:06.119+0000 7fb1c5085640 1 -- 172.21.15.16:0/2355850630 shutdown_connections 2023-05-22T16:06:06.126 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:06.119+0000 7fb1c5085640 1 -- 172.21.15.16:0/2355850630 wait complete. 2023-05-22T16:06:06.141 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":11,"stamp":"2023-05-22T16:06:05.802334+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2132,"num_read_kb":4426,"num_write":579,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":348,"ondisk_log_size":348,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":210,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":259176,"kb_used_data":21992,"kb_used_omap":0,"kb_used_meta":237120,"kb_avail":754715544,"statfs":{"total":773094113280,"available":772828717056,"internally_reserved":0,"allocated":22519808,"data_stored":13208106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":242810880},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-667690,"num_objects":-25,"num_object_clones":0,"num_object_copies":-50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-25,"num_whiteouts":0,"num_read":-317,"num_read_kb":-281,"num_write":-287,"num_write_kb":-2056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.344338"},"pg_stats":[{"pgid":"1.0","version":"198'348","reported_seq":1160,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-05-22T16:05:55.897220+0000","last_change":"2023-05-22T15:26:39.318204+0000","last_active":"2023-05-22T16:05:55.897220+0000","last_peered":"2023-05-22T16:05:55.897220+0000","last_clean":"2023-05-22T16:05:55.897220+0000","last_became_active":"2023-05-22T15:26:39.315370+0000","last_became_peered":"2023-05-22T15:26:39.315370+0000","last_unstale":"2023-05-22T16:05:55.897220+0000","last_undegraded":"2023-05-22T16:05:55.897220+0000","last_fullsized":"2023-05-22T16:05:55.897220+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:38.020076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:38.020076+0000","last_clean_scrub_stamp":"2023-05-22T15:26:38.020076+0000","objects_scrubbed":0,"log_size":348,"log_dups_size":0,"ondisk_log_size":348,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:06:39.207543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2132,"num_read_kb":4426,"num_write":579,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":199,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-05-22T16:05:55.773610+0000","last_change":"2023-05-22T15:26:44.971951+0000","last_active":"2023-05-22T16:05:55.773610+0000","last_peered":"2023-05-22T16:05:55.773610+0000","last_clean":"2023-05-22T16:05:55.773610+0000","last_became_active":"2023-05-22T15:26:44.970872+0000","last_became_peered":"2023-05-22T15:26:44.970872+0000","last_unstale":"2023-05-22T16:05:55.773610+0000","last_undegraded":"2023-05-22T16:05:55.773610+0000","last_fullsized":"2023-05-22T16:05:55.773610+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:55:34.752949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":188,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-05-22T16:05:55.833896+0000","last_change":"2023-05-22T15:26:44.977198+0000","last_active":"2023-05-22T16:05:55.833896+0000","last_peered":"2023-05-22T16:05:55.833896+0000","last_clean":"2023-05-22T16:05:55.833896+0000","last_became_active":"2023-05-22T15:26:44.976445+0000","last_became_peered":"2023-05-22T15:26:44.976445+0000","last_unstale":"2023-05-22T16:05:55.833896+0000","last_undegraded":"2023-05-22T16:05:55.833896+0000","last_fullsized":"2023-05-22T16:05:55.833896+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:12:41.464054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":193,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-05-22T16:05:58.459713+0000","last_change":"2023-05-22T15:26:44.969469+0000","last_active":"2023-05-22T16:05:58.459713+0000","last_peered":"2023-05-22T16:05:58.459713+0000","last_clean":"2023-05-22T16:05:58.459713+0000","last_became_active":"2023-05-22T15:26:44.969083+0000","last_became_peered":"2023-05-22T15:26:44.969083+0000","last_unstale":"2023-05-22T16:05:58.459713+0000","last_undegraded":"2023-05-22T16:05:58.459713+0000","last_fullsized":"2023-05-22T16:05:58.459713+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:29:23.670067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":199,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-05-22T16:05:55.774421+0000","last_change":"2023-05-22T15:26:44.976632+0000","last_active":"2023-05-22T16:05:55.774421+0000","last_peered":"2023-05-22T16:05:55.774421+0000","last_clean":"2023-05-22T16:05:55.774421+0000","last_became_active":"2023-05-22T15:26:44.975546+0000","last_became_peered":"2023-05-22T15:26:44.975546+0000","last_unstale":"2023-05-22T16:05:55.774421+0000","last_undegraded":"2023-05-22T16:05:55.774421+0000","last_fullsized":"2023-05-22T16:05:55.774421+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:42:57.973030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":178,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-05-22T16:05:58.461430+0000","last_change":"2023-05-22T15:26:45.091475+0000","last_active":"2023-05-22T16:05:58.461430+0000","last_peered":"2023-05-22T16:05:58.461430+0000","last_clean":"2023-05-22T16:05:58.461430+0000","last_became_active":"2023-05-22T15:26:45.091143+0000","last_became_peered":"2023-05-22T15:26:45.091143+0000","last_unstale":"2023-05-22T16:05:58.461430+0000","last_undegraded":"2023-05-22T16:05:58.461430+0000","last_fullsized":"2023-05-22T16:05:58.461430+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:12:41.464054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":187,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-05-22T16:05:55.834651+0000","last_change":"2023-05-22T15:26:45.092367+0000","last_active":"2023-05-22T16:05:55.834651+0000","last_peered":"2023-05-22T16:05:55.834651+0000","last_clean":"2023-05-22T16:05:55.834651+0000","last_became_active":"2023-05-22T15:26:45.091951+0000","last_became_peered":"2023-05-22T15:26:45.091951+0000","last_unstale":"2023-05-22T16:05:55.834651+0000","last_undegraded":"2023-05-22T16:05:55.834651+0000","last_fullsized":"2023-05-22T16:05:55.834651+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:42:57.973030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":178,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-05-22T16:05:58.461984+0000","last_change":"2023-05-22T15:26:44.978858+0000","last_active":"2023-05-22T16:05:58.461984+0000","last_peered":"2023-05-22T16:05:58.461984+0000","last_clean":"2023-05-22T16:05:58.461984+0000","last_became_active":"2023-05-22T15:26:44.978443+0000","last_became_peered":"2023-05-22T15:26:44.978443+0000","last_unstale":"2023-05-22T16:05:58.461984+0000","last_undegraded":"2023-05-22T16:05:58.461984+0000","last_fullsized":"2023-05-22T16:05:58.461984+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:46:29.290243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":176,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-05-22T16:05:55.915820+0000","last_change":"2023-05-22T15:26:44.982421+0000","last_active":"2023-05-22T16:05:55.915820+0000","last_peered":"2023-05-22T16:05:55.915820+0000","last_clean":"2023-05-22T16:05:55.915820+0000","last_became_active":"2023-05-22T15:26:44.981586+0000","last_became_peered":"2023-05-22T15:26:44.981586+0000","last_unstale":"2023-05-22T16:05:55.915820+0000","last_undegraded":"2023-05-22T16:05:55.915820+0000","last_fullsized":"2023-05-22T16:05:55.915820+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:14:11.156544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2132,"num_read_kb":4426,"num_write":579,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":348,"ondisk_log_size":348,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575314,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29912,"kb_used_data":3344,"kb_used_omap":0,"kb_used_meta":26560,"kb_avail":94341928,"statfs":{"total":96636764160,"available":96606134272,"internally_reserved":0,"allocated":3424256,"data_stored":2254794,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27197440},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 16:05:53 2023","interfaces":[{"interface":"back","average":{"1min":0.81999999999999995,"5min":0.85099999999999998,"15min":0.86099999999999999},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":1.222,"5min":1.3069999999999999,"15min":1.3069999999999999},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.84999999999999998,"5min":0.88200000000000001,"15min":0.877},"min":{"1min":0.61299999999999999,"5min":0.501,"15min":0.501},"max":{"1min":1.296,"5min":1.296,"15min":1.296},"last":1.036}]},{"osd":1,"last update":"Mon May 22 16:05:53 2023","interfaces":[{"interface":"back","average":{"1min":0.90400000000000003,"5min":0.97299999999999998,"15min":0.98999999999999999},"min":{"1min":0.66000000000000003,"5min":0.63400000000000001,"15min":0.58299999999999996},"max":{"1min":1.274,"5min":1.431,"15min":1.431},"last":1.129},{"interface":"front","average":{"1min":0.95499999999999996,"5min":1.026,"15min":1.026},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":1.2909999999999999,"5min":1.345,"15min":1.345},"last":1.216}]},{"osd":2,"last update":"Mon May 22 16:05:53 2023","interfaces":[{"interface":"back","average":{"1min":0.83899999999999997,"5min":0.89900000000000002,"15min":0.90400000000000003},"min":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.171,"5min":1.286,"15min":1.407},"last":0.73399999999999999},{"interface":"front","average":{"1min":0.89100000000000001,"5min":0.94899999999999995,"15min":0.95899999999999996},"min":{"1min":0.53200000000000003,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":1.3899999999999999,"5min":1.3899999999999999,"15min":1.3899999999999999},"last":0.90500000000000003}]},{"osd":3,"last update":"Mon May 22 16:05:53 2023","interfaces":[{"interface":"back","average":{"1min":0.96399999999999997,"5min":0.93300000000000005,"15min":0.94799999999999995},"min":{"1min":0.755,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.2,"5min":1.2909999999999999,"15min":1.401},"last":1.0700000000000001},{"interface":"front","average":{"1min":0.97199999999999998,"5min":0.98699999999999999,"15min":0.98099999999999998},"min":{"1min":0.60299999999999998,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":1.4179999999999999,"5min":1.4179999999999999,"15min":1.4179999999999999},"last":0.85199999999999998}]},{"osd":4,"last update":"Mon May 22 16:05:53 2023","interfaces":[{"interface":"back","average":{"1min":0.875,"5min":0.88900000000000001,"15min":0.88600000000000001},"min":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51000000000000001},"max":{"1min":1.2310000000000001,"5min":1.3440000000000001,"15min":1.3440000000000001},"last":0.77600000000000002},{"interface":"front","average":{"1min":0.95599999999999996,"5min":0.99099999999999999,"15min":0.96499999999999997},"min":{"1min":0.60899999999999999,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":1.3480000000000001,"5min":1.405,"15min":1.405},"last":1.165}]},{"osd":5,"last update":"Mon May 22 16:05:53 2023","interfaces":[{"interface":"back","average":{"1min":0.96499999999999997,"5min":0.98899999999999999,"15min":0.98999999999999999},"min":{"1min":0.66200000000000003,"5min":0.623,"15min":0.441},"max":{"1min":1.159,"5min":1.2909999999999999,"15min":1.3759999999999999},"last":0.95799999999999996},{"interface":"front","average":{"1min":1.1180000000000001,"5min":1.1339999999999999,"15min":1.135},"min":{"1min":0.91000000000000003,"5min":0.72299999999999998,"15min":0.66600000000000004},"max":{"1min":1.468,"5min":1.468,"15min":1.468},"last":1.194}]},{"osd":6,"last update":"Mon May 22 16:05:53 2023","interfaces":[{"interface":"back","average":{"1min":1.048,"5min":1.0600000000000001,"15min":1.0569999999999999},"min":{"1min":0.72799999999999998,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":1.288,"5min":1.3169999999999999,"15min":1.389},"last":1.105},{"interface":"front","average":{"1min":1.008,"5min":1.0940000000000001,"15min":1.0880000000000001},"min":{"1min":0.70399999999999996,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":1.3149999999999999,"5min":1.391,"15min":1.7050000000000001},"last":0.99399999999999999}]}]},{"osd":6,"up_from":13,"seq":55834575315,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":22668,"kb_used_data":2436,"kb_used_omap":0,"kb_used_meta":20224,"kb_avail":94349172,"statfs":{"total":96636764160,"available":96613552128,"internally_reserved":0,"allocated":2494464,"data_stored":1336191,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":20709376},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 16:05:56 2023","interfaces":[{"interface":"back","average":{"1min":0.85999999999999999,"5min":0.88,"15min":0.89600000000000002},"min":{"1min":0.48599999999999999,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":2.6469999999999998,"5min":2.6469999999999998,"15min":2.6469999999999998},"last":1.093},{"interface":"front","average":{"1min":1.254,"5min":0.96899999999999997,"15min":0.92800000000000005},"min":{"1min":0.64600000000000002,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":6.7439999999999998,"5min":6.7439999999999998,"15min":6.7439999999999998},"last":0.95399999999999996}]},{"osd":1,"last update":"Mon May 22 16:05:56 2023","interfaces":[{"interface":"back","average":{"1min":1.0009999999999999,"5min":0.93000000000000005,"15min":0.94199999999999995},"min":{"1min":0.57399999999999995,"5min":0.55300000000000005,"15min":0.51100000000000001},"max":{"1min":2.3290000000000002,"5min":2.3290000000000002,"15min":2.3290000000000002},"last":0.93200000000000005},{"interface":"front","average":{"1min":1.2330000000000001,"5min":0.97899999999999998,"15min":0.94699999999999995},"min":{"1min":0.58199999999999996,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":6.5010000000000003,"5min":6.5010000000000003,"15min":6.5010000000000003},"last":1.1839999999999999}]},{"osd":2,"last update":"Mon May 22 16:05:56 2023","interfaces":[{"interface":"back","average":{"1min":1.2210000000000001,"5min":1.012,"15min":1.0109999999999999},"min":{"1min":0.57399999999999995,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":7.0170000000000003,"5min":7.0170000000000003,"15min":7.0170000000000003},"last":0.98999999999999999},{"interface":"front","average":{"1min":1.0589999999999999,"5min":1.05,"15min":1.0720000000000001},"min":{"1min":0.66200000000000003,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":2.403,"5min":2.403,"15min":2.403},"last":1.02}]},{"osd":3,"last update":"Mon May 22 16:05:56 2023","interfaces":[{"interface":"back","average":{"1min":1.278,"5min":1.054,"15min":1.0129999999999999},"min":{"1min":0.63800000000000001,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":6.6699999999999999,"5min":6.6699999999999999,"15min":6.6699999999999999},"last":1.133},{"interface":"front","average":{"1min":1.244,"5min":1.02,"15min":0.98899999999999999},"min":{"1min":0.68000000000000005,"5min":0.622,"15min":0.622},"max":{"1min":7.1260000000000003,"5min":7.1260000000000003,"15min":7.1260000000000003},"last":0.90400000000000003}]},{"osd":4,"last update":"Mon May 22 16:05:56 2023","interfaces":[{"interface":"back","average":{"1min":1.161,"5min":0.98099999999999998,"15min":0.96999999999999997},"min":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"max":{"1min":4.0709999999999997,"5min":4.0709999999999997,"15min":4.0709999999999997},"last":0.73599999999999999},{"interface":"front","average":{"1min":1.583,"5min":1.1619999999999999,"15min":1.075},"min":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":11.581,"5min":11.581,"15min":11.581},"last":1.2330000000000001}]},{"osd":5,"last update":"Mon May 22 16:05:56 2023","interfaces":[{"interface":"back","average":{"1min":1.925,"5min":1.1739999999999999,"15min":1.056},"min":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":17.587,"5min":17.587,"15min":17.587},"last":0.70699999999999996},{"interface":"front","average":{"1min":1.204,"5min":1.1180000000000001,"15min":1.1240000000000001},"min":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"max":{"1min":2.9620000000000002,"5min":2.9620000000000002,"15min":2.9620000000000002},"last":0.83099999999999996}]},{"osd":7,"last update":"Mon May 22 16:05:56 2023","interfaces":[{"interface":"back","average":{"1min":1.4319999999999999,"5min":1.202,"15min":1.1559999999999999},"min":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.58599999999999997},"max":{"1min":6.8339999999999996,"5min":6.8339999999999996,"15min":6.8339999999999996},"last":1.0629999999999999},{"interface":"front","average":{"1min":1.5429999999999999,"5min":1.2090000000000001,"15min":1.163},"min":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":8.5990000000000002,"5min":8.5990000000000002,"15min":8.5990000000000002},"last":0.78600000000000003}]}]},{"osd":5,"up_from":13,"seq":55834575315,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32780,"kb_used_data":2436,"kb_used_omap":0,"kb_used_meta":30336,"kb_avail":94339060,"statfs":{"total":96636764160,"available":96603197440,"internally_reserved":0,"allocated":2494464,"data_stored":1336191,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31064064},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 16:05:54 2023","interfaces":[{"interface":"back","average":{"1min":0.76000000000000001,"5min":0.85199999999999998,"15min":0.875},"min":{"1min":0.47899999999999998,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.0820000000000001,"5min":1.2749999999999999,"15min":1.3959999999999999},"last":0.98599999999999999},{"interface":"front","average":{"1min":0.86499999999999999,"5min":0.93999999999999995,"15min":0.94299999999999995},"min":{"1min":0.56499999999999995,"5min":0.56200000000000006,"15min":0.502},"max":{"1min":1.165,"5min":1.325,"15min":1.325},"last":1.0860000000000001}]},{"osd":1,"last update":"Mon May 22 16:05:54 2023","interfaces":[{"interface":"back","average":{"1min":0.86599999999999999,"5min":0.95999999999999996,"15min":1.0009999999999999},"min":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.2909999999999999,"5min":1.365,"15min":1.365},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.89600000000000002,"5min":1.028,"15min":1.0640000000000001},"min":{"1min":0.46899999999999997,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":1.216,"5min":1.3520000000000001,"15min":1.4630000000000001},"last":1.0569999999999999}]},{"osd":2,"last update":"Mon May 22 16:05:54 2023","interfaces":[{"interface":"back","average":{"1min":0.85299999999999998,"5min":0.90500000000000003,"15min":0.92000000000000004},"min":{"1min":0.60899999999999999,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.1859999999999999,"5min":1.3200000000000001,"15min":1.3740000000000001},"last":1.0149999999999999},{"interface":"front","average":{"1min":0.86799999999999999,"5min":0.94699999999999995,"15min":0.94599999999999995},"min":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":1.117,"5min":1.4350000000000001,"15min":1.4350000000000001},"last":0.92700000000000005}]},{"osd":3,"last update":"Mon May 22 16:05:54 2023","interfaces":[{"interface":"back","average":{"1min":0.91300000000000003,"5min":0.97699999999999998,"15min":0.97599999999999998},"min":{"1min":0.63400000000000001,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":1.1759999999999999,"5min":1.349,"15min":1.367},"last":0.83699999999999997},{"interface":"front","average":{"1min":0.97999999999999998,"5min":1.0329999999999999,"15min":1.048},"min":{"1min":0.67600000000000005,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.1779999999999999,"5min":1.3899999999999999,"15min":1.3899999999999999},"last":0.85599999999999998}]},{"osd":4,"last update":"Mon May 22 16:05:54 2023","interfaces":[{"interface":"back","average":{"1min":0.91200000000000003,"5min":0.93600000000000005,"15min":0.93100000000000005},"min":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.2170000000000001,"5min":1.6000000000000001,"15min":1.6000000000000001},"last":0.79900000000000004},{"interface":"front","average":{"1min":0.98899999999999999,"5min":0.98499999999999999,"15min":1.01},"min":{"1min":0.57899999999999996,"5min":0.498,"15min":0.498},"max":{"1min":1.371,"5min":1.4630000000000001,"15min":1.4630000000000001},"last":0.89900000000000002}]},{"osd":6,"last update":"Mon May 22 16:05:54 2023","interfaces":[{"interface":"back","average":{"1min":1.054,"5min":1.0129999999999999,"15min":1.0209999999999999},"min":{"1min":0.68999999999999995,"5min":0.63200000000000001,"15min":0.59799999999999998},"max":{"1min":1.2509999999999999,"5min":1.4630000000000001,"15min":1.4790000000000001},"last":1.147},{"interface":"front","average":{"1min":1.0449999999999999,"5min":1.073,"15min":1.081},"min":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":1.3440000000000001,"5min":1.6080000000000001,"15min":1.6080000000000001},"last":1.0349999999999999}]},{"osd":7,"last update":"Mon May 22 16:05:54 2023","interfaces":[{"interface":"back","average":{"1min":1.133,"5min":1.1180000000000001,"15min":1.1259999999999999},"min":{"1min":0.83899999999999997,"5min":0.57699999999999996,"15min":0.57699999999999996},"max":{"1min":1.3680000000000001,"5min":1.51,"15min":1.5109999999999999},"last":0.77300000000000002},{"interface":"front","average":{"1min":1.1739999999999999,"5min":1.1859999999999999,"15min":1.1930000000000001},"min":{"1min":0.91000000000000003,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":1.353,"5min":1.6679999999999999,"15min":1.7210000000000001},"last":0.95299999999999996}]}]},{"osd":4,"up_from":13,"seq":55834575313,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39584,"kb_used_data":2456,"kb_used_omap":0,"kb_used_meta":37120,"kb_avail":94332256,"statfs":{"total":96636764160,"available":96596230144,"internally_reserved":0,"allocated":2514944,"data_stored":1349186,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":38010880},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 16:05:02 2023","interfaces":[{"interface":"back","average":{"1min":0.90400000000000003,"5min":0.89600000000000002,"15min":0.90000000000000002},"min":{"1min":0.59999999999999998,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.1870000000000001,"5min":1.7210000000000001,"15min":1.7210000000000001},"last":0.72299999999999998},{"interface":"front","average":{"1min":0.995,"5min":0.96599999999999997,"15min":0.95199999999999996},"min":{"1min":0.70899999999999996,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":1.3360000000000001,"5min":1.7290000000000001,"15min":1.7290000000000001},"last":0.57099999999999995}]},{"osd":1,"last update":"Mon May 22 16:05:02 2023","interfaces":[{"interface":"back","average":{"1min":1.0269999999999999,"5min":0.97599999999999998,"15min":0.95899999999999996},"min":{"1min":0.76900000000000002,"5min":0.314,"15min":0.314},"max":{"1min":1.258,"5min":1.9339999999999999,"15min":1.9339999999999999},"last":0.96299999999999997},{"interface":"front","average":{"1min":0.90000000000000002,"5min":0.92800000000000005,"15min":0.93999999999999995},"min":{"1min":0.68799999999999994,"5min":0.504,"15min":0.504},"max":{"1min":1.0900000000000001,"5min":1.365,"15min":1.365},"last":0.79400000000000004}]},{"osd":2,"last update":"Mon May 22 16:05:02 2023","interfaces":[{"interface":"back","average":{"1min":0.98599999999999999,"5min":0.95499999999999996,"15min":0.94999999999999996},"min":{"1min":0.71799999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":1.28,"5min":1.4990000000000001,"15min":1.4990000000000001},"last":1.0129999999999999},{"interface":"front","average":{"1min":1.056,"5min":1.036,"15min":1.052},"min":{"1min":0.76000000000000001,"5min":0.44,"15min":0.44},"max":{"1min":1.2529999999999999,"5min":2.0249999999999999,"15min":2.0249999999999999},"last":0.84899999999999998}]},{"osd":3,"last update":"Mon May 22 16:05:02 2023","interfaces":[{"interface":"back","average":{"1min":1.0329999999999999,"5min":0.96499999999999997,"15min":0.96599999999999997},"min":{"1min":0.79200000000000004,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":1.258,"5min":1.4610000000000001,"15min":1.4610000000000001},"last":0.61699999999999999},{"interface":"front","average":{"1min":1.0760000000000001,"5min":1.0609999999999999,"15min":1.0489999999999999},"min":{"1min":0.83899999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":1.323,"5min":1.349,"15min":1.484},"last":1.125}]},{"osd":5,"last update":"Mon May 22 16:05:02 2023","interfaces":[{"interface":"back","average":{"1min":1.004,"5min":1.036,"15min":1.002},"min":{"1min":0.69599999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":1.325,"5min":2.1110000000000002,"15min":2.1110000000000002},"last":1.0880000000000001},{"interface":"front","average":{"1min":1.0309999999999999,"5min":1.0189999999999999,"15min":0.996},"min":{"1min":0.69999999999999996,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.1990000000000001,"5min":1.5449999999999999,"15min":1.5449999999999999},"last":1.0620000000000001}]},{"osd":6,"last update":"Mon May 22 16:05:02 2023","interfaces":[{"interface":"back","average":{"1min":1.0669999999999999,"5min":1.036,"15min":1.0009999999999999},"min":{"1min":0.75800000000000001,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":1.387,"5min":1.397,"15min":1.4339999999999999},"last":1.2},{"interface":"front","average":{"1min":1.1599999999999999,"5min":1.1599999999999999,"15min":1.137},"min":{"1min":0.875,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.3340000000000001,"5min":2.2690000000000001,"15min":2.2690000000000001},"last":1.1639999999999999}]},{"osd":7,"last update":"Mon May 22 16:05:02 2023","interfaces":[{"interface":"back","average":{"1min":1.093,"5min":1.0960000000000001,"15min":1.069},"min":{"1min":0.83599999999999997,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":1.3029999999999999,"5min":1.623,"15min":1.623},"last":0.91300000000000003},{"interface":"front","average":{"1min":1.2150000000000001,"5min":1.1879999999999999,"15min":1.1599999999999999},"min":{"1min":1.0169999999999999,"5min":0.71199999999999997,"15min":0.68100000000000005},"max":{"1min":1.4390000000000001,"5min":1.448,"15min":1.5089999999999999},"last":1.2529999999999999}]}]},{"osd":3,"up_from":13,"seq":55834575313,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27928,"kb_used_data":2768,"kb_used_omap":0,"kb_used_meta":25152,"kb_avail":94343912,"statfs":{"total":96636764160,"available":96608165888,"internally_reserved":0,"allocated":2834432,"data_stored":1664516,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":25755648},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 16:05:38 2023","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"min":{"1min":0.29299999999999998,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.93700000000000006,"5min":1.0049999999999999,"15min":1.0229999999999999},"last":0.81899999999999995},{"interface":"front","average":{"1min":0.70299999999999996,"5min":0.76200000000000001,"15min":0.76900000000000002},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":1.2749999999999999,"5min":1.7230000000000001,"15min":1.7230000000000001},"last":0.45900000000000002}]},{"osd":1,"last update":"Mon May 22 16:05:38 2023","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.75600000000000001,"15min":0.71999999999999997},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":1.2190000000000001,"5min":2.4969999999999999,"15min":2.4969999999999999},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.76000000000000001,"5min":0.80700000000000005,"15min":0.80100000000000005},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":1.3520000000000001,"5min":2.4769999999999999,"15min":2.4769999999999999},"last":0.66200000000000003}]},{"osd":2,"last update":"Mon May 22 16:05:38 2023","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":0.83699999999999997,"15min":0.84899999999999998},"min":{"1min":0.52300000000000002,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.94799999999999995,"5min":2.2250000000000001,"15min":2.2250000000000001},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.87,"5min":0.83599999999999997,"15min":0.83299999999999996},"min":{"1min":0.55500000000000005,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":1.129,"5min":2.1680000000000001,"15min":2.1680000000000001},"last":0.76400000000000001}]},{"osd":4,"last update":"Mon May 22 16:05:38 2023","interfaces":[{"interface":"back","average":{"1min":0.95199999999999996,"5min":0.98099999999999998,"15min":0.97399999999999998},"min":{"1min":0.36799999999999999,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":1.357,"5min":2.3479999999999999,"15min":2.3479999999999999},"last":1.0009999999999999},{"interface":"front","average":{"1min":1.0009999999999999,"5min":1.022,"15min":1.03},"min":{"1min":0.39400000000000002,"5min":0.37,"15min":0.37},"max":{"1min":1.2210000000000001,"5min":1.788,"15min":1.788},"last":1.135}]},{"osd":5,"last update":"Mon May 22 16:05:38 2023","interfaces":[{"interface":"back","average":{"1min":1.0069999999999999,"5min":1.0409999999999999,"15min":1.05},"min":{"1min":0.496,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":1.21,"5min":1.8129999999999999,"15min":1.8129999999999999},"last":1.0409999999999999},{"interface":"front","average":{"1min":0.98299999999999998,"5min":1.0169999999999999,"15min":1.0369999999999999},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.3140000000000001,"5min":1.571,"15min":1.571},"last":0.86599999999999999}]},{"osd":6,"last update":"Mon May 22 16:05:38 2023","interfaces":[{"interface":"back","average":{"1min":1.0109999999999999,"5min":1.0680000000000001,"15min":1.0880000000000001},"min":{"1min":0.45600000000000002,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":1.274,"5min":1.76,"15min":1.76},"last":1.2090000000000001},{"interface":"front","average":{"1min":1.0609999999999999,"5min":1.0820000000000001,"15min":1.099},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":1.4139999999999999,"5min":1.754,"15min":1.754},"last":0.91000000000000003}]},{"osd":7,"last update":"Mon May 22 16:05:38 2023","interfaces":[{"interface":"back","average":{"1min":1.1020000000000001,"5min":1.111,"15min":1.127},"min":{"1min":0.53800000000000003,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.5049999999999999,"5min":1.673,"15min":1.673},"last":1.081},{"interface":"front","average":{"1min":1.1040000000000001,"5min":1.1160000000000001,"15min":1.109},"min":{"1min":0.41099999999999998,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":1.4670000000000001,"5min":2.3079999999999998,"15min":2.3079999999999998},"last":0.88}]}]},{"osd":2,"up_from":13,"seq":55834575314,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31316,"kb_used_data":2444,"kb_used_omap":0,"kb_used_meta":28864,"kb_avail":94340524,"statfs":{"total":96636764160,"available":96604696576,"internally_reserved":0,"allocated":2502656,"data_stored":1336768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":29556736},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 16:05:35 2023","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.629,"15min":0.67000000000000004},"min":{"1min":0.28599999999999998,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.90100000000000002,"5min":1.0549999999999999,"15min":1.1519999999999999},"last":0.78400000000000003},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.71799999999999997,"15min":0.74199999999999999},"min":{"1min":0.35299999999999998,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":1.0309999999999999,"5min":1.1659999999999999,"15min":1.2090000000000001},"last":0.66100000000000003}]},{"osd":1,"last update":"Mon May 22 16:05:35 2023","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.74399999999999999,"15min":0.75700000000000001},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.875,"5min":1.071,"15min":1.099},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.79300000000000004,"5min":0.82099999999999995,"15min":0.82599999999999996},"min":{"1min":0.39500000000000002,"5min":0.311,"15min":0.311},"max":{"1min":1.069,"5min":1.1739999999999999,"15min":1.2589999999999999},"last":0.875}]},{"osd":3,"last update":"Mon May 22 16:05:35 2023","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":0.75600000000000001,"15min":0.75800000000000001},"min":{"1min":0.32100000000000001,"5min":0.315,"15min":0.315},"max":{"1min":0.91400000000000003,"5min":1.3540000000000001,"15min":1.3540000000000001},"last":0.83899999999999997},{"interface":"front","average":{"1min":0.85199999999999998,"5min":0.87,"15min":0.89300000000000002},"min":{"1min":0.41399999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":1.1859999999999999,"5min":1.2949999999999999,"15min":1.2949999999999999},"last":0.95099999999999996}]},{"osd":4,"last update":"Mon May 22 16:05:35 2023","interfaces":[{"interface":"back","average":{"1min":1.02,"5min":1.024,"15min":1.04},"min":{"1min":0.66100000000000003,"5min":0.441,"15min":0.441},"max":{"1min":1.4590000000000001,"5min":1.516,"15min":1.6539999999999999},"last":1.0249999999999999},{"interface":"front","average":{"1min":0.996,"5min":1.0089999999999999,"15min":1.04},"min":{"1min":0.55600000000000005,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.419,"5min":1.4299999999999999,"15min":1.538},"last":1.1060000000000001}]},{"osd":5,"last update":"Mon May 22 16:05:35 2023","interfaces":[{"interface":"back","average":{"1min":1.03,"5min":1.024,"15min":1.0549999999999999},"min":{"1min":0.53600000000000003,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":1.4139999999999999,"5min":1.4139999999999999,"15min":1.605},"last":1.2350000000000001},{"interface":"front","average":{"1min":1.0069999999999999,"5min":1.038,"15min":1.03},"min":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":1.339,"5min":1.534,"15min":1.534},"last":0.92200000000000004}]},{"osd":6,"last update":"Mon May 22 16:05:35 2023","interfaces":[{"interface":"back","average":{"1min":1.095,"5min":1.0589999999999999,"15min":1.0349999999999999},"min":{"1min":0.72699999999999998,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":1.4259999999999999,"5min":1.429,"15min":1.462},"last":0.99399999999999999},{"interface":"front","average":{"1min":1.127,"5min":1.105,"15min":1.1339999999999999},"min":{"1min":0.59199999999999997,"5min":0.57099999999999995,"15min":0.57099999999999995},"max":{"1min":1.5269999999999999,"5min":1.621,"15min":1.7110000000000001},"last":1.0760000000000001}]},{"osd":7,"last update":"Mon May 22 16:05:35 2023","interfaces":[{"interface":"back","average":{"1min":1.0940000000000001,"5min":1.0860000000000001,"15min":1.1220000000000001},"min":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.5680000000000001,"5min":1.5680000000000001,"15min":1.6399999999999999},"last":1.179},{"interface":"front","average":{"1min":1.0940000000000001,"5min":1.107,"15min":1.1140000000000001},"min":{"1min":0.629,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":1.4610000000000001,"5min":1.673,"15min":1.673},"last":1.0389999999999999}]}]},{"osd":1,"up_from":13,"seq":55834575314,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37724,"kb_used_data":2772,"kb_used_omap":0,"kb_used_meta":34944,"kb_avail":94334116,"statfs":{"total":96636764160,"available":96598134784,"internally_reserved":0,"allocated":2838528,"data_stored":1676300,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":35782656},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 16:05:58 2023","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.64800000000000002,"15min":0.66200000000000003},"min":{"1min":0.34499999999999997,"5min":0.308,"15min":0.308},"max":{"1min":1.0449999999999999,"5min":1.0449999999999999,"15min":1.329},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.72199999999999998,"15min":0.76300000000000001},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.95499999999999996,"5min":1.25,"15min":1.268},"last":0.85099999999999998}]},{"osd":2,"last update":"Mon May 22 16:05:58 2023","interfaces":[{"interface":"back","average":{"1min":0.70999999999999996,"5min":0.73399999999999999,"15min":0.75600000000000001},"min":{"1min":0.372,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":1.141,"5min":1.141,"15min":1.26},"last":0.96099999999999997},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.76300000000000001,"15min":0.77700000000000002},"min":{"1min":0.46999999999999997,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":1.0249999999999999,"5min":1.127,"15min":1.268},"last":0.78900000000000003}]},{"osd":3,"last update":"Mon May 22 16:05:58 2023","interfaces":[{"interface":"back","average":{"1min":0.78300000000000003,"5min":0.83799999999999997,"15min":0.878},"min":{"1min":0.39900000000000002,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":1.1100000000000001,"5min":1.2829999999999999,"15min":1.2829999999999999},"last":1.1100000000000001},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.80300000000000005,"15min":0.81899999999999995},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":1.2050000000000001,"5min":1.2749999999999999,"15min":1.2749999999999999},"last":0.38500000000000001}]},{"osd":4,"last update":"Mon May 22 16:05:58 2023","interfaces":[{"interface":"back","average":{"1min":0.93899999999999995,"5min":0.96599999999999997,"15min":0.98399999999999999},"min":{"1min":0.58499999999999996,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":1.3500000000000001,"5min":1.542,"15min":1.542},"last":1.046},{"interface":"front","average":{"1min":1.006,"5min":0.997,"15min":1.024},"min":{"1min":0.71499999999999997,"5min":0.434,"15min":0.434},"max":{"1min":1.2729999999999999,"5min":1.327,"15min":1.4650000000000001},"last":1.2390000000000001}]},{"osd":5,"last update":"Mon May 22 16:05:58 2023","interfaces":[{"interface":"back","average":{"1min":1.0309999999999999,"5min":1.0349999999999999,"15min":1.048},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.319,"5min":1.605,"15min":1.605},"last":0.997},{"interface":"front","average":{"1min":1.0589999999999999,"5min":1.0620000000000001,"15min":1.0760000000000001},"min":{"1min":0.75900000000000001,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":1.4370000000000001,"5min":1.4370000000000001,"15min":1.5389999999999999},"last":1.151}]},{"osd":6,"last update":"Mon May 22 16:05:58 2023","interfaces":[{"interface":"back","average":{"1min":1.03,"5min":1.0469999999999999,"15min":1.0640000000000001},"min":{"1min":0.56899999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"max":{"1min":1.4770000000000001,"5min":1.4770000000000001,"15min":1.5249999999999999},"last":1.3580000000000001},{"interface":"front","average":{"1min":1.0589999999999999,"5min":1.099,"15min":1.1100000000000001},"min":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"max":{"1min":1.387,"5min":1.4359999999999999,"15min":1.6479999999999999},"last":1.1890000000000001}]},{"osd":7,"last update":"Mon May 22 16:05:58 2023","interfaces":[{"interface":"back","average":{"1min":1.0509999999999999,"5min":1.123,"15min":1.153},"min":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":1.484,"5min":1.7030000000000001,"15min":1.7030000000000001},"last":1.2829999999999999},{"interface":"front","average":{"1min":1.1339999999999999,"5min":1.131,"15min":1.151},"min":{"1min":0.78400000000000003,"5min":0.73599999999999999,"15min":0.73599999999999999},"max":{"1min":1.4810000000000001,"5min":1.6439999999999999,"15min":1.6439999999999999},"last":1.319}]}]},{"osd":0,"up_from":13,"seq":55834575315,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37264,"kb_used_data":3336,"kb_used_omap":0,"kb_used_meta":33920,"kb_avail":94334576,"statfs":{"total":96636764160,"available":96598605824,"internally_reserved":0,"allocated":3416064,"data_stored":2254160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":34734080},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon May 22 16:06:01 2023","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.72399999999999998,"15min":0.71099999999999997},"min":{"1min":0.42299999999999999,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":1.1140000000000001,"5min":1.296,"15min":1.296},"last":0.73799999999999999},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.72999999999999998,"15min":0.73899999999999999},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.86899999999999999,"5min":1.181,"15min":1.3149999999999999},"last":0.60699999999999998}]},{"osd":2,"last update":"Mon May 22 16:06:01 2023","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.751,"15min":0.76900000000000002},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":1.071,"5min":1.22,"15min":1.284},"last":0.877},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.78700000000000003,"15min":0.81299999999999994},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":1.042,"5min":1.323,"15min":1.323},"last":0.70399999999999996}]},{"osd":3,"last update":"Mon May 22 16:06:01 2023","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.746,"15min":0.77100000000000002},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":1.228,"5min":1.2350000000000001,"15min":1.2350000000000001},"last":0.83099999999999996},{"interface":"front","average":{"1min":0.78300000000000003,"5min":0.85899999999999999,"15min":0.874},"min":{"1min":0.43099999999999999,"5min":0.377,"15min":0.377},"max":{"1min":1.22,"5min":1.2949999999999999,"15min":1.331},"last":1.0600000000000001}]},{"osd":4,"last update":"Mon May 22 16:06:01 2023","interfaces":[{"interface":"back","average":{"1min":0.90200000000000002,"5min":0.94599999999999995,"15min":0.95199999999999996},"min":{"1min":0.64100000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"max":{"1min":1.1819999999999999,"5min":1.264,"15min":1.4059999999999999},"last":1.085},{"interface":"front","average":{"1min":0.91500000000000004,"5min":1.042,"15min":1.05},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.1619999999999999,"5min":1.401,"15min":1.482},"last":0.93999999999999995}]},{"osd":5,"last update":"Mon May 22 16:06:01 2023","interfaces":[{"interface":"back","average":{"1min":0.92600000000000005,"5min":1.014,"15min":1.0349999999999999},"min":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":1.1830000000000001,"5min":1.325,"15min":1.6419999999999999},"last":1.016},{"interface":"front","average":{"1min":0.90200000000000002,"5min":1.0009999999999999,"15min":1.0109999999999999},"min":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"max":{"1min":1.1859999999999999,"5min":1.413,"15min":1.458},"last":0.96899999999999997}]},{"osd":6,"last update":"Mon May 22 16:06:01 2023","interfaces":[{"interface":"back","average":{"1min":0.98399999999999999,"5min":1.091,"15min":1.1060000000000001},"min":{"1min":0.66400000000000003,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":1.2729999999999999,"5min":1.407,"15min":1.77},"last":1.1479999999999999},{"interface":"front","average":{"1min":0.96399999999999997,"5min":1.093,"15min":1.1040000000000001},"min":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"max":{"1min":1.2589999999999999,"5min":1.5409999999999999,"15min":1.5409999999999999},"last":1.2070000000000001}]},{"osd":7,"last update":"Mon May 22 16:06:01 2023","interfaces":[{"interface":"back","average":{"1min":1.034,"5min":1.1279999999999999,"15min":1.1359999999999999},"min":{"1min":0.69299999999999995,"5min":0.63900000000000001,"15min":0.63900000000000001},"max":{"1min":1.341,"5min":1.6020000000000001,"15min":1.6020000000000001},"last":1.1259999999999999},{"interface":"front","average":{"1min":1.0109999999999999,"5min":1.0780000000000001,"15min":1.095},"min":{"1min":0.68999999999999995,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.2989999999999999,"5min":1.4650000000000001,"15min":1.4990000000000001},"last":1.173}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-05-22T16:06:06.143 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-05-22T16:06:06.272 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:06.267+0000 7f715851e640 1 Processor -- start 2023-05-22T16:06:06.273 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:06.267+0000 7f715851e640 1 -- start start 2023-05-22T16:06:06.274 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:06.271+0000 7f715851e640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f715006c8e0 0x7f715015cf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:06.274 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:06.271+0000 7f715851e640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f7150158be0 0x7f715006ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:06.274 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:06.271+0000 7f715851e640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f715015d480 con 0x7f71500c8840 2023-05-22T16:06:06.274 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:06.271+0000 7f715851e640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f715006d1f0 con 0x7f7150158be0 2023-05-22T16:06:06.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:06.271+0000 7f715851e640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71500c8e80 con 0x7f715006c8e0 2023-05-22T16:06:06.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:06.271+0000 7f7155ab9640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f7150158be0 0x7f715006ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:06.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:06.271+0000 7f7155ab9640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f7150158be0 0x7f715006ccb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43514/0 (socket says 172.21.15.16:43514) 2023-05-22T16:06:06.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:06.271+0000 7f7155ab9640 1 -- 172.21.15.16:0/2482118355 learned_addr learned my addr 172.21.15.16:0/2482118355 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:06:06.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:06.271+0000 7f71562ba640 1 --2- 172.21.15.16:0/2482118355 >> v2:172.21.15.62:3300/0 conn(0x7f715006c8e0 0x7f715015cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:06.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:06.271+0000 7f7155ab9640 1 -- 172.21.15.16:0/2482118355 >> v2:172.21.15.62:3300/0 conn(0x7f715006c8e0 msgr2=0x7f715015cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:06:06.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:06.271+0000 7f7155ab9640 1 --2- 172.21.15.16:0/2482118355 >> v2:172.21.15.62:3300/0 conn(0x7f715006c8e0 0x7f715015cf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:06.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:06.271+0000 7f7155ab9640 1 -- 172.21.15.16:0/2482118355 >> v1:172.21.15.62:6789/0 conn(0x7f71500c8840 legacy=0x7f71500afb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:06:06.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:06.271+0000 7f7155ab9640 1 -- 172.21.15.16:0/2482118355 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71500c9160 con 0x7f7150158be0 2023-05-22T16:06:06.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:06.271+0000 7f7155ab9640 1 --2- 172.21.15.16:0/2482118355 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7150158be0 0x7f715006ccb0 secure :-1 s=READY pgs=2622 cs=0 l=1 rev1=1 crypto rx=0x7f71500cb910 tx=0x7f714c01ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d54b734ec3d42d1d server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:06:06.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:06.271+0000 7f71552b8640 1 -- 172.21.15.16:0/2482118355 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f714c0a7aa0 con 0x7f7150158be0 2023-05-22T16:06:06.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:06.275+0000 7f71552b8640 1 -- 172.21.15.16:0/2482118355 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f714c0a7c00 con 0x7f7150158be0 2023-05-22T16:06:06.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:06.275+0000 7f71552b8640 1 -- 172.21.15.16:0/2482118355 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f714c0af440 con 0x7f7150158be0 2023-05-22T16:06:06.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:06.275+0000 7f715851e640 1 -- 172.21.15.16:0/2482118355 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7150158be0 msgr2=0x7f715006ccb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:06:06.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:06.275+0000 7f715851e640 1 --2- 172.21.15.16:0/2482118355 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7150158be0 0x7f715006ccb0 secure :-1 s=READY pgs=2622 cs=0 l=1 rev1=1 crypto rx=0x7f71500cb910 tx=0x7f714c01ebd0 comp rx=0 tx=0).stop 2023-05-22T16:06:06.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:06.275+0000 7f715851e640 1 -- 172.21.15.16:0/2482118355 shutdown_connections 2023-05-22T16:06:06.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:06.275+0000 7f715851e640 1 --2- 172.21.15.16:0/2482118355 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7150158be0 0x7f715006ccb0 unknown :-1 s=CLOSED pgs=2622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:06.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:06.275+0000 7f715851e640 1 --2- 172.21.15.16:0/2482118355 >> v2:172.21.15.62:3300/0 conn(0x7f715006c8e0 0x7f715015cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:06.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:06.275+0000 7f715851e640 1 -- 172.21.15.16:0/2482118355 >> 172.21.15.16:0/2482118355 conn(0x7f71500cab50 msgr2=0x7f71500caf50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:06:06.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:06.275+0000 7f715851e640 1 -- 172.21.15.16:0/2482118355 shutdown_connections 2023-05-22T16:06:06.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:06.275+0000 7f715851e640 1 -- 172.21.15.16:0/2482118355 wait complete. 2023-05-22T16:06:06.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:06.275+0000 7f715851e640 1 Processor -- start 2023-05-22T16:06:06.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:06.275+0000 7f715851e640 1 -- start start 2023-05-22T16:06:06.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:06.275+0000 7f715851e640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f715006c8e0 0x7f715014c4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:06.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:06.275+0000 7f715851e640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f71500c8840 0x7f71500b2df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:06.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:06.275+0000 7f715851e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7150158be0 0x7f71500c1c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:06.282 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:06.275+0000 7f715851e640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f715014ca10 con 0x7f7150158be0 2023-05-22T16:06:06.282 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:06.275+0000 7f715851e640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71500b1a40 con 0x7f71500c8840 2023-05-22T16:06:06.282 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:06.275+0000 7f715851e640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71500b1d20 con 0x7f715006c8e0 2023-05-22T16:06:06.282 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:06.275+0000 7f7156abb640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7150158be0 0x7f71500c1c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:06.282 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:06.275+0000 7f7156abb640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7150158be0 0x7f71500c1c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43528/0 (socket says 172.21.15.16:43528) 2023-05-22T16:06:06.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:06.275+0000 7f7156abb640 1 -- 172.21.15.16:0/2719349328 learned_addr learned my addr 172.21.15.16:0/2719349328 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:06:06.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:06.275+0000 7f7155ab9640 1 --2- 172.21.15.16:0/2719349328 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f71500c8840 0x7f71500b2df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:06.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:06.275+0000 7f71562ba640 1 --2- 172.21.15.16:0/2719349328 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f715006c8e0 0x7f715014c4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:06.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:06.275+0000 7f7156abb640 1 -- 172.21.15.16:0/2719349328 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f715006c8e0 msgr2=0x7f715014c4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:06:06.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:06.275+0000 7f7156abb640 1 --2- 172.21.15.16:0/2719349328 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f715006c8e0 0x7f715014c4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:06.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:06.275+0000 7f7156abb640 1 -- 172.21.15.16:0/2719349328 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f71500c8840 msgr2=0x7f71500b2df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:06:06.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:06.275+0000 7f7156abb640 1 --2- 172.21.15.16:0/2719349328 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f71500c8840 0x7f71500b2df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:06.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:06.275+0000 7f7156abb640 1 -- 172.21.15.16:0/2719349328 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f714c00b770 con 0x7f7150158be0 2023-05-22T16:06:06.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:06.275+0000 7f7155ab9640 1 --2- 172.21.15.16:0/2719349328 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f71500c8840 0x7f71500b2df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T16:06:06.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:06.275+0000 7f71562ba640 1 --2- 172.21.15.16:0/2719349328 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f715006c8e0 0x7f715014c4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T16:06:06.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:06.275+0000 7f7156abb640 1 --2- 172.21.15.16:0/2719349328 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7150158be0 0x7f71500c1c00 secure :-1 s=READY pgs=2623 cs=0 l=1 rev1=1 crypto rx=0x7f714800c710 tx=0x7f714800cbd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:06:06.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:06.275+0000 7f7142ffd640 1 -- 172.21.15.16:0/2719349328 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7148006d60 con 0x7f7150158be0 2023-05-22T16:06:06.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:06.275+0000 7f7142ffd640 1 -- 172.21.15.16:0/2719349328 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f714800ce20 con 0x7f7150158be0 2023-05-22T16:06:06.285 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:06.275+0000 7f715851e640 1 -- 172.21.15.16:0/2719349328 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f715006b7d0 con 0x7f7150158be0 2023-05-22T16:06:06.285 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:06.275+0000 7f7142ffd640 1 -- 172.21.15.16:0/2719349328 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7148003bc0 con 0x7f7150158be0 2023-05-22T16:06:06.285 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:06.275+0000 7f715851e640 1 -- 172.21.15.16:0/2719349328 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71500c9a00 con 0x7f7150158be0 2023-05-22T16:06:06.285 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:06.279+0000 7f715851e640 1 -- 172.21.15.16:0/2719349328 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f715006b960 con 0x7f7150158be0 2023-05-22T16:06:06.287 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:06.279+0000 7f7142ffd640 1 -- 172.21.15.16:0/2719349328 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 100) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f7148003d20 con 0x7f7150158be0 2023-05-22T16:06:06.287 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:06.279+0000 7f7142ffd640 1 --2- 172.21.15.16:0/2719349328 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f71340654a0 0x7f7134067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:06.287 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:06.279+0000 7f7142ffd640 1 -- 172.21.15.16:0/2719349328 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(201..201 src has 1..201) v4 ==== 6317+0+0 (secure 0 0 0) 0x7f7148082660 con 0x7f7150158be0 2023-05-22T16:06:06.288 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:06.279+0000 7f71562ba640 1 --2- 172.21.15.16:0/2719349328 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f71340654a0 0x7f7134067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:06.288 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:06.279+0000 7f71562ba640 1 --2- 172.21.15.16:0/2719349328 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f71340654a0 0x7f7134067950 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f71501f3c10 tx=0x7f7144008040 comp rx=0 tx=0).ready entity=mgr.8195 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:06:06.288 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:06.283+0000 7f7142ffd640 1 -- 172.21.15.16:0/2719349328 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f7148053b90 con 0x7f7150158be0 2023-05-22T16:06:06.443 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:06.439+0000 7f715851e640 1 -- 172.21.15.16:0/2719349328 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f71500c4fb0 con 0x7f7150158be0 2023-05-22T16:06:06.445 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:06.443+0000 7f7142ffd640 1 -- 172.21.15.16:0/2719349328 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v201) v1 ==== 74+0+15480 (secure 0 0 0) 0x7f7148053780 con 0x7f7150158be0 2023-05-22T16:06:06.446 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T16:06:06.446 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":201,"fsid":"b2719834-7847-4d98-90a6-f65d57f85c49","created":"2023-05-22T15:26:25.259314+0000","modified":"2023-05-22T16:06:02.330407+0000","last_up_change":"2023-05-22T15:26:36.448524+0000","last_in_change":"2023-05-22T15:26:27.675668+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":25,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-05-22T15:26:38.019781+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-05-22T15:26:43.570514+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9e7ff854-b25f-4cbb-a5c2-1799d9c53d2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":37021},{"type":"v1","addr":"172.21.15.16:6825","nonce":37021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":37021},{"type":"v1","addr":"172.21.15.16:6827","nonce":37021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":37021},{"type":"v1","addr":"172.21.15.16:6831","nonce":37021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":37021},{"type":"v1","addr":"172.21.15.16:6829","nonce":37021}]},"public_addr":"172.21.15.16:6825/37021","cluster_addr":"172.21.15.16:6827/37021","heartbeat_back_addr":"172.21.15.16:6831/37021","heartbeat_front_addr":"172.21.15.16:6829/37021","state":["exists","up"]},{"osd":1,"uuid":"4702fbc9-a927-4565-aca5-41dc69cc8264","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":37008},{"type":"v1","addr":"172.21.15.16:6801","nonce":37008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":37008},{"type":"v1","addr":"172.21.15.16:6803","nonce":37008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":37008},{"type":"v1","addr":"172.21.15.16:6807","nonce":37008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":37008},{"type":"v1","addr":"172.21.15.16:6805","nonce":37008}]},"public_addr":"172.21.15.16:6801/37008","cluster_addr":"172.21.15.16:6803/37008","heartbeat_back_addr":"172.21.15.16:6807/37008","heartbeat_front_addr":"172.21.15.16:6805/37008","state":["exists","up"]},{"osd":2,"uuid":"1e7bba46-4eba-47ef-bb81-5e2d371638ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":37009},{"type":"v1","addr":"172.21.15.16:6809","nonce":37009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":37009},{"type":"v1","addr":"172.21.15.16:6811","nonce":37009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":37009},{"type":"v1","addr":"172.21.15.16:6815","nonce":37009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":37009},{"type":"v1","addr":"172.21.15.16:6813","nonce":37009}]},"public_addr":"172.21.15.16:6809/37009","cluster_addr":"172.21.15.16:6811/37009","heartbeat_back_addr":"172.21.15.16:6815/37009","heartbeat_front_addr":"172.21.15.16:6813/37009","state":["exists","up"]},{"osd":3,"uuid":"383cacc0-5a88-4904-990e-498ca59ef48a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":37020},{"type":"v1","addr":"172.21.15.16:6817","nonce":37020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":37020},{"type":"v1","addr":"172.21.15.16:6819","nonce":37020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":37020},{"type":"v1","addr":"172.21.15.16:6823","nonce":37020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":37020},{"type":"v1","addr":"172.21.15.16:6821","nonce":37020}]},"public_addr":"172.21.15.16:6817/37020","cluster_addr":"172.21.15.16:6819/37020","heartbeat_back_addr":"172.21.15.16:6823/37020","heartbeat_front_addr":"172.21.15.16:6821/37020","state":["exists","up"]},{"osd":4,"uuid":"970500ef-95b7-42c1-b080-eed4fae7013d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":37353},{"type":"v1","addr":"172.21.15.62:6817","nonce":37353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6819","nonce":37353},{"type":"v1","addr":"172.21.15.62:6822","nonce":37353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37353},{"type":"v1","addr":"172.21.15.62:6831","nonce":37353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37353},{"type":"v1","addr":"172.21.15.62:6826","nonce":37353}]},"public_addr":"172.21.15.62:6817/37353","cluster_addr":"172.21.15.62:6822/37353","heartbeat_back_addr":"172.21.15.62:6831/37353","heartbeat_front_addr":"172.21.15.62:6826/37353","state":["exists","up"]},{"osd":5,"uuid":"022d67db-d4d8-445b-b1b9-8b446a00f5c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":37352},{"type":"v1","addr":"172.21.15.62:6801","nonce":37352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":37352},{"type":"v1","addr":"172.21.15.62:6803","nonce":37352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":37352},{"type":"v1","addr":"172.21.15.62:6807","nonce":37352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":37352},{"type":"v1","addr":"172.21.15.62:6805","nonce":37352}]},"public_addr":"172.21.15.62:6801/37352","cluster_addr":"172.21.15.62:6803/37352","heartbeat_back_addr":"172.21.15.62:6807/37352","heartbeat_front_addr":"172.21.15.62:6805/37352","state":["exists","up"]},{"osd":6,"uuid":"42bd372b-6688-4a1b-8407-e16e8c266e33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":37354},{"type":"v1","addr":"172.21.15.62:6820","nonce":37354}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":37354},{"type":"v1","addr":"172.21.15.62:6823","nonce":37354}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6829","nonce":37354},{"type":"v1","addr":"172.21.15.62:6830","nonce":37354}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6825","nonce":37354},{"type":"v1","addr":"172.21.15.62:6827","nonce":37354}]},"public_addr":"172.21.15.62:6820/37354","cluster_addr":"172.21.15.62:6823/37354","heartbeat_back_addr":"172.21.15.62:6830/37354","heartbeat_front_addr":"172.21.15.62:6827/37354","state":["exists","up"]},{"osd":7,"uuid":"ac70b2a3-f41b-4bf2-b227-7f2cbb55426d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37355},{"type":"v1","addr":"172.21.15.62:6809","nonce":37355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37355},{"type":"v1","addr":"172.21.15.62:6811","nonce":37355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":37355},{"type":"v1","addr":"172.21.15.62:6815","nonce":37355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":37355},{"type":"v1","addr":"172.21.15.62:6813","nonce":37355}]},"public_addr":"172.21.15.62:6809/37355","cluster_addr":"172.21.15.62:6811/37355","heartbeat_back_addr":"172.21.15.62:6815/37355","heartbeat_front_addr":"172.21.15.62:6813/37355","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.368896+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.107273+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.276888+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.287502+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.124791+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.251937+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.255679+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-22T15:26:33.232797+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:6837/1705367575":"2023-05-23T16:05:58.453097+0000","172.21.15.16:0/3256922077":"2023-05-23T16:05:55.761117+0000","172.21.15.16:0/1351897283":"2023-05-23T16:05:55.761117+0000","172.21.15.16:0/168959634":"2023-05-23T16:05:55.761117+0000","172.21.15.16:0/3781682956":"2023-05-23T16:05:55.761117+0000","172.21.15.16:0/1599584790":"2023-05-23T16:05:55.761117+0000","172.21.15.16:0/1961044213":"2023-05-23T16:05:35.647562+0000","172.21.15.16:0/238292614":"2023-05-23T16:05:35.647562+0000","172.21.15.16:0/1185672517":"2023-05-23T16:05:55.761117+0000","172.21.15.16:0/2111070480":"2023-05-23T16:05:55.761117+0000","172.21.15.16:6833/65890":"2023-05-23T16:05:35.647562+0000","172.21.15.16:0/1148602276":"2023-05-23T16:05:35.647562+0000","172.21.15.16:6832/65890":"2023-05-23T16:05:35.647562+0000","172.21.15.16:6836/1705367575":"2023-05-23T16:05:58.453097+0000","172.21.15.16:0/456460873":"2023-05-23T16:05:35.647562+0000","172.21.15.16:0/2766506027":"2023-05-23T16:05:35.647562+0000","172.21.15.16:0/2214029832":"2023-05-23T16:05:35.647562+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":17,"snaps":[{"begin":2,"length":2}]},{"pool":21,"snaps":[{"begin":25,"length":9}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-05-22T16:06:06.448 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:06.443+0000 7f715851e640 1 -- 172.21.15.16:0/2719349328 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f71340654a0 msgr2=0x7f7134067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:06:06.448 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:06.443+0000 7f715851e640 1 --2- 172.21.15.16:0/2719349328 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f71340654a0 0x7f7134067950 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f71501f3c10 tx=0x7f7144008040 comp rx=0 tx=0).stop 2023-05-22T16:06:06.448 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:06.443+0000 7f715851e640 1 -- 172.21.15.16:0/2719349328 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7150158be0 msgr2=0x7f71500c1c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:06:06.448 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:06.443+0000 7f715851e640 1 --2- 172.21.15.16:0/2719349328 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7150158be0 0x7f71500c1c00 secure :-1 s=READY pgs=2623 cs=0 l=1 rev1=1 crypto rx=0x7f714800c710 tx=0x7f714800cbd0 comp rx=0 tx=0).stop 2023-05-22T16:06:06.449 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:06.443+0000 7f715851e640 1 -- 172.21.15.16:0/2719349328 shutdown_connections 2023-05-22T16:06:06.449 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:06.443+0000 7f715851e640 1 --2- 172.21.15.16:0/2719349328 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f71340654a0 0x7f7134067950 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:06.449 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:06.443+0000 7f715851e640 1 --2- 172.21.15.16:0/2719349328 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7150158be0 0x7f71500c1c00 unknown :-1 s=CLOSED pgs=2623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:06.449 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:06.443+0000 7f715851e640 1 --2- 172.21.15.16:0/2719349328 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f71500c8840 0x7f71500b2df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:06.449 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:06.443+0000 7f715851e640 1 --2- 172.21.15.16:0/2719349328 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f715006c8e0 0x7f715014c4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:06.450 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:06.443+0000 7f715851e640 1 -- 172.21.15.16:0/2719349328 >> 172.21.15.16:0/2719349328 conn(0x7f71500cab50 msgr2=0x7f71500af910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:06:06.450 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:06.443+0000 7f715851e640 1 -- 172.21.15.16:0/2719349328 shutdown_connections 2023-05-22T16:06:06.450 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:06.443+0000 7f715851e640 1 -- 172.21.15.16:0/2719349328 wait complete. 2023-05-22T16:06:07.466 INFO:tasks.ceph:Scrubbing osd.0 2023-05-22T16:06:07.467 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2023-05-22T16:06:07.606 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:07.603+0000 7f65fccf7640 1 Processor -- start 2023-05-22T16:06:07.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:07.603+0000 7f65fccf7640 1 -- start start 2023-05-22T16:06:07.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:07.603+0000 7f65fccf7640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f65f815bec0 0x7f65f80b53d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:07.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:07.603+0000 7f65fccf7640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f65f815c290 0x7f65f80b1120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:07.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:07.603+0000 7f65fccf7640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65f80b1e30 con 0x7f65f80b1660 2023-05-22T16:06:07.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:07.603+0000 7f65fccf7640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65f80b1b50 con 0x7f65f815bec0 2023-05-22T16:06:07.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:07.603+0000 7f65fccf7640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65f80d0490 con 0x7f65f815c290 2023-05-22T16:06:07.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:07.603+0000 7f65f6d9d640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f65f80b1660 0x7f65f80b4000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:49834/0 (socket says 172.21.15.16:49834) 2023-05-22T16:06:07.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:07.603+0000 7f65f6d9d640 1 -- 172.21.15.16:0/104285802 learned_addr learned my addr 172.21.15.16:0/104285802 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T16:06:07.610 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:07.603+0000 7f65f5d9b640 1 --2- 172.21.15.16:0/104285802 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f65f815c290 0x7f65f80b1120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:07.610 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:07.603+0000 7f65f659c640 1 --2- 172.21.15.16:0/104285802 >> v2:172.21.15.62:3300/0 conn(0x7f65f815bec0 0x7f65f80b53d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:07.610 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:07.603+0000 7f65f559a640 1 -- 172.21.15.16:0/104285802 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 491975683 0 0) 0x7f65f80b1e30 con 0x7f65f80b1660 2023-05-22T16:06:07.610 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:07.603+0000 7f65f559a640 1 -- 172.21.15.16:0/104285802 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f65e0003530 con 0x7f65f80b1660 2023-05-22T16:06:07.610 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:07.603+0000 7f65f559a640 1 -- 172.21.15.16:0/104285802 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3481315817 0 0) 0x7f65e0003530 con 0x7f65f80b1660 2023-05-22T16:06:07.611 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:07.603+0000 7f65f559a640 1 -- 172.21.15.16:0/104285802 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f65f815c290 msgr2=0x7f65f80b1120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:06:07.611 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:07.603+0000 7f65f559a640 1 --2- 172.21.15.16:0/104285802 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f65f815c290 0x7f65f80b1120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:07.611 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:07.603+0000 7f65f559a640 1 -- 172.21.15.16:0/104285802 >> v2:172.21.15.62:3300/0 conn(0x7f65f815bec0 msgr2=0x7f65f80b53d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:06:07.611 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:07.603+0000 7f65f559a640 1 --2- 172.21.15.16:0/104285802 >> v2:172.21.15.62:3300/0 conn(0x7f65f815bec0 0x7f65f80b53d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:07.611 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:07.603+0000 7f65f559a640 1 -- 172.21.15.16:0/104285802 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65f80d0770 con 0x7f65f80b1660 2023-05-22T16:06:07.611 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:07.603+0000 7f65f559a640 1 -- 172.21.15.16:0/104285802 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7f65f0001a00 con 0x7f65f80b1660 2023-05-22T16:06:07.612 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:07.603+0000 7f65f559a640 1 -- 172.21.15.16:0/104285802 >> v1:172.21.15.16:6789/0 conn(0x7f65f80b1660 legacy=0x7f65f80b4000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:06:07.612 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:07.607+0000 7f65f559a640 1 --2- 172.21.15.16:0/104285802 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f65e0003a40 0x7f65e0003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:07.612 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:07.607+0000 7f65f559a640 1 --2- 172.21.15.16:0/104285802 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f65e00047d0 0x7f65e0004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:07.613 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:07.607+0000 7f65f559a640 1 --2- 172.21.15.16:0/104285802 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f65e0005180 0x7f65e000b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:07.613 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:07.607+0000 7f65f559a640 1 -- 172.21.15.16:0/104285802 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65f80b1b50 con 0x7f65e00047d0 2023-05-22T16:06:07.613 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:07.607+0000 7f65f559a640 1 -- 172.21.15.16:0/104285802 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65f80d0490 con 0x7f65e0003a40 2023-05-22T16:06:07.614 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:07.607+0000 7f65f559a640 1 -- 172.21.15.16:0/104285802 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65e000bed0 con 0x7f65e0005180 2023-05-22T16:06:07.614 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:07.607+0000 7f65f5d9b640 1 --2- 172.21.15.16:0/104285802 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f65e00047d0 0x7f65e0004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:07.615 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:07.607+0000 7f65f6d9d640 1 --2- 172.21.15.16:0/104285802 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f65e0005180 0x7f65e000b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:07.615 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:07.607+0000 7f65f659c640 1 --2- 172.21.15.16:0/104285802 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f65e0003a40 0x7f65e0003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:07.615 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:07.607+0000 7f65f5d9b640 1 -- 172.21.15.16:0/104285802 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f65e0005180 msgr2=0x7f65e000b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:06:07.615 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:07.607+0000 7f65f5d9b640 1 --2- 172.21.15.16:0/104285802 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f65e0005180 0x7f65e000b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:07.616 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:07.607+0000 7f65f5d9b640 1 -- 172.21.15.16:0/104285802 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f65e0003a40 msgr2=0x7f65e0003e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:06:07.616 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:07.607+0000 7f65f5d9b640 1 --2- 172.21.15.16:0/104285802 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f65e0003a40 0x7f65e0003e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:07.616 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:07.607+0000 7f65f5d9b640 1 -- 172.21.15.16:0/104285802 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65e0003530 con 0x7f65e00047d0 2023-05-22T16:06:07.626 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:07.607+0000 7f65f5d9b640 1 --2- 172.21.15.16:0/104285802 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f65e00047d0 0x7f65e0004c40 secure :-1 s=READY pgs=2625 cs=0 l=1 rev1=1 crypto rx=0x7f65ec009ae0 tx=0x7f65ec01ca20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30336ceb3f9ab351 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:06:07.626 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:07.607+0000 7f65f559a640 1 -- 172.21.15.16:0/104285802 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65ec0a6510 con 0x7f65e00047d0 2023-05-22T16:06:07.626 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:07.607+0000 7f65f559a640 1 -- 172.21.15.16:0/104285802 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65ec0a6670 con 0x7f65e00047d0 2023-05-22T16:06:07.627 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:07.607+0000 7f65f559a640 1 -- 172.21.15.16:0/104285802 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65ec0a6a20 con 0x7f65e00047d0 2023-05-22T16:06:07.627 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:07.607+0000 7f65fccf7640 1 -- 172.21.15.16:0/104285802 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f65e00047d0 msgr2=0x7f65e0004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:06:07.627 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:07.607+0000 7f65fccf7640 1 --2- 172.21.15.16:0/104285802 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f65e00047d0 0x7f65e0004c40 secure :-1 s=READY pgs=2625 cs=0 l=1 rev1=1 crypto rx=0x7f65ec009ae0 tx=0x7f65ec01ca20 comp rx=0 tx=0).stop 2023-05-22T16:06:07.627 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:07.607+0000 7f65f6d9d640 1 -- 172.21.15.16:0/104285802 reap_dead start 2023-05-22T16:06:07.627 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:07.607+0000 7f65fccf7640 1 -- 172.21.15.16:0/104285802 shutdown_connections 2023-05-22T16:06:07.627 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:07.607+0000 7f65fccf7640 1 -- 172.21.15.16:0/104285802 >> 172.21.15.16:0/104285802 conn(0x7f65f80cf000 msgr2=0x7f65f80cf400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:06:07.627 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:07.607+0000 7f65fccf7640 1 -- 172.21.15.16:0/104285802 shutdown_connections 2023-05-22T16:06:07.628 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:07.607+0000 7f65fccf7640 1 -- 172.21.15.16:0/104285802 wait complete. 2023-05-22T16:06:07.628 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:07.611+0000 7f65fccf7640 1 Processor -- start 2023-05-22T16:06:07.628 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:07.611+0000 7f65fccf7640 1 -- start start 2023-05-22T16:06:07.628 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:07.611+0000 7f65fccf7640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f65e00047d0 0x7f65f81524c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:07.628 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:07.611+0000 7f65fccf7640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f65f8151ec0 0x7f65f815d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:07.628 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:07.611+0000 7f65fccf7640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f65f8159560 0x7f65f81599d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:07.628 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:07.611+0000 7f65fccf7640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65f8152330 con 0x7f65f8151ec0 2023-05-22T16:06:07.629 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:07.611+0000 7f65fccf7640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65f815e8c0 con 0x7f65f8159560 2023-05-22T16:06:07.629 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:07.611+0000 7f65fccf7640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65f815e020 con 0x7f65e00047d0 2023-05-22T16:06:07.629 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:07.611+0000 7f65f5d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f65f8151ec0 0x7f65f815d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:07.629 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:07.611+0000 7f65f5d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f65f8151ec0 0x7f65f815d990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43554/0 (socket says 172.21.15.16:43554) 2023-05-22T16:06:07.629 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:07.611+0000 7f65f5d9b640 1 -- 172.21.15.16:0/2802852931 learned_addr learned my addr 172.21.15.16:0/2802852931 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:06:07.629 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:07.611+0000 7f65f6d9d640 1 --2- 172.21.15.16:0/2802852931 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f65f8159560 0x7f65f81599d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:07.629 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:07.611+0000 7f65f659c640 1 --2- 172.21.15.16:0/2802852931 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f65e00047d0 0x7f65f81524c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:07.629 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:07.611+0000 7f65f5d9b640 1 -- 172.21.15.16:0/2802852931 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f65e00047d0 msgr2=0x7f65f81524c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:06:07.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:07.611+0000 7f65f5d9b640 1 --2- 172.21.15.16:0/2802852931 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f65e00047d0 0x7f65f81524c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:07.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:07.611+0000 7f65f5d9b640 1 -- 172.21.15.16:0/2802852931 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f65f8159560 msgr2=0x7f65f81599d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:06:07.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:07.611+0000 7f65f5d9b640 1 --2- 172.21.15.16:0/2802852931 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f65f8159560 0x7f65f81599d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:07.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:07.611+0000 7f65f5d9b640 1 -- 172.21.15.16:0/2802852931 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65f80b5aa0 con 0x7f65f8151ec0 2023-05-22T16:06:07.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:07.611+0000 7f65f659c640 1 --2- 172.21.15.16:0/2802852931 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f65e00047d0 0x7f65f81524c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T16:06:07.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:07.611+0000 7f65f5d9b640 1 --2- 172.21.15.16:0/2802852931 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f65f8151ec0 0x7f65f815d990 secure :-1 s=READY pgs=2626 cs=0 l=1 rev1=1 crypto rx=0x7f65ec01c850 tx=0x7f65e0005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:06:07.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:07.611+0000 7f65e6ffd640 1 -- 172.21.15.16:0/2802852931 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65ec00f000 con 0x7f65f8151ec0 2023-05-22T16:06:07.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:07.611+0000 7f65e6ffd640 1 -- 172.21.15.16:0/2802852931 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65ec0a67e0 con 0x7f65f8151ec0 2023-05-22T16:06:07.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:07.611+0000 7f65fccf7640 1 -- 172.21.15.16:0/2802852931 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65f80b5910 con 0x7f65f8151ec0 2023-05-22T16:06:07.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:07.611+0000 7f65e6ffd640 1 -- 172.21.15.16:0/2802852931 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65ec0a6b90 con 0x7f65f8151ec0 2023-05-22T16:06:07.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:07.611+0000 7f65fccf7640 1 -- 172.21.15.16:0/2802852931 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65f80d1780 con 0x7f65f8151ec0 2023-05-22T16:06:07.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:07.611+0000 7f65fccf7640 1 -- 172.21.15.16:0/2802852931 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f65f804dcc0 con 0x7f65f8151ec0 2023-05-22T16:06:07.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:07.611+0000 7f65e6ffd640 1 -- 172.21.15.16:0/2802852931 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 100) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f65ec0a6cf0 con 0x7f65f8151ec0 2023-05-22T16:06:07.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:07.615+0000 7f65e6ffd640 1 --2- 172.21.15.16:0/2802852931 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f65d80654a0 0x7f65d8067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:07.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:07.615+0000 7f65e6ffd640 1 -- 172.21.15.16:0/2802852931 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(201..201 src has 1..201) v4 ==== 6317+0+0 (secure 0 0 0) 0x7f65ec0aa070 con 0x7f65f8151ec0 2023-05-22T16:06:07.632 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:07.615+0000 7f65f659c640 1 --2- 172.21.15.16:0/2802852931 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f65d80654a0 0x7f65d8067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:07.632 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:07.615+0000 7f65e6ffd640 1 --2- 172.21.15.16:0/2802852931 >> [v2:172.21.15.16:6824/37021,v1:172.21.15.16:6825/37021] conn(0x7f65d806c350 0x7f65d806e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:07.632 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:07.615+0000 7f65e6ffd640 1 -- 172.21.15.16:0/2802852931 --> [v2:172.21.15.16:6824/37021,v1:172.21.15.16:6825/37021] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f65d806ee90 con 0x7f65d806c350 2023-05-22T16:06:07.632 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:07.615+0000 7f65e6ffd640 1 -- 172.21.15.16:0/2802852931 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_get_version_reply(handle=1 version=201) v2 ==== 24+0+0 (secure 0 0 0) 0x7f65ec11e5f0 con 0x7f65f8151ec0 2023-05-22T16:06:07.632 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:07.615+0000 7f65f6d9d640 1 --2- 172.21.15.16:0/2802852931 >> [v2:172.21.15.16:6824/37021,v1:172.21.15.16:6825/37021] conn(0x7f65d806c350 0x7f65d806e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:07.632 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:07.615+0000 7f65f659c640 1 --2- 172.21.15.16:0/2802852931 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f65d80654a0 0x7f65d8067950 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f65f815a4a0 tx=0x7f65e8008040 comp rx=0 tx=0).ready entity=mgr.8195 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:06:07.632 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:07.615+0000 7f65f6d9d640 1 --2- 172.21.15.16:0/2802852931 >> [v2:172.21.15.16:6824/37021,v1:172.21.15.16:6825/37021] conn(0x7f65d806c350 0x7f65d806e7c0 crc :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:06:07.633 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:07.615+0000 7f65e6ffd640 1 -- 172.21.15.16:0/2802852931 <== osd.0 v2:172.21.15.16:6824/37021 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f65d806ee90 con 0x7f65d806c350 2023-05-22T16:06:07.633 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:07.627+0000 7f65fccf7640 1 -- 172.21.15.16:0/2802852931 --> [v2:172.21.15.16:6824/37021,v1:172.21.15.16:6825/37021] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f65f8152cd0 con 0x7f65d806c350 2023-05-22T16:06:07.633 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:07.627+0000 7f65e6ffd640 1 -- 172.21.15.16:0/2802852931 <== osd.0 v2:172.21.15.16:6824/37021 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f65f8152cd0 con 0x7f65d806c350 2023-05-22T16:06:07.634 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-05-22T16:06:07.634 INFO:teuthology.orchestra.run.smithi016.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-05-22T16:06:07.634 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-05-22T16:06:07.635 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:07.627+0000 7f65fccf7640 1 -- 172.21.15.16:0/2802852931 >> [v2:172.21.15.16:6824/37021,v1:172.21.15.16:6825/37021] conn(0x7f65d806c350 msgr2=0x7f65d806e7c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:06:07.635 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:07.627+0000 7f65fccf7640 1 --2- 172.21.15.16:0/2802852931 >> [v2:172.21.15.16:6824/37021,v1:172.21.15.16:6825/37021] conn(0x7f65d806c350 0x7f65d806e7c0 crc :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:07.635 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:07.627+0000 7f65fccf7640 1 -- 172.21.15.16:0/2802852931 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f65d80654a0 msgr2=0x7f65d8067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:06:07.635 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:07.627+0000 7f65fccf7640 1 --2- 172.21.15.16:0/2802852931 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f65d80654a0 0x7f65d8067950 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f65f815a4a0 tx=0x7f65e8008040 comp rx=0 tx=0).stop 2023-05-22T16:06:07.635 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:07.627+0000 7f65fccf7640 1 -- 172.21.15.16:0/2802852931 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f65f8151ec0 msgr2=0x7f65f815d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:06:07.636 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:07.627+0000 7f65fccf7640 1 --2- 172.21.15.16:0/2802852931 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f65f8151ec0 0x7f65f815d990 secure :-1 s=READY pgs=2626 cs=0 l=1 rev1=1 crypto rx=0x7f65ec01c850 tx=0x7f65e0005800 comp rx=0 tx=0).stop 2023-05-22T16:06:07.636 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:07.627+0000 7f65f6d9d640 1 -- 172.21.15.16:0/2802852931 reap_dead start 2023-05-22T16:06:07.636 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:07.627+0000 7f65fccf7640 1 -- 172.21.15.16:0/2802852931 shutdown_connections 2023-05-22T16:06:07.636 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:07.627+0000 7f65fccf7640 1 -- 172.21.15.16:0/2802852931 >> 172.21.15.16:0/2802852931 conn(0x7f65f80cf000 msgr2=0x7f65f80b7140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:06:07.636 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:07.627+0000 7f65fccf7640 1 -- 172.21.15.16:0/2802852931 shutdown_connections 2023-05-22T16:06:07.637 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:07.627+0000 7f65fccf7640 1 -- 172.21.15.16:0/2802852931 wait complete. 2023-05-22T16:06:07.648 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2023-05-22T16:06:07.765 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:07.763+0000 7fb5226ca640 1 Processor -- start 2023-05-22T16:06:07.766 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:07.763+0000 7fb5226ca640 1 -- start start 2023-05-22T16:06:07.766 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:07.763+0000 7fb5226ca640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb51c070700 0x7fb51c06cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:07.767 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:07.763+0000 7fb5226ca640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb51c070c40 0x7fb51c06d110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:07.767 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:07.763+0000 7fb5226ca640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb51c0b46b0 con 0x7fb51c159da0 2023-05-22T16:06:07.767 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:07.763+0000 7fb5226ca640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb51c06c0f0 con 0x7fb51c070700 2023-05-22T16:06:07.767 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:07.763+0000 7fb5226ca640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb51c06c3d0 con 0x7fb51c070c40 2023-05-22T16:06:07.767 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:07.763+0000 7fb51b7fe640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb51c070700 0x7fb51c06cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:07.767 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:07.763+0000 7fb51bfff640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fb51c159da0 0x7fb51c06fff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:49848/0 (socket says 172.21.15.16:49848) 2023-05-22T16:06:07.767 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:07.763+0000 7fb51bfff640 1 -- 172.21.15.16:0/1319473253 learned_addr learned my addr 172.21.15.16:0/1319473253 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T16:06:07.768 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:07.763+0000 7fb520c67640 1 --2- 172.21.15.16:0/1319473253 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb51c070c40 0x7fb51c06d110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:07.768 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:07.763+0000 7fb51b7fe640 1 -- 172.21.15.16:0/1319473253 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb51c070c40 msgr2=0x7fb51c06d110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:06:07.768 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:07.763+0000 7fb51b7fe640 1 --2- 172.21.15.16:0/1319473253 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb51c070c40 0x7fb51c06d110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:07.768 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:07.763+0000 7fb51b7fe640 1 -- 172.21.15.16:0/1319473253 >> v1:172.21.15.16:6789/0 conn(0x7fb51c159da0 legacy=0x7fb51c06fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:06:07.769 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:07.763+0000 7fb51b7fe640 1 -- 172.21.15.16:0/1319473253 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb51c06c6b0 con 0x7fb51c070700 2023-05-22T16:06:07.769 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:07.767+0000 7fb51b7fe640 1 --2- 172.21.15.16:0/1319473253 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb51c070700 0x7fb51c06cbd0 secure :-1 s=READY pgs=2628 cs=0 l=1 rev1=1 crypto rx=0x7fb5100097c0 tx=0x7fb51001ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=393ed61933b9ade4 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:06:07.770 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:07.767+0000 7fb51affd640 1 -- 172.21.15.16:0/1319473253 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5100a8970 con 0x7fb51c070700 2023-05-22T16:06:07.770 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:07.767+0000 7fb51affd640 1 -- 172.21.15.16:0/1319473253 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5100a8ad0 con 0x7fb51c070700 2023-05-22T16:06:07.770 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:07.767+0000 7fb51affd640 1 -- 172.21.15.16:0/1319473253 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5100a8e60 con 0x7fb51c070700 2023-05-22T16:06:07.770 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:07.767+0000 7fb5226ca640 1 -- 172.21.15.16:0/1319473253 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb51c070700 msgr2=0x7fb51c06cbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:06:07.770 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:07.767+0000 7fb5226ca640 1 --2- 172.21.15.16:0/1319473253 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb51c070700 0x7fb51c06cbd0 secure :-1 s=READY pgs=2628 cs=0 l=1 rev1=1 crypto rx=0x7fb5100097c0 tx=0x7fb51001ec70 comp rx=0 tx=0).stop 2023-05-22T16:06:07.771 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:07.767+0000 7fb5226ca640 1 -- 172.21.15.16:0/1319473253 shutdown_connections 2023-05-22T16:06:07.771 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:07.767+0000 7fb5226ca640 1 --2- 172.21.15.16:0/1319473253 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb51c070c40 0x7fb51c06d110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:07.771 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:07.767+0000 7fb5226ca640 1 --2- 172.21.15.16:0/1319473253 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb51c070700 0x7fb51c06cbd0 unknown :-1 s=CLOSED pgs=2628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:07.771 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:07.767+0000 7fb5226ca640 1 -- 172.21.15.16:0/1319473253 >> 172.21.15.16:0/1319473253 conn(0x7fb51c0cea90 msgr2=0x7fb51c0cee90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:06:07.772 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:07.767+0000 7fb5226ca640 1 -- 172.21.15.16:0/1319473253 shutdown_connections 2023-05-22T16:06:07.772 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:07.767+0000 7fb5226ca640 1 -- 172.21.15.16:0/1319473253 wait complete. 2023-05-22T16:06:07.773 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:07.767+0000 7fb5226ca640 1 Processor -- start 2023-05-22T16:06:07.774 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:07.767+0000 7fb5226ca640 1 -- start start 2023-05-22T16:06:07.774 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:07.771+0000 7fb5226ca640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb51c070700 0x7fb51c148e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:07.774 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:07.771+0000 7fb5226ca640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb51c070c40 0x7fb51c0b3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:07.774 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:07.771+0000 7fb5226ca640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb51c159da0 0x7fb51c0ac970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:07.774 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:07.771+0000 7fb5226ca640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb51c0ab8f0 con 0x7fb51c070700 2023-05-22T16:06:07.774 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:07.771+0000 7fb5226ca640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb51c0abbb0 con 0x7fb51c070c40 2023-05-22T16:06:07.774 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:07.771+0000 7fb5226ca640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb51c0abe90 con 0x7fb51c159da0 2023-05-22T16:06:07.775 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:07.771+0000 7fb51bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb51c070700 0x7fb51c148e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:07.775 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:07.771+0000 7fb51bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb51c070700 0x7fb51c148e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43564/0 (socket says 172.21.15.16:43564) 2023-05-22T16:06:07.775 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:07.771+0000 7fb51bfff640 1 -- 172.21.15.16:0/2419599384 learned_addr learned my addr 172.21.15.16:0/2419599384 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:06:07.775 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:07.771+0000 7fb520c67640 1 --2- 172.21.15.16:0/2419599384 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb51c159da0 0x7fb51c0ac970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:07.775 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:07.771+0000 7fb51b7fe640 1 --2- 172.21.15.16:0/2419599384 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb51c070c40 0x7fb51c0b3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:07.775 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:07.771+0000 7fb51bfff640 1 -- 172.21.15.16:0/2419599384 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb51c159da0 msgr2=0x7fb51c0ac970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:06:07.775 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:07.771+0000 7fb51bfff640 1 --2- 172.21.15.16:0/2419599384 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb51c159da0 0x7fb51c0ac970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:07.775 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:07.771+0000 7fb51bfff640 1 -- 172.21.15.16:0/2419599384 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb51c070c40 msgr2=0x7fb51c0b3680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:06:07.776 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:07.771+0000 7fb51bfff640 1 --2- 172.21.15.16:0/2419599384 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb51c070c40 0x7fb51c0b3680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:07.776 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:07.771+0000 7fb51bfff640 1 -- 172.21.15.16:0/2419599384 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb510009420 con 0x7fb51c070700 2023-05-22T16:06:07.776 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:07.771+0000 7fb51b7fe640 1 --2- 172.21.15.16:0/2419599384 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb51c070c40 0x7fb51c0b3680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T16:06:07.776 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:07.771+0000 7fb51bfff640 1 --2- 172.21.15.16:0/2419599384 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb51c070700 0x7fb51c148e50 secure :-1 s=READY pgs=2629 cs=0 l=1 rev1=1 crypto rx=0x7fb50c00c370 tx=0x7fb50c00c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:06:07.776 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:07.771+0000 7fb518ff9640 1 -- 172.21.15.16:0/2419599384 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb50c005cf0 con 0x7fb51c070700 2023-05-22T16:06:07.776 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:07.771+0000 7fb518ff9640 1 -- 172.21.15.16:0/2419599384 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb50c005e50 con 0x7fb51c070700 2023-05-22T16:06:07.776 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:07.771+0000 7fb5226ca640 1 -- 172.21.15.16:0/2419599384 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb51c0b4340 con 0x7fb51c070700 2023-05-22T16:06:07.777 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:07.771+0000 7fb518ff9640 1 -- 172.21.15.16:0/2419599384 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb50c0072f0 con 0x7fb51c070700 2023-05-22T16:06:07.777 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:07.771+0000 7fb5226ca640 1 -- 172.21.15.16:0/2419599384 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb51c0b4990 con 0x7fb51c070700 2023-05-22T16:06:07.777 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:07.771+0000 7fb5226ca640 1 -- 172.21.15.16:0/2419599384 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb51c0add40 con 0x7fb51c070700 2023-05-22T16:06:07.778 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:07.771+0000 7fb518ff9640 1 -- 172.21.15.16:0/2419599384 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 100) v1 ==== 84332+0+0 (secure 0 0 0) 0x7fb50c007450 con 0x7fb51c070700 2023-05-22T16:06:07.778 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:07.771+0000 7fb518ff9640 1 --2- 172.21.15.16:0/2419599384 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fb5080654a0 0x7fb508067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:07.778 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:07.771+0000 7fb518ff9640 1 -- 172.21.15.16:0/2419599384 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(201..201 src has 1..201) v4 ==== 6317+0+0 (secure 0 0 0) 0x7fb50c081fc0 con 0x7fb51c070700 2023-05-22T16:06:07.778 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:07.771+0000 7fb51b7fe640 1 --2- 172.21.15.16:0/2419599384 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fb5080654a0 0x7fb508067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:07.778 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:07.771+0000 7fb51b7fe640 1 --2- 172.21.15.16:0/2419599384 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fb5080654a0 0x7fb508067950 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fb5100b2000 tx=0x7fb5100aa070 comp rx=0 tx=0).ready entity=mgr.8195 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:06:07.779 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:07.775+0000 7fb518ff9640 1 -- 172.21.15.16:0/2419599384 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fb50c04fac0 con 0x7fb51c070700 2023-05-22T16:06:07.934 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:07.931+0000 7fb5226ca640 1 -- 172.21.15.16:0/2419599384 --> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) v1 -- 0x7fb51c073050 con 0x7fb5080654a0 2023-05-22T16:06:07.934 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:07.931+0000 7fb518ff9640 1 -- 172.21.15.16:0/2419599384 <== mgr.8195 v2:172.21.15.16:6832/65890 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fb51c073050 con 0x7fb5080654a0 2023-05-22T16:06:07.935 INFO:teuthology.orchestra.run.smithi016.stderr:instructed osd(s) 0 to deep-scrub 2023-05-22T16:06:07.937 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:07.931+0000 7fb5226ca640 1 -- 172.21.15.16:0/2419599384 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fb5080654a0 msgr2=0x7fb508067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:06:07.937 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:07.931+0000 7fb5226ca640 1 --2- 172.21.15.16:0/2419599384 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fb5080654a0 0x7fb508067950 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fb5100b2000 tx=0x7fb5100aa070 comp rx=0 tx=0).stop 2023-05-22T16:06:07.937 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:07.931+0000 7fb5226ca640 1 -- 172.21.15.16:0/2419599384 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb51c070700 msgr2=0x7fb51c148e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:06:07.938 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:07.931+0000 7fb5226ca640 1 --2- 172.21.15.16:0/2419599384 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb51c070700 0x7fb51c148e50 secure :-1 s=READY pgs=2629 cs=0 l=1 rev1=1 crypto rx=0x7fb50c00c370 tx=0x7fb50c00c830 comp rx=0 tx=0).stop 2023-05-22T16:06:07.938 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:07.935+0000 7fb5226ca640 1 -- 172.21.15.16:0/2419599384 shutdown_connections 2023-05-22T16:06:07.938 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:07.935+0000 7fb5226ca640 1 --2- 172.21.15.16:0/2419599384 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb51c159da0 0x7fb51c0ac970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:07.938 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:07.935+0000 7fb5226ca640 1 --2- 172.21.15.16:0/2419599384 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb51c070c40 0x7fb51c0b3680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:07.938 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:07.935+0000 7fb5226ca640 1 --2- 172.21.15.16:0/2419599384 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fb5080654a0 0x7fb508067950 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:07.938 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:07.935+0000 7fb5226ca640 1 --2- 172.21.15.16:0/2419599384 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb51c070700 0x7fb51c148e50 unknown :-1 s=CLOSED pgs=2629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:07.938 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:07.935+0000 7fb5226ca640 1 -- 172.21.15.16:0/2419599384 >> 172.21.15.16:0/2419599384 conn(0x7fb51c0cea90 msgr2=0x7fb51c159080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:06:07.939 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:07.935+0000 7fb5226ca640 1 -- 172.21.15.16:0/2419599384 shutdown_connections 2023-05-22T16:06:07.939 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:07.935+0000 7fb5226ca640 1 -- 172.21.15.16:0/2419599384 wait complete. 2023-05-22T16:06:07.954 INFO:tasks.ceph:Scrubbing osd.1 2023-05-22T16:06:07.955 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2023-05-22T16:06:08.087 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.083+0000 7f268dac4640 1 Processor -- start 2023-05-22T16:06:08.087 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.083+0000 7f268dac4640 1 -- start start 2023-05-22T16:06:08.088 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.083+0000 7f268dac4640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2688158a40 0x7f268806ec00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:08.088 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.087+0000 7f268dac4640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f268806f140 con 0x7f2688158e10 2023-05-22T16:06:08.089 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.087+0000 7f268dac4640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f268815c730 con 0x7f26880cc300 2023-05-22T16:06:08.089 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.087+0000 7f268dac4640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f268815ca10 con 0x7f2688158a40 2023-05-22T16:06:08.089 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.087+0000 7f2686ffd640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f2688158e10 0x7f26880cbbf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:49850/0 (socket says 172.21.15.16:49850) 2023-05-22T16:06:08.089 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.087+0000 7f2686ffd640 1 -- 172.21.15.16:0/647295190 learned_addr learned my addr 172.21.15.16:0/647295190 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T16:06:08.092 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.087+0000 7f26877fe640 1 --2- 172.21.15.16:0/647295190 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2688158a40 0x7f268806ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:08.092 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.087+0000 7f26867fc640 1 -- 172.21.15.16:0/647295190 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2678390672 0 0) 0x7f268806f140 con 0x7f2688158e10 2023-05-22T16:06:08.092 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.087+0000 7f26867fc640 1 -- 172.21.15.16:0/647295190 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2670003530 con 0x7f2688158e10 2023-05-22T16:06:08.092 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.087+0000 7f26867fc640 1 -- 172.21.15.16:0/647295190 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1847826229 0 0) 0x7f2670003530 con 0x7f2688158e10 2023-05-22T16:06:08.092 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.087+0000 7f26867fc640 1 -- 172.21.15.16:0/647295190 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2688158a40 msgr2=0x7f268806ec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:06:08.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.087+0000 7f26867fc640 1 --2- 172.21.15.16:0/647295190 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2688158a40 0x7f268806ec00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:08.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.087+0000 7f26867fc640 1 -- 172.21.15.16:0/647295190 >> v1:172.21.15.62:6789/0 conn(0x7f26880cc300 legacy=0x7f26880ca1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:06:08.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.087+0000 7f26867fc640 1 -- 172.21.15.16:0/647295190 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f268815ccf0 con 0x7f2688158e10 2023-05-22T16:06:08.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.087+0000 7f26867fc640 1 -- 172.21.15.16:0/647295190 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7f267c001680 con 0x7f2688158e10 2023-05-22T16:06:08.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.087+0000 7f26867fc640 1 -- 172.21.15.16:0/647295190 >> v1:172.21.15.16:6789/0 conn(0x7f2688158e10 legacy=0x7f26880cbbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:06:08.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.087+0000 7f26867fc640 1 --2- 172.21.15.16:0/647295190 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2670003a70 0x7f2670003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:08.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.087+0000 7f26867fc640 1 --2- 172.21.15.16:0/647295190 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f26700047d0 0x7f2670004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:08.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.087+0000 7f26867fc640 1 --2- 172.21.15.16:0/647295190 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2670005380 0x7f267000b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:08.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.087+0000 7f26867fc640 1 -- 172.21.15.16:0/647295190 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f268815ca10 con 0x7f26700047d0 2023-05-22T16:06:08.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.087+0000 7f26867fc640 1 -- 172.21.15.16:0/647295190 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f267000c180 con 0x7f2670003a70 2023-05-22T16:06:08.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.087+0000 7f26867fc640 1 -- 172.21.15.16:0/647295190 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f267000c340 con 0x7f2670005380 2023-05-22T16:06:08.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.087+0000 7f26877fe640 1 --2- 172.21.15.16:0/647295190 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2688158a40 0x7f268806ec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T16:06:08.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.087+0000 7f2686ffd640 1 --2- 172.21.15.16:0/647295190 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f26700047d0 0x7f2670004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:08.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.087+0000 7f2687fff640 1 --2- 172.21.15.16:0/647295190 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2670005380 0x7f267000b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:08.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.087+0000 7f2686ffd640 1 -- 172.21.15.16:0/647295190 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2670005380 msgr2=0x7f267000b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:06:08.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.087+0000 7f2686ffd640 1 --2- 172.21.15.16:0/647295190 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2670005380 0x7f267000b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:08.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.087+0000 7f2686ffd640 1 -- 172.21.15.16:0/647295190 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2670003a70 msgr2=0x7f2670003e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:06:08.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.087+0000 7f2686ffd640 1 --2- 172.21.15.16:0/647295190 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2670003a70 0x7f2670003e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:08.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.087+0000 7f2686ffd640 1 -- 172.21.15.16:0/647295190 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2670003530 con 0x7f26700047d0 2023-05-22T16:06:08.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.087+0000 7f2687fff640 1 --2- 172.21.15.16:0/647295190 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2670005380 0x7f267000b990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T16:06:08.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.087+0000 7f2686ffd640 1 --2- 172.21.15.16:0/647295190 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f26700047d0 0x7f2670004c40 secure :-1 s=READY pgs=2631 cs=0 l=1 rev1=1 crypto rx=0x7f267c00c210 tx=0x7f267c01cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee4e43676f2138ff server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:06:08.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.087+0000 7f26867fc640 1 -- 172.21.15.16:0/647295190 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f267c005ae0 con 0x7f26700047d0 2023-05-22T16:06:08.096 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.087+0000 7f26867fc640 1 -- 172.21.15.16:0/647295190 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f267c005c40 con 0x7f26700047d0 2023-05-22T16:06:08.096 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.087+0000 7f26867fc640 1 -- 172.21.15.16:0/647295190 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f267c0a57e0 con 0x7f26700047d0 2023-05-22T16:06:08.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.091+0000 7f268dac4640 1 -- 172.21.15.16:0/647295190 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f26700047d0 msgr2=0x7f2670004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:06:08.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.091+0000 7f268dac4640 1 --2- 172.21.15.16:0/647295190 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f26700047d0 0x7f2670004c40 secure :-1 s=READY pgs=2631 cs=0 l=1 rev1=1 crypto rx=0x7f267c00c210 tx=0x7f267c01cbf0 comp rx=0 tx=0).stop 2023-05-22T16:06:08.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.091+0000 7f2687fff640 1 -- 172.21.15.16:0/647295190 reap_dead start 2023-05-22T16:06:08.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.091+0000 7f268dac4640 1 -- 172.21.15.16:0/647295190 shutdown_connections 2023-05-22T16:06:08.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.091+0000 7f268dac4640 1 -- 172.21.15.16:0/647295190 >> 172.21.15.16:0/647295190 conn(0x7f26880b0d50 msgr2=0x7f26880b1150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:06:08.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.091+0000 7f268dac4640 1 -- 172.21.15.16:0/647295190 shutdown_connections 2023-05-22T16:06:08.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.091+0000 7f268dac4640 1 -- 172.21.15.16:0/647295190 wait complete. 2023-05-22T16:06:08.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.091+0000 7f268dac4640 1 Processor -- start 2023-05-22T16:06:08.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.091+0000 7f268dac4640 1 -- start start 2023-05-22T16:06:08.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.091+0000 7f268dac4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f26700047d0 0x7f268814d720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:08.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.091+0000 7f268dac4640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f26881f4d20 0x7f26880accd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:08.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.091+0000 7f268dac4640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f26880ad210 0x7f26880c8300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:08.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.091+0000 7f268dac4640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26880ae3a0 con 0x7f26700047d0 2023-05-22T16:06:08.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.091+0000 7f268dac4640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f268814dc60 con 0x7f26881f4d20 2023-05-22T16:06:08.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.091+0000 7f268dac4640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26880ad720 con 0x7f26880ad210 2023-05-22T16:06:08.103 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.091+0000 7f26877fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f26700047d0 0x7f268814d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:08.103 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.091+0000 7f26877fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f26700047d0 0x7f268814d720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43592/0 (socket says 172.21.15.16:43592) 2023-05-22T16:06:08.103 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.091+0000 7f26877fe640 1 -- 172.21.15.16:0/1706814315 learned_addr learned my addr 172.21.15.16:0/1706814315 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:06:08.103 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.091+0000 7f26877fe640 1 -- 172.21.15.16:0/1706814315 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f26880ad210 msgr2=0x7f26880c8300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:06:08.103 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.091+0000 7f26877fe640 1 --2- 172.21.15.16:0/1706814315 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f26880ad210 0x7f26880c8300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:08.104 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.091+0000 7f26877fe640 1 -- 172.21.15.16:0/1706814315 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f26881f4d20 msgr2=0x7f26880accd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:06:08.104 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.091+0000 7f26877fe640 1 --2- 172.21.15.16:0/1706814315 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f26881f4d20 0x7f26880accd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:08.104 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.091+0000 7f26877fe640 1 -- 172.21.15.16:0/1706814315 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26880ccdf0 con 0x7f26700047d0 2023-05-22T16:06:08.104 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.091+0000 7f26877fe640 1 --2- 172.21.15.16:0/1706814315 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f26700047d0 0x7f268814d720 secure :-1 s=READY pgs=2632 cs=0 l=1 rev1=1 crypto rx=0x7f267800e590 tx=0x7f267800e680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:06:08.104 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.091+0000 7f266ffff640 1 -- 172.21.15.16:0/1706814315 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f267800c070 con 0x7f26700047d0 2023-05-22T16:06:08.105 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.091+0000 7f266ffff640 1 -- 172.21.15.16:0/1706814315 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2678013610 con 0x7f26700047d0 2023-05-22T16:06:08.105 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.091+0000 7f268dac4640 1 -- 172.21.15.16:0/1706814315 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f268815d590 con 0x7f26700047d0 2023-05-22T16:06:08.105 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.091+0000 7f266ffff640 1 -- 172.21.15.16:0/1706814315 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26780139c0 con 0x7f26700047d0 2023-05-22T16:06:08.105 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.091+0000 7f268dac4640 1 -- 172.21.15.16:0/1706814315 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26880cb7a0 con 0x7f26700047d0 2023-05-22T16:06:08.105 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.091+0000 7f268dac4640 1 -- 172.21.15.16:0/1706814315 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f268804dcc0 con 0x7f26700047d0 2023-05-22T16:06:08.105 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.095+0000 7f266ffff640 1 -- 172.21.15.16:0/1706814315 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 100) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f2678013b20 con 0x7f26700047d0 2023-05-22T16:06:08.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.095+0000 7f266ffff640 1 --2- 172.21.15.16:0/1706814315 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f26740654d0 0x7f2674067980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:08.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.095+0000 7f266ffff640 1 -- 172.21.15.16:0/1706814315 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(201..201 src has 1..201) v4 ==== 6317+0+0 (secure 0 0 0) 0x7f2678082fc0 con 0x7f26700047d0 2023-05-22T16:06:08.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.095+0000 7f2686ffd640 1 --2- 172.21.15.16:0/1706814315 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f26740654d0 0x7f2674067980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:08.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.095+0000 7f266ffff640 1 --2- 172.21.15.16:0/1706814315 >> [v2:172.21.15.16:6800/37008,v1:172.21.15.16:6801/37008] conn(0x7f267406c380 0x7f267406e7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:08.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.095+0000 7f266ffff640 1 -- 172.21.15.16:0/1706814315 --> [v2:172.21.15.16:6800/37008,v1:172.21.15.16:6801/37008] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f267406eec0 con 0x7f267406c380 2023-05-22T16:06:08.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.095+0000 7f266ffff640 1 -- 172.21.15.16:0/1706814315 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_get_version_reply(handle=1 version=201) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2678083330 con 0x7f26700047d0 2023-05-22T16:06:08.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.095+0000 7f2687fff640 1 --2- 172.21.15.16:0/1706814315 >> [v2:172.21.15.16:6800/37008,v1:172.21.15.16:6801/37008] conn(0x7f267406c380 0x7f267406e7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:08.107 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.095+0000 7f2686ffd640 1 --2- 172.21.15.16:0/1706814315 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f26740654d0 0x7f2674067980 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f267c0ab000 tx=0x7f267c001a10 comp rx=0 tx=0).ready entity=mgr.8195 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:06:08.107 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.095+0000 7f2687fff640 1 --2- 172.21.15.16:0/1706814315 >> [v2:172.21.15.16:6800/37008,v1:172.21.15.16:6801/37008] conn(0x7f267406c380 0x7f267406e7f0 crc :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:06:08.107 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.095+0000 7f266ffff640 1 -- 172.21.15.16:0/1706814315 <== osd.1 v2:172.21.15.16:6800/37008 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f267406eec0 con 0x7f267406c380 2023-05-22T16:06:08.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.107+0000 7f268dac4640 1 -- 172.21.15.16:0/1706814315 --> [v2:172.21.15.16:6800/37008,v1:172.21.15.16:6801/37008] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f26880b3160 con 0x7f267406c380 2023-05-22T16:06:08.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.107+0000 7f266ffff640 1 -- 172.21.15.16:0/1706814315 <== osd.1 v2:172.21.15.16:6800/37008 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f26880b3160 con 0x7f267406c380 2023-05-22T16:06:08.112 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-05-22T16:06:08.113 INFO:teuthology.orchestra.run.smithi016.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-05-22T16:06:08.113 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-05-22T16:06:08.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.107+0000 7f268dac4640 1 -- 172.21.15.16:0/1706814315 >> [v2:172.21.15.16:6800/37008,v1:172.21.15.16:6801/37008] conn(0x7f267406c380 msgr2=0x7f267406e7f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:06:08.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.107+0000 7f268dac4640 1 --2- 172.21.15.16:0/1706814315 >> [v2:172.21.15.16:6800/37008,v1:172.21.15.16:6801/37008] conn(0x7f267406c380 0x7f267406e7f0 crc :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:08.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.107+0000 7f268dac4640 1 -- 172.21.15.16:0/1706814315 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f26740654d0 msgr2=0x7f2674067980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:06:08.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.107+0000 7f268dac4640 1 --2- 172.21.15.16:0/1706814315 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f26740654d0 0x7f2674067980 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f267c0ab000 tx=0x7f267c001a10 comp rx=0 tx=0).stop 2023-05-22T16:06:08.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.107+0000 7f268dac4640 1 -- 172.21.15.16:0/1706814315 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f26700047d0 msgr2=0x7f268814d720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:06:08.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.107+0000 7f268dac4640 1 --2- 172.21.15.16:0/1706814315 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f26700047d0 0x7f268814d720 secure :-1 s=READY pgs=2632 cs=0 l=1 rev1=1 crypto rx=0x7f267800e590 tx=0x7f267800e680 comp rx=0 tx=0).stop 2023-05-22T16:06:08.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.107+0000 7f2687fff640 1 -- 172.21.15.16:0/1706814315 reap_dead start 2023-05-22T16:06:08.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.111+0000 7f268dac4640 1 -- 172.21.15.16:0/1706814315 shutdown_connections 2023-05-22T16:06:08.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.111+0000 7f268dac4640 1 -- 172.21.15.16:0/1706814315 >> 172.21.15.16:0/1706814315 conn(0x7f26880b0d50 msgr2=0x7f2688070d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:06:08.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.111+0000 7f268dac4640 1 -- 172.21.15.16:0/1706814315 shutdown_connections 2023-05-22T16:06:08.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.111+0000 7f268dac4640 1 -- 172.21.15.16:0/1706814315 wait complete. 2023-05-22T16:06:08.129 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2023-05-22T16:06:08.246 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.243+0000 7ff27605e640 1 Processor -- start 2023-05-22T16:06:08.246 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.243+0000 7ff27605e640 1 -- start start 2023-05-22T16:06:08.247 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.243+0000 7ff27605e640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff2700d07a0 0x7ff27015c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:08.247 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.243+0000 7ff27605e640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff27015cbe0 0x7ff2700b0440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:08.248 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.243+0000 7ff27605e640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2700cd650 con 0x7ff2701565d0 2023-05-22T16:06:08.248 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.243+0000 7ff27605e640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2700ccb10 con 0x7ff2700d07a0 2023-05-22T16:06:08.248 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.243+0000 7ff27605e640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2700ceaf0 con 0x7ff27015cbe0 2023-05-22T16:06:08.248 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.243+0000 7ff26effd640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff2700d07a0 0x7ff27015c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:08.248 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.243+0000 7ff26f7fe640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7ff2701565d0 0x7ff2700d0090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:49856/0 (socket says 172.21.15.16:49856) 2023-05-22T16:06:08.248 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.243+0000 7ff26f7fe640 1 -- 172.21.15.16:0/1851850710 learned_addr learned my addr 172.21.15.16:0/1851850710 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T16:06:08.249 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.243+0000 7ff26ffff640 1 --2- 172.21.15.16:0/1851850710 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff27015cbe0 0x7ff2700b0440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:08.249 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.243+0000 7ff26effd640 1 -- 172.21.15.16:0/1851850710 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff27015cbe0 msgr2=0x7ff2700b0440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:06:08.249 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.243+0000 7ff26effd640 1 --2- 172.21.15.16:0/1851850710 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff27015cbe0 0x7ff2700b0440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:08.249 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.243+0000 7ff26effd640 1 -- 172.21.15.16:0/1851850710 >> v1:172.21.15.16:6789/0 conn(0x7ff2701565d0 legacy=0x7ff2700d0090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:06:08.249 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.243+0000 7ff26effd640 1 -- 172.21.15.16:0/1851850710 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2700ccd20 con 0x7ff2700d07a0 2023-05-22T16:06:08.251 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.247+0000 7ff26effd640 1 --2- 172.21.15.16:0/1851850710 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff2700d07a0 0x7ff27015c6a0 secure :-1 s=READY pgs=2634 cs=0 l=1 rev1=1 crypto rx=0x7ff2600097c0 tx=0x7ff26001c2e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8cf81dd3b6fe35c7 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:06:08.251 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.247+0000 7ff26e7fc640 1 -- 172.21.15.16:0/1851850710 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2600ab600 con 0x7ff2700d07a0 2023-05-22T16:06:08.251 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.247+0000 7ff26e7fc640 1 -- 172.21.15.16:0/1851850710 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2600a7050 con 0x7ff2700d07a0 2023-05-22T16:06:08.251 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.247+0000 7ff26e7fc640 1 -- 172.21.15.16:0/1851850710 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2600ab980 con 0x7ff2700d07a0 2023-05-22T16:06:08.251 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.247+0000 7ff27605e640 1 -- 172.21.15.16:0/1851850710 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff2700d07a0 msgr2=0x7ff27015c6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:06:08.252 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.247+0000 7ff27605e640 1 --2- 172.21.15.16:0/1851850710 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff2700d07a0 0x7ff27015c6a0 secure :-1 s=READY pgs=2634 cs=0 l=1 rev1=1 crypto rx=0x7ff2600097c0 tx=0x7ff26001c2e0 comp rx=0 tx=0).stop 2023-05-22T16:06:08.252 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.247+0000 7ff27605e640 1 -- 172.21.15.16:0/1851850710 shutdown_connections 2023-05-22T16:06:08.252 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.247+0000 7ff27605e640 1 --2- 172.21.15.16:0/1851850710 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff27015cbe0 0x7ff2700b0440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:08.252 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.247+0000 7ff27605e640 1 --2- 172.21.15.16:0/1851850710 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff2700d07a0 0x7ff27015c6a0 unknown :-1 s=CLOSED pgs=2634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:08.252 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.247+0000 7ff27605e640 1 -- 172.21.15.16:0/1851850710 >> 172.21.15.16:0/1851850710 conn(0x7ff270071850 msgr2=0x7ff27009b830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:06:08.252 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.247+0000 7ff27605e640 1 -- 172.21.15.16:0/1851850710 shutdown_connections 2023-05-22T16:06:08.253 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.247+0000 7ff27605e640 1 -- 172.21.15.16:0/1851850710 wait complete. 2023-05-22T16:06:08.254 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.247+0000 7ff27605e640 1 Processor -- start 2023-05-22T16:06:08.254 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.247+0000 7ff27605e640 1 -- start start 2023-05-22T16:06:08.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.247+0000 7ff27605e640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff2700d07a0 0x7ff2700b52f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:08.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.247+0000 7ff27605e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff2701565d0 0x7ff2700b70a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:08.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.247+0000 7ff27605e640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff27015cbe0 0x7ff27014bad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:08.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.247+0000 7ff27605e640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2700b5830 con 0x7ff2701565d0 2023-05-22T16:06:08.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.247+0000 7ff27605e640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2700b5b00 con 0x7ff27015cbe0 2023-05-22T16:06:08.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.251+0000 7ff27605e640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2700b5de0 con 0x7ff2700d07a0 2023-05-22T16:06:08.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.251+0000 7ff26effd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff2701565d0 0x7ff2700b70a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:08.256 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.251+0000 7ff26effd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff2701565d0 0x7ff2700b70a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43608/0 (socket says 172.21.15.16:43608) 2023-05-22T16:06:08.256 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.251+0000 7ff26effd640 1 -- 172.21.15.16:0/257343862 learned_addr learned my addr 172.21.15.16:0/257343862 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:06:08.256 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.251+0000 7ff26f7fe640 1 --2- 172.21.15.16:0/257343862 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff2700d07a0 0x7ff2700b52f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:08.256 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.251+0000 7ff26ffff640 1 --2- 172.21.15.16:0/257343862 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff27015cbe0 0x7ff27014bad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:08.256 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.251+0000 7ff26effd640 1 -- 172.21.15.16:0/257343862 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff2700d07a0 msgr2=0x7ff2700b52f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:06:08.256 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.251+0000 7ff26effd640 1 --2- 172.21.15.16:0/257343862 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff2700d07a0 0x7ff2700b52f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:08.256 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.251+0000 7ff26effd640 1 -- 172.21.15.16:0/257343862 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff27015cbe0 msgr2=0x7ff27014bad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:06:08.257 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.251+0000 7ff26effd640 1 --2- 172.21.15.16:0/257343862 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff27015cbe0 0x7ff27014bad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:08.257 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.251+0000 7ff26effd640 1 -- 172.21.15.16:0/257343862 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff260009420 con 0x7ff2701565d0 2023-05-22T16:06:08.257 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.251+0000 7ff26f7fe640 1 --2- 172.21.15.16:0/257343862 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff2700d07a0 0x7ff2700b52f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T16:06:08.257 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.251+0000 7ff26ffff640 1 --2- 172.21.15.16:0/257343862 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff27015cbe0 0x7ff27014bad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T16:06:08.257 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.251+0000 7ff26effd640 1 --2- 172.21.15.16:0/257343862 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff2701565d0 0x7ff2700b70a0 secure :-1 s=READY pgs=2635 cs=0 l=1 rev1=1 crypto rx=0x7ff26001a990 tx=0x7ff260019080 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:06:08.258 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.251+0000 7ff257fff640 1 -- 172.21.15.16:0/257343862 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2600b4610 con 0x7ff2701565d0 2023-05-22T16:06:08.258 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.251+0000 7ff257fff640 1 -- 172.21.15.16:0/257343862 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2600abe90 con 0x7ff2701565d0 2023-05-22T16:06:08.258 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.251+0000 7ff257fff640 1 -- 172.21.15.16:0/257343862 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2600b49d0 con 0x7ff2701565d0 2023-05-22T16:06:08.258 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.251+0000 7ff27605e640 1 -- 172.21.15.16:0/257343862 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2700af220 con 0x7ff2701565d0 2023-05-22T16:06:08.258 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.251+0000 7ff27605e640 1 -- 172.21.15.16:0/257343862 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2700af090 con 0x7ff2701565d0 2023-05-22T16:06:08.259 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.251+0000 7ff27605e640 1 -- 172.21.15.16:0/257343862 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff27015cfe0 con 0x7ff2701565d0 2023-05-22T16:06:08.259 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.251+0000 7ff257fff640 1 -- 172.21.15.16:0/257343862 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 100) v1 ==== 84332+0+0 (secure 0 0 0) 0x7ff2600b2040 con 0x7ff2701565d0 2023-05-22T16:06:08.259 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.251+0000 7ff257fff640 1 --2- 172.21.15.16:0/257343862 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7ff25c0654a0 0x7ff25c067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:08.259 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.251+0000 7ff257fff640 1 -- 172.21.15.16:0/257343862 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(201..201 src has 1..201) v4 ==== 6317+0+0 (secure 0 0 0) 0x7ff26011cbc0 con 0x7ff2701565d0 2023-05-22T16:06:08.260 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.251+0000 7ff26f7fe640 1 --2- 172.21.15.16:0/257343862 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7ff25c0654a0 0x7ff25c067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:08.260 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.255+0000 7ff26f7fe640 1 --2- 172.21.15.16:0/257343862 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7ff25c0654a0 0x7ff25c067950 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7ff264000f00 tx=0x7ff264009040 comp rx=0 tx=0).ready entity=mgr.8195 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:06:08.260 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.255+0000 7ff257fff640 1 -- 172.21.15.16:0/257343862 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7ff2600ea520 con 0x7ff2701565d0 2023-05-22T16:06:08.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.407+0000 7ff27605e640 1 -- 172.21.15.16:0/257343862 --> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) v1 -- 0x7ff270073830 con 0x7ff25c0654a0 2023-05-22T16:06:08.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.407+0000 7ff257fff640 1 -- 172.21.15.16:0/257343862 <== mgr.8195 v2:172.21.15.16:6832/65890 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7ff270073830 con 0x7ff25c0654a0 2023-05-22T16:06:08.412 INFO:teuthology.orchestra.run.smithi016.stderr:instructed osd(s) 1 to deep-scrub 2023-05-22T16:06:08.413 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.411+0000 7ff27605e640 1 -- 172.21.15.16:0/257343862 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7ff25c0654a0 msgr2=0x7ff25c067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:06:08.413 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.411+0000 7ff27605e640 1 --2- 172.21.15.16:0/257343862 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7ff25c0654a0 0x7ff25c067950 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7ff264000f00 tx=0x7ff264009040 comp rx=0 tx=0).stop 2023-05-22T16:06:08.414 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.411+0000 7ff27605e640 1 -- 172.21.15.16:0/257343862 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff2701565d0 msgr2=0x7ff2700b70a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:06:08.414 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.411+0000 7ff27605e640 1 --2- 172.21.15.16:0/257343862 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff2701565d0 0x7ff2700b70a0 secure :-1 s=READY pgs=2635 cs=0 l=1 rev1=1 crypto rx=0x7ff26001a990 tx=0x7ff260019080 comp rx=0 tx=0).stop 2023-05-22T16:06:08.414 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.411+0000 7ff27605e640 1 -- 172.21.15.16:0/257343862 shutdown_connections 2023-05-22T16:06:08.414 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.411+0000 7ff27605e640 1 --2- 172.21.15.16:0/257343862 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff27015cbe0 0x7ff27014bad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:08.414 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.411+0000 7ff27605e640 1 --2- 172.21.15.16:0/257343862 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7ff25c0654a0 0x7ff25c067950 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:08.415 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.411+0000 7ff27605e640 1 --2- 172.21.15.16:0/257343862 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff2701565d0 0x7ff2700b70a0 unknown :-1 s=CLOSED pgs=2635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:08.415 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.411+0000 7ff27605e640 1 --2- 172.21.15.16:0/257343862 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff2700d07a0 0x7ff2700b52f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:08.415 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.411+0000 7ff27605e640 1 -- 172.21.15.16:0/257343862 >> 172.21.15.16:0/257343862 conn(0x7ff270071850 msgr2=0x7ff27015bc10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:06:08.415 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.411+0000 7ff27605e640 1 -- 172.21.15.16:0/257343862 shutdown_connections 2023-05-22T16:06:08.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.411+0000 7ff27605e640 1 -- 172.21.15.16:0/257343862 wait complete. 2023-05-22T16:06:08.429 INFO:tasks.ceph:Scrubbing osd.2 2023-05-22T16:06:08.429 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2023-05-22T16:06:08.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.543+0000 7fcf98478640 1 Processor -- start 2023-05-22T16:06:08.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.543+0000 7fcf98478640 1 -- start start 2023-05-22T16:06:08.549 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.543+0000 7fcf98478640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fcf90158a40 0x7fcf900ce040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:08.550 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.543+0000 7fcf98478640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf9015ccf0 con 0x7fcf900ce580 2023-05-22T16:06:08.550 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.543+0000 7fcf98478640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf90159bd0 con 0x7fcf9015d180 2023-05-22T16:06:08.550 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.543+0000 7fcf98478640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf90159eb0 con 0x7fcf90158a40 2023-05-22T16:06:08.550 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.543+0000 7fcf96a15640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fcf900ce580 0x7fcf900c8ae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:49862/0 (socket says 172.21.15.16:49862) 2023-05-22T16:06:08.550 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.543+0000 7fcf96a15640 1 -- 172.21.15.16:0/2423122208 learned_addr learned my addr 172.21.15.16:0/2423122208 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T16:06:08.550 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.543+0000 7fcf96214640 1 --2- 172.21.15.16:0/2423122208 >> v2:172.21.15.62:3300/0 conn(0x7fcf90158a40 0x7fcf900ce040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:08.550 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.547+0000 7fcf95212640 1 -- 172.21.15.16:0/2423122208 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3280562020 0 0) 0x7fcf9015ccf0 con 0x7fcf900ce580 2023-05-22T16:06:08.550 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.547+0000 7fcf95212640 1 -- 172.21.15.16:0/2423122208 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcf7c003530 con 0x7fcf900ce580 2023-05-22T16:06:08.551 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.547+0000 7fcf95212640 1 -- 172.21.15.16:0/2423122208 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 810078616 0 0) 0x7fcf7c003530 con 0x7fcf900ce580 2023-05-22T16:06:08.551 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.547+0000 7fcf95212640 1 -- 172.21.15.16:0/2423122208 >> v2:172.21.15.62:3300/0 conn(0x7fcf90158a40 msgr2=0x7fcf900ce040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:06:08.551 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.547+0000 7fcf95212640 1 --2- 172.21.15.16:0/2423122208 >> v2:172.21.15.62:3300/0 conn(0x7fcf90158a40 0x7fcf900ce040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:08.551 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.547+0000 7fcf95212640 1 -- 172.21.15.16:0/2423122208 >> v1:172.21.15.62:6789/0 conn(0x7fcf9015d180 legacy=0x7fcf90158e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:06:08.551 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.547+0000 7fcf95212640 1 -- 172.21.15.16:0/2423122208 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf9015c6a0 con 0x7fcf900ce580 2023-05-22T16:06:08.551 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.547+0000 7fcf95212640 1 -- 172.21.15.16:0/2423122208 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7fcf88001a00 con 0x7fcf900ce580 2023-05-22T16:06:08.551 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.547+0000 7fcf95212640 1 -- 172.21.15.16:0/2423122208 >> v1:172.21.15.16:6789/0 conn(0x7fcf900ce580 legacy=0x7fcf900c8ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:06:08.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.547+0000 7fcf95212640 1 --2- 172.21.15.16:0/2423122208 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fcf7c003a90 0x7fcf7c003ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:08.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.547+0000 7fcf95212640 1 --2- 172.21.15.16:0/2423122208 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcf7c0047d0 0x7fcf7c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:08.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.547+0000 7fcf95212640 1 --2- 172.21.15.16:0/2423122208 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcf7c0051d0 0x7fcf7c00b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:08.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.547+0000 7fcf95212640 1 -- 172.21.15.16:0/2423122208 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf90159eb0 con 0x7fcf7c0051d0 2023-05-22T16:06:08.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.547+0000 7fcf95212640 1 -- 172.21.15.16:0/2423122208 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf7c00c110 con 0x7fcf7c0047d0 2023-05-22T16:06:08.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.547+0000 7fcf95212640 1 -- 172.21.15.16:0/2423122208 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf7c00c2d0 con 0x7fcf7c003a90 2023-05-22T16:06:08.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.547+0000 7fcf96a15640 1 --2- 172.21.15.16:0/2423122208 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcf7c0051d0 0x7fcf7c00b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:08.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.547+0000 7fcf96214640 1 --2- 172.21.15.16:0/2423122208 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fcf7c003a90 0x7fcf7c003ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:08.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.547+0000 7fcf95a13640 1 --2- 172.21.15.16:0/2423122208 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcf7c0047d0 0x7fcf7c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:08.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.547+0000 7fcf96a15640 1 -- 172.21.15.16:0/2423122208 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fcf7c003a90 msgr2=0x7fcf7c003ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:06:08.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.547+0000 7fcf96a15640 1 --2- 172.21.15.16:0/2423122208 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fcf7c003a90 0x7fcf7c003ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:08.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.547+0000 7fcf96a15640 1 -- 172.21.15.16:0/2423122208 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcf7c0047d0 msgr2=0x7fcf7c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:06:08.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.547+0000 7fcf96a15640 1 --2- 172.21.15.16:0/2423122208 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcf7c0047d0 0x7fcf7c004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:08.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.547+0000 7fcf96a15640 1 -- 172.21.15.16:0/2423122208 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf7c003530 con 0x7fcf7c0051d0 2023-05-22T16:06:08.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.547+0000 7fcf96214640 1 --2- 172.21.15.16:0/2423122208 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fcf7c003a90 0x7fcf7c003ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T16:06:08.555 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.547+0000 7fcf96a15640 1 -- 172.21.15.16:0/2423122208 reap_dead start 2023-05-22T16:06:08.556 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.547+0000 7fcf96a15640 1 --2- 172.21.15.16:0/2423122208 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcf7c0051d0 0x7fcf7c00b990 secure :-1 s=READY pgs=2637 cs=0 l=1 rev1=1 crypto rx=0x7fcf88002240 tx=0x7fcf8801d240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a71c7cb9e211b2b9 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:06:08.556 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.551+0000 7fcf95212640 1 -- 172.21.15.16:0/2423122208 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf880ad630 con 0x7fcf7c0051d0 2023-05-22T16:06:08.556 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.551+0000 7fcf95212640 1 -- 172.21.15.16:0/2423122208 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf880a8070 con 0x7fcf7c0051d0 2023-05-22T16:06:08.556 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.551+0000 7fcf95212640 1 -- 172.21.15.16:0/2423122208 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf880ad9d0 con 0x7fcf7c0051d0 2023-05-22T16:06:08.556 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.551+0000 7fcf98478640 1 -- 172.21.15.16:0/2423122208 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcf7c0051d0 msgr2=0x7fcf7c00b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:06:08.556 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.551+0000 7fcf98478640 1 --2- 172.21.15.16:0/2423122208 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcf7c0051d0 0x7fcf7c00b990 secure :-1 s=READY pgs=2637 cs=0 l=1 rev1=1 crypto rx=0x7fcf88002240 tx=0x7fcf8801d240 comp rx=0 tx=0).stop 2023-05-22T16:06:08.557 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.551+0000 7fcf98478640 1 -- 172.21.15.16:0/2423122208 shutdown_connections 2023-05-22T16:06:08.557 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.551+0000 7fcf98478640 1 --2- 172.21.15.16:0/2423122208 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcf7c0051d0 0x7fcf7c00b990 unknown :-1 s=CLOSED pgs=2637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:08.557 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.551+0000 7fcf98478640 1 -- 172.21.15.16:0/2423122208 >> 172.21.15.16:0/2423122208 conn(0x7fcf90069800 msgr2=0x7fcf90069c00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:06:08.557 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.551+0000 7fcf98478640 1 -- 172.21.15.16:0/2423122208 shutdown_connections 2023-05-22T16:06:08.557 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.551+0000 7fcf98478640 1 -- 172.21.15.16:0/2423122208 wait complete. 2023-05-22T16:06:08.557 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.551+0000 7fcf98478640 1 Processor -- start 2023-05-22T16:06:08.557 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.551+0000 7fcf98478640 1 -- start start 2023-05-22T16:06:08.558 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.551+0000 7fcf98478640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcf7c0051d0 0x7fcf90148b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:08.558 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.551+0000 7fcf98478640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fcf900c5890 0x7fcf900cd080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:08.558 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.551+0000 7fcf98478640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcf900c5d00 0x7fcf900c6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:08.558 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.551+0000 7fcf98478640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf900c66c0 con 0x7fcf900c5d00 2023-05-22T16:06:08.558 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.551+0000 7fcf98478640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf900c25f0 con 0x7fcf7c0051d0 2023-05-22T16:06:08.559 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.551+0000 7fcf98478640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf900c2880 con 0x7fcf900c5890 2023-05-22T16:06:08.559 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.551+0000 7fcf96a15640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcf900c5d00 0x7fcf900c6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:08.559 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.551+0000 7fcf96a15640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcf900c5d00 0x7fcf900c6180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43632/0 (socket says 172.21.15.16:43632) 2023-05-22T16:06:08.559 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.551+0000 7fcf96a15640 1 -- 172.21.15.16:0/868749626 learned_addr learned my addr 172.21.15.16:0/868749626 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:06:08.559 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.551+0000 7fcf96214640 1 --2- 172.21.15.16:0/868749626 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcf7c0051d0 0x7fcf90148b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:08.559 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.551+0000 7fcf95a13640 1 --2- 172.21.15.16:0/868749626 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fcf900c5890 0x7fcf900cd080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:08.559 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.551+0000 7fcf96a15640 1 -- 172.21.15.16:0/868749626 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fcf900c5890 msgr2=0x7fcf900cd080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:06:08.560 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.551+0000 7fcf96a15640 1 --2- 172.21.15.16:0/868749626 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fcf900c5890 0x7fcf900cd080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:08.560 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.551+0000 7fcf96a15640 1 -- 172.21.15.16:0/868749626 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcf7c0051d0 msgr2=0x7fcf90148b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:06:08.560 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.551+0000 7fcf96a15640 1 --2- 172.21.15.16:0/868749626 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcf7c0051d0 0x7fcf90148b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:08.560 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.551+0000 7fcf96a15640 1 -- 172.21.15.16:0/868749626 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf900ce040 con 0x7fcf900c5d00 2023-05-22T16:06:08.560 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.551+0000 7fcf96214640 1 --2- 172.21.15.16:0/868749626 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcf7c0051d0 0x7fcf90148b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T16:06:08.560 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.551+0000 7fcf95a13640 1 --2- 172.21.15.16:0/868749626 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fcf900c5890 0x7fcf900cd080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T16:06:08.561 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.551+0000 7fcf96a15640 1 --2- 172.21.15.16:0/868749626 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcf900c5d00 0x7fcf900c6180 secure :-1 s=READY pgs=2638 cs=0 l=1 rev1=1 crypto rx=0x7fcf8801dae0 tx=0x7fcf7c005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:06:08.561 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.551+0000 7fcf86ffd640 1 -- 172.21.15.16:0/868749626 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf880ad630 con 0x7fcf900c5d00 2023-05-22T16:06:08.561 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.551+0000 7fcf86ffd640 1 -- 172.21.15.16:0/868749626 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf880ad790 con 0x7fcf900c5d00 2023-05-22T16:06:08.561 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.551+0000 7fcf98478640 1 -- 172.21.15.16:0/868749626 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf9015c980 con 0x7fcf900c5d00 2023-05-22T16:06:08.561 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.551+0000 7fcf86ffd640 1 -- 172.21.15.16:0/868749626 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf880adb00 con 0x7fcf900c5d00 2023-05-22T16:06:08.561 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.551+0000 7fcf98478640 1 -- 172.21.15.16:0/868749626 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf900ce1d0 con 0x7fcf900c5d00 2023-05-22T16:06:08.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.555+0000 7fcf98478640 1 -- 172.21.15.16:0/868749626 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fcf9004dcc0 con 0x7fcf900c5d00 2023-05-22T16:06:08.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.555+0000 7fcf86ffd640 1 -- 172.21.15.16:0/868749626 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 100) v1 ==== 84332+0+0 (secure 0 0 0) 0x7fcf880aa030 con 0x7fcf900c5d00 2023-05-22T16:06:08.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.555+0000 7fcf86ffd640 1 --2- 172.21.15.16:0/868749626 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fcf740654a0 0x7fcf74067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:08.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.555+0000 7fcf86ffd640 1 -- 172.21.15.16:0/868749626 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(201..201 src has 1..201) v4 ==== 6317+0+0 (secure 0 0 0) 0x7fcf8811d010 con 0x7fcf900c5d00 2023-05-22T16:06:08.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.555+0000 7fcf96214640 1 --2- 172.21.15.16:0/868749626 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fcf740654a0 0x7fcf74067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:08.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.555+0000 7fcf86ffd640 1 --2- 172.21.15.16:0/868749626 >> [v2:172.21.15.16:6808/37009,v1:172.21.15.16:6809/37009] conn(0x7fcf7406c350 0x7fcf7406e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:08.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.555+0000 7fcf86ffd640 1 -- 172.21.15.16:0/868749626 --> [v2:172.21.15.16:6808/37009,v1:172.21.15.16:6809/37009] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fcf7406ee90 con 0x7fcf7406c350 2023-05-22T16:06:08.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.555+0000 7fcf86ffd640 1 -- 172.21.15.16:0/868749626 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_get_version_reply(handle=1 version=201) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcf8811d460 con 0x7fcf900c5d00 2023-05-22T16:06:08.565 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.555+0000 7fcf95a13640 1 --2- 172.21.15.16:0/868749626 >> [v2:172.21.15.16:6808/37009,v1:172.21.15.16:6809/37009] conn(0x7fcf7406c350 0x7fcf7406e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:08.565 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.555+0000 7fcf96214640 1 --2- 172.21.15.16:0/868749626 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fcf740654a0 0x7fcf74067950 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fcf8c006f90 tx=0x7fcf8c008040 comp rx=0 tx=0).ready entity=mgr.8195 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:06:08.565 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.555+0000 7fcf95a13640 1 --2- 172.21.15.16:0/868749626 >> [v2:172.21.15.16:6808/37009,v1:172.21.15.16:6809/37009] conn(0x7fcf7406c350 0x7fcf7406e7c0 crc :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:06:08.565 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.555+0000 7fcf86ffd640 1 -- 172.21.15.16:0/868749626 <== osd.2 v2:172.21.15.16:6808/37009 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7fcf7406ee90 con 0x7fcf7406c350 2023-05-22T16:06:08.571 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.567+0000 7fcf98478640 1 -- 172.21.15.16:0/868749626 --> [v2:172.21.15.16:6808/37009,v1:172.21.15.16:6809/37009] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fcf900c66c0 con 0x7fcf7406c350 2023-05-22T16:06:08.571 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.567+0000 7fcf86ffd640 1 -- 172.21.15.16:0/868749626 <== osd.2 v2:172.21.15.16:6808/37009 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7fcf900c66c0 con 0x7fcf7406c350 2023-05-22T16:06:08.572 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-05-22T16:06:08.572 INFO:teuthology.orchestra.run.smithi016.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-05-22T16:06:08.572 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-05-22T16:06:08.573 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.567+0000 7fcf98478640 1 -- 172.21.15.16:0/868749626 >> [v2:172.21.15.16:6808/37009,v1:172.21.15.16:6809/37009] conn(0x7fcf7406c350 msgr2=0x7fcf7406e7c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:06:08.573 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.567+0000 7fcf98478640 1 --2- 172.21.15.16:0/868749626 >> [v2:172.21.15.16:6808/37009,v1:172.21.15.16:6809/37009] conn(0x7fcf7406c350 0x7fcf7406e7c0 crc :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:08.573 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.567+0000 7fcf98478640 1 -- 172.21.15.16:0/868749626 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fcf740654a0 msgr2=0x7fcf74067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:06:08.573 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.567+0000 7fcf98478640 1 --2- 172.21.15.16:0/868749626 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fcf740654a0 0x7fcf74067950 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fcf8c006f90 tx=0x7fcf8c008040 comp rx=0 tx=0).stop 2023-05-22T16:06:08.574 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.567+0000 7fcf98478640 1 -- 172.21.15.16:0/868749626 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcf900c5d00 msgr2=0x7fcf900c6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:06:08.574 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.567+0000 7fcf98478640 1 --2- 172.21.15.16:0/868749626 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcf900c5d00 0x7fcf900c6180 secure :-1 s=READY pgs=2638 cs=0 l=1 rev1=1 crypto rx=0x7fcf8801dae0 tx=0x7fcf7c005800 comp rx=0 tx=0).stop 2023-05-22T16:06:08.574 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.567+0000 7fcf96a15640 1 -- 172.21.15.16:0/868749626 reap_dead start 2023-05-22T16:06:08.574 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.567+0000 7fcf98478640 1 -- 172.21.15.16:0/868749626 shutdown_connections 2023-05-22T16:06:08.574 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.567+0000 7fcf98478640 1 -- 172.21.15.16:0/868749626 >> 172.21.15.16:0/868749626 conn(0x7fcf90069800 msgr2=0x7fcf9015ab50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:06:08.574 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.567+0000 7fcf98478640 1 -- 172.21.15.16:0/868749626 shutdown_connections 2023-05-22T16:06:08.574 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.567+0000 7fcf98478640 1 -- 172.21.15.16:0/868749626 wait complete. 2023-05-22T16:06:08.586 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2023-05-22T16:06:08.697 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.695+0000 7fd991789640 1 Processor -- start 2023-05-22T16:06:08.697 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.695+0000 7fd991789640 1 -- start start 2023-05-22T16:06:08.698 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.695+0000 7fd991789640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fd98c0c8f80 0x7fd98c1630b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:08.698 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.695+0000 7fd991789640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd98c158610 con 0x7fd98c0cd790 2023-05-22T16:06:08.698 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.695+0000 7fd991789640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd98c0cc9e0 con 0x7fd98c158240 2023-05-22T16:06:08.698 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.695+0000 7fd991789640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd98c06c7e0 con 0x7fd98c0c8f80 2023-05-22T16:06:08.698 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.695+0000 7fd98a7fc640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fd98c0cd790 0x7fd98c0c8870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:49878/0 (socket says 172.21.15.16:49878) 2023-05-22T16:06:08.699 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.695+0000 7fd98a7fc640 1 -- 172.21.15.16:0/1157350395 learned_addr learned my addr 172.21.15.16:0/1157350395 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T16:06:08.700 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.695+0000 7fd989ffb640 1 -- 172.21.15.16:0/1157350395 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3074727878 0 0) 0x7fd98c158610 con 0x7fd98c0cd790 2023-05-22T16:06:08.700 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.695+0000 7fd98b7fe640 1 --2- 172.21.15.16:0/1157350395 >> v2:172.21.15.62:3300/0 conn(0x7fd98c0c8f80 0x7fd98c1630b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:08.700 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.695+0000 7fd989ffb640 1 -- 172.21.15.16:0/1157350395 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd978003530 con 0x7fd98c0cd790 2023-05-22T16:06:08.701 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.695+0000 7fd989ffb640 1 -- 172.21.15.16:0/1157350395 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2075946266 0 0) 0x7fd978003530 con 0x7fd98c0cd790 2023-05-22T16:06:08.701 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.695+0000 7fd989ffb640 1 -- 172.21.15.16:0/1157350395 >> v2:172.21.15.62:3300/0 conn(0x7fd98c0c8f80 msgr2=0x7fd98c1630b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:06:08.701 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.695+0000 7fd989ffb640 1 --2- 172.21.15.16:0/1157350395 >> v2:172.21.15.62:3300/0 conn(0x7fd98c0c8f80 0x7fd98c1630b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:08.701 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.695+0000 7fd989ffb640 1 -- 172.21.15.16:0/1157350395 >> v1:172.21.15.62:6789/0 conn(0x7fd98c158240 legacy=0x7fd98c1591b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:06:08.701 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.695+0000 7fd989ffb640 1 -- 172.21.15.16:0/1157350395 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd98c0cbdd0 con 0x7fd98c0cd790 2023-05-22T16:06:08.701 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.695+0000 7fd989ffb640 1 -- 172.21.15.16:0/1157350395 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7fd974001680 con 0x7fd98c0cd790 2023-05-22T16:06:08.701 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.695+0000 7fd989ffb640 1 -- 172.21.15.16:0/1157350395 >> v1:172.21.15.16:6789/0 conn(0x7fd98c0cd790 legacy=0x7fd98c0c8870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:06:08.701 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.695+0000 7fd989ffb640 1 --2- 172.21.15.16:0/1157350395 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd978003980 0x7fd978003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:08.702 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.695+0000 7fd989ffb640 1 --2- 172.21.15.16:0/1157350395 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd9780047d0 0x7fd978004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:08.702 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.695+0000 7fd989ffb640 1 --2- 172.21.15.16:0/1157350395 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd978005180 0x7fd97800b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:08.702 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.695+0000 7fd989ffb640 1 -- 172.21.15.16:0/1157350395 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd98c06c7e0 con 0x7fd9780047d0 2023-05-22T16:06:08.702 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.695+0000 7fd989ffb640 1 -- 172.21.15.16:0/1157350395 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd97800c090 con 0x7fd978005180 2023-05-22T16:06:08.702 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.695+0000 7fd989ffb640 1 -- 172.21.15.16:0/1157350395 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd97800c250 con 0x7fd978003980 2023-05-22T16:06:08.702 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.695+0000 7fd98a7fc640 1 --2- 172.21.15.16:0/1157350395 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd9780047d0 0x7fd978004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:08.702 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.695+0000 7fd98affd640 1 --2- 172.21.15.16:0/1157350395 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd978003980 0x7fd978003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:08.703 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.695+0000 7fd98a7fc640 1 -- 172.21.15.16:0/1157350395 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd978003980 msgr2=0x7fd978003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:06:08.703 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.695+0000 7fd98a7fc640 1 --2- 172.21.15.16:0/1157350395 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd978003980 0x7fd978003d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:08.703 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.695+0000 7fd98a7fc640 1 -- 172.21.15.16:0/1157350395 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd978005180 msgr2=0x7fd97800b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:06:08.703 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.695+0000 7fd98a7fc640 1 --2- 172.21.15.16:0/1157350395 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd978005180 0x7fd97800b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:08.703 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.695+0000 7fd98a7fc640 1 -- 172.21.15.16:0/1157350395 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd978003530 con 0x7fd9780047d0 2023-05-22T16:06:08.703 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.695+0000 7fd98b7fe640 1 -- 172.21.15.16:0/1157350395 reap_dead start 2023-05-22T16:06:08.705 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.699+0000 7fd98a7fc640 1 --2- 172.21.15.16:0/1157350395 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd9780047d0 0x7fd978004c40 secure :-1 s=READY pgs=2640 cs=0 l=1 rev1=1 crypto rx=0x7fd97400c1c0 tx=0x7fd97401cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=449cdc5b1a1b9048 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:06:08.706 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.699+0000 7fd989ffb640 1 -- 172.21.15.16:0/1157350395 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd974005ae0 con 0x7fd9780047d0 2023-05-22T16:06:08.706 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.699+0000 7fd989ffb640 1 -- 172.21.15.16:0/1157350395 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd974005c40 con 0x7fd9780047d0 2023-05-22T16:06:08.706 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.699+0000 7fd989ffb640 1 -- 172.21.15.16:0/1157350395 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9740a59f0 con 0x7fd9780047d0 2023-05-22T16:06:08.706 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.699+0000 7fd991789640 1 -- 172.21.15.16:0/1157350395 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd9780047d0 msgr2=0x7fd978004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:06:08.706 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.699+0000 7fd991789640 1 --2- 172.21.15.16:0/1157350395 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd9780047d0 0x7fd978004c40 secure :-1 s=READY pgs=2640 cs=0 l=1 rev1=1 crypto rx=0x7fd97400c1c0 tx=0x7fd97401cbf0 comp rx=0 tx=0).stop 2023-05-22T16:06:08.706 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.699+0000 7fd991789640 1 -- 172.21.15.16:0/1157350395 shutdown_connections 2023-05-22T16:06:08.707 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.699+0000 7fd991789640 1 --2- 172.21.15.16:0/1157350395 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd9780047d0 0x7fd978004c40 unknown :-1 s=CLOSED pgs=2640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:08.707 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.699+0000 7fd991789640 1 -- 172.21.15.16:0/1157350395 >> 172.21.15.16:0/1157350395 conn(0x7fd98c0cab50 msgr2=0x7fd98c0caf50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:06:08.707 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.699+0000 7fd991789640 1 -- 172.21.15.16:0/1157350395 shutdown_connections 2023-05-22T16:06:08.707 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.699+0000 7fd991789640 1 -- 172.21.15.16:0/1157350395 wait complete. 2023-05-22T16:06:08.707 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.699+0000 7fd991789640 1 Processor -- start 2023-05-22T16:06:08.708 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.699+0000 7fd991789640 1 -- start start 2023-05-22T16:06:08.708 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.699+0000 7fd991789640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd9780047d0 0x7fd98c14ff70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:08.708 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.699+0000 7fd991789640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd98c14e040 0x7fd98c0c3ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:08.708 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.699+0000 7fd991789640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd98c14e4b0 0x7fd98c0c43e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:08.708 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.699+0000 7fd991789640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd98c0c3a80 con 0x7fd9780047d0 2023-05-22T16:06:08.708 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.699+0000 7fd991789640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd98c1504b0 con 0x7fd98c14e4b0 2023-05-22T16:06:08.709 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.699+0000 7fd991789640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd98c0c2e00 con 0x7fd98c14e040 2023-05-22T16:06:08.709 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.699+0000 7fd98affd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd9780047d0 0x7fd98c14ff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:08.709 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.699+0000 7fd98affd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd9780047d0 0x7fd98c14ff70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43660/0 (socket says 172.21.15.16:43660) 2023-05-22T16:06:08.709 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.699+0000 7fd98affd640 1 -- 172.21.15.16:0/1803259226 learned_addr learned my addr 172.21.15.16:0/1803259226 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:06:08.709 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.703+0000 7fd98b7fe640 1 --2- 172.21.15.16:0/1803259226 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd98c14e4b0 0x7fd98c0c43e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:08.709 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.703+0000 7fd98affd640 1 -- 172.21.15.16:0/1803259226 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd98c14e040 msgr2=0x7fd98c0c3ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:06:08.709 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.703+0000 7fd98a7fc640 1 --2- 172.21.15.16:0/1803259226 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd98c14e040 0x7fd98c0c3ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:08.710 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.703+0000 7fd98affd640 1 --2- 172.21.15.16:0/1803259226 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd98c14e040 0x7fd98c0c3ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:08.710 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.703+0000 7fd98affd640 1 -- 172.21.15.16:0/1803259226 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd98c14e4b0 msgr2=0x7fd98c0c43e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:06:08.710 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.703+0000 7fd98affd640 1 --2- 172.21.15.16:0/1803259226 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd98c14e4b0 0x7fd98c0c43e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:08.710 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.703+0000 7fd98affd640 1 -- 172.21.15.16:0/1803259226 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd98c0c8c80 con 0x7fd9780047d0 2023-05-22T16:06:08.710 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.703+0000 7fd98b7fe640 1 --2- 172.21.15.16:0/1803259226 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd98c14e4b0 0x7fd98c0c43e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T16:06:08.711 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.703+0000 7fd98affd640 1 --2- 172.21.15.16:0/1803259226 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd9780047d0 0x7fd98c14ff70 secure :-1 s=READY pgs=2641 cs=0 l=1 rev1=1 crypto rx=0x7fd97c00d4d0 tx=0x7fd97c00d5c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:06:08.711 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.703+0000 7fd9837fe640 1 -- 172.21.15.16:0/1803259226 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd97c00b070 con 0x7fd9780047d0 2023-05-22T16:06:08.712 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.703+0000 7fd9837fe640 1 -- 172.21.15.16:0/1803259226 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd97c0052d0 con 0x7fd9780047d0 2023-05-22T16:06:08.712 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.703+0000 7fd991789640 1 -- 172.21.15.16:0/1803259226 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd98c0cc670 con 0x7fd9780047d0 2023-05-22T16:06:08.712 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.703+0000 7fd9837fe640 1 -- 172.21.15.16:0/1803259226 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd97c003620 con 0x7fd9780047d0 2023-05-22T16:06:08.712 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.703+0000 7fd991789640 1 -- 172.21.15.16:0/1803259226 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd98c1595c0 con 0x7fd9780047d0 2023-05-22T16:06:08.712 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.703+0000 7fd991789640 1 -- 172.21.15.16:0/1803259226 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd98c073830 con 0x7fd9780047d0 2023-05-22T16:06:08.713 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.703+0000 7fd9837fe640 1 -- 172.21.15.16:0/1803259226 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 100) v1 ==== 84332+0+0 (secure 0 0 0) 0x7fd97c003780 con 0x7fd9780047d0 2023-05-22T16:06:08.714 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.703+0000 7fd9837fe640 1 --2- 172.21.15.16:0/1803259226 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fd96c0654a0 0x7fd96c067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:08.714 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.703+0000 7fd9837fe640 1 -- 172.21.15.16:0/1803259226 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(201..201 src has 1..201) v4 ==== 6317+0+0 (secure 0 0 0) 0x7fd97c082050 con 0x7fd9780047d0 2023-05-22T16:06:08.714 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.703+0000 7fd98a7fc640 1 --2- 172.21.15.16:0/1803259226 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fd96c0654a0 0x7fd96c067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:08.714 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.703+0000 7fd98a7fc640 1 --2- 172.21.15.16:0/1803259226 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fd96c0654a0 0x7fd96c067950 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fd9740ab000 tx=0x7fd9740a7070 comp rx=0 tx=0).ready entity=mgr.8195 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:06:08.714 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.707+0000 7fd9837fe640 1 -- 172.21.15.16:0/1803259226 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fd97c04b620 con 0x7fd9780047d0 2023-05-22T16:06:08.864 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.863+0000 7fd991789640 1 -- 172.21.15.16:0/1803259226 --> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) v1 -- 0x7fd98c0b4050 con 0x7fd96c0654a0 2023-05-22T16:06:08.865 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.863+0000 7fd9837fe640 1 -- 172.21.15.16:0/1803259226 <== mgr.8195 v2:172.21.15.16:6832/65890 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fd98c0b4050 con 0x7fd96c0654a0 2023-05-22T16:06:08.865 INFO:teuthology.orchestra.run.smithi016.stderr:instructed osd(s) 2 to deep-scrub 2023-05-22T16:06:08.868 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.863+0000 7fd991789640 1 -- 172.21.15.16:0/1803259226 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fd96c0654a0 msgr2=0x7fd96c067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:06:08.868 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.863+0000 7fd991789640 1 --2- 172.21.15.16:0/1803259226 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fd96c0654a0 0x7fd96c067950 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fd9740ab000 tx=0x7fd9740a7070 comp rx=0 tx=0).stop 2023-05-22T16:06:08.868 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.863+0000 7fd991789640 1 -- 172.21.15.16:0/1803259226 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd9780047d0 msgr2=0x7fd98c14ff70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:06:08.868 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.863+0000 7fd991789640 1 --2- 172.21.15.16:0/1803259226 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd9780047d0 0x7fd98c14ff70 secure :-1 s=READY pgs=2641 cs=0 l=1 rev1=1 crypto rx=0x7fd97c00d4d0 tx=0x7fd97c00d5c0 comp rx=0 tx=0).stop 2023-05-22T16:06:08.868 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.863+0000 7fd991789640 1 -- 172.21.15.16:0/1803259226 shutdown_connections 2023-05-22T16:06:08.869 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.863+0000 7fd991789640 1 --2- 172.21.15.16:0/1803259226 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd98c14e4b0 0x7fd98c0c43e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:08.869 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.863+0000 7fd991789640 1 --2- 172.21.15.16:0/1803259226 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd98c14e040 0x7fd98c0c3ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:08.869 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.863+0000 7fd991789640 1 --2- 172.21.15.16:0/1803259226 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fd96c0654a0 0x7fd96c067950 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:08.869 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.863+0000 7fd991789640 1 --2- 172.21.15.16:0/1803259226 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd9780047d0 0x7fd98c14ff70 unknown :-1 s=CLOSED pgs=2641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:08.869 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.863+0000 7fd991789640 1 -- 172.21.15.16:0/1803259226 >> 172.21.15.16:0/1803259226 conn(0x7fd98c0cab50 msgr2=0x7fd98c0b2f10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:06:08.869 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.863+0000 7fd991789640 1 -- 172.21.15.16:0/1803259226 shutdown_connections 2023-05-22T16:06:08.869 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:08.863+0000 7fd991789640 1 -- 172.21.15.16:0/1803259226 wait complete. 2023-05-22T16:06:08.884 INFO:tasks.ceph:Scrubbing osd.3 2023-05-22T16:06:08.885 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2023-05-22T16:06:09.008 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.003+0000 7f43ba6db640 1 Processor -- start 2023-05-22T16:06:09.008 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.003+0000 7f43ba6db640 1 -- start start 2023-05-22T16:06:09.010 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.007+0000 7f43ba6db640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f43b4159a50 0x7f43b4070d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:09.010 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.007+0000 7f43ba6db640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f43b40cea90 0x7f43b40712c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:09.010 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.007+0000 7f43ba6db640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43b4071800 con 0x7f43b406c060 2023-05-22T16:06:09.010 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.007+0000 7f43ba6db640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43b415d800 con 0x7f43b40cea90 2023-05-22T16:06:09.010 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.007+0000 7f43ba6db640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43b415dae0 con 0x7f43b4159a50 2023-05-22T16:06:09.010 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.007+0000 7f43b37fe640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f43b40cea90 0x7f43b40712c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:09.010 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.007+0000 7f43b37fe640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f43b40cea90 0x7f43b40712c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43664/0 (socket says 172.21.15.16:43664) 2023-05-22T16:06:09.011 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.007+0000 7f43b37fe640 1 -- 172.21.15.16:0/753600294 learned_addr learned my addr 172.21.15.16:0/753600294 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:06:09.011 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.007+0000 7f43b3fff640 1 --2- 172.21.15.16:0/753600294 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f43b4159a50 0x7f43b4070d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:09.011 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.007+0000 7f43b37fe640 1 -- 172.21.15.16:0/753600294 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f43b4159a50 msgr2=0x7f43b4070d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:06:09.011 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.007+0000 7f43b37fe640 1 --2- 172.21.15.16:0/753600294 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f43b4159a50 0x7f43b4070d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:09.011 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.007+0000 7f43b37fe640 1 -- 172.21.15.16:0/753600294 >> v1:172.21.15.62:6789/0 conn(0x7f43b406c060 legacy=0x7f43b40c8740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:06:09.011 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.007+0000 7f43b37fe640 1 -- 172.21.15.16:0/753600294 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43b415ddc0 con 0x7f43b40cea90 2023-05-22T16:06:09.012 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.007+0000 7f43b37fe640 1 --2- 172.21.15.16:0/753600294 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f43b40cea90 0x7f43b40712c0 secure :-1 s=READY pgs=2642 cs=0 l=1 rev1=1 crypto rx=0x7f43a8009870 tx=0x7f43a801c470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e96cc7ada3a34239 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:06:09.012 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.007+0000 7f43b2ffd640 1 -- 172.21.15.16:0/753600294 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43a80ab600 con 0x7f43b40cea90 2023-05-22T16:06:09.013 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.007+0000 7f43b2ffd640 1 -- 172.21.15.16:0/753600294 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43a80a7050 con 0x7f43b40cea90 2023-05-22T16:06:09.013 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.007+0000 7f43b2ffd640 1 -- 172.21.15.16:0/753600294 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43a80ab980 con 0x7f43b40cea90 2023-05-22T16:06:09.013 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.011+0000 7f43ba6db640 1 -- 172.21.15.16:0/753600294 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f43b40cea90 msgr2=0x7f43b40712c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:06:09.014 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.011+0000 7f43ba6db640 1 --2- 172.21.15.16:0/753600294 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f43b40cea90 0x7f43b40712c0 secure :-1 s=READY pgs=2642 cs=0 l=1 rev1=1 crypto rx=0x7f43a8009870 tx=0x7f43a801c470 comp rx=0 tx=0).stop 2023-05-22T16:06:09.014 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.011+0000 7f43ba6db640 1 -- 172.21.15.16:0/753600294 shutdown_connections 2023-05-22T16:06:09.014 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.011+0000 7f43ba6db640 1 --2- 172.21.15.16:0/753600294 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f43b40cea90 0x7f43b40712c0 unknown :-1 s=CLOSED pgs=2642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:09.014 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.011+0000 7f43ba6db640 1 --2- 172.21.15.16:0/753600294 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f43b4159a50 0x7f43b4070d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:09.014 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.011+0000 7f43ba6db640 1 -- 172.21.15.16:0/753600294 >> 172.21.15.16:0/753600294 conn(0x7f43b4069800 msgr2=0x7f43b4069c00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:06:09.014 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.011+0000 7f43ba6db640 1 -- 172.21.15.16:0/753600294 shutdown_connections 2023-05-22T16:06:09.014 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.011+0000 7f43ba6db640 1 -- 172.21.15.16:0/753600294 wait complete. 2023-05-22T16:06:09.016 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.011+0000 7f43ba6db640 1 Processor -- start 2023-05-22T16:06:09.016 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.011+0000 7f43ba6db640 1 -- start start 2023-05-22T16:06:09.016 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.011+0000 7f43ba6db640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f43b406c060 0x7f43b4146690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:09.016 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.011+0000 7f43ba6db640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f43b40cea90 0x7f43b414bfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:09.017 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.011+0000 7f43ba6db640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f43b4159a50 0x7f43b414ec20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:09.017 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.011+0000 7f43ba6db640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43b414c4f0 con 0x7f43b406c060 2023-05-22T16:06:09.017 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.011+0000 7f43ba6db640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43b414e1d0 con 0x7f43b4159a50 2023-05-22T16:06:09.017 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.011+0000 7f43ba6db640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43b414b750 con 0x7f43b40cea90 2023-05-22T16:06:09.017 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.011+0000 7f43b3fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f43b406c060 0x7f43b4146690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:09.017 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.011+0000 7f43b3fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f43b406c060 0x7f43b4146690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43666/0 (socket says 172.21.15.16:43666) 2023-05-22T16:06:09.018 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.011+0000 7f43b3fff640 1 -- 172.21.15.16:0/2429389362 learned_addr learned my addr 172.21.15.16:0/2429389362 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:06:09.018 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.011+0000 7f43b8c78640 1 --2- 172.21.15.16:0/2429389362 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f43b4159a50 0x7f43b414ec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:09.018 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.011+0000 7f43b37fe640 1 --2- 172.21.15.16:0/2429389362 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f43b40cea90 0x7f43b414bfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:09.018 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.011+0000 7f43b3fff640 1 -- 172.21.15.16:0/2429389362 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f43b40cea90 msgr2=0x7f43b414bfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:06:09.018 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.011+0000 7f43b3fff640 1 --2- 172.21.15.16:0/2429389362 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f43b40cea90 0x7f43b414bfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:09.018 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.011+0000 7f43b3fff640 1 -- 172.21.15.16:0/2429389362 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f43b4159a50 msgr2=0x7f43b414ec20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:06:09.018 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.011+0000 7f43b3fff640 1 --2- 172.21.15.16:0/2429389362 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f43b4159a50 0x7f43b414ec20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:09.018 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.011+0000 7f43b3fff640 1 -- 172.21.15.16:0/2429389362 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43a80094d0 con 0x7f43b406c060 2023-05-22T16:06:09.019 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.011+0000 7f43b3fff640 1 --2- 172.21.15.16:0/2429389362 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f43b406c060 0x7f43b4146690 secure :-1 s=READY pgs=2643 cs=0 l=1 rev1=1 crypto rx=0x7f43a400d5f0 tx=0x7f43a400dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:06:09.019 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.011+0000 7f43b0ff9640 1 -- 172.21.15.16:0/2429389362 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43a40125f0 con 0x7f43b406c060 2023-05-22T16:06:09.019 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.011+0000 7f43b0ff9640 1 -- 172.21.15.16:0/2429389362 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43a4012750 con 0x7f43b406c060 2023-05-22T16:06:09.019 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.011+0000 7f43ba6db640 1 -- 172.21.15.16:0/2429389362 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43b40c9db0 con 0x7f43b406c060 2023-05-22T16:06:09.019 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.011+0000 7f43b0ff9640 1 -- 172.21.15.16:0/2429389362 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43a4012ae0 con 0x7f43b406c060 2023-05-22T16:06:09.019 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.011+0000 7f43ba6db640 1 -- 172.21.15.16:0/2429389362 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43b4071a20 con 0x7f43b406c060 2023-05-22T16:06:09.020 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.015+0000 7f43ba6db640 1 -- 172.21.15.16:0/2429389362 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f43b40c9f40 con 0x7f43b406c060 2023-05-22T16:06:09.020 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.015+0000 7f43b0ff9640 1 -- 172.21.15.16:0/2429389362 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 100) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f43a4012c40 con 0x7f43b406c060 2023-05-22T16:06:09.021 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.015+0000 7f43b0ff9640 1 --2- 172.21.15.16:0/2429389362 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f43a00654a0 0x7f43a0067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:09.021 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.015+0000 7f43b0ff9640 1 -- 172.21.15.16:0/2429389362 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(201..201 src has 1..201) v4 ==== 6317+0+0 (secure 0 0 0) 0x7f43a40839a0 con 0x7f43b406c060 2023-05-22T16:06:09.021 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.015+0000 7f43b37fe640 1 --2- 172.21.15.16:0/2429389362 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f43a00654a0 0x7f43a0067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:09.021 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.015+0000 7f43b0ff9640 1 --2- 172.21.15.16:0/2429389362 >> [v2:172.21.15.16:6816/37020,v1:172.21.15.16:6817/37020] conn(0x7f43a006c350 0x7f43a006e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:09.021 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.015+0000 7f43b0ff9640 1 -- 172.21.15.16:0/2429389362 --> [v2:172.21.15.16:6816/37020,v1:172.21.15.16:6817/37020] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f43a006ee90 con 0x7f43a006c350 2023-05-22T16:06:09.021 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.015+0000 7f43b0ff9640 1 -- 172.21.15.16:0/2429389362 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_get_version_reply(handle=1 version=201) v2 ==== 24+0+0 (secure 0 0 0) 0x7f43a4083df0 con 0x7f43b406c060 2023-05-22T16:06:09.022 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.015+0000 7f43b8c78640 1 --2- 172.21.15.16:0/2429389362 >> [v2:172.21.15.16:6816/37020,v1:172.21.15.16:6817/37020] conn(0x7f43a006c350 0x7f43a006e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:09.022 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.015+0000 7f43b37fe640 1 --2- 172.21.15.16:0/2429389362 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f43a00654a0 0x7f43a0067950 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f43a80ad000 tx=0x7f43a8009040 comp rx=0 tx=0).ready entity=mgr.8195 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:06:09.022 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.015+0000 7f43b8c78640 1 --2- 172.21.15.16:0/2429389362 >> [v2:172.21.15.16:6816/37020,v1:172.21.15.16:6817/37020] conn(0x7f43a006c350 0x7f43a006e7c0 crc :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:06:09.022 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.015+0000 7f43b0ff9640 1 -- 172.21.15.16:0/2429389362 <== osd.3 v2:172.21.15.16:6816/37020 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f43a006ee90 con 0x7f43a006c350 2023-05-22T16:06:09.032 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.027+0000 7f43ba6db640 1 -- 172.21.15.16:0/2429389362 --> [v2:172.21.15.16:6816/37020,v1:172.21.15.16:6817/37020] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f43b404dcc0 con 0x7f43a006c350 2023-05-22T16:06:09.032 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.027+0000 7f43b0ff9640 1 -- 172.21.15.16:0/2429389362 <== osd.3 v2:172.21.15.16:6816/37020 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f43b404dcc0 con 0x7f43a006c350 2023-05-22T16:06:09.032 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-05-22T16:06:09.033 INFO:teuthology.orchestra.run.smithi016.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-05-22T16:06:09.033 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-05-22T16:06:09.034 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.031+0000 7f43ba6db640 1 -- 172.21.15.16:0/2429389362 >> [v2:172.21.15.16:6816/37020,v1:172.21.15.16:6817/37020] conn(0x7f43a006c350 msgr2=0x7f43a006e7c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:06:09.034 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.031+0000 7f43ba6db640 1 --2- 172.21.15.16:0/2429389362 >> [v2:172.21.15.16:6816/37020,v1:172.21.15.16:6817/37020] conn(0x7f43a006c350 0x7f43a006e7c0 crc :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:09.034 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.031+0000 7f43ba6db640 1 -- 172.21.15.16:0/2429389362 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f43a00654a0 msgr2=0x7f43a0067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:06:09.034 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.031+0000 7f43ba6db640 1 --2- 172.21.15.16:0/2429389362 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f43a00654a0 0x7f43a0067950 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f43a80ad000 tx=0x7f43a8009040 comp rx=0 tx=0).stop 2023-05-22T16:06:09.034 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.031+0000 7f43ba6db640 1 -- 172.21.15.16:0/2429389362 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f43b406c060 msgr2=0x7f43b4146690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:06:09.035 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.031+0000 7f43ba6db640 1 --2- 172.21.15.16:0/2429389362 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f43b406c060 0x7f43b4146690 secure :-1 s=READY pgs=2643 cs=0 l=1 rev1=1 crypto rx=0x7f43a400d5f0 tx=0x7f43a400dab0 comp rx=0 tx=0).stop 2023-05-22T16:06:09.035 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.031+0000 7f43b8c78640 1 -- 172.21.15.16:0/2429389362 reap_dead start 2023-05-22T16:06:09.035 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.031+0000 7f43ba6db640 1 -- 172.21.15.16:0/2429389362 shutdown_connections 2023-05-22T16:06:09.035 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.031+0000 7f43ba6db640 1 -- 172.21.15.16:0/2429389362 >> 172.21.15.16:0/2429389362 conn(0x7f43b4069800 msgr2=0x7f43b4157c10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:06:09.035 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.031+0000 7f43ba6db640 1 -- 172.21.15.16:0/2429389362 shutdown_connections 2023-05-22T16:06:09.035 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.031+0000 7f43ba6db640 1 -- 172.21.15.16:0/2429389362 wait complete. 2023-05-22T16:06:09.049 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2023-05-22T16:06:09.164 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.159+0000 7f5647e4d640 1 Processor -- start 2023-05-22T16:06:09.164 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.159+0000 7f5647e4d640 1 -- start start 2023-05-22T16:06:09.165 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.163+0000 7f5647e4d640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f56400cfa20 0x7f56400d0020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:09.165 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.163+0000 7f5647e4d640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f56400d0560 0x7f56400d20c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:09.165 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.163+0000 7f5647e4d640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56400d0990 con 0x7f56400d2600 2023-05-22T16:06:09.166 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.163+0000 7f5647e4d640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f564015c4f0 con 0x7f56400cfa20 2023-05-22T16:06:09.166 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.163+0000 7f5647e4d640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56400cf190 con 0x7f56400d0560 2023-05-22T16:06:09.166 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.163+0000 7f56463ea640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f56400d2600 0x7f56400b23c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:49894/0 (socket says 172.21.15.16:49894) 2023-05-22T16:06:09.166 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.163+0000 7f56463ea640 1 -- 172.21.15.16:0/624681347 learned_addr learned my addr 172.21.15.16:0/624681347 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T16:06:09.168 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.163+0000 7f56453e8640 1 --2- 172.21.15.16:0/624681347 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f56400d0560 0x7f56400d20c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:09.168 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.163+0000 7f5645be9640 1 --2- 172.21.15.16:0/624681347 >> v2:172.21.15.62:3300/0 conn(0x7f56400cfa20 0x7f56400d0020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:09.168 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.163+0000 7f5644be7640 1 -- 172.21.15.16:0/624681347 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 701168937 0 0) 0x7f56400d0990 con 0x7f56400d2600 2023-05-22T16:06:09.168 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.163+0000 7f5644be7640 1 -- 172.21.15.16:0/624681347 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f562c003530 con 0x7f56400d2600 2023-05-22T16:06:09.168 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.163+0000 7f5644be7640 1 -- 172.21.15.16:0/624681347 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3201549273 0 0) 0x7f562c003530 con 0x7f56400d2600 2023-05-22T16:06:09.168 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.163+0000 7f5644be7640 1 -- 172.21.15.16:0/624681347 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f56400d0560 msgr2=0x7f56400d20c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:06:09.169 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.163+0000 7f5644be7640 1 --2- 172.21.15.16:0/624681347 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f56400d0560 0x7f56400d20c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:09.169 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.163+0000 7f5644be7640 1 -- 172.21.15.16:0/624681347 >> v2:172.21.15.62:3300/0 conn(0x7f56400cfa20 msgr2=0x7f56400d0020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:06:09.169 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.163+0000 7f5644be7640 1 --2- 172.21.15.16:0/624681347 >> v2:172.21.15.62:3300/0 conn(0x7f56400cfa20 0x7f56400d0020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:09.169 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.163+0000 7f5644be7640 1 -- 172.21.15.16:0/624681347 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56400d3360 con 0x7f56400d2600 2023-05-22T16:06:09.169 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.163+0000 7f5644be7640 1 -- 172.21.15.16:0/624681347 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7f5638001a50 con 0x7f56400d2600 2023-05-22T16:06:09.169 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.163+0000 7f5644be7640 1 -- 172.21.15.16:0/624681347 >> v1:172.21.15.16:6789/0 conn(0x7f56400d2600 legacy=0x7f56400b23c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:06:09.169 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.163+0000 7f5644be7640 1 --2- 172.21.15.16:0/624681347 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f562c003980 0x7f562c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:09.170 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.163+0000 7f5644be7640 1 --2- 172.21.15.16:0/624681347 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f56400d0560 0x7f562c0047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:09.170 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.163+0000 7f5644be7640 1 --2- 172.21.15.16:0/624681347 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f562c004d10 0x7f562c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:09.170 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.163+0000 7f5644be7640 1 -- 172.21.15.16:0/624681347 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f564015c4f0 con 0x7f562c003980 2023-05-22T16:06:09.170 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.163+0000 7f5644be7640 1 -- 172.21.15.16:0/624681347 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56400cf190 con 0x7f562c004d10 2023-05-22T16:06:09.170 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.163+0000 7f5644be7640 1 -- 172.21.15.16:0/624681347 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f562c00b990 con 0x7f56400d0560 2023-05-22T16:06:09.170 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.163+0000 7f5645be9640 1 --2- 172.21.15.16:0/624681347 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f562c003980 0x7f562c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:09.170 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.163+0000 7f56463ea640 1 --2- 172.21.15.16:0/624681347 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f562c004d10 0x7f562c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:09.171 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.163+0000 7f56453e8640 1 --2- 172.21.15.16:0/624681347 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f56400d0560 0x7f562c0047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:09.171 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.163+0000 7f5645be9640 1 -- 172.21.15.16:0/624681347 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f56400d0560 msgr2=0x7f562c0047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:06:09.171 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.163+0000 7f5645be9640 1 --2- 172.21.15.16:0/624681347 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f56400d0560 0x7f562c0047d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:09.171 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.163+0000 7f5645be9640 1 -- 172.21.15.16:0/624681347 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f562c004d10 msgr2=0x7f562c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:06:09.171 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.163+0000 7f5645be9640 1 --2- 172.21.15.16:0/624681347 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f562c004d10 0x7f562c005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:09.171 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.163+0000 7f5645be9640 1 -- 172.21.15.16:0/624681347 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f562c003530 con 0x7f562c003980 2023-05-22T16:06:09.175 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.167+0000 7f5645be9640 1 --2- 172.21.15.16:0/624681347 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f562c003980 0x7f562c003d90 secure :-1 s=READY pgs=2645 cs=0 l=1 rev1=1 crypto rx=0x7f5634009af0 tx=0x7f563401c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cbf0399260e9fd87 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:06:09.175 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.167+0000 7f5644be7640 1 -- 172.21.15.16:0/624681347 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56340059a0 con 0x7f562c003980 2023-05-22T16:06:09.175 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.167+0000 7f5644be7640 1 -- 172.21.15.16:0/624681347 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5634005b00 con 0x7f562c003980 2023-05-22T16:06:09.176 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.167+0000 7f5644be7640 1 -- 172.21.15.16:0/624681347 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56340a5640 con 0x7f562c003980 2023-05-22T16:06:09.176 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.167+0000 7f5647e4d640 1 -- 172.21.15.16:0/624681347 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f562c003980 msgr2=0x7f562c003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:06:09.176 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.167+0000 7f5647e4d640 1 --2- 172.21.15.16:0/624681347 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f562c003980 0x7f562c003d90 secure :-1 s=READY pgs=2645 cs=0 l=1 rev1=1 crypto rx=0x7f5634009af0 tx=0x7f563401c9f0 comp rx=0 tx=0).stop 2023-05-22T16:06:09.176 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.167+0000 7f56463ea640 1 -- 172.21.15.16:0/624681347 reap_dead start 2023-05-22T16:06:09.177 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.167+0000 7f5647e4d640 1 -- 172.21.15.16:0/624681347 shutdown_connections 2023-05-22T16:06:09.177 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.167+0000 7f5647e4d640 1 -- 172.21.15.16:0/624681347 >> 172.21.15.16:0/624681347 conn(0x7f5640071850 msgr2=0x7f564009b830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:06:09.177 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.167+0000 7f5647e4d640 1 -- 172.21.15.16:0/624681347 shutdown_connections 2023-05-22T16:06:09.177 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.167+0000 7f5647e4d640 1 -- 172.21.15.16:0/624681347 wait complete. 2023-05-22T16:06:09.178 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.167+0000 7f5647e4d640 1 Processor -- start 2023-05-22T16:06:09.178 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.167+0000 7f5647e4d640 1 -- start start 2023-05-22T16:06:09.178 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.167+0000 7f5647e4d640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f562c003980 0x7f564014d820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:09.179 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.167+0000 7f5647e4d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f56400c9520 0x7f56400b79e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:09.179 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.167+0000 7f5647e4d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f564015c750 0x7f564015cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:09.179 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.167+0000 7f5647e4d640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56400ca630 con 0x7f56400c9520 2023-05-22T16:06:09.179 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.167+0000 7f5647e4d640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56400c9a20 con 0x7f564015c750 2023-05-22T16:06:09.179 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.167+0000 7f5647e4d640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56400c9d00 con 0x7f562c003980 2023-05-22T16:06:09.179 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.167+0000 7f56453e8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f56400c9520 0x7f56400b79e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:09.180 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.167+0000 7f56453e8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f56400c9520 0x7f56400b79e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43692/0 (socket says 172.21.15.16:43692) 2023-05-22T16:06:09.180 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.167+0000 7f56453e8640 1 -- 172.21.15.16:0/4060221238 learned_addr learned my addr 172.21.15.16:0/4060221238 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:06:09.180 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.167+0000 7f5645be9640 1 --2- 172.21.15.16:0/4060221238 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f562c003980 0x7f564014d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:09.180 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.167+0000 7f56463ea640 1 --2- 172.21.15.16:0/4060221238 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f564015c750 0x7f564015cbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:09.181 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.167+0000 7f56453e8640 1 -- 172.21.15.16:0/4060221238 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f562c003980 msgr2=0x7f564014d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:06:09.181 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.167+0000 7f56453e8640 1 --2- 172.21.15.16:0/4060221238 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f562c003980 0x7f564014d820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:09.181 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.167+0000 7f56453e8640 1 -- 172.21.15.16:0/4060221238 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f564015c750 msgr2=0x7f564015cbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:06:09.181 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.167+0000 7f56453e8640 1 --2- 172.21.15.16:0/4060221238 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f564015c750 0x7f564015cbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:09.181 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.167+0000 7f56453e8640 1 -- 172.21.15.16:0/4060221238 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56400d3080 con 0x7f56400c9520 2023-05-22T16:06:09.182 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.167+0000 7f56453e8640 1 --2- 172.21.15.16:0/4060221238 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f56400c9520 0x7f56400b79e0 secure :-1 s=READY pgs=2646 cs=0 l=1 rev1=1 crypto rx=0x7f563c010890 tx=0x7f563c010d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:06:09.182 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.167+0000 7f56327fc640 1 -- 172.21.15.16:0/4060221238 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f563c009630 con 0x7f56400c9520 2023-05-22T16:06:09.182 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.167+0000 7f56327fc640 1 -- 172.21.15.16:0/4060221238 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f563c00eb60 con 0x7f56400c9520 2023-05-22T16:06:09.182 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.167+0000 7f5647e4d640 1 -- 172.21.15.16:0/4060221238 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56400d2da0 con 0x7f56400c9520 2023-05-22T16:06:09.182 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.171+0000 7f56327fc640 1 -- 172.21.15.16:0/4060221238 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f563c01c400 con 0x7f56400c9520 2023-05-22T16:06:09.182 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.171+0000 7f5647e4d640 1 -- 172.21.15.16:0/4060221238 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56400ccb40 con 0x7f56400c9520 2023-05-22T16:06:09.183 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.171+0000 7f5647e4d640 1 -- 172.21.15.16:0/4060221238 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5640073830 con 0x7f56400c9520 2023-05-22T16:06:09.183 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.171+0000 7f56327fc640 1 -- 172.21.15.16:0/4060221238 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 100) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f563c01c560 con 0x7f56400c9520 2023-05-22T16:06:09.183 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.171+0000 7f56327fc640 1 --2- 172.21.15.16:0/4060221238 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f56240654a0 0x7f5624067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:09.183 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.171+0000 7f5645be9640 1 --2- 172.21.15.16:0/4060221238 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f56240654a0 0x7f5624067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:09.183 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.171+0000 7f56327fc640 1 -- 172.21.15.16:0/4060221238 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(201..201 src has 1..201) v4 ==== 6317+0+0 (secure 0 0 0) 0x7f563c0831b0 con 0x7f56400c9520 2023-05-22T16:06:09.184 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.171+0000 7f5645be9640 1 --2- 172.21.15.16:0/4060221238 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f56240654a0 0x7f5624067950 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f56340ae000 tx=0x7f56340046d0 comp rx=0 tx=0).ready entity=mgr.8195 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:06:09.184 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.175+0000 7f56327fc640 1 -- 172.21.15.16:0/4060221238 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f563c054520 con 0x7f56400c9520 2023-05-22T16:06:09.330 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.327+0000 7f5647e4d640 1 -- 172.21.15.16:0/4060221238 --> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) v1 -- 0x7f56400cce10 con 0x7f56240654a0 2023-05-22T16:06:09.331 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.327+0000 7f56327fc640 1 -- 172.21.15.16:0/4060221238 <== mgr.8195 v2:172.21.15.16:6832/65890 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f56400cce10 con 0x7f56240654a0 2023-05-22T16:06:09.331 INFO:teuthology.orchestra.run.smithi016.stderr:instructed osd(s) 3 to deep-scrub 2023-05-22T16:06:09.333 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.331+0000 7f5647e4d640 1 -- 172.21.15.16:0/4060221238 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f56240654a0 msgr2=0x7f5624067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:06:09.333 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.331+0000 7f5647e4d640 1 --2- 172.21.15.16:0/4060221238 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f56240654a0 0x7f5624067950 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f56340ae000 tx=0x7f56340046d0 comp rx=0 tx=0).stop 2023-05-22T16:06:09.333 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.331+0000 7f5647e4d640 1 -- 172.21.15.16:0/4060221238 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f56400c9520 msgr2=0x7f56400b79e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:06:09.334 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.331+0000 7f5647e4d640 1 --2- 172.21.15.16:0/4060221238 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f56400c9520 0x7f56400b79e0 secure :-1 s=READY pgs=2646 cs=0 l=1 rev1=1 crypto rx=0x7f563c010890 tx=0x7f563c010d50 comp rx=0 tx=0).stop 2023-05-22T16:06:09.334 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.331+0000 7f5647e4d640 1 -- 172.21.15.16:0/4060221238 shutdown_connections 2023-05-22T16:06:09.334 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.331+0000 7f5647e4d640 1 --2- 172.21.15.16:0/4060221238 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f564015c750 0x7f564015cbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:09.334 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.331+0000 7f5647e4d640 1 --2- 172.21.15.16:0/4060221238 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f56240654a0 0x7f5624067950 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:09.334 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.331+0000 7f5647e4d640 1 --2- 172.21.15.16:0/4060221238 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f56400c9520 0x7f56400b79e0 unknown :-1 s=CLOSED pgs=2646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:09.334 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.331+0000 7f5647e4d640 1 --2- 172.21.15.16:0/4060221238 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f562c003980 0x7f564014d820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:09.334 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.331+0000 7f5647e4d640 1 -- 172.21.15.16:0/4060221238 >> 172.21.15.16:0/4060221238 conn(0x7f5640071850 msgr2=0x7f564006ae80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:06:09.335 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.331+0000 7f5647e4d640 1 -- 172.21.15.16:0/4060221238 shutdown_connections 2023-05-22T16:06:09.335 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.331+0000 7f5647e4d640 1 -- 172.21.15.16:0/4060221238 wait complete. 2023-05-22T16:06:09.350 INFO:tasks.ceph:Scrubbing osd.4 2023-05-22T16:06:09.350 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2023-05-22T16:06:09.479 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.475+0000 7f2228f0b640 1 Processor -- start 2023-05-22T16:06:09.479 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.475+0000 7f2228f0b640 1 -- start start 2023-05-22T16:06:09.480 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.475+0000 7f2228f0b640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f222415f510 0x7f222406ac20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:09.481 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.475+0000 7f2228f0b640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f22241607e0 0x7f222406ff60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:09.481 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.475+0000 7f2228f0b640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22240cc820 con 0x7f22240704a0 2023-05-22T16:06:09.481 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.475+0000 7f2228f0b640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f222406de10 con 0x7f22241607e0 2023-05-22T16:06:09.481 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.475+0000 7f2228f0b640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2224070920 con 0x7f222415f510 2023-05-22T16:06:09.481 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.475+0000 7f221bfff640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f22241607e0 0x7f222406ff60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:09.481 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.475+0000 7f221bfff640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f22241607e0 0x7f222406ff60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57688/0 (socket says 172.21.15.16:57688) 2023-05-22T16:06:09.481 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.475+0000 7f221bfff640 1 -- 172.21.15.16:0/57835495 learned_addr learned my addr 172.21.15.16:0/57835495 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:06:09.481 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.475+0000 7f221bfff640 1 -- 172.21.15.16:0/57835495 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f222415f510 msgr2=0x7f222406ac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:06:09.482 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.475+0000 7f222259c640 1 --2- 172.21.15.16:0/57835495 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f222415f510 0x7f222406ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:09.482 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.475+0000 7f221bfff640 1 --2- 172.21.15.16:0/57835495 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f222415f510 0x7f222406ac20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:09.482 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.475+0000 7f221bfff640 1 -- 172.21.15.16:0/57835495 >> v1:172.21.15.62:6789/0 conn(0x7f22240704a0 legacy=0x7f222406bfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:06:09.482 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.475+0000 7f221bfff640 1 -- 172.21.15.16:0/57835495 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f222415f8e0 con 0x7f22241607e0 2023-05-22T16:06:09.483 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.479+0000 7f221bfff640 1 --2- 172.21.15.16:0/57835495 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f22241607e0 0x7f222406ff60 secure :-1 s=READY pgs=2647 cs=0 l=1 rev1=1 crypto rx=0x7f220c009870 tx=0x7f220c01c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c993c61ae31f951 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:06:09.483 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.479+0000 7f2221d9b640 1 -- 172.21.15.16:0/57835495 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f220c0ac640 con 0x7f22241607e0 2023-05-22T16:06:09.483 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.479+0000 7f2221d9b640 1 -- 172.21.15.16:0/57835495 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f220c0ac7a0 con 0x7f22241607e0 2023-05-22T16:06:09.484 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.479+0000 7f2221d9b640 1 -- 172.21.15.16:0/57835495 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f220c0acb30 con 0x7f22241607e0 2023-05-22T16:06:09.484 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.479+0000 7f2228f0b640 1 -- 172.21.15.16:0/57835495 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f22241607e0 msgr2=0x7f222406ff60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:06:09.484 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.479+0000 7f2228f0b640 1 --2- 172.21.15.16:0/57835495 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f22241607e0 0x7f222406ff60 secure :-1 s=READY pgs=2647 cs=0 l=1 rev1=1 crypto rx=0x7f220c009870 tx=0x7f220c01c500 comp rx=0 tx=0).stop 2023-05-22T16:06:09.484 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.479+0000 7f2228f0b640 1 -- 172.21.15.16:0/57835495 shutdown_connections 2023-05-22T16:06:09.484 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.479+0000 7f2228f0b640 1 --2- 172.21.15.16:0/57835495 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f22241607e0 0x7f222406ff60 unknown :-1 s=CLOSED pgs=2647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:09.484 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.479+0000 7f2228f0b640 1 --2- 172.21.15.16:0/57835495 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f222415f510 0x7f222406ac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:09.484 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.479+0000 7f2228f0b640 1 -- 172.21.15.16:0/57835495 >> 172.21.15.16:0/57835495 conn(0x7f22240ce9c0 msgr2=0x7f22240cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:06:09.485 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.479+0000 7f2228f0b640 1 -- 172.21.15.16:0/57835495 shutdown_connections 2023-05-22T16:06:09.485 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.479+0000 7f2228f0b640 1 -- 172.21.15.16:0/57835495 wait complete. 2023-05-22T16:06:09.486 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.483+0000 7f2228f0b640 1 Processor -- start 2023-05-22T16:06:09.486 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.483+0000 7f2228f0b640 1 -- start start 2023-05-22T16:06:09.487 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.483+0000 7f2228f0b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f22240704a0 0x7f22240ac520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:09.487 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.483+0000 7f2228f0b640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f222415f510 0x7f22240aca60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:09.487 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.483+0000 7f2228f0b640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f22241607e0 0x7f2224155c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:09.487 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.483+0000 7f2228f0b640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22240acfa0 con 0x7f22240704a0 2023-05-22T16:06:09.487 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.483+0000 7f2228f0b640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2224156a80 con 0x7f222415f510 2023-05-22T16:06:09.487 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.483+0000 7f2228f0b640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2224156d60 con 0x7f22241607e0 2023-05-22T16:06:09.487 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.483+0000 7f222259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f22240704a0 0x7f22240ac520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:09.488 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.483+0000 7f222259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f22240704a0 0x7f22240ac520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57690/0 (socket says 172.21.15.16:57690) 2023-05-22T16:06:09.488 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.483+0000 7f222259c640 1 -- 172.21.15.16:0/23439894 learned_addr learned my addr 172.21.15.16:0/23439894 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:06:09.488 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.483+0000 7f2222d9d640 1 --2- 172.21.15.16:0/23439894 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f22241607e0 0x7f2224155c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:09.488 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.483+0000 7f222259c640 1 -- 172.21.15.16:0/23439894 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f22241607e0 msgr2=0x7f2224155c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:06:09.488 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.483+0000 7f221bfff640 1 --2- 172.21.15.16:0/23439894 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f222415f510 0x7f22240aca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:09.488 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.483+0000 7f222259c640 1 --2- 172.21.15.16:0/23439894 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f22241607e0 0x7f2224155c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:09.488 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.483+0000 7f222259c640 1 -- 172.21.15.16:0/23439894 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f222415f510 msgr2=0x7f22240aca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:06:09.488 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.483+0000 7f222259c640 1 --2- 172.21.15.16:0/23439894 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f222415f510 0x7f22240aca60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:09.489 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.483+0000 7f222259c640 1 -- 172.21.15.16:0/23439894 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f220c0094d0 con 0x7f22240704a0 2023-05-22T16:06:09.489 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.483+0000 7f222259c640 1 --2- 172.21.15.16:0/23439894 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f22240704a0 0x7f22240ac520 secure :-1 s=READY pgs=2648 cs=0 l=1 rev1=1 crypto rx=0x7f221400c7f0 tx=0x7f221400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:06:09.489 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.483+0000 7f221affd640 1 -- 172.21.15.16:0/23439894 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2214007820 con 0x7f22240704a0 2023-05-22T16:06:09.489 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.483+0000 7f221affd640 1 -- 172.21.15.16:0/23439894 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2214007980 con 0x7f22240704a0 2023-05-22T16:06:09.489 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.483+0000 7f2228f0b640 1 -- 172.21.15.16:0/23439894 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f222415ffe0 con 0x7f22240704a0 2023-05-22T16:06:09.489 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.483+0000 7f221affd640 1 -- 172.21.15.16:0/23439894 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2214007d30 con 0x7f22240704a0 2023-05-22T16:06:09.489 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.483+0000 7f2228f0b640 1 -- 172.21.15.16:0/23439894 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f222415fd00 con 0x7f22240704a0 2023-05-22T16:06:09.490 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.483+0000 7f2228f0b640 1 -- 172.21.15.16:0/23439894 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f22241602c0 con 0x7f22240704a0 2023-05-22T16:06:09.491 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.483+0000 7f221affd640 1 -- 172.21.15.16:0/23439894 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 100) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f22140113f0 con 0x7f22240704a0 2023-05-22T16:06:09.491 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.487+0000 7f221affd640 1 --2- 172.21.15.16:0/23439894 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f22040654a0 0x7f2204067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:09.491 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.487+0000 7f221affd640 1 -- 172.21.15.16:0/23439894 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(201..201 src has 1..201) v4 ==== 6317+0+0 (secure 0 0 0) 0x7f22140834a0 con 0x7f22240704a0 2023-05-22T16:06:09.491 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.487+0000 7f221bfff640 1 --2- 172.21.15.16:0/23439894 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f22040654a0 0x7f2204067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:09.491 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.487+0000 7f221affd640 1 --2- 172.21.15.16:0/23439894 >> [v2:172.21.15.62:6816/37353,v1:172.21.15.62:6817/37353] conn(0x7f220406c350 0x7f220406e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:09.491 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.487+0000 7f221affd640 1 -- 172.21.15.16:0/23439894 --> [v2:172.21.15.62:6816/37353,v1:172.21.15.62:6817/37353] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f220406ee90 con 0x7f220406c350 2023-05-22T16:06:09.491 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.487+0000 7f221affd640 1 -- 172.21.15.16:0/23439894 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_get_version_reply(handle=1 version=201) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2214083810 con 0x7f22240704a0 2023-05-22T16:06:09.492 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.487+0000 7f221bfff640 1 --2- 172.21.15.16:0/23439894 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f22040654a0 0x7f2204067950 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f220c0ae000 tx=0x7f220c0a7000 comp rx=0 tx=0).ready entity=mgr.8195 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:06:09.492 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.487+0000 7f2222d9d640 1 --2- 172.21.15.16:0/23439894 >> [v2:172.21.15.62:6816/37353,v1:172.21.15.62:6817/37353] conn(0x7f220406c350 0x7f220406e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:09.492 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.487+0000 7f2222d9d640 1 --2- 172.21.15.16:0/23439894 >> [v2:172.21.15.62:6816/37353,v1:172.21.15.62:6817/37353] conn(0x7f220406c350 0x7f220406e7c0 crc :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:06:09.493 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.487+0000 7f221affd640 1 -- 172.21.15.16:0/23439894 <== osd.4 v2:172.21.15.62:6816/37353 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f220406ee90 con 0x7f220406c350 2023-05-22T16:06:09.504 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.499+0000 7f2228f0b640 1 -- 172.21.15.16:0/23439894 --> [v2:172.21.15.62:6816/37353,v1:172.21.15.62:6817/37353] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f22240c2080 con 0x7f220406c350 2023-05-22T16:06:09.505 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.503+0000 7f221affd640 1 -- 172.21.15.16:0/23439894 <== osd.4 v2:172.21.15.62:6816/37353 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f22240c2080 con 0x7f220406c350 2023-05-22T16:06:09.505 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-05-22T16:06:09.505 INFO:teuthology.orchestra.run.smithi016.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-05-22T16:06:09.505 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-05-22T16:06:09.506 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.503+0000 7f2228f0b640 1 -- 172.21.15.16:0/23439894 >> [v2:172.21.15.62:6816/37353,v1:172.21.15.62:6817/37353] conn(0x7f220406c350 msgr2=0x7f220406e7c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:06:09.506 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.503+0000 7f2228f0b640 1 --2- 172.21.15.16:0/23439894 >> [v2:172.21.15.62:6816/37353,v1:172.21.15.62:6817/37353] conn(0x7f220406c350 0x7f220406e7c0 crc :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:09.506 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.503+0000 7f2228f0b640 1 -- 172.21.15.16:0/23439894 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f22040654a0 msgr2=0x7f2204067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:06:09.507 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.503+0000 7f2228f0b640 1 --2- 172.21.15.16:0/23439894 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f22040654a0 0x7f2204067950 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f220c0ae000 tx=0x7f220c0a7000 comp rx=0 tx=0).stop 2023-05-22T16:06:09.507 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.503+0000 7f2228f0b640 1 -- 172.21.15.16:0/23439894 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f22240704a0 msgr2=0x7f22240ac520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:06:09.507 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.503+0000 7f2228f0b640 1 --2- 172.21.15.16:0/23439894 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f22240704a0 0x7f22240ac520 secure :-1 s=READY pgs=2648 cs=0 l=1 rev1=1 crypto rx=0x7f221400c7f0 tx=0x7f221400ccb0 comp rx=0 tx=0).stop 2023-05-22T16:06:09.507 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.503+0000 7f2222d9d640 1 -- 172.21.15.16:0/23439894 reap_dead start 2023-05-22T16:06:09.507 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.503+0000 7f2228f0b640 1 -- 172.21.15.16:0/23439894 shutdown_connections 2023-05-22T16:06:09.507 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.503+0000 7f2228f0b640 1 -- 172.21.15.16:0/23439894 >> 172.21.15.16:0/23439894 conn(0x7f22240ce9c0 msgr2=0x7f222406a090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:06:09.507 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.503+0000 7f2228f0b640 1 -- 172.21.15.16:0/23439894 shutdown_connections 2023-05-22T16:06:09.508 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.503+0000 7f2228f0b640 1 -- 172.21.15.16:0/23439894 wait complete. 2023-05-22T16:06:09.521 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2023-05-22T16:06:09.638 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.635+0000 7f84d5a48640 1 Processor -- start 2023-05-22T16:06:09.638 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.635+0000 7f84d5a48640 1 -- start start 2023-05-22T16:06:09.639 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.635+0000 7f84d5a48640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f84d0154170 0x7f84d006f630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:09.639 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.635+0000 7f84d5a48640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f84d00afb50 0x7f84d00affb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:09.639 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.635+0000 7f84d5a48640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84d0071b40 con 0x7f84d006cc80 2023-05-22T16:06:09.640 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.635+0000 7f84d5a48640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84d00ca9f0 con 0x7f84d0154170 2023-05-22T16:06:09.640 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.635+0000 7f84d5a48640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84d00ca150 con 0x7f84d00afb50 2023-05-22T16:06:09.640 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.635+0000 7f84ceffd640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f84d0154170 0x7f84d006f630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:09.640 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.635+0000 7f84ce7fc640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f84d006cc80 0x7f84d006be90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:46400/0 (socket says 172.21.15.16:46400) 2023-05-22T16:06:09.640 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.635+0000 7f84ce7fc640 1 -- 172.21.15.16:0/1041559892 learned_addr learned my addr 172.21.15.16:0/1041559892 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T16:06:09.640 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.635+0000 7f84cf7fe640 1 --2- 172.21.15.16:0/1041559892 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f84d00afb50 0x7f84d00affb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:09.640 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.635+0000 7f84ceffd640 1 -- 172.21.15.16:0/1041559892 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f84d00afb50 msgr2=0x7f84d00affb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:06:09.641 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.635+0000 7f84ceffd640 1 --2- 172.21.15.16:0/1041559892 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f84d00afb50 0x7f84d00affb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:09.641 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.635+0000 7f84ceffd640 1 -- 172.21.15.16:0/1041559892 >> v1:172.21.15.16:6789/0 conn(0x7f84d006cc80 legacy=0x7f84d006be90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:06:09.641 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.635+0000 7f84cdffb640 1 -- 172.21.15.16:0/1041559892 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3290317105 0 0) 0x7f84d0071b40 con 0x7f84d006cc80 2023-05-22T16:06:09.641 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.635+0000 7f84ceffd640 1 -- 172.21.15.16:0/1041559892 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84d00b04f0 con 0x7f84d0154170 2023-05-22T16:06:09.642 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.639+0000 7f84ceffd640 1 --2- 172.21.15.16:0/1041559892 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f84d0154170 0x7f84d006f630 secure :-1 s=READY pgs=2650 cs=0 l=1 rev1=1 crypto rx=0x7f84c00097c0 tx=0x7f84c001c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af32c4af1f55287c server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:06:09.642 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.639+0000 7f84cdffb640 1 -- 172.21.15.16:0/1041559892 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84c00ab650 con 0x7f84d0154170 2023-05-22T16:06:09.642 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.639+0000 7f84cdffb640 1 -- 172.21.15.16:0/1041559892 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84c00ab7b0 con 0x7f84d0154170 2023-05-22T16:06:09.642 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.639+0000 7f84cdffb640 1 -- 172.21.15.16:0/1041559892 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84c00abb40 con 0x7f84d0154170 2023-05-22T16:06:09.643 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.639+0000 7f84d5a48640 1 -- 172.21.15.16:0/1041559892 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f84d0154170 msgr2=0x7f84d006f630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:06:09.643 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.639+0000 7f84d5a48640 1 --2- 172.21.15.16:0/1041559892 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f84d0154170 0x7f84d006f630 secure :-1 s=READY pgs=2650 cs=0 l=1 rev1=1 crypto rx=0x7f84c00097c0 tx=0x7f84c001c450 comp rx=0 tx=0).stop 2023-05-22T16:06:09.643 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.639+0000 7f84d5a48640 1 -- 172.21.15.16:0/1041559892 shutdown_connections 2023-05-22T16:06:09.643 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.639+0000 7f84d5a48640 1 --2- 172.21.15.16:0/1041559892 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f84d00afb50 0x7f84d00affb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:09.643 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.639+0000 7f84d5a48640 1 --2- 172.21.15.16:0/1041559892 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f84d0154170 0x7f84d006f630 unknown :-1 s=CLOSED pgs=2650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:09.644 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.639+0000 7f84d5a48640 1 -- 172.21.15.16:0/1041559892 >> 172.21.15.16:0/1041559892 conn(0x7f84d00cab50 msgr2=0x7f84d00caf50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:06:09.644 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.639+0000 7f84d5a48640 1 -- 172.21.15.16:0/1041559892 shutdown_connections 2023-05-22T16:06:09.644 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.639+0000 7f84d5a48640 1 -- 172.21.15.16:0/1041559892 wait complete. 2023-05-22T16:06:09.645 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.639+0000 7f84d5a48640 1 Processor -- start 2023-05-22T16:06:09.645 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.639+0000 7f84d5a48640 1 -- start start 2023-05-22T16:06:09.646 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.639+0000 7f84d5a48640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f84d006cc80 0x7f84d0152830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:09.646 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.639+0000 7f84d5a48640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f84d00afb50 0x7f84d00c6d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:09.646 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.639+0000 7f84d5a48640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f84d0154170 0x7f84d0156900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:09.646 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.639+0000 7f84d5a48640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84d00c51a0 con 0x7f84d00afb50 2023-05-22T16:06:09.646 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.639+0000 7f84d5a48640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84d00b3cd0 con 0x7f84d006cc80 2023-05-22T16:06:09.646 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.643+0000 7f84d5a48640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84d00c5440 con 0x7f84d0154170 2023-05-22T16:06:09.646 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.643+0000 7f84ce7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f84d00afb50 0x7f84d00c6d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:09.646 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.643+0000 7f84ce7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f84d00afb50 0x7f84d00c6d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57714/0 (socket says 172.21.15.16:57714) 2023-05-22T16:06:09.647 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.643+0000 7f84ce7fc640 1 -- 172.21.15.16:0/2151901891 learned_addr learned my addr 172.21.15.16:0/2151901891 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:06:09.647 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.643+0000 7f84cf7fe640 1 --2- 172.21.15.16:0/2151901891 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f84d0154170 0x7f84d0156900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:09.647 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.643+0000 7f84ceffd640 1 --2- 172.21.15.16:0/2151901891 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f84d006cc80 0x7f84d0152830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:09.647 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.643+0000 7f84ce7fc640 1 -- 172.21.15.16:0/2151901891 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f84d0154170 msgr2=0x7f84d0156900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:06:09.647 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.643+0000 7f84ce7fc640 1 --2- 172.21.15.16:0/2151901891 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f84d0154170 0x7f84d0156900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:09.647 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.643+0000 7f84ce7fc640 1 -- 172.21.15.16:0/2151901891 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f84d006cc80 msgr2=0x7f84d0152830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:06:09.647 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.643+0000 7f84ce7fc640 1 --2- 172.21.15.16:0/2151901891 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f84d006cc80 0x7f84d0152830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:09.648 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.643+0000 7f84ce7fc640 1 -- 172.21.15.16:0/2151901891 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84c0009420 con 0x7f84d00afb50 2023-05-22T16:06:09.648 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.643+0000 7f84ceffd640 1 --2- 172.21.15.16:0/2151901891 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f84d006cc80 0x7f84d0152830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T16:06:09.648 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.643+0000 7f84ce7fc640 1 --2- 172.21.15.16:0/2151901891 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f84d00afb50 0x7f84d00c6d50 secure :-1 s=READY pgs=2651 cs=0 l=1 rev1=1 crypto rx=0x7f84c400c4b0 tx=0x7f84c400c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:06:09.648 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.643+0000 7f84b7fff640 1 -- 172.21.15.16:0/2151901891 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84c40074e0 con 0x7f84d00afb50 2023-05-22T16:06:09.648 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.643+0000 7f84b7fff640 1 -- 172.21.15.16:0/2151901891 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84c4007660 con 0x7f84d00afb50 2023-05-22T16:06:09.648 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.643+0000 7f84d5a48640 1 -- 172.21.15.16:0/2151901891 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84d006c5a0 con 0x7f84d00afb50 2023-05-22T16:06:09.648 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.643+0000 7f84b7fff640 1 -- 172.21.15.16:0/2151901891 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84c4004a30 con 0x7f84d00afb50 2023-05-22T16:06:09.648 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.643+0000 7f84d5a48640 1 -- 172.21.15.16:0/2151901891 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84d006b970 con 0x7f84d00afb50 2023-05-22T16:06:09.649 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.643+0000 7f84d5a48640 1 -- 172.21.15.16:0/2151901891 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84d0154540 con 0x7f84d00afb50 2023-05-22T16:06:09.649 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.643+0000 7f84b7fff640 1 -- 172.21.15.16:0/2151901891 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 100) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f84c4004b90 con 0x7f84d00afb50 2023-05-22T16:06:09.650 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.643+0000 7f84b7fff640 1 --2- 172.21.15.16:0/2151901891 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f84bc0654a0 0x7f84bc067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:09.650 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.643+0000 7f84b7fff640 1 -- 172.21.15.16:0/2151901891 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(201..201 src has 1..201) v4 ==== 6317+0+0 (secure 0 0 0) 0x7f84c4081d40 con 0x7f84d00afb50 2023-05-22T16:06:09.650 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.643+0000 7f84ceffd640 1 --2- 172.21.15.16:0/2151901891 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f84bc0654a0 0x7f84bc067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:09.650 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.643+0000 7f84ceffd640 1 --2- 172.21.15.16:0/2151901891 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f84bc0654a0 0x7f84bc067950 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f84c00af000 tx=0x7f84c00a7000 comp rx=0 tx=0).ready entity=mgr.8195 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:06:09.651 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.647+0000 7f84b7fff640 1 -- 172.21.15.16:0/2151901891 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f84c404f870 con 0x7f84d00afb50 2023-05-22T16:06:09.807 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.803+0000 7f84d5a48640 1 -- 172.21.15.16:0/2151901891 --> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) v1 -- 0x7f84d0073830 con 0x7f84bc0654a0 2023-05-22T16:06:09.808 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.803+0000 7f84b7fff640 1 -- 172.21.15.16:0/2151901891 <== mgr.8195 v2:172.21.15.16:6832/65890 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f84d0073830 con 0x7f84bc0654a0 2023-05-22T16:06:09.808 INFO:teuthology.orchestra.run.smithi016.stderr:instructed osd(s) 4 to deep-scrub 2023-05-22T16:06:09.809 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.807+0000 7f84d5a48640 1 -- 172.21.15.16:0/2151901891 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f84bc0654a0 msgr2=0x7f84bc067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:06:09.810 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.807+0000 7f84d5a48640 1 --2- 172.21.15.16:0/2151901891 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f84bc0654a0 0x7f84bc067950 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f84c00af000 tx=0x7f84c00a7000 comp rx=0 tx=0).stop 2023-05-22T16:06:09.810 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.807+0000 7f84d5a48640 1 -- 172.21.15.16:0/2151901891 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f84d00afb50 msgr2=0x7f84d00c6d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:06:09.810 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.807+0000 7f84d5a48640 1 --2- 172.21.15.16:0/2151901891 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f84d00afb50 0x7f84d00c6d50 secure :-1 s=READY pgs=2651 cs=0 l=1 rev1=1 crypto rx=0x7f84c400c4b0 tx=0x7f84c400c970 comp rx=0 tx=0).stop 2023-05-22T16:06:09.810 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.807+0000 7f84d5a48640 1 -- 172.21.15.16:0/2151901891 shutdown_connections 2023-05-22T16:06:09.810 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.807+0000 7f84d5a48640 1 --2- 172.21.15.16:0/2151901891 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f84d0154170 0x7f84d0156900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:09.810 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.807+0000 7f84d5a48640 1 --2- 172.21.15.16:0/2151901891 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f84bc0654a0 0x7f84bc067950 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:09.810 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.807+0000 7f84d5a48640 1 --2- 172.21.15.16:0/2151901891 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f84d00afb50 0x7f84d00c6d50 unknown :-1 s=CLOSED pgs=2651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:09.811 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.807+0000 7f84d5a48640 1 --2- 172.21.15.16:0/2151901891 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f84d006cc80 0x7f84d0152830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:09.811 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.807+0000 7f84d5a48640 1 -- 172.21.15.16:0/2151901891 >> 172.21.15.16:0/2151901891 conn(0x7f84d00cab50 msgr2=0x7f84d00caf20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:06:09.811 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.807+0000 7f84d5a48640 1 -- 172.21.15.16:0/2151901891 shutdown_connections 2023-05-22T16:06:09.811 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.807+0000 7f84d5a48640 1 -- 172.21.15.16:0/2151901891 wait complete. 2023-05-22T16:06:09.827 INFO:tasks.ceph:Scrubbing osd.5 2023-05-22T16:06:09.828 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2023-05-22T16:06:09.954 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.951+0000 7f8917792640 1 Processor -- start 2023-05-22T16:06:09.954 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.951+0000 7f8917792640 1 -- start start 2023-05-22T16:06:09.956 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.951+0000 7f8917792640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f89100ce950 0x7f89100ca880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:09.956 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.951+0000 7f8917792640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f89100ced20 0x7f89100cadc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:09.956 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.951+0000 7f8917792640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8910069800 0x7f8910069ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:09.956 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.951+0000 7f8917792640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f891006a1e0 con 0x7f89100ced20 2023-05-22T16:06:09.957 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.951+0000 7f8917792640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89100cc5d0 con 0x7f89100ce950 2023-05-22T16:06:09.957 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.951+0000 7f8917792640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f891015d310 con 0x7f8910069800 2023-05-22T16:06:09.957 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.951+0000 7f8914d2d640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f89100ced20 0x7f89100cadc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:09.957 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.951+0000 7f8914d2d640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f89100ced20 0x7f89100cadc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57722/0 (socket says 172.21.15.16:57722) 2023-05-22T16:06:09.957 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.951+0000 7f8914d2d640 1 -- 172.21.15.16:0/797790673 learned_addr learned my addr 172.21.15.16:0/797790673 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:06:09.958 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.951+0000 7f8915d2f640 1 --2- 172.21.15.16:0/797790673 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8910069800 0x7f8910069ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:09.958 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.951+0000 7f891552e640 1 --2- 172.21.15.16:0/797790673 >> v2:172.21.15.62:3300/0 conn(0x7f89100ce950 0x7f89100ca880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:09.958 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.951+0000 7f8914d2d640 1 -- 172.21.15.16:0/797790673 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8910069800 msgr2=0x7f8910069ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:06:09.958 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.951+0000 7f8914d2d640 1 --2- 172.21.15.16:0/797790673 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8910069800 0x7f8910069ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:09.958 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.951+0000 7f8914d2d640 1 -- 172.21.15.16:0/797790673 >> v2:172.21.15.62:3300/0 conn(0x7f89100ce950 msgr2=0x7f89100ca880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:06:09.958 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.951+0000 7f8914d2d640 1 --2- 172.21.15.16:0/797790673 >> v2:172.21.15.62:3300/0 conn(0x7f89100ce950 0x7f89100ca880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:09.958 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.951+0000 7f8914d2d640 1 -- 172.21.15.16:0/797790673 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89100cbca0 con 0x7f89100ced20 2023-05-22T16:06:09.960 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.955+0000 7f8914d2d640 1 --2- 172.21.15.16:0/797790673 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f89100ced20 0x7f89100cadc0 secure :-1 s=READY pgs=2652 cs=0 l=1 rev1=1 crypto rx=0x7f891006a5d0 tx=0x7f890001c440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7464ae5f2a56c418 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:06:09.960 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.955+0000 7f8907fff640 1 -- 172.21.15.16:0/797790673 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89000ab600 con 0x7f89100ced20 2023-05-22T16:06:09.960 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.955+0000 7f8907fff640 1 -- 172.21.15.16:0/797790673 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89000ab760 con 0x7f89100ced20 2023-05-22T16:06:09.960 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.955+0000 7f8907fff640 1 -- 172.21.15.16:0/797790673 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89000abaf0 con 0x7f89100ced20 2023-05-22T16:06:09.960 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.955+0000 7f8917792640 1 -- 172.21.15.16:0/797790673 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f89100ced20 msgr2=0x7f89100cadc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:06:09.960 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.955+0000 7f8917792640 1 --2- 172.21.15.16:0/797790673 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f89100ced20 0x7f89100cadc0 secure :-1 s=READY pgs=2652 cs=0 l=1 rev1=1 crypto rx=0x7f891006a5d0 tx=0x7f890001c440 comp rx=0 tx=0).stop 2023-05-22T16:06:09.961 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.955+0000 7f8917792640 1 -- 172.21.15.16:0/797790673 shutdown_connections 2023-05-22T16:06:09.961 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.955+0000 7f8917792640 1 --2- 172.21.15.16:0/797790673 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f89100ced20 0x7f89100cadc0 unknown :-1 s=CLOSED pgs=2652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:09.961 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.955+0000 7f8917792640 1 --2- 172.21.15.16:0/797790673 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8910069800 0x7f8910069ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:09.961 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.955+0000 7f8917792640 1 --2- 172.21.15.16:0/797790673 >> v2:172.21.15.62:3300/0 conn(0x7f89100ce950 0x7f89100ca880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:09.961 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.955+0000 7f8917792640 1 -- 172.21.15.16:0/797790673 >> 172.21.15.16:0/797790673 conn(0x7f89100b0d50 msgr2=0x7f89100b1150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:06:09.961 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.955+0000 7f8917792640 1 -- 172.21.15.16:0/797790673 shutdown_connections 2023-05-22T16:06:09.961 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.955+0000 7f8917792640 1 -- 172.21.15.16:0/797790673 wait complete. 2023-05-22T16:06:09.961 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.955+0000 7f8917792640 1 Processor -- start 2023-05-22T16:06:09.962 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.955+0000 7f8917792640 1 -- start start 2023-05-22T16:06:09.964 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.959+0000 7f8917792640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8910069800 0x7f89100b2050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:09.964 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.959+0000 7f8917792640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f89100ce950 0x7f89100b2590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:09.964 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.959+0000 7f8917792640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f89100ced20 0x7f8910154b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:09.965 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.959+0000 7f8917792640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89100cb930 con 0x7f89100ced20 2023-05-22T16:06:09.965 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.959+0000 7f8917792640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89100b2bc0 con 0x7f89100ce950 2023-05-22T16:06:09.965 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.959+0000 7f8917792640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89100b2ea0 con 0x7f8910069800 2023-05-22T16:06:09.965 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.959+0000 7f8915d2f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f89100ced20 0x7f8910154b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:09.965 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.959+0000 7f8915d2f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f89100ced20 0x7f8910154b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57724/0 (socket says 172.21.15.16:57724) 2023-05-22T16:06:09.965 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.959+0000 7f8915d2f640 1 -- 172.21.15.16:0/439587447 learned_addr learned my addr 172.21.15.16:0/439587447 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:06:09.965 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.959+0000 7f891552e640 1 --2- 172.21.15.16:0/439587447 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8910069800 0x7f89100b2050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:09.966 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.959+0000 7f8914d2d640 1 --2- 172.21.15.16:0/439587447 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f89100ce950 0x7f89100b2590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:09.966 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.959+0000 7f8915d2f640 1 -- 172.21.15.16:0/439587447 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8910069800 msgr2=0x7f89100b2050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:06:09.966 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.959+0000 7f8915d2f640 1 --2- 172.21.15.16:0/439587447 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8910069800 0x7f89100b2050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:09.966 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.959+0000 7f8915d2f640 1 -- 172.21.15.16:0/439587447 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f89100ce950 msgr2=0x7f89100b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:06:09.966 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.959+0000 7f8915d2f640 1 --2- 172.21.15.16:0/439587447 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f89100ce950 0x7f89100b2590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:09.966 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.959+0000 7f8915d2f640 1 -- 172.21.15.16:0/439587447 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89000094d0 con 0x7f89100ced20 2023-05-22T16:06:09.966 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.959+0000 7f8914d2d640 1 --2- 172.21.15.16:0/439587447 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f89100ce950 0x7f89100b2590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T16:06:09.967 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.959+0000 7f8915d2f640 1 --2- 172.21.15.16:0/439587447 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f89100ced20 0x7f8910154b80 secure :-1 s=READY pgs=2653 cs=0 l=1 rev1=1 crypto rx=0x7f8908007e60 tx=0x7f890800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:06:09.967 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.959+0000 7f8905ffb640 1 -- 172.21.15.16:0/439587447 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8908014420 con 0x7f89100ced20 2023-05-22T16:06:09.968 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.959+0000 7f8905ffb640 1 -- 172.21.15.16:0/439587447 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89080145a0 con 0x7f89100ced20 2023-05-22T16:06:09.968 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.959+0000 7f8917792640 1 -- 172.21.15.16:0/439587447 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f891015cdd0 con 0x7f89100ced20 2023-05-22T16:06:09.968 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.959+0000 7f8905ffb640 1 -- 172.21.15.16:0/439587447 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8908014950 con 0x7f89100ced20 2023-05-22T16:06:09.968 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.959+0000 7f8917792640 1 -- 172.21.15.16:0/439587447 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89100cca40 con 0x7f89100ced20 2023-05-22T16:06:09.968 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.959+0000 7f8917792640 1 -- 172.21.15.16:0/439587447 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f891015cf60 con 0x7f89100ced20 2023-05-22T16:06:09.968 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.959+0000 7f8905ffb640 1 -- 172.21.15.16:0/439587447 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 100) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f8908014ab0 con 0x7f89100ced20 2023-05-22T16:06:09.969 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.959+0000 7f8905ffb640 1 --2- 172.21.15.16:0/439587447 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f88f40654d0 0x7f88f4067980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:09.969 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.959+0000 7f8905ffb640 1 -- 172.21.15.16:0/439587447 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(201..201 src has 1..201) v4 ==== 6317+0+0 (secure 0 0 0) 0x7f8908084160 con 0x7f89100ced20 2023-05-22T16:06:09.969 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.959+0000 7f891552e640 1 --2- 172.21.15.16:0/439587447 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f88f40654d0 0x7f88f4067980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:09.969 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.959+0000 7f8905ffb640 1 --2- 172.21.15.16:0/439587447 >> [v2:172.21.15.62:6800/37352,v1:172.21.15.62:6801/37352] conn(0x7f88f406c380 0x7f88f406e7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:09.969 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.959+0000 7f8905ffb640 1 -- 172.21.15.16:0/439587447 --> [v2:172.21.15.62:6800/37352,v1:172.21.15.62:6801/37352] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f88f406eec0 con 0x7f88f406c380 2023-05-22T16:06:09.969 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.959+0000 7f8905ffb640 1 -- 172.21.15.16:0/439587447 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_get_version_reply(handle=1 version=201) v2 ==== 24+0+0 (secure 0 0 0) 0x7f89080844d0 con 0x7f89100ced20 2023-05-22T16:06:09.970 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.959+0000 7f891552e640 1 --2- 172.21.15.16:0/439587447 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f88f40654d0 0x7f88f4067980 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f8910159b30 tx=0x7f890c008040 comp rx=0 tx=0).ready entity=mgr.8195 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:06:09.970 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.959+0000 7f8914d2d640 1 --2- 172.21.15.16:0/439587447 >> [v2:172.21.15.62:6800/37352,v1:172.21.15.62:6801/37352] conn(0x7f88f406c380 0x7f88f406e7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:09.971 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.963+0000 7f8914d2d640 1 --2- 172.21.15.16:0/439587447 >> [v2:172.21.15.62:6800/37352,v1:172.21.15.62:6801/37352] conn(0x7f88f406c380 0x7f88f406e7f0 crc :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:06:09.971 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.963+0000 7f8905ffb640 1 -- 172.21.15.16:0/439587447 <== osd.5 v2:172.21.15.62:6800/37352 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f88f406eec0 con 0x7f88f406c380 2023-05-22T16:06:09.979 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.975+0000 7f8917792640 1 -- 172.21.15.16:0/439587447 --> [v2:172.21.15.62:6800/37352,v1:172.21.15.62:6801/37352] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f891004dcc0 con 0x7f88f406c380 2023-05-22T16:06:09.980 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.975+0000 7f8905ffb640 1 -- 172.21.15.16:0/439587447 <== osd.5 v2:172.21.15.62:6800/37352 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f891004dcc0 con 0x7f88f406c380 2023-05-22T16:06:09.981 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-05-22T16:06:09.981 INFO:teuthology.orchestra.run.smithi016.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-05-22T16:06:09.981 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-05-22T16:06:09.982 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.975+0000 7f8917792640 1 -- 172.21.15.16:0/439587447 >> [v2:172.21.15.62:6800/37352,v1:172.21.15.62:6801/37352] conn(0x7f88f406c380 msgr2=0x7f88f406e7f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:06:09.982 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.975+0000 7f8917792640 1 --2- 172.21.15.16:0/439587447 >> [v2:172.21.15.62:6800/37352,v1:172.21.15.62:6801/37352] conn(0x7f88f406c380 0x7f88f406e7f0 crc :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:09.982 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.979+0000 7f8917792640 1 -- 172.21.15.16:0/439587447 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f88f40654d0 msgr2=0x7f88f4067980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:06:09.983 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.979+0000 7f8917792640 1 --2- 172.21.15.16:0/439587447 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f88f40654d0 0x7f88f4067980 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f8910159b30 tx=0x7f890c008040 comp rx=0 tx=0).stop 2023-05-22T16:06:09.983 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.979+0000 7f8917792640 1 -- 172.21.15.16:0/439587447 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f89100ced20 msgr2=0x7f8910154b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:06:09.983 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.979+0000 7f8917792640 1 --2- 172.21.15.16:0/439587447 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f89100ced20 0x7f8910154b80 secure :-1 s=READY pgs=2653 cs=0 l=1 rev1=1 crypto rx=0x7f8908007e60 tx=0x7f890800e4b0 comp rx=0 tx=0).stop 2023-05-22T16:06:09.983 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.979+0000 7f8915d2f640 1 -- 172.21.15.16:0/439587447 reap_dead start 2023-05-22T16:06:09.983 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.979+0000 7f8917792640 1 -- 172.21.15.16:0/439587447 shutdown_connections 2023-05-22T16:06:09.983 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.979+0000 7f8917792640 1 -- 172.21.15.16:0/439587447 >> 172.21.15.16:0/439587447 conn(0x7f89100b0d50 msgr2=0x7f891006ec10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:06:09.984 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.979+0000 7f8917792640 1 -- 172.21.15.16:0/439587447 shutdown_connections 2023-05-22T16:06:09.984 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:09.979+0000 7f8917792640 1 -- 172.21.15.16:0/439587447 wait complete. 2023-05-22T16:06:09.996 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2023-05-22T16:06:10.110 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.107+0000 7f9aa128f640 1 Processor -- start 2023-05-22T16:06:10.111 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.107+0000 7f9aa128f640 1 -- start start 2023-05-22T16:06:10.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.107+0000 7f9aa128f640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f9a9c154170 0x7f9a9c06f630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:10.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.107+0000 7f9aa128f640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f9a9c06cc80 0x7f9a9c06be90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:10.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.107+0000 7f9aa128f640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a9c071b40 con 0x7f9a9c06c3d0 2023-05-22T16:06:10.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.107+0000 7f9aa128f640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a9c0ca9f0 con 0x7f9a9c154170 2023-05-22T16:06:10.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.107+0000 7f9aa128f640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a9c0ca150 con 0x7f9a9c06cc80 2023-05-22T16:06:10.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.107+0000 7f9a9affd640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f9a9c154170 0x7f9a9c06f630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:10.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.107+0000 7f9a9b7fe640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f9a9c06c3d0 0x7f9a9c0afb50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:46404/0 (socket says 172.21.15.16:46404) 2023-05-22T16:06:10.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.107+0000 7f9a9b7fe640 1 -- 172.21.15.16:0/2076787474 learned_addr learned my addr 172.21.15.16:0/2076787474 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T16:06:10.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.107+0000 7f9a9a7fc640 1 --2- 172.21.15.16:0/2076787474 >> v2:172.21.15.62:3300/0 conn(0x7f9a9c06cc80 0x7f9a9c06be90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:10.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.107+0000 7f9a99ffb640 1 -- 172.21.15.16:0/2076787474 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4142733024 0 0) 0x7f9a9c071b40 con 0x7f9a9c06c3d0 2023-05-22T16:06:10.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.107+0000 7f9a99ffb640 1 -- 172.21.15.16:0/2076787474 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a84003530 con 0x7f9a9c06c3d0 2023-05-22T16:06:10.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.107+0000 7f9a9affd640 1 -- 172.21.15.16:0/2076787474 >> v2:172.21.15.62:3300/0 conn(0x7f9a9c06cc80 msgr2=0x7f9a9c06be90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:06:10.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.107+0000 7f9a9affd640 1 --2- 172.21.15.16:0/2076787474 >> v2:172.21.15.62:3300/0 conn(0x7f9a9c06cc80 0x7f9a9c06be90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:10.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.107+0000 7f9a9affd640 1 -- 172.21.15.16:0/2076787474 >> v1:172.21.15.16:6789/0 conn(0x7f9a9c06c3d0 legacy=0x7f9a9c0afb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:06:10.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.107+0000 7f9a9affd640 1 -- 172.21.15.16:0/2076787474 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a9c0ca430 con 0x7f9a9c154170 2023-05-22T16:06:10.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.107+0000 7f9a9a7fc640 1 --2- 172.21.15.16:0/2076787474 >> v2:172.21.15.62:3300/0 conn(0x7f9a9c06cc80 0x7f9a9c06be90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T16:06:10.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.111+0000 7f9a9affd640 1 --2- 172.21.15.16:0/2076787474 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9a9c154170 0x7f9a9c06f630 secure :-1 s=READY pgs=2655 cs=0 l=1 rev1=1 crypto rx=0x7f9a9c0cbc50 tx=0x7f9a8c01c430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6255827bae7e974f server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:06:10.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.111+0000 7f9a99ffb640 1 -- 172.21.15.16:0/2076787474 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a8c0a7070 con 0x7f9a9c154170 2023-05-22T16:06:10.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.111+0000 7f9a99ffb640 1 -- 172.21.15.16:0/2076787474 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a8c0ab430 con 0x7f9a9c154170 2023-05-22T16:06:10.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.111+0000 7f9a99ffb640 1 -- 172.21.15.16:0/2076787474 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a8c0ab7c0 con 0x7f9a9c154170 2023-05-22T16:06:10.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.111+0000 7f9aa128f640 1 -- 172.21.15.16:0/2076787474 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9a9c154170 msgr2=0x7f9a9c06f630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:06:10.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.111+0000 7f9aa128f640 1 --2- 172.21.15.16:0/2076787474 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9a9c154170 0x7f9a9c06f630 secure :-1 s=READY pgs=2655 cs=0 l=1 rev1=1 crypto rx=0x7f9a9c0cbc50 tx=0x7f9a8c01c430 comp rx=0 tx=0).stop 2023-05-22T16:06:10.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.111+0000 7f9aa128f640 1 -- 172.21.15.16:0/2076787474 shutdown_connections 2023-05-22T16:06:10.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.111+0000 7f9aa128f640 1 --2- 172.21.15.16:0/2076787474 >> v2:172.21.15.62:3300/0 conn(0x7f9a9c06cc80 0x7f9a9c06be90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:10.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.111+0000 7f9aa128f640 1 --2- 172.21.15.16:0/2076787474 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9a9c154170 0x7f9a9c06f630 unknown :-1 s=CLOSED pgs=2655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:10.118 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.111+0000 7f9aa128f640 1 -- 172.21.15.16:0/2076787474 >> 172.21.15.16:0/2076787474 conn(0x7f9a9c0cab50 msgr2=0x7f9a9c0caf50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:06:10.118 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.111+0000 7f9aa128f640 1 -- 172.21.15.16:0/2076787474 shutdown_connections 2023-05-22T16:06:10.118 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.111+0000 7f9aa128f640 1 -- 172.21.15.16:0/2076787474 wait complete. 2023-05-22T16:06:10.120 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.115+0000 7f9aa128f640 1 Processor -- start 2023-05-22T16:06:10.120 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.115+0000 7f9aa128f640 1 -- start start 2023-05-22T16:06:10.120 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.115+0000 7f9aa128f640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9a9c06c3d0 0x7f9a9c14cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:10.120 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.115+0000 7f9aa128f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9a9c06cc80 0x7f9a9c0cbca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:10.120 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.115+0000 7f9aa128f640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9a9c154170 0x7f9a9c1524c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:10.121 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.115+0000 7f9aa128f640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a9c0cd370 con 0x7f9a9c06cc80 2023-05-22T16:06:10.121 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.115+0000 7f9aa128f640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a9c0cc760 con 0x7f9a9c154170 2023-05-22T16:06:10.121 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.115+0000 7f9aa128f640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a9c0cca40 con 0x7f9a9c06c3d0 2023-05-22T16:06:10.121 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.115+0000 7f9a9a7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9a9c06cc80 0x7f9a9c0cbca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:10.121 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.115+0000 7f9a9a7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9a9c06cc80 0x7f9a9c0cbca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57738/0 (socket says 172.21.15.16:57738) 2023-05-22T16:06:10.121 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.115+0000 7f9a9a7fc640 1 -- 172.21.15.16:0/189621434 learned_addr learned my addr 172.21.15.16:0/189621434 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:06:10.121 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.115+0000 7f9a9b7fe640 1 --2- 172.21.15.16:0/189621434 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9a9c154170 0x7f9a9c1524c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:10.122 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.115+0000 7f9a9affd640 1 --2- 172.21.15.16:0/189621434 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9a9c06c3d0 0x7f9a9c14cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:10.122 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.115+0000 7f9a9a7fc640 1 -- 172.21.15.16:0/189621434 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9a9c06c3d0 msgr2=0x7f9a9c14cdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:06:10.122 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.115+0000 7f9a9a7fc640 1 --2- 172.21.15.16:0/189621434 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9a9c06c3d0 0x7f9a9c14cdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:10.122 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.115+0000 7f9a9a7fc640 1 -- 172.21.15.16:0/189621434 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9a9c154170 msgr2=0x7f9a9c1524c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:06:10.122 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.115+0000 7f9a9a7fc640 1 --2- 172.21.15.16:0/189621434 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9a9c154170 0x7f9a9c1524c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:10.122 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.115+0000 7f9a9a7fc640 1 -- 172.21.15.16:0/189621434 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a8c009420 con 0x7f9a9c06cc80 2023-05-22T16:06:10.122 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.115+0000 7f9a9b7fe640 1 --2- 172.21.15.16:0/189621434 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9a9c154170 0x7f9a9c1524c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T16:06:10.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.115+0000 7f9a9a7fc640 1 --2- 172.21.15.16:0/189621434 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9a9c06cc80 0x7f9a9c0cbca0 secure :-1 s=READY pgs=2656 cs=0 l=1 rev1=1 crypto rx=0x7f9a9000e810 tx=0x7f9a9000ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:06:10.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.115+0000 7f9a837fe640 1 -- 172.21.15.16:0/189621434 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a9000c6d0 con 0x7f9a9c06cc80 2023-05-22T16:06:10.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.115+0000 7f9a837fe640 1 -- 172.21.15.16:0/189621434 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a9000c830 con 0x7f9a9c06cc80 2023-05-22T16:06:10.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.115+0000 7f9aa128f640 1 -- 172.21.15.16:0/189621434 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a9c0c9d90 con 0x7f9a9c06cc80 2023-05-22T16:06:10.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.115+0000 7f9a837fe640 1 -- 172.21.15.16:0/189621434 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a9000cbe0 con 0x7f9a9c06cc80 2023-05-22T16:06:10.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.115+0000 7f9aa128f640 1 -- 172.21.15.16:0/189621434 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a9c154540 con 0x7f9a9c06cc80 2023-05-22T16:06:10.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.115+0000 7f9aa128f640 1 -- 172.21.15.16:0/189621434 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a9c0c9f20 con 0x7f9a9c06cc80 2023-05-22T16:06:10.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.115+0000 7f9a837fe640 1 -- 172.21.15.16:0/189621434 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 100) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f9a9000cd40 con 0x7f9a9c06cc80 2023-05-22T16:06:10.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.119+0000 7f9a837fe640 1 --2- 172.21.15.16:0/189621434 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f9a880654d0 0x7f9a88067980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:10.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.119+0000 7f9a9affd640 1 --2- 172.21.15.16:0/189621434 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f9a880654d0 0x7f9a88067980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:10.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.119+0000 7f9a837fe640 1 -- 172.21.15.16:0/189621434 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(201..201 src has 1..201) v4 ==== 6317+0+0 (secure 0 0 0) 0x7f9a90082b40 con 0x7f9a9c06cc80 2023-05-22T16:06:10.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.119+0000 7f9a9affd640 1 --2- 172.21.15.16:0/189621434 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f9a880654d0 0x7f9a88067980 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f9a8c0af000 tx=0x7f9a8c008040 comp rx=0 tx=0).ready entity=mgr.8195 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:06:10.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.123+0000 7f9a837fe640 1 -- 172.21.15.16:0/189621434 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f9a90050400 con 0x7f9a9c06cc80 2023-05-22T16:06:10.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.275+0000 7f9aa128f640 1 -- 172.21.15.16:0/189621434 --> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) v1 -- 0x7f9a9c073830 con 0x7f9a880654d0 2023-05-22T16:06:10.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.275+0000 7f9a837fe640 1 -- 172.21.15.16:0/189621434 <== mgr.8195 v2:172.21.15.16:6832/65890 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f9a9c073830 con 0x7f9a880654d0 2023-05-22T16:06:10.280 INFO:teuthology.orchestra.run.smithi016.stderr:instructed osd(s) 5 to deep-scrub 2023-05-22T16:06:10.282 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.279+0000 7f9aa128f640 1 -- 172.21.15.16:0/189621434 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f9a880654d0 msgr2=0x7f9a88067980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:06:10.282 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.279+0000 7f9aa128f640 1 --2- 172.21.15.16:0/189621434 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f9a880654d0 0x7f9a88067980 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f9a8c0af000 tx=0x7f9a8c008040 comp rx=0 tx=0).stop 2023-05-22T16:06:10.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.279+0000 7f9aa128f640 1 -- 172.21.15.16:0/189621434 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9a9c06cc80 msgr2=0x7f9a9c0cbca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:06:10.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.279+0000 7f9aa128f640 1 --2- 172.21.15.16:0/189621434 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9a9c06cc80 0x7f9a9c0cbca0 secure :-1 s=READY pgs=2656 cs=0 l=1 rev1=1 crypto rx=0x7f9a9000e810 tx=0x7f9a9000ecd0 comp rx=0 tx=0).stop 2023-05-22T16:06:10.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.279+0000 7f9aa128f640 1 -- 172.21.15.16:0/189621434 shutdown_connections 2023-05-22T16:06:10.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.279+0000 7f9aa128f640 1 --2- 172.21.15.16:0/189621434 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9a9c154170 0x7f9a9c1524c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:10.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.279+0000 7f9aa128f640 1 --2- 172.21.15.16:0/189621434 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f9a880654d0 0x7f9a88067980 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:10.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.279+0000 7f9aa128f640 1 --2- 172.21.15.16:0/189621434 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9a9c06cc80 0x7f9a9c0cbca0 unknown :-1 s=CLOSED pgs=2656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:10.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.279+0000 7f9aa128f640 1 --2- 172.21.15.16:0/189621434 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9a9c06c3d0 0x7f9a9c14cdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:10.285 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.279+0000 7f9aa128f640 1 -- 172.21.15.16:0/189621434 >> 172.21.15.16:0/189621434 conn(0x7f9a9c0cab50 msgr2=0x7f9a9c0b0f00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:06:10.285 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.279+0000 7f9aa128f640 1 -- 172.21.15.16:0/189621434 shutdown_connections 2023-05-22T16:06:10.285 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.279+0000 7f9aa128f640 1 -- 172.21.15.16:0/189621434 wait complete. 2023-05-22T16:06:10.298 INFO:tasks.ceph:Scrubbing osd.6 2023-05-22T16:06:10.299 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2023-05-22T16:06:10.425 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.423+0000 7f8dcb304640 1 Processor -- start 2023-05-22T16:06:10.426 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.423+0000 7f8dcb304640 1 -- start start 2023-05-22T16:06:10.428 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.423+0000 7f8dcb304640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8dc4159a50 0x7f8dc4070d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:10.428 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.423+0000 7f8dcb304640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f8dc406c060 0x7f8dc40712c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:10.428 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.423+0000 7f8dcb304640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8dc4071800 con 0x7f8dc40cea90 2023-05-22T16:06:10.428 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.423+0000 7f8dcb304640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8dc415d800 con 0x7f8dc406c060 2023-05-22T16:06:10.428 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.423+0000 7f8dcb304640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8dc415dae0 con 0x7f8dc4159a50 2023-05-22T16:06:10.428 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.423+0000 7f8dc889f640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f8dc40cea90 0x7f8dc40c8740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:46410/0 (socket says 172.21.15.16:46410) 2023-05-22T16:06:10.429 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.423+0000 7f8dc889f640 1 -- 172.21.15.16:0/462795239 learned_addr learned my addr 172.21.15.16:0/462795239 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T16:06:10.429 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.423+0000 7f8dc98a1640 1 --2- 172.21.15.16:0/462795239 >> v2:172.21.15.62:3300/0 conn(0x7f8dc406c060 0x7f8dc40712c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:10.429 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.423+0000 7f8dc90a0640 1 --2- 172.21.15.16:0/462795239 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8dc4159a50 0x7f8dc4070d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:10.429 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.423+0000 7f8dbbfff640 1 -- 172.21.15.16:0/462795239 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2416278009 0 0) 0x7f8dc4071800 con 0x7f8dc40cea90 2023-05-22T16:06:10.429 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.423+0000 7f8dbbfff640 1 -- 172.21.15.16:0/462795239 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8db0003530 con 0x7f8dc40cea90 2023-05-22T16:06:10.429 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.423+0000 7f8dbbfff640 1 -- 172.21.15.16:0/462795239 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1915622570 0 0) 0x7f8db0003530 con 0x7f8dc40cea90 2023-05-22T16:06:10.429 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.423+0000 7f8dbbfff640 1 -- 172.21.15.16:0/462795239 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8dc4159a50 msgr2=0x7f8dc4070d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:06:10.430 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.423+0000 7f8dbbfff640 1 --2- 172.21.15.16:0/462795239 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8dc4159a50 0x7f8dc4070d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:10.430 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.423+0000 7f8dbbfff640 1 -- 172.21.15.16:0/462795239 >> v2:172.21.15.62:3300/0 conn(0x7f8dc406c060 msgr2=0x7f8dc40712c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:06:10.430 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.423+0000 7f8dbbfff640 1 --2- 172.21.15.16:0/462795239 >> v2:172.21.15.62:3300/0 conn(0x7f8dc406c060 0x7f8dc40712c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:10.430 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.423+0000 7f8dbbfff640 1 -- 172.21.15.16:0/462795239 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8dc415ddc0 con 0x7f8dc40cea90 2023-05-22T16:06:10.430 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.423+0000 7f8dbbfff640 1 -- 172.21.15.16:0/462795239 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7f8db4001680 con 0x7f8dc40cea90 2023-05-22T16:06:10.430 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.423+0000 7f8dbbfff640 1 -- 172.21.15.16:0/462795239 >> v1:172.21.15.16:6789/0 conn(0x7f8dc40cea90 legacy=0x7f8dc40c8740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:06:10.430 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.423+0000 7f8dbbfff640 1 --2- 172.21.15.16:0/462795239 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8db0003a40 0x7f8db0003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:10.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.423+0000 7f8dbbfff640 1 --2- 172.21.15.16:0/462795239 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8dc4159a50 0x7f8db00047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:10.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.423+0000 7f8dbbfff640 1 --2- 172.21.15.16:0/462795239 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8db0004d10 0x7f8db0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:10.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.423+0000 7f8dbbfff640 1 -- 172.21.15.16:0/462795239 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8dc415d800 con 0x7f8db0003a40 2023-05-22T16:06:10.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.423+0000 7f8dbbfff640 1 -- 172.21.15.16:0/462795239 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8dc415dae0 con 0x7f8db0004d10 2023-05-22T16:06:10.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.423+0000 7f8dbbfff640 1 -- 172.21.15.16:0/462795239 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8db000ba40 con 0x7f8dc4159a50 2023-05-22T16:06:10.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.423+0000 7f8dc90a0640 1 --2- 172.21.15.16:0/462795239 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8db0003a40 0x7f8db0003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:10.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.423+0000 7f8dc98a1640 1 --2- 172.21.15.16:0/462795239 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8db0004d10 0x7f8db0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:10.432 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.423+0000 7f8dc90a0640 1 -- 172.21.15.16:0/462795239 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8dc4159a50 msgr2=0x7f8db00047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:06:10.432 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.423+0000 7f8dc90a0640 1 --2- 172.21.15.16:0/462795239 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8dc4159a50 0x7f8db00047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:10.432 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.423+0000 7f8dc90a0640 1 -- 172.21.15.16:0/462795239 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8db0004d10 msgr2=0x7f8db0005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:06:10.432 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.423+0000 7f8dc90a0640 1 --2- 172.21.15.16:0/462795239 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8db0004d10 0x7f8db0005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:10.432 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.423+0000 7f8dc90a0640 1 -- 172.21.15.16:0/462795239 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8db0003530 con 0x7f8db0003a40 2023-05-22T16:06:10.432 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.423+0000 7f8dc98a1640 1 --2- 172.21.15.16:0/462795239 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8db0004d10 0x7f8db0005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T16:06:10.434 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.427+0000 7f8dc90a0640 1 --2- 172.21.15.16:0/462795239 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8db0003a40 0x7f8db0003e50 secure :-1 s=READY pgs=2658 cs=0 l=1 rev1=1 crypto rx=0x7f8dc0009ae0 tx=0x7f8dc001ca90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52d07adf9f253ea6 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:06:10.435 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.427+0000 7f8dbbfff640 1 -- 172.21.15.16:0/462795239 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dc00a6470 con 0x7f8db0003a40 2023-05-22T16:06:10.435 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.427+0000 7f8dbbfff640 1 -- 172.21.15.16:0/462795239 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8dc00a65d0 con 0x7f8db0003a40 2023-05-22T16:06:10.435 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.427+0000 7f8dbbfff640 1 -- 172.21.15.16:0/462795239 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dc00a6980 con 0x7f8db0003a40 2023-05-22T16:06:10.435 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.427+0000 7f8dcb304640 1 -- 172.21.15.16:0/462795239 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8db0003a40 msgr2=0x7f8db0003e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:06:10.435 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.427+0000 7f8dcb304640 1 --2- 172.21.15.16:0/462795239 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8db0003a40 0x7f8db0003e50 secure :-1 s=READY pgs=2658 cs=0 l=1 rev1=1 crypto rx=0x7f8dc0009ae0 tx=0x7f8dc001ca90 comp rx=0 tx=0).stop 2023-05-22T16:06:10.435 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.427+0000 7f8dc98a1640 1 -- 172.21.15.16:0/462795239 reap_dead start 2023-05-22T16:06:10.435 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.427+0000 7f8dcb304640 1 -- 172.21.15.16:0/462795239 shutdown_connections 2023-05-22T16:06:10.436 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.427+0000 7f8dcb304640 1 -- 172.21.15.16:0/462795239 >> 172.21.15.16:0/462795239 conn(0x7f8dc4069800 msgr2=0x7f8dc4069c00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:06:10.436 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.427+0000 7f8dcb304640 1 -- 172.21.15.16:0/462795239 shutdown_connections 2023-05-22T16:06:10.436 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.427+0000 7f8dcb304640 1 -- 172.21.15.16:0/462795239 wait complete. 2023-05-22T16:06:10.436 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.427+0000 7f8dcb304640 1 Processor -- start 2023-05-22T16:06:10.436 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.427+0000 7f8dcb304640 1 -- start start 2023-05-22T16:06:10.436 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.431+0000 7f8dcb304640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8db0003a40 0x7f8dc41f30f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:10.436 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.431+0000 7f8dcb304640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8dc40b33e0 0x7f8dc41546e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:10.436 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.431+0000 7f8dcb304640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8dc40b0f60 0x7f8dc41573f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:10.437 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.431+0000 7f8dcb304640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8dc4154c20 con 0x7f8dc40b0f60 2023-05-22T16:06:10.437 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.431+0000 7f8dcb304640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8dc41557b0 con 0x7f8dc40b33e0 2023-05-22T16:06:10.437 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.431+0000 7f8dcb304640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8dc4156330 con 0x7f8db0003a40 2023-05-22T16:06:10.437 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.431+0000 7f8dc98a1640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8dc40b0f60 0x7f8dc41573f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:10.437 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.431+0000 7f8dc98a1640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8dc40b0f60 0x7f8dc41573f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57756/0 (socket says 172.21.15.16:57756) 2023-05-22T16:06:10.437 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.431+0000 7f8dc98a1640 1 -- 172.21.15.16:0/828252563 learned_addr learned my addr 172.21.15.16:0/828252563 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:06:10.437 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.431+0000 7f8dc90a0640 1 --2- 172.21.15.16:0/828252563 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8db0003a40 0x7f8dc41f30f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:10.438 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.431+0000 7f8dc889f640 1 --2- 172.21.15.16:0/828252563 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8dc40b33e0 0x7f8dc41546e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:10.438 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.431+0000 7f8dc98a1640 1 -- 172.21.15.16:0/828252563 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8db0003a40 msgr2=0x7f8dc41f30f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:06:10.438 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.431+0000 7f8dc98a1640 1 --2- 172.21.15.16:0/828252563 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8db0003a40 0x7f8dc41f30f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:10.438 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.431+0000 7f8dc98a1640 1 -- 172.21.15.16:0/828252563 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8dc40b33e0 msgr2=0x7f8dc41546e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:06:10.438 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.431+0000 7f8dc98a1640 1 --2- 172.21.15.16:0/828252563 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8dc40b33e0 0x7f8dc41546e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:10.438 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.431+0000 7f8dc98a1640 1 -- 172.21.15.16:0/828252563 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8dc40c9db0 con 0x7f8dc40b0f60 2023-05-22T16:06:10.438 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.431+0000 7f8dc98a1640 1 --2- 172.21.15.16:0/828252563 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8dc40b0f60 0x7f8dc41573f0 secure :-1 s=READY pgs=2659 cs=0 l=1 rev1=1 crypto rx=0x7f8dc40cdff0 tx=0x7f8dbc007f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:06:10.438 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.431+0000 7f8db9ffb640 1 -- 172.21.15.16:0/828252563 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dbc00f840 con 0x7f8dc40b0f60 2023-05-22T16:06:10.439 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.431+0000 7f8db9ffb640 1 -- 172.21.15.16:0/828252563 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8dbc00f9a0 con 0x7f8dc40b0f60 2023-05-22T16:06:10.439 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.431+0000 7f8db9ffb640 1 -- 172.21.15.16:0/828252563 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dbc00fd30 con 0x7f8dc40b0f60 2023-05-22T16:06:10.439 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.431+0000 7f8dcb304640 1 -- 172.21.15.16:0/828252563 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8dc4071a20 con 0x7f8dc40b0f60 2023-05-22T16:06:10.439 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.431+0000 7f8dcb304640 1 -- 172.21.15.16:0/828252563 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8dc4155a00 con 0x7f8dc40b0f60 2023-05-22T16:06:10.439 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.431+0000 7f8dcb304640 1 -- 172.21.15.16:0/828252563 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f8dc40c9f40 con 0x7f8dc40b0f60 2023-05-22T16:06:10.440 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.431+0000 7f8db9ffb640 1 -- 172.21.15.16:0/828252563 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 100) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f8dbc029430 con 0x7f8dc40b0f60 2023-05-22T16:06:10.440 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.431+0000 7f8db9ffb640 1 --2- 172.21.15.16:0/828252563 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f8da80654a0 0x7f8da8067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:10.441 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.431+0000 7f8db9ffb640 1 -- 172.21.15.16:0/828252563 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(201..201 src has 1..201) v4 ==== 6317+0+0 (secure 0 0 0) 0x7f8dbc083ad0 con 0x7f8dc40b0f60 2023-05-22T16:06:10.441 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.431+0000 7f8dc90a0640 1 --2- 172.21.15.16:0/828252563 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f8da80654a0 0x7f8da8067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:10.441 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.431+0000 7f8db9ffb640 1 --2- 172.21.15.16:0/828252563 >> [v2:172.21.15.62:6818/37354,v1:172.21.15.62:6820/37354] conn(0x7f8da806c350 0x7f8da806e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:10.441 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.431+0000 7f8db9ffb640 1 -- 172.21.15.16:0/828252563 --> [v2:172.21.15.62:6818/37354,v1:172.21.15.62:6820/37354] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8da806ee90 con 0x7f8da806c350 2023-05-22T16:06:10.441 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.431+0000 7f8db9ffb640 1 -- 172.21.15.16:0/828252563 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_get_version_reply(handle=1 version=201) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8dbc083f20 con 0x7f8dc40b0f60 2023-05-22T16:06:10.441 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.431+0000 7f8dc90a0640 1 --2- 172.21.15.16:0/828252563 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f8da80654a0 0x7f8da8067950 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f8dc00ad000 tx=0x7f8dc00046d0 comp rx=0 tx=0).ready entity=mgr.8195 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:06:10.441 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.431+0000 7f8dc889f640 1 --2- 172.21.15.16:0/828252563 >> [v2:172.21.15.62:6818/37354,v1:172.21.15.62:6820/37354] conn(0x7f8da806c350 0x7f8da806e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:10.441 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.435+0000 7f8dc889f640 1 --2- 172.21.15.16:0/828252563 >> [v2:172.21.15.62:6818/37354,v1:172.21.15.62:6820/37354] conn(0x7f8da806c350 0x7f8da806e7c0 crc :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:06:10.442 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.435+0000 7f8db9ffb640 1 -- 172.21.15.16:0/828252563 <== osd.6 v2:172.21.15.62:6818/37354 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f8da806ee90 con 0x7f8da806c350 2023-05-22T16:06:10.451 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.447+0000 7f8dcb304640 1 -- 172.21.15.16:0/828252563 --> [v2:172.21.15.62:6818/37354,v1:172.21.15.62:6820/37354] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f8dc41564c0 con 0x7f8da806c350 2023-05-22T16:06:10.452 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.447+0000 7f8db9ffb640 1 -- 172.21.15.16:0/828252563 <== osd.6 v2:172.21.15.62:6818/37354 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f8dc41564c0 con 0x7f8da806c350 2023-05-22T16:06:10.452 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-05-22T16:06:10.453 INFO:teuthology.orchestra.run.smithi016.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-05-22T16:06:10.453 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-05-22T16:06:10.453 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.451+0000 7f8dcb304640 1 -- 172.21.15.16:0/828252563 >> [v2:172.21.15.62:6818/37354,v1:172.21.15.62:6820/37354] conn(0x7f8da806c350 msgr2=0x7f8da806e7c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:06:10.454 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.451+0000 7f8dcb304640 1 --2- 172.21.15.16:0/828252563 >> [v2:172.21.15.62:6818/37354,v1:172.21.15.62:6820/37354] conn(0x7f8da806c350 0x7f8da806e7c0 crc :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:10.454 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.451+0000 7f8dcb304640 1 -- 172.21.15.16:0/828252563 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f8da80654a0 msgr2=0x7f8da8067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:06:10.454 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.451+0000 7f8dcb304640 1 --2- 172.21.15.16:0/828252563 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f8da80654a0 0x7f8da8067950 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f8dc00ad000 tx=0x7f8dc00046d0 comp rx=0 tx=0).stop 2023-05-22T16:06:10.454 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.451+0000 7f8dcb304640 1 -- 172.21.15.16:0/828252563 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8dc40b0f60 msgr2=0x7f8dc41573f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:06:10.454 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.451+0000 7f8dcb304640 1 --2- 172.21.15.16:0/828252563 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8dc40b0f60 0x7f8dc41573f0 secure :-1 s=READY pgs=2659 cs=0 l=1 rev1=1 crypto rx=0x7f8dc40cdff0 tx=0x7f8dbc007f50 comp rx=0 tx=0).stop 2023-05-22T16:06:10.454 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.451+0000 7f8dc98a1640 1 -- 172.21.15.16:0/828252563 reap_dead start 2023-05-22T16:06:10.454 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.451+0000 7f8dcb304640 1 -- 172.21.15.16:0/828252563 shutdown_connections 2023-05-22T16:06:10.455 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.451+0000 7f8dcb304640 1 -- 172.21.15.16:0/828252563 >> 172.21.15.16:0/828252563 conn(0x7f8dc4069800 msgr2=0x7f8dc415d040 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:06:10.455 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.451+0000 7f8dcb304640 1 -- 172.21.15.16:0/828252563 shutdown_connections 2023-05-22T16:06:10.455 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.451+0000 7f8dcb304640 1 -- 172.21.15.16:0/828252563 wait complete. 2023-05-22T16:06:10.468 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2023-05-22T16:06:10.584 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.579+0000 7fde54f11640 1 Processor -- start 2023-05-22T16:06:10.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.579+0000 7fde54f11640 1 -- start start 2023-05-22T16:06:10.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.583+0000 7fde54f11640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fde5006cc80 0x7fde5006f630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:10.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.583+0000 7fde54f11640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fde500afb50 0x7fde500affb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:10.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.583+0000 7fde54f11640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde50071b40 con 0x7fde50154170 2023-05-22T16:06:10.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.583+0000 7fde54f11640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde500ca9f0 con 0x7fde500afb50 2023-05-22T16:06:10.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.583+0000 7fde54f11640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde500ca150 con 0x7fde5006cc80 2023-05-22T16:06:10.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.583+0000 7fde4ed9d640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fde500afb50 0x7fde500affb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:10.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.583+0000 7fde4ed9d640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fde500afb50 0x7fde500affb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57764/0 (socket says 172.21.15.16:57764) 2023-05-22T16:06:10.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.583+0000 7fde4ed9d640 1 -- 172.21.15.16:0/2908736335 learned_addr learned my addr 172.21.15.16:0/2908736335 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:06:10.587 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.583+0000 7fde4dd9b640 1 --2- 172.21.15.16:0/2908736335 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fde5006cc80 0x7fde5006f630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:10.587 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.583+0000 7fde4ed9d640 1 -- 172.21.15.16:0/2908736335 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fde5006cc80 msgr2=0x7fde5006f630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:06:10.587 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.583+0000 7fde4ed9d640 1 --2- 172.21.15.16:0/2908736335 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fde5006cc80 0x7fde5006f630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:10.587 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.583+0000 7fde4ed9d640 1 -- 172.21.15.16:0/2908736335 >> v1:172.21.15.62:6789/0 conn(0x7fde50154170 legacy=0x7fde5006be90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:06:10.588 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.583+0000 7fde4ed9d640 1 -- 172.21.15.16:0/2908736335 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde500b04f0 con 0x7fde500afb50 2023-05-22T16:06:10.588 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.583+0000 7fde4ed9d640 1 --2- 172.21.15.16:0/2908736335 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde500afb50 0x7fde500affb0 secure :-1 s=READY pgs=2660 cs=0 l=1 rev1=1 crypto rx=0x7fde48009e50 tx=0x7fde4801c9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9f2c8791cc73d9d server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:06:10.588 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.587+0000 7fde4d59a640 1 -- 172.21.15.16:0/2908736335 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde480a58a0 con 0x7fde500afb50 2023-05-22T16:06:10.589 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.587+0000 7fde4d59a640 1 -- 172.21.15.16:0/2908736335 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde480a5a00 con 0x7fde500afb50 2023-05-22T16:06:10.589 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.587+0000 7fde4d59a640 1 -- 172.21.15.16:0/2908736335 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde480a5d90 con 0x7fde500afb50 2023-05-22T16:06:10.590 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.587+0000 7fde54f11640 1 -- 172.21.15.16:0/2908736335 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde500afb50 msgr2=0x7fde500affb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:06:10.590 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.587+0000 7fde54f11640 1 --2- 172.21.15.16:0/2908736335 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde500afb50 0x7fde500affb0 secure :-1 s=READY pgs=2660 cs=0 l=1 rev1=1 crypto rx=0x7fde48009e50 tx=0x7fde4801c9e0 comp rx=0 tx=0).stop 2023-05-22T16:06:10.590 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.587+0000 7fde54f11640 1 -- 172.21.15.16:0/2908736335 shutdown_connections 2023-05-22T16:06:10.590 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.587+0000 7fde54f11640 1 --2- 172.21.15.16:0/2908736335 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde500afb50 0x7fde500affb0 unknown :-1 s=CLOSED pgs=2660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:10.590 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.587+0000 7fde54f11640 1 --2- 172.21.15.16:0/2908736335 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fde5006cc80 0x7fde5006f630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:10.590 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.587+0000 7fde54f11640 1 -- 172.21.15.16:0/2908736335 >> 172.21.15.16:0/2908736335 conn(0x7fde500cab50 msgr2=0x7fde500caf50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:06:10.590 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.587+0000 7fde54f11640 1 -- 172.21.15.16:0/2908736335 shutdown_connections 2023-05-22T16:06:10.591 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.587+0000 7fde54f11640 1 -- 172.21.15.16:0/2908736335 wait complete. 2023-05-22T16:06:10.593 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.587+0000 7fde54f11640 1 Processor -- start 2023-05-22T16:06:10.593 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.587+0000 7fde54f11640 1 -- start start 2023-05-22T16:06:10.594 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.587+0000 7fde54f11640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde5006cc80 0x7fde501f4c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:10.594 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.587+0000 7fde54f11640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fde500afb50 0x7fde5014b200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:10.594 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.587+0000 7fde54f11640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fde50154170 0x7fde500cbca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:10.594 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.587+0000 7fde54f11640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde500cd100 con 0x7fde5006cc80 2023-05-22T16:06:10.594 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.587+0000 7fde54f11640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde500cc4f0 con 0x7fde50154170 2023-05-22T16:06:10.594 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.587+0000 7fde54f11640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde500cc7d0 con 0x7fde500afb50 2023-05-22T16:06:10.594 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.587+0000 7fde4e59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde5006cc80 0x7fde501f4c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:10.595 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.587+0000 7fde4e59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde5006cc80 0x7fde501f4c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57774/0 (socket says 172.21.15.16:57774) 2023-05-22T16:06:10.595 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.587+0000 7fde4e59c640 1 -- 172.21.15.16:0/545450705 learned_addr learned my addr 172.21.15.16:0/545450705 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:06:10.595 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.587+0000 7fde4dd9b640 1 --2- 172.21.15.16:0/545450705 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fde500afb50 0x7fde5014b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:10.595 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.587+0000 7fde4ed9d640 1 --2- 172.21.15.16:0/545450705 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fde50154170 0x7fde500cbca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:10.595 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.587+0000 7fde4e59c640 1 -- 172.21.15.16:0/545450705 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fde500afb50 msgr2=0x7fde5014b200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:06:10.595 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.587+0000 7fde4e59c640 1 --2- 172.21.15.16:0/545450705 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fde500afb50 0x7fde5014b200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:10.595 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.587+0000 7fde4e59c640 1 -- 172.21.15.16:0/545450705 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fde50154170 msgr2=0x7fde500cbca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:06:10.595 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.587+0000 7fde4e59c640 1 --2- 172.21.15.16:0/545450705 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fde50154170 0x7fde500cbca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:10.596 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.587+0000 7fde4e59c640 1 -- 172.21.15.16:0/545450705 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde48009b30 con 0x7fde5006cc80 2023-05-22T16:06:10.596 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.587+0000 7fde4e59c640 1 --2- 172.21.15.16:0/545450705 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde5006cc80 0x7fde501f4c60 secure :-1 s=READY pgs=2661 cs=0 l=1 rev1=1 crypto rx=0x7fde4400c3c0 tx=0x7fde4400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:06:10.596 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.591+0000 7fde3effd640 1 -- 172.21.15.16:0/545450705 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde440052d0 con 0x7fde5006cc80 2023-05-22T16:06:10.596 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.591+0000 7fde3effd640 1 -- 172.21.15.16:0/545450705 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde44005430 con 0x7fde5006cc80 2023-05-22T16:06:10.596 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.591+0000 7fde54f11640 1 -- 172.21.15.16:0/545450705 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde5006c5a0 con 0x7fde5006cc80 2023-05-22T16:06:10.596 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.591+0000 7fde3effd640 1 -- 172.21.15.16:0/545450705 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde44003840 con 0x7fde5006cc80 2023-05-22T16:06:10.596 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.591+0000 7fde54f11640 1 -- 172.21.15.16:0/545450705 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde5006b970 con 0x7fde5006cc80 2023-05-22T16:06:10.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.591+0000 7fde54f11640 1 -- 172.21.15.16:0/545450705 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde50154540 con 0x7fde5006cc80 2023-05-22T16:06:10.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.591+0000 7fde3effd640 1 -- 172.21.15.16:0/545450705 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 100) v1 ==== 84332+0+0 (secure 0 0 0) 0x7fde440039a0 con 0x7fde5006cc80 2023-05-22T16:06:10.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.591+0000 7fde3effd640 1 --2- 172.21.15.16:0/545450705 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fde300654a0 0x7fde30067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:10.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.591+0000 7fde3effd640 1 -- 172.21.15.16:0/545450705 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(201..201 src has 1..201) v4 ==== 6317+0+0 (secure 0 0 0) 0x7fde44082030 con 0x7fde5006cc80 2023-05-22T16:06:10.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.591+0000 7fde4dd9b640 1 --2- 172.21.15.16:0/545450705 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fde300654a0 0x7fde30067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:10.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.591+0000 7fde4dd9b640 1 --2- 172.21.15.16:0/545450705 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fde300654a0 0x7fde30067950 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fde5006f650 tx=0x7fde40008040 comp rx=0 tx=0).ready entity=mgr.8195 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:06:10.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.595+0000 7fde3effd640 1 -- 172.21.15.16:0/545450705 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fde4404faa0 con 0x7fde5006cc80 2023-05-22T16:06:10.748 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.747+0000 7fde54f11640 1 -- 172.21.15.16:0/545450705 --> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "6", "target": ["mon-mgr", ""]}) v1 -- 0x7fde500cd240 con 0x7fde300654a0 2023-05-22T16:06:10.749 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.747+0000 7fde3effd640 1 -- 172.21.15.16:0/545450705 <== mgr.8195 v2:172.21.15.16:6832/65890 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 6 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fde40011950 con 0x7fde300654a0 2023-05-22T16:06:10.749 INFO:teuthology.orchestra.run.smithi016.stderr:instructed osd(s) 6 to deep-scrub 2023-05-22T16:06:10.751 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.747+0000 7fde54f11640 1 -- 172.21.15.16:0/545450705 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fde300654a0 msgr2=0x7fde30067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:06:10.751 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.747+0000 7fde54f11640 1 --2- 172.21.15.16:0/545450705 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fde300654a0 0x7fde30067950 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fde5006f650 tx=0x7fde40008040 comp rx=0 tx=0).stop 2023-05-22T16:06:10.751 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.747+0000 7fde54f11640 1 -- 172.21.15.16:0/545450705 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde5006cc80 msgr2=0x7fde501f4c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:06:10.751 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.747+0000 7fde54f11640 1 --2- 172.21.15.16:0/545450705 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde5006cc80 0x7fde501f4c60 secure :-1 s=READY pgs=2661 cs=0 l=1 rev1=1 crypto rx=0x7fde4400c3c0 tx=0x7fde4400c880 comp rx=0 tx=0).stop 2023-05-22T16:06:10.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.747+0000 7fde54f11640 1 -- 172.21.15.16:0/545450705 shutdown_connections 2023-05-22T16:06:10.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.747+0000 7fde54f11640 1 --2- 172.21.15.16:0/545450705 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fde50154170 0x7fde500cbca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:10.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.747+0000 7fde54f11640 1 --2- 172.21.15.16:0/545450705 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fde500afb50 0x7fde5014b200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:10.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.747+0000 7fde54f11640 1 --2- 172.21.15.16:0/545450705 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fde300654a0 0x7fde30067950 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:10.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.747+0000 7fde54f11640 1 --2- 172.21.15.16:0/545450705 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde5006cc80 0x7fde501f4c60 unknown :-1 s=CLOSED pgs=2661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:10.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.747+0000 7fde54f11640 1 -- 172.21.15.16:0/545450705 >> 172.21.15.16:0/545450705 conn(0x7fde500cab50 msgr2=0x7fde500caf20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:06:10.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.747+0000 7fde54f11640 1 -- 172.21.15.16:0/545450705 shutdown_connections 2023-05-22T16:06:10.753 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.747+0000 7fde54f11640 1 -- 172.21.15.16:0/545450705 wait complete. 2023-05-22T16:06:10.766 INFO:tasks.ceph:Scrubbing osd.7 2023-05-22T16:06:10.767 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2023-05-22T16:06:10.894 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.891+0000 7f8da6344640 1 Processor -- start 2023-05-22T16:06:10.895 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.891+0000 7f8da6344640 1 -- start start 2023-05-22T16:06:10.895 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.891+0000 7f8da6344640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8da00ce6a0 0x7f8da00cc920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:10.895 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.891+0000 7f8da6344640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f8da00b29e0 0x7f8da00cfb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:10.896 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.891+0000 7f8da6344640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8da00ced50 con 0x7f8da0159b00 2023-05-22T16:06:10.896 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.891+0000 7f8da6344640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8da00ccfb0 con 0x7f8da00b29e0 2023-05-22T16:06:10.896 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.891+0000 7f8da6344640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8da00cd290 con 0x7f8da00ce6a0 2023-05-22T16:06:10.897 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.891+0000 7f8da48e1640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f8da00b29e0 0x7f8da00cfb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:10.897 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.891+0000 7f8da48e1640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f8da00b29e0 0x7f8da00cfb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.16:34636/0 (socket says 172.21.15.16:34636) 2023-05-22T16:06:10.897 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.891+0000 7f8d9f7fe640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8da00ce6a0 0x7f8da00cc920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:10.897 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.891+0000 7f8da48e1640 1 -- 172.21.15.16:0/4099310994 learned_addr learned my addr 172.21.15.16:0/4099310994 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:06:10.898 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.895+0000 7f8da48e1640 1 -- 172.21.15.16:0/4099310994 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8da00ce6a0 msgr2=0x7f8da00cc920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:06:10.898 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.895+0000 7f8da48e1640 1 --2- 172.21.15.16:0/4099310994 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8da00ce6a0 0x7f8da00cc920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:10.898 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.895+0000 7f8da48e1640 1 -- 172.21.15.16:0/4099310994 >> v1:172.21.15.62:6789/0 conn(0x7f8da0159b00 legacy=0x7f8da00cdf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:06:10.898 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.895+0000 7f8da48e1640 1 -- 172.21.15.16:0/4099310994 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8da00cd570 con 0x7f8da00b29e0 2023-05-22T16:06:10.898 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.895+0000 7f8da48e1640 1 --2- 172.21.15.16:0/4099310994 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8da00b29e0 0x7f8da00cfb00 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f8d98009870 tx=0x7f8d9801c540 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5730ead23c4ef862 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:06:10.899 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.895+0000 7f8d9effd640 1 -- 172.21.15.16:0/4099310994 <== mon.1 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d980ab650 con 0x7f8da00b29e0 2023-05-22T16:06:10.899 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.895+0000 7f8d9effd640 1 -- 172.21.15.16:0/4099310994 <== mon.1 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d980ab7b0 con 0x7f8da00b29e0 2023-05-22T16:06:10.900 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.895+0000 7f8d9effd640 1 -- 172.21.15.16:0/4099310994 <== mon.1 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d980abb40 con 0x7f8da00b29e0 2023-05-22T16:06:10.901 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.895+0000 7f8da6344640 1 -- 172.21.15.16:0/4099310994 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8da00b29e0 msgr2=0x7f8da00cfb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:06:10.902 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.895+0000 7f8da6344640 1 --2- 172.21.15.16:0/4099310994 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8da00b29e0 0x7f8da00cfb00 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f8d98009870 tx=0x7f8d9801c540 comp rx=0 tx=0).stop 2023-05-22T16:06:10.902 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.895+0000 7f8da6344640 1 -- 172.21.15.16:0/4099310994 shutdown_connections 2023-05-22T16:06:10.902 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.895+0000 7f8da6344640 1 --2- 172.21.15.16:0/4099310994 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8da00b29e0 0x7f8da00cfb00 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:10.903 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.895+0000 7f8da6344640 1 --2- 172.21.15.16:0/4099310994 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8da00ce6a0 0x7f8da00cc920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:10.903 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.895+0000 7f8da6344640 1 -- 172.21.15.16:0/4099310994 >> 172.21.15.16:0/4099310994 conn(0x7f8da006a440 msgr2=0x7f8da006db60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:06:10.904 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.895+0000 7f8da6344640 1 -- 172.21.15.16:0/4099310994 shutdown_connections 2023-05-22T16:06:10.904 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.895+0000 7f8da6344640 1 -- 172.21.15.16:0/4099310994 wait complete. 2023-05-22T16:06:10.904 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.899+0000 7f8da6344640 1 Processor -- start 2023-05-22T16:06:10.904 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.899+0000 7f8da6344640 1 -- start start 2023-05-22T16:06:10.907 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.899+0000 7f8da6344640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8da00b29e0 0x7f8da00c9f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:10.908 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.899+0000 7f8da6344640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8da00ce6a0 0x7f8da014e810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:10.908 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.899+0000 7f8da6344640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8da0159b00 0x7f8da0151ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:10.908 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.899+0000 7f8da6344640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8da0151590 con 0x7f8da00ce6a0 2023-05-22T16:06:10.908 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.899+0000 7f8da6344640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8da0151060 con 0x7f8da00b29e0 2023-05-22T16:06:10.908 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.899+0000 7f8da6344640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8da01512b0 con 0x7f8da0159b00 2023-05-22T16:06:10.908 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.899+0000 7f8d9f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8da00ce6a0 0x7f8da014e810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:10.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.899+0000 7f8d9f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8da00ce6a0 0x7f8da014e810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57776/0 (socket says 172.21.15.16:57776) 2023-05-22T16:06:10.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.899+0000 7f8d9f7fe640 1 -- 172.21.15.16:0/187589949 learned_addr learned my addr 172.21.15.16:0/187589949 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:06:10.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.899+0000 7f8da48e1640 1 --2- 172.21.15.16:0/187589949 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8da0159b00 0x7f8da0151ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:10.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.899+0000 7f8d9ffff640 1 --2- 172.21.15.16:0/187589949 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8da00b29e0 0x7f8da00c9f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:10.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.899+0000 7f8d9f7fe640 1 -- 172.21.15.16:0/187589949 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8da0159b00 msgr2=0x7f8da0151ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:06:10.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.899+0000 7f8d9f7fe640 1 --2- 172.21.15.16:0/187589949 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8da0159b00 0x7f8da0151ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:10.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.899+0000 7f8d9f7fe640 1 -- 172.21.15.16:0/187589949 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8da00b29e0 msgr2=0x7f8da00c9f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:06:10.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.899+0000 7f8d9f7fe640 1 --2- 172.21.15.16:0/187589949 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8da00b29e0 0x7f8da00c9f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:10.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.899+0000 7f8d9f7fe640 1 -- 172.21.15.16:0/187589949 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d980094d0 con 0x7f8da00ce6a0 2023-05-22T16:06:10.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.899+0000 7f8da48e1640 1 --2- 172.21.15.16:0/187589949 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8da0159b00 0x7f8da0151ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T16:06:10.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.899+0000 7f8d9ffff640 1 --2- 172.21.15.16:0/187589949 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8da00b29e0 0x7f8da00c9f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-22T16:06:10.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.899+0000 7f8d9f7fe640 1 --2- 172.21.15.16:0/187589949 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8da00ce6a0 0x7f8da014e810 secure :-1 s=READY pgs=2662 cs=0 l=1 rev1=1 crypto rx=0x7f8d8800efc0 tx=0x7f8d8800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:06:10.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.899+0000 7f8d9cff9640 1 -- 172.21.15.16:0/187589949 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d88009be0 con 0x7f8da00ce6a0 2023-05-22T16:06:10.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.899+0000 7f8d9cff9640 1 -- 172.21.15.16:0/187589949 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d8800f040 con 0x7f8da00ce6a0 2023-05-22T16:06:10.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.899+0000 7f8d9cff9640 1 -- 172.21.15.16:0/187589949 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d88014680 con 0x7f8da00ce6a0 2023-05-22T16:06:10.911 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.899+0000 7f8da6344640 1 -- 172.21.15.16:0/187589949 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8da00d09e0 con 0x7f8da00ce6a0 2023-05-22T16:06:10.911 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.899+0000 7f8da6344640 1 -- 172.21.15.16:0/187589949 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8da00d0850 con 0x7f8da00ce6a0 2023-05-22T16:06:10.911 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.899+0000 7f8da6344640 1 -- 172.21.15.16:0/187589949 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f8da00d0b70 con 0x7f8da00ce6a0 2023-05-22T16:06:10.911 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.903+0000 7f8d9cff9640 1 -- 172.21.15.16:0/187589949 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 100) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f8d880147e0 con 0x7f8da00ce6a0 2023-05-22T16:06:10.911 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.903+0000 7f8d9cff9640 1 --2- 172.21.15.16:0/187589949 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f8d800654d0 0x7f8d80067980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:10.911 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.903+0000 7f8d9cff9640 1 -- 172.21.15.16:0/187589949 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(201..201 src has 1..201) v4 ==== 6317+0+0 (secure 0 0 0) 0x7f8d88084390 con 0x7f8da00ce6a0 2023-05-22T16:06:10.912 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.903+0000 7f8d9ffff640 1 --2- 172.21.15.16:0/187589949 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f8d800654d0 0x7f8d80067980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:10.912 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.903+0000 7f8d9cff9640 1 --2- 172.21.15.16:0/187589949 >> [v2:172.21.15.62:6808/37355,v1:172.21.15.62:6809/37355] conn(0x7f8d8006c380 0x7f8d8006e7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:10.912 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.903+0000 7f8d9cff9640 1 -- 172.21.15.16:0/187589949 --> [v2:172.21.15.62:6808/37355,v1:172.21.15.62:6809/37355] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8d8006eec0 con 0x7f8d8006c380 2023-05-22T16:06:10.912 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.903+0000 7f8d9cff9640 1 -- 172.21.15.16:0/187589949 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_get_version_reply(handle=1 version=201) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8d880847e0 con 0x7f8da00ce6a0 2023-05-22T16:06:10.912 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.903+0000 7f8da48e1640 1 --2- 172.21.15.16:0/187589949 >> [v2:172.21.15.62:6808/37355,v1:172.21.15.62:6809/37355] conn(0x7f8d8006c380 0x7f8d8006e7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:10.912 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.903+0000 7f8d9ffff640 1 --2- 172.21.15.16:0/187589949 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f8d800654d0 0x7f8d80067980 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f8da00ca570 tx=0x7f8d90009040 comp rx=0 tx=0).ready entity=mgr.8195 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:06:10.912 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.903+0000 7f8da48e1640 1 --2- 172.21.15.16:0/187589949 >> [v2:172.21.15.62:6808/37355,v1:172.21.15.62:6809/37355] conn(0x7f8d8006c380 0x7f8d8006e7f0 crc :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:06:10.913 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.903+0000 7f8d9cff9640 1 -- 172.21.15.16:0/187589949 <== osd.7 v2:172.21.15.62:6808/37355 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f8d8006eec0 con 0x7f8d8006c380 2023-05-22T16:06:10.920 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.919+0000 7f8da6344640 1 -- 172.21.15.16:0/187589949 --> [v2:172.21.15.62:6808/37355,v1:172.21.15.62:6809/37355] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f8da014f060 con 0x7f8d8006c380 2023-05-22T16:06:10.921 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.919+0000 7f8d9cff9640 1 -- 172.21.15.16:0/187589949 <== osd.7 v2:172.21.15.62:6808/37355 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f8da014f060 con 0x7f8d8006c380 2023-05-22T16:06:10.921 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-05-22T16:06:10.921 INFO:teuthology.orchestra.run.smithi016.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-05-22T16:06:10.922 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-05-22T16:06:10.922 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.919+0000 7f8da6344640 1 -- 172.21.15.16:0/187589949 >> [v2:172.21.15.62:6808/37355,v1:172.21.15.62:6809/37355] conn(0x7f8d8006c380 msgr2=0x7f8d8006e7f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:06:10.922 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.919+0000 7f8da6344640 1 --2- 172.21.15.16:0/187589949 >> [v2:172.21.15.62:6808/37355,v1:172.21.15.62:6809/37355] conn(0x7f8d8006c380 0x7f8d8006e7f0 crc :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:10.923 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.919+0000 7f8da6344640 1 -- 172.21.15.16:0/187589949 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f8d800654d0 msgr2=0x7f8d80067980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:06:10.923 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.919+0000 7f8da6344640 1 --2- 172.21.15.16:0/187589949 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f8d800654d0 0x7f8d80067980 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f8da00ca570 tx=0x7f8d90009040 comp rx=0 tx=0).stop 2023-05-22T16:06:10.923 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.919+0000 7f8da6344640 1 -- 172.21.15.16:0/187589949 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8da00ce6a0 msgr2=0x7f8da014e810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:06:10.923 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.919+0000 7f8da6344640 1 --2- 172.21.15.16:0/187589949 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8da00ce6a0 0x7f8da014e810 secure :-1 s=READY pgs=2662 cs=0 l=1 rev1=1 crypto rx=0x7f8d8800efc0 tx=0x7f8d8800c490 comp rx=0 tx=0).stop 2023-05-22T16:06:10.923 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.919+0000 7f8da48e1640 1 -- 172.21.15.16:0/187589949 reap_dead start 2023-05-22T16:06:10.923 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.919+0000 7f8da6344640 1 -- 172.21.15.16:0/187589949 shutdown_connections 2023-05-22T16:06:10.923 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.919+0000 7f8da6344640 1 -- 172.21.15.16:0/187589949 >> 172.21.15.16:0/187589949 conn(0x7f8da006a440 msgr2=0x7f8da006db60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:06:10.924 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.919+0000 7f8da6344640 1 -- 172.21.15.16:0/187589949 shutdown_connections 2023-05-22T16:06:10.924 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:10.919+0000 7f8da6344640 1 -- 172.21.15.16:0/187589949 wait complete. 2023-05-22T16:06:10.937 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2023-05-22T16:06:11.049 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:11.047+0000 7fa249cbf640 1 Processor -- start 2023-05-22T16:06:11.049 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:11.047+0000 7fa249cbf640 1 -- start start 2023-05-22T16:06:11.050 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:11.047+0000 7fa249cbf640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fa2440d0730 0x7fa2440d20c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:11.050 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:11.047+0000 7fa249cbf640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2440cf190 con 0x7fa2440cfa20 2023-05-22T16:06:11.050 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:11.047+0000 7fa249cbf640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2440d3360 con 0x7fa2440d2600 2023-05-22T16:06:11.050 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:11.047+0000 7fa249cbf640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa24415c4f0 con 0x7fa2440d0730 2023-05-22T16:06:11.050 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:11.047+0000 7fa2437fe640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fa2440cfa20 0x7fa2440d0020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:46422/0 (socket says 172.21.15.16:46422) 2023-05-22T16:06:11.050 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:11.047+0000 7fa2437fe640 1 -- 172.21.15.16:0/804764094 learned_addr learned my addr 172.21.15.16:0/804764094 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T16:06:11.051 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:11.047+0000 7fa242ffd640 1 --2- 172.21.15.16:0/804764094 >> v2:172.21.15.62:3300/0 conn(0x7fa2440d0730 0x7fa2440d20c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:11.051 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:11.047+0000 7fa2427fc640 1 -- 172.21.15.16:0/804764094 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1560294559 0 0) 0x7fa2440cf190 con 0x7fa2440cfa20 2023-05-22T16:06:11.051 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:11.047+0000 7fa2427fc640 1 -- 172.21.15.16:0/804764094 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa22c003530 con 0x7fa2440cfa20 2023-05-22T16:06:11.052 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:11.047+0000 7fa2427fc640 1 -- 172.21.15.16:0/804764094 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1681088082 0 0) 0x7fa22c003530 con 0x7fa2440cfa20 2023-05-22T16:06:11.052 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:11.047+0000 7fa2427fc640 1 -- 172.21.15.16:0/804764094 >> v2:172.21.15.62:3300/0 conn(0x7fa2440d0730 msgr2=0x7fa2440d20c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:06:11.052 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:11.047+0000 7fa2427fc640 1 --2- 172.21.15.16:0/804764094 >> v2:172.21.15.62:3300/0 conn(0x7fa2440d0730 0x7fa2440d20c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:11.052 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:11.047+0000 7fa2427fc640 1 -- 172.21.15.16:0/804764094 >> v1:172.21.15.62:6789/0 conn(0x7fa2440d2600 legacy=0x7fa2440cd340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:06:11.052 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:11.047+0000 7fa2427fc640 1 -- 172.21.15.16:0/804764094 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2440d2da0 con 0x7fa2440cfa20 2023-05-22T16:06:11.053 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:11.047+0000 7fa2427fc640 1 -- 172.21.15.16:0/804764094 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7fa234001800 con 0x7fa2440cfa20 2023-05-22T16:06:11.053 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:11.047+0000 7fa2427fc640 1 -- 172.21.15.16:0/804764094 >> v1:172.21.15.16:6789/0 conn(0x7fa2440cfa20 legacy=0x7fa2440d0020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:06:11.053 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:11.047+0000 7fa2427fc640 1 --2- 172.21.15.16:0/804764094 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa22c003980 0x7fa22c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:11.053 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:11.047+0000 7fa2427fc640 1 --2- 172.21.15.16:0/804764094 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa22c0047d0 0x7fa22c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:11.053 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:11.047+0000 7fa2427fc640 1 --2- 172.21.15.16:0/804764094 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa22c005180 0x7fa22c00b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:11.053 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:11.047+0000 7fa2427fc640 1 -- 172.21.15.16:0/804764094 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa24415c4f0 con 0x7fa22c0047d0 2023-05-22T16:06:11.053 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:11.047+0000 7fa2427fc640 1 -- 172.21.15.16:0/804764094 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa22c00bfe0 con 0x7fa22c005180 2023-05-22T16:06:11.054 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:11.047+0000 7fa2427fc640 1 -- 172.21.15.16:0/804764094 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa22c00c1a0 con 0x7fa22c003980 2023-05-22T16:06:11.054 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:11.047+0000 7fa242ffd640 1 --2- 172.21.15.16:0/804764094 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa22c0047d0 0x7fa22c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:11.054 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:11.047+0000 7fa243fff640 1 --2- 172.21.15.16:0/804764094 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa22c005180 0x7fa22c00b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:11.054 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:11.047+0000 7fa242ffd640 1 -- 172.21.15.16:0/804764094 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa22c003980 msgr2=0x7fa22c003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:06:11.054 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:11.047+0000 7fa242ffd640 1 --2- 172.21.15.16:0/804764094 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa22c003980 0x7fa22c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:11.054 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:11.047+0000 7fa242ffd640 1 -- 172.21.15.16:0/804764094 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa22c005180 msgr2=0x7fa22c00b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:06:11.054 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:11.047+0000 7fa242ffd640 1 --2- 172.21.15.16:0/804764094 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa22c005180 0x7fa22c00b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:11.054 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:11.047+0000 7fa242ffd640 1 -- 172.21.15.16:0/804764094 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa22c003530 con 0x7fa22c0047d0 2023-05-22T16:06:11.055 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:11.047+0000 7fa243fff640 1 --2- 172.21.15.16:0/804764094 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa22c005180 0x7fa22c00b990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T16:06:11.055 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:11.047+0000 7fa243fff640 1 -- 172.21.15.16:0/804764094 reap_dead start 2023-05-22T16:06:11.059 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:11.051+0000 7fa242ffd640 1 --2- 172.21.15.16:0/804764094 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa22c0047d0 0x7fa22c004c40 secure :-1 s=READY pgs=2664 cs=0 l=1 rev1=1 crypto rx=0x7fa238009ab0 tx=0x7fa23801ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eacbadb498c756a7 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:06:11.060 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:11.051+0000 7fa2427fc640 1 -- 172.21.15.16:0/804764094 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2380a5620 con 0x7fa22c0047d0 2023-05-22T16:06:11.060 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:11.051+0000 7fa2427fc640 1 -- 172.21.15.16:0/804764094 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2380a5780 con 0x7fa22c0047d0 2023-05-22T16:06:11.060 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:11.051+0000 7fa2427fc640 1 -- 172.21.15.16:0/804764094 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2380a5b30 con 0x7fa22c0047d0 2023-05-22T16:06:11.060 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:11.051+0000 7fa249cbf640 1 -- 172.21.15.16:0/804764094 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa22c0047d0 msgr2=0x7fa22c004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:06:11.060 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:11.051+0000 7fa249cbf640 1 --2- 172.21.15.16:0/804764094 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa22c0047d0 0x7fa22c004c40 secure :-1 s=READY pgs=2664 cs=0 l=1 rev1=1 crypto rx=0x7fa238009ab0 tx=0x7fa23801ca30 comp rx=0 tx=0).stop 2023-05-22T16:06:11.061 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:11.051+0000 7fa249cbf640 1 -- 172.21.15.16:0/804764094 shutdown_connections 2023-05-22T16:06:11.061 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:11.051+0000 7fa249cbf640 1 --2- 172.21.15.16:0/804764094 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa22c0047d0 0x7fa22c004c40 unknown :-1 s=CLOSED pgs=2664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:11.061 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:11.051+0000 7fa249cbf640 1 -- 172.21.15.16:0/804764094 >> 172.21.15.16:0/804764094 conn(0x7fa244071850 msgr2=0x7fa24409b830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:06:11.061 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:11.051+0000 7fa249cbf640 1 -- 172.21.15.16:0/804764094 shutdown_connections 2023-05-22T16:06:11.061 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:11.051+0000 7fa249cbf640 1 -- 172.21.15.16:0/804764094 wait complete. 2023-05-22T16:06:11.061 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:11.051+0000 7fa249cbf640 1 Processor -- start 2023-05-22T16:06:11.061 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:11.051+0000 7fa249cbf640 1 -- start start 2023-05-22T16:06:11.061 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:11.051+0000 7fa249cbf640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa22c0047d0 0x7fa24414f260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:11.062 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:11.051+0000 7fa249cbf640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa244158bc0 0x7fa244159890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:11.062 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:11.051+0000 7fa249cbf640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa2440c73c0 0x7fa2440c7830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:11.062 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:11.051+0000 7fa249cbf640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2440c7d70 con 0x7fa22c0047d0 2023-05-22T16:06:11.062 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:11.051+0000 7fa249cbf640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa24414f7a0 con 0x7fa244158bc0 2023-05-22T16:06:11.062 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:11.051+0000 7fa249cbf640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa244159dd0 con 0x7fa2440c73c0 2023-05-22T16:06:11.062 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:11.051+0000 7fa2437fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa22c0047d0 0x7fa24414f260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:11.062 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:11.051+0000 7fa2437fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa22c0047d0 0x7fa24414f260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57788/0 (socket says 172.21.15.16:57788) 2023-05-22T16:06:11.063 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:11.051+0000 7fa2437fe640 1 -- 172.21.15.16:0/3731072681 learned_addr learned my addr 172.21.15.16:0/3731072681 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:06:11.063 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:11.051+0000 7fa242ffd640 1 --2- 172.21.15.16:0/3731072681 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa244158bc0 0x7fa244159890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:11.063 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:11.051+0000 7fa2437fe640 1 -- 172.21.15.16:0/3731072681 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa2440c73c0 msgr2=0x7fa2440c7830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:06:11.063 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:11.051+0000 7fa2437fe640 1 --2- 172.21.15.16:0/3731072681 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa2440c73c0 0x7fa2440c7830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:11.063 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:11.051+0000 7fa2437fe640 1 -- 172.21.15.16:0/3731072681 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa244158bc0 msgr2=0x7fa244159890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:06:11.063 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:11.051+0000 7fa2437fe640 1 --2- 172.21.15.16:0/3731072681 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa244158bc0 0x7fa244159890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:11.063 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:11.051+0000 7fa2437fe640 1 -- 172.21.15.16:0/3731072681 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2440ccb40 con 0x7fa22c0047d0 2023-05-22T16:06:11.064 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:11.051+0000 7fa242ffd640 1 --2- 172.21.15.16:0/3731072681 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa244158bc0 0x7fa244159890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T16:06:11.064 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:11.051+0000 7fa2437fe640 1 --2- 172.21.15.16:0/3731072681 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa22c0047d0 0x7fa24414f260 secure :-1 s=READY pgs=2665 cs=0 l=1 rev1=1 crypto rx=0x7fa23400c840 tx=0x7fa23400cd00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:06:11.064 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:11.055+0000 7fa22bfff640 1 -- 172.21.15.16:0/3731072681 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa234007480 con 0x7fa22c0047d0 2023-05-22T16:06:11.064 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:11.055+0000 7fa22bfff640 1 -- 172.21.15.16:0/3731072681 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2340075e0 con 0x7fa22c0047d0 2023-05-22T16:06:11.064 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:11.055+0000 7fa249cbf640 1 -- 172.21.15.16:0/3731072681 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2440d3080 con 0x7fa22c0047d0 2023-05-22T16:06:11.064 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:11.055+0000 7fa22bfff640 1 -- 172.21.15.16:0/3731072681 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2340059f0 con 0x7fa22c0047d0 2023-05-22T16:06:11.065 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:11.055+0000 7fa249cbf640 1 -- 172.21.15.16:0/3731072681 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2440cce20 con 0x7fa22c0047d0 2023-05-22T16:06:11.065 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:11.055+0000 7fa249cbf640 1 -- 172.21.15.16:0/3731072681 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa244073830 con 0x7fa22c0047d0 2023-05-22T16:06:11.065 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:11.055+0000 7fa22bfff640 1 -- 172.21.15.16:0/3731072681 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 100) v1 ==== 84332+0+0 (secure 0 0 0) 0x7fa234005b50 con 0x7fa22c0047d0 2023-05-22T16:06:11.065 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:11.055+0000 7fa22bfff640 1 --2- 172.21.15.16:0/3731072681 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fa2300654d0 0x7fa230067980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:11.066 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:11.055+0000 7fa22bfff640 1 -- 172.21.15.16:0/3731072681 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(201..201 src has 1..201) v4 ==== 6317+0+0 (secure 0 0 0) 0x7fa234082380 con 0x7fa22c0047d0 2023-05-22T16:06:11.066 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:11.055+0000 7fa242ffd640 1 --2- 172.21.15.16:0/3731072681 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fa2300654d0 0x7fa230067980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:11.066 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:11.055+0000 7fa242ffd640 1 --2- 172.21.15.16:0/3731072681 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fa2300654d0 0x7fa230067980 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fa2380af000 tx=0x7fa238008040 comp rx=0 tx=0).ready entity=mgr.8195 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:06:11.067 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:11.059+0000 7fa22bfff640 1 -- 172.21.15.16:0/3731072681 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fa23404fd90 con 0x7fa22c0047d0 2023-05-22T16:06:11.212 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:11.207+0000 7fa249cbf640 1 -- 172.21.15.16:0/3731072681 --> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "7", "target": ["mon-mgr", ""]}) v1 -- 0x7fa24414fa40 con 0x7fa2300654d0 2023-05-22T16:06:11.212 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:11.207+0000 7fa22bfff640 1 -- 172.21.15.16:0/3731072681 <== mgr.8195 v2:172.21.15.16:6832/65890 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 7 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fa24414fa40 con 0x7fa2300654d0 2023-05-22T16:06:11.213 INFO:teuthology.orchestra.run.smithi016.stderr:instructed osd(s) 7 to deep-scrub 2023-05-22T16:06:11.215 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:11.211+0000 7fa249cbf640 1 -- 172.21.15.16:0/3731072681 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fa2300654d0 msgr2=0x7fa230067980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:06:11.215 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:11.211+0000 7fa249cbf640 1 --2- 172.21.15.16:0/3731072681 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fa2300654d0 0x7fa230067980 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fa2380af000 tx=0x7fa238008040 comp rx=0 tx=0).stop 2023-05-22T16:06:11.215 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:11.211+0000 7fa249cbf640 1 -- 172.21.15.16:0/3731072681 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa22c0047d0 msgr2=0x7fa24414f260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:06:11.215 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:11.211+0000 7fa249cbf640 1 --2- 172.21.15.16:0/3731072681 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa22c0047d0 0x7fa24414f260 secure :-1 s=READY pgs=2665 cs=0 l=1 rev1=1 crypto rx=0x7fa23400c840 tx=0x7fa23400cd00 comp rx=0 tx=0).stop 2023-05-22T16:06:11.215 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:11.211+0000 7fa249cbf640 1 -- 172.21.15.16:0/3731072681 shutdown_connections 2023-05-22T16:06:11.216 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:11.211+0000 7fa249cbf640 1 --2- 172.21.15.16:0/3731072681 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa2440c73c0 0x7fa2440c7830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:11.216 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:11.211+0000 7fa249cbf640 1 --2- 172.21.15.16:0/3731072681 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa244158bc0 0x7fa244159890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:11.216 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:11.211+0000 7fa249cbf640 1 --2- 172.21.15.16:0/3731072681 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fa2300654d0 0x7fa230067980 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:11.216 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:11.211+0000 7fa249cbf640 1 --2- 172.21.15.16:0/3731072681 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa22c0047d0 0x7fa24414f260 unknown :-1 s=CLOSED pgs=2665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:11.216 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:11.211+0000 7fa249cbf640 1 -- 172.21.15.16:0/3731072681 >> 172.21.15.16:0/3731072681 conn(0x7fa244071850 msgr2=0x7fa24406bd90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:06:11.216 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:11.211+0000 7fa249cbf640 1 -- 172.21.15.16:0/3731072681 shutdown_connections 2023-05-22T16:06:11.217 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:11.211+0000 7fa249cbf640 1 -- 172.21.15.16:0/3731072681 wait complete. 2023-05-22T16:06:11.230 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-05-22T16:06:11.352 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:11.347+0000 7fc3f1863640 1 Processor -- start 2023-05-22T16:06:11.353 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:11.347+0000 7fc3f1863640 1 -- start start 2023-05-22T16:06:11.354 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:11.351+0000 7fc3f1863640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc3ec1565d0 0x7fc3ec0d0090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:11.355 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:11.351+0000 7fc3f1863640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3ec0cd650 con 0x7fc3ec0d05d0 2023-05-22T16:06:11.355 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:11.351+0000 7fc3f1863640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3ec0ccd20 con 0x7fc3ec15cdb0 2023-05-22T16:06:11.355 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:11.351+0000 7fc3f1863640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3ec0ccb10 con 0x7fc3ec1565d0 2023-05-22T16:06:11.356 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:11.351+0000 7fc3ea7fc640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fc3ec0d05d0 0x7fc3ec15c6a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:46424/0 (socket says 172.21.15.16:46424) 2023-05-22T16:06:11.356 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:11.351+0000 7fc3ea7fc640 1 -- 172.21.15.16:0/3311040287 learned_addr learned my addr 172.21.15.16:0/3311040287 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T16:06:11.356 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:11.351+0000 7fc3eaffd640 1 --2- 172.21.15.16:0/3311040287 >> v2:172.21.15.16:3300/0 conn(0x7fc3ec1565d0 0x7fc3ec0d0090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:11.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:11.351+0000 7fc3e9ffb640 1 -- 172.21.15.16:0/3311040287 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4242592132 0 0) 0x7fc3ec0cd650 con 0x7fc3ec0d05d0 2023-05-22T16:06:11.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:11.351+0000 7fc3e9ffb640 1 -- 172.21.15.16:0/3311040287 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3d4003530 con 0x7fc3ec0d05d0 2023-05-22T16:06:11.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:11.351+0000 7fc3eaffd640 1 -- 172.21.15.16:0/3311040287 >> v1:172.21.15.62:6789/0 conn(0x7fc3ec15cdb0 legacy=0x7fc3ec0b0440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:06:11.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:11.351+0000 7fc3eaffd640 1 -- 172.21.15.16:0/3311040287 >> v1:172.21.15.16:6789/0 conn(0x7fc3ec0d05d0 legacy=0x7fc3ec15c6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:06:11.358 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:11.351+0000 7fc3e9ffb640 1 -- 172.21.15.16:0/3311040287 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1640885360 0 0) 0x7fc3d4003530 con 0x7fc3ec0d05d0 2023-05-22T16:06:11.358 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:11.351+0000 7fc3eaffd640 1 -- 172.21.15.16:0/3311040287 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3ec0cd000 con 0x7fc3ec1565d0 2023-05-22T16:06:11.363 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:11.351+0000 7fc3eaffd640 1 --2- 172.21.15.16:0/3311040287 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc3ec1565d0 0x7fc3ec0d0090 secure :-1 s=READY pgs=2667 cs=0 l=1 rev1=1 crypto rx=0x7fc3dc0097b0 tx=0x7fc3dc01c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=540cf5a6ed74d416 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:06:11.363 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:11.351+0000 7fc3e9ffb640 1 -- 172.21.15.16:0/3311040287 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3e0001680 con 0x7fc3ec1565d0 2023-05-22T16:06:11.363 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:11.351+0000 7fc3e9ffb640 1 -- 172.21.15.16:0/3311040287 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3dc0a7050 con 0x7fc3ec1565d0 2023-05-22T16:06:11.364 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:11.351+0000 7fc3e9ffb640 1 -- 172.21.15.16:0/3311040287 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3dc0ab620 con 0x7fc3ec1565d0 2023-05-22T16:06:11.364 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:11.355+0000 7fc3f1863640 1 -- 172.21.15.16:0/3311040287 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc3ec1565d0 msgr2=0x7fc3ec0d0090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:06:11.364 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:11.355+0000 7fc3f1863640 1 --2- 172.21.15.16:0/3311040287 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc3ec1565d0 0x7fc3ec0d0090 secure :-1 s=READY pgs=2667 cs=0 l=1 rev1=1 crypto rx=0x7fc3dc0097b0 tx=0x7fc3dc01c3a0 comp rx=0 tx=0).stop 2023-05-22T16:06:11.365 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:11.355+0000 7fc3f1863640 1 -- 172.21.15.16:0/3311040287 shutdown_connections 2023-05-22T16:06:11.365 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:11.355+0000 7fc3f1863640 1 --2- 172.21.15.16:0/3311040287 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc3ec1565d0 0x7fc3ec0d0090 unknown :-1 s=CLOSED pgs=2667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:11.365 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:11.355+0000 7fc3f1863640 1 -- 172.21.15.16:0/3311040287 >> 172.21.15.16:0/3311040287 conn(0x7fc3ec071850 msgr2=0x7fc3ec09b830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:06:11.366 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:11.355+0000 7fc3f1863640 1 -- 172.21.15.16:0/3311040287 shutdown_connections 2023-05-22T16:06:11.366 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:11.355+0000 7fc3f1863640 1 -- 172.21.15.16:0/3311040287 wait complete. 2023-05-22T16:06:11.366 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:11.355+0000 7fc3f1863640 1 Processor -- start 2023-05-22T16:06:11.366 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:11.355+0000 7fc3f1863640 1 -- start start 2023-05-22T16:06:11.367 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:11.355+0000 7fc3f1863640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc3ec0d05d0 0x7fc3ec153cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:11.367 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:11.355+0000 7fc3f1863640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc3ec1565d0 0x7fc3ec1407c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:11.367 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:11.355+0000 7fc3f1863640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc3ec15cdb0 0x7fc3ec152170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:11.367 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:11.355+0000 7fc3f1863640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3ec1541f0 con 0x7fc3ec1565d0 2023-05-22T16:06:11.368 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:11.355+0000 7fc3f1863640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3ec140d00 con 0x7fc3ec0d05d0 2023-05-22T16:06:11.368 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:11.355+0000 7fc3f1863640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3ec152c10 con 0x7fc3ec15cdb0 2023-05-22T16:06:11.368 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:11.355+0000 7fc3ea7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc3ec1565d0 0x7fc3ec1407c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:11.369 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:11.355+0000 7fc3ea7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc3ec1565d0 0x7fc3ec1407c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57804/0 (socket says 172.21.15.16:57804) 2023-05-22T16:06:11.369 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:11.355+0000 7fc3ea7fc640 1 -- 172.21.15.16:0/651368446 learned_addr learned my addr 172.21.15.16:0/651368446 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:06:11.369 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:11.355+0000 7fc3eaffd640 1 --2- 172.21.15.16:0/651368446 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc3ec0d05d0 0x7fc3ec153cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:11.369 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:11.355+0000 7fc3eb7fe640 1 --2- 172.21.15.16:0/651368446 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc3ec15cdb0 0x7fc3ec152170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:11.370 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:11.355+0000 7fc3ea7fc640 1 -- 172.21.15.16:0/651368446 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc3ec15cdb0 msgr2=0x7fc3ec152170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:06:11.370 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:11.355+0000 7fc3ea7fc640 1 --2- 172.21.15.16:0/651368446 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc3ec15cdb0 0x7fc3ec152170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:11.370 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:11.355+0000 7fc3ea7fc640 1 -- 172.21.15.16:0/651368446 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc3ec0d05d0 msgr2=0x7fc3ec153cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:06:11.370 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:11.355+0000 7fc3ea7fc640 1 --2- 172.21.15.16:0/651368446 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc3ec0d05d0 0x7fc3ec153cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:11.371 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:11.355+0000 7fc3ea7fc640 1 -- 172.21.15.16:0/651368446 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3dc0093c0 con 0x7fc3ec1565d0 2023-05-22T16:06:11.371 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:11.355+0000 7fc3eaffd640 1 --2- 172.21.15.16:0/651368446 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc3ec0d05d0 0x7fc3ec153cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T16:06:11.371 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:11.355+0000 7fc3eb7fe640 1 --2- 172.21.15.16:0/651368446 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc3ec15cdb0 0x7fc3ec152170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T16:06:11.372 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:11.355+0000 7fc3ea7fc640 1 --2- 172.21.15.16:0/651368446 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc3ec1565d0 0x7fc3ec1407c0 secure :-1 s=READY pgs=2668 cs=0 l=1 rev1=1 crypto rx=0x7fc3e000c780 tx=0x7fc3e000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:06:11.372 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:11.355+0000 7fc3d3fff640 1 -- 172.21.15.16:0/651368446 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3e0005ae0 con 0x7fc3ec1565d0 2023-05-22T16:06:11.372 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:11.355+0000 7fc3d3fff640 1 -- 172.21.15.16:0/651368446 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3e0005c60 con 0x7fc3ec1565d0 2023-05-22T16:06:11.373 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:11.355+0000 7fc3f1863640 1 -- 172.21.15.16:0/651368446 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3ec0af090 con 0x7fc3ec1565d0 2023-05-22T16:06:11.373 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:11.355+0000 7fc3d3fff640 1 -- 172.21.15.16:0/651368446 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3e0007070 con 0x7fc3ec1565d0 2023-05-22T16:06:11.373 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:11.355+0000 7fc3f1863640 1 -- 172.21.15.16:0/651368446 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3ec15d1b0 con 0x7fc3ec1565d0 2023-05-22T16:06:11.373 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:11.359+0000 7fc3f1863640 1 -- 172.21.15.16:0/651368446 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3ec0af220 con 0x7fc3ec1565d0 2023-05-22T16:06:11.374 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:11.359+0000 7fc3d3fff640 1 -- 172.21.15.16:0/651368446 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 100) v1 ==== 84332+0+0 (secure 0 0 0) 0x7fc3e00071d0 con 0x7fc3ec1565d0 2023-05-22T16:06:11.374 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:11.359+0000 7fc3d3fff640 1 --2- 172.21.15.16:0/651368446 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fc3d80654a0 0x7fc3d8067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:11.374 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:11.359+0000 7fc3eaffd640 1 --2- 172.21.15.16:0/651368446 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fc3d80654a0 0x7fc3d8067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:11.375 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:11.359+0000 7fc3d3fff640 1 -- 172.21.15.16:0/651368446 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(201..201 src has 1..201) v4 ==== 6317+0+0 (secure 0 0 0) 0x7fc3e0082150 con 0x7fc3ec1565d0 2023-05-22T16:06:11.375 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:11.359+0000 7fc3eaffd640 1 --2- 172.21.15.16:0/651368446 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fc3d80654a0 0x7fc3d8067950 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fc3dc0ad000 tx=0x7fc3dc008040 comp rx=0 tx=0).ready entity=mgr.8195 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:06:11.376 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:11.363+0000 7fc3d3fff640 1 -- 172.21.15.16:0/651368446 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fc3e004fd10 con 0x7fc3ec1565d0 2023-05-22T16:06:11.516 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:11.511+0000 7fc3f1863640 1 -- 172.21.15.16:0/651368446 --> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fc3ec073830 con 0x7fc3d80654a0 2023-05-22T16:06:11.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:11.519+0000 7fc3d3fff640 1 -- 172.21.15.16:0/651368446 <== mgr.8195 v2:172.21.15.16:6832/65890 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+71134 (secure 0 0 0) 0x7fc3ec073830 con 0x7fc3d80654a0 2023-05-22T16:06:11.522 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T16:06:11.522 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2023-05-22T16:06:11.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:11.519+0000 7fc3f1863640 1 -- 172.21.15.16:0/651368446 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fc3d80654a0 msgr2=0x7fc3d8067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:06:11.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:11.519+0000 7fc3f1863640 1 --2- 172.21.15.16:0/651368446 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fc3d80654a0 0x7fc3d8067950 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fc3dc0ad000 tx=0x7fc3dc008040 comp rx=0 tx=0).stop 2023-05-22T16:06:11.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:11.519+0000 7fc3f1863640 1 -- 172.21.15.16:0/651368446 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc3ec1565d0 msgr2=0x7fc3ec1407c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:06:11.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:11.519+0000 7fc3f1863640 1 --2- 172.21.15.16:0/651368446 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc3ec1565d0 0x7fc3ec1407c0 secure :-1 s=READY pgs=2668 cs=0 l=1 rev1=1 crypto rx=0x7fc3e000c780 tx=0x7fc3e000cc40 comp rx=0 tx=0).stop 2023-05-22T16:06:11.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:11.519+0000 7fc3f1863640 1 -- 172.21.15.16:0/651368446 shutdown_connections 2023-05-22T16:06:11.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:11.519+0000 7fc3f1863640 1 --2- 172.21.15.16:0/651368446 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fc3ec15cdb0 0x7fc3ec152170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:11.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:11.519+0000 7fc3f1863640 1 --2- 172.21.15.16:0/651368446 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fc3d80654a0 0x7fc3d8067950 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:11.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:11.519+0000 7fc3f1863640 1 --2- 172.21.15.16:0/651368446 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc3ec1565d0 0x7fc3ec1407c0 unknown :-1 s=CLOSED pgs=2668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:11.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:11.519+0000 7fc3f1863640 1 --2- 172.21.15.16:0/651368446 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc3ec0d05d0 0x7fc3ec153cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:11.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:11.519+0000 7fc3f1863640 1 -- 172.21.15.16:0/651368446 >> 172.21.15.16:0/651368446 conn(0x7fc3ec071850 msgr2=0x7fc3ec06bcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:06:11.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:11.519+0000 7fc3f1863640 1 -- 172.21.15.16:0/651368446 shutdown_connections 2023-05-22T16:06:11.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:11.519+0000 7fc3f1863640 1 -- 172.21.15.16:0/651368446 wait complete. 2023-05-22T16:06:11.539 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":13,"stamp":"2023-05-22T16:06:09.802974+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2132,"num_read_kb":4426,"num_write":579,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":348,"ondisk_log_size":348,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":41,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":7,"kb":754974720,"kb_used":258760,"kb_used_data":20872,"kb_used_omap":0,"kb_used_meta":237824,"kb_avail":754715960,"statfs":{"total":773094113280,"available":772829143040,"internally_reserved":0,"allocated":21372928,"data_stored":11983984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":243531776},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.468815"},"pg_stats":[{"pgid":"1.0","version":"198'348","reported_seq":1162,"reported_epoch":200,"state":"active+clean","last_fresh":"2023-05-22T16:06:02.067357+0000","last_change":"2023-05-22T15:26:39.318204+0000","last_active":"2023-05-22T16:06:02.067357+0000","last_peered":"2023-05-22T16:06:02.067357+0000","last_clean":"2023-05-22T16:06:02.067357+0000","last_became_active":"2023-05-22T15:26:39.315370+0000","last_became_peered":"2023-05-22T15:26:39.315370+0000","last_unstale":"2023-05-22T16:06:02.067357+0000","last_undegraded":"2023-05-22T16:06:02.067357+0000","last_fullsized":"2023-05-22T16:06:02.067357+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:38.020076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:38.020076+0000","last_clean_scrub_stamp":"2023-05-22T15:26:38.020076+0000","objects_scrubbed":0,"log_size":348,"log_dups_size":0,"ondisk_log_size":348,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:06:39.207543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2132,"num_read_kb":4426,"num_write":579,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":201,"reported_epoch":200,"state":"active+clean","last_fresh":"2023-05-22T16:06:02.122284+0000","last_change":"2023-05-22T15:26:44.971951+0000","last_active":"2023-05-22T16:06:02.122284+0000","last_peered":"2023-05-22T16:06:02.122284+0000","last_clean":"2023-05-22T16:06:02.122284+0000","last_became_active":"2023-05-22T15:26:44.970872+0000","last_became_peered":"2023-05-22T15:26:44.970872+0000","last_unstale":"2023-05-22T16:06:02.122284+0000","last_undegraded":"2023-05-22T16:06:02.122284+0000","last_fullsized":"2023-05-22T16:06:02.122284+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:55:34.752949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":190,"reported_epoch":200,"state":"active+clean","last_fresh":"2023-05-22T16:06:02.067104+0000","last_change":"2023-05-22T15:26:44.977198+0000","last_active":"2023-05-22T16:06:02.067104+0000","last_peered":"2023-05-22T16:06:02.067104+0000","last_clean":"2023-05-22T16:06:02.067104+0000","last_became_active":"2023-05-22T15:26:44.976445+0000","last_became_peered":"2023-05-22T15:26:44.976445+0000","last_unstale":"2023-05-22T16:06:02.067104+0000","last_undegraded":"2023-05-22T16:06:02.067104+0000","last_fullsized":"2023-05-22T16:06:02.067104+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:12:41.464054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":194,"reported_epoch":200,"state":"active+clean","last_fresh":"2023-05-22T16:06:01.336116+0000","last_change":"2023-05-22T15:26:44.969469+0000","last_active":"2023-05-22T16:06:01.336116+0000","last_peered":"2023-05-22T16:06:01.336116+0000","last_clean":"2023-05-22T16:06:01.336116+0000","last_became_active":"2023-05-22T15:26:44.969083+0000","last_became_peered":"2023-05-22T15:26:44.969083+0000","last_unstale":"2023-05-22T16:06:01.336116+0000","last_undegraded":"2023-05-22T16:06:01.336116+0000","last_fullsized":"2023-05-22T16:06:01.336116+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:29:23.670067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":201,"reported_epoch":200,"state":"active+clean","last_fresh":"2023-05-22T16:06:02.122520+0000","last_change":"2023-05-22T15:26:44.976632+0000","last_active":"2023-05-22T16:06:02.122520+0000","last_peered":"2023-05-22T16:06:02.122520+0000","last_clean":"2023-05-22T16:06:02.122520+0000","last_became_active":"2023-05-22T15:26:44.975546+0000","last_became_peered":"2023-05-22T15:26:44.975546+0000","last_unstale":"2023-05-22T16:06:02.122520+0000","last_undegraded":"2023-05-22T16:06:02.122520+0000","last_fullsized":"2023-05-22T16:06:02.122520+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:42:57.973030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":179,"reported_epoch":200,"state":"active+clean","last_fresh":"2023-05-22T16:06:01.343165+0000","last_change":"2023-05-22T15:26:45.091475+0000","last_active":"2023-05-22T16:06:01.343165+0000","last_peered":"2023-05-22T16:06:01.343165+0000","last_clean":"2023-05-22T16:06:01.343165+0000","last_became_active":"2023-05-22T15:26:45.091143+0000","last_became_peered":"2023-05-22T15:26:45.091143+0000","last_unstale":"2023-05-22T16:06:01.343165+0000","last_undegraded":"2023-05-22T16:06:01.343165+0000","last_fullsized":"2023-05-22T16:06:01.343165+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T00:12:41.464054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":189,"reported_epoch":200,"state":"active+clean","last_fresh":"2023-05-22T16:06:02.066368+0000","last_change":"2023-05-22T15:26:45.092367+0000","last_active":"2023-05-22T16:06:02.066368+0000","last_peered":"2023-05-22T16:06:02.066368+0000","last_clean":"2023-05-22T16:06:02.066368+0000","last_became_active":"2023-05-22T15:26:45.091951+0000","last_became_peered":"2023-05-22T15:26:45.091951+0000","last_unstale":"2023-05-22T16:06:02.066368+0000","last_undegraded":"2023-05-22T16:06:02.066368+0000","last_fullsized":"2023-05-22T16:06:02.066368+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:42:57.973030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":179,"reported_epoch":200,"state":"active+clean","last_fresh":"2023-05-22T16:06:01.346627+0000","last_change":"2023-05-22T15:26:44.978858+0000","last_active":"2023-05-22T16:06:01.346627+0000","last_peered":"2023-05-22T16:06:01.346627+0000","last_clean":"2023-05-22T16:06:01.346627+0000","last_became_active":"2023-05-22T15:26:44.978443+0000","last_became_peered":"2023-05-22T15:26:44.978443+0000","last_unstale":"2023-05-22T16:06:01.346627+0000","last_undegraded":"2023-05-22T16:06:01.346627+0000","last_fullsized":"2023-05-22T16:06:01.346627+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T23:46:29.290243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":178,"reported_epoch":200,"state":"active+clean","last_fresh":"2023-05-22T16:06:02.374912+0000","last_change":"2023-05-22T15:26:44.982421+0000","last_active":"2023-05-22T16:06:02.374912+0000","last_peered":"2023-05-22T16:06:02.374912+0000","last_clean":"2023-05-22T16:06:02.374912+0000","last_became_active":"2023-05-22T15:26:44.981586+0000","last_became_peered":"2023-05-22T15:26:44.981586+0000","last_unstale":"2023-05-22T16:06:02.374912+0000","last_undegraded":"2023-05-22T16:06:02.374912+0000","last_fullsized":"2023-05-22T16:06:02.374912+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:14:11.156544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2132,"num_read_kb":4426,"num_write":579,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":348,"ondisk_log_size":348,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575315,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29676,"kb_used_data":3044,"kb_used_omap":0,"kb_used_meta":26624,"kb_avail":94342164,"statfs":{"total":96636764160,"available":96606375936,"internally_reserved":0,"allocated":3117056,"data_stored":1940774,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27262976},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 16:05:53 2023","interfaces":[{"interface":"back","average":{"1min":0.81999999999999995,"5min":0.85099999999999998,"15min":0.86099999999999999},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":1.222,"5min":1.3069999999999999,"15min":1.3069999999999999},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.84999999999999998,"5min":0.88200000000000001,"15min":0.877},"min":{"1min":0.61299999999999999,"5min":0.501,"15min":0.501},"max":{"1min":1.296,"5min":1.296,"15min":1.296},"last":0.63100000000000001}]},{"osd":1,"last update":"Mon May 22 16:05:53 2023","interfaces":[{"interface":"back","average":{"1min":0.90400000000000003,"5min":0.97299999999999998,"15min":0.98999999999999999},"min":{"1min":0.66000000000000003,"5min":0.63400000000000001,"15min":0.58299999999999996},"max":{"1min":1.274,"5min":1.431,"15min":1.431},"last":1.0880000000000001},{"interface":"front","average":{"1min":0.95499999999999996,"5min":1.026,"15min":1.026},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":1.2909999999999999,"5min":1.345,"15min":1.345},"last":0.83899999999999997}]},{"osd":2,"last update":"Mon May 22 16:05:53 2023","interfaces":[{"interface":"back","average":{"1min":0.83899999999999997,"5min":0.89900000000000002,"15min":0.90400000000000003},"min":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.171,"5min":1.286,"15min":1.407},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.89100000000000001,"5min":0.94899999999999995,"15min":0.95899999999999996},"min":{"1min":0.53200000000000003,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":1.3899999999999999,"5min":1.3899999999999999,"15min":1.3899999999999999},"last":0.74299999999999999}]},{"osd":3,"last update":"Mon May 22 16:05:53 2023","interfaces":[{"interface":"back","average":{"1min":0.96399999999999997,"5min":0.93300000000000005,"15min":0.94799999999999995},"min":{"1min":0.755,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.2,"5min":1.2909999999999999,"15min":1.401},"last":1.0109999999999999},{"interface":"front","average":{"1min":0.97199999999999998,"5min":0.98699999999999999,"15min":0.98099999999999998},"min":{"1min":0.60299999999999998,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":1.4179999999999999,"5min":1.4179999999999999,"15min":1.4179999999999999},"last":0.93000000000000005}]},{"osd":4,"last update":"Mon May 22 16:05:53 2023","interfaces":[{"interface":"back","average":{"1min":0.875,"5min":0.88900000000000001,"15min":0.88600000000000001},"min":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51000000000000001},"max":{"1min":1.2310000000000001,"5min":1.3440000000000001,"15min":1.3440000000000001},"last":0.88},{"interface":"front","average":{"1min":0.95599999999999996,"5min":0.99099999999999999,"15min":0.96499999999999997},"min":{"1min":0.60899999999999999,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":1.3480000000000001,"5min":1.405,"15min":1.405},"last":1.1699999999999999}]},{"osd":5,"last update":"Mon May 22 16:05:53 2023","interfaces":[{"interface":"back","average":{"1min":0.96499999999999997,"5min":0.98899999999999999,"15min":0.98999999999999999},"min":{"1min":0.66200000000000003,"5min":0.623,"15min":0.441},"max":{"1min":1.159,"5min":1.2909999999999999,"15min":1.3759999999999999},"last":1.2030000000000001},{"interface":"front","average":{"1min":1.1180000000000001,"5min":1.1339999999999999,"15min":1.135},"min":{"1min":0.91000000000000003,"5min":0.72299999999999998,"15min":0.66600000000000004},"max":{"1min":1.468,"5min":1.468,"15min":1.468},"last":1.129}]},{"osd":6,"last update":"Mon May 22 16:05:53 2023","interfaces":[{"interface":"back","average":{"1min":1.048,"5min":1.0600000000000001,"15min":1.0569999999999999},"min":{"1min":0.72799999999999998,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":1.288,"5min":1.3169999999999999,"15min":1.389},"last":0.97699999999999998},{"interface":"front","average":{"1min":1.008,"5min":1.0940000000000001,"15min":1.0880000000000001},"min":{"1min":0.70399999999999996,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":1.3149999999999999,"5min":1.391,"15min":1.7050000000000001},"last":1.2430000000000001}]}]},{"osd":6,"up_from":13,"seq":55834575316,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":22824,"kb_used_data":2464,"kb_used_omap":0,"kb_used_meta":20352,"kb_avail":94349016,"statfs":{"total":96636764160,"available":96613392384,"internally_reserved":0,"allocated":2523136,"data_stored":1350406,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":20840448},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 16:05:56 2023","interfaces":[{"interface":"back","average":{"1min":0.85999999999999999,"5min":0.88,"15min":0.89600000000000002},"min":{"1min":0.48599999999999999,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":2.6469999999999998,"5min":2.6469999999999998,"15min":2.6469999999999998},"last":0.71299999999999997},{"interface":"front","average":{"1min":1.254,"5min":0.96899999999999997,"15min":0.92800000000000005},"min":{"1min":0.64600000000000002,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":6.7439999999999998,"5min":6.7439999999999998,"15min":6.7439999999999998},"last":0.86899999999999999}]},{"osd":1,"last update":"Mon May 22 16:05:56 2023","interfaces":[{"interface":"back","average":{"1min":1.0009999999999999,"5min":0.93000000000000005,"15min":0.94199999999999995},"min":{"1min":0.57399999999999995,"5min":0.55300000000000005,"15min":0.51100000000000001},"max":{"1min":2.3290000000000002,"5min":2.3290000000000002,"15min":2.3290000000000002},"last":1.147},{"interface":"front","average":{"1min":1.2330000000000001,"5min":0.97899999999999998,"15min":0.94699999999999995},"min":{"1min":0.58199999999999996,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":6.5010000000000003,"5min":6.5010000000000003,"15min":6.5010000000000003},"last":0.60399999999999998}]},{"osd":2,"last update":"Mon May 22 16:05:56 2023","interfaces":[{"interface":"back","average":{"1min":1.2210000000000001,"5min":1.012,"15min":1.0109999999999999},"min":{"1min":0.57399999999999995,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":7.0170000000000003,"5min":7.0170000000000003,"15min":7.0170000000000003},"last":0.81799999999999995},{"interface":"front","average":{"1min":1.0589999999999999,"5min":1.05,"15min":1.0720000000000001},"min":{"1min":0.66200000000000003,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":2.403,"5min":2.403,"15min":2.403},"last":1.276}]},{"osd":3,"last update":"Mon May 22 16:05:56 2023","interfaces":[{"interface":"back","average":{"1min":1.278,"5min":1.054,"15min":1.0129999999999999},"min":{"1min":0.63800000000000001,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":6.6699999999999999,"5min":6.6699999999999999,"15min":6.6699999999999999},"last":0.77400000000000002},{"interface":"front","average":{"1min":1.244,"5min":1.02,"15min":0.98899999999999999},"min":{"1min":0.68000000000000005,"5min":0.622,"15min":0.622},"max":{"1min":7.1260000000000003,"5min":7.1260000000000003,"15min":7.1260000000000003},"last":0.90600000000000003}]},{"osd":4,"last update":"Mon May 22 16:05:56 2023","interfaces":[{"interface":"back","average":{"1min":1.161,"5min":0.98099999999999998,"15min":0.96999999999999997},"min":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"max":{"1min":4.0709999999999997,"5min":4.0709999999999997,"15min":4.0709999999999997},"last":0.84299999999999997},{"interface":"front","average":{"1min":1.583,"5min":1.1619999999999999,"15min":1.075},"min":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":11.581,"5min":11.581,"15min":11.581},"last":0.95399999999999996}]},{"osd":5,"last update":"Mon May 22 16:05:56 2023","interfaces":[{"interface":"back","average":{"1min":1.925,"5min":1.1739999999999999,"15min":1.056},"min":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":17.587,"5min":17.587,"15min":17.587},"last":1.216},{"interface":"front","average":{"1min":1.204,"5min":1.1180000000000001,"15min":1.1240000000000001},"min":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"max":{"1min":2.9620000000000002,"5min":2.9620000000000002,"15min":2.9620000000000002},"last":1.339}]},{"osd":7,"last update":"Mon May 22 16:05:56 2023","interfaces":[{"interface":"back","average":{"1min":1.4319999999999999,"5min":1.202,"15min":1.1559999999999999},"min":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.58599999999999997},"max":{"1min":6.8339999999999996,"5min":6.8339999999999996,"15min":6.8339999999999996},"last":1.034},{"interface":"front","average":{"1min":1.5429999999999999,"5min":1.2090000000000001,"15min":1.163},"min":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":8.5990000000000002,"5min":8.5990000000000002,"15min":8.5990000000000002},"last":0.98399999999999999}]}]},{"osd":5,"up_from":13,"seq":55834575316,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32936,"kb_used_data":2464,"kb_used_omap":0,"kb_used_meta":30464,"kb_avail":94338904,"statfs":{"total":96636764160,"available":96603037696,"internally_reserved":0,"allocated":2523136,"data_stored":1350406,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31195136},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 16:05:54 2023","interfaces":[{"interface":"back","average":{"1min":0.76000000000000001,"5min":0.85199999999999998,"15min":0.875},"min":{"1min":0.47899999999999998,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.0820000000000001,"5min":1.2749999999999999,"15min":1.3959999999999999},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.86499999999999999,"5min":0.93999999999999995,"15min":0.94299999999999995},"min":{"1min":0.56499999999999995,"5min":0.56200000000000006,"15min":0.502},"max":{"1min":1.165,"5min":1.325,"15min":1.325},"last":0.71099999999999997}]},{"osd":1,"last update":"Mon May 22 16:05:54 2023","interfaces":[{"interface":"back","average":{"1min":0.86599999999999999,"5min":0.95999999999999996,"15min":1.0009999999999999},"min":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.2909999999999999,"5min":1.365,"15min":1.365},"last":0.76400000000000001},{"interface":"front","average":{"1min":0.89600000000000002,"5min":1.028,"15min":1.0640000000000001},"min":{"1min":0.46899999999999997,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":1.216,"5min":1.3520000000000001,"15min":1.4630000000000001},"last":0.85699999999999998}]},{"osd":2,"last update":"Mon May 22 16:05:54 2023","interfaces":[{"interface":"back","average":{"1min":0.85299999999999998,"5min":0.90500000000000003,"15min":0.92000000000000004},"min":{"1min":0.60899999999999999,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.1859999999999999,"5min":1.3200000000000001,"15min":1.3740000000000001},"last":0.995},{"interface":"front","average":{"1min":0.86799999999999999,"5min":0.94699999999999995,"15min":0.94599999999999995},"min":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":1.117,"5min":1.4350000000000001,"15min":1.4350000000000001},"last":0.60399999999999998}]},{"osd":3,"last update":"Mon May 22 16:05:54 2023","interfaces":[{"interface":"back","average":{"1min":0.91300000000000003,"5min":0.97699999999999998,"15min":0.97599999999999998},"min":{"1min":0.63400000000000001,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":1.1759999999999999,"5min":1.349,"15min":1.367},"last":0.95199999999999996},{"interface":"front","average":{"1min":0.97999999999999998,"5min":1.0329999999999999,"15min":1.048},"min":{"1min":0.67600000000000005,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.1779999999999999,"5min":1.3899999999999999,"15min":1.3899999999999999},"last":1.087}]},{"osd":4,"last update":"Mon May 22 16:05:54 2023","interfaces":[{"interface":"back","average":{"1min":0.91200000000000003,"5min":0.93600000000000005,"15min":0.93100000000000005},"min":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.2170000000000001,"5min":1.6000000000000001,"15min":1.6000000000000001},"last":0.89800000000000002},{"interface":"front","average":{"1min":0.98899999999999999,"5min":0.98499999999999999,"15min":1.01},"min":{"1min":0.57899999999999996,"5min":0.498,"15min":0.498},"max":{"1min":1.371,"5min":1.4630000000000001,"15min":1.4630000000000001},"last":1.036}]},{"osd":6,"last update":"Mon May 22 16:05:54 2023","interfaces":[{"interface":"back","average":{"1min":1.054,"5min":1.0129999999999999,"15min":1.0209999999999999},"min":{"1min":0.68999999999999995,"5min":0.63200000000000001,"15min":0.59799999999999998},"max":{"1min":1.2509999999999999,"5min":1.4630000000000001,"15min":1.4790000000000001},"last":1.1819999999999999},{"interface":"front","average":{"1min":1.0449999999999999,"5min":1.073,"15min":1.081},"min":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":1.3440000000000001,"5min":1.6080000000000001,"15min":1.6080000000000001},"last":0.97099999999999997}]},{"osd":7,"last update":"Mon May 22 16:05:54 2023","interfaces":[{"interface":"back","average":{"1min":1.133,"5min":1.1180000000000001,"15min":1.1259999999999999},"min":{"1min":0.83899999999999997,"5min":0.57699999999999996,"15min":0.57699999999999996},"max":{"1min":1.3680000000000001,"5min":1.51,"15min":1.5109999999999999},"last":1.125},{"interface":"front","average":{"1min":1.1739999999999999,"5min":1.1859999999999999,"15min":1.1930000000000001},"min":{"1min":0.91000000000000003,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":1.353,"5min":1.6679999999999999,"15min":1.7210000000000001},"last":1.26}]}]},{"osd":4,"up_from":13,"seq":55834575314,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":39656,"kb_used_data":2464,"kb_used_omap":0,"kb_used_meta":37184,"kb_avail":94332184,"statfs":{"total":96636764160,"available":96596156416,"internally_reserved":0,"allocated":2523136,"data_stored":1350406,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":38076416},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 16:06:05 2023","interfaces":[{"interface":"back","average":{"1min":0.88200000000000001,"5min":0.89400000000000002,"15min":0.89800000000000002},"min":{"1min":0.58199999999999996,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.1890000000000001,"5min":1.7210000000000001,"15min":1.7210000000000001},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.86799999999999999,"5min":0.93899999999999995,"15min":0.94799999999999995},"min":{"1min":0.49099999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":1.1419999999999999,"5min":1.7290000000000001,"15min":1.7290000000000001},"last":1.0609999999999999}]},{"osd":1,"last update":"Mon May 22 16:06:05 2023","interfaces":[{"interface":"back","average":{"1min":0.91300000000000003,"5min":0.95099999999999996,"15min":0.96199999999999997},"min":{"1min":0.60699999999999998,"5min":0.314,"15min":0.314},"max":{"1min":1.242,"5min":1.9339999999999999,"15min":1.9339999999999999},"last":0.80700000000000005},{"interface":"front","average":{"1min":0.82299999999999995,"5min":0.89600000000000002,"15min":0.93000000000000005},"min":{"1min":0.54000000000000004,"5min":0.504,"15min":0.504},"max":{"1min":1.149,"5min":1.365,"15min":1.365},"last":0.68899999999999995}]},{"osd":2,"last update":"Mon May 22 16:06:05 2023","interfaces":[{"interface":"back","average":{"1min":0.87,"5min":0.94199999999999995,"15min":0.94399999999999995},"min":{"1min":0.53500000000000003,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":1.1759999999999999,"5min":1.4990000000000001,"15min":1.4990000000000001},"last":0.749},{"interface":"front","average":{"1min":0.92200000000000004,"5min":1.0129999999999999,"15min":1.0449999999999999},"min":{"1min":0.60799999999999998,"5min":0.44,"15min":0.44},"max":{"1min":1.1579999999999999,"5min":2.0249999999999999,"15min":2.0249999999999999},"last":0.93899999999999995}]},{"osd":3,"last update":"Mon May 22 16:06:05 2023","interfaces":[{"interface":"back","average":{"1min":0.93999999999999995,"5min":0.95799999999999996,"15min":0.96399999999999997},"min":{"1min":0.61699999999999999,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":1.2929999999999999,"5min":1.452,"15min":1.4610000000000001},"last":0.874},{"interface":"front","average":{"1min":0.97899999999999998,"5min":1.0329999999999999,"15min":1.044},"min":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":1.3140000000000001,"5min":1.349,"15min":1.484},"last":0.96899999999999997}]},{"osd":5,"last update":"Mon May 22 16:06:05 2023","interfaces":[{"interface":"back","average":{"1min":1.032,"5min":1.0289999999999999,"15min":1.0049999999999999},"min":{"1min":0.63600000000000001,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":1.353,"5min":2.1110000000000002,"15min":2.1110000000000002},"last":1.1259999999999999},{"interface":"front","average":{"1min":1.083,"5min":1.018,"15min":1.002},"min":{"1min":0.77300000000000002,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.413,"5min":1.5449999999999999,"15min":1.5449999999999999},"last":1.0129999999999999}]},{"osd":6,"last update":"Mon May 22 16:06:05 2023","interfaces":[{"interface":"back","average":{"1min":1.0029999999999999,"5min":1.02,"15min":1.006},"min":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"max":{"1min":1.254,"5min":1.397,"15min":1.4339999999999999},"last":1.1819999999999999},{"interface":"front","average":{"1min":1.111,"5min":1.137,"15min":1.1379999999999999},"min":{"1min":0.74399999999999999,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.266,"5min":2.2690000000000001,"15min":2.2690000000000001},"last":1.2150000000000001}]},{"osd":7,"last update":"Mon May 22 16:06:05 2023","interfaces":[{"interface":"back","average":{"1min":1.085,"5min":1.0840000000000001,"15min":1.0720000000000001},"min":{"1min":0.78700000000000003,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":1.4810000000000001,"5min":1.623,"15min":1.623},"last":1.149},{"interface":"front","average":{"1min":1.117,"5min":1.1659999999999999,"15min":1.159},"min":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.68100000000000005},"max":{"1min":1.389,"5min":1.448,"15min":1.5089999999999999},"last":1.099}]}]},{"osd":3,"up_from":13,"seq":55834575314,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27688,"kb_used_data":2464,"kb_used_omap":0,"kb_used_meta":25216,"kb_avail":94344152,"statfs":{"total":96636764160,"available":96608411648,"internally_reserved":0,"allocated":2523136,"data_stored":1350406,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":25821184},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 16:05:38 2023","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"min":{"1min":0.29299999999999998,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.93700000000000006,"5min":1.0049999999999999,"15min":1.0229999999999999},"last":0.20000000000000001},{"interface":"front","average":{"1min":0.70299999999999996,"5min":0.76200000000000001,"15min":0.76900000000000002},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":1.2749999999999999,"5min":1.7230000000000001,"15min":1.7230000000000001},"last":0.36299999999999999}]},{"osd":1,"last update":"Mon May 22 16:05:38 2023","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.75600000000000001,"15min":0.71999999999999997},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":1.2190000000000001,"5min":2.4969999999999999,"15min":2.4969999999999999},"last":0.27300000000000002},{"interface":"front","average":{"1min":0.76000000000000001,"5min":0.80700000000000005,"15min":0.80100000000000005},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":1.3520000000000001,"5min":2.4769999999999999,"15min":2.4769999999999999},"last":0.32300000000000001}]},{"osd":2,"last update":"Mon May 22 16:05:38 2023","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":0.83699999999999997,"15min":0.84899999999999998},"min":{"1min":0.52300000000000002,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.94799999999999995,"5min":2.2250000000000001,"15min":2.2250000000000001},"last":0.34300000000000003},{"interface":"front","average":{"1min":0.87,"5min":0.83599999999999997,"15min":0.83299999999999996},"min":{"1min":0.55500000000000005,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":1.129,"5min":2.1680000000000001,"15min":2.1680000000000001},"last":0.26000000000000001}]},{"osd":4,"last update":"Mon May 22 16:05:38 2023","interfaces":[{"interface":"back","average":{"1min":0.95199999999999996,"5min":0.98099999999999998,"15min":0.97399999999999998},"min":{"1min":0.36799999999999999,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":1.357,"5min":2.3479999999999999,"15min":2.3479999999999999},"last":0.47599999999999998},{"interface":"front","average":{"1min":1.0009999999999999,"5min":1.022,"15min":1.03},"min":{"1min":0.39400000000000002,"5min":0.37,"15min":0.37},"max":{"1min":1.2210000000000001,"5min":1.788,"15min":1.788},"last":0.65900000000000003}]},{"osd":5,"last update":"Mon May 22 16:05:38 2023","interfaces":[{"interface":"back","average":{"1min":1.0069999999999999,"5min":1.0409999999999999,"15min":1.05},"min":{"1min":0.496,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":1.21,"5min":1.8129999999999999,"15min":1.8129999999999999},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.98299999999999998,"5min":1.0169999999999999,"15min":1.0369999999999999},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.3140000000000001,"5min":1.571,"15min":1.571},"last":0.58099999999999996}]},{"osd":6,"last update":"Mon May 22 16:05:38 2023","interfaces":[{"interface":"back","average":{"1min":1.0109999999999999,"5min":1.0680000000000001,"15min":1.0880000000000001},"min":{"1min":0.45600000000000002,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":1.274,"5min":1.76,"15min":1.76},"last":0.625},{"interface":"front","average":{"1min":1.0609999999999999,"5min":1.0820000000000001,"15min":1.099},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":1.4139999999999999,"5min":1.754,"15min":1.754},"last":0.54900000000000004}]},{"osd":7,"last update":"Mon May 22 16:05:38 2023","interfaces":[{"interface":"back","average":{"1min":1.1020000000000001,"5min":1.111,"15min":1.127},"min":{"1min":0.53800000000000003,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.5049999999999999,"5min":1.673,"15min":1.673},"last":0.77500000000000002},{"interface":"front","average":{"1min":1.1040000000000001,"5min":1.1160000000000001,"15min":1.109},"min":{"1min":0.41099999999999998,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":1.4670000000000001,"5min":2.3079999999999998,"15min":2.3079999999999998},"last":0.53800000000000003}]}]},{"osd":2,"up_from":13,"seq":55834575315,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31400,"kb_used_data":2464,"kb_used_omap":0,"kb_used_meta":28928,"kb_avail":94340440,"statfs":{"total":96636764160,"available":96604610560,"internally_reserved":0,"allocated":2523136,"data_stored":1350406,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":29622272},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 16:05:35 2023","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.629,"15min":0.67000000000000004},"min":{"1min":0.28599999999999998,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.90100000000000002,"5min":1.0549999999999999,"15min":1.1519999999999999},"last":0.31},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.71799999999999997,"15min":0.74199999999999999},"min":{"1min":0.35299999999999998,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":1.0309999999999999,"5min":1.1659999999999999,"15min":1.2090000000000001},"last":0.34699999999999998}]},{"osd":1,"last update":"Mon May 22 16:05:35 2023","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.74399999999999999,"15min":0.75700000000000001},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.875,"5min":1.071,"15min":1.099},"last":0.38},{"interface":"front","average":{"1min":0.79300000000000004,"5min":0.82099999999999995,"15min":0.82599999999999996},"min":{"1min":0.39500000000000002,"5min":0.311,"15min":0.311},"max":{"1min":1.069,"5min":1.1739999999999999,"15min":1.2589999999999999},"last":0.45800000000000002}]},{"osd":3,"last update":"Mon May 22 16:05:35 2023","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":0.75600000000000001,"15min":0.75800000000000001},"min":{"1min":0.32100000000000001,"5min":0.315,"15min":0.315},"max":{"1min":0.91400000000000003,"5min":1.3540000000000001,"15min":1.3540000000000001},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.85199999999999998,"5min":0.87,"15min":0.89300000000000002},"min":{"1min":0.41399999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":1.1859999999999999,"5min":1.2949999999999999,"15min":1.2949999999999999},"last":0.41499999999999998}]},{"osd":4,"last update":"Mon May 22 16:05:35 2023","interfaces":[{"interface":"back","average":{"1min":1.02,"5min":1.024,"15min":1.04},"min":{"1min":0.66100000000000003,"5min":0.441,"15min":0.441},"max":{"1min":1.4590000000000001,"5min":1.516,"15min":1.6539999999999999},"last":0.82799999999999996},{"interface":"front","average":{"1min":0.996,"5min":1.0089999999999999,"15min":1.04},"min":{"1min":0.55600000000000005,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.419,"5min":1.4299999999999999,"15min":1.538},"last":0.61699999999999999}]},{"osd":5,"last update":"Mon May 22 16:05:35 2023","interfaces":[{"interface":"back","average":{"1min":1.03,"5min":1.024,"15min":1.0549999999999999},"min":{"1min":0.53600000000000003,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":1.4139999999999999,"5min":1.4139999999999999,"15min":1.605},"last":0.53200000000000003},{"interface":"front","average":{"1min":1.0069999999999999,"5min":1.038,"15min":1.03},"min":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":1.339,"5min":1.534,"15min":1.534},"last":0.72699999999999998}]},{"osd":6,"last update":"Mon May 22 16:05:35 2023","interfaces":[{"interface":"back","average":{"1min":1.095,"5min":1.0589999999999999,"15min":1.0349999999999999},"min":{"1min":0.72699999999999998,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":1.4259999999999999,"5min":1.429,"15min":1.462},"last":0.86499999999999999},{"interface":"front","average":{"1min":1.127,"5min":1.105,"15min":1.1339999999999999},"min":{"1min":0.59199999999999997,"5min":0.57099999999999995,"15min":0.57099999999999995},"max":{"1min":1.5269999999999999,"5min":1.621,"15min":1.7110000000000001},"last":0.78000000000000003}]},{"osd":7,"last update":"Mon May 22 16:05:35 2023","interfaces":[{"interface":"back","average":{"1min":1.0940000000000001,"5min":1.0860000000000001,"15min":1.1220000000000001},"min":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.5680000000000001,"5min":1.5680000000000001,"15min":1.6399999999999999},"last":0.66100000000000003},{"interface":"front","average":{"1min":1.0940000000000001,"5min":1.107,"15min":1.1140000000000001},"min":{"1min":0.629,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":1.4610000000000001,"5min":1.673,"15min":1.673},"last":0.81599999999999995}]}]},{"osd":1,"up_from":13,"seq":55834575315,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37544,"kb_used_data":2464,"kb_used_omap":0,"kb_used_meta":35072,"kb_avail":94334296,"statfs":{"total":96636764160,"available":96598319104,"internally_reserved":0,"allocated":2523136,"data_stored":1350406,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":35913728},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 16:05:58 2023","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.64800000000000002,"15min":0.66200000000000003},"min":{"1min":0.34499999999999997,"5min":0.308,"15min":0.308},"max":{"1min":1.0449999999999999,"5min":1.0449999999999999,"15min":1.329},"last":0.67400000000000004},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.72199999999999998,"15min":0.76300000000000001},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.95499999999999996,"5min":1.25,"15min":1.268},"last":0.57099999999999995}]},{"osd":2,"last update":"Mon May 22 16:05:58 2023","interfaces":[{"interface":"back","average":{"1min":0.70999999999999996,"5min":0.73399999999999999,"15min":0.75600000000000001},"min":{"1min":0.372,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":1.141,"5min":1.141,"15min":1.26},"last":0.88100000000000001},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.76300000000000001,"15min":0.77700000000000002},"min":{"1min":0.46999999999999997,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":1.0249999999999999,"5min":1.127,"15min":1.268},"last":0.64500000000000002}]},{"osd":3,"last update":"Mon May 22 16:05:58 2023","interfaces":[{"interface":"back","average":{"1min":0.78300000000000003,"5min":0.83799999999999997,"15min":0.878},"min":{"1min":0.39900000000000002,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":1.1100000000000001,"5min":1.2829999999999999,"15min":1.2829999999999999},"last":0.80700000000000005},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.80300000000000005,"15min":0.81899999999999995},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":1.2050000000000001,"5min":1.2749999999999999,"15min":1.2749999999999999},"last":0.83899999999999997}]},{"osd":4,"last update":"Mon May 22 16:05:58 2023","interfaces":[{"interface":"back","average":{"1min":0.93899999999999995,"5min":0.96599999999999997,"15min":0.98399999999999999},"min":{"1min":0.58499999999999996,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":1.3500000000000001,"5min":1.542,"15min":1.542},"last":0.78200000000000003},{"interface":"front","average":{"1min":1.006,"5min":0.997,"15min":1.024},"min":{"1min":0.71499999999999997,"5min":0.434,"15min":0.434},"max":{"1min":1.2729999999999999,"5min":1.327,"15min":1.4650000000000001},"last":0.95899999999999996}]},{"osd":5,"last update":"Mon May 22 16:05:58 2023","interfaces":[{"interface":"back","average":{"1min":1.0309999999999999,"5min":1.0349999999999999,"15min":1.048},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.319,"5min":1.605,"15min":1.605},"last":0.99199999999999999},{"interface":"front","average":{"1min":1.0589999999999999,"5min":1.0620000000000001,"15min":1.0760000000000001},"min":{"1min":0.75900000000000001,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":1.4370000000000001,"5min":1.4370000000000001,"15min":1.5389999999999999},"last":1.018}]},{"osd":6,"last update":"Mon May 22 16:05:58 2023","interfaces":[{"interface":"back","average":{"1min":1.03,"5min":1.0469999999999999,"15min":1.0640000000000001},"min":{"1min":0.56899999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"max":{"1min":1.4770000000000001,"5min":1.4770000000000001,"15min":1.5249999999999999},"last":1.042},{"interface":"front","average":{"1min":1.0589999999999999,"5min":1.099,"15min":1.1100000000000001},"min":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"max":{"1min":1.387,"5min":1.4359999999999999,"15min":1.6479999999999999},"last":1.0660000000000001}]},{"osd":7,"last update":"Mon May 22 16:05:58 2023","interfaces":[{"interface":"back","average":{"1min":1.0509999999999999,"5min":1.123,"15min":1.153},"min":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":1.484,"5min":1.7030000000000001,"15min":1.7030000000000001},"last":1.1259999999999999},{"interface":"front","average":{"1min":1.1339999999999999,"5min":1.131,"15min":1.151},"min":{"1min":0.78400000000000003,"5min":0.73599999999999999,"15min":0.73599999999999999},"max":{"1min":1.4810000000000001,"5min":1.6439999999999999,"15min":1.6439999999999999},"last":1.0940000000000001}]}]},{"osd":0,"up_from":13,"seq":55834575316,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37036,"kb_used_data":3044,"kb_used_omap":0,"kb_used_meta":33984,"kb_avail":94334804,"statfs":{"total":96636764160,"available":96598839296,"internally_reserved":0,"allocated":3117056,"data_stored":1940774,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":34799616},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon May 22 16:06:01 2023","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.72399999999999998,"15min":0.71099999999999997},"min":{"1min":0.42299999999999999,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":1.1140000000000001,"5min":1.296,"15min":1.296},"last":1.075},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.72999999999999998,"15min":0.73899999999999999},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.86899999999999999,"5min":1.181,"15min":1.3149999999999999},"last":0.79000000000000004}]},{"osd":2,"last update":"Mon May 22 16:06:01 2023","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.751,"15min":0.76900000000000002},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":1.071,"5min":1.22,"15min":1.284},"last":0.71099999999999997},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.78700000000000003,"15min":0.81299999999999994},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":1.042,"5min":1.323,"15min":1.323},"last":0.95299999999999996}]},{"osd":3,"last update":"Mon May 22 16:06:01 2023","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.746,"15min":0.77100000000000002},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":1.228,"5min":1.2350000000000001,"15min":1.2350000000000001},"last":1.022},{"interface":"front","average":{"1min":0.78300000000000003,"5min":0.85899999999999999,"15min":0.874},"min":{"1min":0.43099999999999999,"5min":0.377,"15min":0.377},"max":{"1min":1.22,"5min":1.2949999999999999,"15min":1.331},"last":0.85499999999999998}]},{"osd":4,"last update":"Mon May 22 16:06:01 2023","interfaces":[{"interface":"back","average":{"1min":0.90200000000000002,"5min":0.94599999999999995,"15min":0.95199999999999996},"min":{"1min":0.64100000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"max":{"1min":1.1819999999999999,"5min":1.264,"15min":1.4059999999999999},"last":1.25},{"interface":"front","average":{"1min":0.91500000000000004,"5min":1.042,"15min":1.05},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.1619999999999999,"5min":1.401,"15min":1.482},"last":1.2849999999999999}]},{"osd":5,"last update":"Mon May 22 16:06:01 2023","interfaces":[{"interface":"back","average":{"1min":0.92600000000000005,"5min":1.014,"15min":1.0349999999999999},"min":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":1.1830000000000001,"5min":1.325,"15min":1.6419999999999999},"last":1.22},{"interface":"front","average":{"1min":0.90200000000000002,"5min":1.0009999999999999,"15min":1.0109999999999999},"min":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"max":{"1min":1.1859999999999999,"5min":1.413,"15min":1.458},"last":1.157}]},{"osd":6,"last update":"Mon May 22 16:06:01 2023","interfaces":[{"interface":"back","average":{"1min":0.98399999999999999,"5min":1.091,"15min":1.1060000000000001},"min":{"1min":0.66400000000000003,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":1.2729999999999999,"5min":1.407,"15min":1.77},"last":1.4099999999999999},{"interface":"front","average":{"1min":0.96399999999999997,"5min":1.093,"15min":1.1040000000000001},"min":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"max":{"1min":1.2589999999999999,"5min":1.5409999999999999,"15min":1.5409999999999999},"last":1.1100000000000001}]},{"osd":7,"last update":"Mon May 22 16:06:01 2023","interfaces":[{"interface":"back","average":{"1min":1.034,"5min":1.1279999999999999,"15min":1.1359999999999999},"min":{"1min":0.69299999999999995,"5min":0.63900000000000001,"15min":0.63900000000000001},"max":{"1min":1.341,"5min":1.6020000000000001,"15min":1.6020000000000001},"last":1.3660000000000001},{"interface":"front","average":{"1min":1.0109999999999999,"5min":1.0780000000000001,"15min":1.095},"min":{"1min":0.68999999999999995,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.2989999999999999,"5min":1.4650000000000001,"15min":1.4990000000000001},"last":1.3280000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-05-22T16:06:11.545 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2023-05-22T15:26:38.020076+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=22, tm_hour=15, tm_min=26, tm_sec=38, tm_wday=0, tm_yday=142, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=22, tm_hour=16, tm_min=6, tm_sec=6, tm_wday=0, tm_yday=142, tm_isdst=0) 2023-05-22T16:06:11.545 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-05-22T15:26:43.943718+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=22, tm_hour=15, tm_min=26, tm_sec=43, tm_wday=0, tm_yday=142, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=22, tm_hour=16, tm_min=6, tm_sec=6, tm_wday=0, tm_yday=142, tm_isdst=0) 2023-05-22T16:06:11.546 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-05-22T15:26:43.943718+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=22, tm_hour=15, tm_min=26, tm_sec=43, tm_wday=0, tm_yday=142, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=22, tm_hour=16, tm_min=6, tm_sec=6, tm_wday=0, tm_yday=142, tm_isdst=0) 2023-05-22T16:06:11.546 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2023-05-22T15:26:43.943718+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=22, tm_hour=15, tm_min=26, tm_sec=43, tm_wday=0, tm_yday=142, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=22, tm_hour=16, tm_min=6, tm_sec=6, tm_wday=0, tm_yday=142, tm_isdst=0) 2023-05-22T16:06:11.546 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2023-05-22T15:26:43.943718+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=22, tm_hour=15, tm_min=26, tm_sec=43, tm_wday=0, tm_yday=142, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=22, tm_hour=16, tm_min=6, tm_sec=6, tm_wday=0, tm_yday=142, tm_isdst=0) 2023-05-22T16:06:11.546 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-05-22T15:26:43.943718+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=22, tm_hour=15, tm_min=26, tm_sec=43, tm_wday=0, tm_yday=142, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=22, tm_hour=16, tm_min=6, tm_sec=6, tm_wday=0, tm_yday=142, tm_isdst=0) 2023-05-22T16:06:11.547 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-05-22T15:26:43.943718+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=22, tm_hour=15, tm_min=26, tm_sec=43, tm_wday=0, tm_yday=142, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=22, tm_hour=16, tm_min=6, tm_sec=6, tm_wday=0, tm_yday=142, tm_isdst=0) 2023-05-22T16:06:11.547 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-05-22T15:26:43.943718+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=22, tm_hour=15, tm_min=26, tm_sec=43, tm_wday=0, tm_yday=142, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=22, tm_hour=16, tm_min=6, tm_sec=6, tm_wday=0, tm_yday=142, tm_isdst=0) 2023-05-22T16:06:11.547 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-05-22T15:26:43.943718+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=22, tm_hour=15, tm_min=26, tm_sec=43, tm_wday=0, tm_yday=142, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=22, tm_hour=16, tm_min=6, tm_sec=6, tm_wday=0, tm_yday=142, tm_isdst=0) 2023-05-22T16:06:11.547 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-05-22T16:06:24.889 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-05-22T16:06:24.892 DEBUG:teuthology.orchestra.run.smithi062:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-05-22T16:06:31.548 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-05-22T16:06:31.680 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:31.675+0000 7fb3e26be640 1 Processor -- start 2023-05-22T16:06:31.680 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:31.675+0000 7fb3e26be640 1 -- start start 2023-05-22T16:06:31.681 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:31.679+0000 7fb3e26be640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb3dc15bea0 0x7fb3dc06c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:31.681 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:31.679+0000 7fb3e26be640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3dc06f6d0 con 0x7fb3dc0cd970 2023-05-22T16:06:31.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:31.679+0000 7fb3e26be640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3dc0cddc0 con 0x7fb3dc0b0260 2023-05-22T16:06:31.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:31.679+0000 7fb3e26be640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3dc0cbca0 con 0x7fb3dc15bea0 2023-05-22T16:06:31.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:31.679+0000 7fb3db7fe640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fb3dc0cd970 0x7fb3dc0afb50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:40662/0 (socket says 172.21.15.16:40662) 2023-05-22T16:06:31.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:31.679+0000 7fb3db7fe640 1 -- 172.21.15.16:0/2873702425 learned_addr learned my addr 172.21.15.16:0/2873702425 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T16:06:31.684 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:31.679+0000 7fb3daffd640 1 -- 172.21.15.16:0/2873702425 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3507687699 0 0) 0x7fb3dc06f6d0 con 0x7fb3dc0cd970 2023-05-22T16:06:31.685 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:31.679+0000 7fb3daffd640 1 -- 172.21.15.16:0/2873702425 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb3c8003530 con 0x7fb3dc0cd970 2023-05-22T16:06:31.685 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:31.679+0000 7fb3dbfff640 1 --2- 172.21.15.16:0/2873702425 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb3dc15bea0 0x7fb3dc06c1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:31.685 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:31.679+0000 7fb3daffd640 1 -- 172.21.15.16:0/2873702425 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2529983906 0 0) 0x7fb3c8003530 con 0x7fb3dc0cd970 2023-05-22T16:06:31.685 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:31.679+0000 7fb3daffd640 1 -- 172.21.15.16:0/2873702425 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb3dc15bea0 msgr2=0x7fb3dc06c1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:06:31.686 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:31.679+0000 7fb3daffd640 1 --2- 172.21.15.16:0/2873702425 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb3dc15bea0 0x7fb3dc06c1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:31.686 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:31.679+0000 7fb3daffd640 1 -- 172.21.15.16:0/2873702425 >> v1:172.21.15.62:6789/0 conn(0x7fb3dc0b0260 legacy=0x7fb3dc06fe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:06:31.686 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:31.679+0000 7fb3daffd640 1 -- 172.21.15.16:0/2873702425 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3dc0cbf80 con 0x7fb3dc0cd970 2023-05-22T16:06:31.686 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:31.679+0000 7fb3daffd640 1 -- 172.21.15.16:0/2873702425 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3138467121 0 0) 0x7fb3cc001a00 con 0x7fb3dc0cd970 2023-05-22T16:06:31.686 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:31.679+0000 7fb3daffd640 1 -- 172.21.15.16:0/2873702425 >> v1:172.21.15.16:6789/0 conn(0x7fb3dc0cd970 legacy=0x7fb3dc0afb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:06:31.686 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:31.679+0000 7fb3daffd640 1 --2- 172.21.15.16:0/2873702425 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb3c80039b0 0x7fb3c8003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:31.686 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:31.679+0000 7fb3daffd640 1 --2- 172.21.15.16:0/2873702425 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb3c80047d0 0x7fb3c8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:31.687 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:31.679+0000 7fb3daffd640 1 --2- 172.21.15.16:0/2873702425 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb3dc15bea0 0x7fb3c80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:31.687 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:31.679+0000 7fb3daffd640 1 -- 172.21.15.16:0/2873702425 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3dc0cbca0 con 0x7fb3c80047d0 2023-05-22T16:06:31.687 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:31.679+0000 7fb3daffd640 1 -- 172.21.15.16:0/2873702425 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3c800bac0 con 0x7fb3c80039b0 2023-05-22T16:06:31.687 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:31.679+0000 7fb3daffd640 1 -- 172.21.15.16:0/2873702425 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3c800bc80 con 0x7fb3dc15bea0 2023-05-22T16:06:31.687 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:31.679+0000 7fb3db7fe640 1 --2- 172.21.15.16:0/2873702425 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb3c80047d0 0x7fb3c8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:31.687 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:31.679+0000 7fb3dbfff640 1 --2- 172.21.15.16:0/2873702425 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb3c80039b0 0x7fb3c8003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:31.688 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:31.679+0000 7fb3db7fe640 1 -- 172.21.15.16:0/2873702425 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb3dc15bea0 msgr2=0x7fb3c80051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:06:31.688 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:31.679+0000 7fb3db7fe640 1 --2- 172.21.15.16:0/2873702425 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb3dc15bea0 0x7fb3c80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:31.688 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:31.679+0000 7fb3db7fe640 1 -- 172.21.15.16:0/2873702425 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb3c80039b0 msgr2=0x7fb3c8003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:06:31.688 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:31.679+0000 7fb3db7fe640 1 --2- 172.21.15.16:0/2873702425 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb3c80039b0 0x7fb3c8003dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:31.688 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:31.679+0000 7fb3db7fe640 1 -- 172.21.15.16:0/2873702425 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3c8003530 con 0x7fb3c80047d0 2023-05-22T16:06:31.689 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:31.683+0000 7fb3db7fe640 1 --2- 172.21.15.16:0/2873702425 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb3c80047d0 0x7fb3c8004c40 secure :-1 s=READY pgs=2670 cs=0 l=1 rev1=1 crypto rx=0x7fb3cc0013e0 tx=0x7fb3cc01f9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28135c617ddf4517 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:06:31.691 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:31.683+0000 7fb3daffd640 1 -- 172.21.15.16:0/2873702425 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3cc0ac650 con 0x7fb3c80047d0 2023-05-22T16:06:31.691 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:31.683+0000 7fb3daffd640 1 -- 172.21.15.16:0/2873702425 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3cc0ac7b0 con 0x7fb3c80047d0 2023-05-22T16:06:31.691 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:31.683+0000 7fb3daffd640 1 -- 172.21.15.16:0/2873702425 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3cc0acba0 con 0x7fb3c80047d0 2023-05-22T16:06:31.692 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:31.683+0000 7fb3e26be640 1 -- 172.21.15.16:0/2873702425 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb3c80047d0 msgr2=0x7fb3c8004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:06:31.692 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:31.683+0000 7fb3e26be640 1 --2- 172.21.15.16:0/2873702425 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb3c80047d0 0x7fb3c8004c40 secure :-1 s=READY pgs=2670 cs=0 l=1 rev1=1 crypto rx=0x7fb3cc0013e0 tx=0x7fb3cc01f9a0 comp rx=0 tx=0).stop 2023-05-22T16:06:31.692 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:31.683+0000 7fb3e0c5b640 1 -- 172.21.15.16:0/2873702425 reap_dead start 2023-05-22T16:06:31.692 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:31.683+0000 7fb3e26be640 1 -- 172.21.15.16:0/2873702425 shutdown_connections 2023-05-22T16:06:31.692 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:31.683+0000 7fb3e26be640 1 -- 172.21.15.16:0/2873702425 >> 172.21.15.16:0/2873702425 conn(0x7fb3dc0cab50 msgr2=0x7fb3dc0caf50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:06:31.693 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:31.683+0000 7fb3e26be640 1 -- 172.21.15.16:0/2873702425 shutdown_connections 2023-05-22T16:06:31.693 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:31.683+0000 7fb3e26be640 1 -- 172.21.15.16:0/2873702425 wait complete. 2023-05-22T16:06:31.693 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:31.683+0000 7fb3e26be640 1 Processor -- start 2023-05-22T16:06:31.693 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:31.683+0000 7fb3e26be640 1 -- start start 2023-05-22T16:06:31.693 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:31.683+0000 7fb3e26be640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb3dc147780 0x7fb3dc151f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:31.693 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:31.683+0000 7fb3e26be640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb3dc1527a0 0x7fb3dc15d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:31.693 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:31.683+0000 7fb3e26be640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb3dc15dbc0 0x7fb3dc153cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:31.694 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:31.683+0000 7fb3e26be640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3dc15c870 con 0x7fb3dc15dbc0 2023-05-22T16:06:31.694 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:31.683+0000 7fb3e26be640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3dc147b50 con 0x7fb3dc1527a0 2023-05-22T16:06:31.694 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:31.683+0000 7fb3e26be640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3dc153820 con 0x7fb3dc147780 2023-05-22T16:06:31.694 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:31.683+0000 7fb3e0c5b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb3dc15dbc0 0x7fb3dc153cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:31.694 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:31.683+0000 7fb3e0c5b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb3dc15dbc0 0x7fb3dc153cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51440/0 (socket says 172.21.15.16:51440) 2023-05-22T16:06:31.694 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:31.683+0000 7fb3e0c5b640 1 -- 172.21.15.16:0/3104265046 learned_addr learned my addr 172.21.15.16:0/3104265046 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:06:31.694 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:31.683+0000 7fb3db7fe640 1 --2- 172.21.15.16:0/3104265046 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb3dc1527a0 0x7fb3dc15d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:31.694 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:31.683+0000 7fb3dbfff640 1 --2- 172.21.15.16:0/3104265046 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb3dc147780 0x7fb3dc151f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:31.695 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:31.683+0000 7fb3e0c5b640 1 -- 172.21.15.16:0/3104265046 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb3dc147780 msgr2=0x7fb3dc151f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:06:31.695 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:31.683+0000 7fb3e0c5b640 1 --2- 172.21.15.16:0/3104265046 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb3dc147780 0x7fb3dc151f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:31.695 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:31.683+0000 7fb3e0c5b640 1 -- 172.21.15.16:0/3104265046 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb3dc1527a0 msgr2=0x7fb3dc15d680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:06:31.695 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:31.683+0000 7fb3e0c5b640 1 --2- 172.21.15.16:0/3104265046 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb3dc1527a0 0x7fb3dc15d680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:31.695 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:31.683+0000 7fb3e0c5b640 1 -- 172.21.15.16:0/3104265046 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3dc0cc260 con 0x7fb3dc15dbc0 2023-05-22T16:06:31.695 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:31.687+0000 7fb3e0c5b640 1 --2- 172.21.15.16:0/3104265046 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb3dc15dbc0 0x7fb3dc153cb0 secure :-1 s=READY pgs=2671 cs=0 l=1 rev1=1 crypto rx=0x7fb3dc06db20 tx=0x7fb3d400c390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:06:31.695 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:31.687+0000 7fb3d8ff9640 1 -- 172.21.15.16:0/3104265046 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3d40074e0 con 0x7fb3dc15dbc0 2023-05-22T16:06:31.696 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:31.687+0000 7fb3d8ff9640 1 -- 172.21.15.16:0/3104265046 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3d4007680 con 0x7fb3dc15dbc0 2023-05-22T16:06:31.696 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:31.687+0000 7fb3e26be640 1 -- 172.21.15.16:0/3104265046 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3dc06f970 con 0x7fb3dc15dbc0 2023-05-22T16:06:31.696 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:31.687+0000 7fb3d8ff9640 1 -- 172.21.15.16:0/3104265046 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3d4004240 con 0x7fb3dc15dbc0 2023-05-22T16:06:31.696 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:31.687+0000 7fb3e26be640 1 -- 172.21.15.16:0/3104265046 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3dc0cc3f0 con 0x7fb3dc15dbc0 2023-05-22T16:06:31.696 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:31.687+0000 7fb3e26be640 1 -- 172.21.15.16:0/3104265046 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3dc1539b0 con 0x7fb3dc15dbc0 2023-05-22T16:06:31.697 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:31.687+0000 7fb3d8ff9640 1 -- 172.21.15.16:0/3104265046 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 100) v1 ==== 84332+0+0 (secure 0 0 0) 0x7fb3d40043a0 con 0x7fb3dc15dbc0 2023-05-22T16:06:31.697 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:31.687+0000 7fb3d8ff9640 1 --2- 172.21.15.16:0/3104265046 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fb3bc0654a0 0x7fb3bc067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:31.697 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:31.687+0000 7fb3d8ff9640 1 -- 172.21.15.16:0/3104265046 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(201..201 src has 1..201) v4 ==== 6317+0+0 (secure 0 0 0) 0x7fb3d4081420 con 0x7fb3dc15dbc0 2023-05-22T16:06:31.697 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:31.687+0000 7fb3dbfff640 1 --2- 172.21.15.16:0/3104265046 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fb3bc0654a0 0x7fb3bc067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:31.697 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:31.687+0000 7fb3dbfff640 1 --2- 172.21.15.16:0/3104265046 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fb3bc0654a0 0x7fb3bc067950 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fb3c400c820 tx=0x7fb3c4008040 comp rx=0 tx=0).ready entity=mgr.8195 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:06:31.698 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:31.691+0000 7fb3d8ff9640 1 -- 172.21.15.16:0/3104265046 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fb3d4049620 con 0x7fb3dc15dbc0 2023-05-22T16:06:31.846 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:31.843+0000 7fb3e26be640 1 -- 172.21.15.16:0/3104265046 --> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb3dc15b9f0 con 0x7fb3bc0654a0 2023-05-22T16:06:31.849 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:31.847+0000 7fb3d8ff9640 1 -- 172.21.15.16:0/3104265046 <== mgr.8195 v2:172.21.15.16:6832/65890 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+71140 (secure 0 0 0) 0x7fb3dc15b9f0 con 0x7fb3bc0654a0 2023-05-22T16:06:31.850 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T16:06:31.850 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2023-05-22T16:06:31.851 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:31.847+0000 7fb3e26be640 1 -- 172.21.15.16:0/3104265046 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fb3bc0654a0 msgr2=0x7fb3bc067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:06:31.851 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:31.847+0000 7fb3e26be640 1 --2- 172.21.15.16:0/3104265046 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fb3bc0654a0 0x7fb3bc067950 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fb3c400c820 tx=0x7fb3c4008040 comp rx=0 tx=0).stop 2023-05-22T16:06:31.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:31.847+0000 7fb3e26be640 1 -- 172.21.15.16:0/3104265046 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb3dc15dbc0 msgr2=0x7fb3dc153cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:06:31.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:31.847+0000 7fb3e26be640 1 --2- 172.21.15.16:0/3104265046 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb3dc15dbc0 0x7fb3dc153cb0 secure :-1 s=READY pgs=2671 cs=0 l=1 rev1=1 crypto rx=0x7fb3dc06db20 tx=0x7fb3d400c390 comp rx=0 tx=0).stop 2023-05-22T16:06:31.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:31.847+0000 7fb3e26be640 1 -- 172.21.15.16:0/3104265046 shutdown_connections 2023-05-22T16:06:31.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:31.847+0000 7fb3e26be640 1 --2- 172.21.15.16:0/3104265046 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fb3bc0654a0 0x7fb3bc067950 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:31.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:31.847+0000 7fb3e26be640 1 --2- 172.21.15.16:0/3104265046 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb3dc15dbc0 0x7fb3dc153cb0 unknown :-1 s=CLOSED pgs=2671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:31.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:31.847+0000 7fb3e26be640 1 --2- 172.21.15.16:0/3104265046 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb3dc1527a0 0x7fb3dc15d680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:31.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:31.847+0000 7fb3e26be640 1 --2- 172.21.15.16:0/3104265046 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb3dc147780 0x7fb3dc151f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:31.853 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:31.847+0000 7fb3e26be640 1 -- 172.21.15.16:0/3104265046 >> 172.21.15.16:0/3104265046 conn(0x7fb3dc0cab50 msgr2=0x7fb3dc06b7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:06:31.853 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:31.847+0000 7fb3e26be640 1 -- 172.21.15.16:0/3104265046 shutdown_connections 2023-05-22T16:06:31.853 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:31.847+0000 7fb3e26be640 1 -- 172.21.15.16:0/3104265046 wait complete. 2023-05-22T16:06:31.867 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2023-05-22T16:06:31.809366+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2132,"num_read_kb":4426,"num_write":579,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":349,"ondisk_log_size":349,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":7,"kb":754974720,"kb_used":258760,"kb_used_data":20872,"kb_used_omap":0,"kb_used_meta":237824,"kb_avail":754715960,"statfs":{"total":773094113280,"available":772829143040,"internally_reserved":0,"allocated":21372928,"data_stored":11983984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":243531776},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003383"},"pg_stats":[{"pgid":"1.0","version":"201'349","reported_seq":1175,"reported_epoch":201,"state":"active+clean","last_fresh":"2023-05-22T16:06:14.813007+0000","last_change":"2023-05-22T16:06:14.813307+0000","last_active":"2023-05-22T16:06:14.813007+0000","last_peered":"2023-05-22T16:06:14.813007+0000","last_clean":"2023-05-22T16:06:14.813007+0000","last_became_active":"2023-05-22T15:26:39.315370+0000","last_became_peered":"2023-05-22T15:26:39.315370+0000","last_unstale":"2023-05-22T16:06:14.813007+0000","last_undegraded":"2023-05-22T16:06:14.813007+0000","last_fullsized":"2023-05-22T16:06:14.813007+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"201'349","last_scrub_stamp":"2023-05-22T16:06:14.812951+0000","last_deep_scrub":"201'349","last_deep_scrub_stamp":"2023-05-22T16:06:14.812951+0000","last_clean_scrub_stamp":"2023-05-22T16:06:14.812951+0000","objects_scrubbed":2,"log_size":349,"log_dups_size":0,"ondisk_log_size":349,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T16:08:32.345880+0000","scrub_duration":2.9823071529999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2132,"num_read_kb":4426,"num_write":579,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":215,"reported_epoch":201,"state":"active+clean","last_fresh":"2023-05-22T16:06:27.300799+0000","last_change":"2023-05-22T16:06:27.301083+0000","last_active":"2023-05-22T16:06:27.300799+0000","last_peered":"2023-05-22T16:06:27.300799+0000","last_clean":"2023-05-22T16:06:27.300799+0000","last_became_active":"2023-05-22T15:26:44.970872+0000","last_became_peered":"2023-05-22T15:26:44.970872+0000","last_unstale":"2023-05-22T16:06:27.300799+0000","last_undegraded":"2023-05-22T16:06:27.300799+0000","last_fullsized":"2023-05-22T16:06:27.300799+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:06:27.300747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:06:27.300747+0000","last_clean_scrub_stamp":"2023-05-22T16:06:27.300747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T16:54:34.454270+0000","scrub_duration":2.2360322450000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":203,"reported_epoch":201,"state":"active+clean","last_fresh":"2023-05-22T16:06:17.795279+0000","last_change":"2023-05-22T16:06:17.795555+0000","last_active":"2023-05-22T16:06:17.795279+0000","last_peered":"2023-05-22T16:06:17.795279+0000","last_clean":"2023-05-22T16:06:17.795279+0000","last_became_active":"2023-05-22T15:26:44.976445+0000","last_became_peered":"2023-05-22T15:26:44.976445+0000","last_unstale":"2023-05-22T16:06:17.795279+0000","last_undegraded":"2023-05-22T16:06:17.795279+0000","last_fullsized":"2023-05-22T16:06:17.795279+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:06:17.795228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:06:17.795228+0000","last_clean_scrub_stamp":"2023-05-22T16:06:17.795228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:12:50.953032+0000","scrub_duration":2.2366842309999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":207,"reported_epoch":201,"state":"active+clean","last_fresh":"2023-05-22T16:06:16.858681+0000","last_change":"2023-05-22T16:06:16.858999+0000","last_active":"2023-05-22T16:06:16.858681+0000","last_peered":"2023-05-22T16:06:16.858681+0000","last_clean":"2023-05-22T16:06:16.858681+0000","last_became_active":"2023-05-22T15:26:44.969083+0000","last_became_peered":"2023-05-22T15:26:44.969083+0000","last_unstale":"2023-05-22T16:06:16.858681+0000","last_undegraded":"2023-05-22T16:06:16.858681+0000","last_fullsized":"2023-05-22T16:06:16.858681+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:06:16.858632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:06:16.858632+0000","last_clean_scrub_stamp":"2023-05-22T16:06:16.858632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:33:23.760170+0000","scrub_duration":2.2363037530000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":214,"reported_epoch":201,"state":"active+clean","last_fresh":"2023-05-22T16:06:13.411231+0000","last_change":"2023-05-22T16:06:13.411376+0000","last_active":"2023-05-22T16:06:13.411231+0000","last_peered":"2023-05-22T16:06:13.411231+0000","last_clean":"2023-05-22T16:06:13.411231+0000","last_became_active":"2023-05-22T15:26:44.975546+0000","last_became_peered":"2023-05-22T15:26:44.975546+0000","last_unstale":"2023-05-22T16:06:13.411231+0000","last_undegraded":"2023-05-22T16:06:13.411231+0000","last_fullsized":"2023-05-22T16:06:13.411231+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:06:13.411211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:06:13.411211+0000","last_clean_scrub_stamp":"2023-05-22T16:06:13.411211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T03:50:54.501993+0000","scrub_duration":2.2360626940000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":191,"reported_epoch":201,"state":"active+clean","last_fresh":"2023-05-22T16:06:29.286583+0000","last_change":"2023-05-22T16:06:29.286873+0000","last_active":"2023-05-22T16:06:29.286583+0000","last_peered":"2023-05-22T16:06:29.286583+0000","last_clean":"2023-05-22T16:06:29.286583+0000","last_became_active":"2023-05-22T15:26:45.091143+0000","last_became_peered":"2023-05-22T15:26:45.091143+0000","last_unstale":"2023-05-22T16:06:29.286583+0000","last_undegraded":"2023-05-22T16:06:29.286583+0000","last_fullsized":"2023-05-22T16:06:29.286583+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:06:29.286530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:06:29.286530+0000","last_clean_scrub_stamp":"2023-05-22T16:06:29.286530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:57:47.242362+0000","scrub_duration":2.2341401730000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":201,"reported_epoch":201,"state":"active+clean","last_fresh":"2023-05-22T16:06:21.078431+0000","last_change":"2023-05-22T16:06:21.078724+0000","last_active":"2023-05-22T16:06:21.078431+0000","last_peered":"2023-05-22T16:06:21.078431+0000","last_clean":"2023-05-22T16:06:21.078431+0000","last_became_active":"2023-05-22T15:26:45.091951+0000","last_became_peered":"2023-05-22T15:26:45.091951+0000","last_unstale":"2023-05-22T16:06:21.078431+0000","last_undegraded":"2023-05-22T16:06:21.078431+0000","last_fullsized":"2023-05-22T16:06:21.078431+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:06:21.078378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:06:21.078378+0000","last_clean_scrub_stamp":"2023-05-22T16:06:21.078378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:28:52.334842+0000","scrub_duration":2.2366510910000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":180,"reported_epoch":201,"state":"active+clean","last_fresh":"2023-05-22T16:06:03.193505+0000","last_change":"2023-05-22T15:26:44.978858+0000","last_active":"2023-05-22T16:06:03.193505+0000","last_peered":"2023-05-22T16:06:03.193505+0000","last_clean":"2023-05-22T16:06:03.193505+0000","last_became_active":"2023-05-22T15:26:44.978443+0000","last_became_peered":"2023-05-22T15:26:44.978443+0000","last_unstale":"2023-05-22T16:06:03.193505+0000","last_undegraded":"2023-05-22T16:06:03.193505+0000","last_fullsized":"2023-05-22T16:06:03.193505+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T15:26:43.943718+0000","last_clean_scrub_stamp":"2023-05-22T15:26:43.943718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":191,"reported_epoch":201,"state":"active+clean","last_fresh":"2023-05-22T16:06:25.091309+0000","last_change":"2023-05-22T16:06:25.091606+0000","last_active":"2023-05-22T16:06:25.091309+0000","last_peered":"2023-05-22T16:06:25.091309+0000","last_clean":"2023-05-22T16:06:25.091309+0000","last_became_active":"2023-05-22T15:26:44.981586+0000","last_became_peered":"2023-05-22T15:26:44.981586+0000","last_unstale":"2023-05-22T16:06:25.091309+0000","last_undegraded":"2023-05-22T16:06:25.091309+0000","last_fullsized":"2023-05-22T16:06:25.091309+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:06:25.091260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:06:25.091260+0000","last_clean_scrub_stamp":"2023-05-22T16:06:25.091260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:57:43.047092+0000","scrub_duration":2.235795135,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2132,"num_read_kb":4426,"num_write":579,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":349,"ondisk_log_size":349,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575320,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29676,"kb_used_data":3044,"kb_used_omap":0,"kb_used_meta":26624,"kb_avail":94342164,"statfs":{"total":96636764160,"available":96606375936,"internally_reserved":0,"allocated":3117056,"data_stored":1940774,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27262976},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 16:05:53 2023","interfaces":[{"interface":"back","average":{"1min":0.81999999999999995,"5min":0.85099999999999998,"15min":0.86099999999999999},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":1.222,"5min":1.3069999999999999,"15min":1.3069999999999999},"last":1.177},{"interface":"front","average":{"1min":0.84999999999999998,"5min":0.88200000000000001,"15min":0.877},"min":{"1min":0.61299999999999999,"5min":0.501,"15min":0.501},"max":{"1min":1.296,"5min":1.296,"15min":1.296},"last":0.68000000000000005}]},{"osd":1,"last update":"Mon May 22 16:05:53 2023","interfaces":[{"interface":"back","average":{"1min":0.90400000000000003,"5min":0.97299999999999998,"15min":0.98999999999999999},"min":{"1min":0.66000000000000003,"5min":0.63400000000000001,"15min":0.58299999999999996},"max":{"1min":1.274,"5min":1.431,"15min":1.431},"last":0.86799999999999999},{"interface":"front","average":{"1min":0.95499999999999996,"5min":1.026,"15min":1.026},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":1.2909999999999999,"5min":1.345,"15min":1.345},"last":0.97899999999999998}]},{"osd":2,"last update":"Mon May 22 16:05:53 2023","interfaces":[{"interface":"back","average":{"1min":0.83899999999999997,"5min":0.89900000000000002,"15min":0.90400000000000003},"min":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.171,"5min":1.286,"15min":1.407},"last":0.76800000000000002},{"interface":"front","average":{"1min":0.89100000000000001,"5min":0.94899999999999995,"15min":0.95899999999999996},"min":{"1min":0.53200000000000003,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":1.3899999999999999,"5min":1.3899999999999999,"15min":1.3899999999999999},"last":0.90200000000000002}]},{"osd":3,"last update":"Mon May 22 16:05:53 2023","interfaces":[{"interface":"back","average":{"1min":0.96399999999999997,"5min":0.93300000000000005,"15min":0.94799999999999995},"min":{"1min":0.755,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.2,"5min":1.2909999999999999,"15min":1.401},"last":1.1080000000000001},{"interface":"front","average":{"1min":0.97199999999999998,"5min":0.98699999999999999,"15min":0.98099999999999998},"min":{"1min":0.60299999999999998,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":1.4179999999999999,"5min":1.4179999999999999,"15min":1.4179999999999999},"last":0.94499999999999995}]},{"osd":4,"last update":"Mon May 22 16:05:53 2023","interfaces":[{"interface":"back","average":{"1min":0.875,"5min":0.88900000000000001,"15min":0.88600000000000001},"min":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51000000000000001},"max":{"1min":1.2310000000000001,"5min":1.3440000000000001,"15min":1.3440000000000001},"last":0.80900000000000005},{"interface":"front","average":{"1min":0.95599999999999996,"5min":0.99099999999999999,"15min":0.96499999999999997},"min":{"1min":0.60899999999999999,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":1.3480000000000001,"5min":1.405,"15min":1.405},"last":1.22}]},{"osd":5,"last update":"Mon May 22 16:05:53 2023","interfaces":[{"interface":"back","average":{"1min":0.96499999999999997,"5min":0.98899999999999999,"15min":0.98999999999999999},"min":{"1min":0.66200000000000003,"5min":0.623,"15min":0.441},"max":{"1min":1.159,"5min":1.2909999999999999,"15min":1.3759999999999999},"last":1.0629999999999999},{"interface":"front","average":{"1min":1.1180000000000001,"5min":1.1339999999999999,"15min":1.135},"min":{"1min":0.91000000000000003,"5min":0.72299999999999998,"15min":0.66600000000000004},"max":{"1min":1.468,"5min":1.468,"15min":1.468},"last":1.0900000000000001}]},{"osd":6,"last update":"Mon May 22 16:05:53 2023","interfaces":[{"interface":"back","average":{"1min":1.048,"5min":1.0600000000000001,"15min":1.0569999999999999},"min":{"1min":0.72799999999999998,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":1.288,"5min":1.3169999999999999,"15min":1.389},"last":1.0269999999999999},{"interface":"front","average":{"1min":1.008,"5min":1.0940000000000001,"15min":1.0880000000000001},"min":{"1min":0.70399999999999996,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":1.3149999999999999,"5min":1.391,"15min":1.7050000000000001},"last":1.268}]}]},{"osd":6,"up_from":13,"seq":55834575321,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":22824,"kb_used_data":2464,"kb_used_omap":0,"kb_used_meta":20352,"kb_avail":94349016,"statfs":{"total":96636764160,"available":96613392384,"internally_reserved":0,"allocated":2523136,"data_stored":1350406,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":20840448},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 16:05:56 2023","interfaces":[{"interface":"back","average":{"1min":0.85999999999999999,"5min":0.88,"15min":0.89600000000000002},"min":{"1min":0.48599999999999999,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":2.6469999999999998,"5min":2.6469999999999998,"15min":2.6469999999999998},"last":0.96299999999999997},{"interface":"front","average":{"1min":1.254,"5min":0.96899999999999997,"15min":0.92800000000000005},"min":{"1min":0.64600000000000002,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":6.7439999999999998,"5min":6.7439999999999998,"15min":6.7439999999999998},"last":0.72999999999999998}]},{"osd":1,"last update":"Mon May 22 16:05:56 2023","interfaces":[{"interface":"back","average":{"1min":1.0009999999999999,"5min":0.93000000000000005,"15min":0.94199999999999995},"min":{"1min":0.57399999999999995,"5min":0.55300000000000005,"15min":0.51100000000000001},"max":{"1min":2.3290000000000002,"5min":2.3290000000000002,"15min":2.3290000000000002},"last":1.1240000000000001},{"interface":"front","average":{"1min":1.2330000000000001,"5min":0.97899999999999998,"15min":0.94699999999999995},"min":{"1min":0.58199999999999996,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":6.5010000000000003,"5min":6.5010000000000003,"15min":6.5010000000000003},"last":0.88600000000000001}]},{"osd":2,"last update":"Mon May 22 16:05:56 2023","interfaces":[{"interface":"back","average":{"1min":1.2210000000000001,"5min":1.012,"15min":1.0109999999999999},"min":{"1min":0.57399999999999995,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":7.0170000000000003,"5min":7.0170000000000003,"15min":7.0170000000000003},"last":1.077},{"interface":"front","average":{"1min":1.0589999999999999,"5min":1.05,"15min":1.0720000000000001},"min":{"1min":0.66200000000000003,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":2.403,"5min":2.403,"15min":2.403},"last":1.254}]},{"osd":3,"last update":"Mon May 22 16:05:56 2023","interfaces":[{"interface":"back","average":{"1min":1.278,"5min":1.054,"15min":1.0129999999999999},"min":{"1min":0.63800000000000001,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":6.6699999999999999,"5min":6.6699999999999999,"15min":6.6699999999999999},"last":1.03},{"interface":"front","average":{"1min":1.244,"5min":1.02,"15min":0.98899999999999999},"min":{"1min":0.68000000000000005,"5min":0.622,"15min":0.622},"max":{"1min":7.1260000000000003,"5min":7.1260000000000003,"15min":7.1260000000000003},"last":1.2729999999999999}]},{"osd":4,"last update":"Mon May 22 16:05:56 2023","interfaces":[{"interface":"back","average":{"1min":1.161,"5min":0.98099999999999998,"15min":0.96999999999999997},"min":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"max":{"1min":4.0709999999999997,"5min":4.0709999999999997,"15min":4.0709999999999997},"last":1.397},{"interface":"front","average":{"1min":1.583,"5min":1.1619999999999999,"15min":1.075},"min":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":11.581,"5min":11.581,"15min":11.581},"last":1.1639999999999999}]},{"osd":5,"last update":"Mon May 22 16:05:56 2023","interfaces":[{"interface":"back","average":{"1min":1.925,"5min":1.1739999999999999,"15min":1.056},"min":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":17.587,"5min":17.587,"15min":17.587},"last":1.2090000000000001},{"interface":"front","average":{"1min":1.204,"5min":1.1180000000000001,"15min":1.1240000000000001},"min":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"max":{"1min":2.9620000000000002,"5min":2.9620000000000002,"15min":2.9620000000000002},"last":1.3300000000000001}]},{"osd":7,"last update":"Mon May 22 16:05:56 2023","interfaces":[{"interface":"back","average":{"1min":1.4319999999999999,"5min":1.202,"15min":1.1559999999999999},"min":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.58599999999999997},"max":{"1min":6.8339999999999996,"5min":6.8339999999999996,"15min":6.8339999999999996},"last":1.2350000000000001},{"interface":"front","average":{"1min":1.5429999999999999,"5min":1.2090000000000001,"15min":1.163},"min":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":8.5990000000000002,"5min":8.5990000000000002,"15min":8.5990000000000002},"last":1.3620000000000001}]}]},{"osd":5,"up_from":13,"seq":55834575321,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32936,"kb_used_data":2464,"kb_used_omap":0,"kb_used_meta":30464,"kb_avail":94338904,"statfs":{"total":96636764160,"available":96603037696,"internally_reserved":0,"allocated":2523136,"data_stored":1350406,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31195136},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 16:05:54 2023","interfaces":[{"interface":"back","average":{"1min":0.76000000000000001,"5min":0.85199999999999998,"15min":0.875},"min":{"1min":0.47899999999999998,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.0820000000000001,"5min":1.2749999999999999,"15min":1.3959999999999999},"last":0.91500000000000004},{"interface":"front","average":{"1min":0.86499999999999999,"5min":0.93999999999999995,"15min":0.94299999999999995},"min":{"1min":0.56499999999999995,"5min":0.56200000000000006,"15min":0.502},"max":{"1min":1.165,"5min":1.325,"15min":1.325},"last":1.042}]},{"osd":1,"last update":"Mon May 22 16:05:54 2023","interfaces":[{"interface":"back","average":{"1min":0.86599999999999999,"5min":0.95999999999999996,"15min":1.0009999999999999},"min":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.2909999999999999,"5min":1.365,"15min":1.365},"last":0.88900000000000001},{"interface":"front","average":{"1min":0.89600000000000002,"5min":1.028,"15min":1.0640000000000001},"min":{"1min":0.46899999999999997,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":1.216,"5min":1.3520000000000001,"15min":1.4630000000000001},"last":1.111}]},{"osd":2,"last update":"Mon May 22 16:05:54 2023","interfaces":[{"interface":"back","average":{"1min":0.85299999999999998,"5min":0.90500000000000003,"15min":0.92000000000000004},"min":{"1min":0.60899999999999999,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.1859999999999999,"5min":1.3200000000000001,"15min":1.3740000000000001},"last":0.98299999999999998},{"interface":"front","average":{"1min":0.86799999999999999,"5min":0.94699999999999995,"15min":0.94599999999999995},"min":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":1.117,"5min":1.4350000000000001,"15min":1.4350000000000001},"last":0.95799999999999996}]},{"osd":3,"last update":"Mon May 22 16:05:54 2023","interfaces":[{"interface":"back","average":{"1min":0.91300000000000003,"5min":0.97699999999999998,"15min":0.97599999999999998},"min":{"1min":0.63400000000000001,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":1.1759999999999999,"5min":1.349,"15min":1.367},"last":1.0069999999999999},{"interface":"front","average":{"1min":0.97999999999999998,"5min":1.0329999999999999,"15min":1.048},"min":{"1min":0.67600000000000005,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.1779999999999999,"5min":1.3899999999999999,"15min":1.3899999999999999},"last":0.86899999999999999}]},{"osd":4,"last update":"Mon May 22 16:05:54 2023","interfaces":[{"interface":"back","average":{"1min":0.91200000000000003,"5min":0.93600000000000005,"15min":0.93100000000000005},"min":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.2170000000000001,"5min":1.6000000000000001,"15min":1.6000000000000001},"last":0.69599999999999995},{"interface":"front","average":{"1min":0.98899999999999999,"5min":0.98499999999999999,"15min":1.01},"min":{"1min":0.57899999999999996,"5min":0.498,"15min":0.498},"max":{"1min":1.371,"5min":1.4630000000000001,"15min":1.4630000000000001},"last":1.2010000000000001}]},{"osd":6,"last update":"Mon May 22 16:05:54 2023","interfaces":[{"interface":"back","average":{"1min":1.054,"5min":1.0129999999999999,"15min":1.0209999999999999},"min":{"1min":0.68999999999999995,"5min":0.63200000000000001,"15min":0.59799999999999998},"max":{"1min":1.2509999999999999,"5min":1.4630000000000001,"15min":1.4790000000000001},"last":0.66500000000000004},{"interface":"front","average":{"1min":1.0449999999999999,"5min":1.073,"15min":1.081},"min":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":1.3440000000000001,"5min":1.6080000000000001,"15min":1.6080000000000001},"last":0.82399999999999995}]},{"osd":7,"last update":"Mon May 22 16:05:54 2023","interfaces":[{"interface":"back","average":{"1min":1.133,"5min":1.1180000000000001,"15min":1.1259999999999999},"min":{"1min":0.83899999999999997,"5min":0.57699999999999996,"15min":0.57699999999999996},"max":{"1min":1.3680000000000001,"5min":1.51,"15min":1.5109999999999999},"last":0.76400000000000001},{"interface":"front","average":{"1min":1.1739999999999999,"5min":1.1859999999999999,"15min":1.1930000000000001},"min":{"1min":0.91000000000000003,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":1.353,"5min":1.6679999999999999,"15min":1.7210000000000001},"last":1.085}]}]},{"osd":4,"up_from":13,"seq":55834575319,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":39656,"kb_used_data":2464,"kb_used_omap":0,"kb_used_meta":37184,"kb_avail":94332184,"statfs":{"total":96636764160,"available":96596156416,"internally_reserved":0,"allocated":2523136,"data_stored":1350406,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":38076416},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 16:06:05 2023","interfaces":[{"interface":"back","average":{"1min":0.88200000000000001,"5min":0.89400000000000002,"15min":0.89800000000000002},"min":{"1min":0.58199999999999996,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.1890000000000001,"5min":1.7210000000000001,"15min":1.7210000000000001},"last":0.83099999999999996},{"interface":"front","average":{"1min":0.86799999999999999,"5min":0.93899999999999995,"15min":0.94799999999999995},"min":{"1min":0.49099999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":1.1419999999999999,"5min":1.7290000000000001,"15min":1.7290000000000001},"last":1.2789999999999999}]},{"osd":1,"last update":"Mon May 22 16:06:05 2023","interfaces":[{"interface":"back","average":{"1min":0.91300000000000003,"5min":0.95099999999999996,"15min":0.96199999999999997},"min":{"1min":0.60699999999999998,"5min":0.314,"15min":0.314},"max":{"1min":1.242,"5min":1.9339999999999999,"15min":1.9339999999999999},"last":0.74299999999999999},{"interface":"front","average":{"1min":0.82299999999999995,"5min":0.89600000000000002,"15min":0.93000000000000005},"min":{"1min":0.54000000000000004,"5min":0.504,"15min":0.504},"max":{"1min":1.149,"5min":1.365,"15min":1.365},"last":0.93999999999999995}]},{"osd":2,"last update":"Mon May 22 16:06:05 2023","interfaces":[{"interface":"back","average":{"1min":0.87,"5min":0.94199999999999995,"15min":0.94399999999999995},"min":{"1min":0.53500000000000003,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":1.1759999999999999,"5min":1.4990000000000001,"15min":1.4990000000000001},"last":0.99099999999999999},{"interface":"front","average":{"1min":0.92200000000000004,"5min":1.0129999999999999,"15min":1.0449999999999999},"min":{"1min":0.60799999999999998,"5min":0.44,"15min":0.44},"max":{"1min":1.1579999999999999,"5min":2.0249999999999999,"15min":2.0249999999999999},"last":1.0369999999999999}]},{"osd":3,"last update":"Mon May 22 16:06:05 2023","interfaces":[{"interface":"back","average":{"1min":0.93999999999999995,"5min":0.95799999999999996,"15min":0.96399999999999997},"min":{"1min":0.61699999999999999,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":1.2929999999999999,"5min":1.452,"15min":1.4610000000000001},"last":1.3200000000000001},{"interface":"front","average":{"1min":0.97899999999999998,"5min":1.0329999999999999,"15min":1.044},"min":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":1.3140000000000001,"5min":1.349,"15min":1.484},"last":1.0780000000000001}]},{"osd":5,"last update":"Mon May 22 16:06:05 2023","interfaces":[{"interface":"back","average":{"1min":1.032,"5min":1.0289999999999999,"15min":1.0049999999999999},"min":{"1min":0.63600000000000001,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":1.353,"5min":2.1110000000000002,"15min":2.1110000000000002},"last":0.88800000000000001},{"interface":"front","average":{"1min":1.083,"5min":1.018,"15min":1.002},"min":{"1min":0.77300000000000002,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.413,"5min":1.5449999999999999,"15min":1.5449999999999999},"last":1.232}]},{"osd":6,"last update":"Mon May 22 16:06:05 2023","interfaces":[{"interface":"back","average":{"1min":1.0029999999999999,"5min":1.02,"15min":1.006},"min":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"max":{"1min":1.254,"5min":1.397,"15min":1.4339999999999999},"last":1.1910000000000001},{"interface":"front","average":{"1min":1.111,"5min":1.137,"15min":1.1379999999999999},"min":{"1min":0.74399999999999999,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.266,"5min":2.2690000000000001,"15min":2.2690000000000001},"last":1.153}]},{"osd":7,"last update":"Mon May 22 16:06:05 2023","interfaces":[{"interface":"back","average":{"1min":1.085,"5min":1.0840000000000001,"15min":1.0720000000000001},"min":{"1min":0.78700000000000003,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":1.4810000000000001,"5min":1.623,"15min":1.623},"last":1.1100000000000001},{"interface":"front","average":{"1min":1.117,"5min":1.1659999999999999,"15min":1.159},"min":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.68100000000000005},"max":{"1min":1.389,"5min":1.448,"15min":1.5089999999999999},"last":1.3540000000000001}]}]},{"osd":3,"up_from":13,"seq":55834575319,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27688,"kb_used_data":2464,"kb_used_omap":0,"kb_used_meta":25216,"kb_avail":94344152,"statfs":{"total":96636764160,"available":96608411648,"internally_reserved":0,"allocated":2523136,"data_stored":1350406,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":25821184},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 16:05:38 2023","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"min":{"1min":0.29299999999999998,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.93700000000000006,"5min":1.0049999999999999,"15min":1.0229999999999999},"last":0.80200000000000005},{"interface":"front","average":{"1min":0.70299999999999996,"5min":0.76200000000000001,"15min":0.76900000000000002},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":1.2749999999999999,"5min":1.7230000000000001,"15min":1.7230000000000001},"last":0.63}]},{"osd":1,"last update":"Mon May 22 16:05:38 2023","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.75600000000000001,"15min":0.71999999999999997},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":1.2190000000000001,"5min":2.4969999999999999,"15min":2.4969999999999999},"last":0.72899999999999998},{"interface":"front","average":{"1min":0.76000000000000001,"5min":0.80700000000000005,"15min":0.80100000000000005},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":1.3520000000000001,"5min":2.4769999999999999,"15min":2.4769999999999999},"last":0.85899999999999999}]},{"osd":2,"last update":"Mon May 22 16:05:38 2023","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":0.83699999999999997,"15min":0.84899999999999998},"min":{"1min":0.52300000000000002,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.94799999999999995,"5min":2.2250000000000001,"15min":2.2250000000000001},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.87,"5min":0.83599999999999997,"15min":0.83299999999999996},"min":{"1min":0.55500000000000005,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":1.129,"5min":2.1680000000000001,"15min":2.1680000000000001},"last":0.92900000000000005}]},{"osd":4,"last update":"Mon May 22 16:05:38 2023","interfaces":[{"interface":"back","average":{"1min":0.95199999999999996,"5min":0.98099999999999998,"15min":0.97399999999999998},"min":{"1min":0.36799999999999999,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":1.357,"5min":2.3479999999999999,"15min":2.3479999999999999},"last":1.0209999999999999},{"interface":"front","average":{"1min":1.0009999999999999,"5min":1.022,"15min":1.03},"min":{"1min":0.39400000000000002,"5min":0.37,"15min":0.37},"max":{"1min":1.2210000000000001,"5min":1.788,"15min":1.788},"last":1.115}]},{"osd":5,"last update":"Mon May 22 16:05:38 2023","interfaces":[{"interface":"back","average":{"1min":1.0069999999999999,"5min":1.0409999999999999,"15min":1.05},"min":{"1min":0.496,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":1.21,"5min":1.8129999999999999,"15min":1.8129999999999999},"last":1.2190000000000001},{"interface":"front","average":{"1min":0.98299999999999998,"5min":1.0169999999999999,"15min":1.0369999999999999},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.3140000000000001,"5min":1.571,"15min":1.571},"last":1.0609999999999999}]},{"osd":6,"last update":"Mon May 22 16:05:38 2023","interfaces":[{"interface":"back","average":{"1min":1.0109999999999999,"5min":1.0680000000000001,"15min":1.0880000000000001},"min":{"1min":0.45600000000000002,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":1.274,"5min":1.76,"15min":1.76},"last":1.29},{"interface":"front","average":{"1min":1.0609999999999999,"5min":1.0820000000000001,"15min":1.099},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":1.4139999999999999,"5min":1.754,"15min":1.754},"last":1.141}]},{"osd":7,"last update":"Mon May 22 16:05:38 2023","interfaces":[{"interface":"back","average":{"1min":1.1020000000000001,"5min":1.111,"15min":1.127},"min":{"1min":0.53800000000000003,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.5049999999999999,"5min":1.673,"15min":1.673},"last":1.1910000000000001},{"interface":"front","average":{"1min":1.1040000000000001,"5min":1.1160000000000001,"15min":1.109},"min":{"1min":0.41099999999999998,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":1.4670000000000001,"5min":2.3079999999999998,"15min":2.3079999999999998},"last":1.3129999999999999}]}]},{"osd":2,"up_from":13,"seq":55834575320,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31400,"kb_used_data":2464,"kb_used_omap":0,"kb_used_meta":28928,"kb_avail":94340440,"statfs":{"total":96636764160,"available":96604610560,"internally_reserved":0,"allocated":2523136,"data_stored":1350406,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":29622272},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 16:05:35 2023","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.629,"15min":0.67000000000000004},"min":{"1min":0.28599999999999998,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.90100000000000002,"5min":1.0549999999999999,"15min":1.1519999999999999},"last":0.67800000000000005},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.71799999999999997,"15min":0.74199999999999999},"min":{"1min":0.35299999999999998,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":1.0309999999999999,"5min":1.1659999999999999,"15min":1.2090000000000001},"last":0.79100000000000004}]},{"osd":1,"last update":"Mon May 22 16:05:35 2023","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.74399999999999999,"15min":0.75700000000000001},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.875,"5min":1.071,"15min":1.099},"last":0.90600000000000003},{"interface":"front","average":{"1min":0.79300000000000004,"5min":0.82099999999999995,"15min":0.82599999999999996},"min":{"1min":0.39500000000000002,"5min":0.311,"15min":0.311},"max":{"1min":1.069,"5min":1.1739999999999999,"15min":1.2589999999999999},"last":1.069}]},{"osd":3,"last update":"Mon May 22 16:05:35 2023","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":0.75600000000000001,"15min":0.75800000000000001},"min":{"1min":0.32100000000000001,"5min":0.315,"15min":0.315},"max":{"1min":0.91400000000000003,"5min":1.3540000000000001,"15min":1.3540000000000001},"last":0.84999999999999998},{"interface":"front","average":{"1min":0.85199999999999998,"5min":0.87,"15min":0.89300000000000002},"min":{"1min":0.41399999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":1.1859999999999999,"5min":1.2949999999999999,"15min":1.2949999999999999},"last":1.02}]},{"osd":4,"last update":"Mon May 22 16:05:35 2023","interfaces":[{"interface":"back","average":{"1min":1.02,"5min":1.024,"15min":1.04},"min":{"1min":0.66100000000000003,"5min":0.441,"15min":0.441},"max":{"1min":1.4590000000000001,"5min":1.516,"15min":1.6539999999999999},"last":1.208},{"interface":"front","average":{"1min":0.996,"5min":1.0089999999999999,"15min":1.04},"min":{"1min":0.55600000000000005,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.419,"5min":1.4299999999999999,"15min":1.538},"last":1.173}]},{"osd":5,"last update":"Mon May 22 16:05:35 2023","interfaces":[{"interface":"back","average":{"1min":1.03,"5min":1.024,"15min":1.0549999999999999},"min":{"1min":0.53600000000000003,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":1.4139999999999999,"5min":1.4139999999999999,"15min":1.605},"last":1.2789999999999999},{"interface":"front","average":{"1min":1.0069999999999999,"5min":1.038,"15min":1.03},"min":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":1.339,"5min":1.534,"15min":1.534},"last":0.95499999999999996}]},{"osd":6,"last update":"Mon May 22 16:05:35 2023","interfaces":[{"interface":"back","average":{"1min":1.095,"5min":1.0589999999999999,"15min":1.0349999999999999},"min":{"1min":0.72699999999999998,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":1.4259999999999999,"5min":1.429,"15min":1.462},"last":1.139},{"interface":"front","average":{"1min":1.127,"5min":1.105,"15min":1.1339999999999999},"min":{"1min":0.59199999999999997,"5min":0.57099999999999995,"15min":0.57099999999999995},"max":{"1min":1.5269999999999999,"5min":1.621,"15min":1.7110000000000001},"last":1.1100000000000001}]},{"osd":7,"last update":"Mon May 22 16:05:35 2023","interfaces":[{"interface":"back","average":{"1min":1.0940000000000001,"5min":1.0860000000000001,"15min":1.1220000000000001},"min":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.5680000000000001,"5min":1.5680000000000001,"15min":1.6399999999999999},"last":1.351},{"interface":"front","average":{"1min":1.0940000000000001,"5min":1.107,"15min":1.1140000000000001},"min":{"1min":0.629,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":1.4610000000000001,"5min":1.673,"15min":1.673},"last":1.248}]}]},{"osd":1,"up_from":13,"seq":55834575320,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37544,"kb_used_data":2464,"kb_used_omap":0,"kb_used_meta":35072,"kb_avail":94334296,"statfs":{"total":96636764160,"available":96598319104,"internally_reserved":0,"allocated":2523136,"data_stored":1350406,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":35913728},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 16:05:58 2023","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.64800000000000002,"15min":0.66200000000000003},"min":{"1min":0.34499999999999997,"5min":0.308,"15min":0.308},"max":{"1min":1.0449999999999999,"5min":1.0449999999999999,"15min":1.329},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.72199999999999998,"15min":0.76300000000000001},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.95499999999999996,"5min":1.25,"15min":1.268},"last":0.67900000000000005}]},{"osd":2,"last update":"Mon May 22 16:05:58 2023","interfaces":[{"interface":"back","average":{"1min":0.70999999999999996,"5min":0.73399999999999999,"15min":0.75600000000000001},"min":{"1min":0.372,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":1.141,"5min":1.141,"15min":1.26},"last":0.73199999999999998},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.76300000000000001,"15min":0.77700000000000002},"min":{"1min":0.46999999999999997,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":1.0249999999999999,"5min":1.127,"15min":1.268},"last":0.63400000000000001}]},{"osd":3,"last update":"Mon May 22 16:05:58 2023","interfaces":[{"interface":"back","average":{"1min":0.78300000000000003,"5min":0.83799999999999997,"15min":0.878},"min":{"1min":0.39900000000000002,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":1.1100000000000001,"5min":1.2829999999999999,"15min":1.2829999999999999},"last":0.91300000000000003},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.80300000000000005,"15min":0.81899999999999995},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":1.2050000000000001,"5min":1.2749999999999999,"15min":1.2749999999999999},"last":0.84699999999999998}]},{"osd":4,"last update":"Mon May 22 16:05:58 2023","interfaces":[{"interface":"back","average":{"1min":0.93899999999999995,"5min":0.96599999999999997,"15min":0.98399999999999999},"min":{"1min":0.58499999999999996,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":1.3500000000000001,"5min":1.542,"15min":1.542},"last":0.88},{"interface":"front","average":{"1min":1.006,"5min":0.997,"15min":1.024},"min":{"1min":0.71499999999999997,"5min":0.434,"15min":0.434},"max":{"1min":1.2729999999999999,"5min":1.327,"15min":1.4650000000000001},"last":0.82599999999999996}]},{"osd":5,"last update":"Mon May 22 16:05:58 2023","interfaces":[{"interface":"back","average":{"1min":1.0309999999999999,"5min":1.0349999999999999,"15min":1.048},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.319,"5min":1.605,"15min":1.605},"last":1.03},{"interface":"front","average":{"1min":1.0589999999999999,"5min":1.0620000000000001,"15min":1.0760000000000001},"min":{"1min":0.75900000000000001,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":1.4370000000000001,"5min":1.4370000000000001,"15min":1.5389999999999999},"last":0.96099999999999997}]},{"osd":6,"last update":"Mon May 22 16:05:58 2023","interfaces":[{"interface":"back","average":{"1min":1.03,"5min":1.0469999999999999,"15min":1.0640000000000001},"min":{"1min":0.56899999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"max":{"1min":1.4770000000000001,"5min":1.4770000000000001,"15min":1.5249999999999999},"last":0.97799999999999998},{"interface":"front","average":{"1min":1.0589999999999999,"5min":1.099,"15min":1.1100000000000001},"min":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"max":{"1min":1.387,"5min":1.4359999999999999,"15min":1.6479999999999999},"last":0.93799999999999994}]},{"osd":7,"last update":"Mon May 22 16:05:58 2023","interfaces":[{"interface":"back","average":{"1min":1.0509999999999999,"5min":1.123,"15min":1.153},"min":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":1.484,"5min":1.7030000000000001,"15min":1.7030000000000001},"last":1.097},{"interface":"front","average":{"1min":1.1339999999999999,"5min":1.131,"15min":1.151},"min":{"1min":0.78400000000000003,"5min":0.73599999999999999,"15min":0.73599999999999999},"max":{"1min":1.4810000000000001,"5min":1.6439999999999999,"15min":1.6439999999999999},"last":1.0089999999999999}]}]},{"osd":0,"up_from":13,"seq":55834575321,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37036,"kb_used_data":3044,"kb_used_omap":0,"kb_used_meta":33984,"kb_avail":94334804,"statfs":{"total":96636764160,"available":96598839296,"internally_reserved":0,"allocated":3117056,"data_stored":1940774,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":34799616},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon May 22 16:06:01 2023","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.72399999999999998,"15min":0.71099999999999997},"min":{"1min":0.42299999999999999,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":1.1140000000000001,"5min":1.296,"15min":1.296},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.72999999999999998,"15min":0.73899999999999999},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.86899999999999999,"5min":1.181,"15min":1.3149999999999999},"last":0.88800000000000001}]},{"osd":2,"last update":"Mon May 22 16:06:01 2023","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.751,"15min":0.76900000000000002},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":1.071,"5min":1.22,"15min":1.284},"last":0.66600000000000004},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.78700000000000003,"15min":0.81299999999999994},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":1.042,"5min":1.323,"15min":1.323},"last":0.746}]},{"osd":3,"last update":"Mon May 22 16:06:01 2023","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.746,"15min":0.77100000000000002},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":1.228,"5min":1.2350000000000001,"15min":1.2350000000000001},"last":0.80600000000000005},{"interface":"front","average":{"1min":0.78300000000000003,"5min":0.85899999999999999,"15min":0.874},"min":{"1min":0.43099999999999999,"5min":0.377,"15min":0.377},"max":{"1min":1.22,"5min":1.2949999999999999,"15min":1.331},"last":0.84699999999999998}]},{"osd":4,"last update":"Mon May 22 16:06:01 2023","interfaces":[{"interface":"back","average":{"1min":0.90200000000000002,"5min":0.94599999999999995,"15min":0.95199999999999996},"min":{"1min":0.64100000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"max":{"1min":1.1819999999999999,"5min":1.264,"15min":1.4059999999999999},"last":0.98299999999999998},{"interface":"front","average":{"1min":0.91500000000000004,"5min":1.042,"15min":1.05},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.1619999999999999,"5min":1.401,"15min":1.482},"last":1.0049999999999999}]},{"osd":5,"last update":"Mon May 22 16:06:01 2023","interfaces":[{"interface":"back","average":{"1min":0.92600000000000005,"5min":1.014,"15min":1.0349999999999999},"min":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":1.1830000000000001,"5min":1.325,"15min":1.6419999999999999},"last":1.1379999999999999},{"interface":"front","average":{"1min":0.90200000000000002,"5min":1.0009999999999999,"15min":1.0109999999999999},"min":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"max":{"1min":1.1859999999999999,"5min":1.413,"15min":1.458},"last":0.93000000000000005}]},{"osd":6,"last update":"Mon May 22 16:06:01 2023","interfaces":[{"interface":"back","average":{"1min":0.98399999999999999,"5min":1.091,"15min":1.1060000000000001},"min":{"1min":0.66400000000000003,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":1.2729999999999999,"5min":1.407,"15min":1.77},"last":0.95499999999999996},{"interface":"front","average":{"1min":0.96399999999999997,"5min":1.093,"15min":1.1040000000000001},"min":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"max":{"1min":1.2589999999999999,"5min":1.5409999999999999,"15min":1.5409999999999999},"last":1.2170000000000001}]},{"osd":7,"last update":"Mon May 22 16:06:01 2023","interfaces":[{"interface":"back","average":{"1min":1.034,"5min":1.1279999999999999,"15min":1.1359999999999999},"min":{"1min":0.69299999999999995,"5min":0.63900000000000001,"15min":0.63900000000000001},"max":{"1min":1.341,"5min":1.6020000000000001,"15min":1.6020000000000001},"last":1.0469999999999999},{"interface":"front","average":{"1min":1.0109999999999999,"5min":1.0780000000000001,"15min":1.095},"min":{"1min":0.68999999999999995,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.2989999999999999,"5min":1.4650000000000001,"15min":1.4990000000000001},"last":1.026}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-05-22T16:06:31.870 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-05-22T15:26:43.943718+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=22, tm_hour=15, tm_min=26, tm_sec=43, tm_wday=0, tm_yday=142, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=22, tm_hour=16, tm_min=6, tm_sec=6, tm_wday=0, tm_yday=142, tm_isdst=0) 2023-05-22T16:06:31.870 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-05-22T16:06:51.873 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-05-22T16:06:52.008 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.002+0000 7f2c6cdd1640 1 Processor -- start 2023-05-22T16:06:52.008 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.002+0000 7f2c6cdd1640 1 -- start start 2023-05-22T16:06:52.009 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.006+0000 7f2c6cdd1640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f2c6806cc80 0x7f2c6806f630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:52.009 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.006+0000 7f2c6cdd1640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2c680afb50 0x7f2c680affb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:52.009 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.006+0000 7f2c6cdd1640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c68071b40 con 0x7f2c68154170 2023-05-22T16:06:52.009 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.006+0000 7f2c6cdd1640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c680ca9f0 con 0x7f2c680afb50 2023-05-22T16:06:52.009 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.006+0000 7f2c6cdd1640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c680ca150 con 0x7f2c6806cc80 2023-05-22T16:06:52.010 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.006+0000 7f2c6659c640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f2c68154170 0x7f2c6806be90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:54636/0 (socket says 172.21.15.16:54636) 2023-05-22T16:06:52.010 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.006+0000 7f2c6659c640 1 -- 172.21.15.16:0/2807589590 learned_addr learned my addr 172.21.15.16:0/2807589590 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-05-22T16:06:52.010 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.006+0000 7f2c66d9d640 1 --2- 172.21.15.16:0/2807589590 >> v2:172.21.15.16:3300/0 conn(0x7f2c680afb50 0x7f2c680affb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:52.011 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.006+0000 7f2c65d9b640 1 --2- 172.21.15.16:0/2807589590 >> v2:172.21.15.62:3300/0 conn(0x7f2c6806cc80 0x7f2c6806f630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:52.011 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.006+0000 7f2c6559a640 1 -- 172.21.15.16:0/2807589590 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3452589733 0 0) 0x7f2c68071b40 con 0x7f2c68154170 2023-05-22T16:06:52.011 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.006+0000 7f2c6559a640 1 -- 172.21.15.16:0/2807589590 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2c50003530 con 0x7f2c68154170 2023-05-22T16:06:52.011 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.006+0000 7f2c66d9d640 1 -- 172.21.15.16:0/2807589590 >> v2:172.21.15.62:3300/0 conn(0x7f2c6806cc80 msgr2=0x7f2c6806f630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:06:52.011 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.006+0000 7f2c66d9d640 1 --2- 172.21.15.16:0/2807589590 >> v2:172.21.15.62:3300/0 conn(0x7f2c6806cc80 0x7f2c6806f630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:52.012 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.006+0000 7f2c66d9d640 1 -- 172.21.15.16:0/2807589590 >> v1:172.21.15.16:6789/0 conn(0x7f2c68154170 legacy=0x7f2c6806be90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:06:52.012 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.006+0000 7f2c66d9d640 1 -- 172.21.15.16:0/2807589590 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c680b04f0 con 0x7f2c680afb50 2023-05-22T16:06:52.013 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.006+0000 7f2c66d9d640 1 --2- 172.21.15.16:0/2807589590 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2c680afb50 0x7f2c680affb0 secure :-1 s=READY pgs=2673 cs=0 l=1 rev1=1 crypto rx=0x7f2c680cbc50 tx=0x7f2c6001c430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7002c8f5cdcc6d3 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:06:52.014 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.006+0000 7f2c6559a640 1 -- 172.21.15.16:0/2807589590 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c600a7070 con 0x7f2c680afb50 2023-05-22T16:06:52.014 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.006+0000 7f2c6559a640 1 -- 172.21.15.16:0/2807589590 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c600ab430 con 0x7f2c680afb50 2023-05-22T16:06:52.014 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.006+0000 7f2c6559a640 1 -- 172.21.15.16:0/2807589590 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c600ab7a0 con 0x7f2c680afb50 2023-05-22T16:06:52.014 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.010+0000 7f2c6cdd1640 1 -- 172.21.15.16:0/2807589590 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2c680afb50 msgr2=0x7f2c680affb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:06:52.015 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.010+0000 7f2c6cdd1640 1 --2- 172.21.15.16:0/2807589590 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2c680afb50 0x7f2c680affb0 secure :-1 s=READY pgs=2673 cs=0 l=1 rev1=1 crypto rx=0x7f2c680cbc50 tx=0x7f2c6001c430 comp rx=0 tx=0).stop 2023-05-22T16:06:52.015 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.010+0000 7f2c6cdd1640 1 -- 172.21.15.16:0/2807589590 shutdown_connections 2023-05-22T16:06:52.015 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.010+0000 7f2c6cdd1640 1 --2- 172.21.15.16:0/2807589590 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2c680afb50 0x7f2c680affb0 unknown :-1 s=CLOSED pgs=2673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:52.015 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.010+0000 7f2c6cdd1640 1 --2- 172.21.15.16:0/2807589590 >> v2:172.21.15.62:3300/0 conn(0x7f2c6806cc80 0x7f2c6806f630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:52.015 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.010+0000 7f2c6cdd1640 1 -- 172.21.15.16:0/2807589590 >> 172.21.15.16:0/2807589590 conn(0x7f2c680cab50 msgr2=0x7f2c680caf50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:06:52.016 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.010+0000 7f2c6cdd1640 1 -- 172.21.15.16:0/2807589590 shutdown_connections 2023-05-22T16:06:52.016 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.010+0000 7f2c6cdd1640 1 -- 172.21.15.16:0/2807589590 wait complete. 2023-05-22T16:06:52.018 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.010+0000 7f2c6cdd1640 1 Processor -- start 2023-05-22T16:06:52.019 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.010+0000 7f2c6cdd1640 1 -- start start 2023-05-22T16:06:52.019 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.010+0000 7f2c6cdd1640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2c6806cc80 0x7f2c68159940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:52.019 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.010+0000 7f2c6cdd1640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2c680afb50 0x7f2c6814d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:52.019 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.010+0000 7f2c6cdd1640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2c68154170 0x7f2c68139f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:52.019 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.010+0000 7f2c6cdd1640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c68137ee0 con 0x7f2c6806cc80 2023-05-22T16:06:52.019 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.010+0000 7f2c6cdd1640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c68150160 con 0x7f2c68154170 2023-05-22T16:06:52.019 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.010+0000 7f2c6cdd1640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c68150440 con 0x7f2c680afb50 2023-05-22T16:06:52.020 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.010+0000 7f2c6659c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2c6806cc80 0x7f2c68159940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:52.020 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.010+0000 7f2c6659c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2c6806cc80 0x7f2c68159940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47050/0 (socket says 172.21.15.16:47050) 2023-05-22T16:06:52.020 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.010+0000 7f2c6659c640 1 -- 172.21.15.16:0/2597895377 learned_addr learned my addr 172.21.15.16:0/2597895377 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:06:52.020 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.010+0000 7f2c65d9b640 1 --2- 172.21.15.16:0/2597895377 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2c680afb50 0x7f2c6814d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:52.020 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.010+0000 7f2c66d9d640 1 --2- 172.21.15.16:0/2597895377 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2c68154170 0x7f2c68139f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:52.020 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.010+0000 7f2c6659c640 1 -- 172.21.15.16:0/2597895377 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2c680afb50 msgr2=0x7f2c6814d5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:06:52.020 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.010+0000 7f2c6659c640 1 --2- 172.21.15.16:0/2597895377 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2c680afb50 0x7f2c6814d5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:52.020 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.010+0000 7f2c6659c640 1 -- 172.21.15.16:0/2597895377 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2c68154170 msgr2=0x7f2c68139f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:06:52.021 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.010+0000 7f2c6659c640 1 --2- 172.21.15.16:0/2597895377 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2c68154170 0x7f2c68139f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:52.021 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.010+0000 7f2c6659c640 1 -- 172.21.15.16:0/2597895377 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c60009420 con 0x7f2c6806cc80 2023-05-22T16:06:52.021 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.010+0000 7f2c6659c640 1 --2- 172.21.15.16:0/2597895377 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2c6806cc80 0x7f2c68159940 secure :-1 s=READY pgs=2674 cs=0 l=1 rev1=1 crypto rx=0x7f2c5800cb40 tx=0x7f2c5800a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:06:52.021 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.010+0000 7f2c56ffd640 1 -- 172.21.15.16:0/2597895377 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c5800aca0 con 0x7f2c6806cc80 2023-05-22T16:06:52.021 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.010+0000 7f2c56ffd640 1 -- 172.21.15.16:0/2597895377 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c5800ae00 con 0x7f2c6806cc80 2023-05-22T16:06:52.021 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.010+0000 7f2c6cdd1640 1 -- 172.21.15.16:0/2597895377 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c6806c5a0 con 0x7f2c6806cc80 2023-05-22T16:06:52.021 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.010+0000 7f2c56ffd640 1 -- 172.21.15.16:0/2597895377 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c58012620 con 0x7f2c6806cc80 2023-05-22T16:06:52.022 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.010+0000 7f2c6cdd1640 1 -- 172.21.15.16:0/2597895377 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c6806b970 con 0x7f2c6806cc80 2023-05-22T16:06:52.022 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.014+0000 7f2c6cdd1640 1 -- 172.21.15.16:0/2597895377 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c68154540 con 0x7f2c6806cc80 2023-05-22T16:06:52.022 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.014+0000 7f2c56ffd640 1 -- 172.21.15.16:0/2597895377 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 100) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f2c58007070 con 0x7f2c6806cc80 2023-05-22T16:06:52.022 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.014+0000 7f2c56ffd640 1 --2- 172.21.15.16:0/2597895377 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f2c480654f0 0x7f2c480679a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:52.022 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.014+0000 7f2c56ffd640 1 -- 172.21.15.16:0/2597895377 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(201..201 src has 1..201) v4 ==== 6317+0+0 (secure 0 0 0) 0x7f2c58082b90 con 0x7f2c6806cc80 2023-05-22T16:06:52.022 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.014+0000 7f2c65d9b640 1 --2- 172.21.15.16:0/2597895377 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f2c480654f0 0x7f2c480679a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:52.022 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.014+0000 7f2c65d9b640 1 --2- 172.21.15.16:0/2597895377 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f2c480654f0 0x7f2c480679a0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f2c680b0100 tx=0x7f2c5c008040 comp rx=0 tx=0).ready entity=mgr.8195 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:06:52.023 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.018+0000 7f2c56ffd640 1 -- 172.21.15.16:0/2597895377 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f2c58050580 con 0x7f2c6806cc80 2023-05-22T16:06:52.173 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.170+0000 7f2c6cdd1640 1 -- 172.21.15.16:0/2597895377 --> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f2c68073830 con 0x7f2c480654f0 2023-05-22T16:06:52.180 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.174+0000 7f2c56ffd640 1 -- 172.21.15.16:0/2597895377 <== mgr.8195 v2:172.21.15.16:6832/65890 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+70926 (secure 0 0 0) 0x7f2c68073830 con 0x7f2c480654f0 2023-05-22T16:06:52.181 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T16:06:52.182 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2023-05-22T16:06:52.183 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.178+0000 7f2c6cdd1640 1 -- 172.21.15.16:0/2597895377 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f2c480654f0 msgr2=0x7f2c480679a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:06:52.183 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.178+0000 7f2c6cdd1640 1 --2- 172.21.15.16:0/2597895377 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f2c480654f0 0x7f2c480679a0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f2c680b0100 tx=0x7f2c5c008040 comp rx=0 tx=0).stop 2023-05-22T16:06:52.183 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.178+0000 7f2c6cdd1640 1 -- 172.21.15.16:0/2597895377 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2c6806cc80 msgr2=0x7f2c68159940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:06:52.183 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.178+0000 7f2c6cdd1640 1 --2- 172.21.15.16:0/2597895377 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2c6806cc80 0x7f2c68159940 secure :-1 s=READY pgs=2674 cs=0 l=1 rev1=1 crypto rx=0x7f2c5800cb40 tx=0x7f2c5800a060 comp rx=0 tx=0).stop 2023-05-22T16:06:52.184 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.178+0000 7f2c6cdd1640 1 -- 172.21.15.16:0/2597895377 shutdown_connections 2023-05-22T16:06:52.184 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.178+0000 7f2c6cdd1640 1 --2- 172.21.15.16:0/2597895377 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2c68154170 0x7f2c68139f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:52.184 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.178+0000 7f2c6cdd1640 1 --2- 172.21.15.16:0/2597895377 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2c680afb50 0x7f2c6814d5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:52.184 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.178+0000 7f2c6cdd1640 1 --2- 172.21.15.16:0/2597895377 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f2c480654f0 0x7f2c480679a0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:52.184 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.178+0000 7f2c6cdd1640 1 --2- 172.21.15.16:0/2597895377 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2c6806cc80 0x7f2c68159940 unknown :-1 s=CLOSED pgs=2674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:52.184 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.178+0000 7f2c6cdd1640 1 -- 172.21.15.16:0/2597895377 >> 172.21.15.16:0/2597895377 conn(0x7f2c680cab50 msgr2=0x7f2c680c3870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:06:52.184 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.178+0000 7f2c6cdd1640 1 -- 172.21.15.16:0/2597895377 shutdown_connections 2023-05-22T16:06:52.185 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.178+0000 7f2c6cdd1640 1 -- 172.21.15.16:0/2597895377 wait complete. 2023-05-22T16:06:52.200 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":34,"stamp":"2023-05-22T16:06:51.814919+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2132,"num_read_kb":4426,"num_write":579,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":349,"ondisk_log_size":349,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":7,"kb":754974720,"kb_used":258760,"kb_used_data":20872,"kb_used_omap":0,"kb_used_meta":237824,"kb_avail":754715960,"statfs":{"total":773094113280,"available":772829143040,"internally_reserved":0,"allocated":21372928,"data_stored":11983984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":243531776},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003328"},"pg_stats":[{"pgid":"1.0","version":"201'349","reported_seq":1175,"reported_epoch":201,"state":"active+clean","last_fresh":"2023-05-22T16:06:14.813007+0000","last_change":"2023-05-22T16:06:14.813307+0000","last_active":"2023-05-22T16:06:14.813007+0000","last_peered":"2023-05-22T16:06:14.813007+0000","last_clean":"2023-05-22T16:06:14.813007+0000","last_became_active":"2023-05-22T15:26:39.315370+0000","last_became_peered":"2023-05-22T15:26:39.315370+0000","last_unstale":"2023-05-22T16:06:14.813007+0000","last_undegraded":"2023-05-22T16:06:14.813007+0000","last_fullsized":"2023-05-22T16:06:14.813007+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"201'349","last_scrub_stamp":"2023-05-22T16:06:14.812951+0000","last_deep_scrub":"201'349","last_deep_scrub_stamp":"2023-05-22T16:06:14.812951+0000","last_clean_scrub_stamp":"2023-05-22T16:06:14.812951+0000","objects_scrubbed":2,"log_size":349,"log_dups_size":0,"ondisk_log_size":349,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T16:08:32.345880+0000","scrub_duration":2.9823071529999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2132,"num_read_kb":4426,"num_write":579,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":215,"reported_epoch":201,"state":"active+clean","last_fresh":"2023-05-22T16:06:27.300799+0000","last_change":"2023-05-22T16:06:27.301083+0000","last_active":"2023-05-22T16:06:27.300799+0000","last_peered":"2023-05-22T16:06:27.300799+0000","last_clean":"2023-05-22T16:06:27.300799+0000","last_became_active":"2023-05-22T15:26:44.970872+0000","last_became_peered":"2023-05-22T15:26:44.970872+0000","last_unstale":"2023-05-22T16:06:27.300799+0000","last_undegraded":"2023-05-22T16:06:27.300799+0000","last_fullsized":"2023-05-22T16:06:27.300799+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:06:27.300747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:06:27.300747+0000","last_clean_scrub_stamp":"2023-05-22T16:06:27.300747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T16:54:34.454270+0000","scrub_duration":2.2360322450000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":203,"reported_epoch":201,"state":"active+clean","last_fresh":"2023-05-22T16:06:17.795279+0000","last_change":"2023-05-22T16:06:17.795555+0000","last_active":"2023-05-22T16:06:17.795279+0000","last_peered":"2023-05-22T16:06:17.795279+0000","last_clean":"2023-05-22T16:06:17.795279+0000","last_became_active":"2023-05-22T15:26:44.976445+0000","last_became_peered":"2023-05-22T15:26:44.976445+0000","last_unstale":"2023-05-22T16:06:17.795279+0000","last_undegraded":"2023-05-22T16:06:17.795279+0000","last_fullsized":"2023-05-22T16:06:17.795279+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:06:17.795228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:06:17.795228+0000","last_clean_scrub_stamp":"2023-05-22T16:06:17.795228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:12:50.953032+0000","scrub_duration":2.2366842309999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":207,"reported_epoch":201,"state":"active+clean","last_fresh":"2023-05-22T16:06:16.858681+0000","last_change":"2023-05-22T16:06:16.858999+0000","last_active":"2023-05-22T16:06:16.858681+0000","last_peered":"2023-05-22T16:06:16.858681+0000","last_clean":"2023-05-22T16:06:16.858681+0000","last_became_active":"2023-05-22T15:26:44.969083+0000","last_became_peered":"2023-05-22T15:26:44.969083+0000","last_unstale":"2023-05-22T16:06:16.858681+0000","last_undegraded":"2023-05-22T16:06:16.858681+0000","last_fullsized":"2023-05-22T16:06:16.858681+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:06:16.858632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:06:16.858632+0000","last_clean_scrub_stamp":"2023-05-22T16:06:16.858632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T21:33:23.760170+0000","scrub_duration":2.2363037530000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":214,"reported_epoch":201,"state":"active+clean","last_fresh":"2023-05-22T16:06:13.411231+0000","last_change":"2023-05-22T16:06:13.411376+0000","last_active":"2023-05-22T16:06:13.411231+0000","last_peered":"2023-05-22T16:06:13.411231+0000","last_clean":"2023-05-22T16:06:13.411231+0000","last_became_active":"2023-05-22T15:26:44.975546+0000","last_became_peered":"2023-05-22T15:26:44.975546+0000","last_unstale":"2023-05-22T16:06:13.411231+0000","last_undegraded":"2023-05-22T16:06:13.411231+0000","last_fullsized":"2023-05-22T16:06:13.411231+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:06:13.411211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:06:13.411211+0000","last_clean_scrub_stamp":"2023-05-22T16:06:13.411211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T03:50:54.501993+0000","scrub_duration":2.2360626940000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":192,"reported_epoch":201,"state":"active+clean","last_fresh":"2023-05-22T16:06:29.286873+0000","last_change":"2023-05-22T16:06:29.286873+0000","last_active":"2023-05-22T16:06:29.286873+0000","last_peered":"2023-05-22T16:06:29.286873+0000","last_clean":"2023-05-22T16:06:29.286873+0000","last_became_active":"2023-05-22T15:26:45.091143+0000","last_became_peered":"2023-05-22T15:26:45.091143+0000","last_unstale":"2023-05-22T16:06:29.286873+0000","last_undegraded":"2023-05-22T16:06:29.286873+0000","last_fullsized":"2023-05-22T16:06:29.286873+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:06:29.286530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:06:29.286530+0000","last_clean_scrub_stamp":"2023-05-22T16:06:29.286530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:57:47.242362+0000","scrub_duration":2.2341401730000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":202,"reported_epoch":201,"state":"active+clean","last_fresh":"2023-05-22T16:06:21.078724+0000","last_change":"2023-05-22T16:06:21.078724+0000","last_active":"2023-05-22T16:06:21.078724+0000","last_peered":"2023-05-22T16:06:21.078724+0000","last_clean":"2023-05-22T16:06:21.078724+0000","last_became_active":"2023-05-22T15:26:45.091951+0000","last_became_peered":"2023-05-22T15:26:45.091951+0000","last_unstale":"2023-05-22T16:06:21.078724+0000","last_undegraded":"2023-05-22T16:06:21.078724+0000","last_fullsized":"2023-05-22T16:06:21.078724+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:06:21.078378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:06:21.078378+0000","last_clean_scrub_stamp":"2023-05-22T16:06:21.078378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2023-05-24T02:28:52.334842+0000","scrub_duration":2.2366510910000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":191,"reported_epoch":201,"state":"active+clean","last_fresh":"2023-05-22T16:06:41.371494+0000","last_change":"2023-05-22T16:06:41.371928+0000","last_active":"2023-05-22T16:06:41.371494+0000","last_peered":"2023-05-22T16:06:41.371494+0000","last_clean":"2023-05-22T16:06:41.371494+0000","last_became_active":"2023-05-22T15:26:44.978443+0000","last_became_peered":"2023-05-22T15:26:44.978443+0000","last_unstale":"2023-05-22T16:06:41.371494+0000","last_undegraded":"2023-05-22T16:06:41.371494+0000","last_fullsized":"2023-05-22T16:06:41.371494+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:06:41.371398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:06:41.371398+0000","last_clean_scrub_stamp":"2023-05-22T16:06:41.371398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T17:31:46.113135+0000","scrub_duration":2.2341064940000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":191,"reported_epoch":201,"state":"active+clean","last_fresh":"2023-05-22T16:06:25.091309+0000","last_change":"2023-05-22T16:06:25.091606+0000","last_active":"2023-05-22T16:06:25.091309+0000","last_peered":"2023-05-22T16:06:25.091309+0000","last_clean":"2023-05-22T16:06:25.091309+0000","last_became_active":"2023-05-22T15:26:44.981586+0000","last_became_peered":"2023-05-22T15:26:44.981586+0000","last_unstale":"2023-05-22T16:06:25.091309+0000","last_undegraded":"2023-05-22T16:06:25.091309+0000","last_fullsized":"2023-05-22T16:06:25.091309+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-22T16:06:25.091260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-22T16:06:25.091260+0000","last_clean_scrub_stamp":"2023-05-22T16:06:25.091260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2023-05-23T19:57:43.047092+0000","scrub_duration":2.235795135,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2132,"num_read_kb":4426,"num_write":579,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":349,"ondisk_log_size":349,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575324,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29676,"kb_used_data":3044,"kb_used_omap":0,"kb_used_meta":26624,"kb_avail":94342164,"statfs":{"total":96636764160,"available":96606375936,"internally_reserved":0,"allocated":3117056,"data_stored":1940774,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27262976},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 16:05:53 2023","interfaces":[{"interface":"back","average":{"1min":0.81999999999999995,"5min":0.85099999999999998,"15min":0.86099999999999999},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":1.222,"5min":1.3069999999999999,"15min":1.3069999999999999},"last":0.99399999999999999},{"interface":"front","average":{"1min":0.84999999999999998,"5min":0.88200000000000001,"15min":0.877},"min":{"1min":0.61299999999999999,"5min":0.501,"15min":0.501},"max":{"1min":1.296,"5min":1.296,"15min":1.296},"last":0.97399999999999998}]},{"osd":1,"last update":"Mon May 22 16:05:53 2023","interfaces":[{"interface":"back","average":{"1min":0.90400000000000003,"5min":0.97299999999999998,"15min":0.98999999999999999},"min":{"1min":0.66000000000000003,"5min":0.63400000000000001,"15min":0.58299999999999996},"max":{"1min":1.274,"5min":1.431,"15min":1.431},"last":1.141},{"interface":"front","average":{"1min":0.95499999999999996,"5min":1.026,"15min":1.026},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":1.2909999999999999,"5min":1.345,"15min":1.345},"last":1.2230000000000001}]},{"osd":2,"last update":"Mon May 22 16:05:53 2023","interfaces":[{"interface":"back","average":{"1min":0.83899999999999997,"5min":0.89900000000000002,"15min":0.90400000000000003},"min":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.171,"5min":1.286,"15min":1.407},"last":1.052},{"interface":"front","average":{"1min":0.89100000000000001,"5min":0.94899999999999995,"15min":0.95899999999999996},"min":{"1min":0.53200000000000003,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":1.3899999999999999,"5min":1.3899999999999999,"15min":1.3899999999999999},"last":0.81100000000000005}]},{"osd":3,"last update":"Mon May 22 16:05:53 2023","interfaces":[{"interface":"back","average":{"1min":0.96399999999999997,"5min":0.93300000000000005,"15min":0.94799999999999995},"min":{"1min":0.755,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.2,"5min":1.2909999999999999,"15min":1.401},"last":1.1200000000000001},{"interface":"front","average":{"1min":0.97199999999999998,"5min":0.98699999999999999,"15min":0.98099999999999998},"min":{"1min":0.60299999999999998,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":1.4179999999999999,"5min":1.4179999999999999,"15min":1.4179999999999999},"last":0.94199999999999995}]},{"osd":4,"last update":"Mon May 22 16:05:53 2023","interfaces":[{"interface":"back","average":{"1min":0.875,"5min":0.88900000000000001,"15min":0.88600000000000001},"min":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51000000000000001},"max":{"1min":1.2310000000000001,"5min":1.3440000000000001,"15min":1.3440000000000001},"last":0.90700000000000003},{"interface":"front","average":{"1min":0.95599999999999996,"5min":0.99099999999999999,"15min":0.96499999999999997},"min":{"1min":0.60899999999999999,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":1.3480000000000001,"5min":1.405,"15min":1.405},"last":0.871}]},{"osd":5,"last update":"Mon May 22 16:05:53 2023","interfaces":[{"interface":"back","average":{"1min":0.96499999999999997,"5min":0.98899999999999999,"15min":0.98999999999999999},"min":{"1min":0.66200000000000003,"5min":0.623,"15min":0.441},"max":{"1min":1.159,"5min":1.2909999999999999,"15min":1.3759999999999999},"last":0.69299999999999995},{"interface":"front","average":{"1min":1.1180000000000001,"5min":1.1339999999999999,"15min":1.135},"min":{"1min":0.91000000000000003,"5min":0.72299999999999998,"15min":0.66600000000000004},"max":{"1min":1.468,"5min":1.468,"15min":1.468},"last":1.095}]},{"osd":6,"last update":"Mon May 22 16:05:53 2023","interfaces":[{"interface":"back","average":{"1min":1.048,"5min":1.0600000000000001,"15min":1.0569999999999999},"min":{"1min":0.72799999999999998,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":1.288,"5min":1.3169999999999999,"15min":1.389},"last":1.0169999999999999},{"interface":"front","average":{"1min":1.008,"5min":1.0940000000000001,"15min":1.0880000000000001},"min":{"1min":0.70399999999999996,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":1.3149999999999999,"5min":1.391,"15min":1.7050000000000001},"last":1.175}]}]},{"osd":6,"up_from":13,"seq":55834575325,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":22824,"kb_used_data":2464,"kb_used_omap":0,"kb_used_meta":20352,"kb_avail":94349016,"statfs":{"total":96636764160,"available":96613392384,"internally_reserved":0,"allocated":2523136,"data_stored":1350406,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":20840448},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 16:05:56 2023","interfaces":[{"interface":"back","average":{"1min":0.85999999999999999,"5min":0.88,"15min":0.89600000000000002},"min":{"1min":0.48599999999999999,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":2.6469999999999998,"5min":2.6469999999999998,"15min":2.6469999999999998},"last":0.80700000000000005},{"interface":"front","average":{"1min":1.254,"5min":0.96899999999999997,"15min":0.92800000000000005},"min":{"1min":0.64600000000000002,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":6.7439999999999998,"5min":6.7439999999999998,"15min":6.7439999999999998},"last":0.66200000000000003}]},{"osd":1,"last update":"Mon May 22 16:05:56 2023","interfaces":[{"interface":"back","average":{"1min":1.0009999999999999,"5min":0.93000000000000005,"15min":0.94199999999999995},"min":{"1min":0.57399999999999995,"5min":0.55300000000000005,"15min":0.51100000000000001},"max":{"1min":2.3290000000000002,"5min":2.3290000000000002,"15min":2.3290000000000002},"last":0.71399999999999997},{"interface":"front","average":{"1min":1.2330000000000001,"5min":0.97899999999999998,"15min":0.94699999999999995},"min":{"1min":0.58199999999999996,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":6.5010000000000003,"5min":6.5010000000000003,"15min":6.5010000000000003},"last":0.76000000000000001}]},{"osd":2,"last update":"Mon May 22 16:05:56 2023","interfaces":[{"interface":"back","average":{"1min":1.2210000000000001,"5min":1.012,"15min":1.0109999999999999},"min":{"1min":0.57399999999999995,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":7.0170000000000003,"5min":7.0170000000000003,"15min":7.0170000000000003},"last":0.84699999999999998},{"interface":"front","average":{"1min":1.0589999999999999,"5min":1.05,"15min":1.0720000000000001},"min":{"1min":0.66200000000000003,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":2.403,"5min":2.403,"15min":2.403},"last":0.93600000000000005}]},{"osd":3,"last update":"Mon May 22 16:05:56 2023","interfaces":[{"interface":"back","average":{"1min":1.278,"5min":1.054,"15min":1.0129999999999999},"min":{"1min":0.63800000000000001,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":6.6699999999999999,"5min":6.6699999999999999,"15min":6.6699999999999999},"last":0.89000000000000001},{"interface":"front","average":{"1min":1.244,"5min":1.02,"15min":0.98899999999999999},"min":{"1min":0.68000000000000005,"5min":0.622,"15min":0.622},"max":{"1min":7.1260000000000003,"5min":7.1260000000000003,"15min":7.1260000000000003},"last":1.1259999999999999}]},{"osd":4,"last update":"Mon May 22 16:05:56 2023","interfaces":[{"interface":"back","average":{"1min":1.161,"5min":0.98099999999999998,"15min":0.96999999999999997},"min":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"max":{"1min":4.0709999999999997,"5min":4.0709999999999997,"15min":4.0709999999999997},"last":1.079},{"interface":"front","average":{"1min":1.583,"5min":1.1619999999999999,"15min":1.075},"min":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":11.581,"5min":11.581,"15min":11.581},"last":1.05}]},{"osd":5,"last update":"Mon May 22 16:05:56 2023","interfaces":[{"interface":"back","average":{"1min":1.925,"5min":1.1739999999999999,"15min":1.056},"min":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":17.587,"5min":17.587,"15min":17.587},"last":1.0069999999999999},{"interface":"front","average":{"1min":1.204,"5min":1.1180000000000001,"15min":1.1240000000000001},"min":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"max":{"1min":2.9620000000000002,"5min":2.9620000000000002,"15min":2.9620000000000002},"last":1.0309999999999999}]},{"osd":7,"last update":"Mon May 22 16:05:56 2023","interfaces":[{"interface":"back","average":{"1min":1.4319999999999999,"5min":1.202,"15min":1.1559999999999999},"min":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.58599999999999997},"max":{"1min":6.8339999999999996,"5min":6.8339999999999996,"15min":6.8339999999999996},"last":0.97999999999999998},{"interface":"front","average":{"1min":1.5429999999999999,"5min":1.2090000000000001,"15min":1.163},"min":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":8.5990000000000002,"5min":8.5990000000000002,"15min":8.5990000000000002},"last":1.3080000000000001}]}]},{"osd":5,"up_from":13,"seq":55834575325,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32936,"kb_used_data":2464,"kb_used_omap":0,"kb_used_meta":30464,"kb_avail":94338904,"statfs":{"total":96636764160,"available":96603037696,"internally_reserved":0,"allocated":2523136,"data_stored":1350406,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31195136},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 16:05:54 2023","interfaces":[{"interface":"back","average":{"1min":0.76000000000000001,"5min":0.85199999999999998,"15min":0.875},"min":{"1min":0.47899999999999998,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.0820000000000001,"5min":1.2749999999999999,"15min":1.3959999999999999},"last":0.74299999999999999},{"interface":"front","average":{"1min":0.86499999999999999,"5min":0.93999999999999995,"15min":0.94299999999999995},"min":{"1min":0.56499999999999995,"5min":0.56200000000000006,"15min":0.502},"max":{"1min":1.165,"5min":1.325,"15min":1.325},"last":0.80500000000000005}]},{"osd":1,"last update":"Mon May 22 16:05:54 2023","interfaces":[{"interface":"back","average":{"1min":0.86599999999999999,"5min":0.95999999999999996,"15min":1.0009999999999999},"min":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.2909999999999999,"5min":1.365,"15min":1.365},"last":1.073},{"interface":"front","average":{"1min":0.89600000000000002,"5min":1.028,"15min":1.0640000000000001},"min":{"1min":0.46899999999999997,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":1.216,"5min":1.3520000000000001,"15min":1.4630000000000001},"last":1.1919999999999999}]},{"osd":2,"last update":"Mon May 22 16:05:54 2023","interfaces":[{"interface":"back","average":{"1min":0.85299999999999998,"5min":0.90500000000000003,"15min":0.92000000000000004},"min":{"1min":0.60899999999999999,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.1859999999999999,"5min":1.3200000000000001,"15min":1.3740000000000001},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.86799999999999999,"5min":0.94699999999999995,"15min":0.94599999999999995},"min":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":1.117,"5min":1.4350000000000001,"15min":1.4350000000000001},"last":0.997}]},{"osd":3,"last update":"Mon May 22 16:05:54 2023","interfaces":[{"interface":"back","average":{"1min":0.91300000000000003,"5min":0.97699999999999998,"15min":0.97599999999999998},"min":{"1min":0.63400000000000001,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":1.1759999999999999,"5min":1.349,"15min":1.367},"last":0.90700000000000003},{"interface":"front","average":{"1min":0.97999999999999998,"5min":1.0329999999999999,"15min":1.048},"min":{"1min":0.67600000000000005,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.1779999999999999,"5min":1.3899999999999999,"15min":1.3899999999999999},"last":1.0389999999999999}]},{"osd":4,"last update":"Mon May 22 16:05:54 2023","interfaces":[{"interface":"back","average":{"1min":0.91200000000000003,"5min":0.93600000000000005,"15min":0.93100000000000005},"min":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.2170000000000001,"5min":1.6000000000000001,"15min":1.6000000000000001},"last":0.85099999999999998},{"interface":"front","average":{"1min":0.98899999999999999,"5min":0.98499999999999999,"15min":1.01},"min":{"1min":0.57899999999999996,"5min":0.498,"15min":0.498},"max":{"1min":1.371,"5min":1.4630000000000001,"15min":1.4630000000000001},"last":1.093}]},{"osd":6,"last update":"Mon May 22 16:05:54 2023","interfaces":[{"interface":"back","average":{"1min":1.054,"5min":1.0129999999999999,"15min":1.0209999999999999},"min":{"1min":0.68999999999999995,"5min":0.63200000000000001,"15min":0.59799999999999998},"max":{"1min":1.2509999999999999,"5min":1.4630000000000001,"15min":1.4790000000000001},"last":0.95599999999999996},{"interface":"front","average":{"1min":1.0449999999999999,"5min":1.073,"15min":1.081},"min":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":1.3440000000000001,"5min":1.6080000000000001,"15min":1.6080000000000001},"last":1.153}]},{"osd":7,"last update":"Mon May 22 16:05:54 2023","interfaces":[{"interface":"back","average":{"1min":1.133,"5min":1.1180000000000001,"15min":1.1259999999999999},"min":{"1min":0.83899999999999997,"5min":0.57699999999999996,"15min":0.57699999999999996},"max":{"1min":1.3680000000000001,"5min":1.51,"15min":1.5109999999999999},"last":1.2490000000000001},{"interface":"front","average":{"1min":1.1739999999999999,"5min":1.1859999999999999,"15min":1.1930000000000001},"min":{"1min":0.91000000000000003,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":1.353,"5min":1.6679999999999999,"15min":1.7210000000000001},"last":1.1240000000000001}]}]},{"osd":4,"up_from":13,"seq":55834575323,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":39656,"kb_used_data":2464,"kb_used_omap":0,"kb_used_meta":37184,"kb_avail":94332184,"statfs":{"total":96636764160,"available":96596156416,"internally_reserved":0,"allocated":2523136,"data_stored":1350406,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":38076416},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 16:06:05 2023","interfaces":[{"interface":"back","average":{"1min":0.88200000000000001,"5min":0.89400000000000002,"15min":0.89800000000000002},"min":{"1min":0.58199999999999996,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.1890000000000001,"5min":1.7210000000000001,"15min":1.7210000000000001},"last":1.121},{"interface":"front","average":{"1min":0.86799999999999999,"5min":0.93899999999999995,"15min":0.94799999999999995},"min":{"1min":0.49099999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":1.1419999999999999,"5min":1.7290000000000001,"15min":1.7290000000000001},"last":0.80700000000000005}]},{"osd":1,"last update":"Mon May 22 16:06:05 2023","interfaces":[{"interface":"back","average":{"1min":0.91300000000000003,"5min":0.95099999999999996,"15min":0.96199999999999997},"min":{"1min":0.60699999999999998,"5min":0.314,"15min":0.314},"max":{"1min":1.242,"5min":1.9339999999999999,"15min":1.9339999999999999},"last":1.095},{"interface":"front","average":{"1min":0.82299999999999995,"5min":0.89600000000000002,"15min":0.93000000000000005},"min":{"1min":0.54000000000000004,"5min":0.504,"15min":0.504},"max":{"1min":1.149,"5min":1.365,"15min":1.365},"last":0.873}]},{"osd":2,"last update":"Mon May 22 16:06:05 2023","interfaces":[{"interface":"back","average":{"1min":0.87,"5min":0.94199999999999995,"15min":0.94399999999999995},"min":{"1min":0.53500000000000003,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":1.1759999999999999,"5min":1.4990000000000001,"15min":1.4990000000000001},"last":0.91100000000000003},{"interface":"front","average":{"1min":0.92200000000000004,"5min":1.0129999999999999,"15min":1.0449999999999999},"min":{"1min":0.60799999999999998,"5min":0.44,"15min":0.44},"max":{"1min":1.1579999999999999,"5min":2.0249999999999999,"15min":2.0249999999999999},"last":1.0429999999999999}]},{"osd":3,"last update":"Mon May 22 16:06:05 2023","interfaces":[{"interface":"back","average":{"1min":0.93999999999999995,"5min":0.95799999999999996,"15min":0.96399999999999997},"min":{"1min":0.61699999999999999,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":1.2929999999999999,"5min":1.452,"15min":1.4610000000000001},"last":1.147},{"interface":"front","average":{"1min":0.97899999999999998,"5min":1.0329999999999999,"15min":1.044},"min":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":1.3140000000000001,"5min":1.349,"15min":1.484},"last":0.84399999999999997}]},{"osd":5,"last update":"Mon May 22 16:06:05 2023","interfaces":[{"interface":"back","average":{"1min":1.032,"5min":1.0289999999999999,"15min":1.0049999999999999},"min":{"1min":0.63600000000000001,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":1.353,"5min":2.1110000000000002,"15min":2.1110000000000002},"last":0.93400000000000005},{"interface":"front","average":{"1min":1.083,"5min":1.018,"15min":1.002},"min":{"1min":0.77300000000000002,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.413,"5min":1.5449999999999999,"15min":1.5449999999999999},"last":1.069}]},{"osd":6,"last update":"Mon May 22 16:06:05 2023","interfaces":[{"interface":"back","average":{"1min":1.0029999999999999,"5min":1.02,"15min":1.006},"min":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"max":{"1min":1.254,"5min":1.397,"15min":1.4339999999999999},"last":1.0129999999999999},{"interface":"front","average":{"1min":1.111,"5min":1.137,"15min":1.1379999999999999},"min":{"1min":0.74399999999999999,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.266,"5min":2.2690000000000001,"15min":2.2690000000000001},"last":0.98999999999999999}]},{"osd":7,"last update":"Mon May 22 16:06:05 2023","interfaces":[{"interface":"back","average":{"1min":1.085,"5min":1.0840000000000001,"15min":1.0720000000000001},"min":{"1min":0.78700000000000003,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":1.4810000000000001,"5min":1.623,"15min":1.623},"last":0.95899999999999996},{"interface":"front","average":{"1min":1.117,"5min":1.1659999999999999,"15min":1.159},"min":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.68100000000000005},"max":{"1min":1.389,"5min":1.448,"15min":1.5089999999999999},"last":1.1759999999999999}]}]},{"osd":3,"up_from":13,"seq":55834575323,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27688,"kb_used_data":2464,"kb_used_omap":0,"kb_used_meta":25216,"kb_avail":94344152,"statfs":{"total":96636764160,"available":96608411648,"internally_reserved":0,"allocated":2523136,"data_stored":1350406,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":25821184},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 16:06:39 2023","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.63100000000000001,"15min":0.63600000000000001},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.92600000000000005,"5min":1.0049999999999999,"15min":1.0229999999999999},"last":0.75800000000000001},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.749,"15min":0.76300000000000001},"min":{"1min":0.36299999999999999,"5min":0.255,"15min":0.255},"max":{"1min":1.1060000000000001,"5min":1.7230000000000001,"15min":1.7230000000000001},"last":0.73799999999999999}]},{"osd":1,"last update":"Mon May 22 16:06:39 2023","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.72999999999999998,"15min":0.71399999999999997},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.96599999999999997,"5min":2.4969999999999999,"15min":2.4969999999999999},"last":0.81100000000000005},{"interface":"front","average":{"1min":0.72799999999999998,"5min":0.79400000000000004,"15min":0.79600000000000004},"min":{"1min":0.32300000000000001,"5min":0.31,"15min":0.31},"max":{"1min":1,"5min":2.4769999999999999,"15min":2.4769999999999999},"last":0.64600000000000002}]},{"osd":2,"last update":"Mon May 22 16:06:39 2023","interfaces":[{"interface":"back","average":{"1min":0.74399999999999999,"5min":0.81399999999999995,"15min":0.83999999999999997},"min":{"1min":0.34300000000000003,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":1.161,"5min":2.2250000000000001,"15min":2.2250000000000001},"last":1.0580000000000001},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.82499999999999996,"15min":0.82599999999999996},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":1.079,"5min":2.1680000000000001,"15min":2.1680000000000001},"last":1.0209999999999999}]},{"osd":4,"last update":"Mon May 22 16:06:39 2023","interfaces":[{"interface":"back","average":{"1min":0.92800000000000005,"5min":0.96299999999999997,"15min":0.96699999999999997},"min":{"1min":0.47599999999999998,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":1.2629999999999999,"5min":2.3479999999999999,"15min":2.3479999999999999},"last":1.194},{"interface":"front","average":{"1min":0.93999999999999995,"5min":1.002,"15min":1.0209999999999999},"min":{"1min":0.58999999999999997,"5min":0.37,"15min":0.37},"max":{"1min":1.2549999999999999,"5min":1.788,"15min":1.788},"last":1.149}]},{"osd":5,"last update":"Mon May 22 16:06:39 2023","interfaces":[{"interface":"back","average":{"1min":0.98799999999999999,"5min":1.0209999999999999,"15min":1.046},"min":{"1min":0.68700000000000006,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":1.363,"5min":1.8129999999999999,"15min":1.8129999999999999},"last":0.877},{"interface":"front","average":{"1min":0.96399999999999997,"5min":1.004,"15min":1.0329999999999999},"min":{"1min":0.53400000000000003,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.331,"5min":1.571,"15min":1.571},"last":1.248}]},{"osd":6,"last update":"Mon May 22 16:06:39 2023","interfaces":[{"interface":"back","average":{"1min":1.042,"5min":1.054,"15min":1.0840000000000001},"min":{"1min":0.625,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":1.593,"5min":1.76,"15min":1.76},"last":0.998},{"interface":"front","average":{"1min":0.96299999999999997,"5min":1.0629999999999999,"15min":1.0880000000000001},"min":{"1min":0.54900000000000004,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":1.4830000000000001,"5min":1.754,"15min":1.754},"last":1.385}]},{"osd":7,"last update":"Mon May 22 16:06:39 2023","interfaces":[{"interface":"back","average":{"1min":1.071,"5min":1.105,"15min":1.1200000000000001},"min":{"1min":0.63900000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.423,"5min":1.673,"15min":1.673},"last":1.468},{"interface":"front","average":{"1min":1.032,"5min":1.1000000000000001,"15min":1.1000000000000001},"min":{"1min":0.53800000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":1.5269999999999999,"5min":2.3079999999999998,"15min":2.3079999999999998},"last":1.3029999999999999}]}]},{"osd":2,"up_from":13,"seq":55834575324,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31400,"kb_used_data":2464,"kb_used_omap":0,"kb_used_meta":28928,"kb_avail":94340440,"statfs":{"total":96636764160,"available":96604610560,"internally_reserved":0,"allocated":2523136,"data_stored":1350406,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":29622272},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 16:06:35 2023","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.626,"15min":0.66300000000000003},"min":{"1min":0.31,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.877,"5min":1.0549999999999999,"15min":1.1519999999999999},"last":0.71899999999999997},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.69499999999999995,"15min":0.73699999999999999},"min":{"1min":0.34499999999999997,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":1.024,"5min":1.1659999999999999,"15min":1.2090000000000001},"last":0.79400000000000004}]},{"osd":1,"last update":"Mon May 22 16:06:35 2023","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.72499999999999998,"15min":0.75},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.97099999999999997,"5min":1.071,"15min":1.099},"last":0.94999999999999996},{"interface":"front","average":{"1min":0.747,"5min":0.79200000000000004,"15min":0.82199999999999995},"min":{"1min":0.373,"5min":0.311,"15min":0.311},"max":{"1min":1.069,"5min":1.099,"15min":1.2589999999999999},"last":0.97999999999999998}]},{"osd":3,"last update":"Mon May 22 16:06:35 2023","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.72799999999999998,"15min":0.751},"min":{"1min":0.35799999999999998,"5min":0.315,"15min":0.315},"max":{"1min":0.84999999999999998,"5min":1.3540000000000001,"15min":1.3540000000000001},"last":1.036},{"interface":"front","average":{"1min":0.84799999999999998,"5min":0.86099999999999999,"15min":0.88800000000000001},"min":{"1min":0.41499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":1.2869999999999999,"5min":1.2949999999999999,"15min":1.2949999999999999},"last":1.01}]},{"osd":4,"last update":"Mon May 22 16:06:35 2023","interfaces":[{"interface":"back","average":{"1min":0.94399999999999995,"5min":1.008,"15min":1.03},"min":{"1min":0.56999999999999995,"5min":0.441,"15min":0.441},"max":{"1min":1.2430000000000001,"5min":1.516,"15min":1.6539999999999999},"last":1.157},{"interface":"front","average":{"1min":1.036,"5min":1.008,"15min":1.0369999999999999},"min":{"1min":0.61699999999999999,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.4219999999999999,"5min":1.4299999999999999,"15min":1.538},"last":1.083}]},{"osd":5,"last update":"Mon May 22 16:06:35 2023","interfaces":[{"interface":"back","average":{"1min":1.101,"5min":1.032,"15min":1.0549999999999999},"min":{"1min":0.53200000000000003,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":1.482,"5min":1.482,"15min":1.605},"last":1.1779999999999999},{"interface":"front","average":{"1min":0.90600000000000003,"5min":1.008,"15min":1.026},"min":{"1min":0.65400000000000003,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":1.0980000000000001,"5min":1.534,"15min":1.534},"last":0.90000000000000002}]},{"osd":6,"last update":"Mon May 22 16:06:35 2023","interfaces":[{"interface":"back","average":{"1min":0.95999999999999996,"5min":1.038,"15min":1.0329999999999999},"min":{"1min":0.623,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":1.218,"5min":1.4259999999999999,"15min":1.462},"last":1.294},{"interface":"front","average":{"1min":1.077,"5min":1.101,"15min":1.1259999999999999},"min":{"1min":0.78000000000000003,"5min":0.57099999999999995,"15min":0.57099999999999995},"max":{"1min":1.415,"5min":1.621,"15min":1.7110000000000001},"last":1.107}]},{"osd":7,"last update":"Mon May 22 16:06:35 2023","interfaces":[{"interface":"back","average":{"1min":1.135,"5min":1.0980000000000001,"15min":1.1220000000000001},"min":{"1min":0.66100000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.4259999999999999,"5min":1.5680000000000001,"15min":1.6399999999999999},"last":1.137},{"interface":"front","average":{"1min":1.018,"5min":1.0840000000000001,"15min":1.107},"min":{"1min":0.63500000000000001,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":1.274,"5min":1.673,"15min":1.673},"last":1.2030000000000001}]}]},{"osd":1,"up_from":13,"seq":55834575324,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37544,"kb_used_data":2464,"kb_used_omap":0,"kb_used_meta":35072,"kb_avail":94334296,"statfs":{"total":96636764160,"available":96598319104,"internally_reserved":0,"allocated":2523136,"data_stored":1350406,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":35913728},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 22 16:05:58 2023","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.64800000000000002,"15min":0.66200000000000003},"min":{"1min":0.34499999999999997,"5min":0.308,"15min":0.308},"max":{"1min":1.0449999999999999,"5min":1.0449999999999999,"15min":1.329},"last":0.82399999999999995},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.72199999999999998,"15min":0.76300000000000001},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.95499999999999996,"5min":1.25,"15min":1.268},"last":0.63600000000000001}]},{"osd":2,"last update":"Mon May 22 16:05:58 2023","interfaces":[{"interface":"back","average":{"1min":0.70999999999999996,"5min":0.73399999999999999,"15min":0.75600000000000001},"min":{"1min":0.372,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":1.141,"5min":1.141,"15min":1.26},"last":1.077},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.76300000000000001,"15min":0.77700000000000002},"min":{"1min":0.46999999999999997,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":1.0249999999999999,"5min":1.127,"15min":1.268},"last":0.72799999999999998}]},{"osd":3,"last update":"Mon May 22 16:05:58 2023","interfaces":[{"interface":"back","average":{"1min":0.78300000000000003,"5min":0.83799999999999997,"15min":0.878},"min":{"1min":0.39900000000000002,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":1.1100000000000001,"5min":1.2829999999999999,"15min":1.2829999999999999},"last":1.03},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.80300000000000005,"15min":0.81899999999999995},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":1.2050000000000001,"5min":1.2749999999999999,"15min":1.2749999999999999},"last":0.76100000000000001}]},{"osd":4,"last update":"Mon May 22 16:05:58 2023","interfaces":[{"interface":"back","average":{"1min":0.93899999999999995,"5min":0.96599999999999997,"15min":0.98399999999999999},"min":{"1min":0.58499999999999996,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":1.3500000000000001,"5min":1.542,"15min":1.542},"last":0.95499999999999996},{"interface":"front","average":{"1min":1.006,"5min":0.997,"15min":1.024},"min":{"1min":0.71499999999999997,"5min":0.434,"15min":0.434},"max":{"1min":1.2729999999999999,"5min":1.327,"15min":1.4650000000000001},"last":1.141}]},{"osd":5,"last update":"Mon May 22 16:05:58 2023","interfaces":[{"interface":"back","average":{"1min":1.0309999999999999,"5min":1.0349999999999999,"15min":1.048},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.319,"5min":1.605,"15min":1.605},"last":1.171},{"interface":"front","average":{"1min":1.0589999999999999,"5min":1.0620000000000001,"15min":1.0760000000000001},"min":{"1min":0.75900000000000001,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":1.4370000000000001,"5min":1.4370000000000001,"15min":1.5389999999999999},"last":1.2709999999999999}]},{"osd":6,"last update":"Mon May 22 16:05:58 2023","interfaces":[{"interface":"back","average":{"1min":1.03,"5min":1.0469999999999999,"15min":1.0640000000000001},"min":{"1min":0.56899999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"max":{"1min":1.4770000000000001,"5min":1.4770000000000001,"15min":1.5249999999999999},"last":1.2310000000000001},{"interface":"front","average":{"1min":1.0589999999999999,"5min":1.099,"15min":1.1100000000000001},"min":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"max":{"1min":1.387,"5min":1.4359999999999999,"15min":1.6479999999999999},"last":1.1120000000000001}]},{"osd":7,"last update":"Mon May 22 16:05:58 2023","interfaces":[{"interface":"back","average":{"1min":1.0509999999999999,"5min":1.123,"15min":1.153},"min":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":1.484,"5min":1.7030000000000001,"15min":1.7030000000000001},"last":0.876},{"interface":"front","average":{"1min":1.1339999999999999,"5min":1.131,"15min":1.151},"min":{"1min":0.78400000000000003,"5min":0.73599999999999999,"15min":0.73599999999999999},"max":{"1min":1.4810000000000001,"5min":1.6439999999999999,"15min":1.6439999999999999},"last":1.1850000000000001}]}]},{"osd":0,"up_from":13,"seq":55834575325,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37036,"kb_used_data":3044,"kb_used_omap":0,"kb_used_meta":33984,"kb_avail":94334804,"statfs":{"total":96636764160,"available":96598839296,"internally_reserved":0,"allocated":3117056,"data_stored":1940774,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":34799616},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon May 22 16:06:01 2023","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.72399999999999998,"15min":0.71099999999999997},"min":{"1min":0.42299999999999999,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":1.1140000000000001,"5min":1.296,"15min":1.296},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.72999999999999998,"15min":0.73899999999999999},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.86899999999999999,"5min":1.181,"15min":1.3149999999999999},"last":0.76600000000000001}]},{"osd":2,"last update":"Mon May 22 16:06:01 2023","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.751,"15min":0.76900000000000002},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":1.071,"5min":1.22,"15min":1.284},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.78700000000000003,"15min":0.81299999999999994},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":1.042,"5min":1.323,"15min":1.323},"last":0.64700000000000002}]},{"osd":3,"last update":"Mon May 22 16:06:01 2023","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.746,"15min":0.77100000000000002},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":1.228,"5min":1.2350000000000001,"15min":1.2350000000000001},"last":0.97599999999999998},{"interface":"front","average":{"1min":0.78300000000000003,"5min":0.85899999999999999,"15min":0.874},"min":{"1min":0.43099999999999999,"5min":0.377,"15min":0.377},"max":{"1min":1.22,"5min":1.2949999999999999,"15min":1.331},"last":0.82699999999999996}]},{"osd":4,"last update":"Mon May 22 16:06:01 2023","interfaces":[{"interface":"back","average":{"1min":0.90200000000000002,"5min":0.94599999999999995,"15min":0.95199999999999996},"min":{"1min":0.64100000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"max":{"1min":1.1819999999999999,"5min":1.264,"15min":1.4059999999999999},"last":0.86699999999999999},{"interface":"front","average":{"1min":0.91500000000000004,"5min":1.042,"15min":1.05},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.1619999999999999,"5min":1.401,"15min":1.482},"last":0.92100000000000004}]},{"osd":5,"last update":"Mon May 22 16:06:01 2023","interfaces":[{"interface":"back","average":{"1min":0.92600000000000005,"5min":1.014,"15min":1.0349999999999999},"min":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":1.1830000000000001,"5min":1.325,"15min":1.6419999999999999},"last":1.0149999999999999},{"interface":"front","average":{"1min":0.90200000000000002,"5min":1.0009999999999999,"15min":1.0109999999999999},"min":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"max":{"1min":1.1859999999999999,"5min":1.413,"15min":1.458},"last":1.0660000000000001}]},{"osd":6,"last update":"Mon May 22 16:06:01 2023","interfaces":[{"interface":"back","average":{"1min":0.98399999999999999,"5min":1.091,"15min":1.1060000000000001},"min":{"1min":0.66400000000000003,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":1.2729999999999999,"5min":1.407,"15min":1.77},"last":1.1399999999999999},{"interface":"front","average":{"1min":0.96399999999999997,"5min":1.093,"15min":1.1040000000000001},"min":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"max":{"1min":1.2589999999999999,"5min":1.5409999999999999,"15min":1.5409999999999999},"last":1.214}]},{"osd":7,"last update":"Mon May 22 16:06:01 2023","interfaces":[{"interface":"back","average":{"1min":1.034,"5min":1.1279999999999999,"15min":1.1359999999999999},"min":{"1min":0.69299999999999995,"5min":0.63900000000000001,"15min":0.63900000000000001},"max":{"1min":1.341,"5min":1.6020000000000001,"15min":1.6020000000000001},"last":1.2909999999999999},{"interface":"front","average":{"1min":1.0109999999999999,"5min":1.0780000000000001,"15min":1.095},"min":{"1min":0.68999999999999995,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.2989999999999999,"5min":1.4650000000000001,"15min":1.4990000000000001},"last":1.1859999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-05-22T16:06:52.203 DEBUG:teuthology.orchestra.run.smithi016:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2023-05-22T16:06:52.317 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.314+0000 7fee275f5640 1 Processor -- start 2023-05-22T16:06:52.317 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.314+0000 7fee275f5640 1 -- start start 2023-05-22T16:06:52.319 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.314+0000 7fee275f5640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fee200ca330 0x7fee20069800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:52.319 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.314+0000 7fee275f5640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fee20069d40 0x7fee2015dd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:52.319 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.314+0000 7fee275f5640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee200cb1d0 con 0x7fee20159a20 2023-05-22T16:06:52.319 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.314+0000 7fee275f5640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee200b4970 con 0x7fee200ca330 2023-05-22T16:06:52.320 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.314+0000 7fee275f5640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee20070900 con 0x7fee20069d40 2023-05-22T16:06:52.320 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.314+0000 7fee24b90640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fee200ca330 0x7fee20069800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:52.320 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.314+0000 7fee24b90640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fee200ca330 0x7fee20069800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47064/0 (socket says 172.21.15.16:47064) 2023-05-22T16:06:52.320 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.314+0000 7fee24b90640 1 -- 172.21.15.16:0/4074612119 learned_addr learned my addr 172.21.15.16:0/4074612119 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:06:52.320 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.314+0000 7fee25b92640 1 --2- 172.21.15.16:0/4074612119 >> v2:172.21.15.62:3300/0 conn(0x7fee20069d40 0x7fee2015dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:52.320 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.314+0000 7fee24b90640 1 -- 172.21.15.16:0/4074612119 >> v2:172.21.15.62:3300/0 conn(0x7fee20069d40 msgr2=0x7fee2015dd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:06:52.320 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.314+0000 7fee24b90640 1 --2- 172.21.15.16:0/4074612119 >> v2:172.21.15.62:3300/0 conn(0x7fee20069d40 0x7fee2015dd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:52.321 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.314+0000 7fee24b90640 1 -- 172.21.15.16:0/4074612119 >> v1:172.21.15.62:6789/0 conn(0x7fee20159a20 legacy=0x7fee200b4260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:06:52.321 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.314+0000 7fee24b90640 1 -- 172.21.15.16:0/4074612119 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee200ca930 con 0x7fee200ca330 2023-05-22T16:06:52.321 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.314+0000 7fee25b92640 1 --2- 172.21.15.16:0/4074612119 >> v2:172.21.15.62:3300/0 conn(0x7fee20069d40 0x7fee2015dd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-05-22T16:06:52.322 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.318+0000 7fee24b90640 1 --2- 172.21.15.16:0/4074612119 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fee200ca330 0x7fee20069800 secure :-1 s=READY pgs=2675 cs=0 l=1 rev1=1 crypto rx=0x7fee2006fe10 tx=0x7fee1001c400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3424da29ee4b46ed server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:06:52.322 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.318+0000 7fee17fff640 1 -- 172.21.15.16:0/4074612119 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee100ab600 con 0x7fee200ca330 2023-05-22T16:06:52.322 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.318+0000 7fee17fff640 1 -- 172.21.15.16:0/4074612119 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee100ab760 con 0x7fee200ca330 2023-05-22T16:06:52.322 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.318+0000 7fee17fff640 1 -- 172.21.15.16:0/4074612119 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee100abaf0 con 0x7fee200ca330 2023-05-22T16:06:52.323 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.318+0000 7fee275f5640 1 -- 172.21.15.16:0/4074612119 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fee200ca330 msgr2=0x7fee20069800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:06:52.323 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.318+0000 7fee275f5640 1 --2- 172.21.15.16:0/4074612119 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fee200ca330 0x7fee20069800 secure :-1 s=READY pgs=2675 cs=0 l=1 rev1=1 crypto rx=0x7fee2006fe10 tx=0x7fee1001c400 comp rx=0 tx=0).stop 2023-05-22T16:06:52.323 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.318+0000 7fee275f5640 1 -- 172.21.15.16:0/4074612119 shutdown_connections 2023-05-22T16:06:52.323 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.318+0000 7fee275f5640 1 --2- 172.21.15.16:0/4074612119 >> v2:172.21.15.62:3300/0 conn(0x7fee20069d40 0x7fee2015dd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:52.323 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.318+0000 7fee275f5640 1 --2- 172.21.15.16:0/4074612119 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fee200ca330 0x7fee20069800 unknown :-1 s=CLOSED pgs=2675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:52.324 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.318+0000 7fee275f5640 1 -- 172.21.15.16:0/4074612119 >> 172.21.15.16:0/4074612119 conn(0x7fee200ce930 msgr2=0x7fee200ced30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:06:52.324 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.318+0000 7fee275f5640 1 -- 172.21.15.16:0/4074612119 shutdown_connections 2023-05-22T16:06:52.324 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.318+0000 7fee275f5640 1 -- 172.21.15.16:0/4074612119 wait complete. 2023-05-22T16:06:52.324 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.318+0000 7fee275f5640 1 Processor -- start 2023-05-22T16:06:52.325 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.322+0000 7fee275f5640 1 -- start start 2023-05-22T16:06:52.326 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.322+0000 7fee275f5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fee20069d40 0x7fee2015b1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:52.326 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.322+0000 7fee275f5640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fee200ca330 0x7fee20151780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:52.327 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.322+0000 7fee275f5640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fee20159a20 0x7fee2014a090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:52.327 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.322+0000 7fee275f5640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee2014d100 con 0x7fee20069d40 2023-05-22T16:06:52.327 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.322+0000 7fee275f5640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee201523a0 con 0x7fee200ca330 2023-05-22T16:06:52.327 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.322+0000 7fee275f5640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee2015a6d0 con 0x7fee20159a20 2023-05-22T16:06:52.327 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.322+0000 7fee25391640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fee20069d40 0x7fee2015b1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:52.327 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.322+0000 7fee25391640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fee20069d40 0x7fee2015b1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47072/0 (socket says 172.21.15.16:47072) 2023-05-22T16:06:52.328 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.322+0000 7fee25391640 1 -- 172.21.15.16:0/1377608844 learned_addr learned my addr 172.21.15.16:0/1377608844 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:06:52.328 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.322+0000 7fee24b90640 1 --2- 172.21.15.16:0/1377608844 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fee200ca330 0x7fee20151780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:52.328 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.322+0000 7fee25391640 1 -- 172.21.15.16:0/1377608844 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fee20159a20 msgr2=0x7fee2014a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:06:52.328 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.322+0000 7fee25391640 1 --2- 172.21.15.16:0/1377608844 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fee20159a20 0x7fee2014a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:52.328 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.322+0000 7fee25391640 1 -- 172.21.15.16:0/1377608844 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fee200ca330 msgr2=0x7fee20151780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:06:52.328 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.322+0000 7fee25391640 1 --2- 172.21.15.16:0/1377608844 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fee200ca330 0x7fee20151780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:52.328 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.322+0000 7fee25391640 1 -- 172.21.15.16:0/1377608844 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee100094d0 con 0x7fee20069d40 2023-05-22T16:06:52.329 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.322+0000 7fee25391640 1 --2- 172.21.15.16:0/1377608844 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fee20069d40 0x7fee2015b1b0 secure :-1 s=READY pgs=2676 cs=0 l=1 rev1=1 crypto rx=0x7fee1c00b410 tx=0x7fee1c00b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:06:52.329 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.322+0000 7fee15ffb640 1 -- 172.21.15.16:0/1377608844 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee1c004020 con 0x7fee20069d40 2023-05-22T16:06:52.329 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.322+0000 7fee15ffb640 1 -- 172.21.15.16:0/1377608844 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee1c004180 con 0x7fee20069d40 2023-05-22T16:06:52.329 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.322+0000 7fee15ffb640 1 -- 172.21.15.16:0/1377608844 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee1c011630 con 0x7fee20069d40 2023-05-22T16:06:52.329 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.322+0000 7fee275f5640 1 -- 172.21.15.16:0/1377608844 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee20159df0 con 0x7fee20069d40 2023-05-22T16:06:52.329 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.322+0000 7fee275f5640 1 -- 172.21.15.16:0/1377608844 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee2015d7e0 con 0x7fee20069d40 2023-05-22T16:06:52.330 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.322+0000 7fee275f5640 1 -- 172.21.15.16:0/1377608844 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee20159f80 con 0x7fee20069d40 2023-05-22T16:06:52.330 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.322+0000 7fee15ffb640 1 -- 172.21.15.16:0/1377608844 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 100) v1 ==== 84332+0+0 (secure 0 0 0) 0x7fee1c011790 con 0x7fee20069d40 2023-05-22T16:06:52.330 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.322+0000 7fee15ffb640 1 --2- 172.21.15.16:0/1377608844 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fee040654a0 0x7fee04067950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:52.330 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.322+0000 7fee15ffb640 1 -- 172.21.15.16:0/1377608844 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(201..201 src has 1..201) v4 ==== 6317+0+0 (secure 0 0 0) 0x7fee1c082840 con 0x7fee20069d40 2023-05-22T16:06:52.331 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.326+0000 7fee24b90640 1 --2- 172.21.15.16:0/1377608844 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fee040654a0 0x7fee04067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:52.331 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.326+0000 7fee24b90640 1 --2- 172.21.15.16:0/1377608844 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fee040654a0 0x7fee04067950 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fee100af000 tx=0x7fee10008040 comp rx=0 tx=0).ready entity=mgr.8195 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:06:52.332 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.326+0000 7fee15ffb640 1 -- 172.21.15.16:0/1377608844 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7fee1c0502b0 con 0x7fee20069d40 2023-05-22T16:06:52.482 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.478+0000 7fee275f5640 1 -- 172.21.15.16:0/1377608844 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) v1 -- 0x7fee20070f20 con 0x7fee20069d40 2023-05-22T16:06:52.490 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.486+0000 7fee15ffb640 1 -- 172.21.15.16:0/1377608844 <== mon.0 v2:172.21.15.16:3300/0 7 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7fee1c0502b0 con 0x7fee20069d40 2023-05-22T16:06:52.490 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.486+0000 7fee15ffb640 1 -- 172.21.15.16:0/1377608844 <== mon.0 v2:172.21.15.16:3300/0 8 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v70)=0 v70) v1 ==== 123+0+0 (secure 0 0 0) 0x7fee1c0538d0 con 0x7fee20069d40 2023-05-22T16:06:52.492 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.486+0000 7fee275f5640 1 -- 172.21.15.16:0/1377608844 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fee040654a0 msgr2=0x7fee04067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:06:52.492 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.486+0000 7fee275f5640 1 --2- 172.21.15.16:0/1377608844 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fee040654a0 0x7fee04067950 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fee100af000 tx=0x7fee10008040 comp rx=0 tx=0).stop 2023-05-22T16:06:52.493 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.486+0000 7fee275f5640 1 -- 172.21.15.16:0/1377608844 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fee20069d40 msgr2=0x7fee2015b1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:06:52.493 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.486+0000 7fee275f5640 1 --2- 172.21.15.16:0/1377608844 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fee20069d40 0x7fee2015b1b0 secure :-1 s=READY pgs=2676 cs=0 l=1 rev1=1 crypto rx=0x7fee1c00b410 tx=0x7fee1c00b8d0 comp rx=0 tx=0).stop 2023-05-22T16:06:52.493 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.486+0000 7fee275f5640 1 -- 172.21.15.16:0/1377608844 shutdown_connections 2023-05-22T16:06:52.493 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.486+0000 7fee275f5640 1 --2- 172.21.15.16:0/1377608844 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fee20159a20 0x7fee2014a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:52.493 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.486+0000 7fee275f5640 1 --2- 172.21.15.16:0/1377608844 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fee200ca330 0x7fee20151780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:52.494 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.486+0000 7fee275f5640 1 --2- 172.21.15.16:0/1377608844 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7fee040654a0 0x7fee04067950 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:52.494 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.486+0000 7fee275f5640 1 --2- 172.21.15.16:0/1377608844 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fee20069d40 0x7fee2015b1b0 unknown :-1 s=CLOSED pgs=2676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:52.494 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.486+0000 7fee275f5640 1 -- 172.21.15.16:0/1377608844 >> 172.21.15.16:0/1377608844 conn(0x7fee200ce930 msgr2=0x7fee200b3bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:06:52.494 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.486+0000 7fee275f5640 1 -- 172.21.15.16:0/1377608844 shutdown_connections 2023-05-22T16:06:52.494 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.486+0000 7fee275f5640 1 -- 172.21.15.16:0/1377608844 wait complete. 2023-05-22T16:06:52.512 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-05-22T16:06:52.513 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-05-22T16:06:52.633 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.630+0000 7f54dc4a9640 1 Processor -- start 2023-05-22T16:06:52.633 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.630+0000 7f54dc4a9640 1 -- start start 2023-05-22T16:06:52.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.630+0000 7f54dc4a9640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f54d406cc80 0x7f54d406f630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:52.635 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.630+0000 7f54dc4a9640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f54d40afb50 0x7f54d40affb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:52.635 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.630+0000 7f54dc4a9640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54d4071b40 con 0x7f54d4154170 2023-05-22T16:06:52.635 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.630+0000 7f54dc4a9640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54d40ca9f0 con 0x7f54d40afb50 2023-05-22T16:06:52.635 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.630+0000 7f54dc4a9640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54d40ca150 con 0x7f54d406cc80 2023-05-22T16:06:52.635 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.630+0000 7f54daa46640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f54d40afb50 0x7f54d40affb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:52.635 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.630+0000 7f54daa46640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f54d40afb50 0x7f54d40affb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47082/0 (socket says 172.21.15.16:47082) 2023-05-22T16:06:52.635 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.630+0000 7f54daa46640 1 -- 172.21.15.16:0/2464448662 learned_addr learned my addr 172.21.15.16:0/2464448662 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:06:52.636 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.630+0000 7f54d9a44640 1 --2- 172.21.15.16:0/2464448662 >> v2:172.21.15.62:3300/0 conn(0x7f54d406cc80 0x7f54d406f630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:52.636 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.630+0000 7f54daa46640 1 -- 172.21.15.16:0/2464448662 >> v2:172.21.15.62:3300/0 conn(0x7f54d406cc80 msgr2=0x7f54d406f630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:06:52.636 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.630+0000 7f54daa46640 1 --2- 172.21.15.16:0/2464448662 >> v2:172.21.15.62:3300/0 conn(0x7f54d406cc80 0x7f54d406f630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:52.636 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.630+0000 7f54daa46640 1 -- 172.21.15.16:0/2464448662 >> v1:172.21.15.62:6789/0 conn(0x7f54d4154170 legacy=0x7f54d406be90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-22T16:06:52.637 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.630+0000 7f54daa46640 1 -- 172.21.15.16:0/2464448662 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54d40b04f0 con 0x7f54d40afb50 2023-05-22T16:06:52.638 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.634+0000 7f54daa46640 1 --2- 172.21.15.16:0/2464448662 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f54d40afb50 0x7f54d40affb0 secure :-1 s=READY pgs=2677 cs=0 l=1 rev1=1 crypto rx=0x7f54d40cbc50 tx=0x7f54cc01c9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d19e2ae287d3ea2 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:06:52.638 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.634+0000 7f54d9243640 1 -- 172.21.15.16:0/2464448662 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54cc0a5860 con 0x7f54d40afb50 2023-05-22T16:06:52.638 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.634+0000 7f54d9243640 1 -- 172.21.15.16:0/2464448662 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f54cc0a59c0 con 0x7f54d40afb50 2023-05-22T16:06:52.638 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.634+0000 7f54d9243640 1 -- 172.21.15.16:0/2464448662 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54cc0a5d70 con 0x7f54d40afb50 2023-05-22T16:06:52.638 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.634+0000 7f54dc4a9640 1 -- 172.21.15.16:0/2464448662 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f54d40afb50 msgr2=0x7f54d40affb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:06:52.639 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.634+0000 7f54dc4a9640 1 --2- 172.21.15.16:0/2464448662 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f54d40afb50 0x7f54d40affb0 secure :-1 s=READY pgs=2677 cs=0 l=1 rev1=1 crypto rx=0x7f54d40cbc50 tx=0x7f54cc01c9a0 comp rx=0 tx=0).stop 2023-05-22T16:06:52.639 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.634+0000 7f54dc4a9640 1 -- 172.21.15.16:0/2464448662 shutdown_connections 2023-05-22T16:06:52.639 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.634+0000 7f54dc4a9640 1 --2- 172.21.15.16:0/2464448662 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f54d40afb50 0x7f54d40affb0 unknown :-1 s=CLOSED pgs=2677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:52.639 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.634+0000 7f54dc4a9640 1 --2- 172.21.15.16:0/2464448662 >> v2:172.21.15.62:3300/0 conn(0x7f54d406cc80 0x7f54d406f630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:52.639 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.634+0000 7f54dc4a9640 1 -- 172.21.15.16:0/2464448662 >> 172.21.15.16:0/2464448662 conn(0x7f54d40cab50 msgr2=0x7f54d40caf50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:06:52.639 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.634+0000 7f54dc4a9640 1 -- 172.21.15.16:0/2464448662 shutdown_connections 2023-05-22T16:06:52.639 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.634+0000 7f54dc4a9640 1 -- 172.21.15.16:0/2464448662 wait complete. 2023-05-22T16:06:52.641 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.634+0000 7f54dc4a9640 1 Processor -- start 2023-05-22T16:06:52.641 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.634+0000 7f54dc4a9640 1 -- start start 2023-05-22T16:06:52.641 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.634+0000 7f54dc4a9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f54d406cc80 0x7f54d414ab10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:52.641 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.634+0000 7f54dc4a9640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f54d40afb50 0x7f54d4151b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:52.641 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.634+0000 7f54dc4a9640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f54d4154170 0x7f54d4149880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:52.641 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.634+0000 7f54dc4a9640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54d414a3c0 con 0x7f54d406cc80 2023-05-22T16:06:52.641 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.634+0000 7f54dc4a9640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54d414b0f0 con 0x7f54d4154170 2023-05-22T16:06:52.642 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.634+0000 7f54dc4a9640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54d414b3d0 con 0x7f54d40afb50 2023-05-22T16:06:52.642 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.634+0000 7f54da245640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f54d406cc80 0x7f54d414ab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:52.642 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.634+0000 7f54da245640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f54d406cc80 0x7f54d414ab10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47098/0 (socket says 172.21.15.16:47098) 2023-05-22T16:06:52.642 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.634+0000 7f54da245640 1 -- 172.21.15.16:0/2039140330 learned_addr learned my addr 172.21.15.16:0/2039140330 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-05-22T16:06:52.642 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.638+0000 7f54daa46640 1 --2- 172.21.15.16:0/2039140330 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f54d4154170 0x7f54d4149880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:52.642 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.638+0000 7f54da245640 1 -- 172.21.15.16:0/2039140330 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f54d40afb50 msgr2=0x7f54d4151b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:06:52.642 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.638+0000 7f54da245640 1 --2- 172.21.15.16:0/2039140330 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f54d40afb50 0x7f54d4151b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:52.643 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.638+0000 7f54da245640 1 -- 172.21.15.16:0/2039140330 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f54d4154170 msgr2=0x7f54d4149880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:06:52.643 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.638+0000 7f54da245640 1 --2- 172.21.15.16:0/2039140330 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f54d4154170 0x7f54d4149880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:52.643 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.638+0000 7f54da245640 1 -- 172.21.15.16:0/2039140330 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54cc009b30 con 0x7f54d406cc80 2023-05-22T16:06:52.643 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.638+0000 7f54da245640 1 --2- 172.21.15.16:0/2039140330 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f54d406cc80 0x7f54d414ab10 secure :-1 s=READY pgs=2678 cs=0 l=1 rev1=1 crypto rx=0x7f54d000c3c0 tx=0x7f54d000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:06:52.643 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.638+0000 7f54caffd640 1 -- 172.21.15.16:0/2039140330 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54d00052d0 con 0x7f54d406cc80 2023-05-22T16:06:52.643 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.638+0000 7f54caffd640 1 -- 172.21.15.16:0/2039140330 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f54d0005430 con 0x7f54d406cc80 2023-05-22T16:06:52.643 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.638+0000 7f54caffd640 1 -- 172.21.15.16:0/2039140330 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54d0003860 con 0x7f54d406cc80 2023-05-22T16:06:52.644 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.638+0000 7f54dc4a9640 1 -- 172.21.15.16:0/2039140330 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54d406c5a0 con 0x7f54d406cc80 2023-05-22T16:06:52.644 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.638+0000 7f54dc4a9640 1 -- 172.21.15.16:0/2039140330 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54d406b970 con 0x7f54d406cc80 2023-05-22T16:06:52.645 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.638+0000 7f54dc4a9640 1 -- 172.21.15.16:0/2039140330 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54d4154540 con 0x7f54d406cc80 2023-05-22T16:06:52.645 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.638+0000 7f54caffd640 1 -- 172.21.15.16:0/2039140330 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 100) v1 ==== 84332+0+0 (secure 0 0 0) 0x7f54d00039c0 con 0x7f54d406cc80 2023-05-22T16:06:52.645 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.638+0000 7f54caffd640 1 --2- 172.21.15.16:0/2039140330 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f54b80654d0 0x7f54b8067980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-22T16:06:52.645 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.638+0000 7f54caffd640 1 -- 172.21.15.16:0/2039140330 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(201..201 src has 1..201) v4 ==== 6317+0+0 (secure 0 0 0) 0x7f54d0082060 con 0x7f54d406cc80 2023-05-22T16:06:52.645 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.638+0000 7f54d9a44640 1 --2- 172.21.15.16:0/2039140330 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f54b80654d0 0x7f54b8067980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-22T16:06:52.645 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.638+0000 7f54d9a44640 1 --2- 172.21.15.16:0/2039140330 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f54b80654d0 0x7f54b8067980 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f54d4153c50 tx=0x7f54c4008040 comp rx=0 tx=0).ready entity=mgr.8195 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-22T16:06:52.647 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.642+0000 7f54caffd640 1 -- 172.21.15.16:0/2039140330 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169485 (secure 0 0 0) 0x7f54d004fa30 con 0x7f54d406cc80 2023-05-22T16:06:52.825 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.822+0000 7f54dc4a9640 1 -- 172.21.15.16:0/2039140330 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f54d4073830 con 0x7f54d406cc80 2023-05-22T16:06:52.826 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.822+0000 7f54caffd640 1 -- 172.21.15.16:0/2039140330 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 83 v83) v1 ==== 94+0+3528 (secure 0 0 0) 0x7f54d0017de0 con 0x7f54d406cc80 2023-05-22T16:06:52.826 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-05-22T16:06:52.827 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":83,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7979,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/1053276484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":1053276484},{"type":"v1","addr":"172.21.15.62:6835","nonce":1053276484}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":76},{"gid":8983,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6833/1372552690","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":1372552690},{"type":"v1","addr":"172.21.15.62:6833","nonce":1372552690}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":76},{"gid":10776,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/982492476","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":982492476},{"type":"v1","addr":"172.21.15.16:6835","nonce":982492476}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":76},{"gid":11310,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/3314899002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":3314899002},{"type":"v1","addr":"172.21.15.16:6837","nonce":3314899002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":83}],"filesystems":[]} 2023-05-22T16:06:52.829 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 83 2023-05-22T16:06:52.829 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.822+0000 7f54dc4a9640 1 -- 172.21.15.16:0/2039140330 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f54b80654d0 msgr2=0x7f54b8067980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:06:52.830 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.822+0000 7f54dc4a9640 1 --2- 172.21.15.16:0/2039140330 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f54b80654d0 0x7f54b8067980 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f54d4153c50 tx=0x7f54c4008040 comp rx=0 tx=0).stop 2023-05-22T16:06:52.830 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.822+0000 7f54dc4a9640 1 -- 172.21.15.16:0/2039140330 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f54d406cc80 msgr2=0x7f54d414ab10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-22T16:06:52.830 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.822+0000 7f54dc4a9640 1 --2- 172.21.15.16:0/2039140330 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f54d406cc80 0x7f54d414ab10 secure :-1 s=READY pgs=2678 cs=0 l=1 rev1=1 crypto rx=0x7f54d000c3c0 tx=0x7f54d000c880 comp rx=0 tx=0).stop 2023-05-22T16:06:52.831 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.822+0000 7f54dc4a9640 1 -- 172.21.15.16:0/2039140330 shutdown_connections 2023-05-22T16:06:52.831 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.822+0000 7f54dc4a9640 1 --2- 172.21.15.16:0/2039140330 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f54d4154170 0x7f54d4149880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:52.831 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.822+0000 7f54dc4a9640 1 --2- 172.21.15.16:0/2039140330 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f54d40afb50 0x7f54d4151b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:52.832 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.822+0000 7f54dc4a9640 1 --2- 172.21.15.16:0/2039140330 >> [v2:172.21.15.16:6832/65890,v1:172.21.15.16:6833/65890] conn(0x7f54b80654d0 0x7f54b8067980 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:52.832 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.822+0000 7f54dc4a9640 1 --2- 172.21.15.16:0/2039140330 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f54d406cc80 0x7f54d414ab10 unknown :-1 s=CLOSED pgs=2678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-22T16:06:52.832 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.822+0000 7f54dc4a9640 1 -- 172.21.15.16:0/2039140330 >> 172.21.15.16:0/2039140330 conn(0x7f54d40cab50 msgr2=0x7f54d40caf20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-22T16:06:52.832 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.822+0000 7f54dc4a9640 1 -- 172.21.15.16:0/2039140330 shutdown_connections 2023-05-22T16:06:52.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-05-22T16:06:52.822+0000 7f54dc4a9640 1 -- 172.21.15.16:0/2039140330 wait complete. 2023-05-22T16:06:52.843 DEBUG:tasks.cephfs.filesystem:already dead... 2023-05-22T16:06:52.843 INFO:teuthology.misc:Shutting down mds daemons... 2023-05-22T16:06:52.844 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-05-22T16:06:52.844 INFO:teuthology.orchestra.run:waiting for 300 2023-05-22T16:06:52.854 INFO:tasks.ceph.mds.a:Stopped 2023-05-22T16:06:52.855 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-05-22T16:06:52.855 INFO:teuthology.orchestra.run:waiting for 300 2023-05-22T16:06:52.906 INFO:tasks.ceph.mds.c:Stopped 2023-05-22T16:06:52.910 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-05-22T16:06:52.910 INFO:teuthology.orchestra.run:waiting for 300 2023-05-22T16:06:52.923 INFO:tasks.ceph.mds.b:Stopped 2023-05-22T16:06:52.924 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-05-22T16:06:52.924 INFO:teuthology.orchestra.run:waiting for 300 2023-05-22T16:06:52.980 INFO:tasks.ceph.mds.d:Stopped 2023-05-22T16:06:52.981 INFO:teuthology.misc:Shutting down osd daemons... 2023-05-22T16:06:52.981 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2023-05-22T16:06:52.981 INFO:teuthology.orchestra.run:waiting for 300 2023-05-22T16:06:53.059 INFO:tasks.ceph.osd.0:Stopped 2023-05-22T16:06:53.060 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2023-05-22T16:06:53.060 INFO:teuthology.orchestra.run:waiting for 300 2023-05-22T16:06:54.934 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-05-22T16:06:54.937 DEBUG:teuthology.orchestra.run.smithi062:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-05-22T16:06:58.226 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~0s 2023-05-22T16:06:59.161 INFO:tasks.ceph.osd.1:Stopped 2023-05-22T16:06:59.162 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2023-05-22T16:06:59.162 INFO:teuthology.orchestra.run:waiting for 300 2023-05-22T16:06:59.238 INFO:tasks.ceph.osd.2:Stopped 2023-05-22T16:06:59.239 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2023-05-22T16:06:59.239 INFO:teuthology.orchestra.run:waiting for 300 2023-05-22T16:06:59.331 INFO:tasks.ceph.osd.3:Stopped 2023-05-22T16:06:59.331 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2023-05-22T16:06:59.331 INFO:teuthology.orchestra.run:waiting for 300 2023-05-22T16:06:59.400 INFO:tasks.ceph.osd.4:Stopped 2023-05-22T16:06:59.401 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2023-05-22T16:06:59.401 INFO:teuthology.orchestra.run:waiting for 300 2023-05-22T16:07:03.933 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~0s 2023-05-22T16:07:03.934 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 has been restored 2023-05-22T16:07:05.503 INFO:tasks.ceph.osd.5:Stopped 2023-05-22T16:07:05.504 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2023-05-22T16:07:05.504 INFO:teuthology.orchestra.run:waiting for 300 2023-05-22T16:07:05.566 INFO:tasks.ceph.osd.6:Stopped 2023-05-22T16:07:05.567 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2023-05-22T16:07:05.567 INFO:teuthology.orchestra.run:waiting for 300 2023-05-22T16:07:09.439 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~0s 2023-05-22T16:07:09.440 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 has been restored 2023-05-22T16:07:11.669 INFO:tasks.ceph.osd.7:Stopped 2023-05-22T16:07:11.670 INFO:teuthology.misc:Shutting down mgr daemons... 2023-05-22T16:07:11.671 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2023-05-22T16:07:11.671 INFO:teuthology.orchestra.run:waiting for 300 2023-05-22T16:07:11.708 INFO:tasks.ceph.mgr.y:Stopped 2023-05-22T16:07:11.709 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2023-05-22T16:07:11.709 INFO:teuthology.orchestra.run:waiting for 300 2023-05-22T16:07:11.739 INFO:tasks.ceph.mgr.x:Stopped 2023-05-22T16:07:11.740 INFO:teuthology.misc:Shutting down mon daemons... 2023-05-22T16:07:11.740 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2023-05-22T16:07:11.741 INFO:teuthology.orchestra.run:waiting for 300 2023-05-22T16:07:11.771 INFO:tasks.ceph.mon.a:Stopped 2023-05-22T16:07:11.771 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2023-05-22T16:07:11.772 INFO:teuthology.orchestra.run:waiting for 300 2023-05-22T16:07:11.809 INFO:tasks.ceph.mon.b:Stopped 2023-05-22T16:07:11.810 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2023-05-22T16:07:11.810 INFO:teuthology.orchestra.run:waiting for 300 2023-05-22T16:07:11.868 INFO:tasks.ceph.mon.c:Stopped 2023-05-22T16:07:11.870 INFO:tasks.ceph:Checking cluster log for badness... 2023-05-22T16:07:11.871 DEBUG:teuthology.orchestra.run.smithi016:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | head -n 1 2023-05-22T16:07:11.887 INFO:teuthology.orchestra.run.smithi016.stdout:1684769373.0744042 mon.a (mon.0) 533 : cluster [WRN] Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY) 2023-05-22T16:07:11.888 WARNING:tasks.ceph:Found errors (ERR|WRN|SEC) in cluster log 2023-05-22T16:07:11.888 DEBUG:teuthology.orchestra.run.smithi016:> sudo egrep '\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | head -n 1 2023-05-22T16:07:11.942 DEBUG:teuthology.orchestra.run.smithi016:> sudo egrep '\[ERR\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | head -n 1 2023-05-22T16:07:11.998 DEBUG:teuthology.orchestra.run.smithi016:> sudo egrep '\[WRN\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | head -n 1 2023-05-22T16:07:12.054 INFO:teuthology.orchestra.run.smithi016.stdout:1684769373.0744042 mon.a (mon.0) 533 : cluster [WRN] Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY) 2023-05-22T16:07:12.055 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi016.front.sepia.ceph.com 2023-05-22T16:07:12.055 DEBUG:teuthology.orchestra.run.smithi016:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2023-05-22T16:07:12.283 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi016.front.sepia.ceph.com 2023-05-22T16:07:12.283 DEBUG:teuthology.orchestra.run.smithi016:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2023-05-22T16:07:12.512 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi016.front.sepia.ceph.com 2023-05-22T16:07:12.512 DEBUG:teuthology.orchestra.run.smithi016:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2023-05-22T16:07:12.655 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi016.front.sepia.ceph.com 2023-05-22T16:07:12.655 DEBUG:teuthology.orchestra.run.smithi016:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2023-05-22T16:07:12.789 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi062.front.sepia.ceph.com 2023-05-22T16:07:12.790 DEBUG:teuthology.orchestra.run.smithi062:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2023-05-22T16:07:13.055 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi062.front.sepia.ceph.com 2023-05-22T16:07:13.055 DEBUG:teuthology.orchestra.run.smithi062:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2023-05-22T16:07:13.248 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi062.front.sepia.ceph.com 2023-05-22T16:07:13.249 DEBUG:teuthology.orchestra.run.smithi062:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2023-05-22T16:07:13.389 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi062.front.sepia.ceph.com 2023-05-22T16:07:13.389 DEBUG:teuthology.orchestra.run.smithi062:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2023-05-22T16:07:13.538 INFO:tasks.ceph:Archiving mon data... 2023-05-22T16:07:13.540 DEBUG:teuthology.misc:Transferring archived files from smithi016:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/yuriw-2023-05-22_14:44:12-fs-wip-yuri3-testing-2023-05-21-0740-reef-distro-default-smithi/7282496/data/mon.a.tgz 2023-05-22T16:07:13.540 DEBUG:teuthology.orchestra.run.smithi016:> mktemp 2023-05-22T16:07:13.546 INFO:teuthology.orchestra.run.smithi016.stdout:/tmp/tmp.LjloeHDp5r 2023-05-22T16:07:13.546 DEBUG:teuthology.orchestra.run.smithi016:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.LjloeHDp5r 2023-05-22T16:07:13.978 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod 0666 /tmp/tmp.LjloeHDp5r 2023-05-22T16:07:14.041 DEBUG:teuthology.orchestra.remote:smithi016:/tmp/tmp.LjloeHDp5r is 2MB 2023-05-22T16:07:14.124 DEBUG:teuthology.orchestra.run.smithi016:> rm -fr /tmp/tmp.LjloeHDp5r 2023-05-22T16:07:14.131 DEBUG:teuthology.misc:Transferring archived files from smithi062:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/yuriw-2023-05-22_14:44:12-fs-wip-yuri3-testing-2023-05-21-0740-reef-distro-default-smithi/7282496/data/mon.b.tgz 2023-05-22T16:07:14.131 DEBUG:teuthology.orchestra.run.smithi062:> mktemp 2023-05-22T16:07:14.137 INFO:teuthology.orchestra.run.smithi062.stdout:/tmp/tmp.XM34r3bIs4 2023-05-22T16:07:14.138 DEBUG:teuthology.orchestra.run.smithi062:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.XM34r3bIs4 2023-05-22T16:07:14.441 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 has been restored 2023-05-22T16:07:14.573 DEBUG:teuthology.orchestra.run.smithi062:> sudo chmod 0666 /tmp/tmp.XM34r3bIs4 2023-05-22T16:07:14.639 DEBUG:teuthology.orchestra.remote:smithi062:/tmp/tmp.XM34r3bIs4 is 2MB 2023-05-22T16:07:14.717 DEBUG:teuthology.orchestra.run.smithi062:> rm -fr /tmp/tmp.XM34r3bIs4 2023-05-22T16:07:14.724 DEBUG:teuthology.misc:Transferring archived files from smithi062:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/yuriw-2023-05-22_14:44:12-fs-wip-yuri3-testing-2023-05-21-0740-reef-distro-default-smithi/7282496/data/mon.c.tgz 2023-05-22T16:07:14.725 DEBUG:teuthology.orchestra.run.smithi062:> mktemp 2023-05-22T16:07:14.771 INFO:teuthology.orchestra.run.smithi062.stdout:/tmp/tmp.zC3Mmw1KZR 2023-05-22T16:07:14.771 DEBUG:teuthology.orchestra.run.smithi062:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.zC3Mmw1KZR 2023-05-22T16:07:15.210 DEBUG:teuthology.orchestra.run.smithi062:> sudo chmod 0666 /tmp/tmp.zC3Mmw1KZR 2023-05-22T16:07:15.325 DEBUG:teuthology.orchestra.remote:smithi062:/tmp/tmp.zC3Mmw1KZR is 2MB 2023-05-22T16:07:15.419 DEBUG:teuthology.orchestra.run.smithi062:> rm -fr /tmp/tmp.zC3Mmw1KZR 2023-05-22T16:07:15.426 INFO:tasks.ceph:Cleaning ceph cluster... 2023-05-22T16:07:15.427 DEBUG:teuthology.orchestra.run.smithi016:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-05-22T16:07:15.429 DEBUG:teuthology.orchestra.run.smithi062:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-05-22T16:07:15.483 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2023-05-22T16:07:15.484 DEBUG:teuthology.orchestra.run.smithi016:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-05-22T16:07:15.486 DEBUG:teuthology.orchestra.run.smithi062:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-05-22T16:07:15.510 INFO:teuthology.orchestra.run.smithi016.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-05-22T16:07:15.545 INFO:teuthology.orchestra.run.smithi062.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-05-22T16:07:15.551 INFO:tasks.ceph:Archiving crash dumps... 2023-05-22T16:07:15.553 DEBUG:teuthology.misc:Transferring archived files from smithi016:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2023-05-22_14:44:12-fs-wip-yuri3-testing-2023-05-21-0740-reef-distro-default-smithi/7282496/remote/smithi016/crash 2023-05-22T16:07:15.554 DEBUG:teuthology.orchestra.run.smithi016:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-05-22T16:07:15.575 DEBUG:teuthology.misc:Transferring archived files from smithi062:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2023-05-22_14:44:12-fs-wip-yuri3-testing-2023-05-21-0740-reef-distro-default-smithi/7282496/remote/smithi062/crash 2023-05-22T16:07:15.576 DEBUG:teuthology.orchestra.run.smithi062:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-05-22T16:07:15.612 INFO:tasks.ceph:Shutting down logrotate 2023-05-22T16:07:15.614 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-05-22T16:07:15.623 DEBUG:teuthology.orchestra.run.smithi062:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-05-22T16:07:15.701 DEBUG:teuthology.orchestra.run.smithi016:> sudo rm /etc/logrotate.d/ceph-test.conf 2023-05-22T16:07:15.715 DEBUG:teuthology.orchestra.run.smithi062:> sudo rm /etc/logrotate.d/ceph-test.conf 2023-05-22T16:07:15.726 INFO:tasks.ceph:Compressing logs... 2023-05-22T16:07:15.726 DEBUG:teuthology.orchestra.run.smithi016:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-05-22T16:07:15.758 DEBUG:teuthology.orchestra.run.smithi062:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-05-22T16:07:35.983 INFO:tasks.ceph:Archiving logs... 2023-05-22T16:07:35.983 DEBUG:teuthology.misc:Transferring archived files from smithi016:/var/log/ceph to /home/teuthworker/archive/yuriw-2023-05-22_14:44:12-fs-wip-yuri3-testing-2023-05-21-0740-reef-distro-default-smithi/7282496/remote/smithi016/log 2023-05-22T16:07:35.985 DEBUG:teuthology.orchestra.run.smithi016:> sudo tar c -f - -C /var/log/ceph -- . 2023-05-22T16:07:38.508 DEBUG:teuthology.misc:Transferring archived files from smithi062:/var/log/ceph to /home/teuthworker/archive/yuriw-2023-05-22_14:44:12-fs-wip-yuri3-testing-2023-05-21-0740-reef-distro-default-smithi/7282496/remote/smithi062/log 2023-05-22T16:07:38.510 DEBUG:teuthology.orchestra.run.smithi062:> sudo tar c -f - -C /var/log/ceph -- . 2023-05-22T16:07:40.014 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-05-22T16:07:40.029 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits... 2023-05-22T16:07:40.029 DEBUG:teuthology.orchestra.run.smithi016:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits 2023-05-22T16:07:40.032 DEBUG:teuthology.orchestra.run.smithi062:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits 2023-05-22T16:07:40.115 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-05-22T16:07:40.140 INFO:teuthology.task.clock:Checking final clock skew... 2023-05-22T16:07:40.140 DEBUG:teuthology.orchestra.run.smithi016:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-05-22T16:07:40.143 DEBUG:teuthology.orchestra.run.smithi062:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-05-22T16:07:40.154 INFO:teuthology.orchestra.run.smithi016.stdout: remote refid st t when poll reach delay offset jitter 2023-05-22T16:07:40.154 INFO:teuthology.orchestra.run.smithi016.stdout:============================================================================== 2023-05-22T16:07:40.155 INFO:teuthology.orchestra.run.smithi016.stdout:*hv01.front.sepi 44.4.53.2 2 u 109 128 377 0.197 -4.944 1.488 2023-05-22T16:07:40.155 INFO:teuthology.orchestra.run.smithi016.stdout:+hv02.front.sepi 63.231.80.2 3 u 26 128 377 0.169 +0.277 1.380 2023-05-22T16:07:40.155 INFO:teuthology.orchestra.run.smithi016.stdout:+hv03.front.sepi 74.6.168.72 3 u 16 128 377 0.182 -7.636 1.130 2023-05-22T16:07:40.155 INFO:teuthology.orchestra.run.smithi016.stdout: hv04.front.sepi .INIT. 16 u - 1024 0 0.000 +0.000 0.000 2023-05-22T16:07:40.158 INFO:teuthology.orchestra.run.smithi062.stdout: remote refid st t when poll reach delay offset jitter 2023-05-22T16:07:40.158 INFO:teuthology.orchestra.run.smithi062.stdout:============================================================================== 2023-05-22T16:07:40.158 INFO:teuthology.orchestra.run.smithi062.stdout:*hv01.front.sepi 44.4.53.2 2 u 109 128 377 0.102 -2.516 0.323 2023-05-22T16:07:40.159 INFO:teuthology.orchestra.run.smithi062.stdout:+hv02.front.sepi 63.231.80.2 3 u 41 128 377 0.102 +1.084 0.136 2023-05-22T16:07:40.159 INFO:teuthology.orchestra.run.smithi062.stdout:+hv03.front.sepi 74.6.168.72 3 u 31 128 377 0.090 -5.320 0.216 2023-05-22T16:07:40.159 INFO:teuthology.orchestra.run.smithi062.stdout: hv04.front.sepi .INIT. 16 u - 1024 0 0.000 +0.000 0.000 2023-05-22T16:07:40.160 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-05-22T16:07:40.171 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-05-22T16:07:40.172 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-05-22T16:07:40.183 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-05-22T16:07:40.196 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-05-22T16:07:40.214 INFO:teuthology.task.internal:Duration was 3123.244646 seconds 2023-05-22T16:07:40.215 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-05-22T16:07:40.226 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-05-22T16:07:40.227 DEBUG:teuthology.orchestra.run.smithi016:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-05-22T16:07:40.232 DEBUG:teuthology.orchestra.run.smithi062:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-05-22T16:07:40.314 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-05-22T16:07:40.314 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi016.front.sepia.ceph.com 2023-05-22T16:07:40.315 DEBUG:teuthology.orchestra.run.smithi016:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-05-22T16:07:40.326 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi062.front.sepia.ceph.com 2023-05-22T16:07:40.327 DEBUG:teuthology.orchestra.run.smithi062:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-05-22T16:07:40.340 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-05-22T16:07:40.340 DEBUG:teuthology.orchestra.run.smithi016:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-05-22T16:07:40.374 DEBUG:teuthology.orchestra.run.smithi062:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-05-22T16:07:40.404 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-05-22T16:07:40.405 DEBUG:teuthology.orchestra.run.smithi016:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-05-22T16:07:40.447 DEBUG:teuthology.orchestra.run.smithi062:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-05-22T16:07:40.527 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-05-22T16:07:40.543 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-05-22T16:07:40.544 DEBUG:teuthology.orchestra.run.smithi016:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-05-22T16:07:40.559 DEBUG:teuthology.orchestra.run.smithi062:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-05-22T16:07:40.571 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-05-22T16:07:40.584 DEBUG:teuthology.orchestra.run.smithi016:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-05-22T16:07:40.602 DEBUG:teuthology.orchestra.run.smithi062:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-05-22T16:07:40.611 INFO:teuthology.orchestra.run.smithi016.stdout:kernel.core_pattern = core 2023-05-22T16:07:40.624 INFO:teuthology.orchestra.run.smithi062.stdout:kernel.core_pattern = core 2023-05-22T16:07:40.635 DEBUG:teuthology.orchestra.run.smithi016:> test -e /home/ubuntu/cephtest/archive/coredump 2023-05-22T16:07:40.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-05-22T16:07:40.673 DEBUG:teuthology.orchestra.run.smithi062:> test -e /home/ubuntu/cephtest/archive/coredump 2023-05-22T16:07:40.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-05-22T16:07:40.683 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-05-22T16:07:40.700 INFO:teuthology.task.internal:Transferring archived files... 2023-05-22T16:07:40.701 DEBUG:teuthology.misc:Transferring archived files from smithi016:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-05-22_14:44:12-fs-wip-yuri3-testing-2023-05-21-0740-reef-distro-default-smithi/7282496/remote/smithi016 2023-05-22T16:07:40.702 DEBUG:teuthology.orchestra.run.smithi016:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-05-22T16:07:40.736 DEBUG:teuthology.misc:Transferring archived files from smithi062:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-05-22_14:44:12-fs-wip-yuri3-testing-2023-05-21-0740-reef-distro-default-smithi/7282496/remote/smithi062 2023-05-22T16:07:40.737 DEBUG:teuthology.orchestra.run.smithi062:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-05-22T16:07:40.757 INFO:teuthology.task.internal:Removing archive directory... 2023-05-22T16:07:40.757 DEBUG:teuthology.orchestra.run.smithi016:> rm -rf -- /home/ubuntu/cephtest/archive 2023-05-22T16:07:40.785 DEBUG:teuthology.orchestra.run.smithi062:> rm -rf -- /home/ubuntu/cephtest/archive 2023-05-22T16:07:40.799 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-05-22T16:07:40.819 INFO:teuthology.task.internal:Not uploading archives. 2023-05-22T16:07:40.820 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-05-22T16:07:40.834 INFO:teuthology.task.internal:Tidying up after the test... 2023-05-22T16:07:40.834 DEBUG:teuthology.orchestra.run.smithi016:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-05-22T16:07:40.837 DEBUG:teuthology.orchestra.run.smithi062:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-05-22T16:07:40.840 INFO:teuthology.orchestra.run.smithi016.stdout: 655472 4 drwxr-xr-x 2 ubuntu ubuntu 4096 May 22 16:07 /home/ubuntu/cephtest 2023-05-22T16:07:40.847 INFO:teuthology.orchestra.run.smithi062.stdout: 655477 4 drwxr-xr-x 2 ubuntu ubuntu 4096 May 22 16:07 /home/ubuntu/cephtest 2023-05-22T16:07:40.849 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-05-22T16:07:40.862 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-05-22T16:07:40.889 INFO:teuthology.nuke:Checking targets against current locks 2023-05-22T16:07:40.919 DEBUG:teuthology.nuke:shortname: smithi016 2023-05-22T16:07:40.920 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-05-22T16:07:40.947 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi016.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-05-22_14:44:12-fs-wip-yuri3-testing-2023-05-21-0740-reef-distro-default-smithi/7282496', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-05-22 15:07:05.114694', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLlVQ8mnKu7mabeHXD6q1SCUofl2/7TQGJtZoDkWRaEN8SjRlNYMe5o9bDtiZPZ2Z7+Mb61B2A6l71rddBvoLjg='} 2023-05-22T16:07:40.951 DEBUG:teuthology.nuke:shortname: smithi062 2023-05-22T16:07:40.952 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-05-22T16:07:40.979 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi062.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-05-22_14:44:12-fs-wip-yuri3-testing-2023-05-21-0740-reef-distro-default-smithi/7282496', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-05-22 15:07:05.113244', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIA1xuKgcCm2/BCZTETfSU7EAds5jWkUYadR3nrazn66o'} 2023-05-22T16:07:41.009 INFO:teuthology.orchestra.console:Power off smithi016 2023-05-22T16:07:41.011 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi016.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-05-22T16:07:41.056 INFO:teuthology.orchestra.console:Power off smithi062 2023-05-22T16:07:41.057 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi062.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-05-22T16:07:41.083 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi062.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-05-22T16:07:41.094 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi016.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-05-22T16:07:45.110 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi062.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-05-22T16:07:45.121 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi016.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-05-22T16:07:49.228 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi062.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-05-22T16:07:49.239 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi016.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-05-22T16:07:49.459 INFO:teuthology.orchestra.console:Power off for smithi016 completed 2023-05-22T16:07:53.344 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi062.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-05-22T16:07:53.555 INFO:teuthology.orchestra.console:Power off for smithi062 completed 2023-05-22T16:07:53.666 INFO:teuthology.run:Summary data: description: fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snap-schedule} duration: 3123.2446463108063 failure_reason: '"1684769373.0744042 mon.a (mon.0) 533 : cluster [WRN] Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY)" in cluster log' flavor: default owner: scheduled_yuriw@teuthology success: false 2023-05-22T16:07:53.667 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-05-22T16:07:53.763 INFO:teuthology.run:FAIL